2024-03-06T07:09:54.834 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-06T07:09:54.835 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_leonidus@teuthology --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays} --archive /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583062 --verbose --name leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi -- /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583062/orig.config.yaml 2024-03-06T07:09:54.878 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-06T07:09:54.964 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583062 branch: wip-lusov-qdb description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays} email: null first_in_suite: false job_id: '7583062' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lusov-qdb ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd subvolume_version: 2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-qdb sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd owner: scheduled_leonidus@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 5072 sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd sleep_before_teardown: 0 subset: 246/10000 suite: fs suite_branch: wip-lusov-qdb suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi083.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE1mL0BgMoVtpCUXipFaslZ37m3Ygh93+0MQR210oMZBMyGW4vvH8MwfKedU06uBiOY5eGNbiCMsbp+TQAsy7ws= smithi183.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAKTQjnjlPE3geR+z/SVB8x392UFEtbZMHxHz917JIHBT0aUi9su9UZc5NVIVWcumwM6TE0zNyML9E9hoUUTpSc= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - ceph-fuse: null - cephfs_test_runner: modules: - tasks.cephfs.test_strays teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-06_07:04:29 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-06T07:09:54.967 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa; will attempt to use it 2024-03-06T07:09:54.968 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks 2024-03-06T07:09:54.968 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-06T07:09:54.969 INFO:teuthology.task.internal:Checking packages... 2024-03-06T07:09:54.994 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd' 2024-03-06T07:09:54.995 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-06T07:09:54.995 INFO:teuthology.packaging:ref: None 2024-03-06T07:09:54.995 INFO:teuthology.packaging:tag: None 2024-03-06T07:09:54.995 INFO:teuthology.packaging:branch: wip-lusov-qdb 2024-03-06T07:09:54.995 INFO:teuthology.packaging:sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:09:54.995 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lusov-qdb 2024-03-06T07:09:55.283 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1739.g1bc250c7 2024-03-06T07:09:55.285 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-06T07:09:55.291 INFO:teuthology.task.internal:no buildpackages task found 2024-03-06T07:09:55.291 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-06T07:09:55.319 INFO:teuthology.task.internal:Saving configuration 2024-03-06T07:09:55.335 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-06T07:09:55.356 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-06T07:09:55.381 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi083.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583062', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-06 07:04:41.435642', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE1mL0BgMoVtpCUXipFaslZ37m3Ygh93+0MQR210oMZBMyGW4vvH8MwfKedU06uBiOY5eGNbiCMsbp+TQAsy7ws='} 2024-03-06T07:09:55.399 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi183.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583062', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-06 07:04:41.433984', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAKTQjnjlPE3geR+z/SVB8x392UFEtbZMHxHz917JIHBT0aUi9su9UZc5NVIVWcumwM6TE0zNyML9E9hoUUTpSc='} 2024-03-06T07:09:55.399 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-06T07:09:55.404 INFO:teuthology.task.internal:roles: ubuntu@smithi083.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-03-06T07:09:55.404 INFO:teuthology.task.internal:roles: ubuntu@smithi183.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-03-06T07:09:55.404 INFO:teuthology.run_tasks:Running task console_log... 2024-03-06T07:09:55.535 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6921ea2f70>, signals=[15]) 2024-03-06T07:09:55.536 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-06T07:09:55.541 INFO:teuthology.task.internal:Opening connections... 2024-03-06T07:09:55.541 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi083.front.sepia.ceph.com 2024-03-06T07:09:55.543 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:09:55.632 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi183.front.sepia.ceph.com 2024-03-06T07:09:55.633 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:09:55.718 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-06T07:09:55.724 DEBUG:teuthology.orchestra.run.smithi083:> uname -m 2024-03-06T07:09:55.740 INFO:teuthology.orchestra.run.smithi083.stdout:x86_64 2024-03-06T07:09:55.740 DEBUG:teuthology.orchestra.run.smithi083:> cat /etc/os-release 2024-03-06T07:09:55.796 INFO:teuthology.orchestra.run.smithi083.stdout:NAME="CentOS Stream" 2024-03-06T07:09:55.796 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION="9" 2024-03-06T07:09:55.796 INFO:teuthology.orchestra.run.smithi083.stdout:ID="centos" 2024-03-06T07:09:55.796 INFO:teuthology.orchestra.run.smithi083.stdout:ID_LIKE="rhel fedora" 2024-03-06T07:09:55.796 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION_ID="9" 2024-03-06T07:09:55.796 INFO:teuthology.orchestra.run.smithi083.stdout:PLATFORM_ID="platform:el9" 2024-03-06T07:09:55.796 INFO:teuthology.orchestra.run.smithi083.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-06T07:09:55.796 INFO:teuthology.orchestra.run.smithi083.stdout:ANSI_COLOR="0;31" 2024-03-06T07:09:55.796 INFO:teuthology.orchestra.run.smithi083.stdout:LOGO="fedora-logo-icon" 2024-03-06T07:09:55.796 INFO:teuthology.orchestra.run.smithi083.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-06T07:09:55.797 INFO:teuthology.orchestra.run.smithi083.stdout:HOME_URL="https://centos.org/" 2024-03-06T07:09:55.797 INFO:teuthology.orchestra.run.smithi083.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-06T07:09:55.797 INFO:teuthology.orchestra.run.smithi083.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-06T07:09:55.797 INFO:teuthology.orchestra.run.smithi083.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-06T07:09:55.797 INFO:teuthology.lock.ops:Updating smithi083.front.sepia.ceph.com on lock server 2024-03-06T07:09:55.822 DEBUG:teuthology.orchestra.run.smithi183:> uname -m 2024-03-06T07:09:55.839 INFO:teuthology.orchestra.run.smithi183.stdout:x86_64 2024-03-06T07:09:55.839 DEBUG:teuthology.orchestra.run.smithi183:> cat /etc/os-release 2024-03-06T07:09:55.895 INFO:teuthology.orchestra.run.smithi183.stdout:NAME="CentOS Stream" 2024-03-06T07:09:55.895 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION="9" 2024-03-06T07:09:55.895 INFO:teuthology.orchestra.run.smithi183.stdout:ID="centos" 2024-03-06T07:09:55.895 INFO:teuthology.orchestra.run.smithi183.stdout:ID_LIKE="rhel fedora" 2024-03-06T07:09:55.895 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION_ID="9" 2024-03-06T07:09:55.895 INFO:teuthology.orchestra.run.smithi183.stdout:PLATFORM_ID="platform:el9" 2024-03-06T07:09:55.896 INFO:teuthology.orchestra.run.smithi183.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-06T07:09:55.896 INFO:teuthology.orchestra.run.smithi183.stdout:ANSI_COLOR="0;31" 2024-03-06T07:09:55.896 INFO:teuthology.orchestra.run.smithi183.stdout:LOGO="fedora-logo-icon" 2024-03-06T07:09:55.896 INFO:teuthology.orchestra.run.smithi183.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-06T07:09:55.896 INFO:teuthology.orchestra.run.smithi183.stdout:HOME_URL="https://centos.org/" 2024-03-06T07:09:55.896 INFO:teuthology.orchestra.run.smithi183.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-06T07:09:55.896 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-06T07:09:55.896 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-06T07:09:55.896 INFO:teuthology.lock.ops:Updating smithi183.front.sepia.ceph.com on lock server 2024-03-06T07:09:55.921 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-06T07:09:55.930 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-06T07:09:55.978 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-06T07:09:55.978 DEBUG:teuthology.orchestra.run.smithi083:> test '!' -e /home/ubuntu/cephtest 2024-03-06T07:09:55.981 DEBUG:teuthology.orchestra.run.smithi183:> test '!' -e /home/ubuntu/cephtest 2024-03-06T07:09:55.996 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-06T07:09:56.034 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-06T07:09:56.034 DEBUG:teuthology.orchestra.run.smithi083:> test -z $(ls -A /var/lib/ceph) 2024-03-06T07:09:56.037 DEBUG:teuthology.orchestra.run.smithi183:> test -z $(ls -A /var/lib/ceph) 2024-03-06T07:09:56.132 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-06T07:09:56.187 INFO:teuthology.run_tasks:Running task kernel... 2024-03-06T07:09:56.203 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-06T07:09:56.203 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-06T07:09:56.204 DEBUG:teuthology.orchestra.run.smithi083:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-06T07:09:56.204 DEBUG:teuthology.orchestra.run.smithi183:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-06T07:09:56.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:09:56.220 DEBUG:teuthology.orchestra.run.smithi183:> uname -r 2024-03-06T07:09:56.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:09:56.221 DEBUG:teuthology.orchestra.run.smithi083:> uname -r 2024-03-06T07:09:56.276 INFO:teuthology.orchestra.run.smithi183.stdout:5.14.0-361.el9.x86_64 2024-03-06T07:09:56.276 INFO:teuthology.task.kernel:Running kernel on smithi183: 5.14.0-361.el9.x86_64 2024-03-06T07:09:56.277 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum install -y kernel 2024-03-06T07:09:56.278 INFO:teuthology.orchestra.run.smithi083.stdout:5.14.0-361.el9.x86_64 2024-03-06T07:09:56.279 INFO:teuthology.task.kernel:Running kernel on smithi083: 5.14.0-361.el9.x86_64 2024-03-06T07:09:56.279 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum install -y kernel 2024-03-06T07:09:58.594 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS-9 - AppStream 9.5 MB/s | 6.7 MB 00:00 2024-03-06T07:09:58.766 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - AppStream 6.8 MB/s | 6.7 MB 00:00 2024-03-06T07:10:00.186 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS-9 - BaseOS 4.2 MB/s | 2.0 MB 00:00 2024-03-06T07:10:00.491 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - BaseOS 3.2 MB/s | 2.0 MB 00:00 2024-03-06T07:10:02.016 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - BaseOS 6.7 MB/s | 8.0 MB 00:01 2024-03-06T07:10:04.226 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - BaseOS 2.2 MB/s | 8.0 MB 00:03 2024-03-06T07:10:06.658 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - AppStream 5.3 MB/s | 19 MB 00:03 2024-03-06T07:10:10.035 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - AppStream 4.0 MB/s | 19 MB 00:04 2024-03-06T07:10:10.525 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.1 MB 00:00 2024-03-06T07:10:12.101 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 15 kB 00:00 2024-03-06T07:10:13.129 INFO:teuthology.orchestra.run.smithi183.stdout:Extra Packages for Enterprise Linux 22 MB/s | 20 MB 00:00 2024-03-06T07:10:17.993 INFO:teuthology.orchestra.run.smithi183.stdout:lab-extras 42 kB/s | 1.7 kB 00:00 2024-03-06T07:10:20.175 INFO:teuthology.orchestra.run.smithi183.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-06T07:10:20.175 INFO:teuthology.orchestra.run.smithi183.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-06T07:10:20.268 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-06T07:10:20.278 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-06T07:10:20.279 INFO:teuthology.orchestra.run.smithi183.stdout: Package Architecture Version Repository Size 2024-03-06T07:10:20.279 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-06T07:10:20.279 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-06T07:10:20.279 INFO:teuthology.orchestra.run.smithi183.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-06T07:10:20.279 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-06T07:10:20.279 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-06T07:10:20.279 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-06T07:10:20.279 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-06T07:10:20.279 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:10:20.279 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-06T07:10:20.279 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-06T07:10:20.279 INFO:teuthology.orchestra.run.smithi183.stdout:Install 4 Packages 2024-03-06T07:10:20.280 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:10:20.280 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 100 M 2024-03-06T07:10:20.280 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 123 M 2024-03-06T07:10:20.280 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-06T07:10:21.310 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - CRB 802 kB/s | 6.1 MB 00:07 2024-03-06T07:10:21.595 INFO:teuthology.orchestra.run.smithi183.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 5.0 MB/s | 6.0 MB 00:01 2024-03-06T07:10:22.927 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 15 kB 00:00 2024-03-06T07:10:23.144 INFO:teuthology.orchestra.run.smithi183.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 7.6 MB/s | 21 MB 00:02 2024-03-06T07:10:23.879 INFO:teuthology.orchestra.run.smithi083.stdout:Extra Packages for Enterprise Linux 25 MB/s | 20 MB 00:00 2024-03-06T07:10:24.761 INFO:teuthology.orchestra.run.smithi183.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 9.0 MB/s | 39 MB 00:04 2024-03-06T07:10:27.786 INFO:teuthology.orchestra.run.smithi183.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 5.4 MB/s | 33 MB 00:06 2024-03-06T07:10:27.787 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:10:27.787 INFO:teuthology.orchestra.run.smithi183.stdout:Total 13 MB/s | 100 MB 00:07 2024-03-06T07:10:28.676 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-06T07:10:28.788 INFO:teuthology.orchestra.run.smithi083.stdout:lab-extras 17 kB/s | 1.7 kB 00:00 2024-03-06T07:10:29.034 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-06T07:10:29.034 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-06T07:10:30.378 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-06T07:10:30.378 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-06T07:10:31.177 INFO:teuthology.orchestra.run.smithi083.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-06T07:10:31.178 INFO:teuthology.orchestra.run.smithi083.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-06T07:10:31.260 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-06T07:10:31.270 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-06T07:10:31.270 INFO:teuthology.orchestra.run.smithi083.stdout: Package Architecture Version Repository Size 2024-03-06T07:10:31.270 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-06T07:10:31.270 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-06T07:10:31.270 INFO:teuthology.orchestra.run.smithi083.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-06T07:10:31.270 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-03-06T07:10:31.270 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-06T07:10:31.271 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-06T07:10:31.271 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-06T07:10:31.271 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:10:31.271 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-06T07:10:31.271 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-06T07:10:31.271 INFO:teuthology.orchestra.run.smithi083.stdout:Install 4 Packages 2024-03-06T07:10:31.271 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:10:31.272 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 100 M 2024-03-06T07:10:31.272 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 123 M 2024-03-06T07:10:31.272 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-06T07:10:33.091 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-06T07:10:33.734 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-06T07:10:33.869 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:10:36.642 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:10:36.822 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:10:41.584 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:10:41.699 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:10:43.071 INFO:teuthology.orchestra.run.smithi083.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 527 kB/s | 6.0 MB 00:11 2024-03-06T07:10:46.497 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:03.186 INFO:teuthology.orchestra.run.smithi083.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 672 kB/s | 21 MB 00:31 2024-03-06T07:11:15.859 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:15.859 INFO:teuthology.orchestra.run.smithi183.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-06T07:11:15.859 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:11:15.944 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:16.399 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:16.400 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-06T07:11:16.400 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:11:16.400 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:11:16.828 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:16.828 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:11:16.828 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-06T07:11:16.828 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-06T07:11:16.828 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-06T07:11:16.828 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-06T07:11:16.829 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-06T07:11:16.829 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:11:16.829 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-06T07:11:17.338 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q kernel | sort -rV | head -n 1 2024-03-06T07:11:17.484 INFO:teuthology.orchestra.run.smithi183.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-06T07:11:17.485 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-06T07:11:17.485 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-06T07:11:17.485 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-06T07:11:17.485 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-06T07:11:17.485 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-06T07:11:17.486 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi183.front.sepia.ceph.com, path=None, version=distro) 2024-03-06T07:11:17.486 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q kernel | sort -rV | head -n 1 2024-03-06T07:11:17.607 INFO:teuthology.orchestra.run.smithi183.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-06T07:11:17.608 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-06T07:11:17.608 DEBUG:teuthology.orchestra.run.smithi183:> sudo rpm -qi grub2-tools 2024-03-06T07:11:17.647 INFO:teuthology.orchestra.run.smithi183.stdout:Name : grub2-tools 2024-03-06T07:11:17.647 INFO:teuthology.orchestra.run.smithi183.stdout:Epoch : 1 2024-03-06T07:11:17.647 INFO:teuthology.orchestra.run.smithi183.stdout:Version : 2.06 2024-03-06T07:11:17.647 INFO:teuthology.orchestra.run.smithi183.stdout:Release : 68.el9 2024-03-06T07:11:17.647 INFO:teuthology.orchestra.run.smithi183.stdout:Architecture: x86_64 2024-03-06T07:11:17.647 INFO:teuthology.orchestra.run.smithi183.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-06T07:11:17.647 INFO:teuthology.orchestra.run.smithi183.stdout:Group : Unspecified 2024-03-06T07:11:17.647 INFO:teuthology.orchestra.run.smithi183.stdout:Size : 8264254 2024-03-06T07:11:17.648 INFO:teuthology.orchestra.run.smithi183.stdout:License : GPLv3+ 2024-03-06T07:11:17.648 INFO:teuthology.orchestra.run.smithi183.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-06T07:11:17.648 INFO:teuthology.orchestra.run.smithi183.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-06T07:11:17.648 INFO:teuthology.orchestra.run.smithi183.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-06T07:11:17.648 INFO:teuthology.orchestra.run.smithi183.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-06T07:11:17.648 INFO:teuthology.orchestra.run.smithi183.stdout:Packager : builder@centos.org 2024-03-06T07:11:17.648 INFO:teuthology.orchestra.run.smithi183.stdout:Vendor : CentOS 2024-03-06T07:11:17.648 INFO:teuthology.orchestra.run.smithi183.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-06T07:11:17.648 INFO:teuthology.orchestra.run.smithi183.stdout:Summary : Support tools for GRUB. 2024-03-06T07:11:17.648 INFO:teuthology.orchestra.run.smithi183.stdout:Description : 2024-03-06T07:11:17.648 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:11:17.649 INFO:teuthology.orchestra.run.smithi183.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-06T07:11:17.649 INFO:teuthology.orchestra.run.smithi183.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-06T07:11:17.649 INFO:teuthology.orchestra.run.smithi183.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-06T07:11:17.649 INFO:teuthology.orchestra.run.smithi183.stdout:hardware devices. 2024-03-06T07:11:17.649 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:11:17.649 INFO:teuthology.orchestra.run.smithi183.stdout:This subpackage provides tools for support of all platforms. 2024-03-06T07:11:17.650 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-06T07:11:17.650 INFO:teuthology.task.kernel:Updating grub on smithi183 to boot 5.14.0-427.el9.x86_64 2024-03-06T07:11:17.650 DEBUG:teuthology.orchestra.run.smithi183:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-06T07:11:18.275 INFO:teuthology.orchestra.run.smithi183.stderr:Generating grub configuration file ... 2024-03-06T07:11:19.668 INFO:teuthology.orchestra.run.smithi183.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-06T07:11:19.705 INFO:teuthology.orchestra.run.smithi183.stderr:done 2024-03-06T07:11:19.706 DEBUG:teuthology.orchestra.run.smithi183:> mktemp 2024-03-06T07:11:19.722 INFO:teuthology.orchestra.run.smithi183.stdout:/tmp/tmp.nH4CMPRo7E 2024-03-06T07:11:19.722 DEBUG:teuthology.orchestra.run.smithi183:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.nH4CMPRo7E 2024-03-06T07:11:19.790 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0666 /tmp/tmp.nH4CMPRo7E 2024-03-06T07:11:19.936 DEBUG:teuthology.orchestra.remote:smithi183:/tmp/tmp.nH4CMPRo7E is 6KB 2024-03-06T07:11:19.987 DEBUG:teuthology.orchestra.run.smithi183:> rm -fr /tmp/tmp.nH4CMPRo7E 2024-03-06T07:11:20.001 DEBUG:teuthology.orchestra.run.smithi183:> sudo /bin/ls /boot/loader/entries || true 2024-03-06T07:11:20.065 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-06T07:11:20.066 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-06T07:11:20.066 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-06T07:11:20.066 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-06T07:11:20.067 DEBUG:teuthology.orchestra.run.smithi183:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-06T07:11:20.226 DEBUG:teuthology.orchestra.run.smithi183:> sudo shutdown -r now 2024-03-06T07:11:37.832 INFO:teuthology.orchestra.run.smithi083.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 606 kB/s | 39 MB 01:06 2024-03-06T07:11:39.707 INFO:teuthology.orchestra.run.smithi083.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 604 kB/s | 33 MB 00:56 2024-03-06T07:11:39.708 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:11:39.708 INFO:teuthology.orchestra.run.smithi083.stdout:Total 1.5 MB/s | 100 MB 01:08 2024-03-06T07:11:40.586 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-06T07:11:40.964 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-06T07:11:40.964 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-06T07:11:42.319 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-06T07:11:42.319 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-06T07:11:45.223 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-06T07:11:45.885 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-06T07:11:46.009 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:11:48.817 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:11:48.996 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:11:50.229 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-06T07:11:50.230 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-06T07:11:50.231 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:11:53.717 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:11:53.935 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:58.691 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:12:27.892 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:12:27.892 INFO:teuthology.orchestra.run.smithi083.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-06T07:12:27.893 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:12:27.975 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:12:28.459 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:12:28.460 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-06T07:12:28.460 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:12:28.460 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:12:28.898 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:12:28.899 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:12:28.899 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-06T07:12:28.899 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-06T07:12:28.899 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-06T07:12:28.899 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-06T07:12:28.899 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-06T07:12:28.899 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:12:28.899 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-06T07:12:29.387 DEBUG:teuthology.orchestra.run.smithi083:> rpm -q kernel | sort -rV | head -n 1 2024-03-06T07:12:29.528 INFO:teuthology.orchestra.run.smithi083.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-06T07:12:29.528 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-06T07:12:29.528 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-06T07:12:29.528 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-06T07:12:29.529 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-06T07:12:29.529 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-06T07:12:29.529 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi083.front.sepia.ceph.com, path=None, version=distro) 2024-03-06T07:12:29.529 DEBUG:teuthology.orchestra.run.smithi083:> rpm -q kernel | sort -rV | head -n 1 2024-03-06T07:12:29.650 INFO:teuthology.orchestra.run.smithi083.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-06T07:12:29.651 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-06T07:12:29.651 DEBUG:teuthology.orchestra.run.smithi083:> sudo rpm -qi grub2-tools 2024-03-06T07:12:29.685 INFO:teuthology.orchestra.run.smithi083.stdout:Name : grub2-tools 2024-03-06T07:12:29.685 INFO:teuthology.orchestra.run.smithi083.stdout:Epoch : 1 2024-03-06T07:12:29.685 INFO:teuthology.orchestra.run.smithi083.stdout:Version : 2.06 2024-03-06T07:12:29.685 INFO:teuthology.orchestra.run.smithi083.stdout:Release : 68.el9 2024-03-06T07:12:29.685 INFO:teuthology.orchestra.run.smithi083.stdout:Architecture: x86_64 2024-03-06T07:12:29.686 INFO:teuthology.orchestra.run.smithi083.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-06T07:12:29.686 INFO:teuthology.orchestra.run.smithi083.stdout:Group : Unspecified 2024-03-06T07:12:29.686 INFO:teuthology.orchestra.run.smithi083.stdout:Size : 8264254 2024-03-06T07:12:29.686 INFO:teuthology.orchestra.run.smithi083.stdout:License : GPLv3+ 2024-03-06T07:12:29.686 INFO:teuthology.orchestra.run.smithi083.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-06T07:12:29.686 INFO:teuthology.orchestra.run.smithi083.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-06T07:12:29.686 INFO:teuthology.orchestra.run.smithi083.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-06T07:12:29.686 INFO:teuthology.orchestra.run.smithi083.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-06T07:12:29.686 INFO:teuthology.orchestra.run.smithi083.stdout:Packager : builder@centos.org 2024-03-06T07:12:29.686 INFO:teuthology.orchestra.run.smithi083.stdout:Vendor : CentOS 2024-03-06T07:12:29.686 INFO:teuthology.orchestra.run.smithi083.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-06T07:12:29.686 INFO:teuthology.orchestra.run.smithi083.stdout:Summary : Support tools for GRUB. 2024-03-06T07:12:29.686 INFO:teuthology.orchestra.run.smithi083.stdout:Description : 2024-03-06T07:12:29.686 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:12:29.687 INFO:teuthology.orchestra.run.smithi083.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-06T07:12:29.687 INFO:teuthology.orchestra.run.smithi083.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-06T07:12:29.687 INFO:teuthology.orchestra.run.smithi083.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-06T07:12:29.687 INFO:teuthology.orchestra.run.smithi083.stdout:hardware devices. 2024-03-06T07:12:29.687 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:12:29.687 INFO:teuthology.orchestra.run.smithi083.stdout:This subpackage provides tools for support of all platforms. 2024-03-06T07:12:29.688 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-06T07:12:29.688 INFO:teuthology.task.kernel:Updating grub on smithi083 to boot 5.14.0-427.el9.x86_64 2024-03-06T07:12:29.688 DEBUG:teuthology.orchestra.run.smithi083:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-06T07:12:30.332 INFO:teuthology.orchestra.run.smithi083.stderr:Generating grub configuration file ... 2024-03-06T07:12:31.804 INFO:teuthology.orchestra.run.smithi083.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-06T07:12:31.839 INFO:teuthology.orchestra.run.smithi083.stderr:done 2024-03-06T07:12:31.841 DEBUG:teuthology.orchestra.run.smithi083:> mktemp 2024-03-06T07:12:31.857 INFO:teuthology.orchestra.run.smithi083.stdout:/tmp/tmp.W1wT9ttbdN 2024-03-06T07:12:31.857 DEBUG:teuthology.orchestra.run.smithi083:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.W1wT9ttbdN 2024-03-06T07:12:31.926 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0666 /tmp/tmp.W1wT9ttbdN 2024-03-06T07:12:32.071 DEBUG:teuthology.orchestra.remote:smithi083:/tmp/tmp.W1wT9ttbdN is 6KB 2024-03-06T07:12:32.122 DEBUG:teuthology.orchestra.run.smithi083:> rm -fr /tmp/tmp.W1wT9ttbdN 2024-03-06T07:12:32.137 DEBUG:teuthology.orchestra.run.smithi083:> sudo /bin/ls /boot/loader/entries || true 2024-03-06T07:12:32.210 INFO:teuthology.orchestra.run.smithi083.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-06T07:12:32.211 INFO:teuthology.orchestra.run.smithi083.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-06T07:12:32.211 INFO:teuthology.orchestra.run.smithi083.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-06T07:12:32.211 INFO:teuthology.orchestra.run.smithi083.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-06T07:12:32.212 DEBUG:teuthology.orchestra.run.smithi083:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-06T07:12:32.372 DEBUG:teuthology.orchestra.run.smithi083:> sudo shutdown -r now 2024-03-06T07:12:50.251 DEBUG:teuthology.orchestra.remote:timed out 2024-03-06T07:12:59.260 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-06T07:12:59.261 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:12:59.608 DEBUG:teuthology.orchestra.run.smithi183:> true 2024-03-06T07:13:00.136 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-06T07:13:00.137 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-427.el9.x86_64"... 2024-03-06T07:13:00.137 DEBUG:teuthology.orchestra.run.smithi183:> uname -r 2024-03-06T07:13:00.151 INFO:teuthology.orchestra.run.smithi183.stdout:5.14.0-427.el9.x86_64 2024-03-06T07:13:00.151 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-06T07:13:00.151 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-06T07:13:00.151 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-06T07:13:01.152 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-06T07:13:01.152 DEBUG:teuthology.orchestra.run.smithi183:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-06T07:13:01.277 INFO:teuthology.orchestra.run.smithi183.stdout:ttyS1 2024-03-06T07:13:01.325 DEBUG:teuthology.parallel:result is None 2024-03-06T07:13:02.376 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-06T07:13:02.376 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-03-06T07:13:02.376 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:13:36.828 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.83 2024-03-06T07:13:45.829 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-03-06T07:13:45.830 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:14:04.220 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.83 2024-03-06T07:14:16.222 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-03-06T07:14:16.223 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:14:16.577 DEBUG:teuthology.orchestra.run.smithi083:> true 2024-03-06T07:14:17.194 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-03-06T07:14:17.195 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-427.el9.x86_64"... 2024-03-06T07:14:17.195 DEBUG:teuthology.orchestra.run.smithi083:> uname -r 2024-03-06T07:14:17.209 INFO:teuthology.orchestra.run.smithi083.stdout:5.14.0-427.el9.x86_64 2024-03-06T07:14:17.210 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-06T07:14:17.210 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-06T07:14:17.210 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-06T07:14:18.211 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-06T07:14:18.211 DEBUG:teuthology.orchestra.run.smithi083:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-06T07:14:18.303 INFO:teuthology.orchestra.run.smithi083.stdout:ttyS1 2024-03-06T07:14:18.353 DEBUG:teuthology.parallel:result is None 2024-03-06T07:14:18.353 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-06T07:14:18.361 INFO:teuthology.task.internal:Creating test directory... 2024-03-06T07:14:18.361 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-06T07:14:18.364 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-06T07:14:18.384 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-06T07:14:18.392 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-06T07:14:18.473 INFO:teuthology.task.internal:Creating archive directory... 2024-03-06T07:14:18.473 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-06T07:14:18.476 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-06T07:14:18.510 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-06T07:14:18.517 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-06T07:14:18.517 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-06T07:14:18.537 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-06T07:14:18.572 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:14:18.585 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:14:18.588 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:14:18.603 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:14:18.604 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-06T07:14:18.611 INFO:teuthology.task.internal:Configuring sudo... 2024-03-06T07:14:18.612 DEBUG:teuthology.orchestra.run.smithi083:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-06T07:14:18.630 DEBUG:teuthology.orchestra.run.smithi183:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-06T07:14:18.661 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-06T07:14:18.679 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-06T07:14:18.679 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-06T07:14:18.698 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-06T07:14:18.716 DEBUG:teuthology.orchestra.run.smithi083:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:14:18.795 DEBUG:teuthology.orchestra.run.smithi083:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:14:18.844 DEBUG:teuthology.orchestra.run.smithi083:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:14:18.925 DEBUG:teuthology.orchestra.run.smithi083:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:14:18.968 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:14:18.968 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-06T07:14:19.035 DEBUG:teuthology.orchestra.run.smithi183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:14:19.078 DEBUG:teuthology.orchestra.run.smithi183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:14:19.128 DEBUG:teuthology.orchestra.run.smithi183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:14:19.210 DEBUG:teuthology.orchestra.run.smithi183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:14:19.256 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:14:19.256 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-06T07:14:19.325 DEBUG:teuthology.orchestra.run.smithi083:> sudo service rsyslog restart 2024-03-06T07:14:19.327 DEBUG:teuthology.orchestra.run.smithi183:> sudo service rsyslog restart 2024-03-06T07:14:19.370 INFO:teuthology.orchestra.run.smithi083.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-06T07:14:19.421 INFO:teuthology.orchestra.run.smithi183.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-06T07:14:19.901 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-06T07:14:19.911 INFO:teuthology.task.internal:Starting timer... 2024-03-06T07:14:19.911 INFO:teuthology.run_tasks:Running task pcp... 2024-03-06T07:14:19.940 INFO:teuthology.run_tasks:Running task selinux... 2024-03-06T07:14:19.951 DEBUG:teuthology.orchestra.run.smithi083:> sudo service auditd rotate 2024-03-06T07:14:19.999 INFO:teuthology.orchestra.run.smithi083.stdout:Rotating logs: 2024-03-06T07:14:19.999 DEBUG:teuthology.orchestra.run.smithi183:> sudo service auditd rotate 2024-03-06T07:14:20.046 INFO:teuthology.orchestra.run.smithi183.stdout:Rotating logs: 2024-03-06T07:14:20.048 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-06T07:14:20.048 DEBUG:teuthology.orchestra.run.smithi083:> /usr/sbin/getenforce 2024-03-06T07:14:20.064 INFO:teuthology.orchestra.run.smithi083.stdout:Permissive 2024-03-06T07:14:20.064 DEBUG:teuthology.orchestra.run.smithi183:> /usr/sbin/getenforce 2024-03-06T07:14:20.097 INFO:teuthology.orchestra.run.smithi183.stdout:Permissive 2024-03-06T07:14:20.097 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi083.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi183.front.sepia.ceph.com': 'permissive'} 2024-03-06T07:14:20.098 DEBUG:teuthology.orchestra.run.smithi083:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-06T07:14:20.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:14:20.133 DEBUG:teuthology.orchestra.run.smithi183:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-06T07:14:20.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:14:20.167 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-06T07:14:20.168 DEBUG:teuthology.orchestra.run.smithi083:> sudo /usr/sbin/setenforce permissive 2024-03-06T07:14:20.215 DEBUG:teuthology.orchestra.run.smithi183:> sudo /usr/sbin/setenforce permissive 2024-03-06T07:14:20.257 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-06T07:14:20.271 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-06T07:14:20.292 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-06T07:14:20.293 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi083.front.sepia.ceph.com,smithi183.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-06T07:18:34.553 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi083.front.sepia.ceph.com'), Remote(name='ubuntu@smithi183.front.sepia.ceph.com')] 2024-03-06T07:18:34.555 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-03-06T07:18:34.556 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:18:34.651 DEBUG:teuthology.orchestra.run.smithi083:> true 2024-03-06T07:18:34.723 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-03-06T07:18:34.724 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-06T07:18:34.724 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:18:34.812 DEBUG:teuthology.orchestra.run.smithi183:> true 2024-03-06T07:18:34.891 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-06T07:18:34.892 INFO:teuthology.run_tasks:Running task clock... 2024-03-06T07:18:34.904 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-06T07:18:34.904 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-06T07:18:34.905 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-06T07:18:34.907 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-06T07:18:34.907 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-06T07:18:34.936 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-06T07:18:34.953 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-06T07:18:34.964 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-06T07:18:34.981 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-06T07:18:34.982 INFO:teuthology.orchestra.run.smithi083.stderr:sudo: ntpd: command not found 2024-03-06T07:18:34.996 INFO:teuthology.orchestra.run.smithi083.stdout:506 Cannot talk to daemon 2024-03-06T07:18:35.011 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: ntpd: command not found 2024-03-06T07:18:35.013 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-06T07:18:35.024 INFO:teuthology.orchestra.run.smithi183.stdout:506 Cannot talk to daemon 2024-03-06T07:18:35.029 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-06T07:18:35.041 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-06T07:18:35.057 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-06T07:18:35.085 INFO:teuthology.orchestra.run.smithi083.stderr:bash: line 1: ntpq: command not found 2024-03-06T07:18:35.088 INFO:teuthology.orchestra.run.smithi083.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-06T07:18:35.088 INFO:teuthology.orchestra.run.smithi083.stdout:=============================================================================== 2024-03-06T07:18:35.088 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:18:35.088 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:18:35.088 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:18:35.088 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:18:35.115 INFO:teuthology.orchestra.run.smithi183.stderr:bash: line 1: ntpq: command not found 2024-03-06T07:18:35.118 INFO:teuthology.orchestra.run.smithi183.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-06T07:18:35.119 INFO:teuthology.orchestra.run.smithi183.stdout:=============================================================================== 2024-03-06T07:18:35.119 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:18:35.119 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:18:35.119 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:18:35.119 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:18:35.119 INFO:teuthology.run_tasks:Running task install... 2024-03-06T07:18:35.128 DEBUG:teuthology.task.install:project ceph 2024-03-06T07:18:35.128 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}} 2024-03-06T07:18:35.129 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'} 2024-03-06T07:18:35.129 INFO:teuthology.task.install:Using flavor: default 2024-03-06T07:18:35.136 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-06T07:18:35.136 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-06T07:18:35.136 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'tag': None, 'wait_for_package': False} 2024-03-06T07:18:35.137 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:18:35.139 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'tag': None, 'wait_for_package': False} 2024-03-06T07:18:35.139 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:18:35.392 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/ 2024-03-06T07:18:35.392 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1739.g1bc250c7 2024-03-06T07:18:35.398 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/ 2024-03-06T07:18:35.398 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1739.g1bc250c7 2024-03-06T07:18:35.687 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-06T07:18:35.687 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:18:35.687 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-06T07:18:35.690 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-06T07:18:35.690 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:18:35.690 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-06T07:18:35.718 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-06T07:18:35.718 DEBUG:teuthology.orchestra.run.smithi083:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-06T07:18:35.723 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-06T07:18:35.723 DEBUG:teuthology.orchestra.run.smithi183:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-06T07:18:35.787 DEBUG:teuthology.orchestra.run.smithi083:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-06T07:18:35.793 DEBUG:teuthology.orchestra.run.smithi183:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-06T07:18:35.873 DEBUG:teuthology.orchestra.run.smithi083:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-06T07:18:35.875 DEBUG:teuthology.orchestra.run.smithi183:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-06T07:18:35.939 INFO:teuthology.orchestra.run.smithi083.stdout:check_obsoletes = 1 2024-03-06T07:18:35.941 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum clean all 2024-03-06T07:18:35.943 INFO:teuthology.orchestra.run.smithi183.stdout:check_obsoletes = 1 2024-03-06T07:18:35.945 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum clean all 2024-03-06T07:18:36.200 INFO:teuthology.orchestra.run.smithi183.stdout:74 files removed 2024-03-06T07:18:36.207 INFO:teuthology.orchestra.run.smithi083.stdout:74 files removed 2024-03-06T07:18:36.230 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-06T07:18:36.236 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-06T07:18:37.632 INFO:teuthology.orchestra.run.smithi083.stdout:ceph packages for x86_64 77 kB/s | 83 kB 00:01 2024-03-06T07:18:38.244 INFO:teuthology.orchestra.run.smithi183.stdout:ceph packages for x86_64 49 kB/s | 83 kB 00:01 2024-03-06T07:18:38.549 INFO:teuthology.orchestra.run.smithi083.stdout:ceph noarch packages 12 kB/s | 10 kB 00:00 2024-03-06T07:18:39.020 INFO:teuthology.orchestra.run.smithi183.stdout:ceph noarch packages 14 kB/s | 10 kB 00:00 2024-03-06T07:18:39.425 INFO:teuthology.orchestra.run.smithi083.stdout:ceph source packages 2.3 kB/s | 1.9 kB 00:00 2024-03-06T07:18:39.821 INFO:teuthology.orchestra.run.smithi183.stdout:ceph source packages 2.5 kB/s | 1.9 kB 00:00 2024-03-06T07:18:39.869 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-06T07:18:40.255 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-06T07:18:41.303 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS-9 - BaseOS 6.1 MB/s | 2.0 MB 00:00 2024-03-06T07:18:41.656 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2024-03-06T07:18:42.185 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-03-06T07:18:42.514 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-03-06T07:18:44.506 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-03-06T07:18:46.527 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - AppStream 5.7 MB/s | 19 MB 00:03 2024-03-06T07:18:49.773 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - CRB 3.4 MB/s | 6.1 MB 00:01 2024-03-06T07:18:52.424 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - Extras packages 11 kB/s | 15 kB 00:01 2024-03-06T07:18:52.879 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - CRB 2.1 MB/s | 6.1 MB 00:02 2024-03-06T07:18:53.911 INFO:teuthology.orchestra.run.smithi183.stdout:Extra Packages for Enterprise Linux 15 MB/s | 20 MB 00:01 2024-03-06T07:18:54.388 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 15 kB 00:00 2024-03-06T07:18:58.875 INFO:teuthology.orchestra.run.smithi183.stdout:lab-extras 38 kB/s | 1.7 kB 00:00 2024-03-06T07:19:01.426 INFO:teuthology.orchestra.run.smithi183.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:19:01.427 INFO:teuthology.orchestra.run.smithi183.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:19:01.441 INFO:teuthology.orchestra.run.smithi183.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-06T07:19:01.443 INFO:teuthology.orchestra.run.smithi183.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-06T07:19:01.444 INFO:teuthology.orchestra.run.smithi183.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-06T07:19:01.445 INFO:teuthology.orchestra.run.smithi183.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-06T07:19:01.447 INFO:teuthology.orchestra.run.smithi183.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-06T07:19:01.450 INFO:teuthology.orchestra.run.smithi183.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-06T07:19:01.451 INFO:teuthology.orchestra.run.smithi183.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-06T07:19:01.532 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-06T07:19:01.556 INFO:teuthology.orchestra.run.smithi183.stdout:============================================================================================ 2024-03-06T07:19:01.556 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-06T07:19:01.556 INFO:teuthology.orchestra.run.smithi183.stdout:============================================================================================ 2024-03-06T07:19:01.556 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-06T07:19:01.556 INFO:teuthology.orchestra.run.smithi183.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-06T07:19:01.556 INFO:teuthology.orchestra.run.smithi183.stdout: ceph x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 6.6 k 2024-03-06T07:19:01.556 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-base x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.3 M 2024-03-06T07:19:01.556 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 888 k 2024-03-06T07:19:01.556 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 145 k 2024-03-06T07:19:01.556 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 1.5 M 2024-03-06T07:19:01.557 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 138 k 2024-03-06T07:19:01.557 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 1.8 M 2024-03-06T07:19:01.557 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 7.4 M 2024-03-06T07:19:01.557 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-rook noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 49 k 2024-03-06T07:19:01.557 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-radosgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 9.8 M 2024-03-06T07:19:01.557 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-test x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 47 M 2024-03-06T07:19:01.557 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-volume noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 264 k 2024-03-06T07:19:01.557 INFO:teuthology.orchestra.run.smithi183.stdout: cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 748 k 2024-03-06T07:19:01.557 INFO:teuthology.orchestra.run.smithi183.stdout: cephfs-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 220 k 2024-03-06T07:19:01.557 INFO:teuthology.orchestra.run.smithi183.stdout: cephfs-shell x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 27 k 2024-03-06T07:19:01.557 INFO:teuthology.orchestra.run.smithi183.stdout: cephfs-top noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 18 k 2024-03-06T07:19:01.557 INFO:teuthology.orchestra.run.smithi183.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-03-06T07:19:01.557 INFO:teuthology.orchestra.run.smithi183.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-06T07:19:01.558 INFO:teuthology.orchestra.run.smithi183.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-06T07:19:01.558 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 31 k 2024-03-06T07:19:01.558 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 747 k 2024-03-06T07:19:01.558 INFO:teuthology.orchestra.run.smithi183.stdout: librados-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 127 k 2024-03-06T07:19:01.558 INFO:teuthology.orchestra.run.smithi183.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-06T07:19:01.558 INFO:teuthology.orchestra.run.smithi183.stdout: libuuid-devel x86_64 2.37.4-18.el9 CentOS-AppStream 24 k 2024-03-06T07:19:01.558 INFO:teuthology.orchestra.run.smithi183.stdout: openssl-devel x86_64 1:3.0.7-27.el9 CentOS-AppStream 3.0 M 2024-03-06T07:19:01.558 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-06T07:19:01.558 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-06T07:19:01.558 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-06T07:19:01.558 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cephfs x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 162 k 2024-03-06T07:19:01.558 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rados x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 321 k 2024-03-06T07:19:01.558 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 298 k 2024-03-06T07:19:01.559 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 99 k 2024-03-06T07:19:01.559 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 85 k 2024-03-06T07:19:01.559 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:19:01.559 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-nbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 171 k 2024-03-06T07:19:01.559 INFO:teuthology.orchestra.run.smithi183.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-03-06T07:19:01.559 INFO:teuthology.orchestra.run.smithi183.stdout:Upgrading: 2024-03-06T07:19:01.559 INFO:teuthology.orchestra.run.smithi183.stdout: NetworkManager x86_64 1:1.46.0-1.el9 CentOS-BaseOS 2.3 M 2024-03-06T07:19:01.559 INFO:teuthology.orchestra.run.smithi183.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 CentOS-BaseOS 1.8 M 2024-03-06T07:19:01.559 INFO:teuthology.orchestra.run.smithi183.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 CentOS-BaseOS 41 k 2024-03-06T07:19:01.559 INFO:teuthology.orchestra.run.smithi183.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 CentOS-BaseOS 247 k 2024-03-06T07:19:01.559 INFO:teuthology.orchestra.run.smithi183.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-03-06T07:19:01.559 INFO:teuthology.orchestra.run.smithi183.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-03-06T07:19:01.559 INFO:teuthology.orchestra.run.smithi183.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-03-06T07:19:01.559 INFO:teuthology.orchestra.run.smithi183.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-03-06T07:19:01.560 INFO:teuthology.orchestra.run.smithi183.stdout: iproute x86_64 6.7.0-2.el9 CentOS-BaseOS 802 k 2024-03-06T07:19:01.560 INFO:teuthology.orchestra.run.smithi183.stdout: iproute-tc x86_64 6.7.0-2.el9 CentOS-BaseOS 433 k 2024-03-06T07:19:01.560 INFO:teuthology.orchestra.run.smithi183.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-03-06T07:19:01.560 INFO:teuthology.orchestra.run.smithi183.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-03-06T07:19:01.560 INFO:teuthology.orchestra.run.smithi183.stdout: libblkid x86_64 2.37.4-18.el9 CentOS-BaseOS 107 k 2024-03-06T07:19:01.560 INFO:teuthology.orchestra.run.smithi183.stdout: libfdisk x86_64 2.37.4-18.el9 CentOS-BaseOS 154 k 2024-03-06T07:19:01.560 INFO:teuthology.orchestra.run.smithi183.stdout: libmount x86_64 2.37.4-18.el9 CentOS-BaseOS 135 k 2024-03-06T07:19:01.560 INFO:teuthology.orchestra.run.smithi183.stdout: librados2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.4 M 2024-03-06T07:19:01.560 INFO:teuthology.orchestra.run.smithi183.stdout: librbd1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:19:01.560 INFO:teuthology.orchestra.run.smithi183.stdout: libsmartcols x86_64 2.37.4-18.el9 CentOS-BaseOS 63 k 2024-03-06T07:19:01.560 INFO:teuthology.orchestra.run.smithi183.stdout: libuuid x86_64 2.37.4-18.el9 CentOS-BaseOS 28 k 2024-03-06T07:19:01.560 INFO:teuthology.orchestra.run.smithi183.stdout: openssl x86_64 1:3.0.7-27.el9 CentOS-BaseOS 1.2 M 2024-03-06T07:19:01.561 INFO:teuthology.orchestra.run.smithi183.stdout: openssl-libs x86_64 1:3.0.7-27.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:19:01.561 INFO:teuthology.orchestra.run.smithi183.stdout: util-linux x86_64 2.37.4-18.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:19:01.561 INFO:teuthology.orchestra.run.smithi183.stdout: util-linux-core x86_64 2.37.4-18.el9 CentOS-BaseOS 438 k 2024-03-06T07:19:01.561 INFO:teuthology.orchestra.run.smithi183.stdout: util-linux-user x86_64 2.37.4-18.el9 CentOS-BaseOS 30 k 2024-03-06T07:19:01.561 INFO:teuthology.orchestra.run.smithi183.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-06T07:19:01.561 INFO:teuthology.orchestra.run.smithi183.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:19:01.561 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-06T07:19:01.561 INFO:teuthology.orchestra.run.smithi183.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-06T07:19:01.561 INFO:teuthology.orchestra.run.smithi183.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-06T07:19:01.561 INFO:teuthology.orchestra.run.smithi183.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-06T07:19:01.561 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 20 M 2024-03-06T07:19:01.561 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 31 k 2024-03-06T07:19:01.561 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mds x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 2.3 M 2024-03-06T07:19:01.561 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 248 k 2024-03-06T07:19:01.561 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mon x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 4.7 M 2024-03-06T07:19:01.562 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-osd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 17 M 2024-03-06T07:19:01.562 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 15 k 2024-03-06T07:19:01.562 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-selinux x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 25 k 2024-03-06T07:19:01.562 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-06T07:19:01.562 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-06T07:19:01.562 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-06T07:19:01.562 INFO:teuthology.orchestra.run.smithi183.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-06T07:19:01.562 INFO:teuthology.orchestra.run.smithi183.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-06T07:19:01.562 INFO:teuthology.orchestra.run.smithi183.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-06T07:19:01.562 INFO:teuthology.orchestra.run.smithi183.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-03-06T07:19:01.562 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-06T07:19:01.562 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-06T07:19:01.562 INFO:teuthology.orchestra.run.smithi183.stdout: libcephsqlite x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 163 k 2024-03-06T07:19:01.562 INFO:teuthology.orchestra.run.smithi183.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-03-06T07:19:01.563 INFO:teuthology.orchestra.run.smithi183.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-06T07:19:01.563 INFO:teuthology.orchestra.run.smithi183.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-06T07:19:01.563 INFO:teuthology.orchestra.run.smithi183.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-03-06T07:19:01.563 INFO:teuthology.orchestra.run.smithi183.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-06T07:19:01.563 INFO:teuthology.orchestra.run.smithi183.stdout: libradosstriper1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 501 k 2024-03-06T07:19:01.563 INFO:teuthology.orchestra.run.smithi183.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-06T07:19:01.563 INFO:teuthology.orchestra.run.smithi183.stdout: librgw2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.1 M 2024-03-06T07:19:01.563 INFO:teuthology.orchestra.run.smithi183.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-06T07:19:01.563 INFO:teuthology.orchestra.run.smithi183.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-06T07:19:01.563 INFO:teuthology.orchestra.run.smithi183.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-06T07:19:01.563 INFO:teuthology.orchestra.run.smithi183.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-06T07:19:01.563 INFO:teuthology.orchestra.run.smithi183.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-06T07:19:01.563 INFO:teuthology.orchestra.run.smithi183.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-06T07:19:01.563 INFO:teuthology.orchestra.run.smithi183.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-03-06T07:19:01.564 INFO:teuthology.orchestra.run.smithi183.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-06T07:19:01.564 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-06T07:19:01.564 INFO:teuthology.orchestra.run.smithi183.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-06T07:19:01.564 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-06T07:19:01.564 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-06T07:19:01.564 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-06T07:19:01.564 INFO:teuthology.orchestra.run.smithi183.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-06T07:19:01.564 INFO:teuthology.orchestra.run.smithi183.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-06T07:19:01.564 INFO:teuthology.orchestra.run.smithi183.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-06T07:19:01.564 INFO:teuthology.orchestra.run.smithi183.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-06T07:19:01.564 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-06T07:19:01.564 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-argparse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 45 k 2024-03-06T07:19:01.564 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 131 k 2024-03-06T07:19:01.564 INFO:teuthology.orchestra.run.smithi183.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-06T07:19:01.565 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-06T07:19:01.565 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-06T07:19:01.565 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-06T07:19:01.565 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-06T07:19:01.565 INFO:teuthology.orchestra.run.smithi183.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-06T07:19:01.565 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:19:01.565 INFO:teuthology.orchestra.run.smithi183.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-06T07:19:01.565 INFO:teuthology.orchestra.run.smithi183.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-06T07:19:01.565 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-06T07:19:01.565 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-06T07:19:01.565 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-06T07:19:01.565 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-06T07:19:01.566 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-06T07:19:01.566 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-06T07:19:01.566 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-06T07:19:01.566 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-03-06T07:19:01.566 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-06T07:19:01.566 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-06T07:19:01.566 INFO:teuthology.orchestra.run.smithi183.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-06T07:19:01.566 INFO:teuthology.orchestra.run.smithi183.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-06T07:19:01.566 INFO:teuthology.orchestra.run.smithi183.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-06T07:19:01.566 INFO:teuthology.orchestra.run.smithi183.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-06T07:19:01.566 INFO:teuthology.orchestra.run.smithi183.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-06T07:19:01.567 INFO:teuthology.orchestra.run.smithi183.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-06T07:19:01.567 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-06T07:19:01.567 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-06T07:19:01.567 INFO:teuthology.orchestra.run.smithi183.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-06T07:19:01.567 INFO:teuthology.orchestra.run.smithi183.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-06T07:19:01.567 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-06T07:19:01.567 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-06T07:19:01.567 INFO:teuthology.orchestra.run.smithi183.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-06T07:19:01.567 INFO:teuthology.orchestra.run.smithi183.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-06T07:19:01.567 INFO:teuthology.orchestra.run.smithi183.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-06T07:19:01.567 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-06T07:19:01.567 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-06T07:19:01.567 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-06T07:19:01.567 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-06T07:19:01.568 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-06T07:19:01.568 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-06T07:19:01.568 INFO:teuthology.orchestra.run.smithi183.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-06T07:19:01.568 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-03-06T07:19:01.568 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-06T07:19:01.568 INFO:teuthology.orchestra.run.smithi183.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-06T07:19:01.568 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-06T07:19:01.568 INFO:teuthology.orchestra.run.smithi183.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-06T07:19:01.568 INFO:teuthology.orchestra.run.smithi183.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-06T07:19:01.568 INFO:teuthology.orchestra.run.smithi183.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-06T07:19:01.568 INFO:teuthology.orchestra.run.smithi183.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-06T07:19:01.568 INFO:teuthology.orchestra.run.smithi183.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-06T07:19:01.568 INFO:teuthology.orchestra.run.smithi183.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-06T07:19:01.568 INFO:teuthology.orchestra.run.smithi183.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-06T07:19:01.568 INFO:teuthology.orchestra.run.smithi183.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-06T07:19:01.569 INFO:teuthology.orchestra.run.smithi183.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-06T07:19:01.569 INFO:teuthology.orchestra.run.smithi183.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-06T07:19:01.569 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-03-06T07:19:01.569 INFO:teuthology.orchestra.run.smithi183.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-03-06T07:19:01.569 INFO:teuthology.orchestra.run.smithi183.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-06T07:19:01.569 INFO:teuthology.orchestra.run.smithi183.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-06T07:19:01.569 INFO:teuthology.orchestra.run.smithi183.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-06T07:19:01.569 INFO:teuthology.orchestra.run.smithi183.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-06T07:19:01.569 INFO:teuthology.orchestra.run.smithi183.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-06T07:19:01.569 INFO:teuthology.orchestra.run.smithi183.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-06T07:19:01.569 INFO:teuthology.orchestra.run.smithi183.stdout:Installing weak dependencies: 2024-03-06T07:19:01.569 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-03-06T07:19:01.569 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:19:01.569 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-06T07:19:01.569 INFO:teuthology.orchestra.run.smithi183.stdout:============================================================================================ 2024-03-06T07:19:01.570 INFO:teuthology.orchestra.run.smithi183.stdout:Install 150 Packages 2024-03-06T07:19:01.570 INFO:teuthology.orchestra.run.smithi183.stdout:Upgrade 26 Packages 2024-03-06T07:19:01.570 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:19:01.570 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 224 M 2024-03-06T07:19:01.570 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-06T07:19:02.359 INFO:teuthology.orchestra.run.smithi183.stdout:(1/176): ceph-19.0.0-1739.g1bc250c7.el9.x86_64. 19 kB/s | 6.6 kB 00:00 2024-03-06T07:19:03.392 INFO:teuthology.orchestra.run.smithi183.stdout:(2/176): ceph-fuse-19.0.0-1739.g1bc250c7.el9.x8 860 kB/s | 888 kB 00:01 2024-03-06T07:19:03.559 INFO:teuthology.orchestra.run.smithi183.stdout:(3/176): ceph-immutable-object-cache-19.0.0-173 870 kB/s | 145 kB 00:00 2024-03-06T07:19:05.118 INFO:teuthology.orchestra.run.smithi183.stdout:(4/176): ceph-mds-19.0.0-1739.g1bc250c7.el9.x86 1.5 MB/s | 2.3 MB 00:01 2024-03-06T07:19:06.884 INFO:teuthology.orchestra.run.smithi183.stdout:(5/176): ceph-mgr-19.0.0-1739.g1bc250c7.el9.x86 866 kB/s | 1.5 MB 00:01 2024-03-06T07:19:11.154 INFO:teuthology.orchestra.run.smithi083.stdout:Extra Packages for Enterprise Linux 1.2 MB/s | 20 MB 00:16 2024-03-06T07:19:11.226 INFO:teuthology.orchestra.run.smithi183.stdout:(6/176): ceph-base-19.0.0-1739.g1bc250c7.el9.x8 594 kB/s | 5.3 MB 00:09 2024-03-06T07:19:12.226 INFO:teuthology.orchestra.run.smithi183.stdout:(7/176): ceph-mon-19.0.0-1739.g1bc250c7.el9.x86 899 kB/s | 4.7 MB 00:05 2024-03-06T07:19:16.120 INFO:teuthology.orchestra.run.smithi083.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-06T07:19:16.434 INFO:teuthology.orchestra.run.smithi183.stdout:(8/176): ceph-common-19.0.0-1739.g1bc250c7.el9. 1.4 MB/s | 20 MB 00:14 2024-03-06T07:19:16.525 INFO:teuthology.orchestra.run.smithi183.stdout:(9/176): ceph-selinux-19.0.0-1739.g1bc250c7.el9 276 kB/s | 25 kB 00:00 2024-03-06T07:19:18.647 INFO:teuthology.orchestra.run.smithi083.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:19:18.648 INFO:teuthology.orchestra.run.smithi083.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:19:18.662 INFO:teuthology.orchestra.run.smithi083.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-06T07:19:18.663 INFO:teuthology.orchestra.run.smithi083.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-06T07:19:18.664 INFO:teuthology.orchestra.run.smithi083.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-06T07:19:18.665 INFO:teuthology.orchestra.run.smithi083.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-06T07:19:18.667 INFO:teuthology.orchestra.run.smithi083.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-06T07:19:18.670 INFO:teuthology.orchestra.run.smithi083.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-06T07:19:18.671 INFO:teuthology.orchestra.run.smithi083.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-06T07:19:18.751 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-06T07:19:18.768 INFO:teuthology.orchestra.run.smithi083.stdout:============================================================================================ 2024-03-06T07:19:18.768 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-03-06T07:19:18.768 INFO:teuthology.orchestra.run.smithi083.stdout:============================================================================================ 2024-03-06T07:19:18.768 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-06T07:19:18.769 INFO:teuthology.orchestra.run.smithi083.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-06T07:19:18.769 INFO:teuthology.orchestra.run.smithi083.stdout: ceph x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 6.6 k 2024-03-06T07:19:18.769 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-base x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.3 M 2024-03-06T07:19:18.769 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 888 k 2024-03-06T07:19:18.769 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 145 k 2024-03-06T07:19:18.769 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 1.5 M 2024-03-06T07:19:18.769 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 138 k 2024-03-06T07:19:18.769 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 1.8 M 2024-03-06T07:19:18.769 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 7.4 M 2024-03-06T07:19:18.769 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-rook noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 49 k 2024-03-06T07:19:18.769 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-radosgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 9.8 M 2024-03-06T07:19:18.769 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-test x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 47 M 2024-03-06T07:19:18.770 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-volume noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 264 k 2024-03-06T07:19:18.770 INFO:teuthology.orchestra.run.smithi083.stdout: cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 748 k 2024-03-06T07:19:18.770 INFO:teuthology.orchestra.run.smithi083.stdout: cephfs-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 220 k 2024-03-06T07:19:18.770 INFO:teuthology.orchestra.run.smithi083.stdout: cephfs-shell x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 27 k 2024-03-06T07:19:18.770 INFO:teuthology.orchestra.run.smithi083.stdout: cephfs-top noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 18 k 2024-03-06T07:19:18.770 INFO:teuthology.orchestra.run.smithi083.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-03-06T07:19:18.770 INFO:teuthology.orchestra.run.smithi083.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-06T07:19:18.770 INFO:teuthology.orchestra.run.smithi083.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-06T07:19:18.770 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 31 k 2024-03-06T07:19:18.770 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 747 k 2024-03-06T07:19:18.770 INFO:teuthology.orchestra.run.smithi083.stdout: librados-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 127 k 2024-03-06T07:19:18.771 INFO:teuthology.orchestra.run.smithi083.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-06T07:19:18.771 INFO:teuthology.orchestra.run.smithi083.stdout: libuuid-devel x86_64 2.37.4-18.el9 CentOS-AppStream 24 k 2024-03-06T07:19:18.771 INFO:teuthology.orchestra.run.smithi083.stdout: openssl-devel x86_64 1:3.0.7-27.el9 CentOS-AppStream 3.0 M 2024-03-06T07:19:18.771 INFO:teuthology.orchestra.run.smithi083.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-06T07:19:18.771 INFO:teuthology.orchestra.run.smithi083.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-06T07:19:18.771 INFO:teuthology.orchestra.run.smithi083.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-06T07:19:18.771 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cephfs x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 162 k 2024-03-06T07:19:18.771 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rados x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 321 k 2024-03-06T07:19:18.771 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 298 k 2024-03-06T07:19:18.771 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 99 k 2024-03-06T07:19:18.771 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 85 k 2024-03-06T07:19:18.771 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:19:18.771 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-nbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 171 k 2024-03-06T07:19:18.771 INFO:teuthology.orchestra.run.smithi083.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-03-06T07:19:18.772 INFO:teuthology.orchestra.run.smithi083.stdout:Upgrading: 2024-03-06T07:19:18.772 INFO:teuthology.orchestra.run.smithi083.stdout: NetworkManager x86_64 1:1.46.0-1.el9 CentOS-BaseOS 2.3 M 2024-03-06T07:19:18.772 INFO:teuthology.orchestra.run.smithi083.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 CentOS-BaseOS 1.8 M 2024-03-06T07:19:18.772 INFO:teuthology.orchestra.run.smithi083.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 CentOS-BaseOS 41 k 2024-03-06T07:19:18.772 INFO:teuthology.orchestra.run.smithi083.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 CentOS-BaseOS 247 k 2024-03-06T07:19:18.772 INFO:teuthology.orchestra.run.smithi083.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-03-06T07:19:18.772 INFO:teuthology.orchestra.run.smithi083.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-03-06T07:19:18.772 INFO:teuthology.orchestra.run.smithi083.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-03-06T07:19:18.772 INFO:teuthology.orchestra.run.smithi083.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-03-06T07:19:18.772 INFO:teuthology.orchestra.run.smithi083.stdout: iproute x86_64 6.7.0-2.el9 CentOS-BaseOS 802 k 2024-03-06T07:19:18.772 INFO:teuthology.orchestra.run.smithi083.stdout: iproute-tc x86_64 6.7.0-2.el9 CentOS-BaseOS 433 k 2024-03-06T07:19:18.773 INFO:teuthology.orchestra.run.smithi083.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-03-06T07:19:18.773 INFO:teuthology.orchestra.run.smithi083.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-03-06T07:19:18.773 INFO:teuthology.orchestra.run.smithi083.stdout: libblkid x86_64 2.37.4-18.el9 CentOS-BaseOS 107 k 2024-03-06T07:19:18.773 INFO:teuthology.orchestra.run.smithi083.stdout: libfdisk x86_64 2.37.4-18.el9 CentOS-BaseOS 154 k 2024-03-06T07:19:18.773 INFO:teuthology.orchestra.run.smithi083.stdout: libmount x86_64 2.37.4-18.el9 CentOS-BaseOS 135 k 2024-03-06T07:19:18.773 INFO:teuthology.orchestra.run.smithi083.stdout: librados2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.4 M 2024-03-06T07:19:18.773 INFO:teuthology.orchestra.run.smithi083.stdout: librbd1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:19:18.773 INFO:teuthology.orchestra.run.smithi083.stdout: libsmartcols x86_64 2.37.4-18.el9 CentOS-BaseOS 63 k 2024-03-06T07:19:18.773 INFO:teuthology.orchestra.run.smithi083.stdout: libuuid x86_64 2.37.4-18.el9 CentOS-BaseOS 28 k 2024-03-06T07:19:18.773 INFO:teuthology.orchestra.run.smithi083.stdout: openssl x86_64 1:3.0.7-27.el9 CentOS-BaseOS 1.2 M 2024-03-06T07:19:18.773 INFO:teuthology.orchestra.run.smithi083.stdout: openssl-libs x86_64 1:3.0.7-27.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:19:18.773 INFO:teuthology.orchestra.run.smithi083.stdout: util-linux x86_64 2.37.4-18.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:19:18.773 INFO:teuthology.orchestra.run.smithi083.stdout: util-linux-core x86_64 2.37.4-18.el9 CentOS-BaseOS 438 k 2024-03-06T07:19:18.773 INFO:teuthology.orchestra.run.smithi083.stdout: util-linux-user x86_64 2.37.4-18.el9 CentOS-BaseOS 30 k 2024-03-06T07:19:18.774 INFO:teuthology.orchestra.run.smithi083.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-06T07:19:18.774 INFO:teuthology.orchestra.run.smithi083.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:19:18.774 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-03-06T07:19:18.774 INFO:teuthology.orchestra.run.smithi083.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-06T07:19:18.774 INFO:teuthology.orchestra.run.smithi083.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-06T07:19:18.774 INFO:teuthology.orchestra.run.smithi083.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-06T07:19:18.774 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 20 M 2024-03-06T07:19:18.774 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 31 k 2024-03-06T07:19:18.774 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mds x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 2.3 M 2024-03-06T07:19:18.774 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 248 k 2024-03-06T07:19:18.774 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mon x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 4.7 M 2024-03-06T07:19:18.774 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-osd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 17 M 2024-03-06T07:19:18.775 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 15 k 2024-03-06T07:19:18.775 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-selinux x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 25 k 2024-03-06T07:19:18.775 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-06T07:19:18.775 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-06T07:19:18.775 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-06T07:19:18.775 INFO:teuthology.orchestra.run.smithi083.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-06T07:19:18.775 INFO:teuthology.orchestra.run.smithi083.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-06T07:19:18.775 INFO:teuthology.orchestra.run.smithi083.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-06T07:19:18.775 INFO:teuthology.orchestra.run.smithi083.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-03-06T07:19:18.775 INFO:teuthology.orchestra.run.smithi083.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-06T07:19:18.775 INFO:teuthology.orchestra.run.smithi083.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-06T07:19:18.775 INFO:teuthology.orchestra.run.smithi083.stdout: libcephsqlite x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 163 k 2024-03-06T07:19:18.776 INFO:teuthology.orchestra.run.smithi083.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-03-06T07:19:18.776 INFO:teuthology.orchestra.run.smithi083.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-06T07:19:18.776 INFO:teuthology.orchestra.run.smithi083.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-06T07:19:18.776 INFO:teuthology.orchestra.run.smithi083.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-03-06T07:19:18.776 INFO:teuthology.orchestra.run.smithi083.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-06T07:19:18.776 INFO:teuthology.orchestra.run.smithi083.stdout: libradosstriper1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 501 k 2024-03-06T07:19:18.776 INFO:teuthology.orchestra.run.smithi083.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-06T07:19:18.776 INFO:teuthology.orchestra.run.smithi083.stdout: librgw2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.1 M 2024-03-06T07:19:18.776 INFO:teuthology.orchestra.run.smithi083.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-06T07:19:18.776 INFO:teuthology.orchestra.run.smithi083.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-06T07:19:18.776 INFO:teuthology.orchestra.run.smithi083.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-06T07:19:18.776 INFO:teuthology.orchestra.run.smithi083.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-06T07:19:18.776 INFO:teuthology.orchestra.run.smithi083.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-06T07:19:18.777 INFO:teuthology.orchestra.run.smithi083.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-06T07:19:18.777 INFO:teuthology.orchestra.run.smithi083.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-03-06T07:19:18.777 INFO:teuthology.orchestra.run.smithi083.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-06T07:19:18.777 INFO:teuthology.orchestra.run.smithi083.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-06T07:19:18.777 INFO:teuthology.orchestra.run.smithi083.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-06T07:19:18.777 INFO:teuthology.orchestra.run.smithi083.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-06T07:19:18.777 INFO:teuthology.orchestra.run.smithi083.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-06T07:19:18.777 INFO:teuthology.orchestra.run.smithi083.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-06T07:19:18.777 INFO:teuthology.orchestra.run.smithi083.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-06T07:19:18.777 INFO:teuthology.orchestra.run.smithi083.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-06T07:19:18.777 INFO:teuthology.orchestra.run.smithi083.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-06T07:19:18.777 INFO:teuthology.orchestra.run.smithi083.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-06T07:19:18.778 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-06T07:19:18.778 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-argparse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 45 k 2024-03-06T07:19:18.778 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 131 k 2024-03-06T07:19:18.778 INFO:teuthology.orchestra.run.smithi083.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-06T07:19:18.778 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-06T07:19:18.778 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-06T07:19:18.778 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-06T07:19:18.778 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-06T07:19:18.778 INFO:teuthology.orchestra.run.smithi083.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-06T07:19:18.778 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:19:18.778 INFO:teuthology.orchestra.run.smithi083.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-06T07:19:18.778 INFO:teuthology.orchestra.run.smithi083.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-06T07:19:18.778 INFO:teuthology.orchestra.run.smithi083.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-06T07:19:18.778 INFO:teuthology.orchestra.run.smithi083.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-06T07:19:18.779 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-06T07:19:18.779 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-06T07:19:18.779 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-06T07:19:18.779 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-06T07:19:18.779 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-06T07:19:18.779 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-03-06T07:19:18.779 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-06T07:19:18.779 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-06T07:19:18.779 INFO:teuthology.orchestra.run.smithi083.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-06T07:19:18.779 INFO:teuthology.orchestra.run.smithi083.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-06T07:19:18.779 INFO:teuthology.orchestra.run.smithi083.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-06T07:19:18.779 INFO:teuthology.orchestra.run.smithi083.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-06T07:19:18.779 INFO:teuthology.orchestra.run.smithi083.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-06T07:19:18.779 INFO:teuthology.orchestra.run.smithi083.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-06T07:19:18.780 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-06T07:19:18.780 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-06T07:19:18.780 INFO:teuthology.orchestra.run.smithi083.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-06T07:19:18.780 INFO:teuthology.orchestra.run.smithi083.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-06T07:19:18.780 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-06T07:19:18.780 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-06T07:19:18.780 INFO:teuthology.orchestra.run.smithi083.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-06T07:19:18.780 INFO:teuthology.orchestra.run.smithi083.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-06T07:19:18.780 INFO:teuthology.orchestra.run.smithi083.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-06T07:19:18.780 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-06T07:19:18.780 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-06T07:19:18.780 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-06T07:19:18.781 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-06T07:19:18.781 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-06T07:19:18.781 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-06T07:19:18.781 INFO:teuthology.orchestra.run.smithi083.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-06T07:19:18.781 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-03-06T07:19:18.781 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-06T07:19:18.781 INFO:teuthology.orchestra.run.smithi083.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-06T07:19:18.781 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-06T07:19:18.781 INFO:teuthology.orchestra.run.smithi083.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-06T07:19:18.781 INFO:teuthology.orchestra.run.smithi083.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-06T07:19:18.781 INFO:teuthology.orchestra.run.smithi083.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-06T07:19:18.781 INFO:teuthology.orchestra.run.smithi083.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-06T07:19:18.781 INFO:teuthology.orchestra.run.smithi083.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-06T07:19:18.781 INFO:teuthology.orchestra.run.smithi083.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-06T07:19:18.782 INFO:teuthology.orchestra.run.smithi083.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-06T07:19:18.782 INFO:teuthology.orchestra.run.smithi083.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-06T07:19:18.782 INFO:teuthology.orchestra.run.smithi083.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-06T07:19:18.782 INFO:teuthology.orchestra.run.smithi083.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-06T07:19:18.782 INFO:teuthology.orchestra.run.smithi083.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-03-06T07:19:18.782 INFO:teuthology.orchestra.run.smithi083.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-03-06T07:19:18.782 INFO:teuthology.orchestra.run.smithi083.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-06T07:19:18.782 INFO:teuthology.orchestra.run.smithi083.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-06T07:19:18.782 INFO:teuthology.orchestra.run.smithi083.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-06T07:19:18.782 INFO:teuthology.orchestra.run.smithi083.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-06T07:19:18.782 INFO:teuthology.orchestra.run.smithi083.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-06T07:19:18.782 INFO:teuthology.orchestra.run.smithi083.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-06T07:19:18.782 INFO:teuthology.orchestra.run.smithi083.stdout:Installing weak dependencies: 2024-03-06T07:19:18.782 INFO:teuthology.orchestra.run.smithi083.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-03-06T07:19:18.783 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:19:18.783 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-06T07:19:18.783 INFO:teuthology.orchestra.run.smithi083.stdout:============================================================================================ 2024-03-06T07:19:18.783 INFO:teuthology.orchestra.run.smithi083.stdout:Install 150 Packages 2024-03-06T07:19:18.783 INFO:teuthology.orchestra.run.smithi083.stdout:Upgrade 26 Packages 2024-03-06T07:19:18.783 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:19:18.783 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 224 M 2024-03-06T07:19:18.783 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-06T07:19:19.355 INFO:teuthology.orchestra.run.smithi083.stdout:(1/176): ceph-19.0.0-1739.g1bc250c7.el9.x86_64. 18 kB/s | 6.6 kB 00:00 2024-03-06T07:19:20.747 INFO:teuthology.orchestra.run.smithi083.stdout:(2/176): ceph-fuse-19.0.0-1739.g1bc250c7.el9.x8 638 kB/s | 888 kB 00:01 2024-03-06T07:19:20.980 INFO:teuthology.orchestra.run.smithi083.stdout:(3/176): ceph-immutable-object-cache-19.0.0-173 622 kB/s | 145 kB 00:00 2024-03-06T07:19:22.472 INFO:teuthology.orchestra.run.smithi083.stdout:(4/176): ceph-mds-19.0.0-1739.g1bc250c7.el9.x86 1.5 MB/s | 2.3 MB 00:01 2024-03-06T07:19:23.331 INFO:teuthology.orchestra.run.smithi083.stdout:(5/176): ceph-mgr-19.0.0-1739.g1bc250c7.el9.x86 1.7 MB/s | 1.5 MB 00:00 2024-03-06T07:19:23.550 INFO:teuthology.orchestra.run.smithi183.stdout:(10/176): ceph-radosgw-19.0.0-1739.g1bc250c7.el 884 kB/s | 9.8 MB 00:11 2024-03-06T07:19:23.942 INFO:teuthology.orchestra.run.smithi183.stdout:(11/176): cephfs-mirror-19.0.0-1739.g1bc250c7.e 563 kB/s | 220 kB 00:00 2024-03-06T07:19:24.033 INFO:teuthology.orchestra.run.smithi183.stdout:(12/176): cephfs-shell-19.0.0-1739.g1bc250c7.el 295 kB/s | 27 kB 00:00 2024-03-06T07:19:24.134 INFO:teuthology.orchestra.run.smithi183.stdout:(13/176): libcephfs-devel-19.0.0-1739.g1bc250c7 313 kB/s | 31 kB 00:00 2024-03-06T07:19:25.281 INFO:teuthology.orchestra.run.smithi083.stdout:(6/176): ceph-base-19.0.0-1739.g1bc250c7.el9.x8 869 kB/s | 5.3 MB 00:06 2024-03-06T07:19:25.484 INFO:teuthology.orchestra.run.smithi183.stdout:(14/176): libcephfs2-19.0.0-1739.g1bc250c7.el9. 553 kB/s | 747 kB 00:01 2024-03-06T07:19:25.498 INFO:teuthology.orchestra.run.smithi083.stdout:(7/176): ceph-mon-19.0.0-1739.g1bc250c7.el9.x86 2.2 MB/s | 4.7 MB 00:02 2024-03-06T07:19:25.817 INFO:teuthology.orchestra.run.smithi183.stdout:(15/176): libcephsqlite-19.0.0-1739.g1bc250c7.e 490 kB/s | 163 kB 00:00 2024-03-06T07:19:26.059 INFO:teuthology.orchestra.run.smithi183.stdout:(16/176): librados-devel-19.0.0-1739.g1bc250c7. 524 kB/s | 127 kB 00:00 2024-03-06T07:19:26.759 INFO:teuthology.orchestra.run.smithi183.stdout:(17/176): libradosstriper1-19.0.0-1739.g1bc250c 716 kB/s | 501 kB 00:00 2024-03-06T07:19:31.490 INFO:teuthology.orchestra.run.smithi083.stdout:(8/176): ceph-radosgw-19.0.0-1739.g1bc250c7.el9 1.6 MB/s | 9.8 MB 00:05 2024-03-06T07:19:31.573 INFO:teuthology.orchestra.run.smithi083.stdout:(9/176): ceph-selinux-19.0.0-1739.g1bc250c7.el9 303 kB/s | 25 kB 00:00 2024-03-06T07:19:31.667 INFO:teuthology.orchestra.run.smithi183.stdout:(18/176): librgw2-19.0.0-1739.g1bc250c7.el9.x86 1.0 MB/s | 5.1 MB 00:04 2024-03-06T07:19:31.759 INFO:teuthology.orchestra.run.smithi183.stdout:(19/176): python3-ceph-argparse-19.0.0-1739.g1b 493 kB/s | 45 kB 00:00 2024-03-06T07:19:32.034 INFO:teuthology.orchestra.run.smithi183.stdout:(20/176): python3-ceph-common-19.0.0-1739.g1bc2 478 kB/s | 131 kB 00:00 2024-03-06T07:19:32.201 INFO:teuthology.orchestra.run.smithi183.stdout:(21/176): python3-cephfs-19.0.0-1739.g1bc250c7. 970 kB/s | 162 kB 00:00 2024-03-06T07:19:32.535 INFO:teuthology.orchestra.run.smithi183.stdout:(22/176): python3-rados-19.0.0-1739.g1bc250c7.e 963 kB/s | 321 kB 00:00 2024-03-06T07:19:32.868 INFO:teuthology.orchestra.run.smithi183.stdout:(23/176): python3-rbd-19.0.0-1739.g1bc250c7.el9 894 kB/s | 298 kB 00:00 2024-03-06T07:19:33.044 INFO:teuthology.orchestra.run.smithi183.stdout:(24/176): python3-rgw-19.0.0-1739.g1bc250c7.el9 567 kB/s | 99 kB 00:00 2024-03-06T07:19:33.144 INFO:teuthology.orchestra.run.smithi183.stdout:(25/176): rbd-fuse-19.0.0-1739.g1bc250c7.el9.x8 852 kB/s | 85 kB 00:00 2024-03-06T07:19:33.994 INFO:teuthology.orchestra.run.smithi183.stdout:(26/176): ceph-osd-19.0.0-1739.g1bc250c7.el9.x8 756 kB/s | 17 MB 00:22 2024-03-06T07:19:34.278 INFO:teuthology.orchestra.run.smithi183.stdout:(27/176): rbd-nbd-19.0.0-1739.g1bc250c7.el9.x86 602 kB/s | 171 kB 00:00 2024-03-06T07:19:34.395 INFO:teuthology.orchestra.run.smithi183.stdout:(28/176): ceph-grafana-dashboards-19.0.0-1739.g 270 kB/s | 31 kB 00:00 2024-03-06T07:19:34.629 INFO:teuthology.orchestra.run.smithi183.stdout:(29/176): ceph-mgr-cephadm-19.0.0-1739.g1bc250c 591 kB/s | 138 kB 00:00 2024-03-06T07:19:35.729 INFO:teuthology.orchestra.run.smithi183.stdout:(30/176): rbd-mirror-19.0.0-1739.g1bc250c7.el9. 1.2 MB/s | 3.1 MB 00:02 2024-03-06T07:19:36.813 INFO:teuthology.orchestra.run.smithi183.stdout:(31/176): ceph-mgr-dashboard-19.0.0-1739.g1bc25 835 kB/s | 1.8 MB 00:02 2024-03-06T07:19:37.146 INFO:teuthology.orchestra.run.smithi183.stdout:(32/176): ceph-mgr-modules-core-19.0.0-1739.g1b 745 kB/s | 248 kB 00:00 2024-03-06T07:19:37.263 INFO:teuthology.orchestra.run.smithi183.stdout:(33/176): ceph-mgr-rook-19.0.0-1739.g1bc250c7.e 423 kB/s | 49 kB 00:00 2024-03-06T07:19:37.372 INFO:teuthology.orchestra.run.smithi183.stdout:(34/176): ceph-prometheus-alerts-19.0.0-1739.g1 142 kB/s | 15 kB 00:00 2024-03-06T07:19:37.688 INFO:teuthology.orchestra.run.smithi183.stdout:(35/176): ceph-volume-19.0.0-1739.g1bc250c7.el9 833 kB/s | 264 kB 00:00 2024-03-06T07:19:38.430 INFO:teuthology.orchestra.run.smithi183.stdout:(36/176): cephadm-19.0.0-1739.g1bc250c7.el9.noa 1.0 MB/s | 748 kB 00:00 2024-03-06T07:19:38.547 INFO:teuthology.orchestra.run.smithi183.stdout:(37/176): cephfs-top-19.0.0-1739.g1bc250c7.el9. 157 kB/s | 18 kB 00:00 2024-03-06T07:19:38.689 INFO:teuthology.orchestra.run.smithi183.stdout:(38/176): automake-1.16.2-8.el9.noarch.rpm 4.6 MB/s | 662 kB 00:00 2024-03-06T07:19:38.781 INFO:teuthology.orchestra.run.smithi183.stdout:(39/176): bison-3.7.4-5.el9.x86_64.rpm 9.8 MB/s | 920 kB 00:00 2024-03-06T07:19:38.815 INFO:teuthology.orchestra.run.smithi183.stdout:(40/176): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 106 kB 00:00 2024-03-06T07:19:38.848 INFO:teuthology.orchestra.run.smithi183.stdout:(41/176): elfutils-libelf-devel-0.190-2.el9.x86 614 kB/s | 20 kB 00:00 2024-03-06T07:19:38.915 INFO:teuthology.orchestra.run.smithi183.stdout:(42/176): flex-2.6.4-9.el9.x86_64.rpm 4.5 MB/s | 307 kB 00:00 2024-03-06T07:19:38.974 INFO:teuthology.orchestra.run.smithi183.stdout:(43/176): flexiblas-3.0.4-8.el9.x86_64.rpm 544 kB/s | 32 kB 00:00 2024-03-06T07:19:39.174 INFO:teuthology.orchestra.run.smithi183.stdout:(44/176): flexiblas-netlib-3.0.4-8.el9.x86_64.r 15 MB/s | 3.0 MB 00:00 2024-03-06T07:19:39.199 INFO:teuthology.orchestra.run.smithi183.stdout:(45/176): flexiblas-openblas-openmp-3.0.4-8.el9 692 kB/s | 17 kB 00:00 2024-03-06T07:19:39.224 INFO:teuthology.orchestra.run.smithi183.stdout:(46/176): libaio-devel-0.3.111-13.el9.x86_64.rp 407 kB/s | 10 kB 00:00 2024-03-06T07:19:39.267 INFO:teuthology.orchestra.run.smithi183.stdout:(47/176): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-06T07:19:39.292 INFO:teuthology.orchestra.run.smithi183.stdout:(48/176): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.7 MB/s | 44 kB 00:00 2024-03-06T07:19:39.342 INFO:teuthology.orchestra.run.smithi183.stdout:(49/176): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-06T07:19:39.384 INFO:teuthology.orchestra.run.smithi183.stdout:(50/176): libtool-2.4.6-45.el9.x86_64.rpm 14 MB/s | 578 kB 00:00 2024-03-06T07:19:39.426 INFO:teuthology.orchestra.run.smithi183.stdout:(51/176): libuuid-devel-2.37.4-18.el9.x86_64.rp 589 kB/s | 24 kB 00:00 2024-03-06T07:19:39.476 INFO:teuthology.orchestra.run.smithi183.stdout:(52/176): libzstd-devel-1.5.1-2.el9.x86_64.rpm 926 kB/s | 46 kB 00:00 2024-03-06T07:19:39.551 INFO:teuthology.orchestra.run.smithi183.stdout:(53/176): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-06T07:19:39.619 INFO:teuthology.orchestra.run.smithi183.stdout:(54/176): lua-5.4.4-4.el9.x86_64.rpm 2.7 MB/s | 187 kB 00:00 2024-03-06T07:19:39.677 INFO:teuthology.orchestra.run.smithi183.stdout:(55/176): openblas-0.3.21-2.el9.x86_64.rpm 585 kB/s | 34 kB 00:00 2024-03-06T07:19:39.853 INFO:teuthology.orchestra.run.smithi183.stdout:(56/176): openblas-openmp-0.3.21-2.el9.x86_64.r 27 MB/s | 4.7 MB 00:00 2024-03-06T07:19:39.944 INFO:teuthology.orchestra.run.smithi183.stdout:(57/176): openssl-devel-3.0.7-27.el9.x86_64.rpm 33 MB/s | 3.0 MB 00:00 2024-03-06T07:19:40.011 INFO:teuthology.orchestra.run.smithi183.stdout:(58/176): postgresql-13.11-1.el9.x86_64.rpm 23 MB/s | 1.5 MB 00:00 2024-03-06T07:19:40.070 INFO:teuthology.orchestra.run.smithi183.stdout:(59/176): postgresql-contrib-13.11-1.el9.x86_64 14 MB/s | 811 kB 00:00 2024-03-06T07:19:40.112 INFO:teuthology.orchestra.run.smithi183.stdout:(60/176): postgresql-private-libs-13.11-1.el9.x 3.1 MB/s | 134 kB 00:00 2024-03-06T07:19:40.204 INFO:teuthology.orchestra.run.smithi183.stdout:(61/176): ceph-mgr-diskprediction-local-19.0.0- 1.7 MB/s | 7.4 MB 00:04 2024-03-06T07:19:40.338 INFO:teuthology.orchestra.run.smithi183.stdout:(62/176): postgresql-server-13.11-1.el9.x86_64. 25 MB/s | 5.7 MB 00:00 2024-03-06T07:19:40.371 INFO:teuthology.orchestra.run.smithi183.stdout:(63/176): protobuf-3.14.0-13.el9.x86_64.rpm 6.1 MB/s | 1.0 MB 00:00 2024-03-06T07:19:40.396 INFO:teuthology.orchestra.run.smithi183.stdout:(64/176): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-06T07:19:40.539 INFO:teuthology.orchestra.run.smithi183.stdout:(65/176): python3-babel-2.9.1-2.el9.noarch.rpm 35 MB/s | 5.8 MB 00:00 2024-03-06T07:19:40.555 INFO:teuthology.orchestra.run.smithi183.stdout:(66/176): python3-jinja2-2.11.3-5.el9.noarch.rp 1.4 MB/s | 227 kB 00:00 2024-03-06T07:19:40.581 INFO:teuthology.orchestra.run.smithi183.stdout:(67/176): python3-jsonpatch-1.21-16.el9.noarch. 590 kB/s | 24 kB 00:00 2024-03-06T07:19:40.606 INFO:teuthology.orchestra.run.smithi183.stdout:(68/176): python3-jsonpointer-2.0-4.el9.noarch. 360 kB/s | 18 kB 00:00 2024-03-06T07:19:40.639 INFO:teuthology.orchestra.run.smithi183.stdout:(69/176): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-03-06T07:19:40.664 INFO:teuthology.orchestra.run.smithi183.stdout:(70/176): python3-markupsafe-1.1.1-12.el9.x86_6 541 kB/s | 32 kB 00:00 2024-03-06T07:19:40.773 INFO:teuthology.orchestra.run.smithi183.stdout:(71/176): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-06T07:19:40.807 INFO:teuthology.orchestra.run.smithi183.stdout:(72/176): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.5 MB/s | 217 kB 00:00 2024-03-06T07:19:40.832 INFO:teuthology.orchestra.run.smithi183.stdout:(73/176): python3-oauthlib-3.1.1-5.el9.noarch.r 2.9 MB/s | 174 kB 00:00 2024-03-06T07:19:40.860 INFO:teuthology.orchestra.run.smithi183.stdout:(74/176): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-06T07:19:40.891 INFO:teuthology.orchestra.run.smithi183.stdout:(75/176): python3-prettytable-0.7.2-27.el9.noar 707 kB/s | 41 kB 00:00 2024-03-06T07:19:40.916 INFO:teuthology.orchestra.run.smithi183.stdout:(76/176): python3-protobuf-3.14.0-13.el9.noarch 4.0 MB/s | 238 kB 00:00 2024-03-06T07:19:40.941 INFO:teuthology.orchestra.run.smithi183.stdout:(77/176): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-06T07:19:40.964 INFO:teuthology.orchestra.run.smithi083.stdout:(10/176): ceph-osd-19.0.0-1739.g1bc250c7.el9.x8 1.1 MB/s | 17 MB 00:15 2024-03-06T07:19:40.966 INFO:teuthology.orchestra.run.smithi183.stdout:(78/176): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-06T07:19:40.992 INFO:teuthology.orchestra.run.smithi183.stdout:(79/176): python3-pytz-2021.1-5.el9.noarch.rpm 947 kB/s | 47 kB 00:00 2024-03-06T07:19:41.017 INFO:teuthology.orchestra.run.smithi183.stdout:(80/176): python3-requests-oauthlib-1.3.0-12.el 852 kB/s | 43 kB 00:00 2024-03-06T07:19:41.089 INFO:teuthology.orchestra.run.smithi083.stdout:(11/176): cephfs-mirror-19.0.0-1739.g1bc250c7.e 1.7 MB/s | 220 kB 00:00 2024-03-06T07:19:41.173 INFO:teuthology.orchestra.run.smithi083.stdout:(12/176): cephfs-shell-19.0.0-1739.g1bc250c7.el 326 kB/s | 27 kB 00:00 2024-03-06T07:19:41.226 INFO:teuthology.orchestra.run.smithi183.stdout:(81/176): python3-toml-0.10.2-6.el9.noarch.rpm 187 kB/s | 39 kB 00:00 2024-03-06T07:19:41.268 INFO:teuthology.orchestra.run.smithi083.stdout:(13/176): libcephfs-devel-19.0.0-1739.g1bc250c7 341 kB/s | 31 kB 00:00 2024-03-06T07:19:41.376 INFO:teuthology.orchestra.run.smithi183.stdout:(82/176): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-03-06T07:19:41.401 INFO:teuthology.orchestra.run.smithi183.stdout:(83/176): python3-wcwidth-0.2.5-8.el9.noarch.rp 237 kB/s | 41 kB 00:00 2024-03-06T07:19:41.427 INFO:teuthology.orchestra.run.smithi183.stdout:(84/176): qatlib-23.11.0-1.el9.x86_64.rpm 4.2 MB/s | 215 kB 00:00 2024-03-06T07:19:41.452 INFO:teuthology.orchestra.run.smithi183.stdout:(85/176): qatlib-service-23.11.0-1.el9.x86_64.r 675 kB/s | 34 kB 00:00 2024-03-06T07:19:41.469 INFO:teuthology.orchestra.run.smithi183.stdout:(86/176): qatzip-libs-1.1.2-2.el9.x86_64.rpm 1.0 MB/s | 43 kB 00:00 2024-03-06T07:19:41.495 INFO:teuthology.orchestra.run.smithi183.stdout:(87/176): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-03-06T07:19:41.511 INFO:teuthology.orchestra.run.smithi183.stdout:(88/176): uuid-1.6.2-55.el9.x86_64.rpm 1.3 MB/s | 56 kB 00:00 2024-03-06T07:19:41.537 INFO:teuthology.orchestra.run.smithi183.stdout:(89/176): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 2.0 MB/s | 87 kB 00:00 2024-03-06T07:19:41.553 INFO:teuthology.orchestra.run.smithi183.stdout:(90/176): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-06T07:19:41.570 INFO:teuthology.orchestra.run.smithi183.stdout:(91/176): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-06T07:19:41.604 INFO:teuthology.orchestra.run.smithi183.stdout:(92/176): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-06T07:19:41.621 INFO:teuthology.orchestra.run.smithi183.stdout:(93/176): jq-1.6-16.el9.x86_64.rpm 3.6 MB/s | 186 kB 00:00 2024-03-06T07:19:41.648 INFO:teuthology.orchestra.run.smithi083.stdout:(14/176): libcephfs2-19.0.0-1739.g1bc250c7.el9. 1.9 MB/s | 747 kB 00:00 2024-03-06T07:19:41.671 INFO:teuthology.orchestra.run.smithi183.stdout:(94/176): libgfortran-11.4.1-3.el9.x86_64.rpm 12 MB/s | 802 kB 00:00 2024-03-06T07:19:41.696 INFO:teuthology.orchestra.run.smithi183.stdout:(95/176): libquadmath-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 194 kB 00:00 2024-03-06T07:19:41.747 INFO:teuthology.orchestra.run.smithi183.stdout:(96/176): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.8 MB/s | 217 kB 00:00 2024-03-06T07:19:41.855 INFO:teuthology.orchestra.run.smithi183.stdout:(97/176): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.5 MB/s | 241 kB 00:00 2024-03-06T07:19:41.881 INFO:teuthology.orchestra.run.smithi183.stdout:(98/176): python3-cryptography-36.0.1-4.el9.x86 8.6 MB/s | 1.1 MB 00:00 2024-03-06T07:19:41.924 INFO:teuthology.orchestra.run.smithi083.stdout:(15/176): libcephsqlite-19.0.0-1739.g1bc250c7.e 594 kB/s | 163 kB 00:00 2024-03-06T07:19:41.931 INFO:teuthology.orchestra.run.smithi183.stdout:(99/176): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2024-03-06T07:19:41.956 INFO:teuthology.orchestra.run.smithi183.stdout:(100/176): python3-pycparser-2.20-6.el9.noarch. 1.6 MB/s | 124 kB 00:00 2024-03-06T07:19:41.981 INFO:teuthology.orchestra.run.smithi183.stdout:(101/176): python3-requests-2.25.1-8.el9.noarch 2.2 MB/s | 113 kB 00:00 2024-03-06T07:19:42.007 INFO:teuthology.orchestra.run.smithi183.stdout:(102/176): python3-urllib3-1.26.5-5.el9.noarch. 3.7 MB/s | 187 kB 00:00 2024-03-06T07:19:42.082 INFO:teuthology.orchestra.run.smithi083.stdout:(16/176): librados-devel-19.0.0-1739.g1bc250c7. 801 kB/s | 127 kB 00:00 2024-03-06T07:19:42.098 INFO:teuthology.orchestra.run.smithi183.stdout:(103/176): lua-devel-5.4.4-4.el9.x86_64.rpm 244 kB/s | 22 kB 00:00 2024-03-06T07:19:42.149 INFO:teuthology.orchestra.run.smithi183.stdout:(104/176): python3-devel-3.9.17-2.el9.x86_64.rp 1.4 MB/s | 245 kB 00:00 2024-03-06T07:19:42.533 INFO:teuthology.orchestra.run.smithi083.stdout:(17/176): libradosstriper1-19.0.0-1739.g1bc250c 1.1 MB/s | 501 kB 00:00 2024-03-06T07:19:42.691 INFO:teuthology.orchestra.run.smithi183.stdout:(105/176): ceph-test-19.0.0-1739.g1bc250c7.el9. 1.8 MB/s | 47 MB 00:26 2024-03-06T07:19:42.891 INFO:teuthology.orchestra.run.smithi183.stdout:(106/176): abseil-cpp-20211102.0-3.el9.x86_64.r 739 kB/s | 548 kB 00:00 2024-03-06T07:19:42.917 INFO:teuthology.orchestra.run.smithi183.stdout:(107/176): gperftools-libs-2.9.1-2.el9.x86_64.r 1.3 MB/s | 309 kB 00:00 2024-03-06T07:19:42.950 INFO:teuthology.orchestra.run.smithi183.stdout:(108/176): grpc-data-1.46.7-10.el9.noarch.rpm 335 kB/s | 19 kB 00:00 2024-03-06T07:19:43.119 INFO:teuthology.orchestra.run.smithi183.stdout:(109/176): libarrow-9.0.0-10.el9.x86_64.rpm 22 MB/s | 4.4 MB 00:00 2024-03-06T07:19:43.142 INFO:teuthology.orchestra.run.smithi183.stdout:(110/176): libarrow-doc-9.0.0-10.el9.noarch.rpm 135 kB/s | 26 kB 00:00 2024-03-06T07:19:43.175 INFO:teuthology.orchestra.run.smithi183.stdout:(111/176): liboath-2.6.7-2.el9.x86_64.rpm 831 kB/s | 49 kB 00:00 2024-03-06T07:19:43.201 INFO:teuthology.orchestra.run.smithi183.stdout:(112/176): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-06T07:19:43.234 INFO:teuthology.orchestra.run.smithi183.stdout:(113/176): luarocks-3.9.2-1.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-03-06T07:19:43.276 INFO:teuthology.orchestra.run.smithi183.stdout:(114/176): parquet-libs-9.0.0-10.el9.x86_64.rpm 11 MB/s | 838 kB 00:00 2024-03-06T07:19:43.301 INFO:teuthology.orchestra.run.smithi183.stdout:(115/176): python3-asyncssh-2.13.2-4.el9.noarch 8.0 MB/s | 548 kB 00:00 2024-03-06T07:19:43.335 INFO:teuthology.orchestra.run.smithi183.stdout:(116/176): python3-bcrypt-3.2.2-1.el9.x86_64.rp 752 kB/s | 43 kB 00:00 2024-03-06T07:19:43.361 INFO:teuthology.orchestra.run.smithi183.stdout:(117/176): python3-cachetools-4.2.4-1.el9.noarc 545 kB/s | 32 kB 00:00 2024-03-06T07:19:43.394 INFO:teuthology.orchestra.run.smithi183.stdout:(118/176): python3-certifi-2021.10.8-2.el9.noar 255 kB/s | 15 kB 00:00 2024-03-06T07:19:43.420 INFO:teuthology.orchestra.run.smithi183.stdout:(119/176): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-03-06T07:19:43.461 INFO:teuthology.orchestra.run.smithi183.stdout:(120/176): python3-cherrypy-18.6.1-2.el9.noarch 5.2 MB/s | 358 kB 00:00 2024-03-06T07:19:43.487 INFO:teuthology.orchestra.run.smithi183.stdout:(121/176): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.4 MB/s | 303 kB 00:00 2024-03-06T07:19:43.520 INFO:teuthology.orchestra.run.smithi183.stdout:(122/176): python3-colorama-0.4.4-12.el9.noarch 608 kB/s | 35 kB 00:00 2024-03-06T07:19:43.545 INFO:teuthology.orchestra.run.smithi183.stdout:(123/176): python3-google-auth-2.28.0-1.el9.noa 3.4 MB/s | 203 kB 00:00 2024-03-06T07:19:43.637 INFO:teuthology.orchestra.run.smithi183.stdout:(124/176): python3-grpcio-1.46.7-10.el9.x86_64. 17 MB/s | 2.0 MB 00:00 2024-03-06T07:19:43.663 INFO:teuthology.orchestra.run.smithi183.stdout:(125/176): python3-grpcio-tools-1.46.7-10.el9.x 1.2 MB/s | 144 kB 00:00 2024-03-06T07:19:43.696 INFO:teuthology.orchestra.run.smithi183.stdout:(126/176): python3-jaraco-8.2.1-3.el9.noarch.rp 183 kB/s | 11 kB 00:00 2024-03-06T07:19:43.713 INFO:teuthology.orchestra.run.smithi183.stdout:(127/176): python3-jaraco-classes-3.2.1-5.el9.n 352 kB/s | 18 kB 00:00 2024-03-06T07:19:43.747 INFO:teuthology.orchestra.run.smithi183.stdout:(128/176): python3-jaraco-collections-3.0.0-8.e 460 kB/s | 23 kB 00:00 2024-03-06T07:19:43.764 INFO:teuthology.orchestra.run.smithi183.stdout:(129/176): python3-jaraco-functools-3.5.0-2.el9 386 kB/s | 19 kB 00:00 2024-03-06T07:19:43.806 INFO:teuthology.orchestra.run.smithi183.stdout:(130/176): python3-jaraco-text-3.2.0-6.el9.noar 336 kB/s | 20 kB 00:00 2024-03-06T07:19:43.848 INFO:teuthology.orchestra.run.smithi183.stdout:(131/176): python3-kubernetes-26.1.0-2.el9.noar 12 MB/s | 1.0 MB 00:00 2024-03-06T07:19:43.873 INFO:teuthology.orchestra.run.smithi183.stdout:(132/176): python3-logutils-0.3.5-21.el9.noarch 702 kB/s | 46 kB 00:00 2024-03-06T07:19:43.906 INFO:teuthology.orchestra.run.smithi183.stdout:(133/176): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2024-03-06T07:19:43.932 INFO:teuthology.orchestra.run.smithi183.stdout:(134/176): python3-natsort-7.1.1-5.el9.noarch.r 982 kB/s | 58 kB 00:00 2024-03-06T07:19:43.974 INFO:teuthology.orchestra.run.smithi183.stdout:(135/176): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-03-06T07:19:43.999 INFO:teuthology.orchestra.run.smithi183.stdout:(136/176): python3-portend-3.1.0-2.el9.noarch.r 245 kB/s | 16 kB 00:00 2024-03-06T07:19:44.033 INFO:teuthology.orchestra.run.smithi183.stdout:(137/176): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.5 MB/s | 90 kB 00:00 2024-03-06T07:19:44.058 INFO:teuthology.orchestra.run.smithi183.stdout:(138/176): python3-pyperclip-1.8.2-2.el9.noarch 484 kB/s | 28 kB 00:00 2024-03-06T07:19:44.091 INFO:teuthology.orchestra.run.smithi183.stdout:(139/176): python3-repoze-lru-0.7-16.el9.noarch 527 kB/s | 31 kB 00:00 2024-03-06T07:19:44.117 INFO:teuthology.orchestra.run.smithi183.stdout:(140/176): python3-routes-2.5.1-5.el9.noarch.rp 3.1 MB/s | 188 kB 00:00 2024-03-06T07:19:44.150 INFO:teuthology.orchestra.run.smithi183.stdout:(141/176): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-03-06T07:19:44.175 INFO:teuthology.orchestra.run.smithi183.stdout:(142/176): python3-tempora-5.0.0-2.el9.noarch.r 611 kB/s | 36 kB 00:00 2024-03-06T07:19:44.209 INFO:teuthology.orchestra.run.smithi183.stdout:(143/176): python3-typing-extensions-4.4.0-2.el 868 kB/s | 51 kB 00:00 2024-03-06T07:19:44.243 INFO:teuthology.orchestra.run.smithi183.stdout:(144/176): python3-webob-1.8.7-6.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2024-03-06T07:19:44.268 INFO:teuthology.orchestra.run.smithi183.stdout:(145/176): python3-websocket-client-1.2.3-2.el9 1.5 MB/s | 90 kB 00:00 2024-03-06T07:19:44.310 INFO:teuthology.orchestra.run.smithi183.stdout:(146/176): python3-werkzeug-2.0.3-3.el9.1.noarc 6.2 MB/s | 427 kB 00:00 2024-03-06T07:19:44.335 INFO:teuthology.orchestra.run.smithi183.stdout:(147/176): python3-zc-lockfile-2.0-10.el9.noarc 297 kB/s | 20 kB 00:00 2024-03-06T07:19:44.369 INFO:teuthology.orchestra.run.smithi183.stdout:(148/176): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-03-06T07:19:44.436 INFO:teuthology.orchestra.run.smithi183.stdout:(149/176): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-03-06T07:19:45.203 INFO:teuthology.orchestra.run.smithi183.stdout:(150/176): protobuf-compiler-3.14.0-13.el9.x86_ 278 kB/s | 863 kB 00:03 2024-03-06T07:19:45.236 INFO:teuthology.orchestra.run.smithi183.stdout:(151/176): libacl-devel-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 68 kB 00:00 2024-03-06T07:19:45.353 INFO:teuthology.orchestra.run.smithi183.stdout:(152/176): NetworkManager-1.46.0-1.el9.x86_64.r 20 MB/s | 2.3 MB 00:00 2024-03-06T07:19:45.429 INFO:teuthology.orchestra.run.smithi183.stdout:(153/176): NetworkManager-libnm-1.46.0-1.el9.x8 24 MB/s | 1.8 MB 00:00 2024-03-06T07:19:45.470 INFO:teuthology.orchestra.run.smithi183.stdout:(154/176): NetworkManager-team-1.46.0-1.el9.x86 996 kB/s | 41 kB 00:00 2024-03-06T07:19:45.512 INFO:teuthology.orchestra.run.smithi183.stdout:(155/176): NetworkManager-tui-1.46.0-1.el9.x86_ 5.8 MB/s | 247 kB 00:00 2024-03-06T07:19:45.546 INFO:teuthology.orchestra.run.smithi183.stdout:(156/176): acl-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 69 kB 00:00 2024-03-06T07:19:45.579 INFO:teuthology.orchestra.run.smithi183.stdout:(157/176): elfutils-debuginfod-client-0.190-2.e 1.0 MB/s | 35 kB 00:00 2024-03-06T07:19:45.613 INFO:teuthology.orchestra.run.smithi183.stdout:(158/176): elfutils-libelf-0.190-2.el9.x86_64.r 5.6 MB/s | 191 kB 00:00 2024-03-06T07:19:45.655 INFO:teuthology.orchestra.run.smithi183.stdout:(159/176): elfutils-libs-0.190-2.el9.x86_64.rpm 6.0 MB/s | 254 kB 00:00 2024-03-06T07:19:45.714 INFO:teuthology.orchestra.run.smithi183.stdout:(160/176): iproute-6.7.0-2.el9.x86_64.rpm 13 MB/s | 802 kB 00:00 2024-03-06T07:19:45.755 INFO:teuthology.orchestra.run.smithi183.stdout:(161/176): iproute-tc-6.7.0-2.el9.x86_64.rpm 10 MB/s | 433 kB 00:00 2024-03-06T07:19:45.797 INFO:teuthology.orchestra.run.smithi183.stdout:(162/176): libacl-2.3.1-4.el9.x86_64.rpm 526 kB/s | 22 kB 00:00 2024-03-06T07:19:45.831 INFO:teuthology.orchestra.run.smithi183.stdout:(163/176): libblkid-2.37.4-18.el9.x86_64.rpm 3.2 MB/s | 107 kB 00:00 2024-03-06T07:19:45.865 INFO:teuthology.orchestra.run.smithi183.stdout:(164/176): libfdisk-2.37.4-18.el9.x86_64.rpm 4.5 MB/s | 154 kB 00:00 2024-03-06T07:19:45.898 INFO:teuthology.orchestra.run.smithi183.stdout:(165/176): libmount-2.37.4-18.el9.x86_64.rpm 4.0 MB/s | 135 kB 00:00 2024-03-06T07:19:45.916 INFO:teuthology.orchestra.run.smithi083.stdout:(18/176): ceph-common-19.0.0-1739.g1bc250c7.el9 776 kB/s | 20 MB 00:26 2024-03-06T07:19:45.932 INFO:teuthology.orchestra.run.smithi183.stdout:(166/176): libsmartcols-2.37.4-18.el9.x86_64.rp 1.8 MB/s | 63 kB 00:00 2024-03-06T07:19:45.957 INFO:teuthology.orchestra.run.smithi183.stdout:(167/176): libuuid-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 28 kB 00:00 2024-03-06T07:19:45.983 INFO:teuthology.orchestra.run.smithi083.stdout:(19/176): librgw2-19.0.0-1739.g1bc250c7.el9.x86 1.5 MB/s | 5.1 MB 00:03 2024-03-06T07:19:46.016 INFO:teuthology.orchestra.run.smithi183.stdout:(168/176): openssl-3.0.7-27.el9.x86_64.rpm 20 MB/s | 1.2 MB 00:00 2024-03-06T07:19:46.083 INFO:teuthology.orchestra.run.smithi083.stdout:(20/176): python3-ceph-argparse-19.0.0-1739.g1b 271 kB/s | 45 kB 00:00 2024-03-06T07:19:46.100 INFO:teuthology.orchestra.run.smithi183.stdout:(169/176): openssl-libs-3.0.7-27.el9.x86_64.rpm 26 MB/s | 2.2 MB 00:00 2024-03-06T07:19:46.159 INFO:teuthology.orchestra.run.smithi083.stdout:(21/176): python3-ceph-common-19.0.0-1739.g1bc2 749 kB/s | 131 kB 00:00 2024-03-06T07:19:46.183 INFO:teuthology.orchestra.run.smithi183.stdout:(170/176): util-linux-2.37.4-18.el9.x86_64.rpm 26 MB/s | 2.2 MB 00:00 2024-03-06T07:19:46.225 INFO:teuthology.orchestra.run.smithi183.stdout:(171/176): util-linux-core-2.37.4-18.el9.x86_64 10 MB/s | 438 kB 00:00 2024-03-06T07:19:46.258 INFO:teuthology.orchestra.run.smithi183.stdout:(172/176): util-linux-user-2.37.4-18.el9.x86_64 898 kB/s | 30 kB 00:00 2024-03-06T07:19:46.300 INFO:teuthology.orchestra.run.smithi183.stdout:(173/176): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-06T07:19:46.334 INFO:teuthology.orchestra.run.smithi083.stdout:(22/176): python3-cephfs-19.0.0-1739.g1bc250c7. 645 kB/s | 162 kB 00:00 2024-03-06T07:19:46.351 INFO:teuthology.orchestra.run.smithi183.stdout:(174/176): xfsprogs-6.3.0-1.el9.x86_64.rpm 21 MB/s | 1.1 MB 00:00 2024-03-06T07:19:46.509 INFO:teuthology.orchestra.run.smithi183.stdout:(175/176): librados2-19.0.0-1739.g1bc250c7.el9. 1.6 MB/s | 3.4 MB 00:02 2024-03-06T07:19:46.527 INFO:teuthology.orchestra.run.smithi083.stdout:(23/176): python3-rados-19.0.0-1739.g1bc250c7.e 875 kB/s | 321 kB 00:00 2024-03-06T07:19:46.626 INFO:teuthology.orchestra.run.smithi083.stdout:(24/176): python3-rgw-19.0.0-1739.g1bc250c7.el9 994 kB/s | 99 kB 00:00 2024-03-06T07:19:46.718 INFO:teuthology.orchestra.run.smithi083.stdout:(25/176): python3-rbd-19.0.0-1739.g1bc250c7.el9 776 kB/s | 298 kB 00:00 2024-03-06T07:19:46.743 INFO:teuthology.orchestra.run.smithi083.stdout:(26/176): rbd-fuse-19.0.0-1739.g1bc250c7.el9.x8 729 kB/s | 85 kB 00:00 2024-03-06T07:19:46.902 INFO:teuthology.orchestra.run.smithi083.stdout:(27/176): rbd-nbd-19.0.0-1739.g1bc250c7.el9.x86 1.1 MB/s | 171 kB 00:00 2024-03-06T07:19:46.977 INFO:teuthology.orchestra.run.smithi083.stdout:(28/176): ceph-grafana-dashboards-19.0.0-1739.g 419 kB/s | 31 kB 00:00 2024-03-06T07:19:47.119 INFO:teuthology.orchestra.run.smithi083.stdout:(29/176): ceph-mgr-cephadm-19.0.0-1739.g1bc250c 977 kB/s | 138 kB 00:00 2024-03-06T07:19:47.193 INFO:teuthology.orchestra.run.smithi183.stdout:(176/176): librbd1-19.0.0-1739.g1bc250c7.el9.x8 1.1 MB/s | 3.1 MB 00:02 2024-03-06T07:19:47.198 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:19:47.198 INFO:teuthology.orchestra.run.smithi183.stdout:Total 4.9 MB/s | 224 MB 00:45 2024-03-06T07:19:47.242 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-06T07:19:47.555 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-06T07:19:47.555 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-06T07:19:47.653 INFO:teuthology.orchestra.run.smithi083.stdout:(30/176): ceph-test-19.0.0-1739.g1bc250c7.el9.x 2.9 MB/s | 47 MB 00:16 2024-03-06T07:19:48.453 INFO:teuthology.orchestra.run.smithi083.stdout:(31/176): ceph-mgr-dashboard-19.0.0-1739.g1bc25 1.3 MB/s | 1.8 MB 00:01 2024-03-06T07:19:48.620 INFO:teuthology.orchestra.run.smithi083.stdout:(32/176): ceph-mgr-modules-core-19.0.0-1739.g1b 1.5 MB/s | 248 kB 00:00 2024-03-06T07:19:48.887 INFO:teuthology.orchestra.run.smithi083.stdout:(33/176): ceph-mgr-rook-19.0.0-1739.g1bc250c7.e 185 kB/s | 49 kB 00:00 2024-03-06T07:19:48.987 INFO:teuthology.orchestra.run.smithi083.stdout:(34/176): ceph-prometheus-alerts-19.0.0-1739.g1 153 kB/s | 15 kB 00:00 2024-03-06T07:19:49.287 INFO:teuthology.orchestra.run.smithi083.stdout:(35/176): rbd-mirror-19.0.0-1739.g1bc250c7.el9. 1.2 MB/s | 3.1 MB 00:02 2024-03-06T07:19:49.346 INFO:teuthology.orchestra.run.smithi083.stdout:(36/176): ceph-volume-19.0.0-1739.g1bc250c7.el9 735 kB/s | 264 kB 00:00 2024-03-06T07:19:49.455 INFO:teuthology.orchestra.run.smithi083.stdout:(37/176): cephfs-top-19.0.0-1739.g1bc250c7.el9. 168 kB/s | 18 kB 00:00 2024-03-06T07:19:49.500 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-06T07:19:49.500 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-06T07:19:49.613 INFO:teuthology.orchestra.run.smithi083.stdout:(38/176): automake-1.16.2-8.el9.noarch.rpm 4.1 MB/s | 662 kB 00:00 2024-03-06T07:19:49.705 INFO:teuthology.orchestra.run.smithi083.stdout:(39/176): ceph-mgr-diskprediction-local-19.0.0- 3.6 MB/s | 7.4 MB 00:02 2024-03-06T07:19:49.747 INFO:teuthology.orchestra.run.smithi083.stdout:(40/176): bison-3.7.4-5.el9.x86_64.rpm 6.7 MB/s | 920 kB 00:00 2024-03-06T07:19:49.772 INFO:teuthology.orchestra.run.smithi083.stdout:(41/176): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 106 kB 00:00 2024-03-06T07:19:49.789 INFO:teuthology.orchestra.run.smithi083.stdout:(42/176): elfutils-libelf-devel-0.190-2.el9.x86 491 kB/s | 20 kB 00:00 2024-03-06T07:19:49.814 INFO:teuthology.orchestra.run.smithi083.stdout:(43/176): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-03-06T07:19:49.832 INFO:teuthology.orchestra.run.smithi083.stdout:(44/176): flexiblas-3.0.4-8.el9.x86_64.rpm 767 kB/s | 32 kB 00:00 2024-03-06T07:19:49.865 INFO:teuthology.orchestra.run.smithi083.stdout:(45/176): cephadm-19.0.0-1739.g1bc250c7.el9.noa 1.3 MB/s | 748 kB 00:00 2024-03-06T07:19:49.915 INFO:teuthology.orchestra.run.smithi083.stdout:(46/176): flexiblas-netlib-3.0.4-8.el9.x86_64.r 30 MB/s | 3.0 MB 00:00 2024-03-06T07:19:49.932 INFO:teuthology.orchestra.run.smithi083.stdout:(47/176): flexiblas-openblas-openmp-3.0.4-8.el9 174 kB/s | 17 kB 00:00 2024-03-06T07:19:49.957 INFO:teuthology.orchestra.run.smithi083.stdout:(48/176): libaio-devel-0.3.111-13.el9.x86_64.rp 112 kB/s | 10 kB 00:00 2024-03-06T07:19:49.983 INFO:teuthology.orchestra.run.smithi083.stdout:(49/176): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-06T07:19:50.008 INFO:teuthology.orchestra.run.smithi083.stdout:(50/176): librabbitmq-0.11.0-7.el9.x86_64.rpm 587 kB/s | 44 kB 00:00 2024-03-06T07:19:50.042 INFO:teuthology.orchestra.run.smithi083.stdout:(51/176): librdkafka-1.6.1-102.el9.x86_64.rpm 7.8 MB/s | 661 kB 00:00 2024-03-06T07:19:50.067 INFO:teuthology.orchestra.run.smithi083.stdout:(52/176): libtool-2.4.6-45.el9.x86_64.rpm 9.9 MB/s | 578 kB 00:00 2024-03-06T07:19:50.092 INFO:teuthology.orchestra.run.smithi083.stdout:(53/176): libuuid-devel-2.37.4-18.el9.x86_64.rp 296 kB/s | 24 kB 00:00 2024-03-06T07:19:50.117 INFO:teuthology.orchestra.run.smithi083.stdout:(54/176): libzstd-devel-1.5.1-2.el9.x86_64.rpm 624 kB/s | 46 kB 00:00 2024-03-06T07:19:50.143 INFO:teuthology.orchestra.run.smithi083.stdout:(55/176): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-06T07:19:50.168 INFO:teuthology.orchestra.run.smithi083.stdout:(56/176): lua-5.4.4-4.el9.x86_64.rpm 2.4 MB/s | 187 kB 00:00 2024-03-06T07:19:50.193 INFO:teuthology.orchestra.run.smithi083.stdout:(57/176): openblas-0.3.21-2.el9.x86_64.rpm 458 kB/s | 34 kB 00:00 2024-03-06T07:19:50.302 INFO:teuthology.orchestra.run.smithi083.stdout:(58/176): openssl-devel-3.0.7-27.el9.x86_64.rpm 22 MB/s | 3.0 MB 00:00 2024-03-06T07:19:50.369 INFO:teuthology.orchestra.run.smithi083.stdout:(59/176): openblas-openmp-0.3.21-2.el9.x86_64.r 24 MB/s | 4.7 MB 00:00 2024-03-06T07:19:50.402 INFO:teuthology.orchestra.run.smithi083.stdout:(60/176): postgresql-13.11-1.el9.x86_64.rpm 7.4 MB/s | 1.5 MB 00:00 2024-03-06T07:19:50.435 INFO:teuthology.orchestra.run.smithi083.stdout:(61/176): postgresql-contrib-13.11-1.el9.x86_64 6.1 MB/s | 811 kB 00:00 2024-03-06T07:19:50.460 INFO:teuthology.orchestra.run.smithi083.stdout:(62/176): postgresql-private-libs-13.11-1.el9.x 1.4 MB/s | 134 kB 00:00 2024-03-06T07:19:50.545 INFO:teuthology.orchestra.run.smithi083.stdout:(63/176): protobuf-3.14.0-13.el9.x86_64.rpm 9.3 MB/s | 1.0 MB 00:00 2024-03-06T07:19:50.620 INFO:teuthology.orchestra.run.smithi083.stdout:(64/176): postgresql-server-13.11-1.el9.x86_64. 31 MB/s | 5.7 MB 00:00 2024-03-06T07:19:50.645 INFO:teuthology.orchestra.run.smithi083.stdout:(65/176): python3-attrs-20.3.0-7.el9.noarch.rpm 427 kB/s | 78 kB 00:00 2024-03-06T07:19:50.754 INFO:teuthology.orchestra.run.smithi083.stdout:(66/176): python3-babel-2.9.1-2.el9.noarch.rpm 28 MB/s | 5.8 MB 00:00 2024-03-06T07:19:50.779 INFO:teuthology.orchestra.run.smithi083.stdout:(67/176): python3-jinja2-2.11.3-5.el9.noarch.rp 1.7 MB/s | 227 kB 00:00 2024-03-06T07:19:50.804 INFO:teuthology.orchestra.run.smithi083.stdout:(68/176): python3-jsonpatch-1.21-16.el9.noarch. 156 kB/s | 24 kB 00:00 2024-03-06T07:19:50.829 INFO:teuthology.orchestra.run.smithi083.stdout:(69/176): python3-jsonpointer-2.0-4.el9.noarch. 241 kB/s | 18 kB 00:00 2024-03-06T07:19:50.855 INFO:teuthology.orchestra.run.smithi083.stdout:(70/176): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-06T07:19:50.880 INFO:teuthology.orchestra.run.smithi083.stdout:(71/176): python3-markupsafe-1.1.1-12.el9.x86_6 420 kB/s | 32 kB 00:00 2024-03-06T07:19:50.963 INFO:teuthology.orchestra.run.smithi083.stdout:(72/176): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-06T07:19:50.989 INFO:teuthology.orchestra.run.smithi083.stdout:(73/176): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.0 MB/s | 217 kB 00:00 2024-03-06T07:19:51.014 INFO:teuthology.orchestra.run.smithi083.stdout:(74/176): python3-oauthlib-3.1.1-5.el9.noarch.r 1.3 MB/s | 174 kB 00:00 2024-03-06T07:19:51.039 INFO:teuthology.orchestra.run.smithi083.stdout:(75/176): python3-packaging-20.9-5.el9.noarch.r 918 kB/s | 69 kB 00:00 2024-03-06T07:19:51.064 INFO:teuthology.orchestra.run.smithi083.stdout:(76/176): python3-prettytable-0.7.2-27.el9.noar 821 kB/s | 41 kB 00:00 2024-03-06T07:19:51.089 INFO:teuthology.orchestra.run.smithi083.stdout:(77/176): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-03-06T07:19:51.116 INFO:teuthology.orchestra.run.smithi083.stdout:(78/176): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-06T07:19:51.140 INFO:teuthology.orchestra.run.smithi083.stdout:(79/176): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-06T07:19:51.157 INFO:teuthology.orchestra.run.smithi083.stdout:(80/176): python3-pytz-2021.1-5.el9.noarch.rpm 710 kB/s | 47 kB 00:00 2024-03-06T07:19:51.182 INFO:teuthology.orchestra.run.smithi083.stdout:(81/176): python3-requests-oauthlib-1.3.0-12.el 642 kB/s | 43 kB 00:00 2024-03-06T07:19:51.224 INFO:teuthology.orchestra.run.smithi083.stdout:(82/176): python3-toml-0.10.2-6.el9.noarch.rpm 579 kB/s | 39 kB 00:00 2024-03-06T07:19:51.374 INFO:teuthology.orchestra.run.smithi083.stdout:(83/176): python3-wcwidth-0.2.5-8.el9.noarch.rp 216 kB/s | 41 kB 00:00 2024-03-06T07:19:51.525 INFO:teuthology.orchestra.run.smithi083.stdout:(84/176): python3-scipy-1.6.2-8.el9.x86_64.rpm 43 MB/s | 16 MB 00:00 2024-03-06T07:19:51.558 INFO:teuthology.orchestra.run.smithi083.stdout:(85/176): qatlib-23.11.0-1.el9.x86_64.rpm 651 kB/s | 215 kB 00:00 2024-03-06T07:19:51.575 INFO:teuthology.orchestra.run.smithi083.stdout:(86/176): qatlib-service-23.11.0-1.el9.x86_64.r 169 kB/s | 34 kB 00:00 2024-03-06T07:19:51.592 INFO:teuthology.orchestra.run.smithi083.stdout:(87/176): qatzip-libs-1.1.2-2.el9.x86_64.rpm 1.3 MB/s | 43 kB 00:00 2024-03-06T07:19:51.617 INFO:teuthology.orchestra.run.smithi083.stdout:(88/176): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-06T07:19:51.634 INFO:teuthology.orchestra.run.smithi083.stdout:(89/176): uuid-1.6.2-55.el9.x86_64.rpm 972 kB/s | 56 kB 00:00 2024-03-06T07:19:51.659 INFO:teuthology.orchestra.run.smithi083.stdout:(90/176): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 2.0 MB/s | 87 kB 00:00 2024-03-06T07:19:51.676 INFO:teuthology.orchestra.run.smithi083.stdout:(91/176): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.0 MB/s | 63 kB 00:00 2024-03-06T07:19:51.693 INFO:teuthology.orchestra.run.smithi083.stdout:(92/176): zlib-devel-1.2.11-40.el9.x86_64.rpm 754 kB/s | 44 kB 00:00 2024-03-06T07:19:51.718 INFO:teuthology.orchestra.run.smithi083.stdout:(93/176): fuse-2.9.9-15.el9.x86_64.rpm 1.8 MB/s | 79 kB 00:00 2024-03-06T07:19:51.744 INFO:teuthology.orchestra.run.smithi083.stdout:(94/176): jq-1.6-16.el9.x86_64.rpm 2.7 MB/s | 186 kB 00:00 2024-03-06T07:19:51.769 INFO:teuthology.orchestra.run.smithi083.stdout:(95/176): libgfortran-11.4.1-3.el9.x86_64.rpm 10 MB/s | 802 kB 00:00 2024-03-06T07:19:51.794 INFO:teuthology.orchestra.run.smithi083.stdout:(96/176): libquadmath-11.4.1-3.el9.x86_64.rpm 3.9 MB/s | 194 kB 00:00 2024-03-06T07:19:51.819 INFO:teuthology.orchestra.run.smithi083.stdout:(97/176): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.9 MB/s | 217 kB 00:00 2024-03-06T07:19:51.844 INFO:teuthology.orchestra.run.smithi083.stdout:(98/176): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-03-06T07:19:51.887 INFO:teuthology.orchestra.run.smithi083.stdout:(99/176): python3-cryptography-36.0.1-4.el9.x86 17 MB/s | 1.1 MB 00:00 2024-03-06T07:19:51.912 INFO:teuthology.orchestra.run.smithi083.stdout:(100/176): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-03-06T07:19:51.937 INFO:teuthology.orchestra.run.smithi083.stdout:(101/176): python3-pycparser-2.20-6.el9.noarch. 1.3 MB/s | 124 kB 00:00 2024-03-06T07:19:51.962 INFO:teuthology.orchestra.run.smithi083.stdout:(102/176): python3-requests-2.25.1-8.el9.noarch 2.2 MB/s | 113 kB 00:00 2024-03-06T07:19:51.987 INFO:teuthology.orchestra.run.smithi083.stdout:(103/176): python3-urllib3-1.26.5-5.el9.noarch. 2.4 MB/s | 187 kB 00:00 2024-03-06T07:19:52.280 INFO:teuthology.orchestra.run.smithi083.stdout:(104/176): lua-devel-5.4.4-4.el9.x86_64.rpm 76 kB/s | 22 kB 00:00 2024-03-06T07:19:52.597 INFO:teuthology.orchestra.run.smithi083.stdout:(105/176): python3-devel-3.9.17-2.el9.x86_64.rp 373 kB/s | 245 kB 00:00 2024-03-06T07:19:52.622 INFO:teuthology.orchestra.run.smithi083.stdout:(106/176): abseil-cpp-20211102.0-3.el9.x86_64.r 1.6 MB/s | 548 kB 00:00 2024-03-06T07:19:52.680 INFO:teuthology.orchestra.run.smithi083.stdout:(107/176): grpc-data-1.46.7-10.el9.noarch.rpm 334 kB/s | 19 kB 00:00 2024-03-06T07:19:52.914 INFO:teuthology.orchestra.run.smithi083.stdout:(108/176): libarrow-9.0.0-10.el9.x86_64.rpm 19 MB/s | 4.4 MB 00:00 2024-03-06T07:19:52.981 INFO:teuthology.orchestra.run.smithi083.stdout:(109/176): gperftools-libs-2.9.1-2.el9.x86_64.r 804 kB/s | 309 kB 00:00 2024-03-06T07:19:53.006 INFO:teuthology.orchestra.run.smithi083.stdout:(110/176): libarrow-doc-9.0.0-10.el9.noarch.rpm 283 kB/s | 26 kB 00:00 2024-03-06T07:19:53.048 INFO:teuthology.orchestra.run.smithi083.stdout:(111/176): liboath-2.6.7-2.el9.x86_64.rpm 727 kB/s | 49 kB 00:00 2024-03-06T07:19:53.081 INFO:teuthology.orchestra.run.smithi083.stdout:(112/176): libunwind-1.6.2-1.el9.x86_64.rpm 897 kB/s | 67 kB 00:00 2024-03-06T07:19:53.149 INFO:teuthology.orchestra.run.smithi083.stdout:(113/176): luarocks-3.9.2-1.el9.noarch.rpm 1.5 MB/s | 151 kB 00:00 2024-03-06T07:19:53.190 INFO:teuthology.orchestra.run.smithi083.stdout:(114/176): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.5 MB/s | 838 kB 00:00 2024-03-06T07:19:53.291 INFO:teuthology.orchestra.run.smithi083.stdout:(115/176): python3-bcrypt-3.2.2-1.el9.x86_64.rp 435 kB/s | 43 kB 00:00 2024-03-06T07:19:53.316 INFO:teuthology.orchestra.run.smithi083.stdout:(116/176): python3-asyncssh-2.13.2-4.el9.noarch 3.2 MB/s | 548 kB 00:00 2024-03-06T07:19:53.366 INFO:teuthology.orchestra.run.smithi083.stdout:(117/176): python3-cachetools-4.2.4-1.el9.noarc 427 kB/s | 32 kB 00:00 2024-03-06T07:19:53.408 INFO:teuthology.orchestra.run.smithi083.stdout:(118/176): python3-certifi-2021.10.8-2.el9.noar 163 kB/s | 15 kB 00:00 2024-03-06T07:19:53.450 INFO:teuthology.orchestra.run.smithi083.stdout:(119/176): python3-cheroot-8.6.0-4.el9.noarch.r 2.0 MB/s | 172 kB 00:00 2024-03-06T07:19:53.484 INFO:teuthology.orchestra.run.smithi083.stdout:(120/176): python3-cherrypy-18.6.1-2.el9.noarch 4.7 MB/s | 358 kB 00:00 2024-03-06T07:19:53.526 INFO:teuthology.orchestra.run.smithi083.stdout:(121/176): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.9 MB/s | 303 kB 00:00 2024-03-06T07:19:53.559 INFO:teuthology.orchestra.run.smithi083.stdout:(122/176): python3-colorama-0.4.4-12.el9.noarch 468 kB/s | 35 kB 00:00 2024-03-06T07:19:53.601 INFO:teuthology.orchestra.run.smithi083.stdout:(123/176): python3-google-auth-2.28.0-1.el9.noa 2.6 MB/s | 203 kB 00:00 2024-03-06T07:19:53.685 INFO:teuthology.orchestra.run.smithi083.stdout:(124/176): python3-grpcio-tools-1.46.7-10.el9.x 1.7 MB/s | 144 kB 00:00 2024-03-06T07:19:53.727 INFO:teuthology.orchestra.run.smithi083.stdout:(125/176): python3-grpcio-1.46.7-10.el9.x86_64. 12 MB/s | 2.0 MB 00:00 2024-03-06T07:19:53.752 INFO:teuthology.orchestra.run.smithi083.stdout:(126/176): python3-jaraco-8.2.1-3.el9.noarch.rp 160 kB/s | 11 kB 00:00 2024-03-06T07:19:53.803 INFO:teuthology.orchestra.run.smithi083.stdout:(127/176): python3-jaraco-classes-3.2.1-5.el9.n 234 kB/s | 18 kB 00:00 2024-03-06T07:19:53.828 INFO:teuthology.orchestra.run.smithi083.stdout:(128/176): python3-jaraco-collections-3.0.0-8.e 307 kB/s | 23 kB 00:00 2024-03-06T07:19:53.878 INFO:teuthology.orchestra.run.smithi083.stdout:(129/176): python3-jaraco-functools-3.5.0-2.el9 258 kB/s | 19 kB 00:00 2024-03-06T07:19:53.904 INFO:teuthology.orchestra.run.smithi083.stdout:(130/176): python3-jaraco-text-3.2.0-6.el9.noar 260 kB/s | 20 kB 00:00 2024-03-06T07:19:53.987 INFO:teuthology.orchestra.run.smithi083.stdout:(131/176): python3-kubernetes-26.1.0-2.el9.noar 9.4 MB/s | 1.0 MB 00:00 2024-03-06T07:19:54.013 INFO:teuthology.orchestra.run.smithi083.stdout:(132/176): python3-logutils-0.3.5-21.el9.noarch 426 kB/s | 46 kB 00:00 2024-03-06T07:19:54.063 INFO:teuthology.orchestra.run.smithi083.stdout:(133/176): python3-more-itertools-8.12.0-2.el9. 1.0 MB/s | 79 kB 00:00 2024-03-06T07:19:54.096 INFO:teuthology.orchestra.run.smithi083.stdout:(134/176): python3-natsort-7.1.1-5.el9.noarch.r 690 kB/s | 58 kB 00:00 2024-03-06T07:19:54.138 INFO:teuthology.orchestra.run.smithi083.stdout:(135/176): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-03-06T07:19:54.172 INFO:teuthology.orchestra.run.smithi083.stdout:(136/176): python3-portend-3.1.0-2.el9.noarch.r 218 kB/s | 16 kB 00:00 2024-03-06T07:19:54.214 INFO:teuthology.orchestra.run.smithi083.stdout:(137/176): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-03-06T07:19:54.239 INFO:teuthology.orchestra.run.smithi083.stdout:(138/176): python3-pyperclip-1.8.2-2.el9.noarch 421 kB/s | 28 kB 00:00 2024-03-06T07:19:54.289 INFO:teuthology.orchestra.run.smithi083.stdout:(139/176): python3-repoze-lru-0.7-16.el9.noarch 409 kB/s | 31 kB 00:00 2024-03-06T07:19:54.376 INFO:teuthology.orchestra.run.smithi083.stdout:(140/176): python3-routes-2.5.1-5.el9.noarch.rp 1.3 MB/s | 188 kB 00:00 2024-03-06T07:19:54.390 INFO:teuthology.orchestra.run.smithi083.stdout:(141/176): python3-rsa-4.9-2.el9.noarch.rpm 589 kB/s | 59 kB 00:00 2024-03-06T07:19:54.440 INFO:teuthology.orchestra.run.smithi083.stdout:(142/176): protobuf-compiler-3.14.0-13.el9.x86_ 352 kB/s | 863 kB 00:02 2024-03-06T07:19:54.458 INFO:teuthology.orchestra.run.smithi083.stdout:(143/176): python3-tempora-5.0.0-2.el9.noarch.r 444 kB/s | 36 kB 00:00 2024-03-06T07:19:54.483 INFO:teuthology.orchestra.run.smithi083.stdout:(144/176): python3-typing-extensions-4.4.0-2.el 553 kB/s | 51 kB 00:00 2024-03-06T07:19:54.525 INFO:teuthology.orchestra.run.smithi083.stdout:(145/176): python3-websocket-client-1.2.3-2.el9 1.3 MB/s | 90 kB 00:00 2024-03-06T07:19:54.575 INFO:teuthology.orchestra.run.smithi083.stdout:(146/176): python3-werkzeug-2.0.3-3.el9.1.noarc 4.5 MB/s | 427 kB 00:00 2024-03-06T07:19:54.600 INFO:teuthology.orchestra.run.smithi083.stdout:(147/176): python3-zc-lockfile-2.0-10.el9.noarc 265 kB/s | 20 kB 00:00 2024-03-06T07:19:54.659 INFO:teuthology.orchestra.run.smithi083.stdout:(148/176): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-03-06T07:19:54.701 INFO:teuthology.orchestra.run.smithi083.stdout:(149/176): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-03-06T07:19:54.726 INFO:teuthology.orchestra.run.smithi083.stdout:(150/176): python3-webob-1.8.7-6.el9.noarch.rpm 805 kB/s | 230 kB 00:00 2024-03-06T07:19:54.760 INFO:teuthology.orchestra.run.smithi083.stdout:(151/176): libacl-devel-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 68 kB 00:00 2024-03-06T07:19:54.843 INFO:teuthology.orchestra.run.smithi083.stdout:(152/176): NetworkManager-1.46.0-1.el9.x86_64.r 27 MB/s | 2.3 MB 00:00 2024-03-06T07:19:54.927 INFO:teuthology.orchestra.run.smithi083.stdout:(153/176): NetworkManager-libnm-1.46.0-1.el9.x8 22 MB/s | 1.8 MB 00:00 2024-03-06T07:19:54.960 INFO:teuthology.orchestra.run.smithi083.stdout:(154/176): NetworkManager-team-1.46.0-1.el9.x86 1.2 MB/s | 41 kB 00:00 2024-03-06T07:19:55.002 INFO:teuthology.orchestra.run.smithi083.stdout:(155/176): NetworkManager-tui-1.46.0-1.el9.x86_ 5.8 MB/s | 247 kB 00:00 2024-03-06T07:19:55.036 INFO:teuthology.orchestra.run.smithi083.stdout:(156/176): acl-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 69 kB 00:00 2024-03-06T07:19:55.070 INFO:teuthology.orchestra.run.smithi083.stdout:(157/176): elfutils-debuginfod-client-0.190-2.e 1.0 MB/s | 35 kB 00:00 2024-03-06T07:19:55.112 INFO:teuthology.orchestra.run.smithi083.stdout:(158/176): elfutils-libelf-0.190-2.el9.x86_64.r 4.5 MB/s | 191 kB 00:00 2024-03-06T07:19:55.154 INFO:teuthology.orchestra.run.smithi083.stdout:(159/176): elfutils-libs-0.190-2.el9.x86_64.rpm 6.0 MB/s | 254 kB 00:00 2024-03-06T07:19:55.204 INFO:teuthology.orchestra.run.smithi083.stdout:(160/176): librados2-19.0.0-1739.g1bc250c7.el9. 6.2 MB/s | 3.4 MB 00:00 2024-03-06T07:19:55.229 INFO:teuthology.orchestra.run.smithi083.stdout:(161/176): iproute-6.7.0-2.el9.x86_64.rpm 10 MB/s | 802 kB 00:00 2024-03-06T07:19:55.254 INFO:teuthology.orchestra.run.smithi083.stdout:(162/176): iproute-tc-6.7.0-2.el9.x86_64.rpm 8.7 MB/s | 433 kB 00:00 2024-03-06T07:19:55.280 INFO:teuthology.orchestra.run.smithi083.stdout:(163/176): libacl-2.3.1-4.el9.x86_64.rpm 445 kB/s | 22 kB 00:00 2024-03-06T07:19:55.305 INFO:teuthology.orchestra.run.smithi083.stdout:(164/176): libblkid-2.37.4-18.el9.x86_64.rpm 2.1 MB/s | 107 kB 00:00 2024-03-06T07:19:55.330 INFO:teuthology.orchestra.run.smithi083.stdout:(165/176): libfdisk-2.37.4-18.el9.x86_64.rpm 3.0 MB/s | 154 kB 00:00 2024-03-06T07:19:55.355 INFO:teuthology.orchestra.run.smithi083.stdout:(166/176): libmount-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 135 kB 00:00 2024-03-06T07:19:55.381 INFO:teuthology.orchestra.run.smithi083.stdout:(167/176): libsmartcols-2.37.4-18.el9.x86_64.rp 1.2 MB/s | 63 kB 00:00 2024-03-06T07:19:55.406 INFO:teuthology.orchestra.run.smithi083.stdout:(168/176): libuuid-2.37.4-18.el9.x86_64.rpm 552 kB/s | 28 kB 00:00 2024-03-06T07:19:55.440 INFO:teuthology.orchestra.run.smithi083.stdout:(169/176): openssl-3.0.7-27.el9.x86_64.rpm 20 MB/s | 1.2 MB 00:00 2024-03-06T07:19:55.490 INFO:teuthology.orchestra.run.smithi083.stdout:(170/176): openssl-libs-3.0.7-27.el9.x86_64.rpm 26 MB/s | 2.2 MB 00:00 2024-03-06T07:19:55.549 INFO:teuthology.orchestra.run.smithi083.stdout:(171/176): util-linux-2.37.4-18.el9.x86_64.rpm 21 MB/s | 2.2 MB 00:00 2024-03-06T07:19:55.574 INFO:teuthology.orchestra.run.smithi083.stdout:(172/176): util-linux-core-2.37.4-18.el9.x86_64 5.3 MB/s | 438 kB 00:00 2024-03-06T07:19:55.599 INFO:teuthology.orchestra.run.smithi083.stdout:(173/176): util-linux-user-2.37.4-18.el9.x86_64 608 kB/s | 30 kB 00:00 2024-03-06T07:19:55.625 INFO:teuthology.orchestra.run.smithi083.stdout:(174/176): xfsdump-3.1.12-4.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-03-06T07:19:55.666 INFO:teuthology.orchestra.run.smithi083.stdout:(175/176): xfsprogs-6.3.0-1.el9.x86_64.rpm 16 MB/s | 1.1 MB 00:00 2024-03-06T07:19:56.250 INFO:teuthology.orchestra.run.smithi083.stdout:(176/176): librbd1-19.0.0-1739.g1bc250c7.el9.x8 2.0 MB/s | 3.1 MB 00:01 2024-03-06T07:19:56.254 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:19:56.254 INFO:teuthology.orchestra.run.smithi083.stdout:Total 6.0 MB/s | 224 MB 00:37 2024-03-06T07:19:56.297 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-06T07:19:56.493 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-06T07:19:56.589 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/202 2024-03-06T07:19:56.613 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-06T07:19:56.613 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-06T07:19:56.614 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/202 2024-03-06T07:19:56.644 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:19:56.672 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:19:56.723 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 4/202 2024-03-06T07:19:56.771 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/202 2024-03-06T07:19:56.832 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/202 2024-03-06T07:19:56.880 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/202 2024-03-06T07:19:56.906 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/202 2024-03-06T07:19:56.954 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/202 2024-03-06T07:19:57.277 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/202 2024-03-06T07:19:57.331 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:19:57.467 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:19:57.507 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:19:57.599 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:19:57.651 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:19:57.814 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/202 2024-03-06T07:19:57.859 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/202 2024-03-06T07:19:57.886 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:19:57.990 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:19:58.020 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:19:58.088 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:19:58.133 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/202 2024-03-06T07:19:58.164 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:19:58.179 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:19:58.337 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:19:58.370 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:19:58.393 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/202 2024-03-06T07:19:58.578 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-06T07:19:58.579 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/202 2024-03-06T07:19:58.579 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-06T07:19:58.712 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:19:58.772 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:19:58.880 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:19:58.916 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:19:59.129 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:19:59.237 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/202 2024-03-06T07:19:59.332 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/202 2024-03-06T07:19:59.367 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/202 2024-03-06T07:19:59.439 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/202 2024-03-06T07:19:59.492 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/202 2024-03-06T07:19:59.527 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/202 2024-03-06T07:19:59.567 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/202 2024-03-06T07:19:59.607 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/202 2024-03-06T07:19:59.663 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/202 2024-03-06T07:19:59.698 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/202 2024-03-06T07:19:59.741 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/202 2024-03-06T07:19:59.774 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/202 2024-03-06T07:19:59.802 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/202 2024-03-06T07:19:59.871 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/202 2024-03-06T07:19:59.907 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 39/202 2024-03-06T07:19:59.950 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 40/202 2024-03-06T07:19:59.979 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 41/202 2024-03-06T07:20:00.048 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/202 2024-03-06T07:20:00.093 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/202 2024-03-06T07:20:00.167 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/202 2024-03-06T07:20:00.300 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/202 2024-03-06T07:20:00.354 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/202 2024-03-06T07:20:00.871 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/202 2024-03-06T07:20:00.949 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/202 2024-03-06T07:20:01.027 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/202 2024-03-06T07:20:01.113 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:20:01.122 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:20:01.158 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/202 2024-03-06T07:20:01.195 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 52/202 2024-03-06T07:20:01.236 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 53/202 2024-03-06T07:20:01.343 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 54/202 2024-03-06T07:20:01.396 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : iproute-6.7.0-2.el9.x86_64 55/202 2024-03-06T07:20:01.646 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/202 2024-03-06T07:20:01.825 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/202 2024-03-06T07:20:01.906 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/202 2024-03-06T07:20:01.949 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/202 2024-03-06T07:20:02.087 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/202 2024-03-06T07:20:02.128 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 61/202 2024-03-06T07:20:02.174 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/202 2024-03-06T07:20:02.246 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/202 2024-03-06T07:20:02.278 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/202 2024-03-06T07:20:02.315 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/202 2024-03-06T07:20:02.362 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/202 2024-03-06T07:20:02.397 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/202 2024-03-06T07:20:02.441 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/202 2024-03-06T07:20:02.483 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/202 2024-03-06T07:20:02.615 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/202 2024-03-06T07:20:02.663 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/202 2024-03-06T07:20:02.694 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/202 2024-03-06T07:20:02.735 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/202 2024-03-06T07:20:02.829 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/202 2024-03-06T07:20:02.869 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 75/202 2024-03-06T07:20:02.909 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/202 2024-03-06T07:20:02.934 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/202 2024-03-06T07:20:02.959 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/202 2024-03-06T07:20:03.064 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/202 2024-03-06T07:20:03.225 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/202 2024-03-06T07:20:03.493 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/202 2024-03-06T07:20:03.576 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/202 2024-03-06T07:20:03.633 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/202 2024-03-06T07:20:03.688 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/202 2024-03-06T07:20:03.877 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/202 2024-03-06T07:20:03.938 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/202 2024-03-06T07:20:04.066 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/202 2024-03-06T07:20:04.103 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 88/202 2024-03-06T07:20:04.144 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/202 2024-03-06T07:20:04.192 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 90/202 2024-03-06T07:20:04.228 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : jq-1.6-16.el9.x86_64 91/202 2024-03-06T07:20:04.259 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : fuse-2.9.9-15.el9.x86_64 92/202 2024-03-06T07:20:04.287 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 93/202 2024-03-06T07:20:04.321 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 94/202 2024-03-06T07:20:04.360 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : uuid-1.6.2-55.el9.x86_64 95/202 2024-03-06T07:20:04.392 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 96/202 2024-03-06T07:20:04.423 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 97/202 2024-03-06T07:20:04.497 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 98/202 2024-03-06T07:20:04.532 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 99/202 2024-03-06T07:20:04.557 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 100/202 2024-03-06T07:20:04.618 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 101/202 2024-03-06T07:20:04.650 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 102/202 2024-03-06T07:20:04.738 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 103/202 2024-03-06T07:20:04.828 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 104/202 2024-03-06T07:20:04.872 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 105/202 2024-03-06T07:20:04.904 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 106/202 2024-03-06T07:20:04.950 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 107/202 2024-03-06T07:20:05.047 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-packaging-20.9-5.el9.noarch 108/202 2024-03-06T07:20:05.105 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 109/202 2024-03-06T07:20:05.552 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-06T07:20:05.629 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/202 2024-03-06T07:20:05.664 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/202 2024-03-06T07:20:05.711 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:20:05.734 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:20:05.798 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 4/202 2024-03-06T07:20:05.821 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 110/202 2024-03-06T07:20:05.837 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/202 2024-03-06T07:20:05.870 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 111/202 2024-03-06T07:20:05.900 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/202 2024-03-06T07:20:05.904 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 112/202 2024-03-06T07:20:05.948 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/202 2024-03-06T07:20:05.952 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 113/202 2024-03-06T07:20:05.974 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/202 2024-03-06T07:20:06.023 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/202 2024-03-06T07:20:06.030 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 114/202 2024-03-06T07:20:06.063 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 115/202 2024-03-06T07:20:06.230 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : openblas-0.3.21-2.el9.x86_64 116/202 2024-03-06T07:20:06.261 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 117/202 2024-03-06T07:20:06.338 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 118/202 2024-03-06T07:20:06.345 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/202 2024-03-06T07:20:06.400 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:20:06.527 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:20:06.551 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:20:06.634 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:20:06.685 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:20:06.846 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/202 2024-03-06T07:20:06.901 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/202 2024-03-06T07:20:06.920 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:20:06.975 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 119/202 2024-03-06T07:20:07.016 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:20:07.038 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:20:07.091 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 120/202 2024-03-06T07:20:07.103 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:20:07.159 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/202 2024-03-06T07:20:07.181 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:20:07.210 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:20:07.406 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:20:07.438 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:20:07.470 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/202 2024-03-06T07:20:07.639 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/202 2024-03-06T07:20:07.771 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:20:07.832 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:20:07.949 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:20:08.002 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:20:08.225 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:20:08.309 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 121/202 2024-03-06T07:20:08.328 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/202 2024-03-06T07:20:08.382 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 122/202 2024-03-06T07:20:08.409 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/202 2024-03-06T07:20:08.417 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/202 2024-03-06T07:20:08.449 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/202 2024-03-06T07:20:08.515 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/202 2024-03-06T07:20:08.562 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/202 2024-03-06T07:20:08.594 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/202 2024-03-06T07:20:08.620 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/202 2024-03-06T07:20:08.652 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/202 2024-03-06T07:20:08.718 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/202 2024-03-06T07:20:08.718 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/202 2024-03-06T07:20:08.754 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/202 2024-03-06T07:20:08.755 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:20:08.797 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/202 2024-03-06T07:20:08.831 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/202 2024-03-06T07:20:08.845 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:20:08.866 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/202 2024-03-06T07:20:08.876 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 126/202 2024-03-06T07:20:08.913 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/202 2024-03-06T07:20:08.935 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/202 2024-03-06T07:20:08.971 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 39/202 2024-03-06T07:20:09.015 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 40/202 2024-03-06T07:20:09.043 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 41/202 2024-03-06T07:20:09.113 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/202 2024-03-06T07:20:09.158 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/202 2024-03-06T07:20:09.223 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/202 2024-03-06T07:20:09.356 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/202 2024-03-06T07:20:09.363 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/202 2024-03-06T07:20:09.392 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:20:09.410 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/202 2024-03-06T07:20:09.462 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:20:09.485 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 130/202 2024-03-06T07:20:09.928 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/202 2024-03-06T07:20:10.006 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/202 2024-03-06T07:20:10.095 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/202 2024-03-06T07:20:10.182 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:20:10.190 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:20:10.239 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/202 2024-03-06T07:20:10.277 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 52/202 2024-03-06T07:20:10.315 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 53/202 2024-03-06T07:20:10.418 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 54/202 2024-03-06T07:20:10.486 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : iproute-6.7.0-2.el9.x86_64 55/202 2024-03-06T07:20:10.966 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/202 2024-03-06T07:20:11.231 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/202 2024-03-06T07:20:11.303 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/202 2024-03-06T07:20:11.340 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/202 2024-03-06T07:20:11.479 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/202 2024-03-06T07:20:11.535 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 61/202 2024-03-06T07:20:11.590 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/202 2024-03-06T07:20:11.662 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/202 2024-03-06T07:20:11.694 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/202 2024-03-06T07:20:11.696 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:20:11.730 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/202 2024-03-06T07:20:11.733 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:20:11.777 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/202 2024-03-06T07:20:11.813 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/202 2024-03-06T07:20:11.856 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/202 2024-03-06T07:20:11.898 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/202 2024-03-06T07:20:12.032 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/202 2024-03-06T07:20:12.078 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/202 2024-03-06T07:20:12.109 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/202 2024-03-06T07:20:12.150 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/202 2024-03-06T07:20:12.244 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/202 2024-03-06T07:20:12.268 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:20:12.276 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 75/202 2024-03-06T07:20:12.303 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:20:12.316 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/202 2024-03-06T07:20:12.341 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/202 2024-03-06T07:20:12.366 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/202 2024-03-06T07:20:12.406 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:20:12.406 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-06T07:20:12.406 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-06T07:20:12.406 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:20:12.444 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/202 2024-03-06T07:20:12.446 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:20:12.575 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/202 2024-03-06T07:20:12.690 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/202 2024-03-06T07:20:12.733 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/202 2024-03-06T07:20:12.798 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/202 2024-03-06T07:20:12.853 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/202 2024-03-06T07:20:13.051 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/202 2024-03-06T07:20:13.095 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/202 2024-03-06T07:20:13.221 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/202 2024-03-06T07:20:13.260 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 88/202 2024-03-06T07:20:13.293 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/202 2024-03-06T07:20:13.333 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 90/202 2024-03-06T07:20:13.368 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : jq-1.6-16.el9.x86_64 91/202 2024-03-06T07:20:13.399 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : fuse-2.9.9-15.el9.x86_64 92/202 2024-03-06T07:20:13.418 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 93/202 2024-03-06T07:20:13.445 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 94/202 2024-03-06T07:20:13.483 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : uuid-1.6.2-55.el9.x86_64 95/202 2024-03-06T07:20:13.515 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 96/202 2024-03-06T07:20:13.546 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 97/202 2024-03-06T07:20:13.621 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 98/202 2024-03-06T07:20:13.674 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 99/202 2024-03-06T07:20:13.706 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 100/202 2024-03-06T07:20:13.747 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 101/202 2024-03-06T07:20:13.773 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 102/202 2024-03-06T07:20:13.862 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 103/202 2024-03-06T07:20:13.952 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 104/202 2024-03-06T07:20:13.996 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 105/202 2024-03-06T07:20:14.028 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 106/202 2024-03-06T07:20:14.074 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 107/202 2024-03-06T07:20:14.171 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-packaging-20.9-5.el9.noarch 108/202 2024-03-06T07:20:14.220 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 109/202 2024-03-06T07:20:14.934 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 110/202 2024-03-06T07:20:14.976 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 111/202 2024-03-06T07:20:15.010 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 112/202 2024-03-06T07:20:15.058 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 113/202 2024-03-06T07:20:15.136 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 114/202 2024-03-06T07:20:15.178 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 115/202 2024-03-06T07:20:15.356 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : openblas-0.3.21-2.el9.x86_64 116/202 2024-03-06T07:20:15.387 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 117/202 2024-03-06T07:20:15.464 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 118/202 2024-03-06T07:20:16.095 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 119/202 2024-03-06T07:20:16.209 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 120/202 2024-03-06T07:20:17.419 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 121/202 2024-03-06T07:20:17.500 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 122/202 2024-03-06T07:20:17.527 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/202 2024-03-06T07:20:17.837 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/202 2024-03-06T07:20:17.865 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:20:17.955 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:20:17.986 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 126/202 2024-03-06T07:20:18.023 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/202 2024-03-06T07:20:18.478 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/202 2024-03-06T07:20:18.502 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:20:18.572 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:20:18.595 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 130/202 2024-03-06T07:20:20.788 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:20:20.833 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:20:21.371 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:20:21.412 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:20:21.575 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:20:21.575 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-06T07:20:21.575 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-06T07:20:21.576 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:20:21.830 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:20:24.278 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:20:24.279 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /sys 2024-03-06T07:20:24.279 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /proc 2024-03-06T07:20:24.279 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /mnt 2024-03-06T07:20:24.279 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /var/tmp 2024-03-06T07:20:24.279 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /home 2024-03-06T07:20:24.279 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /root 2024-03-06T07:20:24.280 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /tmp 2024-03-06T07:20:24.280 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:20:24.442 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:20:24.457 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:20:24.457 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:24.457 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-06T07:20:24.457 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:20:25.937 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:20:25.963 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:20:25.963 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:25.963 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-06T07:20:25.963 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:20:25.963 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:20:25.963 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:20:26.179 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:20:26.202 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:20:26.202 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:26.202 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-06T07:20:26.202 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:20:26.202 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:20:26.202 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:20:26.655 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:20:26.677 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:20:26.678 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:26.678 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-06T07:20:26.678 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:20:26.678 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:20:26.678 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:20:26.774 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : automake-1.16.2-8.el9.noarch 138/202 2024-03-06T07:20:27.116 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:20:27.138 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:20:27.161 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 140/202 2024-03-06T07:20:27.213 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 141/202 2024-03-06T07:20:27.236 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:20:27.463 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:20:27.496 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:20:28.453 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:20:28.479 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:20:28.578 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:20:28.733 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 145/202 2024-03-06T07:20:28.756 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:20:28.779 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:20:28.779 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:28.779 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-06T07:20:28.779 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:20:28.780 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:20:28.780 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:20:28.823 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:20:28.834 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:20:28.884 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 148/202 2024-03-06T07:20:29.762 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/202 2024-03-06T07:20:29.792 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:20:29.815 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:20:29.815 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:29.815 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-06T07:20:29.815 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:20:29.815 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:20:29.815 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:20:29.851 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:20:29.873 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:20:29.873 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:29.873 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-06T07:20:29.873 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:20:29.927 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:20:29.949 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:20:29.950 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:29.950 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-06T07:20:29.950 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:20:29.950 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:20:29.950 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:20:30.269 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:20:30.292 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:20:30.292 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:30.292 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-06T07:20:30.292 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:20:30.292 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:20:30.292 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:20:33.415 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:20:33.416 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /sys 2024-03-06T07:20:33.416 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /proc 2024-03-06T07:20:33.416 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /mnt 2024-03-06T07:20:33.416 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /var/tmp 2024-03-06T07:20:33.416 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /home 2024-03-06T07:20:33.416 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /root 2024-03-06T07:20:33.416 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /tmp 2024-03-06T07:20:33.416 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:20:33.586 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:20:33.602 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:20:33.602 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:33.602 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-06T07:20:33.602 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:20:34.706 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 154/202 2024-03-06T07:20:34.748 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 155/202 2024-03-06T07:20:34.778 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 156/202 2024-03-06T07:20:34.806 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/202 2024-03-06T07:20:35.001 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 158/202 2024-03-06T07:20:35.106 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:20:35.113 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/202 2024-03-06T07:20:35.131 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:20:35.132 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:35.132 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-06T07:20:35.132 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:20:35.132 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:20:35.132 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:20:35.161 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 160/202 2024-03-06T07:20:35.217 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/202 2024-03-06T07:20:35.270 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/202 2024-03-06T07:20:35.355 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/202 2024-03-06T07:20:35.357 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:20:35.379 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:20:35.379 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:35.379 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-06T07:20:35.380 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:20:35.380 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:20:35.380 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:20:35.408 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : iproute-tc-6.7.0-2.el9.x86_64 164/202 2024-03-06T07:20:35.451 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/202 2024-03-06T07:20:35.482 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 166/202 2024-03-06T07:20:35.511 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 167/202 2024-03-06T07:20:35.607 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 168/202 2024-03-06T07:20:35.671 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/202 2024-03-06T07:20:35.710 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 170/202 2024-03-06T07:20:35.825 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:20:35.847 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:20:35.847 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:35.847 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-06T07:20:35.847 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:20:35.847 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:20:35.847 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:20:35.935 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : automake-1.16.2-8.el9.noarch 138/202 2024-03-06T07:20:36.324 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:20:36.358 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:20:36.369 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:20:36.378 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:20:36.400 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:20:36.432 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 140/202 2024-03-06T07:20:36.493 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 141/202 2024-03-06T07:20:36.515 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:20:36.742 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:20:36.774 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:20:37.185 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 172/202 2024-03-06T07:20:37.253 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 173/202 2024-03-06T07:20:37.294 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 174/202 2024-03-06T07:20:37.374 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flex-2.6.4-9.el9.x86_64 175/202 2024-03-06T07:20:37.407 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : bison-3.7.4-5.el9.x86_64 176/202 2024-03-06T07:20:37.432 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 177/202 2024-03-06T07:20:37.457 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 178/202 2024-03-06T07:20:37.484 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 179/202 2024-03-06T07:20:37.484 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:20:37.577 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:20:37.604 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 181/202 2024-03-06T07:20:37.635 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 182/202 2024-03-06T07:20:37.635 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:20:37.720 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:20:37.736 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:20:37.746 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 184/202 2024-03-06T07:20:37.766 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:20:37.788 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 185/202 2024-03-06T07:20:37.813 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : iproute-tc-6.2.0-5.el9.x86_64 186/202 2024-03-06T07:20:37.847 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 187/202 2024-03-06T07:20:37.866 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:20:37.881 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : iproute-6.2.0-5.el9.x86_64 188/202 2024-03-06T07:20:37.915 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 189/202 2024-03-06T07:20:37.953 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 190/202 2024-03-06T07:20:37.986 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 191/202 2024-03-06T07:20:38.011 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 192/202 2024-03-06T07:20:38.036 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 193/202 2024-03-06T07:20:38.037 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 145/202 2024-03-06T07:20:38.062 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:20:38.076 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:20:38.087 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 195/202 2024-03-06T07:20:38.094 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:20:38.094 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:20:38.099 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:20:38.099 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:38.099 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-06T07:20:38.099 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:20:38.099 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:20:38.099 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:20:38.144 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:20:38.154 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:20:38.205 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 148/202 2024-03-06T07:20:38.274 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:20:38.274 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:20:38.340 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:20:38.372 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:20:38.398 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 199/202 2024-03-06T07:20:38.423 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 200/202 2024-03-06T07:20:38.457 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 201/202 2024-03-06T07:20:38.531 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:20:39.065 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/202 2024-03-06T07:20:39.096 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:20:39.118 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:20:39.118 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:39.118 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-06T07:20:39.119 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:20:39.119 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:20:39.119 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:20:39.163 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:20:39.185 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:20:39.185 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:39.185 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-06T07:20:39.185 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:20:39.231 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:20:39.252 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:20:39.252 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:39.253 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-06T07:20:39.253 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:20:39.253 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:20:39.253 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:20:39.556 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:20:39.578 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:20:39.578 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:39.578 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-06T07:20:39.578 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:20:39.579 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:20:39.579 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:20:43.347 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:20:43.347 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 1/202 2024-03-06T07:20:43.347 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2/202 2024-03-06T07:20:43.348 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 3/202 2024-03-06T07:20:43.348 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 4/202 2024-03-06T07:20:43.348 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 5/202 2024-03-06T07:20:43.348 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 6/202 2024-03-06T07:20:43.348 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 7/202 2024-03-06T07:20:43.348 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 8/202 2024-03-06T07:20:43.348 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 9/202 2024-03-06T07:20:43.348 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 10/202 2024-03-06T07:20:43.348 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:20:43.348 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:20:43.348 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:20:43.348 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 14/202 2024-03-06T07:20:43.348 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 15/202 2024-03-06T07:20:43.349 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:20:43.349 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 17/202 2024-03-06T07:20:43.349 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 18/202 2024-03-06T07:20:43.349 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 19/202 2024-03-06T07:20:43.349 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 20/202 2024-03-06T07:20:43.349 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 21/202 2024-03-06T07:20:43.349 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 22/202 2024-03-06T07:20:43.349 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 23/202 2024-03-06T07:20:43.349 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 24/202 2024-03-06T07:20:43.349 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 25/202 2024-03-06T07:20:43.349 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 26/202 2024-03-06T07:20:43.349 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 27/202 2024-03-06T07:20:43.350 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 28/202 2024-03-06T07:20:43.350 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 29/202 2024-03-06T07:20:43.350 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 30/202 2024-03-06T07:20:43.350 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 31/202 2024-03-06T07:20:43.350 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 32/202 2024-03-06T07:20:43.350 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 33/202 2024-03-06T07:20:43.350 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 34/202 2024-03-06T07:20:43.350 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 35/202 2024-03-06T07:20:43.350 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 36/202 2024-03-06T07:20:43.350 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 37/202 2024-03-06T07:20:43.350 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 38/202 2024-03-06T07:20:43.350 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 39/202 2024-03-06T07:20:43.350 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/202 2024-03-06T07:20:43.351 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/202 2024-03-06T07:20:43.351 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/202 2024-03-06T07:20:43.351 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 43/202 2024-03-06T07:20:43.351 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flex-2.6.4-9.el9.x86_64 44/202 2024-03-06T07:20:43.351 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 45/202 2024-03-06T07:20:43.351 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 46/202 2024-03-06T07:20:43.351 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 47/202 2024-03-06T07:20:43.351 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/202 2024-03-06T07:20:43.351 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/202 2024-03-06T07:20:43.351 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/202 2024-03-06T07:20:43.351 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/202 2024-03-06T07:20:43.351 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/202 2024-03-06T07:20:43.351 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 53/202 2024-03-06T07:20:43.351 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/202 2024-03-06T07:20:43.352 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/202 2024-03-06T07:20:43.352 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/202 2024-03-06T07:20:43.352 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 57/202 2024-03-06T07:20:43.352 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 58/202 2024-03-06T07:20:43.352 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 59/202 2024-03-06T07:20:43.352 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/202 2024-03-06T07:20:43.352 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/202 2024-03-06T07:20:43.352 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/202 2024-03-06T07:20:43.352 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/202 2024-03-06T07:20:43.352 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 64/202 2024-03-06T07:20:43.352 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 65/202 2024-03-06T07:20:43.352 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 66/202 2024-03-06T07:20:43.352 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 67/202 2024-03-06T07:20:43.353 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 68/202 2024-03-06T07:20:43.353 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 69/202 2024-03-06T07:20:43.353 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 70/202 2024-03-06T07:20:43.353 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/202 2024-03-06T07:20:43.353 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 72/202 2024-03-06T07:20:43.353 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 73/202 2024-03-06T07:20:43.353 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 74/202 2024-03-06T07:20:43.353 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 75/202 2024-03-06T07:20:43.353 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 76/202 2024-03-06T07:20:43.353 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 77/202 2024-03-06T07:20:43.353 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 78/202 2024-03-06T07:20:43.353 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 79/202 2024-03-06T07:20:43.353 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 80/202 2024-03-06T07:20:43.354 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/202 2024-03-06T07:20:43.354 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 82/202 2024-03-06T07:20:43.354 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 83/202 2024-03-06T07:20:43.354 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 84/202 2024-03-06T07:20:43.354 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 85/202 2024-03-06T07:20:43.354 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 86/202 2024-03-06T07:20:43.354 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 87/202 2024-03-06T07:20:43.354 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 88/202 2024-03-06T07:20:43.354 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 89/202 2024-03-06T07:20:43.354 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 90/202 2024-03-06T07:20:43.354 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 91/202 2024-03-06T07:20:43.354 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 92/202 2024-03-06T07:20:43.354 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 93/202 2024-03-06T07:20:43.354 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : jq-1.6-16.el9.x86_64 94/202 2024-03-06T07:20:43.355 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 95/202 2024-03-06T07:20:43.355 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 96/202 2024-03-06T07:20:43.355 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 97/202 2024-03-06T07:20:43.355 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 98/202 2024-03-06T07:20:43.355 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 99/202 2024-03-06T07:20:43.355 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ply-3.11-14.el9.noarch 100/202 2024-03-06T07:20:43.355 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 101/202 2024-03-06T07:20:43.355 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 102/202 2024-03-06T07:20:43.355 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 103/202 2024-03-06T07:20:43.355 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 104/202 2024-03-06T07:20:43.355 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/202 2024-03-06T07:20:43.355 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/202 2024-03-06T07:20:43.355 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/202 2024-03-06T07:20:43.356 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/202 2024-03-06T07:20:43.356 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/202 2024-03-06T07:20:43.356 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/202 2024-03-06T07:20:43.356 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/202 2024-03-06T07:20:43.356 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/202 2024-03-06T07:20:43.356 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/202 2024-03-06T07:20:43.356 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/202 2024-03-06T07:20:43.356 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/202 2024-03-06T07:20:43.356 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/202 2024-03-06T07:20:43.356 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/202 2024-03-06T07:20:43.356 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/202 2024-03-06T07:20:43.356 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/202 2024-03-06T07:20:43.356 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/202 2024-03-06T07:20:43.357 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/202 2024-03-06T07:20:43.357 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/202 2024-03-06T07:20:43.357 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/202 2024-03-06T07:20:43.357 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/202 2024-03-06T07:20:43.357 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/202 2024-03-06T07:20:43.357 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/202 2024-03-06T07:20:43.357 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/202 2024-03-06T07:20:43.357 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/202 2024-03-06T07:20:43.357 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/202 2024-03-06T07:20:43.357 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/202 2024-03-06T07:20:43.357 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/202 2024-03-06T07:20:43.357 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/202 2024-03-06T07:20:43.357 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/202 2024-03-06T07:20:43.357 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/202 2024-03-06T07:20:43.358 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/202 2024-03-06T07:20:43.358 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/202 2024-03-06T07:20:43.358 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/202 2024-03-06T07:20:43.358 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/202 2024-03-06T07:20:43.358 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/202 2024-03-06T07:20:43.358 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/202 2024-03-06T07:20:43.358 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/202 2024-03-06T07:20:43.358 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/202 2024-03-06T07:20:43.358 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/202 2024-03-06T07:20:43.358 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/202 2024-03-06T07:20:43.358 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/202 2024-03-06T07:20:43.358 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/202 2024-03-06T07:20:43.358 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/202 2024-03-06T07:20:43.358 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/202 2024-03-06T07:20:43.359 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/202 2024-03-06T07:20:43.359 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/202 2024-03-06T07:20:43.359 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 151/202 2024-03-06T07:20:43.359 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/202 2024-03-06T07:20:43.359 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:20:43.359 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/202 2024-03-06T07:20:43.359 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 155/202 2024-03-06T07:20:43.359 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 156/202 2024-03-06T07:20:43.359 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/202 2024-03-06T07:20:43.359 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/202 2024-03-06T07:20:43.359 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/202 2024-03-06T07:20:43.359 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/202 2024-03-06T07:20:43.359 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/202 2024-03-06T07:20:43.359 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/202 2024-03-06T07:20:43.360 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/202 2024-03-06T07:20:43.360 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/202 2024-03-06T07:20:43.360 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/202 2024-03-06T07:20:43.360 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/202 2024-03-06T07:20:43.360 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/202 2024-03-06T07:20:43.360 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/202 2024-03-06T07:20:43.360 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/202 2024-03-06T07:20:43.360 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/202 2024-03-06T07:20:43.360 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/202 2024-03-06T07:20:43.360 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/202 2024-03-06T07:20:43.360 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : iproute-6.7.0-2.el9.x86_64 173/202 2024-03-06T07:20:43.360 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : iproute-6.2.0-5.el9.x86_64 174/202 2024-03-06T07:20:43.360 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : iproute-tc-6.7.0-2.el9.x86_64 175/202 2024-03-06T07:20:43.361 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : iproute-tc-6.2.0-5.el9.x86_64 176/202 2024-03-06T07:20:43.361 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 177/202 2024-03-06T07:20:43.361 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 178/202 2024-03-06T07:20:43.361 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 179/202 2024-03-06T07:20:43.361 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 180/202 2024-03-06T07:20:43.361 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 181/202 2024-03-06T07:20:43.361 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 182/202 2024-03-06T07:20:43.361 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 183/202 2024-03-06T07:20:43.361 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 184/202 2024-03-06T07:20:43.361 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 185/202 2024-03-06T07:20:43.361 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 186/202 2024-03-06T07:20:43.361 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 187/202 2024-03-06T07:20:43.361 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 188/202 2024-03-06T07:20:43.361 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 189/202 2024-03-06T07:20:43.362 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 190/202 2024-03-06T07:20:43.362 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 191/202 2024-03-06T07:20:43.362 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 192/202 2024-03-06T07:20:43.362 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 193/202 2024-03-06T07:20:43.362 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:20:43.362 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 195/202 2024-03-06T07:20:43.362 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 196/202 2024-03-06T07:20:43.362 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 197/202 2024-03-06T07:20:43.362 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:20:43.362 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 199/202 2024-03-06T07:20:43.362 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 200/202 2024-03-06T07:20:43.362 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 201/202 2024-03-06T07:20:43.995 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 154/202 2024-03-06T07:20:44.035 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 155/202 2024-03-06T07:20:44.066 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 156/202 2024-03-06T07:20:44.093 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/202 2024-03-06T07:20:44.280 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 158/202 2024-03-06T07:20:44.401 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/202 2024-03-06T07:20:44.441 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 160/202 2024-03-06T07:20:44.498 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/202 2024-03-06T07:20:44.550 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/202 2024-03-06T07:20:44.621 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/202 2024-03-06T07:20:44.652 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : iproute-tc-6.7.0-2.el9.x86_64 164/202 2024-03-06T07:20:44.687 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/202 2024-03-06T07:20:44.718 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 166/202 2024-03-06T07:20:44.747 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 167/202 2024-03-06T07:20:44.824 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 168/202 2024-03-06T07:20:44.886 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/202 2024-03-06T07:20:44.937 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 170/202 2024-03-06T07:20:45.534 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:20:45.569 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:20:45.588 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:20:46.100 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 202/202 2024-03-06T07:20:46.100 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:20:46.100 INFO:teuthology.orchestra.run.smithi183.stdout:Upgraded: 2024-03-06T07:20:46.100 INFO:teuthology.orchestra.run.smithi183.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-06T07:20:46.100 INFO:teuthology.orchestra.run.smithi183.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-06T07:20:46.101 INFO:teuthology.orchestra.run.smithi183.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-06T07:20:46.101 INFO:teuthology.orchestra.run.smithi183.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-06T07:20:46.101 INFO:teuthology.orchestra.run.smithi183.stdout: acl-2.3.1-4.el9.x86_64 2024-03-06T07:20:46.101 INFO:teuthology.orchestra.run.smithi183.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-06T07:20:46.101 INFO:teuthology.orchestra.run.smithi183.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-06T07:20:46.101 INFO:teuthology.orchestra.run.smithi183.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-06T07:20:46.101 INFO:teuthology.orchestra.run.smithi183.stdout: iproute-6.7.0-2.el9.x86_64 2024-03-06T07:20:46.101 INFO:teuthology.orchestra.run.smithi183.stdout: iproute-tc-6.7.0-2.el9.x86_64 2024-03-06T07:20:46.101 INFO:teuthology.orchestra.run.smithi183.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-06T07:20:46.101 INFO:teuthology.orchestra.run.smithi183.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-06T07:20:46.101 INFO:teuthology.orchestra.run.smithi183.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-06T07:20:46.101 INFO:teuthology.orchestra.run.smithi183.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-06T07:20:46.101 INFO:teuthology.orchestra.run.smithi183.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-06T07:20:46.102 INFO:teuthology.orchestra.run.smithi183.stdout: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.102 INFO:teuthology.orchestra.run.smithi183.stdout: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.102 INFO:teuthology.orchestra.run.smithi183.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-06T07:20:46.102 INFO:teuthology.orchestra.run.smithi183.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-06T07:20:46.102 INFO:teuthology.orchestra.run.smithi183.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-06T07:20:46.102 INFO:teuthology.orchestra.run.smithi183.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-06T07:20:46.102 INFO:teuthology.orchestra.run.smithi183.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-06T07:20:46.102 INFO:teuthology.orchestra.run.smithi183.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-06T07:20:46.102 INFO:teuthology.orchestra.run.smithi183.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-06T07:20:46.102 INFO:teuthology.orchestra.run.smithi183.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-06T07:20:46.102 INFO:teuthology.orchestra.run.smithi183.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-06T07:20:46.102 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-06T07:20:46.102 INFO:teuthology.orchestra.run.smithi183.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-06T07:20:46.102 INFO:teuthology.orchestra.run.smithi183.stdout: automake-1.16.2-8.el9.noarch 2024-03-06T07:20:46.103 INFO:teuthology.orchestra.run.smithi183.stdout: bison-3.7.4-5.el9.x86_64 2024-03-06T07:20:46.103 INFO:teuthology.orchestra.run.smithi183.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-06T07:20:46.103 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.103 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.103 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.103 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.103 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:46.103 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-immutable-object-cache-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.103 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.103 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.103 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:46.103 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:46.103 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:46.104 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:46.104 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:46.104 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.104 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.104 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:46.104 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.104 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.104 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.104 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:46.104 INFO:teuthology.orchestra.run.smithi183.stdout: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:46.104 INFO:teuthology.orchestra.run.smithi183.stdout: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.104 INFO:teuthology.orchestra.run.smithi183.stdout: cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.105 INFO:teuthology.orchestra.run.smithi183.stdout: cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:46.105 INFO:teuthology.orchestra.run.smithi183.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-06T07:20:46.105 INFO:teuthology.orchestra.run.smithi183.stdout: flex-2.6.4-9.el9.x86_64 2024-03-06T07:20:46.105 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-06T07:20:46.105 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-06T07:20:46.105 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-06T07:20:46.105 INFO:teuthology.orchestra.run.smithi183.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-06T07:20:46.105 INFO:teuthology.orchestra.run.smithi183.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-06T07:20:46.105 INFO:teuthology.orchestra.run.smithi183.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-06T07:20:46.105 INFO:teuthology.orchestra.run.smithi183.stdout: jq-1.6-16.el9.x86_64 2024-03-06T07:20:46.105 INFO:teuthology.orchestra.run.smithi183.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-06T07:20:46.105 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-06T07:20:46.105 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-06T07:20:46.105 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.106 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.106 INFO:teuthology.orchestra.run.smithi183.stdout: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.106 INFO:teuthology.orchestra.run.smithi183.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-06T07:20:46.106 INFO:teuthology.orchestra.run.smithi183.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-06T07:20:46.106 INFO:teuthology.orchestra.run.smithi183.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-06T07:20:46.106 INFO:teuthology.orchestra.run.smithi183.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-06T07:20:46.106 INFO:teuthology.orchestra.run.smithi183.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-06T07:20:46.106 INFO:teuthology.orchestra.run.smithi183.stdout: librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.106 INFO:teuthology.orchestra.run.smithi183.stdout: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.106 INFO:teuthology.orchestra.run.smithi183.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-06T07:20:46.106 INFO:teuthology.orchestra.run.smithi183.stdout: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.106 INFO:teuthology.orchestra.run.smithi183.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-06T07:20:46.107 INFO:teuthology.orchestra.run.smithi183.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-06T07:20:46.107 INFO:teuthology.orchestra.run.smithi183.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-06T07:20:46.107 INFO:teuthology.orchestra.run.smithi183.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-06T07:20:46.107 INFO:teuthology.orchestra.run.smithi183.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-06T07:20:46.107 INFO:teuthology.orchestra.run.smithi183.stdout: lua-5.4.4-4.el9.x86_64 2024-03-06T07:20:46.107 INFO:teuthology.orchestra.run.smithi183.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-06T07:20:46.107 INFO:teuthology.orchestra.run.smithi183.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-06T07:20:46.107 INFO:teuthology.orchestra.run.smithi183.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-06T07:20:46.107 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-06T07:20:46.107 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-06T07:20:46.107 INFO:teuthology.orchestra.run.smithi183.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-06T07:20:46.108 INFO:teuthology.orchestra.run.smithi183.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-06T07:20:46.108 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql-13.11-1.el9.x86_64 2024-03-06T07:20:46.108 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-06T07:20:46.108 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-06T07:20:46.108 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-06T07:20:46.108 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-06T07:20:46.108 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-06T07:20:46.108 INFO:teuthology.orchestra.run.smithi183.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-06T07:20:46.108 INFO:teuthology.orchestra.run.smithi183.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-06T07:20:46.108 INFO:teuthology.orchestra.run.smithi183.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-06T07:20:46.108 INFO:teuthology.orchestra.run.smithi183.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-06T07:20:46.108 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-06T07:20:46.108 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.109 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.109 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.109 INFO:teuthology.orchestra.run.smithi183.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-06T07:20:46.109 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-06T07:20:46.109 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-06T07:20:46.109 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-06T07:20:46.109 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-06T07:20:46.109 INFO:teuthology.orchestra.run.smithi183.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-06T07:20:46.109 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-06T07:20:46.109 INFO:teuthology.orchestra.run.smithi183.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-06T07:20:46.109 INFO:teuthology.orchestra.run.smithi183.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-06T07:20:46.109 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-06T07:20:46.109 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-06T07:20:46.109 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-06T07:20:46.110 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-06T07:20:46.110 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-06T07:20:46.110 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-06T07:20:46.110 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-06T07:20:46.110 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-06T07:20:46.110 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-06T07:20:46.110 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-06T07:20:46.110 INFO:teuthology.orchestra.run.smithi183.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-06T07:20:46.110 INFO:teuthology.orchestra.run.smithi183.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-06T07:20:46.111 INFO:teuthology.orchestra.run.smithi183.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-06T07:20:46.111 INFO:teuthology.orchestra.run.smithi183.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-06T07:20:46.111 INFO:teuthology.orchestra.run.smithi183.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-06T07:20:46.111 INFO:teuthology.orchestra.run.smithi183.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-06T07:20:46.111 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-06T07:20:46.111 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-06T07:20:46.111 INFO:teuthology.orchestra.run.smithi183.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-06T07:20:46.111 INFO:teuthology.orchestra.run.smithi183.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-06T07:20:46.111 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-06T07:20:46.111 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ply-3.11-14.el9.noarch 2024-03-06T07:20:46.111 INFO:teuthology.orchestra.run.smithi183.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-06T07:20:46.111 INFO:teuthology.orchestra.run.smithi183.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-06T07:20:46.111 INFO:teuthology.orchestra.run.smithi183.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-06T07:20:46.111 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-06T07:20:46.112 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-06T07:20:46.112 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-06T07:20:46.112 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-06T07:20:46.112 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-06T07:20:46.112 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-06T07:20:46.112 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.112 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.112 INFO:teuthology.orchestra.run.smithi183.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-06T07:20:46.112 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-06T07:20:46.112 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-06T07:20:46.112 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.112 INFO:teuthology.orchestra.run.smithi183.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-06T07:20:46.113 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-06T07:20:46.113 INFO:teuthology.orchestra.run.smithi183.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-06T07:20:46.113 INFO:teuthology.orchestra.run.smithi183.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-06T07:20:46.113 INFO:teuthology.orchestra.run.smithi183.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-06T07:20:46.113 INFO:teuthology.orchestra.run.smithi183.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-06T07:20:46.113 INFO:teuthology.orchestra.run.smithi183.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-06T07:20:46.113 INFO:teuthology.orchestra.run.smithi183.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-06T07:20:46.113 INFO:teuthology.orchestra.run.smithi183.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-06T07:20:46.113 INFO:teuthology.orchestra.run.smithi183.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-06T07:20:46.113 INFO:teuthology.orchestra.run.smithi183.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-06T07:20:46.113 INFO:teuthology.orchestra.run.smithi183.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-06T07:20:46.113 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-06T07:20:46.113 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-06T07:20:46.114 INFO:teuthology.orchestra.run.smithi183.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-06T07:20:46.114 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.114 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.114 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.114 INFO:teuthology.orchestra.run.smithi183.stdout: re2-1:20211101-3.el9.x86_64 2024-03-06T07:20:46.114 INFO:teuthology.orchestra.run.smithi183.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-06T07:20:46.114 INFO:teuthology.orchestra.run.smithi183.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-06T07:20:46.114 INFO:teuthology.orchestra.run.smithi183.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-06T07:20:46.114 INFO:teuthology.orchestra.run.smithi183.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-06T07:20:46.114 INFO:teuthology.orchestra.run.smithi183.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-06T07:20:46.115 INFO:teuthology.orchestra.run.smithi183.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-06T07:20:46.115 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:20:46.115 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-06T07:20:46.396 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 172/202 2024-03-06T07:20:46.458 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 173/202 2024-03-06T07:20:46.499 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 174/202 2024-03-06T07:20:46.581 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : flex-2.6.4-9.el9.x86_64 175/202 2024-03-06T07:20:46.611 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : bison-3.7.4-5.el9.x86_64 176/202 2024-03-06T07:20:46.637 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 177/202 2024-03-06T07:20:46.662 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 178/202 2024-03-06T07:20:46.688 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 179/202 2024-03-06T07:20:46.689 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:20:46.777 DEBUG:teuthology.parallel:result is None 2024-03-06T07:20:46.781 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:20:46.820 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 181/202 2024-03-06T07:20:46.848 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 182/202 2024-03-06T07:20:46.848 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:20:46.916 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:20:46.943 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 184/202 2024-03-06T07:20:46.976 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 185/202 2024-03-06T07:20:47.001 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : iproute-tc-6.2.0-5.el9.x86_64 186/202 2024-03-06T07:20:47.035 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 187/202 2024-03-06T07:20:47.069 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : iproute-6.2.0-5.el9.x86_64 188/202 2024-03-06T07:20:47.102 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 189/202 2024-03-06T07:20:47.150 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 190/202 2024-03-06T07:20:47.182 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 191/202 2024-03-06T07:20:47.207 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 192/202 2024-03-06T07:20:47.241 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 193/202 2024-03-06T07:20:47.267 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:20:47.292 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 195/202 2024-03-06T07:20:47.298 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:20:47.298 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:20:47.478 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:20:47.479 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:20:47.561 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:20:47.593 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:20:47.619 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 199/202 2024-03-06T07:20:47.645 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 200/202 2024-03-06T07:20:47.671 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 201/202 2024-03-06T07:20:47.736 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:20:52.697 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:20:52.697 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 1/202 2024-03-06T07:20:52.697 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2/202 2024-03-06T07:20:52.697 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 3/202 2024-03-06T07:20:52.697 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 4/202 2024-03-06T07:20:52.697 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 5/202 2024-03-06T07:20:52.697 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 6/202 2024-03-06T07:20:52.697 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 7/202 2024-03-06T07:20:52.698 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 8/202 2024-03-06T07:20:52.698 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 9/202 2024-03-06T07:20:52.698 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 10/202 2024-03-06T07:20:52.698 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:20:52.698 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:20:52.698 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:20:52.698 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 14/202 2024-03-06T07:20:52.698 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 15/202 2024-03-06T07:20:52.698 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:20:52.698 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 17/202 2024-03-06T07:20:52.698 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 18/202 2024-03-06T07:20:52.698 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 19/202 2024-03-06T07:20:52.698 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 20/202 2024-03-06T07:20:52.698 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 21/202 2024-03-06T07:20:52.699 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 22/202 2024-03-06T07:20:52.699 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 23/202 2024-03-06T07:20:52.699 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 24/202 2024-03-06T07:20:52.699 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 25/202 2024-03-06T07:20:52.699 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 26/202 2024-03-06T07:20:52.699 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 27/202 2024-03-06T07:20:52.699 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 28/202 2024-03-06T07:20:52.699 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 29/202 2024-03-06T07:20:52.699 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 30/202 2024-03-06T07:20:52.699 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 31/202 2024-03-06T07:20:52.699 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 32/202 2024-03-06T07:20:52.699 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 33/202 2024-03-06T07:20:52.699 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 34/202 2024-03-06T07:20:52.699 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 35/202 2024-03-06T07:20:52.700 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 36/202 2024-03-06T07:20:52.700 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 37/202 2024-03-06T07:20:52.700 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 38/202 2024-03-06T07:20:52.700 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 39/202 2024-03-06T07:20:52.700 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/202 2024-03-06T07:20:52.700 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/202 2024-03-06T07:20:52.700 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/202 2024-03-06T07:20:52.700 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 43/202 2024-03-06T07:20:52.700 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : flex-2.6.4-9.el9.x86_64 44/202 2024-03-06T07:20:52.700 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 45/202 2024-03-06T07:20:52.700 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 46/202 2024-03-06T07:20:52.700 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 47/202 2024-03-06T07:20:52.700 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/202 2024-03-06T07:20:52.700 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/202 2024-03-06T07:20:52.701 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/202 2024-03-06T07:20:52.701 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/202 2024-03-06T07:20:52.701 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/202 2024-03-06T07:20:52.701 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 53/202 2024-03-06T07:20:52.701 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/202 2024-03-06T07:20:52.701 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/202 2024-03-06T07:20:52.701 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/202 2024-03-06T07:20:52.701 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 57/202 2024-03-06T07:20:52.701 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 58/202 2024-03-06T07:20:52.701 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 59/202 2024-03-06T07:20:52.701 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/202 2024-03-06T07:20:52.701 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/202 2024-03-06T07:20:52.701 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/202 2024-03-06T07:20:52.701 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/202 2024-03-06T07:20:52.702 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 64/202 2024-03-06T07:20:52.702 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 65/202 2024-03-06T07:20:52.702 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 66/202 2024-03-06T07:20:52.702 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 67/202 2024-03-06T07:20:52.702 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 68/202 2024-03-06T07:20:52.702 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 69/202 2024-03-06T07:20:52.702 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 70/202 2024-03-06T07:20:52.702 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/202 2024-03-06T07:20:52.702 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 72/202 2024-03-06T07:20:52.702 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 73/202 2024-03-06T07:20:52.702 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 74/202 2024-03-06T07:20:52.702 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 75/202 2024-03-06T07:20:52.702 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 76/202 2024-03-06T07:20:52.702 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 77/202 2024-03-06T07:20:52.703 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 78/202 2024-03-06T07:20:52.703 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 79/202 2024-03-06T07:20:52.703 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 80/202 2024-03-06T07:20:52.703 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/202 2024-03-06T07:20:52.703 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 82/202 2024-03-06T07:20:52.703 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 83/202 2024-03-06T07:20:52.703 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 84/202 2024-03-06T07:20:52.703 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 85/202 2024-03-06T07:20:52.703 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 86/202 2024-03-06T07:20:52.703 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 87/202 2024-03-06T07:20:52.703 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 88/202 2024-03-06T07:20:52.703 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 89/202 2024-03-06T07:20:52.703 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 90/202 2024-03-06T07:20:52.703 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 91/202 2024-03-06T07:20:52.704 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 92/202 2024-03-06T07:20:52.704 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 93/202 2024-03-06T07:20:52.704 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : jq-1.6-16.el9.x86_64 94/202 2024-03-06T07:20:52.704 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 95/202 2024-03-06T07:20:52.704 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 96/202 2024-03-06T07:20:52.704 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 97/202 2024-03-06T07:20:52.704 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 98/202 2024-03-06T07:20:52.704 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 99/202 2024-03-06T07:20:52.704 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-ply-3.11-14.el9.noarch 100/202 2024-03-06T07:20:52.704 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 101/202 2024-03-06T07:20:52.704 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 102/202 2024-03-06T07:20:52.704 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 103/202 2024-03-06T07:20:52.704 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 104/202 2024-03-06T07:20:52.705 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/202 2024-03-06T07:20:52.705 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/202 2024-03-06T07:20:52.705 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/202 2024-03-06T07:20:52.705 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/202 2024-03-06T07:20:52.705 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/202 2024-03-06T07:20:52.705 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/202 2024-03-06T07:20:52.705 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/202 2024-03-06T07:20:52.705 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/202 2024-03-06T07:20:52.705 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/202 2024-03-06T07:20:52.705 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/202 2024-03-06T07:20:52.705 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/202 2024-03-06T07:20:52.705 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/202 2024-03-06T07:20:52.705 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/202 2024-03-06T07:20:52.705 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/202 2024-03-06T07:20:52.706 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/202 2024-03-06T07:20:52.706 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/202 2024-03-06T07:20:52.706 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/202 2024-03-06T07:20:52.706 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/202 2024-03-06T07:20:52.706 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/202 2024-03-06T07:20:52.706 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/202 2024-03-06T07:20:52.706 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/202 2024-03-06T07:20:52.706 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/202 2024-03-06T07:20:52.706 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/202 2024-03-06T07:20:52.706 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/202 2024-03-06T07:20:52.706 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/202 2024-03-06T07:20:52.706 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/202 2024-03-06T07:20:52.706 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/202 2024-03-06T07:20:52.706 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/202 2024-03-06T07:20:52.707 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/202 2024-03-06T07:20:52.707 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/202 2024-03-06T07:20:52.707 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/202 2024-03-06T07:20:52.707 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/202 2024-03-06T07:20:52.707 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/202 2024-03-06T07:20:52.707 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/202 2024-03-06T07:20:52.707 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/202 2024-03-06T07:20:52.707 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/202 2024-03-06T07:20:52.707 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/202 2024-03-06T07:20:52.707 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/202 2024-03-06T07:20:52.707 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/202 2024-03-06T07:20:52.708 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/202 2024-03-06T07:20:52.708 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/202 2024-03-06T07:20:52.708 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/202 2024-03-06T07:20:52.708 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/202 2024-03-06T07:20:52.708 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/202 2024-03-06T07:20:52.708 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/202 2024-03-06T07:20:52.708 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/202 2024-03-06T07:20:52.708 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 151/202 2024-03-06T07:20:52.708 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/202 2024-03-06T07:20:52.708 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:20:52.708 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/202 2024-03-06T07:20:52.708 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 155/202 2024-03-06T07:20:52.709 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 156/202 2024-03-06T07:20:52.709 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/202 2024-03-06T07:20:52.709 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/202 2024-03-06T07:20:52.709 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/202 2024-03-06T07:20:52.709 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/202 2024-03-06T07:20:52.709 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/202 2024-03-06T07:20:52.709 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/202 2024-03-06T07:20:52.709 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/202 2024-03-06T07:20:52.709 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/202 2024-03-06T07:20:52.709 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/202 2024-03-06T07:20:52.709 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/202 2024-03-06T07:20:52.709 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/202 2024-03-06T07:20:52.709 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/202 2024-03-06T07:20:52.709 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/202 2024-03-06T07:20:52.710 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/202 2024-03-06T07:20:52.710 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/202 2024-03-06T07:20:52.710 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/202 2024-03-06T07:20:52.710 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : iproute-6.7.0-2.el9.x86_64 173/202 2024-03-06T07:20:52.710 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : iproute-6.2.0-5.el9.x86_64 174/202 2024-03-06T07:20:52.710 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : iproute-tc-6.7.0-2.el9.x86_64 175/202 2024-03-06T07:20:52.710 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : iproute-tc-6.2.0-5.el9.x86_64 176/202 2024-03-06T07:20:52.710 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 177/202 2024-03-06T07:20:52.710 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 178/202 2024-03-06T07:20:52.710 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 179/202 2024-03-06T07:20:52.710 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 180/202 2024-03-06T07:20:52.710 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 181/202 2024-03-06T07:20:52.710 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 182/202 2024-03-06T07:20:52.710 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 183/202 2024-03-06T07:20:52.710 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 184/202 2024-03-06T07:20:52.711 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 185/202 2024-03-06T07:20:52.711 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 186/202 2024-03-06T07:20:52.711 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 187/202 2024-03-06T07:20:52.711 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 188/202 2024-03-06T07:20:52.711 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 189/202 2024-03-06T07:20:52.711 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 190/202 2024-03-06T07:20:52.711 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 191/202 2024-03-06T07:20:52.711 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 192/202 2024-03-06T07:20:52.711 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 193/202 2024-03-06T07:20:52.711 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:20:52.711 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 195/202 2024-03-06T07:20:52.711 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 196/202 2024-03-06T07:20:52.711 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 197/202 2024-03-06T07:20:52.711 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:20:52.712 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 199/202 2024-03-06T07:20:52.712 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 200/202 2024-03-06T07:20:52.712 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 201/202 2024-03-06T07:20:55.519 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 202/202 2024-03-06T07:20:55.519 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:20:55.520 INFO:teuthology.orchestra.run.smithi083.stdout:Upgraded: 2024-03-06T07:20:55.520 INFO:teuthology.orchestra.run.smithi083.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-06T07:20:55.520 INFO:teuthology.orchestra.run.smithi083.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-06T07:20:55.520 INFO:teuthology.orchestra.run.smithi083.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-06T07:20:55.520 INFO:teuthology.orchestra.run.smithi083.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-06T07:20:55.520 INFO:teuthology.orchestra.run.smithi083.stdout: acl-2.3.1-4.el9.x86_64 2024-03-06T07:20:55.520 INFO:teuthology.orchestra.run.smithi083.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-06T07:20:55.520 INFO:teuthology.orchestra.run.smithi083.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-06T07:20:55.520 INFO:teuthology.orchestra.run.smithi083.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-06T07:20:55.520 INFO:teuthology.orchestra.run.smithi083.stdout: iproute-6.7.0-2.el9.x86_64 2024-03-06T07:20:55.520 INFO:teuthology.orchestra.run.smithi083.stdout: iproute-tc-6.7.0-2.el9.x86_64 2024-03-06T07:20:55.521 INFO:teuthology.orchestra.run.smithi083.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-06T07:20:55.521 INFO:teuthology.orchestra.run.smithi083.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-06T07:20:55.521 INFO:teuthology.orchestra.run.smithi083.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-06T07:20:55.521 INFO:teuthology.orchestra.run.smithi083.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-06T07:20:55.521 INFO:teuthology.orchestra.run.smithi083.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-06T07:20:55.521 INFO:teuthology.orchestra.run.smithi083.stdout: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:55.521 INFO:teuthology.orchestra.run.smithi083.stdout: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:55.521 INFO:teuthology.orchestra.run.smithi083.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-06T07:20:55.521 INFO:teuthology.orchestra.run.smithi083.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-06T07:20:55.521 INFO:teuthology.orchestra.run.smithi083.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-06T07:20:55.521 INFO:teuthology.orchestra.run.smithi083.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-06T07:20:55.521 INFO:teuthology.orchestra.run.smithi083.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-06T07:20:55.522 INFO:teuthology.orchestra.run.smithi083.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-06T07:20:55.522 INFO:teuthology.orchestra.run.smithi083.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-06T07:20:55.522 INFO:teuthology.orchestra.run.smithi083.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-06T07:20:55.522 INFO:teuthology.orchestra.run.smithi083.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-06T07:20:55.522 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-06T07:20:55.522 INFO:teuthology.orchestra.run.smithi083.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-06T07:20:55.522 INFO:teuthology.orchestra.run.smithi083.stdout: automake-1.16.2-8.el9.noarch 2024-03-06T07:20:55.522 INFO:teuthology.orchestra.run.smithi083.stdout: bison-3.7.4-5.el9.x86_64 2024-03-06T07:20:55.522 INFO:teuthology.orchestra.run.smithi083.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-06T07:20:55.522 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:55.522 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:55.522 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:55.523 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:55.523 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:55.523 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-immutable-object-cache-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:55.523 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:55.523 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:55.523 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:55.523 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:55.523 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:55.523 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:55.523 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:55.523 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:55.523 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:55.523 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:55.524 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:55.524 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:55.524 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:55.524 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:55.524 INFO:teuthology.orchestra.run.smithi083.stdout: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:55.524 INFO:teuthology.orchestra.run.smithi083.stdout: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:55.525 INFO:teuthology.orchestra.run.smithi083.stdout: cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:55.525 INFO:teuthology.orchestra.run.smithi083.stdout: cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:55.525 INFO:teuthology.orchestra.run.smithi083.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-06T07:20:55.525 INFO:teuthology.orchestra.run.smithi083.stdout: flex-2.6.4-9.el9.x86_64 2024-03-06T07:20:55.525 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-06T07:20:55.525 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-06T07:20:55.525 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-06T07:20:55.526 INFO:teuthology.orchestra.run.smithi083.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-06T07:20:55.526 INFO:teuthology.orchestra.run.smithi083.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-06T07:20:55.526 INFO:teuthology.orchestra.run.smithi083.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-06T07:20:55.526 INFO:teuthology.orchestra.run.smithi083.stdout: jq-1.6-16.el9.x86_64 2024-03-06T07:20:55.526 INFO:teuthology.orchestra.run.smithi083.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-06T07:20:55.526 INFO:teuthology.orchestra.run.smithi083.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-06T07:20:55.527 INFO:teuthology.orchestra.run.smithi083.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-06T07:20:55.527 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:55.527 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:55.527 INFO:teuthology.orchestra.run.smithi083.stdout: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:55.527 INFO:teuthology.orchestra.run.smithi083.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-06T07:20:55.527 INFO:teuthology.orchestra.run.smithi083.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-06T07:20:55.528 INFO:teuthology.orchestra.run.smithi083.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-06T07:20:55.528 INFO:teuthology.orchestra.run.smithi083.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-06T07:20:55.528 INFO:teuthology.orchestra.run.smithi083.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-06T07:20:55.528 INFO:teuthology.orchestra.run.smithi083.stdout: librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:55.528 INFO:teuthology.orchestra.run.smithi083.stdout: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:55.528 INFO:teuthology.orchestra.run.smithi083.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-06T07:20:55.529 INFO:teuthology.orchestra.run.smithi083.stdout: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:55.529 INFO:teuthology.orchestra.run.smithi083.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-06T07:20:55.529 INFO:teuthology.orchestra.run.smithi083.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-06T07:20:55.529 INFO:teuthology.orchestra.run.smithi083.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-06T07:20:55.529 INFO:teuthology.orchestra.run.smithi083.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-06T07:20:55.529 INFO:teuthology.orchestra.run.smithi083.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-06T07:20:55.530 INFO:teuthology.orchestra.run.smithi083.stdout: lua-5.4.4-4.el9.x86_64 2024-03-06T07:20:55.530 INFO:teuthology.orchestra.run.smithi083.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-06T07:20:55.530 INFO:teuthology.orchestra.run.smithi083.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-06T07:20:55.530 INFO:teuthology.orchestra.run.smithi083.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-06T07:20:55.530 INFO:teuthology.orchestra.run.smithi083.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-06T07:20:55.530 INFO:teuthology.orchestra.run.smithi083.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-06T07:20:55.531 INFO:teuthology.orchestra.run.smithi083.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-06T07:20:55.531 INFO:teuthology.orchestra.run.smithi083.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-06T07:20:55.531 INFO:teuthology.orchestra.run.smithi083.stdout: postgresql-13.11-1.el9.x86_64 2024-03-06T07:20:55.531 INFO:teuthology.orchestra.run.smithi083.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-06T07:20:55.531 INFO:teuthology.orchestra.run.smithi083.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-06T07:20:55.531 INFO:teuthology.orchestra.run.smithi083.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-06T07:20:55.532 INFO:teuthology.orchestra.run.smithi083.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-06T07:20:55.532 INFO:teuthology.orchestra.run.smithi083.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-06T07:20:55.532 INFO:teuthology.orchestra.run.smithi083.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-06T07:20:55.532 INFO:teuthology.orchestra.run.smithi083.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-06T07:20:55.532 INFO:teuthology.orchestra.run.smithi083.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-06T07:20:55.533 INFO:teuthology.orchestra.run.smithi083.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-06T07:20:55.533 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-06T07:20:55.533 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:55.533 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:55.533 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:55.533 INFO:teuthology.orchestra.run.smithi083.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-06T07:20:55.534 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-06T07:20:55.534 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-06T07:20:55.534 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-06T07:20:55.534 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-06T07:20:55.534 INFO:teuthology.orchestra.run.smithi083.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-06T07:20:55.535 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-06T07:20:55.535 INFO:teuthology.orchestra.run.smithi083.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-06T07:20:55.535 INFO:teuthology.orchestra.run.smithi083.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-06T07:20:55.535 INFO:teuthology.orchestra.run.smithi083.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-06T07:20:55.535 INFO:teuthology.orchestra.run.smithi083.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-06T07:20:55.535 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-06T07:20:55.536 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-06T07:20:55.536 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-06T07:20:55.536 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-06T07:20:55.536 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-06T07:20:55.536 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-06T07:20:55.536 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-06T07:20:55.537 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-06T07:20:55.537 INFO:teuthology.orchestra.run.smithi083.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-06T07:20:55.537 INFO:teuthology.orchestra.run.smithi083.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-06T07:20:55.537 INFO:teuthology.orchestra.run.smithi083.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-06T07:20:55.537 INFO:teuthology.orchestra.run.smithi083.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-06T07:20:55.537 INFO:teuthology.orchestra.run.smithi083.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-06T07:20:55.538 INFO:teuthology.orchestra.run.smithi083.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-06T07:20:55.538 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-06T07:20:55.538 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-06T07:20:55.538 INFO:teuthology.orchestra.run.smithi083.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-06T07:20:55.538 INFO:teuthology.orchestra.run.smithi083.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-06T07:20:55.539 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-06T07:20:55.539 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ply-3.11-14.el9.noarch 2024-03-06T07:20:55.539 INFO:teuthology.orchestra.run.smithi083.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-06T07:20:55.539 INFO:teuthology.orchestra.run.smithi083.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-06T07:20:55.539 INFO:teuthology.orchestra.run.smithi083.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-06T07:20:55.539 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-06T07:20:55.540 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-06T07:20:55.540 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-06T07:20:55.540 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-06T07:20:55.540 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-06T07:20:55.540 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-06T07:20:55.540 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:55.541 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:55.541 INFO:teuthology.orchestra.run.smithi083.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-06T07:20:55.541 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-06T07:20:55.541 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-06T07:20:55.541 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:55.542 INFO:teuthology.orchestra.run.smithi083.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-06T07:20:55.542 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-06T07:20:55.542 INFO:teuthology.orchestra.run.smithi083.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-06T07:20:55.542 INFO:teuthology.orchestra.run.smithi083.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-06T07:20:55.542 INFO:teuthology.orchestra.run.smithi083.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-06T07:20:55.542 INFO:teuthology.orchestra.run.smithi083.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-06T07:20:55.543 INFO:teuthology.orchestra.run.smithi083.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-06T07:20:55.543 INFO:teuthology.orchestra.run.smithi083.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-06T07:20:55.543 INFO:teuthology.orchestra.run.smithi083.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-06T07:20:55.543 INFO:teuthology.orchestra.run.smithi083.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-06T07:20:55.543 INFO:teuthology.orchestra.run.smithi083.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-06T07:20:55.543 INFO:teuthology.orchestra.run.smithi083.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-06T07:20:55.544 INFO:teuthology.orchestra.run.smithi083.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-06T07:20:55.544 INFO:teuthology.orchestra.run.smithi083.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-06T07:20:55.544 INFO:teuthology.orchestra.run.smithi083.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-06T07:20:55.544 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:55.544 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:55.545 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:55.545 INFO:teuthology.orchestra.run.smithi083.stdout: re2-1:20211101-3.el9.x86_64 2024-03-06T07:20:55.545 INFO:teuthology.orchestra.run.smithi083.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-06T07:20:55.545 INFO:teuthology.orchestra.run.smithi083.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-06T07:20:55.545 INFO:teuthology.orchestra.run.smithi083.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-06T07:20:55.545 INFO:teuthology.orchestra.run.smithi083.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-06T07:20:55.546 INFO:teuthology.orchestra.run.smithi083.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-06T07:20:55.546 INFO:teuthology.orchestra.run.smithi083.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-06T07:20:55.547 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:20:55.547 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-06T07:20:56.175 DEBUG:teuthology.parallel:result is None 2024-03-06T07:20:56.176 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:20:56.415 DEBUG:teuthology.orchestra.run.smithi083:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-06T07:20:56.437 INFO:teuthology.orchestra.run.smithi083.stdout:19.0.0-1739.g1bc250c7.el9 2024-03-06T07:20:56.438 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1739.g1bc250c7.el9 2024-03-06T07:20:56.438 INFO:teuthology.task.install:The correct ceph version 19.0.0-1739.g1bc250c7 is installed. 2024-03-06T07:20:56.440 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:20:56.581 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-06T07:20:56.607 INFO:teuthology.orchestra.run.smithi183.stdout:19.0.0-1739.g1bc250c7.el9 2024-03-06T07:20:56.608 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1739.g1bc250c7.el9 2024-03-06T07:20:56.608 INFO:teuthology.task.install:The correct ceph version 19.0.0-1739.g1bc250c7 is installed. 2024-03-06T07:20:56.610 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-06T07:20:56.610 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:20:56.610 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-06T07:20:56.642 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:20:56.642 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-06T07:20:56.683 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-06T07:20:56.683 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:20:56.683 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/daemon-helper 2024-03-06T07:20:56.712 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-06T07:20:56.780 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:20:56.780 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/daemon-helper 2024-03-06T07:20:56.810 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-06T07:20:56.884 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-06T07:20:56.884 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:20:56.884 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-06T07:20:56.912 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-06T07:20:56.979 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:20:56.979 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-06T07:20:57.010 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-06T07:20:57.080 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-06T07:20:57.080 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:20:57.080 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/stdin-killer 2024-03-06T07:20:57.107 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-06T07:20:57.174 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:20:57.175 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/stdin-killer 2024-03-06T07:20:57.204 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-06T07:20:57.273 INFO:teuthology.run_tasks:Running task ceph... 2024-03-06T07:20:57.366 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-06T07:20:57.366 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 777 /var/log/ceph 2024-03-06T07:20:57.368 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 777 /var/log/ceph 2024-03-06T07:20:57.396 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-06T07:20:57.396 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-06T07:20:57.435 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-06T07:20:57.464 INFO:tasks.ceph:Creating extra log directories... 2024-03-06T07:20:57.464 DEBUG:teuthology.orchestra.run.smithi083:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-06T07:20:57.503 DEBUG:teuthology.orchestra.run.smithi183:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-06T07:20:57.537 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-06T07:20:57.537 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-06T07:20:57.537 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-06T07:20:57.537 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:20:57.537 DEBUG:teuthology.orchestra.run.smithi083:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-06T07:20:57.582 DEBUG:teuthology.orchestra.run.smithi083:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-06T07:20:57.701 DEBUG:teuthology.orchestra.run.smithi083:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-06T07:20:57.727 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:20:57.727 DEBUG:teuthology.orchestra.run.smithi183:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-06T07:20:57.742 DEBUG:teuthology.orchestra.run.smithi183:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-06T07:20:57.859 DEBUG:teuthology.orchestra.run.smithi183:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-06T07:20:57.893 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-06T07:20:57.893 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-06T07:20:57.893 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583062', 'branch': 'wip-lusov-qdb', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays}', 'email': None, 'first_in_suite': False, 'job_id': '7583062', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-lusov-qdb'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}}, 'subvolume_version': 2, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-lusov-qdb', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}}, 'owner': 'scheduled_leonidus@teuthology', 'priority': 70, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 5072, 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'sleep_before_teardown': 0, 'subset': '246/10000', 'suite': 'fs', 'suite_branch': 'wip-lusov-qdb', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi083.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE1mL0BgMoVtpCUXipFaslZ37m3Ygh93+0MQR210oMZBMyGW4vvH8MwfKedU06uBiOY5eGNbiCMsbp+TQAsy7ws=', 'smithi183.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAKTQjnjlPE3geR+z/SVB8x392UFEtbZMHxHz917JIHBT0aUi9su9UZc5NVIVWcumwM6TE0zNyML9E9hoUUTpSc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'cluster': 'ceph'}}, {'ceph-fuse': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_strays']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-06_07:04:29', 'tube': 'smithi', 'user': 'leonidus', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-06T07:20:57.894 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-06T07:20:57.896 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-06T07:20:57.941 DEBUG:teuthology.orchestra.run.smithi083:> sudo install -d -m0777 -- /var/run/ceph 2024-03-06T07:20:57.952 DEBUG:teuthology.orchestra.run.smithi183:> sudo install -d -m0777 -- /var/run/ceph 2024-03-06T07:20:58.008 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:20:58.008 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/scratch_devs of=/dev/stdout 2024-03-06T07:20:58.032 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-06T07:20:58.032 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_1 2024-03-06T07:20:58.086 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-06T07:20:58.086 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:20:58.086 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-06T07:20:58.086 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:20:58.086 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:20:58.086 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-03-06 07:20:48.087356411 +0000 2024-03-06T07:20:58.087 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-03-06 07:18:11.731004251 +0000 2024-03-06T07:20:58.087 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-03-06 07:18:11.731004251 +0000 2024-03-06T07:20:58.087 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-03-06 07:18:11.731004251 +0000 2024-03-06T07:20:58.087 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-06T07:20:58.152 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-03-06T07:20:58.152 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-03-06T07:20:58.152 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000203422 s, 2.5 MB/s 2024-03-06T07:20:58.153 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-06T07:20:58.209 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_2 2024-03-06T07:20:58.264 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-06T07:20:58.264 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:20:58.264 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-06T07:20:58.264 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:20:58.264 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:20:58.264 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-03-06 07:20:48.088356388 +0000 2024-03-06T07:20:58.265 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-03-06 07:18:12.076996365 +0000 2024-03-06T07:20:58.265 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-03-06 07:18:12.076996365 +0000 2024-03-06T07:20:58.265 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-03-06 07:18:12.076996365 +0000 2024-03-06T07:20:58.265 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-06T07:20:58.330 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-03-06T07:20:58.330 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-03-06T07:20:58.330 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000200328 s, 2.6 MB/s 2024-03-06T07:20:58.331 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-06T07:20:58.387 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_3 2024-03-06T07:20:58.441 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-06T07:20:58.441 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:20:58.441 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-06T07:20:58.441 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:20:58.441 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:20:58.441 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-03-06 07:20:48.088356388 +0000 2024-03-06T07:20:58.441 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-03-06 07:18:12.403988911 +0000 2024-03-06T07:20:58.441 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-03-06 07:18:12.403988911 +0000 2024-03-06T07:20:58.442 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-03-06 07:18:12.403988911 +0000 2024-03-06T07:20:58.442 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-06T07:20:58.507 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-03-06T07:20:58.507 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-03-06T07:20:58.507 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000210832 s, 2.4 MB/s 2024-03-06T07:20:58.508 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-06T07:20:58.564 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_4 2024-03-06T07:20:58.618 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-06T07:20:58.619 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:20:58.619 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-06T07:20:58.619 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:20:58.619 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:20:58.620 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-03-06 07:20:48.088356388 +0000 2024-03-06T07:20:58.620 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-03-06 07:18:12.739981252 +0000 2024-03-06T07:20:58.620 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-03-06 07:18:12.739981252 +0000 2024-03-06T07:20:58.620 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-03-06 07:18:12.739981252 +0000 2024-03-06T07:20:58.620 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-06T07:20:58.684 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-03-06T07:20:58.684 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-03-06T07:20:58.684 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000181026 s, 2.8 MB/s 2024-03-06T07:20:58.685 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-06T07:20:58.741 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:20:58.741 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:20:58.741 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/scratch_devs of=/dev/stdout 2024-03-06T07:20:58.757 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-06T07:20:58.758 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_1 2024-03-06T07:20:58.812 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-06T07:20:58.813 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:20:58.813 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-06T07:20:58.813 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:20:58.813 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:20:58.813 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-06 07:20:38.870531901 +0000 2024-03-06T07:20:58.813 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-06 07:18:11.728726100 +0000 2024-03-06T07:20:58.813 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-06 07:18:11.728726100 +0000 2024-03-06T07:20:58.813 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-06 07:18:11.728726100 +0000 2024-03-06T07:20:58.813 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-06T07:20:58.879 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-06T07:20:58.879 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-06T07:20:58.879 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000331205 s, 1.5 MB/s 2024-03-06T07:20:58.880 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-06T07:20:58.936 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_2 2024-03-06T07:20:58.991 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-06T07:20:58.991 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:20:58.992 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-06T07:20:58.992 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:20:58.992 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:20:58.992 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-06 07:20:38.870531901 +0000 2024-03-06T07:20:58.992 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-06 07:18:12.073717044 +0000 2024-03-06T07:20:58.992 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-06 07:18:12.073717044 +0000 2024-03-06T07:20:58.992 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-06 07:18:12.073717044 +0000 2024-03-06T07:20:58.992 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-06T07:20:59.056 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-06T07:20:59.057 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-06T07:20:59.057 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000184775 s, 2.8 MB/s 2024-03-06T07:20:59.058 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-06T07:20:59.114 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_3 2024-03-06T07:20:59.169 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-06T07:20:59.169 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:20:59.169 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-06T07:20:59.169 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:20:59.169 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:20:59.169 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-06 07:20:38.870531901 +0000 2024-03-06T07:20:59.169 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-06 07:18:12.410708199 +0000 2024-03-06T07:20:59.169 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-06 07:18:12.410708199 +0000 2024-03-06T07:20:59.169 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-06 07:18:12.410708199 +0000 2024-03-06T07:20:59.170 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-06T07:20:59.234 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-06T07:20:59.235 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-06T07:20:59.235 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000204702 s, 2.5 MB/s 2024-03-06T07:20:59.236 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-06T07:20:59.292 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_4 2024-03-06T07:20:59.347 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-06T07:20:59.347 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:20:59.347 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-06T07:20:59.347 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:20:59.347 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:20:59.347 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-06 07:20:38.871531875 +0000 2024-03-06T07:20:59.347 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-06 07:18:12.733699721 +0000 2024-03-06T07:20:59.347 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-06 07:18:12.733699721 +0000 2024-03-06T07:20:59.347 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-06 07:18:12.733699721 +0000 2024-03-06T07:20:59.348 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-06T07:20:59.412 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-06T07:20:59.412 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-06T07:20:59.412 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.00019095 s, 2.7 MB/s 2024-03-06T07:20:59.413 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-06T07:20:59.468 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:20:59.469 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi083.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi183.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-06T07:20:59.469 INFO:tasks.ceph:Generating config... 2024-03-06T07:20:59.470 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-06T07:20:59.470 INFO:tasks.ceph:[client] debug client = 20 2024-03-06T07:20:59.470 INFO:tasks.ceph:[client] debug ms = 1 2024-03-06T07:20:59.470 INFO:tasks.ceph:[client] pid file = 2024-03-06T07:20:59.470 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-06T07:20:59.470 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-06T07:20:59.470 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-06T07:20:59.470 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-06T07:20:59.470 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-06T07:20:59.470 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-06T07:20:59.470 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-06T07:20:59.471 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-06T07:20:59.471 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-06T07:20:59.471 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-06T07:20:59.471 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-06T07:20:59.471 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-06T07:20:59.471 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-06T07:20:59.471 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-06T07:20:59.471 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-06T07:20:59.471 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-06T07:20:59.471 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-06T07:20:59.471 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-06T07:20:59.471 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-06T07:20:59.471 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-06T07:20:59.471 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-03-06T07:20:59.471 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-06T07:20:59.472 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-06T07:20:59.472 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-03-06T07:20:59.472 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-03-06T07:20:59.472 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-06T07:20:59.472 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-06T07:20:59.472 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-03-06T07:20:59.472 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-06T07:20:59.472 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-06T07:20:59.472 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-06T07:20:59.472 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-06T07:20:59.472 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-06T07:20:59.472 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-06T07:20:59.472 INFO:tasks.ceph:Setting up mon.a... 2024-03-06T07:20:59.473 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-06T07:20:59.523 INFO:teuthology.orchestra.run.smithi083.stdout:creating /etc/ceph/ceph.keyring 2024-03-06T07:20:59.527 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-06T07:20:59.573 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-06T07:20:59.601 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.83'), ('mon.b', '172.21.15.183'), ('mon.c', '[v2:172.21.15.183:3301,v1:172.21.15.183:6790]')] 2024-03-06T07:20:59.601 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.83,172.21.15.183,[v2:172.21.15.183:3301,v1:172.21.15.183:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-06T07:20:59.602 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:20:59.602 DEBUG:teuthology.orchestra.run.smithi083:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-06T07:20:59.656 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.83 --add b 172.21.15.183 --addv c '[v2:172.21.15.183:3301,v1:172.21.15.183:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:20:59.740 INFO:teuthology.orchestra.run.smithi083.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:20:59.741 INFO:teuthology.orchestra.run.smithi083.stdout:monmaptool: generated fsid 039877dd-d2ce-45d2-8e77-ac5eb7e1d55a 2024-03-06T07:20:59.741 INFO:teuthology.orchestra.run.smithi083.stdout:setting min_mon_release = quincy 2024-03-06T07:20:59.741 INFO:teuthology.orchestra.run.smithi083.stdout:epoch 0 2024-03-06T07:20:59.741 INFO:teuthology.orchestra.run.smithi083.stdout:fsid 039877dd-d2ce-45d2-8e77-ac5eb7e1d55a 2024-03-06T07:20:59.741 INFO:teuthology.orchestra.run.smithi083.stdout:last_changed 2024-03-06T07:20:59.740088+0000 2024-03-06T07:20:59.741 INFO:teuthology.orchestra.run.smithi083.stdout:created 2024-03-06T07:20:59.740088+0000 2024-03-06T07:20:59.741 INFO:teuthology.orchestra.run.smithi083.stdout:min_mon_release 17 (quincy) 2024-03-06T07:20:59.741 INFO:teuthology.orchestra.run.smithi083.stdout:election_strategy: 1 2024-03-06T07:20:59.741 INFO:teuthology.orchestra.run.smithi083.stdout:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-06T07:20:59.741 INFO:teuthology.orchestra.run.smithi083.stdout:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2024-03-06T07:20:59.741 INFO:teuthology.orchestra.run.smithi083.stdout:2: [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] mon.c 2024-03-06T07:20:59.741 INFO:teuthology.orchestra.run.smithi083.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-06T07:20:59.743 DEBUG:teuthology.orchestra.run.smithi083:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-06T07:20:59.757 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 039877dd-d2ce-45d2-8e77-ac5eb7e1d55a... 2024-03-06T07:20:59.758 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:20:59.800 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:20:59.848 INFO:teuthology.orchestra.run.smithi083.stdout:[global] 2024-03-06T07:20:59.849 INFO:teuthology.orchestra.run.smithi083.stdout: chdir = "" 2024-03-06T07:20:59.849 INFO:teuthology.orchestra.run.smithi083.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:20:59.849 INFO:teuthology.orchestra.run.smithi083.stdout: auth supported = cephx 2024-03-06T07:20:59.849 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:20:59.849 INFO:teuthology.orchestra.run.smithi083.stdout: filestore xattr use omap = true 2024-03-06T07:20:59.849 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:20:59.849 INFO:teuthology.orchestra.run.smithi083.stdout: mon clock drift allowed = 1.000 2024-03-06T07:20:59.849 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:20:59.849 INFO:teuthology.orchestra.run.smithi083.stdout: osd crush chooseleaf type = 0 2024-03-06T07:20:59.849 INFO:teuthology.orchestra.run.smithi083.stdout: auth debug = true 2024-03-06T07:20:59.850 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:20:59.850 INFO:teuthology.orchestra.run.smithi083.stdout: ms die on old message = true 2024-03-06T07:20:59.850 INFO:teuthology.orchestra.run.smithi083.stdout: ms die on bug = true 2024-03-06T07:20:59.851 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:20:59.851 INFO:teuthology.orchestra.run.smithi083.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:20:59.851 INFO:teuthology.orchestra.run.smithi083.stdout: mon pg warn max object skew = 0 2024-03-06T07:20:59.851 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:20:59.851 INFO:teuthology.orchestra.run.smithi083.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:20:59.851 INFO:teuthology.orchestra.run.smithi083.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:20:59.851 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:20:59.851 INFO:teuthology.orchestra.run.smithi083.stdout: osd pool default size = 2 2024-03-06T07:20:59.851 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:20:59.851 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd allow primary affinity = true 2024-03-06T07:20:59.851 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd allow pg remap = true 2024-03-06T07:20:59.851 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:20:59.851 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:20:59.851 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on no sortbitwise = false 2024-03-06T07:20:59.852 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:20:59.852 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on too few osds = false 2024-03-06T07:20:59.852 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:20:59.852 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:20:59.852 INFO:teuthology.orchestra.run.smithi083.stdout: mon_allow_pool_size_one = true 2024-03-06T07:20:59.852 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:20:59.852 INFO:teuthology.orchestra.run.smithi083.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:20:59.852 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:20:59.852 INFO:teuthology.orchestra.run.smithi083.stdout: osd default data pool replay window = 5 2024-03-06T07:20:59.852 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:20:59.852 INFO:teuthology.orchestra.run.smithi083.stdout: mon allow pool delete = true 2024-03-06T07:20:59.852 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:20:59.852 INFO:teuthology.orchestra.run.smithi083.stdout: mon cluster log file level = debug 2024-03-06T07:20:59.852 INFO:teuthology.orchestra.run.smithi083.stdout: debug asserts on shutdown = true 2024-03-06T07:20:59.853 INFO:teuthology.orchestra.run.smithi083.stdout: mon health detail to clog = false 2024-03-06T07:20:59.853 INFO:teuthology.orchestra.run.smithi083.stdout: mon host = "172.21.15.83,172.21.15.183,[v2:172.21.15.183:3301,v1:172.21.15.183:6790]" 2024-03-06T07:20:59.853 INFO:teuthology.orchestra.run.smithi083.stdout: fsid = 039877dd-d2ce-45d2-8e77-ac5eb7e1d55a 2024-03-06T07:20:59.853 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:20:59.853 INFO:teuthology.orchestra.run.smithi083.stdout:[osd] 2024-03-06T07:20:59.853 INFO:teuthology.orchestra.run.smithi083.stdout: osd journal size = 100 2024-03-06T07:20:59.853 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:20:59.853 INFO:teuthology.orchestra.run.smithi083.stdout: osd scrub load threshold = 5.0 2024-03-06T07:20:59.853 INFO:teuthology.orchestra.run.smithi083.stdout: osd scrub max interval = 600 2024-03-06T07:20:59.853 INFO:teuthology.orchestra.run.smithi083.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:20:59.853 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:20:59.853 INFO:teuthology.orchestra.run.smithi083.stdout: osd recover clone overlap = true 2024-03-06T07:20:59.853 INFO:teuthology.orchestra.run.smithi083.stdout: osd recovery max chunk = 1048576 2024-03-06T07:20:59.853 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:20:59.853 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug shutdown = true 2024-03-06T07:20:59.854 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug op order = true 2024-03-06T07:20:59.854 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug verify stray on activate = true 2024-03-06T07:20:59.854 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug trim objects = true 2024-03-06T07:20:59.854 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:20:59.854 INFO:teuthology.orchestra.run.smithi083.stdout: osd open classes on start = true 2024-03-06T07:20:59.854 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug pg log writeout = true 2024-03-06T07:20:59.854 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:20:59.854 INFO:teuthology.orchestra.run.smithi083.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:20:59.854 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:20:59.854 INFO:teuthology.orchestra.run.smithi083.stdout: osd map max advance = 10 2024-03-06T07:20:59.854 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:20:59.854 INFO:teuthology.orchestra.run.smithi083.stdout: journal zero on create = true 2024-03-06T07:20:59.854 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:20:59.854 INFO:teuthology.orchestra.run.smithi083.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:20:59.855 INFO:teuthology.orchestra.run.smithi083.stdout: filestore apply finisher threads = 3 2024-03-06T07:20:59.855 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:20:59.855 INFO:teuthology.orchestra.run.smithi083.stdout: bdev debug aio = true 2024-03-06T07:20:59.855 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug misdirected ops = true 2024-03-06T07:20:59.855 INFO:teuthology.orchestra.run.smithi083.stdout: bdev async discard = True 2024-03-06T07:20:59.855 INFO:teuthology.orchestra.run.smithi083.stdout: bdev enable discard = True 2024-03-06T07:20:59.855 INFO:teuthology.orchestra.run.smithi083.stdout: bluestore allocator = bitmap 2024-03-06T07:20:59.855 INFO:teuthology.orchestra.run.smithi083.stdout: bluestore block size = 96636764160 2024-03-06T07:20:59.855 INFO:teuthology.orchestra.run.smithi083.stdout: bluestore fsck on mount = True 2024-03-06T07:20:59.855 INFO:teuthology.orchestra.run.smithi083.stdout: debug bluefs = 1/20 2024-03-06T07:20:59.855 INFO:teuthology.orchestra.run.smithi083.stdout: debug bluestore = 1/20 2024-03-06T07:20:59.855 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-06T07:20:59.855 INFO:teuthology.orchestra.run.smithi083.stdout: debug osd = 20 2024-03-06T07:20:59.855 INFO:teuthology.orchestra.run.smithi083.stdout: debug rocksdb = 4/10 2024-03-06T07:20:59.855 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:20:59.856 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd full ratio = 0.9 2024-03-06T07:20:59.856 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:20:59.856 INFO:teuthology.orchestra.run.smithi083.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:20:59.856 INFO:teuthology.orchestra.run.smithi083.stdout: osd objectstore = bluestore 2024-03-06T07:20:59.856 INFO:teuthology.orchestra.run.smithi083.stdout: osd op complaint time = 180 2024-03-06T07:20:59.856 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:20:59.856 INFO:teuthology.orchestra.run.smithi083.stdout:[mgr] 2024-03-06T07:20:59.856 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-06T07:20:59.856 INFO:teuthology.orchestra.run.smithi083.stdout: debug mgr = 20 2024-03-06T07:20:59.856 INFO:teuthology.orchestra.run.smithi083.stdout: debug mon = 20 2024-03-06T07:20:59.856 INFO:teuthology.orchestra.run.smithi083.stdout: debug auth = 20 2024-03-06T07:20:59.856 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:20:59.856 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:20:59.856 INFO:teuthology.orchestra.run.smithi083.stdout: mgr/telemetry/nag = false 2024-03-06T07:20:59.856 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:20:59.857 INFO:teuthology.orchestra.run.smithi083.stdout:[mon] 2024-03-06T07:20:59.857 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-06T07:20:59.857 INFO:teuthology.orchestra.run.smithi083.stdout: debug mon = 20 2024-03-06T07:20:59.857 INFO:teuthology.orchestra.run.smithi083.stdout: debug paxos = 20 2024-03-06T07:20:59.857 INFO:teuthology.orchestra.run.smithi083.stdout: debug auth = 20 2024-03-06T07:20:59.857 INFO:teuthology.orchestra.run.smithi083.stdout: mon data avail warn = 5 2024-03-06T07:20:59.857 INFO:teuthology.orchestra.run.smithi083.stdout: mon mgr mkfs grace = 240 2024-03-06T07:20:59.857 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:20:59.857 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd reporter subtree level = osd 2024-03-06T07:20:59.857 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd prime pg temp = true 2024-03-06T07:20:59.857 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:20:59.857 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:20:59.857 INFO:teuthology.orchestra.run.smithi083.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:20:59.857 INFO:teuthology.orchestra.run.smithi083.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:20:59.858 INFO:teuthology.orchestra.run.smithi083.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:20:59.858 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:20:59.858 INFO:teuthology.orchestra.run.smithi083.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:20:59.858 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:20:59.858 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:20:59.858 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:20:59.858 INFO:teuthology.orchestra.run.smithi083.stdout: # 1m isn't quite enough 2024-03-06T07:20:59.858 INFO:teuthology.orchestra.run.smithi083.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:20:59.858 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:20:59.858 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:20:59.858 INFO:teuthology.orchestra.run.smithi083.stdout: mon op complaint time = 120 2024-03-06T07:20:59.858 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:20:59.858 INFO:teuthology.orchestra.run.smithi083.stdout:[client] 2024-03-06T07:20:59.858 INFO:teuthology.orchestra.run.smithi083.stdout: rgw cache enabled = true 2024-03-06T07:20:59.859 INFO:teuthology.orchestra.run.smithi083.stdout: rgw enable ops log = true 2024-03-06T07:20:59.859 INFO:teuthology.orchestra.run.smithi083.stdout: rgw enable usage log = true 2024-03-06T07:20:59.859 INFO:teuthology.orchestra.run.smithi083.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:20:59.859 INFO:teuthology.orchestra.run.smithi083.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:20:59.859 INFO:teuthology.orchestra.run.smithi083.stdout: client mount timeout = 600 2024-03-06T07:20:59.859 INFO:teuthology.orchestra.run.smithi083.stdout: debug client = 20 2024-03-06T07:20:59.859 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-06T07:20:59.859 INFO:teuthology.orchestra.run.smithi083.stdout: pid file = "" 2024-03-06T07:20:59.859 INFO:teuthology.orchestra.run.smithi083.stdout: rados mon op timeout = 900 2024-03-06T07:20:59.859 INFO:teuthology.orchestra.run.smithi083.stdout: rados osd op timeout = 900 2024-03-06T07:20:59.859 INFO:teuthology.orchestra.run.smithi083.stdout:[mon.a] 2024-03-06T07:20:59.859 INFO:teuthology.orchestra.run.smithi083.stdout:[mon.b] 2024-03-06T07:20:59.859 INFO:teuthology.orchestra.run.smithi083.stdout:[mon.c] 2024-03-06T07:20:59.859 INFO:teuthology.orchestra.run.smithi083.stdout:[mds.a] 2024-03-06T07:20:59.860 INFO:teuthology.orchestra.run.smithi083.stdout:[mds.c] 2024-03-06T07:20:59.860 INFO:teuthology.orchestra.run.smithi083.stdout:[mds.b] 2024-03-06T07:20:59.860 INFO:teuthology.orchestra.run.smithi083.stdout:[mds.d] 2024-03-06T07:20:59.860 INFO:teuthology.orchestra.run.smithi083.stdout:[mds] 2024-03-06T07:20:59.860 INFO:teuthology.orchestra.run.smithi083.stdout: debug mds = 20 2024-03-06T07:20:59.860 INFO:teuthology.orchestra.run.smithi083.stdout: debug mds balancer = 20 2024-03-06T07:20:59.860 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-06T07:20:59.860 INFO:teuthology.orchestra.run.smithi083.stdout: mds debug frag = True 2024-03-06T07:20:59.860 INFO:teuthology.orchestra.run.smithi083.stdout: mds debug scatterstat = True 2024-03-06T07:20:59.860 INFO:teuthology.orchestra.run.smithi083.stdout: mds op complaint time = 180 2024-03-06T07:20:59.860 INFO:teuthology.orchestra.run.smithi083.stdout: mds verify scatter = True 2024-03-06T07:20:59.860 INFO:teuthology.orchestra.run.smithi083.stdout: osd op complaint time = 180 2024-03-06T07:20:59.860 INFO:teuthology.orchestra.run.smithi083.stdout: rados mon op timeout = 900 2024-03-06T07:20:59.861 INFO:teuthology.orchestra.run.smithi083.stdout: rados osd op timeout = 900 2024-03-06T07:20:59.861 INFO:teuthology.orchestra.run.smithi183.stdout:[global] 2024-03-06T07:20:59.861 INFO:teuthology.orchestra.run.smithi183.stdout: chdir = "" 2024-03-06T07:20:59.861 INFO:teuthology.orchestra.run.smithi183.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:20:59.861 INFO:teuthology.orchestra.run.smithi183.stdout: auth supported = cephx 2024-03-06T07:20:59.861 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:20:59.861 INFO:teuthology.orchestra.run.smithi183.stdout: filestore xattr use omap = true 2024-03-06T07:20:59.862 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:20:59.862 INFO:teuthology.orchestra.run.smithi183.stdout: mon clock drift allowed = 1.000 2024-03-06T07:20:59.862 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:20:59.862 INFO:teuthology.orchestra.run.smithi183.stdout: osd crush chooseleaf type = 0 2024-03-06T07:20:59.862 INFO:teuthology.orchestra.run.smithi183.stdout: auth debug = true 2024-03-06T07:20:59.862 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:20:59.862 INFO:teuthology.orchestra.run.smithi183.stdout: ms die on old message = true 2024-03-06T07:20:59.862 INFO:teuthology.orchestra.run.smithi183.stdout: ms die on bug = true 2024-03-06T07:20:59.862 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:20:59.862 INFO:teuthology.orchestra.run.smithi183.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:20:59.862 INFO:teuthology.orchestra.run.smithi183.stdout: mon pg warn max object skew = 0 2024-03-06T07:20:59.862 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:20:59.862 INFO:teuthology.orchestra.run.smithi183.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:20:59.863 INFO:teuthology.orchestra.run.smithi183.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:20:59.863 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:20:59.863 INFO:teuthology.orchestra.run.smithi183.stdout: osd pool default size = 2 2024-03-06T07:20:59.863 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:20:59.863 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd allow primary affinity = true 2024-03-06T07:20:59.863 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd allow pg remap = true 2024-03-06T07:20:59.863 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:20:59.863 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:20:59.863 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on no sortbitwise = false 2024-03-06T07:20:59.863 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:20:59.863 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on too few osds = false 2024-03-06T07:20:59.863 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:20:59.863 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:20:59.863 INFO:teuthology.orchestra.run.smithi183.stdout: mon_allow_pool_size_one = true 2024-03-06T07:20:59.863 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:20:59.864 INFO:teuthology.orchestra.run.smithi183.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:20:59.864 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:20:59.864 INFO:teuthology.orchestra.run.smithi183.stdout: osd default data pool replay window = 5 2024-03-06T07:20:59.864 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:20:59.864 INFO:teuthology.orchestra.run.smithi183.stdout: mon allow pool delete = true 2024-03-06T07:20:59.864 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:20:59.864 INFO:teuthology.orchestra.run.smithi183.stdout: mon cluster log file level = debug 2024-03-06T07:20:59.864 INFO:teuthology.orchestra.run.smithi183.stdout: debug asserts on shutdown = true 2024-03-06T07:20:59.864 INFO:teuthology.orchestra.run.smithi183.stdout: mon health detail to clog = false 2024-03-06T07:20:59.864 INFO:teuthology.orchestra.run.smithi183.stdout: mon host = "172.21.15.83,172.21.15.183,[v2:172.21.15.183:3301,v1:172.21.15.183:6790]" 2024-03-06T07:20:59.864 INFO:teuthology.orchestra.run.smithi183.stdout: fsid = 039877dd-d2ce-45d2-8e77-ac5eb7e1d55a 2024-03-06T07:20:59.864 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:20:59.864 INFO:teuthology.orchestra.run.smithi183.stdout:[osd] 2024-03-06T07:20:59.865 INFO:teuthology.orchestra.run.smithi183.stdout: osd journal size = 100 2024-03-06T07:20:59.865 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:20:59.865 INFO:teuthology.orchestra.run.smithi183.stdout: osd scrub load threshold = 5.0 2024-03-06T07:20:59.865 INFO:teuthology.orchestra.run.smithi183.stdout: osd scrub max interval = 600 2024-03-06T07:20:59.865 INFO:teuthology.orchestra.run.smithi183.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:20:59.865 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:20:59.865 INFO:teuthology.orchestra.run.smithi183.stdout: osd recover clone overlap = true 2024-03-06T07:20:59.865 INFO:teuthology.orchestra.run.smithi183.stdout: osd recovery max chunk = 1048576 2024-03-06T07:20:59.865 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:20:59.865 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug shutdown = true 2024-03-06T07:20:59.865 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug op order = true 2024-03-06T07:20:59.865 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug verify stray on activate = true 2024-03-06T07:20:59.865 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug trim objects = true 2024-03-06T07:20:59.865 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:20:59.866 INFO:teuthology.orchestra.run.smithi183.stdout: osd open classes on start = true 2024-03-06T07:20:59.866 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug pg log writeout = true 2024-03-06T07:20:59.866 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:20:59.866 INFO:teuthology.orchestra.run.smithi183.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:20:59.866 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:20:59.866 INFO:teuthology.orchestra.run.smithi183.stdout: osd map max advance = 10 2024-03-06T07:20:59.866 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:20:59.866 INFO:teuthology.orchestra.run.smithi183.stdout: journal zero on create = true 2024-03-06T07:20:59.866 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:20:59.866 INFO:teuthology.orchestra.run.smithi183.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:20:59.866 INFO:teuthology.orchestra.run.smithi183.stdout: filestore apply finisher threads = 3 2024-03-06T07:20:59.866 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:20:59.866 INFO:teuthology.orchestra.run.smithi183.stdout: bdev debug aio = true 2024-03-06T07:20:59.866 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug misdirected ops = true 2024-03-06T07:20:59.867 INFO:teuthology.orchestra.run.smithi183.stdout: bdev async discard = True 2024-03-06T07:20:59.867 INFO:teuthology.orchestra.run.smithi183.stdout: bdev enable discard = True 2024-03-06T07:20:59.867 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore allocator = bitmap 2024-03-06T07:20:59.867 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore block size = 96636764160 2024-03-06T07:20:59.867 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore fsck on mount = True 2024-03-06T07:20:59.867 INFO:teuthology.orchestra.run.smithi183.stdout: debug bluefs = 1/20 2024-03-06T07:20:59.867 INFO:teuthology.orchestra.run.smithi183.stdout: debug bluestore = 1/20 2024-03-06T07:20:59.867 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-06T07:20:59.867 INFO:teuthology.orchestra.run.smithi183.stdout: debug osd = 20 2024-03-06T07:20:59.867 INFO:teuthology.orchestra.run.smithi183.stdout: debug rocksdb = 4/10 2024-03-06T07:20:59.867 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:20:59.867 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd full ratio = 0.9 2024-03-06T07:20:59.867 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:20:59.867 INFO:teuthology.orchestra.run.smithi183.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:20:59.868 INFO:teuthology.orchestra.run.smithi183.stdout: osd objectstore = bluestore 2024-03-06T07:20:59.868 INFO:teuthology.orchestra.run.smithi183.stdout: osd op complaint time = 180 2024-03-06T07:20:59.868 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:20:59.868 INFO:teuthology.orchestra.run.smithi183.stdout:[mgr] 2024-03-06T07:20:59.868 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-06T07:20:59.868 INFO:teuthology.orchestra.run.smithi183.stdout: debug mgr = 20 2024-03-06T07:20:59.868 INFO:teuthology.orchestra.run.smithi183.stdout: debug mon = 20 2024-03-06T07:20:59.868 INFO:teuthology.orchestra.run.smithi183.stdout: debug auth = 20 2024-03-06T07:20:59.868 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:20:59.868 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:20:59.868 INFO:teuthology.orchestra.run.smithi183.stdout: mgr/telemetry/nag = false 2024-03-06T07:20:59.868 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:20:59.868 INFO:teuthology.orchestra.run.smithi183.stdout:[mon] 2024-03-06T07:20:59.869 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-06T07:20:59.869 INFO:teuthology.orchestra.run.smithi183.stdout: debug mon = 20 2024-03-06T07:20:59.869 INFO:teuthology.orchestra.run.smithi183.stdout: debug paxos = 20 2024-03-06T07:20:59.869 INFO:teuthology.orchestra.run.smithi183.stdout: debug auth = 20 2024-03-06T07:20:59.869 INFO:teuthology.orchestra.run.smithi183.stdout: mon data avail warn = 5 2024-03-06T07:20:59.869 INFO:teuthology.orchestra.run.smithi183.stdout: mon mgr mkfs grace = 240 2024-03-06T07:20:59.869 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:20:59.869 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd reporter subtree level = osd 2024-03-06T07:20:59.869 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd prime pg temp = true 2024-03-06T07:20:59.869 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:20:59.869 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:20:59.869 INFO:teuthology.orchestra.run.smithi183.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:20:59.869 INFO:teuthology.orchestra.run.smithi183.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:20:59.869 INFO:teuthology.orchestra.run.smithi183.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:20:59.869 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:20:59.870 INFO:teuthology.orchestra.run.smithi183.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:20:59.870 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:20:59.870 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:20:59.870 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:20:59.870 INFO:teuthology.orchestra.run.smithi183.stdout: # 1m isn't quite enough 2024-03-06T07:20:59.870 INFO:teuthology.orchestra.run.smithi183.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:20:59.870 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:20:59.870 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:20:59.870 INFO:teuthology.orchestra.run.smithi183.stdout: mon op complaint time = 120 2024-03-06T07:20:59.870 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:20:59.870 INFO:teuthology.orchestra.run.smithi183.stdout:[client] 2024-03-06T07:20:59.870 INFO:teuthology.orchestra.run.smithi183.stdout: rgw cache enabled = true 2024-03-06T07:20:59.870 INFO:teuthology.orchestra.run.smithi183.stdout: rgw enable ops log = true 2024-03-06T07:20:59.870 INFO:teuthology.orchestra.run.smithi183.stdout: rgw enable usage log = true 2024-03-06T07:20:59.870 INFO:teuthology.orchestra.run.smithi183.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:20:59.871 INFO:teuthology.orchestra.run.smithi183.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:20:59.871 INFO:teuthology.orchestra.run.smithi183.stdout: client mount timeout = 600 2024-03-06T07:20:59.871 INFO:teuthology.orchestra.run.smithi183.stdout: debug client = 20 2024-03-06T07:20:59.871 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-06T07:20:59.871 INFO:teuthology.orchestra.run.smithi183.stdout: pid file = "" 2024-03-06T07:20:59.871 INFO:teuthology.orchestra.run.smithi183.stdout: rados mon op timeout = 900 2024-03-06T07:20:59.871 INFO:teuthology.orchestra.run.smithi183.stdout: rados osd op timeout = 900 2024-03-06T07:20:59.871 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.a] 2024-03-06T07:20:59.871 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.b] 2024-03-06T07:20:59.871 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.c] 2024-03-06T07:20:59.871 INFO:teuthology.orchestra.run.smithi183.stdout:[mds.a] 2024-03-06T07:20:59.871 INFO:teuthology.orchestra.run.smithi183.stdout:[mds.c] 2024-03-06T07:20:59.871 INFO:teuthology.orchestra.run.smithi183.stdout:[mds.b] 2024-03-06T07:20:59.871 INFO:teuthology.orchestra.run.smithi183.stdout:[mds.d] 2024-03-06T07:20:59.872 INFO:teuthology.orchestra.run.smithi183.stdout:[mds] 2024-03-06T07:20:59.872 INFO:teuthology.orchestra.run.smithi183.stdout: debug mds = 20 2024-03-06T07:20:59.872 INFO:teuthology.orchestra.run.smithi183.stdout: debug mds balancer = 20 2024-03-06T07:20:59.872 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-06T07:20:59.872 INFO:teuthology.orchestra.run.smithi183.stdout: mds debug frag = True 2024-03-06T07:20:59.872 INFO:teuthology.orchestra.run.smithi183.stdout: mds debug scatterstat = True 2024-03-06T07:20:59.872 INFO:teuthology.orchestra.run.smithi183.stdout: mds op complaint time = 180 2024-03-06T07:20:59.872 INFO:teuthology.orchestra.run.smithi183.stdout: mds verify scatter = True 2024-03-06T07:20:59.872 INFO:teuthology.orchestra.run.smithi183.stdout: osd op complaint time = 180 2024-03-06T07:20:59.872 INFO:teuthology.orchestra.run.smithi183.stdout: rados mon op timeout = 900 2024-03-06T07:20:59.872 INFO:teuthology.orchestra.run.smithi183.stdout: rados osd op timeout = 900 2024-03-06T07:20:59.874 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-06T07:20:59.874 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-06T07:20:59.969 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-06T07:20:59.969 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:20:59.969 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-06T07:20:59.983 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:20:59.984 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-06T07:21:00.038 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi083.front.sepia.ceph.com 2024-03-06T07:21:00.038 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:21:00.038 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-06T07:21:00.038 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-06T07:21:00.116 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:21:00.116 DEBUG:teuthology.orchestra.run.smithi083:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:00.171 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi183.front.sepia.ceph.com 2024-03-06T07:21:00.171 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:21:00.171 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-06T07:21:00.171 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-06T07:21:00.211 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:21:00.211 DEBUG:teuthology.orchestra.run.smithi183:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:00.266 INFO:tasks.ceph:Setting up mon nodes... 2024-03-06T07:21:00.266 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-06T07:21:00.266 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-06T07:21:00.321 INFO:teuthology.orchestra.run.smithi083.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-06T07:21:00.323 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-06T07:21:00.389 INFO:teuthology.orchestra.run.smithi183.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-06T07:21:00.392 INFO:tasks.ceph:Setting up mds nodes... 2024-03-06T07:21:00.392 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-06T07:21:00.446 INFO:teuthology.orchestra.run.smithi083.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-06T07:21:00.449 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-06T07:21:00.475 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-03-06T07:21:00.570 INFO:teuthology.orchestra.run.smithi083.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-03-06T07:21:00.572 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-03-06T07:21:00.599 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-06T07:21:00.653 INFO:teuthology.orchestra.run.smithi183.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-06T07:21:00.656 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-06T07:21:00.682 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-03-06T07:21:00.777 INFO:teuthology.orchestra.run.smithi183.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-03-06T07:21:00.779 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-03-06T07:21:00.805 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-06T07:21:00.806 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-06T07:21:00.848 INFO:teuthology.orchestra.run.smithi083.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-06T07:21:00.863 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-06T07:21:00.945 INFO:teuthology.orchestra.run.smithi083.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-03-06T07:21:00.961 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-06T07:21:01.003 INFO:teuthology.orchestra.run.smithi183.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-06T07:21:01.018 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-03-06T07:21:01.100 INFO:teuthology.orchestra.run.smithi183.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-03-06T07:21:01.116 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-06T07:21:01.116 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi083.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi183.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-06T07:21:01.116 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-06T07:21:01.145 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:21:01.145 INFO:tasks.ceph:role: osd.0 2024-03-06T07:21:01.145 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi083.front.sepia.ceph.com 2024-03-06T07:21:01.145 DEBUG:teuthology.orchestra.run.smithi083:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-06T07:21:01.218 INFO:teuthology.orchestra.run.smithi083.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:21:01.219 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:21:01.219 INFO:teuthology.orchestra.run.smithi083.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:21:01.219 INFO:teuthology.orchestra.run.smithi083.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:21:01.219 INFO:teuthology.orchestra.run.smithi083.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:21:01.219 INFO:teuthology.orchestra.run.smithi083.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:21:01.219 INFO:teuthology.orchestra.run.smithi083.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:21:01.219 INFO:teuthology.orchestra.run.smithi083.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:21:01.219 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:21:01.220 INFO:teuthology.orchestra.run.smithi083.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:21:01.705 INFO:teuthology.orchestra.run.smithi083.stdout:Discarding blocks...Done. 2024-03-06T07:21:01.707 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi083.front.sepia.ceph.com -o noatime 2024-03-06T07:21:01.707 DEBUG:teuthology.orchestra.run.smithi083:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-06T07:21:01.818 DEBUG:teuthology.orchestra.run.smithi083:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-06T07:21:01.852 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-06T07:21:01.919 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:21:01.919 INFO:tasks.ceph:role: osd.1 2024-03-06T07:21:01.919 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi083.front.sepia.ceph.com 2024-03-06T07:21:01.920 DEBUG:teuthology.orchestra.run.smithi083:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-06T07:21:01.995 INFO:teuthology.orchestra.run.smithi083.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:21:01.995 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:21:01.995 INFO:teuthology.orchestra.run.smithi083.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:21:01.995 INFO:teuthology.orchestra.run.smithi083.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:21:01.995 INFO:teuthology.orchestra.run.smithi083.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:21:01.995 INFO:teuthology.orchestra.run.smithi083.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:21:01.995 INFO:teuthology.orchestra.run.smithi083.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:21:01.995 INFO:teuthology.orchestra.run.smithi083.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:21:01.995 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:21:01.996 INFO:teuthology.orchestra.run.smithi083.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:21:02.463 INFO:teuthology.orchestra.run.smithi083.stdout:Discarding blocks...Done. 2024-03-06T07:21:02.465 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi083.front.sepia.ceph.com -o noatime 2024-03-06T07:21:02.465 DEBUG:teuthology.orchestra.run.smithi083:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-06T07:21:02.549 DEBUG:teuthology.orchestra.run.smithi083:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-06T07:21:02.620 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-06T07:21:02.688 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:21:02.688 INFO:tasks.ceph:role: osd.2 2024-03-06T07:21:02.688 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi083.front.sepia.ceph.com 2024-03-06T07:21:02.688 DEBUG:teuthology.orchestra.run.smithi083:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-06T07:21:02.761 INFO:teuthology.orchestra.run.smithi083.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:21:02.762 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:21:02.762 INFO:teuthology.orchestra.run.smithi083.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:21:02.762 INFO:teuthology.orchestra.run.smithi083.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:21:02.762 INFO:teuthology.orchestra.run.smithi083.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:21:02.762 INFO:teuthology.orchestra.run.smithi083.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:21:02.762 INFO:teuthology.orchestra.run.smithi083.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:21:02.762 INFO:teuthology.orchestra.run.smithi083.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:21:02.762 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:21:02.762 INFO:teuthology.orchestra.run.smithi083.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:21:03.277 INFO:teuthology.orchestra.run.smithi083.stdout:Discarding blocks...Done. 2024-03-06T07:21:03.279 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi083.front.sepia.ceph.com -o noatime 2024-03-06T07:21:03.279 DEBUG:teuthology.orchestra.run.smithi083:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-06T07:21:03.405 DEBUG:teuthology.orchestra.run.smithi083:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-06T07:21:03.437 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-06T07:21:03.506 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:21:03.506 INFO:tasks.ceph:role: osd.3 2024-03-06T07:21:03.506 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi083.front.sepia.ceph.com 2024-03-06T07:21:03.506 DEBUG:teuthology.orchestra.run.smithi083:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-06T07:21:03.582 INFO:teuthology.orchestra.run.smithi083.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:21:03.582 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:21:03.582 INFO:teuthology.orchestra.run.smithi083.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:21:03.582 INFO:teuthology.orchestra.run.smithi083.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:21:03.582 INFO:teuthology.orchestra.run.smithi083.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:21:03.582 INFO:teuthology.orchestra.run.smithi083.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:21:03.582 INFO:teuthology.orchestra.run.smithi083.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:21:03.582 INFO:teuthology.orchestra.run.smithi083.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:21:03.583 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:21:03.583 INFO:teuthology.orchestra.run.smithi083.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:21:04.096 INFO:teuthology.orchestra.run.smithi083.stdout:Discarding blocks...Done. 2024-03-06T07:21:04.098 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi083.front.sepia.ceph.com -o noatime 2024-03-06T07:21:04.098 DEBUG:teuthology.orchestra.run.smithi083:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-06T07:21:04.199 DEBUG:teuthology.orchestra.run.smithi083:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-06T07:21:04.233 DEBUG:teuthology.orchestra.run.smithi083:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:04.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:04.329+0000 7f2af34aa640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-06T07:21:04.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:04.330+0000 7f2af34aa640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-06T07:21:04.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:04.330+0000 7f2af34aa640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-06T07:21:04.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:04.330+0000 7f2af34aa640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-06T07:21:04.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:04.330+0000 7f2af34aa640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-06T07:21:04.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:04.330+0000 7f2af34aa640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-06T07:21:07.186 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-06T07:21:07.217 DEBUG:teuthology.orchestra.run.smithi083:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:07.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:07.303+0000 7fd6484f9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-06T07:21:07.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:07.304+0000 7fd6484f9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-06T07:21:07.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:07.304+0000 7fd6484f9640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-06T07:21:07.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:07.304+0000 7fd6484f9640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-06T07:21:07.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:07.304+0000 7fd6484f9640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-06T07:21:07.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:07.304+0000 7fd6484f9640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-06T07:21:10.154 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-06T07:21:10.184 DEBUG:teuthology.orchestra.run.smithi083:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:10.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:10.273+0000 7f47cbe1b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-06T07:21:10.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:10.273+0000 7f47cbe1b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-06T07:21:10.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:10.273+0000 7f47cbe1b640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-06T07:21:10.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:10.273+0000 7f47cbe1b640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-06T07:21:10.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:10.273+0000 7f47cbe1b640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-06T07:21:10.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:10.273+0000 7f47cbe1b640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-06T07:21:13.117 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-06T07:21:13.148 DEBUG:teuthology.orchestra.run.smithi083:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:13.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:13.236+0000 7f71f2620640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-06T07:21:13.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:13.237+0000 7f71f2620640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-06T07:21:13.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:13.237+0000 7f71f2620640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-06T07:21:13.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:13.237+0000 7f71f2620640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-06T07:21:13.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:13.237+0000 7f71f2620640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-06T07:21:13.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:13.237+0000 7f71f2620640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-06T07:21:16.091 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-06T07:21:16.122 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-06T07:21:16.156 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:21:16.156 INFO:tasks.ceph:role: osd.4 2024-03-06T07:21:16.156 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-06T07:21:16.156 DEBUG:teuthology.orchestra.run.smithi183:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-06T07:21:16.233 INFO:teuthology.orchestra.run.smithi183.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:21:16.233 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:21:16.233 INFO:teuthology.orchestra.run.smithi183.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:21:16.233 INFO:teuthology.orchestra.run.smithi183.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:21:16.233 INFO:teuthology.orchestra.run.smithi183.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:21:16.233 INFO:teuthology.orchestra.run.smithi183.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:21:16.233 INFO:teuthology.orchestra.run.smithi183.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:21:16.233 INFO:teuthology.orchestra.run.smithi183.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:21:16.233 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:21:16.234 INFO:teuthology.orchestra.run.smithi183.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:21:16.737 INFO:teuthology.orchestra.run.smithi183.stdout:Discarding blocks...Done. 2024-03-06T07:21:16.739 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi183.front.sepia.ceph.com -o noatime 2024-03-06T07:21:16.739 DEBUG:teuthology.orchestra.run.smithi183:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-06T07:21:16.841 DEBUG:teuthology.orchestra.run.smithi183:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-06T07:21:16.875 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-06T07:21:16.943 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:21:16.943 INFO:tasks.ceph:role: osd.5 2024-03-06T07:21:16.943 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-06T07:21:16.943 DEBUG:teuthology.orchestra.run.smithi183:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-06T07:21:17.018 INFO:teuthology.orchestra.run.smithi183.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:21:17.019 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:21:17.019 INFO:teuthology.orchestra.run.smithi183.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:21:17.019 INFO:teuthology.orchestra.run.smithi183.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:21:17.019 INFO:teuthology.orchestra.run.smithi183.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:21:17.019 INFO:teuthology.orchestra.run.smithi183.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:21:17.019 INFO:teuthology.orchestra.run.smithi183.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:21:17.019 INFO:teuthology.orchestra.run.smithi183.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:21:17.019 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:21:17.019 INFO:teuthology.orchestra.run.smithi183.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:21:17.483 INFO:teuthology.orchestra.run.smithi183.stdout:Discarding blocks...Done. 2024-03-06T07:21:17.486 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi183.front.sepia.ceph.com -o noatime 2024-03-06T07:21:17.486 DEBUG:teuthology.orchestra.run.smithi183:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-06T07:21:17.577 DEBUG:teuthology.orchestra.run.smithi183:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-06T07:21:17.608 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-06T07:21:17.677 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:21:17.677 INFO:tasks.ceph:role: osd.6 2024-03-06T07:21:17.677 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-06T07:21:17.677 DEBUG:teuthology.orchestra.run.smithi183:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-06T07:21:17.751 INFO:teuthology.orchestra.run.smithi183.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:21:17.751 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:21:17.751 INFO:teuthology.orchestra.run.smithi183.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:21:17.751 INFO:teuthology.orchestra.run.smithi183.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:21:17.751 INFO:teuthology.orchestra.run.smithi183.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:21:17.751 INFO:teuthology.orchestra.run.smithi183.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:21:17.752 INFO:teuthology.orchestra.run.smithi183.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:21:17.752 INFO:teuthology.orchestra.run.smithi183.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:21:17.752 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:21:17.752 INFO:teuthology.orchestra.run.smithi183.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:21:18.237 INFO:teuthology.orchestra.run.smithi183.stdout:Discarding blocks...Done. 2024-03-06T07:21:18.239 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi183.front.sepia.ceph.com -o noatime 2024-03-06T07:21:18.239 DEBUG:teuthology.orchestra.run.smithi183:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-06T07:21:18.400 DEBUG:teuthology.orchestra.run.smithi183:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-06T07:21:18.432 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-06T07:21:18.500 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:21:18.500 INFO:tasks.ceph:role: osd.7 2024-03-06T07:21:18.500 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-06T07:21:18.500 DEBUG:teuthology.orchestra.run.smithi183:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-06T07:21:18.575 INFO:teuthology.orchestra.run.smithi183.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:21:18.576 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:21:18.576 INFO:teuthology.orchestra.run.smithi183.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:21:18.576 INFO:teuthology.orchestra.run.smithi183.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:21:18.576 INFO:teuthology.orchestra.run.smithi183.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:21:18.576 INFO:teuthology.orchestra.run.smithi183.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:21:18.576 INFO:teuthology.orchestra.run.smithi183.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:21:18.576 INFO:teuthology.orchestra.run.smithi183.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:21:18.576 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:21:18.576 INFO:teuthology.orchestra.run.smithi183.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:21:19.087 INFO:teuthology.orchestra.run.smithi183.stdout:Discarding blocks...Done. 2024-03-06T07:21:19.089 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi183.front.sepia.ceph.com -o noatime 2024-03-06T07:21:19.089 DEBUG:teuthology.orchestra.run.smithi183:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-06T07:21:19.217 DEBUG:teuthology.orchestra.run.smithi183:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-06T07:21:19.253 DEBUG:teuthology.orchestra.run.smithi183:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:19.350 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:19.348+0000 7fc10de10640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-06T07:21:19.350 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:19.349+0000 7fc10de10640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-06T07:21:19.351 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:19.349+0000 7fc10de10640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-06T07:21:19.351 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:19.349+0000 7fc10de10640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-06T07:21:19.351 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:19.349+0000 7fc10de10640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-06T07:21:19.351 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:19.349+0000 7fc10de10640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-06T07:21:22.196 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-06T07:21:22.227 DEBUG:teuthology.orchestra.run.smithi183:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:22.321 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:22.319+0000 7fd8a8615640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-06T07:21:22.321 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:22.319+0000 7fd8a8615640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-06T07:21:22.321 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:22.319+0000 7fd8a8615640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-06T07:21:22.321 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:22.319+0000 7fd8a8615640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-06T07:21:22.321 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:22.319+0000 7fd8a8615640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-06T07:21:22.322 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:22.320+0000 7fd8a8615640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-06T07:21:25.177 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-06T07:21:25.208 DEBUG:teuthology.orchestra.run.smithi183:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:25.300 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:25.299+0000 7fb3849fd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-06T07:21:25.301 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:25.299+0000 7fb3849fd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-06T07:21:25.301 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:25.299+0000 7fb3849fd640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-06T07:21:25.301 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:25.299+0000 7fb3849fd640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-06T07:21:25.301 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:25.299+0000 7fb3849fd640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-06T07:21:25.301 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:25.300+0000 7fb3849fd640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-06T07:21:27.895 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:21:27.898 DEBUG:teuthology.orchestra.run.smithi183:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:21:28.158 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-06T07:21:28.187 DEBUG:teuthology.orchestra.run.smithi183:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:28.276 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:28.274+0000 7fdd767bb640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-06T07:21:28.276 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:28.274+0000 7fdd767bb640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-06T07:21:28.276 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:28.274+0000 7fdd767bb640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-06T07:21:28.276 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:28.274+0000 7fdd767bb640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-06T07:21:28.276 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:28.274+0000 7fdd767bb640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-06T07:21:28.276 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:28.275+0000 7fdd767bb640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-06T07:21:31.139 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-06T07:21:31.171 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-06T07:21:31.172 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:21:31.172 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-06T07:21:31.204 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:21:31.204 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-06T07:21:31.274 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:21:31.274 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-03-06T07:21:31.343 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:21:31.344 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-06T07:21:31.412 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:21:31.412 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-06T07:21:31.481 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:21:31.481 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-06T07:21:31.548 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:21:31.548 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-06T07:21:31.614 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:21:31.614 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-06T07:21:31.647 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:21:31.647 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-06T07:21:31.718 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:21:31.718 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-03-06T07:21:31.788 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:21:31.788 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-06T07:21:31.857 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:21:31.857 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-06T07:21:31.926 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:21:31.926 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-06T07:21:31.995 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:21:31.995 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-06T07:21:32.063 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:21:32.063 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-06T07:21:32.079 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:21:32.079 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-03-06T07:21:32.134 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:21:32.134 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-06T07:21:32.150 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:21:32.150 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-03-06T07:21:32.205 INFO:tasks.ceph:Adding keys to all mons... 2024-03-06T07:21:32.205 DEBUG:teuthology.orchestra.run.smithi083:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-06T07:21:32.208 DEBUG:teuthology.orchestra.run.smithi183:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-06T07:21:32.248 INFO:teuthology.orchestra.run.smithi083.stdout:[mgr.y] 2024-03-06T07:21:32.248 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBcGehlvdcdExAAySXlEfJqLnpMySiiYGFkMQ== 2024-03-06T07:21:32.248 INFO:teuthology.orchestra.run.smithi083.stdout:[mds.a] 2024-03-06T07:21:32.248 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBcGehlVHaUGhAAVRpLsHYTUkaYDmceq5Rlgw== 2024-03-06T07:21:32.248 INFO:teuthology.orchestra.run.smithi083.stdout:[mds.c] 2024-03-06T07:21:32.249 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBcGehlX0jxIRAAxYNi39qxIFXazlL2qNzLCg== 2024-03-06T07:21:32.249 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.0] 2024-03-06T07:21:32.249 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBgGehlkezBExAAZOLMH2ua3CTWAL7H6LGeZw== 2024-03-06T07:21:32.249 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.1] 2024-03-06T07:21:32.249 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBjGehlC04qEhAAqirTFR6y3NOGw6N+iz3/pg== 2024-03-06T07:21:32.249 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.2] 2024-03-06T07:21:32.249 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBmGehlfuFWEBAAj/vjZXrv1wVNSJ1R74UmnQ== 2024-03-06T07:21:32.249 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.3] 2024-03-06T07:21:32.249 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBpGehl5+ItDhAA0YNc83MhKcgb3JgRuP87aw== 2024-03-06T07:21:32.249 INFO:teuthology.orchestra.run.smithi083.stdout:[mgr.x] 2024-03-06T07:21:32.249 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBcGehlfikrFxAAoZkTUtpHlBcUAMpwWABRxg== 2024-03-06T07:21:32.249 INFO:teuthology.orchestra.run.smithi083.stdout:[mds.b] 2024-03-06T07:21:32.249 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBcGehlFlrrJhAAQbY6xU910eYRg1zwgpj9vg== 2024-03-06T07:21:32.249 INFO:teuthology.orchestra.run.smithi083.stdout:[mds.d] 2024-03-06T07:21:32.250 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBcGehl3utCLhAAzEjASLnYpPYrJsWOt3WwpQ== 2024-03-06T07:21:32.250 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.4] 2024-03-06T07:21:32.250 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBvGehlYa/TFBAAvGcizpRQ21pvgpEGgSET0w== 2024-03-06T07:21:32.250 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.5] 2024-03-06T07:21:32.250 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQByGehlE80TExAAFcWDgnLtW8OKlyu4B0jnmg== 2024-03-06T07:21:32.250 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.6] 2024-03-06T07:21:32.250 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQB1Gehla/LbERAAYfAjTKL11VMVnGUGV5GVcg== 2024-03-06T07:21:32.250 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.7] 2024-03-06T07:21:32.250 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQB4GehlpSJkEBAA17tUgMf94jfqvXlD51P8DQ== 2024-03-06T07:21:32.250 INFO:teuthology.orchestra.run.smithi083.stdout:[client.0] 2024-03-06T07:21:32.250 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBcGehlRwCIMhAA1DdbdQZPplu7LQOTK3BmSg== 2024-03-06T07:21:32.250 INFO:teuthology.orchestra.run.smithi083.stdout:[client.2] 2024-03-06T07:21:32.250 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBcGehltY5SOBAAI8rcIEVqMZaWQGgNaYKJTA== 2024-03-06T07:21:32.250 INFO:teuthology.orchestra.run.smithi083.stdout:[client.1] 2024-03-06T07:21:32.251 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBdGehl77spABAAsiEf+tXos3MwFrXToEQ8dQ== 2024-03-06T07:21:32.251 INFO:teuthology.orchestra.run.smithi083.stdout:[client.3] 2024-03-06T07:21:32.251 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBdGehl9ZX1BRAAA7fKnRyaQ+zqHoSGs4UvDw== 2024-03-06T07:21:32.270 INFO:teuthology.orchestra.run.smithi183.stdout:[mgr.y] 2024-03-06T07:21:32.270 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBcGehlvdcdExAAySXlEfJqLnpMySiiYGFkMQ== 2024-03-06T07:21:32.270 INFO:teuthology.orchestra.run.smithi183.stdout:[mds.a] 2024-03-06T07:21:32.271 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBcGehlVHaUGhAAVRpLsHYTUkaYDmceq5Rlgw== 2024-03-06T07:21:32.271 INFO:teuthology.orchestra.run.smithi183.stdout:[mds.c] 2024-03-06T07:21:32.271 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBcGehlX0jxIRAAxYNi39qxIFXazlL2qNzLCg== 2024-03-06T07:21:32.271 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.0] 2024-03-06T07:21:32.271 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBgGehlkezBExAAZOLMH2ua3CTWAL7H6LGeZw== 2024-03-06T07:21:32.271 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.1] 2024-03-06T07:21:32.271 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBjGehlC04qEhAAqirTFR6y3NOGw6N+iz3/pg== 2024-03-06T07:21:32.271 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.2] 2024-03-06T07:21:32.271 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBmGehlfuFWEBAAj/vjZXrv1wVNSJ1R74UmnQ== 2024-03-06T07:21:32.271 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.3] 2024-03-06T07:21:32.271 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBpGehl5+ItDhAA0YNc83MhKcgb3JgRuP87aw== 2024-03-06T07:21:32.271 INFO:teuthology.orchestra.run.smithi183.stdout:[mgr.x] 2024-03-06T07:21:32.271 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBcGehlfikrFxAAoZkTUtpHlBcUAMpwWABRxg== 2024-03-06T07:21:32.271 INFO:teuthology.orchestra.run.smithi183.stdout:[mds.b] 2024-03-06T07:21:32.271 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBcGehlFlrrJhAAQbY6xU910eYRg1zwgpj9vg== 2024-03-06T07:21:32.272 INFO:teuthology.orchestra.run.smithi183.stdout:[mds.d] 2024-03-06T07:21:32.272 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBcGehl3utCLhAAzEjASLnYpPYrJsWOt3WwpQ== 2024-03-06T07:21:32.272 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.4] 2024-03-06T07:21:32.272 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBvGehlYa/TFBAAvGcizpRQ21pvgpEGgSET0w== 2024-03-06T07:21:32.272 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.5] 2024-03-06T07:21:32.272 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQByGehlE80TExAAFcWDgnLtW8OKlyu4B0jnmg== 2024-03-06T07:21:32.272 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.6] 2024-03-06T07:21:32.272 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQB1Gehla/LbERAAYfAjTKL11VMVnGUGV5GVcg== 2024-03-06T07:21:32.272 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.7] 2024-03-06T07:21:32.272 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQB4GehlpSJkEBAA17tUgMf94jfqvXlD51P8DQ== 2024-03-06T07:21:32.272 INFO:teuthology.orchestra.run.smithi183.stdout:[client.0] 2024-03-06T07:21:32.272 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBcGehlRwCIMhAA1DdbdQZPplu7LQOTK3BmSg== 2024-03-06T07:21:32.272 INFO:teuthology.orchestra.run.smithi183.stdout:[client.2] 2024-03-06T07:21:32.273 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBcGehltY5SOBAAI8rcIEVqMZaWQGgNaYKJTA== 2024-03-06T07:21:32.273 INFO:teuthology.orchestra.run.smithi183.stdout:[client.1] 2024-03-06T07:21:32.273 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBdGehl77spABAAsiEf+tXos3MwFrXToEQ8dQ== 2024-03-06T07:21:32.273 INFO:teuthology.orchestra.run.smithi183.stdout:[client.3] 2024-03-06T07:21:32.273 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBdGehl9ZX1BRAAA7fKnRyaQ+zqHoSGs4UvDw== 2024-03-06T07:21:32.274 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:21:32.295 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:21:32.367 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:21:32.369 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:21:32.423 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:21:32.425 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:21:32.480 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:32.482 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:32.534 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:32.536 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:32.588 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:32.590 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:32.642 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:32.644 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:32.696 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:21:32.699 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:21:32.751 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:21:32.755 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:21:32.809 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:21:32.812 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:21:32.866 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:32.870 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:32.922 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:32.924 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:32.976 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:32.978 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:33.029 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:33.032 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:33.085 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:21:33.088 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:21:33.142 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:21:33.145 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:21:33.199 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:21:33.202 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:21:33.257 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:21:33.260 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:21:33.315 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-06T07:21:33.315 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-06T07:21:33.343 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-06T07:21:33.439 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-06T07:21:33.466 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-06T07:21:33.494 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-06T07:21:33.591 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-06T07:21:33.618 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-06T07:21:33.685 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-06T07:21:33.782 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-06T07:21:33.809 DEBUG:teuthology.orchestra.run.smithi083:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:33.811 DEBUG:teuthology.orchestra.run.smithi183:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:33.864 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-06T07:21:33.865 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-06T07:21:33.865 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-06T07:21:33.867 INFO:tasks.ceph.mon.a:Started 2024-03-06T07:21:33.867 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-06T07:21:33.867 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-06T07:21:33.906 INFO:tasks.ceph.mon.b:Started 2024-03-06T07:21:33.907 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-06T07:21:33.907 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-06T07:21:33.909 INFO:tasks.ceph.mon.c:Started 2024-03-06T07:21:33.909 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-06T07:21:33.909 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-06T07:21:33.909 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-06T07:21:33.911 INFO:tasks.ceph.mgr.y:Started 2024-03-06T07:21:33.911 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-06T07:21:33.911 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-06T07:21:33.913 INFO:tasks.ceph.mgr.x:Started 2024-03-06T07:21:33.914 INFO:tasks.ceph:Setting crush tunables to default 2024-03-06T07:21:33.914 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster ceph osd crush tunables default 2024-03-06T07:21:34.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.014+0000 7fed26e88640 1 Processor -- start 2024-03-06T07:21:34.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.015+0000 7fed26e88640 1 -- start start 2024-03-06T07:21:34.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.015+0000 7fed26e88640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fed2015de80 0x7fed20161b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:34.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.015+0000 7fed26e88640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fed2015e260 0x7fed200d2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:34.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.015+0000 7fed26e88640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed20162ab0 con 0x7fed200cc780 2024-03-06T07:21:34.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.015+0000 7fed26e88640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed20162210 con 0x7fed2015e260 2024-03-06T07:21:34.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.015+0000 7fed26e88640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed201624f0 con 0x7fed2015de80 2024-03-06T07:21:34.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.015+0000 7fed1ffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fed2015e260 0x7fed200d2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:34.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.016+0000 7fed1ffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fed2015e260 0x7fed200d2010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:42014/0 (socket says 172.21.15.83:42014) 2024-03-06T07:21:34.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.016+0000 7fed1ffff640 1 -- 172.21.15.83:0/3092658757 learned_addr learned my addr 172.21.15.83:0/3092658757 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:21:34.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.016+0000 7fed24c24640 1 --2- 172.21.15.83:0/3092658757 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fed2015de80 0x7fed20161b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:34.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.016+0000 7fed1ffff640 1 -- 172.21.15.83:0/3092658757 >> v2:172.21.15.83:3300/0 conn(0x7fed2015e260 msgr2=0x7fed200d2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-06T07:21:34.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.016+0000 7fed1ffff640 1 -- 172.21.15.83:0/3092658757 >> v2:172.21.15.83:3300/0 conn(0x7fed2015e260 msgr2=0x7fed200d2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:21:34.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.016+0000 7fed1ffff640 1 --2- 172.21.15.83:0/3092658757 >> v2:172.21.15.83:3300/0 conn(0x7fed2015e260 0x7fed200d2010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:21:34.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.016+0000 7fed1ffff640 1 --2- 172.21.15.83:0/3092658757 >> v2:172.21.15.83:3300/0 conn(0x7fed2015e260 0x7fed200d2010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:21:34.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.016+0000 7fed24c24640 1 -- 172.21.15.83:0/3092658757 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fed2015de80 msgr2=0x7fed20161b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-06T07:21:34.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.016+0000 7fed24c24640 1 -- 172.21.15.83:0/3092658757 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fed2015de80 msgr2=0x7fed20161b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:21:34.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.016+0000 7fed24c24640 1 --2- 172.21.15.83:0/3092658757 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fed2015de80 0x7fed20161b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:21:34.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.016+0000 7fed24c24640 1 --2- 172.21.15.83:0/3092658757 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fed2015de80 0x7fed20161b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:21:34.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.196+0000 7fed1f7fe640 1 -- 172.21.15.83:0/3092658757 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1093656665 0 0) 0x7fed20162ab0 con 0x7fed200cc780 2024-03-06T07:21:34.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.196+0000 7fed1f7fe640 1 -- 172.21.15.83:0/3092658757 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed10003520 con 0x7fed200cc780 2024-03-06T07:21:34.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.196+0000 7fed1f7fe640 1 -- 172.21.15.83:0/3092658757 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4087249113 0 0) 0x7fed10003520 con 0x7fed200cc780 2024-03-06T07:21:34.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.196+0000 7fed1f7fe640 1 -- 172.21.15.83:0/3092658757 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fed2015de80 msgr2=0x7fed20161b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:21:34.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.196+0000 7fed1f7fe640 1 --2- 172.21.15.83:0/3092658757 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fed2015de80 0x7fed20161b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:34.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.196+0000 7fed1f7fe640 1 -- 172.21.15.83:0/3092658757 >> v2:172.21.15.83:3300/0 conn(0x7fed2015e260 msgr2=0x7fed200d2010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:21:34.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.196+0000 7fed1f7fe640 1 --2- 172.21.15.83:0/3092658757 >> v2:172.21.15.83:3300/0 conn(0x7fed2015e260 0x7fed200d2010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:34.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.196+0000 7fed1f7fe640 1 -- 172.21.15.83:0/3092658757 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed201627d0 con 0x7fed200cc780 2024-03-06T07:21:34.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.196+0000 7fed1f7fe640 1 -- 172.21.15.83:0/3092658757 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7fed18001a10 con 0x7fed200cc780 2024-03-06T07:21:34.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.196+0000 7fed1f7fe640 1 -- 172.21.15.83:0/3092658757 >> v1:172.21.15.183:6789/0 conn(0x7fed200cc780 legacy=0x7fed200d01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:34.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.196+0000 7fed1f7fe640 1 --2- 172.21.15.83:0/3092658757 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fed2015de80 0x7fed100039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:34.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.196+0000 7fed1f7fe640 1 --2- 172.21.15.83:0/3092658757 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fed10004030 0x7fed10004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:34.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.196+0000 7fed1f7fe640 1 --2- 172.21.15.83:0/3092658757 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed10004d40 0x7fed100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:34.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.196+0000 7fed1f7fe640 1 -- 172.21.15.83:0/3092658757 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed20162210 con 0x7fed10004d40 2024-03-06T07:21:34.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.196+0000 7fed1f7fe640 1 -- 172.21.15.83:0/3092658757 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed201624f0 con 0x7fed10004030 2024-03-06T07:21:34.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.196+0000 7fed1f7fe640 1 -- 172.21.15.83:0/3092658757 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed1000bbd0 con 0x7fed2015de80 2024-03-06T07:21:34.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.196+0000 7fed25425640 1 --2- 172.21.15.83:0/3092658757 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed10004d40 0x7fed100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:34.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.196+0000 7fed24c24640 1 --2- 172.21.15.83:0/3092658757 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fed2015de80 0x7fed100039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:34.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.196+0000 7fed1ffff640 1 --2- 172.21.15.83:0/3092658757 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fed10004030 0x7fed10004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:34.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.196+0000 7fed25425640 1 -- 172.21.15.83:0/3092658757 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fed2015de80 msgr2=0x7fed100039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:34.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.196+0000 7fed25425640 1 --2- 172.21.15.83:0/3092658757 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fed2015de80 0x7fed100039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:34.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.196+0000 7fed25425640 1 -- 172.21.15.83:0/3092658757 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fed10004030 msgr2=0x7fed10004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:34.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.196+0000 7fed25425640 1 --2- 172.21.15.83:0/3092658757 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fed10004030 0x7fed10004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:34.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.196+0000 7fed25425640 1 -- 172.21.15.83:0/3092658757 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed10003520 con 0x7fed10004d40 2024-03-06T07:21:34.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.196+0000 7fed1ffff640 1 --2- 172.21.15.83:0/3092658757 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fed10004030 0x7fed10004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:34.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.197+0000 7fed25425640 1 --2- 172.21.15.83:0/3092658757 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed10004d40 0x7fed100051c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fed1800c490 tx=0x7fed10005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cbc475226ec347a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:34.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.197+0000 7fed1f7fe640 1 -- 172.21.15.83:0/3092658757 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed18039070 con 0x7fed10004d40 2024-03-06T07:21:34.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.197+0000 7fed1f7fe640 1 -- 172.21.15.83:0/3092658757 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed1802ccd0 con 0x7fed10004d40 2024-03-06T07:21:34.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.197+0000 7fed1f7fe640 1 -- 172.21.15.83:0/3092658757 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1800c0e0 con 0x7fed10004d40 2024-03-06T07:21:34.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.198+0000 7fed26e88640 1 -- 172.21.15.83:0/3092658757 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed10004d40 msgr2=0x7fed100051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:34.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.198+0000 7fed26e88640 1 --2- 172.21.15.83:0/3092658757 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed10004d40 0x7fed100051c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fed1800c490 tx=0x7fed10005830 comp rx=0 tx=0).stop 2024-03-06T07:21:34.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.198+0000 7fed25425640 1 -- 172.21.15.83:0/3092658757 reap_dead start 2024-03-06T07:21:34.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.198+0000 7fed26e88640 1 -- 172.21.15.83:0/3092658757 shutdown_connections 2024-03-06T07:21:34.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.198+0000 7fed26e88640 1 -- 172.21.15.83:0/3092658757 >> 172.21.15.83:0/3092658757 conn(0x7fed2009a9f0 msgr2=0x7fed2009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:34.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.198+0000 7fed26e88640 1 -- 172.21.15.83:0/3092658757 shutdown_connections 2024-03-06T07:21:34.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.198+0000 7fed26e88640 1 -- 172.21.15.83:0/3092658757 wait complete. 2024-03-06T07:21:34.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.199+0000 7fed26e88640 1 Processor -- start 2024-03-06T07:21:34.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.200+0000 7fed26e88640 1 -- start start 2024-03-06T07:21:34.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.200+0000 7fed26e88640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed10004d40 0x7fed201542a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:34.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.200+0000 7fed26e88640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fed200c9530 0x7fed200ccd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:34.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.201+0000 7fed26e88640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fed200cd2b0 0x7fed201f7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:34.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.201+0000 7fed26e88640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed200d08e0 con 0x7fed10004d40 2024-03-06T07:21:34.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.201+0000 7fed26e88640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed200ccbf0 con 0x7fed200c9530 2024-03-06T07:21:34.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.201+0000 7fed26e88640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed200d0a60 con 0x7fed200cd2b0 2024-03-06T07:21:34.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.201+0000 7fed24c24640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed10004d40 0x7fed201542a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:34.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.201+0000 7fed24c24640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed10004d40 0x7fed201542a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:42038/0 (socket says 172.21.15.83:42038) 2024-03-06T07:21:34.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.201+0000 7fed24c24640 1 -- 172.21.15.83:0/1025280412 learned_addr learned my addr 172.21.15.83:0/1025280412 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:21:34.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.201+0000 7fed1ffff640 1 --2- 172.21.15.83:0/1025280412 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fed200c9530 0x7fed200ccd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:34.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.201+0000 7fed25425640 1 --2- 172.21.15.83:0/1025280412 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fed200cd2b0 0x7fed201f7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:34.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.201+0000 7fed1ffff640 1 -- 172.21.15.83:0/1025280412 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fed200cd2b0 msgr2=0x7fed201f7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:34.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.201+0000 7fed1ffff640 1 --2- 172.21.15.83:0/1025280412 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fed200cd2b0 0x7fed201f7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:34.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.201+0000 7fed1ffff640 1 -- 172.21.15.83:0/1025280412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed10004d40 msgr2=0x7fed201542a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:34.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.201+0000 7fed1ffff640 1 --2- 172.21.15.83:0/1025280412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed10004d40 0x7fed201542a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:34.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.201+0000 7fed1ffff640 1 -- 172.21.15.83:0/1025280412 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed2015e680 con 0x7fed200c9530 2024-03-06T07:21:34.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.201+0000 7fed24c24640 1 --2- 172.21.15.83:0/1025280412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed10004d40 0x7fed201542a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:34.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.201+0000 7fed25425640 1 --2- 172.21.15.83:0/1025280412 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fed200cd2b0 0x7fed201f7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:34.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.201+0000 7fed1ffff640 1 --2- 172.21.15.83:0/1025280412 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fed200c9530 0x7fed200ccd70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fed0c00f540 tx=0x7fed0c00fa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:34.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.203+0000 7fed1d7fa640 1 -- 172.21.15.83:0/1025280412 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0c019030 con 0x7fed200c9530 2024-03-06T07:21:34.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.204+0000 7fed26e88640 1 -- 172.21.15.83:0/1025280412 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed200c8de0 con 0x7fed200c9530 2024-03-06T07:21:34.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.204+0000 7fed1d7fa640 1 -- 172.21.15.83:0/1025280412 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed0c004020 con 0x7fed200c9530 2024-03-06T07:21:34.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.204+0000 7fed1d7fa640 1 -- 172.21.15.83:0/1025280412 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0c0043f0 con 0x7fed200c9530 2024-03-06T07:21:34.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.204+0000 7fed26e88640 1 -- 172.21.15.83:0/1025280412 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed2015e940 con 0x7fed200c9530 2024-03-06T07:21:34.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.204+0000 7fed1d7fa640 1 -- 172.21.15.83:0/1025280412 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fed0c01c070 con 0x7fed200c9530 2024-03-06T07:21:34.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.204+0000 7fed1d7fa640 1 -- 172.21.15.83:0/1025280412 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fed0c003da0 con 0x7fed200c9530 2024-03-06T07:21:34.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.205+0000 7fed26e88640 1 -- 172.21.15.83:0/1025280412 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed200c8b70 con 0x7fed200c9530 2024-03-06T07:21:34.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.209+0000 7fed1d7fa640 1 -- 172.21.15.83:0/1025280412 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fed0c014800 con 0x7fed200c9530 2024-03-06T07:21:34.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.274+0000 7fed26e88640 1 -- 172.21.15.83:0/1025280412 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fed2004f330 con 0x7fed200c9530 2024-03-06T07:21:34.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.283+0000 7fed1d7fa640 1 -- 172.21.15.83:0/1025280412 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7fed0c0149e0 con 0x7fed200c9530 2024-03-06T07:21:34.284 INFO:teuthology.orchestra.run.smithi083.stderr:adjusted tunables profile to default 2024-03-06T07:21:34.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.283+0000 7fed26e88640 1 -- 172.21.15.83:0/1025280412 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fed200c9530 msgr2=0x7fed200ccd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:34.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.283+0000 7fed26e88640 1 --2- 172.21.15.83:0/1025280412 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fed200c9530 0x7fed200ccd70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fed0c00f540 tx=0x7fed0c00fa00 comp rx=0 tx=0).stop 2024-03-06T07:21:34.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.284+0000 7fed26e88640 1 -- 172.21.15.83:0/1025280412 shutdown_connections 2024-03-06T07:21:34.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.284+0000 7fed26e88640 1 --2- 172.21.15.83:0/1025280412 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fed200cd2b0 0x7fed201f7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:34.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.284+0000 7fed26e88640 1 --2- 172.21.15.83:0/1025280412 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fed200c9530 0x7fed200ccd70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:34.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.284+0000 7fed26e88640 1 --2- 172.21.15.83:0/1025280412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed10004d40 0x7fed201542a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:34.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.284+0000 7fed26e88640 1 -- 172.21.15.83:0/1025280412 >> 172.21.15.83:0/1025280412 conn(0x7fed2009a9f0 msgr2=0x7fed2006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:34.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.284+0000 7fed26e88640 1 -- 172.21.15.83:0/1025280412 shutdown_connections 2024-03-06T07:21:34.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:34.284+0000 7fed26e88640 1 -- 172.21.15.83:0/1025280412 wait complete. 2024-03-06T07:21:34.296 INFO:tasks.ceph:check_enable_crimson: False 2024-03-06T07:21:34.296 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-06T07:21:34.297 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:21:34.297 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-06T07:21:34.324 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:21:34.324 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-06T07:21:34.391 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:21:34.391 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-06T07:21:34.458 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:21:34.458 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-06T07:21:34.526 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:21:34.526 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-06T07:21:34.553 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:21:34.553 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-06T07:21:34.620 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:21:34.620 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-06T07:21:34.669 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-06T07:21:34.667+0000 7f1929873040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-06T07:21:34.689 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:21:34.689 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-06T07:21:34.717 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-06T07:21:34.715+0000 7f38bb43e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-06T07:21:34.722 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-06T07:21:34.721+0000 7f1929873040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-06T07:21:34.743 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new c61c90c5-4903-4b8c-acf9-0544bf2d2d4d 0 2024-03-06T07:21:34.773 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-06T07:21:34.772+0000 7f38bb43e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-06T07:21:34.872 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.871+0000 7f4651132640 1 Processor -- start 2024-03-06T07:21:34.872 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.871+0000 7f4651132640 1 -- start start 2024-03-06T07:21:34.874 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.871+0000 7f4651132640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f464c06bd80 0x7f464c06c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:34.875 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.871+0000 7f4651132640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f464c072e90 con 0x7f464c06b9a0 2024-03-06T07:21:34.875 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.871+0000 7f4651132640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f464c073170 con 0x7f464c0b5e20 2024-03-06T07:21:34.875 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.871+0000 7f4651132640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f464c1619c0 con 0x7f464c06bd80 2024-03-06T07:21:34.875 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.872+0000 7f464a59c640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f464c0b5e20 0x7f464c072590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:36944/0 (socket says 172.21.15.183:36944) 2024-03-06T07:21:34.875 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.872+0000 7f464a59c640 1 -- 172.21.15.183:0/2784840816 learned_addr learned my addr 172.21.15.183:0/2784840816 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-06T07:21:34.875 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.872+0000 7f464b59e640 1 --2- 172.21.15.183:0/2784840816 >> v2:172.21.15.83:3300/0 conn(0x7f464c06bd80 0x7f464c06c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:34.875 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.872+0000 7f4649d9b640 1 -- 172.21.15.183:0/2784840816 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2484217413 0 0) 0x7f464c073170 con 0x7f464c0b5e20 2024-03-06T07:21:34.875 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.872+0000 7f4649d9b640 1 -- 172.21.15.183:0/2784840816 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4634003520 con 0x7f464c0b5e20 2024-03-06T07:21:34.875 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.872+0000 7f4649d9b640 1 -- 172.21.15.183:0/2784840816 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4000510324 0 0) 0x7f4634003520 con 0x7f464c0b5e20 2024-03-06T07:21:34.875 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.872+0000 7f464b59e640 1 -- 172.21.15.183:0/2784840816 >> v1:172.21.15.183:6789/0 conn(0x7f464c0b5e20 legacy=0x7f464c072590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:34.875 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.872+0000 7f464b59e640 1 -- 172.21.15.183:0/2784840816 >> v1:172.21.15.83:6789/0 conn(0x7f464c06b9a0 legacy=0x7f464c0ad160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:34.875 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.872+0000 7f464b59e640 1 -- 172.21.15.183:0/2784840816 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f464c073450 con 0x7f464c06bd80 2024-03-06T07:21:34.875 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.873+0000 7f464b59e640 1 --2- 172.21.15.183:0/2784840816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f464c06bd80 0x7f464c06c1f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4644009d90 tx=0x7f464402bbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b3363a128d6cebd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:34.875 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.873+0000 7f4649d9b640 1 -- 172.21.15.183:0/2784840816 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4640001670 con 0x7f464c06bd80 2024-03-06T07:21:34.875 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.873+0000 7f4649d9b640 1 -- 172.21.15.183:0/2784840816 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f464402d6a0 con 0x7f464c06bd80 2024-03-06T07:21:34.876 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.873+0000 7f4649d9b640 1 -- 172.21.15.183:0/2784840816 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f464402da50 con 0x7f464c06bd80 2024-03-06T07:21:34.879 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.873+0000 7f4651132640 1 -- 172.21.15.183:0/2784840816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f464c06bd80 msgr2=0x7f464c06c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:34.879 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.873+0000 7f4651132640 1 --2- 172.21.15.183:0/2784840816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f464c06bd80 0x7f464c06c1f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4644009d90 tx=0x7f464402bbe0 comp rx=0 tx=0).stop 2024-03-06T07:21:34.879 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.874+0000 7f4651132640 1 -- 172.21.15.183:0/2784840816 shutdown_connections 2024-03-06T07:21:34.879 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.874+0000 7f4651132640 1 --2- 172.21.15.183:0/2784840816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f464c06bd80 0x7f464c06c1f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:34.879 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.874+0000 7f4651132640 1 -- 172.21.15.183:0/2784840816 >> 172.21.15.183:0/2784840816 conn(0x7f464c0b4fe0 msgr2=0x7f464c0b56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:34.879 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.874+0000 7f4651132640 1 -- 172.21.15.183:0/2784840816 shutdown_connections 2024-03-06T07:21:34.879 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.874+0000 7f4651132640 1 -- 172.21.15.183:0/2784840816 wait complete. 2024-03-06T07:21:34.879 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.875+0000 7f4651132640 1 Processor -- start 2024-03-06T07:21:34.879 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.875+0000 7f4651132640 1 -- start start 2024-03-06T07:21:34.879 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.876+0000 7f4651132640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f464c06b9a0 0x7f464c157c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:34.879 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.876+0000 7f4651132640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f464c06bd80 0x7f464c153b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:34.879 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.876+0000 7f4651132640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f464c0b5e20 0x7f464c0cf240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:34.879 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.876+0000 7f4651132640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f464c06c730 con 0x7f464c06bd80 2024-03-06T07:21:34.880 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.876+0000 7f4651132640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f464c0cfed0 con 0x7f464c06b9a0 2024-03-06T07:21:34.880 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.876+0000 7f4651132640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f464c0d0640 con 0x7f464c0b5e20 2024-03-06T07:21:34.880 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.876+0000 7f464ad9d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f464c06b9a0 0x7f464c157c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:34.880 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.876+0000 7f464b59e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f464c0b5e20 0x7f464c0cf240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:34.880 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.876+0000 7f464b59e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f464c0b5e20 0x7f464c0cf240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.183:51144/0 (socket says 172.21.15.183:51144) 2024-03-06T07:21:34.880 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.876+0000 7f464ad9d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f464c06b9a0 0x7f464c157c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:59034/0 (socket says 172.21.15.183:59034) 2024-03-06T07:21:34.880 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.876+0000 7f464b59e640 1 -- 172.21.15.183:0/1506432148 learned_addr learned my addr 172.21.15.183:0/1506432148 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-03-06T07:21:34.880 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.876+0000 7f464a59c640 1 --2- 172.21.15.183:0/1506432148 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f464c06bd80 0x7f464c153b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:34.880 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.876+0000 7f464b59e640 1 -- 172.21.15.183:0/1506432148 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f464c06b9a0 msgr2=0x7f464c157c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:34.880 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.876+0000 7f464b59e640 1 --2- 172.21.15.183:0/1506432148 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f464c06b9a0 0x7f464c157c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:34.880 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.876+0000 7f464b59e640 1 -- 172.21.15.183:0/1506432148 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f464c06bd80 msgr2=0x7f464c153b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:34.880 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.876+0000 7f464b59e640 1 --2- 172.21.15.183:0/1506432148 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f464c06bd80 0x7f464c153b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:34.881 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.876+0000 7f464b59e640 1 -- 172.21.15.183:0/1506432148 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f464c0cbf60 con 0x7f464c0b5e20 2024-03-06T07:21:34.881 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.876+0000 7f464ad9d640 1 --2- 172.21.15.183:0/1506432148 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f464c06b9a0 0x7f464c157c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:34.881 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.876+0000 7f464b59e640 1 --2- 172.21.15.183:0/1506432148 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f464c0b5e20 0x7f464c0cf240 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f464402c1f0 tx=0x7f4644004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:34.881 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.876+0000 7f46337fe640 1 -- 172.21.15.183:0/1506432148 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4644046020 con 0x7f464c0b5e20 2024-03-06T07:21:34.881 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.877+0000 7f46337fe640 1 -- 172.21.15.183:0/1506432148 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f464402ecd0 con 0x7f464c0b5e20 2024-03-06T07:21:34.881 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.877+0000 7f4651132640 1 -- 172.21.15.183:0/1506432148 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f464c0ce1a0 con 0x7f464c0b5e20 2024-03-06T07:21:34.881 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.877+0000 7f46337fe640 1 -- 172.21.15.183:0/1506432148 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4644042680 con 0x7f464c0b5e20 2024-03-06T07:21:34.881 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.877+0000 7f4651132640 1 -- 172.21.15.183:0/1506432148 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f464c0ce760 con 0x7f464c0b5e20 2024-03-06T07:21:34.881 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.877+0000 7f46337fe640 1 -- 172.21.15.183:0/1506432148 <== mon.2 v2:172.21.15.183:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f4644049070 con 0x7f464c0b5e20 2024-03-06T07:21:34.881 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.877+0000 7f46337fe640 1 -- 172.21.15.183:0/1506432148 <== mon.2 v2:172.21.15.183:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f464404b9d0 con 0x7f464c0b5e20 2024-03-06T07:21:34.882 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.878+0000 7f4651132640 1 -- 172.21.15.183:0/1506432148 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f464c09e4e0 con 0x7f464c0b5e20 2024-03-06T07:21:34.882 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.880+0000 7f46337fe640 1 -- 172.21.15.183:0/1506432148 <== mon.2 v2:172.21.15.183:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f4644050070 con 0x7f464c0b5e20 2024-03-06T07:21:34.952 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.951+0000 7f4651132640 1 -- 172.21.15.183:0/1506432148 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_command({"prefix": "osd new", "uuid": "c61c90c5-4903-4b8c-acf9-0544bf2d2d4d", "id": 0} v 0) v1 -- 0x7f464c068ff0 con 0x7f464c0b5e20 2024-03-06T07:21:34.958 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.956+0000 7f46337fe640 1 -- 172.21.15.183:0/1506432148 <== mon.2 v2:172.21.15.183:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c61c90c5-4903-4b8c-acf9-0544bf2d2d4d", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f4644033030 con 0x7f464c0b5e20 2024-03-06T07:21:34.958 INFO:teuthology.orchestra.run.smithi183.stdout:0 2024-03-06T07:21:34.959 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.957+0000 7f4651132640 1 -- 172.21.15.183:0/1506432148 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f464c0b5e20 msgr2=0x7f464c0cf240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:34.959 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.957+0000 7f4651132640 1 --2- 172.21.15.183:0/1506432148 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f464c0b5e20 0x7f464c0cf240 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f464402c1f0 tx=0x7f4644004a40 comp rx=0 tx=0).stop 2024-03-06T07:21:34.959 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.957+0000 7f4651132640 1 -- 172.21.15.183:0/1506432148 shutdown_connections 2024-03-06T07:21:34.959 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.957+0000 7f4651132640 1 --2- 172.21.15.183:0/1506432148 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f464c0b5e20 0x7f464c0cf240 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:34.959 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.957+0000 7f4651132640 1 --2- 172.21.15.183:0/1506432148 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f464c06bd80 0x7f464c153b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:34.959 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.957+0000 7f4651132640 1 --2- 172.21.15.183:0/1506432148 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f464c06b9a0 0x7f464c157c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:34.960 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.957+0000 7f4651132640 1 -- 172.21.15.183:0/1506432148 >> 172.21.15.183:0/1506432148 conn(0x7f464c0b4fe0 msgr2=0x7f464c06b1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:34.960 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.957+0000 7f4651132640 1 -- 172.21.15.183:0/1506432148 shutdown_connections 2024-03-06T07:21:34.960 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:34.957+0000 7f4651132640 1 -- 172.21.15.183:0/1506432148 wait complete. 2024-03-06T07:21:34.973 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new 9d6eddc0-6755-4bfc-bdf8-5d76456e7b99 1 2024-03-06T07:21:35.073 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.071+0000 7f60f0257640 1 Processor -- start 2024-03-06T07:21:35.073 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.071+0000 7f60f0257640 1 -- start start 2024-03-06T07:21:35.074 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.072+0000 7f60f0257640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f60e815fc30 0x7f60e80b47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:35.074 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.072+0000 7f60f0257640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f60e80b4d10 0x7f60e80cf200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:35.074 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.072+0000 7f60f0257640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60e80d47f0 con 0x7f60e80d06e0 2024-03-06T07:21:35.074 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.072+0000 7f60f0257640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60e80ced70 con 0x7f60e80b4d10 2024-03-06T07:21:35.075 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.072+0000 7f60f0257640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60e8160070 con 0x7f60e815fc30 2024-03-06T07:21:35.075 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.072+0000 7f60edff3640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f60e80d06e0 0x7f60e80d0ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:36952/0 (socket says 172.21.15.183:36952) 2024-03-06T07:21:35.075 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.072+0000 7f60ed7f2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f60e815fc30 0x7f60e80b47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:35.075 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.072+0000 7f60edff3640 1 -- 172.21.15.183:0/861675503 learned_addr learned my addr 172.21.15.183:0/861675503 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-06T07:21:35.075 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.072+0000 7f60ee7f4640 1 --2- 172.21.15.183:0/861675503 >> v2:172.21.15.83:3300/0 conn(0x7f60e80b4d10 0x7f60e80cf200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:35.075 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.073+0000 7f60ed7f2640 1 -- 172.21.15.183:0/861675503 >> v2:172.21.15.83:3300/0 conn(0x7f60e80b4d10 msgr2=0x7f60e80cf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:35.075 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.073+0000 7f60ecff1640 1 -- 172.21.15.183:0/861675503 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2996243223 0 0) 0x7f60e80d47f0 con 0x7f60e80d06e0 2024-03-06T07:21:35.075 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.073+0000 7f60ed7f2640 1 --2- 172.21.15.183:0/861675503 >> v2:172.21.15.83:3300/0 conn(0x7f60e80b4d10 0x7f60e80cf200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.075 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.073+0000 7f60ed7f2640 1 -- 172.21.15.183:0/861675503 >> v1:172.21.15.183:6789/0 conn(0x7f60e80d06e0 legacy=0x7f60e80d0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:35.075 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.073+0000 7f60ed7f2640 1 -- 172.21.15.183:0/861675503 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60e80d1ab0 con 0x7f60e815fc30 2024-03-06T07:21:35.075 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.073+0000 7f60ee7f4640 1 --2- 172.21.15.183:0/861675503 >> v2:172.21.15.83:3300/0 conn(0x7f60e80b4d10 0x7f60e80cf200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:35.081 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.073+0000 7f60ed7f2640 1 --2- 172.21.15.183:0/861675503 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f60e815fc30 0x7f60e80b47d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f60dc0097a0 tx=0x7f60dc02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55e1a85e4c646582 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:35.081 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.073+0000 7f60ecff1640 1 -- 172.21.15.183:0/861675503 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60dc035020 con 0x7f60e815fc30 2024-03-06T07:21:35.081 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.074+0000 7f60ecff1640 1 -- 172.21.15.183:0/861675503 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60dc031830 con 0x7f60e815fc30 2024-03-06T07:21:35.081 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.074+0000 7f60ecff1640 1 -- 172.21.15.183:0/861675503 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60dc031c00 con 0x7f60e815fc30 2024-03-06T07:21:35.081 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.074+0000 7f60f0257640 1 -- 172.21.15.183:0/861675503 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f60e815fc30 msgr2=0x7f60e80b47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:35.081 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.074+0000 7f60f0257640 1 --2- 172.21.15.183:0/861675503 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f60e815fc30 0x7f60e80b47d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f60dc0097a0 tx=0x7f60dc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.082 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.074+0000 7f60f0257640 1 -- 172.21.15.183:0/861675503 shutdown_connections 2024-03-06T07:21:35.082 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.074+0000 7f60f0257640 1 --2- 172.21.15.183:0/861675503 >> v2:172.21.15.83:3300/0 conn(0x7f60e80b4d10 0x7f60e80cf200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.082 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.074+0000 7f60f0257640 1 --2- 172.21.15.183:0/861675503 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f60e815fc30 0x7f60e80b47d0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.082 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.074+0000 7f60f0257640 1 -- 172.21.15.183:0/861675503 >> 172.21.15.183:0/861675503 conn(0x7f60e80d3cd0 msgr2=0x7f60e80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:35.082 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.074+0000 7f60f0257640 1 -- 172.21.15.183:0/861675503 shutdown_connections 2024-03-06T07:21:35.082 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.074+0000 7f60f0257640 1 -- 172.21.15.183:0/861675503 wait complete. 2024-03-06T07:21:35.082 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.075+0000 7f60f0257640 1 Processor -- start 2024-03-06T07:21:35.083 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.075+0000 7f60f0257640 1 -- start start 2024-03-06T07:21:35.083 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.076+0000 7f60f0257640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f60e80b4d10 0x7f60e81f7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:35.083 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.076+0000 7f60f0257640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f60e80d06e0 0x7f60e81f83e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:35.083 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.076+0000 7f60f0257640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60e815fc30 0x7f60e80c97c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:35.083 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.076+0000 7f60f0257640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e80ce990 con 0x7f60e815fc30 2024-03-06T07:21:35.083 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.076+0000 7f60f0257640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e80d4ab0 con 0x7f60e80d06e0 2024-03-06T07:21:35.083 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.076+0000 7f60f0257640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60e80b44d0 con 0x7f60e80b4d10 2024-03-06T07:21:35.083 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.076+0000 7f60edff3640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f60e80b4d10 0x7f60e81f7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:35.083 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.076+0000 7f60edff3640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f60e80b4d10 0x7f60e81f7350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.183:51162/0 (socket says 172.21.15.183:51162) 2024-03-06T07:21:35.084 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.076+0000 7f60ed7f2640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f60e80d06e0 0x7f60e81f83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:35.084 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.076+0000 7f60edff3640 1 -- 172.21.15.183:0/3351516913 learned_addr learned my addr 172.21.15.183:0/3351516913 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-03-06T07:21:35.084 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.076+0000 7f60ee7f4640 1 --2- 172.21.15.183:0/3351516913 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60e815fc30 0x7f60e80c97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:35.084 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.077+0000 7f60edff3640 1 -- 172.21.15.183:0/3351516913 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f60e80d06e0 msgr2=0x7f60e81f83e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:35.084 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.077+0000 7f60edff3640 1 --2- 172.21.15.183:0/3351516913 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f60e80d06e0 0x7f60e81f83e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.084 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.077+0000 7f60edff3640 1 -- 172.21.15.183:0/3351516913 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60e815fc30 msgr2=0x7f60e80c97c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:35.084 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.077+0000 7f60edff3640 1 --2- 172.21.15.183:0/3351516913 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60e815fc30 0x7f60e80c97c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.085 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.077+0000 7f60edff3640 1 -- 172.21.15.183:0/3351516913 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60e80c8720 con 0x7f60e80b4d10 2024-03-06T07:21:35.085 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.077+0000 7f60ed7f2640 1 --2- 172.21.15.183:0/3351516913 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f60e80d06e0 0x7f60e81f83e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:35.085 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.077+0000 7f60edff3640 1 --2- 172.21.15.183:0/3351516913 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f60e80b4d10 0x7f60e81f7350 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f60e400c430 tx=0x7f60e400c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:35.085 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.077+0000 7f60da7fc640 1 -- 172.21.15.183:0/3351516913 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60e40074e0 con 0x7f60e80b4d10 2024-03-06T07:21:35.085 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.077+0000 7f60da7fc640 1 -- 172.21.15.183:0/3351516913 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60e40047e0 con 0x7f60e80b4d10 2024-03-06T07:21:35.085 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.077+0000 7f60f0257640 1 -- 172.21.15.183:0/3351516913 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60e80c8a60 con 0x7f60e80b4d10 2024-03-06T07:21:35.085 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.077+0000 7f60da7fc640 1 -- 172.21.15.183:0/3351516913 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60e4004bd0 con 0x7f60e80b4d10 2024-03-06T07:21:35.085 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.077+0000 7f60f0257640 1 -- 172.21.15.183:0/3351516913 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60e80c92a0 con 0x7f60e80b4d10 2024-03-06T07:21:35.086 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.077+0000 7f60da7fc640 1 -- 172.21.15.183:0/3351516913 <== mon.2 v2:172.21.15.183:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f60e4011070 con 0x7f60e80b4d10 2024-03-06T07:21:35.086 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.077+0000 7f60da7fc640 1 -- 172.21.15.183:0/3351516913 <== mon.2 v2:172.21.15.183:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f60e4013770 con 0x7f60e80b4d10 2024-03-06T07:21:35.086 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.078+0000 7f60f0257640 1 -- 172.21.15.183:0/3351516913 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60e80a1c80 con 0x7f60e80b4d10 2024-03-06T07:21:35.086 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.080+0000 7f60da7fc640 1 -- 172.21.15.183:0/3351516913 <== mon.2 v2:172.21.15.183:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f60e80c8a60 con 0x7f60e80b4d10 2024-03-06T07:21:35.151 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.150+0000 7f60f0257640 1 -- 172.21.15.183:0/3351516913 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_command({"prefix": "osd new", "uuid": "9d6eddc0-6755-4bfc-bdf8-5d76456e7b99", "id": 1} v 0) v1 -- 0x7f60e80d2c60 con 0x7f60e80b4d10 2024-03-06T07:21:35.157 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.156+0000 7f60da7fc640 1 -- 172.21.15.183:0/3351516913 <== mon.2 v2:172.21.15.183:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9d6eddc0-6755-4bfc-bdf8-5d76456e7b99", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f60e4013970 con 0x7f60e80b4d10 2024-03-06T07:21:35.157 INFO:teuthology.orchestra.run.smithi183.stdout:1 2024-03-06T07:21:35.159 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.157+0000 7f60f0257640 1 -- 172.21.15.183:0/3351516913 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f60e80b4d10 msgr2=0x7f60e81f7350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:35.159 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.157+0000 7f60f0257640 1 --2- 172.21.15.183:0/3351516913 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f60e80b4d10 0x7f60e81f7350 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f60e400c430 tx=0x7f60e400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.159 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.157+0000 7f60f0257640 1 -- 172.21.15.183:0/3351516913 shutdown_connections 2024-03-06T07:21:35.159 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.157+0000 7f60f0257640 1 --2- 172.21.15.183:0/3351516913 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60e815fc30 0x7f60e80c97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.159 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.157+0000 7f60f0257640 1 --2- 172.21.15.183:0/3351516913 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f60e80d06e0 0x7f60e81f83e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.159 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.157+0000 7f60f0257640 1 --2- 172.21.15.183:0/3351516913 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f60e80b4d10 0x7f60e81f7350 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.159 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.157+0000 7f60f0257640 1 -- 172.21.15.183:0/3351516913 >> 172.21.15.183:0/3351516913 conn(0x7f60e80d3cd0 msgr2=0x7f60e80cfe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:35.159 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.157+0000 7f60f0257640 1 -- 172.21.15.183:0/3351516913 shutdown_connections 2024-03-06T07:21:35.159 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.157+0000 7f60f0257640 1 -- 172.21.15.183:0/3351516913 wait complete. 2024-03-06T07:21:35.169 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new 6d74c643-7301-4daf-91f4-cfd3203a7ec4 2 2024-03-06T07:21:35.249 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-06T07:21:35.247+0000 7f1929873040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-06T07:21:35.270 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.268+0000 7fbadc755640 1 Processor -- start 2024-03-06T07:21:35.270 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.268+0000 7fbadc755640 1 -- start start 2024-03-06T07:21:35.271 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.269+0000 7fbadc755640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fbad415bbd0 0x7fbad415cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:35.271 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.269+0000 7fbadc755640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbad40b5b60 0x7fbad41658a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:35.271 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.269+0000 7fbadc755640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbad415d650 con 0x7fbad40b4f10 2024-03-06T07:21:35.271 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.269+0000 7fbadc755640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbad415e260 con 0x7fbad415bbd0 2024-03-06T07:21:35.271 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.269+0000 7fbadc755640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbad415d930 con 0x7fbad40b5b60 2024-03-06T07:21:35.272 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.269+0000 7fbadacf2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbad40b5b60 0x7fbad41658a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:35.272 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.269+0000 7fbada4f1640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fbad415bbd0 0x7fbad415cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:35.272 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.269+0000 7fbada4f1640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fbad415bbd0 0x7fbad415cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:59056/0 (socket says 172.21.15.183:59056) 2024-03-06T07:21:35.272 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.269+0000 7fbada4f1640 1 -- 172.21.15.183:0/1706653150 learned_addr learned my addr 172.21.15.183:0/1706653150 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-03-06T07:21:35.272 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.270+0000 7fbada4f1640 1 -- 172.21.15.183:0/1706653150 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbad40b5b60 msgr2=0x7fbad41658a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:35.272 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.270+0000 7fbada4f1640 1 --2- 172.21.15.183:0/1706653150 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbad40b5b60 0x7fbad41658a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.272 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.270+0000 7fbada4f1640 1 -- 172.21.15.183:0/1706653150 >> v1:172.21.15.83:6789/0 conn(0x7fbad40b4f10 legacy=0x7fbad40b5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:35.272 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.270+0000 7fbada4f1640 1 -- 172.21.15.183:0/1706653150 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad415dc10 con 0x7fbad415bbd0 2024-03-06T07:21:35.272 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.270+0000 7fbadacf2640 1 --2- 172.21.15.183:0/1706653150 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbad40b5b60 0x7fbad41658a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:35.274 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.270+0000 7fbada4f1640 1 --2- 172.21.15.183:0/1706653150 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbad415bbd0 0x7fbad415cd00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fbac8009870 tx=0x7fbac802ba20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d02c12f23a72b5f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:35.274 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.270+0000 7fbad94ef640 1 -- 172.21.15.183:0/1706653150 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac802f840 con 0x7fbad415bbd0 2024-03-06T07:21:35.274 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.270+0000 7fbad94ef640 1 -- 172.21.15.183:0/1706653150 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbac802f9e0 con 0x7fbad415bbd0 2024-03-06T07:21:35.274 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.270+0000 7fbad94ef640 1 -- 172.21.15.183:0/1706653150 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac802fdb0 con 0x7fbad415bbd0 2024-03-06T07:21:35.274 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.271+0000 7fbadc755640 1 -- 172.21.15.183:0/1706653150 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbad415bbd0 msgr2=0x7fbad415cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:35.274 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.271+0000 7fbadc755640 1 --2- 172.21.15.183:0/1706653150 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbad415bbd0 0x7fbad415cd00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fbac8009870 tx=0x7fbac802ba20 comp rx=0 tx=0).stop 2024-03-06T07:21:35.274 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.271+0000 7fbadc755640 1 -- 172.21.15.183:0/1706653150 shutdown_connections 2024-03-06T07:21:35.274 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.271+0000 7fbadc755640 1 --2- 172.21.15.183:0/1706653150 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbad40b5b60 0x7fbad41658a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.274 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.271+0000 7fbadc755640 1 --2- 172.21.15.183:0/1706653150 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbad415bbd0 0x7fbad415cd00 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.274 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.271+0000 7fbadc755640 1 -- 172.21.15.183:0/1706653150 >> 172.21.15.183:0/1706653150 conn(0x7fbad40b6ea0 msgr2=0x7fbad40b72b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:35.274 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.271+0000 7fbadc755640 1 -- 172.21.15.183:0/1706653150 shutdown_connections 2024-03-06T07:21:35.274 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.271+0000 7fbadc755640 1 -- 172.21.15.183:0/1706653150 wait complete. 2024-03-06T07:21:35.274 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.272+0000 7fbadc755640 1 Processor -- start 2024-03-06T07:21:35.274 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.272+0000 7fbadc755640 1 -- start start 2024-03-06T07:21:35.278 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.273+0000 7fbadc755640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbad40b4f10 0x7fbad41e6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:35.278 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.273+0000 7fbadc755640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbad40b5b60 0x7fbad409b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:35.278 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.273+0000 7fbadc755640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbad415bbd0 0x7fbad4141c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:35.278 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.273+0000 7fbadc755640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad415c130 con 0x7fbad415bbd0 2024-03-06T07:21:35.278 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.273+0000 7fbadc755640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad415bfb0 con 0x7fbad40b5b60 2024-03-06T07:21:35.278 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.273+0000 7fbadc755640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbad415d240 con 0x7fbad40b4f10 2024-03-06T07:21:35.278 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.273+0000 7fbada4f1640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbad40b4f10 0x7fbad41e6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:35.278 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.273+0000 7fbad9cf0640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbad40b5b60 0x7fbad409b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:35.279 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.273+0000 7fbada4f1640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbad40b4f10 0x7fbad41e6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.183:51174/0 (socket says 172.21.15.183:51174) 2024-03-06T07:21:35.279 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.273+0000 7fbad9cf0640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbad40b5b60 0x7fbad409b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:59066/0 (socket says 172.21.15.183:59066) 2024-03-06T07:21:35.279 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.273+0000 7fbada4f1640 1 -- 172.21.15.183:0/4277102635 learned_addr learned my addr 172.21.15.183:0/4277102635 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-03-06T07:21:35.279 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.273+0000 7fbadacf2640 1 --2- 172.21.15.183:0/4277102635 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbad415bbd0 0x7fbad4141c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:35.279 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.273+0000 7fbad9cf0640 1 -- 172.21.15.183:0/4277102635 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbad40b4f10 msgr2=0x7fbad41e6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:35.279 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.273+0000 7fbad9cf0640 1 --2- 172.21.15.183:0/4277102635 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbad40b4f10 0x7fbad41e6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.279 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.273+0000 7fbad9cf0640 1 -- 172.21.15.183:0/4277102635 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbad415bbd0 msgr2=0x7fbad4141c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:35.279 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.273+0000 7fbad9cf0640 1 --2- 172.21.15.183:0/4277102635 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbad415bbd0 0x7fbad4141c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.279 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.273+0000 7fbad9cf0640 1 -- 172.21.15.183:0/4277102635 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad41e74d0 con 0x7fbad40b5b60 2024-03-06T07:21:35.279 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.273+0000 7fbada4f1640 1 --2- 172.21.15.183:0/4277102635 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbad40b4f10 0x7fbad41e6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:35.279 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.274+0000 7fbad9cf0640 1 --2- 172.21.15.183:0/4277102635 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbad40b5b60 0x7fbad409b9a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fbad000c340 tx=0x7fbad000c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:35.279 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.274+0000 7fbac6ffd640 1 -- 172.21.15.183:0/4277102635 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad00052d0 con 0x7fbad40b5b60 2024-03-06T07:21:35.279 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.274+0000 7fbac6ffd640 1 -- 172.21.15.183:0/4277102635 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbad0005490 con 0x7fbad40b5b60 2024-03-06T07:21:35.279 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.274+0000 7fbadc755640 1 -- 172.21.15.183:0/4277102635 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbad41e77b0 con 0x7fbad40b5b60 2024-03-06T07:21:35.280 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.274+0000 7fbac6ffd640 1 -- 172.21.15.183:0/4277102635 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad0007260 con 0x7fbad40b5b60 2024-03-06T07:21:35.280 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.274+0000 7fbadc755640 1 -- 172.21.15.183:0/4277102635 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbad41e7d70 con 0x7fbad40b5b60 2024-03-06T07:21:35.280 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.274+0000 7fbac6ffd640 1 -- 172.21.15.183:0/4277102635 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fbad0007400 con 0x7fbad40b5b60 2024-03-06T07:21:35.280 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.274+0000 7fbac6ffd640 1 -- 172.21.15.183:0/4277102635 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7fbad0011590 con 0x7fbad40b5b60 2024-03-06T07:21:35.280 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.275+0000 7fbadc755640 1 -- 172.21.15.183:0/4277102635 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbad409fca0 con 0x7fbad40b5b60 2024-03-06T07:21:35.280 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.279+0000 7fbac6ffd640 1 -- 172.21.15.183:0/4277102635 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fbad0003a00 con 0x7fbad40b5b60 2024-03-06T07:21:35.303 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-06T07:21:35.302+0000 7f1929873040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-06T07:21:35.330 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-06T07:21:35.329+0000 7f38bb43e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-06T07:21:35.348 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-06T07:21:35.346+0000 7f1929873040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-06T07:21:35.351 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.350+0000 7fbadc755640 1 -- 172.21.15.183:0/4277102635 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd new", "uuid": "6d74c643-7301-4daf-91f4-cfd3203a7ec4", "id": 2} v 0) v1 -- 0x7fbad40b6ba0 con 0x7fbad40b5b60 2024-03-06T07:21:35.355 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.354+0000 7fbac6ffd640 1 -- 172.21.15.183:0/4277102635 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6d74c643-7301-4daf-91f4-cfd3203a7ec4", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fbad0003cc0 con 0x7fbad40b5b60 2024-03-06T07:21:35.355 INFO:teuthology.orchestra.run.smithi183.stdout:2 2024-03-06T07:21:35.357 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.355+0000 7fbadc755640 1 -- 172.21.15.183:0/4277102635 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbad40b5b60 msgr2=0x7fbad409b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:35.357 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.355+0000 7fbadc755640 1 --2- 172.21.15.183:0/4277102635 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbad40b5b60 0x7fbad409b9a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fbad000c340 tx=0x7fbad000c800 comp rx=0 tx=0).stop 2024-03-06T07:21:35.357 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.355+0000 7fbadc755640 1 -- 172.21.15.183:0/4277102635 shutdown_connections 2024-03-06T07:21:35.357 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.355+0000 7fbadc755640 1 --2- 172.21.15.183:0/4277102635 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbad415bbd0 0x7fbad4141c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.357 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.355+0000 7fbadc755640 1 --2- 172.21.15.183:0/4277102635 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbad40b5b60 0x7fbad409b9a0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.357 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.355+0000 7fbadc755640 1 --2- 172.21.15.183:0/4277102635 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbad40b4f10 0x7fbad41e6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.357 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.355+0000 7fbadc755640 1 -- 172.21.15.183:0/4277102635 >> 172.21.15.183:0/4277102635 conn(0x7fbad40b6ea0 msgr2=0x7fbad40b3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:35.358 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.355+0000 7fbadc755640 1 -- 172.21.15.183:0/4277102635 shutdown_connections 2024-03-06T07:21:35.358 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.355+0000 7fbadc755640 1 -- 172.21.15.183:0/4277102635 wait complete. 2024-03-06T07:21:35.367 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new fa07c172-62c0-485f-89ec-495300de270d 3 2024-03-06T07:21:35.388 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-06T07:21:35.387+0000 7f38bb43e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-06T07:21:35.402 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-06T07:21:35.400+0000 7f1929873040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-06T07:21:35.435 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-06T07:21:35.434+0000 7f38bb43e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-06T07:21:35.444 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-06T07:21:35.443+0000 7f1929873040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-06T07:21:35.466 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.465+0000 7ff8a62e0640 1 Processor -- start 2024-03-06T07:21:35.466 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.465+0000 7ff8a62e0640 1 -- start start 2024-03-06T07:21:35.468 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.466+0000 7ff8a62e0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8a0071a70 0x7ff8a0071e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:35.468 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.466+0000 7ff8a62e0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff8a00b42e0 0x7ff8a015c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:35.468 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.466+0000 7ff8a62e0640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8a0072390 con 0x7ff8a015ad90 2024-03-06T07:21:35.468 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.466+0000 7ff8a62e0640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8a015c020 con 0x7ff8a00b42e0 2024-03-06T07:21:35.468 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.466+0000 7ff8a62e0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a00b46c0 con 0x7ff8a0071a70 2024-03-06T07:21:35.468 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.466+0000 7ff89ffff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8a0071a70 0x7ff8a0071e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:35.468 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.466+0000 7ff89ffff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8a0071a70 0x7ff8a0071e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.183:51182/0 (socket says 172.21.15.183:51182) 2024-03-06T07:21:35.468 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.466+0000 7ff89ffff640 1 -- 172.21.15.183:0/2275120826 learned_addr learned my addr 172.21.15.183:0/2275120826 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-03-06T07:21:35.468 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.466+0000 7ff8a487d640 1 --2- 172.21.15.183:0/2275120826 >> v2:172.21.15.83:3300/0 conn(0x7ff8a00b42e0 0x7ff8a015c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:35.469 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.466+0000 7ff89ffff640 1 -- 172.21.15.183:0/2275120826 >> v2:172.21.15.83:3300/0 conn(0x7ff8a00b42e0 msgr2=0x7ff8a015c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:35.469 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.466+0000 7ff89ffff640 1 --2- 172.21.15.183:0/2275120826 >> v2:172.21.15.83:3300/0 conn(0x7ff8a00b42e0 0x7ff8a015c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.469 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.466+0000 7ff89ffff640 1 -- 172.21.15.183:0/2275120826 >> v1:172.21.15.83:6789/0 conn(0x7ff8a015ad90 legacy=0x7ff8a00b3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:35.469 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.466+0000 7ff89ffff640 1 -- 172.21.15.183:0/2275120826 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a015b260 con 0x7ff8a0071a70 2024-03-06T07:21:35.470 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.467+0000 7ff89ffff640 1 --2- 172.21.15.183:0/2275120826 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8a0071a70 0x7ff8a0071e50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff894009870 tx=0x7ff89402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e140f6095ab85f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:35.470 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.467+0000 7ff89effd640 1 -- 172.21.15.183:0/2275120826 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff894035020 con 0x7ff8a0071a70 2024-03-06T07:21:35.470 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.467+0000 7ff89effd640 1 -- 172.21.15.183:0/2275120826 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff894031640 con 0x7ff8a0071a70 2024-03-06T07:21:35.470 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.467+0000 7ff89effd640 1 -- 172.21.15.183:0/2275120826 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89402d040 con 0x7ff8a0071a70 2024-03-06T07:21:35.470 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.468+0000 7ff8a62e0640 1 -- 172.21.15.183:0/2275120826 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8a0071a70 msgr2=0x7ff8a0071e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:35.470 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.468+0000 7ff8a62e0640 1 --2- 172.21.15.183:0/2275120826 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8a0071a70 0x7ff8a0071e50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff894009870 tx=0x7ff89402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.471 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.468+0000 7ff8a62e0640 1 -- 172.21.15.183:0/2275120826 shutdown_connections 2024-03-06T07:21:35.471 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.468+0000 7ff8a62e0640 1 --2- 172.21.15.183:0/2275120826 >> v2:172.21.15.83:3300/0 conn(0x7ff8a00b42e0 0x7ff8a015c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.471 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.468+0000 7ff8a62e0640 1 --2- 172.21.15.183:0/2275120826 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8a0071a70 0x7ff8a0071e50 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.471 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.468+0000 7ff8a62e0640 1 -- 172.21.15.183:0/2275120826 >> 172.21.15.183:0/2275120826 conn(0x7ff8a00b2660 msgr2=0x7ff8a00b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:35.471 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.468+0000 7ff8a62e0640 1 -- 172.21.15.183:0/2275120826 shutdown_connections 2024-03-06T07:21:35.471 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.468+0000 7ff8a62e0640 1 -- 172.21.15.183:0/2275120826 wait complete. 2024-03-06T07:21:35.471 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.469+0000 7ff8a62e0640 1 Processor -- start 2024-03-06T07:21:35.471 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.469+0000 7ff8a62e0640 1 -- start start 2024-03-06T07:21:35.473 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.470+0000 7ff8a62e0640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff8a0071a70 0x7ff8a00cc170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:35.473 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.470+0000 7ff8a62e0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8a00b42e0 0x7ff8a014a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:35.473 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.470+0000 7ff8a62e0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff8a015ad90 0x7ff8a0151f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:35.473 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.470+0000 7ff8a62e0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a015bb00 con 0x7ff8a015ad90 2024-03-06T07:21:35.473 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.470+0000 7ff8a62e0640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a015b820 con 0x7ff8a0071a70 2024-03-06T07:21:35.473 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.470+0000 7ff8a62e0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a015c750 con 0x7ff8a00b42e0 2024-03-06T07:21:35.474 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.470+0000 7ff89ffff640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff8a0071a70 0x7ff8a00cc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:35.474 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.470+0000 7ff89f7fe640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8a00b42e0 0x7ff8a014a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:35.474 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.470+0000 7ff89ffff640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff8a0071a70 0x7ff8a00cc170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:59068/0 (socket says 172.21.15.183:59068) 2024-03-06T07:21:35.474 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.470+0000 7ff89ffff640 1 -- 172.21.15.183:0/3067208594 learned_addr learned my addr 172.21.15.183:0/3067208594 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-03-06T07:21:35.474 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.470+0000 7ff8a487d640 1 --2- 172.21.15.183:0/3067208594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff8a015ad90 0x7ff8a0151f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:35.474 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.470+0000 7ff89ffff640 1 -- 172.21.15.183:0/3067208594 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8a00b42e0 msgr2=0x7ff8a014a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:35.474 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.470+0000 7ff89ffff640 1 --2- 172.21.15.183:0/3067208594 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8a00b42e0 0x7ff8a014a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.474 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.470+0000 7ff89ffff640 1 -- 172.21.15.183:0/3067208594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff8a015ad90 msgr2=0x7ff8a0151f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:35.474 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.470+0000 7ff89ffff640 1 --2- 172.21.15.183:0/3067208594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff8a015ad90 0x7ff8a0151f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.474 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.470+0000 7ff89ffff640 1 -- 172.21.15.183:0/3067208594 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a00cb0d0 con 0x7ff8a0071a70 2024-03-06T07:21:35.474 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.470+0000 7ff89ffff640 1 --2- 172.21.15.183:0/3067208594 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff8a0071a70 0x7ff8a00cc170 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff894000c00 tx=0x7ff8940046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:35.474 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.470+0000 7ff89cff9640 1 -- 172.21.15.183:0/3067208594 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff894040070 con 0x7ff8a0071a70 2024-03-06T07:21:35.474 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.470+0000 7ff89cff9640 1 -- 172.21.15.183:0/3067208594 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8940399f0 con 0x7ff8a0071a70 2024-03-06T07:21:35.475 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.470+0000 7ff8a62e0640 1 -- 172.21.15.183:0/3067208594 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8a00cb3b0 con 0x7ff8a0071a70 2024-03-06T07:21:35.475 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.470+0000 7ff89cff9640 1 -- 172.21.15.183:0/3067208594 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff894035040 con 0x7ff8a0071a70 2024-03-06T07:21:35.475 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.470+0000 7ff8a62e0640 1 -- 172.21.15.183:0/3067208594 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8a00cbc50 con 0x7ff8a0071a70 2024-03-06T07:21:35.475 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.471+0000 7ff89cff9640 1 -- 172.21.15.183:0/3067208594 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7ff89404b060 con 0x7ff8a0071a70 2024-03-06T07:21:35.475 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.471+0000 7ff89cff9640 1 -- 172.21.15.183:0/3067208594 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7ff89403a8b0 con 0x7ff8a0071a70 2024-03-06T07:21:35.475 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.471+0000 7ff8a62e0640 1 -- 172.21.15.183:0/3067208594 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8a0073390 con 0x7ff8a0071a70 2024-03-06T07:21:35.475 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.473+0000 7ff89cff9640 1 -- 172.21.15.183:0/3067208594 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7ff89403a420 con 0x7ff8a0071a70 2024-03-06T07:21:35.492 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-06T07:21:35.491+0000 7f38bb43e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-06T07:21:35.535 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-06T07:21:35.534+0000 7f38bb43e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-06T07:21:35.545 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.544+0000 7ff8a62e0640 1 -- 172.21.15.183:0/3067208594 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd new", "uuid": "fa07c172-62c0-485f-89ec-495300de270d", "id": 3} v 0) v1 -- 0x7ff8a00d07d0 con 0x7ff8a0071a70 2024-03-06T07:21:35.549 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.547+0000 7ff89cff9640 1 -- 172.21.15.183:0/3067208594 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fa07c172-62c0-485f-89ec-495300de270d", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff89403a6c0 con 0x7ff8a0071a70 2024-03-06T07:21:35.549 INFO:teuthology.orchestra.run.smithi183.stdout:3 2024-03-06T07:21:35.550 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.548+0000 7ff8a62e0640 1 -- 172.21.15.183:0/3067208594 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff8a0071a70 msgr2=0x7ff8a00cc170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:35.550 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.548+0000 7ff8a62e0640 1 --2- 172.21.15.183:0/3067208594 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff8a0071a70 0x7ff8a00cc170 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff894000c00 tx=0x7ff8940046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.550 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.548+0000 7ff8a62e0640 1 -- 172.21.15.183:0/3067208594 shutdown_connections 2024-03-06T07:21:35.550 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.548+0000 7ff8a62e0640 1 --2- 172.21.15.183:0/3067208594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff8a015ad90 0x7ff8a0151f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.550 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.548+0000 7ff8a62e0640 1 --2- 172.21.15.183:0/3067208594 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8a00b42e0 0x7ff8a014a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.550 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.548+0000 7ff8a62e0640 1 --2- 172.21.15.183:0/3067208594 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff8a0071a70 0x7ff8a00cc170 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.551 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.548+0000 7ff8a62e0640 1 -- 172.21.15.183:0/3067208594 >> 172.21.15.183:0/3067208594 conn(0x7ff8a00b2660 msgr2=0x7ff8a00b19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:35.551 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.549+0000 7ff8a62e0640 1 -- 172.21.15.183:0/3067208594 shutdown_connections 2024-03-06T07:21:35.551 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.549+0000 7ff8a62e0640 1 -- 172.21.15.183:0/3067208594 wait complete. 2024-03-06T07:21:35.560 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-06T07:21:35.558+0000 7f1929873040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-06T07:21:35.561 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new 3b420d33-2d34-4f12-ae2e-98ed0188d304 4 2024-03-06T07:21:35.629 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-06T07:21:35.628+0000 7f1929873040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-06T07:21:35.656 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-06T07:21:35.655+0000 7f38bb43e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-06T07:21:35.675 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-06T07:21:35.673+0000 7f1929873040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-06T07:21:35.701 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.699+0000 7fa214cd4640 1 Processor -- start 2024-03-06T07:21:35.701 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.699+0000 7fa214cd4640 1 -- start start 2024-03-06T07:21:35.703 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.700+0000 7fa214cd4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa2100b42f0 0x7fa21015fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:35.703 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.700+0000 7fa214cd4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa21015c470 con 0x7fa21006f0b0 2024-03-06T07:21:35.703 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.700+0000 7fa214cd4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2101602a0 con 0x7fa21015c090 2024-03-06T07:21:35.703 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.700+0000 7fa214cd4640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2100cddc0 con 0x7fa2100b42f0 2024-03-06T07:21:35.703 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.700+0000 7fa20e59c640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fa21015c090 0x7fa2100cebe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:36954/0 (socket says 172.21.15.183:36954) 2024-03-06T07:21:35.703 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.700+0000 7fa20e59c640 1 -- 172.21.15.183:0/3632479396 learned_addr learned my addr 172.21.15.183:0/3632479396 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-06T07:21:35.703 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.700+0000 7fa20dd9b640 1 --2- 172.21.15.183:0/3632479396 >> v2:172.21.15.83:3300/0 conn(0x7fa2100b42f0 0x7fa21015fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:35.703 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.700+0000 7fa20d59a640 1 -- 172.21.15.183:0/3632479396 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 461540294 0 0) 0x7fa2101602a0 con 0x7fa21015c090 2024-03-06T07:21:35.703 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.700+0000 7fa20d59a640 1 -- 172.21.15.183:0/3632479396 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1fc003520 con 0x7fa21015c090 2024-03-06T07:21:35.704 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.700+0000 7fa20d59a640 1 -- 172.21.15.183:0/3632479396 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 401555772 0 0) 0x7fa21015c470 con 0x7fa21006f0b0 2024-03-06T07:21:35.704 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.700+0000 7fa20d59a640 1 -- 172.21.15.183:0/3632479396 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2101602a0 con 0x7fa21006f0b0 2024-03-06T07:21:35.704 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.701+0000 7fa20dd9b640 1 -- 172.21.15.183:0/3632479396 >> v1:172.21.15.183:6789/0 conn(0x7fa21015c090 legacy=0x7fa2100cebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:35.704 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.701+0000 7fa20d59a640 1 -- 172.21.15.183:0/3632479396 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4038081468 0 0) 0x7fa1fc003520 con 0x7fa21015c090 2024-03-06T07:21:35.704 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.701+0000 7fa20dd9b640 1 -- 172.21.15.183:0/3632479396 >> v1:172.21.15.83:6789/0 conn(0x7fa21006f0b0 legacy=0x7fa2100cc660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:35.704 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.701+0000 7fa20dd9b640 1 -- 172.21.15.183:0/3632479396 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa210160580 con 0x7fa2100b42f0 2024-03-06T07:21:35.704 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.701+0000 7fa20dd9b640 1 --2- 172.21.15.183:0/3632479396 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2100b42f0 0x7fa21015fb70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa200009720 tx=0x7fa20002b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baf8289a88859922 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:35.704 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.701+0000 7fa20d59a640 1 -- 172.21.15.183:0/3632479396 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa200038070 con 0x7fa2100b42f0 2024-03-06T07:21:35.704 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.702+0000 7fa20d59a640 1 -- 172.21.15.183:0/3632479396 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa200031520 con 0x7fa2100b42f0 2024-03-06T07:21:35.704 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.702+0000 7fa20d59a640 1 -- 172.21.15.183:0/3632479396 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f8001670 con 0x7fa2100b42f0 2024-03-06T07:21:35.705 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.702+0000 7fa214cd4640 1 -- 172.21.15.183:0/3632479396 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2100b42f0 msgr2=0x7fa21015fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:35.705 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.702+0000 7fa214cd4640 1 --2- 172.21.15.183:0/3632479396 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2100b42f0 0x7fa21015fb70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa200009720 tx=0x7fa20002b7c0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.706 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.702+0000 7fa214cd4640 1 -- 172.21.15.183:0/3632479396 shutdown_connections 2024-03-06T07:21:35.706 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.702+0000 7fa214cd4640 1 --2- 172.21.15.183:0/3632479396 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2100b42f0 0x7fa21015fb70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.706 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.702+0000 7fa214cd4640 1 -- 172.21.15.183:0/3632479396 >> 172.21.15.183:0/3632479396 conn(0x7fa2100b3800 msgr2=0x7fa2100b3be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:35.706 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.702+0000 7fa214cd4640 1 -- 172.21.15.183:0/3632479396 shutdown_connections 2024-03-06T07:21:35.706 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.702+0000 7fa214cd4640 1 -- 172.21.15.183:0/3632479396 wait complete. 2024-03-06T07:21:35.706 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.704+0000 7fa214cd4640 1 Processor -- start 2024-03-06T07:21:35.706 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.704+0000 7fa214cd4640 1 -- start start 2024-03-06T07:21:35.708 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.704+0000 7fa214cd4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa21006f0b0 0x7fa21014a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:35.708 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.704+0000 7fa214cd4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2100b42f0 0x7fa2100b7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:35.708 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.704+0000 7fa214cd4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa21015c090 0x7fa21015e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:35.709 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.704+0000 7fa214cd4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2100cf470 con 0x7fa2100b42f0 2024-03-06T07:21:35.709 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.704+0000 7fa214cd4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2100cf2f0 con 0x7fa21015c090 2024-03-06T07:21:35.709 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.704+0000 7fa214cd4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2100ccd70 con 0x7fa21006f0b0 2024-03-06T07:21:35.709 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.705+0000 7fa20e59c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa21006f0b0 0x7fa21014a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:35.709 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.705+0000 7fa20e59c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa21006f0b0 0x7fa21014a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.183:51196/0 (socket says 172.21.15.183:51196) 2024-03-06T07:21:35.709 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.705+0000 7fa20e59c640 1 -- 172.21.15.183:0/101618624 learned_addr learned my addr 172.21.15.183:0/101618624 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-03-06T07:21:35.709 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.705+0000 7fa20ed9d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa21015c090 0x7fa21015e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:35.709 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.705+0000 7fa20dd9b640 1 --2- 172.21.15.183:0/101618624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2100b42f0 0x7fa2100b7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:35.709 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.705+0000 7fa20ed9d640 1 -- 172.21.15.183:0/101618624 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa21006f0b0 msgr2=0x7fa21014a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:35.709 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.705+0000 7fa20ed9d640 1 --2- 172.21.15.183:0/101618624 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa21006f0b0 0x7fa21014a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.709 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.705+0000 7fa20ed9d640 1 -- 172.21.15.183:0/101618624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2100b42f0 msgr2=0x7fa2100b7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:35.709 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.705+0000 7fa20ed9d640 1 --2- 172.21.15.183:0/101618624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2100b42f0 0x7fa2100b7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.710 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.705+0000 7fa20ed9d640 1 -- 172.21.15.183:0/101618624 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa21014ab30 con 0x7fa21015c090 2024-03-06T07:21:35.710 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.705+0000 7fa20ed9d640 1 --2- 172.21.15.183:0/101618624 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa21015c090 0x7fa21015e040 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa20800c6c0 tx=0x7fa20800cb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:35.710 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.705+0000 7fa206ffd640 1 -- 172.21.15.183:0/101618624 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa208003b70 con 0x7fa21015c090 2024-03-06T07:21:35.710 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.705+0000 7fa206ffd640 1 -- 172.21.15.183:0/101618624 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa208003d10 con 0x7fa21015c090 2024-03-06T07:21:35.710 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.705+0000 7fa206ffd640 1 -- 172.21.15.183:0/101618624 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa208005310 con 0x7fa21015c090 2024-03-06T07:21:35.710 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.705+0000 7fa214cd4640 1 -- 172.21.15.183:0/101618624 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2100b5fb0 con 0x7fa21015c090 2024-03-06T07:21:35.710 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.705+0000 7fa214cd4640 1 -- 172.21.15.183:0/101618624 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2100b6570 con 0x7fa21015c090 2024-03-06T07:21:35.710 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.705+0000 7fa206ffd640 1 -- 172.21.15.183:0/101618624 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fa2080054b0 con 0x7fa21015c090 2024-03-06T07:21:35.710 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.705+0000 7fa206ffd640 1 -- 172.21.15.183:0/101618624 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7fa2080129d0 con 0x7fa21015c090 2024-03-06T07:21:35.710 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.706+0000 7fa214cd4640 1 -- 172.21.15.183:0/101618624 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa21009fe60 con 0x7fa21015c090 2024-03-06T07:21:35.710 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.708+0000 7fa206ffd640 1 -- 172.21.15.183:0/101618624 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fa208012cf0 con 0x7fa21015c090 2024-03-06T07:21:35.728 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-06T07:21:35.726+0000 7f1929873040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-06T07:21:35.729 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-06T07:21:35.728+0000 7f38bb43e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-06T07:21:35.774 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-06T07:21:35.773+0000 7f1929873040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-06T07:21:35.776 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-06T07:21:35.774+0000 7f38bb43e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-06T07:21:35.784 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.779+0000 7fa214cd4640 1 -- 172.21.15.183:0/101618624 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd new", "uuid": "3b420d33-2d34-4f12-ae2e-98ed0188d304", "id": 4} v 0) v1 -- 0x7fa210068ff0 con 0x7fa21015c090 2024-03-06T07:21:35.786 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.784+0000 7fa206ffd640 1 -- 172.21.15.183:0/101618624 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3b420d33-2d34-4f12-ae2e-98ed0188d304", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fa2080124c0 con 0x7fa21015c090 2024-03-06T07:21:35.786 INFO:teuthology.orchestra.run.smithi183.stdout:4 2024-03-06T07:21:35.787 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.785+0000 7fa214cd4640 1 -- 172.21.15.183:0/101618624 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa21015c090 msgr2=0x7fa21015e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:35.787 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.785+0000 7fa214cd4640 1 --2- 172.21.15.183:0/101618624 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa21015c090 0x7fa21015e040 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa20800c6c0 tx=0x7fa20800cb80 comp rx=0 tx=0).stop 2024-03-06T07:21:35.787 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.785+0000 7fa214cd4640 1 -- 172.21.15.183:0/101618624 shutdown_connections 2024-03-06T07:21:35.787 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.785+0000 7fa214cd4640 1 --2- 172.21.15.183:0/101618624 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa21015c090 0x7fa21015e040 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.787 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.785+0000 7fa214cd4640 1 --2- 172.21.15.183:0/101618624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2100b42f0 0x7fa2100b7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.787 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.785+0000 7fa214cd4640 1 --2- 172.21.15.183:0/101618624 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa21006f0b0 0x7fa21014a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.787 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.785+0000 7fa214cd4640 1 -- 172.21.15.183:0/101618624 >> 172.21.15.183:0/101618624 conn(0x7fa2100b3800 msgr2=0x7fa2100b2920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:35.787 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.785+0000 7fa214cd4640 1 -- 172.21.15.183:0/101618624 shutdown_connections 2024-03-06T07:21:35.788 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.785+0000 7fa214cd4640 1 -- 172.21.15.183:0/101618624 wait complete. 2024-03-06T07:21:35.797 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new 0b171206-23f4-4d38-b977-4a88e15eebb1 5 2024-03-06T07:21:35.833 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-06T07:21:35.831+0000 7f38bb43e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-06T07:21:35.881 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-06T07:21:35.880+0000 7f38bb43e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-06T07:21:35.897 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.895+0000 7f0f2c283640 1 Processor -- start 2024-03-06T07:21:35.897 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.896+0000 7f0f2c283640 1 -- start start 2024-03-06T07:21:35.898 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.896+0000 7f0f2c283640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0f240701a0 0x7f0f2406da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:35.898 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.896+0000 7f0f2c283640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f240cf430 con 0x7f0f240b42f0 2024-03-06T07:21:35.899 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.896+0000 7f0f2c283640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f2406e240 con 0x7f0f2415e6d0 2024-03-06T07:21:35.899 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.896+0000 7f0f2c283640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f2406dfa0 con 0x7f0f240701a0 2024-03-06T07:21:35.899 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.897+0000 7f0f2a820640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f0f2415e6d0 0x7f0f2415eb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:36962/0 (socket says 172.21.15.183:36962) 2024-03-06T07:21:35.899 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.897+0000 7f0f2a820640 1 -- 172.21.15.183:0/1641959109 learned_addr learned my addr 172.21.15.183:0/1641959109 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-06T07:21:35.899 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.897+0000 7f0f2a01f640 1 --2- 172.21.15.183:0/1641959109 >> v2:172.21.15.83:3300/0 conn(0x7f0f240701a0 0x7f0f2406da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:35.899 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.897+0000 7f0f2901d640 1 -- 172.21.15.183:0/1641959109 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 69949581 0 0) 0x7f0f2406e240 con 0x7f0f2415e6d0 2024-03-06T07:21:35.899 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.897+0000 7f0f2901d640 1 -- 172.21.15.183:0/1641959109 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f10003520 con 0x7f0f2415e6d0 2024-03-06T07:21:35.899 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.897+0000 7f0f2a01f640 1 -- 172.21.15.183:0/1641959109 >> v1:172.21.15.183:6789/0 conn(0x7f0f2415e6d0 legacy=0x7f0f2415eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:35.899 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.897+0000 7f0f2901d640 1 -- 172.21.15.183:0/1641959109 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2012012068 0 0) 0x7f0f10003520 con 0x7f0f2415e6d0 2024-03-06T07:21:35.899 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.897+0000 7f0f2a01f640 1 -- 172.21.15.183:0/1641959109 >> v1:172.21.15.83:6789/0 conn(0x7f0f240b42f0 legacy=0x7f0f24070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:35.899 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.897+0000 7f0f2a01f640 1 -- 172.21.15.183:0/1641959109 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f240cae80 con 0x7f0f240701a0 2024-03-06T07:21:35.900 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.898+0000 7f0f2a01f640 1 --2- 172.21.15.183:0/1641959109 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0f240701a0 0x7f0f2406da60 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0f14009720 tx=0x7f0f1402b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebf2180d964da08f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:35.901 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.898+0000 7f0f2901d640 1 -- 172.21.15.183:0/1641959109 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f14038070 con 0x7f0f240701a0 2024-03-06T07:21:35.901 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.898+0000 7f0f2901d640 1 -- 172.21.15.183:0/1641959109 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f14031520 con 0x7f0f240701a0 2024-03-06T07:21:35.901 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.898+0000 7f0f2901d640 1 -- 172.21.15.183:0/1641959109 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f1c0019f0 con 0x7f0f240701a0 2024-03-06T07:21:35.901 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.899+0000 7f0f2c283640 1 -- 172.21.15.183:0/1641959109 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0f240701a0 msgr2=0x7f0f2406da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:35.901 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.899+0000 7f0f2c283640 1 --2- 172.21.15.183:0/1641959109 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0f240701a0 0x7f0f2406da60 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0f14009720 tx=0x7f0f1402b7c0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.901 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.899+0000 7f0f2c283640 1 -- 172.21.15.183:0/1641959109 shutdown_connections 2024-03-06T07:21:35.901 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.899+0000 7f0f2c283640 1 --2- 172.21.15.183:0/1641959109 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0f240701a0 0x7f0f2406da60 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.901 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.899+0000 7f0f2c283640 1 -- 172.21.15.183:0/1641959109 >> 172.21.15.183:0/1641959109 conn(0x7f0f240b3800 msgr2=0x7f0f240b3be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:35.901 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.899+0000 7f0f2c283640 1 -- 172.21.15.183:0/1641959109 shutdown_connections 2024-03-06T07:21:35.901 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.899+0000 7f0f2c283640 1 -- 172.21.15.183:0/1641959109 wait complete. 2024-03-06T07:21:35.904 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.900+0000 7f0f2c283640 1 Processor -- start 2024-03-06T07:21:35.904 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.900+0000 7f0f2c283640 1 -- start start 2024-03-06T07:21:35.904 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.901+0000 7f0f2c283640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0f240701a0 0x7f0f24147ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:35.904 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.901+0000 7f0f2c283640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0f240b42f0 0x7f0f2414a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:35.904 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.901+0000 7f0f2c283640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f2415e6d0 0x7f0f2414ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:35.904 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.901+0000 7f0f2c283640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f240cec30 con 0x7f0f240701a0 2024-03-06T07:21:35.904 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.901+0000 7f0f2c283640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f240ce950 con 0x7f0f2415e6d0 2024-03-06T07:21:35.904 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.901+0000 7f0f2c283640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f240cef10 con 0x7f0f240b42f0 2024-03-06T07:21:35.904 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.901+0000 7f0f2a820640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f2415e6d0 0x7f0f2414ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:35.904 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.901+0000 7f0f2981e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0f240b42f0 0x7f0f2414a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:35.905 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.901+0000 7f0f2a820640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f2415e6d0 0x7f0f2414ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:59086/0 (socket says 172.21.15.183:59086) 2024-03-06T07:21:35.905 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.901+0000 7f0f2a820640 1 -- 172.21.15.183:0/2262122233 learned_addr learned my addr 172.21.15.183:0/2262122233 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-03-06T07:21:35.905 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.901+0000 7f0f2981e640 1 --2- 172.21.15.183:0/2262122233 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0f240b42f0 0x7f0f2414a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.183:51198/0 (socket says 172.21.15.183:51198) 2024-03-06T07:21:35.905 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.901+0000 7f0f2a01f640 1 --2- 172.21.15.183:0/2262122233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0f240701a0 0x7f0f24147ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:35.905 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.901+0000 7f0f2a820640 1 -- 172.21.15.183:0/2262122233 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0f240b42f0 msgr2=0x7f0f2414a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:35.905 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.901+0000 7f0f2a820640 1 --2- 172.21.15.183:0/2262122233 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0f240b42f0 0x7f0f2414a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.905 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.901+0000 7f0f2a820640 1 -- 172.21.15.183:0/2262122233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0f240701a0 msgr2=0x7f0f24147ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:35.905 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.901+0000 7f0f2a820640 1 --2- 172.21.15.183:0/2262122233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0f240701a0 0x7f0f24147ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.905 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.901+0000 7f0f2a820640 1 -- 172.21.15.183:0/2262122233 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f24157520 con 0x7f0f2415e6d0 2024-03-06T07:21:35.905 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.901+0000 7f0f2a820640 1 --2- 172.21.15.183:0/2262122233 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f2415e6d0 0x7f0f2414ad90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c00ca70 tx=0x7f0f1c00cf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:35.905 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.901+0000 7f0f1affd640 1 -- 172.21.15.183:0/2262122233 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f1c018070 con 0x7f0f2415e6d0 2024-03-06T07:21:35.905 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.901+0000 7f0f1affd640 1 -- 172.21.15.183:0/2262122233 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f1c00a6a0 con 0x7f0f2415e6d0 2024-03-06T07:21:35.905 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.901+0000 7f0f2c283640 1 -- 172.21.15.183:0/2262122233 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f24157800 con 0x7f0f2415e6d0 2024-03-06T07:21:35.906 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.901+0000 7f0f1affd640 1 -- 172.21.15.183:0/2262122233 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f1c00aa90 con 0x7f0f2415e6d0 2024-03-06T07:21:35.906 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.901+0000 7f0f2c283640 1 -- 172.21.15.183:0/2262122233 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f24148b40 con 0x7f0f2415e6d0 2024-03-06T07:21:35.906 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.902+0000 7f0f1affd640 1 -- 172.21.15.183:0/2262122233 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f0f1c01b070 con 0x7f0f2415e6d0 2024-03-06T07:21:35.906 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.902+0000 7f0f1affd640 1 -- 172.21.15.183:0/2262122233 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f0f1c012980 con 0x7f0f2415e6d0 2024-03-06T07:21:35.906 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.902+0000 7f0f2c283640 1 -- 172.21.15.183:0/2262122233 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f2409fe60 con 0x7f0f2415e6d0 2024-03-06T07:21:35.906 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.904+0000 7f0f1affd640 1 -- 172.21.15.183:0/2262122233 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f0f1c00ad20 con 0x7f0f2415e6d0 2024-03-06T07:21:35.976 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.974+0000 7f0f2c283640 1 -- 172.21.15.183:0/2262122233 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd new", "uuid": "0b171206-23f4-4d38-b977-4a88e15eebb1", "id": 5} v 0) v1 -- 0x7f0f24068ff0 con 0x7f0f2415e6d0 2024-03-06T07:21:35.981 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.980+0000 7f0f1affd640 1 -- 172.21.15.183:0/2262122233 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0b171206-23f4-4d38-b977-4a88e15eebb1", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f0f1c012bd0 con 0x7f0f2415e6d0 2024-03-06T07:21:35.981 INFO:teuthology.orchestra.run.smithi183.stdout:5 2024-03-06T07:21:35.983 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.981+0000 7f0f2c283640 1 -- 172.21.15.183:0/2262122233 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f2415e6d0 msgr2=0x7f0f2414ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:35.983 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.981+0000 7f0f2c283640 1 --2- 172.21.15.183:0/2262122233 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f2415e6d0 0x7f0f2414ad90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c00ca70 tx=0x7f0f1c00cf30 comp rx=0 tx=0).stop 2024-03-06T07:21:35.983 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.981+0000 7f0f2c283640 1 -- 172.21.15.183:0/2262122233 shutdown_connections 2024-03-06T07:21:35.983 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.981+0000 7f0f2c283640 1 --2- 172.21.15.183:0/2262122233 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f2415e6d0 0x7f0f2414ad90 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.983 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.981+0000 7f0f2c283640 1 --2- 172.21.15.183:0/2262122233 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0f240b42f0 0x7f0f2414a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.983 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.981+0000 7f0f2c283640 1 --2- 172.21.15.183:0/2262122233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0f240701a0 0x7f0f24147ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.983 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.981+0000 7f0f2c283640 1 -- 172.21.15.183:0/2262122233 >> 172.21.15.183:0/2262122233 conn(0x7f0f240b3800 msgr2=0x7f0f2406cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:35.984 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.981+0000 7f0f2c283640 1 -- 172.21.15.183:0/2262122233 shutdown_connections 2024-03-06T07:21:35.984 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:35.981+0000 7f0f2c283640 1 -- 172.21.15.183:0/2262122233 wait complete. 2024-03-06T07:21:35.993 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new 5573c862-78b3-4c8a-8443-9288d6872d31 6 2024-03-06T07:21:36.133 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.132+0000 7f891ab65640 1 Processor -- start 2024-03-06T07:21:36.134 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.132+0000 7f891ab65640 1 -- start start 2024-03-06T07:21:36.135 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.133+0000 7f891ab65640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f891415c050 0x7f891406f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.135 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.133+0000 7f891ab65640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89140cd4b0 con 0x7f89140d0af0 2024-03-06T07:21:36.135 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.133+0000 7f891ab65640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89140cd790 con 0x7f891406fb40 2024-03-06T07:21:36.135 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.133+0000 7f891ab65640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f891406ffb0 con 0x7f891415c050 2024-03-06T07:21:36.135 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.133+0000 7f8918901640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f891415c050 0x7f891406f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:36.135 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.133+0000 7f8913fff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f891406fb40 0x7f89140cc760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:36972/0 (socket says 172.21.15.183:36972) 2024-03-06T07:21:36.135 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.133+0000 7f8913fff640 1 -- 172.21.15.183:0/3701523131 learned_addr learned my addr 172.21.15.183:0/3701523131 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-06T07:21:36.135 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.133+0000 7f8918901640 1 -- 172.21.15.183:0/3701523131 >> v1:172.21.15.183:6789/0 conn(0x7f891406fb40 legacy=0x7f89140cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.135 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.133+0000 7f8918901640 1 -- 172.21.15.183:0/3701523131 >> v1:172.21.15.83:6789/0 conn(0x7f89140d0af0 legacy=0x7f8914165ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.135 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.133+0000 7f8918901640 1 -- 172.21.15.183:0/3701523131 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89140cda70 con 0x7f891415c050 2024-03-06T07:21:36.137 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.134+0000 7f8918901640 1 --2- 172.21.15.183:0/3701523131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f891415c050 0x7f891406f600 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f89080097a0 tx=0x7f890802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ecf5a4a63269711 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:36.137 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.134+0000 7f89137fe640 1 -- 172.21.15.183:0/3701523131 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8908035020 con 0x7f891415c050 2024-03-06T07:21:36.137 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.134+0000 7f89137fe640 1 -- 172.21.15.183:0/3701523131 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8908031830 con 0x7f891415c050 2024-03-06T07:21:36.137 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.134+0000 7f89137fe640 1 -- 172.21.15.183:0/3701523131 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8908031c00 con 0x7f891415c050 2024-03-06T07:21:36.137 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.135+0000 7f891ab65640 1 -- 172.21.15.183:0/3701523131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f891415c050 msgr2=0x7f891406f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.137 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.135+0000 7f891ab65640 1 --2- 172.21.15.183:0/3701523131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f891415c050 0x7f891406f600 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f89080097a0 tx=0x7f890802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.137 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.135+0000 7f891ab65640 1 -- 172.21.15.183:0/3701523131 shutdown_connections 2024-03-06T07:21:36.137 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.135+0000 7f891ab65640 1 --2- 172.21.15.183:0/3701523131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f891415c050 0x7f891406f600 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.137 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.135+0000 7f891ab65640 1 -- 172.21.15.183:0/3701523131 >> 172.21.15.183:0/3701523131 conn(0x7f89140b3800 msgr2=0x7f89140b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:36.137 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.135+0000 7f891ab65640 1 -- 172.21.15.183:0/3701523131 shutdown_connections 2024-03-06T07:21:36.137 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.135+0000 7f891ab65640 1 -- 172.21.15.183:0/3701523131 wait complete. 2024-03-06T07:21:36.138 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.136+0000 7f891ab65640 1 Processor -- start 2024-03-06T07:21:36.138 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.136+0000 7f891ab65640 1 -- start start 2024-03-06T07:21:36.140 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.137+0000 7f891ab65640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f891406fb40 0x7f89140b7430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.140 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.137+0000 7f891ab65640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f89140d0af0 0x7f89140af200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.140 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.137+0000 7f891ab65640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f891415c050 0x7f89140cae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.140 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.137+0000 7f891ab65640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89140ccff0 con 0x7f891415c050 2024-03-06T07:21:36.140 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.137+0000 7f891ab65640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89140cce70 con 0x7f891406fb40 2024-03-06T07:21:36.141 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.137+0000 7f891ab65640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89140cd170 con 0x7f89140d0af0 2024-03-06T07:21:36.141 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.137+0000 7f8913fff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f89140d0af0 0x7f89140af200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:36.141 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.137+0000 7f8918901640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f891406fb40 0x7f89140b7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:36.141 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.137+0000 7f8913fff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f89140d0af0 0x7f89140af200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.183:51208/0 (socket says 172.21.15.183:51208) 2024-03-06T07:21:36.141 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.137+0000 7f8913fff640 1 -- 172.21.15.183:0/1074627754 learned_addr learned my addr 172.21.15.183:0/1074627754 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-03-06T07:21:36.141 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.137+0000 7f8919102640 1 --2- 172.21.15.183:0/1074627754 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f891415c050 0x7f89140cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:36.141 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.137+0000 7f8913fff640 1 -- 172.21.15.183:0/1074627754 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f891406fb40 msgr2=0x7f89140b7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.141 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.137+0000 7f8913fff640 1 --2- 172.21.15.183:0/1074627754 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f891406fb40 0x7f89140b7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.141 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.137+0000 7f8913fff640 1 -- 172.21.15.183:0/1074627754 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f891415c050 msgr2=0x7f89140cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.141 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.137+0000 7f8913fff640 1 --2- 172.21.15.183:0/1074627754 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f891415c050 0x7f89140cae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.141 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.137+0000 7f8913fff640 1 -- 172.21.15.183:0/1074627754 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f891415a0a0 con 0x7f89140d0af0 2024-03-06T07:21:36.141 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.138+0000 7f8913fff640 1 --2- 172.21.15.183:0/1074627754 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f89140d0af0 0x7f89140af200 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f890000c4b0 tx=0x7f890000c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:36.141 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.138+0000 7f89117fa640 1 -- 172.21.15.183:0/1074627754 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8900017070 con 0x7f89140d0af0 2024-03-06T07:21:36.142 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.138+0000 7f89117fa640 1 -- 172.21.15.183:0/1074627754 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8900005ce0 con 0x7f89140d0af0 2024-03-06T07:21:36.142 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.138+0000 7f891ab65640 1 -- 172.21.15.183:0/1074627754 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f891415a380 con 0x7f89140d0af0 2024-03-06T07:21:36.142 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.138+0000 7f89117fa640 1 -- 172.21.15.183:0/1074627754 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8900007270 con 0x7f89140d0af0 2024-03-06T07:21:36.142 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.138+0000 7f891ab65640 1 -- 172.21.15.183:0/1074627754 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f891415a940 con 0x7f89140d0af0 2024-03-06T07:21:36.142 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.138+0000 7f89117fa640 1 -- 172.21.15.183:0/1074627754 <== mon.2 v2:172.21.15.183:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f890001a070 con 0x7f89140d0af0 2024-03-06T07:21:36.142 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.138+0000 7f89117fa640 1 -- 172.21.15.183:0/1074627754 <== mon.2 v2:172.21.15.183:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f890001cc10 con 0x7f89140d0af0 2024-03-06T07:21:36.142 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.139+0000 7f891ab65640 1 -- 172.21.15.183:0/1074627754 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f891409fe60 con 0x7f89140d0af0 2024-03-06T07:21:36.142 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.141+0000 7f89117fa640 1 -- 172.21.15.183:0/1074627754 <== mon.2 v2:172.21.15.183:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f8900007470 con 0x7f89140d0af0 2024-03-06T07:21:36.151 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-06T07:21:36.150+0000 7f1929873040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-06T07:21:36.215 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.213+0000 7f891ab65640 1 -- 172.21.15.183:0/1074627754 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_command({"prefix": "osd new", "uuid": "5573c862-78b3-4c8a-8443-9288d6872d31", "id": 6} v 0) v1 -- 0x7f89140d2920 con 0x7f89140d0af0 2024-03-06T07:21:36.220 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.219+0000 7f89117fa640 1 -- 172.21.15.183:0/1074627754 <== mon.2 v2:172.21.15.183:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5573c862-78b3-4c8a-8443-9288d6872d31", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f890001c440 con 0x7f89140d0af0 2024-03-06T07:21:36.220 INFO:teuthology.orchestra.run.smithi183.stdout:6 2024-03-06T07:21:36.221 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.219+0000 7f891ab65640 1 -- 172.21.15.183:0/1074627754 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f89140d0af0 msgr2=0x7f89140af200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.221 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.219+0000 7f891ab65640 1 --2- 172.21.15.183:0/1074627754 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f89140d0af0 0x7f89140af200 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f890000c4b0 tx=0x7f890000c970 comp rx=0 tx=0).stop 2024-03-06T07:21:36.221 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.220+0000 7f891ab65640 1 -- 172.21.15.183:0/1074627754 shutdown_connections 2024-03-06T07:21:36.221 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.220+0000 7f891ab65640 1 --2- 172.21.15.183:0/1074627754 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f891415c050 0x7f89140cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.222 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.220+0000 7f891ab65640 1 --2- 172.21.15.183:0/1074627754 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f89140d0af0 0x7f89140af200 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.222 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.220+0000 7f891ab65640 1 --2- 172.21.15.183:0/1074627754 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f891406fb40 0x7f89140b7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.222 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.220+0000 7f891ab65640 1 -- 172.21.15.183:0/1074627754 >> 172.21.15.183:0/1074627754 conn(0x7f89140b3800 msgr2=0x7f89140b3be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:36.222 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.220+0000 7f891ab65640 1 -- 172.21.15.183:0/1074627754 shutdown_connections 2024-03-06T07:21:36.222 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.220+0000 7f891ab65640 1 -- 172.21.15.183:0/1074627754 wait complete. 2024-03-06T07:21:36.232 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new 6f176f58-87b8-4b3c-9526-316b7024e6d8 7 2024-03-06T07:21:36.280 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-06T07:21:36.278+0000 7f38bb43e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-06T07:21:36.286 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-06T07:21:36.284+0000 7f1929873040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-06T07:21:36.332 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.330+0000 7f914456b640 1 Processor -- start 2024-03-06T07:21:36.332 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.330+0000 7f914456b640 1 -- start start 2024-03-06T07:21:36.334 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.331+0000 7f914456b640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f913c071a70 0x7f913c071e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.334 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.331+0000 7f914456b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f913c15ad90 0x7f913c0b3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.334 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.331+0000 7f914456b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f913c072390 con 0x7f913c0b4110 2024-03-06T07:21:36.334 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.331+0000 7f914456b640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f913c15c020 con 0x7f913c071a70 2024-03-06T07:21:36.334 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.331+0000 7f914456b640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f913c15b260 con 0x7f913c15ad90 2024-03-06T07:21:36.334 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.331+0000 7f9141b06640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f913c15ad90 0x7f913c0b3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:36.334 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.331+0000 7f9142307640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f913c071a70 0x7f913c071e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:36.334 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.331+0000 7f9142307640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f913c071a70 0x7f913c071e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:59126/0 (socket says 172.21.15.183:59126) 2024-03-06T07:21:36.334 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.331+0000 7f9141b06640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f913c15ad90 0x7f913c0b3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.183:51220/0 (socket says 172.21.15.183:51220) 2024-03-06T07:21:36.334 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.331+0000 7f9142307640 1 -- 172.21.15.183:0/2799494407 learned_addr learned my addr 172.21.15.183:0/2799494407 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-03-06T07:21:36.335 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.332+0000 7f9141b06640 1 -- 172.21.15.183:0/2799494407 >> v2:172.21.15.183:3300/0 conn(0x7f913c071a70 msgr2=0x7f913c071e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:36.335 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.332+0000 7f9141b06640 1 --2- 172.21.15.183:0/2799494407 >> v2:172.21.15.183:3300/0 conn(0x7f913c071a70 0x7f913c071e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.335 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.332+0000 7f9141b06640 1 -- 172.21.15.183:0/2799494407 >> v1:172.21.15.83:6789/0 conn(0x7f913c0b4110 legacy=0x7f913c06da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.335 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.332+0000 7f9141b06640 1 -- 172.21.15.183:0/2799494407 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f913c15b540 con 0x7f913c15ad90 2024-03-06T07:21:36.335 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.332+0000 7f9142307640 1 --2- 172.21.15.183:0/2799494407 >> v2:172.21.15.183:3300/0 conn(0x7f913c071a70 0x7f913c071e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:36.335 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.332+0000 7f9141b06640 1 --2- 172.21.15.183:0/2799494407 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f913c15ad90 0x7f913c0b3bd0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f91380097a0 tx=0x7f913802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d700aa5c7d620144 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:36.335 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.333+0000 7f9141305640 1 -- 172.21.15.183:0/2799494407 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91380315a0 con 0x7f913c15ad90 2024-03-06T07:21:36.335 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.333+0000 7f9141305640 1 -- 172.21.15.183:0/2799494407 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9138037070 con 0x7f913c15ad90 2024-03-06T07:21:36.335 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.333+0000 7f9141305640 1 -- 172.21.15.183:0/2799494407 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9138031960 con 0x7f913c15ad90 2024-03-06T07:21:36.338 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.333+0000 7f914456b640 1 -- 172.21.15.183:0/2799494407 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f913c15ad90 msgr2=0x7f913c0b3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.338 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.333+0000 7f914456b640 1 --2- 172.21.15.183:0/2799494407 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f913c15ad90 0x7f913c0b3bd0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f91380097a0 tx=0x7f913802b840 comp rx=0 tx=0).stop 2024-03-06T07:21:36.339 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.333+0000 7f914456b640 1 -- 172.21.15.183:0/2799494407 shutdown_connections 2024-03-06T07:21:36.339 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.333+0000 7f914456b640 1 --2- 172.21.15.183:0/2799494407 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f913c15ad90 0x7f913c0b3bd0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.339 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.333+0000 7f914456b640 1 --2- 172.21.15.183:0/2799494407 >> v2:172.21.15.183:3300/0 conn(0x7f913c071a70 0x7f913c071e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.339 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.333+0000 7f914456b640 1 -- 172.21.15.183:0/2799494407 >> 172.21.15.183:0/2799494407 conn(0x7f913c0b2660 msgr2=0x7f913c0b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:36.339 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.333+0000 7f914456b640 1 -- 172.21.15.183:0/2799494407 shutdown_connections 2024-03-06T07:21:36.339 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.333+0000 7f914456b640 1 -- 172.21.15.183:0/2799494407 wait complete. 2024-03-06T07:21:36.339 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.334+0000 7f914456b640 1 Processor -- start 2024-03-06T07:21:36.339 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.334+0000 7f914456b640 1 -- start start 2024-03-06T07:21:36.339 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.335+0000 7f914456b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f913c071a70 0x7f913c1f59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.339 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.335+0000 7f914456b640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f913c0b4110 0x7f913c153a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.339 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.335+0000 7f914456b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f913c15ad90 0x7f913c152bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.339 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.335+0000 7f914456b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f913c15d220 con 0x7f913c071a70 2024-03-06T07:21:36.340 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.335+0000 7f914456b640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f913c15bb00 con 0x7f913c0b4110 2024-03-06T07:21:36.340 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.335+0000 7f914456b640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f913c15d3a0 con 0x7f913c15ad90 2024-03-06T07:21:36.340 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.335+0000 7f9141b06640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f913c0b4110 0x7f913c153a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:36.340 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.335+0000 7f9142b08640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f913c15ad90 0x7f913c152bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:36.340 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.335+0000 7f9141b06640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f913c0b4110 0x7f913c153a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:59140/0 (socket says 172.21.15.183:59140) 2024-03-06T07:21:36.340 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.335+0000 7f9141b06640 1 -- 172.21.15.183:0/777404534 learned_addr learned my addr 172.21.15.183:0/777404534 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-03-06T07:21:36.340 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.335+0000 7f9142307640 1 --2- 172.21.15.183:0/777404534 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f913c071a70 0x7f913c1f59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:36.340 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.335+0000 7f9142b08640 1 -- 172.21.15.183:0/777404534 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f913c0b4110 msgr2=0x7f913c153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.340 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.335+0000 7f9142b08640 1 --2- 172.21.15.183:0/777404534 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f913c0b4110 0x7f913c153a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.340 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.335+0000 7f9142b08640 1 -- 172.21.15.183:0/777404534 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f913c071a70 msgr2=0x7f913c1f59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.340 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.335+0000 7f9142b08640 1 --2- 172.21.15.183:0/777404534 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f913c071a70 0x7f913c1f59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.341 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.335+0000 7f9142b08640 1 -- 172.21.15.183:0/777404534 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f913c153f70 con 0x7f913c15ad90 2024-03-06T07:21:36.341 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.336+0000 7f9142b08640 1 --2- 172.21.15.183:0/777404534 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f913c15ad90 0x7f913c152bb0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f913400c690 tx=0x7f913400cb50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:36.341 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.336+0000 7f9132ffd640 1 -- 172.21.15.183:0/777404534 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9134003b70 con 0x7f913c15ad90 2024-03-06T07:21:36.341 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.336+0000 7f9132ffd640 1 -- 172.21.15.183:0/777404534 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9134003d10 con 0x7f913c15ad90 2024-03-06T07:21:36.341 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.336+0000 7f914456b640 1 -- 172.21.15.183:0/777404534 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f913c1f1ae0 con 0x7f913c15ad90 2024-03-06T07:21:36.341 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.336+0000 7f9132ffd640 1 -- 172.21.15.183:0/777404534 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9134005310 con 0x7f913c15ad90 2024-03-06T07:21:36.341 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.336+0000 7f914456b640 1 -- 172.21.15.183:0/777404534 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f913c1f20a0 con 0x7f913c15ad90 2024-03-06T07:21:36.341 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.336+0000 7f9132ffd640 1 -- 172.21.15.183:0/777404534 <== mon.2 v2:172.21.15.183:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f91340054b0 con 0x7f913c15ad90 2024-03-06T07:21:36.341 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.336+0000 7f9132ffd640 1 -- 172.21.15.183:0/777404534 <== mon.2 v2:172.21.15.183:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f9134012ad0 con 0x7f913c15ad90 2024-03-06T07:21:36.341 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.337+0000 7f914456b640 1 -- 172.21.15.183:0/777404534 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f913c073390 con 0x7f913c15ad90 2024-03-06T07:21:36.342 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.339+0000 7f9132ffd640 1 -- 172.21.15.183:0/777404534 <== mon.2 v2:172.21.15.183:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f9134012db0 con 0x7f913c15ad90 2024-03-06T07:21:36.393 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-06T07:21:36.391+0000 7f1929873040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-06T07:21:36.411 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.410+0000 7f914456b640 1 -- 172.21.15.183:0/777404534 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_command({"prefix": "osd new", "uuid": "6f176f58-87b8-4b3c-9526-316b7024e6d8", "id": 7} v 0) v1 -- 0x7f913c152230 con 0x7f913c15ad90 2024-03-06T07:21:36.418 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.416+0000 7f9132ffd640 1 -- 172.21.15.183:0/777404534 <== mon.2 v2:172.21.15.183:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6f176f58-87b8-4b3c-9526-316b7024e6d8", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9134012470 con 0x7f913c15ad90 2024-03-06T07:21:36.419 INFO:teuthology.orchestra.run.smithi183.stdout:7 2024-03-06T07:21:36.419 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.417+0000 7f914456b640 1 -- 172.21.15.183:0/777404534 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f913c15ad90 msgr2=0x7f913c152bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.419 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.417+0000 7f914456b640 1 --2- 172.21.15.183:0/777404534 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f913c15ad90 0x7f913c152bb0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f913400c690 tx=0x7f913400cb50 comp rx=0 tx=0).stop 2024-03-06T07:21:36.419 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.417+0000 7f914456b640 1 -- 172.21.15.183:0/777404534 shutdown_connections 2024-03-06T07:21:36.420 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.417+0000 7f914456b640 1 --2- 172.21.15.183:0/777404534 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f913c15ad90 0x7f913c152bb0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.420 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.417+0000 7f914456b640 1 --2- 172.21.15.183:0/777404534 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f913c0b4110 0x7f913c153a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.420 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.417+0000 7f914456b640 1 --2- 172.21.15.183:0/777404534 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f913c071a70 0x7f913c1f59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.420 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.417+0000 7f914456b640 1 -- 172.21.15.183:0/777404534 >> 172.21.15.183:0/777404534 conn(0x7f913c0b2660 msgr2=0x7f913c15ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:36.420 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-06T07:21:36.418+0000 7f38bb43e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-06T07:21:36.421 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.418+0000 7f914456b640 1 -- 172.21.15.183:0/777404534 shutdown_connections 2024-03-06T07:21:36.421 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:21:36.418+0000 7f914456b640 1 -- 172.21.15.183:0/777404534 wait complete. 2024-03-06T07:21:36.429 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-06T07:21:36.430 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-06T07:21:36.432 INFO:tasks.ceph.osd.0:Started 2024-03-06T07:21:36.432 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-06T07:21:36.432 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-06T07:21:36.434 INFO:tasks.ceph.osd.1:Started 2024-03-06T07:21:36.434 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-06T07:21:36.434 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-06T07:21:36.436 INFO:tasks.ceph.osd.2:Started 2024-03-06T07:21:36.437 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-06T07:21:36.437 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-06T07:21:36.439 INFO:tasks.ceph.osd.3:Started 2024-03-06T07:21:36.439 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-06T07:21:36.440 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-06T07:21:36.440 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-06T07:21:36.439+0000 7f1929873040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-06T07:21:36.442 INFO:tasks.ceph.osd.4:Started 2024-03-06T07:21:36.442 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-06T07:21:36.442 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-06T07:21:36.444 INFO:tasks.ceph.osd.5:Started 2024-03-06T07:21:36.444 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-06T07:21:36.444 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-06T07:21:36.446 INFO:tasks.ceph.osd.6:Started 2024-03-06T07:21:36.446 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-06T07:21:36.447 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-06T07:21:36.449 INFO:tasks.ceph.osd.7:Started 2024-03-06T07:21:36.449 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:36.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.564+0000 7fa4bc555640 1 Processor -- start 2024-03-06T07:21:36.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.564+0000 7fa4bc555640 1 -- start start 2024-03-06T07:21:36.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.565+0000 7fa4bc555640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fa4b415dfe0 0x7fa4b40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.565+0000 7fa4bc555640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4b40b3430 con 0x7fa4b40d29c0 2024-03-06T07:21:36.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.565+0000 7fa4bc555640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4b40b3710 con 0x7fa4b40cf830 2024-03-06T07:21:36.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.565+0000 7fa4bc555640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4b40b31e0 con 0x7fa4b415dfe0 2024-03-06T07:21:36.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.565+0000 7fa4baaf2640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fa4b40d29c0 0x7fa4b40b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:46588/0 (socket says 172.21.15.83:46588) 2024-03-06T07:21:36.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.565+0000 7fa4baaf2640 1 -- 172.21.15.83:0/4181566091 learned_addr learned my addr 172.21.15.83:0/4181566091 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:21:36.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.565+0000 7fa4ba2f1640 1 --2- 172.21.15.83:0/4181566091 >> v2:172.21.15.183:3300/0 conn(0x7fa4b415dfe0 0x7fa4b40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:36.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.565+0000 7fa4b92ef640 1 -- 172.21.15.83:0/4181566091 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2166998537 0 0) 0x7fa4b40b3430 con 0x7fa4b40d29c0 2024-03-06T07:21:36.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.565+0000 7fa4b92ef640 1 -- 172.21.15.83:0/4181566091 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4a0003520 con 0x7fa4b40d29c0 2024-03-06T07:21:36.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.565+0000 7fa4b92ef640 1 -- 172.21.15.83:0/4181566091 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4271932567 0 0) 0x7fa4b40b3710 con 0x7fa4b40cf830 2024-03-06T07:21:36.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.565+0000 7fa4b92ef640 1 -- 172.21.15.83:0/4181566091 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4b40b3430 con 0x7fa4b40cf830 2024-03-06T07:21:36.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.565+0000 7fa4ba2f1640 1 -- 172.21.15.83:0/4181566091 >> v1:172.21.15.183:6789/0 conn(0x7fa4b40cf830 legacy=0x7fa4b40d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.565+0000 7fa4ba2f1640 1 -- 172.21.15.83:0/4181566091 >> v1:172.21.15.83:6789/0 conn(0x7fa4b40d29c0 legacy=0x7fa4b40b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.565+0000 7fa4ba2f1640 1 -- 172.21.15.83:0/4181566091 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b40b39f0 con 0x7fa4b415dfe0 2024-03-06T07:21:36.569 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-06T07:21:36.566+0000 7f38bb43e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-06T07:21:36.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.566+0000 7fa4ba2f1640 1 --2- 172.21.15.83:0/4181566091 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa4b415dfe0 0x7fa4b40cf2f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa4b40b3360 tx=0x7fa4ac02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8353be1fd0abf78 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:36.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.566+0000 7fa4b92ef640 1 -- 172.21.15.83:0/4181566091 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac034070 con 0x7fa4b415dfe0 2024-03-06T07:21:36.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.566+0000 7fa4b92ef640 1 -- 172.21.15.83:0/4181566091 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4ac030610 con 0x7fa4b415dfe0 2024-03-06T07:21:36.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.566+0000 7fa4b92ef640 1 -- 172.21.15.83:0/4181566091 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac0309e0 con 0x7fa4b415dfe0 2024-03-06T07:21:36.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.567+0000 7fa4bc555640 1 -- 172.21.15.83:0/4181566091 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa4b415dfe0 msgr2=0x7fa4b40cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.567+0000 7fa4bc555640 1 --2- 172.21.15.83:0/4181566091 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa4b415dfe0 0x7fa4b40cf2f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa4b40b3360 tx=0x7fa4ac02b960 comp rx=0 tx=0).stop 2024-03-06T07:21:36.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.567+0000 7fa4bc555640 1 -- 172.21.15.83:0/4181566091 shutdown_connections 2024-03-06T07:21:36.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.567+0000 7fa4bc555640 1 --2- 172.21.15.83:0/4181566091 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa4b415dfe0 0x7fa4b40cf2f0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.567+0000 7fa4bc555640 1 -- 172.21.15.83:0/4181566091 >> 172.21.15.83:0/4181566091 conn(0x7fa4b40d3cd0 msgr2=0x7fa4b40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:36.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.567+0000 7fa4bc555640 1 -- 172.21.15.83:0/4181566091 shutdown_connections 2024-03-06T07:21:36.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.567+0000 7fa4bc555640 1 -- 172.21.15.83:0/4181566091 wait complete. 2024-03-06T07:21:36.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.568+0000 7fa4bc555640 1 Processor -- start 2024-03-06T07:21:36.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.568+0000 7fa4bc555640 1 -- start start 2024-03-06T07:21:36.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.569+0000 7fa4bc555640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa4b40cf830 0x7fa4b40cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.569+0000 7fa4bc555640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa4b40d29c0 0x7fa4b406bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.569+0000 7fa4bc555640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa4b415dfe0 0x7fa4b414ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.569+0000 7fa4bc555640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b415e3c0 con 0x7fa4b40cf830 2024-03-06T07:21:36.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.569+0000 7fa4bc555640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b40b3d60 con 0x7fa4b415dfe0 2024-03-06T07:21:36.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.569+0000 7fa4bc555640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b415e540 con 0x7fa4b40d29c0 2024-03-06T07:21:36.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.569+0000 7fa4ba2f1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa4b40cf830 0x7fa4b40cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:36.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.569+0000 7fa4ba2f1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa4b40cf830 0x7fa4b40cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:42070/0 (socket says 172.21.15.83:42070) 2024-03-06T07:21:36.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.569+0000 7fa4baaf2640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa4b415dfe0 0x7fa4b414ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:36.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.569+0000 7fa4ba2f1640 1 -- 172.21.15.83:0/816463575 learned_addr learned my addr 172.21.15.83:0/816463575 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:21:36.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.569+0000 7fa4b9af0640 1 --2- 172.21.15.83:0/816463575 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa4b40d29c0 0x7fa4b406bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:36.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.569+0000 7fa4baaf2640 1 -- 172.21.15.83:0/816463575 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa4b40d29c0 msgr2=0x7fa4b406bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.569+0000 7fa4baaf2640 1 --2- 172.21.15.83:0/816463575 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa4b40d29c0 0x7fa4b406bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.569+0000 7fa4baaf2640 1 -- 172.21.15.83:0/816463575 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa4b40cf830 msgr2=0x7fa4b40cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.569+0000 7fa4baaf2640 1 --2- 172.21.15.83:0/816463575 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa4b40cf830 0x7fa4b40cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.569+0000 7fa4baaf2640 1 -- 172.21.15.83:0/816463575 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b41f75a0 con 0x7fa4b415dfe0 2024-03-06T07:21:36.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.569+0000 7fa4b9af0640 1 --2- 172.21.15.83:0/816463575 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa4b40d29c0 0x7fa4b406bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:36.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.570+0000 7fa4baaf2640 1 --2- 172.21.15.83:0/816463575 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa4b415dfe0 0x7fa4b414ec80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa4b40cff60 tx=0x7fa4b000cf00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:36.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.570+0000 7fa4aaffd640 1 -- 172.21.15.83:0/816463575 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b0017070 con 0x7fa4b415dfe0 2024-03-06T07:21:36.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.570+0000 7fa4aaffd640 1 -- 172.21.15.83:0/816463575 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4b000abd0 con 0x7fa4b415dfe0 2024-03-06T07:21:36.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.570+0000 7fa4aaffd640 1 -- 172.21.15.83:0/816463575 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b0012400 con 0x7fa4b415dfe0 2024-03-06T07:21:36.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.570+0000 7fa4bc555640 1 -- 172.21.15.83:0/816463575 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4b41f7880 con 0x7fa4b415dfe0 2024-03-06T07:21:36.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.570+0000 7fa4bc555640 1 -- 172.21.15.83:0/816463575 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4b41f7e40 con 0x7fa4b415dfe0 2024-03-06T07:21:36.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.570+0000 7fa4aaffd640 1 -- 172.21.15.83:0/816463575 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fa4b001a070 con 0x7fa4b415dfe0 2024-03-06T07:21:36.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.571+0000 7fa4aaffd640 1 -- 172.21.15.83:0/816463575 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fa4b00125a0 con 0x7fa4b415dfe0 2024-03-06T07:21:36.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.571+0000 7fa4bc555640 1 -- 172.21.15.83:0/816463575 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4b404f330 con 0x7fa4b415dfe0 2024-03-06T07:21:36.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.573+0000 7fa4aaffd640 1 -- 172.21.15.83:0/816463575 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fa4b00128c0 con 0x7fa4b415dfe0 2024-03-06T07:21:36.582 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-06T07:21:36.580+0000 7f1929873040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:21:36.613 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-06T07:21:36.612+0000 7f38bb43e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-06T07:21:36.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.634+0000 7fa4bc555640 1 -- 172.21.15.83:0/816463575 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa4b40d1ee0 con 0x7fa4b415dfe0 2024-03-06T07:21:36.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.634+0000 7fa4aaffd640 1 -- 172.21.15.83:0/816463575 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fa4b0012aa0 con 0x7fa4b415dfe0 2024-03-06T07:21:36.636 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:21:36.636 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":10,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:21:36.411792+0000","last_up_change":"0.000000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:36.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.635+0000 7fa4bc555640 1 -- 172.21.15.83:0/816463575 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa4b415dfe0 msgr2=0x7fa4b414ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.635+0000 7fa4bc555640 1 --2- 172.21.15.83:0/816463575 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa4b415dfe0 0x7fa4b414ec80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa4b40cff60 tx=0x7fa4b000cf00 comp rx=0 tx=0).stop 2024-03-06T07:21:36.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.635+0000 7fa4bc555640 1 -- 172.21.15.83:0/816463575 shutdown_connections 2024-03-06T07:21:36.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.635+0000 7fa4bc555640 1 --2- 172.21.15.83:0/816463575 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa4b415dfe0 0x7fa4b414ec80 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.635+0000 7fa4bc555640 1 --2- 172.21.15.83:0/816463575 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa4b40d29c0 0x7fa4b406bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.635+0000 7fa4bc555640 1 --2- 172.21.15.83:0/816463575 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa4b40cf830 0x7fa4b40cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.635+0000 7fa4bc555640 1 -- 172.21.15.83:0/816463575 >> 172.21.15.83:0/816463575 conn(0x7fa4b40d3cd0 msgr2=0x7fa4b40b5b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:36.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.635+0000 7fa4bc555640 1 -- 172.21.15.83:0/816463575 shutdown_connections 2024-03-06T07:21:36.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:36.635+0000 7fa4bc555640 1 -- 172.21.15.83:0/816463575 wait complete. 2024-03-06T07:21:36.647 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-06T07:21:36.648 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-06T07:21:36.680 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-06T07:21:36.678+0000 7f1929873040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-06T07:21:36.725 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-06T07:21:36.723+0000 7f38bb43e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:21:36.820 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-06T07:21:36.818+0000 7f38bb43e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-06T07:21:36.836 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-06T07:21:36.834+0000 7f1929873040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:21:36.878 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-06T07:21:36.877+0000 7f1929873040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-06T07:21:36.972 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-06T07:21:36.971+0000 7f38bb43e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:21:37.016 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-06T07:21:37.012+0000 7f38bb43e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-06T07:21:37.032 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-06T07:21:37.030+0000 7f1929873040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-06T07:21:37.049 INFO:tasks.ceph.osd.2.smithi083.stderr:2024-03-06T07:21:37.047+0000 7f3c4b2a4640 -1 Falling back to public interface 2024-03-06T07:21:37.052 INFO:tasks.ceph.osd.6.smithi183.stderr:2024-03-06T07:21:37.050+0000 7fc10fff6640 -1 Falling back to public interface 2024-03-06T07:21:37.052 INFO:tasks.ceph.osd.5.smithi183.stderr:2024-03-06T07:21:37.051+0000 7f7007bac640 -1 Falling back to public interface 2024-03-06T07:21:37.064 INFO:tasks.ceph.osd.4.smithi183.stderr:2024-03-06T07:21:37.061+0000 7fc646e0f640 -1 Falling back to public interface 2024-03-06T07:21:37.064 INFO:tasks.ceph.osd.7.smithi183.stderr:2024-03-06T07:21:37.062+0000 7f4015a8b640 -1 Falling back to public interface 2024-03-06T07:21:37.067 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-06T07:21:37.065+0000 7f20b1d73640 -1 Falling back to public interface 2024-03-06T07:21:37.068 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-06T07:21:37.066+0000 7f82c9026640 -1 Falling back to public interface 2024-03-06T07:21:37.068 INFO:tasks.ceph.osd.1.smithi083.stderr:2024-03-06T07:21:37.066+0000 7ffa37a24640 -1 Falling back to public interface 2024-03-06T07:21:37.070 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-06T07:21:37.069+0000 7f82c7a33640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6826/0: (98) Address already in use 2024-03-06T07:21:37.172 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-06T07:21:37.170+0000 7f38bb43e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-06T07:21:37.452 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:37.542 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.540+0000 7f3171bc7640 1 Processor -- start 2024-03-06T07:21:37.543 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.541+0000 7f3171bc7640 1 -- start start 2024-03-06T07:21:37.544 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.541+0000 7f3171bc7640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f316c06c800 0x7f316c0b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:37.544 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.541+0000 7f3171bc7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f316c067020 con 0x7f316c0d0d80 2024-03-06T07:21:37.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.541+0000 7f3171bc7640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f316c067300 con 0x7f316c0b1570 2024-03-06T07:21:37.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.541+0000 7f3171bc7640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f316c06b650 con 0x7f316c06c800 2024-03-06T07:21:37.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.541+0000 7f316bfff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f316c0d0d80 0x7f316c0cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45160/0 (socket says 172.21.15.83:45160) 2024-03-06T07:21:37.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.541+0000 7f316bfff640 1 -- 172.21.15.83:0/1619688244 learned_addr learned my addr 172.21.15.83:0/1619688244 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:21:37.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.542+0000 7f316affd640 1 --2- 172.21.15.83:0/1619688244 >> v2:172.21.15.183:3300/0 conn(0x7f316c06c800 0x7f316c0b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:37.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.542+0000 7f316a7fc640 1 -- 172.21.15.83:0/1619688244 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2978901026 0 0) 0x7f316c067020 con 0x7f316c0d0d80 2024-03-06T07:21:37.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.542+0000 7f316a7fc640 1 -- 172.21.15.83:0/1619688244 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3154003520 con 0x7f316c0d0d80 2024-03-06T07:21:37.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.542+0000 7f316a7fc640 1 -- 172.21.15.83:0/1619688244 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1249397345 0 0) 0x7f3154003520 con 0x7f316c0d0d80 2024-03-06T07:21:37.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.542+0000 7f316affd640 1 -- 172.21.15.83:0/1619688244 >> v1:172.21.15.183:6789/0 conn(0x7f316c0b1570 legacy=0x7f316c06c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:37.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.542+0000 7f316affd640 1 -- 172.21.15.83:0/1619688244 >> v1:172.21.15.83:6789/0 conn(0x7f316c0d0d80 legacy=0x7f316c0cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:37.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.542+0000 7f316affd640 1 -- 172.21.15.83:0/1619688244 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f316c06b090 con 0x7f316c06c800 2024-03-06T07:21:37.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.543+0000 7f316affd640 1 --2- 172.21.15.83:0/1619688244 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f316c06c800 0x7f316c0b3e90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f31600097a0 tx=0x7f316002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e3390667dcd87f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:37.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.543+0000 7f316a7fc640 1 -- 172.21.15.83:0/1619688244 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3160038070 con 0x7f316c06c800 2024-03-06T07:21:37.546 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.543+0000 7f316a7fc640 1 -- 172.21.15.83:0/1619688244 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3160031500 con 0x7f316c06c800 2024-03-06T07:21:37.546 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.543+0000 7f316a7fc640 1 -- 172.21.15.83:0/1619688244 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315c0019f0 con 0x7f316c06c800 2024-03-06T07:21:37.549 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.543+0000 7f3171bc7640 1 -- 172.21.15.83:0/1619688244 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f316c06c800 msgr2=0x7f316c0b3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:37.549 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.543+0000 7f3171bc7640 1 --2- 172.21.15.83:0/1619688244 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f316c06c800 0x7f316c0b3e90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f31600097a0 tx=0x7f316002b820 comp rx=0 tx=0).stop 2024-03-06T07:21:37.549 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.543+0000 7f3171bc7640 1 -- 172.21.15.83:0/1619688244 shutdown_connections 2024-03-06T07:21:37.549 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.543+0000 7f3171bc7640 1 --2- 172.21.15.83:0/1619688244 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f316c06c800 0x7f316c0b3e90 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:37.549 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.543+0000 7f3171bc7640 1 -- 172.21.15.83:0/1619688244 >> 172.21.15.83:0/1619688244 conn(0x7f316c0b3370 msgr2=0x7f316c0b3780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:37.549 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.543+0000 7f3171bc7640 1 -- 172.21.15.83:0/1619688244 shutdown_connections 2024-03-06T07:21:37.549 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.543+0000 7f3171bc7640 1 -- 172.21.15.83:0/1619688244 wait complete. 2024-03-06T07:21:37.549 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.544+0000 7f3171bc7640 1 Processor -- start 2024-03-06T07:21:37.549 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.545+0000 7f3171bc7640 1 -- start start 2024-03-06T07:21:37.549 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.545+0000 7f3171bc7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f316c06c800 0x7f316c0c6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:37.549 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.545+0000 7f3171bc7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f316c0b1570 0x7f316c099090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:37.549 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.545+0000 7f3171bc7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f316c0d0d80 0x7f316c1f20d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:37.550 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.545+0000 7f3171bc7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316c06b990 con 0x7f316c0b1570 2024-03-06T07:21:37.550 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.545+0000 7f3171bc7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316c0cf6d0 con 0x7f316c0d0d80 2024-03-06T07:21:37.550 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.545+0000 7f3171bc7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f316c06bb10 con 0x7f316c06c800 2024-03-06T07:21:37.550 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.545+0000 7f316affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f316c0b1570 0x7f316c099090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:37.550 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.545+0000 7f316affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f316c0b1570 0x7f316c099090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46616/0 (socket says 172.21.15.83:46616) 2024-03-06T07:21:37.550 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.545+0000 7f316affd640 1 -- 172.21.15.83:0/1947927382 learned_addr learned my addr 172.21.15.83:0/1947927382 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:21:37.550 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.546+0000 7f316bfff640 1 --2- 172.21.15.83:0/1947927382 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f316c0d0d80 0x7f316c1f20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:37.550 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.546+0000 7f316b7fe640 1 --2- 172.21.15.83:0/1947927382 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f316c06c800 0x7f316c0c6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:37.550 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.546+0000 7f316affd640 1 -- 172.21.15.83:0/1947927382 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f316c06c800 msgr2=0x7f316c0c6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:37.550 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.546+0000 7f316affd640 1 --2- 172.21.15.83:0/1947927382 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f316c06c800 0x7f316c0c6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:37.550 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.546+0000 7f316affd640 1 -- 172.21.15.83:0/1947927382 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f316c0d0d80 msgr2=0x7f316c1f20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:37.550 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.546+0000 7f316affd640 1 --2- 172.21.15.83:0/1947927382 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f316c0d0d80 0x7f316c1f20d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:37.550 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.546+0000 7f316affd640 1 -- 172.21.15.83:0/1947927382 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f316c0c4ac0 con 0x7f316c0b1570 2024-03-06T07:21:37.550 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.546+0000 7f316affd640 1 --2- 172.21.15.83:0/1947927382 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f316c0b1570 0x7f316c099090 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3160036040 tx=0x7f31600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:37.551 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.546+0000 7f3153fff640 1 -- 172.21.15.83:0/1947927382 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3160043070 con 0x7f316c0b1570 2024-03-06T07:21:37.551 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.546+0000 7f3153fff640 1 -- 172.21.15.83:0/1947927382 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f316003f870 con 0x7f316c0b1570 2024-03-06T07:21:37.551 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.546+0000 7f3171bc7640 1 -- 172.21.15.83:0/1947927382 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f316c155930 con 0x7f316c0b1570 2024-03-06T07:21:37.551 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.546+0000 7f3153fff640 1 -- 172.21.15.83:0/1947927382 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3160038040 con 0x7f316c0b1570 2024-03-06T07:21:37.551 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.546+0000 7f3171bc7640 1 -- 172.21.15.83:0/1947927382 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f316c155370 con 0x7f316c0b1570 2024-03-06T07:21:37.551 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.546+0000 7f3153fff640 1 -- 172.21.15.83:0/1947927382 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f3160046070 con 0x7f316c0b1570 2024-03-06T07:21:37.551 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.546+0000 7f3153fff640 1 -- 172.21.15.83:0/1947927382 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f316003db00 con 0x7f316c0b1570 2024-03-06T07:21:37.551 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.547+0000 7f3171bc7640 1 -- 172.21.15.83:0/1947927382 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f316c04f330 con 0x7f316c0b1570 2024-03-06T07:21:37.551 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.549+0000 7f3153fff640 1 -- 172.21.15.83:0/1947927382 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f3160030070 con 0x7f316c0b1570 2024-03-06T07:21:37.612 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.610+0000 7f3171bc7640 1 -- 172.21.15.83:0/1947927382 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f316c0cd960 con 0x7f316c0b1570 2024-03-06T07:21:37.613 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.611+0000 7f3153fff640 1 -- 172.21.15.83:0/1947927382 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f316003dda0 con 0x7f316c0b1570 2024-03-06T07:21:37.613 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-06T07:21:37.613 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:21:36.411792+0000","last_up_change":"0.000000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:37.614 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.612+0000 7f3171bc7640 1 -- 172.21.15.83:0/1947927382 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f316c0b1570 msgr2=0x7f316c099090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:37.614 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.612+0000 7f3171bc7640 1 --2- 172.21.15.83:0/1947927382 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f316c0b1570 0x7f316c099090 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3160036040 tx=0x7f31600046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:37.614 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.612+0000 7f3171bc7640 1 -- 172.21.15.83:0/1947927382 shutdown_connections 2024-03-06T07:21:37.614 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.612+0000 7f3171bc7640 1 --2- 172.21.15.83:0/1947927382 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f316c0d0d80 0x7f316c1f20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:37.615 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.612+0000 7f3171bc7640 1 --2- 172.21.15.83:0/1947927382 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f316c0b1570 0x7f316c099090 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:37.615 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.612+0000 7f3171bc7640 1 --2- 172.21.15.83:0/1947927382 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f316c06c800 0x7f316c0c6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:37.615 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.612+0000 7f3171bc7640 1 -- 172.21.15.83:0/1947927382 >> 172.21.15.83:0/1947927382 conn(0x7f316c0b3370 msgr2=0x7f316c1594e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:37.615 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.612+0000 7f3171bc7640 1 -- 172.21.15.83:0/1947927382 shutdown_connections 2024-03-06T07:21:37.615 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:37.612+0000 7f3171bc7640 1 -- 172.21.15.83:0/1947927382 wait complete. 2024-03-06T07:21:37.623 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-06T07:21:37.855 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-06T07:21:37.854+0000 7f1929873040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-06T07:21:37.991 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-06T07:21:37.989+0000 7f38bb43e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-06T07:21:38.130 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-06T07:21:38.128+0000 7f1929873040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-06T07:21:38.183 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-06T07:21:38.181+0000 7f1929873040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-06T07:21:38.267 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-06T07:21:38.265+0000 7f38bb43e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-06T07:21:38.298 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-06T07:21:38.296+0000 7f1929873040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-06T07:21:38.318 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-06T07:21:38.316+0000 7f38bb43e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-06T07:21:38.436 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-06T07:21:38.434+0000 7f38bb43e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-06T07:21:39.327 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-06T07:21:39.325+0000 7f18cace2640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-06T07:21:39.327 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-06T07:21:39.325+0000 7f18cace2640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-06T07:21:40.532 INFO:tasks.ceph.osd.2.smithi083.stderr:2024-03-06T07:21:40.531+0000 7f3c4b2a4640 -1 osd.2 0 log_to_monitors true 2024-03-06T07:21:40.564 INFO:tasks.ceph.osd.5.smithi183.stderr:2024-03-06T07:21:40.561+0000 7f7007bac640 -1 osd.5 0 log_to_monitors true 2024-03-06T07:21:40.577 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-06T07:21:40.575+0000 7f82c9026640 -1 osd.3 0 log_to_monitors true 2024-03-06T07:21:40.591 INFO:tasks.ceph.osd.6.smithi183.stderr:2024-03-06T07:21:40.589+0000 7fc10fff6640 -1 osd.6 0 log_to_monitors true 2024-03-06T07:21:40.591 INFO:tasks.ceph.osd.4.smithi183.stderr:2024-03-06T07:21:40.589+0000 7fc646e0f640 -1 osd.4 0 log_to_monitors true 2024-03-06T07:21:40.599 INFO:tasks.ceph.osd.1.smithi083.stderr:2024-03-06T07:21:40.597+0000 7ffa37a24640 -1 osd.1 0 log_to_monitors true 2024-03-06T07:21:40.599 INFO:tasks.ceph.osd.7.smithi183.stderr:2024-03-06T07:21:40.597+0000 7f4015a8b640 -1 osd.7 0 log_to_monitors true 2024-03-06T07:21:40.605 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-06T07:21:40.603+0000 7f20b1d73640 -1 osd.0 0 log_to_monitors true 2024-03-06T07:21:44.228 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-06T07:21:44.226+0000 7f20ad4e6640 -1 osd.0 0 waiting for initial osdmap 2024-03-06T07:21:44.248 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-06T07:21:44.246+0000 7f20a5b18640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:21:44.332 INFO:tasks.ceph.osd.2.smithi083.stderr:2024-03-06T07:21:44.330+0000 7f3c46a17640 -1 osd.2 0 waiting for initial osdmap 2024-03-06T07:21:44.343 INFO:tasks.ceph.osd.2.smithi083.stderr:2024-03-06T07:21:44.342+0000 7f3c3f049640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:21:44.372 INFO:tasks.ceph.osd.4.smithi183.stderr:2024-03-06T07:21:44.370+0000 7fc641ec2640 -1 osd.4 0 waiting for initial osdmap 2024-03-06T07:21:44.378 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-06T07:21:44.377+0000 7f82c41f7640 -1 osd.3 0 waiting for initial osdmap 2024-03-06T07:21:44.383 INFO:tasks.ceph.osd.4.smithi183.stderr:2024-03-06T07:21:44.381+0000 7fc63a4f4640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:21:44.388 INFO:tasks.ceph.osd.1.smithi083.stderr:2024-03-06T07:21:44.387+0000 7ffa33197640 -1 osd.1 0 waiting for initial osdmap 2024-03-06T07:21:44.390 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-06T07:21:44.389+0000 7f82bc829640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:21:44.394 INFO:tasks.ceph.osd.5.smithi183.stderr:2024-03-06T07:21:44.392+0000 7f700331f640 -1 osd.5 0 waiting for initial osdmap 2024-03-06T07:21:44.404 INFO:tasks.ceph.osd.1.smithi083.stderr:2024-03-06T07:21:44.403+0000 7ffa2b7c9640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:21:44.416 INFO:tasks.ceph.osd.5.smithi183.stderr:2024-03-06T07:21:44.414+0000 7f6ffb951640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:21:44.420 INFO:tasks.ceph.osd.6.smithi183.stderr:2024-03-06T07:21:44.417+0000 7fc10b769640 -1 osd.6 0 waiting for initial osdmap 2024-03-06T07:21:44.421 INFO:tasks.ceph.osd.7.smithi183.stderr:2024-03-06T07:21:44.419+0000 7f40111fe640 -1 osd.7 0 waiting for initial osdmap 2024-03-06T07:21:44.429 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:44.430 INFO:tasks.ceph.osd.6.smithi183.stderr:2024-03-06T07:21:44.427+0000 7fc103d9b640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:21:44.432 INFO:tasks.ceph.osd.7.smithi183.stderr:2024-03-06T07:21:44.430+0000 7f4009830640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:21:44.519 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.516+0000 7fdf76064640 1 Processor -- start 2024-03-06T07:21:44.519 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.517+0000 7fdf76064640 1 -- start start 2024-03-06T07:21:44.521 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.517+0000 7fdf76064640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdf7006ca10 0x7fdf7006cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.521 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.517+0000 7fdf76064640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fdf700d25c0 0x7fdf700ccd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.521 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.517+0000 7fdf76064640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf7006d330 con 0x7fdf700cd2b0 2024-03-06T07:21:44.521 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.517+0000 7fdf76064640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf700b1130 con 0x7fdf700d25c0 2024-03-06T07:21:44.522 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.517+0000 7fdf76064640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7006c440 con 0x7fdf7006ca10 2024-03-06T07:21:44.522 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.517+0000 7fdf6ffff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fdf700cd2b0 0x7fdf700cd720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45164/0 (socket says 172.21.15.83:45164) 2024-03-06T07:21:44.522 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.517+0000 7fdf6ffff640 1 -- 172.21.15.83:0/2431885431 learned_addr learned my addr 172.21.15.83:0/2431885431 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:21:44.522 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.517+0000 7fdf6f7fe640 1 --2- 172.21.15.83:0/2431885431 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdf7006ca10 0x7fdf7006cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.522 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.518+0000 7fdf6effd640 1 --2- 172.21.15.83:0/2431885431 >> v2:172.21.15.183:3300/0 conn(0x7fdf700d25c0 0x7fdf700ccd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.522 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.518+0000 7fdf6e7fc640 1 -- 172.21.15.83:0/2431885431 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4064499980 0 0) 0x7fdf7006d330 con 0x7fdf700cd2b0 2024-03-06T07:21:44.522 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.518+0000 7fdf6e7fc640 1 -- 172.21.15.83:0/2431885431 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf58003520 con 0x7fdf700cd2b0 2024-03-06T07:21:44.522 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.518+0000 7fdf6f7fe640 1 -- 172.21.15.83:0/2431885431 >> v2:172.21.15.183:3300/0 conn(0x7fdf700d25c0 msgr2=0x7fdf700ccd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:44.522 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.518+0000 7fdf6f7fe640 1 --2- 172.21.15.83:0/2431885431 >> v2:172.21.15.183:3300/0 conn(0x7fdf700d25c0 0x7fdf700ccd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.522 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.518+0000 7fdf6f7fe640 1 -- 172.21.15.83:0/2431885431 >> v1:172.21.15.83:6789/0 conn(0x7fdf700cd2b0 legacy=0x7fdf700cd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:44.522 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.518+0000 7fdf6f7fe640 1 -- 172.21.15.83:0/2431885431 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf700cc290 con 0x7fdf7006ca10 2024-03-06T07:21:44.522 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.519+0000 7fdf6f7fe640 1 --2- 172.21.15.83:0/2431885431 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdf7006ca10 0x7fdf7006cdf0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fdf600097a0 tx=0x7fdf6002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0397d923be3ba56 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:44.523 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.519+0000 7fdf6e7fc640 1 -- 172.21.15.83:0/2431885431 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf60035020 con 0x7fdf7006ca10 2024-03-06T07:21:44.523 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.519+0000 7fdf6e7fc640 1 -- 172.21.15.83:0/2431885431 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf60031830 con 0x7fdf7006ca10 2024-03-06T07:21:44.523 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.519+0000 7fdf6e7fc640 1 -- 172.21.15.83:0/2431885431 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf60031be0 con 0x7fdf7006ca10 2024-03-06T07:21:44.523 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.519+0000 7fdf76064640 1 -- 172.21.15.83:0/2431885431 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdf7006ca10 msgr2=0x7fdf7006cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:44.523 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.519+0000 7fdf76064640 1 --2- 172.21.15.83:0/2431885431 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdf7006ca10 0x7fdf7006cdf0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fdf600097a0 tx=0x7fdf6002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.523 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.519+0000 7fdf76064640 1 -- 172.21.15.83:0/2431885431 shutdown_connections 2024-03-06T07:21:44.523 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.519+0000 7fdf76064640 1 --2- 172.21.15.83:0/2431885431 >> v2:172.21.15.183:3300/0 conn(0x7fdf700d25c0 0x7fdf700ccd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.523 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.519+0000 7fdf76064640 1 --2- 172.21.15.83:0/2431885431 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdf7006ca10 0x7fdf7006cdf0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.523 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.519+0000 7fdf76064640 1 -- 172.21.15.83:0/2431885431 >> 172.21.15.83:0/2431885431 conn(0x7fdf700d1aa0 msgr2=0x7fdf700d1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:44.523 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.519+0000 7fdf76064640 1 -- 172.21.15.83:0/2431885431 shutdown_connections 2024-03-06T07:21:44.523 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.519+0000 7fdf76064640 1 -- 172.21.15.83:0/2431885431 wait complete. 2024-03-06T07:21:44.526 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.520+0000 7fdf76064640 1 Processor -- start 2024-03-06T07:21:44.527 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.521+0000 7fdf76064640 1 -- start start 2024-03-06T07:21:44.527 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.521+0000 7fdf76064640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf7006ca10 0x7fdf700b59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.527 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.521+0000 7fdf76064640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdf700cd2b0 0x7fdf700c7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.527 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.521+0000 7fdf76064640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf700d25c0 0x7fdf700af7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.527 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.521+0000 7fdf76064640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf70068070 con 0x7fdf700d25c0 2024-03-06T07:21:44.527 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.521+0000 7fdf76064640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf700cc850 con 0x7fdf7006ca10 2024-03-06T07:21:44.527 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.521+0000 7fdf76064640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf700cde30 con 0x7fdf700cd2b0 2024-03-06T07:21:44.527 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.521+0000 7fdf6ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf700d25c0 0x7fdf700af7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.527 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.521+0000 7fdf6ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf700d25c0 0x7fdf700af7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46728/0 (socket says 172.21.15.83:46728) 2024-03-06T07:21:44.527 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.521+0000 7fdf6ffff640 1 -- 172.21.15.83:0/2835760115 learned_addr learned my addr 172.21.15.83:0/2835760115 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:21:44.527 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.521+0000 7fdf6effd640 1 --2- 172.21.15.83:0/2835760115 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdf700cd2b0 0x7fdf700c7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.527 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.521+0000 7fdf6f7fe640 1 --2- 172.21.15.83:0/2835760115 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf7006ca10 0x7fdf700b59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.528 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.522+0000 7fdf6ffff640 1 -- 172.21.15.83:0/2835760115 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdf700cd2b0 msgr2=0x7fdf700c7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:44.528 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.522+0000 7fdf6ffff640 1 --2- 172.21.15.83:0/2835760115 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdf700cd2b0 0x7fdf700c7b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.528 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.522+0000 7fdf6ffff640 1 -- 172.21.15.83:0/2835760115 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf7006ca10 msgr2=0x7fdf700b59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:44.528 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.522+0000 7fdf6ffff640 1 --2- 172.21.15.83:0/2835760115 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf7006ca10 0x7fdf700b59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.528 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.522+0000 7fdf6ffff640 1 -- 172.21.15.83:0/2835760115 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf700b4620 con 0x7fdf700d25c0 2024-03-06T07:21:44.528 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.522+0000 7fdf6ffff640 1 --2- 172.21.15.83:0/2835760115 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf700d25c0 0x7fdf700af7c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdf6800cb10 tx=0x7fdf6800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:44.528 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.522+0000 7fdf57fff640 1 -- 172.21.15.83:0/2835760115 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf68017070 con 0x7fdf700d25c0 2024-03-06T07:21:44.528 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.522+0000 7fdf57fff640 1 -- 172.21.15.83:0/2835760115 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf6800ac70 con 0x7fdf700d25c0 2024-03-06T07:21:44.528 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.522+0000 7fdf76064640 1 -- 172.21.15.83:0/2835760115 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf700b4900 con 0x7fdf700d25c0 2024-03-06T07:21:44.528 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.522+0000 7fdf57fff640 1 -- 172.21.15.83:0/2835760115 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf68012620 con 0x7fdf700d25c0 2024-03-06T07:21:44.528 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.522+0000 7fdf76064640 1 -- 172.21.15.83:0/2835760115 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf700b4ec0 con 0x7fdf700d25c0 2024-03-06T07:21:44.528 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.523+0000 7fdf76064640 1 -- 172.21.15.83:0/2835760115 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf7004f330 con 0x7fdf700d25c0 2024-03-06T07:21:44.529 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.523+0000 7fdf57fff640 1 -- 172.21.15.83:0/2835760115 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7fdf6801a070 con 0x7fdf700d25c0 2024-03-06T07:21:44.529 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.524+0000 7fdf57fff640 1 --2- 172.21.15.83:0/2835760115 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdf5c06b440 0x7fdf5c06d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.529 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.524+0000 7fdf6f7fe640 1 --2- 172.21.15.83:0/2835760115 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdf5c06b440 0x7fdf5c06d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.529 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.524+0000 7fdf57fff640 1 -- 172.21.15.83:0/2835760115 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7fdf68089050 con 0x7fdf700d25c0 2024-03-06T07:21:44.529 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.524+0000 7fdf6f7fe640 1 --2- 172.21.15.83:0/2835760115 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdf5c06b440 0x7fdf5c06d900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fdf60036000 tx=0x7fdf6003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:44.530 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.529+0000 7fdf57fff640 1 -- 172.21.15.83:0/2835760115 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdf6800ae20 con 0x7fdf700d25c0 2024-03-06T07:21:44.705 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.703+0000 7fdf76064640 1 -- 172.21.15.83:0/2835760115 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdf700cbe40 con 0x7fdf700d25c0 2024-03-06T07:21:44.705 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.704+0000 7fdf57fff640 1 -- 172.21.15.83:0/2835760115 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fdf680586f0 con 0x7fdf700d25c0 2024-03-06T07:21:44.706 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-06T07:21:44.706 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":12,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:21:42.327171+0000","last_up_change":"0.000000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:44.708 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.705+0000 7fdf76064640 1 -- 172.21.15.83:0/2835760115 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdf5c06b440 msgr2=0x7fdf5c06d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:44.708 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.705+0000 7fdf76064640 1 --2- 172.21.15.83:0/2835760115 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdf5c06b440 0x7fdf5c06d900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fdf60036000 tx=0x7fdf6003c040 comp rx=0 tx=0).stop 2024-03-06T07:21:44.708 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.705+0000 7fdf76064640 1 -- 172.21.15.83:0/2835760115 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf700d25c0 msgr2=0x7fdf700af7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:44.708 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.705+0000 7fdf76064640 1 --2- 172.21.15.83:0/2835760115 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf700d25c0 0x7fdf700af7c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdf6800cb10 tx=0x7fdf6800a060 comp rx=0 tx=0).stop 2024-03-06T07:21:44.708 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.706+0000 7fdf76064640 1 -- 172.21.15.83:0/2835760115 shutdown_connections 2024-03-06T07:21:44.708 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.706+0000 7fdf76064640 1 --2- 172.21.15.83:0/2835760115 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdf5c06b440 0x7fdf5c06d900 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.708 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.706+0000 7fdf76064640 1 --2- 172.21.15.83:0/2835760115 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf700d25c0 0x7fdf700af7c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.708 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.706+0000 7fdf76064640 1 --2- 172.21.15.83:0/2835760115 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdf700cd2b0 0x7fdf700c7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.709 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.706+0000 7fdf76064640 1 --2- 172.21.15.83:0/2835760115 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf7006ca10 0x7fdf700b59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.709 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.706+0000 7fdf76064640 1 -- 172.21.15.83:0/2835760115 >> 172.21.15.83:0/2835760115 conn(0x7fdf700d1aa0 msgr2=0x7fdf7015b060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:44.709 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.706+0000 7fdf76064640 1 -- 172.21.15.83:0/2835760115 shutdown_connections 2024-03-06T07:21:44.709 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:44.706+0000 7fdf76064640 1 -- 172.21.15.83:0/2835760115 wait complete. 2024-03-06T07:21:44.717 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-06T07:21:51.524 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:51.624 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.622+0000 7f5fb8d3e640 1 Processor -- start 2024-03-06T07:21:51.624 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.622+0000 7f5fb8d3e640 1 -- start start 2024-03-06T07:21:51.628 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.623+0000 7f5fb8d3e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f5fb415e2b0 0x7f5fb40d1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.628 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.623+0000 7f5fb8d3e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fb40d4a50 con 0x7f5fb40b2c50 2024-03-06T07:21:51.628 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.623+0000 7f5fb8d3e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fb40cf550 con 0x7f5fb40d29d0 2024-03-06T07:21:51.628 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.623+0000 7f5fb8d3e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fb40d4800 con 0x7f5fb415e2b0 2024-03-06T07:21:51.628 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.623+0000 7f5fb1d9b640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f5fb40b2c50 0x7f5fb40d22c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52320/0 (socket says 172.21.15.83:52320) 2024-03-06T07:21:51.628 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.623+0000 7f5fb1d9b640 1 -- 172.21.15.83:0/1866530622 learned_addr learned my addr 172.21.15.83:0/1866530622 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:21:51.628 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.624+0000 7f5fb259c640 1 --2- 172.21.15.83:0/1866530622 >> v2:172.21.15.183:3300/0 conn(0x7f5fb415e2b0 0x7f5fb40d1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.629 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.624+0000 7f5fb159a640 1 -- 172.21.15.83:0/1866530622 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4197088430 0 0) 0x7f5fb40d4a50 con 0x7f5fb40b2c50 2024-03-06T07:21:51.629 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.624+0000 7f5fb159a640 1 -- 172.21.15.83:0/1866530622 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f9c003520 con 0x7f5fb40b2c50 2024-03-06T07:21:51.629 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.624+0000 7f5fb159a640 1 -- 172.21.15.83:0/1866530622 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 373531605 0 0) 0x7f5f9c003520 con 0x7f5fb40b2c50 2024-03-06T07:21:51.629 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.624+0000 7f5fb159a640 1 -- 172.21.15.83:0/1866530622 >> v2:172.21.15.183:3300/0 conn(0x7f5fb415e2b0 msgr2=0x7f5fb40d1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:51.629 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.624+0000 7f5fb159a640 1 --2- 172.21.15.83:0/1866530622 >> v2:172.21.15.183:3300/0 conn(0x7f5fb415e2b0 0x7f5fb40d1ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.629 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.624+0000 7f5fb159a640 1 -- 172.21.15.83:0/1866530622 >> v1:172.21.15.183:6789/0 conn(0x7f5fb40d29d0 legacy=0x7f5fb40b0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.629 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.624+0000 7f5fb159a640 1 -- 172.21.15.83:0/1866530622 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fb40cf830 con 0x7f5fb40b2c50 2024-03-06T07:21:51.629 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.624+0000 7f5fb159a640 1 -- 172.21.15.83:0/1866530622 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f5fa8001670 con 0x7f5fb40b2c50 2024-03-06T07:21:51.629 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.624+0000 7f5fb159a640 1 -- 172.21.15.83:0/1866530622 >> v1:172.21.15.83:6789/0 conn(0x7f5fb40b2c50 legacy=0x7f5fb40d22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.629 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.624+0000 7f5fb159a640 1 --2- 172.21.15.83:0/1866530622 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f9c0039b0 0x7f5f9c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.629 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.624+0000 7f5fb159a640 1 --2- 172.21.15.83:0/1866530622 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f9c004800 0x7f5f9c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.629 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.624+0000 7f5fb159a640 1 --2- 172.21.15.83:0/1866530622 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f9c0051c0 0x7f5f9c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.629 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.624+0000 7f5fb159a640 1 -- 172.21.15.83:0/1866530622 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb40d4800 con 0x7f5f9c0039b0 2024-03-06T07:21:51.630 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.624+0000 7f5fb159a640 1 -- 172.21.15.83:0/1866530622 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9c00c160 con 0x7f5f9c004800 2024-03-06T07:21:51.630 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.624+0000 7f5fb159a640 1 -- 172.21.15.83:0/1866530622 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9c00c360 con 0x7f5f9c0051c0 2024-03-06T07:21:51.630 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.624+0000 7f5fb259c640 1 --2- 172.21.15.83:0/1866530622 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f9c0039b0 0x7f5f9c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.630 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.624+0000 7f5fb1d9b640 1 --2- 172.21.15.83:0/1866530622 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f9c004800 0x7f5f9c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.630 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.624+0000 7f5fb2d9d640 1 --2- 172.21.15.83:0/1866530622 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f9c0051c0 0x7f5f9c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.630 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.624+0000 7f5fb259c640 1 -- 172.21.15.83:0/1866530622 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f9c0051c0 msgr2=0x7f5f9c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:51.630 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.624+0000 7f5fb259c640 1 --2- 172.21.15.83:0/1866530622 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f9c0051c0 0x7f5f9c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.630 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.624+0000 7f5fb259c640 1 -- 172.21.15.83:0/1866530622 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f9c004800 msgr2=0x7f5f9c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:51.630 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.624+0000 7f5fb259c640 1 --2- 172.21.15.83:0/1866530622 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f9c004800 0x7f5f9c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.630 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.624+0000 7f5fb259c640 1 -- 172.21.15.83:0/1866530622 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f9c003520 con 0x7f5f9c0039b0 2024-03-06T07:21:51.630 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.624+0000 7f5fb2d9d640 1 --2- 172.21.15.83:0/1866530622 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f9c0051c0 0x7f5f9c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:51.630 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.624+0000 7f5fb1d9b640 1 --2- 172.21.15.83:0/1866530622 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f9c004800 0x7f5f9c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:51.630 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.624+0000 7f5fb2d9d640 1 -- 172.21.15.83:0/1866530622 reap_dead start 2024-03-06T07:21:51.630 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.625+0000 7f5fb259c640 1 --2- 172.21.15.83:0/1866530622 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f9c0039b0 0x7f5f9c003dd0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4009fc0 tx=0x7f5f9c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0c39f5e331bedad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.631 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.625+0000 7f5fb159a640 1 -- 172.21.15.83:0/1866530622 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa4039070 con 0x7f5f9c0039b0 2024-03-06T07:21:51.631 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.625+0000 7f5fb159a640 1 -- 172.21.15.83:0/1866530622 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fa402cc20 con 0x7f5f9c0039b0 2024-03-06T07:21:51.631 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.625+0000 7f5fb159a640 1 -- 172.21.15.83:0/1866530622 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa4009c10 con 0x7f5f9c0039b0 2024-03-06T07:21:51.631 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.626+0000 7f5fb8d3e640 1 -- 172.21.15.83:0/1866530622 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f9c0039b0 msgr2=0x7f5f9c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.631 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.626+0000 7f5fb8d3e640 1 --2- 172.21.15.83:0/1866530622 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f9c0039b0 0x7f5f9c003dd0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4009fc0 tx=0x7f5f9c005830 comp rx=0 tx=0).stop 2024-03-06T07:21:51.631 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.626+0000 7f5fb8d3e640 1 -- 172.21.15.83:0/1866530622 shutdown_connections 2024-03-06T07:21:51.631 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.626+0000 7f5fb8d3e640 1 --2- 172.21.15.83:0/1866530622 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f9c0039b0 0x7f5f9c003dd0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.631 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.626+0000 7f5fb8d3e640 1 -- 172.21.15.83:0/1866530622 >> 172.21.15.83:0/1866530622 conn(0x7f5fb40d3ce0 msgr2=0x7f5fb40d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:51.631 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.626+0000 7f5fb8d3e640 1 -- 172.21.15.83:0/1866530622 shutdown_connections 2024-03-06T07:21:51.631 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.626+0000 7f5fb8d3e640 1 -- 172.21.15.83:0/1866530622 wait complete. 2024-03-06T07:21:51.635 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.627+0000 7f5fb8d3e640 1 Processor -- start 2024-03-06T07:21:51.635 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.627+0000 7f5fb8d3e640 1 -- start start 2024-03-06T07:21:51.635 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.628+0000 7f5fb8d3e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f9c0039b0 0x7f5fb415cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.635 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.628+0000 7f5fb8d3e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5fb40c75b0 0x7f5fb40c9000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.635 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.628+0000 7f5fb8d3e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5fb4161300 0x7f5fb40c9540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.635 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.628+0000 7f5fb8d3e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb40d1ac0 con 0x7f5f9c0039b0 2024-03-06T07:21:51.635 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.628+0000 7f5fb8d3e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb40cf300 con 0x7f5fb4161300 2024-03-06T07:21:51.635 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.628+0000 7f5fb8d3e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb40d1c40 con 0x7f5fb40c75b0 2024-03-06T07:21:51.635 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.628+0000 7f5fb259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f9c0039b0 0x7f5fb415cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.635 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.628+0000 7f5fb259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f9c0039b0 0x7f5fb415cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51370/0 (socket says 172.21.15.83:51370) 2024-03-06T07:21:51.635 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.628+0000 7f5fb259c640 1 -- 172.21.15.83:0/3723669056 learned_addr learned my addr 172.21.15.83:0/3723669056 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:21:51.635 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.629+0000 7f5fb1d9b640 1 --2- 172.21.15.83:0/3723669056 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5fb40c75b0 0x7f5fb40c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.636 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.629+0000 7f5fb2d9d640 1 --2- 172.21.15.83:0/3723669056 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5fb4161300 0x7f5fb40c9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.636 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.629+0000 7f5fb259c640 1 -- 172.21.15.83:0/3723669056 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5fb40c75b0 msgr2=0x7f5fb40c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.636 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.629+0000 7f5fb259c640 1 --2- 172.21.15.83:0/3723669056 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5fb40c75b0 0x7f5fb40c9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.636 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.629+0000 7f5fb259c640 1 -- 172.21.15.83:0/3723669056 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5fb4161300 msgr2=0x7f5fb40c9540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.636 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.629+0000 7f5fb259c640 1 --2- 172.21.15.83:0/3723669056 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5fb4161300 0x7f5fb40c9540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.636 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.629+0000 7f5fb259c640 1 -- 172.21.15.83:0/3723669056 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fb415f930 con 0x7f5f9c0039b0 2024-03-06T07:21:51.636 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.629+0000 7f5fb2d9d640 1 --2- 172.21.15.83:0/3723669056 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5fb4161300 0x7f5fb40c9540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:51.636 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.629+0000 7f5fb1d9b640 1 --2- 172.21.15.83:0/3723669056 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5fb40c75b0 0x7f5fb40c9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:51.636 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.629+0000 7f5fb259c640 1 --2- 172.21.15.83:0/3723669056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f9c0039b0 0x7f5fb415cd30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5fa400e230 tx=0x7f5fa4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.636 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.629+0000 7f5fa2ffd640 1 -- 172.21.15.83:0/3723669056 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa4039050 con 0x7f5f9c0039b0 2024-03-06T07:21:51.637 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.629+0000 7f5fa2ffd640 1 -- 172.21.15.83:0/3723669056 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fa403a040 con 0x7f5f9c0039b0 2024-03-06T07:21:51.637 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.629+0000 7f5fb8d3e640 1 -- 172.21.15.83:0/3723669056 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fb415fc10 con 0x7f5f9c0039b0 2024-03-06T07:21:51.637 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.629+0000 7f5fa2ffd640 1 -- 172.21.15.83:0/3723669056 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa4009c10 con 0x7f5f9c0039b0 2024-03-06T07:21:51.637 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.629+0000 7f5fb8d3e640 1 -- 172.21.15.83:0/3723669056 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fb41601d0 con 0x7f5f9c0039b0 2024-03-06T07:21:51.637 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.630+0000 7f5fb8d3e640 1 -- 172.21.15.83:0/3723669056 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fb404f330 con 0x7f5f9c0039b0 2024-03-06T07:21:51.637 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.631+0000 7f5fa2ffd640 1 -- 172.21.15.83:0/3723669056 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5fa4003bb0 con 0x7f5f9c0039b0 2024-03-06T07:21:51.637 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.631+0000 7f5fa2ffd640 1 --2- 172.21.15.83:0/3723669056 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5f9406b4d0 0x7f5f9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.637 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.631+0000 7f5fa2ffd640 1 -- 172.21.15.83:0/3723669056 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5fa40ac760 con 0x7f5f9c0039b0 2024-03-06T07:21:51.637 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.631+0000 7f5fb1d9b640 1 --2- 172.21.15.83:0/3723669056 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5f9406b4d0 0x7f5f9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.637 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.631+0000 7f5fb1d9b640 1 --2- 172.21.15.83:0/3723669056 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5f9406b4d0 0x7f5f9406d990 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5fb40c79f0 tx=0x7f5fa80014e0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.638 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.635+0000 7f5fa2ffd640 1 -- 172.21.15.83:0/3723669056 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5fa407bae0 con 0x7f5f9c0039b0 2024-03-06T07:21:51.809 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.807+0000 7f5fb8d3e640 1 -- 172.21.15.83:0/3723669056 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5fb406c8d0 con 0x7f5f9c0039b0 2024-03-06T07:21:51.810 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.808+0000 7f5fa2ffd640 1 -- 172.21.15.83:0/3723669056 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12896 (secure 0 0 0) 0x7f5fa407b3e0 con 0x7f5f9c0039b0 2024-03-06T07:21:51.810 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-06T07:21:51.810 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":16,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:21:48.365486+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:51.812 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.810+0000 7f5fb8d3e640 1 -- 172.21.15.83:0/3723669056 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5f9406b4d0 msgr2=0x7f5f9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.813 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.810+0000 7f5fb8d3e640 1 --2- 172.21.15.83:0/3723669056 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5f9406b4d0 0x7f5f9406d990 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5fb40c79f0 tx=0x7f5fa80014e0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.813 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.810+0000 7f5fb8d3e640 1 -- 172.21.15.83:0/3723669056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f9c0039b0 msgr2=0x7f5fb415cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.813 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.810+0000 7f5fb8d3e640 1 --2- 172.21.15.83:0/3723669056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f9c0039b0 0x7f5fb415cd30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5fa400e230 tx=0x7f5fa4009750 comp rx=0 tx=0).stop 2024-03-06T07:21:51.813 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.810+0000 7f5fb8d3e640 1 -- 172.21.15.83:0/3723669056 shutdown_connections 2024-03-06T07:21:51.813 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.810+0000 7f5fb8d3e640 1 --2- 172.21.15.83:0/3723669056 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5f9406b4d0 0x7f5f9406d990 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.813 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.810+0000 7f5fb8d3e640 1 --2- 172.21.15.83:0/3723669056 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5fb4161300 0x7f5fb40c9540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.813 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.810+0000 7f5fb8d3e640 1 --2- 172.21.15.83:0/3723669056 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5fb40c75b0 0x7f5fb40c9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.813 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.810+0000 7f5fb8d3e640 1 --2- 172.21.15.83:0/3723669056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f9c0039b0 0x7f5fb415cd30 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.813 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.810+0000 7f5fb8d3e640 1 -- 172.21.15.83:0/3723669056 >> 172.21.15.83:0/3723669056 conn(0x7f5fb40d3ce0 msgr2=0x7f5fb406caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:51.813 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.810+0000 7f5fb8d3e640 1 -- 172.21.15.83:0/3723669056 shutdown_connections 2024-03-06T07:21:51.813 INFO:teuthology.misc.health.smithi083.stderr:2024-03-06T07:21:51.810+0000 7f5fb8d3e640 1 -- 172.21.15.83:0/3723669056 wait complete. 2024-03-06T07:21:51.822 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-06T07:21:51.822 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-06T07:21:51.822 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:21:51.822 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:21:51.825 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:21:51.825 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:21:51.825 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:21:51.827 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:21:51.827 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:21:51.827 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:21:51.830 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:21:51.830 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:21:51.830 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:21:51.832 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:21:51.832 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-06T07:21:51.835 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:51.900 INFO:tasks.ceph.mds.c.smithi083.stdout:starting mds.c at 2024-03-06T07:21:51.900 INFO:tasks.ceph.mds.a.smithi083.stdout:starting mds.a at 2024-03-06T07:21:51.901 INFO:tasks.ceph.mds.c.smithi083.stderr:2024-03-06T07:21:51.898+0000 7f363946a640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-03-06T07:21:51.915 INFO:tasks.ceph.mds.d.smithi183.stdout:starting mds.d at 2024-03-06T07:21:51.916 INFO:tasks.ceph.mds.b.smithi183.stdout:starting mds.b at 2024-03-06T07:21:51.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.939+0000 7fd9e149d640 1 Processor -- start 2024-03-06T07:21:51.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.939+0000 7fd9e149d640 1 -- start start 2024-03-06T07:21:51.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.940+0000 7fd9e149d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd9dc0d4760 0x7fd9dc0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.940+0000 7fd9e149d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9dc0d0a80 0x7fd9dc0cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.940+0000 7fd9e149d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9dc0d25d0 con 0x7fd9dc0b4170 2024-03-06T07:21:51.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.940+0000 7fd9e149d640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9dc0d4bf0 con 0x7fd9dc0d4760 2024-03-06T07:21:51.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.940+0000 7fd9e149d640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9dc0ce9e0 con 0x7fd9dc0d0a80 2024-03-06T07:21:51.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.940+0000 7fd9da7fc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd9dc0d4760 0x7fd9dc0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.940+0000 7fd9daffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fd9dc0b4170 0x7fd9dc0d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52328/0 (socket says 172.21.15.83:52328) 2024-03-06T07:21:51.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.940+0000 7fd9daffd640 1 -- 172.21.15.83:0/1610657843 learned_addr learned my addr 172.21.15.83:0/1610657843 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:21:51.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.940+0000 7fd9db7fe640 1 --2- 172.21.15.83:0/1610657843 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9dc0d0a80 0x7fd9dc0cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.940+0000 7fd9da7fc640 1 -- 172.21.15.83:0/1610657843 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9dc0d0a80 msgr2=0x7fd9dc0cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:51.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.940+0000 7fd9da7fc640 1 --2- 172.21.15.83:0/1610657843 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9dc0d0a80 0x7fd9dc0cee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.940+0000 7fd9da7fc640 1 -- 172.21.15.83:0/1610657843 >> v1:172.21.15.83:6789/0 conn(0x7fd9dc0b4170 legacy=0x7fd9dc0d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.940+0000 7fd9da7fc640 1 -- 172.21.15.83:0/1610657843 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9dc0d2870 con 0x7fd9dc0d4760 2024-03-06T07:21:51.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.940+0000 7fd9d9ffb640 1 -- 172.21.15.83:0/1610657843 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3716044513 0 0) 0x7fd9dc0d25d0 con 0x7fd9dc0b4170 2024-03-06T07:21:51.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.940+0000 7fd9db7fe640 1 --2- 172.21.15.83:0/1610657843 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9dc0d0a80 0x7fd9dc0cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:51.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.941+0000 7fd9da7fc640 1 --2- 172.21.15.83:0/1610657843 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9dc0d4760 0x7fd9dc0b6a90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd9d00097a0 tx=0x7fd9d002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fafcd5a06e6ab57b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.941+0000 7fd9d9ffb640 1 -- 172.21.15.83:0/1610657843 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d0035020 con 0x7fd9dc0d4760 2024-03-06T07:21:51.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.941+0000 7fd9d9ffb640 1 -- 172.21.15.83:0/1610657843 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9d0031830 con 0x7fd9dc0d4760 2024-03-06T07:21:51.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.941+0000 7fd9d9ffb640 1 -- 172.21.15.83:0/1610657843 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d0031c00 con 0x7fd9dc0d4760 2024-03-06T07:21:51.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.942+0000 7fd9e149d640 1 -- 172.21.15.83:0/1610657843 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9dc0d4760 msgr2=0x7fd9dc0b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.942+0000 7fd9e149d640 1 --2- 172.21.15.83:0/1610657843 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9dc0d4760 0x7fd9dc0b6a90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd9d00097a0 tx=0x7fd9d002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.942+0000 7fd9e149d640 1 -- 172.21.15.83:0/1610657843 shutdown_connections 2024-03-06T07:21:51.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.942+0000 7fd9e149d640 1 --2- 172.21.15.83:0/1610657843 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9dc0d0a80 0x7fd9dc0cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.942+0000 7fd9e149d640 1 --2- 172.21.15.83:0/1610657843 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9dc0d4760 0x7fd9dc0b6a90 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.942+0000 7fd9e149d640 1 -- 172.21.15.83:0/1610657843 >> 172.21.15.83:0/1610657843 conn(0x7fd9dc0d3c40 msgr2=0x7fd9dc0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:51.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.942+0000 7fd9e149d640 1 -- 172.21.15.83:0/1610657843 shutdown_connections 2024-03-06T07:21:51.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.942+0000 7fd9e149d640 1 -- 172.21.15.83:0/1610657843 wait complete. 2024-03-06T07:21:51.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.943+0000 7fd9e149d640 1 Processor -- start 2024-03-06T07:21:51.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.943+0000 7fd9e149d640 1 -- start start 2024-03-06T07:21:51.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.944+0000 7fd9e149d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9dc0b4170 0x7fd9dc0cc990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.944+0000 7fd9e149d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9dc0d0a80 0x7fd9dc0cb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.944+0000 7fd9e149d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9dc1f4b60 0x7fd9dc14ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.944+0000 7fd9e149d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9dc0b4dd0 con 0x7fd9dc1f4b60 2024-03-06T07:21:51.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.944+0000 7fd9e149d640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9dc0b4550 con 0x7fd9dc0b4170 2024-03-06T07:21:51.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.944+0000 7fd9e149d640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9dc0b4f50 con 0x7fd9dc0d0a80 2024-03-06T07:21:51.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.944+0000 7fd9db7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9dc1f4b60 0x7fd9dc14ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.944+0000 7fd9db7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9dc1f4b60 0x7fd9dc14ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51434/0 (socket says 172.21.15.83:51434) 2024-03-06T07:21:51.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.944+0000 7fd9db7fe640 1 -- 172.21.15.83:0/2088668489 learned_addr learned my addr 172.21.15.83:0/2088668489 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:21:51.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.944+0000 7fd9daffd640 1 --2- 172.21.15.83:0/2088668489 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9dc0b4170 0x7fd9dc0cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.944+0000 7fd9da7fc640 1 --2- 172.21.15.83:0/2088668489 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9dc0d0a80 0x7fd9dc0cb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.944+0000 7fd9db7fe640 1 -- 172.21.15.83:0/2088668489 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9dc0d0a80 msgr2=0x7fd9dc0cb5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.944+0000 7fd9db7fe640 1 --2- 172.21.15.83:0/2088668489 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9dc0d0a80 0x7fd9dc0cb5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.944+0000 7fd9db7fe640 1 -- 172.21.15.83:0/2088668489 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9dc0b4170 msgr2=0x7fd9dc0cc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.944+0000 7fd9db7fe640 1 --2- 172.21.15.83:0/2088668489 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9dc0b4170 0x7fd9dc0cc990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.944+0000 7fd9db7fe640 1 -- 172.21.15.83:0/2088668489 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9dc0cd1e0 con 0x7fd9dc1f4b60 2024-03-06T07:21:51.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.944+0000 7fd9db7fe640 1 --2- 172.21.15.83:0/2088668489 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9dc1f4b60 0x7fd9dc14ef60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd9d4007f20 tx=0x7fd9d400d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.944+0000 7fd9c37fe640 1 -- 172.21.15.83:0/2088668489 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d401a070 con 0x7fd9dc1f4b60 2024-03-06T07:21:51.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.944+0000 7fd9c37fe640 1 -- 172.21.15.83:0/2088668489 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9d4003b90 con 0x7fd9dc1f4b60 2024-03-06T07:21:51.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.944+0000 7fd9c37fe640 1 -- 172.21.15.83:0/2088668489 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d4004030 con 0x7fd9dc1f4b60 2024-03-06T07:21:51.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.944+0000 7fd9e149d640 1 -- 172.21.15.83:0/2088668489 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9dc0cd4c0 con 0x7fd9dc1f4b60 2024-03-06T07:21:51.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.944+0000 7fd9e149d640 1 -- 172.21.15.83:0/2088668489 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9dc0cced0 con 0x7fd9dc1f4b60 2024-03-06T07:21:51.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.945+0000 7fd9e149d640 1 -- 172.21.15.83:0/2088668489 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9dc04f330 con 0x7fd9dc1f4b60 2024-03-06T07:21:51.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.946+0000 7fd9c37fe640 1 -- 172.21.15.83:0/2088668489 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd9d401d070 con 0x7fd9dc1f4b60 2024-03-06T07:21:51.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.946+0000 7fd9c37fe640 1 --2- 172.21.15.83:0/2088668489 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd9c806b500 0x7fd9c806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.946+0000 7fd9c37fe640 1 -- 172.21.15.83:0/2088668489 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd9d408a1b0 con 0x7fd9dc1f4b60 2024-03-06T07:21:51.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.946+0000 7fd9daffd640 1 --2- 172.21.15.83:0/2088668489 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd9c806b500 0x7fd9c806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.947+0000 7fd9daffd640 1 --2- 172.21.15.83:0/2088668489 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd9c806b500 0x7fd9c806d9c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc000f60 tx=0x7fd9cc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:51.951+0000 7fd9c37fe640 1 -- 172.21.15.83:0/2088668489 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd9d4010070 con 0x7fd9dc1f4b60 2024-03-06T07:21:52.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.122+0000 7fd9e149d640 1 -- 172.21.15.83:0/2088668489 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9dc0733b0 con 0x7fd9dc1f4b60 2024-03-06T07:21:52.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.122+0000 7fd9c37fe640 1 -- 172.21.15.83:0/2088668489 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12896 (secure 0 0 0) 0x7fd9d4059380 con 0x7fd9dc1f4b60 2024-03-06T07:21:52.124 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:21:52.124 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":16,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:21:48.365486+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:52.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.124+0000 7fd9e149d640 1 -- 172.21.15.83:0/2088668489 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd9c806b500 msgr2=0x7fd9c806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.124+0000 7fd9e149d640 1 --2- 172.21.15.83:0/2088668489 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd9c806b500 0x7fd9c806d9c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc000f60 tx=0x7fd9cc009040 comp rx=0 tx=0).stop 2024-03-06T07:21:52.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.124+0000 7fd9e149d640 1 -- 172.21.15.83:0/2088668489 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9dc1f4b60 msgr2=0x7fd9dc14ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.124+0000 7fd9e149d640 1 --2- 172.21.15.83:0/2088668489 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9dc1f4b60 0x7fd9dc14ef60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd9d4007f20 tx=0x7fd9d400d560 comp rx=0 tx=0).stop 2024-03-06T07:21:52.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.124+0000 7fd9e149d640 1 -- 172.21.15.83:0/2088668489 shutdown_connections 2024-03-06T07:21:52.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.124+0000 7fd9e149d640 1 --2- 172.21.15.83:0/2088668489 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd9c806b500 0x7fd9c806d9c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.124+0000 7fd9e149d640 1 --2- 172.21.15.83:0/2088668489 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9dc1f4b60 0x7fd9dc14ef60 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.124+0000 7fd9e149d640 1 --2- 172.21.15.83:0/2088668489 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9dc0d0a80 0x7fd9dc0cb5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.124+0000 7fd9e149d640 1 --2- 172.21.15.83:0/2088668489 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9dc0b4170 0x7fd9dc0cc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.124+0000 7fd9e149d640 1 -- 172.21.15.83:0/2088668489 >> 172.21.15.83:0/2088668489 conn(0x7fd9dc0d3c40 msgr2=0x7fd9dc0b55c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:52.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.124+0000 7fd9e149d640 1 -- 172.21.15.83:0/2088668489 shutdown_connections 2024-03-06T07:21:52.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.124+0000 7fd9e149d640 1 -- 172.21.15.83:0/2088668489 wait complete. 2024-03-06T07:21:52.137 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:21:52.137 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:21:52.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.239+0000 7f9cc975e640 1 Processor -- start 2024-03-06T07:21:52.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.239+0000 7f9cc975e640 1 -- start start 2024-03-06T07:21:52.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.240+0000 7f9cc975e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f9cc40d4860 0x7f9cc40d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.240+0000 7f9cc975e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9cc40b0e30 0x7f9cc40b1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.240+0000 7f9cc975e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cc41618b0 con 0x7f9cc41614d0 2024-03-06T07:21:52.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.240+0000 7f9cc975e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cc40d4cf0 con 0x7f9cc40b0e30 2024-03-06T07:21:52.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.240+0000 7f9cc975e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cc40cf400 con 0x7f9cc40d4860 2024-03-06T07:21:52.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.240+0000 7f9cc37fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9cc40b0e30 0x7f9cc40b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.240+0000 7f9cc37fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9cc40b0e30 0x7f9cc40b1260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51450/0 (socket says 172.21.15.83:51450) 2024-03-06T07:21:52.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.240+0000 7f9cc37fe640 1 -- 172.21.15.83:0/3282328936 learned_addr learned my addr 172.21.15.83:0/3282328936 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:21:52.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.240+0000 7f9cc27fc640 1 --2- 172.21.15.83:0/3282328936 >> v2:172.21.15.183:3300/0 conn(0x7f9cc40d4860 0x7f9cc40d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.240+0000 7f9cc37fe640 1 -- 172.21.15.83:0/3282328936 >> v2:172.21.15.183:3300/0 conn(0x7f9cc40d4860 msgr2=0x7f9cc40d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:52.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.240+0000 7f9cc37fe640 1 --2- 172.21.15.83:0/3282328936 >> v2:172.21.15.183:3300/0 conn(0x7f9cc40d4860 0x7f9cc40d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.240+0000 7f9cc37fe640 1 -- 172.21.15.83:0/3282328936 >> v1:172.21.15.183:6789/0 conn(0x7f9cc41614d0 legacy=0x7f9cc40d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:52.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.240+0000 7f9cc37fe640 1 -- 172.21.15.83:0/3282328936 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cc40cf6a0 con 0x7f9cc40b0e30 2024-03-06T07:21:52.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.240+0000 7f9cc27fc640 1 --2- 172.21.15.83:0/3282328936 >> v2:172.21.15.183:3300/0 conn(0x7f9cc40d4860 0x7f9cc40d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:52.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.241+0000 7f9cc37fe640 1 --2- 172.21.15.83:0/3282328936 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9cc40b0e30 0x7f9cc40b1260 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9cc40b2a70 tx=0x7f9cbc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9ab8370efd99d4e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:52.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.241+0000 7f9cc1ffb640 1 -- 172.21.15.83:0/3282328936 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cbc035020 con 0x7f9cc40b0e30 2024-03-06T07:21:52.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.241+0000 7f9cc1ffb640 1 -- 172.21.15.83:0/3282328936 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cbc031640 con 0x7f9cc40b0e30 2024-03-06T07:21:52.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.241+0000 7f9cc1ffb640 1 -- 172.21.15.83:0/3282328936 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cbc02d040 con 0x7f9cc40b0e30 2024-03-06T07:21:52.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.242+0000 7f9cc975e640 1 -- 172.21.15.83:0/3282328936 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9cc40b0e30 msgr2=0x7f9cc40b1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.242+0000 7f9cc975e640 1 --2- 172.21.15.83:0/3282328936 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9cc40b0e30 0x7f9cc40b1260 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9cc40b2a70 tx=0x7f9cbc02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.242+0000 7f9cc975e640 1 -- 172.21.15.83:0/3282328936 shutdown_connections 2024-03-06T07:21:52.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.242+0000 7f9cc975e640 1 --2- 172.21.15.83:0/3282328936 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9cc40b0e30 0x7f9cc40b1260 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.242+0000 7f9cc975e640 1 --2- 172.21.15.83:0/3282328936 >> v2:172.21.15.183:3300/0 conn(0x7f9cc40d4860 0x7f9cc40d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.242+0000 7f9cc975e640 1 -- 172.21.15.83:0/3282328936 >> 172.21.15.83:0/3282328936 conn(0x7f9cc40d3d40 msgr2=0x7f9cc40d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:52.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.242+0000 7f9cc975e640 1 -- 172.21.15.83:0/3282328936 shutdown_connections 2024-03-06T07:21:52.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.242+0000 7f9cc975e640 1 -- 172.21.15.83:0/3282328936 wait complete. 2024-03-06T07:21:52.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.243+0000 7f9cc975e640 1 Processor -- start 2024-03-06T07:21:52.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.243+0000 7f9cc975e640 1 -- start start 2024-03-06T07:21:52.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.244+0000 7f9cc975e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9cc40b0e30 0x7f9cc40b98b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.244+0000 7f9cc975e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9cc40d4860 0x7f9cc40b41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.244+0000 7f9cc975e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9cc41614d0 0x7f9cc414aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.244+0000 7f9cc975e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc40b0910 con 0x7f9cc40d4860 2024-03-06T07:21:52.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.244+0000 7f9cc975e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc40cfc60 con 0x7f9cc40b0e30 2024-03-06T07:21:52.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.244+0000 7f9cc975e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc40d2560 con 0x7f9cc41614d0 2024-03-06T07:21:52.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.244+0000 7f9cc27fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9cc40d4860 0x7f9cc40b41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.244+0000 7f9cc27fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9cc40d4860 0x7f9cc40b41a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51454/0 (socket says 172.21.15.83:51454) 2024-03-06T07:21:52.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.244+0000 7f9cc27fc640 1 -- 172.21.15.83:0/2558765469 learned_addr learned my addr 172.21.15.83:0/2558765469 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:21:52.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.244+0000 7f9cc2ffd640 1 --2- 172.21.15.83:0/2558765469 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9cc40b0e30 0x7f9cc40b98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.244+0000 7f9cc37fe640 1 --2- 172.21.15.83:0/2558765469 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9cc41614d0 0x7f9cc414aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.244+0000 7f9cc27fc640 1 -- 172.21.15.83:0/2558765469 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9cc41614d0 msgr2=0x7f9cc414aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.244+0000 7f9cc27fc640 1 --2- 172.21.15.83:0/2558765469 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9cc41614d0 0x7f9cc414aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.244+0000 7f9cc27fc640 1 -- 172.21.15.83:0/2558765469 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9cc40b0e30 msgr2=0x7f9cc40b98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.244+0000 7f9cc27fc640 1 --2- 172.21.15.83:0/2558765469 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9cc40b0e30 0x7f9cc40b98b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.244+0000 7f9cc27fc640 1 -- 172.21.15.83:0/2558765469 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cc414b010 con 0x7f9cc40d4860 2024-03-06T07:21:52.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.244+0000 7f9cc27fc640 1 --2- 172.21.15.83:0/2558765469 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9cc40d4860 0x7f9cc40b41a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9cb800c820 tx=0x7f9cb800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:52.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.245+0000 7f9cab7fe640 1 -- 172.21.15.83:0/2558765469 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb8015070 con 0x7f9cc40d4860 2024-03-06T07:21:52.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.245+0000 7f9cab7fe640 1 -- 172.21.15.83:0/2558765469 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cb8007a30 con 0x7f9cc40d4860 2024-03-06T07:21:52.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.245+0000 7f9cc975e640 1 -- 172.21.15.83:0/2558765469 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cc40b8af0 con 0x7f9cc40d4860 2024-03-06T07:21:52.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.245+0000 7f9cab7fe640 1 -- 172.21.15.83:0/2558765469 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb8007e00 con 0x7f9cc40d4860 2024-03-06T07:21:52.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.245+0000 7f9cc975e640 1 -- 172.21.15.83:0/2558765469 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cc40b90b0 con 0x7f9cc40d4860 2024-03-06T07:21:52.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.246+0000 7f9cc975e640 1 -- 172.21.15.83:0/2558765469 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cc40b95a0 con 0x7f9cc40d4860 2024-03-06T07:21:52.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.246+0000 7f9cab7fe640 1 -- 172.21.15.83:0/2558765469 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9cb8018070 con 0x7f9cc40d4860 2024-03-06T07:21:52.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.246+0000 7f9cab7fe640 1 --2- 172.21.15.83:0/2558765469 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9cb006b4d0 0x7f9cb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.246+0000 7f9cab7fe640 1 -- 172.21.15.83:0/2558765469 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9cb8089d50 con 0x7f9cc40d4860 2024-03-06T07:21:52.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.246+0000 7f9cc2ffd640 1 --2- 172.21.15.83:0/2558765469 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9cb006b4d0 0x7f9cb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.247+0000 7f9cc2ffd640 1 --2- 172.21.15.83:0/2558765469 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9cb006b4d0 0x7f9cb006d990 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9cb400b450 tx=0x7f9cb400d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:52.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.251+0000 7f9cab7fe640 1 -- 172.21.15.83:0/2558765469 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9cb808d030 con 0x7f9cc40d4860 2024-03-06T07:21:52.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.421+0000 7f9cc975e640 1 -- 172.21.15.83:0/2558765469 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9cc40cecc0 con 0x7f9cc40d4860 2024-03-06T07:21:52.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.422+0000 7f9cab7fe640 1 -- 172.21.15.83:0/2558765469 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9cb8058960 con 0x7f9cc40d4860 2024-03-06T07:21:52.424 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:21:52.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.424+0000 7f9cc975e640 1 -- 172.21.15.83:0/2558765469 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9cb006b4d0 msgr2=0x7f9cb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.424+0000 7f9cc975e640 1 --2- 172.21.15.83:0/2558765469 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9cb006b4d0 0x7f9cb006d990 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9cb400b450 tx=0x7f9cb400d040 comp rx=0 tx=0).stop 2024-03-06T07:21:52.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.424+0000 7f9cc975e640 1 -- 172.21.15.83:0/2558765469 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9cc40d4860 msgr2=0x7f9cc40b41a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.424+0000 7f9cc975e640 1 --2- 172.21.15.83:0/2558765469 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9cc40d4860 0x7f9cc40b41a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9cb800c820 tx=0x7f9cb800cc20 comp rx=0 tx=0).stop 2024-03-06T07:21:52.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.424+0000 7f9cc975e640 1 -- 172.21.15.83:0/2558765469 shutdown_connections 2024-03-06T07:21:52.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.424+0000 7f9cc975e640 1 --2- 172.21.15.83:0/2558765469 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9cb006b4d0 0x7f9cb006d990 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.424+0000 7f9cc975e640 1 --2- 172.21.15.83:0/2558765469 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9cc41614d0 0x7f9cc414aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.424+0000 7f9cc975e640 1 --2- 172.21.15.83:0/2558765469 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9cc40d4860 0x7f9cc40b41a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.424+0000 7f9cc975e640 1 --2- 172.21.15.83:0/2558765469 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9cc40b0e30 0x7f9cc40b98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.424+0000 7f9cc975e640 1 -- 172.21.15.83:0/2558765469 >> 172.21.15.83:0/2558765469 conn(0x7f9cc40d3d40 msgr2=0x7f9cc406bc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:52.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.424+0000 7f9cc975e640 1 -- 172.21.15.83:0/2558765469 shutdown_connections 2024-03-06T07:21:52.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.424+0000 7f9cc975e640 1 -- 172.21.15.83:0/2558765469 wait complete. 2024-03-06T07:21:52.436 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:52.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.534+0000 7f96e2446640 1 Processor -- start 2024-03-06T07:21:52.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.534+0000 7f96e2446640 1 -- start start 2024-03-06T07:21:52.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.535+0000 7f96e2446640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f96dc0d00c0 0x7f96dc0d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.535+0000 7f96e2446640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f96dc06cf20 0x7f96dc06d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.535+0000 7f96e2446640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96dc0d09e0 con 0x7f96dc0cf850 2024-03-06T07:21:52.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.535+0000 7f96e2446640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96dc06ca90 con 0x7f96dc0d00c0 2024-03-06T07:21:52.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.535+0000 7f96e2446640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96dc15e380 con 0x7f96dc06cf20 2024-03-06T07:21:52.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.535+0000 7f96dbfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f96dc0d00c0 0x7f96dc0d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.535+0000 7f96dbfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f96dc0d00c0 0x7f96dc0d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51462/0 (socket says 172.21.15.83:51462) 2024-03-06T07:21:52.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.535+0000 7f96dbfff640 1 -- 172.21.15.83:0/2804433145 learned_addr learned my addr 172.21.15.83:0/2804433145 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:21:52.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.535+0000 7f96e09e3640 1 --2- 172.21.15.83:0/2804433145 >> v2:172.21.15.183:3300/0 conn(0x7f96dc06cf20 0x7f96dc06d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.535+0000 7f96dbfff640 1 -- 172.21.15.83:0/2804433145 >> v2:172.21.15.183:3300/0 conn(0x7f96dc06cf20 msgr2=0x7f96dc06d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:52.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.535+0000 7f96dbfff640 1 --2- 172.21.15.83:0/2804433145 >> v2:172.21.15.183:3300/0 conn(0x7f96dc06cf20 0x7f96dc06d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.535+0000 7f96dbfff640 1 -- 172.21.15.83:0/2804433145 >> v1:172.21.15.183:6789/0 conn(0x7f96dc0cf850 legacy=0x7f96dc0cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.535+0000 7f96dbfff640 1 -- 172.21.15.83:0/2804433145 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96dc0cdc10 con 0x7f96dc0d00c0 2024-03-06T07:21:52.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.536+0000 7f96dbfff640 1 --2- 172.21.15.83:0/2804433145 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96dc0d00c0 0x7f96dc0d04a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f96dc0cfda0 tx=0x7f96cc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=872f5f18d15d5cca server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:52.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.536+0000 7f96daffd640 1 -- 172.21.15.83:0/2804433145 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96cc034070 con 0x7f96dc0d00c0 2024-03-06T07:21:52.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.536+0000 7f96daffd640 1 -- 172.21.15.83:0/2804433145 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96cc030860 con 0x7f96dc0d00c0 2024-03-06T07:21:52.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.536+0000 7f96daffd640 1 -- 172.21.15.83:0/2804433145 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96cc030c30 con 0x7f96dc0d00c0 2024-03-06T07:21:52.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.537+0000 7f96e2446640 1 -- 172.21.15.83:0/2804433145 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96dc0d00c0 msgr2=0x7f96dc0d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.537+0000 7f96e2446640 1 --2- 172.21.15.83:0/2804433145 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96dc0d00c0 0x7f96dc0d04a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f96dc0cfda0 tx=0x7f96cc02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.537+0000 7f96e2446640 1 -- 172.21.15.83:0/2804433145 shutdown_connections 2024-03-06T07:21:52.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.537+0000 7f96e2446640 1 --2- 172.21.15.83:0/2804433145 >> v2:172.21.15.183:3300/0 conn(0x7f96dc06cf20 0x7f96dc06d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.537+0000 7f96e2446640 1 --2- 172.21.15.83:0/2804433145 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96dc0d00c0 0x7f96dc0d04a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.537+0000 7f96e2446640 1 -- 172.21.15.83:0/2804433145 >> 172.21.15.83:0/2804433145 conn(0x7f96dc0b35a0 msgr2=0x7f96dc0b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:52.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.537+0000 7f96e2446640 1 -- 172.21.15.83:0/2804433145 shutdown_connections 2024-03-06T07:21:52.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.537+0000 7f96e2446640 1 -- 172.21.15.83:0/2804433145 wait complete. 2024-03-06T07:21:52.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.538+0000 7f96e2446640 1 Processor -- start 2024-03-06T07:21:52.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.538+0000 7f96e2446640 1 -- start start 2024-03-06T07:21:52.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.539+0000 7f96e2446640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96dc06cf20 0x7f96dc14a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.539+0000 7f96e2446640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96dc0cf850 0x7f96dc15a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.539+0000 7f96e2446640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96dc0d00c0 0x7f96dc15aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.539+0000 7f96e2446640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96dc06c720 con 0x7f96dc0d00c0 2024-03-06T07:21:52.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.539+0000 7f96e2446640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96dc06c440 con 0x7f96dc0cf850 2024-03-06T07:21:52.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.539+0000 7f96e2446640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96dc0b1030 con 0x7f96dc06cf20 2024-03-06T07:21:52.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.539+0000 7f96e09e3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96dc0d00c0 0x7f96dc15aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.539+0000 7f96e09e3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96dc0d00c0 0x7f96dc15aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51474/0 (socket says 172.21.15.83:51474) 2024-03-06T07:21:52.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.539+0000 7f96e09e3640 1 -- 172.21.15.83:0/1946980594 learned_addr learned my addr 172.21.15.83:0/1946980594 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:21:52.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.539+0000 7f96db7fe640 1 --2- 172.21.15.83:0/1946980594 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96dc0cf850 0x7f96dc15a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.539+0000 7f96dbfff640 1 --2- 172.21.15.83:0/1946980594 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96dc06cf20 0x7f96dc14a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.539+0000 7f96e09e3640 1 -- 172.21.15.83:0/1946980594 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96dc06cf20 msgr2=0x7f96dc14a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.539+0000 7f96e09e3640 1 --2- 172.21.15.83:0/1946980594 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96dc06cf20 0x7f96dc14a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.539+0000 7f96e09e3640 1 -- 172.21.15.83:0/1946980594 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96dc0cf850 msgr2=0x7f96dc15a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.539+0000 7f96e09e3640 1 --2- 172.21.15.83:0/1946980594 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96dc0cf850 0x7f96dc15a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.539+0000 7f96e09e3640 1 -- 172.21.15.83:0/1946980594 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96dc0c9420 con 0x7f96dc0d00c0 2024-03-06T07:21:52.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.539+0000 7f96db7fe640 1 --2- 172.21.15.83:0/1946980594 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96dc0cf850 0x7f96dc15a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:52.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.539+0000 7f96e09e3640 1 --2- 172.21.15.83:0/1946980594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96dc0d00c0 0x7f96dc15aaf0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f96d4007e60 tx=0x7f96d400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:52.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.539+0000 7f96d8ff9640 1 -- 172.21.15.83:0/1946980594 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d4014420 con 0x7f96dc0d00c0 2024-03-06T07:21:52.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.539+0000 7f96d8ff9640 1 -- 172.21.15.83:0/1946980594 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96d4019070 con 0x7f96dc0d00c0 2024-03-06T07:21:52.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.539+0000 7f96d8ff9640 1 -- 172.21.15.83:0/1946980594 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d4014820 con 0x7f96dc0d00c0 2024-03-06T07:21:52.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.540+0000 7f96e2446640 1 -- 172.21.15.83:0/1946980594 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96dc0c9700 con 0x7f96dc0d00c0 2024-03-06T07:21:52.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.540+0000 7f96e2446640 1 -- 172.21.15.83:0/1946980594 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96dc0c9cc0 con 0x7f96dc0d00c0 2024-03-06T07:21:52.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.540+0000 7f96e2446640 1 -- 172.21.15.83:0/1946980594 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96dc04f330 con 0x7f96dc0d00c0 2024-03-06T07:21:52.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.541+0000 7f96d8ff9640 1 -- 172.21.15.83:0/1946980594 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f96d40149c0 con 0x7f96dc0d00c0 2024-03-06T07:21:52.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.541+0000 7f96d8ff9640 1 --2- 172.21.15.83:0/1946980594 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f96c806b500 0x7f96c806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.541+0000 7f96d8ff9640 1 -- 172.21.15.83:0/1946980594 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f96d408a780 con 0x7f96dc0d00c0 2024-03-06T07:21:52.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.541+0000 7f96dbfff640 1 --2- 172.21.15.83:0/1946980594 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f96c806b500 0x7f96c806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.542+0000 7f96dbfff640 1 --2- 172.21.15.83:0/1946980594 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f96c806b500 0x7f96c806d9c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f96cc036000 tx=0x7f96cc03b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:52.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.546+0000 7f96d8ff9640 1 -- 172.21.15.83:0/1946980594 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f96d40598b0 con 0x7f96dc0d00c0 2024-03-06T07:21:52.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.753+0000 7f96e2446640 1 -- 172.21.15.83:0/1946980594 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f96dc06ff60 con 0x7f96dc0d00c0 2024-03-06T07:21:52.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.753+0000 7f96d8ff9640 1 -- 172.21.15.83:0/1946980594 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1324 (secure 0 0 0) 0x7f96d40591b0 con 0x7f96dc0d00c0 2024-03-06T07:21:52.755 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:21:52.756 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4287,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1597344199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1597344199},{"type":"v1","addr":"172.21.15.83:6837","nonce":1597344199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-06T07:21:52.756 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 2 2024-03-06T07:21:52.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.755+0000 7f96e2446640 1 -- 172.21.15.83:0/1946980594 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f96c806b500 msgr2=0x7f96c806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.755+0000 7f96e2446640 1 --2- 172.21.15.83:0/1946980594 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f96c806b500 0x7f96c806d9c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f96cc036000 tx=0x7f96cc03b040 comp rx=0 tx=0).stop 2024-03-06T07:21:52.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.755+0000 7f96e2446640 1 -- 172.21.15.83:0/1946980594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96dc0d00c0 msgr2=0x7f96dc15aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.755+0000 7f96e2446640 1 --2- 172.21.15.83:0/1946980594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96dc0d00c0 0x7f96dc15aaf0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f96d4007e60 tx=0x7f96d400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.755+0000 7f96e2446640 1 -- 172.21.15.83:0/1946980594 shutdown_connections 2024-03-06T07:21:52.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.755+0000 7f96e2446640 1 --2- 172.21.15.83:0/1946980594 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f96c806b500 0x7f96c806d9c0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.755+0000 7f96e2446640 1 --2- 172.21.15.83:0/1946980594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96dc0d00c0 0x7f96dc15aaf0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.755+0000 7f96e2446640 1 --2- 172.21.15.83:0/1946980594 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96dc0cf850 0x7f96dc15a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.755+0000 7f96e2446640 1 --2- 172.21.15.83:0/1946980594 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96dc06cf20 0x7f96dc14a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.755+0000 7f96e2446640 1 -- 172.21.15.83:0/1946980594 >> 172.21.15.83:0/1946980594 conn(0x7f96dc0b35a0 msgr2=0x7f96dc06f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:52.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.755+0000 7f96e2446640 1 -- 172.21.15.83:0/1946980594 shutdown_connections 2024-03-06T07:21:52.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:52.755+0000 7f96e2446640 1 -- 172.21.15.83:0/1946980594 wait complete. 2024-03-06T07:21:54.769 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:54.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.878+0000 7f577f2eb640 1 Processor -- start 2024-03-06T07:21:54.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.878+0000 7f577f2eb640 1 -- start start 2024-03-06T07:21:54.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.879+0000 7f577f2eb640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f57781515e0 0x7f57780cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.879+0000 7f577f2eb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f57780cd7a0 0x7f57780cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.879+0000 7f577f2eb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5778071f40 con 0x7f577815b430 2024-03-06T07:21:54.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.879+0000 7f577f2eb640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f577815b8e0 con 0x7f57781515e0 2024-03-06T07:21:54.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.879+0000 7f577f2eb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5778070a70 con 0x7f57780cd7a0 2024-03-06T07:21:54.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.879+0000 7f577c886640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f577815b430 0x7f5778071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52342/0 (socket says 172.21.15.83:52342) 2024-03-06T07:21:54.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.879+0000 7f577c886640 1 -- 172.21.15.83:0/70508414 learned_addr learned my addr 172.21.15.83:0/70508414 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:21:54.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.879+0000 7f576ffff640 1 -- 172.21.15.83:0/70508414 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3327017291 0 0) 0x7f5778071f40 con 0x7f577815b430 2024-03-06T07:21:54.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.879+0000 7f576ffff640 1 -- 172.21.15.83:0/70508414 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5764003520 con 0x7f577815b430 2024-03-06T07:21:54.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.879+0000 7f577d087640 1 --2- 172.21.15.83:0/70508414 >> v2:172.21.15.183:3300/0 conn(0x7f57781515e0 0x7f57780cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.879+0000 7f577d888640 1 --2- 172.21.15.83:0/70508414 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f57780cd7a0 0x7f57780cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.880+0000 7f576ffff640 1 -- 172.21.15.83:0/70508414 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1815094837 0 0) 0x7f5764003520 con 0x7f577815b430 2024-03-06T07:21:54.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.880+0000 7f576ffff640 1 -- 172.21.15.83:0/70508414 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f57780cd7a0 msgr2=0x7f57780cdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:54.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.880+0000 7f576ffff640 1 --2- 172.21.15.83:0/70508414 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f57780cd7a0 0x7f57780cdbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.880+0000 7f576ffff640 1 -- 172.21.15.83:0/70508414 >> v2:172.21.15.183:3300/0 conn(0x7f57781515e0 msgr2=0x7f57780cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:54.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.880+0000 7f576ffff640 1 --2- 172.21.15.83:0/70508414 >> v2:172.21.15.183:3300/0 conn(0x7f57781515e0 0x7f57780cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.880+0000 7f576ffff640 1 -- 172.21.15.83:0/70508414 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5778070d50 con 0x7f577815b430 2024-03-06T07:21:54.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.880+0000 7f576ffff640 1 -- 172.21.15.83:0/70508414 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f5768001690 con 0x7f577815b430 2024-03-06T07:21:54.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.880+0000 7f576ffff640 1 -- 172.21.15.83:0/70508414 >> v1:172.21.15.83:6789/0 conn(0x7f577815b430 legacy=0x7f5778071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.880+0000 7f576ffff640 1 --2- 172.21.15.83:0/70508414 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f57780cd7a0 0x7f57640039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.880+0000 7f576ffff640 1 --2- 172.21.15.83:0/70508414 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5764003f60 0x7f5764004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.880+0000 7f576ffff640 1 --2- 172.21.15.83:0/70508414 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5764004d40 0x7f57640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.880+0000 7f576ffff640 1 -- 172.21.15.83:0/70508414 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f577815b8e0 con 0x7f5764003f60 2024-03-06T07:21:54.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.880+0000 7f576ffff640 1 -- 172.21.15.83:0/70508414 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5778070a70 con 0x7f5764004d40 2024-03-06T07:21:54.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.880+0000 7f576ffff640 1 -- 172.21.15.83:0/70508414 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f576400b9c0 con 0x7f57780cd7a0 2024-03-06T07:21:54.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.880+0000 7f577c886640 1 --2- 172.21.15.83:0/70508414 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5764003f60 0x7f5764004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.880+0000 7f577d087640 1 --2- 172.21.15.83:0/70508414 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f57780cd7a0 0x7f57640039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.880+0000 7f577d888640 1 --2- 172.21.15.83:0/70508414 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5764004d40 0x7f57640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.880+0000 7f577c886640 1 -- 172.21.15.83:0/70508414 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f57780cd7a0 msgr2=0x7f57640039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:54.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.880+0000 7f577c886640 1 --2- 172.21.15.83:0/70508414 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f57780cd7a0 0x7f57640039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.880+0000 7f577c886640 1 -- 172.21.15.83:0/70508414 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5764004d40 msgr2=0x7f57640051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:54.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.880+0000 7f577c886640 1 --2- 172.21.15.83:0/70508414 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5764004d40 0x7f57640051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.880+0000 7f577c886640 1 -- 172.21.15.83:0/70508414 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5764003520 con 0x7f5764003f60 2024-03-06T07:21:54.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.881+0000 7f577c886640 1 --2- 172.21.15.83:0/70508414 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5764003f60 0x7f5764004800 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5768007cf0 tx=0x7f5764005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddce8f5f5116c41d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:54.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.881+0000 7f576ffff640 1 -- 172.21.15.83:0/70508414 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5768007d30 con 0x7f5764003f60 2024-03-06T07:21:54.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.881+0000 7f576ffff640 1 -- 172.21.15.83:0/70508414 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5768032070 con 0x7f5764003f60 2024-03-06T07:21:54.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.881+0000 7f576ffff640 1 -- 172.21.15.83:0/70508414 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57680058d0 con 0x7f5764003f60 2024-03-06T07:21:54.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.881+0000 7f577f2eb640 1 -- 172.21.15.83:0/70508414 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5764003f60 msgr2=0x7f5764004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.881+0000 7f577f2eb640 1 --2- 172.21.15.83:0/70508414 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5764003f60 0x7f5764004800 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5768007cf0 tx=0x7f5764005830 comp rx=0 tx=0).stop 2024-03-06T07:21:54.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.881+0000 7f577d888640 1 -- 172.21.15.83:0/70508414 reap_dead start 2024-03-06T07:21:54.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.881+0000 7f577f2eb640 1 -- 172.21.15.83:0/70508414 shutdown_connections 2024-03-06T07:21:54.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.881+0000 7f577f2eb640 1 -- 172.21.15.83:0/70508414 >> 172.21.15.83:0/70508414 conn(0x7f57780cf8a0 msgr2=0x7f57780cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:54.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.882+0000 7f577f2eb640 1 -- 172.21.15.83:0/70508414 shutdown_connections 2024-03-06T07:21:54.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.882+0000 7f577f2eb640 1 -- 172.21.15.83:0/70508414 wait complete. 2024-03-06T07:21:54.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.883+0000 7f577f2eb640 1 Processor -- start 2024-03-06T07:21:54.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.883+0000 7f577f2eb640 1 -- start start 2024-03-06T07:21:54.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.883+0000 7f577f2eb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5764003f60 0x7f5778159660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.883+0000 7f577f2eb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5778152a70 0x7f577814cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.883+0000 7f577f2eb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5778152ed0 0x7f577814d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.883+0000 7f577f2eb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57780cdbb0 con 0x7f5764003f60 2024-03-06T07:21:54.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.883+0000 7f577f2eb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5778071310 con 0x7f5778152a70 2024-03-06T07:21:54.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.883+0000 7f577f2eb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57780cdd30 con 0x7f5778152ed0 2024-03-06T07:21:54.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.884+0000 7f577d087640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5764003f60 0x7f5778159660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.884+0000 7f577d087640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5764003f60 0x7f5778159660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51492/0 (socket says 172.21.15.83:51492) 2024-03-06T07:21:54.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.884+0000 7f577d087640 1 -- 172.21.15.83:0/214542595 learned_addr learned my addr 172.21.15.83:0/214542595 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:21:54.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.884+0000 7f577c886640 1 --2- 172.21.15.83:0/214542595 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5778152a70 0x7f577814cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.884+0000 7f577d888640 1 --2- 172.21.15.83:0/214542595 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5778152ed0 0x7f577814d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.884+0000 7f577d087640 1 -- 172.21.15.83:0/214542595 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5778152ed0 msgr2=0x7f577814d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.884+0000 7f577d087640 1 --2- 172.21.15.83:0/214542595 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5778152ed0 0x7f577814d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.884+0000 7f577d087640 1 -- 172.21.15.83:0/214542595 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5778152a70 msgr2=0x7f577814cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.884+0000 7f577d087640 1 --2- 172.21.15.83:0/214542595 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5778152a70 0x7f577814cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.884+0000 7f577d087640 1 -- 172.21.15.83:0/214542595 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57781f79c0 con 0x7f5764003f60 2024-03-06T07:21:54.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.884+0000 7f577d888640 1 --2- 172.21.15.83:0/214542595 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5778152ed0 0x7f577814d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:54.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.884+0000 7f577d087640 1 --2- 172.21.15.83:0/214542595 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5764003f60 0x7f5778159660 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5774011990 tx=0x7f5774011e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:54.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.884+0000 7f576dffb640 1 -- 172.21.15.83:0/214542595 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f577400fad0 con 0x7f5764003f60 2024-03-06T07:21:54.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.884+0000 7f576dffb640 1 -- 172.21.15.83:0/214542595 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f577400fc70 con 0x7f5764003f60 2024-03-06T07:21:54.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.884+0000 7f577f2eb640 1 -- 172.21.15.83:0/214542595 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57781f7ca0 con 0x7f5764003f60 2024-03-06T07:21:54.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.884+0000 7f576dffb640 1 -- 172.21.15.83:0/214542595 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5774014440 con 0x7f5764003f60 2024-03-06T07:21:54.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.884+0000 7f577f2eb640 1 -- 172.21.15.83:0/214542595 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57781f8260 con 0x7f5764003f60 2024-03-06T07:21:54.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.885+0000 7f577f2eb640 1 -- 172.21.15.83:0/214542595 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5778068840 con 0x7f5764003f60 2024-03-06T07:21:54.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.886+0000 7f576dffb640 1 -- 172.21.15.83:0/214542595 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5774012040 con 0x7f5764003f60 2024-03-06T07:21:54.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.886+0000 7f576dffb640 1 --2- 172.21.15.83:0/214542595 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f575c06b500 0x7f575c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.886+0000 7f576dffb640 1 -- 172.21.15.83:0/214542595 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f577408a470 con 0x7f5764003f60 2024-03-06T07:21:54.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.886+0000 7f577c886640 1 --2- 172.21.15.83:0/214542595 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f575c06b500 0x7f575c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.887+0000 7f577c886640 1 --2- 172.21.15.83:0/214542595 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f575c06b500 0x7f575c06d9c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5768036000 tx=0x7f5768009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:54.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:54.891+0000 7f576dffb640 1 -- 172.21.15.83:0/214542595 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5774019030 con 0x7f5764003f60 2024-03-06T07:21:55.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.100+0000 7f577f2eb640 1 -- 172.21.15.83:0/214542595 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f577804f330 con 0x7f5764003f60 2024-03-06T07:21:55.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.100+0000 7f576dffb640 1 -- 172.21.15.83:0/214542595 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3697 (secure 0 0 0) 0x7f57740596d0 con 0x7f5764003f60 2024-03-06T07:21:55.103 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:21:55.103 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/4249608353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":4249608353},{"type":"v1","addr":"172.21.15.83:6836","nonce":4249608353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/714452530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":714452530},{"type":"v1","addr":"172.21.15.183:6835","nonce":714452530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4282,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/2723013431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2723013431},{"type":"v1","addr":"172.21.15.183:6834","nonce":2723013431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4287,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1597344199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1597344199},{"type":"v1","addr":"172.21.15.83:6837","nonce":1597344199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-06T07:21:55.103 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 3 2024-03-06T07:21:55.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.102+0000 7f577f2eb640 1 -- 172.21.15.83:0/214542595 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f575c06b500 msgr2=0x7f575c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.102+0000 7f577f2eb640 1 --2- 172.21.15.83:0/214542595 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f575c06b500 0x7f575c06d9c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5768036000 tx=0x7f5768009030 comp rx=0 tx=0).stop 2024-03-06T07:21:55.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.102+0000 7f577f2eb640 1 -- 172.21.15.83:0/214542595 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5764003f60 msgr2=0x7f5778159660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.102+0000 7f577f2eb640 1 --2- 172.21.15.83:0/214542595 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5764003f60 0x7f5778159660 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5774011990 tx=0x7f5774011e50 comp rx=0 tx=0).stop 2024-03-06T07:21:55.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.102+0000 7f577f2eb640 1 -- 172.21.15.83:0/214542595 shutdown_connections 2024-03-06T07:21:55.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.103+0000 7f577f2eb640 1 --2- 172.21.15.83:0/214542595 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f575c06b500 0x7f575c06d9c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.103+0000 7f577f2eb640 1 --2- 172.21.15.83:0/214542595 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5778152ed0 0x7f577814d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.103+0000 7f577f2eb640 1 --2- 172.21.15.83:0/214542595 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5778152a70 0x7f577814cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.103+0000 7f577f2eb640 1 --2- 172.21.15.83:0/214542595 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5764003f60 0x7f5778159660 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.103+0000 7f577f2eb640 1 -- 172.21.15.83:0/214542595 >> 172.21.15.83:0/214542595 conn(0x7f57780cf8a0 msgr2=0x7f5778072210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:55.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.103+0000 7f577f2eb640 1 -- 172.21.15.83:0/214542595 shutdown_connections 2024-03-06T07:21:55.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.103+0000 7f577f2eb640 1 -- 172.21.15.83:0/214542595 wait complete. 2024-03-06T07:21:55.116 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:55.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.214+0000 7f6df110c640 1 Processor -- start 2024-03-06T07:21:55.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.214+0000 7f6df110c640 1 -- start start 2024-03-06T07:21:55.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.215+0000 7f6df110c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6dec0d06a0 0x7f6dec0cf850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.215+0000 7f6df110c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dec0cef50 con 0x7f6dec0b6550 2024-03-06T07:21:55.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.215+0000 7f6df110c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dec15c620 con 0x7f6dec0d02c0 2024-03-06T07:21:55.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.215+0000 7f6df110c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dec0b50b0 con 0x7f6dec0d06a0 2024-03-06T07:21:55.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.215+0000 7f6dea59c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f6dec0b6550 0x7f6dec0ce840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52350/0 (socket says 172.21.15.83:52350) 2024-03-06T07:21:55.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.215+0000 7f6dea59c640 1 -- 172.21.15.83:0/135877411 learned_addr learned my addr 172.21.15.83:0/135877411 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:21:55.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.215+0000 7f6deb59e640 1 --2- 172.21.15.83:0/135877411 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6dec0d06a0 0x7f6dec0cf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.215+0000 7f6de9d9b640 1 -- 172.21.15.83:0/135877411 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3088112304 0 0) 0x7f6dec0cef50 con 0x7f6dec0b6550 2024-03-06T07:21:55.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.215+0000 7f6de9d9b640 1 -- 172.21.15.83:0/135877411 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dd4003520 con 0x7f6dec0b6550 2024-03-06T07:21:55.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.215+0000 7f6de9d9b640 1 -- 172.21.15.83:0/135877411 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1416662875 0 0) 0x7f6dd4003520 con 0x7f6dec0b6550 2024-03-06T07:21:55.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.216+0000 7f6de9d9b640 1 -- 172.21.15.83:0/135877411 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6dec0d06a0 msgr2=0x7f6dec0cf850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:55.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.216+0000 7f6de9d9b640 1 --2- 172.21.15.83:0/135877411 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6dec0d06a0 0x7f6dec0cf850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.216+0000 7f6de9d9b640 1 -- 172.21.15.83:0/135877411 >> v1:172.21.15.183:6789/0 conn(0x7f6dec0d02c0 legacy=0x7f6dec0ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.216+0000 7f6de9d9b640 1 -- 172.21.15.83:0/135877411 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dec0cd800 con 0x7f6dec0b6550 2024-03-06T07:21:55.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.216+0000 7f6de9d9b640 1 -- 172.21.15.83:0/135877411 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f6de0001670 con 0x7f6dec0b6550 2024-03-06T07:21:55.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.216+0000 7f6de9d9b640 1 -- 172.21.15.83:0/135877411 >> v1:172.21.15.83:6789/0 conn(0x7f6dec0b6550 legacy=0x7f6dec0ce840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.216+0000 7f6de9d9b640 1 --2- 172.21.15.83:0/135877411 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6dec0d06a0 0x7f6dd40039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.216+0000 7f6de9d9b640 1 --2- 172.21.15.83:0/135877411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6dd4003f20 0x7f6dd4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.216+0000 7f6de9d9b640 1 --2- 172.21.15.83:0/135877411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6dd4004d40 0x7f6dd40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.216+0000 7f6de9d9b640 1 -- 172.21.15.83:0/135877411 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dec0b50b0 con 0x7f6dd4004d40 2024-03-06T07:21:55.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.216+0000 7f6de9d9b640 1 -- 172.21.15.83:0/135877411 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dd400bad0 con 0x7f6dd4003f20 2024-03-06T07:21:55.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.216+0000 7f6de9d9b640 1 -- 172.21.15.83:0/135877411 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dd400bcd0 con 0x7f6dec0d06a0 2024-03-06T07:21:55.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.216+0000 7f6deb59e640 1 --2- 172.21.15.83:0/135877411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6dd4004d40 0x7f6dd40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.216+0000 7f6dea59c640 1 --2- 172.21.15.83:0/135877411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6dd4003f20 0x7f6dd4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.216+0000 7f6dead9d640 1 --2- 172.21.15.83:0/135877411 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6dec0d06a0 0x7f6dd40039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.216+0000 7f6deb59e640 1 -- 172.21.15.83:0/135877411 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6dec0d06a0 msgr2=0x7f6dd40039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:55.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.216+0000 7f6deb59e640 1 --2- 172.21.15.83:0/135877411 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6dec0d06a0 0x7f6dd40039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.216+0000 7f6deb59e640 1 -- 172.21.15.83:0/135877411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6dd4003f20 msgr2=0x7f6dd4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:55.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.216+0000 7f6deb59e640 1 --2- 172.21.15.83:0/135877411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6dd4003f20 0x7f6dd4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.216+0000 7f6deb59e640 1 -- 172.21.15.83:0/135877411 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dd4003520 con 0x7f6dd4004d40 2024-03-06T07:21:55.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.217+0000 7f6deb59e640 1 --2- 172.21.15.83:0/135877411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6dd4004d40 0x7f6dd40051c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6de4009ac0 tx=0x7f6dd4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a07534a25be43ec server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.217+0000 7f6de9d9b640 1 -- 172.21.15.83:0/135877411 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de4039070 con 0x7f6dd4004d40 2024-03-06T07:21:55.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.217+0000 7f6de9d9b640 1 -- 172.21.15.83:0/135877411 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6de402c6b0 con 0x7f6dd4004d40 2024-03-06T07:21:55.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.217+0000 7f6de9d9b640 1 -- 172.21.15.83:0/135877411 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de4009690 con 0x7f6dd4004d40 2024-03-06T07:21:55.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.217+0000 7f6df110c640 1 -- 172.21.15.83:0/135877411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6dd4004d40 msgr2=0x7f6dd40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.217+0000 7f6df110c640 1 --2- 172.21.15.83:0/135877411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6dd4004d40 0x7f6dd40051c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6de4009ac0 tx=0x7f6dd4005830 comp rx=0 tx=0).stop 2024-03-06T07:21:55.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.217+0000 7f6deb59e640 1 -- 172.21.15.83:0/135877411 reap_dead start 2024-03-06T07:21:55.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.217+0000 7f6df110c640 1 -- 172.21.15.83:0/135877411 shutdown_connections 2024-03-06T07:21:55.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.217+0000 7f6df110c640 1 -- 172.21.15.83:0/135877411 >> 172.21.15.83:0/135877411 conn(0x7f6dec0b35a0 msgr2=0x7f6dec0b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:55.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.217+0000 7f6df110c640 1 -- 172.21.15.83:0/135877411 shutdown_connections 2024-03-06T07:21:55.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.217+0000 7f6df110c640 1 -- 172.21.15.83:0/135877411 wait complete. 2024-03-06T07:21:55.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.219+0000 7f6df110c640 1 Processor -- start 2024-03-06T07:21:55.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.219+0000 7f6df110c640 1 -- start start 2024-03-06T07:21:55.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.220+0000 7f6df110c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6dec14d050 0x7f6dec0cab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.220+0000 7f6df110c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6dec0cb0d0 0x7f6dec0c6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.220+0000 7f6df110c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6dec0c6da0 0x7f6dec0c8b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.220+0000 7f6df110c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dec0cf850 con 0x7f6dec14d050 2024-03-06T07:21:55.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.220+0000 7f6df110c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dec0b4d40 con 0x7f6dec0c6da0 2024-03-06T07:21:55.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.220+0000 7f6df110c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dec0cf9d0 con 0x7f6dec0cb0d0 2024-03-06T07:21:55.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.220+0000 7f6dead9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6dec14d050 0x7f6dec0cab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.220+0000 7f6dead9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6dec14d050 0x7f6dec0cab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51502/0 (socket says 172.21.15.83:51502) 2024-03-06T07:21:55.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.220+0000 7f6dead9d640 1 -- 172.21.15.83:0/348056921 learned_addr learned my addr 172.21.15.83:0/348056921 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:21:55.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.220+0000 7f6deb59e640 1 --2- 172.21.15.83:0/348056921 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6dec0c6da0 0x7f6dec0c8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.220+0000 7f6dea59c640 1 --2- 172.21.15.83:0/348056921 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6dec0cb0d0 0x7f6dec0c6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.220+0000 7f6dead9d640 1 -- 172.21.15.83:0/348056921 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6dec0cb0d0 msgr2=0x7f6dec0c6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.220+0000 7f6dead9d640 1 --2- 172.21.15.83:0/348056921 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6dec0cb0d0 0x7f6dec0c6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.220+0000 7f6dead9d640 1 -- 172.21.15.83:0/348056921 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6dec0c6da0 msgr2=0x7f6dec0c8b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.220+0000 7f6dead9d640 1 --2- 172.21.15.83:0/348056921 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6dec0c6da0 0x7f6dec0c8b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.220+0000 7f6dead9d640 1 -- 172.21.15.83:0/348056921 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dec0c7600 con 0x7f6dec14d050 2024-03-06T07:21:55.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.220+0000 7f6deb59e640 1 --2- 172.21.15.83:0/348056921 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6dec0c6da0 0x7f6dec0c8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:55.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.220+0000 7f6dea59c640 1 --2- 172.21.15.83:0/348056921 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6dec0cb0d0 0x7f6dec0c6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:55.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.220+0000 7f6dead9d640 1 --2- 172.21.15.83:0/348056921 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6dec14d050 0x7f6dec0cab90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc00fbc0 tx=0x7f6ddc00d520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.220+0000 7f6dd37fe640 1 -- 172.21.15.83:0/348056921 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ddc00dc30 con 0x7f6dec14d050 2024-03-06T07:21:55.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.220+0000 7f6dd37fe640 1 -- 172.21.15.83:0/348056921 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ddc018070 con 0x7f6dec14d050 2024-03-06T07:21:55.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.220+0000 7f6dd37fe640 1 -- 172.21.15.83:0/348056921 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ddc003970 con 0x7f6dec14d050 2024-03-06T07:21:55.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.220+0000 7f6df110c640 1 -- 172.21.15.83:0/348056921 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dec0c78e0 con 0x7f6dec14d050 2024-03-06T07:21:55.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.220+0000 7f6df110c640 1 -- 172.21.15.83:0/348056921 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dec0c7ea0 con 0x7f6dec14d050 2024-03-06T07:21:55.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.221+0000 7f6df110c640 1 -- 172.21.15.83:0/348056921 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dec04f330 con 0x7f6dec14d050 2024-03-06T07:21:55.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.222+0000 7f6dd37fe640 1 -- 172.21.15.83:0/348056921 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6ddc003b30 con 0x7f6dec14d050 2024-03-06T07:21:55.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.222+0000 7f6dd37fe640 1 --2- 172.21.15.83:0/348056921 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6dd806b500 0x7f6dd806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.222+0000 7f6dea59c640 1 --2- 172.21.15.83:0/348056921 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6dd806b500 0x7f6dd806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.222+0000 7f6dd37fe640 1 -- 172.21.15.83:0/348056921 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6ddc08a1e0 con 0x7f6dec14d050 2024-03-06T07:21:55.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.223+0000 7f6dea59c640 1 --2- 172.21.15.83:0/348056921 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6dd806b500 0x7f6dd806d9c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6dec0c54b0 tx=0x7f6de0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.226+0000 7f6dd37fe640 1 -- 172.21.15.83:0/348056921 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6ddc003e20 con 0x7f6dec14d050 2024-03-06T07:21:55.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.398+0000 7f6df110c640 1 -- 172.21.15.83:0/348056921 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6dec0cb4b0 con 0x7f6dec14d050 2024-03-06T07:21:55.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.399+0000 7f6dd37fe640 1 -- 172.21.15.83:0/348056921 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12896 (secure 0 0 0) 0x7f6ddc0593b0 con 0x7f6dec14d050 2024-03-06T07:21:55.401 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:21:55.401 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":16,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:21:48.365486+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:55.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.401+0000 7f6df110c640 1 -- 172.21.15.83:0/348056921 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6dd806b500 msgr2=0x7f6dd806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.401+0000 7f6df110c640 1 --2- 172.21.15.83:0/348056921 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6dd806b500 0x7f6dd806d9c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6dec0c54b0 tx=0x7f6de0009040 comp rx=0 tx=0).stop 2024-03-06T07:21:55.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.401+0000 7f6df110c640 1 -- 172.21.15.83:0/348056921 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6dec14d050 msgr2=0x7f6dec0cab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.401+0000 7f6df110c640 1 --2- 172.21.15.83:0/348056921 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6dec14d050 0x7f6dec0cab90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc00fbc0 tx=0x7f6ddc00d520 comp rx=0 tx=0).stop 2024-03-06T07:21:55.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.401+0000 7f6df110c640 1 -- 172.21.15.83:0/348056921 shutdown_connections 2024-03-06T07:21:55.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.401+0000 7f6df110c640 1 --2- 172.21.15.83:0/348056921 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6dd806b500 0x7f6dd806d9c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.401+0000 7f6df110c640 1 --2- 172.21.15.83:0/348056921 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6dec0c6da0 0x7f6dec0c8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.401+0000 7f6df110c640 1 --2- 172.21.15.83:0/348056921 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6dec0cb0d0 0x7f6dec0c6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.401+0000 7f6df110c640 1 --2- 172.21.15.83:0/348056921 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6dec14d050 0x7f6dec0cab90 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.401+0000 7f6df110c640 1 -- 172.21.15.83:0/348056921 >> 172.21.15.83:0/348056921 conn(0x7f6dec0b35a0 msgr2=0x7f6dec0ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:55.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.401+0000 7f6df110c640 1 -- 172.21.15.83:0/348056921 shutdown_connections 2024-03-06T07:21:55.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.401+0000 7f6df110c640 1 -- 172.21.15.83:0/348056921 wait complete. 2024-03-06T07:21:55.414 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:21:55.414 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:21:55.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.515+0000 7f2e147e9640 1 Processor -- start 2024-03-06T07:21:55.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.515+0000 7f2e147e9640 1 -- start start 2024-03-06T07:21:55.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.516+0000 7f2e147e9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2e0c06c860 0x7f2e0c1619d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.516+0000 7f2e147e9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2e0c161f10 0x7f2e0c15f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.516+0000 7f2e147e9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e0c06cd20 con 0x7f2e0c1614f0 2024-03-06T07:21:55.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.516+0000 7f2e147e9640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e0c06d000 con 0x7f2e0c161f10 2024-03-06T07:21:55.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.516+0000 7f2e147e9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e0c06d2e0 con 0x7f2e0c06c860 2024-03-06T07:21:55.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.516+0000 7f2e12d86640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2e0c161f10 0x7f2e0c15f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.516+0000 7f2e12d86640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2e0c161f10 0x7f2e0c15f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51510/0 (socket says 172.21.15.83:51510) 2024-03-06T07:21:55.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.516+0000 7f2e12d86640 1 -- 172.21.15.83:0/89370702 learned_addr learned my addr 172.21.15.83:0/89370702 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:21:55.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.517+0000 7f2e11d84640 1 --2- 172.21.15.83:0/89370702 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2e0c06c860 0x7f2e0c1619d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.517+0000 7f2e12d86640 1 -- 172.21.15.83:0/89370702 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2e0c06c860 msgr2=0x7f2e0c1619d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:55.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.517+0000 7f2e12d86640 1 --2- 172.21.15.83:0/89370702 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2e0c06c860 0x7f2e0c1619d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.517+0000 7f2e12d86640 1 -- 172.21.15.83:0/89370702 >> v1:172.21.15.183:6789/0 conn(0x7f2e0c1614f0 legacy=0x7f2e0c0c77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.517+0000 7f2e12d86640 1 -- 172.21.15.83:0/89370702 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e0c06d5c0 con 0x7f2e0c161f10 2024-03-06T07:21:55.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.517+0000 7f2e11583640 1 -- 172.21.15.83:0/89370702 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 764810463 0 0) 0x7f2e0c06cd20 con 0x7f2e0c1614f0 2024-03-06T07:21:55.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.517+0000 7f2e12d86640 1 --2- 172.21.15.83:0/89370702 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2e0c161f10 0x7f2e0c15f5a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2e04009870 tx=0x7f2e0402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39028362c3f66165 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.518+0000 7f2e11583640 1 -- 172.21.15.83:0/89370702 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e04035020 con 0x7f2e0c161f10 2024-03-06T07:21:55.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.518+0000 7f2e11583640 1 -- 172.21.15.83:0/89370702 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e04031640 con 0x7f2e0c161f10 2024-03-06T07:21:55.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.518+0000 7f2e11583640 1 -- 172.21.15.83:0/89370702 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e0402d040 con 0x7f2e0c161f10 2024-03-06T07:21:55.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.518+0000 7f2e147e9640 1 -- 172.21.15.83:0/89370702 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2e0c161f10 msgr2=0x7f2e0c15f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.518+0000 7f2e147e9640 1 --2- 172.21.15.83:0/89370702 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2e0c161f10 0x7f2e0c15f5a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2e04009870 tx=0x7f2e0402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.518+0000 7f2e147e9640 1 -- 172.21.15.83:0/89370702 shutdown_connections 2024-03-06T07:21:55.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.518+0000 7f2e147e9640 1 --2- 172.21.15.83:0/89370702 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2e0c161f10 0x7f2e0c15f5a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.518+0000 7f2e147e9640 1 --2- 172.21.15.83:0/89370702 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2e0c06c860 0x7f2e0c1619d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.518+0000 7f2e147e9640 1 -- 172.21.15.83:0/89370702 >> 172.21.15.83:0/89370702 conn(0x7f2e0c0d1b40 msgr2=0x7f2e0c0d1f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:55.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.518+0000 7f2e147e9640 1 -- 172.21.15.83:0/89370702 shutdown_connections 2024-03-06T07:21:55.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.518+0000 7f2e147e9640 1 -- 172.21.15.83:0/89370702 wait complete. 2024-03-06T07:21:55.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.519+0000 7f2e147e9640 1 Processor -- start 2024-03-06T07:21:55.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.519+0000 7f2e147e9640 1 -- start start 2024-03-06T07:21:55.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.520+0000 7f2e147e9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2e0c06c860 0x7f2e0c0b74f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.520+0000 7f2e147e9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2e0c1614f0 0x7f2e0c1f3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.520+0000 7f2e147e9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2e0c161f10 0x7f2e0c1f43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.520+0000 7f2e147e9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e0c15fae0 con 0x7f2e0c06c860 2024-03-06T07:21:55.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.520+0000 7f2e147e9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e0c0d28f0 con 0x7f2e0c161f10 2024-03-06T07:21:55.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.520+0000 7f2e147e9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e0c15fc60 con 0x7f2e0c1614f0 2024-03-06T07:21:55.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.520+0000 7f2e12585640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2e0c06c860 0x7f2e0c0b74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.520+0000 7f2e12585640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2e0c06c860 0x7f2e0c0b74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51512/0 (socket says 172.21.15.83:51512) 2024-03-06T07:21:55.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.520+0000 7f2e12585640 1 -- 172.21.15.83:0/1075957586 learned_addr learned my addr 172.21.15.83:0/1075957586 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:21:55.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.520+0000 7f2e12d86640 1 --2- 172.21.15.83:0/1075957586 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2e0c161f10 0x7f2e0c1f43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.520+0000 7f2e11d84640 1 --2- 172.21.15.83:0/1075957586 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2e0c1614f0 0x7f2e0c1f3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.520+0000 7f2e12585640 1 -- 172.21.15.83:0/1075957586 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2e0c1614f0 msgr2=0x7f2e0c1f3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.520+0000 7f2e12585640 1 --2- 172.21.15.83:0/1075957586 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2e0c1614f0 0x7f2e0c1f3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.520+0000 7f2e12585640 1 -- 172.21.15.83:0/1075957586 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2e0c161f10 msgr2=0x7f2e0c1f43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.520+0000 7f2e12585640 1 --2- 172.21.15.83:0/1075957586 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2e0c161f10 0x7f2e0c1f43a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.520+0000 7f2e12585640 1 -- 172.21.15.83:0/1075957586 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e0c0b7a30 con 0x7f2e0c06c860 2024-03-06T07:21:55.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.520+0000 7f2e12d86640 1 --2- 172.21.15.83:0/1075957586 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2e0c161f10 0x7f2e0c1f43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:55.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.520+0000 7f2e11d84640 1 --2- 172.21.15.83:0/1075957586 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2e0c1614f0 0x7f2e0c1f3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:55.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.520+0000 7f2e12585640 1 --2- 172.21.15.83:0/1075957586 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2e0c06c860 0x7f2e0c0b74f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2e0800c890 tx=0x7f2e0800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.521+0000 7f2e02ffd640 1 -- 172.21.15.83:0/1075957586 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e08017070 con 0x7f2e0c06c860 2024-03-06T07:21:55.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.521+0000 7f2e02ffd640 1 -- 172.21.15.83:0/1075957586 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e08003dc0 con 0x7f2e0c06c860 2024-03-06T07:21:55.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.521+0000 7f2e147e9640 1 -- 172.21.15.83:0/1075957586 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e0c0b4e40 con 0x7f2e0c06c860 2024-03-06T07:21:55.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.521+0000 7f2e02ffd640 1 -- 172.21.15.83:0/1075957586 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e080052b0 con 0x7f2e0c06c860 2024-03-06T07:21:55.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.521+0000 7f2e147e9640 1 -- 172.21.15.83:0/1075957586 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e0c0b5400 con 0x7f2e0c06c860 2024-03-06T07:21:55.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.522+0000 7f2e147e9640 1 -- 172.21.15.83:0/1075957586 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e0c15ac20 con 0x7f2e0c06c860 2024-03-06T07:21:55.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.522+0000 7f2e02ffd640 1 -- 172.21.15.83:0/1075957586 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2e0801a070 con 0x7f2e0c06c860 2024-03-06T07:21:55.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.522+0000 7f2e02ffd640 1 --2- 172.21.15.83:0/1075957586 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2df006b500 0x7f2df006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.522+0000 7f2e02ffd640 1 -- 172.21.15.83:0/1075957586 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2e08089520 con 0x7f2e0c06c860 2024-03-06T07:21:55.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.522+0000 7f2e11d84640 1 --2- 172.21.15.83:0/1075957586 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2df006b500 0x7f2df006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.523+0000 7f2e11d84640 1 --2- 172.21.15.83:0/1075957586 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2df006b500 0x7f2df006d9c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c0b3f50 tx=0x7f2dfc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.527+0000 7f2e02ffd640 1 -- 172.21.15.83:0/1075957586 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2e08058780 con 0x7f2e0c06c860 2024-03-06T07:21:55.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.699+0000 7f2e147e9640 1 -- 172.21.15.83:0/1075957586 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2e0c0c82e0 con 0x7f2e0c06c860 2024-03-06T07:21:55.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.700+0000 7f2e02ffd640 1 -- 172.21.15.83:0/1075957586 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2e0c0c82e0 con 0x7f2e0c06c860 2024-03-06T07:21:55.702 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:21:55.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.702+0000 7f2e147e9640 1 -- 172.21.15.83:0/1075957586 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2df006b500 msgr2=0x7f2df006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.702+0000 7f2e147e9640 1 --2- 172.21.15.83:0/1075957586 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2df006b500 0x7f2df006d9c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c0b3f50 tx=0x7f2dfc008040 comp rx=0 tx=0).stop 2024-03-06T07:21:55.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.702+0000 7f2e147e9640 1 -- 172.21.15.83:0/1075957586 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2e0c06c860 msgr2=0x7f2e0c0b74f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.702+0000 7f2e147e9640 1 --2- 172.21.15.83:0/1075957586 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2e0c06c860 0x7f2e0c0b74f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2e0800c890 tx=0x7f2e0800cd50 comp rx=0 tx=0).stop 2024-03-06T07:21:55.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.702+0000 7f2e147e9640 1 -- 172.21.15.83:0/1075957586 shutdown_connections 2024-03-06T07:21:55.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.702+0000 7f2e147e9640 1 --2- 172.21.15.83:0/1075957586 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2df006b500 0x7f2df006d9c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.702+0000 7f2e147e9640 1 --2- 172.21.15.83:0/1075957586 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2e0c161f10 0x7f2e0c1f43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.702+0000 7f2e147e9640 1 --2- 172.21.15.83:0/1075957586 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2e0c1614f0 0x7f2e0c1f3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.702+0000 7f2e147e9640 1 --2- 172.21.15.83:0/1075957586 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2e0c06c860 0x7f2e0c0b74f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.702+0000 7f2e147e9640 1 -- 172.21.15.83:0/1075957586 >> 172.21.15.83:0/1075957586 conn(0x7f2e0c0d1b40 msgr2=0x7f2e0c15c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:55.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.702+0000 7f2e147e9640 1 -- 172.21.15.83:0/1075957586 shutdown_connections 2024-03-06T07:21:55.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.702+0000 7f2e147e9640 1 -- 172.21.15.83:0/1075957586 wait complete. 2024-03-06T07:21:55.714 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:21:55.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.812+0000 7f43841c9640 1 Processor -- start 2024-03-06T07:21:55.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.812+0000 7f43841c9640 1 -- start start 2024-03-06T07:21:55.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.813+0000 7f43841c9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f437c0d0700 0x7f437c0b0970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.813+0000 7f43841c9640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f437c0b0eb0 0x7f437c0b1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.813+0000 7f43841c9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f437c15c270 con 0x7f437c0d0320 2024-03-06T07:21:55.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.813+0000 7f43841c9640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f437c0b1870 con 0x7f437c0b0eb0 2024-03-06T07:21:55.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.813+0000 7f43841c9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f437c0b04e0 con 0x7f437c0d0700 2024-03-06T07:21:55.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.813+0000 7f4381764640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f437c0d0700 0x7f437c0b0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.813+0000 7f4382766640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f437c0b0eb0 0x7f437c0b1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.813+0000 7f4381764640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f437c0d0700 0x7f437c0b0970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:37164/0 (socket says 172.21.15.83:37164) 2024-03-06T07:21:55.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.813+0000 7f4381764640 1 -- 172.21.15.83:0/12214092 learned_addr learned my addr 172.21.15.83:0/12214092 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:21:55.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.813+0000 7f4381f65640 1 --1- 172.21.15.83:0/12214092 >> v1:172.21.15.183:6789/0 conn(0x7f437c0d0320 0x7f437c0b4f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:33084/0 (socket says 172.21.15.83:33084) 2024-03-06T07:21:55.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.813+0000 7f4382766640 1 -- 172.21.15.83:0/12214092 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f437c0d0700 msgr2=0x7f437c0b0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:55.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.813+0000 7f4382766640 1 --2- 172.21.15.83:0/12214092 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f437c0d0700 0x7f437c0b0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.813+0000 7f4382766640 1 -- 172.21.15.83:0/12214092 >> v1:172.21.15.183:6789/0 conn(0x7f437c0d0320 legacy=0x7f437c0b4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.813+0000 7f4382766640 1 -- 172.21.15.83:0/12214092 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f437c0d0e90 con 0x7f437c0b0eb0 2024-03-06T07:21:55.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.813+0000 7f4381764640 1 --2- 172.21.15.83:0/12214092 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f437c0d0700 0x7f437c0b0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:55.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.814+0000 7f4382766640 1 --2- 172.21.15.83:0/12214092 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f437c0b0eb0 0x7f437c0b1330 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4374009d90 tx=0x7f437402bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=122a6a1d74901950 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.814+0000 7f4380f63640 1 -- 172.21.15.83:0/12214092 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4374039070 con 0x7f437c0b0eb0 2024-03-06T07:21:55.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.814+0000 7f4380f63640 1 -- 172.21.15.83:0/12214092 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f437402cce0 con 0x7f437c0b0eb0 2024-03-06T07:21:55.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.814+0000 7f4380f63640 1 -- 172.21.15.83:0/12214092 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4374034660 con 0x7f437c0b0eb0 2024-03-06T07:21:55.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.815+0000 7f43841c9640 1 -- 172.21.15.83:0/12214092 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f437c0b0eb0 msgr2=0x7f437c0b1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.815+0000 7f43841c9640 1 --2- 172.21.15.83:0/12214092 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f437c0b0eb0 0x7f437c0b1330 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4374009d90 tx=0x7f437402bf30 comp rx=0 tx=0).stop 2024-03-06T07:21:55.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.815+0000 7f43841c9640 1 -- 172.21.15.83:0/12214092 shutdown_connections 2024-03-06T07:21:55.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.815+0000 7f43841c9640 1 --2- 172.21.15.83:0/12214092 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f437c0b0eb0 0x7f437c0b1330 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.815+0000 7f43841c9640 1 --2- 172.21.15.83:0/12214092 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f437c0d0700 0x7f437c0b0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.815+0000 7f43841c9640 1 -- 172.21.15.83:0/12214092 >> 172.21.15.83:0/12214092 conn(0x7f437c0724d0 msgr2=0x7f437c0728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:55.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.815+0000 7f43841c9640 1 -- 172.21.15.83:0/12214092 shutdown_connections 2024-03-06T07:21:55.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.815+0000 7f43841c9640 1 -- 172.21.15.83:0/12214092 wait complete. 2024-03-06T07:21:55.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.816+0000 7f43841c9640 1 Processor -- start 2024-03-06T07:21:55.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.816+0000 7f43841c9640 1 -- start start 2024-03-06T07:21:55.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.817+0000 7f43841c9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f437c0b0eb0 0x7f437c14a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.817+0000 7f43841c9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f437c0d0320 0x7f437c0ae6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.817+0000 7f43841c9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f437c0d0700 0x7f437c0c5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.817+0000 7f43841c9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f437c0b0170 con 0x7f437c0b0eb0 2024-03-06T07:21:55.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.817+0000 7f43841c9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f437c0b5dc0 con 0x7f437c0d0320 2024-03-06T07:21:55.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.817+0000 7f43841c9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f437c0b60a0 con 0x7f437c0d0700 2024-03-06T07:21:55.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.817+0000 7f4381f65640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f437c0b0eb0 0x7f437c14a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.817+0000 7f4381f65640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f437c0b0eb0 0x7f437c14a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51524/0 (socket says 172.21.15.83:51524) 2024-03-06T07:21:55.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.817+0000 7f4381f65640 1 -- 172.21.15.83:0/250957771 learned_addr learned my addr 172.21.15.83:0/250957771 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:21:55.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.817+0000 7f4381764640 1 --2- 172.21.15.83:0/250957771 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f437c0d0320 0x7f437c0ae6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.817+0000 7f4382766640 1 --2- 172.21.15.83:0/250957771 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f437c0d0700 0x7f437c0c5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.817+0000 7f4381f65640 1 -- 172.21.15.83:0/250957771 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f437c0d0700 msgr2=0x7f437c0c5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.817+0000 7f4381f65640 1 --2- 172.21.15.83:0/250957771 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f437c0d0700 0x7f437c0c5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.817+0000 7f4381f65640 1 -- 172.21.15.83:0/250957771 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f437c0d0320 msgr2=0x7f437c0ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.817+0000 7f4381f65640 1 --2- 172.21.15.83:0/250957771 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f437c0d0320 0x7f437c0ae6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.817+0000 7f4381f65640 1 -- 172.21.15.83:0/250957771 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f437c09d070 con 0x7f437c0b0eb0 2024-03-06T07:21:55.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.817+0000 7f4381764640 1 --2- 172.21.15.83:0/250957771 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f437c0d0320 0x7f437c0ae6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:55.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.817+0000 7f4381f65640 1 --2- 172.21.15.83:0/250957771 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f437c0b0eb0 0x7f437c14a450 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f437800c400 tx=0x7f437800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.818+0000 7f43727fc640 1 -- 172.21.15.83:0/250957771 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43780074e0 con 0x7f437c0b0eb0 2024-03-06T07:21:55.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.818+0000 7f43727fc640 1 -- 172.21.15.83:0/250957771 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4378007680 con 0x7f437c0b0eb0 2024-03-06T07:21:55.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.818+0000 7f43841c9640 1 -- 172.21.15.83:0/250957771 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f437c09d350 con 0x7f437c0b0eb0 2024-03-06T07:21:55.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.818+0000 7f43727fc640 1 -- 172.21.15.83:0/250957771 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4378004a00 con 0x7f437c0b0eb0 2024-03-06T07:21:55.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.818+0000 7f43841c9640 1 -- 172.21.15.83:0/250957771 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f437c09d910 con 0x7f437c0b0eb0 2024-03-06T07:21:55.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.818+0000 7f43841c9640 1 -- 172.21.15.83:0/250957771 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f437c09de00 con 0x7f437c0b0eb0 2024-03-06T07:21:55.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.819+0000 7f43727fc640 1 -- 172.21.15.83:0/250957771 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f437801a020 con 0x7f437c0b0eb0 2024-03-06T07:21:55.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.819+0000 7f43727fc640 1 --2- 172.21.15.83:0/250957771 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f436006b4d0 0x7f436006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.820+0000 7f4381764640 1 --2- 172.21.15.83:0/250957771 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f436006b4d0 0x7f436006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.820+0000 7f43727fc640 1 -- 172.21.15.83:0/250957771 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f43780896d0 con 0x7f437c0b0eb0 2024-03-06T07:21:55.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.820+0000 7f4381764640 1 --2- 172.21.15.83:0/250957771 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f436006b4d0 0x7f436006d990 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f437c15b210 tx=0x7f436c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.824+0000 7f43727fc640 1 -- 172.21.15.83:0/250957771 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f43780588a0 con 0x7f437c0b0eb0 2024-03-06T07:21:56.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.998+0000 7f43841c9640 1 -- 172.21.15.83:0/250957771 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f437c04f330 con 0x7f437c0b0eb0 2024-03-06T07:21:56.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:55.999+0000 7f43727fc640 1 -- 172.21.15.83:0/250957771 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+65 (secure 0 0 0) 0x7f43780581a0 con 0x7f437c0b0eb0 2024-03-06T07:21:56.001 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:21:56.001 INFO:teuthology.orchestra.run.smithi083.stdout:[ 2024-03-06T07:21:56.001 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:21:56.001 INFO:teuthology.orchestra.run.smithi083.stdout: "poolnum": 1, 2024-03-06T07:21:56.001 INFO:teuthology.orchestra.run.smithi083.stdout: "poolname": ".mgr" 2024-03-06T07:21:56.001 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:21:56.001 INFO:teuthology.orchestra.run.smithi083.stdout:] 2024-03-06T07:21:56.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.001+0000 7f43841c9640 1 -- 172.21.15.83:0/250957771 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f436006b4d0 msgr2=0x7f436006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.001+0000 7f43841c9640 1 --2- 172.21.15.83:0/250957771 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f436006b4d0 0x7f436006d990 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f437c15b210 tx=0x7f436c008040 comp rx=0 tx=0).stop 2024-03-06T07:21:56.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.001+0000 7f43841c9640 1 -- 172.21.15.83:0/250957771 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f437c0b0eb0 msgr2=0x7f437c14a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.001+0000 7f43841c9640 1 --2- 172.21.15.83:0/250957771 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f437c0b0eb0 0x7f437c14a450 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f437800c400 tx=0x7f437800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.001+0000 7f43841c9640 1 -- 172.21.15.83:0/250957771 shutdown_connections 2024-03-06T07:21:56.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.001+0000 7f43841c9640 1 --2- 172.21.15.83:0/250957771 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f436006b4d0 0x7f436006d990 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.001+0000 7f43841c9640 1 --2- 172.21.15.83:0/250957771 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f437c0d0700 0x7f437c0c5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.001+0000 7f43841c9640 1 --2- 172.21.15.83:0/250957771 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f437c0d0320 0x7f437c0ae6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.001+0000 7f43841c9640 1 --2- 172.21.15.83:0/250957771 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f437c0b0eb0 0x7f437c14a450 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.001+0000 7f43841c9640 1 -- 172.21.15.83:0/250957771 >> 172.21.15.83:0/250957771 conn(0x7f437c0724d0 msgr2=0x7f437c0b2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:56.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.001+0000 7f43841c9640 1 -- 172.21.15.83:0/250957771 shutdown_connections 2024-03-06T07:21:56.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.001+0000 7f43841c9640 1 -- 172.21.15.83:0/250957771 wait complete. 2024-03-06T07:21:56.013 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:21:56.013 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:21:56.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.115+0000 7f59944b5640 1 Processor -- start 2024-03-06T07:21:56.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.115+0000 7f59944b5640 1 -- start start 2024-03-06T07:21:56.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.116+0000 7f59944b5640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f598c161440 0x7f598c0b4be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.116+0000 7f59944b5640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f598c161820 0x7f598c0d0610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.116+0000 7f59944b5640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f598c0d1300 0x7f598c0d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.116+0000 7f59944b5640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f598c0cf2d0 con 0x7f598c161440 2024-03-06T07:21:56.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.116+0000 7f59944b5640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f598c0b6b00 con 0x7f598c161820 2024-03-06T07:21:56.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.116+0000 7f59944b5640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f598c0cf520 con 0x7f598c0d1300 2024-03-06T07:21:56.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.116+0000 7f5992251640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f598c161440 0x7f598c0b4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.116+0000 7f5992251640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f598c161440 0x7f598c0b4be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51540/0 (socket says 172.21.15.83:51540) 2024-03-06T07:21:56.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.116+0000 7f5992251640 1 -- 172.21.15.83:0/48435526 learned_addr learned my addr 172.21.15.83:0/48435526 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:21:56.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.116+0000 7f5991a50640 1 --2- 172.21.15.83:0/48435526 >> v2:172.21.15.183:3300/0 conn(0x7f598c161820 0x7f598c0d0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.116+0000 7f5992a52640 1 --2- 172.21.15.83:0/48435526 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f598c0d1300 0x7f598c0d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.116+0000 7f5992251640 1 -- 172.21.15.83:0/48435526 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f598c0d1300 msgr2=0x7f598c0d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:56.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.116+0000 7f5992251640 1 --2- 172.21.15.83:0/48435526 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f598c0d1300 0x7f598c0d0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.116+0000 7f5992251640 1 -- 172.21.15.83:0/48435526 >> v2:172.21.15.183:3300/0 conn(0x7f598c161820 msgr2=0x7f598c0d0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:56.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.116+0000 7f5992251640 1 --2- 172.21.15.83:0/48435526 >> v2:172.21.15.183:3300/0 conn(0x7f598c161820 0x7f598c0d0610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.116+0000 7f5992251640 1 -- 172.21.15.83:0/48435526 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f598c0cf800 con 0x7f598c161440 2024-03-06T07:21:56.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.117+0000 7f5992251640 1 --2- 172.21.15.83:0/48435526 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f598c161440 0x7f598c0b4be0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f598c08dc80 tx=0x7f598802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faebcaf431079bb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.117+0000 7f599124f640 1 -- 172.21.15.83:0/48435526 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5988038070 con 0x7f598c161440 2024-03-06T07:21:56.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.117+0000 7f599124f640 1 -- 172.21.15.83:0/48435526 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f598802f050 con 0x7f598c161440 2024-03-06T07:21:56.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.117+0000 7f599124f640 1 -- 172.21.15.83:0/48435526 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5988031970 con 0x7f598c161440 2024-03-06T07:21:56.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.117+0000 7f59944b5640 1 -- 172.21.15.83:0/48435526 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f598c161440 msgr2=0x7f598c0b4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.117+0000 7f59944b5640 1 --2- 172.21.15.83:0/48435526 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f598c161440 0x7f598c0b4be0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f598c08dc80 tx=0x7f598802b7f0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.117+0000 7f59944b5640 1 -- 172.21.15.83:0/48435526 shutdown_connections 2024-03-06T07:21:56.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.117+0000 7f59944b5640 1 --2- 172.21.15.83:0/48435526 >> v2:172.21.15.183:3300/0 conn(0x7f598c161820 0x7f598c0d0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.117+0000 7f59944b5640 1 --2- 172.21.15.83:0/48435526 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f598c0d1300 0x7f598c0d0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.117+0000 7f59944b5640 1 --2- 172.21.15.83:0/48435526 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f598c161440 0x7f598c0b4be0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.117+0000 7f59944b5640 1 -- 172.21.15.83:0/48435526 >> 172.21.15.83:0/48435526 conn(0x7f598c0d3cb0 msgr2=0x7f598c0d40c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:56.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.118+0000 7f59944b5640 1 -- 172.21.15.83:0/48435526 shutdown_connections 2024-03-06T07:21:56.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.118+0000 7f59944b5640 1 -- 172.21.15.83:0/48435526 wait complete. 2024-03-06T07:21:56.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.119+0000 7f59944b5640 1 Processor -- start 2024-03-06T07:21:56.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.119+0000 7f59944b5640 1 -- start start 2024-03-06T07:21:56.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.119+0000 7f59944b5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f598c0d1300 0x7f598c14a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.119+0000 7f59944b5640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f598c161440 0x7f598c0cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.119+0000 7f59944b5640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f598c161820 0x7f598c157090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.119+0000 7f59944b5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598c0d1880 con 0x7f598c0d1300 2024-03-06T07:21:56.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.119+0000 7f59944b5640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598c0b5120 con 0x7f598c161820 2024-03-06T07:21:56.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.119+0000 7f59944b5640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f598c0d1a00 con 0x7f598c161440 2024-03-06T07:21:56.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.119+0000 7f5992251640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f598c0d1300 0x7f598c14a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.119+0000 7f5992251640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f598c0d1300 0x7f598c14a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51548/0 (socket says 172.21.15.83:51548) 2024-03-06T07:21:56.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.119+0000 7f5992251640 1 -- 172.21.15.83:0/3640594618 learned_addr learned my addr 172.21.15.83:0/3640594618 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:21:56.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.120+0000 7f5992a52640 1 --2- 172.21.15.83:0/3640594618 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f598c161820 0x7f598c157090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.120+0000 7f5991a50640 1 --2- 172.21.15.83:0/3640594618 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f598c161440 0x7f598c0cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.120+0000 7f5992251640 1 -- 172.21.15.83:0/3640594618 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f598c161440 msgr2=0x7f598c0cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.120+0000 7f5992251640 1 --2- 172.21.15.83:0/3640594618 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f598c161440 0x7f598c0cc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.120+0000 7f5992251640 1 -- 172.21.15.83:0/3640594618 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f598c161820 msgr2=0x7f598c157090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.120+0000 7f5992251640 1 --2- 172.21.15.83:0/3640594618 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f598c161820 0x7f598c157090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.120+0000 7f5992251640 1 -- 172.21.15.83:0/3640594618 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f598c14a5f0 con 0x7f598c0d1300 2024-03-06T07:21:56.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.120+0000 7f5992251640 1 --2- 172.21.15.83:0/3640594618 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f598c0d1300 0x7f598c14a0b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5988000c00 tx=0x7f59880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.120+0000 7f5982ffd640 1 -- 172.21.15.83:0/3640594618 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5988042020 con 0x7f598c0d1300 2024-03-06T07:21:56.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.120+0000 7f5982ffd640 1 -- 172.21.15.83:0/3640594618 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f598803d870 con 0x7f598c0d1300 2024-03-06T07:21:56.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.120+0000 7f59944b5640 1 -- 172.21.15.83:0/3640594618 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f598c0cd6f0 con 0x7f598c0d1300 2024-03-06T07:21:56.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.120+0000 7f5982ffd640 1 -- 172.21.15.83:0/3640594618 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5988038040 con 0x7f598c0d1300 2024-03-06T07:21:56.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.120+0000 7f59944b5640 1 -- 172.21.15.83:0/3640594618 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f598c0cd410 con 0x7f598c0d1300 2024-03-06T07:21:56.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.121+0000 7f59944b5640 1 -- 172.21.15.83:0/3640594618 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f598c04f330 con 0x7f598c0d1300 2024-03-06T07:21:56.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.121+0000 7f5982ffd640 1 -- 172.21.15.83:0/3640594618 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5988045020 con 0x7f598c0d1300 2024-03-06T07:21:56.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.122+0000 7f5982ffd640 1 --2- 172.21.15.83:0/3640594618 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f597006b500 0x7f597006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.122+0000 7f5982ffd640 1 -- 172.21.15.83:0/3640594618 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f59880abc00 con 0x7f598c0d1300 2024-03-06T07:21:56.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.122+0000 7f5991a50640 1 --2- 172.21.15.83:0/3640594618 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f597006b500 0x7f597006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.122+0000 7f5991a50640 1 --2- 172.21.15.83:0/3640594618 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f597006b500 0x7f597006d9c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f597c006fd0 tx=0x7f597c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.126+0000 7f5982ffd640 1 -- 172.21.15.83:0/3640594618 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f59880b2080 con 0x7f598c0d1300 2024-03-06T07:21:56.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.300+0000 7f59944b5640 1 -- 172.21.15.83:0/3640594618 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f598c0b9250 con 0x7f598c0d1300 2024-03-06T07:21:56.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.926+0000 7f5982ffd640 1 -- 172.21.15.83:0/3640594618 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7f598804c030 con 0x7f598c0d1300 2024-03-06T07:21:56.928 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'cephfs_metadata' created 2024-03-06T07:21:56.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.929+0000 7f59944b5640 1 -- 172.21.15.83:0/3640594618 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f597006b500 msgr2=0x7f597006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.929+0000 7f59944b5640 1 --2- 172.21.15.83:0/3640594618 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f597006b500 0x7f597006d9c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f597c006fd0 tx=0x7f597c008040 comp rx=0 tx=0).stop 2024-03-06T07:21:56.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.929+0000 7f59944b5640 1 -- 172.21.15.83:0/3640594618 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f598c0d1300 msgr2=0x7f598c14a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.929+0000 7f59944b5640 1 --2- 172.21.15.83:0/3640594618 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f598c0d1300 0x7f598c14a0b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5988000c00 tx=0x7f59880046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.929+0000 7f59944b5640 1 -- 172.21.15.83:0/3640594618 shutdown_connections 2024-03-06T07:21:56.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.929+0000 7f59944b5640 1 --2- 172.21.15.83:0/3640594618 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f597006b500 0x7f597006d9c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.929+0000 7f59944b5640 1 --2- 172.21.15.83:0/3640594618 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f598c161820 0x7f598c157090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.929+0000 7f59944b5640 1 --2- 172.21.15.83:0/3640594618 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f598c161440 0x7f598c0cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.929+0000 7f59944b5640 1 --2- 172.21.15.83:0/3640594618 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f598c0d1300 0x7f598c14a0b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.929+0000 7f59944b5640 1 -- 172.21.15.83:0/3640594618 >> 172.21.15.83:0/3640594618 conn(0x7f598c0d3cb0 msgr2=0x7f598c0b7470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:56.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.930+0000 7f59944b5640 1 -- 172.21.15.83:0/3640594618 shutdown_connections 2024-03-06T07:21:56.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:56.930+0000 7f59944b5640 1 -- 172.21.15.83:0/3640594618 wait complete. 2024-03-06T07:21:56.946 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:21:57.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.053+0000 7f1ec63d7640 1 Processor -- start 2024-03-06T07:21:57.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.053+0000 7f1ec63d7640 1 -- start start 2024-03-06T07:21:57.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.053+0000 7f1ec63d7640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f1ec01614d0 0x7f1ec00d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.053+0000 7f1ec63d7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1ec00d1910 0x7f1ec00d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.053+0000 7f1ec63d7640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ec01618b0 con 0x7f1ec00d22e0 2024-03-06T07:21:57.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.053+0000 7f1ec63d7640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ec00cf400 con 0x7f1ec00d1910 2024-03-06T07:21:57.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.053+0000 7f1ec63d7640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ec00d2710 con 0x7f1ec01614d0 2024-03-06T07:21:57.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.054+0000 7f1ebf7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1ec00d1910 0x7f1ec00d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.054+0000 7f1ebf7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1ec00d1910 0x7f1ec00d1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33804/0 (socket says 172.21.15.83:33804) 2024-03-06T07:21:57.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.054+0000 7f1ebf7fe640 1 -- 172.21.15.83:0/3002275728 learned_addr learned my addr 172.21.15.83:0/3002275728 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:21:57.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.054+0000 7f1ebffff640 1 --2- 172.21.15.83:0/3002275728 >> v2:172.21.15.183:3300/0 conn(0x7f1ec01614d0 0x7f1ec00d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.054+0000 7f1ebf7fe640 1 -- 172.21.15.83:0/3002275728 >> v2:172.21.15.183:3300/0 conn(0x7f1ec01614d0 msgr2=0x7f1ec00d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:57.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.054+0000 7f1ebf7fe640 1 --2- 172.21.15.83:0/3002275728 >> v2:172.21.15.183:3300/0 conn(0x7f1ec01614d0 0x7f1ec00d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.054+0000 7f1ebf7fe640 1 -- 172.21.15.83:0/3002275728 >> v1:172.21.15.183:6789/0 conn(0x7f1ec00d22e0 legacy=0x7f1ec00d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:57.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.054+0000 7f1ebf7fe640 1 -- 172.21.15.83:0/3002275728 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec00cf6a0 con 0x7f1ec00d1910 2024-03-06T07:21:57.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.055+0000 7f1ebf7fe640 1 --2- 172.21.15.83:0/3002275728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ec00d1910 0x7f1ec00d1da0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1ec008c470 tx=0x7f1eb402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3433fe67b26d552 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:57.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.055+0000 7f1ebeffd640 1 -- 172.21.15.83:0/3002275728 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb4034070 con 0x7f1ec00d1910 2024-03-06T07:21:57.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.055+0000 7f1ebeffd640 1 -- 172.21.15.83:0/3002275728 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eb4030860 con 0x7f1ec00d1910 2024-03-06T07:21:57.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.055+0000 7f1ebeffd640 1 -- 172.21.15.83:0/3002275728 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb4030c30 con 0x7f1ec00d1910 2024-03-06T07:21:57.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.055+0000 7f1ec63d7640 1 -- 172.21.15.83:0/3002275728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ec00d1910 msgr2=0x7f1ec00d1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.055+0000 7f1ec63d7640 1 --2- 172.21.15.83:0/3002275728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ec00d1910 0x7f1ec00d1da0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1ec008c470 tx=0x7f1eb402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.055+0000 7f1ec63d7640 1 -- 172.21.15.83:0/3002275728 shutdown_connections 2024-03-06T07:21:57.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.055+0000 7f1ec63d7640 1 --2- 172.21.15.83:0/3002275728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ec00d1910 0x7f1ec00d1da0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.055+0000 7f1ec63d7640 1 --2- 172.21.15.83:0/3002275728 >> v2:172.21.15.183:3300/0 conn(0x7f1ec01614d0 0x7f1ec00d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.055+0000 7f1ec63d7640 1 -- 172.21.15.83:0/3002275728 >> 172.21.15.83:0/3002275728 conn(0x7f1ec00d3d40 msgr2=0x7f1ec00d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:57.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.055+0000 7f1ec63d7640 1 -- 172.21.15.83:0/3002275728 shutdown_connections 2024-03-06T07:21:57.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.055+0000 7f1ec63d7640 1 -- 172.21.15.83:0/3002275728 wait complete. 2024-03-06T07:21:57.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.056+0000 7f1ec63d7640 1 Processor -- start 2024-03-06T07:21:57.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.057+0000 7f1ec63d7640 1 -- start start 2024-03-06T07:21:57.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.057+0000 7f1ec63d7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ec00d1910 0x7f1ec00cb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.057+0000 7f1ec63d7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1ec00d22e0 0x7f1ec014f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.057+0000 7f1ec63d7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1ec01614d0 0x7f1ec00b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.057+0000 7f1ec63d7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec00b0910 con 0x7f1ec00d1910 2024-03-06T07:21:57.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.057+0000 7f1ec63d7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec00cfc60 con 0x7f1ec00d22e0 2024-03-06T07:21:57.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.057+0000 7f1ec63d7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec00b1e40 con 0x7f1ec01614d0 2024-03-06T07:21:57.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.057+0000 7f1ebffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ec00d1910 0x7f1ec00cb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.057+0000 7f1ebffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ec00d1910 0x7f1ec00cb1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33818/0 (socket says 172.21.15.83:33818) 2024-03-06T07:21:57.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.057+0000 7f1ebffff640 1 -- 172.21.15.83:0/490693169 learned_addr learned my addr 172.21.15.83:0/490693169 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:21:57.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.057+0000 7f1ebf7fe640 1 --2- 172.21.15.83:0/490693169 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1ec00d22e0 0x7f1ec014f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.057+0000 7f1ec4974640 1 --2- 172.21.15.83:0/490693169 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1ec01614d0 0x7f1ec00b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.058+0000 7f1ebffff640 1 -- 172.21.15.83:0/490693169 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1ec01614d0 msgr2=0x7f1ec00b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.058+0000 7f1ebffff640 1 --2- 172.21.15.83:0/490693169 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1ec01614d0 0x7f1ec00b4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.058+0000 7f1ebffff640 1 -- 172.21.15.83:0/490693169 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1ec00d22e0 msgr2=0x7f1ec014f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.058+0000 7f1ebffff640 1 --2- 172.21.15.83:0/490693169 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1ec00d22e0 0x7f1ec014f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.058+0000 7f1ebffff640 1 -- 172.21.15.83:0/490693169 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec00b63e0 con 0x7f1ec00d1910 2024-03-06T07:21:57.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.058+0000 7f1ec4974640 1 --2- 172.21.15.83:0/490693169 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1ec01614d0 0x7f1ec00b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:57.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.058+0000 7f1ebffff640 1 --2- 172.21.15.83:0/490693169 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ec00d1910 0x7f1ec00cb1d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1eb000d620 tx=0x7f1eb000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:57.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.058+0000 7f1ebcff9640 1 -- 172.21.15.83:0/490693169 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb000b620 con 0x7f1ec00d1910 2024-03-06T07:21:57.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.058+0000 7f1ebcff9640 1 -- 172.21.15.83:0/490693169 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eb0019070 con 0x7f1ec00d1910 2024-03-06T07:21:57.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.058+0000 7f1ec63d7640 1 -- 172.21.15.83:0/490693169 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ec00b5340 con 0x7f1ec00d1910 2024-03-06T07:21:57.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.058+0000 7f1ebcff9640 1 -- 172.21.15.83:0/490693169 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb000ba00 con 0x7f1ec00d1910 2024-03-06T07:21:57.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.058+0000 7f1ec63d7640 1 -- 172.21.15.83:0/490693169 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ec00b5900 con 0x7f1ec00d1910 2024-03-06T07:21:57.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.059+0000 7f1ec63d7640 1 -- 172.21.15.83:0/490693169 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ec0068ff0 con 0x7f1ec00d1910 2024-03-06T07:21:57.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.059+0000 7f1ebcff9640 1 -- 172.21.15.83:0/490693169 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1eb000bba0 con 0x7f1ec00d1910 2024-03-06T07:21:57.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.060+0000 7f1ebcff9640 1 --2- 172.21.15.83:0/490693169 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1eac06b500 0x7f1eac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.060+0000 7f1ebf7fe640 1 --2- 172.21.15.83:0/490693169 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1eac06b500 0x7f1eac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.060+0000 7f1ebcff9640 1 -- 172.21.15.83:0/490693169 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f1eb008a400 con 0x7f1ec00d1910 2024-03-06T07:21:57.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.060+0000 7f1ebf7fe640 1 --2- 172.21.15.83:0/490693169 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1eac06b500 0x7f1eac06d9c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1eb4036000 tx=0x7f1eb403b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:57.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.064+0000 7f1ebcff9640 1 -- 172.21.15.83:0/490693169 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1eb0059460 con 0x7f1ec00d1910 2024-03-06T07:21:57.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.236+0000 7f1ec63d7640 1 -- 172.21.15.83:0/490693169 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f1ec00caab0 con 0x7f1ec00d1910 2024-03-06T07:21:57.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.929+0000 7f1ebcff9640 1 -- 172.21.15.83:0/490693169 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v18) v1 ==== 170+0+0 (secure 0 0 0) 0x7f1eb0058d60 con 0x7f1ec00d1910 2024-03-06T07:21:57.931 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'cephfs_data' created 2024-03-06T07:21:57.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.932+0000 7f1ec63d7640 1 -- 172.21.15.83:0/490693169 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1eac06b500 msgr2=0x7f1eac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.932+0000 7f1ec63d7640 1 --2- 172.21.15.83:0/490693169 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1eac06b500 0x7f1eac06d9c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1eb4036000 tx=0x7f1eb403b040 comp rx=0 tx=0).stop 2024-03-06T07:21:57.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.932+0000 7f1ec63d7640 1 -- 172.21.15.83:0/490693169 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ec00d1910 msgr2=0x7f1ec00cb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.932+0000 7f1ec63d7640 1 --2- 172.21.15.83:0/490693169 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ec00d1910 0x7f1ec00cb1d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1eb000d620 tx=0x7f1eb000dae0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.932+0000 7f1ec63d7640 1 -- 172.21.15.83:0/490693169 shutdown_connections 2024-03-06T07:21:57.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.932+0000 7f1ec63d7640 1 --2- 172.21.15.83:0/490693169 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1eac06b500 0x7f1eac06d9c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.932+0000 7f1ec63d7640 1 --2- 172.21.15.83:0/490693169 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1ec01614d0 0x7f1ec00b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.932+0000 7f1ec63d7640 1 --2- 172.21.15.83:0/490693169 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1ec00d22e0 0x7f1ec014f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.932+0000 7f1ec63d7640 1 --2- 172.21.15.83:0/490693169 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ec00d1910 0x7f1ec00cb1d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.932+0000 7f1ec63d7640 1 -- 172.21.15.83:0/490693169 >> 172.21.15.83:0/490693169 conn(0x7f1ec00d3d40 msgr2=0x7f1ec00cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:57.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.932+0000 7f1ec63d7640 1 -- 172.21.15.83:0/490693169 shutdown_connections 2024-03-06T07:21:57.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:57.932+0000 7f1ec63d7640 1 -- 172.21.15.83:0/490693169 wait complete. 2024-03-06T07:21:57.956 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:21:58.055 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:21:58.058 DEBUG:teuthology.orchestra.run.smithi183:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:21:58.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.059+0000 7fca6bb7a640 1 Processor -- start 2024-03-06T07:21:58.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.059+0000 7fca6bb7a640 1 -- start start 2024-03-06T07:21:58.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.060+0000 7fca6bb7a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fca64161980 0x7fca640cf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.060+0000 7fca6bb7a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fca640c72a0 0x7fca6406afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.060+0000 7fca6bb7a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca640ceb30 con 0x7fca6406b4f0 2024-03-06T07:21:58.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.060+0000 7fca6bb7a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca640c6ff0 con 0x7fca640c72a0 2024-03-06T07:21:58.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.060+0000 7fca6bb7a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca640ce570 con 0x7fca64161980 2024-03-06T07:21:58.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.060+0000 7fca6a117640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fca6406b4f0 0x7fca6406e380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:40906/0 (socket says 172.21.15.83:40906) 2024-03-06T07:21:58.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.060+0000 7fca6a117640 1 -- 172.21.15.83:0/2868603873 learned_addr learned my addr 172.21.15.83:0/2868603873 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:21:58.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.060+0000 7fca69916640 1 --2- 172.21.15.83:0/2868603873 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fca64161980 0x7fca640cf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.060+0000 7fca69115640 1 --2- 172.21.15.83:0/2868603873 >> v2:172.21.15.183:3300/0 conn(0x7fca640c72a0 0x7fca6406afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.060+0000 7fca68914640 1 -- 172.21.15.83:0/2868603873 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 459446145 0 0) 0x7fca5c001490 con 0x7fca6406b4f0 2024-03-06T07:21:58.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.060+0000 7fca68914640 1 -- 172.21.15.83:0/2868603873 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca50003520 con 0x7fca6406b4f0 2024-03-06T07:21:58.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.060+0000 7fca69916640 1 -- 172.21.15.83:0/2868603873 >> v2:172.21.15.183:3300/0 conn(0x7fca640c72a0 msgr2=0x7fca6406afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:58.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.060+0000 7fca69916640 1 --2- 172.21.15.83:0/2868603873 >> v2:172.21.15.183:3300/0 conn(0x7fca640c72a0 0x7fca6406afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.060+0000 7fca69916640 1 -- 172.21.15.83:0/2868603873 >> v1:172.21.15.83:6789/0 conn(0x7fca6406b4f0 legacy=0x7fca6406e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.060+0000 7fca69916640 1 -- 172.21.15.83:0/2868603873 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca640ce850 con 0x7fca64161980 2024-03-06T07:21:58.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.060+0000 7fca69115640 1 --2- 172.21.15.83:0/2868603873 >> v2:172.21.15.183:3300/0 conn(0x7fca640c72a0 0x7fca6406afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:58.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.061+0000 7fca69916640 1 --2- 172.21.15.83:0/2868603873 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fca64161980 0x7fca640cf1f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fca600097a0 tx=0x7fca6002b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74d1fcc6b90ea699 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:58.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.061+0000 7fca68914640 1 -- 172.21.15.83:0/2868603873 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca60038070 con 0x7fca64161980 2024-03-06T07:21:58.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.061+0000 7fca68914640 1 -- 172.21.15.83:0/2868603873 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca60031700 con 0x7fca64161980 2024-03-06T07:21:58.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.061+0000 7fca68914640 1 -- 172.21.15.83:0/2868603873 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca60031ad0 con 0x7fca64161980 2024-03-06T07:21:58.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.062+0000 7fca6bb7a640 1 -- 172.21.15.83:0/2868603873 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fca64161980 msgr2=0x7fca640cf1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.062+0000 7fca6bb7a640 1 --2- 172.21.15.83:0/2868603873 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fca64161980 0x7fca640cf1f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fca600097a0 tx=0x7fca6002b7a0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.062+0000 7fca6bb7a640 1 -- 172.21.15.83:0/2868603873 shutdown_connections 2024-03-06T07:21:58.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.062+0000 7fca6bb7a640 1 --2- 172.21.15.83:0/2868603873 >> v2:172.21.15.183:3300/0 conn(0x7fca640c72a0 0x7fca6406afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.062+0000 7fca6bb7a640 1 --2- 172.21.15.83:0/2868603873 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fca64161980 0x7fca640cf1f0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.062+0000 7fca6bb7a640 1 -- 172.21.15.83:0/2868603873 >> 172.21.15.83:0/2868603873 conn(0x7fca640cd420 msgr2=0x7fca640cd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:58.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.062+0000 7fca6bb7a640 1 -- 172.21.15.83:0/2868603873 shutdown_connections 2024-03-06T07:21:58.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.062+0000 7fca6bb7a640 1 -- 172.21.15.83:0/2868603873 wait complete. 2024-03-06T07:21:58.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.063+0000 7fca6bb7a640 1 Processor -- start 2024-03-06T07:21:58.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.063+0000 7fca6bb7a640 1 -- start start 2024-03-06T07:21:58.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.064+0000 7fca6bb7a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fca6406b4f0 0x7fca640c8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.064+0000 7fca6bb7a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fca640c72a0 0x7fca640b4150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.064+0000 7fca6bb7a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fca64161980 0x7fca640ca9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.064+0000 7fca6bb7a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca640c7700 con 0x7fca6406b4f0 2024-03-06T07:21:58.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.064+0000 7fca6bb7a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca640c6c80 con 0x7fca64161980 2024-03-06T07:21:58.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.064+0000 7fca6bb7a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca640ce090 con 0x7fca640c72a0 2024-03-06T07:21:58.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.064+0000 7fca69916640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fca6406b4f0 0x7fca640c8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.064+0000 7fca69916640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fca6406b4f0 0x7fca640c8640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33826/0 (socket says 172.21.15.83:33826) 2024-03-06T07:21:58.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.064+0000 7fca69916640 1 -- 172.21.15.83:0/899469065 learned_addr learned my addr 172.21.15.83:0/899469065 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:21:58.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.064+0000 7fca69115640 1 --2- 172.21.15.83:0/899469065 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fca640c72a0 0x7fca640b4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.064+0000 7fca6a117640 1 --2- 172.21.15.83:0/899469065 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fca64161980 0x7fca640ca9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.064+0000 7fca69916640 1 -- 172.21.15.83:0/899469065 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fca640c72a0 msgr2=0x7fca640b4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.064+0000 7fca69916640 1 --2- 172.21.15.83:0/899469065 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fca640c72a0 0x7fca640b4150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.064+0000 7fca69916640 1 -- 172.21.15.83:0/899469065 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fca64161980 msgr2=0x7fca640ca9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.064+0000 7fca69916640 1 --2- 172.21.15.83:0/899469065 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fca64161980 0x7fca640ca9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.064+0000 7fca69916640 1 -- 172.21.15.83:0/899469065 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca640c8d00 con 0x7fca6406b4f0 2024-03-06T07:21:58.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.065+0000 7fca69916640 1 --2- 172.21.15.83:0/899469065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fca6406b4f0 0x7fca640c8640 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fca60036040 tx=0x7fca600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:58.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.065+0000 7fca5a7fc640 1 -- 172.21.15.83:0/899469065 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca60042070 con 0x7fca6406b4f0 2024-03-06T07:21:58.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.065+0000 7fca5a7fc640 1 -- 172.21.15.83:0/899469065 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca6003d890 con 0x7fca6406b4f0 2024-03-06T07:21:58.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.065+0000 7fca6bb7a640 1 -- 172.21.15.83:0/899469065 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca640c8fe0 con 0x7fca6406b4f0 2024-03-06T07:21:58.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.065+0000 7fca5a7fc640 1 -- 172.21.15.83:0/899469065 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca60038040 con 0x7fca6406b4f0 2024-03-06T07:21:58.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.065+0000 7fca6bb7a640 1 -- 172.21.15.83:0/899469065 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca640b3950 con 0x7fca6406b4f0 2024-03-06T07:21:58.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.066+0000 7fca6bb7a640 1 -- 172.21.15.83:0/899469065 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca6404f330 con 0x7fca6406b4f0 2024-03-06T07:21:58.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.066+0000 7fca5a7fc640 1 -- 172.21.15.83:0/899469065 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fca6003b070 con 0x7fca6406b4f0 2024-03-06T07:21:58.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.067+0000 7fca5a7fc640 1 --2- 172.21.15.83:0/899469065 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fca4806b500 0x7fca4806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.067+0000 7fca5a7fc640 1 -- 172.21.15.83:0/899469065 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fca600ab7b0 con 0x7fca6406b4f0 2024-03-06T07:21:58.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.067+0000 7fca69115640 1 --2- 172.21.15.83:0/899469065 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fca4806b500 0x7fca4806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.067+0000 7fca69115640 1 --2- 172.21.15.83:0/899469065 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fca4806b500 0x7fca4806d9c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fca640b47e0 tx=0x7fca54009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:58.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.071+0000 7fca5a7fc640 1 -- 172.21.15.83:0/899469065 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fca60030070 con 0x7fca6406b4f0 2024-03-06T07:21:58.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.285+0000 7fca6bb7a640 1 -- 172.21.15.83:0/899469065 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fca64074290 con 0x7fca6406b4f0 2024-03-06T07:21:58.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.948+0000 7fca5a7fc640 1 -- 172.21.15.83:0/899469065 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7fca60079fb0 con 0x7fca6406b4f0 2024-03-06T07:21:58.951 INFO:teuthology.orchestra.run.smithi083.stderr:new fs with metadata pool 2 and data pool 3 2024-03-06T07:21:58.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.955+0000 7fca6bb7a640 1 -- 172.21.15.83:0/899469065 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fca4806b500 msgr2=0x7fca4806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.955+0000 7fca6bb7a640 1 --2- 172.21.15.83:0/899469065 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fca4806b500 0x7fca4806d9c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fca640b47e0 tx=0x7fca54009190 comp rx=0 tx=0).stop 2024-03-06T07:21:58.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.955+0000 7fca6bb7a640 1 -- 172.21.15.83:0/899469065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fca6406b4f0 msgr2=0x7fca640c8640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.955+0000 7fca6bb7a640 1 --2- 172.21.15.83:0/899469065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fca6406b4f0 0x7fca640c8640 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fca60036040 tx=0x7fca600046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.957+0000 7fca6bb7a640 1 -- 172.21.15.83:0/899469065 shutdown_connections 2024-03-06T07:21:58.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.958+0000 7fca6bb7a640 1 --2- 172.21.15.83:0/899469065 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fca4806b500 0x7fca4806d9c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.958+0000 7fca6bb7a640 1 --2- 172.21.15.83:0/899469065 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fca64161980 0x7fca640ca9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.958+0000 7fca6bb7a640 1 --2- 172.21.15.83:0/899469065 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fca640c72a0 0x7fca640b4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.958+0000 7fca6bb7a640 1 --2- 172.21.15.83:0/899469065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fca6406b4f0 0x7fca640c8640 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.958+0000 7fca6bb7a640 1 -- 172.21.15.83:0/899469065 >> 172.21.15.83:0/899469065 conn(0x7fca640cd420 msgr2=0x7fca64071850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:58.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.961+0000 7fca6bb7a640 1 -- 172.21.15.83:0/899469065 shutdown_connections 2024-03-06T07:21:58.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:58.963+0000 7fca6bb7a640 1 -- 172.21.15.83:0/899469065 wait complete. 2024-03-06T07:21:58.980 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:59.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.079+0000 7f0b1a35e640 1 Processor -- start 2024-03-06T07:21:59.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.079+0000 7f0b1a35e640 1 -- start start 2024-03-06T07:21:59.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.080+0000 7f0b1a35e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b141424f0 0x7f0b140cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.080+0000 7f0b1a35e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0b1406da40 0x7f0b1415cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.080+0000 7f0b1a35e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b140cc980 con 0x7f0b140cd7a0 2024-03-06T07:21:59.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.080+0000 7f0b1a35e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b140cd220 con 0x7f0b1406da40 2024-03-06T07:21:59.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.080+0000 7f0b1a35e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b140ccc60 con 0x7f0b141424f0 2024-03-06T07:21:59.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.080+0000 7f0b188fb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0b1406da40 0x7f0b1415cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.080+0000 7f0b188fb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0b1406da40 0x7f0b1415cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33838/0 (socket says 172.21.15.83:33838) 2024-03-06T07:21:59.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.080+0000 7f0b137fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f0b140cd7a0 0x7f0b1406d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:40918/0 (socket says 172.21.15.83:40918) 2024-03-06T07:21:59.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.080+0000 7f0b188fb640 1 -- 172.21.15.83:0/853208739 learned_addr learned my addr 172.21.15.83:0/853208739 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:21:59.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.080+0000 7f0b188fb640 1 -- 172.21.15.83:0/853208739 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b141424f0 msgr2=0x7f0b140cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:59.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.080+0000 7f0b188fb640 1 --2- 172.21.15.83:0/853208739 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b141424f0 0x7f0b140cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.080+0000 7f0b188fb640 1 -- 172.21.15.83:0/853208739 >> v1:172.21.15.83:6789/0 conn(0x7f0b140cd7a0 legacy=0x7f0b1406d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.080+0000 7f0b188fb640 1 -- 172.21.15.83:0/853208739 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b140ccf40 con 0x7f0b1406da40 2024-03-06T07:21:59.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.080+0000 7f0b12ffd640 1 -- 172.21.15.83:0/853208739 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 275869020 0 0) 0x7f0b140cc980 con 0x7f0b140cd7a0 2024-03-06T07:21:59.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.081+0000 7f0b188fb640 1 --2- 172.21.15.83:0/853208739 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b1406da40 0x7f0b1415cef0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c009e50 tx=0x7f0b0c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=237030822741333a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.081+0000 7f0b12ffd640 1 -- 172.21.15.83:0/853208739 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c02cb00 con 0x7f0b1406da40 2024-03-06T07:21:59.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.081+0000 7f0b12ffd640 1 -- 172.21.15.83:0/853208739 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b0c02cca0 con 0x7f0b1406da40 2024-03-06T07:21:59.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.081+0000 7f0b12ffd640 1 -- 172.21.15.83:0/853208739 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c034610 con 0x7f0b1406da40 2024-03-06T07:21:59.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.081+0000 7f0b1a35e640 1 -- 172.21.15.83:0/853208739 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b1406da40 msgr2=0x7f0b1415cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.081+0000 7f0b1a35e640 1 --2- 172.21.15.83:0/853208739 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b1406da40 0x7f0b1415cef0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c009e50 tx=0x7f0b0c02bf60 comp rx=0 tx=0).stop 2024-03-06T07:21:59.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.082+0000 7f0b1a35e640 1 -- 172.21.15.83:0/853208739 shutdown_connections 2024-03-06T07:21:59.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.082+0000 7f0b1a35e640 1 --2- 172.21.15.83:0/853208739 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b1406da40 0x7f0b1415cef0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.082+0000 7f0b1a35e640 1 --2- 172.21.15.83:0/853208739 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b141424f0 0x7f0b140cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.082+0000 7f0b1a35e640 1 -- 172.21.15.83:0/853208739 >> 172.21.15.83:0/853208739 conn(0x7f0b140cf8a0 msgr2=0x7f0b140cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:59.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.082+0000 7f0b1a35e640 1 -- 172.21.15.83:0/853208739 shutdown_connections 2024-03-06T07:21:59.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.082+0000 7f0b1a35e640 1 -- 172.21.15.83:0/853208739 wait complete. 2024-03-06T07:21:59.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.083+0000 7f0b1a35e640 1 Processor -- start 2024-03-06T07:21:59.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.083+0000 7f0b1a35e640 1 -- start start 2024-03-06T07:21:59.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.084+0000 7f0b1a35e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b1406da40 0x7f0b1415af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.084+0000 7f0b1a35e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b140cd7a0 0x7f0b1406b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.084+0000 7f0b1a35e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b141424f0 0x7f0b140b0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.084+0000 7f0b1a35e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b14071ce0 con 0x7f0b1406da40 2024-03-06T07:21:59.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.084+0000 7f0b1a35e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b14071b60 con 0x7f0b140cd7a0 2024-03-06T07:21:59.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.084+0000 7f0b1a35e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b14071e60 con 0x7f0b141424f0 2024-03-06T07:21:59.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.084+0000 7f0b13fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b1406da40 0x7f0b1415af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.084+0000 7f0b13fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b1406da40 0x7f0b1415af90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33848/0 (socket says 172.21.15.83:33848) 2024-03-06T07:21:59.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.084+0000 7f0b13fff640 1 -- 172.21.15.83:0/1284369338 learned_addr learned my addr 172.21.15.83:0/1284369338 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:21:59.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.084+0000 7f0b137fe640 1 --2- 172.21.15.83:0/1284369338 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b140cd7a0 0x7f0b1406b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.084+0000 7f0b188fb640 1 --2- 172.21.15.83:0/1284369338 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b141424f0 0x7f0b140b0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.084+0000 7f0b13fff640 1 -- 172.21.15.83:0/1284369338 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b141424f0 msgr2=0x7f0b140b0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.084+0000 7f0b13fff640 1 --2- 172.21.15.83:0/1284369338 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b141424f0 0x7f0b140b0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.084+0000 7f0b13fff640 1 -- 172.21.15.83:0/1284369338 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b140cd7a0 msgr2=0x7f0b1406b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.084+0000 7f0b13fff640 1 --2- 172.21.15.83:0/1284369338 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b140cd7a0 0x7f0b1406b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.084+0000 7f0b13fff640 1 -- 172.21.15.83:0/1284369338 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b1415b4d0 con 0x7f0b1406da40 2024-03-06T07:21:59.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.084+0000 7f0b137fe640 1 --2- 172.21.15.83:0/1284369338 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b140cd7a0 0x7f0b1406b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:59.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.084+0000 7f0b188fb640 1 --2- 172.21.15.83:0/1284369338 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b141424f0 0x7f0b140b0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:59.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.084+0000 7f0b13fff640 1 --2- 172.21.15.83:0/1284369338 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b1406da40 0x7f0b1415af90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0b0400b440 tx=0x7f0b0400b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.084+0000 7f0b10ff9640 1 -- 172.21.15.83:0/1284369338 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b04015020 con 0x7f0b1406da40 2024-03-06T07:21:59.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.084+0000 7f0b10ff9640 1 -- 172.21.15.83:0/1284369338 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b04011890 con 0x7f0b1406da40 2024-03-06T07:21:59.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.084+0000 7f0b10ff9640 1 -- 172.21.15.83:0/1284369338 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b04011c80 con 0x7f0b1406da40 2024-03-06T07:21:59.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.084+0000 7f0b1a35e640 1 -- 172.21.15.83:0/1284369338 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b14143e60 con 0x7f0b1406da40 2024-03-06T07:21:59.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.084+0000 7f0b1a35e640 1 -- 172.21.15.83:0/1284369338 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b14144420 con 0x7f0b1406da40 2024-03-06T07:21:59.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.085+0000 7f0b1a35e640 1 -- 172.21.15.83:0/1284369338 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b1404f330 con 0x7f0b1406da40 2024-03-06T07:21:59.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.086+0000 7f0b10ff9640 1 -- 172.21.15.83:0/1284369338 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0b04018020 con 0x7f0b1406da40 2024-03-06T07:21:59.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.086+0000 7f0b10ff9640 1 --2- 172.21.15.83:0/1284369338 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0b0006b4d0 0x7f0b0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.086+0000 7f0b10ff9640 1 -- 172.21.15.83:0/1284369338 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0b0408a380 con 0x7f0b1406da40 2024-03-06T07:21:59.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.086+0000 7f0b137fe640 1 --2- 172.21.15.83:0/1284369338 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0b0006b4d0 0x7f0b0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.086+0000 7f0b137fe640 1 --2- 172.21.15.83:0/1284369338 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0b0006b4d0 0x7f0b0006d990 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0b1406c870 tx=0x7f0b08009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.091+0000 7f0b10ff9640 1 -- 172.21.15.83:0/1284369338 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0b0401f020 con 0x7f0b1406da40 2024-03-06T07:21:59.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.264+0000 7f0b1a35e640 1 -- 172.21.15.83:0/1284369338 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b1415d8e0 con 0x7f0b1406da40 2024-03-06T07:21:59.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.265+0000 7f0b10ff9640 1 -- 172.21.15.83:0/1284369338 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16630 (secure 0 0 0) 0x7f0b0400d040 con 0x7f0b1406da40 2024-03-06T07:21:59.267 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:21:59.267 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":19,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:21:58.286605+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.318942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.254795+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:59.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.267+0000 7f0b1a35e640 1 -- 172.21.15.83:0/1284369338 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0b0006b4d0 msgr2=0x7f0b0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.267+0000 7f0b1a35e640 1 --2- 172.21.15.83:0/1284369338 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0b0006b4d0 0x7f0b0006d990 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0b1406c870 tx=0x7f0b08009040 comp rx=0 tx=0).stop 2024-03-06T07:21:59.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.267+0000 7f0b1a35e640 1 -- 172.21.15.83:0/1284369338 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b1406da40 msgr2=0x7f0b1415af90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.267+0000 7f0b1a35e640 1 --2- 172.21.15.83:0/1284369338 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b1406da40 0x7f0b1415af90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0b0400b440 tx=0x7f0b0400b900 comp rx=0 tx=0).stop 2024-03-06T07:21:59.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.267+0000 7f0b1a35e640 1 -- 172.21.15.83:0/1284369338 shutdown_connections 2024-03-06T07:21:59.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.267+0000 7f0b1a35e640 1 --2- 172.21.15.83:0/1284369338 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0b0006b4d0 0x7f0b0006d990 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.267+0000 7f0b1a35e640 1 --2- 172.21.15.83:0/1284369338 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b141424f0 0x7f0b140b0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.267+0000 7f0b1a35e640 1 --2- 172.21.15.83:0/1284369338 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b140cd7a0 0x7f0b1406b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.267+0000 7f0b1a35e640 1 --2- 172.21.15.83:0/1284369338 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b1406da40 0x7f0b1415af90 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.267+0000 7f0b1a35e640 1 -- 172.21.15.83:0/1284369338 >> 172.21.15.83:0/1284369338 conn(0x7f0b140cf8a0 msgr2=0x7f0b140cdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:59.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.267+0000 7f0b1a35e640 1 -- 172.21.15.83:0/1284369338 shutdown_connections 2024-03-06T07:21:59.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.267+0000 7f0b1a35e640 1 -- 172.21.15.83:0/1284369338 wait complete. 2024-03-06T07:21:59.279 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:59.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.376+0000 7ff586b7a640 1 Processor -- start 2024-03-06T07:21:59.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.376+0000 7ff586b7a640 1 -- start start 2024-03-06T07:21:59.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.377+0000 7ff586b7a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ff58013dc10 0x7ff5800b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.377+0000 7ff586b7a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff58015bf30 0x7ff5800b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.377+0000 7ff586b7a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff58015e850 0x7ff58015eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.377+0000 7ff586b7a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5800cf3f0 con 0x7ff58015e850 2024-03-06T07:21:59.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.377+0000 7ff586b7a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff58015b210 con 0x7ff58013dc10 2024-03-06T07:21:59.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.377+0000 7ff586b7a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff58015f580 con 0x7ff58015bf30 2024-03-06T07:21:59.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.377+0000 7ff585117640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff58015e850 0x7ff58015eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.377+0000 7ff585117640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff58015e850 0x7ff58015eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33854/0 (socket says 172.21.15.83:33854) 2024-03-06T07:21:59.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.377+0000 7ff585117640 1 -- 172.21.15.83:0/1804623623 learned_addr learned my addr 172.21.15.83:0/1804623623 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:21:59.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.378+0000 7ff584916640 1 --2- 172.21.15.83:0/1804623623 >> v2:172.21.15.183:3300/0 conn(0x7ff58013dc10 0x7ff5800b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.378+0000 7ff585117640 1 -- 172.21.15.83:0/1804623623 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff58015bf30 msgr2=0x7ff5800b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:59.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.378+0000 7ff585117640 1 --2- 172.21.15.83:0/1804623623 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff58015bf30 0x7ff5800b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.378+0000 7ff585117640 1 -- 172.21.15.83:0/1804623623 >> v2:172.21.15.183:3300/0 conn(0x7ff58013dc10 msgr2=0x7ff5800b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:59.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.378+0000 7ff585117640 1 --2- 172.21.15.83:0/1804623623 >> v2:172.21.15.183:3300/0 conn(0x7ff58013dc10 0x7ff5800b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.378+0000 7ff585117640 1 -- 172.21.15.83:0/1804623623 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff58015b450 con 0x7ff58015e850 2024-03-06T07:21:59.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.378+0000 7ff584916640 1 --2- 172.21.15.83:0/1804623623 >> v2:172.21.15.183:3300/0 conn(0x7ff58013dc10 0x7ff5800b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:59.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.378+0000 7ff585117640 1 --2- 172.21.15.83:0/1804623623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff58015e850 0x7ff58015eca0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff5800731a0 tx=0x7ff57802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b204ae0d5a6dfc32 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.379+0000 7ff57f7fe640 1 -- 172.21.15.83:0/1804623623 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57802c9e0 con 0x7ff58015e850 2024-03-06T07:21:59.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.379+0000 7ff57f7fe640 1 -- 172.21.15.83:0/1804623623 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff57802cb80 con 0x7ff58015e850 2024-03-06T07:21:59.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.379+0000 7ff57f7fe640 1 -- 172.21.15.83:0/1804623623 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5780343f0 con 0x7ff58015e850 2024-03-06T07:21:59.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.379+0000 7ff586b7a640 1 -- 172.21.15.83:0/1804623623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff58015e850 msgr2=0x7ff58015eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.379+0000 7ff586b7a640 1 --2- 172.21.15.83:0/1804623623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff58015e850 0x7ff58015eca0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff5800731a0 tx=0x7ff57802be40 comp rx=0 tx=0).stop 2024-03-06T07:21:59.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.379+0000 7ff586b7a640 1 -- 172.21.15.83:0/1804623623 shutdown_connections 2024-03-06T07:21:59.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.379+0000 7ff586b7a640 1 --2- 172.21.15.83:0/1804623623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff58015e850 0x7ff58015eca0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.379+0000 7ff586b7a640 1 --2- 172.21.15.83:0/1804623623 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff58015bf30 0x7ff5800b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.379+0000 7ff586b7a640 1 --2- 172.21.15.83:0/1804623623 >> v2:172.21.15.183:3300/0 conn(0x7ff58013dc10 0x7ff5800b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.379+0000 7ff586b7a640 1 -- 172.21.15.83:0/1804623623 >> 172.21.15.83:0/1804623623 conn(0x7ff5800cf8a0 msgr2=0x7ff5800cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:59.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.379+0000 7ff586b7a640 1 -- 172.21.15.83:0/1804623623 shutdown_connections 2024-03-06T07:21:59.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.379+0000 7ff586b7a640 1 -- 172.21.15.83:0/1804623623 wait complete. 2024-03-06T07:21:59.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.380+0000 7ff586b7a640 1 Processor -- start 2024-03-06T07:21:59.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.380+0000 7ff586b7a640 1 -- start start 2024-03-06T07:21:59.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.381+0000 7ff586b7a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff58013dc10 0x7ff58013a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.381+0000 7ff586b7a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff58015bf30 0x7ff580155920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.381+0000 7ff586b7a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff58015e850 0x7ff580159ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.381+0000 7ff586b7a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff58015f1e0 con 0x7ff58013dc10 2024-03-06T07:21:59.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.381+0000 7ff586b7a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff58013dff0 con 0x7ff58015bf30 2024-03-06T07:21:59.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.381+0000 7ff586b7a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff58015f360 con 0x7ff58015e850 2024-03-06T07:21:59.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.381+0000 7ff584916640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff58013dc10 0x7ff58013a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.381+0000 7ff584916640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff58013dc10 0x7ff58013a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33866/0 (socket says 172.21.15.83:33866) 2024-03-06T07:21:59.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.381+0000 7ff584916640 1 -- 172.21.15.83:0/1163141477 learned_addr learned my addr 172.21.15.83:0/1163141477 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:21:59.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.381+0000 7ff57ffff640 1 --2- 172.21.15.83:0/1163141477 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff58015bf30 0x7ff580155920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.381+0000 7ff584916640 1 -- 172.21.15.83:0/1163141477 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff58015e850 msgr2=0x7ff580159ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.381+0000 7ff584916640 1 --2- 172.21.15.83:0/1163141477 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff58015e850 0x7ff580159ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.381+0000 7ff584916640 1 -- 172.21.15.83:0/1163141477 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff58015bf30 msgr2=0x7ff580155920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.381+0000 7ff584916640 1 --2- 172.21.15.83:0/1163141477 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff58015bf30 0x7ff580155920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.382+0000 7ff584916640 1 -- 172.21.15.83:0/1163141477 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff580156290 con 0x7ff58013dc10 2024-03-06T07:21:59.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.382+0000 7ff584916640 1 --2- 172.21.15.83:0/1163141477 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff58013dc10 0x7ff58013a8a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff57400eb70 tx=0x7ff574007410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.382+0000 7ff57d7fa640 1 -- 172.21.15.83:0/1163141477 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff574007d20 con 0x7ff58013dc10 2024-03-06T07:21:59.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.382+0000 7ff57d7fa640 1 -- 172.21.15.83:0/1163141477 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff574010440 con 0x7ff58013dc10 2024-03-06T07:21:59.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.382+0000 7ff586b7a640 1 -- 172.21.15.83:0/1163141477 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff580156570 con 0x7ff58013dc10 2024-03-06T07:21:59.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.382+0000 7ff57d7fa640 1 -- 172.21.15.83:0/1163141477 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5740107f0 con 0x7ff58013dc10 2024-03-06T07:21:59.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.382+0000 7ff586b7a640 1 -- 172.21.15.83:0/1163141477 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff580156b30 con 0x7ff58013dc10 2024-03-06T07:21:59.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.383+0000 7ff586b7a640 1 -- 172.21.15.83:0/1163141477 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff58004f330 con 0x7ff58013dc10 2024-03-06T07:21:59.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.383+0000 7ff57d7fa640 1 -- 172.21.15.83:0/1163141477 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff57400b030 con 0x7ff58013dc10 2024-03-06T07:21:59.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.384+0000 7ff57d7fa640 1 --2- 172.21.15.83:0/1163141477 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff56406b500 0x7ff56406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.384+0000 7ff57d7fa640 1 -- 172.21.15.83:0/1163141477 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff57408c440 con 0x7ff58013dc10 2024-03-06T07:21:59.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.384+0000 7ff57ffff640 1 --2- 172.21.15.83:0/1163141477 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff56406b500 0x7ff56406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.384+0000 7ff57ffff640 1 --2- 172.21.15.83:0/1163141477 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff56406b500 0x7ff56406d9c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff5800ccd50 tx=0x7ff56c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.388+0000 7ff57d7fa640 1 -- 172.21.15.83:0/1163141477 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff57405b350 con 0x7ff58013dc10 2024-03-06T07:21:59.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.560+0000 7ff586b7a640 1 -- 172.21.15.83:0/1163141477 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff58006eb00 con 0x7ff58013dc10 2024-03-06T07:21:59.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.561+0000 7ff57d7fa640 1 -- 172.21.15.83:0/1163141477 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16630 (secure 0 0 0) 0x7ff57405ac50 con 0x7ff58013dc10 2024-03-06T07:21:59.563 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:21:59.563 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":19,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:21:58.286605+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.318942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.254795+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:59.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.563+0000 7ff586b7a640 1 -- 172.21.15.83:0/1163141477 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff56406b500 msgr2=0x7ff56406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.563+0000 7ff586b7a640 1 --2- 172.21.15.83:0/1163141477 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff56406b500 0x7ff56406d9c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff5800ccd50 tx=0x7ff56c008040 comp rx=0 tx=0).stop 2024-03-06T07:21:59.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.563+0000 7ff586b7a640 1 -- 172.21.15.83:0/1163141477 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff58013dc10 msgr2=0x7ff58013a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.563+0000 7ff586b7a640 1 --2- 172.21.15.83:0/1163141477 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff58013dc10 0x7ff58013a8a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff57400eb70 tx=0x7ff574007410 comp rx=0 tx=0).stop 2024-03-06T07:21:59.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.563+0000 7ff586b7a640 1 -- 172.21.15.83:0/1163141477 shutdown_connections 2024-03-06T07:21:59.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.563+0000 7ff586b7a640 1 --2- 172.21.15.83:0/1163141477 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff56406b500 0x7ff56406d9c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.563+0000 7ff586b7a640 1 --2- 172.21.15.83:0/1163141477 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff58015e850 0x7ff580159ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.563+0000 7ff586b7a640 1 --2- 172.21.15.83:0/1163141477 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff58015bf30 0x7ff580155920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.563+0000 7ff586b7a640 1 --2- 172.21.15.83:0/1163141477 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff58013dc10 0x7ff58013a8a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.563+0000 7ff586b7a640 1 -- 172.21.15.83:0/1163141477 >> 172.21.15.83:0/1163141477 conn(0x7ff5800cf8a0 msgr2=0x7ff580071dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:59.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.563+0000 7ff586b7a640 1 -- 172.21.15.83:0/1163141477 shutdown_connections 2024-03-06T07:21:59.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.563+0000 7ff586b7a640 1 -- 172.21.15.83:0/1163141477 wait complete. 2024-03-06T07:21:59.576 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:21:59.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.677+0000 7f6b26df6640 1 Processor -- start 2024-03-06T07:21:59.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.677+0000 7f6b26df6640 1 -- start start 2024-03-06T07:21:59.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.678+0000 7f6b26df6640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6b20071a70 0x7f6b20071e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.678+0000 7f6b26df6640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f6b200b42e0 0x7f6b2015c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.678+0000 7f6b26df6640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b20072390 con 0x7f6b2015ad90 2024-03-06T07:21:59.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.678+0000 7f6b26df6640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b2015c020 con 0x7f6b200b42e0 2024-03-06T07:21:59.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.678+0000 7f6b26df6640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b200b46c0 con 0x7f6b20071a70 2024-03-06T07:21:59.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.679+0000 7f6b25393640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f6b200b42e0 0x7f6b2015c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.679+0000 7f6b24b92640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6b20071a70 0x7f6b20071e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.679+0000 7f6b1ffff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f6b2015ad90 0x7f6b200b3bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:44222/0 (socket says 172.21.15.83:44222) 2024-03-06T07:21:59.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.679+0000 7f6b1ffff640 1 -- 172.21.15.83:0/1635792300 learned_addr learned my addr 172.21.15.83:0/1635792300 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:21:59.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.679+0000 7f6b25393640 1 -- 172.21.15.83:0/1635792300 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6b20071a70 msgr2=0x7f6b20071e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:59.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.679+0000 7f6b25393640 1 --2- 172.21.15.83:0/1635792300 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6b20071a70 0x7f6b20071e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.679+0000 7f6b25393640 1 -- 172.21.15.83:0/1635792300 >> v1:172.21.15.183:6789/0 conn(0x7f6b2015ad90 legacy=0x7f6b200b3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.679+0000 7f6b25393640 1 -- 172.21.15.83:0/1635792300 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b2015b260 con 0x7f6b200b42e0 2024-03-06T07:21:59.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.679+0000 7f6b24b92640 1 --2- 172.21.15.83:0/1635792300 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6b20071a70 0x7f6b20071e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:59.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.680+0000 7f6b25393640 1 --2- 172.21.15.83:0/1635792300 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6b200b42e0 0x7f6b2015c210 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6b18009d90 tx=0x7f6b1802be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f28f2c476111bea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.680+0000 7f6b1f7fe640 1 -- 172.21.15.83:0/1635792300 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b18039070 con 0x7f6b200b42e0 2024-03-06T07:21:59.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.680+0000 7f6b1f7fe640 1 -- 172.21.15.83:0/1635792300 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b1802cbc0 con 0x7f6b200b42e0 2024-03-06T07:21:59.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.680+0000 7f6b1f7fe640 1 -- 172.21.15.83:0/1635792300 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b18034440 con 0x7f6b200b42e0 2024-03-06T07:21:59.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.680+0000 7f6b26df6640 1 -- 172.21.15.83:0/1635792300 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6b200b42e0 msgr2=0x7f6b2015c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.680+0000 7f6b26df6640 1 --2- 172.21.15.83:0/1635792300 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6b200b42e0 0x7f6b2015c210 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6b18009d90 tx=0x7f6b1802be10 comp rx=0 tx=0).stop 2024-03-06T07:21:59.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.680+0000 7f6b26df6640 1 -- 172.21.15.83:0/1635792300 shutdown_connections 2024-03-06T07:21:59.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.680+0000 7f6b26df6640 1 --2- 172.21.15.83:0/1635792300 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6b200b42e0 0x7f6b2015c210 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.680+0000 7f6b26df6640 1 --2- 172.21.15.83:0/1635792300 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6b20071a70 0x7f6b20071e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.680+0000 7f6b26df6640 1 -- 172.21.15.83:0/1635792300 >> 172.21.15.83:0/1635792300 conn(0x7f6b200b2660 msgr2=0x7f6b200b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:59.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.680+0000 7f6b26df6640 1 -- 172.21.15.83:0/1635792300 shutdown_connections 2024-03-06T07:21:59.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.680+0000 7f6b26df6640 1 -- 172.21.15.83:0/1635792300 wait complete. 2024-03-06T07:21:59.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.681+0000 7f6b26df6640 1 Processor -- start 2024-03-06T07:21:59.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.682+0000 7f6b26df6640 1 -- start start 2024-03-06T07:21:59.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.682+0000 7f6b26df6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b20071a70 0x7f6b2013ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.682+0000 7f6b26df6640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6b200b42e0 0x7f6b201516e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.682+0000 7f6b26df6640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6b2015ad90 0x7f6b201e32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.682+0000 7f6b26df6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2015bb00 con 0x7f6b20071a70 2024-03-06T07:21:59.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.682+0000 7f6b26df6640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2015b820 con 0x7f6b200b42e0 2024-03-06T07:21:59.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.682+0000 7f6b26df6640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2015c750 con 0x7f6b2015ad90 2024-03-06T07:21:59.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.682+0000 7f6b24b92640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b20071a70 0x7f6b2013ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.683+0000 7f6b24b92640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b20071a70 0x7f6b2013ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33870/0 (socket says 172.21.15.83:33870) 2024-03-06T07:21:59.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.683+0000 7f6b24b92640 1 -- 172.21.15.83:0/1280796186 learned_addr learned my addr 172.21.15.83:0/1280796186 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:21:59.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.683+0000 7f6b1ffff640 1 --2- 172.21.15.83:0/1280796186 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6b200b42e0 0x7f6b201516e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.683+0000 7f6b25393640 1 --2- 172.21.15.83:0/1280796186 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6b2015ad90 0x7f6b201e32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.683+0000 7f6b24b92640 1 -- 172.21.15.83:0/1280796186 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6b2015ad90 msgr2=0x7f6b201e32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.683+0000 7f6b24b92640 1 --2- 172.21.15.83:0/1280796186 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6b2015ad90 0x7f6b201e32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.683+0000 7f6b24b92640 1 -- 172.21.15.83:0/1280796186 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6b200b42e0 msgr2=0x7f6b201516e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.683+0000 7f6b24b92640 1 --2- 172.21.15.83:0/1280796186 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6b200b42e0 0x7f6b201516e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.683+0000 7f6b24b92640 1 -- 172.21.15.83:0/1280796186 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b2013f280 con 0x7f6b20071a70 2024-03-06T07:21:59.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.683+0000 7f6b24b92640 1 --2- 172.21.15.83:0/1280796186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b20071a70 0x7f6b2013ed40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6b1000ebe0 tx=0x7f6b1000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.683+0000 7f6b1d7fa640 1 -- 172.21.15.83:0/1280796186 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b10017070 con 0x7f6b20071a70 2024-03-06T07:21:59.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.683+0000 7f6b1d7fa640 1 -- 172.21.15.83:0/1280796186 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b1000cc30 con 0x7f6b20071a70 2024-03-06T07:21:59.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.683+0000 7f6b26df6640 1 -- 172.21.15.83:0/1280796186 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b2013f7b0 con 0x7f6b20071a70 2024-03-06T07:21:59.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.683+0000 7f6b1d7fa640 1 -- 172.21.15.83:0/1280796186 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b10013470 con 0x7f6b20071a70 2024-03-06T07:21:59.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.683+0000 7f6b26df6640 1 -- 172.21.15.83:0/1280796186 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b2013fd70 con 0x7f6b20071a70 2024-03-06T07:21:59.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.684+0000 7f6b26df6640 1 -- 172.21.15.83:0/1280796186 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b20159b50 con 0x7f6b20071a70 2024-03-06T07:21:59.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.684+0000 7f6b1d7fa640 1 -- 172.21.15.83:0/1280796186 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6b1001a070 con 0x7f6b20071a70 2024-03-06T07:21:59.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.685+0000 7f6b1d7fa640 1 --2- 172.21.15.83:0/1280796186 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6b0c06b500 0x7f6b0c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.685+0000 7f6b1d7fa640 1 -- 172.21.15.83:0/1280796186 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6b1008a120 con 0x7f6b20071a70 2024-03-06T07:21:59.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.685+0000 7f6b1ffff640 1 --2- 172.21.15.83:0/1280796186 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6b0c06b500 0x7f6b0c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.685+0000 7f6b1ffff640 1 --2- 172.21.15.83:0/1280796186 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6b0c06b500 0x7f6b0c06d9c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6b20159d50 tx=0x7f6b14009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.689+0000 7f6b1d7fa640 1 -- 172.21.15.83:0/1280796186 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6b20159b50 con 0x7f6b20071a70 2024-03-06T07:21:59.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.897+0000 7f6b26df6640 1 -- 172.21.15.83:0/1280796186 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f6b2006fcd0 con 0x7f6b20071a70 2024-03-06T07:21:59.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.955+0000 7f6b1d7fa640 1 -- 172.21.15.83:0/1280796186 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v5) v1 ==== 118+0+0 (secure 0 0 0) 0x7f6b2013f280 con 0x7f6b20071a70 2024-03-06T07:21:59.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.958+0000 7f6b26df6640 1 -- 172.21.15.83:0/1280796186 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6b0c06b500 msgr2=0x7f6b0c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.958+0000 7f6b26df6640 1 --2- 172.21.15.83:0/1280796186 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6b0c06b500 0x7f6b0c06d9c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6b20159d50 tx=0x7f6b14009040 comp rx=0 tx=0).stop 2024-03-06T07:21:59.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.958+0000 7f6b26df6640 1 -- 172.21.15.83:0/1280796186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b20071a70 msgr2=0x7f6b2013ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.958+0000 7f6b26df6640 1 --2- 172.21.15.83:0/1280796186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b20071a70 0x7f6b2013ed40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6b1000ebe0 tx=0x7f6b1000c520 comp rx=0 tx=0).stop 2024-03-06T07:21:59.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.958+0000 7f6b26df6640 1 -- 172.21.15.83:0/1280796186 shutdown_connections 2024-03-06T07:21:59.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.958+0000 7f6b26df6640 1 --2- 172.21.15.83:0/1280796186 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6b0c06b500 0x7f6b0c06d9c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.958+0000 7f6b26df6640 1 --2- 172.21.15.83:0/1280796186 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6b2015ad90 0x7f6b201e32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.958+0000 7f6b26df6640 1 --2- 172.21.15.83:0/1280796186 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6b200b42e0 0x7f6b201516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.958+0000 7f6b26df6640 1 --2- 172.21.15.83:0/1280796186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b20071a70 0x7f6b2013ed40 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.958+0000 7f6b26df6640 1 -- 172.21.15.83:0/1280796186 >> 172.21.15.83:0/1280796186 conn(0x7f6b200b2660 msgr2=0x7f6b2006da00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:59.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.958+0000 7f6b26df6640 1 -- 172.21.15.83:0/1280796186 shutdown_connections 2024-03-06T07:21:59.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:21:59.958+0000 7f6b26df6640 1 -- 172.21.15.83:0/1280796186 wait complete. 2024-03-06T07:21:59.971 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:21:59.971 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:22:00.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.073+0000 7f369fe9d640 1 Processor -- start 2024-03-06T07:22:00.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.073+0000 7f369fe9d640 1 -- start start 2024-03-06T07:22:00.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.074+0000 7f369fe9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f369815c050 0x7f369806f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.074+0000 7f369fe9d640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f369806fb40 0x7f36980cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.074+0000 7f369fe9d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36980cd4b0 con 0x7f36980ccca0 2024-03-06T07:22:00.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.074+0000 7f369fe9d640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f369806ffb0 con 0x7f369815c050 2024-03-06T07:22:00.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.074+0000 7f369fe9d640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36980cd790 con 0x7f369806fb40 2024-03-06T07:22:00.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.074+0000 7f369dc39640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f369815c050 0x7f369806f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.074+0000 7f369dc39640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f369815c050 0x7f369806f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33878/0 (socket says 172.21.15.83:33878) 2024-03-06T07:22:00.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.074+0000 7f369e43a640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f36980ccca0 0x7f36980cfae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:40926/0 (socket says 172.21.15.83:40926) 2024-03-06T07:22:00.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.074+0000 7f369dc39640 1 -- 172.21.15.83:0/3181348667 learned_addr learned my addr 172.21.15.83:0/3181348667 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:00.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.074+0000 7f369d438640 1 --2- 172.21.15.83:0/3181348667 >> v2:172.21.15.183:3300/0 conn(0x7f369806fb40 0x7f36980cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.074+0000 7f369dc39640 1 -- 172.21.15.83:0/3181348667 >> v2:172.21.15.183:3300/0 conn(0x7f369806fb40 msgr2=0x7f36980cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:00.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.074+0000 7f369dc39640 1 --2- 172.21.15.83:0/3181348667 >> v2:172.21.15.183:3300/0 conn(0x7f369806fb40 0x7f36980cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.074+0000 7f369dc39640 1 -- 172.21.15.83:0/3181348667 >> v1:172.21.15.83:6789/0 conn(0x7f36980ccca0 legacy=0x7f36980cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.074+0000 7f369dc39640 1 -- 172.21.15.83:0/3181348667 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36980cda70 con 0x7f369815c050 2024-03-06T07:22:00.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.074+0000 7f369d438640 1 --2- 172.21.15.83:0/3181348667 >> v2:172.21.15.183:3300/0 conn(0x7f369806fb40 0x7f36980cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:00.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.075+0000 7f369dc39640 1 --2- 172.21.15.83:0/3181348667 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f369815c050 0x7f369806f600 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f36980b7dc0 tx=0x7f368802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41c950690858436e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.075+0000 7f369cc37640 1 -- 172.21.15.83:0/3181348667 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3688035020 con 0x7f369815c050 2024-03-06T07:22:00.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.075+0000 7f369cc37640 1 -- 172.21.15.83:0/3181348667 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3688031870 con 0x7f369815c050 2024-03-06T07:22:00.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.075+0000 7f369cc37640 1 -- 172.21.15.83:0/3181348667 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3688031c40 con 0x7f369815c050 2024-03-06T07:22:00.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.075+0000 7f369fe9d640 1 -- 172.21.15.83:0/3181348667 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f369815c050 msgr2=0x7f369806f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.075+0000 7f369fe9d640 1 --2- 172.21.15.83:0/3181348667 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f369815c050 0x7f369806f600 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f36980b7dc0 tx=0x7f368802b910 comp rx=0 tx=0).stop 2024-03-06T07:22:00.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.076+0000 7f369fe9d640 1 -- 172.21.15.83:0/3181348667 shutdown_connections 2024-03-06T07:22:00.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.076+0000 7f369fe9d640 1 --2- 172.21.15.83:0/3181348667 >> v2:172.21.15.183:3300/0 conn(0x7f369806fb40 0x7f36980cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.076+0000 7f369fe9d640 1 --2- 172.21.15.83:0/3181348667 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f369815c050 0x7f369806f600 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.076+0000 7f369fe9d640 1 -- 172.21.15.83:0/3181348667 >> 172.21.15.83:0/3181348667 conn(0x7f36980b3800 msgr2=0x7f36980b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.076+0000 7f369fe9d640 1 -- 172.21.15.83:0/3181348667 shutdown_connections 2024-03-06T07:22:00.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.076+0000 7f369fe9d640 1 -- 172.21.15.83:0/3181348667 wait complete. 2024-03-06T07:22:00.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.077+0000 7f369fe9d640 1 Processor -- start 2024-03-06T07:22:00.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.077+0000 7f369fe9d640 1 -- start start 2024-03-06T07:22:00.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.077+0000 7f369fe9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f369806fb40 0x7f369814a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.077+0000 7f369fe9d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f36980ccca0 0x7f36980b48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.077+0000 7f369fe9d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f369815c050 0x7f3698155d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.077+0000 7f369fe9d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36980d01f0 con 0x7f369806fb40 2024-03-06T07:22:00.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.077+0000 7f369fe9d640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36980cd130 con 0x7f36980ccca0 2024-03-06T07:22:00.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.077+0000 7f369fe9d640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36980d0370 con 0x7f369815c050 2024-03-06T07:22:00.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.077+0000 7f369dc39640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f369806fb40 0x7f369814a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.077+0000 7f369dc39640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f369806fb40 0x7f369814a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33890/0 (socket says 172.21.15.83:33890) 2024-03-06T07:22:00.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.078+0000 7f369dc39640 1 -- 172.21.15.83:0/449242066 learned_addr learned my addr 172.21.15.83:0/449242066 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:00.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.078+0000 7f369d438640 1 --2- 172.21.15.83:0/449242066 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f36980ccca0 0x7f36980b48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.078+0000 7f369e43a640 1 --2- 172.21.15.83:0/449242066 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f369815c050 0x7f3698155d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.078+0000 7f369dc39640 1 -- 172.21.15.83:0/449242066 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f369815c050 msgr2=0x7f3698155d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.078+0000 7f369dc39640 1 --2- 172.21.15.83:0/449242066 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f369815c050 0x7f3698155d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.078+0000 7f369dc39640 1 -- 172.21.15.83:0/449242066 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f36980ccca0 msgr2=0x7f36980b48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.078+0000 7f369dc39640 1 --2- 172.21.15.83:0/449242066 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f36980ccca0 0x7f36980b48f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.078+0000 7f369dc39640 1 -- 172.21.15.83:0/449242066 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36980b5240 con 0x7f369806fb40 2024-03-06T07:22:00.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.078+0000 7f369d438640 1 --2- 172.21.15.83:0/449242066 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f36980ccca0 0x7f36980b48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:00.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.078+0000 7f369e43a640 1 --2- 172.21.15.83:0/449242066 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f369815c050 0x7f3698155d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:00.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.078+0000 7f369dc39640 1 --2- 172.21.15.83:0/449242066 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f369806fb40 0x7f369814a580 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f368802b950 tx=0x7f36880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.078+0000 7f368e7fc640 1 -- 172.21.15.83:0/449242066 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3688040070 con 0x7f369806fb40 2024-03-06T07:22:00.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.078+0000 7f368e7fc640 1 -- 172.21.15.83:0/449242066 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f368803a420 con 0x7f369806fb40 2024-03-06T07:22:00.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.078+0000 7f369fe9d640 1 -- 172.21.15.83:0/449242066 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36980b5520 con 0x7f369806fb40 2024-03-06T07:22:00.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.078+0000 7f368e7fc640 1 -- 172.21.15.83:0/449242066 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3688035040 con 0x7f369806fb40 2024-03-06T07:22:00.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.078+0000 7f369fe9d640 1 -- 172.21.15.83:0/449242066 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36980b5ae0 con 0x7f369806fb40 2024-03-06T07:22:00.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.079+0000 7f369fe9d640 1 -- 172.21.15.83:0/449242066 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3698068ff0 con 0x7f369806fb40 2024-03-06T07:22:00.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.080+0000 7f368e7fc640 1 -- 172.21.15.83:0/449242066 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3688038020 con 0x7f369806fb40 2024-03-06T07:22:00.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.080+0000 7f368e7fc640 1 --2- 172.21.15.83:0/449242066 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f367c06b4d0 0x7f367c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.080+0000 7f368e7fc640 1 -- 172.21.15.83:0/449242066 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f36880ac290 con 0x7f369806fb40 2024-03-06T07:22:00.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.080+0000 7f369d438640 1 --2- 172.21.15.83:0/449242066 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f367c06b4d0 0x7f367c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.080+0000 7f369d438640 1 --2- 172.21.15.83:0/449242066 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f367c06b4d0 0x7f367c06d990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f36981f5040 tx=0x7f3694008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.088+0000 7f368e7fc640 1 -- 172.21.15.83:0/449242066 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f368807b080 con 0x7f369806fb40 2024-03-06T07:22:00.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.305+0000 7f369fe9d640 1 -- 172.21.15.83:0/449242066 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f36980ce9b0 con 0x7f369806fb40 2024-03-06T07:22:00.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.962+0000 7f368e7fc640 1 -- 172.21.15.83:0/449242066 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v6) v1 ==== 122+0+0 (secure 0 0 0) 0x7f368807a980 con 0x7f369806fb40 2024-03-06T07:22:00.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.964+0000 7f369fe9d640 1 -- 172.21.15.83:0/449242066 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f367c06b4d0 msgr2=0x7f367c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.964+0000 7f369fe9d640 1 --2- 172.21.15.83:0/449242066 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f367c06b4d0 0x7f367c06d990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f36981f5040 tx=0x7f3694008040 comp rx=0 tx=0).stop 2024-03-06T07:22:00.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.964+0000 7f369fe9d640 1 -- 172.21.15.83:0/449242066 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f369806fb40 msgr2=0x7f369814a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.964+0000 7f369fe9d640 1 --2- 172.21.15.83:0/449242066 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f369806fb40 0x7f369814a580 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f368802b950 tx=0x7f36880046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.964+0000 7f369fe9d640 1 -- 172.21.15.83:0/449242066 shutdown_connections 2024-03-06T07:22:00.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.964+0000 7f369fe9d640 1 --2- 172.21.15.83:0/449242066 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f367c06b4d0 0x7f367c06d990 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.964+0000 7f369fe9d640 1 --2- 172.21.15.83:0/449242066 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f369815c050 0x7f3698155d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.965+0000 7f369fe9d640 1 --2- 172.21.15.83:0/449242066 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f36980ccca0 0x7f36980b48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.965+0000 7f369fe9d640 1 --2- 172.21.15.83:0/449242066 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f369806fb40 0x7f369814a580 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.965+0000 7f369fe9d640 1 -- 172.21.15.83:0/449242066 >> 172.21.15.83:0/449242066 conn(0x7f36980b3800 msgr2=0x7f36980b3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.965+0000 7f369fe9d640 1 -- 172.21.15.83:0/449242066 shutdown_connections 2024-03-06T07:22:00.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:00.965+0000 7f369fe9d640 1 -- 172.21.15.83:0/449242066 wait complete. 2024-03-06T07:22:00.979 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:22:00.979 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:01.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.077+0000 7fcba6b9a640 1 Processor -- start 2024-03-06T07:22:01.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.077+0000 7fcba6b9a640 1 -- start start 2024-03-06T07:22:01.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.078+0000 7fcba6b9a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fcba015bdd0 0x7fcba00c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.078+0000 7fcba6b9a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcba015fa10 0x7fcba00ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.078+0000 7fcba6b9a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcba00cebe0 con 0x7fcba00d16e0 2024-03-06T07:22:01.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.078+0000 7fcba6b9a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcba00ceec0 con 0x7fcba015bdd0 2024-03-06T07:22:01.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.078+0000 7fcba6b9a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcba00cf1a0 con 0x7fcba015fa10 2024-03-06T07:22:01.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.078+0000 7fcba5137640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fcba00d16e0 0x7fcba00d1b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:40932/0 (socket says 172.21.15.83:40932) 2024-03-06T07:22:01.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.078+0000 7fcba5137640 1 -- 172.21.15.83:0/4263131379 learned_addr learned my addr 172.21.15.83:0/4263131379 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:01.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.078+0000 7fcba4936640 1 --2- 172.21.15.83:0/4263131379 >> v2:172.21.15.183:3300/0 conn(0x7fcba015bdd0 0x7fcba00c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.078+0000 7fcb9ffff640 1 --2- 172.21.15.83:0/4263131379 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcba015fa10 0x7fcba00ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.078+0000 7fcb9f7fe640 1 -- 172.21.15.83:0/4263131379 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3643847351 0 0) 0x7fcba00cebe0 con 0x7fcba00d16e0 2024-03-06T07:22:01.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.078+0000 7fcb9f7fe640 1 -- 172.21.15.83:0/4263131379 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb90003520 con 0x7fcba00d16e0 2024-03-06T07:22:01.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.078+0000 7fcb9f7fe640 1 -- 172.21.15.83:0/4263131379 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4201916366 0 0) 0x7fcb90003520 con 0x7fcba00d16e0 2024-03-06T07:22:01.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.078+0000 7fcb9f7fe640 1 -- 172.21.15.83:0/4263131379 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcba015fa10 msgr2=0x7fcba00ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:01.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.078+0000 7fcb9f7fe640 1 --2- 172.21.15.83:0/4263131379 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcba015fa10 0x7fcba00ce270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.078+0000 7fcb9f7fe640 1 -- 172.21.15.83:0/4263131379 >> v2:172.21.15.183:3300/0 conn(0x7fcba015bdd0 msgr2=0x7fcba00c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:01.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.078+0000 7fcb9f7fe640 1 --2- 172.21.15.83:0/4263131379 >> v2:172.21.15.183:3300/0 conn(0x7fcba015bdd0 0x7fcba00c5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.078+0000 7fcb9f7fe640 1 -- 172.21.15.83:0/4263131379 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcba00cc8e0 con 0x7fcba00d16e0 2024-03-06T07:22:01.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.078+0000 7fcb9f7fe640 1 -- 172.21.15.83:0/4263131379 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7fcb980019f0 con 0x7fcba00d16e0 2024-03-06T07:22:01.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.079+0000 7fcb9f7fe640 1 -- 172.21.15.83:0/4263131379 >> v1:172.21.15.83:6789/0 conn(0x7fcba00d16e0 legacy=0x7fcba00d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.079+0000 7fcb9f7fe640 1 --2- 172.21.15.83:0/4263131379 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcba015fa10 0x7fcb900039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.079+0000 7fcb9f7fe640 1 --2- 172.21.15.83:0/4263131379 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcb90003f30 0x7fcb90004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.079+0000 7fcb9f7fe640 1 --2- 172.21.15.83:0/4263131379 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcb90004d40 0x7fcb900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.079+0000 7fcb9f7fe640 1 -- 172.21.15.83:0/4263131379 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba00ceec0 con 0x7fcb90004d40 2024-03-06T07:22:01.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.079+0000 7fcb9f7fe640 1 -- 172.21.15.83:0/4263131379 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba00cf1a0 con 0x7fcb90003f30 2024-03-06T07:22:01.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.079+0000 7fcb9f7fe640 1 -- 172.21.15.83:0/4263131379 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9000ba10 con 0x7fcba015fa10 2024-03-06T07:22:01.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.079+0000 7fcba5137640 1 --2- 172.21.15.83:0/4263131379 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcb90004d40 0x7fcb900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.079+0000 7fcba4936640 1 --2- 172.21.15.83:0/4263131379 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcba015fa10 0x7fcb900039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.079+0000 7fcb9ffff640 1 --2- 172.21.15.83:0/4263131379 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcb90003f30 0x7fcb90004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.079+0000 7fcba5137640 1 -- 172.21.15.83:0/4263131379 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcba015fa10 msgr2=0x7fcb900039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:01.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.079+0000 7fcba5137640 1 --2- 172.21.15.83:0/4263131379 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcba015fa10 0x7fcb900039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.079+0000 7fcba5137640 1 -- 172.21.15.83:0/4263131379 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcb90003f30 msgr2=0x7fcb90004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:01.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.079+0000 7fcba5137640 1 --2- 172.21.15.83:0/4263131379 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcb90003f30 0x7fcb90004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.079+0000 7fcba5137640 1 -- 172.21.15.83:0/4263131379 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb90003520 con 0x7fcb90004d40 2024-03-06T07:22:01.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.079+0000 7fcba4936640 1 --2- 172.21.15.83:0/4263131379 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcba015fa10 0x7fcb900039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:01.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.080+0000 7fcba5137640 1 --2- 172.21.15.83:0/4263131379 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcb90004d40 0x7fcb900051c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fcb9800a8d0 tx=0x7fcb90005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c54f29898b26b920 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.080+0000 7fcb9f7fe640 1 -- 172.21.15.83:0/4263131379 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9800a4c0 con 0x7fcb90004d40 2024-03-06T07:22:01.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.080+0000 7fcb9f7fe640 1 -- 172.21.15.83:0/4263131379 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb98034650 con 0x7fcb90004d40 2024-03-06T07:22:01.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.080+0000 7fcb9f7fe640 1 -- 172.21.15.83:0/4263131379 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb98034a80 con 0x7fcb90004d40 2024-03-06T07:22:01.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.081+0000 7fcba6b9a640 1 -- 172.21.15.83:0/4263131379 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcb90004d40 msgr2=0x7fcb900051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.081+0000 7fcba6b9a640 1 --2- 172.21.15.83:0/4263131379 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcb90004d40 0x7fcb900051c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fcb9800a8d0 tx=0x7fcb90005830 comp rx=0 tx=0).stop 2024-03-06T07:22:01.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.081+0000 7fcba5137640 1 -- 172.21.15.83:0/4263131379 reap_dead start 2024-03-06T07:22:01.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.081+0000 7fcba6b9a640 1 -- 172.21.15.83:0/4263131379 shutdown_connections 2024-03-06T07:22:01.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.081+0000 7fcba6b9a640 1 -- 172.21.15.83:0/4263131379 >> 172.21.15.83:0/4263131379 conn(0x7fcba009a9f0 msgr2=0x7fcba009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:01.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.081+0000 7fcba6b9a640 1 -- 172.21.15.83:0/4263131379 shutdown_connections 2024-03-06T07:22:01.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.081+0000 7fcba6b9a640 1 -- 172.21.15.83:0/4263131379 wait complete. 2024-03-06T07:22:01.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.082+0000 7fcba6b9a640 1 Processor -- start 2024-03-06T07:22:01.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.082+0000 7fcba6b9a640 1 -- start start 2024-03-06T07:22:01.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.083+0000 7fcba6b9a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcb90004d40 0x7fcba015eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.083+0000 7fcba6b9a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcba00c90b0 0x7fcba00c9500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.083+0000 7fcba6b9a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcba00c9a40 0x7fcba00cb280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.083+0000 7fcba6b9a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba015c1b0 con 0x7fcb90004d40 2024-03-06T07:22:01.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.083+0000 7fcba6b9a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba00cc5d0 con 0x7fcba00c90b0 2024-03-06T07:22:01.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.083+0000 7fcba6b9a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcba00ce270 con 0x7fcba00c9a40 2024-03-06T07:22:01.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.083+0000 7fcba4936640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcb90004d40 0x7fcba015eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.083+0000 7fcba4936640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcb90004d40 0x7fcba015eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33904/0 (socket says 172.21.15.83:33904) 2024-03-06T07:22:01.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.083+0000 7fcba4936640 1 -- 172.21.15.83:0/213387191 learned_addr learned my addr 172.21.15.83:0/213387191 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:01.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.083+0000 7fcb9ffff640 1 --2- 172.21.15.83:0/213387191 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcba00c90b0 0x7fcba00c9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.083+0000 7fcba5137640 1 --2- 172.21.15.83:0/213387191 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcba00c9a40 0x7fcba00cb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.084+0000 7fcba4936640 1 -- 172.21.15.83:0/213387191 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcba00c9a40 msgr2=0x7fcba00cb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.084+0000 7fcba4936640 1 --2- 172.21.15.83:0/213387191 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcba00c9a40 0x7fcba00cb280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.084+0000 7fcba4936640 1 -- 172.21.15.83:0/213387191 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcba00c90b0 msgr2=0x7fcba00c9500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.084+0000 7fcba4936640 1 --2- 172.21.15.83:0/213387191 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcba00c90b0 0x7fcba00c9500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.084+0000 7fcba4936640 1 -- 172.21.15.83:0/213387191 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcba015f3f0 con 0x7fcb90004d40 2024-03-06T07:22:01.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.084+0000 7fcba4936640 1 --2- 172.21.15.83:0/213387191 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcb90004d40 0x7fcba015eeb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fcb94011cd0 tx=0x7fcb9400f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.084+0000 7fcb9d7fa640 1 -- 172.21.15.83:0/213387191 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb94011d00 con 0x7fcb90004d40 2024-03-06T07:22:01.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.084+0000 7fcb9d7fa640 1 -- 172.21.15.83:0/213387191 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb94014440 con 0x7fcb90004d40 2024-03-06T07:22:01.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.084+0000 7fcb9d7fa640 1 -- 172.21.15.83:0/213387191 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb94011ea0 con 0x7fcb90004d40 2024-03-06T07:22:01.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.084+0000 7fcba6b9a640 1 -- 172.21.15.83:0/213387191 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcba00b61d0 con 0x7fcb90004d40 2024-03-06T07:22:01.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.084+0000 7fcba6b9a640 1 -- 172.21.15.83:0/213387191 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcba00b6790 con 0x7fcb90004d40 2024-03-06T07:22:01.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.085+0000 7fcba6b9a640 1 -- 172.21.15.83:0/213387191 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcba0069e80 con 0x7fcb90004d40 2024-03-06T07:22:01.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.085+0000 7fcb9d7fa640 1 -- 172.21.15.83:0/213387191 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcb94012040 con 0x7fcb90004d40 2024-03-06T07:22:01.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.086+0000 7fcb9d7fa640 1 --2- 172.21.15.83:0/213387191 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcb8406b500 0x7fcb8406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.086+0000 7fcb9d7fa640 1 -- 172.21.15.83:0/213387191 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcb9408aab0 con 0x7fcb90004d40 2024-03-06T07:22:01.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.086+0000 7fcb9ffff640 1 --2- 172.21.15.83:0/213387191 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcb8406b500 0x7fcb8406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.086+0000 7fcb9ffff640 1 --2- 172.21.15.83:0/213387191 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcb8406b500 0x7fcb8406d9c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcba006f770 tx=0x7fcb8c009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.090+0000 7fcb9d7fa640 1 -- 172.21.15.83:0/213387191 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcba0069e80 con 0x7fcb90004d40 2024-03-06T07:22:01.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.307+0000 7fcba6b9a640 1 -- 172.21.15.83:0/213387191 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcba00cbed0 con 0x7fcb90004d40 2024-03-06T07:22:01.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.307+0000 7fcb9d7fa640 1 -- 172.21.15.83:0/213387191 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4974 (secure 0 0 0) 0x7fcba015f3f0 con 0x7fcb90004d40 2024-03-06T07:22:01.309 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:22:01.309 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/4249608353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":4249608353},{"type":"v1","addr":"172.21.15.83:6836","nonce":4249608353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/714452530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":714452530},{"type":"v1","addr":"172.21.15.183:6835","nonce":714452530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4282,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/2723013431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2723013431},{"type":"v1","addr":"172.21.15.183:6834","nonce":2723013431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:58.286994+0000","modified":"2024-03-06T07:22:00.307284+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/1597344199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1597344199},{"type":"v1","addr":"172.21.15.83:6837","nonce":1597344199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4287,"qdb_cluster":[4287]},"id":1}]} 2024-03-06T07:22:01.310 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 6 2024-03-06T07:22:01.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.309+0000 7fcba6b9a640 1 -- 172.21.15.83:0/213387191 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcb8406b500 msgr2=0x7fcb8406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.309+0000 7fcba6b9a640 1 --2- 172.21.15.83:0/213387191 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcb8406b500 0x7fcb8406d9c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcba006f770 tx=0x7fcb8c009190 comp rx=0 tx=0).stop 2024-03-06T07:22:01.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.309+0000 7fcba6b9a640 1 -- 172.21.15.83:0/213387191 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcb90004d40 msgr2=0x7fcba015eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.309+0000 7fcba6b9a640 1 --2- 172.21.15.83:0/213387191 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcb90004d40 0x7fcba015eeb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fcb94011cd0 tx=0x7fcb9400f610 comp rx=0 tx=0).stop 2024-03-06T07:22:01.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.309+0000 7fcba6b9a640 1 -- 172.21.15.83:0/213387191 shutdown_connections 2024-03-06T07:22:01.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.309+0000 7fcba6b9a640 1 --2- 172.21.15.83:0/213387191 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcb8406b500 0x7fcb8406d9c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.309+0000 7fcba6b9a640 1 --2- 172.21.15.83:0/213387191 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcba00c9a40 0x7fcba00cb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.309+0000 7fcba6b9a640 1 --2- 172.21.15.83:0/213387191 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcba00c90b0 0x7fcba00c9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.309+0000 7fcba6b9a640 1 --2- 172.21.15.83:0/213387191 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcb90004d40 0x7fcba015eeb0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.309+0000 7fcba6b9a640 1 -- 172.21.15.83:0/213387191 >> 172.21.15.83:0/213387191 conn(0x7fcba009a9f0 msgr2=0x7fcba009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:01.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.309+0000 7fcba6b9a640 1 -- 172.21.15.83:0/213387191 shutdown_connections 2024-03-06T07:22:01.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.309+0000 7fcba6b9a640 1 -- 172.21.15.83:0/213387191 wait complete. 2024-03-06T07:22:01.322 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:01.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.422+0000 7f1425bf6640 1 Processor -- start 2024-03-06T07:22:01.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.422+0000 7f1425bf6640 1 -- start start 2024-03-06T07:22:01.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.422+0000 7f1425bf6640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f14200b4170 0x7f14200b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.422+0000 7f1425bf6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14200d25d0 con 0x7f14200d0a80 2024-03-06T07:22:01.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.422+0000 7f1425bf6640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14200d2870 con 0x7f14200d4760 2024-03-06T07:22:01.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.422+0000 7f1425bf6640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14200d4bf0 con 0x7f14200b4170 2024-03-06T07:22:01.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.422+0000 7f141ffff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f14200d0a80 0x7f14200cee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:40940/0 (socket says 172.21.15.83:40940) 2024-03-06T07:22:01.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.422+0000 7f141ffff640 1 -- 172.21.15.83:0/2090836688 learned_addr learned my addr 172.21.15.83:0/2090836688 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:01.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.423+0000 7f141f7fe640 1 --2- 172.21.15.83:0/2090836688 >> v2:172.21.15.183:3300/0 conn(0x7f14200b4170 0x7f14200b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.423+0000 7f141effd640 1 -- 172.21.15.83:0/2090836688 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1019551360 0 0) 0x7f14200d25d0 con 0x7f14200d0a80 2024-03-06T07:22:01.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.423+0000 7f141effd640 1 -- 172.21.15.83:0/2090836688 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f140c003520 con 0x7f14200d0a80 2024-03-06T07:22:01.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.423+0000 7f141f7fe640 1 -- 172.21.15.83:0/2090836688 >> v1:172.21.15.183:6789/0 conn(0x7f14200d4760 legacy=0x7f14200d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.423+0000 7f141f7fe640 1 -- 172.21.15.83:0/2090836688 >> v1:172.21.15.83:6789/0 conn(0x7f14200d0a80 legacy=0x7f14200cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.423+0000 7f141f7fe640 1 -- 172.21.15.83:0/2090836688 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14200d2b50 con 0x7f14200b4170 2024-03-06T07:22:01.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.424+0000 7f141f7fe640 1 --2- 172.21.15.83:0/2090836688 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14200b4170 0x7f14200b6a90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f14100097a0 tx=0x7f141002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a934bc4a659b47ca server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.424+0000 7f141effd640 1 -- 172.21.15.83:0/2090836688 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1410034070 con 0x7f14200b4170 2024-03-06T07:22:01.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.424+0000 7f141effd640 1 -- 172.21.15.83:0/2090836688 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1410030610 con 0x7f14200b4170 2024-03-06T07:22:01.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.424+0000 7f141effd640 1 -- 172.21.15.83:0/2090836688 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14100309c0 con 0x7f14200b4170 2024-03-06T07:22:01.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.425+0000 7f1425bf6640 1 -- 172.21.15.83:0/2090836688 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14200b4170 msgr2=0x7f14200b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.425+0000 7f1425bf6640 1 --2- 172.21.15.83:0/2090836688 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14200b4170 0x7f14200b6a90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f14100097a0 tx=0x7f141002b960 comp rx=0 tx=0).stop 2024-03-06T07:22:01.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.425+0000 7f1425bf6640 1 -- 172.21.15.83:0/2090836688 shutdown_connections 2024-03-06T07:22:01.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.425+0000 7f1425bf6640 1 --2- 172.21.15.83:0/2090836688 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14200b4170 0x7f14200b6a90 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.425+0000 7f1425bf6640 1 -- 172.21.15.83:0/2090836688 >> 172.21.15.83:0/2090836688 conn(0x7f14200d3c40 msgr2=0x7f14200d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:01.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.425+0000 7f1425bf6640 1 -- 172.21.15.83:0/2090836688 shutdown_connections 2024-03-06T07:22:01.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.425+0000 7f1425bf6640 1 -- 172.21.15.83:0/2090836688 wait complete. 2024-03-06T07:22:01.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.426+0000 7f1425bf6640 1 Processor -- start 2024-03-06T07:22:01.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.426+0000 7f1425bf6640 1 -- start start 2024-03-06T07:22:01.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.427+0000 7f1425bf6640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f14200b4170 0x7f14201599a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.427+0000 7f1425bf6640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14200d0a80 0x7f14201567e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.427+0000 7f1425bf6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1420156ef0 0x7f142015b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.427+0000 7f1425bf6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14200b4dd0 con 0x7f1420156ef0 2024-03-06T07:22:01.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.427+0000 7f1425bf6640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14200ce9e0 con 0x7f14200d0a80 2024-03-06T07:22:01.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.427+0000 7f1425bf6640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14200b4f50 con 0x7f14200b4170 2024-03-06T07:22:01.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.427+0000 7f141ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1420156ef0 0x7f142015b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.427+0000 7f141ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1420156ef0 0x7f142015b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33906/0 (socket says 172.21.15.83:33906) 2024-03-06T07:22:01.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.427+0000 7f141ffff640 1 -- 172.21.15.83:0/57166442 learned_addr learned my addr 172.21.15.83:0/57166442 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:01.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.427+0000 7f141f7fe640 1 --2- 172.21.15.83:0/57166442 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f14200b4170 0x7f14201599a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.427+0000 7f1417fff640 1 --2- 172.21.15.83:0/57166442 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14200d0a80 0x7f14201567e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.427+0000 7f141ffff640 1 -- 172.21.15.83:0/57166442 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f14200b4170 msgr2=0x7f14201599a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.427+0000 7f141ffff640 1 --2- 172.21.15.83:0/57166442 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f14200b4170 0x7f14201599a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.427+0000 7f141ffff640 1 -- 172.21.15.83:0/57166442 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14200d0a80 msgr2=0x7f14201567e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.427+0000 7f141ffff640 1 --2- 172.21.15.83:0/57166442 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14200d0a80 0x7f14201567e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.427+0000 7f141ffff640 1 -- 172.21.15.83:0/57166442 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f142015a110 con 0x7f1420156ef0 2024-03-06T07:22:01.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.427+0000 7f1417fff640 1 --2- 172.21.15.83:0/57166442 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14200d0a80 0x7f14201567e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:01.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.427+0000 7f141ffff640 1 --2- 172.21.15.83:0/57166442 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1420156ef0 0x7f142015b390 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f141800cb40 tx=0x7f141800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.427+0000 7f141cff9640 1 -- 172.21.15.83:0/57166442 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1418017070 con 0x7f1420156ef0 2024-03-06T07:22:01.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.427+0000 7f141cff9640 1 -- 172.21.15.83:0/57166442 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f141800ac40 con 0x7f1420156ef0 2024-03-06T07:22:01.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.427+0000 7f141cff9640 1 -- 172.21.15.83:0/57166442 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1418012620 con 0x7f1420156ef0 2024-03-06T07:22:01.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.428+0000 7f1425bf6640 1 -- 172.21.15.83:0/57166442 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f142015a3f0 con 0x7f1420156ef0 2024-03-06T07:22:01.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.428+0000 7f1425bf6640 1 -- 172.21.15.83:0/57166442 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f142015b8d0 con 0x7f1420156ef0 2024-03-06T07:22:01.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.428+0000 7f1425bf6640 1 -- 172.21.15.83:0/57166442 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f142004f330 con 0x7f1420156ef0 2024-03-06T07:22:01.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.429+0000 7f141cff9640 1 -- 172.21.15.83:0/57166442 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f141801a070 con 0x7f1420156ef0 2024-03-06T07:22:01.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.429+0000 7f141cff9640 1 --2- 172.21.15.83:0/57166442 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f140006b500 0x7f140006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.429+0000 7f141f7fe640 1 --2- 172.21.15.83:0/57166442 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f140006b500 0x7f140006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.429+0000 7f141cff9640 1 -- 172.21.15.83:0/57166442 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1418089f90 con 0x7f1420156ef0 2024-03-06T07:22:01.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.430+0000 7f141f7fe640 1 --2- 172.21.15.83:0/57166442 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f140006b500 0x7f140006d9c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1410036000 tx=0x7f1410008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.434+0000 7f141cff9640 1 -- 172.21.15.83:0/57166442 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1418058730 con 0x7f1420156ef0 2024-03-06T07:22:01.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.612+0000 7f1425bf6640 1 -- 172.21.15.83:0/57166442 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14200733b0 con 0x7f1420156ef0 2024-03-06T07:22:01.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.613+0000 7f141cff9640 1 -- 172.21.15.83:0/57166442 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16617 (secure 0 0 0) 0x7f1418081e20 con 0x7f1420156ef0 2024-03-06T07:22:01.615 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:22:01.616 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":20,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:21:59.938137+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.318942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.254795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:01.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.615+0000 7f1425bf6640 1 -- 172.21.15.83:0/57166442 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f140006b500 msgr2=0x7f140006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.615+0000 7f1425bf6640 1 --2- 172.21.15.83:0/57166442 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f140006b500 0x7f140006d9c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1410036000 tx=0x7f1410008030 comp rx=0 tx=0).stop 2024-03-06T07:22:01.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.615+0000 7f1425bf6640 1 -- 172.21.15.83:0/57166442 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1420156ef0 msgr2=0x7f142015b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.615+0000 7f1425bf6640 1 --2- 172.21.15.83:0/57166442 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1420156ef0 0x7f142015b390 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f141800cb40 tx=0x7f141800a060 comp rx=0 tx=0).stop 2024-03-06T07:22:01.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.615+0000 7f1425bf6640 1 -- 172.21.15.83:0/57166442 shutdown_connections 2024-03-06T07:22:01.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.615+0000 7f1425bf6640 1 --2- 172.21.15.83:0/57166442 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f140006b500 0x7f140006d9c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.615+0000 7f1425bf6640 1 --2- 172.21.15.83:0/57166442 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1420156ef0 0x7f142015b390 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.615+0000 7f1425bf6640 1 --2- 172.21.15.83:0/57166442 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14200d0a80 0x7f14201567e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.615+0000 7f1425bf6640 1 --2- 172.21.15.83:0/57166442 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f14200b4170 0x7f14201599a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.615+0000 7f1425bf6640 1 -- 172.21.15.83:0/57166442 >> 172.21.15.83:0/57166442 conn(0x7f14200d3c40 msgr2=0x7f142006b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:01.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.615+0000 7f1425bf6640 1 -- 172.21.15.83:0/57166442 shutdown_connections 2024-03-06T07:22:01.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.615+0000 7f1425bf6640 1 -- 172.21.15.83:0/57166442 wait complete. 2024-03-06T07:22:01.628 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:22:01.628 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:01.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.725+0000 7f4762046640 1 Processor -- start 2024-03-06T07:22:01.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.725+0000 7f4762046640 1 -- start start 2024-03-06T07:22:01.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.727+0000 7f4762046640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f475c15ab00 0x7f475c0b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.727+0000 7f4762046640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f475c0b4e30 con 0x7f475c0b48f0 2024-03-06T07:22:01.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.727+0000 7f4762046640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f475c0b5110 con 0x7f475c0afdd0 2024-03-06T07:22:01.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.727+0000 7f4762046640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f475c15b310 con 0x7f475c15ab00 2024-03-06T07:22:01.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.727+0000 7f4753fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f475c0b48f0 0x7f475c0af6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:40948/0 (socket says 172.21.15.83:40948) 2024-03-06T07:22:01.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.727+0000 7f4753fff640 1 -- 172.21.15.83:0/4242264517 learned_addr learned my addr 172.21.15.83:0/4242264517 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:01.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.727+0000 7f475b7fe640 1 --2- 172.21.15.83:0/4242264517 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f475c15ab00 0x7f475c0b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.727+0000 7f475affd640 1 -- 172.21.15.83:0/4242264517 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1355536088 0 0) 0x7f475c0b4e30 con 0x7f475c0b48f0 2024-03-06T07:22:01.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.727+0000 7f475affd640 1 -- 172.21.15.83:0/4242264517 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4748003520 con 0x7f475c0b48f0 2024-03-06T07:22:01.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.727+0000 7f475affd640 1 -- 172.21.15.83:0/4242264517 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2752800649 0 0) 0x7f475c0b5110 con 0x7f475c0afdd0 2024-03-06T07:22:01.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.727+0000 7f475affd640 1 -- 172.21.15.83:0/4242264517 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f475c0b4e30 con 0x7f475c0afdd0 2024-03-06T07:22:01.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.727+0000 7f475affd640 1 -- 172.21.15.83:0/4242264517 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2951730021 0 0) 0x7f4748003520 con 0x7f475c0b48f0 2024-03-06T07:22:01.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.727+0000 7f475b7fe640 1 -- 172.21.15.83:0/4242264517 >> v1:172.21.15.183:6789/0 conn(0x7f475c0afdd0 legacy=0x7f475c15bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.727+0000 7f475b7fe640 1 -- 172.21.15.83:0/4242264517 >> v1:172.21.15.83:6789/0 conn(0x7f475c0b48f0 legacy=0x7f475c0af6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.727+0000 7f475b7fe640 1 -- 172.21.15.83:0/4242264517 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f475c15b5f0 con 0x7f475c15ab00 2024-03-06T07:22:01.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.728+0000 7f475b7fe640 1 --2- 172.21.15.83:0/4242264517 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f475c15ab00 0x7f475c0b43b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f474c0097d0 tx=0x7f474c02b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e530b9ea2f3c38de server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.728+0000 7f475affd640 1 -- 172.21.15.83:0/4242264517 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f474c037070 con 0x7f475c15ab00 2024-03-06T07:22:01.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.728+0000 7f475affd640 1 -- 172.21.15.83:0/4242264517 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f474c030510 con 0x7f475c15ab00 2024-03-06T07:22:01.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.728+0000 7f475affd640 1 -- 172.21.15.83:0/4242264517 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4744001670 con 0x7f475c15ab00 2024-03-06T07:22:01.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.729+0000 7f4762046640 1 -- 172.21.15.83:0/4242264517 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f475c15ab00 msgr2=0x7f475c0b43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.729+0000 7f4762046640 1 --2- 172.21.15.83:0/4242264517 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f475c15ab00 0x7f475c0b43b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f474c0097d0 tx=0x7f474c02b800 comp rx=0 tx=0).stop 2024-03-06T07:22:01.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.729+0000 7f4762046640 1 -- 172.21.15.83:0/4242264517 shutdown_connections 2024-03-06T07:22:01.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.729+0000 7f4762046640 1 --2- 172.21.15.83:0/4242264517 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f475c15ab00 0x7f475c0b43b0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.729+0000 7f4762046640 1 -- 172.21.15.83:0/4242264517 >> 172.21.15.83:0/4242264517 conn(0x7f475c06c0c0 msgr2=0x7f475c06c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:01.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.729+0000 7f4762046640 1 -- 172.21.15.83:0/4242264517 shutdown_connections 2024-03-06T07:22:01.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.729+0000 7f4762046640 1 -- 172.21.15.83:0/4242264517 wait complete. 2024-03-06T07:22:01.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.730+0000 7f4762046640 1 Processor -- start 2024-03-06T07:22:01.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.730+0000 7f4762046640 1 -- start start 2024-03-06T07:22:01.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.731+0000 7f4762046640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f475c0afdd0 0x7f475c0d0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.731+0000 7f4762046640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f475c0b48f0 0x7f475c0a9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.731+0000 7f4762046640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f475c15ab00 0x7f475c1494a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.731+0000 7f4762046640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f475c15b060 con 0x7f475c0b48f0 2024-03-06T07:22:01.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.731+0000 7f4762046640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f475c15aee0 con 0x7f475c0afdd0 2024-03-06T07:22:01.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.731+0000 7f4762046640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f475c15c220 con 0x7f475c15ab00 2024-03-06T07:22:01.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.731+0000 7f4753fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f475c0b48f0 0x7f475c0a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.731+0000 7f4753fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f475c0b48f0 0x7f475c0a9450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33922/0 (socket says 172.21.15.83:33922) 2024-03-06T07:22:01.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.731+0000 7f4753fff640 1 -- 172.21.15.83:0/4064506814 learned_addr learned my addr 172.21.15.83:0/4064506814 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:01.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.731+0000 7f475bfff640 1 --2- 172.21.15.83:0/4064506814 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f475c15ab00 0x7f475c1494a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.731+0000 7f475b7fe640 1 --2- 172.21.15.83:0/4064506814 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f475c0afdd0 0x7f475c0d0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.731+0000 7f4753fff640 1 -- 172.21.15.83:0/4064506814 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f475c15ab00 msgr2=0x7f475c1494a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.731+0000 7f4753fff640 1 --2- 172.21.15.83:0/4064506814 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f475c15ab00 0x7f475c1494a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.731+0000 7f4753fff640 1 -- 172.21.15.83:0/4064506814 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f475c0afdd0 msgr2=0x7f475c0d0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.731+0000 7f4753fff640 1 --2- 172.21.15.83:0/4064506814 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f475c0afdd0 0x7f475c0d0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.731+0000 7f4753fff640 1 -- 172.21.15.83:0/4064506814 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f475c0a9ae0 con 0x7f475c0b48f0 2024-03-06T07:22:01.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.731+0000 7f4753fff640 1 --2- 172.21.15.83:0/4064506814 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f475c0b48f0 0x7f475c0a9450 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f474400c780 tx=0x7f474400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.731+0000 7f4758ff9640 1 -- 172.21.15.83:0/4064506814 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4744017070 con 0x7f475c0b48f0 2024-03-06T07:22:01.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.731+0000 7f4758ff9640 1 -- 172.21.15.83:0/4064506814 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4744005af0 con 0x7f475c0b48f0 2024-03-06T07:22:01.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.731+0000 7f4762046640 1 -- 172.21.15.83:0/4064506814 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f475c0ceda0 con 0x7f475c0b48f0 2024-03-06T07:22:01.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.731+0000 7f4758ff9640 1 -- 172.21.15.83:0/4064506814 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4744007070 con 0x7f475c0b48f0 2024-03-06T07:22:01.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.731+0000 7f4762046640 1 -- 172.21.15.83:0/4064506814 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f475c0cf360 con 0x7f475c0b48f0 2024-03-06T07:22:01.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.733+0000 7f4762046640 1 -- 172.21.15.83:0/4064506814 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f475c04f330 con 0x7f475c0b48f0 2024-03-06T07:22:01.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.733+0000 7f4758ff9640 1 -- 172.21.15.83:0/4064506814 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f474401a070 con 0x7f475c0b48f0 2024-03-06T07:22:01.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.734+0000 7f4758ff9640 1 --2- 172.21.15.83:0/4064506814 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f473c06b4d0 0x7f473c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.734+0000 7f4758ff9640 1 -- 172.21.15.83:0/4064506814 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f47440897e0 con 0x7f475c0b48f0 2024-03-06T07:22:01.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.734+0000 7f475b7fe640 1 --2- 172.21.15.83:0/4064506814 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f473c06b4d0 0x7f473c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.734+0000 7f475b7fe640 1 --2- 172.21.15.83:0/4064506814 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f473c06b4d0 0x7f473c06d990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f474c034000 tx=0x7f474c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.738+0000 7f4758ff9640 1 -- 172.21.15.83:0/4064506814 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4744058660 con 0x7f475c0b48f0 2024-03-06T07:22:01.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.914+0000 7f4762046640 1 -- 172.21.15.83:0/4064506814 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f475c153590 con 0x7f473c06b4d0 2024-03-06T07:22:01.921 INFO:tasks.ceph.mds.c.smithi083.stderr:2024-03-06T07:22:01.919+0000 7f363345e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:22:01.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.920+0000 7f4758ff9640 1 -- 172.21.15.83:0/4064506814 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+237103 (secure 0 0 0) 0x7f474c030610 con 0x7f473c06b4d0 2024-03-06T07:22:01.925 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:22:01.926 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-06T07:22:01.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.922+0000 7f4762046640 1 -- 172.21.15.83:0/4064506814 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f473c06b4d0 msgr2=0x7f473c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.922+0000 7f4762046640 1 --2- 172.21.15.83:0/4064506814 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f473c06b4d0 0x7f473c06d990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f474c034000 tx=0x7f474c008030 comp rx=0 tx=0).stop 2024-03-06T07:22:01.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.922+0000 7f4762046640 1 -- 172.21.15.83:0/4064506814 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f475c0b48f0 msgr2=0x7f475c0a9450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.922+0000 7f4762046640 1 --2- 172.21.15.83:0/4064506814 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f475c0b48f0 0x7f475c0a9450 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f474400c780 tx=0x7f474400cc40 comp rx=0 tx=0).stop 2024-03-06T07:22:01.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.922+0000 7f4762046640 1 -- 172.21.15.83:0/4064506814 shutdown_connections 2024-03-06T07:22:01.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.922+0000 7f4762046640 1 --2- 172.21.15.83:0/4064506814 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f473c06b4d0 0x7f473c06d990 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.922+0000 7f4762046640 1 --2- 172.21.15.83:0/4064506814 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f475c15ab00 0x7f475c1494a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.922+0000 7f4762046640 1 --2- 172.21.15.83:0/4064506814 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f475c0b48f0 0x7f475c0a9450 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.922+0000 7f4762046640 1 --2- 172.21.15.83:0/4064506814 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f475c0afdd0 0x7f475c0d0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.922+0000 7f4762046640 1 -- 172.21.15.83:0/4064506814 >> 172.21.15.83:0/4064506814 conn(0x7f475c06c0c0 msgr2=0x7f475c15d420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:01.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.923+0000 7f4762046640 1 -- 172.21.15.83:0/4064506814 shutdown_connections 2024-03-06T07:22:01.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:01.923+0000 7f4762046640 1 -- 172.21.15.83:0/4064506814 wait complete. 2024-03-06T07:22:01.935 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-03-06T07:22:00.321569+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":218800,"kb_used_data":2376,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755920,"statfs":{"total":773094113280,"available":772870062080,"internally_reserved":0,"allocated":2433024,"data_stored":1546872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12757,"internal_metadata":221236779},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000736"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":162,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-03-06T07:21:50.467028+0000","last_change":"2024-03-06T07:21:47.553693+0000","last_active":"2024-03-06T07:21:50.467028+0000","last_peered":"2024-03-06T07:21:50.467028+0000","last_clean":"2024-03-06T07:21:50.467028+0000","last_became_active":"2024-03-06T07:21:47.552664+0000","last_became_peered":"2024-03-06T07:21:47.552664+0000","last_unstale":"2024-03-06T07:21:50.467028+0000","last_undegraded":"2024-03-06T07:21:50.467028+0000","last_fullsized":"2024-03-06T07:21:50.467028+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_clean_scrub_stamp":"2024-03-06T07:21:46.362028+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:02.916426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27752,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344088,"statfs":{"total":96636764160,"available":96608346112,"internally_reserved":0,"allocated":749568,"data_stored":636135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":45767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27180,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344660,"statfs":{"total":96636764160,"available":96608931840,"internally_reserved":0,"allocated":155648,"data_stored":45767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27172,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":155648,"data_stored":45767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27172,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":155648,"data_stored":45767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27172,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":155648,"data_stored":45767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":45767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28000,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343840,"statfs":{"total":96636764160,"available":96608092160,"internally_reserved":0,"allocated":749568,"data_stored":636135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:01.940 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:02.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.074+0000 7f4ffaa88640 1 Processor -- start 2024-03-06T07:22:02.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.074+0000 7f4ffaa88640 1 -- start start 2024-03-06T07:22:02.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.075+0000 7f4ffaa88640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4ff415bb20 0x7f4ff40d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.075+0000 7f4ffaa88640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f4ff40cfff0 0x7f4ff40d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.075+0000 7f4ffaa88640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4ff40ce4b0 0x7f4ff40ce920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.075+0000 7f4ffaa88640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ff4161980 con 0x7f4ff415bb20 2024-03-06T07:22:02.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.075+0000 7f4ffaa88640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ff4161c60 con 0x7f4ff40cfff0 2024-03-06T07:22:02.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.075+0000 7f4ffaa88640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff4161f40 con 0x7f4ff40ce4b0 2024-03-06T07:22:02.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.075+0000 7f4ff8824640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4ff415bb20 0x7f4ff40d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.075+0000 7f4ff8824640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4ff415bb20 0x7f4ff40d1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33938/0 (socket says 172.21.15.83:33938) 2024-03-06T07:22:02.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.075+0000 7f4ff8824640 1 -- 172.21.15.83:0/3048821571 learned_addr learned my addr 172.21.15.83:0/3048821571 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:02.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.075+0000 7f4ff9025640 1 --2- 172.21.15.83:0/3048821571 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4ff40ce4b0 0x7f4ff40ce920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.075+0000 7f4febfff640 1 --2- 172.21.15.83:0/3048821571 >> v2:172.21.15.183:3300/0 conn(0x7f4ff40cfff0 0x7f4ff40d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.075+0000 7f4ff8824640 1 -- 172.21.15.83:0/3048821571 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4ff40ce4b0 msgr2=0x7f4ff40ce920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:02.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.075+0000 7f4ff8824640 1 --2- 172.21.15.83:0/3048821571 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4ff40ce4b0 0x7f4ff40ce920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.075+0000 7f4ff8824640 1 -- 172.21.15.83:0/3048821571 >> v2:172.21.15.183:3300/0 conn(0x7f4ff40cfff0 msgr2=0x7f4ff40d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:02.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.075+0000 7f4ff8824640 1 --2- 172.21.15.83:0/3048821571 >> v2:172.21.15.183:3300/0 conn(0x7f4ff40cfff0 0x7f4ff40d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.075+0000 7f4ff8824640 1 -- 172.21.15.83:0/3048821571 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff4162220 con 0x7f4ff415bb20 2024-03-06T07:22:02.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.076+0000 7f4ff8824640 1 --2- 172.21.15.83:0/3048821571 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ff415bb20 0x7f4ff40d1df0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4ff4071290 tx=0x7f4ff002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d136a6cdf12b08a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.076+0000 7f4feb7fe640 1 -- 172.21.15.83:0/3048821571 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff0035020 con 0x7f4ff415bb20 2024-03-06T07:22:02.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.076+0000 7f4feb7fe640 1 -- 172.21.15.83:0/3048821571 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ff0031870 con 0x7f4ff415bb20 2024-03-06T07:22:02.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.076+0000 7f4feb7fe640 1 -- 172.21.15.83:0/3048821571 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff0031c40 con 0x7f4ff415bb20 2024-03-06T07:22:02.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.077+0000 7f4ffaa88640 1 -- 172.21.15.83:0/3048821571 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ff415bb20 msgr2=0x7f4ff40d1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.077+0000 7f4ffaa88640 1 --2- 172.21.15.83:0/3048821571 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ff415bb20 0x7f4ff40d1df0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4ff4071290 tx=0x7f4ff002b910 comp rx=0 tx=0).stop 2024-03-06T07:22:02.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.077+0000 7f4ffaa88640 1 -- 172.21.15.83:0/3048821571 shutdown_connections 2024-03-06T07:22:02.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.077+0000 7f4ffaa88640 1 --2- 172.21.15.83:0/3048821571 >> v2:172.21.15.183:3300/0 conn(0x7f4ff40cfff0 0x7f4ff40d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.077+0000 7f4ffaa88640 1 --2- 172.21.15.83:0/3048821571 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4ff40ce4b0 0x7f4ff40ce920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.077+0000 7f4ffaa88640 1 --2- 172.21.15.83:0/3048821571 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ff415bb20 0x7f4ff40d1df0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.077+0000 7f4ffaa88640 1 -- 172.21.15.83:0/3048821571 >> 172.21.15.83:0/3048821571 conn(0x7f4ff409a9f0 msgr2=0x7f4ff409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.077+0000 7f4ffaa88640 1 -- 172.21.15.83:0/3048821571 shutdown_connections 2024-03-06T07:22:02.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.077+0000 7f4ffaa88640 1 -- 172.21.15.83:0/3048821571 wait complete. 2024-03-06T07:22:02.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.078+0000 7f4ffaa88640 1 Processor -- start 2024-03-06T07:22:02.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.078+0000 7f4ffaa88640 1 -- start start 2024-03-06T07:22:02.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.079+0000 7f4ffaa88640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4ff40ce4b0 0x7f4ff41568a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.079+0000 7f4ffaa88640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4ff40cfff0 0x7f4ff40c5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.079+0000 7f4ffaa88640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ff415bb20 0x7f4ff40c77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.079+0000 7f4ffaa88640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff40d24b0 con 0x7f4ff415bb20 2024-03-06T07:22:02.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.079+0000 7f4ffaa88640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff40d2330 con 0x7f4ff40ce4b0 2024-03-06T07:22:02.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.079+0000 7f4ffaa88640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff40d1980 con 0x7f4ff40cfff0 2024-03-06T07:22:02.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.079+0000 7f4ff9025640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ff415bb20 0x7f4ff40c77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.079+0000 7f4ff9025640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ff415bb20 0x7f4ff40c77d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33950/0 (socket says 172.21.15.83:33950) 2024-03-06T07:22:02.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.079+0000 7f4ff9025640 1 -- 172.21.15.83:0/4063356593 learned_addr learned my addr 172.21.15.83:0/4063356593 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:02.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.079+0000 7f4ff9025640 1 -- 172.21.15.83:0/4063356593 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4ff40cfff0 msgr2=0x7f4ff40c5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.079+0000 7f4ff9025640 1 --2- 172.21.15.83:0/4063356593 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4ff40cfff0 0x7f4ff40c5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.079+0000 7f4ff8824640 1 --2- 172.21.15.83:0/4063356593 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4ff40ce4b0 0x7f4ff41568a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.079+0000 7f4ff9025640 1 -- 172.21.15.83:0/4063356593 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4ff40ce4b0 msgr2=0x7f4ff41568a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.079+0000 7f4ff9025640 1 --2- 172.21.15.83:0/4063356593 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4ff40ce4b0 0x7f4ff41568a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.079+0000 7f4ff9025640 1 -- 172.21.15.83:0/4063356593 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff40c5f90 con 0x7f4ff415bb20 2024-03-06T07:22:02.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.079+0000 7f4ff9025640 1 --2- 172.21.15.83:0/4063356593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ff415bb20 0x7f4ff40c77d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4fec007e60 tx=0x7f4fec00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.079+0000 7f4fe97fa640 1 -- 172.21.15.83:0/4063356593 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fec014620 con 0x7f4ff415bb20 2024-03-06T07:22:02.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.080+0000 7f4fe97fa640 1 -- 172.21.15.83:0/4063356593 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fec019070 con 0x7f4ff415bb20 2024-03-06T07:22:02.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.080+0000 7f4ffaa88640 1 -- 172.21.15.83:0/4063356593 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ff406bf00 con 0x7f4ff415bb20 2024-03-06T07:22:02.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.080+0000 7f4fe97fa640 1 -- 172.21.15.83:0/4063356593 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fec014a20 con 0x7f4ff415bb20 2024-03-06T07:22:02.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.080+0000 7f4ffaa88640 1 -- 172.21.15.83:0/4063356593 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ff4156de0 con 0x7f4ff415bb20 2024-03-06T07:22:02.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.080+0000 7f4ffaa88640 1 -- 172.21.15.83:0/4063356593 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ff404f330 con 0x7f4ff415bb20 2024-03-06T07:22:02.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.081+0000 7f4fe97fa640 1 -- 172.21.15.83:0/4063356593 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4fec014bc0 con 0x7f4ff415bb20 2024-03-06T07:22:02.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.081+0000 7f4fe97fa640 1 --2- 172.21.15.83:0/4063356593 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4fe006b500 0x7f4fe006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.082+0000 7f4fe97fa640 1 -- 172.21.15.83:0/4063356593 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4fec08b7e0 con 0x7f4ff415bb20 2024-03-06T07:22:02.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.082+0000 7f4ff8824640 1 --2- 172.21.15.83:0/4063356593 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4fe006b500 0x7f4fe006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.082+0000 7f4ff8824640 1 --2- 172.21.15.83:0/4063356593 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4fe006b500 0x7f4fe006d9c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0036000 tx=0x7f4ff0008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.085+0000 7f4fe97fa640 1 -- 172.21.15.83:0/4063356593 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4fec05a7d0 con 0x7f4ff415bb20 2024-03-06T07:22:02.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.282+0000 7f4ffaa88640 1 -- 172.21.15.83:0/4063356593 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4ff40b1820 con 0x7f4fe006b500 2024-03-06T07:22:02.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.288+0000 7f4fe97fa640 1 -- 172.21.15.83:0/4063356593 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+237103 (secure 0 0 0) 0x7f4ff40b1820 con 0x7f4fe006b500 2024-03-06T07:22:02.292 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:22:02.293 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-06T07:22:02.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.290+0000 7f4ffaa88640 1 -- 172.21.15.83:0/4063356593 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4fe006b500 msgr2=0x7f4fe006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.290+0000 7f4ffaa88640 1 --2- 172.21.15.83:0/4063356593 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4fe006b500 0x7f4fe006d9c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0036000 tx=0x7f4ff0008030 comp rx=0 tx=0).stop 2024-03-06T07:22:02.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.290+0000 7f4ffaa88640 1 -- 172.21.15.83:0/4063356593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ff415bb20 msgr2=0x7f4ff40c77d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.290+0000 7f4ffaa88640 1 --2- 172.21.15.83:0/4063356593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ff415bb20 0x7f4ff40c77d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4fec007e60 tx=0x7f4fec00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.291+0000 7f4ffaa88640 1 -- 172.21.15.83:0/4063356593 shutdown_connections 2024-03-06T07:22:02.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.291+0000 7f4ffaa88640 1 --2- 172.21.15.83:0/4063356593 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4fe006b500 0x7f4fe006d9c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.291+0000 7f4ffaa88640 1 --2- 172.21.15.83:0/4063356593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ff415bb20 0x7f4ff40c77d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.291+0000 7f4ffaa88640 1 --2- 172.21.15.83:0/4063356593 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4ff40cfff0 0x7f4ff40c5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.291+0000 7f4ffaa88640 1 --2- 172.21.15.83:0/4063356593 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4ff40ce4b0 0x7f4ff41568a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.291+0000 7f4ffaa88640 1 -- 172.21.15.83:0/4063356593 >> 172.21.15.83:0/4063356593 conn(0x7f4ff409a9f0 msgr2=0x7f4ff406ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.291+0000 7f4ffaa88640 1 -- 172.21.15.83:0/4063356593 shutdown_connections 2024-03-06T07:22:02.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.291+0000 7f4ffaa88640 1 -- 172.21.15.83:0/4063356593 wait complete. 2024-03-06T07:22:02.303 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-03-06T07:22:00.321569+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":218800,"kb_used_data":2376,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755920,"statfs":{"total":773094113280,"available":772870062080,"internally_reserved":0,"allocated":2433024,"data_stored":1546872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12757,"internal_metadata":221236779},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000736"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":162,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-03-06T07:21:50.467028+0000","last_change":"2024-03-06T07:21:47.553693+0000","last_active":"2024-03-06T07:21:50.467028+0000","last_peered":"2024-03-06T07:21:50.467028+0000","last_clean":"2024-03-06T07:21:50.467028+0000","last_became_active":"2024-03-06T07:21:47.552664+0000","last_became_peered":"2024-03-06T07:21:47.552664+0000","last_unstale":"2024-03-06T07:21:50.467028+0000","last_undegraded":"2024-03-06T07:21:50.467028+0000","last_fullsized":"2024-03-06T07:21:50.467028+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_clean_scrub_stamp":"2024-03-06T07:21:46.362028+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:02.916426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.927774+0000","last_change":"2024-03-06T07:21:57.927774+0000","last_active":"2024-03-06T07:21:57.927774+0000","last_peered":"2024-03-06T07:21:57.927774+0000","last_clean":"2024-03-06T07:21:57.927774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.927774+0000","last_undegraded":"2024-03-06T07:21:57.927774+0000","last_fullsized":"2024-03-06T07:21:57.927774+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:56.923997+0000","last_change":"2024-03-06T07:21:56.923997+0000","last_active":"2024-03-06T07:21:56.923997+0000","last_peered":"2024-03-06T07:21:56.923997+0000","last_clean":"2024-03-06T07:21:56.923997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.923997+0000","last_undegraded":"2024-03-06T07:21:56.923997+0000","last_fullsized":"2024-03-06T07:21:56.923997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27752,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344088,"statfs":{"total":96636764160,"available":96608346112,"internally_reserved":0,"allocated":749568,"data_stored":636135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":45767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27180,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344660,"statfs":{"total":96636764160,"available":96608931840,"internally_reserved":0,"allocated":155648,"data_stored":45767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27172,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":155648,"data_stored":45767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27172,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":155648,"data_stored":45767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27172,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":155648,"data_stored":45767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":45767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28000,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343840,"statfs":{"total":96636764160,"available":96608092160,"internally_reserved":0,"allocated":749568,"data_stored":636135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:02.307 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:02.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.441+0000 7fcacf08d640 1 Processor -- start 2024-03-06T07:22:02.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.442+0000 7fcacf08d640 1 -- start start 2024-03-06T07:22:02.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.442+0000 7fcacf08d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcac815bdd0 0x7fcac80c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.442+0000 7fcacf08d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcac80d16e0 0x7fcac80d1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.442+0000 7fcacf08d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcac80cebe0 con 0x7fcac815fa10 2024-03-06T07:22:02.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.442+0000 7fcacf08d640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcac80ce980 con 0x7fcac80d16e0 2024-03-06T07:22:02.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.442+0000 7fcacf08d640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcac80ceec0 con 0x7fcac815bdd0 2024-03-06T07:22:02.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.442+0000 7fcacd62a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcac80d16e0 0x7fcac80d1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.442+0000 7fcacd62a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcac80d16e0 0x7fcac80d1b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33966/0 (socket says 172.21.15.83:33966) 2024-03-06T07:22:02.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.442+0000 7fcacd62a640 1 -- 172.21.15.83:0/932162719 learned_addr learned my addr 172.21.15.83:0/932162719 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:02.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.442+0000 7fcacce29640 1 --2- 172.21.15.83:0/932162719 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcac815bdd0 0x7fcac80c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.443+0000 7fcacd62a640 1 -- 172.21.15.83:0/932162719 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcac815bdd0 msgr2=0x7fcac80c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:02.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.443+0000 7fcacd62a640 1 --2- 172.21.15.83:0/932162719 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcac815bdd0 0x7fcac80c5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.443+0000 7fcacd62a640 1 -- 172.21.15.83:0/932162719 >> v1:172.21.15.183:6789/0 conn(0x7fcac815fa10 legacy=0x7fcac80ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.443+0000 7fcacd62a640 1 -- 172.21.15.83:0/932162719 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac80cf1a0 con 0x7fcac80d16e0 2024-03-06T07:22:02.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.443+0000 7fcacd62a640 1 --2- 172.21.15.83:0/932162719 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcac80d16e0 0x7fcac80d1b30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcac0009e50 tx=0x7fcac002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e487dee39678e49 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.444+0000 7fcac77fe640 1 -- 172.21.15.83:0/932162719 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac002ca70 con 0x7fcac80d16e0 2024-03-06T07:22:02.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.444+0000 7fcac77fe640 1 -- 172.21.15.83:0/932162719 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcac002cc10 con 0x7fcac80d16e0 2024-03-06T07:22:02.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.444+0000 7fcac77fe640 1 -- 172.21.15.83:0/932162719 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac00343f0 con 0x7fcac80d16e0 2024-03-06T07:22:02.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.444+0000 7fcacf08d640 1 -- 172.21.15.83:0/932162719 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcac80d16e0 msgr2=0x7fcac80d1b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.444+0000 7fcacf08d640 1 --2- 172.21.15.83:0/932162719 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcac80d16e0 0x7fcac80d1b30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcac0009e50 tx=0x7fcac002bed0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.444+0000 7fcacf08d640 1 -- 172.21.15.83:0/932162719 shutdown_connections 2024-03-06T07:22:02.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.444+0000 7fcacf08d640 1 --2- 172.21.15.83:0/932162719 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcac80d16e0 0x7fcac80d1b30 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.444+0000 7fcacf08d640 1 --2- 172.21.15.83:0/932162719 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcac815bdd0 0x7fcac80c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.444+0000 7fcacf08d640 1 -- 172.21.15.83:0/932162719 >> 172.21.15.83:0/932162719 conn(0x7fcac809a9f0 msgr2=0x7fcac809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.444+0000 7fcacf08d640 1 -- 172.21.15.83:0/932162719 shutdown_connections 2024-03-06T07:22:02.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.444+0000 7fcacf08d640 1 -- 172.21.15.83:0/932162719 wait complete. 2024-03-06T07:22:02.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.445+0000 7fcacf08d640 1 Processor -- start 2024-03-06T07:22:02.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.445+0000 7fcacf08d640 1 -- start start 2024-03-06T07:22:02.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.446+0000 7fcacf08d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcac80d16e0 0x7fcac80c9260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.446+0000 7fcacf08d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcac815bdd0 0x7fcac815e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.446+0000 7fcacf08d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcac815fa10 0x7fcac81e5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.446+0000 7fcacf08d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac80d2070 con 0x7fcac815fa10 2024-03-06T07:22:02.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.446+0000 7fcacf08d640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac80c5b50 con 0x7fcac80d16e0 2024-03-06T07:22:02.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.446+0000 7fcacf08d640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcac80cc5d0 con 0x7fcac815bdd0 2024-03-06T07:22:02.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.446+0000 7fcacd62a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcac815fa10 0x7fcac81e5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.446+0000 7fcacd62a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcac815fa10 0x7fcac81e5730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33982/0 (socket says 172.21.15.83:33982) 2024-03-06T07:22:02.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.446+0000 7fcacd62a640 1 -- 172.21.15.83:0/2254253775 learned_addr learned my addr 172.21.15.83:0/2254253775 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:02.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.447+0000 7fcacce29640 1 --2- 172.21.15.83:0/2254253775 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcac80d16e0 0x7fcac80c9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.447+0000 7fcacd62a640 1 -- 172.21.15.83:0/2254253775 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcac815bdd0 msgr2=0x7fcac815e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.447+0000 7fcacd62a640 1 --2- 172.21.15.83:0/2254253775 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcac815bdd0 0x7fcac815e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.447+0000 7fcacd62a640 1 -- 172.21.15.83:0/2254253775 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcac80d16e0 msgr2=0x7fcac80c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.447+0000 7fcacd62a640 1 --2- 172.21.15.83:0/2254253775 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcac80d16e0 0x7fcac80c9260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.447+0000 7fcacd62a640 1 -- 172.21.15.83:0/2254253775 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac815b310 con 0x7fcac815fa10 2024-03-06T07:22:02.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.447+0000 7fcacd62a640 1 --2- 172.21.15.83:0/2254253775 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcac815fa10 0x7fcac81e5730 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fcac0009e50 tx=0x7fcac0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.447+0000 7fcac57fa640 1 -- 172.21.15.83:0/2254253775 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac0043020 con 0x7fcac815fa10 2024-03-06T07:22:02.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.447+0000 7fcac57fa640 1 -- 172.21.15.83:0/2254253775 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcac003e450 con 0x7fcac815fa10 2024-03-06T07:22:02.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.447+0000 7fcacf08d640 1 -- 172.21.15.83:0/2254253775 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcac80c97a0 con 0x7fcac815fa10 2024-03-06T07:22:02.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.447+0000 7fcac57fa640 1 -- 172.21.15.83:0/2254253775 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac0009510 con 0x7fcac815fa10 2024-03-06T07:22:02.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.447+0000 7fcacf08d640 1 -- 172.21.15.83:0/2254253775 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcac8147e80 con 0x7fcac815fa10 2024-03-06T07:22:02.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.448+0000 7fcacf08d640 1 -- 172.21.15.83:0/2254253775 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcac804f330 con 0x7fcac815fa10 2024-03-06T07:22:02.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.449+0000 7fcac57fa640 1 -- 172.21.15.83:0/2254253775 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcac0046020 con 0x7fcac815fa10 2024-03-06T07:22:02.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.449+0000 7fcac57fa640 1 --2- 172.21.15.83:0/2254253775 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcaac06b4d0 0x7fcaac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.449+0000 7fcac57fa640 1 -- 172.21.15.83:0/2254253775 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcac00aca50 con 0x7fcac815fa10 2024-03-06T07:22:02.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.449+0000 7fcacce29640 1 --2- 172.21.15.83:0/2254253775 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcaac06b4d0 0x7fcaac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.449+0000 7fcacce29640 1 --2- 172.21.15.83:0/2254253775 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcaac06b4d0 0x7fcaac06d990 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fcac815b650 tx=0x7fcabc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.453+0000 7fcac57fa640 1 -- 172.21.15.83:0/2254253775 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcac004d030 con 0x7fcac815fa10 2024-03-06T07:22:02.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.628+0000 7fcacf08d640 1 -- 172.21.15.83:0/2254253775 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcac809b290 con 0x7fcaac06b4d0 2024-03-06T07:22:02.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.640+0000 7fcac57fa640 1 -- 172.21.15.83:0/2254253775 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251946 (secure 0 0 0) 0x7fcac809b290 con 0x7fcaac06b4d0 2024-03-06T07:22:02.644 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:22:02.645 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-06T07:22:02.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.643+0000 7fcacf08d640 1 -- 172.21.15.83:0/2254253775 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcaac06b4d0 msgr2=0x7fcaac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.643+0000 7fcacf08d640 1 --2- 172.21.15.83:0/2254253775 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcaac06b4d0 0x7fcaac06d990 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fcac815b650 tx=0x7fcabc008040 comp rx=0 tx=0).stop 2024-03-06T07:22:02.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.643+0000 7fcacf08d640 1 -- 172.21.15.83:0/2254253775 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcac815fa10 msgr2=0x7fcac81e5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.643+0000 7fcacf08d640 1 --2- 172.21.15.83:0/2254253775 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcac815fa10 0x7fcac81e5730 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fcac0009e50 tx=0x7fcac0004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:02.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.643+0000 7fcacf08d640 1 -- 172.21.15.83:0/2254253775 shutdown_connections 2024-03-06T07:22:02.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.643+0000 7fcacf08d640 1 --2- 172.21.15.83:0/2254253775 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcaac06b4d0 0x7fcaac06d990 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.643+0000 7fcacf08d640 1 --2- 172.21.15.83:0/2254253775 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcac815fa10 0x7fcac81e5730 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.643+0000 7fcacf08d640 1 --2- 172.21.15.83:0/2254253775 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcac815bdd0 0x7fcac815e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.643+0000 7fcacf08d640 1 --2- 172.21.15.83:0/2254253775 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcac80d16e0 0x7fcac80c9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.643+0000 7fcacf08d640 1 -- 172.21.15.83:0/2254253775 >> 172.21.15.83:0/2254253775 conn(0x7fcac809a9f0 msgr2=0x7fcac80ccd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.643+0000 7fcacf08d640 1 -- 172.21.15.83:0/2254253775 shutdown_connections 2024-03-06T07:22:02.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.643+0000 7fcacf08d640 1 -- 172.21.15.83:0/2254253775 wait complete. 2024-03-06T07:22:02.659 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-03-06T07:22:02.321898+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":167,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218892,"kb_used_data":2728,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755828,"statfs":{"total":773094113280,"available":772869967872,"internally_reserved":0,"allocated":2793472,"data_stored":1702628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12757,"internal_metadata":221236779},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.393612"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299166+0000","last_change":"2024-03-06T07:21:58.961070+0000","last_active":"2024-03-06T07:22:00.299166+0000","last_peered":"2024-03-06T07:22:00.299166+0000","last_clean":"2024-03-06T07:22:00.299166+0000","last_became_active":"2024-03-06T07:21:58.960784+0000","last_became_peered":"2024-03-06T07:21:58.960784+0000","last_unstale":"2024-03-06T07:22:00.299166+0000","last_undegraded":"2024-03-06T07:22:00.299166+0000","last_fullsized":"2024-03-06T07:22:00.299166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:06.413968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943975+0000","last_change":"2024-03-06T07:21:58.958978+0000","last_active":"2024-03-06T07:21:59.943975+0000","last_peered":"2024-03-06T07:21:59.943975+0000","last_clean":"2024-03-06T07:21:59.943975+0000","last_became_active":"2024-03-06T07:21:58.958728+0000","last_became_peered":"2024-03-06T07:21:58.958728+0000","last_unstale":"2024-03-06T07:21:59.943975+0000","last_undegraded":"2024-03-06T07:21:59.943975+0000","last_fullsized":"2024-03-06T07:21:59.943975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:54.495097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298515+0000","last_change":"2024-03-06T07:21:58.953968+0000","last_active":"2024-03-06T07:22:00.298515+0000","last_peered":"2024-03-06T07:22:00.298515+0000","last_clean":"2024-03-06T07:22:00.298515+0000","last_became_active":"2024-03-06T07:21:58.953678+0000","last_became_peered":"2024-03-06T07:21:58.953678+0000","last_unstale":"2024-03-06T07:22:00.298515+0000","last_undegraded":"2024-03-06T07:22:00.298515+0000","last_fullsized":"2024-03-06T07:22:00.298515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:51:39.557766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298069+0000","last_change":"2024-03-06T07:21:58.970682+0000","last_active":"2024-03-06T07:22:00.298069+0000","last_peered":"2024-03-06T07:22:00.298069+0000","last_clean":"2024-03-06T07:22:00.298069+0000","last_became_active":"2024-03-06T07:21:58.970397+0000","last_became_peered":"2024-03-06T07:21:58.970397+0000","last_unstale":"2024-03-06T07:22:00.298069+0000","last_undegraded":"2024-03-06T07:22:00.298069+0000","last_fullsized":"2024-03-06T07:22:00.298069+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:44.732365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298736+0000","last_change":"2024-03-06T07:21:58.965912+0000","last_active":"2024-03-06T07:22:00.298736+0000","last_peered":"2024-03-06T07:22:00.298736+0000","last_clean":"2024-03-06T07:22:00.298736+0000","last_became_active":"2024-03-06T07:21:58.965567+0000","last_became_peered":"2024-03-06T07:21:58.965567+0000","last_unstale":"2024-03-06T07:22:00.298736+0000","last_undegraded":"2024-03-06T07:22:00.298736+0000","last_fullsized":"2024-03-06T07:22:00.298736+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:10:08.320367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298064+0000","last_change":"2024-03-06T07:21:58.963244+0000","last_active":"2024-03-06T07:22:00.298064+0000","last_peered":"2024-03-06T07:22:00.298064+0000","last_clean":"2024-03-06T07:22:00.298064+0000","last_became_active":"2024-03-06T07:21:58.962903+0000","last_became_peered":"2024-03-06T07:21:58.962903+0000","last_unstale":"2024-03-06T07:22:00.298064+0000","last_undegraded":"2024-03-06T07:22:00.298064+0000","last_fullsized":"2024-03-06T07:22:00.298064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:30:54.760925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.297983+0000","last_change":"2024-03-06T07:21:58.964015+0000","last_active":"2024-03-06T07:22:00.297983+0000","last_peered":"2024-03-06T07:22:00.297983+0000","last_clean":"2024-03-06T07:22:00.297983+0000","last_became_active":"2024-03-06T07:21:58.962062+0000","last_became_peered":"2024-03-06T07:21:58.962062+0000","last_unstale":"2024-03-06T07:22:00.297983+0000","last_undegraded":"2024-03-06T07:22:00.297983+0000","last_fullsized":"2024-03-06T07:22:00.297983+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:44.732365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298608+0000","last_change":"2024-03-06T07:21:58.968307+0000","last_active":"2024-03-06T07:22:00.298608+0000","last_peered":"2024-03-06T07:22:00.298608+0000","last_clean":"2024-03-06T07:22:00.298608+0000","last_became_active":"2024-03-06T07:21:58.967917+0000","last_became_peered":"2024-03-06T07:21:58.967917+0000","last_unstale":"2024-03-06T07:22:00.298608+0000","last_undegraded":"2024-03-06T07:22:00.298608+0000","last_fullsized":"2024-03-06T07:22:00.298608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:21:17.637762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299130+0000","last_change":"2024-03-06T07:21:58.971497+0000","last_active":"2024-03-06T07:22:00.299130+0000","last_peered":"2024-03-06T07:22:00.299130+0000","last_clean":"2024-03-06T07:22:00.299130+0000","last_became_active":"2024-03-06T07:21:58.970710+0000","last_became_peered":"2024-03-06T07:21:58.970710+0000","last_unstale":"2024-03-06T07:22:00.299130+0000","last_undegraded":"2024-03-06T07:22:00.299130+0000","last_fullsized":"2024-03-06T07:22:00.299130+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:13.580324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943457+0000","last_change":"2024-03-06T07:21:58.956333+0000","last_active":"2024-03-06T07:21:59.943457+0000","last_peered":"2024-03-06T07:21:59.943457+0000","last_clean":"2024-03-06T07:21:59.943457+0000","last_became_active":"2024-03-06T07:21:58.956073+0000","last_became_peered":"2024-03-06T07:21:58.956073+0000","last_unstale":"2024-03-06T07:21:59.943457+0000","last_undegraded":"2024-03-06T07:21:59.943457+0000","last_fullsized":"2024-03-06T07:21:59.943457+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:30:54.760925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943172+0000","last_change":"2024-03-06T07:21:58.956210+0000","last_active":"2024-03-06T07:21:59.943172+0000","last_peered":"2024-03-06T07:21:59.943172+0000","last_clean":"2024-03-06T07:21:59.943172+0000","last_became_active":"2024-03-06T07:21:58.955949+0000","last_became_peered":"2024-03-06T07:21:58.955949+0000","last_unstale":"2024-03-06T07:21:59.943172+0000","last_undegraded":"2024-03-06T07:21:59.943172+0000","last_fullsized":"2024-03-06T07:21:59.943172+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:54.495097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943018+0000","last_change":"2024-03-06T07:21:58.950931+0000","last_active":"2024-03-06T07:21:59.943018+0000","last_peered":"2024-03-06T07:21:59.943018+0000","last_clean":"2024-03-06T07:21:59.943018+0000","last_became_active":"2024-03-06T07:21:58.949704+0000","last_became_peered":"2024-03-06T07:21:58.949704+0000","last_unstale":"2024-03-06T07:21:59.943018+0000","last_undegraded":"2024-03-06T07:21:59.943018+0000","last_fullsized":"2024-03-06T07:21:59.943018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:02.162943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299186+0000","last_change":"2024-03-06T07:21:58.956175+0000","last_active":"2024-03-06T07:22:00.299186+0000","last_peered":"2024-03-06T07:22:00.299186+0000","last_clean":"2024-03-06T07:22:00.299186+0000","last_became_active":"2024-03-06T07:21:58.955857+0000","last_became_peered":"2024-03-06T07:21:58.955857+0000","last_unstale":"2024-03-06T07:22:00.299186+0000","last_undegraded":"2024-03-06T07:22:00.299186+0000","last_fullsized":"2024-03-06T07:22:00.299186+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:44.732365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298168+0000","last_change":"2024-03-06T07:21:58.951176+0000","last_active":"2024-03-06T07:22:00.298168+0000","last_peered":"2024-03-06T07:22:00.298168+0000","last_clean":"2024-03-06T07:22:00.298168+0000","last_became_active":"2024-03-06T07:21:58.950874+0000","last_became_peered":"2024-03-06T07:21:58.950874+0000","last_unstale":"2024-03-06T07:22:00.298168+0000","last_undegraded":"2024-03-06T07:22:00.298168+0000","last_fullsized":"2024-03-06T07:22:00.298168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:42:14.052025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943706+0000","last_change":"2024-03-06T07:21:58.954562+0000","last_active":"2024-03-06T07:21:59.943706+0000","last_peered":"2024-03-06T07:21:59.943706+0000","last_clean":"2024-03-06T07:21:59.943706+0000","last_became_active":"2024-03-06T07:21:58.954242+0000","last_became_peered":"2024-03-06T07:21:58.954242+0000","last_unstale":"2024-03-06T07:21:59.943706+0000","last_undegraded":"2024-03-06T07:21:59.943706+0000","last_fullsized":"2024-03-06T07:21:59.943706+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:22.038204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.297923+0000","last_change":"2024-03-06T07:21:58.963478+0000","last_active":"2024-03-06T07:22:00.297923+0000","last_peered":"2024-03-06T07:22:00.297923+0000","last_clean":"2024-03-06T07:22:00.297923+0000","last_became_active":"2024-03-06T07:21:58.962917+0000","last_became_peered":"2024-03-06T07:21:58.962917+0000","last_unstale":"2024-03-06T07:22:00.297923+0000","last_undegraded":"2024-03-06T07:22:00.297923+0000","last_fullsized":"2024-03-06T07:22:00.297923+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:54.495097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298958+0000","last_change":"2024-03-06T07:21:58.965994+0000","last_active":"2024-03-06T07:22:00.298958+0000","last_peered":"2024-03-06T07:22:00.298958+0000","last_clean":"2024-03-06T07:22:00.298958+0000","last_became_active":"2024-03-06T07:21:58.965655+0000","last_became_peered":"2024-03-06T07:21:58.965655+0000","last_unstale":"2024-03-06T07:22:00.298958+0000","last_undegraded":"2024-03-06T07:22:00.298958+0000","last_fullsized":"2024-03-06T07:22:00.298958+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:06.413968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298767+0000","last_change":"2024-03-06T07:21:58.955322+0000","last_active":"2024-03-06T07:22:00.298767+0000","last_peered":"2024-03-06T07:22:00.298767+0000","last_clean":"2024-03-06T07:22:00.298767+0000","last_became_active":"2024-03-06T07:21:58.955080+0000","last_became_peered":"2024-03-06T07:21:58.955080+0000","last_unstale":"2024-03-06T07:22:00.298767+0000","last_undegraded":"2024-03-06T07:22:00.298767+0000","last_fullsized":"2024-03-06T07:22:00.298767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:20.851108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298974+0000","last_change":"2024-03-06T07:21:58.966461+0000","last_active":"2024-03-06T07:22:00.298974+0000","last_peered":"2024-03-06T07:22:00.298974+0000","last_clean":"2024-03-06T07:22:00.298974+0000","last_became_active":"2024-03-06T07:21:58.966154+0000","last_became_peered":"2024-03-06T07:21:58.966154+0000","last_unstale":"2024-03-06T07:22:00.298974+0000","last_undegraded":"2024-03-06T07:22:00.298974+0000","last_fullsized":"2024-03-06T07:22:00.298974+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:37:30.965258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942831+0000","last_change":"2024-03-06T07:21:58.953916+0000","last_active":"2024-03-06T07:21:59.942831+0000","last_peered":"2024-03-06T07:21:59.942831+0000","last_clean":"2024-03-06T07:21:59.942831+0000","last_became_active":"2024-03-06T07:21:58.953623+0000","last_became_peered":"2024-03-06T07:21:58.953623+0000","last_unstale":"2024-03-06T07:21:59.942831+0000","last_undegraded":"2024-03-06T07:21:59.942831+0000","last_fullsized":"2024-03-06T07:21:59.942831+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:21:17.637762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299245+0000","last_change":"2024-03-06T07:21:58.955378+0000","last_active":"2024-03-06T07:22:00.299245+0000","last_peered":"2024-03-06T07:22:00.299245+0000","last_clean":"2024-03-06T07:22:00.299245+0000","last_became_active":"2024-03-06T07:21:58.955118+0000","last_became_peered":"2024-03-06T07:21:58.955118+0000","last_unstale":"2024-03-06T07:22:00.299245+0000","last_undegraded":"2024-03-06T07:22:00.299245+0000","last_fullsized":"2024-03-06T07:22:00.299245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:22.038204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298483+0000","last_change":"2024-03-06T07:21:58.963849+0000","last_active":"2024-03-06T07:22:00.298483+0000","last_peered":"2024-03-06T07:22:00.298483+0000","last_clean":"2024-03-06T07:22:00.298483+0000","last_became_active":"2024-03-06T07:21:58.963573+0000","last_became_peered":"2024-03-06T07:21:58.963573+0000","last_unstale":"2024-03-06T07:22:00.298483+0000","last_undegraded":"2024-03-06T07:22:00.298483+0000","last_fullsized":"2024-03-06T07:22:00.298483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:20.758138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299836+0000","last_change":"2024-03-06T07:21:58.962026+0000","last_active":"2024-03-06T07:22:00.299836+0000","last_peered":"2024-03-06T07:22:00.299836+0000","last_clean":"2024-03-06T07:22:00.299836+0000","last_became_active":"2024-03-06T07:21:58.961691+0000","last_became_peered":"2024-03-06T07:21:58.961691+0000","last_unstale":"2024-03-06T07:22:00.299836+0000","last_undegraded":"2024-03-06T07:22:00.299836+0000","last_fullsized":"2024-03-06T07:22:00.299836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:02:11.315323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298380+0000","last_change":"2024-03-06T07:21:58.970098+0000","last_active":"2024-03-06T07:22:00.298380+0000","last_peered":"2024-03-06T07:22:00.298380+0000","last_clean":"2024-03-06T07:22:00.298380+0000","last_became_active":"2024-03-06T07:21:58.969786+0000","last_became_peered":"2024-03-06T07:21:58.969786+0000","last_unstale":"2024-03-06T07:22:00.298380+0000","last_undegraded":"2024-03-06T07:22:00.298380+0000","last_fullsized":"2024-03-06T07:22:00.298380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:07.060045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.297875+0000","last_change":"2024-03-06T07:21:58.961107+0000","last_active":"2024-03-06T07:22:00.297875+0000","last_peered":"2024-03-06T07:22:00.297875+0000","last_clean":"2024-03-06T07:22:00.297875+0000","last_became_active":"2024-03-06T07:21:58.960826+0000","last_became_peered":"2024-03-06T07:21:58.960826+0000","last_unstale":"2024-03-06T07:22:00.297875+0000","last_undegraded":"2024-03-06T07:22:00.297875+0000","last_fullsized":"2024-03-06T07:22:00.297875+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:36:02.122800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942422+0000","last_change":"2024-03-06T07:21:58.954994+0000","last_active":"2024-03-06T07:21:59.942422+0000","last_peered":"2024-03-06T07:21:59.942422+0000","last_clean":"2024-03-06T07:21:59.942422+0000","last_became_active":"2024-03-06T07:21:58.954710+0000","last_became_peered":"2024-03-06T07:21:58.954710+0000","last_unstale":"2024-03-06T07:21:59.942422+0000","last_undegraded":"2024-03-06T07:21:59.942422+0000","last_fullsized":"2024-03-06T07:21:59.942422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:12.378982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943938+0000","last_change":"2024-03-06T07:21:58.956241+0000","last_active":"2024-03-06T07:21:59.943938+0000","last_peered":"2024-03-06T07:21:59.943938+0000","last_clean":"2024-03-06T07:21:59.943938+0000","last_became_active":"2024-03-06T07:21:58.955977+0000","last_became_peered":"2024-03-06T07:21:58.955977+0000","last_unstale":"2024-03-06T07:21:59.943938+0000","last_undegraded":"2024-03-06T07:21:59.943938+0000","last_fullsized":"2024-03-06T07:21:59.943938+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:21:17.637762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298498+0000","last_change":"2024-03-06T07:21:58.963500+0000","last_active":"2024-03-06T07:22:00.298498+0000","last_peered":"2024-03-06T07:22:00.298498+0000","last_clean":"2024-03-06T07:22:00.298498+0000","last_became_active":"2024-03-06T07:21:58.963218+0000","last_became_peered":"2024-03-06T07:21:58.963218+0000","last_unstale":"2024-03-06T07:22:00.298498+0000","last_undegraded":"2024-03-06T07:22:00.298498+0000","last_fullsized":"2024-03-06T07:22:00.298498+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:12.378982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299375+0000","last_change":"2024-03-06T07:21:58.957981+0000","last_active":"2024-03-06T07:22:00.299375+0000","last_peered":"2024-03-06T07:22:00.299375+0000","last_clean":"2024-03-06T07:22:00.299375+0000","last_became_active":"2024-03-06T07:21:58.957690+0000","last_became_peered":"2024-03-06T07:21:58.957690+0000","last_unstale":"2024-03-06T07:22:00.299375+0000","last_undegraded":"2024-03-06T07:22:00.299375+0000","last_fullsized":"2024-03-06T07:22:00.299375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:59.261197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298443+0000","last_change":"2024-03-06T07:21:58.968221+0000","last_active":"2024-03-06T07:22:00.298443+0000","last_peered":"2024-03-06T07:22:00.298443+0000","last_clean":"2024-03-06T07:22:00.298443+0000","last_became_active":"2024-03-06T07:21:58.967798+0000","last_became_peered":"2024-03-06T07:21:58.967798+0000","last_unstale":"2024-03-06T07:22:00.298443+0000","last_undegraded":"2024-03-06T07:22:00.298443+0000","last_fullsized":"2024-03-06T07:22:00.298443+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:37:30.965258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298249+0000","last_change":"2024-03-06T07:21:58.966486+0000","last_active":"2024-03-06T07:22:00.298249+0000","last_peered":"2024-03-06T07:22:00.298249+0000","last_clean":"2024-03-06T07:22:00.298249+0000","last_became_active":"2024-03-06T07:21:58.966202+0000","last_became_peered":"2024-03-06T07:21:58.966202+0000","last_unstale":"2024-03-06T07:22:00.298249+0000","last_undegraded":"2024-03-06T07:22:00.298249+0000","last_fullsized":"2024-03-06T07:22:00.298249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:20.908786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942026+0000","last_change":"2024-03-06T07:21:58.952310+0000","last_active":"2024-03-06T07:21:59.942026+0000","last_peered":"2024-03-06T07:21:59.942026+0000","last_clean":"2024-03-06T07:21:59.942026+0000","last_became_active":"2024-03-06T07:21:58.952054+0000","last_became_peered":"2024-03-06T07:21:58.952054+0000","last_unstale":"2024-03-06T07:21:59.942026+0000","last_undegraded":"2024-03-06T07:21:59.942026+0000","last_fullsized":"2024-03-06T07:21:59.942026+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:55.294652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942908+0000","last_change":"2024-03-06T07:21:58.951201+0000","last_active":"2024-03-06T07:21:59.942908+0000","last_peered":"2024-03-06T07:21:59.942908+0000","last_clean":"2024-03-06T07:21:59.942908+0000","last_became_active":"2024-03-06T07:21:58.950746+0000","last_became_peered":"2024-03-06T07:21:58.950746+0000","last_unstale":"2024-03-06T07:21:59.942908+0000","last_undegraded":"2024-03-06T07:21:59.942908+0000","last_fullsized":"2024-03-06T07:21:59.942908+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:52.785061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299467+0000","last_change":"2024-03-06T07:21:57.947333+0000","last_active":"2024-03-06T07:22:00.299467+0000","last_peered":"2024-03-06T07:22:00.299467+0000","last_clean":"2024-03-06T07:22:00.299467+0000","last_became_active":"2024-03-06T07:21:57.947066+0000","last_became_peered":"2024-03-06T07:21:57.947066+0000","last_unstale":"2024-03-06T07:22:00.299467+0000","last_undegraded":"2024-03-06T07:22:00.299467+0000","last_fullsized":"2024-03-06T07:22:00.299467+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:18:20.077304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299505+0000","last_change":"2024-03-06T07:21:58.954116+0000","last_active":"2024-03-06T07:22:00.299505+0000","last_peered":"2024-03-06T07:22:00.299505+0000","last_clean":"2024-03-06T07:22:00.299505+0000","last_became_active":"2024-03-06T07:21:58.953858+0000","last_became_peered":"2024-03-06T07:21:58.953858+0000","last_unstale":"2024-03-06T07:22:00.299505+0000","last_undegraded":"2024-03-06T07:22:00.299505+0000","last_fullsized":"2024-03-06T07:22:00.299505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:12.378982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942428+0000","last_change":"2024-03-06T07:21:57.937765+0000","last_active":"2024-03-06T07:21:59.942428+0000","last_peered":"2024-03-06T07:21:59.942428+0000","last_clean":"2024-03-06T07:21:59.942428+0000","last_became_active":"2024-03-06T07:21:57.937476+0000","last_became_peered":"2024-03-06T07:21:57.937476+0000","last_unstale":"2024-03-06T07:21:59.942428+0000","last_undegraded":"2024-03-06T07:21:59.942428+0000","last_fullsized":"2024-03-06T07:21:59.942428+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:07:58.548928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299878+0000","last_change":"2024-03-06T07:21:57.943799+0000","last_active":"2024-03-06T07:22:00.299878+0000","last_peered":"2024-03-06T07:22:00.299878+0000","last_clean":"2024-03-06T07:22:00.299878+0000","last_became_active":"2024-03-06T07:21:57.943507+0000","last_became_peered":"2024-03-06T07:21:57.943507+0000","last_unstale":"2024-03-06T07:22:00.299878+0000","last_undegraded":"2024-03-06T07:22:00.299878+0000","last_fullsized":"2024-03-06T07:22:00.299878+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:31.179338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298759+0000","last_change":"2024-03-06T07:21:58.965398+0000","last_active":"2024-03-06T07:22:00.298759+0000","last_peered":"2024-03-06T07:22:00.298759+0000","last_clean":"2024-03-06T07:22:00.298759+0000","last_became_active":"2024-03-06T07:21:58.964367+0000","last_became_peered":"2024-03-06T07:21:58.964367+0000","last_unstale":"2024-03-06T07:22:00.298759+0000","last_undegraded":"2024-03-06T07:22:00.298759+0000","last_fullsized":"2024-03-06T07:22:00.298759+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:54.495097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298098+0000","last_change":"2024-03-06T07:21:57.951473+0000","last_active":"2024-03-06T07:22:00.298098+0000","last_peered":"2024-03-06T07:22:00.298098+0000","last_clean":"2024-03-06T07:22:00.298098+0000","last_became_active":"2024-03-06T07:21:57.951208+0000","last_became_peered":"2024-03-06T07:21:57.951208+0000","last_unstale":"2024-03-06T07:22:00.298098+0000","last_undegraded":"2024-03-06T07:22:00.298098+0000","last_fullsized":"2024-03-06T07:22:00.298098+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:51.781285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943705+0000","last_change":"2024-03-06T07:21:58.961873+0000","last_active":"2024-03-06T07:21:59.943705+0000","last_peered":"2024-03-06T07:21:59.943705+0000","last_clean":"2024-03-06T07:21:59.943705+0000","last_became_active":"2024-03-06T07:21:58.961630+0000","last_became_peered":"2024-03-06T07:21:58.961630+0000","last_unstale":"2024-03-06T07:21:59.943705+0000","last_undegraded":"2024-03-06T07:21:59.943705+0000","last_fullsized":"2024-03-06T07:21:59.943705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:55.294652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942284+0000","last_change":"2024-03-06T07:21:57.948765+0000","last_active":"2024-03-06T07:21:59.942284+0000","last_peered":"2024-03-06T07:21:59.942284+0000","last_clean":"2024-03-06T07:21:59.942284+0000","last_became_active":"2024-03-06T07:21:57.948498+0000","last_became_peered":"2024-03-06T07:21:57.948498+0000","last_unstale":"2024-03-06T07:21:59.942284+0000","last_undegraded":"2024-03-06T07:21:59.942284+0000","last_fullsized":"2024-03-06T07:21:59.942284+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:51.781285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298709+0000","last_change":"2024-03-06T07:21:58.971817+0000","last_active":"2024-03-06T07:22:00.298709+0000","last_peered":"2024-03-06T07:22:00.298709+0000","last_clean":"2024-03-06T07:22:00.298709+0000","last_became_active":"2024-03-06T07:21:58.971481+0000","last_became_peered":"2024-03-06T07:21:58.971481+0000","last_unstale":"2024-03-06T07:22:00.298709+0000","last_undegraded":"2024-03-06T07:22:00.298709+0000","last_fullsized":"2024-03-06T07:22:00.298709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:13.580324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298587+0000","last_change":"2024-03-06T07:21:57.950501+0000","last_active":"2024-03-06T07:22:00.298587+0000","last_peered":"2024-03-06T07:22:00.298587+0000","last_clean":"2024-03-06T07:22:00.298587+0000","last_became_active":"2024-03-06T07:21:57.950201+0000","last_became_peered":"2024-03-06T07:21:57.950201+0000","last_unstale":"2024-03-06T07:22:00.298587+0000","last_undegraded":"2024-03-06T07:22:00.298587+0000","last_fullsized":"2024-03-06T07:22:00.298587+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:13.736718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298791+0000","last_change":"2024-03-06T07:21:58.958089+0000","last_active":"2024-03-06T07:22:00.298791+0000","last_peered":"2024-03-06T07:22:00.298791+0000","last_clean":"2024-03-06T07:22:00.298791+0000","last_became_active":"2024-03-06T07:21:58.957814+0000","last_became_peered":"2024-03-06T07:21:58.957814+0000","last_unstale":"2024-03-06T07:22:00.298791+0000","last_undegraded":"2024-03-06T07:22:00.298791+0000","last_fullsized":"2024-03-06T07:22:00.298791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:55.294652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943941+0000","last_change":"2024-03-06T07:21:57.946124+0000","last_active":"2024-03-06T07:21:59.943941+0000","last_peered":"2024-03-06T07:21:59.943941+0000","last_clean":"2024-03-06T07:21:59.943941+0000","last_became_active":"2024-03-06T07:21:57.945856+0000","last_became_peered":"2024-03-06T07:21:57.945856+0000","last_unstale":"2024-03-06T07:21:59.943941+0000","last_undegraded":"2024-03-06T07:21:59.943941+0000","last_fullsized":"2024-03-06T07:21:59.943941+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:01.159166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943172+0000","last_change":"2024-03-06T07:21:58.958734+0000","last_active":"2024-03-06T07:21:59.943172+0000","last_peered":"2024-03-06T07:21:59.943172+0000","last_clean":"2024-03-06T07:21:59.943172+0000","last_became_active":"2024-03-06T07:21:58.958451+0000","last_became_peered":"2024-03-06T07:21:58.958451+0000","last_unstale":"2024-03-06T07:21:59.943172+0000","last_undegraded":"2024-03-06T07:21:59.943172+0000","last_fullsized":"2024-03-06T07:21:59.943172+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:20.908786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942962+0000","last_change":"2024-03-06T07:21:57.947199+0000","last_active":"2024-03-06T07:21:59.942962+0000","last_peered":"2024-03-06T07:21:59.942962+0000","last_clean":"2024-03-06T07:21:59.942962+0000","last_became_active":"2024-03-06T07:21:57.946927+0000","last_became_peered":"2024-03-06T07:21:57.946927+0000","last_unstale":"2024-03-06T07:21:59.942962+0000","last_undegraded":"2024-03-06T07:21:59.942962+0000","last_fullsized":"2024-03-06T07:21:59.942962+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:01.159166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298650+0000","last_change":"2024-03-06T07:21:58.970850+0000","last_active":"2024-03-06T07:22:00.298650+0000","last_peered":"2024-03-06T07:22:00.298650+0000","last_clean":"2024-03-06T07:22:00.298650+0000","last_became_active":"2024-03-06T07:21:58.970512+0000","last_became_peered":"2024-03-06T07:21:58.970512+0000","last_unstale":"2024-03-06T07:22:00.298650+0000","last_undegraded":"2024-03-06T07:22:00.298650+0000","last_fullsized":"2024-03-06T07:22:00.298650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:02:11.315323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299603+0000","last_change":"2024-03-06T07:21:57.951893+0000","last_active":"2024-03-06T07:22:00.299603+0000","last_peered":"2024-03-06T07:22:00.299603+0000","last_clean":"2024-03-06T07:22:00.299603+0000","last_became_active":"2024-03-06T07:21:57.951696+0000","last_became_peered":"2024-03-06T07:21:57.951696+0000","last_unstale":"2024-03-06T07:22:00.299603+0000","last_undegraded":"2024-03-06T07:22:00.299603+0000","last_fullsized":"2024-03-06T07:22:00.299603+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:27:31.864292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298310+0000","last_change":"2024-03-06T07:21:58.954594+0000","last_active":"2024-03-06T07:22:00.298310+0000","last_peered":"2024-03-06T07:22:00.298310+0000","last_clean":"2024-03-06T07:22:00.298310+0000","last_became_active":"2024-03-06T07:21:58.954326+0000","last_became_peered":"2024-03-06T07:21:58.954326+0000","last_unstale":"2024-03-06T07:22:00.298310+0000","last_undegraded":"2024-03-06T07:22:00.298310+0000","last_fullsized":"2024-03-06T07:22:00.298310+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:06:11.334051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298716+0000","last_change":"2024-03-06T07:21:57.956339+0000","last_active":"2024-03-06T07:22:00.298716+0000","last_peered":"2024-03-06T07:22:00.298716+0000","last_clean":"2024-03-06T07:22:00.298716+0000","last_became_active":"2024-03-06T07:21:57.956135+0000","last_became_peered":"2024-03-06T07:21:57.956135+0000","last_unstale":"2024-03-06T07:22:00.298716+0000","last_undegraded":"2024-03-06T07:22:00.298716+0000","last_fullsized":"2024-03-06T07:22:00.298716+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:01.159166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299011+0000","last_change":"2024-03-06T07:21:58.962391+0000","last_active":"2024-03-06T07:22:00.299011+0000","last_peered":"2024-03-06T07:22:00.299011+0000","last_clean":"2024-03-06T07:22:00.299011+0000","last_became_active":"2024-03-06T07:21:58.961887+0000","last_became_peered":"2024-03-06T07:21:58.961887+0000","last_unstale":"2024-03-06T07:22:00.299011+0000","last_undegraded":"2024-03-06T07:22:00.299011+0000","last_fullsized":"2024-03-06T07:22:00.299011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:20.758138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942138+0000","last_change":"2024-03-06T07:21:57.939474+0000","last_active":"2024-03-06T07:21:59.942138+0000","last_peered":"2024-03-06T07:21:59.942138+0000","last_clean":"2024-03-06T07:21:59.942138+0000","last_became_active":"2024-03-06T07:21:57.939053+0000","last_became_peered":"2024-03-06T07:21:57.939053+0000","last_unstale":"2024-03-06T07:21:59.942138+0000","last_undegraded":"2024-03-06T07:21:59.942138+0000","last_fullsized":"2024-03-06T07:21:59.942138+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:35.501272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942064+0000","last_change":"2024-03-06T07:21:58.958835+0000","last_active":"2024-03-06T07:21:59.942064+0000","last_peered":"2024-03-06T07:21:59.942064+0000","last_clean":"2024-03-06T07:21:59.942064+0000","last_became_active":"2024-03-06T07:21:58.958518+0000","last_became_peered":"2024-03-06T07:21:58.958518+0000","last_unstale":"2024-03-06T07:21:59.942064+0000","last_undegraded":"2024-03-06T07:21:59.942064+0000","last_fullsized":"2024-03-06T07:21:59.942064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:22.038204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298611+0000","last_change":"2024-03-06T07:21:57.940312+0000","last_active":"2024-03-06T07:22:00.298611+0000","last_peered":"2024-03-06T07:22:00.298611+0000","last_clean":"2024-03-06T07:22:00.298611+0000","last_became_active":"2024-03-06T07:21:57.940030+0000","last_became_peered":"2024-03-06T07:21:57.940030+0000","last_unstale":"2024-03-06T07:22:00.298611+0000","last_undegraded":"2024-03-06T07:22:00.298611+0000","last_fullsized":"2024-03-06T07:22:00.298611+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:35.501272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298350+0000","last_change":"2024-03-06T07:21:58.961737+0000","last_active":"2024-03-06T07:22:00.298350+0000","last_peered":"2024-03-06T07:22:00.298350+0000","last_clean":"2024-03-06T07:22:00.298350+0000","last_became_active":"2024-03-06T07:21:58.961365+0000","last_became_peered":"2024-03-06T07:21:58.961365+0000","last_unstale":"2024-03-06T07:22:00.298350+0000","last_undegraded":"2024-03-06T07:22:00.298350+0000","last_fullsized":"2024-03-06T07:22:00.298350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:44.889543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943032+0000","last_change":"2024-03-06T07:21:57.948442+0000","last_active":"2024-03-06T07:21:59.943032+0000","last_peered":"2024-03-06T07:21:59.943032+0000","last_clean":"2024-03-06T07:21:59.943032+0000","last_became_active":"2024-03-06T07:21:57.948170+0000","last_became_peered":"2024-03-06T07:21:57.948170+0000","last_unstale":"2024-03-06T07:21:59.943032+0000","last_undegraded":"2024-03-06T07:21:59.943032+0000","last_fullsized":"2024-03-06T07:21:59.943032+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:10.622090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298979+0000","last_change":"2024-03-06T07:21:58.961595+0000","last_active":"2024-03-06T07:22:00.298979+0000","last_peered":"2024-03-06T07:22:00.298979+0000","last_clean":"2024-03-06T07:22:00.298979+0000","last_became_active":"2024-03-06T07:21:58.961259+0000","last_became_peered":"2024-03-06T07:21:58.961259+0000","last_unstale":"2024-03-06T07:22:00.298979+0000","last_undegraded":"2024-03-06T07:22:00.298979+0000","last_fullsized":"2024-03-06T07:22:00.298979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:43.260641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298869+0000","last_change":"2024-03-06T07:21:57.942300+0000","last_active":"2024-03-06T07:22:00.298869+0000","last_peered":"2024-03-06T07:22:00.298869+0000","last_clean":"2024-03-06T07:22:00.298869+0000","last_became_active":"2024-03-06T07:21:57.942023+0000","last_became_peered":"2024-03-06T07:21:57.942023+0000","last_unstale":"2024-03-06T07:22:00.298869+0000","last_undegraded":"2024-03-06T07:22:00.298869+0000","last_fullsized":"2024-03-06T07:22:00.298869+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:07:58.548928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299225+0000","last_change":"2024-03-06T07:21:58.970895+0000","last_active":"2024-03-06T07:22:00.299225+0000","last_peered":"2024-03-06T07:22:00.299225+0000","last_clean":"2024-03-06T07:22:00.299225+0000","last_became_active":"2024-03-06T07:21:58.970418+0000","last_became_peered":"2024-03-06T07:21:58.970418+0000","last_unstale":"2024-03-06T07:22:00.299225+0000","last_undegraded":"2024-03-06T07:22:00.299225+0000","last_fullsized":"2024-03-06T07:22:00.299225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:19:55.013458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944014+0000","last_change":"2024-03-06T07:21:57.942325+0000","last_active":"2024-03-06T07:21:59.944014+0000","last_peered":"2024-03-06T07:21:59.944014+0000","last_clean":"2024-03-06T07:21:59.944014+0000","last_became_active":"2024-03-06T07:21:57.942045+0000","last_became_peered":"2024-03-06T07:21:57.942045+0000","last_unstale":"2024-03-06T07:21:59.944014+0000","last_undegraded":"2024-03-06T07:21:59.944014+0000","last_fullsized":"2024-03-06T07:21:59.944014+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:19.905010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298465+0000","last_change":"2024-03-06T07:21:58.967944+0000","last_active":"2024-03-06T07:22:00.298465+0000","last_peered":"2024-03-06T07:22:00.298465+0000","last_clean":"2024-03-06T07:22:00.298465+0000","last_became_active":"2024-03-06T07:21:58.967521+0000","last_became_peered":"2024-03-06T07:21:58.967521+0000","last_unstale":"2024-03-06T07:22:00.298465+0000","last_undegraded":"2024-03-06T07:22:00.298465+0000","last_fullsized":"2024-03-06T07:22:00.298465+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:43.260641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.297951+0000","last_change":"2024-03-06T07:21:57.947477+0000","last_active":"2024-03-06T07:22:00.297951+0000","last_peered":"2024-03-06T07:22:00.297951+0000","last_clean":"2024-03-06T07:22:00.297951+0000","last_became_active":"2024-03-06T07:21:57.946678+0000","last_became_peered":"2024-03-06T07:21:57.946678+0000","last_unstale":"2024-03-06T07:22:00.297951+0000","last_undegraded":"2024-03-06T07:22:00.297951+0000","last_fullsized":"2024-03-06T07:22:00.297951+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:13.736718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298718+0000","last_change":"2024-03-06T07:21:58.970473+0000","last_active":"2024-03-06T07:22:00.298718+0000","last_peered":"2024-03-06T07:22:00.298718+0000","last_clean":"2024-03-06T07:22:00.298718+0000","last_became_active":"2024-03-06T07:21:58.969810+0000","last_became_peered":"2024-03-06T07:21:58.969810+0000","last_unstale":"2024-03-06T07:22:00.298718+0000","last_undegraded":"2024-03-06T07:22:00.298718+0000","last_fullsized":"2024-03-06T07:22:00.298718+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:07.060045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298038+0000","last_change":"2024-03-06T07:21:47.553693+0000","last_active":"2024-03-06T07:22:00.298038+0000","last_peered":"2024-03-06T07:22:00.298038+0000","last_clean":"2024-03-06T07:22:00.298038+0000","last_became_active":"2024-03-06T07:21:47.552664+0000","last_became_peered":"2024-03-06T07:21:47.552664+0000","last_unstale":"2024-03-06T07:22:00.298038+0000","last_undegraded":"2024-03-06T07:22:00.298038+0000","last_fullsized":"2024-03-06T07:22:00.298038+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_clean_scrub_stamp":"2024-03-06T07:21:46.362028+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:02.916426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.297955+0000","last_change":"2024-03-06T07:21:57.949622+0000","last_active":"2024-03-06T07:22:00.297955+0000","last_peered":"2024-03-06T07:22:00.297955+0000","last_clean":"2024-03-06T07:22:00.297955+0000","last_became_active":"2024-03-06T07:21:57.948874+0000","last_became_peered":"2024-03-06T07:21:57.948874+0000","last_unstale":"2024-03-06T07:22:00.297955+0000","last_undegraded":"2024-03-06T07:22:00.297955+0000","last_fullsized":"2024-03-06T07:22:00.297955+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:35.501272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298026+0000","last_change":"2024-03-06T07:21:58.968667+0000","last_active":"2024-03-06T07:22:00.298026+0000","last_peered":"2024-03-06T07:22:00.298026+0000","last_clean":"2024-03-06T07:22:00.298026+0000","last_became_active":"2024-03-06T07:21:58.968213+0000","last_became_peered":"2024-03-06T07:21:58.968213+0000","last_unstale":"2024-03-06T07:22:00.298026+0000","last_undegraded":"2024-03-06T07:22:00.298026+0000","last_fullsized":"2024-03-06T07:22:00.298026+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:22.038204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.297808+0000","last_change":"2024-03-06T07:21:57.951697+0000","last_active":"2024-03-06T07:22:00.297808+0000","last_peered":"2024-03-06T07:22:00.297808+0000","last_clean":"2024-03-06T07:22:00.297808+0000","last_became_active":"2024-03-06T07:21:57.951421+0000","last_became_peered":"2024-03-06T07:21:57.951421+0000","last_unstale":"2024-03-06T07:22:00.297808+0000","last_undegraded":"2024-03-06T07:22:00.297808+0000","last_fullsized":"2024-03-06T07:22:00.297808+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:51.781285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.297721+0000","last_change":"2024-03-06T07:21:58.948562+0000","last_active":"2024-03-06T07:22:00.297721+0000","last_peered":"2024-03-06T07:22:00.297721+0000","last_clean":"2024-03-06T07:22:00.297721+0000","last_became_active":"2024-03-06T07:21:58.948249+0000","last_became_peered":"2024-03-06T07:21:58.948249+0000","last_unstale":"2024-03-06T07:22:00.297721+0000","last_undegraded":"2024-03-06T07:22:00.297721+0000","last_fullsized":"2024-03-06T07:22:00.297721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:53.360751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298409+0000","last_change":"2024-03-06T07:21:57.945890+0000","last_active":"2024-03-06T07:22:00.298409+0000","last_peered":"2024-03-06T07:22:00.298409+0000","last_clean":"2024-03-06T07:22:00.298409+0000","last_became_active":"2024-03-06T07:21:57.945527+0000","last_became_peered":"2024-03-06T07:21:57.945527+0000","last_unstale":"2024-03-06T07:22:00.298409+0000","last_undegraded":"2024-03-06T07:22:00.298409+0000","last_fullsized":"2024-03-06T07:22:00.298409+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:31.179338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943753+0000","last_change":"2024-03-06T07:21:58.952436+0000","last_active":"2024-03-06T07:21:59.943753+0000","last_peered":"2024-03-06T07:21:59.943753+0000","last_clean":"2024-03-06T07:21:59.943753+0000","last_became_active":"2024-03-06T07:21:58.952135+0000","last_became_peered":"2024-03-06T07:21:58.952135+0000","last_unstale":"2024-03-06T07:21:59.943753+0000","last_undegraded":"2024-03-06T07:21:59.943753+0000","last_fullsized":"2024-03-06T07:21:59.943753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:59.261197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298778+0000","last_change":"2024-03-06T07:21:57.951597+0000","last_active":"2024-03-06T07:22:00.298778+0000","last_peered":"2024-03-06T07:22:00.298778+0000","last_clean":"2024-03-06T07:22:00.298778+0000","last_became_active":"2024-03-06T07:21:57.951299+0000","last_became_peered":"2024-03-06T07:21:57.951299+0000","last_unstale":"2024-03-06T07:22:00.298778+0000","last_undegraded":"2024-03-06T07:22:00.298778+0000","last_fullsized":"2024-03-06T07:22:00.298778+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:19.847331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943217+0000","last_change":"2024-03-06T07:21:58.951633+0000","last_active":"2024-03-06T07:21:59.943217+0000","last_peered":"2024-03-06T07:21:59.943217+0000","last_clean":"2024-03-06T07:21:59.943217+0000","last_became_active":"2024-03-06T07:21:58.951315+0000","last_became_peered":"2024-03-06T07:21:58.951315+0000","last_unstale":"2024-03-06T07:21:59.943217+0000","last_undegraded":"2024-03-06T07:21:59.943217+0000","last_fullsized":"2024-03-06T07:21:59.943217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:20.851108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943826+0000","last_change":"2024-03-06T07:21:57.945253+0000","last_active":"2024-03-06T07:21:59.943826+0000","last_peered":"2024-03-06T07:21:59.943826+0000","last_clean":"2024-03-06T07:21:59.943826+0000","last_became_active":"2024-03-06T07:21:57.944983+0000","last_became_peered":"2024-03-06T07:21:57.944983+0000","last_unstale":"2024-03-06T07:21:59.943826+0000","last_undegraded":"2024-03-06T07:21:59.943826+0000","last_fullsized":"2024-03-06T07:21:59.943826+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:07:58.548928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298511+0000","last_change":"2024-03-06T07:21:58.964580+0000","last_active":"2024-03-06T07:22:00.298511+0000","last_peered":"2024-03-06T07:22:00.298511+0000","last_clean":"2024-03-06T07:22:00.298511+0000","last_became_active":"2024-03-06T07:21:58.964272+0000","last_became_peered":"2024-03-06T07:21:58.964272+0000","last_unstale":"2024-03-06T07:22:00.298511+0000","last_undegraded":"2024-03-06T07:22:00.298511+0000","last_fullsized":"2024-03-06T07:22:00.298511+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:02:11.315323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298902+0000","last_change":"2024-03-06T07:21:57.952261+0000","last_active":"2024-03-06T07:22:00.298902+0000","last_peered":"2024-03-06T07:22:00.298902+0000","last_clean":"2024-03-06T07:22:00.298902+0000","last_became_active":"2024-03-06T07:21:57.951991+0000","last_became_peered":"2024-03-06T07:21:57.951991+0000","last_unstale":"2024-03-06T07:22:00.298902+0000","last_undegraded":"2024-03-06T07:22:00.298902+0000","last_fullsized":"2024-03-06T07:22:00.298902+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:52.356974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943810+0000","last_change":"2024-03-06T07:21:58.955953+0000","last_active":"2024-03-06T07:21:59.943810+0000","last_peered":"2024-03-06T07:21:59.943810+0000","last_clean":"2024-03-06T07:21:59.943810+0000","last_became_active":"2024-03-06T07:21:58.955634+0000","last_became_peered":"2024-03-06T07:21:58.955634+0000","last_unstale":"2024-03-06T07:21:59.943810+0000","last_undegraded":"2024-03-06T07:21:59.943810+0000","last_fullsized":"2024-03-06T07:21:59.943810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:12.378982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943960+0000","last_change":"2024-03-06T07:21:57.943165+0000","last_active":"2024-03-06T07:21:59.943960+0000","last_peered":"2024-03-06T07:21:59.943960+0000","last_clean":"2024-03-06T07:21:59.943960+0000","last_became_active":"2024-03-06T07:21:57.942914+0000","last_became_peered":"2024-03-06T07:21:57.942914+0000","last_unstale":"2024-03-06T07:21:59.943960+0000","last_undegraded":"2024-03-06T07:21:59.943960+0000","last_fullsized":"2024-03-06T07:21:59.943960+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:19.847331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298017+0000","last_change":"2024-03-06T07:21:58.960410+0000","last_active":"2024-03-06T07:22:00.298017+0000","last_peered":"2024-03-06T07:22:00.298017+0000","last_clean":"2024-03-06T07:22:00.298017+0000","last_became_active":"2024-03-06T07:21:58.960121+0000","last_became_peered":"2024-03-06T07:21:58.960121+0000","last_unstale":"2024-03-06T07:22:00.298017+0000","last_undegraded":"2024-03-06T07:22:00.298017+0000","last_fullsized":"2024-03-06T07:22:00.298017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:55.294652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942389+0000","last_change":"2024-03-06T07:21:57.944106+0000","last_active":"2024-03-06T07:21:59.942389+0000","last_peered":"2024-03-06T07:21:59.942389+0000","last_clean":"2024-03-06T07:21:59.942389+0000","last_became_active":"2024-03-06T07:21:57.943815+0000","last_became_peered":"2024-03-06T07:21:57.943815+0000","last_unstale":"2024-03-06T07:21:59.942389+0000","last_undegraded":"2024-03-06T07:21:59.942389+0000","last_fullsized":"2024-03-06T07:21:59.942389+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:13.736718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298012+0000","last_change":"2024-03-06T07:21:58.951840+0000","last_active":"2024-03-06T07:22:00.298012+0000","last_peered":"2024-03-06T07:22:00.298012+0000","last_clean":"2024-03-06T07:22:00.298012+0000","last_became_active":"2024-03-06T07:21:58.951594+0000","last_became_peered":"2024-03-06T07:21:58.951594+0000","last_unstale":"2024-03-06T07:22:00.298012+0000","last_undegraded":"2024-03-06T07:22:00.298012+0000","last_fullsized":"2024-03-06T07:22:00.298012+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:34.528149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.297795+0000","last_change":"2024-03-06T07:21:57.951916+0000","last_active":"2024-03-06T07:22:00.297795+0000","last_peered":"2024-03-06T07:22:00.297795+0000","last_clean":"2024-03-06T07:22:00.297795+0000","last_became_active":"2024-03-06T07:21:57.951647+0000","last_became_peered":"2024-03-06T07:21:57.951647+0000","last_unstale":"2024-03-06T07:22:00.297795+0000","last_undegraded":"2024-03-06T07:22:00.297795+0000","last_fullsized":"2024-03-06T07:22:00.297795+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:13.736718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942859+0000","last_change":"2024-03-06T07:21:58.953147+0000","last_active":"2024-03-06T07:21:59.942859+0000","last_peered":"2024-03-06T07:21:59.942859+0000","last_clean":"2024-03-06T07:21:59.942859+0000","last_became_active":"2024-03-06T07:21:58.952862+0000","last_became_peered":"2024-03-06T07:21:58.952862+0000","last_unstale":"2024-03-06T07:21:59.942859+0000","last_undegraded":"2024-03-06T07:21:59.942859+0000","last_fullsized":"2024-03-06T07:21:59.942859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:14.740494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943795+0000","last_change":"2024-03-06T07:21:57.948593+0000","last_active":"2024-03-06T07:21:59.943795+0000","last_peered":"2024-03-06T07:21:59.943795+0000","last_clean":"2024-03-06T07:21:59.943795+0000","last_became_active":"2024-03-06T07:21:57.948302+0000","last_became_peered":"2024-03-06T07:21:57.948302+0000","last_unstale":"2024-03-06T07:21:59.943795+0000","last_undegraded":"2024-03-06T07:21:59.943795+0000","last_fullsized":"2024-03-06T07:21:59.943795+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:13.736718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298923+0000","last_change":"2024-03-06T07:21:58.953065+0000","last_active":"2024-03-06T07:22:00.298923+0000","last_peered":"2024-03-06T07:22:00.298923+0000","last_clean":"2024-03-06T07:22:00.298923+0000","last_became_active":"2024-03-06T07:21:58.952790+0000","last_became_peered":"2024-03-06T07:21:58.952790+0000","last_unstale":"2024-03-06T07:22:00.298923+0000","last_undegraded":"2024-03-06T07:22:00.298923+0000","last_fullsized":"2024-03-06T07:22:00.298923+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:30:54.760925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943752+0000","last_change":"2024-03-06T07:21:57.945134+0000","last_active":"2024-03-06T07:21:59.943752+0000","last_peered":"2024-03-06T07:21:59.943752+0000","last_clean":"2024-03-06T07:21:59.943752+0000","last_became_active":"2024-03-06T07:21:57.944858+0000","last_became_peered":"2024-03-06T07:21:57.944858+0000","last_unstale":"2024-03-06T07:21:59.943752+0000","last_undegraded":"2024-03-06T07:21:59.943752+0000","last_fullsized":"2024-03-06T07:21:59.943752+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:51.781285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299349+0000","last_change":"2024-03-06T07:21:58.964930+0000","last_active":"2024-03-06T07:22:00.299349+0000","last_peered":"2024-03-06T07:22:00.299349+0000","last_clean":"2024-03-06T07:22:00.299349+0000","last_became_active":"2024-03-06T07:21:58.964474+0000","last_became_peered":"2024-03-06T07:21:58.964474+0000","last_unstale":"2024-03-06T07:22:00.299349+0000","last_undegraded":"2024-03-06T07:22:00.299349+0000","last_fullsized":"2024-03-06T07:22:00.299349+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:44.889543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299058+0000","last_change":"2024-03-06T07:21:57.953248+0000","last_active":"2024-03-06T07:22:00.299058+0000","last_peered":"2024-03-06T07:22:00.299058+0000","last_clean":"2024-03-06T07:22:00.299058+0000","last_became_active":"2024-03-06T07:21:57.952969+0000","last_became_peered":"2024-03-06T07:21:57.952969+0000","last_unstale":"2024-03-06T07:22:00.299058+0000","last_undegraded":"2024-03-06T07:22:00.299058+0000","last_fullsized":"2024-03-06T07:22:00.299058+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:19.905010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942240+0000","last_change":"2024-03-06T07:21:58.954261+0000","last_active":"2024-03-06T07:21:59.942240+0000","last_peered":"2024-03-06T07:21:59.942240+0000","last_clean":"2024-03-06T07:21:59.942240+0000","last_became_active":"2024-03-06T07:21:58.953584+0000","last_became_peered":"2024-03-06T07:21:58.953584+0000","last_unstale":"2024-03-06T07:21:59.942240+0000","last_undegraded":"2024-03-06T07:21:59.942240+0000","last_fullsized":"2024-03-06T07:21:59.942240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:59.261197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298322+0000","last_change":"2024-03-06T07:21:58.959725+0000","last_active":"2024-03-06T07:22:00.298322+0000","last_peered":"2024-03-06T07:22:00.298322+0000","last_clean":"2024-03-06T07:22:00.298322+0000","last_became_active":"2024-03-06T07:21:58.959078+0000","last_became_peered":"2024-03-06T07:21:58.959078+0000","last_unstale":"2024-03-06T07:22:00.298322+0000","last_undegraded":"2024-03-06T07:22:00.298322+0000","last_fullsized":"2024-03-06T07:22:00.298322+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:44.889543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299074+0000","last_change":"2024-03-06T07:21:57.951758+0000","last_active":"2024-03-06T07:22:00.299074+0000","last_peered":"2024-03-06T07:22:00.299074+0000","last_clean":"2024-03-06T07:22:00.299074+0000","last_became_active":"2024-03-06T07:21:57.951452+0000","last_became_peered":"2024-03-06T07:21:57.951452+0000","last_unstale":"2024-03-06T07:22:00.299074+0000","last_undegraded":"2024-03-06T07:22:00.299074+0000","last_fullsized":"2024-03-06T07:22:00.299074+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:01.159166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298378+0000","last_change":"2024-03-06T07:21:58.961541+0000","last_active":"2024-03-06T07:22:00.298378+0000","last_peered":"2024-03-06T07:22:00.298378+0000","last_clean":"2024-03-06T07:22:00.298378+0000","last_became_active":"2024-03-06T07:21:58.961271+0000","last_became_peered":"2024-03-06T07:21:58.961271+0000","last_unstale":"2024-03-06T07:22:00.298378+0000","last_undegraded":"2024-03-06T07:22:00.298378+0000","last_fullsized":"2024-03-06T07:22:00.298378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:10:08.320367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298788+0000","last_change":"2024-03-06T07:21:57.947161+0000","last_active":"2024-03-06T07:22:00.298788+0000","last_peered":"2024-03-06T07:22:00.298788+0000","last_clean":"2024-03-06T07:22:00.298788+0000","last_became_active":"2024-03-06T07:21:57.946849+0000","last_became_peered":"2024-03-06T07:21:57.946849+0000","last_unstale":"2024-03-06T07:22:00.298788+0000","last_undegraded":"2024-03-06T07:22:00.298788+0000","last_fullsized":"2024-03-06T07:22:00.298788+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:19.847331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298661+0000","last_change":"2024-03-06T07:21:58.958814+0000","last_active":"2024-03-06T07:22:00.298661+0000","last_peered":"2024-03-06T07:22:00.298661+0000","last_clean":"2024-03-06T07:22:00.298661+0000","last_became_active":"2024-03-06T07:21:58.958550+0000","last_became_peered":"2024-03-06T07:21:58.958550+0000","last_unstale":"2024-03-06T07:22:00.298661+0000","last_undegraded":"2024-03-06T07:22:00.298661+0000","last_fullsized":"2024-03-06T07:22:00.298661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:36:02.122800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298388+0000","last_change":"2024-03-06T07:21:57.949031+0000","last_active":"2024-03-06T07:22:00.298388+0000","last_peered":"2024-03-06T07:22:00.298388+0000","last_clean":"2024-03-06T07:22:00.298388+0000","last_became_active":"2024-03-06T07:21:57.948741+0000","last_became_peered":"2024-03-06T07:21:57.948741+0000","last_unstale":"2024-03-06T07:22:00.298388+0000","last_undegraded":"2024-03-06T07:22:00.298388+0000","last_fullsized":"2024-03-06T07:22:00.298388+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:35.501272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298230+0000","last_change":"2024-03-06T07:21:58.957355+0000","last_active":"2024-03-06T07:22:00.298230+0000","last_peered":"2024-03-06T07:22:00.298230+0000","last_clean":"2024-03-06T07:22:00.298230+0000","last_became_active":"2024-03-06T07:21:58.956672+0000","last_became_peered":"2024-03-06T07:21:58.956672+0000","last_unstale":"2024-03-06T07:22:00.298230+0000","last_undegraded":"2024-03-06T07:22:00.298230+0000","last_fullsized":"2024-03-06T07:22:00.298230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:59.261197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298955+0000","last_change":"2024-03-06T07:21:57.951293+0000","last_active":"2024-03-06T07:22:00.298955+0000","last_peered":"2024-03-06T07:22:00.298955+0000","last_clean":"2024-03-06T07:22:00.298955+0000","last_became_active":"2024-03-06T07:21:57.950995+0000","last_became_peered":"2024-03-06T07:21:57.950995+0000","last_unstale":"2024-03-06T07:22:00.298955+0000","last_undegraded":"2024-03-06T07:22:00.298955+0000","last_fullsized":"2024-03-06T07:22:00.298955+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:10.622090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27792,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":790528,"data_stored":655584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":66246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":196608,"data_stored":64724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":64760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":66246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":192512,"data_stored":64634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":200704,"data_stored":64794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":798720,"data_stored":655640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:02.663 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:22:02.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.796+0000 7fbe6cf20640 1 Processor -- start 2024-03-06T07:22:02.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.796+0000 7fbe6cf20640 1 -- start start 2024-03-06T07:22:02.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.797+0000 7fbe6cf20640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fbe680b4460 0x7fbe680d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.797+0000 7fbe6cf20640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbe68162970 0x7fbe680b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.797+0000 7fbe6cf20640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbe68161cf0 0x7fbe68162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.797+0000 7fbe6cf20640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe68161b00 con 0x7fbe68162970 2024-03-06T07:22:02.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.797+0000 7fbe6cf20640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe680cecc0 con 0x7fbe680b4460 2024-03-06T07:22:02.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.797+0000 7fbe6cf20640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe680d22c0 con 0x7fbe68161cf0 2024-03-06T07:22:02.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.797+0000 7fbe65d9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbe68162970 0x7fbe680b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.798+0000 7fbe65d9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbe68162970 0x7fbe680b3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33990/0 (socket says 172.21.15.83:33990) 2024-03-06T07:22:02.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.798+0000 7fbe65d9b640 1 -- 172.21.15.83:0/193774086 learned_addr learned my addr 172.21.15.83:0/193774086 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:02.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.798+0000 7fbe66d9d640 1 --2- 172.21.15.83:0/193774086 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbe68161cf0 0x7fbe68162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.798+0000 7fbe6659c640 1 --2- 172.21.15.83:0/193774086 >> v2:172.21.15.183:3300/0 conn(0x7fbe680b4460 0x7fbe680d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.798+0000 7fbe65d9b640 1 -- 172.21.15.83:0/193774086 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbe68161cf0 msgr2=0x7fbe68162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:02.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.798+0000 7fbe65d9b640 1 --2- 172.21.15.83:0/193774086 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbe68161cf0 0x7fbe68162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.798+0000 7fbe65d9b640 1 -- 172.21.15.83:0/193774086 >> v2:172.21.15.183:3300/0 conn(0x7fbe680b4460 msgr2=0x7fbe680d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:02.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.798+0000 7fbe65d9b640 1 --2- 172.21.15.83:0/193774086 >> v2:172.21.15.183:3300/0 conn(0x7fbe680b4460 0x7fbe680d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.798+0000 7fbe65d9b640 1 -- 172.21.15.83:0/193774086 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe680d25a0 con 0x7fbe68162970 2024-03-06T07:22:02.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.798+0000 7fbe65d9b640 1 --2- 172.21.15.83:0/193774086 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe68162970 0x7fbe680b3580 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbe6808ddd0 tx=0x7fbe5c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56ffc76288f869cf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.799+0000 7fbe6559a640 1 -- 172.21.15.83:0/193774086 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c035020 con 0x7fbe68162970 2024-03-06T07:22:02.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.799+0000 7fbe6559a640 1 -- 172.21.15.83:0/193774086 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe5c031870 con 0x7fbe68162970 2024-03-06T07:22:02.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.799+0000 7fbe6559a640 1 -- 172.21.15.83:0/193774086 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c031c40 con 0x7fbe68162970 2024-03-06T07:22:02.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.799+0000 7fbe6cf20640 1 -- 172.21.15.83:0/193774086 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe68162970 msgr2=0x7fbe680b3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.799+0000 7fbe6cf20640 1 --2- 172.21.15.83:0/193774086 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe68162970 0x7fbe680b3580 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbe6808ddd0 tx=0x7fbe5c02b910 comp rx=0 tx=0).stop 2024-03-06T07:22:02.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.799+0000 7fbe6cf20640 1 -- 172.21.15.83:0/193774086 shutdown_connections 2024-03-06T07:22:02.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.799+0000 7fbe6cf20640 1 --2- 172.21.15.83:0/193774086 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbe68161cf0 0x7fbe68162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.799+0000 7fbe6cf20640 1 --2- 172.21.15.83:0/193774086 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe68162970 0x7fbe680b3580 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.799+0000 7fbe6cf20640 1 --2- 172.21.15.83:0/193774086 >> v2:172.21.15.183:3300/0 conn(0x7fbe680b4460 0x7fbe680d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.799+0000 7fbe6cf20640 1 -- 172.21.15.83:0/193774086 >> 172.21.15.83:0/193774086 conn(0x7fbe680d3ce0 msgr2=0x7fbe680d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.799+0000 7fbe6cf20640 1 -- 172.21.15.83:0/193774086 shutdown_connections 2024-03-06T07:22:02.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.799+0000 7fbe6cf20640 1 -- 172.21.15.83:0/193774086 wait complete. 2024-03-06T07:22:02.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.801+0000 7fbe6cf20640 1 Processor -- start 2024-03-06T07:22:02.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.801+0000 7fbe6cf20640 1 -- start start 2024-03-06T07:22:02.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.801+0000 7fbe6cf20640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbe680b4460 0x7fbe680ca070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.801+0000 7fbe6cf20640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe68161cf0 0x7fbe680c91e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.801+0000 7fbe6cf20640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe68162970 0x7fbe680b8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.801+0000 7fbe6cf20640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe680ce950 con 0x7fbe68161cf0 2024-03-06T07:22:02.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.801+0000 7fbe6cf20640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe680d2b60 con 0x7fbe68162970 2024-03-06T07:22:02.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.801+0000 7fbe6cf20640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe680b3ac0 con 0x7fbe680b4460 2024-03-06T07:22:02.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.801+0000 7fbe65d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe68161cf0 0x7fbe680c91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.801+0000 7fbe65d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe68161cf0 0x7fbe680c91e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33994/0 (socket says 172.21.15.83:33994) 2024-03-06T07:22:02.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.801+0000 7fbe65d9b640 1 -- 172.21.15.83:0/2756691846 learned_addr learned my addr 172.21.15.83:0/2756691846 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:02.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.802+0000 7fbe6659c640 1 --2- 172.21.15.83:0/2756691846 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbe680b4460 0x7fbe680ca070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.802+0000 7fbe66d9d640 1 --2- 172.21.15.83:0/2756691846 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe68162970 0x7fbe680b8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.802+0000 7fbe65d9b640 1 -- 172.21.15.83:0/2756691846 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbe680b4460 msgr2=0x7fbe680ca070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.802+0000 7fbe65d9b640 1 --2- 172.21.15.83:0/2756691846 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbe680b4460 0x7fbe680ca070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.802+0000 7fbe65d9b640 1 -- 172.21.15.83:0/2756691846 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe68162970 msgr2=0x7fbe680b8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.802+0000 7fbe65d9b640 1 --2- 172.21.15.83:0/2756691846 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe68162970 0x7fbe680b8030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.802+0000 7fbe65d9b640 1 -- 172.21.15.83:0/2756691846 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe680b7a10 con 0x7fbe68161cf0 2024-03-06T07:22:02.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.802+0000 7fbe65d9b640 1 --2- 172.21.15.83:0/2756691846 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe68161cf0 0x7fbe680c91e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c02b950 tx=0x7fbe5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.802+0000 7fbe56ffd640 1 -- 172.21.15.83:0/2756691846 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c041070 con 0x7fbe68161cf0 2024-03-06T07:22:02.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.802+0000 7fbe56ffd640 1 -- 172.21.15.83:0/2756691846 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe5c031e10 con 0x7fbe68161cf0 2024-03-06T07:22:02.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.802+0000 7fbe56ffd640 1 -- 172.21.15.83:0/2756691846 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c035040 con 0x7fbe68161cf0 2024-03-06T07:22:02.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.802+0000 7fbe6cf20640 1 -- 172.21.15.83:0/2756691846 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe680ccb30 con 0x7fbe68161cf0 2024-03-06T07:22:02.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.802+0000 7fbe6cf20640 1 -- 172.21.15.83:0/2756691846 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe680ca5b0 con 0x7fbe68161cf0 2024-03-06T07:22:02.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.803+0000 7fbe6cf20640 1 -- 172.21.15.83:0/2756691846 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe6804f330 con 0x7fbe68161cf0 2024-03-06T07:22:02.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.804+0000 7fbe56ffd640 1 -- 172.21.15.83:0/2756691846 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbe5c04c060 con 0x7fbe68161cf0 2024-03-06T07:22:02.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.804+0000 7fbe56ffd640 1 --2- 172.21.15.83:0/2756691846 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbe4806b4d0 0x7fbe4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.805+0000 7fbe6659c640 1 --2- 172.21.15.83:0/2756691846 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbe4806b4d0 0x7fbe4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.805+0000 7fbe56ffd640 1 -- 172.21.15.83:0/2756691846 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbe5c0abd40 con 0x7fbe68161cf0 2024-03-06T07:22:02.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.805+0000 7fbe6659c640 1 --2- 172.21.15.83:0/2756691846 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbe4806b4d0 0x7fbe4806d990 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbe60011340 tx=0x7fbe6000b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:02.809+0000 7fbe56ffd640 1 -- 172.21.15.83:0/2756691846 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbe5c07a570 con 0x7fbe68161cf0 2024-03-06T07:22:03.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.056+0000 7fbe6cf20640 1 -- 172.21.15.83:0/2756691846 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fbe680cdf00 con 0x7fbe68161cf0 2024-03-06T07:22:03.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.056+0000 7fbe56ffd640 1 -- 172.21.15.83:0/2756691846 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1011 (secure 0 0 0) 0x7fbe5c07e610 con 0x7fbe68161cf0 2024-03-06T07:22:03.059 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:22:03.059 INFO:teuthology.orchestra.run.smithi083.stdout:{"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":28,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":20,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709705,"num_in_osds":8,"osd_in_since":1709709696,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":23,"data_bytes":592778,"bytes_used":224145408,"bytes_avail":772869967872,"bytes_total":773094113280,"write_bytes_sec":2657,"read_op_per_sec":0,"write_op_per_sec":8},"fsmap":{"epoch":6,"id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"c","status":"up:active","gid":4287}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-03-06T07:21:34.187896+0000","services":{}},"progress_events":{}} 2024-03-06T07:22:03.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.059+0000 7fbe6cf20640 1 -- 172.21.15.83:0/2756691846 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbe4806b4d0 msgr2=0x7fbe4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.059+0000 7fbe6cf20640 1 --2- 172.21.15.83:0/2756691846 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbe4806b4d0 0x7fbe4806d990 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbe60011340 tx=0x7fbe6000b040 comp rx=0 tx=0).stop 2024-03-06T07:22:03.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.059+0000 7fbe6cf20640 1 -- 172.21.15.83:0/2756691846 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe68161cf0 msgr2=0x7fbe680c91e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.059+0000 7fbe6cf20640 1 --2- 172.21.15.83:0/2756691846 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe68161cf0 0x7fbe680c91e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c02b950 tx=0x7fbe5c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.059+0000 7fbe6cf20640 1 -- 172.21.15.83:0/2756691846 shutdown_connections 2024-03-06T07:22:03.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.059+0000 7fbe6cf20640 1 --2- 172.21.15.83:0/2756691846 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbe4806b4d0 0x7fbe4806d990 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.059+0000 7fbe6cf20640 1 --2- 172.21.15.83:0/2756691846 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe68162970 0x7fbe680b8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.059+0000 7fbe6cf20640 1 --2- 172.21.15.83:0/2756691846 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe68161cf0 0x7fbe680c91e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.059+0000 7fbe6cf20640 1 --2- 172.21.15.83:0/2756691846 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbe680b4460 0x7fbe680ca070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.059+0000 7fbe6cf20640 1 -- 172.21.15.83:0/2756691846 >> 172.21.15.83:0/2756691846 conn(0x7fbe680d3ce0 msgr2=0x7fbe680cf150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:03.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.059+0000 7fbe6cf20640 1 -- 172.21.15.83:0/2756691846 shutdown_connections 2024-03-06T07:22:03.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.059+0000 7fbe6cf20640 1 -- 172.21.15.83:0/2756691846 wait complete. 2024-03-06T07:22:03.072 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:22:03.072 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:03.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.171+0000 7f72f295c640 1 Processor -- start 2024-03-06T07:22:03.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.171+0000 7f72f295c640 1 -- start start 2024-03-06T07:22:03.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.172+0000 7f72f295c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f72ec0cd390 0x7f72ec072700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.172+0000 7f72f295c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72ec0d0610 con 0x7f72ec06b6c0 2024-03-06T07:22:03.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.172+0000 7f72f295c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72ec0d08f0 con 0x7f72ec15baf0 2024-03-06T07:22:03.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.172+0000 7f72f295c640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72ec072c40 con 0x7f72ec0cd390 2024-03-06T07:22:03.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.172+0000 7f72f0ef9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f72ec0cd390 0x7f72ec072700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.172+0000 7f72f0ef9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f72ec0cd390 0x7f72ec072700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34006/0 (socket says 172.21.15.83:34006) 2024-03-06T07:22:03.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.172+0000 7f72eb7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f72ec06b6c0 0x7f72ec0ced90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:40964/0 (socket says 172.21.15.83:40964) 2024-03-06T07:22:03.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.172+0000 7f72f0ef9640 1 -- 172.21.15.83:0/3345728895 learned_addr learned my addr 172.21.15.83:0/3345728895 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:03.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.172+0000 7f72f0ef9640 1 -- 172.21.15.83:0/3345728895 >> v1:172.21.15.183:6789/0 conn(0x7f72ec15baf0 legacy=0x7f72ec06afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.172+0000 7f72f0ef9640 1 -- 172.21.15.83:0/3345728895 >> v1:172.21.15.83:6789/0 conn(0x7f72ec06b6c0 legacy=0x7f72ec0ced90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.172+0000 7f72f0ef9640 1 -- 172.21.15.83:0/3345728895 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ec0d0bd0 con 0x7f72ec0cd390 2024-03-06T07:22:03.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.173+0000 7f72f0ef9640 1 --2- 172.21.15.83:0/3345728895 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72ec0cd390 0x7f72ec072700 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f72e4009e50 tx=0x7f72e402c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33fa7bee33d3970f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.173+0000 7f72eaffd640 1 -- 172.21.15.83:0/3345728895 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e402cbb0 con 0x7f72ec0cd390 2024-03-06T07:22:03.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.173+0000 7f72eaffd640 1 -- 172.21.15.83:0/3345728895 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e402cd50 con 0x7f72ec0cd390 2024-03-06T07:22:03.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.173+0000 7f72eaffd640 1 -- 172.21.15.83:0/3345728895 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e4034610 con 0x7f72ec0cd390 2024-03-06T07:22:03.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.174+0000 7f72f295c640 1 -- 172.21.15.83:0/3345728895 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72ec0cd390 msgr2=0x7f72ec072700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.174+0000 7f72f295c640 1 --2- 172.21.15.83:0/3345728895 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72ec0cd390 0x7f72ec072700 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f72e4009e50 tx=0x7f72e402c010 comp rx=0 tx=0).stop 2024-03-06T07:22:03.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.174+0000 7f72f295c640 1 -- 172.21.15.83:0/3345728895 shutdown_connections 2024-03-06T07:22:03.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.174+0000 7f72f295c640 1 --2- 172.21.15.83:0/3345728895 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72ec0cd390 0x7f72ec072700 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.174+0000 7f72f295c640 1 -- 172.21.15.83:0/3345728895 >> 172.21.15.83:0/3345728895 conn(0x7f72ec06c0c0 msgr2=0x7f72ec06c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:03.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.174+0000 7f72f295c640 1 -- 172.21.15.83:0/3345728895 shutdown_connections 2024-03-06T07:22:03.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.174+0000 7f72f295c640 1 -- 172.21.15.83:0/3345728895 wait complete. 2024-03-06T07:22:03.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.175+0000 7f72f295c640 1 Processor -- start 2024-03-06T07:22:03.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.175+0000 7f72f295c640 1 -- start start 2024-03-06T07:22:03.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.176+0000 7f72f295c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72ec06b6c0 0x7f72ec143070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.176+0000 7f72f295c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f72ec0cd390 0x7f72ec1435b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.176+0000 7f72f295c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f72ec15baf0 0x7f72ec15e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.176+0000 7f72f295c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec15bed0 con 0x7f72ec06b6c0 2024-03-06T07:22:03.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.176+0000 7f72f295c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec0d1190 con 0x7f72ec15baf0 2024-03-06T07:22:03.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.176+0000 7f72f295c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec0d16b0 con 0x7f72ec0cd390 2024-03-06T07:22:03.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.176+0000 7f72ebfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72ec06b6c0 0x7f72ec143070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.176+0000 7f72ebfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72ec06b6c0 0x7f72ec143070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34008/0 (socket says 172.21.15.83:34008) 2024-03-06T07:22:03.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.176+0000 7f72ebfff640 1 -- 172.21.15.83:0/1275653272 learned_addr learned my addr 172.21.15.83:0/1275653272 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:03.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.176+0000 7f72f0ef9640 1 --2- 172.21.15.83:0/1275653272 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f72ec15baf0 0x7f72ec15e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.176+0000 7f72eb7fe640 1 --2- 172.21.15.83:0/1275653272 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f72ec0cd390 0x7f72ec1435b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.176+0000 7f72ebfff640 1 -- 172.21.15.83:0/1275653272 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f72ec0cd390 msgr2=0x7f72ec1435b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.176+0000 7f72ebfff640 1 --2- 172.21.15.83:0/1275653272 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f72ec0cd390 0x7f72ec1435b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.176+0000 7f72ebfff640 1 -- 172.21.15.83:0/1275653272 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f72ec15baf0 msgr2=0x7f72ec15e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.176+0000 7f72ebfff640 1 --2- 172.21.15.83:0/1275653272 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f72ec15baf0 0x7f72ec15e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.176+0000 7f72ebfff640 1 -- 172.21.15.83:0/1275653272 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ec1548a0 con 0x7f72ec06b6c0 2024-03-06T07:22:03.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.176+0000 7f72ebfff640 1 --2- 172.21.15.83:0/1275653272 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72ec06b6c0 0x7f72ec143070 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f72e000c430 tx=0x7f72e000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.176+0000 7f72e8ff9640 1 -- 172.21.15.83:0/1275653272 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e00052d0 con 0x7f72ec06b6c0 2024-03-06T07:22:03.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.176+0000 7f72e8ff9640 1 -- 172.21.15.83:0/1275653272 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e0005490 con 0x7f72ec06b6c0 2024-03-06T07:22:03.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.176+0000 7f72f295c640 1 -- 172.21.15.83:0/1275653272 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72ec154b80 con 0x7f72ec06b6c0 2024-03-06T07:22:03.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.176+0000 7f72e8ff9640 1 -- 172.21.15.83:0/1275653272 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e0007260 con 0x7f72ec06b6c0 2024-03-06T07:22:03.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.177+0000 7f72f295c640 1 -- 172.21.15.83:0/1275653272 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72ec155140 con 0x7f72ec06b6c0 2024-03-06T07:22:03.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.178+0000 7f72f295c640 1 -- 172.21.15.83:0/1275653272 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72ec04f330 con 0x7f72ec06b6c0 2024-03-06T07:22:03.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.178+0000 7f72e8ff9640 1 -- 172.21.15.83:0/1275653272 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f72e0007400 con 0x7f72ec06b6c0 2024-03-06T07:22:03.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.178+0000 7f72e8ff9640 1 --2- 172.21.15.83:0/1275653272 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f72d806b4d0 0x7f72d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.178+0000 7f72e8ff9640 1 -- 172.21.15.83:0/1275653272 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f72e0089680 con 0x7f72ec06b6c0 2024-03-06T07:22:03.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.178+0000 7f72eb7fe640 1 --2- 172.21.15.83:0/1275653272 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f72d806b4d0 0x7f72d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.179+0000 7f72eb7fe640 1 --2- 172.21.15.83:0/1275653272 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f72d806b4d0 0x7f72d806d990 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f72ec144710 tx=0x7f72dc007c80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.183+0000 7f72e8ff9640 1 -- 172.21.15.83:0/1275653272 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f72e0057eb0 con 0x7f72ec06b6c0 2024-03-06T07:22:03.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.356+0000 7f72f295c640 1 -- 172.21.15.83:0/1275653272 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f72ec071f00 con 0x7f72d806b4d0 2024-03-06T07:22:03.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.362+0000 7f72e8ff9640 1 -- 172.21.15.83:0/1275653272 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251946 (secure 0 0 0) 0x7f72ec071f00 con 0x7f72d806b4d0 2024-03-06T07:22:03.366 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:22:03.368 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-06T07:22:03.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.365+0000 7f72f295c640 1 -- 172.21.15.83:0/1275653272 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f72d806b4d0 msgr2=0x7f72d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.365+0000 7f72f295c640 1 --2- 172.21.15.83:0/1275653272 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f72d806b4d0 0x7f72d806d990 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f72ec144710 tx=0x7f72dc007c80 comp rx=0 tx=0).stop 2024-03-06T07:22:03.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.365+0000 7f72f295c640 1 -- 172.21.15.83:0/1275653272 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72ec06b6c0 msgr2=0x7f72ec143070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.365+0000 7f72f295c640 1 --2- 172.21.15.83:0/1275653272 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72ec06b6c0 0x7f72ec143070 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f72e000c430 tx=0x7f72e000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.365+0000 7f72f295c640 1 -- 172.21.15.83:0/1275653272 shutdown_connections 2024-03-06T07:22:03.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.365+0000 7f72f295c640 1 --2- 172.21.15.83:0/1275653272 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f72d806b4d0 0x7f72d806d990 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.365+0000 7f72f295c640 1 --2- 172.21.15.83:0/1275653272 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f72ec15baf0 0x7f72ec15e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.365+0000 7f72f295c640 1 --2- 172.21.15.83:0/1275653272 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f72ec0cd390 0x7f72ec1435b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.365+0000 7f72f295c640 1 --2- 172.21.15.83:0/1275653272 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72ec06b6c0 0x7f72ec143070 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.365+0000 7f72f295c640 1 -- 172.21.15.83:0/1275653272 >> 172.21.15.83:0/1275653272 conn(0x7f72ec06c0c0 msgr2=0x7f72ec1590e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:03.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.365+0000 7f72f295c640 1 -- 172.21.15.83:0/1275653272 shutdown_connections 2024-03-06T07:22:03.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:03.365+0000 7f72f295c640 1 -- 172.21.15.83:0/1275653272 wait complete. 2024-03-06T07:22:03.378 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-03-06T07:22:02.321898+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":167,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218892,"kb_used_data":2728,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755828,"statfs":{"total":773094113280,"available":772869967872,"internally_reserved":0,"allocated":2793472,"data_stored":1702628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12757,"internal_metadata":221236779},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.393612"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299166+0000","last_change":"2024-03-06T07:21:58.961070+0000","last_active":"2024-03-06T07:22:00.299166+0000","last_peered":"2024-03-06T07:22:00.299166+0000","last_clean":"2024-03-06T07:22:00.299166+0000","last_became_active":"2024-03-06T07:21:58.960784+0000","last_became_peered":"2024-03-06T07:21:58.960784+0000","last_unstale":"2024-03-06T07:22:00.299166+0000","last_undegraded":"2024-03-06T07:22:00.299166+0000","last_fullsized":"2024-03-06T07:22:00.299166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:06.413968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943975+0000","last_change":"2024-03-06T07:21:58.958978+0000","last_active":"2024-03-06T07:21:59.943975+0000","last_peered":"2024-03-06T07:21:59.943975+0000","last_clean":"2024-03-06T07:21:59.943975+0000","last_became_active":"2024-03-06T07:21:58.958728+0000","last_became_peered":"2024-03-06T07:21:58.958728+0000","last_unstale":"2024-03-06T07:21:59.943975+0000","last_undegraded":"2024-03-06T07:21:59.943975+0000","last_fullsized":"2024-03-06T07:21:59.943975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:54.495097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298515+0000","last_change":"2024-03-06T07:21:58.953968+0000","last_active":"2024-03-06T07:22:00.298515+0000","last_peered":"2024-03-06T07:22:00.298515+0000","last_clean":"2024-03-06T07:22:00.298515+0000","last_became_active":"2024-03-06T07:21:58.953678+0000","last_became_peered":"2024-03-06T07:21:58.953678+0000","last_unstale":"2024-03-06T07:22:00.298515+0000","last_undegraded":"2024-03-06T07:22:00.298515+0000","last_fullsized":"2024-03-06T07:22:00.298515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:51:39.557766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298069+0000","last_change":"2024-03-06T07:21:58.970682+0000","last_active":"2024-03-06T07:22:00.298069+0000","last_peered":"2024-03-06T07:22:00.298069+0000","last_clean":"2024-03-06T07:22:00.298069+0000","last_became_active":"2024-03-06T07:21:58.970397+0000","last_became_peered":"2024-03-06T07:21:58.970397+0000","last_unstale":"2024-03-06T07:22:00.298069+0000","last_undegraded":"2024-03-06T07:22:00.298069+0000","last_fullsized":"2024-03-06T07:22:00.298069+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:44.732365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298736+0000","last_change":"2024-03-06T07:21:58.965912+0000","last_active":"2024-03-06T07:22:00.298736+0000","last_peered":"2024-03-06T07:22:00.298736+0000","last_clean":"2024-03-06T07:22:00.298736+0000","last_became_active":"2024-03-06T07:21:58.965567+0000","last_became_peered":"2024-03-06T07:21:58.965567+0000","last_unstale":"2024-03-06T07:22:00.298736+0000","last_undegraded":"2024-03-06T07:22:00.298736+0000","last_fullsized":"2024-03-06T07:22:00.298736+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:10:08.320367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298064+0000","last_change":"2024-03-06T07:21:58.963244+0000","last_active":"2024-03-06T07:22:00.298064+0000","last_peered":"2024-03-06T07:22:00.298064+0000","last_clean":"2024-03-06T07:22:00.298064+0000","last_became_active":"2024-03-06T07:21:58.962903+0000","last_became_peered":"2024-03-06T07:21:58.962903+0000","last_unstale":"2024-03-06T07:22:00.298064+0000","last_undegraded":"2024-03-06T07:22:00.298064+0000","last_fullsized":"2024-03-06T07:22:00.298064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:30:54.760925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.297983+0000","last_change":"2024-03-06T07:21:58.964015+0000","last_active":"2024-03-06T07:22:00.297983+0000","last_peered":"2024-03-06T07:22:00.297983+0000","last_clean":"2024-03-06T07:22:00.297983+0000","last_became_active":"2024-03-06T07:21:58.962062+0000","last_became_peered":"2024-03-06T07:21:58.962062+0000","last_unstale":"2024-03-06T07:22:00.297983+0000","last_undegraded":"2024-03-06T07:22:00.297983+0000","last_fullsized":"2024-03-06T07:22:00.297983+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:44.732365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298608+0000","last_change":"2024-03-06T07:21:58.968307+0000","last_active":"2024-03-06T07:22:00.298608+0000","last_peered":"2024-03-06T07:22:00.298608+0000","last_clean":"2024-03-06T07:22:00.298608+0000","last_became_active":"2024-03-06T07:21:58.967917+0000","last_became_peered":"2024-03-06T07:21:58.967917+0000","last_unstale":"2024-03-06T07:22:00.298608+0000","last_undegraded":"2024-03-06T07:22:00.298608+0000","last_fullsized":"2024-03-06T07:22:00.298608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:21:17.637762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299130+0000","last_change":"2024-03-06T07:21:58.971497+0000","last_active":"2024-03-06T07:22:00.299130+0000","last_peered":"2024-03-06T07:22:00.299130+0000","last_clean":"2024-03-06T07:22:00.299130+0000","last_became_active":"2024-03-06T07:21:58.970710+0000","last_became_peered":"2024-03-06T07:21:58.970710+0000","last_unstale":"2024-03-06T07:22:00.299130+0000","last_undegraded":"2024-03-06T07:22:00.299130+0000","last_fullsized":"2024-03-06T07:22:00.299130+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:13.580324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943457+0000","last_change":"2024-03-06T07:21:58.956333+0000","last_active":"2024-03-06T07:21:59.943457+0000","last_peered":"2024-03-06T07:21:59.943457+0000","last_clean":"2024-03-06T07:21:59.943457+0000","last_became_active":"2024-03-06T07:21:58.956073+0000","last_became_peered":"2024-03-06T07:21:58.956073+0000","last_unstale":"2024-03-06T07:21:59.943457+0000","last_undegraded":"2024-03-06T07:21:59.943457+0000","last_fullsized":"2024-03-06T07:21:59.943457+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:30:54.760925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943172+0000","last_change":"2024-03-06T07:21:58.956210+0000","last_active":"2024-03-06T07:21:59.943172+0000","last_peered":"2024-03-06T07:21:59.943172+0000","last_clean":"2024-03-06T07:21:59.943172+0000","last_became_active":"2024-03-06T07:21:58.955949+0000","last_became_peered":"2024-03-06T07:21:58.955949+0000","last_unstale":"2024-03-06T07:21:59.943172+0000","last_undegraded":"2024-03-06T07:21:59.943172+0000","last_fullsized":"2024-03-06T07:21:59.943172+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:54.495097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943018+0000","last_change":"2024-03-06T07:21:58.950931+0000","last_active":"2024-03-06T07:21:59.943018+0000","last_peered":"2024-03-06T07:21:59.943018+0000","last_clean":"2024-03-06T07:21:59.943018+0000","last_became_active":"2024-03-06T07:21:58.949704+0000","last_became_peered":"2024-03-06T07:21:58.949704+0000","last_unstale":"2024-03-06T07:21:59.943018+0000","last_undegraded":"2024-03-06T07:21:59.943018+0000","last_fullsized":"2024-03-06T07:21:59.943018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:02.162943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299186+0000","last_change":"2024-03-06T07:21:58.956175+0000","last_active":"2024-03-06T07:22:00.299186+0000","last_peered":"2024-03-06T07:22:00.299186+0000","last_clean":"2024-03-06T07:22:00.299186+0000","last_became_active":"2024-03-06T07:21:58.955857+0000","last_became_peered":"2024-03-06T07:21:58.955857+0000","last_unstale":"2024-03-06T07:22:00.299186+0000","last_undegraded":"2024-03-06T07:22:00.299186+0000","last_fullsized":"2024-03-06T07:22:00.299186+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:44.732365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298168+0000","last_change":"2024-03-06T07:21:58.951176+0000","last_active":"2024-03-06T07:22:00.298168+0000","last_peered":"2024-03-06T07:22:00.298168+0000","last_clean":"2024-03-06T07:22:00.298168+0000","last_became_active":"2024-03-06T07:21:58.950874+0000","last_became_peered":"2024-03-06T07:21:58.950874+0000","last_unstale":"2024-03-06T07:22:00.298168+0000","last_undegraded":"2024-03-06T07:22:00.298168+0000","last_fullsized":"2024-03-06T07:22:00.298168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:42:14.052025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943706+0000","last_change":"2024-03-06T07:21:58.954562+0000","last_active":"2024-03-06T07:21:59.943706+0000","last_peered":"2024-03-06T07:21:59.943706+0000","last_clean":"2024-03-06T07:21:59.943706+0000","last_became_active":"2024-03-06T07:21:58.954242+0000","last_became_peered":"2024-03-06T07:21:58.954242+0000","last_unstale":"2024-03-06T07:21:59.943706+0000","last_undegraded":"2024-03-06T07:21:59.943706+0000","last_fullsized":"2024-03-06T07:21:59.943706+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:22.038204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.297923+0000","last_change":"2024-03-06T07:21:58.963478+0000","last_active":"2024-03-06T07:22:00.297923+0000","last_peered":"2024-03-06T07:22:00.297923+0000","last_clean":"2024-03-06T07:22:00.297923+0000","last_became_active":"2024-03-06T07:21:58.962917+0000","last_became_peered":"2024-03-06T07:21:58.962917+0000","last_unstale":"2024-03-06T07:22:00.297923+0000","last_undegraded":"2024-03-06T07:22:00.297923+0000","last_fullsized":"2024-03-06T07:22:00.297923+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:54.495097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298958+0000","last_change":"2024-03-06T07:21:58.965994+0000","last_active":"2024-03-06T07:22:00.298958+0000","last_peered":"2024-03-06T07:22:00.298958+0000","last_clean":"2024-03-06T07:22:00.298958+0000","last_became_active":"2024-03-06T07:21:58.965655+0000","last_became_peered":"2024-03-06T07:21:58.965655+0000","last_unstale":"2024-03-06T07:22:00.298958+0000","last_undegraded":"2024-03-06T07:22:00.298958+0000","last_fullsized":"2024-03-06T07:22:00.298958+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:06.413968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298767+0000","last_change":"2024-03-06T07:21:58.955322+0000","last_active":"2024-03-06T07:22:00.298767+0000","last_peered":"2024-03-06T07:22:00.298767+0000","last_clean":"2024-03-06T07:22:00.298767+0000","last_became_active":"2024-03-06T07:21:58.955080+0000","last_became_peered":"2024-03-06T07:21:58.955080+0000","last_unstale":"2024-03-06T07:22:00.298767+0000","last_undegraded":"2024-03-06T07:22:00.298767+0000","last_fullsized":"2024-03-06T07:22:00.298767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:20.851108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298974+0000","last_change":"2024-03-06T07:21:58.966461+0000","last_active":"2024-03-06T07:22:00.298974+0000","last_peered":"2024-03-06T07:22:00.298974+0000","last_clean":"2024-03-06T07:22:00.298974+0000","last_became_active":"2024-03-06T07:21:58.966154+0000","last_became_peered":"2024-03-06T07:21:58.966154+0000","last_unstale":"2024-03-06T07:22:00.298974+0000","last_undegraded":"2024-03-06T07:22:00.298974+0000","last_fullsized":"2024-03-06T07:22:00.298974+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:37:30.965258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942831+0000","last_change":"2024-03-06T07:21:58.953916+0000","last_active":"2024-03-06T07:21:59.942831+0000","last_peered":"2024-03-06T07:21:59.942831+0000","last_clean":"2024-03-06T07:21:59.942831+0000","last_became_active":"2024-03-06T07:21:58.953623+0000","last_became_peered":"2024-03-06T07:21:58.953623+0000","last_unstale":"2024-03-06T07:21:59.942831+0000","last_undegraded":"2024-03-06T07:21:59.942831+0000","last_fullsized":"2024-03-06T07:21:59.942831+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:21:17.637762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299245+0000","last_change":"2024-03-06T07:21:58.955378+0000","last_active":"2024-03-06T07:22:00.299245+0000","last_peered":"2024-03-06T07:22:00.299245+0000","last_clean":"2024-03-06T07:22:00.299245+0000","last_became_active":"2024-03-06T07:21:58.955118+0000","last_became_peered":"2024-03-06T07:21:58.955118+0000","last_unstale":"2024-03-06T07:22:00.299245+0000","last_undegraded":"2024-03-06T07:22:00.299245+0000","last_fullsized":"2024-03-06T07:22:00.299245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:22.038204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298483+0000","last_change":"2024-03-06T07:21:58.963849+0000","last_active":"2024-03-06T07:22:00.298483+0000","last_peered":"2024-03-06T07:22:00.298483+0000","last_clean":"2024-03-06T07:22:00.298483+0000","last_became_active":"2024-03-06T07:21:58.963573+0000","last_became_peered":"2024-03-06T07:21:58.963573+0000","last_unstale":"2024-03-06T07:22:00.298483+0000","last_undegraded":"2024-03-06T07:22:00.298483+0000","last_fullsized":"2024-03-06T07:22:00.298483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:20.758138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299836+0000","last_change":"2024-03-06T07:21:58.962026+0000","last_active":"2024-03-06T07:22:00.299836+0000","last_peered":"2024-03-06T07:22:00.299836+0000","last_clean":"2024-03-06T07:22:00.299836+0000","last_became_active":"2024-03-06T07:21:58.961691+0000","last_became_peered":"2024-03-06T07:21:58.961691+0000","last_unstale":"2024-03-06T07:22:00.299836+0000","last_undegraded":"2024-03-06T07:22:00.299836+0000","last_fullsized":"2024-03-06T07:22:00.299836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:02:11.315323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298380+0000","last_change":"2024-03-06T07:21:58.970098+0000","last_active":"2024-03-06T07:22:00.298380+0000","last_peered":"2024-03-06T07:22:00.298380+0000","last_clean":"2024-03-06T07:22:00.298380+0000","last_became_active":"2024-03-06T07:21:58.969786+0000","last_became_peered":"2024-03-06T07:21:58.969786+0000","last_unstale":"2024-03-06T07:22:00.298380+0000","last_undegraded":"2024-03-06T07:22:00.298380+0000","last_fullsized":"2024-03-06T07:22:00.298380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:07.060045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.297875+0000","last_change":"2024-03-06T07:21:58.961107+0000","last_active":"2024-03-06T07:22:00.297875+0000","last_peered":"2024-03-06T07:22:00.297875+0000","last_clean":"2024-03-06T07:22:00.297875+0000","last_became_active":"2024-03-06T07:21:58.960826+0000","last_became_peered":"2024-03-06T07:21:58.960826+0000","last_unstale":"2024-03-06T07:22:00.297875+0000","last_undegraded":"2024-03-06T07:22:00.297875+0000","last_fullsized":"2024-03-06T07:22:00.297875+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:36:02.122800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942422+0000","last_change":"2024-03-06T07:21:58.954994+0000","last_active":"2024-03-06T07:21:59.942422+0000","last_peered":"2024-03-06T07:21:59.942422+0000","last_clean":"2024-03-06T07:21:59.942422+0000","last_became_active":"2024-03-06T07:21:58.954710+0000","last_became_peered":"2024-03-06T07:21:58.954710+0000","last_unstale":"2024-03-06T07:21:59.942422+0000","last_undegraded":"2024-03-06T07:21:59.942422+0000","last_fullsized":"2024-03-06T07:21:59.942422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:12.378982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943938+0000","last_change":"2024-03-06T07:21:58.956241+0000","last_active":"2024-03-06T07:21:59.943938+0000","last_peered":"2024-03-06T07:21:59.943938+0000","last_clean":"2024-03-06T07:21:59.943938+0000","last_became_active":"2024-03-06T07:21:58.955977+0000","last_became_peered":"2024-03-06T07:21:58.955977+0000","last_unstale":"2024-03-06T07:21:59.943938+0000","last_undegraded":"2024-03-06T07:21:59.943938+0000","last_fullsized":"2024-03-06T07:21:59.943938+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:21:17.637762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298498+0000","last_change":"2024-03-06T07:21:58.963500+0000","last_active":"2024-03-06T07:22:00.298498+0000","last_peered":"2024-03-06T07:22:00.298498+0000","last_clean":"2024-03-06T07:22:00.298498+0000","last_became_active":"2024-03-06T07:21:58.963218+0000","last_became_peered":"2024-03-06T07:21:58.963218+0000","last_unstale":"2024-03-06T07:22:00.298498+0000","last_undegraded":"2024-03-06T07:22:00.298498+0000","last_fullsized":"2024-03-06T07:22:00.298498+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:12.378982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299375+0000","last_change":"2024-03-06T07:21:58.957981+0000","last_active":"2024-03-06T07:22:00.299375+0000","last_peered":"2024-03-06T07:22:00.299375+0000","last_clean":"2024-03-06T07:22:00.299375+0000","last_became_active":"2024-03-06T07:21:58.957690+0000","last_became_peered":"2024-03-06T07:21:58.957690+0000","last_unstale":"2024-03-06T07:22:00.299375+0000","last_undegraded":"2024-03-06T07:22:00.299375+0000","last_fullsized":"2024-03-06T07:22:00.299375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:59.261197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298443+0000","last_change":"2024-03-06T07:21:58.968221+0000","last_active":"2024-03-06T07:22:00.298443+0000","last_peered":"2024-03-06T07:22:00.298443+0000","last_clean":"2024-03-06T07:22:00.298443+0000","last_became_active":"2024-03-06T07:21:58.967798+0000","last_became_peered":"2024-03-06T07:21:58.967798+0000","last_unstale":"2024-03-06T07:22:00.298443+0000","last_undegraded":"2024-03-06T07:22:00.298443+0000","last_fullsized":"2024-03-06T07:22:00.298443+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:37:30.965258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298249+0000","last_change":"2024-03-06T07:21:58.966486+0000","last_active":"2024-03-06T07:22:00.298249+0000","last_peered":"2024-03-06T07:22:00.298249+0000","last_clean":"2024-03-06T07:22:00.298249+0000","last_became_active":"2024-03-06T07:21:58.966202+0000","last_became_peered":"2024-03-06T07:21:58.966202+0000","last_unstale":"2024-03-06T07:22:00.298249+0000","last_undegraded":"2024-03-06T07:22:00.298249+0000","last_fullsized":"2024-03-06T07:22:00.298249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:20.908786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942026+0000","last_change":"2024-03-06T07:21:58.952310+0000","last_active":"2024-03-06T07:21:59.942026+0000","last_peered":"2024-03-06T07:21:59.942026+0000","last_clean":"2024-03-06T07:21:59.942026+0000","last_became_active":"2024-03-06T07:21:58.952054+0000","last_became_peered":"2024-03-06T07:21:58.952054+0000","last_unstale":"2024-03-06T07:21:59.942026+0000","last_undegraded":"2024-03-06T07:21:59.942026+0000","last_fullsized":"2024-03-06T07:21:59.942026+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:55.294652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942908+0000","last_change":"2024-03-06T07:21:58.951201+0000","last_active":"2024-03-06T07:21:59.942908+0000","last_peered":"2024-03-06T07:21:59.942908+0000","last_clean":"2024-03-06T07:21:59.942908+0000","last_became_active":"2024-03-06T07:21:58.950746+0000","last_became_peered":"2024-03-06T07:21:58.950746+0000","last_unstale":"2024-03-06T07:21:59.942908+0000","last_undegraded":"2024-03-06T07:21:59.942908+0000","last_fullsized":"2024-03-06T07:21:59.942908+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:52.785061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299467+0000","last_change":"2024-03-06T07:21:57.947333+0000","last_active":"2024-03-06T07:22:00.299467+0000","last_peered":"2024-03-06T07:22:00.299467+0000","last_clean":"2024-03-06T07:22:00.299467+0000","last_became_active":"2024-03-06T07:21:57.947066+0000","last_became_peered":"2024-03-06T07:21:57.947066+0000","last_unstale":"2024-03-06T07:22:00.299467+0000","last_undegraded":"2024-03-06T07:22:00.299467+0000","last_fullsized":"2024-03-06T07:22:00.299467+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:18:20.077304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299505+0000","last_change":"2024-03-06T07:21:58.954116+0000","last_active":"2024-03-06T07:22:00.299505+0000","last_peered":"2024-03-06T07:22:00.299505+0000","last_clean":"2024-03-06T07:22:00.299505+0000","last_became_active":"2024-03-06T07:21:58.953858+0000","last_became_peered":"2024-03-06T07:21:58.953858+0000","last_unstale":"2024-03-06T07:22:00.299505+0000","last_undegraded":"2024-03-06T07:22:00.299505+0000","last_fullsized":"2024-03-06T07:22:00.299505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:12.378982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942428+0000","last_change":"2024-03-06T07:21:57.937765+0000","last_active":"2024-03-06T07:21:59.942428+0000","last_peered":"2024-03-06T07:21:59.942428+0000","last_clean":"2024-03-06T07:21:59.942428+0000","last_became_active":"2024-03-06T07:21:57.937476+0000","last_became_peered":"2024-03-06T07:21:57.937476+0000","last_unstale":"2024-03-06T07:21:59.942428+0000","last_undegraded":"2024-03-06T07:21:59.942428+0000","last_fullsized":"2024-03-06T07:21:59.942428+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:07:58.548928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299878+0000","last_change":"2024-03-06T07:21:57.943799+0000","last_active":"2024-03-06T07:22:00.299878+0000","last_peered":"2024-03-06T07:22:00.299878+0000","last_clean":"2024-03-06T07:22:00.299878+0000","last_became_active":"2024-03-06T07:21:57.943507+0000","last_became_peered":"2024-03-06T07:21:57.943507+0000","last_unstale":"2024-03-06T07:22:00.299878+0000","last_undegraded":"2024-03-06T07:22:00.299878+0000","last_fullsized":"2024-03-06T07:22:00.299878+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:31.179338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298759+0000","last_change":"2024-03-06T07:21:58.965398+0000","last_active":"2024-03-06T07:22:00.298759+0000","last_peered":"2024-03-06T07:22:00.298759+0000","last_clean":"2024-03-06T07:22:00.298759+0000","last_became_active":"2024-03-06T07:21:58.964367+0000","last_became_peered":"2024-03-06T07:21:58.964367+0000","last_unstale":"2024-03-06T07:22:00.298759+0000","last_undegraded":"2024-03-06T07:22:00.298759+0000","last_fullsized":"2024-03-06T07:22:00.298759+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:54.495097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298098+0000","last_change":"2024-03-06T07:21:57.951473+0000","last_active":"2024-03-06T07:22:00.298098+0000","last_peered":"2024-03-06T07:22:00.298098+0000","last_clean":"2024-03-06T07:22:00.298098+0000","last_became_active":"2024-03-06T07:21:57.951208+0000","last_became_peered":"2024-03-06T07:21:57.951208+0000","last_unstale":"2024-03-06T07:22:00.298098+0000","last_undegraded":"2024-03-06T07:22:00.298098+0000","last_fullsized":"2024-03-06T07:22:00.298098+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:51.781285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943705+0000","last_change":"2024-03-06T07:21:58.961873+0000","last_active":"2024-03-06T07:21:59.943705+0000","last_peered":"2024-03-06T07:21:59.943705+0000","last_clean":"2024-03-06T07:21:59.943705+0000","last_became_active":"2024-03-06T07:21:58.961630+0000","last_became_peered":"2024-03-06T07:21:58.961630+0000","last_unstale":"2024-03-06T07:21:59.943705+0000","last_undegraded":"2024-03-06T07:21:59.943705+0000","last_fullsized":"2024-03-06T07:21:59.943705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:55.294652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942284+0000","last_change":"2024-03-06T07:21:57.948765+0000","last_active":"2024-03-06T07:21:59.942284+0000","last_peered":"2024-03-06T07:21:59.942284+0000","last_clean":"2024-03-06T07:21:59.942284+0000","last_became_active":"2024-03-06T07:21:57.948498+0000","last_became_peered":"2024-03-06T07:21:57.948498+0000","last_unstale":"2024-03-06T07:21:59.942284+0000","last_undegraded":"2024-03-06T07:21:59.942284+0000","last_fullsized":"2024-03-06T07:21:59.942284+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:51.781285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298709+0000","last_change":"2024-03-06T07:21:58.971817+0000","last_active":"2024-03-06T07:22:00.298709+0000","last_peered":"2024-03-06T07:22:00.298709+0000","last_clean":"2024-03-06T07:22:00.298709+0000","last_became_active":"2024-03-06T07:21:58.971481+0000","last_became_peered":"2024-03-06T07:21:58.971481+0000","last_unstale":"2024-03-06T07:22:00.298709+0000","last_undegraded":"2024-03-06T07:22:00.298709+0000","last_fullsized":"2024-03-06T07:22:00.298709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:13.580324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298587+0000","last_change":"2024-03-06T07:21:57.950501+0000","last_active":"2024-03-06T07:22:00.298587+0000","last_peered":"2024-03-06T07:22:00.298587+0000","last_clean":"2024-03-06T07:22:00.298587+0000","last_became_active":"2024-03-06T07:21:57.950201+0000","last_became_peered":"2024-03-06T07:21:57.950201+0000","last_unstale":"2024-03-06T07:22:00.298587+0000","last_undegraded":"2024-03-06T07:22:00.298587+0000","last_fullsized":"2024-03-06T07:22:00.298587+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:13.736718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298791+0000","last_change":"2024-03-06T07:21:58.958089+0000","last_active":"2024-03-06T07:22:00.298791+0000","last_peered":"2024-03-06T07:22:00.298791+0000","last_clean":"2024-03-06T07:22:00.298791+0000","last_became_active":"2024-03-06T07:21:58.957814+0000","last_became_peered":"2024-03-06T07:21:58.957814+0000","last_unstale":"2024-03-06T07:22:00.298791+0000","last_undegraded":"2024-03-06T07:22:00.298791+0000","last_fullsized":"2024-03-06T07:22:00.298791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:55.294652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943941+0000","last_change":"2024-03-06T07:21:57.946124+0000","last_active":"2024-03-06T07:21:59.943941+0000","last_peered":"2024-03-06T07:21:59.943941+0000","last_clean":"2024-03-06T07:21:59.943941+0000","last_became_active":"2024-03-06T07:21:57.945856+0000","last_became_peered":"2024-03-06T07:21:57.945856+0000","last_unstale":"2024-03-06T07:21:59.943941+0000","last_undegraded":"2024-03-06T07:21:59.943941+0000","last_fullsized":"2024-03-06T07:21:59.943941+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:01.159166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943172+0000","last_change":"2024-03-06T07:21:58.958734+0000","last_active":"2024-03-06T07:21:59.943172+0000","last_peered":"2024-03-06T07:21:59.943172+0000","last_clean":"2024-03-06T07:21:59.943172+0000","last_became_active":"2024-03-06T07:21:58.958451+0000","last_became_peered":"2024-03-06T07:21:58.958451+0000","last_unstale":"2024-03-06T07:21:59.943172+0000","last_undegraded":"2024-03-06T07:21:59.943172+0000","last_fullsized":"2024-03-06T07:21:59.943172+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:20.908786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942962+0000","last_change":"2024-03-06T07:21:57.947199+0000","last_active":"2024-03-06T07:21:59.942962+0000","last_peered":"2024-03-06T07:21:59.942962+0000","last_clean":"2024-03-06T07:21:59.942962+0000","last_became_active":"2024-03-06T07:21:57.946927+0000","last_became_peered":"2024-03-06T07:21:57.946927+0000","last_unstale":"2024-03-06T07:21:59.942962+0000","last_undegraded":"2024-03-06T07:21:59.942962+0000","last_fullsized":"2024-03-06T07:21:59.942962+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:01.159166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298650+0000","last_change":"2024-03-06T07:21:58.970850+0000","last_active":"2024-03-06T07:22:00.298650+0000","last_peered":"2024-03-06T07:22:00.298650+0000","last_clean":"2024-03-06T07:22:00.298650+0000","last_became_active":"2024-03-06T07:21:58.970512+0000","last_became_peered":"2024-03-06T07:21:58.970512+0000","last_unstale":"2024-03-06T07:22:00.298650+0000","last_undegraded":"2024-03-06T07:22:00.298650+0000","last_fullsized":"2024-03-06T07:22:00.298650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:02:11.315323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299603+0000","last_change":"2024-03-06T07:21:57.951893+0000","last_active":"2024-03-06T07:22:00.299603+0000","last_peered":"2024-03-06T07:22:00.299603+0000","last_clean":"2024-03-06T07:22:00.299603+0000","last_became_active":"2024-03-06T07:21:57.951696+0000","last_became_peered":"2024-03-06T07:21:57.951696+0000","last_unstale":"2024-03-06T07:22:00.299603+0000","last_undegraded":"2024-03-06T07:22:00.299603+0000","last_fullsized":"2024-03-06T07:22:00.299603+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:27:31.864292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298310+0000","last_change":"2024-03-06T07:21:58.954594+0000","last_active":"2024-03-06T07:22:00.298310+0000","last_peered":"2024-03-06T07:22:00.298310+0000","last_clean":"2024-03-06T07:22:00.298310+0000","last_became_active":"2024-03-06T07:21:58.954326+0000","last_became_peered":"2024-03-06T07:21:58.954326+0000","last_unstale":"2024-03-06T07:22:00.298310+0000","last_undegraded":"2024-03-06T07:22:00.298310+0000","last_fullsized":"2024-03-06T07:22:00.298310+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:06:11.334051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298716+0000","last_change":"2024-03-06T07:21:57.956339+0000","last_active":"2024-03-06T07:22:00.298716+0000","last_peered":"2024-03-06T07:22:00.298716+0000","last_clean":"2024-03-06T07:22:00.298716+0000","last_became_active":"2024-03-06T07:21:57.956135+0000","last_became_peered":"2024-03-06T07:21:57.956135+0000","last_unstale":"2024-03-06T07:22:00.298716+0000","last_undegraded":"2024-03-06T07:22:00.298716+0000","last_fullsized":"2024-03-06T07:22:00.298716+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:01.159166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299011+0000","last_change":"2024-03-06T07:21:58.962391+0000","last_active":"2024-03-06T07:22:00.299011+0000","last_peered":"2024-03-06T07:22:00.299011+0000","last_clean":"2024-03-06T07:22:00.299011+0000","last_became_active":"2024-03-06T07:21:58.961887+0000","last_became_peered":"2024-03-06T07:21:58.961887+0000","last_unstale":"2024-03-06T07:22:00.299011+0000","last_undegraded":"2024-03-06T07:22:00.299011+0000","last_fullsized":"2024-03-06T07:22:00.299011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:20.758138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942138+0000","last_change":"2024-03-06T07:21:57.939474+0000","last_active":"2024-03-06T07:21:59.942138+0000","last_peered":"2024-03-06T07:21:59.942138+0000","last_clean":"2024-03-06T07:21:59.942138+0000","last_became_active":"2024-03-06T07:21:57.939053+0000","last_became_peered":"2024-03-06T07:21:57.939053+0000","last_unstale":"2024-03-06T07:21:59.942138+0000","last_undegraded":"2024-03-06T07:21:59.942138+0000","last_fullsized":"2024-03-06T07:21:59.942138+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:35.501272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942064+0000","last_change":"2024-03-06T07:21:58.958835+0000","last_active":"2024-03-06T07:21:59.942064+0000","last_peered":"2024-03-06T07:21:59.942064+0000","last_clean":"2024-03-06T07:21:59.942064+0000","last_became_active":"2024-03-06T07:21:58.958518+0000","last_became_peered":"2024-03-06T07:21:58.958518+0000","last_unstale":"2024-03-06T07:21:59.942064+0000","last_undegraded":"2024-03-06T07:21:59.942064+0000","last_fullsized":"2024-03-06T07:21:59.942064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:22.038204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298611+0000","last_change":"2024-03-06T07:21:57.940312+0000","last_active":"2024-03-06T07:22:00.298611+0000","last_peered":"2024-03-06T07:22:00.298611+0000","last_clean":"2024-03-06T07:22:00.298611+0000","last_became_active":"2024-03-06T07:21:57.940030+0000","last_became_peered":"2024-03-06T07:21:57.940030+0000","last_unstale":"2024-03-06T07:22:00.298611+0000","last_undegraded":"2024-03-06T07:22:00.298611+0000","last_fullsized":"2024-03-06T07:22:00.298611+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:35.501272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298350+0000","last_change":"2024-03-06T07:21:58.961737+0000","last_active":"2024-03-06T07:22:00.298350+0000","last_peered":"2024-03-06T07:22:00.298350+0000","last_clean":"2024-03-06T07:22:00.298350+0000","last_became_active":"2024-03-06T07:21:58.961365+0000","last_became_peered":"2024-03-06T07:21:58.961365+0000","last_unstale":"2024-03-06T07:22:00.298350+0000","last_undegraded":"2024-03-06T07:22:00.298350+0000","last_fullsized":"2024-03-06T07:22:00.298350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:44.889543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943032+0000","last_change":"2024-03-06T07:21:57.948442+0000","last_active":"2024-03-06T07:21:59.943032+0000","last_peered":"2024-03-06T07:21:59.943032+0000","last_clean":"2024-03-06T07:21:59.943032+0000","last_became_active":"2024-03-06T07:21:57.948170+0000","last_became_peered":"2024-03-06T07:21:57.948170+0000","last_unstale":"2024-03-06T07:21:59.943032+0000","last_undegraded":"2024-03-06T07:21:59.943032+0000","last_fullsized":"2024-03-06T07:21:59.943032+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:10.622090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298979+0000","last_change":"2024-03-06T07:21:58.961595+0000","last_active":"2024-03-06T07:22:00.298979+0000","last_peered":"2024-03-06T07:22:00.298979+0000","last_clean":"2024-03-06T07:22:00.298979+0000","last_became_active":"2024-03-06T07:21:58.961259+0000","last_became_peered":"2024-03-06T07:21:58.961259+0000","last_unstale":"2024-03-06T07:22:00.298979+0000","last_undegraded":"2024-03-06T07:22:00.298979+0000","last_fullsized":"2024-03-06T07:22:00.298979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:43.260641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298869+0000","last_change":"2024-03-06T07:21:57.942300+0000","last_active":"2024-03-06T07:22:00.298869+0000","last_peered":"2024-03-06T07:22:00.298869+0000","last_clean":"2024-03-06T07:22:00.298869+0000","last_became_active":"2024-03-06T07:21:57.942023+0000","last_became_peered":"2024-03-06T07:21:57.942023+0000","last_unstale":"2024-03-06T07:22:00.298869+0000","last_undegraded":"2024-03-06T07:22:00.298869+0000","last_fullsized":"2024-03-06T07:22:00.298869+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:07:58.548928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299225+0000","last_change":"2024-03-06T07:21:58.970895+0000","last_active":"2024-03-06T07:22:00.299225+0000","last_peered":"2024-03-06T07:22:00.299225+0000","last_clean":"2024-03-06T07:22:00.299225+0000","last_became_active":"2024-03-06T07:21:58.970418+0000","last_became_peered":"2024-03-06T07:21:58.970418+0000","last_unstale":"2024-03-06T07:22:00.299225+0000","last_undegraded":"2024-03-06T07:22:00.299225+0000","last_fullsized":"2024-03-06T07:22:00.299225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:19:55.013458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944014+0000","last_change":"2024-03-06T07:21:57.942325+0000","last_active":"2024-03-06T07:21:59.944014+0000","last_peered":"2024-03-06T07:21:59.944014+0000","last_clean":"2024-03-06T07:21:59.944014+0000","last_became_active":"2024-03-06T07:21:57.942045+0000","last_became_peered":"2024-03-06T07:21:57.942045+0000","last_unstale":"2024-03-06T07:21:59.944014+0000","last_undegraded":"2024-03-06T07:21:59.944014+0000","last_fullsized":"2024-03-06T07:21:59.944014+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:19.905010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298465+0000","last_change":"2024-03-06T07:21:58.967944+0000","last_active":"2024-03-06T07:22:00.298465+0000","last_peered":"2024-03-06T07:22:00.298465+0000","last_clean":"2024-03-06T07:22:00.298465+0000","last_became_active":"2024-03-06T07:21:58.967521+0000","last_became_peered":"2024-03-06T07:21:58.967521+0000","last_unstale":"2024-03-06T07:22:00.298465+0000","last_undegraded":"2024-03-06T07:22:00.298465+0000","last_fullsized":"2024-03-06T07:22:00.298465+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:43.260641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.297951+0000","last_change":"2024-03-06T07:21:57.947477+0000","last_active":"2024-03-06T07:22:00.297951+0000","last_peered":"2024-03-06T07:22:00.297951+0000","last_clean":"2024-03-06T07:22:00.297951+0000","last_became_active":"2024-03-06T07:21:57.946678+0000","last_became_peered":"2024-03-06T07:21:57.946678+0000","last_unstale":"2024-03-06T07:22:00.297951+0000","last_undegraded":"2024-03-06T07:22:00.297951+0000","last_fullsized":"2024-03-06T07:22:00.297951+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:13.736718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298718+0000","last_change":"2024-03-06T07:21:58.970473+0000","last_active":"2024-03-06T07:22:00.298718+0000","last_peered":"2024-03-06T07:22:00.298718+0000","last_clean":"2024-03-06T07:22:00.298718+0000","last_became_active":"2024-03-06T07:21:58.969810+0000","last_became_peered":"2024-03-06T07:21:58.969810+0000","last_unstale":"2024-03-06T07:22:00.298718+0000","last_undegraded":"2024-03-06T07:22:00.298718+0000","last_fullsized":"2024-03-06T07:22:00.298718+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:07.060045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298038+0000","last_change":"2024-03-06T07:21:47.553693+0000","last_active":"2024-03-06T07:22:00.298038+0000","last_peered":"2024-03-06T07:22:00.298038+0000","last_clean":"2024-03-06T07:22:00.298038+0000","last_became_active":"2024-03-06T07:21:47.552664+0000","last_became_peered":"2024-03-06T07:21:47.552664+0000","last_unstale":"2024-03-06T07:22:00.298038+0000","last_undegraded":"2024-03-06T07:22:00.298038+0000","last_fullsized":"2024-03-06T07:22:00.298038+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_clean_scrub_stamp":"2024-03-06T07:21:46.362028+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:02.916426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.297955+0000","last_change":"2024-03-06T07:21:57.949622+0000","last_active":"2024-03-06T07:22:00.297955+0000","last_peered":"2024-03-06T07:22:00.297955+0000","last_clean":"2024-03-06T07:22:00.297955+0000","last_became_active":"2024-03-06T07:21:57.948874+0000","last_became_peered":"2024-03-06T07:21:57.948874+0000","last_unstale":"2024-03-06T07:22:00.297955+0000","last_undegraded":"2024-03-06T07:22:00.297955+0000","last_fullsized":"2024-03-06T07:22:00.297955+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:35.501272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298026+0000","last_change":"2024-03-06T07:21:58.968667+0000","last_active":"2024-03-06T07:22:00.298026+0000","last_peered":"2024-03-06T07:22:00.298026+0000","last_clean":"2024-03-06T07:22:00.298026+0000","last_became_active":"2024-03-06T07:21:58.968213+0000","last_became_peered":"2024-03-06T07:21:58.968213+0000","last_unstale":"2024-03-06T07:22:00.298026+0000","last_undegraded":"2024-03-06T07:22:00.298026+0000","last_fullsized":"2024-03-06T07:22:00.298026+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:22.038204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.297808+0000","last_change":"2024-03-06T07:21:57.951697+0000","last_active":"2024-03-06T07:22:00.297808+0000","last_peered":"2024-03-06T07:22:00.297808+0000","last_clean":"2024-03-06T07:22:00.297808+0000","last_became_active":"2024-03-06T07:21:57.951421+0000","last_became_peered":"2024-03-06T07:21:57.951421+0000","last_unstale":"2024-03-06T07:22:00.297808+0000","last_undegraded":"2024-03-06T07:22:00.297808+0000","last_fullsized":"2024-03-06T07:22:00.297808+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:51.781285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.297721+0000","last_change":"2024-03-06T07:21:58.948562+0000","last_active":"2024-03-06T07:22:00.297721+0000","last_peered":"2024-03-06T07:22:00.297721+0000","last_clean":"2024-03-06T07:22:00.297721+0000","last_became_active":"2024-03-06T07:21:58.948249+0000","last_became_peered":"2024-03-06T07:21:58.948249+0000","last_unstale":"2024-03-06T07:22:00.297721+0000","last_undegraded":"2024-03-06T07:22:00.297721+0000","last_fullsized":"2024-03-06T07:22:00.297721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:53.360751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298409+0000","last_change":"2024-03-06T07:21:57.945890+0000","last_active":"2024-03-06T07:22:00.298409+0000","last_peered":"2024-03-06T07:22:00.298409+0000","last_clean":"2024-03-06T07:22:00.298409+0000","last_became_active":"2024-03-06T07:21:57.945527+0000","last_became_peered":"2024-03-06T07:21:57.945527+0000","last_unstale":"2024-03-06T07:22:00.298409+0000","last_undegraded":"2024-03-06T07:22:00.298409+0000","last_fullsized":"2024-03-06T07:22:00.298409+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:31.179338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943753+0000","last_change":"2024-03-06T07:21:58.952436+0000","last_active":"2024-03-06T07:21:59.943753+0000","last_peered":"2024-03-06T07:21:59.943753+0000","last_clean":"2024-03-06T07:21:59.943753+0000","last_became_active":"2024-03-06T07:21:58.952135+0000","last_became_peered":"2024-03-06T07:21:58.952135+0000","last_unstale":"2024-03-06T07:21:59.943753+0000","last_undegraded":"2024-03-06T07:21:59.943753+0000","last_fullsized":"2024-03-06T07:21:59.943753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:59.261197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298778+0000","last_change":"2024-03-06T07:21:57.951597+0000","last_active":"2024-03-06T07:22:00.298778+0000","last_peered":"2024-03-06T07:22:00.298778+0000","last_clean":"2024-03-06T07:22:00.298778+0000","last_became_active":"2024-03-06T07:21:57.951299+0000","last_became_peered":"2024-03-06T07:21:57.951299+0000","last_unstale":"2024-03-06T07:22:00.298778+0000","last_undegraded":"2024-03-06T07:22:00.298778+0000","last_fullsized":"2024-03-06T07:22:00.298778+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:19.847331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943217+0000","last_change":"2024-03-06T07:21:58.951633+0000","last_active":"2024-03-06T07:21:59.943217+0000","last_peered":"2024-03-06T07:21:59.943217+0000","last_clean":"2024-03-06T07:21:59.943217+0000","last_became_active":"2024-03-06T07:21:58.951315+0000","last_became_peered":"2024-03-06T07:21:58.951315+0000","last_unstale":"2024-03-06T07:21:59.943217+0000","last_undegraded":"2024-03-06T07:21:59.943217+0000","last_fullsized":"2024-03-06T07:21:59.943217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:20.851108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943826+0000","last_change":"2024-03-06T07:21:57.945253+0000","last_active":"2024-03-06T07:21:59.943826+0000","last_peered":"2024-03-06T07:21:59.943826+0000","last_clean":"2024-03-06T07:21:59.943826+0000","last_became_active":"2024-03-06T07:21:57.944983+0000","last_became_peered":"2024-03-06T07:21:57.944983+0000","last_unstale":"2024-03-06T07:21:59.943826+0000","last_undegraded":"2024-03-06T07:21:59.943826+0000","last_fullsized":"2024-03-06T07:21:59.943826+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:07:58.548928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298511+0000","last_change":"2024-03-06T07:21:58.964580+0000","last_active":"2024-03-06T07:22:00.298511+0000","last_peered":"2024-03-06T07:22:00.298511+0000","last_clean":"2024-03-06T07:22:00.298511+0000","last_became_active":"2024-03-06T07:21:58.964272+0000","last_became_peered":"2024-03-06T07:21:58.964272+0000","last_unstale":"2024-03-06T07:22:00.298511+0000","last_undegraded":"2024-03-06T07:22:00.298511+0000","last_fullsized":"2024-03-06T07:22:00.298511+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:02:11.315323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298902+0000","last_change":"2024-03-06T07:21:57.952261+0000","last_active":"2024-03-06T07:22:00.298902+0000","last_peered":"2024-03-06T07:22:00.298902+0000","last_clean":"2024-03-06T07:22:00.298902+0000","last_became_active":"2024-03-06T07:21:57.951991+0000","last_became_peered":"2024-03-06T07:21:57.951991+0000","last_unstale":"2024-03-06T07:22:00.298902+0000","last_undegraded":"2024-03-06T07:22:00.298902+0000","last_fullsized":"2024-03-06T07:22:00.298902+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:52.356974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943810+0000","last_change":"2024-03-06T07:21:58.955953+0000","last_active":"2024-03-06T07:21:59.943810+0000","last_peered":"2024-03-06T07:21:59.943810+0000","last_clean":"2024-03-06T07:21:59.943810+0000","last_became_active":"2024-03-06T07:21:58.955634+0000","last_became_peered":"2024-03-06T07:21:58.955634+0000","last_unstale":"2024-03-06T07:21:59.943810+0000","last_undegraded":"2024-03-06T07:21:59.943810+0000","last_fullsized":"2024-03-06T07:21:59.943810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:12.378982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943960+0000","last_change":"2024-03-06T07:21:57.943165+0000","last_active":"2024-03-06T07:21:59.943960+0000","last_peered":"2024-03-06T07:21:59.943960+0000","last_clean":"2024-03-06T07:21:59.943960+0000","last_became_active":"2024-03-06T07:21:57.942914+0000","last_became_peered":"2024-03-06T07:21:57.942914+0000","last_unstale":"2024-03-06T07:21:59.943960+0000","last_undegraded":"2024-03-06T07:21:59.943960+0000","last_fullsized":"2024-03-06T07:21:59.943960+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:19.847331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298017+0000","last_change":"2024-03-06T07:21:58.960410+0000","last_active":"2024-03-06T07:22:00.298017+0000","last_peered":"2024-03-06T07:22:00.298017+0000","last_clean":"2024-03-06T07:22:00.298017+0000","last_became_active":"2024-03-06T07:21:58.960121+0000","last_became_peered":"2024-03-06T07:21:58.960121+0000","last_unstale":"2024-03-06T07:22:00.298017+0000","last_undegraded":"2024-03-06T07:22:00.298017+0000","last_fullsized":"2024-03-06T07:22:00.298017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:55.294652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942389+0000","last_change":"2024-03-06T07:21:57.944106+0000","last_active":"2024-03-06T07:21:59.942389+0000","last_peered":"2024-03-06T07:21:59.942389+0000","last_clean":"2024-03-06T07:21:59.942389+0000","last_became_active":"2024-03-06T07:21:57.943815+0000","last_became_peered":"2024-03-06T07:21:57.943815+0000","last_unstale":"2024-03-06T07:21:59.942389+0000","last_undegraded":"2024-03-06T07:21:59.942389+0000","last_fullsized":"2024-03-06T07:21:59.942389+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:13.736718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298012+0000","last_change":"2024-03-06T07:21:58.951840+0000","last_active":"2024-03-06T07:22:00.298012+0000","last_peered":"2024-03-06T07:22:00.298012+0000","last_clean":"2024-03-06T07:22:00.298012+0000","last_became_active":"2024-03-06T07:21:58.951594+0000","last_became_peered":"2024-03-06T07:21:58.951594+0000","last_unstale":"2024-03-06T07:22:00.298012+0000","last_undegraded":"2024-03-06T07:22:00.298012+0000","last_fullsized":"2024-03-06T07:22:00.298012+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:34.528149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.297795+0000","last_change":"2024-03-06T07:21:57.951916+0000","last_active":"2024-03-06T07:22:00.297795+0000","last_peered":"2024-03-06T07:22:00.297795+0000","last_clean":"2024-03-06T07:22:00.297795+0000","last_became_active":"2024-03-06T07:21:57.951647+0000","last_became_peered":"2024-03-06T07:21:57.951647+0000","last_unstale":"2024-03-06T07:22:00.297795+0000","last_undegraded":"2024-03-06T07:22:00.297795+0000","last_fullsized":"2024-03-06T07:22:00.297795+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:13.736718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942859+0000","last_change":"2024-03-06T07:21:58.953147+0000","last_active":"2024-03-06T07:21:59.942859+0000","last_peered":"2024-03-06T07:21:59.942859+0000","last_clean":"2024-03-06T07:21:59.942859+0000","last_became_active":"2024-03-06T07:21:58.952862+0000","last_became_peered":"2024-03-06T07:21:58.952862+0000","last_unstale":"2024-03-06T07:21:59.942859+0000","last_undegraded":"2024-03-06T07:21:59.942859+0000","last_fullsized":"2024-03-06T07:21:59.942859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:14.740494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943795+0000","last_change":"2024-03-06T07:21:57.948593+0000","last_active":"2024-03-06T07:21:59.943795+0000","last_peered":"2024-03-06T07:21:59.943795+0000","last_clean":"2024-03-06T07:21:59.943795+0000","last_became_active":"2024-03-06T07:21:57.948302+0000","last_became_peered":"2024-03-06T07:21:57.948302+0000","last_unstale":"2024-03-06T07:21:59.943795+0000","last_undegraded":"2024-03-06T07:21:59.943795+0000","last_fullsized":"2024-03-06T07:21:59.943795+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:13.736718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298923+0000","last_change":"2024-03-06T07:21:58.953065+0000","last_active":"2024-03-06T07:22:00.298923+0000","last_peered":"2024-03-06T07:22:00.298923+0000","last_clean":"2024-03-06T07:22:00.298923+0000","last_became_active":"2024-03-06T07:21:58.952790+0000","last_became_peered":"2024-03-06T07:21:58.952790+0000","last_unstale":"2024-03-06T07:22:00.298923+0000","last_undegraded":"2024-03-06T07:22:00.298923+0000","last_fullsized":"2024-03-06T07:22:00.298923+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:30:54.760925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943752+0000","last_change":"2024-03-06T07:21:57.945134+0000","last_active":"2024-03-06T07:21:59.943752+0000","last_peered":"2024-03-06T07:21:59.943752+0000","last_clean":"2024-03-06T07:21:59.943752+0000","last_became_active":"2024-03-06T07:21:57.944858+0000","last_became_peered":"2024-03-06T07:21:57.944858+0000","last_unstale":"2024-03-06T07:21:59.943752+0000","last_undegraded":"2024-03-06T07:21:59.943752+0000","last_fullsized":"2024-03-06T07:21:59.943752+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:51.781285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299349+0000","last_change":"2024-03-06T07:21:58.964930+0000","last_active":"2024-03-06T07:22:00.299349+0000","last_peered":"2024-03-06T07:22:00.299349+0000","last_clean":"2024-03-06T07:22:00.299349+0000","last_became_active":"2024-03-06T07:21:58.964474+0000","last_became_peered":"2024-03-06T07:21:58.964474+0000","last_unstale":"2024-03-06T07:22:00.299349+0000","last_undegraded":"2024-03-06T07:22:00.299349+0000","last_fullsized":"2024-03-06T07:22:00.299349+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:44.889543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299058+0000","last_change":"2024-03-06T07:21:57.953248+0000","last_active":"2024-03-06T07:22:00.299058+0000","last_peered":"2024-03-06T07:22:00.299058+0000","last_clean":"2024-03-06T07:22:00.299058+0000","last_became_active":"2024-03-06T07:21:57.952969+0000","last_became_peered":"2024-03-06T07:21:57.952969+0000","last_unstale":"2024-03-06T07:22:00.299058+0000","last_undegraded":"2024-03-06T07:22:00.299058+0000","last_fullsized":"2024-03-06T07:22:00.299058+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:19.905010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942240+0000","last_change":"2024-03-06T07:21:58.954261+0000","last_active":"2024-03-06T07:21:59.942240+0000","last_peered":"2024-03-06T07:21:59.942240+0000","last_clean":"2024-03-06T07:21:59.942240+0000","last_became_active":"2024-03-06T07:21:58.953584+0000","last_became_peered":"2024-03-06T07:21:58.953584+0000","last_unstale":"2024-03-06T07:21:59.942240+0000","last_undegraded":"2024-03-06T07:21:59.942240+0000","last_fullsized":"2024-03-06T07:21:59.942240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:59.261197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298322+0000","last_change":"2024-03-06T07:21:58.959725+0000","last_active":"2024-03-06T07:22:00.298322+0000","last_peered":"2024-03-06T07:22:00.298322+0000","last_clean":"2024-03-06T07:22:00.298322+0000","last_became_active":"2024-03-06T07:21:58.959078+0000","last_became_peered":"2024-03-06T07:21:58.959078+0000","last_unstale":"2024-03-06T07:22:00.298322+0000","last_undegraded":"2024-03-06T07:22:00.298322+0000","last_fullsized":"2024-03-06T07:22:00.298322+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:44.889543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299074+0000","last_change":"2024-03-06T07:21:57.951758+0000","last_active":"2024-03-06T07:22:00.299074+0000","last_peered":"2024-03-06T07:22:00.299074+0000","last_clean":"2024-03-06T07:22:00.299074+0000","last_became_active":"2024-03-06T07:21:57.951452+0000","last_became_peered":"2024-03-06T07:21:57.951452+0000","last_unstale":"2024-03-06T07:22:00.299074+0000","last_undegraded":"2024-03-06T07:22:00.299074+0000","last_fullsized":"2024-03-06T07:22:00.299074+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:01.159166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298378+0000","last_change":"2024-03-06T07:21:58.961541+0000","last_active":"2024-03-06T07:22:00.298378+0000","last_peered":"2024-03-06T07:22:00.298378+0000","last_clean":"2024-03-06T07:22:00.298378+0000","last_became_active":"2024-03-06T07:21:58.961271+0000","last_became_peered":"2024-03-06T07:21:58.961271+0000","last_unstale":"2024-03-06T07:22:00.298378+0000","last_undegraded":"2024-03-06T07:22:00.298378+0000","last_fullsized":"2024-03-06T07:22:00.298378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:10:08.320367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298788+0000","last_change":"2024-03-06T07:21:57.947161+0000","last_active":"2024-03-06T07:22:00.298788+0000","last_peered":"2024-03-06T07:22:00.298788+0000","last_clean":"2024-03-06T07:22:00.298788+0000","last_became_active":"2024-03-06T07:21:57.946849+0000","last_became_peered":"2024-03-06T07:21:57.946849+0000","last_unstale":"2024-03-06T07:22:00.298788+0000","last_undegraded":"2024-03-06T07:22:00.298788+0000","last_fullsized":"2024-03-06T07:22:00.298788+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:19.847331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298661+0000","last_change":"2024-03-06T07:21:58.958814+0000","last_active":"2024-03-06T07:22:00.298661+0000","last_peered":"2024-03-06T07:22:00.298661+0000","last_clean":"2024-03-06T07:22:00.298661+0000","last_became_active":"2024-03-06T07:21:58.958550+0000","last_became_peered":"2024-03-06T07:21:58.958550+0000","last_unstale":"2024-03-06T07:22:00.298661+0000","last_undegraded":"2024-03-06T07:22:00.298661+0000","last_fullsized":"2024-03-06T07:22:00.298661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:36:02.122800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298388+0000","last_change":"2024-03-06T07:21:57.949031+0000","last_active":"2024-03-06T07:22:00.298388+0000","last_peered":"2024-03-06T07:22:00.298388+0000","last_clean":"2024-03-06T07:22:00.298388+0000","last_became_active":"2024-03-06T07:21:57.948741+0000","last_became_peered":"2024-03-06T07:21:57.948741+0000","last_unstale":"2024-03-06T07:22:00.298388+0000","last_undegraded":"2024-03-06T07:22:00.298388+0000","last_fullsized":"2024-03-06T07:22:00.298388+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:35.501272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298230+0000","last_change":"2024-03-06T07:21:58.957355+0000","last_active":"2024-03-06T07:22:00.298230+0000","last_peered":"2024-03-06T07:22:00.298230+0000","last_clean":"2024-03-06T07:22:00.298230+0000","last_became_active":"2024-03-06T07:21:58.956672+0000","last_became_peered":"2024-03-06T07:21:58.956672+0000","last_unstale":"2024-03-06T07:22:00.298230+0000","last_undegraded":"2024-03-06T07:22:00.298230+0000","last_fullsized":"2024-03-06T07:22:00.298230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:59.261197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298955+0000","last_change":"2024-03-06T07:21:57.951293+0000","last_active":"2024-03-06T07:22:00.298955+0000","last_peered":"2024-03-06T07:22:00.298955+0000","last_clean":"2024-03-06T07:22:00.298955+0000","last_became_active":"2024-03-06T07:21:57.950995+0000","last_became_peered":"2024-03-06T07:21:57.950995+0000","last_unstale":"2024-03-06T07:22:00.298955+0000","last_undegraded":"2024-03-06T07:22:00.298955+0000","last_fullsized":"2024-03-06T07:22:00.298955+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:10.622090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27792,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":790528,"data_stored":655584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":66246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":196608,"data_stored":64724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":64760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":66246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":192512,"data_stored":64634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":200704,"data_stored":64794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":798720,"data_stored":655640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:06.382 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:06.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.482+0000 7efcd8bb6640 1 Processor -- start 2024-03-06T07:22:06.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.482+0000 7efcd8bb6640 1 -- start start 2024-03-06T07:22:06.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.483+0000 7efcd8bb6640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7efcd41621c0 0x7efcd40cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.483+0000 7efcd8bb6640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efcd41625a0 0x7efcd415dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.483+0000 7efcd8bb6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcd40ceac0 con 0x7efcd40d4920 2024-03-06T07:22:06.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.483+0000 7efcd8bb6640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcd40cf690 con 0x7efcd41621c0 2024-03-06T07:22:06.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.483+0000 7efcd8bb6640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcd40d2dc0 con 0x7efcd41625a0 2024-03-06T07:22:06.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.483+0000 7efcd259c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7efcd41621c0 0x7efcd40cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.483+0000 7efccbfff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7efcd40d4920 0x7efcd40d26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:40978/0 (socket says 172.21.15.83:40978) 2024-03-06T07:22:06.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.483+0000 7efccbfff640 1 -- 172.21.15.83:0/2261890960 learned_addr learned my addr 172.21.15.83:0/2261890960 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:06.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.483+0000 7efcd2d9d640 1 --2- 172.21.15.83:0/2261890960 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efcd41625a0 0x7efcd415dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.483+0000 7efcd259c640 1 -- 172.21.15.83:0/2261890960 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efcd41625a0 msgr2=0x7efcd415dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:06.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.483+0000 7efcd259c640 1 --2- 172.21.15.83:0/2261890960 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efcd41625a0 0x7efcd415dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.483+0000 7efcd259c640 1 -- 172.21.15.83:0/2261890960 >> v1:172.21.15.83:6789/0 conn(0x7efcd40d4920 legacy=0x7efcd40d26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.483+0000 7efcd1d9b640 1 -- 172.21.15.83:0/2261890960 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 162152374 0 0) 0x7efcd40ceac0 con 0x7efcd40d4920 2024-03-06T07:22:06.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.483+0000 7efcd259c640 1 -- 172.21.15.83:0/2261890960 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd40ced60 con 0x7efcd41621c0 2024-03-06T07:22:06.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.484+0000 7efcd259c640 1 --2- 172.21.15.83:0/2261890960 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcd41621c0 0x7efcd40cfb20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7efcc40097a0 tx=0x7efcc402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c6be4ce8d4786a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.484+0000 7efcd1d9b640 1 -- 172.21.15.83:0/2261890960 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc4035020 con 0x7efcd41621c0 2024-03-06T07:22:06.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.484+0000 7efcd1d9b640 1 -- 172.21.15.83:0/2261890960 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcc4031830 con 0x7efcd41621c0 2024-03-06T07:22:06.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.484+0000 7efcd1d9b640 1 -- 172.21.15.83:0/2261890960 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc4031c00 con 0x7efcd41621c0 2024-03-06T07:22:06.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.485+0000 7efcd8bb6640 1 -- 172.21.15.83:0/2261890960 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcd41621c0 msgr2=0x7efcd40cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.485+0000 7efcd8bb6640 1 --2- 172.21.15.83:0/2261890960 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcd41621c0 0x7efcd40cfb20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7efcc40097a0 tx=0x7efcc402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.485+0000 7efcd8bb6640 1 -- 172.21.15.83:0/2261890960 shutdown_connections 2024-03-06T07:22:06.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.485+0000 7efcd8bb6640 1 --2- 172.21.15.83:0/2261890960 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efcd41625a0 0x7efcd415dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.485+0000 7efcd8bb6640 1 --2- 172.21.15.83:0/2261890960 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcd41621c0 0x7efcd40cfb20 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.485+0000 7efcd8bb6640 1 -- 172.21.15.83:0/2261890960 >> 172.21.15.83:0/2261890960 conn(0x7efcd40d3d10 msgr2=0x7efcd40d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:06.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.485+0000 7efcd8bb6640 1 -- 172.21.15.83:0/2261890960 shutdown_connections 2024-03-06T07:22:06.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.485+0000 7efcd8bb6640 1 -- 172.21.15.83:0/2261890960 wait complete. 2024-03-06T07:22:06.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.486+0000 7efcd8bb6640 1 Processor -- start 2024-03-06T07:22:06.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.486+0000 7efcd8bb6640 1 -- start start 2024-03-06T07:22:06.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.487+0000 7efcd8bb6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcd40d4920 0x7efcd40b8e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.487+0000 7efcd8bb6640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efcd41621c0 0x7efcd4157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.487+0000 7efcd8bb6640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efcd41625a0 0x7efcd40b6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.487+0000 7efcd8bb6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd415e280 con 0x7efcd40d4920 2024-03-06T07:22:06.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.487+0000 7efcd8bb6640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd40cf320 con 0x7efcd41625a0 2024-03-06T07:22:06.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.487+0000 7efcd8bb6640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcd415e400 con 0x7efcd41621c0 2024-03-06T07:22:06.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.487+0000 7efcd259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcd40d4920 0x7efcd40b8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.487+0000 7efcd259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcd40d4920 0x7efcd40b8e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34026/0 (socket says 172.21.15.83:34026) 2024-03-06T07:22:06.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.487+0000 7efcd259c640 1 -- 172.21.15.83:0/2914970021 learned_addr learned my addr 172.21.15.83:0/2914970021 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:06.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.487+0000 7efcd2d9d640 1 --2- 172.21.15.83:0/2914970021 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efcd41625a0 0x7efcd40b6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.487+0000 7efccbfff640 1 --2- 172.21.15.83:0/2914970021 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efcd41621c0 0x7efcd4157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.488+0000 7efcd259c640 1 -- 172.21.15.83:0/2914970021 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efcd41621c0 msgr2=0x7efcd4157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.488+0000 7efcd259c640 1 --2- 172.21.15.83:0/2914970021 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efcd41621c0 0x7efcd4157dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.488+0000 7efcd259c640 1 -- 172.21.15.83:0/2914970021 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efcd41625a0 msgr2=0x7efcd40b6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.488+0000 7efcd259c640 1 --2- 172.21.15.83:0/2914970021 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efcd41625a0 0x7efcd40b6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.488+0000 7efcd259c640 1 -- 172.21.15.83:0/2914970021 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd40b6d50 con 0x7efcd40d4920 2024-03-06T07:22:06.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.488+0000 7efccbfff640 1 --2- 172.21.15.83:0/2914970021 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efcd41621c0 0x7efcd4157dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:06.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.488+0000 7efcd2d9d640 1 --2- 172.21.15.83:0/2914970021 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efcd41625a0 0x7efcd40b6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:06.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.488+0000 7efcd259c640 1 --2- 172.21.15.83:0/2914970021 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcd40d4920 0x7efcd40b8e90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7efcc40098c0 tx=0x7efcc40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.488+0000 7efccaffd640 1 -- 172.21.15.83:0/2914970021 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc4035050 con 0x7efcd40d4920 2024-03-06T07:22:06.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.488+0000 7efccaffd640 1 -- 172.21.15.83:0/2914970021 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcc4004230 con 0x7efcd40d4920 2024-03-06T07:22:06.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.488+0000 7efcd8bb6640 1 -- 172.21.15.83:0/2914970021 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcd41584d0 con 0x7efcd40d4920 2024-03-06T07:22:06.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.488+0000 7efccaffd640 1 -- 172.21.15.83:0/2914970021 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc403b800 con 0x7efcd40d4920 2024-03-06T07:22:06.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.488+0000 7efcd8bb6640 1 -- 172.21.15.83:0/2914970021 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcd40b97e0 con 0x7efcd40d4920 2024-03-06T07:22:06.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.489+0000 7efcd8bb6640 1 -- 172.21.15.83:0/2914970021 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcd40b9a20 con 0x7efcd40d4920 2024-03-06T07:22:06.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.489+0000 7efccaffd640 1 -- 172.21.15.83:0/2914970021 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efcc4038020 con 0x7efcd40d4920 2024-03-06T07:22:06.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.490+0000 7efccaffd640 1 --2- 172.21.15.83:0/2914970021 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efcb406b4d0 0x7efcb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.490+0000 7efccaffd640 1 -- 172.21.15.83:0/2914970021 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efcc40abfe0 con 0x7efcd40d4920 2024-03-06T07:22:06.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.490+0000 7efccbfff640 1 --2- 172.21.15.83:0/2914970021 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efcb406b4d0 0x7efcb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.490+0000 7efccbfff640 1 --2- 172.21.15.83:0/2914970021 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efcb406b4d0 0x7efcb406d990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7efcd415f450 tx=0x7efcbc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.494+0000 7efccaffd640 1 -- 172.21.15.83:0/2914970021 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efcd40b9a20 con 0x7efcd40d4920 2024-03-06T07:22:06.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.666+0000 7efcd8bb6640 1 -- 172.21.15.83:0/2914970021 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efcd404f330 con 0x7efcb406b4d0 2024-03-06T07:22:06.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.672+0000 7efccaffd640 1 -- 172.21.15.83:0/2914970021 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251836 (secure 0 0 0) 0x7efcd404f330 con 0x7efcb406b4d0 2024-03-06T07:22:06.676 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:22:06.677 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-06T07:22:06.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.674+0000 7efcd8bb6640 1 -- 172.21.15.83:0/2914970021 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efcb406b4d0 msgr2=0x7efcb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.674+0000 7efcd8bb6640 1 --2- 172.21.15.83:0/2914970021 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efcb406b4d0 0x7efcb406d990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7efcd415f450 tx=0x7efcbc009040 comp rx=0 tx=0).stop 2024-03-06T07:22:06.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.674+0000 7efcd8bb6640 1 -- 172.21.15.83:0/2914970021 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcd40d4920 msgr2=0x7efcd40b8e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.674+0000 7efcd8bb6640 1 --2- 172.21.15.83:0/2914970021 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcd40d4920 0x7efcd40b8e90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7efcc40098c0 tx=0x7efcc40046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.675+0000 7efcd8bb6640 1 -- 172.21.15.83:0/2914970021 shutdown_connections 2024-03-06T07:22:06.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.675+0000 7efcd8bb6640 1 --2- 172.21.15.83:0/2914970021 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efcb406b4d0 0x7efcb406d990 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.675+0000 7efcd8bb6640 1 --2- 172.21.15.83:0/2914970021 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efcd41625a0 0x7efcd40b6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.675+0000 7efcd8bb6640 1 --2- 172.21.15.83:0/2914970021 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efcd41621c0 0x7efcd4157dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.675+0000 7efcd8bb6640 1 --2- 172.21.15.83:0/2914970021 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcd40d4920 0x7efcd40b8e90 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.675+0000 7efcd8bb6640 1 -- 172.21.15.83:0/2914970021 >> 172.21.15.83:0/2914970021 conn(0x7efcd40d3d10 msgr2=0x7efcd40d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:06.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.675+0000 7efcd8bb6640 1 -- 172.21.15.83:0/2914970021 shutdown_connections 2024-03-06T07:22:06.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.675+0000 7efcd8bb6640 1 -- 172.21.15.83:0/2914970021 wait complete. 2024-03-06T07:22:06.691 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-03-06T07:22:06.322623+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219020,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755700,"statfs":{"total":773094113280,"available":772869836800,"internally_reserved":0,"allocated":2891776,"data_stored":1753028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12757,"internal_metadata":221236779},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001390"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299166+0000","last_change":"2024-03-06T07:21:58.961070+0000","last_active":"2024-03-06T07:22:00.299166+0000","last_peered":"2024-03-06T07:22:00.299166+0000","last_clean":"2024-03-06T07:22:00.299166+0000","last_became_active":"2024-03-06T07:21:58.960784+0000","last_became_peered":"2024-03-06T07:21:58.960784+0000","last_unstale":"2024-03-06T07:22:00.299166+0000","last_undegraded":"2024-03-06T07:22:00.299166+0000","last_fullsized":"2024-03-06T07:22:00.299166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:06.413968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943975+0000","last_change":"2024-03-06T07:21:58.958978+0000","last_active":"2024-03-06T07:21:59.943975+0000","last_peered":"2024-03-06T07:21:59.943975+0000","last_clean":"2024-03-06T07:21:59.943975+0000","last_became_active":"2024-03-06T07:21:58.958728+0000","last_became_peered":"2024-03-06T07:21:58.958728+0000","last_unstale":"2024-03-06T07:21:59.943975+0000","last_undegraded":"2024-03-06T07:21:59.943975+0000","last_fullsized":"2024-03-06T07:21:59.943975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:54.495097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298515+0000","last_change":"2024-03-06T07:21:58.953968+0000","last_active":"2024-03-06T07:22:00.298515+0000","last_peered":"2024-03-06T07:22:00.298515+0000","last_clean":"2024-03-06T07:22:00.298515+0000","last_became_active":"2024-03-06T07:21:58.953678+0000","last_became_peered":"2024-03-06T07:21:58.953678+0000","last_unstale":"2024-03-06T07:22:00.298515+0000","last_undegraded":"2024-03-06T07:22:00.298515+0000","last_fullsized":"2024-03-06T07:22:00.298515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:51:39.557766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298069+0000","last_change":"2024-03-06T07:21:58.970682+0000","last_active":"2024-03-06T07:22:00.298069+0000","last_peered":"2024-03-06T07:22:00.298069+0000","last_clean":"2024-03-06T07:22:00.298069+0000","last_became_active":"2024-03-06T07:21:58.970397+0000","last_became_peered":"2024-03-06T07:21:58.970397+0000","last_unstale":"2024-03-06T07:22:00.298069+0000","last_undegraded":"2024-03-06T07:22:00.298069+0000","last_fullsized":"2024-03-06T07:22:00.298069+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:44.732365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298736+0000","last_change":"2024-03-06T07:21:58.965912+0000","last_active":"2024-03-06T07:22:00.298736+0000","last_peered":"2024-03-06T07:22:00.298736+0000","last_clean":"2024-03-06T07:22:00.298736+0000","last_became_active":"2024-03-06T07:21:58.965567+0000","last_became_peered":"2024-03-06T07:21:58.965567+0000","last_unstale":"2024-03-06T07:22:00.298736+0000","last_undegraded":"2024-03-06T07:22:00.298736+0000","last_fullsized":"2024-03-06T07:22:00.298736+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:10:08.320367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298064+0000","last_change":"2024-03-06T07:21:58.963244+0000","last_active":"2024-03-06T07:22:00.298064+0000","last_peered":"2024-03-06T07:22:00.298064+0000","last_clean":"2024-03-06T07:22:00.298064+0000","last_became_active":"2024-03-06T07:21:58.962903+0000","last_became_peered":"2024-03-06T07:21:58.962903+0000","last_unstale":"2024-03-06T07:22:00.298064+0000","last_undegraded":"2024-03-06T07:22:00.298064+0000","last_fullsized":"2024-03-06T07:22:00.298064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:30:54.760925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.297983+0000","last_change":"2024-03-06T07:21:58.964015+0000","last_active":"2024-03-06T07:22:00.297983+0000","last_peered":"2024-03-06T07:22:00.297983+0000","last_clean":"2024-03-06T07:22:00.297983+0000","last_became_active":"2024-03-06T07:21:58.962062+0000","last_became_peered":"2024-03-06T07:21:58.962062+0000","last_unstale":"2024-03-06T07:22:00.297983+0000","last_undegraded":"2024-03-06T07:22:00.297983+0000","last_fullsized":"2024-03-06T07:22:00.297983+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:44.732365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298608+0000","last_change":"2024-03-06T07:21:58.968307+0000","last_active":"2024-03-06T07:22:00.298608+0000","last_peered":"2024-03-06T07:22:00.298608+0000","last_clean":"2024-03-06T07:22:00.298608+0000","last_became_active":"2024-03-06T07:21:58.967917+0000","last_became_peered":"2024-03-06T07:21:58.967917+0000","last_unstale":"2024-03-06T07:22:00.298608+0000","last_undegraded":"2024-03-06T07:22:00.298608+0000","last_fullsized":"2024-03-06T07:22:00.298608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:21:17.637762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299130+0000","last_change":"2024-03-06T07:21:58.971497+0000","last_active":"2024-03-06T07:22:00.299130+0000","last_peered":"2024-03-06T07:22:00.299130+0000","last_clean":"2024-03-06T07:22:00.299130+0000","last_became_active":"2024-03-06T07:21:58.970710+0000","last_became_peered":"2024-03-06T07:21:58.970710+0000","last_unstale":"2024-03-06T07:22:00.299130+0000","last_undegraded":"2024-03-06T07:22:00.299130+0000","last_fullsized":"2024-03-06T07:22:00.299130+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:13.580324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943457+0000","last_change":"2024-03-06T07:21:58.956333+0000","last_active":"2024-03-06T07:21:59.943457+0000","last_peered":"2024-03-06T07:21:59.943457+0000","last_clean":"2024-03-06T07:21:59.943457+0000","last_became_active":"2024-03-06T07:21:58.956073+0000","last_became_peered":"2024-03-06T07:21:58.956073+0000","last_unstale":"2024-03-06T07:21:59.943457+0000","last_undegraded":"2024-03-06T07:21:59.943457+0000","last_fullsized":"2024-03-06T07:21:59.943457+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:30:54.760925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943172+0000","last_change":"2024-03-06T07:21:58.956210+0000","last_active":"2024-03-06T07:21:59.943172+0000","last_peered":"2024-03-06T07:21:59.943172+0000","last_clean":"2024-03-06T07:21:59.943172+0000","last_became_active":"2024-03-06T07:21:58.955949+0000","last_became_peered":"2024-03-06T07:21:58.955949+0000","last_unstale":"2024-03-06T07:21:59.943172+0000","last_undegraded":"2024-03-06T07:21:59.943172+0000","last_fullsized":"2024-03-06T07:21:59.943172+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:54.495097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943018+0000","last_change":"2024-03-06T07:21:58.950931+0000","last_active":"2024-03-06T07:21:59.943018+0000","last_peered":"2024-03-06T07:21:59.943018+0000","last_clean":"2024-03-06T07:21:59.943018+0000","last_became_active":"2024-03-06T07:21:58.949704+0000","last_became_peered":"2024-03-06T07:21:58.949704+0000","last_unstale":"2024-03-06T07:21:59.943018+0000","last_undegraded":"2024-03-06T07:21:59.943018+0000","last_fullsized":"2024-03-06T07:21:59.943018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:02.162943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299186+0000","last_change":"2024-03-06T07:21:58.956175+0000","last_active":"2024-03-06T07:22:00.299186+0000","last_peered":"2024-03-06T07:22:00.299186+0000","last_clean":"2024-03-06T07:22:00.299186+0000","last_became_active":"2024-03-06T07:21:58.955857+0000","last_became_peered":"2024-03-06T07:21:58.955857+0000","last_unstale":"2024-03-06T07:22:00.299186+0000","last_undegraded":"2024-03-06T07:22:00.299186+0000","last_fullsized":"2024-03-06T07:22:00.299186+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:44.732365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298168+0000","last_change":"2024-03-06T07:21:58.951176+0000","last_active":"2024-03-06T07:22:00.298168+0000","last_peered":"2024-03-06T07:22:00.298168+0000","last_clean":"2024-03-06T07:22:00.298168+0000","last_became_active":"2024-03-06T07:21:58.950874+0000","last_became_peered":"2024-03-06T07:21:58.950874+0000","last_unstale":"2024-03-06T07:22:00.298168+0000","last_undegraded":"2024-03-06T07:22:00.298168+0000","last_fullsized":"2024-03-06T07:22:00.298168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:42:14.052025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943706+0000","last_change":"2024-03-06T07:21:58.954562+0000","last_active":"2024-03-06T07:21:59.943706+0000","last_peered":"2024-03-06T07:21:59.943706+0000","last_clean":"2024-03-06T07:21:59.943706+0000","last_became_active":"2024-03-06T07:21:58.954242+0000","last_became_peered":"2024-03-06T07:21:58.954242+0000","last_unstale":"2024-03-06T07:21:59.943706+0000","last_undegraded":"2024-03-06T07:21:59.943706+0000","last_fullsized":"2024-03-06T07:21:59.943706+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:22.038204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.297923+0000","last_change":"2024-03-06T07:21:58.963478+0000","last_active":"2024-03-06T07:22:00.297923+0000","last_peered":"2024-03-06T07:22:00.297923+0000","last_clean":"2024-03-06T07:22:00.297923+0000","last_became_active":"2024-03-06T07:21:58.962917+0000","last_became_peered":"2024-03-06T07:21:58.962917+0000","last_unstale":"2024-03-06T07:22:00.297923+0000","last_undegraded":"2024-03-06T07:22:00.297923+0000","last_fullsized":"2024-03-06T07:22:00.297923+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:54.495097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298958+0000","last_change":"2024-03-06T07:21:58.965994+0000","last_active":"2024-03-06T07:22:00.298958+0000","last_peered":"2024-03-06T07:22:00.298958+0000","last_clean":"2024-03-06T07:22:00.298958+0000","last_became_active":"2024-03-06T07:21:58.965655+0000","last_became_peered":"2024-03-06T07:21:58.965655+0000","last_unstale":"2024-03-06T07:22:00.298958+0000","last_undegraded":"2024-03-06T07:22:00.298958+0000","last_fullsized":"2024-03-06T07:22:00.298958+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:06.413968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298767+0000","last_change":"2024-03-06T07:21:58.955322+0000","last_active":"2024-03-06T07:22:00.298767+0000","last_peered":"2024-03-06T07:22:00.298767+0000","last_clean":"2024-03-06T07:22:00.298767+0000","last_became_active":"2024-03-06T07:21:58.955080+0000","last_became_peered":"2024-03-06T07:21:58.955080+0000","last_unstale":"2024-03-06T07:22:00.298767+0000","last_undegraded":"2024-03-06T07:22:00.298767+0000","last_fullsized":"2024-03-06T07:22:00.298767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:20.851108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298974+0000","last_change":"2024-03-06T07:21:58.966461+0000","last_active":"2024-03-06T07:22:00.298974+0000","last_peered":"2024-03-06T07:22:00.298974+0000","last_clean":"2024-03-06T07:22:00.298974+0000","last_became_active":"2024-03-06T07:21:58.966154+0000","last_became_peered":"2024-03-06T07:21:58.966154+0000","last_unstale":"2024-03-06T07:22:00.298974+0000","last_undegraded":"2024-03-06T07:22:00.298974+0000","last_fullsized":"2024-03-06T07:22:00.298974+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:37:30.965258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942831+0000","last_change":"2024-03-06T07:21:58.953916+0000","last_active":"2024-03-06T07:21:59.942831+0000","last_peered":"2024-03-06T07:21:59.942831+0000","last_clean":"2024-03-06T07:21:59.942831+0000","last_became_active":"2024-03-06T07:21:58.953623+0000","last_became_peered":"2024-03-06T07:21:58.953623+0000","last_unstale":"2024-03-06T07:21:59.942831+0000","last_undegraded":"2024-03-06T07:21:59.942831+0000","last_fullsized":"2024-03-06T07:21:59.942831+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:21:17.637762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299245+0000","last_change":"2024-03-06T07:21:58.955378+0000","last_active":"2024-03-06T07:22:00.299245+0000","last_peered":"2024-03-06T07:22:00.299245+0000","last_clean":"2024-03-06T07:22:00.299245+0000","last_became_active":"2024-03-06T07:21:58.955118+0000","last_became_peered":"2024-03-06T07:21:58.955118+0000","last_unstale":"2024-03-06T07:22:00.299245+0000","last_undegraded":"2024-03-06T07:22:00.299245+0000","last_fullsized":"2024-03-06T07:22:00.299245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:22.038204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298483+0000","last_change":"2024-03-06T07:21:58.963849+0000","last_active":"2024-03-06T07:22:00.298483+0000","last_peered":"2024-03-06T07:22:00.298483+0000","last_clean":"2024-03-06T07:22:00.298483+0000","last_became_active":"2024-03-06T07:21:58.963573+0000","last_became_peered":"2024-03-06T07:21:58.963573+0000","last_unstale":"2024-03-06T07:22:00.298483+0000","last_undegraded":"2024-03-06T07:22:00.298483+0000","last_fullsized":"2024-03-06T07:22:00.298483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:20.758138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299836+0000","last_change":"2024-03-06T07:21:58.962026+0000","last_active":"2024-03-06T07:22:00.299836+0000","last_peered":"2024-03-06T07:22:00.299836+0000","last_clean":"2024-03-06T07:22:00.299836+0000","last_became_active":"2024-03-06T07:21:58.961691+0000","last_became_peered":"2024-03-06T07:21:58.961691+0000","last_unstale":"2024-03-06T07:22:00.299836+0000","last_undegraded":"2024-03-06T07:22:00.299836+0000","last_fullsized":"2024-03-06T07:22:00.299836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:02:11.315323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298380+0000","last_change":"2024-03-06T07:21:58.970098+0000","last_active":"2024-03-06T07:22:00.298380+0000","last_peered":"2024-03-06T07:22:00.298380+0000","last_clean":"2024-03-06T07:22:00.298380+0000","last_became_active":"2024-03-06T07:21:58.969786+0000","last_became_peered":"2024-03-06T07:21:58.969786+0000","last_unstale":"2024-03-06T07:22:00.298380+0000","last_undegraded":"2024-03-06T07:22:00.298380+0000","last_fullsized":"2024-03-06T07:22:00.298380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:07.060045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.297875+0000","last_change":"2024-03-06T07:21:58.961107+0000","last_active":"2024-03-06T07:22:00.297875+0000","last_peered":"2024-03-06T07:22:00.297875+0000","last_clean":"2024-03-06T07:22:00.297875+0000","last_became_active":"2024-03-06T07:21:58.960826+0000","last_became_peered":"2024-03-06T07:21:58.960826+0000","last_unstale":"2024-03-06T07:22:00.297875+0000","last_undegraded":"2024-03-06T07:22:00.297875+0000","last_fullsized":"2024-03-06T07:22:00.297875+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:36:02.122800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942422+0000","last_change":"2024-03-06T07:21:58.954994+0000","last_active":"2024-03-06T07:21:59.942422+0000","last_peered":"2024-03-06T07:21:59.942422+0000","last_clean":"2024-03-06T07:21:59.942422+0000","last_became_active":"2024-03-06T07:21:58.954710+0000","last_became_peered":"2024-03-06T07:21:58.954710+0000","last_unstale":"2024-03-06T07:21:59.942422+0000","last_undegraded":"2024-03-06T07:21:59.942422+0000","last_fullsized":"2024-03-06T07:21:59.942422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:12.378982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943938+0000","last_change":"2024-03-06T07:21:58.956241+0000","last_active":"2024-03-06T07:21:59.943938+0000","last_peered":"2024-03-06T07:21:59.943938+0000","last_clean":"2024-03-06T07:21:59.943938+0000","last_became_active":"2024-03-06T07:21:58.955977+0000","last_became_peered":"2024-03-06T07:21:58.955977+0000","last_unstale":"2024-03-06T07:21:59.943938+0000","last_undegraded":"2024-03-06T07:21:59.943938+0000","last_fullsized":"2024-03-06T07:21:59.943938+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:21:17.637762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298498+0000","last_change":"2024-03-06T07:21:58.963500+0000","last_active":"2024-03-06T07:22:00.298498+0000","last_peered":"2024-03-06T07:22:00.298498+0000","last_clean":"2024-03-06T07:22:00.298498+0000","last_became_active":"2024-03-06T07:21:58.963218+0000","last_became_peered":"2024-03-06T07:21:58.963218+0000","last_unstale":"2024-03-06T07:22:00.298498+0000","last_undegraded":"2024-03-06T07:22:00.298498+0000","last_fullsized":"2024-03-06T07:22:00.298498+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:12.378982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299375+0000","last_change":"2024-03-06T07:21:58.957981+0000","last_active":"2024-03-06T07:22:00.299375+0000","last_peered":"2024-03-06T07:22:00.299375+0000","last_clean":"2024-03-06T07:22:00.299375+0000","last_became_active":"2024-03-06T07:21:58.957690+0000","last_became_peered":"2024-03-06T07:21:58.957690+0000","last_unstale":"2024-03-06T07:22:00.299375+0000","last_undegraded":"2024-03-06T07:22:00.299375+0000","last_fullsized":"2024-03-06T07:22:00.299375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:59.261197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298443+0000","last_change":"2024-03-06T07:21:58.968221+0000","last_active":"2024-03-06T07:22:00.298443+0000","last_peered":"2024-03-06T07:22:00.298443+0000","last_clean":"2024-03-06T07:22:00.298443+0000","last_became_active":"2024-03-06T07:21:58.967798+0000","last_became_peered":"2024-03-06T07:21:58.967798+0000","last_unstale":"2024-03-06T07:22:00.298443+0000","last_undegraded":"2024-03-06T07:22:00.298443+0000","last_fullsized":"2024-03-06T07:22:00.298443+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:37:30.965258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298249+0000","last_change":"2024-03-06T07:21:58.966486+0000","last_active":"2024-03-06T07:22:00.298249+0000","last_peered":"2024-03-06T07:22:00.298249+0000","last_clean":"2024-03-06T07:22:00.298249+0000","last_became_active":"2024-03-06T07:21:58.966202+0000","last_became_peered":"2024-03-06T07:21:58.966202+0000","last_unstale":"2024-03-06T07:22:00.298249+0000","last_undegraded":"2024-03-06T07:22:00.298249+0000","last_fullsized":"2024-03-06T07:22:00.298249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:20.908786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942026+0000","last_change":"2024-03-06T07:21:58.952310+0000","last_active":"2024-03-06T07:21:59.942026+0000","last_peered":"2024-03-06T07:21:59.942026+0000","last_clean":"2024-03-06T07:21:59.942026+0000","last_became_active":"2024-03-06T07:21:58.952054+0000","last_became_peered":"2024-03-06T07:21:58.952054+0000","last_unstale":"2024-03-06T07:21:59.942026+0000","last_undegraded":"2024-03-06T07:21:59.942026+0000","last_fullsized":"2024-03-06T07:21:59.942026+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:55.294652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942908+0000","last_change":"2024-03-06T07:21:58.951201+0000","last_active":"2024-03-06T07:21:59.942908+0000","last_peered":"2024-03-06T07:21:59.942908+0000","last_clean":"2024-03-06T07:21:59.942908+0000","last_became_active":"2024-03-06T07:21:58.950746+0000","last_became_peered":"2024-03-06T07:21:58.950746+0000","last_unstale":"2024-03-06T07:21:59.942908+0000","last_undegraded":"2024-03-06T07:21:59.942908+0000","last_fullsized":"2024-03-06T07:21:59.942908+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:52.785061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299467+0000","last_change":"2024-03-06T07:21:57.947333+0000","last_active":"2024-03-06T07:22:00.299467+0000","last_peered":"2024-03-06T07:22:00.299467+0000","last_clean":"2024-03-06T07:22:00.299467+0000","last_became_active":"2024-03-06T07:21:57.947066+0000","last_became_peered":"2024-03-06T07:21:57.947066+0000","last_unstale":"2024-03-06T07:22:00.299467+0000","last_undegraded":"2024-03-06T07:22:00.299467+0000","last_fullsized":"2024-03-06T07:22:00.299467+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:18:20.077304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299505+0000","last_change":"2024-03-06T07:21:58.954116+0000","last_active":"2024-03-06T07:22:00.299505+0000","last_peered":"2024-03-06T07:22:00.299505+0000","last_clean":"2024-03-06T07:22:00.299505+0000","last_became_active":"2024-03-06T07:21:58.953858+0000","last_became_peered":"2024-03-06T07:21:58.953858+0000","last_unstale":"2024-03-06T07:22:00.299505+0000","last_undegraded":"2024-03-06T07:22:00.299505+0000","last_fullsized":"2024-03-06T07:22:00.299505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:12.378982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942428+0000","last_change":"2024-03-06T07:21:57.937765+0000","last_active":"2024-03-06T07:21:59.942428+0000","last_peered":"2024-03-06T07:21:59.942428+0000","last_clean":"2024-03-06T07:21:59.942428+0000","last_became_active":"2024-03-06T07:21:57.937476+0000","last_became_peered":"2024-03-06T07:21:57.937476+0000","last_unstale":"2024-03-06T07:21:59.942428+0000","last_undegraded":"2024-03-06T07:21:59.942428+0000","last_fullsized":"2024-03-06T07:21:59.942428+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:07:58.548928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299878+0000","last_change":"2024-03-06T07:21:57.943799+0000","last_active":"2024-03-06T07:22:00.299878+0000","last_peered":"2024-03-06T07:22:00.299878+0000","last_clean":"2024-03-06T07:22:00.299878+0000","last_became_active":"2024-03-06T07:21:57.943507+0000","last_became_peered":"2024-03-06T07:21:57.943507+0000","last_unstale":"2024-03-06T07:22:00.299878+0000","last_undegraded":"2024-03-06T07:22:00.299878+0000","last_fullsized":"2024-03-06T07:22:00.299878+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:31.179338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298759+0000","last_change":"2024-03-06T07:21:58.965398+0000","last_active":"2024-03-06T07:22:00.298759+0000","last_peered":"2024-03-06T07:22:00.298759+0000","last_clean":"2024-03-06T07:22:00.298759+0000","last_became_active":"2024-03-06T07:21:58.964367+0000","last_became_peered":"2024-03-06T07:21:58.964367+0000","last_unstale":"2024-03-06T07:22:00.298759+0000","last_undegraded":"2024-03-06T07:22:00.298759+0000","last_fullsized":"2024-03-06T07:22:00.298759+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:54.495097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298098+0000","last_change":"2024-03-06T07:21:57.951473+0000","last_active":"2024-03-06T07:22:00.298098+0000","last_peered":"2024-03-06T07:22:00.298098+0000","last_clean":"2024-03-06T07:22:00.298098+0000","last_became_active":"2024-03-06T07:21:57.951208+0000","last_became_peered":"2024-03-06T07:21:57.951208+0000","last_unstale":"2024-03-06T07:22:00.298098+0000","last_undegraded":"2024-03-06T07:22:00.298098+0000","last_fullsized":"2024-03-06T07:22:00.298098+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:51.781285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943705+0000","last_change":"2024-03-06T07:21:58.961873+0000","last_active":"2024-03-06T07:21:59.943705+0000","last_peered":"2024-03-06T07:21:59.943705+0000","last_clean":"2024-03-06T07:21:59.943705+0000","last_became_active":"2024-03-06T07:21:58.961630+0000","last_became_peered":"2024-03-06T07:21:58.961630+0000","last_unstale":"2024-03-06T07:21:59.943705+0000","last_undegraded":"2024-03-06T07:21:59.943705+0000","last_fullsized":"2024-03-06T07:21:59.943705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:55.294652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942284+0000","last_change":"2024-03-06T07:21:57.948765+0000","last_active":"2024-03-06T07:21:59.942284+0000","last_peered":"2024-03-06T07:21:59.942284+0000","last_clean":"2024-03-06T07:21:59.942284+0000","last_became_active":"2024-03-06T07:21:57.948498+0000","last_became_peered":"2024-03-06T07:21:57.948498+0000","last_unstale":"2024-03-06T07:21:59.942284+0000","last_undegraded":"2024-03-06T07:21:59.942284+0000","last_fullsized":"2024-03-06T07:21:59.942284+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:51.781285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298709+0000","last_change":"2024-03-06T07:21:58.971817+0000","last_active":"2024-03-06T07:22:00.298709+0000","last_peered":"2024-03-06T07:22:00.298709+0000","last_clean":"2024-03-06T07:22:00.298709+0000","last_became_active":"2024-03-06T07:21:58.971481+0000","last_became_peered":"2024-03-06T07:21:58.971481+0000","last_unstale":"2024-03-06T07:22:00.298709+0000","last_undegraded":"2024-03-06T07:22:00.298709+0000","last_fullsized":"2024-03-06T07:22:00.298709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:13.580324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298587+0000","last_change":"2024-03-06T07:21:57.950501+0000","last_active":"2024-03-06T07:22:00.298587+0000","last_peered":"2024-03-06T07:22:00.298587+0000","last_clean":"2024-03-06T07:22:00.298587+0000","last_became_active":"2024-03-06T07:21:57.950201+0000","last_became_peered":"2024-03-06T07:21:57.950201+0000","last_unstale":"2024-03-06T07:22:00.298587+0000","last_undegraded":"2024-03-06T07:22:00.298587+0000","last_fullsized":"2024-03-06T07:22:00.298587+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:13.736718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298791+0000","last_change":"2024-03-06T07:21:58.958089+0000","last_active":"2024-03-06T07:22:00.298791+0000","last_peered":"2024-03-06T07:22:00.298791+0000","last_clean":"2024-03-06T07:22:00.298791+0000","last_became_active":"2024-03-06T07:21:58.957814+0000","last_became_peered":"2024-03-06T07:21:58.957814+0000","last_unstale":"2024-03-06T07:22:00.298791+0000","last_undegraded":"2024-03-06T07:22:00.298791+0000","last_fullsized":"2024-03-06T07:22:00.298791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:55.294652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943941+0000","last_change":"2024-03-06T07:21:57.946124+0000","last_active":"2024-03-06T07:21:59.943941+0000","last_peered":"2024-03-06T07:21:59.943941+0000","last_clean":"2024-03-06T07:21:59.943941+0000","last_became_active":"2024-03-06T07:21:57.945856+0000","last_became_peered":"2024-03-06T07:21:57.945856+0000","last_unstale":"2024-03-06T07:21:59.943941+0000","last_undegraded":"2024-03-06T07:21:59.943941+0000","last_fullsized":"2024-03-06T07:21:59.943941+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:01.159166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943172+0000","last_change":"2024-03-06T07:21:58.958734+0000","last_active":"2024-03-06T07:21:59.943172+0000","last_peered":"2024-03-06T07:21:59.943172+0000","last_clean":"2024-03-06T07:21:59.943172+0000","last_became_active":"2024-03-06T07:21:58.958451+0000","last_became_peered":"2024-03-06T07:21:58.958451+0000","last_unstale":"2024-03-06T07:21:59.943172+0000","last_undegraded":"2024-03-06T07:21:59.943172+0000","last_fullsized":"2024-03-06T07:21:59.943172+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:20.908786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942962+0000","last_change":"2024-03-06T07:21:57.947199+0000","last_active":"2024-03-06T07:21:59.942962+0000","last_peered":"2024-03-06T07:21:59.942962+0000","last_clean":"2024-03-06T07:21:59.942962+0000","last_became_active":"2024-03-06T07:21:57.946927+0000","last_became_peered":"2024-03-06T07:21:57.946927+0000","last_unstale":"2024-03-06T07:21:59.942962+0000","last_undegraded":"2024-03-06T07:21:59.942962+0000","last_fullsized":"2024-03-06T07:21:59.942962+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:01.159166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298650+0000","last_change":"2024-03-06T07:21:58.970850+0000","last_active":"2024-03-06T07:22:00.298650+0000","last_peered":"2024-03-06T07:22:00.298650+0000","last_clean":"2024-03-06T07:22:00.298650+0000","last_became_active":"2024-03-06T07:21:58.970512+0000","last_became_peered":"2024-03-06T07:21:58.970512+0000","last_unstale":"2024-03-06T07:22:00.298650+0000","last_undegraded":"2024-03-06T07:22:00.298650+0000","last_fullsized":"2024-03-06T07:22:00.298650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:02:11.315323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299603+0000","last_change":"2024-03-06T07:21:57.951893+0000","last_active":"2024-03-06T07:22:00.299603+0000","last_peered":"2024-03-06T07:22:00.299603+0000","last_clean":"2024-03-06T07:22:00.299603+0000","last_became_active":"2024-03-06T07:21:57.951696+0000","last_became_peered":"2024-03-06T07:21:57.951696+0000","last_unstale":"2024-03-06T07:22:00.299603+0000","last_undegraded":"2024-03-06T07:22:00.299603+0000","last_fullsized":"2024-03-06T07:22:00.299603+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:27:31.864292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298310+0000","last_change":"2024-03-06T07:21:58.954594+0000","last_active":"2024-03-06T07:22:00.298310+0000","last_peered":"2024-03-06T07:22:00.298310+0000","last_clean":"2024-03-06T07:22:00.298310+0000","last_became_active":"2024-03-06T07:21:58.954326+0000","last_became_peered":"2024-03-06T07:21:58.954326+0000","last_unstale":"2024-03-06T07:22:00.298310+0000","last_undegraded":"2024-03-06T07:22:00.298310+0000","last_fullsized":"2024-03-06T07:22:00.298310+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:06:11.334051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298716+0000","last_change":"2024-03-06T07:21:57.956339+0000","last_active":"2024-03-06T07:22:00.298716+0000","last_peered":"2024-03-06T07:22:00.298716+0000","last_clean":"2024-03-06T07:22:00.298716+0000","last_became_active":"2024-03-06T07:21:57.956135+0000","last_became_peered":"2024-03-06T07:21:57.956135+0000","last_unstale":"2024-03-06T07:22:00.298716+0000","last_undegraded":"2024-03-06T07:22:00.298716+0000","last_fullsized":"2024-03-06T07:22:00.298716+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:01.159166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299011+0000","last_change":"2024-03-06T07:21:58.962391+0000","last_active":"2024-03-06T07:22:00.299011+0000","last_peered":"2024-03-06T07:22:00.299011+0000","last_clean":"2024-03-06T07:22:00.299011+0000","last_became_active":"2024-03-06T07:21:58.961887+0000","last_became_peered":"2024-03-06T07:21:58.961887+0000","last_unstale":"2024-03-06T07:22:00.299011+0000","last_undegraded":"2024-03-06T07:22:00.299011+0000","last_fullsized":"2024-03-06T07:22:00.299011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:20.758138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942138+0000","last_change":"2024-03-06T07:21:57.939474+0000","last_active":"2024-03-06T07:21:59.942138+0000","last_peered":"2024-03-06T07:21:59.942138+0000","last_clean":"2024-03-06T07:21:59.942138+0000","last_became_active":"2024-03-06T07:21:57.939053+0000","last_became_peered":"2024-03-06T07:21:57.939053+0000","last_unstale":"2024-03-06T07:21:59.942138+0000","last_undegraded":"2024-03-06T07:21:59.942138+0000","last_fullsized":"2024-03-06T07:21:59.942138+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:35.501272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942064+0000","last_change":"2024-03-06T07:21:58.958835+0000","last_active":"2024-03-06T07:21:59.942064+0000","last_peered":"2024-03-06T07:21:59.942064+0000","last_clean":"2024-03-06T07:21:59.942064+0000","last_became_active":"2024-03-06T07:21:58.958518+0000","last_became_peered":"2024-03-06T07:21:58.958518+0000","last_unstale":"2024-03-06T07:21:59.942064+0000","last_undegraded":"2024-03-06T07:21:59.942064+0000","last_fullsized":"2024-03-06T07:21:59.942064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:22.038204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298611+0000","last_change":"2024-03-06T07:21:57.940312+0000","last_active":"2024-03-06T07:22:00.298611+0000","last_peered":"2024-03-06T07:22:00.298611+0000","last_clean":"2024-03-06T07:22:00.298611+0000","last_became_active":"2024-03-06T07:21:57.940030+0000","last_became_peered":"2024-03-06T07:21:57.940030+0000","last_unstale":"2024-03-06T07:22:00.298611+0000","last_undegraded":"2024-03-06T07:22:00.298611+0000","last_fullsized":"2024-03-06T07:22:00.298611+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:35.501272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298350+0000","last_change":"2024-03-06T07:21:58.961737+0000","last_active":"2024-03-06T07:22:00.298350+0000","last_peered":"2024-03-06T07:22:00.298350+0000","last_clean":"2024-03-06T07:22:00.298350+0000","last_became_active":"2024-03-06T07:21:58.961365+0000","last_became_peered":"2024-03-06T07:21:58.961365+0000","last_unstale":"2024-03-06T07:22:00.298350+0000","last_undegraded":"2024-03-06T07:22:00.298350+0000","last_fullsized":"2024-03-06T07:22:00.298350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:44.889543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943032+0000","last_change":"2024-03-06T07:21:57.948442+0000","last_active":"2024-03-06T07:21:59.943032+0000","last_peered":"2024-03-06T07:21:59.943032+0000","last_clean":"2024-03-06T07:21:59.943032+0000","last_became_active":"2024-03-06T07:21:57.948170+0000","last_became_peered":"2024-03-06T07:21:57.948170+0000","last_unstale":"2024-03-06T07:21:59.943032+0000","last_undegraded":"2024-03-06T07:21:59.943032+0000","last_fullsized":"2024-03-06T07:21:59.943032+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:10.622090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298979+0000","last_change":"2024-03-06T07:21:58.961595+0000","last_active":"2024-03-06T07:22:00.298979+0000","last_peered":"2024-03-06T07:22:00.298979+0000","last_clean":"2024-03-06T07:22:00.298979+0000","last_became_active":"2024-03-06T07:21:58.961259+0000","last_became_peered":"2024-03-06T07:21:58.961259+0000","last_unstale":"2024-03-06T07:22:00.298979+0000","last_undegraded":"2024-03-06T07:22:00.298979+0000","last_fullsized":"2024-03-06T07:22:00.298979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:43.260641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298869+0000","last_change":"2024-03-06T07:21:57.942300+0000","last_active":"2024-03-06T07:22:00.298869+0000","last_peered":"2024-03-06T07:22:00.298869+0000","last_clean":"2024-03-06T07:22:00.298869+0000","last_became_active":"2024-03-06T07:21:57.942023+0000","last_became_peered":"2024-03-06T07:21:57.942023+0000","last_unstale":"2024-03-06T07:22:00.298869+0000","last_undegraded":"2024-03-06T07:22:00.298869+0000","last_fullsized":"2024-03-06T07:22:00.298869+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:07:58.548928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299225+0000","last_change":"2024-03-06T07:21:58.970895+0000","last_active":"2024-03-06T07:22:00.299225+0000","last_peered":"2024-03-06T07:22:00.299225+0000","last_clean":"2024-03-06T07:22:00.299225+0000","last_became_active":"2024-03-06T07:21:58.970418+0000","last_became_peered":"2024-03-06T07:21:58.970418+0000","last_unstale":"2024-03-06T07:22:00.299225+0000","last_undegraded":"2024-03-06T07:22:00.299225+0000","last_fullsized":"2024-03-06T07:22:00.299225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:19:55.013458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944014+0000","last_change":"2024-03-06T07:21:57.942325+0000","last_active":"2024-03-06T07:21:59.944014+0000","last_peered":"2024-03-06T07:21:59.944014+0000","last_clean":"2024-03-06T07:21:59.944014+0000","last_became_active":"2024-03-06T07:21:57.942045+0000","last_became_peered":"2024-03-06T07:21:57.942045+0000","last_unstale":"2024-03-06T07:21:59.944014+0000","last_undegraded":"2024-03-06T07:21:59.944014+0000","last_fullsized":"2024-03-06T07:21:59.944014+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:19.905010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298465+0000","last_change":"2024-03-06T07:21:58.967944+0000","last_active":"2024-03-06T07:22:00.298465+0000","last_peered":"2024-03-06T07:22:00.298465+0000","last_clean":"2024-03-06T07:22:00.298465+0000","last_became_active":"2024-03-06T07:21:58.967521+0000","last_became_peered":"2024-03-06T07:21:58.967521+0000","last_unstale":"2024-03-06T07:22:00.298465+0000","last_undegraded":"2024-03-06T07:22:00.298465+0000","last_fullsized":"2024-03-06T07:22:00.298465+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:43.260641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.297951+0000","last_change":"2024-03-06T07:21:57.947477+0000","last_active":"2024-03-06T07:22:00.297951+0000","last_peered":"2024-03-06T07:22:00.297951+0000","last_clean":"2024-03-06T07:22:00.297951+0000","last_became_active":"2024-03-06T07:21:57.946678+0000","last_became_peered":"2024-03-06T07:21:57.946678+0000","last_unstale":"2024-03-06T07:22:00.297951+0000","last_undegraded":"2024-03-06T07:22:00.297951+0000","last_fullsized":"2024-03-06T07:22:00.297951+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:13.736718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298718+0000","last_change":"2024-03-06T07:21:58.970473+0000","last_active":"2024-03-06T07:22:00.298718+0000","last_peered":"2024-03-06T07:22:00.298718+0000","last_clean":"2024-03-06T07:22:00.298718+0000","last_became_active":"2024-03-06T07:21:58.969810+0000","last_became_peered":"2024-03-06T07:21:58.969810+0000","last_unstale":"2024-03-06T07:22:00.298718+0000","last_undegraded":"2024-03-06T07:22:00.298718+0000","last_fullsized":"2024-03-06T07:22:00.298718+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:07.060045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298038+0000","last_change":"2024-03-06T07:21:47.553693+0000","last_active":"2024-03-06T07:22:00.298038+0000","last_peered":"2024-03-06T07:22:00.298038+0000","last_clean":"2024-03-06T07:22:00.298038+0000","last_became_active":"2024-03-06T07:21:47.552664+0000","last_became_peered":"2024-03-06T07:21:47.552664+0000","last_unstale":"2024-03-06T07:22:00.298038+0000","last_undegraded":"2024-03-06T07:22:00.298038+0000","last_fullsized":"2024-03-06T07:22:00.298038+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_clean_scrub_stamp":"2024-03-06T07:21:46.362028+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:02.916426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.297955+0000","last_change":"2024-03-06T07:21:57.949622+0000","last_active":"2024-03-06T07:22:00.297955+0000","last_peered":"2024-03-06T07:22:00.297955+0000","last_clean":"2024-03-06T07:22:00.297955+0000","last_became_active":"2024-03-06T07:21:57.948874+0000","last_became_peered":"2024-03-06T07:21:57.948874+0000","last_unstale":"2024-03-06T07:22:00.297955+0000","last_undegraded":"2024-03-06T07:22:00.297955+0000","last_fullsized":"2024-03-06T07:22:00.297955+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:35.501272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298026+0000","last_change":"2024-03-06T07:21:58.968667+0000","last_active":"2024-03-06T07:22:00.298026+0000","last_peered":"2024-03-06T07:22:00.298026+0000","last_clean":"2024-03-06T07:22:00.298026+0000","last_became_active":"2024-03-06T07:21:58.968213+0000","last_became_peered":"2024-03-06T07:21:58.968213+0000","last_unstale":"2024-03-06T07:22:00.298026+0000","last_undegraded":"2024-03-06T07:22:00.298026+0000","last_fullsized":"2024-03-06T07:22:00.298026+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:22.038204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.297808+0000","last_change":"2024-03-06T07:21:57.951697+0000","last_active":"2024-03-06T07:22:00.297808+0000","last_peered":"2024-03-06T07:22:00.297808+0000","last_clean":"2024-03-06T07:22:00.297808+0000","last_became_active":"2024-03-06T07:21:57.951421+0000","last_became_peered":"2024-03-06T07:21:57.951421+0000","last_unstale":"2024-03-06T07:22:00.297808+0000","last_undegraded":"2024-03-06T07:22:00.297808+0000","last_fullsized":"2024-03-06T07:22:00.297808+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:51.781285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.297721+0000","last_change":"2024-03-06T07:21:58.948562+0000","last_active":"2024-03-06T07:22:00.297721+0000","last_peered":"2024-03-06T07:22:00.297721+0000","last_clean":"2024-03-06T07:22:00.297721+0000","last_became_active":"2024-03-06T07:21:58.948249+0000","last_became_peered":"2024-03-06T07:21:58.948249+0000","last_unstale":"2024-03-06T07:22:00.297721+0000","last_undegraded":"2024-03-06T07:22:00.297721+0000","last_fullsized":"2024-03-06T07:22:00.297721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:53.360751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298409+0000","last_change":"2024-03-06T07:21:57.945890+0000","last_active":"2024-03-06T07:22:00.298409+0000","last_peered":"2024-03-06T07:22:00.298409+0000","last_clean":"2024-03-06T07:22:00.298409+0000","last_became_active":"2024-03-06T07:21:57.945527+0000","last_became_peered":"2024-03-06T07:21:57.945527+0000","last_unstale":"2024-03-06T07:22:00.298409+0000","last_undegraded":"2024-03-06T07:22:00.298409+0000","last_fullsized":"2024-03-06T07:22:00.298409+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:31.179338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943753+0000","last_change":"2024-03-06T07:21:58.952436+0000","last_active":"2024-03-06T07:21:59.943753+0000","last_peered":"2024-03-06T07:21:59.943753+0000","last_clean":"2024-03-06T07:21:59.943753+0000","last_became_active":"2024-03-06T07:21:58.952135+0000","last_became_peered":"2024-03-06T07:21:58.952135+0000","last_unstale":"2024-03-06T07:21:59.943753+0000","last_undegraded":"2024-03-06T07:21:59.943753+0000","last_fullsized":"2024-03-06T07:21:59.943753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:59.261197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298778+0000","last_change":"2024-03-06T07:21:57.951597+0000","last_active":"2024-03-06T07:22:00.298778+0000","last_peered":"2024-03-06T07:22:00.298778+0000","last_clean":"2024-03-06T07:22:00.298778+0000","last_became_active":"2024-03-06T07:21:57.951299+0000","last_became_peered":"2024-03-06T07:21:57.951299+0000","last_unstale":"2024-03-06T07:22:00.298778+0000","last_undegraded":"2024-03-06T07:22:00.298778+0000","last_fullsized":"2024-03-06T07:22:00.298778+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:19.847331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943217+0000","last_change":"2024-03-06T07:21:58.951633+0000","last_active":"2024-03-06T07:21:59.943217+0000","last_peered":"2024-03-06T07:21:59.943217+0000","last_clean":"2024-03-06T07:21:59.943217+0000","last_became_active":"2024-03-06T07:21:58.951315+0000","last_became_peered":"2024-03-06T07:21:58.951315+0000","last_unstale":"2024-03-06T07:21:59.943217+0000","last_undegraded":"2024-03-06T07:21:59.943217+0000","last_fullsized":"2024-03-06T07:21:59.943217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:20.851108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943826+0000","last_change":"2024-03-06T07:21:57.945253+0000","last_active":"2024-03-06T07:21:59.943826+0000","last_peered":"2024-03-06T07:21:59.943826+0000","last_clean":"2024-03-06T07:21:59.943826+0000","last_became_active":"2024-03-06T07:21:57.944983+0000","last_became_peered":"2024-03-06T07:21:57.944983+0000","last_unstale":"2024-03-06T07:21:59.943826+0000","last_undegraded":"2024-03-06T07:21:59.943826+0000","last_fullsized":"2024-03-06T07:21:59.943826+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:07:58.548928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298511+0000","last_change":"2024-03-06T07:21:58.964580+0000","last_active":"2024-03-06T07:22:00.298511+0000","last_peered":"2024-03-06T07:22:00.298511+0000","last_clean":"2024-03-06T07:22:00.298511+0000","last_became_active":"2024-03-06T07:21:58.964272+0000","last_became_peered":"2024-03-06T07:21:58.964272+0000","last_unstale":"2024-03-06T07:22:00.298511+0000","last_undegraded":"2024-03-06T07:22:00.298511+0000","last_fullsized":"2024-03-06T07:22:00.298511+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:02:11.315323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298902+0000","last_change":"2024-03-06T07:21:57.952261+0000","last_active":"2024-03-06T07:22:00.298902+0000","last_peered":"2024-03-06T07:22:00.298902+0000","last_clean":"2024-03-06T07:22:00.298902+0000","last_became_active":"2024-03-06T07:21:57.951991+0000","last_became_peered":"2024-03-06T07:21:57.951991+0000","last_unstale":"2024-03-06T07:22:00.298902+0000","last_undegraded":"2024-03-06T07:22:00.298902+0000","last_fullsized":"2024-03-06T07:22:00.298902+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:52.356974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943810+0000","last_change":"2024-03-06T07:21:58.955953+0000","last_active":"2024-03-06T07:21:59.943810+0000","last_peered":"2024-03-06T07:21:59.943810+0000","last_clean":"2024-03-06T07:21:59.943810+0000","last_became_active":"2024-03-06T07:21:58.955634+0000","last_became_peered":"2024-03-06T07:21:58.955634+0000","last_unstale":"2024-03-06T07:21:59.943810+0000","last_undegraded":"2024-03-06T07:21:59.943810+0000","last_fullsized":"2024-03-06T07:21:59.943810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:12.378982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.342768+0000","last_change":"2024-03-06T07:21:57.943165+0000","last_active":"2024-03-06T07:22:00.342768+0000","last_peered":"2024-03-06T07:22:00.342768+0000","last_clean":"2024-03-06T07:22:00.342768+0000","last_became_active":"2024-03-06T07:21:57.942914+0000","last_became_peered":"2024-03-06T07:21:57.942914+0000","last_unstale":"2024-03-06T07:22:00.342768+0000","last_undegraded":"2024-03-06T07:22:00.342768+0000","last_fullsized":"2024-03-06T07:22:00.342768+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:19.847331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298017+0000","last_change":"2024-03-06T07:21:58.960410+0000","last_active":"2024-03-06T07:22:00.298017+0000","last_peered":"2024-03-06T07:22:00.298017+0000","last_clean":"2024-03-06T07:22:00.298017+0000","last_became_active":"2024-03-06T07:21:58.960121+0000","last_became_peered":"2024-03-06T07:21:58.960121+0000","last_unstale":"2024-03-06T07:22:00.298017+0000","last_undegraded":"2024-03-06T07:22:00.298017+0000","last_fullsized":"2024-03-06T07:22:00.298017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:55.294652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942389+0000","last_change":"2024-03-06T07:21:57.944106+0000","last_active":"2024-03-06T07:21:59.942389+0000","last_peered":"2024-03-06T07:21:59.942389+0000","last_clean":"2024-03-06T07:21:59.942389+0000","last_became_active":"2024-03-06T07:21:57.943815+0000","last_became_peered":"2024-03-06T07:21:57.943815+0000","last_unstale":"2024-03-06T07:21:59.942389+0000","last_undegraded":"2024-03-06T07:21:59.942389+0000","last_fullsized":"2024-03-06T07:21:59.942389+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:13.736718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298012+0000","last_change":"2024-03-06T07:21:58.951840+0000","last_active":"2024-03-06T07:22:00.298012+0000","last_peered":"2024-03-06T07:22:00.298012+0000","last_clean":"2024-03-06T07:22:00.298012+0000","last_became_active":"2024-03-06T07:21:58.951594+0000","last_became_peered":"2024-03-06T07:21:58.951594+0000","last_unstale":"2024-03-06T07:22:00.298012+0000","last_undegraded":"2024-03-06T07:22:00.298012+0000","last_fullsized":"2024-03-06T07:22:00.298012+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:34.528149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.297795+0000","last_change":"2024-03-06T07:21:57.951916+0000","last_active":"2024-03-06T07:22:00.297795+0000","last_peered":"2024-03-06T07:22:00.297795+0000","last_clean":"2024-03-06T07:22:00.297795+0000","last_became_active":"2024-03-06T07:21:57.951647+0000","last_became_peered":"2024-03-06T07:21:57.951647+0000","last_unstale":"2024-03-06T07:22:00.297795+0000","last_undegraded":"2024-03-06T07:22:00.297795+0000","last_fullsized":"2024-03-06T07:22:00.297795+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:13.736718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942859+0000","last_change":"2024-03-06T07:21:58.953147+0000","last_active":"2024-03-06T07:21:59.942859+0000","last_peered":"2024-03-06T07:21:59.942859+0000","last_clean":"2024-03-06T07:21:59.942859+0000","last_became_active":"2024-03-06T07:21:58.952862+0000","last_became_peered":"2024-03-06T07:21:58.952862+0000","last_unstale":"2024-03-06T07:21:59.942859+0000","last_undegraded":"2024-03-06T07:21:59.942859+0000","last_fullsized":"2024-03-06T07:21:59.942859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:14.740494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943795+0000","last_change":"2024-03-06T07:21:57.948593+0000","last_active":"2024-03-06T07:21:59.943795+0000","last_peered":"2024-03-06T07:21:59.943795+0000","last_clean":"2024-03-06T07:21:59.943795+0000","last_became_active":"2024-03-06T07:21:57.948302+0000","last_became_peered":"2024-03-06T07:21:57.948302+0000","last_unstale":"2024-03-06T07:21:59.943795+0000","last_undegraded":"2024-03-06T07:21:59.943795+0000","last_fullsized":"2024-03-06T07:21:59.943795+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:13.736718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298923+0000","last_change":"2024-03-06T07:21:58.953065+0000","last_active":"2024-03-06T07:22:00.298923+0000","last_peered":"2024-03-06T07:22:00.298923+0000","last_clean":"2024-03-06T07:22:00.298923+0000","last_became_active":"2024-03-06T07:21:58.952790+0000","last_became_peered":"2024-03-06T07:21:58.952790+0000","last_unstale":"2024-03-06T07:22:00.298923+0000","last_undegraded":"2024-03-06T07:22:00.298923+0000","last_fullsized":"2024-03-06T07:22:00.298923+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:30:54.760925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943752+0000","last_change":"2024-03-06T07:21:57.945134+0000","last_active":"2024-03-06T07:21:59.943752+0000","last_peered":"2024-03-06T07:21:59.943752+0000","last_clean":"2024-03-06T07:21:59.943752+0000","last_became_active":"2024-03-06T07:21:57.944858+0000","last_became_peered":"2024-03-06T07:21:57.944858+0000","last_unstale":"2024-03-06T07:21:59.943752+0000","last_undegraded":"2024-03-06T07:21:59.943752+0000","last_fullsized":"2024-03-06T07:21:59.943752+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:51.781285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299349+0000","last_change":"2024-03-06T07:21:58.964930+0000","last_active":"2024-03-06T07:22:00.299349+0000","last_peered":"2024-03-06T07:22:00.299349+0000","last_clean":"2024-03-06T07:22:00.299349+0000","last_became_active":"2024-03-06T07:21:58.964474+0000","last_became_peered":"2024-03-06T07:21:58.964474+0000","last_unstale":"2024-03-06T07:22:00.299349+0000","last_undegraded":"2024-03-06T07:22:00.299349+0000","last_fullsized":"2024-03-06T07:22:00.299349+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:44.889543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299058+0000","last_change":"2024-03-06T07:21:57.953248+0000","last_active":"2024-03-06T07:22:00.299058+0000","last_peered":"2024-03-06T07:22:00.299058+0000","last_clean":"2024-03-06T07:22:00.299058+0000","last_became_active":"2024-03-06T07:21:57.952969+0000","last_became_peered":"2024-03-06T07:21:57.952969+0000","last_unstale":"2024-03-06T07:22:00.299058+0000","last_undegraded":"2024-03-06T07:22:00.299058+0000","last_fullsized":"2024-03-06T07:22:00.299058+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:19.905010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942240+0000","last_change":"2024-03-06T07:21:58.954261+0000","last_active":"2024-03-06T07:21:59.942240+0000","last_peered":"2024-03-06T07:21:59.942240+0000","last_clean":"2024-03-06T07:21:59.942240+0000","last_became_active":"2024-03-06T07:21:58.953584+0000","last_became_peered":"2024-03-06T07:21:58.953584+0000","last_unstale":"2024-03-06T07:21:59.942240+0000","last_undegraded":"2024-03-06T07:21:59.942240+0000","last_fullsized":"2024-03-06T07:21:59.942240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:59.261197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298322+0000","last_change":"2024-03-06T07:21:58.959725+0000","last_active":"2024-03-06T07:22:00.298322+0000","last_peered":"2024-03-06T07:22:00.298322+0000","last_clean":"2024-03-06T07:22:00.298322+0000","last_became_active":"2024-03-06T07:21:58.959078+0000","last_became_peered":"2024-03-06T07:21:58.959078+0000","last_unstale":"2024-03-06T07:22:00.298322+0000","last_undegraded":"2024-03-06T07:22:00.298322+0000","last_fullsized":"2024-03-06T07:22:00.298322+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:44.889543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299074+0000","last_change":"2024-03-06T07:21:57.951758+0000","last_active":"2024-03-06T07:22:00.299074+0000","last_peered":"2024-03-06T07:22:00.299074+0000","last_clean":"2024-03-06T07:22:00.299074+0000","last_became_active":"2024-03-06T07:21:57.951452+0000","last_became_peered":"2024-03-06T07:21:57.951452+0000","last_unstale":"2024-03-06T07:22:00.299074+0000","last_undegraded":"2024-03-06T07:22:00.299074+0000","last_fullsized":"2024-03-06T07:22:00.299074+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:01.159166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298378+0000","last_change":"2024-03-06T07:21:58.961541+0000","last_active":"2024-03-06T07:22:00.298378+0000","last_peered":"2024-03-06T07:22:00.298378+0000","last_clean":"2024-03-06T07:22:00.298378+0000","last_became_active":"2024-03-06T07:21:58.961271+0000","last_became_peered":"2024-03-06T07:21:58.961271+0000","last_unstale":"2024-03-06T07:22:00.298378+0000","last_undegraded":"2024-03-06T07:22:00.298378+0000","last_fullsized":"2024-03-06T07:22:00.298378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:10:08.320367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298788+0000","last_change":"2024-03-06T07:21:57.947161+0000","last_active":"2024-03-06T07:22:00.298788+0000","last_peered":"2024-03-06T07:22:00.298788+0000","last_clean":"2024-03-06T07:22:00.298788+0000","last_became_active":"2024-03-06T07:21:57.946849+0000","last_became_peered":"2024-03-06T07:21:57.946849+0000","last_unstale":"2024-03-06T07:22:00.298788+0000","last_undegraded":"2024-03-06T07:22:00.298788+0000","last_fullsized":"2024-03-06T07:22:00.298788+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:19.847331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298661+0000","last_change":"2024-03-06T07:21:58.958814+0000","last_active":"2024-03-06T07:22:00.298661+0000","last_peered":"2024-03-06T07:22:00.298661+0000","last_clean":"2024-03-06T07:22:00.298661+0000","last_became_active":"2024-03-06T07:21:58.958550+0000","last_became_peered":"2024-03-06T07:21:58.958550+0000","last_unstale":"2024-03-06T07:22:00.298661+0000","last_undegraded":"2024-03-06T07:22:00.298661+0000","last_fullsized":"2024-03-06T07:22:00.298661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:36:02.122800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298388+0000","last_change":"2024-03-06T07:21:57.949031+0000","last_active":"2024-03-06T07:22:00.298388+0000","last_peered":"2024-03-06T07:22:00.298388+0000","last_clean":"2024-03-06T07:22:00.298388+0000","last_became_active":"2024-03-06T07:21:57.948741+0000","last_became_peered":"2024-03-06T07:21:57.948741+0000","last_unstale":"2024-03-06T07:22:00.298388+0000","last_undegraded":"2024-03-06T07:22:00.298388+0000","last_fullsized":"2024-03-06T07:22:00.298388+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:35.501272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298230+0000","last_change":"2024-03-06T07:21:58.957355+0000","last_active":"2024-03-06T07:22:00.298230+0000","last_peered":"2024-03-06T07:22:00.298230+0000","last_clean":"2024-03-06T07:22:00.298230+0000","last_became_active":"2024-03-06T07:21:58.956672+0000","last_became_peered":"2024-03-06T07:21:58.956672+0000","last_unstale":"2024-03-06T07:22:00.298230+0000","last_undegraded":"2024-03-06T07:22:00.298230+0000","last_fullsized":"2024-03-06T07:22:00.298230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:59.261197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298955+0000","last_change":"2024-03-06T07:21:57.951293+0000","last_active":"2024-03-06T07:22:00.298955+0000","last_peered":"2024-03-06T07:22:00.298955+0000","last_clean":"2024-03-06T07:22:00.298955+0000","last_became_active":"2024-03-06T07:21:57.950995+0000","last_became_peered":"2024-03-06T07:21:57.950995+0000","last_unstale":"2024-03-06T07:22:00.298955+0000","last_undegraded":"2024-03-06T07:22:00.298955+0000","last_fullsized":"2024-03-06T07:22:00.298955+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:10.622090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:06.695 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:22:06.695 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-06T07:22:06.695 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-06T07:22:06.695 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-06T07:22:06.696 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-06T07:22:06.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.834+0000 7f372ec6f640 1 Processor -- start 2024-03-06T07:22:06.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.834+0000 7f372ec6f640 1 -- start start 2024-03-06T07:22:06.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.835+0000 7f372ec6f640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f37280d00c0 0x7f37280d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.835+0000 7f372ec6f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f372806cf20 0x7f372806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.835+0000 7f372ec6f640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37280d09e0 con 0x7f37280cf850 2024-03-06T07:22:06.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.835+0000 7f372ec6f640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f372806ca90 con 0x7f37280d00c0 2024-03-06T07:22:06.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.835+0000 7f372ec6f640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f372815e380 con 0x7f372806cf20 2024-03-06T07:22:06.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.835+0000 7f372d20c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f372806cf20 0x7f372806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.835+0000 7f372ca0b640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f37280d00c0 0x7f37280d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.835+0000 7f372d20c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f372806cf20 0x7f372806d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:59080/0 (socket says 172.21.15.83:59080) 2024-03-06T07:22:06.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.835+0000 7f372d20c640 1 -- 172.21.15.83:0/1001996080 learned_addr learned my addr 172.21.15.83:0/1001996080 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:06.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.835+0000 7f3727fff640 1 --1- 172.21.15.83:0/1001996080 >> v1:172.21.15.183:6789/0 conn(0x7f37280cf850 0x7f37280cd3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:37828/0 (socket says 172.21.15.83:37828) 2024-03-06T07:22:06.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.836+0000 7f372ca0b640 1 -- 172.21.15.83:0/1001996080 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f372806cf20 msgr2=0x7f372806d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:06.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.836+0000 7f372ca0b640 1 --2- 172.21.15.83:0/1001996080 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f372806cf20 0x7f372806d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.836+0000 7f372ca0b640 1 -- 172.21.15.83:0/1001996080 >> v1:172.21.15.183:6789/0 conn(0x7f37280cf850 legacy=0x7f37280cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.836+0000 7f372ca0b640 1 -- 172.21.15.83:0/1001996080 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37280cdc10 con 0x7f37280d00c0 2024-03-06T07:22:06.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.836+0000 7f372d20c640 1 --2- 172.21.15.83:0/1001996080 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f372806cf20 0x7f372806d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:06.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.837+0000 7f372ca0b640 1 --2- 172.21.15.83:0/1001996080 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f37280d00c0 0x7f37280d04a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f37100097a0 tx=0x7f371002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=387322f53c2c78ab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.837+0000 7f37277fe640 1 -- 172.21.15.83:0/1001996080 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3710035020 con 0x7f37280d00c0 2024-03-06T07:22:06.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.837+0000 7f37277fe640 1 -- 172.21.15.83:0/1001996080 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3710031830 con 0x7f37280d00c0 2024-03-06T07:22:06.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.837+0000 7f37277fe640 1 -- 172.21.15.83:0/1001996080 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3710031c00 con 0x7f37280d00c0 2024-03-06T07:22:06.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.837+0000 7f372ec6f640 1 -- 172.21.15.83:0/1001996080 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f37280d00c0 msgr2=0x7f37280d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.837+0000 7f372ec6f640 1 --2- 172.21.15.83:0/1001996080 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f37280d00c0 0x7f37280d04a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f37100097a0 tx=0x7f371002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.837+0000 7f372ec6f640 1 -- 172.21.15.83:0/1001996080 shutdown_connections 2024-03-06T07:22:06.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.837+0000 7f372ec6f640 1 --2- 172.21.15.83:0/1001996080 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f372806cf20 0x7f372806d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.837+0000 7f372ec6f640 1 --2- 172.21.15.83:0/1001996080 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f37280d00c0 0x7f37280d04a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.837+0000 7f372ec6f640 1 -- 172.21.15.83:0/1001996080 >> 172.21.15.83:0/1001996080 conn(0x7f37280b35a0 msgr2=0x7f37280b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:06.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.837+0000 7f372ec6f640 1 -- 172.21.15.83:0/1001996080 shutdown_connections 2024-03-06T07:22:06.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.837+0000 7f372ec6f640 1 -- 172.21.15.83:0/1001996080 wait complete. 2024-03-06T07:22:06.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.838+0000 7f372ec6f640 1 Processor -- start 2024-03-06T07:22:06.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.838+0000 7f372ec6f640 1 -- start start 2024-03-06T07:22:06.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.839+0000 7f372ec6f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f372806cf20 0x7f37280c7710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.839+0000 7f372ec6f640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f37280cf850 0x7f3728152df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.839+0000 7f372ec6f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37280d00c0 0x7f372815e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.839+0000 7f372ec6f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372806c720 con 0x7f37280d00c0 2024-03-06T07:22:06.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.839+0000 7f372ec6f640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372806c440 con 0x7f37280cf850 2024-03-06T07:22:06.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.839+0000 7f372ec6f640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37280b1030 con 0x7f372806cf20 2024-03-06T07:22:06.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.839+0000 7f372d20c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37280d00c0 0x7f372815e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.839+0000 7f372d20c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37280d00c0 0x7f372815e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49302/0 (socket says 172.21.15.83:49302) 2024-03-06T07:22:06.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.839+0000 7f372d20c640 1 -- 172.21.15.83:0/3387275432 learned_addr learned my addr 172.21.15.83:0/3387275432 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:06.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.839+0000 7f372ca0b640 1 --2- 172.21.15.83:0/3387275432 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f372806cf20 0x7f37280c7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.839+0000 7f3727fff640 1 --2- 172.21.15.83:0/3387275432 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f37280cf850 0x7f3728152df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.840+0000 7f372d20c640 1 -- 172.21.15.83:0/3387275432 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f372806cf20 msgr2=0x7f37280c7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.840+0000 7f372d20c640 1 --2- 172.21.15.83:0/3387275432 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f372806cf20 0x7f37280c7710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.840+0000 7f372d20c640 1 -- 172.21.15.83:0/3387275432 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f37280cf850 msgr2=0x7f3728152df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.840+0000 7f372d20c640 1 --2- 172.21.15.83:0/3387275432 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f37280cf850 0x7f3728152df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.840+0000 7f372d20c640 1 -- 172.21.15.83:0/3387275432 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3728153330 con 0x7f37280d00c0 2024-03-06T07:22:06.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.840+0000 7f372d20c640 1 --2- 172.21.15.83:0/3387275432 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37280d00c0 0x7f372815e590 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f372000efc0 tx=0x7f372000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.840+0000 7f37257fa640 1 -- 172.21.15.83:0/3387275432 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f372000cb80 con 0x7f37280d00c0 2024-03-06T07:22:06.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.840+0000 7f37257fa640 1 -- 172.21.15.83:0/3387275432 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f372000cd20 con 0x7f37280d00c0 2024-03-06T07:22:06.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.840+0000 7f372ec6f640 1 -- 172.21.15.83:0/3387275432 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3728161160 con 0x7f37280d00c0 2024-03-06T07:22:06.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.840+0000 7f37257fa640 1 -- 172.21.15.83:0/3387275432 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3720003bb0 con 0x7f37280d00c0 2024-03-06T07:22:06.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.840+0000 7f372ec6f640 1 -- 172.21.15.83:0/3387275432 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37280c6950 con 0x7f37280d00c0 2024-03-06T07:22:06.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.841+0000 7f372ec6f640 1 -- 172.21.15.83:0/3387275432 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f372804f330 con 0x7f37280d00c0 2024-03-06T07:22:06.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.841+0000 7f37257fa640 1 -- 172.21.15.83:0/3387275432 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f372001b030 con 0x7f37280d00c0 2024-03-06T07:22:06.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.842+0000 7f37257fa640 1 --2- 172.21.15.83:0/3387275432 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f370806b500 0x7f370806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.842+0000 7f372ca0b640 1 --2- 172.21.15.83:0/3387275432 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f370806b500 0x7f370806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.842+0000 7f37257fa640 1 -- 172.21.15.83:0/3387275432 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f372008a410 con 0x7f37280d00c0 2024-03-06T07:22:06.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.842+0000 7f372ca0b640 1 --2- 172.21.15.83:0/3387275432 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f370806b500 0x7f370806d9c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3710036000 tx=0x7f3710008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:06.846+0000 7f37257fa640 1 -- 172.21.15.83:0/3387275432 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3720022020 con 0x7f37280d00c0 2024-03-06T07:22:07.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.081+0000 7f372ec6f640 1 -- 172.21.15.83:0/3387275432 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f3728153520 con 0x7f37280d00c0 2024-03-06T07:22:07.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.084+0000 7f37257fa640 1 -- 172.21.15.83:0/3387275432 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171785 (secure 0 0 0) 0x7f3720010040 con 0x7f37280d00c0 2024-03-06T07:22:07.088 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:22:07.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.087+0000 7f372ec6f640 1 -- 172.21.15.83:0/3387275432 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f370806b500 msgr2=0x7f370806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.087+0000 7f372ec6f640 1 --2- 172.21.15.83:0/3387275432 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f370806b500 0x7f370806d9c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3710036000 tx=0x7f3710008030 comp rx=0 tx=0).stop 2024-03-06T07:22:07.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.087+0000 7f372ec6f640 1 -- 172.21.15.83:0/3387275432 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37280d00c0 msgr2=0x7f372815e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.087+0000 7f372ec6f640 1 --2- 172.21.15.83:0/3387275432 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37280d00c0 0x7f372815e590 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f372000efc0 tx=0x7f372000c490 comp rx=0 tx=0).stop 2024-03-06T07:22:07.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.087+0000 7f372ec6f640 1 -- 172.21.15.83:0/3387275432 shutdown_connections 2024-03-06T07:22:07.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.087+0000 7f372ec6f640 1 --2- 172.21.15.83:0/3387275432 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f370806b500 0x7f370806d9c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.087+0000 7f372ec6f640 1 --2- 172.21.15.83:0/3387275432 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37280d00c0 0x7f372815e590 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.087+0000 7f372ec6f640 1 --2- 172.21.15.83:0/3387275432 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f37280cf850 0x7f3728152df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.087+0000 7f372ec6f640 1 --2- 172.21.15.83:0/3387275432 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f372806cf20 0x7f37280c7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.087+0000 7f372ec6f640 1 -- 172.21.15.83:0/3387275432 >> 172.21.15.83:0/3387275432 conn(0x7f37280b35a0 msgr2=0x7f37280cc870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.087+0000 7f372ec6f640 1 -- 172.21.15.83:0/3387275432 shutdown_connections 2024-03-06T07:22:07.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.087+0000 7f372ec6f640 1 -- 172.21.15.83:0/3387275432 wait complete. 2024-03-06T07:22:07.102 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":5,"flags":0,"active_gid":4110,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":2272295342},{"type":"v1","addr":"172.21.15.83:6833","nonce":2272295342}]},"active_addr":"172.21.15.83:6833/2272295342","active_change":"2024-03-06T07:21:38.310905+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4102,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":3291978586}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":3744162245}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":3800443316}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":3552029378}]}]} 2024-03-06T07:22:07.106 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-06T07:22:07.106 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-06T07:22:07.106 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:07.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.207+0000 7f4f9ad7a640 1 Processor -- start 2024-03-06T07:22:07.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.207+0000 7f4f9ad7a640 1 -- start start 2024-03-06T07:22:07.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.208+0000 7f4f9ad7a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f4f9415bb20 0x7f4f940d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.208+0000 7f4f9ad7a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4f940cfff0 0x7f4f940d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.208+0000 7f4f9ad7a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4f940ce4b0 0x7f4f940ce920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.208+0000 7f4f9ad7a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f94161980 con 0x7f4f940cfff0 2024-03-06T07:22:07.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.208+0000 7f4f9ad7a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f94161c60 con 0x7f4f9415bb20 2024-03-06T07:22:07.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.208+0000 7f4f9ad7a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f94161f40 con 0x7f4f940ce4b0 2024-03-06T07:22:07.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.208+0000 7f4f93fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4f940cfff0 0x7f4f940d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.208+0000 7f4f93fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4f940cfff0 0x7f4f940d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49318/0 (socket says 172.21.15.83:49318) 2024-03-06T07:22:07.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.208+0000 7f4f93fff640 1 -- 172.21.15.83:0/3220207720 learned_addr learned my addr 172.21.15.83:0/3220207720 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:07.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.209+0000 7f4f93fff640 1 -- 172.21.15.83:0/3220207720 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4f940ce4b0 msgr2=0x7f4f940ce920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:07.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.209+0000 7f4f93fff640 1 --2- 172.21.15.83:0/3220207720 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4f940ce4b0 0x7f4f940ce920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.209+0000 7f4f93fff640 1 -- 172.21.15.83:0/3220207720 >> v2:172.21.15.183:3300/0 conn(0x7f4f9415bb20 msgr2=0x7f4f940d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:07.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.209+0000 7f4f98b16640 1 --2- 172.21.15.83:0/3220207720 >> v2:172.21.15.183:3300/0 conn(0x7f4f9415bb20 0x7f4f940d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.209+0000 7f4f93fff640 1 --2- 172.21.15.83:0/3220207720 >> v2:172.21.15.183:3300/0 conn(0x7f4f9415bb20 0x7f4f940d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.209+0000 7f4f93fff640 1 -- 172.21.15.83:0/3220207720 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f94162220 con 0x7f4f940cfff0 2024-03-06T07:22:07.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.209+0000 7f4f98b16640 1 --2- 172.21.15.83:0/3220207720 >> v2:172.21.15.183:3300/0 conn(0x7f4f9415bb20 0x7f4f940d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:07.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.209+0000 7f4f93fff640 1 --2- 172.21.15.83:0/3220207720 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f940cfff0 0x7f4f940d04a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4f940d2bc0 tx=0x7f4f8002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c71722ba6399ead server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.209+0000 7f4f937fe640 1 -- 172.21.15.83:0/3220207720 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f80035020 con 0x7f4f940cfff0 2024-03-06T07:22:07.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.209+0000 7f4f937fe640 1 -- 172.21.15.83:0/3220207720 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f80031870 con 0x7f4f940cfff0 2024-03-06T07:22:07.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.209+0000 7f4f937fe640 1 -- 172.21.15.83:0/3220207720 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f80031c40 con 0x7f4f940cfff0 2024-03-06T07:22:07.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.210+0000 7f4f9ad7a640 1 -- 172.21.15.83:0/3220207720 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f940cfff0 msgr2=0x7f4f940d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.210+0000 7f4f9ad7a640 1 --2- 172.21.15.83:0/3220207720 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f940cfff0 0x7f4f940d04a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4f940d2bc0 tx=0x7f4f8002b910 comp rx=0 tx=0).stop 2024-03-06T07:22:07.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.210+0000 7f4f9ad7a640 1 -- 172.21.15.83:0/3220207720 shutdown_connections 2024-03-06T07:22:07.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.210+0000 7f4f9ad7a640 1 --2- 172.21.15.83:0/3220207720 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4f940ce4b0 0x7f4f940ce920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.210+0000 7f4f9ad7a640 1 --2- 172.21.15.83:0/3220207720 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f940cfff0 0x7f4f940d04a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.210+0000 7f4f9ad7a640 1 --2- 172.21.15.83:0/3220207720 >> v2:172.21.15.183:3300/0 conn(0x7f4f9415bb20 0x7f4f940d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.210+0000 7f4f9ad7a640 1 -- 172.21.15.83:0/3220207720 >> 172.21.15.83:0/3220207720 conn(0x7f4f9409a9f0 msgr2=0x7f4f9409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.210+0000 7f4f9ad7a640 1 -- 172.21.15.83:0/3220207720 shutdown_connections 2024-03-06T07:22:07.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.210+0000 7f4f9ad7a640 1 -- 172.21.15.83:0/3220207720 wait complete. 2024-03-06T07:22:07.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.211+0000 7f4f9ad7a640 1 Processor -- start 2024-03-06T07:22:07.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.211+0000 7f4f9ad7a640 1 -- start start 2024-03-06T07:22:07.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.212+0000 7f4f9ad7a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f940ce4b0 0x7f4f9414e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.212+0000 7f4f9ad7a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4f940cfff0 0x7f4f941f7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.212+0000 7f4f9ad7a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4f9415bb20 0x7f4f9406ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.212+0000 7f4f9ad7a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f940d24b0 con 0x7f4f940ce4b0 2024-03-06T07:22:07.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.212+0000 7f4f9ad7a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f940d2330 con 0x7f4f9415bb20 2024-03-06T07:22:07.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.212+0000 7f4f9ad7a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f940d1980 con 0x7f4f940cfff0 2024-03-06T07:22:07.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.212+0000 7f4f98b16640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f940ce4b0 0x7f4f9414e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.212+0000 7f4f98b16640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f940ce4b0 0x7f4f9414e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49324/0 (socket says 172.21.15.83:49324) 2024-03-06T07:22:07.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.212+0000 7f4f98b16640 1 -- 172.21.15.83:0/3256221061 learned_addr learned my addr 172.21.15.83:0/3256221061 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:07.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.212+0000 7f4f98b16640 1 -- 172.21.15.83:0/3256221061 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4f940cfff0 msgr2=0x7f4f941f7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.212+0000 7f4f98b16640 1 --2- 172.21.15.83:0/3256221061 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4f940cfff0 0x7f4f941f7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.212+0000 7f4f98b16640 1 -- 172.21.15.83:0/3256221061 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4f9415bb20 msgr2=0x7f4f9406ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.212+0000 7f4f98b16640 1 --2- 172.21.15.83:0/3256221061 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4f9415bb20 0x7f4f9406ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.212+0000 7f4f98b16640 1 -- 172.21.15.83:0/3256221061 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f941f7f70 con 0x7f4f940ce4b0 2024-03-06T07:22:07.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.212+0000 7f4f98b16640 1 --2- 172.21.15.83:0/3256221061 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f940ce4b0 0x7f4f9414e140 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4f8800c820 tx=0x7f4f8800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.213+0000 7f4f917fa640 1 -- 172.21.15.83:0/3256221061 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f88015070 con 0x7f4f940ce4b0 2024-03-06T07:22:07.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.213+0000 7f4f917fa640 1 -- 172.21.15.83:0/3256221061 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f88007a30 con 0x7f4f940ce4b0 2024-03-06T07:22:07.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.213+0000 7f4f9ad7a640 1 -- 172.21.15.83:0/3256221061 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f941f8250 con 0x7f4f940ce4b0 2024-03-06T07:22:07.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.213+0000 7f4f917fa640 1 -- 172.21.15.83:0/3256221061 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f88007de0 con 0x7f4f940ce4b0 2024-03-06T07:22:07.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.213+0000 7f4f9ad7a640 1 -- 172.21.15.83:0/3256221061 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f940b5f30 con 0x7f4f940ce4b0 2024-03-06T07:22:07.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.214+0000 7f4f9ad7a640 1 -- 172.21.15.83:0/3256221061 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f94070750 con 0x7f4f940ce4b0 2024-03-06T07:22:07.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.214+0000 7f4f917fa640 1 -- 172.21.15.83:0/3256221061 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4f88018070 con 0x7f4f940ce4b0 2024-03-06T07:22:07.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.214+0000 7f4f917fa640 1 --2- 172.21.15.83:0/3256221061 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4f7806b4d0 0x7f4f7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.215+0000 7f4f917fa640 1 -- 172.21.15.83:0/3256221061 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4f8808a110 con 0x7f4f940ce4b0 2024-03-06T07:22:07.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.215+0000 7f4f93fff640 1 --2- 172.21.15.83:0/3256221061 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4f7806b4d0 0x7f4f7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.215+0000 7f4f93fff640 1 --2- 172.21.15.83:0/3256221061 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4f7806b4d0 0x7f4f7806d990 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4f80036000 tx=0x7f4f80030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.219+0000 7f4f917fa640 1 -- 172.21.15.83:0/3256221061 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4f94070750 con 0x7f4f940ce4b0 2024-03-06T07:22:07.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.403+0000 7f4f9ad7a640 1 -- 172.21.15.83:0/3256221061 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f94069e80 con 0x7f4f940ce4b0 2024-03-06T07:22:07.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.404+0000 7f4f917fa640 1 -- 172.21.15.83:0/3256221061 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16617 (secure 0 0 0) 0x7f4f94069e80 con 0x7f4f940ce4b0 2024-03-06T07:22:07.406 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:22:07.406 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":20,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:21:59.938137+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.318942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.254795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:07.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.406+0000 7f4f9ad7a640 1 -- 172.21.15.83:0/3256221061 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4f7806b4d0 msgr2=0x7f4f7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.406+0000 7f4f9ad7a640 1 --2- 172.21.15.83:0/3256221061 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4f7806b4d0 0x7f4f7806d990 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4f80036000 tx=0x7f4f80030000 comp rx=0 tx=0).stop 2024-03-06T07:22:07.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.406+0000 7f4f9ad7a640 1 -- 172.21.15.83:0/3256221061 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f940ce4b0 msgr2=0x7f4f9414e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.406+0000 7f4f9ad7a640 1 --2- 172.21.15.83:0/3256221061 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f940ce4b0 0x7f4f9414e140 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4f8800c820 tx=0x7f4f8800cce0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.406+0000 7f4f9ad7a640 1 -- 172.21.15.83:0/3256221061 shutdown_connections 2024-03-06T07:22:07.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.406+0000 7f4f9ad7a640 1 --2- 172.21.15.83:0/3256221061 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4f7806b4d0 0x7f4f7806d990 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.406+0000 7f4f9ad7a640 1 --2- 172.21.15.83:0/3256221061 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4f9415bb20 0x7f4f9406ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.406+0000 7f4f9ad7a640 1 --2- 172.21.15.83:0/3256221061 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4f940cfff0 0x7f4f941f7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.406+0000 7f4f9ad7a640 1 --2- 172.21.15.83:0/3256221061 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f940ce4b0 0x7f4f9414e140 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.406+0000 7f4f9ad7a640 1 -- 172.21.15.83:0/3256221061 >> 172.21.15.83:0/3256221061 conn(0x7f4f9409a9f0 msgr2=0x7f4f9409add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.406+0000 7f4f9ad7a640 1 -- 172.21.15.83:0/3256221061 shutdown_connections 2024-03-06T07:22:07.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.406+0000 7f4f9ad7a640 1 -- 172.21.15.83:0/3256221061 wait complete. 2024-03-06T07:22:07.419 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-06T07:22:07.419 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:07.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.518+0000 7f4fcfe8a640 1 Processor -- start 2024-03-06T07:22:07.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.518+0000 7f4fcfe8a640 1 -- start start 2024-03-06T07:22:07.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.519+0000 7f4fcfe8a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4fc815dcd0 0x7f4fc80d1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.519+0000 7f4fcfe8a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f4fc80727f0 0x7f4fc806afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.519+0000 7f4fcfe8a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fc8072c50 con 0x7f4fc806b4f0 2024-03-06T07:22:07.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.519+0000 7f4fcfe8a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fc80b3fe0 con 0x7f4fc80727f0 2024-03-06T07:22:07.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.519+0000 7f4fcfe8a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc80b5c30 con 0x7f4fc815dcd0 2024-03-06T07:22:07.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.519+0000 7f4fce427640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f4fc806b4f0 0x7f4fc80ad860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:42820/0 (socket says 172.21.15.83:42820) 2024-03-06T07:22:07.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.519+0000 7f4fce427640 1 -- 172.21.15.83:0/1731728286 learned_addr learned my addr 172.21.15.83:0/1731728286 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:07.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.520+0000 7f4fcd425640 1 --2- 172.21.15.83:0/1731728286 >> v2:172.21.15.183:3300/0 conn(0x7f4fc80727f0 0x7f4fc806afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.520+0000 7f4fcdc26640 1 --2- 172.21.15.83:0/1731728286 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4fc815dcd0 0x7f4fc80d1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.520+0000 7f4fccc24640 1 -- 172.21.15.83:0/1731728286 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1019639547 0 0) 0x7f4fc8072c50 con 0x7f4fc806b4f0 2024-03-06T07:22:07.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.520+0000 7f4fccc24640 1 -- 172.21.15.83:0/1731728286 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fb4003520 con 0x7f4fc806b4f0 2024-03-06T07:22:07.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.520+0000 7f4fccc24640 1 -- 172.21.15.83:0/1731728286 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4103598838 0 0) 0x7f4fb4003520 con 0x7f4fc806b4f0 2024-03-06T07:22:07.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.520+0000 7f4fccc24640 1 -- 172.21.15.83:0/1731728286 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4fc815dcd0 msgr2=0x7f4fc80d1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:07.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.520+0000 7f4fccc24640 1 --2- 172.21.15.83:0/1731728286 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4fc815dcd0 0x7f4fc80d1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.520+0000 7f4fccc24640 1 -- 172.21.15.83:0/1731728286 >> v2:172.21.15.183:3300/0 conn(0x7f4fc80727f0 msgr2=0x7f4fc806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:07.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.520+0000 7f4fccc24640 1 --2- 172.21.15.83:0/1731728286 >> v2:172.21.15.183:3300/0 conn(0x7f4fc80727f0 0x7f4fc806afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.520+0000 7f4fccc24640 1 -- 172.21.15.83:0/1731728286 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc80b5950 con 0x7f4fc806b4f0 2024-03-06T07:22:07.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.520+0000 7f4fccc24640 1 -- 172.21.15.83:0/1731728286 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f4fc00019f0 con 0x7f4fc806b4f0 2024-03-06T07:22:07.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.520+0000 7f4fccc24640 1 -- 172.21.15.83:0/1731728286 >> v1:172.21.15.83:6789/0 conn(0x7f4fc806b4f0 legacy=0x7f4fc80ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.520+0000 7f4fccc24640 1 --2- 172.21.15.83:0/1731728286 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4fb40039b0 0x7f4fb4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.520+0000 7f4fccc24640 1 --2- 172.21.15.83:0/1731728286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4fb4004800 0x7f4fb4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.520+0000 7f4fccc24640 1 --2- 172.21.15.83:0/1731728286 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4fc815dcd0 0x7f4fb40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.520+0000 7f4fccc24640 1 -- 172.21.15.83:0/1731728286 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc80b3fe0 con 0x7f4fb4004800 2024-03-06T07:22:07.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.520+0000 7f4fccc24640 1 -- 172.21.15.83:0/1731728286 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc80b5c30 con 0x7f4fb40039b0 2024-03-06T07:22:07.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.520+0000 7f4fccc24640 1 -- 172.21.15.83:0/1731728286 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb400b9c0 con 0x7f4fc815dcd0 2024-03-06T07:22:07.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.520+0000 7f4fcd425640 1 --2- 172.21.15.83:0/1731728286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4fb4004800 0x7f4fb4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.520+0000 7f4fcdc26640 1 --2- 172.21.15.83:0/1731728286 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4fb40039b0 0x7f4fb4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.520+0000 7f4fce427640 1 --2- 172.21.15.83:0/1731728286 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4fc815dcd0 0x7f4fb40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.520+0000 7f4fcd425640 1 -- 172.21.15.83:0/1731728286 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4fc815dcd0 msgr2=0x7f4fb40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:07.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.520+0000 7f4fcd425640 1 --2- 172.21.15.83:0/1731728286 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4fc815dcd0 0x7f4fb40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.520+0000 7f4fcd425640 1 -- 172.21.15.83:0/1731728286 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4fb40039b0 msgr2=0x7f4fb4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:07.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.520+0000 7f4fcd425640 1 --2- 172.21.15.83:0/1731728286 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4fb40039b0 0x7f4fb4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.520+0000 7f4fcd425640 1 -- 172.21.15.83:0/1731728286 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fb4003520 con 0x7f4fb4004800 2024-03-06T07:22:07.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.521+0000 7f4fcd425640 1 --2- 172.21.15.83:0/1731728286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4fb4004800 0x7f4fb4004c80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4fc4009af0 tx=0x7f4fb4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1b7f0821f4162c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.521+0000 7f4fccc24640 1 -- 172.21.15.83:0/1731728286 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fc4039070 con 0x7f4fb4004800 2024-03-06T07:22:07.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.521+0000 7f4fccc24640 1 -- 172.21.15.83:0/1731728286 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fc402c710 con 0x7f4fb4004800 2024-03-06T07:22:07.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.521+0000 7f4fccc24640 1 -- 172.21.15.83:0/1731728286 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fc4009690 con 0x7f4fb4004800 2024-03-06T07:22:07.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.522+0000 7f4fcfe8a640 1 -- 172.21.15.83:0/1731728286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4fb4004800 msgr2=0x7f4fb4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.522+0000 7f4fcfe8a640 1 --2- 172.21.15.83:0/1731728286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4fb4004800 0x7f4fb4004c80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4fc4009af0 tx=0x7f4fb4005830 comp rx=0 tx=0).stop 2024-03-06T07:22:07.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.522+0000 7f4fce427640 1 -- 172.21.15.83:0/1731728286 reap_dead start 2024-03-06T07:22:07.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.522+0000 7f4fcfe8a640 1 -- 172.21.15.83:0/1731728286 shutdown_connections 2024-03-06T07:22:07.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.522+0000 7f4fcfe8a640 1 -- 172.21.15.83:0/1731728286 >> 172.21.15.83:0/1731728286 conn(0x7f4fc806c0c0 msgr2=0x7f4fc806c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.522+0000 7f4fcfe8a640 1 -- 172.21.15.83:0/1731728286 shutdown_connections 2024-03-06T07:22:07.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.522+0000 7f4fcfe8a640 1 -- 172.21.15.83:0/1731728286 wait complete. 2024-03-06T07:22:07.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.524+0000 7f4fcfe8a640 1 Processor -- start 2024-03-06T07:22:07.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.524+0000 7f4fcfe8a640 1 -- start start 2024-03-06T07:22:07.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.525+0000 7f4fcfe8a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4fb4004800 0x7f4fc815e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.525+0000 7f4fcfe8a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4fc815ffc0 0x7f4fc814b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.525+0000 7f4fcfe8a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4fc806ea70 0x7f4fc806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.525+0000 7f4fcfe8a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc815e0b0 con 0x7f4fb4004800 2024-03-06T07:22:07.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.525+0000 7f4fcfe8a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc80b3d00 con 0x7f4fc815ffc0 2024-03-06T07:22:07.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.525+0000 7f4fcfe8a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc80afee0 con 0x7f4fc806ea70 2024-03-06T07:22:07.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.525+0000 7f4fcdc26640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4fb4004800 0x7f4fc815e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.525+0000 7f4fcdc26640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4fb4004800 0x7f4fc815e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49348/0 (socket says 172.21.15.83:49348) 2024-03-06T07:22:07.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.525+0000 7f4fcdc26640 1 -- 172.21.15.83:0/2546108390 learned_addr learned my addr 172.21.15.83:0/2546108390 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:07.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.525+0000 7f4fcd425640 1 --2- 172.21.15.83:0/2546108390 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4fc815ffc0 0x7f4fc814b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.525+0000 7f4fce427640 1 --2- 172.21.15.83:0/2546108390 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4fc806ea70 0x7f4fc806ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.525+0000 7f4fcdc26640 1 -- 172.21.15.83:0/2546108390 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4fc806ea70 msgr2=0x7f4fc806ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.525+0000 7f4fcdc26640 1 --2- 172.21.15.83:0/2546108390 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4fc806ea70 0x7f4fc806ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.525+0000 7f4fcdc26640 1 -- 172.21.15.83:0/2546108390 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4fc815ffc0 msgr2=0x7f4fc814b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.525+0000 7f4fcdc26640 1 --2- 172.21.15.83:0/2546108390 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4fc815ffc0 0x7f4fc814b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.525+0000 7f4fcdc26640 1 -- 172.21.15.83:0/2546108390 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc8160440 con 0x7f4fb4004800 2024-03-06T07:22:07.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.525+0000 7f4fcd425640 1 --2- 172.21.15.83:0/2546108390 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4fc815ffc0 0x7f4fc814b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:07.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.525+0000 7f4fce427640 1 --2- 172.21.15.83:0/2546108390 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4fc806ea70 0x7f4fc806ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:07.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.525+0000 7f4fcdc26640 1 --2- 172.21.15.83:0/2546108390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4fb4004800 0x7f4fc815e230 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4fb80119a0 tx=0x7f4fb8011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.525+0000 7f4fbe7fc640 1 -- 172.21.15.83:0/2546108390 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb800faf0 con 0x7f4fb4004800 2024-03-06T07:22:07.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.526+0000 7f4fbe7fc640 1 -- 172.21.15.83:0/2546108390 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fb800fc90 con 0x7f4fb4004800 2024-03-06T07:22:07.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.526+0000 7f4fcfe8a640 1 -- 172.21.15.83:0/2546108390 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fc815e930 con 0x7f4fb4004800 2024-03-06T07:22:07.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.526+0000 7f4fbe7fc640 1 -- 172.21.15.83:0/2546108390 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb8014440 con 0x7f4fb4004800 2024-03-06T07:22:07.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.526+0000 7f4fcfe8a640 1 -- 172.21.15.83:0/2546108390 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fc815eef0 con 0x7f4fb4004800 2024-03-06T07:22:07.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.527+0000 7f4fcfe8a640 1 -- 172.21.15.83:0/2546108390 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fc80b4b60 con 0x7f4fb4004800 2024-03-06T07:22:07.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.527+0000 7f4fbe7fc640 1 -- 172.21.15.83:0/2546108390 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4fb8012040 con 0x7f4fb4004800 2024-03-06T07:22:07.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.528+0000 7f4fbe7fc640 1 --2- 172.21.15.83:0/2546108390 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4fac06b4d0 0x7f4fac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.528+0000 7f4fbe7fc640 1 -- 172.21.15.83:0/2546108390 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4fb808a730 con 0x7f4fb4004800 2024-03-06T07:22:07.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.528+0000 7f4fcd425640 1 --2- 172.21.15.83:0/2546108390 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4fac06b4d0 0x7f4fac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.528+0000 7f4fcd425640 1 --2- 172.21.15.83:0/2546108390 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4fac06b4d0 0x7f4fac06d990 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4fc4036000 tx=0x7f4fc4030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.532+0000 7f4fbe7fc640 1 -- 172.21.15.83:0/2546108390 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4fc80b4b60 con 0x7f4fb4004800 2024-03-06T07:22:07.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.711+0000 7f4fcfe8a640 1 -- 172.21.15.83:0/2546108390 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4fc804f330 con 0x7f4fb4004800 2024-03-06T07:22:07.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.713+0000 7f4fbe7fc640 1 -- 172.21.15.83:0/2546108390 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16617 (secure 0 0 0) 0x7f4fb8019030 con 0x7f4fb4004800 2024-03-06T07:22:07.715 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:22:07.715 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":20,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:21:59.938137+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.318942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.254795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:07.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.715+0000 7f4fcfe8a640 1 -- 172.21.15.83:0/2546108390 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4fac06b4d0 msgr2=0x7f4fac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.715+0000 7f4fcfe8a640 1 --2- 172.21.15.83:0/2546108390 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4fac06b4d0 0x7f4fac06d990 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4fc4036000 tx=0x7f4fc4030000 comp rx=0 tx=0).stop 2024-03-06T07:22:07.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.715+0000 7f4fcfe8a640 1 -- 172.21.15.83:0/2546108390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4fb4004800 msgr2=0x7f4fc815e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.715+0000 7f4fcfe8a640 1 --2- 172.21.15.83:0/2546108390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4fb4004800 0x7f4fc815e230 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4fb80119a0 tx=0x7f4fb8011e60 comp rx=0 tx=0).stop 2024-03-06T07:22:07.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.715+0000 7f4fcfe8a640 1 -- 172.21.15.83:0/2546108390 shutdown_connections 2024-03-06T07:22:07.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.715+0000 7f4fcfe8a640 1 --2- 172.21.15.83:0/2546108390 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4fac06b4d0 0x7f4fac06d990 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.715+0000 7f4fcfe8a640 1 --2- 172.21.15.83:0/2546108390 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4fc806ea70 0x7f4fc806ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.715+0000 7f4fcfe8a640 1 --2- 172.21.15.83:0/2546108390 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4fc815ffc0 0x7f4fc814b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.715+0000 7f4fcfe8a640 1 --2- 172.21.15.83:0/2546108390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4fb4004800 0x7f4fc815e230 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.715+0000 7f4fcfe8a640 1 -- 172.21.15.83:0/2546108390 >> 172.21.15.83:0/2546108390 conn(0x7f4fc806c0c0 msgr2=0x7f4fc80b04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.715+0000 7f4fcfe8a640 1 -- 172.21.15.83:0/2546108390 shutdown_connections 2024-03-06T07:22:07.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.715+0000 7f4fcfe8a640 1 -- 172.21.15.83:0/2546108390 wait complete. 2024-03-06T07:22:07.728 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-06T07:22:07.728 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-06T07:22:07.729 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-06T07:22:07.729 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-06T07:22:07.729 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-06T07:22:07.730 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-06T07:22:07.730 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-06T07:22:07.731 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-06T07:22:07.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.897+0000 7f79e9ceb640 1 Processor -- start 2024-03-06T07:22:07.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.897+0000 7f79e9ceb640 1 -- start start 2024-03-06T07:22:07.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.898+0000 7f79e9ceb640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f79dc009190 0x7f79dc052f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.898+0000 7f79e9ceb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79dc0ec0c0 con 0x7f79dc048c40 2024-03-06T07:22:07.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.899+0000 7f79e9ceb640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79dc009570 con 0x7f79dc049070 2024-03-06T07:22:07.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.899+0000 7f79e9ceb640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79dc035970 con 0x7f79dc009190 2024-03-06T07:22:07.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.899+0000 7f79e37fe640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f79dc009190 0x7f79dc052f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.899+0000 7f79e2ffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f79dc048c40 0x7f79dc0eb2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:42828/0 (socket says 172.21.15.83:42828) 2024-03-06T07:22:07.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.899+0000 7f79e2ffd640 1 -- 172.21.15.83:0/77463281 learned_addr learned my addr 172.21.15.83:0/77463281 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:07.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.899+0000 7fb8f3481640 1 Processor -- start 2024-03-06T07:22:07.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.900+0000 7fb8f3481640 1 -- start start 2024-03-06T07:22:07.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.900+0000 7f79e27fc640 1 -- 172.21.15.83:0/77463281 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3698541356 0 0) 0x7f79dc0ec0c0 con 0x7f79dc048c40 2024-03-06T07:22:07.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.900+0000 7f79e27fc640 1 -- 172.21.15.83:0/77463281 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79d4003520 con 0x7f79dc048c40 2024-03-06T07:22:07.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.900+0000 7f49284fb640 1 Processor -- start 2024-03-06T07:22:07.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.900+0000 7f79e27fc640 1 -- 172.21.15.83:0/77463281 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4257204783 0 0) 0x7f79d4003520 con 0x7f79dc048c40 2024-03-06T07:22:07.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.900+0000 7f79e27fc640 1 -- 172.21.15.83:0/77463281 >> v2:172.21.15.183:3300/0 conn(0x7f79dc009190 msgr2=0x7f79dc052f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:07.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.900+0000 7f79e27fc640 1 --2- 172.21.15.83:0/77463281 >> v2:172.21.15.183:3300/0 conn(0x7f79dc009190 0x7f79dc052f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.900+0000 7f79e27fc640 1 -- 172.21.15.83:0/77463281 >> v1:172.21.15.183:6789/0 conn(0x7f79dc049070 legacy=0x7f79dc0eb9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.900+0000 7f79e27fc640 1 -- 172.21.15.83:0/77463281 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79dc035c30 con 0x7f79dc048c40 2024-03-06T07:22:07.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.900+0000 7f79e27fc640 1 -- 172.21.15.83:0/77463281 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f79e404fb80 con 0x7f79dc048c40 2024-03-06T07:22:07.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.900+0000 7f79e27fc640 1 -- 172.21.15.83:0/77463281 >> v1:172.21.15.83:6789/0 conn(0x7f79dc048c40 legacy=0x7f79dc0eb2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.900+0000 7f79e27fc640 1 --2- 172.21.15.83:0/77463281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79d40039b0 0x7f79d4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.900+0000 7f79e27fc640 1 --2- 172.21.15.83:0/77463281 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f79d4004350 0x7f79d4004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.900+0000 7f79e27fc640 1 --2- 172.21.15.83:0/77463281 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f79d4005020 0x7f79d400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.900+0000 7f79e27fc640 1 -- 172.21.15.83:0/77463281 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79dc035970 con 0x7f79d40039b0 2024-03-06T07:22:07.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.900+0000 7f79e27fc640 1 -- 172.21.15.83:0/77463281 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d400bf60 con 0x7f79d4004350 2024-03-06T07:22:07.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.900+0000 7f79e27fc640 1 -- 172.21.15.83:0/77463281 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79d400c160 con 0x7f79d4005020 2024-03-06T07:22:07.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.901+0000 7f79e2ffd640 1 --2- 172.21.15.83:0/77463281 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f79d4004350 0x7f79d4004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.901+0000 7fb8f3481640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb8e40df800 0x7fb8e4055d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.901+0000 7fb8f3481640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fb8e40366a0 0x7fb8e404fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.901+0000 7fb8f3481640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e4006c30 con 0x7fb8e403a0a0 2024-03-06T07:22:07.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.901+0000 7fb8f3481640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8e4006f10 con 0x7fb8e40366a0 2024-03-06T07:22:07.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.901+0000 7fb8f3481640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e40071f0 con 0x7fb8e40df800 2024-03-06T07:22:07.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.901+0000 7fb8f1a1e640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fb8e403a0a0 0x7fb8e40079f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:37844/0 (socket says 172.21.15.83:37844) 2024-03-06T07:22:07.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.901+0000 7fb8f1a1e640 1 -- 172.21.15.83:0/2370598389 learned_addr learned my addr 172.21.15.83:0/2370598389 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:07.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.900+0000 7f49284fb640 1 -- start start 2024-03-06T07:22:07.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.900+0000 7f4577053640 1 Processor -- start 2024-03-06T07:22:07.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.900+0000 7f4577053640 1 -- start start 2024-03-06T07:22:07.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.901+0000 7f4577053640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f457008b990 0x7f45700b2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.901+0000 7f4577053640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f45700b42b0 0x7f457016f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.901+0000 7f4577053640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45700724e0 con 0x7f457016ab10 2024-03-06T07:22:07.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.901+0000 7f4577053640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45700708e0 con 0x7f457008b990 2024-03-06T07:22:07.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.901+0000 7f4577053640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4570072760 con 0x7f45700b42b0 2024-03-06T07:22:07.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.901+0000 7f4576051640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f457016ab10 0x7f457008b280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:37848/0 (socket says 172.21.15.83:37848) 2024-03-06T07:22:07.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.902+0000 7f4576051640 1 -- 172.21.15.83:0/1337497136 learned_addr learned my addr 172.21.15.83:0/1337497136 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:07.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.902+0000 7f4575850640 1 --2- 172.21.15.83:0/1337497136 >> v2:172.21.15.183:3300/0 conn(0x7f457008b990 0x7f45700b2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.902+0000 7f4576852640 1 --2- 172.21.15.83:0/1337497136 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f45700b42b0 0x7f457016f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.901+0000 7f79e37fe640 1 --2- 172.21.15.83:0/77463281 >> v2:172.21.15.183:3300/0 conn(0x7f79dc009190 0x7f79dc052f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:07.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.901+0000 7f79e2ffd640 1 -- 172.21.15.83:0/77463281 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f79d4005020 msgr2=0x7f79d400b9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:22:07.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.901+0000 7f79e2ffd640 1 --2- 172.21.15.83:0/77463281 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f79d4005020 0x7f79d400b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.901+0000 7f79e2ffd640 1 -- 172.21.15.83:0/77463281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79d40039b0 msgr2=0x7f79d4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:07.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.901+0000 7f79e2ffd640 1 --2- 172.21.15.83:0/77463281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79d40039b0 0x7f79d4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.901+0000 7f79e2ffd640 1 -- 172.21.15.83:0/77463281 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d4003520 con 0x7f79d4004350 2024-03-06T07:22:07.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.902+0000 7f49284fb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f49201621c0 0x7f49200b4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.902+0000 7f49284fb640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f4920165cc0 0x7f4920166100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.902+0000 7f49284fb640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49200cf4e0 con 0x7f49200b29c0 2024-03-06T07:22:07.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.902+0000 7f49284fb640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4920070a10 con 0x7f4920165cc0 2024-03-06T07:22:07.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.902+0000 7f49284fb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4920166640 con 0x7f49201621c0 2024-03-06T07:22:07.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.902+0000 7f79e3fff640 1 -- 172.21.15.83:0/77463281 reap_dead start 2024-03-06T07:22:07.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.902+0000 7f457504f640 1 -- 172.21.15.83:0/1337497136 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2075791751 0 0) 0x7f456c001110 con 0x7f457016ab10 2024-03-06T07:22:07.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.902+0000 7f457504f640 1 -- 172.21.15.83:0/1337497136 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f455c003520 con 0x7f457016ab10 2024-03-06T07:22:07.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.902+0000 7f4926297640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f49201621c0 0x7f49200b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.902+0000 7f4926297640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f49201621c0 0x7f49200b4810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:59162/0 (socket says 172.21.15.83:59162) 2024-03-06T07:22:07.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.902+0000 7f4926297640 1 -- 172.21.15.83:0/2665213794 learned_addr learned my addr 172.21.15.83:0/2665213794 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:07.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.902+0000 7f4925a96640 1 --2- 172.21.15.83:0/2665213794 >> v2:172.21.15.183:3300/0 conn(0x7f4920165cc0 0x7f4920166100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.902+0000 7fb8e3fff640 1 -- 172.21.15.83:0/2370598389 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 798203590 0 0) 0x7fb8e4006c30 con 0x7fb8e403a0a0 2024-03-06T07:22:07.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.902+0000 7fb8e3fff640 1 -- 172.21.15.83:0/2370598389 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8d8003520 con 0x7fb8e403a0a0 2024-03-06T07:22:07.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.902+0000 7f79e2ffd640 1 --2- 172.21.15.83:0/77463281 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f79d4004350 0x7f79d4004820 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f79e4000b60 tx=0x7f79d4005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be1285cd6bd5a60f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.902+0000 7fb8f0a1c640 1 --2- 172.21.15.83:0/2370598389 >> v2:172.21.15.183:3300/0 conn(0x7fb8e40366a0 0x7fb8e404fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.902+0000 7f4926297640 1 -- 172.21.15.83:0/2665213794 >> v2:172.21.15.183:3300/0 conn(0x7f4920165cc0 msgr2=0x7f4920166100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:07.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.902+0000 7f4926297640 1 --2- 172.21.15.83:0/2665213794 >> v2:172.21.15.183:3300/0 conn(0x7f4920165cc0 0x7f4920166100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.902+0000 7f4926297640 1 -- 172.21.15.83:0/2665213794 >> v1:172.21.15.183:6789/0 conn(0x7f49200b29c0 legacy=0x7f49200b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.902+0000 7f4926297640 1 -- 172.21.15.83:0/2665213794 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4920070c60 con 0x7f49201621c0 2024-03-06T07:22:07.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.903+0000 7f4925a96640 1 --2- 172.21.15.83:0/2665213794 >> v2:172.21.15.183:3300/0 conn(0x7f4920165cc0 0x7f4920166100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:07.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.903+0000 7f79e27fc640 1 -- 172.21.15.83:0/77463281 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e40b4070 con 0x7f79d4004350 2024-03-06T07:22:07.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.903+0000 7fb8f0a1c640 1 -- 172.21.15.83:0/2370598389 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb8e40df800 msgr2=0x7fb8e4055d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:07.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.903+0000 7fb8f0a1c640 1 --2- 172.21.15.83:0/2370598389 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb8e40df800 0x7fb8e4055d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.903+0000 7f79e27fc640 1 -- 172.21.15.83:0/77463281 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79e40a7c50 con 0x7f79d4004350 2024-03-06T07:22:07.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.903+0000 7f79e27fc640 1 -- 172.21.15.83:0/77463281 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e4068f10 con 0x7f79d4004350 2024-03-06T07:22:07.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.903+0000 7fb8f0a1c640 1 -- 172.21.15.83:0/2370598389 >> v1:172.21.15.183:6789/0 conn(0x7fb8e403a0a0 legacy=0x7fb8e40079f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.903+0000 7fb8f0a1c640 1 -- 172.21.15.83:0/2370598389 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e40074d0 con 0x7fb8e40366a0 2024-03-06T07:22:07.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.902+0000 7f4575850640 1 -- 172.21.15.83:0/1337497136 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f45700b42b0 msgr2=0x7f457016f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:07.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.902+0000 7f4575850640 1 --2- 172.21.15.83:0/1337497136 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f45700b42b0 0x7f457016f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.902+0000 7f4575850640 1 -- 172.21.15.83:0/1337497136 >> v1:172.21.15.183:6789/0 conn(0x7f457016ab10 legacy=0x7f457008b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.902+0000 7f4575850640 1 -- 172.21.15.83:0/1337497136 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4570070b80 con 0x7f457008b990 2024-03-06T07:22:07.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.903+0000 7f4575850640 1 --2- 172.21.15.83:0/1337497136 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f457008b990 0x7f45700b2800 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f45680077a0 tx=0x7f456802e650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97b20b18f527675f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.904+0000 7f4926297640 1 --2- 172.21.15.83:0/2665213794 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f49201621c0 0x7f49200b4810 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4918009e50 tx=0x7f491802bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba751ae4d29fc1a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.904+0000 7f79e9ceb640 1 -- 172.21.15.83:0/77463281 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f79d4004350 msgr2=0x7f79d4004820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.904+0000 7f79e9ceb640 1 --2- 172.21.15.83:0/77463281 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f79d4004350 0x7f79d4004820 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f79e4000b60 tx=0x7f79d4005830 comp rx=0 tx=0).stop 2024-03-06T07:22:07.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.904+0000 7f79e9ceb640 1 -- 172.21.15.83:0/77463281 shutdown_connections 2024-03-06T07:22:07.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.904+0000 7f79e9ceb640 1 --2- 172.21.15.83:0/77463281 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f79d4004350 0x7f79d4004820 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.904+0000 7f79e9ceb640 1 -- 172.21.15.83:0/77463281 >> 172.21.15.83:0/77463281 conn(0x7f79dc01ef40 msgr2=0x7f79dc055c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.904+0000 7f4925295640 1 -- 172.21.15.83:0/2665213794 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f491802ca80 con 0x7f49201621c0 2024-03-06T07:22:07.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.904+0000 7f79e9ceb640 1 -- 172.21.15.83:0/77463281 shutdown_connections 2024-03-06T07:22:07.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.904+0000 7f79e9ceb640 1 -- 172.21.15.83:0/77463281 wait complete. 2024-03-06T07:22:07.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.904+0000 7f4925295640 1 -- 172.21.15.83:0/2665213794 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f491802cc20 con 0x7f49201621c0 2024-03-06T07:22:07.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.904+0000 7fb8f0a1c640 1 --2- 172.21.15.83:0/2370598389 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb8e40366a0 0x7fb8e404fa40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb8e80097a0 tx=0x7fb8e802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb93a5ea4698c97a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.904+0000 7f4c07981640 1 Processor -- start 2024-03-06T07:22:07.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.904+0000 7fb8e3fff640 1 -- 172.21.15.83:0/2370598389 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e8034070 con 0x7fb8e40366a0 2024-03-06T07:22:07.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.904+0000 7fb8e3fff640 1 -- 172.21.15.83:0/2370598389 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8e8030610 con 0x7fb8e40366a0 2024-03-06T07:22:07.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.905+0000 7fb8e3fff640 1 -- 172.21.15.83:0/2370598389 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e80309e0 con 0x7fb8e40366a0 2024-03-06T07:22:07.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.904+0000 7f457504f640 1 -- 172.21.15.83:0/1337497136 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4568003b90 con 0x7f457008b990 2024-03-06T07:22:07.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.904+0000 7f457504f640 1 -- 172.21.15.83:0/1337497136 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4568003d30 con 0x7f457008b990 2024-03-06T07:22:07.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.904+0000 7f457504f640 1 -- 172.21.15.83:0/1337497136 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4568004250 con 0x7f457008b990 2024-03-06T07:22:07.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.905+0000 7f4577053640 1 -- 172.21.15.83:0/1337497136 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f457008b990 msgr2=0x7f45700b2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.905+0000 7f4577053640 1 --2- 172.21.15.83:0/1337497136 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f457008b990 0x7f45700b2800 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f45680077a0 tx=0x7f456802e650 comp rx=0 tx=0).stop 2024-03-06T07:22:07.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.905+0000 7f4577053640 1 -- 172.21.15.83:0/1337497136 shutdown_connections 2024-03-06T07:22:07.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.905+0000 7f4577053640 1 --2- 172.21.15.83:0/1337497136 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f45700b42b0 0x7f457016f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.905+0000 7f4577053640 1 --2- 172.21.15.83:0/1337497136 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f457008b990 0x7f45700b2800 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.905+0000 7f4577053640 1 -- 172.21.15.83:0/1337497136 >> 172.21.15.83:0/1337497136 conn(0x7f457006afb0 msgr2=0x7f457006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.905+0000 7f4577053640 1 -- 172.21.15.83:0/1337497136 shutdown_connections 2024-03-06T07:22:07.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.905+0000 7f4577053640 1 -- 172.21.15.83:0/1337497136 wait complete. 2024-03-06T07:22:07.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.905+0000 7f4925295640 1 -- 172.21.15.83:0/2665213794 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49180343f0 con 0x7f49201621c0 2024-03-06T07:22:07.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.905+0000 7f49284fb640 1 -- 172.21.15.83:0/2665213794 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f49201621c0 msgr2=0x7f49200b4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.905+0000 7f49284fb640 1 --2- 172.21.15.83:0/2665213794 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f49201621c0 0x7f49200b4810 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4918009e50 tx=0x7f491802bee0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.905+0000 7f49284fb640 1 -- 172.21.15.83:0/2665213794 shutdown_connections 2024-03-06T07:22:07.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.905+0000 7f49284fb640 1 --2- 172.21.15.83:0/2665213794 >> v2:172.21.15.183:3300/0 conn(0x7f4920165cc0 0x7f4920166100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.905+0000 7f49284fb640 1 --2- 172.21.15.83:0/2665213794 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f49201621c0 0x7f49200b4810 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.905+0000 7f49284fb640 1 -- 172.21.15.83:0/2665213794 >> 172.21.15.83:0/2665213794 conn(0x7f49200b16c0 msgr2=0x7f49200b1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.905+0000 7fb8f3481640 1 -- 172.21.15.83:0/2370598389 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb8e40366a0 msgr2=0x7fb8e404fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.905+0000 7fb8f3481640 1 --2- 172.21.15.83:0/2370598389 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb8e40366a0 0x7fb8e404fa40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb8e80097a0 tx=0x7fb8e802b960 comp rx=0 tx=0).stop 2024-03-06T07:22:07.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.905+0000 7f49284fb640 1 -- 172.21.15.83:0/2665213794 shutdown_connections 2024-03-06T07:22:07.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.906+0000 7f49284fb640 1 -- 172.21.15.83:0/2665213794 wait complete. 2024-03-06T07:22:07.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.905+0000 7f4c07981640 1 -- start start 2024-03-06T07:22:07.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.906+0000 7f4c07981640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f4c0006dc30 0x7f4c0016d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.906+0000 7fb8f3481640 1 -- 172.21.15.83:0/2370598389 shutdown_connections 2024-03-06T07:22:07.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.906+0000 7fb8f3481640 1 --2- 172.21.15.83:0/2370598389 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb8e40366a0 0x7fb8e404fa40 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.906+0000 7fb8f3481640 1 --2- 172.21.15.83:0/2370598389 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb8e40df800 0x7fb8e4055d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.906+0000 7fb8f3481640 1 -- 172.21.15.83:0/2370598389 >> 172.21.15.83:0/2370598389 conn(0x7fb8e401e6c0 msgr2=0x7fb8e40553f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.906+0000 7f4577053640 1 Processor -- start 2024-03-06T07:22:07.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.907+0000 7f4577053640 1 -- start start 2024-03-06T07:22:07.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.907+0000 7f79e9ceb640 1 Processor -- start 2024-03-06T07:22:07.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.907+0000 7f4c07981640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c0006e170 con 0x7f4c0006d000 2024-03-06T07:22:07.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.907+0000 7f4c04f1c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f4c0006d000 0x7f4c0006d520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:42838/0 (socket says 172.21.15.83:42838) 2024-03-06T07:22:07.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.907+0000 7f4c04f1c640 1 -- 172.21.15.83:0/652866998 learned_addr learned my addr 172.21.15.83:0/652866998 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:07.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.907+0000 7f4c07981640 1 -- 172.21.15.83:0/652866998 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c00165ff0 con 0x7f4c0015ff70 2024-03-06T07:22:07.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.907+0000 7f4c07981640 1 -- 172.21.15.83:0/652866998 --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c000717e0 con 0x7f4c0006dc30 2024-03-06T07:22:07.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.907+0000 7f4c05f1e640 1 --2- 172.21.15.83:0/652866998 >> v2:172.21.15.183:3300/0 conn(0x7f4c0006dc30 0x7f4c0016d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.907+0000 7f49284fb640 1 Processor -- start 2024-03-06T07:22:07.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.907+0000 7f49284fb640 1 -- start start 2024-03-06T07:22:07.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.907+0000 7f4577053640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f457008b990 0x7f4570162d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.907+0000 7f4577053640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f45700b42b0 0x7f4570169cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.908+0000 7f4577053640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f457016ab10 0x7f457015f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.908+0000 7f4577053640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f457006ca10 con 0x7f457008b990 2024-03-06T07:22:07.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.908+0000 7f4577053640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f457016aef0 con 0x7f45700b42b0 2024-03-06T07:22:07.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.908+0000 7f4577053640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f457006cb90 con 0x7f457016ab10 2024-03-06T07:22:07.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.908+0000 7f4575850640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f45700b42b0 0x7f4570169cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.908+0000 7f4575850640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f45700b42b0 0x7f4570169cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:59970/0 (socket says 172.21.15.83:59970) 2024-03-06T07:22:07.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.908+0000 7f4575850640 1 -- 172.21.15.83:0/3444966563 learned_addr learned my addr 172.21.15.83:0/3444966563 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:07.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.908+0000 7f4576852640 1 --2- 172.21.15.83:0/3444966563 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f457016ab10 0x7f457015f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.908+0000 7f4576051640 1 --2- 172.21.15.83:0/3444966563 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f457008b990 0x7f4570162d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.908+0000 7f4575850640 1 -- 172.21.15.83:0/3444966563 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f457016ab10 msgr2=0x7f457015f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.908+0000 7f4575850640 1 --2- 172.21.15.83:0/3444966563 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f457016ab10 0x7f457015f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.908+0000 7f4575850640 1 -- 172.21.15.83:0/3444966563 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f457008b990 msgr2=0x7f4570162d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.908+0000 7f4575850640 1 --2- 172.21.15.83:0/3444966563 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f457008b990 0x7f4570162d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.908+0000 7f4575850640 1 -- 172.21.15.83:0/3444966563 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4570160be0 con 0x7f45700b42b0 2024-03-06T07:22:07.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.907+0000 7f79e9ceb640 1 -- start start 2024-03-06T07:22:07.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.908+0000 7f79e9ceb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79d4004350 0x7f79dc0e9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.907+0000 7f4bf7fff640 1 -- 172.21.15.83:0/652866998 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3771586868 0 0) 0x7f4c00165ff0 con 0x7f4c0015ff70 2024-03-06T07:22:07.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.907+0000 7f4bf7fff640 1 -- 172.21.15.83:0/652866998 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bec003520 con 0x7f4c0015ff70 2024-03-06T07:22:07.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.908+0000 7fb8f3481640 1 -- 172.21.15.83:0/2370598389 shutdown_connections 2024-03-06T07:22:07.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.908+0000 7fb8f3481640 1 -- 172.21.15.83:0/2370598389 wait complete. 2024-03-06T07:22:07.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.908+0000 7f4575850640 1 --2- 172.21.15.83:0/3444966563 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f45700b42b0 0x7f4570169cf0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f45680078c0 tx=0x7f4568004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.907+0000 7f4c05f1e640 1 -- 172.21.15.83:0/652866998 >> v1:172.21.15.183:6789/0 conn(0x7f4c0015ff70 legacy=0x7f4c0006c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.909+0000 7f49284fb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f49200b29c0 0x7f4920153830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.909+0000 7f49284fb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49201621c0 0x7f49201eb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.909+0000 7f49284fb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4920165cc0 0x7f492014d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.907+0000 7f4c05f1e640 1 -- 172.21.15.83:0/652866998 >> v1:172.21.15.83:6789/0 conn(0x7f4c0006d000 legacy=0x7f4c0006d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.907+0000 7f4c05f1e640 1 -- 172.21.15.83:0/652866998 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c001662d0 con 0x7f4c0006dc30 2024-03-06T07:22:07.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.909+0000 7f4566ffd640 1 -- 172.21.15.83:0/3444966563 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4568044020 con 0x7f45700b42b0 2024-03-06T07:22:07.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.909+0000 7f4566ffd640 1 -- 172.21.15.83:0/3444966563 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4568040680 con 0x7f45700b42b0 2024-03-06T07:22:07.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.909+0000 7f4566ffd640 1 -- 172.21.15.83:0/3444966563 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4568040a50 con 0x7f45700b42b0 2024-03-06T07:22:07.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.909+0000 7f4577053640 1 -- 172.21.15.83:0/3444966563 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4570160ec0 con 0x7f45700b42b0 2024-03-06T07:22:07.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.909+0000 7f4c05f1e640 1 --2- 172.21.15.83:0/652866998 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4c0006dc30 0x7f4c0016d150 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4bf8009d90 tx=0x7f4bf802bf00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a73030d3fcf89a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.909+0000 7f4577053640 1 -- 172.21.15.83:0/3444966563 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4570161480 con 0x7f45700b42b0 2024-03-06T07:22:07.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.909+0000 7f4bf7fff640 1 -- 172.21.15.83:0/652866998 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf8039070 con 0x7f4c0006dc30 2024-03-06T07:22:07.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.909+0000 7f4bf7fff640 1 -- 172.21.15.83:0/652866998 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bf802ccb0 con 0x7f4c0006dc30 2024-03-06T07:22:07.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.909+0000 7f4bf7fff640 1 -- 172.21.15.83:0/652866998 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf8034660 con 0x7f4c0006dc30 2024-03-06T07:22:07.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.910+0000 7fb8f3481640 1 Processor -- start 2024-03-06T07:22:07.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.909+0000 7f4926297640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f49200b29c0 0x7f4920153830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.909+0000 7f4926297640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f49200b29c0 0x7f4920153830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:59982/0 (socket says 172.21.15.83:59982) 2024-03-06T07:22:07.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.909+0000 7f4925a96640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49201621c0 0x7f49201eb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.909+0000 7f4925a96640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49201621c0 0x7f49201eb9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49384/0 (socket says 172.21.15.83:49384) 2024-03-06T07:22:07.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.909+0000 7f49284fb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49201669a0 con 0x7f49201621c0 2024-03-06T07:22:07.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.909+0000 7f49284fb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49201625a0 con 0x7f49200b29c0 2024-03-06T07:22:07.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.909+0000 7f49284fb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4920166b20 con 0x7f4920165cc0 2024-03-06T07:22:07.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.909+0000 7f4926297640 1 -- 172.21.15.83:0/886514769 learned_addr learned my addr 172.21.15.83:0/886514769 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:07.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.909+0000 7f4926a98640 1 --2- 172.21.15.83:0/886514769 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4920165cc0 0x7f492014d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.910+0000 7f4926a98640 1 -- 172.21.15.83:0/886514769 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f49200b29c0 msgr2=0x7f4920153830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.910+0000 7f4564ff9640 1 -- 172.21.15.83:0/3444966563 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4558001030 con 0x7f45700b42b0 2024-03-06T07:22:07.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.910+0000 7f4c07981640 1 -- 172.21.15.83:0/652866998 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4c0006dc30 msgr2=0x7f4c0016d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.910+0000 7f4c07981640 1 --2- 172.21.15.83:0/652866998 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4c0006dc30 0x7f4c0016d150 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4bf8009d90 tx=0x7f4bf802bf00 comp rx=0 tx=0).stop 2024-03-06T07:22:07.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.911+0000 7f4c07981640 1 -- 172.21.15.83:0/652866998 shutdown_connections 2024-03-06T07:22:07.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.910+0000 7f4926a98640 1 --2- 172.21.15.83:0/886514769 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f49200b29c0 0x7f4920153830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.910+0000 7f4926a98640 1 -- 172.21.15.83:0/886514769 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49201621c0 msgr2=0x7f49201eb9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.910+0000 7f4926a98640 1 --2- 172.21.15.83:0/886514769 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49201621c0 0x7f49201eb9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.910+0000 7f4926a98640 1 -- 172.21.15.83:0/886514769 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49201ed4a0 con 0x7f4920165cc0 2024-03-06T07:22:07.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.910+0000 7f4925a96640 1 --2- 172.21.15.83:0/886514769 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49201621c0 0x7f49201eb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:07.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.910+0000 7f4926a98640 1 --2- 172.21.15.83:0/886514769 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4920165cc0 0x7f492014d090 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f491c00c3f0 tx=0x7f491c00c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.911+0000 7f4916ffd640 1 -- 172.21.15.83:0/886514769 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f491c0052d0 con 0x7f4920165cc0 2024-03-06T07:22:07.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.911+0000 7f4c07981640 1 --2- 172.21.15.83:0/652866998 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4c0006dc30 0x7f4c0016d150 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.911+0000 7f4c07981640 1 -- 172.21.15.83:0/652866998 >> 172.21.15.83:0/652866998 conn(0x7f4c000b1c80 msgr2=0x7f4c000cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.911+0000 7f4c07981640 1 -- 172.21.15.83:0/652866998 shutdown_connections 2024-03-06T07:22:07.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.909+0000 7f79e9ceb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f79dc04fbf0 0x7f79dc006ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.909+0000 7f79e9ceb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f79dc051420 0x7f79dc0518d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.909+0000 7f79e9ceb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79dc0364d0 con 0x7f79d4004350 2024-03-06T07:22:07.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.909+0000 7f79e9ceb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79dc0361f0 con 0x7f79dc04fbf0 2024-03-06T07:22:07.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.909+0000 7f79e9ceb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79dc0367b0 con 0x7f79dc051420 2024-03-06T07:22:07.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.912+0000 7f79e3fff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f79dc051420 0x7f79dc0518d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.912+0000 7f79e3fff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f79dc051420 0x7f79dc0518d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:59190/0 (socket says 172.21.15.83:59190) 2024-03-06T07:22:07.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.912+0000 7f79e3fff640 1 -- 172.21.15.83:0/4052470765 learned_addr learned my addr 172.21.15.83:0/4052470765 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:07.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.911+0000 7f4c07981640 1 -- 172.21.15.83:0/652866998 wait complete. 2024-03-06T07:22:07.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.913+0000 7f4c07981640 1 Processor -- start 2024-03-06T07:22:07.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.912+0000 7f79e37fe640 1 --2- 172.21.15.83:0/4052470765 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79d4004350 0x7f79dc0e9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.913+0000 7f79e3fff640 1 -- 172.21.15.83:0/4052470765 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f79dc04fbf0 msgr2=0x7f79dc006ab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-06T07:22:07.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.913+0000 7f79e3fff640 1 --2- 172.21.15.83:0/4052470765 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f79dc04fbf0 0x7f79dc006ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.913+0000 7f79e3fff640 1 -- 172.21.15.83:0/4052470765 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79d4004350 msgr2=0x7f79dc0e9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.913+0000 7f4c07981640 1 -- start start 2024-03-06T07:22:07.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.913+0000 7f79e3fff640 1 --2- 172.21.15.83:0/4052470765 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79d4004350 0x7f79dc0e9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.913+0000 7f79e3fff640 1 -- 172.21.15.83:0/4052470765 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79dc006ff0 con 0x7f79dc051420 2024-03-06T07:22:07.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.911+0000 7f49284fb640 1 -- 172.21.15.83:0/886514769 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49201ec400 con 0x7f4920165cc0 2024-03-06T07:22:07.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.911+0000 7f49284fb640 1 -- 172.21.15.83:0/886514769 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49201ec9c0 con 0x7f4920165cc0 2024-03-06T07:22:07.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.912+0000 7f4914ff9640 1 -- 172.21.15.83:0/886514769 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4908001030 con 0x7f4920165cc0 2024-03-06T07:22:07.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.913+0000 7f4916ffd640 1 -- 172.21.15.83:0/886514769 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f491c005470 con 0x7f4920165cc0 2024-03-06T07:22:07.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.913+0000 7f4916ffd640 1 -- 172.21.15.83:0/886514769 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f491c007260 con 0x7f4920165cc0 2024-03-06T07:22:07.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.913+0000 7f4916ffd640 1 -- 172.21.15.83:0/886514769 <== mon.2 v2:172.21.15.183:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f491c007480 con 0x7f4920165cc0 2024-03-06T07:22:07.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.911+0000 7f4566ffd640 1 -- 172.21.15.83:0/3444966563 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4568047020 con 0x7f45700b42b0 2024-03-06T07:22:07.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.913+0000 7f79e37fe640 1 --2- 172.21.15.83:0/4052470765 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79d4004350 0x7f79dc0e9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:07.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.912+0000 7f4566ffd640 1 --2- 172.21.15.83:0/3444966563 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f455406b5a0 0x7f455406da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.912+0000 7f4566ffd640 1 -- 172.21.15.83:0/3444966563 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f45680adb60 con 0x7f45700b42b0 2024-03-06T07:22:07.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.914+0000 7f79e3fff640 1 --2- 172.21.15.83:0/4052470765 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f79dc051420 0x7f79dc0518d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f79dc038b70 tx=0x7f79d000c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.910+0000 7fb8f3481640 1 -- start start 2024-03-06T07:22:07.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.911+0000 7fb8f3481640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb8e40366a0 0x7fb8e4181310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.911+0000 7fb8f3481640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb8e403a0a0 0x7fb8e40d6830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.911+0000 7fb8f3481640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb8e40df800 0x7fb8e4048f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.911+0000 7fb8f3481640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e404ff80 con 0x7fb8e40df800 2024-03-06T07:22:07.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.911+0000 7fb8f3481640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e4036ae0 con 0x7fb8e403a0a0 2024-03-06T07:22:07.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.911+0000 7fb8f3481640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e4050100 con 0x7fb8e40366a0 2024-03-06T07:22:07.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.912+0000 7f4566ffd640 1 --2- 172.21.15.83:0/3444966563 >> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] conn(0x7f4554073740 0x7f4554075b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.912+0000 7f4566ffd640 1 -- 172.21.15.83:0/3444966563 --> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f456804ae50 con 0x7f4554073740 2024-03-06T07:22:07.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.912+0000 7f4566ffd640 1 -- 172.21.15.83:0/3444966563 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4568078680 con 0x7f45700b42b0 2024-03-06T07:22:07.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.912+0000 7f4576852640 1 --2- 172.21.15.83:0/3444966563 >> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] conn(0x7f4554073740 0x7f4554075b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.913+0000 7f4576852640 1 --2- 172.21.15.83:0/3444966563 >> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] conn(0x7f4554073740 0x7f4554075b60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.914+0000 7fb8f1a1e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb8e40df800 0x7fb8e4048f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.914+0000 7f4576051640 1 --2- 172.21.15.83:0/3444966563 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f455406b5a0 0x7f455406da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.914+0000 7f79cffff640 1 -- 172.21.15.83:0/4052470765 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d00074e0 con 0x7f79dc051420 2024-03-06T07:22:07.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.914+0000 7f79e9ceb640 1 -- 172.21.15.83:0/4052470765 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79dc007290 con 0x7f79dc051420 2024-03-06T07:22:07.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.914+0000 7f79e9ceb640 1 -- 172.21.15.83:0/4052470765 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79dc007850 con 0x7f79dc051420 2024-03-06T07:22:07.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.914+0000 7f79cffff640 1 -- 172.21.15.83:0/4052470765 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79d0004020 con 0x7f79dc051420 2024-03-06T07:22:07.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.914+0000 7fb8f1a1e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb8e40df800 0x7fb8e4048f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49392/0 (socket says 172.21.15.83:49392) 2024-03-06T07:22:07.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.914+0000 7fb8f1a1e640 1 -- 172.21.15.83:0/2633103697 learned_addr learned my addr 172.21.15.83:0/2633103697 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:07.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.914+0000 7f4576051640 1 --2- 172.21.15.83:0/3444966563 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f455406b5a0 0x7f455406da60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f456c007fd0 tx=0x7f456c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.915+0000 7f4c07981640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4c0006d000 0x7f4c0015a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.915+0000 7f4916ffd640 1 --2- 172.21.15.83:0/886514769 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f490406b5d0 0x7f490406da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.915+0000 7f4c07981640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4c0006dc30 0x7f4c00160ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.915+0000 7f4c07981640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c0015ff70 0x7f4c0015b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.915+0000 7f4c04f1c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4c0006dc30 0x7f4c00160ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.915+0000 7f4c05f1e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c0015ff70 0x7f4c0015b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.915+0000 7f4c05f1e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c0015ff70 0x7f4c0015b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49406/0 (socket says 172.21.15.83:49406) 2024-03-06T07:22:07.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.915+0000 7f4c04f1c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4c0006dc30 0x7f4c00160ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:59992/0 (socket says 172.21.15.83:59992) 2024-03-06T07:22:07.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.915+0000 7f4c05f1e640 1 -- 172.21.15.83:0/3952791940 learned_addr learned my addr 172.21.15.83:0/3952791940 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:07.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.915+0000 7f4c07981640 1 -- 172.21.15.83:0/3952791940 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c00166db0 con 0x7f4c0015ff70 2024-03-06T07:22:07.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.915+0000 7f4566ffd640 1 -- 172.21.15.83:0/3444966563 <== osd.0 v2:172.21.15.83:6808/1635427425 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f456804ae50 con 0x7f4554073740 2024-03-06T07:22:07.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.915+0000 7f79cffff640 1 -- 172.21.15.83:0/4052470765 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d0004410 con 0x7f79dc051420 2024-03-06T07:22:07.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.915+0000 7f79cdffb640 1 -- 172.21.15.83:0/4052470765 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f79bc001030 con 0x7f79dc051420 2024-03-06T07:22:07.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.915+0000 7f4926297640 1 --2- 172.21.15.83:0/886514769 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f490406b5d0 0x7f490406da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.916+0000 7f4916ffd640 1 -- 172.21.15.83:0/886514769 <== mon.2 v2:172.21.15.183:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f491c0895d0 con 0x7f4920165cc0 2024-03-06T07:22:07.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.916+0000 7f4916ffd640 1 --2- 172.21.15.83:0/886514769 >> [v2:172.21.15.83:6800/2591342133,v1:172.21.15.83:6801/2591342133] conn(0x7f49040737f0 0x7f4904075c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.916+0000 7f4916ffd640 1 -- 172.21.15.83:0/886514769 --> [v2:172.21.15.83:6800/2591342133,v1:172.21.15.83:6801/2591342133] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f491c011e50 con 0x7f49040737f0 2024-03-06T07:22:07.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.916+0000 7f4916ffd640 1 -- 172.21.15.83:0/886514769 <== mon.2 v2:172.21.15.183:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f491c090050 con 0x7f4920165cc0 2024-03-06T07:22:07.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.916+0000 7f4926297640 1 --2- 172.21.15.83:0/886514769 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f490406b5d0 0x7f490406da90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4918009e50 tx=0x7f4918030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.916+0000 7f4925a96640 1 --2- 172.21.15.83:0/886514769 >> [v2:172.21.15.83:6800/2591342133,v1:172.21.15.83:6801/2591342133] conn(0x7f49040737f0 0x7f4904075c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.915+0000 7f4c0571d640 1 --2- 172.21.15.83:0/3952791940 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4c0006d000 0x7f4c0015a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.917+0000 7f03d6226640 1 Processor -- start 2024-03-06T07:22:07.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.915+0000 7fb8f1a1e640 1 -- 172.21.15.83:0/2633103697 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb8e40366a0 msgr2=0x7fb8e4181310 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-06T07:22:07.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.915+0000 7fb8f1a1e640 1 --2- 172.21.15.83:0/2633103697 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb8e40366a0 0x7fb8e4181310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.915+0000 7fb8f1a1e640 1 -- 172.21.15.83:0/2633103697 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb8e403a0a0 msgr2=0x7fb8e40d6830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-06T07:22:07.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.915+0000 7fb8f1a1e640 1 --2- 172.21.15.83:0/2633103697 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb8e403a0a0 0x7fb8e40d6830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.915+0000 7fb8f1a1e640 1 -- 172.21.15.83:0/2633103697 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e4049820 con 0x7fb8e40df800 2024-03-06T07:22:07.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.917+0000 7f4925a96640 1 --2- 172.21.15.83:0/886514769 >> [v2:172.21.15.83:6800/2591342133,v1:172.21.15.83:6801/2591342133] conn(0x7f49040737f0 0x7f4904075c10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.916+0000 7f79cffff640 1 -- 172.21.15.83:0/4052470765 <== mon.2 v2:172.21.15.183:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f79d00057c0 con 0x7f79dc051420 2024-03-06T07:22:07.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.917+0000 7f79cffff640 1 --2- 172.21.15.83:0/4052470765 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f79c806b550 0x7f79c806da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.917+0000 7f79cffff640 1 -- 172.21.15.83:0/4052470765 <== mon.2 v2:172.21.15.183:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f79d0089360 con 0x7f79dc051420 2024-03-06T07:22:07.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.917+0000 7f79cffff640 1 --2- 172.21.15.83:0/4052470765 >> [v2:172.21.15.83:6814/1822925240,v1:172.21.15.83:6817/1822925240] conn(0x7f79c80736f0 0x7f79c8075b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.917+0000 7f79cffff640 1 -- 172.21.15.83:0/4052470765 --> [v2:172.21.15.83:6814/1822925240,v1:172.21.15.83:6817/1822925240] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f79d0004020 con 0x7f79c80736f0 2024-03-06T07:22:07.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.917+0000 7fb8f1a1e640 1 --2- 172.21.15.83:0/2633103697 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb8e40df800 0x7fb8e4048f20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec04fec0 tx=0x7fb8ec068ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.917+0000 7f4c07981640 1 -- 172.21.15.83:0/3952791940 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c00160350 con 0x7f4c0006dc30 2024-03-06T07:22:07.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.917+0000 7f4c07981640 1 -- 172.21.15.83:0/3952791940 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c00166f30 con 0x7f4c0006d000 2024-03-06T07:22:07.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.918+0000 7f79cffff640 1 -- 172.21.15.83:0/4052470765 <== mon.2 v2:172.21.15.183:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f79d0090050 con 0x7f79dc051420 2024-03-06T07:22:07.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.917+0000 7f03d6226640 1 -- start start 2024-03-06T07:22:07.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.918+0000 7f4916ffd640 1 -- 172.21.15.83:0/886514769 <== osd.2 v2:172.21.15.83:6800/2591342133 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f491c011e50 con 0x7f49040737f0 2024-03-06T07:22:07.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.918+0000 7f03d6226640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f03d00d3710 0x7f03d016d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.918+0000 7f79e37fe640 1 --2- 172.21.15.83:0/4052470765 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f79c806b550 0x7f79c806da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.917+0000 7fb8e1ffb640 1 -- 172.21.15.83:0/2633103697 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec068360 con 0x7fb8e40df800 2024-03-06T07:22:07.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.918+0000 7fb8e1ffb640 1 -- 172.21.15.83:0/2633103697 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8ec073260 con 0x7fb8e40df800 2024-03-06T07:22:07.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.918+0000 7fb8e1ffb640 1 -- 172.21.15.83:0/2633103697 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec073650 con 0x7fb8e40df800 2024-03-06T07:22:07.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.918+0000 7f03d5224640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f03d0162140 0x7f03d00729b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:42848/0 (socket says 172.21.15.83:42848) 2024-03-06T07:22:07.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.918+0000 7f03d6226640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03d00b8700 con 0x7f03d0162140 2024-03-06T07:22:07.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.918+0000 7f03d6226640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03d0165f30 con 0x7f03d00730c0 2024-03-06T07:22:07.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.919+0000 7f03d6226640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03d0166cf0 con 0x7f03d00d3710 2024-03-06T07:22:07.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.919+0000 7f03d5a25640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f03d00d3710 0x7f03d016d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.919+0000 7f79e2ffd640 1 --2- 172.21.15.83:0/4052470765 >> [v2:172.21.15.83:6814/1822925240,v1:172.21.15.83:6817/1822925240] conn(0x7f79c80736f0 0x7f79c8075b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.919+0000 7f03d5a25640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f03d00d3710 0x7f03d016d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:59212/0 (socket says 172.21.15.83:59212) 2024-03-06T07:22:07.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.919+0000 7f03d5a25640 1 -- 172.21.15.83:0/2062158780 learned_addr learned my addr 172.21.15.83:0/2062158780 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:07.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.919+0000 7fb8f3481640 1 -- 172.21.15.83:0/2633103697 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8e4049b00 con 0x7fb8e40df800 2024-03-06T07:22:07.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.919+0000 7f79e37fe640 1 --2- 172.21.15.83:0/4052470765 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f79c806b550 0x7f79c806da10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f79dc003e80 tx=0x7f79d8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.920+0000 7f4c0571d640 1 -- 172.21.15.83:0/3952791940 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4c0006dc30 msgr2=0x7f4c00160ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.920+0000 7f4c0571d640 1 --2- 172.21.15.83:0/3952791940 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4c0006dc30 0x7f4c00160ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.920+0000 7f4c0571d640 1 -- 172.21.15.83:0/3952791940 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c0015ff70 msgr2=0x7f4c0015b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.920+0000 7f4c0571d640 1 --2- 172.21.15.83:0/3952791940 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c0015ff70 0x7f4c0015b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.920+0000 7f4c0571d640 1 -- 172.21.15.83:0/3952791940 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c001f8040 con 0x7f4c0006d000 2024-03-06T07:22:07.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.920+0000 7f4c0571d640 1 --2- 172.21.15.83:0/3952791940 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4c0006d000 0x7f4c0015a030 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc00c3c0 tx=0x7f4bfc00c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.919+0000 7f03d5a25640 1 -- 172.21.15.83:0/2062158780 >> v1:172.21.15.183:6789/0 conn(0x7f03d00730c0 legacy=0x7f03d00b7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:07.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.919+0000 7f03d5a25640 1 -- 172.21.15.83:0/2062158780 >> v1:172.21.15.83:6789/0 conn(0x7f03d0162140 legacy=0x7f03d00729b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:07.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.919+0000 7f03d5a25640 1 -- 172.21.15.83:0/2062158780 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d0166210 con 0x7f03d00d3710 2024-03-06T07:22:07.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.920+0000 7f79e2ffd640 1 --2- 172.21.15.83:0/4052470765 >> [v2:172.21.15.83:6814/1822925240,v1:172.21.15.83:6817/1822925240] conn(0x7f79c80736f0 0x7f79c8075b10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.921+0000 7f4c05f1e640 1 --2- 172.21.15.83:0/3952791940 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c0015ff70 0x7f4c0015b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:07.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.921+0000 7f03d5a25640 1 --2- 172.21.15.83:0/2062158780 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f03d00d3710 0x7f03d016d0c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f03c8009e80 tx=0x7f03c802beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49b6d3445cf67183 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.922+0000 7f4bf5ffb640 1 -- 172.21.15.83:0/3952791940 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc017070 con 0x7f4c0006d000 2024-03-06T07:22:07.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.922+0000 7f4bf5ffb640 1 -- 172.21.15.83:0/3952791940 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bfc0052d0 con 0x7f4c0006d000 2024-03-06T07:22:07.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.920+0000 7fb8f3481640 1 -- 172.21.15.83:0/2633103697 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8e40d6d70 con 0x7fb8e40df800 2024-03-06T07:22:07.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.921+0000 7fb8cf7fe640 1 -- 172.21.15.83:0/2633103697 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb8c8001030 con 0x7fb8e40df800 2024-03-06T07:22:07.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.922+0000 7f03c7fff640 1 -- 172.21.15.83:0/2062158780 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c802ca50 con 0x7f03d00d3710 2024-03-06T07:22:07.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.922+0000 7f03c7fff640 1 -- 172.21.15.83:0/2062158780 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03c802cbf0 con 0x7f03d00d3710 2024-03-06T07:22:07.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.922+0000 7f03c7fff640 1 -- 172.21.15.83:0/2062158780 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c80343f0 con 0x7f03d00d3710 2024-03-06T07:22:07.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.922+0000 7f79cffff640 1 -- 172.21.15.83:0/4052470765 <== osd.1 v2:172.21.15.83:6814/1822925240 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f79d0004020 con 0x7f79c80736f0 2024-03-06T07:22:07.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.923+0000 7f4bf5ffb640 1 -- 172.21.15.83:0/3952791940 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc003810 con 0x7f4c0006d000 2024-03-06T07:22:07.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.923+0000 7f4c07981640 1 -- 172.21.15.83:0/3952791940 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c00161710 con 0x7f4c0006d000 2024-03-06T07:22:07.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.923+0000 7f4c07981640 1 -- 172.21.15.83:0/3952791940 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c00161cd0 con 0x7f4c0006d000 2024-03-06T07:22:07.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.923+0000 7f03d6226640 1 -- 172.21.15.83:0/2062158780 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f03d00d3710 msgr2=0x7f03d016d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.923+0000 7f03d6226640 1 --2- 172.21.15.83:0/2062158780 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f03d00d3710 0x7f03d016d0c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f03c8009e80 tx=0x7f03c802beb0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.923+0000 7fb8e1ffb640 1 -- 172.21.15.83:0/2633103697 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb8ec070690 con 0x7fb8e40df800 2024-03-06T07:22:07.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.923+0000 7fb8e1ffb640 1 --2- 172.21.15.83:0/2633103697 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb8d006b740 0x7fb8d006dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.923+0000 7fb8e1ffb640 1 -- 172.21.15.83:0/2633103697 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb8ec105370 con 0x7fb8e40df800 2024-03-06T07:22:07.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.923+0000 7fb8e1ffb640 1 --2- 172.21.15.83:0/2633103697 >> [v2:172.21.15.183:6800/2649185225,v1:172.21.15.183:6801/2649185225] conn(0x7fb8d00739c0 0x7fb8d0075de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.923+0000 7fb8e1ffb640 1 -- 172.21.15.83:0/2633103697 --> [v2:172.21.15.183:6800/2649185225,v1:172.21.15.183:6801/2649185225] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb8ec06ff30 con 0x7fb8d00739c0 2024-03-06T07:22:07.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.923+0000 7fb8e1ffb640 1 -- 172.21.15.83:0/2633103697 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb8ec1056e0 con 0x7fb8e40df800 2024-03-06T07:22:07.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.924+0000 7f03d6226640 1 -- 172.21.15.83:0/2062158780 shutdown_connections 2024-03-06T07:22:07.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.924+0000 7f03d6226640 1 --2- 172.21.15.83:0/2062158780 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f03d00d3710 0x7f03d016d0c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.924+0000 7f03d6226640 1 -- 172.21.15.83:0/2062158780 >> 172.21.15.83:0/2062158780 conn(0x7f03d00d3cd0 msgr2=0x7f03d00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.924+0000 7fb8f121d640 1 --2- 172.21.15.83:0/2633103697 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb8d006b740 0x7fb8d006dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.924+0000 7fb8f0a1c640 1 --2- 172.21.15.83:0/2633103697 >> [v2:172.21.15.183:6800/2649185225,v1:172.21.15.183:6801/2649185225] conn(0x7fb8d00739c0 0x7fb8d0075de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.924+0000 7f03d6226640 1 -- 172.21.15.83:0/2062158780 shutdown_connections 2024-03-06T07:22:07.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.925+0000 7fb8f121d640 1 --2- 172.21.15.83:0/2633103697 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb8d006b740 0x7fb8d006dc00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc009660 tx=0x7fb8dc006b80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.925+0000 7fb8f0a1c640 1 --2- 172.21.15.83:0/2633103697 >> [v2:172.21.15.183:6800/2649185225,v1:172.21.15.183:6801/2649185225] conn(0x7fb8d00739c0 0x7fb8d0075de0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.924+0000 7f03d6226640 1 -- 172.21.15.83:0/2062158780 wait complete. 2024-03-06T07:22:07.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.926+0000 7f03d6226640 1 Processor -- start 2024-03-06T07:22:07.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.926+0000 7f03d6226640 1 -- start start 2024-03-06T07:22:07.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.926+0000 7f03d6226640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f03d00730c0 0x7f03d00c68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.926+0000 7fb8e1ffb640 1 -- 172.21.15.83:0/2633103697 <== osd.6 v2:172.21.15.183:6800/2649185225 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb8ec06ff30 con 0x7fb8d00739c0 2024-03-06T07:22:07.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.926+0000 7f03d6226640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f03d00d3710 0x7f03d01f5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.926+0000 7f03d6226640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f03d0162140 0x7f03d01f5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.926+0000 7f03d6226640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d0162520 con 0x7f03d00730c0 2024-03-06T07:22:07.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.926+0000 7f03d6226640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d01667d0 con 0x7f03d00d3710 2024-03-06T07:22:07.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.926+0000 7f03d6226640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03d01626a0 con 0x7f03d0162140 2024-03-06T07:22:07.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.927+0000 7f03d5224640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f03d00730c0 0x7f03d00c68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.927+0000 7f03d5224640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f03d00730c0 0x7f03d00c68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49414/0 (socket says 172.21.15.83:49414) 2024-03-06T07:22:07.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.927+0000 7f03d5224640 1 -- 172.21.15.83:0/1864028690 learned_addr learned my addr 172.21.15.83:0/1864028690 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:07.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.927+0000 7f03d5a25640 1 --2- 172.21.15.83:0/1864028690 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f03d0162140 0x7f03d01f5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.927+0000 7f03d5224640 1 -- 172.21.15.83:0/1864028690 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f03d0162140 msgr2=0x7f03d01f5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.927+0000 7f03d5224640 1 --2- 172.21.15.83:0/1864028690 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f03d0162140 0x7f03d01f5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.927+0000 7f03d5224640 1 -- 172.21.15.83:0/1864028690 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f03d00d3710 msgr2=0x7f03d01f5250 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-06T07:22:07.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.927+0000 7f03d5224640 1 --2- 172.21.15.83:0/1864028690 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f03d00d3710 0x7f03d01f5250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.927+0000 7f03d5224640 1 -- 172.21.15.83:0/1864028690 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d01f4630 con 0x7f03d00730c0 2024-03-06T07:22:07.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.928+0000 7f03d5224640 1 --2- 172.21.15.83:0/1864028690 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f03d00730c0 0x7f03d00c68d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f03cc00b7b0 tx=0x7f03cc00bc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.928+0000 7f03c5ffb640 1 -- 172.21.15.83:0/1864028690 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03cc019070 con 0x7f03d00730c0 2024-03-06T07:22:07.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.928+0000 7f03c5ffb640 1 -- 172.21.15.83:0/1864028690 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03cc00c720 con 0x7f03d00730c0 2024-03-06T07:22:07.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.928+0000 7f03d6226640 1 -- 172.21.15.83:0/1864028690 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03d00c6e10 con 0x7f03d00730c0 2024-03-06T07:22:07.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.928+0000 7f03c5ffb640 1 -- 172.21.15.83:0/1864028690 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03cc00caf0 con 0x7f03d00730c0 2024-03-06T07:22:07.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.928+0000 7f03d6226640 1 -- 172.21.15.83:0/1864028690 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03d01f60a0 con 0x7f03d00730c0 2024-03-06T07:22:07.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.929+0000 7f03d6226640 1 -- 172.21.15.83:0/1864028690 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f03d0068f80 con 0x7f03d00730c0 2024-03-06T07:22:07.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.931+0000 7f03c5ffb640 1 -- 172.21.15.83:0/1864028690 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f03cc01c070 con 0x7f03d00730c0 2024-03-06T07:22:07.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.931+0000 7f03c5ffb640 1 --2- 172.21.15.83:0/1864028690 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f03b406b5a0 0x7f03b406da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.931+0000 7f03c5ffb640 1 -- 172.21.15.83:0/1864028690 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f03cc08a100 con 0x7f03d00730c0 2024-03-06T07:22:07.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.931+0000 7f03c5ffb640 1 --2- 172.21.15.83:0/1864028690 >> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] conn(0x7f03b4073820 0x7f03b4075c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.931+0000 7f03c5ffb640 1 -- 172.21.15.83:0/1864028690 --> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f03cc010e50 con 0x7f03b4073820 2024-03-06T07:22:07.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.931+0000 7f03c5ffb640 1 -- 172.21.15.83:0/1864028690 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f03cc08a4f0 con 0x7f03d00730c0 2024-03-06T07:22:07.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.932+0000 7f03d5a25640 1 --2- 172.21.15.83:0/1864028690 >> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] conn(0x7f03b4073820 0x7f03b4075c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.932+0000 7f03d4a23640 1 --2- 172.21.15.83:0/1864028690 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f03b406b5a0 0x7f03b406da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.932+0000 7f03d5a25640 1 --2- 172.21.15.83:0/1864028690 >> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] conn(0x7f03b4073820 0x7f03b4075c40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.932+0000 7f03d4a23640 1 --2- 172.21.15.83:0/1864028690 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f03b406b5a0 0x7f03b406da60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f03d006d580 tx=0x7f03c00079b0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.933+0000 7f03c5ffb640 1 -- 172.21.15.83:0/1864028690 <== osd.3 v2:172.21.15.83:6815/2656563756 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f03cc010e50 con 0x7f03b4073820 2024-03-06T07:22:07.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.940+0000 7f4bf5ffb640 1 -- 172.21.15.83:0/3952791940 <== mon.2 v2:172.21.15.183:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4bfc01a070 con 0x7f4c0006d000 2024-03-06T07:22:07.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.943+0000 7f4c07981640 1 -- 172.21.15.83:0/3952791940 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4c0004f330 con 0x7f4c0006d000 2024-03-06T07:22:07.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.943+0000 7f5bed365640 1 Processor -- start 2024-03-06T07:22:07.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.943+0000 7f5bed365640 1 -- start start 2024-03-06T07:22:07.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.945+0000 7f5bed365640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5be806ebe0 0x7f5be80b4060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.945+0000 7f5bed365640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5be80b45a0 0x7f5be806cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.945+0000 7f5bed365640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5be815b380 con 0x7f5be8157b20 2024-03-06T07:22:07.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.945+0000 7f5bed365640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5be80b4a10 con 0x7f5be806ebe0 2024-03-06T07:22:07.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.945+0000 7f5bed365640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5be815e420 con 0x7f5be80b45a0 2024-03-06T07:22:07.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.945+0000 7f5be6ffd640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f5be8157b20 0x7f5be806e4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:37882/0 (socket says 172.21.15.83:37882) 2024-03-06T07:22:07.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.945+0000 7f5be6ffd640 1 -- 172.21.15.83:0/2725495788 learned_addr learned my addr 172.21.15.83:0/2725495788 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:07.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.945+0000 7f5be77fe640 1 --2- 172.21.15.83:0/2725495788 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5be80b45a0 0x7f5be806cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.945+0000 7f5be5ffb640 1 -- 172.21.15.83:0/2725495788 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3780165291 0 0) 0x7f5be815b380 con 0x7f5be8157b20 2024-03-06T07:22:07.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.945+0000 7f5be5ffb640 1 -- 172.21.15.83:0/2725495788 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bd0003520 con 0x7f5be8157b20 2024-03-06T07:22:07.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.945+0000 7f5be77fe640 1 -- 172.21.15.83:0/2725495788 >> v2:172.21.15.83:3300/0 conn(0x7f5be806ebe0 msgr2=0x7f5be80b4060 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:22:07.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.945+0000 7f5be77fe640 1 --2- 172.21.15.83:0/2725495788 >> v2:172.21.15.83:3300/0 conn(0x7f5be806ebe0 0x7f5be80b4060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.945+0000 7f5be77fe640 1 -- 172.21.15.83:0/2725495788 >> v1:172.21.15.183:6789/0 conn(0x7f5be8157b20 legacy=0x7f5be806e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.945+0000 7f5be77fe640 1 -- 172.21.15.83:0/2725495788 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5be815e8b0 con 0x7f5be80b45a0 2024-03-06T07:22:07.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.946+0000 7f4564ff9640 1 -- 172.21.15.83:0/3444966563 --> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4558002dd0 con 0x7f4554073740 2024-03-06T07:22:07.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.944+0000 7f4bf5ffb640 1 --2- 172.21.15.83:0/3952791940 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4be406b520 0x7f4be406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.945+0000 7f4bf5ffb640 1 -- 172.21.15.83:0/3952791940 <== mon.2 v2:172.21.15.183:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4bfc0891b0 con 0x7f4c0006d000 2024-03-06T07:22:07.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.945+0000 7f4bf5ffb640 1 --2- 172.21.15.83:0/3952791940 >> [v2:172.21.15.183:6808/2957306877,v1:172.21.15.183:6809/2957306877] conn(0x7f4be40736c0 0x7f4be4075ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.945+0000 7f4bf5ffb640 1 -- 172.21.15.83:0/3952791940 --> [v2:172.21.15.183:6808/2957306877,v1:172.21.15.183:6809/2957306877] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4bfc009e50 con 0x7f4be40736c0 2024-03-06T07:22:07.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.945+0000 7f4bf5ffb640 1 -- 172.21.15.83:0/3952791940 <== mon.2 v2:172.21.15.183:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4bfc090050 con 0x7f4c0006d000 2024-03-06T07:22:07.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.945+0000 7f4c04f1c640 1 --2- 172.21.15.83:0/3952791940 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4be406b520 0x7f4be406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.946+0000 7f4c05f1e640 1 --2- 172.21.15.83:0/3952791940 >> [v2:172.21.15.183:6808/2957306877,v1:172.21.15.183:6809/2957306877] conn(0x7f4be40736c0 0x7f4be4075ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.946+0000 7f4c04f1c640 1 --2- 172.21.15.83:0/3952791940 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4be406b520 0x7f4be406d9e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4c000cf5c0 tx=0x7f4bf0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.947+0000 7f5be77fe640 1 --2- 172.21.15.83:0/2725495788 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5be80b45a0 0x7f5be806cfa0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5bd800ba50 tx=0x7f5bd802e090 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df4621696d62df8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.947+0000 7f4c05f1e640 1 --2- 172.21.15.83:0/3952791940 >> [v2:172.21.15.183:6808/2957306877,v1:172.21.15.183:6809/2957306877] conn(0x7f4be40736c0 0x7f4be4075ae0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.947+0000 7f5be5ffb640 1 -- 172.21.15.83:0/2725495788 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd8034070 con 0x7f5be80b45a0 2024-03-06T07:22:07.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.947+0000 7f4566ffd640 1 -- 172.21.15.83:0/3444966563 <== osd.0 v2:172.21.15.83:6808/1635427425 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f4558002dd0 con 0x7f4554073740 2024-03-06T07:22:07.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.947+0000 7f4bf5ffb640 1 -- 172.21.15.83:0/3952791940 <== osd.5 v2:172.21.15.183:6808/2957306877 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f4bfc009e50 con 0x7f4be40736c0 2024-03-06T07:22:07.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.947+0000 7f5be5ffb640 1 -- 172.21.15.83:0/2725495788 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bd802ee40 con 0x7f5be80b45a0 2024-03-06T07:22:07.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.947+0000 7f5be5ffb640 1 -- 172.21.15.83:0/2725495788 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd8034070 con 0x7f5be80b45a0 2024-03-06T07:22:07.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.948+0000 7f4914ff9640 1 -- 172.21.15.83:0/886514769 --> [v2:172.21.15.83:6800/2591342133,v1:172.21.15.83:6801/2591342133] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4908002dd0 con 0x7f49040737f0 2024-03-06T07:22:07.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.948+0000 7f5bed365640 1 -- 172.21.15.83:0/2725495788 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5be80b45a0 msgr2=0x7f5be806cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.948+0000 7f5bed365640 1 --2- 172.21.15.83:0/2725495788 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5be80b45a0 0x7f5be806cfa0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5bd800ba50 tx=0x7f5bd802e090 comp rx=0 tx=0).stop 2024-03-06T07:22:07.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.948+0000 7f5bed365640 1 -- 172.21.15.83:0/2725495788 shutdown_connections 2024-03-06T07:22:07.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.948+0000 7f5bed365640 1 --2- 172.21.15.83:0/2725495788 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5be80b45a0 0x7f5be806cfa0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.948+0000 7f5bed365640 1 --2- 172.21.15.83:0/2725495788 >> v2:172.21.15.83:3300/0 conn(0x7f5be806ebe0 0x7f5be80b4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.948+0000 7f4916ffd640 1 -- 172.21.15.83:0/886514769 <== osd.2 v2:172.21.15.83:6800/2591342133 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f4908002dd0 con 0x7f49040737f0 2024-03-06T07:22:07.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.948+0000 7f5bed365640 1 -- 172.21.15.83:0/2725495788 >> 172.21.15.83:0/2725495788 conn(0x7f5be80b1c50 msgr2=0x7f5be80b2030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.948+0000 7f5bed365640 1 -- 172.21.15.83:0/2725495788 shutdown_connections 2024-03-06T07:22:07.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.948+0000 7f5bed365640 1 -- 172.21.15.83:0/2725495788 wait complete. 2024-03-06T07:22:07.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.949+0000 7f49284fb640 1 -- 172.21.15.83:0/886514769 >> [v2:172.21.15.83:6800/2591342133,v1:172.21.15.83:6801/2591342133] conn(0x7f49040737f0 msgr2=0x7f4904075c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.949+0000 7f49284fb640 1 --2- 172.21.15.83:0/886514769 >> [v2:172.21.15.83:6800/2591342133,v1:172.21.15.83:6801/2591342133] conn(0x7f49040737f0 0x7f4904075c10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.949+0000 7f4564ff9640 1 -- 172.21.15.83:0/3444966563 >> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] conn(0x7f4554073740 msgr2=0x7f4554075b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.949+0000 7f4564ff9640 1 --2- 172.21.15.83:0/3444966563 >> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] conn(0x7f4554073740 0x7f4554075b60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.949+0000 7f49284fb640 1 -- 172.21.15.83:0/886514769 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f490406b5d0 msgr2=0x7f490406da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.949+0000 7f49284fb640 1 --2- 172.21.15.83:0/886514769 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f490406b5d0 0x7f490406da90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4918009e50 tx=0x7f4918030000 comp rx=0 tx=0).stop 2024-03-06T07:22:07.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.949+0000 7f49284fb640 1 -- 172.21.15.83:0/886514769 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4920165cc0 msgr2=0x7f492014d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.949+0000 7f49284fb640 1 --2- 172.21.15.83:0/886514769 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4920165cc0 0x7f492014d090 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f491c00c3f0 tx=0x7f491c00c8b0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.950+0000 7f4564ff9640 1 -- 172.21.15.83:0/3444966563 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f455406b5a0 msgr2=0x7f455406da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.950+0000 7f4564ff9640 1 --2- 172.21.15.83:0/3444966563 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f455406b5a0 0x7f455406da60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f456c007fd0 tx=0x7f456c009040 comp rx=0 tx=0).stop 2024-03-06T07:22:07.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.950+0000 7f4564ff9640 1 -- 172.21.15.83:0/3444966563 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f45700b42b0 msgr2=0x7f4570169cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.950+0000 7f4564ff9640 1 --2- 172.21.15.83:0/3444966563 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f45700b42b0 0x7f4570169cf0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f45680078c0 tx=0x7f4568004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:07.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.950+0000 7f4576852640 1 -- 172.21.15.83:0/3444966563 reap_dead start 2024-03-06T07:22:07.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.950+0000 7f4926a98640 1 -- 172.21.15.83:0/886514769 reap_dead start 2024-03-06T07:22:07.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.950+0000 7f5bed365640 1 Processor -- start 2024-03-06T07:22:07.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.950+0000 7f5bed365640 1 -- start start 2024-03-06T07:22:07.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.951+0000 7f4564ff9640 1 -- 172.21.15.83:0/3444966563 shutdown_connections 2024-03-06T07:22:07.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.951+0000 7f4564ff9640 1 -- 172.21.15.83:0/3444966563 >> 172.21.15.83:0/3444966563 conn(0x7f457006afb0 msgr2=0x7f45700af630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.951+0000 7f49284fb640 1 -- 172.21.15.83:0/886514769 shutdown_connections 2024-03-06T07:22:07.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.951+0000 7f49284fb640 1 -- 172.21.15.83:0/886514769 >> 172.21.15.83:0/886514769 conn(0x7f49200b16c0 msgr2=0x7f49200afca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.952+0000 7f5bed365640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5be806ebe0 0x7f5be81510b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.952+0000 7f5bed365640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5be80b45a0 0x7f5be80b3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.952+0000 7f4564ff9640 1 -- 172.21.15.83:0/3444966563 shutdown_connections 2024-03-06T07:22:07.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.952+0000 7f4564ff9640 1 -- 172.21.15.83:0/3444966563 wait complete. 2024-03-06T07:22:07.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.952+0000 7f5bed365640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5be8157b20 0x7f5be80a9680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.952+0000 7f5bed365640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be815ddd0 con 0x7f5be8157b20 2024-03-06T07:22:07.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.952+0000 7f5bed365640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be815daf0 con 0x7f5be80b45a0 2024-03-06T07:22:07.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.952+0000 7f5bed365640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5be815e0b0 con 0x7f5be806ebe0 2024-03-06T07:22:07.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.952+0000 7f5be67fc640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5be80b45a0 0x7f5be80b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.952+0000 7f5be6ffd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5be806ebe0 0x7f5be81510b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.952+0000 7f5be6ffd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5be806ebe0 0x7f5be81510b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:59242/0 (socket says 172.21.15.83:59242) 2024-03-06T07:22:07.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.952+0000 7f5be6ffd640 1 -- 172.21.15.83:0/975970051 learned_addr learned my addr 172.21.15.83:0/975970051 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:07.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.952+0000 7f5be77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5be8157b20 0x7f5be80a9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.952+0000 7f5be6ffd640 1 -- 172.21.15.83:0/975970051 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5be80b45a0 msgr2=0x7f5be80b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.953+0000 7f5be6ffd640 1 --2- 172.21.15.83:0/975970051 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5be80b45a0 0x7f5be80b3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.953+0000 7f5be6ffd640 1 -- 172.21.15.83:0/975970051 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5be8157b20 msgr2=0x7f5be80a9680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.953+0000 7f5be6ffd640 1 --2- 172.21.15.83:0/975970051 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5be8157b20 0x7f5be80a9680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.953+0000 7f5be6ffd640 1 -- 172.21.15.83:0/975970051 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5be8151a00 con 0x7f5be806ebe0 2024-03-06T07:22:07.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.953+0000 7f5be6ffd640 1 --2- 172.21.15.83:0/975970051 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5be806ebe0 0x7f5be81510b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5be000c700 tx=0x7f5be000cbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.953+0000 7f5bcf7fe640 1 -- 172.21.15.83:0/975970051 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be0016020 con 0x7f5be806ebe0 2024-03-06T07:22:07.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.953+0000 7f5bed365640 1 -- 172.21.15.83:0/975970051 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5be8151ce0 con 0x7f5be806ebe0 2024-03-06T07:22:07.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.953+0000 7f5bed365640 1 -- 172.21.15.83:0/975970051 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5be81522a0 con 0x7f5be806ebe0 2024-03-06T07:22:07.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.954+0000 7f5bcf7fe640 1 -- 172.21.15.83:0/975970051 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5be000aa70 con 0x7f5be806ebe0 2024-03-06T07:22:07.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.954+0000 7f5bcf7fe640 1 -- 172.21.15.83:0/975970051 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be000ae60 con 0x7f5be806ebe0 2024-03-06T07:22:07.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.956+0000 7f5bcf7fe640 1 -- 172.21.15.83:0/975970051 <== mon.2 v2:172.21.15.183:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5be0019020 con 0x7f5be806ebe0 2024-03-06T07:22:07.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.956+0000 7f49284fb640 1 -- 172.21.15.83:0/886514769 shutdown_connections 2024-03-06T07:22:07.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.956+0000 7f49284fb640 1 -- 172.21.15.83:0/886514769 wait complete. 2024-03-06T07:22:07.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.958+0000 7f5bcf7fe640 1 --2- 172.21.15.83:0/975970051 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5bd406b4d0 0x7f5bd406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.958+0000 7f5bcf7fe640 1 -- 172.21.15.83:0/975970051 <== mon.2 v2:172.21.15.183:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5be008a4b0 con 0x7f5be806ebe0 2024-03-06T07:22:07.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.958+0000 7f5bed365640 1 --2- 172.21.15.83:0/975970051 >> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] conn(0x7f5bc8001650 0x7f5bc8003b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.958+0000 7f03d6226640 1 -- 172.21.15.83:0/1864028690 --> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f03d00b37d0 con 0x7f03b4073820 2024-03-06T07:22:07.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.958+0000 7f5be67fc640 1 --2- 172.21.15.83:0/975970051 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5bd406b4d0 0x7f5bd406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.958+0000 7f5bed365640 1 -- 172.21.15.83:0/975970051 --> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5bc8006bb0 con 0x7f5bc8001650 2024-03-06T07:22:07.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.959+0000 7f5be77fe640 1 --2- 172.21.15.83:0/975970051 >> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] conn(0x7f5bc8001650 0x7f5bc8003b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.959+0000 7f03c5ffb640 1 -- 172.21.15.83:0/1864028690 <== osd.3 v2:172.21.15.83:6815/2656563756 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f03d00b37d0 con 0x7f03b4073820 2024-03-06T07:22:07.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.959+0000 7f5be67fc640 1 --2- 172.21.15.83:0/975970051 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5bd406b4d0 0x7f5bd406d990 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc006fd0 tx=0x7f5bdc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.959+0000 7f5be77fe640 1 --2- 172.21.15.83:0/975970051 >> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] conn(0x7f5bc8001650 0x7f5bc8003b10 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.960+0000 7f03bb7fe640 1 -- 172.21.15.83:0/1864028690 >> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] conn(0x7f03b4073820 msgr2=0x7f03b4075c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.960+0000 7f03bb7fe640 1 --2- 172.21.15.83:0/1864028690 >> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] conn(0x7f03b4073820 0x7f03b4075c40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.960+0000 7f03bb7fe640 1 -- 172.21.15.83:0/1864028690 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f03b406b5a0 msgr2=0x7f03b406da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.960+0000 7f03bb7fe640 1 --2- 172.21.15.83:0/1864028690 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f03b406b5a0 0x7f03b406da60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f03d006d580 tx=0x7f03c00079b0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.960+0000 7f03bb7fe640 1 -- 172.21.15.83:0/1864028690 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f03d00730c0 msgr2=0x7f03d00c68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.960+0000 7f03bb7fe640 1 --2- 172.21.15.83:0/1864028690 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f03d00730c0 0x7f03d00c68d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f03cc00b7b0 tx=0x7f03cc00bc70 comp rx=0 tx=0).stop 2024-03-06T07:22:07.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.960+0000 7f03d5a25640 1 -- 172.21.15.83:0/1864028690 reap_dead start 2024-03-06T07:22:07.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.960+0000 7f5bcf7fe640 1 -- 172.21.15.83:0/975970051 <== osd.7 v2:172.21.15.183:6820/3336566695 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5bc8006bb0 con 0x7f5bc8001650 2024-03-06T07:22:07.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.960+0000 7f79cdffb640 1 -- 172.21.15.83:0/4052470765 --> [v2:172.21.15.83:6814/1822925240,v1:172.21.15.83:6817/1822925240] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f79bc002dd0 con 0x7f79c80736f0 2024-03-06T07:22:07.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.960+0000 7f03bb7fe640 1 -- 172.21.15.83:0/1864028690 shutdown_connections 2024-03-06T07:22:07.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.960+0000 7f03bb7fe640 1 -- 172.21.15.83:0/1864028690 >> 172.21.15.83:0/1864028690 conn(0x7f03d00d3cd0 msgr2=0x7f03d00b10e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.961+0000 7f79cffff640 1 -- 172.21.15.83:0/4052470765 <== osd.1 v2:172.21.15.83:6814/1822925240 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f79bc002dd0 con 0x7f79c80736f0 2024-03-06T07:22:07.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.962+0000 7f79e9ceb640 1 -- 172.21.15.83:0/4052470765 >> [v2:172.21.15.83:6814/1822925240,v1:172.21.15.83:6817/1822925240] conn(0x7f79c80736f0 msgr2=0x7f79c8075b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.962+0000 7f79e9ceb640 1 --2- 172.21.15.83:0/4052470765 >> [v2:172.21.15.83:6814/1822925240,v1:172.21.15.83:6817/1822925240] conn(0x7f79c80736f0 0x7f79c8075b10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.962+0000 7fb8cf7fe640 1 -- 172.21.15.83:0/2633103697 --> [v2:172.21.15.183:6800/2649185225,v1:172.21.15.183:6801/2649185225] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb8c8002dd0 con 0x7fb8d00739c0 2024-03-06T07:22:07.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.962+0000 7f79e9ceb640 1 -- 172.21.15.83:0/4052470765 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f79c806b550 msgr2=0x7f79c806da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.962+0000 7f79e9ceb640 1 --2- 172.21.15.83:0/4052470765 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f79c806b550 0x7f79c806da10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f79dc003e80 tx=0x7f79d8008040 comp rx=0 tx=0).stop 2024-03-06T07:22:07.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.962+0000 7f79e9ceb640 1 -- 172.21.15.83:0/4052470765 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f79dc051420 msgr2=0x7f79dc0518d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.962+0000 7f79e9ceb640 1 --2- 172.21.15.83:0/4052470765 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f79dc051420 0x7f79dc0518d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f79dc038b70 tx=0x7f79d000c820 comp rx=0 tx=0).stop 2024-03-06T07:22:07.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.962+0000 7f79e3fff640 1 -- 172.21.15.83:0/4052470765 reap_dead start 2024-03-06T07:22:07.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.962+0000 7f03bb7fe640 1 -- 172.21.15.83:0/1864028690 shutdown_connections 2024-03-06T07:22:07.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.963+0000 7f79e9ceb640 1 -- 172.21.15.83:0/4052470765 shutdown_connections 2024-03-06T07:22:07.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.963+0000 7f79e9ceb640 1 -- 172.21.15.83:0/4052470765 >> 172.21.15.83:0/4052470765 conn(0x7f79dc01ef40 msgr2=0x7f79dc037cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.963+0000 7f79e9ceb640 1 -- 172.21.15.83:0/4052470765 shutdown_connections 2024-03-06T07:22:07.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.963+0000 7f79e9ceb640 1 -- 172.21.15.83:0/4052470765 wait complete. 2024-03-06T07:22:07.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.963+0000 7fb8e1ffb640 1 -- 172.21.15.83:0/2633103697 <== osd.6 v2:172.21.15.183:6800/2649185225 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb8c8002dd0 con 0x7fb8d00739c0 2024-03-06T07:22:07.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.964+0000 7fb8cf7fe640 1 -- 172.21.15.83:0/2633103697 >> [v2:172.21.15.183:6800/2649185225,v1:172.21.15.183:6801/2649185225] conn(0x7fb8d00739c0 msgr2=0x7fb8d0075de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.964+0000 7fb8cf7fe640 1 --2- 172.21.15.83:0/2633103697 >> [v2:172.21.15.183:6800/2649185225,v1:172.21.15.183:6801/2649185225] conn(0x7fb8d00739c0 0x7fb8d0075de0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.963+0000 7f03bb7fe640 1 -- 172.21.15.83:0/1864028690 wait complete. 2024-03-06T07:22:07.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.966+0000 7fb8cf7fe640 1 -- 172.21.15.83:0/2633103697 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb8d006b740 msgr2=0x7fb8d006dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.966+0000 7fb8cf7fe640 1 --2- 172.21.15.83:0/2633103697 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb8d006b740 0x7fb8d006dc00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc009660 tx=0x7fb8dc006b80 comp rx=0 tx=0).stop 2024-03-06T07:22:07.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.966+0000 7fb8cf7fe640 1 -- 172.21.15.83:0/2633103697 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb8e40df800 msgr2=0x7fb8e4048f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.966+0000 7fb8cf7fe640 1 --2- 172.21.15.83:0/2633103697 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb8e40df800 0x7fb8e4048f20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec04fec0 tx=0x7fb8ec068ad0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.966+0000 7fb8f1a1e640 1 -- 172.21.15.83:0/2633103697 reap_dead start 2024-03-06T07:22:08.000 INFO:teuthology.orchestra.run.smithi083.stdout:55834574856 2024-03-06T07:22:08.000 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-06T07:22:08.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.968+0000 7fb8cf7fe640 1 -- 172.21.15.83:0/2633103697 shutdown_connections 2024-03-06T07:22:08.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.968+0000 7fb8cf7fe640 1 -- 172.21.15.83:0/2633103697 >> 172.21.15.83:0/2633103697 conn(0x7fb8e401e6c0 msgr2=0x7fb8e40ddac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.968+0000 7fb8cf7fe640 1 -- 172.21.15.83:0/2633103697 shutdown_connections 2024-03-06T07:22:08.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.968+0000 7fb8cf7fe640 1 -- 172.21.15.83:0/2633103697 wait complete. 2024-03-06T07:22:08.002 INFO:teuthology.orchestra.run.smithi083.stdout:55834574856 2024-03-06T07:22:08.002 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-06T07:22:08.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.980+0000 7f4c07981640 1 -- 172.21.15.83:0/3952791940 --> [v2:172.21.15.183:6808/2957306877,v1:172.21.15.183:6809/2957306877] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4c001f8210 con 0x7f4be40736c0 2024-03-06T07:22:08.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.980+0000 7f4bf5ffb640 1 -- 172.21.15.83:0/3952791940 <== osd.5 v2:172.21.15.183:6808/2957306877 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f4c001f8210 con 0x7f4be40736c0 2024-03-06T07:22:08.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.981+0000 7f4c07981640 1 -- 172.21.15.83:0/3952791940 >> [v2:172.21.15.183:6808/2957306877,v1:172.21.15.183:6809/2957306877] conn(0x7f4be40736c0 msgr2=0x7f4be4075ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.981+0000 7f4c07981640 1 --2- 172.21.15.83:0/3952791940 >> [v2:172.21.15.183:6808/2957306877,v1:172.21.15.183:6809/2957306877] conn(0x7f4be40736c0 0x7f4be4075ae0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.981+0000 7f4c07981640 1 -- 172.21.15.83:0/3952791940 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4be406b520 msgr2=0x7f4be406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.981+0000 7f4c07981640 1 --2- 172.21.15.83:0/3952791940 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4be406b520 0x7f4be406d9e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4c000cf5c0 tx=0x7f4bf0009040 comp rx=0 tx=0).stop 2024-03-06T07:22:08.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.982+0000 7f4c07981640 1 -- 172.21.15.83:0/3952791940 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4c0006d000 msgr2=0x7f4c0015a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.982+0000 7f4c07981640 1 --2- 172.21.15.83:0/3952791940 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4c0006d000 0x7f4c0015a030 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc00c3c0 tx=0x7f4bfc00c880 comp rx=0 tx=0).stop 2024-03-06T07:22:08.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.982+0000 7f4c05f1e640 1 -- 172.21.15.83:0/3952791940 reap_dead start 2024-03-06T07:22:08.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.982+0000 7f4c07981640 1 -- 172.21.15.83:0/3952791940 shutdown_connections 2024-03-06T07:22:08.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.982+0000 7f4c07981640 1 -- 172.21.15.83:0/3952791940 >> 172.21.15.83:0/3952791940 conn(0x7f4c000b1c80 msgr2=0x7f4c00071960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.005 INFO:teuthology.orchestra.run.smithi083.stdout:55834574856 2024-03-06T07:22:08.006 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-06T07:22:08.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.982+0000 7f4c07981640 1 -- 172.21.15.83:0/3952791940 shutdown_connections 2024-03-06T07:22:08.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.982+0000 7f4c07981640 1 -- 172.21.15.83:0/3952791940 wait complete. 2024-03-06T07:22:08.007 INFO:teuthology.orchestra.run.smithi083.stdout:55834574857 2024-03-06T07:22:08.007 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-06T07:22:08.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.983+0000 7fe2061b6640 1 Processor -- start 2024-03-06T07:22:08.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.983+0000 7fe2061b6640 1 -- start start 2024-03-06T07:22:08.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.983+0000 7fe2061b6640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fe200158880 0x7fe20006c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.983+0000 7fe2061b6640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe20015b380 0x7fe2000708d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.984+0000 7fe2061b6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe20006d030 con 0x7fe20006cb90 2024-03-06T07:22:08.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.984+0000 7fe2061b6640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe20006d310 con 0x7fe20015b380 2024-03-06T07:22:08.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.984+0000 7fe2061b6640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe20006d5f0 con 0x7fe200158880 2024-03-06T07:22:08.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.984+0000 7fe1fffff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fe20006cb90 0x7fe20015e800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:42864/0 (socket says 172.21.15.83:42864) 2024-03-06T07:22:08.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.984+0000 7fe1fffff640 1 -- 172.21.15.83:0/3826581103 learned_addr learned my addr 172.21.15.83:0/3826581103 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:08.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.984+0000 7fe1feffd640 1 --2- 172.21.15.83:0/3826581103 >> v2:172.21.15.83:3300/0 conn(0x7fe20015b380 0x7fe2000708d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.984+0000 7fe1ff7fe640 1 --2- 172.21.15.83:0/3826581103 >> v2:172.21.15.183:3300/0 conn(0x7fe200158880 0x7fe20006c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.984+0000 7fe1feffd640 1 -- 172.21.15.83:0/3826581103 >> v2:172.21.15.183:3300/0 conn(0x7fe200158880 msgr2=0x7fe20006c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:08.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.984+0000 7fe1feffd640 1 --2- 172.21.15.83:0/3826581103 >> v2:172.21.15.183:3300/0 conn(0x7fe200158880 0x7fe20006c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.984+0000 7fe1feffd640 1 -- 172.21.15.83:0/3826581103 >> v1:172.21.15.83:6789/0 conn(0x7fe20006cb90 legacy=0x7fe20015e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.984+0000 7fe1feffd640 1 -- 172.21.15.83:0/3826581103 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe20006d8d0 con 0x7fe20015b380 2024-03-06T07:22:08.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.985+0000 7fe1feffd640 1 --2- 172.21.15.83:0/3826581103 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe20015b380 0x7fe2000708d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe2000cd510 tx=0x7fe1f002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=756fce58a8eaafc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.011 INFO:teuthology.orchestra.run.smithi083.stdout:55834574855 2024-03-06T07:22:08.011 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-06T07:22:08.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.985+0000 7fe1fe7fc640 1 -- 172.21.15.83:0/3826581103 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f0035020 con 0x7fe20015b380 2024-03-06T07:22:08.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.985+0000 7fe1fe7fc640 1 -- 172.21.15.83:0/3826581103 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1f0031830 con 0x7fe20015b380 2024-03-06T07:22:08.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.985+0000 7fe1fe7fc640 1 -- 172.21.15.83:0/3826581103 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f0031c00 con 0x7fe20015b380 2024-03-06T07:22:08.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.986+0000 7fe2061b6640 1 -- 172.21.15.83:0/3826581103 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe20015b380 msgr2=0x7fe2000708d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.986+0000 7fe2061b6640 1 --2- 172.21.15.83:0/3826581103 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe20015b380 0x7fe2000708d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe2000cd510 tx=0x7fe1f002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.986+0000 7fe2061b6640 1 -- 172.21.15.83:0/3826581103 shutdown_connections 2024-03-06T07:22:08.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.986+0000 7fe2061b6640 1 --2- 172.21.15.83:0/3826581103 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe20015b380 0x7fe2000708d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.986+0000 7fe2061b6640 1 --2- 172.21.15.83:0/3826581103 >> v2:172.21.15.183:3300/0 conn(0x7fe200158880 0x7fe20006c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.986+0000 7fe2061b6640 1 -- 172.21.15.83:0/3826581103 >> 172.21.15.83:0/3826581103 conn(0x7fe2000b1c80 msgr2=0x7fe2000cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.986+0000 7fe2061b6640 1 -- 172.21.15.83:0/3826581103 shutdown_connections 2024-03-06T07:22:08.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.986+0000 7fe2061b6640 1 -- 172.21.15.83:0/3826581103 wait complete. 2024-03-06T07:22:08.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.987+0000 7f5bed365640 1 -- 172.21.15.83:0/975970051 --> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5bc8005c40 con 0x7f5bc8001650 2024-03-06T07:22:08.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.988+0000 7f5bcf7fe640 1 -- 172.21.15.83:0/975970051 <== osd.7 v2:172.21.15.183:6820/3336566695 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5bc8005c40 con 0x7f5bc8001650 2024-03-06T07:22:08.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.988+0000 7f5bed365640 1 -- 172.21.15.83:0/975970051 >> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] conn(0x7f5bc8001650 msgr2=0x7f5bc8003b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.988+0000 7f5bed365640 1 --2- 172.21.15.83:0/975970051 >> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] conn(0x7f5bc8001650 0x7f5bc8003b10 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.988+0000 7fe2061b6640 1 Processor -- start 2024-03-06T07:22:08.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.988+0000 7f5bed365640 1 -- 172.21.15.83:0/975970051 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5bd406b4d0 msgr2=0x7f5bd406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.988+0000 7f5bed365640 1 --2- 172.21.15.83:0/975970051 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5bd406b4d0 0x7f5bd406d990 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc006fd0 tx=0x7f5bdc008040 comp rx=0 tx=0).stop 2024-03-06T07:22:08.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.988+0000 7f5bed365640 1 -- 172.21.15.83:0/975970051 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5be806ebe0 msgr2=0x7f5be81510b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.988+0000 7f5bed365640 1 --2- 172.21.15.83:0/975970051 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5be806ebe0 0x7f5be81510b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5be000c700 tx=0x7f5be000cbc0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.988+0000 7f5be77fe640 1 -- 172.21.15.83:0/975970051 reap_dead start 2024-03-06T07:22:08.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.988+0000 7f5bed365640 1 -- 172.21.15.83:0/975970051 shutdown_connections 2024-03-06T07:22:08.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.988+0000 7f5bed365640 1 -- 172.21.15.83:0/975970051 >> 172.21.15.83:0/975970051 conn(0x7f5be80b1c50 msgr2=0x7f5be80afbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.988+0000 7fe2061b6640 1 -- start start 2024-03-06T07:22:08.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.988+0000 7f5bed365640 1 -- 172.21.15.83:0/975970051 shutdown_connections 2024-03-06T07:22:08.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.988+0000 7f5bed365640 1 -- 172.21.15.83:0/975970051 wait complete. 2024-03-06T07:22:08.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.989+0000 7fe2061b6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe20006cb90 0x7fe20014f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.989+0000 7fe2061b6640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe200158880 0x7fe200147610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.989+0000 7fe2061b6640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe20015b380 0x7fe20014cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.989+0000 7fe2061b6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2000ce950 con 0x7fe20006cb90 2024-03-06T07:22:08.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.989+0000 7fe2061b6640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2000ce7d0 con 0x7fe200158880 2024-03-06T07:22:08.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.989+0000 7fe2061b6640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe200070e10 con 0x7fe20015b380 2024-03-06T07:22:08.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.989+0000 7fe1ff7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe20006cb90 0x7fe20014f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.989+0000 7fe1ff7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe20006cb90 0x7fe20014f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49450/0 (socket says 172.21.15.83:49450) 2024-03-06T07:22:08.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.989+0000 7fe1ff7fe640 1 -- 172.21.15.83:0/414245410 learned_addr learned my addr 172.21.15.83:0/414245410 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:08.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.989+0000 7fe1feffd640 1 --2- 172.21.15.83:0/414245410 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe200158880 0x7fe200147610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.989+0000 7fe1fffff640 1 --2- 172.21.15.83:0/414245410 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe20015b380 0x7fe20014cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.990+0000 7fe1ff7fe640 1 -- 172.21.15.83:0/414245410 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe20015b380 msgr2=0x7fe20014cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.990+0000 7fe1ff7fe640 1 --2- 172.21.15.83:0/414245410 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe20015b380 0x7fe20014cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.990+0000 7fe1ff7fe640 1 -- 172.21.15.83:0/414245410 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe200158880 msgr2=0x7fe200147610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.990+0000 7fe1ff7fe640 1 --2- 172.21.15.83:0/414245410 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe200158880 0x7fe200147610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.990+0000 7fe1ff7fe640 1 -- 172.21.15.83:0/414245410 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe20014fc30 con 0x7fe20006cb90 2024-03-06T07:22:08.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.990+0000 7fe1fffff640 1 --2- 172.21.15.83:0/414245410 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe20015b380 0x7fe20014cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:08.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.990+0000 7fe1feffd640 1 --2- 172.21.15.83:0/414245410 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe200158880 0x7fe200147610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:08.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.990+0000 7fe1ff7fe640 1 --2- 172.21.15.83:0/414245410 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe20006cb90 0x7fe20014f630 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe1f400e9b0 tx=0x7fe1f400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.990+0000 7fe1e7fff640 1 -- 172.21.15.83:0/414245410 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f4017070 con 0x7fe20006cb90 2024-03-06T07:22:08.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.990+0000 7fe1e7fff640 1 -- 172.21.15.83:0/414245410 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1f400c9a0 con 0x7fe20006cb90 2024-03-06T07:22:08.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.990+0000 7fe2061b6640 1 -- 172.21.15.83:0/414245410 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe20014ff10 con 0x7fe20006cb90 2024-03-06T07:22:08.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.990+0000 7fe1e7fff640 1 -- 172.21.15.83:0/414245410 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f400cd90 con 0x7fe20006cb90 2024-03-06T07:22:08.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.990+0000 7fe2061b6640 1 -- 172.21.15.83:0/414245410 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe20014b580 con 0x7fe20006cb90 2024-03-06T07:22:08.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.991+0000 7fe2061b6640 1 -- 172.21.15.83:0/414245410 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe20004f330 con 0x7fe20006cb90 2024-03-06T07:22:08.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.992+0000 7fe1e7fff640 1 -- 172.21.15.83:0/414245410 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe1f401a070 con 0x7fe20006cb90 2024-03-06T07:22:08.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.992+0000 7fe1e7fff640 1 --2- 172.21.15.83:0/414245410 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe1ec06b500 0x7fe1ec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.992+0000 7fe1e7fff640 1 -- 172.21.15.83:0/414245410 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe1f408a140 con 0x7fe20006cb90 2024-03-06T07:22:08.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.992+0000 7fe1feffd640 1 --2- 172.21.15.83:0/414245410 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe1ec06b500 0x7fe1ec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.992+0000 7fe1e7fff640 1 --2- 172.21.15.83:0/414245410 >> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] conn(0x7fe1ec0736a0 0x7fe1ec075ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.992+0000 7fe1e7fff640 1 -- 172.21.15.83:0/414245410 --> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe1ec076170 con 0x7fe1ec0736a0 2024-03-06T07:22:08.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.992+0000 7fe1e7fff640 1 -- 172.21.15.83:0/414245410 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe1f4091050 con 0x7fe20006cb90 2024-03-06T07:22:08.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.993+0000 7fe1feffd640 1 --2- 172.21.15.83:0/414245410 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe1ec06b500 0x7fe1ec06d9c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0036000 tx=0x7fe1f0040040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.993+0000 7fe1fffff640 1 --2- 172.21.15.83:0/414245410 >> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] conn(0x7fe1ec0736a0 0x7fe1ec075ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.993+0000 7fe1fffff640 1 --2- 172.21.15.83:0/414245410 >> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] conn(0x7fe1ec0736a0 0x7fe1ec075ac0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:07.994+0000 7fe1e7fff640 1 -- 172.21.15.83:0/414245410 <== osd.4 v2:172.21.15.183:6816/3256618830 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe1ec076170 con 0x7fe1ec0736a0 2024-03-06T07:22:08.023 INFO:teuthology.orchestra.run.smithi083.stdout:55834574856 2024-03-06T07:22:08.023 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-06T07:22:08.024 INFO:teuthology.orchestra.run.smithi083.stdout:55834574857 2024-03-06T07:22:08.025 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-06T07:22:08.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.013+0000 7fe2061b6640 1 -- 172.21.15.83:0/414245410 --> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe2000d02e0 con 0x7fe1ec0736a0 2024-03-06T07:22:08.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.013+0000 7fe1e7fff640 1 -- 172.21.15.83:0/414245410 <== osd.4 v2:172.21.15.183:6816/3256618830 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe2000d02e0 con 0x7fe1ec0736a0 2024-03-06T07:22:08.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.013+0000 7fe2061b6640 1 -- 172.21.15.83:0/414245410 >> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] conn(0x7fe1ec0736a0 msgr2=0x7fe1ec075ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.013+0000 7fe2061b6640 1 --2- 172.21.15.83:0/414245410 >> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] conn(0x7fe1ec0736a0 0x7fe1ec075ac0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.013+0000 7fe2061b6640 1 -- 172.21.15.83:0/414245410 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe1ec06b500 msgr2=0x7fe1ec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.013+0000 7fe2061b6640 1 --2- 172.21.15.83:0/414245410 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe1ec06b500 0x7fe1ec06d9c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0036000 tx=0x7fe1f0040040 comp rx=0 tx=0).stop 2024-03-06T07:22:08.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.013+0000 7fe2061b6640 1 -- 172.21.15.83:0/414245410 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe20006cb90 msgr2=0x7fe20014f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.013+0000 7fe2061b6640 1 --2- 172.21.15.83:0/414245410 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe20006cb90 0x7fe20014f630 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe1f400e9b0 tx=0x7fe1f400ee70 comp rx=0 tx=0).stop 2024-03-06T07:22:08.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.013+0000 7fe1fffff640 1 -- 172.21.15.83:0/414245410 reap_dead start 2024-03-06T07:22:08.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.013+0000 7fe2061b6640 1 -- 172.21.15.83:0/414245410 shutdown_connections 2024-03-06T07:22:08.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.013+0000 7fe2061b6640 1 -- 172.21.15.83:0/414245410 >> 172.21.15.83:0/414245410 conn(0x7fe2000b1c80 msgr2=0x7fe2000afeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.014+0000 7fe2061b6640 1 -- 172.21.15.83:0/414245410 shutdown_connections 2024-03-06T07:22:08.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.014+0000 7fe2061b6640 1 -- 172.21.15.83:0/414245410 wait complete. 2024-03-06T07:22:08.029 INFO:teuthology.orchestra.run.smithi083.stdout:55834574857 2024-03-06T07:22:08.029 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-06T07:22:08.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.178+0000 7f92d3fe2640 1 Processor -- start 2024-03-06T07:22:08.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.179+0000 7f92d3fe2640 1 -- start start 2024-03-06T07:22:08.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.180+0000 7f92d3fe2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f92cc1620a0 0x7f92cc163020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.180+0000 7f92d3fe2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92cc06c4c0 0x7f92cc0b3730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.180+0000 7f92d1d7e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f92cc1620a0 0x7f92cc163020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.180+0000 7f92d1d7e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f92cc1620a0 0x7f92cc163020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49462/0 (socket says 172.21.15.83:49462) 2024-03-06T07:22:08.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.181+0000 7f92d3fe2640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92cc163560 con 0x7f92cc06c920 2024-03-06T07:22:08.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.181+0000 7f92d3fe2640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92cc162480 con 0x7f92cc1620a0 2024-03-06T07:22:08.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.181+0000 7f92d3fe2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92cc0ae6b0 con 0x7f92cc06c4c0 2024-03-06T07:22:08.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.181+0000 7f92d157d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92cc06c4c0 0x7f92cc0b3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.181+0000 7f92d157d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92cc06c4c0 0x7f92cc0b3730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:59264/0 (socket says 172.21.15.83:59264) 2024-03-06T07:22:08.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.181+0000 7f92d157d640 1 -- 172.21.15.83:0/1023132653 learned_addr learned my addr 172.21.15.83:0/1023132653 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:08.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.181+0000 7f92d0d7c640 1 -- 172.21.15.83:0/1023132653 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1581306904 0 0) 0x7f92c4001490 con 0x7f92cc06c920 2024-03-06T07:22:08.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.181+0000 7f92d0d7c640 1 -- 172.21.15.83:0/1023132653 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92b8003520 con 0x7f92cc06c920 2024-03-06T07:22:08.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.182+0000 7f92d0d7c640 1 -- 172.21.15.83:0/1023132653 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2769813606 0 0) 0x7f92b8003520 con 0x7f92cc06c920 2024-03-06T07:22:08.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.182+0000 7f92d0d7c640 1 -- 172.21.15.83:0/1023132653 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92cc06c4c0 msgr2=0x7f92cc0b3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:08.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.182+0000 7f92d0d7c640 1 --2- 172.21.15.83:0/1023132653 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92cc06c4c0 0x7f92cc0b3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.182+0000 7f92d0d7c640 1 -- 172.21.15.83:0/1023132653 >> v2:172.21.15.83:3300/0 conn(0x7f92cc1620a0 msgr2=0x7f92cc163020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:08.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.182+0000 7f92d0d7c640 1 --2- 172.21.15.83:0/1023132653 >> v2:172.21.15.83:3300/0 conn(0x7f92cc1620a0 0x7f92cc163020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.182+0000 7f92d0d7c640 1 -- 172.21.15.83:0/1023132653 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92cc0ae990 con 0x7f92cc06c920 2024-03-06T07:22:08.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.182+0000 7f92d0d7c640 1 -- 172.21.15.83:0/1023132653 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f92c4001be0 con 0x7f92cc06c920 2024-03-06T07:22:08.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.182+0000 7f92d0d7c640 1 -- 172.21.15.83:0/1023132653 >> v1:172.21.15.83:6789/0 conn(0x7f92cc06c920 legacy=0x7f92cc0713f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.182+0000 7f92d0d7c640 1 --2- 172.21.15.83:0/1023132653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92b80039b0 0x7f92b8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.182+0000 7f92d0d7c640 1 --2- 172.21.15.83:0/1023132653 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92b8004800 0x7f92b8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.182+0000 7f92d0d7c640 1 --2- 172.21.15.83:0/1023132653 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92cc06c4c0 0x7f92b80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.182+0000 7f92d0d7c640 1 -- 172.21.15.83:0/1023132653 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92cc0ae6b0 con 0x7f92b80039b0 2024-03-06T07:22:08.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.182+0000 7f92d0d7c640 1 -- 172.21.15.83:0/1023132653 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b800ba70 con 0x7f92b8004800 2024-03-06T07:22:08.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.182+0000 7f92d0d7c640 1 -- 172.21.15.83:0/1023132653 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92b800bc70 con 0x7f92cc06c4c0 2024-03-06T07:22:08.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.185+0000 7f92d257f640 1 --2- 172.21.15.83:0/1023132653 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92cc06c4c0 0x7f92b80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.185+0000 7f92d157d640 1 --2- 172.21.15.83:0/1023132653 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92b8004800 0x7f92b8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.186+0000 7f92d257f640 1 -- 172.21.15.83:0/1023132653 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92b8004800 msgr2=0x7f92b8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:08.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.186+0000 7f92d257f640 1 --2- 172.21.15.83:0/1023132653 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92b8004800 0x7f92b8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.186+0000 7f92d257f640 1 -- 172.21.15.83:0/1023132653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92b80039b0 msgr2=0x7f92b8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:08.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.186+0000 7f92d1d7e640 1 --2- 172.21.15.83:0/1023132653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92b80039b0 0x7f92b8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.186+0000 7f92d257f640 1 --2- 172.21.15.83:0/1023132653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92b80039b0 0x7f92b8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.186+0000 7f92d257f640 1 -- 172.21.15.83:0/1023132653 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b8003520 con 0x7f92cc06c4c0 2024-03-06T07:22:08.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.187+0000 7f92d1d7e640 1 --2- 172.21.15.83:0/1023132653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92b80039b0 0x7f92b8003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:08.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.187+0000 7f2c40ce9640 1 Processor -- start 2024-03-06T07:22:08.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.187+0000 7f2c40ce9640 1 -- start start 2024-03-06T07:22:08.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.187+0000 7f92d257f640 1 --2- 172.21.15.83:0/1023132653 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92cc06c4c0 0x7f92b80051c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f92cc0cbfd0 tx=0x7f92b8005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=267b8d2bd8f4bbd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.188+0000 7f92d0d7c640 1 -- 172.21.15.83:0/1023132653 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c400c1e0 con 0x7f92cc06c4c0 2024-03-06T07:22:08.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.188+0000 7f92d0d7c640 1 -- 172.21.15.83:0/1023132653 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92c4030070 con 0x7f92cc06c4c0 2024-03-06T07:22:08.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.188+0000 7f92d0d7c640 1 -- 172.21.15.83:0/1023132653 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c4034650 con 0x7f92cc06c4c0 2024-03-06T07:22:08.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.189+0000 7f92d3fe2640 1 -- 172.21.15.83:0/1023132653 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92cc06c4c0 msgr2=0x7f92b80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.189+0000 7f92d3fe2640 1 --2- 172.21.15.83:0/1023132653 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92cc06c4c0 0x7f92b80051c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f92cc0cbfd0 tx=0x7f92b8005830 comp rx=0 tx=0).stop 2024-03-06T07:22:08.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.189+0000 7f92d257f640 1 -- 172.21.15.83:0/1023132653 reap_dead start 2024-03-06T07:22:08.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.189+0000 7f2c40ce9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2c3c0afd10 0x7f2c3c15b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.189+0000 7f2c40ce9640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c3c072330 con 0x7f2c3c0af930 2024-03-06T07:22:08.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.189+0000 7f2c40ce9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c3c072610 con 0x7f2c3c071950 2024-03-06T07:22:08.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.189+0000 7f2c40ce9640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c3c06e360 con 0x7f2c3c0afd10 2024-03-06T07:22:08.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.189+0000 7f92d3fe2640 1 -- 172.21.15.83:0/1023132653 shutdown_connections 2024-03-06T07:22:08.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.189+0000 7f92d3fe2640 1 -- 172.21.15.83:0/1023132653 >> 172.21.15.83:0/1023132653 conn(0x7f92cc0b16e0 msgr2=0x7f92cc0b1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.188+0000 7f4948cc3640 1 Processor -- start 2024-03-06T07:22:08.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.188+0000 7f4948cc3640 1 -- start start 2024-03-06T07:22:08.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.189+0000 7f4948cc3640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f493c048040 0x7f493c04fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.189+0000 7f4948cc3640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f493c038720 0x7f493c006ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.189+0000 7f4948cc3640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f493c048420 con 0x7f493c037bb0 2024-03-06T07:22:08.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.189+0000 7f4948cc3640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f493c0547a0 con 0x7f493c048040 2024-03-06T07:22:08.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.189+0000 7f4948cc3640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f493c053700 con 0x7f493c038720 2024-03-06T07:22:08.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.189+0000 7f4943fff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f493c038720 0x7f493c006ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.189+0000 7f4943fff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f493c038720 0x7f493c006ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:59282/0 (socket says 172.21.15.83:59282) 2024-03-06T07:22:08.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.189+0000 7f4943fff640 1 -- 172.21.15.83:0/4079698166 learned_addr learned my addr 172.21.15.83:0/4079698166 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:08.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.189+0000 7f2c3affd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2c3c0afd10 0x7f2c3c15b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.190+0000 7f2c3affd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2c3c0afd10 0x7f2c3c15b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49482/0 (socket says 172.21.15.83:49482) 2024-03-06T07:22:08.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.190+0000 7f2c3affd640 1 -- 172.21.15.83:0/2756483698 learned_addr learned my addr 172.21.15.83:0/2756483698 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:08.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.190+0000 7f92d3fe2640 1 -- 172.21.15.83:0/1023132653 shutdown_connections 2024-03-06T07:22:08.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.190+0000 7f4943fff640 1 -- 172.21.15.83:0/4079698166 >> v2:172.21.15.183:3300/0 conn(0x7f493c048040 msgr2=0x7f493c04fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:08.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.190+0000 7f49427fc640 1 -- 172.21.15.83:0/4079698166 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2966517000 0 0) 0x7f493c048420 con 0x7f493c037bb0 2024-03-06T07:22:08.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.190+0000 7f49437fe640 1 --2- 172.21.15.83:0/4079698166 >> v2:172.21.15.183:3300/0 conn(0x7f493c048040 0x7f493c04fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.190+0000 7f4943fff640 1 --2- 172.21.15.83:0/4079698166 >> v2:172.21.15.183:3300/0 conn(0x7f493c048040 0x7f493c04fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.190+0000 7f4943fff640 1 -- 172.21.15.83:0/4079698166 >> v1:172.21.15.183:6789/0 conn(0x7f493c037bb0 legacy=0x7f493c038010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.190+0000 7f4943fff640 1 -- 172.21.15.83:0/4079698166 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f493c0539e0 con 0x7f493c038720 2024-03-06T07:22:08.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.190+0000 7f49437fe640 1 --2- 172.21.15.83:0/4079698166 >> v2:172.21.15.183:3300/0 conn(0x7f493c048040 0x7f493c04fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:08.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.190+0000 7f92d3fe2640 1 -- 172.21.15.83:0/1023132653 wait complete. 2024-03-06T07:22:08.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.191+0000 7f9c23fff640 1 Processor -- start 2024-03-06T07:22:08.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.191+0000 7f9c23fff640 1 -- start start 2024-03-06T07:22:08.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.190+0000 7f2c3affd640 1 -- 172.21.15.83:0/2756483698 >> v1:172.21.15.183:6789/0 conn(0x7f2c3c071950 legacy=0x7f2c3c0b3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:08.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.190+0000 7f2c3affd640 1 -- 172.21.15.83:0/2756483698 >> v1:172.21.15.83:6789/0 conn(0x7f2c3c0af930 legacy=0x7f2c3c06eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:08.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.190+0000 7f2c3affd640 1 -- 172.21.15.83:0/2756483698 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c3c06e640 con 0x7f2c3c0afd10 2024-03-06T07:22:08.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.192+0000 7f2c3affd640 1 --2- 172.21.15.83:0/2756483698 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c3c0afd10 0x7f2c3c15b710 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2c30009870 tx=0x7f2c3002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbacfc44051b297a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.192+0000 7f9c23fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9c1c0381e0 0x7f9c1c0500e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.192+0000 7f9c23fff640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c1c039f50 con 0x7f9c1c037d50 2024-03-06T07:22:08.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.192+0000 7f9c23fff640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c1c03a230 con 0x7f9c1c0e0ff0 2024-03-06T07:22:08.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.192+0000 7f9c23fff640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c1c03a510 con 0x7f9c1c0381e0 2024-03-06T07:22:08.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.192+0000 7f9c227fc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f9c1c037d50 0x7f9c1c04f9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:42904/0 (socket says 172.21.15.83:42904) 2024-03-06T07:22:08.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.192+0000 7f9c227fc640 1 -- 172.21.15.83:0/2776578963 learned_addr learned my addr 172.21.15.83:0/2776578963 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:08.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.192+0000 7f2c3a7fc640 1 -- 172.21.15.83:0/2756483698 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3002e070 con 0x7f2c3c0afd10 2024-03-06T07:22:08.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.193+0000 7f2c3a7fc640 1 -- 172.21.15.83:0/2756483698 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c30037070 con 0x7f2c3c0afd10 2024-03-06T07:22:08.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.193+0000 7f2c3a7fc640 1 -- 172.21.15.83:0/2756483698 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c30031a00 con 0x7f2c3c0afd10 2024-03-06T07:22:08.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.193+0000 7f92d3fe2640 1 Processor -- start 2024-03-06T07:22:08.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.193+0000 7f9c237fe640 1 --2- 172.21.15.83:0/2776578963 >> v2:172.21.15.83:3300/0 conn(0x7f9c1c0381e0 0x7f9c1c0500e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.193+0000 7f92d3fe2640 1 -- start start 2024-03-06T07:22:08.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.193+0000 7f9c237fe640 1 -- 172.21.15.83:0/2776578963 >> v1:172.21.15.183:6789/0 conn(0x7f9c1c0e0ff0 legacy=0x7f9c1c033e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.193+0000 7f9c237fe640 1 -- 172.21.15.83:0/2776578963 >> v1:172.21.15.83:6789/0 conn(0x7f9c1c037d50 legacy=0x7f9c1c04f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.193+0000 7f9c237fe640 1 -- 172.21.15.83:0/2776578963 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c1c03a7f0 con 0x7f9c1c0381e0 2024-03-06T07:22:08.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.194+0000 7f2c40ce9640 1 -- 172.21.15.83:0/2756483698 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c3c0afd10 msgr2=0x7f2c3c15b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.194+0000 7f2c40ce9640 1 --2- 172.21.15.83:0/2756483698 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c3c0afd10 0x7f2c3c15b710 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2c30009870 tx=0x7f2c3002b880 comp rx=0 tx=0).stop 2024-03-06T07:22:08.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.194+0000 7f92d3fe2640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92cc06c4c0 0x7f92cc1f85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.195+0000 7f92d3fe2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92cc1fb0e0 0x7f92cc0afe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.195+0000 7f2c40ce9640 1 -- 172.21.15.83:0/2756483698 shutdown_connections 2024-03-06T07:22:08.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.195+0000 7f2c40ce9640 1 --2- 172.21.15.83:0/2756483698 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c3c0afd10 0x7f2c3c15b710 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.195+0000 7f2c40ce9640 1 -- 172.21.15.83:0/2756483698 >> 172.21.15.83:0/2756483698 conn(0x7f2c3c06c0c0 msgr2=0x7f2c3c06c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.195+0000 7f9c237fe640 1 --2- 172.21.15.83:0/2776578963 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c1c0381e0 0x7f9c1c0500e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9c2402d580 tx=0x7f9c240a6620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9ef5ae47ab6a396 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.195+0000 7f92d1d7e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92cc06c4c0 0x7f92cc1f85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.195+0000 7f92d1d7e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92cc06c4c0 0x7f92cc1f85e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:60052/0 (socket says 172.21.15.83:60052) 2024-03-06T07:22:08.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.195+0000 7f9c21ffb640 1 -- 172.21.15.83:0/2776578963 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c240b2070 con 0x7f9c1c0381e0 2024-03-06T07:22:08.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.195+0000 7f92d1d7e640 1 -- 172.21.15.83:0/729821505 learned_addr learned my addr 172.21.15.83:0/729821505 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:08.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.195+0000 7f92d3fe2640 1 --2- 172.21.15.83:0/729821505 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92cc14da80 0x7f92cc14dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.195+0000 7f92d157d640 1 --2- 172.21.15.83:0/729821505 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92cc1fb0e0 0x7f92cc0afe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.195+0000 7f92d3fe2640 1 -- 172.21.15.83:0/729821505 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92cc070ed0 con 0x7f92cc14da80 2024-03-06T07:22:08.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.195+0000 7f92d257f640 1 --2- 172.21.15.83:0/729821505 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92cc14da80 0x7f92cc14dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.195+0000 7f9c21ffb640 1 -- 172.21.15.83:0/2776578963 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c240ab7d0 con 0x7f9c1c0381e0 2024-03-06T07:22:08.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.195+0000 7f92d3fe2640 1 -- 172.21.15.83:0/729821505 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92cc0aef50 con 0x7f92cc06c4c0 2024-03-06T07:22:08.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.195+0000 7f92d3fe2640 1 -- 172.21.15.83:0/729821505 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92cc0b4420 con 0x7f92cc1fb0e0 2024-03-06T07:22:08.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.196+0000 7f9c21ffb640 1 -- 172.21.15.83:0/2776578963 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c240abc40 con 0x7f9c1c0381e0 2024-03-06T07:22:08.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.196+0000 7f2c40ce9640 1 -- 172.21.15.83:0/2756483698 shutdown_connections 2024-03-06T07:22:08.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.196+0000 7f2c40ce9640 1 -- 172.21.15.83:0/2756483698 wait complete. 2024-03-06T07:22:08.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.196+0000 7f4943fff640 1 --2- 172.21.15.83:0/4079698166 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f493c038720 0x7f493c006ea0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f494402d580 tx=0x7f49440a6960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5ac8d1a501820b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.196+0000 7f9c23fff640 1 -- 172.21.15.83:0/2776578963 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c1c0381e0 msgr2=0x7f9c1c0500e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.196+0000 7f9c23fff640 1 --2- 172.21.15.83:0/2776578963 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c1c0381e0 0x7f9c1c0500e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9c2402d580 tx=0x7f9c240a6620 comp rx=0 tx=0).stop 2024-03-06T07:22:08.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.196+0000 7f9c23fff640 1 -- 172.21.15.83:0/2776578963 shutdown_connections 2024-03-06T07:22:08.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.196+0000 7f9c23fff640 1 --2- 172.21.15.83:0/2776578963 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c1c0381e0 0x7f9c1c0500e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.196+0000 7f9c23fff640 1 -- 172.21.15.83:0/2776578963 >> 172.21.15.83:0/2776578963 conn(0x7f9c1c00a3f0 msgr2=0x7f9c1c00a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.196+0000 7f49427fc640 1 -- 172.21.15.83:0/4079698166 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49440af020 con 0x7f493c038720 2024-03-06T07:22:08.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.196+0000 7f9c23fff640 1 -- 172.21.15.83:0/2776578963 shutdown_connections 2024-03-06T07:22:08.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.196+0000 7f9c23fff640 1 -- 172.21.15.83:0/2776578963 wait complete. 2024-03-06T07:22:08.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.196+0000 7f49427fc640 1 -- 172.21.15.83:0/4079698166 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49440ab750 con 0x7f493c038720 2024-03-06T07:22:08.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.197+0000 7f92d257f640 1 -- 172.21.15.83:0/729821505 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92cc1fb0e0 msgr2=0x7f92cc0afe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.197+0000 7f92d257f640 1 --2- 172.21.15.83:0/729821505 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92cc1fb0e0 0x7f92cc0afe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.197+0000 7f92d257f640 1 -- 172.21.15.83:0/729821505 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92cc06c4c0 msgr2=0x7f92cc1f85e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.197+0000 7f92d257f640 1 --2- 172.21.15.83:0/729821505 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92cc06c4c0 0x7f92cc1f85e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.197+0000 7f49427fc640 1 -- 172.21.15.83:0/4079698166 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49440abc00 con 0x7f493c038720 2024-03-06T07:22:08.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.197+0000 7f92d257f640 1 -- 172.21.15.83:0/729821505 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92cc1fb560 con 0x7f92cc14da80 2024-03-06T07:22:08.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.197+0000 7f92d157d640 1 --2- 172.21.15.83:0/729821505 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92cc1fb0e0 0x7f92cc0afe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:08.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.198+0000 7f92d257f640 1 --2- 172.21.15.83:0/729821505 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92cc14da80 0x7f92cc14dee0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f92c4032000 tx=0x7f92cc0aeb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.197+0000 7f7d595b5640 1 Processor -- start 2024-03-06T07:22:08.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.197+0000 7f7d595b5640 1 -- start start 2024-03-06T07:22:08.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.198+0000 7f2c40ce9640 1 Processor -- start 2024-03-06T07:22:08.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.198+0000 7f9c23fff640 1 Processor -- start 2024-03-06T07:22:08.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.198+0000 7f9c23fff640 1 -- start start 2024-03-06T07:22:08.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.198+0000 7f92c27fc640 1 -- 172.21.15.83:0/729821505 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c400c1e0 con 0x7f92cc14da80 2024-03-06T07:22:08.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.198+0000 7f7d595b5640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7d4c0e78c0 0x7f7d4c03aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.198+0000 7f7d595b5640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d4c000e20 0x7f7d4c037500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.198+0000 7f7d595b5640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d4c055f20 con 0x7f7d4c009500 2024-03-06T07:22:08.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.198+0000 7f7d595b5640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d4c057160 con 0x7f7d4c0e78c0 2024-03-06T07:22:08.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.198+0000 7f7d595b5640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4c03afc0 con 0x7f7d4c000e20 2024-03-06T07:22:08.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.199+0000 7f7d527fc640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d4c000e20 0x7f7d4c037500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.198+0000 7f92c27fc640 1 -- 172.21.15.83:0/729821505 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92c4044070 con 0x7f92cc14da80 2024-03-06T07:22:08.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.199+0000 7f92c27fc640 1 -- 172.21.15.83:0/729821505 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c400c1e0 con 0x7f92cc14da80 2024-03-06T07:22:08.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.199+0000 7f4948cc3640 1 -- 172.21.15.83:0/4079698166 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f493c038720 msgr2=0x7f493c006ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.199+0000 7f4948cc3640 1 --2- 172.21.15.83:0/4079698166 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f493c038720 0x7f493c006ea0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f494402d580 tx=0x7f49440a6960 comp rx=0 tx=0).stop 2024-03-06T07:22:08.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.199+0000 7f2c40ce9640 1 -- start start 2024-03-06T07:22:08.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.199+0000 7f7d527fc640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d4c000e20 0x7f7d4c037500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:59312/0 (socket says 172.21.15.83:59312) 2024-03-06T07:22:08.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.199+0000 7f7d527fc640 1 -- 172.21.15.83:0/537311539 learned_addr learned my addr 172.21.15.83:0/537311539 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:08.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.199+0000 7f7d527fc640 1 -- 172.21.15.83:0/537311539 >> v2:172.21.15.83:3300/0 conn(0x7f7d4c0e78c0 msgr2=0x7f7d4c03aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:08.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.199+0000 7f7d527fc640 1 --2- 172.21.15.83:0/537311539 >> v2:172.21.15.83:3300/0 conn(0x7f7d4c0e78c0 0x7f7d4c03aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.199+0000 7f7d527fc640 1 -- 172.21.15.83:0/537311539 >> v1:172.21.15.83:6789/0 conn(0x7f7d4c009500 legacy=0x7f7d4c0575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:08.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.199+0000 7f7d527fc640 1 -- 172.21.15.83:0/537311539 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d4c001300 con 0x7f7d4c000e20 2024-03-06T07:22:08.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.200+0000 7f9c23fff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c1c037d50 0x7f9c1c0dce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.200+0000 7f2c40ce9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c3c071950 0x7f2c3c0aed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.200+0000 7f9c23fff640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c1c0381e0 0x7f9c1c0dd540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.200+0000 7f4948cc3640 1 -- 172.21.15.83:0/4079698166 shutdown_connections 2024-03-06T07:22:08.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.200+0000 7f4948cc3640 1 --2- 172.21.15.83:0/4079698166 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f493c038720 0x7f493c006ea0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.200+0000 7f4948cc3640 1 --2- 172.21.15.83:0/4079698166 >> v2:172.21.15.183:3300/0 conn(0x7f493c048040 0x7f493c04fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.200+0000 7f4948cc3640 1 -- 172.21.15.83:0/4079698166 >> 172.21.15.83:0/4079698166 conn(0x7f493c009fb0 msgr2=0x7f493c00a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.200+0000 7f9c23fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c1c0e0ff0 0x7f9c1c0e35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.200+0000 7f9c22ffd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c1c037d50 0x7f9c1c0dce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.200+0000 7f9c22ffd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c1c037d50 0x7f9c1c0dce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:59324/0 (socket says 172.21.15.83:59324) 2024-03-06T07:22:08.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.201+0000 7f9c227fc640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c1c0381e0 0x7f9c1c0dd540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.200+0000 7f2c40ce9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c3c0af930 0x7f2c3c06b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.201+0000 7f2c3b7fe640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c3c071950 0x7f2c3c0aed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.201+0000 7f2c3b7fe640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c3c071950 0x7f2c3c0aed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:60068/0 (socket says 172.21.15.83:60068) 2024-03-06T07:22:08.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.201+0000 7f2c3b7fe640 1 -- 172.21.15.83:0/3571182664 learned_addr learned my addr 172.21.15.83:0/3571182664 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:08.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.201+0000 7f2c40ce9640 1 --2- 172.21.15.83:0/3571182664 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2c3c0afd10 0x7f2c3c145c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.201+0000 7f2c40ce9640 1 -- 172.21.15.83:0/3571182664 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c3c15e780 con 0x7f2c3c0af930 2024-03-06T07:22:08.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.201+0000 7f2c40ce9640 1 -- 172.21.15.83:0/3571182664 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c3c15e600 con 0x7f2c3c071950 2024-03-06T07:22:08.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.201+0000 7f2c40ce9640 1 -- 172.21.15.83:0/3571182664 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c3c071d80 con 0x7f2c3c0afd10 2024-03-06T07:22:08.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.201+0000 7f2c3bfff640 1 --2- 172.21.15.83:0/3571182664 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2c3c0afd10 0x7f2c3c145c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.201+0000 7f2c3affd640 1 --2- 172.21.15.83:0/3571182664 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c3c0af930 0x7f2c3c06b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.199+0000 7f92d3fe2640 1 -- 172.21.15.83:0/729821505 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92cc1fb830 con 0x7f92cc14da80 2024-03-06T07:22:08.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.199+0000 7f92d3fe2640 1 -- 172.21.15.83:0/729821505 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92cc1fbdf0 con 0x7f92cc14da80 2024-03-06T07:22:08.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.201+0000 7f2c3b7fe640 1 -- 172.21.15.83:0/3571182664 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2c3c0afd10 msgr2=0x7f2c3c145c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.201+0000 7f2c3b7fe640 1 --2- 172.21.15.83:0/3571182664 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2c3c0afd10 0x7f2c3c145c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.201+0000 7f2c3b7fe640 1 -- 172.21.15.83:0/3571182664 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c3c0af930 msgr2=0x7f2c3c06b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.201+0000 7f2c3b7fe640 1 --2- 172.21.15.83:0/3571182664 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c3c0af930 0x7f2c3c06b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.202+0000 7f2c3b7fe640 1 -- 172.21.15.83:0/3571182664 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c3c0b2860 con 0x7f2c3c071950 2024-03-06T07:22:08.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.200+0000 7f4948cc3640 1 -- 172.21.15.83:0/4079698166 shutdown_connections 2024-03-06T07:22:08.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.200+0000 7f4948cc3640 1 -- 172.21.15.83:0/4079698166 wait complete. 2024-03-06T07:22:08.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.201+0000 7f9c237fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c1c0e0ff0 0x7f9c1c0e35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.202+0000 7f9c227fc640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c1c0381e0 0x7f9c1c0dd540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:60064/0 (socket says 172.21.15.83:60064) 2024-03-06T07:22:08.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.202+0000 7f9c237fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c1c0e0ff0 0x7f9c1c0e35f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49516/0 (socket says 172.21.15.83:49516) 2024-03-06T07:22:08.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.202+0000 7f9c23fff640 1 -- 172.21.15.83:0/2142934492 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1c034570 con 0x7f9c1c0e0ff0 2024-03-06T07:22:08.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.202+0000 7f9c22ffd640 1 -- 172.21.15.83:0/2142934492 learned_addr learned my addr 172.21.15.83:0/2142934492 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:08.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.202+0000 7f2c3b7fe640 1 --2- 172.21.15.83:0/3571182664 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c3c071950 0x7f2c3c0aed10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c00c410 tx=0x7f2c2c00c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.202+0000 7f9c23fff640 1 -- 172.21.15.83:0/2142934492 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1c038630 con 0x7f9c1c0381e0 2024-03-06T07:22:08.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.202+0000 7f7d527fc640 1 --2- 172.21.15.83:0/537311539 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d4c000e20 0x7f7d4c037500 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7d48009870 tx=0x7f7d4802b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b20b2afa2e3b1e15 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.202+0000 7f9c23fff640 1 -- 172.21.15.83:0/2142934492 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1c0e2280 con 0x7f9c1c037d50 2024-03-06T07:22:08.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.203+0000 7f7d51ffb640 1 -- 172.21.15.83:0/537311539 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d48035020 con 0x7f7d4c000e20 2024-03-06T07:22:08.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.202+0000 7f2c23fff640 1 -- 172.21.15.83:0/3571182664 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c017070 con 0x7f2c3c071950 2024-03-06T07:22:08.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.202+0000 7f2c23fff640 1 -- 172.21.15.83:0/3571182664 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c2c0035f0 con 0x7f2c3c071950 2024-03-06T07:22:08.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.202+0000 7f2c23fff640 1 -- 172.21.15.83:0/3571182664 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c003a20 con 0x7f2c3c071950 2024-03-06T07:22:08.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.203+0000 7f7d51ffb640 1 -- 172.21.15.83:0/537311539 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d48031870 con 0x7f7d4c000e20 2024-03-06T07:22:08.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.203+0000 7f7d51ffb640 1 -- 172.21.15.83:0/537311539 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d48035020 con 0x7f7d4c000e20 2024-03-06T07:22:08.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.203+0000 7f9c237fe640 1 -- 172.21.15.83:0/2142934492 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c1c037d50 msgr2=0x7f9c1c0dce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.203+0000 7f9c237fe640 1 --2- 172.21.15.83:0/2142934492 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c1c037d50 0x7f9c1c0dce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.203+0000 7f9c237fe640 1 -- 172.21.15.83:0/2142934492 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c1c0381e0 msgr2=0x7f9c1c0dd540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.203+0000 7f9c237fe640 1 --2- 172.21.15.83:0/2142934492 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c1c0381e0 0x7f9c1c0dd540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.203+0000 7f9c237fe640 1 -- 172.21.15.83:0/2142934492 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c1c0de000 con 0x7f9c1c0e0ff0 2024-03-06T07:22:08.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.204+0000 7f9c237fe640 1 --2- 172.21.15.83:0/2142934492 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c1c0e0ff0 0x7f9c1c0e35f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9c240645f0 tx=0x7f9c24069a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.204+0000 7f9c227fc640 1 --2- 172.21.15.83:0/2142934492 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c1c0381e0 0x7f9c1c0dd540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:08.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.204+0000 7f7d595b5640 1 -- 172.21.15.83:0/537311539 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d4c000e20 msgr2=0x7f7d4c037500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.204+0000 7f9c0b7fe640 1 -- 172.21.15.83:0/2142934492 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c240b2040 con 0x7f9c1c0e0ff0 2024-03-06T07:22:08.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.204+0000 7f7d595b5640 1 --2- 172.21.15.83:0/537311539 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d4c000e20 0x7f7d4c037500 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7d48009870 tx=0x7f7d4802b910 comp rx=0 tx=0).stop 2024-03-06T07:22:08.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.204+0000 7f2c40ce9640 1 -- 172.21.15.83:0/3571182664 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c3c0b2b40 con 0x7f2c3c071950 2024-03-06T07:22:08.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.204+0000 7f2c40ce9640 1 -- 172.21.15.83:0/3571182664 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c3c0b2550 con 0x7f2c3c071950 2024-03-06T07:22:08.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.202+0000 7f4948cc3640 1 Processor -- start 2024-03-06T07:22:08.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.202+0000 7f4948cc3640 1 -- start start 2024-03-06T07:22:08.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.203+0000 7f4948cc3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f493c037bb0 0x7f493c04ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.203+0000 7f4948cc3640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f493c038720 0x7f493c0d9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.203+0000 7f4948cc3640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f493c048040 0x7f493c0da0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.203+0000 7f4948cc3640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f493c054280 con 0x7f493c037bb0 2024-03-06T07:22:08.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.203+0000 7f4948cc3640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f493c053fa0 con 0x7f493c038720 2024-03-06T07:22:08.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.203+0000 7f4948cc3640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f493c04ffd0 con 0x7f493c048040 2024-03-06T07:22:08.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.205+0000 7f7d595b5640 1 -- 172.21.15.83:0/537311539 shutdown_connections 2024-03-06T07:22:08.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.205+0000 7f7d595b5640 1 --2- 172.21.15.83:0/537311539 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d4c000e20 0x7f7d4c037500 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.205+0000 7f7d595b5640 1 --2- 172.21.15.83:0/537311539 >> v2:172.21.15.83:3300/0 conn(0x7f7d4c0e78c0 0x7f7d4c03aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.205+0000 7f7d595b5640 1 -- 172.21.15.83:0/537311539 >> 172.21.15.83:0/537311539 conn(0x7f7d4c055070 msgr2=0x7f7d4c055480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.205+0000 7f92c27fc640 1 -- 172.21.15.83:0/729821505 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f92c4032030 con 0x7f92cc14da80 2024-03-06T07:22:08.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.205+0000 7f4942ffd640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f493c038720 0x7f493c0d9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.205+0000 7f4942ffd640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f493c038720 0x7f493c0d9bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:60084/0 (socket says 172.21.15.83:60084) 2024-03-06T07:22:08.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.205+0000 7f4942ffd640 1 -- 172.21.15.83:0/3581370308 learned_addr learned my addr 172.21.15.83:0/3581370308 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:08.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.205+0000 7f4943fff640 1 --2- 172.21.15.83:0/3581370308 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f493c048040 0x7f493c0da0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.204+0000 7f9c23fff640 1 -- 172.21.15.83:0/2142934492 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c1c0de2e0 con 0x7f9c1c0e0ff0 2024-03-06T07:22:08.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.204+0000 7f9c23fff640 1 -- 172.21.15.83:0/2142934492 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c1c177530 con 0x7f9c1c0e0ff0 2024-03-06T07:22:08.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.205+0000 7f92c27fc640 1 --2- 172.21.15.83:0/729821505 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f92b006b900 0x7f92b006ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.205+0000 7f49437fe640 1 --2- 172.21.15.83:0/3581370308 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f493c037bb0 0x7f493c04ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.205+0000 7f4943fff640 1 -- 172.21.15.83:0/3581370308 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f493c038720 msgr2=0x7f493c0d9bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.205+0000 7f4943fff640 1 --2- 172.21.15.83:0/3581370308 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f493c038720 0x7f493c0d9bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.206+0000 7f2c40ce9640 1 -- 172.21.15.83:0/3571182664 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c3c068f80 con 0x7f2c3c071950 2024-03-06T07:22:08.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.206+0000 7f7d595b5640 1 -- 172.21.15.83:0/537311539 shutdown_connections 2024-03-06T07:22:08.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.206+0000 7f7d595b5640 1 -- 172.21.15.83:0/537311539 wait complete. 2024-03-06T07:22:08.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.206+0000 7f9c0b7fe640 1 -- 172.21.15.83:0/2142934492 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c240adc30 con 0x7f9c1c0e0ff0 2024-03-06T07:22:08.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.206+0000 7f9c0b7fe640 1 -- 172.21.15.83:0/2142934492 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c240b8670 con 0x7f9c1c0e0ff0 2024-03-06T07:22:08.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.206+0000 7f92c27fc640 1 -- 172.21.15.83:0/729821505 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f92c4003ba0 con 0x7f92cc14da80 2024-03-06T07:22:08.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.206+0000 7f92d1d7e640 1 --2- 172.21.15.83:0/729821505 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f92b006b900 0x7f92b006ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.205+0000 7f4943fff640 1 -- 172.21.15.83:0/3581370308 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f493c037bb0 msgr2=0x7f493c04ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.205+0000 7f4943fff640 1 --2- 172.21.15.83:0/3581370308 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f493c037bb0 0x7f493c04ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.206+0000 7f4943fff640 1 -- 172.21.15.83:0/3581370308 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f493c02efd0 con 0x7f493c048040 2024-03-06T07:22:08.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.206+0000 7f4943fff640 1 --2- 172.21.15.83:0/3581370308 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f493c048040 0x7f493c0da0f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4944075030 tx=0x7f49440693f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.206+0000 7f492ffff640 1 -- 172.21.15.83:0/3581370308 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49440af050 con 0x7f493c048040 2024-03-06T07:22:08.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.207+0000 7f92d1d7e640 1 --2- 172.21.15.83:0/729821505 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f92b006b900 0x7f92b006ddc0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f92bc006e80 tx=0x7f92bc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.208+0000 7f2c23fff640 1 -- 172.21.15.83:0/3571182664 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2c2c01a070 con 0x7f2c3c071950 2024-03-06T07:22:08.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.207+0000 7f4948cc3640 1 -- 172.21.15.83:0/3581370308 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f493c030650 con 0x7f493c048040 2024-03-06T07:22:08.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.207+0000 7f4948cc3640 1 -- 172.21.15.83:0/3581370308 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f493c006920 con 0x7f493c048040 2024-03-06T07:22:08.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.208+0000 7f492ffff640 1 -- 172.21.15.83:0/3581370308 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49440bf3f0 con 0x7f493c048040 2024-03-06T07:22:08.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.208+0000 7f2c23fff640 1 --2- 172.21.15.83:0/3571182664 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2c2806b5a0 0x7f2c2806da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.208+0000 7f492ffff640 1 -- 172.21.15.83:0/3581370308 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49440bf820 con 0x7f493c048040 2024-03-06T07:22:08.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.208+0000 7f7d595b5640 1 Processor -- start 2024-03-06T07:22:08.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.208+0000 7f7d595b5640 1 -- start start 2024-03-06T07:22:08.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.209+0000 7f2c3affd640 1 --2- 172.21.15.83:0/3571182664 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2c2806b5a0 0x7f2c2806da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.209+0000 7f4948cc3640 1 -- 172.21.15.83:0/3581370308 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4924005290 con 0x7f493c048040 2024-03-06T07:22:08.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.209+0000 7f9c0b7fe640 1 -- 172.21.15.83:0/2142934492 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9c240b8810 con 0x7f9c1c0e0ff0 2024-03-06T07:22:08.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.209+0000 7f7d595b5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d4c000e20 0x7f7d4c17d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.209+0000 7f7d595b5640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d4c009500 0x7f7d4c0390f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.209+0000 7f7d595b5640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d4c0e78c0 0x7f7d4c039630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.209+0000 7f7d595b5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4c001ba0 con 0x7f7d4c000e20 2024-03-06T07:22:08.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.209+0000 7f7d595b5640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4c0018c0 con 0x7f7d4c0e78c0 2024-03-06T07:22:08.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.209+0000 7f7d595b5640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4c037a40 con 0x7f7d4c009500 2024-03-06T07:22:08.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.209+0000 7f7d52ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d4c000e20 0x7f7d4c17d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.209+0000 7f7d52ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d4c000e20 0x7f7d4c17d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49564/0 (socket says 172.21.15.83:49564) 2024-03-06T07:22:08.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.209+0000 7f7d52ffd640 1 -- 172.21.15.83:0/73000528 learned_addr learned my addr 172.21.15.83:0/73000528 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:08.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.209+0000 7f7d53fff640 1 --2- 172.21.15.83:0/73000528 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d4c0e78c0 0x7f7d4c039630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.209+0000 7f7d527fc640 1 --2- 172.21.15.83:0/73000528 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d4c009500 0x7f7d4c0390f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.209+0000 7f2c3affd640 1 --2- 172.21.15.83:0/3571182664 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2c2806b5a0 0x7f2c2806da60 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2c30036000 tx=0x7f2c3003d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.210+0000 7f9c23fff640 1 -- 172.21.15.83:0/2142934492 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c04005290 con 0x7f9c1c0e0ff0 2024-03-06T07:22:08.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.210+0000 7f2c23fff640 1 -- 172.21.15.83:0/3571182664 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2c2c089630 con 0x7f2c3c071950 2024-03-06T07:22:08.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.210+0000 7f7d53fff640 1 -- 172.21.15.83:0/73000528 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d4c009500 msgr2=0x7f7d4c0390f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.210+0000 7f7d53fff640 1 --2- 172.21.15.83:0/73000528 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d4c009500 0x7f7d4c0390f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.210+0000 7f7d53fff640 1 -- 172.21.15.83:0/73000528 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d4c000e20 msgr2=0x7f7d4c17d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.210+0000 7f7d53fff640 1 --2- 172.21.15.83:0/73000528 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d4c000e20 0x7f7d4c17d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.210+0000 7f7d53fff640 1 -- 172.21.15.83:0/73000528 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d4c0e8ab0 con 0x7f7d4c0e78c0 2024-03-06T07:22:08.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.210+0000 7f7d53fff640 1 --2- 172.21.15.83:0/73000528 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d4c0e78c0 0x7f7d4c039630 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7d4400c3e0 tx=0x7f7d4400c8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.210+0000 7f9c0b7fe640 1 --2- 172.21.15.83:0/2142934492 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9c1006b5a0 0x7f9c1006da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.210+0000 7f92d3fe2640 1 -- 172.21.15.83:0/729821505 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92cc068f80 con 0x7f92cc14da80 2024-03-06T07:22:08.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.211+0000 7f9c0b7fe640 1 -- 172.21.15.83:0/2142934492 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9c2412ad00 con 0x7f9c1c0e0ff0 2024-03-06T07:22:08.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.211+0000 7f9c22ffd640 1 --2- 172.21.15.83:0/2142934492 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9c1006b5a0 0x7f9c1006da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.211+0000 7f492ffff640 1 -- 172.21.15.83:0/3581370308 <== mon.2 v2:172.21.15.183:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f49440c6060 con 0x7f493c048040 2024-03-06T07:22:08.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.211+0000 7f9c22ffd640 1 --2- 172.21.15.83:0/2142934492 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9c1006b5a0 0x7f9c1006da60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9c140019e0 tx=0x7f9c14009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.211+0000 7f492ffff640 1 --2- 172.21.15.83:0/3581370308 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f492806b4d0 0x7f492806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.212+0000 7f492ffff640 1 -- 172.21.15.83:0/3581370308 <== mon.2 v2:172.21.15.183:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f49441271e0 con 0x7f493c048040 2024-03-06T07:22:08.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.212+0000 7f49437fe640 1 --2- 172.21.15.83:0/3581370308 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f492806b4d0 0x7f492806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.212+0000 7f49437fe640 1 --2- 172.21.15.83:0/3581370308 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f492806b4d0 0x7f492806d990 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f493c035190 tx=0x7f4938008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.212+0000 7f2c23fff640 1 -- 172.21.15.83:0/3571182664 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2c2c0010c0 con 0x7f2c3c071950 2024-03-06T07:22:08.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.212+0000 7f7d3b7fe640 1 -- 172.21.15.83:0/73000528 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d44017070 con 0x7f7d4c0e78c0 2024-03-06T07:22:08.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.212+0000 7f7d3b7fe640 1 -- 172.21.15.83:0/73000528 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d440035f0 con 0x7f7d4c0e78c0 2024-03-06T07:22:08.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.212+0000 7f7d3b7fe640 1 -- 172.21.15.83:0/73000528 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d440039e0 con 0x7f7d4c0e78c0 2024-03-06T07:22:08.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.212+0000 7f7d595b5640 1 -- 172.21.15.83:0/73000528 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d4c04fa60 con 0x7f7d4c0e78c0 2024-03-06T07:22:08.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.212+0000 7f7d595b5640 1 -- 172.21.15.83:0/73000528 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d4c050020 con 0x7f7d4c0e78c0 2024-03-06T07:22:08.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.214+0000 7f7d397fa640 1 -- 172.21.15.83:0/73000528 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d4c007100 con 0x7f7d4c0e78c0 2024-03-06T07:22:08.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.214+0000 7f7d3b7fe640 1 -- 172.21.15.83:0/73000528 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7d4401a070 con 0x7f7d4c0e78c0 2024-03-06T07:22:08.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.215+0000 7f7d3b7fe640 1 --2- 172.21.15.83:0/73000528 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7d4006b550 0x7f7d4006da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.215+0000 7f7d52ffd640 1 --2- 172.21.15.83:0/73000528 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7d4006b550 0x7f7d4006da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.216+0000 7f7d3b7fe640 1 -- 172.21.15.83:0/73000528 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7d440894f0 con 0x7f7d4c0e78c0 2024-03-06T07:22:08.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.216+0000 7f7d52ffd640 1 --2- 172.21.15.83:0/73000528 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7d4006b550 0x7f7d4006da10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7d540645f0 tx=0x7f7d5406d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.216+0000 7f492ffff640 1 -- 172.21.15.83:0/3581370308 <== mon.2 v2:172.21.15.183:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f49440ba070 con 0x7f493c048040 2024-03-06T07:22:08.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.220+0000 7f9c0b7fe640 1 -- 172.21.15.83:0/2142934492 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9c240a6e20 con 0x7f9c1c0e0ff0 2024-03-06T07:22:08.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.220+0000 7f7d3b7fe640 1 -- 172.21.15.83:0/73000528 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7d44057d00 con 0x7f7d4c0e78c0 2024-03-06T07:22:08.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.231+0000 7f92c27fc640 1 -- 172.21.15.83:0/729821505 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f92cc163560 con 0x7f92cc14da80 2024-03-06T07:22:08.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.239+0000 7fd16fdc9640 1 Processor -- start 2024-03-06T07:22:08.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.239+0000 7fd16fdc9640 1 -- start start 2024-03-06T07:22:08.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.240+0000 7fd16fdc9640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fd1680d1a40 0x7fd1680d1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.240+0000 7fd16fdc9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd16806c610 0x7fd1680d2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.240+0000 7fd16fdc9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1680b5910 con 0x7fd1680b30c0 2024-03-06T07:22:08.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.240+0000 7fd16fdc9640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd16806d8c0 con 0x7fd16806c610 2024-03-06T07:22:08.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.240+0000 7fd16fdc9640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1680d28a0 con 0x7fd1680d1a40 2024-03-06T07:22:08.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.241+0000 7fd16d364640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd16806c610 0x7fd1680d2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.241+0000 7fd16db65640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fd1680d1a40 0x7fd1680d1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.241+0000 7fd16db65640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fd1680d1a40 0x7fd1680d1e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:60098/0 (socket says 172.21.15.83:60098) 2024-03-06T07:22:08.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.241+0000 7fd16db65640 1 -- 172.21.15.83:0/912532161 learned_addr learned my addr 172.21.15.83:0/912532161 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:08.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.242+0000 7fd16d364640 1 -- 172.21.15.83:0/912532161 >> v2:172.21.15.183:3300/0 conn(0x7fd1680d1a40 msgr2=0x7fd1680d1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:08.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.242+0000 7fd16d364640 1 --2- 172.21.15.83:0/912532161 >> v2:172.21.15.183:3300/0 conn(0x7fd1680d1a40 0x7fd1680d1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.242+0000 7fd16d364640 1 -- 172.21.15.83:0/912532161 >> v1:172.21.15.183:6789/0 conn(0x7fd1680b30c0 legacy=0x7fd1680b1430 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-06T07:22:08.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.242+0000 7fd16d364640 1 -- 172.21.15.83:0/912532161 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1680b5bf0 con 0x7fd16806c610 2024-03-06T07:22:08.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.243+0000 7fd16d364640 1 --2- 172.21.15.83:0/912532161 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd16806c610 0x7fd1680d2360 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd168071ba0 tx=0x7fd16402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=970fbc813805286e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.244+0000 7fd16cb63640 1 -- 172.21.15.83:0/912532161 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1640317c0 con 0x7fd16806c610 2024-03-06T07:22:08.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.244+0000 7fd16cb63640 1 -- 172.21.15.83:0/912532161 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd164037070 con 0x7fd16806c610 2024-03-06T07:22:08.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.244+0000 7fd16cb63640 1 -- 172.21.15.83:0/912532161 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd164031bc0 con 0x7fd16806c610 2024-03-06T07:22:08.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.244+0000 7f443573e640 1 Processor -- start 2024-03-06T07:22:08.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.244+0000 7f443573e640 1 -- start start 2024-03-06T07:22:08.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.245+0000 7f443573e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44280eaca0 0x7f44280526b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.245+0000 7f443573e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f442803b0e0 0x7f44280483f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.245+0000 7f443573e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f44280eb080 0x7f4428048930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.245+0000 7f443573e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4428052bf0 con 0x7f442803b0e0 2024-03-06T07:22:08.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.245+0000 7f443573e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f442803a0d0 con 0x7f44280eb080 2024-03-06T07:22:08.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.245+0000 7f443573e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f442803ac50 con 0x7f44280eaca0 2024-03-06T07:22:08.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.246+0000 7f442effd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44280eaca0 0x7f44280526b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.246+0000 7f442effd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44280eaca0 0x7f44280526b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:59348/0 (socket says 172.21.15.83:59348) 2024-03-06T07:22:08.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.246+0000 7f442effd640 1 -- 172.21.15.83:0/2585056701 learned_addr learned my addr 172.21.15.83:0/2585056701 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:08.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.246+0000 7f442effd640 1 -- 172.21.15.83:0/2585056701 >> v2:172.21.15.183:3300/0 conn(0x7f44280eb080 msgr2=0x7f4428048930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:22:08.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.246+0000 7f442effd640 1 --2- 172.21.15.83:0/2585056701 >> v2:172.21.15.183:3300/0 conn(0x7f44280eb080 0x7f4428048930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.246+0000 7f442effd640 1 -- 172.21.15.83:0/2585056701 >> v2:172.21.15.83:3300/0 conn(0x7f442803b0e0 msgr2=0x7f44280483f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:22:08.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.246+0000 7f442e7fc640 1 --2- 172.21.15.83:0/2585056701 >> v2:172.21.15.83:3300/0 conn(0x7f442803b0e0 0x7f44280483f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.246+0000 7f442effd640 1 --2- 172.21.15.83:0/2585056701 >> v2:172.21.15.83:3300/0 conn(0x7f442803b0e0 0x7f44280483f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.246+0000 7f442effd640 1 -- 172.21.15.83:0/2585056701 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f442803a320 con 0x7f44280eaca0 2024-03-06T07:22:08.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.246+0000 7f442e7fc640 1 --2- 172.21.15.83:0/2585056701 >> v2:172.21.15.83:3300/0 conn(0x7f442803b0e0 0x7f44280483f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:08.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.248+0000 7f442effd640 1 --2- 172.21.15.83:0/2585056701 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44280eaca0 0x7f44280526b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4430000b60 tx=0x7f44300a6810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a174b50c64642c45 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.249+0000 7f442dffb640 1 -- 172.21.15.83:0/2585056701 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44300b3070 con 0x7f44280eaca0 2024-03-06T07:22:08.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.249+0000 7f442dffb640 1 -- 172.21.15.83:0/2585056701 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44300ab7f0 con 0x7f44280eaca0 2024-03-06T07:22:08.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.250+0000 7f443573e640 1 -- 172.21.15.83:0/2585056701 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44280eaca0 msgr2=0x7f44280526b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.250+0000 7f443573e640 1 --2- 172.21.15.83:0/2585056701 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44280eaca0 0x7f44280526b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4430000b60 tx=0x7f44300a6810 comp rx=0 tx=0).stop 2024-03-06T07:22:08.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.250+0000 7f443573e640 1 -- 172.21.15.83:0/2585056701 shutdown_connections 2024-03-06T07:22:08.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.250+0000 7f443573e640 1 --2- 172.21.15.83:0/2585056701 >> v2:172.21.15.183:3300/0 conn(0x7f44280eb080 0x7f4428048930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.250+0000 7f443573e640 1 --2- 172.21.15.83:0/2585056701 >> v2:172.21.15.83:3300/0 conn(0x7f442803b0e0 0x7f44280483f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.250+0000 7f443573e640 1 --2- 172.21.15.83:0/2585056701 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44280eaca0 0x7f44280526b0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.250+0000 7f443573e640 1 -- 172.21.15.83:0/2585056701 >> 172.21.15.83:0/2585056701 conn(0x7f442801e6f0 msgr2=0x7f4428055420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.250+0000 7f443573e640 1 -- 172.21.15.83:0/2585056701 shutdown_connections 2024-03-06T07:22:08.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.251+0000 7f443573e640 1 -- 172.21.15.83:0/2585056701 wait complete. 2024-03-06T07:22:08.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.253+0000 7f443573e640 1 Processor -- start 2024-03-06T07:22:08.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.253+0000 7fd16fdc9640 1 -- 172.21.15.83:0/912532161 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd16806c610 msgr2=0x7fd1680d2360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.253+0000 7fd16fdc9640 1 --2- 172.21.15.83:0/912532161 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd16806c610 0x7fd1680d2360 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd168071ba0 tx=0x7fd16402b860 comp rx=0 tx=0).stop 2024-03-06T07:22:08.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.253+0000 7f443573e640 1 -- start start 2024-03-06T07:22:08.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.253+0000 7fd16fdc9640 1 -- 172.21.15.83:0/912532161 shutdown_connections 2024-03-06T07:22:08.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.253+0000 7fd16fdc9640 1 --2- 172.21.15.83:0/912532161 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd16806c610 0x7fd1680d2360 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.253+0000 7fd16fdc9640 1 --2- 172.21.15.83:0/912532161 >> v2:172.21.15.183:3300/0 conn(0x7fd1680d1a40 0x7fd1680d1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.253+0000 7fd16fdc9640 1 -- 172.21.15.83:0/912532161 >> 172.21.15.83:0/912532161 conn(0x7fd16809a9f0 msgr2=0x7fd16809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.254+0000 7f443573e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f442803b0e0 0x7f4428020b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.254+0000 7f443573e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f44280eaca0 0x7f44280e6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.254+0000 7f443573e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44280eb080 0x7f44280e6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.254+0000 7f443573e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4428056000 con 0x7f44280eb080 2024-03-06T07:22:08.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.254+0000 7f443573e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f442803a8e0 con 0x7f44280eaca0 2024-03-06T07:22:08.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.254+0000 7f443573e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4428056180 con 0x7f442803b0e0 2024-03-06T07:22:08.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.255+0000 7f442effd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f442803b0e0 0x7f4428020b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.255+0000 7f442effd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f442803b0e0 0x7f4428020b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:59364/0 (socket says 172.21.15.83:59364) 2024-03-06T07:22:08.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.255+0000 7f442effd640 1 -- 172.21.15.83:0/330878663 learned_addr learned my addr 172.21.15.83:0/330878663 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:08.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.255+0000 7f442e7fc640 1 --2- 172.21.15.83:0/330878663 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f44280eaca0 0x7f44280e6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.255+0000 7f442effd640 1 -- 172.21.15.83:0/330878663 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f44280eaca0 msgr2=0x7f44280e6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.255+0000 7f442effd640 1 --2- 172.21.15.83:0/330878663 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f44280eaca0 0x7f44280e6210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.255+0000 7f442effd640 1 -- 172.21.15.83:0/330878663 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44280eb080 msgr2=0x7f44280e6750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-06T07:22:08.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.256+0000 7f442f7fe640 1 --2- 172.21.15.83:0/330878663 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44280eb080 0x7f44280e6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.256+0000 7f442effd640 1 --2- 172.21.15.83:0/330878663 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44280eb080 0x7f44280e6750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.256+0000 7f442effd640 1 -- 172.21.15.83:0/330878663 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4428021220 con 0x7f442803b0e0 2024-03-06T07:22:08.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.256+0000 7f442f7fe640 1 --2- 172.21.15.83:0/330878663 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44280eb080 0x7f44280e6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:08.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.256+0000 7f442effd640 1 --2- 172.21.15.83:0/330878663 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f442803b0e0 0x7f4428020b40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4430075030 tx=0x7f44300693f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.256+0000 7f44277fe640 1 -- 172.21.15.83:0/330878663 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44300bc070 con 0x7f442803b0e0 2024-03-06T07:22:08.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.256+0000 7f443573e640 1 -- 172.21.15.83:0/330878663 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4428021500 con 0x7f442803b0e0 2024-03-06T07:22:08.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.257+0000 7f443573e640 1 -- 172.21.15.83:0/330878663 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4428021ac0 con 0x7f442803b0e0 2024-03-06T07:22:08.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.257+0000 7f44277fe640 1 -- 172.21.15.83:0/330878663 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44300aee50 con 0x7f442803b0e0 2024-03-06T07:22:08.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.257+0000 7f44277fe640 1 -- 172.21.15.83:0/330878663 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44300b3040 con 0x7f442803b0e0 2024-03-06T07:22:08.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.254+0000 7fd16fdc9640 1 -- 172.21.15.83:0/912532161 shutdown_connections 2024-03-06T07:22:08.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.254+0000 7fd16fdc9640 1 -- 172.21.15.83:0/912532161 wait complete. 2024-03-06T07:22:08.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.255+0000 7fd16fdc9640 1 Processor -- start 2024-03-06T07:22:08.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.259+0000 7f44277fe640 1 -- 172.21.15.83:0/330878663 <== mon.2 v2:172.21.15.183:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f44300bf070 con 0x7f442803b0e0 2024-03-06T07:22:08.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.256+0000 7fd16fdc9640 1 -- start start 2024-03-06T07:22:08.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.257+0000 7fd16fdc9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd16806c610 0x7fd168135e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.257+0000 7fd16fdc9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd1680b30c0 0x7fd1681356c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.257+0000 7fd16fdc9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1680d1a40 0x7fd16815dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.257+0000 7fd16fdc9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1680b6490 con 0x7fd1680d1a40 2024-03-06T07:22:08.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.257+0000 7fd16fdc9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1680b61b0 con 0x7fd16806c610 2024-03-06T07:22:08.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.257+0000 7fd16fdc9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1680b54a0 con 0x7fd1680b30c0 2024-03-06T07:22:08.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.257+0000 7fd16e366640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1680d1a40 0x7fd16815dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.257+0000 7fd16d364640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd1680b30c0 0x7fd1681356c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.257+0000 7fd16e366640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1680d1a40 0x7fd16815dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49580/0 (socket says 172.21.15.83:49580) 2024-03-06T07:22:08.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.257+0000 7fd16e366640 1 -- 172.21.15.83:0/459725552 learned_addr learned my addr 172.21.15.83:0/459725552 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:08.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.257+0000 7fd16db65640 1 --2- 172.21.15.83:0/459725552 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd16806c610 0x7fd168135e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.259+0000 7fd16e366640 1 -- 172.21.15.83:0/459725552 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd1680b30c0 msgr2=0x7fd1681356c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.259+0000 7fd16e366640 1 --2- 172.21.15.83:0/459725552 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd1680b30c0 0x7fd1681356c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.259+0000 7fd16e366640 1 -- 172.21.15.83:0/459725552 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd16806c610 msgr2=0x7fd168135e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.259+0000 7fd16e366640 1 --2- 172.21.15.83:0/459725552 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd16806c610 0x7fd168135e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.259+0000 7fd16e366640 1 -- 172.21.15.83:0/459725552 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd168135c00 con 0x7fd1680d1a40 2024-03-06T07:22:08.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.260+0000 7fd16e366640 1 --2- 172.21.15.83:0/459725552 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1680d1a40 0x7fd16815dc50 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd15800b3f0 tx=0x7fd15800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.261+0000 7f44277fe640 1 --2- 172.21.15.83:0/330878663 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f441006b4d0 0x7f441006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.261+0000 7f443573e640 1 -- 172.21.15.83:0/330878663 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4414005290 con 0x7f442803b0e0 2024-03-06T07:22:08.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.262+0000 7f442e7fc640 1 --2- 172.21.15.83:0/330878663 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f441006b4d0 0x7f441006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.261+0000 7fd15e7fc640 1 -- 172.21.15.83:0/459725552 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd158015020 con 0x7fd1680d1a40 2024-03-06T07:22:08.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.262+0000 7fd15e7fc640 1 -- 172.21.15.83:0/459725552 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd158011850 con 0x7fd1680d1a40 2024-03-06T07:22:08.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.262+0000 7fd15e7fc640 1 -- 172.21.15.83:0/459725552 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd158011c40 con 0x7fd1680d1a40 2024-03-06T07:22:08.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.263+0000 7fd16fdc9640 1 -- 172.21.15.83:0/459725552 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd168162b30 con 0x7fd1680d1a40 2024-03-06T07:22:08.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.263+0000 7fd16fdc9640 1 -- 172.21.15.83:0/459725552 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1681630f0 con 0x7fd1680d1a40 2024-03-06T07:22:08.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.265+0000 7fd15e7fc640 1 -- 172.21.15.83:0/459725552 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd158018020 con 0x7fd1680d1a40 2024-03-06T07:22:08.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.266+0000 7fd16fdc9640 1 -- 172.21.15.83:0/459725552 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd16804f330 con 0x7fd1680d1a40 2024-03-06T07:22:08.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.268+0000 7fd15e7fc640 1 --2- 172.21.15.83:0/459725552 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd14c06b550 0x7fd14c06da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.270+0000 7fd15e7fc640 1 -- 172.21.15.83:0/459725552 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd158089f70 con 0x7fd1680d1a40 2024-03-06T07:22:08.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.274+0000 7fd16db65640 1 --2- 172.21.15.83:0/459725552 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd14c06b550 0x7fd14c06da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.274+0000 7fd16db65640 1 --2- 172.21.15.83:0/459725552 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd14c06b550 0x7fd14c06da10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd1681381b0 tx=0x7fd16000b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.276+0000 7f442e7fc640 1 --2- 172.21.15.83:0/330878663 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f441006b4d0 0x7f441006d990 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f44280365d0 tx=0x7f4420008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.276+0000 7f44277fe640 1 -- 172.21.15.83:0/330878663 <== mon.2 v2:172.21.15.183:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f44301272b0 con 0x7f442803b0e0 2024-03-06T07:22:08.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.276+0000 7fd15e7fc640 1 -- 172.21.15.83:0/459725552 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd15801f020 con 0x7fd1680d1a40 2024-03-06T07:22:08.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.283+0000 7f44277fe640 1 -- 172.21.15.83:0/330878663 <== mon.2 v2:172.21.15.183:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f44300f6090 con 0x7f442803b0e0 2024-03-06T07:22:08.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.291+0000 7f42c9513640 1 Processor -- start 2024-03-06T07:22:08.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.291+0000 7f42c9513640 1 -- start start 2024-03-06T07:22:08.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.292+0000 7f42c9513640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f42c406ee10 0x7f42c406f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.292+0000 7f42c9513640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f42c406e7b0 0x7f42c40cdde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.292+0000 7f42c9513640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42c40b4450 con 0x7f42c4144030 2024-03-06T07:22:08.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.292+0000 7f42c9513640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42c40b4a60 con 0x7f42c406e7b0 2024-03-06T07:22:08.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.292+0000 7f42c9513640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42c406f730 con 0x7f42c406ee10 2024-03-06T07:22:08.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.292+0000 7f42c37fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f42c406e7b0 0x7f42c40cdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.293+0000 7f42c27fc640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f42c4144030 0x7f42c40b4ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:37930/0 (socket says 172.21.15.83:37930) 2024-03-06T07:22:08.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.293+0000 7f42c27fc640 1 -- 172.21.15.83:0/2741563328 learned_addr learned my addr 172.21.15.83:0/2741563328 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:08.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.293+0000 7f42c37fe640 1 -- 172.21.15.83:0/2741563328 >> v2:172.21.15.183:3300/0 conn(0x7f42c406ee10 msgr2=0x7f42c406f1f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:22:08.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.293+0000 7f42c37fe640 1 --2- 172.21.15.83:0/2741563328 >> v2:172.21.15.183:3300/0 conn(0x7f42c406ee10 0x7f42c406f1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.293+0000 7f42c37fe640 1 -- 172.21.15.83:0/2741563328 >> v1:172.21.15.183:6789/0 conn(0x7f42c4144030 legacy=0x7f42c40b4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.293+0000 7f42c37fe640 1 -- 172.21.15.83:0/2741563328 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42c40b5b60 con 0x7f42c406e7b0 2024-03-06T07:22:08.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.295+0000 7f42c37fe640 1 --2- 172.21.15.83:0/2741563328 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42c406e7b0 0x7f42c40cdde0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f42c40705d0 tx=0x7f42b402b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f7371b6ca913e77 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.296+0000 7f42c1ffb640 1 -- 172.21.15.83:0/2741563328 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b4035020 con 0x7f42c406e7b0 2024-03-06T07:22:08.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.296+0000 7f42c1ffb640 1 -- 172.21.15.83:0/2741563328 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42b4031660 con 0x7f42c406e7b0 2024-03-06T07:22:08.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.297+0000 7f42c9513640 1 -- 172.21.15.83:0/2741563328 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42c406e7b0 msgr2=0x7f42c40cdde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.297+0000 7f42c9513640 1 --2- 172.21.15.83:0/2741563328 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42c406e7b0 0x7f42c40cdde0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f42c40705d0 tx=0x7f42b402b990 comp rx=0 tx=0).stop 2024-03-06T07:22:08.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.297+0000 7f42c9513640 1 -- 172.21.15.83:0/2741563328 shutdown_connections 2024-03-06T07:22:08.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.297+0000 7f42c9513640 1 --2- 172.21.15.83:0/2741563328 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42c406e7b0 0x7f42c40cdde0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.297+0000 7f42c9513640 1 --2- 172.21.15.83:0/2741563328 >> v2:172.21.15.183:3300/0 conn(0x7f42c406ee10 0x7f42c406f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.297+0000 7f42c9513640 1 -- 172.21.15.83:0/2741563328 >> 172.21.15.83:0/2741563328 conn(0x7f42c40d0a10 msgr2=0x7f42c40d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.298+0000 7f42c9513640 1 -- 172.21.15.83:0/2741563328 shutdown_connections 2024-03-06T07:22:08.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.298+0000 7f42c9513640 1 -- 172.21.15.83:0/2741563328 wait complete. 2024-03-06T07:22:08.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.300+0000 7f42c9513640 1 Processor -- start 2024-03-06T07:22:08.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.300+0000 7f42c9513640 1 -- start start 2024-03-06T07:22:08.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.301+0000 7f42c9513640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f42c406e7b0 0x7f42c41ec2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.301+0000 7f42c9513640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42c406ee10 0x7f42c40c8980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.302+0000 7f42c27fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42c406ee10 0x7f42c40c8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.302+0000 7f42c27fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42c406ee10 0x7f42c40c8980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49594/0 (socket says 172.21.15.83:49594) 2024-03-06T07:22:08.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.302+0000 7f42c27fc640 1 -- 172.21.15.83:0/2658360624 learned_addr learned my addr 172.21.15.83:0/2658360624 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:08.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.302+0000 7f42c9513640 1 --2- 172.21.15.83:0/2658360624 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f42c4144030 0x7f42c40ca700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.302+0000 7f42c9513640 1 -- 172.21.15.83:0/2658360624 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c4144840 con 0x7f42c406ee10 2024-03-06T07:22:08.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.302+0000 7f42c9513640 1 -- 172.21.15.83:0/2658360624 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c40b46f0 con 0x7f42c4144030 2024-03-06T07:22:08.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.302+0000 7f42c9513640 1 -- 172.21.15.83:0/2658360624 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42c4143c40 con 0x7f42c406e7b0 2024-03-06T07:22:08.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.302+0000 7f42c27fc640 1 -- 172.21.15.83:0/2658360624 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f42c406e7b0 msgr2=0x7f42c41ec2a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-06T07:22:08.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.302+0000 7f42c27fc640 1 --2- 172.21.15.83:0/2658360624 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f42c406e7b0 0x7f42c41ec2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.302+0000 7f42c27fc640 1 -- 172.21.15.83:0/2658360624 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f42c4144030 msgr2=0x7f42c40ca700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-06T07:22:08.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.302+0000 7f42c27fc640 1 --2- 172.21.15.83:0/2658360624 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f42c4144030 0x7f42c40ca700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.302+0000 7f42c27fc640 1 -- 172.21.15.83:0/2658360624 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42c40c6fc0 con 0x7f42c406ee10 2024-03-06T07:22:08.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.303+0000 7f42c27fc640 1 --2- 172.21.15.83:0/2658360624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42c406ee10 0x7f42c40c8980 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f42bc00cc90 tx=0x7f42bc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.304+0000 7f42ab7fe640 1 -- 172.21.15.83:0/2658360624 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42bc018070 con 0x7f42c406ee10 2024-03-06T07:22:08.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.304+0000 7f42ab7fe640 1 -- 172.21.15.83:0/2658360624 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42bc004d80 con 0x7f42c406ee10 2024-03-06T07:22:08.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.304+0000 7f42ab7fe640 1 -- 172.21.15.83:0/2658360624 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42bc00ad80 con 0x7f42c406ee10 2024-03-06T07:22:08.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.304+0000 7f42c9513640 1 -- 172.21.15.83:0/2658360624 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42c40c72a0 con 0x7f42c406ee10 2024-03-06T07:22:08.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.304+0000 7f42c9513640 1 -- 172.21.15.83:0/2658360624 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42c41e92d0 con 0x7f42c406ee10 2024-03-06T07:22:08.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.307+0000 7f42ab7fe640 1 -- 172.21.15.83:0/2658360624 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f42bc01b070 con 0x7f42c406ee10 2024-03-06T07:22:08.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.308+0000 7f42ab7fe640 1 --2- 172.21.15.83:0/2658360624 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f42ac06b6e0 0x7f42ac06dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.308+0000 7f42ab7fe640 1 -- 172.21.15.83:0/2658360624 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f42bc0899b0 con 0x7f42c406ee10 2024-03-06T07:22:08.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.308+0000 7f42c2ffd640 1 --2- 172.21.15.83:0/2658360624 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f42ac06b6e0 0x7f42ac06dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.309+0000 7f42c2ffd640 1 --2- 172.21.15.83:0/2658360624 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f42ac06b6e0 0x7f42ac06dba0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f42c40cd300 tx=0x7f42b8006bd0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.310+0000 7f42c9513640 1 -- 172.21.15.83:0/2658360624 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42a4005290 con 0x7f42c406ee10 2024-03-06T07:22:08.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.319+0000 7f42ab7fe640 1 -- 172.21.15.83:0/2658360624 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f42bc0588a0 con 0x7f42c406ee10 2024-03-06T07:22:08.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.600+0000 7f7d397fa640 1 -- 172.21.15.83:0/73000528 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f7d4c03e570 con 0x7f7d4c0e78c0 2024-03-06T07:22:08.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.600+0000 7f2c40ce9640 1 -- 172.21.15.83:0/3571182664 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f2c3c06dbf0 con 0x7f2c3c071950 2024-03-06T07:22:08.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.601+0000 7f7d3b7fe640 1 -- 172.21.15.83:0/73000528 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7d4405bda0 con 0x7f7d4c0e78c0 2024-03-06T07:22:08.603 INFO:teuthology.orchestra.run.smithi083.stdout:55834574855 2024-03-06T07:22:08.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.602+0000 7f2c23fff640 1 -- 172.21.15.83:0/3571182664 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2c2c057e40 con 0x7f2c3c071950 2024-03-06T07:22:08.604 INFO:teuthology.orchestra.run.smithi083.stdout:55834574856 2024-03-06T07:22:08.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.605+0000 7f7d595b5640 1 -- 172.21.15.83:0/73000528 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7d4006b550 msgr2=0x7f7d4006da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.605+0000 7f7d595b5640 1 --2- 172.21.15.83:0/73000528 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7d4006b550 0x7f7d4006da10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7d540645f0 tx=0x7f7d5406d040 comp rx=0 tx=0).stop 2024-03-06T07:22:08.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.605+0000 7f7d595b5640 1 -- 172.21.15.83:0/73000528 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d4c0e78c0 msgr2=0x7f7d4c039630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.605+0000 7f2c21ffb640 1 -- 172.21.15.83:0/3571182664 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2c2806b5a0 msgr2=0x7f2c2806da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.605+0000 7f7d595b5640 1 --2- 172.21.15.83:0/73000528 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d4c0e78c0 0x7f7d4c039630 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7d4400c3e0 tx=0x7f7d4400c8a0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.605+0000 7f7d595b5640 1 -- 172.21.15.83:0/73000528 shutdown_connections 2024-03-06T07:22:08.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.605+0000 7f7d595b5640 1 --2- 172.21.15.83:0/73000528 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7d4006b550 0x7f7d4006da10 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.605+0000 7f2c21ffb640 1 --2- 172.21.15.83:0/3571182664 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2c2806b5a0 0x7f2c2806da60 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2c30036000 tx=0x7f2c3003d040 comp rx=0 tx=0).stop 2024-03-06T07:22:08.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.605+0000 7f7d595b5640 1 --2- 172.21.15.83:0/73000528 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d4c0e78c0 0x7f7d4c039630 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.605+0000 7f2c21ffb640 1 -- 172.21.15.83:0/3571182664 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c3c071950 msgr2=0x7f2c3c0aed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.605+0000 7f2c21ffb640 1 --2- 172.21.15.83:0/3571182664 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c3c071950 0x7f2c3c0aed10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c00c410 tx=0x7f2c2c00c8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.605+0000 7f7d595b5640 1 --2- 172.21.15.83:0/73000528 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d4c009500 0x7f7d4c0390f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.605+0000 7f7d595b5640 1 --2- 172.21.15.83:0/73000528 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d4c000e20 0x7f7d4c17d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.606+0000 7f2c21ffb640 1 -- 172.21.15.83:0/3571182664 shutdown_connections 2024-03-06T07:22:08.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.606+0000 7f2c21ffb640 1 --2- 172.21.15.83:0/3571182664 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2c2806b5a0 0x7f2c2806da60 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.606+0000 7f2c21ffb640 1 --2- 172.21.15.83:0/3571182664 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2c3c0afd10 0x7f2c3c145c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.606+0000 7f2c21ffb640 1 --2- 172.21.15.83:0/3571182664 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c3c0af930 0x7f2c3c06b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.606+0000 7f2c21ffb640 1 --2- 172.21.15.83:0/3571182664 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c3c071950 0x7f2c3c0aed10 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.606+0000 7f2c21ffb640 1 -- 172.21.15.83:0/3571182664 >> 172.21.15.83:0/3571182664 conn(0x7f2c3c06c0c0 msgr2=0x7f2c3c0d0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.605+0000 7f7d595b5640 1 -- 172.21.15.83:0/73000528 >> 172.21.15.83:0/73000528 conn(0x7f7d4c055070 msgr2=0x7f7d4c04e600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.606+0000 7f7d595b5640 1 -- 172.21.15.83:0/73000528 shutdown_connections 2024-03-06T07:22:08.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.606+0000 7f2c21ffb640 1 -- 172.21.15.83:0/3571182664 shutdown_connections 2024-03-06T07:22:08.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.606+0000 7f2c21ffb640 1 -- 172.21.15.83:0/3571182664 wait complete. 2024-03-06T07:22:08.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.606+0000 7f7d595b5640 1 -- 172.21.15.83:0/73000528 wait complete. 2024-03-06T07:22:08.625 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.3 2024-03-06T07:22:08.627 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.1 2024-03-06T07:22:08.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.648+0000 7f92d3fe2640 1 -- 172.21.15.83:0/729821505 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f92cc0b3aa0 con 0x7f92cc14da80 2024-03-06T07:22:08.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.648+0000 7f4948cc3640 1 -- 172.21.15.83:0/3581370308 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f4924005580 con 0x7f493c048040 2024-03-06T07:22:08.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.648+0000 7f92c27fc640 1 -- 172.21.15.83:0/729821505 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f92cc163560 con 0x7f92cc14da80 2024-03-06T07:22:08.651 INFO:teuthology.orchestra.run.smithi083.stdout:55834574855 2024-03-06T07:22:08.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.649+0000 7f492ffff640 1 -- 172.21.15.83:0/3581370308 <== mon.2 v2:172.21.15.183:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f49440f6170 con 0x7f493c048040 2024-03-06T07:22:08.651 INFO:teuthology.orchestra.run.smithi083.stdout:55834574854 2024-03-06T07:22:08.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.650+0000 7fd16fdc9640 1 -- 172.21.15.83:0/459725552 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fd1681610b0 con 0x7fd1680d1a40 2024-03-06T07:22:08.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.650+0000 7fd15e7fc640 1 -- 172.21.15.83:0/459725552 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd15801f020 con 0x7fd1680d1a40 2024-03-06T07:22:08.652 INFO:teuthology.orchestra.run.smithi083.stdout:55834574855 2024-03-06T07:22:08.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.651+0000 7f92d3fe2640 1 -- 172.21.15.83:0/729821505 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f92b006b900 msgr2=0x7f92b006ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.651+0000 7f92d3fe2640 1 --2- 172.21.15.83:0/729821505 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f92b006b900 0x7f92b006ddc0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f92bc006e80 tx=0x7f92bc008040 comp rx=0 tx=0).stop 2024-03-06T07:22:08.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.651+0000 7f92d3fe2640 1 -- 172.21.15.83:0/729821505 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92cc14da80 msgr2=0x7f92cc14dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.651+0000 7f92d3fe2640 1 --2- 172.21.15.83:0/729821505 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92cc14da80 0x7f92cc14dee0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f92c4032000 tx=0x7f92cc0aeb60 comp rx=0 tx=0).stop 2024-03-06T07:22:08.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.652+0000 7f92d3fe2640 1 -- 172.21.15.83:0/729821505 shutdown_connections 2024-03-06T07:22:08.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.652+0000 7f92d3fe2640 1 --2- 172.21.15.83:0/729821505 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f92b006b900 0x7f92b006ddc0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.652+0000 7f92d3fe2640 1 --2- 172.21.15.83:0/729821505 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92cc14da80 0x7f92cc14dee0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.652+0000 7f92d3fe2640 1 --2- 172.21.15.83:0/729821505 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92cc1fb0e0 0x7f92cc0afe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.652+0000 7f92d3fe2640 1 --2- 172.21.15.83:0/729821505 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92cc06c4c0 0x7f92cc1f85e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.652+0000 7f92d3fe2640 1 -- 172.21.15.83:0/729821505 >> 172.21.15.83:0/729821505 conn(0x7f92cc0b16e0 msgr2=0x7f92cc0cc950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.652+0000 7f92d3fe2640 1 -- 172.21.15.83:0/729821505 shutdown_connections 2024-03-06T07:22:08.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.652+0000 7f92d3fe2640 1 -- 172.21.15.83:0/729821505 wait complete. 2024-03-06T07:22:08.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.652+0000 7f4948cc3640 1 -- 172.21.15.83:0/3581370308 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f492806b4d0 msgr2=0x7f492806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.652+0000 7f4948cc3640 1 --2- 172.21.15.83:0/3581370308 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f492806b4d0 0x7f492806d990 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f493c035190 tx=0x7f4938008040 comp rx=0 tx=0).stop 2024-03-06T07:22:08.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.653+0000 7f4948cc3640 1 -- 172.21.15.83:0/3581370308 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f493c048040 msgr2=0x7f493c0da0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.653+0000 7f4948cc3640 1 --2- 172.21.15.83:0/3581370308 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f493c048040 0x7f493c0da0f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4944075030 tx=0x7f49440693f0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.653+0000 7f4948cc3640 1 -- 172.21.15.83:0/3581370308 shutdown_connections 2024-03-06T07:22:08.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.653+0000 7f4948cc3640 1 --2- 172.21.15.83:0/3581370308 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f492806b4d0 0x7f492806d990 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.653+0000 7f4948cc3640 1 --2- 172.21.15.83:0/3581370308 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f493c048040 0x7f493c0da0f0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.653+0000 7f4948cc3640 1 --2- 172.21.15.83:0/3581370308 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f493c038720 0x7f493c0d9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.653+0000 7f4948cc3640 1 --2- 172.21.15.83:0/3581370308 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f493c037bb0 0x7f493c04ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.653+0000 7f4948cc3640 1 -- 172.21.15.83:0/3581370308 >> 172.21.15.83:0/3581370308 conn(0x7f493c009fb0 msgr2=0x7f493c04dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.653+0000 7fd16fdc9640 1 -- 172.21.15.83:0/459725552 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd14c06b550 msgr2=0x7fd14c06da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.653+0000 7fd16fdc9640 1 --2- 172.21.15.83:0/459725552 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd14c06b550 0x7fd14c06da10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd1681381b0 tx=0x7fd16000b040 comp rx=0 tx=0).stop 2024-03-06T07:22:08.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.653+0000 7f4948cc3640 1 -- 172.21.15.83:0/3581370308 shutdown_connections 2024-03-06T07:22:08.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.653+0000 7fd16fdc9640 1 -- 172.21.15.83:0/459725552 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1680d1a40 msgr2=0x7fd16815dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.653+0000 7fd16fdc9640 1 --2- 172.21.15.83:0/459725552 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1680d1a40 0x7fd16815dc50 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd15800b3f0 tx=0x7fd15800b8b0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.653+0000 7f4948cc3640 1 -- 172.21.15.83:0/3581370308 wait complete. 2024-03-06T07:22:08.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.653+0000 7fd16fdc9640 1 -- 172.21.15.83:0/459725552 shutdown_connections 2024-03-06T07:22:08.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.653+0000 7fd16fdc9640 1 --2- 172.21.15.83:0/459725552 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd14c06b550 0x7fd14c06da10 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.653+0000 7fd16fdc9640 1 --2- 172.21.15.83:0/459725552 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1680d1a40 0x7fd16815dc50 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.653+0000 7fd16fdc9640 1 --2- 172.21.15.83:0/459725552 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd1680b30c0 0x7fd1681356c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.653+0000 7fd16fdc9640 1 --2- 172.21.15.83:0/459725552 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd16806c610 0x7fd168135e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.653+0000 7fd16fdc9640 1 -- 172.21.15.83:0/459725552 >> 172.21.15.83:0/459725552 conn(0x7fd16809a9f0 msgr2=0x7fd16815ff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.653+0000 7fd16fdc9640 1 -- 172.21.15.83:0/459725552 shutdown_connections 2024-03-06T07:22:08.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.653+0000 7fd16fdc9640 1 -- 172.21.15.83:0/459725552 wait complete. 2024-03-06T07:22:08.672 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.2 2024-03-06T07:22:08.673 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.6 2024-03-06T07:22:08.673 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.5 2024-03-06T07:22:08.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.680+0000 7f42c9513640 1 -- 172.21.15.83:0/2658360624 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f42a4005580 con 0x7f42c406ee10 2024-03-06T07:22:08.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.680+0000 7f42ab7fe640 1 -- 172.21.15.83:0/2658360624 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f42bc0581a0 con 0x7f42c406ee10 2024-03-06T07:22:08.682 INFO:teuthology.orchestra.run.smithi083.stdout:55834574856 2024-03-06T07:22:08.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.683+0000 7f42c9513640 1 -- 172.21.15.83:0/2658360624 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f42ac06b6e0 msgr2=0x7f42ac06dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.683+0000 7f42c9513640 1 --2- 172.21.15.83:0/2658360624 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f42ac06b6e0 0x7f42ac06dba0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f42c40cd300 tx=0x7f42b8006bd0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.683+0000 7f42c9513640 1 -- 172.21.15.83:0/2658360624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42c406ee10 msgr2=0x7f42c40c8980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.683+0000 7f42c9513640 1 --2- 172.21.15.83:0/2658360624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42c406ee10 0x7f42c40c8980 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f42bc00cc90 tx=0x7f42bc00a430 comp rx=0 tx=0).stop 2024-03-06T07:22:08.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.683+0000 7f42c9513640 1 -- 172.21.15.83:0/2658360624 shutdown_connections 2024-03-06T07:22:08.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.683+0000 7f42c9513640 1 --2- 172.21.15.83:0/2658360624 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f42ac06b6e0 0x7f42ac06dba0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.683+0000 7f42c9513640 1 --2- 172.21.15.83:0/2658360624 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f42c4144030 0x7f42c40ca700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.683+0000 7f42c9513640 1 --2- 172.21.15.83:0/2658360624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42c406ee10 0x7f42c40c8980 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.683+0000 7f42c9513640 1 --2- 172.21.15.83:0/2658360624 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f42c406e7b0 0x7f42c41ec2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.683+0000 7f42c9513640 1 -- 172.21.15.83:0/2658360624 >> 172.21.15.83:0/2658360624 conn(0x7f42c40d0a10 msgr2=0x7f42c4141910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.683+0000 7f42c9513640 1 -- 172.21.15.83:0/2658360624 shutdown_connections 2024-03-06T07:22:08.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.683+0000 7f42c9513640 1 -- 172.21.15.83:0/2658360624 wait complete. 2024-03-06T07:22:08.696 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.4 2024-03-06T07:22:08.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.724+0000 7f9c23fff640 1 -- 172.21.15.83:0/2142934492 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f9c04005580 con 0x7f9c1c0e0ff0 2024-03-06T07:22:08.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.724+0000 7f9c0b7fe640 1 -- 172.21.15.83:0/2142934492 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9c240f9bf0 con 0x7f9c1c0e0ff0 2024-03-06T07:22:08.726 INFO:teuthology.orchestra.run.smithi083.stdout:55834574855 2024-03-06T07:22:08.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.727+0000 7f9c23fff640 1 -- 172.21.15.83:0/2142934492 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9c1006b5a0 msgr2=0x7f9c1006da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.727+0000 7f9c23fff640 1 --2- 172.21.15.83:0/2142934492 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9c1006b5a0 0x7f9c1006da60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9c140019e0 tx=0x7f9c14009040 comp rx=0 tx=0).stop 2024-03-06T07:22:08.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.727+0000 7f9c23fff640 1 -- 172.21.15.83:0/2142934492 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c1c0e0ff0 msgr2=0x7f9c1c0e35f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.727+0000 7f9c23fff640 1 --2- 172.21.15.83:0/2142934492 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c1c0e0ff0 0x7f9c1c0e35f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9c240645f0 tx=0x7f9c24069a60 comp rx=0 tx=0).stop 2024-03-06T07:22:08.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.727+0000 7f9c23fff640 1 -- 172.21.15.83:0/2142934492 shutdown_connections 2024-03-06T07:22:08.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.727+0000 7f9c23fff640 1 --2- 172.21.15.83:0/2142934492 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9c1006b5a0 0x7f9c1006da60 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.727+0000 7f9c23fff640 1 --2- 172.21.15.83:0/2142934492 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c1c0e0ff0 0x7f9c1c0e35f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.727+0000 7f9c23fff640 1 --2- 172.21.15.83:0/2142934492 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c1c0381e0 0x7f9c1c0dd540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.727+0000 7f9c23fff640 1 --2- 172.21.15.83:0/2142934492 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c1c037d50 0x7f9c1c0dce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.727+0000 7f9c23fff640 1 -- 172.21.15.83:0/2142934492 >> 172.21.15.83:0/2142934492 conn(0x7f9c1c00a3f0 msgr2=0x7f9c1c033bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.727+0000 7f9c23fff640 1 -- 172.21.15.83:0/2142934492 shutdown_connections 2024-03-06T07:22:08.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.727+0000 7f9c23fff640 1 -- 172.21.15.83:0/2142934492 wait complete. 2024-03-06T07:22:08.743 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.0 2024-03-06T07:22:08.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.786+0000 7f443573e640 1 -- 172.21.15.83:0/330878663 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f4414005580 con 0x7f442803b0e0 2024-03-06T07:22:08.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.786+0000 7f44277fe640 1 -- 172.21.15.83:0/330878663 <== mon.2 v2:172.21.15.183:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f44300f5990 con 0x7f442803b0e0 2024-03-06T07:22:08.788 INFO:teuthology.orchestra.run.smithi083.stdout:55834574856 2024-03-06T07:22:08.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.788+0000 7f443573e640 1 -- 172.21.15.83:0/330878663 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f441006b4d0 msgr2=0x7f441006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.788+0000 7f443573e640 1 --2- 172.21.15.83:0/330878663 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f441006b4d0 0x7f441006d990 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f44280365d0 tx=0x7f4420008040 comp rx=0 tx=0).stop 2024-03-06T07:22:08.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.788+0000 7f443573e640 1 -- 172.21.15.83:0/330878663 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f442803b0e0 msgr2=0x7f4428020b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.788+0000 7f443573e640 1 --2- 172.21.15.83:0/330878663 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f442803b0e0 0x7f4428020b40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4430075030 tx=0x7f44300693f0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.788+0000 7f443573e640 1 -- 172.21.15.83:0/330878663 shutdown_connections 2024-03-06T07:22:08.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.788+0000 7f443573e640 1 --2- 172.21.15.83:0/330878663 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f441006b4d0 0x7f441006d990 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.788+0000 7f443573e640 1 --2- 172.21.15.83:0/330878663 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44280eb080 0x7f44280e6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.788+0000 7f443573e640 1 --2- 172.21.15.83:0/330878663 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f44280eaca0 0x7f44280e6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.788+0000 7f443573e640 1 --2- 172.21.15.83:0/330878663 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f442803b0e0 0x7f4428020b40 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.788+0000 7f443573e640 1 -- 172.21.15.83:0/330878663 >> 172.21.15.83:0/330878663 conn(0x7f442801e6f0 msgr2=0x7f44280e7720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.788+0000 7f443573e640 1 -- 172.21.15.83:0/330878663 shutdown_connections 2024-03-06T07:22:08.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:08.788+0000 7f443573e640 1 -- 172.21.15.83:0/330878663 wait complete. 2024-03-06T07:22:08.801 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.7 2024-03-06T07:22:09.626 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-06T07:22:09.627 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-06T07:22:09.673 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-06T07:22:09.674 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-06T07:22:09.674 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-06T07:22:09.696 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-06T07:22:09.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.741+0000 7f209aa71640 1 Processor -- start 2024-03-06T07:22:09.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.741+0000 7f209aa71640 1 -- start start 2024-03-06T07:22:09.744 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-06T07:22:09.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.743+0000 7f209aa71640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f20941627a0 0x7f20941646a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.743+0000 7f209aa71640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2094162e80 con 0x7f209406fe70 2024-03-06T07:22:09.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.743+0000 7f2093fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f209406fe70 0x7f2094163f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:42922/0 (socket says 172.21.15.83:42922) 2024-03-06T07:22:09.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.743+0000 7f2093fff640 1 -- 172.21.15.83:0/149834614 learned_addr learned my addr 172.21.15.83:0/149834614 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:09.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.743+0000 7f209900e640 1 --2- 172.21.15.83:0/149834614 >> v2:172.21.15.183:3300/0 conn(0x7f20941627a0 0x7f20941646a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.743+0000 7f209aa71640 1 -- 172.21.15.83:0/149834614 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2094163160 con 0x7f209406de20 2024-03-06T07:22:09.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.743+0000 7f209aa71640 1 -- 172.21.15.83:0/149834614 --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2094163720 con 0x7f20941627a0 2024-03-06T07:22:09.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.743+0000 7f20937fe640 1 -- 172.21.15.83:0/149834614 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2824553322 0 0) 0x7f2094162e80 con 0x7f209406fe70 2024-03-06T07:22:09.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.743+0000 7f20937fe640 1 -- 172.21.15.83:0/149834614 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2084003520 con 0x7f209406fe70 2024-03-06T07:22:09.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.743+0000 7f20937fe640 1 -- 172.21.15.83:0/149834614 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4167383541 0 0) 0x7f2094163160 con 0x7f209406de20 2024-03-06T07:22:09.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.743+0000 7f20937fe640 1 -- 172.21.15.83:0/149834614 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20840048e0 con 0x7f209406de20 2024-03-06T07:22:09.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.744+0000 7f209900e640 1 -- 172.21.15.83:0/149834614 >> v1:172.21.15.183:6789/0 conn(0x7f209406de20 legacy=0x7f209406e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.744+0000 7f209900e640 1 -- 172.21.15.83:0/149834614 >> v1:172.21.15.83:6789/0 conn(0x7f209406fe70 legacy=0x7f2094163f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.744+0000 7f209900e640 1 -- 172.21.15.83:0/149834614 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2094163440 con 0x7f20941627a0 2024-03-06T07:22:09.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.745+0000 7f209900e640 1 --2- 172.21.15.83:0/149834614 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f20941627a0 0x7f20941646a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f208c009d90 tx=0x7f208c02bec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e79db45026deced7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.745+0000 7f20937fe640 1 -- 172.21.15.83:0/149834614 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f208c039070 con 0x7f20941627a0 2024-03-06T07:22:09.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.745+0000 7f20937fe640 1 -- 172.21.15.83:0/149834614 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f208c02cc70 con 0x7f20941627a0 2024-03-06T07:22:09.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.745+0000 7f20937fe640 1 -- 172.21.15.83:0/149834614 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f208c034640 con 0x7f20941627a0 2024-03-06T07:22:09.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.746+0000 7f209aa71640 1 -- 172.21.15.83:0/149834614 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f20941627a0 msgr2=0x7f20941646a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.746+0000 7f209aa71640 1 --2- 172.21.15.83:0/149834614 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f20941627a0 0x7f20941646a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f208c009d90 tx=0x7f208c02bec0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.747+0000 7f209aa71640 1 -- 172.21.15.83:0/149834614 shutdown_connections 2024-03-06T07:22:09.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.747+0000 7f209aa71640 1 --2- 172.21.15.83:0/149834614 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f20941627a0 0x7f20941646a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.747+0000 7f209aa71640 1 -- 172.21.15.83:0/149834614 >> 172.21.15.83:0/149834614 conn(0x7f20940d0a10 msgr2=0x7f20940d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:09.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.747+0000 7f209aa71640 1 -- 172.21.15.83:0/149834614 shutdown_connections 2024-03-06T07:22:09.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.747+0000 7f209aa71640 1 -- 172.21.15.83:0/149834614 wait complete. 2024-03-06T07:22:09.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.749+0000 7f209aa71640 1 Processor -- start 2024-03-06T07:22:09.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.749+0000 7f209aa71640 1 -- start start 2024-03-06T07:22:09.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.751+0000 7f209aa71640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f209406de20 0x7f209414af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.751+0000 7f209aa71640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f209406fe70 0x7f209414a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.751+0000 7f209aa71640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f20941627a0 0x7f2094160d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.751+0000 7f209880d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f209406de20 0x7f209414af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.751+0000 7f209880d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f209406de20 0x7f209414af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49608/0 (socket says 172.21.15.83:49608) 2024-03-06T07:22:09.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.752+0000 7f2093fff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f209406fe70 0x7f209414a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.752+0000 7f2093fff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f209406fe70 0x7f209414a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:59374/0 (socket says 172.21.15.83:59374) 2024-03-06T07:22:09.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.752+0000 7f209900e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f20941627a0 0x7f2094160d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.752+0000 7f209900e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f20941627a0 0x7f2094160d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:60122/0 (socket says 172.21.15.83:60122) 2024-03-06T07:22:09.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.752+0000 7f209880d640 1 -- 172.21.15.83:0/4267029696 learned_addr learned my addr 172.21.15.83:0/4267029696 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:09.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.752+0000 7f209aa71640 1 -- 172.21.15.83:0/4267029696 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2094162c30 con 0x7f209406de20 2024-03-06T07:22:09.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.752+0000 7f209aa71640 1 -- 172.21.15.83:0/4267029696 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2094163b00 con 0x7f20941627a0 2024-03-06T07:22:09.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.752+0000 7f209aa71640 1 -- 172.21.15.83:0/4267029696 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f209406b930 con 0x7f209406fe70 2024-03-06T07:22:09.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.753+0000 7f209880d640 1 -- 172.21.15.83:0/4267029696 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f209406fe70 msgr2=0x7f209414a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.753+0000 7f209880d640 1 --2- 172.21.15.83:0/4267029696 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f209406fe70 0x7f209414a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.753+0000 7f209880d640 1 -- 172.21.15.83:0/4267029696 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f20941627a0 msgr2=0x7f2094160d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.753+0000 7f209880d640 1 --2- 172.21.15.83:0/4267029696 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f20941627a0 0x7f2094160d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.753+0000 7f209880d640 1 -- 172.21.15.83:0/4267029696 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20940cd3d0 con 0x7f209406de20 2024-03-06T07:22:09.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.753+0000 7f2093fff640 1 --2- 172.21.15.83:0/4267029696 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f209406fe70 0x7f209414a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:09.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.753+0000 7f209900e640 1 --2- 172.21.15.83:0/4267029696 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f20941627a0 0x7f2094160d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:09.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.753+0000 7f209880d640 1 --2- 172.21.15.83:0/4267029696 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f209406de20 0x7f209414af50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f208800c430 tx=0x7f208800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.754+0000 7f20917fa640 1 -- 172.21.15.83:0/4267029696 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2088017070 con 0x7f209406de20 2024-03-06T07:22:09.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.754+0000 7f20917fa640 1 -- 172.21.15.83:0/4267029696 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20880052f0 con 0x7f209406de20 2024-03-06T07:22:09.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.755+0000 7f209aa71640 1 -- 172.21.15.83:0/4267029696 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20940cd690 con 0x7f209406de20 2024-03-06T07:22:09.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.755+0000 7f209aa71640 1 -- 172.21.15.83:0/4267029696 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2094160590 con 0x7f209406de20 2024-03-06T07:22:09.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.756+0000 7f20917fa640 1 -- 172.21.15.83:0/4267029696 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2088003830 con 0x7f209406de20 2024-03-06T07:22:09.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.756+0000 7f209aa71640 1 -- 172.21.15.83:0/4267029696 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f209404f330 con 0x7f209406de20 2024-03-06T07:22:09.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.757+0000 7f20917fa640 1 -- 172.21.15.83:0/4267029696 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f208801a070 con 0x7f209406de20 2024-03-06T07:22:09.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.757+0000 7fcad5e1c640 1 Processor -- start 2024-03-06T07:22:09.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.758+0000 7fcad5e1c640 1 -- start start 2024-03-06T07:22:09.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.758+0000 7f20917fa640 1 --2- 172.21.15.83:0/4267029696 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f207806b5a0 0x7f207806da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.758+0000 7f20917fa640 1 -- 172.21.15.83:0/4267029696 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2088089a00 con 0x7f209406de20 2024-03-06T07:22:09.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.759+0000 7f2093fff640 1 --2- 172.21.15.83:0/4267029696 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f207806b5a0 0x7f207806da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.759+0000 7fcad5e1c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcad006d3c0 0x7fcad00b1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.759+0000 7fcad5e1c640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fcad00b1850 0x7fcad00b5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.759+0000 7fcad5e1c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcad00af4b0 con 0x7fcad01614a0 2024-03-06T07:22:09.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.759+0000 7fcad5e1c640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcad00af790 con 0x7fcad006d3c0 2024-03-06T07:22:09.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.759+0000 7fcad5e1c640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcad00afa70 con 0x7fcad00b1850 2024-03-06T07:22:09.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.759+0000 7fcacf7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fcad01614a0 0x7fcad00b0270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:42932/0 (socket says 172.21.15.83:42932) 2024-03-06T07:22:09.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.759+0000 7fcacf7fe640 1 -- 172.21.15.83:0/1424661671 learned_addr learned my addr 172.21.15.83:0/1424661671 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:09.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.760+0000 7fcaceffd640 1 --2- 172.21.15.83:0/1424661671 >> v2:172.21.15.83:3300/0 conn(0x7fcad006d3c0 0x7fcad00b1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.760+0000 7fcacffff640 1 --2- 172.21.15.83:0/1424661671 >> v2:172.21.15.183:3300/0 conn(0x7fcad00b1850 0x7fcad00b5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.760+0000 7fcacffff640 1 -- 172.21.15.83:0/1424661671 >> v2:172.21.15.83:3300/0 conn(0x7fcad006d3c0 msgr2=0x7fcad00b1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:09.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.760+0000 7fcacffff640 1 --2- 172.21.15.83:0/1424661671 >> v2:172.21.15.83:3300/0 conn(0x7fcad006d3c0 0x7fcad00b1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.760+0000 7f2093fff640 1 --2- 172.21.15.83:0/4267029696 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f207806b5a0 0x7f207806da60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2094158c90 tx=0x7f2080007d30 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.760+0000 7fcacffff640 1 -- 172.21.15.83:0/1424661671 >> v1:172.21.15.83:6789/0 conn(0x7fcad01614a0 legacy=0x7fcad00b0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.760+0000 7fcacffff640 1 -- 172.21.15.83:0/1424661671 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcad00afd50 con 0x7fcad00b1850 2024-03-06T07:22:09.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.761+0000 7fcacffff640 1 --2- 172.21.15.83:0/1424661671 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcad00b1850 0x7fcad00b5510 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcad00b34f0 tx=0x7fcac002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a064a0bc5b595c0e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.763+0000 7fcace7fc640 1 -- 172.21.15.83:0/1424661671 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac0035020 con 0x7fcad00b1850 2024-03-06T07:22:09.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.763+0000 7fcace7fc640 1 -- 172.21.15.83:0/1424661671 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcac0031830 con 0x7fcad00b1850 2024-03-06T07:22:09.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.764+0000 7fcace7fc640 1 -- 172.21.15.83:0/1424661671 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac0031c00 con 0x7fcad00b1850 2024-03-06T07:22:09.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.764+0000 7fcad5e1c640 1 -- 172.21.15.83:0/1424661671 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcad00b1850 msgr2=0x7fcad00b5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.764+0000 7fcad5e1c640 1 --2- 172.21.15.83:0/1424661671 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcad00b1850 0x7fcad00b5510 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcad00b34f0 tx=0x7fcac002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.764+0000 7fcad5e1c640 1 -- 172.21.15.83:0/1424661671 shutdown_connections 2024-03-06T07:22:09.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.764+0000 7fcad5e1c640 1 --2- 172.21.15.83:0/1424661671 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcad00b1850 0x7fcad00b5510 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.764+0000 7fcad5e1c640 1 --2- 172.21.15.83:0/1424661671 >> v2:172.21.15.83:3300/0 conn(0x7fcad006d3c0 0x7fcad00b1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.764+0000 7fcad5e1c640 1 -- 172.21.15.83:0/1424661671 >> 172.21.15.83:0/1424661671 conn(0x7fcad009a9f0 msgr2=0x7fcad009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:09.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.764+0000 7fcad5e1c640 1 -- 172.21.15.83:0/1424661671 shutdown_connections 2024-03-06T07:22:09.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.764+0000 7fcad5e1c640 1 -- 172.21.15.83:0/1424661671 wait complete. 2024-03-06T07:22:09.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.766+0000 7fcad5e1c640 1 Processor -- start 2024-03-06T07:22:09.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.766+0000 7fcad5e1c640 1 -- start start 2024-03-06T07:22:09.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.767+0000 7fcad5e1c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcad006d3c0 0x7fcad0150920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.768+0000 7fcad5e1c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcad00b1850 0x7fcad015ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.768+0000 7fcad5e1c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcad01614a0 0x7fcad006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.768+0000 7fcacf7fe640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcad006d3c0 0x7fcad0150920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.768+0000 7fcacf7fe640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcad006d3c0 0x7fcad0150920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:60128/0 (socket says 172.21.15.83:60128) 2024-03-06T07:22:09.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.768+0000 7fcaceffd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcad00b1850 0x7fcad015ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.768+0000 7fcaceffd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcad00b1850 0x7fcad015ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:59380/0 (socket says 172.21.15.83:59380) 2024-03-06T07:22:09.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.768+0000 7fcacf7fe640 1 -- 172.21.15.83:0/1386484533 learned_addr learned my addr 172.21.15.83:0/1386484533 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:09.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.768+0000 7fcacffff640 1 --2- 172.21.15.83:0/1386484533 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcad01614a0 0x7fcad006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.768+0000 7fcad5e1c640 1 -- 172.21.15.83:0/1386484533 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad00b1db0 con 0x7fcad01614a0 2024-03-06T07:22:09.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.769+0000 7fcad5e1c640 1 -- 172.21.15.83:0/1386484533 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad00b1c30 con 0x7fcad006d3c0 2024-03-06T07:22:09.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.769+0000 7fcad5e1c640 1 -- 172.21.15.83:0/1386484533 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcad00b1f30 con 0x7fcad00b1850 2024-03-06T07:22:09.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.769+0000 7f20917fa640 1 -- 172.21.15.83:0/4267029696 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2088021020 con 0x7f209406de20 2024-03-06T07:22:09.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.770+0000 7fcacf7fe640 1 -- 172.21.15.83:0/1386484533 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcad00b1850 msgr2=0x7fcad015ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.770+0000 7fcacf7fe640 1 --2- 172.21.15.83:0/1386484533 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcad00b1850 0x7fcad015ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.770+0000 7fcacf7fe640 1 -- 172.21.15.83:0/1386484533 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcad01614a0 msgr2=0x7fcad006ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.770+0000 7fcacf7fe640 1 --2- 172.21.15.83:0/1386484533 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcad01614a0 0x7fcad006ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.770+0000 7fcacf7fe640 1 -- 172.21.15.83:0/1386484533 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcad015cbf0 con 0x7fcad006d3c0 2024-03-06T07:22:09.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.771+0000 7fcacf7fe640 1 --2- 172.21.15.83:0/1386484533 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcad006d3c0 0x7fcad0150920 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcac400c4a0 tx=0x7fcac400c960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.771+0000 7fcab7fff640 1 -- 172.21.15.83:0/1386484533 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac4017070 con 0x7fcad006d3c0 2024-03-06T07:22:09.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.771+0000 7fcad5e1c640 1 -- 172.21.15.83:0/1386484533 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcad015ced0 con 0x7fcad006d3c0 2024-03-06T07:22:09.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.771+0000 7fcad5e1c640 1 -- 172.21.15.83:0/1386484533 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcad015d490 con 0x7fcad006d3c0 2024-03-06T07:22:09.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.771+0000 7fcab7fff640 1 -- 172.21.15.83:0/1386484533 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcac4005ce0 con 0x7fcad006d3c0 2024-03-06T07:22:09.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.771+0000 7fcab7fff640 1 -- 172.21.15.83:0/1386484533 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac4007270 con 0x7fcad006d3c0 2024-03-06T07:22:09.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.775+0000 7fcad5e1c640 1 -- 172.21.15.83:0/1386484533 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcab0005290 con 0x7fcad006d3c0 2024-03-06T07:22:09.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.780+0000 7fcab7fff640 1 -- 172.21.15.83:0/1386484533 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcac401a070 con 0x7fcad006d3c0 2024-03-06T07:22:09.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.781+0000 7fcab7fff640 1 --2- 172.21.15.83:0/1386484533 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcabc06b550 0x7fcabc06da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.781+0000 7fcab7fff640 1 -- 172.21.15.83:0/1386484533 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcac40892f0 con 0x7fcad006d3c0 2024-03-06T07:22:09.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.782+0000 7fcaceffd640 1 --2- 172.21.15.83:0/1386484533 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcabc06b550 0x7fcabc06da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.782+0000 7fcaceffd640 1 --2- 172.21.15.83:0/1386484533 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcabc06b550 0x7fcabc06da10 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fcad015c2d0 tx=0x7fcac800b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.787+0000 7fcab7fff640 1 -- 172.21.15.83:0/1386484533 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcac4057c90 con 0x7fcad006d3c0 2024-03-06T07:22:09.802 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-06T07:22:09.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.808+0000 7fc72eed4640 1 Processor -- start 2024-03-06T07:22:09.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.809+0000 7fc72eed4640 1 -- start start 2024-03-06T07:22:09.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.810+0000 7fc72eed4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc7280af7d0 0x7fc7280d0510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.810+0000 7fc72eed4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc72806b730 con 0x7fc7280d0130 2024-03-06T07:22:09.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.810+0000 7fc72eed4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc72806ba10 con 0x7fc7280d0a50 2024-03-06T07:22:09.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.810+0000 7fc72eed4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc72815dd50 con 0x7fc7280af7d0 2024-03-06T07:22:09.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.810+0000 7fc72cc70640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fc7280d0130 0x7fc7280cd430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:42936/0 (socket says 172.21.15.83:42936) 2024-03-06T07:22:09.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.810+0000 7fc72cc70640 1 -- 172.21.15.83:0/4212684929 learned_addr learned my addr 172.21.15.83:0/4212684929 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:09.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.811+0000 7fc7277fe640 1 -- 172.21.15.83:0/4212684929 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3892400218 0 0) 0x7fc72806ba10 con 0x7fc7280d0a50 2024-03-06T07:22:09.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.811+0000 7fc7277fe640 1 -- 172.21.15.83:0/4212684929 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc710003520 con 0x7fc7280d0a50 2024-03-06T07:22:09.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.811+0000 7fc7277fe640 1 -- 172.21.15.83:0/4212684929 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2957127057 0 0) 0x7fc72806b730 con 0x7fc7280d0130 2024-03-06T07:22:09.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.811+0000 7fc7277fe640 1 -- 172.21.15.83:0/4212684929 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc72806ba10 con 0x7fc7280d0130 2024-03-06T07:22:09.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.811+0000 7fc7277fe640 1 -- 172.21.15.83:0/4212684929 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1409653667 0 0) 0x7fc710003520 con 0x7fc7280d0a50 2024-03-06T07:22:09.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.812+0000 7fc7277fe640 1 -- 172.21.15.83:0/4212684929 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc7280af7d0 msgr2=0x7fc7280d0510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:22:09.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.812+0000 7fc7277fe640 1 --2- 172.21.15.83:0/4212684929 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc7280af7d0 0x7fc7280d0510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.812+0000 7fc7277fe640 1 -- 172.21.15.83:0/4212684929 >> v1:172.21.15.83:6789/0 conn(0x7fc7280d0130 legacy=0x7fc7280cd430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.812+0000 7fc7277fe640 1 -- 172.21.15.83:0/4212684929 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc72806bcf0 con 0x7fc7280d0a50 2024-03-06T07:22:09.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.812+0000 7fc7277fe640 1 -- 172.21.15.83:0/4212684929 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7fc71c0015a0 con 0x7fc7280d0a50 2024-03-06T07:22:09.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.812+0000 7fc7277fe640 1 -- 172.21.15.83:0/4212684929 >> v1:172.21.15.183:6789/0 conn(0x7fc7280d0a50 legacy=0x7fc728163f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.812+0000 7fc7277fe640 1 --2- 172.21.15.83:0/4212684929 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc710003b70 0x7fc710003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.812+0000 7fc7277fe640 1 --2- 172.21.15.83:0/4212684929 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc710004800 0x7fc710004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.812+0000 7fc7277fe640 1 --2- 172.21.15.83:0/4212684929 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc710005290 0x7fc71000cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.812+0000 7fc7277fe640 1 -- 172.21.15.83:0/4212684929 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72815dd50 con 0x7fc710004800 2024-03-06T07:22:09.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.812+0000 7fc7277fe640 1 -- 172.21.15.83:0/4212684929 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71000d2f0 con 0x7fc710003b70 2024-03-06T07:22:09.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.812+0000 7fc7277fe640 1 -- 172.21.15.83:0/4212684929 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc71000d4f0 con 0x7fc710005290 2024-03-06T07:22:09.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.812+0000 7fc72cc70640 1 --2- 172.21.15.83:0/4212684929 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc710003b70 0x7fc710003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.812+0000 7fc72d471640 1 --2- 172.21.15.83:0/4212684929 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc710005290 0x7fc71000cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.813+0000 7fc72cc70640 1 -- 172.21.15.83:0/4212684929 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc710005290 msgr2=0x7fc71000cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:09.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.813+0000 7fc72cc70640 1 --2- 172.21.15.83:0/4212684929 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc710005290 0x7fc71000cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.813+0000 7fc72cc70640 1 -- 172.21.15.83:0/4212684929 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc710004800 msgr2=0x7fc710004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:22:09.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.813+0000 7fc72cc70640 1 --2- 172.21.15.83:0/4212684929 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc710004800 0x7fc710004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.813+0000 7fc72cc70640 1 -- 172.21.15.83:0/4212684929 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc710003520 con 0x7fc710003b70 2024-03-06T07:22:09.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.815+0000 7fc72cc70640 1 --2- 172.21.15.83:0/4212684929 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc710003b70 0x7fc710003f50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc72000a760 tx=0x7fc710005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f34a6bbbe8a82409 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.816+0000 7fc7277fe640 1 -- 172.21.15.83:0/4212684929 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72000a330 con 0x7fc710003b70 2024-03-06T07:22:09.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.816+0000 7fc7277fe640 1 -- 172.21.15.83:0/4212684929 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7200072f0 con 0x7fc710003b70 2024-03-06T07:22:09.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.816+0000 7fc72eed4640 1 -- 172.21.15.83:0/4212684929 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc710003b70 msgr2=0x7fc710003f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.816+0000 7fc72eed4640 1 --2- 172.21.15.83:0/4212684929 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc710003b70 0x7fc710003f50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc72000a760 tx=0x7fc710005830 comp rx=0 tx=0).stop 2024-03-06T07:22:09.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.817+0000 7fc72d471640 1 -- 172.21.15.83:0/4212684929 reap_dead start 2024-03-06T07:22:09.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.817+0000 7fc72eed4640 1 -- 172.21.15.83:0/4212684929 shutdown_connections 2024-03-06T07:22:09.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.817+0000 7fc72eed4640 1 -- 172.21.15.83:0/4212684929 >> 172.21.15.83:0/4212684929 conn(0x7fc728072520 msgr2=0x7fc728072930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:09.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.817+0000 7fc72eed4640 1 -- 172.21.15.83:0/4212684929 shutdown_connections 2024-03-06T07:22:09.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.817+0000 7fc72eed4640 1 -- 172.21.15.83:0/4212684929 wait complete. 2024-03-06T07:22:09.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.819+0000 7fc72eed4640 1 Processor -- start 2024-03-06T07:22:09.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.819+0000 7fc72eed4640 1 -- start start 2024-03-06T07:22:09.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.820+0000 7fc72eed4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc728151b40 0x7fc7280cd000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.820+0000 7fc72eed4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc728151f20 0x7fc7280c5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.820+0000 7fc72eed4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7280c60f0 0x7fc7280c6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.820+0000 7fc72eed4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72806c7b0 con 0x7fc728151f20 2024-03-06T07:22:09.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.820+0000 7fc72eed4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72806c340 con 0x7fc7280c60f0 2024-03-06T07:22:09.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.820+0000 7fc72eed4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7280d0e80 con 0x7fc728151b40 2024-03-06T07:22:09.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.821+0000 7fc72d471640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7280c60f0 0x7fc7280c6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.821+0000 7fc72d471640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7280c60f0 0x7fc7280c6570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:60142/0 (socket says 172.21.15.83:60142) 2024-03-06T07:22:09.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.821+0000 7fc72d471640 1 -- 172.21.15.83:0/34234774 learned_addr learned my addr 172.21.15.83:0/34234774 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:09.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.821+0000 7fc72cc70640 1 --2- 172.21.15.83:0/34234774 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc728151b40 0x7fc7280cd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.821+0000 7fc72d471640 1 -- 172.21.15.83:0/34234774 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc728151b40 msgr2=0x7fc7280cd000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.821+0000 7fc72d471640 1 --2- 172.21.15.83:0/34234774 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc728151b40 0x7fc7280cd000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.821+0000 7fc72d471640 1 -- 172.21.15.83:0/34234774 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc728151f20 msgr2=0x7fc7280c5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.821+0000 7fc727fff640 1 --2- 172.21.15.83:0/34234774 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc728151f20 0x7fc7280c5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.821+0000 7fc72d471640 1 --2- 172.21.15.83:0/34234774 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc728151f20 0x7fc7280c5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.821+0000 7fc72d471640 1 -- 172.21.15.83:0/34234774 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc72809d0f0 con 0x7fc7280c60f0 2024-03-06T07:22:09.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.822+0000 7fc727fff640 1 --2- 172.21.15.83:0/34234774 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc728151f20 0x7fc7280c5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:09.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.822+0000 7fc72d471640 1 --2- 172.21.15.83:0/34234774 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7280c60f0 0x7fc7280c6570 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc71c001eb0 tx=0x7fc71c00e9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.823+0000 7fc7257fa640 1 -- 172.21.15.83:0/34234774 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71c0052e0 con 0x7fc7280c60f0 2024-03-06T07:22:09.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.823+0000 7fc7257fa640 1 -- 172.21.15.83:0/34234774 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc71c018070 con 0x7fc7280c60f0 2024-03-06T07:22:09.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.823+0000 7fc7257fa640 1 -- 172.21.15.83:0/34234774 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71c003830 con 0x7fc7280c60f0 2024-03-06T07:22:09.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.823+0000 7fc72eed4640 1 -- 172.21.15.83:0/34234774 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc72809d3d0 con 0x7fc7280c60f0 2024-03-06T07:22:09.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.823+0000 7fc72eed4640 1 -- 172.21.15.83:0/34234774 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc72809d990 con 0x7fc7280c60f0 2024-03-06T07:22:09.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.826+0000 7fc7257fa640 1 -- 172.21.15.83:0/34234774 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc71c0039d0 con 0x7fc7280c60f0 2024-03-06T07:22:09.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.827+0000 7fc7257fa640 1 --2- 172.21.15.83:0/34234774 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc71406b5a0 0x7fc71406da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.827+0000 7fc7257fa640 1 -- 172.21.15.83:0/34234774 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc71c089cb0 con 0x7fc7280c60f0 2024-03-06T07:22:09.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.827+0000 7fc72cc70640 1 --2- 172.21.15.83:0/34234774 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc71406b5a0 0x7fc71406da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.828+0000 7fc72eed4640 1 -- 172.21.15.83:0/34234774 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc708005290 con 0x7fc7280c60f0 2024-03-06T07:22:09.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.830+0000 7fc72cc70640 1 --2- 172.21.15.83:0/34234774 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc71406b5a0 0x7fc71406da60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc72000a730 tx=0x7fc720009000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.832+0000 7f5f3581c640 1 Processor -- start 2024-03-06T07:22:09.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.832+0000 7f5f3581c640 1 -- start start 2024-03-06T07:22:09.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.833+0000 7f5f3581c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f28100cf0 0x7f5f2803a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.833+0000 7f5f3581c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5f28054690 0x7f5f28104dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.833+0000 7f5f3581c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f28037b50 con 0x7f5f28104970 2024-03-06T07:22:09.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.833+0000 7f5f3581c640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f28105310 con 0x7f5f28054690 2024-03-06T07:22:09.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.833+0000 7f5f3581c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f28007340 con 0x7f5f28100cf0 2024-03-06T07:22:09.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.834+0000 7f5f2effd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f28100cf0 0x7f5f2803a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.834+0000 7f5f2effd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f28100cf0 0x7f5f2803a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:59398/0 (socket says 172.21.15.83:59398) 2024-03-06T07:22:09.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.834+0000 7f5f2f7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5f28054690 0x7f5f28104dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.834+0000 7f5f2effd640 1 -- 172.21.15.83:0/3777634073 learned_addr learned my addr 172.21.15.83:0/3777634073 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:09.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.834+0000 7f5f2dffb640 1 -- 172.21.15.83:0/3777634073 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2972370648 0 0) 0x7f5f28037b50 con 0x7f5f28104970 2024-03-06T07:22:09.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.834+0000 7f5f2dffb640 1 -- 172.21.15.83:0/3777634073 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f18003500 con 0x7f5f28104970 2024-03-06T07:22:09.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.835+0000 7f5f2dffb640 1 -- 172.21.15.83:0/3777634073 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1342462025 0 0) 0x7f5f18003500 con 0x7f5f28104970 2024-03-06T07:22:09.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.835+0000 7f5f2dffb640 1 -- 172.21.15.83:0/3777634073 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f28100cf0 msgr2=0x7f5f2803a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:09.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.835+0000 7f5f2dffb640 1 --2- 172.21.15.83:0/3777634073 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f28100cf0 0x7f5f2803a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.835+0000 7f5f2dffb640 1 -- 172.21.15.83:0/3777634073 >> v2:172.21.15.83:3300/0 conn(0x7f5f28054690 msgr2=0x7f5f28104dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:09.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.835+0000 7f5f2dffb640 1 --2- 172.21.15.83:0/3777634073 >> v2:172.21.15.83:3300/0 conn(0x7f5f28054690 0x7f5f28104dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.835+0000 7f5f2dffb640 1 -- 172.21.15.83:0/3777634073 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f28007620 con 0x7f5f28104970 2024-03-06T07:22:09.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.835+0000 7f5f2dffb640 1 -- 172.21.15.83:0/3777634073 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f5f20001670 con 0x7f5f28104970 2024-03-06T07:22:09.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.835+0000 7f5f2dffb640 1 -- 172.21.15.83:0/3777634073 >> v1:172.21.15.183:6789/0 conn(0x7f5f28104970 legacy=0x7f5f28005ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.835+0000 7f5f2dffb640 1 --2- 172.21.15.83:0/3777634073 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f18003b10 0x7f5f18005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.835+0000 7f5f2dffb640 1 --2- 172.21.15.83:0/3777634073 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f18006580 0x7f5f18006a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.835+0000 7f5f2dffb640 1 --2- 172.21.15.83:0/3777634073 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f18007020 0x7f5f180074a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.835+0000 7f5f2dffb640 1 -- 172.21.15.83:0/3777634073 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f28007340 con 0x7f5f18007020 2024-03-06T07:22:09.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.835+0000 7f5f2dffb640 1 -- 172.21.15.83:0/3777634073 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f180102a0 con 0x7f5f18003b10 2024-03-06T07:22:09.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.835+0000 7f5f2dffb640 1 -- 172.21.15.83:0/3777634073 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f180104a0 con 0x7f5f18006580 2024-03-06T07:22:09.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.835+0000 7fc7257fa640 1 -- 172.21.15.83:0/34234774 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc71c058b20 con 0x7fc7280c60f0 2024-03-06T07:22:09.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.835+0000 7f5f2effd640 1 --2- 172.21.15.83:0/3777634073 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f18003b10 0x7f5f18005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.835+0000 7f5f2e7fc640 1 --2- 172.21.15.83:0/3777634073 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f18006580 0x7f5f18006a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.835+0000 7f5f2f7fe640 1 --2- 172.21.15.83:0/3777634073 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f18007020 0x7f5f180074a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.836+0000 7f5f2e7fc640 1 -- 172.21.15.83:0/3777634073 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f18003b10 msgr2=0x7f5f18005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:09.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.836+0000 7f5f2e7fc640 1 --2- 172.21.15.83:0/3777634073 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f18003b10 0x7f5f18005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.836+0000 7f5f2e7fc640 1 -- 172.21.15.83:0/3777634073 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f18007020 msgr2=0x7f5f180074a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:09.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.836+0000 7f5f2e7fc640 1 --2- 172.21.15.83:0/3777634073 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f18007020 0x7f5f180074a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.835+0000 7f2c36b67640 1 Processor -- start 2024-03-06T07:22:09.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.836+0000 7f2c36b67640 1 -- start start 2024-03-06T07:22:09.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.837+0000 7f2c36b67640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2c280df220 0x7f2c28038080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.837+0000 7f2c36b67640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2c28053c70 0x7f2c2804fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.837+0000 7f2c36b67640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f2c28033ef0 0x7f2c2804ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.836+0000 7f5f2e7fc640 1 -- 172.21.15.83:0/3777634073 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f28007620 con 0x7f5f18006580 2024-03-06T07:22:09.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.837+0000 7f5f2f7fe640 1 --2- 172.21.15.83:0/3777634073 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f18007020 0x7f5f180074a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:09.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.837+0000 7f2c36b67640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c280544c0 con 0x7f2c280df220 2024-03-06T07:22:09.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.837+0000 7f2c36b67640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c280541e0 con 0x7f2c28033ef0 2024-03-06T07:22:09.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.837+0000 7f2c36b67640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c28005f30 con 0x7f2c28053c70 2024-03-06T07:22:09.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.837+0000 7f2c35b65640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2c280df220 0x7f2c28038080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.837+0000 7f2c35b65640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2c280df220 0x7f2c28038080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49676/0 (socket says 172.21.15.83:49676) 2024-03-06T07:22:09.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.837+0000 7f2c35b65640 1 -- 172.21.15.83:0/180967713 learned_addr learned my addr 172.21.15.83:0/180967713 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:09.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.837+0000 7f2c36366640 1 --2- 172.21.15.83:0/180967713 >> v2:172.21.15.183:3300/0 conn(0x7f2c28033ef0 0x7f2c2804ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.837+0000 7f2c35364640 1 --2- 172.21.15.83:0/180967713 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2c28053c70 0x7f2c2804fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.838+0000 7f2c36366640 1 -- 172.21.15.83:0/180967713 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2c28053c70 msgr2=0x7f2c2804fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:09.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.838+0000 7f2c36366640 1 --2- 172.21.15.83:0/180967713 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2c28053c70 0x7f2c2804fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.838+0000 7f2c36366640 1 -- 172.21.15.83:0/180967713 >> v2:172.21.15.83:3300/0 conn(0x7f2c280df220 msgr2=0x7f2c28038080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:09.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.838+0000 7f2c36366640 1 --2- 172.21.15.83:0/180967713 >> v2:172.21.15.83:3300/0 conn(0x7f2c280df220 0x7f2c28038080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.838+0000 7f2c36366640 1 -- 172.21.15.83:0/180967713 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c28006210 con 0x7f2c28033ef0 2024-03-06T07:22:09.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.838+0000 7f5f2e7fc640 1 --2- 172.21.15.83:0/3777634073 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f18006580 0x7f5f18006a00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5f2000c190 tx=0x7f5f18003890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb67ff15439b3eac server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.838+0000 7f2c35364640 1 --2- 172.21.15.83:0/180967713 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2c28053c70 0x7f2c2804fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:09.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.838+0000 7f5f2dffb640 1 -- 172.21.15.83:0/3777634073 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f20039070 con 0x7f5f18006580 2024-03-06T07:22:09.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.838+0000 7f5f2dffb640 1 -- 172.21.15.83:0/3777634073 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f2002ca70 con 0x7f5f18006580 2024-03-06T07:22:09.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.838+0000 7f5f2dffb640 1 -- 172.21.15.83:0/3777634073 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f20007c80 con 0x7f5f18006580 2024-03-06T07:22:09.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.839+0000 7f2c36366640 1 --2- 172.21.15.83:0/180967713 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c28033ef0 0x7f2c2804ffd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2c28033c60 tx=0x7f2c20033fc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1ca12d52c102a4f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.839+0000 7f2c34b63640 1 -- 172.21.15.83:0/180967713 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c20034b60 con 0x7f2c28033ef0 2024-03-06T07:22:09.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.839+0000 7f2c34b63640 1 -- 172.21.15.83:0/180967713 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c20034d00 con 0x7f2c28033ef0 2024-03-06T07:22:09.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.839+0000 7f5f3581c640 1 -- 172.21.15.83:0/3777634073 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f18006580 msgr2=0x7f5f18006a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.839+0000 7f5f3581c640 1 --2- 172.21.15.83:0/3777634073 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f18006580 0x7f5f18006a00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5f2000c190 tx=0x7f5f18003890 comp rx=0 tx=0).stop 2024-03-06T07:22:09.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.840+0000 7f5f2f7fe640 1 -- 172.21.15.83:0/3777634073 reap_dead start 2024-03-06T07:22:09.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.840+0000 7f5f3581c640 1 -- 172.21.15.83:0/3777634073 shutdown_connections 2024-03-06T07:22:09.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.840+0000 7f5f3581c640 1 -- 172.21.15.83:0/3777634073 >> 172.21.15.83:0/3777634073 conn(0x7f5f28009fe0 msgr2=0x7f5f2800a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:09.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.840+0000 7f2c34b63640 1 -- 172.21.15.83:0/180967713 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c2000b670 con 0x7f2c28033ef0 2024-03-06T07:22:09.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.840+0000 7f5f3581c640 1 -- 172.21.15.83:0/3777634073 shutdown_connections 2024-03-06T07:22:09.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.840+0000 7f5f3581c640 1 -- 172.21.15.83:0/3777634073 wait complete. 2024-03-06T07:22:09.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.840+0000 7f2c36b67640 1 -- 172.21.15.83:0/180967713 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c28033ef0 msgr2=0x7f2c2804ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.840+0000 7f2c36b67640 1 --2- 172.21.15.83:0/180967713 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c28033ef0 0x7f2c2804ffd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2c28033c60 tx=0x7f2c20033fc0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.840+0000 7f2c36b67640 1 -- 172.21.15.83:0/180967713 shutdown_connections 2024-03-06T07:22:09.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.840+0000 7f2c36b67640 1 --2- 172.21.15.83:0/180967713 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c28033ef0 0x7f2c2804ffd0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.841+0000 7f2c36b67640 1 --2- 172.21.15.83:0/180967713 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2c28053c70 0x7f2c2804fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.841+0000 7f2c36b67640 1 --2- 172.21.15.83:0/180967713 >> v2:172.21.15.83:3300/0 conn(0x7f2c280df220 0x7f2c28038080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.841+0000 7f2c36b67640 1 -- 172.21.15.83:0/180967713 >> 172.21.15.83:0/180967713 conn(0x7f2c28009fb0 msgr2=0x7f2c2800a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:09.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.842+0000 7f2c36b67640 1 -- 172.21.15.83:0/180967713 shutdown_connections 2024-03-06T07:22:09.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.842+0000 7f5f3581c640 1 Processor -- start 2024-03-06T07:22:09.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.842+0000 7f5f3581c640 1 -- start start 2024-03-06T07:22:09.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.842+0000 7f2c36b67640 1 -- 172.21.15.83:0/180967713 wait complete. 2024-03-06T07:22:09.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.844+0000 7f5f3581c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f280f1490 0x7f5f280fceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.844+0000 7f5f2effd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f280f1490 0x7f5f280fceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.844+0000 7f5f2effd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f280f1490 0x7f5f280fceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:59430/0 (socket says 172.21.15.83:59430) 2024-03-06T07:22:09.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.845+0000 7f5f3581c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f280fd3f0 0x7f5f2819a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.845+0000 7f5f2effd640 1 -- 172.21.15.83:0/1663044662 learned_addr learned my addr 172.21.15.83:0/1663044662 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:09.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.845+0000 7f5f3581c640 1 --2- 172.21.15.83:0/1663044662 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f2819a6d0 0x7f5f2804d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.845+0000 7f5f2e7fc640 1 --2- 172.21.15.83:0/1663044662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f280fd3f0 0x7f5f2819a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.845+0000 7f5f3581c640 1 -- 172.21.15.83:0/1663044662 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f281010d0 con 0x7f5f280fd3f0 2024-03-06T07:22:09.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.845+0000 7f5f3581c640 1 -- 172.21.15.83:0/1663044662 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f28053160 con 0x7f5f2819a6d0 2024-03-06T07:22:09.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.845+0000 7f5f3581c640 1 -- 172.21.15.83:0/1663044662 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f28101250 con 0x7f5f280f1490 2024-03-06T07:22:09.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.846+0000 7f5f2effd640 1 -- 172.21.15.83:0/1663044662 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f2819a6d0 msgr2=0x7f5f2804d710 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-06T07:22:09.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.846+0000 7f5f2effd640 1 --2- 172.21.15.83:0/1663044662 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f2819a6d0 0x7f5f2804d710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.846+0000 7f5f2effd640 1 -- 172.21.15.83:0/1663044662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f280fd3f0 msgr2=0x7f5f2819a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.846+0000 7f5f2effd640 1 --2- 172.21.15.83:0/1663044662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f280fd3f0 0x7f5f2819a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.846+0000 7f5f2effd640 1 -- 172.21.15.83:0/1663044662 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f2819aee0 con 0x7f5f280f1490 2024-03-06T07:22:09.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.848+0000 7f5f2e7fc640 1 --2- 172.21.15.83:0/1663044662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f280fd3f0 0x7f5f2819a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:09.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.848+0000 7f5f2effd640 1 --2- 172.21.15.83:0/1663044662 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f280f1490 0x7f5f280fceb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5f3002d580 tx=0x7f5f300683c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.848+0000 7f5f17fff640 1 -- 172.21.15.83:0/1663044662 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f30089070 con 0x7f5f280f1490 2024-03-06T07:22:09.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.848+0000 7f5f17fff640 1 -- 172.21.15.83:0/1663044662 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f30084480 con 0x7f5f280f1490 2024-03-06T07:22:09.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.848+0000 7f5f17fff640 1 -- 172.21.15.83:0/1663044662 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f30073660 con 0x7f5f280f1490 2024-03-06T07:22:09.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.850+0000 7f5f3581c640 1 -- 172.21.15.83:0/1663044662 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f2804bc60 con 0x7f5f280f1490 2024-03-06T07:22:09.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.850+0000 7f5f3581c640 1 -- 172.21.15.83:0/1663044662 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f2804c220 con 0x7f5f280f1490 2024-03-06T07:22:09.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.851+0000 7f5f15ffb640 1 -- 172.21.15.83:0/1663044662 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f28008a00 con 0x7f5f280f1490 2024-03-06T07:22:09.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.853+0000 7f5f17fff640 1 -- 172.21.15.83:0/1663044662 <== mon.2 v2:172.21.15.183:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5f3008c070 con 0x7f5f280f1490 2024-03-06T07:22:09.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.853+0000 7f5f17fff640 1 --2- 172.21.15.83:0/1663044662 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5f1c06b5a0 0x7f5f1c06da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.853+0000 7f5f17fff640 1 -- 172.21.15.83:0/1663044662 <== mon.2 v2:172.21.15.183:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5f300f3f50 con 0x7f5f280f1490 2024-03-06T07:22:09.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.854+0000 7f5f2e7fc640 1 --2- 172.21.15.83:0/1663044662 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5f1c06b5a0 0x7f5f1c06da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.854+0000 7f2c36b67640 1 Processor -- start 2024-03-06T07:22:09.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.854+0000 7f5f2e7fc640 1 --2- 172.21.15.83:0/1663044662 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5f1c06b5a0 0x7f5f1c06da60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5f20036000 tx=0x7f5f20030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.855+0000 7f2c36b67640 1 -- start start 2024-03-06T07:22:09.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.857+0000 7f5f17fff640 1 -- 172.21.15.83:0/1663044662 <== mon.2 v2:172.21.15.183:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5f30073070 con 0x7f5f280f1490 2024-03-06T07:22:09.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.864+0000 7f2c36b67640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c28033ef0 0x7f2c280485f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.864+0000 7f2c36b67640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2c28053c70 0x7f2c280d0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.864+0000 7f2c36b67640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c280df220 0x7f2c280e0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.865+0000 7f2c36366640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c280df220 0x7f2c280e0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.865+0000 7f2c36366640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c280df220 0x7f2c280e0970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49692/0 (socket says 172.21.15.83:49692) 2024-03-06T07:22:09.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.866+0000 7f2c36366640 1 -- 172.21.15.83:0/714377576 learned_addr learned my addr 172.21.15.83:0/714377576 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:09.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.866+0000 7f2c36b67640 1 -- 172.21.15.83:0/714377576 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c28037a90 con 0x7f2c280df220 2024-03-06T07:22:09.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.866+0000 7f2c36b67640 1 -- 172.21.15.83:0/714377576 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c28037910 con 0x7f2c28033ef0 2024-03-06T07:22:09.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.866+0000 7f2c36b67640 1 -- 172.21.15.83:0/714377576 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c280df600 con 0x7f2c28053c70 2024-03-06T07:22:09.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.867+0000 7f2c35364640 1 --2- 172.21.15.83:0/714377576 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2c28053c70 0x7f2c280d0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.867+0000 7f2c35b65640 1 --2- 172.21.15.83:0/714377576 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c28033ef0 0x7f2c280485f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.870+0000 7f2c36366640 1 -- 172.21.15.83:0/714377576 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2c28053c70 msgr2=0x7f2c280d0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.870+0000 7f2c36366640 1 --2- 172.21.15.83:0/714377576 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2c28053c70 0x7f2c280d0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.870+0000 7f2c36366640 1 -- 172.21.15.83:0/714377576 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c28033ef0 msgr2=0x7f2c280485f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.874+0000 7f2c36366640 1 --2- 172.21.15.83:0/714377576 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c28033ef0 0x7f2c280485f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.874+0000 7f2c36366640 1 -- 172.21.15.83:0/714377576 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c2804cd80 con 0x7f2c280df220 2024-03-06T07:22:09.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.874+0000 7f2c35364640 1 --2- 172.21.15.83:0/714377576 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2c28053c70 0x7f2c280d0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:09.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.874+0000 7f2c35b65640 1 --2- 172.21.15.83:0/714377576 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c28033ef0 0x7f2c280485f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:09.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.874+0000 7f2c36366640 1 --2- 172.21.15.83:0/714377576 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c280df220 0x7f2c280e0970 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2c2000dcb0 tx=0x7f2c200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.875+0000 7f2c267fc640 1 -- 172.21.15.83:0/714377576 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c2004b020 con 0x7f2c280df220 2024-03-06T07:22:09.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.875+0000 7f2c267fc640 1 -- 172.21.15.83:0/714377576 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c20047670 con 0x7f2c280df220 2024-03-06T07:22:09.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.875+0000 7f2c267fc640 1 -- 172.21.15.83:0/714377576 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c20006e10 con 0x7f2c280df220 2024-03-06T07:22:09.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.876+0000 7f2c36b67640 1 -- 172.21.15.83:0/714377576 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c280e13c0 con 0x7f2c280df220 2024-03-06T07:22:09.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.876+0000 7f2c36b67640 1 -- 172.21.15.83:0/714377576 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c280e1980 con 0x7f2c280df220 2024-03-06T07:22:09.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.879+0000 7f2c267fc640 1 -- 172.21.15.83:0/714377576 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2c2004e020 con 0x7f2c280df220 2024-03-06T07:22:09.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.879+0000 7f2c267fc640 1 --2- 172.21.15.83:0/714377576 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2c1406b4d0 0x7f2c1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.879+0000 7f2c267fc640 1 -- 172.21.15.83:0/714377576 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2c200b4d30 con 0x7f2c280df220 2024-03-06T07:22:09.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.881+0000 7f2c36b67640 1 -- 172.21.15.83:0/714377576 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c280e1bc0 con 0x7f2c280df220 2024-03-06T07:22:09.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.888+0000 7f403e8f4640 1 Processor -- start 2024-03-06T07:22:09.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.888+0000 7f403e8f4640 1 -- start start 2024-03-06T07:22:09.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.887+0000 7f2c35b65640 1 --2- 172.21.15.83:0/714377576 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2c1406b4d0 0x7f2c1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.888+0000 7f2c35b65640 1 --2- 172.21.15.83:0/714377576 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2c1406b4d0 0x7f2c1406d990 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2c3006a8c0 tx=0x7f2c3008d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.890+0000 7f403e8f4640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f40380b3450 0x7f4038071150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.890+0000 7f403e8f4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4038071690 0x7f40380ae4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.890+0000 7f403e8f4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40380b3b10 con 0x7f403816aab0 2024-03-06T07:22:09.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.890+0000 7f403e8f4640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40380b38b0 con 0x7f40380b3450 2024-03-06T07:22:09.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.890+0000 7f403e8f4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40380b3df0 con 0x7f4038071690 2024-03-06T07:22:09.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.890+0000 7f403d8f2640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f403816aab0 0x7f40380b2d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:42940/0 (socket says 172.21.15.83:42940) 2024-03-06T07:22:09.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.890+0000 7f403d8f2640 1 -- 172.21.15.83:0/3095693875 learned_addr learned my addr 172.21.15.83:0/3095693875 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:09.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.890+0000 7f403e0f3640 1 --2- 172.21.15.83:0/3095693875 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4038071690 0x7f40380ae4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.891+0000 7f403d0f1640 1 --2- 172.21.15.83:0/3095693875 >> v2:172.21.15.183:3300/0 conn(0x7f40380b3450 0x7f4038071150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.891+0000 7f403e0f3640 1 -- 172.21.15.83:0/3095693875 >> v2:172.21.15.183:3300/0 conn(0x7f40380b3450 msgr2=0x7f4038071150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:09.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.891+0000 7f403e0f3640 1 --2- 172.21.15.83:0/3095693875 >> v2:172.21.15.183:3300/0 conn(0x7f40380b3450 0x7f4038071150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.891+0000 7f403e0f3640 1 -- 172.21.15.83:0/3095693875 >> v1:172.21.15.83:6789/0 conn(0x7f403816aab0 legacy=0x7f40380b2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.891+0000 7f403e0f3640 1 -- 172.21.15.83:0/3095693875 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40380b40d0 con 0x7f4038071690 2024-03-06T07:22:09.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.892+0000 7f403e0f3640 1 --2- 172.21.15.83:0/3095693875 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4038071690 0x7f40380ae4a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4030009d90 tx=0x7f403002bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6198fa797e9fb14 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.893+0000 7f403c8f0640 1 -- 172.21.15.83:0/3095693875 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4030039070 con 0x7f4038071690 2024-03-06T07:22:09.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.893+0000 7f403c8f0640 1 -- 172.21.15.83:0/3095693875 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f403002cd00 con 0x7f4038071690 2024-03-06T07:22:09.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.894+0000 7f403e8f4640 1 -- 172.21.15.83:0/3095693875 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4038071690 msgr2=0x7f40380ae4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.894+0000 7f403e8f4640 1 --2- 172.21.15.83:0/3095693875 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4038071690 0x7f40380ae4a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4030009d90 tx=0x7f403002bf50 comp rx=0 tx=0).stop 2024-03-06T07:22:09.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.894+0000 7f403e8f4640 1 -- 172.21.15.83:0/3095693875 shutdown_connections 2024-03-06T07:22:09.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.894+0000 7f403e8f4640 1 --2- 172.21.15.83:0/3095693875 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4038071690 0x7f40380ae4a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.894+0000 7f403e8f4640 1 --2- 172.21.15.83:0/3095693875 >> v2:172.21.15.183:3300/0 conn(0x7f40380b3450 0x7f4038071150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.894+0000 7f403e8f4640 1 -- 172.21.15.83:0/3095693875 >> 172.21.15.83:0/3095693875 conn(0x7f40380b1500 msgr2=0x7f40380b18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:09.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.895+0000 7f403e8f4640 1 -- 172.21.15.83:0/3095693875 shutdown_connections 2024-03-06T07:22:09.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.895+0000 7f403e8f4640 1 -- 172.21.15.83:0/3095693875 wait complete. 2024-03-06T07:22:09.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.897+0000 7f403e8f4640 1 Processor -- start 2024-03-06T07:22:09.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.897+0000 7f403e8f4640 1 -- start start 2024-03-06T07:22:09.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.898+0000 7f403e8f4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4038071690 0x7f4038211660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.898+0000 7f403e8f4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f40380b3450 0x7f403808a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.898+0000 7f403e8f4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40380adcb0 0x7f40380b25d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.898+0000 7f403d8f2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4038071690 0x7f4038211660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.899+0000 7f403d8f2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4038071690 0x7f4038211660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:59452/0 (socket says 172.21.15.83:59452) 2024-03-06T07:22:09.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.899+0000 7f403d0f1640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f40380b3450 0x7f403808a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.899+0000 7f403d0f1640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f40380b3450 0x7f403808a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:60180/0 (socket says 172.21.15.83:60180) 2024-03-06T07:22:09.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.899+0000 7f403d8f2640 1 -- 172.21.15.83:0/698695133 learned_addr learned my addr 172.21.15.83:0/698695133 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:09.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.899+0000 7f403e0f3640 1 --2- 172.21.15.83:0/698695133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40380adcb0 0x7f40380b25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.899+0000 7f403e8f4640 1 -- 172.21.15.83:0/698695133 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4038071ca0 con 0x7f40380adcb0 2024-03-06T07:22:09.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.899+0000 7f403e8f4640 1 -- 172.21.15.83:0/698695133 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4038071b20 con 0x7f40380b3450 2024-03-06T07:22:09.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.899+0000 7f403e8f4640 1 -- 172.21.15.83:0/698695133 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4038071e20 con 0x7f4038071690 2024-03-06T07:22:09.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.900+0000 7f403d8f2640 1 -- 172.21.15.83:0/698695133 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f40380b3450 msgr2=0x7f403808a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.900+0000 7f403d8f2640 1 --2- 172.21.15.83:0/698695133 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f40380b3450 0x7f403808a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.900+0000 7f403d8f2640 1 -- 172.21.15.83:0/698695133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40380adcb0 msgr2=0x7f40380b25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.900+0000 7f403d8f2640 1 --2- 172.21.15.83:0/698695133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40380adcb0 0x7f40380b25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.900+0000 7f403d8f2640 1 -- 172.21.15.83:0/698695133 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4038211cf0 con 0x7f4038071690 2024-03-06T07:22:09.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.901+0000 7f403d8f2640 1 --2- 172.21.15.83:0/698695133 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4038071690 0x7f4038211660 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f402800c480 tx=0x7f402800c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.901+0000 7f403e0f3640 1 --2- 172.21.15.83:0/698695133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40380adcb0 0x7f40380b25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:09.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.903+0000 7f402e7fc640 1 -- 172.21.15.83:0/698695133 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4028017070 con 0x7f4038071690 2024-03-06T07:22:09.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.903+0000 7f402e7fc640 1 -- 172.21.15.83:0/698695133 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4028005ce0 con 0x7f4038071690 2024-03-06T07:22:09.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.903+0000 7f402e7fc640 1 -- 172.21.15.83:0/698695133 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4028007290 con 0x7f4038071690 2024-03-06T07:22:09.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.903+0000 7f403e8f4640 1 -- 172.21.15.83:0/698695133 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4038211fd0 con 0x7f4038071690 2024-03-06T07:22:09.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.903+0000 7f403e8f4640 1 -- 172.21.15.83:0/698695133 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4038212590 con 0x7f4038071690 2024-03-06T07:22:09.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.903+0000 7f2c267fc640 1 -- 172.21.15.83:0/714377576 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2c20055030 con 0x7f2c280df220 2024-03-06T07:22:09.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.907+0000 7f4023fff640 1 -- 172.21.15.83:0/698695133 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4038068f80 con 0x7f4038071690 2024-03-06T07:22:09.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.907+0000 7f402e7fc640 1 -- 172.21.15.83:0/698695133 <== mon.2 v2:172.21.15.183:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f402801a070 con 0x7f4038071690 2024-03-06T07:22:09.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.908+0000 7f402e7fc640 1 --2- 172.21.15.83:0/698695133 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f401c06b5a0 0x7f401c06da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.908+0000 7f403d0f1640 1 --2- 172.21.15.83:0/698695133 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f401c06b5a0 0x7f401c06da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.909+0000 7f403d0f1640 1 --2- 172.21.15.83:0/698695133 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f401c06b5a0 0x7f401c06da60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f403806d7a0 tx=0x7f4034009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.909+0000 7f402e7fc640 1 -- 172.21.15.83:0/698695133 <== mon.2 v2:172.21.15.183:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f40280894e0 con 0x7f4038071690 2024-03-06T07:22:09.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.913+0000 7f402e7fc640 1 -- 172.21.15.83:0/698695133 <== mon.2 v2:172.21.15.183:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4028057d80 con 0x7f4038071690 2024-03-06T07:22:09.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.918+0000 7fa9f7e78640 1 Processor -- start 2024-03-06T07:22:09.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.918+0000 7fa9f7e78640 1 -- start start 2024-03-06T07:22:09.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.920+0000 7fa9f7e78640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa9e80e84a0 0x7fa9e8035b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.920+0000 7fa9f7e78640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fa9e80ebf30 0x7fa9e803b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.920+0000 7fa9f7e78640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e80e8c30 con 0x7fa9e8052950 2024-03-06T07:22:09.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.920+0000 7fa9f7e78640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9e80e8950 con 0x7fa9e80e84a0 2024-03-06T07:22:09.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.920+0000 7fa9f7e78640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9e80eb450 con 0x7fa9e80ebf30 2024-03-06T07:22:09.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.921+0000 7fa9f5c14640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fa9e8052950 0x7fa9e8052d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:42950/0 (socket says 172.21.15.83:42950) 2024-03-06T07:22:09.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.921+0000 7fa9f5c14640 1 -- 172.21.15.83:0/210403336 learned_addr learned my addr 172.21.15.83:0/210403336 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:09.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.929+0000 7fa9f6415640 1 --2- 172.21.15.83:0/210403336 >> v2:172.21.15.183:3300/0 conn(0x7fa9e80ebf30 0x7fa9e803b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.931+0000 7fa9f4c12640 1 -- 172.21.15.83:0/210403336 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1650280221 0 0) 0x7fa9e80e8c30 con 0x7fa9e8052950 2024-03-06T07:22:09.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.931+0000 7fa9f4c12640 1 -- 172.21.15.83:0/210403336 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9dc003520 con 0x7fa9e8052950 2024-03-06T07:22:09.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.931+0000 7fa9f5413640 1 --2- 172.21.15.83:0/210403336 >> v2:172.21.15.83:3300/0 conn(0x7fa9e80e84a0 0x7fa9e8035b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.935+0000 7fa9f6415640 1 -- 172.21.15.83:0/210403336 >> v2:172.21.15.83:3300/0 conn(0x7fa9e80e84a0 msgr2=0x7fa9e8035b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:09.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.935+0000 7fa9f6415640 1 --2- 172.21.15.83:0/210403336 >> v2:172.21.15.83:3300/0 conn(0x7fa9e80e84a0 0x7fa9e8035b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.935+0000 7fa9f6415640 1 -- 172.21.15.83:0/210403336 >> v1:172.21.15.83:6789/0 conn(0x7fa9e8052950 legacy=0x7fa9e8052d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.935+0000 7fa9f6415640 1 -- 172.21.15.83:0/210403336 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e80eb730 con 0x7fa9e80ebf30 2024-03-06T07:22:09.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.936+0000 7fa9f6415640 1 --2- 172.21.15.83:0/210403336 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa9e80ebf30 0x7fa9e803b290 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa9e803cd90 tx=0x7fa9ec02b650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9cda404a05fedfd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.943+0000 7fa9f4c12640 1 -- 172.21.15.83:0/210403336 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9ec037070 con 0x7fa9e80ebf30 2024-03-06T07:22:09.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.943+0000 7fa9f4c12640 1 -- 172.21.15.83:0/210403336 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9ec02be50 con 0x7fa9e80ebf30 2024-03-06T07:22:09.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.944+0000 7fa9f7e78640 1 -- 172.21.15.83:0/210403336 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa9e80ebf30 msgr2=0x7fa9e803b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.944+0000 7fa9f7e78640 1 --2- 172.21.15.83:0/210403336 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa9e80ebf30 0x7fa9e803b290 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa9e803cd90 tx=0x7fa9ec02b650 comp rx=0 tx=0).stop 2024-03-06T07:22:09.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.951+0000 7fa9f7e78640 1 -- 172.21.15.83:0/210403336 shutdown_connections 2024-03-06T07:22:09.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.951+0000 7fa9f7e78640 1 --2- 172.21.15.83:0/210403336 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa9e80ebf30 0x7fa9e803b290 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.951+0000 7fa9f7e78640 1 --2- 172.21.15.83:0/210403336 >> v2:172.21.15.83:3300/0 conn(0x7fa9e80e84a0 0x7fa9e8035b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.951+0000 7fa9f7e78640 1 -- 172.21.15.83:0/210403336 >> 172.21.15.83:0/210403336 conn(0x7fa9e80537e0 msgr2=0x7fa9e8053bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:09.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.961+0000 7fa9f7e78640 1 -- 172.21.15.83:0/210403336 shutdown_connections 2024-03-06T07:22:09.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.962+0000 7fa9f7e78640 1 -- 172.21.15.83:0/210403336 wait complete. 2024-03-06T07:22:09.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.963+0000 7fa9f7e78640 1 Processor -- start 2024-03-06T07:22:09.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.966+0000 7fa9f7e78640 1 -- start start 2024-03-06T07:22:09.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.967+0000 7fa9f7e78640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa9e8052950 0x7fa9e8045920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.967+0000 7fa9f7e78640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9e80e84a0 0x7fa9e80d7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.967+0000 7fa9f7e78640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa9e80ebf30 0x7fa9e80d77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.967+0000 7fa9f7e78640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e8036040 con 0x7fa9e80e84a0 2024-03-06T07:22:09.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.967+0000 7fa9f7e78640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e8039d40 con 0x7fa9e8052950 2024-03-06T07:22:09.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.967+0000 7fa9f7e78640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e80361c0 con 0x7fa9e80ebf30 2024-03-06T07:22:09.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.967+0000 7fa9f5413640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9e80e84a0 0x7fa9e80d7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.968+0000 7fa9f6415640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa9e80ebf30 0x7fa9e80d77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.968+0000 7fa9f5c14640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa9e8052950 0x7fa9e8045920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.968+0000 7fa9f5c14640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa9e8052950 0x7fa9e8045920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:60204/0 (socket says 172.21.15.83:60204) 2024-03-06T07:22:09.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.968+0000 7fa9f5c14640 1 -- 172.21.15.83:0/1430912704 learned_addr learned my addr 172.21.15.83:0/1430912704 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:09.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.969+0000 7fa9f5c14640 1 -- 172.21.15.83:0/1430912704 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa9e80ebf30 msgr2=0x7fa9e80d77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.969+0000 7fa9f5c14640 1 --2- 172.21.15.83:0/1430912704 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa9e80ebf30 0x7fa9e80d77d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.969+0000 7fa9f5c14640 1 -- 172.21.15.83:0/1430912704 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9e80e84a0 msgr2=0x7fa9e80d7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.969+0000 7fa9f5c14640 1 --2- 172.21.15.83:0/1430912704 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9e80e84a0 0x7fa9e80d7290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.969+0000 7fa9f5c14640 1 -- 172.21.15.83:0/1430912704 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e8047260 con 0x7fa9e8052950 2024-03-06T07:22:09.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.969+0000 7fa9f5413640 1 --2- 172.21.15.83:0/1430912704 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9e80e84a0 0x7fa9e80d7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:09.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.975+0000 7fa9f5c14640 1 --2- 172.21.15.83:0/1430912704 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa9e8052950 0x7fa9e8045920 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa9e000cac0 tx=0x7fa9e000cf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.975+0000 7fa9e67fc640 1 -- 172.21.15.83:0/1430912704 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e0018070 con 0x7fa9e8052950 2024-03-06T07:22:09.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.975+0000 7fa9e67fc640 1 -- 172.21.15.83:0/1430912704 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9e00075a0 con 0x7fa9e8052950 2024-03-06T07:22:09.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.976+0000 7fa9f7e78640 1 -- 172.21.15.83:0/1430912704 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9e804d700 con 0x7fa9e8052950 2024-03-06T07:22:09.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.976+0000 7fa9f7e78640 1 -- 172.21.15.83:0/1430912704 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9e80d5660 con 0x7fa9e8052950 2024-03-06T07:22:09.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.977+0000 7fa9e67fc640 1 -- 172.21.15.83:0/1430912704 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e0004a20 con 0x7fa9e8052950 2024-03-06T07:22:09.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.977+0000 7f5838657640 1 Processor -- start 2024-03-06T07:22:09.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.977+0000 7f5838657640 1 -- start start 2024-03-06T07:22:09.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.978+0000 7f5838657640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5830165190 0x7f5830071770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.978+0000 7f5838657640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f58300cf3f0 0x7f583006b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.978+0000 7f5838657640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f583016b3a0 con 0x7f583006afb0 2024-03-06T07:22:09.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.978+0000 7f5838657640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f583006b950 con 0x7f5830165190 2024-03-06T07:22:09.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.978+0000 7f5838657640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f583016bfb0 con 0x7f58300cf3f0 2024-03-06T07:22:09.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.979+0000 7f5835bf2640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f583006afb0 0x7f58300cece0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:37952/0 (socket says 172.21.15.83:37952) 2024-03-06T07:22:09.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.979+0000 7f5835bf2640 1 -- 172.21.15.83:0/372259551 learned_addr learned my addr 172.21.15.83:0/372259551 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:09.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.980+0000 7fa9e67fc640 1 -- 172.21.15.83:0/1430912704 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa9e001b070 con 0x7fa9e8052950 2024-03-06T07:22:09.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.980+0000 7f5836bf4640 1 --2- 172.21.15.83:0/372259551 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f58300cf3f0 0x7f583006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.980+0000 7f58363f3640 1 --2- 172.21.15.83:0/372259551 >> v2:172.21.15.83:3300/0 conn(0x7f5830165190 0x7f5830071770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.981+0000 7f58363f3640 1 -- 172.21.15.83:0/372259551 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f58300cf3f0 msgr2=0x7f583006b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:09.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.981+0000 7f58363f3640 1 --2- 172.21.15.83:0/372259551 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f58300cf3f0 0x7f583006b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.981+0000 7f58363f3640 1 -- 172.21.15.83:0/372259551 >> v1:172.21.15.183:6789/0 conn(0x7f583006afb0 legacy=0x7f58300cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.981+0000 7f58363f3640 1 -- 172.21.15.83:0/372259551 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f583016b680 con 0x7f5830165190 2024-03-06T07:22:09.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.982+0000 7f58363f3640 1 --2- 172.21.15.83:0/372259551 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5830165190 0x7f5830071770 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f582c0097a0 tx=0x7f582c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92f2501bf90bde5e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.982+0000 7f58353f1640 1 -- 172.21.15.83:0/372259551 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582c035020 con 0x7f5830165190 2024-03-06T07:22:09.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.983+0000 7fa9e67fc640 1 --2- 172.21.15.83:0/1430912704 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa9d406b5a0 0x7fa9d406da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.983+0000 7fa9e67fc640 1 -- 172.21.15.83:0/1430912704 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa9e0089940 con 0x7fa9e8052950 2024-03-06T07:22:09.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.983+0000 7f58353f1640 1 -- 172.21.15.83:0/372259551 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f582c031640 con 0x7f5830165190 2024-03-06T07:22:09.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.983+0000 7f58353f1640 1 -- 172.21.15.83:0/372259551 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582c02d040 con 0x7f5830165190 2024-03-06T07:22:09.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.984+0000 7f5838657640 1 -- 172.21.15.83:0/372259551 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5830165190 msgr2=0x7f5830071770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.984+0000 7f5838657640 1 --2- 172.21.15.83:0/372259551 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5830165190 0x7f5830071770 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f582c0097a0 tx=0x7f582c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.985+0000 7fa9f5413640 1 --2- 172.21.15.83:0/1430912704 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa9d406b5a0 0x7fa9d406da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.986+0000 7f5838657640 1 -- 172.21.15.83:0/372259551 shutdown_connections 2024-03-06T07:22:09.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.986+0000 7f5838657640 1 --2- 172.21.15.83:0/372259551 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f58300cf3f0 0x7f583006b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.986+0000 7f5838657640 1 --2- 172.21.15.83:0/372259551 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5830165190 0x7f5830071770 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.986+0000 7f5838657640 1 -- 172.21.15.83:0/372259551 >> 172.21.15.83:0/372259551 conn(0x7f58300b1830 msgr2=0x7f58300b1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:09.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.986+0000 7fa9f5413640 1 --2- 172.21.15.83:0/1430912704 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa9d406b5a0 0x7fa9d406da60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa9f0075030 tx=0x7fa9f008d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.986+0000 7f5838657640 1 -- 172.21.15.83:0/372259551 shutdown_connections 2024-03-06T07:22:09.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.987+0000 7fa9f7e78640 1 -- 172.21.15.83:0/1430912704 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9e8005810 con 0x7fa9e8052950 2024-03-06T07:22:09.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.988+0000 7f5838657640 1 -- 172.21.15.83:0/372259551 wait complete. 2024-03-06T07:22:09.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.989+0000 7f5838657640 1 Processor -- start 2024-03-06T07:22:09.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.990+0000 7f5838657640 1 -- start start 2024-03-06T07:22:09.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.991+0000 7f5838657640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f58300cf3f0 0x7f5830214b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.991+0000 7f5838657640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5830165190 0x7f583016d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.991+0000 7f5838657640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f583016d820 0x7f5830166820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.991+0000 7f5838657640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5830165570 con 0x7f5830165190 2024-03-06T07:22:09.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.991+0000 7f5838657640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583016bc40 con 0x7f583016d820 2024-03-06T07:22:09.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.991+0000 7f5838657640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f583016e880 con 0x7f58300cf3f0 2024-03-06T07:22:09.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.991+0000 7f5835bf2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5830165190 0x7f583016d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.992+0000 7f5835bf2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5830165190 0x7f583016d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49732/0 (socket says 172.21.15.83:49732) 2024-03-06T07:22:09.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.992+0000 7f5835bf2640 1 -- 172.21.15.83:0/3044015526 learned_addr learned my addr 172.21.15.83:0/3044015526 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:09.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.994+0000 7f5835bf2640 1 -- 172.21.15.83:0/3044015526 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f58300cf3f0 msgr2=0x7f5830214b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-06T07:22:09.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.994+0000 7f5835bf2640 1 --2- 172.21.15.83:0/3044015526 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f58300cf3f0 0x7f5830214b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.994+0000 7f5835bf2640 1 -- 172.21.15.83:0/3044015526 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f583016d820 msgr2=0x7f5830166820 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-06T07:22:09.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.994+0000 7f5835bf2640 1 --2- 172.21.15.83:0/3044015526 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f583016d820 0x7f5830166820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.994+0000 7f5835bf2640 1 -- 172.21.15.83:0/3044015526 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5830167530 con 0x7f5830165190 2024-03-06T07:22:09.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.995+0000 7f5835bf2640 1 --2- 172.21.15.83:0/3044015526 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5830165190 0x7f583016d2e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f582000b5f0 tx=0x7f582000bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.996+0000 7fa9e67fc640 1 -- 172.21.15.83:0/1430912704 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa9e8047260 con 0x7fa9e8052950 2024-03-06T07:22:09.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.997+0000 7f5826ffd640 1 -- 172.21.15.83:0/3044015526 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582000c9c0 con 0x7f5830165190 2024-03-06T07:22:09.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.997+0000 7f5838657640 1 -- 172.21.15.83:0/3044015526 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5830167810 con 0x7f5830165190 2024-03-06T07:22:09.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.997+0000 7f5826ffd640 1 -- 172.21.15.83:0/3044015526 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5820010070 con 0x7f5830165190 2024-03-06T07:22:09.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.997+0000 7f5838657640 1 -- 172.21.15.83:0/3044015526 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5830167dd0 con 0x7f5830165190 2024-03-06T07:22:09.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:09.997+0000 7f5826ffd640 1 -- 172.21.15.83:0/3044015526 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582000cdc0 con 0x7f5830165190 2024-03-06T07:22:10.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.002+0000 7f5838657640 1 -- 172.21.15.83:0/3044015526 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5818005290 con 0x7f5830165190 2024-03-06T07:22:10.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.005+0000 7f5826ffd640 1 -- 172.21.15.83:0/3044015526 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5820006070 con 0x7f5830165190 2024-03-06T07:22:10.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.005+0000 7f5826ffd640 1 --2- 172.21.15.83:0/3044015526 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f581406b4d0 0x7f581406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.005+0000 7f5826ffd640 1 -- 172.21.15.83:0/3044015526 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f582008a2f0 con 0x7f5830165190 2024-03-06T07:22:10.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.013+0000 7f58363f3640 1 --2- 172.21.15.83:0/3044015526 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f581406b4d0 0x7f581406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.013+0000 7f58363f3640 1 --2- 172.21.15.83:0/3044015526 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f581406b4d0 0x7f581406d990 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f582c036000 tx=0x7f582c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.014+0000 7f5826ffd640 1 -- 172.21.15.83:0/3044015526 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5820058c90 con 0x7f5830165190 2024-03-06T07:22:10.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.147+0000 7f209aa71640 1 -- 172.21.15.83:0/4267029696 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f20940703e0 con 0x7f209406de20 2024-03-06T07:22:10.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.148+0000 7f20917fa640 1 -- 172.21.15.83:0/4267029696 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2088058210 con 0x7f209406de20 2024-03-06T07:22:10.152 INFO:teuthology.orchestra.run.smithi083.stdout:55834574856 2024-03-06T07:22:10.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.151+0000 7f209aa71640 1 -- 172.21.15.83:0/4267029696 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f207806b5a0 msgr2=0x7f207806da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.151+0000 7f209aa71640 1 --2- 172.21.15.83:0/4267029696 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f207806b5a0 0x7f207806da60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2094158c90 tx=0x7f2080007d30 comp rx=0 tx=0).stop 2024-03-06T07:22:10.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.151+0000 7f209aa71640 1 -- 172.21.15.83:0/4267029696 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f209406de20 msgr2=0x7f209414af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.151+0000 7f209aa71640 1 --2- 172.21.15.83:0/4267029696 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f209406de20 0x7f209414af50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f208800c430 tx=0x7f208800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.156+0000 7f209aa71640 1 -- 172.21.15.83:0/4267029696 shutdown_connections 2024-03-06T07:22:10.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.156+0000 7f209aa71640 1 --2- 172.21.15.83:0/4267029696 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f207806b5a0 0x7f207806da60 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.156+0000 7f209aa71640 1 --2- 172.21.15.83:0/4267029696 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f20941627a0 0x7f2094160d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.156+0000 7f209aa71640 1 --2- 172.21.15.83:0/4267029696 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f209406fe70 0x7f209414a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.156+0000 7f209aa71640 1 --2- 172.21.15.83:0/4267029696 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f209406de20 0x7f209414af50 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.156+0000 7f209aa71640 1 -- 172.21.15.83:0/4267029696 >> 172.21.15.83:0/4267029696 conn(0x7f20940d0a10 msgr2=0x7f20941666b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.156+0000 7f209aa71640 1 -- 172.21.15.83:0/4267029696 shutdown_connections 2024-03-06T07:22:10.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.156+0000 7f209aa71640 1 -- 172.21.15.83:0/4267029696 wait complete. 2024-03-06T07:22:10.174 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.3 2024-03-06T07:22:10.174 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:10.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.226+0000 7fcad5e1c640 1 -- 172.21.15.83:0/1386484533 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fcab0005850 con 0x7fcad006d3c0 2024-03-06T07:22:10.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.228+0000 7fcab7fff640 1 -- 172.21.15.83:0/1386484533 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcac405bd30 con 0x7fcad006d3c0 2024-03-06T07:22:10.229 INFO:teuthology.orchestra.run.smithi083.stdout:55834574857 2024-03-06T07:22:10.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.230+0000 7f5f15ffb640 1 -- 172.21.15.83:0/1663044662 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f5f280053c0 con 0x7f5f280f1490 2024-03-06T07:22:10.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.231+0000 7fcad5e1c640 1 -- 172.21.15.83:0/1386484533 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcabc06b550 msgr2=0x7fcabc06da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.231+0000 7fcad5e1c640 1 --2- 172.21.15.83:0/1386484533 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcabc06b550 0x7fcabc06da10 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fcad015c2d0 tx=0x7fcac800b040 comp rx=0 tx=0).stop 2024-03-06T07:22:10.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.231+0000 7fcad5e1c640 1 -- 172.21.15.83:0/1386484533 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcad006d3c0 msgr2=0x7fcad0150920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.231+0000 7f5f17fff640 1 -- 172.21.15.83:0/1663044662 <== mon.2 v2:172.21.15.183:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5f300684f0 con 0x7f5f280f1490 2024-03-06T07:22:10.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.231+0000 7fcad5e1c640 1 --2- 172.21.15.83:0/1386484533 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcad006d3c0 0x7fcad0150920 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcac400c4a0 tx=0x7fcac400c960 comp rx=0 tx=0).stop 2024-03-06T07:22:10.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.231+0000 7fcad5e1c640 1 -- 172.21.15.83:0/1386484533 shutdown_connections 2024-03-06T07:22:10.234 INFO:teuthology.orchestra.run.smithi083.stdout:55834574855 2024-03-06T07:22:10.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.231+0000 7fcad5e1c640 1 --2- 172.21.15.83:0/1386484533 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcabc06b550 0x7fcabc06da10 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.231+0000 7fcad5e1c640 1 --2- 172.21.15.83:0/1386484533 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcad01614a0 0x7fcad006ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.231+0000 7fcad5e1c640 1 --2- 172.21.15.83:0/1386484533 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcad00b1850 0x7fcad015ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.231+0000 7fcad5e1c640 1 --2- 172.21.15.83:0/1386484533 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcad006d3c0 0x7fcad0150920 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.231+0000 7fcad5e1c640 1 -- 172.21.15.83:0/1386484533 >> 172.21.15.83:0/1386484533 conn(0x7fcad009a9f0 msgr2=0x7fcad00b0170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.231+0000 7fcad5e1c640 1 -- 172.21.15.83:0/1386484533 shutdown_connections 2024-03-06T07:22:10.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.231+0000 7fcad5e1c640 1 -- 172.21.15.83:0/1386484533 wait complete. 2024-03-06T07:22:10.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.234+0000 7f5f15ffb640 1 -- 172.21.15.83:0/1663044662 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5f1c06b5a0 msgr2=0x7f5f1c06da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.234+0000 7f5f15ffb640 1 --2- 172.21.15.83:0/1663044662 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5f1c06b5a0 0x7f5f1c06da60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5f20036000 tx=0x7f5f20030000 comp rx=0 tx=0).stop 2024-03-06T07:22:10.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.234+0000 7f5f15ffb640 1 -- 172.21.15.83:0/1663044662 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f280f1490 msgr2=0x7f5f280fceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.234+0000 7f5f15ffb640 1 --2- 172.21.15.83:0/1663044662 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f280f1490 0x7f5f280fceb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5f3002d580 tx=0x7f5f300683c0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.234+0000 7f5f15ffb640 1 -- 172.21.15.83:0/1663044662 shutdown_connections 2024-03-06T07:22:10.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.234+0000 7f5f15ffb640 1 --2- 172.21.15.83:0/1663044662 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5f1c06b5a0 0x7f5f1c06da60 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.234+0000 7f5f15ffb640 1 --2- 172.21.15.83:0/1663044662 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f2819a6d0 0x7f5f2804d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.234+0000 7f5f15ffb640 1 --2- 172.21.15.83:0/1663044662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f280fd3f0 0x7f5f2819a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.234+0000 7f5f15ffb640 1 --2- 172.21.15.83:0/1663044662 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f280f1490 0x7f5f280fceb0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.234+0000 7f5f15ffb640 1 -- 172.21.15.83:0/1663044662 >> 172.21.15.83:0/1663044662 conn(0x7f5f28009fe0 msgr2=0x7f5f2804f570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.239+0000 7f5f15ffb640 1 -- 172.21.15.83:0/1663044662 shutdown_connections 2024-03-06T07:22:10.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.239+0000 7f5f15ffb640 1 -- 172.21.15.83:0/1663044662 wait complete. 2024-03-06T07:22:10.250 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.1 2024-03-06T07:22:10.250 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:10.256 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.6 2024-03-06T07:22:10.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.254+0000 7fc72eed4640 1 -- 172.21.15.83:0/34234774 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fc708005580 con 0x7fc7280c60f0 2024-03-06T07:22:10.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.255+0000 7fc7257fa640 1 -- 172.21.15.83:0/34234774 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc71c058420 con 0x7fc7280c60f0 2024-03-06T07:22:10.257 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:10.257 INFO:teuthology.orchestra.run.smithi083.stdout:55834574856 2024-03-06T07:22:10.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.257+0000 7fc72eed4640 1 -- 172.21.15.83:0/34234774 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc71406b5a0 msgr2=0x7fc71406da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.257+0000 7fc72eed4640 1 --2- 172.21.15.83:0/34234774 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc71406b5a0 0x7fc71406da60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc72000a730 tx=0x7fc720009000 comp rx=0 tx=0).stop 2024-03-06T07:22:10.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.257+0000 7fc72eed4640 1 -- 172.21.15.83:0/34234774 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7280c60f0 msgr2=0x7fc7280c6570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.257+0000 7fc72eed4640 1 --2- 172.21.15.83:0/34234774 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7280c60f0 0x7fc7280c6570 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc71c001eb0 tx=0x7fc71c00e9e0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.258+0000 7fc72eed4640 1 -- 172.21.15.83:0/34234774 shutdown_connections 2024-03-06T07:22:10.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.258+0000 7fc72eed4640 1 --2- 172.21.15.83:0/34234774 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc71406b5a0 0x7fc71406da60 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.258+0000 7fc72eed4640 1 --2- 172.21.15.83:0/34234774 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7280c60f0 0x7fc7280c6570 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.258+0000 7fc72eed4640 1 --2- 172.21.15.83:0/34234774 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc728151f20 0x7fc7280c5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.258+0000 7fc72eed4640 1 --2- 172.21.15.83:0/34234774 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc728151b40 0x7fc7280cd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.258+0000 7fc72eed4640 1 -- 172.21.15.83:0/34234774 >> 172.21.15.83:0/34234774 conn(0x7fc728072520 msgr2=0x7fc72806ec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.258+0000 7fc72eed4640 1 -- 172.21.15.83:0/34234774 shutdown_connections 2024-03-06T07:22:10.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.258+0000 7fc72eed4640 1 -- 172.21.15.83:0/34234774 wait complete. 2024-03-06T07:22:10.271 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.2 2024-03-06T07:22:10.271 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:10.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.289+0000 7f4023fff640 1 -- 172.21.15.83:0/698695133 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f40380cc1f0 con 0x7f4038071690 2024-03-06T07:22:10.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.290+0000 7f402e7fc640 1 -- 172.21.15.83:0/698695133 <== mon.2 v2:172.21.15.183:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f402805e0d0 con 0x7f4038071690 2024-03-06T07:22:10.292 INFO:teuthology.orchestra.run.smithi083.stdout:55834574857 2024-03-06T07:22:10.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.291+0000 7f4023fff640 1 -- 172.21.15.83:0/698695133 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f401c06b5a0 msgr2=0x7f401c06da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.291+0000 7f4023fff640 1 --2- 172.21.15.83:0/698695133 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f401c06b5a0 0x7f401c06da60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f403806d7a0 tx=0x7f4034009190 comp rx=0 tx=0).stop 2024-03-06T07:22:10.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.291+0000 7f4023fff640 1 -- 172.21.15.83:0/698695133 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4038071690 msgr2=0x7f4038211660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.291+0000 7f4023fff640 1 --2- 172.21.15.83:0/698695133 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4038071690 0x7f4038211660 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f402800c480 tx=0x7f402800c940 comp rx=0 tx=0).stop 2024-03-06T07:22:10.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.292+0000 7f4023fff640 1 -- 172.21.15.83:0/698695133 shutdown_connections 2024-03-06T07:22:10.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.292+0000 7f4023fff640 1 --2- 172.21.15.83:0/698695133 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f401c06b5a0 0x7f401c06da60 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.292+0000 7f4023fff640 1 --2- 172.21.15.83:0/698695133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40380adcb0 0x7f40380b25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.292+0000 7f4023fff640 1 --2- 172.21.15.83:0/698695133 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f40380b3450 0x7f403808a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.292+0000 7f4023fff640 1 --2- 172.21.15.83:0/698695133 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4038071690 0x7f4038211660 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.292+0000 7f4023fff640 1 -- 172.21.15.83:0/698695133 >> 172.21.15.83:0/698695133 conn(0x7f40380b1500 msgr2=0x7f40380b0780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.292+0000 7f4023fff640 1 -- 172.21.15.83:0/698695133 shutdown_connections 2024-03-06T07:22:10.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.292+0000 7f4023fff640 1 -- 172.21.15.83:0/698695133 wait complete. 2024-03-06T07:22:10.305 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.4 2024-03-06T07:22:10.305 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:10.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.313+0000 7f2c36b67640 1 -- 172.21.15.83:0/714377576 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f2c28005390 con 0x7f2c280df220 2024-03-06T07:22:10.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.313+0000 7f2c267fc640 1 -- 172.21.15.83:0/714377576 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2c2804cd80 con 0x7f2c280df220 2024-03-06T07:22:10.315 INFO:teuthology.orchestra.run.smithi083.stdout:55834574856 2024-03-06T07:22:10.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.315+0000 7f2c36b67640 1 -- 172.21.15.83:0/714377576 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2c1406b4d0 msgr2=0x7f2c1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.315+0000 7f2c36b67640 1 --2- 172.21.15.83:0/714377576 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2c1406b4d0 0x7f2c1406d990 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2c3006a8c0 tx=0x7f2c3008d040 comp rx=0 tx=0).stop 2024-03-06T07:22:10.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.315+0000 7f2c36b67640 1 -- 172.21.15.83:0/714377576 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c280df220 msgr2=0x7f2c280e0970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.315+0000 7f2c36b67640 1 --2- 172.21.15.83:0/714377576 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c280df220 0x7f2c280e0970 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2c2000dcb0 tx=0x7f2c200046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.315+0000 7f2c36b67640 1 -- 172.21.15.83:0/714377576 shutdown_connections 2024-03-06T07:22:10.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.315+0000 7f2c36b67640 1 --2- 172.21.15.83:0/714377576 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2c1406b4d0 0x7f2c1406d990 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.315+0000 7f2c36b67640 1 --2- 172.21.15.83:0/714377576 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c280df220 0x7f2c280e0970 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.315+0000 7f2c36b67640 1 --2- 172.21.15.83:0/714377576 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2c28053c70 0x7f2c280d0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.315+0000 7f2c36b67640 1 --2- 172.21.15.83:0/714377576 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c28033ef0 0x7f2c280485f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.315+0000 7f2c36b67640 1 -- 172.21.15.83:0/714377576 >> 172.21.15.83:0/714377576 conn(0x7f2c28009fb0 msgr2=0x7f2c280dcbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.316+0000 7f2c36b67640 1 -- 172.21.15.83:0/714377576 shutdown_connections 2024-03-06T07:22:10.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.316+0000 7f2c36b67640 1 -- 172.21.15.83:0/714377576 wait complete. 2024-03-06T07:22:10.328 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.5 2024-03-06T07:22:10.328 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:10.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.331+0000 7fa9f7e78640 1 -- 172.21.15.83:0/1430912704 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fa9e80365a0 con 0x7fa9e8052950 2024-03-06T07:22:10.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.332+0000 7fa9e67fc640 1 -- 172.21.15.83:0/1430912704 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa9e00580c0 con 0x7fa9e8052950 2024-03-06T07:22:10.333 INFO:teuthology.orchestra.run.smithi083.stdout:55834574856 2024-03-06T07:22:10.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.333+0000 7fa9f7e78640 1 -- 172.21.15.83:0/1430912704 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa9d406b5a0 msgr2=0x7fa9d406da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.333+0000 7fa9f7e78640 1 --2- 172.21.15.83:0/1430912704 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa9d406b5a0 0x7fa9d406da60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa9f0075030 tx=0x7fa9f008d040 comp rx=0 tx=0).stop 2024-03-06T07:22:10.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.333+0000 7fa9f7e78640 1 -- 172.21.15.83:0/1430912704 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa9e8052950 msgr2=0x7fa9e8045920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.333+0000 7fa9f7e78640 1 --2- 172.21.15.83:0/1430912704 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa9e8052950 0x7fa9e8045920 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa9e000cac0 tx=0x7fa9e000cf80 comp rx=0 tx=0).stop 2024-03-06T07:22:10.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.334+0000 7fa9f7e78640 1 -- 172.21.15.83:0/1430912704 shutdown_connections 2024-03-06T07:22:10.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.334+0000 7fa9f7e78640 1 --2- 172.21.15.83:0/1430912704 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa9d406b5a0 0x7fa9d406da60 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.334+0000 7fa9f7e78640 1 --2- 172.21.15.83:0/1430912704 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa9e80ebf30 0x7fa9e80d77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.334+0000 7fa9f7e78640 1 --2- 172.21.15.83:0/1430912704 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9e80e84a0 0x7fa9e80d7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.334+0000 7fa9f7e78640 1 --2- 172.21.15.83:0/1430912704 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa9e8052950 0x7fa9e8045920 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.334+0000 7fa9f7e78640 1 -- 172.21.15.83:0/1430912704 >> 172.21.15.83:0/1430912704 conn(0x7fa9e80537e0 msgr2=0x7fa9e80e5a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.334+0000 7fa9f7e78640 1 -- 172.21.15.83:0/1430912704 shutdown_connections 2024-03-06T07:22:10.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.334+0000 7fa9f7e78640 1 -- 172.21.15.83:0/1430912704 wait complete. 2024-03-06T07:22:10.346 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.0 2024-03-06T07:22:10.347 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:10.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.402+0000 7f5838657640 1 -- 172.21.15.83:0/3044015526 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f5818005850 con 0x7f5830165190 2024-03-06T07:22:10.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.402+0000 7f5826ffd640 1 -- 172.21.15.83:0/3044015526 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f582005cd30 con 0x7f5830165190 2024-03-06T07:22:10.404 INFO:teuthology.orchestra.run.smithi083.stdout:55834574857 2024-03-06T07:22:10.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.404+0000 7f5838657640 1 -- 172.21.15.83:0/3044015526 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f581406b4d0 msgr2=0x7f581406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.404+0000 7f5838657640 1 --2- 172.21.15.83:0/3044015526 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f581406b4d0 0x7f581406d990 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f582c036000 tx=0x7f582c008030 comp rx=0 tx=0).stop 2024-03-06T07:22:10.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.404+0000 7f5838657640 1 -- 172.21.15.83:0/3044015526 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5830165190 msgr2=0x7f583016d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.404+0000 7f5838657640 1 --2- 172.21.15.83:0/3044015526 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5830165190 0x7f583016d2e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f582000b5f0 tx=0x7f582000bab0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.404+0000 7f5838657640 1 -- 172.21.15.83:0/3044015526 shutdown_connections 2024-03-06T07:22:10.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.404+0000 7f5838657640 1 --2- 172.21.15.83:0/3044015526 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f581406b4d0 0x7f581406d990 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.404+0000 7f5838657640 1 --2- 172.21.15.83:0/3044015526 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f583016d820 0x7f5830166820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.404+0000 7f5838657640 1 --2- 172.21.15.83:0/3044015526 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5830165190 0x7f583016d2e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.404+0000 7f5838657640 1 --2- 172.21.15.83:0/3044015526 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f58300cf3f0 0x7f5830214b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.404+0000 7f5838657640 1 -- 172.21.15.83:0/3044015526 >> 172.21.15.83:0/3044015526 conn(0x7f58300b1830 msgr2=0x7f58300cebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.404+0000 7f5838657640 1 -- 172.21.15.83:0/3044015526 shutdown_connections 2024-03-06T07:22:10.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.404+0000 7f5838657640 1 -- 172.21.15.83:0/3044015526 wait complete. 2024-03-06T07:22:10.416 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.7 2024-03-06T07:22:10.417 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:10.417 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-06T07:22:10.417 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:10.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.516+0000 7f7fc102a640 1 Processor -- start 2024-03-06T07:22:10.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.516+0000 7f7fc102a640 1 -- start start 2024-03-06T07:22:10.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.517+0000 7f7fc102a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7fbc0d2920 0x7f7fbc0d47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.517+0000 7f7fc102a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f7fbc0d0650 0x7f7fbc0d0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.517+0000 7f7fc102a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fbc15e470 con 0x7f7fbc15e090 2024-03-06T07:22:10.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.517+0000 7f7fc102a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fbc0d0fe0 con 0x7f7fbc0d2920 2024-03-06T07:22:10.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.517+0000 7f7fc102a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fbc0d18f0 con 0x7f7fbc0d0650 2024-03-06T07:22:10.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.517+0000 7f7fba59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7fbc0d2920 0x7f7fbc0d47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.517+0000 7f7fba59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7fbc0d2920 0x7f7fbc0d47a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49736/0 (socket says 172.21.15.83:49736) 2024-03-06T07:22:10.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.517+0000 7f7fba59c640 1 -- 172.21.15.83:0/2903340866 learned_addr learned my addr 172.21.15.83:0/2903340866 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:10.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.517+0000 7f7fbb59e640 1 --2- 172.21.15.83:0/2903340866 >> v2:172.21.15.183:3300/0 conn(0x7f7fbc0d0650 0x7f7fbc0d0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.517+0000 7f7fba59c640 1 -- 172.21.15.83:0/2903340866 >> v2:172.21.15.183:3300/0 conn(0x7f7fbc0d0650 msgr2=0x7f7fbc0d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:10.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.517+0000 7f7fba59c640 1 --2- 172.21.15.83:0/2903340866 >> v2:172.21.15.183:3300/0 conn(0x7f7fbc0d0650 0x7f7fbc0d0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.517+0000 7f7fba59c640 1 -- 172.21.15.83:0/2903340866 >> v1:172.21.15.183:6789/0 conn(0x7f7fbc15e090 legacy=0x7f7fbc0b35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:10.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.517+0000 7f7fba59c640 1 -- 172.21.15.83:0/2903340866 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fbc0b3cf0 con 0x7f7fbc0d2920 2024-03-06T07:22:10.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.518+0000 7f7fba59c640 1 --2- 172.21.15.83:0/2903340866 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fbc0d2920 0x7f7fbc0d47a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc08c550 tx=0x7f7fac02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=937076b82b44dbf7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.518+0000 7f7fb9d9b640 1 -- 172.21.15.83:0/2903340866 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fac035020 con 0x7f7fbc0d2920 2024-03-06T07:22:10.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.518+0000 7f7fb9d9b640 1 -- 172.21.15.83:0/2903340866 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fac031870 con 0x7f7fbc0d2920 2024-03-06T07:22:10.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.518+0000 7f7fb9d9b640 1 -- 172.21.15.83:0/2903340866 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fac031c40 con 0x7f7fbc0d2920 2024-03-06T07:22:10.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.518+0000 7f7fc102a640 1 -- 172.21.15.83:0/2903340866 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fbc0d2920 msgr2=0x7f7fbc0d47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.519+0000 7f7fc102a640 1 --2- 172.21.15.83:0/2903340866 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fbc0d2920 0x7f7fbc0d47a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc08c550 tx=0x7f7fac02b910 comp rx=0 tx=0).stop 2024-03-06T07:22:10.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.519+0000 7f7fc102a640 1 -- 172.21.15.83:0/2903340866 shutdown_connections 2024-03-06T07:22:10.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.519+0000 7f7fc102a640 1 --2- 172.21.15.83:0/2903340866 >> v2:172.21.15.183:3300/0 conn(0x7f7fbc0d0650 0x7f7fbc0d0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.519+0000 7f7fc102a640 1 --2- 172.21.15.83:0/2903340866 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fbc0d2920 0x7f7fbc0d47a0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.519+0000 7f7fc102a640 1 -- 172.21.15.83:0/2903340866 >> 172.21.15.83:0/2903340866 conn(0x7f7fbc0d3c80 msgr2=0x7f7fbc0d4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.519+0000 7f7fc102a640 1 -- 172.21.15.83:0/2903340866 shutdown_connections 2024-03-06T07:22:10.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.519+0000 7f7fc102a640 1 -- 172.21.15.83:0/2903340866 wait complete. 2024-03-06T07:22:10.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.520+0000 7f7fc102a640 1 Processor -- start 2024-03-06T07:22:10.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.520+0000 7f7fc102a640 1 -- start start 2024-03-06T07:22:10.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.521+0000 7f7fc102a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7fbc0d0650 0x7f7fbc1582f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.521+0000 7f7fc102a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fbc0d2920 0x7f7fbc1f40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.521+0000 7f7fc102a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7fbc15e090 0x7f7fbc1f7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.521+0000 7f7fc102a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fbc0cee30 con 0x7f7fbc0d2920 2024-03-06T07:22:10.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.521+0000 7f7fc102a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fbc0ceb50 con 0x7f7fbc15e090 2024-03-06T07:22:10.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.521+0000 7f7fc102a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fbc0b2df0 con 0x7f7fbc0d0650 2024-03-06T07:22:10.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.521+0000 7f7fba59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fbc0d2920 0x7f7fbc1f40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.521+0000 7f7fba59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fbc0d2920 0x7f7fbc1f40a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49752/0 (socket says 172.21.15.83:49752) 2024-03-06T07:22:10.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.521+0000 7f7fba59c640 1 -- 172.21.15.83:0/347585172 learned_addr learned my addr 172.21.15.83:0/347585172 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:10.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.521+0000 7f7fba59c640 1 -- 172.21.15.83:0/347585172 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7fbc0d0650 msgr2=0x7f7fbc1582f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.521+0000 7f7fba59c640 1 --2- 172.21.15.83:0/347585172 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7fbc0d0650 0x7f7fbc1582f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.521+0000 7f7fba59c640 1 -- 172.21.15.83:0/347585172 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7fbc15e090 msgr2=0x7f7fbc1f7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.521+0000 7f7fba59c640 1 --2- 172.21.15.83:0/347585172 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7fbc15e090 0x7f7fbc1f7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.521+0000 7f7fba59c640 1 -- 172.21.15.83:0/347585172 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fbc0b8b20 con 0x7f7fbc0d2920 2024-03-06T07:22:10.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.521+0000 7f7fba59c640 1 --2- 172.21.15.83:0/347585172 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fbc0d2920 0x7f7fbc1f40a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7fac02b950 tx=0x7f7fac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.521+0000 7f7fb37fe640 1 -- 172.21.15.83:0/347585172 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fac040070 con 0x7f7fbc0d2920 2024-03-06T07:22:10.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.521+0000 7f7fb37fe640 1 -- 172.21.15.83:0/347585172 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fac030d00 con 0x7f7fbc0d2920 2024-03-06T07:22:10.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.521+0000 7f7fc102a640 1 -- 172.21.15.83:0/347585172 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fbc1f45e0 con 0x7f7fbc0d2920 2024-03-06T07:22:10.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.521+0000 7f7fb37fe640 1 -- 172.21.15.83:0/347585172 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fac035040 con 0x7f7fbc0d2920 2024-03-06T07:22:10.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.521+0000 7f7fc102a640 1 -- 172.21.15.83:0/347585172 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fbc1f4b40 con 0x7f7fbc0d2920 2024-03-06T07:22:10.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.522+0000 7f7fc102a640 1 -- 172.21.15.83:0/347585172 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fbc04f330 con 0x7f7fbc0d2920 2024-03-06T07:22:10.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.523+0000 7f7fb37fe640 1 -- 172.21.15.83:0/347585172 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7fac04b060 con 0x7f7fbc0d2920 2024-03-06T07:22:10.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.523+0000 7f7fb37fe640 1 --2- 172.21.15.83:0/347585172 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7f9c06b4d0 0x7f7f9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.523+0000 7f7fbad9d640 1 --2- 172.21.15.83:0/347585172 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7f9c06b4d0 0x7f7f9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.524+0000 7f7fb37fe640 1 -- 172.21.15.83:0/347585172 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7fac0abf40 con 0x7f7fbc0d2920 2024-03-06T07:22:10.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.524+0000 7f7fbad9d640 1 --2- 172.21.15.83:0/347585172 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7f9c06b4d0 0x7f7f9c06d990 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7fa4003cc0 tx=0x7f7fa40079b0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.528+0000 7f7fb37fe640 1 -- 172.21.15.83:0/347585172 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7fbc1f45e0 con 0x7f7fbc0d2920 2024-03-06T07:22:10.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.703+0000 7f7fc102a640 1 -- 172.21.15.83:0/347585172 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7fbc0d0400 con 0x7f7f9c06b4d0 2024-03-06T07:22:10.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.713+0000 7f7fb37fe640 1 -- 172.21.15.83:0/347585172 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251837 (secure 0 0 0) 0x7f7fbc0d0400 con 0x7f7f9c06b4d0 2024-03-06T07:22:10.717 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:22:10.719 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-06T07:22:10.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.716+0000 7f7fc102a640 1 -- 172.21.15.83:0/347585172 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7f9c06b4d0 msgr2=0x7f7f9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.716+0000 7f7fc102a640 1 --2- 172.21.15.83:0/347585172 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7f9c06b4d0 0x7f7f9c06d990 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7fa4003cc0 tx=0x7f7fa40079b0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.716+0000 7f7fc102a640 1 -- 172.21.15.83:0/347585172 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fbc0d2920 msgr2=0x7f7fbc1f40a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.716+0000 7f7fc102a640 1 --2- 172.21.15.83:0/347585172 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fbc0d2920 0x7f7fbc1f40a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7fac02b950 tx=0x7f7fac0046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.716+0000 7f7fc102a640 1 -- 172.21.15.83:0/347585172 shutdown_connections 2024-03-06T07:22:10.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.716+0000 7f7fc102a640 1 --2- 172.21.15.83:0/347585172 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7f9c06b4d0 0x7f7f9c06d990 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.716+0000 7f7fc102a640 1 --2- 172.21.15.83:0/347585172 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7fbc15e090 0x7f7fbc1f7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.716+0000 7f7fc102a640 1 --2- 172.21.15.83:0/347585172 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fbc0d2920 0x7f7fbc1f40a0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.716+0000 7f7fc102a640 1 --2- 172.21.15.83:0/347585172 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7fbc0d0650 0x7f7fbc1582f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.716+0000 7f7fc102a640 1 -- 172.21.15.83:0/347585172 >> 172.21.15.83:0/347585172 conn(0x7f7fbc0d3c80 msgr2=0x7f7fbc0b47a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.716+0000 7f7fc102a640 1 -- 172.21.15.83:0/347585172 shutdown_connections 2024-03-06T07:22:10.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.716+0000 7f7fc102a640 1 -- 172.21.15.83:0/347585172 wait complete. 2024-03-06T07:22:10.732 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-03-06T07:22:10.323658+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219020,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755700,"statfs":{"total":773094113280,"available":772869836800,"internally_reserved":0,"allocated":2891776,"data_stored":1753028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12757,"internal_metadata":221236779},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.381376"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299166+0000","last_change":"2024-03-06T07:21:58.961070+0000","last_active":"2024-03-06T07:22:00.299166+0000","last_peered":"2024-03-06T07:22:00.299166+0000","last_clean":"2024-03-06T07:22:00.299166+0000","last_became_active":"2024-03-06T07:21:58.960784+0000","last_became_peered":"2024-03-06T07:21:58.960784+0000","last_unstale":"2024-03-06T07:22:00.299166+0000","last_undegraded":"2024-03-06T07:22:00.299166+0000","last_fullsized":"2024-03-06T07:22:00.299166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:06.413968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943975+0000","last_change":"2024-03-06T07:21:58.958978+0000","last_active":"2024-03-06T07:21:59.943975+0000","last_peered":"2024-03-06T07:21:59.943975+0000","last_clean":"2024-03-06T07:21:59.943975+0000","last_became_active":"2024-03-06T07:21:58.958728+0000","last_became_peered":"2024-03-06T07:21:58.958728+0000","last_unstale":"2024-03-06T07:21:59.943975+0000","last_undegraded":"2024-03-06T07:21:59.943975+0000","last_fullsized":"2024-03-06T07:21:59.943975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:54.495097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298515+0000","last_change":"2024-03-06T07:21:58.953968+0000","last_active":"2024-03-06T07:22:00.298515+0000","last_peered":"2024-03-06T07:22:00.298515+0000","last_clean":"2024-03-06T07:22:00.298515+0000","last_became_active":"2024-03-06T07:21:58.953678+0000","last_became_peered":"2024-03-06T07:21:58.953678+0000","last_unstale":"2024-03-06T07:22:00.298515+0000","last_undegraded":"2024-03-06T07:22:00.298515+0000","last_fullsized":"2024-03-06T07:22:00.298515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:51:39.557766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298069+0000","last_change":"2024-03-06T07:21:58.970682+0000","last_active":"2024-03-06T07:22:00.298069+0000","last_peered":"2024-03-06T07:22:00.298069+0000","last_clean":"2024-03-06T07:22:00.298069+0000","last_became_active":"2024-03-06T07:21:58.970397+0000","last_became_peered":"2024-03-06T07:21:58.970397+0000","last_unstale":"2024-03-06T07:22:00.298069+0000","last_undegraded":"2024-03-06T07:22:00.298069+0000","last_fullsized":"2024-03-06T07:22:00.298069+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:44.732365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298736+0000","last_change":"2024-03-06T07:21:58.965912+0000","last_active":"2024-03-06T07:22:00.298736+0000","last_peered":"2024-03-06T07:22:00.298736+0000","last_clean":"2024-03-06T07:22:00.298736+0000","last_became_active":"2024-03-06T07:21:58.965567+0000","last_became_peered":"2024-03-06T07:21:58.965567+0000","last_unstale":"2024-03-06T07:22:00.298736+0000","last_undegraded":"2024-03-06T07:22:00.298736+0000","last_fullsized":"2024-03-06T07:22:00.298736+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:10:08.320367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298064+0000","last_change":"2024-03-06T07:21:58.963244+0000","last_active":"2024-03-06T07:22:00.298064+0000","last_peered":"2024-03-06T07:22:00.298064+0000","last_clean":"2024-03-06T07:22:00.298064+0000","last_became_active":"2024-03-06T07:21:58.962903+0000","last_became_peered":"2024-03-06T07:21:58.962903+0000","last_unstale":"2024-03-06T07:22:00.298064+0000","last_undegraded":"2024-03-06T07:22:00.298064+0000","last_fullsized":"2024-03-06T07:22:00.298064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:30:54.760925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.297983+0000","last_change":"2024-03-06T07:21:58.964015+0000","last_active":"2024-03-06T07:22:00.297983+0000","last_peered":"2024-03-06T07:22:00.297983+0000","last_clean":"2024-03-06T07:22:00.297983+0000","last_became_active":"2024-03-06T07:21:58.962062+0000","last_became_peered":"2024-03-06T07:21:58.962062+0000","last_unstale":"2024-03-06T07:22:00.297983+0000","last_undegraded":"2024-03-06T07:22:00.297983+0000","last_fullsized":"2024-03-06T07:22:00.297983+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:44.732365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298608+0000","last_change":"2024-03-06T07:21:58.968307+0000","last_active":"2024-03-06T07:22:00.298608+0000","last_peered":"2024-03-06T07:22:00.298608+0000","last_clean":"2024-03-06T07:22:00.298608+0000","last_became_active":"2024-03-06T07:21:58.967917+0000","last_became_peered":"2024-03-06T07:21:58.967917+0000","last_unstale":"2024-03-06T07:22:00.298608+0000","last_undegraded":"2024-03-06T07:22:00.298608+0000","last_fullsized":"2024-03-06T07:22:00.298608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:21:17.637762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299130+0000","last_change":"2024-03-06T07:21:58.971497+0000","last_active":"2024-03-06T07:22:00.299130+0000","last_peered":"2024-03-06T07:22:00.299130+0000","last_clean":"2024-03-06T07:22:00.299130+0000","last_became_active":"2024-03-06T07:21:58.970710+0000","last_became_peered":"2024-03-06T07:21:58.970710+0000","last_unstale":"2024-03-06T07:22:00.299130+0000","last_undegraded":"2024-03-06T07:22:00.299130+0000","last_fullsized":"2024-03-06T07:22:00.299130+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:13.580324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943457+0000","last_change":"2024-03-06T07:21:58.956333+0000","last_active":"2024-03-06T07:21:59.943457+0000","last_peered":"2024-03-06T07:21:59.943457+0000","last_clean":"2024-03-06T07:21:59.943457+0000","last_became_active":"2024-03-06T07:21:58.956073+0000","last_became_peered":"2024-03-06T07:21:58.956073+0000","last_unstale":"2024-03-06T07:21:59.943457+0000","last_undegraded":"2024-03-06T07:21:59.943457+0000","last_fullsized":"2024-03-06T07:21:59.943457+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:30:54.760925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943172+0000","last_change":"2024-03-06T07:21:58.956210+0000","last_active":"2024-03-06T07:21:59.943172+0000","last_peered":"2024-03-06T07:21:59.943172+0000","last_clean":"2024-03-06T07:21:59.943172+0000","last_became_active":"2024-03-06T07:21:58.955949+0000","last_became_peered":"2024-03-06T07:21:58.955949+0000","last_unstale":"2024-03-06T07:21:59.943172+0000","last_undegraded":"2024-03-06T07:21:59.943172+0000","last_fullsized":"2024-03-06T07:21:59.943172+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:54.495097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943018+0000","last_change":"2024-03-06T07:21:58.950931+0000","last_active":"2024-03-06T07:21:59.943018+0000","last_peered":"2024-03-06T07:21:59.943018+0000","last_clean":"2024-03-06T07:21:59.943018+0000","last_became_active":"2024-03-06T07:21:58.949704+0000","last_became_peered":"2024-03-06T07:21:58.949704+0000","last_unstale":"2024-03-06T07:21:59.943018+0000","last_undegraded":"2024-03-06T07:21:59.943018+0000","last_fullsized":"2024-03-06T07:21:59.943018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:02.162943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299186+0000","last_change":"2024-03-06T07:21:58.956175+0000","last_active":"2024-03-06T07:22:00.299186+0000","last_peered":"2024-03-06T07:22:00.299186+0000","last_clean":"2024-03-06T07:22:00.299186+0000","last_became_active":"2024-03-06T07:21:58.955857+0000","last_became_peered":"2024-03-06T07:21:58.955857+0000","last_unstale":"2024-03-06T07:22:00.299186+0000","last_undegraded":"2024-03-06T07:22:00.299186+0000","last_fullsized":"2024-03-06T07:22:00.299186+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:44.732365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298168+0000","last_change":"2024-03-06T07:21:58.951176+0000","last_active":"2024-03-06T07:22:00.298168+0000","last_peered":"2024-03-06T07:22:00.298168+0000","last_clean":"2024-03-06T07:22:00.298168+0000","last_became_active":"2024-03-06T07:21:58.950874+0000","last_became_peered":"2024-03-06T07:21:58.950874+0000","last_unstale":"2024-03-06T07:22:00.298168+0000","last_undegraded":"2024-03-06T07:22:00.298168+0000","last_fullsized":"2024-03-06T07:22:00.298168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:42:14.052025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943706+0000","last_change":"2024-03-06T07:21:58.954562+0000","last_active":"2024-03-06T07:21:59.943706+0000","last_peered":"2024-03-06T07:21:59.943706+0000","last_clean":"2024-03-06T07:21:59.943706+0000","last_became_active":"2024-03-06T07:21:58.954242+0000","last_became_peered":"2024-03-06T07:21:58.954242+0000","last_unstale":"2024-03-06T07:21:59.943706+0000","last_undegraded":"2024-03-06T07:21:59.943706+0000","last_fullsized":"2024-03-06T07:21:59.943706+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:22.038204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.297923+0000","last_change":"2024-03-06T07:21:58.963478+0000","last_active":"2024-03-06T07:22:00.297923+0000","last_peered":"2024-03-06T07:22:00.297923+0000","last_clean":"2024-03-06T07:22:00.297923+0000","last_became_active":"2024-03-06T07:21:58.962917+0000","last_became_peered":"2024-03-06T07:21:58.962917+0000","last_unstale":"2024-03-06T07:22:00.297923+0000","last_undegraded":"2024-03-06T07:22:00.297923+0000","last_fullsized":"2024-03-06T07:22:00.297923+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:54.495097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298958+0000","last_change":"2024-03-06T07:21:58.965994+0000","last_active":"2024-03-06T07:22:00.298958+0000","last_peered":"2024-03-06T07:22:00.298958+0000","last_clean":"2024-03-06T07:22:00.298958+0000","last_became_active":"2024-03-06T07:21:58.965655+0000","last_became_peered":"2024-03-06T07:21:58.965655+0000","last_unstale":"2024-03-06T07:22:00.298958+0000","last_undegraded":"2024-03-06T07:22:00.298958+0000","last_fullsized":"2024-03-06T07:22:00.298958+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:06.413968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298767+0000","last_change":"2024-03-06T07:21:58.955322+0000","last_active":"2024-03-06T07:22:00.298767+0000","last_peered":"2024-03-06T07:22:00.298767+0000","last_clean":"2024-03-06T07:22:00.298767+0000","last_became_active":"2024-03-06T07:21:58.955080+0000","last_became_peered":"2024-03-06T07:21:58.955080+0000","last_unstale":"2024-03-06T07:22:00.298767+0000","last_undegraded":"2024-03-06T07:22:00.298767+0000","last_fullsized":"2024-03-06T07:22:00.298767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:20.851108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298974+0000","last_change":"2024-03-06T07:21:58.966461+0000","last_active":"2024-03-06T07:22:00.298974+0000","last_peered":"2024-03-06T07:22:00.298974+0000","last_clean":"2024-03-06T07:22:00.298974+0000","last_became_active":"2024-03-06T07:21:58.966154+0000","last_became_peered":"2024-03-06T07:21:58.966154+0000","last_unstale":"2024-03-06T07:22:00.298974+0000","last_undegraded":"2024-03-06T07:22:00.298974+0000","last_fullsized":"2024-03-06T07:22:00.298974+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:37:30.965258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942831+0000","last_change":"2024-03-06T07:21:58.953916+0000","last_active":"2024-03-06T07:21:59.942831+0000","last_peered":"2024-03-06T07:21:59.942831+0000","last_clean":"2024-03-06T07:21:59.942831+0000","last_became_active":"2024-03-06T07:21:58.953623+0000","last_became_peered":"2024-03-06T07:21:58.953623+0000","last_unstale":"2024-03-06T07:21:59.942831+0000","last_undegraded":"2024-03-06T07:21:59.942831+0000","last_fullsized":"2024-03-06T07:21:59.942831+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:21:17.637762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299245+0000","last_change":"2024-03-06T07:21:58.955378+0000","last_active":"2024-03-06T07:22:00.299245+0000","last_peered":"2024-03-06T07:22:00.299245+0000","last_clean":"2024-03-06T07:22:00.299245+0000","last_became_active":"2024-03-06T07:21:58.955118+0000","last_became_peered":"2024-03-06T07:21:58.955118+0000","last_unstale":"2024-03-06T07:22:00.299245+0000","last_undegraded":"2024-03-06T07:22:00.299245+0000","last_fullsized":"2024-03-06T07:22:00.299245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:22.038204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298483+0000","last_change":"2024-03-06T07:21:58.963849+0000","last_active":"2024-03-06T07:22:00.298483+0000","last_peered":"2024-03-06T07:22:00.298483+0000","last_clean":"2024-03-06T07:22:00.298483+0000","last_became_active":"2024-03-06T07:21:58.963573+0000","last_became_peered":"2024-03-06T07:21:58.963573+0000","last_unstale":"2024-03-06T07:22:00.298483+0000","last_undegraded":"2024-03-06T07:22:00.298483+0000","last_fullsized":"2024-03-06T07:22:00.298483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:20.758138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299836+0000","last_change":"2024-03-06T07:21:58.962026+0000","last_active":"2024-03-06T07:22:00.299836+0000","last_peered":"2024-03-06T07:22:00.299836+0000","last_clean":"2024-03-06T07:22:00.299836+0000","last_became_active":"2024-03-06T07:21:58.961691+0000","last_became_peered":"2024-03-06T07:21:58.961691+0000","last_unstale":"2024-03-06T07:22:00.299836+0000","last_undegraded":"2024-03-06T07:22:00.299836+0000","last_fullsized":"2024-03-06T07:22:00.299836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:02:11.315323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298380+0000","last_change":"2024-03-06T07:21:58.970098+0000","last_active":"2024-03-06T07:22:00.298380+0000","last_peered":"2024-03-06T07:22:00.298380+0000","last_clean":"2024-03-06T07:22:00.298380+0000","last_became_active":"2024-03-06T07:21:58.969786+0000","last_became_peered":"2024-03-06T07:21:58.969786+0000","last_unstale":"2024-03-06T07:22:00.298380+0000","last_undegraded":"2024-03-06T07:22:00.298380+0000","last_fullsized":"2024-03-06T07:22:00.298380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:07.060045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.297875+0000","last_change":"2024-03-06T07:21:58.961107+0000","last_active":"2024-03-06T07:22:00.297875+0000","last_peered":"2024-03-06T07:22:00.297875+0000","last_clean":"2024-03-06T07:22:00.297875+0000","last_became_active":"2024-03-06T07:21:58.960826+0000","last_became_peered":"2024-03-06T07:21:58.960826+0000","last_unstale":"2024-03-06T07:22:00.297875+0000","last_undegraded":"2024-03-06T07:22:00.297875+0000","last_fullsized":"2024-03-06T07:22:00.297875+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:36:02.122800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942422+0000","last_change":"2024-03-06T07:21:58.954994+0000","last_active":"2024-03-06T07:21:59.942422+0000","last_peered":"2024-03-06T07:21:59.942422+0000","last_clean":"2024-03-06T07:21:59.942422+0000","last_became_active":"2024-03-06T07:21:58.954710+0000","last_became_peered":"2024-03-06T07:21:58.954710+0000","last_unstale":"2024-03-06T07:21:59.942422+0000","last_undegraded":"2024-03-06T07:21:59.942422+0000","last_fullsized":"2024-03-06T07:21:59.942422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:12.378982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943938+0000","last_change":"2024-03-06T07:21:58.956241+0000","last_active":"2024-03-06T07:21:59.943938+0000","last_peered":"2024-03-06T07:21:59.943938+0000","last_clean":"2024-03-06T07:21:59.943938+0000","last_became_active":"2024-03-06T07:21:58.955977+0000","last_became_peered":"2024-03-06T07:21:58.955977+0000","last_unstale":"2024-03-06T07:21:59.943938+0000","last_undegraded":"2024-03-06T07:21:59.943938+0000","last_fullsized":"2024-03-06T07:21:59.943938+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:21:17.637762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298498+0000","last_change":"2024-03-06T07:21:58.963500+0000","last_active":"2024-03-06T07:22:00.298498+0000","last_peered":"2024-03-06T07:22:00.298498+0000","last_clean":"2024-03-06T07:22:00.298498+0000","last_became_active":"2024-03-06T07:21:58.963218+0000","last_became_peered":"2024-03-06T07:21:58.963218+0000","last_unstale":"2024-03-06T07:22:00.298498+0000","last_undegraded":"2024-03-06T07:22:00.298498+0000","last_fullsized":"2024-03-06T07:22:00.298498+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:12.378982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299375+0000","last_change":"2024-03-06T07:21:58.957981+0000","last_active":"2024-03-06T07:22:00.299375+0000","last_peered":"2024-03-06T07:22:00.299375+0000","last_clean":"2024-03-06T07:22:00.299375+0000","last_became_active":"2024-03-06T07:21:58.957690+0000","last_became_peered":"2024-03-06T07:21:58.957690+0000","last_unstale":"2024-03-06T07:22:00.299375+0000","last_undegraded":"2024-03-06T07:22:00.299375+0000","last_fullsized":"2024-03-06T07:22:00.299375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:59.261197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298443+0000","last_change":"2024-03-06T07:21:58.968221+0000","last_active":"2024-03-06T07:22:00.298443+0000","last_peered":"2024-03-06T07:22:00.298443+0000","last_clean":"2024-03-06T07:22:00.298443+0000","last_became_active":"2024-03-06T07:21:58.967798+0000","last_became_peered":"2024-03-06T07:21:58.967798+0000","last_unstale":"2024-03-06T07:22:00.298443+0000","last_undegraded":"2024-03-06T07:22:00.298443+0000","last_fullsized":"2024-03-06T07:22:00.298443+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:37:30.965258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298249+0000","last_change":"2024-03-06T07:21:58.966486+0000","last_active":"2024-03-06T07:22:00.298249+0000","last_peered":"2024-03-06T07:22:00.298249+0000","last_clean":"2024-03-06T07:22:00.298249+0000","last_became_active":"2024-03-06T07:21:58.966202+0000","last_became_peered":"2024-03-06T07:21:58.966202+0000","last_unstale":"2024-03-06T07:22:00.298249+0000","last_undegraded":"2024-03-06T07:22:00.298249+0000","last_fullsized":"2024-03-06T07:22:00.298249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:20.908786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942026+0000","last_change":"2024-03-06T07:21:58.952310+0000","last_active":"2024-03-06T07:21:59.942026+0000","last_peered":"2024-03-06T07:21:59.942026+0000","last_clean":"2024-03-06T07:21:59.942026+0000","last_became_active":"2024-03-06T07:21:58.952054+0000","last_became_peered":"2024-03-06T07:21:58.952054+0000","last_unstale":"2024-03-06T07:21:59.942026+0000","last_undegraded":"2024-03-06T07:21:59.942026+0000","last_fullsized":"2024-03-06T07:21:59.942026+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:55.294652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942908+0000","last_change":"2024-03-06T07:21:58.951201+0000","last_active":"2024-03-06T07:21:59.942908+0000","last_peered":"2024-03-06T07:21:59.942908+0000","last_clean":"2024-03-06T07:21:59.942908+0000","last_became_active":"2024-03-06T07:21:58.950746+0000","last_became_peered":"2024-03-06T07:21:58.950746+0000","last_unstale":"2024-03-06T07:21:59.942908+0000","last_undegraded":"2024-03-06T07:21:59.942908+0000","last_fullsized":"2024-03-06T07:21:59.942908+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:52.785061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299467+0000","last_change":"2024-03-06T07:21:57.947333+0000","last_active":"2024-03-06T07:22:00.299467+0000","last_peered":"2024-03-06T07:22:00.299467+0000","last_clean":"2024-03-06T07:22:00.299467+0000","last_became_active":"2024-03-06T07:21:57.947066+0000","last_became_peered":"2024-03-06T07:21:57.947066+0000","last_unstale":"2024-03-06T07:22:00.299467+0000","last_undegraded":"2024-03-06T07:22:00.299467+0000","last_fullsized":"2024-03-06T07:22:00.299467+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:18:20.077304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299505+0000","last_change":"2024-03-06T07:21:58.954116+0000","last_active":"2024-03-06T07:22:00.299505+0000","last_peered":"2024-03-06T07:22:00.299505+0000","last_clean":"2024-03-06T07:22:00.299505+0000","last_became_active":"2024-03-06T07:21:58.953858+0000","last_became_peered":"2024-03-06T07:21:58.953858+0000","last_unstale":"2024-03-06T07:22:00.299505+0000","last_undegraded":"2024-03-06T07:22:00.299505+0000","last_fullsized":"2024-03-06T07:22:00.299505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:12.378982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942428+0000","last_change":"2024-03-06T07:21:57.937765+0000","last_active":"2024-03-06T07:21:59.942428+0000","last_peered":"2024-03-06T07:21:59.942428+0000","last_clean":"2024-03-06T07:21:59.942428+0000","last_became_active":"2024-03-06T07:21:57.937476+0000","last_became_peered":"2024-03-06T07:21:57.937476+0000","last_unstale":"2024-03-06T07:21:59.942428+0000","last_undegraded":"2024-03-06T07:21:59.942428+0000","last_fullsized":"2024-03-06T07:21:59.942428+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:07:58.548928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299878+0000","last_change":"2024-03-06T07:21:57.943799+0000","last_active":"2024-03-06T07:22:00.299878+0000","last_peered":"2024-03-06T07:22:00.299878+0000","last_clean":"2024-03-06T07:22:00.299878+0000","last_became_active":"2024-03-06T07:21:57.943507+0000","last_became_peered":"2024-03-06T07:21:57.943507+0000","last_unstale":"2024-03-06T07:22:00.299878+0000","last_undegraded":"2024-03-06T07:22:00.299878+0000","last_fullsized":"2024-03-06T07:22:00.299878+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:31.179338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298759+0000","last_change":"2024-03-06T07:21:58.965398+0000","last_active":"2024-03-06T07:22:00.298759+0000","last_peered":"2024-03-06T07:22:00.298759+0000","last_clean":"2024-03-06T07:22:00.298759+0000","last_became_active":"2024-03-06T07:21:58.964367+0000","last_became_peered":"2024-03-06T07:21:58.964367+0000","last_unstale":"2024-03-06T07:22:00.298759+0000","last_undegraded":"2024-03-06T07:22:00.298759+0000","last_fullsized":"2024-03-06T07:22:00.298759+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:54.495097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298098+0000","last_change":"2024-03-06T07:21:57.951473+0000","last_active":"2024-03-06T07:22:00.298098+0000","last_peered":"2024-03-06T07:22:00.298098+0000","last_clean":"2024-03-06T07:22:00.298098+0000","last_became_active":"2024-03-06T07:21:57.951208+0000","last_became_peered":"2024-03-06T07:21:57.951208+0000","last_unstale":"2024-03-06T07:22:00.298098+0000","last_undegraded":"2024-03-06T07:22:00.298098+0000","last_fullsized":"2024-03-06T07:22:00.298098+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:51.781285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943705+0000","last_change":"2024-03-06T07:21:58.961873+0000","last_active":"2024-03-06T07:21:59.943705+0000","last_peered":"2024-03-06T07:21:59.943705+0000","last_clean":"2024-03-06T07:21:59.943705+0000","last_became_active":"2024-03-06T07:21:58.961630+0000","last_became_peered":"2024-03-06T07:21:58.961630+0000","last_unstale":"2024-03-06T07:21:59.943705+0000","last_undegraded":"2024-03-06T07:21:59.943705+0000","last_fullsized":"2024-03-06T07:21:59.943705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:55.294652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942284+0000","last_change":"2024-03-06T07:21:57.948765+0000","last_active":"2024-03-06T07:21:59.942284+0000","last_peered":"2024-03-06T07:21:59.942284+0000","last_clean":"2024-03-06T07:21:59.942284+0000","last_became_active":"2024-03-06T07:21:57.948498+0000","last_became_peered":"2024-03-06T07:21:57.948498+0000","last_unstale":"2024-03-06T07:21:59.942284+0000","last_undegraded":"2024-03-06T07:21:59.942284+0000","last_fullsized":"2024-03-06T07:21:59.942284+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:51.781285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298709+0000","last_change":"2024-03-06T07:21:58.971817+0000","last_active":"2024-03-06T07:22:00.298709+0000","last_peered":"2024-03-06T07:22:00.298709+0000","last_clean":"2024-03-06T07:22:00.298709+0000","last_became_active":"2024-03-06T07:21:58.971481+0000","last_became_peered":"2024-03-06T07:21:58.971481+0000","last_unstale":"2024-03-06T07:22:00.298709+0000","last_undegraded":"2024-03-06T07:22:00.298709+0000","last_fullsized":"2024-03-06T07:22:00.298709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:13.580324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298587+0000","last_change":"2024-03-06T07:21:57.950501+0000","last_active":"2024-03-06T07:22:00.298587+0000","last_peered":"2024-03-06T07:22:00.298587+0000","last_clean":"2024-03-06T07:22:00.298587+0000","last_became_active":"2024-03-06T07:21:57.950201+0000","last_became_peered":"2024-03-06T07:21:57.950201+0000","last_unstale":"2024-03-06T07:22:00.298587+0000","last_undegraded":"2024-03-06T07:22:00.298587+0000","last_fullsized":"2024-03-06T07:22:00.298587+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:13.736718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298791+0000","last_change":"2024-03-06T07:21:58.958089+0000","last_active":"2024-03-06T07:22:00.298791+0000","last_peered":"2024-03-06T07:22:00.298791+0000","last_clean":"2024-03-06T07:22:00.298791+0000","last_became_active":"2024-03-06T07:21:58.957814+0000","last_became_peered":"2024-03-06T07:21:58.957814+0000","last_unstale":"2024-03-06T07:22:00.298791+0000","last_undegraded":"2024-03-06T07:22:00.298791+0000","last_fullsized":"2024-03-06T07:22:00.298791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:55.294652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943941+0000","last_change":"2024-03-06T07:21:57.946124+0000","last_active":"2024-03-06T07:21:59.943941+0000","last_peered":"2024-03-06T07:21:59.943941+0000","last_clean":"2024-03-06T07:21:59.943941+0000","last_became_active":"2024-03-06T07:21:57.945856+0000","last_became_peered":"2024-03-06T07:21:57.945856+0000","last_unstale":"2024-03-06T07:21:59.943941+0000","last_undegraded":"2024-03-06T07:21:59.943941+0000","last_fullsized":"2024-03-06T07:21:59.943941+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:01.159166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943172+0000","last_change":"2024-03-06T07:21:58.958734+0000","last_active":"2024-03-06T07:21:59.943172+0000","last_peered":"2024-03-06T07:21:59.943172+0000","last_clean":"2024-03-06T07:21:59.943172+0000","last_became_active":"2024-03-06T07:21:58.958451+0000","last_became_peered":"2024-03-06T07:21:58.958451+0000","last_unstale":"2024-03-06T07:21:59.943172+0000","last_undegraded":"2024-03-06T07:21:59.943172+0000","last_fullsized":"2024-03-06T07:21:59.943172+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:20.908786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942962+0000","last_change":"2024-03-06T07:21:57.947199+0000","last_active":"2024-03-06T07:21:59.942962+0000","last_peered":"2024-03-06T07:21:59.942962+0000","last_clean":"2024-03-06T07:21:59.942962+0000","last_became_active":"2024-03-06T07:21:57.946927+0000","last_became_peered":"2024-03-06T07:21:57.946927+0000","last_unstale":"2024-03-06T07:21:59.942962+0000","last_undegraded":"2024-03-06T07:21:59.942962+0000","last_fullsized":"2024-03-06T07:21:59.942962+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:01.159166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298650+0000","last_change":"2024-03-06T07:21:58.970850+0000","last_active":"2024-03-06T07:22:00.298650+0000","last_peered":"2024-03-06T07:22:00.298650+0000","last_clean":"2024-03-06T07:22:00.298650+0000","last_became_active":"2024-03-06T07:21:58.970512+0000","last_became_peered":"2024-03-06T07:21:58.970512+0000","last_unstale":"2024-03-06T07:22:00.298650+0000","last_undegraded":"2024-03-06T07:22:00.298650+0000","last_fullsized":"2024-03-06T07:22:00.298650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:02:11.315323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299603+0000","last_change":"2024-03-06T07:21:57.951893+0000","last_active":"2024-03-06T07:22:00.299603+0000","last_peered":"2024-03-06T07:22:00.299603+0000","last_clean":"2024-03-06T07:22:00.299603+0000","last_became_active":"2024-03-06T07:21:57.951696+0000","last_became_peered":"2024-03-06T07:21:57.951696+0000","last_unstale":"2024-03-06T07:22:00.299603+0000","last_undegraded":"2024-03-06T07:22:00.299603+0000","last_fullsized":"2024-03-06T07:22:00.299603+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:27:31.864292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298310+0000","last_change":"2024-03-06T07:21:58.954594+0000","last_active":"2024-03-06T07:22:00.298310+0000","last_peered":"2024-03-06T07:22:00.298310+0000","last_clean":"2024-03-06T07:22:00.298310+0000","last_became_active":"2024-03-06T07:21:58.954326+0000","last_became_peered":"2024-03-06T07:21:58.954326+0000","last_unstale":"2024-03-06T07:22:00.298310+0000","last_undegraded":"2024-03-06T07:22:00.298310+0000","last_fullsized":"2024-03-06T07:22:00.298310+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:06:11.334051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298716+0000","last_change":"2024-03-06T07:21:57.956339+0000","last_active":"2024-03-06T07:22:00.298716+0000","last_peered":"2024-03-06T07:22:00.298716+0000","last_clean":"2024-03-06T07:22:00.298716+0000","last_became_active":"2024-03-06T07:21:57.956135+0000","last_became_peered":"2024-03-06T07:21:57.956135+0000","last_unstale":"2024-03-06T07:22:00.298716+0000","last_undegraded":"2024-03-06T07:22:00.298716+0000","last_fullsized":"2024-03-06T07:22:00.298716+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:01.159166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299011+0000","last_change":"2024-03-06T07:21:58.962391+0000","last_active":"2024-03-06T07:22:00.299011+0000","last_peered":"2024-03-06T07:22:00.299011+0000","last_clean":"2024-03-06T07:22:00.299011+0000","last_became_active":"2024-03-06T07:21:58.961887+0000","last_became_peered":"2024-03-06T07:21:58.961887+0000","last_unstale":"2024-03-06T07:22:00.299011+0000","last_undegraded":"2024-03-06T07:22:00.299011+0000","last_fullsized":"2024-03-06T07:22:00.299011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:20.758138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942138+0000","last_change":"2024-03-06T07:21:57.939474+0000","last_active":"2024-03-06T07:21:59.942138+0000","last_peered":"2024-03-06T07:21:59.942138+0000","last_clean":"2024-03-06T07:21:59.942138+0000","last_became_active":"2024-03-06T07:21:57.939053+0000","last_became_peered":"2024-03-06T07:21:57.939053+0000","last_unstale":"2024-03-06T07:21:59.942138+0000","last_undegraded":"2024-03-06T07:21:59.942138+0000","last_fullsized":"2024-03-06T07:21:59.942138+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:35.501272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942064+0000","last_change":"2024-03-06T07:21:58.958835+0000","last_active":"2024-03-06T07:21:59.942064+0000","last_peered":"2024-03-06T07:21:59.942064+0000","last_clean":"2024-03-06T07:21:59.942064+0000","last_became_active":"2024-03-06T07:21:58.958518+0000","last_became_peered":"2024-03-06T07:21:58.958518+0000","last_unstale":"2024-03-06T07:21:59.942064+0000","last_undegraded":"2024-03-06T07:21:59.942064+0000","last_fullsized":"2024-03-06T07:21:59.942064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:22.038204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298611+0000","last_change":"2024-03-06T07:21:57.940312+0000","last_active":"2024-03-06T07:22:00.298611+0000","last_peered":"2024-03-06T07:22:00.298611+0000","last_clean":"2024-03-06T07:22:00.298611+0000","last_became_active":"2024-03-06T07:21:57.940030+0000","last_became_peered":"2024-03-06T07:21:57.940030+0000","last_unstale":"2024-03-06T07:22:00.298611+0000","last_undegraded":"2024-03-06T07:22:00.298611+0000","last_fullsized":"2024-03-06T07:22:00.298611+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:35.501272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298350+0000","last_change":"2024-03-06T07:21:58.961737+0000","last_active":"2024-03-06T07:22:00.298350+0000","last_peered":"2024-03-06T07:22:00.298350+0000","last_clean":"2024-03-06T07:22:00.298350+0000","last_became_active":"2024-03-06T07:21:58.961365+0000","last_became_peered":"2024-03-06T07:21:58.961365+0000","last_unstale":"2024-03-06T07:22:00.298350+0000","last_undegraded":"2024-03-06T07:22:00.298350+0000","last_fullsized":"2024-03-06T07:22:00.298350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:44.889543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943032+0000","last_change":"2024-03-06T07:21:57.948442+0000","last_active":"2024-03-06T07:21:59.943032+0000","last_peered":"2024-03-06T07:21:59.943032+0000","last_clean":"2024-03-06T07:21:59.943032+0000","last_became_active":"2024-03-06T07:21:57.948170+0000","last_became_peered":"2024-03-06T07:21:57.948170+0000","last_unstale":"2024-03-06T07:21:59.943032+0000","last_undegraded":"2024-03-06T07:21:59.943032+0000","last_fullsized":"2024-03-06T07:21:59.943032+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:10.622090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298979+0000","last_change":"2024-03-06T07:21:58.961595+0000","last_active":"2024-03-06T07:22:00.298979+0000","last_peered":"2024-03-06T07:22:00.298979+0000","last_clean":"2024-03-06T07:22:00.298979+0000","last_became_active":"2024-03-06T07:21:58.961259+0000","last_became_peered":"2024-03-06T07:21:58.961259+0000","last_unstale":"2024-03-06T07:22:00.298979+0000","last_undegraded":"2024-03-06T07:22:00.298979+0000","last_fullsized":"2024-03-06T07:22:00.298979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:43.260641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298869+0000","last_change":"2024-03-06T07:21:57.942300+0000","last_active":"2024-03-06T07:22:00.298869+0000","last_peered":"2024-03-06T07:22:00.298869+0000","last_clean":"2024-03-06T07:22:00.298869+0000","last_became_active":"2024-03-06T07:21:57.942023+0000","last_became_peered":"2024-03-06T07:21:57.942023+0000","last_unstale":"2024-03-06T07:22:00.298869+0000","last_undegraded":"2024-03-06T07:22:00.298869+0000","last_fullsized":"2024-03-06T07:22:00.298869+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:07:58.548928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299225+0000","last_change":"2024-03-06T07:21:58.970895+0000","last_active":"2024-03-06T07:22:00.299225+0000","last_peered":"2024-03-06T07:22:00.299225+0000","last_clean":"2024-03-06T07:22:00.299225+0000","last_became_active":"2024-03-06T07:21:58.970418+0000","last_became_peered":"2024-03-06T07:21:58.970418+0000","last_unstale":"2024-03-06T07:22:00.299225+0000","last_undegraded":"2024-03-06T07:22:00.299225+0000","last_fullsized":"2024-03-06T07:22:00.299225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:19:55.013458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944014+0000","last_change":"2024-03-06T07:21:57.942325+0000","last_active":"2024-03-06T07:21:59.944014+0000","last_peered":"2024-03-06T07:21:59.944014+0000","last_clean":"2024-03-06T07:21:59.944014+0000","last_became_active":"2024-03-06T07:21:57.942045+0000","last_became_peered":"2024-03-06T07:21:57.942045+0000","last_unstale":"2024-03-06T07:21:59.944014+0000","last_undegraded":"2024-03-06T07:21:59.944014+0000","last_fullsized":"2024-03-06T07:21:59.944014+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:19.905010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298465+0000","last_change":"2024-03-06T07:21:58.967944+0000","last_active":"2024-03-06T07:22:00.298465+0000","last_peered":"2024-03-06T07:22:00.298465+0000","last_clean":"2024-03-06T07:22:00.298465+0000","last_became_active":"2024-03-06T07:21:58.967521+0000","last_became_peered":"2024-03-06T07:21:58.967521+0000","last_unstale":"2024-03-06T07:22:00.298465+0000","last_undegraded":"2024-03-06T07:22:00.298465+0000","last_fullsized":"2024-03-06T07:22:00.298465+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:43.260641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.297951+0000","last_change":"2024-03-06T07:21:57.947477+0000","last_active":"2024-03-06T07:22:00.297951+0000","last_peered":"2024-03-06T07:22:00.297951+0000","last_clean":"2024-03-06T07:22:00.297951+0000","last_became_active":"2024-03-06T07:21:57.946678+0000","last_became_peered":"2024-03-06T07:21:57.946678+0000","last_unstale":"2024-03-06T07:22:00.297951+0000","last_undegraded":"2024-03-06T07:22:00.297951+0000","last_fullsized":"2024-03-06T07:22:00.297951+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:13.736718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298718+0000","last_change":"2024-03-06T07:21:58.970473+0000","last_active":"2024-03-06T07:22:00.298718+0000","last_peered":"2024-03-06T07:22:00.298718+0000","last_clean":"2024-03-06T07:22:00.298718+0000","last_became_active":"2024-03-06T07:21:58.969810+0000","last_became_peered":"2024-03-06T07:21:58.969810+0000","last_unstale":"2024-03-06T07:22:00.298718+0000","last_undegraded":"2024-03-06T07:22:00.298718+0000","last_fullsized":"2024-03-06T07:22:00.298718+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:07.060045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298038+0000","last_change":"2024-03-06T07:21:47.553693+0000","last_active":"2024-03-06T07:22:00.298038+0000","last_peered":"2024-03-06T07:22:00.298038+0000","last_clean":"2024-03-06T07:22:00.298038+0000","last_became_active":"2024-03-06T07:21:47.552664+0000","last_became_peered":"2024-03-06T07:21:47.552664+0000","last_unstale":"2024-03-06T07:22:00.298038+0000","last_undegraded":"2024-03-06T07:22:00.298038+0000","last_fullsized":"2024-03-06T07:22:00.298038+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_clean_scrub_stamp":"2024-03-06T07:21:46.362028+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:02.916426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.297955+0000","last_change":"2024-03-06T07:21:57.949622+0000","last_active":"2024-03-06T07:22:00.297955+0000","last_peered":"2024-03-06T07:22:00.297955+0000","last_clean":"2024-03-06T07:22:00.297955+0000","last_became_active":"2024-03-06T07:21:57.948874+0000","last_became_peered":"2024-03-06T07:21:57.948874+0000","last_unstale":"2024-03-06T07:22:00.297955+0000","last_undegraded":"2024-03-06T07:22:00.297955+0000","last_fullsized":"2024-03-06T07:22:00.297955+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:35.501272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298026+0000","last_change":"2024-03-06T07:21:58.968667+0000","last_active":"2024-03-06T07:22:00.298026+0000","last_peered":"2024-03-06T07:22:00.298026+0000","last_clean":"2024-03-06T07:22:00.298026+0000","last_became_active":"2024-03-06T07:21:58.968213+0000","last_became_peered":"2024-03-06T07:21:58.968213+0000","last_unstale":"2024-03-06T07:22:00.298026+0000","last_undegraded":"2024-03-06T07:22:00.298026+0000","last_fullsized":"2024-03-06T07:22:00.298026+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:22.038204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.297808+0000","last_change":"2024-03-06T07:21:57.951697+0000","last_active":"2024-03-06T07:22:00.297808+0000","last_peered":"2024-03-06T07:22:00.297808+0000","last_clean":"2024-03-06T07:22:00.297808+0000","last_became_active":"2024-03-06T07:21:57.951421+0000","last_became_peered":"2024-03-06T07:21:57.951421+0000","last_unstale":"2024-03-06T07:22:00.297808+0000","last_undegraded":"2024-03-06T07:22:00.297808+0000","last_fullsized":"2024-03-06T07:22:00.297808+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:51.781285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.297721+0000","last_change":"2024-03-06T07:21:58.948562+0000","last_active":"2024-03-06T07:22:00.297721+0000","last_peered":"2024-03-06T07:22:00.297721+0000","last_clean":"2024-03-06T07:22:00.297721+0000","last_became_active":"2024-03-06T07:21:58.948249+0000","last_became_peered":"2024-03-06T07:21:58.948249+0000","last_unstale":"2024-03-06T07:22:00.297721+0000","last_undegraded":"2024-03-06T07:22:00.297721+0000","last_fullsized":"2024-03-06T07:22:00.297721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:53.360751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298409+0000","last_change":"2024-03-06T07:21:57.945890+0000","last_active":"2024-03-06T07:22:00.298409+0000","last_peered":"2024-03-06T07:22:00.298409+0000","last_clean":"2024-03-06T07:22:00.298409+0000","last_became_active":"2024-03-06T07:21:57.945527+0000","last_became_peered":"2024-03-06T07:21:57.945527+0000","last_unstale":"2024-03-06T07:22:00.298409+0000","last_undegraded":"2024-03-06T07:22:00.298409+0000","last_fullsized":"2024-03-06T07:22:00.298409+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:31.179338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943753+0000","last_change":"2024-03-06T07:21:58.952436+0000","last_active":"2024-03-06T07:21:59.943753+0000","last_peered":"2024-03-06T07:21:59.943753+0000","last_clean":"2024-03-06T07:21:59.943753+0000","last_became_active":"2024-03-06T07:21:58.952135+0000","last_became_peered":"2024-03-06T07:21:58.952135+0000","last_unstale":"2024-03-06T07:21:59.943753+0000","last_undegraded":"2024-03-06T07:21:59.943753+0000","last_fullsized":"2024-03-06T07:21:59.943753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:59.261197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298778+0000","last_change":"2024-03-06T07:21:57.951597+0000","last_active":"2024-03-06T07:22:00.298778+0000","last_peered":"2024-03-06T07:22:00.298778+0000","last_clean":"2024-03-06T07:22:00.298778+0000","last_became_active":"2024-03-06T07:21:57.951299+0000","last_became_peered":"2024-03-06T07:21:57.951299+0000","last_unstale":"2024-03-06T07:22:00.298778+0000","last_undegraded":"2024-03-06T07:22:00.298778+0000","last_fullsized":"2024-03-06T07:22:00.298778+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:19.847331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943217+0000","last_change":"2024-03-06T07:21:58.951633+0000","last_active":"2024-03-06T07:21:59.943217+0000","last_peered":"2024-03-06T07:21:59.943217+0000","last_clean":"2024-03-06T07:21:59.943217+0000","last_became_active":"2024-03-06T07:21:58.951315+0000","last_became_peered":"2024-03-06T07:21:58.951315+0000","last_unstale":"2024-03-06T07:21:59.943217+0000","last_undegraded":"2024-03-06T07:21:59.943217+0000","last_fullsized":"2024-03-06T07:21:59.943217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:20.851108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943826+0000","last_change":"2024-03-06T07:21:57.945253+0000","last_active":"2024-03-06T07:21:59.943826+0000","last_peered":"2024-03-06T07:21:59.943826+0000","last_clean":"2024-03-06T07:21:59.943826+0000","last_became_active":"2024-03-06T07:21:57.944983+0000","last_became_peered":"2024-03-06T07:21:57.944983+0000","last_unstale":"2024-03-06T07:21:59.943826+0000","last_undegraded":"2024-03-06T07:21:59.943826+0000","last_fullsized":"2024-03-06T07:21:59.943826+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:07:58.548928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298511+0000","last_change":"2024-03-06T07:21:58.964580+0000","last_active":"2024-03-06T07:22:00.298511+0000","last_peered":"2024-03-06T07:22:00.298511+0000","last_clean":"2024-03-06T07:22:00.298511+0000","last_became_active":"2024-03-06T07:21:58.964272+0000","last_became_peered":"2024-03-06T07:21:58.964272+0000","last_unstale":"2024-03-06T07:22:00.298511+0000","last_undegraded":"2024-03-06T07:22:00.298511+0000","last_fullsized":"2024-03-06T07:22:00.298511+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:02:11.315323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298902+0000","last_change":"2024-03-06T07:21:57.952261+0000","last_active":"2024-03-06T07:22:00.298902+0000","last_peered":"2024-03-06T07:22:00.298902+0000","last_clean":"2024-03-06T07:22:00.298902+0000","last_became_active":"2024-03-06T07:21:57.951991+0000","last_became_peered":"2024-03-06T07:21:57.951991+0000","last_unstale":"2024-03-06T07:22:00.298902+0000","last_undegraded":"2024-03-06T07:22:00.298902+0000","last_fullsized":"2024-03-06T07:22:00.298902+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:52.356974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943810+0000","last_change":"2024-03-06T07:21:58.955953+0000","last_active":"2024-03-06T07:21:59.943810+0000","last_peered":"2024-03-06T07:21:59.943810+0000","last_clean":"2024-03-06T07:21:59.943810+0000","last_became_active":"2024-03-06T07:21:58.955634+0000","last_became_peered":"2024-03-06T07:21:58.955634+0000","last_unstale":"2024-03-06T07:21:59.943810+0000","last_undegraded":"2024-03-06T07:21:59.943810+0000","last_fullsized":"2024-03-06T07:21:59.943810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:12.378982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.342768+0000","last_change":"2024-03-06T07:21:57.943165+0000","last_active":"2024-03-06T07:22:00.342768+0000","last_peered":"2024-03-06T07:22:00.342768+0000","last_clean":"2024-03-06T07:22:00.342768+0000","last_became_active":"2024-03-06T07:21:57.942914+0000","last_became_peered":"2024-03-06T07:21:57.942914+0000","last_unstale":"2024-03-06T07:22:00.342768+0000","last_undegraded":"2024-03-06T07:22:00.342768+0000","last_fullsized":"2024-03-06T07:22:00.342768+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:19.847331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298017+0000","last_change":"2024-03-06T07:21:58.960410+0000","last_active":"2024-03-06T07:22:00.298017+0000","last_peered":"2024-03-06T07:22:00.298017+0000","last_clean":"2024-03-06T07:22:00.298017+0000","last_became_active":"2024-03-06T07:21:58.960121+0000","last_became_peered":"2024-03-06T07:21:58.960121+0000","last_unstale":"2024-03-06T07:22:00.298017+0000","last_undegraded":"2024-03-06T07:22:00.298017+0000","last_fullsized":"2024-03-06T07:22:00.298017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:55.294652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942389+0000","last_change":"2024-03-06T07:21:57.944106+0000","last_active":"2024-03-06T07:21:59.942389+0000","last_peered":"2024-03-06T07:21:59.942389+0000","last_clean":"2024-03-06T07:21:59.942389+0000","last_became_active":"2024-03-06T07:21:57.943815+0000","last_became_peered":"2024-03-06T07:21:57.943815+0000","last_unstale":"2024-03-06T07:21:59.942389+0000","last_undegraded":"2024-03-06T07:21:59.942389+0000","last_fullsized":"2024-03-06T07:21:59.942389+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:13.736718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298012+0000","last_change":"2024-03-06T07:21:58.951840+0000","last_active":"2024-03-06T07:22:00.298012+0000","last_peered":"2024-03-06T07:22:00.298012+0000","last_clean":"2024-03-06T07:22:00.298012+0000","last_became_active":"2024-03-06T07:21:58.951594+0000","last_became_peered":"2024-03-06T07:21:58.951594+0000","last_unstale":"2024-03-06T07:22:00.298012+0000","last_undegraded":"2024-03-06T07:22:00.298012+0000","last_fullsized":"2024-03-06T07:22:00.298012+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:34.528149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.297795+0000","last_change":"2024-03-06T07:21:57.951916+0000","last_active":"2024-03-06T07:22:00.297795+0000","last_peered":"2024-03-06T07:22:00.297795+0000","last_clean":"2024-03-06T07:22:00.297795+0000","last_became_active":"2024-03-06T07:21:57.951647+0000","last_became_peered":"2024-03-06T07:21:57.951647+0000","last_unstale":"2024-03-06T07:22:00.297795+0000","last_undegraded":"2024-03-06T07:22:00.297795+0000","last_fullsized":"2024-03-06T07:22:00.297795+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:13.736718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942859+0000","last_change":"2024-03-06T07:21:58.953147+0000","last_active":"2024-03-06T07:21:59.942859+0000","last_peered":"2024-03-06T07:21:59.942859+0000","last_clean":"2024-03-06T07:21:59.942859+0000","last_became_active":"2024-03-06T07:21:58.952862+0000","last_became_peered":"2024-03-06T07:21:58.952862+0000","last_unstale":"2024-03-06T07:21:59.942859+0000","last_undegraded":"2024-03-06T07:21:59.942859+0000","last_fullsized":"2024-03-06T07:21:59.942859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:14.740494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943795+0000","last_change":"2024-03-06T07:21:57.948593+0000","last_active":"2024-03-06T07:21:59.943795+0000","last_peered":"2024-03-06T07:21:59.943795+0000","last_clean":"2024-03-06T07:21:59.943795+0000","last_became_active":"2024-03-06T07:21:57.948302+0000","last_became_peered":"2024-03-06T07:21:57.948302+0000","last_unstale":"2024-03-06T07:21:59.943795+0000","last_undegraded":"2024-03-06T07:21:59.943795+0000","last_fullsized":"2024-03-06T07:21:59.943795+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:13.736718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298923+0000","last_change":"2024-03-06T07:21:58.953065+0000","last_active":"2024-03-06T07:22:00.298923+0000","last_peered":"2024-03-06T07:22:00.298923+0000","last_clean":"2024-03-06T07:22:00.298923+0000","last_became_active":"2024-03-06T07:21:58.952790+0000","last_became_peered":"2024-03-06T07:21:58.952790+0000","last_unstale":"2024-03-06T07:22:00.298923+0000","last_undegraded":"2024-03-06T07:22:00.298923+0000","last_fullsized":"2024-03-06T07:22:00.298923+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:30:54.760925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943752+0000","last_change":"2024-03-06T07:21:57.945134+0000","last_active":"2024-03-06T07:21:59.943752+0000","last_peered":"2024-03-06T07:21:59.943752+0000","last_clean":"2024-03-06T07:21:59.943752+0000","last_became_active":"2024-03-06T07:21:57.944858+0000","last_became_peered":"2024-03-06T07:21:57.944858+0000","last_unstale":"2024-03-06T07:21:59.943752+0000","last_undegraded":"2024-03-06T07:21:59.943752+0000","last_fullsized":"2024-03-06T07:21:59.943752+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:51.781285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299349+0000","last_change":"2024-03-06T07:21:58.964930+0000","last_active":"2024-03-06T07:22:00.299349+0000","last_peered":"2024-03-06T07:22:00.299349+0000","last_clean":"2024-03-06T07:22:00.299349+0000","last_became_active":"2024-03-06T07:21:58.964474+0000","last_became_peered":"2024-03-06T07:21:58.964474+0000","last_unstale":"2024-03-06T07:22:00.299349+0000","last_undegraded":"2024-03-06T07:22:00.299349+0000","last_fullsized":"2024-03-06T07:22:00.299349+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:44.889543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299058+0000","last_change":"2024-03-06T07:21:57.953248+0000","last_active":"2024-03-06T07:22:00.299058+0000","last_peered":"2024-03-06T07:22:00.299058+0000","last_clean":"2024-03-06T07:22:00.299058+0000","last_became_active":"2024-03-06T07:21:57.952969+0000","last_became_peered":"2024-03-06T07:21:57.952969+0000","last_unstale":"2024-03-06T07:22:00.299058+0000","last_undegraded":"2024-03-06T07:22:00.299058+0000","last_fullsized":"2024-03-06T07:22:00.299058+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:19.905010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942240+0000","last_change":"2024-03-06T07:21:58.954261+0000","last_active":"2024-03-06T07:21:59.942240+0000","last_peered":"2024-03-06T07:21:59.942240+0000","last_clean":"2024-03-06T07:21:59.942240+0000","last_became_active":"2024-03-06T07:21:58.953584+0000","last_became_peered":"2024-03-06T07:21:58.953584+0000","last_unstale":"2024-03-06T07:21:59.942240+0000","last_undegraded":"2024-03-06T07:21:59.942240+0000","last_fullsized":"2024-03-06T07:21:59.942240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:59.261197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298322+0000","last_change":"2024-03-06T07:21:58.959725+0000","last_active":"2024-03-06T07:22:00.298322+0000","last_peered":"2024-03-06T07:22:00.298322+0000","last_clean":"2024-03-06T07:22:00.298322+0000","last_became_active":"2024-03-06T07:21:58.959078+0000","last_became_peered":"2024-03-06T07:21:58.959078+0000","last_unstale":"2024-03-06T07:22:00.298322+0000","last_undegraded":"2024-03-06T07:22:00.298322+0000","last_fullsized":"2024-03-06T07:22:00.298322+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:44.889543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299074+0000","last_change":"2024-03-06T07:21:57.951758+0000","last_active":"2024-03-06T07:22:00.299074+0000","last_peered":"2024-03-06T07:22:00.299074+0000","last_clean":"2024-03-06T07:22:00.299074+0000","last_became_active":"2024-03-06T07:21:57.951452+0000","last_became_peered":"2024-03-06T07:21:57.951452+0000","last_unstale":"2024-03-06T07:22:00.299074+0000","last_undegraded":"2024-03-06T07:22:00.299074+0000","last_fullsized":"2024-03-06T07:22:00.299074+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:01.159166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298378+0000","last_change":"2024-03-06T07:21:58.961541+0000","last_active":"2024-03-06T07:22:00.298378+0000","last_peered":"2024-03-06T07:22:00.298378+0000","last_clean":"2024-03-06T07:22:00.298378+0000","last_became_active":"2024-03-06T07:21:58.961271+0000","last_became_peered":"2024-03-06T07:21:58.961271+0000","last_unstale":"2024-03-06T07:22:00.298378+0000","last_undegraded":"2024-03-06T07:22:00.298378+0000","last_fullsized":"2024-03-06T07:22:00.298378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:10:08.320367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298788+0000","last_change":"2024-03-06T07:21:57.947161+0000","last_active":"2024-03-06T07:22:00.298788+0000","last_peered":"2024-03-06T07:22:00.298788+0000","last_clean":"2024-03-06T07:22:00.298788+0000","last_became_active":"2024-03-06T07:21:57.946849+0000","last_became_peered":"2024-03-06T07:21:57.946849+0000","last_unstale":"2024-03-06T07:22:00.298788+0000","last_undegraded":"2024-03-06T07:22:00.298788+0000","last_fullsized":"2024-03-06T07:22:00.298788+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:19.847331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298661+0000","last_change":"2024-03-06T07:21:58.958814+0000","last_active":"2024-03-06T07:22:00.298661+0000","last_peered":"2024-03-06T07:22:00.298661+0000","last_clean":"2024-03-06T07:22:00.298661+0000","last_became_active":"2024-03-06T07:21:58.958550+0000","last_became_peered":"2024-03-06T07:21:58.958550+0000","last_unstale":"2024-03-06T07:22:00.298661+0000","last_undegraded":"2024-03-06T07:22:00.298661+0000","last_fullsized":"2024-03-06T07:22:00.298661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:36:02.122800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298388+0000","last_change":"2024-03-06T07:21:57.949031+0000","last_active":"2024-03-06T07:22:00.298388+0000","last_peered":"2024-03-06T07:22:00.298388+0000","last_clean":"2024-03-06T07:22:00.298388+0000","last_became_active":"2024-03-06T07:21:57.948741+0000","last_became_peered":"2024-03-06T07:21:57.948741+0000","last_unstale":"2024-03-06T07:22:00.298388+0000","last_undegraded":"2024-03-06T07:22:00.298388+0000","last_fullsized":"2024-03-06T07:22:00.298388+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:35.501272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298230+0000","last_change":"2024-03-06T07:21:58.957355+0000","last_active":"2024-03-06T07:22:00.298230+0000","last_peered":"2024-03-06T07:22:00.298230+0000","last_clean":"2024-03-06T07:22:00.298230+0000","last_became_active":"2024-03-06T07:21:58.956672+0000","last_became_peered":"2024-03-06T07:21:58.956672+0000","last_unstale":"2024-03-06T07:22:00.298230+0000","last_undegraded":"2024-03-06T07:22:00.298230+0000","last_fullsized":"2024-03-06T07:22:00.298230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:59.261197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298955+0000","last_change":"2024-03-06T07:21:57.951293+0000","last_active":"2024-03-06T07:22:00.298955+0000","last_peered":"2024-03-06T07:22:00.298955+0000","last_clean":"2024-03-06T07:22:00.298955+0000","last_became_active":"2024-03-06T07:21:57.950995+0000","last_became_peered":"2024-03-06T07:21:57.950995+0000","last_unstale":"2024-03-06T07:22:00.298955+0000","last_undegraded":"2024-03-06T07:22:00.298955+0000","last_fullsized":"2024-03-06T07:22:00.298955+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:10.622090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:10.737 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:10.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.872+0000 7fe6949c7640 1 Processor -- start 2024-03-06T07:22:10.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.872+0000 7fe6949c7640 1 -- start start 2024-03-06T07:22:10.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.873+0000 7fe6949c7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe69015bdd0 0x7fe6900c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.873+0000 7fe6949c7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6900cebe0 con 0x7fe69015fa10 2024-03-06T07:22:10.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.873+0000 7fe6949c7640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6900ceec0 con 0x7fe6900d16e0 2024-03-06T07:22:10.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.873+0000 7fe6949c7640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6900ce980 con 0x7fe69015bdd0 2024-03-06T07:22:10.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.873+0000 7fe68e59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe69015bdd0 0x7fe6900c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.873+0000 7fe687fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fe69015fa10 0x7fe6900ce270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:42964/0 (socket says 172.21.15.83:42964) 2024-03-06T07:22:10.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.873+0000 7fe68e59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe69015bdd0 0x7fe6900c5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49768/0 (socket says 172.21.15.83:49768) 2024-03-06T07:22:10.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.873+0000 7fe687fff640 1 -- 172.21.15.83:0/750858353 learned_addr learned my addr 172.21.15.83:0/750858353 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:10.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.874+0000 7fe68e59c640 1 -- 172.21.15.83:0/750858353 >> v1:172.21.15.183:6789/0 conn(0x7fe6900d16e0 legacy=0x7fe6900d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.874+0000 7fe68e59c640 1 -- 172.21.15.83:0/750858353 >> v1:172.21.15.83:6789/0 conn(0x7fe69015fa10 legacy=0x7fe6900ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.874+0000 7fe68e59c640 1 -- 172.21.15.83:0/750858353 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6900cf1a0 con 0x7fe69015bdd0 2024-03-06T07:22:10.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.874+0000 7fe68dd9b640 1 -- 172.21.15.83:0/750858353 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1410685283 0 0) 0x7fe6900cebe0 con 0x7fe69015fa10 2024-03-06T07:22:10.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.874+0000 7fe68e59c640 1 --2- 172.21.15.83:0/750858353 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe69015bdd0 0x7fe6900c5610 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe6800097a0 tx=0x7fe68002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c01bc7cdbf493f72 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.875+0000 7fe68dd9b640 1 -- 172.21.15.83:0/750858353 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe680035020 con 0x7fe69015bdd0 2024-03-06T07:22:10.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.875+0000 7fe68dd9b640 1 -- 172.21.15.83:0/750858353 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe680031830 con 0x7fe69015bdd0 2024-03-06T07:22:10.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.875+0000 7fe68dd9b640 1 -- 172.21.15.83:0/750858353 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe680031c00 con 0x7fe69015bdd0 2024-03-06T07:22:10.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.875+0000 7fe6949c7640 1 -- 172.21.15.83:0/750858353 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe69015bdd0 msgr2=0x7fe6900c5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.875+0000 7fe6949c7640 1 --2- 172.21.15.83:0/750858353 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe69015bdd0 0x7fe6900c5610 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe6800097a0 tx=0x7fe68002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.875+0000 7fe6949c7640 1 -- 172.21.15.83:0/750858353 shutdown_connections 2024-03-06T07:22:10.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.875+0000 7fe6949c7640 1 --2- 172.21.15.83:0/750858353 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe69015bdd0 0x7fe6900c5610 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.875+0000 7fe6949c7640 1 -- 172.21.15.83:0/750858353 >> 172.21.15.83:0/750858353 conn(0x7fe69009a9f0 msgr2=0x7fe69009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.875+0000 7fe6949c7640 1 -- 172.21.15.83:0/750858353 shutdown_connections 2024-03-06T07:22:10.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.875+0000 7fe6949c7640 1 -- 172.21.15.83:0/750858353 wait complete. 2024-03-06T07:22:10.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.877+0000 7fe6949c7640 1 Processor -- start 2024-03-06T07:22:10.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.877+0000 7fe6949c7640 1 -- start start 2024-03-06T07:22:10.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.878+0000 7fe6949c7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6900d16e0 0x7fe6900c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.878+0000 7fe6949c7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe69015bdd0 0x7fe690154910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.878+0000 7fe6949c7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe69015fa10 0x7fe69015cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.878+0000 7fe6949c7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6900cc5d0 con 0x7fe6900d16e0 2024-03-06T07:22:10.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.878+0000 7fe6949c7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6900c5b50 con 0x7fe69015fa10 2024-03-06T07:22:10.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.878+0000 7fe6949c7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe69015c1b0 con 0x7fe69015bdd0 2024-03-06T07:22:10.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.878+0000 7fe68e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6900d16e0 0x7fe6900c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.878+0000 7fe68e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6900d16e0 0x7fe6900c8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49782/0 (socket says 172.21.15.83:49782) 2024-03-06T07:22:10.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.878+0000 7fe68e59c640 1 -- 172.21.15.83:0/1333063850 learned_addr learned my addr 172.21.15.83:0/1333063850 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:10.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.879+0000 7fe68ed9d640 1 --2- 172.21.15.83:0/1333063850 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe69015fa10 0x7fe69015cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.879+0000 7fe687fff640 1 --2- 172.21.15.83:0/1333063850 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe69015bdd0 0x7fe690154910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.879+0000 7fe68e59c640 1 -- 172.21.15.83:0/1333063850 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe69015bdd0 msgr2=0x7fe690154910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.879+0000 7fe68e59c640 1 --2- 172.21.15.83:0/1333063850 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe69015bdd0 0x7fe690154910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.879+0000 7fe68e59c640 1 -- 172.21.15.83:0/1333063850 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe69015fa10 msgr2=0x7fe69015cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.879+0000 7fe68e59c640 1 --2- 172.21.15.83:0/1333063850 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe69015fa10 0x7fe69015cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.879+0000 7fe68e59c640 1 -- 172.21.15.83:0/1333063850 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6900b5880 con 0x7fe6900d16e0 2024-03-06T07:22:10.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.879+0000 7fe68ed9d640 1 --2- 172.21.15.83:0/1333063850 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe69015fa10 0x7fe69015cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:10.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.879+0000 7fe68e59c640 1 --2- 172.21.15.83:0/1333063850 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6900d16e0 0x7fe6900c8b90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe6800098c0 tx=0x7fe6800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.879+0000 7fe686ffd640 1 -- 172.21.15.83:0/1333063850 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe680041070 con 0x7fe6900d16e0 2024-03-06T07:22:10.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.879+0000 7fe686ffd640 1 -- 172.21.15.83:0/1333063850 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe68003aa30 con 0x7fe6900d16e0 2024-03-06T07:22:10.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.879+0000 7fe6949c7640 1 -- 172.21.15.83:0/1333063850 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6900b5b60 con 0x7fe6900d16e0 2024-03-06T07:22:10.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.879+0000 7fe686ffd640 1 -- 172.21.15.83:0/1333063850 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe680035040 con 0x7fe6900d16e0 2024-03-06T07:22:10.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.879+0000 7fe6949c7640 1 -- 172.21.15.83:0/1333063850 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6900b6120 con 0x7fe6900d16e0 2024-03-06T07:22:10.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.880+0000 7fe6949c7640 1 -- 172.21.15.83:0/1333063850 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe69004f330 con 0x7fe6900d16e0 2024-03-06T07:22:10.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.880+0000 7fe686ffd640 1 -- 172.21.15.83:0/1333063850 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe68004c060 con 0x7fe6900d16e0 2024-03-06T07:22:10.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.881+0000 7fe686ffd640 1 --2- 172.21.15.83:0/1333063850 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe67006b4d0 0x7fe67006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.881+0000 7fe686ffd640 1 -- 172.21.15.83:0/1333063850 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe6800abc90 con 0x7fe6900d16e0 2024-03-06T07:22:10.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.881+0000 7fe687fff640 1 --2- 172.21.15.83:0/1333063850 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe67006b4d0 0x7fe67006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.881+0000 7fe687fff640 1 --2- 172.21.15.83:0/1333063850 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe67006b4d0 0x7fe67006d990 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe6901f5020 tx=0x7fe678009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:10.885+0000 7fe686ffd640 1 -- 172.21.15.83:0/1333063850 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe68007a550 con 0x7fe6900d16e0 2024-03-06T07:22:11.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.061+0000 7fe6949c7640 1 -- 172.21.15.83:0/1333063850 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe6900d1e40 con 0x7fe67006b4d0 2024-03-06T07:22:11.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.072+0000 7fe686ffd640 1 -- 172.21.15.83:0/1333063850 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251837 (secure 0 0 0) 0x7fe6900d1e40 con 0x7fe67006b4d0 2024-03-06T07:22:11.077 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:22:11.078 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-06T07:22:11.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.075+0000 7fe6949c7640 1 -- 172.21.15.83:0/1333063850 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe67006b4d0 msgr2=0x7fe67006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.075+0000 7fe6949c7640 1 --2- 172.21.15.83:0/1333063850 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe67006b4d0 0x7fe67006d990 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe6901f5020 tx=0x7fe678009040 comp rx=0 tx=0).stop 2024-03-06T07:22:11.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.075+0000 7fe6949c7640 1 -- 172.21.15.83:0/1333063850 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6900d16e0 msgr2=0x7fe6900c8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.075+0000 7fe6949c7640 1 --2- 172.21.15.83:0/1333063850 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6900d16e0 0x7fe6900c8b90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe6800098c0 tx=0x7fe6800046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.076+0000 7fe6949c7640 1 -- 172.21.15.83:0/1333063850 shutdown_connections 2024-03-06T07:22:11.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.076+0000 7fe6949c7640 1 --2- 172.21.15.83:0/1333063850 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe67006b4d0 0x7fe67006d990 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.076+0000 7fe6949c7640 1 --2- 172.21.15.83:0/1333063850 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe69015fa10 0x7fe69015cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.076+0000 7fe6949c7640 1 --2- 172.21.15.83:0/1333063850 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe69015bdd0 0x7fe690154910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.076+0000 7fe6949c7640 1 --2- 172.21.15.83:0/1333063850 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6900d16e0 0x7fe6900c8b90 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.076+0000 7fe6949c7640 1 -- 172.21.15.83:0/1333063850 >> 172.21.15.83:0/1333063850 conn(0x7fe69009a9f0 msgr2=0x7fe69006eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:11.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.076+0000 7fe6949c7640 1 -- 172.21.15.83:0/1333063850 shutdown_connections 2024-03-06T07:22:11.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.076+0000 7fe6949c7640 1 -- 172.21.15.83:0/1333063850 wait complete. 2024-03-06T07:22:11.088 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-03-06T07:22:10.323658+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219020,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755700,"statfs":{"total":773094113280,"available":772869836800,"internally_reserved":0,"allocated":2891776,"data_stored":1753028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12757,"internal_metadata":221236779},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.381376"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299166+0000","last_change":"2024-03-06T07:21:58.961070+0000","last_active":"2024-03-06T07:22:00.299166+0000","last_peered":"2024-03-06T07:22:00.299166+0000","last_clean":"2024-03-06T07:22:00.299166+0000","last_became_active":"2024-03-06T07:21:58.960784+0000","last_became_peered":"2024-03-06T07:21:58.960784+0000","last_unstale":"2024-03-06T07:22:00.299166+0000","last_undegraded":"2024-03-06T07:22:00.299166+0000","last_fullsized":"2024-03-06T07:22:00.299166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:06.413968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943975+0000","last_change":"2024-03-06T07:21:58.958978+0000","last_active":"2024-03-06T07:21:59.943975+0000","last_peered":"2024-03-06T07:21:59.943975+0000","last_clean":"2024-03-06T07:21:59.943975+0000","last_became_active":"2024-03-06T07:21:58.958728+0000","last_became_peered":"2024-03-06T07:21:58.958728+0000","last_unstale":"2024-03-06T07:21:59.943975+0000","last_undegraded":"2024-03-06T07:21:59.943975+0000","last_fullsized":"2024-03-06T07:21:59.943975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:54.495097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298515+0000","last_change":"2024-03-06T07:21:58.953968+0000","last_active":"2024-03-06T07:22:00.298515+0000","last_peered":"2024-03-06T07:22:00.298515+0000","last_clean":"2024-03-06T07:22:00.298515+0000","last_became_active":"2024-03-06T07:21:58.953678+0000","last_became_peered":"2024-03-06T07:21:58.953678+0000","last_unstale":"2024-03-06T07:22:00.298515+0000","last_undegraded":"2024-03-06T07:22:00.298515+0000","last_fullsized":"2024-03-06T07:22:00.298515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:51:39.557766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298069+0000","last_change":"2024-03-06T07:21:58.970682+0000","last_active":"2024-03-06T07:22:00.298069+0000","last_peered":"2024-03-06T07:22:00.298069+0000","last_clean":"2024-03-06T07:22:00.298069+0000","last_became_active":"2024-03-06T07:21:58.970397+0000","last_became_peered":"2024-03-06T07:21:58.970397+0000","last_unstale":"2024-03-06T07:22:00.298069+0000","last_undegraded":"2024-03-06T07:22:00.298069+0000","last_fullsized":"2024-03-06T07:22:00.298069+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:44.732365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298736+0000","last_change":"2024-03-06T07:21:58.965912+0000","last_active":"2024-03-06T07:22:00.298736+0000","last_peered":"2024-03-06T07:22:00.298736+0000","last_clean":"2024-03-06T07:22:00.298736+0000","last_became_active":"2024-03-06T07:21:58.965567+0000","last_became_peered":"2024-03-06T07:21:58.965567+0000","last_unstale":"2024-03-06T07:22:00.298736+0000","last_undegraded":"2024-03-06T07:22:00.298736+0000","last_fullsized":"2024-03-06T07:22:00.298736+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:10:08.320367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298064+0000","last_change":"2024-03-06T07:21:58.963244+0000","last_active":"2024-03-06T07:22:00.298064+0000","last_peered":"2024-03-06T07:22:00.298064+0000","last_clean":"2024-03-06T07:22:00.298064+0000","last_became_active":"2024-03-06T07:21:58.962903+0000","last_became_peered":"2024-03-06T07:21:58.962903+0000","last_unstale":"2024-03-06T07:22:00.298064+0000","last_undegraded":"2024-03-06T07:22:00.298064+0000","last_fullsized":"2024-03-06T07:22:00.298064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:30:54.760925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.297983+0000","last_change":"2024-03-06T07:21:58.964015+0000","last_active":"2024-03-06T07:22:00.297983+0000","last_peered":"2024-03-06T07:22:00.297983+0000","last_clean":"2024-03-06T07:22:00.297983+0000","last_became_active":"2024-03-06T07:21:58.962062+0000","last_became_peered":"2024-03-06T07:21:58.962062+0000","last_unstale":"2024-03-06T07:22:00.297983+0000","last_undegraded":"2024-03-06T07:22:00.297983+0000","last_fullsized":"2024-03-06T07:22:00.297983+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:44.732365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298608+0000","last_change":"2024-03-06T07:21:58.968307+0000","last_active":"2024-03-06T07:22:00.298608+0000","last_peered":"2024-03-06T07:22:00.298608+0000","last_clean":"2024-03-06T07:22:00.298608+0000","last_became_active":"2024-03-06T07:21:58.967917+0000","last_became_peered":"2024-03-06T07:21:58.967917+0000","last_unstale":"2024-03-06T07:22:00.298608+0000","last_undegraded":"2024-03-06T07:22:00.298608+0000","last_fullsized":"2024-03-06T07:22:00.298608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:21:17.637762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299130+0000","last_change":"2024-03-06T07:21:58.971497+0000","last_active":"2024-03-06T07:22:00.299130+0000","last_peered":"2024-03-06T07:22:00.299130+0000","last_clean":"2024-03-06T07:22:00.299130+0000","last_became_active":"2024-03-06T07:21:58.970710+0000","last_became_peered":"2024-03-06T07:21:58.970710+0000","last_unstale":"2024-03-06T07:22:00.299130+0000","last_undegraded":"2024-03-06T07:22:00.299130+0000","last_fullsized":"2024-03-06T07:22:00.299130+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:13.580324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943457+0000","last_change":"2024-03-06T07:21:58.956333+0000","last_active":"2024-03-06T07:21:59.943457+0000","last_peered":"2024-03-06T07:21:59.943457+0000","last_clean":"2024-03-06T07:21:59.943457+0000","last_became_active":"2024-03-06T07:21:58.956073+0000","last_became_peered":"2024-03-06T07:21:58.956073+0000","last_unstale":"2024-03-06T07:21:59.943457+0000","last_undegraded":"2024-03-06T07:21:59.943457+0000","last_fullsized":"2024-03-06T07:21:59.943457+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:30:54.760925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943172+0000","last_change":"2024-03-06T07:21:58.956210+0000","last_active":"2024-03-06T07:21:59.943172+0000","last_peered":"2024-03-06T07:21:59.943172+0000","last_clean":"2024-03-06T07:21:59.943172+0000","last_became_active":"2024-03-06T07:21:58.955949+0000","last_became_peered":"2024-03-06T07:21:58.955949+0000","last_unstale":"2024-03-06T07:21:59.943172+0000","last_undegraded":"2024-03-06T07:21:59.943172+0000","last_fullsized":"2024-03-06T07:21:59.943172+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:54.495097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943018+0000","last_change":"2024-03-06T07:21:58.950931+0000","last_active":"2024-03-06T07:21:59.943018+0000","last_peered":"2024-03-06T07:21:59.943018+0000","last_clean":"2024-03-06T07:21:59.943018+0000","last_became_active":"2024-03-06T07:21:58.949704+0000","last_became_peered":"2024-03-06T07:21:58.949704+0000","last_unstale":"2024-03-06T07:21:59.943018+0000","last_undegraded":"2024-03-06T07:21:59.943018+0000","last_fullsized":"2024-03-06T07:21:59.943018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:02.162943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299186+0000","last_change":"2024-03-06T07:21:58.956175+0000","last_active":"2024-03-06T07:22:00.299186+0000","last_peered":"2024-03-06T07:22:00.299186+0000","last_clean":"2024-03-06T07:22:00.299186+0000","last_became_active":"2024-03-06T07:21:58.955857+0000","last_became_peered":"2024-03-06T07:21:58.955857+0000","last_unstale":"2024-03-06T07:22:00.299186+0000","last_undegraded":"2024-03-06T07:22:00.299186+0000","last_fullsized":"2024-03-06T07:22:00.299186+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:44.732365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298168+0000","last_change":"2024-03-06T07:21:58.951176+0000","last_active":"2024-03-06T07:22:00.298168+0000","last_peered":"2024-03-06T07:22:00.298168+0000","last_clean":"2024-03-06T07:22:00.298168+0000","last_became_active":"2024-03-06T07:21:58.950874+0000","last_became_peered":"2024-03-06T07:21:58.950874+0000","last_unstale":"2024-03-06T07:22:00.298168+0000","last_undegraded":"2024-03-06T07:22:00.298168+0000","last_fullsized":"2024-03-06T07:22:00.298168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:42:14.052025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943706+0000","last_change":"2024-03-06T07:21:58.954562+0000","last_active":"2024-03-06T07:21:59.943706+0000","last_peered":"2024-03-06T07:21:59.943706+0000","last_clean":"2024-03-06T07:21:59.943706+0000","last_became_active":"2024-03-06T07:21:58.954242+0000","last_became_peered":"2024-03-06T07:21:58.954242+0000","last_unstale":"2024-03-06T07:21:59.943706+0000","last_undegraded":"2024-03-06T07:21:59.943706+0000","last_fullsized":"2024-03-06T07:21:59.943706+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:22.038204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.297923+0000","last_change":"2024-03-06T07:21:58.963478+0000","last_active":"2024-03-06T07:22:00.297923+0000","last_peered":"2024-03-06T07:22:00.297923+0000","last_clean":"2024-03-06T07:22:00.297923+0000","last_became_active":"2024-03-06T07:21:58.962917+0000","last_became_peered":"2024-03-06T07:21:58.962917+0000","last_unstale":"2024-03-06T07:22:00.297923+0000","last_undegraded":"2024-03-06T07:22:00.297923+0000","last_fullsized":"2024-03-06T07:22:00.297923+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:54.495097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298958+0000","last_change":"2024-03-06T07:21:58.965994+0000","last_active":"2024-03-06T07:22:00.298958+0000","last_peered":"2024-03-06T07:22:00.298958+0000","last_clean":"2024-03-06T07:22:00.298958+0000","last_became_active":"2024-03-06T07:21:58.965655+0000","last_became_peered":"2024-03-06T07:21:58.965655+0000","last_unstale":"2024-03-06T07:22:00.298958+0000","last_undegraded":"2024-03-06T07:22:00.298958+0000","last_fullsized":"2024-03-06T07:22:00.298958+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:06.413968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298767+0000","last_change":"2024-03-06T07:21:58.955322+0000","last_active":"2024-03-06T07:22:00.298767+0000","last_peered":"2024-03-06T07:22:00.298767+0000","last_clean":"2024-03-06T07:22:00.298767+0000","last_became_active":"2024-03-06T07:21:58.955080+0000","last_became_peered":"2024-03-06T07:21:58.955080+0000","last_unstale":"2024-03-06T07:22:00.298767+0000","last_undegraded":"2024-03-06T07:22:00.298767+0000","last_fullsized":"2024-03-06T07:22:00.298767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:20.851108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298974+0000","last_change":"2024-03-06T07:21:58.966461+0000","last_active":"2024-03-06T07:22:00.298974+0000","last_peered":"2024-03-06T07:22:00.298974+0000","last_clean":"2024-03-06T07:22:00.298974+0000","last_became_active":"2024-03-06T07:21:58.966154+0000","last_became_peered":"2024-03-06T07:21:58.966154+0000","last_unstale":"2024-03-06T07:22:00.298974+0000","last_undegraded":"2024-03-06T07:22:00.298974+0000","last_fullsized":"2024-03-06T07:22:00.298974+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:37:30.965258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942831+0000","last_change":"2024-03-06T07:21:58.953916+0000","last_active":"2024-03-06T07:21:59.942831+0000","last_peered":"2024-03-06T07:21:59.942831+0000","last_clean":"2024-03-06T07:21:59.942831+0000","last_became_active":"2024-03-06T07:21:58.953623+0000","last_became_peered":"2024-03-06T07:21:58.953623+0000","last_unstale":"2024-03-06T07:21:59.942831+0000","last_undegraded":"2024-03-06T07:21:59.942831+0000","last_fullsized":"2024-03-06T07:21:59.942831+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:21:17.637762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299245+0000","last_change":"2024-03-06T07:21:58.955378+0000","last_active":"2024-03-06T07:22:00.299245+0000","last_peered":"2024-03-06T07:22:00.299245+0000","last_clean":"2024-03-06T07:22:00.299245+0000","last_became_active":"2024-03-06T07:21:58.955118+0000","last_became_peered":"2024-03-06T07:21:58.955118+0000","last_unstale":"2024-03-06T07:22:00.299245+0000","last_undegraded":"2024-03-06T07:22:00.299245+0000","last_fullsized":"2024-03-06T07:22:00.299245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:22.038204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298483+0000","last_change":"2024-03-06T07:21:58.963849+0000","last_active":"2024-03-06T07:22:00.298483+0000","last_peered":"2024-03-06T07:22:00.298483+0000","last_clean":"2024-03-06T07:22:00.298483+0000","last_became_active":"2024-03-06T07:21:58.963573+0000","last_became_peered":"2024-03-06T07:21:58.963573+0000","last_unstale":"2024-03-06T07:22:00.298483+0000","last_undegraded":"2024-03-06T07:22:00.298483+0000","last_fullsized":"2024-03-06T07:22:00.298483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:20.758138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299836+0000","last_change":"2024-03-06T07:21:58.962026+0000","last_active":"2024-03-06T07:22:00.299836+0000","last_peered":"2024-03-06T07:22:00.299836+0000","last_clean":"2024-03-06T07:22:00.299836+0000","last_became_active":"2024-03-06T07:21:58.961691+0000","last_became_peered":"2024-03-06T07:21:58.961691+0000","last_unstale":"2024-03-06T07:22:00.299836+0000","last_undegraded":"2024-03-06T07:22:00.299836+0000","last_fullsized":"2024-03-06T07:22:00.299836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:02:11.315323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298380+0000","last_change":"2024-03-06T07:21:58.970098+0000","last_active":"2024-03-06T07:22:00.298380+0000","last_peered":"2024-03-06T07:22:00.298380+0000","last_clean":"2024-03-06T07:22:00.298380+0000","last_became_active":"2024-03-06T07:21:58.969786+0000","last_became_peered":"2024-03-06T07:21:58.969786+0000","last_unstale":"2024-03-06T07:22:00.298380+0000","last_undegraded":"2024-03-06T07:22:00.298380+0000","last_fullsized":"2024-03-06T07:22:00.298380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:07.060045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.297875+0000","last_change":"2024-03-06T07:21:58.961107+0000","last_active":"2024-03-06T07:22:00.297875+0000","last_peered":"2024-03-06T07:22:00.297875+0000","last_clean":"2024-03-06T07:22:00.297875+0000","last_became_active":"2024-03-06T07:21:58.960826+0000","last_became_peered":"2024-03-06T07:21:58.960826+0000","last_unstale":"2024-03-06T07:22:00.297875+0000","last_undegraded":"2024-03-06T07:22:00.297875+0000","last_fullsized":"2024-03-06T07:22:00.297875+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:36:02.122800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942422+0000","last_change":"2024-03-06T07:21:58.954994+0000","last_active":"2024-03-06T07:21:59.942422+0000","last_peered":"2024-03-06T07:21:59.942422+0000","last_clean":"2024-03-06T07:21:59.942422+0000","last_became_active":"2024-03-06T07:21:58.954710+0000","last_became_peered":"2024-03-06T07:21:58.954710+0000","last_unstale":"2024-03-06T07:21:59.942422+0000","last_undegraded":"2024-03-06T07:21:59.942422+0000","last_fullsized":"2024-03-06T07:21:59.942422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:12.378982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943938+0000","last_change":"2024-03-06T07:21:58.956241+0000","last_active":"2024-03-06T07:21:59.943938+0000","last_peered":"2024-03-06T07:21:59.943938+0000","last_clean":"2024-03-06T07:21:59.943938+0000","last_became_active":"2024-03-06T07:21:58.955977+0000","last_became_peered":"2024-03-06T07:21:58.955977+0000","last_unstale":"2024-03-06T07:21:59.943938+0000","last_undegraded":"2024-03-06T07:21:59.943938+0000","last_fullsized":"2024-03-06T07:21:59.943938+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:21:17.637762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298498+0000","last_change":"2024-03-06T07:21:58.963500+0000","last_active":"2024-03-06T07:22:00.298498+0000","last_peered":"2024-03-06T07:22:00.298498+0000","last_clean":"2024-03-06T07:22:00.298498+0000","last_became_active":"2024-03-06T07:21:58.963218+0000","last_became_peered":"2024-03-06T07:21:58.963218+0000","last_unstale":"2024-03-06T07:22:00.298498+0000","last_undegraded":"2024-03-06T07:22:00.298498+0000","last_fullsized":"2024-03-06T07:22:00.298498+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:12.378982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299375+0000","last_change":"2024-03-06T07:21:58.957981+0000","last_active":"2024-03-06T07:22:00.299375+0000","last_peered":"2024-03-06T07:22:00.299375+0000","last_clean":"2024-03-06T07:22:00.299375+0000","last_became_active":"2024-03-06T07:21:58.957690+0000","last_became_peered":"2024-03-06T07:21:58.957690+0000","last_unstale":"2024-03-06T07:22:00.299375+0000","last_undegraded":"2024-03-06T07:22:00.299375+0000","last_fullsized":"2024-03-06T07:22:00.299375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:59.261197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298443+0000","last_change":"2024-03-06T07:21:58.968221+0000","last_active":"2024-03-06T07:22:00.298443+0000","last_peered":"2024-03-06T07:22:00.298443+0000","last_clean":"2024-03-06T07:22:00.298443+0000","last_became_active":"2024-03-06T07:21:58.967798+0000","last_became_peered":"2024-03-06T07:21:58.967798+0000","last_unstale":"2024-03-06T07:22:00.298443+0000","last_undegraded":"2024-03-06T07:22:00.298443+0000","last_fullsized":"2024-03-06T07:22:00.298443+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:37:30.965258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298249+0000","last_change":"2024-03-06T07:21:58.966486+0000","last_active":"2024-03-06T07:22:00.298249+0000","last_peered":"2024-03-06T07:22:00.298249+0000","last_clean":"2024-03-06T07:22:00.298249+0000","last_became_active":"2024-03-06T07:21:58.966202+0000","last_became_peered":"2024-03-06T07:21:58.966202+0000","last_unstale":"2024-03-06T07:22:00.298249+0000","last_undegraded":"2024-03-06T07:22:00.298249+0000","last_fullsized":"2024-03-06T07:22:00.298249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:20.908786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942026+0000","last_change":"2024-03-06T07:21:58.952310+0000","last_active":"2024-03-06T07:21:59.942026+0000","last_peered":"2024-03-06T07:21:59.942026+0000","last_clean":"2024-03-06T07:21:59.942026+0000","last_became_active":"2024-03-06T07:21:58.952054+0000","last_became_peered":"2024-03-06T07:21:58.952054+0000","last_unstale":"2024-03-06T07:21:59.942026+0000","last_undegraded":"2024-03-06T07:21:59.942026+0000","last_fullsized":"2024-03-06T07:21:59.942026+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:55.294652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942908+0000","last_change":"2024-03-06T07:21:58.951201+0000","last_active":"2024-03-06T07:21:59.942908+0000","last_peered":"2024-03-06T07:21:59.942908+0000","last_clean":"2024-03-06T07:21:59.942908+0000","last_became_active":"2024-03-06T07:21:58.950746+0000","last_became_peered":"2024-03-06T07:21:58.950746+0000","last_unstale":"2024-03-06T07:21:59.942908+0000","last_undegraded":"2024-03-06T07:21:59.942908+0000","last_fullsized":"2024-03-06T07:21:59.942908+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:52.785061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299467+0000","last_change":"2024-03-06T07:21:57.947333+0000","last_active":"2024-03-06T07:22:00.299467+0000","last_peered":"2024-03-06T07:22:00.299467+0000","last_clean":"2024-03-06T07:22:00.299467+0000","last_became_active":"2024-03-06T07:21:57.947066+0000","last_became_peered":"2024-03-06T07:21:57.947066+0000","last_unstale":"2024-03-06T07:22:00.299467+0000","last_undegraded":"2024-03-06T07:22:00.299467+0000","last_fullsized":"2024-03-06T07:22:00.299467+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:18:20.077304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299505+0000","last_change":"2024-03-06T07:21:58.954116+0000","last_active":"2024-03-06T07:22:00.299505+0000","last_peered":"2024-03-06T07:22:00.299505+0000","last_clean":"2024-03-06T07:22:00.299505+0000","last_became_active":"2024-03-06T07:21:58.953858+0000","last_became_peered":"2024-03-06T07:21:58.953858+0000","last_unstale":"2024-03-06T07:22:00.299505+0000","last_undegraded":"2024-03-06T07:22:00.299505+0000","last_fullsized":"2024-03-06T07:22:00.299505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:12.378982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942428+0000","last_change":"2024-03-06T07:21:57.937765+0000","last_active":"2024-03-06T07:21:59.942428+0000","last_peered":"2024-03-06T07:21:59.942428+0000","last_clean":"2024-03-06T07:21:59.942428+0000","last_became_active":"2024-03-06T07:21:57.937476+0000","last_became_peered":"2024-03-06T07:21:57.937476+0000","last_unstale":"2024-03-06T07:21:59.942428+0000","last_undegraded":"2024-03-06T07:21:59.942428+0000","last_fullsized":"2024-03-06T07:21:59.942428+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:07:58.548928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299878+0000","last_change":"2024-03-06T07:21:57.943799+0000","last_active":"2024-03-06T07:22:00.299878+0000","last_peered":"2024-03-06T07:22:00.299878+0000","last_clean":"2024-03-06T07:22:00.299878+0000","last_became_active":"2024-03-06T07:21:57.943507+0000","last_became_peered":"2024-03-06T07:21:57.943507+0000","last_unstale":"2024-03-06T07:22:00.299878+0000","last_undegraded":"2024-03-06T07:22:00.299878+0000","last_fullsized":"2024-03-06T07:22:00.299878+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:31.179338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298759+0000","last_change":"2024-03-06T07:21:58.965398+0000","last_active":"2024-03-06T07:22:00.298759+0000","last_peered":"2024-03-06T07:22:00.298759+0000","last_clean":"2024-03-06T07:22:00.298759+0000","last_became_active":"2024-03-06T07:21:58.964367+0000","last_became_peered":"2024-03-06T07:21:58.964367+0000","last_unstale":"2024-03-06T07:22:00.298759+0000","last_undegraded":"2024-03-06T07:22:00.298759+0000","last_fullsized":"2024-03-06T07:22:00.298759+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:54.495097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298098+0000","last_change":"2024-03-06T07:21:57.951473+0000","last_active":"2024-03-06T07:22:00.298098+0000","last_peered":"2024-03-06T07:22:00.298098+0000","last_clean":"2024-03-06T07:22:00.298098+0000","last_became_active":"2024-03-06T07:21:57.951208+0000","last_became_peered":"2024-03-06T07:21:57.951208+0000","last_unstale":"2024-03-06T07:22:00.298098+0000","last_undegraded":"2024-03-06T07:22:00.298098+0000","last_fullsized":"2024-03-06T07:22:00.298098+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:51.781285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943705+0000","last_change":"2024-03-06T07:21:58.961873+0000","last_active":"2024-03-06T07:21:59.943705+0000","last_peered":"2024-03-06T07:21:59.943705+0000","last_clean":"2024-03-06T07:21:59.943705+0000","last_became_active":"2024-03-06T07:21:58.961630+0000","last_became_peered":"2024-03-06T07:21:58.961630+0000","last_unstale":"2024-03-06T07:21:59.943705+0000","last_undegraded":"2024-03-06T07:21:59.943705+0000","last_fullsized":"2024-03-06T07:21:59.943705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:55.294652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942284+0000","last_change":"2024-03-06T07:21:57.948765+0000","last_active":"2024-03-06T07:21:59.942284+0000","last_peered":"2024-03-06T07:21:59.942284+0000","last_clean":"2024-03-06T07:21:59.942284+0000","last_became_active":"2024-03-06T07:21:57.948498+0000","last_became_peered":"2024-03-06T07:21:57.948498+0000","last_unstale":"2024-03-06T07:21:59.942284+0000","last_undegraded":"2024-03-06T07:21:59.942284+0000","last_fullsized":"2024-03-06T07:21:59.942284+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:51.781285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298709+0000","last_change":"2024-03-06T07:21:58.971817+0000","last_active":"2024-03-06T07:22:00.298709+0000","last_peered":"2024-03-06T07:22:00.298709+0000","last_clean":"2024-03-06T07:22:00.298709+0000","last_became_active":"2024-03-06T07:21:58.971481+0000","last_became_peered":"2024-03-06T07:21:58.971481+0000","last_unstale":"2024-03-06T07:22:00.298709+0000","last_undegraded":"2024-03-06T07:22:00.298709+0000","last_fullsized":"2024-03-06T07:22:00.298709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:13.580324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298587+0000","last_change":"2024-03-06T07:21:57.950501+0000","last_active":"2024-03-06T07:22:00.298587+0000","last_peered":"2024-03-06T07:22:00.298587+0000","last_clean":"2024-03-06T07:22:00.298587+0000","last_became_active":"2024-03-06T07:21:57.950201+0000","last_became_peered":"2024-03-06T07:21:57.950201+0000","last_unstale":"2024-03-06T07:22:00.298587+0000","last_undegraded":"2024-03-06T07:22:00.298587+0000","last_fullsized":"2024-03-06T07:22:00.298587+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:13.736718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298791+0000","last_change":"2024-03-06T07:21:58.958089+0000","last_active":"2024-03-06T07:22:00.298791+0000","last_peered":"2024-03-06T07:22:00.298791+0000","last_clean":"2024-03-06T07:22:00.298791+0000","last_became_active":"2024-03-06T07:21:58.957814+0000","last_became_peered":"2024-03-06T07:21:58.957814+0000","last_unstale":"2024-03-06T07:22:00.298791+0000","last_undegraded":"2024-03-06T07:22:00.298791+0000","last_fullsized":"2024-03-06T07:22:00.298791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:55.294652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943941+0000","last_change":"2024-03-06T07:21:57.946124+0000","last_active":"2024-03-06T07:21:59.943941+0000","last_peered":"2024-03-06T07:21:59.943941+0000","last_clean":"2024-03-06T07:21:59.943941+0000","last_became_active":"2024-03-06T07:21:57.945856+0000","last_became_peered":"2024-03-06T07:21:57.945856+0000","last_unstale":"2024-03-06T07:21:59.943941+0000","last_undegraded":"2024-03-06T07:21:59.943941+0000","last_fullsized":"2024-03-06T07:21:59.943941+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:01.159166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943172+0000","last_change":"2024-03-06T07:21:58.958734+0000","last_active":"2024-03-06T07:21:59.943172+0000","last_peered":"2024-03-06T07:21:59.943172+0000","last_clean":"2024-03-06T07:21:59.943172+0000","last_became_active":"2024-03-06T07:21:58.958451+0000","last_became_peered":"2024-03-06T07:21:58.958451+0000","last_unstale":"2024-03-06T07:21:59.943172+0000","last_undegraded":"2024-03-06T07:21:59.943172+0000","last_fullsized":"2024-03-06T07:21:59.943172+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:20.908786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942962+0000","last_change":"2024-03-06T07:21:57.947199+0000","last_active":"2024-03-06T07:21:59.942962+0000","last_peered":"2024-03-06T07:21:59.942962+0000","last_clean":"2024-03-06T07:21:59.942962+0000","last_became_active":"2024-03-06T07:21:57.946927+0000","last_became_peered":"2024-03-06T07:21:57.946927+0000","last_unstale":"2024-03-06T07:21:59.942962+0000","last_undegraded":"2024-03-06T07:21:59.942962+0000","last_fullsized":"2024-03-06T07:21:59.942962+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:01.159166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298650+0000","last_change":"2024-03-06T07:21:58.970850+0000","last_active":"2024-03-06T07:22:00.298650+0000","last_peered":"2024-03-06T07:22:00.298650+0000","last_clean":"2024-03-06T07:22:00.298650+0000","last_became_active":"2024-03-06T07:21:58.970512+0000","last_became_peered":"2024-03-06T07:21:58.970512+0000","last_unstale":"2024-03-06T07:22:00.298650+0000","last_undegraded":"2024-03-06T07:22:00.298650+0000","last_fullsized":"2024-03-06T07:22:00.298650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:02:11.315323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299603+0000","last_change":"2024-03-06T07:21:57.951893+0000","last_active":"2024-03-06T07:22:00.299603+0000","last_peered":"2024-03-06T07:22:00.299603+0000","last_clean":"2024-03-06T07:22:00.299603+0000","last_became_active":"2024-03-06T07:21:57.951696+0000","last_became_peered":"2024-03-06T07:21:57.951696+0000","last_unstale":"2024-03-06T07:22:00.299603+0000","last_undegraded":"2024-03-06T07:22:00.299603+0000","last_fullsized":"2024-03-06T07:22:00.299603+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:27:31.864292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298310+0000","last_change":"2024-03-06T07:21:58.954594+0000","last_active":"2024-03-06T07:22:00.298310+0000","last_peered":"2024-03-06T07:22:00.298310+0000","last_clean":"2024-03-06T07:22:00.298310+0000","last_became_active":"2024-03-06T07:21:58.954326+0000","last_became_peered":"2024-03-06T07:21:58.954326+0000","last_unstale":"2024-03-06T07:22:00.298310+0000","last_undegraded":"2024-03-06T07:22:00.298310+0000","last_fullsized":"2024-03-06T07:22:00.298310+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:06:11.334051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298716+0000","last_change":"2024-03-06T07:21:57.956339+0000","last_active":"2024-03-06T07:22:00.298716+0000","last_peered":"2024-03-06T07:22:00.298716+0000","last_clean":"2024-03-06T07:22:00.298716+0000","last_became_active":"2024-03-06T07:21:57.956135+0000","last_became_peered":"2024-03-06T07:21:57.956135+0000","last_unstale":"2024-03-06T07:22:00.298716+0000","last_undegraded":"2024-03-06T07:22:00.298716+0000","last_fullsized":"2024-03-06T07:22:00.298716+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:01.159166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299011+0000","last_change":"2024-03-06T07:21:58.962391+0000","last_active":"2024-03-06T07:22:00.299011+0000","last_peered":"2024-03-06T07:22:00.299011+0000","last_clean":"2024-03-06T07:22:00.299011+0000","last_became_active":"2024-03-06T07:21:58.961887+0000","last_became_peered":"2024-03-06T07:21:58.961887+0000","last_unstale":"2024-03-06T07:22:00.299011+0000","last_undegraded":"2024-03-06T07:22:00.299011+0000","last_fullsized":"2024-03-06T07:22:00.299011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:20.758138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942138+0000","last_change":"2024-03-06T07:21:57.939474+0000","last_active":"2024-03-06T07:21:59.942138+0000","last_peered":"2024-03-06T07:21:59.942138+0000","last_clean":"2024-03-06T07:21:59.942138+0000","last_became_active":"2024-03-06T07:21:57.939053+0000","last_became_peered":"2024-03-06T07:21:57.939053+0000","last_unstale":"2024-03-06T07:21:59.942138+0000","last_undegraded":"2024-03-06T07:21:59.942138+0000","last_fullsized":"2024-03-06T07:21:59.942138+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:35.501272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942064+0000","last_change":"2024-03-06T07:21:58.958835+0000","last_active":"2024-03-06T07:21:59.942064+0000","last_peered":"2024-03-06T07:21:59.942064+0000","last_clean":"2024-03-06T07:21:59.942064+0000","last_became_active":"2024-03-06T07:21:58.958518+0000","last_became_peered":"2024-03-06T07:21:58.958518+0000","last_unstale":"2024-03-06T07:21:59.942064+0000","last_undegraded":"2024-03-06T07:21:59.942064+0000","last_fullsized":"2024-03-06T07:21:59.942064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:22.038204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298611+0000","last_change":"2024-03-06T07:21:57.940312+0000","last_active":"2024-03-06T07:22:00.298611+0000","last_peered":"2024-03-06T07:22:00.298611+0000","last_clean":"2024-03-06T07:22:00.298611+0000","last_became_active":"2024-03-06T07:21:57.940030+0000","last_became_peered":"2024-03-06T07:21:57.940030+0000","last_unstale":"2024-03-06T07:22:00.298611+0000","last_undegraded":"2024-03-06T07:22:00.298611+0000","last_fullsized":"2024-03-06T07:22:00.298611+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:35.501272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298350+0000","last_change":"2024-03-06T07:21:58.961737+0000","last_active":"2024-03-06T07:22:00.298350+0000","last_peered":"2024-03-06T07:22:00.298350+0000","last_clean":"2024-03-06T07:22:00.298350+0000","last_became_active":"2024-03-06T07:21:58.961365+0000","last_became_peered":"2024-03-06T07:21:58.961365+0000","last_unstale":"2024-03-06T07:22:00.298350+0000","last_undegraded":"2024-03-06T07:22:00.298350+0000","last_fullsized":"2024-03-06T07:22:00.298350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:44.889543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943032+0000","last_change":"2024-03-06T07:21:57.948442+0000","last_active":"2024-03-06T07:21:59.943032+0000","last_peered":"2024-03-06T07:21:59.943032+0000","last_clean":"2024-03-06T07:21:59.943032+0000","last_became_active":"2024-03-06T07:21:57.948170+0000","last_became_peered":"2024-03-06T07:21:57.948170+0000","last_unstale":"2024-03-06T07:21:59.943032+0000","last_undegraded":"2024-03-06T07:21:59.943032+0000","last_fullsized":"2024-03-06T07:21:59.943032+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:10.622090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298979+0000","last_change":"2024-03-06T07:21:58.961595+0000","last_active":"2024-03-06T07:22:00.298979+0000","last_peered":"2024-03-06T07:22:00.298979+0000","last_clean":"2024-03-06T07:22:00.298979+0000","last_became_active":"2024-03-06T07:21:58.961259+0000","last_became_peered":"2024-03-06T07:21:58.961259+0000","last_unstale":"2024-03-06T07:22:00.298979+0000","last_undegraded":"2024-03-06T07:22:00.298979+0000","last_fullsized":"2024-03-06T07:22:00.298979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:43.260641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298869+0000","last_change":"2024-03-06T07:21:57.942300+0000","last_active":"2024-03-06T07:22:00.298869+0000","last_peered":"2024-03-06T07:22:00.298869+0000","last_clean":"2024-03-06T07:22:00.298869+0000","last_became_active":"2024-03-06T07:21:57.942023+0000","last_became_peered":"2024-03-06T07:21:57.942023+0000","last_unstale":"2024-03-06T07:22:00.298869+0000","last_undegraded":"2024-03-06T07:22:00.298869+0000","last_fullsized":"2024-03-06T07:22:00.298869+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:07:58.548928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299225+0000","last_change":"2024-03-06T07:21:58.970895+0000","last_active":"2024-03-06T07:22:00.299225+0000","last_peered":"2024-03-06T07:22:00.299225+0000","last_clean":"2024-03-06T07:22:00.299225+0000","last_became_active":"2024-03-06T07:21:58.970418+0000","last_became_peered":"2024-03-06T07:21:58.970418+0000","last_unstale":"2024-03-06T07:22:00.299225+0000","last_undegraded":"2024-03-06T07:22:00.299225+0000","last_fullsized":"2024-03-06T07:22:00.299225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:19:55.013458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944014+0000","last_change":"2024-03-06T07:21:57.942325+0000","last_active":"2024-03-06T07:21:59.944014+0000","last_peered":"2024-03-06T07:21:59.944014+0000","last_clean":"2024-03-06T07:21:59.944014+0000","last_became_active":"2024-03-06T07:21:57.942045+0000","last_became_peered":"2024-03-06T07:21:57.942045+0000","last_unstale":"2024-03-06T07:21:59.944014+0000","last_undegraded":"2024-03-06T07:21:59.944014+0000","last_fullsized":"2024-03-06T07:21:59.944014+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:19.905010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298465+0000","last_change":"2024-03-06T07:21:58.967944+0000","last_active":"2024-03-06T07:22:00.298465+0000","last_peered":"2024-03-06T07:22:00.298465+0000","last_clean":"2024-03-06T07:22:00.298465+0000","last_became_active":"2024-03-06T07:21:58.967521+0000","last_became_peered":"2024-03-06T07:21:58.967521+0000","last_unstale":"2024-03-06T07:22:00.298465+0000","last_undegraded":"2024-03-06T07:22:00.298465+0000","last_fullsized":"2024-03-06T07:22:00.298465+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:43.260641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.297951+0000","last_change":"2024-03-06T07:21:57.947477+0000","last_active":"2024-03-06T07:22:00.297951+0000","last_peered":"2024-03-06T07:22:00.297951+0000","last_clean":"2024-03-06T07:22:00.297951+0000","last_became_active":"2024-03-06T07:21:57.946678+0000","last_became_peered":"2024-03-06T07:21:57.946678+0000","last_unstale":"2024-03-06T07:22:00.297951+0000","last_undegraded":"2024-03-06T07:22:00.297951+0000","last_fullsized":"2024-03-06T07:22:00.297951+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:13.736718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298718+0000","last_change":"2024-03-06T07:21:58.970473+0000","last_active":"2024-03-06T07:22:00.298718+0000","last_peered":"2024-03-06T07:22:00.298718+0000","last_clean":"2024-03-06T07:22:00.298718+0000","last_became_active":"2024-03-06T07:21:58.969810+0000","last_became_peered":"2024-03-06T07:21:58.969810+0000","last_unstale":"2024-03-06T07:22:00.298718+0000","last_undegraded":"2024-03-06T07:22:00.298718+0000","last_fullsized":"2024-03-06T07:22:00.298718+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:07.060045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298038+0000","last_change":"2024-03-06T07:21:47.553693+0000","last_active":"2024-03-06T07:22:00.298038+0000","last_peered":"2024-03-06T07:22:00.298038+0000","last_clean":"2024-03-06T07:22:00.298038+0000","last_became_active":"2024-03-06T07:21:47.552664+0000","last_became_peered":"2024-03-06T07:21:47.552664+0000","last_unstale":"2024-03-06T07:22:00.298038+0000","last_undegraded":"2024-03-06T07:22:00.298038+0000","last_fullsized":"2024-03-06T07:22:00.298038+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_clean_scrub_stamp":"2024-03-06T07:21:46.362028+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:02.916426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.297955+0000","last_change":"2024-03-06T07:21:57.949622+0000","last_active":"2024-03-06T07:22:00.297955+0000","last_peered":"2024-03-06T07:22:00.297955+0000","last_clean":"2024-03-06T07:22:00.297955+0000","last_became_active":"2024-03-06T07:21:57.948874+0000","last_became_peered":"2024-03-06T07:21:57.948874+0000","last_unstale":"2024-03-06T07:22:00.297955+0000","last_undegraded":"2024-03-06T07:22:00.297955+0000","last_fullsized":"2024-03-06T07:22:00.297955+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:35.501272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298026+0000","last_change":"2024-03-06T07:21:58.968667+0000","last_active":"2024-03-06T07:22:00.298026+0000","last_peered":"2024-03-06T07:22:00.298026+0000","last_clean":"2024-03-06T07:22:00.298026+0000","last_became_active":"2024-03-06T07:21:58.968213+0000","last_became_peered":"2024-03-06T07:21:58.968213+0000","last_unstale":"2024-03-06T07:22:00.298026+0000","last_undegraded":"2024-03-06T07:22:00.298026+0000","last_fullsized":"2024-03-06T07:22:00.298026+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:22.038204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.297808+0000","last_change":"2024-03-06T07:21:57.951697+0000","last_active":"2024-03-06T07:22:00.297808+0000","last_peered":"2024-03-06T07:22:00.297808+0000","last_clean":"2024-03-06T07:22:00.297808+0000","last_became_active":"2024-03-06T07:21:57.951421+0000","last_became_peered":"2024-03-06T07:21:57.951421+0000","last_unstale":"2024-03-06T07:22:00.297808+0000","last_undegraded":"2024-03-06T07:22:00.297808+0000","last_fullsized":"2024-03-06T07:22:00.297808+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:51.781285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.297721+0000","last_change":"2024-03-06T07:21:58.948562+0000","last_active":"2024-03-06T07:22:00.297721+0000","last_peered":"2024-03-06T07:22:00.297721+0000","last_clean":"2024-03-06T07:22:00.297721+0000","last_became_active":"2024-03-06T07:21:58.948249+0000","last_became_peered":"2024-03-06T07:21:58.948249+0000","last_unstale":"2024-03-06T07:22:00.297721+0000","last_undegraded":"2024-03-06T07:22:00.297721+0000","last_fullsized":"2024-03-06T07:22:00.297721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:53.360751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298409+0000","last_change":"2024-03-06T07:21:57.945890+0000","last_active":"2024-03-06T07:22:00.298409+0000","last_peered":"2024-03-06T07:22:00.298409+0000","last_clean":"2024-03-06T07:22:00.298409+0000","last_became_active":"2024-03-06T07:21:57.945527+0000","last_became_peered":"2024-03-06T07:21:57.945527+0000","last_unstale":"2024-03-06T07:22:00.298409+0000","last_undegraded":"2024-03-06T07:22:00.298409+0000","last_fullsized":"2024-03-06T07:22:00.298409+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:31.179338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943753+0000","last_change":"2024-03-06T07:21:58.952436+0000","last_active":"2024-03-06T07:21:59.943753+0000","last_peered":"2024-03-06T07:21:59.943753+0000","last_clean":"2024-03-06T07:21:59.943753+0000","last_became_active":"2024-03-06T07:21:58.952135+0000","last_became_peered":"2024-03-06T07:21:58.952135+0000","last_unstale":"2024-03-06T07:21:59.943753+0000","last_undegraded":"2024-03-06T07:21:59.943753+0000","last_fullsized":"2024-03-06T07:21:59.943753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:59.261197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298778+0000","last_change":"2024-03-06T07:21:57.951597+0000","last_active":"2024-03-06T07:22:00.298778+0000","last_peered":"2024-03-06T07:22:00.298778+0000","last_clean":"2024-03-06T07:22:00.298778+0000","last_became_active":"2024-03-06T07:21:57.951299+0000","last_became_peered":"2024-03-06T07:21:57.951299+0000","last_unstale":"2024-03-06T07:22:00.298778+0000","last_undegraded":"2024-03-06T07:22:00.298778+0000","last_fullsized":"2024-03-06T07:22:00.298778+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:19.847331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943217+0000","last_change":"2024-03-06T07:21:58.951633+0000","last_active":"2024-03-06T07:21:59.943217+0000","last_peered":"2024-03-06T07:21:59.943217+0000","last_clean":"2024-03-06T07:21:59.943217+0000","last_became_active":"2024-03-06T07:21:58.951315+0000","last_became_peered":"2024-03-06T07:21:58.951315+0000","last_unstale":"2024-03-06T07:21:59.943217+0000","last_undegraded":"2024-03-06T07:21:59.943217+0000","last_fullsized":"2024-03-06T07:21:59.943217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:20.851108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943826+0000","last_change":"2024-03-06T07:21:57.945253+0000","last_active":"2024-03-06T07:21:59.943826+0000","last_peered":"2024-03-06T07:21:59.943826+0000","last_clean":"2024-03-06T07:21:59.943826+0000","last_became_active":"2024-03-06T07:21:57.944983+0000","last_became_peered":"2024-03-06T07:21:57.944983+0000","last_unstale":"2024-03-06T07:21:59.943826+0000","last_undegraded":"2024-03-06T07:21:59.943826+0000","last_fullsized":"2024-03-06T07:21:59.943826+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:07:58.548928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298511+0000","last_change":"2024-03-06T07:21:58.964580+0000","last_active":"2024-03-06T07:22:00.298511+0000","last_peered":"2024-03-06T07:22:00.298511+0000","last_clean":"2024-03-06T07:22:00.298511+0000","last_became_active":"2024-03-06T07:21:58.964272+0000","last_became_peered":"2024-03-06T07:21:58.964272+0000","last_unstale":"2024-03-06T07:22:00.298511+0000","last_undegraded":"2024-03-06T07:22:00.298511+0000","last_fullsized":"2024-03-06T07:22:00.298511+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:02:11.315323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298902+0000","last_change":"2024-03-06T07:21:57.952261+0000","last_active":"2024-03-06T07:22:00.298902+0000","last_peered":"2024-03-06T07:22:00.298902+0000","last_clean":"2024-03-06T07:22:00.298902+0000","last_became_active":"2024-03-06T07:21:57.951991+0000","last_became_peered":"2024-03-06T07:21:57.951991+0000","last_unstale":"2024-03-06T07:22:00.298902+0000","last_undegraded":"2024-03-06T07:22:00.298902+0000","last_fullsized":"2024-03-06T07:22:00.298902+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:52.356974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943810+0000","last_change":"2024-03-06T07:21:58.955953+0000","last_active":"2024-03-06T07:21:59.943810+0000","last_peered":"2024-03-06T07:21:59.943810+0000","last_clean":"2024-03-06T07:21:59.943810+0000","last_became_active":"2024-03-06T07:21:58.955634+0000","last_became_peered":"2024-03-06T07:21:58.955634+0000","last_unstale":"2024-03-06T07:21:59.943810+0000","last_undegraded":"2024-03-06T07:21:59.943810+0000","last_fullsized":"2024-03-06T07:21:59.943810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:12.378982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.342768+0000","last_change":"2024-03-06T07:21:57.943165+0000","last_active":"2024-03-06T07:22:00.342768+0000","last_peered":"2024-03-06T07:22:00.342768+0000","last_clean":"2024-03-06T07:22:00.342768+0000","last_became_active":"2024-03-06T07:21:57.942914+0000","last_became_peered":"2024-03-06T07:21:57.942914+0000","last_unstale":"2024-03-06T07:22:00.342768+0000","last_undegraded":"2024-03-06T07:22:00.342768+0000","last_fullsized":"2024-03-06T07:22:00.342768+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:19.847331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298017+0000","last_change":"2024-03-06T07:21:58.960410+0000","last_active":"2024-03-06T07:22:00.298017+0000","last_peered":"2024-03-06T07:22:00.298017+0000","last_clean":"2024-03-06T07:22:00.298017+0000","last_became_active":"2024-03-06T07:21:58.960121+0000","last_became_peered":"2024-03-06T07:21:58.960121+0000","last_unstale":"2024-03-06T07:22:00.298017+0000","last_undegraded":"2024-03-06T07:22:00.298017+0000","last_fullsized":"2024-03-06T07:22:00.298017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:55.294652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942389+0000","last_change":"2024-03-06T07:21:57.944106+0000","last_active":"2024-03-06T07:21:59.942389+0000","last_peered":"2024-03-06T07:21:59.942389+0000","last_clean":"2024-03-06T07:21:59.942389+0000","last_became_active":"2024-03-06T07:21:57.943815+0000","last_became_peered":"2024-03-06T07:21:57.943815+0000","last_unstale":"2024-03-06T07:21:59.942389+0000","last_undegraded":"2024-03-06T07:21:59.942389+0000","last_fullsized":"2024-03-06T07:21:59.942389+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:13.736718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298012+0000","last_change":"2024-03-06T07:21:58.951840+0000","last_active":"2024-03-06T07:22:00.298012+0000","last_peered":"2024-03-06T07:22:00.298012+0000","last_clean":"2024-03-06T07:22:00.298012+0000","last_became_active":"2024-03-06T07:21:58.951594+0000","last_became_peered":"2024-03-06T07:21:58.951594+0000","last_unstale":"2024-03-06T07:22:00.298012+0000","last_undegraded":"2024-03-06T07:22:00.298012+0000","last_fullsized":"2024-03-06T07:22:00.298012+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:34.528149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.297795+0000","last_change":"2024-03-06T07:21:57.951916+0000","last_active":"2024-03-06T07:22:00.297795+0000","last_peered":"2024-03-06T07:22:00.297795+0000","last_clean":"2024-03-06T07:22:00.297795+0000","last_became_active":"2024-03-06T07:21:57.951647+0000","last_became_peered":"2024-03-06T07:21:57.951647+0000","last_unstale":"2024-03-06T07:22:00.297795+0000","last_undegraded":"2024-03-06T07:22:00.297795+0000","last_fullsized":"2024-03-06T07:22:00.297795+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:13.736718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942859+0000","last_change":"2024-03-06T07:21:58.953147+0000","last_active":"2024-03-06T07:21:59.942859+0000","last_peered":"2024-03-06T07:21:59.942859+0000","last_clean":"2024-03-06T07:21:59.942859+0000","last_became_active":"2024-03-06T07:21:58.952862+0000","last_became_peered":"2024-03-06T07:21:58.952862+0000","last_unstale":"2024-03-06T07:21:59.942859+0000","last_undegraded":"2024-03-06T07:21:59.942859+0000","last_fullsized":"2024-03-06T07:21:59.942859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:14.740494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943795+0000","last_change":"2024-03-06T07:21:57.948593+0000","last_active":"2024-03-06T07:21:59.943795+0000","last_peered":"2024-03-06T07:21:59.943795+0000","last_clean":"2024-03-06T07:21:59.943795+0000","last_became_active":"2024-03-06T07:21:57.948302+0000","last_became_peered":"2024-03-06T07:21:57.948302+0000","last_unstale":"2024-03-06T07:21:59.943795+0000","last_undegraded":"2024-03-06T07:21:59.943795+0000","last_fullsized":"2024-03-06T07:21:59.943795+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:13.736718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298923+0000","last_change":"2024-03-06T07:21:58.953065+0000","last_active":"2024-03-06T07:22:00.298923+0000","last_peered":"2024-03-06T07:22:00.298923+0000","last_clean":"2024-03-06T07:22:00.298923+0000","last_became_active":"2024-03-06T07:21:58.952790+0000","last_became_peered":"2024-03-06T07:21:58.952790+0000","last_unstale":"2024-03-06T07:22:00.298923+0000","last_undegraded":"2024-03-06T07:22:00.298923+0000","last_fullsized":"2024-03-06T07:22:00.298923+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:30:54.760925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943752+0000","last_change":"2024-03-06T07:21:57.945134+0000","last_active":"2024-03-06T07:21:59.943752+0000","last_peered":"2024-03-06T07:21:59.943752+0000","last_clean":"2024-03-06T07:21:59.943752+0000","last_became_active":"2024-03-06T07:21:57.944858+0000","last_became_peered":"2024-03-06T07:21:57.944858+0000","last_unstale":"2024-03-06T07:21:59.943752+0000","last_undegraded":"2024-03-06T07:21:59.943752+0000","last_fullsized":"2024-03-06T07:21:59.943752+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:51.781285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299349+0000","last_change":"2024-03-06T07:21:58.964930+0000","last_active":"2024-03-06T07:22:00.299349+0000","last_peered":"2024-03-06T07:22:00.299349+0000","last_clean":"2024-03-06T07:22:00.299349+0000","last_became_active":"2024-03-06T07:21:58.964474+0000","last_became_peered":"2024-03-06T07:21:58.964474+0000","last_unstale":"2024-03-06T07:22:00.299349+0000","last_undegraded":"2024-03-06T07:22:00.299349+0000","last_fullsized":"2024-03-06T07:22:00.299349+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:44.889543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299058+0000","last_change":"2024-03-06T07:21:57.953248+0000","last_active":"2024-03-06T07:22:00.299058+0000","last_peered":"2024-03-06T07:22:00.299058+0000","last_clean":"2024-03-06T07:22:00.299058+0000","last_became_active":"2024-03-06T07:21:57.952969+0000","last_became_peered":"2024-03-06T07:21:57.952969+0000","last_unstale":"2024-03-06T07:22:00.299058+0000","last_undegraded":"2024-03-06T07:22:00.299058+0000","last_fullsized":"2024-03-06T07:22:00.299058+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:19.905010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942240+0000","last_change":"2024-03-06T07:21:58.954261+0000","last_active":"2024-03-06T07:21:59.942240+0000","last_peered":"2024-03-06T07:21:59.942240+0000","last_clean":"2024-03-06T07:21:59.942240+0000","last_became_active":"2024-03-06T07:21:58.953584+0000","last_became_peered":"2024-03-06T07:21:58.953584+0000","last_unstale":"2024-03-06T07:21:59.942240+0000","last_undegraded":"2024-03-06T07:21:59.942240+0000","last_fullsized":"2024-03-06T07:21:59.942240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:59.261197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298322+0000","last_change":"2024-03-06T07:21:58.959725+0000","last_active":"2024-03-06T07:22:00.298322+0000","last_peered":"2024-03-06T07:22:00.298322+0000","last_clean":"2024-03-06T07:22:00.298322+0000","last_became_active":"2024-03-06T07:21:58.959078+0000","last_became_peered":"2024-03-06T07:21:58.959078+0000","last_unstale":"2024-03-06T07:22:00.298322+0000","last_undegraded":"2024-03-06T07:22:00.298322+0000","last_fullsized":"2024-03-06T07:22:00.298322+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:44.889543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.299074+0000","last_change":"2024-03-06T07:21:57.951758+0000","last_active":"2024-03-06T07:22:00.299074+0000","last_peered":"2024-03-06T07:22:00.299074+0000","last_clean":"2024-03-06T07:22:00.299074+0000","last_became_active":"2024-03-06T07:21:57.951452+0000","last_became_peered":"2024-03-06T07:21:57.951452+0000","last_unstale":"2024-03-06T07:22:00.299074+0000","last_undegraded":"2024-03-06T07:22:00.299074+0000","last_fullsized":"2024-03-06T07:22:00.299074+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:01.159166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298378+0000","last_change":"2024-03-06T07:21:58.961541+0000","last_active":"2024-03-06T07:22:00.298378+0000","last_peered":"2024-03-06T07:22:00.298378+0000","last_clean":"2024-03-06T07:22:00.298378+0000","last_became_active":"2024-03-06T07:21:58.961271+0000","last_became_peered":"2024-03-06T07:21:58.961271+0000","last_unstale":"2024-03-06T07:22:00.298378+0000","last_undegraded":"2024-03-06T07:22:00.298378+0000","last_fullsized":"2024-03-06T07:22:00.298378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:10:08.320367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298788+0000","last_change":"2024-03-06T07:21:57.947161+0000","last_active":"2024-03-06T07:22:00.298788+0000","last_peered":"2024-03-06T07:22:00.298788+0000","last_clean":"2024-03-06T07:22:00.298788+0000","last_became_active":"2024-03-06T07:21:57.946849+0000","last_became_peered":"2024-03-06T07:21:57.946849+0000","last_unstale":"2024-03-06T07:22:00.298788+0000","last_undegraded":"2024-03-06T07:22:00.298788+0000","last_fullsized":"2024-03-06T07:22:00.298788+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:19.847331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298661+0000","last_change":"2024-03-06T07:21:58.958814+0000","last_active":"2024-03-06T07:22:00.298661+0000","last_peered":"2024-03-06T07:22:00.298661+0000","last_clean":"2024-03-06T07:22:00.298661+0000","last_became_active":"2024-03-06T07:21:58.958550+0000","last_became_peered":"2024-03-06T07:21:58.958550+0000","last_unstale":"2024-03-06T07:22:00.298661+0000","last_undegraded":"2024-03-06T07:22:00.298661+0000","last_fullsized":"2024-03-06T07:22:00.298661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:36:02.122800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298388+0000","last_change":"2024-03-06T07:21:57.949031+0000","last_active":"2024-03-06T07:22:00.298388+0000","last_peered":"2024-03-06T07:22:00.298388+0000","last_clean":"2024-03-06T07:22:00.298388+0000","last_became_active":"2024-03-06T07:21:57.948741+0000","last_became_peered":"2024-03-06T07:21:57.948741+0000","last_unstale":"2024-03-06T07:22:00.298388+0000","last_undegraded":"2024-03-06T07:22:00.298388+0000","last_fullsized":"2024-03-06T07:22:00.298388+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:35.501272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298230+0000","last_change":"2024-03-06T07:21:58.957355+0000","last_active":"2024-03-06T07:22:00.298230+0000","last_peered":"2024-03-06T07:22:00.298230+0000","last_clean":"2024-03-06T07:22:00.298230+0000","last_became_active":"2024-03-06T07:21:58.956672+0000","last_became_peered":"2024-03-06T07:21:58.956672+0000","last_unstale":"2024-03-06T07:22:00.298230+0000","last_undegraded":"2024-03-06T07:22:00.298230+0000","last_fullsized":"2024-03-06T07:22:00.298230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.927774+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.927774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:59.261197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.298955+0000","last_change":"2024-03-06T07:21:57.951293+0000","last_active":"2024-03-06T07:22:00.298955+0000","last_peered":"2024-03-06T07:22:00.298955+0000","last_clean":"2024-03-06T07:22:00.298955+0000","last_became_active":"2024-03-06T07:21:57.950995+0000","last_became_peered":"2024-03-06T07:21:57.950995+0000","last_unstale":"2024-03-06T07:22:00.298955+0000","last_undegraded":"2024-03-06T07:22:00.298955+0000","last_fullsized":"2024-03-06T07:22:00.298955+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.923997+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.923997+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:10.622090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:11.092 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-06T07:22:11.092 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-06T07:22:11.092 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-06T07:22:11.092 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-06T07:22:11.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.227+0000 7fa8b1c88640 1 Processor -- start 2024-03-06T07:22:11.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.227+0000 7fa8b1c88640 1 -- start start 2024-03-06T07:22:11.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.228+0000 7fa8b1c88640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa8ac0b4c00 0x7fa8ac0b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.228+0000 7fa8b1c88640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8ac0b28a0 0x7fa8ac0b2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.228+0000 7fa8b1c88640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8ac15d510 con 0x7fa8ac15e540 2024-03-06T07:22:11.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.228+0000 7fa8b1c88640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8ac15e0b0 con 0x7fa8ac0b4c00 2024-03-06T07:22:11.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.228+0000 7fa8b1c88640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ac15d780 con 0x7fa8ac0b28a0 2024-03-06T07:22:11.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.228+0000 7fa8ab7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa8ac0b4c00 0x7fa8ac0b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.228+0000 7fa8abfff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fa8ac15e540 0x7fa8ac15e980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:42978/0 (socket says 172.21.15.83:42978) 2024-03-06T07:22:11.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.228+0000 7fa8ab7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa8ac0b4c00 0x7fa8ac0b4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49786/0 (socket says 172.21.15.83:49786) 2024-03-06T07:22:11.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.228+0000 7fa8abfff640 1 -- 172.21.15.83:0/195596534 learned_addr learned my addr 172.21.15.83:0/195596534 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:11.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.228+0000 7fa8aaffd640 1 --2- 172.21.15.83:0/195596534 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8ac0b28a0 0x7fa8ac0b2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.228+0000 7fa8ab7fe640 1 -- 172.21.15.83:0/195596534 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8ac0b28a0 msgr2=0x7fa8ac0b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:11.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.228+0000 7fa8ab7fe640 1 --2- 172.21.15.83:0/195596534 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8ac0b28a0 0x7fa8ac0b2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.228+0000 7fa8ab7fe640 1 -- 172.21.15.83:0/195596534 >> v1:172.21.15.83:6789/0 conn(0x7fa8ac15e540 legacy=0x7fa8ac15e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.228+0000 7fa8ab7fe640 1 -- 172.21.15.83:0/195596534 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8ac15da60 con 0x7fa8ac0b4c00 2024-03-06T07:22:11.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.228+0000 7fa8aa7fc640 1 -- 172.21.15.83:0/195596534 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2397989792 0 0) 0x7fa8ac15d510 con 0x7fa8ac15e540 2024-03-06T07:22:11.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.229+0000 7fa8ab7fe640 1 --2- 172.21.15.83:0/195596534 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8ac0b4c00 0x7fa8ac0b4fe0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa89c0097a0 tx=0x7fa89c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dce74a2db5d46e61 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.229+0000 7fa8aa7fc640 1 -- 172.21.15.83:0/195596534 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa89c034070 con 0x7fa8ac0b4c00 2024-03-06T07:22:11.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.229+0000 7fa8aa7fc640 1 -- 172.21.15.83:0/195596534 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa89c030610 con 0x7fa8ac0b4c00 2024-03-06T07:22:11.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.229+0000 7fa8aa7fc640 1 -- 172.21.15.83:0/195596534 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa89c0309e0 con 0x7fa8ac0b4c00 2024-03-06T07:22:11.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.230+0000 7fa8b1c88640 1 -- 172.21.15.83:0/195596534 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8ac0b4c00 msgr2=0x7fa8ac0b4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.230+0000 7fa8b1c88640 1 --2- 172.21.15.83:0/195596534 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8ac0b4c00 0x7fa8ac0b4fe0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa89c0097a0 tx=0x7fa89c02b960 comp rx=0 tx=0).stop 2024-03-06T07:22:11.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.230+0000 7fa8b1c88640 1 -- 172.21.15.83:0/195596534 shutdown_connections 2024-03-06T07:22:11.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.230+0000 7fa8b1c88640 1 --2- 172.21.15.83:0/195596534 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8ac0b28a0 0x7fa8ac0b2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.230+0000 7fa8b1c88640 1 --2- 172.21.15.83:0/195596534 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8ac0b4c00 0x7fa8ac0b4fe0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.230+0000 7fa8b1c88640 1 -- 172.21.15.83:0/195596534 >> 172.21.15.83:0/195596534 conn(0x7fa8ac09a9f0 msgr2=0x7fa8ac09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:11.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.230+0000 7fa8b1c88640 1 -- 172.21.15.83:0/195596534 shutdown_connections 2024-03-06T07:22:11.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.230+0000 7fa8b1c88640 1 -- 172.21.15.83:0/195596534 wait complete. 2024-03-06T07:22:11.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.231+0000 7fa8b1c88640 1 Processor -- start 2024-03-06T07:22:11.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.231+0000 7fa8b1c88640 1 -- start start 2024-03-06T07:22:11.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.233+0000 7fa8b1c88640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8ac0b28a0 0x7fa8ac09c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.233+0000 7fa8b1c88640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8ac0b4c00 0x7fa8ac151380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.233+0000 7fa8b1c88640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8ac15e540 0x7fa8ac1518c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.233+0000 7fa8b1c88640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ac0b33a0 con 0x7fa8ac0b4c00 2024-03-06T07:22:11.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.233+0000 7fa8b1c88640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ac0b3220 con 0x7fa8ac0b28a0 2024-03-06T07:22:11.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.233+0000 7fa8b1c88640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ac15f090 con 0x7fa8ac15e540 2024-03-06T07:22:11.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.233+0000 7fa8aaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8ac0b4c00 0x7fa8ac151380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.233+0000 7fa8aaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8ac0b4c00 0x7fa8ac151380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49802/0 (socket says 172.21.15.83:49802) 2024-03-06T07:22:11.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.233+0000 7fa8aaffd640 1 -- 172.21.15.83:0/2824531904 learned_addr learned my addr 172.21.15.83:0/2824531904 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:11.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.233+0000 7fa8ab7fe640 1 --2- 172.21.15.83:0/2824531904 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8ac0b28a0 0x7fa8ac09c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.233+0000 7fa8abfff640 1 --2- 172.21.15.83:0/2824531904 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8ac15e540 0x7fa8ac1518c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.233+0000 7fa8aaffd640 1 -- 172.21.15.83:0/2824531904 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8ac15e540 msgr2=0x7fa8ac1518c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.233+0000 7fa8aaffd640 1 --2- 172.21.15.83:0/2824531904 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8ac15e540 0x7fa8ac1518c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.233+0000 7fa8aaffd640 1 -- 172.21.15.83:0/2824531904 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8ac0b28a0 msgr2=0x7fa8ac09c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.233+0000 7fa8aaffd640 1 --2- 172.21.15.83:0/2824531904 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8ac0b28a0 0x7fa8ac09c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.233+0000 7fa8aaffd640 1 -- 172.21.15.83:0/2824531904 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8ac0b01b0 con 0x7fa8ac0b4c00 2024-03-06T07:22:11.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.233+0000 7fa8ab7fe640 1 --2- 172.21.15.83:0/2824531904 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8ac0b28a0 0x7fa8ac09c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:11.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.233+0000 7fa8abfff640 1 --2- 172.21.15.83:0/2824531904 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8ac15e540 0x7fa8ac1518c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:11.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.233+0000 7fa8aaffd640 1 --2- 172.21.15.83:0/2824531904 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8ac0b4c00 0x7fa8ac151380 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa8a000d800 tx=0x7fa8a000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.233+0000 7fa893fff640 1 -- 172.21.15.83:0/2824531904 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a000b6c0 con 0x7fa8ac0b4c00 2024-03-06T07:22:11.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.233+0000 7fa893fff640 1 -- 172.21.15.83:0/2824531904 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8a0019070 con 0x7fa8ac0b4c00 2024-03-06T07:22:11.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.233+0000 7fa893fff640 1 -- 172.21.15.83:0/2824531904 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a000ba80 con 0x7fa8ac0b4c00 2024-03-06T07:22:11.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.233+0000 7fa8b1c88640 1 -- 172.21.15.83:0/2824531904 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8ac09f410 con 0x7fa8ac0b4c00 2024-03-06T07:22:11.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.233+0000 7fa8b1c88640 1 -- 172.21.15.83:0/2824531904 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8ac09f9d0 con 0x7fa8ac0b4c00 2024-03-06T07:22:11.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.234+0000 7fa8b1c88640 1 -- 172.21.15.83:0/2824531904 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8ac04f330 con 0x7fa8ac0b4c00 2024-03-06T07:22:11.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.235+0000 7fa893fff640 1 -- 172.21.15.83:0/2824531904 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa8a000bc20 con 0x7fa8ac0b4c00 2024-03-06T07:22:11.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.235+0000 7fa893fff640 1 --2- 172.21.15.83:0/2824531904 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa89806b4d0 0x7fa89806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.236+0000 7fa8ab7fe640 1 --2- 172.21.15.83:0/2824531904 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa89806b4d0 0x7fa89806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.236+0000 7fa893fff640 1 -- 172.21.15.83:0/2824531904 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa8a008a620 con 0x7fa8ac0b4c00 2024-03-06T07:22:11.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.236+0000 7fa8ab7fe640 1 --2- 172.21.15.83:0/2824531904 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa89806b4d0 0x7fa89806d990 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa89c036000 tx=0x7fa89c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.240+0000 7fa893fff640 1 -- 172.21.15.83:0/2824531904 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa8a0059530 con 0x7fa8ac0b4c00 2024-03-06T07:22:11.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.452+0000 7fa8b1c88640 1 -- 172.21.15.83:0/2824531904 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa8ac0d2040 con 0x7fa8ac0b4c00 2024-03-06T07:22:11.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.453+0000 7fa893fff640 1 -- 172.21.15.83:0/2824531904 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fa8a0058e30 con 0x7fa8ac0b4c00 2024-03-06T07:22:11.454 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:22:11.455 INFO:teuthology.orchestra.run.smithi083.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-06T07:22:11.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.454+0000 7fa8b1c88640 1 -- 172.21.15.83:0/2824531904 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa89806b4d0 msgr2=0x7fa89806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.454+0000 7fa8b1c88640 1 --2- 172.21.15.83:0/2824531904 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa89806b4d0 0x7fa89806d990 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa89c036000 tx=0x7fa89c008030 comp rx=0 tx=0).stop 2024-03-06T07:22:11.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.454+0000 7fa8b1c88640 1 -- 172.21.15.83:0/2824531904 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8ac0b4c00 msgr2=0x7fa8ac151380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.454+0000 7fa8b1c88640 1 --2- 172.21.15.83:0/2824531904 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8ac0b4c00 0x7fa8ac151380 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa8a000d800 tx=0x7fa8a000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.454+0000 7fa8b1c88640 1 -- 172.21.15.83:0/2824531904 shutdown_connections 2024-03-06T07:22:11.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.454+0000 7fa8b1c88640 1 --2- 172.21.15.83:0/2824531904 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa89806b4d0 0x7fa89806d990 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.454+0000 7fa8b1c88640 1 --2- 172.21.15.83:0/2824531904 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8ac15e540 0x7fa8ac1518c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.454+0000 7fa8b1c88640 1 --2- 172.21.15.83:0/2824531904 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8ac0b4c00 0x7fa8ac151380 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.454+0000 7fa8b1c88640 1 --2- 172.21.15.83:0/2824531904 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8ac0b28a0 0x7fa8ac09c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.454+0000 7fa8b1c88640 1 -- 172.21.15.83:0/2824531904 >> 172.21.15.83:0/2824531904 conn(0x7fa8ac09a9f0 msgr2=0x7fa8ac159130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:11.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.455+0000 7fa8b1c88640 1 -- 172.21.15.83:0/2824531904 shutdown_connections 2024-03-06T07:22:11.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.455+0000 7fa8b1c88640 1 -- 172.21.15.83:0/2824531904 wait complete. 2024-03-06T07:22:11.467 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-06T07:22:11.467 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:11.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.565+0000 7f36ca9e2640 1 Processor -- start 2024-03-06T07:22:11.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.565+0000 7f36ca9e2640 1 -- start start 2024-03-06T07:22:11.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.566+0000 7f36ca9e2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f36c40b4170 0x7f36c40b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.566+0000 7f36ca9e2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f36c40d4760 0x7f36c40d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.566+0000 7f36ca9e2640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36c40d25d0 con 0x7f36c40b4dd0 2024-03-06T07:22:11.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.566+0000 7f36ca9e2640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36c40d4bf0 con 0x7f36c40b4170 2024-03-06T07:22:11.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.566+0000 7f36ca9e2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36c40ce9e0 con 0x7f36c40d4760 2024-03-06T07:22:11.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.566+0000 7f36c3fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f36c40b4170 0x7f36c40b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.566+0000 7f36c3fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f36c40b4170 0x7f36c40b6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49804/0 (socket says 172.21.15.83:49804) 2024-03-06T07:22:11.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.566+0000 7f36c3fff640 1 -- 172.21.15.83:0/2661489441 learned_addr learned my addr 172.21.15.83:0/2661489441 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:11.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.566+0000 7f36c37fe640 1 --2- 172.21.15.83:0/2661489441 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f36c40d4760 0x7f36c40d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.567+0000 7f36c3fff640 1 -- 172.21.15.83:0/2661489441 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f36c40d4760 msgr2=0x7f36c40d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:11.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.567+0000 7f36c3fff640 1 --2- 172.21.15.83:0/2661489441 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f36c40d4760 0x7f36c40d0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.567+0000 7f36c3fff640 1 -- 172.21.15.83:0/2661489441 >> v1:172.21.15.183:6789/0 conn(0x7f36c40b4dd0 legacy=0x7f36c40cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.567+0000 7f36c3fff640 1 -- 172.21.15.83:0/2661489441 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c40d2870 con 0x7f36c40b4170 2024-03-06T07:22:11.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.567+0000 7f36c37fe640 1 --2- 172.21.15.83:0/2661489441 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f36c40d4760 0x7f36c40d0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:11.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.567+0000 7f36c3fff640 1 --2- 172.21.15.83:0/2661489441 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f36c40b4170 0x7f36c40b6a90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f36b4009870 tx=0x7f36b402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aff5d6d3aa011551 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.567+0000 7f36c2ffd640 1 -- 172.21.15.83:0/2661489441 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b4034070 con 0x7f36c40b4170 2024-03-06T07:22:11.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.568+0000 7f36c2ffd640 1 -- 172.21.15.83:0/2661489441 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36b4030840 con 0x7f36c40b4170 2024-03-06T07:22:11.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.568+0000 7f36c2ffd640 1 -- 172.21.15.83:0/2661489441 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b4030c10 con 0x7f36c40b4170 2024-03-06T07:22:11.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.568+0000 7f36ca9e2640 1 -- 172.21.15.83:0/2661489441 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f36c40b4170 msgr2=0x7f36c40b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.568+0000 7f36ca9e2640 1 --2- 172.21.15.83:0/2661489441 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f36c40b4170 0x7f36c40b6a90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f36b4009870 tx=0x7f36b402b980 comp rx=0 tx=0).stop 2024-03-06T07:22:11.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.568+0000 7f36ca9e2640 1 -- 172.21.15.83:0/2661489441 shutdown_connections 2024-03-06T07:22:11.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.568+0000 7f36ca9e2640 1 --2- 172.21.15.83:0/2661489441 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f36c40d4760 0x7f36c40d0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.568+0000 7f36ca9e2640 1 --2- 172.21.15.83:0/2661489441 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f36c40b4170 0x7f36c40b6a90 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.568+0000 7f36ca9e2640 1 -- 172.21.15.83:0/2661489441 >> 172.21.15.83:0/2661489441 conn(0x7f36c40d3c40 msgr2=0x7f36c40d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:11.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.568+0000 7f36ca9e2640 1 -- 172.21.15.83:0/2661489441 shutdown_connections 2024-03-06T07:22:11.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.568+0000 7f36ca9e2640 1 -- 172.21.15.83:0/2661489441 wait complete. 2024-03-06T07:22:11.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.569+0000 7f36ca9e2640 1 Processor -- start 2024-03-06T07:22:11.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.569+0000 7f36ca9e2640 1 -- start start 2024-03-06T07:22:11.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.570+0000 7f36ca9e2640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f36c40b4170 0x7f36c4156820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.570+0000 7f36ca9e2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f36c40b4dd0 0x7f36c41f4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.570+0000 7f36ca9e2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f36c41f4560 0x7f36c40cbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.570+0000 7f36ca9e2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c40b47d0 con 0x7f36c41f4560 2024-03-06T07:22:11.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.570+0000 7f36ca9e2640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c40b4550 con 0x7f36c40b4170 2024-03-06T07:22:11.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.570+0000 7f36ca9e2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36c40b4950 con 0x7f36c40b4dd0 2024-03-06T07:22:11.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.570+0000 7f36c8f7f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f36c41f4560 0x7f36c40cbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.570+0000 7f36c8f7f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f36c41f4560 0x7f36c40cbcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49820/0 (socket says 172.21.15.83:49820) 2024-03-06T07:22:11.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.570+0000 7f36c8f7f640 1 -- 172.21.15.83:0/2258834856 learned_addr learned my addr 172.21.15.83:0/2258834856 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:11.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.570+0000 7f36c37fe640 1 --2- 172.21.15.83:0/2258834856 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f36c40b4dd0 0x7f36c41f4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.570+0000 7f36c3fff640 1 --2- 172.21.15.83:0/2258834856 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f36c40b4170 0x7f36c4156820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.571+0000 7f36c8f7f640 1 -- 172.21.15.83:0/2258834856 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f36c40b4dd0 msgr2=0x7f36c41f4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.571+0000 7f36c8f7f640 1 --2- 172.21.15.83:0/2258834856 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f36c40b4dd0 0x7f36c41f4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.571+0000 7f36c8f7f640 1 -- 172.21.15.83:0/2258834856 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f36c40b4170 msgr2=0x7f36c4156820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.571+0000 7f36c8f7f640 1 --2- 172.21.15.83:0/2258834856 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f36c40b4170 0x7f36c4156820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.571+0000 7f36c8f7f640 1 -- 172.21.15.83:0/2258834856 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c41596a0 con 0x7f36c41f4560 2024-03-06T07:22:11.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.571+0000 7f36c3fff640 1 --2- 172.21.15.83:0/2258834856 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f36c40b4170 0x7f36c4156820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:11.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.571+0000 7f36c8f7f640 1 --2- 172.21.15.83:0/2258834856 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f36c41f4560 0x7f36c40cbcf0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f36bc00c7a0 tx=0x7f36bc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.571+0000 7f36c0ff9640 1 -- 172.21.15.83:0/2258834856 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36bc003b70 con 0x7f36c41f4560 2024-03-06T07:22:11.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.571+0000 7f36c0ff9640 1 -- 172.21.15.83:0/2258834856 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36bc003d30 con 0x7f36c41f4560 2024-03-06T07:22:11.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.571+0000 7f36ca9e2640 1 -- 172.21.15.83:0/2258834856 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36c4159980 con 0x7f36c41f4560 2024-03-06T07:22:11.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.571+0000 7f36c0ff9640 1 -- 172.21.15.83:0/2258834856 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36bc00aa90 con 0x7f36c41f4560 2024-03-06T07:22:11.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.571+0000 7f36ca9e2640 1 -- 172.21.15.83:0/2258834856 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36c4157850 con 0x7f36c41f4560 2024-03-06T07:22:11.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.572+0000 7f36ca9e2640 1 -- 172.21.15.83:0/2258834856 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36c404f330 con 0x7f36c41f4560 2024-03-06T07:22:11.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.572+0000 7f36c0ff9640 1 -- 172.21.15.83:0/2258834856 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f36bc00ac30 con 0x7f36c41f4560 2024-03-06T07:22:11.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.573+0000 7f36c0ff9640 1 --2- 172.21.15.83:0/2258834856 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f36b006b500 0x7f36b006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.573+0000 7f36c0ff9640 1 -- 172.21.15.83:0/2258834856 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f36bc08a050 con 0x7f36c41f4560 2024-03-06T07:22:11.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.573+0000 7f36c3fff640 1 --2- 172.21.15.83:0/2258834856 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f36b006b500 0x7f36b006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.573+0000 7f36c3fff640 1 --2- 172.21.15.83:0/2258834856 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f36b006b500 0x7f36b006d9c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f36b4036000 tx=0x7f36b403b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.577+0000 7f36c0ff9640 1 -- 172.21.15.83:0/2258834856 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f36bc058ed0 con 0x7f36c41f4560 2024-03-06T07:22:11.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.755+0000 7f36ca9e2640 1 -- 172.21.15.83:0/2258834856 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36c40733b0 con 0x7f36c41f4560 2024-03-06T07:22:11.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.756+0000 7f36c0ff9640 1 -- 172.21.15.83:0/2258834856 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16617 (secure 0 0 0) 0x7f36bc0587d0 con 0x7f36c41f4560 2024-03-06T07:22:11.758 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:22:11.758 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":20,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:21:59.938137+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.318942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.254795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:11.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.758+0000 7f36ca9e2640 1 -- 172.21.15.83:0/2258834856 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f36b006b500 msgr2=0x7f36b006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.758+0000 7f36ca9e2640 1 --2- 172.21.15.83:0/2258834856 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f36b006b500 0x7f36b006d9c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f36b4036000 tx=0x7f36b403b040 comp rx=0 tx=0).stop 2024-03-06T07:22:11.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.758+0000 7f36ca9e2640 1 -- 172.21.15.83:0/2258834856 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f36c41f4560 msgr2=0x7f36c40cbcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.758+0000 7f36ca9e2640 1 --2- 172.21.15.83:0/2258834856 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f36c41f4560 0x7f36c40cbcf0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f36bc00c7a0 tx=0x7f36bc00cc60 comp rx=0 tx=0).stop 2024-03-06T07:22:11.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.758+0000 7f36ca9e2640 1 -- 172.21.15.83:0/2258834856 shutdown_connections 2024-03-06T07:22:11.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.758+0000 7f36ca9e2640 1 --2- 172.21.15.83:0/2258834856 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f36b006b500 0x7f36b006d9c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.758+0000 7f36ca9e2640 1 --2- 172.21.15.83:0/2258834856 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f36c41f4560 0x7f36c40cbcf0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.758+0000 7f36ca9e2640 1 --2- 172.21.15.83:0/2258834856 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f36c40b4dd0 0x7f36c41f4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.758+0000 7f36ca9e2640 1 --2- 172.21.15.83:0/2258834856 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f36c40b4170 0x7f36c4156820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.758+0000 7f36ca9e2640 1 -- 172.21.15.83:0/2258834856 >> 172.21.15.83:0/2258834856 conn(0x7f36c40d3c40 msgr2=0x7f36c40d4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:11.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.758+0000 7f36ca9e2640 1 -- 172.21.15.83:0/2258834856 shutdown_connections 2024-03-06T07:22:11.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.758+0000 7f36ca9e2640 1 -- 172.21.15.83:0/2258834856 wait complete. 2024-03-06T07:22:11.771 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:56.318942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:57.254795+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:11.771 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:11.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.873+0000 7ff634e8f640 1 Processor -- start 2024-03-06T07:22:11.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.873+0000 7ff634e8f640 1 -- start start 2024-03-06T07:22:11.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.874+0000 7ff634e8f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff6300d4860 0x7ff6300d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.874+0000 7ff634e8f640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ff6300b0e30 0x7ff6300b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.874+0000 7ff634e8f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6301618b0 con 0x7ff6301614d0 2024-03-06T07:22:11.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.874+0000 7ff634e8f640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6300d4cc0 con 0x7ff6300b0e30 2024-03-06T07:22:11.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.874+0000 7ff634e8f640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6300cf400 con 0x7ff6300d4860 2024-03-06T07:22:11.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.874+0000 7ff62e59c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff6301614d0 0x7ff6300d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:42982/0 (socket says 172.21.15.83:42982) 2024-03-06T07:22:11.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.874+0000 7ff62e59c640 1 -- 172.21.15.83:0/2143710106 learned_addr learned my addr 172.21.15.83:0/2143710106 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:11.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.874+0000 7ff627fff640 1 --2- 172.21.15.83:0/2143710106 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff6300d4860 0x7ff6300d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.874+0000 7ff62ed9d640 1 --2- 172.21.15.83:0/2143710106 >> v2:172.21.15.183:3300/0 conn(0x7ff6300b0e30 0x7ff6300b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.874+0000 7ff62dd9b640 1 -- 172.21.15.83:0/2143710106 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1622546058 0 0) 0x7ff6301618b0 con 0x7ff6301614d0 2024-03-06T07:22:11.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.874+0000 7ff62dd9b640 1 -- 172.21.15.83:0/2143710106 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff61c003520 con 0x7ff6301614d0 2024-03-06T07:22:11.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.874+0000 7ff62dd9b640 1 -- 172.21.15.83:0/2143710106 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3740608717 0 0) 0x7ff61c003520 con 0x7ff6301614d0 2024-03-06T07:22:11.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.874+0000 7ff62dd9b640 1 -- 172.21.15.83:0/2143710106 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff6300d4860 msgr2=0x7ff6300d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:11.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.874+0000 7ff62dd9b640 1 --2- 172.21.15.83:0/2143710106 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff6300d4860 0x7ff6300d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.874+0000 7ff62dd9b640 1 -- 172.21.15.83:0/2143710106 >> v2:172.21.15.183:3300/0 conn(0x7ff6300b0e30 msgr2=0x7ff6300b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:11.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.874+0000 7ff62dd9b640 1 --2- 172.21.15.83:0/2143710106 >> v2:172.21.15.183:3300/0 conn(0x7ff6300b0e30 0x7ff6300b12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.874+0000 7ff62dd9b640 1 -- 172.21.15.83:0/2143710106 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6300cf6a0 con 0x7ff6301614d0 2024-03-06T07:22:11.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.874+0000 7ff62dd9b640 1 -- 172.21.15.83:0/2143710106 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7ff620001670 con 0x7ff6301614d0 2024-03-06T07:22:11.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.874+0000 7ff627fff640 1 --2- 172.21.15.83:0/2143710106 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff6300d4860 0x7ff6300d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:11.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.874+0000 7ff62dd9b640 1 -- 172.21.15.83:0/2143710106 >> v1:172.21.15.83:6789/0 conn(0x7ff6301614d0 legacy=0x7ff6300d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.874+0000 7ff62dd9b640 1 --2- 172.21.15.83:0/2143710106 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff61c0039b0 0x7ff61c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.874+0000 7ff62dd9b640 1 --2- 172.21.15.83:0/2143710106 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff6300d4860 0x7ff61c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.875+0000 7ff62dd9b640 1 --2- 172.21.15.83:0/2143710106 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff61c004d40 0x7ff61c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.875+0000 7ff62dd9b640 1 -- 172.21.15.83:0/2143710106 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6300d4cc0 con 0x7ff61c004d40 2024-03-06T07:22:11.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.875+0000 7ff62dd9b640 1 -- 172.21.15.83:0/2143710106 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6300cf400 con 0x7ff61c0039b0 2024-03-06T07:22:11.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.875+0000 7ff62dd9b640 1 -- 172.21.15.83:0/2143710106 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff61c00b9c0 con 0x7ff6300d4860 2024-03-06T07:22:11.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.875+0000 7ff62ed9d640 1 --2- 172.21.15.83:0/2143710106 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff61c004d40 0x7ff61c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.875+0000 7ff62e59c640 1 --2- 172.21.15.83:0/2143710106 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff61c0039b0 0x7ff61c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.875+0000 7ff627fff640 1 --2- 172.21.15.83:0/2143710106 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff6300d4860 0x7ff61c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.875+0000 7ff62ed9d640 1 -- 172.21.15.83:0/2143710106 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff6300d4860 msgr2=0x7ff61c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:11.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.875+0000 7ff62ed9d640 1 --2- 172.21.15.83:0/2143710106 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff6300d4860 0x7ff61c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.875+0000 7ff62ed9d640 1 -- 172.21.15.83:0/2143710106 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff61c0039b0 msgr2=0x7ff61c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:11.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.875+0000 7ff62ed9d640 1 --2- 172.21.15.83:0/2143710106 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff61c0039b0 0x7ff61c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.875+0000 7ff62ed9d640 1 -- 172.21.15.83:0/2143710106 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff61c003520 con 0x7ff61c004d40 2024-03-06T07:22:11.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.876+0000 7ff62ed9d640 1 --2- 172.21.15.83:0/2143710106 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff61c004d40 0x7ff61c0051c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff628009fc0 tx=0x7ff61c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a1acf0843ef1b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.876+0000 7ff62dd9b640 1 -- 172.21.15.83:0/2143710106 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff628039070 con 0x7ff61c004d40 2024-03-06T07:22:11.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.876+0000 7ff62dd9b640 1 -- 172.21.15.83:0/2143710106 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff62802cc60 con 0x7ff61c004d40 2024-03-06T07:22:11.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.876+0000 7ff62dd9b640 1 -- 172.21.15.83:0/2143710106 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff628009c10 con 0x7ff61c004d40 2024-03-06T07:22:11.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.876+0000 7ff634e8f640 1 -- 172.21.15.83:0/2143710106 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff61c004d40 msgr2=0x7ff61c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.876+0000 7ff634e8f640 1 --2- 172.21.15.83:0/2143710106 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff61c004d40 0x7ff61c0051c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff628009fc0 tx=0x7ff61c005830 comp rx=0 tx=0).stop 2024-03-06T07:22:11.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.876+0000 7ff62ed9d640 1 -- 172.21.15.83:0/2143710106 reap_dead start 2024-03-06T07:22:11.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.876+0000 7ff634e8f640 1 -- 172.21.15.83:0/2143710106 shutdown_connections 2024-03-06T07:22:11.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.876+0000 7ff634e8f640 1 -- 172.21.15.83:0/2143710106 >> 172.21.15.83:0/2143710106 conn(0x7ff6300d3d40 msgr2=0x7ff6300d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:11.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.876+0000 7ff634e8f640 1 -- 172.21.15.83:0/2143710106 shutdown_connections 2024-03-06T07:22:11.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.876+0000 7ff634e8f640 1 -- 172.21.15.83:0/2143710106 wait complete. 2024-03-06T07:22:11.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.878+0000 7ff634e8f640 1 Processor -- start 2024-03-06T07:22:11.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.878+0000 7ff634e8f640 1 -- start start 2024-03-06T07:22:11.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.878+0000 7ff634e8f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff61c004d40 0x7ff63015bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.878+0000 7ff634e8f640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff630156380 0x7ff630156800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.878+0000 7ff634e8f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff630159450 0x7ff6301598d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.878+0000 7ff634e8f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6300b0910 con 0x7ff61c004d40 2024-03-06T07:22:11.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.878+0000 7ff634e8f640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6300cfc60 con 0x7ff630156380 2024-03-06T07:22:11.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.878+0000 7ff634e8f640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6300d2020 con 0x7ff630159450 2024-03-06T07:22:11.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.878+0000 7ff62e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff61c004d40 0x7ff63015bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.878+0000 7ff62e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff61c004d40 0x7ff63015bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49832/0 (socket says 172.21.15.83:49832) 2024-03-06T07:22:11.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.878+0000 7ff62e59c640 1 -- 172.21.15.83:0/1773108937 learned_addr learned my addr 172.21.15.83:0/1773108937 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:11.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.878+0000 7ff62ed9d640 1 --2- 172.21.15.83:0/1773108937 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff630159450 0x7ff6301598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.878+0000 7ff627fff640 1 --2- 172.21.15.83:0/1773108937 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff630156380 0x7ff630156800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.879+0000 7ff62e59c640 1 -- 172.21.15.83:0/1773108937 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff630159450 msgr2=0x7ff6301598d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.879+0000 7ff62e59c640 1 --2- 172.21.15.83:0/1773108937 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff630159450 0x7ff6301598d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.879+0000 7ff62e59c640 1 -- 172.21.15.83:0/1773108937 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff630156380 msgr2=0x7ff630156800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.879+0000 7ff62e59c640 1 --2- 172.21.15.83:0/1773108937 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff630156380 0x7ff630156800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.879+0000 7ff62e59c640 1 -- 172.21.15.83:0/1773108937 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff63015a210 con 0x7ff61c004d40 2024-03-06T07:22:11.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.879+0000 7ff62e59c640 1 --2- 172.21.15.83:0/1773108937 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff61c004d40 0x7ff63015bcc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff620007290 tx=0x7ff620007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.879+0000 7ff626ffd640 1 -- 172.21.15.83:0/1773108937 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff62000f9c0 con 0x7ff61c004d40 2024-03-06T07:22:11.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.879+0000 7ff626ffd640 1 -- 172.21.15.83:0/1773108937 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff620018070 con 0x7ff61c004d40 2024-03-06T07:22:11.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.879+0000 7ff634e8f640 1 -- 172.21.15.83:0/1773108937 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff63015a4f0 con 0x7ff61c004d40 2024-03-06T07:22:11.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.879+0000 7ff626ffd640 1 -- 172.21.15.83:0/1773108937 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff62000fdc0 con 0x7ff61c004d40 2024-03-06T07:22:11.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.879+0000 7ff634e8f640 1 -- 172.21.15.83:0/1773108937 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff630157960 con 0x7ff61c004d40 2024-03-06T07:22:11.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.880+0000 7ff626ffd640 1 -- 172.21.15.83:0/1773108937 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff62001d460 con 0x7ff61c004d40 2024-03-06T07:22:11.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.880+0000 7ff634e8f640 1 -- 172.21.15.83:0/1773108937 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff630157b30 con 0x7ff61c004d40 2024-03-06T07:22:11.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.881+0000 7ff626ffd640 1 --2- 172.21.15.83:0/1773108937 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff61006b500 0x7ff61006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.881+0000 7ff627fff640 1 --2- 172.21.15.83:0/1773108937 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff61006b500 0x7ff61006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.881+0000 7ff626ffd640 1 -- 172.21.15.83:0/1773108937 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff62008a100 con 0x7ff61c004d40 2024-03-06T07:22:11.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.881+0000 7ff627fff640 1 --2- 172.21.15.83:0/1773108937 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff61006b500 0x7ff61006d9c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff630156f40 tx=0x7ff618009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:11.885+0000 7ff626ffd640 1 -- 172.21.15.83:0/1773108937 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff6200589c0 con 0x7ff61c004d40 2024-03-06T07:22:12.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.061+0000 7ff634e8f640 1 -- 172.21.15.83:0/1773108937 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff6300c7040 con 0x7ff61c004d40 2024-03-06T07:22:12.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.061+0000 7ff626ffd640 1 -- 172.21.15.83:0/1773108937 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff62005ca60 con 0x7ff61c004d40 2024-03-06T07:22:12.063 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:22:12.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.064+0000 7ff634e8f640 1 -- 172.21.15.83:0/1773108937 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff61006b500 msgr2=0x7ff61006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.064+0000 7ff634e8f640 1 --2- 172.21.15.83:0/1773108937 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff61006b500 0x7ff61006d9c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff630156f40 tx=0x7ff618009190 comp rx=0 tx=0).stop 2024-03-06T07:22:12.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.064+0000 7ff634e8f640 1 -- 172.21.15.83:0/1773108937 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff61c004d40 msgr2=0x7ff63015bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.064+0000 7ff634e8f640 1 --2- 172.21.15.83:0/1773108937 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff61c004d40 0x7ff63015bcc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff620007290 tx=0x7ff620007750 comp rx=0 tx=0).stop 2024-03-06T07:22:12.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.064+0000 7ff634e8f640 1 -- 172.21.15.83:0/1773108937 shutdown_connections 2024-03-06T07:22:12.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.064+0000 7ff634e8f640 1 --2- 172.21.15.83:0/1773108937 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff61006b500 0x7ff61006d9c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.064+0000 7ff634e8f640 1 --2- 172.21.15.83:0/1773108937 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff630159450 0x7ff6301598d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.064+0000 7ff634e8f640 1 --2- 172.21.15.83:0/1773108937 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff630156380 0x7ff630156800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.064+0000 7ff634e8f640 1 --2- 172.21.15.83:0/1773108937 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff61c004d40 0x7ff63015bcc0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.064+0000 7ff634e8f640 1 -- 172.21.15.83:0/1773108937 >> 172.21.15.83:0/1773108937 conn(0x7ff6300d3d40 msgr2=0x7ff6300d23e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:12.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.064+0000 7ff634e8f640 1 -- 172.21.15.83:0/1773108937 shutdown_connections 2024-03-06T07:22:12.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.064+0000 7ff634e8f640 1 -- 172.21.15.83:0/1773108937 wait complete. 2024-03-06T07:22:12.076 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:12.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.181+0000 7f15380df640 1 Processor -- start 2024-03-06T07:22:12.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.181+0000 7f15380df640 1 -- start start 2024-03-06T07:22:12.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.182+0000 7f15380df640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f15300cea50 0x7f15300b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.182+0000 7f15380df640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f153006d030 con 0x7f153015def0 2024-03-06T07:22:12.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.182+0000 7f15380df640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f153006d310 con 0x7f153006cab0 2024-03-06T07:22:12.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.182+0000 7f15380df640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15300cf1c0 con 0x7f15300cea50 2024-03-06T07:22:12.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.182+0000 7f1535e7b640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f153015def0 0x7f15300ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:42996/0 (socket says 172.21.15.83:42996) 2024-03-06T07:22:12.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.182+0000 7f1535e7b640 1 -- 172.21.15.83:0/4160622390 learned_addr learned my addr 172.21.15.83:0/4160622390 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:12.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.182+0000 7f153567a640 1 --2- 172.21.15.83:0/4160622390 >> v2:172.21.15.183:3300/0 conn(0x7f15300cea50 0x7f15300b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.182+0000 7f1534e79640 1 -- 172.21.15.83:0/4160622390 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 47367334 0 0) 0x7f153006d030 con 0x7f153015def0 2024-03-06T07:22:12.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.182+0000 7f1534e79640 1 -- 172.21.15.83:0/4160622390 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f151c003520 con 0x7f153015def0 2024-03-06T07:22:12.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.182+0000 7f1534e79640 1 -- 172.21.15.83:0/4160622390 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2379337639 0 0) 0x7f151c003520 con 0x7f153015def0 2024-03-06T07:22:12.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.182+0000 7f1534e79640 1 -- 172.21.15.83:0/4160622390 >> v2:172.21.15.183:3300/0 conn(0x7f15300cea50 msgr2=0x7f15300b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:12.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.182+0000 7f1534e79640 1 --2- 172.21.15.83:0/4160622390 >> v2:172.21.15.183:3300/0 conn(0x7f15300cea50 0x7f15300b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.182+0000 7f1534e79640 1 -- 172.21.15.83:0/4160622390 >> v1:172.21.15.183:6789/0 conn(0x7f153006cab0 legacy=0x7f15300ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.182+0000 7f1534e79640 1 -- 172.21.15.83:0/4160622390 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15300cc110 con 0x7f153015def0 2024-03-06T07:22:12.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.183+0000 7f1534e79640 1 -- 172.21.15.83:0/4160622390 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f1520001670 con 0x7f153015def0 2024-03-06T07:22:12.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.183+0000 7f1534e79640 1 -- 172.21.15.83:0/4160622390 >> v1:172.21.15.83:6789/0 conn(0x7f153015def0 legacy=0x7f15300ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.183+0000 7f1534e79640 1 --2- 172.21.15.83:0/4160622390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f151c0039b0 0x7f151c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.183+0000 7f1534e79640 1 --2- 172.21.15.83:0/4160622390 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f151c004800 0x7f151c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.183+0000 7f1534e79640 1 --2- 172.21.15.83:0/4160622390 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f151c0051c0 0x7f151c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.183+0000 7f1534e79640 1 -- 172.21.15.83:0/4160622390 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151c00c030 con 0x7f151c0039b0 2024-03-06T07:22:12.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.183+0000 7f1534e79640 1 -- 172.21.15.83:0/4160622390 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151c00c1e0 con 0x7f151c0051c0 2024-03-06T07:22:12.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.183+0000 7f1534e79640 1 -- 172.21.15.83:0/4160622390 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f151c00c3e0 con 0x7f151c004800 2024-03-06T07:22:12.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.183+0000 7f1535e7b640 1 --2- 172.21.15.83:0/4160622390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f151c0039b0 0x7f151c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.183+0000 7f153667c640 1 --2- 172.21.15.83:0/4160622390 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f151c0051c0 0x7f151c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.183+0000 7f153567a640 1 --2- 172.21.15.83:0/4160622390 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f151c004800 0x7f151c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.183+0000 7f1535e7b640 1 -- 172.21.15.83:0/4160622390 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f151c004800 msgr2=0x7f151c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:12.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.183+0000 7f1535e7b640 1 --2- 172.21.15.83:0/4160622390 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f151c004800 0x7f151c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.183+0000 7f1535e7b640 1 -- 172.21.15.83:0/4160622390 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f151c0051c0 msgr2=0x7f151c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:12.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.183+0000 7f1535e7b640 1 --2- 172.21.15.83:0/4160622390 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f151c0051c0 0x7f151c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.183+0000 7f1535e7b640 1 -- 172.21.15.83:0/4160622390 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f151c003520 con 0x7f151c0039b0 2024-03-06T07:22:12.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.183+0000 7f153667c640 1 -- 172.21.15.83:0/4160622390 reap_dead start 2024-03-06T07:22:12.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.184+0000 7f1535e7b640 1 --2- 172.21.15.83:0/4160622390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f151c0039b0 0x7f151c003dd0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f152000c210 tx=0x7f151c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a47923fd8871d5ad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:12.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.184+0000 7f1534e79640 1 -- 172.21.15.83:0/4160622390 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1520007d30 con 0x7f151c0039b0 2024-03-06T07:22:12.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.184+0000 7f1534e79640 1 -- 172.21.15.83:0/4160622390 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1520030070 con 0x7f151c0039b0 2024-03-06T07:22:12.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.184+0000 7f1534e79640 1 -- 172.21.15.83:0/4160622390 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f152002cda0 con 0x7f151c0039b0 2024-03-06T07:22:12.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.184+0000 7f15380df640 1 -- 172.21.15.83:0/4160622390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f151c0039b0 msgr2=0x7f151c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.184+0000 7f15380df640 1 --2- 172.21.15.83:0/4160622390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f151c0039b0 0x7f151c003dd0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f152000c210 tx=0x7f151c005830 comp rx=0 tx=0).stop 2024-03-06T07:22:12.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.184+0000 7f15380df640 1 -- 172.21.15.83:0/4160622390 shutdown_connections 2024-03-06T07:22:12.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.184+0000 7f15380df640 1 --2- 172.21.15.83:0/4160622390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f151c0039b0 0x7f151c003dd0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.184+0000 7f15380df640 1 -- 172.21.15.83:0/4160622390 >> 172.21.15.83:0/4160622390 conn(0x7f15300b3690 msgr2=0x7f15300b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:12.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.185+0000 7f15380df640 1 -- 172.21.15.83:0/4160622390 shutdown_connections 2024-03-06T07:22:12.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.185+0000 7f15380df640 1 -- 172.21.15.83:0/4160622390 wait complete. 2024-03-06T07:22:12.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.186+0000 7f15380df640 1 Processor -- start 2024-03-06T07:22:12.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.186+0000 7f15380df640 1 -- start start 2024-03-06T07:22:12.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.186+0000 7f15380df640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f151c0039b0 0x7f1530153de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.186+0000 7f15380df640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f15300c75a0 0x7f15300c5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.186+0000 7f15380df640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f15300c5a80 0x7f15300c5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.186+0000 7f15380df640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f153015e2d0 con 0x7f151c0039b0 2024-03-06T07:22:12.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.186+0000 7f15380df640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15300cc760 con 0x7f15300c5a80 2024-03-06T07:22:12.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.186+0000 7f15380df640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f153015e450 con 0x7f15300c75a0 2024-03-06T07:22:12.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.186+0000 7f1535e7b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f151c0039b0 0x7f1530153de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.186+0000 7f1535e7b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f151c0039b0 0x7f1530153de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49856/0 (socket says 172.21.15.83:49856) 2024-03-06T07:22:12.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.186+0000 7f1535e7b640 1 -- 172.21.15.83:0/464201465 learned_addr learned my addr 172.21.15.83:0/464201465 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:12.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.187+0000 7f153567a640 1 --2- 172.21.15.83:0/464201465 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f15300c75a0 0x7f15300c5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.187+0000 7f153667c640 1 --2- 172.21.15.83:0/464201465 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f15300c5a80 0x7f15300c5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.187+0000 7f1535e7b640 1 -- 172.21.15.83:0/464201465 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f15300c75a0 msgr2=0x7f15300c5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.187+0000 7f1535e7b640 1 --2- 172.21.15.83:0/464201465 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f15300c75a0 0x7f15300c5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.187+0000 7f1535e7b640 1 -- 172.21.15.83:0/464201465 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f15300c5a80 msgr2=0x7f15300c5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.187+0000 7f1535e7b640 1 --2- 172.21.15.83:0/464201465 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f15300c5a80 0x7f15300c5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.187+0000 7f1535e7b640 1 -- 172.21.15.83:0/464201465 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15300b49d0 con 0x7f151c0039b0 2024-03-06T07:22:12.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.187+0000 7f153667c640 1 --2- 172.21.15.83:0/464201465 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f15300c5a80 0x7f15300c5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:12.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.187+0000 7f1535e7b640 1 --2- 172.21.15.83:0/464201465 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f151c0039b0 0x7f1530153de0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f152000c210 tx=0x7f152002e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:12.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.187+0000 7f15267fc640 1 -- 172.21.15.83:0/464201465 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1520004bf0 con 0x7f151c0039b0 2024-03-06T07:22:12.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.187+0000 7f15267fc640 1 -- 172.21.15.83:0/464201465 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1520004d90 con 0x7f151c0039b0 2024-03-06T07:22:12.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.187+0000 7f15380df640 1 -- 172.21.15.83:0/464201465 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15300b4cb0 con 0x7f151c0039b0 2024-03-06T07:22:12.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.187+0000 7f15267fc640 1 -- 172.21.15.83:0/464201465 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1520007270 con 0x7f151c0039b0 2024-03-06T07:22:12.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.187+0000 7f15380df640 1 -- 172.21.15.83:0/464201465 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f153015d030 con 0x7f151c0039b0 2024-03-06T07:22:12.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.188+0000 7f15380df640 1 -- 172.21.15.83:0/464201465 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f153009fcf0 con 0x7f151c0039b0 2024-03-06T07:22:12.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.189+0000 7f15267fc640 1 -- 172.21.15.83:0/464201465 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1520048020 con 0x7f151c0039b0 2024-03-06T07:22:12.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.189+0000 7f15267fc640 1 --2- 172.21.15.83:0/464201465 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f151406b500 0x7f151406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.189+0000 7f153567a640 1 --2- 172.21.15.83:0/464201465 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f151406b500 0x7f151406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.189+0000 7f15267fc640 1 -- 172.21.15.83:0/464201465 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f15200ab4d0 con 0x7f151c0039b0 2024-03-06T07:22:12.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.190+0000 7f153567a640 1 --2- 172.21.15.83:0/464201465 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f151406b500 0x7f151406d9c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1530070070 tx=0x7f152c009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:12.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.193+0000 7f15267fc640 1 -- 172.21.15.83:0/464201465 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f152007a500 con 0x7f151c0039b0 2024-03-06T07:22:12.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.364+0000 7f15380df640 1 -- 172.21.15.83:0/464201465 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f15300c8350 con 0x7f151c0039b0 2024-03-06T07:22:12.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.364+0000 7f15267fc640 1 -- 172.21.15.83:0/464201465 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f15200a9e20 con 0x7f151c0039b0 2024-03-06T07:22:12.367 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-03-06T07:22:12.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.366+0000 7f15380df640 1 -- 172.21.15.83:0/464201465 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f151406b500 msgr2=0x7f151406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.366+0000 7f15380df640 1 --2- 172.21.15.83:0/464201465 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f151406b500 0x7f151406d9c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1530070070 tx=0x7f152c009190 comp rx=0 tx=0).stop 2024-03-06T07:22:12.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.366+0000 7f15380df640 1 -- 172.21.15.83:0/464201465 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f151c0039b0 msgr2=0x7f1530153de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.366+0000 7f15380df640 1 --2- 172.21.15.83:0/464201465 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f151c0039b0 0x7f1530153de0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f152000c210 tx=0x7f152002e070 comp rx=0 tx=0).stop 2024-03-06T07:22:12.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.366+0000 7f15380df640 1 -- 172.21.15.83:0/464201465 shutdown_connections 2024-03-06T07:22:12.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.366+0000 7f15380df640 1 --2- 172.21.15.83:0/464201465 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f151406b500 0x7f151406d9c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.366+0000 7f15380df640 1 --2- 172.21.15.83:0/464201465 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f15300c5a80 0x7f15300c5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.366+0000 7f15380df640 1 --2- 172.21.15.83:0/464201465 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f15300c75a0 0x7f15300c5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.366+0000 7f15380df640 1 --2- 172.21.15.83:0/464201465 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f151c0039b0 0x7f1530153de0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.366+0000 7f15380df640 1 -- 172.21.15.83:0/464201465 >> 172.21.15.83:0/464201465 conn(0x7f15300b3690 msgr2=0x7f15300caba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:12.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.366+0000 7f15380df640 1 -- 172.21.15.83:0/464201465 shutdown_connections 2024-03-06T07:22:12.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.366+0000 7f15380df640 1 -- 172.21.15.83:0/464201465 wait complete. 2024-03-06T07:22:12.379 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:12.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.483+0000 7fef1fd7a640 1 Processor -- start 2024-03-06T07:22:12.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.484+0000 7fef1fd7a640 1 -- start start 2024-03-06T07:22:12.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.484+0000 7fef1fd7a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fef181586e0 0x7fef180aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.484+0000 7fef1fd7a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fef180d01a0 0x7fef180b33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.484+0000 7fef1fd7a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef18158ac0 con 0x7fef180aef80 2024-03-06T07:22:12.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.484+0000 7fef1fd7a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef1815b2a0 con 0x7fef181586e0 2024-03-06T07:22:12.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.484+0000 7fef1fd7a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef1815b540 con 0x7fef180d01a0 2024-03-06T07:22:12.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.485+0000 7fef1db16640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fef181586e0 0x7fef180aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.485+0000 7fef1db16640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fef181586e0 0x7fef180aea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:60316/0 (socket says 172.21.15.83:60316) 2024-03-06T07:22:12.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.485+0000 7fef1d315640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fef180aef80 0x7fef180cfa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:37996/0 (socket says 172.21.15.83:37996) 2024-03-06T07:22:12.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.485+0000 7fef1db16640 1 -- 172.21.15.83:0/3537672074 learned_addr learned my addr 172.21.15.83:0/3537672074 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:12.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.485+0000 7fef1e317640 1 --2- 172.21.15.83:0/3537672074 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fef180d01a0 0x7fef180b33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.485+0000 7fef1db16640 1 -- 172.21.15.83:0/3537672074 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fef180d01a0 msgr2=0x7fef180b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:12.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.485+0000 7fef1db16640 1 --2- 172.21.15.83:0/3537672074 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fef180d01a0 0x7fef180b33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.485+0000 7fef1db16640 1 -- 172.21.15.83:0/3537672074 >> v1:172.21.15.183:6789/0 conn(0x7fef180aef80 legacy=0x7fef180cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.485+0000 7fef1db16640 1 -- 172.21.15.83:0/3537672074 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef1815b820 con 0x7fef181586e0 2024-03-06T07:22:12.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.485+0000 7fef1e317640 1 --2- 172.21.15.83:0/3537672074 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fef180d01a0 0x7fef180b33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:12.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.486+0000 7fef1db16640 1 --2- 172.21.15.83:0/3537672074 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef181586e0 0x7fef180aea40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fef08009870 tx=0x7fef0802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c71d464f8998036 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:12.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.486+0000 7fef1cb14640 1 -- 172.21.15.83:0/3537672074 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef08034070 con 0x7fef181586e0 2024-03-06T07:22:12.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.486+0000 7fef1cb14640 1 -- 172.21.15.83:0/3537672074 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef08030840 con 0x7fef181586e0 2024-03-06T07:22:12.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.486+0000 7fef1cb14640 1 -- 172.21.15.83:0/3537672074 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef08030c10 con 0x7fef181586e0 2024-03-06T07:22:12.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.486+0000 7fef1fd7a640 1 -- 172.21.15.83:0/3537672074 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef181586e0 msgr2=0x7fef180aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.486+0000 7fef1fd7a640 1 --2- 172.21.15.83:0/3537672074 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef181586e0 0x7fef180aea40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fef08009870 tx=0x7fef0802b980 comp rx=0 tx=0).stop 2024-03-06T07:22:12.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.486+0000 7fef1fd7a640 1 -- 172.21.15.83:0/3537672074 shutdown_connections 2024-03-06T07:22:12.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.486+0000 7fef1fd7a640 1 --2- 172.21.15.83:0/3537672074 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fef180d01a0 0x7fef180b33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.486+0000 7fef1fd7a640 1 --2- 172.21.15.83:0/3537672074 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef181586e0 0x7fef180aea40 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.486+0000 7fef1fd7a640 1 -- 172.21.15.83:0/3537672074 >> 172.21.15.83:0/3537672074 conn(0x7fef1806b190 msgr2=0x7fef1806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:12.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.486+0000 7fef1fd7a640 1 -- 172.21.15.83:0/3537672074 shutdown_connections 2024-03-06T07:22:12.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.486+0000 7fef1fd7a640 1 -- 172.21.15.83:0/3537672074 wait complete. 2024-03-06T07:22:12.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.488+0000 7fef1fd7a640 1 Processor -- start 2024-03-06T07:22:12.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.488+0000 7fef1fd7a640 1 -- start start 2024-03-06T07:22:12.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.488+0000 7fef1fd7a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fef180aef80 0x7fef180cd680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.488+0000 7fef1fd7a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef180d01a0 0x7fef18156a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.488+0000 7fef1fd7a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef181586e0 0x7fef18156f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.488+0000 7fef1fd7a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef18071350 con 0x7fef180d01a0 2024-03-06T07:22:12.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.488+0000 7fef1fd7a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef1815bde0 con 0x7fef181586e0 2024-03-06T07:22:12.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.488+0000 7fef1fd7a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef180714d0 con 0x7fef180aef80 2024-03-06T07:22:12.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.488+0000 7fef1d315640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef180d01a0 0x7fef18156a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.488+0000 7fef1d315640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef180d01a0 0x7fef18156a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49872/0 (socket says 172.21.15.83:49872) 2024-03-06T07:22:12.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.488+0000 7fef1d315640 1 -- 172.21.15.83:0/60828470 learned_addr learned my addr 172.21.15.83:0/60828470 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:12.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.488+0000 7fef1db16640 1 --2- 172.21.15.83:0/60828470 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fef180aef80 0x7fef180cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.488+0000 7fef1e317640 1 --2- 172.21.15.83:0/60828470 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef181586e0 0x7fef18156f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.489+0000 7fef1d315640 1 -- 172.21.15.83:0/60828470 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fef180aef80 msgr2=0x7fef180cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.489+0000 7fef1d315640 1 --2- 172.21.15.83:0/60828470 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fef180aef80 0x7fef180cd680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.489+0000 7fef1d315640 1 -- 172.21.15.83:0/60828470 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef181586e0 msgr2=0x7fef18156f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.489+0000 7fef1d315640 1 --2- 172.21.15.83:0/60828470 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef181586e0 0x7fef18156f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.489+0000 7fef1d315640 1 -- 172.21.15.83:0/60828470 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef181411e0 con 0x7fef180d01a0 2024-03-06T07:22:12.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.489+0000 7fef1d315640 1 --2- 172.21.15.83:0/60828470 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef180d01a0 0x7fef18156a10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fef1400c440 tx=0x7fef14001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:12.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.489+0000 7fef0e7fc640 1 -- 172.21.15.83:0/60828470 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef140074e0 con 0x7fef180d01a0 2024-03-06T07:22:12.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.489+0000 7fef0e7fc640 1 -- 172.21.15.83:0/60828470 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef14007680 con 0x7fef180d01a0 2024-03-06T07:22:12.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.489+0000 7fef1fd7a640 1 -- 172.21.15.83:0/60828470 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef181414c0 con 0x7fef180d01a0 2024-03-06T07:22:12.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.489+0000 7fef0e7fc640 1 -- 172.21.15.83:0/60828470 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef14004a20 con 0x7fef180d01a0 2024-03-06T07:22:12.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.489+0000 7fef1fd7a640 1 -- 172.21.15.83:0/60828470 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef18141ea0 con 0x7fef180d01a0 2024-03-06T07:22:12.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.490+0000 7fef1fd7a640 1 -- 172.21.15.83:0/60828470 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef180732b0 con 0x7fef180d01a0 2024-03-06T07:22:12.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.491+0000 7fef0e7fc640 1 -- 172.21.15.83:0/60828470 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fef1401a020 con 0x7fef180d01a0 2024-03-06T07:22:12.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.491+0000 7fef0e7fc640 1 --2- 172.21.15.83:0/60828470 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7feefc06b500 0x7feefc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.491+0000 7fef0e7fc640 1 -- 172.21.15.83:0/60828470 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fef140898d0 con 0x7fef180d01a0 2024-03-06T07:22:12.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.491+0000 7fef1db16640 1 --2- 172.21.15.83:0/60828470 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7feefc06b500 0x7feefc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.491+0000 7fef1db16640 1 --2- 172.21.15.83:0/60828470 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7feefc06b500 0x7feefc06d9c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fef08036000 tx=0x7fef0803b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:12.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.499+0000 7fef0e7fc640 1 -- 172.21.15.83:0/60828470 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fef140586c0 con 0x7fef180d01a0 2024-03-06T07:22:12.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.711+0000 7fef1fd7a640 1 -- 172.21.15.83:0/60828470 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fef18068ff0 con 0x7fef180d01a0 2024-03-06T07:22:12.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.711+0000 7fef0e7fc640 1 -- 172.21.15.83:0/60828470 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fef14057fc0 con 0x7fef180d01a0 2024-03-06T07:22:12.713 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 64 2024-03-06T07:22:12.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.713+0000 7fef1fd7a640 1 -- 172.21.15.83:0/60828470 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7feefc06b500 msgr2=0x7feefc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.713+0000 7fef1fd7a640 1 --2- 172.21.15.83:0/60828470 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7feefc06b500 0x7feefc06d9c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fef08036000 tx=0x7fef0803b040 comp rx=0 tx=0).stop 2024-03-06T07:22:12.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.713+0000 7fef1fd7a640 1 -- 172.21.15.83:0/60828470 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef180d01a0 msgr2=0x7fef18156a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.713+0000 7fef1fd7a640 1 --2- 172.21.15.83:0/60828470 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef180d01a0 0x7fef18156a10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fef1400c440 tx=0x7fef14001e70 comp rx=0 tx=0).stop 2024-03-06T07:22:12.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.713+0000 7fef1fd7a640 1 -- 172.21.15.83:0/60828470 shutdown_connections 2024-03-06T07:22:12.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.713+0000 7fef1fd7a640 1 --2- 172.21.15.83:0/60828470 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7feefc06b500 0x7feefc06d9c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.713+0000 7fef1fd7a640 1 --2- 172.21.15.83:0/60828470 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef181586e0 0x7fef18156f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.713+0000 7fef1fd7a640 1 --2- 172.21.15.83:0/60828470 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef180d01a0 0x7fef18156a10 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.713+0000 7fef1fd7a640 1 --2- 172.21.15.83:0/60828470 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fef180aef80 0x7fef180cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.713+0000 7fef1fd7a640 1 -- 172.21.15.83:0/60828470 >> 172.21.15.83:0/60828470 conn(0x7fef1806b190 msgr2=0x7fef180723a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:12.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.713+0000 7fef1fd7a640 1 -- 172.21.15.83:0/60828470 shutdown_connections 2024-03-06T07:22:12.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.713+0000 7fef1fd7a640 1 -- 172.21.15.83:0/60828470 wait complete. 2024-03-06T07:22:12.725 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:12.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.834+0000 7fc5bcb25640 1 Processor -- start 2024-03-06T07:22:12.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.834+0000 7fc5bcb25640 1 -- start start 2024-03-06T07:22:12.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.835+0000 7fc5bcb25640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fc5b815fa10 0x7fc5b80ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.835+0000 7fc5bcb25640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b80cebe0 con 0x7fc5b80d16e0 2024-03-06T07:22:12.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.835+0000 7fc5bcb25640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b80ceec0 con 0x7fc5b815bdd0 2024-03-06T07:22:12.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.835+0000 7fc5bcb25640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5b80cf1a0 con 0x7fc5b815fa10 2024-03-06T07:22:12.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.835+0000 7fc5b6d9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fc5b80d16e0 0x7fc5b80d1af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43000/0 (socket says 172.21.15.83:43000) 2024-03-06T07:22:12.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.835+0000 7fc5b6d9d640 1 -- 172.21.15.83:0/509776981 learned_addr learned my addr 172.21.15.83:0/509776981 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:12.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.835+0000 7fc5affff640 1 --2- 172.21.15.83:0/509776981 >> v2:172.21.15.183:3300/0 conn(0x7fc5b815fa10 0x7fc5b80ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.835+0000 7fc5b5d9b640 1 -- 172.21.15.83:0/509776981 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3904435767 0 0) 0x7fc5b80cebe0 con 0x7fc5b80d16e0 2024-03-06T07:22:12.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.835+0000 7fc5b5d9b640 1 -- 172.21.15.83:0/509776981 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5a4003520 con 0x7fc5b80d16e0 2024-03-06T07:22:12.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.836+0000 7fc5b5d9b640 1 -- 172.21.15.83:0/509776981 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 617056465 0 0) 0x7fc5a4003520 con 0x7fc5b80d16e0 2024-03-06T07:22:12.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.836+0000 7fc5affff640 1 -- 172.21.15.83:0/509776981 >> v1:172.21.15.183:6789/0 conn(0x7fc5b815bdd0 legacy=0x7fc5b80c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.836+0000 7fc5affff640 1 -- 172.21.15.83:0/509776981 >> v1:172.21.15.83:6789/0 conn(0x7fc5b80d16e0 legacy=0x7fc5b80d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.836+0000 7fc5affff640 1 -- 172.21.15.83:0/509776981 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b80cc8e0 con 0x7fc5b815fa10 2024-03-06T07:22:12.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.836+0000 7fc5affff640 1 --2- 172.21.15.83:0/509776981 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5b815fa10 0x7fc5b80ce270 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc5a00097a0 tx=0x7fc5a002b600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c87a4c73887c9204 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:12.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.837+0000 7fc5b5d9b640 1 -- 172.21.15.83:0/509776981 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a0037070 con 0x7fc5b815fa10 2024-03-06T07:22:12.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.837+0000 7fc5b5d9b640 1 -- 172.21.15.83:0/509776981 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5a0030510 con 0x7fc5b815fa10 2024-03-06T07:22:12.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.837+0000 7fc5b5d9b640 1 -- 172.21.15.83:0/509776981 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0001a10 con 0x7fc5b815fa10 2024-03-06T07:22:12.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.837+0000 7fc5bcb25640 1 -- 172.21.15.83:0/509776981 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5b815fa10 msgr2=0x7fc5b80ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.837+0000 7fc5bcb25640 1 --2- 172.21.15.83:0/509776981 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5b815fa10 0x7fc5b80ce270 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc5a00097a0 tx=0x7fc5a002b600 comp rx=0 tx=0).stop 2024-03-06T07:22:12.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.837+0000 7fc5bcb25640 1 -- 172.21.15.83:0/509776981 shutdown_connections 2024-03-06T07:22:12.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.837+0000 7fc5bcb25640 1 --2- 172.21.15.83:0/509776981 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5b815fa10 0x7fc5b80ce270 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.837+0000 7fc5bcb25640 1 -- 172.21.15.83:0/509776981 >> 172.21.15.83:0/509776981 conn(0x7fc5b809a9f0 msgr2=0x7fc5b809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:12.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.837+0000 7fc5bcb25640 1 -- 172.21.15.83:0/509776981 shutdown_connections 2024-03-06T07:22:12.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.837+0000 7fc5bcb25640 1 -- 172.21.15.83:0/509776981 wait complete. 2024-03-06T07:22:12.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.839+0000 7fc5bcb25640 1 Processor -- start 2024-03-06T07:22:12.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.839+0000 7fc5bcb25640 1 -- start start 2024-03-06T07:22:12.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.840+0000 7fc5bcb25640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc5b80d16e0 0x7fc5b80c46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.840+0000 7fc5bcb25640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5b815bdd0 0x7fc5b80cb310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.840+0000 7fc5bcb25640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5b815fa10 0x7fc5b815c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.840+0000 7fc5bcb25640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b80ce7b0 con 0x7fc5b815fa10 2024-03-06T07:22:12.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.840+0000 7fc5bcb25640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b815c1b0 con 0x7fc5b815bdd0 2024-03-06T07:22:12.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.840+0000 7fc5bcb25640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b80ce930 con 0x7fc5b80d16e0 2024-03-06T07:22:12.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.840+0000 7fc5b6d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5b815fa10 0x7fc5b815c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.840+0000 7fc5b6d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5b815fa10 0x7fc5b815c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49884/0 (socket says 172.21.15.83:49884) 2024-03-06T07:22:12.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.840+0000 7fc5b6d9d640 1 -- 172.21.15.83:0/3097711671 learned_addr learned my addr 172.21.15.83:0/3097711671 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:12.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.840+0000 7fc5affff640 1 --2- 172.21.15.83:0/3097711671 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5b815bdd0 0x7fc5b80cb310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.840+0000 7fc5b659c640 1 --2- 172.21.15.83:0/3097711671 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc5b80d16e0 0x7fc5b80c46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.840+0000 7fc5b6d9d640 1 -- 172.21.15.83:0/3097711671 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc5b80d16e0 msgr2=0x7fc5b80c46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.840+0000 7fc5b6d9d640 1 --2- 172.21.15.83:0/3097711671 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc5b80d16e0 0x7fc5b80c46a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.840+0000 7fc5b6d9d640 1 -- 172.21.15.83:0/3097711671 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5b815bdd0 msgr2=0x7fc5b80cb310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.840+0000 7fc5b6d9d640 1 --2- 172.21.15.83:0/3097711671 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5b815bdd0 0x7fc5b80cb310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.840+0000 7fc5b6d9d640 1 -- 172.21.15.83:0/3097711671 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b80ca270 con 0x7fc5b815fa10 2024-03-06T07:22:12.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.840+0000 7fc5b6d9d640 1 --2- 172.21.15.83:0/3097711671 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5b815fa10 0x7fc5b815c330 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc5b000c8a0 tx=0x7fc5b000cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:12.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.840+0000 7fc5aeffd640 1 -- 172.21.15.83:0/3097711671 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0018070 con 0x7fc5b815fa10 2024-03-06T07:22:12.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.840+0000 7fc5aeffd640 1 -- 172.21.15.83:0/3097711671 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b000a5e0 con 0x7fc5b815fa10 2024-03-06T07:22:12.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.840+0000 7fc5bcb25640 1 -- 172.21.15.83:0/3097711671 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5b80ca550 con 0x7fc5b815fa10 2024-03-06T07:22:12.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.841+0000 7fc5aeffd640 1 -- 172.21.15.83:0/3097711671 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b000a9d0 con 0x7fc5b815fa10 2024-03-06T07:22:12.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.841+0000 7fc5bcb25640 1 -- 172.21.15.83:0/3097711671 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5b80cab10 con 0x7fc5b815fa10 2024-03-06T07:22:12.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.842+0000 7fc5bcb25640 1 -- 172.21.15.83:0/3097711671 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5b804f330 con 0x7fc5b815fa10 2024-03-06T07:22:12.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.842+0000 7fc5aeffd640 1 -- 172.21.15.83:0/3097711671 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc5b001b070 con 0x7fc5b815fa10 2024-03-06T07:22:12.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.842+0000 7fc5aeffd640 1 --2- 172.21.15.83:0/3097711671 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc59806b4d0 0x7fc59806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.843+0000 7fc5b659c640 1 --2- 172.21.15.83:0/3097711671 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc59806b4d0 0x7fc59806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.843+0000 7fc5aeffd640 1 -- 172.21.15.83:0/3097711671 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc5b0089950 con 0x7fc5b815fa10 2024-03-06T07:22:12.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.843+0000 7fc5b659c640 1 --2- 172.21.15.83:0/3097711671 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc59806b4d0 0x7fc59806d990 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc5a8000f60 tx=0x7fc5a8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:12.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:12.847+0000 7fc5aeffd640 1 -- 172.21.15.83:0/3097711671 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc5b00587d0 con 0x7fc5b815fa10 2024-03-06T07:22:13.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.057+0000 7fc5bcb25640 1 -- 172.21.15.83:0/3097711671 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5b80ce3f0 con 0x7fc5b815fa10 2024-03-06T07:22:13.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.058+0000 7fc5aeffd640 1 -- 172.21.15.83:0/3097711671 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4974 (secure 0 0 0) 0x7fc5b00580d0 con 0x7fc5b815fa10 2024-03-06T07:22:13.060 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:22:13.060 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/4249608353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":4249608353},{"type":"v1","addr":"172.21.15.83:6836","nonce":4249608353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/714452530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":714452530},{"type":"v1","addr":"172.21.15.183:6835","nonce":714452530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4282,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/2723013431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2723013431},{"type":"v1","addr":"172.21.15.183:6834","nonce":2723013431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:58.286994+0000","modified":"2024-03-06T07:22:00.307284+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/1597344199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1597344199},{"type":"v1","addr":"172.21.15.83:6837","nonce":1597344199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4287,"qdb_cluster":[4287]},"id":1}]} 2024-03-06T07:22:13.060 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 6 2024-03-06T07:22:13.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.060+0000 7fc5bcb25640 1 -- 172.21.15.83:0/3097711671 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc59806b4d0 msgr2=0x7fc59806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.060+0000 7fc5bcb25640 1 --2- 172.21.15.83:0/3097711671 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc59806b4d0 0x7fc59806d990 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc5a8000f60 tx=0x7fc5a8009040 comp rx=0 tx=0).stop 2024-03-06T07:22:13.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.060+0000 7fc5bcb25640 1 -- 172.21.15.83:0/3097711671 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5b815fa10 msgr2=0x7fc5b815c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.060+0000 7fc5bcb25640 1 --2- 172.21.15.83:0/3097711671 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5b815fa10 0x7fc5b815c330 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc5b000c8a0 tx=0x7fc5b000cd60 comp rx=0 tx=0).stop 2024-03-06T07:22:13.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.060+0000 7fc5bcb25640 1 -- 172.21.15.83:0/3097711671 shutdown_connections 2024-03-06T07:22:13.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.060+0000 7fc5bcb25640 1 --2- 172.21.15.83:0/3097711671 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc59806b4d0 0x7fc59806d990 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.060+0000 7fc5bcb25640 1 --2- 172.21.15.83:0/3097711671 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5b815fa10 0x7fc5b815c330 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.060+0000 7fc5bcb25640 1 --2- 172.21.15.83:0/3097711671 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5b815bdd0 0x7fc5b80cb310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.060+0000 7fc5bcb25640 1 --2- 172.21.15.83:0/3097711671 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc5b80d16e0 0x7fc5b80c46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.060+0000 7fc5bcb25640 1 -- 172.21.15.83:0/3097711671 >> 172.21.15.83:0/3097711671 conn(0x7fc5b809a9f0 msgr2=0x7fc5b809add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:13.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.060+0000 7fc5bcb25640 1 -- 172.21.15.83:0/3097711671 shutdown_connections 2024-03-06T07:22:13.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.060+0000 7fc5bcb25640 1 -- 172.21.15.83:0/3097711671 wait complete. 2024-03-06T07:22:13.072 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:13.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.175+0000 7f5933d56640 1 Processor -- start 2024-03-06T07:22:13.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.175+0000 7f5933d56640 1 -- start start 2024-03-06T07:22:13.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.176+0000 7f5933d56640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f592c15dfe0 0x7f592c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.176+0000 7f5933d56640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f592c0b3430 con 0x7f592c0d29c0 2024-03-06T07:22:13.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.176+0000 7f5933d56640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f592c0b3710 con 0x7f592c0cf830 2024-03-06T07:22:13.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.176+0000 7f5933d56640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f592c0b31e0 con 0x7f592c15dfe0 2024-03-06T07:22:13.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.176+0000 7f59322f3640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f592c0d29c0 0x7f592c0b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43012/0 (socket says 172.21.15.83:43012) 2024-03-06T07:22:13.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.176+0000 7f59322f3640 1 -- 172.21.15.83:0/1956550275 learned_addr learned my addr 172.21.15.83:0/1956550275 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:13.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.176+0000 7f5931af2640 1 --2- 172.21.15.83:0/1956550275 >> v2:172.21.15.183:3300/0 conn(0x7f592c15dfe0 0x7f592c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.176+0000 7f5930af0640 1 -- 172.21.15.83:0/1956550275 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2929567917 0 0) 0x7f592c0b3430 con 0x7f592c0d29c0 2024-03-06T07:22:13.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.176+0000 7f5930af0640 1 -- 172.21.15.83:0/1956550275 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5918003520 con 0x7f592c0d29c0 2024-03-06T07:22:13.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.176+0000 7f5930af0640 1 -- 172.21.15.83:0/1956550275 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 946927611 0 0) 0x7f5918003520 con 0x7f592c0d29c0 2024-03-06T07:22:13.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.176+0000 7f5930af0640 1 -- 172.21.15.83:0/1956550275 >> v2:172.21.15.183:3300/0 conn(0x7f592c15dfe0 msgr2=0x7f592c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:13.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.176+0000 7f5930af0640 1 --2- 172.21.15.83:0/1956550275 >> v2:172.21.15.183:3300/0 conn(0x7f592c15dfe0 0x7f592c0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.176+0000 7f5930af0640 1 -- 172.21.15.83:0/1956550275 >> v1:172.21.15.183:6789/0 conn(0x7f592c0cf830 legacy=0x7f592c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.176+0000 7f5930af0640 1 -- 172.21.15.83:0/1956550275 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f592c0b39f0 con 0x7f592c0d29c0 2024-03-06T07:22:13.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.176+0000 7f5930af0640 1 -- 172.21.15.83:0/1956550275 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f59240019f0 con 0x7f592c0d29c0 2024-03-06T07:22:13.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.176+0000 7f5930af0640 1 -- 172.21.15.83:0/1956550275 >> v1:172.21.15.83:6789/0 conn(0x7f592c0d29c0 legacy=0x7f592c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.176+0000 7f5930af0640 1 --2- 172.21.15.83:0/1956550275 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59180039b0 0x7f5918003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.176+0000 7f5930af0640 1 --2- 172.21.15.83:0/1956550275 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5918004800 0x7f5918004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.176+0000 7f5930af0640 1 --2- 172.21.15.83:0/1956550275 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f59180051c0 0x7f591800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.176+0000 7f5930af0640 1 -- 172.21.15.83:0/1956550275 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f592c0b31e0 con 0x7f59180039b0 2024-03-06T07:22:13.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.176+0000 7f5930af0640 1 -- 172.21.15.83:0/1956550275 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f591800c090 con 0x7f5918004800 2024-03-06T07:22:13.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.176+0000 7f5930af0640 1 -- 172.21.15.83:0/1956550275 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f591800c290 con 0x7f59180051c0 2024-03-06T07:22:13.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.177+0000 7f5931af2640 1 --2- 172.21.15.83:0/1956550275 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59180039b0 0x7f5918003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.177+0000 7f59312f1640 1 --2- 172.21.15.83:0/1956550275 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5918004800 0x7f5918004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.177+0000 7f59322f3640 1 --2- 172.21.15.83:0/1956550275 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f59180051c0 0x7f591800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.177+0000 7f5931af2640 1 -- 172.21.15.83:0/1956550275 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f59180051c0 msgr2=0x7f591800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:13.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.177+0000 7f5931af2640 1 --2- 172.21.15.83:0/1956550275 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f59180051c0 0x7f591800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.177+0000 7f5931af2640 1 -- 172.21.15.83:0/1956550275 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5918004800 msgr2=0x7f5918004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:13.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.177+0000 7f5931af2640 1 --2- 172.21.15.83:0/1956550275 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5918004800 0x7f5918004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.177+0000 7f5931af2640 1 -- 172.21.15.83:0/1956550275 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5918003520 con 0x7f59180039b0 2024-03-06T07:22:13.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.177+0000 7f59322f3640 1 -- 172.21.15.83:0/1956550275 reap_dead start 2024-03-06T07:22:13.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.177+0000 7f59312f1640 1 --2- 172.21.15.83:0/1956550275 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5918004800 0x7f5918004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:13.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.177+0000 7f5931af2640 1 --2- 172.21.15.83:0/1956550275 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59180039b0 0x7f5918003df0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5928009ac0 tx=0x7f5918005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f60461a7db41010d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:13.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.178+0000 7f5930af0640 1 -- 172.21.15.83:0/1956550275 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5928039070 con 0x7f59180039b0 2024-03-06T07:22:13.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.178+0000 7f5930af0640 1 -- 172.21.15.83:0/1956550275 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f592802c6d0 con 0x7f59180039b0 2024-03-06T07:22:13.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.178+0000 7f5930af0640 1 -- 172.21.15.83:0/1956550275 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5928009690 con 0x7f59180039b0 2024-03-06T07:22:13.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.178+0000 7f5933d56640 1 -- 172.21.15.83:0/1956550275 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59180039b0 msgr2=0x7f5918003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.178+0000 7f5933d56640 1 --2- 172.21.15.83:0/1956550275 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59180039b0 0x7f5918003df0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5928009ac0 tx=0x7f5918005830 comp rx=0 tx=0).stop 2024-03-06T07:22:13.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.178+0000 7f5933d56640 1 -- 172.21.15.83:0/1956550275 shutdown_connections 2024-03-06T07:22:13.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.178+0000 7f5933d56640 1 --2- 172.21.15.83:0/1956550275 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59180039b0 0x7f5918003df0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.178+0000 7f5933d56640 1 -- 172.21.15.83:0/1956550275 >> 172.21.15.83:0/1956550275 conn(0x7f592c0d3cd0 msgr2=0x7f592c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:13.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.178+0000 7f5933d56640 1 -- 172.21.15.83:0/1956550275 shutdown_connections 2024-03-06T07:22:13.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.178+0000 7f5933d56640 1 -- 172.21.15.83:0/1956550275 wait complete. 2024-03-06T07:22:13.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.180+0000 7f5933d56640 1 Processor -- start 2024-03-06T07:22:13.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.180+0000 7f5933d56640 1 -- start start 2024-03-06T07:22:13.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.181+0000 7f5933d56640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f59180039b0 0x7f592c0b8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.181+0000 7f5933d56640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f592c150c10 0x7f592c151070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.181+0000 7f5933d56640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f592c0cbfc0 0x7f592c0cc440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.181+0000 7f5933d56640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f592c15e3c0 con 0x7f592c150c10 2024-03-06T07:22:13.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.181+0000 7f5933d56640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f592c0b3d60 con 0x7f59180039b0 2024-03-06T07:22:13.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.181+0000 7f5933d56640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f592c15e540 con 0x7f592c0cbfc0 2024-03-06T07:22:13.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.182+0000 7f59312f1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f592c150c10 0x7f592c151070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.182+0000 7f59312f1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f592c150c10 0x7f592c151070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49890/0 (socket says 172.21.15.83:49890) 2024-03-06T07:22:13.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.182+0000 7f59312f1640 1 -- 172.21.15.83:0/807674288 learned_addr learned my addr 172.21.15.83:0/807674288 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:13.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.182+0000 7f5931af2640 1 --2- 172.21.15.83:0/807674288 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f59180039b0 0x7f592c0b8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.182+0000 7f59322f3640 1 --2- 172.21.15.83:0/807674288 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f592c0cbfc0 0x7f592c0cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.182+0000 7f59312f1640 1 -- 172.21.15.83:0/807674288 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f592c0cbfc0 msgr2=0x7f592c0cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.182+0000 7f59312f1640 1 --2- 172.21.15.83:0/807674288 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f592c0cbfc0 0x7f592c0cc440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.182+0000 7f59312f1640 1 -- 172.21.15.83:0/807674288 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f59180039b0 msgr2=0x7f592c0b8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.182+0000 7f59312f1640 1 --2- 172.21.15.83:0/807674288 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f59180039b0 0x7f592c0b8b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.182+0000 7f59312f1640 1 -- 172.21.15.83:0/807674288 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f592c0caf20 con 0x7f592c150c10 2024-03-06T07:22:13.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.182+0000 7f5931af2640 1 --2- 172.21.15.83:0/807674288 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f59180039b0 0x7f592c0b8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:13.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.182+0000 7f59322f3640 1 --2- 172.21.15.83:0/807674288 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f592c0cbfc0 0x7f592c0cc440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:13.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.182+0000 7f59312f1640 1 --2- 172.21.15.83:0/807674288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f592c150c10 0x7f592c151070 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f591c00e480 tx=0x7f591c00e940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:13.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.182+0000 7f59227fc640 1 -- 172.21.15.83:0/807674288 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591c0052e0 con 0x7f592c150c10 2024-03-06T07:22:13.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.182+0000 7f59227fc640 1 -- 172.21.15.83:0/807674288 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f591c018070 con 0x7f592c150c10 2024-03-06T07:22:13.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.182+0000 7f5933d56640 1 -- 172.21.15.83:0/807674288 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f592c0cb200 con 0x7f592c150c10 2024-03-06T07:22:13.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.182+0000 7f59227fc640 1 -- 172.21.15.83:0/807674288 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591c003830 con 0x7f592c150c10 2024-03-06T07:22:13.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.182+0000 7f5933d56640 1 -- 172.21.15.83:0/807674288 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f592c0cb7c0 con 0x7f592c150c10 2024-03-06T07:22:13.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.183+0000 7f5933d56640 1 -- 172.21.15.83:0/807674288 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f592c04f330 con 0x7f592c150c10 2024-03-06T07:22:13.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.183+0000 7f59227fc640 1 -- 172.21.15.83:0/807674288 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f591c0039d0 con 0x7f592c150c10 2024-03-06T07:22:13.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.184+0000 7f59227fc640 1 --2- 172.21.15.83:0/807674288 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f591006b500 0x7f591006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.184+0000 7f5931af2640 1 --2- 172.21.15.83:0/807674288 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f591006b500 0x7f591006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.184+0000 7f59227fc640 1 -- 172.21.15.83:0/807674288 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f591c089f00 con 0x7f592c150c10 2024-03-06T07:22:13.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.185+0000 7f5931af2640 1 --2- 172.21.15.83:0/807674288 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f591006b500 0x7f591006d9c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5928009ac0 tx=0x7f5928030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:13.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.188+0000 7f59227fc640 1 -- 172.21.15.83:0/807674288 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f591c058d00 con 0x7f592c150c10 2024-03-06T07:22:13.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.367+0000 7f5933d56640 1 -- 172.21.15.83:0/807674288 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f592c0b53d0 con 0x7f592c150c10 2024-03-06T07:22:13.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.368+0000 7f59227fc640 1 -- 172.21.15.83:0/807674288 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16617 (secure 0 0 0) 0x7f591c058600 con 0x7f592c150c10 2024-03-06T07:22:13.370 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:22:13.371 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":20,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:21:59.938137+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.318942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.254795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:13.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.370+0000 7f5933d56640 1 -- 172.21.15.83:0/807674288 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f591006b500 msgr2=0x7f591006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.370+0000 7f5933d56640 1 --2- 172.21.15.83:0/807674288 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f591006b500 0x7f591006d9c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5928009ac0 tx=0x7f5928030000 comp rx=0 tx=0).stop 2024-03-06T07:22:13.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.370+0000 7f5933d56640 1 -- 172.21.15.83:0/807674288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f592c150c10 msgr2=0x7f592c151070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.370+0000 7f5933d56640 1 --2- 172.21.15.83:0/807674288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f592c150c10 0x7f592c151070 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f591c00e480 tx=0x7f591c00e940 comp rx=0 tx=0).stop 2024-03-06T07:22:13.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.370+0000 7f5933d56640 1 -- 172.21.15.83:0/807674288 shutdown_connections 2024-03-06T07:22:13.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.370+0000 7f5933d56640 1 --2- 172.21.15.83:0/807674288 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f591006b500 0x7f591006d9c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.370+0000 7f5933d56640 1 --2- 172.21.15.83:0/807674288 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f592c0cbfc0 0x7f592c0cc440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.370+0000 7f5933d56640 1 --2- 172.21.15.83:0/807674288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f592c150c10 0x7f592c151070 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.370+0000 7f5933d56640 1 --2- 172.21.15.83:0/807674288 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f59180039b0 0x7f592c0b8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.370+0000 7f5933d56640 1 -- 172.21.15.83:0/807674288 >> 172.21.15.83:0/807674288 conn(0x7f592c0d3cd0 msgr2=0x7f592c0ced30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:13.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.370+0000 7f5933d56640 1 -- 172.21.15.83:0/807674288 shutdown_connections 2024-03-06T07:22:13.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.370+0000 7f5933d56640 1 -- 172.21.15.83:0/807674288 wait complete. 2024-03-06T07:22:13.383 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:56.318942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:57.254795+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:13.384 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:13.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.485+0000 7fc07af01640 1 Processor -- start 2024-03-06T07:22:13.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.485+0000 7fc07af01640 1 -- start start 2024-03-06T07:22:13.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.487+0000 7fc07af01640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc07415ecd0 0x7fc0740cae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.487+0000 7fc07af01640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0740cfa50 con 0x7fc0740d10c0 2024-03-06T07:22:13.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.487+0000 7fc07af01640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0740b45c0 con 0x7fc0740d14a0 2024-03-06T07:22:13.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.487+0000 7fc07af01640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc07415c050 con 0x7fc07415ecd0 2024-03-06T07:22:13.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.487+0000 7fc07949e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc07415ecd0 0x7fc0740cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.487+0000 7fc078c9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fc0740d10c0 0x7fc0740d04f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43014/0 (socket says 172.21.15.83:43014) 2024-03-06T07:22:13.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.487+0000 7fc078c9d640 1 -- 172.21.15.83:0/4082841220 learned_addr learned my addr 172.21.15.83:0/4082841220 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:13.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.487+0000 7fc07949e640 1 -- 172.21.15.83:0/4082841220 >> v1:172.21.15.183:6789/0 conn(0x7fc0740d14a0 legacy=0x7fc07415e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:13.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.487+0000 7fc07949e640 1 -- 172.21.15.83:0/4082841220 >> v1:172.21.15.83:6789/0 conn(0x7fc0740d10c0 legacy=0x7fc0740d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.487+0000 7fc07949e640 1 -- 172.21.15.83:0/4082841220 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc07415f160 con 0x7fc07415ecd0 2024-03-06T07:22:13.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.487+0000 7fc0737fe640 1 -- 172.21.15.83:0/4082841220 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 415853101 0 0) 0x7fc0740cfa50 con 0x7fc0740d10c0 2024-03-06T07:22:13.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.488+0000 7fc07949e640 1 --2- 172.21.15.83:0/4082841220 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc07415ecd0 0x7fc0740cae80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc06c009d90 tx=0x7fc06c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14b7dd38135dc9e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:13.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.488+0000 7fc0737fe640 1 -- 172.21.15.83:0/4082841220 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc06c039070 con 0x7fc07415ecd0 2024-03-06T07:22:13.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.488+0000 7fc0737fe640 1 -- 172.21.15.83:0/4082841220 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc06c02cc60 con 0x7fc07415ecd0 2024-03-06T07:22:13.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.488+0000 7fc0737fe640 1 -- 172.21.15.83:0/4082841220 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc06c034660 con 0x7fc07415ecd0 2024-03-06T07:22:13.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.489+0000 7fc07af01640 1 -- 172.21.15.83:0/4082841220 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc07415ecd0 msgr2=0x7fc0740cae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.489+0000 7fc07af01640 1 --2- 172.21.15.83:0/4082841220 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc07415ecd0 0x7fc0740cae80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc06c009d90 tx=0x7fc06c02beb0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.489+0000 7fc07af01640 1 -- 172.21.15.83:0/4082841220 shutdown_connections 2024-03-06T07:22:13.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.489+0000 7fc07af01640 1 --2- 172.21.15.83:0/4082841220 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc07415ecd0 0x7fc0740cae80 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.489+0000 7fc07af01640 1 -- 172.21.15.83:0/4082841220 >> 172.21.15.83:0/4082841220 conn(0x7fc0740b3800 msgr2=0x7fc0740b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:13.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.489+0000 7fc07af01640 1 -- 172.21.15.83:0/4082841220 shutdown_connections 2024-03-06T07:22:13.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.489+0000 7fc07af01640 1 -- 172.21.15.83:0/4082841220 wait complete. 2024-03-06T07:22:13.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.490+0000 7fc07af01640 1 Processor -- start 2024-03-06T07:22:13.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.490+0000 7fc07af01640 1 -- start start 2024-03-06T07:22:13.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.491+0000 7fc07af01640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0740d10c0 0x7fc0740b19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.491+0000 7fc07af01640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc0740d14a0 0x7fc0740b4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.491+0000 7fc07af01640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0740b5060 0x7fc0740b6990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.491+0000 7fc07af01640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0740cfcf0 con 0x7fc0740b5060 2024-03-06T07:22:13.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.491+0000 7fc07af01640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0740d0060 con 0x7fc0740d10c0 2024-03-06T07:22:13.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.491+0000 7fc07af01640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0740cb3c0 con 0x7fc0740d14a0 2024-03-06T07:22:13.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.491+0000 7fc07949e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0740b5060 0x7fc0740b6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.491+0000 7fc07949e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0740b5060 0x7fc0740b6990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49906/0 (socket says 172.21.15.83:49906) 2024-03-06T07:22:13.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.491+0000 7fc07949e640 1 -- 172.21.15.83:0/997334288 learned_addr learned my addr 172.21.15.83:0/997334288 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:13.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.491+0000 7fc07949e640 1 -- 172.21.15.83:0/997334288 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc0740d14a0 msgr2=0x7fc0740b4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.491+0000 7fc07949e640 1 --2- 172.21.15.83:0/997334288 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc0740d14a0 0x7fc0740b4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.491+0000 7fc078c9d640 1 --2- 172.21.15.83:0/997334288 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0740d10c0 0x7fc0740b19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.491+0000 7fc07949e640 1 -- 172.21.15.83:0/997334288 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0740d10c0 msgr2=0x7fc0740b19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.491+0000 7fc07949e640 1 --2- 172.21.15.83:0/997334288 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0740d10c0 0x7fc0740b19e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.491+0000 7fc07949e640 1 -- 172.21.15.83:0/997334288 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0740b54f0 con 0x7fc0740b5060 2024-03-06T07:22:13.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.491+0000 7fc07949e640 1 --2- 172.21.15.83:0/997334288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0740b5060 0x7fc0740b6990 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc06c037040 tx=0x7fc06c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:13.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.491+0000 7fc0717fa640 1 -- 172.21.15.83:0/997334288 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc06c044030 con 0x7fc0740b5060 2024-03-06T07:22:13.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.491+0000 7fc0717fa640 1 -- 172.21.15.83:0/997334288 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc06c02ca60 con 0x7fc0740b5060 2024-03-06T07:22:13.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.491+0000 7fc0717fa640 1 -- 172.21.15.83:0/997334288 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc06c039040 con 0x7fc0740b5060 2024-03-06T07:22:13.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.491+0000 7fc07af01640 1 -- 172.21.15.83:0/997334288 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0740b7750 con 0x7fc0740b5060 2024-03-06T07:22:13.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.491+0000 7fc07af01640 1 -- 172.21.15.83:0/997334288 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0741f7cb0 con 0x7fc0740b5060 2024-03-06T07:22:13.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.492+0000 7fc07af01640 1 -- 172.21.15.83:0/997334288 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0741f77c0 con 0x7fc0740b5060 2024-03-06T07:22:13.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.493+0000 7fc0717fa640 1 -- 172.21.15.83:0/997334288 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc06c047070 con 0x7fc0740b5060 2024-03-06T07:22:13.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.493+0000 7fc0717fa640 1 --2- 172.21.15.83:0/997334288 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc05806b4d0 0x7fc05806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.494+0000 7fc078c9d640 1 --2- 172.21.15.83:0/997334288 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc05806b4d0 0x7fc05806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.494+0000 7fc0717fa640 1 -- 172.21.15.83:0/997334288 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc06c0ad1c0 con 0x7fc0740b5060 2024-03-06T07:22:13.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.494+0000 7fc078c9d640 1 --2- 172.21.15.83:0/997334288 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc05806b4d0 0x7fc05806d990 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc068007fd0 tx=0x7fc068009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:13.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.497+0000 7fc0717fa640 1 -- 172.21.15.83:0/997334288 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc06c04e070 con 0x7fc0740b5060 2024-03-06T07:22:13.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.687+0000 7fc07af01640 1 -- 172.21.15.83:0/997334288 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc07409fe60 con 0x7fc0740b5060 2024-03-06T07:22:13.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.687+0000 7fc0717fa640 1 -- 172.21.15.83:0/997334288 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc06c02cc30 con 0x7fc0740b5060 2024-03-06T07:22:13.689 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:22:13.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.690+0000 7fc07af01640 1 -- 172.21.15.83:0/997334288 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc05806b4d0 msgr2=0x7fc05806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.690+0000 7fc07af01640 1 --2- 172.21.15.83:0/997334288 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc05806b4d0 0x7fc05806d990 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc068007fd0 tx=0x7fc068009040 comp rx=0 tx=0).stop 2024-03-06T07:22:13.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.690+0000 7fc07af01640 1 -- 172.21.15.83:0/997334288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0740b5060 msgr2=0x7fc0740b6990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.690+0000 7fc07af01640 1 --2- 172.21.15.83:0/997334288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0740b5060 0x7fc0740b6990 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc06c037040 tx=0x7fc06c004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:13.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.690+0000 7fc07af01640 1 -- 172.21.15.83:0/997334288 shutdown_connections 2024-03-06T07:22:13.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.690+0000 7fc07af01640 1 --2- 172.21.15.83:0/997334288 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc05806b4d0 0x7fc05806d990 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.690+0000 7fc07af01640 1 --2- 172.21.15.83:0/997334288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0740b5060 0x7fc0740b6990 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.690+0000 7fc07af01640 1 --2- 172.21.15.83:0/997334288 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc0740d14a0 0x7fc0740b4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.690+0000 7fc07af01640 1 --2- 172.21.15.83:0/997334288 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0740d10c0 0x7fc0740b19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.690+0000 7fc07af01640 1 -- 172.21.15.83:0/997334288 >> 172.21.15.83:0/997334288 conn(0x7fc0740b3800 msgr2=0x7fc0740b3be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:13.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.690+0000 7fc07af01640 1 -- 172.21.15.83:0/997334288 shutdown_connections 2024-03-06T07:22:13.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.690+0000 7fc07af01640 1 -- 172.21.15.83:0/997334288 wait complete. 2024-03-06T07:22:13.702 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:13.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.804+0000 7f5dc18df640 1 Processor -- start 2024-03-06T07:22:13.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.804+0000 7f5dc18df640 1 -- start start 2024-03-06T07:22:13.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.805+0000 7f5dc18df640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f5dbc0d06e0 0x7f5dbc15fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.805+0000 7f5dc18df640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dbc0d47f0 con 0x7f5dbc0d0ac0 2024-03-06T07:22:13.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.805+0000 7f5dc18df640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dbc0d1ab0 con 0x7f5dbc0cf200 2024-03-06T07:22:13.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.805+0000 7f5dc18df640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dbc0ced70 con 0x7f5dbc0d06e0 2024-03-06T07:22:13.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.805+0000 7f5dba7fc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f5dbc0d0ac0 0x7f5dbc0b47d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43020/0 (socket says 172.21.15.83:43020) 2024-03-06T07:22:13.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.805+0000 7f5dba7fc640 1 -- 172.21.15.83:0/2441902600 learned_addr learned my addr 172.21.15.83:0/2441902600 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:13.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.805+0000 7f5dbaffd640 1 --2- 172.21.15.83:0/2441902600 >> v2:172.21.15.183:3300/0 conn(0x7f5dbc0d06e0 0x7f5dbc15fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.805+0000 7f5db9ffb640 1 -- 172.21.15.83:0/2441902600 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 978463903 0 0) 0x7f5dbc0d47f0 con 0x7f5dbc0d0ac0 2024-03-06T07:22:13.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.805+0000 7f5db9ffb640 1 -- 172.21.15.83:0/2441902600 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5da4003520 con 0x7f5dbc0d0ac0 2024-03-06T07:22:13.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.805+0000 7f5db9ffb640 1 -- 172.21.15.83:0/2441902600 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1828664441 0 0) 0x7f5da4003520 con 0x7f5dbc0d0ac0 2024-03-06T07:22:13.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.805+0000 7f5db9ffb640 1 -- 172.21.15.83:0/2441902600 >> v2:172.21.15.183:3300/0 conn(0x7f5dbc0d06e0 msgr2=0x7f5dbc15fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:13.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.805+0000 7f5db9ffb640 1 --2- 172.21.15.83:0/2441902600 >> v2:172.21.15.183:3300/0 conn(0x7f5dbc0d06e0 0x7f5dbc15fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.806+0000 7f5db9ffb640 1 -- 172.21.15.83:0/2441902600 >> v1:172.21.15.183:6789/0 conn(0x7f5dbc0cf200 legacy=0x7f5dbc0cf650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.806+0000 7f5db9ffb640 1 -- 172.21.15.83:0/2441902600 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dbc0d1d90 con 0x7f5dbc0d0ac0 2024-03-06T07:22:13.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.806+0000 7f5db9ffb640 1 -- 172.21.15.83:0/2441902600 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f5db0001670 con 0x7f5dbc0d0ac0 2024-03-06T07:22:13.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.806+0000 7f5db9ffb640 1 -- 172.21.15.83:0/2441902600 >> v1:172.21.15.83:6789/0 conn(0x7f5dbc0d0ac0 legacy=0x7f5dbc0b47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.806+0000 7f5db9ffb640 1 --2- 172.21.15.83:0/2441902600 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5da40039b0 0x7f5da4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.806+0000 7f5db9ffb640 1 --2- 172.21.15.83:0/2441902600 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5da4004800 0x7f5da4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.806+0000 7f5db9ffb640 1 --2- 172.21.15.83:0/2441902600 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5da40051c0 0x7f5da400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.806+0000 7f5db9ffb640 1 -- 172.21.15.83:0/2441902600 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dbc0ced70 con 0x7f5da40039b0 2024-03-06T07:22:13.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.806+0000 7f5db9ffb640 1 -- 172.21.15.83:0/2441902600 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da400c060 con 0x7f5da4004800 2024-03-06T07:22:13.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.806+0000 7f5db9ffb640 1 -- 172.21.15.83:0/2441902600 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5da400c260 con 0x7f5da40051c0 2024-03-06T07:22:13.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.806+0000 7f5dbaffd640 1 --2- 172.21.15.83:0/2441902600 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5da40039b0 0x7f5da4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.806+0000 7f5dba7fc640 1 --2- 172.21.15.83:0/2441902600 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5da4004800 0x7f5da4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.806+0000 7f5dbb7fe640 1 --2- 172.21.15.83:0/2441902600 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5da40051c0 0x7f5da400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.806+0000 7f5dbaffd640 1 -- 172.21.15.83:0/2441902600 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5da40051c0 msgr2=0x7f5da400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:13.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.806+0000 7f5dbaffd640 1 --2- 172.21.15.83:0/2441902600 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5da40051c0 0x7f5da400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.806+0000 7f5dbaffd640 1 -- 172.21.15.83:0/2441902600 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5da4004800 msgr2=0x7f5da4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:13.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.806+0000 7f5dbaffd640 1 --2- 172.21.15.83:0/2441902600 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5da4004800 0x7f5da4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.806+0000 7f5dbaffd640 1 -- 172.21.15.83:0/2441902600 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da4003520 con 0x7f5da40039b0 2024-03-06T07:22:13.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.806+0000 7f5dbb7fe640 1 --2- 172.21.15.83:0/2441902600 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5da40051c0 0x7f5da400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:13.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.806+0000 7f5dba7fc640 1 --2- 172.21.15.83:0/2441902600 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5da4004800 0x7f5da4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:13.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.806+0000 7f5dbb7fe640 1 -- 172.21.15.83:0/2441902600 reap_dead start 2024-03-06T07:22:13.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.807+0000 7f5dbaffd640 1 --2- 172.21.15.83:0/2441902600 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5da40039b0 0x7f5da4003dd0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5db4009fc0 tx=0x7f5da4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b75881f54a18fab9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:13.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.807+0000 7f5db9ffb640 1 -- 172.21.15.83:0/2441902600 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db4039070 con 0x7f5da40039b0 2024-03-06T07:22:13.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.807+0000 7f5db9ffb640 1 -- 172.21.15.83:0/2441902600 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5db402cc20 con 0x7f5da40039b0 2024-03-06T07:22:13.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.807+0000 7f5db9ffb640 1 -- 172.21.15.83:0/2441902600 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db4009c10 con 0x7f5da40039b0 2024-03-06T07:22:13.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.807+0000 7f5dc18df640 1 -- 172.21.15.83:0/2441902600 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5da40039b0 msgr2=0x7f5da4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.807+0000 7f5dc18df640 1 --2- 172.21.15.83:0/2441902600 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5da40039b0 0x7f5da4003dd0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5db4009fc0 tx=0x7f5da4005830 comp rx=0 tx=0).stop 2024-03-06T07:22:13.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.807+0000 7f5dc18df640 1 -- 172.21.15.83:0/2441902600 shutdown_connections 2024-03-06T07:22:13.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.807+0000 7f5dc18df640 1 --2- 172.21.15.83:0/2441902600 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5da40039b0 0x7f5da4003dd0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.807+0000 7f5dc18df640 1 -- 172.21.15.83:0/2441902600 >> 172.21.15.83:0/2441902600 conn(0x7f5dbc0d3cd0 msgr2=0x7f5dbc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:13.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.807+0000 7f5dc18df640 1 -- 172.21.15.83:0/2441902600 shutdown_connections 2024-03-06T07:22:13.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.807+0000 7f5dc18df640 1 -- 172.21.15.83:0/2441902600 wait complete. 2024-03-06T07:22:13.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.808+0000 7f5dc18df640 1 Processor -- start 2024-03-06T07:22:13.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.809+0000 7f5dc18df640 1 -- start start 2024-03-06T07:22:13.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.809+0000 7f5dc18df640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5da40039b0 0x7f5dbc0c9c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.809+0000 7f5dc18df640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5dbc1f8910 0x7f5dbc14e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.809+0000 7f5dc18df640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5dbc15fc20 0x7f5dbc1600b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.809+0000 7f5dc18df640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dbc0b44d0 con 0x7f5dbc1f8910 2024-03-06T07:22:13.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.809+0000 7f5dc18df640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dbc0ce990 con 0x7f5dbc15fc20 2024-03-06T07:22:13.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.809+0000 7f5dc18df640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dbc0d0f30 con 0x7f5da40039b0 2024-03-06T07:22:13.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.809+0000 7f5dba7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5dbc1f8910 0x7f5dbc14e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.809+0000 7f5dba7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5dbc1f8910 0x7f5dbc14e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49928/0 (socket says 172.21.15.83:49928) 2024-03-06T07:22:13.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.809+0000 7f5dba7fc640 1 -- 172.21.15.83:0/1234960685 learned_addr learned my addr 172.21.15.83:0/1234960685 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:13.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.810+0000 7f5dbaffd640 1 --2- 172.21.15.83:0/1234960685 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5da40039b0 0x7f5dbc0c9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.810+0000 7f5dbb7fe640 1 --2- 172.21.15.83:0/1234960685 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5dbc15fc20 0x7f5dbc1600b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.810+0000 7f5dba7fc640 1 -- 172.21.15.83:0/1234960685 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5da40039b0 msgr2=0x7f5dbc0c9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.810+0000 7f5dba7fc640 1 --2- 172.21.15.83:0/1234960685 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5da40039b0 0x7f5dbc0c9c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.810+0000 7f5dba7fc640 1 -- 172.21.15.83:0/1234960685 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5dbc15fc20 msgr2=0x7f5dbc1600b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.810+0000 7f5dba7fc640 1 --2- 172.21.15.83:0/1234960685 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5dbc15fc20 0x7f5dbc1600b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.810+0000 7f5dba7fc640 1 -- 172.21.15.83:0/1234960685 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dbc0ca180 con 0x7f5dbc1f8910 2024-03-06T07:22:13.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.810+0000 7f5dba7fc640 1 --2- 172.21.15.83:0/1234960685 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5dbc1f8910 0x7f5dbc14e110 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5db000eaf0 tx=0x7f5db0005e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:13.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.810+0000 7f5da3fff640 1 -- 172.21.15.83:0/1234960685 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db0019070 con 0x7f5dbc1f8910 2024-03-06T07:22:13.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.810+0000 7f5da3fff640 1 -- 172.21.15.83:0/1234960685 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5db00049f0 con 0x7f5dbc1f8910 2024-03-06T07:22:13.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.810+0000 7f5dc18df640 1 -- 172.21.15.83:0/1234960685 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dbc0b1af0 con 0x7f5dbc1f8910 2024-03-06T07:22:13.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.810+0000 7f5da3fff640 1 -- 172.21.15.83:0/1234960685 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db0004dc0 con 0x7f5dbc1f8910 2024-03-06T07:22:13.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.810+0000 7f5dc18df640 1 -- 172.21.15.83:0/1234960685 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dbc0b20b0 con 0x7f5dbc1f8910 2024-03-06T07:22:13.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.811+0000 7f5dc18df640 1 -- 172.21.15.83:0/1234960685 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dbc1624d0 con 0x7f5dbc1f8910 2024-03-06T07:22:13.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.811+0000 7f5da3fff640 1 -- 172.21.15.83:0/1234960685 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5db000c7e0 con 0x7f5dbc1f8910 2024-03-06T07:22:13.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.812+0000 7f5da3fff640 1 --2- 172.21.15.83:0/1234960685 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5da806b500 0x7f5da806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.812+0000 7f5dbaffd640 1 --2- 172.21.15.83:0/1234960685 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5da806b500 0x7f5da806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.812+0000 7f5da3fff640 1 -- 172.21.15.83:0/1234960685 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5db0089af0 con 0x7f5dbc1f8910 2024-03-06T07:22:13.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.812+0000 7f5dbaffd640 1 --2- 172.21.15.83:0/1234960685 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5da806b500 0x7f5da806d9c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5db4036000 tx=0x7f5db402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:13.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.816+0000 7f5da3fff640 1 -- 172.21.15.83:0/1234960685 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5dbc1624d0 con 0x7f5dbc1f8910 2024-03-06T07:22:13.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.993+0000 7f5dc18df640 1 -- 172.21.15.83:0/1234960685 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5dbc06c100 con 0x7f5dbc1f8910 2024-03-06T07:22:13.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.994+0000 7f5da3fff640 1 -- 172.21.15.83:0/1234960685 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5dbc06c100 con 0x7f5dbc1f8910 2024-03-06T07:22:13.996 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-03-06T07:22:13.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.996+0000 7f5dc18df640 1 -- 172.21.15.83:0/1234960685 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5da806b500 msgr2=0x7f5da806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.996+0000 7f5dc18df640 1 --2- 172.21.15.83:0/1234960685 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5da806b500 0x7f5da806d9c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5db4036000 tx=0x7f5db402e000 comp rx=0 tx=0).stop 2024-03-06T07:22:13.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.996+0000 7f5dc18df640 1 -- 172.21.15.83:0/1234960685 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5dbc1f8910 msgr2=0x7f5dbc14e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.996+0000 7f5dc18df640 1 --2- 172.21.15.83:0/1234960685 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5dbc1f8910 0x7f5dbc14e110 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5db000eaf0 tx=0x7f5db0005e90 comp rx=0 tx=0).stop 2024-03-06T07:22:13.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.996+0000 7f5dc18df640 1 -- 172.21.15.83:0/1234960685 shutdown_connections 2024-03-06T07:22:13.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.996+0000 7f5dc18df640 1 --2- 172.21.15.83:0/1234960685 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5da806b500 0x7f5da806d9c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.996+0000 7f5dc18df640 1 --2- 172.21.15.83:0/1234960685 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5dbc15fc20 0x7f5dbc1600b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.996+0000 7f5dc18df640 1 --2- 172.21.15.83:0/1234960685 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5dbc1f8910 0x7f5dbc14e110 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.996+0000 7f5dc18df640 1 --2- 172.21.15.83:0/1234960685 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5da40039b0 0x7f5dbc0c9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.996+0000 7f5dc18df640 1 -- 172.21.15.83:0/1234960685 >> 172.21.15.83:0/1234960685 conn(0x7f5dbc0d3cd0 msgr2=0x7f5dbc161dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:13.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.996+0000 7f5dc18df640 1 -- 172.21.15.83:0/1234960685 shutdown_connections 2024-03-06T07:22:13.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:13.996+0000 7f5dc18df640 1 -- 172.21.15.83:0/1234960685 wait complete. 2024-03-06T07:22:14.009 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:14.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.111+0000 7efcc28d4640 1 Processor -- start 2024-03-06T07:22:14.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.111+0000 7efcc28d4640 1 -- start start 2024-03-06T07:22:14.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.112+0000 7efcc28d4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7efcbc15ecd0 0x7efcbc0cae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.112+0000 7efcc28d4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcbc0cfa50 con 0x7efcbc0d10c0 2024-03-06T07:22:14.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.112+0000 7efcc28d4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcbc0b45c0 con 0x7efcbc0d14a0 2024-03-06T07:22:14.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.112+0000 7efcc28d4640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcbc15c050 con 0x7efcbc15ecd0 2024-03-06T07:22:14.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.112+0000 7efcc0e71640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7efcbc15ecd0 0x7efcbc0cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.112+0000 7efcbbfff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7efcbc0d10c0 0x7efcbc0d04f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43022/0 (socket says 172.21.15.83:43022) 2024-03-06T07:22:14.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.112+0000 7efcbbfff640 1 -- 172.21.15.83:0/2965838548 learned_addr learned my addr 172.21.15.83:0/2965838548 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:14.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.112+0000 7efcc0e71640 1 -- 172.21.15.83:0/2965838548 >> v1:172.21.15.183:6789/0 conn(0x7efcbc0d14a0 legacy=0x7efcbc15e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:14.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.112+0000 7efcc0e71640 1 -- 172.21.15.83:0/2965838548 >> v1:172.21.15.83:6789/0 conn(0x7efcbc0d10c0 legacy=0x7efcbc0d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.112+0000 7efcbaffd640 1 -- 172.21.15.83:0/2965838548 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3104210303 0 0) 0x7efcbc0cfa50 con 0x7efcbc0d10c0 2024-03-06T07:22:14.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.112+0000 7efcc0e71640 1 -- 172.21.15.83:0/2965838548 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcbc15f160 con 0x7efcbc15ecd0 2024-03-06T07:22:14.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.113+0000 7efcc0e71640 1 --2- 172.21.15.83:0/2965838548 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcbc15ecd0 0x7efcbc0cae80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7efcb4009d90 tx=0x7efcb402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95a0c22983d5c185 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:14.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.113+0000 7efcbaffd640 1 -- 172.21.15.83:0/2965838548 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb4039070 con 0x7efcbc15ecd0 2024-03-06T07:22:14.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.113+0000 7efcbaffd640 1 -- 172.21.15.83:0/2965838548 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcb402cc60 con 0x7efcbc15ecd0 2024-03-06T07:22:14.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.113+0000 7efcbaffd640 1 -- 172.21.15.83:0/2965838548 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb4034660 con 0x7efcbc15ecd0 2024-03-06T07:22:14.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.114+0000 7efcc28d4640 1 -- 172.21.15.83:0/2965838548 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcbc15ecd0 msgr2=0x7efcbc0cae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.114+0000 7efcc28d4640 1 --2- 172.21.15.83:0/2965838548 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcbc15ecd0 0x7efcbc0cae80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7efcb4009d90 tx=0x7efcb402beb0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.114+0000 7efcc28d4640 1 -- 172.21.15.83:0/2965838548 shutdown_connections 2024-03-06T07:22:14.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.114+0000 7efcc28d4640 1 --2- 172.21.15.83:0/2965838548 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcbc15ecd0 0x7efcbc0cae80 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.114+0000 7efcc28d4640 1 -- 172.21.15.83:0/2965838548 >> 172.21.15.83:0/2965838548 conn(0x7efcbc0b3800 msgr2=0x7efcbc0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:14.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.114+0000 7efcc28d4640 1 -- 172.21.15.83:0/2965838548 shutdown_connections 2024-03-06T07:22:14.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.114+0000 7efcc28d4640 1 -- 172.21.15.83:0/2965838548 wait complete. 2024-03-06T07:22:14.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.115+0000 7efcc28d4640 1 Processor -- start 2024-03-06T07:22:14.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.115+0000 7efcc28d4640 1 -- start start 2024-03-06T07:22:14.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.116+0000 7efcc28d4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efcbc0d10c0 0x7efcbc147200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.116+0000 7efcc28d4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcbc0d14a0 0x7efcbc0b48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.116+0000 7efcc28d4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efcbc147aa0 0x7efcbc0b4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.116+0000 7efcc28d4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcbc0cfcf0 con 0x7efcbc0d14a0 2024-03-06T07:22:14.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.116+0000 7efcc28d4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcbc0d0060 con 0x7efcbc147aa0 2024-03-06T07:22:14.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.116+0000 7efcc28d4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcbc0cb3c0 con 0x7efcbc0d10c0 2024-03-06T07:22:14.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.116+0000 7efcbb7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcbc0d14a0 0x7efcbc0b48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.116+0000 7efcbb7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcbc0d14a0 0x7efcbc0b48f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49950/0 (socket says 172.21.15.83:49950) 2024-03-06T07:22:14.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.116+0000 7efcbb7fe640 1 -- 172.21.15.83:0/2852569062 learned_addr learned my addr 172.21.15.83:0/2852569062 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:14.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.116+0000 7efcc0e71640 1 --2- 172.21.15.83:0/2852569062 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efcbc147aa0 0x7efcbc0b4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.116+0000 7efcbbfff640 1 --2- 172.21.15.83:0/2852569062 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efcbc0d10c0 0x7efcbc147200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.116+0000 7efcbb7fe640 1 -- 172.21.15.83:0/2852569062 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efcbc0d10c0 msgr2=0x7efcbc147200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.116+0000 7efcbb7fe640 1 --2- 172.21.15.83:0/2852569062 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efcbc0d10c0 0x7efcbc147200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.116+0000 7efcbb7fe640 1 -- 172.21.15.83:0/2852569062 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efcbc147aa0 msgr2=0x7efcbc0b4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.116+0000 7efcbb7fe640 1 --2- 172.21.15.83:0/2852569062 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efcbc147aa0 0x7efcbc0b4e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.116+0000 7efcbb7fe640 1 -- 172.21.15.83:0/2852569062 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcbc0b7a20 con 0x7efcbc0d14a0 2024-03-06T07:22:14.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.116+0000 7efcbbfff640 1 --2- 172.21.15.83:0/2852569062 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efcbc0d10c0 0x7efcbc147200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:14.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.116+0000 7efcbb7fe640 1 --2- 172.21.15.83:0/2852569062 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcbc0d14a0 0x7efcbc0b48f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7efcac00c430 tx=0x7efcac00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:14.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.116+0000 7efcb8ff9640 1 -- 172.21.15.83:0/2852569062 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcac011790 con 0x7efcbc0d14a0 2024-03-06T07:22:14.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.117+0000 7efcb8ff9640 1 -- 172.21.15.83:0/2852569062 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcac017070 con 0x7efcbc0d14a0 2024-03-06T07:22:14.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.117+0000 7efcc28d4640 1 -- 172.21.15.83:0/2852569062 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcbc0b7740 con 0x7efcbc0d14a0 2024-03-06T07:22:14.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.117+0000 7efcb8ff9640 1 -- 172.21.15.83:0/2852569062 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcac011bb0 con 0x7efcbc0d14a0 2024-03-06T07:22:14.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.117+0000 7efcc28d4640 1 -- 172.21.15.83:0/2852569062 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcbc0aeeb0 con 0x7efcbc0d14a0 2024-03-06T07:22:14.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.118+0000 7efcc28d4640 1 -- 172.21.15.83:0/2852569062 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcbc09fe60 con 0x7efcbc0d14a0 2024-03-06T07:22:14.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.118+0000 7efcb8ff9640 1 -- 172.21.15.83:0/2852569062 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efcac01c460 con 0x7efcbc0d14a0 2024-03-06T07:22:14.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.118+0000 7efcb8ff9640 1 --2- 172.21.15.83:0/2852569062 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efca806b4d0 0x7efca806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.118+0000 7efcb8ff9640 1 -- 172.21.15.83:0/2852569062 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efcac08a0e0 con 0x7efcbc0d14a0 2024-03-06T07:22:14.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.118+0000 7efcbbfff640 1 --2- 172.21.15.83:0/2852569062 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efca806b4d0 0x7efca806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.119+0000 7efcbbfff640 1 --2- 172.21.15.83:0/2852569062 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efca806b4d0 0x7efca806d990 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7efcb0000f60 tx=0x7efcb0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:14.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.127+0000 7efcb8ff9640 1 -- 172.21.15.83:0/2852569062 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efcac058f60 con 0x7efcbc0d14a0 2024-03-06T07:22:14.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.299+0000 7efcc28d4640 1 -- 172.21.15.83:0/2852569062 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7efcbc0d0c30 con 0x7efcbc0d14a0 2024-03-06T07:22:14.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.299+0000 7efcb8ff9640 1 -- 172.21.15.83:0/2852569062 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7efcac058860 con 0x7efcbc0d14a0 2024-03-06T07:22:14.301 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 64 2024-03-06T07:22:14.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.301+0000 7efcc28d4640 1 -- 172.21.15.83:0/2852569062 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efca806b4d0 msgr2=0x7efca806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.301+0000 7efcc28d4640 1 --2- 172.21.15.83:0/2852569062 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efca806b4d0 0x7efca806d990 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7efcb0000f60 tx=0x7efcb0009040 comp rx=0 tx=0).stop 2024-03-06T07:22:14.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.301+0000 7efcc28d4640 1 -- 172.21.15.83:0/2852569062 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcbc0d14a0 msgr2=0x7efcbc0b48f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.301+0000 7efcc28d4640 1 --2- 172.21.15.83:0/2852569062 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcbc0d14a0 0x7efcbc0b48f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7efcac00c430 tx=0x7efcac00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.301+0000 7efcc28d4640 1 -- 172.21.15.83:0/2852569062 shutdown_connections 2024-03-06T07:22:14.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.301+0000 7efcc28d4640 1 --2- 172.21.15.83:0/2852569062 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efca806b4d0 0x7efca806d990 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.301+0000 7efcc28d4640 1 --2- 172.21.15.83:0/2852569062 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efcbc147aa0 0x7efcbc0b4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.301+0000 7efcc28d4640 1 --2- 172.21.15.83:0/2852569062 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcbc0d14a0 0x7efcbc0b48f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.301+0000 7efcc28d4640 1 --2- 172.21.15.83:0/2852569062 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efcbc0d10c0 0x7efcbc147200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.301+0000 7efcc28d4640 1 -- 172.21.15.83:0/2852569062 >> 172.21.15.83:0/2852569062 conn(0x7efcbc0b3800 msgr2=0x7efcbc15c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:14.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.301+0000 7efcc28d4640 1 -- 172.21.15.83:0/2852569062 shutdown_connections 2024-03-06T07:22:14.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.301+0000 7efcc28d4640 1 -- 172.21.15.83:0/2852569062 wait complete. 2024-03-06T07:22:14.314 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:14.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.414+0000 7f6652a84640 1 Processor -- start 2024-03-06T07:22:14.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.414+0000 7f6652a84640 1 -- start start 2024-03-06T07:22:14.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.415+0000 7f6652a84640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f664c15fa10 0x7f664c0ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.415+0000 7f6652a84640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f664c0d16e0 0x7f664c0d1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.415+0000 7f6652a84640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f664c0cebe0 con 0x7f664c15bdd0 2024-03-06T07:22:14.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.415+0000 7f6652a84640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f664c0ceec0 con 0x7f664c15fa10 2024-03-06T07:22:14.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.415+0000 7f6652a84640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f664c0cf1a0 con 0x7f664c0d16e0 2024-03-06T07:22:14.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.415+0000 7f6650820640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f664c15bdd0 0x7f664c0c5610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43036/0 (socket says 172.21.15.83:43036) 2024-03-06T07:22:14.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.415+0000 7f6650820640 1 -- 172.21.15.83:0/41511271 learned_addr learned my addr 172.21.15.83:0/41511271 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:14.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.415+0000 7f664bfff640 1 --2- 172.21.15.83:0/41511271 >> v2:172.21.15.83:3300/0 conn(0x7f664c15fa10 0x7f664c0ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.415+0000 7f664b7fe640 1 -- 172.21.15.83:0/41511271 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4247395308 0 0) 0x7f664c0cebe0 con 0x7f664c15bdd0 2024-03-06T07:22:14.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.415+0000 7f6651021640 1 --2- 172.21.15.83:0/41511271 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f664c0d16e0 0x7f664c0d1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.415+0000 7f664b7fe640 1 -- 172.21.15.83:0/41511271 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6634003520 con 0x7f664c15bdd0 2024-03-06T07:22:14.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.415+0000 7f664bfff640 1 -- 172.21.15.83:0/41511271 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f664c0d16e0 msgr2=0x7f664c0d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:14.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.415+0000 7f664bfff640 1 --2- 172.21.15.83:0/41511271 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f664c0d16e0 0x7f664c0d1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.415+0000 7f664bfff640 1 -- 172.21.15.83:0/41511271 >> v1:172.21.15.83:6789/0 conn(0x7f664c15bdd0 legacy=0x7f664c0c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.415+0000 7f664bfff640 1 -- 172.21.15.83:0/41511271 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f664c0cc8e0 con 0x7f664c15fa10 2024-03-06T07:22:14.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.416+0000 7f664bfff640 1 --2- 172.21.15.83:0/41511271 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f664c15fa10 0x7f664c0ce270 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f66400097a0 tx=0x7f664002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e51f83285b3d1eb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:14.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.416+0000 7f664b7fe640 1 -- 172.21.15.83:0/41511271 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66400315a0 con 0x7f664c15fa10 2024-03-06T07:22:14.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.416+0000 7f664b7fe640 1 -- 172.21.15.83:0/41511271 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6640037070 con 0x7f664c15fa10 2024-03-06T07:22:14.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.416+0000 7f664b7fe640 1 -- 172.21.15.83:0/41511271 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6640031940 con 0x7f664c15fa10 2024-03-06T07:22:14.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.417+0000 7f6652a84640 1 -- 172.21.15.83:0/41511271 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f664c15fa10 msgr2=0x7f664c0ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.417+0000 7f6652a84640 1 --2- 172.21.15.83:0/41511271 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f664c15fa10 0x7f664c0ce270 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f66400097a0 tx=0x7f664002b840 comp rx=0 tx=0).stop 2024-03-06T07:22:14.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.417+0000 7f6652a84640 1 -- 172.21.15.83:0/41511271 shutdown_connections 2024-03-06T07:22:14.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.417+0000 7f6652a84640 1 --2- 172.21.15.83:0/41511271 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f664c0d16e0 0x7f664c0d1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.417+0000 7f6652a84640 1 --2- 172.21.15.83:0/41511271 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f664c15fa10 0x7f664c0ce270 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.417+0000 7f6652a84640 1 -- 172.21.15.83:0/41511271 >> 172.21.15.83:0/41511271 conn(0x7f664c09a9f0 msgr2=0x7f664c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:14.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.417+0000 7f6652a84640 1 -- 172.21.15.83:0/41511271 shutdown_connections 2024-03-06T07:22:14.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.417+0000 7f6652a84640 1 -- 172.21.15.83:0/41511271 wait complete. 2024-03-06T07:22:14.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.418+0000 7f6652a84640 1 Processor -- start 2024-03-06T07:22:14.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.418+0000 7f6652a84640 1 -- start start 2024-03-06T07:22:14.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.419+0000 7f6652a84640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f664c0d16e0 0x7f664c0b5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.419+0000 7f6652a84640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f664c15bdd0 0x7f664c152e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.419+0000 7f6652a84640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f664c15fa10 0x7f664c0b7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.419+0000 7f6652a84640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f664c15c1b0 con 0x7f664c15bdd0 2024-03-06T07:22:14.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.419+0000 7f6652a84640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f664c0cc5d0 con 0x7f664c0d16e0 2024-03-06T07:22:14.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.419+0000 7f6652a84640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f664c0ce7b0 con 0x7f664c15fa10 2024-03-06T07:22:14.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.419+0000 7f664bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f664c15bdd0 0x7f664c152e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.419+0000 7f664bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f664c15bdd0 0x7f664c152e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49974/0 (socket says 172.21.15.83:49974) 2024-03-06T07:22:14.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.419+0000 7f664bfff640 1 -- 172.21.15.83:0/3742263251 learned_addr learned my addr 172.21.15.83:0/3742263251 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:14.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.419+0000 7f6650820640 1 --2- 172.21.15.83:0/3742263251 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f664c0d16e0 0x7f664c0b5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.419+0000 7f6651021640 1 --2- 172.21.15.83:0/3742263251 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f664c15fa10 0x7f664c0b7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.419+0000 7f664bfff640 1 -- 172.21.15.83:0/3742263251 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f664c15fa10 msgr2=0x7f664c0b7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.419+0000 7f664bfff640 1 --2- 172.21.15.83:0/3742263251 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f664c15fa10 0x7f664c0b7040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.419+0000 7f664bfff640 1 -- 172.21.15.83:0/3742263251 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f664c0d16e0 msgr2=0x7f664c0b5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.419+0000 7f664bfff640 1 --2- 172.21.15.83:0/3742263251 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f664c0d16e0 0x7f664c0b5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.419+0000 7f664bfff640 1 -- 172.21.15.83:0/3742263251 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f664c0b5af0 con 0x7f664c15bdd0 2024-03-06T07:22:14.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.419+0000 7f664bfff640 1 --2- 172.21.15.83:0/3742263251 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f664c15bdd0 0x7f664c152e40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f66400098c0 tx=0x7f66400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:14.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.419+0000 7f66497fa640 1 -- 172.21.15.83:0/3742263251 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6640006e10 con 0x7f664c15bdd0 2024-03-06T07:22:14.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.419+0000 7f66497fa640 1 -- 172.21.15.83:0/3742263251 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66400317a0 con 0x7f664c15bdd0 2024-03-06T07:22:14.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.420+0000 7f66497fa640 1 -- 172.21.15.83:0/3742263251 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6640031b90 con 0x7f664c15bdd0 2024-03-06T07:22:14.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.420+0000 7f6652a84640 1 -- 172.21.15.83:0/3742263251 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f664c0b5dd0 con 0x7f664c15bdd0 2024-03-06T07:22:14.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.420+0000 7f6652a84640 1 -- 172.21.15.83:0/3742263251 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f664c0b6390 con 0x7f664c15bdd0 2024-03-06T07:22:14.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.421+0000 7f6652a84640 1 -- 172.21.15.83:0/3742263251 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f664c04f330 con 0x7f664c15bdd0 2024-03-06T07:22:14.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.421+0000 7f66497fa640 1 -- 172.21.15.83:0/3742263251 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6640031d30 con 0x7f664c15bdd0 2024-03-06T07:22:14.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.421+0000 7f66497fa640 1 --2- 172.21.15.83:0/3742263251 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f663806b4d0 0x7f663806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.422+0000 7f66497fa640 1 -- 172.21.15.83:0/3742263251 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f664002f030 con 0x7f664c15bdd0 2024-03-06T07:22:14.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.422+0000 7f6650820640 1 --2- 172.21.15.83:0/3742263251 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f663806b4d0 0x7f663806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.422+0000 7f6650820640 1 --2- 172.21.15.83:0/3742263251 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f663806b4d0 0x7f663806d990 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f663c005880 tx=0x7f663c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:14.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.426+0000 7f66497fa640 1 -- 172.21.15.83:0/3742263251 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f664004c060 con 0x7f664c15bdd0 2024-03-06T07:22:14.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.692+0000 7f6652a84640 1 -- 172.21.15.83:0/3742263251 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f664c0b7a40 con 0x7f664c15bdd0 2024-03-06T07:22:14.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.693+0000 7f66497fa640 1 -- 172.21.15.83:0/3742263251 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4974 (secure 0 0 0) 0x7f664003a380 con 0x7f664c15bdd0 2024-03-06T07:22:14.695 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:22:14.695 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/4249608353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":4249608353},{"type":"v1","addr":"172.21.15.83:6836","nonce":4249608353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/714452530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":714452530},{"type":"v1","addr":"172.21.15.183:6835","nonce":714452530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4282,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/2723013431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2723013431},{"type":"v1","addr":"172.21.15.183:6834","nonce":2723013431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:58.286994+0000","modified":"2024-03-06T07:22:00.307284+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/1597344199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1597344199},{"type":"v1","addr":"172.21.15.83:6837","nonce":1597344199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4287,"qdb_cluster":[4287]},"id":1}]} 2024-03-06T07:22:14.695 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 6 2024-03-06T07:22:14.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.695+0000 7f6652a84640 1 -- 172.21.15.83:0/3742263251 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f663806b4d0 msgr2=0x7f663806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.695+0000 7f6652a84640 1 --2- 172.21.15.83:0/3742263251 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f663806b4d0 0x7f663806d990 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f663c005880 tx=0x7f663c009040 comp rx=0 tx=0).stop 2024-03-06T07:22:14.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.695+0000 7f6652a84640 1 -- 172.21.15.83:0/3742263251 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f664c15bdd0 msgr2=0x7f664c152e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.695+0000 7f6652a84640 1 --2- 172.21.15.83:0/3742263251 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f664c15bdd0 0x7f664c152e40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f66400098c0 tx=0x7f66400046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.695+0000 7f6652a84640 1 -- 172.21.15.83:0/3742263251 shutdown_connections 2024-03-06T07:22:14.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.695+0000 7f6652a84640 1 --2- 172.21.15.83:0/3742263251 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f663806b4d0 0x7f663806d990 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.695+0000 7f6652a84640 1 --2- 172.21.15.83:0/3742263251 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f664c15fa10 0x7f664c0b7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.695+0000 7f6652a84640 1 --2- 172.21.15.83:0/3742263251 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f664c15bdd0 0x7f664c152e40 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.695+0000 7f6652a84640 1 --2- 172.21.15.83:0/3742263251 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f664c0d16e0 0x7f664c0b5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.695+0000 7f6652a84640 1 -- 172.21.15.83:0/3742263251 >> 172.21.15.83:0/3742263251 conn(0x7f664c09a9f0 msgr2=0x7f664c15eb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:14.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.695+0000 7f6652a84640 1 -- 172.21.15.83:0/3742263251 shutdown_connections 2024-03-06T07:22:14.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.695+0000 7f6652a84640 1 -- 172.21.15.83:0/3742263251 wait complete. 2024-03-06T07:22:14.711 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:14.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.809+0000 7fa8cc288640 1 Processor -- start 2024-03-06T07:22:14.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.809+0000 7fa8cc288640 1 -- start start 2024-03-06T07:22:14.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.810+0000 7fa8cc288640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8c415bdd0 0x7fa8c40c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.810+0000 7fa8cc288640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8c40cebe0 con 0x7fa8c415fa10 2024-03-06T07:22:14.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.810+0000 7fa8cc288640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8c40ceec0 con 0x7fa8c40d16e0 2024-03-06T07:22:14.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.810+0000 7fa8cc288640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c40ce980 con 0x7fa8c415bdd0 2024-03-06T07:22:14.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.810+0000 7fa8c9823640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fa8c415fa10 0x7fa8c40ce270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43050/0 (socket says 172.21.15.83:43050) 2024-03-06T07:22:14.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.810+0000 7fa8c9823640 1 -- 172.21.15.83:0/1500361762 learned_addr learned my addr 172.21.15.83:0/1500361762 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:14.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.810+0000 7fa8c9022640 1 -- 172.21.15.83:0/1500361762 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3623170275 0 0) 0x7fa8c40cebe0 con 0x7fa8c415fa10 2024-03-06T07:22:14.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.810+0000 7fa8ca024640 1 --2- 172.21.15.83:0/1500361762 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8c415bdd0 0x7fa8c40c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.810+0000 7fa8c9022640 1 -- 172.21.15.83:0/1500361762 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8b0003520 con 0x7fa8c415fa10 2024-03-06T07:22:14.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.810+0000 7fa8c9022640 1 -- 172.21.15.83:0/1500361762 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1722471527 0 0) 0x7fa8b0003520 con 0x7fa8c415fa10 2024-03-06T07:22:14.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.810+0000 7fa8c9022640 1 -- 172.21.15.83:0/1500361762 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8c415bdd0 msgr2=0x7fa8c40c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:14.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.810+0000 7fa8c9022640 1 --2- 172.21.15.83:0/1500361762 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8c415bdd0 0x7fa8c40c5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.810+0000 7fa8c9022640 1 -- 172.21.15.83:0/1500361762 >> v1:172.21.15.183:6789/0 conn(0x7fa8c40d16e0 legacy=0x7fa8c40d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.810+0000 7fa8c9022640 1 -- 172.21.15.83:0/1500361762 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c40cf1a0 con 0x7fa8c415fa10 2024-03-06T07:22:14.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.810+0000 7fa8c9022640 1 -- 172.21.15.83:0/1500361762 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7fa8b4001670 con 0x7fa8c415fa10 2024-03-06T07:22:14.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.810+0000 7fa8c9022640 1 -- 172.21.15.83:0/1500361762 >> v1:172.21.15.83:6789/0 conn(0x7fa8c415fa10 legacy=0x7fa8c40ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.810+0000 7fa8c9022640 1 --2- 172.21.15.83:0/1500361762 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8b00039e0 0x7fa8b0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.810+0000 7fa8c9022640 1 --2- 172.21.15.83:0/1500361762 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8c415bdd0 0x7fa8b0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.810+0000 7fa8c9022640 1 --2- 172.21.15.83:0/1500361762 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8b0004d40 0x7fa8b00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.810+0000 7fa8c9022640 1 -- 172.21.15.83:0/1500361762 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c40ce980 con 0x7fa8b00039e0 2024-03-06T07:22:14.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.810+0000 7fa8c9022640 1 -- 172.21.15.83:0/1500361762 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b000ba70 con 0x7fa8b0004d40 2024-03-06T07:22:14.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.810+0000 7fa8c9022640 1 -- 172.21.15.83:0/1500361762 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b000bc70 con 0x7fa8c415bdd0 2024-03-06T07:22:14.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.810+0000 7fa8ca024640 1 --2- 172.21.15.83:0/1500361762 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8b00039e0 0x7fa8b0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.811+0000 7fa8c9823640 1 --2- 172.21.15.83:0/1500361762 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8c415bdd0 0x7fa8b0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.811+0000 7fa8ca825640 1 --2- 172.21.15.83:0/1500361762 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8b0004d40 0x7fa8b00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.811+0000 7fa8ca024640 1 -- 172.21.15.83:0/1500361762 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8c415bdd0 msgr2=0x7fa8b0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:14.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.811+0000 7fa8ca024640 1 --2- 172.21.15.83:0/1500361762 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8c415bdd0 0x7fa8b0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.811+0000 7fa8ca024640 1 -- 172.21.15.83:0/1500361762 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8b0004d40 msgr2=0x7fa8b00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:14.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.811+0000 7fa8ca024640 1 --2- 172.21.15.83:0/1500361762 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8b0004d40 0x7fa8b00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.811+0000 7fa8ca024640 1 -- 172.21.15.83:0/1500361762 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b0003520 con 0x7fa8b00039e0 2024-03-06T07:22:14.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.811+0000 7fa8c9823640 1 --2- 172.21.15.83:0/1500361762 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8c415bdd0 0x7fa8b0004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:14.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.811+0000 7fa8ca825640 1 --2- 172.21.15.83:0/1500361762 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8b0004d40 0x7fa8b00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:14.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.811+0000 7fa8ca024640 1 --2- 172.21.15.83:0/1500361762 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8b00039e0 0x7fa8b0003dc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0009ac0 tx=0x7fa8b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d54558fe22b346d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:14.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.812+0000 7fa8c9022640 1 -- 172.21.15.83:0/1500361762 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c0039070 con 0x7fa8b00039e0 2024-03-06T07:22:14.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.812+0000 7fa8c9022640 1 -- 172.21.15.83:0/1500361762 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c002c6b0 con 0x7fa8b00039e0 2024-03-06T07:22:14.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.812+0000 7fa8c9022640 1 -- 172.21.15.83:0/1500361762 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c0009690 con 0x7fa8b00039e0 2024-03-06T07:22:14.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.812+0000 7fa8cc288640 1 -- 172.21.15.83:0/1500361762 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8b00039e0 msgr2=0x7fa8b0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.812+0000 7fa8cc288640 1 --2- 172.21.15.83:0/1500361762 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8b00039e0 0x7fa8b0003dc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0009ac0 tx=0x7fa8b0005830 comp rx=0 tx=0).stop 2024-03-06T07:22:14.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.812+0000 7fa8ca825640 1 -- 172.21.15.83:0/1500361762 reap_dead start 2024-03-06T07:22:14.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.812+0000 7fa8cc288640 1 -- 172.21.15.83:0/1500361762 shutdown_connections 2024-03-06T07:22:14.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.812+0000 7fa8cc288640 1 -- 172.21.15.83:0/1500361762 >> 172.21.15.83:0/1500361762 conn(0x7fa8c409a9f0 msgr2=0x7fa8c409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:14.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.812+0000 7fa8cc288640 1 -- 172.21.15.83:0/1500361762 shutdown_connections 2024-03-06T07:22:14.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.812+0000 7fa8cc288640 1 -- 172.21.15.83:0/1500361762 wait complete. 2024-03-06T07:22:14.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.813+0000 7fa8cc288640 1 Processor -- start 2024-03-06T07:22:14.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.813+0000 7fa8cc288640 1 -- start start 2024-03-06T07:22:14.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.814+0000 7fa8cc288640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8b00039e0 0x7fa8c40c8c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.814+0000 7fa8cc288640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8c414a380 0x7fa8c414a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.814+0000 7fa8cc288640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8c41475d0 0x7fa8c4147a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.814+0000 7fa8cc288640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c415c1b0 con 0x7fa8c414a380 2024-03-06T07:22:14.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.814+0000 7fa8cc288640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c40cc5d0 con 0x7fa8b00039e0 2024-03-06T07:22:14.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.814+0000 7fa8cc288640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c40c5610 con 0x7fa8c41475d0 2024-03-06T07:22:14.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.814+0000 7fa8c9823640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8c414a380 0x7fa8c414a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.814+0000 7fa8c9823640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8c414a380 0x7fa8c414a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50000/0 (socket says 172.21.15.83:50000) 2024-03-06T07:22:14.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.814+0000 7fa8c9823640 1 -- 172.21.15.83:0/110564974 learned_addr learned my addr 172.21.15.83:0/110564974 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:14.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.814+0000 7fa8ca024640 1 --2- 172.21.15.83:0/110564974 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8b00039e0 0x7fa8c40c8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.814+0000 7fa8ca825640 1 --2- 172.21.15.83:0/110564974 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8c41475d0 0x7fa8c4147a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.814+0000 7fa8c9823640 1 -- 172.21.15.83:0/110564974 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8c41475d0 msgr2=0x7fa8c4147a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.814+0000 7fa8c9823640 1 --2- 172.21.15.83:0/110564974 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8c41475d0 0x7fa8c4147a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.814+0000 7fa8c9823640 1 -- 172.21.15.83:0/110564974 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8b00039e0 msgr2=0x7fa8c40c8c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.814+0000 7fa8c9823640 1 --2- 172.21.15.83:0/110564974 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8b00039e0 0x7fa8c40c8c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.814+0000 7fa8c9823640 1 -- 172.21.15.83:0/110564974 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c414ad40 con 0x7fa8c414a380 2024-03-06T07:22:14.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.814+0000 7fa8ca024640 1 --2- 172.21.15.83:0/110564974 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8b00039e0 0x7fa8c40c8c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:14.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.814+0000 7fa8ca825640 1 --2- 172.21.15.83:0/110564974 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8c41475d0 0x7fa8c4147a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:14.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.815+0000 7fa8c9823640 1 --2- 172.21.15.83:0/110564974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8c414a380 0x7fa8c414a800 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa8b400e7b0 tx=0x7fa8b400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:14.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.815+0000 7fa8baffd640 1 -- 172.21.15.83:0/110564974 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b400c8d0 con 0x7fa8c414a380 2024-03-06T07:22:14.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.815+0000 7fa8baffd640 1 -- 172.21.15.83:0/110564974 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8b4018070 con 0x7fa8c414a380 2024-03-06T07:22:14.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.815+0000 7fa8cc288640 1 -- 172.21.15.83:0/110564974 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8c40c9160 con 0x7fa8c414a380 2024-03-06T07:22:14.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.815+0000 7fa8baffd640 1 -- 172.21.15.83:0/110564974 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b400cc90 con 0x7fa8c414a380 2024-03-06T07:22:14.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.815+0000 7fa8cc288640 1 -- 172.21.15.83:0/110564974 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8c40b6660 con 0x7fa8c414a380 2024-03-06T07:22:14.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.816+0000 7fa8cc288640 1 -- 172.21.15.83:0/110564974 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8c404f330 con 0x7fa8c414a380 2024-03-06T07:22:14.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.816+0000 7fa8baffd640 1 -- 172.21.15.83:0/110564974 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa8b4013420 con 0x7fa8c414a380 2024-03-06T07:22:14.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.817+0000 7fa8baffd640 1 --2- 172.21.15.83:0/110564974 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa8a806b500 0x7fa8a806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.817+0000 7fa8ca024640 1 --2- 172.21.15.83:0/110564974 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa8a806b500 0x7fa8a806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.817+0000 7fa8baffd640 1 -- 172.21.15.83:0/110564974 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa8b4089dc0 con 0x7fa8c414a380 2024-03-06T07:22:14.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.817+0000 7fa8ca024640 1 --2- 172.21.15.83:0/110564974 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa8a806b500 0x7fa8a806d9c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0036000 tx=0x7fa8c0030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:14.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:14.821+0000 7fa8baffd640 1 -- 172.21.15.83:0/110564974 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa8b4058c40 con 0x7fa8c414a380 2024-03-06T07:22:15.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.003+0000 7fa8cc288640 1 -- 172.21.15.83:0/110564974 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8c40cbed0 con 0x7fa8c414a380 2024-03-06T07:22:15.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.005+0000 7fa8baffd640 1 -- 172.21.15.83:0/110564974 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16617 (secure 0 0 0) 0x7fa8b4058540 con 0x7fa8c414a380 2024-03-06T07:22:15.007 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:22:15.007 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":20,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:21:59.938137+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.318942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.254795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:15.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.007+0000 7fa8cc288640 1 -- 172.21.15.83:0/110564974 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa8a806b500 msgr2=0x7fa8a806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.007+0000 7fa8cc288640 1 --2- 172.21.15.83:0/110564974 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa8a806b500 0x7fa8a806d9c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0036000 tx=0x7fa8c0030000 comp rx=0 tx=0).stop 2024-03-06T07:22:15.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.007+0000 7fa8cc288640 1 -- 172.21.15.83:0/110564974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8c414a380 msgr2=0x7fa8c414a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.007+0000 7fa8cc288640 1 --2- 172.21.15.83:0/110564974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8c414a380 0x7fa8c414a800 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa8b400e7b0 tx=0x7fa8b400ec70 comp rx=0 tx=0).stop 2024-03-06T07:22:15.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.007+0000 7fa8cc288640 1 -- 172.21.15.83:0/110564974 shutdown_connections 2024-03-06T07:22:15.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.007+0000 7fa8cc288640 1 --2- 172.21.15.83:0/110564974 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa8a806b500 0x7fa8a806d9c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.007+0000 7fa8cc288640 1 --2- 172.21.15.83:0/110564974 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8c41475d0 0x7fa8c4147a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.007+0000 7fa8cc288640 1 --2- 172.21.15.83:0/110564974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8c414a380 0x7fa8c414a800 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.007+0000 7fa8cc288640 1 --2- 172.21.15.83:0/110564974 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8b00039e0 0x7fa8c40c8c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.007+0000 7fa8cc288640 1 -- 172.21.15.83:0/110564974 >> 172.21.15.83:0/110564974 conn(0x7fa8c409a9f0 msgr2=0x7fa8c406eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:15.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.007+0000 7fa8cc288640 1 -- 172.21.15.83:0/110564974 shutdown_connections 2024-03-06T07:22:15.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.007+0000 7fa8cc288640 1 -- 172.21.15.83:0/110564974 wait complete. 2024-03-06T07:22:15.024 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:15.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.123+0000 7fb302637640 1 Processor -- start 2024-03-06T07:22:15.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.123+0000 7fb302637640 1 -- start start 2024-03-06T07:22:15.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.124+0000 7fb302637640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb2fc15dfe0 0x7fb2fc0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.124+0000 7fb302637640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb2fc0cf830 0x7fb2fc0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.124+0000 7fb302637640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2fc0b3430 con 0x7fb2fc0d27f0 2024-03-06T07:22:15.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.124+0000 7fb302637640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2fc0d2c50 con 0x7fb2fc0cf830 2024-03-06T07:22:15.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.124+0000 7fb302637640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2fc0b31e0 con 0x7fb2fc15dfe0 2024-03-06T07:22:15.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.124+0000 7fb2fb7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb2fc0cf830 0x7fb2fc0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.124+0000 7fb2fb7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb2fc0cf830 0x7fb2fc0d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50014/0 (socket says 172.21.15.83:50014) 2024-03-06T07:22:15.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.124+0000 7fb2fb7fe640 1 -- 172.21.15.83:0/1932020603 learned_addr learned my addr 172.21.15.83:0/1932020603 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:15.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.124+0000 7fb2fbfff640 1 --2- 172.21.15.83:0/1932020603 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb2fc15dfe0 0x7fb2fc0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.124+0000 7fb2fb7fe640 1 -- 172.21.15.83:0/1932020603 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb2fc15dfe0 msgr2=0x7fb2fc0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:15.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.124+0000 7fb2fb7fe640 1 --2- 172.21.15.83:0/1932020603 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb2fc15dfe0 0x7fb2fc0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.124+0000 7fb2fb7fe640 1 -- 172.21.15.83:0/1932020603 >> v1:172.21.15.183:6789/0 conn(0x7fb2fc0d27f0 legacy=0x7fb2fc0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:15.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.124+0000 7fb2fb7fe640 1 -- 172.21.15.83:0/1932020603 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2fc0b3710 con 0x7fb2fc0cf830 2024-03-06T07:22:15.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.124+0000 7fb2fbfff640 1 --2- 172.21.15.83:0/1932020603 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb2fc15dfe0 0x7fb2fc0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:15.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.125+0000 7fb2fb7fe640 1 --2- 172.21.15.83:0/1932020603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2fc0cf830 0x7fb2fc0d22b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb2f0009870 tx=0x7fb2f002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb7e2753409aecbe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:15.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.125+0000 7fb2faffd640 1 -- 172.21.15.83:0/1932020603 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f0035020 con 0x7fb2fc0cf830 2024-03-06T07:22:15.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.125+0000 7fb2faffd640 1 -- 172.21.15.83:0/1932020603 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2f0031640 con 0x7fb2fc0cf830 2024-03-06T07:22:15.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.125+0000 7fb2faffd640 1 -- 172.21.15.83:0/1932020603 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f002d040 con 0x7fb2fc0cf830 2024-03-06T07:22:15.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.126+0000 7fb302637640 1 -- 172.21.15.83:0/1932020603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2fc0cf830 msgr2=0x7fb2fc0d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.126+0000 7fb302637640 1 --2- 172.21.15.83:0/1932020603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2fc0cf830 0x7fb2fc0d22b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb2f0009870 tx=0x7fb2f002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.126+0000 7fb302637640 1 -- 172.21.15.83:0/1932020603 shutdown_connections 2024-03-06T07:22:15.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.126+0000 7fb302637640 1 --2- 172.21.15.83:0/1932020603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2fc0cf830 0x7fb2fc0d22b0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.126+0000 7fb302637640 1 --2- 172.21.15.83:0/1932020603 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb2fc15dfe0 0x7fb2fc0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.126+0000 7fb302637640 1 -- 172.21.15.83:0/1932020603 >> 172.21.15.83:0/1932020603 conn(0x7fb2fc0d3cd0 msgr2=0x7fb2fc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:15.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.126+0000 7fb302637640 1 -- 172.21.15.83:0/1932020603 shutdown_connections 2024-03-06T07:22:15.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.126+0000 7fb302637640 1 -- 172.21.15.83:0/1932020603 wait complete. 2024-03-06T07:22:15.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.127+0000 7fb302637640 1 Processor -- start 2024-03-06T07:22:15.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.127+0000 7fb302637640 1 -- start start 2024-03-06T07:22:15.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.128+0000 7fb302637640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb2fc0cf830 0x7fb2fc0cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.128+0000 7fb302637640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2fc0d27f0 0x7fb2fc0b25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.128+0000 7fb302637640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb2fc15dfe0 0x7fb2fc0b7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.128+0000 7fb302637640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2fc0d1c70 con 0x7fb2fc0d27f0 2024-03-06T07:22:15.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.128+0000 7fb302637640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2fc0b3d60 con 0x7fb2fc15dfe0 2024-03-06T07:22:15.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.128+0000 7fb302637640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2fc0d1df0 con 0x7fb2fc0cf830 2024-03-06T07:22:15.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.128+0000 7fb2fb7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2fc0d27f0 0x7fb2fc0b25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.128+0000 7fb2fb7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2fc0d27f0 0x7fb2fc0b25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50026/0 (socket says 172.21.15.83:50026) 2024-03-06T07:22:15.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.128+0000 7fb2fb7fe640 1 -- 172.21.15.83:0/2041057406 learned_addr learned my addr 172.21.15.83:0/2041057406 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:15.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.128+0000 7fb2fbfff640 1 --2- 172.21.15.83:0/2041057406 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb2fc0cf830 0x7fb2fc0cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.128+0000 7fb2fb7fe640 1 -- 172.21.15.83:0/2041057406 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb2fc0cf830 msgr2=0x7fb2fc0cd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.128+0000 7fb2fb7fe640 1 --2- 172.21.15.83:0/2041057406 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb2fc0cf830 0x7fb2fc0cd8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.128+0000 7fb2fb7fe640 1 -- 172.21.15.83:0/2041057406 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb2fc15dfe0 msgr2=0x7fb2fc0b7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.128+0000 7fb300bd4640 1 --2- 172.21.15.83:0/2041057406 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb2fc15dfe0 0x7fb2fc0b7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.128+0000 7fb2fb7fe640 1 --2- 172.21.15.83:0/2041057406 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb2fc15dfe0 0x7fb2fc0b7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.128+0000 7fb2fb7fe640 1 -- 172.21.15.83:0/2041057406 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2fc15f340 con 0x7fb2fc0d27f0 2024-03-06T07:22:15.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.128+0000 7fb300bd4640 1 --2- 172.21.15.83:0/2041057406 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb2fc15dfe0 0x7fb2fc0b7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:15.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.128+0000 7fb2fb7fe640 1 --2- 172.21.15.83:0/2041057406 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2fc0d27f0 0x7fb2fc0b25b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb2f002b930 tx=0x7fb2f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:15.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.128+0000 7fb2f8ff9640 1 -- 172.21.15.83:0/2041057406 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f0040070 con 0x7fb2fc0d27f0 2024-03-06T07:22:15.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.128+0000 7fb2f8ff9640 1 -- 172.21.15.83:0/2041057406 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2f0039bf0 con 0x7fb2fc0d27f0 2024-03-06T07:22:15.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.128+0000 7fb302637640 1 -- 172.21.15.83:0/2041057406 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2fc15f620 con 0x7fb2fc0d27f0 2024-03-06T07:22:15.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.128+0000 7fb2f8ff9640 1 -- 172.21.15.83:0/2041057406 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f0035040 con 0x7fb2fc0d27f0 2024-03-06T07:22:15.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.128+0000 7fb302637640 1 -- 172.21.15.83:0/2041057406 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2fc160340 con 0x7fb2fc0d27f0 2024-03-06T07:22:15.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.129+0000 7fb302637640 1 -- 172.21.15.83:0/2041057406 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2fc04f330 con 0x7fb2fc0d27f0 2024-03-06T07:22:15.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.130+0000 7fb2f8ff9640 1 -- 172.21.15.83:0/2041057406 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb2f004b060 con 0x7fb2fc0d27f0 2024-03-06T07:22:15.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.130+0000 7fb2f8ff9640 1 --2- 172.21.15.83:0/2041057406 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb2e806b500 0x7fb2e806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.130+0000 7fb2f8ff9640 1 -- 172.21.15.83:0/2041057406 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb2f00abd60 con 0x7fb2fc0d27f0 2024-03-06T07:22:15.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.130+0000 7fb2fbfff640 1 --2- 172.21.15.83:0/2041057406 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb2e806b500 0x7fb2e806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.131+0000 7fb2fbfff640 1 --2- 172.21.15.83:0/2041057406 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb2e806b500 0x7fb2e806d9c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec00b840 tx=0x7fb2ec008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:15.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.139+0000 7fb2f8ff9640 1 -- 172.21.15.83:0/2041057406 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb2f002f300 con 0x7fb2fc0d27f0 2024-03-06T07:22:15.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.449+0000 7fb302637640 1 -- 172.21.15.83:0/2041057406 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2fc0d10f0 con 0x7fb2fc0d27f0 2024-03-06T07:22:15.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.449+0000 7fb2f8ff9640 1 -- 172.21.15.83:0/2041057406 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4974 (secure 0 0 0) 0x7fb2f007abe0 con 0x7fb2fc0d27f0 2024-03-06T07:22:15.451 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:22:15.452 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/4249608353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":4249608353},{"type":"v1","addr":"172.21.15.83:6836","nonce":4249608353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/714452530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":714452530},{"type":"v1","addr":"172.21.15.183:6835","nonce":714452530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4282,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/2723013431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2723013431},{"type":"v1","addr":"172.21.15.183:6834","nonce":2723013431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:58.286994+0000","modified":"2024-03-06T07:22:00.307284+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/1597344199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1597344199},{"type":"v1","addr":"172.21.15.83:6837","nonce":1597344199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4287,"qdb_cluster":[4287]},"id":1}]} 2024-03-06T07:22:15.452 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 6 2024-03-06T07:22:15.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.451+0000 7fb302637640 1 -- 172.21.15.83:0/2041057406 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb2e806b500 msgr2=0x7fb2e806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.451+0000 7fb302637640 1 --2- 172.21.15.83:0/2041057406 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb2e806b500 0x7fb2e806d9c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec00b840 tx=0x7fb2ec008040 comp rx=0 tx=0).stop 2024-03-06T07:22:15.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.451+0000 7fb302637640 1 -- 172.21.15.83:0/2041057406 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2fc0d27f0 msgr2=0x7fb2fc0b25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.451+0000 7fb302637640 1 --2- 172.21.15.83:0/2041057406 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2fc0d27f0 0x7fb2fc0b25b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb2f002b930 tx=0x7fb2f00046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.451+0000 7fb302637640 1 -- 172.21.15.83:0/2041057406 shutdown_connections 2024-03-06T07:22:15.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.451+0000 7fb302637640 1 --2- 172.21.15.83:0/2041057406 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb2e806b500 0x7fb2e806d9c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.451+0000 7fb302637640 1 --2- 172.21.15.83:0/2041057406 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb2fc15dfe0 0x7fb2fc0b7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.451+0000 7fb302637640 1 --2- 172.21.15.83:0/2041057406 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2fc0d27f0 0x7fb2fc0b25b0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.451+0000 7fb302637640 1 --2- 172.21.15.83:0/2041057406 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb2fc0cf830 0x7fb2fc0cd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.451+0000 7fb302637640 1 -- 172.21.15.83:0/2041057406 >> 172.21.15.83:0/2041057406 conn(0x7fb2fc0d3cd0 msgr2=0x7fb2fc0b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:15.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.451+0000 7fb302637640 1 -- 172.21.15.83:0/2041057406 shutdown_connections 2024-03-06T07:22:15.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.451+0000 7fb302637640 1 -- 172.21.15.83:0/2041057406 wait complete. 2024-03-06T07:22:15.869 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:21:58.286994+0000', 'modified': '2024-03-06T07:22:00.307284+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4287}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4287': {'gid': 4287, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.83:6837/1597344199', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1597344199}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1597344199}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4287, 'qdb_cluster': [4287]} 2024-03-06T07:22:15.870 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:22:15.870 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:22:15.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.973+0000 7fb87fa9d640 1 Processor -- start 2024-03-06T07:22:15.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.973+0000 7fb87fa9d640 1 -- start start 2024-03-06T07:22:15.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.974+0000 7fb87fa9d640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fb8781614a0 0x7fb87806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.974+0000 7fb87fa9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb8780b1a20 0x7fb8780b5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.974+0000 7fb87fa9d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8780af4b0 con 0x7fb8780b0270 2024-03-06T07:22:15.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.974+0000 7fb87fa9d640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8780af790 con 0x7fb8780b1a20 2024-03-06T07:22:15.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.974+0000 7fb87fa9d640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8780afa70 con 0x7fb8781614a0 2024-03-06T07:22:15.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.974+0000 7fb87e03a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb8780b1a20 0x7fb8780b5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.974+0000 7fb87e03a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb8780b1a20 0x7fb8780b5510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50036/0 (socket says 172.21.15.83:50036) 2024-03-06T07:22:15.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.974+0000 7fb87e03a640 1 -- 172.21.15.83:0/3275398463 learned_addr learned my addr 172.21.15.83:0/3275398463 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:15.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.975+0000 7fb87e03a640 1 -- 172.21.15.83:0/3275398463 >> v2:172.21.15.183:3300/0 conn(0x7fb8781614a0 msgr2=0x7fb87806d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:15.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.975+0000 7fb87e03a640 1 --2- 172.21.15.83:0/3275398463 >> v2:172.21.15.183:3300/0 conn(0x7fb8781614a0 0x7fb87806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.975+0000 7fb87e03a640 1 -- 172.21.15.83:0/3275398463 >> v1:172.21.15.183:6789/0 conn(0x7fb8780b0270 legacy=0x7fb8780b1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:15.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.975+0000 7fb87e03a640 1 -- 172.21.15.83:0/3275398463 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8780afd50 con 0x7fb8780b1a20 2024-03-06T07:22:15.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.975+0000 7fb87e03a640 1 --2- 172.21.15.83:0/3275398463 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb8780b1a20 0x7fb8780b5510 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb8780730b0 tx=0x7fb87002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ffa63c93f720bbc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:15.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.975+0000 7fb87c837640 1 -- 172.21.15.83:0/3275398463 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87002c9f0 con 0x7fb8780b1a20 2024-03-06T07:22:15.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.975+0000 7fb87c837640 1 -- 172.21.15.83:0/3275398463 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb87002cb90 con 0x7fb8780b1a20 2024-03-06T07:22:15.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.975+0000 7fb87c837640 1 -- 172.21.15.83:0/3275398463 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8700343f0 con 0x7fb8780b1a20 2024-03-06T07:22:15.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.976+0000 7fb87fa9d640 1 -- 172.21.15.83:0/3275398463 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb8780b1a20 msgr2=0x7fb8780b5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.976+0000 7fb87fa9d640 1 --2- 172.21.15.83:0/3275398463 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb8780b1a20 0x7fb8780b5510 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb8780730b0 tx=0x7fb87002be50 comp rx=0 tx=0).stop 2024-03-06T07:22:15.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.976+0000 7fb87fa9d640 1 -- 172.21.15.83:0/3275398463 shutdown_connections 2024-03-06T07:22:15.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.976+0000 7fb87fa9d640 1 --2- 172.21.15.83:0/3275398463 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb8780b1a20 0x7fb8780b5510 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.976+0000 7fb87fa9d640 1 --2- 172.21.15.83:0/3275398463 >> v2:172.21.15.183:3300/0 conn(0x7fb8781614a0 0x7fb87806d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.976+0000 7fb87fa9d640 1 -- 172.21.15.83:0/3275398463 >> 172.21.15.83:0/3275398463 conn(0x7fb87809a9f0 msgr2=0x7fb87809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:15.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.976+0000 7fb87fa9d640 1 -- 172.21.15.83:0/3275398463 shutdown_connections 2024-03-06T07:22:15.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.976+0000 7fb87fa9d640 1 -- 172.21.15.83:0/3275398463 wait complete. 2024-03-06T07:22:15.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.977+0000 7fb87fa9d640 1 Processor -- start 2024-03-06T07:22:15.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.977+0000 7fb87fa9d640 1 -- start start 2024-03-06T07:22:15.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.978+0000 7fb87fa9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb8780b0270 0x7fb878149a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.978+0000 7fb87fa9d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb8780b1a20 0x7fb8781f3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.978+0000 7fb87fa9d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb8781614a0 0x7fb87815bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.978+0000 7fb87fa9d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8780b1e00 con 0x7fb8780b0270 2024-03-06T07:22:15.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.978+0000 7fb87fa9d640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8780b06e0 con 0x7fb8780b1a20 2024-03-06T07:22:15.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.978+0000 7fb87fa9d640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8780b1f80 con 0x7fb8781614a0 2024-03-06T07:22:15.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.978+0000 7fb87d839640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb8780b0270 0x7fb878149a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.978+0000 7fb87d839640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb8780b0270 0x7fb878149a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50050/0 (socket says 172.21.15.83:50050) 2024-03-06T07:22:15.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.978+0000 7fb87d839640 1 -- 172.21.15.83:0/1900453791 learned_addr learned my addr 172.21.15.83:0/1900453791 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:15.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.978+0000 7fb87d839640 1 -- 172.21.15.83:0/1900453791 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb8781614a0 msgr2=0x7fb87815bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.978+0000 7fb87d839640 1 --2- 172.21.15.83:0/1900453791 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb8781614a0 0x7fb87815bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.978+0000 7fb87d839640 1 -- 172.21.15.83:0/1900453791 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb8780b1a20 msgr2=0x7fb8781f3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.978+0000 7fb87d038640 1 --2- 172.21.15.83:0/1900453791 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb8780b1a20 0x7fb8781f3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.978+0000 7fb87d839640 1 --2- 172.21.15.83:0/1900453791 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb8780b1a20 0x7fb8781f3a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.978+0000 7fb87d839640 1 -- 172.21.15.83:0/1900453791 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8780cc2b0 con 0x7fb8780b0270 2024-03-06T07:22:15.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.978+0000 7fb87d839640 1 --2- 172.21.15.83:0/1900453791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb8780b0270 0x7fb878149a50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb86800b410 tx=0x7fb86800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:15.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.979+0000 7fb86e7fc640 1 -- 172.21.15.83:0/1900453791 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb868015020 con 0x7fb8780b0270 2024-03-06T07:22:15.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.979+0000 7fb86e7fc640 1 -- 172.21.15.83:0/1900453791 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb868011850 con 0x7fb8780b0270 2024-03-06T07:22:15.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.979+0000 7fb87fa9d640 1 -- 172.21.15.83:0/1900453791 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8780cc590 con 0x7fb8780b0270 2024-03-06T07:22:15.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.979+0000 7fb86e7fc640 1 -- 172.21.15.83:0/1900453791 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb868011c40 con 0x7fb8780b0270 2024-03-06T07:22:15.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.979+0000 7fb87fa9d640 1 -- 172.21.15.83:0/1900453791 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8780ccb50 con 0x7fb8780b0270 2024-03-06T07:22:15.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.980+0000 7fb86e7fc640 1 -- 172.21.15.83:0/1900453791 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb868018020 con 0x7fb8780b0270 2024-03-06T07:22:15.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.980+0000 7fb86e7fc640 1 --2- 172.21.15.83:0/1900453791 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb85c06b4d0 0x7fb85c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.981+0000 7fb87d038640 1 --2- 172.21.15.83:0/1900453791 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb85c06b4d0 0x7fb85c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.981+0000 7fb86e7fc640 1 -- 172.21.15.83:0/1900453791 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb86808a110 con 0x7fb8780b0270 2024-03-06T07:22:15.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.981+0000 7fb87d038640 1 --2- 172.21.15.83:0/1900453791 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb85c06b4d0 0x7fb85c06d990 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb878070720 tx=0x7fb874008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:15.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.982+0000 7fb882766740 1 Processor -- start 2024-03-06T07:22:15.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.982+0000 7fb882766740 1 -- start start 2024-03-06T07:22:15.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.983+0000 7fb882766740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562936b3f870 0x562936b3fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.983+0000 7fb882766740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x562936b438c0 0x562936b45360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.983+0000 7fb882766740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562936b6f600 0x562936b6fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.983+0000 7fb882766740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x562936aa96b0 con 0x562936b3f870 2024-03-06T07:22:15.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.983+0000 7fb882766740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x562936a52a60 con 0x562936b6f600 2024-03-06T07:22:15.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.983+0000 7fb882766740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x562936aa5750 con 0x562936b438c0 2024-03-06T07:22:15.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.984+0000 7fb87d839640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562936b3f870 0x562936b3fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.984+0000 7fb87d839640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562936b3f870 0x562936b3fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50060/0 (socket says 172.21.15.83:50060) 2024-03-06T07:22:15.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.984+0000 7fb87d839640 1 -- 172.21.15.83:0/2218153038 learned_addr learned my addr 172.21.15.83:0/2218153038 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:15.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.984+0000 7fb87e03a640 1 --2- 172.21.15.83:0/2218153038 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562936b6f600 0x562936b6fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.984+0000 7fb87d038640 1 --2- 172.21.15.83:0/2218153038 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x562936b438c0 0x562936b45360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.984+0000 7fb87d839640 1 -- 172.21.15.83:0/2218153038 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x562936b438c0 msgr2=0x562936b45360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:15.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.984+0000 7fb87d839640 1 --2- 172.21.15.83:0/2218153038 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x562936b438c0 0x562936b45360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.984+0000 7fb87d839640 1 -- 172.21.15.83:0/2218153038 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562936b6f600 msgr2=0x562936b6fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:15.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.984+0000 7fb87d839640 1 --2- 172.21.15.83:0/2218153038 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562936b6f600 0x562936b6fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.984+0000 7fb87d839640 1 -- 172.21.15.83:0/2218153038 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562936a70f30 con 0x562936b3f870 2024-03-06T07:22:15.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.984+0000 7fb87e03a640 1 --2- 172.21.15.83:0/2218153038 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562936b6f600 0x562936b6fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:15.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.984+0000 7fb87d839640 1 --2- 172.21.15.83:0/2218153038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562936b3f870 0x562936b3fc50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb868088640 tx=0x7fb868059160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f9db9ff59c7f1a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:15.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.984+0000 7fb85b7fe640 1 -- 172.21.15.83:0/2218153038 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb868070020 con 0x562936b3f870 2024-03-06T07:22:15.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.984+0000 7fb85b7fe640 1 -- 172.21.15.83:0/2218153038 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb868086070 con 0x562936b3f870 2024-03-06T07:22:15.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.984+0000 7fb85b7fe640 1 -- 172.21.15.83:0/2218153038 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb868061030 con 0x562936b3f870 2024-03-06T07:22:15.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.985+0000 7fb882766740 1 -- 172.21.15.83:0/2218153038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562936b3f870 msgr2=0x562936b3fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.985+0000 7fb882766740 1 --2- 172.21.15.83:0/2218153038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562936b3f870 0x562936b3fc50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb868088640 tx=0x7fb868059160 comp rx=0 tx=0).stop 2024-03-06T07:22:15.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.985+0000 7fb882766740 1 -- 172.21.15.83:0/2218153038 shutdown_connections 2024-03-06T07:22:15.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.985+0000 7fb882766740 1 --2- 172.21.15.83:0/2218153038 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562936b6f600 0x562936b6fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.985+0000 7fb882766740 1 --2- 172.21.15.83:0/2218153038 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x562936b438c0 0x562936b45360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.985+0000 7fb882766740 1 --2- 172.21.15.83:0/2218153038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562936b3f870 0x562936b3fc50 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.985+0000 7fb882766740 1 -- 172.21.15.83:0/2218153038 >> 172.21.15.83:0/2218153038 conn(0x562936b418f0 msgr2=0x562936b431b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:15.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.985+0000 7fb882766740 1 -- 172.21.15.83:0/2218153038 shutdown_connections 2024-03-06T07:22:15.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.985+0000 7fb882766740 1 -- 172.21.15.83:0/2218153038 wait complete. 2024-03-06T07:22:15.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.986+0000 7fb882766740 1 -- start start 2024-03-06T07:22:15.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.986+0000 7fb882766740 1 Processor -- start 2024-03-06T07:22:15.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.987+0000 7fb882766740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562936b6f600 0x562936b68cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.987+0000 7fb882766740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x562936b64d00 0x562936b691f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.987+0000 7fb882766740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562936b87590 0x562936b87a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.987+0000 7fb882766740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x562936b43d50 con 0x562936b6f600 2024-03-06T07:22:15.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.987+0000 7fb882766740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x562936a65ea0 con 0x562936b87590 2024-03-06T07:22:15.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.987+0000 7fb882766740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x562936b44030 con 0x562936b64d00 2024-03-06T07:22:15.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.987+0000 7fb87d839640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562936b6f600 0x562936b68cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.987+0000 7fb882766740 10 client.0 _ll_register_callbacks cb 0x562936a65870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:22:15.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.987+0000 7fb87d839640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562936b6f600 0x562936b68cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50062/0 (socket says 172.21.15.83:50062) 2024-03-06T07:22:15.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.987+0000 7fb87d839640 1 -- 172.21.15.83:0/1335309519 learned_addr learned my addr 172.21.15.83:0/1335309519 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:15.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.987+0000 7fb87d839640 1 -- 172.21.15.83:0/1335309519 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x562936b64d00 msgr2=0x562936b691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:15.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.987+0000 7fb87d839640 1 --2- 172.21.15.83:0/1335309519 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x562936b64d00 0x562936b691f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.987+0000 7fb87d839640 1 -- 172.21.15.83:0/1335309519 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562936b87590 msgr2=0x562936b87a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:15.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.987+0000 7fb87d839640 1 --2- 172.21.15.83:0/1335309519 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562936b87590 0x562936b87a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.987+0000 7fb87d839640 1 -- 172.21.15.83:0/1335309519 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562936a66270 con 0x562936b6f600 2024-03-06T07:22:15.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.988+0000 7fb87d839640 1 --2- 172.21.15.83:0/1335309519 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562936b6f600 0x562936b68cb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb86800bdd0 tx=0x7fb86801c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16a4ed3ef1142a17 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:15.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.988+0000 7fb858ff9640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-06T07:22:15.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.988+0000 7fb858ff9640 1 -- 172.21.15.83:0/1335309519 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb868074030 con 0x562936b6f600 2024-03-06T07:22:15.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.988+0000 7fb858ff9640 1 -- 172.21.15.83:0/1335309519 <== mon.0 v2:172.21.15.83:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb868011de0 con 0x562936b6f600 2024-03-06T07:22:15.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.988+0000 7fb882766740 1 -- 172.21.15.83:0/1335309519 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb868086070 con 0x562936b6f600 2024-03-06T07:22:15.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.988+0000 7fb858ff9640 1 -- 172.21.15.83:0/1335309519 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb86807f050 con 0x562936b6f600 2024-03-06T07:22:15.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.988+0000 7fb882766740 10 client.4683 fetch_fsmap learned FSMap version 6 2024-03-06T07:22:15.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.988+0000 7fb882766740 1 -- 172.21.15.83:0/1335309519 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) v3 -- 0x5629367be2d0 con 0x562936b6f600 2024-03-06T07:22:15.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.988+0000 7fb858ff9640 1 -- 172.21.15.83:0/1335309519 <== mon.0 v2:172.21.15.83:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb868086070 con 0x562936b6f600 2024-03-06T07:22:15.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.988+0000 7fb858ff9640 1 -- 172.21.15.83:0/1335309519 <== mon.0 v2:172.21.15.83:3300/0 5 ==== fsmap(e 6) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fb868054f90 con 0x562936b6f600 2024-03-06T07:22:15.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.988+0000 7fb882766740 10 client.4683 fetch_fsmap finished waiting for FSMap version 6 2024-03-06T07:22:15.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.988+0000 7fb882766740 10 client.4683 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:22:15.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.988+0000 7fb882766740 20 client.4683 populate_metadata read hostname 'smithi083' 2024-03-06T07:22:15.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.988+0000 7fb882766740 1 --2- 172.21.15.83:0/1335309519 >> [v2:172.21.15.83:6835/1597344199,v1:172.21.15.83:6837/1597344199] conn(0x562936b72c30 0x562936b73030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.988+0000 7fb882766740 4 client.4683 mds_command: new command op to 4287 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:22:15.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.988+0000 7fb882766740 1 -- 172.21.15.83:0/1335309519 --> [v2:172.21.15.83:6835/1597344199,v1:172.21.15.83:6837/1597344199] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562936ab6bf0 con 0x562936b72c30 2024-03-06T07:22:15.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.988+0000 7fb87d038640 1 --2- 172.21.15.83:0/1335309519 >> [v2:172.21.15.83:6835/1597344199,v1:172.21.15.83:6837/1597344199] conn(0x562936b72c30 0x562936b73030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.988+0000 7fb858ff9640 10 client.4683 ms_handle_connect on v2:172.21.15.83:6835/1597344199 2024-03-06T07:22:15.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.988+0000 7fb87d038640 1 --2- 172.21.15.83:0/1335309519 >> [v2:172.21.15.83:6835/1597344199,v1:172.21.15.83:6837/1597344199] conn(0x562936b72c30 0x562936b73030 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4883d3ddeb872927 server_cookie=76f7cc40b1281011 in_seq=0 out_seq=0 2024-03-06T07:22:15.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.989+0000 7fb858ff9640 1 -- 172.21.15.83:0/1335309519 <== mds.0 v2:172.21.15.83:6835/1597344199 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562936ab6bf0 con 0x562936b72c30 2024-03-06T07:22:15.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.989+0000 7fb858ff9640 10 client.4683 handle_command_reply: tid=0 2024-03-06T07:22:15.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.989+0000 7fb882766740 1 client.4683 shutdown 2024-03-06T07:22:15.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.990+0000 7fb882766740 1 -- 172.21.15.83:0/1335309519 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562936b6f600 msgr2=0x562936b68cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.990+0000 7fb882766740 1 --2- 172.21.15.83:0/1335309519 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562936b6f600 0x562936b68cb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb86800bdd0 tx=0x7fb86801c040 comp rx=0 tx=0).stop 2024-03-06T07:22:15.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.990+0000 7fb882766740 1 -- 172.21.15.83:0/1335309519 shutdown_connections 2024-03-06T07:22:15.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.990+0000 7fb882766740 1 --2- 172.21.15.83:0/1335309519 >> [v2:172.21.15.83:6835/1597344199,v1:172.21.15.83:6837/1597344199] conn(0x562936b72c30 0x562936b73030 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.990+0000 7fb882766740 1 --2- 172.21.15.83:0/1335309519 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562936b87590 0x562936b87a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.990+0000 7fb858ff9640 0 client.4683 ms_handle_reset on v2:172.21.15.83:6835/1597344199 2024-03-06T07:22:15.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.990+0000 7fb882766740 1 --2- 172.21.15.83:0/1335309519 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x562936b64d00 0x562936b691f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.990+0000 7fb882766740 1 --2- 172.21.15.83:0/1335309519 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562936b6f600 0x562936b68cb0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.990+0000 7fb882766740 1 -- 172.21.15.83:0/1335309519 >> 172.21.15.83:0/1335309519 conn(0x562936b3f870 msgr2=0x562936b6d5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:16.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.990+0000 7fb882766740 1 -- 172.21.15.83:0/1335309519 shutdown_connections 2024-03-06T07:22:16.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.990+0000 7fb882766740 1 -- 172.21.15.83:0/1335309519 wait complete. 2024-03-06T07:22:16.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:15.990+0000 7fb882766740 20 client.4683 trim_cache size 0 max 16384 2024-03-06T07:22:16.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.007+0000 7fb882766740 1 Processor -- start 2024-03-06T07:22:16.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.007+0000 7fb882766740 1 -- start start 2024-03-06T07:22:16.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.008+0000 7fb882766740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x562936b64d00 0x562936b699d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.008+0000 7fb882766740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562936b6f600 0x562936b6f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.008+0000 7fb882766740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562936b87590 0x562936b6ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.008+0000 7fb882766740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8781ff3c0 con 0x562936b87590 2024-03-06T07:22:16.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.008+0000 7fb882766740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8780cba30 con 0x562936b6f600 2024-03-06T07:22:16.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.008+0000 7fb882766740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8780cdbe0 con 0x562936b64d00 2024-03-06T07:22:16.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.008+0000 7fb87e03a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562936b87590 0x562936b6ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.008+0000 7fb87e03a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562936b87590 0x562936b6ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50076/0 (socket says 172.21.15.83:50076) 2024-03-06T07:22:16.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.008+0000 7fb87e03a640 1 -- 172.21.15.83:0/4076409684 learned_addr learned my addr 172.21.15.83:0/4076409684 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:16.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.008+0000 7fb87d839640 1 --2- 172.21.15.83:0/4076409684 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x562936b64d00 0x562936b699d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.008+0000 7fb87d038640 1 --2- 172.21.15.83:0/4076409684 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562936b6f600 0x562936b6f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.008+0000 7fb87e03a640 1 -- 172.21.15.83:0/4076409684 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x562936b64d00 msgr2=0x562936b699d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:16.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.008+0000 7fb87e03a640 1 --2- 172.21.15.83:0/4076409684 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x562936b64d00 0x562936b699d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.008+0000 7fb87e03a640 1 -- 172.21.15.83:0/4076409684 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562936b6f600 msgr2=0x562936b6f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:16.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.008+0000 7fb87e03a640 1 --2- 172.21.15.83:0/4076409684 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562936b6f600 0x562936b6f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.008+0000 7fb87e03a640 1 -- 172.21.15.83:0/4076409684 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5629368886b0 con 0x562936b87590 2024-03-06T07:22:16.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.009+0000 7fb87d038640 1 --2- 172.21.15.83:0/4076409684 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562936b6f600 0x562936b6f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:16.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.009+0000 7fb87e03a640 1 --2- 172.21.15.83:0/4076409684 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562936b87590 0x562936b6ff20 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb8700093e0 tx=0x7fb870004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10dde2dbeb3a7a8c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:16.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.009+0000 7fb85affd640 1 -- 172.21.15.83:0/4076409684 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb870044070 con 0x562936b87590 2024-03-06T07:22:16.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.009+0000 7fb85affd640 1 -- 172.21.15.83:0/4076409684 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb87002ca10 con 0x562936b87590 2024-03-06T07:22:16.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.009+0000 7fb85affd640 1 -- 172.21.15.83:0/4076409684 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87002ce00 con 0x562936b87590 2024-03-06T07:22:16.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.010+0000 7fb882766740 1 -- 172.21.15.83:0/4076409684 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562936b87590 msgr2=0x562936b6ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.010+0000 7fb882766740 1 --2- 172.21.15.83:0/4076409684 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562936b87590 0x562936b6ff20 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb8700093e0 tx=0x7fb870004ab0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.010+0000 7fb882766740 1 -- 172.21.15.83:0/4076409684 shutdown_connections 2024-03-06T07:22:16.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.010+0000 7fb882766740 1 --2- 172.21.15.83:0/4076409684 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562936b87590 0x562936b6ff20 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.010+0000 7fb882766740 1 --2- 172.21.15.83:0/4076409684 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562936b6f600 0x562936b6f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.010+0000 7fb882766740 1 --2- 172.21.15.83:0/4076409684 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x562936b64d00 0x562936b699d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.010+0000 7fb882766740 1 -- 172.21.15.83:0/4076409684 >> 172.21.15.83:0/4076409684 conn(0x562936b3f870 msgr2=0x562936b45f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:16.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.010+0000 7fb882766740 1 -- 172.21.15.83:0/4076409684 shutdown_connections 2024-03-06T07:22:16.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.010+0000 7fb882766740 1 -- 172.21.15.83:0/4076409684 wait complete. 2024-03-06T07:22:16.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.011+0000 7fb882766740 1 -- start start 2024-03-06T07:22:16.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.011+0000 7fb882766740 1 Processor -- start 2024-03-06T07:22:16.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.012+0000 7fb882766740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x562936b64d00 0x562936bf4150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.012+0000 7fb882766740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562936b6f600 0x562936b70560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.012+0000 7fb882766740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562936b87590 0x562936b6f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.012+0000 7fb882766740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x562936b43d50 con 0x562936b6f600 2024-03-06T07:22:16.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.012+0000 7fb882766740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x562936b44030 con 0x562936b87590 2024-03-06T07:22:16.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.012+0000 7fb882766740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x562936a65ea0 con 0x562936b64d00 2024-03-06T07:22:16.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.012+0000 7fb882766740 10 client.0 _ll_register_callbacks cb 0x7fb868088640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:22:16.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.012+0000 7fb87d038640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562936b6f600 0x562936b70560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.012+0000 7fb87d038640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562936b6f600 0x562936b70560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50092/0 (socket says 172.21.15.83:50092) 2024-03-06T07:22:16.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.012+0000 7fb87d038640 1 -- 172.21.15.83:0/2981859062 learned_addr learned my addr 172.21.15.83:0/2981859062 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:16.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.012+0000 7fb87e03a640 1 --2- 172.21.15.83:0/2981859062 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562936b87590 0x562936b6f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.012+0000 7fb87d839640 1 --2- 172.21.15.83:0/2981859062 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x562936b64d00 0x562936bf4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.012+0000 7fb87d038640 1 -- 172.21.15.83:0/2981859062 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x562936b64d00 msgr2=0x562936bf4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:16.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.012+0000 7fb87d038640 1 --2- 172.21.15.83:0/2981859062 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x562936b64d00 0x562936bf4150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.012+0000 7fb87d038640 1 -- 172.21.15.83:0/2981859062 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562936b87590 msgr2=0x562936b6f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:16.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.012+0000 7fb87d038640 1 --2- 172.21.15.83:0/2981859062 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562936b87590 0x562936b6f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.012+0000 7fb87d038640 1 -- 172.21.15.83:0/2981859062 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562936b7e6c0 con 0x562936b6f600 2024-03-06T07:22:16.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.012+0000 7fb87e03a640 1 --2- 172.21.15.83:0/2981859062 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562936b87590 0x562936b6f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:16.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.013+0000 7fb87d038640 1 --2- 172.21.15.83:0/2981859062 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562936b6f600 0x562936b70560 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb87814aa90 tx=0x7fb8740046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=308aff77a8e8b422 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:16.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.013+0000 7fb858ff9640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-06T07:22:16.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.013+0000 7fb858ff9640 1 -- 172.21.15.83:0/2981859062 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb874019040 con 0x562936b6f600 2024-03-06T07:22:16.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.013+0000 7fb858ff9640 1 -- 172.21.15.83:0/2981859062 <== mon.0 v2:172.21.15.83:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb87401f420 con 0x562936b6f600 2024-03-06T07:22:16.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.013+0000 7fb882766740 1 -- 172.21.15.83:0/2981859062 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb87002ca10 con 0x562936b6f600 2024-03-06T07:22:16.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.013+0000 7fb858ff9640 1 -- 172.21.15.83:0/2981859062 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb874004020 con 0x562936b6f600 2024-03-06T07:22:16.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.013+0000 7fb882766740 10 client.4689 fetch_fsmap learned FSMap version 6 2024-03-06T07:22:16.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.013+0000 7fb882766740 1 -- 172.21.15.83:0/2981859062 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) v3 -- 0x562936b7e3e0 con 0x562936b6f600 2024-03-06T07:22:16.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.013+0000 7fb858ff9640 1 -- 172.21.15.83:0/2981859062 <== mon.0 v2:172.21.15.83:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb87002ca10 con 0x562936b6f600 2024-03-06T07:22:16.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.013+0000 7fb858ff9640 1 -- 172.21.15.83:0/2981859062 <== mon.0 v2:172.21.15.83:3300/0 5 ==== fsmap(e 6) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fb8740041c0 con 0x562936b6f600 2024-03-06T07:22:16.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.013+0000 7fb882766740 10 client.4689 fetch_fsmap finished waiting for FSMap version 6 2024-03-06T07:22:16.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.013+0000 7fb882766740 10 client.4689 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:22:16.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.013+0000 7fb882766740 20 client.4689 populate_metadata read hostname 'smithi083' 2024-03-06T07:22:16.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.013+0000 7fb882766740 1 --2- 172.21.15.83:0/2981859062 >> [v2:172.21.15.83:6835/1597344199,v1:172.21.15.83:6837/1597344199] conn(0x562936bfab10 0x562936bfd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.013+0000 7fb882766740 4 client.4689 mds_command: new command op to 4287 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:22:16.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.013+0000 7fb882766740 1 -- 172.21.15.83:0/2981859062 --> [v2:172.21.15.83:6835/1597344199,v1:172.21.15.83:6837/1597344199] -- command(tid 0: {"prefix": "status"}) v1 -- 0x562936ab0f40 con 0x562936bfab10 2024-03-06T07:22:16.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.013+0000 7fb87d839640 1 --2- 172.21.15.83:0/2981859062 >> [v2:172.21.15.83:6835/1597344199,v1:172.21.15.83:6837/1597344199] conn(0x562936bfab10 0x562936bfd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.014+0000 7fb858ff9640 10 client.4689 ms_handle_connect on v2:172.21.15.83:6835/1597344199 2024-03-06T07:22:16.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.014+0000 7fb87d839640 1 --2- 172.21.15.83:0/2981859062 >> [v2:172.21.15.83:6835/1597344199,v1:172.21.15.83:6837/1597344199] conn(0x562936bfab10 0x562936bfd680 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39a8b2bbfe3a9cc9 server_cookie=c36e8f51114f395e in_seq=0 out_seq=0 2024-03-06T07:22:16.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.014+0000 7fb858ff9640 1 -- 172.21.15.83:0/2981859062 <== mds.0 v2:172.21.15.83:6835/1597344199 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x562936ab0f40 con 0x562936bfab10 2024-03-06T07:22:16.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.014+0000 7fb858ff9640 10 client.4689 handle_command_reply: tid=0 2024-03-06T07:22:16.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.014+0000 7fb882766740 1 client.4689 shutdown 2024-03-06T07:22:16.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.014+0000 7fb882766740 1 -- 172.21.15.83:0/2981859062 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562936b6f600 msgr2=0x562936b70560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.014+0000 7fb882766740 1 --2- 172.21.15.83:0/2981859062 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562936b6f600 0x562936b70560 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb87814aa90 tx=0x7fb8740046b0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.014+0000 7fb882766740 1 -- 172.21.15.83:0/2981859062 shutdown_connections 2024-03-06T07:22:16.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.014+0000 7fb882766740 1 --2- 172.21.15.83:0/2981859062 >> [v2:172.21.15.83:6835/1597344199,v1:172.21.15.83:6837/1597344199] conn(0x562936bfab10 0x562936bfd680 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.014+0000 7fb882766740 1 --2- 172.21.15.83:0/2981859062 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562936b87590 0x562936b6f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.014+0000 7fb858ff9640 0 client.4689 ms_handle_reset on v2:172.21.15.83:6835/1597344199 2024-03-06T07:22:16.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.014+0000 7fb882766740 1 --2- 172.21.15.83:0/2981859062 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562936b6f600 0x562936b70560 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.014+0000 7fb882766740 1 --2- 172.21.15.83:0/2981859062 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x562936b64d00 0x562936bf4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.014+0000 7fb882766740 1 -- 172.21.15.83:0/2981859062 >> 172.21.15.83:0/2981859062 conn(0x562936b3f870 msgr2=0x562936b6d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:16.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.014+0000 7fb882766740 1 -- 172.21.15.83:0/2981859062 shutdown_connections 2024-03-06T07:22:16.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.014+0000 7fb882766740 1 -- 172.21.15.83:0/2981859062 wait complete. 2024-03-06T07:22:16.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.014+0000 7fb882766740 20 client.4689 trim_cache size 0 max 16384 2024-03-06T07:22:16.024 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:22:16.025 INFO:teuthology.orchestra.run.smithi083.stdout: "cluster_fsid": "039877dd-d2ce-45d2-8e77-ac5eb7e1d55a", 2024-03-06T07:22:16.025 INFO:teuthology.orchestra.run.smithi083.stdout: "whoami": 0, 2024-03-06T07:22:16.025 INFO:teuthology.orchestra.run.smithi083.stdout: "id": 4287, 2024-03-06T07:22:16.025 INFO:teuthology.orchestra.run.smithi083.stdout: "want_state": "up:active", 2024-03-06T07:22:16.025 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "up:active", 2024-03-06T07:22:16.025 INFO:teuthology.orchestra.run.smithi083.stdout: "fs_name": "cephfs", 2024-03-06T07:22:16.025 INFO:teuthology.orchestra.run.smithi083.stdout: "rank_uptime": 17.063294785, 2024-03-06T07:22:16.025 INFO:teuthology.orchestra.run.smithi083.stdout: "mdsmap_epoch": 6, 2024-03-06T07:22:16.025 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_epoch": 20, 2024-03-06T07:22:16.025 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_epoch_barrier": 20, 2024-03-06T07:22:16.026 INFO:teuthology.orchestra.run.smithi083.stdout: "uptime": 24.114587966999999 2024-03-06T07:22:16.026 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:22:16.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.015+0000 7fb87fa9d640 1 -- 172.21.15.83:0/1900453791 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb85c06b4d0 msgr2=0x7fb85c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.015+0000 7fb87fa9d640 1 --2- 172.21.15.83:0/1900453791 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb85c06b4d0 0x7fb85c06d990 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb878070720 tx=0x7fb874008040 comp rx=0 tx=0).stop 2024-03-06T07:22:16.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.015+0000 7fb87fa9d640 1 -- 172.21.15.83:0/1900453791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb8780b0270 msgr2=0x7fb878149a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.015+0000 7fb87fa9d640 1 --2- 172.21.15.83:0/1900453791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb8780b0270 0x7fb878149a50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb86800b410 tx=0x7fb86800b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.015+0000 7fb87fa9d640 1 -- 172.21.15.83:0/1900453791 shutdown_connections 2024-03-06T07:22:16.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.015+0000 7fb87fa9d640 1 --2- 172.21.15.83:0/1900453791 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb85c06b4d0 0x7fb85c06d990 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.015+0000 7fb87fa9d640 1 --2- 172.21.15.83:0/1900453791 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb8781614a0 0x7fb87815bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.015+0000 7fb87fa9d640 1 --2- 172.21.15.83:0/1900453791 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb8780b1a20 0x7fb8781f3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.015+0000 7fb87fa9d640 1 --2- 172.21.15.83:0/1900453791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb8780b0270 0x7fb878149a50 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.015+0000 7fb87fa9d640 1 -- 172.21.15.83:0/1900453791 >> 172.21.15.83:0/1900453791 conn(0x7fb87809a9f0 msgr2=0x7fb87806ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:16.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.015+0000 7fb87fa9d640 1 -- 172.21.15.83:0/1900453791 shutdown_connections 2024-03-06T07:22:16.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.015+0000 7fb87fa9d640 1 -- 172.21.15.83:0/1900453791 wait complete. 2024-03-06T07:22:16.028 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-03-06T07:22:16.053 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-03-06T07:22:16.053 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-03-06T07:22:16.053 INFO:tasks.ceph_fuse:client.0 config is {} 2024-03-06T07:22:16.053 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:22:16.053 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:16.053 INFO:tasks.ceph_fuse:client.2 config is {} 2024-03-06T07:22:16.054 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:22:16.054 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-06T07:22:16.054 INFO:tasks.ceph_fuse:client.1 config is {} 2024-03-06T07:22:16.054 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:22:16.054 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:16.054 INFO:tasks.ceph_fuse:client.3 config is {} 2024-03-06T07:22:16.054 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:22:16.054 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-06T07:22:16.054 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:16.054 DEBUG:teuthology.orchestra.run.smithi183:> ip netns list 2024-03-06T07:22:16.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:16.074 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip link delete ceph-brx 2024-03-06T07:22:16.146 INFO:teuthology.orchestra.run.smithi183.stderr:Cannot find device "ceph-brx" 2024-03-06T07:22:16.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:22:16.148 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:16.148 DEBUG:teuthology.orchestra.run.smithi083:> ip netns list 2024-03-06T07:22:16.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:16.164 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip link delete ceph-brx 2024-03-06T07:22:16.232 INFO:teuthology.orchestra.run.smithi083.stderr:Cannot find device "ceph-brx" 2024-03-06T07:22:16.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:22:16.233 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-03-06T07:22:16.234 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:22:16.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.372+0000 7fee9b36b640 1 Processor -- start 2024-03-06T07:22:16.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.372+0000 7fee9b36b640 1 -- start start 2024-03-06T07:22:16.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.373+0000 7fee9b36b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fee9415bdd0 0x7fee940c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.373+0000 7fee9b36b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fee940d16e0 0x7fee940d1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.373+0000 7fee9b36b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee940cebe0 con 0x7fee9415fa10 2024-03-06T07:22:16.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.373+0000 7fee9b36b640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee940ce980 con 0x7fee9415bdd0 2024-03-06T07:22:16.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.373+0000 7fee9b36b640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee940ceec0 con 0x7fee940d16e0 2024-03-06T07:22:16.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.373+0000 7fee99107640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fee9415bdd0 0x7fee940c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.373+0000 7fee99107640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fee9415bdd0 0x7fee940c5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50100/0 (socket says 172.21.15.83:50100) 2024-03-06T07:22:16.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.373+0000 7fee99107640 1 -- 172.21.15.83:0/3642767697 learned_addr learned my addr 172.21.15.83:0/3642767697 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:16.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.373+0000 7fee99908640 1 --2- 172.21.15.83:0/3642767697 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fee940d16e0 0x7fee940d1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.373+0000 7fee99107640 1 -- 172.21.15.83:0/3642767697 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fee940d16e0 msgr2=0x7fee940d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:16.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.373+0000 7fee99107640 1 --2- 172.21.15.83:0/3642767697 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fee940d16e0 0x7fee940d1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.373+0000 7fee99107640 1 -- 172.21.15.83:0/3642767697 >> v1:172.21.15.183:6789/0 conn(0x7fee9415fa10 legacy=0x7fee940ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.373+0000 7fee99107640 1 -- 172.21.15.83:0/3642767697 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee940cf1a0 con 0x7fee9415bdd0 2024-03-06T07:22:16.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.374+0000 7fee99107640 1 --2- 172.21.15.83:0/3642767697 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee9415bdd0 0x7fee940c5610 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fee8c009e50 tx=0x7fee8c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=766e68bcbba5d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:16.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.374+0000 7fee8bfff640 1 -- 172.21.15.83:0/3642767697 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee8c02ca70 con 0x7fee9415bdd0 2024-03-06T07:22:16.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.374+0000 7fee8bfff640 1 -- 172.21.15.83:0/3642767697 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee8c02cc10 con 0x7fee9415bdd0 2024-03-06T07:22:16.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.374+0000 7fee8bfff640 1 -- 172.21.15.83:0/3642767697 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee8c0343f0 con 0x7fee9415bdd0 2024-03-06T07:22:16.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.375+0000 7fee9b36b640 1 -- 172.21.15.83:0/3642767697 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee9415bdd0 msgr2=0x7fee940c5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.375+0000 7fee9b36b640 1 --2- 172.21.15.83:0/3642767697 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee9415bdd0 0x7fee940c5610 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fee8c009e50 tx=0x7fee8c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.375+0000 7fee9b36b640 1 -- 172.21.15.83:0/3642767697 shutdown_connections 2024-03-06T07:22:16.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.375+0000 7fee9b36b640 1 --2- 172.21.15.83:0/3642767697 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fee940d16e0 0x7fee940d1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.375+0000 7fee9b36b640 1 --2- 172.21.15.83:0/3642767697 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee9415bdd0 0x7fee940c5610 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.375+0000 7fee9b36b640 1 -- 172.21.15.83:0/3642767697 >> 172.21.15.83:0/3642767697 conn(0x7fee9409a9f0 msgr2=0x7fee9409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:16.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.375+0000 7fee9b36b640 1 -- 172.21.15.83:0/3642767697 shutdown_connections 2024-03-06T07:22:16.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.375+0000 7fee9b36b640 1 -- 172.21.15.83:0/3642767697 wait complete. 2024-03-06T07:22:16.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.376+0000 7fee9b36b640 1 Processor -- start 2024-03-06T07:22:16.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.376+0000 7fee9b36b640 1 -- start start 2024-03-06T07:22:16.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.377+0000 7fee9b36b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fee940d16e0 0x7fee941e6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.377+0000 7fee9b36b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee9415bdd0 0x7fee9415a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.377+0000 7fee9b36b640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fee9415fa10 0x7fee941e5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.377+0000 7fee9b36b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee940d2050 con 0x7fee9415bdd0 2024-03-06T07:22:16.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.377+0000 7fee9b36b640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee940c5b50 con 0x7fee9415fa10 2024-03-06T07:22:16.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.377+0000 7fee9b36b640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee940cc5d0 con 0x7fee940d16e0 2024-03-06T07:22:16.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.377+0000 7fee98906640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee9415bdd0 0x7fee9415a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.377+0000 7fee98906640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee9415bdd0 0x7fee9415a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50116/0 (socket says 172.21.15.83:50116) 2024-03-06T07:22:16.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.377+0000 7fee98906640 1 -- 172.21.15.83:0/84319540 learned_addr learned my addr 172.21.15.83:0/84319540 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:16.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.377+0000 7fee99908640 1 --2- 172.21.15.83:0/84319540 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fee9415fa10 0x7fee941e5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.377+0000 7fee99107640 1 --2- 172.21.15.83:0/84319540 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fee940d16e0 0x7fee941e6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.377+0000 7fee98906640 1 -- 172.21.15.83:0/84319540 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fee940d16e0 msgr2=0x7fee941e6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.377+0000 7fee98906640 1 --2- 172.21.15.83:0/84319540 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fee940d16e0 0x7fee941e6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.377+0000 7fee98906640 1 -- 172.21.15.83:0/84319540 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fee9415fa10 msgr2=0x7fee941e5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.377+0000 7fee98906640 1 --2- 172.21.15.83:0/84319540 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fee9415fa10 0x7fee941e5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.377+0000 7fee98906640 1 -- 172.21.15.83:0/84319540 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee941e5c70 con 0x7fee9415bdd0 2024-03-06T07:22:16.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.377+0000 7fee99908640 1 --2- 172.21.15.83:0/84319540 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fee9415fa10 0x7fee941e5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:16.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.377+0000 7fee98906640 1 --2- 172.21.15.83:0/84319540 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee9415bdd0 0x7fee9415a8e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fee8400c440 tx=0x7fee8400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:16.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.377+0000 7fee89ffb640 1 -- 172.21.15.83:0/84319540 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee840052d0 con 0x7fee9415bdd0 2024-03-06T07:22:16.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.377+0000 7fee89ffb640 1 -- 172.21.15.83:0/84319540 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee84005470 con 0x7fee9415bdd0 2024-03-06T07:22:16.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.377+0000 7fee89ffb640 1 -- 172.21.15.83:0/84319540 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee84007260 con 0x7fee9415bdd0 2024-03-06T07:22:16.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.377+0000 7fee9b36b640 1 -- 172.21.15.83:0/84319540 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee940ca0c0 con 0x7fee9415bdd0 2024-03-06T07:22:16.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.378+0000 7fee9b36b640 1 -- 172.21.15.83:0/84319540 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee940ca680 con 0x7fee9415bdd0 2024-03-06T07:22:16.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.379+0000 7fee89ffb640 1 -- 172.21.15.83:0/84319540 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fee840074c0 con 0x7fee9415bdd0 2024-03-06T07:22:16.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.379+0000 7fee9b36b640 1 -- 172.21.15.83:0/84319540 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee9404f330 con 0x7fee9415bdd0 2024-03-06T07:22:16.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.379+0000 7fee89ffb640 1 --2- 172.21.15.83:0/84319540 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fee7806b4d0 0x7fee7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.379+0000 7fee89ffb640 1 -- 172.21.15.83:0/84319540 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fee84089650 con 0x7fee9415bdd0 2024-03-06T07:22:16.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.379+0000 7fee99107640 1 --2- 172.21.15.83:0/84319540 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fee7806b4d0 0x7fee7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.380+0000 7fee99107640 1 --2- 172.21.15.83:0/84319540 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fee7806b4d0 0x7fee7806d990 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fee8c035000 tx=0x7fee8c030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:16.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.384+0000 7fee89ffb640 1 -- 172.21.15.83:0/84319540 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fee840584d0 con 0x7fee9415bdd0 2024-03-06T07:22:16.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.593+0000 7fee9b36b640 1 -- 172.21.15.83:0/84319540 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fee940b7a40 con 0x7fee9415bdd0 2024-03-06T07:22:16.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.594+0000 7fee89ffb640 1 -- 172.21.15.83:0/84319540 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) v1 ==== 53+0+73 (secure 0 0 0) 0x7fee84057dd0 con 0x7fee9415bdd0 2024-03-06T07:22:16.596 INFO:teuthology.orchestra.run.smithi083.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:22:16.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.596+0000 7fee9b36b640 1 -- 172.21.15.83:0/84319540 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fee7806b4d0 msgr2=0x7fee7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.596+0000 7fee9b36b640 1 --2- 172.21.15.83:0/84319540 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fee7806b4d0 0x7fee7806d990 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fee8c035000 tx=0x7fee8c030000 comp rx=0 tx=0).stop 2024-03-06T07:22:16.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.596+0000 7fee9b36b640 1 -- 172.21.15.83:0/84319540 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee9415bdd0 msgr2=0x7fee9415a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.596+0000 7fee9b36b640 1 --2- 172.21.15.83:0/84319540 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee9415bdd0 0x7fee9415a8e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fee8400c440 tx=0x7fee8400c900 comp rx=0 tx=0).stop 2024-03-06T07:22:16.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.596+0000 7fee9b36b640 1 -- 172.21.15.83:0/84319540 shutdown_connections 2024-03-06T07:22:16.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.596+0000 7fee9b36b640 1 --2- 172.21.15.83:0/84319540 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fee7806b4d0 0x7fee7806d990 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.596+0000 7fee9b36b640 1 --2- 172.21.15.83:0/84319540 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fee9415fa10 0x7fee941e5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.596+0000 7fee9b36b640 1 --2- 172.21.15.83:0/84319540 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee9415bdd0 0x7fee9415a8e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.596+0000 7fee9b36b640 1 --2- 172.21.15.83:0/84319540 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fee940d16e0 0x7fee941e6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.596+0000 7fee9b36b640 1 -- 172.21.15.83:0/84319540 >> 172.21.15.83:0/84319540 conn(0x7fee9409a9f0 msgr2=0x7fee94070300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:16.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.596+0000 7fee9b36b640 1 -- 172.21.15.83:0/84319540 shutdown_connections 2024-03-06T07:22:16.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:16.596+0000 7fee9b36b640 1 -- 172.21.15.83:0/84319540 wait complete. 2024-03-06T07:22:16.612 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:22:16.612 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:22:16.612 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi083.front.sepia.ceph.com 2024-03-06T07:22:16.612 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:22:16.612 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:16.612 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:22:16.612 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:22:16.612 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:22:16.613 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:22:16.613 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:16.613 DEBUG:teuthology.orchestra.run.smithi083:> ip addr 2024-03-06T07:22:16.629 INFO:teuthology.orchestra.run.smithi083.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:22:16.629 INFO:teuthology.orchestra.run.smithi083.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:22:16.629 INFO:teuthology.orchestra.run.smithi083.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:22:16.629 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:16.629 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 ::1/128 scope host 2024-03-06T07:22:16.629 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:16.629 INFO:teuthology.orchestra.run.smithi083.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:16.629 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:d8:da:1e brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:16.629 INFO:teuthology.orchestra.run.smithi083.stdout: altname enp7s0f0 2024-03-06T07:22:16.629 INFO:teuthology.orchestra.run.smithi083.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:16.629 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:d8:da:1f brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:16.630 INFO:teuthology.orchestra.run.smithi083.stdout: altname enp7s0f1 2024-03-06T07:22:16.630 INFO:teuthology.orchestra.run.smithi083.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:16.630 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:8f:ce:90 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:16.630 INFO:teuthology.orchestra.run.smithi083.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:22:16.630 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:8f:ce:91 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:16.630 INFO:teuthology.orchestra.run.smithi083.stdout: inet 172.21.15.83/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:22:16.630 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft 42719sec preferred_lft 42719sec 2024-03-06T07:22:16.630 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::ec4:7aff:fe8f:ce91/64 scope link 2024-03-06T07:22:16.630 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:16.630 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-06T07:22:16.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:16.631 DEBUG:teuthology.orchestra.run.smithi083:> (cd / && exec bash -c ' 2024-03-06T07:22:16.631 DEBUG:teuthology.orchestra.run.smithi083:> set -e 2024-03-06T07:22:16.631 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip link add name ceph-brx type bridge 2024-03-06T07:22:16.631 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip addr flush dev ceph-brx 2024-03-06T07:22:16.631 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip link set ceph-brx up 2024-03-06T07:22:16.631 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-06T07:22:16.631 DEBUG:teuthology.orchestra.run.smithi083:> ') 2024-03-06T07:22:16.798 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:16.798 DEBUG:teuthology.orchestra.run.smithi083:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:22:16.825 INFO:teuthology.orchestra.run.smithi083.stdout:1 2024-03-06T07:22:16.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:16.828 DEBUG:teuthology.orchestra.run.smithi083:> route 2024-03-06T07:22:16.897 INFO:teuthology.orchestra.run.smithi083.stdout:Kernel IP routing table 2024-03-06T07:22:16.897 INFO:teuthology.orchestra.run.smithi083.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:22:16.897 INFO:teuthology.orchestra.run.smithi083.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:22:16.897 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:22:16.897 INFO:teuthology.orchestra.run.smithi083.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:22:16.897 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:16.898 DEBUG:teuthology.orchestra.run.smithi083:> (cd / && exec bash -c ' 2024-03-06T07:22:16.898 DEBUG:teuthology.orchestra.run.smithi083:> set -e 2024-03-06T07:22:16.898 DEBUG:teuthology.orchestra.run.smithi083:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:22:16.898 DEBUG:teuthology.orchestra.run.smithi083:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:22:16.898 DEBUG:teuthology.orchestra.run.smithi083:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:22:16.898 DEBUG:teuthology.orchestra.run.smithi083:> ') 2024-03-06T07:22:17.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:17.150 DEBUG:teuthology.orchestra.run.smithi083:> ip netns list 2024-03-06T07:22:17.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:17.166 DEBUG:teuthology.orchestra.run.smithi083:> ip netns list-id 2024-03-06T07:22:17.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:17.220 DEBUG:teuthology.orchestra.run.smithi083:> (cd / && exec bash -c ' 2024-03-06T07:22:17.220 DEBUG:teuthology.orchestra.run.smithi083:> set -e 2024-03-06T07:22:17.220 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:17.221 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-06T07:22:17.221 DEBUG:teuthology.orchestra.run.smithi083:> ') 2024-03-06T07:22:17.304 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-06T07:22:17.304 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:17.304 DEBUG:teuthology.orchestra.run.smithi083:> (cd / && exec bash -c ' 2024-03-06T07:22:17.305 DEBUG:teuthology.orchestra.run.smithi083:> set -e 2024-03-06T07:22:17.305 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-06T07:22:17.305 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-06T07:22:17.305 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-06T07:22:17.305 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-06T07:22:17.305 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-06T07:22:17.305 DEBUG:teuthology.orchestra.run.smithi083:> ') 2024-03-06T07:22:17.446 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:17.446 DEBUG:teuthology.orchestra.run.smithi083:> (cd / && exec bash -c ' 2024-03-06T07:22:17.446 DEBUG:teuthology.orchestra.run.smithi083:> set -e 2024-03-06T07:22:17.446 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip link set brx.0 up 2024-03-06T07:22:17.446 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip link set dev brx.0 master ceph-brx 2024-03-06T07:22:17.446 DEBUG:teuthology.orchestra.run.smithi083:> ') 2024-03-06T07:22:17.490 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:22:17.490 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:17.490 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:17.548 INFO:teuthology.orchestra.run.smithi083.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:22:17.549 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:17.549 DEBUG:teuthology.orchestra.run.smithi083:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:17.603 DEBUG:teuthology.orchestra.run.smithi083:> sudo modprobe fuse 2024-03-06T07:22:17.673 DEBUG:teuthology.orchestra.run.smithi083:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:17.728 INFO:teuthology.orchestra.run.smithi083.stdout:/proc 2024-03-06T07:22:17.729 INFO:teuthology.orchestra.run.smithi083.stdout:/sys 2024-03-06T07:22:17.729 INFO:teuthology.orchestra.run.smithi083.stdout:/dev 2024-03-06T07:22:17.729 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/security 2024-03-06T07:22:17.729 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/shm 2024-03-06T07:22:17.729 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/pts 2024-03-06T07:22:17.729 INFO:teuthology.orchestra.run.smithi083.stdout:/run 2024-03-06T07:22:17.729 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/cgroup 2024-03-06T07:22:17.729 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/pstore 2024-03-06T07:22:17.729 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/bpf 2024-03-06T07:22:17.729 INFO:teuthology.orchestra.run.smithi083.stdout:/ 2024-03-06T07:22:17.730 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/selinux 2024-03-06T07:22:17.730 INFO:teuthology.orchestra.run.smithi083.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:17.730 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/mqueue 2024-03-06T07:22:17.730 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/hugepages 2024-03-06T07:22:17.730 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/debug 2024-03-06T07:22:17.730 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/tracing 2024-03-06T07:22:17.730 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:17.730 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:17.730 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/config 2024-03-06T07:22:17.730 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:17.730 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:17.731 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:17.731 INFO:teuthology.orchestra.run.smithi083.stdout:/run/user/1000 2024-03-06T07:22:17.731 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph 2024-03-06T07:22:17.731 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:22:17.731 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:22:17.731 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:22:17.731 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:22:17.731 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns 2024-03-06T07:22:17.731 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:17.731 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:17.731 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:17.732 DEBUG:teuthology.orchestra.run.smithi083:> ls /sys/fs/fuse/connections 2024-03-06T07:22:17.783 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:22:17.783 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:22:17.826 DEBUG:teuthology.orchestra.run.smithi083:> sudo modprobe fuse 2024-03-06T07:22:17.853 DEBUG:teuthology.orchestra.run.smithi083:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:17.909 INFO:teuthology.orchestra.run.smithi083.stdout:/proc 2024-03-06T07:22:17.909 INFO:teuthology.orchestra.run.smithi083.stdout:/sys 2024-03-06T07:22:17.909 INFO:teuthology.orchestra.run.smithi083.stdout:/dev 2024-03-06T07:22:17.909 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/security 2024-03-06T07:22:17.909 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/shm 2024-03-06T07:22:17.909 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/pts 2024-03-06T07:22:17.909 INFO:teuthology.orchestra.run.smithi083.stdout:/run 2024-03-06T07:22:17.909 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/cgroup 2024-03-06T07:22:17.909 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/pstore 2024-03-06T07:22:17.910 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/bpf 2024-03-06T07:22:17.910 INFO:teuthology.orchestra.run.smithi083.stdout:/ 2024-03-06T07:22:17.910 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/selinux 2024-03-06T07:22:17.910 INFO:teuthology.orchestra.run.smithi083.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:17.910 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/mqueue 2024-03-06T07:22:17.910 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/hugepages 2024-03-06T07:22:17.910 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/debug 2024-03-06T07:22:17.910 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/tracing 2024-03-06T07:22:17.910 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:17.910 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:17.910 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/config 2024-03-06T07:22:17.910 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:17.910 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:17.910 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:17.911 INFO:teuthology.orchestra.run.smithi083.stdout:/run/user/1000 2024-03-06T07:22:17.911 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph 2024-03-06T07:22:17.911 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:22:17.911 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:22:17.911 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:22:17.911 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:22:17.911 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns 2024-03-06T07:22:17.911 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:17.911 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:17.911 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:17.911 DEBUG:teuthology.orchestra.run.smithi083:> ls /sys/fs/fuse/connections 2024-03-06T07:22:17.919 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:2024-03-06T07:22:17.917+0000 7f2085a48480 -1 init, newargv = 0x55d18a7b9730 newargc=15 2024-03-06T07:22:17.919 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:ceph-fuse[47084]: starting ceph client 2024-03-06T07:22:17.924 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:ceph-fuse[47084]: starting fuse 2024-03-06T07:22:17.934 INFO:teuthology.orchestra.run.smithi083.stdout:41 2024-03-06T07:22:17.935 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-06T07:22:17.935 DEBUG:teuthology.orchestra.run.smithi083:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:22:17.935 DEBUG:teuthology.orchestra.run.smithi083:> import glob 2024-03-06T07:22:17.935 DEBUG:teuthology.orchestra.run.smithi083:> import re 2024-03-06T07:22:17.935 DEBUG:teuthology.orchestra.run.smithi083:> import os 2024-03-06T07:22:17.935 DEBUG:teuthology.orchestra.run.smithi083:> import subprocess 2024-03-06T07:22:17.935 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:22:17.935 DEBUG:teuthology.orchestra.run.smithi083:> def _find_admin_socket(client_name): 2024-03-06T07:22:17.935 DEBUG:teuthology.orchestra.run.smithi083:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:22:17.935 DEBUG:teuthology.orchestra.run.smithi083:> files = glob.glob(asok_path) 2024-03-06T07:22:17.936 DEBUG:teuthology.orchestra.run.smithi083:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:22:17.936 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:22:17.936 DEBUG:teuthology.orchestra.run.smithi083:> # Given a non-glob path, it better be there 2024-03-06T07:22:17.936 DEBUG:teuthology.orchestra.run.smithi083:> if "*" not in asok_path: 2024-03-06T07:22:17.936 DEBUG:teuthology.orchestra.run.smithi083:> assert(len(files) == 1) 2024-03-06T07:22:17.936 DEBUG:teuthology.orchestra.run.smithi083:> return files[0] 2024-03-06T07:22:17.936 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:22:17.936 DEBUG:teuthology.orchestra.run.smithi083:> for f in files: 2024-03-06T07:22:17.936 DEBUG:teuthology.orchestra.run.smithi083:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:22:17.936 DEBUG:teuthology.orchestra.run.smithi083:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:22:17.936 DEBUG:teuthology.orchestra.run.smithi083:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:22:17.936 DEBUG:teuthology.orchestra.run.smithi083:> contents = proc_f.read() 2024-03-06T07:22:17.936 DEBUG:teuthology.orchestra.run.smithi083:> if mountpoint in contents: 2024-03-06T07:22:17.936 DEBUG:teuthology.orchestra.run.smithi083:> return f 2024-03-06T07:22:17.936 DEBUG:teuthology.orchestra.run.smithi083:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:22:17.937 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:22:17.937 DEBUG:teuthology.orchestra.run.smithi083:> print(_find_admin_socket("client.0")) 2024-03-06T07:22:17.937 DEBUG:teuthology.orchestra.run.smithi083:> ' 2024-03-06T07:22:18.059 INFO:teuthology.orchestra.run.smithi083.stdout:/var/run/ceph/ceph-client.0.47084.asok 2024-03-06T07:22:18.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:22:18.069 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.47084.asok 2024-03-06T07:22:18.069 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:18.069 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.47084.asok status 2024-03-06T07:22:18.208 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:22:18.208 INFO:teuthology.orchestra.run.smithi083.stdout: "metadata": { 2024-03-06T07:22:18.208 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:22:18.208 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:22:18.208 INFO:teuthology.orchestra.run.smithi083.stdout: "entity_id": "0", 2024-03-06T07:22:18.208 INFO:teuthology.orchestra.run.smithi083.stdout: "hostname": "smithi083", 2024-03-06T07:22:18.208 INFO:teuthology.orchestra.run.smithi083.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:22:18.209 INFO:teuthology.orchestra.run.smithi083.stdout: "pid": "47084", 2024-03-06T07:22:18.209 INFO:teuthology.orchestra.run.smithi083.stdout: "root": "/" 2024-03-06T07:22:18.209 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:22:18.209 INFO:teuthology.orchestra.run.smithi083.stdout: "dentry_count": 0, 2024-03-06T07:22:18.209 INFO:teuthology.orchestra.run.smithi083.stdout: "dentry_pinned_count": 0, 2024-03-06T07:22:18.209 INFO:teuthology.orchestra.run.smithi083.stdout: "id": 4704, 2024-03-06T07:22:18.209 INFO:teuthology.orchestra.run.smithi083.stdout: "inst": { 2024-03-06T07:22:18.209 INFO:teuthology.orchestra.run.smithi083.stdout: "name": { 2024-03-06T07:22:18.209 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "client", 2024-03-06T07:22:18.209 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 4704 2024-03-06T07:22:18.209 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:22:18.209 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": { 2024-03-06T07:22:18.209 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-03-06T07:22:18.209 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:22:18.210 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 2245015985 2024-03-06T07:22:18.210 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:22:18.210 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:22:18.210 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": { 2024-03-06T07:22:18.210 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-03-06T07:22:18.210 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:22:18.210 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 2245015985 2024-03-06T07:22:18.210 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:22:18.210 INFO:teuthology.orchestra.run.smithi083.stdout: "inst_str": "client.4704 192.168.0.1:0/2245015985", 2024-03-06T07:22:18.210 INFO:teuthology.orchestra.run.smithi083.stdout: "addr_str": "192.168.0.1:0/2245015985", 2024-03-06T07:22:18.210 INFO:teuthology.orchestra.run.smithi083.stdout: "inode_count": 1, 2024-03-06T07:22:18.210 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_epoch": 6, 2024-03-06T07:22:18.210 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_epoch": 20, 2024-03-06T07:22:18.210 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:22:18.211 INFO:teuthology.orchestra.run.smithi083.stdout: "blocklisted": false, 2024-03-06T07:22:18.211 INFO:teuthology.orchestra.run.smithi083.stdout: "fs_name": "cephfs" 2024-03-06T07:22:18.211 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:22:18.216 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:22:18.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.317+0000 7feb578df640 1 Processor -- start 2024-03-06T07:22:18.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.318+0000 7feb578df640 1 -- start start 2024-03-06T07:22:18.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.318+0000 7feb578df640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feb500d4760 0x7feb500b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.318+0000 7feb578df640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7feb500d0a80 0x7feb500cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.318+0000 7feb578df640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb500d25d0 con 0x7feb500b4170 2024-03-06T07:22:18.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.318+0000 7feb578df640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb500d4bf0 con 0x7feb500d0a80 2024-03-06T07:22:18.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.318+0000 7feb578df640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb500ce9e0 con 0x7feb500d4760 2024-03-06T07:22:18.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.319+0000 7feb55e7c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7feb500d0a80 0x7feb500cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.319+0000 7feb55e7c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7feb500d0a80 0x7feb500cee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48692/0 (socket says 172.21.15.83:48692) 2024-03-06T07:22:18.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.319+0000 7feb55e7c640 1 -- 172.21.15.83:0/452908791 learned_addr learned my addr 172.21.15.83:0/452908791 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:18.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.319+0000 7feb55e7c640 1 -- 172.21.15.83:0/452908791 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feb500d4760 msgr2=0x7feb500b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:18.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.319+0000 7feb55e7c640 1 --2- 172.21.15.83:0/452908791 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feb500d4760 0x7feb500b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.319+0000 7feb55e7c640 1 -- 172.21.15.83:0/452908791 >> v1:172.21.15.183:6789/0 conn(0x7feb500b4170 legacy=0x7feb500d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:18.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.319+0000 7feb55e7c640 1 -- 172.21.15.83:0/452908791 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb500d2870 con 0x7feb500d0a80 2024-03-06T07:22:18.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.320+0000 7feb55e7c640 1 --2- 172.21.15.83:0/452908791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb500d0a80 0x7feb500cee70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7feb48009e50 tx=0x7feb4802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b010ef168f4061cb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:18.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.320+0000 7feb47fff640 1 -- 172.21.15.83:0/452908791 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb4802ca70 con 0x7feb500d0a80 2024-03-06T07:22:18.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.320+0000 7feb47fff640 1 -- 172.21.15.83:0/452908791 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb4802cc10 con 0x7feb500d0a80 2024-03-06T07:22:18.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.320+0000 7feb47fff640 1 -- 172.21.15.83:0/452908791 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb480343f0 con 0x7feb500d0a80 2024-03-06T07:22:18.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.320+0000 7feb578df640 1 -- 172.21.15.83:0/452908791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb500d0a80 msgr2=0x7feb500cee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.320+0000 7feb578df640 1 --2- 172.21.15.83:0/452908791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb500d0a80 0x7feb500cee70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7feb48009e50 tx=0x7feb4802bed0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.320+0000 7feb578df640 1 -- 172.21.15.83:0/452908791 shutdown_connections 2024-03-06T07:22:18.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.320+0000 7feb578df640 1 --2- 172.21.15.83:0/452908791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb500d0a80 0x7feb500cee70 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.320+0000 7feb578df640 1 --2- 172.21.15.83:0/452908791 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feb500d4760 0x7feb500b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.320+0000 7feb578df640 1 -- 172.21.15.83:0/452908791 >> 172.21.15.83:0/452908791 conn(0x7feb500d3c40 msgr2=0x7feb500d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:18.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.320+0000 7feb578df640 1 -- 172.21.15.83:0/452908791 shutdown_connections 2024-03-06T07:22:18.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.320+0000 7feb578df640 1 -- 172.21.15.83:0/452908791 wait complete. 2024-03-06T07:22:18.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.322+0000 7feb578df640 1 Processor -- start 2024-03-06T07:22:18.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.322+0000 7feb578df640 1 -- start start 2024-03-06T07:22:18.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.322+0000 7feb578df640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb500b4170 0x7feb501f7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.322+0000 7feb578df640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feb500d0a80 0x7feb500c80b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.322+0000 7feb578df640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feb500c9d10 0x7feb500b7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.322+0000 7feb578df640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb500b4dd0 con 0x7feb500b4170 2024-03-06T07:22:18.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.322+0000 7feb578df640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb500b4550 con 0x7feb500c9d10 2024-03-06T07:22:18.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.322+0000 7feb578df640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb500b4f50 con 0x7feb500d0a80 2024-03-06T07:22:18.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.322+0000 7feb5567b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb500b4170 0x7feb501f7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.322+0000 7feb5567b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb500b4170 0x7feb501f7330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48696/0 (socket says 172.21.15.83:48696) 2024-03-06T07:22:18.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.322+0000 7feb5567b640 1 -- 172.21.15.83:0/2990295276 learned_addr learned my addr 172.21.15.83:0/2990295276 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:18.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.323+0000 7feb54e7a640 1 --2- 172.21.15.83:0/2990295276 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feb500d0a80 0x7feb500c80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.323+0000 7feb55e7c640 1 --2- 172.21.15.83:0/2990295276 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feb500c9d10 0x7feb500b7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.323+0000 7feb5567b640 1 -- 172.21.15.83:0/2990295276 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feb500d0a80 msgr2=0x7feb500c80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.323+0000 7feb5567b640 1 --2- 172.21.15.83:0/2990295276 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feb500d0a80 0x7feb500c80b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.323+0000 7feb5567b640 1 -- 172.21.15.83:0/2990295276 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feb500c9d10 msgr2=0x7feb500b7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.323+0000 7feb5567b640 1 --2- 172.21.15.83:0/2990295276 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feb500c9d10 0x7feb500b7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.323+0000 7feb5567b640 1 -- 172.21.15.83:0/2990295276 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb500b7580 con 0x7feb500b4170 2024-03-06T07:22:18.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.323+0000 7feb54e7a640 1 --2- 172.21.15.83:0/2990295276 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feb500d0a80 0x7feb500c80b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:18.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.323+0000 7feb5567b640 1 --2- 172.21.15.83:0/2990295276 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb500b4170 0x7feb501f7330 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7feb4c00c430 tx=0x7feb4c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:18.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.323+0000 7feb45ffb640 1 -- 172.21.15.83:0/2990295276 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb4c011790 con 0x7feb500b4170 2024-03-06T07:22:18.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.323+0000 7feb45ffb640 1 -- 172.21.15.83:0/2990295276 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb4c017070 con 0x7feb500b4170 2024-03-06T07:22:18.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.323+0000 7feb578df640 1 -- 172.21.15.83:0/2990295276 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb50154b50 con 0x7feb500b4170 2024-03-06T07:22:18.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.323+0000 7feb45ffb640 1 -- 172.21.15.83:0/2990295276 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb4c011bb0 con 0x7feb500b4170 2024-03-06T07:22:18.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.323+0000 7feb578df640 1 -- 172.21.15.83:0/2990295276 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb501f7bf0 con 0x7feb500b4170 2024-03-06T07:22:18.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.324+0000 7feb578df640 1 -- 172.21.15.83:0/2990295276 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb5004f330 con 0x7feb500b4170 2024-03-06T07:22:18.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.325+0000 7feb45ffb640 1 -- 172.21.15.83:0/2990295276 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feb4c01c460 con 0x7feb500b4170 2024-03-06T07:22:18.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.325+0000 7feb45ffb640 1 --2- 172.21.15.83:0/2990295276 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7feb3406b4d0 0x7feb3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.325+0000 7feb45ffb640 1 -- 172.21.15.83:0/2990295276 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feb4c089ff0 con 0x7feb500b4170 2024-03-06T07:22:18.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.325+0000 7feb54e7a640 1 --2- 172.21.15.83:0/2990295276 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7feb3406b4d0 0x7feb3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.326+0000 7feb54e7a640 1 --2- 172.21.15.83:0/2990295276 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7feb3406b4d0 0x7feb3406d990 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7feb501518f0 tx=0x7feb40008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:18.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.329+0000 7feb45ffb640 1 -- 172.21.15.83:0/2990295276 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feb4c058e70 con 0x7feb500b4170 2024-03-06T07:22:18.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.543+0000 7feb578df640 1 -- 172.21.15.83:0/2990295276 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7feb5015e880 con 0x7feb500b4170 2024-03-06T07:22:18.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.543+0000 7feb45ffb640 1 -- 172.21.15.83:0/2990295276 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) v1 ==== 53+0+73 (secure 0 0 0) 0x7feb4c058770 con 0x7feb500b4170 2024-03-06T07:22:18.545 INFO:teuthology.orchestra.run.smithi083.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:22:18.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.545+0000 7feb578df640 1 -- 172.21.15.83:0/2990295276 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7feb3406b4d0 msgr2=0x7feb3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.545+0000 7feb578df640 1 --2- 172.21.15.83:0/2990295276 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7feb3406b4d0 0x7feb3406d990 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7feb501518f0 tx=0x7feb40008040 comp rx=0 tx=0).stop 2024-03-06T07:22:18.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.545+0000 7feb578df640 1 -- 172.21.15.83:0/2990295276 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb500b4170 msgr2=0x7feb501f7330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.545+0000 7feb578df640 1 --2- 172.21.15.83:0/2990295276 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb500b4170 0x7feb501f7330 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7feb4c00c430 tx=0x7feb4c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.545+0000 7feb578df640 1 -- 172.21.15.83:0/2990295276 shutdown_connections 2024-03-06T07:22:18.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.545+0000 7feb578df640 1 --2- 172.21.15.83:0/2990295276 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7feb3406b4d0 0x7feb3406d990 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.545+0000 7feb578df640 1 --2- 172.21.15.83:0/2990295276 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feb500c9d10 0x7feb500b7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.545+0000 7feb578df640 1 --2- 172.21.15.83:0/2990295276 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feb500d0a80 0x7feb500c80b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.545+0000 7feb578df640 1 --2- 172.21.15.83:0/2990295276 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb500b4170 0x7feb501f7330 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.545+0000 7feb578df640 1 -- 172.21.15.83:0/2990295276 >> 172.21.15.83:0/2990295276 conn(0x7feb500d3c40 msgr2=0x7feb500d4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:18.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.545+0000 7feb578df640 1 -- 172.21.15.83:0/2990295276 shutdown_connections 2024-03-06T07:22:18.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:18.546+0000 7feb578df640 1 -- 172.21.15.83:0/2990295276 wait complete. 2024-03-06T07:22:18.558 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:22:18.558 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:22:18.558 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi083.front.sepia.ceph.com 2024-03-06T07:22:18.558 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-03-06T07:22:18.558 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-06T07:22:18.558 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:22:18.558 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:22:18.558 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:22:18.558 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-03-06T07:22:18.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:18.558 DEBUG:teuthology.orchestra.run.smithi083:> ip addr 2024-03-06T07:22:18.574 INFO:teuthology.orchestra.run.smithi083.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:22:18.574 INFO:teuthology.orchestra.run.smithi083.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:22:18.574 INFO:teuthology.orchestra.run.smithi083.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:22:18.575 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:18.575 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 ::1/128 scope host 2024-03-06T07:22:18.575 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:18.575 INFO:teuthology.orchestra.run.smithi083.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:18.575 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:d8:da:1e brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:18.575 INFO:teuthology.orchestra.run.smithi083.stdout: altname enp7s0f0 2024-03-06T07:22:18.575 INFO:teuthology.orchestra.run.smithi083.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:18.575 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:d8:da:1f brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:18.575 INFO:teuthology.orchestra.run.smithi083.stdout: altname enp7s0f1 2024-03-06T07:22:18.575 INFO:teuthology.orchestra.run.smithi083.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:18.575 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:8f:ce:90 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:18.575 INFO:teuthology.orchestra.run.smithi083.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:22:18.575 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:8f:ce:91 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:18.576 INFO:teuthology.orchestra.run.smithi083.stdout: inet 172.21.15.83/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:22:18.576 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft 42717sec preferred_lft 42717sec 2024-03-06T07:22:18.576 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::ec4:7aff:fe8f:ce91/64 scope link 2024-03-06T07:22:18.576 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:18.576 INFO:teuthology.orchestra.run.smithi083.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:22:18.576 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 1e:29:46:c9:e7:c8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:18.576 INFO:teuthology.orchestra.run.smithi083.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:22:18.576 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:18.576 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::60c5:d7ff:fe70:2b1c/64 scope link 2024-03-06T07:22:18.576 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:18.576 INFO:teuthology.orchestra.run.smithi083.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:22:18.576 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 1e:29:46:c9:e7:c8 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:18.576 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::1c29:46ff:fec9:e7c8/64 scope link tentative 2024-03-06T07:22:18.576 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:18.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:18.577 DEBUG:teuthology.orchestra.run.smithi083:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:22:18.640 INFO:teuthology.orchestra.run.smithi083.stdout:1 2024-03-06T07:22:18.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:18.642 DEBUG:teuthology.orchestra.run.smithi083:> route 2024-03-06T07:22:18.698 INFO:teuthology.orchestra.run.smithi083.stdout:Kernel IP routing table 2024-03-06T07:22:18.698 INFO:teuthology.orchestra.run.smithi083.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:22:18.698 INFO:teuthology.orchestra.run.smithi083.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:22:18.698 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:22:18.699 INFO:teuthology.orchestra.run.smithi083.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:22:18.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:18.699 DEBUG:teuthology.orchestra.run.smithi083:> (cd / && exec bash -c ' 2024-03-06T07:22:18.699 DEBUG:teuthology.orchestra.run.smithi083:> set -e 2024-03-06T07:22:18.699 DEBUG:teuthology.orchestra.run.smithi083:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:22:18.699 DEBUG:teuthology.orchestra.run.smithi083:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:22:18.699 DEBUG:teuthology.orchestra.run.smithi083:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:22:18.699 DEBUG:teuthology.orchestra.run.smithi083:> ') 2024-03-06T07:22:18.795 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:18.795 DEBUG:teuthology.orchestra.run.smithi083:> ip netns list 2024-03-06T07:22:18.810 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:22:18.811 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:18.811 DEBUG:teuthology.orchestra.run.smithi083:> ip netns list-id 2024-03-06T07:22:18.866 INFO:teuthology.orchestra.run.smithi083.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-03-06T07:22:18.866 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:18.866 DEBUG:teuthology.orchestra.run.smithi083:> (cd / && exec bash -c ' 2024-03-06T07:22:18.866 DEBUG:teuthology.orchestra.run.smithi083:> set -e 2024-03-06T07:22:18.866 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:22:18.867 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-03-06T07:22:18.867 DEBUG:teuthology.orchestra.run.smithi083:> ') 2024-03-06T07:22:18.950 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:18.950 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-06T07:22:19.018 INFO:teuthology.orchestra.run.smithi083.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:22:19.018 INFO:teuthology.orchestra.run.smithi083.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:22:19.018 INFO:teuthology.orchestra.run.smithi083.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:22:19.018 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:19.018 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 ::1/128 scope host 2024-03-06T07:22:19.018 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:19.018 INFO:teuthology.orchestra.run.smithi083.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:22:19.018 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 1a:20:49:33:c0:29 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-06T07:22:19.019 INFO:teuthology.orchestra.run.smithi083.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-06T07:22:19.019 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:19.019 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::1820:49ff:fe33:c029/64 scope link tentative 2024-03-06T07:22:19.019 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:19.020 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:19.020 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-06T07:22:19.086 INFO:teuthology.orchestra.run.smithi083.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:22:19.087 INFO:teuthology.orchestra.run.smithi083.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:22:19.087 INFO:teuthology.orchestra.run.smithi083.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:22:19.087 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:19.087 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 ::1/128 scope host 2024-03-06T07:22:19.087 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:19.087 INFO:teuthology.orchestra.run.smithi083.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:22:19.087 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 1a:20:49:33:c0:29 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-06T07:22:19.087 INFO:teuthology.orchestra.run.smithi083.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-06T07:22:19.088 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:19.088 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::1820:49ff:fe33:c029/64 scope link tentative 2024-03-06T07:22:19.088 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:19.089 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-03-06T07:22:19.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:19.089 DEBUG:teuthology.orchestra.run.smithi083:> (cd / && exec bash -c ' 2024-03-06T07:22:19.089 DEBUG:teuthology.orchestra.run.smithi083:> set -e 2024-03-06T07:22:19.089 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-03-06T07:22:19.090 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-06T07:22:19.090 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-03-06T07:22:19.090 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-03-06T07:22:19.090 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-03-06T07:22:19.090 DEBUG:teuthology.orchestra.run.smithi083:> ') 2024-03-06T07:22:19.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:19.215 DEBUG:teuthology.orchestra.run.smithi083:> (cd / && exec bash -c ' 2024-03-06T07:22:19.216 DEBUG:teuthology.orchestra.run.smithi083:> set -e 2024-03-06T07:22:19.216 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip link set brx.1 up 2024-03-06T07:22:19.216 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip link set dev brx.1 master ceph-brx 2024-03-06T07:22:19.216 DEBUG:teuthology.orchestra.run.smithi083:> ') 2024-03-06T07:22:19.257 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-03-06T07:22:19.257 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:19.257 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-03-06T07:22:19.312 INFO:teuthology.orchestra.run.smithi083.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-03-06T07:22:19.312 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:19.312 DEBUG:teuthology.orchestra.run.smithi083:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-03-06T07:22:19.366 DEBUG:teuthology.orchestra.run.smithi083:> sudo modprobe fuse 2024-03-06T07:22:19.434 DEBUG:teuthology.orchestra.run.smithi083:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:19.490 INFO:teuthology.orchestra.run.smithi083.stdout:/proc 2024-03-06T07:22:19.490 INFO:teuthology.orchestra.run.smithi083.stdout:/sys 2024-03-06T07:22:19.491 INFO:teuthology.orchestra.run.smithi083.stdout:/dev 2024-03-06T07:22:19.491 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/security 2024-03-06T07:22:19.491 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/shm 2024-03-06T07:22:19.491 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/pts 2024-03-06T07:22:19.491 INFO:teuthology.orchestra.run.smithi083.stdout:/run 2024-03-06T07:22:19.491 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/cgroup 2024-03-06T07:22:19.491 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/pstore 2024-03-06T07:22:19.491 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/bpf 2024-03-06T07:22:19.491 INFO:teuthology.orchestra.run.smithi083.stdout:/ 2024-03-06T07:22:19.491 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/selinux 2024-03-06T07:22:19.491 INFO:teuthology.orchestra.run.smithi083.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:19.491 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/mqueue 2024-03-06T07:22:19.491 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/hugepages 2024-03-06T07:22:19.491 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/debug 2024-03-06T07:22:19.492 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/tracing 2024-03-06T07:22:19.492 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:19.492 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:19.492 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/config 2024-03-06T07:22:19.492 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:19.492 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:19.492 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:19.492 INFO:teuthology.orchestra.run.smithi083.stdout:/run/user/1000 2024-03-06T07:22:19.492 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph 2024-03-06T07:22:19.492 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:22:19.492 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:22:19.492 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:22:19.492 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:22:19.492 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns 2024-03-06T07:22:19.493 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:19.493 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:19.493 INFO:teuthology.orchestra.run.smithi083.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:19.493 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:22:19.493 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:22:19.493 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:19.493 DEBUG:teuthology.orchestra.run.smithi083:> ls /sys/fs/fuse/connections 2024-03-06T07:22:19.544 INFO:teuthology.orchestra.run.smithi083.stdout:41 2024-03-06T07:22:19.545 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [41] 2024-03-06T07:22:19.545 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2024-03-06T07:22:19.587 DEBUG:teuthology.orchestra.run.smithi083:> sudo modprobe fuse 2024-03-06T07:22:19.613 DEBUG:teuthology.orchestra.run.smithi083:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:19.669 INFO:teuthology.orchestra.run.smithi083.stdout:/proc 2024-03-06T07:22:19.670 INFO:teuthology.orchestra.run.smithi083.stdout:/sys 2024-03-06T07:22:19.670 INFO:teuthology.orchestra.run.smithi083.stdout:/dev 2024-03-06T07:22:19.670 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/security 2024-03-06T07:22:19.670 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/shm 2024-03-06T07:22:19.670 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/pts 2024-03-06T07:22:19.670 INFO:teuthology.orchestra.run.smithi083.stdout:/run 2024-03-06T07:22:19.670 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/cgroup 2024-03-06T07:22:19.670 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/pstore 2024-03-06T07:22:19.670 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/bpf 2024-03-06T07:22:19.670 INFO:teuthology.orchestra.run.smithi083.stdout:/ 2024-03-06T07:22:19.670 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/selinux 2024-03-06T07:22:19.670 INFO:teuthology.orchestra.run.smithi083.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:19.670 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/mqueue 2024-03-06T07:22:19.671 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/hugepages 2024-03-06T07:22:19.671 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/debug 2024-03-06T07:22:19.671 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/tracing 2024-03-06T07:22:19.671 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:19.671 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:19.671 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/config 2024-03-06T07:22:19.671 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:19.671 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:19.671 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:19.671 INFO:teuthology.orchestra.run.smithi083.stdout:/run/user/1000 2024-03-06T07:22:19.671 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph 2024-03-06T07:22:19.671 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:22:19.671 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:22:19.671 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:22:19.671 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:22:19.672 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns 2024-03-06T07:22:19.672 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:19.672 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:19.672 INFO:teuthology.orchestra.run.smithi083.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:19.672 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:22:19.672 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:22:19.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:19.672 DEBUG:teuthology.orchestra.run.smithi083:> ls /sys/fs/fuse/connections 2024-03-06T07:22:19.673 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi083.stderr:2024-03-06T07:22:19.670+0000 7fb36463a480 -1 init, newargv = 0x5633047585f0 newargc=15 2024-03-06T07:22:19.673 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi083.stderr:ceph-fuse[47578]: starting ceph client 2024-03-06T07:22:19.676 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi083.stderr:ceph-fuse[47578]: starting fuse 2024-03-06T07:22:19.688 INFO:teuthology.orchestra.run.smithi083.stdout:41 2024-03-06T07:22:19.688 INFO:teuthology.orchestra.run.smithi083.stdout:44 2024-03-06T07:22:19.688 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41, 44] 2024-03-06T07:22:19.688 DEBUG:teuthology.orchestra.run.smithi083:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:22:19.688 DEBUG:teuthology.orchestra.run.smithi083:> import glob 2024-03-06T07:22:19.688 DEBUG:teuthology.orchestra.run.smithi083:> import re 2024-03-06T07:22:19.689 DEBUG:teuthology.orchestra.run.smithi083:> import os 2024-03-06T07:22:19.689 DEBUG:teuthology.orchestra.run.smithi083:> import subprocess 2024-03-06T07:22:19.689 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:22:19.689 DEBUG:teuthology.orchestra.run.smithi083:> def _find_admin_socket(client_name): 2024-03-06T07:22:19.689 DEBUG:teuthology.orchestra.run.smithi083:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-03-06T07:22:19.689 DEBUG:teuthology.orchestra.run.smithi083:> files = glob.glob(asok_path) 2024-03-06T07:22:19.689 DEBUG:teuthology.orchestra.run.smithi083:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-03-06T07:22:19.689 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:22:19.689 DEBUG:teuthology.orchestra.run.smithi083:> # Given a non-glob path, it better be there 2024-03-06T07:22:19.689 DEBUG:teuthology.orchestra.run.smithi083:> if "*" not in asok_path: 2024-03-06T07:22:19.689 DEBUG:teuthology.orchestra.run.smithi083:> assert(len(files) == 1) 2024-03-06T07:22:19.689 DEBUG:teuthology.orchestra.run.smithi083:> return files[0] 2024-03-06T07:22:19.689 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:22:19.689 DEBUG:teuthology.orchestra.run.smithi083:> for f in files: 2024-03-06T07:22:19.689 DEBUG:teuthology.orchestra.run.smithi083:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:22:19.690 DEBUG:teuthology.orchestra.run.smithi083:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:22:19.690 DEBUG:teuthology.orchestra.run.smithi083:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:22:19.690 DEBUG:teuthology.orchestra.run.smithi083:> contents = proc_f.read() 2024-03-06T07:22:19.690 DEBUG:teuthology.orchestra.run.smithi083:> if mountpoint in contents: 2024-03-06T07:22:19.690 DEBUG:teuthology.orchestra.run.smithi083:> return f 2024-03-06T07:22:19.690 DEBUG:teuthology.orchestra.run.smithi083:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:22:19.690 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:22:19.690 DEBUG:teuthology.orchestra.run.smithi083:> print(_find_admin_socket("client.2")) 2024-03-06T07:22:19.690 DEBUG:teuthology.orchestra.run.smithi083:> ' 2024-03-06T07:22:19.809 INFO:teuthology.orchestra.run.smithi083.stdout:/var/run/ceph/ceph-client.2.47578.asok 2024-03-06T07:22:19.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:22:19.819 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.47578.asok 2024-03-06T07:22:19.819 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:19.819 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.47578.asok status 2024-03-06T07:22:19.988 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:22:19.988 INFO:teuthology.orchestra.run.smithi083.stdout: "metadata": { 2024-03-06T07:22:19.989 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:22:19.989 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:22:19.989 INFO:teuthology.orchestra.run.smithi083.stdout: "entity_id": "2", 2024-03-06T07:22:19.989 INFO:teuthology.orchestra.run.smithi083.stdout: "hostname": "smithi083", 2024-03-06T07:22:19.989 INFO:teuthology.orchestra.run.smithi083.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-03-06T07:22:19.989 INFO:teuthology.orchestra.run.smithi083.stdout: "pid": "47578", 2024-03-06T07:22:19.989 INFO:teuthology.orchestra.run.smithi083.stdout: "root": "/" 2024-03-06T07:22:19.989 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:22:19.989 INFO:teuthology.orchestra.run.smithi083.stdout: "dentry_count": 0, 2024-03-06T07:22:19.989 INFO:teuthology.orchestra.run.smithi083.stdout: "dentry_pinned_count": 0, 2024-03-06T07:22:19.989 INFO:teuthology.orchestra.run.smithi083.stdout: "id": 4716, 2024-03-06T07:22:19.990 INFO:teuthology.orchestra.run.smithi083.stdout: "inst": { 2024-03-06T07:22:19.990 INFO:teuthology.orchestra.run.smithi083.stdout: "name": { 2024-03-06T07:22:19.990 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "client", 2024-03-06T07:22:19.990 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 4716 2024-03-06T07:22:19.990 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:22:19.990 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": { 2024-03-06T07:22:19.990 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-03-06T07:22:19.990 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "192.168.0.2:0", 2024-03-06T07:22:19.990 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 3640935036 2024-03-06T07:22:19.990 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:22:19.990 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:22:19.990 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": { 2024-03-06T07:22:19.991 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-03-06T07:22:19.991 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "192.168.0.2:0", 2024-03-06T07:22:19.991 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 3640935036 2024-03-06T07:22:19.991 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:22:19.991 INFO:teuthology.orchestra.run.smithi083.stdout: "inst_str": "client.4716 192.168.0.2:0/3640935036", 2024-03-06T07:22:19.991 INFO:teuthology.orchestra.run.smithi083.stdout: "addr_str": "192.168.0.2:0/3640935036", 2024-03-06T07:22:19.991 INFO:teuthology.orchestra.run.smithi083.stdout: "inode_count": 1, 2024-03-06T07:22:19.991 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_epoch": 6, 2024-03-06T07:22:19.991 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_epoch": 20, 2024-03-06T07:22:19.991 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:22:19.991 INFO:teuthology.orchestra.run.smithi083.stdout: "blocklisted": false, 2024-03-06T07:22:19.991 INFO:teuthology.orchestra.run.smithi083.stdout: "fs_name": "cephfs" 2024-03-06T07:22:19.992 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:22:20.000 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:22:20.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.100+0000 7f65a5dbd640 1 Processor -- start 2024-03-06T07:22:20.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.100+0000 7f65a5dbd640 1 -- start start 2024-03-06T07:22:20.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.100+0000 7f65a5dbd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f65a015dfe0 0x7f65a00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.100+0000 7f65a5dbd640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65a00b3430 con 0x7f65a00d29c0 2024-03-06T07:22:20.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.100+0000 7f65a5dbd640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65a00b3710 con 0x7f65a00cf830 2024-03-06T07:22:20.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.100+0000 7f65a5dbd640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65a00b31e0 con 0x7f65a015dfe0 2024-03-06T07:22:20.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.101+0000 7f659ffff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f65a00d29c0 0x7f65a00b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:55766/0 (socket says 172.21.15.83:55766) 2024-03-06T07:22:20.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.101+0000 7f659ffff640 1 -- 172.21.15.83:0/1167429591 learned_addr learned my addr 172.21.15.83:0/1167429591 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:20.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.101+0000 7f659f7fe640 1 --2- 172.21.15.83:0/1167429591 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f65a015dfe0 0x7f65a00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.101+0000 7f659e7fc640 1 -- 172.21.15.83:0/1167429591 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2407583798 0 0) 0x7f65a00b3430 con 0x7f65a00d29c0 2024-03-06T07:22:20.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.101+0000 7f659e7fc640 1 -- 172.21.15.83:0/1167429591 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6588003520 con 0x7f65a00d29c0 2024-03-06T07:22:20.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.101+0000 7f659e7fc640 1 -- 172.21.15.83:0/1167429591 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2502152574 0 0) 0x7f65a00b3710 con 0x7f65a00cf830 2024-03-06T07:22:20.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.101+0000 7f659e7fc640 1 -- 172.21.15.83:0/1167429591 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65a00b3430 con 0x7f65a00cf830 2024-03-06T07:22:20.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.101+0000 7f659f7fe640 1 -- 172.21.15.83:0/1167429591 >> v1:172.21.15.183:6789/0 conn(0x7f65a00cf830 legacy=0x7f65a00d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.101+0000 7f659f7fe640 1 -- 172.21.15.83:0/1167429591 >> v1:172.21.15.83:6789/0 conn(0x7f65a00d29c0 legacy=0x7f65a00b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.101+0000 7f659f7fe640 1 -- 172.21.15.83:0/1167429591 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65a00b39f0 con 0x7f65a015dfe0 2024-03-06T07:22:20.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.102+0000 7f659f7fe640 1 --2- 172.21.15.83:0/1167429591 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f65a015dfe0 0x7f65a00cf2f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f65900097d0 tx=0x7f659002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79e19b81adcc3ed8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:20.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.102+0000 7f659e7fc640 1 -- 172.21.15.83:0/1167429591 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6590035020 con 0x7f65a015dfe0 2024-03-06T07:22:20.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.102+0000 7f659e7fc640 1 -- 172.21.15.83:0/1167429591 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6590031890 con 0x7f65a015dfe0 2024-03-06T07:22:20.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.102+0000 7f659e7fc640 1 -- 172.21.15.83:0/1167429591 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6590031c60 con 0x7f65a015dfe0 2024-03-06T07:22:20.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.102+0000 7f65a5dbd640 1 -- 172.21.15.83:0/1167429591 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f65a015dfe0 msgr2=0x7f65a00cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.102+0000 7f65a5dbd640 1 --2- 172.21.15.83:0/1167429591 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f65a015dfe0 0x7f65a00cf2f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f65900097d0 tx=0x7f659002b930 comp rx=0 tx=0).stop 2024-03-06T07:22:20.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.102+0000 7f65a5dbd640 1 -- 172.21.15.83:0/1167429591 shutdown_connections 2024-03-06T07:22:20.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.102+0000 7f65a5dbd640 1 --2- 172.21.15.83:0/1167429591 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f65a015dfe0 0x7f65a00cf2f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.102+0000 7f65a5dbd640 1 -- 172.21.15.83:0/1167429591 >> 172.21.15.83:0/1167429591 conn(0x7f65a00d3cd0 msgr2=0x7f65a00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:20.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.102+0000 7f65a5dbd640 1 -- 172.21.15.83:0/1167429591 shutdown_connections 2024-03-06T07:22:20.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.103+0000 7f65a5dbd640 1 -- 172.21.15.83:0/1167429591 wait complete. 2024-03-06T07:22:20.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.104+0000 7f65a5dbd640 1 Processor -- start 2024-03-06T07:22:20.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.104+0000 7f65a5dbd640 1 -- start start 2024-03-06T07:22:20.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.104+0000 7f65a5dbd640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f65a00cf830 0x7f65a00a1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.104+0000 7f65a5dbd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f65a00d29c0 0x7f65a014dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.104+0000 7f65a5dbd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65a015dfe0 0x7f65a014e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.104+0000 7f65a5dbd640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a015e3c0 con 0x7f65a015dfe0 2024-03-06T07:22:20.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.104+0000 7f65a5dbd640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a00b3d60 con 0x7f65a00cf830 2024-03-06T07:22:20.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.104+0000 7f65a5dbd640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65a015e540 con 0x7f65a00d29c0 2024-03-06T07:22:20.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.105+0000 7f659ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65a015dfe0 0x7f65a014e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.105+0000 7f659ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65a015dfe0 0x7f65a014e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48704/0 (socket says 172.21.15.83:48704) 2024-03-06T07:22:20.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.105+0000 7f659ffff640 1 -- 172.21.15.83:0/3680655827 learned_addr learned my addr 172.21.15.83:0/3680655827 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:20.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.105+0000 7f659f7fe640 1 --2- 172.21.15.83:0/3680655827 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f65a00cf830 0x7f65a00a1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.105+0000 7f659effd640 1 --2- 172.21.15.83:0/3680655827 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f65a00d29c0 0x7f65a014dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.105+0000 7f659ffff640 1 -- 172.21.15.83:0/3680655827 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f65a00d29c0 msgr2=0x7f65a014dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.105+0000 7f659ffff640 1 --2- 172.21.15.83:0/3680655827 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f65a00d29c0 0x7f65a014dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.105+0000 7f659ffff640 1 -- 172.21.15.83:0/3680655827 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f65a00cf830 msgr2=0x7f65a00a1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.105+0000 7f659ffff640 1 --2- 172.21.15.83:0/3680655827 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f65a00cf830 0x7f65a00a1670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.105+0000 7f659ffff640 1 -- 172.21.15.83:0/3680655827 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65a014e8e0 con 0x7f65a015dfe0 2024-03-06T07:22:20.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.105+0000 7f659ffff640 1 --2- 172.21.15.83:0/3680655827 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65a015dfe0 0x7f65a014e2b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f659800cba0 tx=0x7f659800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:20.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.105+0000 7f6587fff640 1 -- 172.21.15.83:0/3680655827 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6598017070 con 0x7f65a015dfe0 2024-03-06T07:22:20.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.105+0000 7f6587fff640 1 -- 172.21.15.83:0/3680655827 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f659800ace0 con 0x7f65a015dfe0 2024-03-06T07:22:20.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.105+0000 7f65a5dbd640 1 -- 172.21.15.83:0/3680655827 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65a014ebc0 con 0x7f65a015dfe0 2024-03-06T07:22:20.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.105+0000 7f6587fff640 1 -- 172.21.15.83:0/3680655827 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6598012620 con 0x7f65a015dfe0 2024-03-06T07:22:20.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.105+0000 7f65a5dbd640 1 -- 172.21.15.83:0/3680655827 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65a014f180 con 0x7f65a015dfe0 2024-03-06T07:22:20.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.106+0000 7f65a5dbd640 1 -- 172.21.15.83:0/3680655827 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65a004f330 con 0x7f65a015dfe0 2024-03-06T07:22:20.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.107+0000 7f6587fff640 1 -- 172.21.15.83:0/3680655827 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f659801a070 con 0x7f65a015dfe0 2024-03-06T07:22:20.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.107+0000 7f6587fff640 1 --2- 172.21.15.83:0/3680655827 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f658c06b4d0 0x7f658c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.107+0000 7f6587fff640 1 -- 172.21.15.83:0/3680655827 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f659808a020 con 0x7f65a015dfe0 2024-03-06T07:22:20.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.107+0000 7f659f7fe640 1 --2- 172.21.15.83:0/3680655827 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f658c06b4d0 0x7f658c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.108+0000 7f659f7fe640 1 --2- 172.21.15.83:0/3680655827 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f658c06b4d0 0x7f658c06d990 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6590036000 tx=0x7f659003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:20.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.112+0000 7f6587fff640 1 -- 172.21.15.83:0/3680655827 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f65980587d0 con 0x7f65a015dfe0 2024-03-06T07:22:20.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.321+0000 7f65a5dbd640 1 -- 172.21.15.83:0/3680655827 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f65a006a090 con 0x7f65a015dfe0 2024-03-06T07:22:20.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.322+0000 7f6587fff640 1 -- 172.21.15.83:0/3680655827 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) v1 ==== 53+0+73 (secure 0 0 0) 0x7f659805c870 con 0x7f65a015dfe0 2024-03-06T07:22:20.324 INFO:teuthology.orchestra.run.smithi083.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:22:20.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.324+0000 7f65a5dbd640 1 -- 172.21.15.83:0/3680655827 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f658c06b4d0 msgr2=0x7f658c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.324+0000 7f65a5dbd640 1 --2- 172.21.15.83:0/3680655827 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f658c06b4d0 0x7f658c06d990 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6590036000 tx=0x7f659003c040 comp rx=0 tx=0).stop 2024-03-06T07:22:20.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.324+0000 7f65a5dbd640 1 -- 172.21.15.83:0/3680655827 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65a015dfe0 msgr2=0x7f65a014e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.324+0000 7f65a5dbd640 1 --2- 172.21.15.83:0/3680655827 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65a015dfe0 0x7f65a014e2b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f659800cba0 tx=0x7f659800a430 comp rx=0 tx=0).stop 2024-03-06T07:22:20.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.324+0000 7f65a5dbd640 1 -- 172.21.15.83:0/3680655827 shutdown_connections 2024-03-06T07:22:20.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.324+0000 7f65a5dbd640 1 --2- 172.21.15.83:0/3680655827 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f658c06b4d0 0x7f658c06d990 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.324+0000 7f65a5dbd640 1 --2- 172.21.15.83:0/3680655827 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65a015dfe0 0x7f65a014e2b0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.324+0000 7f65a5dbd640 1 --2- 172.21.15.83:0/3680655827 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f65a00d29c0 0x7f65a014dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.324+0000 7f65a5dbd640 1 --2- 172.21.15.83:0/3680655827 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f65a00cf830 0x7f65a00a1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.324+0000 7f65a5dbd640 1 -- 172.21.15.83:0/3680655827 >> 172.21.15.83:0/3680655827 conn(0x7f65a00d3cd0 msgr2=0x7f65a00b4d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:20.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.324+0000 7f65a5dbd640 1 -- 172.21.15.83:0/3680655827 shutdown_connections 2024-03-06T07:22:20.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:20.324+0000 7f65a5dbd640 1 -- 172.21.15.83:0/3680655827 wait complete. 2024-03-06T07:22:20.337 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:22:20.337 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:22:20.337 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi183.front.sepia.ceph.com 2024-03-06T07:22:20.337 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-06T07:22:20.337 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:20.337 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:22:20.337 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:22:20.337 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:22:20.337 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-06T07:22:20.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:20.337 DEBUG:teuthology.orchestra.run.smithi183:> ip addr 2024-03-06T07:22:20.357 INFO:teuthology.orchestra.run.smithi183.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:22:20.357 INFO:teuthology.orchestra.run.smithi183.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:22:20.357 INFO:teuthology.orchestra.run.smithi183.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:22:20.357 INFO:teuthology.orchestra.run.smithi183.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:20.357 INFO:teuthology.orchestra.run.smithi183.stdout: inet6 ::1/128 scope host 2024-03-06T07:22:20.357 INFO:teuthology.orchestra.run.smithi183.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:20.357 INFO:teuthology.orchestra.run.smithi183.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:20.358 INFO:teuthology.orchestra.run.smithi183.stdout: link/ether 0c:c4:7a:d9:38:aa brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:20.358 INFO:teuthology.orchestra.run.smithi183.stdout: altname enp7s0f0 2024-03-06T07:22:20.358 INFO:teuthology.orchestra.run.smithi183.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:20.358 INFO:teuthology.orchestra.run.smithi183.stdout: link/ether 0c:c4:7a:d9:38:ab brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:20.358 INFO:teuthology.orchestra.run.smithi183.stdout: altname enp7s0f1 2024-03-06T07:22:20.358 INFO:teuthology.orchestra.run.smithi183.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:20.358 INFO:teuthology.orchestra.run.smithi183.stdout: link/ether 0c:c4:7a:88:80:84 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:20.358 INFO:teuthology.orchestra.run.smithi183.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:22:20.358 INFO:teuthology.orchestra.run.smithi183.stdout: link/ether 0c:c4:7a:88:80:85 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:20.358 INFO:teuthology.orchestra.run.smithi183.stdout: inet 172.21.15.183/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:22:20.358 INFO:teuthology.orchestra.run.smithi183.stdout: valid_lft 42639sec preferred_lft 42639sec 2024-03-06T07:22:20.358 INFO:teuthology.orchestra.run.smithi183.stdout: inet6 fe80::ec4:7aff:fe88:8085/64 scope link 2024-03-06T07:22:20.358 INFO:teuthology.orchestra.run.smithi183.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:20.358 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-06T07:22:20.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:20.359 DEBUG:teuthology.orchestra.run.smithi183:> (cd / && exec bash -c ' 2024-03-06T07:22:20.359 DEBUG:teuthology.orchestra.run.smithi183:> set -e 2024-03-06T07:22:20.359 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip link add name ceph-brx type bridge 2024-03-06T07:22:20.359 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip addr flush dev ceph-brx 2024-03-06T07:22:20.359 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip link set ceph-brx up 2024-03-06T07:22:20.359 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-06T07:22:20.359 DEBUG:teuthology.orchestra.run.smithi183:> ') 2024-03-06T07:22:20.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:20.544 DEBUG:teuthology.orchestra.run.smithi183:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:22:20.571 INFO:teuthology.orchestra.run.smithi183.stdout:1 2024-03-06T07:22:20.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:20.573 DEBUG:teuthology.orchestra.run.smithi183:> route 2024-03-06T07:22:20.650 INFO:teuthology.orchestra.run.smithi183.stdout:Kernel IP routing table 2024-03-06T07:22:20.650 INFO:teuthology.orchestra.run.smithi183.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:22:20.651 INFO:teuthology.orchestra.run.smithi183.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:22:20.651 INFO:teuthology.orchestra.run.smithi183.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:22:20.651 INFO:teuthology.orchestra.run.smithi183.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:22:20.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:20.651 DEBUG:teuthology.orchestra.run.smithi183:> (cd / && exec bash -c ' 2024-03-06T07:22:20.651 DEBUG:teuthology.orchestra.run.smithi183:> set -e 2024-03-06T07:22:20.651 DEBUG:teuthology.orchestra.run.smithi183:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:22:20.651 DEBUG:teuthology.orchestra.run.smithi183:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:22:20.652 DEBUG:teuthology.orchestra.run.smithi183:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:22:20.652 DEBUG:teuthology.orchestra.run.smithi183:> ') 2024-03-06T07:22:20.932 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:20.932 DEBUG:teuthology.orchestra.run.smithi183:> ip netns list 2024-03-06T07:22:20.947 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:20.947 DEBUG:teuthology.orchestra.run.smithi183:> ip netns list-id 2024-03-06T07:22:21.003 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:21.003 DEBUG:teuthology.orchestra.run.smithi183:> (cd / && exec bash -c ' 2024-03-06T07:22:21.003 DEBUG:teuthology.orchestra.run.smithi183:> set -e 2024-03-06T07:22:21.003 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:21.004 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-06T07:22:21.004 DEBUG:teuthology.orchestra.run.smithi183:> ') 2024-03-06T07:22:21.087 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-06T07:22:21.088 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:21.088 DEBUG:teuthology.orchestra.run.smithi183:> (cd / && exec bash -c ' 2024-03-06T07:22:21.088 DEBUG:teuthology.orchestra.run.smithi183:> set -e 2024-03-06T07:22:21.088 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-06T07:22:21.088 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-06T07:22:21.088 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-06T07:22:21.088 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-06T07:22:21.088 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-06T07:22:21.088 DEBUG:teuthology.orchestra.run.smithi183:> ') 2024-03-06T07:22:21.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:21.241 DEBUG:teuthology.orchestra.run.smithi183:> (cd / && exec bash -c ' 2024-03-06T07:22:21.241 DEBUG:teuthology.orchestra.run.smithi183:> set -e 2024-03-06T07:22:21.241 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip link set brx.0 up 2024-03-06T07:22:21.241 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip link set dev brx.0 master ceph-brx 2024-03-06T07:22:21.241 DEBUG:teuthology.orchestra.run.smithi183:> ') 2024-03-06T07:22:21.284 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-03-06T07:22:21.284 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:21.284 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:21.339 INFO:teuthology.orchestra.run.smithi183.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-06T07:22:21.339 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:21.339 DEBUG:teuthology.orchestra.run.smithi183:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:21.394 DEBUG:teuthology.orchestra.run.smithi183:> sudo modprobe fuse 2024-03-06T07:22:21.462 DEBUG:teuthology.orchestra.run.smithi183:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:21.518 INFO:teuthology.orchestra.run.smithi183.stdout:/proc 2024-03-06T07:22:21.518 INFO:teuthology.orchestra.run.smithi183.stdout:/sys 2024-03-06T07:22:21.518 INFO:teuthology.orchestra.run.smithi183.stdout:/dev 2024-03-06T07:22:21.518 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/kernel/security 2024-03-06T07:22:21.518 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/shm 2024-03-06T07:22:21.518 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/pts 2024-03-06T07:22:21.518 INFO:teuthology.orchestra.run.smithi183.stdout:/run 2024-03-06T07:22:21.518 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/fs/cgroup 2024-03-06T07:22:21.518 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/fs/pstore 2024-03-06T07:22:21.518 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/fs/bpf 2024-03-06T07:22:21.518 INFO:teuthology.orchestra.run.smithi183.stdout:/ 2024-03-06T07:22:21.519 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/fs/selinux 2024-03-06T07:22:21.519 INFO:teuthology.orchestra.run.smithi183.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:21.519 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/hugepages 2024-03-06T07:22:21.519 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/mqueue 2024-03-06T07:22:21.519 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/kernel/debug 2024-03-06T07:22:21.519 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/kernel/tracing 2024-03-06T07:22:21.519 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:21.519 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/kernel/config 2024-03-06T07:22:21.519 INFO:teuthology.orchestra.run.smithi183.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:21.519 INFO:teuthology.orchestra.run.smithi183.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:21.519 INFO:teuthology.orchestra.run.smithi183.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:21.519 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:21.519 INFO:teuthology.orchestra.run.smithi183.stdout:/run/user/1000 2024-03-06T07:22:21.519 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/ceph 2024-03-06T07:22:21.519 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/ceph/osd/ceph-4 2024-03-06T07:22:21.520 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/ceph/osd/ceph-5 2024-03-06T07:22:21.520 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/ceph/osd/ceph-6 2024-03-06T07:22:21.520 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/ceph/osd/ceph-7 2024-03-06T07:22:21.520 INFO:teuthology.orchestra.run.smithi183.stdout:/run/netns 2024-03-06T07:22:21.520 INFO:teuthology.orchestra.run.smithi183.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:21.520 INFO:teuthology.orchestra.run.smithi183.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:21.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:21.520 DEBUG:teuthology.orchestra.run.smithi183:> ls /sys/fs/fuse/connections 2024-03-06T07:22:21.572 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:22:21.572 DEBUG:teuthology.orchestra.run.smithi183:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-03-06T07:22:21.614 DEBUG:teuthology.orchestra.run.smithi183:> sudo modprobe fuse 2024-03-06T07:22:21.644 DEBUG:teuthology.orchestra.run.smithi183:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:21.699 INFO:teuthology.orchestra.run.smithi183.stdout:/proc 2024-03-06T07:22:21.699 INFO:teuthology.orchestra.run.smithi183.stdout:/sys 2024-03-06T07:22:21.699 INFO:teuthology.orchestra.run.smithi183.stdout:/dev 2024-03-06T07:22:21.699 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/kernel/security 2024-03-06T07:22:21.699 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/shm 2024-03-06T07:22:21.699 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/pts 2024-03-06T07:22:21.699 INFO:teuthology.orchestra.run.smithi183.stdout:/run 2024-03-06T07:22:21.699 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/fs/cgroup 2024-03-06T07:22:21.699 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/fs/pstore 2024-03-06T07:22:21.700 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/fs/bpf 2024-03-06T07:22:21.700 INFO:teuthology.orchestra.run.smithi183.stdout:/ 2024-03-06T07:22:21.700 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/fs/selinux 2024-03-06T07:22:21.700 INFO:teuthology.orchestra.run.smithi183.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:21.700 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/hugepages 2024-03-06T07:22:21.700 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/mqueue 2024-03-06T07:22:21.700 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/kernel/debug 2024-03-06T07:22:21.700 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/kernel/tracing 2024-03-06T07:22:21.700 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:21.700 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/kernel/config 2024-03-06T07:22:21.701 INFO:teuthology.orchestra.run.smithi183.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:21.701 INFO:teuthology.orchestra.run.smithi183.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:21.701 INFO:teuthology.orchestra.run.smithi183.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:21.701 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:21.701 INFO:teuthology.orchestra.run.smithi183.stdout:/run/user/1000 2024-03-06T07:22:21.701 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/ceph 2024-03-06T07:22:21.701 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/ceph/osd/ceph-4 2024-03-06T07:22:21.701 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/ceph/osd/ceph-5 2024-03-06T07:22:21.701 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/ceph/osd/ceph-6 2024-03-06T07:22:21.701 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/ceph/osd/ceph-7 2024-03-06T07:22:21.701 INFO:teuthology.orchestra.run.smithi183.stdout:/run/netns 2024-03-06T07:22:21.701 INFO:teuthology.orchestra.run.smithi183.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:21.701 INFO:teuthology.orchestra.run.smithi183.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:21.702 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:21.702 DEBUG:teuthology.orchestra.run.smithi183:> ls /sys/fs/fuse/connections 2024-03-06T07:22:21.703 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi183.stderr:2024-03-06T07:22:21.699+0000 7f02d1c3b480 -1 init, newargv = 0x556a4ef26810 newargc=15 2024-03-06T07:22:21.703 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi183.stderr:ceph-fuse[44426]: starting ceph client 2024-03-06T07:22:21.705 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi183.stderr:ceph-fuse[44426]: starting fuse 2024-03-06T07:22:21.717 INFO:teuthology.orchestra.run.smithi183.stdout:42 2024-03-06T07:22:21.717 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-03-06T07:22:21.717 DEBUG:teuthology.orchestra.run.smithi183:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:22:21.717 DEBUG:teuthology.orchestra.run.smithi183:> import glob 2024-03-06T07:22:21.717 DEBUG:teuthology.orchestra.run.smithi183:> import re 2024-03-06T07:22:21.718 DEBUG:teuthology.orchestra.run.smithi183:> import os 2024-03-06T07:22:21.718 DEBUG:teuthology.orchestra.run.smithi183:> import subprocess 2024-03-06T07:22:21.718 DEBUG:teuthology.orchestra.run.smithi183:> 2024-03-06T07:22:21.718 DEBUG:teuthology.orchestra.run.smithi183:> def _find_admin_socket(client_name): 2024-03-06T07:22:21.718 DEBUG:teuthology.orchestra.run.smithi183:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-03-06T07:22:21.718 DEBUG:teuthology.orchestra.run.smithi183:> files = glob.glob(asok_path) 2024-03-06T07:22:21.718 DEBUG:teuthology.orchestra.run.smithi183:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-03-06T07:22:21.718 DEBUG:teuthology.orchestra.run.smithi183:> 2024-03-06T07:22:21.718 DEBUG:teuthology.orchestra.run.smithi183:> # Given a non-glob path, it better be there 2024-03-06T07:22:21.718 DEBUG:teuthology.orchestra.run.smithi183:> if "*" not in asok_path: 2024-03-06T07:22:21.718 DEBUG:teuthology.orchestra.run.smithi183:> assert(len(files) == 1) 2024-03-06T07:22:21.718 DEBUG:teuthology.orchestra.run.smithi183:> return files[0] 2024-03-06T07:22:21.718 DEBUG:teuthology.orchestra.run.smithi183:> 2024-03-06T07:22:21.718 DEBUG:teuthology.orchestra.run.smithi183:> for f in files: 2024-03-06T07:22:21.718 DEBUG:teuthology.orchestra.run.smithi183:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:22:21.718 DEBUG:teuthology.orchestra.run.smithi183:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:22:21.719 DEBUG:teuthology.orchestra.run.smithi183:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:22:21.719 DEBUG:teuthology.orchestra.run.smithi183:> contents = proc_f.read() 2024-03-06T07:22:21.719 DEBUG:teuthology.orchestra.run.smithi183:> if mountpoint in contents: 2024-03-06T07:22:21.719 DEBUG:teuthology.orchestra.run.smithi183:> return f 2024-03-06T07:22:21.719 DEBUG:teuthology.orchestra.run.smithi183:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:22:21.719 DEBUG:teuthology.orchestra.run.smithi183:> 2024-03-06T07:22:21.719 DEBUG:teuthology.orchestra.run.smithi183:> print(_find_admin_socket("client.1")) 2024-03-06T07:22:21.719 DEBUG:teuthology.orchestra.run.smithi183:> ' 2024-03-06T07:22:21.839 INFO:teuthology.orchestra.run.smithi183.stdout:/var/run/ceph/ceph-client.1.44426.asok 2024-03-06T07:22:21.842 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:22:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:22:21.849 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.44426.asok 2024-03-06T07:22:21.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:21.850 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.44426.asok status 2024-03-06T07:22:21.989 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-03-06T07:22:21.990 INFO:teuthology.orchestra.run.smithi183.stdout: "metadata": { 2024-03-06T07:22:21.990 INFO:teuthology.orchestra.run.smithi183.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:22:21.990 INFO:teuthology.orchestra.run.smithi183.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:22:21.990 INFO:teuthology.orchestra.run.smithi183.stdout: "entity_id": "1", 2024-03-06T07:22:21.990 INFO:teuthology.orchestra.run.smithi183.stdout: "hostname": "smithi183", 2024-03-06T07:22:21.990 INFO:teuthology.orchestra.run.smithi183.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-03-06T07:22:21.990 INFO:teuthology.orchestra.run.smithi183.stdout: "pid": "44426", 2024-03-06T07:22:21.990 INFO:teuthology.orchestra.run.smithi183.stdout: "root": "/" 2024-03-06T07:22:21.990 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-03-06T07:22:21.990 INFO:teuthology.orchestra.run.smithi183.stdout: "dentry_count": 0, 2024-03-06T07:22:21.990 INFO:teuthology.orchestra.run.smithi183.stdout: "dentry_pinned_count": 0, 2024-03-06T07:22:21.991 INFO:teuthology.orchestra.run.smithi183.stdout: "id": 4630, 2024-03-06T07:22:21.991 INFO:teuthology.orchestra.run.smithi183.stdout: "inst": { 2024-03-06T07:22:21.991 INFO:teuthology.orchestra.run.smithi183.stdout: "name": { 2024-03-06T07:22:21.991 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "client", 2024-03-06T07:22:21.991 INFO:teuthology.orchestra.run.smithi183.stdout: "num": 4630 2024-03-06T07:22:21.991 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-03-06T07:22:21.991 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": { 2024-03-06T07:22:21.991 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-03-06T07:22:21.991 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:22:21.991 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 295576134 2024-03-06T07:22:21.992 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-03-06T07:22:21.992 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-03-06T07:22:21.992 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": { 2024-03-06T07:22:21.992 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-03-06T07:22:21.992 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:22:21.992 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 295576134 2024-03-06T07:22:21.992 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-03-06T07:22:21.992 INFO:teuthology.orchestra.run.smithi183.stdout: "inst_str": "client.4630 192.168.0.1:0/295576134", 2024-03-06T07:22:21.992 INFO:teuthology.orchestra.run.smithi183.stdout: "addr_str": "192.168.0.1:0/295576134", 2024-03-06T07:22:21.992 INFO:teuthology.orchestra.run.smithi183.stdout: "inode_count": 1, 2024-03-06T07:22:21.992 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_epoch": 6, 2024-03-06T07:22:21.992 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_epoch": 20, 2024-03-06T07:22:21.993 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:22:21.993 INFO:teuthology.orchestra.run.smithi183.stdout: "blocklisted": false, 2024-03-06T07:22:21.993 INFO:teuthology.orchestra.run.smithi183.stdout: "fs_name": "cephfs" 2024-03-06T07:22:21.993 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-03-06T07:22:21.998 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:22:22.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.104+0000 7f937f30c640 1 Processor -- start 2024-03-06T07:22:22.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.104+0000 7f937f30c640 1 -- start start 2024-03-06T07:22:22.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.105+0000 7f937f30c640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f937815fa10 0x7f93780ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.105+0000 7f937f30c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93780cebe0 con 0x7f937815bdd0 2024-03-06T07:22:22.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.105+0000 7f937f30c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93780ceec0 con 0x7f93780d16e0 2024-03-06T07:22:22.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.105+0000 7f937f30c640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93780cf1a0 con 0x7f937815fa10 2024-03-06T07:22:22.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.105+0000 7f937d0a8640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f937815bdd0 0x7f93780c5610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:55768/0 (socket says 172.21.15.83:55768) 2024-03-06T07:22:22.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.105+0000 7f937d0a8640 1 -- 172.21.15.83:0/3172862710 learned_addr learned my addr 172.21.15.83:0/3172862710 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:22.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.105+0000 7f937c8a7640 1 --2- 172.21.15.83:0/3172862710 >> v2:172.21.15.183:3300/0 conn(0x7f937815fa10 0x7f93780ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.105+0000 7f9377fff640 1 -- 172.21.15.83:0/3172862710 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4014369970 0 0) 0x7f93780cebe0 con 0x7f937815bdd0 2024-03-06T07:22:22.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.105+0000 7f9377fff640 1 -- 172.21.15.83:0/3172862710 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9360003520 con 0x7f937815bdd0 2024-03-06T07:22:22.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.105+0000 7f9377fff640 1 -- 172.21.15.83:0/3172862710 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1729108288 0 0) 0x7f93780ceec0 con 0x7f93780d16e0 2024-03-06T07:22:22.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.105+0000 7f9377fff640 1 -- 172.21.15.83:0/3172862710 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93780cebe0 con 0x7f93780d16e0 2024-03-06T07:22:22.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.105+0000 7f9377fff640 1 -- 172.21.15.83:0/3172862710 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3949848861 0 0) 0x7f9360003520 con 0x7f937815bdd0 2024-03-06T07:22:22.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.105+0000 7f937c8a7640 1 -- 172.21.15.83:0/3172862710 >> v1:172.21.15.183:6789/0 conn(0x7f93780d16e0 legacy=0x7f93780d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.105+0000 7f937c8a7640 1 -- 172.21.15.83:0/3172862710 >> v1:172.21.15.83:6789/0 conn(0x7f937815bdd0 legacy=0x7f93780c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.105+0000 7f937c8a7640 1 -- 172.21.15.83:0/3172862710 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93780cc8e0 con 0x7f937815fa10 2024-03-06T07:22:22.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.106+0000 7f937c8a7640 1 --2- 172.21.15.83:0/3172862710 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f937815fa10 0x7f93780ce270 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f936c0097a0 tx=0x7f936c02b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53e5840532e856bf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:22.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.106+0000 7f9377fff640 1 -- 172.21.15.83:0/3172862710 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f936c037070 con 0x7f937815fa10 2024-03-06T07:22:22.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.106+0000 7f9377fff640 1 -- 172.21.15.83:0/3172862710 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f936c030510 con 0x7f937815fa10 2024-03-06T07:22:22.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.106+0000 7f9377fff640 1 -- 172.21.15.83:0/3172862710 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9368001690 con 0x7f937815fa10 2024-03-06T07:22:22.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.107+0000 7f937f30c640 1 -- 172.21.15.83:0/3172862710 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f937815fa10 msgr2=0x7f93780ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.107+0000 7f937f30c640 1 --2- 172.21.15.83:0/3172862710 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f937815fa10 0x7f93780ce270 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f936c0097a0 tx=0x7f936c02b790 comp rx=0 tx=0).stop 2024-03-06T07:22:22.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.107+0000 7f937f30c640 1 -- 172.21.15.83:0/3172862710 shutdown_connections 2024-03-06T07:22:22.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.107+0000 7f937f30c640 1 --2- 172.21.15.83:0/3172862710 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f937815fa10 0x7f93780ce270 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.107+0000 7f937f30c640 1 -- 172.21.15.83:0/3172862710 >> 172.21.15.83:0/3172862710 conn(0x7f937809a9f0 msgr2=0x7f937809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:22.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.107+0000 7f937f30c640 1 -- 172.21.15.83:0/3172862710 shutdown_connections 2024-03-06T07:22:22.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.107+0000 7f937f30c640 1 -- 172.21.15.83:0/3172862710 wait complete. 2024-03-06T07:22:22.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.108+0000 7f937f30c640 1 Processor -- start 2024-03-06T07:22:22.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.108+0000 7f937f30c640 1 -- start start 2024-03-06T07:22:22.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.109+0000 7f937f30c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f93780d16e0 0x7f93780b5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.109+0000 7f937f30c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f937815bdd0 0x7f93780b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.109+0000 7f937f30c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f937815fa10 0x7f9378155470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.109+0000 7f937f30c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93780ce7b0 con 0x7f937815bdd0 2024-03-06T07:22:22.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.109+0000 7f937f30c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f937815c1b0 con 0x7f93780d16e0 2024-03-06T07:22:22.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.109+0000 7f937f30c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93780ce930 con 0x7f937815fa10 2024-03-06T07:22:22.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.109+0000 7f937c8a7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f937815bdd0 0x7f93780b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.109+0000 7f937c8a7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f937815bdd0 0x7f93780b7230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48706/0 (socket says 172.21.15.83:48706) 2024-03-06T07:22:22.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.109+0000 7f937c8a7640 1 -- 172.21.15.83:0/4205841851 learned_addr learned my addr 172.21.15.83:0/4205841851 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:22.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.109+0000 7f937d0a8640 1 --2- 172.21.15.83:0/4205841851 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f93780d16e0 0x7f93780b5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.109+0000 7f937d8a9640 1 --2- 172.21.15.83:0/4205841851 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f937815fa10 0x7f9378155470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.109+0000 7f937c8a7640 1 -- 172.21.15.83:0/4205841851 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f937815fa10 msgr2=0x7f9378155470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.109+0000 7f937c8a7640 1 --2- 172.21.15.83:0/4205841851 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f937815fa10 0x7f9378155470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.109+0000 7f937c8a7640 1 -- 172.21.15.83:0/4205841851 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f93780d16e0 msgr2=0x7f93780b5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.109+0000 7f937c8a7640 1 --2- 172.21.15.83:0/4205841851 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f93780d16e0 0x7f93780b5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.109+0000 7f937c8a7640 1 -- 172.21.15.83:0/4205841851 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93780b5750 con 0x7f937815bdd0 2024-03-06T07:22:22.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.110+0000 7f937c8a7640 1 --2- 172.21.15.83:0/4205841851 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f937815bdd0 0x7f93780b7230 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f936c0098c0 tx=0x7f936c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:22.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.110+0000 7f9375ffb640 1 -- 172.21.15.83:0/4205841851 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f936c042070 con 0x7f937815bdd0 2024-03-06T07:22:22.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.110+0000 7f9375ffb640 1 -- 172.21.15.83:0/4205841851 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f936c03e870 con 0x7f937815bdd0 2024-03-06T07:22:22.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.110+0000 7f937f30c640 1 -- 172.21.15.83:0/4205841851 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93780b6190 con 0x7f937815bdd0 2024-03-06T07:22:22.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.110+0000 7f9375ffb640 1 -- 172.21.15.83:0/4205841851 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f936c037040 con 0x7f937815bdd0 2024-03-06T07:22:22.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.110+0000 7f937f30c640 1 -- 172.21.15.83:0/4205841851 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93780b6750 con 0x7f937815bdd0 2024-03-06T07:22:22.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.111+0000 7f937f30c640 1 -- 172.21.15.83:0/4205841851 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f937804f330 con 0x7f937815bdd0 2024-03-06T07:22:22.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.111+0000 7f9375ffb640 1 -- 172.21.15.83:0/4205841851 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f936c045070 con 0x7f937815bdd0 2024-03-06T07:22:22.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.112+0000 7f9375ffb640 1 --2- 172.21.15.83:0/4205841851 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f936406b500 0x7f936406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.112+0000 7f9375ffb640 1 -- 172.21.15.83:0/4205841851 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f936c0abea0 con 0x7f937815bdd0 2024-03-06T07:22:22.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.112+0000 7f937d0a8640 1 --2- 172.21.15.83:0/4205841851 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f936406b500 0x7f936406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.112+0000 7f937d0a8640 1 --2- 172.21.15.83:0/4205841851 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f936406b500 0x7f936406d9c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9368000c00 tx=0x7f9368001040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:22.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.116+0000 7f9375ffb640 1 -- 172.21.15.83:0/4205841851 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f936c07a800 con 0x7f937815bdd0 2024-03-06T07:22:22.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.392+0000 7f937f30c640 1 -- 172.21.15.83:0/4205841851 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f93780b7a40 con 0x7f937815bdd0 2024-03-06T07:22:22.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.392+0000 7f9375ffb640 1 -- 172.21.15.83:0/4205841851 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) v1 ==== 53+0+73 (secure 0 0 0) 0x7f936c07e8a0 con 0x7f937815bdd0 2024-03-06T07:22:22.395 INFO:teuthology.orchestra.run.smithi083.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:22:22.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.394+0000 7f937f30c640 1 -- 172.21.15.83:0/4205841851 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f936406b500 msgr2=0x7f936406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.394+0000 7f937f30c640 1 --2- 172.21.15.83:0/4205841851 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f936406b500 0x7f936406d9c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9368000c00 tx=0x7f9368001040 comp rx=0 tx=0).stop 2024-03-06T07:22:22.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.394+0000 7f937f30c640 1 -- 172.21.15.83:0/4205841851 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f937815bdd0 msgr2=0x7f93780b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.394+0000 7f937f30c640 1 --2- 172.21.15.83:0/4205841851 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f937815bdd0 0x7f93780b7230 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f936c0098c0 tx=0x7f936c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.394+0000 7f937f30c640 1 -- 172.21.15.83:0/4205841851 shutdown_connections 2024-03-06T07:22:22.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.394+0000 7f937f30c640 1 --2- 172.21.15.83:0/4205841851 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f936406b500 0x7f936406d9c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.394+0000 7f937f30c640 1 --2- 172.21.15.83:0/4205841851 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f937815fa10 0x7f9378155470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.394+0000 7f937f30c640 1 --2- 172.21.15.83:0/4205841851 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f937815bdd0 0x7f93780b7230 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.394+0000 7f937f30c640 1 --2- 172.21.15.83:0/4205841851 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f93780d16e0 0x7f93780b5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.394+0000 7f937f30c640 1 -- 172.21.15.83:0/4205841851 >> 172.21.15.83:0/4205841851 conn(0x7f937809a9f0 msgr2=0x7f937809add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:22.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.395+0000 7f937f30c640 1 -- 172.21.15.83:0/4205841851 shutdown_connections 2024-03-06T07:22:22.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:22.395+0000 7f937f30c640 1 -- 172.21.15.83:0/4205841851 wait complete. 2024-03-06T07:22:22.408 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:22:22.408 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:22:22.408 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi183.front.sepia.ceph.com 2024-03-06T07:22:22.408 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-03-06T07:22:22.408 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-06T07:22:22.408 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:22:22.408 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:22:22.408 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:22:22.409 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-03-06T07:22:22.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:22.409 DEBUG:teuthology.orchestra.run.smithi183:> ip addr 2024-03-06T07:22:22.424 INFO:teuthology.orchestra.run.smithi183.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:22:22.424 INFO:teuthology.orchestra.run.smithi183.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:22:22.424 INFO:teuthology.orchestra.run.smithi183.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:22:22.424 INFO:teuthology.orchestra.run.smithi183.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:22.425 INFO:teuthology.orchestra.run.smithi183.stdout: inet6 ::1/128 scope host 2024-03-06T07:22:22.425 INFO:teuthology.orchestra.run.smithi183.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:22.425 INFO:teuthology.orchestra.run.smithi183.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:22.425 INFO:teuthology.orchestra.run.smithi183.stdout: link/ether 0c:c4:7a:d9:38:aa brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:22.425 INFO:teuthology.orchestra.run.smithi183.stdout: altname enp7s0f0 2024-03-06T07:22:22.425 INFO:teuthology.orchestra.run.smithi183.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:22.425 INFO:teuthology.orchestra.run.smithi183.stdout: link/ether 0c:c4:7a:d9:38:ab brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:22.425 INFO:teuthology.orchestra.run.smithi183.stdout: altname enp7s0f1 2024-03-06T07:22:22.425 INFO:teuthology.orchestra.run.smithi183.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:22.425 INFO:teuthology.orchestra.run.smithi183.stdout: link/ether 0c:c4:7a:88:80:84 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:22.425 INFO:teuthology.orchestra.run.smithi183.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:22:22.425 INFO:teuthology.orchestra.run.smithi183.stdout: link/ether 0c:c4:7a:88:80:85 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:22.425 INFO:teuthology.orchestra.run.smithi183.stdout: inet 172.21.15.183/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:22:22.425 INFO:teuthology.orchestra.run.smithi183.stdout: valid_lft 42637sec preferred_lft 42637sec 2024-03-06T07:22:22.426 INFO:teuthology.orchestra.run.smithi183.stdout: inet6 fe80::ec4:7aff:fe88:8085/64 scope link 2024-03-06T07:22:22.426 INFO:teuthology.orchestra.run.smithi183.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:22.426 INFO:teuthology.orchestra.run.smithi183.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:22:22.426 INFO:teuthology.orchestra.run.smithi183.stdout: link/ether 92:70:03:c6:6c:9b brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:22.426 INFO:teuthology.orchestra.run.smithi183.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:22:22.426 INFO:teuthology.orchestra.run.smithi183.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:22.426 INFO:teuthology.orchestra.run.smithi183.stdout: inet6 fe80::88a3:15ff:feb2:f609/64 scope link 2024-03-06T07:22:22.426 INFO:teuthology.orchestra.run.smithi183.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:22.426 INFO:teuthology.orchestra.run.smithi183.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:22:22.426 INFO:teuthology.orchestra.run.smithi183.stdout: link/ether 92:70:03:c6:6c:9b brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:22.426 INFO:teuthology.orchestra.run.smithi183.stdout: inet6 fe80::9070:3ff:fec6:6c9b/64 scope link tentative 2024-03-06T07:22:22.426 INFO:teuthology.orchestra.run.smithi183.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:22.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:22.427 DEBUG:teuthology.orchestra.run.smithi183:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:22:22.491 INFO:teuthology.orchestra.run.smithi183.stdout:1 2024-03-06T07:22:22.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:22.492 DEBUG:teuthology.orchestra.run.smithi183:> route 2024-03-06T07:22:22.548 INFO:teuthology.orchestra.run.smithi183.stdout:Kernel IP routing table 2024-03-06T07:22:22.548 INFO:teuthology.orchestra.run.smithi183.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:22:22.548 INFO:teuthology.orchestra.run.smithi183.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:22:22.548 INFO:teuthology.orchestra.run.smithi183.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:22:22.548 INFO:teuthology.orchestra.run.smithi183.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:22:22.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:22.549 DEBUG:teuthology.orchestra.run.smithi183:> (cd / && exec bash -c ' 2024-03-06T07:22:22.549 DEBUG:teuthology.orchestra.run.smithi183:> set -e 2024-03-06T07:22:22.549 DEBUG:teuthology.orchestra.run.smithi183:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:22:22.549 DEBUG:teuthology.orchestra.run.smithi183:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:22:22.549 DEBUG:teuthology.orchestra.run.smithi183:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:22:22.549 DEBUG:teuthology.orchestra.run.smithi183:> ') 2024-03-06T07:22:22.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:22.644 DEBUG:teuthology.orchestra.run.smithi183:> ip netns list 2024-03-06T07:22:22.659 INFO:teuthology.orchestra.run.smithi183.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-06T07:22:22.659 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:22.659 DEBUG:teuthology.orchestra.run.smithi183:> ip netns list-id 2024-03-06T07:22:22.715 INFO:teuthology.orchestra.run.smithi183.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-03-06T07:22:22.715 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:22.715 DEBUG:teuthology.orchestra.run.smithi183:> (cd / && exec bash -c ' 2024-03-06T07:22:22.715 DEBUG:teuthology.orchestra.run.smithi183:> set -e 2024-03-06T07:22:22.715 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-06T07:22:22.715 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-03-06T07:22:22.716 DEBUG:teuthology.orchestra.run.smithi183:> ') 2024-03-06T07:22:22.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:22.798 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-06T07:22:22.864 INFO:teuthology.orchestra.run.smithi183.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:22:22.865 INFO:teuthology.orchestra.run.smithi183.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:22:22.865 INFO:teuthology.orchestra.run.smithi183.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:22:22.865 INFO:teuthology.orchestra.run.smithi183.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:22.865 INFO:teuthology.orchestra.run.smithi183.stdout: inet6 ::1/128 scope host 2024-03-06T07:22:22.865 INFO:teuthology.orchestra.run.smithi183.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:22.865 INFO:teuthology.orchestra.run.smithi183.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:22:22.865 INFO:teuthology.orchestra.run.smithi183.stdout: link/ether 6e:d0:46:e9:ee:83 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-06T07:22:22.865 INFO:teuthology.orchestra.run.smithi183.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-06T07:22:22.865 INFO:teuthology.orchestra.run.smithi183.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:22.865 INFO:teuthology.orchestra.run.smithi183.stdout: inet6 fe80::6cd0:46ff:fee9:ee83/64 scope link tentative 2024-03-06T07:22:22.865 INFO:teuthology.orchestra.run.smithi183.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:22.866 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:22.866 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-06T07:22:22.933 INFO:teuthology.orchestra.run.smithi183.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:22:22.933 INFO:teuthology.orchestra.run.smithi183.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:22:22.933 INFO:teuthology.orchestra.run.smithi183.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:22:22.933 INFO:teuthology.orchestra.run.smithi183.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:22.933 INFO:teuthology.orchestra.run.smithi183.stdout: inet6 ::1/128 scope host 2024-03-06T07:22:22.933 INFO:teuthology.orchestra.run.smithi183.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:22.933 INFO:teuthology.orchestra.run.smithi183.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:22:22.933 INFO:teuthology.orchestra.run.smithi183.stdout: link/ether 6e:d0:46:e9:ee:83 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-06T07:22:22.933 INFO:teuthology.orchestra.run.smithi183.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-06T07:22:22.933 INFO:teuthology.orchestra.run.smithi183.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:22.934 INFO:teuthology.orchestra.run.smithi183.stdout: inet6 fe80::6cd0:46ff:fee9:ee83/64 scope link tentative 2024-03-06T07:22:22.934 INFO:teuthology.orchestra.run.smithi183.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:22.935 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-03-06T07:22:22.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:22.935 DEBUG:teuthology.orchestra.run.smithi183:> (cd / && exec bash -c ' 2024-03-06T07:22:22.935 DEBUG:teuthology.orchestra.run.smithi183:> set -e 2024-03-06T07:22:22.935 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-03-06T07:22:22.935 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-06T07:22:22.935 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-03-06T07:22:22.935 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-03-06T07:22:22.935 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-03-06T07:22:22.935 DEBUG:teuthology.orchestra.run.smithi183:> ') 2024-03-06T07:22:23.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:23.063 DEBUG:teuthology.orchestra.run.smithi183:> (cd / && exec bash -c ' 2024-03-06T07:22:23.063 DEBUG:teuthology.orchestra.run.smithi183:> set -e 2024-03-06T07:22:23.063 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip link set brx.1 up 2024-03-06T07:22:23.063 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip link set dev brx.1 master ceph-brx 2024-03-06T07:22:23.063 DEBUG:teuthology.orchestra.run.smithi183:> ') 2024-03-06T07:22:23.105 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-03-06T07:22:23.105 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:23.105 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-03-06T07:22:23.161 INFO:teuthology.orchestra.run.smithi183.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-03-06T07:22:23.161 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:23.162 DEBUG:teuthology.orchestra.run.smithi183:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-03-06T07:22:23.216 DEBUG:teuthology.orchestra.run.smithi183:> sudo modprobe fuse 2024-03-06T07:22:23.285 DEBUG:teuthology.orchestra.run.smithi183:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:23.341 INFO:teuthology.orchestra.run.smithi183.stdout:/proc 2024-03-06T07:22:23.341 INFO:teuthology.orchestra.run.smithi183.stdout:/sys 2024-03-06T07:22:23.341 INFO:teuthology.orchestra.run.smithi183.stdout:/dev 2024-03-06T07:22:23.342 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/kernel/security 2024-03-06T07:22:23.342 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/shm 2024-03-06T07:22:23.342 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/pts 2024-03-06T07:22:23.342 INFO:teuthology.orchestra.run.smithi183.stdout:/run 2024-03-06T07:22:23.342 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/fs/cgroup 2024-03-06T07:22:23.342 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/fs/pstore 2024-03-06T07:22:23.342 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/fs/bpf 2024-03-06T07:22:23.342 INFO:teuthology.orchestra.run.smithi183.stdout:/ 2024-03-06T07:22:23.342 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/fs/selinux 2024-03-06T07:22:23.342 INFO:teuthology.orchestra.run.smithi183.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:23.342 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/hugepages 2024-03-06T07:22:23.342 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/mqueue 2024-03-06T07:22:23.342 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/kernel/debug 2024-03-06T07:22:23.342 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/kernel/tracing 2024-03-06T07:22:23.343 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:23.343 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/kernel/config 2024-03-06T07:22:23.343 INFO:teuthology.orchestra.run.smithi183.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:23.343 INFO:teuthology.orchestra.run.smithi183.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:23.343 INFO:teuthology.orchestra.run.smithi183.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:23.343 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:23.343 INFO:teuthology.orchestra.run.smithi183.stdout:/run/user/1000 2024-03-06T07:22:23.343 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/ceph 2024-03-06T07:22:23.343 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/ceph/osd/ceph-4 2024-03-06T07:22:23.343 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/ceph/osd/ceph-5 2024-03-06T07:22:23.343 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/ceph/osd/ceph-6 2024-03-06T07:22:23.343 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/ceph/osd/ceph-7 2024-03-06T07:22:23.344 INFO:teuthology.orchestra.run.smithi183.stdout:/run/netns 2024-03-06T07:22:23.344 INFO:teuthology.orchestra.run.smithi183.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:23.344 INFO:teuthology.orchestra.run.smithi183.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:23.344 INFO:teuthology.orchestra.run.smithi183.stdout:/home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:23.344 INFO:teuthology.orchestra.run.smithi183.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-06T07:22:23.344 INFO:teuthology.orchestra.run.smithi183.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-06T07:22:23.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:23.344 DEBUG:teuthology.orchestra.run.smithi183:> ls /sys/fs/fuse/connections 2024-03-06T07:22:23.396 INFO:teuthology.orchestra.run.smithi183.stdout:42 2024-03-06T07:22:23.397 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2024-03-06T07:22:23.397 DEBUG:teuthology.orchestra.run.smithi183:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2024-03-06T07:22:23.439 DEBUG:teuthology.orchestra.run.smithi183:> sudo modprobe fuse 2024-03-06T07:22:23.465 DEBUG:teuthology.orchestra.run.smithi183:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:23.521 INFO:teuthology.orchestra.run.smithi183.stdout:/proc 2024-03-06T07:22:23.521 INFO:teuthology.orchestra.run.smithi183.stdout:/sys 2024-03-06T07:22:23.522 INFO:teuthology.orchestra.run.smithi183.stdout:/dev 2024-03-06T07:22:23.522 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/kernel/security 2024-03-06T07:22:23.522 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/shm 2024-03-06T07:22:23.522 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/pts 2024-03-06T07:22:23.522 INFO:teuthology.orchestra.run.smithi183.stdout:/run 2024-03-06T07:22:23.522 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/fs/cgroup 2024-03-06T07:22:23.522 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/fs/pstore 2024-03-06T07:22:23.522 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/fs/bpf 2024-03-06T07:22:23.522 INFO:teuthology.orchestra.run.smithi183.stdout:/ 2024-03-06T07:22:23.522 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/fs/selinux 2024-03-06T07:22:23.522 INFO:teuthology.orchestra.run.smithi183.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:23.522 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/hugepages 2024-03-06T07:22:23.523 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/mqueue 2024-03-06T07:22:23.523 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/kernel/debug 2024-03-06T07:22:23.523 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/kernel/tracing 2024-03-06T07:22:23.523 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:23.523 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/kernel/config 2024-03-06T07:22:23.523 INFO:teuthology.orchestra.run.smithi183.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:23.523 INFO:teuthology.orchestra.run.smithi183.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:23.523 INFO:teuthology.orchestra.run.smithi183.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:23.523 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:23.523 INFO:teuthology.orchestra.run.smithi183.stdout:/run/user/1000 2024-03-06T07:22:23.523 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/ceph 2024-03-06T07:22:23.523 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/ceph/osd/ceph-4 2024-03-06T07:22:23.523 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/ceph/osd/ceph-5 2024-03-06T07:22:23.524 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/ceph/osd/ceph-6 2024-03-06T07:22:23.524 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/ceph/osd/ceph-7 2024-03-06T07:22:23.524 INFO:teuthology.orchestra.run.smithi183.stdout:/run/netns 2024-03-06T07:22:23.524 INFO:teuthology.orchestra.run.smithi183.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:23.524 INFO:teuthology.orchestra.run.smithi183.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:23.524 INFO:teuthology.orchestra.run.smithi183.stdout:/home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:23.524 INFO:teuthology.orchestra.run.smithi183.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-06T07:22:23.524 INFO:teuthology.orchestra.run.smithi183.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-06T07:22:23.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:23.524 DEBUG:teuthology.orchestra.run.smithi183:> ls /sys/fs/fuse/connections 2024-03-06T07:22:23.525 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi183.stderr:2024-03-06T07:22:23.521+0000 7f682df56480 -1 init, newargv = 0x55cfca494b30 newargc=15 2024-03-06T07:22:23.525 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi183.stderr:ceph-fuse[44879]: starting ceph client 2024-03-06T07:22:23.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi183.stderr:ceph-fuse[44879]: starting fuse 2024-03-06T07:22:23.539 INFO:teuthology.orchestra.run.smithi183.stdout:42 2024-03-06T07:22:23.539 INFO:teuthology.orchestra.run.smithi183.stdout:44 2024-03-06T07:22:23.539 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 44] 2024-03-06T07:22:23.540 DEBUG:teuthology.orchestra.run.smithi183:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:22:23.540 DEBUG:teuthology.orchestra.run.smithi183:> import glob 2024-03-06T07:22:23.540 DEBUG:teuthology.orchestra.run.smithi183:> import re 2024-03-06T07:22:23.540 DEBUG:teuthology.orchestra.run.smithi183:> import os 2024-03-06T07:22:23.540 DEBUG:teuthology.orchestra.run.smithi183:> import subprocess 2024-03-06T07:22:23.540 DEBUG:teuthology.orchestra.run.smithi183:> 2024-03-06T07:22:23.540 DEBUG:teuthology.orchestra.run.smithi183:> def _find_admin_socket(client_name): 2024-03-06T07:22:23.540 DEBUG:teuthology.orchestra.run.smithi183:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-03-06T07:22:23.540 DEBUG:teuthology.orchestra.run.smithi183:> files = glob.glob(asok_path) 2024-03-06T07:22:23.540 DEBUG:teuthology.orchestra.run.smithi183:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-03-06T07:22:23.540 DEBUG:teuthology.orchestra.run.smithi183:> 2024-03-06T07:22:23.540 DEBUG:teuthology.orchestra.run.smithi183:> # Given a non-glob path, it better be there 2024-03-06T07:22:23.540 DEBUG:teuthology.orchestra.run.smithi183:> if "*" not in asok_path: 2024-03-06T07:22:23.540 DEBUG:teuthology.orchestra.run.smithi183:> assert(len(files) == 1) 2024-03-06T07:22:23.541 DEBUG:teuthology.orchestra.run.smithi183:> return files[0] 2024-03-06T07:22:23.541 DEBUG:teuthology.orchestra.run.smithi183:> 2024-03-06T07:22:23.541 DEBUG:teuthology.orchestra.run.smithi183:> for f in files: 2024-03-06T07:22:23.541 DEBUG:teuthology.orchestra.run.smithi183:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:22:23.541 DEBUG:teuthology.orchestra.run.smithi183:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:22:23.541 DEBUG:teuthology.orchestra.run.smithi183:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:22:23.541 DEBUG:teuthology.orchestra.run.smithi183:> contents = proc_f.read() 2024-03-06T07:22:23.541 DEBUG:teuthology.orchestra.run.smithi183:> if mountpoint in contents: 2024-03-06T07:22:23.541 DEBUG:teuthology.orchestra.run.smithi183:> return f 2024-03-06T07:22:23.541 DEBUG:teuthology.orchestra.run.smithi183:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:22:23.541 DEBUG:teuthology.orchestra.run.smithi183:> 2024-03-06T07:22:23.541 DEBUG:teuthology.orchestra.run.smithi183:> print(_find_admin_socket("client.3")) 2024-03-06T07:22:23.541 DEBUG:teuthology.orchestra.run.smithi183:> ' 2024-03-06T07:22:23.659 INFO:teuthology.orchestra.run.smithi183.stdout:/var/run/ceph/ceph-client.3.44879.asok 2024-03-06T07:22:23.662 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-06T07:22:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:22:23.670 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.44879.asok 2024-03-06T07:22:23.670 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:23.670 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.44879.asok status 2024-03-06T07:22:23.805 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-03-06T07:22:23.806 INFO:teuthology.orchestra.run.smithi183.stdout: "metadata": { 2024-03-06T07:22:23.806 INFO:teuthology.orchestra.run.smithi183.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:22:23.806 INFO:teuthology.orchestra.run.smithi183.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:22:23.806 INFO:teuthology.orchestra.run.smithi183.stdout: "entity_id": "3", 2024-03-06T07:22:23.806 INFO:teuthology.orchestra.run.smithi183.stdout: "hostname": "smithi183", 2024-03-06T07:22:23.806 INFO:teuthology.orchestra.run.smithi183.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-03-06T07:22:23.806 INFO:teuthology.orchestra.run.smithi183.stdout: "pid": "44879", 2024-03-06T07:22:23.806 INFO:teuthology.orchestra.run.smithi183.stdout: "root": "/" 2024-03-06T07:22:23.806 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-03-06T07:22:23.806 INFO:teuthology.orchestra.run.smithi183.stdout: "dentry_count": 0, 2024-03-06T07:22:23.807 INFO:teuthology.orchestra.run.smithi183.stdout: "dentry_pinned_count": 0, 2024-03-06T07:22:23.807 INFO:teuthology.orchestra.run.smithi183.stdout: "id": 4645, 2024-03-06T07:22:23.807 INFO:teuthology.orchestra.run.smithi183.stdout: "inst": { 2024-03-06T07:22:23.807 INFO:teuthology.orchestra.run.smithi183.stdout: "name": { 2024-03-06T07:22:23.807 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "client", 2024-03-06T07:22:23.807 INFO:teuthology.orchestra.run.smithi183.stdout: "num": 4645 2024-03-06T07:22:23.807 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-03-06T07:22:23.807 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": { 2024-03-06T07:22:23.807 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-03-06T07:22:23.807 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "192.168.0.2:0", 2024-03-06T07:22:23.807 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 841376101 2024-03-06T07:22:23.807 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-03-06T07:22:23.807 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-03-06T07:22:23.807 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": { 2024-03-06T07:22:23.808 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-03-06T07:22:23.808 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "192.168.0.2:0", 2024-03-06T07:22:23.808 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 841376101 2024-03-06T07:22:23.808 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-03-06T07:22:23.808 INFO:teuthology.orchestra.run.smithi183.stdout: "inst_str": "client.4645 192.168.0.2:0/841376101", 2024-03-06T07:22:23.808 INFO:teuthology.orchestra.run.smithi183.stdout: "addr_str": "192.168.0.2:0/841376101", 2024-03-06T07:22:23.808 INFO:teuthology.orchestra.run.smithi183.stdout: "inode_count": 1, 2024-03-06T07:22:23.808 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_epoch": 6, 2024-03-06T07:22:23.808 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_epoch": 20, 2024-03-06T07:22:23.808 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:22:23.808 INFO:teuthology.orchestra.run.smithi183.stdout: "blocklisted": false, 2024-03-06T07:22:23.808 INFO:teuthology.orchestra.run.smithi183.stdout: "fs_name": "cephfs" 2024-03-06T07:22:23.808 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-03-06T07:22:23.814 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:23.814 DEBUG:teuthology.orchestra.run.smithi083:> stat --file-system '--printf=%T 2024-03-06T07:22:23.814 DEBUG:teuthology.orchestra.run.smithi083:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:23.829 INFO:teuthology.orchestra.run.smithi083.stdout:fuseblk 2024-03-06T07:22:23.829 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:23.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:23.830 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:23.902 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:23.903 DEBUG:teuthology.orchestra.run.smithi083:> stat --file-system '--printf=%T 2024-03-06T07:22:23.903 DEBUG:teuthology.orchestra.run.smithi083:> ' -- /home/ubuntu/cephtest/mnt.2 2024-03-06T07:22:23.957 INFO:teuthology.orchestra.run.smithi083.stdout:fuseblk 2024-03-06T07:22:23.958 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2024-03-06T07:22:23.958 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:23.958 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-03-06T07:22:24.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:24.027 DEBUG:teuthology.orchestra.run.smithi183:> stat --file-system '--printf=%T 2024-03-06T07:22:24.027 DEBUG:teuthology.orchestra.run.smithi183:> ' -- /home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:24.043 INFO:teuthology.orchestra.run.smithi183.stdout:fuseblk 2024-03-06T07:22:24.043 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:24.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:24.043 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:24.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:24.111 DEBUG:teuthology.orchestra.run.smithi183:> stat --file-system '--printf=%T 2024-03-06T07:22:24.111 DEBUG:teuthology.orchestra.run.smithi183:> ' -- /home/ubuntu/cephtest/mnt.3 2024-03-06T07:22:24.167 INFO:teuthology.orchestra.run.smithi183.stdout:fuseblk 2024-03-06T07:22:24.167 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2024-03-06T07:22:24.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:24.167 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-03-06T07:22:24.234 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-03-06T07:22:24.252 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:24.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.352+0000 7f8f00c47640 1 Processor -- start 2024-03-06T07:22:24.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.352+0000 7f8f00c47640 1 -- start start 2024-03-06T07:22:24.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.352+0000 7f8f00c47640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f8efc0d4760 0x7f8efc0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.352+0000 7f8f00c47640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8efc0d25d0 con 0x7f8efc0b4170 2024-03-06T07:22:24.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.352+0000 7f8f00c47640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8efc0d2870 con 0x7f8efc0d0a80 2024-03-06T07:22:24.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.352+0000 7f8f00c47640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8efc0d4bf0 con 0x7f8efc0d4760 2024-03-06T07:22:24.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.353+0000 7f8efa59c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f8efc0b4170 0x7f8efc0d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:55780/0 (socket says 172.21.15.83:55780) 2024-03-06T07:22:24.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.353+0000 7f8efa59c640 1 -- 172.21.15.83:0/2770360998 learned_addr learned my addr 172.21.15.83:0/2770360998 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:24.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.353+0000 7f8ef3fff640 1 --2- 172.21.15.83:0/2770360998 >> v2:172.21.15.183:3300/0 conn(0x7f8efc0d4760 0x7f8efc0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.353+0000 7f8ef9d9b640 1 -- 172.21.15.83:0/2770360998 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3393584870 0 0) 0x7f8efc0d25d0 con 0x7f8efc0b4170 2024-03-06T07:22:24.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.353+0000 7f8ef9d9b640 1 -- 172.21.15.83:0/2770360998 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ee8003520 con 0x7f8efc0b4170 2024-03-06T07:22:24.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.353+0000 7f8ef3fff640 1 -- 172.21.15.83:0/2770360998 >> v1:172.21.15.183:6789/0 conn(0x7f8efc0d0a80 legacy=0x7f8efc0cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.353+0000 7f8ef3fff640 1 -- 172.21.15.83:0/2770360998 >> v1:172.21.15.83:6789/0 conn(0x7f8efc0b4170 legacy=0x7f8efc0d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.353+0000 7f8ef3fff640 1 -- 172.21.15.83:0/2770360998 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8efc0d2b50 con 0x7f8efc0d4760 2024-03-06T07:22:24.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.354+0000 7f8ef3fff640 1 --2- 172.21.15.83:0/2770360998 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8efc0d4760 0x7f8efc0b6a90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8ee40097a0 tx=0x7f8ee402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a358c00aa1ef9747 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.354+0000 7f8ef9d9b640 1 -- 172.21.15.83:0/2770360998 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee4034070 con 0x7f8efc0d4760 2024-03-06T07:22:24.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.354+0000 7f8ef9d9b640 1 -- 172.21.15.83:0/2770360998 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ee4030610 con 0x7f8efc0d4760 2024-03-06T07:22:24.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.354+0000 7f8ef9d9b640 1 -- 172.21.15.83:0/2770360998 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee40309e0 con 0x7f8efc0d4760 2024-03-06T07:22:24.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.355+0000 7f8f00c47640 1 -- 172.21.15.83:0/2770360998 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8efc0d4760 msgr2=0x7f8efc0b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.355+0000 7f8f00c47640 1 --2- 172.21.15.83:0/2770360998 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8efc0d4760 0x7f8efc0b6a90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8ee40097a0 tx=0x7f8ee402b960 comp rx=0 tx=0).stop 2024-03-06T07:22:24.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.355+0000 7f8f00c47640 1 -- 172.21.15.83:0/2770360998 shutdown_connections 2024-03-06T07:22:24.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.355+0000 7f8f00c47640 1 --2- 172.21.15.83:0/2770360998 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8efc0d4760 0x7f8efc0b6a90 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.355+0000 7f8f00c47640 1 -- 172.21.15.83:0/2770360998 >> 172.21.15.83:0/2770360998 conn(0x7f8efc0d3c40 msgr2=0x7f8efc0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:24.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.355+0000 7f8f00c47640 1 -- 172.21.15.83:0/2770360998 shutdown_connections 2024-03-06T07:22:24.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.355+0000 7f8f00c47640 1 -- 172.21.15.83:0/2770360998 wait complete. 2024-03-06T07:22:24.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.356+0000 7f8f00c47640 1 Processor -- start 2024-03-06T07:22:24.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.356+0000 7f8f00c47640 1 -- start start 2024-03-06T07:22:24.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.357+0000 7f8f00c47640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8efc0b4170 0x7f8efc0cb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.357+0000 7f8f00c47640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8efc0d0a80 0x7f8efc15b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.357+0000 7f8f00c47640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8efc156f50 0x7f8efc0b21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.357+0000 7f8f00c47640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8efc0b4dd0 con 0x7f8efc0d0a80 2024-03-06T07:22:24.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.357+0000 7f8f00c47640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8efc0ce9e0 con 0x7f8efc0b4170 2024-03-06T07:22:24.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.357+0000 7f8f00c47640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8efc0b4f50 con 0x7f8efc156f50 2024-03-06T07:22:24.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.357+0000 7f8ef3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8efc0d0a80 0x7f8efc15b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.357+0000 7f8ef3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8efc0d0a80 0x7f8efc15b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48716/0 (socket says 172.21.15.83:48716) 2024-03-06T07:22:24.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.357+0000 7f8ef3fff640 1 -- 172.21.15.83:0/2531207319 learned_addr learned my addr 172.21.15.83:0/2531207319 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:24.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.357+0000 7f8efad9d640 1 --2- 172.21.15.83:0/2531207319 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8efc156f50 0x7f8efc0b21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.357+0000 7f8efa59c640 1 --2- 172.21.15.83:0/2531207319 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8efc0b4170 0x7f8efc0cb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.357+0000 7f8ef3fff640 1 -- 172.21.15.83:0/2531207319 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8efc156f50 msgr2=0x7f8efc0b21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.357+0000 7f8ef3fff640 1 --2- 172.21.15.83:0/2531207319 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8efc156f50 0x7f8efc0b21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.357+0000 7f8ef3fff640 1 -- 172.21.15.83:0/2531207319 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8efc0b4170 msgr2=0x7f8efc0cb8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.357+0000 7f8ef3fff640 1 --2- 172.21.15.83:0/2531207319 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8efc0b4170 0x7f8efc0cb8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.357+0000 7f8ef3fff640 1 -- 172.21.15.83:0/2531207319 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8efc15c140 con 0x7f8efc0d0a80 2024-03-06T07:22:24.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.357+0000 7f8efa59c640 1 --2- 172.21.15.83:0/2531207319 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8efc0b4170 0x7f8efc0cb8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:24.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.357+0000 7f8ef3fff640 1 --2- 172.21.15.83:0/2531207319 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8efc0d0a80 0x7f8efc15b9d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8ee40098c0 tx=0x7f8ee40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.357+0000 7f8ef2ffd640 1 -- 172.21.15.83:0/2531207319 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee4034040 con 0x7f8efc0d0a80 2024-03-06T07:22:24.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.358+0000 7f8ef2ffd640 1 -- 172.21.15.83:0/2531207319 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ee403aaa0 con 0x7f8efc0d0a80 2024-03-06T07:22:24.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.358+0000 7f8ef2ffd640 1 -- 172.21.15.83:0/2531207319 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee40443f0 con 0x7f8efc0d0a80 2024-03-06T07:22:24.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.358+0000 7f8f00c47640 1 -- 172.21.15.83:0/2531207319 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8efc15c420 con 0x7f8efc0d0a80 2024-03-06T07:22:24.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.358+0000 7f8f00c47640 1 -- 172.21.15.83:0/2531207319 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8efc15c9e0 con 0x7f8efc0d0a80 2024-03-06T07:22:24.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.359+0000 7f8f00c47640 1 -- 172.21.15.83:0/2531207319 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8efc04f330 con 0x7f8efc0d0a80 2024-03-06T07:22:24.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.359+0000 7f8ef2ffd640 1 -- 172.21.15.83:0/2531207319 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8ee404b060 con 0x7f8efc0d0a80 2024-03-06T07:22:24.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.360+0000 7f8ef2ffd640 1 --2- 172.21.15.83:0/2531207319 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8edc06b4d0 0x7f8edc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.360+0000 7f8ef2ffd640 1 -- 172.21.15.83:0/2531207319 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8ee40abee0 con 0x7f8efc0d0a80 2024-03-06T07:22:24.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.360+0000 7f8efa59c640 1 --2- 172.21.15.83:0/2531207319 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8edc06b4d0 0x7f8edc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.360+0000 7f8efa59c640 1 --2- 172.21.15.83:0/2531207319 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8edc06b4d0 0x7f8edc06d990 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8eec005880 tx=0x7f8eec009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.364+0000 7f8ef2ffd640 1 -- 172.21.15.83:0/2531207319 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8ee403f070 con 0x7f8efc0d0a80 2024-03-06T07:22:24.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.646+0000 7f8f00c47640 1 -- 172.21.15.83:0/2531207319 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8efc0733b0 con 0x7f8efc0d0a80 2024-03-06T07:22:24.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.647+0000 7f8ef2ffd640 1 -- 172.21.15.83:0/2531207319 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16617 (secure 0 0 0) 0x7f8ee407ad70 con 0x7f8efc0d0a80 2024-03-06T07:22:24.649 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:22:24.650 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":20,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:21:59.938137+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.318942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.254795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:24.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.649+0000 7f8f00c47640 1 -- 172.21.15.83:0/2531207319 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8edc06b4d0 msgr2=0x7f8edc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.649+0000 7f8f00c47640 1 --2- 172.21.15.83:0/2531207319 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8edc06b4d0 0x7f8edc06d990 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8eec005880 tx=0x7f8eec009040 comp rx=0 tx=0).stop 2024-03-06T07:22:24.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.649+0000 7f8f00c47640 1 -- 172.21.15.83:0/2531207319 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8efc0d0a80 msgr2=0x7f8efc15b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.649+0000 7f8f00c47640 1 --2- 172.21.15.83:0/2531207319 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8efc0d0a80 0x7f8efc15b9d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8ee40098c0 tx=0x7f8ee40046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.649+0000 7f8f00c47640 1 -- 172.21.15.83:0/2531207319 shutdown_connections 2024-03-06T07:22:24.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.649+0000 7f8f00c47640 1 --2- 172.21.15.83:0/2531207319 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8edc06b4d0 0x7f8edc06d990 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.649+0000 7f8f00c47640 1 --2- 172.21.15.83:0/2531207319 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8efc156f50 0x7f8efc0b21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.649+0000 7f8f00c47640 1 --2- 172.21.15.83:0/2531207319 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8efc0d0a80 0x7f8efc15b9d0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.649+0000 7f8f00c47640 1 --2- 172.21.15.83:0/2531207319 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8efc0b4170 0x7f8efc0cb8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.649+0000 7f8f00c47640 1 -- 172.21.15.83:0/2531207319 >> 172.21.15.83:0/2531207319 conn(0x7f8efc0d3c40 msgr2=0x7f8efc0cee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:24.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.649+0000 7f8f00c47640 1 -- 172.21.15.83:0/2531207319 shutdown_connections 2024-03-06T07:22:24.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.649+0000 7f8f00c47640 1 -- 172.21.15.83:0/2531207319 wait complete. 2024-03-06T07:22:24.665 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:56.318942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:57.254795+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:24.665 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:24.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.767+0000 7f2ece4b5640 1 Processor -- start 2024-03-06T07:22:24.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.767+0000 7f2ece4b5640 1 -- start start 2024-03-06T07:22:24.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.768+0000 7f2ece4b5640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2ec80b1e40 0x7f2ec8168b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.768+0000 7f2ece4b5640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ec80d2730 con 0x7f2ec81614d0 2024-03-06T07:22:24.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.768+0000 7f2ece4b5640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ec81618b0 con 0x7f2ec80d4860 2024-03-06T07:22:24.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.768+0000 7f2ece4b5640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ec80d4cf0 con 0x7f2ec80b1e40 2024-03-06T07:22:24.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.768+0000 7f2ecca52640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2ec80b1e40 0x7f2ec8168b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.768+0000 7f2ecca52640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2ec80b1e40 0x7f2ec8168b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48726/0 (socket says 172.21.15.83:48726) 2024-03-06T07:22:24.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.768+0000 7f2ecca52640 1 -- 172.21.15.83:0/1743526308 learned_addr learned my addr 172.21.15.83:0/1743526308 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:24.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.768+0000 7f2ec7fff640 1 --1- 172.21.15.83:0/1743526308 >> v1:172.21.15.83:6789/0 conn(0x7f2ec81614d0 0x7f2ec80d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:55794/0 (socket says 172.21.15.83:55794) 2024-03-06T07:22:24.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.769+0000 7f2ecca52640 1 -- 172.21.15.83:0/1743526308 >> v1:172.21.15.183:6789/0 conn(0x7f2ec80d4860 legacy=0x7f2ec80d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.769+0000 7f2ecca52640 1 -- 172.21.15.83:0/1743526308 >> v1:172.21.15.83:6789/0 conn(0x7f2ec81614d0 legacy=0x7f2ec80d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.769+0000 7f2ecca52640 1 -- 172.21.15.83:0/1743526308 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ec80cf400 con 0x7f2ec80b1e40 2024-03-06T07:22:24.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.769+0000 7f2ec6ffd640 1 -- 172.21.15.83:0/1743526308 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3834393552 0 0) 0x7f2ec80d2730 con 0x7f2ec81614d0 2024-03-06T07:22:24.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.769+0000 7f2ecca52640 1 --2- 172.21.15.83:0/1743526308 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ec80b1e40 0x7f2ec8168b60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2ec80cf5d0 tx=0x7f2ec002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a678e59efc7574e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.770+0000 7f2ec6ffd640 1 -- 172.21.15.83:0/1743526308 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec002caf0 con 0x7f2ec80b1e40 2024-03-06T07:22:24.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.770+0000 7f2ec6ffd640 1 -- 172.21.15.83:0/1743526308 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ec002cc90 con 0x7f2ec80b1e40 2024-03-06T07:22:24.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.770+0000 7f2ec6ffd640 1 -- 172.21.15.83:0/1743526308 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec0034610 con 0x7f2ec80b1e40 2024-03-06T07:22:24.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.770+0000 7f2ece4b5640 1 -- 172.21.15.83:0/1743526308 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ec80b1e40 msgr2=0x7f2ec8168b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.770+0000 7f2ece4b5640 1 --2- 172.21.15.83:0/1743526308 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ec80b1e40 0x7f2ec8168b60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2ec80cf5d0 tx=0x7f2ec002bf50 comp rx=0 tx=0).stop 2024-03-06T07:22:24.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.770+0000 7f2ece4b5640 1 -- 172.21.15.83:0/1743526308 shutdown_connections 2024-03-06T07:22:24.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.770+0000 7f2ece4b5640 1 --2- 172.21.15.83:0/1743526308 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ec80b1e40 0x7f2ec8168b60 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.770+0000 7f2ece4b5640 1 -- 172.21.15.83:0/1743526308 >> 172.21.15.83:0/1743526308 conn(0x7f2ec80d3d40 msgr2=0x7f2ec80d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:24.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.770+0000 7f2ece4b5640 1 -- 172.21.15.83:0/1743526308 shutdown_connections 2024-03-06T07:22:24.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.770+0000 7f2ece4b5640 1 -- 172.21.15.83:0/1743526308 wait complete. 2024-03-06T07:22:24.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.772+0000 7f2ece4b5640 1 Processor -- start 2024-03-06T07:22:24.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.772+0000 7f2ece4b5640 1 -- start start 2024-03-06T07:22:24.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.773+0000 7f2ece4b5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ec80b1e40 0x7f2ec8150070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.773+0000 7f2ece4b5640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2ec80d4860 0x7f2ec80cbe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.773+0000 7f2ece4b5640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2ec81614d0 0x7f2ec80b0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.773+0000 7f2ece4b5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec80cfc60 con 0x7f2ec80b1e40 2024-03-06T07:22:24.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.773+0000 7f2ece4b5640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec80cf980 con 0x7f2ec80d4860 2024-03-06T07:22:24.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.773+0000 7f2ece4b5640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec80b0910 con 0x7f2ec81614d0 2024-03-06T07:22:24.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.773+0000 7f2ec7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ec80b1e40 0x7f2ec8150070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.773+0000 7f2ec7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ec80b1e40 0x7f2ec8150070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48740/0 (socket says 172.21.15.83:48740) 2024-03-06T07:22:24.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.773+0000 7f2ec7fff640 1 -- 172.21.15.83:0/859419088 learned_addr learned my addr 172.21.15.83:0/859419088 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:24.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.774+0000 7f2ec77fe640 1 --2- 172.21.15.83:0/859419088 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2ec80d4860 0x7f2ec80cbe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.774+0000 7f2ecca52640 1 --2- 172.21.15.83:0/859419088 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2ec81614d0 0x7f2ec80b0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.774+0000 7f2ec7fff640 1 -- 172.21.15.83:0/859419088 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2ec81614d0 msgr2=0x7f2ec80b0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.774+0000 7f2ec7fff640 1 --2- 172.21.15.83:0/859419088 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2ec81614d0 0x7f2ec80b0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.774+0000 7f2ec7fff640 1 -- 172.21.15.83:0/859419088 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2ec80d4860 msgr2=0x7f2ec80cbe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.774+0000 7f2ec7fff640 1 --2- 172.21.15.83:0/859419088 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2ec80d4860 0x7f2ec80cbe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.774+0000 7f2ec7fff640 1 -- 172.21.15.83:0/859419088 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ec80cb060 con 0x7f2ec80b1e40 2024-03-06T07:22:24.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.774+0000 7f2ec7fff640 1 --2- 172.21.15.83:0/859419088 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ec80b1e40 0x7f2ec8150070 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2eb800c4d0 tx=0x7f2eb800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.774+0000 7f2ec4ff9640 1 -- 172.21.15.83:0/859419088 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb8018070 con 0x7f2ec80b1e40 2024-03-06T07:22:24.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.774+0000 7f2ec4ff9640 1 -- 172.21.15.83:0/859419088 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eb80074e0 con 0x7f2ec80b1e40 2024-03-06T07:22:24.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.774+0000 7f2ece4b5640 1 -- 172.21.15.83:0/859419088 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ec80cb340 con 0x7f2ec80b1e40 2024-03-06T07:22:24.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.774+0000 7f2ec4ff9640 1 -- 172.21.15.83:0/859419088 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb8004a20 con 0x7f2ec80b1e40 2024-03-06T07:22:24.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.774+0000 7f2ece4b5640 1 -- 172.21.15.83:0/859419088 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ec80cb900 con 0x7f2ec80b1e40 2024-03-06T07:22:24.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.775+0000 7f2ece4b5640 1 -- 172.21.15.83:0/859419088 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ec80b28d0 con 0x7f2ec80b1e40 2024-03-06T07:22:24.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.776+0000 7f2ec4ff9640 1 -- 172.21.15.83:0/859419088 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2eb801b070 con 0x7f2ec80b1e40 2024-03-06T07:22:24.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.776+0000 7f2ec4ff9640 1 --2- 172.21.15.83:0/859419088 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2eb406b4d0 0x7f2eb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.776+0000 7f2ec4ff9640 1 -- 172.21.15.83:0/859419088 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2eb8089450 con 0x7f2ec80b1e40 2024-03-06T07:22:24.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.776+0000 7f2ec77fe640 1 --2- 172.21.15.83:0/859419088 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2eb406b4d0 0x7f2eb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.777+0000 7f2ec77fe640 1 --2- 172.21.15.83:0/859419088 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2eb406b4d0 0x7f2eb406d990 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2ec806d590 tx=0x7f2ebc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.781+0000 7f2ec4ff9640 1 -- 172.21.15.83:0/859419088 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2eb8057c80 con 0x7f2ec80b1e40 2024-03-06T07:22:24.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.957+0000 7f2ece4b5640 1 -- 172.21.15.83:0/859419088 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2ec8068ff0 con 0x7f2ec80b1e40 2024-03-06T07:22:24.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.957+0000 7f2ec4ff9640 1 -- 172.21.15.83:0/859419088 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2eb805bd20 con 0x7f2ec80b1e40 2024-03-06T07:22:24.959 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:22:24.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.959+0000 7f2ece4b5640 1 -- 172.21.15.83:0/859419088 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2eb406b4d0 msgr2=0x7f2eb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.959+0000 7f2ece4b5640 1 --2- 172.21.15.83:0/859419088 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2eb406b4d0 0x7f2eb406d990 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2ec806d590 tx=0x7f2ebc009040 comp rx=0 tx=0).stop 2024-03-06T07:22:24.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.959+0000 7f2ece4b5640 1 -- 172.21.15.83:0/859419088 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ec80b1e40 msgr2=0x7f2ec8150070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.959+0000 7f2ece4b5640 1 --2- 172.21.15.83:0/859419088 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ec80b1e40 0x7f2ec8150070 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2eb800c4d0 tx=0x7f2eb800c990 comp rx=0 tx=0).stop 2024-03-06T07:22:24.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.959+0000 7f2ece4b5640 1 -- 172.21.15.83:0/859419088 shutdown_connections 2024-03-06T07:22:24.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.959+0000 7f2ece4b5640 1 --2- 172.21.15.83:0/859419088 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2eb406b4d0 0x7f2eb406d990 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.959+0000 7f2ece4b5640 1 --2- 172.21.15.83:0/859419088 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2ec81614d0 0x7f2ec80b0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.959+0000 7f2ece4b5640 1 --2- 172.21.15.83:0/859419088 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2ec80d4860 0x7f2ec80cbe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.959+0000 7f2ece4b5640 1 --2- 172.21.15.83:0/859419088 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ec80b1e40 0x7f2ec8150070 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.959+0000 7f2ece4b5640 1 -- 172.21.15.83:0/859419088 >> 172.21.15.83:0/859419088 conn(0x7f2ec80d3d40 msgr2=0x7f2ec80d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:24.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.959+0000 7f2ece4b5640 1 -- 172.21.15.83:0/859419088 shutdown_connections 2024-03-06T07:22:24.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:24.959+0000 7f2ece4b5640 1 -- 172.21.15.83:0/859419088 wait complete. 2024-03-06T07:22:24.972 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:25.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.073+0000 7f9f1270f640 1 Processor -- start 2024-03-06T07:22:25.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.073+0000 7f9f1270f640 1 -- start start 2024-03-06T07:22:25.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.074+0000 7f9f1270f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9f0c15ad90 0x7f9f0c0b3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.074+0000 7f9f1270f640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f9f0c0b4110 0x7f9f0c15c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.074+0000 7f9f1270f640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f0c15b260 con 0x7f9f0c071a70 2024-03-06T07:22:25.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.074+0000 7f9f1270f640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f0c15c020 con 0x7f9f0c0b4110 2024-03-06T07:22:25.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.074+0000 7f9f1270f640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f0c15b540 con 0x7f9f0c15ad90 2024-03-06T07:22:25.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.075+0000 7f9f10cac640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f9f0c0b4110 0x7f9f0c15c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.075+0000 7f9f0b7fe640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9f0c15ad90 0x7f9f0c0b3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.075+0000 7f9f10cac640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f9f0c0b4110 0x7f9f0c15c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:41346/0 (socket says 172.21.15.83:41346) 2024-03-06T07:22:25.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.075+0000 7f9f0bfff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f9f0c071a70 0x7f9f0c071e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:36640/0 (socket says 172.21.15.83:36640) 2024-03-06T07:22:25.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.075+0000 7f9f10cac640 1 -- 172.21.15.83:0/192704873 learned_addr learned my addr 172.21.15.83:0/192704873 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:25.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.075+0000 7f9f0b7fe640 1 -- 172.21.15.83:0/192704873 >> v2:172.21.15.183:3300/0 conn(0x7f9f0c0b4110 msgr2=0x7f9f0c15c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:25.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.075+0000 7f9f0b7fe640 1 --2- 172.21.15.83:0/192704873 >> v2:172.21.15.183:3300/0 conn(0x7f9f0c0b4110 0x7f9f0c15c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.075+0000 7f9f0b7fe640 1 -- 172.21.15.83:0/192704873 >> v1:172.21.15.183:6789/0 conn(0x7f9f0c071a70 legacy=0x7f9f0c071e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.075+0000 7f9f0b7fe640 1 -- 172.21.15.83:0/192704873 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f0c15b820 con 0x7f9f0c15ad90 2024-03-06T07:22:25.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.075+0000 7f9f10cac640 1 --2- 172.21.15.83:0/192704873 >> v2:172.21.15.183:3300/0 conn(0x7f9f0c0b4110 0x7f9f0c15c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:25.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.076+0000 7f9f0b7fe640 1 --2- 172.21.15.83:0/192704873 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9f0c15ad90 0x7f9f0c0b3bd0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9f000097a0 tx=0x7f9f0002da50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10e4aee26798c7f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.076+0000 7f9f0affd640 1 -- 172.21.15.83:0/192704873 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f00031870 con 0x7f9f0c15ad90 2024-03-06T07:22:25.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.076+0000 7f9f0affd640 1 -- 172.21.15.83:0/192704873 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f00031a10 con 0x7f9f0c15ad90 2024-03-06T07:22:25.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.076+0000 7f9f0affd640 1 -- 172.21.15.83:0/192704873 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f00031de0 con 0x7f9f0c15ad90 2024-03-06T07:22:25.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.076+0000 7f9f1270f640 1 -- 172.21.15.83:0/192704873 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9f0c15ad90 msgr2=0x7f9f0c0b3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.076+0000 7f9f1270f640 1 --2- 172.21.15.83:0/192704873 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9f0c15ad90 0x7f9f0c0b3bd0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9f000097a0 tx=0x7f9f0002da50 comp rx=0 tx=0).stop 2024-03-06T07:22:25.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.076+0000 7f9f1270f640 1 -- 172.21.15.83:0/192704873 shutdown_connections 2024-03-06T07:22:25.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.076+0000 7f9f1270f640 1 --2- 172.21.15.83:0/192704873 >> v2:172.21.15.183:3300/0 conn(0x7f9f0c0b4110 0x7f9f0c15c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.076+0000 7f9f1270f640 1 --2- 172.21.15.83:0/192704873 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9f0c15ad90 0x7f9f0c0b3bd0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.076+0000 7f9f1270f640 1 -- 172.21.15.83:0/192704873 >> 172.21.15.83:0/192704873 conn(0x7f9f0c0b2660 msgr2=0x7f9f0c0b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:25.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.076+0000 7f9f1270f640 1 -- 172.21.15.83:0/192704873 shutdown_connections 2024-03-06T07:22:25.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.076+0000 7f9f1270f640 1 -- 172.21.15.83:0/192704873 wait complete. 2024-03-06T07:22:25.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.078+0000 7f9f1270f640 1 Processor -- start 2024-03-06T07:22:25.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.078+0000 7f9f1270f640 1 -- start start 2024-03-06T07:22:25.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.078+0000 7f9f1270f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f0c071a70 0x7f9f0c0afd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.078+0000 7f9f1270f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9f0c0b4110 0x7f9f0c1e4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.078+0000 7f9f1270f640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9f0c15ad90 0x7f9f0c1e5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.078+0000 7f9f1270f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f0c0b46e0 con 0x7f9f0c071a70 2024-03-06T07:22:25.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.078+0000 7f9f1270f640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f0c0b4560 con 0x7f9f0c15ad90 2024-03-06T07:22:25.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.078+0000 7f9f1270f640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f0c15c750 con 0x7f9f0c0b4110 2024-03-06T07:22:25.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.079+0000 7f9f0bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f0c071a70 0x7f9f0c0afd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.079+0000 7f9f0bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f0c071a70 0x7f9f0c0afd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48742/0 (socket says 172.21.15.83:48742) 2024-03-06T07:22:25.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.079+0000 7f9f0bfff640 1 -- 172.21.15.83:0/253587821 learned_addr learned my addr 172.21.15.83:0/253587821 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:25.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.079+0000 7f9f0b7fe640 1 --2- 172.21.15.83:0/253587821 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9f0c0b4110 0x7f9f0c1e4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.079+0000 7f9f10cac640 1 --2- 172.21.15.83:0/253587821 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9f0c15ad90 0x7f9f0c1e5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.079+0000 7f9f0bfff640 1 -- 172.21.15.83:0/253587821 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9f0c0b4110 msgr2=0x7f9f0c1e4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.079+0000 7f9f0bfff640 1 --2- 172.21.15.83:0/253587821 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9f0c0b4110 0x7f9f0c1e4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.079+0000 7f9f0bfff640 1 -- 172.21.15.83:0/253587821 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9f0c15ad90 msgr2=0x7f9f0c1e5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.079+0000 7f9f0bfff640 1 --2- 172.21.15.83:0/253587821 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9f0c15ad90 0x7f9f0c1e5130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.079+0000 7f9f0bfff640 1 -- 172.21.15.83:0/253587821 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f0c1424f0 con 0x7f9f0c071a70 2024-03-06T07:22:25.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.079+0000 7f9f10cac640 1 --2- 172.21.15.83:0/253587821 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9f0c15ad90 0x7f9f0c1e5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:25.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.079+0000 7f9f0bfff640 1 --2- 172.21.15.83:0/253587821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f0c071a70 0x7f9f0c0afd00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9efc00c430 tx=0x7f9efc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.079+0000 7f9f08ff9640 1 -- 172.21.15.83:0/253587821 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9efc0074e0 con 0x7f9f0c071a70 2024-03-06T07:22:25.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.079+0000 7f9f08ff9640 1 -- 172.21.15.83:0/253587821 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9efc007680 con 0x7f9f0c071a70 2024-03-06T07:22:25.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.079+0000 7f9f08ff9640 1 -- 172.21.15.83:0/253587821 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9efc004a00 con 0x7f9f0c071a70 2024-03-06T07:22:25.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.079+0000 7f9f1270f640 1 -- 172.21.15.83:0/253587821 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f0c1427d0 con 0x7f9f0c071a70 2024-03-06T07:22:25.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.079+0000 7f9f1270f640 1 -- 172.21.15.83:0/253587821 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f0c142d90 con 0x7f9f0c071a70 2024-03-06T07:22:25.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.080+0000 7f9f1270f640 1 -- 172.21.15.83:0/253587821 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f0c073390 con 0x7f9f0c071a70 2024-03-06T07:22:25.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.081+0000 7f9f08ff9640 1 -- 172.21.15.83:0/253587821 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9efc01a020 con 0x7f9f0c071a70 2024-03-06T07:22:25.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.082+0000 7f9f08ff9640 1 --2- 172.21.15.83:0/253587821 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9ef806b4d0 0x7f9ef806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.082+0000 7f9f0b7fe640 1 --2- 172.21.15.83:0/253587821 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9ef806b4d0 0x7f9ef806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.082+0000 7f9f08ff9640 1 -- 172.21.15.83:0/253587821 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9efc089960 con 0x7f9f0c071a70 2024-03-06T07:22:25.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.082+0000 7f9f0b7fe640 1 --2- 172.21.15.83:0/253587821 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9ef806b4d0 0x7f9ef806d990 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9f00038000 tx=0x7f9f00008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.086+0000 7f9f08ff9640 1 -- 172.21.15.83:0/253587821 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9efc0587e0 con 0x7f9f0c071a70 2024-03-06T07:22:25.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.258+0000 7f9f1270f640 1 -- 172.21.15.83:0/253587821 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9f0c0d07d0 con 0x7f9f0c071a70 2024-03-06T07:22:25.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.258+0000 7f9f08ff9640 1 -- 172.21.15.83:0/253587821 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9efc0580e0 con 0x7f9f0c071a70 2024-03-06T07:22:25.260 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-03-06T07:22:25.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.261+0000 7f9f1270f640 1 -- 172.21.15.83:0/253587821 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9ef806b4d0 msgr2=0x7f9ef806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.261+0000 7f9f1270f640 1 --2- 172.21.15.83:0/253587821 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9ef806b4d0 0x7f9ef806d990 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9f00038000 tx=0x7f9f00008030 comp rx=0 tx=0).stop 2024-03-06T07:22:25.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.261+0000 7f9f1270f640 1 -- 172.21.15.83:0/253587821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f0c071a70 msgr2=0x7f9f0c0afd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.261+0000 7f9f1270f640 1 --2- 172.21.15.83:0/253587821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f0c071a70 0x7f9f0c0afd00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9efc00c430 tx=0x7f9efc00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.261+0000 7f9f1270f640 1 -- 172.21.15.83:0/253587821 shutdown_connections 2024-03-06T07:22:25.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.261+0000 7f9f1270f640 1 --2- 172.21.15.83:0/253587821 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9ef806b4d0 0x7f9ef806d990 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.261+0000 7f9f1270f640 1 --2- 172.21.15.83:0/253587821 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9f0c15ad90 0x7f9f0c1e5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.261+0000 7f9f1270f640 1 --2- 172.21.15.83:0/253587821 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9f0c0b4110 0x7f9f0c1e4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.261+0000 7f9f1270f640 1 --2- 172.21.15.83:0/253587821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f0c071a70 0x7f9f0c0afd00 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.261+0000 7f9f1270f640 1 -- 172.21.15.83:0/253587821 >> 172.21.15.83:0/253587821 conn(0x7f9f0c0b2660 msgr2=0x7f9f0c06e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:25.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.261+0000 7f9f1270f640 1 -- 172.21.15.83:0/253587821 shutdown_connections 2024-03-06T07:22:25.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.261+0000 7f9f1270f640 1 -- 172.21.15.83:0/253587821 wait complete. 2024-03-06T07:22:25.273 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:25.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.375+0000 7f1249ec0640 1 Processor -- start 2024-03-06T07:22:25.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.375+0000 7f1249ec0640 1 -- start start 2024-03-06T07:22:25.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.376+0000 7f1249ec0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f12441455d0 0x7f1244072480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.376+0000 7f1249ec0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f124406afb0 0x7f124406b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.376+0000 7f1249ec0640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12441459b0 con 0x7f124406ded0 2024-03-06T07:22:25.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.376+0000 7f1249ec0640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12440cfdf0 con 0x7f124406afb0 2024-03-06T07:22:25.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.376+0000 7f1249ec0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f124406b930 con 0x7f12441455d0 2024-03-06T07:22:25.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.376+0000 7f123affd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f124406afb0 0x7f124406b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.376+0000 7f1243fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f124406ded0 0x7f124406e360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:55806/0 (socket says 172.21.15.83:55806) 2024-03-06T07:22:25.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.376+0000 7f1243fff640 1 -- 172.21.15.83:0/599208610 learned_addr learned my addr 172.21.15.83:0/599208610 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:25.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.377+0000 7f12437fe640 1 --2- 172.21.15.83:0/599208610 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f12441455d0 0x7f1244072480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.377+0000 7f123affd640 1 -- 172.21.15.83:0/599208610 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f12441455d0 msgr2=0x7f1244072480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:25.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.377+0000 7f123affd640 1 --2- 172.21.15.83:0/599208610 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f12441455d0 0x7f1244072480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.377+0000 7f123affd640 1 -- 172.21.15.83:0/599208610 >> v1:172.21.15.83:6789/0 conn(0x7f124406ded0 legacy=0x7f124406e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.377+0000 7f1242ffd640 1 -- 172.21.15.83:0/599208610 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 592442487 0 0) 0x7f12441459b0 con 0x7f124406ded0 2024-03-06T07:22:25.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.377+0000 7f123affd640 1 -- 172.21.15.83:0/599208610 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12440cf1e0 con 0x7f124406afb0 2024-03-06T07:22:25.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.377+0000 7f123affd640 1 --2- 172.21.15.83:0/599208610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f124406afb0 0x7f124406b3f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f122c0097a0 tx=0x7f122c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ca27fd88e6675c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.378+0000 7f1242ffd640 1 -- 172.21.15.83:0/599208610 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f122c035020 con 0x7f124406afb0 2024-03-06T07:22:25.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.378+0000 7f1242ffd640 1 -- 172.21.15.83:0/599208610 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f122c031830 con 0x7f124406afb0 2024-03-06T07:22:25.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.378+0000 7f1242ffd640 1 -- 172.21.15.83:0/599208610 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f122c031c00 con 0x7f124406afb0 2024-03-06T07:22:25.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.378+0000 7f1249ec0640 1 -- 172.21.15.83:0/599208610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f124406afb0 msgr2=0x7f124406b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.378+0000 7f1249ec0640 1 --2- 172.21.15.83:0/599208610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f124406afb0 0x7f124406b3f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f122c0097a0 tx=0x7f122c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.378+0000 7f1249ec0640 1 -- 172.21.15.83:0/599208610 shutdown_connections 2024-03-06T07:22:25.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.378+0000 7f1249ec0640 1 --2- 172.21.15.83:0/599208610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f124406afb0 0x7f124406b3f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.378+0000 7f1249ec0640 1 --2- 172.21.15.83:0/599208610 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f12441455d0 0x7f1244072480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.378+0000 7f1249ec0640 1 -- 172.21.15.83:0/599208610 >> 172.21.15.83:0/599208610 conn(0x7f12440b2660 msgr2=0x7f12440b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:25.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.378+0000 7f1249ec0640 1 -- 172.21.15.83:0/599208610 shutdown_connections 2024-03-06T07:22:25.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.378+0000 7f1249ec0640 1 -- 172.21.15.83:0/599208610 wait complete. 2024-03-06T07:22:25.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.379+0000 7f1249ec0640 1 Processor -- start 2024-03-06T07:22:25.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.380+0000 7f1249ec0640 1 -- start start 2024-03-06T07:22:25.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.380+0000 7f1249ec0640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f124406afb0 0x7f124412b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.380+0000 7f1249ec0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f124406ded0 0x7f12440a9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.380+0000 7f1249ec0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f12441455d0 0x7f12440a9990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.380+0000 7f1249ec0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12440cfa80 con 0x7f12441455d0 2024-03-06T07:22:25.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.380+0000 7f1249ec0640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12440cf7a0 con 0x7f124406afb0 2024-03-06T07:22:25.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.380+0000 7f1249ec0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12440729c0 con 0x7f124406ded0 2024-03-06T07:22:25.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.380+0000 7f1243fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f12441455d0 0x7f12440a9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.380+0000 7f1243fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f12441455d0 0x7f12440a9990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48772/0 (socket says 172.21.15.83:48772) 2024-03-06T07:22:25.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.380+0000 7f1243fff640 1 -- 172.21.15.83:0/569157884 learned_addr learned my addr 172.21.15.83:0/569157884 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:25.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.381+0000 7f12437fe640 1 --2- 172.21.15.83:0/569157884 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f124406afb0 0x7f124412b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.381+0000 7f123affd640 1 --2- 172.21.15.83:0/569157884 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f124406ded0 0x7f12440a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.381+0000 7f1243fff640 1 -- 172.21.15.83:0/569157884 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f124406ded0 msgr2=0x7f12440a9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.381+0000 7f1243fff640 1 --2- 172.21.15.83:0/569157884 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f124406ded0 0x7f12440a9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.381+0000 7f1243fff640 1 -- 172.21.15.83:0/569157884 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f124406afb0 msgr2=0x7f124412b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.381+0000 7f1243fff640 1 --2- 172.21.15.83:0/569157884 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f124406afb0 0x7f124412b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.381+0000 7f1243fff640 1 -- 172.21.15.83:0/569157884 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f124412be30 con 0x7f12441455d0 2024-03-06T07:22:25.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.381+0000 7f12437fe640 1 --2- 172.21.15.83:0/569157884 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f124406afb0 0x7f124412b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:25.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.381+0000 7f1243fff640 1 --2- 172.21.15.83:0/569157884 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f12441455d0 0x7f12440a9990 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f123c00c7d0 tx=0x7f123c00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.381+0000 7f1240ff9640 1 -- 172.21.15.83:0/569157884 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f123c018070 con 0x7f12441455d0 2024-03-06T07:22:25.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.381+0000 7f1240ff9640 1 -- 172.21.15.83:0/569157884 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f123c00a640 con 0x7f12441455d0 2024-03-06T07:22:25.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.381+0000 7f1249ec0640 1 -- 172.21.15.83:0/569157884 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f124412c0d0 con 0x7f12441455d0 2024-03-06T07:22:25.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.381+0000 7f1240ff9640 1 -- 172.21.15.83:0/569157884 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f123c00aa70 con 0x7f12441455d0 2024-03-06T07:22:25.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.381+0000 7f1249ec0640 1 -- 172.21.15.83:0/569157884 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f124412c690 con 0x7f12441455d0 2024-03-06T07:22:25.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.382+0000 7f1249ec0640 1 -- 172.21.15.83:0/569157884 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1244073390 con 0x7f12441455d0 2024-03-06T07:22:25.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.382+0000 7f1240ff9640 1 -- 172.21.15.83:0/569157884 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f123c00ac10 con 0x7f12441455d0 2024-03-06T07:22:25.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.383+0000 7f1240ff9640 1 --2- 172.21.15.83:0/569157884 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f122406b500 0x7f122406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.383+0000 7f1240ff9640 1 -- 172.21.15.83:0/569157884 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f123c089750 con 0x7f12441455d0 2024-03-06T07:22:25.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.383+0000 7f12437fe640 1 --2- 172.21.15.83:0/569157884 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f122406b500 0x7f122406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.383+0000 7f12437fe640 1 --2- 172.21.15.83:0/569157884 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f122406b500 0x7f122406d9c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1234006fd0 tx=0x7f1234008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.387+0000 7f1240ff9640 1 -- 172.21.15.83:0/569157884 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f123c057f00 con 0x7f12441455d0 2024-03-06T07:22:25.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.558+0000 7f1249ec0640 1 -- 172.21.15.83:0/569157884 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f124406bea0 con 0x7f12441455d0 2024-03-06T07:22:25.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.559+0000 7f1240ff9640 1 -- 172.21.15.83:0/569157884 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f123c05bfa0 con 0x7f12441455d0 2024-03-06T07:22:25.561 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 64 2024-03-06T07:22:25.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.560+0000 7f1249ec0640 1 -- 172.21.15.83:0/569157884 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f122406b500 msgr2=0x7f122406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.560+0000 7f1249ec0640 1 --2- 172.21.15.83:0/569157884 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f122406b500 0x7f122406d9c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1234006fd0 tx=0x7f1234008040 comp rx=0 tx=0).stop 2024-03-06T07:22:25.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.560+0000 7f1249ec0640 1 -- 172.21.15.83:0/569157884 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f12441455d0 msgr2=0x7f12440a9990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.560+0000 7f1249ec0640 1 --2- 172.21.15.83:0/569157884 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f12441455d0 0x7f12440a9990 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f123c00c7d0 tx=0x7f123c00cc90 comp rx=0 tx=0).stop 2024-03-06T07:22:25.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.561+0000 7f1249ec0640 1 -- 172.21.15.83:0/569157884 shutdown_connections 2024-03-06T07:22:25.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.561+0000 7f1249ec0640 1 --2- 172.21.15.83:0/569157884 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f122406b500 0x7f122406d9c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.561+0000 7f1249ec0640 1 --2- 172.21.15.83:0/569157884 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f12441455d0 0x7f12440a9990 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.561+0000 7f1249ec0640 1 --2- 172.21.15.83:0/569157884 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f124406ded0 0x7f12440a9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.561+0000 7f1249ec0640 1 --2- 172.21.15.83:0/569157884 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f124406afb0 0x7f124412b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.561+0000 7f1249ec0640 1 -- 172.21.15.83:0/569157884 >> 172.21.15.83:0/569157884 conn(0x7f12440b2660 msgr2=0x7f12440d09f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:25.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.561+0000 7f1249ec0640 1 -- 172.21.15.83:0/569157884 shutdown_connections 2024-03-06T07:22:25.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.561+0000 7f1249ec0640 1 -- 172.21.15.83:0/569157884 wait complete. 2024-03-06T07:22:25.574 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:25.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.673+0000 7f01551ac640 1 Processor -- start 2024-03-06T07:22:25.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.673+0000 7f01551ac640 1 -- start start 2024-03-06T07:22:25.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.674+0000 7f01551ac640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f015015dfe0 0x7f01500cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.674+0000 7f01551ac640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f01500cf830 0x7f01500d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.674+0000 7f01551ac640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01500b3430 con 0x7f01500d27f0 2024-03-06T07:22:25.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.674+0000 7f01551ac640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01500d2c50 con 0x7f015015dfe0 2024-03-06T07:22:25.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.674+0000 7f01551ac640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01500b31e0 con 0x7f01500cf830 2024-03-06T07:22:25.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.674+0000 7f014ed9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f015015dfe0 0x7f01500cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.674+0000 7f014ed9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f015015dfe0 0x7f01500cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48778/0 (socket says 172.21.15.83:48778) 2024-03-06T07:22:25.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.674+0000 7f014ed9d640 1 -- 172.21.15.83:0/30157878 learned_addr learned my addr 172.21.15.83:0/30157878 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:25.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.674+0000 7f014ed9d640 1 -- 172.21.15.83:0/30157878 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f01500cf830 msgr2=0x7f01500d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:25.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.674+0000 7f014ed9d640 1 --2- 172.21.15.83:0/30157878 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f01500cf830 0x7f01500d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.674+0000 7f014ed9d640 1 -- 172.21.15.83:0/30157878 >> v1:172.21.15.183:6789/0 conn(0x7f01500d27f0 legacy=0x7f01500d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:25.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.674+0000 7f014ed9d640 1 -- 172.21.15.83:0/30157878 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01500b3710 con 0x7f015015dfe0 2024-03-06T07:22:25.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.675+0000 7f014ed9d640 1 --2- 172.21.15.83:0/30157878 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f015015dfe0 0x7f01500cf2f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0140009870 tx=0x7f014002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2614fe53c036e8ff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.675+0000 7f014e59c640 1 -- 172.21.15.83:0/30157878 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0140035020 con 0x7f015015dfe0 2024-03-06T07:22:25.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.675+0000 7f014e59c640 1 -- 172.21.15.83:0/30157878 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0140031640 con 0x7f015015dfe0 2024-03-06T07:22:25.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.675+0000 7f014e59c640 1 -- 172.21.15.83:0/30157878 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014002d040 con 0x7f015015dfe0 2024-03-06T07:22:25.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.676+0000 7f01551ac640 1 -- 172.21.15.83:0/30157878 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f015015dfe0 msgr2=0x7f01500cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.676+0000 7f01551ac640 1 --2- 172.21.15.83:0/30157878 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f015015dfe0 0x7f01500cf2f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0140009870 tx=0x7f014002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.676+0000 7f01551ac640 1 -- 172.21.15.83:0/30157878 shutdown_connections 2024-03-06T07:22:25.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.676+0000 7f01551ac640 1 --2- 172.21.15.83:0/30157878 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f01500cf830 0x7f01500d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.676+0000 7f01551ac640 1 --2- 172.21.15.83:0/30157878 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f015015dfe0 0x7f01500cf2f0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.676+0000 7f01551ac640 1 -- 172.21.15.83:0/30157878 >> 172.21.15.83:0/30157878 conn(0x7f01500d3cd0 msgr2=0x7f01500d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:25.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.676+0000 7f01551ac640 1 -- 172.21.15.83:0/30157878 shutdown_connections 2024-03-06T07:22:25.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.676+0000 7f01551ac640 1 -- 172.21.15.83:0/30157878 wait complete. 2024-03-06T07:22:25.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.678+0000 7f01551ac640 1 Processor -- start 2024-03-06T07:22:25.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.678+0000 7f01551ac640 1 -- start start 2024-03-06T07:22:25.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.679+0000 7f01551ac640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f01500cf830 0x7f015015eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.679+0000 7f01551ac640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01500d27f0 0x7f01500cc370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.679+0000 7f01551ac640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f015015dfe0 0x7f0150154f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.679+0000 7f01551ac640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01500d1c70 con 0x7f01500d27f0 2024-03-06T07:22:25.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.679+0000 7f01551ac640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01500b3d60 con 0x7f015015dfe0 2024-03-06T07:22:25.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.679+0000 7f01551ac640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01500d1df0 con 0x7f01500cf830 2024-03-06T07:22:25.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.680+0000 7f014659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01500d27f0 0x7f01500cc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.680+0000 7f014659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01500d27f0 0x7f01500cc370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48786/0 (socket says 172.21.15.83:48786) 2024-03-06T07:22:25.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.680+0000 7f014659c640 1 -- 172.21.15.83:0/740628227 learned_addr learned my addr 172.21.15.83:0/740628227 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:25.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.680+0000 7f014ed9d640 1 --2- 172.21.15.83:0/740628227 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f01500cf830 0x7f015015eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.680+0000 7f014f59e640 1 --2- 172.21.15.83:0/740628227 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f015015dfe0 0x7f0150154f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.680+0000 7f014659c640 1 -- 172.21.15.83:0/740628227 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f01500cf830 msgr2=0x7f015015eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.680+0000 7f014659c640 1 --2- 172.21.15.83:0/740628227 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f01500cf830 0x7f015015eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.680+0000 7f014659c640 1 -- 172.21.15.83:0/740628227 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f015015dfe0 msgr2=0x7f0150154f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.680+0000 7f014659c640 1 --2- 172.21.15.83:0/740628227 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f015015dfe0 0x7f0150154f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.680+0000 7f014659c640 1 -- 172.21.15.83:0/740628227 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01500cca30 con 0x7f01500d27f0 2024-03-06T07:22:25.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.680+0000 7f014659c640 1 --2- 172.21.15.83:0/740628227 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01500d27f0 0x7f01500cc370 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f013800b450 tx=0x7f013800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.680+0000 7f0147fff640 1 -- 172.21.15.83:0/740628227 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0138015020 con 0x7f01500d27f0 2024-03-06T07:22:25.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.680+0000 7f0147fff640 1 -- 172.21.15.83:0/740628227 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0138011890 con 0x7f01500d27f0 2024-03-06T07:22:25.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.680+0000 7f01551ac640 1 -- 172.21.15.83:0/740628227 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01500ccd10 con 0x7f01500d27f0 2024-03-06T07:22:25.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.680+0000 7f0147fff640 1 -- 172.21.15.83:0/740628227 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0138011c80 con 0x7f01500d27f0 2024-03-06T07:22:25.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.680+0000 7f01551ac640 1 -- 172.21.15.83:0/740628227 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0150155490 con 0x7f01500d27f0 2024-03-06T07:22:25.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.681+0000 7f01551ac640 1 -- 172.21.15.83:0/740628227 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01500d3740 con 0x7f01500d27f0 2024-03-06T07:22:25.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.682+0000 7f0147fff640 1 -- 172.21.15.83:0/740628227 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0138018020 con 0x7f01500d27f0 2024-03-06T07:22:25.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.682+0000 7f0147fff640 1 --2- 172.21.15.83:0/740628227 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f013006b4d0 0x7f013006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.682+0000 7f014ed9d640 1 --2- 172.21.15.83:0/740628227 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f013006b4d0 0x7f013006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.682+0000 7f0147fff640 1 -- 172.21.15.83:0/740628227 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f013808a290 con 0x7f01500d27f0 2024-03-06T07:22:25.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.683+0000 7f014ed9d640 1 --2- 172.21.15.83:0/740628227 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f013006b4d0 0x7f013006d990 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0140036000 tx=0x7f014003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:25.686+0000 7f0147fff640 1 -- 172.21.15.83:0/740628227 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f01500d3740 con 0x7f01500d27f0 2024-03-06T07:22:26.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.001+0000 7f01551ac640 1 -- 172.21.15.83:0/740628227 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01500cebf0 con 0x7f01500d27f0 2024-03-06T07:22:26.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.002+0000 7f0147fff640 1 -- 172.21.15.83:0/740628227 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16617 (secure 0 0 0) 0x7f013801f020 con 0x7f01500d27f0 2024-03-06T07:22:26.004 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:22:26.004 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":20,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:21:59.938137+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.318942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.254795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:26.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.005+0000 7f01551ac640 1 -- 172.21.15.83:0/740628227 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f013006b4d0 msgr2=0x7f013006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.005+0000 7f01551ac640 1 --2- 172.21.15.83:0/740628227 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f013006b4d0 0x7f013006d990 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0140036000 tx=0x7f014003c040 comp rx=0 tx=0).stop 2024-03-06T07:22:26.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.005+0000 7f01551ac640 1 -- 172.21.15.83:0/740628227 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01500d27f0 msgr2=0x7f01500cc370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.005+0000 7f01551ac640 1 --2- 172.21.15.83:0/740628227 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01500d27f0 0x7f01500cc370 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f013800b450 tx=0x7f013800b910 comp rx=0 tx=0).stop 2024-03-06T07:22:26.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.005+0000 7f01551ac640 1 -- 172.21.15.83:0/740628227 shutdown_connections 2024-03-06T07:22:26.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.005+0000 7f01551ac640 1 --2- 172.21.15.83:0/740628227 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f013006b4d0 0x7f013006d990 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.005+0000 7f01551ac640 1 --2- 172.21.15.83:0/740628227 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f015015dfe0 0x7f0150154f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.005+0000 7f01551ac640 1 --2- 172.21.15.83:0/740628227 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01500d27f0 0x7f01500cc370 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.005+0000 7f01551ac640 1 --2- 172.21.15.83:0/740628227 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f01500cf830 0x7f015015eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.005+0000 7f01551ac640 1 -- 172.21.15.83:0/740628227 >> 172.21.15.83:0/740628227 conn(0x7f01500d3cd0 msgr2=0x7f01500cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:26.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.005+0000 7f01551ac640 1 -- 172.21.15.83:0/740628227 shutdown_connections 2024-03-06T07:22:26.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.005+0000 7f01551ac640 1 -- 172.21.15.83:0/740628227 wait complete. 2024-03-06T07:22:26.018 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:56.318942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:57.254795+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:26.018 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:26.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.124+0000 7fcfd773a640 1 Processor -- start 2024-03-06T07:22:26.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.124+0000 7fcfd773a640 1 -- start start 2024-03-06T07:22:26.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.125+0000 7fcfd773a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcfd0071a70 0x7fcfd0071e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.125+0000 7fcfd773a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcfd00b42e0 0x7fcfd015c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.125+0000 7fcfd773a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfd0072390 con 0x7fcfd015ad90 2024-03-06T07:22:26.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.125+0000 7fcfd773a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfd015c020 con 0x7fcfd00b42e0 2024-03-06T07:22:26.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.125+0000 7fcfd773a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd00b46c0 con 0x7fcfd0071a70 2024-03-06T07:22:26.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.125+0000 7fcfd5cd7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcfd00b42e0 0x7fcfd015c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.125+0000 7fcfd4cd5640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fcfd015ad90 0x7fcfd00b3bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:55818/0 (socket says 172.21.15.83:55818) 2024-03-06T07:22:26.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.125+0000 7fcfd4cd5640 1 -- 172.21.15.83:0/1464263735 learned_addr learned my addr 172.21.15.83:0/1464263735 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:26.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.125+0000 7fcfd54d6640 1 --2- 172.21.15.83:0/1464263735 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcfd0071a70 0x7fcfd0071e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.125+0000 7fcfd5cd7640 1 -- 172.21.15.83:0/1464263735 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcfd0071a70 msgr2=0x7fcfd0071e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:26.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.125+0000 7fcfd5cd7640 1 --2- 172.21.15.83:0/1464263735 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcfd0071a70 0x7fcfd0071e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.126+0000 7fcfd5cd7640 1 -- 172.21.15.83:0/1464263735 >> v1:172.21.15.83:6789/0 conn(0x7fcfd015ad90 legacy=0x7fcfd00b3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.126+0000 7fcfd5cd7640 1 -- 172.21.15.83:0/1464263735 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd015b260 con 0x7fcfd00b42e0 2024-03-06T07:22:26.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.126+0000 7fcfd54d6640 1 --2- 172.21.15.83:0/1464263735 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcfd0071a70 0x7fcfd0071e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:26.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.126+0000 7fcfd5cd7640 1 --2- 172.21.15.83:0/1464263735 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcfd00b42e0 0x7fcfd015c210 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8009d90 tx=0x7fcfc802e020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdbe8563f7581e70 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.126+0000 7fcfc7fff640 1 -- 172.21.15.83:0/1464263735 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc803b070 con 0x7fcfd00b42e0 2024-03-06T07:22:26.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.126+0000 7fcfc7fff640 1 -- 172.21.15.83:0/1464263735 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfc802edd0 con 0x7fcfd00b42e0 2024-03-06T07:22:26.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.126+0000 7fcfc7fff640 1 -- 172.21.15.83:0/1464263735 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc8036660 con 0x7fcfd00b42e0 2024-03-06T07:22:26.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.127+0000 7fcfd773a640 1 -- 172.21.15.83:0/1464263735 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcfd00b42e0 msgr2=0x7fcfd015c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.127+0000 7fcfd773a640 1 --2- 172.21.15.83:0/1464263735 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcfd00b42e0 0x7fcfd015c210 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8009d90 tx=0x7fcfc802e020 comp rx=0 tx=0).stop 2024-03-06T07:22:26.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.127+0000 7fcfd773a640 1 -- 172.21.15.83:0/1464263735 shutdown_connections 2024-03-06T07:22:26.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.127+0000 7fcfd773a640 1 --2- 172.21.15.83:0/1464263735 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcfd00b42e0 0x7fcfd015c210 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.127+0000 7fcfd773a640 1 --2- 172.21.15.83:0/1464263735 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcfd0071a70 0x7fcfd0071e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.127+0000 7fcfd773a640 1 -- 172.21.15.83:0/1464263735 >> 172.21.15.83:0/1464263735 conn(0x7fcfd00b2660 msgr2=0x7fcfd00b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:26.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.127+0000 7fcfd773a640 1 -- 172.21.15.83:0/1464263735 shutdown_connections 2024-03-06T07:22:26.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.127+0000 7fcfd773a640 1 -- 172.21.15.83:0/1464263735 wait complete. 2024-03-06T07:22:26.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.128+0000 7fcfd773a640 1 Processor -- start 2024-03-06T07:22:26.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.128+0000 7fcfd773a640 1 -- start start 2024-03-06T07:22:26.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.129+0000 7fcfd773a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcfd0071a70 0x7fcfd00c9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.129+0000 7fcfd773a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcfd00b42e0 0x7fcfd01f4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.129+0000 7fcfd773a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcfd015ad90 0x7fcfd0158050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.129+0000 7fcfd773a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd015bb00 con 0x7fcfd0071a70 2024-03-06T07:22:26.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.129+0000 7fcfd773a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd015b820 con 0x7fcfd015ad90 2024-03-06T07:22:26.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.129+0000 7fcfd773a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd015c750 con 0x7fcfd00b42e0 2024-03-06T07:22:26.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.129+0000 7fcfd54d6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcfd0071a70 0x7fcfd00c9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.129+0000 7fcfd54d6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcfd0071a70 0x7fcfd00c9ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48806/0 (socket says 172.21.15.83:48806) 2024-03-06T07:22:26.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.129+0000 7fcfd54d6640 1 -- 172.21.15.83:0/1518538365 learned_addr learned my addr 172.21.15.83:0/1518538365 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:26.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.129+0000 7fcfd54d6640 1 -- 172.21.15.83:0/1518538365 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcfd00b42e0 msgr2=0x7fcfd01f4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.129+0000 7fcfd4cd5640 1 --2- 172.21.15.83:0/1518538365 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcfd00b42e0 0x7fcfd01f4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.129+0000 7fcfd54d6640 1 --2- 172.21.15.83:0/1518538365 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcfd00b42e0 0x7fcfd01f4380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.129+0000 7fcfd54d6640 1 -- 172.21.15.83:0/1518538365 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcfd015ad90 msgr2=0x7fcfd0158050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.129+0000 7fcfd54d6640 1 --2- 172.21.15.83:0/1518538365 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcfd015ad90 0x7fcfd0158050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.129+0000 7fcfd54d6640 1 -- 172.21.15.83:0/1518538365 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd00ca3e0 con 0x7fcfd0071a70 2024-03-06T07:22:26.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.129+0000 7fcfd54d6640 1 --2- 172.21.15.83:0/1518538365 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcfd0071a70 0x7fcfd00c9ce0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fcfc000c7f0 tx=0x7fcfc000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.130+0000 7fcfc5ffb640 1 -- 172.21.15.83:0/1518538365 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc0015070 con 0x7fcfd0071a70 2024-03-06T07:22:26.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.130+0000 7fcfc5ffb640 1 -- 172.21.15.83:0/1518538365 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfc0007a30 con 0x7fcfd0071a70 2024-03-06T07:22:26.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.130+0000 7fcfd773a640 1 -- 172.21.15.83:0/1518538365 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfd00ca6c0 con 0x7fcfd0071a70 2024-03-06T07:22:26.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.130+0000 7fcfc5ffb640 1 -- 172.21.15.83:0/1518538365 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc0007e00 con 0x7fcfd0071a70 2024-03-06T07:22:26.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.130+0000 7fcfd773a640 1 -- 172.21.15.83:0/1518538365 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfd00cac80 con 0x7fcfd0071a70 2024-03-06T07:22:26.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.131+0000 7fcfd773a640 1 -- 172.21.15.83:0/1518538365 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfd0073390 con 0x7fcfd0071a70 2024-03-06T07:22:26.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.131+0000 7fcfc5ffb640 1 -- 172.21.15.83:0/1518538365 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcfc0018070 con 0x7fcfd0071a70 2024-03-06T07:22:26.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.131+0000 7fcfc5ffb640 1 --2- 172.21.15.83:0/1518538365 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcfb406b4d0 0x7fcfb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.132+0000 7fcfc5ffb640 1 -- 172.21.15.83:0/1518538365 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcfc008a1b0 con 0x7fcfd0071a70 2024-03-06T07:22:26.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.132+0000 7fcfd4cd5640 1 --2- 172.21.15.83:0/1518538365 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcfb406b4d0 0x7fcfb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.132+0000 7fcfd4cd5640 1 --2- 172.21.15.83:0/1518538365 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcfb406b4d0 0x7fcfb406d990 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fcfd01f14b0 tx=0x7fcfcc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.136+0000 7fcfc5ffb640 1 -- 172.21.15.83:0/1518538365 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcfc0059030 con 0x7fcfd0071a70 2024-03-06T07:22:26.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.306+0000 7fcfd773a640 1 -- 172.21.15.83:0/1518538365 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcfd00d07d0 con 0x7fcfd0071a70 2024-03-06T07:22:26.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.306+0000 7fcfc5ffb640 1 -- 172.21.15.83:0/1518538365 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcfc0058930 con 0x7fcfd0071a70 2024-03-06T07:22:26.308 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:22:26.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.308+0000 7fcfd773a640 1 -- 172.21.15.83:0/1518538365 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcfb406b4d0 msgr2=0x7fcfb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.308+0000 7fcfd773a640 1 --2- 172.21.15.83:0/1518538365 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcfb406b4d0 0x7fcfb406d990 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fcfd01f14b0 tx=0x7fcfcc009040 comp rx=0 tx=0).stop 2024-03-06T07:22:26.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.308+0000 7fcfd773a640 1 -- 172.21.15.83:0/1518538365 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcfd0071a70 msgr2=0x7fcfd00c9ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.308+0000 7fcfd773a640 1 --2- 172.21.15.83:0/1518538365 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcfd0071a70 0x7fcfd00c9ce0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fcfc000c7f0 tx=0x7fcfc000cbf0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.308+0000 7fcfd773a640 1 -- 172.21.15.83:0/1518538365 shutdown_connections 2024-03-06T07:22:26.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.308+0000 7fcfd773a640 1 --2- 172.21.15.83:0/1518538365 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcfb406b4d0 0x7fcfb406d990 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.308+0000 7fcfd773a640 1 --2- 172.21.15.83:0/1518538365 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcfd015ad90 0x7fcfd0158050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.308+0000 7fcfd773a640 1 --2- 172.21.15.83:0/1518538365 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcfd00b42e0 0x7fcfd01f4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.308+0000 7fcfd773a640 1 --2- 172.21.15.83:0/1518538365 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcfd0071a70 0x7fcfd00c9ce0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.308+0000 7fcfd773a640 1 -- 172.21.15.83:0/1518538365 >> 172.21.15.83:0/1518538365 conn(0x7fcfd00b2660 msgr2=0x7fcfd00b3660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:26.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.308+0000 7fcfd773a640 1 -- 172.21.15.83:0/1518538365 shutdown_connections 2024-03-06T07:22:26.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.308+0000 7fcfd773a640 1 -- 172.21.15.83:0/1518538365 wait complete. 2024-03-06T07:22:26.321 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:26.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.425+0000 7f9619078640 1 Processor -- start 2024-03-06T07:22:26.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.425+0000 7f9619078640 1 -- start start 2024-03-06T07:22:26.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.426+0000 7f9619078640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96140d10c0 0x7f96140d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.426+0000 7f9619078640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96140cfa50 con 0x7f96140d14a0 2024-03-06T07:22:26.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.426+0000 7f9619078640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96140d0a30 con 0x7f961415ecd0 2024-03-06T07:22:26.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.426+0000 7f9619078640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f961415c050 con 0x7f96140d10c0 2024-03-06T07:22:26.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.426+0000 7f960a59c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f96140d14a0 0x7f961415e5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:55834/0 (socket says 172.21.15.83:55834) 2024-03-06T07:22:26.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.426+0000 7f960a59c640 1 -- 172.21.15.83:0/1089951140 learned_addr learned my addr 172.21.15.83:0/1089951140 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:26.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.426+0000 7f9612d9d640 1 --2- 172.21.15.83:0/1089951140 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96140d10c0 0x7f96140d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.426+0000 7f961259c640 1 -- 172.21.15.83:0/1089951140 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2479571756 0 0) 0x7f96140cfa50 con 0x7f96140d14a0 2024-03-06T07:22:26.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.426+0000 7f961259c640 1 -- 172.21.15.83:0/1089951140 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9600003520 con 0x7f96140d14a0 2024-03-06T07:22:26.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.427+0000 7f961259c640 1 -- 172.21.15.83:0/1089951140 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 199565883 0 0) 0x7f9600003520 con 0x7f96140d14a0 2024-03-06T07:22:26.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.427+0000 7f961259c640 1 -- 172.21.15.83:0/1089951140 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96140d10c0 msgr2=0x7f96140d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:26.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.427+0000 7f961259c640 1 --2- 172.21.15.83:0/1089951140 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96140d10c0 0x7f96140d04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.427+0000 7f961259c640 1 -- 172.21.15.83:0/1089951140 >> v1:172.21.15.183:6789/0 conn(0x7f961415ecd0 legacy=0x7f96140cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.427+0000 7f961259c640 1 -- 172.21.15.83:0/1089951140 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96140b45c0 con 0x7f96140d14a0 2024-03-06T07:22:26.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.427+0000 7f961259c640 1 -- 172.21.15.83:0/1089951140 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f95fc001670 con 0x7f96140d14a0 2024-03-06T07:22:26.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.427+0000 7f961259c640 1 -- 172.21.15.83:0/1089951140 >> v1:172.21.15.83:6789/0 conn(0x7f96140d14a0 legacy=0x7f961415e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.427+0000 7f961259c640 1 --2- 172.21.15.83:0/1089951140 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96000039e0 0x7f9600003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.427+0000 7f961259c640 1 --2- 172.21.15.83:0/1089951140 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96140d10c0 0x7f9600004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.427+0000 7f961259c640 1 --2- 172.21.15.83:0/1089951140 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9600004d40 0x7f96000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.427+0000 7f961259c640 1 -- 172.21.15.83:0/1089951140 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f960000ba70 con 0x7f96000039e0 2024-03-06T07:22:26.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.427+0000 7f961259c640 1 -- 172.21.15.83:0/1089951140 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f960000bbf0 con 0x7f9600004d40 2024-03-06T07:22:26.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.427+0000 7f961259c640 1 -- 172.21.15.83:0/1089951140 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f960000bdf0 con 0x7f96140d10c0 2024-03-06T07:22:26.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.427+0000 7f9612d9d640 1 --2- 172.21.15.83:0/1089951140 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96000039e0 0x7f9600003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.427+0000 7f960a59c640 1 --2- 172.21.15.83:0/1089951140 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96140d10c0 0x7f9600004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.427+0000 7f961359e640 1 --2- 172.21.15.83:0/1089951140 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9600004d40 0x7f96000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.427+0000 7f9612d9d640 1 -- 172.21.15.83:0/1089951140 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96140d10c0 msgr2=0x7f9600004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:26.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.427+0000 7f9612d9d640 1 --2- 172.21.15.83:0/1089951140 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96140d10c0 0x7f9600004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.427+0000 7f9612d9d640 1 -- 172.21.15.83:0/1089951140 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9600004d40 msgr2=0x7f96000051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:26.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.427+0000 7f9612d9d640 1 --2- 172.21.15.83:0/1089951140 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9600004d40 0x7f96000051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.427+0000 7f9612d9d640 1 -- 172.21.15.83:0/1089951140 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9600003520 con 0x7f96000039e0 2024-03-06T07:22:26.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.427+0000 7f960a59c640 1 --2- 172.21.15.83:0/1089951140 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96140d10c0 0x7f9600004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:26.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.428+0000 7f9612d9d640 1 --2- 172.21.15.83:0/1089951140 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96000039e0 0x7f9600003e00 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9604009a90 tx=0x7f9600005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7de86d36554f39d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.428+0000 7f961259c640 1 -- 172.21.15.83:0/1089951140 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9604039070 con 0x7f96000039e0 2024-03-06T07:22:26.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.428+0000 7f961259c640 1 -- 172.21.15.83:0/1089951140 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f960402c680 con 0x7f96000039e0 2024-03-06T07:22:26.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.428+0000 7f961259c640 1 -- 172.21.15.83:0/1089951140 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9604009690 con 0x7f96000039e0 2024-03-06T07:22:26.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.429+0000 7f9619078640 1 -- 172.21.15.83:0/1089951140 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96000039e0 msgr2=0x7f9600003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.429+0000 7f9619078640 1 --2- 172.21.15.83:0/1089951140 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96000039e0 0x7f9600003e00 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9604009a90 tx=0x7f9600005830 comp rx=0 tx=0).stop 2024-03-06T07:22:26.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.429+0000 7f961359e640 1 -- 172.21.15.83:0/1089951140 reap_dead start 2024-03-06T07:22:26.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.429+0000 7f9619078640 1 -- 172.21.15.83:0/1089951140 shutdown_connections 2024-03-06T07:22:26.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.429+0000 7f9619078640 1 -- 172.21.15.83:0/1089951140 >> 172.21.15.83:0/1089951140 conn(0x7f96140b3800 msgr2=0x7f96140b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:26.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.429+0000 7f9619078640 1 -- 172.21.15.83:0/1089951140 shutdown_connections 2024-03-06T07:22:26.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.429+0000 7f9619078640 1 -- 172.21.15.83:0/1089951140 wait complete. 2024-03-06T07:22:26.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.431+0000 7f9619078640 1 Processor -- start 2024-03-06T07:22:26.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.431+0000 7f9619078640 1 -- start start 2024-03-06T07:22:26.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.432+0000 7f9619078640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96000039e0 0x7f9614159b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.432+0000 7f9619078640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f961414f510 0x7f96140b6b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.432+0000 7f9619078640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96140b7060 0x7f96140cae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.432+0000 7f9619078640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f961415f440 con 0x7f96140b7060 2024-03-06T07:22:26.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.432+0000 7f9619078640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96140d0060 con 0x7f96000039e0 2024-03-06T07:22:26.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.432+0000 7f9619078640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96140cfcf0 con 0x7f961414f510 2024-03-06T07:22:26.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.432+0000 7f961359e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96140b7060 0x7f96140cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.432+0000 7f961359e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96140b7060 0x7f96140cae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48826/0 (socket says 172.21.15.83:48826) 2024-03-06T07:22:26.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.432+0000 7f961359e640 1 -- 172.21.15.83:0/3303940440 learned_addr learned my addr 172.21.15.83:0/3303940440 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:26.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.432+0000 7f960a59c640 1 --2- 172.21.15.83:0/3303940440 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f961414f510 0x7f96140b6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.432+0000 7f9612d9d640 1 --2- 172.21.15.83:0/3303940440 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96000039e0 0x7f9614159b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.432+0000 7f961359e640 1 -- 172.21.15.83:0/3303940440 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f961414f510 msgr2=0x7f96140b6b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.432+0000 7f961359e640 1 --2- 172.21.15.83:0/3303940440 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f961414f510 0x7f96140b6b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.432+0000 7f961359e640 1 -- 172.21.15.83:0/3303940440 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96000039e0 msgr2=0x7f9614159b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.433+0000 7f961359e640 1 --2- 172.21.15.83:0/3303940440 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96000039e0 0x7f9614159b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.433+0000 7f961359e640 1 -- 172.21.15.83:0/3303940440 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96140b75e0 con 0x7f96140b7060 2024-03-06T07:22:26.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.433+0000 7f961359e640 1 --2- 172.21.15.83:0/3303940440 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96140b7060 0x7f96140cae80 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9614071930 tx=0x7f960c00eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.433+0000 7f960bfff640 1 -- 172.21.15.83:0/3303940440 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960c003970 con 0x7f96140b7060 2024-03-06T07:22:26.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.433+0000 7f960bfff640 1 -- 172.21.15.83:0/3303940440 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f960c018070 con 0x7f96140b7060 2024-03-06T07:22:26.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.433+0000 7f960bfff640 1 -- 172.21.15.83:0/3303940440 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960c003d50 con 0x7f96140b7060 2024-03-06T07:22:26.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.433+0000 7f9619078640 1 -- 172.21.15.83:0/3303940440 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9614159090 con 0x7f96140b7060 2024-03-06T07:22:26.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.433+0000 7f9619078640 1 -- 172.21.15.83:0/3303940440 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9614159650 con 0x7f96140b7060 2024-03-06T07:22:26.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.434+0000 7f9619078640 1 -- 172.21.15.83:0/3303940440 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9614159850 con 0x7f96140b7060 2024-03-06T07:22:26.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.434+0000 7f960bfff640 1 -- 172.21.15.83:0/3303940440 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f960c0050d0 con 0x7f96140b7060 2024-03-06T07:22:26.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.435+0000 7f960bfff640 1 --2- 172.21.15.83:0/3303940440 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f95f406b4d0 0x7f95f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.435+0000 7f9612d9d640 1 --2- 172.21.15.83:0/3303940440 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f95f406b4d0 0x7f95f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.435+0000 7f960bfff640 1 -- 172.21.15.83:0/3303940440 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f960c089e30 con 0x7f96140b7060 2024-03-06T07:22:26.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.436+0000 7f9612d9d640 1 --2- 172.21.15.83:0/3303940440 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f95f406b4d0 0x7f95f406d990 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9604036000 tx=0x7f9604030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.439+0000 7f960bfff640 1 -- 172.21.15.83:0/3303940440 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9614159850 con 0x7f96140b7060 2024-03-06T07:22:26.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.617+0000 7f9619078640 1 -- 172.21.15.83:0/3303940440 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f961409fe60 con 0x7f96140b7060 2024-03-06T07:22:26.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.617+0000 7f960bfff640 1 -- 172.21.15.83:0/3303940440 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f960c058cb0 con 0x7f96140b7060 2024-03-06T07:22:26.619 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-03-06T07:22:26.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.619+0000 7f9619078640 1 -- 172.21.15.83:0/3303940440 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f95f406b4d0 msgr2=0x7f95f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.619+0000 7f9619078640 1 --2- 172.21.15.83:0/3303940440 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f95f406b4d0 0x7f95f406d990 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9604036000 tx=0x7f9604030000 comp rx=0 tx=0).stop 2024-03-06T07:22:26.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.619+0000 7f9619078640 1 -- 172.21.15.83:0/3303940440 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96140b7060 msgr2=0x7f96140cae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.619+0000 7f9619078640 1 --2- 172.21.15.83:0/3303940440 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96140b7060 0x7f96140cae80 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9614071930 tx=0x7f960c00eb90 comp rx=0 tx=0).stop 2024-03-06T07:22:26.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.619+0000 7f9619078640 1 -- 172.21.15.83:0/3303940440 shutdown_connections 2024-03-06T07:22:26.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.619+0000 7f9619078640 1 --2- 172.21.15.83:0/3303940440 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f95f406b4d0 0x7f95f406d990 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.619+0000 7f9619078640 1 --2- 172.21.15.83:0/3303940440 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f96140b7060 0x7f96140cae80 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.619+0000 7f9619078640 1 --2- 172.21.15.83:0/3303940440 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f961414f510 0x7f96140b6b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.619+0000 7f9619078640 1 --2- 172.21.15.83:0/3303940440 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96000039e0 0x7f9614159b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.619+0000 7f9619078640 1 -- 172.21.15.83:0/3303940440 >> 172.21.15.83:0/3303940440 conn(0x7f96140b3800 msgr2=0x7f96140b0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:26.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.619+0000 7f9619078640 1 -- 172.21.15.83:0/3303940440 shutdown_connections 2024-03-06T07:22:26.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.619+0000 7f9619078640 1 -- 172.21.15.83:0/3303940440 wait complete. 2024-03-06T07:22:26.632 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:26.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.761+0000 7f9e77b55640 1 Processor -- start 2024-03-06T07:22:26.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.762+0000 7f9e77b55640 1 -- start start 2024-03-06T07:22:26.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.762+0000 7f9e77b55640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9e701614d0 0x7f9e700d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.762+0000 7f9e77b55640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9e700d2480 0x7f9e700b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.762+0000 7f9e77b55640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e701618b0 con 0x7f9e700d1910 2024-03-06T07:22:26.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.762+0000 7f9e77b55640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e700cf400 con 0x7f9e700d2480 2024-03-06T07:22:26.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.762+0000 7f9e77b55640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e700cf6a0 con 0x7f9e701614d0 2024-03-06T07:22:26.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.762+0000 7f9e760f2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9e700d2480 0x7f9e700b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.763+0000 7f9e760f2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9e700d2480 0x7f9e700b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48440/0 (socket says 172.21.15.83:48440) 2024-03-06T07:22:26.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.763+0000 7f9e760f2640 1 -- 172.21.15.83:0/872923751 learned_addr learned my addr 172.21.15.83:0/872923751 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:26.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.763+0000 7f9e758f1640 1 --2- 172.21.15.83:0/872923751 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9e701614d0 0x7f9e700d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.763+0000 7f9e760f2640 1 -- 172.21.15.83:0/872923751 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9e701614d0 msgr2=0x7f9e700d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:26.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.763+0000 7f9e760f2640 1 --2- 172.21.15.83:0/872923751 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9e701614d0 0x7f9e700d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.763+0000 7f9e760f2640 1 -- 172.21.15.83:0/872923751 >> v1:172.21.15.183:6789/0 conn(0x7f9e700d1910 legacy=0x7f9e700d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:26.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.763+0000 7f9e760f2640 1 -- 172.21.15.83:0/872923751 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e700cf980 con 0x7f9e700d2480 2024-03-06T07:22:26.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.763+0000 7f9e760f2640 1 --2- 172.21.15.83:0/872923751 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e700d2480 0x7f9e700b0e30 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9e68009e50 tx=0x7f9e6802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceed4dc73f6ac28 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.764+0000 7f9e748ef640 1 -- 172.21.15.83:0/872923751 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6802ca70 con 0x7f9e700d2480 2024-03-06T07:22:26.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.764+0000 7f9e748ef640 1 -- 172.21.15.83:0/872923751 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e6802cc10 con 0x7f9e700d2480 2024-03-06T07:22:26.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.764+0000 7f9e748ef640 1 -- 172.21.15.83:0/872923751 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e680343f0 con 0x7f9e700d2480 2024-03-06T07:22:26.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.764+0000 7f9e77b55640 1 -- 172.21.15.83:0/872923751 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e700d2480 msgr2=0x7f9e700b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.764+0000 7f9e77b55640 1 --2- 172.21.15.83:0/872923751 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e700d2480 0x7f9e700b0e30 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9e68009e50 tx=0x7f9e6802bed0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.764+0000 7f9e77b55640 1 -- 172.21.15.83:0/872923751 shutdown_connections 2024-03-06T07:22:26.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.764+0000 7f9e77b55640 1 --2- 172.21.15.83:0/872923751 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e700d2480 0x7f9e700b0e30 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.764+0000 7f9e77b55640 1 --2- 172.21.15.83:0/872923751 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9e701614d0 0x7f9e700d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.764+0000 7f9e77b55640 1 -- 172.21.15.83:0/872923751 >> 172.21.15.83:0/872923751 conn(0x7f9e700d3d40 msgr2=0x7f9e700d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:26.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.765+0000 7f9e77b55640 1 -- 172.21.15.83:0/872923751 shutdown_connections 2024-03-06T07:22:26.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.765+0000 7f9e77b55640 1 -- 172.21.15.83:0/872923751 wait complete. 2024-03-06T07:22:26.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.766+0000 7f9e77b55640 1 Processor -- start 2024-03-06T07:22:26.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.766+0000 7f9e77b55640 1 -- start start 2024-03-06T07:22:26.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.766+0000 7f9e77b55640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9e700d1910 0x7f9e70153260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.766+0000 7f9e77b55640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9e700d2480 0x7f9e700cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.766+0000 7f9e77b55640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e701614d0 0x7f9e700b8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.766+0000 7f9e77b55640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e700b1370 con 0x7f9e701614d0 2024-03-06T07:22:26.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.766+0000 7f9e77b55640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e700b0910 con 0x7f9e700d1910 2024-03-06T07:22:26.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.766+0000 7f9e77b55640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e700b14f0 con 0x7f9e700d2480 2024-03-06T07:22:26.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.766+0000 7f9e760f2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e701614d0 0x7f9e700b8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.766+0000 7f9e760f2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e701614d0 0x7f9e700b8670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48446/0 (socket says 172.21.15.83:48446) 2024-03-06T07:22:26.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.766+0000 7f9e760f2640 1 -- 172.21.15.83:0/2894862400 learned_addr learned my addr 172.21.15.83:0/2894862400 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:26.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.767+0000 7f9e750f0640 1 --2- 172.21.15.83:0/2894862400 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9e700d2480 0x7f9e700cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.767+0000 7f9e758f1640 1 --2- 172.21.15.83:0/2894862400 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9e700d1910 0x7f9e70153260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.767+0000 7f9e760f2640 1 -- 172.21.15.83:0/2894862400 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9e700d2480 msgr2=0x7f9e700cae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.767+0000 7f9e760f2640 1 --2- 172.21.15.83:0/2894862400 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9e700d2480 0x7f9e700cae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.767+0000 7f9e760f2640 1 -- 172.21.15.83:0/2894862400 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9e700d1910 msgr2=0x7f9e70153260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.767+0000 7f9e760f2640 1 --2- 172.21.15.83:0/2894862400 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9e700d1910 0x7f9e70153260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.767+0000 7f9e760f2640 1 -- 172.21.15.83:0/2894862400 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e700cb780 con 0x7f9e701614d0 2024-03-06T07:22:26.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.767+0000 7f9e750f0640 1 --2- 172.21.15.83:0/2894862400 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9e700d2480 0x7f9e700cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:26.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.767+0000 7f9e760f2640 1 --2- 172.21.15.83:0/2894862400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e701614d0 0x7f9e700b8670 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f9e68009e50 tx=0x7f9e68004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.767+0000 7f9e667fc640 1 -- 172.21.15.83:0/2894862400 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e68043020 con 0x7f9e701614d0 2024-03-06T07:22:26.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.767+0000 7f9e667fc640 1 -- 172.21.15.83:0/2894862400 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e6803e450 con 0x7f9e701614d0 2024-03-06T07:22:26.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.767+0000 7f9e77b55640 1 -- 172.21.15.83:0/2894862400 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e700cba60 con 0x7f9e701614d0 2024-03-06T07:22:26.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.767+0000 7f9e667fc640 1 -- 172.21.15.83:0/2894862400 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e68009510 con 0x7f9e701614d0 2024-03-06T07:22:26.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.767+0000 7f9e77b55640 1 -- 172.21.15.83:0/2894862400 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e700cc020 con 0x7f9e701614d0 2024-03-06T07:22:26.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.768+0000 7f9e77b55640 1 -- 172.21.15.83:0/2894862400 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e700d1e70 con 0x7f9e701614d0 2024-03-06T07:22:26.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.769+0000 7f9e667fc640 1 -- 172.21.15.83:0/2894862400 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9e68046020 con 0x7f9e701614d0 2024-03-06T07:22:26.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.769+0000 7f9e667fc640 1 --2- 172.21.15.83:0/2894862400 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9e5406b4d0 0x7f9e5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.769+0000 7f9e758f1640 1 --2- 172.21.15.83:0/2894862400 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9e5406b4d0 0x7f9e5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.769+0000 7f9e667fc640 1 -- 172.21.15.83:0/2894862400 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9e680acaa0 con 0x7f9e701614d0 2024-03-06T07:22:26.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.770+0000 7f9e758f1640 1 --2- 172.21.15.83:0/2894862400 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9e5406b4d0 0x7f9e5406d990 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9e60006fd0 tx=0x7f9e60008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.774+0000 7f9e667fc640 1 -- 172.21.15.83:0/2894862400 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9e6804d030 con 0x7f9e701614d0 2024-03-06T07:22:26.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.952+0000 7f9e77b55640 1 -- 172.21.15.83:0/2894862400 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9e7006a0d0 con 0x7f9e701614d0 2024-03-06T07:22:26.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.952+0000 7f9e667fc640 1 -- 172.21.15.83:0/2894862400 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9e6807b890 con 0x7f9e701614d0 2024-03-06T07:22:26.954 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 64 2024-03-06T07:22:26.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.954+0000 7f9e77b55640 1 -- 172.21.15.83:0/2894862400 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9e5406b4d0 msgr2=0x7f9e5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.954+0000 7f9e77b55640 1 --2- 172.21.15.83:0/2894862400 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9e5406b4d0 0x7f9e5406d990 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9e60006fd0 tx=0x7f9e60008040 comp rx=0 tx=0).stop 2024-03-06T07:22:26.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.954+0000 7f9e77b55640 1 -- 172.21.15.83:0/2894862400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e701614d0 msgr2=0x7f9e700b8670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.954+0000 7f9e77b55640 1 --2- 172.21.15.83:0/2894862400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e701614d0 0x7f9e700b8670 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f9e68009e50 tx=0x7f9e68004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:26.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.954+0000 7f9e77b55640 1 -- 172.21.15.83:0/2894862400 shutdown_connections 2024-03-06T07:22:26.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.954+0000 7f9e77b55640 1 --2- 172.21.15.83:0/2894862400 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9e5406b4d0 0x7f9e5406d990 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.954+0000 7f9e77b55640 1 --2- 172.21.15.83:0/2894862400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e701614d0 0x7f9e700b8670 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.954+0000 7f9e77b55640 1 --2- 172.21.15.83:0/2894862400 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9e700d2480 0x7f9e700cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.954+0000 7f9e77b55640 1 --2- 172.21.15.83:0/2894862400 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9e700d1910 0x7f9e70153260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.954+0000 7f9e77b55640 1 -- 172.21.15.83:0/2894862400 >> 172.21.15.83:0/2894862400 conn(0x7f9e700d3d40 msgr2=0x7f9e7006cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:26.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.954+0000 7f9e77b55640 1 -- 172.21.15.83:0/2894862400 shutdown_connections 2024-03-06T07:22:26.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:26.954+0000 7f9e77b55640 1 -- 172.21.15.83:0/2894862400 wait complete. 2024-03-06T07:22:26.971 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:27.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.073+0000 7fdd6fb63640 1 Processor -- start 2024-03-06T07:22:27.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.073+0000 7fdd6fb63640 1 -- start start 2024-03-06T07:22:27.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.075+0000 7fdd6fb63640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd680d4920 0x7fdd680cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.075+0000 7fdd6fb63640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd680ceac0 con 0x7fdd681621c0 2024-03-06T07:22:27.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.075+0000 7fdd6fb63640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd680d2dc0 con 0x7fdd681625a0 2024-03-06T07:22:27.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.075+0000 7fdd6fb63640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd680cf690 con 0x7fdd680d4920 2024-03-06T07:22:27.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.075+0000 7fdd6d8ff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fdd681621c0 0x7fdd680d26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39398/0 (socket says 172.21.15.83:39398) 2024-03-06T07:22:27.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.075+0000 7fdd6d8ff640 1 -- 172.21.15.83:0/544353546 learned_addr learned my addr 172.21.15.83:0/544353546 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:27.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.075+0000 7fdd6c8fd640 1 -- 172.21.15.83:0/544353546 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 972870476 0 0) 0x7fdd680ceac0 con 0x7fdd681621c0 2024-03-06T07:22:27.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.075+0000 7fdd6d0fe640 1 --2- 172.21.15.83:0/544353546 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd680d4920 0x7fdd680cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.075+0000 7fdd6c8fd640 1 -- 172.21.15.83:0/544353546 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd54003520 con 0x7fdd681621c0 2024-03-06T07:22:27.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.075+0000 7fdd6c8fd640 1 -- 172.21.15.83:0/544353546 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1895131113 0 0) 0x7fdd54003520 con 0x7fdd681621c0 2024-03-06T07:22:27.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.075+0000 7fdd6c8fd640 1 -- 172.21.15.83:0/544353546 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd680d4920 msgr2=0x7fdd680cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:27.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.075+0000 7fdd6c8fd640 1 --2- 172.21.15.83:0/544353546 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd680d4920 0x7fdd680cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.075+0000 7fdd6c8fd640 1 -- 172.21.15.83:0/544353546 >> v1:172.21.15.183:6789/0 conn(0x7fdd681625a0 legacy=0x7fdd6815dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.075+0000 7fdd6c8fd640 1 -- 172.21.15.83:0/544353546 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd680ced60 con 0x7fdd681621c0 2024-03-06T07:22:27.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.075+0000 7fdd6c8fd640 1 -- 172.21.15.83:0/544353546 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7fdd600019f0 con 0x7fdd681621c0 2024-03-06T07:22:27.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.075+0000 7fdd6c8fd640 1 -- 172.21.15.83:0/544353546 >> v1:172.21.15.83:6789/0 conn(0x7fdd681621c0 legacy=0x7fdd680d26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.075+0000 7fdd6c8fd640 1 --2- 172.21.15.83:0/544353546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd540039e0 0x7fdd54003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.076+0000 7fdd6c8fd640 1 --2- 172.21.15.83:0/544353546 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd54004390 0x7fdd54004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.076+0000 7fdd6c8fd640 1 --2- 172.21.15.83:0/544353546 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd680d4920 0x7fdd540050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.076+0000 7fdd6c8fd640 1 -- 172.21.15.83:0/544353546 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd680cf690 con 0x7fdd540039e0 2024-03-06T07:22:27.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.076+0000 7fdd6c8fd640 1 -- 172.21.15.83:0/544353546 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5400b9c0 con 0x7fdd54004390 2024-03-06T07:22:27.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.076+0000 7fdd6c8fd640 1 -- 172.21.15.83:0/544353546 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5400bbc0 con 0x7fdd680d4920 2024-03-06T07:22:27.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.076+0000 7fdd6d8ff640 1 --2- 172.21.15.83:0/544353546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd540039e0 0x7fdd54003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.076+0000 7fdd6d0fe640 1 --2- 172.21.15.83:0/544353546 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd54004390 0x7fdd54004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.076+0000 7fdd6e100640 1 --2- 172.21.15.83:0/544353546 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd680d4920 0x7fdd540050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.076+0000 7fdd6d8ff640 1 -- 172.21.15.83:0/544353546 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd680d4920 msgr2=0x7fdd540050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:27.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.076+0000 7fdd6d8ff640 1 --2- 172.21.15.83:0/544353546 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd680d4920 0x7fdd540050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.076+0000 7fdd6d8ff640 1 -- 172.21.15.83:0/544353546 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd54004390 msgr2=0x7fdd54004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:27.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.076+0000 7fdd6d8ff640 1 --2- 172.21.15.83:0/544353546 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd54004390 0x7fdd54004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.076+0000 7fdd6d8ff640 1 -- 172.21.15.83:0/544353546 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd54003520 con 0x7fdd540039e0 2024-03-06T07:22:27.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.077+0000 7fdd6d8ff640 1 --2- 172.21.15.83:0/544353546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd540039e0 0x7fdd54003dc0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fdd6000a920 tx=0x7fdd54005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85a5a096630f81fd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.077+0000 7fdd6c8fd640 1 -- 172.21.15.83:0/544353546 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd6000a4c0 con 0x7fdd540039e0 2024-03-06T07:22:27.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.077+0000 7fdd6c8fd640 1 -- 172.21.15.83:0/544353546 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd60034650 con 0x7fdd540039e0 2024-03-06T07:22:27.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.077+0000 7fdd6c8fd640 1 -- 172.21.15.83:0/544353546 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd60034a40 con 0x7fdd540039e0 2024-03-06T07:22:27.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.077+0000 7fdd6fb63640 1 -- 172.21.15.83:0/544353546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd540039e0 msgr2=0x7fdd54003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.077+0000 7fdd6fb63640 1 --2- 172.21.15.83:0/544353546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd540039e0 0x7fdd54003dc0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fdd6000a920 tx=0x7fdd54005830 comp rx=0 tx=0).stop 2024-03-06T07:22:27.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.077+0000 7fdd6e100640 1 -- 172.21.15.83:0/544353546 reap_dead start 2024-03-06T07:22:27.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.077+0000 7fdd6fb63640 1 -- 172.21.15.83:0/544353546 shutdown_connections 2024-03-06T07:22:27.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.077+0000 7fdd6fb63640 1 -- 172.21.15.83:0/544353546 >> 172.21.15.83:0/544353546 conn(0x7fdd680d3d10 msgr2=0x7fdd680d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:27.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.077+0000 7fdd6fb63640 1 -- 172.21.15.83:0/544353546 shutdown_connections 2024-03-06T07:22:27.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.077+0000 7fdd6fb63640 1 -- 172.21.15.83:0/544353546 wait complete. 2024-03-06T07:22:27.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.079+0000 7fdd6fb63640 1 Processor -- start 2024-03-06T07:22:27.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.079+0000 7fdd6fb63640 1 -- start start 2024-03-06T07:22:27.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.079+0000 7fdd6fb63640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd540039e0 0x7fdd68142040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.079+0000 7fdd6fb63640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd68155830 0x7fdd681f3e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.079+0000 7fdd6fb63640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd681f4370 0x7fdd681539a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.079+0000 7fdd6fb63640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd680cfb20 con 0x7fdd681f4370 2024-03-06T07:22:27.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.079+0000 7fdd6fb63640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd680cf320 con 0x7fdd68155830 2024-03-06T07:22:27.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.079+0000 7fdd6fb63640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd680cfca0 con 0x7fdd540039e0 2024-03-06T07:22:27.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.079+0000 7fdd6e100640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd681f4370 0x7fdd681539a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.079+0000 7fdd6e100640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd681f4370 0x7fdd681539a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48460/0 (socket says 172.21.15.83:48460) 2024-03-06T07:22:27.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.079+0000 7fdd6e100640 1 -- 172.21.15.83:0/4570268 learned_addr learned my addr 172.21.15.83:0/4570268 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:27.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.079+0000 7fdd6d8ff640 1 --2- 172.21.15.83:0/4570268 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd540039e0 0x7fdd68142040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.079+0000 7fdd6d0fe640 1 --2- 172.21.15.83:0/4570268 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd68155830 0x7fdd681f3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.080+0000 7fdd6e100640 1 -- 172.21.15.83:0/4570268 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd540039e0 msgr2=0x7fdd68142040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.080+0000 7fdd6e100640 1 --2- 172.21.15.83:0/4570268 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd540039e0 0x7fdd68142040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.080+0000 7fdd6e100640 1 -- 172.21.15.83:0/4570268 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd68155830 msgr2=0x7fdd681f3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.080+0000 7fdd6e100640 1 --2- 172.21.15.83:0/4570268 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd68155830 0x7fdd681f3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.080+0000 7fdd6e100640 1 -- 172.21.15.83:0/4570268 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd681f4820 con 0x7fdd681f4370 2024-03-06T07:22:27.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.080+0000 7fdd6d8ff640 1 --2- 172.21.15.83:0/4570268 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd540039e0 0x7fdd68142040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:27.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.080+0000 7fdd6e100640 1 --2- 172.21.15.83:0/4570268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd681f4370 0x7fdd681539a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fdd6808ded0 tx=0x7fdd6400fb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.080+0000 7fdd5e7fc640 1 -- 172.21.15.83:0/4570268 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd64019070 con 0x7fdd681f4370 2024-03-06T07:22:27.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.080+0000 7fdd5e7fc640 1 -- 172.21.15.83:0/4570268 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd64005030 con 0x7fdd681f4370 2024-03-06T07:22:27.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.080+0000 7fdd6fb63640 1 -- 172.21.15.83:0/4570268 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd68152be0 con 0x7fdd681f4370 2024-03-06T07:22:27.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.080+0000 7fdd5e7fc640 1 -- 172.21.15.83:0/4570268 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd64005400 con 0x7fdd681f4370 2024-03-06T07:22:27.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.080+0000 7fdd6fb63640 1 -- 172.21.15.83:0/4570268 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd681531a0 con 0x7fdd681f4370 2024-03-06T07:22:27.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.081+0000 7fdd6fb63640 1 -- 172.21.15.83:0/4570268 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd6804f330 con 0x7fdd681f4370 2024-03-06T07:22:27.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.081+0000 7fdd5e7fc640 1 -- 172.21.15.83:0/4570268 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdd6401c070 con 0x7fdd681f4370 2024-03-06T07:22:27.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.082+0000 7fdd5e7fc640 1 --2- 172.21.15.83:0/4570268 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdd4c06b4d0 0x7fdd4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.082+0000 7fdd5e7fc640 1 -- 172.21.15.83:0/4570268 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdd640891a0 con 0x7fdd681f4370 2024-03-06T07:22:27.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.082+0000 7fdd6d8ff640 1 --2- 172.21.15.83:0/4570268 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdd4c06b4d0 0x7fdd4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.082+0000 7fdd6d8ff640 1 --2- 172.21.15.83:0/4570268 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdd4c06b4d0 0x7fdd4c06d990 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fdd6000a920 tx=0x7fdd6002f000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.090+0000 7fdd5e7fc640 1 -- 172.21.15.83:0/4570268 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdd64058020 con 0x7fdd681f4370 2024-03-06T07:22:27.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.264+0000 7fdd6fb63640 1 -- 172.21.15.83:0/4570268 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd68073380 con 0x7fdd681f4370 2024-03-06T07:22:27.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.265+0000 7fdd5e7fc640 1 -- 172.21.15.83:0/4570268 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16617 (secure 0 0 0) 0x7fdd64057920 con 0x7fdd681f4370 2024-03-06T07:22:27.267 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:22:27.267 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":20,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:21:59.938137+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.318942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.254795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:27.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.267+0000 7fdd6fb63640 1 -- 172.21.15.83:0/4570268 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdd4c06b4d0 msgr2=0x7fdd4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.267+0000 7fdd6fb63640 1 --2- 172.21.15.83:0/4570268 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdd4c06b4d0 0x7fdd4c06d990 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fdd6000a920 tx=0x7fdd6002f000 comp rx=0 tx=0).stop 2024-03-06T07:22:27.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.267+0000 7fdd6fb63640 1 -- 172.21.15.83:0/4570268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd681f4370 msgr2=0x7fdd681539a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.267+0000 7fdd6fb63640 1 --2- 172.21.15.83:0/4570268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd681f4370 0x7fdd681539a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fdd6808ded0 tx=0x7fdd6400fb40 comp rx=0 tx=0).stop 2024-03-06T07:22:27.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.267+0000 7fdd6fb63640 1 -- 172.21.15.83:0/4570268 shutdown_connections 2024-03-06T07:22:27.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.267+0000 7fdd6fb63640 1 --2- 172.21.15.83:0/4570268 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdd4c06b4d0 0x7fdd4c06d990 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.267+0000 7fdd6fb63640 1 --2- 172.21.15.83:0/4570268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd681f4370 0x7fdd681539a0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.267+0000 7fdd6fb63640 1 --2- 172.21.15.83:0/4570268 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd68155830 0x7fdd681f3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.267+0000 7fdd6fb63640 1 --2- 172.21.15.83:0/4570268 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd540039e0 0x7fdd68142040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.267+0000 7fdd6fb63640 1 -- 172.21.15.83:0/4570268 >> 172.21.15.83:0/4570268 conn(0x7fdd680d3d10 msgr2=0x7fdd6806c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:27.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.267+0000 7fdd6fb63640 1 -- 172.21.15.83:0/4570268 shutdown_connections 2024-03-06T07:22:27.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.267+0000 7fdd6fb63640 1 -- 172.21.15.83:0/4570268 wait complete. 2024-03-06T07:22:27.280 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:56.318942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:57.254795+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:27.280 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:27.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.382+0000 7effe1d7a640 1 Processor -- start 2024-03-06T07:22:27.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.382+0000 7effe1d7a640 1 -- start start 2024-03-06T07:22:27.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.383+0000 7effe1d7a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7effdc1614d0 0x7effdc0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.383+0000 7effe1d7a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7effdc0d24b0 0x7effdc0b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.383+0000 7effe1d7a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effdc1618b0 con 0x7effdc0d1910 2024-03-06T07:22:27.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.383+0000 7effe1d7a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effdc0cf400 con 0x7effdc1614d0 2024-03-06T07:22:27.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.383+0000 7effe1d7a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effdc0cf6a0 con 0x7effdc0d24b0 2024-03-06T07:22:27.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.383+0000 7effdb7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7effdc1614d0 0x7effdc0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.383+0000 7effdb7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7effdc1614d0 0x7effdc0d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48476/0 (socket says 172.21.15.83:48476) 2024-03-06T07:22:27.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.383+0000 7effdb7fe640 1 -- 172.21.15.83:0/527843260 learned_addr learned my addr 172.21.15.83:0/527843260 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:27.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.383+0000 7effdbfff640 1 --2- 172.21.15.83:0/527843260 >> v2:172.21.15.183:3300/0 conn(0x7effdc0d24b0 0x7effdc0b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.383+0000 7effdb7fe640 1 -- 172.21.15.83:0/527843260 >> v2:172.21.15.183:3300/0 conn(0x7effdc0d24b0 msgr2=0x7effdc0b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:27.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.383+0000 7effdb7fe640 1 --2- 172.21.15.83:0/527843260 >> v2:172.21.15.183:3300/0 conn(0x7effdc0d24b0 0x7effdc0b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.384+0000 7effdb7fe640 1 -- 172.21.15.83:0/527843260 >> v1:172.21.15.183:6789/0 conn(0x7effdc0d1910 legacy=0x7effdc0d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.384+0000 7effdb7fe640 1 -- 172.21.15.83:0/527843260 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effdc0cf980 con 0x7effdc1614d0 2024-03-06T07:22:27.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.384+0000 7effdb7fe640 1 --2- 172.21.15.83:0/527843260 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effdc1614d0 0x7effdc0d4860 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7effdc0b2a70 tx=0x7effd002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c362e2fa94bf24d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.384+0000 7effda7fc640 1 -- 172.21.15.83:0/527843260 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd0034070 con 0x7effdc1614d0 2024-03-06T07:22:27.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.384+0000 7effda7fc640 1 -- 172.21.15.83:0/527843260 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effd0030860 con 0x7effdc1614d0 2024-03-06T07:22:27.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.384+0000 7effda7fc640 1 -- 172.21.15.83:0/527843260 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd0030c30 con 0x7effdc1614d0 2024-03-06T07:22:27.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.385+0000 7effe1d7a640 1 -- 172.21.15.83:0/527843260 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effdc1614d0 msgr2=0x7effdc0d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.385+0000 7effe1d7a640 1 --2- 172.21.15.83:0/527843260 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effdc1614d0 0x7effdc0d4860 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7effdc0b2a70 tx=0x7effd002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.385+0000 7effe1d7a640 1 -- 172.21.15.83:0/527843260 shutdown_connections 2024-03-06T07:22:27.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.385+0000 7effe1d7a640 1 --2- 172.21.15.83:0/527843260 >> v2:172.21.15.183:3300/0 conn(0x7effdc0d24b0 0x7effdc0b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.385+0000 7effe1d7a640 1 --2- 172.21.15.83:0/527843260 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effdc1614d0 0x7effdc0d4860 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.385+0000 7effe1d7a640 1 -- 172.21.15.83:0/527843260 >> 172.21.15.83:0/527843260 conn(0x7effdc0d3d40 msgr2=0x7effdc0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:27.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.385+0000 7effe1d7a640 1 -- 172.21.15.83:0/527843260 shutdown_connections 2024-03-06T07:22:27.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.385+0000 7effe1d7a640 1 -- 172.21.15.83:0/527843260 wait complete. 2024-03-06T07:22:27.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.387+0000 7effe1d7a640 1 Processor -- start 2024-03-06T07:22:27.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.387+0000 7effe1d7a640 1 -- start start 2024-03-06T07:22:27.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.388+0000 7effe1d7a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7effdc0d1910 0x7effdc154e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.388+0000 7effe1d7a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7effdc0d24b0 0x7effdc1f7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.388+0000 7effe1d7a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effdc1614d0 0x7effdc0ca570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.388+0000 7effe1d7a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effdc0b1370 con 0x7effdc1614d0 2024-03-06T07:22:27.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.388+0000 7effe1d7a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effdc0b0910 con 0x7effdc0d1910 2024-03-06T07:22:27.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.388+0000 7effe1d7a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effdc0b14f0 con 0x7effdc0d24b0 2024-03-06T07:22:27.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.388+0000 7effdbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effdc1614d0 0x7effdc0ca570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.388+0000 7effdbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effdc1614d0 0x7effdc0ca570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48478/0 (socket says 172.21.15.83:48478) 2024-03-06T07:22:27.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.388+0000 7effdbfff640 1 -- 172.21.15.83:0/2687660486 learned_addr learned my addr 172.21.15.83:0/2687660486 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:27.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.388+0000 7effdb7fe640 1 --2- 172.21.15.83:0/2687660486 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7effdc0d1910 0x7effdc154e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.388+0000 7effdaffd640 1 --2- 172.21.15.83:0/2687660486 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7effdc0d24b0 0x7effdc1f7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.389+0000 7effdbfff640 1 -- 172.21.15.83:0/2687660486 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7effdc0d24b0 msgr2=0x7effdc1f7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.389+0000 7effdbfff640 1 --2- 172.21.15.83:0/2687660486 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7effdc0d24b0 0x7effdc1f7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.389+0000 7effdbfff640 1 -- 172.21.15.83:0/2687660486 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7effdc0d1910 msgr2=0x7effdc154e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.389+0000 7effdbfff640 1 --2- 172.21.15.83:0/2687660486 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7effdc0d1910 0x7effdc154e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.389+0000 7effdbfff640 1 -- 172.21.15.83:0/2687660486 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effdc0cc970 con 0x7effdc1614d0 2024-03-06T07:22:27.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.389+0000 7effdb7fe640 1 --2- 172.21.15.83:0/2687660486 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7effdc0d1910 0x7effdc154e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:27.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.389+0000 7effdbfff640 1 --2- 172.21.15.83:0/2687660486 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effdc1614d0 0x7effdc0ca570 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7effd4007e70 tx=0x7effd400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.389+0000 7effc3fff640 1 -- 172.21.15.83:0/2687660486 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd4014620 con 0x7effdc1614d0 2024-03-06T07:22:27.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.389+0000 7effc3fff640 1 -- 172.21.15.83:0/2687660486 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effd4019070 con 0x7effdc1614d0 2024-03-06T07:22:27.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.389+0000 7effe1d7a640 1 -- 172.21.15.83:0/2687660486 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effdc0ccc50 con 0x7effdc1614d0 2024-03-06T07:22:27.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.389+0000 7effc3fff640 1 -- 172.21.15.83:0/2687660486 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd4014a20 con 0x7effdc1614d0 2024-03-06T07:22:27.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.389+0000 7effe1d7a640 1 -- 172.21.15.83:0/2687660486 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effdc155370 con 0x7effdc1614d0 2024-03-06T07:22:27.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.390+0000 7effe1d7a640 1 -- 172.21.15.83:0/2687660486 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effdc1f75f0 con 0x7effdc1614d0 2024-03-06T07:22:27.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.390+0000 7effc3fff640 1 -- 172.21.15.83:0/2687660486 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7effd4014bc0 con 0x7effdc1614d0 2024-03-06T07:22:27.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.391+0000 7effc3fff640 1 --2- 172.21.15.83:0/2687660486 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7effc806b4d0 0x7effc806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.391+0000 7effc3fff640 1 -- 172.21.15.83:0/2687660486 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7effd408aaf0 con 0x7effdc1614d0 2024-03-06T07:22:27.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.391+0000 7effdb7fe640 1 --2- 172.21.15.83:0/2687660486 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7effc806b4d0 0x7effc806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.391+0000 7effdb7fe640 1 --2- 172.21.15.83:0/2687660486 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7effc806b4d0 0x7effc806d990 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7effd0036000 tx=0x7effd003b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.395+0000 7effc3fff640 1 -- 172.21.15.83:0/2687660486 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7effd4059970 con 0x7effdc1614d0 2024-03-06T07:22:27.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.572+0000 7effe1d7a640 1 -- 172.21.15.83:0/2687660486 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7effdc15a8b0 con 0x7effdc1614d0 2024-03-06T07:22:27.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.573+0000 7effc3fff640 1 -- 172.21.15.83:0/2687660486 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7effd4059270 con 0x7effdc1614d0 2024-03-06T07:22:27.575 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:22:27.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.575+0000 7effe1d7a640 1 -- 172.21.15.83:0/2687660486 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7effc806b4d0 msgr2=0x7effc806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.575+0000 7effe1d7a640 1 --2- 172.21.15.83:0/2687660486 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7effc806b4d0 0x7effc806d990 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7effd0036000 tx=0x7effd003b040 comp rx=0 tx=0).stop 2024-03-06T07:22:27.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.575+0000 7effe1d7a640 1 -- 172.21.15.83:0/2687660486 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effdc1614d0 msgr2=0x7effdc0ca570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.575+0000 7effe1d7a640 1 --2- 172.21.15.83:0/2687660486 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effdc1614d0 0x7effdc0ca570 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7effd4007e70 tx=0x7effd400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.575+0000 7effe1d7a640 1 -- 172.21.15.83:0/2687660486 shutdown_connections 2024-03-06T07:22:27.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.575+0000 7effe1d7a640 1 --2- 172.21.15.83:0/2687660486 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7effc806b4d0 0x7effc806d990 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.575+0000 7effe1d7a640 1 --2- 172.21.15.83:0/2687660486 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effdc1614d0 0x7effdc0ca570 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.575+0000 7effe1d7a640 1 --2- 172.21.15.83:0/2687660486 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7effdc0d24b0 0x7effdc1f7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.575+0000 7effe1d7a640 1 --2- 172.21.15.83:0/2687660486 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7effdc0d1910 0x7effdc154e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.575+0000 7effe1d7a640 1 -- 172.21.15.83:0/2687660486 >> 172.21.15.83:0/2687660486 conn(0x7effdc0d3d40 msgr2=0x7effdc0b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:27.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.575+0000 7effe1d7a640 1 -- 172.21.15.83:0/2687660486 shutdown_connections 2024-03-06T07:22:27.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.575+0000 7effe1d7a640 1 -- 172.21.15.83:0/2687660486 wait complete. 2024-03-06T07:22:27.588 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:27.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.698+0000 7efccbf25640 1 Processor -- start 2024-03-06T07:22:27.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.698+0000 7efccbf25640 1 -- start start 2024-03-06T07:22:27.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.699+0000 7efccbf25640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efcc406afb0 0x7efcc406b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.699+0000 7efccbf25640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7efcc4163870 0x7efcc4165c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.699+0000 7efccbf25640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcc41459b0 con 0x7efcc41455d0 2024-03-06T07:22:27.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.699+0000 7efccbf25640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcc40cfdf0 con 0x7efcc4163870 2024-03-06T07:22:27.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.699+0000 7efccbf25640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcc406b930 con 0x7efcc406afb0 2024-03-06T07:22:27.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.699+0000 7efcca4c2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7efcc4163870 0x7efcc4165c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.699+0000 7efcca4c2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7efcc4163870 0x7efcc4165c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48484/0 (socket says 172.21.15.83:48484) 2024-03-06T07:22:27.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.699+0000 7efcca4c2640 1 -- 172.21.15.83:0/3344042416 learned_addr learned my addr 172.21.15.83:0/3344042416 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:27.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.699+0000 7efcc94c0640 1 --2- 172.21.15.83:0/3344042416 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efcc406afb0 0x7efcc406b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.700+0000 7efcca4c2640 1 -- 172.21.15.83:0/3344042416 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efcc406afb0 msgr2=0x7efcc406b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:27.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.700+0000 7efcca4c2640 1 --2- 172.21.15.83:0/3344042416 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efcc406afb0 0x7efcc406b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.700+0000 7efcca4c2640 1 -- 172.21.15.83:0/3344042416 >> v1:172.21.15.183:6789/0 conn(0x7efcc41455d0 legacy=0x7efcc4072480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.700+0000 7efcca4c2640 1 -- 172.21.15.83:0/3344042416 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc40cf1e0 con 0x7efcc4163870 2024-03-06T07:22:27.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.700+0000 7efcca4c2640 1 --2- 172.21.15.83:0/3344042416 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcc4163870 0x7efcc4165c60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7efcbc009e50 tx=0x7efcbc02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e472ecf0587b09d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.700+0000 7efcc8cbf640 1 -- 172.21.15.83:0/3344042416 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcbc02cb00 con 0x7efcc4163870 2024-03-06T07:22:27.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.701+0000 7efcc8cbf640 1 -- 172.21.15.83:0/3344042416 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcbc02cca0 con 0x7efcc4163870 2024-03-06T07:22:27.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.701+0000 7efcc8cbf640 1 -- 172.21.15.83:0/3344042416 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcbc034610 con 0x7efcc4163870 2024-03-06T07:22:27.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.701+0000 7efccbf25640 1 -- 172.21.15.83:0/3344042416 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcc4163870 msgr2=0x7efcc4165c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.701+0000 7efccbf25640 1 --2- 172.21.15.83:0/3344042416 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcc4163870 0x7efcc4165c60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7efcbc009e50 tx=0x7efcbc02bf60 comp rx=0 tx=0).stop 2024-03-06T07:22:27.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.701+0000 7efccbf25640 1 -- 172.21.15.83:0/3344042416 shutdown_connections 2024-03-06T07:22:27.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.701+0000 7efccbf25640 1 --2- 172.21.15.83:0/3344042416 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcc4163870 0x7efcc4165c60 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.701+0000 7efccbf25640 1 --2- 172.21.15.83:0/3344042416 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efcc406afb0 0x7efcc406b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.701+0000 7efccbf25640 1 -- 172.21.15.83:0/3344042416 >> 172.21.15.83:0/3344042416 conn(0x7efcc40b2660 msgr2=0x7efcc40b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:27.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.701+0000 7efccbf25640 1 -- 172.21.15.83:0/3344042416 shutdown_connections 2024-03-06T07:22:27.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.701+0000 7efccbf25640 1 -- 172.21.15.83:0/3344042416 wait complete. 2024-03-06T07:22:27.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.702+0000 7efccbf25640 1 Processor -- start 2024-03-06T07:22:27.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.702+0000 7efccbf25640 1 -- start start 2024-03-06T07:22:27.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.703+0000 7efccbf25640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efcc406afb0 0x7efcc41410b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.703+0000 7efccbf25640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcc41455d0 0x7efcc40ae490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.703+0000 7efccbf25640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efcc4163870 0x7efcc4144510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.703+0000 7efccbf25640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc40cfa80 con 0x7efcc41455d0 2024-03-06T07:22:27.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.703+0000 7efccbf25640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc40cf7a0 con 0x7efcc406afb0 2024-03-06T07:22:27.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.703+0000 7efccbf25640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcc40b3400 con 0x7efcc4163870 2024-03-06T07:22:27.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.703+0000 7efcc94c0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcc41455d0 0x7efcc40ae490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.703+0000 7efcc94c0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcc41455d0 0x7efcc40ae490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48490/0 (socket says 172.21.15.83:48490) 2024-03-06T07:22:27.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.703+0000 7efcc94c0640 1 -- 172.21.15.83:0/1855165594 learned_addr learned my addr 172.21.15.83:0/1855165594 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:27.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.703+0000 7efcc9cc1640 1 --2- 172.21.15.83:0/1855165594 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efcc406afb0 0x7efcc41410b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.704+0000 7efcca4c2640 1 --2- 172.21.15.83:0/1855165594 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efcc4163870 0x7efcc4144510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.704+0000 7efcc94c0640 1 -- 172.21.15.83:0/1855165594 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efcc4163870 msgr2=0x7efcc4144510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.704+0000 7efcc94c0640 1 --2- 172.21.15.83:0/1855165594 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efcc4163870 0x7efcc4144510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.704+0000 7efcc94c0640 1 -- 172.21.15.83:0/1855165594 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efcc406afb0 msgr2=0x7efcc41410b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.704+0000 7efcc94c0640 1 --2- 172.21.15.83:0/1855165594 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efcc406afb0 0x7efcc41410b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.704+0000 7efcc94c0640 1 -- 172.21.15.83:0/1855165594 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc415d650 con 0x7efcc41455d0 2024-03-06T07:22:27.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.704+0000 7efcc9cc1640 1 --2- 172.21.15.83:0/1855165594 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efcc406afb0 0x7efcc41410b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:27.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.704+0000 7efcc94c0640 1 --2- 172.21.15.83:0/1855165594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcc41455d0 0x7efcc40ae490 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7efcb400d800 tx=0x7efcb400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.704+0000 7efcba7fc640 1 -- 172.21.15.83:0/1855165594 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb400b6c0 con 0x7efcc41455d0 2024-03-06T07:22:27.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.704+0000 7efcba7fc640 1 -- 172.21.15.83:0/1855165594 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcb4019070 con 0x7efcc41455d0 2024-03-06T07:22:27.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.704+0000 7efcba7fc640 1 -- 172.21.15.83:0/1855165594 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb400bac0 con 0x7efcc41455d0 2024-03-06T07:22:27.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.704+0000 7efccbf25640 1 -- 172.21.15.83:0/1855165594 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcc415d930 con 0x7efcc41455d0 2024-03-06T07:22:27.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.704+0000 7efccbf25640 1 -- 172.21.15.83:0/1855165594 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcc415def0 con 0x7efcc41455d0 2024-03-06T07:22:27.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.705+0000 7efccbf25640 1 -- 172.21.15.83:0/1855165594 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcc4073390 con 0x7efcc41455d0 2024-03-06T07:22:27.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.705+0000 7efcba7fc640 1 -- 172.21.15.83:0/1855165594 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efcb400bc60 con 0x7efcc41455d0 2024-03-06T07:22:27.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.706+0000 7efcba7fc640 1 --2- 172.21.15.83:0/1855165594 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efca806b4d0 0x7efca806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.706+0000 7efcba7fc640 1 -- 172.21.15.83:0/1855165594 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efcb408a6c0 con 0x7efcc41455d0 2024-03-06T07:22:27.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.706+0000 7efcc9cc1640 1 --2- 172.21.15.83:0/1855165594 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efca806b4d0 0x7efca806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.706+0000 7efcc9cc1640 1 --2- 172.21.15.83:0/1855165594 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efca806b4d0 0x7efca806d990 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7efcc41442d0 tx=0x7efcc0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.714+0000 7efcba7fc640 1 -- 172.21.15.83:0/1855165594 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efcb4059540 con 0x7efcc41455d0 2024-03-06T07:22:27.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.884+0000 7efccbf25640 1 -- 172.21.15.83:0/1855165594 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efcc4072660 con 0x7efcc41455d0 2024-03-06T07:22:27.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.885+0000 7efcba7fc640 1 -- 172.21.15.83:0/1855165594 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7efcb4058e40 con 0x7efcc41455d0 2024-03-06T07:22:27.886 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-03-06T07:22:27.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.887+0000 7efccbf25640 1 -- 172.21.15.83:0/1855165594 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efca806b4d0 msgr2=0x7efca806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.887+0000 7efccbf25640 1 --2- 172.21.15.83:0/1855165594 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efca806b4d0 0x7efca806d990 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7efcc41442d0 tx=0x7efcc0009040 comp rx=0 tx=0).stop 2024-03-06T07:22:27.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.887+0000 7efccbf25640 1 -- 172.21.15.83:0/1855165594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcc41455d0 msgr2=0x7efcc40ae490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.887+0000 7efccbf25640 1 --2- 172.21.15.83:0/1855165594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcc41455d0 0x7efcc40ae490 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7efcb400d800 tx=0x7efcb400dcc0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.887+0000 7efccbf25640 1 -- 172.21.15.83:0/1855165594 shutdown_connections 2024-03-06T07:22:27.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.887+0000 7efccbf25640 1 --2- 172.21.15.83:0/1855165594 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efca806b4d0 0x7efca806d990 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.887+0000 7efccbf25640 1 --2- 172.21.15.83:0/1855165594 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efcc4163870 0x7efcc4144510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.887+0000 7efccbf25640 1 --2- 172.21.15.83:0/1855165594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcc41455d0 0x7efcc40ae490 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.887+0000 7efccbf25640 1 --2- 172.21.15.83:0/1855165594 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efcc406afb0 0x7efcc41410b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.887+0000 7efccbf25640 1 -- 172.21.15.83:0/1855165594 >> 172.21.15.83:0/1855165594 conn(0x7efcc40b2660 msgr2=0x7efcc40b1ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:27.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.887+0000 7efccbf25640 1 -- 172.21.15.83:0/1855165594 shutdown_connections 2024-03-06T07:22:27.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:27.887+0000 7efccbf25640 1 -- 172.21.15.83:0/1855165594 wait complete. 2024-03-06T07:22:27.900 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:28.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.010+0000 7f84da5f7640 1 Processor -- start 2024-03-06T07:22:28.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.010+0000 7f84da5f7640 1 -- start start 2024-03-06T07:22:28.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.011+0000 7f84da5f7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f84d415c050 0x7f84d406f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.011+0000 7f84da5f7640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f84d40d0af0 0x7f84d40cce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.011+0000 7f84da5f7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84d40cd4b0 con 0x7f84d406fb40 2024-03-06T07:22:28.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.011+0000 7f84da5f7640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84d406ffb0 con 0x7f84d415c050 2024-03-06T07:22:28.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.011+0000 7f84da5f7640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84d40cd790 con 0x7f84d40d0af0 2024-03-06T07:22:28.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.011+0000 7f84d3fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f84d415c050 0x7f84d406f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.011+0000 7f84d37fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f84d406fb40 0x7f84d40cc760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39404/0 (socket says 172.21.15.83:39404) 2024-03-06T07:22:28.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.011+0000 7f84d37fe640 1 -- 172.21.15.83:0/2301047041 learned_addr learned my addr 172.21.15.83:0/2301047041 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:28.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.011+0000 7f84d8b94640 1 --2- 172.21.15.83:0/2301047041 >> v2:172.21.15.183:3300/0 conn(0x7f84d40d0af0 0x7f84d40cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.011+0000 7f84d3fff640 1 -- 172.21.15.83:0/2301047041 >> v2:172.21.15.183:3300/0 conn(0x7f84d40d0af0 msgr2=0x7f84d40cce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:28.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.011+0000 7f84d3fff640 1 --2- 172.21.15.83:0/2301047041 >> v2:172.21.15.183:3300/0 conn(0x7f84d40d0af0 0x7f84d40cce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.011+0000 7f84d3fff640 1 -- 172.21.15.83:0/2301047041 >> v1:172.21.15.83:6789/0 conn(0x7f84d406fb40 legacy=0x7f84d40cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.011+0000 7f84d3fff640 1 -- 172.21.15.83:0/2301047041 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d40cda70 con 0x7f84d415c050 2024-03-06T07:22:28.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.011+0000 7f84d2ffd640 1 -- 172.21.15.83:0/2301047041 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4145476602 0 0) 0x7f84d40cd4b0 con 0x7f84d406fb40 2024-03-06T07:22:28.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.012+0000 7f84d3fff640 1 --2- 172.21.15.83:0/2301047041 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84d415c050 0x7f84d406f600 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f84d406b9c0 tx=0x7f84c402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3fe035e0329df95 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.012+0000 7f84d2ffd640 1 -- 172.21.15.83:0/2301047041 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c40315a0 con 0x7f84d415c050 2024-03-06T07:22:28.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.012+0000 7f84d2ffd640 1 -- 172.21.15.83:0/2301047041 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84c4037070 con 0x7f84d415c050 2024-03-06T07:22:28.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.012+0000 7f84d2ffd640 1 -- 172.21.15.83:0/2301047041 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c4031960 con 0x7f84d415c050 2024-03-06T07:22:28.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.013+0000 7f84da5f7640 1 -- 172.21.15.83:0/2301047041 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84d415c050 msgr2=0x7f84d406f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.013+0000 7f84da5f7640 1 --2- 172.21.15.83:0/2301047041 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84d415c050 0x7f84d406f600 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f84d406b9c0 tx=0x7f84c402b840 comp rx=0 tx=0).stop 2024-03-06T07:22:28.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.013+0000 7f84da5f7640 1 -- 172.21.15.83:0/2301047041 shutdown_connections 2024-03-06T07:22:28.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.013+0000 7f84da5f7640 1 --2- 172.21.15.83:0/2301047041 >> v2:172.21.15.183:3300/0 conn(0x7f84d40d0af0 0x7f84d40cce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.013+0000 7f84da5f7640 1 --2- 172.21.15.83:0/2301047041 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84d415c050 0x7f84d406f600 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.013+0000 7f84da5f7640 1 -- 172.21.15.83:0/2301047041 >> 172.21.15.83:0/2301047041 conn(0x7f84d40b3800 msgr2=0x7f84d40b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:28.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.013+0000 7f84da5f7640 1 -- 172.21.15.83:0/2301047041 shutdown_connections 2024-03-06T07:22:28.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.013+0000 7f84da5f7640 1 -- 172.21.15.83:0/2301047041 wait complete. 2024-03-06T07:22:28.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.014+0000 7f84da5f7640 1 Processor -- start 2024-03-06T07:22:28.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.014+0000 7f84da5f7640 1 -- start start 2024-03-06T07:22:28.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.015+0000 7f84da5f7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84d406fb40 0x7f84d4152970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.015+0000 7f84da5f7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84d40d0af0 0x7f84d415aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.015+0000 7f84da5f7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f84d415c050 0x7f84d4153130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.015+0000 7f84da5f7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d4165c30 con 0x7f84d406fb40 2024-03-06T07:22:28.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.015+0000 7f84da5f7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d4165ab0 con 0x7f84d40d0af0 2024-03-06T07:22:28.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.015+0000 7f84da5f7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84d4165db0 con 0x7f84d415c050 2024-03-06T07:22:28.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.015+0000 7f84d3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84d406fb40 0x7f84d4152970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.015+0000 7f84d3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84d406fb40 0x7f84d4152970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48508/0 (socket says 172.21.15.83:48508) 2024-03-06T07:22:28.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.015+0000 7f84d3fff640 1 -- 172.21.15.83:0/3294061397 learned_addr learned my addr 172.21.15.83:0/3294061397 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:28.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.015+0000 7f84d8b94640 1 --2- 172.21.15.83:0/3294061397 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f84d415c050 0x7f84d4153130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.015+0000 7f84d37fe640 1 --2- 172.21.15.83:0/3294061397 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84d40d0af0 0x7f84d415aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.015+0000 7f84d3fff640 1 -- 172.21.15.83:0/3294061397 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f84d415c050 msgr2=0x7f84d4153130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.015+0000 7f84d3fff640 1 --2- 172.21.15.83:0/3294061397 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f84d415c050 0x7f84d4153130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.015+0000 7f84d3fff640 1 -- 172.21.15.83:0/3294061397 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84d40d0af0 msgr2=0x7f84d415aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.015+0000 7f84d3fff640 1 --2- 172.21.15.83:0/3294061397 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84d40d0af0 0x7f84d415aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.015+0000 7f84d3fff640 1 -- 172.21.15.83:0/3294061397 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d41599a0 con 0x7f84d406fb40 2024-03-06T07:22:28.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.015+0000 7f84d8b94640 1 --2- 172.21.15.83:0/3294061397 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f84d415c050 0x7f84d4153130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:28.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.015+0000 7f84d37fe640 1 --2- 172.21.15.83:0/3294061397 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84d40d0af0 0x7f84d415aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:28.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.015+0000 7f84d3fff640 1 --2- 172.21.15.83:0/3294061397 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84d406fb40 0x7f84d4152970 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f84c4034040 tx=0x7f84c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.015+0000 7f84d0ff9640 1 -- 172.21.15.83:0/3294061397 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c4032630 con 0x7f84d406fb40 2024-03-06T07:22:28.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.015+0000 7f84d0ff9640 1 -- 172.21.15.83:0/3294061397 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84c40327d0 con 0x7f84d406fb40 2024-03-06T07:22:28.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.016+0000 7f84da5f7640 1 -- 172.21.15.83:0/3294061397 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84d4159c80 con 0x7f84d406fb40 2024-03-06T07:22:28.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.016+0000 7f84d0ff9640 1 -- 172.21.15.83:0/3294061397 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c4032bc0 con 0x7f84d406fb40 2024-03-06T07:22:28.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.016+0000 7f84da5f7640 1 -- 172.21.15.83:0/3294061397 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84d415a240 con 0x7f84d406fb40 2024-03-06T07:22:28.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.017+0000 7f84da5f7640 1 -- 172.21.15.83:0/3294061397 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84d409fe60 con 0x7f84d406fb40 2024-03-06T07:22:28.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.017+0000 7f84d0ff9640 1 -- 172.21.15.83:0/3294061397 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f84c403b3f0 con 0x7f84d406fb40 2024-03-06T07:22:28.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.017+0000 7f84d0ff9640 1 --2- 172.21.15.83:0/3294061397 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f84c006b4d0 0x7f84c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.018+0000 7f84d0ff9640 1 -- 172.21.15.83:0/3294061397 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f84c402f030 con 0x7f84d406fb40 2024-03-06T07:22:28.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.018+0000 7f84d37fe640 1 --2- 172.21.15.83:0/3294061397 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f84c006b4d0 0x7f84c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.018+0000 7f84d37fe640 1 --2- 172.21.15.83:0/3294061397 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f84c006b4d0 0x7f84c006d990 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f84d4153f80 tx=0x7f84c8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.022+0000 7f84d0ff9640 1 -- 172.21.15.83:0/3294061397 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f84c4040070 con 0x7f84d406fb40 2024-03-06T07:22:28.191 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:22:28.194 DEBUG:teuthology.orchestra.run.smithi183:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:22:28.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.209+0000 7f84da5f7640 1 -- 172.21.15.83:0/3294061397 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f84d4068ff0 con 0x7f84d406fb40 2024-03-06T07:22:28.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.210+0000 7f84d0ff9640 1 -- 172.21.15.83:0/3294061397 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f84c407af60 con 0x7f84d406fb40 2024-03-06T07:22:28.211 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 64 2024-03-06T07:22:28.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.211+0000 7f84da5f7640 1 -- 172.21.15.83:0/3294061397 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f84c006b4d0 msgr2=0x7f84c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.211+0000 7f84da5f7640 1 --2- 172.21.15.83:0/3294061397 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f84c006b4d0 0x7f84c006d990 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f84d4153f80 tx=0x7f84c8009040 comp rx=0 tx=0).stop 2024-03-06T07:22:28.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.211+0000 7f84da5f7640 1 -- 172.21.15.83:0/3294061397 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84d406fb40 msgr2=0x7f84d4152970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.211+0000 7f84da5f7640 1 --2- 172.21.15.83:0/3294061397 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84d406fb40 0x7f84d4152970 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f84c4034040 tx=0x7f84c40046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.212+0000 7f84da5f7640 1 -- 172.21.15.83:0/3294061397 shutdown_connections 2024-03-06T07:22:28.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.212+0000 7f84da5f7640 1 --2- 172.21.15.83:0/3294061397 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f84c006b4d0 0x7f84c006d990 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.212+0000 7f84da5f7640 1 --2- 172.21.15.83:0/3294061397 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f84d415c050 0x7f84d4153130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.212+0000 7f84da5f7640 1 --2- 172.21.15.83:0/3294061397 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84d40d0af0 0x7f84d415aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.212+0000 7f84da5f7640 1 --2- 172.21.15.83:0/3294061397 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84d406fb40 0x7f84d4152970 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.212+0000 7f84da5f7640 1 -- 172.21.15.83:0/3294061397 >> 172.21.15.83:0/3294061397 conn(0x7f84d40b3800 msgr2=0x7f84d40cc720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:28.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.212+0000 7f84da5f7640 1 -- 172.21.15.83:0/3294061397 shutdown_connections 2024-03-06T07:22:28.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.212+0000 7f84da5f7640 1 -- 172.21.15.83:0/3294061397 wait complete. 2024-03-06T07:22:28.225 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:28.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.400+0000 7f1c9eff6640 1 Processor -- start 2024-03-06T07:22:28.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.400+0000 7f1c9eff6640 1 -- start start 2024-03-06T07:22:28.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.401+0000 7f1c9eff6640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1c9815bf30 0x7f1c9815ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.401+0000 7f1c9eff6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c9815f580 con 0x7f1c980b3060 2024-03-06T07:22:28.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.401+0000 7f1c9eff6640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c9815b450 con 0x7f1c9813dc10 2024-03-06T07:22:28.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.401+0000 7f1c9eff6640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c980cf3f0 con 0x7f1c9815bf30 2024-03-06T07:22:28.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.401+0000 7f1c9d593640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1c9815bf30 0x7f1c9815ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.401+0000 7f1c9d593640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1c9815bf30 0x7f1c9815ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48510/0 (socket says 172.21.15.83:48510) 2024-03-06T07:22:28.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.401+0000 7f1c9d593640 1 -- 172.21.15.83:0/448686640 learned_addr learned my addr 172.21.15.83:0/448686640 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:28.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.401+0000 7f1c97fff640 1 --1- 172.21.15.83:0/448686640 >> v1:172.21.15.83:6789/0 conn(0x7f1c980b3060 0x7f1c9815e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39408/0 (socket says 172.21.15.83:39408) 2024-03-06T07:22:28.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.401+0000 7f1c9d593640 1 -- 172.21.15.83:0/448686640 >> v1:172.21.15.183:6789/0 conn(0x7f1c9813dc10 legacy=0x7f1c980b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:28.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.401+0000 7f1c9d593640 1 -- 172.21.15.83:0/448686640 >> v1:172.21.15.83:6789/0 conn(0x7f1c980b3060 legacy=0x7f1c9815e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.401+0000 7f1c977fe640 1 -- 172.21.15.83:0/448686640 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1394089890 0 0) 0x7f1c9815f580 con 0x7f1c980b3060 2024-03-06T07:22:28.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.401+0000 7f1c9d593640 1 -- 172.21.15.83:0/448686640 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c9815b730 con 0x7f1c9815bf30 2024-03-06T07:22:28.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.402+0000 7f1c9d593640 1 --2- 172.21.15.83:0/448686640 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c9815bf30 0x7f1c9815ef60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1c90009e50 tx=0x7f1c9002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a4cb8661c8b86d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.402+0000 7f1c977fe640 1 -- 172.21.15.83:0/448686640 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9002ca80 con 0x7f1c9815bf30 2024-03-06T07:22:28.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.402+0000 7f1c977fe640 1 -- 172.21.15.83:0/448686640 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c9002cc20 con 0x7f1c9815bf30 2024-03-06T07:22:28.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.402+0000 7f1c977fe640 1 -- 172.21.15.83:0/448686640 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c900343f0 con 0x7f1c9815bf30 2024-03-06T07:22:28.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.403+0000 7f1c9eff6640 1 -- 172.21.15.83:0/448686640 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c9815bf30 msgr2=0x7f1c9815ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.403+0000 7f1c9eff6640 1 --2- 172.21.15.83:0/448686640 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c9815bf30 0x7f1c9815ef60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1c90009e50 tx=0x7f1c9002bee0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.403+0000 7f1c9eff6640 1 -- 172.21.15.83:0/448686640 shutdown_connections 2024-03-06T07:22:28.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.403+0000 7f1c9eff6640 1 --2- 172.21.15.83:0/448686640 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c9815bf30 0x7f1c9815ef60 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.403+0000 7f1c9eff6640 1 -- 172.21.15.83:0/448686640 >> 172.21.15.83:0/448686640 conn(0x7f1c980cf8a0 msgr2=0x7f1c980cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:28.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.403+0000 7f1c9eff6640 1 -- 172.21.15.83:0/448686640 shutdown_connections 2024-03-06T07:22:28.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.403+0000 7f1c9eff6640 1 -- 172.21.15.83:0/448686640 wait complete. 2024-03-06T07:22:28.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.405+0000 7f1c9eff6640 1 Processor -- start 2024-03-06T07:22:28.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.405+0000 7f1c9eff6640 1 -- start start 2024-03-06T07:22:28.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.406+0000 7f1c9eff6640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1c980b3060 0x7f1c9815ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.406+0000 7f1c9eff6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c9813dc10 0x7f1c98151290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.406+0000 7f1c9eff6640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1c9815bf30 0x7f1c98140220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.406+0000 7f1c9eff6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c980718e0 con 0x7f1c9813dc10 2024-03-06T07:22:28.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.406+0000 7f1c9eff6640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9815b210 con 0x7f1c980b3060 2024-03-06T07:22:28.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.406+0000 7f1c9eff6640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c98071a60 con 0x7f1c9815bf30 2024-03-06T07:22:28.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.406+0000 7f1c97fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c9813dc10 0x7f1c98151290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.407+0000 7f1c97fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c9813dc10 0x7f1c98151290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48522/0 (socket says 172.21.15.83:48522) 2024-03-06T07:22:28.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.407+0000 7f1c97fff640 1 -- 172.21.15.83:0/265818245 learned_addr learned my addr 172.21.15.83:0/265818245 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:28.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.407+0000 7f1c9cd92640 1 --2- 172.21.15.83:0/265818245 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1c980b3060 0x7f1c9815ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.407+0000 7f1c9d593640 1 --2- 172.21.15.83:0/265818245 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1c9815bf30 0x7f1c98140220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.407+0000 7f1c97fff640 1 -- 172.21.15.83:0/265818245 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1c9815bf30 msgr2=0x7f1c98140220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.407+0000 7f1c97fff640 1 --2- 172.21.15.83:0/265818245 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1c9815bf30 0x7f1c98140220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.407+0000 7f1c97fff640 1 -- 172.21.15.83:0/265818245 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1c980b3060 msgr2=0x7f1c9815ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.407+0000 7f1c97fff640 1 --2- 172.21.15.83:0/265818245 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1c980b3060 0x7f1c9815ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.407+0000 7f1c97fff640 1 -- 172.21.15.83:0/265818245 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c9813d480 con 0x7f1c9813dc10 2024-03-06T07:22:28.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.407+0000 7f1c9d593640 1 --2- 172.21.15.83:0/265818245 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1c9815bf30 0x7f1c98140220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:28.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.407+0000 7f1c97fff640 1 --2- 172.21.15.83:0/265818245 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c9813dc10 0x7f1c98151290 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1c8400c4d0 tx=0x7f1c8400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.407+0000 7f1c957fa640 1 -- 172.21.15.83:0/265818245 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c84018070 con 0x7f1c9813dc10 2024-03-06T07:22:28.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.407+0000 7f1c957fa640 1 -- 172.21.15.83:0/265818245 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c84007500 con 0x7f1c9813dc10 2024-03-06T07:22:28.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.407+0000 7f1c9eff6640 1 -- 172.21.15.83:0/265818245 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c9813e2a0 con 0x7f1c9813dc10 2024-03-06T07:22:28.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.407+0000 7f1c957fa640 1 -- 172.21.15.83:0/265818245 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c84004a20 con 0x7f1c9813dc10 2024-03-06T07:22:28.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.407+0000 7f1c9eff6640 1 -- 172.21.15.83:0/265818245 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c9813e860 con 0x7f1c9813dc10 2024-03-06T07:22:28.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.408+0000 7f1c9eff6640 1 -- 172.21.15.83:0/265818245 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c9804f330 con 0x7f1c9813dc10 2024-03-06T07:22:28.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.409+0000 7f1c957fa640 1 -- 172.21.15.83:0/265818245 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1c8401b070 con 0x7f1c9813dc10 2024-03-06T07:22:28.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.409+0000 7f1c957fa640 1 --2- 172.21.15.83:0/265818245 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1c7c06b4d0 0x7f1c7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.410+0000 7f1c957fa640 1 -- 172.21.15.83:0/265818245 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1c840894f0 con 0x7f1c9813dc10 2024-03-06T07:22:28.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.410+0000 7f1c9cd92640 1 --2- 172.21.15.83:0/265818245 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1c7c06b4d0 0x7f1c7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.410+0000 7f1c9cd92640 1 --2- 172.21.15.83:0/265818245 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1c7c06b4d0 0x7f1c7c06d990 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c004f60 tx=0x7f1c8c007e30 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.418+0000 7f1c957fa640 1 -- 172.21.15.83:0/265818245 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1c84057c90 con 0x7f1c9813dc10 2024-03-06T07:22:28.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.644+0000 7f1c9eff6640 1 -- 172.21.15.83:0/265818245 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c9806eb00 con 0x7f1c9813dc10 2024-03-06T07:22:28.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.646+0000 7f1c957fa640 1 -- 172.21.15.83:0/265818245 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16617 (secure 0 0 0) 0x7f1c8405bd30 con 0x7f1c9813dc10 2024-03-06T07:22:28.648 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:22:28.648 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":20,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:21:59.938137+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.318942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.254795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:28.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.648+0000 7f1c9eff6640 1 -- 172.21.15.83:0/265818245 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1c7c06b4d0 msgr2=0x7f1c7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.648+0000 7f1c9eff6640 1 --2- 172.21.15.83:0/265818245 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1c7c06b4d0 0x7f1c7c06d990 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c004f60 tx=0x7f1c8c007e30 comp rx=0 tx=0).stop 2024-03-06T07:22:28.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.648+0000 7f1c9eff6640 1 -- 172.21.15.83:0/265818245 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c9813dc10 msgr2=0x7f1c98151290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.648+0000 7f1c9eff6640 1 --2- 172.21.15.83:0/265818245 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c9813dc10 0x7f1c98151290 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1c8400c4d0 tx=0x7f1c8400c990 comp rx=0 tx=0).stop 2024-03-06T07:22:28.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.648+0000 7f1c9eff6640 1 -- 172.21.15.83:0/265818245 shutdown_connections 2024-03-06T07:22:28.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.648+0000 7f1c9eff6640 1 --2- 172.21.15.83:0/265818245 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1c7c06b4d0 0x7f1c7c06d990 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.648+0000 7f1c9eff6640 1 --2- 172.21.15.83:0/265818245 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1c9815bf30 0x7f1c98140220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.648+0000 7f1c9eff6640 1 --2- 172.21.15.83:0/265818245 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c9813dc10 0x7f1c98151290 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.648+0000 7f1c9eff6640 1 --2- 172.21.15.83:0/265818245 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1c980b3060 0x7f1c9815ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.648+0000 7f1c9eff6640 1 -- 172.21.15.83:0/265818245 >> 172.21.15.83:0/265818245 conn(0x7f1c980cf8a0 msgr2=0x7f1c980af5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:28.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.648+0000 7f1c9eff6640 1 -- 172.21.15.83:0/265818245 shutdown_connections 2024-03-06T07:22:28.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.648+0000 7f1c9eff6640 1 -- 172.21.15.83:0/265818245 wait complete. 2024-03-06T07:22:28.661 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:56.318942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:57.254795+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:28.661 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:28.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.763+0000 7f1fa309d640 1 Processor -- start 2024-03-06T07:22:28.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.763+0000 7f1fa309d640 1 -- start start 2024-03-06T07:22:28.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.764+0000 7f1fa309d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1f9c1614d0 0x7f1f9c0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.764+0000 7f1fa309d640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f1f9c0d1910 0x7f1f9c0d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.764+0000 7f1fa309d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f9c1618b0 con 0x7f1f9c0d22e0 2024-03-06T07:22:28.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.764+0000 7f1fa309d640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f9c0cf400 con 0x7f1f9c1614d0 2024-03-06T07:22:28.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.764+0000 7f1fa309d640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f9c0d2710 con 0x7f1f9c0d1910 2024-03-06T07:22:28.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.764+0000 7f1fa0e39640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1f9c1614d0 0x7f1f9c0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.764+0000 7f1fa0e39640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1f9c1614d0 0x7f1f9c0d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48534/0 (socket says 172.21.15.83:48534) 2024-03-06T07:22:28.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.764+0000 7f1fa0e39640 1 -- 172.21.15.83:0/3983167552 learned_addr learned my addr 172.21.15.83:0/3983167552 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:28.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.764+0000 7f1fa0e39640 1 -- 172.21.15.83:0/3983167552 >> v2:172.21.15.183:3300/0 conn(0x7f1f9c0d1910 msgr2=0x7f1f9c0d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:28.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.764+0000 7f1fa0e39640 1 --2- 172.21.15.83:0/3983167552 >> v2:172.21.15.183:3300/0 conn(0x7f1f9c0d1910 0x7f1f9c0d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.764+0000 7f1fa0e39640 1 -- 172.21.15.83:0/3983167552 >> v1:172.21.15.183:6789/0 conn(0x7f1f9c0d22e0 legacy=0x7f1f9c0d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:28.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.764+0000 7f1fa0e39640 1 -- 172.21.15.83:0/3983167552 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f9c0cf6a0 con 0x7f1f9c1614d0 2024-03-06T07:22:28.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.765+0000 7f1fa0e39640 1 --2- 172.21.15.83:0/3983167552 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f9c1614d0 0x7f1f9c0d4860 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c08c470 tx=0x7f1f9402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79b6f9a7d0a6e7a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.765+0000 7f1f9b7fe640 1 -- 172.21.15.83:0/3983167552 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f9402ca70 con 0x7f1f9c1614d0 2024-03-06T07:22:28.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.765+0000 7f1f9b7fe640 1 -- 172.21.15.83:0/3983167552 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f9402cc10 con 0x7f1f9c1614d0 2024-03-06T07:22:28.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.765+0000 7f1f9b7fe640 1 -- 172.21.15.83:0/3983167552 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f940343f0 con 0x7f1f9c1614d0 2024-03-06T07:22:28.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.766+0000 7f1fa309d640 1 -- 172.21.15.83:0/3983167552 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f9c1614d0 msgr2=0x7f1f9c0d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.766+0000 7f1fa309d640 1 --2- 172.21.15.83:0/3983167552 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f9c1614d0 0x7f1f9c0d4860 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c08c470 tx=0x7f1f9402bed0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.766+0000 7f1fa309d640 1 -- 172.21.15.83:0/3983167552 shutdown_connections 2024-03-06T07:22:28.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.766+0000 7f1fa309d640 1 --2- 172.21.15.83:0/3983167552 >> v2:172.21.15.183:3300/0 conn(0x7f1f9c0d1910 0x7f1f9c0d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.766+0000 7f1fa309d640 1 --2- 172.21.15.83:0/3983167552 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f9c1614d0 0x7f1f9c0d4860 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.766+0000 7f1fa309d640 1 -- 172.21.15.83:0/3983167552 >> 172.21.15.83:0/3983167552 conn(0x7f1f9c0d3d40 msgr2=0x7f1f9c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:28.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.766+0000 7f1fa309d640 1 -- 172.21.15.83:0/3983167552 shutdown_connections 2024-03-06T07:22:28.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.766+0000 7f1fa309d640 1 -- 172.21.15.83:0/3983167552 wait complete. 2024-03-06T07:22:28.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.767+0000 7f1fa309d640 1 Processor -- start 2024-03-06T07:22:28.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.767+0000 7f1fa309d640 1 -- start start 2024-03-06T07:22:28.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.768+0000 7f1fa309d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f9c0d1910 0x7f1f9c15f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.768+0000 7f1fa309d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1f9c0d22e0 0x7f1f9c09edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.768+0000 7f1fa309d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f9c1614d0 0x7f1f9c0b7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.768+0000 7f1fa309d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9c0b0910 con 0x7f1f9c0d1910 2024-03-06T07:22:28.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.768+0000 7f1fa309d640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9c0cfc60 con 0x7f1f9c1614d0 2024-03-06T07:22:28.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.768+0000 7f1fa309d640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9c0b1e40 con 0x7f1f9c0d22e0 2024-03-06T07:22:28.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.768+0000 7f1fa0e39640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f9c0d1910 0x7f1f9c15f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.768+0000 7f1fa0e39640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f9c0d1910 0x7f1f9c15f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48538/0 (socket says 172.21.15.83:48538) 2024-03-06T07:22:28.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.768+0000 7f1fa0e39640 1 -- 172.21.15.83:0/1948384557 learned_addr learned my addr 172.21.15.83:0/1948384557 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:28.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.768+0000 7f1fa0e39640 1 -- 172.21.15.83:0/1948384557 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1f9c0d22e0 msgr2=0x7f1f9c09edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.768+0000 7f1fa0e39640 1 --2- 172.21.15.83:0/1948384557 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1f9c0d22e0 0x7f1f9c09edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.768+0000 7f1fa0e39640 1 -- 172.21.15.83:0/1948384557 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f9c1614d0 msgr2=0x7f1f9c0b7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.768+0000 7f1fa0e39640 1 --2- 172.21.15.83:0/1948384557 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f9c1614d0 0x7f1f9c0b7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.768+0000 7f1fa0e39640 1 -- 172.21.15.83:0/1948384557 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f9c0caf40 con 0x7f1f9c0d1910 2024-03-06T07:22:28.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.768+0000 7f1fa0e39640 1 --2- 172.21.15.83:0/1948384557 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f9c0d1910 0x7f1f9c15f320 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1f940093e0 tx=0x7f1f94004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.768+0000 7f1f997fa640 1 -- 172.21.15.83:0/1948384557 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f94043020 con 0x7f1f9c0d1910 2024-03-06T07:22:28.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.768+0000 7f1f997fa640 1 -- 172.21.15.83:0/1948384557 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f9403e450 con 0x7f1f9c0d1910 2024-03-06T07:22:28.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.768+0000 7f1fa309d640 1 -- 172.21.15.83:0/1948384557 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f9c0cb220 con 0x7f1f9c0d1910 2024-03-06T07:22:28.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.768+0000 7f1f997fa640 1 -- 172.21.15.83:0/1948384557 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f9403e840 con 0x7f1f9c0d1910 2024-03-06T07:22:28.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.768+0000 7f1fa309d640 1 -- 172.21.15.83:0/1948384557 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f9c0cb7e0 con 0x7f1f9c0d1910 2024-03-06T07:22:28.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.769+0000 7f1fa309d640 1 -- 172.21.15.83:0/1948384557 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f9c068ff0 con 0x7f1f9c0d1910 2024-03-06T07:22:28.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.770+0000 7f1f997fa640 1 -- 172.21.15.83:0/1948384557 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1f94046020 con 0x7f1f9c0d1910 2024-03-06T07:22:28.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.771+0000 7f1f997fa640 1 --2- 172.21.15.83:0/1948384557 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1f7c06b4d0 0x7f1f7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.771+0000 7f1f997fa640 1 -- 172.21.15.83:0/1948384557 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1f940acbb0 con 0x7f1f9c0d1910 2024-03-06T07:22:28.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.771+0000 7f1f9bfff640 1 --2- 172.21.15.83:0/1948384557 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1f7c06b4d0 0x7f1f7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.771+0000 7f1f9bfff640 1 --2- 172.21.15.83:0/1948384557 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1f7c06b4d0 0x7f1f7c06d990 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c1f7bc0 tx=0x7f1f84006c90 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.775+0000 7f1f997fa640 1 -- 172.21.15.83:0/1948384557 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1f9404d030 con 0x7f1f9c0d1910 2024-03-06T07:22:28.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.952+0000 7f1fa309d640 1 -- 172.21.15.83:0/1948384557 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1f9c0d0d00 con 0x7f1f9c0d1910 2024-03-06T07:22:28.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.952+0000 7f1f997fa640 1 -- 172.21.15.83:0/1948384557 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1f9407ba30 con 0x7f1f9c0d1910 2024-03-06T07:22:28.954 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:22:28.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.954+0000 7f1fa309d640 1 -- 172.21.15.83:0/1948384557 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1f7c06b4d0 msgr2=0x7f1f7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.954+0000 7f1fa309d640 1 --2- 172.21.15.83:0/1948384557 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1f7c06b4d0 0x7f1f7c06d990 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c1f7bc0 tx=0x7f1f84006c90 comp rx=0 tx=0).stop 2024-03-06T07:22:28.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.954+0000 7f1fa309d640 1 -- 172.21.15.83:0/1948384557 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f9c0d1910 msgr2=0x7f1f9c15f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.954+0000 7f1fa309d640 1 --2- 172.21.15.83:0/1948384557 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f9c0d1910 0x7f1f9c15f320 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1f940093e0 tx=0x7f1f94004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:28.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.954+0000 7f1fa309d640 1 -- 172.21.15.83:0/1948384557 shutdown_connections 2024-03-06T07:22:28.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.954+0000 7f1fa309d640 1 --2- 172.21.15.83:0/1948384557 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1f7c06b4d0 0x7f1f7c06d990 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.954+0000 7f1fa309d640 1 --2- 172.21.15.83:0/1948384557 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f9c1614d0 0x7f1f9c0b7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.954+0000 7f1fa309d640 1 --2- 172.21.15.83:0/1948384557 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1f9c0d22e0 0x7f1f9c09edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.954+0000 7f1fa309d640 1 --2- 172.21.15.83:0/1948384557 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f9c0d1910 0x7f1f9c15f320 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.954+0000 7f1fa309d640 1 -- 172.21.15.83:0/1948384557 >> 172.21.15.83:0/1948384557 conn(0x7f1f9c0d3d40 msgr2=0x7f1f9c0d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:28.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.954+0000 7f1fa309d640 1 -- 172.21.15.83:0/1948384557 shutdown_connections 2024-03-06T07:22:28.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:28.954+0000 7f1fa309d640 1 -- 172.21.15.83:0/1948384557 wait complete. 2024-03-06T07:22:28.967 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:29.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.072+0000 7fe6fe113640 1 Processor -- start 2024-03-06T07:22:29.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.072+0000 7fe6fe113640 1 -- start start 2024-03-06T07:22:29.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.072+0000 7fe6fe113640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe6f80d4860 0x7fe6f80d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.072+0000 7fe6fe113640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fe6f80b0e30 0x7fe6f80b1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.072+0000 7fe6fe113640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6f81618b0 con 0x7fe6f81614d0 2024-03-06T07:22:29.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.072+0000 7fe6fe113640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6f80d4cf0 con 0x7fe6f80d4860 2024-03-06T07:22:29.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.072+0000 7fe6fe113640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6f80cf400 con 0x7fe6f80b0e30 2024-03-06T07:22:29.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.073+0000 7fe6f6ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe6f80d4860 0x7fe6f80d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.073+0000 7fe6f7fff640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fe6f80b0e30 0x7fe6f80b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.073+0000 7fe6f6ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe6f80d4860 0x7fe6f80d2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48542/0 (socket says 172.21.15.83:48542) 2024-03-06T07:22:29.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.073+0000 7fe6f6ffd640 1 -- 172.21.15.83:0/3601358400 learned_addr learned my addr 172.21.15.83:0/3601358400 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:29.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.073+0000 7fe6f6ffd640 1 -- 172.21.15.83:0/3601358400 >> v2:172.21.15.183:3300/0 conn(0x7fe6f80b0e30 msgr2=0x7fe6f80b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:29.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.073+0000 7fe6f6ffd640 1 --2- 172.21.15.83:0/3601358400 >> v2:172.21.15.183:3300/0 conn(0x7fe6f80b0e30 0x7fe6f80b1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.073+0000 7fe6f6ffd640 1 -- 172.21.15.83:0/3601358400 >> v1:172.21.15.183:6789/0 conn(0x7fe6f81614d0 legacy=0x7fe6f80d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.073+0000 7fe6f6ffd640 1 -- 172.21.15.83:0/3601358400 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f80cf6a0 con 0x7fe6f80d4860 2024-03-06T07:22:29.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.074+0000 7fe6f6ffd640 1 --2- 172.21.15.83:0/3601358400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6f80d4860 0x7fe6f80d2020 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe6f80b2a70 tx=0x7fe6ec02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57028071ee74d6cd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.074+0000 7fe6f67fc640 1 -- 172.21.15.83:0/3601358400 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ec034070 con 0x7fe6f80d4860 2024-03-06T07:22:29.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.074+0000 7fe6f67fc640 1 -- 172.21.15.83:0/3601358400 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6ec030860 con 0x7fe6f80d4860 2024-03-06T07:22:29.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.074+0000 7fe6f67fc640 1 -- 172.21.15.83:0/3601358400 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ec030c30 con 0x7fe6f80d4860 2024-03-06T07:22:29.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.074+0000 7fe6fe113640 1 -- 172.21.15.83:0/3601358400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6f80d4860 msgr2=0x7fe6f80d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.074+0000 7fe6fe113640 1 --2- 172.21.15.83:0/3601358400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6f80d4860 0x7fe6f80d2020 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe6f80b2a70 tx=0x7fe6ec02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.074+0000 7fe6fe113640 1 -- 172.21.15.83:0/3601358400 shutdown_connections 2024-03-06T07:22:29.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.074+0000 7fe6fe113640 1 --2- 172.21.15.83:0/3601358400 >> v2:172.21.15.183:3300/0 conn(0x7fe6f80b0e30 0x7fe6f80b1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.074+0000 7fe6fe113640 1 --2- 172.21.15.83:0/3601358400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6f80d4860 0x7fe6f80d2020 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.074+0000 7fe6fe113640 1 -- 172.21.15.83:0/3601358400 >> 172.21.15.83:0/3601358400 conn(0x7fe6f80d3d40 msgr2=0x7fe6f80d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:29.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.074+0000 7fe6fe113640 1 -- 172.21.15.83:0/3601358400 shutdown_connections 2024-03-06T07:22:29.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.074+0000 7fe6fe113640 1 -- 172.21.15.83:0/3601358400 wait complete. 2024-03-06T07:22:29.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.075+0000 7fe6fe113640 1 Processor -- start 2024-03-06T07:22:29.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.076+0000 7fe6fe113640 1 -- start start 2024-03-06T07:22:29.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.076+0000 7fe6fe113640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6f80b0e30 0x7fe6f80b51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.076+0000 7fe6fe113640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6f80d4860 0x7fe6f8159df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.076+0000 7fe6fe113640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe6f81614d0 0x7fe6f81e9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.076+0000 7fe6fe113640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f80b0910 con 0x7fe6f80b0e30 2024-03-06T07:22:29.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.076+0000 7fe6fe113640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f80cfc60 con 0x7fe6f80d4860 2024-03-06T07:22:29.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.076+0000 7fe6fe113640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f80d2560 con 0x7fe6f81614d0 2024-03-06T07:22:29.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.076+0000 7fe6f77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6f80b0e30 0x7fe6f80b51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.076+0000 7fe6f77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6f80b0e30 0x7fe6f80b51b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48548/0 (socket says 172.21.15.83:48548) 2024-03-06T07:22:29.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.076+0000 7fe6f77fe640 1 -- 172.21.15.83:0/1895266195 learned_addr learned my addr 172.21.15.83:0/1895266195 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:29.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.077+0000 7fe6f6ffd640 1 --2- 172.21.15.83:0/1895266195 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6f80d4860 0x7fe6f8159df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.077+0000 7fe6f7fff640 1 --2- 172.21.15.83:0/1895266195 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe6f81614d0 0x7fe6f81e9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.077+0000 7fe6f77fe640 1 -- 172.21.15.83:0/1895266195 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe6f81614d0 msgr2=0x7fe6f81e9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.077+0000 7fe6f77fe640 1 --2- 172.21.15.83:0/1895266195 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe6f81614d0 0x7fe6f81e9930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.077+0000 7fe6f77fe640 1 -- 172.21.15.83:0/1895266195 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6f80d4860 msgr2=0x7fe6f8159df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.077+0000 7fe6f77fe640 1 --2- 172.21.15.83:0/1895266195 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6f80d4860 0x7fe6f8159df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.077+0000 7fe6f77fe640 1 -- 172.21.15.83:0/1895266195 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f80cb310 con 0x7fe6f80b0e30 2024-03-06T07:22:29.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.077+0000 7fe6f6ffd640 1 --2- 172.21.15.83:0/1895266195 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6f80d4860 0x7fe6f8159df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:29.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.077+0000 7fe6f7fff640 1 --2- 172.21.15.83:0/1895266195 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe6f81614d0 0x7fe6f81e9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:29.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.077+0000 7fe6f77fe640 1 --2- 172.21.15.83:0/1895266195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6f80b0e30 0x7fe6f80b51b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe6e800c340 tx=0x7fe6e800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.077+0000 7fe6dffff640 1 -- 172.21.15.83:0/1895266195 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e8017070 con 0x7fe6f80b0e30 2024-03-06T07:22:29.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.077+0000 7fe6dffff640 1 -- 172.21.15.83:0/1895266195 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6e80052d0 con 0x7fe6f80b0e30 2024-03-06T07:22:29.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.077+0000 7fe6dffff640 1 -- 172.21.15.83:0/1895266195 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e8003850 con 0x7fe6f80b0e30 2024-03-06T07:22:29.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.077+0000 7fe6fe113640 1 -- 172.21.15.83:0/1895266195 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6f80cb5f0 con 0x7fe6f80b0e30 2024-03-06T07:22:29.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.077+0000 7fe6fe113640 1 -- 172.21.15.83:0/1895266195 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6f80cbbb0 con 0x7fe6f80b0e30 2024-03-06T07:22:29.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.078+0000 7fe6fe113640 1 -- 172.21.15.83:0/1895266195 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6f8068ff0 con 0x7fe6f80b0e30 2024-03-06T07:22:29.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.079+0000 7fe6dffff640 1 -- 172.21.15.83:0/1895266195 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe6e801a070 con 0x7fe6f80b0e30 2024-03-06T07:22:29.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.079+0000 7fe6dffff640 1 --2- 172.21.15.83:0/1895266195 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe6e406b4d0 0x7fe6e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.079+0000 7fe6dffff640 1 -- 172.21.15.83:0/1895266195 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe6e8089510 con 0x7fe6f80b0e30 2024-03-06T07:22:29.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.079+0000 7fe6f6ffd640 1 --2- 172.21.15.83:0/1895266195 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe6e406b4d0 0x7fe6e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.079+0000 7fe6f6ffd640 1 --2- 172.21.15.83:0/1895266195 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe6e406b4d0 0x7fe6e406d990 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec036000 tx=0x7fe6ec03b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.083+0000 7fe6dffff640 1 -- 172.21.15.83:0/1895266195 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe6e8058390 con 0x7fe6f80b0e30 2024-03-06T07:22:29.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.281+0000 7fe6fe113640 1 -- 172.21.15.83:0/1895266195 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe6f80cbdf0 con 0x7fe6f80b0e30 2024-03-06T07:22:29.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.281+0000 7fe6dffff640 1 -- 172.21.15.83:0/1895266195 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe6e8057c90 con 0x7fe6f80b0e30 2024-03-06T07:22:29.284 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-03-06T07:22:29.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.283+0000 7fe6fe113640 1 -- 172.21.15.83:0/1895266195 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe6e406b4d0 msgr2=0x7fe6e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.283+0000 7fe6fe113640 1 --2- 172.21.15.83:0/1895266195 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe6e406b4d0 0x7fe6e406d990 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec036000 tx=0x7fe6ec03b040 comp rx=0 tx=0).stop 2024-03-06T07:22:29.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.283+0000 7fe6fe113640 1 -- 172.21.15.83:0/1895266195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6f80b0e30 msgr2=0x7fe6f80b51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.283+0000 7fe6fe113640 1 --2- 172.21.15.83:0/1895266195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6f80b0e30 0x7fe6f80b51b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe6e800c340 tx=0x7fe6e800c800 comp rx=0 tx=0).stop 2024-03-06T07:22:29.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.283+0000 7fe6fe113640 1 -- 172.21.15.83:0/1895266195 shutdown_connections 2024-03-06T07:22:29.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.283+0000 7fe6fe113640 1 --2- 172.21.15.83:0/1895266195 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe6e406b4d0 0x7fe6e406d990 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.283+0000 7fe6fe113640 1 --2- 172.21.15.83:0/1895266195 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe6f81614d0 0x7fe6f81e9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.283+0000 7fe6fe113640 1 --2- 172.21.15.83:0/1895266195 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6f80d4860 0x7fe6f8159df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.283+0000 7fe6fe113640 1 --2- 172.21.15.83:0/1895266195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6f80b0e30 0x7fe6f80b51b0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.283+0000 7fe6fe113640 1 -- 172.21.15.83:0/1895266195 >> 172.21.15.83:0/1895266195 conn(0x7fe6f80d3d40 msgr2=0x7fe6f80cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:29.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.284+0000 7fe6fe113640 1 -- 172.21.15.83:0/1895266195 shutdown_connections 2024-03-06T07:22:29.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.284+0000 7fe6fe113640 1 -- 172.21.15.83:0/1895266195 wait complete. 2024-03-06T07:22:29.300 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:29.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.401+0000 7f15ed0eb640 1 Processor -- start 2024-03-06T07:22:29.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.401+0000 7f15ed0eb640 1 -- start start 2024-03-06T07:22:29.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.402+0000 7f15ed0eb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f15e81614d0 0x7f15e80d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.402+0000 7f15ed0eb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f15e80d1910 0x7f15e80d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.402+0000 7f15ed0eb640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f15e80d22b0 0x7f15e80d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.402+0000 7f15ed0eb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15e80d2740 con 0x7f15e80d1910 2024-03-06T07:22:29.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.402+0000 7f15ed0eb640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15e81618b0 con 0x7f15e80d22b0 2024-03-06T07:22:29.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.402+0000 7f15ed0eb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15e80cf400 con 0x7f15e81614d0 2024-03-06T07:22:29.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.402+0000 7f15de59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f15e80d1910 0x7f15e80d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.402+0000 7f15de59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f15e80d1910 0x7f15e80d1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48554/0 (socket says 172.21.15.83:48554) 2024-03-06T07:22:29.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.402+0000 7f15de59c640 1 -- 172.21.15.83:0/2868360749 learned_addr learned my addr 172.21.15.83:0/2868360749 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:29.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.402+0000 7f15de59c640 1 -- 172.21.15.83:0/2868360749 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f15e81614d0 msgr2=0x7f15e80d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:29.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.402+0000 7f15de59c640 1 --2- 172.21.15.83:0/2868360749 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f15e81614d0 0x7f15e80d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.402+0000 7f15de59c640 1 -- 172.21.15.83:0/2868360749 >> v2:172.21.15.183:3300/0 conn(0x7f15e80d22b0 msgr2=0x7f15e80d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:29.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.402+0000 7f15e759e640 1 --2- 172.21.15.83:0/2868360749 >> v2:172.21.15.183:3300/0 conn(0x7f15e80d22b0 0x7f15e80d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.402+0000 7f15de59c640 1 --2- 172.21.15.83:0/2868360749 >> v2:172.21.15.183:3300/0 conn(0x7f15e80d22b0 0x7f15e80d06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.402+0000 7f15de59c640 1 -- 172.21.15.83:0/2868360749 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e80cf6a0 con 0x7f15e80d1910 2024-03-06T07:22:29.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.403+0000 7f15de59c640 1 --2- 172.21.15.83:0/2868360749 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15e80d1910 0x7f15e80d1d70 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f15e80b57d0 tx=0x7f15d002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf424f5cb7ca5942 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.403+0000 7f15e659c640 1 -- 172.21.15.83:0/2868360749 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d0035020 con 0x7f15e80d1910 2024-03-06T07:22:29.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.403+0000 7f15e659c640 1 -- 172.21.15.83:0/2868360749 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15d0031870 con 0x7f15e80d1910 2024-03-06T07:22:29.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.403+0000 7f15e659c640 1 -- 172.21.15.83:0/2868360749 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d0031c40 con 0x7f15e80d1910 2024-03-06T07:22:29.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.403+0000 7f15ed0eb640 1 -- 172.21.15.83:0/2868360749 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15e80d1910 msgr2=0x7f15e80d1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.404+0000 7f15ed0eb640 1 --2- 172.21.15.83:0/2868360749 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15e80d1910 0x7f15e80d1d70 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f15e80b57d0 tx=0x7f15d002b910 comp rx=0 tx=0).stop 2024-03-06T07:22:29.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.404+0000 7f15ed0eb640 1 -- 172.21.15.83:0/2868360749 shutdown_connections 2024-03-06T07:22:29.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.404+0000 7f15ed0eb640 1 --2- 172.21.15.83:0/2868360749 >> v2:172.21.15.183:3300/0 conn(0x7f15e80d22b0 0x7f15e80d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.404+0000 7f15ed0eb640 1 --2- 172.21.15.83:0/2868360749 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15e80d1910 0x7f15e80d1d70 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.404+0000 7f15ed0eb640 1 --2- 172.21.15.83:0/2868360749 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f15e81614d0 0x7f15e80d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.404+0000 7f15ed0eb640 1 -- 172.21.15.83:0/2868360749 >> 172.21.15.83:0/2868360749 conn(0x7f15e80d3d40 msgr2=0x7f15e80d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:29.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.404+0000 7f15ed0eb640 1 -- 172.21.15.83:0/2868360749 shutdown_connections 2024-03-06T07:22:29.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.404+0000 7f15ed0eb640 1 -- 172.21.15.83:0/2868360749 wait complete. 2024-03-06T07:22:29.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.406+0000 7f15ed0eb640 1 Processor -- start 2024-03-06T07:22:29.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.406+0000 7f15ed0eb640 1 -- start start 2024-03-06T07:22:29.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.407+0000 7f15ed0eb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f15e80d1910 0x7f15e80b80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.407+0000 7f15ed0eb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15e80d22b0 0x7f15e80b42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.407+0000 7f15ed0eb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f15e81614d0 0x7f15e81f3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.407+0000 7f15ed0eb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e80b0910 con 0x7f15e80d22b0 2024-03-06T07:22:29.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.407+0000 7f15ed0eb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e80cfc60 con 0x7f15e81614d0 2024-03-06T07:22:29.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.407+0000 7f15ed0eb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15e80d0be0 con 0x7f15e80d1910 2024-03-06T07:22:29.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.407+0000 7f15de59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15e80d22b0 0x7f15e80b42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.407+0000 7f15de59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15e80d22b0 0x7f15e80b42e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48556/0 (socket says 172.21.15.83:48556) 2024-03-06T07:22:29.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.407+0000 7f15de59c640 1 -- 172.21.15.83:0/780993873 learned_addr learned my addr 172.21.15.83:0/780993873 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:29.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.407+0000 7f15e759e640 1 --2- 172.21.15.83:0/780993873 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f15e81614d0 0x7f15e81f3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.407+0000 7f15e6d9d640 1 --2- 172.21.15.83:0/780993873 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f15e80d1910 0x7f15e80b80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.407+0000 7f15de59c640 1 -- 172.21.15.83:0/780993873 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f15e80d1910 msgr2=0x7f15e80b80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.407+0000 7f15de59c640 1 --2- 172.21.15.83:0/780993873 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f15e80d1910 0x7f15e80b80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.407+0000 7f15de59c640 1 -- 172.21.15.83:0/780993873 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f15e81614d0 msgr2=0x7f15e81f3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.407+0000 7f15de59c640 1 --2- 172.21.15.83:0/780993873 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f15e81614d0 0x7f15e81f3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.407+0000 7f15de59c640 1 -- 172.21.15.83:0/780993873 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e80caf40 con 0x7f15e80d22b0 2024-03-06T07:22:29.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.407+0000 7f15e759e640 1 --2- 172.21.15.83:0/780993873 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f15e81614d0 0x7f15e81f3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:29.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.407+0000 7f15e6d9d640 1 --2- 172.21.15.83:0/780993873 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f15e80d1910 0x7f15e80b80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:29.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.408+0000 7f15de59c640 1 --2- 172.21.15.83:0/780993873 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15e80d22b0 0x7f15e80b42e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f15d002b950 tx=0x7f15d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.408+0000 7f15dffff640 1 -- 172.21.15.83:0/780993873 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d0040070 con 0x7f15e80d22b0 2024-03-06T07:22:29.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.408+0000 7f15dffff640 1 -- 172.21.15.83:0/780993873 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15d003a420 con 0x7f15e80d22b0 2024-03-06T07:22:29.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.408+0000 7f15ed0eb640 1 -- 172.21.15.83:0/780993873 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15e80cb220 con 0x7f15e80d22b0 2024-03-06T07:22:29.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.408+0000 7f15dffff640 1 -- 172.21.15.83:0/780993873 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d0035040 con 0x7f15e80d22b0 2024-03-06T07:22:29.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.408+0000 7f15ed0eb640 1 -- 172.21.15.83:0/780993873 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15e80cb7e0 con 0x7f15e80d22b0 2024-03-06T07:22:29.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.409+0000 7f15ed0eb640 1 -- 172.21.15.83:0/780993873 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15e80ca680 con 0x7f15e80d22b0 2024-03-06T07:22:29.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.409+0000 7f15dffff640 1 -- 172.21.15.83:0/780993873 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f15d004b060 con 0x7f15e80d22b0 2024-03-06T07:22:29.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.410+0000 7f15dffff640 1 --2- 172.21.15.83:0/780993873 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f15c806b4d0 0x7f15c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.410+0000 7f15e6d9d640 1 --2- 172.21.15.83:0/780993873 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f15c806b4d0 0x7f15c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.410+0000 7f15dffff640 1 -- 172.21.15.83:0/780993873 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f15d00ac110 con 0x7f15e80d22b0 2024-03-06T07:22:29.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.410+0000 7f15e6d9d640 1 --2- 172.21.15.83:0/780993873 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f15c806b4d0 0x7f15c806d990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f15d800a870 tx=0x7f15d8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.414+0000 7f15dffff640 1 -- 172.21.15.83:0/780993873 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f15d007b020 con 0x7f15e80d22b0 2024-03-06T07:22:29.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.591+0000 7f15ed0eb640 1 -- 172.21.15.83:0/780993873 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f15e80cba20 con 0x7f15e80d22b0 2024-03-06T07:22:29.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.591+0000 7f15dffff640 1 -- 172.21.15.83:0/780993873 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f15d007a920 con 0x7f15e80d22b0 2024-03-06T07:22:29.593 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 64 2024-03-06T07:22:29.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.593+0000 7f15ed0eb640 1 -- 172.21.15.83:0/780993873 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f15c806b4d0 msgr2=0x7f15c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.593+0000 7f15ed0eb640 1 --2- 172.21.15.83:0/780993873 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f15c806b4d0 0x7f15c806d990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f15d800a870 tx=0x7f15d8008040 comp rx=0 tx=0).stop 2024-03-06T07:22:29.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.593+0000 7f15ed0eb640 1 -- 172.21.15.83:0/780993873 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15e80d22b0 msgr2=0x7f15e80b42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.593+0000 7f15ed0eb640 1 --2- 172.21.15.83:0/780993873 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15e80d22b0 0x7f15e80b42e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f15d002b950 tx=0x7f15d00046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.593+0000 7f15ed0eb640 1 -- 172.21.15.83:0/780993873 shutdown_connections 2024-03-06T07:22:29.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.593+0000 7f15ed0eb640 1 --2- 172.21.15.83:0/780993873 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f15c806b4d0 0x7f15c806d990 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.593+0000 7f15ed0eb640 1 --2- 172.21.15.83:0/780993873 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f15e81614d0 0x7f15e81f3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.593+0000 7f15ed0eb640 1 --2- 172.21.15.83:0/780993873 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15e80d22b0 0x7f15e80b42e0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.593+0000 7f15ed0eb640 1 --2- 172.21.15.83:0/780993873 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f15e80d1910 0x7f15e80b80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.593+0000 7f15ed0eb640 1 -- 172.21.15.83:0/780993873 >> 172.21.15.83:0/780993873 conn(0x7f15e80d3d40 msgr2=0x7f15e806ba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:29.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.593+0000 7f15ed0eb640 1 -- 172.21.15.83:0/780993873 shutdown_connections 2024-03-06T07:22:29.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.593+0000 7f15ed0eb640 1 -- 172.21.15.83:0/780993873 wait complete. 2024-03-06T07:22:29.616 INFO:tasks.cephfs_test_runner:Starting test: test_dir_deletion (tasks.cephfs.test_strays.TestStrays) That when deleting a bunch of dentries and the containing 2024-03-06T07:22:29.616 INFO:tasks.cephfs_test_runner:test_dir_deletion (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:22:29.617 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:29.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.727+0000 7f468224e640 1 Processor -- start 2024-03-06T07:22:29.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.727+0000 7f468224e640 1 -- start start 2024-03-06T07:22:29.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.728+0000 7f468224e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f467c1515e0 0x7f467c0cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.728+0000 7f468224e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f467c15b430 0x7f467c071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.728+0000 7f468224e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f467c0cc340 0x7f467c0cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.728+0000 7f468224e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f467c15b8b0 con 0x7f467c0cc340 2024-03-06T07:22:29.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.728+0000 7f468224e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f467c070a70 con 0x7f467c15b430 2024-03-06T07:22:29.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.728+0000 7f468224e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f467c070d50 con 0x7f467c1515e0 2024-03-06T07:22:29.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.728+0000 7f467bfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f467c0cc340 0x7f467c0cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.728+0000 7f467bfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f467c0cc340 0x7f467c0cd7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48564/0 (socket says 172.21.15.83:48564) 2024-03-06T07:22:29.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.728+0000 7f467bfff640 1 -- 172.21.15.83:0/6265312 learned_addr learned my addr 172.21.15.83:0/6265312 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:29.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.728+0000 7f467b7fe640 1 --2- 172.21.15.83:0/6265312 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f467c1515e0 0x7f467c0cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.728+0000 7f4672ffd640 1 --2- 172.21.15.83:0/6265312 >> v2:172.21.15.183:3300/0 conn(0x7f467c15b430 0x7f467c071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.728+0000 7f467bfff640 1 -- 172.21.15.83:0/6265312 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f467c1515e0 msgr2=0x7f467c0cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:29.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.728+0000 7f467bfff640 1 --2- 172.21.15.83:0/6265312 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f467c1515e0 0x7f467c0cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.728+0000 7f467bfff640 1 -- 172.21.15.83:0/6265312 >> v2:172.21.15.183:3300/0 conn(0x7f467c15b430 msgr2=0x7f467c071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:29.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.728+0000 7f467bfff640 1 --2- 172.21.15.83:0/6265312 >> v2:172.21.15.183:3300/0 conn(0x7f467c15b430 0x7f467c071830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.728+0000 7f467bfff640 1 -- 172.21.15.83:0/6265312 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f467c071030 con 0x7f467c0cc340 2024-03-06T07:22:29.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.729+0000 7f467bfff640 1 --2- 172.21.15.83:0/6265312 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f467c0cc340 0x7f467c0cd7a0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f467c073220 tx=0x7f467402e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad0c6fcde2d17a3e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.729+0000 7f467affd640 1 -- 172.21.15.83:0/6265312 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f467402ea80 con 0x7f467c0cc340 2024-03-06T07:22:29.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.729+0000 7f467affd640 1 -- 172.21.15.83:0/6265312 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f467402ec20 con 0x7f467c0cc340 2024-03-06T07:22:29.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.729+0000 7f467affd640 1 -- 172.21.15.83:0/6265312 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4674009600 con 0x7f467c0cc340 2024-03-06T07:22:29.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.730+0000 7f468224e640 1 -- 172.21.15.83:0/6265312 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f467c0cc340 msgr2=0x7f467c0cd7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.730+0000 7f468224e640 1 --2- 172.21.15.83:0/6265312 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f467c0cc340 0x7f467c0cd7a0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f467c073220 tx=0x7f467402e0e0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.730+0000 7f468224e640 1 -- 172.21.15.83:0/6265312 shutdown_connections 2024-03-06T07:22:29.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.730+0000 7f468224e640 1 --2- 172.21.15.83:0/6265312 >> v2:172.21.15.183:3300/0 conn(0x7f467c15b430 0x7f467c071830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.730+0000 7f468224e640 1 --2- 172.21.15.83:0/6265312 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f467c0cc340 0x7f467c0cd7a0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.730+0000 7f468224e640 1 --2- 172.21.15.83:0/6265312 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f467c1515e0 0x7f467c0cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.730+0000 7f468224e640 1 -- 172.21.15.83:0/6265312 >> 172.21.15.83:0/6265312 conn(0x7f467c0cf8a0 msgr2=0x7f467c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:29.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.730+0000 7f468224e640 1 -- 172.21.15.83:0/6265312 shutdown_connections 2024-03-06T07:22:29.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.730+0000 7f468224e640 1 -- 172.21.15.83:0/6265312 wait complete. 2024-03-06T07:22:29.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.731+0000 7f468224e640 1 Processor -- start 2024-03-06T07:22:29.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.731+0000 7f468224e640 1 -- start start 2024-03-06T07:22:29.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.732+0000 7f468224e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f467c0cc340 0x7f467c1f29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.732+0000 7f468224e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f467c1515e0 0x7f467c15d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.732+0000 7f468224e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f467c15b430 0x7f467c14df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.732+0000 7f468224e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f467c071ef0 con 0x7f467c1515e0 2024-03-06T07:22:29.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.732+0000 7f468224e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f467c071d70 con 0x7f467c0cc340 2024-03-06T07:22:29.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.732+0000 7f468224e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f467c0cc770 con 0x7f467c15b430 2024-03-06T07:22:29.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.732+0000 7f4672ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f467c1515e0 0x7f467c15d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.732+0000 7f4672ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f467c1515e0 0x7f467c15d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48566/0 (socket says 172.21.15.83:48566) 2024-03-06T07:22:29.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.732+0000 7f4672ffd640 1 -- 172.21.15.83:0/2726323317 learned_addr learned my addr 172.21.15.83:0/2726323317 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:29.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.732+0000 7f467b7fe640 1 --2- 172.21.15.83:0/2726323317 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f467c0cc340 0x7f467c1f29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.732+0000 7f467bfff640 1 --2- 172.21.15.83:0/2726323317 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f467c15b430 0x7f467c14df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.732+0000 7f4672ffd640 1 -- 172.21.15.83:0/2726323317 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f467c15b430 msgr2=0x7f467c14df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.732+0000 7f4672ffd640 1 --2- 172.21.15.83:0/2726323317 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f467c15b430 0x7f467c14df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.732+0000 7f4672ffd640 1 -- 172.21.15.83:0/2726323317 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f467c0cc340 msgr2=0x7f467c1f29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.732+0000 7f4672ffd640 1 --2- 172.21.15.83:0/2726323317 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f467c0cc340 0x7f467c1f29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.732+0000 7f4672ffd640 1 -- 172.21.15.83:0/2726323317 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f467c0c5f10 con 0x7f467c1515e0 2024-03-06T07:22:29.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.732+0000 7f467b7fe640 1 --2- 172.21.15.83:0/2726323317 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f467c0cc340 0x7f467c1f29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:29.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.732+0000 7f467bfff640 1 --2- 172.21.15.83:0/2726323317 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f467c15b430 0x7f467c14df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:29.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.733+0000 7f4672ffd640 1 --2- 172.21.15.83:0/2726323317 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f467c1515e0 0x7f467c15d490 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f466400d5f0 tx=0x7f466400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.733+0000 7f4678ff9640 1 -- 172.21.15.83:0/2726323317 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46640125f0 con 0x7f467c1515e0 2024-03-06T07:22:29.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.733+0000 7f4678ff9640 1 -- 172.21.15.83:0/2726323317 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4664019070 con 0x7f467c1515e0 2024-03-06T07:22:29.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.733+0000 7f4678ff9640 1 -- 172.21.15.83:0/2726323317 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46640129d0 con 0x7f467c1515e0 2024-03-06T07:22:29.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.733+0000 7f468224e640 1 -- 172.21.15.83:0/2726323317 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f467c0c61f0 con 0x7f467c1515e0 2024-03-06T07:22:29.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.733+0000 7f468224e640 1 -- 172.21.15.83:0/2726323317 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f467c0c67b0 con 0x7f467c1515e0 2024-03-06T07:22:29.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.734+0000 7f468224e640 1 -- 172.21.15.83:0/2726323317 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f467c068840 con 0x7f467c1515e0 2024-03-06T07:22:29.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.734+0000 7f4678ff9640 1 -- 172.21.15.83:0/2726323317 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4664012b70 con 0x7f467c1515e0 2024-03-06T07:22:29.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.734+0000 7f4678ff9640 1 --2- 172.21.15.83:0/2726323317 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f466006b4d0 0x7f466006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.735+0000 7f4678ff9640 1 -- 172.21.15.83:0/2726323317 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f466408a600 con 0x7f467c1515e0 2024-03-06T07:22:29.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.735+0000 7f467b7fe640 1 --2- 172.21.15.83:0/2726323317 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f466006b4d0 0x7f466006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.735+0000 7f467b7fe640 1 --2- 172.21.15.83:0/2726323317 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f466006b4d0 0x7f466006d990 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f466c006fd0 tx=0x7f466c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:29.739+0000 7f4678ff9640 1 -- 172.21.15.83:0/2726323317 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4664059480 con 0x7f467c1515e0 2024-03-06T07:22:30.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.005+0000 7f468224e640 1 -- 172.21.15.83:0/2726323317 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f467c04f330 con 0x7f467c1515e0 2024-03-06T07:22:30.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.006+0000 7f4678ff9640 1 -- 172.21.15.83:0/2726323317 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16617 (secure 0 0 0) 0x7f4664058d80 con 0x7f467c1515e0 2024-03-06T07:22:30.008 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:22:30.008 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":20,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:21:59.938137+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.318942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.254795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:30.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.008+0000 7f468224e640 1 -- 172.21.15.83:0/2726323317 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f466006b4d0 msgr2=0x7f466006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.008+0000 7f468224e640 1 --2- 172.21.15.83:0/2726323317 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f466006b4d0 0x7f466006d990 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f466c006fd0 tx=0x7f466c008040 comp rx=0 tx=0).stop 2024-03-06T07:22:30.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.008+0000 7f468224e640 1 -- 172.21.15.83:0/2726323317 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f467c1515e0 msgr2=0x7f467c15d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.008+0000 7f468224e640 1 --2- 172.21.15.83:0/2726323317 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f467c1515e0 0x7f467c15d490 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f466400d5f0 tx=0x7f466400dab0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.008+0000 7f468224e640 1 -- 172.21.15.83:0/2726323317 shutdown_connections 2024-03-06T07:22:30.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.008+0000 7f468224e640 1 --2- 172.21.15.83:0/2726323317 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f466006b4d0 0x7f466006d990 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.008+0000 7f468224e640 1 --2- 172.21.15.83:0/2726323317 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f467c15b430 0x7f467c14df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.008+0000 7f468224e640 1 --2- 172.21.15.83:0/2726323317 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f467c1515e0 0x7f467c15d490 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.008+0000 7f468224e640 1 --2- 172.21.15.83:0/2726323317 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f467c0cc340 0x7f467c1f29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.008+0000 7f468224e640 1 -- 172.21.15.83:0/2726323317 >> 172.21.15.83:0/2726323317 conn(0x7f467c0cf8a0 msgr2=0x7f467c150850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:30.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.008+0000 7f468224e640 1 -- 172.21.15.83:0/2726323317 shutdown_connections 2024-03-06T07:22:30.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.008+0000 7f468224e640 1 -- 172.21.15.83:0/2726323317 wait complete. 2024-03-06T07:22:30.021 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:56.318942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:57.254795+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:30.021 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:30.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.129+0000 7f671aaed640 1 Processor -- start 2024-03-06T07:22:30.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.129+0000 7f671aaed640 1 -- start start 2024-03-06T07:22:30.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.130+0000 7f671aaed640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6714145590 0x7f671406afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.130+0000 7f671aaed640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f671406b4f0 0x7f671406efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.130+0000 7f671aaed640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f671406f510 0x7f671406daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.130+0000 7f671aaed640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6714145970 con 0x7f6714145590 2024-03-06T07:22:30.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.130+0000 7f671aaed640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6714160110 con 0x7f671406f510 2024-03-06T07:22:30.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.130+0000 7f671aaed640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f671406b970 con 0x7f671406b4f0 2024-03-06T07:22:30.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.130+0000 7f6718889640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6714145590 0x7f671406afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.130+0000 7f6718889640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6714145590 0x7f671406afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48580/0 (socket says 172.21.15.83:48580) 2024-03-06T07:22:30.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.130+0000 7f6718889640 1 -- 172.21.15.83:0/3687325661 learned_addr learned my addr 172.21.15.83:0/3687325661 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:30.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.130+0000 7f6713fff640 1 --2- 172.21.15.83:0/3687325661 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f671406b4f0 0x7f671406efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.130+0000 7f671908a640 1 --2- 172.21.15.83:0/3687325661 >> v2:172.21.15.183:3300/0 conn(0x7f671406f510 0x7f671406daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.130+0000 7f6718889640 1 -- 172.21.15.83:0/3687325661 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f671406b4f0 msgr2=0x7f671406efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:30.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.130+0000 7f6718889640 1 --2- 172.21.15.83:0/3687325661 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f671406b4f0 0x7f671406efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.130+0000 7f6718889640 1 -- 172.21.15.83:0/3687325661 >> v2:172.21.15.183:3300/0 conn(0x7f671406f510 msgr2=0x7f671406daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:30.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.130+0000 7f6718889640 1 --2- 172.21.15.83:0/3687325661 >> v2:172.21.15.183:3300/0 conn(0x7f671406f510 0x7f671406daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.130+0000 7f6718889640 1 -- 172.21.15.83:0/3687325661 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f671415f7e0 con 0x7f6714145590 2024-03-06T07:22:30.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.130+0000 7f6713fff640 1 --2- 172.21.15.83:0/3687325661 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f671406b4f0 0x7f671406efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:30.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.131+0000 7f6718889640 1 --2- 172.21.15.83:0/3687325661 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6714145590 0x7f671406afb0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f671406be50 tx=0x7f670402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bda55fc5f1d2d0f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.131+0000 7f67137fe640 1 -- 172.21.15.83:0/3687325661 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6704035020 con 0x7f6714145590 2024-03-06T07:22:30.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.131+0000 7f67137fe640 1 -- 172.21.15.83:0/3687325661 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6704031870 con 0x7f6714145590 2024-03-06T07:22:30.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.131+0000 7f67137fe640 1 -- 172.21.15.83:0/3687325661 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6704031c40 con 0x7f6714145590 2024-03-06T07:22:30.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.132+0000 7f671aaed640 1 -- 172.21.15.83:0/3687325661 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6714145590 msgr2=0x7f671406afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.132+0000 7f671aaed640 1 --2- 172.21.15.83:0/3687325661 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6714145590 0x7f671406afb0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f671406be50 tx=0x7f670402b910 comp rx=0 tx=0).stop 2024-03-06T07:22:30.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.132+0000 7f671aaed640 1 -- 172.21.15.83:0/3687325661 shutdown_connections 2024-03-06T07:22:30.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.132+0000 7f671aaed640 1 --2- 172.21.15.83:0/3687325661 >> v2:172.21.15.183:3300/0 conn(0x7f671406f510 0x7f671406daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.132+0000 7f671aaed640 1 --2- 172.21.15.83:0/3687325661 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f671406b4f0 0x7f671406efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.132+0000 7f671aaed640 1 --2- 172.21.15.83:0/3687325661 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6714145590 0x7f671406afb0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.132+0000 7f671aaed640 1 -- 172.21.15.83:0/3687325661 >> 172.21.15.83:0/3687325661 conn(0x7f67140b2660 msgr2=0x7f67140b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:30.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.132+0000 7f671aaed640 1 -- 172.21.15.83:0/3687325661 shutdown_connections 2024-03-06T07:22:30.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.132+0000 7f671aaed640 1 -- 172.21.15.83:0/3687325661 wait complete. 2024-03-06T07:22:30.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.134+0000 7f671aaed640 1 Processor -- start 2024-03-06T07:22:30.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.134+0000 7f671aaed640 1 -- start start 2024-03-06T07:22:30.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.135+0000 7f671aaed640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f671406b4f0 0x7f671415b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.135+0000 7f671aaed640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f671406f510 0x7f67141e5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.135+0000 7f671aaed640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6714145590 0x7f67140b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.135+0000 7f671aaed640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67141605a0 con 0x7f6714145590 2024-03-06T07:22:30.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.135+0000 7f671aaed640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671415fda0 con 0x7f671406f510 2024-03-06T07:22:30.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.135+0000 7f671aaed640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67141e7180 con 0x7f671406b4f0 2024-03-06T07:22:30.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.135+0000 7f671908a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6714145590 0x7f67140b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.135+0000 7f671908a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6714145590 0x7f67140b4f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48586/0 (socket says 172.21.15.83:48586) 2024-03-06T07:22:30.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.135+0000 7f671908a640 1 -- 172.21.15.83:0/2256264443 learned_addr learned my addr 172.21.15.83:0/2256264443 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:30.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.135+0000 7f6713fff640 1 --2- 172.21.15.83:0/2256264443 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f671406f510 0x7f67141e5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.135+0000 7f6718889640 1 --2- 172.21.15.83:0/2256264443 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f671406b4f0 0x7f671415b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.135+0000 7f671908a640 1 -- 172.21.15.83:0/2256264443 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f671406b4f0 msgr2=0x7f671415b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.135+0000 7f671908a640 1 --2- 172.21.15.83:0/2256264443 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f671406b4f0 0x7f671415b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.135+0000 7f671908a640 1 -- 172.21.15.83:0/2256264443 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f671406f510 msgr2=0x7f67141e5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.135+0000 7f671908a640 1 --2- 172.21.15.83:0/2256264443 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f671406f510 0x7f67141e5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.135+0000 7f671908a640 1 -- 172.21.15.83:0/2256264443 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f671413e6a0 con 0x7f6714145590 2024-03-06T07:22:30.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.135+0000 7f6713fff640 1 --2- 172.21.15.83:0/2256264443 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f671406f510 0x7f67141e5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:30.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.136+0000 7f671908a640 1 --2- 172.21.15.83:0/2256264443 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6714145590 0x7f67140b4f80 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f670c007e40 tx=0x7f670c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.136+0000 7f67117fa640 1 -- 172.21.15.83:0/2256264443 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670c014620 con 0x7f6714145590 2024-03-06T07:22:30.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.136+0000 7f67117fa640 1 -- 172.21.15.83:0/2256264443 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f670c019070 con 0x7f6714145590 2024-03-06T07:22:30.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.136+0000 7f67117fa640 1 -- 172.21.15.83:0/2256264443 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670c014a20 con 0x7f6714145590 2024-03-06T07:22:30.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.136+0000 7f671aaed640 1 -- 172.21.15.83:0/2256264443 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67141e68e0 con 0x7f6714145590 2024-03-06T07:22:30.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.136+0000 7f671aaed640 1 -- 172.21.15.83:0/2256264443 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67141e6ea0 con 0x7f6714145590 2024-03-06T07:22:30.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.137+0000 7f671aaed640 1 -- 172.21.15.83:0/2256264443 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67140733b0 con 0x7f6714145590 2024-03-06T07:22:30.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.137+0000 7f67117fa640 1 -- 172.21.15.83:0/2256264443 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f670c014bc0 con 0x7f6714145590 2024-03-06T07:22:30.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.138+0000 7f67117fa640 1 --2- 172.21.15.83:0/2256264443 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f66f406b4d0 0x7f66f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.138+0000 7f67117fa640 1 -- 172.21.15.83:0/2256264443 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f670c08ab70 con 0x7f6714145590 2024-03-06T07:22:30.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.138+0000 7f6718889640 1 --2- 172.21.15.83:0/2256264443 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f66f406b4d0 0x7f66f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.138+0000 7f6718889640 1 --2- 172.21.15.83:0/2256264443 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f66f406b4d0 0x7f66f406d990 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6704036000 tx=0x7f670403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.142+0000 7f67117fa640 1 -- 172.21.15.83:0/2256264443 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f670c059970 con 0x7f6714145590 2024-03-06T07:22:30.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.346+0000 7f671aaed640 1 -- 172.21.15.83:0/2256264443 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6714068ff0 con 0x7f6714145590 2024-03-06T07:22:30.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.346+0000 7f67117fa640 1 -- 172.21.15.83:0/2256264443 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f670c059270 con 0x7f6714145590 2024-03-06T07:22:30.348 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:22:30.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.349+0000 7f671aaed640 1 -- 172.21.15.83:0/2256264443 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f66f406b4d0 msgr2=0x7f66f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.349+0000 7f671aaed640 1 --2- 172.21.15.83:0/2256264443 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f66f406b4d0 0x7f66f406d990 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6704036000 tx=0x7f670403c040 comp rx=0 tx=0).stop 2024-03-06T07:22:30.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.349+0000 7f671aaed640 1 -- 172.21.15.83:0/2256264443 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6714145590 msgr2=0x7f67140b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.349+0000 7f671aaed640 1 --2- 172.21.15.83:0/2256264443 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6714145590 0x7f67140b4f80 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f670c007e40 tx=0x7f670c00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.349+0000 7f671aaed640 1 -- 172.21.15.83:0/2256264443 shutdown_connections 2024-03-06T07:22:30.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.349+0000 7f671aaed640 1 --2- 172.21.15.83:0/2256264443 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f66f406b4d0 0x7f66f406d990 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.349+0000 7f671aaed640 1 --2- 172.21.15.83:0/2256264443 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6714145590 0x7f67140b4f80 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.349+0000 7f671aaed640 1 --2- 172.21.15.83:0/2256264443 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f671406f510 0x7f67141e5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.349+0000 7f671aaed640 1 --2- 172.21.15.83:0/2256264443 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f671406b4f0 0x7f671415b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.349+0000 7f671aaed640 1 -- 172.21.15.83:0/2256264443 >> 172.21.15.83:0/2256264443 conn(0x7f67140b2660 msgr2=0x7f67140d0860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:30.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.349+0000 7f671aaed640 1 -- 172.21.15.83:0/2256264443 shutdown_connections 2024-03-06T07:22:30.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.349+0000 7f671aaed640 1 -- 172.21.15.83:0/2256264443 wait complete. 2024-03-06T07:22:30.361 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:30.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.462+0000 7f823b6d6640 1 Processor -- start 2024-03-06T07:22:30.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.463+0000 7f823b6d6640 1 -- start start 2024-03-06T07:22:30.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.464+0000 7f823b6d6640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f82341614d0 0x7f82340d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.464+0000 7f823b6d6640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f82340d2480 0x7f82340b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.464+0000 7f823b6d6640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82341618b0 con 0x7f82340d1910 2024-03-06T07:22:30.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.464+0000 7f823b6d6640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82340cf400 con 0x7f82340d2480 2024-03-06T07:22:30.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.464+0000 7f823b6d6640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82340cf6a0 con 0x7f82341614d0 2024-03-06T07:22:30.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.464+0000 7f8239c73640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f82340d2480 0x7f82340b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.464+0000 7f8239472640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f82341614d0 0x7f82340d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.464+0000 7f8238c71640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f82340d1910 0x7f82340d1d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:54942/0 (socket says 172.21.15.83:54942) 2024-03-06T07:22:30.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.464+0000 7f8238c71640 1 -- 172.21.15.83:0/1577064153 learned_addr learned my addr 172.21.15.83:0/1577064153 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:30.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.465+0000 7f8239472640 1 -- 172.21.15.83:0/1577064153 >> v2:172.21.15.183:3300/0 conn(0x7f82340d2480 msgr2=0x7f82340b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:30.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.465+0000 7f8239472640 1 --2- 172.21.15.83:0/1577064153 >> v2:172.21.15.183:3300/0 conn(0x7f82340d2480 0x7f82340b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.465+0000 7f8239472640 1 -- 172.21.15.83:0/1577064153 >> v1:172.21.15.183:6789/0 conn(0x7f82340d1910 legacy=0x7f82340d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.465+0000 7f8239472640 1 -- 172.21.15.83:0/1577064153 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82340cf980 con 0x7f82341614d0 2024-03-06T07:22:30.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.465+0000 7f8239c73640 1 --2- 172.21.15.83:0/1577064153 >> v2:172.21.15.183:3300/0 conn(0x7f82340d2480 0x7f82340b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:30.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.466+0000 7f8239472640 1 --2- 172.21.15.83:0/1577064153 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f82341614d0 0x7f82340d4860 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f822c0097a0 tx=0x7f822c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5e8ef5067ce789e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.466+0000 7f8233fff640 1 -- 172.21.15.83:0/1577064153 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f822c035020 con 0x7f82341614d0 2024-03-06T07:22:30.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.466+0000 7f8233fff640 1 -- 172.21.15.83:0/1577064153 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f822c031830 con 0x7f82341614d0 2024-03-06T07:22:30.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.466+0000 7f8233fff640 1 -- 172.21.15.83:0/1577064153 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f822c031c00 con 0x7f82341614d0 2024-03-06T07:22:30.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.466+0000 7f823b6d6640 1 -- 172.21.15.83:0/1577064153 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f82341614d0 msgr2=0x7f82340d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.466+0000 7f823b6d6640 1 --2- 172.21.15.83:0/1577064153 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f82341614d0 0x7f82340d4860 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f822c0097a0 tx=0x7f822c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.467+0000 7f823b6d6640 1 -- 172.21.15.83:0/1577064153 shutdown_connections 2024-03-06T07:22:30.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.467+0000 7f823b6d6640 1 --2- 172.21.15.83:0/1577064153 >> v2:172.21.15.183:3300/0 conn(0x7f82340d2480 0x7f82340b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.467+0000 7f823b6d6640 1 --2- 172.21.15.83:0/1577064153 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f82341614d0 0x7f82340d4860 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.467+0000 7f823b6d6640 1 -- 172.21.15.83:0/1577064153 >> 172.21.15.83:0/1577064153 conn(0x7f82340d3d40 msgr2=0x7f82340d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:30.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.467+0000 7f823b6d6640 1 -- 172.21.15.83:0/1577064153 shutdown_connections 2024-03-06T07:22:30.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.467+0000 7f823b6d6640 1 -- 172.21.15.83:0/1577064153 wait complete. 2024-03-06T07:22:30.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.468+0000 7f823b6d6640 1 Processor -- start 2024-03-06T07:22:30.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.468+0000 7f823b6d6640 1 -- start start 2024-03-06T07:22:30.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.470+0000 7f823b6d6640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82340d1910 0x7f82340b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.470+0000 7f823b6d6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82340d2480 0x7f82341f3e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.470+0000 7f823b6d6640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f82341614d0 0x7f82340b8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.470+0000 7f823b6d6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82340b1370 con 0x7f82340d2480 2024-03-06T07:22:30.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.470+0000 7f823b6d6640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82340b0910 con 0x7f82340d1910 2024-03-06T07:22:30.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.470+0000 7f823b6d6640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82340b14f0 con 0x7f82341614d0 2024-03-06T07:22:30.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.470+0000 7f8238c71640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82340d2480 0x7f82341f3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.470+0000 7f8238c71640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82340d2480 0x7f82341f3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48600/0 (socket says 172.21.15.83:48600) 2024-03-06T07:22:30.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.470+0000 7f8238c71640 1 -- 172.21.15.83:0/3638260052 learned_addr learned my addr 172.21.15.83:0/3638260052 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:30.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.470+0000 7f8239472640 1 --2- 172.21.15.83:0/3638260052 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82340d1910 0x7f82340b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.470+0000 7f8239c73640 1 --2- 172.21.15.83:0/3638260052 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f82341614d0 0x7f82340b8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.470+0000 7f8238c71640 1 -- 172.21.15.83:0/3638260052 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f82341614d0 msgr2=0x7f82340b8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.470+0000 7f8238c71640 1 --2- 172.21.15.83:0/3638260052 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f82341614d0 0x7f82340b8fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.470+0000 7f8238c71640 1 -- 172.21.15.83:0/3638260052 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82340d1910 msgr2=0x7f82340b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.470+0000 7f8238c71640 1 --2- 172.21.15.83:0/3638260052 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82340d1910 0x7f82340b68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.470+0000 7f8238c71640 1 -- 172.21.15.83:0/3638260052 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82340b7a30 con 0x7f82340d2480 2024-03-06T07:22:30.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.470+0000 7f8239472640 1 --2- 172.21.15.83:0/3638260052 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82340d1910 0x7f82340b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:30.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.470+0000 7f8239c73640 1 --2- 172.21.15.83:0/3638260052 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f82341614d0 0x7f82340b8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:30.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.470+0000 7f8238c71640 1 --2- 172.21.15.83:0/3638260052 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82340d2480 0x7f82341f3e30 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f822800cb10 tx=0x7f822800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.470+0000 7f8231ffb640 1 -- 172.21.15.83:0/3638260052 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8228017070 con 0x7f82340d2480 2024-03-06T07:22:30.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.470+0000 7f8231ffb640 1 -- 172.21.15.83:0/3638260052 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f822800ac70 con 0x7f82340d2480 2024-03-06T07:22:30.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.470+0000 7f823b6d6640 1 -- 172.21.15.83:0/3638260052 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82340b7750 con 0x7f82340d2480 2024-03-06T07:22:30.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.470+0000 7f8231ffb640 1 -- 172.21.15.83:0/3638260052 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f822800ae60 con 0x7f82340d2480 2024-03-06T07:22:30.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.471+0000 7f823b6d6640 1 -- 172.21.15.83:0/3638260052 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8234159dd0 con 0x7f82340d2480 2024-03-06T07:22:30.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.472+0000 7f823b6d6640 1 -- 172.21.15.83:0/3638260052 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82340ce170 con 0x7f82340d2480 2024-03-06T07:22:30.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.472+0000 7f8231ffb640 1 -- 172.21.15.83:0/3638260052 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f822801a070 con 0x7f82340d2480 2024-03-06T07:22:30.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.472+0000 7f8231ffb640 1 --2- 172.21.15.83:0/3638260052 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f822006b4d0 0x7f822006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.472+0000 7f8231ffb640 1 -- 172.21.15.83:0/3638260052 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f822808a030 con 0x7f82340d2480 2024-03-06T07:22:30.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.472+0000 7f8239472640 1 --2- 172.21.15.83:0/3638260052 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f822006b4d0 0x7f822006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.473+0000 7f8239472640 1 --2- 172.21.15.83:0/3638260052 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f822006b4d0 0x7f822006d990 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f822c036000 tx=0x7f822c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.477+0000 7f8231ffb640 1 -- 172.21.15.83:0/3638260052 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8228058e30 con 0x7f82340d2480 2024-03-06T07:22:30.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.664+0000 7f823b6d6640 1 -- 172.21.15.83:0/3638260052 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8234068ff0 con 0x7f82340d2480 2024-03-06T07:22:30.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.664+0000 7f8231ffb640 1 -- 172.21.15.83:0/3638260052 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8228058730 con 0x7f82340d2480 2024-03-06T07:22:30.667 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-03-06T07:22:30.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.667+0000 7f823b6d6640 1 -- 172.21.15.83:0/3638260052 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f822006b4d0 msgr2=0x7f822006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.667+0000 7f823b6d6640 1 --2- 172.21.15.83:0/3638260052 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f822006b4d0 0x7f822006d990 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f822c036000 tx=0x7f822c008030 comp rx=0 tx=0).stop 2024-03-06T07:22:30.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.667+0000 7f823b6d6640 1 -- 172.21.15.83:0/3638260052 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82340d2480 msgr2=0x7f82341f3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.667+0000 7f823b6d6640 1 --2- 172.21.15.83:0/3638260052 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82340d2480 0x7f82341f3e30 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f822800cb10 tx=0x7f822800a060 comp rx=0 tx=0).stop 2024-03-06T07:22:30.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.667+0000 7f823b6d6640 1 -- 172.21.15.83:0/3638260052 shutdown_connections 2024-03-06T07:22:30.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.667+0000 7f823b6d6640 1 --2- 172.21.15.83:0/3638260052 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f822006b4d0 0x7f822006d990 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.667+0000 7f823b6d6640 1 --2- 172.21.15.83:0/3638260052 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f82341614d0 0x7f82340b8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.667+0000 7f823b6d6640 1 --2- 172.21.15.83:0/3638260052 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82340d2480 0x7f82341f3e30 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.667+0000 7f823b6d6640 1 --2- 172.21.15.83:0/3638260052 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82340d1910 0x7f82340b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.667+0000 7f823b6d6640 1 -- 172.21.15.83:0/3638260052 >> 172.21.15.83:0/3638260052 conn(0x7f82340d3d40 msgr2=0x7f823406cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:30.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.667+0000 7f823b6d6640 1 -- 172.21.15.83:0/3638260052 shutdown_connections 2024-03-06T07:22:30.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.667+0000 7f823b6d6640 1 -- 172.21.15.83:0/3638260052 wait complete. 2024-03-06T07:22:30.680 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:30.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.814+0000 7f444ea4e640 1 Processor -- start 2024-03-06T07:22:30.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.814+0000 7f444ea4e640 1 -- start start 2024-03-06T07:22:30.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.815+0000 7f444ea4e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f444815c050 0x7f444806f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.815+0000 7f444ea4e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f44480d0af0 0x7f44480cce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.815+0000 7f444ea4e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44480cd4b0 con 0x7f444806fb40 2024-03-06T07:22:30.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.815+0000 7f444ea4e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f444806ff80 con 0x7f44480d0af0 2024-03-06T07:22:30.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.815+0000 7f444ea4e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44480cd790 con 0x7f444815c050 2024-03-06T07:22:30.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.815+0000 7f444cfeb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f44480d0af0 0x7f44480cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.815+0000 7f444cfeb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f44480d0af0 0x7f44480cce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48606/0 (socket says 172.21.15.83:48606) 2024-03-06T07:22:30.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.815+0000 7f444cfeb640 1 -- 172.21.15.83:0/3524936658 learned_addr learned my addr 172.21.15.83:0/3524936658 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:30.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.815+0000 7f4447fff640 1 --2- 172.21.15.83:0/3524936658 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f444815c050 0x7f444806f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.815+0000 7f444cfeb640 1 -- 172.21.15.83:0/3524936658 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f444815c050 msgr2=0x7f444806f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:30.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.815+0000 7f444cfeb640 1 --2- 172.21.15.83:0/3524936658 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f444815c050 0x7f444806f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.815+0000 7f444cfeb640 1 -- 172.21.15.83:0/3524936658 >> v1:172.21.15.183:6789/0 conn(0x7f444806fb40 legacy=0x7f44480cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:30.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.815+0000 7f444cfeb640 1 -- 172.21.15.83:0/3524936658 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44480cda70 con 0x7f44480d0af0 2024-03-06T07:22:30.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.816+0000 7f444cfeb640 1 --2- 172.21.15.83:0/3524936658 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44480d0af0 0x7f44480cce70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4440009e50 tx=0x7f444002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c077d464f465cd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.816+0000 7f4446ffd640 1 -- 172.21.15.83:0/3524936658 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444002c9e0 con 0x7f44480d0af0 2024-03-06T07:22:30.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.816+0000 7f4446ffd640 1 -- 172.21.15.83:0/3524936658 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f444002cb80 con 0x7f44480d0af0 2024-03-06T07:22:30.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.816+0000 7f4446ffd640 1 -- 172.21.15.83:0/3524936658 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44400343f0 con 0x7f44480d0af0 2024-03-06T07:22:30.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.817+0000 7f444ea4e640 1 -- 172.21.15.83:0/3524936658 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44480d0af0 msgr2=0x7f44480cce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.817+0000 7f444ea4e640 1 --2- 172.21.15.83:0/3524936658 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44480d0af0 0x7f44480cce70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4440009e50 tx=0x7f444002be40 comp rx=0 tx=0).stop 2024-03-06T07:22:30.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.817+0000 7f444ea4e640 1 -- 172.21.15.83:0/3524936658 shutdown_connections 2024-03-06T07:22:30.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.817+0000 7f444ea4e640 1 --2- 172.21.15.83:0/3524936658 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44480d0af0 0x7f44480cce70 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.817+0000 7f444ea4e640 1 --2- 172.21.15.83:0/3524936658 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f444815c050 0x7f444806f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.817+0000 7f444ea4e640 1 -- 172.21.15.83:0/3524936658 >> 172.21.15.83:0/3524936658 conn(0x7f44480b3800 msgr2=0x7f44480b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:30.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.817+0000 7f444ea4e640 1 -- 172.21.15.83:0/3524936658 shutdown_connections 2024-03-06T07:22:30.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.817+0000 7f444ea4e640 1 -- 172.21.15.83:0/3524936658 wait complete. 2024-03-06T07:22:30.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.818+0000 7f444ea4e640 1 Processor -- start 2024-03-06T07:22:30.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.818+0000 7f444ea4e640 1 -- start start 2024-03-06T07:22:30.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.819+0000 7f444ea4e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f444806fb40 0x7f44481f5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.819+0000 7f444ea4e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44480d0af0 0x7f444809f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.819+0000 7f444ea4e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f444815c050 0x7f44480b4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.819+0000 7f444ea4e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4448165c30 con 0x7f444806fb40 2024-03-06T07:22:30.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.819+0000 7f444ea4e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4448165ab0 con 0x7f444815c050 2024-03-06T07:22:30.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.819+0000 7f444ea4e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4448165db0 con 0x7f44480d0af0 2024-03-06T07:22:30.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.819+0000 7f4447fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f444806fb40 0x7f44481f5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.819+0000 7f4447fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f444806fb40 0x7f44481f5170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48612/0 (socket says 172.21.15.83:48612) 2024-03-06T07:22:30.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.819+0000 7f4447fff640 1 -- 172.21.15.83:0/2448842341 learned_addr learned my addr 172.21.15.83:0/2448842341 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:30.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.819+0000 7f4447fff640 1 -- 172.21.15.83:0/2448842341 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44480d0af0 msgr2=0x7f444809f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.819+0000 7f4447fff640 1 --2- 172.21.15.83:0/2448842341 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44480d0af0 0x7f444809f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.819+0000 7f4447fff640 1 -- 172.21.15.83:0/2448842341 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f444815c050 msgr2=0x7f44480b4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.819+0000 7f444cfeb640 1 --2- 172.21.15.83:0/2448842341 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f444815c050 0x7f44480b4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.819+0000 7f4447fff640 1 --2- 172.21.15.83:0/2448842341 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f444815c050 0x7f44480b4fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.819+0000 7f4447fff640 1 -- 172.21.15.83:0/2448842341 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f444809f930 con 0x7f444806fb40 2024-03-06T07:22:30.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.819+0000 7f4447fff640 1 --2- 172.21.15.83:0/2448842341 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f444806fb40 0x7f44481f5170 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f443c00d5f0 tx=0x7f443c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.819+0000 7f4444ff9640 1 -- 172.21.15.83:0/2448842341 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443c0125f0 con 0x7f444806fb40 2024-03-06T07:22:30.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.819+0000 7f4444ff9640 1 -- 172.21.15.83:0/2448842341 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f443c019070 con 0x7f444806fb40 2024-03-06T07:22:30.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.819+0000 7f4444ff9640 1 -- 172.21.15.83:0/2448842341 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443c0129d0 con 0x7f444806fb40 2024-03-06T07:22:30.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.819+0000 7f444ea4e640 1 -- 172.21.15.83:0/2448842341 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44480b5500 con 0x7f444806fb40 2024-03-06T07:22:30.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.820+0000 7f444ea4e640 1 -- 172.21.15.83:0/2448842341 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44480b5a80 con 0x7f444806fb40 2024-03-06T07:22:30.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.821+0000 7f444ea4e640 1 -- 172.21.15.83:0/2448842341 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44480b6420 con 0x7f444806fb40 2024-03-06T07:22:30.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.821+0000 7f4444ff9640 1 -- 172.21.15.83:0/2448842341 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f443c012b70 con 0x7f444806fb40 2024-03-06T07:22:30.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.822+0000 7f4444ff9640 1 --2- 172.21.15.83:0/2448842341 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f443406b500 0x7f443406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.822+0000 7f44477fe640 1 --2- 172.21.15.83:0/2448842341 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f443406b500 0x7f443406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.822+0000 7f4444ff9640 1 -- 172.21.15.83:0/2448842341 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f443c08a5c0 con 0x7f444806fb40 2024-03-06T07:22:30.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.822+0000 7f44477fe640 1 --2- 172.21.15.83:0/2448842341 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f443406b500 0x7f443406d9c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4438006fd0 tx=0x7f4438006bc0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:30.826+0000 7f4444ff9640 1 -- 172.21.15.83:0/2448842341 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f44480b6420 con 0x7f444806fb40 2024-03-06T07:22:31.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.011+0000 7f444ea4e640 1 -- 172.21.15.83:0/2448842341 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f444806f600 con 0x7f444806fb40 2024-03-06T07:22:31.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.011+0000 7f4444ff9640 1 -- 172.21.15.83:0/2448842341 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f444806f600 con 0x7f444806fb40 2024-03-06T07:22:31.015 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 64 2024-03-06T07:22:31.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.014+0000 7f444ea4e640 1 -- 172.21.15.83:0/2448842341 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f443406b500 msgr2=0x7f443406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.014+0000 7f444ea4e640 1 --2- 172.21.15.83:0/2448842341 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f443406b500 0x7f443406d9c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4438006fd0 tx=0x7f4438006bc0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.014+0000 7f444ea4e640 1 -- 172.21.15.83:0/2448842341 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f444806fb40 msgr2=0x7f44481f5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.014+0000 7f444ea4e640 1 --2- 172.21.15.83:0/2448842341 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f444806fb40 0x7f44481f5170 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f443c00d5f0 tx=0x7f443c00dab0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.014+0000 7f444ea4e640 1 -- 172.21.15.83:0/2448842341 shutdown_connections 2024-03-06T07:22:31.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.014+0000 7f444ea4e640 1 --2- 172.21.15.83:0/2448842341 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f443406b500 0x7f443406d9c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.014+0000 7f444ea4e640 1 --2- 172.21.15.83:0/2448842341 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f444815c050 0x7f44480b4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.014+0000 7f444ea4e640 1 --2- 172.21.15.83:0/2448842341 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44480d0af0 0x7f444809f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.014+0000 7f444ea4e640 1 --2- 172.21.15.83:0/2448842341 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f444806fb40 0x7f44481f5170 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.014+0000 7f444ea4e640 1 -- 172.21.15.83:0/2448842341 >> 172.21.15.83:0/2448842341 conn(0x7f44480b3800 msgr2=0x7f44480cc720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:31.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.014+0000 7f444ea4e640 1 -- 172.21.15.83:0/2448842341 shutdown_connections 2024-03-06T07:22:31.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.014+0000 7f444ea4e640 1 -- 172.21.15.83:0/2448842341 wait complete. 2024-03-06T07:22:31.031 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dir_deletion' 2024-03-06T07:22:31.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.129+0000 7f2b1520e640 1 Processor -- start 2024-03-06T07:22:31.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.130+0000 7f2b1520e640 1 -- start start 2024-03-06T07:22:31.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.130+0000 7f2b1520e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2b1015b7a0 0x7f2b100b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.130+0000 7f2b1520e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f2b10070930 0x7f2b100b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.130+0000 7f2b1520e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b10070dc0 con 0x7f2b10144640 2024-03-06T07:22:31.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.130+0000 7f2b1520e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b100b4fe0 con 0x7f2b1015b7a0 2024-03-06T07:22:31.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.130+0000 7f2b1520e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b100ce5d0 con 0x7f2b10070930 2024-03-06T07:22:31.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.130+0000 7f2b0e59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2b1015b7a0 0x7f2b100b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.131+0000 7f2b0ed9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f2b10144640 0x7f2b1015b090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39420/0 (socket says 172.21.15.83:39420) 2024-03-06T07:22:31.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.131+0000 7f2b0ed9d640 1 -- 172.21.15.83:0/2344442361 learned_addr learned my addr 172.21.15.83:0/2344442361 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:31.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.131+0000 7f2b0f59e640 1 --2- 172.21.15.83:0/2344442361 >> v2:172.21.15.183:3300/0 conn(0x7f2b10070930 0x7f2b100b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.131+0000 7f2b0e59c640 1 -- 172.21.15.83:0/2344442361 >> v2:172.21.15.183:3300/0 conn(0x7f2b10070930 msgr2=0x7f2b100b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:31.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.131+0000 7f2b0e59c640 1 --2- 172.21.15.83:0/2344442361 >> v2:172.21.15.183:3300/0 conn(0x7f2b10070930 0x7f2b100b4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.131+0000 7f2b0e59c640 1 -- 172.21.15.83:0/2344442361 >> v1:172.21.15.83:6789/0 conn(0x7f2b10144640 legacy=0x7f2b1015b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.131+0000 7f2b0e59c640 1 -- 172.21.15.83:0/2344442361 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b100b3bb0 con 0x7f2b1015b7a0 2024-03-06T07:22:31.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.131+0000 7f2b0dd9b640 1 -- 172.21.15.83:0/2344442361 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3463704483 0 0) 0x7f2b10070dc0 con 0x7f2b10144640 2024-03-06T07:22:31.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.132+0000 7f2b0e59c640 1 --2- 172.21.15.83:0/2344442361 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b1015b7a0 0x7f2b100b34d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2b1006cc50 tx=0x7f2b0402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fc389babe01e9e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.132+0000 7f2b0dd9b640 1 -- 172.21.15.83:0/2344442361 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b040315a0 con 0x7f2b1015b7a0 2024-03-06T07:22:31.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.132+0000 7f2b0dd9b640 1 -- 172.21.15.83:0/2344442361 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b04037070 con 0x7f2b1015b7a0 2024-03-06T07:22:31.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.132+0000 7f2b0dd9b640 1 -- 172.21.15.83:0/2344442361 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b04031960 con 0x7f2b1015b7a0 2024-03-06T07:22:31.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.132+0000 7f2b1520e640 1 -- 172.21.15.83:0/2344442361 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b1015b7a0 msgr2=0x7f2b100b34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.132+0000 7f2b1520e640 1 --2- 172.21.15.83:0/2344442361 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b1015b7a0 0x7f2b100b34d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2b1006cc50 tx=0x7f2b0402b840 comp rx=0 tx=0).stop 2024-03-06T07:22:31.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.132+0000 7f2b1520e640 1 -- 172.21.15.83:0/2344442361 shutdown_connections 2024-03-06T07:22:31.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.132+0000 7f2b1520e640 1 --2- 172.21.15.83:0/2344442361 >> v2:172.21.15.183:3300/0 conn(0x7f2b10070930 0x7f2b100b4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.132+0000 7f2b1520e640 1 --2- 172.21.15.83:0/2344442361 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b1015b7a0 0x7f2b100b34d0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.132+0000 7f2b1520e640 1 -- 172.21.15.83:0/2344442361 >> 172.21.15.83:0/2344442361 conn(0x7f2b1006b3d0 msgr2=0x7f2b1006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:31.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.132+0000 7f2b1520e640 1 -- 172.21.15.83:0/2344442361 shutdown_connections 2024-03-06T07:22:31.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.132+0000 7f2b1520e640 1 -- 172.21.15.83:0/2344442361 wait complete. 2024-03-06T07:22:31.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.134+0000 7f2b1520e640 1 Processor -- start 2024-03-06T07:22:31.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.134+0000 7f2b1520e640 1 -- start start 2024-03-06T07:22:31.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.134+0000 7f2b1520e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b10070930 0x7f2b1015a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.134+0000 7f2b1520e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b10144640 0x7f2b100ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.134+0000 7f2b1520e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2b1015b7a0 0x7f2b10142030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.134+0000 7f2b1520e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b100cdee0 con 0x7f2b10144640 2024-03-06T07:22:31.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.134+0000 7f2b1520e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b100ce260 con 0x7f2b10070930 2024-03-06T07:22:31.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.134+0000 7f2b1520e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b100cea60 con 0x7f2b1015b7a0 2024-03-06T07:22:31.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.135+0000 7f2b0e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b10144640 0x7f2b100ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.135+0000 7f2b0e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b10144640 0x7f2b100ad860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48640/0 (socket says 172.21.15.83:48640) 2024-03-06T07:22:31.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.135+0000 7f2b0e59c640 1 -- 172.21.15.83:0/2184973286 learned_addr learned my addr 172.21.15.83:0/2184973286 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:31.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.135+0000 7f2b0f59e640 1 --2- 172.21.15.83:0/2184973286 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2b1015b7a0 0x7f2b10142030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.135+0000 7f2b0ed9d640 1 --2- 172.21.15.83:0/2184973286 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b10070930 0x7f2b1015a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.135+0000 7f2b0e59c640 1 -- 172.21.15.83:0/2184973286 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2b1015b7a0 msgr2=0x7f2b10142030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.135+0000 7f2b0e59c640 1 --2- 172.21.15.83:0/2184973286 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2b1015b7a0 0x7f2b10142030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.135+0000 7f2b0e59c640 1 -- 172.21.15.83:0/2184973286 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b10070930 msgr2=0x7f2b1015a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.135+0000 7f2b0e59c640 1 --2- 172.21.15.83:0/2184973286 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b10070930 0x7f2b1015a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.135+0000 7f2b0e59c640 1 -- 172.21.15.83:0/2184973286 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b10142ad0 con 0x7f2b10144640 2024-03-06T07:22:31.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.135+0000 7f2b0f59e640 1 --2- 172.21.15.83:0/2184973286 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2b1015b7a0 0x7f2b10142030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:31.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.135+0000 7f2b0ed9d640 1 --2- 172.21.15.83:0/2184973286 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b10070930 0x7f2b1015a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:31.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.135+0000 7f2b0e59c640 1 --2- 172.21.15.83:0/2184973286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b10144640 0x7f2b100ad860 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f2b04000c00 tx=0x7f2b040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.135+0000 7f2af77fe640 1 -- 172.21.15.83:0/2184973286 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b04004230 con 0x7f2b10144640 2024-03-06T07:22:31.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.135+0000 7f2af77fe640 1 -- 172.21.15.83:0/2184973286 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b040043d0 con 0x7f2b10144640 2024-03-06T07:22:31.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.135+0000 7f2af77fe640 1 -- 172.21.15.83:0/2184973286 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b04003850 con 0x7f2b10144640 2024-03-06T07:22:31.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.135+0000 7f2b1520e640 1 -- 172.21.15.83:0/2184973286 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b10140670 con 0x7f2b10144640 2024-03-06T07:22:31.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.135+0000 7f2b1520e640 1 -- 172.21.15.83:0/2184973286 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b10140c30 con 0x7f2b10144640 2024-03-06T07:22:31.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.136+0000 7f2b1520e640 1 -- 172.21.15.83:0/2184973286 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b1004f330 con 0x7f2b10144640 2024-03-06T07:22:31.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.137+0000 7f2af77fe640 1 -- 172.21.15.83:0/2184973286 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2b040039f0 con 0x7f2b10144640 2024-03-06T07:22:31.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.137+0000 7f2af77fe640 1 --2- 172.21.15.83:0/2184973286 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2afc06b4d0 0x7f2afc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.137+0000 7f2af77fe640 1 -- 172.21.15.83:0/2184973286 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2b0402f030 con 0x7f2b10144640 2024-03-06T07:22:31.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.137+0000 7f2b0ed9d640 1 --2- 172.21.15.83:0/2184973286 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2afc06b4d0 0x7f2afc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.137+0000 7f2b0ed9d640 1 --2- 172.21.15.83:0/2184973286 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2afc06b4d0 0x7f2afc06d990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2b080012e0 tx=0x7f2b08007f90 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.141+0000 7f2af77fe640 1 -- 172.21.15.83:0/2184973286 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2b04041070 con 0x7f2b10144640 2024-03-06T07:22:31.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:31.384+0000 7f2b1520e640 1 -- 172.21.15.83:0/2184973286 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]} v 0) v1 -- 0x7f2b10072650 con 0x7f2b10144640 2024-03-06T07:22:32.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.076+0000 7f2af77fe640 1 -- 172.21.15.83:0/2184973286 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]}]=0 v56) v1 ==== 135+0+0 (secure 0 0 0) 0x7f2b0407af70 con 0x7f2b10144640 2024-03-06T07:22:32.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.078+0000 7f2b1520e640 1 -- 172.21.15.83:0/2184973286 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2afc06b4d0 msgr2=0x7f2afc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.078+0000 7f2b1520e640 1 --2- 172.21.15.83:0/2184973286 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2afc06b4d0 0x7f2afc06d990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2b080012e0 tx=0x7f2b08007f90 comp rx=0 tx=0).stop 2024-03-06T07:22:32.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.078+0000 7f2b1520e640 1 -- 172.21.15.83:0/2184973286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b10144640 msgr2=0x7f2b100ad860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.078+0000 7f2b1520e640 1 --2- 172.21.15.83:0/2184973286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b10144640 0x7f2b100ad860 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f2b04000c00 tx=0x7f2b040046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.078+0000 7f2b1520e640 1 -- 172.21.15.83:0/2184973286 shutdown_connections 2024-03-06T07:22:32.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.078+0000 7f2b1520e640 1 --2- 172.21.15.83:0/2184973286 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2afc06b4d0 0x7f2afc06d990 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.078+0000 7f2b1520e640 1 --2- 172.21.15.83:0/2184973286 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2b1015b7a0 0x7f2b10142030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.078+0000 7f2b1520e640 1 --2- 172.21.15.83:0/2184973286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b10144640 0x7f2b100ad860 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.078+0000 7f2b1520e640 1 --2- 172.21.15.83:0/2184973286 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b10070930 0x7f2b1015a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.078+0000 7f2b1520e640 1 -- 172.21.15.83:0/2184973286 >> 172.21.15.83:0/2184973286 conn(0x7f2b1006b3d0 msgr2=0x7f2b100af730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:32.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.078+0000 7f2b1520e640 1 -- 172.21.15.83:0/2184973286 shutdown_connections 2024-03-06T07:22:32.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.078+0000 7f2b1520e640 1 -- 172.21.15.83:0/2184973286 wait complete. 2024-03-06T07:22:32.092 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:22:32.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.193+0000 7f0ede9d7640 1 Processor -- start 2024-03-06T07:22:32.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.193+0000 7f0ede9d7640 1 -- start start 2024-03-06T07:22:32.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.193+0000 7f0ede9d7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0ed815c050 0x7f0ed806f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.193+0000 7f0ede9d7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0ed80d0af0 0x7f0ed80cce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.193+0000 7f0ede9d7640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ed80cd4b0 con 0x7f0ed806fb40 2024-03-06T07:22:32.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.193+0000 7f0ede9d7640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ed806ffb0 con 0x7f0ed815c050 2024-03-06T07:22:32.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.193+0000 7f0ede9d7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed80cd790 con 0x7f0ed80d0af0 2024-03-06T07:22:32.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.193+0000 7f0ed7fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0ed815c050 0x7f0ed806f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.193+0000 7f0ed7fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0ed815c050 0x7f0ed806f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48652/0 (socket says 172.21.15.83:48652) 2024-03-06T07:22:32.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.194+0000 7f0ed7fff640 1 -- 172.21.15.83:0/3274950317 learned_addr learned my addr 172.21.15.83:0/3274950317 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:32.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.194+0000 7f0edcf74640 1 --2- 172.21.15.83:0/3274950317 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0ed80d0af0 0x7f0ed80cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.194+0000 7f0ed7fff640 1 -- 172.21.15.83:0/3274950317 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0ed80d0af0 msgr2=0x7f0ed80cce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:32.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.194+0000 7f0ed7fff640 1 --2- 172.21.15.83:0/3274950317 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0ed80d0af0 0x7f0ed80cce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.194+0000 7f0ed7fff640 1 -- 172.21.15.83:0/3274950317 >> v1:172.21.15.183:6789/0 conn(0x7f0ed806fb40 legacy=0x7f0ed80cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.194+0000 7f0ed7fff640 1 -- 172.21.15.83:0/3274950317 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed80cda70 con 0x7f0ed815c050 2024-03-06T07:22:32.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.195+0000 7f0ed7fff640 1 --2- 172.21.15.83:0/3274950317 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ed815c050 0x7f0ed806f600 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0ec8009870 tx=0x7f0ec802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e4b6ad1f977097d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.195+0000 7f0ed6ffd640 1 -- 172.21.15.83:0/3274950317 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec8035020 con 0x7f0ed815c050 2024-03-06T07:22:32.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.195+0000 7f0ed6ffd640 1 -- 172.21.15.83:0/3274950317 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ec8031640 con 0x7f0ed815c050 2024-03-06T07:22:32.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.195+0000 7f0ed6ffd640 1 -- 172.21.15.83:0/3274950317 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec802d040 con 0x7f0ed815c050 2024-03-06T07:22:32.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.195+0000 7f0ede9d7640 1 -- 172.21.15.83:0/3274950317 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ed815c050 msgr2=0x7f0ed806f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.195+0000 7f0ede9d7640 1 --2- 172.21.15.83:0/3274950317 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ed815c050 0x7f0ed806f600 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0ec8009870 tx=0x7f0ec802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.195+0000 7f0ede9d7640 1 -- 172.21.15.83:0/3274950317 shutdown_connections 2024-03-06T07:22:32.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.195+0000 7f0ede9d7640 1 --2- 172.21.15.83:0/3274950317 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0ed80d0af0 0x7f0ed80cce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.195+0000 7f0ede9d7640 1 --2- 172.21.15.83:0/3274950317 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ed815c050 0x7f0ed806f600 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.195+0000 7f0ede9d7640 1 -- 172.21.15.83:0/3274950317 >> 172.21.15.83:0/3274950317 conn(0x7f0ed80b3800 msgr2=0x7f0ed80b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:32.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.195+0000 7f0ede9d7640 1 -- 172.21.15.83:0/3274950317 shutdown_connections 2024-03-06T07:22:32.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.195+0000 7f0ede9d7640 1 -- 172.21.15.83:0/3274950317 wait complete. 2024-03-06T07:22:32.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.196+0000 7f0ede9d7640 1 Processor -- start 2024-03-06T07:22:32.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.197+0000 7f0ede9d7640 1 -- start start 2024-03-06T07:22:32.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.197+0000 7f0ede9d7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0ed806fb40 0x7f0ed8152a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.197+0000 7f0ede9d7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ed80d0af0 0x7f0ed81543f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.197+0000 7f0ede9d7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0ed815c050 0x7f0ed8158990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.197+0000 7f0ede9d7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed8165c30 con 0x7f0ed80d0af0 2024-03-06T07:22:32.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.197+0000 7f0ede9d7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed8165ab0 con 0x7f0ed815c050 2024-03-06T07:22:32.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.197+0000 7f0ede9d7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed8165db0 con 0x7f0ed806fb40 2024-03-06T07:22:32.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.197+0000 7f0ed77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ed80d0af0 0x7f0ed81543f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.197+0000 7f0ed77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ed80d0af0 0x7f0ed81543f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48660/0 (socket says 172.21.15.83:48660) 2024-03-06T07:22:32.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.197+0000 7f0ed77fe640 1 -- 172.21.15.83:0/772720557 learned_addr learned my addr 172.21.15.83:0/772720557 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:32.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.198+0000 7f0ed7fff640 1 --2- 172.21.15.83:0/772720557 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0ed806fb40 0x7f0ed8152a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.198+0000 7f0edcf74640 1 --2- 172.21.15.83:0/772720557 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0ed815c050 0x7f0ed8158990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.198+0000 7f0ed77fe640 1 -- 172.21.15.83:0/772720557 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0ed806fb40 msgr2=0x7f0ed8152a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.198+0000 7f0ed77fe640 1 --2- 172.21.15.83:0/772720557 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0ed806fb40 0x7f0ed8152a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.198+0000 7f0ed77fe640 1 -- 172.21.15.83:0/772720557 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0ed815c050 msgr2=0x7f0ed8158990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.198+0000 7f0ed77fe640 1 --2- 172.21.15.83:0/772720557 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0ed815c050 0x7f0ed8158990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.198+0000 7f0ed77fe640 1 -- 172.21.15.83:0/772720557 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed8152fd0 con 0x7f0ed80d0af0 2024-03-06T07:22:32.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.198+0000 7f0ed77fe640 1 --2- 172.21.15.83:0/772720557 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ed80d0af0 0x7f0ed81543f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc00c3c0 tx=0x7f0ecc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.198+0000 7f0ed4ff9640 1 -- 172.21.15.83:0/772720557 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc0052d0 con 0x7f0ed80d0af0 2024-03-06T07:22:32.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.198+0000 7f0ed4ff9640 1 -- 172.21.15.83:0/772720557 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ecc005490 con 0x7f0ed80d0af0 2024-03-06T07:22:32.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.198+0000 7f0ed4ff9640 1 -- 172.21.15.83:0/772720557 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc007260 con 0x7f0ed80d0af0 2024-03-06T07:22:32.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.198+0000 7f0ede9d7640 1 -- 172.21.15.83:0/772720557 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ed80b6880 con 0x7f0ed80d0af0 2024-03-06T07:22:32.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.198+0000 7f0ede9d7640 1 -- 172.21.15.83:0/772720557 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ed80b6e40 con 0x7f0ed80d0af0 2024-03-06T07:22:32.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.199+0000 7f0ede9d7640 1 -- 172.21.15.83:0/772720557 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ed809fe60 con 0x7f0ed80d0af0 2024-03-06T07:22:32.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.199+0000 7f0ed4ff9640 1 -- 172.21.15.83:0/772720557 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0ecc007400 con 0x7f0ed80d0af0 2024-03-06T07:22:32.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.200+0000 7f0ed4ff9640 1 --2- 172.21.15.83:0/772720557 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0ec406b4d0 0x7f0ec406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.200+0000 7f0ed4ff9640 1 -- 172.21.15.83:0/772720557 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0ecc0895a0 con 0x7f0ed80d0af0 2024-03-06T07:22:32.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.200+0000 7f0ed7fff640 1 --2- 172.21.15.83:0/772720557 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0ec406b4d0 0x7f0ec406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.200+0000 7f0ed7fff640 1 --2- 172.21.15.83:0/772720557 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0ec406b4d0 0x7f0ec406d990 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0ec8036000 tx=0x7f0ec803c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.204+0000 7f0ed4ff9640 1 -- 172.21.15.83:0/772720557 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0ecc057e60 con 0x7f0ed80d0af0 2024-03-06T07:22:32.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.378+0000 7f0ede9d7640 1 -- 172.21.15.83:0/772720557 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f0ed8155380 con 0x7f0ed80d0af0 2024-03-06T07:22:32.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.381+0000 7f0ed4ff9640 1 -- 172.21.15.83:0/772720557 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0ecc05ce20 con 0x7f0ed80d0af0 2024-03-06T07:22:32.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.383+0000 7f0ede9d7640 1 -- 172.21.15.83:0/772720557 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0ec406b4d0 msgr2=0x7f0ec406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.383+0000 7f0ede9d7640 1 --2- 172.21.15.83:0/772720557 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0ec406b4d0 0x7f0ec406d990 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0ec8036000 tx=0x7f0ec803c040 comp rx=0 tx=0).stop 2024-03-06T07:22:32.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.383+0000 7f0ede9d7640 1 -- 172.21.15.83:0/772720557 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ed80d0af0 msgr2=0x7f0ed81543f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.383+0000 7f0ede9d7640 1 --2- 172.21.15.83:0/772720557 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ed80d0af0 0x7f0ed81543f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc00c3c0 tx=0x7f0ecc00c880 comp rx=0 tx=0).stop 2024-03-06T07:22:32.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.383+0000 7f0ede9d7640 1 -- 172.21.15.83:0/772720557 shutdown_connections 2024-03-06T07:22:32.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.383+0000 7f0ede9d7640 1 --2- 172.21.15.83:0/772720557 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0ec406b4d0 0x7f0ec406d990 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.383+0000 7f0ede9d7640 1 --2- 172.21.15.83:0/772720557 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0ed815c050 0x7f0ed8158990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.383+0000 7f0ede9d7640 1 --2- 172.21.15.83:0/772720557 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ed80d0af0 0x7f0ed81543f0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.383+0000 7f0ede9d7640 1 --2- 172.21.15.83:0/772720557 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0ed806fb40 0x7f0ed8152a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.383+0000 7f0ede9d7640 1 -- 172.21.15.83:0/772720557 >> 172.21.15.83:0/772720557 conn(0x7f0ed80b3800 msgr2=0x7f0ed815b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:32.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.383+0000 7f0ede9d7640 1 -- 172.21.15.83:0/772720557 shutdown_connections 2024-03-06T07:22:32.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.383+0000 7f0ede9d7640 1 -- 172.21.15.83:0/772720557 wait complete. 2024-03-06T07:22:32.397 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:22:32.397 DEBUG:teuthology.orchestra.run.smithi083:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:22:32.443 DEBUG:teuthology.orchestra.run.smithi183:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:22:32.496 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:22:32.496 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:22:32.496 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:32.512 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:22:32.512 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:32.567 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:32.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.708+0000 7fccef6d4640 1 Processor -- start 2024-03-06T07:22:32.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.708+0000 7fccef6d4640 1 -- start start 2024-03-06T07:22:32.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.708+0000 7fccef6d4640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fcce815bdd0 0x7fcce80c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.708+0000 7fccef6d4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcce80d16e0 0x7fcce80d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.708+0000 7fccef6d4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce80cebe0 con 0x7fcce815fa10 2024-03-06T07:22:32.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.709+0000 7fccef6d4640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcce80ce980 con 0x7fcce80d16e0 2024-03-06T07:22:32.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.709+0000 7fccef6d4640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcce80ceec0 con 0x7fcce815bdd0 2024-03-06T07:22:32.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.709+0000 7fccedc71640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcce80d16e0 0x7fcce80d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.709+0000 7fccedc71640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcce80d16e0 0x7fcce80d1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48664/0 (socket says 172.21.15.83:48664) 2024-03-06T07:22:32.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.709+0000 7fccedc71640 1 -- 172.21.15.83:0/121464910 learned_addr learned my addr 172.21.15.83:0/121464910 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:32.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.709+0000 7fccedc71640 1 -- 172.21.15.83:0/121464910 >> v2:172.21.15.183:3300/0 conn(0x7fcce815bdd0 msgr2=0x7fcce80c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:32.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.709+0000 7fccedc71640 1 --2- 172.21.15.83:0/121464910 >> v2:172.21.15.183:3300/0 conn(0x7fcce815bdd0 0x7fcce80c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.709+0000 7fccedc71640 1 -- 172.21.15.83:0/121464910 >> v1:172.21.15.183:6789/0 conn(0x7fcce815fa10 legacy=0x7fcce80ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:32.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.709+0000 7fccedc71640 1 -- 172.21.15.83:0/121464910 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce80cf1a0 con 0x7fcce80d16e0 2024-03-06T07:22:32.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.710+0000 7fccedc71640 1 --2- 172.21.15.83:0/121464910 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcce80d16e0 0x7fcce80d1af0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fcce80c5520 tx=0x7fcce002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62eadd694732690f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.710+0000 7fccdffff640 1 -- 172.21.15.83:0/121464910 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce002c9e0 con 0x7fcce80d16e0 2024-03-06T07:22:32.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.710+0000 7fccdffff640 1 -- 172.21.15.83:0/121464910 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcce002cb80 con 0x7fcce80d16e0 2024-03-06T07:22:32.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.710+0000 7fccdffff640 1 -- 172.21.15.83:0/121464910 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce00343f0 con 0x7fcce80d16e0 2024-03-06T07:22:32.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.710+0000 7fccef6d4640 1 -- 172.21.15.83:0/121464910 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcce80d16e0 msgr2=0x7fcce80d1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.710+0000 7fccef6d4640 1 --2- 172.21.15.83:0/121464910 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcce80d16e0 0x7fcce80d1af0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fcce80c5520 tx=0x7fcce002be40 comp rx=0 tx=0).stop 2024-03-06T07:22:32.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.710+0000 7fccef6d4640 1 -- 172.21.15.83:0/121464910 shutdown_connections 2024-03-06T07:22:32.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.710+0000 7fccef6d4640 1 --2- 172.21.15.83:0/121464910 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcce80d16e0 0x7fcce80d1af0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.710+0000 7fccef6d4640 1 --2- 172.21.15.83:0/121464910 >> v2:172.21.15.183:3300/0 conn(0x7fcce815bdd0 0x7fcce80c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.710+0000 7fccef6d4640 1 -- 172.21.15.83:0/121464910 >> 172.21.15.83:0/121464910 conn(0x7fcce809a9f0 msgr2=0x7fcce809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:32.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.712+0000 7fccef6d4640 1 -- 172.21.15.83:0/121464910 shutdown_connections 2024-03-06T07:22:32.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.712+0000 7fccef6d4640 1 -- 172.21.15.83:0/121464910 wait complete. 2024-03-06T07:22:32.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.713+0000 7fccef6d4640 1 Processor -- start 2024-03-06T07:22:32.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.713+0000 7fccef6d4640 1 -- start start 2024-03-06T07:22:32.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.713+0000 7fccef6d4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcce80d16e0 0x7fcce81f2290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.713+0000 7fccef6d4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcce815bdd0 0x7fcce80c9250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.713+0000 7fccef6d4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcce815fa10 0x7fcce80cf8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.713+0000 7fccef6d4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce80cc5d0 con 0x7fcce815bdd0 2024-03-06T07:22:32.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.713+0000 7fccef6d4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce80c5b50 con 0x7fcce80d16e0 2024-03-06T07:22:32.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.713+0000 7fccef6d4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcce815c1b0 con 0x7fcce815fa10 2024-03-06T07:22:32.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.713+0000 7fccecc6f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcce815bdd0 0x7fcce80c9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.714+0000 7fccecc6f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcce815bdd0 0x7fcce80c9250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48670/0 (socket says 172.21.15.83:48670) 2024-03-06T07:22:32.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.714+0000 7fccecc6f640 1 -- 172.21.15.83:0/4176276037 learned_addr learned my addr 172.21.15.83:0/4176276037 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:32.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.714+0000 7fcced470640 1 --2- 172.21.15.83:0/4176276037 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcce80d16e0 0x7fcce81f2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.714+0000 7fccedc71640 1 --2- 172.21.15.83:0/4176276037 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcce815fa10 0x7fcce80cf8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.714+0000 7fccecc6f640 1 -- 172.21.15.83:0/4176276037 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcce815fa10 msgr2=0x7fcce80cf8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.714+0000 7fccecc6f640 1 --2- 172.21.15.83:0/4176276037 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcce815fa10 0x7fcce80cf8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.714+0000 7fccecc6f640 1 -- 172.21.15.83:0/4176276037 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcce80d16e0 msgr2=0x7fcce81f2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.714+0000 7fccecc6f640 1 --2- 172.21.15.83:0/4176276037 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcce80d16e0 0x7fcce81f2290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.714+0000 7fccecc6f640 1 -- 172.21.15.83:0/4176276037 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce81f2b90 con 0x7fcce815bdd0 2024-03-06T07:22:32.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.714+0000 7fcced470640 1 --2- 172.21.15.83:0/4176276037 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcce80d16e0 0x7fcce81f2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:32.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.714+0000 7fccedc71640 1 --2- 172.21.15.83:0/4176276037 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcce815fa10 0x7fcce80cf8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:32.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.714+0000 7fccecc6f640 1 --2- 172.21.15.83:0/4176276037 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcce815bdd0 0x7fcce80c9250 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fccd800b3c0 tx=0x7fccd800b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.714+0000 7fccddffb640 1 -- 172.21.15.83:0/4176276037 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd8011830 con 0x7fcce815bdd0 2024-03-06T07:22:32.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.714+0000 7fccddffb640 1 -- 172.21.15.83:0/4176276037 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccd8016070 con 0x7fcce815bdd0 2024-03-06T07:22:32.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.714+0000 7fccef6d4640 1 -- 172.21.15.83:0/4176276037 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcce81f2e70 con 0x7fcce815bdd0 2024-03-06T07:22:32.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.714+0000 7fccddffb640 1 -- 172.21.15.83:0/4176276037 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd8011c50 con 0x7fcce815bdd0 2024-03-06T07:22:32.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.714+0000 7fccef6d4640 1 -- 172.21.15.83:0/4176276037 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcce81f3710 con 0x7fcce815bdd0 2024-03-06T07:22:32.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.715+0000 7fccef6d4640 1 -- 172.21.15.83:0/4176276037 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcce81f3040 con 0x7fcce815bdd0 2024-03-06T07:22:32.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.715+0000 7fccddffb640 1 -- 172.21.15.83:0/4176276037 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fccd801b460 con 0x7fcce815bdd0 2024-03-06T07:22:32.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.715+0000 7fccddffb640 1 --2- 172.21.15.83:0/4176276037 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcccc06b4d0 0x7fcccc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.715+0000 7fccddffb640 1 -- 172.21.15.83:0/4176276037 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fccd8089d90 con 0x7fcce815bdd0 2024-03-06T07:22:32.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.715+0000 7fcced470640 1 --2- 172.21.15.83:0/4176276037 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcccc06b4d0 0x7fcccc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.715+0000 7fcced470640 1 --2- 172.21.15.83:0/4176276037 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcccc06b4d0 0x7fcccc06d990 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fcce400a870 tx=0x7fcce4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.724+0000 7fccddffb640 1 -- 172.21.15.83:0/4176276037 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fccd8020020 con 0x7fcce815bdd0 2024-03-06T07:22:32.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.903+0000 7fccef6d4640 1 -- 172.21.15.83:0/4176276037 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcce804f330 con 0x7fcce815bdd0 2024-03-06T07:22:32.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.904+0000 7fccddffb640 1 -- 172.21.15.83:0/4176276037 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16617 (secure 0 0 0) 0x7fccd8058b80 con 0x7fcce815bdd0 2024-03-06T07:22:32.906 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:22:32.907 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":20,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:21:59.938137+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.318942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.254795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:32.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.906+0000 7fccef6d4640 1 -- 172.21.15.83:0/4176276037 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcccc06b4d0 msgr2=0x7fcccc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.906+0000 7fccef6d4640 1 --2- 172.21.15.83:0/4176276037 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcccc06b4d0 0x7fcccc06d990 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fcce400a870 tx=0x7fcce4008040 comp rx=0 tx=0).stop 2024-03-06T07:22:32.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.906+0000 7fccef6d4640 1 -- 172.21.15.83:0/4176276037 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcce815bdd0 msgr2=0x7fcce80c9250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.906+0000 7fccef6d4640 1 --2- 172.21.15.83:0/4176276037 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcce815bdd0 0x7fcce80c9250 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fccd800b3c0 tx=0x7fccd800b880 comp rx=0 tx=0).stop 2024-03-06T07:22:32.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.906+0000 7fccef6d4640 1 -- 172.21.15.83:0/4176276037 shutdown_connections 2024-03-06T07:22:32.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.906+0000 7fccef6d4640 1 --2- 172.21.15.83:0/4176276037 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcccc06b4d0 0x7fcccc06d990 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.906+0000 7fccef6d4640 1 --2- 172.21.15.83:0/4176276037 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcce815fa10 0x7fcce80cf8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.906+0000 7fccef6d4640 1 --2- 172.21.15.83:0/4176276037 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcce815bdd0 0x7fcce80c9250 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.906+0000 7fccef6d4640 1 --2- 172.21.15.83:0/4176276037 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcce80d16e0 0x7fcce81f2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.906+0000 7fccef6d4640 1 -- 172.21.15.83:0/4176276037 >> 172.21.15.83:0/4176276037 conn(0x7fcce809a9f0 msgr2=0x7fcce806fc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:32.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.906+0000 7fccef6d4640 1 -- 172.21.15.83:0/4176276037 shutdown_connections 2024-03-06T07:22:32.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:32.907+0000 7fccef6d4640 1 -- 172.21.15.83:0/4176276037 wait complete. 2024-03-06T07:22:32.923 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:56.318942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:57.254795+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:32.924 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:33.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.026+0000 7ff572325640 1 Processor -- start 2024-03-06T07:22:33.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.026+0000 7ff572325640 1 -- start start 2024-03-06T07:22:33.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.027+0000 7ff572325640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff56c162230 0x7ff56c0b5db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.027+0000 7ff572325640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff56c0ae270 0x7ff56c0b11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.027+0000 7ff572325640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff56c06bf90 con 0x7ff56c1613f0 2024-03-06T07:22:33.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.027+0000 7ff572325640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff56c0b26f0 con 0x7ff56c162230 2024-03-06T07:22:33.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.027+0000 7ff572325640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c06c270 con 0x7ff56c0ae270 2024-03-06T07:22:33.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.027+0000 7ff56b7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff56c162230 0x7ff56c0b5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.027+0000 7ff56b7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff56c162230 0x7ff56c0b5db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48682/0 (socket says 172.21.15.83:48682) 2024-03-06T07:22:33.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.027+0000 7ff56b7fe640 1 -- 172.21.15.83:0/1366254925 learned_addr learned my addr 172.21.15.83:0/1366254925 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:33.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.028+0000 7ff5708c2640 1 --2- 172.21.15.83:0/1366254925 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff56c0ae270 0x7ff56c0b11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.028+0000 7ff56b7fe640 1 -- 172.21.15.83:0/1366254925 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff56c0ae270 msgr2=0x7ff56c0b11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:33.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.028+0000 7ff56b7fe640 1 --2- 172.21.15.83:0/1366254925 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff56c0ae270 0x7ff56c0b11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.028+0000 7ff56b7fe640 1 -- 172.21.15.83:0/1366254925 >> v1:172.21.15.83:6789/0 conn(0x7ff56c1613f0 legacy=0x7ff56c161b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.028+0000 7ff56b7fe640 1 -- 172.21.15.83:0/1366254925 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff56c06c550 con 0x7ff56c162230 2024-03-06T07:22:33.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.028+0000 7ff56b7fe640 1 --2- 172.21.15.83:0/1366254925 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff56c162230 0x7ff56c0b5db0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff560009870 tx=0x7ff56002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a03852db22b0b5d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.029+0000 7ff56affd640 1 -- 172.21.15.83:0/1366254925 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff560035020 con 0x7ff56c162230 2024-03-06T07:22:33.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.029+0000 7ff56affd640 1 -- 172.21.15.83:0/1366254925 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff560031640 con 0x7ff56c162230 2024-03-06T07:22:33.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.029+0000 7ff56affd640 1 -- 172.21.15.83:0/1366254925 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56002d040 con 0x7ff56c162230 2024-03-06T07:22:33.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.029+0000 7ff572325640 1 -- 172.21.15.83:0/1366254925 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff56c162230 msgr2=0x7ff56c0b5db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.029+0000 7ff572325640 1 --2- 172.21.15.83:0/1366254925 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff56c162230 0x7ff56c0b5db0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff560009870 tx=0x7ff56002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.029+0000 7ff572325640 1 -- 172.21.15.83:0/1366254925 shutdown_connections 2024-03-06T07:22:33.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.029+0000 7ff572325640 1 --2- 172.21.15.83:0/1366254925 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff56c0ae270 0x7ff56c0b11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.029+0000 7ff572325640 1 --2- 172.21.15.83:0/1366254925 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff56c162230 0x7ff56c0b5db0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.029+0000 7ff572325640 1 -- 172.21.15.83:0/1366254925 >> 172.21.15.83:0/1366254925 conn(0x7ff56c0b4fe0 msgr2=0x7ff56c0b56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:33.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.029+0000 7ff572325640 1 -- 172.21.15.83:0/1366254925 shutdown_connections 2024-03-06T07:22:33.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.029+0000 7ff572325640 1 -- 172.21.15.83:0/1366254925 wait complete. 2024-03-06T07:22:33.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.030+0000 7ff572325640 1 Processor -- start 2024-03-06T07:22:33.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.030+0000 7ff572325640 1 -- start start 2024-03-06T07:22:33.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.031+0000 7ff572325640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff56c0ae270 0x7ff56c150e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.031+0000 7ff572325640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff56c1613f0 0x7ff56c0999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.031+0000 7ff572325640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff56c162230 0x7ff56c15d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.031+0000 7ff572325640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c0b16e0 con 0x7ff56c162230 2024-03-06T07:22:33.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.031+0000 7ff572325640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c1617d0 con 0x7ff56c1613f0 2024-03-06T07:22:33.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.031+0000 7ff572325640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c0b1860 con 0x7ff56c0ae270 2024-03-06T07:22:33.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.031+0000 7ff5708c2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff56c162230 0x7ff56c15d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.031+0000 7ff5708c2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff56c162230 0x7ff56c15d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48684/0 (socket says 172.21.15.83:48684) 2024-03-06T07:22:33.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.031+0000 7ff5708c2640 1 -- 172.21.15.83:0/1152789985 learned_addr learned my addr 172.21.15.83:0/1152789985 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:33.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.031+0000 7ff56b7fe640 1 --2- 172.21.15.83:0/1152789985 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff56c1613f0 0x7ff56c0999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.031+0000 7ff56bfff640 1 --2- 172.21.15.83:0/1152789985 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff56c0ae270 0x7ff56c150e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.032+0000 7ff5708c2640 1 -- 172.21.15.83:0/1152789985 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff56c0ae270 msgr2=0x7ff56c150e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.032+0000 7ff5708c2640 1 --2- 172.21.15.83:0/1152789985 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff56c0ae270 0x7ff56c150e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.032+0000 7ff5708c2640 1 -- 172.21.15.83:0/1152789985 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff56c1613f0 msgr2=0x7ff56c0999b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.032+0000 7ff5708c2640 1 --2- 172.21.15.83:0/1152789985 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff56c1613f0 0x7ff56c0999b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.032+0000 7ff5708c2640 1 -- 172.21.15.83:0/1152789985 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff56c151500 con 0x7ff56c162230 2024-03-06T07:22:33.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.032+0000 7ff5708c2640 1 --2- 172.21.15.83:0/1152789985 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff56c162230 0x7ff56c15d360 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff564007e30 tx=0x7ff56400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.032+0000 7ff568ff9640 1 -- 172.21.15.83:0/1152789985 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff564014620 con 0x7ff56c162230 2024-03-06T07:22:33.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.032+0000 7ff568ff9640 1 -- 172.21.15.83:0/1152789985 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff564019070 con 0x7ff56c162230 2024-03-06T07:22:33.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.032+0000 7ff572325640 1 -- 172.21.15.83:0/1152789985 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff56c1517e0 con 0x7ff56c162230 2024-03-06T07:22:33.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.032+0000 7ff568ff9640 1 -- 172.21.15.83:0/1152789985 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff564014a00 con 0x7ff56c162230 2024-03-06T07:22:33.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.032+0000 7ff572325640 1 -- 172.21.15.83:0/1152789985 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff56c151da0 con 0x7ff56c162230 2024-03-06T07:22:33.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.033+0000 7ff572325640 1 -- 172.21.15.83:0/1152789985 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff56c09e4e0 con 0x7ff56c162230 2024-03-06T07:22:33.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.033+0000 7ff568ff9640 1 -- 172.21.15.83:0/1152789985 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff564014ba0 con 0x7ff56c162230 2024-03-06T07:22:33.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.034+0000 7ff568ff9640 1 --2- 172.21.15.83:0/1152789985 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff55806b4d0 0x7ff55806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.034+0000 7ff568ff9640 1 -- 172.21.15.83:0/1152789985 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff56408aaa0 con 0x7ff56c162230 2024-03-06T07:22:33.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.034+0000 7ff56bfff640 1 --2- 172.21.15.83:0/1152789985 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff55806b4d0 0x7ff55806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.034+0000 7ff56bfff640 1 --2- 172.21.15.83:0/1152789985 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff55806b4d0 0x7ff55806d990 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff55c001640 tx=0x7ff55c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.038+0000 7ff568ff9640 1 -- 172.21.15.83:0/1152789985 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff564059920 con 0x7ff56c162230 2024-03-06T07:22:33.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.215+0000 7ff572325640 1 -- 172.21.15.83:0/1152789985 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff56c068ff0 con 0x7ff56c162230 2024-03-06T07:22:33.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.216+0000 7ff568ff9640 1 -- 172.21.15.83:0/1152789985 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff564059220 con 0x7ff56c162230 2024-03-06T07:22:33.218 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:22:33.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.217+0000 7ff572325640 1 -- 172.21.15.83:0/1152789985 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff55806b4d0 msgr2=0x7ff55806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.217+0000 7ff572325640 1 --2- 172.21.15.83:0/1152789985 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff55806b4d0 0x7ff55806d990 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff55c001640 tx=0x7ff55c009040 comp rx=0 tx=0).stop 2024-03-06T07:22:33.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.217+0000 7ff572325640 1 -- 172.21.15.83:0/1152789985 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff56c162230 msgr2=0x7ff56c15d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.217+0000 7ff572325640 1 --2- 172.21.15.83:0/1152789985 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff56c162230 0x7ff56c15d360 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff564007e30 tx=0x7ff56400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.217+0000 7ff572325640 1 -- 172.21.15.83:0/1152789985 shutdown_connections 2024-03-06T07:22:33.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.217+0000 7ff572325640 1 --2- 172.21.15.83:0/1152789985 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff55806b4d0 0x7ff55806d990 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.217+0000 7ff572325640 1 --2- 172.21.15.83:0/1152789985 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff56c162230 0x7ff56c15d360 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.217+0000 7ff572325640 1 --2- 172.21.15.83:0/1152789985 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff56c1613f0 0x7ff56c0999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.217+0000 7ff572325640 1 --2- 172.21.15.83:0/1152789985 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff56c0ae270 0x7ff56c150e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.218+0000 7ff572325640 1 -- 172.21.15.83:0/1152789985 >> 172.21.15.83:0/1152789985 conn(0x7ff56c0b4fe0 msgr2=0x7ff56c06d720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:33.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.218+0000 7ff572325640 1 -- 172.21.15.83:0/1152789985 shutdown_connections 2024-03-06T07:22:33.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.218+0000 7ff572325640 1 -- 172.21.15.83:0/1152789985 wait complete. 2024-03-06T07:22:33.230 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:33.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.332+0000 7f6470d5e640 1 Processor -- start 2024-03-06T07:22:33.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.332+0000 7f6470d5e640 1 -- start start 2024-03-06T07:22:33.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.333+0000 7f6470d5e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f646c15ad90 0x7f646c0b3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.333+0000 7f6470d5e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f646c0b4110 0x7f646c15c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.333+0000 7f6470d5e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f646c15b260 con 0x7f646c071a70 2024-03-06T07:22:33.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.333+0000 7f6470d5e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f646c15c020 con 0x7f646c0b4110 2024-03-06T07:22:33.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.333+0000 7f6470d5e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f646c15b540 con 0x7f646c15ad90 2024-03-06T07:22:33.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.333+0000 7f646ad9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f646c0b4110 0x7f646c15c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.333+0000 7f646ad9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f646c0b4110 0x7f646c15c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48694/0 (socket says 172.21.15.83:48694) 2024-03-06T07:22:33.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.333+0000 7f646ad9d640 1 -- 172.21.15.83:0/1925263799 learned_addr learned my addr 172.21.15.83:0/1925263799 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:33.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.333+0000 7f6469d9b640 1 --2- 172.21.15.83:0/1925263799 >> v2:172.21.15.183:3300/0 conn(0x7f646c15ad90 0x7f646c0b3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.333+0000 7f646ad9d640 1 -- 172.21.15.83:0/1925263799 >> v2:172.21.15.183:3300/0 conn(0x7f646c15ad90 msgr2=0x7f646c0b3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:33.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.333+0000 7f646ad9d640 1 --2- 172.21.15.83:0/1925263799 >> v2:172.21.15.183:3300/0 conn(0x7f646c15ad90 0x7f646c0b3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.333+0000 7f646ad9d640 1 -- 172.21.15.83:0/1925263799 >> v1:172.21.15.183:6789/0 conn(0x7f646c071a70 legacy=0x7f646c071e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.333+0000 7f646ad9d640 1 -- 172.21.15.83:0/1925263799 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f646c15b820 con 0x7f646c0b4110 2024-03-06T07:22:33.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.335+0000 7f646ad9d640 1 --2- 172.21.15.83:0/1925263799 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f646c0b4110 0x7f646c15c210 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f646c0b4ee0 tx=0x7f645c02e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c40a85d00d902fc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.335+0000 7f646959a640 1 -- 172.21.15.83:0/1925263799 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645c03b070 con 0x7f646c0b4110 2024-03-06T07:22:33.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.335+0000 7f646959a640 1 -- 172.21.15.83:0/1925263799 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f645c036440 con 0x7f646c0b4110 2024-03-06T07:22:33.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.335+0000 7f646959a640 1 -- 172.21.15.83:0/1925263799 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645c036810 con 0x7f646c0b4110 2024-03-06T07:22:33.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.335+0000 7f6470d5e640 1 -- 172.21.15.83:0/1925263799 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f646c0b4110 msgr2=0x7f646c15c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.335+0000 7f6470d5e640 1 --2- 172.21.15.83:0/1925263799 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f646c0b4110 0x7f646c15c210 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f646c0b4ee0 tx=0x7f645c02e0e0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.335+0000 7f6470d5e640 1 -- 172.21.15.83:0/1925263799 shutdown_connections 2024-03-06T07:22:33.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.335+0000 7f6470d5e640 1 --2- 172.21.15.83:0/1925263799 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f646c0b4110 0x7f646c15c210 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.335+0000 7f6470d5e640 1 --2- 172.21.15.83:0/1925263799 >> v2:172.21.15.183:3300/0 conn(0x7f646c15ad90 0x7f646c0b3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.335+0000 7f6470d5e640 1 -- 172.21.15.83:0/1925263799 >> 172.21.15.83:0/1925263799 conn(0x7f646c0b2660 msgr2=0x7f646c0b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:33.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.335+0000 7f6470d5e640 1 -- 172.21.15.83:0/1925263799 shutdown_connections 2024-03-06T07:22:33.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.335+0000 7f6470d5e640 1 -- 172.21.15.83:0/1925263799 wait complete. 2024-03-06T07:22:33.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.337+0000 7f6470d5e640 1 Processor -- start 2024-03-06T07:22:33.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.337+0000 7f6470d5e640 1 -- start start 2024-03-06T07:22:33.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.337+0000 7f6470d5e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f646c071a70 0x7f646c1e55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.337+0000 7f6470d5e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f646c0b4110 0x7f646c1e5af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.337+0000 7f6470d5e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f646c15ad90 0x7f646c1eff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.337+0000 7f6470d5e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f646c0b4700 con 0x7f646c071a70 2024-03-06T07:22:33.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.337+0000 7f6470d5e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f646c0b4580 con 0x7f646c0b4110 2024-03-06T07:22:33.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.337+0000 7f6470d5e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f646c15c750 con 0x7f646c15ad90 2024-03-06T07:22:33.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.338+0000 7f646a59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f646c071a70 0x7f646c1e55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.338+0000 7f646a59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f646c071a70 0x7f646c1e55b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48708/0 (socket says 172.21.15.83:48708) 2024-03-06T07:22:33.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.338+0000 7f646a59c640 1 -- 172.21.15.83:0/4052328981 learned_addr learned my addr 172.21.15.83:0/4052328981 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:33.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.338+0000 7f6469d9b640 1 --2- 172.21.15.83:0/4052328981 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f646c0b4110 0x7f646c1e5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.338+0000 7f646ad9d640 1 --2- 172.21.15.83:0/4052328981 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f646c15ad90 0x7f646c1eff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.338+0000 7f646a59c640 1 -- 172.21.15.83:0/4052328981 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f646c15ad90 msgr2=0x7f646c1eff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.338+0000 7f646a59c640 1 --2- 172.21.15.83:0/4052328981 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f646c15ad90 0x7f646c1eff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.338+0000 7f646a59c640 1 -- 172.21.15.83:0/4052328981 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f646c0b4110 msgr2=0x7f646c1e5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.338+0000 7f646a59c640 1 --2- 172.21.15.83:0/4052328981 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f646c0b4110 0x7f646c1e5af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.338+0000 7f646a59c640 1 -- 172.21.15.83:0/4052328981 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f646c156820 con 0x7f646c071a70 2024-03-06T07:22:33.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.338+0000 7f646a59c640 1 --2- 172.21.15.83:0/4052328981 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f646c071a70 0x7f646c1e55b0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f646400c7c0 tx=0x7f646400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.339+0000 7f6462ffd640 1 -- 172.21.15.83:0/4052328981 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6464016020 con 0x7f646c071a70 2024-03-06T07:22:33.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.339+0000 7f6462ffd640 1 -- 172.21.15.83:0/4052328981 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6464006d50 con 0x7f646c071a70 2024-03-06T07:22:33.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.339+0000 7f6470d5e640 1 -- 172.21.15.83:0/4052328981 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f646c156b00 con 0x7f646c071a70 2024-03-06T07:22:33.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.339+0000 7f6462ffd640 1 -- 172.21.15.83:0/4052328981 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6464003bb0 con 0x7f646c071a70 2024-03-06T07:22:33.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.339+0000 7f6470d5e640 1 -- 172.21.15.83:0/4052328981 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f646c1570c0 con 0x7f646c071a70 2024-03-06T07:22:33.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.340+0000 7f6470d5e640 1 -- 172.21.15.83:0/4052328981 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f646c073390 con 0x7f646c071a70 2024-03-06T07:22:33.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.340+0000 7f6462ffd640 1 -- 172.21.15.83:0/4052328981 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f646400a850 con 0x7f646c071a70 2024-03-06T07:22:33.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.341+0000 7f6462ffd640 1 --2- 172.21.15.83:0/4052328981 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f644c06b4d0 0x7f644c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.341+0000 7f6469d9b640 1 --2- 172.21.15.83:0/4052328981 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f644c06b4d0 0x7f644c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.341+0000 7f6462ffd640 1 -- 172.21.15.83:0/4052328981 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6464089da0 con 0x7f646c071a70 2024-03-06T07:22:33.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.341+0000 7f6469d9b640 1 --2- 172.21.15.83:0/4052328981 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f644c06b4d0 0x7f644c06d990 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f646c06e770 tx=0x7f6454008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.345+0000 7f6462ffd640 1 -- 172.21.15.83:0/4052328981 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6464020020 con 0x7f646c071a70 2024-03-06T07:22:33.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.516+0000 7f6470d5e640 1 -- 172.21.15.83:0/4052328981 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f646c0d07d0 con 0x7f646c071a70 2024-03-06T07:22:33.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.516+0000 7f6462ffd640 1 -- 172.21.15.83:0/4052328981 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f64640585e0 con 0x7f646c071a70 2024-03-06T07:22:33.518 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-03-06T07:22:33.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.518+0000 7f6470d5e640 1 -- 172.21.15.83:0/4052328981 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f644c06b4d0 msgr2=0x7f644c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.518+0000 7f6470d5e640 1 --2- 172.21.15.83:0/4052328981 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f644c06b4d0 0x7f644c06d990 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f646c06e770 tx=0x7f6454008040 comp rx=0 tx=0).stop 2024-03-06T07:22:33.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.518+0000 7f6470d5e640 1 -- 172.21.15.83:0/4052328981 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f646c071a70 msgr2=0x7f646c1e55b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.518+0000 7f6470d5e640 1 --2- 172.21.15.83:0/4052328981 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f646c071a70 0x7f646c1e55b0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f646400c7c0 tx=0x7f646400cc80 comp rx=0 tx=0).stop 2024-03-06T07:22:33.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.518+0000 7f6470d5e640 1 -- 172.21.15.83:0/4052328981 shutdown_connections 2024-03-06T07:22:33.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.518+0000 7f6470d5e640 1 --2- 172.21.15.83:0/4052328981 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f644c06b4d0 0x7f644c06d990 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.518+0000 7f6470d5e640 1 --2- 172.21.15.83:0/4052328981 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f646c15ad90 0x7f646c1eff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.518+0000 7f6470d5e640 1 --2- 172.21.15.83:0/4052328981 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f646c0b4110 0x7f646c1e5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.518+0000 7f6470d5e640 1 --2- 172.21.15.83:0/4052328981 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f646c071a70 0x7f646c1e55b0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.518+0000 7f6470d5e640 1 -- 172.21.15.83:0/4052328981 >> 172.21.15.83:0/4052328981 conn(0x7f646c0b2660 msgr2=0x7f646c06e900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:33.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.518+0000 7f6470d5e640 1 -- 172.21.15.83:0/4052328981 shutdown_connections 2024-03-06T07:22:33.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.518+0000 7f6470d5e640 1 -- 172.21.15.83:0/4052328981 wait complete. 2024-03-06T07:22:33.531 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:33.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.633+0000 7f17f39bf640 1 Processor -- start 2024-03-06T07:22:33.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.633+0000 7f17f39bf640 1 -- start start 2024-03-06T07:22:33.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.635+0000 7f17f39bf640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f17ec06fd10 0x7f17ec0cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.635+0000 7f17f39bf640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17ec0cd4b0 con 0x7f17ec0d0af0 2024-03-06T07:22:33.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.635+0000 7f17f39bf640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17ec0cd790 con 0x7f17ec15c050 2024-03-06T07:22:33.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.635+0000 7f17f39bf640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17ec0cda70 con 0x7f17ec06fd10 2024-03-06T07:22:33.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.635+0000 7f17f1f5c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f17ec0d0af0 0x7f17ec0ccca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39448/0 (socket says 172.21.15.83:39448) 2024-03-06T07:22:33.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.635+0000 7f17f1f5c640 1 -- 172.21.15.83:0/1476925357 learned_addr learned my addr 172.21.15.83:0/1476925357 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:33.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.635+0000 7f17f0f5a640 1 --2- 172.21.15.83:0/1476925357 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f17ec06fd10 0x7f17ec0cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.635+0000 7f17dffff640 1 -- 172.21.15.83:0/1476925357 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 362008944 0 0) 0x7f17ec0cd4b0 con 0x7f17ec0d0af0 2024-03-06T07:22:33.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.635+0000 7f17dffff640 1 -- 172.21.15.83:0/1476925357 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17d8003520 con 0x7f17ec0d0af0 2024-03-06T07:22:33.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.635+0000 7f17dffff640 1 -- 172.21.15.83:0/1476925357 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1378248372 0 0) 0x7f17d8003520 con 0x7f17ec0d0af0 2024-03-06T07:22:33.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.635+0000 7f17dffff640 1 -- 172.21.15.83:0/1476925357 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f17ec06fd10 msgr2=0x7f17ec0cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:33.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.635+0000 7f17dffff640 1 --2- 172.21.15.83:0/1476925357 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f17ec06fd10 0x7f17ec0cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.635+0000 7f17dffff640 1 -- 172.21.15.83:0/1476925357 >> v1:172.21.15.183:6789/0 conn(0x7f17ec15c050 legacy=0x7f17ec06f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.635+0000 7f17dffff640 1 -- 172.21.15.83:0/1476925357 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17ec0cdd50 con 0x7f17ec0d0af0 2024-03-06T07:22:33.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.635+0000 7f17dffff640 1 -- 172.21.15.83:0/1476925357 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f17e40019f0 con 0x7f17ec0d0af0 2024-03-06T07:22:33.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.635+0000 7f17dffff640 1 -- 172.21.15.83:0/1476925357 >> v1:172.21.15.83:6789/0 conn(0x7f17ec0d0af0 legacy=0x7f17ec0ccca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.636+0000 7f17dffff640 1 --2- 172.21.15.83:0/1476925357 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f17ec06fd10 0x7f17d80039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.636+0000 7f17dffff640 1 --2- 172.21.15.83:0/1476925357 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17d8003f20 0x7f17d8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.636+0000 7f17dffff640 1 --2- 172.21.15.83:0/1476925357 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17d8004d40 0x7f17d80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.636+0000 7f17dffff640 1 -- 172.21.15.83:0/1476925357 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ec0cda70 con 0x7f17d8004d40 2024-03-06T07:22:33.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.636+0000 7f17dffff640 1 -- 172.21.15.83:0/1476925357 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d800bad0 con 0x7f17d8003f20 2024-03-06T07:22:33.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.636+0000 7f17dffff640 1 -- 172.21.15.83:0/1476925357 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17d800bcd0 con 0x7f17ec06fd10 2024-03-06T07:22:33.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.636+0000 7f17f1f5c640 1 --2- 172.21.15.83:0/1476925357 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17d8004d40 0x7f17d80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.636+0000 7f17f175b640 1 --2- 172.21.15.83:0/1476925357 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f17ec06fd10 0x7f17d80039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.636+0000 7f17f0f5a640 1 --2- 172.21.15.83:0/1476925357 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17d8003f20 0x7f17d8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.636+0000 7f17f1f5c640 1 -- 172.21.15.83:0/1476925357 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f17ec06fd10 msgr2=0x7f17d80039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:33.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.636+0000 7f17f1f5c640 1 --2- 172.21.15.83:0/1476925357 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f17ec06fd10 0x7f17d80039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.636+0000 7f17f1f5c640 1 -- 172.21.15.83:0/1476925357 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17d8003f20 msgr2=0x7f17d8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:33.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.636+0000 7f17f1f5c640 1 --2- 172.21.15.83:0/1476925357 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17d8003f20 0x7f17d8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.636+0000 7f17f1f5c640 1 -- 172.21.15.83:0/1476925357 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d8003520 con 0x7f17d8004d40 2024-03-06T07:22:33.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.637+0000 7f17f1f5c640 1 --2- 172.21.15.83:0/1476925357 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17d8004d40 0x7f17d80051c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f17e400a8f0 tx=0x7f17d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e2f3cb1c5b8bd32 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.637+0000 7f17dffff640 1 -- 172.21.15.83:0/1476925357 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e400a4c0 con 0x7f17d8004d40 2024-03-06T07:22:33.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.637+0000 7f17dffff640 1 -- 172.21.15.83:0/1476925357 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17e4034650 con 0x7f17d8004d40 2024-03-06T07:22:33.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.637+0000 7f17dffff640 1 -- 172.21.15.83:0/1476925357 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e4034a80 con 0x7f17d8004d40 2024-03-06T07:22:33.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.637+0000 7f17f39bf640 1 -- 172.21.15.83:0/1476925357 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17d8004d40 msgr2=0x7f17d80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.637+0000 7f17f39bf640 1 --2- 172.21.15.83:0/1476925357 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17d8004d40 0x7f17d80051c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f17e400a8f0 tx=0x7f17d8005830 comp rx=0 tx=0).stop 2024-03-06T07:22:33.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.637+0000 7f17f1f5c640 1 -- 172.21.15.83:0/1476925357 reap_dead start 2024-03-06T07:22:33.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.637+0000 7f17f39bf640 1 -- 172.21.15.83:0/1476925357 shutdown_connections 2024-03-06T07:22:33.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.637+0000 7f17f39bf640 1 -- 172.21.15.83:0/1476925357 >> 172.21.15.83:0/1476925357 conn(0x7f17ec0b3800 msgr2=0x7f17ec0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:33.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.637+0000 7f17f39bf640 1 -- 172.21.15.83:0/1476925357 shutdown_connections 2024-03-06T07:22:33.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.638+0000 7f17f39bf640 1 -- 172.21.15.83:0/1476925357 wait complete. 2024-03-06T07:22:33.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.639+0000 7f17f39bf640 1 Processor -- start 2024-03-06T07:22:33.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.639+0000 7f17f39bf640 1 -- start start 2024-03-06T07:22:33.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.639+0000 7f17f39bf640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17ec1567b0 0x7f17ec1f2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.639+0000 7f17f39bf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17ec156b90 0x7f17ec09d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.639+0000 7f17f39bf640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f17ec09de30 0x7f17ec0b6f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.639+0000 7f17f39bf640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ec0ccaf0 con 0x7f17ec156b90 2024-03-06T07:22:33.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.639+0000 7f17f39bf640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ec0cc970 con 0x7f17ec1567b0 2024-03-06T07:22:33.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.639+0000 7f17f39bf640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17ec161a90 con 0x7f17ec09de30 2024-03-06T07:22:33.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.639+0000 7f17f0f5a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17ec156b90 0x7f17ec09d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.639+0000 7f17f0f5a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17ec156b90 0x7f17ec09d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48730/0 (socket says 172.21.15.83:48730) 2024-03-06T07:22:33.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.639+0000 7f17f0f5a640 1 -- 172.21.15.83:0/1455878502 learned_addr learned my addr 172.21.15.83:0/1455878502 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:33.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.640+0000 7f17f175b640 1 --2- 172.21.15.83:0/1455878502 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17ec1567b0 0x7f17ec1f2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.640+0000 7f17f1f5c640 1 --2- 172.21.15.83:0/1455878502 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f17ec09de30 0x7f17ec0b6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.640+0000 7f17f0f5a640 1 -- 172.21.15.83:0/1455878502 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f17ec09de30 msgr2=0x7f17ec0b6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.640+0000 7f17f0f5a640 1 --2- 172.21.15.83:0/1455878502 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f17ec09de30 0x7f17ec0b6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.640+0000 7f17f0f5a640 1 -- 172.21.15.83:0/1455878502 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17ec1567b0 msgr2=0x7f17ec1f2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.640+0000 7f17f0f5a640 1 --2- 172.21.15.83:0/1455878502 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17ec1567b0 0x7f17ec1f2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.640+0000 7f17f0f5a640 1 -- 172.21.15.83:0/1455878502 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17ec0b4990 con 0x7f17ec156b90 2024-03-06T07:22:33.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.640+0000 7f17f175b640 1 --2- 172.21.15.83:0/1455878502 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17ec1567b0 0x7f17ec1f2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:33.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.640+0000 7f17f1f5c640 1 --2- 172.21.15.83:0/1455878502 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f17ec09de30 0x7f17ec0b6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:33.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.640+0000 7f17f0f5a640 1 --2- 172.21.15.83:0/1455878502 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17ec156b90 0x7f17ec09d8f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f17e00119a0 tx=0x7f17e0011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.640+0000 7f17ddffb640 1 -- 172.21.15.83:0/1455878502 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e000faf0 con 0x7f17ec156b90 2024-03-06T07:22:33.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.640+0000 7f17ddffb640 1 -- 172.21.15.83:0/1455878502 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17e000fc90 con 0x7f17ec156b90 2024-03-06T07:22:33.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.640+0000 7f17ddffb640 1 -- 172.21.15.83:0/1455878502 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e0014440 con 0x7f17ec156b90 2024-03-06T07:22:33.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.640+0000 7f17f39bf640 1 -- 172.21.15.83:0/1455878502 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17ec0b4c70 con 0x7f17ec156b90 2024-03-06T07:22:33.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.640+0000 7f17f39bf640 1 -- 172.21.15.83:0/1455878502 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17ec0b5230 con 0x7f17ec156b90 2024-03-06T07:22:33.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.642+0000 7f17f39bf640 1 -- 172.21.15.83:0/1455878502 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17ec09fe60 con 0x7f17ec156b90 2024-03-06T07:22:33.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.642+0000 7f17ddffb640 1 -- 172.21.15.83:0/1455878502 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f17e0012040 con 0x7f17ec156b90 2024-03-06T07:22:33.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.642+0000 7f17ddffb640 1 --2- 172.21.15.83:0/1455878502 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f17d006b500 0x7f17d006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.642+0000 7f17ddffb640 1 -- 172.21.15.83:0/1455878502 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f17e008a870 con 0x7f17ec156b90 2024-03-06T07:22:33.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.642+0000 7f17f175b640 1 --2- 172.21.15.83:0/1455878502 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f17d006b500 0x7f17d006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.643+0000 7f17f175b640 1 --2- 172.21.15.83:0/1455878502 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f17d006b500 0x7f17d006d9c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f17ec0cbb90 tx=0x7f17e8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.647+0000 7f17ddffb640 1 -- 172.21.15.83:0/1455878502 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f17e0019030 con 0x7f17ec156b90 2024-03-06T07:22:33.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.887+0000 7f17f39bf640 1 -- 172.21.15.83:0/1455878502 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f17ec068ff0 con 0x7f17ec156b90 2024-03-06T07:22:33.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.887+0000 7f17ddffb640 1 -- 172.21.15.83:0/1455878502 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f17e00596f0 con 0x7f17ec156b90 2024-03-06T07:22:33.889 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 64 2024-03-06T07:22:33.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.889+0000 7f17f39bf640 1 -- 172.21.15.83:0/1455878502 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f17d006b500 msgr2=0x7f17d006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.889+0000 7f17f39bf640 1 --2- 172.21.15.83:0/1455878502 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f17d006b500 0x7f17d006d9c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f17ec0cbb90 tx=0x7f17e8009040 comp rx=0 tx=0).stop 2024-03-06T07:22:33.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.889+0000 7f17f39bf640 1 -- 172.21.15.83:0/1455878502 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17ec156b90 msgr2=0x7f17ec09d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.889+0000 7f17f39bf640 1 --2- 172.21.15.83:0/1455878502 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17ec156b90 0x7f17ec09d8f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f17e00119a0 tx=0x7f17e0011e60 comp rx=0 tx=0).stop 2024-03-06T07:22:33.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.889+0000 7f17f39bf640 1 -- 172.21.15.83:0/1455878502 shutdown_connections 2024-03-06T07:22:33.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.889+0000 7f17f39bf640 1 --2- 172.21.15.83:0/1455878502 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f17d006b500 0x7f17d006d9c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.889+0000 7f17f39bf640 1 --2- 172.21.15.83:0/1455878502 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f17ec09de30 0x7f17ec0b6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.889+0000 7f17f39bf640 1 --2- 172.21.15.83:0/1455878502 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17ec156b90 0x7f17ec09d8f0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.889+0000 7f17f39bf640 1 --2- 172.21.15.83:0/1455878502 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17ec1567b0 0x7f17ec1f2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.889+0000 7f17f39bf640 1 -- 172.21.15.83:0/1455878502 >> 172.21.15.83:0/1455878502 conn(0x7f17ec0b3800 msgr2=0x7f17ec0b2f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:33.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.889+0000 7f17f39bf640 1 -- 172.21.15.83:0/1455878502 shutdown_connections 2024-03-06T07:22:33.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:33.889+0000 7f17f39bf640 1 -- 172.21.15.83:0/1455878502 wait complete. 2024-03-06T07:22:33.902 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:22:34.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.002+0000 7ff7065bc640 1 Processor -- start 2024-03-06T07:22:34.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.002+0000 7ff7065bc640 1 -- start start 2024-03-06T07:22:34.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.003+0000 7ff7065bc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff7001614a0 0x7ff70006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.003+0000 7ff7065bc640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ff7000b1a20 0x7ff7000b5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.003+0000 7ff7065bc640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7000af4b0 con 0x7ff7000b0270 2024-03-06T07:22:34.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.003+0000 7ff7065bc640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7000af790 con 0x7ff7001614a0 2024-03-06T07:22:34.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.003+0000 7ff7065bc640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7000afa70 con 0x7ff7000b1a20 2024-03-06T07:22:34.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.003+0000 7ff6fffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff7001614a0 0x7ff70006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.003+0000 7ff6fffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff7001614a0 0x7ff70006d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48736/0 (socket says 172.21.15.83:48736) 2024-03-06T07:22:34.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.003+0000 7ff6fffff640 1 -- 172.21.15.83:0/257508690 learned_addr learned my addr 172.21.15.83:0/257508690 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:34.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.004+0000 7ff704b59640 1 --2- 172.21.15.83:0/257508690 >> v2:172.21.15.183:3300/0 conn(0x7ff7000b1a20 0x7ff7000b5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.004+0000 7ff6fffff640 1 -- 172.21.15.83:0/257508690 >> v2:172.21.15.183:3300/0 conn(0x7ff7000b1a20 msgr2=0x7ff7000b5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:34.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.004+0000 7ff6fffff640 1 --2- 172.21.15.83:0/257508690 >> v2:172.21.15.183:3300/0 conn(0x7ff7000b1a20 0x7ff7000b5510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.004+0000 7ff6fffff640 1 -- 172.21.15.83:0/257508690 >> v1:172.21.15.83:6789/0 conn(0x7ff7000b0270 legacy=0x7ff7000b1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.004+0000 7ff6fffff640 1 -- 172.21.15.83:0/257508690 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7000afd50 con 0x7ff7001614a0 2024-03-06T07:22:34.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.004+0000 7ff6fffff640 1 --2- 172.21.15.83:0/257508690 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7001614a0 0x7ff70006d3c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff7000b34f0 tx=0x7ff6e802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7166a3924d98e3d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.004+0000 7ff6feffd640 1 -- 172.21.15.83:0/257508690 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e8035020 con 0x7ff7001614a0 2024-03-06T07:22:34.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.005+0000 7ff6feffd640 1 -- 172.21.15.83:0/257508690 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6e8031870 con 0x7ff7001614a0 2024-03-06T07:22:34.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.005+0000 7ff6feffd640 1 -- 172.21.15.83:0/257508690 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e8031c40 con 0x7ff7001614a0 2024-03-06T07:22:34.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.005+0000 7ff7065bc640 1 -- 172.21.15.83:0/257508690 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7001614a0 msgr2=0x7ff70006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.005+0000 7ff7065bc640 1 --2- 172.21.15.83:0/257508690 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7001614a0 0x7ff70006d3c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff7000b34f0 tx=0x7ff6e802b910 comp rx=0 tx=0).stop 2024-03-06T07:22:34.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.005+0000 7ff7065bc640 1 -- 172.21.15.83:0/257508690 shutdown_connections 2024-03-06T07:22:34.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.005+0000 7ff7065bc640 1 --2- 172.21.15.83:0/257508690 >> v2:172.21.15.183:3300/0 conn(0x7ff7000b1a20 0x7ff7000b5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.005+0000 7ff7065bc640 1 --2- 172.21.15.83:0/257508690 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7001614a0 0x7ff70006d3c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.005+0000 7ff7065bc640 1 -- 172.21.15.83:0/257508690 >> 172.21.15.83:0/257508690 conn(0x7ff70009a9f0 msgr2=0x7ff70009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.005+0000 7ff7065bc640 1 -- 172.21.15.83:0/257508690 shutdown_connections 2024-03-06T07:22:34.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.005+0000 7ff7065bc640 1 -- 172.21.15.83:0/257508690 wait complete. 2024-03-06T07:22:34.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.006+0000 7ff7065bc640 1 Processor -- start 2024-03-06T07:22:34.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.006+0000 7ff7065bc640 1 -- start start 2024-03-06T07:22:34.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.007+0000 7ff7065bc640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7000b0270 0x7ff7000cad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.007+0000 7ff7065bc640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff7000b1a20 0x7ff7000cc390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.007+0000 7ff7065bc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7001614a0 0x7ff700148430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.007+0000 7ff7065bc640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7000b1e00 con 0x7ff7001614a0 2024-03-06T07:22:34.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.007+0000 7ff7065bc640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7000b06e0 con 0x7ff7000b0270 2024-03-06T07:22:34.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.007+0000 7ff7065bc640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7000b1f80 con 0x7ff7000b1a20 2024-03-06T07:22:34.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.007+0000 7ff704b59640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7001614a0 0x7ff700148430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.007+0000 7ff704b59640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7001614a0 0x7ff700148430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48746/0 (socket says 172.21.15.83:48746) 2024-03-06T07:22:34.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.007+0000 7ff704b59640 1 -- 172.21.15.83:0/1879097657 learned_addr learned my addr 172.21.15.83:0/1879097657 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:34.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.007+0000 7ff6fffff640 1 --2- 172.21.15.83:0/1879097657 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7000b0270 0x7ff7000cad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.007+0000 7ff6ff7fe640 1 --2- 172.21.15.83:0/1879097657 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff7000b1a20 0x7ff7000cc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.008+0000 7ff704b59640 1 -- 172.21.15.83:0/1879097657 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff7000b1a20 msgr2=0x7ff7000cc390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.008+0000 7ff704b59640 1 --2- 172.21.15.83:0/1879097657 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff7000b1a20 0x7ff7000cc390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.008+0000 7ff704b59640 1 -- 172.21.15.83:0/1879097657 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7000b0270 msgr2=0x7ff7000cad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.008+0000 7ff704b59640 1 --2- 172.21.15.83:0/1879097657 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7000b0270 0x7ff7000cad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.008+0000 7ff704b59640 1 -- 172.21.15.83:0/1879097657 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff700148970 con 0x7ff7001614a0 2024-03-06T07:22:34.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.008+0000 7ff6fffff640 1 --2- 172.21.15.83:0/1879097657 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7000b0270 0x7ff7000cad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:34.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.008+0000 7ff6ff7fe640 1 --2- 172.21.15.83:0/1879097657 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff7000b1a20 0x7ff7000cc390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:34.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.008+0000 7ff704b59640 1 --2- 172.21.15.83:0/1879097657 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7001614a0 0x7ff700148430 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff6f800cc30 tx=0x7ff6f800a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.008+0000 7ff6fcff9640 1 -- 172.21.15.83:0/1879097657 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f800ce40 con 0x7ff7001614a0 2024-03-06T07:22:34.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.008+0000 7ff6fcff9640 1 -- 172.21.15.83:0/1879097657 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6f8003990 con 0x7ff7001614a0 2024-03-06T07:22:34.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.008+0000 7ff7065bc640 1 -- 172.21.15.83:0/1879097657 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7000cb2f0 con 0x7ff7001614a0 2024-03-06T07:22:34.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.008+0000 7ff6fcff9640 1 -- 172.21.15.83:0/1879097657 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f8003d40 con 0x7ff7001614a0 2024-03-06T07:22:34.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.008+0000 7ff7065bc640 1 -- 172.21.15.83:0/1879097657 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7000cb8b0 con 0x7ff7001614a0 2024-03-06T07:22:34.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.009+0000 7ff7065bc640 1 -- 172.21.15.83:0/1879097657 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff70004f330 con 0x7ff7001614a0 2024-03-06T07:22:34.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.009+0000 7ff6fcff9640 1 -- 172.21.15.83:0/1879097657 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff6f8003ee0 con 0x7ff7001614a0 2024-03-06T07:22:34.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.010+0000 7ff6fcff9640 1 --2- 172.21.15.83:0/1879097657 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff6e006b500 0x7ff6e006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.010+0000 7ff6fffff640 1 --2- 172.21.15.83:0/1879097657 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff6e006b500 0x7ff6e006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.010+0000 7ff6fcff9640 1 -- 172.21.15.83:0/1879097657 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff6f808a680 con 0x7ff7001614a0 2024-03-06T07:22:34.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.010+0000 7ff6fffff640 1 --2- 172.21.15.83:0/1879097657 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff6e006b500 0x7ff6e006d9c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8036000 tx=0x7ff6e803c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.014+0000 7ff6fcff9640 1 -- 172.21.15.83:0/1879097657 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff6f8014030 con 0x7ff7001614a0 2024-03-06T07:22:34.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.205+0000 7ff7065bc640 1 -- 172.21.15.83:0/1879097657 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff7000b6430 con 0x7ff7001614a0 2024-03-06T07:22:34.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.205+0000 7ff6fcff9640 1 -- 172.21.15.83:0/1879097657 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) v1 ==== 80+0+0 (secure 0 0 0) 0x7ff6f8059470 con 0x7ff7001614a0 2024-03-06T07:22:34.207 INFO:teuthology.orchestra.run.smithi083.stderr:listed 0 entries 2024-03-06T07:22:34.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.207+0000 7ff7065bc640 1 -- 172.21.15.83:0/1879097657 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff6e006b500 msgr2=0x7ff6e006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.207+0000 7ff7065bc640 1 --2- 172.21.15.83:0/1879097657 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff6e006b500 0x7ff6e006d9c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8036000 tx=0x7ff6e803c040 comp rx=0 tx=0).stop 2024-03-06T07:22:34.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.207+0000 7ff7065bc640 1 -- 172.21.15.83:0/1879097657 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7001614a0 msgr2=0x7ff700148430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.207+0000 7ff7065bc640 1 --2- 172.21.15.83:0/1879097657 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7001614a0 0x7ff700148430 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff6f800cc30 tx=0x7ff6f800a5c0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.207+0000 7ff7065bc640 1 -- 172.21.15.83:0/1879097657 shutdown_connections 2024-03-06T07:22:34.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.207+0000 7ff7065bc640 1 --2- 172.21.15.83:0/1879097657 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff6e006b500 0x7ff6e006d9c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.207+0000 7ff7065bc640 1 --2- 172.21.15.83:0/1879097657 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7001614a0 0x7ff700148430 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.207+0000 7ff7065bc640 1 --2- 172.21.15.83:0/1879097657 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff7000b1a20 0x7ff7000cc390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.207+0000 7ff7065bc640 1 --2- 172.21.15.83:0/1879097657 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7000b0270 0x7ff7000cad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.207+0000 7ff7065bc640 1 -- 172.21.15.83:0/1879097657 >> 172.21.15.83:0/1879097657 conn(0x7ff70009a9f0 msgr2=0x7ff7000b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.207+0000 7ff7065bc640 1 -- 172.21.15.83:0/1879097657 shutdown_connections 2024-03-06T07:22:34.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.207+0000 7ff7065bc640 1 -- 172.21.15.83:0/1879097657 wait complete. 2024-03-06T07:22:34.220 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:22:34.220 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:34.235 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:34.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.374+0000 7f5354bd7640 1 Processor -- start 2024-03-06T07:22:34.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.374+0000 7f5354bd7640 1 -- start start 2024-03-06T07:22:34.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.374+0000 7f5354bd7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f535015bb20 0x7f53500d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.375+0000 7f5354bd7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f53500cfff0 0x7f53500d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.375+0000 7f5354bd7640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f53500ce4b0 0x7f53500ce940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.375+0000 7f5354bd7640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5350161980 con 0x7f53500cfff0 2024-03-06T07:22:34.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.375+0000 7f5354bd7640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5350161c60 con 0x7f53500ce4b0 2024-03-06T07:22:34.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.375+0000 7f5354bd7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5350161f40 con 0x7f535015bb20 2024-03-06T07:22:34.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.375+0000 7f5345d9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f53500cfff0 0x7f53500d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.375+0000 7f5345d9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f53500cfff0 0x7f53500d0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48752/0 (socket says 172.21.15.83:48752) 2024-03-06T07:22:34.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.375+0000 7f5345d9b640 1 -- 172.21.15.83:0/1117487517 learned_addr learned my addr 172.21.15.83:0/1117487517 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:34.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.375+0000 7f534e59c640 1 --2- 172.21.15.83:0/1117487517 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f535015bb20 0x7f53500d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.375+0000 7f534ed9d640 1 --2- 172.21.15.83:0/1117487517 >> v2:172.21.15.183:3300/0 conn(0x7f53500ce4b0 0x7f53500ce940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.375+0000 7f5345d9b640 1 -- 172.21.15.83:0/1117487517 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f535015bb20 msgr2=0x7f53500d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:34.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.375+0000 7f5345d9b640 1 --2- 172.21.15.83:0/1117487517 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f535015bb20 0x7f53500d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.375+0000 7f5345d9b640 1 -- 172.21.15.83:0/1117487517 >> v2:172.21.15.183:3300/0 conn(0x7f53500ce4b0 msgr2=0x7f53500ce940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:34.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.375+0000 7f5345d9b640 1 --2- 172.21.15.83:0/1117487517 >> v2:172.21.15.183:3300/0 conn(0x7f53500ce4b0 0x7f53500ce940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.375+0000 7f5345d9b640 1 -- 172.21.15.83:0/1117487517 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5350162220 con 0x7f53500cfff0 2024-03-06T07:22:34.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.375+0000 7f534ed9d640 1 --2- 172.21.15.83:0/1117487517 >> v2:172.21.15.183:3300/0 conn(0x7f53500ce4b0 0x7f53500ce940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:34.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.375+0000 7f534e59c640 1 --2- 172.21.15.83:0/1117487517 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f535015bb20 0x7f53500d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:34.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.376+0000 7f5345d9b640 1 --2- 172.21.15.83:0/1117487517 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53500cfff0 0x7f53500d0470 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f53500b13b0 tx=0x7f533802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82ac3f586a36a904 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.376+0000 7f534dd9b640 1 -- 172.21.15.83:0/1117487517 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5338035020 con 0x7f53500cfff0 2024-03-06T07:22:34.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.376+0000 7f534dd9b640 1 -- 172.21.15.83:0/1117487517 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f533802d040 con 0x7f53500cfff0 2024-03-06T07:22:34.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.376+0000 7f534dd9b640 1 -- 172.21.15.83:0/1117487517 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5338031a80 con 0x7f53500cfff0 2024-03-06T07:22:34.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.376+0000 7f5354bd7640 1 -- 172.21.15.83:0/1117487517 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53500cfff0 msgr2=0x7f53500d0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.376+0000 7f5354bd7640 1 --2- 172.21.15.83:0/1117487517 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53500cfff0 0x7f53500d0470 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f53500b13b0 tx=0x7f533802b900 comp rx=0 tx=0).stop 2024-03-06T07:22:34.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.376+0000 7f5354bd7640 1 -- 172.21.15.83:0/1117487517 shutdown_connections 2024-03-06T07:22:34.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.376+0000 7f5354bd7640 1 --2- 172.21.15.83:0/1117487517 >> v2:172.21.15.183:3300/0 conn(0x7f53500ce4b0 0x7f53500ce940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.376+0000 7f5354bd7640 1 --2- 172.21.15.83:0/1117487517 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53500cfff0 0x7f53500d0470 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.376+0000 7f5354bd7640 1 --2- 172.21.15.83:0/1117487517 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f535015bb20 0x7f53500d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.376+0000 7f5354bd7640 1 -- 172.21.15.83:0/1117487517 >> 172.21.15.83:0/1117487517 conn(0x7f535009a9f0 msgr2=0x7f535009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.376+0000 7f5354bd7640 1 -- 172.21.15.83:0/1117487517 shutdown_connections 2024-03-06T07:22:34.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.377+0000 7f5354bd7640 1 -- 172.21.15.83:0/1117487517 wait complete. 2024-03-06T07:22:34.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.378+0000 7f5354bd7640 1 Processor -- start 2024-03-06T07:22:34.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.378+0000 7f5354bd7640 1 -- start start 2024-03-06T07:22:34.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.379+0000 7f5354bd7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53500ce4b0 0x7f53500b4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.379+0000 7f5354bd7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53500cfff0 0x7f535015cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.379+0000 7f5354bd7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f535015bb20 0x7f535015d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.379+0000 7f5354bd7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53500d24b0 con 0x7f53500ce4b0 2024-03-06T07:22:34.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.379+0000 7f5354bd7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53500d2330 con 0x7f53500cfff0 2024-03-06T07:22:34.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.379+0000 7f5354bd7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53500d1980 con 0x7f535015bb20 2024-03-06T07:22:34.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.379+0000 7f534e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53500ce4b0 0x7f53500b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.379+0000 7f534e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53500ce4b0 0x7f53500b4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48760/0 (socket says 172.21.15.83:48760) 2024-03-06T07:22:34.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.379+0000 7f534e59c640 1 -- 172.21.15.83:0/1140882415 learned_addr learned my addr 172.21.15.83:0/1140882415 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:34.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.379+0000 7f534ed9d640 1 --2- 172.21.15.83:0/1140882415 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f535015bb20 0x7f535015d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.379+0000 7f5345d9b640 1 --2- 172.21.15.83:0/1140882415 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53500cfff0 0x7f535015cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.379+0000 7f534e59c640 1 -- 172.21.15.83:0/1140882415 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f535015bb20 msgr2=0x7f535015d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.379+0000 7f534e59c640 1 --2- 172.21.15.83:0/1140882415 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f535015bb20 0x7f535015d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.379+0000 7f534e59c640 1 -- 172.21.15.83:0/1140882415 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53500cfff0 msgr2=0x7f535015cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.379+0000 7f534e59c640 1 --2- 172.21.15.83:0/1140882415 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53500cfff0 0x7f535015cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.379+0000 7f534e59c640 1 -- 172.21.15.83:0/1140882415 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f535006bf00 con 0x7f53500ce4b0 2024-03-06T07:22:34.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.379+0000 7f5345d9b640 1 --2- 172.21.15.83:0/1140882415 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53500cfff0 0x7f535015cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:34.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.379+0000 7f534e59c640 1 --2- 172.21.15.83:0/1140882415 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53500ce4b0 0x7f53500b4650 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f534000d7d0 tx=0x7f534000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.379+0000 7f53477fe640 1 -- 172.21.15.83:0/1140882415 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5340004030 con 0x7f53500ce4b0 2024-03-06T07:22:34.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.379+0000 7f53477fe640 1 -- 172.21.15.83:0/1140882415 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53400041d0 con 0x7f53500ce4b0 2024-03-06T07:22:34.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.379+0000 7f5354bd7640 1 -- 172.21.15.83:0/1140882415 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f535006c170 con 0x7f53500ce4b0 2024-03-06T07:22:34.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.379+0000 7f53477fe640 1 -- 172.21.15.83:0/1140882415 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53400096a0 con 0x7f53500ce4b0 2024-03-06T07:22:34.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.380+0000 7f5354bd7640 1 -- 172.21.15.83:0/1140882415 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53500b4b90 con 0x7f53500ce4b0 2024-03-06T07:22:34.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.381+0000 7f5354bd7640 1 -- 172.21.15.83:0/1140882415 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f535004f330 con 0x7f53500ce4b0 2024-03-06T07:22:34.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.381+0000 7f53477fe640 1 -- 172.21.15.83:0/1140882415 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5340009840 con 0x7f53500ce4b0 2024-03-06T07:22:34.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.381+0000 7f53477fe640 1 --2- 172.21.15.83:0/1140882415 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f533006b4d0 0x7f533006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.381+0000 7f53477fe640 1 -- 172.21.15.83:0/1140882415 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f534008a4a0 con 0x7f53500ce4b0 2024-03-06T07:22:34.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.381+0000 7f5345d9b640 1 --2- 172.21.15.83:0/1140882415 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f533006b4d0 0x7f533006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.382+0000 7f5345d9b640 1 --2- 172.21.15.83:0/1140882415 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f533006b4d0 0x7f533006d990 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5338036000 tx=0x7f533803c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.386+0000 7f53477fe640 1 -- 172.21.15.83:0/1140882415 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f53400593b0 con 0x7f53500ce4b0 2024-03-06T07:22:34.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.671+0000 7f5354bd7640 1 -- 172.21.15.83:0/1140882415 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53500b1820 con 0x7f53500ce4b0 2024-03-06T07:22:34.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.672+0000 7f53477fe640 1 -- 172.21.15.83:0/1140882415 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16617 (secure 0 0 0) 0x7f5340058cb0 con 0x7f53500ce4b0 2024-03-06T07:22:34.674 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:22:34.674 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":20,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:21:59.938137+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.318942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.254795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:34.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.674+0000 7f5354bd7640 1 -- 172.21.15.83:0/1140882415 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f533006b4d0 msgr2=0x7f533006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.674+0000 7f5354bd7640 1 --2- 172.21.15.83:0/1140882415 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f533006b4d0 0x7f533006d990 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5338036000 tx=0x7f533803c040 comp rx=0 tx=0).stop 2024-03-06T07:22:34.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.674+0000 7f5354bd7640 1 -- 172.21.15.83:0/1140882415 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53500ce4b0 msgr2=0x7f53500b4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.674+0000 7f5354bd7640 1 --2- 172.21.15.83:0/1140882415 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53500ce4b0 0x7f53500b4650 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f534000d7d0 tx=0x7f534000dc90 comp rx=0 tx=0).stop 2024-03-06T07:22:34.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.674+0000 7f5354bd7640 1 -- 172.21.15.83:0/1140882415 shutdown_connections 2024-03-06T07:22:34.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.674+0000 7f5354bd7640 1 --2- 172.21.15.83:0/1140882415 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f533006b4d0 0x7f533006d990 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.674+0000 7f5354bd7640 1 --2- 172.21.15.83:0/1140882415 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f535015bb20 0x7f535015d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.674+0000 7f5354bd7640 1 --2- 172.21.15.83:0/1140882415 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53500cfff0 0x7f535015cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.674+0000 7f5354bd7640 1 --2- 172.21.15.83:0/1140882415 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53500ce4b0 0x7f53500b4650 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.674+0000 7f5354bd7640 1 -- 172.21.15.83:0/1140882415 >> 172.21.15.83:0/1140882415 conn(0x7f535009a9f0 msgr2=0x7f535009add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.674+0000 7f5354bd7640 1 -- 172.21.15.83:0/1140882415 shutdown_connections 2024-03-06T07:22:34.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.674+0000 7f5354bd7640 1 -- 172.21.15.83:0/1140882415 wait complete. 2024-03-06T07:22:34.687 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:56.318942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:57.254795+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:34.688 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:34.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.789+0000 7f06905a1640 1 Processor -- start 2024-03-06T07:22:34.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.789+0000 7f06905a1640 1 -- start start 2024-03-06T07:22:34.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.790+0000 7f06905a1640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f06881586e0 0x7f06880aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.790+0000 7f06905a1640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f06880d01a0 0x7f06880b33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.790+0000 7f06905a1640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0688158ac0 con 0x7f06880aef80 2024-03-06T07:22:34.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.790+0000 7f06905a1640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f068815b2a0 con 0x7f06881586e0 2024-03-06T07:22:34.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.790+0000 7f06905a1640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f068815b540 con 0x7f06880d01a0 2024-03-06T07:22:34.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.790+0000 7f068db3c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f06880aef80 0x7f06880cfa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39464/0 (socket says 172.21.15.83:39464) 2024-03-06T07:22:34.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.790+0000 7f068db3c640 1 -- 172.21.15.83:0/2002966015 learned_addr learned my addr 172.21.15.83:0/2002966015 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:34.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.790+0000 7f068eb3e640 1 --2- 172.21.15.83:0/2002966015 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f06880d01a0 0x7f06880b33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.790+0000 7f068e33d640 1 --2- 172.21.15.83:0/2002966015 >> v2:172.21.15.183:3300/0 conn(0x7f06881586e0 0x7f06880aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.791+0000 7f068d33b640 1 -- 172.21.15.83:0/2002966015 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3003740293 0 0) 0x7f0688158ac0 con 0x7f06880aef80 2024-03-06T07:22:34.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.791+0000 7f068d33b640 1 -- 172.21.15.83:0/2002966015 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0674003520 con 0x7f06880aef80 2024-03-06T07:22:34.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.791+0000 7f068eb3e640 1 -- 172.21.15.83:0/2002966015 >> v2:172.21.15.183:3300/0 conn(0x7f06881586e0 msgr2=0x7f06880aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:34.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.791+0000 7f068eb3e640 1 --2- 172.21.15.83:0/2002966015 >> v2:172.21.15.183:3300/0 conn(0x7f06881586e0 0x7f06880aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.791+0000 7f068eb3e640 1 -- 172.21.15.83:0/2002966015 >> v1:172.21.15.83:6789/0 conn(0x7f06880aef80 legacy=0x7f06880cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.791+0000 7f068eb3e640 1 -- 172.21.15.83:0/2002966015 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f068815b820 con 0x7f06880d01a0 2024-03-06T07:22:34.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.791+0000 7f068e33d640 1 --2- 172.21.15.83:0/2002966015 >> v2:172.21.15.183:3300/0 conn(0x7f06881586e0 0x7f06880aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:34.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.792+0000 7f068eb3e640 1 --2- 172.21.15.83:0/2002966015 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f06880d01a0 0x7f06880b33a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f06840097a0 tx=0x7f068402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c00fff17b42de46 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.792+0000 7f068d33b640 1 -- 172.21.15.83:0/2002966015 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0684035020 con 0x7f06880d01a0 2024-03-06T07:22:34.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.792+0000 7f068d33b640 1 -- 172.21.15.83:0/2002966015 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0684031830 con 0x7f06880d01a0 2024-03-06T07:22:34.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.792+0000 7f068d33b640 1 -- 172.21.15.83:0/2002966015 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0684031be0 con 0x7f06880d01a0 2024-03-06T07:22:34.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.792+0000 7f06905a1640 1 -- 172.21.15.83:0/2002966015 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f06880d01a0 msgr2=0x7f06880b33a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.792+0000 7f06905a1640 1 --2- 172.21.15.83:0/2002966015 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f06880d01a0 0x7f06880b33a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f06840097a0 tx=0x7f068402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.792+0000 7f06905a1640 1 -- 172.21.15.83:0/2002966015 shutdown_connections 2024-03-06T07:22:34.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.792+0000 7f06905a1640 1 --2- 172.21.15.83:0/2002966015 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f06880d01a0 0x7f06880b33a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.792+0000 7f06905a1640 1 --2- 172.21.15.83:0/2002966015 >> v2:172.21.15.183:3300/0 conn(0x7f06881586e0 0x7f06880aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.792+0000 7f06905a1640 1 -- 172.21.15.83:0/2002966015 >> 172.21.15.83:0/2002966015 conn(0x7f068806b190 msgr2=0x7f068806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.792+0000 7f06905a1640 1 -- 172.21.15.83:0/2002966015 shutdown_connections 2024-03-06T07:22:34.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.792+0000 7f06905a1640 1 -- 172.21.15.83:0/2002966015 wait complete. 2024-03-06T07:22:34.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.794+0000 7f06905a1640 1 Processor -- start 2024-03-06T07:22:34.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.794+0000 7f06905a1640 1 -- start start 2024-03-06T07:22:34.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.794+0000 7f06905a1640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f06880aef80 0x7f068814a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.794+0000 7f06905a1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06880d01a0 0x7f06881527a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.794+0000 7f06905a1640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f06881586e0 0x7f06880ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.794+0000 7f06905a1640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0688071350 con 0x7f06880d01a0 2024-03-06T07:22:34.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.794+0000 7f06905a1640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f068815bde0 con 0x7f06880aef80 2024-03-06T07:22:34.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.794+0000 7f06905a1640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06880714d0 con 0x7f06881586e0 2024-03-06T07:22:34.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.794+0000 7f068db3c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06880d01a0 0x7f06881527a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.794+0000 7f068db3c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06880d01a0 0x7f06881527a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48764/0 (socket says 172.21.15.83:48764) 2024-03-06T07:22:34.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.794+0000 7f068db3c640 1 -- 172.21.15.83:0/3432894591 learned_addr learned my addr 172.21.15.83:0/3432894591 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:34.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.794+0000 7f068eb3e640 1 --2- 172.21.15.83:0/3432894591 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f06881586e0 0x7f06880ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.794+0000 7f068e33d640 1 --2- 172.21.15.83:0/3432894591 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f06880aef80 0x7f068814a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.795+0000 7f068db3c640 1 -- 172.21.15.83:0/3432894591 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f06881586e0 msgr2=0x7f06880ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.795+0000 7f068db3c640 1 --2- 172.21.15.83:0/3432894591 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f06881586e0 0x7f06880ad860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.795+0000 7f068db3c640 1 -- 172.21.15.83:0/3432894591 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f06880aef80 msgr2=0x7f068814a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.795+0000 7f068db3c640 1 --2- 172.21.15.83:0/3432894591 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f06880aef80 0x7f068814a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.795+0000 7f068db3c640 1 -- 172.21.15.83:0/3432894591 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06881f80c0 con 0x7f06880d01a0 2024-03-06T07:22:34.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.795+0000 7f068db3c640 1 --2- 172.21.15.83:0/3432894591 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06880d01a0 0x7f06881527a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f067c00c400 tx=0x7f067c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.795+0000 7f067affd640 1 -- 172.21.15.83:0/3432894591 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f067c017070 con 0x7f06880d01a0 2024-03-06T07:22:34.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.795+0000 7f067affd640 1 -- 172.21.15.83:0/3432894591 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f067c005d00 con 0x7f06880d01a0 2024-03-06T07:22:34.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.795+0000 7f06905a1640 1 -- 172.21.15.83:0/3432894591 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06881f7540 con 0x7f06880d01a0 2024-03-06T07:22:34.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.795+0000 7f067affd640 1 -- 172.21.15.83:0/3432894591 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f067c007310 con 0x7f06880d01a0 2024-03-06T07:22:34.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.795+0000 7f06905a1640 1 -- 172.21.15.83:0/3432894591 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06881f7b00 con 0x7f06880d01a0 2024-03-06T07:22:34.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.796+0000 7f06905a1640 1 -- 172.21.15.83:0/3432894591 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06880732b0 con 0x7f06880d01a0 2024-03-06T07:22:34.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.796+0000 7f067affd640 1 -- 172.21.15.83:0/3432894591 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f067c01a070 con 0x7f06880d01a0 2024-03-06T07:22:34.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.797+0000 7f067affd640 1 --2- 172.21.15.83:0/3432894591 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f066c06b500 0x7f066c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.797+0000 7f068e33d640 1 --2- 172.21.15.83:0/3432894591 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f066c06b500 0x7f066c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.797+0000 7f067affd640 1 -- 172.21.15.83:0/3432894591 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f067c089590 con 0x7f06880d01a0 2024-03-06T07:22:34.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.797+0000 7f068e33d640 1 --2- 172.21.15.83:0/3432894591 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f066c06b500 0x7f066c06d9c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0680009bd0 tx=0x7f068000b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.801+0000 7f067affd640 1 -- 172.21.15.83:0/3432894591 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f067c057dd0 con 0x7f06880d01a0 2024-03-06T07:22:34.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.977+0000 7f06905a1640 1 -- 172.21.15.83:0/3432894591 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0688068ff0 con 0x7f06880d01a0 2024-03-06T07:22:34.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.978+0000 7f067affd640 1 -- 172.21.15.83:0/3432894591 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f067c05be70 con 0x7f06880d01a0 2024-03-06T07:22:34.980 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:22:34.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.980+0000 7f06905a1640 1 -- 172.21.15.83:0/3432894591 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f066c06b500 msgr2=0x7f066c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.980+0000 7f06905a1640 1 --2- 172.21.15.83:0/3432894591 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f066c06b500 0x7f066c06d9c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0680009bd0 tx=0x7f068000b040 comp rx=0 tx=0).stop 2024-03-06T07:22:34.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.980+0000 7f06905a1640 1 -- 172.21.15.83:0/3432894591 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06880d01a0 msgr2=0x7f06881527a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.980+0000 7f06905a1640 1 --2- 172.21.15.83:0/3432894591 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06880d01a0 0x7f06881527a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f067c00c400 tx=0x7f067c00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.980+0000 7f06905a1640 1 -- 172.21.15.83:0/3432894591 shutdown_connections 2024-03-06T07:22:34.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.980+0000 7f06905a1640 1 --2- 172.21.15.83:0/3432894591 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f066c06b500 0x7f066c06d9c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.980+0000 7f06905a1640 1 --2- 172.21.15.83:0/3432894591 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f06881586e0 0x7f06880ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.980+0000 7f06905a1640 1 --2- 172.21.15.83:0/3432894591 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06880d01a0 0x7f06881527a0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.980+0000 7f06905a1640 1 --2- 172.21.15.83:0/3432894591 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f06880aef80 0x7f068814a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.980+0000 7f06905a1640 1 -- 172.21.15.83:0/3432894591 >> 172.21.15.83:0/3432894591 conn(0x7f068806b190 msgr2=0x7f06880b2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.980+0000 7f06905a1640 1 -- 172.21.15.83:0/3432894591 shutdown_connections 2024-03-06T07:22:34.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:34.980+0000 7f06905a1640 1 -- 172.21.15.83:0/3432894591 wait complete. 2024-03-06T07:22:34.992 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:35.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.107+0000 7f7416cec640 1 Processor -- start 2024-03-06T07:22:35.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.107+0000 7f7416cec640 1 -- start start 2024-03-06T07:22:35.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.108+0000 7f7416cec640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f741015bbb0 0x7f741006afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.108+0000 7f7416cec640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74100cea90 con 0x7f741015ee10 2024-03-06T07:22:35.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.108+0000 7f7416cec640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7410161bc0 con 0x7f741006b4f0 2024-03-06T07:22:35.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.108+0000 7f7416cec640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7410161970 con 0x7f741015bbb0 2024-03-06T07:22:35.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.108+0000 7f7415289640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f741015ee10 0x7f7410071ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39470/0 (socket says 172.21.15.83:39470) 2024-03-06T07:22:35.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.108+0000 7f7415289640 1 -- 172.21.15.83:0/1009363840 learned_addr learned my addr 172.21.15.83:0/1009363840 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:35.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.108+0000 7f740f7fe640 1 -- 172.21.15.83:0/1009363840 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3298539264 0 0) 0x7f73f8001110 con 0x7f741015ee10 2024-03-06T07:22:35.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.108+0000 7f7414a88640 1 --2- 172.21.15.83:0/1009363840 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f741015bbb0 0x7f741006afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.108+0000 7f740f7fe640 1 -- 172.21.15.83:0/1009363840 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73fc003520 con 0x7f741015ee10 2024-03-06T07:22:35.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.109+0000 7f740f7fe640 1 -- 172.21.15.83:0/1009363840 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2149412746 0 0) 0x7f73fc003520 con 0x7f741015ee10 2024-03-06T07:22:35.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.109+0000 7f740f7fe640 1 -- 172.21.15.83:0/1009363840 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f741015bbb0 msgr2=0x7f741006afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:35.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.109+0000 7f740f7fe640 1 --2- 172.21.15.83:0/1009363840 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f741015bbb0 0x7f741006afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.109+0000 7f740f7fe640 1 -- 172.21.15.83:0/1009363840 >> v1:172.21.15.183:6789/0 conn(0x7f741006b4f0 legacy=0x7f741015e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.109+0000 7f740f7fe640 1 -- 172.21.15.83:0/1009363840 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7410161ea0 con 0x7f741015ee10 2024-03-06T07:22:35.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.109+0000 7f740f7fe640 1 -- 172.21.15.83:0/1009363840 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f73f8001840 con 0x7f741015ee10 2024-03-06T07:22:35.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.109+0000 7f740f7fe640 1 -- 172.21.15.83:0/1009363840 >> v1:172.21.15.83:6789/0 conn(0x7f741015ee10 legacy=0x7f7410071ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.109+0000 7f740f7fe640 1 --2- 172.21.15.83:0/1009363840 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73fc0039e0 0x7f73fc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.109+0000 7f740f7fe640 1 --2- 172.21.15.83:0/1009363840 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73fc004800 0x7f73fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.109+0000 7f740f7fe640 1 --2- 172.21.15.83:0/1009363840 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f73fc0051c0 0x7f73fc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.109+0000 7f740f7fe640 1 -- 172.21.15.83:0/1009363840 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7410161970 con 0x7f73fc004800 2024-03-06T07:22:35.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.109+0000 7f740f7fe640 1 -- 172.21.15.83:0/1009363840 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc00bf00 con 0x7f73fc0039e0 2024-03-06T07:22:35.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.109+0000 7f740f7fe640 1 -- 172.21.15.83:0/1009363840 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc00c080 con 0x7f73fc0051c0 2024-03-06T07:22:35.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.109+0000 7f740ffff640 1 --2- 172.21.15.83:0/1009363840 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73fc004800 0x7f73fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.109+0000 7f7414a88640 1 --2- 172.21.15.83:0/1009363840 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73fc0039e0 0x7f73fc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.109+0000 7f7415289640 1 --2- 172.21.15.83:0/1009363840 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f73fc0051c0 0x7f73fc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.109+0000 7f740ffff640 1 -- 172.21.15.83:0/1009363840 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f73fc0051c0 msgr2=0x7f73fc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:35.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.109+0000 7f740ffff640 1 --2- 172.21.15.83:0/1009363840 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f73fc0051c0 0x7f73fc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.109+0000 7f740ffff640 1 -- 172.21.15.83:0/1009363840 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73fc0039e0 msgr2=0x7f73fc003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:35.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.109+0000 7f740ffff640 1 --2- 172.21.15.83:0/1009363840 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73fc0039e0 0x7f73fc003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.109+0000 7f740ffff640 1 -- 172.21.15.83:0/1009363840 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73fc003520 con 0x7f73fc004800 2024-03-06T07:22:35.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.110+0000 7f740ffff640 1 --2- 172.21.15.83:0/1009363840 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73fc004800 0x7f73fc004c80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7400001750 tx=0x7f73fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72e7c845d27d7458 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.110+0000 7f740f7fe640 1 -- 172.21.15.83:0/1009363840 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7400001370 con 0x7f73fc004800 2024-03-06T07:22:35.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.110+0000 7f740f7fe640 1 -- 172.21.15.83:0/1009363840 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74000052d0 con 0x7f73fc004800 2024-03-06T07:22:35.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.110+0000 7f740f7fe640 1 -- 172.21.15.83:0/1009363840 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7400003830 con 0x7f73fc004800 2024-03-06T07:22:35.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.110+0000 7f7416cec640 1 -- 172.21.15.83:0/1009363840 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73fc004800 msgr2=0x7f73fc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.110+0000 7f7416cec640 1 --2- 172.21.15.83:0/1009363840 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73fc004800 0x7f73fc004c80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7400001750 tx=0x7f73fc005830 comp rx=0 tx=0).stop 2024-03-06T07:22:35.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.110+0000 7f7415289640 1 -- 172.21.15.83:0/1009363840 reap_dead start 2024-03-06T07:22:35.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.111+0000 7f7416cec640 1 -- 172.21.15.83:0/1009363840 shutdown_connections 2024-03-06T07:22:35.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.111+0000 7f7416cec640 1 -- 172.21.15.83:0/1009363840 >> 172.21.15.83:0/1009363840 conn(0x7f74100b2660 msgr2=0x7f74100b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:35.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.111+0000 7f7416cec640 1 -- 172.21.15.83:0/1009363840 shutdown_connections 2024-03-06T07:22:35.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.111+0000 7f7416cec640 1 -- 172.21.15.83:0/1009363840 wait complete. 2024-03-06T07:22:35.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.112+0000 7f7416cec640 1 Processor -- start 2024-03-06T07:22:35.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.112+0000 7f7416cec640 1 -- start start 2024-03-06T07:22:35.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.113+0000 7f7416cec640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f73fc004800 0x7f7410160090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.113+0000 7f7416cec640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f741014ce00 0x7f74100b3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.113+0000 7f7416cec640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f741015a160 0x7f74101607d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.113+0000 7f7416cec640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f741006afb0 con 0x7f741014ce00 2024-03-06T07:22:35.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.113+0000 7f7416cec640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74101624f0 con 0x7f741015a160 2024-03-06T07:22:35.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.113+0000 7f7416cec640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f741006b130 con 0x7f73fc004800 2024-03-06T07:22:35.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.113+0000 7f740ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f741014ce00 0x7f74100b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.113+0000 7f740ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f741014ce00 0x7f74100b3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48790/0 (socket says 172.21.15.83:48790) 2024-03-06T07:22:35.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.113+0000 7f740ffff640 1 -- 172.21.15.83:0/3773535410 learned_addr learned my addr 172.21.15.83:0/3773535410 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:35.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.113+0000 7f7414a88640 1 --2- 172.21.15.83:0/3773535410 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f73fc004800 0x7f7410160090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.113+0000 7f740ffff640 1 -- 172.21.15.83:0/3773535410 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f73fc004800 msgr2=0x7f7410160090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.113+0000 7f7415289640 1 --2- 172.21.15.83:0/3773535410 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f741015a160 0x7f74101607d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.113+0000 7f740ffff640 1 --2- 172.21.15.83:0/3773535410 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f73fc004800 0x7f7410160090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.113+0000 7f740ffff640 1 -- 172.21.15.83:0/3773535410 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f741015a160 msgr2=0x7f74101607d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.113+0000 7f740ffff640 1 --2- 172.21.15.83:0/3773535410 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f741015a160 0x7f74101607d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.113+0000 7f740ffff640 1 -- 172.21.15.83:0/3773535410 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f741015aad0 con 0x7f741014ce00 2024-03-06T07:22:35.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.113+0000 7f7414a88640 1 --2- 172.21.15.83:0/3773535410 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f73fc004800 0x7f7410160090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:35.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.113+0000 7f7415289640 1 --2- 172.21.15.83:0/3773535410 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f741015a160 0x7f74101607d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:35.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.113+0000 7f740ffff640 1 --2- 172.21.15.83:0/3773535410 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f741014ce00 0x7f74100b3750 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7400008fd0 tx=0x7f7400001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.113+0000 7f740d7fa640 1 -- 172.21.15.83:0/3773535410 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7400041070 con 0x7f741014ce00 2024-03-06T07:22:35.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.113+0000 7f740d7fa640 1 -- 172.21.15.83:0/3773535410 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7400005480 con 0x7f741014ce00 2024-03-06T07:22:35.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.113+0000 7f7416cec640 1 -- 172.21.15.83:0/3773535410 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74100cb1d0 con 0x7f741014ce00 2024-03-06T07:22:35.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.113+0000 7f740d7fa640 1 -- 172.21.15.83:0/3773535410 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7400007a30 con 0x7f741014ce00 2024-03-06T07:22:35.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.113+0000 7f7416cec640 1 -- 172.21.15.83:0/3773535410 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f741015a5e0 con 0x7f741014ce00 2024-03-06T07:22:35.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.114+0000 7f7416cec640 1 -- 172.21.15.83:0/3773535410 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74100733b0 con 0x7f741014ce00 2024-03-06T07:22:35.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.115+0000 7f740d7fa640 1 -- 172.21.15.83:0/3773535410 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7400044070 con 0x7f741014ce00 2024-03-06T07:22:35.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.115+0000 7f740d7fa640 1 --2- 172.21.15.83:0/3773535410 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f73f006b4d0 0x7f73f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.115+0000 7f740d7fa640 1 -- 172.21.15.83:0/3773535410 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f74000abc80 con 0x7f741014ce00 2024-03-06T07:22:35.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.115+0000 7f7414a88640 1 --2- 172.21.15.83:0/3773535410 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f73f006b4d0 0x7f73f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.116+0000 7f7414a88640 1 --2- 172.21.15.83:0/3773535410 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f73f006b4d0 0x7f73f006d990 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7408007440 tx=0x7f740800b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.120+0000 7f740d7fa640 1 -- 172.21.15.83:0/3773535410 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f740007a660 con 0x7f741014ce00 2024-03-06T07:22:35.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.296+0000 7f7416cec640 1 -- 172.21.15.83:0/3773535410 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7410068ff0 con 0x7f741014ce00 2024-03-06T07:22:35.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.297+0000 7f740d7fa640 1 -- 172.21.15.83:0/3773535410 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f740007e700 con 0x7f741014ce00 2024-03-06T07:22:35.299 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-03-06T07:22:35.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.299+0000 7f7416cec640 1 -- 172.21.15.83:0/3773535410 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f73f006b4d0 msgr2=0x7f73f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.299+0000 7f7416cec640 1 --2- 172.21.15.83:0/3773535410 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f73f006b4d0 0x7f73f006d990 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7408007440 tx=0x7f740800b040 comp rx=0 tx=0).stop 2024-03-06T07:22:35.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.299+0000 7f7416cec640 1 -- 172.21.15.83:0/3773535410 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f741014ce00 msgr2=0x7f74100b3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.299+0000 7f7416cec640 1 --2- 172.21.15.83:0/3773535410 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f741014ce00 0x7f74100b3750 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7400008fd0 tx=0x7f7400001120 comp rx=0 tx=0).stop 2024-03-06T07:22:35.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.299+0000 7f7416cec640 1 -- 172.21.15.83:0/3773535410 shutdown_connections 2024-03-06T07:22:35.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.299+0000 7f7416cec640 1 --2- 172.21.15.83:0/3773535410 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f73f006b4d0 0x7f73f006d990 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.299+0000 7f7416cec640 1 --2- 172.21.15.83:0/3773535410 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f741015a160 0x7f74101607d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.299+0000 7f7416cec640 1 --2- 172.21.15.83:0/3773535410 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f741014ce00 0x7f74100b3750 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.299+0000 7f7416cec640 1 --2- 172.21.15.83:0/3773535410 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f73fc004800 0x7f7410160090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.299+0000 7f7416cec640 1 -- 172.21.15.83:0/3773535410 >> 172.21.15.83:0/3773535410 conn(0x7f74100b2660 msgr2=0x7f74100b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:35.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.299+0000 7f7416cec640 1 -- 172.21.15.83:0/3773535410 shutdown_connections 2024-03-06T07:22:35.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.299+0000 7f7416cec640 1 -- 172.21.15.83:0/3773535410 wait complete. 2024-03-06T07:22:35.312 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:35.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.414+0000 7f900a11e640 1 Processor -- start 2024-03-06T07:22:35.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.414+0000 7f900a11e640 1 -- start start 2024-03-06T07:22:35.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.415+0000 7f900a11e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f900415ecd0 0x7f90040cae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.415+0000 7f900a11e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90040cfa50 con 0x7f90040d10c0 2024-03-06T07:22:35.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.415+0000 7f900a11e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90040b45c0 con 0x7f90040d14a0 2024-03-06T07:22:35.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.415+0000 7f900a11e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f900415c050 con 0x7f900415ecd0 2024-03-06T07:22:35.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.415+0000 7f9003fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f900415ecd0 0x7f90040cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.415+0000 7f9003fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f900415ecd0 0x7f90040cae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48792/0 (socket says 172.21.15.83:48792) 2024-03-06T07:22:35.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.415+0000 7f9003fff640 1 -- 172.21.15.83:0/480870866 learned_addr learned my addr 172.21.15.83:0/480870866 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:35.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.415+0000 7f9003fff640 1 -- 172.21.15.83:0/480870866 >> v1:172.21.15.183:6789/0 conn(0x7f90040d14a0 legacy=0x7f900415e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:35.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.415+0000 7f9003fff640 1 -- 172.21.15.83:0/480870866 >> v1:172.21.15.83:6789/0 conn(0x7f90040d10c0 legacy=0x7f90040d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.415+0000 7f9003fff640 1 -- 172.21.15.83:0/480870866 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f900415f160 con 0x7f900415ecd0 2024-03-06T07:22:35.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.416+0000 7f9003fff640 1 --2- 172.21.15.83:0/480870866 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f900415ecd0 0x7f90040cae80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8ff4009870 tx=0x7f8ff402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1dd6da2391fa2b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.416+0000 7f90027fc640 1 -- 172.21.15.83:0/480870866 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff4034070 con 0x7f900415ecd0 2024-03-06T07:22:35.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.416+0000 7f90027fc640 1 -- 172.21.15.83:0/480870866 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ff4030860 con 0x7f900415ecd0 2024-03-06T07:22:35.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.416+0000 7f90027fc640 1 -- 172.21.15.83:0/480870866 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff4030c30 con 0x7f900415ecd0 2024-03-06T07:22:35.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.417+0000 7f900a11e640 1 -- 172.21.15.83:0/480870866 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f900415ecd0 msgr2=0x7f90040cae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.417+0000 7f900a11e640 1 --2- 172.21.15.83:0/480870866 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f900415ecd0 0x7f90040cae80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8ff4009870 tx=0x7f8ff402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.417+0000 7f900a11e640 1 -- 172.21.15.83:0/480870866 shutdown_connections 2024-03-06T07:22:35.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.417+0000 7f900a11e640 1 --2- 172.21.15.83:0/480870866 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f900415ecd0 0x7f90040cae80 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.417+0000 7f900a11e640 1 -- 172.21.15.83:0/480870866 >> 172.21.15.83:0/480870866 conn(0x7f90040b3800 msgr2=0x7f90040b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:35.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.417+0000 7f900a11e640 1 -- 172.21.15.83:0/480870866 shutdown_connections 2024-03-06T07:22:35.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.417+0000 7f900a11e640 1 -- 172.21.15.83:0/480870866 wait complete. 2024-03-06T07:22:35.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.418+0000 7f900a11e640 1 Processor -- start 2024-03-06T07:22:35.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.418+0000 7f900a11e640 1 -- start start 2024-03-06T07:22:35.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.419+0000 7f900a11e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90040d10c0 0x7f9004152290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.419+0000 7f900a11e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90040d14a0 0x7f90040cc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.419+0000 7f900a11e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9004152980 0x7f90040cdb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.419+0000 7f900a11e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90040cfcf0 con 0x7f90040d10c0 2024-03-06T07:22:35.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.419+0000 7f900a11e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90040d0060 con 0x7f90040d14a0 2024-03-06T07:22:35.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.419+0000 7f900a11e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90040cb3c0 con 0x7f9004152980 2024-03-06T07:22:35.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.419+0000 7f90037fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90040d10c0 0x7f9004152290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.419+0000 7f90037fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90040d10c0 0x7f9004152290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48808/0 (socket says 172.21.15.83:48808) 2024-03-06T07:22:35.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.419+0000 7f90037fe640 1 -- 172.21.15.83:0/4042700951 learned_addr learned my addr 172.21.15.83:0/4042700951 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:35.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.419+0000 7f9002ffd640 1 --2- 172.21.15.83:0/4042700951 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90040d14a0 0x7f90040cc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.419+0000 7f9003fff640 1 --2- 172.21.15.83:0/4042700951 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9004152980 0x7f90040cdb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.419+0000 7f90037fe640 1 -- 172.21.15.83:0/4042700951 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9004152980 msgr2=0x7f90040cdb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.419+0000 7f90037fe640 1 --2- 172.21.15.83:0/4042700951 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9004152980 0x7f90040cdb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.419+0000 7f90037fe640 1 -- 172.21.15.83:0/4042700951 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90040d14a0 msgr2=0x7f90040cc180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.419+0000 7f90037fe640 1 --2- 172.21.15.83:0/4042700951 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90040d14a0 0x7f90040cc180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.419+0000 7f90037fe640 1 -- 172.21.15.83:0/4042700951 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90040cca60 con 0x7f90040d10c0 2024-03-06T07:22:35.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.420+0000 7f90037fe640 1 --2- 172.21.15.83:0/4042700951 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90040d10c0 0x7f9004152290 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc00c400 tx=0x7f8ffc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.420+0000 7f8ffbfff640 1 -- 172.21.15.83:0/4042700951 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc0052d0 con 0x7f90040d10c0 2024-03-06T07:22:35.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.420+0000 7f8ffbfff640 1 -- 172.21.15.83:0/4042700951 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ffc005490 con 0x7f90040d10c0 2024-03-06T07:22:35.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.420+0000 7f900a11e640 1 -- 172.21.15.83:0/4042700951 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90040ccd40 con 0x7f90040d10c0 2024-03-06T07:22:35.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.420+0000 7f8ffbfff640 1 -- 172.21.15.83:0/4042700951 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc007260 con 0x7f90040d10c0 2024-03-06T07:22:35.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.420+0000 7f900a11e640 1 -- 172.21.15.83:0/4042700951 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90040cd300 con 0x7f90040d10c0 2024-03-06T07:22:35.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.421+0000 7f900a11e640 1 -- 172.21.15.83:0/4042700951 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9004068ff0 con 0x7f90040d10c0 2024-03-06T07:22:35.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.421+0000 7f8ffbfff640 1 -- 172.21.15.83:0/4042700951 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8ffc007400 con 0x7f90040d10c0 2024-03-06T07:22:35.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.421+0000 7f8ffbfff640 1 --2- 172.21.15.83:0/4042700951 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8fe406b4d0 0x7f8fe406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.422+0000 7f8ffbfff640 1 -- 172.21.15.83:0/4042700951 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8ffc089750 con 0x7f90040d10c0 2024-03-06T07:22:35.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.422+0000 7f9002ffd640 1 --2- 172.21.15.83:0/4042700951 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8fe406b4d0 0x7f8fe406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.422+0000 7f9002ffd640 1 --2- 172.21.15.83:0/4042700951 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8fe406b4d0 0x7f8fe406d990 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f90040b31a0 tx=0x7f8fec009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.430+0000 7f8ffbfff640 1 -- 172.21.15.83:0/4042700951 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8ffc0585d0 con 0x7f90040d10c0 2024-03-06T07:22:35.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.605+0000 7f900a11e640 1 -- 172.21.15.83:0/4042700951 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f90040d0c30 con 0x7f90040d10c0 2024-03-06T07:22:35.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.606+0000 7f8ffbfff640 1 -- 172.21.15.83:0/4042700951 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8ffc057ed0 con 0x7f90040d10c0 2024-03-06T07:22:35.608 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 64 2024-03-06T07:22:35.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.608+0000 7f900a11e640 1 -- 172.21.15.83:0/4042700951 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8fe406b4d0 msgr2=0x7f8fe406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.608+0000 7f900a11e640 1 --2- 172.21.15.83:0/4042700951 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8fe406b4d0 0x7f8fe406d990 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f90040b31a0 tx=0x7f8fec009040 comp rx=0 tx=0).stop 2024-03-06T07:22:35.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.608+0000 7f900a11e640 1 -- 172.21.15.83:0/4042700951 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90040d10c0 msgr2=0x7f9004152290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.608+0000 7f900a11e640 1 --2- 172.21.15.83:0/4042700951 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90040d10c0 0x7f9004152290 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc00c400 tx=0x7f8ffc00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.608+0000 7f900a11e640 1 -- 172.21.15.83:0/4042700951 shutdown_connections 2024-03-06T07:22:35.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.608+0000 7f900a11e640 1 --2- 172.21.15.83:0/4042700951 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8fe406b4d0 0x7f8fe406d990 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.608+0000 7f900a11e640 1 --2- 172.21.15.83:0/4042700951 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9004152980 0x7f90040cdb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.608+0000 7f900a11e640 1 --2- 172.21.15.83:0/4042700951 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90040d14a0 0x7f90040cc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.608+0000 7f900a11e640 1 --2- 172.21.15.83:0/4042700951 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90040d10c0 0x7f9004152290 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.608+0000 7f900a11e640 1 -- 172.21.15.83:0/4042700951 >> 172.21.15.83:0/4042700951 conn(0x7f90040b3800 msgr2=0x7f900415c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:35.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.608+0000 7f900a11e640 1 -- 172.21.15.83:0/4042700951 shutdown_connections 2024-03-06T07:22:35.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.608+0000 7f900a11e640 1 -- 172.21.15.83:0/4042700951 wait complete. 2024-03-06T07:22:35.621 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:22:35.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.720+0000 7fc87a91e640 1 Processor -- start 2024-03-06T07:22:35.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.720+0000 7fc87a91e640 1 -- start start 2024-03-06T07:22:35.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.720+0000 7fc87a91e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc8740b4c00 0x7fc8740b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.720+0000 7fc87a91e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc8740b28a0 0x7fc8740b2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.720+0000 7fc87a91e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fc87415e540 0x7fc87415e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.720+0000 7fc87a91e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc87415e0b0 con 0x7fc8740b4c00 2024-03-06T07:22:35.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.720+0000 7fc87a91e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc87415d510 con 0x7fc87415e540 2024-03-06T07:22:35.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.720+0000 7fc87a91e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc87415d780 con 0x7fc8740b28a0 2024-03-06T07:22:35.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.721+0000 7fc873fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc8740b4c00 0x7fc8740b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.721+0000 7fc873fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc8740b4c00 0x7fc8740b4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48812/0 (socket says 172.21.15.83:48812) 2024-03-06T07:22:35.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.721+0000 7fc873fff640 1 -- 172.21.15.83:0/2480737599 learned_addr learned my addr 172.21.15.83:0/2480737599 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:35.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.721+0000 7fc878ebb640 1 --2- 172.21.15.83:0/2480737599 >> v2:172.21.15.183:3300/0 conn(0x7fc87415e540 0x7fc87415e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.721+0000 7fc8737fe640 1 --2- 172.21.15.83:0/2480737599 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc8740b28a0 0x7fc8740b2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.721+0000 7fc873fff640 1 -- 172.21.15.83:0/2480737599 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc8740b28a0 msgr2=0x7fc8740b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:35.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.721+0000 7fc873fff640 1 --2- 172.21.15.83:0/2480737599 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc8740b28a0 0x7fc8740b2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.721+0000 7fc873fff640 1 -- 172.21.15.83:0/2480737599 >> v2:172.21.15.183:3300/0 conn(0x7fc87415e540 msgr2=0x7fc87415e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:35.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.721+0000 7fc873fff640 1 --2- 172.21.15.83:0/2480737599 >> v2:172.21.15.183:3300/0 conn(0x7fc87415e540 0x7fc87415e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.721+0000 7fc873fff640 1 -- 172.21.15.83:0/2480737599 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc87415da60 con 0x7fc8740b4c00 2024-03-06T07:22:35.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.722+0000 7fc873fff640 1 --2- 172.21.15.83:0/2480737599 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc8740b4c00 0x7fc8740b4fe0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fc87406ba80 tx=0x7fc86402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19b2b855e1fda6df server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.722+0000 7fc872ffd640 1 -- 172.21.15.83:0/2480737599 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc864034070 con 0x7fc8740b4c00 2024-03-06T07:22:35.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.722+0000 7fc872ffd640 1 -- 172.21.15.83:0/2480737599 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc864030860 con 0x7fc8740b4c00 2024-03-06T07:22:35.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.722+0000 7fc872ffd640 1 -- 172.21.15.83:0/2480737599 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc864030c30 con 0x7fc8740b4c00 2024-03-06T07:22:35.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.722+0000 7fc87a91e640 1 -- 172.21.15.83:0/2480737599 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc8740b4c00 msgr2=0x7fc8740b4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.722+0000 7fc87a91e640 1 --2- 172.21.15.83:0/2480737599 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc8740b4c00 0x7fc8740b4fe0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fc87406ba80 tx=0x7fc86402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.722+0000 7fc87a91e640 1 -- 172.21.15.83:0/2480737599 shutdown_connections 2024-03-06T07:22:35.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.722+0000 7fc87a91e640 1 --2- 172.21.15.83:0/2480737599 >> v2:172.21.15.183:3300/0 conn(0x7fc87415e540 0x7fc87415e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.722+0000 7fc87a91e640 1 --2- 172.21.15.83:0/2480737599 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc8740b28a0 0x7fc8740b2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.722+0000 7fc87a91e640 1 --2- 172.21.15.83:0/2480737599 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc8740b4c00 0x7fc8740b4fe0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.722+0000 7fc87a91e640 1 -- 172.21.15.83:0/2480737599 >> 172.21.15.83:0/2480737599 conn(0x7fc87409a9f0 msgr2=0x7fc87409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:35.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.722+0000 7fc87a91e640 1 -- 172.21.15.83:0/2480737599 shutdown_connections 2024-03-06T07:22:35.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.722+0000 7fc87a91e640 1 -- 172.21.15.83:0/2480737599 wait complete. 2024-03-06T07:22:35.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.724+0000 7fc87a91e640 1 Processor -- start 2024-03-06T07:22:35.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.724+0000 7fc87a91e640 1 -- start start 2024-03-06T07:22:35.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.724+0000 7fc87a91e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc8740b28a0 0x7fc87409ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.724+0000 7fc87a91e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc8740b4c00 0x7fc87415a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.724+0000 7fc87a91e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc87415e540 0x7fc8741f4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.724+0000 7fc87a91e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8740b33a0 con 0x7fc8740b4c00 2024-03-06T07:22:35.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.724+0000 7fc87a91e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8740b3220 con 0x7fc8740b28a0 2024-03-06T07:22:35.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.724+0000 7fc87a91e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc87415eec0 con 0x7fc87415e540 2024-03-06T07:22:35.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.725+0000 7fc8737fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc8740b4c00 0x7fc87415a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.725+0000 7fc8737fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc8740b4c00 0x7fc87415a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48828/0 (socket says 172.21.15.83:48828) 2024-03-06T07:22:35.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.725+0000 7fc8737fe640 1 -- 172.21.15.83:0/1643739688 learned_addr learned my addr 172.21.15.83:0/1643739688 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:35.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.725+0000 7fc873fff640 1 --2- 172.21.15.83:0/1643739688 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc8740b28a0 0x7fc87409ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.725+0000 7fc878ebb640 1 --2- 172.21.15.83:0/1643739688 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc87415e540 0x7fc8741f4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.725+0000 7fc8737fe640 1 -- 172.21.15.83:0/1643739688 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc87415e540 msgr2=0x7fc8741f4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.725+0000 7fc8737fe640 1 --2- 172.21.15.83:0/1643739688 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc87415e540 0x7fc8741f4350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.725+0000 7fc8737fe640 1 -- 172.21.15.83:0/1643739688 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc8740b28a0 msgr2=0x7fc87409ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.725+0000 7fc8737fe640 1 --2- 172.21.15.83:0/1643739688 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc8740b28a0 0x7fc87409ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.725+0000 7fc8737fe640 1 -- 172.21.15.83:0/1643739688 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc87409eb40 con 0x7fc8740b4c00 2024-03-06T07:22:35.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.725+0000 7fc8737fe640 1 --2- 172.21.15.83:0/1643739688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc8740b4c00 0x7fc87415a9b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fc86800e8c0 tx=0x7fc86800ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.725+0000 7fc870ff9640 1 -- 172.21.15.83:0/1643739688 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc868017070 con 0x7fc8740b4c00 2024-03-06T07:22:35.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.725+0000 7fc870ff9640 1 -- 172.21.15.83:0/1643739688 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc86800c940 con 0x7fc8740b4c00 2024-03-06T07:22:35.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.725+0000 7fc87a91e640 1 -- 172.21.15.83:0/1643739688 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc87409ee20 con 0x7fc8740b4c00 2024-03-06T07:22:35.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.725+0000 7fc870ff9640 1 -- 172.21.15.83:0/1643739688 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc86800cd10 con 0x7fc8740b4c00 2024-03-06T07:22:35.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.725+0000 7fc87a91e640 1 -- 172.21.15.83:0/1643739688 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc87409f3e0 con 0x7fc8740b4c00 2024-03-06T07:22:35.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.726+0000 7fc87a91e640 1 -- 172.21.15.83:0/1643739688 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc87404f330 con 0x7fc8740b4c00 2024-03-06T07:22:35.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.727+0000 7fc870ff9640 1 -- 172.21.15.83:0/1643739688 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc86801a070 con 0x7fc8740b4c00 2024-03-06T07:22:35.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.727+0000 7fc870ff9640 1 --2- 172.21.15.83:0/1643739688 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc86006b4d0 0x7fc86006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.727+0000 7fc870ff9640 1 -- 172.21.15.83:0/1643739688 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc86808a570 con 0x7fc8740b4c00 2024-03-06T07:22:35.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.727+0000 7fc873fff640 1 --2- 172.21.15.83:0/1643739688 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc86006b4d0 0x7fc86006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.727+0000 7fc873fff640 1 --2- 172.21.15.83:0/1643739688 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc86006b4d0 0x7fc86006d990 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc864036000 tx=0x7fc86403b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.731+0000 7fc870ff9640 1 -- 172.21.15.83:0/1643739688 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc8680593f0 con 0x7fc8740b4c00 2024-03-06T07:22:35.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.969+0000 7fc87a91e640 1 -- 172.21.15.83:0/1643739688 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc87415a220 con 0x7fc8740b4c00 2024-03-06T07:22:35.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.969+0000 7fc870ff9640 1 -- 172.21.15.83:0/1643739688 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) v1 ==== 80+0+0 (secure 0 0 0) 0x7fc868058cf0 con 0x7fc8740b4c00 2024-03-06T07:22:35.971 INFO:teuthology.orchestra.run.smithi083.stderr:listed 0 entries 2024-03-06T07:22:35.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.971+0000 7fc87a91e640 1 -- 172.21.15.83:0/1643739688 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc86006b4d0 msgr2=0x7fc86006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.971+0000 7fc87a91e640 1 --2- 172.21.15.83:0/1643739688 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc86006b4d0 0x7fc86006d990 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc864036000 tx=0x7fc86403b040 comp rx=0 tx=0).stop 2024-03-06T07:22:35.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.971+0000 7fc87a91e640 1 -- 172.21.15.83:0/1643739688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc8740b4c00 msgr2=0x7fc87415a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.971+0000 7fc87a91e640 1 --2- 172.21.15.83:0/1643739688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc8740b4c00 0x7fc87415a9b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fc86800e8c0 tx=0x7fc86800ed80 comp rx=0 tx=0).stop 2024-03-06T07:22:35.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.971+0000 7fc87a91e640 1 -- 172.21.15.83:0/1643739688 shutdown_connections 2024-03-06T07:22:35.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.971+0000 7fc87a91e640 1 --2- 172.21.15.83:0/1643739688 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc86006b4d0 0x7fc86006d990 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.971+0000 7fc87a91e640 1 --2- 172.21.15.83:0/1643739688 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc87415e540 0x7fc8741f4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.971+0000 7fc87a91e640 1 --2- 172.21.15.83:0/1643739688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc8740b4c00 0x7fc87415a9b0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.971+0000 7fc87a91e640 1 --2- 172.21.15.83:0/1643739688 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc8740b28a0 0x7fc87409ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.971+0000 7fc87a91e640 1 -- 172.21.15.83:0/1643739688 >> 172.21.15.83:0/1643739688 conn(0x7fc87409a9f0 msgr2=0x7fc874070470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:35.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.971+0000 7fc87a91e640 1 -- 172.21.15.83:0/1643739688 shutdown_connections 2024-03-06T07:22:35.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:35.971+0000 7fc87a91e640 1 -- 172.21.15.83:0/1643739688 wait complete. 2024-03-06T07:22:35.984 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi083.front.sepia.ceph.com... 2024-03-06T07:22:35.984 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:35.984 DEBUG:teuthology.orchestra.run.smithi083:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:36.011 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:36.011 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi083.front.sepia.ceph.com 2024-03-06T07:22:36.011 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:36.011 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:22:36.077 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:22:36.077 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:36.096 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:22:36.096 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:36.155 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:36.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.289+0000 7fedcb866640 1 Processor -- start 2024-03-06T07:22:36.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.289+0000 7fedcb866640 1 -- start start 2024-03-06T07:22:36.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.290+0000 7fedcb866640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fedc40d0320 0x7fedc40b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.290+0000 7fedcb866640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fedc40b54b0 0x7fedc40cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.290+0000 7fedcb866640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedc415c270 con 0x7fedc40d0700 2024-03-06T07:22:36.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.290+0000 7fedcb866640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedc40b04e0 con 0x7fedc40d0320 2024-03-06T07:22:36.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.290+0000 7fedcb866640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedc40d0e90 con 0x7fedc40b54b0 2024-03-06T07:22:36.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.290+0000 7fedc9602640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fedc40d0320 0x7fedc40b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.290+0000 7fedc9602640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fedc40d0320 0x7fedc40b4f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48830/0 (socket says 172.21.15.83:48830) 2024-03-06T07:22:36.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.290+0000 7fedc9602640 1 -- 172.21.15.83:0/1769869817 learned_addr learned my addr 172.21.15.83:0/1769869817 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:36.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.290+0000 7fedc9e03640 1 --2- 172.21.15.83:0/1769869817 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fedc40b54b0 0x7fedc40cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.290+0000 7fedc9602640 1 -- 172.21.15.83:0/1769869817 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fedc40b54b0 msgr2=0x7fedc40cd1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:36.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.290+0000 7fedc9602640 1 --2- 172.21.15.83:0/1769869817 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fedc40b54b0 0x7fedc40cd1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.290+0000 7fedc9602640 1 -- 172.21.15.83:0/1769869817 >> v1:172.21.15.183:6789/0 conn(0x7fedc40d0700 legacy=0x7fedc40ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.290+0000 7fedc9602640 1 -- 172.21.15.83:0/1769869817 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc40b5ae0 con 0x7fedc40d0320 2024-03-06T07:22:36.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.290+0000 7fedc9e03640 1 --2- 172.21.15.83:0/1769869817 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fedc40b54b0 0x7fedc40cd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:36.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.291+0000 7fedc9602640 1 --2- 172.21.15.83:0/1769869817 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedc40d0320 0x7fedc40b4f70 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fedc0009870 tx=0x7fedc002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4df3e1b5d500b72b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.291+0000 7fedbbfff640 1 -- 172.21.15.83:0/1769869817 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc0034070 con 0x7fedc40d0320 2024-03-06T07:22:36.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.291+0000 7fedbbfff640 1 -- 172.21.15.83:0/1769869817 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedc0030840 con 0x7fedc40d0320 2024-03-06T07:22:36.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.291+0000 7fedbbfff640 1 -- 172.21.15.83:0/1769869817 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc0030c10 con 0x7fedc40d0320 2024-03-06T07:22:36.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.292+0000 7fedcb866640 1 -- 172.21.15.83:0/1769869817 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedc40d0320 msgr2=0x7fedc40b4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.292+0000 7fedcb866640 1 --2- 172.21.15.83:0/1769869817 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedc40d0320 0x7fedc40b4f70 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fedc0009870 tx=0x7fedc002b980 comp rx=0 tx=0).stop 2024-03-06T07:22:36.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.292+0000 7fedcb866640 1 -- 172.21.15.83:0/1769869817 shutdown_connections 2024-03-06T07:22:36.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.292+0000 7fedcb866640 1 --2- 172.21.15.83:0/1769869817 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fedc40b54b0 0x7fedc40cd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.292+0000 7fedcb866640 1 --2- 172.21.15.83:0/1769869817 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedc40d0320 0x7fedc40b4f70 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.292+0000 7fedcb866640 1 -- 172.21.15.83:0/1769869817 >> 172.21.15.83:0/1769869817 conn(0x7fedc40724d0 msgr2=0x7fedc40728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:36.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.292+0000 7fedcb866640 1 -- 172.21.15.83:0/1769869817 shutdown_connections 2024-03-06T07:22:36.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.292+0000 7fedcb866640 1 -- 172.21.15.83:0/1769869817 wait complete. 2024-03-06T07:22:36.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.293+0000 7fedcb866640 1 Processor -- start 2024-03-06T07:22:36.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.293+0000 7fedcb866640 1 -- start start 2024-03-06T07:22:36.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.294+0000 7fedcb866640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fedc40b54b0 0x7fedc40c9700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.294+0000 7fedcb866640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedc40d0320 0x7fedc409d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.294+0000 7fedcb866640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fedc40d0700 0x7fedc409dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.294+0000 7fedcb866640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc40b60a0 con 0x7fedc40d0320 2024-03-06T07:22:36.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.294+0000 7fedcb866640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc40b0170 con 0x7fedc40d0700 2024-03-06T07:22:36.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.294+0000 7fedcb866640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedc40ca790 con 0x7fedc40b54b0 2024-03-06T07:22:36.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.294+0000 7fedc8e01640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedc40d0320 0x7fedc409d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.294+0000 7fedc8e01640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedc40d0320 0x7fedc409d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48846/0 (socket says 172.21.15.83:48846) 2024-03-06T07:22:36.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.294+0000 7fedc8e01640 1 -- 172.21.15.83:0/2160134376 learned_addr learned my addr 172.21.15.83:0/2160134376 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:36.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.294+0000 7fedc9602640 1 --2- 172.21.15.83:0/2160134376 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fedc40b54b0 0x7fedc40c9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.294+0000 7fedc9e03640 1 --2- 172.21.15.83:0/2160134376 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fedc40d0700 0x7fedc409dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.294+0000 7fedc8e01640 1 -- 172.21.15.83:0/2160134376 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fedc40b54b0 msgr2=0x7fedc40c9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.294+0000 7fedc8e01640 1 --2- 172.21.15.83:0/2160134376 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fedc40b54b0 0x7fedc40c9700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.294+0000 7fedc8e01640 1 -- 172.21.15.83:0/2160134376 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fedc40d0700 msgr2=0x7fedc409dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.294+0000 7fedc8e01640 1 --2- 172.21.15.83:0/2160134376 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fedc40d0700 0x7fedc409dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.294+0000 7fedc8e01640 1 -- 172.21.15.83:0/2160134376 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc41501d0 con 0x7fedc40d0320 2024-03-06T07:22:36.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.295+0000 7fedc8e01640 1 --2- 172.21.15.83:0/2160134376 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedc40d0320 0x7fedc409d750 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fedb400c440 tx=0x7fedb4001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.295+0000 7fedb9ffb640 1 -- 172.21.15.83:0/2160134376 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb40052d0 con 0x7fedc40d0320 2024-03-06T07:22:36.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.295+0000 7fedb9ffb640 1 -- 172.21.15.83:0/2160134376 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedb4005470 con 0x7fedc40d0320 2024-03-06T07:22:36.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.295+0000 7fedcb866640 1 -- 172.21.15.83:0/2160134376 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedc414d5b0 con 0x7fedc40d0320 2024-03-06T07:22:36.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.295+0000 7fedb9ffb640 1 -- 172.21.15.83:0/2160134376 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb4007260 con 0x7fedc40d0320 2024-03-06T07:22:36.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.295+0000 7fedcb866640 1 -- 172.21.15.83:0/2160134376 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedc414db70 con 0x7fedc40d0320 2024-03-06T07:22:36.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.296+0000 7fedcb866640 1 -- 172.21.15.83:0/2160134376 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedc404f330 con 0x7fedc40d0320 2024-03-06T07:22:36.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.297+0000 7fedb9ffb640 1 -- 172.21.15.83:0/2160134376 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fedb4007400 con 0x7fedc40d0320 2024-03-06T07:22:36.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.297+0000 7fedb9ffb640 1 --2- 172.21.15.83:0/2160134376 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7feda806b4d0 0x7feda806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.297+0000 7fedc9602640 1 --2- 172.21.15.83:0/2160134376 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7feda806b4d0 0x7feda806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.297+0000 7fedb9ffb640 1 -- 172.21.15.83:0/2160134376 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fedb4089600 con 0x7fedc40d0320 2024-03-06T07:22:36.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.297+0000 7fedc9602640 1 --2- 172.21.15.83:0/2160134376 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7feda806b4d0 0x7feda806d990 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fedc0036000 tx=0x7fedc0008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.301+0000 7fedb9ffb640 1 -- 172.21.15.83:0/2160134376 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fedb4057ec0 con 0x7fedc40d0320 2024-03-06T07:22:36.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.547+0000 7fedcb866640 1 -- 172.21.15.83:0/2160134376 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fedc414dd40 con 0x7fedc40d0320 2024-03-06T07:22:36.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.548+0000 7fedb9ffb640 1 -- 172.21.15.83:0/2160134376 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16617 (secure 0 0 0) 0x7fedb405bf60 con 0x7fedc40d0320 2024-03-06T07:22:36.550 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:22:36.550 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":20,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:21:59.938137+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.318942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.254795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:36.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.550+0000 7fedcb866640 1 -- 172.21.15.83:0/2160134376 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7feda806b4d0 msgr2=0x7feda806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.550+0000 7fedcb866640 1 --2- 172.21.15.83:0/2160134376 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7feda806b4d0 0x7feda806d990 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fedc0036000 tx=0x7fedc0008030 comp rx=0 tx=0).stop 2024-03-06T07:22:36.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.550+0000 7fedcb866640 1 -- 172.21.15.83:0/2160134376 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedc40d0320 msgr2=0x7fedc409d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.550+0000 7fedcb866640 1 --2- 172.21.15.83:0/2160134376 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedc40d0320 0x7fedc409d750 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fedb400c440 tx=0x7fedb4001e70 comp rx=0 tx=0).stop 2024-03-06T07:22:36.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.550+0000 7fedcb866640 1 -- 172.21.15.83:0/2160134376 shutdown_connections 2024-03-06T07:22:36.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.550+0000 7fedcb866640 1 --2- 172.21.15.83:0/2160134376 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7feda806b4d0 0x7feda806d990 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.550+0000 7fedcb866640 1 --2- 172.21.15.83:0/2160134376 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fedc40d0700 0x7fedc409dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.550+0000 7fedcb866640 1 --2- 172.21.15.83:0/2160134376 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedc40d0320 0x7fedc409d750 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.550+0000 7fedcb866640 1 --2- 172.21.15.83:0/2160134376 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fedc40b54b0 0x7fedc40c9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.550+0000 7fedcb866640 1 -- 172.21.15.83:0/2160134376 >> 172.21.15.83:0/2160134376 conn(0x7fedc40724d0 msgr2=0x7fedc40cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:36.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.550+0000 7fedcb866640 1 -- 172.21.15.83:0/2160134376 shutdown_connections 2024-03-06T07:22:36.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.550+0000 7fedcb866640 1 -- 172.21.15.83:0/2160134376 wait complete. 2024-03-06T07:22:36.563 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:56.318942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:57.254795+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:36.564 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:36.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.665+0000 7fdcdbbd0640 1 Processor -- start 2024-03-06T07:22:36.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.665+0000 7fdcdbbd0640 1 -- start start 2024-03-06T07:22:36.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.666+0000 7fdcdbbd0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdcd40d14a0 0x7fdcd415e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.666+0000 7fdcdbbd0640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fdcd415eb00 0x7fdcd40cae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.666+0000 7fdcdbbd0640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcd40cfa50 con 0x7fdcd40d10c0 2024-03-06T07:22:36.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.666+0000 7fdcdbbd0640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcd415c050 con 0x7fdcd415eb00 2024-03-06T07:22:36.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.666+0000 7fdcdbbd0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd40d0060 con 0x7fdcd40d14a0 2024-03-06T07:22:36.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.666+0000 7fdcda16d640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fdcd415eb00 0x7fdcd40cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.666+0000 7fdcd916b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdcd40d14a0 0x7fdcd415e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.667+0000 7fdcd996c640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fdcd40d10c0 0x7fdcd40d04f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:55878/0 (socket says 172.21.15.83:55878) 2024-03-06T07:22:36.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.667+0000 7fdcd996c640 1 -- 172.21.15.83:0/1211201827 learned_addr learned my addr 172.21.15.83:0/1211201827 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:36.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.667+0000 7fdcd916b640 1 -- 172.21.15.83:0/1211201827 >> v2:172.21.15.183:3300/0 conn(0x7fdcd415eb00 msgr2=0x7fdcd40cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:36.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.667+0000 7fdcd916b640 1 --2- 172.21.15.83:0/1211201827 >> v2:172.21.15.183:3300/0 conn(0x7fdcd415eb00 0x7fdcd40cae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.667+0000 7fdcd916b640 1 -- 172.21.15.83:0/1211201827 >> v1:172.21.15.183:6789/0 conn(0x7fdcd40d10c0 legacy=0x7fdcd40d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.667+0000 7fdcd916b640 1 -- 172.21.15.83:0/1211201827 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd40b45c0 con 0x7fdcd40d14a0 2024-03-06T07:22:36.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.667+0000 7fdcda16d640 1 --2- 172.21.15.83:0/1211201827 >> v2:172.21.15.183:3300/0 conn(0x7fdcd415eb00 0x7fdcd40cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:36.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.668+0000 7fdcd916b640 1 --2- 172.21.15.83:0/1211201827 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdcd40d14a0 0x7fdcd415e5c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fdcd00097a0 tx=0x7fdcd002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24075548aac368b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.668+0000 7fdcd896a640 1 -- 172.21.15.83:0/1211201827 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd0035020 con 0x7fdcd40d14a0 2024-03-06T07:22:36.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.668+0000 7fdcd896a640 1 -- 172.21.15.83:0/1211201827 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcd0031830 con 0x7fdcd40d14a0 2024-03-06T07:22:36.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.668+0000 7fdcd896a640 1 -- 172.21.15.83:0/1211201827 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd0031c00 con 0x7fdcd40d14a0 2024-03-06T07:22:36.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.668+0000 7fdcdbbd0640 1 -- 172.21.15.83:0/1211201827 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdcd40d14a0 msgr2=0x7fdcd415e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.668+0000 7fdcdbbd0640 1 --2- 172.21.15.83:0/1211201827 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdcd40d14a0 0x7fdcd415e5c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fdcd00097a0 tx=0x7fdcd002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.668+0000 7fdcdbbd0640 1 -- 172.21.15.83:0/1211201827 shutdown_connections 2024-03-06T07:22:36.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.668+0000 7fdcdbbd0640 1 --2- 172.21.15.83:0/1211201827 >> v2:172.21.15.183:3300/0 conn(0x7fdcd415eb00 0x7fdcd40cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.668+0000 7fdcdbbd0640 1 --2- 172.21.15.83:0/1211201827 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdcd40d14a0 0x7fdcd415e5c0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.668+0000 7fdcdbbd0640 1 -- 172.21.15.83:0/1211201827 >> 172.21.15.83:0/1211201827 conn(0x7fdcd40b3800 msgr2=0x7fdcd40b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:36.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.669+0000 7fdcdbbd0640 1 -- 172.21.15.83:0/1211201827 shutdown_connections 2024-03-06T07:22:36.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.669+0000 7fdcdbbd0640 1 -- 172.21.15.83:0/1211201827 wait complete. 2024-03-06T07:22:36.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.670+0000 7fdcdbbd0640 1 Processor -- start 2024-03-06T07:22:36.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.670+0000 7fdcdbbd0640 1 -- start start 2024-03-06T07:22:36.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.670+0000 7fdcdbbd0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdcd40d10c0 0x7fdcd40b7420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.670+0000 7fdcdbbd0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdcd40d14a0 0x7fdcd40b4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.670+0000 7fdcdbbd0640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdcd415eb00 0x7fdcd40b66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.670+0000 7fdcdbbd0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd40cfcf0 con 0x7fdcd40d10c0 2024-03-06T07:22:36.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.670+0000 7fdcdbbd0640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd415f440 con 0x7fdcd415eb00 2024-03-06T07:22:36.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.670+0000 7fdcdbbd0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd40cb3c0 con 0x7fdcd40d14a0 2024-03-06T07:22:36.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.671+0000 7fdcd996c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdcd40d10c0 0x7fdcd40b7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.671+0000 7fdcd996c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdcd40d10c0 0x7fdcd40b7420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53542/0 (socket says 172.21.15.83:53542) 2024-03-06T07:22:36.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.671+0000 7fdcd996c640 1 -- 172.21.15.83:0/3995652719 learned_addr learned my addr 172.21.15.83:0/3995652719 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:36.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.671+0000 7fdcda16d640 1 --2- 172.21.15.83:0/3995652719 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdcd415eb00 0x7fdcd40b66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.671+0000 7fdcd916b640 1 --2- 172.21.15.83:0/3995652719 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdcd40d14a0 0x7fdcd40b4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.671+0000 7fdcd996c640 1 -- 172.21.15.83:0/3995652719 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdcd40d14a0 msgr2=0x7fdcd40b4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.671+0000 7fdcd996c640 1 --2- 172.21.15.83:0/3995652719 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdcd40d14a0 0x7fdcd40b4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.671+0000 7fdcd996c640 1 -- 172.21.15.83:0/3995652719 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdcd415eb00 msgr2=0x7fdcd40b66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.671+0000 7fdcd996c640 1 --2- 172.21.15.83:0/3995652719 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdcd415eb00 0x7fdcd40b66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.671+0000 7fdcd996c640 1 -- 172.21.15.83:0/3995652719 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd40cc840 con 0x7fdcd40d10c0 2024-03-06T07:22:36.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.671+0000 7fdcd916b640 1 --2- 172.21.15.83:0/3995652719 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdcd40d14a0 0x7fdcd40b4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:36.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.671+0000 7fdcd996c640 1 --2- 172.21.15.83:0/3995652719 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdcd40d10c0 0x7fdcd40b7420 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fdcc800c430 tx=0x7fdcc800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.671+0000 7fdcc67fc640 1 -- 172.21.15.83:0/3995652719 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc8017070 con 0x7fdcd40d10c0 2024-03-06T07:22:36.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.671+0000 7fdcc67fc640 1 -- 172.21.15.83:0/3995652719 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcc8005ce0 con 0x7fdcd40d10c0 2024-03-06T07:22:36.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.671+0000 7fdcc67fc640 1 -- 172.21.15.83:0/3995652719 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc80072f0 con 0x7fdcd40d10c0 2024-03-06T07:22:36.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.671+0000 7fdcdbbd0640 1 -- 172.21.15.83:0/3995652719 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcd40cc560 con 0x7fdcd40d10c0 2024-03-06T07:22:36.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.671+0000 7fdcdbbd0640 1 -- 172.21.15.83:0/3995652719 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcd40cf230 con 0x7fdcd40d10c0 2024-03-06T07:22:36.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.672+0000 7fdcdbbd0640 1 -- 172.21.15.83:0/3995652719 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcd409fe60 con 0x7fdcd40d10c0 2024-03-06T07:22:36.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.673+0000 7fdcc67fc640 1 -- 172.21.15.83:0/3995652719 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdcc801a070 con 0x7fdcd40d10c0 2024-03-06T07:22:36.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.673+0000 7fdcc67fc640 1 --2- 172.21.15.83:0/3995652719 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdcb806b4d0 0x7fdcb806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.673+0000 7fdcc67fc640 1 -- 172.21.15.83:0/3995652719 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdcc80894d0 con 0x7fdcd40d10c0 2024-03-06T07:22:36.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.673+0000 7fdcd916b640 1 --2- 172.21.15.83:0/3995652719 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdcb806b4d0 0x7fdcb806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.674+0000 7fdcd916b640 1 --2- 172.21.15.83:0/3995652719 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdcb806b4d0 0x7fdcb806d990 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fdcd0036000 tx=0x7fdcd0008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.677+0000 7fdcc67fc640 1 -- 172.21.15.83:0/3995652719 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdcc80583e0 con 0x7fdcd40d10c0 2024-03-06T07:22:36.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.925+0000 7fdcdbbd0640 1 -- 172.21.15.83:0/3995652719 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdcd40af4f0 con 0x7fdcd40d10c0 2024-03-06T07:22:36.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.926+0000 7fdcc67fc640 1 -- 172.21.15.83:0/3995652719 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdcc8087e20 con 0x7fdcd40d10c0 2024-03-06T07:22:36.928 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:22:36.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.928+0000 7fdcdbbd0640 1 -- 172.21.15.83:0/3995652719 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdcb806b4d0 msgr2=0x7fdcb806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.928+0000 7fdcdbbd0640 1 --2- 172.21.15.83:0/3995652719 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdcb806b4d0 0x7fdcb806d990 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fdcd0036000 tx=0x7fdcd0008030 comp rx=0 tx=0).stop 2024-03-06T07:22:36.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.928+0000 7fdcdbbd0640 1 -- 172.21.15.83:0/3995652719 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdcd40d10c0 msgr2=0x7fdcd40b7420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.928+0000 7fdcdbbd0640 1 --2- 172.21.15.83:0/3995652719 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdcd40d10c0 0x7fdcd40b7420 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fdcc800c430 tx=0x7fdcc800c830 comp rx=0 tx=0).stop 2024-03-06T07:22:36.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.928+0000 7fdcdbbd0640 1 -- 172.21.15.83:0/3995652719 shutdown_connections 2024-03-06T07:22:36.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.928+0000 7fdcdbbd0640 1 --2- 172.21.15.83:0/3995652719 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdcb806b4d0 0x7fdcb806d990 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.928+0000 7fdcdbbd0640 1 --2- 172.21.15.83:0/3995652719 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdcd415eb00 0x7fdcd40b66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.928+0000 7fdcdbbd0640 1 --2- 172.21.15.83:0/3995652719 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdcd40d14a0 0x7fdcd40b4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.928+0000 7fdcdbbd0640 1 --2- 172.21.15.83:0/3995652719 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdcd40d10c0 0x7fdcd40b7420 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.928+0000 7fdcdbbd0640 1 -- 172.21.15.83:0/3995652719 >> 172.21.15.83:0/3995652719 conn(0x7fdcd40b3800 msgr2=0x7fdcd415b4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:36.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.928+0000 7fdcdbbd0640 1 -- 172.21.15.83:0/3995652719 shutdown_connections 2024-03-06T07:22:36.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:36.928+0000 7fdcdbbd0640 1 -- 172.21.15.83:0/3995652719 wait complete. 2024-03-06T07:22:36.941 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:37.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.042+0000 7f5cc830f640 1 Processor -- start 2024-03-06T07:22:37.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.042+0000 7f5cc830f640 1 -- start start 2024-03-06T07:22:37.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.043+0000 7f5cc830f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5cc01614d0 0x7f5cc00d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.043+0000 7f5cc830f640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f5cc00d1910 0x7f5cc00d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.043+0000 7f5cc830f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5cc00d22b0 0x7f5cc00d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.043+0000 7f5cc830f640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cc00d2740 con 0x7f5cc00d22b0 2024-03-06T07:22:37.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.043+0000 7f5cc830f640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cc01618b0 con 0x7f5cc00d1910 2024-03-06T07:22:37.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.043+0000 7f5cc830f640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc00cf400 con 0x7f5cc01614d0 2024-03-06T07:22:37.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.044+0000 7f5cc68ac640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5cc00d22b0 0x7f5cc00d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.044+0000 7f5cc68ac640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5cc00d22b0 0x7f5cc00d06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53546/0 (socket says 172.21.15.83:53546) 2024-03-06T07:22:37.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.044+0000 7f5cc68ac640 1 -- 172.21.15.83:0/1275195923 learned_addr learned my addr 172.21.15.83:0/1275195923 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:37.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.044+0000 7f5cc58aa640 1 --2- 172.21.15.83:0/1275195923 >> v2:172.21.15.183:3300/0 conn(0x7f5cc00d1910 0x7f5cc00d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.044+0000 7f5cc60ab640 1 --2- 172.21.15.83:0/1275195923 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5cc01614d0 0x7f5cc00d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.044+0000 7f5cc68ac640 1 -- 172.21.15.83:0/1275195923 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5cc01614d0 msgr2=0x7f5cc00d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:37.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.044+0000 7f5cc68ac640 1 --2- 172.21.15.83:0/1275195923 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5cc01614d0 0x7f5cc00d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.044+0000 7f5cc68ac640 1 -- 172.21.15.83:0/1275195923 >> v2:172.21.15.183:3300/0 conn(0x7f5cc00d1910 msgr2=0x7f5cc00d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:37.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.044+0000 7f5cc68ac640 1 --2- 172.21.15.83:0/1275195923 >> v2:172.21.15.183:3300/0 conn(0x7f5cc00d1910 0x7f5cc00d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.044+0000 7f5cc68ac640 1 -- 172.21.15.83:0/1275195923 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cc00cf6a0 con 0x7f5cc00d22b0 2024-03-06T07:22:37.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.044+0000 7f5cc58aa640 1 --2- 172.21.15.83:0/1275195923 >> v2:172.21.15.183:3300/0 conn(0x7f5cc00d1910 0x7f5cc00d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:37.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.045+0000 7f5cc68ac640 1 --2- 172.21.15.83:0/1275195923 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5cc00d22b0 0x7f5cc00d06a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5cc008c4f0 tx=0x7f5cbc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c006b4f8f0a7aeb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.045+0000 7f5cc50a9640 1 -- 172.21.15.83:0/1275195923 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc02ca70 con 0x7f5cc00d22b0 2024-03-06T07:22:37.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.045+0000 7f5cc50a9640 1 -- 172.21.15.83:0/1275195923 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cbc02cc10 con 0x7f5cc00d22b0 2024-03-06T07:22:37.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.045+0000 7f5cc50a9640 1 -- 172.21.15.83:0/1275195923 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc0343f0 con 0x7f5cc00d22b0 2024-03-06T07:22:37.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.045+0000 7f5cc830f640 1 -- 172.21.15.83:0/1275195923 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5cc00d22b0 msgr2=0x7f5cc00d06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.045+0000 7f5cc830f640 1 --2- 172.21.15.83:0/1275195923 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5cc00d22b0 0x7f5cc00d06a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5cc008c4f0 tx=0x7f5cbc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.045+0000 7f5cc830f640 1 -- 172.21.15.83:0/1275195923 shutdown_connections 2024-03-06T07:22:37.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.045+0000 7f5cc830f640 1 --2- 172.21.15.83:0/1275195923 >> v2:172.21.15.183:3300/0 conn(0x7f5cc00d1910 0x7f5cc00d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.045+0000 7f5cc830f640 1 --2- 172.21.15.83:0/1275195923 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5cc00d22b0 0x7f5cc00d06a0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.045+0000 7f5cc830f640 1 --2- 172.21.15.83:0/1275195923 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5cc01614d0 0x7f5cc00d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.045+0000 7f5cc830f640 1 -- 172.21.15.83:0/1275195923 >> 172.21.15.83:0/1275195923 conn(0x7f5cc00d3d40 msgr2=0x7f5cc00d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:37.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.045+0000 7f5cc830f640 1 -- 172.21.15.83:0/1275195923 shutdown_connections 2024-03-06T07:22:37.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.045+0000 7f5cc830f640 1 -- 172.21.15.83:0/1275195923 wait complete. 2024-03-06T07:22:37.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.047+0000 7f5cc830f640 1 Processor -- start 2024-03-06T07:22:37.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.047+0000 7f5cc830f640 1 -- start start 2024-03-06T07:22:37.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.047+0000 7f5cc830f640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5cc00d1910 0x7f5cc0157ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.047+0000 7f5cc830f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5cc00d22b0 0x7f5cc00c9860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.047+0000 7f5cc830f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5cc01614d0 0x7f5cc00c9da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.047+0000 7f5cc830f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc00b0910 con 0x7f5cc00d22b0 2024-03-06T07:22:37.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.047+0000 7f5cc830f640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc00cfc60 con 0x7f5cc00d1910 2024-03-06T07:22:37.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.047+0000 7f5cc830f640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc00d0be0 con 0x7f5cc01614d0 2024-03-06T07:22:37.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.047+0000 7f5cc58aa640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5cc00d22b0 0x7f5cc00c9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.047+0000 7f5cc58aa640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5cc00d22b0 0x7f5cc00c9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53548/0 (socket says 172.21.15.83:53548) 2024-03-06T07:22:37.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.047+0000 7f5cc58aa640 1 -- 172.21.15.83:0/3134810478 learned_addr learned my addr 172.21.15.83:0/3134810478 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:37.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.048+0000 7f5cc60ab640 1 --2- 172.21.15.83:0/3134810478 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5cc00d1910 0x7f5cc0157ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.048+0000 7f5cc68ac640 1 --2- 172.21.15.83:0/3134810478 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5cc01614d0 0x7f5cc00c9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.048+0000 7f5cc58aa640 1 -- 172.21.15.83:0/3134810478 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5cc01614d0 msgr2=0x7f5cc00c9da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.048+0000 7f5cc58aa640 1 --2- 172.21.15.83:0/3134810478 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5cc01614d0 0x7f5cc00c9da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.048+0000 7f5cc58aa640 1 -- 172.21.15.83:0/3134810478 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5cc00d1910 msgr2=0x7f5cc0157ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.048+0000 7f5cc58aa640 1 --2- 172.21.15.83:0/3134810478 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5cc00d1910 0x7f5cc0157ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.048+0000 7f5cc58aa640 1 -- 172.21.15.83:0/3134810478 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cc0158430 con 0x7f5cc00d22b0 2024-03-06T07:22:37.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.048+0000 7f5cc58aa640 1 --2- 172.21.15.83:0/3134810478 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5cc00d22b0 0x7f5cc00c9860 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f5cb000e730 tx=0x7f5cb000ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.048+0000 7f5cb6ffd640 1 -- 172.21.15.83:0/3134810478 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb000f040 con 0x7f5cc00d22b0 2024-03-06T07:22:37.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.048+0000 7f5cb6ffd640 1 -- 172.21.15.83:0/3134810478 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cb0004230 con 0x7f5cc00d22b0 2024-03-06T07:22:37.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.048+0000 7f5cc830f640 1 -- 172.21.15.83:0/3134810478 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cc00ca3b0 con 0x7f5cc00d22b0 2024-03-06T07:22:37.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.048+0000 7f5cb6ffd640 1 -- 172.21.15.83:0/3134810478 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb000ee40 con 0x7f5cc00d22b0 2024-03-06T07:22:37.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.048+0000 7f5cc830f640 1 -- 172.21.15.83:0/3134810478 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cc01f5990 con 0x7f5cc00d22b0 2024-03-06T07:22:37.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.049+0000 7f5cc830f640 1 -- 172.21.15.83:0/3134810478 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cc00a1c90 con 0x7f5cc00d22b0 2024-03-06T07:22:37.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.050+0000 7f5cb6ffd640 1 -- 172.21.15.83:0/3134810478 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5cb000c6d0 con 0x7f5cc00d22b0 2024-03-06T07:22:37.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.050+0000 7f5cb6ffd640 1 --2- 172.21.15.83:0/3134810478 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5ca406b4d0 0x7f5ca406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.050+0000 7f5cb6ffd640 1 -- 172.21.15.83:0/3134810478 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5cb0089e70 con 0x7f5cc00d22b0 2024-03-06T07:22:37.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.050+0000 7f5cc60ab640 1 --2- 172.21.15.83:0/3134810478 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5ca406b4d0 0x7f5ca406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.050+0000 7f5cc60ab640 1 --2- 172.21.15.83:0/3134810478 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5ca406b4d0 0x7f5ca406d990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5cb8007920 tx=0x7f5cb8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.054+0000 7f5cb6ffd640 1 -- 172.21.15.83:0/3134810478 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5cb0017020 con 0x7f5cc00d22b0 2024-03-06T07:22:37.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.226+0000 7f5cc830f640 1 -- 172.21.15.83:0/3134810478 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5cc00ce930 con 0x7f5cc00d22b0 2024-03-06T07:22:37.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.226+0000 7f5cb6ffd640 1 -- 172.21.15.83:0/3134810478 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5cb00586a0 con 0x7f5cc00d22b0 2024-03-06T07:22:37.228 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-03-06T07:22:37.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.228+0000 7f5cc830f640 1 -- 172.21.15.83:0/3134810478 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5ca406b4d0 msgr2=0x7f5ca406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.228+0000 7f5cc830f640 1 --2- 172.21.15.83:0/3134810478 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5ca406b4d0 0x7f5ca406d990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5cb8007920 tx=0x7f5cb8008040 comp rx=0 tx=0).stop 2024-03-06T07:22:37.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.228+0000 7f5cc830f640 1 -- 172.21.15.83:0/3134810478 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5cc00d22b0 msgr2=0x7f5cc00c9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.228+0000 7f5cc830f640 1 --2- 172.21.15.83:0/3134810478 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5cc00d22b0 0x7f5cc00c9860 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f5cb000e730 tx=0x7f5cb000ebf0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.228+0000 7f5cc830f640 1 -- 172.21.15.83:0/3134810478 shutdown_connections 2024-03-06T07:22:37.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.228+0000 7f5cc830f640 1 --2- 172.21.15.83:0/3134810478 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5ca406b4d0 0x7f5ca406d990 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.228+0000 7f5cc830f640 1 --2- 172.21.15.83:0/3134810478 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5cc01614d0 0x7f5cc00c9da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.228+0000 7f5cc830f640 1 --2- 172.21.15.83:0/3134810478 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5cc00d22b0 0x7f5cc00c9860 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.228+0000 7f5cc830f640 1 --2- 172.21.15.83:0/3134810478 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5cc00d1910 0x7f5cc0157ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.228+0000 7f5cc830f640 1 -- 172.21.15.83:0/3134810478 >> 172.21.15.83:0/3134810478 conn(0x7f5cc00d3d40 msgr2=0x7f5cc006ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:37.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.228+0000 7f5cc830f640 1 -- 172.21.15.83:0/3134810478 shutdown_connections 2024-03-06T07:22:37.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.228+0000 7f5cc830f640 1 -- 172.21.15.83:0/3134810478 wait complete. 2024-03-06T07:22:37.241 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:37.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.343+0000 7f14b67e9640 1 Processor -- start 2024-03-06T07:22:37.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.343+0000 7f14b67e9640 1 -- start start 2024-03-06T07:22:37.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.344+0000 7f14b67e9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f14b006b6c0 0x7f14b006e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.344+0000 7f14b67e9640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14b0161bd0 con 0x7f14b015e0e0 2024-03-06T07:22:37.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.344+0000 7f14b67e9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14b0070220 con 0x7f14b015dd00 2024-03-06T07:22:37.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.344+0000 7f14b67e9640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14b0161e90 con 0x7f14b006b6c0 2024-03-06T07:22:37.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.344+0000 7f14b4d86640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f14b006b6c0 0x7f14b006e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.344+0000 7f14b4d86640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f14b006b6c0 0x7f14b006e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53560/0 (socket says 172.21.15.83:53560) 2024-03-06T07:22:37.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.344+0000 7f14b4d86640 1 -- 172.21.15.83:0/2398355400 learned_addr learned my addr 172.21.15.83:0/2398355400 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:37.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.344+0000 7f14b4d86640 1 -- 172.21.15.83:0/2398355400 >> v1:172.21.15.183:6789/0 conn(0x7f14b015dd00 legacy=0x7f14b006afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:37.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.344+0000 7f14b4d86640 1 -- 172.21.15.83:0/2398355400 >> v1:172.21.15.83:6789/0 conn(0x7f14b015e0e0 legacy=0x7f14b00b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.344+0000 7f14b4d86640 1 -- 172.21.15.83:0/2398355400 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14b0162170 con 0x7f14b006b6c0 2024-03-06T07:22:37.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.345+0000 7f14b4d86640 1 --2- 172.21.15.83:0/2398355400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14b006b6c0 0x7f14b006e620 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f14a8009e50 tx=0x7f14a802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49d2a1b31fa2a910 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.345+0000 7f14aeffd640 1 -- 172.21.15.83:0/2398355400 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a802cb20 con 0x7f14b006b6c0 2024-03-06T07:22:37.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.345+0000 7f14aeffd640 1 -- 172.21.15.83:0/2398355400 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14a802ccc0 con 0x7f14b006b6c0 2024-03-06T07:22:37.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.345+0000 7f14aeffd640 1 -- 172.21.15.83:0/2398355400 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a8034610 con 0x7f14b006b6c0 2024-03-06T07:22:37.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.346+0000 7f14b67e9640 1 -- 172.21.15.83:0/2398355400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14b006b6c0 msgr2=0x7f14b006e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.346+0000 7f14b67e9640 1 --2- 172.21.15.83:0/2398355400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14b006b6c0 0x7f14b006e620 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f14a8009e50 tx=0x7f14a802bf80 comp rx=0 tx=0).stop 2024-03-06T07:22:37.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.346+0000 7f14b67e9640 1 -- 172.21.15.83:0/2398355400 shutdown_connections 2024-03-06T07:22:37.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.346+0000 7f14b67e9640 1 --2- 172.21.15.83:0/2398355400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14b006b6c0 0x7f14b006e620 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.346+0000 7f14b67e9640 1 -- 172.21.15.83:0/2398355400 >> 172.21.15.83:0/2398355400 conn(0x7f14b00b2660 msgr2=0x7f14b00b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:37.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.346+0000 7f14b67e9640 1 -- 172.21.15.83:0/2398355400 shutdown_connections 2024-03-06T07:22:37.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.346+0000 7f14b67e9640 1 -- 172.21.15.83:0/2398355400 wait complete. 2024-03-06T07:22:37.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.347+0000 7f14b67e9640 1 Processor -- start 2024-03-06T07:22:37.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.347+0000 7f14b67e9640 1 -- start start 2024-03-06T07:22:37.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.348+0000 7f14b67e9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14b006b6c0 0x7f14b01404f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.348+0000 7f14b67e9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f14b015dd00 0x7f14b0140d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.348+0000 7f14b67e9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14b0141280 0x7f14b015edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.348+0000 7f14b67e9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b00b0180 con 0x7f14b0141280 2024-03-06T07:22:37.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.348+0000 7f14b67e9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b00b0000 con 0x7f14b006b6c0 2024-03-06T07:22:37.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.348+0000 7f14b67e9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14b00b0300 con 0x7f14b015dd00 2024-03-06T07:22:37.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.348+0000 7f14b4d86640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14b0141280 0x7f14b015edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.348+0000 7f14b4d86640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14b0141280 0x7f14b015edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53576/0 (socket says 172.21.15.83:53576) 2024-03-06T07:22:37.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.348+0000 7f14b4d86640 1 -- 172.21.15.83:0/2575143365 learned_addr learned my addr 172.21.15.83:0/2575143365 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:37.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.348+0000 7f14affff640 1 --2- 172.21.15.83:0/2575143365 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14b006b6c0 0x7f14b01404f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.348+0000 7f14af7fe640 1 --2- 172.21.15.83:0/2575143365 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f14b015dd00 0x7f14b0140d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.348+0000 7f14b4d86640 1 -- 172.21.15.83:0/2575143365 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f14b015dd00 msgr2=0x7f14b0140d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.348+0000 7f14b4d86640 1 --2- 172.21.15.83:0/2575143365 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f14b015dd00 0x7f14b0140d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.348+0000 7f14b4d86640 1 -- 172.21.15.83:0/2575143365 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14b006b6c0 msgr2=0x7f14b01404f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.348+0000 7f14b4d86640 1 --2- 172.21.15.83:0/2575143365 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14b006b6c0 0x7f14b01404f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.348+0000 7f14b4d86640 1 -- 172.21.15.83:0/2575143365 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14b015f8d0 con 0x7f14b0141280 2024-03-06T07:22:37.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.348+0000 7f14af7fe640 1 --2- 172.21.15.83:0/2575143365 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f14b015dd00 0x7f14b0140d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:37.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.348+0000 7f14b4d86640 1 --2- 172.21.15.83:0/2575143365 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14b0141280 0x7f14b015edd0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f14a8036040 tx=0x7f14a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.349+0000 7f14acff9640 1 -- 172.21.15.83:0/2575143365 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a8043020 con 0x7f14b0141280 2024-03-06T07:22:37.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.349+0000 7f14acff9640 1 -- 172.21.15.83:0/2575143365 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14a803e450 con 0x7f14b0141280 2024-03-06T07:22:37.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.349+0000 7f14b67e9640 1 -- 172.21.15.83:0/2575143365 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14b015fbb0 con 0x7f14b0141280 2024-03-06T07:22:37.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.349+0000 7f14acff9640 1 -- 172.21.15.83:0/2575143365 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a803e820 con 0x7f14b0141280 2024-03-06T07:22:37.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.349+0000 7f14b67e9640 1 -- 172.21.15.83:0/2575143365 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14b0160170 con 0x7f14b0141280 2024-03-06T07:22:37.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.350+0000 7f14b67e9640 1 -- 172.21.15.83:0/2575143365 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14b0073390 con 0x7f14b0141280 2024-03-06T07:22:37.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.350+0000 7f14acff9640 1 -- 172.21.15.83:0/2575143365 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f14a8046020 con 0x7f14b0141280 2024-03-06T07:22:37.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.350+0000 7f14acff9640 1 --2- 172.21.15.83:0/2575143365 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f149006b4d0 0x7f149006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.351+0000 7f14acff9640 1 -- 172.21.15.83:0/2575143365 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f14a80acc00 con 0x7f14b0141280 2024-03-06T07:22:37.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.351+0000 7f14affff640 1 --2- 172.21.15.83:0/2575143365 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f149006b4d0 0x7f149006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.351+0000 7f14affff640 1 --2- 172.21.15.83:0/2575143365 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f149006b4d0 0x7f149006d990 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f14a0007fd0 tx=0x7f14a0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.355+0000 7f14acff9640 1 -- 172.21.15.83:0/2575143365 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f14a80b3080 con 0x7f14b0141280 2024-03-06T07:22:37.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.533+0000 7f14b67e9640 1 -- 172.21.15.83:0/2575143365 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f14b00b3ba0 con 0x7f14b0141280 2024-03-06T07:22:37.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.533+0000 7f14acff9640 1 -- 172.21.15.83:0/2575143365 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f14a804d030 con 0x7f14b0141280 2024-03-06T07:22:37.535 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 64 2024-03-06T07:22:37.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.535+0000 7f14b67e9640 1 -- 172.21.15.83:0/2575143365 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f149006b4d0 msgr2=0x7f149006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.535+0000 7f14b67e9640 1 --2- 172.21.15.83:0/2575143365 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f149006b4d0 0x7f149006d990 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f14a0007fd0 tx=0x7f14a0009040 comp rx=0 tx=0).stop 2024-03-06T07:22:37.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.535+0000 7f14b67e9640 1 -- 172.21.15.83:0/2575143365 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14b0141280 msgr2=0x7f14b015edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.535+0000 7f14b67e9640 1 --2- 172.21.15.83:0/2575143365 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14b0141280 0x7f14b015edd0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f14a8036040 tx=0x7f14a8004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:37.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.535+0000 7f14b67e9640 1 -- 172.21.15.83:0/2575143365 shutdown_connections 2024-03-06T07:22:37.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.535+0000 7f14b67e9640 1 --2- 172.21.15.83:0/2575143365 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f149006b4d0 0x7f149006d990 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.535+0000 7f14b67e9640 1 --2- 172.21.15.83:0/2575143365 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14b0141280 0x7f14b015edd0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.535+0000 7f14b67e9640 1 --2- 172.21.15.83:0/2575143365 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f14b015dd00 0x7f14b0140d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.535+0000 7f14b67e9640 1 --2- 172.21.15.83:0/2575143365 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14b006b6c0 0x7f14b01404f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.535+0000 7f14b67e9640 1 -- 172.21.15.83:0/2575143365 >> 172.21.15.83:0/2575143365 conn(0x7f14b00b2660 msgr2=0x7f14b00ae190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:37.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.535+0000 7f14b67e9640 1 -- 172.21.15.83:0/2575143365 shutdown_connections 2024-03-06T07:22:37.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.535+0000 7f14b67e9640 1 -- 172.21.15.83:0/2575143365 wait complete. 2024-03-06T07:22:37.552 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:22:37.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.652+0000 7fefbdb55640 1 Processor -- start 2024-03-06T07:22:37.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.652+0000 7fefbdb55640 1 -- start start 2024-03-06T07:22:37.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.653+0000 7fefbdb55640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fefb80b4460 0x7fefb80d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.654+0000 7fefbdb55640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fefb8162970 0x7fefb80b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.654+0000 7fefbdb55640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fefb8161cf0 0x7fefb8162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.654+0000 7fefbdb55640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefb8161b00 con 0x7fefb8162970 2024-03-06T07:22:37.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.654+0000 7fefbdb55640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefb80cecc0 con 0x7fefb80b4460 2024-03-06T07:22:37.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.654+0000 7fefbdb55640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefb80d22c0 con 0x7fefb8161cf0 2024-03-06T07:22:37.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.654+0000 7fefb6ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fefb8162970 0x7fefb80b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.654+0000 7fefb6ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fefb8162970 0x7fefb80b3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53578/0 (socket says 172.21.15.83:53578) 2024-03-06T07:22:37.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.654+0000 7fefb6ffd640 1 -- 172.21.15.83:0/1535502321 learned_addr learned my addr 172.21.15.83:0/1535502321 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:37.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.654+0000 7fefb77fe640 1 --2- 172.21.15.83:0/1535502321 >> v2:172.21.15.183:3300/0 conn(0x7fefb80b4460 0x7fefb80d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.654+0000 7fefb7fff640 1 --2- 172.21.15.83:0/1535502321 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fefb8161cf0 0x7fefb8162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.654+0000 7fefb6ffd640 1 -- 172.21.15.83:0/1535502321 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fefb8161cf0 msgr2=0x7fefb8162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:37.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.654+0000 7fefb6ffd640 1 --2- 172.21.15.83:0/1535502321 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fefb8161cf0 0x7fefb8162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.654+0000 7fefb6ffd640 1 -- 172.21.15.83:0/1535502321 >> v2:172.21.15.183:3300/0 conn(0x7fefb80b4460 msgr2=0x7fefb80d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:37.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.654+0000 7fefb6ffd640 1 --2- 172.21.15.83:0/1535502321 >> v2:172.21.15.183:3300/0 conn(0x7fefb80b4460 0x7fefb80d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.654+0000 7fefb6ffd640 1 -- 172.21.15.83:0/1535502321 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb80d25a0 con 0x7fefb8162970 2024-03-06T07:22:37.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.655+0000 7fefb6ffd640 1 --2- 172.21.15.83:0/1535502321 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fefb8162970 0x7fefb80b3580 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fefb808ddd0 tx=0x7fefa002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e39f2994be4cbe0b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.655+0000 7fefb67fc640 1 -- 172.21.15.83:0/1535502321 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa0035020 con 0x7fefb8162970 2024-03-06T07:22:37.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.655+0000 7fefb67fc640 1 -- 172.21.15.83:0/1535502321 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefa0031870 con 0x7fefb8162970 2024-03-06T07:22:37.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.655+0000 7fefb67fc640 1 -- 172.21.15.83:0/1535502321 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa0031c40 con 0x7fefb8162970 2024-03-06T07:22:37.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.655+0000 7fefbdb55640 1 -- 172.21.15.83:0/1535502321 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fefb8162970 msgr2=0x7fefb80b3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.655+0000 7fefbdb55640 1 --2- 172.21.15.83:0/1535502321 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fefb8162970 0x7fefb80b3580 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fefb808ddd0 tx=0x7fefa002b910 comp rx=0 tx=0).stop 2024-03-06T07:22:37.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.655+0000 7fefbdb55640 1 -- 172.21.15.83:0/1535502321 shutdown_connections 2024-03-06T07:22:37.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.655+0000 7fefbdb55640 1 --2- 172.21.15.83:0/1535502321 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fefb8161cf0 0x7fefb8162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.655+0000 7fefbdb55640 1 --2- 172.21.15.83:0/1535502321 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fefb8162970 0x7fefb80b3580 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.655+0000 7fefbdb55640 1 --2- 172.21.15.83:0/1535502321 >> v2:172.21.15.183:3300/0 conn(0x7fefb80b4460 0x7fefb80d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.655+0000 7fefbdb55640 1 -- 172.21.15.83:0/1535502321 >> 172.21.15.83:0/1535502321 conn(0x7fefb80d3ce0 msgr2=0x7fefb80d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:37.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.656+0000 7fefbdb55640 1 -- 172.21.15.83:0/1535502321 shutdown_connections 2024-03-06T07:22:37.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.656+0000 7fefbdb55640 1 -- 172.21.15.83:0/1535502321 wait complete. 2024-03-06T07:22:37.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.657+0000 7fefbdb55640 1 Processor -- start 2024-03-06T07:22:37.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.657+0000 7fefbdb55640 1 -- start start 2024-03-06T07:22:37.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.657+0000 7fefbdb55640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fefb80b4460 0x7fefb80b8210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.657+0000 7fefbdb55640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fefb8161cf0 0x7fefb80b7240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.657+0000 7fefbdb55640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fefb8162970 0x7fefb80b1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.657+0000 7fefbdb55640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb80ce950 con 0x7fefb8161cf0 2024-03-06T07:22:37.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.657+0000 7fefbdb55640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb80d2b60 con 0x7fefb80b4460 2024-03-06T07:22:37.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.657+0000 7fefbdb55640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefb80b3ac0 con 0x7fefb8162970 2024-03-06T07:22:37.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.658+0000 7fefb6ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fefb8161cf0 0x7fefb80b7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.658+0000 7fefb6ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fefb8161cf0 0x7fefb80b7240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53580/0 (socket says 172.21.15.83:53580) 2024-03-06T07:22:37.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.658+0000 7fefb6ffd640 1 -- 172.21.15.83:0/3733838969 learned_addr learned my addr 172.21.15.83:0/3733838969 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:37.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.658+0000 7fefb77fe640 1 --2- 172.21.15.83:0/3733838969 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fefb80b4460 0x7fefb80b8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.658+0000 7fefb7fff640 1 --2- 172.21.15.83:0/3733838969 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fefb8162970 0x7fefb80b1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.658+0000 7fefb6ffd640 1 -- 172.21.15.83:0/3733838969 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fefb8162970 msgr2=0x7fefb80b1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.658+0000 7fefb6ffd640 1 --2- 172.21.15.83:0/3733838969 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fefb8162970 0x7fefb80b1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.658+0000 7fefb6ffd640 1 -- 172.21.15.83:0/3733838969 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fefb80b4460 msgr2=0x7fefb80b8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.658+0000 7fefb6ffd640 1 --2- 172.21.15.83:0/3733838969 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fefb80b4460 0x7fefb80b8210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.658+0000 7fefb6ffd640 1 -- 172.21.15.83:0/3733838969 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb80c8da0 con 0x7fefb8161cf0 2024-03-06T07:22:37.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.658+0000 7fefb77fe640 1 --2- 172.21.15.83:0/3733838969 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fefb80b4460 0x7fefb80b8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:37.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.658+0000 7fefb6ffd640 1 --2- 172.21.15.83:0/3733838969 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fefb8161cf0 0x7fefb80b7240 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fefa002b950 tx=0x7fefa00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.658+0000 7fefaffff640 1 -- 172.21.15.83:0/3733838969 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa0041070 con 0x7fefb8161cf0 2024-03-06T07:22:37.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.658+0000 7fefaffff640 1 -- 172.21.15.83:0/3733838969 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefa003ac80 con 0x7fefb8161cf0 2024-03-06T07:22:37.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.658+0000 7fefbdb55640 1 -- 172.21.15.83:0/3733838969 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefb80b8b10 con 0x7fefb8161cf0 2024-03-06T07:22:37.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.658+0000 7fefaffff640 1 -- 172.21.15.83:0/3733838969 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa0035040 con 0x7fefb8161cf0 2024-03-06T07:22:37.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.658+0000 7fefbdb55640 1 -- 172.21.15.83:0/3733838969 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefb80b90d0 con 0x7fefb8161cf0 2024-03-06T07:22:37.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.659+0000 7fefbdb55640 1 -- 172.21.15.83:0/3733838969 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefb804f330 con 0x7fefb8161cf0 2024-03-06T07:22:37.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.660+0000 7fefaffff640 1 -- 172.21.15.83:0/3733838969 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fefa0038020 con 0x7fefb8161cf0 2024-03-06T07:22:37.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.660+0000 7fefaffff640 1 --2- 172.21.15.83:0/3733838969 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fef9806b4d0 0x7fef9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.660+0000 7fefaffff640 1 -- 172.21.15.83:0/3733838969 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fefa00ac0c0 con 0x7fefb8161cf0 2024-03-06T07:22:37.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.660+0000 7fefb77fe640 1 --2- 172.21.15.83:0/3733838969 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fef9806b4d0 0x7fef9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.661+0000 7fefb77fe640 1 --2- 172.21.15.83:0/3733838969 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fef9806b4d0 0x7fef9806d990 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fefa8006fd0 tx=0x7fefa8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.665+0000 7fefaffff640 1 -- 172.21.15.83:0/3733838969 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fefa007af40 con 0x7fefb8161cf0 2024-03-06T07:22:37.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.847+0000 7fefbdb55640 1 -- 172.21.15.83:0/3733838969 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fefb80d20a0 con 0x7fefb8161cf0 2024-03-06T07:22:37.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.848+0000 7fefaffff640 1 -- 172.21.15.83:0/3733838969 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) v1 ==== 80+0+0 (secure 0 0 0) 0x7fefa007a840 con 0x7fefb8161cf0 2024-03-06T07:22:37.850 INFO:teuthology.orchestra.run.smithi083.stderr:listed 0 entries 2024-03-06T07:22:37.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.850+0000 7fefbdb55640 1 -- 172.21.15.83:0/3733838969 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fef9806b4d0 msgr2=0x7fef9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.850+0000 7fefbdb55640 1 --2- 172.21.15.83:0/3733838969 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fef9806b4d0 0x7fef9806d990 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fefa8006fd0 tx=0x7fefa8008040 comp rx=0 tx=0).stop 2024-03-06T07:22:37.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.850+0000 7fefbdb55640 1 -- 172.21.15.83:0/3733838969 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fefb8161cf0 msgr2=0x7fefb80b7240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.850+0000 7fefbdb55640 1 --2- 172.21.15.83:0/3733838969 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fefb8161cf0 0x7fefb80b7240 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fefa002b950 tx=0x7fefa00046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.850+0000 7fefbdb55640 1 -- 172.21.15.83:0/3733838969 shutdown_connections 2024-03-06T07:22:37.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.850+0000 7fefbdb55640 1 --2- 172.21.15.83:0/3733838969 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fef9806b4d0 0x7fef9806d990 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.850+0000 7fefbdb55640 1 --2- 172.21.15.83:0/3733838969 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fefb8162970 0x7fefb80b1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.850+0000 7fefbdb55640 1 --2- 172.21.15.83:0/3733838969 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fefb8161cf0 0x7fefb80b7240 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.850+0000 7fefbdb55640 1 --2- 172.21.15.83:0/3733838969 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fefb80b4460 0x7fefb80b8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.850+0000 7fefbdb55640 1 -- 172.21.15.83:0/3733838969 >> 172.21.15.83:0/3733838969 conn(0x7fefb80d3ce0 msgr2=0x7fefb80d0630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:37.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.850+0000 7fefbdb55640 1 -- 172.21.15.83:0/3733838969 shutdown_connections 2024-03-06T07:22:37.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.851+0000 7fefbdb55640 1 -- 172.21.15.83:0/3733838969 wait complete. 2024-03-06T07:22:37.863 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:22:37.863 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:37.882 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:37.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.985+0000 7fca191b5640 1 Processor -- start 2024-03-06T07:22:37.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.985+0000 7fca191b5640 1 -- start start 2024-03-06T07:22:37.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.986+0000 7fca191b5640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fca1415e3a0 0x7fca1406afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.986+0000 7fca191b5640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca1415eb40 con 0x7fca14158600 2024-03-06T07:22:37.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.986+0000 7fca191b5640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca1415ee20 con 0x7fca14072730 2024-03-06T07:22:37.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.986+0000 7fca191b5640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca14161ba0 con 0x7fca1415e3a0 2024-03-06T07:22:37.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.986+0000 7fca12d9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fca14158600 0x7fca1406d970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39128/0 (socket says 172.21.15.83:39128) 2024-03-06T07:22:37.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.986+0000 7fca12d9d640 1 -- 172.21.15.83:0/1103407608 learned_addr learned my addr 172.21.15.83:0/1103407608 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:37.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.986+0000 7fca1359e640 1 --2- 172.21.15.83:0/1103407608 >> v2:172.21.15.183:3300/0 conn(0x7fca1415e3a0 0x7fca1406afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.986+0000 7fca1259c640 1 -- 172.21.15.83:0/1103407608 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1868731361 0 0) 0x7fca1415eb40 con 0x7fca14158600 2024-03-06T07:22:37.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.986+0000 7fca1259c640 1 -- 172.21.15.83:0/1103407608 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca00003520 con 0x7fca14158600 2024-03-06T07:22:37.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.986+0000 7fca1259c640 1 -- 172.21.15.83:0/1103407608 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1729167696 0 0) 0x7fca00003520 con 0x7fca14158600 2024-03-06T07:22:37.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.986+0000 7fca1359e640 1 -- 172.21.15.83:0/1103407608 >> v1:172.21.15.183:6789/0 conn(0x7fca14072730 legacy=0x7fca14162030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.986+0000 7fca1359e640 1 -- 172.21.15.83:0/1103407608 >> v1:172.21.15.83:6789/0 conn(0x7fca14158600 legacy=0x7fca1406d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.986+0000 7fca1359e640 1 -- 172.21.15.83:0/1103407608 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca1415f100 con 0x7fca1415e3a0 2024-03-06T07:22:37.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.987+0000 7fca1359e640 1 --2- 172.21.15.83:0/1103407608 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fca1415e3a0 0x7fca1406afb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fca0c009d90 tx=0x7fca0c02bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=797d85d759c7bba8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.987+0000 7fca1259c640 1 -- 172.21.15.83:0/1103407608 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0c039070 con 0x7fca1415e3a0 2024-03-06T07:22:37.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.987+0000 7fca1259c640 1 -- 172.21.15.83:0/1103407608 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca0c02c8b0 con 0x7fca1415e3a0 2024-03-06T07:22:37.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.987+0000 7fca1259c640 1 -- 172.21.15.83:0/1103407608 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca04001670 con 0x7fca1415e3a0 2024-03-06T07:22:37.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.988+0000 7fca191b5640 1 -- 172.21.15.83:0/1103407608 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fca1415e3a0 msgr2=0x7fca1406afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.988+0000 7fca191b5640 1 --2- 172.21.15.83:0/1103407608 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fca1415e3a0 0x7fca1406afb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fca0c009d90 tx=0x7fca0c02bd80 comp rx=0 tx=0).stop 2024-03-06T07:22:37.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.988+0000 7fca191b5640 1 -- 172.21.15.83:0/1103407608 shutdown_connections 2024-03-06T07:22:37.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.988+0000 7fca191b5640 1 --2- 172.21.15.83:0/1103407608 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fca1415e3a0 0x7fca1406afb0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.988+0000 7fca191b5640 1 -- 172.21.15.83:0/1103407608 >> 172.21.15.83:0/1103407608 conn(0x7fca1406c0c0 msgr2=0x7fca1406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:37.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.988+0000 7fca191b5640 1 -- 172.21.15.83:0/1103407608 shutdown_connections 2024-03-06T07:22:37.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.988+0000 7fca191b5640 1 -- 172.21.15.83:0/1103407608 wait complete. 2024-03-06T07:22:37.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.989+0000 7fca191b5640 1 Processor -- start 2024-03-06T07:22:37.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.990+0000 7fca191b5640 1 -- start start 2024-03-06T07:22:37.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.990+0000 7fca191b5640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fca14072730 0x7fca14144220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.990+0000 7fca191b5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fca14158600 0x7fca14153700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.990+0000 7fca191b5640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fca1415e3a0 0x7fca141509a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.990+0000 7fca191b5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca1406e080 con 0x7fca14158600 2024-03-06T07:22:37.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.990+0000 7fca191b5640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca14072be0 con 0x7fca14072730 2024-03-06T07:22:37.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.990+0000 7fca191b5640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca1414e050 con 0x7fca1415e3a0 2024-03-06T07:22:37.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.990+0000 7fca0bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fca14158600 0x7fca14153700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.990+0000 7fca0bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fca14158600 0x7fca14153700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53592/0 (socket says 172.21.15.83:53592) 2024-03-06T07:22:37.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.990+0000 7fca0bfff640 1 -- 172.21.15.83:0/372600939 learned_addr learned my addr 172.21.15.83:0/372600939 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:37.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.990+0000 7fca12d9d640 1 --2- 172.21.15.83:0/372600939 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fca14072730 0x7fca14144220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.990+0000 7fca1359e640 1 --2- 172.21.15.83:0/372600939 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fca1415e3a0 0x7fca141509a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.991+0000 7fca0bfff640 1 -- 172.21.15.83:0/372600939 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fca1415e3a0 msgr2=0x7fca141509a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.991+0000 7fca0bfff640 1 --2- 172.21.15.83:0/372600939 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fca1415e3a0 0x7fca141509a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.991+0000 7fca0bfff640 1 -- 172.21.15.83:0/372600939 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fca14072730 msgr2=0x7fca14144220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.991+0000 7fca0bfff640 1 --2- 172.21.15.83:0/372600939 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fca14072730 0x7fca14144220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.991+0000 7fca0bfff640 1 -- 172.21.15.83:0/372600939 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca141541f0 con 0x7fca14158600 2024-03-06T07:22:37.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.991+0000 7fca0bfff640 1 --2- 172.21.15.83:0/372600939 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fca14158600 0x7fca14153700 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc00c400 tx=0x7fc9fc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.991+0000 7fca0b7fe640 1 -- 172.21.15.83:0/372600939 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9fc017070 con 0x7fca14158600 2024-03-06T07:22:37.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.991+0000 7fca0b7fe640 1 -- 172.21.15.83:0/372600939 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9fc0052d0 con 0x7fca14158600 2024-03-06T07:22:37.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.991+0000 7fca191b5640 1 -- 172.21.15.83:0/372600939 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca141544d0 con 0x7fca14158600 2024-03-06T07:22:37.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.991+0000 7fca0b7fe640 1 -- 172.21.15.83:0/372600939 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9fc003830 con 0x7fca14158600 2024-03-06T07:22:37.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.991+0000 7fca191b5640 1 -- 172.21.15.83:0/372600939 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca1414e2a0 con 0x7fca14158600 2024-03-06T07:22:37.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.992+0000 7fca191b5640 1 -- 172.21.15.83:0/372600939 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca1414e4a0 con 0x7fca14158600 2024-03-06T07:22:37.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.992+0000 7fca0b7fe640 1 -- 172.21.15.83:0/372600939 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc9fc01a070 con 0x7fca14158600 2024-03-06T07:22:37.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.993+0000 7fca0b7fe640 1 --2- 172.21.15.83:0/372600939 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc9f406b4d0 0x7fc9f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.993+0000 7fca0b7fe640 1 -- 172.21.15.83:0/372600939 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc9fc0894b0 con 0x7fca14158600 2024-03-06T07:22:37.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.993+0000 7fca12d9d640 1 --2- 172.21.15.83:0/372600939 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc9f406b4d0 0x7fc9f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:37.993+0000 7fca12d9d640 1 --2- 172.21.15.83:0/372600939 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc9f406b4d0 0x7fc9f406d990 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fca04005880 tx=0x7fca040057f0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.001+0000 7fca0b7fe640 1 -- 172.21.15.83:0/372600939 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fca1414e4a0 con 0x7fca14158600 2024-03-06T07:22:38.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.182+0000 7fca191b5640 1 -- 172.21.15.83:0/372600939 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca1404f330 con 0x7fca14158600 2024-03-06T07:22:38.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.183+0000 7fca0b7fe640 1 -- 172.21.15.83:0/372600939 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16617 (secure 0 0 0) 0x7fc9fc057ce0 con 0x7fca14158600 2024-03-06T07:22:38.185 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:22:38.185 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":20,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:21:59.938137+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.318942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.254795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:38.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.185+0000 7fca191b5640 1 -- 172.21.15.83:0/372600939 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc9f406b4d0 msgr2=0x7fc9f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.185+0000 7fca191b5640 1 --2- 172.21.15.83:0/372600939 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc9f406b4d0 0x7fc9f406d990 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fca04005880 tx=0x7fca040057f0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.185+0000 7fca191b5640 1 -- 172.21.15.83:0/372600939 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fca14158600 msgr2=0x7fca14153700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.185+0000 7fca191b5640 1 --2- 172.21.15.83:0/372600939 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fca14158600 0x7fca14153700 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc00c400 tx=0x7fc9fc00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.185+0000 7fca191b5640 1 -- 172.21.15.83:0/372600939 shutdown_connections 2024-03-06T07:22:38.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.185+0000 7fca191b5640 1 --2- 172.21.15.83:0/372600939 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc9f406b4d0 0x7fc9f406d990 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.185+0000 7fca191b5640 1 --2- 172.21.15.83:0/372600939 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fca1415e3a0 0x7fca141509a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.185+0000 7fca191b5640 1 --2- 172.21.15.83:0/372600939 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fca14158600 0x7fca14153700 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.185+0000 7fca191b5640 1 --2- 172.21.15.83:0/372600939 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fca14072730 0x7fca14144220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.185+0000 7fca191b5640 1 -- 172.21.15.83:0/372600939 >> 172.21.15.83:0/372600939 conn(0x7fca1406c0c0 msgr2=0x7fca140b3880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:38.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.185+0000 7fca191b5640 1 -- 172.21.15.83:0/372600939 shutdown_connections 2024-03-06T07:22:38.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.185+0000 7fca191b5640 1 -- 172.21.15.83:0/372600939 wait complete. 2024-03-06T07:22:38.198 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:56.318942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:57.254795+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:38.198 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:38.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.300+0000 7f16daac9640 1 Processor -- start 2024-03-06T07:22:38.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.300+0000 7f16daac9640 1 -- start start 2024-03-06T07:22:38.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.301+0000 7f16daac9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f16d415c050 0x7f16d406f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.301+0000 7f16daac9640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f16d40d0af0 0x7f16d40cce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.301+0000 7f16daac9640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16d40cd4b0 con 0x7f16d406fb40 2024-03-06T07:22:38.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.301+0000 7f16daac9640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16d406ffb0 con 0x7f16d415c050 2024-03-06T07:22:38.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.301+0000 7f16daac9640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16d40cd790 con 0x7f16d40d0af0 2024-03-06T07:22:38.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.301+0000 7f16d8865640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f16d415c050 0x7f16d406f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.301+0000 7f16d8865640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f16d415c050 0x7f16d406f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53604/0 (socket says 172.21.15.83:53604) 2024-03-06T07:22:38.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.301+0000 7f16d8865640 1 -- 172.21.15.83:0/1307552295 learned_addr learned my addr 172.21.15.83:0/1307552295 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:38.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.301+0000 7f16d3fff640 1 --1- 172.21.15.83:0/1307552295 >> v1:172.21.15.83:6789/0 conn(0x7f16d406fb40 0x7f16d40cc760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39144/0 (socket says 172.21.15.83:39144) 2024-03-06T07:22:38.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.302+0000 7f16d9066640 1 --2- 172.21.15.83:0/1307552295 >> v2:172.21.15.183:3300/0 conn(0x7f16d40d0af0 0x7f16d40cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.302+0000 7f16d8865640 1 -- 172.21.15.83:0/1307552295 >> v2:172.21.15.183:3300/0 conn(0x7f16d40d0af0 msgr2=0x7f16d40cce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:38.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.302+0000 7f16d8865640 1 --2- 172.21.15.83:0/1307552295 >> v2:172.21.15.183:3300/0 conn(0x7f16d40d0af0 0x7f16d40cce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.302+0000 7f16d8865640 1 -- 172.21.15.83:0/1307552295 >> v1:172.21.15.83:6789/0 conn(0x7f16d406fb40 legacy=0x7f16d40cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.302+0000 7f16d8865640 1 -- 172.21.15.83:0/1307552295 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16d40cda70 con 0x7f16d415c050 2024-03-06T07:22:38.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.302+0000 7f16d8865640 1 --2- 172.21.15.83:0/1307552295 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16d415c050 0x7f16d406f600 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f16d406b9c0 tx=0x7f16c802b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a95234f402dee567 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.303+0000 7f16d37fe640 1 -- 172.21.15.83:0/1307552295 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16c8035020 con 0x7f16d415c050 2024-03-06T07:22:38.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.303+0000 7f16d37fe640 1 -- 172.21.15.83:0/1307552295 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16c8031890 con 0x7f16d415c050 2024-03-06T07:22:38.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.303+0000 7f16d37fe640 1 -- 172.21.15.83:0/1307552295 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16c8031c60 con 0x7f16d415c050 2024-03-06T07:22:38.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.303+0000 7f16daac9640 1 -- 172.21.15.83:0/1307552295 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16d415c050 msgr2=0x7f16d406f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.303+0000 7f16daac9640 1 --2- 172.21.15.83:0/1307552295 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16d415c050 0x7f16d406f600 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f16d406b9c0 tx=0x7f16c802b930 comp rx=0 tx=0).stop 2024-03-06T07:22:38.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.303+0000 7f16daac9640 1 -- 172.21.15.83:0/1307552295 shutdown_connections 2024-03-06T07:22:38.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.303+0000 7f16daac9640 1 --2- 172.21.15.83:0/1307552295 >> v2:172.21.15.183:3300/0 conn(0x7f16d40d0af0 0x7f16d40cce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.303+0000 7f16daac9640 1 --2- 172.21.15.83:0/1307552295 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16d415c050 0x7f16d406f600 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.303+0000 7f16daac9640 1 -- 172.21.15.83:0/1307552295 >> 172.21.15.83:0/1307552295 conn(0x7f16d40b3800 msgr2=0x7f16d40b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:38.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.303+0000 7f16daac9640 1 -- 172.21.15.83:0/1307552295 shutdown_connections 2024-03-06T07:22:38.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.303+0000 7f16daac9640 1 -- 172.21.15.83:0/1307552295 wait complete. 2024-03-06T07:22:38.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.304+0000 7f16daac9640 1 Processor -- start 2024-03-06T07:22:38.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.304+0000 7f16daac9640 1 -- start start 2024-03-06T07:22:38.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.305+0000 7f16daac9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16d406fb40 0x7f16d40cb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.305+0000 7f16daac9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16d40d0af0 0x7f16d4145270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.305+0000 7f16daac9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16d415c050 0x7f16d41457b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.305+0000 7f16daac9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16d4165c30 con 0x7f16d406fb40 2024-03-06T07:22:38.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.305+0000 7f16daac9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16d4165ab0 con 0x7f16d40d0af0 2024-03-06T07:22:38.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.305+0000 7f16daac9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16d4165db0 con 0x7f16d415c050 2024-03-06T07:22:38.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.305+0000 7f16d8865640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16d406fb40 0x7f16d40cb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.305+0000 7f16d8865640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16d406fb40 0x7f16d40cb0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53618/0 (socket says 172.21.15.83:53618) 2024-03-06T07:22:38.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.305+0000 7f16d8865640 1 -- 172.21.15.83:0/3423683604 learned_addr learned my addr 172.21.15.83:0/3423683604 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:38.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.305+0000 7f16d3fff640 1 --2- 172.21.15.83:0/3423683604 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16d40d0af0 0x7f16d4145270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.306+0000 7f16d9066640 1 --2- 172.21.15.83:0/3423683604 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16d415c050 0x7f16d41457b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.306+0000 7f16d8865640 1 -- 172.21.15.83:0/3423683604 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16d415c050 msgr2=0x7f16d41457b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.306+0000 7f16d8865640 1 --2- 172.21.15.83:0/3423683604 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16d415c050 0x7f16d41457b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.306+0000 7f16d8865640 1 -- 172.21.15.83:0/3423683604 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16d40d0af0 msgr2=0x7f16d4145270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.306+0000 7f16d8865640 1 --2- 172.21.15.83:0/3423683604 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16d40d0af0 0x7f16d4145270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.306+0000 7f16d8865640 1 -- 172.21.15.83:0/3423683604 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16d406e070 con 0x7f16d406fb40 2024-03-06T07:22:38.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.306+0000 7f16d3fff640 1 --2- 172.21.15.83:0/3423683604 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16d40d0af0 0x7f16d4145270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:38.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.306+0000 7f16d8865640 1 --2- 172.21.15.83:0/3423683604 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16d406fb40 0x7f16d40cb0f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f16c8000c00 tx=0x7f16c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.306+0000 7f16d17fa640 1 -- 172.21.15.83:0/3423683604 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16c803ad60 con 0x7f16d406fb40 2024-03-06T07:22:38.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.306+0000 7f16d17fa640 1 -- 172.21.15.83:0/3423683604 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16c80443f0 con 0x7f16d406fb40 2024-03-06T07:22:38.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.306+0000 7f16daac9640 1 -- 172.21.15.83:0/3423683604 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16d406cfd0 con 0x7f16d406fb40 2024-03-06T07:22:38.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.306+0000 7f16d17fa640 1 -- 172.21.15.83:0/3423683604 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16c80447e0 con 0x7f16d406fb40 2024-03-06T07:22:38.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.306+0000 7f16daac9640 1 -- 172.21.15.83:0/3423683604 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16d406d590 con 0x7f16d406fb40 2024-03-06T07:22:38.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.307+0000 7f16daac9640 1 -- 172.21.15.83:0/3423683604 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16d409fe60 con 0x7f16d406fb40 2024-03-06T07:22:38.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.307+0000 7f16d17fa640 1 -- 172.21.15.83:0/3423683604 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f16c804b060 con 0x7f16d406fb40 2024-03-06T07:22:38.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.308+0000 7f16d17fa640 1 --2- 172.21.15.83:0/3423683604 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f16b806b4d0 0x7f16b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.308+0000 7f16d17fa640 1 -- 172.21.15.83:0/3423683604 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f16c80ac410 con 0x7f16d406fb40 2024-03-06T07:22:38.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.308+0000 7f16d3fff640 1 --2- 172.21.15.83:0/3423683604 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f16b806b4d0 0x7f16b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.308+0000 7f16d3fff640 1 --2- 172.21.15.83:0/3423683604 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f16b806b4d0 0x7f16b806d990 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f16c0007fd0 tx=0x7f16c0007d60 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.312+0000 7f16d17fa640 1 -- 172.21.15.83:0/3423683604 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f16c803f070 con 0x7f16d406fb40 2024-03-06T07:22:38.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.491+0000 7f16daac9640 1 -- 172.21.15.83:0/3423683604 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f16d4068ff0 con 0x7f16d406fb40 2024-03-06T07:22:38.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.491+0000 7f16d17fa640 1 -- 172.21.15.83:0/3423683604 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f16c802d040 con 0x7f16d406fb40 2024-03-06T07:22:38.493 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:22:38.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.493+0000 7f16daac9640 1 -- 172.21.15.83:0/3423683604 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f16b806b4d0 msgr2=0x7f16b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.493+0000 7f16daac9640 1 --2- 172.21.15.83:0/3423683604 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f16b806b4d0 0x7f16b806d990 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f16c0007fd0 tx=0x7f16c0007d60 comp rx=0 tx=0).stop 2024-03-06T07:22:38.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.493+0000 7f16daac9640 1 -- 172.21.15.83:0/3423683604 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16d406fb40 msgr2=0x7f16d40cb0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.493+0000 7f16daac9640 1 --2- 172.21.15.83:0/3423683604 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16d406fb40 0x7f16d40cb0f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f16c8000c00 tx=0x7f16c80046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.493+0000 7f16daac9640 1 -- 172.21.15.83:0/3423683604 shutdown_connections 2024-03-06T07:22:38.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.493+0000 7f16daac9640 1 --2- 172.21.15.83:0/3423683604 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f16b806b4d0 0x7f16b806d990 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.493+0000 7f16daac9640 1 --2- 172.21.15.83:0/3423683604 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16d415c050 0x7f16d41457b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.493+0000 7f16daac9640 1 --2- 172.21.15.83:0/3423683604 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16d40d0af0 0x7f16d4145270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.493+0000 7f16daac9640 1 --2- 172.21.15.83:0/3423683604 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16d406fb40 0x7f16d40cb0f0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.493+0000 7f16daac9640 1 -- 172.21.15.83:0/3423683604 >> 172.21.15.83:0/3423683604 conn(0x7f16d40b3800 msgr2=0x7f16d4159070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:38.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.493+0000 7f16daac9640 1 -- 172.21.15.83:0/3423683604 shutdown_connections 2024-03-06T07:22:38.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.493+0000 7f16daac9640 1 -- 172.21.15.83:0/3423683604 wait complete. 2024-03-06T07:22:38.507 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:38.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.609+0000 7f42ab439640 1 Processor -- start 2024-03-06T07:22:38.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.609+0000 7f42ab439640 1 -- start start 2024-03-06T07:22:38.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.610+0000 7f42ab439640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f42a415c050 0x7f42a406f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.610+0000 7f42ab439640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f42a406fb40 0x7f42a40cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.610+0000 7f42ab439640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42a40cd4b0 con 0x7f42a40ccca0 2024-03-06T07:22:38.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.610+0000 7f42ab439640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42a406ff80 con 0x7f42a415c050 2024-03-06T07:22:38.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.610+0000 7f42ab439640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42a40cd790 con 0x7f42a406fb40 2024-03-06T07:22:38.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.611+0000 7f42a89d4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f42a406fb40 0x7f42a40cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.611+0000 7f42a89d4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f42a406fb40 0x7f42a40cc760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:40490/0 (socket says 172.21.15.83:40490) 2024-03-06T07:22:38.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.611+0000 7f42a89d4640 1 -- 172.21.15.83:0/260224311 learned_addr learned my addr 172.21.15.83:0/260224311 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:38.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.611+0000 7f42a91d5640 1 --2- 172.21.15.83:0/260224311 >> v2:172.21.15.183:3300/0 conn(0x7f42a415c050 0x7f42a406f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.611+0000 7f42a89d4640 1 -- 172.21.15.83:0/260224311 >> v2:172.21.15.183:3300/0 conn(0x7f42a415c050 msgr2=0x7f42a406f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:38.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.611+0000 7f42a89d4640 1 --2- 172.21.15.83:0/260224311 >> v2:172.21.15.183:3300/0 conn(0x7f42a415c050 0x7f42a406f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.611+0000 7f42a89d4640 1 -- 172.21.15.83:0/260224311 >> v1:172.21.15.183:6789/0 conn(0x7f42a40ccca0 legacy=0x7f42a40cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.611+0000 7f42a89d4640 1 -- 172.21.15.83:0/260224311 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42a40cda70 con 0x7f42a406fb40 2024-03-06T07:22:38.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.612+0000 7f42a89d4640 1 --2- 172.21.15.83:0/260224311 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f42a406fb40 0x7f42a40cc760 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4294009870 tx=0x7f429402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a72324dfcc09bfbc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.612+0000 7f429bfff640 1 -- 172.21.15.83:0/260224311 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4294035020 con 0x7f42a406fb40 2024-03-06T07:22:38.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.612+0000 7f429bfff640 1 -- 172.21.15.83:0/260224311 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4294031640 con 0x7f42a406fb40 2024-03-06T07:22:38.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.612+0000 7f429bfff640 1 -- 172.21.15.83:0/260224311 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f429402d040 con 0x7f42a406fb40 2024-03-06T07:22:38.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.612+0000 7f42ab439640 1 -- 172.21.15.83:0/260224311 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f42a406fb40 msgr2=0x7f42a40cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.612+0000 7f42ab439640 1 --2- 172.21.15.83:0/260224311 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f42a406fb40 0x7f42a40cc760 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4294009870 tx=0x7f429402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.612+0000 7f42ab439640 1 -- 172.21.15.83:0/260224311 shutdown_connections 2024-03-06T07:22:38.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.612+0000 7f42ab439640 1 --2- 172.21.15.83:0/260224311 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f42a406fb40 0x7f42a40cc760 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.612+0000 7f42ab439640 1 --2- 172.21.15.83:0/260224311 >> v2:172.21.15.183:3300/0 conn(0x7f42a415c050 0x7f42a406f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.612+0000 7f42ab439640 1 -- 172.21.15.83:0/260224311 >> 172.21.15.83:0/260224311 conn(0x7f42a40b3800 msgr2=0x7f42a40b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:38.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.612+0000 7f42ab439640 1 -- 172.21.15.83:0/260224311 shutdown_connections 2024-03-06T07:22:38.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.613+0000 7f42ab439640 1 -- 172.21.15.83:0/260224311 wait complete. 2024-03-06T07:22:38.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.614+0000 7f42ab439640 1 Processor -- start 2024-03-06T07:22:38.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.614+0000 7f42ab439640 1 -- start start 2024-03-06T07:22:38.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.614+0000 7f42ab439640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42a406fb40 0x7f42a4149870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.614+0000 7f42ab439640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f42a40ccca0 0x7f42a41f74b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.614+0000 7f42ab439640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f42a415c050 0x7f42a40cae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.614+0000 7f42ab439640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a40d01f0 con 0x7f42a406fb40 2024-03-06T07:22:38.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.614+0000 7f42ab439640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a40cd130 con 0x7f42a415c050 2024-03-06T07:22:38.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.614+0000 7f42ab439640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42a40d0370 con 0x7f42a40ccca0 2024-03-06T07:22:38.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.614+0000 7f42a91d5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42a406fb40 0x7f42a4149870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.614+0000 7f42a91d5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42a406fb40 0x7f42a4149870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53626/0 (socket says 172.21.15.83:53626) 2024-03-06T07:22:38.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.614+0000 7f42a91d5640 1 -- 172.21.15.83:0/1766313037 learned_addr learned my addr 172.21.15.83:0/1766313037 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:38.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.615+0000 7f42a89d4640 1 --2- 172.21.15.83:0/1766313037 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f42a40ccca0 0x7f42a41f74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.615+0000 7f42a99d6640 1 --2- 172.21.15.83:0/1766313037 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f42a415c050 0x7f42a40cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.615+0000 7f42a91d5640 1 -- 172.21.15.83:0/1766313037 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f42a40ccca0 msgr2=0x7f42a41f74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.615+0000 7f42a91d5640 1 --2- 172.21.15.83:0/1766313037 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f42a40ccca0 0x7f42a41f74b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.615+0000 7f42a91d5640 1 -- 172.21.15.83:0/1766313037 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f42a415c050 msgr2=0x7f42a40cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.615+0000 7f42a91d5640 1 --2- 172.21.15.83:0/1766313037 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f42a415c050 0x7f42a40cae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.615+0000 7f42a91d5640 1 -- 172.21.15.83:0/1766313037 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42a41f7f00 con 0x7f42a406fb40 2024-03-06T07:22:38.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.615+0000 7f42a91d5640 1 --2- 172.21.15.83:0/1766313037 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42a406fb40 0x7f42a4149870 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f42a000e9b0 tx=0x7f42a000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.615+0000 7f4299ffb640 1 -- 172.21.15.83:0/1766313037 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a0017070 con 0x7f42a406fb40 2024-03-06T07:22:38.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.615+0000 7f4299ffb640 1 -- 172.21.15.83:0/1766313037 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42a000c9a0 con 0x7f42a406fb40 2024-03-06T07:22:38.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.615+0000 7f42ab439640 1 -- 172.21.15.83:0/1766313037 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42a41f81e0 con 0x7f42a406fb40 2024-03-06T07:22:38.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.615+0000 7f4299ffb640 1 -- 172.21.15.83:0/1766313037 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a000cd90 con 0x7f42a406fb40 2024-03-06T07:22:38.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.615+0000 7f42ab439640 1 -- 172.21.15.83:0/1766313037 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42a40b4e20 con 0x7f42a406fb40 2024-03-06T07:22:38.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.616+0000 7f42ab439640 1 -- 172.21.15.83:0/1766313037 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42a409fe60 con 0x7f42a406fb40 2024-03-06T07:22:38.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.617+0000 7f4299ffb640 1 -- 172.21.15.83:0/1766313037 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f42a001a070 con 0x7f42a406fb40 2024-03-06T07:22:38.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.617+0000 7f4299ffb640 1 --2- 172.21.15.83:0/1766313037 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f428806b4d0 0x7f428806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.617+0000 7f4299ffb640 1 -- 172.21.15.83:0/1766313037 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f42a008a510 con 0x7f42a406fb40 2024-03-06T07:22:38.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.617+0000 7f42a89d4640 1 --2- 172.21.15.83:0/1766313037 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f428806b4d0 0x7f428806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.618+0000 7f42a89d4640 1 --2- 172.21.15.83:0/1766313037 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f428806b4d0 0x7f428806d990 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4294036000 tx=0x7f429403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.625+0000 7f4299ffb640 1 -- 172.21.15.83:0/1766313037 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f42a0058d50 con 0x7f42a406fb40 2024-03-06T07:22:38.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.802+0000 7f42ab439640 1 -- 172.21.15.83:0/1766313037 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f42a4068ff0 con 0x7f42a406fb40 2024-03-06T07:22:38.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.802+0000 7f4299ffb640 1 -- 172.21.15.83:0/1766313037 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f42a005cdf0 con 0x7f42a406fb40 2024-03-06T07:22:38.804 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-03-06T07:22:38.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.804+0000 7f42ab439640 1 -- 172.21.15.83:0/1766313037 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f428806b4d0 msgr2=0x7f428806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.804+0000 7f42ab439640 1 --2- 172.21.15.83:0/1766313037 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f428806b4d0 0x7f428806d990 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4294036000 tx=0x7f429403c040 comp rx=0 tx=0).stop 2024-03-06T07:22:38.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.804+0000 7f42ab439640 1 -- 172.21.15.83:0/1766313037 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42a406fb40 msgr2=0x7f42a4149870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.804+0000 7f42ab439640 1 --2- 172.21.15.83:0/1766313037 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42a406fb40 0x7f42a4149870 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f42a000e9b0 tx=0x7f42a000ee70 comp rx=0 tx=0).stop 2024-03-06T07:22:38.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.804+0000 7f42ab439640 1 -- 172.21.15.83:0/1766313037 shutdown_connections 2024-03-06T07:22:38.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.804+0000 7f42ab439640 1 --2- 172.21.15.83:0/1766313037 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f428806b4d0 0x7f428806d990 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.804+0000 7f42ab439640 1 --2- 172.21.15.83:0/1766313037 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f42a415c050 0x7f42a40cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.804+0000 7f42ab439640 1 --2- 172.21.15.83:0/1766313037 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f42a40ccca0 0x7f42a41f74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.804+0000 7f42ab439640 1 --2- 172.21.15.83:0/1766313037 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42a406fb40 0x7f42a4149870 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.804+0000 7f42ab439640 1 -- 172.21.15.83:0/1766313037 >> 172.21.15.83:0/1766313037 conn(0x7f42a40b3800 msgr2=0x7f42a40d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:38.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.804+0000 7f42ab439640 1 -- 172.21.15.83:0/1766313037 shutdown_connections 2024-03-06T07:22:38.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.804+0000 7f42ab439640 1 -- 172.21.15.83:0/1766313037 wait complete. 2024-03-06T07:22:38.817 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:38.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.922+0000 7f1106839640 1 Processor -- start 2024-03-06T07:22:38.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.922+0000 7f1106839640 1 -- start start 2024-03-06T07:22:38.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.923+0000 7f1106839640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f11001614d0 0x7f11000d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.923+0000 7f1106839640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11000d24b0 0x7f11000b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.923+0000 7f1106839640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11001618b0 con 0x7f11000d1910 2024-03-06T07:22:38.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.923+0000 7f1106839640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11000cf400 con 0x7f11001614d0 2024-03-06T07:22:38.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.923+0000 7f1106839640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11000cf6a0 con 0x7f11000d24b0 2024-03-06T07:22:38.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.923+0000 7f10ff7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f11000d1910 0x7f11000d1da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39150/0 (socket says 172.21.15.83:39150) 2024-03-06T07:22:38.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.923+0000 7f10ff7fe640 1 -- 172.21.15.83:0/1374140030 learned_addr learned my addr 172.21.15.83:0/1374140030 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:38.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.924+0000 7f1104dd6640 1 --2- 172.21.15.83:0/1374140030 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11000d24b0 0x7f11000b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.924+0000 7f10fffff640 1 --2- 172.21.15.83:0/1374140030 >> v2:172.21.15.183:3300/0 conn(0x7f11001614d0 0x7f11000d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.924+0000 7f10feffd640 1 -- 172.21.15.83:0/1374140030 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3211801624 0 0) 0x7f11001618b0 con 0x7f11000d1910 2024-03-06T07:22:38.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.924+0000 7f10feffd640 1 -- 172.21.15.83:0/1374140030 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10e8003520 con 0x7f11000d1910 2024-03-06T07:22:38.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.924+0000 7f10feffd640 1 -- 172.21.15.83:0/1374140030 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1754830281 0 0) 0x7f10e8003520 con 0x7f11000d1910 2024-03-06T07:22:38.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.924+0000 7f10fffff640 1 -- 172.21.15.83:0/1374140030 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11000d24b0 msgr2=0x7f11000b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:38.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.924+0000 7f10fffff640 1 --2- 172.21.15.83:0/1374140030 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11000d24b0 0x7f11000b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.924+0000 7f10fffff640 1 -- 172.21.15.83:0/1374140030 >> v1:172.21.15.83:6789/0 conn(0x7f11000d1910 legacy=0x7f11000d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.924+0000 7f10fffff640 1 -- 172.21.15.83:0/1374140030 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11000cf980 con 0x7f11001614d0 2024-03-06T07:22:38.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.924+0000 7f1104dd6640 1 --2- 172.21.15.83:0/1374140030 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11000d24b0 0x7f11000b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:38.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.925+0000 7f10fffff640 1 --2- 172.21.15.83:0/1374140030 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11001614d0 0x7f11000d4860 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f10f00097a0 tx=0x7f10f002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4edc77fd9d49fb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.925+0000 7f10feffd640 1 -- 172.21.15.83:0/1374140030 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f0037070 con 0x7f11001614d0 2024-03-06T07:22:38.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.925+0000 7f10feffd640 1 -- 172.21.15.83:0/1374140030 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f0030510 con 0x7f11001614d0 2024-03-06T07:22:38.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.925+0000 7f10feffd640 1 -- 172.21.15.83:0/1374140030 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f4001670 con 0x7f11001614d0 2024-03-06T07:22:38.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.925+0000 7f1106839640 1 -- 172.21.15.83:0/1374140030 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11001614d0 msgr2=0x7f11000d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.925+0000 7f1106839640 1 --2- 172.21.15.83:0/1374140030 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11001614d0 0x7f11000d4860 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f10f00097a0 tx=0x7f10f002b790 comp rx=0 tx=0).stop 2024-03-06T07:22:38.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.926+0000 7f1106839640 1 -- 172.21.15.83:0/1374140030 shutdown_connections 2024-03-06T07:22:38.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.926+0000 7f1106839640 1 --2- 172.21.15.83:0/1374140030 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11000d24b0 0x7f11000b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.926+0000 7f1106839640 1 --2- 172.21.15.83:0/1374140030 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11001614d0 0x7f11000d4860 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.926+0000 7f1106839640 1 -- 172.21.15.83:0/1374140030 >> 172.21.15.83:0/1374140030 conn(0x7f11000d3d40 msgr2=0x7f11000d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:38.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.926+0000 7f1106839640 1 -- 172.21.15.83:0/1374140030 shutdown_connections 2024-03-06T07:22:38.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.926+0000 7f1106839640 1 -- 172.21.15.83:0/1374140030 wait complete. 2024-03-06T07:22:38.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.927+0000 7f1106839640 1 Processor -- start 2024-03-06T07:22:38.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.927+0000 7f1106839640 1 -- start start 2024-03-06T07:22:38.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.928+0000 7f1106839640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11000d1910 0x7f1100155fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.928+0000 7f1106839640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11000d24b0 0x7f11000c8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.928+0000 7f1106839640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11001614d0 0x7f11000cce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.928+0000 7f1106839640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11000b1370 con 0x7f11000d24b0 2024-03-06T07:22:38.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.928+0000 7f1106839640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11000b0910 con 0x7f11001614d0 2024-03-06T07:22:38.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.928+0000 7f1106839640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11000b14f0 con 0x7f11000d1910 2024-03-06T07:22:38.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.928+0000 7f10ff7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11000d24b0 0x7f11000c8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.928+0000 7f10ff7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11000d24b0 0x7f11000c8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53642/0 (socket says 172.21.15.83:53642) 2024-03-06T07:22:38.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.928+0000 7f10ff7fe640 1 -- 172.21.15.83:0/1139815847 learned_addr learned my addr 172.21.15.83:0/1139815847 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:38.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.928+0000 7f1104dd6640 1 --2- 172.21.15.83:0/1139815847 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11001614d0 0x7f11000cce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.928+0000 7f10fffff640 1 --2- 172.21.15.83:0/1139815847 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11000d1910 0x7f1100155fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.928+0000 7f10ff7fe640 1 -- 172.21.15.83:0/1139815847 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11000d1910 msgr2=0x7f1100155fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.928+0000 7f10ff7fe640 1 --2- 172.21.15.83:0/1139815847 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11000d1910 0x7f1100155fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.928+0000 7f10ff7fe640 1 -- 172.21.15.83:0/1139815847 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11001614d0 msgr2=0x7f11000cce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.928+0000 7f10ff7fe640 1 --2- 172.21.15.83:0/1139815847 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11001614d0 0x7f11000cce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.928+0000 7f10ff7fe640 1 -- 172.21.15.83:0/1139815847 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11001564e0 con 0x7f11000d24b0 2024-03-06T07:22:38.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.928+0000 7f10ff7fe640 1 --2- 172.21.15.83:0/1139815847 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11000d24b0 0x7f11000c8760 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f10f400c750 tx=0x7f10f400cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.928+0000 7f10fcff9640 1 -- 172.21.15.83:0/1139815847 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f4005ad0 con 0x7f11000d24b0 2024-03-06T07:22:38.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.928+0000 7f10fcff9640 1 -- 172.21.15.83:0/1139815847 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f4005c90 con 0x7f11000d24b0 2024-03-06T07:22:38.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.928+0000 7f1106839640 1 -- 172.21.15.83:0/1139815847 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11000c79a0 con 0x7f11000d24b0 2024-03-06T07:22:38.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.928+0000 7f10fcff9640 1 -- 172.21.15.83:0/1139815847 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f40072b0 con 0x7f11000d24b0 2024-03-06T07:22:38.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.928+0000 7f1106839640 1 -- 172.21.15.83:0/1139815847 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11000c7f60 con 0x7f11000d24b0 2024-03-06T07:22:38.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.929+0000 7f1106839640 1 -- 172.21.15.83:0/1139815847 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1100068ff0 con 0x7f11000d24b0 2024-03-06T07:22:38.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.930+0000 7f10fcff9640 1 -- 172.21.15.83:0/1139815847 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f10f4007450 con 0x7f11000d24b0 2024-03-06T07:22:38.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.930+0000 7f10fcff9640 1 --2- 172.21.15.83:0/1139815847 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f10ec06b4d0 0x7f10ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.930+0000 7f10fffff640 1 --2- 172.21.15.83:0/1139815847 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f10ec06b4d0 0x7f10ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.930+0000 7f10fcff9640 1 -- 172.21.15.83:0/1139815847 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f10f4089f90 con 0x7f11000d24b0 2024-03-06T07:22:38.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.931+0000 7f10fffff640 1 --2- 172.21.15.83:0/1139815847 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f10ec06b4d0 0x7f10ec06d990 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f10f0034000 tx=0x7f10f002f000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:38.935+0000 7f10fcff9640 1 -- 172.21.15.83:0/1139815847 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f10f4010300 con 0x7f11000d24b0 2024-03-06T07:22:39.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.122+0000 7f1106839640 1 -- 172.21.15.83:0/1139815847 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f11000d0a20 con 0x7f11000d24b0 2024-03-06T07:22:39.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.123+0000 7f10fcff9640 1 -- 172.21.15.83:0/1139815847 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f10f4058ea0 con 0x7f11000d24b0 2024-03-06T07:22:39.125 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 64 2024-03-06T07:22:39.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.125+0000 7f1106839640 1 -- 172.21.15.83:0/1139815847 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f10ec06b4d0 msgr2=0x7f10ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.125+0000 7f1106839640 1 --2- 172.21.15.83:0/1139815847 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f10ec06b4d0 0x7f10ec06d990 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f10f0034000 tx=0x7f10f002f000 comp rx=0 tx=0).stop 2024-03-06T07:22:39.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.125+0000 7f1106839640 1 -- 172.21.15.83:0/1139815847 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11000d24b0 msgr2=0x7f11000c8760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.125+0000 7f1106839640 1 --2- 172.21.15.83:0/1139815847 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11000d24b0 0x7f11000c8760 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f10f400c750 tx=0x7f10f400cc10 comp rx=0 tx=0).stop 2024-03-06T07:22:39.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.125+0000 7f1106839640 1 -- 172.21.15.83:0/1139815847 shutdown_connections 2024-03-06T07:22:39.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.125+0000 7f1106839640 1 --2- 172.21.15.83:0/1139815847 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f10ec06b4d0 0x7f10ec06d990 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.125+0000 7f1106839640 1 --2- 172.21.15.83:0/1139815847 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11001614d0 0x7f11000cce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.125+0000 7f1106839640 1 --2- 172.21.15.83:0/1139815847 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11000d24b0 0x7f11000c8760 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.125+0000 7f1106839640 1 --2- 172.21.15.83:0/1139815847 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11000d1910 0x7f1100155fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.125+0000 7f1106839640 1 -- 172.21.15.83:0/1139815847 >> 172.21.15.83:0/1139815847 conn(0x7f11000d3d40 msgr2=0x7f110006b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:39.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.125+0000 7f1106839640 1 -- 172.21.15.83:0/1139815847 shutdown_connections 2024-03-06T07:22:39.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.125+0000 7f1106839640 1 -- 172.21.15.83:0/1139815847 wait complete. 2024-03-06T07:22:39.138 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:22:39.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.237+0000 7f54c2639640 1 Processor -- start 2024-03-06T07:22:39.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.237+0000 7f54c2639640 1 -- start start 2024-03-06T07:22:39.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.238+0000 7f54c2639640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f54bc15e010 0x7f54bc0cedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.238+0000 7f54c2639640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f54bc0b3240 0x7f54bc0b23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.238+0000 7f54c2639640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54bc15e3f0 con 0x7f54bc0b2910 2024-03-06T07:22:39.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.238+0000 7f54c2639640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54bc0b1f40 con 0x7f54bc15e010 2024-03-06T07:22:39.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.238+0000 7f54c2639640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54bc0b1c10 con 0x7f54bc0b3240 2024-03-06T07:22:39.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.238+0000 7f54bbfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f54bc15e010 0x7f54bc0cedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.238+0000 7f54bbfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f54bc15e010 0x7f54bc0cedd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53656/0 (socket says 172.21.15.83:53656) 2024-03-06T07:22:39.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.238+0000 7f54bbfff640 1 -- 172.21.15.83:0/2287580690 learned_addr learned my addr 172.21.15.83:0/2287580690 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:39.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.239+0000 7f54bbfff640 1 -- 172.21.15.83:0/2287580690 >> v2:172.21.15.183:3300/0 conn(0x7f54bc0b3240 msgr2=0x7f54bc0b23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:39.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.239+0000 7f54bbfff640 1 --2- 172.21.15.83:0/2287580690 >> v2:172.21.15.183:3300/0 conn(0x7f54bc0b3240 0x7f54bc0b23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.239+0000 7f54bbfff640 1 -- 172.21.15.83:0/2287580690 >> v1:172.21.15.183:6789/0 conn(0x7f54bc0b2910 legacy=0x7f54bc0d1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:39.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.239+0000 7f54bbfff640 1 -- 172.21.15.83:0/2287580690 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54bc06d1b0 con 0x7f54bc15e010 2024-03-06T07:22:39.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.239+0000 7f54bbfff640 1 --2- 172.21.15.83:0/2287580690 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54bc15e010 0x7f54bc0cedd0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f54bc06e2a0 tx=0x7f54a402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbb720edb8ccc0a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.239+0000 7f54baffd640 1 -- 172.21.15.83:0/2287580690 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a4035020 con 0x7f54bc15e010 2024-03-06T07:22:39.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.239+0000 7f54baffd640 1 -- 172.21.15.83:0/2287580690 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54a4031870 con 0x7f54bc15e010 2024-03-06T07:22:39.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.239+0000 7f54baffd640 1 -- 172.21.15.83:0/2287580690 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a4031c40 con 0x7f54bc15e010 2024-03-06T07:22:39.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.240+0000 7f54c2639640 1 -- 172.21.15.83:0/2287580690 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54bc15e010 msgr2=0x7f54bc0cedd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.240+0000 7f54c2639640 1 --2- 172.21.15.83:0/2287580690 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54bc15e010 0x7f54bc0cedd0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f54bc06e2a0 tx=0x7f54a402b910 comp rx=0 tx=0).stop 2024-03-06T07:22:39.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.240+0000 7f54c2639640 1 -- 172.21.15.83:0/2287580690 shutdown_connections 2024-03-06T07:22:39.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.240+0000 7f54c2639640 1 --2- 172.21.15.83:0/2287580690 >> v2:172.21.15.183:3300/0 conn(0x7f54bc0b3240 0x7f54bc0b23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.240+0000 7f54c2639640 1 --2- 172.21.15.83:0/2287580690 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54bc15e010 0x7f54bc0cedd0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.240+0000 7f54c2639640 1 -- 172.21.15.83:0/2287580690 >> 172.21.15.83:0/2287580690 conn(0x7f54bc09a9f0 msgr2=0x7f54bc09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:39.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.240+0000 7f54c2639640 1 -- 172.21.15.83:0/2287580690 shutdown_connections 2024-03-06T07:22:39.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.240+0000 7f54c2639640 1 -- 172.21.15.83:0/2287580690 wait complete. 2024-03-06T07:22:39.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.241+0000 7f54c2639640 1 Processor -- start 2024-03-06T07:22:39.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.241+0000 7f54c2639640 1 -- start start 2024-03-06T07:22:39.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.242+0000 7f54c2639640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54bc15e010 0x7f54bc1522b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.242+0000 7f54c2639640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f54bc156cf0 0x7f54bc0b6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.242+0000 7f54c2639640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f54bc157170 0x7f54bc0b6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.242+0000 7f54c2639640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc0cf310 con 0x7f54bc15e010 2024-03-06T07:22:39.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.242+0000 7f54c2639640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc06ced0 con 0x7f54bc157170 2024-03-06T07:22:39.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.242+0000 7f54c2639640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc0d21e0 con 0x7f54bc156cf0 2024-03-06T07:22:39.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.242+0000 7f54bbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54bc15e010 0x7f54bc1522b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.242+0000 7f54bbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54bc15e010 0x7f54bc1522b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53664/0 (socket says 172.21.15.83:53664) 2024-03-06T07:22:39.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.242+0000 7f54bbfff640 1 -- 172.21.15.83:0/2250069938 learned_addr learned my addr 172.21.15.83:0/2250069938 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:39.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.242+0000 7f54c0bd6640 1 --2- 172.21.15.83:0/2250069938 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f54bc157170 0x7f54bc0b6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.242+0000 7f54bb7fe640 1 --2- 172.21.15.83:0/2250069938 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f54bc156cf0 0x7f54bc0b6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.242+0000 7f54bbfff640 1 -- 172.21.15.83:0/2250069938 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f54bc156cf0 msgr2=0x7f54bc0b6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.242+0000 7f54bbfff640 1 --2- 172.21.15.83:0/2250069938 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f54bc156cf0 0x7f54bc0b6a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.242+0000 7f54bbfff640 1 -- 172.21.15.83:0/2250069938 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f54bc157170 msgr2=0x7f54bc0b6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.242+0000 7f54bbfff640 1 --2- 172.21.15.83:0/2250069938 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f54bc157170 0x7f54bc0b6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.242+0000 7f54bbfff640 1 -- 172.21.15.83:0/2250069938 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54bc0b4870 con 0x7f54bc15e010 2024-03-06T07:22:39.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.242+0000 7f54c0bd6640 1 --2- 172.21.15.83:0/2250069938 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f54bc157170 0x7f54bc0b6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:39.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.242+0000 7f54bb7fe640 1 --2- 172.21.15.83:0/2250069938 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f54bc156cf0 0x7f54bc0b6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:39.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.243+0000 7f54bbfff640 1 --2- 172.21.15.83:0/2250069938 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54bc15e010 0x7f54bc1522b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f54a402b950 tx=0x7f54a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.243+0000 7f54b8ff9640 1 -- 172.21.15.83:0/2250069938 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a403ad60 con 0x7f54bc15e010 2024-03-06T07:22:39.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.243+0000 7f54b8ff9640 1 -- 172.21.15.83:0/2250069938 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54a40443f0 con 0x7f54bc15e010 2024-03-06T07:22:39.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.243+0000 7f54b8ff9640 1 -- 172.21.15.83:0/2250069938 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a40447e0 con 0x7f54bc15e010 2024-03-06T07:22:39.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.243+0000 7f54c2639640 1 -- 172.21.15.83:0/2250069938 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54bc0b4b50 con 0x7f54bc15e010 2024-03-06T07:22:39.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.243+0000 7f54c2639640 1 -- 172.21.15.83:0/2250069938 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54bc0c7df0 con 0x7f54bc15e010 2024-03-06T07:22:39.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.244+0000 7f54c2639640 1 -- 172.21.15.83:0/2250069938 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54bc0d2c90 con 0x7f54bc15e010 2024-03-06T07:22:39.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.244+0000 7f54b8ff9640 1 -- 172.21.15.83:0/2250069938 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f54a4038020 con 0x7f54bc15e010 2024-03-06T07:22:39.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.245+0000 7f54b8ff9640 1 --2- 172.21.15.83:0/2250069938 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f549c06b500 0x7f549c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.245+0000 7f54bb7fe640 1 --2- 172.21.15.83:0/2250069938 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f549c06b500 0x7f549c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.245+0000 7f54b8ff9640 1 -- 172.21.15.83:0/2250069938 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5912+0+0 (secure 0 0 0) 0x7f54a40ac520 con 0x7f54bc15e010 2024-03-06T07:22:39.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.245+0000 7f54bb7fe640 1 --2- 172.21.15.83:0/2250069938 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f549c06b500 0x7f549c06d9c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f54ac006fd0 tx=0x7f54ac008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.249+0000 7f54b8ff9640 1 -- 172.21.15.83:0/2250069938 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f54bc0d2c90 con 0x7f54bc15e010 2024-03-06T07:22:39.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.542+0000 7f54c2639640 1 -- 172.21.15.83:0/2250069938 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f54bc04f330 con 0x7f54bc15e010 2024-03-06T07:22:39.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.542+0000 7f54b8ff9640 1 -- 172.21.15.83:0/2250069938 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7f54a403f070 con 0x7f54bc15e010 2024-03-06T07:22:39.544 INFO:teuthology.orchestra.run.smithi083.stderr:listed 0 entries 2024-03-06T07:22:39.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.544+0000 7f54c2639640 1 -- 172.21.15.83:0/2250069938 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f549c06b500 msgr2=0x7f549c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.544+0000 7f54c2639640 1 --2- 172.21.15.83:0/2250069938 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f549c06b500 0x7f549c06d9c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f54ac006fd0 tx=0x7f54ac008040 comp rx=0 tx=0).stop 2024-03-06T07:22:39.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.544+0000 7f54c2639640 1 -- 172.21.15.83:0/2250069938 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54bc15e010 msgr2=0x7f54bc1522b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.544+0000 7f54c2639640 1 --2- 172.21.15.83:0/2250069938 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54bc15e010 0x7f54bc1522b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f54a402b950 tx=0x7f54a40046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.544+0000 7f54c2639640 1 -- 172.21.15.83:0/2250069938 shutdown_connections 2024-03-06T07:22:39.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.544+0000 7f54c2639640 1 --2- 172.21.15.83:0/2250069938 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f549c06b500 0x7f549c06d9c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.544+0000 7f54c2639640 1 --2- 172.21.15.83:0/2250069938 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f54bc157170 0x7f54bc0b6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.544+0000 7f54c2639640 1 --2- 172.21.15.83:0/2250069938 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f54bc156cf0 0x7f54bc0b6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.544+0000 7f54c2639640 1 --2- 172.21.15.83:0/2250069938 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54bc15e010 0x7f54bc1522b0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.544+0000 7f54c2639640 1 -- 172.21.15.83:0/2250069938 >> 172.21.15.83:0/2250069938 conn(0x7f54bc09a9f0 msgr2=0x7f54bc0ae890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:39.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.544+0000 7f54c2639640 1 -- 172.21.15.83:0/2250069938 shutdown_connections 2024-03-06T07:22:39.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.544+0000 7f54c2639640 1 -- 172.21.15.83:0/2250069938 wait complete. 2024-03-06T07:22:39.556 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi183.front.sepia.ceph.com... 2024-03-06T07:22:39.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:39.557 DEBUG:teuthology.orchestra.run.smithi183:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:39.590 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:39.590 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi183.front.sepia.ceph.com 2024-03-06T07:22:39.590 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:39.590 DEBUG:teuthology.orchestra.run.smithi183:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:22:39.656 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:22:39.656 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:39.672 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:22:39.672 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:39.728 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:39.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.867+0000 7f1ad18af640 1 Processor -- start 2024-03-06T07:22:39.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.867+0000 7f1ad18af640 1 -- start start 2024-03-06T07:22:39.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.868+0000 7f1ad18af640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f1acc0d4760 0x7f1acc0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.868+0000 7f1ad18af640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1acc0d0a80 0x7f1acc0cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.868+0000 7f1ad18af640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1acc0d25d0 con 0x7f1acc0b4170 2024-03-06T07:22:39.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.868+0000 7f1ad18af640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1acc0d4bf0 con 0x7f1acc0d0a80 2024-03-06T07:22:39.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.868+0000 7f1ad18af640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1acc0ce9e0 con 0x7f1acc0d4760 2024-03-06T07:22:39.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.868+0000 7f1acb7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1acc0d0a80 0x7f1acc0cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.868+0000 7f1acb7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1acc0d0a80 0x7f1acc0cee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53668/0 (socket says 172.21.15.83:53668) 2024-03-06T07:22:39.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.868+0000 7f1acb7fe640 1 -- 172.21.15.83:0/329148624 learned_addr learned my addr 172.21.15.83:0/329148624 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:39.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.868+0000 7f1acaffd640 1 --1- 172.21.15.83:0/329148624 >> v1:172.21.15.83:6789/0 conn(0x7f1acc0b4170 0x7f1acc0d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39158/0 (socket says 172.21.15.83:39158) 2024-03-06T07:22:39.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.868+0000 7f1acb7fe640 1 -- 172.21.15.83:0/329148624 >> v2:172.21.15.183:3300/0 conn(0x7f1acc0d4760 msgr2=0x7f1acc0b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:39.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.868+0000 7f1ac3fff640 1 --2- 172.21.15.83:0/329148624 >> v2:172.21.15.183:3300/0 conn(0x7f1acc0d4760 0x7f1acc0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.868+0000 7f1acb7fe640 1 --2- 172.21.15.83:0/329148624 >> v2:172.21.15.183:3300/0 conn(0x7f1acc0d4760 0x7f1acc0b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.868+0000 7f1acb7fe640 1 -- 172.21.15.83:0/329148624 >> v1:172.21.15.83:6789/0 conn(0x7f1acc0b4170 legacy=0x7f1acc0d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.868+0000 7f1aca7fc640 1 -- 172.21.15.83:0/329148624 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 944848972 0 0) 0x7f1acc0d25d0 con 0x7f1acc0b4170 2024-03-06T07:22:39.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.868+0000 7f1acb7fe640 1 -- 172.21.15.83:0/329148624 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1acc0d2870 con 0x7f1acc0d0a80 2024-03-06T07:22:39.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.868+0000 7f1ac3fff640 1 --2- 172.21.15.83:0/329148624 >> v2:172.21.15.183:3300/0 conn(0x7f1acc0d4760 0x7f1acc0b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:39.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.869+0000 7f1acb7fe640 1 --2- 172.21.15.83:0/329148624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1acc0d0a80 0x7f1acc0cee70 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1acc067920 tx=0x7f1ac402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34f95241e52a46b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.869+0000 7f1aca7fc640 1 -- 172.21.15.83:0/329148624 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac402cb20 con 0x7f1acc0d0a80 2024-03-06T07:22:39.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.869+0000 7f1aca7fc640 1 -- 172.21.15.83:0/329148624 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ac402ccc0 con 0x7f1acc0d0a80 2024-03-06T07:22:39.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.869+0000 7f1aca7fc640 1 -- 172.21.15.83:0/329148624 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac4034610 con 0x7f1acc0d0a80 2024-03-06T07:22:39.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.870+0000 7f1ad18af640 1 -- 172.21.15.83:0/329148624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1acc0d0a80 msgr2=0x7f1acc0cee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.870+0000 7f1ad18af640 1 --2- 172.21.15.83:0/329148624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1acc0d0a80 0x7f1acc0cee70 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1acc067920 tx=0x7f1ac402bf80 comp rx=0 tx=0).stop 2024-03-06T07:22:39.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.870+0000 7f1ad18af640 1 -- 172.21.15.83:0/329148624 shutdown_connections 2024-03-06T07:22:39.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.870+0000 7f1ad18af640 1 --2- 172.21.15.83:0/329148624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1acc0d0a80 0x7f1acc0cee70 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.870+0000 7f1ad18af640 1 --2- 172.21.15.83:0/329148624 >> v2:172.21.15.183:3300/0 conn(0x7f1acc0d4760 0x7f1acc0b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.870+0000 7f1ad18af640 1 -- 172.21.15.83:0/329148624 >> 172.21.15.83:0/329148624 conn(0x7f1acc0d3c40 msgr2=0x7f1acc0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:39.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.870+0000 7f1ad18af640 1 -- 172.21.15.83:0/329148624 shutdown_connections 2024-03-06T07:22:39.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.870+0000 7f1ad18af640 1 -- 172.21.15.83:0/329148624 wait complete. 2024-03-06T07:22:39.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.871+0000 7f1ad18af640 1 Processor -- start 2024-03-06T07:22:39.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.871+0000 7f1ad18af640 1 -- start start 2024-03-06T07:22:39.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.872+0000 7f1ad18af640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1acc0b4170 0x7f1acc1521c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.872+0000 7f1ad18af640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1acc0d0a80 0x7f1acc15bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.872+0000 7f1ad18af640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1acc15b3f0 0x7f1acc0ca050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.872+0000 7f1ad18af640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1acc0b4dd0 con 0x7f1acc15b3f0 2024-03-06T07:22:39.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.872+0000 7f1ad18af640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1acc0b4550 con 0x7f1acc0b4170 2024-03-06T07:22:39.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.872+0000 7f1ad18af640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1acc0b4f50 con 0x7f1acc0d0a80 2024-03-06T07:22:39.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.872+0000 7f1acb7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1acc15b3f0 0x7f1acc0ca050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.872+0000 7f1acb7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1acc15b3f0 0x7f1acc0ca050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53676/0 (socket says 172.21.15.83:53676) 2024-03-06T07:22:39.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.872+0000 7f1acb7fe640 1 -- 172.21.15.83:0/616593990 learned_addr learned my addr 172.21.15.83:0/616593990 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:39.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.872+0000 7f1acaffd640 1 --2- 172.21.15.83:0/616593990 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1acc0b4170 0x7f1acc1521c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.872+0000 7f1ac3fff640 1 --2- 172.21.15.83:0/616593990 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1acc0d0a80 0x7f1acc15bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.872+0000 7f1acb7fe640 1 -- 172.21.15.83:0/616593990 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1acc0d0a80 msgr2=0x7f1acc15bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.872+0000 7f1acb7fe640 1 --2- 172.21.15.83:0/616593990 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1acc0d0a80 0x7f1acc15bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.872+0000 7f1acb7fe640 1 -- 172.21.15.83:0/616593990 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1acc0b4170 msgr2=0x7f1acc1521c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.872+0000 7f1acb7fe640 1 --2- 172.21.15.83:0/616593990 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1acc0b4170 0x7f1acc1521c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.872+0000 7f1acb7fe640 1 -- 172.21.15.83:0/616593990 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1acc0c8fb0 con 0x7f1acc15b3f0 2024-03-06T07:22:39.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.872+0000 7f1acaffd640 1 --2- 172.21.15.83:0/616593990 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1acc0b4170 0x7f1acc1521c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:39.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.872+0000 7f1acb7fe640 1 --2- 172.21.15.83:0/616593990 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1acc15b3f0 0x7f1acc0ca050 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4036040 tx=0x7f1ac4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.872+0000 7f1ac8ff9640 1 -- 172.21.15.83:0/616593990 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac4043030 con 0x7f1acc15b3f0 2024-03-06T07:22:39.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.872+0000 7f1ac8ff9640 1 -- 172.21.15.83:0/616593990 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ac4003990 con 0x7f1acc15b3f0 2024-03-06T07:22:39.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.872+0000 7f1ad18af640 1 -- 172.21.15.83:0/616593990 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1acc0c9290 con 0x7f1acc15b3f0 2024-03-06T07:22:39.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.872+0000 7f1ac8ff9640 1 -- 172.21.15.83:0/616593990 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac4009510 con 0x7f1acc15b3f0 2024-03-06T07:22:39.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.873+0000 7f1ad18af640 1 -- 172.21.15.83:0/616593990 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1acc0c9850 con 0x7f1acc15b3f0 2024-03-06T07:22:39.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.874+0000 7f1ad18af640 1 -- 172.21.15.83:0/616593990 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1acc04f330 con 0x7f1acc15b3f0 2024-03-06T07:22:39.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.875+0000 7f1ac8ff9640 1 -- 172.21.15.83:0/616593990 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1ac402cb30 con 0x7f1acc15b3f0 2024-03-06T07:22:39.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.875+0000 7f1ac8ff9640 1 --2- 172.21.15.83:0/616593990 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1aac06b500 0x7f1aac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.875+0000 7f1acaffd640 1 --2- 172.21.15.83:0/616593990 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1aac06b500 0x7f1aac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.875+0000 7f1ac8ff9640 1 -- 172.21.15.83:0/616593990 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5912+0+0 (secure 0 0 0) 0x7f1ac40ac960 con 0x7f1acc15b3f0 2024-03-06T07:22:39.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.875+0000 7f1acaffd640 1 --2- 172.21.15.83:0/616593990 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1aac06b500 0x7f1aac06d9c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1abc000f60 tx=0x7f1abc007c30 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:39.879+0000 7f1ac8ff9640 1 -- 172.21.15.83:0/616593990 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1ac407b7a0 con 0x7f1acc15b3f0 2024-03-06T07:22:40.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.059+0000 7f1ad18af640 1 -- 172.21.15.83:0/616593990 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1acc0733b0 con 0x7f1acc15b3f0 2024-03-06T07:22:40.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.060+0000 7f1ac8ff9640 1 -- 172.21.15.83:0/616593990 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+17006 (secure 0 0 0) 0x7f1ac407b0a0 con 0x7f1acc15b3f0 2024-03-06T07:22:40.064 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:22:40.065 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":21,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:22:39.099304+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.318942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.254795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.1f","mappings":[{"from":5,"to":1}]},{"pgid":"3.29","mappings":[{"from":3,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:40.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.062+0000 7f1ad18af640 1 -- 172.21.15.83:0/616593990 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1aac06b500 msgr2=0x7f1aac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.062+0000 7f1ad18af640 1 --2- 172.21.15.83:0/616593990 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1aac06b500 0x7f1aac06d9c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1abc000f60 tx=0x7f1abc007c30 comp rx=0 tx=0).stop 2024-03-06T07:22:40.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.062+0000 7f1ad18af640 1 -- 172.21.15.83:0/616593990 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1acc15b3f0 msgr2=0x7f1acc0ca050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.062+0000 7f1ad18af640 1 --2- 172.21.15.83:0/616593990 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1acc15b3f0 0x7f1acc0ca050 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4036040 tx=0x7f1ac4004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:40.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.062+0000 7f1ad18af640 1 -- 172.21.15.83:0/616593990 shutdown_connections 2024-03-06T07:22:40.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.062+0000 7f1ad18af640 1 --2- 172.21.15.83:0/616593990 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1aac06b500 0x7f1aac06d9c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.062+0000 7f1ad18af640 1 --2- 172.21.15.83:0/616593990 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1acc15b3f0 0x7f1acc0ca050 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.062+0000 7f1ad18af640 1 --2- 172.21.15.83:0/616593990 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1acc0d0a80 0x7f1acc15bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.062+0000 7f1ad18af640 1 --2- 172.21.15.83:0/616593990 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1acc0b4170 0x7f1acc1521c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.062+0000 7f1ad18af640 1 -- 172.21.15.83:0/616593990 >> 172.21.15.83:0/616593990 conn(0x7f1acc0d3c40 msgr2=0x7f1acc0d0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:40.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.062+0000 7f1ad18af640 1 -- 172.21.15.83:0/616593990 shutdown_connections 2024-03-06T07:22:40.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.062+0000 7f1ad18af640 1 -- 172.21.15.83:0/616593990 wait complete. 2024-03-06T07:22:40.079 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:56.318942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:57.254795+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:40.079 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:40.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.206+0000 7f090f69a640 1 Processor -- start 2024-03-06T07:22:40.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.206+0000 7f090f69a640 1 -- start start 2024-03-06T07:22:40.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.207+0000 7f090f69a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f09081455d0 0x7f0908072480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.207+0000 7f090f69a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f090806afb0 0x7f090806b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.207+0000 7f090f69a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f090806ded0 0x7f090806e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.207+0000 7f090f69a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09080cfdf0 con 0x7f090806ded0 2024-03-06T07:22:40.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.207+0000 7f090f69a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f090806b930 con 0x7f090806afb0 2024-03-06T07:22:40.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.207+0000 7f090f69a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09081459b0 con 0x7f09081455d0 2024-03-06T07:22:40.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.207+0000 7f090dc37640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f090806ded0 0x7f090806e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.207+0000 7f090dc37640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f090806ded0 0x7f090806e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53686/0 (socket says 172.21.15.83:53686) 2024-03-06T07:22:40.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.207+0000 7f090dc37640 1 -- 172.21.15.83:0/1940001316 learned_addr learned my addr 172.21.15.83:0/1940001316 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:40.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.208+0000 7f090d436640 1 --2- 172.21.15.83:0/1940001316 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f09081455d0 0x7f0908072480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.208+0000 7f090dc37640 1 -- 172.21.15.83:0/1940001316 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f09081455d0 msgr2=0x7f0908072480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:40.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.208+0000 7f090dc37640 1 --2- 172.21.15.83:0/1940001316 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f09081455d0 0x7f0908072480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.208+0000 7f090dc37640 1 -- 172.21.15.83:0/1940001316 >> v2:172.21.15.183:3300/0 conn(0x7f090806afb0 msgr2=0x7f090806b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:40.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.208+0000 7f090cc35640 1 --2- 172.21.15.83:0/1940001316 >> v2:172.21.15.183:3300/0 conn(0x7f090806afb0 0x7f090806b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.208+0000 7f090dc37640 1 --2- 172.21.15.83:0/1940001316 >> v2:172.21.15.183:3300/0 conn(0x7f090806afb0 0x7f090806b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.208+0000 7f090dc37640 1 -- 172.21.15.83:0/1940001316 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09080cf1e0 con 0x7f090806ded0 2024-03-06T07:22:40.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.208+0000 7f090cc35640 1 --2- 172.21.15.83:0/1940001316 >> v2:172.21.15.183:3300/0 conn(0x7f090806afb0 0x7f090806b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:40.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.209+0000 7f090dc37640 1 --2- 172.21.15.83:0/1940001316 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f090806ded0 0x7f090806e360 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f090806d080 tx=0x7f090002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26e97e88bb3f182b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.209+0000 7f08fffff640 1 -- 172.21.15.83:0/1940001316 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f090002ca70 con 0x7f090806ded0 2024-03-06T07:22:40.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.209+0000 7f08fffff640 1 -- 172.21.15.83:0/1940001316 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f090002cc10 con 0x7f090806ded0 2024-03-06T07:22:40.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.209+0000 7f08fffff640 1 -- 172.21.15.83:0/1940001316 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09000343f0 con 0x7f090806ded0 2024-03-06T07:22:40.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.210+0000 7f090f69a640 1 -- 172.21.15.83:0/1940001316 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f090806ded0 msgr2=0x7f090806e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.210+0000 7f090f69a640 1 --2- 172.21.15.83:0/1940001316 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f090806ded0 0x7f090806e360 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f090806d080 tx=0x7f090002bed0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.210+0000 7f090f69a640 1 -- 172.21.15.83:0/1940001316 shutdown_connections 2024-03-06T07:22:40.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.210+0000 7f090f69a640 1 --2- 172.21.15.83:0/1940001316 >> v2:172.21.15.183:3300/0 conn(0x7f090806afb0 0x7f090806b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.210+0000 7f090f69a640 1 --2- 172.21.15.83:0/1940001316 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f090806ded0 0x7f090806e360 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.210+0000 7f090f69a640 1 --2- 172.21.15.83:0/1940001316 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f09081455d0 0x7f0908072480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.210+0000 7f090f69a640 1 -- 172.21.15.83:0/1940001316 >> 172.21.15.83:0/1940001316 conn(0x7f09080b2660 msgr2=0x7f09080b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:40.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.210+0000 7f090f69a640 1 -- 172.21.15.83:0/1940001316 shutdown_connections 2024-03-06T07:22:40.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.210+0000 7f090f69a640 1 -- 172.21.15.83:0/1940001316 wait complete. 2024-03-06T07:22:40.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.212+0000 7f090f69a640 1 Processor -- start 2024-03-06T07:22:40.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.212+0000 7f090f69a640 1 -- start start 2024-03-06T07:22:40.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.213+0000 7f090f69a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f090806afb0 0x7f090813be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.213+0000 7f090f69a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f090806ded0 0x7f09080b53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.213+0000 7f090f69a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f09081455d0 0x7f09080aecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.213+0000 7f090f69a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09080cfa80 con 0x7f090806ded0 2024-03-06T07:22:40.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.213+0000 7f090f69a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09080cf7a0 con 0x7f090806afb0 2024-03-06T07:22:40.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.213+0000 7f090f69a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09080729c0 con 0x7f09081455d0 2024-03-06T07:22:40.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.213+0000 7f090cc35640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f090806ded0 0x7f09080b53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.213+0000 7f090cc35640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f090806ded0 0x7f09080b53f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53702/0 (socket says 172.21.15.83:53702) 2024-03-06T07:22:40.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.213+0000 7f090cc35640 1 -- 172.21.15.83:0/3953201959 learned_addr learned my addr 172.21.15.83:0/3953201959 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:40.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.213+0000 7f090dc37640 1 --2- 172.21.15.83:0/3953201959 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f09081455d0 0x7f09080aecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.213+0000 7f090d436640 1 --2- 172.21.15.83:0/3953201959 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f090806afb0 0x7f090813be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.213+0000 7f090cc35640 1 -- 172.21.15.83:0/3953201959 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f09081455d0 msgr2=0x7f09080aecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.213+0000 7f090cc35640 1 --2- 172.21.15.83:0/3953201959 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f09081455d0 0x7f09080aecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.213+0000 7f090cc35640 1 -- 172.21.15.83:0/3953201959 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f090806afb0 msgr2=0x7f090813be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.213+0000 7f090cc35640 1 --2- 172.21.15.83:0/3953201959 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f090806afb0 0x7f090813be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.213+0000 7f090cc35640 1 -- 172.21.15.83:0/3953201959 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f090815bd50 con 0x7f090806ded0 2024-03-06T07:22:40.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.213+0000 7f090d436640 1 --2- 172.21.15.83:0/3953201959 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f090806afb0 0x7f090813be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:40.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.214+0000 7f090cc35640 1 --2- 172.21.15.83:0/3953201959 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f090806ded0 0x7f09080b53f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f08f800c820 tx=0x7f08f800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.214+0000 7f08fdffb640 1 -- 172.21.15.83:0/3953201959 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f8015070 con 0x7f090806ded0 2024-03-06T07:22:40.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.214+0000 7f08fdffb640 1 -- 172.21.15.83:0/3953201959 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08f80079d0 con 0x7f090806ded0 2024-03-06T07:22:40.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.214+0000 7f08fdffb640 1 -- 172.21.15.83:0/3953201959 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f8007dc0 con 0x7f090806ded0 2024-03-06T07:22:40.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.214+0000 7f090f69a640 1 -- 172.21.15.83:0/3953201959 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09080adc20 con 0x7f090806ded0 2024-03-06T07:22:40.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.214+0000 7f090f69a640 1 -- 172.21.15.83:0/3953201959 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09080ae1e0 con 0x7f090806ded0 2024-03-06T07:22:40.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.215+0000 7f090f69a640 1 -- 172.21.15.83:0/3953201959 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09080703f0 con 0x7f090806ded0 2024-03-06T07:22:40.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.215+0000 7f08fdffb640 1 -- 172.21.15.83:0/3953201959 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f08f8018070 con 0x7f090806ded0 2024-03-06T07:22:40.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.216+0000 7f08fdffb640 1 --2- 172.21.15.83:0/3953201959 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f08ec06b4d0 0x7f08ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.216+0000 7f08fdffb640 1 -- 172.21.15.83:0/3953201959 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5912+0+0 (secure 0 0 0) 0x7f08f808a140 con 0x7f090806ded0 2024-03-06T07:22:40.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.216+0000 7f090d436640 1 --2- 172.21.15.83:0/3953201959 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f08ec06b4d0 0x7f08ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.216+0000 7f090d436640 1 --2- 172.21.15.83:0/3953201959 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f08ec06b4d0 0x7f08ec06d990 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0904008890 tx=0x7f0904008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.224+0000 7f08fdffb640 1 -- 172.21.15.83:0/3953201959 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f09080703f0 con 0x7f090806ded0 2024-03-06T07:22:40.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.423+0000 7f090f69a640 1 -- 172.21.15.83:0/3953201959 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f09080b6c30 con 0x7f090806ded0 2024-03-06T07:22:40.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.424+0000 7f08fdffb640 1 -- 172.21.15.83:0/3953201959 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f09080b6c30 con 0x7f090806ded0 2024-03-06T07:22:40.426 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:22:40.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.426+0000 7f090f69a640 1 -- 172.21.15.83:0/3953201959 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f08ec06b4d0 msgr2=0x7f08ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.426+0000 7f090f69a640 1 --2- 172.21.15.83:0/3953201959 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f08ec06b4d0 0x7f08ec06d990 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0904008890 tx=0x7f0904008040 comp rx=0 tx=0).stop 2024-03-06T07:22:40.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.426+0000 7f090f69a640 1 -- 172.21.15.83:0/3953201959 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f090806ded0 msgr2=0x7f09080b53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.426+0000 7f090f69a640 1 --2- 172.21.15.83:0/3953201959 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f090806ded0 0x7f09080b53f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f08f800c820 tx=0x7f08f800cce0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.426+0000 7f090f69a640 1 -- 172.21.15.83:0/3953201959 shutdown_connections 2024-03-06T07:22:40.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.426+0000 7f090f69a640 1 --2- 172.21.15.83:0/3953201959 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f08ec06b4d0 0x7f08ec06d990 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.426+0000 7f090f69a640 1 --2- 172.21.15.83:0/3953201959 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f09081455d0 0x7f09080aecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.426+0000 7f090f69a640 1 --2- 172.21.15.83:0/3953201959 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f090806ded0 0x7f09080b53f0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.426+0000 7f090f69a640 1 --2- 172.21.15.83:0/3953201959 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f090806afb0 0x7f090813be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.426+0000 7f090f69a640 1 -- 172.21.15.83:0/3953201959 >> 172.21.15.83:0/3953201959 conn(0x7f09080b2660 msgr2=0x7f090806f460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:40.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.426+0000 7f090f69a640 1 -- 172.21.15.83:0/3953201959 shutdown_connections 2024-03-06T07:22:40.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.426+0000 7f090f69a640 1 -- 172.21.15.83:0/3953201959 wait complete. 2024-03-06T07:22:40.438 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:40.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.540+0000 7f74dbcc9640 1 Processor -- start 2024-03-06T07:22:40.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.540+0000 7f74dbcc9640 1 -- start start 2024-03-06T07:22:40.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.541+0000 7f74dbcc9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f74d415c050 0x7f74d406f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.541+0000 7f74dbcc9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f74d406fb40 0x7f74d40cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.541+0000 7f74dbcc9640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74d40cd4b0 con 0x7f74d40ccca0 2024-03-06T07:22:40.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.541+0000 7f74dbcc9640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74d406ff80 con 0x7f74d406fb40 2024-03-06T07:22:40.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.541+0000 7f74dbcc9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74d40cd790 con 0x7f74d415c050 2024-03-06T07:22:40.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.541+0000 7f74d9264640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f74d406fb40 0x7f74d40cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.541+0000 7f74da266640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f74d40ccca0 0x7f74d40cfae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39168/0 (socket says 172.21.15.83:39168) 2024-03-06T07:22:40.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.541+0000 7f74da266640 1 -- 172.21.15.83:0/2498571881 learned_addr learned my addr 172.21.15.83:0/2498571881 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:40.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.541+0000 7f74d9a65640 1 --2- 172.21.15.83:0/2498571881 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f74d415c050 0x7f74d406f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.541+0000 7f74d9264640 1 -- 172.21.15.83:0/2498571881 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f74d415c050 msgr2=0x7f74d406f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:40.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.541+0000 7f74d9264640 1 --2- 172.21.15.83:0/2498571881 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f74d415c050 0x7f74d406f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.541+0000 7f74d9264640 1 -- 172.21.15.83:0/2498571881 >> v1:172.21.15.83:6789/0 conn(0x7f74d40ccca0 legacy=0x7f74d40cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.541+0000 7f74d9264640 1 -- 172.21.15.83:0/2498571881 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74d40cda70 con 0x7f74d406fb40 2024-03-06T07:22:40.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.541+0000 7f74d8a63640 1 -- 172.21.15.83:0/2498571881 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1131785349 0 0) 0x7f74d40cd4b0 con 0x7f74d40ccca0 2024-03-06T07:22:40.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.542+0000 7f74d9264640 1 --2- 172.21.15.83:0/2498571881 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74d406fb40 0x7f74d40cc760 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f74c40097a0 tx=0x7f74c402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6013337a98e63bb4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.542+0000 7f74d8a63640 1 -- 172.21.15.83:0/2498571881 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c40315a0 con 0x7f74d406fb40 2024-03-06T07:22:40.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.542+0000 7f74d8a63640 1 -- 172.21.15.83:0/2498571881 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74c4037070 con 0x7f74d406fb40 2024-03-06T07:22:40.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.542+0000 7f74d8a63640 1 -- 172.21.15.83:0/2498571881 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c4031960 con 0x7f74d406fb40 2024-03-06T07:22:40.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.543+0000 7f74dbcc9640 1 -- 172.21.15.83:0/2498571881 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74d406fb40 msgr2=0x7f74d40cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.543+0000 7f74dbcc9640 1 --2- 172.21.15.83:0/2498571881 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74d406fb40 0x7f74d40cc760 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f74c40097a0 tx=0x7f74c402b840 comp rx=0 tx=0).stop 2024-03-06T07:22:40.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.543+0000 7f74dbcc9640 1 -- 172.21.15.83:0/2498571881 shutdown_connections 2024-03-06T07:22:40.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.543+0000 7f74dbcc9640 1 --2- 172.21.15.83:0/2498571881 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74d406fb40 0x7f74d40cc760 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.543+0000 7f74dbcc9640 1 --2- 172.21.15.83:0/2498571881 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f74d415c050 0x7f74d406f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.543+0000 7f74dbcc9640 1 -- 172.21.15.83:0/2498571881 >> 172.21.15.83:0/2498571881 conn(0x7f74d40b3800 msgr2=0x7f74d40b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:40.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.543+0000 7f74dbcc9640 1 -- 172.21.15.83:0/2498571881 shutdown_connections 2024-03-06T07:22:40.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.543+0000 7f74dbcc9640 1 -- 172.21.15.83:0/2498571881 wait complete. 2024-03-06T07:22:40.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.544+0000 7f74dbcc9640 1 Processor -- start 2024-03-06T07:22:40.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.544+0000 7f74dbcc9640 1 -- start start 2024-03-06T07:22:40.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.545+0000 7f74dbcc9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74d406fb40 0x7f74d409d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.545+0000 7f74dbcc9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f74d40ccca0 0x7f74d409bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.545+0000 7f74dbcc9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f74d415c050 0x7f74d409f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.545+0000 7f74dbcc9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d40d01f0 con 0x7f74d406fb40 2024-03-06T07:22:40.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.545+0000 7f74dbcc9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d40cd130 con 0x7f74d40ccca0 2024-03-06T07:22:40.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.545+0000 7f74dbcc9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74d40d0370 con 0x7f74d415c050 2024-03-06T07:22:40.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.545+0000 7f74d9a65640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74d406fb40 0x7f74d409d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.545+0000 7f74d9a65640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74d406fb40 0x7f74d409d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53722/0 (socket says 172.21.15.83:53722) 2024-03-06T07:22:40.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.545+0000 7f74d9a65640 1 -- 172.21.15.83:0/1420380704 learned_addr learned my addr 172.21.15.83:0/1420380704 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:40.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.545+0000 7f74d9a65640 1 -- 172.21.15.83:0/1420380704 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f74d415c050 msgr2=0x7f74d409f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.545+0000 7f74d9a65640 1 --2- 172.21.15.83:0/1420380704 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f74d415c050 0x7f74d409f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.545+0000 7f74d9a65640 1 -- 172.21.15.83:0/1420380704 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f74d40ccca0 msgr2=0x7f74d409bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.545+0000 7f74d9a65640 1 --2- 172.21.15.83:0/1420380704 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f74d40ccca0 0x7f74d409bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.545+0000 7f74d9a65640 1 -- 172.21.15.83:0/1420380704 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74d409df70 con 0x7f74d406fb40 2024-03-06T07:22:40.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.545+0000 7f74d9a65640 1 --2- 172.21.15.83:0/1420380704 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74d406fb40 0x7f74d409d890 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f74d000c7f0 tx=0x7f74d000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.545+0000 7f74ca7fc640 1 -- 172.21.15.83:0/1420380704 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d0015070 con 0x7f74d406fb40 2024-03-06T07:22:40.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.545+0000 7f74ca7fc640 1 -- 172.21.15.83:0/1420380704 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74d0004240 con 0x7f74d406fb40 2024-03-06T07:22:40.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.545+0000 7f74dbcc9640 1 -- 172.21.15.83:0/1420380704 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74d409e250 con 0x7f74d406fb40 2024-03-06T07:22:40.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.545+0000 7f74ca7fc640 1 -- 172.21.15.83:0/1420380704 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d0007620 con 0x7f74d406fb40 2024-03-06T07:22:40.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.545+0000 7f74dbcc9640 1 -- 172.21.15.83:0/1420380704 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74d409e810 con 0x7f74d406fb40 2024-03-06T07:22:40.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.546+0000 7f74dbcc9640 1 -- 172.21.15.83:0/1420380704 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74d409fe60 con 0x7f74d406fb40 2024-03-06T07:22:40.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.547+0000 7f74ca7fc640 1 -- 172.21.15.83:0/1420380704 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f74d00079d0 con 0x7f74d406fb40 2024-03-06T07:22:40.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.547+0000 7f74ca7fc640 1 --2- 172.21.15.83:0/1420380704 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f74b806b4d0 0x7f74b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.547+0000 7f74d9264640 1 --2- 172.21.15.83:0/1420380704 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f74b806b4d0 0x7f74b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.547+0000 7f74ca7fc640 1 -- 172.21.15.83:0/1420380704 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5912+0+0 (secure 0 0 0) 0x7f74d008a6c0 con 0x7f74d406fb40 2024-03-06T07:22:40.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.548+0000 7f74d9264640 1 --2- 172.21.15.83:0/1420380704 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f74b806b4d0 0x7f74b806d990 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f74c4036000 tx=0x7f74c4030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.552+0000 7f74ca7fc640 1 -- 172.21.15.83:0/1420380704 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f74d0059500 con 0x7f74d406fb40 2024-03-06T07:22:40.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.728+0000 7f74dbcc9640 1 -- 172.21.15.83:0/1420380704 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f74d4070900 con 0x7f74d406fb40 2024-03-06T07:22:40.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.728+0000 7f74ca7fc640 1 -- 172.21.15.83:0/1420380704 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f74d0058e00 con 0x7f74d406fb40 2024-03-06T07:22:40.731 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-03-06T07:22:40.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.731+0000 7f74dbcc9640 1 -- 172.21.15.83:0/1420380704 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f74b806b4d0 msgr2=0x7f74b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.731+0000 7f74dbcc9640 1 --2- 172.21.15.83:0/1420380704 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f74b806b4d0 0x7f74b806d990 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f74c4036000 tx=0x7f74c4030000 comp rx=0 tx=0).stop 2024-03-06T07:22:40.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.731+0000 7f74dbcc9640 1 -- 172.21.15.83:0/1420380704 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74d406fb40 msgr2=0x7f74d409d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.731+0000 7f74dbcc9640 1 --2- 172.21.15.83:0/1420380704 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74d406fb40 0x7f74d409d890 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f74d000c7f0 tx=0x7f74d000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.731+0000 7f74dbcc9640 1 -- 172.21.15.83:0/1420380704 shutdown_connections 2024-03-06T07:22:40.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.731+0000 7f74dbcc9640 1 --2- 172.21.15.83:0/1420380704 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f74b806b4d0 0x7f74b806d990 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.731+0000 7f74dbcc9640 1 --2- 172.21.15.83:0/1420380704 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f74d415c050 0x7f74d409f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.731+0000 7f74dbcc9640 1 --2- 172.21.15.83:0/1420380704 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f74d40ccca0 0x7f74d409bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.731+0000 7f74dbcc9640 1 --2- 172.21.15.83:0/1420380704 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74d406fb40 0x7f74d409d890 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.731+0000 7f74dbcc9640 1 -- 172.21.15.83:0/1420380704 >> 172.21.15.83:0/1420380704 conn(0x7f74d40b3800 msgr2=0x7f74d40b0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:40.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.731+0000 7f74dbcc9640 1 -- 172.21.15.83:0/1420380704 shutdown_connections 2024-03-06T07:22:40.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.731+0000 7f74dbcc9640 1 -- 172.21.15.83:0/1420380704 wait complete. 2024-03-06T07:22:40.743 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:40.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.845+0000 7f482510d640 1 Processor -- start 2024-03-06T07:22:40.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.845+0000 7f482510d640 1 -- start start 2024-03-06T07:22:40.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.846+0000 7f482510d640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f482015f4f0 0x7f4820066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.846+0000 7f482510d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f482006b050 con 0x7f48200ce750 2024-03-06T07:22:40.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.846+0000 7f482510d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f482006b2c0 con 0x7f48200b0db0 2024-03-06T07:22:40.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.846+0000 7f482510d640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48200b0890 con 0x7f482015f4f0 2024-03-06T07:22:40.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.846+0000 7f481f59e640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f48200ce750 0x7f48200b4ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39180/0 (socket says 172.21.15.83:39180) 2024-03-06T07:22:40.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.846+0000 7f481f59e640 1 -- 172.21.15.83:0/4233219648 learned_addr learned my addr 172.21.15.83:0/4233219648 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:40.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.847+0000 7f481ed9d640 1 --2- 172.21.15.83:0/4233219648 >> v2:172.21.15.183:3300/0 conn(0x7f482015f4f0 0x7f4820066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.847+0000 7f481dd9b640 1 -- 172.21.15.83:0/4233219648 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1652427073 0 0) 0x7f482006b050 con 0x7f48200ce750 2024-03-06T07:22:40.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.847+0000 7f481dd9b640 1 -- 172.21.15.83:0/4233219648 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4808003520 con 0x7f48200ce750 2024-03-06T07:22:40.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.847+0000 7f481dd9b640 1 -- 172.21.15.83:0/4233219648 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 90023909 0 0) 0x7f4808003520 con 0x7f48200ce750 2024-03-06T07:22:40.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.847+0000 7f481dd9b640 1 -- 172.21.15.83:0/4233219648 >> v2:172.21.15.183:3300/0 conn(0x7f482015f4f0 msgr2=0x7f4820066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:40.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.847+0000 7f481dd9b640 1 --2- 172.21.15.83:0/4233219648 >> v2:172.21.15.183:3300/0 conn(0x7f482015f4f0 0x7f4820066d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.847+0000 7f481dd9b640 1 -- 172.21.15.83:0/4233219648 >> v1:172.21.15.183:6789/0 conn(0x7f48200b0db0 legacy=0x7f48200ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.847+0000 7f481dd9b640 1 -- 172.21.15.83:0/4233219648 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48200672e0 con 0x7f48200ce750 2024-03-06T07:22:40.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.847+0000 7f481dd9b640 1 -- 172.21.15.83:0/4233219648 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f48100019f0 con 0x7f48200ce750 2024-03-06T07:22:40.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.847+0000 7f481dd9b640 1 -- 172.21.15.83:0/4233219648 >> v1:172.21.15.83:6789/0 conn(0x7f48200ce750 legacy=0x7f48200b4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.847+0000 7f481dd9b640 1 --2- 172.21.15.83:0/4233219648 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f48080039b0 0x7f4808003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.847+0000 7f481dd9b640 1 --2- 172.21.15.83:0/4233219648 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4808004800 0x7f4808004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.847+0000 7f481dd9b640 1 --2- 172.21.15.83:0/4233219648 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f48080051c0 0x7f480800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.847+0000 7f481dd9b640 1 -- 172.21.15.83:0/4233219648 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48200b0890 con 0x7f48080051c0 2024-03-06T07:22:40.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.847+0000 7f481dd9b640 1 -- 172.21.15.83:0/4233219648 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f480800c060 con 0x7f48080039b0 2024-03-06T07:22:40.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.847+0000 7f481dd9b640 1 -- 172.21.15.83:0/4233219648 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f480800c260 con 0x7f4808004800 2024-03-06T07:22:40.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.847+0000 7f481f59e640 1 --2- 172.21.15.83:0/4233219648 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f48080051c0 0x7f480800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.847+0000 7f481ed9d640 1 --2- 172.21.15.83:0/4233219648 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f48080039b0 0x7f4808003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.847+0000 7f481e59c640 1 --2- 172.21.15.83:0/4233219648 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4808004800 0x7f4808004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.847+0000 7f481f59e640 1 -- 172.21.15.83:0/4233219648 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4808004800 msgr2=0x7f4808004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:40.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.847+0000 7f481f59e640 1 --2- 172.21.15.83:0/4233219648 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4808004800 0x7f4808004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.847+0000 7f481f59e640 1 -- 172.21.15.83:0/4233219648 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f48080039b0 msgr2=0x7f4808003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:40.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.847+0000 7f481f59e640 1 --2- 172.21.15.83:0/4233219648 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f48080039b0 0x7f4808003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.847+0000 7f481f59e640 1 -- 172.21.15.83:0/4233219648 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4808003520 con 0x7f48080051c0 2024-03-06T07:22:40.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.847+0000 7f481e59c640 1 --2- 172.21.15.83:0/4233219648 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4808004800 0x7f4808004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:40.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.848+0000 7f481f59e640 1 -- 172.21.15.83:0/4233219648 reap_dead start 2024-03-06T07:22:40.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.848+0000 7f481f59e640 1 --2- 172.21.15.83:0/4233219648 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f48080051c0 0x7f480800b9c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f481000a850 tx=0x7f4808005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=444de87b9c7db2dc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.848+0000 7f481dd9b640 1 -- 172.21.15.83:0/4233219648 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f481000a4a0 con 0x7f48080051c0 2024-03-06T07:22:40.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.848+0000 7f481dd9b640 1 -- 172.21.15.83:0/4233219648 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4810034650 con 0x7f48080051c0 2024-03-06T07:22:40.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.848+0000 7f481dd9b640 1 -- 172.21.15.83:0/4233219648 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4810034a40 con 0x7f48080051c0 2024-03-06T07:22:40.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.849+0000 7f482510d640 1 -- 172.21.15.83:0/4233219648 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f48080051c0 msgr2=0x7f480800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.849+0000 7f482510d640 1 --2- 172.21.15.83:0/4233219648 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f48080051c0 0x7f480800b9c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f481000a850 tx=0x7f4808005830 comp rx=0 tx=0).stop 2024-03-06T07:22:40.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.849+0000 7f482510d640 1 -- 172.21.15.83:0/4233219648 shutdown_connections 2024-03-06T07:22:40.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.849+0000 7f482510d640 1 --2- 172.21.15.83:0/4233219648 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f48080051c0 0x7f480800b9c0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.849+0000 7f482510d640 1 -- 172.21.15.83:0/4233219648 >> 172.21.15.83:0/4233219648 conn(0x7f48200b1d70 msgr2=0x7f48200b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:40.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.849+0000 7f482510d640 1 -- 172.21.15.83:0/4233219648 shutdown_connections 2024-03-06T07:22:40.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.849+0000 7f482510d640 1 -- 172.21.15.83:0/4233219648 wait complete. 2024-03-06T07:22:40.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.850+0000 7f482510d640 1 Processor -- start 2024-03-06T07:22:40.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.850+0000 7f482510d640 1 -- start start 2024-03-06T07:22:40.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.851+0000 7f482510d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f48080051c0 0x7f4820149600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.851+0000 7f482510d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f48200c99e0 0x7f482006e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.851+0000 7f482510d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f48200c9e40 0x7f482006ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.851+0000 7f482510d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f482015f8d0 con 0x7f48200c9e40 2024-03-06T07:22:40.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.851+0000 7f482510d640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f482006b5a0 con 0x7f48080051c0 2024-03-06T07:22:40.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.851+0000 7f482510d640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4820155580 con 0x7f48200c99e0 2024-03-06T07:22:40.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.851+0000 7f481f59e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f48200c9e40 0x7f482006ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.851+0000 7f481f59e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f48200c9e40 0x7f482006ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53736/0 (socket says 172.21.15.83:53736) 2024-03-06T07:22:40.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.851+0000 7f481f59e640 1 -- 172.21.15.83:0/4208423685 learned_addr learned my addr 172.21.15.83:0/4208423685 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:40.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.851+0000 7f481ed9d640 1 --2- 172.21.15.83:0/4208423685 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f48080051c0 0x7f4820149600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.851+0000 7f481e59c640 1 --2- 172.21.15.83:0/4208423685 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f48200c99e0 0x7f482006e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.851+0000 7f481f59e640 1 -- 172.21.15.83:0/4208423685 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f48200c99e0 msgr2=0x7f482006e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.851+0000 7f481f59e640 1 --2- 172.21.15.83:0/4208423685 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f48200c99e0 0x7f482006e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.851+0000 7f481f59e640 1 -- 172.21.15.83:0/4208423685 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f48080051c0 msgr2=0x7f4820149600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.851+0000 7f481f59e640 1 --2- 172.21.15.83:0/4208423685 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f48080051c0 0x7f4820149600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.851+0000 7f481f59e640 1 -- 172.21.15.83:0/4208423685 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4820155860 con 0x7f48200c9e40 2024-03-06T07:22:40.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.851+0000 7f481e59c640 1 --2- 172.21.15.83:0/4208423685 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f48200c99e0 0x7f482006e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:40.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.852+0000 7f481f59e640 1 --2- 172.21.15.83:0/4208423685 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f48200c9e40 0x7f482006ec40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f4810002230 tx=0x7f4810001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.852+0000 7f48077fe640 1 -- 172.21.15.83:0/4208423685 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4810039040 con 0x7f48200c9e40 2024-03-06T07:22:40.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.852+0000 7f48077fe640 1 -- 172.21.15.83:0/4208423685 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4810043030 con 0x7f48200c9e40 2024-03-06T07:22:40.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.852+0000 7f48077fe640 1 -- 172.21.15.83:0/4208423685 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48100017b0 con 0x7f48200c9e40 2024-03-06T07:22:40.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.852+0000 7f482510d640 1 -- 172.21.15.83:0/4208423685 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4820155b40 con 0x7f48200c9e40 2024-03-06T07:22:40.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.852+0000 7f482510d640 1 -- 172.21.15.83:0/4208423685 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48200c74f0 con 0x7f48200c9e40 2024-03-06T07:22:40.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.853+0000 7f482510d640 1 -- 172.21.15.83:0/4208423685 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f482009e3d0 con 0x7f48200c9e40 2024-03-06T07:22:40.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.854+0000 7f48077fe640 1 -- 172.21.15.83:0/4208423685 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4810001950 con 0x7f48200c9e40 2024-03-06T07:22:40.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.854+0000 7f48077fe640 1 --2- 172.21.15.83:0/4208423685 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f480c06b500 0x7f480c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.854+0000 7f48077fe640 1 -- 172.21.15.83:0/4208423685 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5912+0+0 (secure 0 0 0) 0x7f48100b0050 con 0x7f48200c9e40 2024-03-06T07:22:40.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.854+0000 7f481ed9d640 1 --2- 172.21.15.83:0/4208423685 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f480c06b500 0x7f480c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.855+0000 7f481ed9d640 1 --2- 172.21.15.83:0/4208423685 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f480c06b500 0x7f480c06d9c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4818009a80 tx=0x7f4818008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:40.858+0000 7f48077fe640 1 -- 172.21.15.83:0/4208423685 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f481007e720 con 0x7f48200c9e40 2024-03-06T07:22:41.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.031+0000 7f482510d640 1 -- 172.21.15.83:0/4208423685 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f48200c76c0 con 0x7f48200c9e40 2024-03-06T07:22:41.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.032+0000 7f48077fe640 1 -- 172.21.15.83:0/4208423685 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f48100827c0 con 0x7f48200c9e40 2024-03-06T07:22:41.034 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 64 2024-03-06T07:22:41.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.033+0000 7f482510d640 1 -- 172.21.15.83:0/4208423685 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f480c06b500 msgr2=0x7f480c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.033+0000 7f482510d640 1 --2- 172.21.15.83:0/4208423685 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f480c06b500 0x7f480c06d9c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4818009a80 tx=0x7f4818008040 comp rx=0 tx=0).stop 2024-03-06T07:22:41.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.033+0000 7f482510d640 1 -- 172.21.15.83:0/4208423685 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f48200c9e40 msgr2=0x7f482006ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.033+0000 7f482510d640 1 --2- 172.21.15.83:0/4208423685 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f48200c9e40 0x7f482006ec40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f4810002230 tx=0x7f4810001dc0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.034+0000 7f482510d640 1 -- 172.21.15.83:0/4208423685 shutdown_connections 2024-03-06T07:22:41.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.034+0000 7f482510d640 1 --2- 172.21.15.83:0/4208423685 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f480c06b500 0x7f480c06d9c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.034+0000 7f482510d640 1 --2- 172.21.15.83:0/4208423685 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f48200c9e40 0x7f482006ec40 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.034+0000 7f482510d640 1 --2- 172.21.15.83:0/4208423685 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f48200c99e0 0x7f482006e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.034+0000 7f482510d640 1 --2- 172.21.15.83:0/4208423685 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f48080051c0 0x7f4820149600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.034+0000 7f482510d640 1 -- 172.21.15.83:0/4208423685 >> 172.21.15.83:0/4208423685 conn(0x7f48200b1d70 msgr2=0x7f48200b2150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:41.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.034+0000 7f482510d640 1 -- 172.21.15.83:0/4208423685 shutdown_connections 2024-03-06T07:22:41.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.034+0000 7f482510d640 1 -- 172.21.15.83:0/4208423685 wait complete. 2024-03-06T07:22:41.050 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:22:41.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.150+0000 7f2f722fb640 1 Processor -- start 2024-03-06T07:22:41.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.150+0000 7f2f722fb640 1 -- start start 2024-03-06T07:22:41.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.151+0000 7f2f722fb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2f6c15e000 0x7f2f6c0d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.151+0000 7f2f722fb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f6c0d25e0 con 0x7f2f6c0cf7d0 2024-03-06T07:22:41.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.151+0000 7f2f722fb640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f6c0d2880 con 0x7f2f6c0d4810 2024-03-06T07:22:41.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.151+0000 7f2f722fb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f6c0c7c50 con 0x7f2f6c15e000 2024-03-06T07:22:41.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.151+0000 7f2f6bfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2f6c15e000 0x7f2f6c0d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.151+0000 7f2f6bfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2f6c15e000 0x7f2f6c0d0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53748/0 (socket says 172.21.15.83:53748) 2024-03-06T07:22:41.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.151+0000 7f2f6bfff640 1 -- 172.21.15.83:0/3789862850 learned_addr learned my addr 172.21.15.83:0/3789862850 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:41.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.151+0000 7f2f70898640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f2f6c0cf7d0 0x7f2f6c0c87f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39182/0 (socket says 172.21.15.83:39182) 2024-03-06T07:22:41.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.152+0000 7f2f6bfff640 1 -- 172.21.15.83:0/3789862850 >> v1:172.21.15.183:6789/0 conn(0x7f2f6c0d4810 legacy=0x7f2f6c0c80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:41.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.152+0000 7f2f6bfff640 1 -- 172.21.15.83:0/3789862850 >> v1:172.21.15.83:6789/0 conn(0x7f2f6c0cf7d0 legacy=0x7f2f6c0c87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.152+0000 7f2f6bfff640 1 -- 172.21.15.83:0/3789862850 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f6c0d2b40 con 0x7f2f6c15e000 2024-03-06T07:22:41.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.152+0000 7f2f6affd640 1 -- 172.21.15.83:0/3789862850 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 298061378 0 0) 0x7f2f6c0d25e0 con 0x7f2f6c0cf7d0 2024-03-06T07:22:41.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.152+0000 7f2f6bfff640 1 --2- 172.21.15.83:0/3789862850 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2f6c15e000 0x7f2f6c0d0680 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c009870 tx=0x7f2f5c02ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4abf02d359d7c2b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.153+0000 7f2f6affd640 1 -- 172.21.15.83:0/3789862850 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5c02f840 con 0x7f2f6c15e000 2024-03-06T07:22:41.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.153+0000 7f2f6affd640 1 -- 172.21.15.83:0/3789862850 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f5c02f9e0 con 0x7f2f6c15e000 2024-03-06T07:22:41.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.153+0000 7f2f6affd640 1 -- 172.21.15.83:0/3789862850 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5c02fdb0 con 0x7f2f6c15e000 2024-03-06T07:22:41.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.153+0000 7f2f722fb640 1 -- 172.21.15.83:0/3789862850 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2f6c15e000 msgr2=0x7f2f6c0d0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.153+0000 7f2f722fb640 1 --2- 172.21.15.83:0/3789862850 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2f6c15e000 0x7f2f6c0d0680 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c009870 tx=0x7f2f5c02ba20 comp rx=0 tx=0).stop 2024-03-06T07:22:41.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.153+0000 7f2f722fb640 1 -- 172.21.15.83:0/3789862850 shutdown_connections 2024-03-06T07:22:41.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.153+0000 7f2f722fb640 1 --2- 172.21.15.83:0/3789862850 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2f6c15e000 0x7f2f6c0d0680 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.153+0000 7f2f722fb640 1 -- 172.21.15.83:0/3789862850 >> 172.21.15.83:0/3789862850 conn(0x7f2f6c0d3cf0 msgr2=0x7f2f6c0d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:41.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.153+0000 7f2f722fb640 1 -- 172.21.15.83:0/3789862850 shutdown_connections 2024-03-06T07:22:41.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.153+0000 7f2f722fb640 1 -- 172.21.15.83:0/3789862850 wait complete. 2024-03-06T07:22:41.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.154+0000 7f2f722fb640 1 Processor -- start 2024-03-06T07:22:41.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.155+0000 7f2f722fb640 1 -- start start 2024-03-06T07:22:41.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.155+0000 7f2f722fb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2f6c0cf7d0 0x7f2f6c14f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.155+0000 7f2f722fb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2f6c0d4810 0x7f2f6c0b7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.155+0000 7f2f722fb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2f6c15e000 0x7f2f6c0cfc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.155+0000 7f2f722fb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c15e3e0 con 0x7f2f6c15e000 2024-03-06T07:22:41.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.155+0000 7f2f722fb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c0d22d0 con 0x7f2f6c0cf7d0 2024-03-06T07:22:41.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.155+0000 7f2f722fb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c0d0bc0 con 0x7f2f6c0d4810 2024-03-06T07:22:41.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.155+0000 7f2f70898640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2f6c15e000 0x7f2f6c0cfc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.155+0000 7f2f70898640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2f6c15e000 0x7f2f6c0cfc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53758/0 (socket says 172.21.15.83:53758) 2024-03-06T07:22:41.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.155+0000 7f2f70898640 1 -- 172.21.15.83:0/1202590055 learned_addr learned my addr 172.21.15.83:0/1202590055 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:41.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.156+0000 7f2f6bfff640 1 --2- 172.21.15.83:0/1202590055 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2f6c0cf7d0 0x7f2f6c14f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.156+0000 7f2f6b7fe640 1 --2- 172.21.15.83:0/1202590055 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2f6c0d4810 0x7f2f6c0b7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.156+0000 7f2f70898640 1 -- 172.21.15.83:0/1202590055 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2f6c0d4810 msgr2=0x7f2f6c0b7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.156+0000 7f2f70898640 1 --2- 172.21.15.83:0/1202590055 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2f6c0d4810 0x7f2f6c0b7170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.156+0000 7f2f70898640 1 -- 172.21.15.83:0/1202590055 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2f6c0cf7d0 msgr2=0x7f2f6c14f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.156+0000 7f2f70898640 1 --2- 172.21.15.83:0/1202590055 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2f6c0cf7d0 0x7f2f6c14f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.156+0000 7f2f70898640 1 -- 172.21.15.83:0/1202590055 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f6c09fe20 con 0x7f2f6c15e000 2024-03-06T07:22:41.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.156+0000 7f2f70898640 1 --2- 172.21.15.83:0/1202590055 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2f6c15e000 0x7f2f6c0cfc60 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f2f6400c7a0 tx=0x7f2f6400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.156+0000 7f2f68ff9640 1 -- 172.21.15.83:0/1202590055 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f64018070 con 0x7f2f6c15e000 2024-03-06T07:22:41.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.156+0000 7f2f68ff9640 1 -- 172.21.15.83:0/1202590055 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f6400a640 con 0x7f2f6c15e000 2024-03-06T07:22:41.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.156+0000 7f2f722fb640 1 -- 172.21.15.83:0/1202590055 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f6c0a0100 con 0x7f2f6c15e000 2024-03-06T07:22:41.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.156+0000 7f2f68ff9640 1 -- 172.21.15.83:0/1202590055 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6400aa70 con 0x7f2f6c15e000 2024-03-06T07:22:41.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.156+0000 7f2f722fb640 1 -- 172.21.15.83:0/1202590055 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f6c0a06c0 con 0x7f2f6c15e000 2024-03-06T07:22:41.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.157+0000 7f2f722fb640 1 -- 172.21.15.83:0/1202590055 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f6c04f330 con 0x7f2f6c15e000 2024-03-06T07:22:41.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.158+0000 7f2f68ff9640 1 -- 172.21.15.83:0/1202590055 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2f6401b070 con 0x7f2f6c15e000 2024-03-06T07:22:41.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.158+0000 7f2f68ff9640 1 --2- 172.21.15.83:0/1202590055 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2f4c06b4d0 0x7f2f4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.158+0000 7f2f68ff9640 1 -- 172.21.15.83:0/1202590055 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5912+0+0 (secure 0 0 0) 0x7f2f64089850 con 0x7f2f6c15e000 2024-03-06T07:22:41.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.158+0000 7f2f6bfff640 1 --2- 172.21.15.83:0/1202590055 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2f4c06b4d0 0x7f2f4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.158+0000 7f2f6bfff640 1 --2- 172.21.15.83:0/1202590055 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2f4c06b4d0 0x7f2f4c06d990 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c036000 tx=0x7f2f5c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.162+0000 7f2f68ff9640 1 -- 172.21.15.83:0/1202590055 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2f64058690 con 0x7f2f6c15e000 2024-03-06T07:22:41.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.353+0000 7f2f722fb640 1 -- 172.21.15.83:0/1202590055 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f2f6c0d0f90 con 0x7f2f6c15e000 2024-03-06T07:22:41.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.353+0000 7f2f68ff9640 1 -- 172.21.15.83:0/1202590055 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f2f64057f90 con 0x7f2f6c15e000 2024-03-06T07:22:41.355 INFO:teuthology.orchestra.run.smithi083.stderr:listed 0 entries 2024-03-06T07:22:41.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.355+0000 7f2f722fb640 1 -- 172.21.15.83:0/1202590055 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2f4c06b4d0 msgr2=0x7f2f4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.355+0000 7f2f722fb640 1 --2- 172.21.15.83:0/1202590055 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2f4c06b4d0 0x7f2f4c06d990 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c036000 tx=0x7f2f5c008030 comp rx=0 tx=0).stop 2024-03-06T07:22:41.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.355+0000 7f2f722fb640 1 -- 172.21.15.83:0/1202590055 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2f6c15e000 msgr2=0x7f2f6c0cfc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.355+0000 7f2f722fb640 1 --2- 172.21.15.83:0/1202590055 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2f6c15e000 0x7f2f6c0cfc60 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f2f6400c7a0 tx=0x7f2f6400cc60 comp rx=0 tx=0).stop 2024-03-06T07:22:41.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.355+0000 7f2f722fb640 1 -- 172.21.15.83:0/1202590055 shutdown_connections 2024-03-06T07:22:41.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.355+0000 7f2f722fb640 1 --2- 172.21.15.83:0/1202590055 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2f4c06b4d0 0x7f2f4c06d990 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.355+0000 7f2f722fb640 1 --2- 172.21.15.83:0/1202590055 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2f6c15e000 0x7f2f6c0cfc60 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.355+0000 7f2f722fb640 1 --2- 172.21.15.83:0/1202590055 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2f6c0d4810 0x7f2f6c0b7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.355+0000 7f2f722fb640 1 --2- 172.21.15.83:0/1202590055 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2f6c0cf7d0 0x7f2f6c14f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.355+0000 7f2f722fb640 1 -- 172.21.15.83:0/1202590055 >> 172.21.15.83:0/1202590055 conn(0x7f2f6c0d3cf0 msgr2=0x7f2f6c0c80e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:41.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.355+0000 7f2f722fb640 1 -- 172.21.15.83:0/1202590055 shutdown_connections 2024-03-06T07:22:41.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.356+0000 7f2f722fb640 1 -- 172.21.15.83:0/1202590055 wait complete. 2024-03-06T07:22:41.368 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:22:41.368 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:41.385 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:41.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.527+0000 7fa5cb0d4640 1 Processor -- start 2024-03-06T07:22:41.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.527+0000 7fa5cb0d4640 1 -- start start 2024-03-06T07:22:41.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.528+0000 7fa5cb0d4640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fa5c4166af0 0x7fa5c4168ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.528+0000 7fa5cb0d4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5c415e380 con 0x7fa5c40cf850 2024-03-06T07:22:41.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.528+0000 7fa5cb0d4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5c40cdc10 con 0x7fa5c40d00c0 2024-03-06T07:22:41.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.528+0000 7fa5cb0d4640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5c406ca90 con 0x7fa5c4166af0 2024-03-06T07:22:41.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.528+0000 7fa5c3fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fa5c40cf850 0x7fa5c40cd3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39192/0 (socket says 172.21.15.83:39192) 2024-03-06T07:22:41.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.528+0000 7fa5c3fff640 1 -- 172.21.15.83:0/51217536 learned_addr learned my addr 172.21.15.83:0/51217536 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:41.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.528+0000 7fa5c9671640 1 --2- 172.21.15.83:0/51217536 >> v2:172.21.15.183:3300/0 conn(0x7fa5c4166af0 0x7fa5c4168ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.529+0000 7fa5c37fe640 1 -- 172.21.15.83:0/51217536 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1368066408 0 0) 0x7fa5c415e380 con 0x7fa5c40cf850 2024-03-06T07:22:41.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.529+0000 7fa5c37fe640 1 -- 172.21.15.83:0/51217536 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5b0003520 con 0x7fa5c40cf850 2024-03-06T07:22:41.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.529+0000 7fa5c37fe640 1 -- 172.21.15.83:0/51217536 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 616391478 0 0) 0x7fa5b0003520 con 0x7fa5c40cf850 2024-03-06T07:22:41.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.529+0000 7fa5c9671640 1 -- 172.21.15.83:0/51217536 >> v1:172.21.15.183:6789/0 conn(0x7fa5c40d00c0 legacy=0x7fa5c40d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.529+0000 7fa5c9671640 1 -- 172.21.15.83:0/51217536 >> v1:172.21.15.83:6789/0 conn(0x7fa5c40cf850 legacy=0x7fa5c40cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.529+0000 7fa5c9671640 1 -- 172.21.15.83:0/51217536 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c406c160 con 0x7fa5c4166af0 2024-03-06T07:22:41.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.530+0000 7fa5c9671640 1 --2- 172.21.15.83:0/51217536 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5c4166af0 0x7fa5c4168ee0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa5b40097a0 tx=0x7fa5b402b680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5e7e67a9ffc0647 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.530+0000 7fa5c37fe640 1 -- 172.21.15.83:0/51217536 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b4037070 con 0x7fa5c4166af0 2024-03-06T07:22:41.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.530+0000 7fa5c37fe640 1 -- 172.21.15.83:0/51217536 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5b4030510 con 0x7fa5c4166af0 2024-03-06T07:22:41.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.530+0000 7fa5c37fe640 1 -- 172.21.15.83:0/51217536 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ac001670 con 0x7fa5c4166af0 2024-03-06T07:22:41.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.530+0000 7fa5cb0d4640 1 -- 172.21.15.83:0/51217536 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5c4166af0 msgr2=0x7fa5c4168ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.530+0000 7fa5cb0d4640 1 --2- 172.21.15.83:0/51217536 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5c4166af0 0x7fa5c4168ee0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa5b40097a0 tx=0x7fa5b402b680 comp rx=0 tx=0).stop 2024-03-06T07:22:41.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.530+0000 7fa5cb0d4640 1 -- 172.21.15.83:0/51217536 shutdown_connections 2024-03-06T07:22:41.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.530+0000 7fa5cb0d4640 1 --2- 172.21.15.83:0/51217536 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5c4166af0 0x7fa5c4168ee0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.530+0000 7fa5cb0d4640 1 -- 172.21.15.83:0/51217536 >> 172.21.15.83:0/51217536 conn(0x7fa5c40b35a0 msgr2=0x7fa5c40b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:41.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.530+0000 7fa5cb0d4640 1 -- 172.21.15.83:0/51217536 shutdown_connections 2024-03-06T07:22:41.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.530+0000 7fa5cb0d4640 1 -- 172.21.15.83:0/51217536 wait complete. 2024-03-06T07:22:41.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.531+0000 7fa5cb0d4640 1 Processor -- start 2024-03-06T07:22:41.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.531+0000 7fa5cb0d4640 1 -- start start 2024-03-06T07:22:41.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.532+0000 7fa5cb0d4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5c40cf850 0x7fa5c40c8b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.532+0000 7fa5cb0d4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa5c40d00c0 0x7fa5c41547a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.532+0000 7fa5cb0d4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5c4166af0 0x7fa5c40cc870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.532+0000 7fa5cb0d4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c406c720 con 0x7fa5c40cf850 2024-03-06T07:22:41.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.532+0000 7fa5cb0d4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c406df30 con 0x7fa5c4166af0 2024-03-06T07:22:41.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.532+0000 7fa5cb0d4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c40b1030 con 0x7fa5c40d00c0 2024-03-06T07:22:41.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.532+0000 7fa5c8e70640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5c40cf850 0x7fa5c40c8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.532+0000 7fa5c8e70640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5c40cf850 0x7fa5c40c8b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53766/0 (socket says 172.21.15.83:53766) 2024-03-06T07:22:41.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.532+0000 7fa5c8e70640 1 -- 172.21.15.83:0/3498274672 learned_addr learned my addr 172.21.15.83:0/3498274672 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:41.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.533+0000 7fa5c9671640 1 --2- 172.21.15.83:0/3498274672 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5c4166af0 0x7fa5c40cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.533+0000 7fa5c3fff640 1 --2- 172.21.15.83:0/3498274672 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa5c40d00c0 0x7fa5c41547a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.533+0000 7fa5c8e70640 1 -- 172.21.15.83:0/3498274672 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa5c40d00c0 msgr2=0x7fa5c41547a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.533+0000 7fa5c8e70640 1 --2- 172.21.15.83:0/3498274672 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa5c40d00c0 0x7fa5c41547a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.533+0000 7fa5c8e70640 1 -- 172.21.15.83:0/3498274672 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5c4166af0 msgr2=0x7fa5c40cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.533+0000 7fa5c8e70640 1 --2- 172.21.15.83:0/3498274672 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5c4166af0 0x7fa5c40cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.533+0000 7fa5c8e70640 1 -- 172.21.15.83:0/3498274672 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c40c9220 con 0x7fa5c40cf850 2024-03-06T07:22:41.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.533+0000 7fa5c8e70640 1 --2- 172.21.15.83:0/3498274672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5c40cf850 0x7fa5c40c8b40 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc00caf0 tx=0x7fa5bc002080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.533+0000 7fa5c17fa640 1 -- 172.21.15.83:0/3498274672 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5bc017070 con 0x7fa5c40cf850 2024-03-06T07:22:41.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.533+0000 7fa5c17fa640 1 -- 172.21.15.83:0/3498274672 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5bc003da0 con 0x7fa5c40cf850 2024-03-06T07:22:41.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.533+0000 7fa5cb0d4640 1 -- 172.21.15.83:0/3498274672 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5c40c9500 con 0x7fa5c40cf850 2024-03-06T07:22:41.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.533+0000 7fa5c17fa640 1 -- 172.21.15.83:0/3498274672 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5bc00a940 con 0x7fa5c40cf850 2024-03-06T07:22:41.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.533+0000 7fa5cb0d4640 1 -- 172.21.15.83:0/3498274672 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5c40c9ac0 con 0x7fa5c40cf850 2024-03-06T07:22:41.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.534+0000 7fa5cb0d4640 1 -- 172.21.15.83:0/3498274672 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5c404f330 con 0x7fa5c40cf850 2024-03-06T07:22:41.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.534+0000 7fa5c17fa640 1 -- 172.21.15.83:0/3498274672 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa5bc01a070 con 0x7fa5c40cf850 2024-03-06T07:22:41.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.535+0000 7fa5c17fa640 1 --2- 172.21.15.83:0/3498274672 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa5a406b4d0 0x7fa5a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.535+0000 7fa5c3fff640 1 --2- 172.21.15.83:0/3498274672 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa5a406b4d0 0x7fa5a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.535+0000 7fa5c17fa640 1 -- 172.21.15.83:0/3498274672 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5912+0+0 (secure 0 0 0) 0x7fa5bc089ae0 con 0x7fa5c40cf850 2024-03-06T07:22:41.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.535+0000 7fa5c3fff640 1 --2- 172.21.15.83:0/3498274672 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa5a406b4d0 0x7fa5a406d990 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac0017e0 tx=0x7fa5ac001610 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.539+0000 7fa5c17fa640 1 -- 172.21.15.83:0/3498274672 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa5bc058240 con 0x7fa5c40cf850 2024-03-06T07:22:41.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.719+0000 7fa5cb0d4640 1 -- 172.21.15.83:0/3498274672 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5c40d15c0 con 0x7fa5c40cf850 2024-03-06T07:22:41.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.720+0000 7fa5c17fa640 1 -- 172.21.15.83:0/3498274672 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+17006 (secure 0 0 0) 0x7fa5bc05c2e0 con 0x7fa5c40cf850 2024-03-06T07:22:41.722 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:22:41.722 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":22,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:22:40.105898+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.318942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.254795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.1f","mappings":[{"from":5,"to":1}]},{"pgid":"3.29","mappings":[{"from":3,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:41.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.722+0000 7fa5cb0d4640 1 -- 172.21.15.83:0/3498274672 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa5a406b4d0 msgr2=0x7fa5a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.722+0000 7fa5cb0d4640 1 --2- 172.21.15.83:0/3498274672 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa5a406b4d0 0x7fa5a406d990 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac0017e0 tx=0x7fa5ac001610 comp rx=0 tx=0).stop 2024-03-06T07:22:41.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.722+0000 7fa5cb0d4640 1 -- 172.21.15.83:0/3498274672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5c40cf850 msgr2=0x7fa5c40c8b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.722+0000 7fa5cb0d4640 1 --2- 172.21.15.83:0/3498274672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5c40cf850 0x7fa5c40c8b40 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc00caf0 tx=0x7fa5bc002080 comp rx=0 tx=0).stop 2024-03-06T07:22:41.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.722+0000 7fa5cb0d4640 1 -- 172.21.15.83:0/3498274672 shutdown_connections 2024-03-06T07:22:41.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.722+0000 7fa5cb0d4640 1 --2- 172.21.15.83:0/3498274672 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa5a406b4d0 0x7fa5a406d990 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.722+0000 7fa5cb0d4640 1 --2- 172.21.15.83:0/3498274672 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5c4166af0 0x7fa5c40cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.722+0000 7fa5cb0d4640 1 --2- 172.21.15.83:0/3498274672 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa5c40d00c0 0x7fa5c41547a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.722+0000 7fa5cb0d4640 1 --2- 172.21.15.83:0/3498274672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5c40cf850 0x7fa5c40c8b40 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.722+0000 7fa5cb0d4640 1 -- 172.21.15.83:0/3498274672 >> 172.21.15.83:0/3498274672 conn(0x7fa5c40b35a0 msgr2=0x7fa5c40b0820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:41.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.722+0000 7fa5cb0d4640 1 -- 172.21.15.83:0/3498274672 shutdown_connections 2024-03-06T07:22:41.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.722+0000 7fa5cb0d4640 1 -- 172.21.15.83:0/3498274672 wait complete. 2024-03-06T07:22:41.735 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:56.318942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:57.254795+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:41.736 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:41.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.837+0000 7fcf5a5b5640 1 Processor -- start 2024-03-06T07:22:41.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.837+0000 7fcf5a5b5640 1 -- start start 2024-03-06T07:22:41.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.838+0000 7fcf5a5b5640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcf541614d0 0x7fcf540d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.838+0000 7fcf5a5b5640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fcf540d1910 0x7fcf540d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.838+0000 7fcf5a5b5640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcf540d22e0 0x7fcf540d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.838+0000 7fcf5a5b5640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf540d2750 con 0x7fcf541614d0 2024-03-06T07:22:41.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.838+0000 7fcf5a5b5640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf541618b0 con 0x7fcf540d1910 2024-03-06T07:22:41.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.838+0000 7fcf5a5b5640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf540cf400 con 0x7fcf540d22e0 2024-03-06T07:22:41.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.838+0000 7fcf53fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcf541614d0 0x7fcf540d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.838+0000 7fcf53fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcf541614d0 0x7fcf540d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53772/0 (socket says 172.21.15.83:53772) 2024-03-06T07:22:41.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.838+0000 7fcf53fff640 1 -- 172.21.15.83:0/1350408821 learned_addr learned my addr 172.21.15.83:0/1350408821 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:41.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.839+0000 7fcf537fe640 1 --2- 172.21.15.83:0/1350408821 >> v2:172.21.15.183:3300/0 conn(0x7fcf540d1910 0x7fcf540d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.839+0000 7fcf58b52640 1 --2- 172.21.15.83:0/1350408821 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcf540d22e0 0x7fcf540d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.839+0000 7fcf53fff640 1 -- 172.21.15.83:0/1350408821 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcf540d22e0 msgr2=0x7fcf540d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:41.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.839+0000 7fcf53fff640 1 --2- 172.21.15.83:0/1350408821 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcf540d22e0 0x7fcf540d06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.839+0000 7fcf53fff640 1 -- 172.21.15.83:0/1350408821 >> v2:172.21.15.183:3300/0 conn(0x7fcf540d1910 msgr2=0x7fcf540d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:41.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.839+0000 7fcf53fff640 1 --2- 172.21.15.83:0/1350408821 >> v2:172.21.15.183:3300/0 conn(0x7fcf540d1910 0x7fcf540d1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.839+0000 7fcf53fff640 1 -- 172.21.15.83:0/1350408821 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf540cf6a0 con 0x7fcf541614d0 2024-03-06T07:22:41.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.839+0000 7fcf53fff640 1 --2- 172.21.15.83:0/1350408821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf541614d0 0x7fcf540d4860 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fcf540ce550 tx=0x7fcf4402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c63c6f783a9f1e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.840+0000 7fcf52ffd640 1 -- 172.21.15.83:0/1350408821 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf44035020 con 0x7fcf541614d0 2024-03-06T07:22:41.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.840+0000 7fcf52ffd640 1 -- 172.21.15.83:0/1350408821 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf44031870 con 0x7fcf541614d0 2024-03-06T07:22:41.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.840+0000 7fcf52ffd640 1 -- 172.21.15.83:0/1350408821 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf44031c40 con 0x7fcf541614d0 2024-03-06T07:22:41.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.840+0000 7fcf5a5b5640 1 -- 172.21.15.83:0/1350408821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf541614d0 msgr2=0x7fcf540d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.840+0000 7fcf5a5b5640 1 --2- 172.21.15.83:0/1350408821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf541614d0 0x7fcf540d4860 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fcf540ce550 tx=0x7fcf4402b910 comp rx=0 tx=0).stop 2024-03-06T07:22:41.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.840+0000 7fcf5a5b5640 1 -- 172.21.15.83:0/1350408821 shutdown_connections 2024-03-06T07:22:41.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.840+0000 7fcf5a5b5640 1 --2- 172.21.15.83:0/1350408821 >> v2:172.21.15.183:3300/0 conn(0x7fcf540d1910 0x7fcf540d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.840+0000 7fcf5a5b5640 1 --2- 172.21.15.83:0/1350408821 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcf540d22e0 0x7fcf540d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.840+0000 7fcf5a5b5640 1 --2- 172.21.15.83:0/1350408821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf541614d0 0x7fcf540d4860 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.840+0000 7fcf5a5b5640 1 -- 172.21.15.83:0/1350408821 >> 172.21.15.83:0/1350408821 conn(0x7fcf540d3d40 msgr2=0x7fcf540d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:41.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.840+0000 7fcf5a5b5640 1 -- 172.21.15.83:0/1350408821 shutdown_connections 2024-03-06T07:22:41.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.840+0000 7fcf5a5b5640 1 -- 172.21.15.83:0/1350408821 wait complete. 2024-03-06T07:22:41.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.841+0000 7fcf5a5b5640 1 Processor -- start 2024-03-06T07:22:41.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.841+0000 7fcf5a5b5640 1 -- start start 2024-03-06T07:22:41.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.842+0000 7fcf5a5b5640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcf540d1910 0x7fcf541f3e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.842+0000 7fcf5a5b5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf540d22e0 0x7fcf540cb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.842+0000 7fcf5a5b5640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcf541614d0 0x7fcf540cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.842+0000 7fcf5a5b5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf540b0910 con 0x7fcf540d22e0 2024-03-06T07:22:41.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.842+0000 7fcf5a5b5640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf540cfc60 con 0x7fcf540d1910 2024-03-06T07:22:41.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.842+0000 7fcf5a5b5640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf540d0be0 con 0x7fcf541614d0 2024-03-06T07:22:41.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.842+0000 7fcf537fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf540d22e0 0x7fcf540cb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.842+0000 7fcf537fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf540d22e0 0x7fcf540cb890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53774/0 (socket says 172.21.15.83:53774) 2024-03-06T07:22:41.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.842+0000 7fcf537fe640 1 -- 172.21.15.83:0/2045779088 learned_addr learned my addr 172.21.15.83:0/2045779088 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:41.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.842+0000 7fcf58b52640 1 --2- 172.21.15.83:0/2045779088 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcf541614d0 0x7fcf540cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.842+0000 7fcf53fff640 1 --2- 172.21.15.83:0/2045779088 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcf540d1910 0x7fcf541f3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.842+0000 7fcf537fe640 1 -- 172.21.15.83:0/2045779088 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcf541614d0 msgr2=0x7fcf540cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.842+0000 7fcf537fe640 1 --2- 172.21.15.83:0/2045779088 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcf541614d0 0x7fcf540cd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.842+0000 7fcf537fe640 1 -- 172.21.15.83:0/2045779088 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcf540d1910 msgr2=0x7fcf541f3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.842+0000 7fcf537fe640 1 --2- 172.21.15.83:0/2045779088 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcf540d1910 0x7fcf541f3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.842+0000 7fcf537fe640 1 -- 172.21.15.83:0/2045779088 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf5415b580 con 0x7fcf540d22e0 2024-03-06T07:22:41.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.843+0000 7fcf537fe640 1 --2- 172.21.15.83:0/2045779088 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf540d22e0 0x7fcf540cb890 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fcf4800d800 tx=0x7fcf4800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.843+0000 7fcf50ff9640 1 -- 172.21.15.83:0/2045779088 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf4800b6c0 con 0x7fcf540d22e0 2024-03-06T07:22:41.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.843+0000 7fcf50ff9640 1 -- 172.21.15.83:0/2045779088 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf48019070 con 0x7fcf540d22e0 2024-03-06T07:22:41.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.843+0000 7fcf50ff9640 1 -- 172.21.15.83:0/2045779088 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf4800ba80 con 0x7fcf540d22e0 2024-03-06T07:22:41.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.843+0000 7fcf5a5b5640 1 -- 172.21.15.83:0/2045779088 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf540c8b30 con 0x7fcf540d22e0 2024-03-06T07:22:41.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.843+0000 7fcf5a5b5640 1 -- 172.21.15.83:0/2045779088 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf540c90f0 con 0x7fcf540d22e0 2024-03-06T07:22:41.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.844+0000 7fcf5a5b5640 1 -- 172.21.15.83:0/2045779088 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf54068ff0 con 0x7fcf540d22e0 2024-03-06T07:22:41.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.844+0000 7fcf50ff9640 1 -- 172.21.15.83:0/2045779088 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcf4800bc20 con 0x7fcf540d22e0 2024-03-06T07:22:41.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.845+0000 7fcf50ff9640 1 --2- 172.21.15.83:0/2045779088 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcf4006b4d0 0x7fcf4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.845+0000 7fcf53fff640 1 --2- 172.21.15.83:0/2045779088 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcf4006b4d0 0x7fcf4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.845+0000 7fcf50ff9640 1 -- 172.21.15.83:0/2045779088 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5912+0+0 (secure 0 0 0) 0x7fcf4808a7b0 con 0x7fcf540d22e0 2024-03-06T07:22:41.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.845+0000 7fcf53fff640 1 --2- 172.21.15.83:0/2045779088 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcf4006b4d0 0x7fcf4006d990 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fcf44036000 tx=0x7fcf4403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:41.849+0000 7fcf50ff9640 1 -- 172.21.15.83:0/2045779088 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcf48059560 con 0x7fcf540d22e0 2024-03-06T07:22:42.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.161+0000 7fcf5a5b5640 1 -- 172.21.15.83:0/2045779088 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcf540b9ad0 con 0x7fcf540d22e0 2024-03-06T07:22:42.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.162+0000 7fcf50ff9640 1 -- 172.21.15.83:0/2045779088 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcf48058e60 con 0x7fcf540d22e0 2024-03-06T07:22:42.164 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:22:42.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.164+0000 7fcf5a5b5640 1 -- 172.21.15.83:0/2045779088 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcf4006b4d0 msgr2=0x7fcf4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.164+0000 7fcf5a5b5640 1 --2- 172.21.15.83:0/2045779088 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcf4006b4d0 0x7fcf4006d990 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fcf44036000 tx=0x7fcf4403c040 comp rx=0 tx=0).stop 2024-03-06T07:22:42.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.164+0000 7fcf5a5b5640 1 -- 172.21.15.83:0/2045779088 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf540d22e0 msgr2=0x7fcf540cb890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.164+0000 7fcf5a5b5640 1 --2- 172.21.15.83:0/2045779088 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf540d22e0 0x7fcf540cb890 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fcf4800d800 tx=0x7fcf4800dcc0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.164+0000 7fcf5a5b5640 1 -- 172.21.15.83:0/2045779088 shutdown_connections 2024-03-06T07:22:42.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.164+0000 7fcf5a5b5640 1 --2- 172.21.15.83:0/2045779088 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcf4006b4d0 0x7fcf4006d990 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.164+0000 7fcf5a5b5640 1 --2- 172.21.15.83:0/2045779088 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcf541614d0 0x7fcf540cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.164+0000 7fcf5a5b5640 1 --2- 172.21.15.83:0/2045779088 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf540d22e0 0x7fcf540cb890 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.164+0000 7fcf5a5b5640 1 --2- 172.21.15.83:0/2045779088 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcf540d1910 0x7fcf541f3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.164+0000 7fcf5a5b5640 1 -- 172.21.15.83:0/2045779088 >> 172.21.15.83:0/2045779088 conn(0x7fcf540d3d40 msgr2=0x7fcf5406b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:42.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.164+0000 7fcf5a5b5640 1 -- 172.21.15.83:0/2045779088 shutdown_connections 2024-03-06T07:22:42.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.164+0000 7fcf5a5b5640 1 -- 172.21.15.83:0/2045779088 wait complete. 2024-03-06T07:22:42.176 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:42.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.278+0000 7f117aaeb640 1 Processor -- start 2024-03-06T07:22:42.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.278+0000 7f117aaeb640 1 -- start start 2024-03-06T07:22:42.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.279+0000 7f117aaeb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f11741614d0 0x7f11740d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.279+0000 7f117aaeb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11740d1910 0x7f11740d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.279+0000 7f117aaeb640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f11740d22b0 0x7f11740d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.279+0000 7f117aaeb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11740d2740 con 0x7f11741614d0 2024-03-06T07:22:42.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.279+0000 7f117aaeb640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11741618b0 con 0x7f11740d22b0 2024-03-06T07:22:42.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.279+0000 7f117aaeb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11740cf400 con 0x7f11740d1910 2024-03-06T07:22:42.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.279+0000 7f1178887640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f11741614d0 0x7f11740d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.279+0000 7f1178887640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f11741614d0 0x7f11740d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53790/0 (socket says 172.21.15.83:53790) 2024-03-06T07:22:42.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.279+0000 7f1178887640 1 -- 172.21.15.83:0/1094592511 learned_addr learned my addr 172.21.15.83:0/1094592511 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:42.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.279+0000 7f1179088640 1 --2- 172.21.15.83:0/1094592511 >> v2:172.21.15.183:3300/0 conn(0x7f11740d22b0 0x7f11740d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.279+0000 7f1173fff640 1 --2- 172.21.15.83:0/1094592511 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11740d1910 0x7f11740d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.279+0000 7f1178887640 1 -- 172.21.15.83:0/1094592511 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11740d1910 msgr2=0x7f11740d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:42.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.279+0000 7f1178887640 1 --2- 172.21.15.83:0/1094592511 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11740d1910 0x7f11740d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.279+0000 7f1178887640 1 -- 172.21.15.83:0/1094592511 >> v2:172.21.15.183:3300/0 conn(0x7f11740d22b0 msgr2=0x7f11740d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:42.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.279+0000 7f1178887640 1 --2- 172.21.15.83:0/1094592511 >> v2:172.21.15.183:3300/0 conn(0x7f11740d22b0 0x7f11740d06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.279+0000 7f1178887640 1 -- 172.21.15.83:0/1094592511 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11740cf6a0 con 0x7f11741614d0 2024-03-06T07:22:42.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.279+0000 7f1173fff640 1 --2- 172.21.15.83:0/1094592511 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11740d1910 0x7f11740d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:42.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.279+0000 7f1179088640 1 --2- 172.21.15.83:0/1094592511 >> v2:172.21.15.183:3300/0 conn(0x7f11740d22b0 0x7f11740d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:42.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.280+0000 7f1178887640 1 --2- 172.21.15.83:0/1094592511 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11741614d0 0x7f11740d4860 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f11740b57d0 tx=0x7f116802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3858b992309c58f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.280+0000 7f11737fe640 1 -- 172.21.15.83:0/1094592511 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1168035020 con 0x7f11741614d0 2024-03-06T07:22:42.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.280+0000 7f11737fe640 1 -- 172.21.15.83:0/1094592511 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1168031870 con 0x7f11741614d0 2024-03-06T07:22:42.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.280+0000 7f11737fe640 1 -- 172.21.15.83:0/1094592511 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1168031c40 con 0x7f11741614d0 2024-03-06T07:22:42.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.281+0000 7f117aaeb640 1 -- 172.21.15.83:0/1094592511 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11741614d0 msgr2=0x7f11740d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.281+0000 7f117aaeb640 1 --2- 172.21.15.83:0/1094592511 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11741614d0 0x7f11740d4860 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f11740b57d0 tx=0x7f116802b910 comp rx=0 tx=0).stop 2024-03-06T07:22:42.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.281+0000 7f117aaeb640 1 -- 172.21.15.83:0/1094592511 shutdown_connections 2024-03-06T07:22:42.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.281+0000 7f117aaeb640 1 --2- 172.21.15.83:0/1094592511 >> v2:172.21.15.183:3300/0 conn(0x7f11740d22b0 0x7f11740d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.281+0000 7f117aaeb640 1 --2- 172.21.15.83:0/1094592511 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11740d1910 0x7f11740d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.281+0000 7f117aaeb640 1 --2- 172.21.15.83:0/1094592511 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11741614d0 0x7f11740d4860 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.281+0000 7f117aaeb640 1 -- 172.21.15.83:0/1094592511 >> 172.21.15.83:0/1094592511 conn(0x7f11740d3d40 msgr2=0x7f11740d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:42.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.281+0000 7f117aaeb640 1 -- 172.21.15.83:0/1094592511 shutdown_connections 2024-03-06T07:22:42.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.281+0000 7f117aaeb640 1 -- 172.21.15.83:0/1094592511 wait complete. 2024-03-06T07:22:42.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.282+0000 7f117aaeb640 1 Processor -- start 2024-03-06T07:22:42.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.282+0000 7f117aaeb640 1 -- start start 2024-03-06T07:22:42.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.283+0000 7f117aaeb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11740d1910 0x7f117414bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.283+0000 7f117aaeb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11740d22b0 0x7f117414b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.283+0000 7f117aaeb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11741614d0 0x7f11740cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.283+0000 7f117aaeb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11740b0910 con 0x7f11741614d0 2024-03-06T07:22:42.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.283+0000 7f117aaeb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11740cfc60 con 0x7f11740d1910 2024-03-06T07:22:42.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.283+0000 7f117aaeb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11740d0be0 con 0x7f11740d22b0 2024-03-06T07:22:42.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.283+0000 7f1179088640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11741614d0 0x7f11740cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.283+0000 7f1179088640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11741614d0 0x7f11740cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53804/0 (socket says 172.21.15.83:53804) 2024-03-06T07:22:42.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.283+0000 7f1179088640 1 -- 172.21.15.83:0/361633771 learned_addr learned my addr 172.21.15.83:0/361633771 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:42.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.283+0000 7f1178887640 1 --2- 172.21.15.83:0/361633771 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11740d1910 0x7f117414bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.283+0000 7f1179088640 1 -- 172.21.15.83:0/361633771 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11740d22b0 msgr2=0x7f117414b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.283+0000 7f1179088640 1 --2- 172.21.15.83:0/361633771 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11740d22b0 0x7f117414b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.283+0000 7f1179088640 1 -- 172.21.15.83:0/361633771 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11740d1910 msgr2=0x7f117414bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.283+0000 7f1179088640 1 --2- 172.21.15.83:0/361633771 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11740d1910 0x7f117414bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.283+0000 7f1179088640 1 -- 172.21.15.83:0/361633771 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f117414b940 con 0x7f11741614d0 2024-03-06T07:22:42.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.283+0000 7f1179088640 1 --2- 172.21.15.83:0/361633771 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11741614d0 0x7f11740cd7c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f116c009fa0 tx=0x7f116c00d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.283+0000 7f11717fa640 1 -- 172.21.15.83:0/361633771 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116c01a070 con 0x7f11741614d0 2024-03-06T07:22:42.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.283+0000 7f11717fa640 1 -- 172.21.15.83:0/361633771 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f116c003bb0 con 0x7f11741614d0 2024-03-06T07:22:42.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.283+0000 7f117aaeb640 1 -- 172.21.15.83:0/361633771 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f117414df10 con 0x7f11741614d0 2024-03-06T07:22:42.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.283+0000 7f11717fa640 1 -- 172.21.15.83:0/361633771 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116c004030 con 0x7f11741614d0 2024-03-06T07:22:42.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.283+0000 7f117aaeb640 1 -- 172.21.15.83:0/361633771 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f117414e4d0 con 0x7f11741614d0 2024-03-06T07:22:42.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.284+0000 7f117aaeb640 1 -- 172.21.15.83:0/361633771 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f117414eb90 con 0x7f11741614d0 2024-03-06T07:22:42.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.285+0000 7f11717fa640 1 -- 172.21.15.83:0/361633771 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f116c01d070 con 0x7f11741614d0 2024-03-06T07:22:42.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.285+0000 7f11717fa640 1 --2- 172.21.15.83:0/361633771 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f115806b4d0 0x7f115806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.285+0000 7f1178887640 1 --2- 172.21.15.83:0/361633771 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f115806b4d0 0x7f115806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.285+0000 7f11717fa640 1 -- 172.21.15.83:0/361633771 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5912+0+0 (secure 0 0 0) 0x7f116c08a5d0 con 0x7f11741614d0 2024-03-06T07:22:42.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.286+0000 7f1178887640 1 --2- 172.21.15.83:0/361633771 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f115806b4d0 0x7f115806d990 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1168036000 tx=0x7f1168008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.290+0000 7f11717fa640 1 -- 172.21.15.83:0/361633771 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f117414eb90 con 0x7f11741614d0 2024-03-06T07:22:42.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.465+0000 7f117aaeb640 1 -- 172.21.15.83:0/361633771 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f11740a1c90 con 0x7f11741614d0 2024-03-06T07:22:42.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.465+0000 7f11717fa640 1 -- 172.21.15.83:0/361633771 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f116c010070 con 0x7f11741614d0 2024-03-06T07:22:42.467 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-03-06T07:22:42.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.467+0000 7f117aaeb640 1 -- 172.21.15.83:0/361633771 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f115806b4d0 msgr2=0x7f115806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.467+0000 7f117aaeb640 1 --2- 172.21.15.83:0/361633771 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f115806b4d0 0x7f115806d990 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1168036000 tx=0x7f1168008030 comp rx=0 tx=0).stop 2024-03-06T07:22:42.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.467+0000 7f117aaeb640 1 -- 172.21.15.83:0/361633771 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11741614d0 msgr2=0x7f11740cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.467+0000 7f117aaeb640 1 --2- 172.21.15.83:0/361633771 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11741614d0 0x7f11740cd7c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f116c009fa0 tx=0x7f116c00d560 comp rx=0 tx=0).stop 2024-03-06T07:22:42.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.467+0000 7f117aaeb640 1 -- 172.21.15.83:0/361633771 shutdown_connections 2024-03-06T07:22:42.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.467+0000 7f117aaeb640 1 --2- 172.21.15.83:0/361633771 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f115806b4d0 0x7f115806d990 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.467+0000 7f117aaeb640 1 --2- 172.21.15.83:0/361633771 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11741614d0 0x7f11740cd7c0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.467+0000 7f117aaeb640 1 --2- 172.21.15.83:0/361633771 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11740d22b0 0x7f117414b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.467+0000 7f117aaeb640 1 --2- 172.21.15.83:0/361633771 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11740d1910 0x7f117414bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.467+0000 7f117aaeb640 1 -- 172.21.15.83:0/361633771 >> 172.21.15.83:0/361633771 conn(0x7f11740d3d40 msgr2=0x7f11740cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:42.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.467+0000 7f117aaeb640 1 -- 172.21.15.83:0/361633771 shutdown_connections 2024-03-06T07:22:42.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.467+0000 7f117aaeb640 1 -- 172.21.15.83:0/361633771 wait complete. 2024-03-06T07:22:42.480 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:42.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.582+0000 7f1d7799a640 1 Processor -- start 2024-03-06T07:22:42.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.582+0000 7f1d7799a640 1 -- start start 2024-03-06T07:22:42.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.583+0000 7f1d7799a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1d700cf200 0x7f1d700cf670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.583+0000 7f1d7799a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d701600a0 con 0x7f1d7015fc30 2024-03-06T07:22:42.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.583+0000 7f1d7799a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d700d47f0 con 0x7f1d700d06e0 2024-03-06T07:22:42.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.583+0000 7f1d7799a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d700ced70 con 0x7f1d700cf200 2024-03-06T07:22:42.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.583+0000 7f1d75f37640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1d700cf200 0x7f1d700cf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.583+0000 7f1d75f37640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1d700cf200 0x7f1d700cf670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53818/0 (socket says 172.21.15.83:53818) 2024-03-06T07:22:42.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.583+0000 7f1d74f35640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f1d7015fc30 0x7f1d700b47d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39198/0 (socket says 172.21.15.83:39198) 2024-03-06T07:22:42.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.583+0000 7f1d75f37640 1 -- 172.21.15.83:0/2769542033 learned_addr learned my addr 172.21.15.83:0/2769542033 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:42.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.584+0000 7f1d75f37640 1 -- 172.21.15.83:0/2769542033 >> v1:172.21.15.183:6789/0 conn(0x7f1d700d06e0 legacy=0x7f1d700d0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.584+0000 7f1d75f37640 1 -- 172.21.15.83:0/2769542033 >> v1:172.21.15.83:6789/0 conn(0x7f1d7015fc30 legacy=0x7f1d700b47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.584+0000 7f1d75f37640 1 -- 172.21.15.83:0/2769542033 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d700d1ab0 con 0x7f1d700cf200 2024-03-06T07:22:42.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.584+0000 7f1d75f37640 1 --2- 172.21.15.83:0/2769542033 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d700cf200 0x7f1d700cf670 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1d68009e50 tx=0x7f1d6802c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=523d9dc6cd8dcb7b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.585+0000 7f1d67fff640 1 -- 172.21.15.83:0/2769542033 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d6802cbb0 con 0x7f1d700cf200 2024-03-06T07:22:42.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.585+0000 7f1d67fff640 1 -- 172.21.15.83:0/2769542033 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d6802cd50 con 0x7f1d700cf200 2024-03-06T07:22:42.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.585+0000 7f1d67fff640 1 -- 172.21.15.83:0/2769542033 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d68034610 con 0x7f1d700cf200 2024-03-06T07:22:42.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.585+0000 7f1d7799a640 1 -- 172.21.15.83:0/2769542033 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d700cf200 msgr2=0x7f1d700cf670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.585+0000 7f1d7799a640 1 --2- 172.21.15.83:0/2769542033 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d700cf200 0x7f1d700cf670 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1d68009e50 tx=0x7f1d6802c010 comp rx=0 tx=0).stop 2024-03-06T07:22:42.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.585+0000 7f1d7799a640 1 -- 172.21.15.83:0/2769542033 shutdown_connections 2024-03-06T07:22:42.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.585+0000 7f1d7799a640 1 --2- 172.21.15.83:0/2769542033 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d700cf200 0x7f1d700cf670 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.585+0000 7f1d7799a640 1 -- 172.21.15.83:0/2769542033 >> 172.21.15.83:0/2769542033 conn(0x7f1d700d3cd0 msgr2=0x7f1d700d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:42.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.585+0000 7f1d7799a640 1 -- 172.21.15.83:0/2769542033 shutdown_connections 2024-03-06T07:22:42.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.585+0000 7f1d7799a640 1 -- 172.21.15.83:0/2769542033 wait complete. 2024-03-06T07:22:42.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.586+0000 7f1d7799a640 1 Processor -- start 2024-03-06T07:22:42.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.587+0000 7f1d7799a640 1 -- start start 2024-03-06T07:22:42.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.587+0000 7f1d7799a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1d700cf200 0x7f1d700b3450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.587+0000 7f1d7799a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1d700d06e0 0x7f1d700cd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.587+0000 7f1d7799a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d7015fc30 0x7f1d700cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.587+0000 7f1d7799a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d700ce990 con 0x7f1d7015fc30 2024-03-06T07:22:42.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.587+0000 7f1d7799a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d700d4ab0 con 0x7f1d700cf200 2024-03-06T07:22:42.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.587+0000 7f1d7799a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d700b44d0 con 0x7f1d700d06e0 2024-03-06T07:22:42.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.587+0000 7f1d75f37640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d7015fc30 0x7f1d700cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.587+0000 7f1d75f37640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d7015fc30 0x7f1d700cbdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53824/0 (socket says 172.21.15.83:53824) 2024-03-06T07:22:42.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.587+0000 7f1d75f37640 1 -- 172.21.15.83:0/1383023312 learned_addr learned my addr 172.21.15.83:0/1383023312 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:42.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.587+0000 7f1d75736640 1 --2- 172.21.15.83:0/1383023312 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1d700cf200 0x7f1d700b3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.587+0000 7f1d74f35640 1 --2- 172.21.15.83:0/1383023312 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1d700d06e0 0x7f1d700cd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.588+0000 7f1d75f37640 1 -- 172.21.15.83:0/1383023312 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1d700d06e0 msgr2=0x7f1d700cd2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.588+0000 7f1d75f37640 1 --2- 172.21.15.83:0/1383023312 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1d700d06e0 0x7f1d700cd2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.588+0000 7f1d75f37640 1 -- 172.21.15.83:0/1383023312 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1d700cf200 msgr2=0x7f1d700b3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.588+0000 7f1d75f37640 1 --2- 172.21.15.83:0/1383023312 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1d700cf200 0x7f1d700b3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.588+0000 7f1d75f37640 1 -- 172.21.15.83:0/1383023312 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d700cd7e0 con 0x7f1d7015fc30 2024-03-06T07:22:42.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.588+0000 7f1d74f35640 1 --2- 172.21.15.83:0/1383023312 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1d700d06e0 0x7f1d700cd2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:42.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.588+0000 7f1d75f37640 1 --2- 172.21.15.83:0/1383023312 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d7015fc30 0x7f1d700cbdf0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f1d68009e50 tx=0x7f1d68004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.588+0000 7f1d65ffb640 1 -- 172.21.15.83:0/1383023312 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d68043020 con 0x7f1d7015fc30 2024-03-06T07:22:42.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.588+0000 7f1d65ffb640 1 -- 172.21.15.83:0/1383023312 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d6803e450 con 0x7f1d7015fc30 2024-03-06T07:22:42.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.588+0000 7f1d7799a640 1 -- 172.21.15.83:0/1383023312 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d700ca450 con 0x7f1d7015fc30 2024-03-06T07:22:42.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.588+0000 7f1d65ffb640 1 -- 172.21.15.83:0/1383023312 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d68009510 con 0x7f1d7015fc30 2024-03-06T07:22:42.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.588+0000 7f1d7799a640 1 -- 172.21.15.83:0/1383023312 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d700b0a30 con 0x7f1d7015fc30 2024-03-06T07:22:42.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.589+0000 7f1d7799a640 1 -- 172.21.15.83:0/1383023312 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d700a1c80 con 0x7f1d7015fc30 2024-03-06T07:22:42.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.590+0000 7f1d65ffb640 1 -- 172.21.15.83:0/1383023312 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1d68046020 con 0x7f1d7015fc30 2024-03-06T07:22:42.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.590+0000 7f1d65ffb640 1 --2- 172.21.15.83:0/1383023312 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1d5406b4d0 0x7f1d5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.590+0000 7f1d65ffb640 1 -- 172.21.15.83:0/1383023312 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5912+0+0 (secure 0 0 0) 0x7f1d680acb50 con 0x7f1d7015fc30 2024-03-06T07:22:42.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.590+0000 7f1d75736640 1 --2- 172.21.15.83:0/1383023312 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1d5406b4d0 0x7f1d5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.591+0000 7f1d75736640 1 --2- 172.21.15.83:0/1383023312 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1d5406b4d0 0x7f1d5406d990 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1d600016e0 tx=0x7f1d60009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.599+0000 7f1d65ffb640 1 -- 172.21.15.83:0/1383023312 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1d6804d030 con 0x7f1d7015fc30 2024-03-06T07:22:42.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.777+0000 7f1d7799a640 1 -- 172.21.15.83:0/1383023312 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1d700d2090 con 0x7f1d7015fc30 2024-03-06T07:22:42.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.778+0000 7f1d65ffb640 1 -- 172.21.15.83:0/1383023312 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1d6807b900 con 0x7f1d7015fc30 2024-03-06T07:22:42.780 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 64 2024-03-06T07:22:42.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.779+0000 7f1d7799a640 1 -- 172.21.15.83:0/1383023312 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1d5406b4d0 msgr2=0x7f1d5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.780+0000 7f1d7799a640 1 --2- 172.21.15.83:0/1383023312 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1d5406b4d0 0x7f1d5406d990 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1d600016e0 tx=0x7f1d60009040 comp rx=0 tx=0).stop 2024-03-06T07:22:42.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.780+0000 7f1d7799a640 1 -- 172.21.15.83:0/1383023312 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d7015fc30 msgr2=0x7f1d700cbdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.780+0000 7f1d7799a640 1 --2- 172.21.15.83:0/1383023312 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d7015fc30 0x7f1d700cbdf0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f1d68009e50 tx=0x7f1d68004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:42.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.780+0000 7f1d7799a640 1 -- 172.21.15.83:0/1383023312 shutdown_connections 2024-03-06T07:22:42.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.780+0000 7f1d7799a640 1 --2- 172.21.15.83:0/1383023312 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1d5406b4d0 0x7f1d5406d990 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.780+0000 7f1d7799a640 1 --2- 172.21.15.83:0/1383023312 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d7015fc30 0x7f1d700cbdf0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.780+0000 7f1d7799a640 1 --2- 172.21.15.83:0/1383023312 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1d700d06e0 0x7f1d700cd2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.780+0000 7f1d7799a640 1 --2- 172.21.15.83:0/1383023312 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1d700cf200 0x7f1d700b3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.780+0000 7f1d7799a640 1 -- 172.21.15.83:0/1383023312 >> 172.21.15.83:0/1383023312 conn(0x7f1d700d3cd0 msgr2=0x7f1d70160d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:42.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.780+0000 7f1d7799a640 1 -- 172.21.15.83:0/1383023312 shutdown_connections 2024-03-06T07:22:42.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.780+0000 7f1d7799a640 1 -- 172.21.15.83:0/1383023312 wait complete. 2024-03-06T07:22:42.792 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:22:42.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.892+0000 7f183225c640 1 Processor -- start 2024-03-06T07:22:42.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.892+0000 7f183225c640 1 -- start start 2024-03-06T07:22:42.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.893+0000 7f183225c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f182c161670 0x7f182c072520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.893+0000 7f183225c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f182c072a60 0x7f182c072ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.893+0000 7f183225c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f182c066d00 con 0x7f182c0adf10 2024-03-06T07:22:42.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.893+0000 7f183225c640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f182c0b6170 con 0x7f182c072a60 2024-03-06T07:22:42.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.893+0000 7f183225c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f182c0ada80 con 0x7f182c161670 2024-03-06T07:22:42.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.893+0000 7f182affd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f182c072a60 0x7f182c072ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.893+0000 7f182affd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f182c072a60 0x7f182c072ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53826/0 (socket says 172.21.15.83:53826) 2024-03-06T07:22:42.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.893+0000 7f182affd640 1 -- 172.21.15.83:0/558297193 learned_addr learned my addr 172.21.15.83:0/558297193 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:42.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.893+0000 7f182b7fe640 1 --2- 172.21.15.83:0/558297193 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f182c161670 0x7f182c072520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.893+0000 7f182affd640 1 -- 172.21.15.83:0/558297193 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f182c161670 msgr2=0x7f182c072520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:42.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.893+0000 7f182affd640 1 --2- 172.21.15.83:0/558297193 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f182c161670 0x7f182c072520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.894+0000 7f182affd640 1 -- 172.21.15.83:0/558297193 >> v1:172.21.15.183:6789/0 conn(0x7f182c0adf10 legacy=0x7f182c0ae360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.894+0000 7f182affd640 1 -- 172.21.15.83:0/558297193 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f182c066fc0 con 0x7f182c072a60 2024-03-06T07:22:42.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.894+0000 7f182affd640 1 --2- 172.21.15.83:0/558297193 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f182c072a60 0x7f182c072ec0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f1814009870 tx=0x7f181402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1f14d994a786c87 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.894+0000 7f182a7fc640 1 -- 172.21.15.83:0/558297193 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1814034070 con 0x7f182c072a60 2024-03-06T07:22:42.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.894+0000 7f182a7fc640 1 -- 172.21.15.83:0/558297193 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1814030840 con 0x7f182c072a60 2024-03-06T07:22:42.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.894+0000 7f182a7fc640 1 -- 172.21.15.83:0/558297193 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1814030c10 con 0x7f182c072a60 2024-03-06T07:22:42.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.895+0000 7f183225c640 1 -- 172.21.15.83:0/558297193 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f182c072a60 msgr2=0x7f182c072ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.895+0000 7f183225c640 1 --2- 172.21.15.83:0/558297193 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f182c072a60 0x7f182c072ec0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f1814009870 tx=0x7f181402b980 comp rx=0 tx=0).stop 2024-03-06T07:22:42.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.895+0000 7f183225c640 1 -- 172.21.15.83:0/558297193 shutdown_connections 2024-03-06T07:22:42.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.895+0000 7f183225c640 1 --2- 172.21.15.83:0/558297193 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f182c072a60 0x7f182c072ec0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.895+0000 7f183225c640 1 --2- 172.21.15.83:0/558297193 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f182c161670 0x7f182c072520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.895+0000 7f183225c640 1 -- 172.21.15.83:0/558297193 >> 172.21.15.83:0/558297193 conn(0x7f182c0b4e60 msgr2=0x7f182c0b5520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:42.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.895+0000 7f183225c640 1 -- 172.21.15.83:0/558297193 shutdown_connections 2024-03-06T07:22:42.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.895+0000 7f183225c640 1 -- 172.21.15.83:0/558297193 wait complete. 2024-03-06T07:22:42.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.896+0000 7f183225c640 1 Processor -- start 2024-03-06T07:22:42.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.896+0000 7f183225c640 1 -- start start 2024-03-06T07:22:42.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.897+0000 7f183225c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f182c072a60 0x7f182c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.897+0000 7f183225c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f182c0adf10 0x7f182c154500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.897+0000 7f183225c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f182c161670 0x7f182c0afd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.897+0000 7f183225c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f182c06b330 con 0x7f182c072a60 2024-03-06T07:22:42.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.897+0000 7f183225c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f182c161a50 con 0x7f182c0adf10 2024-03-06T07:22:42.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.897+0000 7f183225c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f182c06b610 con 0x7f182c161670 2024-03-06T07:22:42.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.897+0000 7f182b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f182c072a60 0x7f182c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.897+0000 7f182b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f182c072a60 0x7f182c06e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53836/0 (socket says 172.21.15.83:53836) 2024-03-06T07:22:42.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.897+0000 7f182b7fe640 1 -- 172.21.15.83:0/4050448378 learned_addr learned my addr 172.21.15.83:0/4050448378 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:42.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.897+0000 7f182affd640 1 --2- 172.21.15.83:0/4050448378 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f182c0adf10 0x7f182c154500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.897+0000 7f182bfff640 1 --2- 172.21.15.83:0/4050448378 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f182c161670 0x7f182c0afd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.897+0000 7f182b7fe640 1 -- 172.21.15.83:0/4050448378 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f182c161670 msgr2=0x7f182c0afd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.897+0000 7f182b7fe640 1 --2- 172.21.15.83:0/4050448378 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f182c161670 0x7f182c0afd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.897+0000 7f182b7fe640 1 -- 172.21.15.83:0/4050448378 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f182c0adf10 msgr2=0x7f182c154500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.897+0000 7f182b7fe640 1 --2- 172.21.15.83:0/4050448378 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f182c0adf10 0x7f182c154500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.898+0000 7f182b7fe640 1 -- 172.21.15.83:0/4050448378 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f182c08ca30 con 0x7f182c072a60 2024-03-06T07:22:42.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.898+0000 7f182affd640 1 --2- 172.21.15.83:0/4050448378 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f182c0adf10 0x7f182c154500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:42.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.898+0000 7f182b7fe640 1 --2- 172.21.15.83:0/4050448378 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f182c072a60 0x7f182c06e2f0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f181c007ef0 tx=0x7f181c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.898+0000 7f1823fff640 1 -- 172.21.15.83:0/4050448378 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181c014620 con 0x7f182c072a60 2024-03-06T07:22:42.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.898+0000 7f1823fff640 1 -- 172.21.15.83:0/4050448378 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f181c019070 con 0x7f182c072a60 2024-03-06T07:22:42.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.898+0000 7f1823fff640 1 -- 172.21.15.83:0/4050448378 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181c014a00 con 0x7f182c072a60 2024-03-06T07:22:42.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.898+0000 7f183225c640 1 -- 172.21.15.83:0/4050448378 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f182c08cd10 con 0x7f182c072a60 2024-03-06T07:22:42.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.898+0000 7f183225c640 1 -- 172.21.15.83:0/4050448378 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f182c08d2d0 con 0x7f182c072a60 2024-03-06T07:22:42.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.899+0000 7f183225c640 1 -- 172.21.15.83:0/4050448378 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f182c04f330 con 0x7f182c072a60 2024-03-06T07:22:42.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.900+0000 7f1823fff640 1 -- 172.21.15.83:0/4050448378 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f181c014ba0 con 0x7f182c072a60 2024-03-06T07:22:42.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.900+0000 7f1823fff640 1 --2- 172.21.15.83:0/4050448378 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f180c06b4d0 0x7f180c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.900+0000 7f182affd640 1 --2- 172.21.15.83:0/4050448378 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f180c06b4d0 0x7f180c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.900+0000 7f1823fff640 1 -- 172.21.15.83:0/4050448378 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5912+0+0 (secure 0 0 0) 0x7f181c08ac40 con 0x7f182c072a60 2024-03-06T07:22:42.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.901+0000 7f182affd640 1 --2- 172.21.15.83:0/4050448378 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f180c06b4d0 0x7f180c06d990 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1814036000 tx=0x7f181403b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:42.905+0000 7f1823fff640 1 -- 172.21.15.83:0/4050448378 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f181c0599f0 con 0x7f182c072a60 2024-03-06T07:22:43.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.230+0000 7f183225c640 1 -- 172.21.15.83:0/4050448378 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f182c0b09d0 con 0x7f182c072a60 2024-03-06T07:22:43.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.230+0000 7f1823fff640 1 -- 172.21.15.83:0/4050448378 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f181c0592f0 con 0x7f182c072a60 2024-03-06T07:22:43.232 INFO:teuthology.orchestra.run.smithi083.stderr:listed 0 entries 2024-03-06T07:22:43.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.233+0000 7f183225c640 1 -- 172.21.15.83:0/4050448378 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f180c06b4d0 msgr2=0x7f180c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.233+0000 7f183225c640 1 --2- 172.21.15.83:0/4050448378 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f180c06b4d0 0x7f180c06d990 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1814036000 tx=0x7f181403b040 comp rx=0 tx=0).stop 2024-03-06T07:22:43.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.233+0000 7f183225c640 1 -- 172.21.15.83:0/4050448378 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f182c072a60 msgr2=0x7f182c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.233+0000 7f183225c640 1 --2- 172.21.15.83:0/4050448378 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f182c072a60 0x7f182c06e2f0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f181c007ef0 tx=0x7f181c00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.233+0000 7f183225c640 1 -- 172.21.15.83:0/4050448378 shutdown_connections 2024-03-06T07:22:43.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.233+0000 7f183225c640 1 --2- 172.21.15.83:0/4050448378 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f180c06b4d0 0x7f180c06d990 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.233+0000 7f183225c640 1 --2- 172.21.15.83:0/4050448378 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f182c161670 0x7f182c0afd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.233+0000 7f183225c640 1 --2- 172.21.15.83:0/4050448378 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f182c0adf10 0x7f182c154500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.233+0000 7f183225c640 1 --2- 172.21.15.83:0/4050448378 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f182c072a60 0x7f182c06e2f0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.233+0000 7f183225c640 1 -- 172.21.15.83:0/4050448378 >> 172.21.15.83:0/4050448378 conn(0x7f182c0b4e60 msgr2=0x7f182c0cf7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:43.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.233+0000 7f183225c640 1 -- 172.21.15.83:0/4050448378 shutdown_connections 2024-03-06T07:22:43.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.233+0000 7f183225c640 1 -- 172.21.15.83:0/4050448378 wait complete. 2024-03-06T07:22:43.249 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi083.front.sepia.ceph.com... 2024-03-06T07:22:43.249 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:43.249 DEBUG:teuthology.orchestra.run.smithi083:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-03-06T07:22:43.276 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:43.277 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi083.front.sepia.ceph.com 2024-03-06T07:22:43.277 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:43.277 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:22:43.332 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:22:43.332 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:43.351 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:22:43.351 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:43.409 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:43.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.510+0000 7f0b822fb640 1 Processor -- start 2024-03-06T07:22:43.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.510+0000 7f0b822fb640 1 -- start start 2024-03-06T07:22:43.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.511+0000 7f0b822fb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b7c1515e0 0x7f0b7c0cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.511+0000 7f0b822fb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b7c071f40 con 0x7f0b7c0cd7a0 2024-03-06T07:22:43.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.511+0000 7f0b822fb640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b7c15b8b0 con 0x7f0b7c15b430 2024-03-06T07:22:43.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.511+0000 7f0b822fb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c070a70 con 0x7f0b7c1515e0 2024-03-06T07:22:43.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.511+0000 7f0b80898640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f0b7c0cd7a0 0x7f0b7c0cdbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39200/0 (socket says 172.21.15.83:39200) 2024-03-06T07:22:43.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.511+0000 7f0b80898640 1 -- 172.21.15.83:0/1254201874 learned_addr learned my addr 172.21.15.83:0/1254201874 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:43.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.511+0000 7f0b7affd640 1 -- 172.21.15.83:0/1254201874 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2971014660 0 0) 0x7f0b7c071f40 con 0x7f0b7c0cd7a0 2024-03-06T07:22:43.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.511+0000 7f0b7affd640 1 -- 172.21.15.83:0/1254201874 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b64003520 con 0x7f0b7c0cd7a0 2024-03-06T07:22:43.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.511+0000 7f0b7bfff640 1 --2- 172.21.15.83:0/1254201874 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b7c1515e0 0x7f0b7c0cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.511+0000 7f0b7affd640 1 -- 172.21.15.83:0/1254201874 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4244000054 0 0) 0x7f0b64003520 con 0x7f0b7c0cd7a0 2024-03-06T07:22:43.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.511+0000 7f0b7affd640 1 -- 172.21.15.83:0/1254201874 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b7c1515e0 msgr2=0x7f0b7c0cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:43.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.511+0000 7f0b7affd640 1 --2- 172.21.15.83:0/1254201874 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b7c1515e0 0x7f0b7c0cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.511+0000 7f0b7affd640 1 -- 172.21.15.83:0/1254201874 >> v1:172.21.15.183:6789/0 conn(0x7f0b7c15b430 legacy=0x7f0b7c071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.511+0000 7f0b7affd640 1 -- 172.21.15.83:0/1254201874 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b7c070d50 con 0x7f0b7c0cd7a0 2024-03-06T07:22:43.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.511+0000 7f0b7affd640 1 -- 172.21.15.83:0/1254201874 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f0b740019f0 con 0x7f0b7c0cd7a0 2024-03-06T07:22:43.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.512+0000 7f0b7affd640 1 -- 172.21.15.83:0/1254201874 >> v1:172.21.15.83:6789/0 conn(0x7f0b7c0cd7a0 legacy=0x7f0b7c0cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.512+0000 7f0b7affd640 1 --2- 172.21.15.83:0/1254201874 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b640039e0 0x7f0b64003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.512+0000 7f0b7affd640 1 --2- 172.21.15.83:0/1254201874 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b64004800 0x7f0b64004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.512+0000 7f0b7affd640 1 --2- 172.21.15.83:0/1254201874 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b640051c0 0x7f0b6400cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.512+0000 7f0b7affd640 1 -- 172.21.15.83:0/1254201874 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c070a70 con 0x7f0b640051c0 2024-03-06T07:22:43.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.512+0000 7f0b7affd640 1 -- 172.21.15.83:0/1254201874 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b6400d1a0 con 0x7f0b640039e0 2024-03-06T07:22:43.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.512+0000 7f0b7affd640 1 -- 172.21.15.83:0/1254201874 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b6400d3a0 con 0x7f0b64004800 2024-03-06T07:22:43.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.512+0000 7f0b7bfff640 1 --2- 172.21.15.83:0/1254201874 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b7c1515e0 0x7f0b7c0cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:43.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.512+0000 7f0b80898640 1 --2- 172.21.15.83:0/1254201874 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b640051c0 0x7f0b6400cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.512+0000 7f0b7b7fe640 1 --2- 172.21.15.83:0/1254201874 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b64004800 0x7f0b64004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.512+0000 7f0b7bfff640 1 --2- 172.21.15.83:0/1254201874 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b640039e0 0x7f0b64003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.512+0000 7f0b80898640 1 -- 172.21.15.83:0/1254201874 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b64004800 msgr2=0x7f0b64004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:43.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.512+0000 7f0b80898640 1 --2- 172.21.15.83:0/1254201874 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b64004800 0x7f0b64004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.512+0000 7f0b80898640 1 -- 172.21.15.83:0/1254201874 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b640039e0 msgr2=0x7f0b64003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:43.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.512+0000 7f0b80898640 1 --2- 172.21.15.83:0/1254201874 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b640039e0 0x7f0b64003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.512+0000 7f0b80898640 1 -- 172.21.15.83:0/1254201874 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b64003520 con 0x7f0b640051c0 2024-03-06T07:22:43.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.512+0000 7f0b7bfff640 1 --2- 172.21.15.83:0/1254201874 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b640039e0 0x7f0b64003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:43.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.513+0000 7f0b80898640 1 --2- 172.21.15.83:0/1254201874 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b640051c0 0x7f0b6400cad0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0b7400a850 tx=0x7f0b64005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97db5bd325682cf3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.513+0000 7f0b7affd640 1 -- 172.21.15.83:0/1254201874 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7400a4a0 con 0x7f0b640051c0 2024-03-06T07:22:43.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.513+0000 7f0b7affd640 1 -- 172.21.15.83:0/1254201874 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b74034650 con 0x7f0b640051c0 2024-03-06T07:22:43.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.513+0000 7f0b7affd640 1 -- 172.21.15.83:0/1254201874 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b74034a40 con 0x7f0b640051c0 2024-03-06T07:22:43.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.513+0000 7f0b822fb640 1 -- 172.21.15.83:0/1254201874 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b640051c0 msgr2=0x7f0b6400cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.513+0000 7f0b822fb640 1 --2- 172.21.15.83:0/1254201874 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b640051c0 0x7f0b6400cad0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0b7400a850 tx=0x7f0b64005830 comp rx=0 tx=0).stop 2024-03-06T07:22:43.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.513+0000 7f0b80898640 1 -- 172.21.15.83:0/1254201874 reap_dead start 2024-03-06T07:22:43.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.513+0000 7f0b822fb640 1 -- 172.21.15.83:0/1254201874 shutdown_connections 2024-03-06T07:22:43.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.513+0000 7f0b822fb640 1 -- 172.21.15.83:0/1254201874 >> 172.21.15.83:0/1254201874 conn(0x7f0b7c0cf8a0 msgr2=0x7f0b7c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:43.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.513+0000 7f0b822fb640 1 -- 172.21.15.83:0/1254201874 shutdown_connections 2024-03-06T07:22:43.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.513+0000 7f0b822fb640 1 -- 172.21.15.83:0/1254201874 wait complete. 2024-03-06T07:22:43.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.515+0000 7f0b822fb640 1 Processor -- start 2024-03-06T07:22:43.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.515+0000 7f0b822fb640 1 -- start start 2024-03-06T07:22:43.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.517+0000 7f0b822fb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b7c156a00 0x7f0b7c0c49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.517+0000 7f0b822fb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b7c1570f0 0x7f0b7c154f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.517+0000 7f0b822fb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b7c1554b0 0x7f0b7c155910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.517+0000 7f0b822fb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c0cce40 con 0x7f0b7c1554b0 2024-03-06T07:22:43.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.517+0000 7f0b822fb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c071310 con 0x7f0b7c1570f0 2024-03-06T07:22:43.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.517+0000 7f0b822fb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c0ccfc0 con 0x7f0b7c156a00 2024-03-06T07:22:43.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.517+0000 7f0b80898640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b7c1554b0 0x7f0b7c155910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.517+0000 7f0b80898640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b7c1554b0 0x7f0b7c155910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53848/0 (socket says 172.21.15.83:53848) 2024-03-06T07:22:43.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.517+0000 7f0b80898640 1 -- 172.21.15.83:0/2489351586 learned_addr learned my addr 172.21.15.83:0/2489351586 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:43.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.517+0000 7f0b7b7fe640 1 --2- 172.21.15.83:0/2489351586 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b7c1570f0 0x7f0b7c154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.517+0000 7f0b7bfff640 1 --2- 172.21.15.83:0/2489351586 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b7c156a00 0x7f0b7c0c49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.517+0000 7f0b80898640 1 -- 172.21.15.83:0/2489351586 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b7c156a00 msgr2=0x7f0b7c0c49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.517+0000 7f0b80898640 1 --2- 172.21.15.83:0/2489351586 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b7c156a00 0x7f0b7c0c49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.517+0000 7f0b80898640 1 -- 172.21.15.83:0/2489351586 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b7c1570f0 msgr2=0x7f0b7c154f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.517+0000 7f0b80898640 1 --2- 172.21.15.83:0/2489351586 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b7c1570f0 0x7f0b7c154f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.517+0000 7f0b80898640 1 -- 172.21.15.83:0/2489351586 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b7c157860 con 0x7f0b7c1554b0 2024-03-06T07:22:43.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.517+0000 7f0b7b7fe640 1 --2- 172.21.15.83:0/2489351586 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b7c1570f0 0x7f0b7c154f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:43.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.517+0000 7f0b7bfff640 1 --2- 172.21.15.83:0/2489351586 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b7c156a00 0x7f0b7c0c49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:43.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.517+0000 7f0b80898640 1 --2- 172.21.15.83:0/2489351586 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b7c1554b0 0x7f0b7c155910 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f0b7402cc70 tx=0x7f0b74001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.517+0000 7f0b78ff9640 1 -- 172.21.15.83:0/2489351586 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7400a480 con 0x7f0b7c1554b0 2024-03-06T07:22:43.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.517+0000 7f0b78ff9640 1 -- 172.21.15.83:0/2489351586 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b74045070 con 0x7f0b7c1554b0 2024-03-06T07:22:43.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.517+0000 7f0b822fb640 1 -- 172.21.15.83:0/2489351586 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b7c157b40 con 0x7f0b7c1554b0 2024-03-06T07:22:43.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.517+0000 7f0b78ff9640 1 -- 172.21.15.83:0/2489351586 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b74034a00 con 0x7f0b7c1554b0 2024-03-06T07:22:43.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.518+0000 7f0b822fb640 1 -- 172.21.15.83:0/2489351586 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b7c158100 con 0x7f0b7c1554b0 2024-03-06T07:22:43.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.518+0000 7f0b822fb640 1 -- 172.21.15.83:0/2489351586 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b7c068840 con 0x7f0b7c1554b0 2024-03-06T07:22:43.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.519+0000 7f0b78ff9640 1 -- 172.21.15.83:0/2489351586 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0b74034ba0 con 0x7f0b7c1554b0 2024-03-06T07:22:43.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.519+0000 7f0b78ff9640 1 --2- 172.21.15.83:0/2489351586 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0b6806b4d0 0x7f0b6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.519+0000 7f0b78ff9640 1 -- 172.21.15.83:0/2489351586 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5912+0+0 (secure 0 0 0) 0x7f0b740ac520 con 0x7f0b7c1554b0 2024-03-06T07:22:43.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.519+0000 7f0b7bfff640 1 --2- 172.21.15.83:0/2489351586 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0b6806b4d0 0x7f0b6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.520+0000 7f0b7bfff640 1 --2- 172.21.15.83:0/2489351586 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0b6806b4d0 0x7f0b6806d990 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0b70006fd0 tx=0x7f0b70008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.524+0000 7f0b78ff9640 1 -- 172.21.15.83:0/2489351586 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0b7407b240 con 0x7f0b7c1554b0 2024-03-06T07:22:43.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.702+0000 7f0b822fb640 1 -- 172.21.15.83:0/2489351586 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b7c04f330 con 0x7f0b7c1554b0 2024-03-06T07:22:43.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.703+0000 7f0b78ff9640 1 -- 172.21.15.83:0/2489351586 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+17006 (secure 0 0 0) 0x7f0b7407ab40 con 0x7f0b7c1554b0 2024-03-06T07:22:43.705 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:22:43.705 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":22,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:22:40.105898+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.318942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.254795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.1f","mappings":[{"from":5,"to":1}]},{"pgid":"3.29","mappings":[{"from":3,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:43.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.705+0000 7f0b822fb640 1 -- 172.21.15.83:0/2489351586 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0b6806b4d0 msgr2=0x7f0b6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.705+0000 7f0b822fb640 1 --2- 172.21.15.83:0/2489351586 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0b6806b4d0 0x7f0b6806d990 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0b70006fd0 tx=0x7f0b70008040 comp rx=0 tx=0).stop 2024-03-06T07:22:43.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.705+0000 7f0b822fb640 1 -- 172.21.15.83:0/2489351586 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b7c1554b0 msgr2=0x7f0b7c155910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.705+0000 7f0b822fb640 1 --2- 172.21.15.83:0/2489351586 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b7c1554b0 0x7f0b7c155910 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f0b7402cc70 tx=0x7f0b74001dc0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.705+0000 7f0b822fb640 1 -- 172.21.15.83:0/2489351586 shutdown_connections 2024-03-06T07:22:43.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.705+0000 7f0b822fb640 1 --2- 172.21.15.83:0/2489351586 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0b6806b4d0 0x7f0b6806d990 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.705+0000 7f0b822fb640 1 --2- 172.21.15.83:0/2489351586 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b7c1554b0 0x7f0b7c155910 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.705+0000 7f0b822fb640 1 --2- 172.21.15.83:0/2489351586 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b7c1570f0 0x7f0b7c154f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.705+0000 7f0b822fb640 1 --2- 172.21.15.83:0/2489351586 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b7c156a00 0x7f0b7c0c49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.705+0000 7f0b822fb640 1 -- 172.21.15.83:0/2489351586 >> 172.21.15.83:0/2489351586 conn(0x7f0b7c0cf8a0 msgr2=0x7f0b7c0cdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:43.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.705+0000 7f0b822fb640 1 -- 172.21.15.83:0/2489351586 shutdown_connections 2024-03-06T07:22:43.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.705+0000 7f0b822fb640 1 -- 172.21.15.83:0/2489351586 wait complete. 2024-03-06T07:22:43.718 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:56.318942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:57.254795+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:43.718 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:43.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.821+0000 7fe33cc76640 1 Processor -- start 2024-03-06T07:22:43.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.821+0000 7fe33cc76640 1 -- start start 2024-03-06T07:22:43.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.822+0000 7fe33cc76640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fe3380d10c0 0x7fe3380d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.822+0000 7fe33cc76640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3380cfa50 con 0x7fe3380d14a0 2024-03-06T07:22:43.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.822+0000 7fe33cc76640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3380d0a30 con 0x7fe33815ecd0 2024-03-06T07:22:43.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.822+0000 7fe33cc76640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe33815c050 con 0x7fe3380d10c0 2024-03-06T07:22:43.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.822+0000 7fe32ffff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fe3380d14a0 0x7fe33815e5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39212/0 (socket says 172.21.15.83:39212) 2024-03-06T07:22:43.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.822+0000 7fe32ffff640 1 -- 172.21.15.83:0/953334736 learned_addr learned my addr 172.21.15.83:0/953334736 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:43.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.822+0000 7fe33659c640 1 --2- 172.21.15.83:0/953334736 >> v2:172.21.15.183:3300/0 conn(0x7fe3380d10c0 0x7fe3380d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.822+0000 7fe335d9b640 1 -- 172.21.15.83:0/953334736 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4204945615 0 0) 0x7fe3380cfa50 con 0x7fe3380d14a0 2024-03-06T07:22:43.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.822+0000 7fe335d9b640 1 -- 172.21.15.83:0/953334736 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe324003520 con 0x7fe3380d14a0 2024-03-06T07:22:43.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.822+0000 7fe335d9b640 1 -- 172.21.15.83:0/953334736 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2922750430 0 0) 0x7fe324003520 con 0x7fe3380d14a0 2024-03-06T07:22:43.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.822+0000 7fe335d9b640 1 -- 172.21.15.83:0/953334736 >> v2:172.21.15.183:3300/0 conn(0x7fe3380d10c0 msgr2=0x7fe3380d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:43.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.822+0000 7fe335d9b640 1 --2- 172.21.15.83:0/953334736 >> v2:172.21.15.183:3300/0 conn(0x7fe3380d10c0 0x7fe3380d04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.822+0000 7fe335d9b640 1 -- 172.21.15.83:0/953334736 >> v1:172.21.15.183:6789/0 conn(0x7fe33815ecd0 legacy=0x7fe3380cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.822+0000 7fe335d9b640 1 -- 172.21.15.83:0/953334736 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3380b45c0 con 0x7fe3380d14a0 2024-03-06T07:22:43.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.822+0000 7fe335d9b640 1 -- 172.21.15.83:0/953334736 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7fe320001670 con 0x7fe3380d14a0 2024-03-06T07:22:43.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.822+0000 7fe33659c640 1 --2- 172.21.15.83:0/953334736 >> v2:172.21.15.183:3300/0 conn(0x7fe3380d10c0 0x7fe3380d04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:43.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.822+0000 7fe335d9b640 1 -- 172.21.15.83:0/953334736 >> v1:172.21.15.83:6789/0 conn(0x7fe3380d14a0 legacy=0x7fe33815e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.822+0000 7fe335d9b640 1 --2- 172.21.15.83:0/953334736 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe3240039b0 0x7fe324003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.822+0000 7fe335d9b640 1 --2- 172.21.15.83:0/953334736 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe324004800 0x7fe324004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.823+0000 7fe335d9b640 1 --2- 172.21.15.83:0/953334736 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3240051c0 0x7fe32400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.823+0000 7fe335d9b640 1 -- 172.21.15.83:0/953334736 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe32400c030 con 0x7fe324004800 2024-03-06T07:22:43.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.823+0000 7fe335d9b640 1 -- 172.21.15.83:0/953334736 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe32400c1e0 con 0x7fe3240051c0 2024-03-06T07:22:43.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.823+0000 7fe335d9b640 1 -- 172.21.15.83:0/953334736 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe32400c3e0 con 0x7fe3240039b0 2024-03-06T07:22:43.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.823+0000 7fe32ffff640 1 --2- 172.21.15.83:0/953334736 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe324004800 0x7fe324004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.823+0000 7fe33659c640 1 --2- 172.21.15.83:0/953334736 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe3240039b0 0x7fe324003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.823+0000 7fe336d9d640 1 --2- 172.21.15.83:0/953334736 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3240051c0 0x7fe32400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.823+0000 7fe32ffff640 1 -- 172.21.15.83:0/953334736 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe3240039b0 msgr2=0x7fe324003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:43.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.823+0000 7fe32ffff640 1 --2- 172.21.15.83:0/953334736 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe3240039b0 0x7fe324003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.823+0000 7fe32ffff640 1 -- 172.21.15.83:0/953334736 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3240051c0 msgr2=0x7fe32400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:43.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.823+0000 7fe32ffff640 1 --2- 172.21.15.83:0/953334736 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3240051c0 0x7fe32400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.823+0000 7fe32ffff640 1 -- 172.21.15.83:0/953334736 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe324003520 con 0x7fe324004800 2024-03-06T07:22:43.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.823+0000 7fe336d9d640 1 -- 172.21.15.83:0/953334736 reap_dead start 2024-03-06T07:22:43.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.824+0000 7fe32ffff640 1 --2- 172.21.15.83:0/953334736 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe324004800 0x7fe324004c80 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe32000c210 tx=0x7fe324005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b3557d2e95d1c06 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.824+0000 7fe335d9b640 1 -- 172.21.15.83:0/953334736 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe320007d30 con 0x7fe324004800 2024-03-06T07:22:43.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.824+0000 7fe335d9b640 1 -- 172.21.15.83:0/953334736 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe320030070 con 0x7fe324004800 2024-03-06T07:22:43.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.824+0000 7fe335d9b640 1 -- 172.21.15.83:0/953334736 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe32002cda0 con 0x7fe324004800 2024-03-06T07:22:43.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.824+0000 7fe33cc76640 1 -- 172.21.15.83:0/953334736 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe324004800 msgr2=0x7fe324004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.824+0000 7fe33cc76640 1 --2- 172.21.15.83:0/953334736 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe324004800 0x7fe324004c80 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe32000c210 tx=0x7fe324005830 comp rx=0 tx=0).stop 2024-03-06T07:22:43.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.824+0000 7fe33cc76640 1 -- 172.21.15.83:0/953334736 shutdown_connections 2024-03-06T07:22:43.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.824+0000 7fe33cc76640 1 --2- 172.21.15.83:0/953334736 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe324004800 0x7fe324004c80 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.824+0000 7fe33cc76640 1 -- 172.21.15.83:0/953334736 >> 172.21.15.83:0/953334736 conn(0x7fe3380b3800 msgr2=0x7fe3380b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:43.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.825+0000 7fe33cc76640 1 -- 172.21.15.83:0/953334736 shutdown_connections 2024-03-06T07:22:43.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.825+0000 7fe33cc76640 1 -- 172.21.15.83:0/953334736 wait complete. 2024-03-06T07:22:43.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.826+0000 7fe33cc76640 1 Processor -- start 2024-03-06T07:22:43.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.826+0000 7fe33cc76640 1 -- start start 2024-03-06T07:22:43.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.827+0000 7fe33cc76640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe324004800 0x7fe33814a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.827+0000 7fe33cc76640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe3380b5290 0x7fe3380b4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.827+0000 7fe33cc76640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3380afb30 0x7fe3380affc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.827+0000 7fe33cc76640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe33815f440 con 0x7fe324004800 2024-03-06T07:22:43.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.827+0000 7fe33cc76640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3380d0060 con 0x7fe3380afb30 2024-03-06T07:22:43.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.827+0000 7fe33cc76640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3380cfcf0 con 0x7fe3380b5290 2024-03-06T07:22:43.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.827+0000 7fe33659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe324004800 0x7fe33814a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.827+0000 7fe33659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe324004800 0x7fe33814a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53856/0 (socket says 172.21.15.83:53856) 2024-03-06T07:22:43.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.827+0000 7fe33659c640 1 -- 172.21.15.83:0/1763867935 learned_addr learned my addr 172.21.15.83:0/1763867935 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:43.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.827+0000 7fe32ffff640 1 --2- 172.21.15.83:0/1763867935 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe3380b5290 0x7fe3380b4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.827+0000 7fe336d9d640 1 --2- 172.21.15.83:0/1763867935 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3380afb30 0x7fe3380affc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.827+0000 7fe33659c640 1 -- 172.21.15.83:0/1763867935 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe3380b5290 msgr2=0x7fe3380b4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.827+0000 7fe33659c640 1 --2- 172.21.15.83:0/1763867935 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe3380b5290 0x7fe3380b4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.827+0000 7fe33659c640 1 -- 172.21.15.83:0/1763867935 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3380afb30 msgr2=0x7fe3380affc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.827+0000 7fe33659c640 1 --2- 172.21.15.83:0/1763867935 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3380afb30 0x7fe3380affc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.827+0000 7fe33659c640 1 -- 172.21.15.83:0/1763867935 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3380b5790 con 0x7fe324004800 2024-03-06T07:22:43.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.827+0000 7fe336d9d640 1 --2- 172.21.15.83:0/1763867935 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3380afb30 0x7fe3380affc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:43.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.827+0000 7fe33659c640 1 --2- 172.21.15.83:0/1763867935 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe324004800 0x7fe33814a660 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe328011d00 tx=0x7fe32800f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.827+0000 7fe32effd640 1 -- 172.21.15.83:0/1763867935 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe32800fd20 con 0x7fe324004800 2024-03-06T07:22:43.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.827+0000 7fe32effd640 1 -- 172.21.15.83:0/1763867935 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe328004020 con 0x7fe324004800 2024-03-06T07:22:43.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.827+0000 7fe33cc76640 1 -- 172.21.15.83:0/1763867935 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3380b5a70 con 0x7fe324004800 2024-03-06T07:22:43.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.827+0000 7fe32effd640 1 -- 172.21.15.83:0/1763867935 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe328004410 con 0x7fe324004800 2024-03-06T07:22:43.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.827+0000 7fe33cc76640 1 -- 172.21.15.83:0/1763867935 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3380cc180 con 0x7fe324004800 2024-03-06T07:22:43.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.828+0000 7fe33cc76640 1 -- 172.21.15.83:0/1763867935 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe33809fe60 con 0x7fe324004800 2024-03-06T07:22:43.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.829+0000 7fe32effd640 1 -- 172.21.15.83:0/1763867935 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe328012040 con 0x7fe324004800 2024-03-06T07:22:43.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.829+0000 7fe32effd640 1 --2- 172.21.15.83:0/1763867935 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe31806b4d0 0x7fe31806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.829+0000 7fe32effd640 1 -- 172.21.15.83:0/1763867935 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5912+0+0 (secure 0 0 0) 0x7fe32808a900 con 0x7fe324004800 2024-03-06T07:22:43.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.829+0000 7fe32ffff640 1 --2- 172.21.15.83:0/1763867935 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe31806b4d0 0x7fe31806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.830+0000 7fe32ffff640 1 --2- 172.21.15.83:0/1763867935 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe31806b4d0 0x7fe31806d990 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe320034000 tx=0x7fe320009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:43.838+0000 7fe32effd640 1 -- 172.21.15.83:0/1763867935 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe328019030 con 0x7fe324004800 2024-03-06T07:22:44.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.014+0000 7fe33cc76640 1 -- 172.21.15.83:0/1763867935 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe3380b5c40 con 0x7fe324004800 2024-03-06T07:22:44.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.014+0000 7fe32effd640 1 -- 172.21.15.83:0/1763867935 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe3280596b0 con 0x7fe324004800 2024-03-06T07:22:44.016 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:22:44.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.016+0000 7fe33cc76640 1 -- 172.21.15.83:0/1763867935 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe31806b4d0 msgr2=0x7fe31806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.016+0000 7fe33cc76640 1 --2- 172.21.15.83:0/1763867935 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe31806b4d0 0x7fe31806d990 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe320034000 tx=0x7fe320009030 comp rx=0 tx=0).stop 2024-03-06T07:22:44.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.016+0000 7fe33cc76640 1 -- 172.21.15.83:0/1763867935 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe324004800 msgr2=0x7fe33814a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.016+0000 7fe33cc76640 1 --2- 172.21.15.83:0/1763867935 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe324004800 0x7fe33814a660 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe328011d00 tx=0x7fe32800f610 comp rx=0 tx=0).stop 2024-03-06T07:22:44.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.016+0000 7fe33cc76640 1 -- 172.21.15.83:0/1763867935 shutdown_connections 2024-03-06T07:22:44.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.016+0000 7fe33cc76640 1 --2- 172.21.15.83:0/1763867935 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe31806b4d0 0x7fe31806d990 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.016+0000 7fe33cc76640 1 --2- 172.21.15.83:0/1763867935 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3380afb30 0x7fe3380affc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.016+0000 7fe33cc76640 1 --2- 172.21.15.83:0/1763867935 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe3380b5290 0x7fe3380b4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.016+0000 7fe33cc76640 1 --2- 172.21.15.83:0/1763867935 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe324004800 0x7fe33814a660 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.016+0000 7fe33cc76640 1 -- 172.21.15.83:0/1763867935 >> 172.21.15.83:0/1763867935 conn(0x7fe3380b3800 msgr2=0x7fe3380cae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:44.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.016+0000 7fe33cc76640 1 -- 172.21.15.83:0/1763867935 shutdown_connections 2024-03-06T07:22:44.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.016+0000 7fe33cc76640 1 -- 172.21.15.83:0/1763867935 wait complete. 2024-03-06T07:22:44.029 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:44.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.131+0000 7fb2d24af640 1 Processor -- start 2024-03-06T07:22:44.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.131+0000 7fb2d24af640 1 -- start start 2024-03-06T07:22:44.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.131+0000 7fb2d24af640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb2cc15ad90 0x7fb2cc0b3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.131+0000 7fb2d24af640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fb2cc0b4110 0x7fb2cc15c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.131+0000 7fb2d24af640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2cc15b260 con 0x7fb2cc071a70 2024-03-06T07:22:44.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.131+0000 7fb2d24af640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2cc15c020 con 0x7fb2cc15ad90 2024-03-06T07:22:44.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.131+0000 7fb2d24af640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2cc15b540 con 0x7fb2cc0b4110 2024-03-06T07:22:44.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.132+0000 7fb2cb7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb2cc15ad90 0x7fb2cc0b3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.132+0000 7fb2cb7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb2cc15ad90 0x7fb2cc0b3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53872/0 (socket says 172.21.15.83:53872) 2024-03-06T07:22:44.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.132+0000 7fb2cb7fe640 1 -- 172.21.15.83:0/2624539044 learned_addr learned my addr 172.21.15.83:0/2624539044 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:44.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.132+0000 7fb2cbfff640 1 --1- 172.21.15.83:0/2624539044 >> v1:172.21.15.183:6789/0 conn(0x7fb2cc071a70 0x7fb2cc071e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:55994/0 (socket says 172.21.15.83:55994) 2024-03-06T07:22:44.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.132+0000 7fb2d0a4c640 1 --2- 172.21.15.83:0/2624539044 >> v2:172.21.15.183:3300/0 conn(0x7fb2cc0b4110 0x7fb2cc15c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.132+0000 7fb2cb7fe640 1 -- 172.21.15.83:0/2624539044 >> v2:172.21.15.183:3300/0 conn(0x7fb2cc0b4110 msgr2=0x7fb2cc15c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:44.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.132+0000 7fb2cb7fe640 1 --2- 172.21.15.83:0/2624539044 >> v2:172.21.15.183:3300/0 conn(0x7fb2cc0b4110 0x7fb2cc15c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.132+0000 7fb2cb7fe640 1 -- 172.21.15.83:0/2624539044 >> v1:172.21.15.183:6789/0 conn(0x7fb2cc071a70 legacy=0x7fb2cc071e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.132+0000 7fb2cb7fe640 1 -- 172.21.15.83:0/2624539044 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2cc15b820 con 0x7fb2cc15ad90 2024-03-06T07:22:44.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.132+0000 7fb2caffd640 1 -- 172.21.15.83:0/2624539044 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1890490864 0 0) 0x7fb2cc15b260 con 0x7fb2cc071a70 2024-03-06T07:22:44.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.133+0000 7fb2cb7fe640 1 --2- 172.21.15.83:0/2624539044 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2cc15ad90 0x7fb2cc0b3bd0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc15e5a0 tx=0x7fb2c002e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1f6cd331c5b1d11 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.133+0000 7fb2caffd640 1 -- 172.21.15.83:0/2624539044 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c003b070 con 0x7fb2cc15ad90 2024-03-06T07:22:44.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.133+0000 7fb2caffd640 1 -- 172.21.15.83:0/2624539044 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c0036440 con 0x7fb2cc15ad90 2024-03-06T07:22:44.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.133+0000 7fb2caffd640 1 -- 172.21.15.83:0/2624539044 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c0036810 con 0x7fb2cc15ad90 2024-03-06T07:22:44.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.133+0000 7fb2d24af640 1 -- 172.21.15.83:0/2624539044 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2cc15ad90 msgr2=0x7fb2cc0b3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.133+0000 7fb2d24af640 1 --2- 172.21.15.83:0/2624539044 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2cc15ad90 0x7fb2cc0b3bd0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc15e5a0 tx=0x7fb2c002e0e0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.133+0000 7fb2d24af640 1 -- 172.21.15.83:0/2624539044 shutdown_connections 2024-03-06T07:22:44.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.133+0000 7fb2d24af640 1 --2- 172.21.15.83:0/2624539044 >> v2:172.21.15.183:3300/0 conn(0x7fb2cc0b4110 0x7fb2cc15c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.133+0000 7fb2d24af640 1 --2- 172.21.15.83:0/2624539044 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2cc15ad90 0x7fb2cc0b3bd0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.133+0000 7fb2d24af640 1 -- 172.21.15.83:0/2624539044 >> 172.21.15.83:0/2624539044 conn(0x7fb2cc0b2660 msgr2=0x7fb2cc0b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:44.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.133+0000 7fb2d24af640 1 -- 172.21.15.83:0/2624539044 shutdown_connections 2024-03-06T07:22:44.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.133+0000 7fb2d24af640 1 -- 172.21.15.83:0/2624539044 wait complete. 2024-03-06T07:22:44.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.134+0000 7fb2d24af640 1 Processor -- start 2024-03-06T07:22:44.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.135+0000 7fb2d24af640 1 -- start start 2024-03-06T07:22:44.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.136+0000 7fb2d24af640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2cc071a70 0x7fb2cc1494a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.136+0000 7fb2d24af640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb2cc0b4110 0x7fb2cc0aeda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.136+0000 7fb2d24af640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb2cc15ad90 0x7fb2cc0af2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.136+0000 7fb2d24af640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2cc0b4700 con 0x7fb2cc071a70 2024-03-06T07:22:44.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.136+0000 7fb2d24af640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2cc0b4580 con 0x7fb2cc0b4110 2024-03-06T07:22:44.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.136+0000 7fb2d24af640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2cc15c750 con 0x7fb2cc15ad90 2024-03-06T07:22:44.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.136+0000 7fb2cbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2cc071a70 0x7fb2cc1494a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.136+0000 7fb2cbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2cc071a70 0x7fb2cc1494a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53882/0 (socket says 172.21.15.83:53882) 2024-03-06T07:22:44.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.136+0000 7fb2cbfff640 1 -- 172.21.15.83:0/2002836033 learned_addr learned my addr 172.21.15.83:0/2002836033 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:44.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.136+0000 7fb2cb7fe640 1 --2- 172.21.15.83:0/2002836033 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb2cc0b4110 0x7fb2cc0aeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.136+0000 7fb2d0a4c640 1 --2- 172.21.15.83:0/2002836033 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb2cc15ad90 0x7fb2cc0af2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.136+0000 7fb2cbfff640 1 -- 172.21.15.83:0/2002836033 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb2cc15ad90 msgr2=0x7fb2cc0af2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.136+0000 7fb2cbfff640 1 --2- 172.21.15.83:0/2002836033 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb2cc15ad90 0x7fb2cc0af2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.136+0000 7fb2cbfff640 1 -- 172.21.15.83:0/2002836033 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb2cc0b4110 msgr2=0x7fb2cc0aeda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.136+0000 7fb2cbfff640 1 --2- 172.21.15.83:0/2002836033 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb2cc0b4110 0x7fb2cc0aeda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.136+0000 7fb2cbfff640 1 -- 172.21.15.83:0/2002836033 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2cc1499e0 con 0x7fb2cc071a70 2024-03-06T07:22:44.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.136+0000 7fb2d0a4c640 1 --2- 172.21.15.83:0/2002836033 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb2cc15ad90 0x7fb2cc0af2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:44.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.136+0000 7fb2cbfff640 1 --2- 172.21.15.83:0/2002836033 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2cc071a70 0x7fb2cc1494a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc00c4d0 tx=0x7fb2bc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.137+0000 7fb2c8ff9640 1 -- 172.21.15.83:0/2002836033 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc018070 con 0x7fb2cc071a70 2024-03-06T07:22:44.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.137+0000 7fb2c8ff9640 1 -- 172.21.15.83:0/2002836033 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2bc007500 con 0x7fb2cc071a70 2024-03-06T07:22:44.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.137+0000 7fb2d24af640 1 -- 172.21.15.83:0/2002836033 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2cc0af8f0 con 0x7fb2cc071a70 2024-03-06T07:22:44.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.137+0000 7fb2c8ff9640 1 -- 172.21.15.83:0/2002836033 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc004a20 con 0x7fb2cc071a70 2024-03-06T07:22:44.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.137+0000 7fb2d24af640 1 -- 172.21.15.83:0/2002836033 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2cc0afeb0 con 0x7fb2cc071a70 2024-03-06T07:22:44.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.138+0000 7fb2d24af640 1 -- 172.21.15.83:0/2002836033 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2cc0cd8f0 con 0x7fb2cc071a70 2024-03-06T07:22:44.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.138+0000 7fb2c8ff9640 1 -- 172.21.15.83:0/2002836033 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb2bc01b070 con 0x7fb2cc071a70 2024-03-06T07:22:44.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.138+0000 7fb2c8ff9640 1 --2- 172.21.15.83:0/2002836033 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb2b806b4d0 0x7fb2b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.139+0000 7fb2c8ff9640 1 -- 172.21.15.83:0/2002836033 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5912+0+0 (secure 0 0 0) 0x7fb2bc089540 con 0x7fb2cc071a70 2024-03-06T07:22:44.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.139+0000 7fb2cb7fe640 1 --2- 172.21.15.83:0/2002836033 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb2b806b4d0 0x7fb2b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.139+0000 7fb2cb7fe640 1 --2- 172.21.15.83:0/2002836033 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb2b806b4d0 0x7fb2b806d990 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb2c0038000 tx=0x7fb2c0032000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.143+0000 7fb2c8ff9640 1 -- 172.21.15.83:0/2002836033 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb2cc0cd8f0 con 0x7fb2cc071a70 2024-03-06T07:22:44.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.319+0000 7fb2d24af640 1 -- 172.21.15.83:0/2002836033 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb2cc0d07d0 con 0x7fb2cc071a70 2024-03-06T07:22:44.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.320+0000 7fb2c8ff9640 1 -- 172.21.15.83:0/2002836033 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb2cc1499e0 con 0x7fb2cc071a70 2024-03-06T07:22:44.322 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-03-06T07:22:44.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.321+0000 7fb2d24af640 1 -- 172.21.15.83:0/2002836033 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb2b806b4d0 msgr2=0x7fb2b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.321+0000 7fb2d24af640 1 --2- 172.21.15.83:0/2002836033 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb2b806b4d0 0x7fb2b806d990 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb2c0038000 tx=0x7fb2c0032000 comp rx=0 tx=0).stop 2024-03-06T07:22:44.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.321+0000 7fb2d24af640 1 -- 172.21.15.83:0/2002836033 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2cc071a70 msgr2=0x7fb2cc1494a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.321+0000 7fb2d24af640 1 --2- 172.21.15.83:0/2002836033 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2cc071a70 0x7fb2cc1494a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc00c4d0 tx=0x7fb2bc00c990 comp rx=0 tx=0).stop 2024-03-06T07:22:44.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.321+0000 7fb2d24af640 1 -- 172.21.15.83:0/2002836033 shutdown_connections 2024-03-06T07:22:44.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.321+0000 7fb2d24af640 1 --2- 172.21.15.83:0/2002836033 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb2b806b4d0 0x7fb2b806d990 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.321+0000 7fb2d24af640 1 --2- 172.21.15.83:0/2002836033 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb2cc15ad90 0x7fb2cc0af2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.321+0000 7fb2d24af640 1 --2- 172.21.15.83:0/2002836033 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb2cc0b4110 0x7fb2cc0aeda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.321+0000 7fb2d24af640 1 --2- 172.21.15.83:0/2002836033 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2cc071a70 0x7fb2cc1494a0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.322+0000 7fb2d24af640 1 -- 172.21.15.83:0/2002836033 >> 172.21.15.83:0/2002836033 conn(0x7fb2cc0b2660 msgr2=0x7fb2cc155510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:44.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.322+0000 7fb2d24af640 1 -- 172.21.15.83:0/2002836033 shutdown_connections 2024-03-06T07:22:44.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.322+0000 7fb2d24af640 1 -- 172.21.15.83:0/2002836033 wait complete. 2024-03-06T07:22:44.335 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:44.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.441+0000 7f92257e9640 1 Processor -- start 2024-03-06T07:22:44.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.441+0000 7f92257e9640 1 -- start start 2024-03-06T07:22:44.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.442+0000 7f92257e9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92200b42e0 0x7f9220165680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.442+0000 7f92257e9640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f922015b260 con 0x7f9220071a70 2024-03-06T07:22:44.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.442+0000 7f92257e9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f922015b540 con 0x7f922015ad90 2024-03-06T07:22:44.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.442+0000 7f92257e9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f922015c020 con 0x7f92200b42e0 2024-03-06T07:22:44.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.442+0000 7f921effd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f9220071a70 0x7f9220071e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39226/0 (socket says 172.21.15.83:39226) 2024-03-06T07:22:44.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.442+0000 7f921effd640 1 -- 172.21.15.83:0/2614319306 learned_addr learned my addr 172.21.15.83:0/2614319306 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:44.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.442+0000 7f921dffb640 1 -- 172.21.15.83:0/2614319306 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1010105296 0 0) 0x7f922015b260 con 0x7f9220071a70 2024-03-06T07:22:44.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.442+0000 7f921dffb640 1 -- 172.21.15.83:0/2614319306 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f920c003520 con 0x7f9220071a70 2024-03-06T07:22:44.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.442+0000 7f921f7fe640 1 --2- 172.21.15.83:0/2614319306 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92200b42e0 0x7f9220165680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.443+0000 7f921dffb640 1 -- 172.21.15.83:0/2614319306 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4126825788 0 0) 0x7f920c003520 con 0x7f9220071a70 2024-03-06T07:22:44.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.443+0000 7f921dffb640 1 -- 172.21.15.83:0/2614319306 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92200b42e0 msgr2=0x7f9220165680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:44.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.443+0000 7f921dffb640 1 --2- 172.21.15.83:0/2614319306 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92200b42e0 0x7f9220165680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.443+0000 7f921dffb640 1 -- 172.21.15.83:0/2614319306 >> v1:172.21.15.183:6789/0 conn(0x7f922015ad90 legacy=0x7f92200b3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:44.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.443+0000 7f921dffb640 1 -- 172.21.15.83:0/2614319306 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f922015b820 con 0x7f9220071a70 2024-03-06T07:22:44.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.443+0000 7f921dffb640 1 -- 172.21.15.83:0/2614319306 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f9218001690 con 0x7f9220071a70 2024-03-06T07:22:44.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.443+0000 7f921dffb640 1 -- 172.21.15.83:0/2614319306 >> v1:172.21.15.83:6789/0 conn(0x7f9220071a70 legacy=0x7f9220071e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.443+0000 7f921dffb640 1 --2- 172.21.15.83:0/2614319306 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f920c0039e0 0x7f920c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.443+0000 7f921dffb640 1 --2- 172.21.15.83:0/2614319306 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f920c004800 0x7f920c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.443+0000 7f921dffb640 1 --2- 172.21.15.83:0/2614319306 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92200b42e0 0x7f920c005290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.443+0000 7f921dffb640 1 -- 172.21.15.83:0/2614319306 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f922015c020 con 0x7f920c004800 2024-03-06T07:22:44.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.443+0000 7f921dffb640 1 -- 172.21.15.83:0/2614319306 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f920c00bbb0 con 0x7f920c0039e0 2024-03-06T07:22:44.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.443+0000 7f921dffb640 1 -- 172.21.15.83:0/2614319306 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f920c00dfc0 con 0x7f92200b42e0 2024-03-06T07:22:44.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.443+0000 7f921e7fc640 1 --2- 172.21.15.83:0/2614319306 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f920c004800 0x7f920c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.443+0000 7f921effd640 1 --2- 172.21.15.83:0/2614319306 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f920c0039e0 0x7f920c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.443+0000 7f921f7fe640 1 --2- 172.21.15.83:0/2614319306 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92200b42e0 0x7f920c005290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.443+0000 7f921e7fc640 1 -- 172.21.15.83:0/2614319306 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92200b42e0 msgr2=0x7f920c005290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:44.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.443+0000 7f921e7fc640 1 --2- 172.21.15.83:0/2614319306 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92200b42e0 0x7f920c005290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.443+0000 7f921e7fc640 1 -- 172.21.15.83:0/2614319306 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f920c0039e0 msgr2=0x7f920c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:44.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.443+0000 7f921e7fc640 1 --2- 172.21.15.83:0/2614319306 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f920c0039e0 0x7f920c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.443+0000 7f921e7fc640 1 -- 172.21.15.83:0/2614319306 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f920c003520 con 0x7f920c004800 2024-03-06T07:22:44.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.444+0000 7f921e7fc640 1 --2- 172.21.15.83:0/2614319306 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f920c004800 0x7f920c004c80 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f9220070620 tx=0x7f920c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=119ee089e4d3d99e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.444+0000 7f921dffb640 1 -- 172.21.15.83:0/2614319306 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9208037070 con 0x7f920c004800 2024-03-06T07:22:44.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.444+0000 7f921dffb640 1 -- 172.21.15.83:0/2614319306 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9208032600 con 0x7f920c004800 2024-03-06T07:22:44.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.444+0000 7f921dffb640 1 -- 172.21.15.83:0/2614319306 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92080039a0 con 0x7f920c004800 2024-03-06T07:22:44.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.444+0000 7f92257e9640 1 -- 172.21.15.83:0/2614319306 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f920c004800 msgr2=0x7f920c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.444+0000 7f92257e9640 1 --2- 172.21.15.83:0/2614319306 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f920c004800 0x7f920c004c80 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f9220070620 tx=0x7f920c005830 comp rx=0 tx=0).stop 2024-03-06T07:22:44.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.444+0000 7f921f7fe640 1 -- 172.21.15.83:0/2614319306 reap_dead start 2024-03-06T07:22:44.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.444+0000 7f92257e9640 1 -- 172.21.15.83:0/2614319306 shutdown_connections 2024-03-06T07:22:44.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.444+0000 7f92257e9640 1 -- 172.21.15.83:0/2614319306 >> 172.21.15.83:0/2614319306 conn(0x7f92200b2660 msgr2=0x7f92200b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:44.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.445+0000 7f92257e9640 1 -- 172.21.15.83:0/2614319306 shutdown_connections 2024-03-06T07:22:44.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.445+0000 7f92257e9640 1 -- 172.21.15.83:0/2614319306 wait complete. 2024-03-06T07:22:44.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.446+0000 7f92257e9640 1 Processor -- start 2024-03-06T07:22:44.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.446+0000 7f92257e9640 1 -- start start 2024-03-06T07:22:44.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.447+0000 7f92257e9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f920c004800 0x7f92200b0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.447+0000 7f92257e9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9220153230 0x7f9220150510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.447+0000 7f92257e9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92200ae190 0x7f92200ae610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.447+0000 7f92257e9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f922015d220 con 0x7f92200ae190 2024-03-06T07:22:44.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.447+0000 7f92257e9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92200b4750 con 0x7f920c004800 2024-03-06T07:22:44.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.447+0000 7f92257e9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f922015d3a0 con 0x7f9220153230 2024-03-06T07:22:44.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.447+0000 7f921f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92200ae190 0x7f92200ae610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.447+0000 7f921f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92200ae190 0x7f92200ae610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53912/0 (socket says 172.21.15.83:53912) 2024-03-06T07:22:44.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.447+0000 7f921f7fe640 1 -- 172.21.15.83:0/2618141466 learned_addr learned my addr 172.21.15.83:0/2618141466 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:44.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.447+0000 7f921effd640 1 --2- 172.21.15.83:0/2618141466 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f920c004800 0x7f92200b0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.447+0000 7f921e7fc640 1 --2- 172.21.15.83:0/2618141466 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9220153230 0x7f9220150510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.447+0000 7f921f7fe640 1 -- 172.21.15.83:0/2618141466 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9220153230 msgr2=0x7f9220150510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.447+0000 7f921f7fe640 1 --2- 172.21.15.83:0/2618141466 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9220153230 0x7f9220150510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.447+0000 7f921f7fe640 1 -- 172.21.15.83:0/2618141466 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f920c004800 msgr2=0x7f92200b0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.447+0000 7f921f7fe640 1 --2- 172.21.15.83:0/2618141466 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f920c004800 0x7f92200b0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.447+0000 7f921f7fe640 1 -- 172.21.15.83:0/2618141466 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92200b1750 con 0x7f92200ae190 2024-03-06T07:22:44.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.447+0000 7f921effd640 1 --2- 172.21.15.83:0/2618141466 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f920c004800 0x7f92200b0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:44.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.447+0000 7f921f7fe640 1 --2- 172.21.15.83:0/2618141466 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92200ae190 0x7f92200ae610 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f922015b170 tx=0x7f9210007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.447+0000 7f92177fe640 1 -- 172.21.15.83:0/2618141466 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9210007c80 con 0x7f92200ae190 2024-03-06T07:22:44.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.447+0000 7f92177fe640 1 -- 172.21.15.83:0/2618141466 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f921001a070 con 0x7f92200ae190 2024-03-06T07:22:44.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.447+0000 7f92257e9640 1 -- 172.21.15.83:0/2618141466 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92200b1a30 con 0x7f92200ae190 2024-03-06T07:22:44.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.447+0000 7f92177fe640 1 -- 172.21.15.83:0/2618141466 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9210015640 con 0x7f92200ae190 2024-03-06T07:22:44.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.447+0000 7f92257e9640 1 -- 172.21.15.83:0/2618141466 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92200ce0f0 con 0x7f92200ae190 2024-03-06T07:22:44.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.448+0000 7f92257e9640 1 -- 172.21.15.83:0/2618141466 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92200ce2f0 con 0x7f92200ae190 2024-03-06T07:22:44.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.449+0000 7f92177fe640 1 -- 172.21.15.83:0/2618141466 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f92100157e0 con 0x7f92200ae190 2024-03-06T07:22:44.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.449+0000 7f92177fe640 1 --2- 172.21.15.83:0/2618141466 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f920006b500 0x7f920006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.449+0000 7f92177fe640 1 -- 172.21.15.83:0/2618141466 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5912+0+0 (secure 0 0 0) 0x7f921008aa10 con 0x7f92200ae190 2024-03-06T07:22:44.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.449+0000 7f921effd640 1 --2- 172.21.15.83:0/2618141466 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f920006b500 0x7f920006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.450+0000 7f921effd640 1 --2- 172.21.15.83:0/2618141466 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f920006b500 0x7f920006d9c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f922006f080 tx=0x7f9218009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.454+0000 7f92177fe640 1 -- 172.21.15.83:0/2618141466 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f92200ce2f0 con 0x7f92200ae190 2024-03-06T07:22:44.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.787+0000 7f92257e9640 1 -- 172.21.15.83:0/2618141466 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9220068ff0 con 0x7f92200ae190 2024-03-06T07:22:44.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.787+0000 7f92177fe640 1 -- 172.21.15.83:0/2618141466 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9210024020 con 0x7f92200ae190 2024-03-06T07:22:44.789 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 64 2024-03-06T07:22:44.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.789+0000 7f92257e9640 1 -- 172.21.15.83:0/2618141466 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f920006b500 msgr2=0x7f920006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.789+0000 7f92257e9640 1 --2- 172.21.15.83:0/2618141466 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f920006b500 0x7f920006d9c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f922006f080 tx=0x7f9218009040 comp rx=0 tx=0).stop 2024-03-06T07:22:44.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.789+0000 7f92257e9640 1 -- 172.21.15.83:0/2618141466 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92200ae190 msgr2=0x7f92200ae610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.789+0000 7f92257e9640 1 --2- 172.21.15.83:0/2618141466 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92200ae190 0x7f92200ae610 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f922015b170 tx=0x7f9210007700 comp rx=0 tx=0).stop 2024-03-06T07:22:44.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.789+0000 7f92257e9640 1 -- 172.21.15.83:0/2618141466 shutdown_connections 2024-03-06T07:22:44.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.789+0000 7f92257e9640 1 --2- 172.21.15.83:0/2618141466 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f920006b500 0x7f920006d9c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.790+0000 7f92257e9640 1 --2- 172.21.15.83:0/2618141466 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92200ae190 0x7f92200ae610 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.790+0000 7f92257e9640 1 --2- 172.21.15.83:0/2618141466 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9220153230 0x7f9220150510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.790+0000 7f92257e9640 1 --2- 172.21.15.83:0/2618141466 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f920c004800 0x7f92200b0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.790+0000 7f92257e9640 1 -- 172.21.15.83:0/2618141466 >> 172.21.15.83:0/2618141466 conn(0x7f92200b2660 msgr2=0x7f92200b53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:44.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.790+0000 7f92257e9640 1 -- 172.21.15.83:0/2618141466 shutdown_connections 2024-03-06T07:22:44.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.790+0000 7f92257e9640 1 -- 172.21.15.83:0/2618141466 wait complete. 2024-03-06T07:22:44.806 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:22:44.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.906+0000 7f8ab186b640 1 Processor -- start 2024-03-06T07:22:44.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.907+0000 7f8ab186b640 1 -- start start 2024-03-06T07:22:44.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.907+0000 7f8ab186b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f8aac1614a0 0x7f8aac06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.907+0000 7f8ab186b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8aac0b1a20 0x7f8aac0b5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.907+0000 7f8ab186b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aac0af4b0 con 0x7f8aac0b0270 2024-03-06T07:22:44.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.907+0000 7f8ab186b640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8aac0af790 con 0x7f8aac1614a0 2024-03-06T07:22:44.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.907+0000 7f8ab186b640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac0afa70 con 0x7f8aac0b1a20 2024-03-06T07:22:44.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.907+0000 7f8aaaffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f8aac1614a0 0x7f8aac06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.908+0000 7f8aaaffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f8aac1614a0 0x7f8aac06d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53924/0 (socket says 172.21.15.83:53924) 2024-03-06T07:22:44.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.908+0000 7f8aaaffd640 1 -- 172.21.15.83:0/3481888313 learned_addr learned my addr 172.21.15.83:0/3481888313 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:44.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.908+0000 7f8aab7fe640 1 --2- 172.21.15.83:0/3481888313 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8aac0b1a20 0x7f8aac0b5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.908+0000 7f8aaaffd640 1 -- 172.21.15.83:0/3481888313 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8aac0b1a20 msgr2=0x7f8aac0b5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:44.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.908+0000 7f8aaaffd640 1 --2- 172.21.15.83:0/3481888313 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8aac0b1a20 0x7f8aac0b5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.908+0000 7f8aaaffd640 1 -- 172.21.15.83:0/3481888313 >> v1:172.21.15.183:6789/0 conn(0x7f8aac0b0270 legacy=0x7f8aac0b1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:44.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.908+0000 7f8aaaffd640 1 -- 172.21.15.83:0/3481888313 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aac0afd50 con 0x7f8aac1614a0 2024-03-06T07:22:44.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.909+0000 7f8aaaffd640 1 --2- 172.21.15.83:0/3481888313 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8aac1614a0 0x7f8aac06d3c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f8aac0730b0 tx=0x7f8aa402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3bacdf60b6730a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.909+0000 7f8aa9ffb640 1 -- 172.21.15.83:0/3481888313 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa40315a0 con 0x7f8aac1614a0 2024-03-06T07:22:44.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.909+0000 7f8aa9ffb640 1 -- 172.21.15.83:0/3481888313 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aa4037070 con 0x7f8aac1614a0 2024-03-06T07:22:44.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.909+0000 7f8aa9ffb640 1 -- 172.21.15.83:0/3481888313 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa4031960 con 0x7f8aac1614a0 2024-03-06T07:22:44.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.909+0000 7f8ab186b640 1 -- 172.21.15.83:0/3481888313 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8aac1614a0 msgr2=0x7f8aac06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.909+0000 7f8ab186b640 1 --2- 172.21.15.83:0/3481888313 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8aac1614a0 0x7f8aac06d3c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f8aac0730b0 tx=0x7f8aa402b840 comp rx=0 tx=0).stop 2024-03-06T07:22:44.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.909+0000 7f8ab186b640 1 -- 172.21.15.83:0/3481888313 shutdown_connections 2024-03-06T07:22:44.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.909+0000 7f8ab186b640 1 --2- 172.21.15.83:0/3481888313 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8aac0b1a20 0x7f8aac0b5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.909+0000 7f8ab186b640 1 --2- 172.21.15.83:0/3481888313 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8aac1614a0 0x7f8aac06d3c0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.909+0000 7f8ab186b640 1 -- 172.21.15.83:0/3481888313 >> 172.21.15.83:0/3481888313 conn(0x7f8aac09a9f0 msgr2=0x7f8aac09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:44.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.909+0000 7f8ab186b640 1 -- 172.21.15.83:0/3481888313 shutdown_connections 2024-03-06T07:22:44.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.909+0000 7f8ab186b640 1 -- 172.21.15.83:0/3481888313 wait complete. 2024-03-06T07:22:44.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.910+0000 7f8ab186b640 1 Processor -- start 2024-03-06T07:22:44.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.911+0000 7f8ab186b640 1 -- start start 2024-03-06T07:22:44.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.911+0000 7f8ab186b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8aac0b0270 0x7f8aac0b3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.911+0000 7f8ab186b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8aac0b1a20 0x7f8aac15e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.911+0000 7f8ab186b640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8aac1614a0 0x7f8aac146180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.911+0000 7f8ab186b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac0b1e30 con 0x7f8aac0b0270 2024-03-06T07:22:44.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.911+0000 7f8ab186b640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac0b06e0 con 0x7f8aac1614a0 2024-03-06T07:22:44.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.911+0000 7f8ab186b640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac0b5a50 con 0x7f8aac0b1a20 2024-03-06T07:22:44.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.911+0000 7f8aaaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8aac0b0270 0x7f8aac0b3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.911+0000 7f8aaaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8aac0b0270 0x7f8aac0b3b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53934/0 (socket says 172.21.15.83:53934) 2024-03-06T07:22:44.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.911+0000 7f8aaaffd640 1 -- 172.21.15.83:0/1866804885 learned_addr learned my addr 172.21.15.83:0/1866804885 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:44.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.911+0000 7f8aab7fe640 1 --2- 172.21.15.83:0/1866804885 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8aac1614a0 0x7f8aac146180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.911+0000 7f8aaa7fc640 1 --2- 172.21.15.83:0/1866804885 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8aac0b1a20 0x7f8aac15e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.912+0000 7f8aaaffd640 1 -- 172.21.15.83:0/1866804885 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8aac0b1a20 msgr2=0x7f8aac15e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.912+0000 7f8aaaffd640 1 --2- 172.21.15.83:0/1866804885 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8aac0b1a20 0x7f8aac15e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.912+0000 7f8aaaffd640 1 -- 172.21.15.83:0/1866804885 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8aac1614a0 msgr2=0x7f8aac146180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.912+0000 7f8aaaffd640 1 --2- 172.21.15.83:0/1866804885 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8aac1614a0 0x7f8aac146180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.912+0000 7f8aaaffd640 1 -- 172.21.15.83:0/1866804885 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aac1f3e50 con 0x7f8aac0b0270 2024-03-06T07:22:44.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.912+0000 7f8aaaffd640 1 --2- 172.21.15.83:0/1866804885 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8aac0b0270 0x7f8aac0b3b00 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f8aa402b880 tx=0x7f8aa40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.912+0000 7f8a93fff640 1 -- 172.21.15.83:0/1866804885 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa4032630 con 0x7f8aac0b0270 2024-03-06T07:22:44.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.912+0000 7f8a93fff640 1 -- 172.21.15.83:0/1866804885 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aa40327f0 con 0x7f8aac0b0270 2024-03-06T07:22:44.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.912+0000 7f8ab186b640 1 -- 172.21.15.83:0/1866804885 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aac15ed50 con 0x7f8aac0b0270 2024-03-06T07:22:44.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.912+0000 7f8a93fff640 1 -- 172.21.15.83:0/1866804885 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa4032be0 con 0x7f8aac0b0270 2024-03-06T07:22:44.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.912+0000 7f8ab186b640 1 -- 172.21.15.83:0/1866804885 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aac1e6a80 con 0x7f8aac0b0270 2024-03-06T07:22:44.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.913+0000 7f8ab186b640 1 -- 172.21.15.83:0/1866804885 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aac1e6890 con 0x7f8aac0b0270 2024-03-06T07:22:44.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.914+0000 7f8a93fff640 1 -- 172.21.15.83:0/1866804885 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8aa404d3f0 con 0x7f8aac0b0270 2024-03-06T07:22:44.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.914+0000 7f8a93fff640 1 --2- 172.21.15.83:0/1866804885 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8a9806b4d0 0x7f8a9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.914+0000 7f8a93fff640 1 -- 172.21.15.83:0/1866804885 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5912+0+0 (secure 0 0 0) 0x7f8aa402f030 con 0x7f8aac0b0270 2024-03-06T07:22:44.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.914+0000 7f8aaa7fc640 1 --2- 172.21.15.83:0/1866804885 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8a9806b4d0 0x7f8a9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.914+0000 7f8aaa7fc640 1 --2- 172.21.15.83:0/1866804885 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8a9806b4d0 0x7f8a9806d990 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8aac0b4cd0 tx=0x7f8aa0003d90 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:44.918+0000 7f8a93fff640 1 -- 172.21.15.83:0/1866804885 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8aa4040070 con 0x7f8aac0b0270 2024-03-06T07:22:45.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.103+0000 7f8ab186b640 1 -- 172.21.15.83:0/1866804885 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f8aac06e8e0 con 0x7f8aac0b0270 2024-03-06T07:22:45.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.104+0000 7f8a93fff640 1 -- 172.21.15.83:0/1866804885 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f8aa407a270 con 0x7f8aac0b0270 2024-03-06T07:22:45.107 INFO:teuthology.orchestra.run.smithi083.stderr:listed 0 entries 2024-03-06T07:22:45.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.106+0000 7f8ab186b640 1 -- 172.21.15.83:0/1866804885 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8a9806b4d0 msgr2=0x7f8a9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.106+0000 7f8ab186b640 1 --2- 172.21.15.83:0/1866804885 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8a9806b4d0 0x7f8a9806d990 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8aac0b4cd0 tx=0x7f8aa0003d90 comp rx=0 tx=0).stop 2024-03-06T07:22:45.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.106+0000 7f8ab186b640 1 -- 172.21.15.83:0/1866804885 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8aac0b0270 msgr2=0x7f8aac0b3b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.106+0000 7f8ab186b640 1 --2- 172.21.15.83:0/1866804885 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8aac0b0270 0x7f8aac0b3b00 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f8aa402b880 tx=0x7f8aa40046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.106+0000 7f8ab186b640 1 -- 172.21.15.83:0/1866804885 shutdown_connections 2024-03-06T07:22:45.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.106+0000 7f8ab186b640 1 --2- 172.21.15.83:0/1866804885 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8a9806b4d0 0x7f8a9806d990 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.106+0000 7f8ab186b640 1 --2- 172.21.15.83:0/1866804885 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8aac1614a0 0x7f8aac146180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.106+0000 7f8ab186b640 1 --2- 172.21.15.83:0/1866804885 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8aac0b1a20 0x7f8aac15e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.106+0000 7f8ab186b640 1 --2- 172.21.15.83:0/1866804885 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8aac0b0270 0x7f8aac0b3b00 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.106+0000 7f8ab186b640 1 -- 172.21.15.83:0/1866804885 >> 172.21.15.83:0/1866804885 conn(0x7f8aac09a9f0 msgr2=0x7f8aac0aeb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:45.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.106+0000 7f8ab186b640 1 -- 172.21.15.83:0/1866804885 shutdown_connections 2024-03-06T07:22:45.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.106+0000 7f8ab186b640 1 -- 172.21.15.83:0/1866804885 wait complete. 2024-03-06T07:22:45.118 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:22:45.119 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:45.138 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:45.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.238+0000 7fa8b19e2640 1 Processor -- start 2024-03-06T07:22:45.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.238+0000 7fa8b19e2640 1 -- start start 2024-03-06T07:22:45.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.239+0000 7fa8b19e2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa8ac0d0a80 0x7fa8ac0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.239+0000 7fa8b19e2640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8ac0d25d0 con 0x7fa8ac0b4170 2024-03-06T07:22:45.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.239+0000 7fa8b19e2640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8ac0d2870 con 0x7fa8ac0d4760 2024-03-06T07:22:45.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.239+0000 7fa8b19e2640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8ac0d4bf0 con 0x7fa8ac0d0a80 2024-03-06T07:22:45.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.239+0000 7fa8ab7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa8ac0d0a80 0x7fa8ac0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.239+0000 7fa8aaffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fa8ac0b4170 0x7fa8ac0d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39240/0 (socket says 172.21.15.83:39240) 2024-03-06T07:22:45.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.239+0000 7fa8ab7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa8ac0d0a80 0x7fa8ac0b6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53944/0 (socket says 172.21.15.83:53944) 2024-03-06T07:22:45.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.239+0000 7fa8aaffd640 1 -- 172.21.15.83:0/2049566486 learned_addr learned my addr 172.21.15.83:0/2049566486 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:45.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.239+0000 7fa8ab7fe640 1 -- 172.21.15.83:0/2049566486 >> v1:172.21.15.183:6789/0 conn(0x7fa8ac0d4760 legacy=0x7fa8ac0cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:45.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.239+0000 7fa8ab7fe640 1 -- 172.21.15.83:0/2049566486 >> v1:172.21.15.83:6789/0 conn(0x7fa8ac0b4170 legacy=0x7fa8ac0d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.239+0000 7fa8ab7fe640 1 -- 172.21.15.83:0/2049566486 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8ac0d2b50 con 0x7fa8ac0d0a80 2024-03-06T07:22:45.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.239+0000 7fa8aa7fc640 1 -- 172.21.15.83:0/2049566486 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 254805269 0 0) 0x7fa8ac0d25d0 con 0x7fa8ac0b4170 2024-03-06T07:22:45.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.240+0000 7fa8ab7fe640 1 --2- 172.21.15.83:0/2049566486 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8ac0d0a80 0x7fa8ac0b6a90 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa8a4009d90 tx=0x7fa8a402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf148f434143c534 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.240+0000 7fa8aa7fc640 1 -- 172.21.15.83:0/2049566486 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a4039070 con 0x7fa8ac0d0a80 2024-03-06T07:22:45.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.240+0000 7fa8aa7fc640 1 -- 172.21.15.83:0/2049566486 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8a402cc60 con 0x7fa8ac0d0a80 2024-03-06T07:22:45.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.240+0000 7fa8aa7fc640 1 -- 172.21.15.83:0/2049566486 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a4034660 con 0x7fa8ac0d0a80 2024-03-06T07:22:45.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.241+0000 7fa8b19e2640 1 -- 172.21.15.83:0/2049566486 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8ac0d0a80 msgr2=0x7fa8ac0b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.241+0000 7fa8b19e2640 1 --2- 172.21.15.83:0/2049566486 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8ac0d0a80 0x7fa8ac0b6a90 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa8a4009d90 tx=0x7fa8a402beb0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.241+0000 7fa8b19e2640 1 -- 172.21.15.83:0/2049566486 shutdown_connections 2024-03-06T07:22:45.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.241+0000 7fa8b19e2640 1 --2- 172.21.15.83:0/2049566486 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8ac0d0a80 0x7fa8ac0b6a90 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.241+0000 7fa8b19e2640 1 -- 172.21.15.83:0/2049566486 >> 172.21.15.83:0/2049566486 conn(0x7fa8ac0d3c40 msgr2=0x7fa8ac0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:45.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.241+0000 7fa8b19e2640 1 -- 172.21.15.83:0/2049566486 shutdown_connections 2024-03-06T07:22:45.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.241+0000 7fa8b19e2640 1 -- 172.21.15.83:0/2049566486 wait complete. 2024-03-06T07:22:45.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.242+0000 7fa8b19e2640 1 Processor -- start 2024-03-06T07:22:45.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.242+0000 7fa8b19e2640 1 -- start start 2024-03-06T07:22:45.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.243+0000 7fa8b19e2640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8ac0b4170 0x7fa8ac0cc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.243+0000 7fa8b19e2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8ac0d0a80 0x7fa8ac15ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.243+0000 7fa8b19e2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8ac158220 0x7fa8ac0b1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.243+0000 7fa8b19e2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ac0b4dd0 con 0x7fa8ac0d0a80 2024-03-06T07:22:45.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.243+0000 7fa8b19e2640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ac0ce9e0 con 0x7fa8ac0b4170 2024-03-06T07:22:45.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.243+0000 7fa8b19e2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ac0b4f50 con 0x7fa8ac158220 2024-03-06T07:22:45.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.243+0000 7fa8a3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8ac0d0a80 0x7fa8ac15ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.243+0000 7fa8a3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8ac0d0a80 0x7fa8ac15ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53948/0 (socket says 172.21.15.83:53948) 2024-03-06T07:22:45.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.243+0000 7fa8a3fff640 1 -- 172.21.15.83:0/1556785636 learned_addr learned my addr 172.21.15.83:0/1556785636 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:45.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.243+0000 7fa8aaffd640 1 --2- 172.21.15.83:0/1556785636 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8ac0b4170 0x7fa8ac0cc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.243+0000 7fa8ab7fe640 1 --2- 172.21.15.83:0/1556785636 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8ac158220 0x7fa8ac0b1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.243+0000 7fa8a3fff640 1 -- 172.21.15.83:0/1556785636 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8ac158220 msgr2=0x7fa8ac0b1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.243+0000 7fa8a3fff640 1 --2- 172.21.15.83:0/1556785636 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8ac158220 0x7fa8ac0b1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.243+0000 7fa8a3fff640 1 -- 172.21.15.83:0/1556785636 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8ac0b4170 msgr2=0x7fa8ac0cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.243+0000 7fa8a3fff640 1 --2- 172.21.15.83:0/1556785636 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8ac0b4170 0x7fa8ac0cc9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.243+0000 7fa8a3fff640 1 -- 172.21.15.83:0/1556785636 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8ac0cd0c0 con 0x7fa8ac0d0a80 2024-03-06T07:22:45.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.243+0000 7fa8aaffd640 1 --2- 172.21.15.83:0/1556785636 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8ac0b4170 0x7fa8ac0cc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:45.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.243+0000 7fa8ab7fe640 1 --2- 172.21.15.83:0/1556785636 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8ac158220 0x7fa8ac0b1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:45.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.243+0000 7fa8a3fff640 1 --2- 172.21.15.83:0/1556785636 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8ac0d0a80 0x7fa8ac15ef20 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fa89400b3c0 tx=0x7fa89400b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.243+0000 7fa8a8ff9640 1 -- 172.21.15.83:0/1556785636 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa894011830 con 0x7fa8ac0d0a80 2024-03-06T07:22:45.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.243+0000 7fa8a8ff9640 1 -- 172.21.15.83:0/1556785636 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa894016070 con 0x7fa8ac0d0a80 2024-03-06T07:22:45.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.243+0000 7fa8b19e2640 1 -- 172.21.15.83:0/1556785636 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8ac0cd3a0 con 0x7fa8ac0d0a80 2024-03-06T07:22:45.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.243+0000 7fa8a8ff9640 1 -- 172.21.15.83:0/1556785636 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa894011c50 con 0x7fa8ac0d0a80 2024-03-06T07:22:45.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.243+0000 7fa8b19e2640 1 -- 172.21.15.83:0/1556785636 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8ac0b0c70 con 0x7fa8ac0d0a80 2024-03-06T07:22:45.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.245+0000 7fa8b19e2640 1 -- 172.21.15.83:0/1556785636 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8ac0b0e70 con 0x7fa8ac0d0a80 2024-03-06T07:22:45.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.245+0000 7fa8a8ff9640 1 -- 172.21.15.83:0/1556785636 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa89401b460 con 0x7fa8ac0d0a80 2024-03-06T07:22:45.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.245+0000 7fa8a8ff9640 1 --2- 172.21.15.83:0/1556785636 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa88c06b4d0 0x7fa88c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.245+0000 7fa8a8ff9640 1 -- 172.21.15.83:0/1556785636 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5912+0+0 (secure 0 0 0) 0x7fa89408a060 con 0x7fa8ac0d0a80 2024-03-06T07:22:45.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.245+0000 7fa8aaffd640 1 --2- 172.21.15.83:0/1556785636 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa88c06b4d0 0x7fa88c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.246+0000 7fa8aaffd640 1 --2- 172.21.15.83:0/1556785636 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa88c06b4d0 0x7fa88c06d990 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa89c000f60 tx=0x7fa89c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.250+0000 7fa8a8ff9640 1 -- 172.21.15.83:0/1556785636 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa8ac0b0e70 con 0x7fa8ac0d0a80 2024-03-06T07:22:45.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.457+0000 7fa8b19e2640 1 -- 172.21.15.83:0/1556785636 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8ac04f330 con 0x7fa8ac0d0a80 2024-03-06T07:22:45.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.458+0000 7fa8a8ff9640 1 -- 172.21.15.83:0/1556785636 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+17006 (secure 0 0 0) 0x7fa894081e20 con 0x7fa8ac0d0a80 2024-03-06T07:22:45.460 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:22:45.460 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":22,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:22:40.105898+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.318942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.254795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.1f","mappings":[{"from":5,"to":1}]},{"pgid":"3.29","mappings":[{"from":3,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:45.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.460+0000 7fa8b19e2640 1 -- 172.21.15.83:0/1556785636 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa88c06b4d0 msgr2=0x7fa88c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.460+0000 7fa8b19e2640 1 --2- 172.21.15.83:0/1556785636 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa88c06b4d0 0x7fa88c06d990 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa89c000f60 tx=0x7fa89c009040 comp rx=0 tx=0).stop 2024-03-06T07:22:45.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.460+0000 7fa8b19e2640 1 -- 172.21.15.83:0/1556785636 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8ac0d0a80 msgr2=0x7fa8ac15ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.460+0000 7fa8b19e2640 1 --2- 172.21.15.83:0/1556785636 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8ac0d0a80 0x7fa8ac15ef20 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fa89400b3c0 tx=0x7fa89400b880 comp rx=0 tx=0).stop 2024-03-06T07:22:45.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.460+0000 7fa8b19e2640 1 -- 172.21.15.83:0/1556785636 shutdown_connections 2024-03-06T07:22:45.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.460+0000 7fa8b19e2640 1 --2- 172.21.15.83:0/1556785636 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa88c06b4d0 0x7fa88c06d990 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.460+0000 7fa8b19e2640 1 --2- 172.21.15.83:0/1556785636 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8ac158220 0x7fa8ac0b1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.460+0000 7fa8b19e2640 1 --2- 172.21.15.83:0/1556785636 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8ac0d0a80 0x7fa8ac15ef20 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.460+0000 7fa8b19e2640 1 --2- 172.21.15.83:0/1556785636 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8ac0b4170 0x7fa8ac0cc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.460+0000 7fa8b19e2640 1 -- 172.21.15.83:0/1556785636 >> 172.21.15.83:0/1556785636 conn(0x7fa8ac0d3c40 msgr2=0x7fa8ac0b9550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:45.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.460+0000 7fa8b19e2640 1 -- 172.21.15.83:0/1556785636 shutdown_connections 2024-03-06T07:22:45.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.460+0000 7fa8b19e2640 1 -- 172.21.15.83:0/1556785636 wait complete. 2024-03-06T07:22:45.473 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:56.318942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:57.254795+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:45.473 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:45.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.575+0000 7efebda66640 1 Processor -- start 2024-03-06T07:22:45.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.575+0000 7efebda66640 1 -- start start 2024-03-06T07:22:45.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.576+0000 7efebda66640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7efeb80d0af0 0x7efeb80cce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.576+0000 7efebda66640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb80cd4b0 con 0x7efeb815c050 2024-03-06T07:22:45.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.576+0000 7efebda66640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb80cd790 con 0x7efeb806fd10 2024-03-06T07:22:45.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.576+0000 7efebda66640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efeb80cda70 con 0x7efeb80d0af0 2024-03-06T07:22:45.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.576+0000 7efeb77fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7efeb815c050 0x7efeb806f600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39246/0 (socket says 172.21.15.83:39246) 2024-03-06T07:22:45.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.576+0000 7efeb77fe640 1 -- 172.21.15.83:0/2237913475 learned_addr learned my addr 172.21.15.83:0/2237913475 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:45.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.577+0000 7efeb7fff640 1 --2- 172.21.15.83:0/2237913475 >> v2:172.21.15.183:3300/0 conn(0x7efeb80d0af0 0x7efeb80cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.577+0000 7efeb67fc640 1 -- 172.21.15.83:0/2237913475 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 420724460 0 0) 0x7efeb80cd4b0 con 0x7efeb815c050 2024-03-06T07:22:45.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.577+0000 7efeb67fc640 1 -- 172.21.15.83:0/2237913475 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efea0003520 con 0x7efeb815c050 2024-03-06T07:22:45.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.577+0000 7efeb67fc640 1 -- 172.21.15.83:0/2237913475 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1042687160 0 0) 0x7efea0003520 con 0x7efeb815c050 2024-03-06T07:22:45.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.577+0000 7efeb67fc640 1 -- 172.21.15.83:0/2237913475 >> v2:172.21.15.183:3300/0 conn(0x7efeb80d0af0 msgr2=0x7efeb80cce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:45.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.577+0000 7efeb67fc640 1 --2- 172.21.15.83:0/2237913475 >> v2:172.21.15.183:3300/0 conn(0x7efeb80d0af0 0x7efeb80cce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.577+0000 7efeb67fc640 1 -- 172.21.15.83:0/2237913475 >> v1:172.21.15.183:6789/0 conn(0x7efeb806fd10 legacy=0x7efeb80cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.577+0000 7efeb67fc640 1 -- 172.21.15.83:0/2237913475 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb80cdd50 con 0x7efeb815c050 2024-03-06T07:22:45.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.577+0000 7efeb67fc640 1 -- 172.21.15.83:0/2237913475 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7efea8001670 con 0x7efeb815c050 2024-03-06T07:22:45.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.577+0000 7efeb67fc640 1 -- 172.21.15.83:0/2237913475 >> v1:172.21.15.83:6789/0 conn(0x7efeb815c050 legacy=0x7efeb806f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.577+0000 7efeb67fc640 1 --2- 172.21.15.83:0/2237913475 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efea00039b0 0x7efea0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.577+0000 7efeb67fc640 1 --2- 172.21.15.83:0/2237913475 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efea0004800 0x7efea0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.577+0000 7efeb67fc640 1 --2- 172.21.15.83:0/2237913475 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efea00051c0 0x7efea000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.577+0000 7efeb67fc640 1 -- 172.21.15.83:0/2237913475 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb80cda70 con 0x7efea00051c0 2024-03-06T07:22:45.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.577+0000 7efeb67fc640 1 -- 172.21.15.83:0/2237913475 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea000c160 con 0x7efea00039b0 2024-03-06T07:22:45.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.577+0000 7efeb67fc640 1 -- 172.21.15.83:0/2237913475 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efea000c360 con 0x7efea0004800 2024-03-06T07:22:45.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.577+0000 7efeb7fff640 1 --2- 172.21.15.83:0/2237913475 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efea00051c0 0x7efea000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.577+0000 7efeb6ffd640 1 --2- 172.21.15.83:0/2237913475 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efea0004800 0x7efea0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.577+0000 7efeb77fe640 1 --2- 172.21.15.83:0/2237913475 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efea00039b0 0x7efea0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.577+0000 7efeb7fff640 1 -- 172.21.15.83:0/2237913475 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efea0004800 msgr2=0x7efea0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:45.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.577+0000 7efeb7fff640 1 --2- 172.21.15.83:0/2237913475 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efea0004800 0x7efea0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.577+0000 7efeb7fff640 1 -- 172.21.15.83:0/2237913475 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efea00039b0 msgr2=0x7efea0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:45.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.577+0000 7efeb7fff640 1 --2- 172.21.15.83:0/2237913475 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efea00039b0 0x7efea0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.577+0000 7efeb7fff640 1 -- 172.21.15.83:0/2237913475 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea0003520 con 0x7efea00051c0 2024-03-06T07:22:45.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.578+0000 7efeb7fff640 1 -- 172.21.15.83:0/2237913475 reap_dead start 2024-03-06T07:22:45.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.578+0000 7efeb7fff640 1 --2- 172.21.15.83:0/2237913475 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efea00051c0 0x7efea000b9c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7efeb0009f90 tx=0x7efea0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1fc7f6c88340e36 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.578+0000 7efeb67fc640 1 -- 172.21.15.83:0/2237913475 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb0039070 con 0x7efea00051c0 2024-03-06T07:22:45.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.578+0000 7efeb67fc640 1 -- 172.21.15.83:0/2237913475 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeb002cc20 con 0x7efea00051c0 2024-03-06T07:22:45.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.578+0000 7efeb67fc640 1 -- 172.21.15.83:0/2237913475 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb0009c10 con 0x7efea00051c0 2024-03-06T07:22:45.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.579+0000 7efebda66640 1 -- 172.21.15.83:0/2237913475 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efea00051c0 msgr2=0x7efea000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.579+0000 7efebda66640 1 --2- 172.21.15.83:0/2237913475 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efea00051c0 0x7efea000b9c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7efeb0009f90 tx=0x7efea0005830 comp rx=0 tx=0).stop 2024-03-06T07:22:45.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.579+0000 7efebda66640 1 -- 172.21.15.83:0/2237913475 shutdown_connections 2024-03-06T07:22:45.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.579+0000 7efebda66640 1 --2- 172.21.15.83:0/2237913475 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efea00051c0 0x7efea000b9c0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.579+0000 7efebda66640 1 -- 172.21.15.83:0/2237913475 >> 172.21.15.83:0/2237913475 conn(0x7efeb80b3800 msgr2=0x7efeb80b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:45.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.579+0000 7efebda66640 1 -- 172.21.15.83:0/2237913475 shutdown_connections 2024-03-06T07:22:45.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.579+0000 7efebda66640 1 -- 172.21.15.83:0/2237913475 wait complete. 2024-03-06T07:22:45.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.580+0000 7efebda66640 1 Processor -- start 2024-03-06T07:22:45.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.580+0000 7efebda66640 1 -- start start 2024-03-06T07:22:45.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.581+0000 7efebda66640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efea00051c0 0x7efeb8154650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.581+0000 7efebda66640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efeb809edf0 0x7efeb809c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.581+0000 7efebda66640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efeb80cc760 0x7efeb80ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.581+0000 7efebda66640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb806f990 con 0x7efeb80cc760 2024-03-06T07:22:45.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.581+0000 7efebda66640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb806f810 con 0x7efeb809edf0 2024-03-06T07:22:45.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.581+0000 7efebda66640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeb806fb10 con 0x7efea00051c0 2024-03-06T07:22:45.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.581+0000 7efeb7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efeb80cc760 0x7efeb80ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.581+0000 7efeb7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efeb80cc760 0x7efeb80ccbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53956/0 (socket says 172.21.15.83:53956) 2024-03-06T07:22:45.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.581+0000 7efeb7fff640 1 -- 172.21.15.83:0/3547018226 learned_addr learned my addr 172.21.15.83:0/3547018226 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:45.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.581+0000 7efeb6ffd640 1 --2- 172.21.15.83:0/3547018226 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efeb809edf0 0x7efeb809c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.581+0000 7efeb77fe640 1 --2- 172.21.15.83:0/3547018226 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efea00051c0 0x7efeb8154650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.581+0000 7efeb7fff640 1 -- 172.21.15.83:0/3547018226 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efea00051c0 msgr2=0x7efeb8154650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.581+0000 7efeb7fff640 1 --2- 172.21.15.83:0/3547018226 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efea00051c0 0x7efeb8154650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.581+0000 7efeb7fff640 1 -- 172.21.15.83:0/3547018226 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efeb809edf0 msgr2=0x7efeb809c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.581+0000 7efeb7fff640 1 --2- 172.21.15.83:0/3547018226 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efeb809edf0 0x7efeb809c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.581+0000 7efeb7fff640 1 -- 172.21.15.83:0/3547018226 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb8154b90 con 0x7efeb80cc760 2024-03-06T07:22:45.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.581+0000 7efeb7fff640 1 --2- 172.21.15.83:0/3547018226 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efeb80cc760 0x7efeb80ccbf0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7efeb000e230 tx=0x7efeb0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.582+0000 7efe9ffff640 1 -- 172.21.15.83:0/3547018226 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb0039040 con 0x7efeb80cc760 2024-03-06T07:22:45.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.582+0000 7efe9ffff640 1 -- 172.21.15.83:0/3547018226 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeb0045070 con 0x7efeb80cc760 2024-03-06T07:22:45.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.582+0000 7efebda66640 1 -- 172.21.15.83:0/3547018226 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeb809f2b0 con 0x7efeb80cc760 2024-03-06T07:22:45.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.582+0000 7efe9ffff640 1 -- 172.21.15.83:0/3547018226 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb003e6c0 con 0x7efeb80cc760 2024-03-06T07:22:45.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.582+0000 7efebda66640 1 -- 172.21.15.83:0/3547018226 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeb809f870 con 0x7efeb80cc760 2024-03-06T07:22:45.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.583+0000 7efebda66640 1 -- 172.21.15.83:0/3547018226 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeb809fe60 con 0x7efeb80cc760 2024-03-06T07:22:45.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.583+0000 7efe9ffff640 1 -- 172.21.15.83:0/3547018226 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efeb0005d10 con 0x7efeb80cc760 2024-03-06T07:22:45.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.584+0000 7efe9ffff640 1 --2- 172.21.15.83:0/3547018226 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efea406b4d0 0x7efea406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.584+0000 7efeb77fe640 1 --2- 172.21.15.83:0/3547018226 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efea406b4d0 0x7efea406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.584+0000 7efe9ffff640 1 -- 172.21.15.83:0/3547018226 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5912+0+0 (secure 0 0 0) 0x7efeb00ac1e0 con 0x7efeb80cc760 2024-03-06T07:22:45.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.584+0000 7efeb77fe640 1 --2- 172.21.15.83:0/3547018226 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efea406b4d0 0x7efea406d990 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7efea8001eb0 tx=0x7efea8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.588+0000 7efe9ffff640 1 -- 172.21.15.83:0/3547018226 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efeb007af90 con 0x7efeb80cc760 2024-03-06T07:22:45.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.764+0000 7efebda66640 1 -- 172.21.15.83:0/3547018226 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efeb8068ff0 con 0x7efeb80cc760 2024-03-06T07:22:45.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.764+0000 7efe9ffff640 1 -- 172.21.15.83:0/3547018226 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7efeb007a890 con 0x7efeb80cc760 2024-03-06T07:22:45.766 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:22:45.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.766+0000 7efebda66640 1 -- 172.21.15.83:0/3547018226 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efea406b4d0 msgr2=0x7efea406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.766+0000 7efebda66640 1 --2- 172.21.15.83:0/3547018226 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efea406b4d0 0x7efea406d990 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7efea8001eb0 tx=0x7efea8009040 comp rx=0 tx=0).stop 2024-03-06T07:22:45.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.766+0000 7efebda66640 1 -- 172.21.15.83:0/3547018226 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efeb80cc760 msgr2=0x7efeb80ccbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.766+0000 7efebda66640 1 --2- 172.21.15.83:0/3547018226 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efeb80cc760 0x7efeb80ccbf0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7efeb000e230 tx=0x7efeb0009750 comp rx=0 tx=0).stop 2024-03-06T07:22:45.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.766+0000 7efebda66640 1 -- 172.21.15.83:0/3547018226 shutdown_connections 2024-03-06T07:22:45.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.766+0000 7efebda66640 1 --2- 172.21.15.83:0/3547018226 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efea406b4d0 0x7efea406d990 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.766+0000 7efebda66640 1 --2- 172.21.15.83:0/3547018226 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efeb80cc760 0x7efeb80ccbf0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.766+0000 7efebda66640 1 --2- 172.21.15.83:0/3547018226 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efeb809edf0 0x7efeb809c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.766+0000 7efebda66640 1 --2- 172.21.15.83:0/3547018226 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efea00051c0 0x7efeb8154650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.766+0000 7efebda66640 1 -- 172.21.15.83:0/3547018226 >> 172.21.15.83:0/3547018226 conn(0x7efeb80b3800 msgr2=0x7efeb815a860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:45.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.766+0000 7efebda66640 1 -- 172.21.15.83:0/3547018226 shutdown_connections 2024-03-06T07:22:45.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.766+0000 7efebda66640 1 -- 172.21.15.83:0/3547018226 wait complete. 2024-03-06T07:22:45.779 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:45.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.881+0000 7f0a3ab5e640 1 Processor -- start 2024-03-06T07:22:45.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.881+0000 7f0a3ab5e640 1 -- start start 2024-03-06T07:22:45.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.882+0000 7f0a3ab5e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0a3415ecd0 0x7f0a340cae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.882+0000 7f0a3ab5e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a340cfa50 con 0x7f0a340d14a0 2024-03-06T07:22:45.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.882+0000 7f0a3ab5e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a340b45c0 con 0x7f0a340d10c0 2024-03-06T07:22:45.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.882+0000 7f0a3ab5e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a3415c050 con 0x7f0a3415ecd0 2024-03-06T07:22:45.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.882+0000 7f0a390fb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0a3415ecd0 0x7f0a340cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.882+0000 7f0a390fb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0a3415ecd0 0x7f0a340cae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53970/0 (socket says 172.21.15.83:53970) 2024-03-06T07:22:45.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.882+0000 7f0a390fb640 1 -- 172.21.15.83:0/3463759835 learned_addr learned my addr 172.21.15.83:0/3463759835 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:45.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.882+0000 7f0a33fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f0a340d14a0 0x7f0a3415e5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39256/0 (socket says 172.21.15.83:39256) 2024-03-06T07:22:45.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.883+0000 7f0a390fb640 1 -- 172.21.15.83:0/3463759835 >> v1:172.21.15.183:6789/0 conn(0x7f0a340d10c0 legacy=0x7f0a340d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.883+0000 7f0a390fb640 1 -- 172.21.15.83:0/3463759835 >> v1:172.21.15.83:6789/0 conn(0x7f0a340d14a0 legacy=0x7f0a3415e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.883+0000 7f0a337fe640 1 -- 172.21.15.83:0/3463759835 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2783226431 0 0) 0x7f0a340cfa50 con 0x7f0a340d14a0 2024-03-06T07:22:45.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.883+0000 7f0a390fb640 1 -- 172.21.15.83:0/3463759835 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a3415f160 con 0x7f0a3415ecd0 2024-03-06T07:22:45.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.884+0000 7f0a390fb640 1 --2- 172.21.15.83:0/3463759835 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a3415ecd0 0x7f0a340cae80 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c009e50 tx=0x7f0a2c02c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7e11dca59550c6f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.884+0000 7f0a337fe640 1 -- 172.21.15.83:0/3463759835 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a2c02cbb0 con 0x7f0a3415ecd0 2024-03-06T07:22:45.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.884+0000 7f0a337fe640 1 -- 172.21.15.83:0/3463759835 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a2c02cd50 con 0x7f0a3415ecd0 2024-03-06T07:22:45.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.884+0000 7f0a337fe640 1 -- 172.21.15.83:0/3463759835 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a2c034610 con 0x7f0a3415ecd0 2024-03-06T07:22:45.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.884+0000 7f0a3ab5e640 1 -- 172.21.15.83:0/3463759835 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a3415ecd0 msgr2=0x7f0a340cae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.884+0000 7f0a3ab5e640 1 --2- 172.21.15.83:0/3463759835 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a3415ecd0 0x7f0a340cae80 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c009e50 tx=0x7f0a2c02c010 comp rx=0 tx=0).stop 2024-03-06T07:22:45.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.885+0000 7f0a3ab5e640 1 -- 172.21.15.83:0/3463759835 shutdown_connections 2024-03-06T07:22:45.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.885+0000 7f0a3ab5e640 1 --2- 172.21.15.83:0/3463759835 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a3415ecd0 0x7f0a340cae80 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.885+0000 7f0a3ab5e640 1 -- 172.21.15.83:0/3463759835 >> 172.21.15.83:0/3463759835 conn(0x7f0a340b3800 msgr2=0x7f0a340b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:45.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.885+0000 7f0a3ab5e640 1 -- 172.21.15.83:0/3463759835 shutdown_connections 2024-03-06T07:22:45.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.885+0000 7f0a3ab5e640 1 -- 172.21.15.83:0/3463759835 wait complete. 2024-03-06T07:22:45.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.886+0000 7f0a3ab5e640 1 Processor -- start 2024-03-06T07:22:45.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.886+0000 7f0a3ab5e640 1 -- start start 2024-03-06T07:22:45.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.887+0000 7f0a3ab5e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a340d10c0 0x7f0a34152540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.887+0000 7f0a3ab5e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0a340d14a0 0x7f0a3406c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.887+0000 7f0a3ab5e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a3406c5e0 0x7f0a340b66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.887+0000 7f0a3ab5e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a340cfcf0 con 0x7f0a340d10c0 2024-03-06T07:22:45.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.887+0000 7f0a3ab5e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a340d0060 con 0x7f0a3406c5e0 2024-03-06T07:22:45.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.887+0000 7f0a3ab5e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a340cb3c0 con 0x7f0a340d14a0 2024-03-06T07:22:45.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.887+0000 7f0a388fa640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a340d10c0 0x7f0a34152540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.887+0000 7f0a388fa640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a340d10c0 0x7f0a34152540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53974/0 (socket says 172.21.15.83:53974) 2024-03-06T07:22:45.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.887+0000 7f0a388fa640 1 -- 172.21.15.83:0/1187377253 learned_addr learned my addr 172.21.15.83:0/1187377253 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:45.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.887+0000 7f0a33fff640 1 --2- 172.21.15.83:0/1187377253 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0a340d14a0 0x7f0a3406c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.887+0000 7f0a390fb640 1 --2- 172.21.15.83:0/1187377253 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a3406c5e0 0x7f0a340b66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.887+0000 7f0a388fa640 1 -- 172.21.15.83:0/1187377253 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0a340d14a0 msgr2=0x7f0a3406c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.887+0000 7f0a388fa640 1 --2- 172.21.15.83:0/1187377253 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0a340d14a0 0x7f0a3406c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.887+0000 7f0a388fa640 1 -- 172.21.15.83:0/1187377253 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a3406c5e0 msgr2=0x7f0a340b66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.887+0000 7f0a388fa640 1 --2- 172.21.15.83:0/1187377253 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a3406c5e0 0x7f0a340b66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.887+0000 7f0a388fa640 1 -- 172.21.15.83:0/1187377253 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a340b7a00 con 0x7f0a340d10c0 2024-03-06T07:22:45.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.887+0000 7f0a388fa640 1 --2- 172.21.15.83:0/1187377253 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a340d10c0 0x7f0a34152540 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f0a2800c480 tx=0x7f0a2800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.887+0000 7f0a317fa640 1 -- 172.21.15.83:0/1187377253 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a280052d0 con 0x7f0a340d10c0 2024-03-06T07:22:45.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.887+0000 7f0a317fa640 1 -- 172.21.15.83:0/1187377253 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a28005470 con 0x7f0a340d10c0 2024-03-06T07:22:45.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.887+0000 7f0a3ab5e640 1 -- 172.21.15.83:0/1187377253 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a340b7440 con 0x7f0a340d10c0 2024-03-06T07:22:45.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.887+0000 7f0a317fa640 1 -- 172.21.15.83:0/1187377253 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a28012870 con 0x7f0a340d10c0 2024-03-06T07:22:45.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.887+0000 7f0a3ab5e640 1 -- 172.21.15.83:0/1187377253 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a340cc220 con 0x7f0a340d10c0 2024-03-06T07:22:45.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.888+0000 7f0a3ab5e640 1 -- 172.21.15.83:0/1187377253 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a3409fe60 con 0x7f0a340d10c0 2024-03-06T07:22:45.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.889+0000 7f0a317fa640 1 -- 172.21.15.83:0/1187377253 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0a28012a10 con 0x7f0a340d10c0 2024-03-06T07:22:45.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.889+0000 7f0a317fa640 1 --2- 172.21.15.83:0/1187377253 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0a1806b4d0 0x7f0a1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.889+0000 7f0a317fa640 1 -- 172.21.15.83:0/1187377253 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5912+0+0 (secure 0 0 0) 0x7f0a28089ed0 con 0x7f0a340d10c0 2024-03-06T07:22:45.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.889+0000 7f0a33fff640 1 --2- 172.21.15.83:0/1187377253 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0a1806b4d0 0x7f0a1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.890+0000 7f0a33fff640 1 --2- 172.21.15.83:0/1187377253 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0a1806b4d0 0x7f0a1806d990 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0a3409f6c0 tx=0x7f0a20007b90 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:45.893+0000 7f0a317fa640 1 -- 172.21.15.83:0/1187377253 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0a28058d10 con 0x7f0a340d10c0 2024-03-06T07:22:46.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.169+0000 7f0a3ab5e640 1 -- 172.21.15.83:0/1187377253 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0a34068ff0 con 0x7f0a340d10c0 2024-03-06T07:22:46.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.169+0000 7f0a317fa640 1 -- 172.21.15.83:0/1187377253 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0a28058610 con 0x7f0a340d10c0 2024-03-06T07:22:46.171 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-03-06T07:22:46.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.171+0000 7f0a3ab5e640 1 -- 172.21.15.83:0/1187377253 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0a1806b4d0 msgr2=0x7f0a1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.171+0000 7f0a3ab5e640 1 --2- 172.21.15.83:0/1187377253 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0a1806b4d0 0x7f0a1806d990 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0a3409f6c0 tx=0x7f0a20007b90 comp rx=0 tx=0).stop 2024-03-06T07:22:46.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.171+0000 7f0a3ab5e640 1 -- 172.21.15.83:0/1187377253 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a340d10c0 msgr2=0x7f0a34152540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.171+0000 7f0a3ab5e640 1 --2- 172.21.15.83:0/1187377253 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a340d10c0 0x7f0a34152540 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f0a2800c480 tx=0x7f0a2800c940 comp rx=0 tx=0).stop 2024-03-06T07:22:46.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.171+0000 7f0a3ab5e640 1 -- 172.21.15.83:0/1187377253 shutdown_connections 2024-03-06T07:22:46.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.171+0000 7f0a3ab5e640 1 --2- 172.21.15.83:0/1187377253 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0a1806b4d0 0x7f0a1806d990 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.171+0000 7f0a3ab5e640 1 --2- 172.21.15.83:0/1187377253 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a3406c5e0 0x7f0a340b66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.171+0000 7f0a3ab5e640 1 --2- 172.21.15.83:0/1187377253 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0a340d14a0 0x7f0a3406c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.171+0000 7f0a3ab5e640 1 --2- 172.21.15.83:0/1187377253 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a340d10c0 0x7f0a34152540 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.171+0000 7f0a3ab5e640 1 -- 172.21.15.83:0/1187377253 >> 172.21.15.83:0/1187377253 conn(0x7f0a340b3800 msgr2=0x7f0a3415e510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:46.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.171+0000 7f0a3ab5e640 1 -- 172.21.15.83:0/1187377253 shutdown_connections 2024-03-06T07:22:46.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.171+0000 7f0a3ab5e640 1 -- 172.21.15.83:0/1187377253 wait complete. 2024-03-06T07:22:46.184 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:46.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.291+0000 7f0980aeb640 1 Processor -- start 2024-03-06T07:22:46.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.291+0000 7f0980aeb640 1 -- start start 2024-03-06T07:22:46.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.292+0000 7f0980aeb640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f097c15ee10 0x7f097c168d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.292+0000 7f0980aeb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f097c0cea90 con 0x7f097c15bbb0 2024-03-06T07:22:46.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.292+0000 7f0980aeb640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f097c161bc0 con 0x7f097c06b6c0 2024-03-06T07:22:46.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.292+0000 7f0980aeb640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f097c161970 con 0x7f097c15ee10 2024-03-06T07:22:46.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.292+0000 7f097a59c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f097c15bbb0 0x7f097c06afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39272/0 (socket says 172.21.15.83:39272) 2024-03-06T07:22:46.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.292+0000 7f097a59c640 1 -- 172.21.15.83:0/2982194065 learned_addr learned my addr 172.21.15.83:0/2982194065 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:46.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.292+0000 7f0979d9b640 1 -- 172.21.15.83:0/2982194065 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3725974354 0 0) 0x7f096c001110 con 0x7f097c15bbb0 2024-03-06T07:22:46.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.292+0000 7f0979d9b640 1 -- 172.21.15.83:0/2982194065 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0968003520 con 0x7f097c15bbb0 2024-03-06T07:22:46.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.292+0000 7f097ad9d640 1 --2- 172.21.15.83:0/2982194065 >> v2:172.21.15.183:3300/0 conn(0x7f097c15ee10 0x7f097c168d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.293+0000 7f0979d9b640 1 -- 172.21.15.83:0/2982194065 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 912216963 0 0) 0x7f0968003520 con 0x7f097c15bbb0 2024-03-06T07:22:46.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.293+0000 7f0979d9b640 1 -- 172.21.15.83:0/2982194065 >> v2:172.21.15.183:3300/0 conn(0x7f097c15ee10 msgr2=0x7f097c168d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:46.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.293+0000 7f0979d9b640 1 --2- 172.21.15.83:0/2982194065 >> v2:172.21.15.183:3300/0 conn(0x7f097c15ee10 0x7f097c168d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.293+0000 7f0979d9b640 1 -- 172.21.15.83:0/2982194065 >> v1:172.21.15.183:6789/0 conn(0x7f097c06b6c0 legacy=0x7f097c15e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.293+0000 7f0979d9b640 1 -- 172.21.15.83:0/2982194065 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f097c161ea0 con 0x7f097c15bbb0 2024-03-06T07:22:46.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.293+0000 7f0979d9b640 1 -- 172.21.15.83:0/2982194065 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f096c001840 con 0x7f097c15bbb0 2024-03-06T07:22:46.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.293+0000 7f0979d9b640 1 -- 172.21.15.83:0/2982194065 >> v1:172.21.15.83:6789/0 conn(0x7f097c15bbb0 legacy=0x7f097c06afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.293+0000 7f0979d9b640 1 --2- 172.21.15.83:0/2982194065 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f09680039b0 0x7f0968003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.293+0000 7f0979d9b640 1 --2- 172.21.15.83:0/2982194065 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0968004800 0x7f0968004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.293+0000 7f0979d9b640 1 --2- 172.21.15.83:0/2982194065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09680051c0 0x7f096800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.293+0000 7f0979d9b640 1 -- 172.21.15.83:0/2982194065 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f097c161970 con 0x7f09680051c0 2024-03-06T07:22:46.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.293+0000 7f0979d9b640 1 -- 172.21.15.83:0/2982194065 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f096800c110 con 0x7f0968004800 2024-03-06T07:22:46.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.293+0000 7f0979d9b640 1 -- 172.21.15.83:0/2982194065 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f096800c310 con 0x7f09680039b0 2024-03-06T07:22:46.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.293+0000 7f097ad9d640 1 --2- 172.21.15.83:0/2982194065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09680051c0 0x7f096800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.293+0000 7f0973fff640 1 --2- 172.21.15.83:0/2982194065 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0968004800 0x7f0968004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.293+0000 7f097a59c640 1 --2- 172.21.15.83:0/2982194065 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f09680039b0 0x7f0968003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.293+0000 7f097ad9d640 1 -- 172.21.15.83:0/2982194065 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f09680039b0 msgr2=0x7f0968003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:46.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.293+0000 7f097ad9d640 1 --2- 172.21.15.83:0/2982194065 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f09680039b0 0x7f0968003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.293+0000 7f097ad9d640 1 -- 172.21.15.83:0/2982194065 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0968004800 msgr2=0x7f0968004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:46.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.293+0000 7f097ad9d640 1 --2- 172.21.15.83:0/2982194065 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0968004800 0x7f0968004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.293+0000 7f097ad9d640 1 -- 172.21.15.83:0/2982194065 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0968003520 con 0x7f09680051c0 2024-03-06T07:22:46.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.294+0000 7f097ad9d640 1 -- 172.21.15.83:0/2982194065 reap_dead start 2024-03-06T07:22:46.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.294+0000 7f097ad9d640 1 --2- 172.21.15.83:0/2982194065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09680051c0 0x7f096800b9c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f0974007360 tx=0x7f0968005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53db88a4946309aa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.294+0000 7f0979d9b640 1 -- 172.21.15.83:0/2982194065 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0974039070 con 0x7f09680051c0 2024-03-06T07:22:46.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.294+0000 7f0979d9b640 1 -- 172.21.15.83:0/2982194065 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f097402ccf0 con 0x7f09680051c0 2024-03-06T07:22:46.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.294+0000 7f0979d9b640 1 -- 172.21.15.83:0/2982194065 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0974009bc0 con 0x7f09680051c0 2024-03-06T07:22:46.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.294+0000 7f0980aeb640 1 -- 172.21.15.83:0/2982194065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09680051c0 msgr2=0x7f096800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.294+0000 7f0980aeb640 1 --2- 172.21.15.83:0/2982194065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09680051c0 0x7f096800b9c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f0974007360 tx=0x7f0968005830 comp rx=0 tx=0).stop 2024-03-06T07:22:46.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.294+0000 7f0980aeb640 1 -- 172.21.15.83:0/2982194065 shutdown_connections 2024-03-06T07:22:46.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.294+0000 7f0980aeb640 1 --2- 172.21.15.83:0/2982194065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09680051c0 0x7f096800b9c0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.294+0000 7f0980aeb640 1 -- 172.21.15.83:0/2982194065 >> 172.21.15.83:0/2982194065 conn(0x7f097c0b2660 msgr2=0x7f097c0b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:46.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.295+0000 7f0980aeb640 1 -- 172.21.15.83:0/2982194065 shutdown_connections 2024-03-06T07:22:46.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.295+0000 7f0980aeb640 1 -- 172.21.15.83:0/2982194065 wait complete. 2024-03-06T07:22:46.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.296+0000 7f0980aeb640 1 Processor -- start 2024-03-06T07:22:46.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.296+0000 7f0980aeb640 1 -- start start 2024-03-06T07:22:46.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.296+0000 7f0980aeb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f09680051c0 0x7f097c1464a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.296+0000 7f0980aeb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f097c15a320 0x7f097c15a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.297+0000 7f0980aeb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f097c155ae0 0x7f097c155f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.297+0000 7f0980aeb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f097c15e700 con 0x7f097c155ae0 2024-03-06T07:22:46.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.297+0000 7f0980aeb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f097c1624f0 con 0x7f097c15a320 2024-03-06T07:22:46.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.297+0000 7f0980aeb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f097c15e880 con 0x7f09680051c0 2024-03-06T07:22:46.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.297+0000 7f097ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f097c155ae0 0x7f097c155f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.297+0000 7f097ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f097c155ae0 0x7f097c155f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53980/0 (socket says 172.21.15.83:53980) 2024-03-06T07:22:46.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.297+0000 7f097ad9d640 1 -- 172.21.15.83:0/2143963995 learned_addr learned my addr 172.21.15.83:0/2143963995 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:46.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.297+0000 7f0973fff640 1 --2- 172.21.15.83:0/2143963995 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f097c15a320 0x7f097c15a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.297+0000 7f097a59c640 1 --2- 172.21.15.83:0/2143963995 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f09680051c0 0x7f097c1464a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.297+0000 7f097ad9d640 1 -- 172.21.15.83:0/2143963995 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f09680051c0 msgr2=0x7f097c1464a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.297+0000 7f097ad9d640 1 --2- 172.21.15.83:0/2143963995 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f09680051c0 0x7f097c1464a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.297+0000 7f097ad9d640 1 -- 172.21.15.83:0/2143963995 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f097c15a320 msgr2=0x7f097c15a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.297+0000 7f097ad9d640 1 --2- 172.21.15.83:0/2143963995 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f097c15a320 0x7f097c15a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.297+0000 7f097ad9d640 1 -- 172.21.15.83:0/2143963995 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f097c0b3750 con 0x7f097c155ae0 2024-03-06T07:22:46.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.297+0000 7f097ad9d640 1 --2- 172.21.15.83:0/2143963995 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f097c155ae0 0x7f097c155f60 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f0974007480 tx=0x7f0974004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.297+0000 7f0972ffd640 1 -- 172.21.15.83:0/2143963995 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0974039040 con 0x7f097c155ae0 2024-03-06T07:22:46.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.297+0000 7f0972ffd640 1 -- 172.21.15.83:0/2143963995 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0974045070 con 0x7f097c155ae0 2024-03-06T07:22:46.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.297+0000 7f0980aeb640 1 -- 172.21.15.83:0/2143963995 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f097c15f5d0 con 0x7f097c155ae0 2024-03-06T07:22:46.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.297+0000 7f0972ffd640 1 -- 172.21.15.83:0/2143963995 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f097403f420 con 0x7f097c155ae0 2024-03-06T07:22:46.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.297+0000 7f0980aeb640 1 -- 172.21.15.83:0/2143963995 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f097c15fb90 con 0x7f097c155ae0 2024-03-06T07:22:46.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.298+0000 7f0980aeb640 1 -- 172.21.15.83:0/2143963995 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f097c0b22a0 con 0x7f097c155ae0 2024-03-06T07:22:46.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.299+0000 7f0972ffd640 1 -- 172.21.15.83:0/2143963995 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f097403f5c0 con 0x7f097c155ae0 2024-03-06T07:22:46.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.299+0000 7f0972ffd640 1 --2- 172.21.15.83:0/2143963995 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f095c06b4d0 0x7f095c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.299+0000 7f0972ffd640 1 -- 172.21.15.83:0/2143963995 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5912+0+0 (secure 0 0 0) 0x7f09740ac500 con 0x7f097c155ae0 2024-03-06T07:22:46.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.299+0000 7f097a59c640 1 --2- 172.21.15.83:0/2143963995 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f095c06b4d0 0x7f095c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.299+0000 7f097a59c640 1 --2- 172.21.15.83:0/2143963995 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f095c06b4d0 0x7f095c06d990 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f096c0012f0 tx=0x7f096c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.304+0000 7f0972ffd640 1 -- 172.21.15.83:0/2143963995 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f097c0b22a0 con 0x7f097c155ae0 2024-03-06T07:22:46.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.476+0000 7f0980aeb640 1 -- 172.21.15.83:0/2143963995 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f097c0733b0 con 0x7f097c155ae0 2024-03-06T07:22:46.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.476+0000 7f0972ffd640 1 -- 172.21.15.83:0/2143963995 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f097407b3d0 con 0x7f097c155ae0 2024-03-06T07:22:46.478 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 64 2024-03-06T07:22:46.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.478+0000 7f0980aeb640 1 -- 172.21.15.83:0/2143963995 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f095c06b4d0 msgr2=0x7f095c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.478+0000 7f0980aeb640 1 --2- 172.21.15.83:0/2143963995 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f095c06b4d0 0x7f095c06d990 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f096c0012f0 tx=0x7f096c009040 comp rx=0 tx=0).stop 2024-03-06T07:22:46.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.478+0000 7f0980aeb640 1 -- 172.21.15.83:0/2143963995 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f097c155ae0 msgr2=0x7f097c155f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.478+0000 7f0980aeb640 1 --2- 172.21.15.83:0/2143963995 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f097c155ae0 0x7f097c155f60 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f0974007480 tx=0x7f0974004ab0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.478+0000 7f0980aeb640 1 -- 172.21.15.83:0/2143963995 shutdown_connections 2024-03-06T07:22:46.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.478+0000 7f0980aeb640 1 --2- 172.21.15.83:0/2143963995 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f095c06b4d0 0x7f095c06d990 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.478+0000 7f0980aeb640 1 --2- 172.21.15.83:0/2143963995 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f097c155ae0 0x7f097c155f60 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.478+0000 7f0980aeb640 1 --2- 172.21.15.83:0/2143963995 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f097c15a320 0x7f097c15a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.478+0000 7f0980aeb640 1 --2- 172.21.15.83:0/2143963995 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f09680051c0 0x7f097c1464a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.478+0000 7f0980aeb640 1 -- 172.21.15.83:0/2143963995 >> 172.21.15.83:0/2143963995 conn(0x7f097c0b2660 msgr2=0x7f097c157140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:46.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.478+0000 7f0980aeb640 1 -- 172.21.15.83:0/2143963995 shutdown_connections 2024-03-06T07:22:46.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.478+0000 7f0980aeb640 1 -- 172.21.15.83:0/2143963995 wait complete. 2024-03-06T07:22:46.491 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:22:46.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.594+0000 7f6782084640 1 Processor -- start 2024-03-06T07:22:46.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.594+0000 7f6782084640 1 -- start start 2024-03-06T07:22:46.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.595+0000 7f6782084640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f677c0b4460 0x7f677c0d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.595+0000 7f6782084640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f677c162970 0x7f677c0b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.595+0000 7f6782084640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f677c161cf0 0x7f677c162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.595+0000 7f6782084640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f677c161b00 con 0x7f677c0b4460 2024-03-06T07:22:46.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.595+0000 7f6782084640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f677c0cecc0 con 0x7f677c161cf0 2024-03-06T07:22:46.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.595+0000 7f6782084640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f677c0d22c0 con 0x7f677c162970 2024-03-06T07:22:46.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.595+0000 7f677b7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f677c0b4460 0x7f677c0d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.595+0000 7f677b7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f677c0b4460 0x7f677c0d4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36490/0 (socket says 172.21.15.83:36490) 2024-03-06T07:22:46.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.595+0000 7f677b7fe640 1 -- 172.21.15.83:0/3904203839 learned_addr learned my addr 172.21.15.83:0/3904203839 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:46.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.595+0000 7f677b7fe640 1 -- 172.21.15.83:0/3904203839 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f677c162970 msgr2=0x7f677c0b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:46.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.595+0000 7f677b7fe640 1 --2- 172.21.15.83:0/3904203839 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f677c162970 0x7f677c0b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.595+0000 7f677b7fe640 1 -- 172.21.15.83:0/3904203839 >> v2:172.21.15.183:3300/0 conn(0x7f677c161cf0 msgr2=0x7f677c162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:46.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.595+0000 7f677bfff640 1 --2- 172.21.15.83:0/3904203839 >> v2:172.21.15.183:3300/0 conn(0x7f677c161cf0 0x7f677c162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.595+0000 7f677b7fe640 1 --2- 172.21.15.83:0/3904203839 >> v2:172.21.15.183:3300/0 conn(0x7f677c161cf0 0x7f677c162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.595+0000 7f677b7fe640 1 -- 172.21.15.83:0/3904203839 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f677c0d25a0 con 0x7f677c0b4460 2024-03-06T07:22:46.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.595+0000 7f677bfff640 1 --2- 172.21.15.83:0/3904203839 >> v2:172.21.15.183:3300/0 conn(0x7f677c161cf0 0x7f677c162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:46.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.596+0000 7f677b7fe640 1 --2- 172.21.15.83:0/3904203839 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f677c0b4460 0x7f677c0d4800 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f677c0ce2c0 tx=0x7f676c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de75fd3aaeb3d8bc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.596+0000 7f677a7fc640 1 -- 172.21.15.83:0/3904203839 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f676c035020 con 0x7f677c0b4460 2024-03-06T07:22:46.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.596+0000 7f677a7fc640 1 -- 172.21.15.83:0/3904203839 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f676c031870 con 0x7f677c0b4460 2024-03-06T07:22:46.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.596+0000 7f677a7fc640 1 -- 172.21.15.83:0/3904203839 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f676c031c40 con 0x7f677c0b4460 2024-03-06T07:22:46.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.596+0000 7f6782084640 1 -- 172.21.15.83:0/3904203839 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f677c0b4460 msgr2=0x7f677c0d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.596+0000 7f6782084640 1 --2- 172.21.15.83:0/3904203839 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f677c0b4460 0x7f677c0d4800 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f677c0ce2c0 tx=0x7f676c02b910 comp rx=0 tx=0).stop 2024-03-06T07:22:46.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.597+0000 7f6782084640 1 -- 172.21.15.83:0/3904203839 shutdown_connections 2024-03-06T07:22:46.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.597+0000 7f6782084640 1 --2- 172.21.15.83:0/3904203839 >> v2:172.21.15.183:3300/0 conn(0x7f677c161cf0 0x7f677c162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.597+0000 7f6782084640 1 --2- 172.21.15.83:0/3904203839 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f677c162970 0x7f677c0b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.597+0000 7f6782084640 1 --2- 172.21.15.83:0/3904203839 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f677c0b4460 0x7f677c0d4800 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.597+0000 7f6782084640 1 -- 172.21.15.83:0/3904203839 >> 172.21.15.83:0/3904203839 conn(0x7f677c0d3ce0 msgr2=0x7f677c0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:46.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.597+0000 7f6782084640 1 -- 172.21.15.83:0/3904203839 shutdown_connections 2024-03-06T07:22:46.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.597+0000 7f6782084640 1 -- 172.21.15.83:0/3904203839 wait complete. 2024-03-06T07:22:46.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.598+0000 7f6782084640 1 Processor -- start 2024-03-06T07:22:46.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.598+0000 7f6782084640 1 -- start start 2024-03-06T07:22:46.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.599+0000 7f6782084640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f677c0b4460 0x7f677c0c94d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.599+0000 7f6782084640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f677c161cf0 0x7f677c1f4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.599+0000 7f6782084640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f677c162970 0x7f677c0b1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.599+0000 7f6782084640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f677c0ce950 con 0x7f677c0b4460 2024-03-06T07:22:46.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.599+0000 7f6782084640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f677c0d2b60 con 0x7f677c162970 2024-03-06T07:22:46.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.599+0000 7f6782084640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f677c0b3ac0 con 0x7f677c161cf0 2024-03-06T07:22:46.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.599+0000 7f677b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f677c0b4460 0x7f677c0c94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.599+0000 7f677b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f677c0b4460 0x7f677c0c94d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36496/0 (socket says 172.21.15.83:36496) 2024-03-06T07:22:46.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.599+0000 7f677b7fe640 1 -- 172.21.15.83:0/748614526 learned_addr learned my addr 172.21.15.83:0/748614526 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:46.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.599+0000 7f677bfff640 1 --2- 172.21.15.83:0/748614526 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f677c162970 0x7f677c0b1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.599+0000 7f677affd640 1 --2- 172.21.15.83:0/748614526 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f677c161cf0 0x7f677c1f4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.599+0000 7f677b7fe640 1 -- 172.21.15.83:0/748614526 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f677c161cf0 msgr2=0x7f677c1f4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.599+0000 7f677b7fe640 1 --2- 172.21.15.83:0/748614526 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f677c161cf0 0x7f677c1f4220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.599+0000 7f677b7fe640 1 -- 172.21.15.83:0/748614526 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f677c162970 msgr2=0x7f677c0b1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.599+0000 7f677b7fe640 1 --2- 172.21.15.83:0/748614526 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f677c162970 0x7f677c0b1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.599+0000 7f677b7fe640 1 -- 172.21.15.83:0/748614526 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f677c1f4760 con 0x7f677c0b4460 2024-03-06T07:22:46.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.599+0000 7f677bfff640 1 --2- 172.21.15.83:0/748614526 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f677c162970 0x7f677c0b1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:46.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.599+0000 7f677b7fe640 1 --2- 172.21.15.83:0/748614526 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f677c0b4460 0x7f677c0c94d0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f676c000c00 tx=0x7f676c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.599+0000 7f6763fff640 1 -- 172.21.15.83:0/748614526 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f676c040070 con 0x7f677c0b4460 2024-03-06T07:22:46.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.599+0000 7f6763fff640 1 -- 172.21.15.83:0/748614526 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f676c03a420 con 0x7f677c0b4460 2024-03-06T07:22:46.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.599+0000 7f6782084640 1 -- 172.21.15.83:0/748614526 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f677c1f4a30 con 0x7f677c0b4460 2024-03-06T07:22:46.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.599+0000 7f6763fff640 1 -- 172.21.15.83:0/748614526 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f676c035040 con 0x7f677c0b4460 2024-03-06T07:22:46.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.599+0000 7f6782084640 1 -- 172.21.15.83:0/748614526 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f677c1f4ff0 con 0x7f677c0b4460 2024-03-06T07:22:46.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.600+0000 7f6782084640 1 -- 172.21.15.83:0/748614526 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f677c04f330 con 0x7f677c0b4460 2024-03-06T07:22:46.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.601+0000 7f6763fff640 1 -- 172.21.15.83:0/748614526 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f676c038020 con 0x7f677c0b4460 2024-03-06T07:22:46.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.602+0000 7f6763fff640 1 --2- 172.21.15.83:0/748614526 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f676806b4d0 0x7f676806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.602+0000 7f6763fff640 1 -- 172.21.15.83:0/748614526 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5912+0+0 (secure 0 0 0) 0x7f676c0ac200 con 0x7f677c0b4460 2024-03-06T07:22:46.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.602+0000 7f677affd640 1 --2- 172.21.15.83:0/748614526 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f676806b4d0 0x7f676806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.602+0000 7f677affd640 1 --2- 172.21.15.83:0/748614526 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f676806b4d0 0x7f676806d990 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f677c0cded0 tx=0x7f6770008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.606+0000 7f6763fff640 1 -- 172.21.15.83:0/748614526 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f676c07b040 con 0x7f677c0b4460 2024-03-06T07:22:46.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.930+0000 7f6782084640 1 -- 172.21.15.83:0/748614526 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f677c06c800 con 0x7f677c0b4460 2024-03-06T07:22:46.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.930+0000 7f6763fff640 1 -- 172.21.15.83:0/748614526 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f676c07a940 con 0x7f677c0b4460 2024-03-06T07:22:46.932 INFO:teuthology.orchestra.run.smithi083.stderr:listed 0 entries 2024-03-06T07:22:46.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.933+0000 7f6782084640 1 -- 172.21.15.83:0/748614526 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f676806b4d0 msgr2=0x7f676806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.933+0000 7f6782084640 1 --2- 172.21.15.83:0/748614526 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f676806b4d0 0x7f676806d990 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f677c0cded0 tx=0x7f6770008040 comp rx=0 tx=0).stop 2024-03-06T07:22:46.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.933+0000 7f6782084640 1 -- 172.21.15.83:0/748614526 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f677c0b4460 msgr2=0x7f677c0c94d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.933+0000 7f6782084640 1 --2- 172.21.15.83:0/748614526 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f677c0b4460 0x7f677c0c94d0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f676c000c00 tx=0x7f676c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.933+0000 7f6782084640 1 -- 172.21.15.83:0/748614526 shutdown_connections 2024-03-06T07:22:46.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.933+0000 7f6782084640 1 --2- 172.21.15.83:0/748614526 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f676806b4d0 0x7f676806d990 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.933+0000 7f6782084640 1 --2- 172.21.15.83:0/748614526 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f677c162970 0x7f677c0b1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.933+0000 7f6782084640 1 --2- 172.21.15.83:0/748614526 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f677c161cf0 0x7f677c1f4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.933+0000 7f6782084640 1 --2- 172.21.15.83:0/748614526 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f677c0b4460 0x7f677c0c94d0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.933+0000 7f6782084640 1 -- 172.21.15.83:0/748614526 >> 172.21.15.83:0/748614526 conn(0x7f677c0d3ce0 msgr2=0x7f677c06ca20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:46.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.933+0000 7f6782084640 1 -- 172.21.15.83:0/748614526 shutdown_connections 2024-03-06T07:22:46.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:46.933+0000 7f6782084640 1 -- 172.21.15.83:0/748614526 wait complete. 2024-03-06T07:22:46.945 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi183.front.sepia.ceph.com... 2024-03-06T07:22:46.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:46.946 DEBUG:teuthology.orchestra.run.smithi183:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-03-06T07:22:46.979 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:46.979 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi183.front.sepia.ceph.com 2024-03-06T07:22:46.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:46.979 DEBUG:teuthology.orchestra.run.smithi183:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:22:47.037 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:22:47.037 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:47.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.137+0000 7f462b935640 1 Processor -- start 2024-03-06T07:22:47.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.137+0000 7f462b935640 1 -- start start 2024-03-06T07:22:47.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.139+0000 7f462b935640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f462415dfe0 0x7f46240cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.139+0000 7f462b935640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f46240cf830 0x7f46240d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.139+0000 7f462b935640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46240b3430 con 0x7f46240d27f0 2024-03-06T07:22:47.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.139+0000 7f462b935640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46240d2c50 con 0x7f46240cf830 2024-03-06T07:22:47.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.139+0000 7f462b935640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46240b31e0 con 0x7f462415dfe0 2024-03-06T07:22:47.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.139+0000 7f4628ed0640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f46240cf830 0x7f46240d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.139+0000 7f46296d1640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f462415dfe0 0x7f46240cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.139+0000 7f4628ed0640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f46240cf830 0x7f46240d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:38056/0 (socket says 172.21.15.83:38056) 2024-03-06T07:22:47.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.139+0000 7f46296d1640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f462415dfe0 0x7f46240cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:35198/0 (socket says 172.21.15.83:35198) 2024-03-06T07:22:47.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.139+0000 7f4629ed2640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f46240d27f0 0x7f46240d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:35836/0 (socket says 172.21.15.83:35836) 2024-03-06T07:22:47.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.139+0000 7f4628ed0640 1 -- 172.21.15.83:0/4245851567 learned_addr learned my addr 172.21.15.83:0/4245851567 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:47.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.139+0000 7f4628ed0640 1 -- 172.21.15.83:0/4245851567 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f462415dfe0 msgr2=0x7f46240cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:47.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.139+0000 7f4628ed0640 1 --2- 172.21.15.83:0/4245851567 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f462415dfe0 0x7f46240cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.139+0000 7f4628ed0640 1 -- 172.21.15.83:0/4245851567 >> v1:172.21.15.183:6789/0 conn(0x7f46240d27f0 legacy=0x7f46240d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.139+0000 7f4628ed0640 1 -- 172.21.15.83:0/4245851567 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46240b3710 con 0x7f46240cf830 2024-03-06T07:22:47.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.139+0000 7f46296d1640 1 --2- 172.21.15.83:0/4245851567 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f462415dfe0 0x7f46240cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:47.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.140+0000 7f4628ed0640 1 --2- 172.21.15.83:0/4245851567 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46240cf830 0x7f46240d22b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4614009870 tx=0x7f461402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9b70d185aa124dd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.140+0000 7f461bfff640 1 -- 172.21.15.83:0/4245851567 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4614034070 con 0x7f46240cf830 2024-03-06T07:22:47.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.140+0000 7f461bfff640 1 -- 172.21.15.83:0/4245851567 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4614030840 con 0x7f46240cf830 2024-03-06T07:22:47.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.140+0000 7f461bfff640 1 -- 172.21.15.83:0/4245851567 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4614030c10 con 0x7f46240cf830 2024-03-06T07:22:47.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.141+0000 7f462b935640 1 -- 172.21.15.83:0/4245851567 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46240cf830 msgr2=0x7f46240d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.141+0000 7f462b935640 1 --2- 172.21.15.83:0/4245851567 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46240cf830 0x7f46240d22b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4614009870 tx=0x7f461402b980 comp rx=0 tx=0).stop 2024-03-06T07:22:47.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.141+0000 7f462b935640 1 -- 172.21.15.83:0/4245851567 shutdown_connections 2024-03-06T07:22:47.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.141+0000 7f462b935640 1 --2- 172.21.15.83:0/4245851567 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46240cf830 0x7f46240d22b0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.141+0000 7f462b935640 1 --2- 172.21.15.83:0/4245851567 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f462415dfe0 0x7f46240cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.141+0000 7f462b935640 1 -- 172.21.15.83:0/4245851567 >> 172.21.15.83:0/4245851567 conn(0x7f46240d3cd0 msgr2=0x7f46240d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:47.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.141+0000 7f462b935640 1 -- 172.21.15.83:0/4245851567 shutdown_connections 2024-03-06T07:22:47.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.141+0000 7f462b935640 1 -- 172.21.15.83:0/4245851567 wait complete. 2024-03-06T07:22:47.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.143+0000 7f462b935640 1 Processor -- start 2024-03-06T07:22:47.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.143+0000 7f462b935640 1 -- start start 2024-03-06T07:22:47.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.144+0000 7f462b935640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46240cf830 0x7f46241f7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.144+0000 7f462b935640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46240d27f0 0x7f4624155210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.144+0000 7f462b935640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f462415dfe0 0x7f46240b2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.144+0000 7f462b935640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46240d1c70 con 0x7f46240cf830 2024-03-06T07:22:47.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.144+0000 7f462b935640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46240b3d60 con 0x7f462415dfe0 2024-03-06T07:22:47.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.144+0000 7f462b935640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46240d1df0 con 0x7f46240d27f0 2024-03-06T07:22:47.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.144+0000 7f46296d1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46240cf830 0x7f46241f7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.144+0000 7f46296d1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46240cf830 0x7f46241f7930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36498/0 (socket says 172.21.15.83:36498) 2024-03-06T07:22:47.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.144+0000 7f46296d1640 1 -- 172.21.15.83:0/2594284461 learned_addr learned my addr 172.21.15.83:0/2594284461 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:47.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.144+0000 7f4629ed2640 1 --2- 172.21.15.83:0/2594284461 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f462415dfe0 0x7f46240b2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.144+0000 7f4628ed0640 1 --2- 172.21.15.83:0/2594284461 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46240d27f0 0x7f4624155210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.145+0000 7f46296d1640 1 -- 172.21.15.83:0/2594284461 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46240d27f0 msgr2=0x7f4624155210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.145+0000 7f46296d1640 1 --2- 172.21.15.83:0/2594284461 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46240d27f0 0x7f4624155210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.145+0000 7f46296d1640 1 -- 172.21.15.83:0/2594284461 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f462415dfe0 msgr2=0x7f46240b2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.145+0000 7f46296d1640 1 --2- 172.21.15.83:0/2594284461 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f462415dfe0 0x7f46240b2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.145+0000 7f46296d1640 1 -- 172.21.15.83:0/2594284461 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46241f7e70 con 0x7f46240cf830 2024-03-06T07:22:47.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.145+0000 7f4629ed2640 1 --2- 172.21.15.83:0/2594284461 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f462415dfe0 0x7f46240b2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:47.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.145+0000 7f46296d1640 1 --2- 172.21.15.83:0/2594284461 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46240cf830 0x7f46241f7930 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f462000e990 tx=0x7f462000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.145+0000 7f4619ffb640 1 -- 172.21.15.83:0/2594284461 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4620017070 con 0x7f46240cf830 2024-03-06T07:22:47.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.145+0000 7f4619ffb640 1 -- 172.21.15.83:0/2594284461 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f462000c9e0 con 0x7f46240cf830 2024-03-06T07:22:47.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.145+0000 7f462b935640 1 -- 172.21.15.83:0/2594284461 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4624160620 con 0x7f46240cf830 2024-03-06T07:22:47.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.145+0000 7f4619ffb640 1 -- 172.21.15.83:0/2594284461 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f462000cdb0 con 0x7f46240cf830 2024-03-06T07:22:47.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.145+0000 7f462b935640 1 -- 172.21.15.83:0/2594284461 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46240b6ce0 con 0x7f46240cf830 2024-03-06T07:22:47.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.146+0000 7f462b935640 1 -- 172.21.15.83:0/2594284461 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f462404f330 con 0x7f46240cf830 2024-03-06T07:22:47.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.147+0000 7f4619ffb640 1 -- 172.21.15.83:0/2594284461 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f462001a070 con 0x7f46240cf830 2024-03-06T07:22:47.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.147+0000 7f4619ffb640 1 --2- 172.21.15.83:0/2594284461 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f460806b4d0 0x7f460806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.147+0000 7f4619ffb640 1 -- 172.21.15.83:0/2594284461 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5912+0+0 (secure 0 0 0) 0x7f462008a1b0 con 0x7f46240cf830 2024-03-06T07:22:47.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.147+0000 7f4628ed0640 1 --2- 172.21.15.83:0/2594284461 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f460806b4d0 0x7f460806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.148+0000 7f4628ed0640 1 --2- 172.21.15.83:0/2594284461 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f460806b4d0 0x7f460806d990 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4614036000 tx=0x7f461403b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.151+0000 7f4619ffb640 1 -- 172.21.15.83:0/2594284461 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f46241f7e70 con 0x7f46240cf830 2024-03-06T07:22:47.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.360+0000 7f462b935640 1 -- 172.21.15.83:0/2594284461 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f46240cea90 con 0x7f46240cf830 2024-03-06T07:22:47.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.360+0000 7f4619ffb640 1 -- 172.21.15.83:0/2594284461 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4974 (secure 0 0 0) 0x7f4620058f70 con 0x7f46240cf830 2024-03-06T07:22:47.362 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:22:47.362 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/4249608353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":4249608353},{"type":"v1","addr":"172.21.15.83:6836","nonce":4249608353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/714452530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":714452530},{"type":"v1","addr":"172.21.15.183:6835","nonce":714452530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4282,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/2723013431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2723013431},{"type":"v1","addr":"172.21.15.183:6834","nonce":2723013431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:58.286994+0000","modified":"2024-03-06T07:22:00.307284+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/1597344199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1597344199},{"type":"v1","addr":"172.21.15.83:6837","nonce":1597344199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4287,"qdb_cluster":[4287]},"id":1}]} 2024-03-06T07:22:47.363 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 6 2024-03-06T07:22:47.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.362+0000 7f462b935640 1 -- 172.21.15.83:0/2594284461 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f460806b4d0 msgr2=0x7f460806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.362+0000 7f462b935640 1 --2- 172.21.15.83:0/2594284461 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f460806b4d0 0x7f460806d990 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4614036000 tx=0x7f461403b040 comp rx=0 tx=0).stop 2024-03-06T07:22:47.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.362+0000 7f462b935640 1 -- 172.21.15.83:0/2594284461 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46240cf830 msgr2=0x7f46241f7930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.362+0000 7f462b935640 1 --2- 172.21.15.83:0/2594284461 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46240cf830 0x7f46241f7930 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f462000e990 tx=0x7f462000ee50 comp rx=0 tx=0).stop 2024-03-06T07:22:47.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.362+0000 7f462b935640 1 -- 172.21.15.83:0/2594284461 shutdown_connections 2024-03-06T07:22:47.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.362+0000 7f462b935640 1 --2- 172.21.15.83:0/2594284461 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f460806b4d0 0x7f460806d990 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.362+0000 7f462b935640 1 --2- 172.21.15.83:0/2594284461 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f462415dfe0 0x7f46240b2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.362+0000 7f462b935640 1 --2- 172.21.15.83:0/2594284461 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46240d27f0 0x7f4624155210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.362+0000 7f462b935640 1 --2- 172.21.15.83:0/2594284461 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46240cf830 0x7f46241f7930 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.362+0000 7f462b935640 1 -- 172.21.15.83:0/2594284461 >> 172.21.15.83:0/2594284461 conn(0x7f46240d3cd0 msgr2=0x7f46240cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:47.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.362+0000 7f462b935640 1 -- 172.21.15.83:0/2594284461 shutdown_connections 2024-03-06T07:22:47.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.362+0000 7f462b935640 1 -- 172.21.15.83:0/2594284461 wait complete. 2024-03-06T07:22:47.376 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:47.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.474+0000 7fda25916640 1 Processor -- start 2024-03-06T07:22:47.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.474+0000 7fda25916640 1 -- start start 2024-03-06T07:22:47.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.475+0000 7fda25916640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fda2015bb20 0x7fda200d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.475+0000 7fda25916640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fda200cd8c0 0x7fda200d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.475+0000 7fda25916640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda20161980 con 0x7fda200cfff0 2024-03-06T07:22:47.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.475+0000 7fda25916640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda20161c60 con 0x7fda2015bb20 2024-03-06T07:22:47.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.475+0000 7fda25916640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda20161f40 con 0x7fda200cd8c0 2024-03-06T07:22:47.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.475+0000 7fda167fc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fda200cfff0 0x7fda200ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:47108/0 (socket says 172.21.15.83:47108) 2024-03-06T07:22:47.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.475+0000 7fda167fc640 1 -- 172.21.15.83:0/3006682355 learned_addr learned my addr 172.21.15.83:0/3006682355 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:47.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.476+0000 7fda1effd640 1 --2- 172.21.15.83:0/3006682355 >> v2:172.21.15.183:3300/0 conn(0x7fda2015bb20 0x7fda200d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.476+0000 7fda1f7fe640 1 --2- 172.21.15.83:0/3006682355 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fda200cd8c0 0x7fda200d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.476+0000 7fda1e7fc640 1 -- 172.21.15.83:0/3006682355 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3922068418 0 0) 0x7fda20161980 con 0x7fda200cfff0 2024-03-06T07:22:47.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.476+0000 7fda1e7fc640 1 -- 172.21.15.83:0/3006682355 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda0c003520 con 0x7fda200cfff0 2024-03-06T07:22:47.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.476+0000 7fda1e7fc640 1 -- 172.21.15.83:0/3006682355 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2871174562 0 0) 0x7fda0c003520 con 0x7fda200cfff0 2024-03-06T07:22:47.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.476+0000 7fda1effd640 1 -- 172.21.15.83:0/3006682355 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fda200cd8c0 msgr2=0x7fda200d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:47.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.476+0000 7fda1effd640 1 --2- 172.21.15.83:0/3006682355 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fda200cd8c0 0x7fda200d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.476+0000 7fda1effd640 1 -- 172.21.15.83:0/3006682355 >> v1:172.21.15.83:6789/0 conn(0x7fda200cfff0 legacy=0x7fda200ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.476+0000 7fda1effd640 1 -- 172.21.15.83:0/3006682355 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda20162220 con 0x7fda2015bb20 2024-03-06T07:22:47.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.476+0000 7fda1f7fe640 1 --2- 172.21.15.83:0/3006682355 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fda200cd8c0 0x7fda200d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:47.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.477+0000 7fda1effd640 1 --2- 172.21.15.83:0/3006682355 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fda2015bb20 0x7fda200d1df0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fda100097a0 tx=0x7fda1002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23787e65e9832185 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.477+0000 7fda1e7fc640 1 -- 172.21.15.83:0/3006682355 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda10037070 con 0x7fda2015bb20 2024-03-06T07:22:47.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.477+0000 7fda1e7fc640 1 -- 172.21.15.83:0/3006682355 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda10030510 con 0x7fda2015bb20 2024-03-06T07:22:47.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.477+0000 7fda1e7fc640 1 -- 172.21.15.83:0/3006682355 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda08001670 con 0x7fda2015bb20 2024-03-06T07:22:47.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.478+0000 7fda25916640 1 -- 172.21.15.83:0/3006682355 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fda2015bb20 msgr2=0x7fda200d1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.478+0000 7fda25916640 1 --2- 172.21.15.83:0/3006682355 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fda2015bb20 0x7fda200d1df0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fda100097a0 tx=0x7fda1002b790 comp rx=0 tx=0).stop 2024-03-06T07:22:47.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.478+0000 7fda25916640 1 -- 172.21.15.83:0/3006682355 shutdown_connections 2024-03-06T07:22:47.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.478+0000 7fda25916640 1 --2- 172.21.15.83:0/3006682355 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fda200cd8c0 0x7fda200d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.478+0000 7fda25916640 1 --2- 172.21.15.83:0/3006682355 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fda2015bb20 0x7fda200d1df0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.478+0000 7fda25916640 1 -- 172.21.15.83:0/3006682355 >> 172.21.15.83:0/3006682355 conn(0x7fda2009a9f0 msgr2=0x7fda2009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:47.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.478+0000 7fda25916640 1 -- 172.21.15.83:0/3006682355 shutdown_connections 2024-03-06T07:22:47.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.478+0000 7fda25916640 1 -- 172.21.15.83:0/3006682355 wait complete. 2024-03-06T07:22:47.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.480+0000 7fda25916640 1 Processor -- start 2024-03-06T07:22:47.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.480+0000 7fda25916640 1 -- start start 2024-03-06T07:22:47.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.481+0000 7fda25916640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fda200cd8c0 0x7fda2015ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.481+0000 7fda25916640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fda200cfff0 0x7fda200c73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.481+0000 7fda25916640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fda2015bb20 0x7fda200afa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.481+0000 7fda25916640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda200d24b0 con 0x7fda2015bb20 2024-03-06T07:22:47.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.481+0000 7fda25916640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda200d2330 con 0x7fda200cfff0 2024-03-06T07:22:47.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.481+0000 7fda25916640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda200d1980 con 0x7fda200cd8c0 2024-03-06T07:22:47.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.481+0000 7fda1f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fda2015bb20 0x7fda200afa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.481+0000 7fda1f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fda2015bb20 0x7fda200afa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36512/0 (socket says 172.21.15.83:36512) 2024-03-06T07:22:47.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.481+0000 7fda1f7fe640 1 -- 172.21.15.83:0/689514716 learned_addr learned my addr 172.21.15.83:0/689514716 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:47.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.481+0000 7fda1effd640 1 --2- 172.21.15.83:0/689514716 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fda200cd8c0 0x7fda2015ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.481+0000 7fda1f7fe640 1 -- 172.21.15.83:0/689514716 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fda200cd8c0 msgr2=0x7fda2015ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.481+0000 7fda1f7fe640 1 --2- 172.21.15.83:0/689514716 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fda200cd8c0 0x7fda2015ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.481+0000 7fda1f7fe640 1 -- 172.21.15.83:0/689514716 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fda200cfff0 msgr2=0x7fda200c73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.481+0000 7fda1f7fe640 1 --2- 172.21.15.83:0/689514716 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fda200cfff0 0x7fda200c73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.481+0000 7fda1f7fe640 1 -- 172.21.15.83:0/689514716 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda200c7d50 con 0x7fda2015bb20 2024-03-06T07:22:47.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.482+0000 7fda1f7fe640 1 --2- 172.21.15.83:0/689514716 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fda2015bb20 0x7fda200afa70 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fda1800efc0 tx=0x7fda18009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.482+0000 7fda1cff9640 1 -- 172.21.15.83:0/689514716 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda18018030 con 0x7fda2015bb20 2024-03-06T07:22:47.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.482+0000 7fda1cff9640 1 -- 172.21.15.83:0/689514716 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda1800cde0 con 0x7fda2015bb20 2024-03-06T07:22:47.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.482+0000 7fda1cff9640 1 -- 172.21.15.83:0/689514716 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda18003bb0 con 0x7fda2015bb20 2024-03-06T07:22:47.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.482+0000 7fda25916640 1 -- 172.21.15.83:0/689514716 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda2015f450 con 0x7fda2015bb20 2024-03-06T07:22:47.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.482+0000 7fda25916640 1 -- 172.21.15.83:0/689514716 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda20160470 con 0x7fda2015bb20 2024-03-06T07:22:47.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.483+0000 7fda25916640 1 -- 172.21.15.83:0/689514716 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda2004f330 con 0x7fda2015bb20 2024-03-06T07:22:47.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.483+0000 7fda1cff9640 1 -- 172.21.15.83:0/689514716 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fda1801b070 con 0x7fda2015bb20 2024-03-06T07:22:47.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.484+0000 7fda1cff9640 1 --2- 172.21.15.83:0/689514716 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fda0006b4d0 0x7fda0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.484+0000 7fda1effd640 1 --2- 172.21.15.83:0/689514716 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fda0006b4d0 0x7fda0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.484+0000 7fda1cff9640 1 -- 172.21.15.83:0/689514716 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5912+0+0 (secure 0 0 0) 0x7fda1808acd0 con 0x7fda2015bb20 2024-03-06T07:22:47.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.484+0000 7fda1effd640 1 --2- 172.21.15.83:0/689514716 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fda0006b4d0 0x7fda0006d990 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fda10034000 tx=0x7fda1002f000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.488+0000 7fda1cff9640 1 -- 172.21.15.83:0/689514716 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fda18010040 con 0x7fda2015bb20 2024-03-06T07:22:47.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.667+0000 7fda25916640 1 -- 172.21.15.83:0/689514716 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda200d0640 con 0x7fda2015bb20 2024-03-06T07:22:47.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.668+0000 7fda1cff9640 1 -- 172.21.15.83:0/689514716 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+17006 (secure 0 0 0) 0x7fda18059b10 con 0x7fda2015bb20 2024-03-06T07:22:47.671 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:22:47.671 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":22,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:22:40.105898+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.318942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.254795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.1f","mappings":[{"from":5,"to":1}]},{"pgid":"3.29","mappings":[{"from":3,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:47.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.670+0000 7fda25916640 1 -- 172.21.15.83:0/689514716 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fda0006b4d0 msgr2=0x7fda0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.670+0000 7fda25916640 1 --2- 172.21.15.83:0/689514716 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fda0006b4d0 0x7fda0006d990 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fda10034000 tx=0x7fda1002f000 comp rx=0 tx=0).stop 2024-03-06T07:22:47.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.670+0000 7fda25916640 1 -- 172.21.15.83:0/689514716 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fda2015bb20 msgr2=0x7fda200afa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.670+0000 7fda25916640 1 --2- 172.21.15.83:0/689514716 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fda2015bb20 0x7fda200afa70 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fda1800efc0 tx=0x7fda18009f90 comp rx=0 tx=0).stop 2024-03-06T07:22:47.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.670+0000 7fda25916640 1 -- 172.21.15.83:0/689514716 shutdown_connections 2024-03-06T07:22:47.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.670+0000 7fda25916640 1 --2- 172.21.15.83:0/689514716 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fda0006b4d0 0x7fda0006d990 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.670+0000 7fda25916640 1 --2- 172.21.15.83:0/689514716 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fda2015bb20 0x7fda200afa70 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.670+0000 7fda25916640 1 --2- 172.21.15.83:0/689514716 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fda200cfff0 0x7fda200c73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.670+0000 7fda25916640 1 --2- 172.21.15.83:0/689514716 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fda200cd8c0 0x7fda2015ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.670+0000 7fda25916640 1 -- 172.21.15.83:0/689514716 >> 172.21.15.83:0/689514716 conn(0x7fda2009a9f0 msgr2=0x7fda2009add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:47.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.671+0000 7fda25916640 1 -- 172.21.15.83:0/689514716 shutdown_connections 2024-03-06T07:22:47.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.671+0000 7fda25916640 1 -- 172.21.15.83:0/689514716 wait complete. 2024-03-06T07:22:47.683 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:56.318942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:57.254795+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:47.684 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:47.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.785+0000 7f1750647640 1 Processor -- start 2024-03-06T07:22:47.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.785+0000 7f1750647640 1 -- start start 2024-03-06T07:22:47.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.786+0000 7f1750647640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f17481614d0 0x7f17480d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.786+0000 7f1750647640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f17480d1910 0x7f17480d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.786+0000 7f1750647640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17480d2740 con 0x7f17480d22b0 2024-03-06T07:22:47.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.786+0000 7f1750647640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17481618b0 con 0x7f17481614d0 2024-03-06T07:22:47.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.786+0000 7f1750647640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17480cf400 con 0x7f17480d1910 2024-03-06T07:22:47.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.786+0000 7f174e3e3640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f17481614d0 0x7f17480d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.786+0000 7f174e3e3640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f17481614d0 0x7f17480d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36526/0 (socket says 172.21.15.83:36526) 2024-03-06T07:22:47.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.786+0000 7f174e3e3640 1 -- 172.21.15.83:0/2333072957 learned_addr learned my addr 172.21.15.83:0/2333072957 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:47.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.787+0000 7f174dbe2640 1 --2- 172.21.15.83:0/2333072957 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f17480d1910 0x7f17480d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.787+0000 7f174e3e3640 1 -- 172.21.15.83:0/2333072957 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f17480d1910 msgr2=0x7f17480d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:47.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.787+0000 7f174e3e3640 1 --2- 172.21.15.83:0/2333072957 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f17480d1910 0x7f17480d1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.787+0000 7f174e3e3640 1 -- 172.21.15.83:0/2333072957 >> v1:172.21.15.183:6789/0 conn(0x7f17480d22b0 legacy=0x7f17480d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:47.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.787+0000 7f174e3e3640 1 -- 172.21.15.83:0/2333072957 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17480cf6a0 con 0x7f17481614d0 2024-03-06T07:22:47.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.788+0000 7f174e3e3640 1 --2- 172.21.15.83:0/2333072957 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17481614d0 0x7f17480d4860 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f1738009870 tx=0x7f173802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26c57932421cda1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.788+0000 7f174d3e1640 1 -- 172.21.15.83:0/2333072957 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1738035020 con 0x7f17481614d0 2024-03-06T07:22:47.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.788+0000 7f174d3e1640 1 -- 172.21.15.83:0/2333072957 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1738031640 con 0x7f17481614d0 2024-03-06T07:22:47.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.788+0000 7f174d3e1640 1 -- 172.21.15.83:0/2333072957 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173802d040 con 0x7f17481614d0 2024-03-06T07:22:47.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.789+0000 7f1750647640 1 -- 172.21.15.83:0/2333072957 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17481614d0 msgr2=0x7f17480d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.789+0000 7f1750647640 1 --2- 172.21.15.83:0/2333072957 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17481614d0 0x7f17480d4860 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f1738009870 tx=0x7f173802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.789+0000 7f1750647640 1 -- 172.21.15.83:0/2333072957 shutdown_connections 2024-03-06T07:22:47.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.789+0000 7f1750647640 1 --2- 172.21.15.83:0/2333072957 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f17480d1910 0x7f17480d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.789+0000 7f1750647640 1 --2- 172.21.15.83:0/2333072957 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17481614d0 0x7f17480d4860 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.789+0000 7f1750647640 1 -- 172.21.15.83:0/2333072957 >> 172.21.15.83:0/2333072957 conn(0x7f17480d3d40 msgr2=0x7f17480d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:47.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.789+0000 7f1750647640 1 -- 172.21.15.83:0/2333072957 shutdown_connections 2024-03-06T07:22:47.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.789+0000 7f1750647640 1 -- 172.21.15.83:0/2333072957 wait complete. 2024-03-06T07:22:47.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.790+0000 7f1750647640 1 Processor -- start 2024-03-06T07:22:47.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.790+0000 7f1750647640 1 -- start start 2024-03-06T07:22:47.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.791+0000 7f1750647640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f17480d1910 0x7f17480cdc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.791+0000 7f1750647640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17480d22b0 0x7f1748151060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.791+0000 7f1750647640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17481614d0 0x7f17480c92f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.791+0000 7f1750647640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17480b0910 con 0x7f17480d22b0 2024-03-06T07:22:47.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.791+0000 7f1750647640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17480cfc60 con 0x7f17481614d0 2024-03-06T07:22:47.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.791+0000 7f1750647640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17480b1e40 con 0x7f17480d1910 2024-03-06T07:22:47.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.791+0000 7f174dbe2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17480d22b0 0x7f1748151060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.791+0000 7f174dbe2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17480d22b0 0x7f1748151060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36532/0 (socket says 172.21.15.83:36532) 2024-03-06T07:22:47.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.791+0000 7f174dbe2640 1 -- 172.21.15.83:0/2864015900 learned_addr learned my addr 172.21.15.83:0/2864015900 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:47.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.791+0000 7f174e3e3640 1 --2- 172.21.15.83:0/2864015900 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f17480d1910 0x7f17480cdc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.791+0000 7f174ebe4640 1 --2- 172.21.15.83:0/2864015900 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17481614d0 0x7f17480c92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.791+0000 7f174dbe2640 1 -- 172.21.15.83:0/2864015900 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f17480d1910 msgr2=0x7f17480cdc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.791+0000 7f174dbe2640 1 --2- 172.21.15.83:0/2864015900 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f17480d1910 0x7f17480cdc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.791+0000 7f174dbe2640 1 -- 172.21.15.83:0/2864015900 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17481614d0 msgr2=0x7f17480c92f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.791+0000 7f174dbe2640 1 --2- 172.21.15.83:0/2864015900 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17481614d0 0x7f17480c92f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.791+0000 7f174dbe2640 1 -- 172.21.15.83:0/2864015900 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17480c7e30 con 0x7f17480d22b0 2024-03-06T07:22:47.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.791+0000 7f174e3e3640 1 --2- 172.21.15.83:0/2864015900 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f17480d1910 0x7f17480cdc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:47.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.791+0000 7f174dbe2640 1 --2- 172.21.15.83:0/2864015900 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17480d22b0 0x7f1748151060 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f174400c7f0 tx=0x7f174400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.791+0000 7f173effd640 1 -- 172.21.15.83:0/2864015900 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1744015070 con 0x7f17480d22b0 2024-03-06T07:22:47.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.791+0000 7f173effd640 1 -- 172.21.15.83:0/2864015900 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1744004240 con 0x7f17480d22b0 2024-03-06T07:22:47.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.791+0000 7f1750647640 1 -- 172.21.15.83:0/2864015900 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17480ca0c0 con 0x7f17480d22b0 2024-03-06T07:22:47.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.791+0000 7f173effd640 1 -- 172.21.15.83:0/2864015900 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1744007620 con 0x7f17480d22b0 2024-03-06T07:22:47.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.791+0000 7f1750647640 1 -- 172.21.15.83:0/2864015900 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17480ca680 con 0x7f17480d22b0 2024-03-06T07:22:47.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.793+0000 7f1750647640 1 -- 172.21.15.83:0/2864015900 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1748068ff0 con 0x7f17480d22b0 2024-03-06T07:22:47.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.793+0000 7f173effd640 1 -- 172.21.15.83:0/2864015900 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f17440077c0 con 0x7f17480d22b0 2024-03-06T07:22:47.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.793+0000 7f173effd640 1 --2- 172.21.15.83:0/2864015900 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f172c06b4d0 0x7f172c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.793+0000 7f173effd640 1 -- 172.21.15.83:0/2864015900 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5912+0+0 (secure 0 0 0) 0x7f174408a170 con 0x7f17480d22b0 2024-03-06T07:22:47.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.793+0000 7f174e3e3640 1 --2- 172.21.15.83:0/2864015900 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f172c06b4d0 0x7f172c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.794+0000 7f174e3e3640 1 --2- 172.21.15.83:0/2864015900 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f172c06b4d0 0x7f172c06d990 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1738036000 tx=0x7f1738008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.798+0000 7f173effd640 1 -- 172.21.15.83:0/2864015900 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1744058fb0 con 0x7f17480d22b0 2024-03-06T07:22:47.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.976+0000 7f1750647640 1 -- 172.21.15.83:0/2864015900 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f17481515a0 con 0x7f17480d22b0 2024-03-06T07:22:47.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.976+0000 7f173effd640 1 -- 172.21.15.83:0/2864015900 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f17440588b0 con 0x7f17480d22b0 2024-03-06T07:22:47.978 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:22:47.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.978+0000 7f1750647640 1 -- 172.21.15.83:0/2864015900 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f172c06b4d0 msgr2=0x7f172c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.978+0000 7f1750647640 1 --2- 172.21.15.83:0/2864015900 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f172c06b4d0 0x7f172c06d990 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1738036000 tx=0x7f1738008030 comp rx=0 tx=0).stop 2024-03-06T07:22:47.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.978+0000 7f1750647640 1 -- 172.21.15.83:0/2864015900 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17480d22b0 msgr2=0x7f1748151060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.978+0000 7f1750647640 1 --2- 172.21.15.83:0/2864015900 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17480d22b0 0x7f1748151060 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f174400c7f0 tx=0x7f174400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.978+0000 7f1750647640 1 -- 172.21.15.83:0/2864015900 shutdown_connections 2024-03-06T07:22:47.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.978+0000 7f1750647640 1 --2- 172.21.15.83:0/2864015900 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f172c06b4d0 0x7f172c06d990 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.978+0000 7f1750647640 1 --2- 172.21.15.83:0/2864015900 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17481614d0 0x7f17480c92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.978+0000 7f1750647640 1 --2- 172.21.15.83:0/2864015900 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17480d22b0 0x7f1748151060 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.978+0000 7f1750647640 1 --2- 172.21.15.83:0/2864015900 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f17480d1910 0x7f17480cdc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.978+0000 7f1750647640 1 -- 172.21.15.83:0/2864015900 >> 172.21.15.83:0/2864015900 conn(0x7f17480d3d40 msgr2=0x7f174806ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:47.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.979+0000 7f1750647640 1 -- 172.21.15.83:0/2864015900 shutdown_connections 2024-03-06T07:22:47.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:47.979+0000 7f1750647640 1 -- 172.21.15.83:0/2864015900 wait complete. 2024-03-06T07:22:47.992 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:48.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.094+0000 7f25b1264640 1 Processor -- start 2024-03-06T07:22:48.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.095+0000 7f25b1264640 1 -- start start 2024-03-06T07:22:48.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.095+0000 7f25b1264640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f25ac15ecd0 0x7f25ac0cae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.095+0000 7f25b1264640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25ac0cfa50 con 0x7f25ac0d10c0 2024-03-06T07:22:48.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.095+0000 7f25b1264640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25ac0b45c0 con 0x7f25ac0d14a0 2024-03-06T07:22:48.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.095+0000 7f25b1264640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac15c050 con 0x7f25ac15ecd0 2024-03-06T07:22:48.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.095+0000 7f25aaffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f25ac0d10c0 0x7f25ac0d04f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:47112/0 (socket says 172.21.15.83:47112) 2024-03-06T07:22:48.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.095+0000 7f25aaffd640 1 -- 172.21.15.83:0/3160189748 learned_addr learned my addr 172.21.15.83:0/3160189748 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:48.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.096+0000 7f25ab7fe640 1 --2- 172.21.15.83:0/3160189748 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f25ac15ecd0 0x7f25ac0cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.096+0000 7f25aa7fc640 1 -- 172.21.15.83:0/3160189748 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3202973696 0 0) 0x7f25ac0cfa50 con 0x7f25ac0d10c0 2024-03-06T07:22:48.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.096+0000 7f25aa7fc640 1 -- 172.21.15.83:0/3160189748 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2598003520 con 0x7f25ac0d10c0 2024-03-06T07:22:48.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.096+0000 7f25aa7fc640 1 -- 172.21.15.83:0/3160189748 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 87181067 0 0) 0x7f2598003520 con 0x7f25ac0d10c0 2024-03-06T07:22:48.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.096+0000 7f25aa7fc640 1 -- 172.21.15.83:0/3160189748 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f25ac15ecd0 msgr2=0x7f25ac0cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:48.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.096+0000 7f25aa7fc640 1 --2- 172.21.15.83:0/3160189748 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f25ac15ecd0 0x7f25ac0cae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.096+0000 7f25aa7fc640 1 -- 172.21.15.83:0/3160189748 >> v1:172.21.15.183:6789/0 conn(0x7f25ac0d14a0 legacy=0x7f25ac15e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.096+0000 7f25aa7fc640 1 -- 172.21.15.83:0/3160189748 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ac15f160 con 0x7f25ac0d10c0 2024-03-06T07:22:48.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.096+0000 7f25aa7fc640 1 -- 172.21.15.83:0/3160189748 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f259c001670 con 0x7f25ac0d10c0 2024-03-06T07:22:48.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.096+0000 7f25aa7fc640 1 -- 172.21.15.83:0/3160189748 >> v1:172.21.15.83:6789/0 conn(0x7f25ac0d10c0 legacy=0x7f25ac0d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.096+0000 7f25aa7fc640 1 --2- 172.21.15.83:0/3160189748 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f25980039e0 0x7f2598003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.096+0000 7f25aa7fc640 1 --2- 172.21.15.83:0/3160189748 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2598004340 0x7f2598004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.096+0000 7f25aa7fc640 1 --2- 172.21.15.83:0/3160189748 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2598005020 0x7f259800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.096+0000 7f25aa7fc640 1 -- 172.21.15.83:0/3160189748 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f259800bf00 con 0x7f2598004340 2024-03-06T07:22:48.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.096+0000 7f25aa7fc640 1 -- 172.21.15.83:0/3160189748 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f259800c080 con 0x7f2598005020 2024-03-06T07:22:48.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.096+0000 7f25aa7fc640 1 -- 172.21.15.83:0/3160189748 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f259800c230 con 0x7f25980039e0 2024-03-06T07:22:48.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.096+0000 7f25a3fff640 1 --2- 172.21.15.83:0/3160189748 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2598004340 0x7f2598004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.096+0000 7f25aaffd640 1 --2- 172.21.15.83:0/3160189748 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f25980039e0 0x7f2598003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.096+0000 7f25ab7fe640 1 --2- 172.21.15.83:0/3160189748 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2598005020 0x7f259800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.097+0000 7f25a3fff640 1 -- 172.21.15.83:0/3160189748 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f25980039e0 msgr2=0x7f2598003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:48.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.097+0000 7f25a3fff640 1 --2- 172.21.15.83:0/3160189748 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f25980039e0 0x7f2598003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.097+0000 7f25a3fff640 1 -- 172.21.15.83:0/3160189748 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2598005020 msgr2=0x7f259800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:48.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.097+0000 7f25a3fff640 1 --2- 172.21.15.83:0/3160189748 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2598005020 0x7f259800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.097+0000 7f25a3fff640 1 -- 172.21.15.83:0/3160189748 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2598003520 con 0x7f2598004340 2024-03-06T07:22:48.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.097+0000 7f25a3fff640 1 --2- 172.21.15.83:0/3160189748 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2598004340 0x7f2598004820 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f2594001750 tx=0x7f2598005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d26ada8f80be044e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:48.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.097+0000 7f25aa7fc640 1 -- 172.21.15.83:0/3160189748 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2594001370 con 0x7f2598004340 2024-03-06T07:22:48.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.097+0000 7f25aa7fc640 1 -- 172.21.15.83:0/3160189748 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25940052d0 con 0x7f2598004340 2024-03-06T07:22:48.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.097+0000 7f25aa7fc640 1 -- 172.21.15.83:0/3160189748 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2594003830 con 0x7f2598004340 2024-03-06T07:22:48.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.098+0000 7f25b1264640 1 -- 172.21.15.83:0/3160189748 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2598004340 msgr2=0x7f2598004820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.098+0000 7f25b1264640 1 --2- 172.21.15.83:0/3160189748 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2598004340 0x7f2598004820 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f2594001750 tx=0x7f2598005830 comp rx=0 tx=0).stop 2024-03-06T07:22:48.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.098+0000 7f25ab7fe640 1 -- 172.21.15.83:0/3160189748 reap_dead start 2024-03-06T07:22:48.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.098+0000 7f25b1264640 1 -- 172.21.15.83:0/3160189748 shutdown_connections 2024-03-06T07:22:48.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.098+0000 7f25b1264640 1 -- 172.21.15.83:0/3160189748 >> 172.21.15.83:0/3160189748 conn(0x7f25ac0b3800 msgr2=0x7f25ac0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:48.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.098+0000 7f25b1264640 1 -- 172.21.15.83:0/3160189748 shutdown_connections 2024-03-06T07:22:48.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.098+0000 7f25b1264640 1 -- 172.21.15.83:0/3160189748 wait complete. 2024-03-06T07:22:48.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.099+0000 7f25b1264640 1 Processor -- start 2024-03-06T07:22:48.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.099+0000 7f25b1264640 1 -- start start 2024-03-06T07:22:48.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.100+0000 7f25b1264640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2598004340 0x7f25ac0cdb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.100+0000 7f25b1264640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f25ac15e6a0 0x7f25ac1f37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.100+0000 7f25b1264640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f25ac15eb20 0x7f25ac15c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.100+0000 7f25b1264640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac0cfcf0 con 0x7f2598004340 2024-03-06T07:22:48.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.100+0000 7f25b1264640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac0d0060 con 0x7f25ac15e6a0 2024-03-06T07:22:48.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.100+0000 7f25b1264640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac0cae80 con 0x7f25ac15eb20 2024-03-06T07:22:48.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.100+0000 7f25aaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2598004340 0x7f25ac0cdb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.100+0000 7f25aaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2598004340 0x7f25ac0cdb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36548/0 (socket says 172.21.15.83:36548) 2024-03-06T07:22:48.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.100+0000 7f25aaffd640 1 -- 172.21.15.83:0/336094838 learned_addr learned my addr 172.21.15.83:0/336094838 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:48.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.100+0000 7f25a3fff640 1 --2- 172.21.15.83:0/336094838 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f25ac15e6a0 0x7f25ac1f37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.100+0000 7f25ab7fe640 1 --2- 172.21.15.83:0/336094838 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f25ac15eb20 0x7f25ac15c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.100+0000 7f25aaffd640 1 -- 172.21.15.83:0/336094838 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f25ac15eb20 msgr2=0x7f25ac15c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.100+0000 7f25aaffd640 1 --2- 172.21.15.83:0/336094838 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f25ac15eb20 0x7f25ac15c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.100+0000 7f25aaffd640 1 -- 172.21.15.83:0/336094838 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f25ac15e6a0 msgr2=0x7f25ac1f37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.100+0000 7f25aaffd640 1 --2- 172.21.15.83:0/336094838 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f25ac15e6a0 0x7f25ac1f37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.100+0000 7f25aaffd640 1 -- 172.21.15.83:0/336094838 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ac1f41f0 con 0x7f2598004340 2024-03-06T07:22:48.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.101+0000 7f25aaffd640 1 --2- 172.21.15.83:0/336094838 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2598004340 0x7f25ac0cdb30 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f259c0072f0 tx=0x7f259c0077b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:48.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.101+0000 7f25a37fe640 1 -- 172.21.15.83:0/336094838 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259c001660 con 0x7f2598004340 2024-03-06T07:22:48.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.101+0000 7f25a37fe640 1 -- 172.21.15.83:0/336094838 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f259c018070 con 0x7f2598004340 2024-03-06T07:22:48.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.101+0000 7f25b1264640 1 -- 172.21.15.83:0/336094838 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25ac1f44d0 con 0x7f2598004340 2024-03-06T07:22:48.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.101+0000 7f25a37fe640 1 -- 172.21.15.83:0/336094838 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259c00fb80 con 0x7f2598004340 2024-03-06T07:22:48.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.101+0000 7f25b1264640 1 -- 172.21.15.83:0/336094838 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25ac0cd630 con 0x7f2598004340 2024-03-06T07:22:48.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.102+0000 7f25b1264640 1 -- 172.21.15.83:0/336094838 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25ac09fe60 con 0x7f2598004340 2024-03-06T07:22:48.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.103+0000 7f25a37fe640 1 -- 172.21.15.83:0/336094838 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f259c00fd20 con 0x7f2598004340 2024-03-06T07:22:48.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.103+0000 7f25a37fe640 1 --2- 172.21.15.83:0/336094838 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f258c06b670 0x7f258c06db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.103+0000 7f25a3fff640 1 --2- 172.21.15.83:0/336094838 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f258c06b670 0x7f258c06db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.103+0000 7f25a37fe640 1 -- 172.21.15.83:0/336094838 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5912+0+0 (secure 0 0 0) 0x7f259c08a320 con 0x7f2598004340 2024-03-06T07:22:48.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.104+0000 7f25a3fff640 1 --2- 172.21.15.83:0/336094838 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f258c06b670 0x7f258c06db30 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2594033000 tx=0x7f2594009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:48.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.108+0000 7f25a37fe640 1 -- 172.21.15.83:0/336094838 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f259c0590d0 con 0x7f2598004340 2024-03-06T07:22:48.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.284+0000 7f25b1264640 1 -- 172.21.15.83:0/336094838 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f25ac068ff0 con 0x7f2598004340 2024-03-06T07:22:48.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.285+0000 7f25a37fe640 1 -- 172.21.15.83:0/336094838 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f259c0589d0 con 0x7f2598004340 2024-03-06T07:22:48.287 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-03-06T07:22:48.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.286+0000 7f25b1264640 1 -- 172.21.15.83:0/336094838 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f258c06b670 msgr2=0x7f258c06db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.286+0000 7f25b1264640 1 --2- 172.21.15.83:0/336094838 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f258c06b670 0x7f258c06db30 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2594033000 tx=0x7f2594009030 comp rx=0 tx=0).stop 2024-03-06T07:22:48.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.286+0000 7f25b1264640 1 -- 172.21.15.83:0/336094838 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2598004340 msgr2=0x7f25ac0cdb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.286+0000 7f25b1264640 1 --2- 172.21.15.83:0/336094838 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2598004340 0x7f25ac0cdb30 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f259c0072f0 tx=0x7f259c0077b0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.286+0000 7f25b1264640 1 -- 172.21.15.83:0/336094838 shutdown_connections 2024-03-06T07:22:48.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.286+0000 7f25b1264640 1 --2- 172.21.15.83:0/336094838 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f258c06b670 0x7f258c06db30 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.286+0000 7f25b1264640 1 --2- 172.21.15.83:0/336094838 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f25ac15eb20 0x7f25ac15c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.286+0000 7f25b1264640 1 --2- 172.21.15.83:0/336094838 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f25ac15e6a0 0x7f25ac1f37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.286+0000 7f25b1264640 1 --2- 172.21.15.83:0/336094838 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2598004340 0x7f25ac0cdb30 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.286+0000 7f25b1264640 1 -- 172.21.15.83:0/336094838 >> 172.21.15.83:0/336094838 conn(0x7f25ac0b3800 msgr2=0x7f25ac0b3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:48.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.286+0000 7f25b1264640 1 -- 172.21.15.83:0/336094838 shutdown_connections 2024-03-06T07:22:48.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.286+0000 7f25b1264640 1 -- 172.21.15.83:0/336094838 wait complete. 2024-03-06T07:22:48.303 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:48.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.406+0000 7f28dd602640 1 Processor -- start 2024-03-06T07:22:48.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.407+0000 7f28dd602640 1 -- start start 2024-03-06T07:22:48.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.407+0000 7f28dd602640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f28d80d06c0 0x7f28d81620d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.407+0000 7f28dd602640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d806fa90 con 0x7f28d815ded0 2024-03-06T07:22:48.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.407+0000 7f28dd602640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d815e2b0 con 0x7f28d806cb10 2024-03-06T07:22:48.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.407+0000 7f28dd602640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28d806c0c0 con 0x7f28d80d06c0 2024-03-06T07:22:48.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.408+0000 7f28d6ffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f28d815ded0 0x7f28d81619c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:47116/0 (socket says 172.21.15.83:47116) 2024-03-06T07:22:48.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.408+0000 7f28d6ffd640 1 -- 172.21.15.83:0/2505054721 learned_addr learned my addr 172.21.15.83:0/2505054721 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:48.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.408+0000 7f28d67fc640 1 --2- 172.21.15.83:0/2505054721 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f28d80d06c0 0x7f28d81620d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.408+0000 7f28d5ffb640 1 -- 172.21.15.83:0/2505054721 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2043456580 0 0) 0x7f28c8001110 con 0x7f28d815ded0 2024-03-06T07:22:48.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.408+0000 7f28d5ffb640 1 -- 172.21.15.83:0/2505054721 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28c0003520 con 0x7f28d815ded0 2024-03-06T07:22:48.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.408+0000 7f28d5ffb640 1 -- 172.21.15.83:0/2505054721 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1213288833 0 0) 0x7f28d815e2b0 con 0x7f28d806cb10 2024-03-06T07:22:48.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.408+0000 7f28d5ffb640 1 -- 172.21.15.83:0/2505054721 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28c8001110 con 0x7f28d806cb10 2024-03-06T07:22:48.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.408+0000 7f28d67fc640 1 -- 172.21.15.83:0/2505054721 >> v1:172.21.15.183:6789/0 conn(0x7f28d806cb10 legacy=0x7f28d806cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.408+0000 7f28d67fc640 1 -- 172.21.15.83:0/2505054721 >> v1:172.21.15.83:6789/0 conn(0x7f28d815ded0 legacy=0x7f28d81619c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.408+0000 7f28d5ffb640 1 -- 172.21.15.83:0/2505054721 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2518301461 0 0) 0x7f28c0003520 con 0x7f28d815ded0 2024-03-06T07:22:48.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.408+0000 7f28d67fc640 1 -- 172.21.15.83:0/2505054721 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d80d0b50 con 0x7f28d80d06c0 2024-03-06T07:22:48.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.409+0000 7f28d67fc640 1 --2- 172.21.15.83:0/2505054721 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f28d80d06c0 0x7f28d81620d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f28cc0097d0 tx=0x7f28cc02b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42b1d95d49a4b6e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:48.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.409+0000 7f28d5ffb640 1 -- 172.21.15.83:0/2505054721 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28cc037070 con 0x7f28d80d06c0 2024-03-06T07:22:48.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.409+0000 7f28d5ffb640 1 -- 172.21.15.83:0/2505054721 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28cc030510 con 0x7f28d80d06c0 2024-03-06T07:22:48.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.409+0000 7f28d5ffb640 1 -- 172.21.15.83:0/2505054721 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c8001840 con 0x7f28d80d06c0 2024-03-06T07:22:48.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.410+0000 7f28dd602640 1 -- 172.21.15.83:0/2505054721 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f28d80d06c0 msgr2=0x7f28d81620d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.410+0000 7f28dd602640 1 --2- 172.21.15.83:0/2505054721 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f28d80d06c0 0x7f28d81620d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f28cc0097d0 tx=0x7f28cc02b800 comp rx=0 tx=0).stop 2024-03-06T07:22:48.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.410+0000 7f28dd602640 1 -- 172.21.15.83:0/2505054721 shutdown_connections 2024-03-06T07:22:48.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.410+0000 7f28dd602640 1 --2- 172.21.15.83:0/2505054721 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f28d80d06c0 0x7f28d81620d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.410+0000 7f28dd602640 1 -- 172.21.15.83:0/2505054721 >> 172.21.15.83:0/2505054721 conn(0x7f28d80b34c0 msgr2=0x7f28d80b38d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:48.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.410+0000 7f28dd602640 1 -- 172.21.15.83:0/2505054721 shutdown_connections 2024-03-06T07:22:48.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.410+0000 7f28dd602640 1 -- 172.21.15.83:0/2505054721 wait complete. 2024-03-06T07:22:48.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.411+0000 7f28dd602640 1 Processor -- start 2024-03-06T07:22:48.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.411+0000 7f28dd602640 1 -- start start 2024-03-06T07:22:48.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.412+0000 7f28dd602640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28d806cb10 0x7f28d814c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.412+0000 7f28dd602640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f28d815ded0 0x7f28d815e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.412+0000 7f28dd602640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f28d815e970 0x7f28d80b5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.412+0000 7f28dd602640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d80b11c0 con 0x7f28d806cb10 2024-03-06T07:22:48.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.412+0000 7f28dd602640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d806c310 con 0x7f28d815e970 2024-03-06T07:22:48.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.412+0000 7f28dd602640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28d8162610 con 0x7f28d815ded0 2024-03-06T07:22:48.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.412+0000 7f28d6ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28d806cb10 0x7f28d814c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.412+0000 7f28d6ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28d806cb10 0x7f28d814c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36554/0 (socket says 172.21.15.83:36554) 2024-03-06T07:22:48.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.412+0000 7f28d6ffd640 1 -- 172.21.15.83:0/142127831 learned_addr learned my addr 172.21.15.83:0/142127831 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:48.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.412+0000 7f28d77fe640 1 --2- 172.21.15.83:0/142127831 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f28d815e970 0x7f28d80b5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.412+0000 7f28d67fc640 1 --2- 172.21.15.83:0/142127831 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f28d815ded0 0x7f28d815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.412+0000 7f28d6ffd640 1 -- 172.21.15.83:0/142127831 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f28d815ded0 msgr2=0x7f28d815e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.412+0000 7f28d6ffd640 1 --2- 172.21.15.83:0/142127831 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f28d815ded0 0x7f28d815e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.412+0000 7f28d6ffd640 1 -- 172.21.15.83:0/142127831 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f28d815e970 msgr2=0x7f28d80b5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.412+0000 7f28d6ffd640 1 --2- 172.21.15.83:0/142127831 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f28d815e970 0x7f28d80b5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.412+0000 7f28d6ffd640 1 -- 172.21.15.83:0/142127831 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d80b6330 con 0x7f28d806cb10 2024-03-06T07:22:48.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.412+0000 7f28d77fe640 1 --2- 172.21.15.83:0/142127831 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f28d815e970 0x7f28d80b5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:48.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.413+0000 7f28d6ffd640 1 --2- 172.21.15.83:0/142127831 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28d806cb10 0x7f28d814c740 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f28c800caf0 tx=0x7f28c8001840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:48.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.413+0000 7f28bf7fe640 1 -- 172.21.15.83:0/142127831 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c8017070 con 0x7f28d806cb10 2024-03-06T07:22:48.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.413+0000 7f28bf7fe640 1 -- 172.21.15.83:0/142127831 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28c8005d00 con 0x7f28d806cb10 2024-03-06T07:22:48.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.413+0000 7f28dd602640 1 -- 172.21.15.83:0/142127831 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28d80b6610 con 0x7f28d806cb10 2024-03-06T07:22:48.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.413+0000 7f28bf7fe640 1 -- 172.21.15.83:0/142127831 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c80072b0 con 0x7f28d806cb10 2024-03-06T07:22:48.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.413+0000 7f28dd602640 1 -- 172.21.15.83:0/142127831 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28d80b6bd0 con 0x7f28d806cb10 2024-03-06T07:22:48.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.414+0000 7f28dd602640 1 -- 172.21.15.83:0/142127831 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28d815ed80 con 0x7f28d806cb10 2024-03-06T07:22:48.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.414+0000 7f28bf7fe640 1 -- 172.21.15.83:0/142127831 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f28c801a070 con 0x7f28d806cb10 2024-03-06T07:22:48.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.415+0000 7f28bf7fe640 1 --2- 172.21.15.83:0/142127831 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f28c406b4d0 0x7f28c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.415+0000 7f28bf7fe640 1 -- 172.21.15.83:0/142127831 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5912+0+0 (secure 0 0 0) 0x7f28c8089ac0 con 0x7f28d806cb10 2024-03-06T07:22:48.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.415+0000 7f28d67fc640 1 --2- 172.21.15.83:0/142127831 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f28c406b4d0 0x7f28c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.415+0000 7f28d67fc640 1 --2- 172.21.15.83:0/142127831 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f28c406b4d0 0x7f28c406d990 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f28cc034000 tx=0x7f28cc008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:48.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.423+0000 7f28bf7fe640 1 -- 172.21.15.83:0/142127831 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f28d815ed80 con 0x7f28d806cb10 2024-03-06T07:22:48.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.620+0000 7f28dd602640 1 -- 172.21.15.83:0/142127831 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f28d8068f10 con 0x7f28d806cb10 2024-03-06T07:22:48.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.620+0000 7f28bf7fe640 1 -- 172.21.15.83:0/142127831 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f28d806fa90 con 0x7f28d806cb10 2024-03-06T07:22:48.622 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 64 2024-03-06T07:22:48.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.623+0000 7f28dd602640 1 -- 172.21.15.83:0/142127831 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f28c406b4d0 msgr2=0x7f28c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.623+0000 7f28dd602640 1 --2- 172.21.15.83:0/142127831 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f28c406b4d0 0x7f28c406d990 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f28cc034000 tx=0x7f28cc008030 comp rx=0 tx=0).stop 2024-03-06T07:22:48.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.623+0000 7f28dd602640 1 -- 172.21.15.83:0/142127831 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28d806cb10 msgr2=0x7f28d814c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.623+0000 7f28dd602640 1 --2- 172.21.15.83:0/142127831 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28d806cb10 0x7f28d814c740 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f28c800caf0 tx=0x7f28c8001840 comp rx=0 tx=0).stop 2024-03-06T07:22:48.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.623+0000 7f28dd602640 1 -- 172.21.15.83:0/142127831 shutdown_connections 2024-03-06T07:22:48.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.623+0000 7f28dd602640 1 --2- 172.21.15.83:0/142127831 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f28c406b4d0 0x7f28c406d990 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.623+0000 7f28dd602640 1 --2- 172.21.15.83:0/142127831 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f28d815e970 0x7f28d80b5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.623+0000 7f28dd602640 1 --2- 172.21.15.83:0/142127831 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f28d815ded0 0x7f28d815e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.623+0000 7f28dd602640 1 --2- 172.21.15.83:0/142127831 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28d806cb10 0x7f28d814c740 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.623+0000 7f28dd602640 1 -- 172.21.15.83:0/142127831 >> 172.21.15.83:0/142127831 conn(0x7f28d80b34c0 msgr2=0x7f28d80cdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:48.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.623+0000 7f28dd602640 1 -- 172.21.15.83:0/142127831 shutdown_connections 2024-03-06T07:22:48.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.623+0000 7f28dd602640 1 -- 172.21.15.83:0/142127831 wait complete. 2024-03-06T07:22:48.637 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:48.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.748+0000 7eff50481640 1 Processor -- start 2024-03-06T07:22:48.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.748+0000 7eff50481640 1 -- start start 2024-03-06T07:22:48.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.749+0000 7eff50481640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7eff4815bb20 0x7eff480d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.749+0000 7eff50481640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7eff480cd8c0 0x7eff480d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.749+0000 7eff50481640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff48161980 con 0x7eff480cfff0 2024-03-06T07:22:48.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.749+0000 7eff50481640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff48161c60 con 0x7eff480cd8c0 2024-03-06T07:22:48.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.749+0000 7eff50481640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff48161f40 con 0x7eff4815bb20 2024-03-06T07:22:48.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.749+0000 7eff4da1c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7eff480cfff0 0x7eff480ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:47128/0 (socket says 172.21.15.83:47128) 2024-03-06T07:22:48.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.749+0000 7eff4da1c640 1 -- 172.21.15.83:0/3460611639 learned_addr learned my addr 172.21.15.83:0/3460611639 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:48.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.750+0000 7eff4ea1e640 1 --2- 172.21.15.83:0/3460611639 >> v2:172.21.15.183:3300/0 conn(0x7eff480cd8c0 0x7eff480d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.750+0000 7eff4e21d640 1 --2- 172.21.15.83:0/3460611639 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7eff4815bb20 0x7eff480d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.750+0000 7eff4d21b640 1 -- 172.21.15.83:0/3460611639 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3591622357 0 0) 0x7eff48161980 con 0x7eff480cfff0 2024-03-06T07:22:48.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.750+0000 7eff4d21b640 1 -- 172.21.15.83:0/3460611639 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff34003520 con 0x7eff480cfff0 2024-03-06T07:22:48.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.750+0000 7eff4d21b640 1 -- 172.21.15.83:0/3460611639 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3305574231 0 0) 0x7eff34003520 con 0x7eff480cfff0 2024-03-06T07:22:48.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.750+0000 7eff4d21b640 1 -- 172.21.15.83:0/3460611639 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7eff4815bb20 msgr2=0x7eff480d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:48.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.750+0000 7eff4d21b640 1 --2- 172.21.15.83:0/3460611639 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7eff4815bb20 0x7eff480d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.750+0000 7eff4d21b640 1 -- 172.21.15.83:0/3460611639 >> v2:172.21.15.183:3300/0 conn(0x7eff480cd8c0 msgr2=0x7eff480d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:48.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.750+0000 7eff4d21b640 1 --2- 172.21.15.83:0/3460611639 >> v2:172.21.15.183:3300/0 conn(0x7eff480cd8c0 0x7eff480d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.750+0000 7eff4d21b640 1 -- 172.21.15.83:0/3460611639 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff48162220 con 0x7eff480cfff0 2024-03-06T07:22:48.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.750+0000 7eff4d21b640 1 -- 172.21.15.83:0/3460611639 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7eff44001670 con 0x7eff480cfff0 2024-03-06T07:22:48.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.750+0000 7eff4d21b640 1 -- 172.21.15.83:0/3460611639 >> v1:172.21.15.83:6789/0 conn(0x7eff480cfff0 legacy=0x7eff480ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.750+0000 7eff4d21b640 1 --2- 172.21.15.83:0/3460611639 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7eff340039b0 0x7eff34003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.750+0000 7eff4d21b640 1 --2- 172.21.15.83:0/3460611639 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7eff34004800 0x7eff34004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.750+0000 7eff4d21b640 1 --2- 172.21.15.83:0/3460611639 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff340051c0 0x7eff3400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.750+0000 7eff4d21b640 1 -- 172.21.15.83:0/3460611639 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff48161c60 con 0x7eff340051c0 2024-03-06T07:22:48.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.750+0000 7eff4d21b640 1 -- 172.21.15.83:0/3460611639 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff48161f40 con 0x7eff340039b0 2024-03-06T07:22:48.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.750+0000 7eff4d21b640 1 -- 172.21.15.83:0/3460611639 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff3400bf50 con 0x7eff34004800 2024-03-06T07:22:48.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.750+0000 7eff4e21d640 1 --2- 172.21.15.83:0/3460611639 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7eff4815bb20 0x7eff480d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:48.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.750+0000 7eff4ea1e640 1 --2- 172.21.15.83:0/3460611639 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff340051c0 0x7eff3400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.750+0000 7eff4da1c640 1 --2- 172.21.15.83:0/3460611639 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7eff34004800 0x7eff34004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.750+0000 7eff4e21d640 1 --2- 172.21.15.83:0/3460611639 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7eff340039b0 0x7eff34003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.750+0000 7eff4ea1e640 1 -- 172.21.15.83:0/3460611639 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7eff34004800 msgr2=0x7eff34004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:48.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.750+0000 7eff4ea1e640 1 --2- 172.21.15.83:0/3460611639 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7eff34004800 0x7eff34004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.750+0000 7eff4ea1e640 1 -- 172.21.15.83:0/3460611639 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7eff340039b0 msgr2=0x7eff34003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:48.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.751+0000 7eff4ea1e640 1 --2- 172.21.15.83:0/3460611639 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7eff340039b0 0x7eff34003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.751+0000 7eff4ea1e640 1 -- 172.21.15.83:0/3460611639 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff34003520 con 0x7eff340051c0 2024-03-06T07:22:48.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.751+0000 7eff4e21d640 1 --2- 172.21.15.83:0/3460611639 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7eff340039b0 0x7eff34003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:48.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.752+0000 7eff4ea1e640 1 --2- 172.21.15.83:0/3460611639 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff340051c0 0x7eff3400b9c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7eff4000e170 tx=0x7eff34005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10f1f5d436bcde50 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:48.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.752+0000 7eff4d21b640 1 -- 172.21.15.83:0/3460611639 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff40039070 con 0x7eff340051c0 2024-03-06T07:22:48.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.752+0000 7eff4d21b640 1 -- 172.21.15.83:0/3460611639 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff4002cc20 con 0x7eff340051c0 2024-03-06T07:22:48.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.752+0000 7eff4d21b640 1 -- 172.21.15.83:0/3460611639 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff40009c10 con 0x7eff340051c0 2024-03-06T07:22:48.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.752+0000 7eff50481640 1 -- 172.21.15.83:0/3460611639 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff340051c0 msgr2=0x7eff3400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.752+0000 7eff50481640 1 --2- 172.21.15.83:0/3460611639 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff340051c0 0x7eff3400b9c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7eff4000e170 tx=0x7eff34005830 comp rx=0 tx=0).stop 2024-03-06T07:22:48.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.752+0000 7eff4ea1e640 1 -- 172.21.15.83:0/3460611639 reap_dead start 2024-03-06T07:22:48.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.753+0000 7eff50481640 1 -- 172.21.15.83:0/3460611639 shutdown_connections 2024-03-06T07:22:48.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.753+0000 7eff50481640 1 -- 172.21.15.83:0/3460611639 >> 172.21.15.83:0/3460611639 conn(0x7eff4809a9f0 msgr2=0x7eff4809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:48.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.753+0000 7eff50481640 1 -- 172.21.15.83:0/3460611639 shutdown_connections 2024-03-06T07:22:48.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.753+0000 7eff50481640 1 -- 172.21.15.83:0/3460611639 wait complete. 2024-03-06T07:22:48.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.754+0000 7eff50481640 1 Processor -- start 2024-03-06T07:22:48.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.754+0000 7eff50481640 1 -- start start 2024-03-06T07:22:48.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.756+0000 7eff50481640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7eff340051c0 0x7eff481582e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.756+0000 7eff50481640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff481f6d30 0x7eff481f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.756+0000 7eff50481640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7eff4815cb20 0x7eff4815cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.756+0000 7eff50481640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff480d1b00 con 0x7eff481f6d30 2024-03-06T07:22:48.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.756+0000 7eff50481640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff480d1980 con 0x7eff340051c0 2024-03-06T07:22:48.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.756+0000 7eff50481640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff480d1df0 con 0x7eff4815cb20 2024-03-06T07:22:48.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.756+0000 7eff4da1c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff481f6d30 0x7eff481f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.756+0000 7eff4da1c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff481f6d30 0x7eff481f71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36562/0 (socket says 172.21.15.83:36562) 2024-03-06T07:22:48.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.756+0000 7eff4da1c640 1 -- 172.21.15.83:0/2012317561 learned_addr learned my addr 172.21.15.83:0/2012317561 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:48.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.756+0000 7eff4da1c640 1 -- 172.21.15.83:0/2012317561 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7eff4815cb20 msgr2=0x7eff4815cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.756+0000 7eff4da1c640 1 --2- 172.21.15.83:0/2012317561 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7eff4815cb20 0x7eff4815cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.756+0000 7eff4da1c640 1 -- 172.21.15.83:0/2012317561 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7eff340051c0 msgr2=0x7eff481582e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.756+0000 7eff4da1c640 1 --2- 172.21.15.83:0/2012317561 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7eff340051c0 0x7eff481582e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.756+0000 7eff4da1c640 1 -- 172.21.15.83:0/2012317561 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff48158b30 con 0x7eff481f6d30 2024-03-06T07:22:48.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.756+0000 7eff4da1c640 1 --2- 172.21.15.83:0/2012317561 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff481f6d30 0x7eff481f71b0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7eff44007340 tx=0x7eff44007740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:48.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.756+0000 7eff3effd640 1 -- 172.21.15.83:0/2012317561 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff4400f860 con 0x7eff481f6d30 2024-03-06T07:22:48.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.756+0000 7eff3effd640 1 -- 172.21.15.83:0/2012317561 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff44018070 con 0x7eff481f6d30 2024-03-06T07:22:48.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.756+0000 7eff50481640 1 -- 172.21.15.83:0/2012317561 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff48158e10 con 0x7eff481f6d30 2024-03-06T07:22:48.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.756+0000 7eff3effd640 1 -- 172.21.15.83:0/2012317561 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff4400fc80 con 0x7eff481f6d30 2024-03-06T07:22:48.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.756+0000 7eff50481640 1 -- 172.21.15.83:0/2012317561 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff481593d0 con 0x7eff481f6d30 2024-03-06T07:22:48.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.758+0000 7eff50481640 1 -- 172.21.15.83:0/2012317561 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff4804f330 con 0x7eff481f6d30 2024-03-06T07:22:48.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.758+0000 7eff3effd640 1 -- 172.21.15.83:0/2012317561 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7eff4401d460 con 0x7eff481f6d30 2024-03-06T07:22:48.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.758+0000 7eff3effd640 1 --2- 172.21.15.83:0/2012317561 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7eff2c06b500 0x7eff2c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.759+0000 7eff3effd640 1 -- 172.21.15.83:0/2012317561 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5912+0+0 (secure 0 0 0) 0x7eff4408a3b0 con 0x7eff481f6d30 2024-03-06T07:22:48.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.759+0000 7eff4e21d640 1 --2- 172.21.15.83:0/2012317561 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7eff2c06b500 0x7eff2c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.759+0000 7eff4e21d640 1 --2- 172.21.15.83:0/2012317561 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7eff2c06b500 0x7eff2c06d9c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7eff480d2bc0 tx=0x7eff38008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:48.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.763+0000 7eff3effd640 1 -- 172.21.15.83:0/2012317561 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7eff44059160 con 0x7eff481f6d30 2024-03-06T07:22:48.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.980+0000 7eff50481640 1 -- 172.21.15.83:0/2012317561 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff480b29a0 con 0x7eff481f6d30 2024-03-06T07:22:48.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.980+0000 7eff3effd640 1 -- 172.21.15.83:0/2012317561 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4974 (secure 0 0 0) 0x7eff44058a60 con 0x7eff481f6d30 2024-03-06T07:22:48.983 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:22:48.983 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/4249608353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":4249608353},{"type":"v1","addr":"172.21.15.83:6836","nonce":4249608353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/714452530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":714452530},{"type":"v1","addr":"172.21.15.183:6835","nonce":714452530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4282,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/2723013431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2723013431},{"type":"v1","addr":"172.21.15.183:6834","nonce":2723013431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:58.286994+0000","modified":"2024-03-06T07:22:00.307284+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/1597344199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1597344199},{"type":"v1","addr":"172.21.15.83:6837","nonce":1597344199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4287,"qdb_cluster":[4287]},"id":1}]} 2024-03-06T07:22:48.983 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 6 2024-03-06T07:22:48.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.983+0000 7eff50481640 1 -- 172.21.15.83:0/2012317561 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7eff2c06b500 msgr2=0x7eff2c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.983+0000 7eff50481640 1 --2- 172.21.15.83:0/2012317561 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7eff2c06b500 0x7eff2c06d9c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7eff480d2bc0 tx=0x7eff38008040 comp rx=0 tx=0).stop 2024-03-06T07:22:48.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.983+0000 7eff50481640 1 -- 172.21.15.83:0/2012317561 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff481f6d30 msgr2=0x7eff481f71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.983+0000 7eff50481640 1 --2- 172.21.15.83:0/2012317561 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff481f6d30 0x7eff481f71b0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7eff44007340 tx=0x7eff44007740 comp rx=0 tx=0).stop 2024-03-06T07:22:48.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.983+0000 7eff50481640 1 -- 172.21.15.83:0/2012317561 shutdown_connections 2024-03-06T07:22:48.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.983+0000 7eff50481640 1 --2- 172.21.15.83:0/2012317561 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7eff2c06b500 0x7eff2c06d9c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.983+0000 7eff50481640 1 --2- 172.21.15.83:0/2012317561 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7eff4815cb20 0x7eff4815cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.983+0000 7eff50481640 1 --2- 172.21.15.83:0/2012317561 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff481f6d30 0x7eff481f71b0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.983+0000 7eff50481640 1 --2- 172.21.15.83:0/2012317561 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7eff340051c0 0x7eff481582e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.983+0000 7eff50481640 1 -- 172.21.15.83:0/2012317561 >> 172.21.15.83:0/2012317561 conn(0x7eff4809a9f0 msgr2=0x7eff480ce4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:48.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.983+0000 7eff50481640 1 -- 172.21.15.83:0/2012317561 shutdown_connections 2024-03-06T07:22:48.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:48.983+0000 7eff50481640 1 -- 172.21.15.83:0/2012317561 wait complete. 2024-03-06T07:22:48.996 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:49.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.096+0000 7f02eb7f7640 1 Processor -- start 2024-03-06T07:22:49.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.096+0000 7f02eb7f7640 1 -- start start 2024-03-06T07:22:49.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.097+0000 7f02eb7f7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f02e415dfe0 0x7f02e40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.097+0000 7f02eb7f7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e40b3430 con 0x7f02e40d29c0 2024-03-06T07:22:49.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.097+0000 7f02eb7f7640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e40b3710 con 0x7f02e40cf830 2024-03-06T07:22:49.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.097+0000 7f02eb7f7640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02e40b31e0 con 0x7f02e415dfe0 2024-03-06T07:22:49.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.097+0000 7f02e9593640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f02e415dfe0 0x7f02e40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.097+0000 7f02e9d94640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f02e40d29c0 0x7f02e40b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:47142/0 (socket says 172.21.15.83:47142) 2024-03-06T07:22:49.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.097+0000 7f02e9593640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f02e415dfe0 0x7f02e40cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36578/0 (socket says 172.21.15.83:36578) 2024-03-06T07:22:49.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.097+0000 7f02e9d94640 1 -- 172.21.15.83:0/2898439171 learned_addr learned my addr 172.21.15.83:0/2898439171 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:49.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.098+0000 7f02e9593640 1 -- 172.21.15.83:0/2898439171 >> v1:172.21.15.183:6789/0 conn(0x7f02e40cf830 legacy=0x7f02e40d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.098+0000 7f02e9593640 1 -- 172.21.15.83:0/2898439171 >> v1:172.21.15.83:6789/0 conn(0x7f02e40d29c0 legacy=0x7f02e40b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.098+0000 7f02dbfff640 1 -- 172.21.15.83:0/2898439171 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1736139123 0 0) 0x7f02e40b3430 con 0x7f02e40d29c0 2024-03-06T07:22:49.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.098+0000 7f02e9593640 1 -- 172.21.15.83:0/2898439171 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e40b39f0 con 0x7f02e415dfe0 2024-03-06T07:22:49.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.099+0000 7f02e9593640 1 --2- 172.21.15.83:0/2898439171 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02e415dfe0 0x7f02e40cf2f0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f02e40b3360 tx=0x7f02e002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66c7e65100a5b816 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.099+0000 7f02dbfff640 1 -- 172.21.15.83:0/2898439171 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e0034070 con 0x7f02e415dfe0 2024-03-06T07:22:49.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.099+0000 7f02dbfff640 1 -- 172.21.15.83:0/2898439171 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02e0030610 con 0x7f02e415dfe0 2024-03-06T07:22:49.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.099+0000 7f02dbfff640 1 -- 172.21.15.83:0/2898439171 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e00309e0 con 0x7f02e415dfe0 2024-03-06T07:22:49.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.100+0000 7f02eb7f7640 1 -- 172.21.15.83:0/2898439171 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02e415dfe0 msgr2=0x7f02e40cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.100+0000 7f02eb7f7640 1 --2- 172.21.15.83:0/2898439171 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02e415dfe0 0x7f02e40cf2f0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f02e40b3360 tx=0x7f02e002b960 comp rx=0 tx=0).stop 2024-03-06T07:22:49.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.100+0000 7f02eb7f7640 1 -- 172.21.15.83:0/2898439171 shutdown_connections 2024-03-06T07:22:49.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.100+0000 7f02eb7f7640 1 --2- 172.21.15.83:0/2898439171 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02e415dfe0 0x7f02e40cf2f0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.100+0000 7f02eb7f7640 1 -- 172.21.15.83:0/2898439171 >> 172.21.15.83:0/2898439171 conn(0x7f02e40d3cd0 msgr2=0x7f02e40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:49.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.100+0000 7f02eb7f7640 1 -- 172.21.15.83:0/2898439171 shutdown_connections 2024-03-06T07:22:49.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.100+0000 7f02eb7f7640 1 -- 172.21.15.83:0/2898439171 wait complete. 2024-03-06T07:22:49.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.101+0000 7f02eb7f7640 1 Processor -- start 2024-03-06T07:22:49.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.101+0000 7f02eb7f7640 1 -- start start 2024-03-06T07:22:49.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.102+0000 7f02eb7f7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f02e40cf830 0x7f02e40b0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.102+0000 7f02eb7f7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02e40d29c0 0x7f02e41f3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.102+0000 7f02eb7f7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f02e415dfe0 0x7f02e40cb920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.102+0000 7f02eb7f7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e415e3c0 con 0x7f02e40d29c0 2024-03-06T07:22:49.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.102+0000 7f02eb7f7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e40b3d60 con 0x7f02e40cf830 2024-03-06T07:22:49.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.102+0000 7f02eb7f7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02e415e540 con 0x7f02e415dfe0 2024-03-06T07:22:49.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.102+0000 7f02e8d92640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02e40d29c0 0x7f02e41f3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.102+0000 7f02e8d92640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02e40d29c0 0x7f02e41f3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36592/0 (socket says 172.21.15.83:36592) 2024-03-06T07:22:49.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.102+0000 7f02e8d92640 1 -- 172.21.15.83:0/1286119683 learned_addr learned my addr 172.21.15.83:0/1286119683 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:49.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.102+0000 7f02e9593640 1 --2- 172.21.15.83:0/1286119683 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f02e40cf830 0x7f02e40b0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.102+0000 7f02e8d92640 1 -- 172.21.15.83:0/1286119683 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f02e415dfe0 msgr2=0x7f02e40cb920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.102+0000 7f02e8d92640 1 --2- 172.21.15.83:0/1286119683 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f02e415dfe0 0x7f02e40cb920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.102+0000 7f02e8d92640 1 -- 172.21.15.83:0/1286119683 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f02e40cf830 msgr2=0x7f02e40b0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.102+0000 7f02e8d92640 1 --2- 172.21.15.83:0/1286119683 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f02e40cf830 0x7f02e40b0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.102+0000 7f02e8d92640 1 -- 172.21.15.83:0/1286119683 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e40b1880 con 0x7f02e40d29c0 2024-03-06T07:22:49.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.102+0000 7f02e8d92640 1 --2- 172.21.15.83:0/1286119683 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02e40d29c0 0x7f02e41f3250 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f02d400c430 tx=0x7f02d400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.102+0000 7f02d9ffb640 1 -- 172.21.15.83:0/1286119683 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d4017070 con 0x7f02e40d29c0 2024-03-06T07:22:49.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.103+0000 7f02d9ffb640 1 -- 172.21.15.83:0/1286119683 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02d40052d0 con 0x7f02e40d29c0 2024-03-06T07:22:49.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.103+0000 7f02d9ffb640 1 -- 172.21.15.83:0/1286119683 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d4003830 con 0x7f02e40d29c0 2024-03-06T07:22:49.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.103+0000 7f02eb7f7640 1 -- 172.21.15.83:0/1286119683 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02e40b15a0 con 0x7f02e40d29c0 2024-03-06T07:22:49.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.103+0000 7f02eb7f7640 1 -- 172.21.15.83:0/1286119683 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02e40cb120 con 0x7f02e40d29c0 2024-03-06T07:22:49.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.104+0000 7f02eb7f7640 1 -- 172.21.15.83:0/1286119683 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02e404f330 con 0x7f02e40d29c0 2024-03-06T07:22:49.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.104+0000 7f02d9ffb640 1 -- 172.21.15.83:0/1286119683 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f02d401a070 con 0x7f02e40d29c0 2024-03-06T07:22:49.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.105+0000 7f02d9ffb640 1 --2- 172.21.15.83:0/1286119683 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f02c806b500 0x7f02c806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.105+0000 7f02e9593640 1 --2- 172.21.15.83:0/1286119683 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f02c806b500 0x7f02c806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.105+0000 7f02d9ffb640 1 -- 172.21.15.83:0/1286119683 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5912+0+0 (secure 0 0 0) 0x7f02d40896d0 con 0x7f02e40d29c0 2024-03-06T07:22:49.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.105+0000 7f02e9593640 1 --2- 172.21.15.83:0/1286119683 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f02c806b500 0x7f02c806d9c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f02e0036000 tx=0x7f02e0008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.113+0000 7f02d9ffb640 1 -- 172.21.15.83:0/1286119683 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f02d4058480 con 0x7f02e40d29c0 2024-03-06T07:22:49.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.318+0000 7f02eb7f7640 1 -- 172.21.15.83:0/1286119683 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02e41f3790 con 0x7f02e40d29c0 2024-03-06T07:22:49.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.319+0000 7f02d9ffb640 1 -- 172.21.15.83:0/1286119683 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+17006 (secure 0 0 0) 0x7f02d4057d80 con 0x7f02e40d29c0 2024-03-06T07:22:49.321 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:22:49.321 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":22,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:22:40.105898+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.318942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.254795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.1f","mappings":[{"from":5,"to":1}]},{"pgid":"3.29","mappings":[{"from":3,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:49.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.321+0000 7f02eb7f7640 1 -- 172.21.15.83:0/1286119683 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f02c806b500 msgr2=0x7f02c806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.321+0000 7f02eb7f7640 1 --2- 172.21.15.83:0/1286119683 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f02c806b500 0x7f02c806d9c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f02e0036000 tx=0x7f02e0008030 comp rx=0 tx=0).stop 2024-03-06T07:22:49.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.321+0000 7f02eb7f7640 1 -- 172.21.15.83:0/1286119683 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02e40d29c0 msgr2=0x7f02e41f3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.321+0000 7f02eb7f7640 1 --2- 172.21.15.83:0/1286119683 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02e40d29c0 0x7f02e41f3250 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f02d400c430 tx=0x7f02d400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.321+0000 7f02eb7f7640 1 -- 172.21.15.83:0/1286119683 shutdown_connections 2024-03-06T07:22:49.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.321+0000 7f02eb7f7640 1 --2- 172.21.15.83:0/1286119683 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f02c806b500 0x7f02c806d9c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.321+0000 7f02eb7f7640 1 --2- 172.21.15.83:0/1286119683 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f02e415dfe0 0x7f02e40cb920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.321+0000 7f02eb7f7640 1 --2- 172.21.15.83:0/1286119683 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02e40d29c0 0x7f02e41f3250 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.321+0000 7f02eb7f7640 1 --2- 172.21.15.83:0/1286119683 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f02e40cf830 0x7f02e40b0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.321+0000 7f02eb7f7640 1 -- 172.21.15.83:0/1286119683 >> 172.21.15.83:0/1286119683 conn(0x7f02e40d3cd0 msgr2=0x7f02e40b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:49.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.321+0000 7f02eb7f7640 1 -- 172.21.15.83:0/1286119683 shutdown_connections 2024-03-06T07:22:49.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.321+0000 7f02eb7f7640 1 -- 172.21.15.83:0/1286119683 wait complete. 2024-03-06T07:22:49.334 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:22:49.334 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:49.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.434+0000 7f6827156640 1 Processor -- start 2024-03-06T07:22:49.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.434+0000 7f6827156640 1 -- start start 2024-03-06T07:22:49.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.434+0000 7f6827156640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f682015dfe0 0x7f68200cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.434+0000 7f6827156640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f68200d29c0 0x7f68200b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.434+0000 7f6827156640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68200b3430 con 0x7f68200cf830 2024-03-06T07:22:49.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.434+0000 7f6827156640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68200b31e0 con 0x7f68200d29c0 2024-03-06T07:22:49.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.434+0000 7f6827156640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68200b3d60 con 0x7f682015dfe0 2024-03-06T07:22:49.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.435+0000 7f6824ef2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f682015dfe0 0x7f68200cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.435+0000 7f68256f3640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f68200d29c0 0x7f68200b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.435+0000 7f681ffff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f68200cf830 0x7f68200d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:35878/0 (socket says 172.21.15.83:35878) 2024-03-06T07:22:49.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.435+0000 7f681ffff640 1 -- 172.21.15.83:0/937159724 learned_addr learned my addr 172.21.15.83:0/937159724 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:49.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.435+0000 7f681f7fe640 1 -- 172.21.15.83:0/937159724 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 891427774 0 0) 0x7f68200b3430 con 0x7f68200cf830 2024-03-06T07:22:49.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.435+0000 7f681f7fe640 1 -- 172.21.15.83:0/937159724 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6810003520 con 0x7f68200cf830 2024-03-06T07:22:49.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.435+0000 7f6824ef2640 1 -- 172.21.15.83:0/937159724 >> v2:172.21.15.183:3300/0 conn(0x7f68200d29c0 msgr2=0x7f68200b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:49.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.435+0000 7f6824ef2640 1 --2- 172.21.15.83:0/937159724 >> v2:172.21.15.183:3300/0 conn(0x7f68200d29c0 0x7f68200b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.435+0000 7f6824ef2640 1 -- 172.21.15.83:0/937159724 >> v1:172.21.15.183:6789/0 conn(0x7f68200cf830 legacy=0x7f68200d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.435+0000 7f6824ef2640 1 -- 172.21.15.83:0/937159724 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68200b3710 con 0x7f682015dfe0 2024-03-06T07:22:49.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.435+0000 7f68256f3640 1 --2- 172.21.15.83:0/937159724 >> v2:172.21.15.183:3300/0 conn(0x7f68200d29c0 0x7f68200b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:49.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.436+0000 7f6824ef2640 1 --2- 172.21.15.83:0/937159724 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f682015dfe0 0x7f68200cf2f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f68140097a0 tx=0x7f681402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a80d367456b1d319 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.436+0000 7f681f7fe640 1 -- 172.21.15.83:0/937159724 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6814035020 con 0x7f682015dfe0 2024-03-06T07:22:49.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.436+0000 7f681f7fe640 1 -- 172.21.15.83:0/937159724 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6814031830 con 0x7f682015dfe0 2024-03-06T07:22:49.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.436+0000 7f681f7fe640 1 -- 172.21.15.83:0/937159724 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6814031be0 con 0x7f682015dfe0 2024-03-06T07:22:49.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.437+0000 7f6827156640 1 -- 172.21.15.83:0/937159724 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f682015dfe0 msgr2=0x7f68200cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.437+0000 7f6827156640 1 --2- 172.21.15.83:0/937159724 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f682015dfe0 0x7f68200cf2f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f68140097a0 tx=0x7f681402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.437+0000 7f6827156640 1 -- 172.21.15.83:0/937159724 shutdown_connections 2024-03-06T07:22:49.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.437+0000 7f6827156640 1 --2- 172.21.15.83:0/937159724 >> v2:172.21.15.183:3300/0 conn(0x7f68200d29c0 0x7f68200b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.437+0000 7f6827156640 1 --2- 172.21.15.83:0/937159724 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f682015dfe0 0x7f68200cf2f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.437+0000 7f6827156640 1 -- 172.21.15.83:0/937159724 >> 172.21.15.83:0/937159724 conn(0x7f68200d3cd0 msgr2=0x7f68200d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:49.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.437+0000 7f6827156640 1 -- 172.21.15.83:0/937159724 shutdown_connections 2024-03-06T07:22:49.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.437+0000 7f6827156640 1 -- 172.21.15.83:0/937159724 wait complete. 2024-03-06T07:22:49.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.438+0000 7f6827156640 1 Processor -- start 2024-03-06T07:22:49.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.438+0000 7f6827156640 1 -- start start 2024-03-06T07:22:49.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.439+0000 7f6827156640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f68200cf830 0x7f68200cb730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.439+0000 7f6827156640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f68200d29c0 0x7f68200b1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.439+0000 7f6827156640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f682015dfe0 0x7f682015afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.439+0000 7f6827156640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68200d1df0 con 0x7f682015dfe0 2024-03-06T07:22:49.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.439+0000 7f6827156640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68200d1c70 con 0x7f68200d29c0 2024-03-06T07:22:49.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.439+0000 7f6827156640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f682015e3c0 con 0x7f68200cf830 2024-03-06T07:22:49.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.439+0000 7f68256f3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f682015dfe0 0x7f682015afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.439+0000 7f68256f3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f682015dfe0 0x7f682015afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36596/0 (socket says 172.21.15.83:36596) 2024-03-06T07:22:49.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.439+0000 7f68256f3640 1 -- 172.21.15.83:0/860669610 learned_addr learned my addr 172.21.15.83:0/860669610 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:49.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.439+0000 7f681ffff640 1 --2- 172.21.15.83:0/860669610 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f68200d29c0 0x7f68200b1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.439+0000 7f6824ef2640 1 --2- 172.21.15.83:0/860669610 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f68200cf830 0x7f68200cb730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.439+0000 7f68256f3640 1 -- 172.21.15.83:0/860669610 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f68200cf830 msgr2=0x7f68200cb730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.439+0000 7f68256f3640 1 --2- 172.21.15.83:0/860669610 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f68200cf830 0x7f68200cb730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.439+0000 7f68256f3640 1 -- 172.21.15.83:0/860669610 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f68200d29c0 msgr2=0x7f68200b1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.439+0000 7f68256f3640 1 --2- 172.21.15.83:0/860669610 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f68200d29c0 0x7f68200b1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.439+0000 7f68256f3640 1 -- 172.21.15.83:0/860669610 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68200b6a70 con 0x7f682015dfe0 2024-03-06T07:22:49.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.439+0000 7f68256f3640 1 --2- 172.21.15.83:0/860669610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f682015dfe0 0x7f682015afe0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f681800efc0 tx=0x7f6818009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.440+0000 7f681d7fa640 1 -- 172.21.15.83:0/860669610 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6818018030 con 0x7f682015dfe0 2024-03-06T07:22:49.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.440+0000 7f681d7fa640 1 -- 172.21.15.83:0/860669610 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f681800cde0 con 0x7f682015dfe0 2024-03-06T07:22:49.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.440+0000 7f6827156640 1 -- 172.21.15.83:0/860669610 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68200cbc70 con 0x7f682015dfe0 2024-03-06T07:22:49.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.440+0000 7f681d7fa640 1 -- 172.21.15.83:0/860669610 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6818003bb0 con 0x7f682015dfe0 2024-03-06T07:22:49.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.440+0000 7f6827156640 1 -- 172.21.15.83:0/860669610 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68200b6d10 con 0x7f682015dfe0 2024-03-06T07:22:49.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.441+0000 7f6827156640 1 -- 172.21.15.83:0/860669610 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f682004f330 con 0x7f682015dfe0 2024-03-06T07:22:49.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.441+0000 7f681d7fa640 1 -- 172.21.15.83:0/860669610 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6818005d10 con 0x7f682015dfe0 2024-03-06T07:22:49.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.442+0000 7f681d7fa640 1 --2- 172.21.15.83:0/860669610 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f680406b4d0 0x7f680406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.442+0000 7f681d7fa640 1 -- 172.21.15.83:0/860669610 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5912+0+0 (secure 0 0 0) 0x7f681808ab20 con 0x7f682015dfe0 2024-03-06T07:22:49.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.442+0000 7f6824ef2640 1 --2- 172.21.15.83:0/860669610 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f680406b4d0 0x7f680406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.442+0000 7f6824ef2640 1 --2- 172.21.15.83:0/860669610 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f680406b4d0 0x7f680406d990 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6814036000 tx=0x7f681403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.450+0000 7f681d7fa640 1 -- 172.21.15.83:0/860669610 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6818022070 con 0x7f682015dfe0 2024-03-06T07:22:49.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.748+0000 7f6827156640 1 -- 172.21.15.83:0/860669610 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68200d2630 con 0x7f682015dfe0 2024-03-06T07:22:49.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.748+0000 7f681d7fa640 1 -- 172.21.15.83:0/860669610 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4974 (secure 0 0 0) 0x7f6818010040 con 0x7f682015dfe0 2024-03-06T07:22:49.750 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:22:49.751 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/4249608353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":4249608353},{"type":"v1","addr":"172.21.15.83:6836","nonce":4249608353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/714452530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":714452530},{"type":"v1","addr":"172.21.15.183:6835","nonce":714452530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4282,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/2723013431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2723013431},{"type":"v1","addr":"172.21.15.183:6834","nonce":2723013431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:58.286994+0000","modified":"2024-03-06T07:22:00.307284+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/1597344199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1597344199},{"type":"v1","addr":"172.21.15.83:6837","nonce":1597344199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4287,"qdb_cluster":[4287]},"id":1}]} 2024-03-06T07:22:49.751 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 6 2024-03-06T07:22:49.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.750+0000 7f6827156640 1 -- 172.21.15.83:0/860669610 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f680406b4d0 msgr2=0x7f680406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.750+0000 7f6827156640 1 --2- 172.21.15.83:0/860669610 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f680406b4d0 0x7f680406d990 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6814036000 tx=0x7f681403c040 comp rx=0 tx=0).stop 2024-03-06T07:22:49.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.750+0000 7f6827156640 1 -- 172.21.15.83:0/860669610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f682015dfe0 msgr2=0x7f682015afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.750+0000 7f6827156640 1 --2- 172.21.15.83:0/860669610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f682015dfe0 0x7f682015afe0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f681800efc0 tx=0x7f6818009f90 comp rx=0 tx=0).stop 2024-03-06T07:22:49.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.751+0000 7f6827156640 1 -- 172.21.15.83:0/860669610 shutdown_connections 2024-03-06T07:22:49.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.751+0000 7f6827156640 1 --2- 172.21.15.83:0/860669610 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f680406b4d0 0x7f680406d990 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.751+0000 7f6827156640 1 --2- 172.21.15.83:0/860669610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f682015dfe0 0x7f682015afe0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.751+0000 7f6827156640 1 --2- 172.21.15.83:0/860669610 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f68200d29c0 0x7f68200b1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.751+0000 7f6827156640 1 --2- 172.21.15.83:0/860669610 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f68200cf830 0x7f68200cb730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.751+0000 7f6827156640 1 -- 172.21.15.83:0/860669610 >> 172.21.15.83:0/860669610 conn(0x7f68200d3cd0 msgr2=0x7f682006d070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:49.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.751+0000 7f6827156640 1 -- 172.21.15.83:0/860669610 shutdown_connections 2024-03-06T07:22:49.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.751+0000 7f6827156640 1 -- 172.21.15.83:0/860669610 wait complete. 2024-03-06T07:22:49.763 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:49.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.863+0000 7f915bcc9640 1 Processor -- start 2024-03-06T07:22:49.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.863+0000 7f915bcc9640 1 -- start start 2024-03-06T07:22:49.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.864+0000 7f915bcc9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f915415bdd0 0x7f91540c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.864+0000 7f915bcc9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f915415fa10 0x7f91540ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.864+0000 7f915bcc9640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91540cebe0 con 0x7f91540d16e0 2024-03-06T07:22:49.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.864+0000 7f915bcc9640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91540ceec0 con 0x7f915415bdd0 2024-03-06T07:22:49.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.864+0000 7f915bcc9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91540cf1a0 con 0x7f915415fa10 2024-03-06T07:22:49.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.864+0000 7f9159a65640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f915415bdd0 0x7f91540c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.864+0000 7f915a266640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f91540d16e0 0x7f91540d1b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:47158/0 (socket says 172.21.15.83:47158) 2024-03-06T07:22:49.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.864+0000 7f915a266640 1 -- 172.21.15.83:0/1067337168 learned_addr learned my addr 172.21.15.83:0/1067337168 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:49.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.864+0000 7f9159a65640 1 -- 172.21.15.83:0/1067337168 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f915415fa10 msgr2=0x7f91540ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:49.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.864+0000 7f9159264640 1 --2- 172.21.15.83:0/1067337168 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f915415fa10 0x7f91540ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.864+0000 7f9159a65640 1 --2- 172.21.15.83:0/1067337168 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f915415fa10 0x7f91540ce270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.864+0000 7f9159a65640 1 -- 172.21.15.83:0/1067337168 >> v1:172.21.15.83:6789/0 conn(0x7f91540d16e0 legacy=0x7f91540d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.864+0000 7f9158a63640 1 -- 172.21.15.83:0/1067337168 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3174725985 0 0) 0x7f91540cebe0 con 0x7f91540d16e0 2024-03-06T07:22:49.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.864+0000 7f9159a65640 1 -- 172.21.15.83:0/1067337168 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91540cc8e0 con 0x7f915415bdd0 2024-03-06T07:22:49.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.865+0000 7f9159a65640 1 --2- 172.21.15.83:0/1067337168 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f915415bdd0 0x7f91540c5610 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f914c009d90 tx=0x7f914c02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5080ab1e9cc36b7d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.865+0000 7f9158a63640 1 -- 172.21.15.83:0/1067337168 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f914c039070 con 0x7f915415bdd0 2024-03-06T07:22:49.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.865+0000 7f9158a63640 1 -- 172.21.15.83:0/1067337168 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f914c02cbc0 con 0x7f915415bdd0 2024-03-06T07:22:49.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.865+0000 7f9158a63640 1 -- 172.21.15.83:0/1067337168 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f914c034440 con 0x7f915415bdd0 2024-03-06T07:22:49.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.866+0000 7f915bcc9640 1 -- 172.21.15.83:0/1067337168 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f915415bdd0 msgr2=0x7f91540c5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.866+0000 7f915bcc9640 1 --2- 172.21.15.83:0/1067337168 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f915415bdd0 0x7f91540c5610 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f914c009d90 tx=0x7f914c02be10 comp rx=0 tx=0).stop 2024-03-06T07:22:49.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.866+0000 7f915bcc9640 1 -- 172.21.15.83:0/1067337168 shutdown_connections 2024-03-06T07:22:49.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.866+0000 7f915bcc9640 1 --2- 172.21.15.83:0/1067337168 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f915415fa10 0x7f91540ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.866+0000 7f915bcc9640 1 --2- 172.21.15.83:0/1067337168 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f915415bdd0 0x7f91540c5610 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.866+0000 7f915bcc9640 1 -- 172.21.15.83:0/1067337168 >> 172.21.15.83:0/1067337168 conn(0x7f915409a9f0 msgr2=0x7f915409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:49.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.866+0000 7f915bcc9640 1 -- 172.21.15.83:0/1067337168 shutdown_connections 2024-03-06T07:22:49.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.866+0000 7f915bcc9640 1 -- 172.21.15.83:0/1067337168 wait complete. 2024-03-06T07:22:49.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.867+0000 7f915bcc9640 1 Processor -- start 2024-03-06T07:22:49.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.867+0000 7f915bcc9640 1 -- start start 2024-03-06T07:22:49.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.868+0000 7f915bcc9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f91540d16e0 0x7f91541f3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.868+0000 7f915bcc9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f915415bdd0 0x7f9154153100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.868+0000 7f915bcc9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f915415fa10 0x7f915415eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.868+0000 7f915bcc9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f915415c1b0 con 0x7f915415fa10 2024-03-06T07:22:49.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.868+0000 7f915bcc9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91540cc5d0 con 0x7f91540d16e0 2024-03-06T07:22:49.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.868+0000 7f915bcc9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91540ce7b0 con 0x7f915415bdd0 2024-03-06T07:22:49.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.868+0000 7f915a266640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f915415fa10 0x7f915415eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.868+0000 7f915a266640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f915415fa10 0x7f915415eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36622/0 (socket says 172.21.15.83:36622) 2024-03-06T07:22:49.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.868+0000 7f915a266640 1 -- 172.21.15.83:0/919417322 learned_addr learned my addr 172.21.15.83:0/919417322 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:49.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.868+0000 7f9159a65640 1 --2- 172.21.15.83:0/919417322 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f91540d16e0 0x7f91541f3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.868+0000 7f915a266640 1 -- 172.21.15.83:0/919417322 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f915415bdd0 msgr2=0x7f9154153100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.868+0000 7f9159264640 1 --2- 172.21.15.83:0/919417322 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f915415bdd0 0x7f9154153100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.868+0000 7f915a266640 1 --2- 172.21.15.83:0/919417322 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f915415bdd0 0x7f9154153100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.868+0000 7f915a266640 1 -- 172.21.15.83:0/919417322 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f91540d16e0 msgr2=0x7f91541f3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.868+0000 7f915a266640 1 --2- 172.21.15.83:0/919417322 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f91540d16e0 0x7f91541f3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.868+0000 7f915a266640 1 -- 172.21.15.83:0/919417322 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91540c7430 con 0x7f915415fa10 2024-03-06T07:22:49.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.868+0000 7f9159a65640 1 --2- 172.21.15.83:0/919417322 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f91540d16e0 0x7f91541f3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:49.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.868+0000 7f915a266640 1 --2- 172.21.15.83:0/919417322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f915415fa10 0x7f915415eb40 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f915000c450 tx=0x7f915000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.869+0000 7f914a7fc640 1 -- 172.21.15.83:0/919417322 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9150018070 con 0x7f915415fa10 2024-03-06T07:22:49.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.869+0000 7f914a7fc640 1 -- 172.21.15.83:0/919417322 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9150007500 con 0x7f915415fa10 2024-03-06T07:22:49.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.869+0000 7f915bcc9640 1 -- 172.21.15.83:0/919417322 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91540c68b0 con 0x7f915415fa10 2024-03-06T07:22:49.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.869+0000 7f914a7fc640 1 -- 172.21.15.83:0/919417322 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9150004a20 con 0x7f915415fa10 2024-03-06T07:22:49.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.869+0000 7f915bcc9640 1 -- 172.21.15.83:0/919417322 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91540c6e70 con 0x7f915415fa10 2024-03-06T07:22:49.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.870+0000 7f915bcc9640 1 -- 172.21.15.83:0/919417322 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f915404f330 con 0x7f915415fa10 2024-03-06T07:22:49.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.870+0000 7f914a7fc640 1 -- 172.21.15.83:0/919417322 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f915001b070 con 0x7f915415fa10 2024-03-06T07:22:49.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.871+0000 7f914a7fc640 1 --2- 172.21.15.83:0/919417322 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f913806b4d0 0x7f913806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.871+0000 7f914a7fc640 1 -- 172.21.15.83:0/919417322 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5912+0+0 (secure 0 0 0) 0x7f91500894e0 con 0x7f915415fa10 2024-03-06T07:22:49.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.871+0000 7f9159a65640 1 --2- 172.21.15.83:0/919417322 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f913806b4d0 0x7f913806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.871+0000 7f9159a65640 1 --2- 172.21.15.83:0/919417322 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f913806b4d0 0x7f913806d990 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f914c036000 tx=0x7f914c030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:49.875+0000 7f914a7fc640 1 -- 172.21.15.83:0/919417322 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9150058290 con 0x7f915415fa10 2024-03-06T07:22:50.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.084+0000 7f915bcc9640 1 -- 172.21.15.83:0/919417322 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91540d13d0 con 0x7f915415fa10 2024-03-06T07:22:50.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.085+0000 7f914a7fc640 1 -- 172.21.15.83:0/919417322 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4974 (secure 0 0 0) 0x7f9150057b90 con 0x7f915415fa10 2024-03-06T07:22:50.087 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:22:50.087 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/4249608353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":4249608353},{"type":"v1","addr":"172.21.15.83:6836","nonce":4249608353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/714452530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":714452530},{"type":"v1","addr":"172.21.15.183:6835","nonce":714452530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4282,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/2723013431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2723013431},{"type":"v1","addr":"172.21.15.183:6834","nonce":2723013431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:58.286994+0000","modified":"2024-03-06T07:22:00.307284+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/1597344199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1597344199},{"type":"v1","addr":"172.21.15.83:6837","nonce":1597344199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4287,"qdb_cluster":[4287]},"id":1}]} 2024-03-06T07:22:50.087 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 6 2024-03-06T07:22:50.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.087+0000 7f915bcc9640 1 -- 172.21.15.83:0/919417322 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f913806b4d0 msgr2=0x7f913806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.087+0000 7f915bcc9640 1 --2- 172.21.15.83:0/919417322 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f913806b4d0 0x7f913806d990 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f914c036000 tx=0x7f914c030000 comp rx=0 tx=0).stop 2024-03-06T07:22:50.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.087+0000 7f915bcc9640 1 -- 172.21.15.83:0/919417322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f915415fa10 msgr2=0x7f915415eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.087+0000 7f915bcc9640 1 --2- 172.21.15.83:0/919417322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f915415fa10 0x7f915415eb40 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f915000c450 tx=0x7f915000c910 comp rx=0 tx=0).stop 2024-03-06T07:22:50.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.087+0000 7f915bcc9640 1 -- 172.21.15.83:0/919417322 shutdown_connections 2024-03-06T07:22:50.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.087+0000 7f915bcc9640 1 --2- 172.21.15.83:0/919417322 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f913806b4d0 0x7f913806d990 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.087+0000 7f915bcc9640 1 --2- 172.21.15.83:0/919417322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f915415fa10 0x7f915415eb40 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.087+0000 7f915bcc9640 1 --2- 172.21.15.83:0/919417322 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f915415bdd0 0x7f9154153100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.087+0000 7f915bcc9640 1 --2- 172.21.15.83:0/919417322 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f91540d16e0 0x7f91541f3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.087+0000 7f915bcc9640 1 -- 172.21.15.83:0/919417322 >> 172.21.15.83:0/919417322 conn(0x7f915409a9f0 msgr2=0x7f91540c6030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:50.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.087+0000 7f915bcc9640 1 -- 172.21.15.83:0/919417322 shutdown_connections 2024-03-06T07:22:50.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.087+0000 7f915bcc9640 1 -- 172.21.15.83:0/919417322 wait complete. 2024-03-06T07:22:50.100 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:50.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.201+0000 7fdd5b535640 1 Processor -- start 2024-03-06T07:22:50.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.201+0000 7fdd5b535640 1 -- start start 2024-03-06T07:22:50.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.202+0000 7fdd5b535640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd540d04a0 0x7fdd540ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.202+0000 7fdd5b535640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd540d0910 con 0x7fdd540cfff0 2024-03-06T07:22:50.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.202+0000 7fdd5b535640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd54161980 con 0x7fdd5415bb20 2024-03-06T07:22:50.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.202+0000 7fdd5b535640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd540d2500 con 0x7fdd540d04a0 2024-03-06T07:22:50.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.202+0000 7fdd58ad0640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fdd540cfff0 0x7fdd540cc8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:47170/0 (socket says 172.21.15.83:47170) 2024-03-06T07:22:50.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.202+0000 7fdd58ad0640 1 -- 172.21.15.83:0/2615832119 learned_addr learned my addr 172.21.15.83:0/2615832119 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:50.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.202+0000 7fdd59ad2640 1 --2- 172.21.15.83:0/2615832119 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd540d04a0 0x7fdd540ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.202+0000 7fdd4bfff640 1 -- 172.21.15.83:0/2615832119 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1353806025 0 0) 0x7fdd540d0910 con 0x7fdd540cfff0 2024-03-06T07:22:50.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.202+0000 7fdd4bfff640 1 -- 172.21.15.83:0/2615832119 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd40003520 con 0x7fdd540cfff0 2024-03-06T07:22:50.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.202+0000 7fdd4bfff640 1 -- 172.21.15.83:0/2615832119 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 794239033 0 0) 0x7fdd54161980 con 0x7fdd5415bb20 2024-03-06T07:22:50.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.202+0000 7fdd4bfff640 1 -- 172.21.15.83:0/2615832119 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd540d0910 con 0x7fdd5415bb20 2024-03-06T07:22:50.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.202+0000 7fdd4bfff640 1 -- 172.21.15.83:0/2615832119 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3188267894 0 0) 0x7fdd40003520 con 0x7fdd540cfff0 2024-03-06T07:22:50.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.202+0000 7fdd4bfff640 1 -- 172.21.15.83:0/2615832119 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd540d04a0 msgr2=0x7fdd540ccfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:50.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.202+0000 7fdd4bfff640 1 --2- 172.21.15.83:0/2615832119 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd540d04a0 0x7fdd540ccfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.202+0000 7fdd4bfff640 1 -- 172.21.15.83:0/2615832119 >> v1:172.21.15.183:6789/0 conn(0x7fdd5415bb20 legacy=0x7fdd540d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.202+0000 7fdd4bfff640 1 -- 172.21.15.83:0/2615832119 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd54161c60 con 0x7fdd540cfff0 2024-03-06T07:22:50.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.202+0000 7fdd4bfff640 1 -- 172.21.15.83:0/2615832119 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7fdd44001670 con 0x7fdd540cfff0 2024-03-06T07:22:50.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.202+0000 7fdd4bfff640 1 -- 172.21.15.83:0/2615832119 >> v1:172.21.15.83:6789/0 conn(0x7fdd540cfff0 legacy=0x7fdd540cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.202+0000 7fdd4bfff640 1 --2- 172.21.15.83:0/2615832119 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd40003b70 0x7fdd40003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.202+0000 7fdd4bfff640 1 --2- 172.21.15.83:0/2615832119 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd540d04a0 0x7fdd40004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.202+0000 7fdd4bfff640 1 --2- 172.21.15.83:0/2615832119 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd40004d40 0x7fdd400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.202+0000 7fdd4bfff640 1 -- 172.21.15.83:0/2615832119 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd540d2500 con 0x7fdd40003b70 2024-03-06T07:22:50.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.202+0000 7fdd4bfff640 1 -- 172.21.15.83:0/2615832119 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4000ba70 con 0x7fdd40004d40 2024-03-06T07:22:50.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.202+0000 7fdd4bfff640 1 -- 172.21.15.83:0/2615832119 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4000bc20 con 0x7fdd540d04a0 2024-03-06T07:22:50.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.203+0000 7fdd592d1640 1 --2- 172.21.15.83:0/2615832119 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd40003b70 0x7fdd40003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.203+0000 7fdd59ad2640 1 --2- 172.21.15.83:0/2615832119 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd40004d40 0x7fdd400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.203+0000 7fdd58ad0640 1 --2- 172.21.15.83:0/2615832119 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd540d04a0 0x7fdd40004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.203+0000 7fdd592d1640 1 -- 172.21.15.83:0/2615832119 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd540d04a0 msgr2=0x7fdd40004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:50.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.203+0000 7fdd592d1640 1 --2- 172.21.15.83:0/2615832119 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd540d04a0 0x7fdd40004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.203+0000 7fdd592d1640 1 -- 172.21.15.83:0/2615832119 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd40004d40 msgr2=0x7fdd400051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:50.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.203+0000 7fdd592d1640 1 --2- 172.21.15.83:0/2615832119 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd40004d40 0x7fdd400051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.203+0000 7fdd592d1640 1 -- 172.21.15.83:0/2615832119 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd40003520 con 0x7fdd40003b70 2024-03-06T07:22:50.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.204+0000 7fdd592d1640 1 --2- 172.21.15.83:0/2615832119 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd40003b70 0x7fdd40003f70 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fdd500018d0 tx=0x7fdd40005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4867ac3ffb5959d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:50.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.204+0000 7fdd4bfff640 1 -- 172.21.15.83:0/2615832119 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd50037070 con 0x7fdd40003b70 2024-03-06T07:22:50.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.204+0000 7fdd4bfff640 1 -- 172.21.15.83:0/2615832119 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd500052d0 con 0x7fdd40003b70 2024-03-06T07:22:50.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.204+0000 7fdd4bfff640 1 -- 172.21.15.83:0/2615832119 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd500014f0 con 0x7fdd40003b70 2024-03-06T07:22:50.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.204+0000 7fdd5b535640 1 -- 172.21.15.83:0/2615832119 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd40003b70 msgr2=0x7fdd40003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.204+0000 7fdd5b535640 1 --2- 172.21.15.83:0/2615832119 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd40003b70 0x7fdd40003f70 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fdd500018d0 tx=0x7fdd40005830 comp rx=0 tx=0).stop 2024-03-06T07:22:50.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.205+0000 7fdd59ad2640 1 -- 172.21.15.83:0/2615832119 reap_dead start 2024-03-06T07:22:50.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.205+0000 7fdd5b535640 1 -- 172.21.15.83:0/2615832119 shutdown_connections 2024-03-06T07:22:50.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.205+0000 7fdd5b535640 1 -- 172.21.15.83:0/2615832119 >> 172.21.15.83:0/2615832119 conn(0x7fdd5409a9f0 msgr2=0x7fdd5409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:50.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.205+0000 7fdd5b535640 1 -- 172.21.15.83:0/2615832119 shutdown_connections 2024-03-06T07:22:50.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.205+0000 7fdd5b535640 1 -- 172.21.15.83:0/2615832119 wait complete. 2024-03-06T07:22:50.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.206+0000 7fdd5b535640 1 Processor -- start 2024-03-06T07:22:50.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.206+0000 7fdd5b535640 1 -- start start 2024-03-06T07:22:50.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.206+0000 7fdd5b535640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd40003b70 0x7fdd54155960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.206+0000 7fdd5b535640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd540ca590 0x7fdd5414aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.206+0000 7fdd5b535640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd54159e10 0x7fdd5415a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.206+0000 7fdd5b535640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd54162500 con 0x7fdd40003b70 2024-03-06T07:22:50.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.206+0000 7fdd5b535640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd54162220 con 0x7fdd540ca590 2024-03-06T07:22:50.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.206+0000 7fdd5b535640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd540d1980 con 0x7fdd54159e10 2024-03-06T07:22:50.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.207+0000 7fdd592d1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd40003b70 0x7fdd54155960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.207+0000 7fdd592d1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd40003b70 0x7fdd54155960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36644/0 (socket says 172.21.15.83:36644) 2024-03-06T07:22:50.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.207+0000 7fdd592d1640 1 -- 172.21.15.83:0/2185811159 learned_addr learned my addr 172.21.15.83:0/2185811159 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:50.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.207+0000 7fdd58ad0640 1 --2- 172.21.15.83:0/2185811159 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd540ca590 0x7fdd5414aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.207+0000 7fdd59ad2640 1 --2- 172.21.15.83:0/2185811159 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd54159e10 0x7fdd5415a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.207+0000 7fdd592d1640 1 -- 172.21.15.83:0/2185811159 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd54159e10 msgr2=0x7fdd5415a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.207+0000 7fdd592d1640 1 --2- 172.21.15.83:0/2185811159 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd54159e10 0x7fdd5415a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.207+0000 7fdd592d1640 1 -- 172.21.15.83:0/2185811159 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd540ca590 msgr2=0x7fdd5414aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.207+0000 7fdd592d1640 1 --2- 172.21.15.83:0/2185811159 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd540ca590 0x7fdd5414aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.207+0000 7fdd592d1640 1 -- 172.21.15.83:0/2185811159 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd540caa10 con 0x7fdd40003b70 2024-03-06T07:22:50.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.207+0000 7fdd592d1640 1 --2- 172.21.15.83:0/2185811159 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd40003b70 0x7fdd54155960 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fdd50035040 tx=0x7fdd500012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:50.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.207+0000 7fdd49ffb640 1 -- 172.21.15.83:0/2185811159 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd50037070 con 0x7fdd40003b70 2024-03-06T07:22:50.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.207+0000 7fdd49ffb640 1 -- 172.21.15.83:0/2185811159 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd50038040 con 0x7fdd40003b70 2024-03-06T07:22:50.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.207+0000 7fdd5b535640 1 -- 172.21.15.83:0/2185811159 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd540cac90 con 0x7fdd40003b70 2024-03-06T07:22:50.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.207+0000 7fdd49ffb640 1 -- 172.21.15.83:0/2185811159 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd500014f0 con 0x7fdd40003b70 2024-03-06T07:22:50.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.207+0000 7fdd5b535640 1 -- 172.21.15.83:0/2185811159 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd540cb250 con 0x7fdd40003b70 2024-03-06T07:22:50.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.209+0000 7fdd5b535640 1 -- 172.21.15.83:0/2185811159 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd5406f970 con 0x7fdd40003b70 2024-03-06T07:22:50.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.209+0000 7fdd49ffb640 1 -- 172.21.15.83:0/2185811159 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdd500052d0 con 0x7fdd40003b70 2024-03-06T07:22:50.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.209+0000 7fdd49ffb640 1 --2- 172.21.15.83:0/2185811159 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdd3806b500 0x7fdd3806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.210+0000 7fdd49ffb640 1 -- 172.21.15.83:0/2185811159 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5912+0+0 (secure 0 0 0) 0x7fdd500ab990 con 0x7fdd40003b70 2024-03-06T07:22:50.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.210+0000 7fdd58ad0640 1 --2- 172.21.15.83:0/2185811159 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdd3806b500 0x7fdd3806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.210+0000 7fdd58ad0640 1 --2- 172.21.15.83:0/2185811159 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdd3806b500 0x7fdd3806d9c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fdd44001c60 tx=0x7fdd44001e40 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:50.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.214+0000 7fdd49ffb640 1 -- 172.21.15.83:0/2185811159 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdd5406f970 con 0x7fdd40003b70 2024-03-06T07:22:50.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.393+0000 7fdd5b535640 1 -- 172.21.15.83:0/2185811159 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd5404f330 con 0x7fdd40003b70 2024-03-06T07:22:50.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.394+0000 7fdd49ffb640 1 -- 172.21.15.83:0/2185811159 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+17006 (secure 0 0 0) 0x7fdd540caa10 con 0x7fdd40003b70 2024-03-06T07:22:50.396 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:22:50.396 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":22,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:22:40.105898+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.318942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.254795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.1f","mappings":[{"from":5,"to":1}]},{"pgid":"3.29","mappings":[{"from":3,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:50.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.396+0000 7fdd5b535640 1 -- 172.21.15.83:0/2185811159 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdd3806b500 msgr2=0x7fdd3806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.396+0000 7fdd5b535640 1 --2- 172.21.15.83:0/2185811159 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdd3806b500 0x7fdd3806d9c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fdd44001c60 tx=0x7fdd44001e40 comp rx=0 tx=0).stop 2024-03-06T07:22:50.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.396+0000 7fdd5b535640 1 -- 172.21.15.83:0/2185811159 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd40003b70 msgr2=0x7fdd54155960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.396+0000 7fdd5b535640 1 --2- 172.21.15.83:0/2185811159 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd40003b70 0x7fdd54155960 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fdd50035040 tx=0x7fdd500012d0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.396+0000 7fdd5b535640 1 -- 172.21.15.83:0/2185811159 shutdown_connections 2024-03-06T07:22:50.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.396+0000 7fdd5b535640 1 --2- 172.21.15.83:0/2185811159 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdd3806b500 0x7fdd3806d9c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.396+0000 7fdd5b535640 1 --2- 172.21.15.83:0/2185811159 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd54159e10 0x7fdd5415a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.396+0000 7fdd5b535640 1 --2- 172.21.15.83:0/2185811159 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd540ca590 0x7fdd5414aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.396+0000 7fdd5b535640 1 --2- 172.21.15.83:0/2185811159 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd40003b70 0x7fdd54155960 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.396+0000 7fdd5b535640 1 -- 172.21.15.83:0/2185811159 >> 172.21.15.83:0/2185811159 conn(0x7fdd5409a9f0 msgr2=0x7fdd5406f260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:50.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.396+0000 7fdd5b535640 1 -- 172.21.15.83:0/2185811159 shutdown_connections 2024-03-06T07:22:50.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.396+0000 7fdd5b535640 1 -- 172.21.15.83:0/2185811159 wait complete. 2024-03-06T07:22:50.409 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:22:50.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.508+0000 7ff350bc7640 1 Processor -- start 2024-03-06T07:22:50.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.508+0000 7ff350bc7640 1 -- start start 2024-03-06T07:22:50.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.509+0000 7ff350bc7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff34c0d0d90 0x7ff34c0ce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.509+0000 7ff350bc7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff34c0d25e0 con 0x7ff34c0c90f0 2024-03-06T07:22:50.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.509+0000 7ff350bc7640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff34c0d2880 con 0x7ff34c15e000 2024-03-06T07:22:50.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.509+0000 7ff350bc7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff34c0c7c50 con 0x7ff34c0d0d90 2024-03-06T07:22:50.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.509+0000 7ff34ad9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff34c0c90f0 0x7ff34c0ced00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:47182/0 (socket says 172.21.15.83:47182) 2024-03-06T07:22:50.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.509+0000 7ff34ad9d640 1 -- 172.21.15.83:0/4052090088 learned_addr learned my addr 172.21.15.83:0/4052090088 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:50.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.509+0000 7ff341d9b640 1 --2- 172.21.15.83:0/4052090088 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff34c0d0d90 0x7ff34c0ce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.509+0000 7ff349d9b640 1 -- 172.21.15.83:0/4052090088 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1110294949 0 0) 0x7ff34c0d25e0 con 0x7ff34c0c90f0 2024-03-06T07:22:50.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.509+0000 7ff349d9b640 1 -- 172.21.15.83:0/4052090088 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff338003520 con 0x7ff34c0c90f0 2024-03-06T07:22:50.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.509+0000 7ff349d9b640 1 -- 172.21.15.83:0/4052090088 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1639246533 0 0) 0x7ff34c0d2880 con 0x7ff34c15e000 2024-03-06T07:22:50.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.509+0000 7ff349d9b640 1 -- 172.21.15.83:0/4052090088 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff34c0d25e0 con 0x7ff34c15e000 2024-03-06T07:22:50.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.509+0000 7ff341d9b640 1 -- 172.21.15.83:0/4052090088 >> v1:172.21.15.183:6789/0 conn(0x7ff34c15e000 legacy=0x7ff34c0d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.509+0000 7ff341d9b640 1 -- 172.21.15.83:0/4052090088 >> v1:172.21.15.83:6789/0 conn(0x7ff34c0c90f0 legacy=0x7ff34c0ced00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.509+0000 7ff341d9b640 1 -- 172.21.15.83:0/4052090088 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff34c0d2b40 con 0x7ff34c0d0d90 2024-03-06T07:22:50.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.510+0000 7ff341d9b640 1 --2- 172.21.15.83:0/4052090088 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff34c0d0d90 0x7ff34c0ce7c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff334009800 tx=0x7ff33402b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2235217f938093be server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:50.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.510+0000 7ff349d9b640 1 -- 172.21.15.83:0/4052090088 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff334034070 con 0x7ff34c0d0d90 2024-03-06T07:22:50.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.510+0000 7ff349d9b640 1 -- 172.21.15.83:0/4052090088 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff334030880 con 0x7ff34c0d0d90 2024-03-06T07:22:50.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.510+0000 7ff349d9b640 1 -- 172.21.15.83:0/4052090088 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff334030c30 con 0x7ff34c0d0d90 2024-03-06T07:22:50.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.511+0000 7ff350bc7640 1 -- 172.21.15.83:0/4052090088 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff34c0d0d90 msgr2=0x7ff34c0ce7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.511+0000 7ff350bc7640 1 --2- 172.21.15.83:0/4052090088 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff34c0d0d90 0x7ff34c0ce7c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff334009800 tx=0x7ff33402b9c0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.511+0000 7ff350bc7640 1 -- 172.21.15.83:0/4052090088 shutdown_connections 2024-03-06T07:22:50.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.511+0000 7ff350bc7640 1 --2- 172.21.15.83:0/4052090088 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff34c0d0d90 0x7ff34c0ce7c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.511+0000 7ff350bc7640 1 -- 172.21.15.83:0/4052090088 >> 172.21.15.83:0/4052090088 conn(0x7ff34c0d3cf0 msgr2=0x7ff34c0d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:50.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.511+0000 7ff350bc7640 1 -- 172.21.15.83:0/4052090088 shutdown_connections 2024-03-06T07:22:50.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.511+0000 7ff350bc7640 1 -- 172.21.15.83:0/4052090088 wait complete. 2024-03-06T07:22:50.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.513+0000 7ff350bc7640 1 Processor -- start 2024-03-06T07:22:50.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.513+0000 7ff350bc7640 1 -- start start 2024-03-06T07:22:50.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.514+0000 7ff350bc7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff34c0c90f0 0x7ff34c1f3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.514+0000 7ff350bc7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff34c0d0d90 0x7ff34c09da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.514+0000 7ff350bc7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff34c15e000 0x7ff34c09dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.514+0000 7ff350bc7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff34c15e3e0 con 0x7ff34c0c90f0 2024-03-06T07:22:50.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.514+0000 7ff350bc7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff34c0d22d0 con 0x7ff34c15e000 2024-03-06T07:22:50.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.514+0000 7ff350bc7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff34c0d4810 con 0x7ff34c0d0d90 2024-03-06T07:22:50.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.514+0000 7ff34a59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff34c0c90f0 0x7ff34c1f3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.514+0000 7ff34a59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff34c0c90f0 0x7ff34c1f3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36650/0 (socket says 172.21.15.83:36650) 2024-03-06T07:22:50.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.514+0000 7ff34a59c640 1 -- 172.21.15.83:0/466655266 learned_addr learned my addr 172.21.15.83:0/466655266 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:50.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.515+0000 7ff341d9b640 1 --2- 172.21.15.83:0/466655266 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff34c0d0d90 0x7ff34c09da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.515+0000 7ff34ad9d640 1 --2- 172.21.15.83:0/466655266 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff34c15e000 0x7ff34c09dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.515+0000 7ff34a59c640 1 -- 172.21.15.83:0/466655266 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff34c0d0d90 msgr2=0x7ff34c09da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.515+0000 7ff34a59c640 1 --2- 172.21.15.83:0/466655266 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff34c0d0d90 0x7ff34c09da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.515+0000 7ff34a59c640 1 -- 172.21.15.83:0/466655266 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff34c15e000 msgr2=0x7ff34c09dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.515+0000 7ff34a59c640 1 --2- 172.21.15.83:0/466655266 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff34c15e000 0x7ff34c09dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.515+0000 7ff34a59c640 1 -- 172.21.15.83:0/466655266 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff34c1f4400 con 0x7ff34c0c90f0 2024-03-06T07:22:50.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.515+0000 7ff341d9b640 1 --2- 172.21.15.83:0/466655266 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff34c0d0d90 0x7ff34c09da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:50.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.515+0000 7ff34a59c640 1 --2- 172.21.15.83:0/466655266 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff34c0c90f0 0x7ff34c1f3ec0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ff33c00c460 tx=0x7ff33c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:50.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.515+0000 7ff3437fe640 1 -- 172.21.15.83:0/466655266 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33c017070 con 0x7ff34c0c90f0 2024-03-06T07:22:50.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.515+0000 7ff3437fe640 1 -- 172.21.15.83:0/466655266 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff33c0052f0 con 0x7ff34c0c90f0 2024-03-06T07:22:50.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.515+0000 7ff350bc7640 1 -- 172.21.15.83:0/466655266 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff34c09f9d0 con 0x7ff34c0c90f0 2024-03-06T07:22:50.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.515+0000 7ff3437fe640 1 -- 172.21.15.83:0/466655266 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33c003830 con 0x7ff34c0c90f0 2024-03-06T07:22:50.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.515+0000 7ff350bc7640 1 -- 172.21.15.83:0/466655266 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff34c0ccf70 con 0x7ff34c0c90f0 2024-03-06T07:22:50.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.516+0000 7ff350bc7640 1 -- 172.21.15.83:0/466655266 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff34c04f330 con 0x7ff34c0c90f0 2024-03-06T07:22:50.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.517+0000 7ff3437fe640 1 -- 172.21.15.83:0/466655266 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff33c01a070 con 0x7ff34c0c90f0 2024-03-06T07:22:50.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.517+0000 7ff3437fe640 1 --2- 172.21.15.83:0/466655266 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff32c06b4d0 0x7ff32c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.517+0000 7ff3437fe640 1 -- 172.21.15.83:0/466655266 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5912+0+0 (secure 0 0 0) 0x7ff33c089690 con 0x7ff34c0c90f0 2024-03-06T07:22:50.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.517+0000 7ff341d9b640 1 --2- 172.21.15.83:0/466655266 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff32c06b4d0 0x7ff32c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.517+0000 7ff341d9b640 1 --2- 172.21.15.83:0/466655266 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff32c06b4d0 0x7ff32c06d990 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff334036000 tx=0x7ff33403b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:50.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.525+0000 7ff3437fe640 1 -- 172.21.15.83:0/466655266 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff34c09f9d0 con 0x7ff34c0c90f0 2024-03-06T07:22:50.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.749+0000 7ff350bc7640 1 -- 172.21.15.83:0/466655266 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7ff34c0747a0 con 0x7ff34c0c90f0 2024-03-06T07:22:50.751 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-03-06T07:22:50.749+0000 7fb0dae11640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:22:50.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.751+0000 7ff3437fe640 1 -- 172.21.15.83:0/466655266 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v7) v1 ==== 161+0+0 (secure 0 0 0) 0x7ff33c058450 con 0x7ff34c0c90f0 2024-03-06T07:22:50.753 INFO:teuthology.orchestra.run.smithi083.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:22:50.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.753+0000 7ff350bc7640 1 -- 172.21.15.83:0/466655266 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff32c06b4d0 msgr2=0x7ff32c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.753+0000 7ff350bc7640 1 --2- 172.21.15.83:0/466655266 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff32c06b4d0 0x7ff32c06d990 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff334036000 tx=0x7ff33403b040 comp rx=0 tx=0).stop 2024-03-06T07:22:50.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.753+0000 7ff350bc7640 1 -- 172.21.15.83:0/466655266 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff34c0c90f0 msgr2=0x7ff34c1f3ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.753+0000 7ff350bc7640 1 --2- 172.21.15.83:0/466655266 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff34c0c90f0 0x7ff34c1f3ec0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ff33c00c460 tx=0x7ff33c00c920 comp rx=0 tx=0).stop 2024-03-06T07:22:50.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.753+0000 7ff350bc7640 1 -- 172.21.15.83:0/466655266 shutdown_connections 2024-03-06T07:22:50.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.753+0000 7ff350bc7640 1 --2- 172.21.15.83:0/466655266 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff32c06b4d0 0x7ff32c06d990 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.753+0000 7ff350bc7640 1 --2- 172.21.15.83:0/466655266 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff34c15e000 0x7ff34c09dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.753+0000 7ff350bc7640 1 --2- 172.21.15.83:0/466655266 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff34c0d0d90 0x7ff34c09da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.753+0000 7ff350bc7640 1 --2- 172.21.15.83:0/466655266 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff34c0c90f0 0x7ff34c1f3ec0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.753+0000 7ff350bc7640 1 -- 172.21.15.83:0/466655266 >> 172.21.15.83:0/466655266 conn(0x7ff34c0d3cf0 msgr2=0x7ff34c06b990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:50.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.753+0000 7ff350bc7640 1 -- 172.21.15.83:0/466655266 shutdown_connections 2024-03-06T07:22:50.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.753+0000 7ff350bc7640 1 -- 172.21.15.83:0/466655266 wait complete. 2024-03-06T07:22:50.758 INFO:tasks.ceph.mds.c.smithi083.stderr: -4406> 2024-03-06T07:21:51.898+0000 7f363946a640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-03-06T07:22:50.758 INFO:tasks.ceph.mds.c.smithi083.stderr: -3209> 2024-03-06T07:22:01.919+0000 7f363345e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:22:50.768 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:22:50.786 INFO:tasks.ceph.mds.c.smithi083.stdout:starting mds.c at 2024-03-06T07:22:50.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.869+0000 7f1385f79640 1 Processor -- start 2024-03-06T07:22:50.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.869+0000 7f1385f79640 1 -- start start 2024-03-06T07:22:50.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.870+0000 7f1385f79640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f13800b4970 0x7f138006d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.870+0000 7f1385f79640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f138006d5c0 0x7f13800cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.870+0000 7f1385f79640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f138015ff90 con 0x7f138015bb40 2024-03-06T07:22:50.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.870+0000 7f1385f79640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13800b4dd0 con 0x7f13800b4970 2024-03-06T07:22:50.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.870+0000 7f1385f79640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f138006c910 con 0x7f138006d5c0 2024-03-06T07:22:50.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.870+0000 7f137effd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f13800b4970 0x7f138006d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.870+0000 7f137f7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f138015bb40 0x7f138015f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:47200/0 (socket says 172.21.15.83:47200) 2024-03-06T07:22:50.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.870+0000 7f137f7fe640 1 -- 172.21.15.83:0/2143822317 learned_addr learned my addr 172.21.15.83:0/2143822317 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:50.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.870+0000 7f137ffff640 1 --2- 172.21.15.83:0/2143822317 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f138006d5c0 0x7f13800cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.870+0000 7f137effd640 1 -- 172.21.15.83:0/2143822317 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f138006d5c0 msgr2=0x7f13800cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:50.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.870+0000 7f137effd640 1 --2- 172.21.15.83:0/2143822317 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f138006d5c0 0x7f13800cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.870+0000 7f137effd640 1 -- 172.21.15.83:0/2143822317 >> v1:172.21.15.83:6789/0 conn(0x7f138015bb40 legacy=0x7f138015f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.870+0000 7f137e7fc640 1 -- 172.21.15.83:0/2143822317 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3529345126 0 0) 0x7f138015ff90 con 0x7f138015bb40 2024-03-06T07:22:50.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.870+0000 7f137effd640 1 -- 172.21.15.83:0/2143822317 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1380160270 con 0x7f13800b4970 2024-03-06T07:22:50.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.871+0000 7f137effd640 1 --2- 172.21.15.83:0/2143822317 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13800b4970 0x7f138006d080 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f13700097a0 tx=0x7f137002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d86b6b1186da560a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:50.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.871+0000 7f137e7fc640 1 -- 172.21.15.83:0/2143822317 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1370035020 con 0x7f13800b4970 2024-03-06T07:22:50.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.871+0000 7f137e7fc640 1 -- 172.21.15.83:0/2143822317 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1370031830 con 0x7f13800b4970 2024-03-06T07:22:50.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.871+0000 7f137e7fc640 1 -- 172.21.15.83:0/2143822317 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1370031c00 con 0x7f13800b4970 2024-03-06T07:22:50.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.871+0000 7f1385f79640 1 -- 172.21.15.83:0/2143822317 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13800b4970 msgr2=0x7f138006d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.871+0000 7f1385f79640 1 --2- 172.21.15.83:0/2143822317 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13800b4970 0x7f138006d080 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f13700097a0 tx=0x7f137002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.871+0000 7f1385f79640 1 -- 172.21.15.83:0/2143822317 shutdown_connections 2024-03-06T07:22:50.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.871+0000 7f1385f79640 1 --2- 172.21.15.83:0/2143822317 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f138006d5c0 0x7f13800cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.871+0000 7f1385f79640 1 --2- 172.21.15.83:0/2143822317 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13800b4970 0x7f138006d080 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.871+0000 7f1385f79640 1 -- 172.21.15.83:0/2143822317 >> 172.21.15.83:0/2143822317 conn(0x7f138006b190 msgr2=0x7f138006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:50.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.871+0000 7f1385f79640 1 -- 172.21.15.83:0/2143822317 shutdown_connections 2024-03-06T07:22:50.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.872+0000 7f1385f79640 1 -- 172.21.15.83:0/2143822317 wait complete. 2024-03-06T07:22:50.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.873+0000 7f1385f79640 1 Processor -- start 2024-03-06T07:22:50.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.873+0000 7f1385f79640 1 -- start start 2024-03-06T07:22:50.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.873+0000 7f1385f79640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f138006d5c0 0x7f1380153260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.873+0000 7f1385f79640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f13800b4970 0x7f13800b0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.873+0000 7f1385f79640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f138015bb40 0x7f1380155bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.873+0000 7f1385f79640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138015bf20 con 0x7f138015bb40 2024-03-06T07:22:50.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.873+0000 7f1385f79640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138006cb60 con 0x7f13800b4970 2024-03-06T07:22:50.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.873+0000 7f1385f79640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f138015c0a0 con 0x7f138006d5c0 2024-03-06T07:22:50.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.874+0000 7f137ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f138015bb40 0x7f1380155bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.874+0000 7f137ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f138015bb40 0x7f1380155bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36680/0 (socket says 172.21.15.83:36680) 2024-03-06T07:22:50.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.874+0000 7f137ffff640 1 -- 172.21.15.83:0/20689810 learned_addr learned my addr 172.21.15.83:0/20689810 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:50.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.874+0000 7f137f7fe640 1 --2- 172.21.15.83:0/20689810 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f138006d5c0 0x7f1380153260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.874+0000 7f137effd640 1 --2- 172.21.15.83:0/20689810 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f13800b4970 0x7f13800b0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.874+0000 7f137ffff640 1 -- 172.21.15.83:0/20689810 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f138006d5c0 msgr2=0x7f1380153260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.874+0000 7f137ffff640 1 --2- 172.21.15.83:0/20689810 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f138006d5c0 0x7f1380153260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.874+0000 7f137ffff640 1 -- 172.21.15.83:0/20689810 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f13800b4970 msgr2=0x7f13800b0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.874+0000 7f137ffff640 1 --2- 172.21.15.83:0/20689810 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f13800b4970 0x7f13800b0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.874+0000 7f137ffff640 1 -- 172.21.15.83:0/20689810 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13801537a0 con 0x7f138015bb40 2024-03-06T07:22:50.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.874+0000 7f137ffff640 1 --2- 172.21.15.83:0/20689810 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f138015bb40 0x7f1380155bb0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f1378007e40 tx=0x7f137800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:50.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.874+0000 7f1367fff640 1 -- 172.21.15.83:0/20689810 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1378014620 con 0x7f138015bb40 2024-03-06T07:22:50.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.874+0000 7f1367fff640 1 -- 172.21.15.83:0/20689810 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1378019070 con 0x7f138015bb40 2024-03-06T07:22:50.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.874+0000 7f1385f79640 1 -- 172.21.15.83:0/20689810 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13800af730 con 0x7f138015bb40 2024-03-06T07:22:50.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.874+0000 7f1367fff640 1 -- 172.21.15.83:0/20689810 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1378014a20 con 0x7f138015bb40 2024-03-06T07:22:50.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.874+0000 7f1385f79640 1 -- 172.21.15.83:0/20689810 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13800afcf0 con 0x7f138015bb40 2024-03-06T07:22:50.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.875+0000 7f1385f79640 1 -- 172.21.15.83:0/20689810 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13800732b0 con 0x7f138015bb40 2024-03-06T07:22:50.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.876+0000 7f1367fff640 1 -- 172.21.15.83:0/20689810 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1378014bc0 con 0x7f138015bb40 2024-03-06T07:22:50.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.876+0000 7f1367fff640 1 --2- 172.21.15.83:0/20689810 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f136c06b4d0 0x7f136c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.876+0000 7f1367fff640 1 -- 172.21.15.83:0/20689810 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5998+0+0 (secure 0 0 0) 0x7f137808ac40 con 0x7f138015bb40 2024-03-06T07:22:50.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.876+0000 7f137f7fe640 1 --2- 172.21.15.83:0/20689810 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f136c06b4d0 0x7f136c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.876+0000 7f137f7fe640 1 --2- 172.21.15.83:0/20689810 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f136c06b4d0 0x7f136c06d990 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1374000f60 tx=0x7f1374007c00 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:50.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:50.880+0000 7f1367fff640 1 -- 172.21.15.83:0/20689810 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f13801537a0 con 0x7f138015bb40 2024-03-06T07:22:51.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.092+0000 7f1385f79640 1 -- 172.21.15.83:0/20689810 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f1380068ff0 con 0x7f138015bb40 2024-03-06T07:22:51.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.752+0000 7f1367fff640 1 -- 172.21.15.83:0/20689810 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v8) v1 ==== 104+0+0 (secure 0 0 0) 0x7f13780599a0 con 0x7f138015bb40 2024-03-06T07:22:51.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.755+0000 7f1385f79640 1 -- 172.21.15.83:0/20689810 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f136c06b4d0 msgr2=0x7f136c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.755+0000 7f1385f79640 1 --2- 172.21.15.83:0/20689810 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f136c06b4d0 0x7f136c06d990 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1374000f60 tx=0x7f1374007c00 comp rx=0 tx=0).stop 2024-03-06T07:22:51.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.755+0000 7f1385f79640 1 -- 172.21.15.83:0/20689810 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f138015bb40 msgr2=0x7f1380155bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.755+0000 7f1385f79640 1 --2- 172.21.15.83:0/20689810 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f138015bb40 0x7f1380155bb0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f1378007e40 tx=0x7f137800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.755+0000 7f1385f79640 1 -- 172.21.15.83:0/20689810 shutdown_connections 2024-03-06T07:22:51.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.755+0000 7f1385f79640 1 --2- 172.21.15.83:0/20689810 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f136c06b4d0 0x7f136c06d990 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.755+0000 7f1385f79640 1 --2- 172.21.15.83:0/20689810 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f138015bb40 0x7f1380155bb0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.755+0000 7f1385f79640 1 --2- 172.21.15.83:0/20689810 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f13800b4970 0x7f13800b0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.755+0000 7f1385f79640 1 --2- 172.21.15.83:0/20689810 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f138006d5c0 0x7f1380153260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.756+0000 7f1385f79640 1 -- 172.21.15.83:0/20689810 >> 172.21.15.83:0/20689810 conn(0x7f138006b190 msgr2=0x7f13801540e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:51.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.756+0000 7f1385f79640 1 -- 172.21.15.83:0/20689810 shutdown_connections 2024-03-06T07:22:51.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.756+0000 7f1385f79640 1 -- 172.21.15.83:0/20689810 wait complete. 2024-03-06T07:22:51.771 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:22:51.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.919+0000 7f4e82a17640 1 Processor -- start 2024-03-06T07:22:51.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.919+0000 7f4e82a17640 1 -- start start 2024-03-06T07:22:51.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.920+0000 7f4e82a17640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4e7c15c5a0 0x7f4e7c06c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.920+0000 7f4e82a17640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f4e7c15c980 0x7f4e7c071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.920+0000 7f4e82a17640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e7c06ce50 con 0x7f4e7c0ce8c0 2024-03-06T07:22:51.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.920+0000 7f4e82a17640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e7c072a20 con 0x7f4e7c15c980 2024-03-06T07:22:51.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.920+0000 7f4e82a17640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7c0708b0 con 0x7f4e7c15c5a0 2024-03-06T07:22:51.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.920+0000 7f4e80fb4640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f4e7c15c980 0x7f4e7c071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.920+0000 7f4e7bfff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4e7c15c5a0 0x7f4e7c06c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.920+0000 7f4e7b7fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f4e7c0ce8c0 0x7f4e7c0cc330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:35924/0 (socket says 172.21.15.83:35924) 2024-03-06T07:22:51.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.920+0000 7f4e7b7fe640 1 -- 172.21.15.83:0/4274196481 learned_addr learned my addr 172.21.15.83:0/4274196481 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:51.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.921+0000 7f4e80fb4640 1 -- 172.21.15.83:0/4274196481 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4e7c15c5a0 msgr2=0x7f4e7c06c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:51.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.921+0000 7f4e80fb4640 1 --2- 172.21.15.83:0/4274196481 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4e7c15c5a0 0x7f4e7c06c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.921+0000 7f4e80fb4640 1 -- 172.21.15.83:0/4274196481 >> v1:172.21.15.183:6789/0 conn(0x7f4e7c0ce8c0 legacy=0x7f4e7c0cc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.921+0000 7f4e80fb4640 1 -- 172.21.15.83:0/4274196481 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e7c0ced50 con 0x7f4e7c15c980 2024-03-06T07:22:51.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.921+0000 7f4e7bfff640 1 --2- 172.21.15.83:0/4274196481 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4e7c15c5a0 0x7f4e7c06c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:51.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.921+0000 7f4e80fb4640 1 --2- 172.21.15.83:0/4274196481 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4e7c15c980 0x7f4e7c071300 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4e74009d90 tx=0x7f4e7402be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2982e017f8bdbe9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.922+0000 7f4e7affd640 1 -- 172.21.15.83:0/4274196481 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e74039070 con 0x7f4e7c15c980 2024-03-06T07:22:51.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.922+0000 7f4e7affd640 1 -- 172.21.15.83:0/4274196481 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e7402cbc0 con 0x7f4e7c15c980 2024-03-06T07:22:51.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.922+0000 7f4e7affd640 1 -- 172.21.15.83:0/4274196481 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e74034440 con 0x7f4e7c15c980 2024-03-06T07:22:51.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.922+0000 7f4e82a17640 1 -- 172.21.15.83:0/4274196481 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4e7c15c980 msgr2=0x7f4e7c071300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.922+0000 7f4e82a17640 1 --2- 172.21.15.83:0/4274196481 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4e7c15c980 0x7f4e7c071300 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4e74009d90 tx=0x7f4e7402be10 comp rx=0 tx=0).stop 2024-03-06T07:22:51.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.922+0000 7f4e82a17640 1 -- 172.21.15.83:0/4274196481 shutdown_connections 2024-03-06T07:22:51.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.922+0000 7f4e82a17640 1 --2- 172.21.15.83:0/4274196481 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4e7c15c980 0x7f4e7c071300 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.922+0000 7f4e82a17640 1 --2- 172.21.15.83:0/4274196481 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4e7c15c5a0 0x7f4e7c06c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.922+0000 7f4e82a17640 1 -- 172.21.15.83:0/4274196481 >> 172.21.15.83:0/4274196481 conn(0x7f4e7c0b1650 msgr2=0x7f4e7c0b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:51.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.922+0000 7f4e82a17640 1 -- 172.21.15.83:0/4274196481 shutdown_connections 2024-03-06T07:22:51.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.922+0000 7f4e82a17640 1 -- 172.21.15.83:0/4274196481 wait complete. 2024-03-06T07:22:51.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.923+0000 7f4e82a17640 1 Processor -- start 2024-03-06T07:22:51.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.924+0000 7f4e82a17640 1 -- start start 2024-03-06T07:22:51.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.924+0000 7f4e82a17640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4e7c15c5a0 0x7f4e7c13fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.924+0000 7f4e82a17640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4e7c15c980 0x7f4e7c133a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.924+0000 7f4e82a17640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e7c157840 0x7f4e7c155010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.924+0000 7f4e82a17640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7c070b00 con 0x7f4e7c157840 2024-03-06T07:22:51.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.924+0000 7f4e82a17640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7c070e70 con 0x7f4e7c15c980 2024-03-06T07:22:51.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.924+0000 7f4e82a17640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7c15ce30 con 0x7f4e7c15c5a0 2024-03-06T07:22:51.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.924+0000 7f4e80fb4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e7c157840 0x7f4e7c155010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.924+0000 7f4e80fb4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e7c157840 0x7f4e7c155010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36686/0 (socket says 172.21.15.83:36686) 2024-03-06T07:22:51.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.924+0000 7f4e80fb4640 1 -- 172.21.15.83:0/131419692 learned_addr learned my addr 172.21.15.83:0/131419692 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:51.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.925+0000 7f4e7bfff640 1 --2- 172.21.15.83:0/131419692 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4e7c15c5a0 0x7f4e7c13fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.925+0000 7f4e7b7fe640 1 --2- 172.21.15.83:0/131419692 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4e7c15c980 0x7f4e7c133a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.925+0000 7f4e80fb4640 1 -- 172.21.15.83:0/131419692 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4e7c15c5a0 msgr2=0x7f4e7c13fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.925+0000 7f4e80fb4640 1 --2- 172.21.15.83:0/131419692 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4e7c15c5a0 0x7f4e7c13fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.925+0000 7f4e80fb4640 1 -- 172.21.15.83:0/131419692 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4e7c15c980 msgr2=0x7f4e7c133a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.925+0000 7f4e80fb4640 1 --2- 172.21.15.83:0/131419692 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4e7c15c980 0x7f4e7c133a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.925+0000 7f4e80fb4640 1 -- 172.21.15.83:0/131419692 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e7c155550 con 0x7f4e7c157840 2024-03-06T07:22:51.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.925+0000 7f4e80fb4640 1 --2- 172.21.15.83:0/131419692 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e7c157840 0x7f4e7c155010 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f4e74009d90 tx=0x7f4e74004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.925+0000 7f4e78ff9640 1 -- 172.21.15.83:0/131419692 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e74039050 con 0x7f4e7c157840 2024-03-06T07:22:51.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.925+0000 7f4e78ff9640 1 -- 172.21.15.83:0/131419692 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e74045070 con 0x7f4e7c157840 2024-03-06T07:22:51.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.925+0000 7f4e82a17640 1 -- 172.21.15.83:0/131419692 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e7c142820 con 0x7f4e7c157840 2024-03-06T07:22:51.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.925+0000 7f4e78ff9640 1 -- 172.21.15.83:0/131419692 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7403e830 con 0x7f4e7c157840 2024-03-06T07:22:51.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.925+0000 7f4e82a17640 1 -- 172.21.15.83:0/131419692 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e7c142de0 con 0x7f4e7c157840 2024-03-06T07:22:51.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.926+0000 7f4e82a17640 1 -- 172.21.15.83:0/131419692 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e7c0687d0 con 0x7f4e7c157840 2024-03-06T07:22:51.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.926+0000 7f4e78ff9640 1 -- 172.21.15.83:0/131419692 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4e7403e9d0 con 0x7f4e7c157840 2024-03-06T07:22:51.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.927+0000 7f4e78ff9640 1 --2- 172.21.15.83:0/131419692 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4e6806b4d0 0x7f4e6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.927+0000 7f4e78ff9640 1 -- 172.21.15.83:0/131419692 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5998+0+0 (secure 0 0 0) 0x7f4e740ad0d0 con 0x7f4e7c157840 2024-03-06T07:22:51.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.927+0000 7f4e7bfff640 1 --2- 172.21.15.83:0/131419692 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4e6806b4d0 0x7f4e6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.927+0000 7f4e7bfff640 1 --2- 172.21.15.83:0/131419692 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4e6806b4d0 0x7f4e6806d990 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c009560 tx=0x7f4e6c009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:51.931+0000 7f4e78ff9640 1 -- 172.21.15.83:0/131419692 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4e7407b910 con 0x7f4e7c157840 2024-03-06T07:22:52.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.102+0000 7f4e82a17640 1 -- 172.21.15.83:0/131419692 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4e7c04f330 con 0x7f4e7c157840 2024-03-06T07:22:52.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.754+0000 7f4e78ff9640 1 -- 172.21.15.83:0/131419692 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v24) v1 ==== 181+0+0 (secure 0 0 0) 0x7f4e7407f9b0 con 0x7f4e7c157840 2024-03-06T07:22:52.756 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:22:52.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.757+0000 7f4e82a17640 1 -- 172.21.15.83:0/131419692 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4e6806b4d0 msgr2=0x7f4e6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.757+0000 7f4e82a17640 1 --2- 172.21.15.83:0/131419692 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4e6806b4d0 0x7f4e6806d990 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c009560 tx=0x7f4e6c009190 comp rx=0 tx=0).stop 2024-03-06T07:22:52.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.757+0000 7f4e82a17640 1 -- 172.21.15.83:0/131419692 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e7c157840 msgr2=0x7f4e7c155010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.757+0000 7f4e82a17640 1 --2- 172.21.15.83:0/131419692 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e7c157840 0x7f4e7c155010 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f4e74009d90 tx=0x7f4e74004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:52.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.757+0000 7f4e82a17640 1 -- 172.21.15.83:0/131419692 shutdown_connections 2024-03-06T07:22:52.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.757+0000 7f4e82a17640 1 --2- 172.21.15.83:0/131419692 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4e6806b4d0 0x7f4e6806d990 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.757+0000 7f4e82a17640 1 --2- 172.21.15.83:0/131419692 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e7c157840 0x7f4e7c155010 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.757+0000 7f4e82a17640 1 --2- 172.21.15.83:0/131419692 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4e7c15c980 0x7f4e7c133a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.757+0000 7f4e82a17640 1 --2- 172.21.15.83:0/131419692 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4e7c15c5a0 0x7f4e7c13fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.757+0000 7f4e82a17640 1 -- 172.21.15.83:0/131419692 >> 172.21.15.83:0/131419692 conn(0x7f4e7c0b1650 msgr2=0x7f4e7c0cf7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:52.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.757+0000 7f4e82a17640 1 -- 172.21.15.83:0/131419692 shutdown_connections 2024-03-06T07:22:52.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.757+0000 7f4e82a17640 1 -- 172.21.15.83:0/131419692 wait complete. 2024-03-06T07:22:52.772 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:22:52.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.877+0000 7fa5dedae640 1 Processor -- start 2024-03-06T07:22:52.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.877+0000 7fa5dedae640 1 -- start start 2024-03-06T07:22:52.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.878+0000 7fa5dedae640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa5d8162850 0x7fa5d80d1a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.878+0000 7fa5dedae640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fa5d80d1f90 0x7fa5d806c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.878+0000 7fa5dedae640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5d80cf3c0 con 0x7fa5d8162470 2024-03-06T07:22:52.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.878+0000 7fa5dedae640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5d80d23e0 con 0x7fa5d8162850 2024-03-06T07:22:52.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.878+0000 7fa5dedae640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5d806f270 con 0x7fa5d80d1f90 2024-03-06T07:22:52.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.878+0000 7fa5d7fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa5d8162850 0x7fa5d80d1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.878+0000 7fa5d7fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa5d8162850 0x7fa5d80d1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36698/0 (socket says 172.21.15.83:36698) 2024-03-06T07:22:52.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.878+0000 7fa5d7fff640 1 -- 172.21.15.83:0/44947419 learned_addr learned my addr 172.21.15.83:0/44947419 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:52.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.879+0000 7fa5dd34b640 1 --2- 172.21.15.83:0/44947419 >> v2:172.21.15.183:3300/0 conn(0x7fa5d80d1f90 0x7fa5d806c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.879+0000 7fa5d7fff640 1 -- 172.21.15.83:0/44947419 >> v2:172.21.15.183:3300/0 conn(0x7fa5d80d1f90 msgr2=0x7fa5d806c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:52.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.879+0000 7fa5d7fff640 1 --2- 172.21.15.83:0/44947419 >> v2:172.21.15.183:3300/0 conn(0x7fa5d80d1f90 0x7fa5d806c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.879+0000 7fa5d7fff640 1 -- 172.21.15.83:0/44947419 >> v1:172.21.15.183:6789/0 conn(0x7fa5d8162470 legacy=0x7fa5d806f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:52.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.879+0000 7fa5d7fff640 1 -- 172.21.15.83:0/44947419 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5d80cf0e0 con 0x7fa5d8162850 2024-03-06T07:22:52.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.879+0000 7fa5d7fff640 1 --2- 172.21.15.83:0/44947419 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5d8162850 0x7fa5d80d1a50 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa5d80b15a0 tx=0x7fa5cc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59805b6a0de30d0b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.880+0000 7fa5d77fe640 1 -- 172.21.15.83:0/44947419 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc035020 con 0x7fa5d8162850 2024-03-06T07:22:52.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.880+0000 7fa5d77fe640 1 -- 172.21.15.83:0/44947419 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5cc031870 con 0x7fa5d8162850 2024-03-06T07:22:52.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.880+0000 7fa5d77fe640 1 -- 172.21.15.83:0/44947419 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc031c40 con 0x7fa5d8162850 2024-03-06T07:22:52.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.880+0000 7fa5dedae640 1 -- 172.21.15.83:0/44947419 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5d8162850 msgr2=0x7fa5d80d1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.880+0000 7fa5dedae640 1 --2- 172.21.15.83:0/44947419 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5d8162850 0x7fa5d80d1a50 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa5d80b15a0 tx=0x7fa5cc02b910 comp rx=0 tx=0).stop 2024-03-06T07:22:52.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.880+0000 7fa5dedae640 1 -- 172.21.15.83:0/44947419 shutdown_connections 2024-03-06T07:22:52.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.880+0000 7fa5dedae640 1 --2- 172.21.15.83:0/44947419 >> v2:172.21.15.183:3300/0 conn(0x7fa5d80d1f90 0x7fa5d806c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.880+0000 7fa5dedae640 1 --2- 172.21.15.83:0/44947419 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5d8162850 0x7fa5d80d1a50 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.880+0000 7fa5dedae640 1 -- 172.21.15.83:0/44947419 >> 172.21.15.83:0/44947419 conn(0x7fa5d809a9f0 msgr2=0x7fa5d809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:52.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.880+0000 7fa5dedae640 1 -- 172.21.15.83:0/44947419 shutdown_connections 2024-03-06T07:22:52.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.880+0000 7fa5dedae640 1 -- 172.21.15.83:0/44947419 wait complete. 2024-03-06T07:22:52.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.881+0000 7fa5dedae640 1 Processor -- start 2024-03-06T07:22:52.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.881+0000 7fa5dedae640 1 -- start start 2024-03-06T07:22:52.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.882+0000 7fa5dedae640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa5d80d1f90 0x7fa5d81f81d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.882+0000 7fa5dedae640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5d8162470 0x7fa5d815ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.882+0000 7fa5dedae640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5d8162850 0x7fa5d8158770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.882+0000 7fa5dedae640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d806fe10 con 0x7fa5d8162850 2024-03-06T07:22:52.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.882+0000 7fa5dedae640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d806ef00 con 0x7fa5d8162470 2024-03-06T07:22:52.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.882+0000 7fa5dedae640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d806ff90 con 0x7fa5d80d1f90 2024-03-06T07:22:52.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.882+0000 7fa5dd34b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5d8162850 0x7fa5d8158770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.882+0000 7fa5dd34b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5d8162850 0x7fa5d8158770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36708/0 (socket says 172.21.15.83:36708) 2024-03-06T07:22:52.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.882+0000 7fa5dd34b640 1 -- 172.21.15.83:0/2705238487 learned_addr learned my addr 172.21.15.83:0/2705238487 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:52.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.882+0000 7fa5d7fff640 1 --2- 172.21.15.83:0/2705238487 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5d8162470 0x7fa5d815ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.882+0000 7fa5dcb4a640 1 --2- 172.21.15.83:0/2705238487 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa5d80d1f90 0x7fa5d81f81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.883+0000 7fa5dd34b640 1 -- 172.21.15.83:0/2705238487 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa5d80d1f90 msgr2=0x7fa5d81f81d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.883+0000 7fa5dd34b640 1 --2- 172.21.15.83:0/2705238487 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa5d80d1f90 0x7fa5d81f81d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.883+0000 7fa5dd34b640 1 -- 172.21.15.83:0/2705238487 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5d8162470 msgr2=0x7fa5d815ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.883+0000 7fa5dd34b640 1 --2- 172.21.15.83:0/2705238487 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5d8162470 0x7fa5d815ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.883+0000 7fa5dd34b640 1 -- 172.21.15.83:0/2705238487 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5d815f360 con 0x7fa5d8162850 2024-03-06T07:22:52.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.883+0000 7fa5d7fff640 1 --2- 172.21.15.83:0/2705238487 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5d8162470 0x7fa5d815ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:52.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.883+0000 7fa5dd34b640 1 --2- 172.21.15.83:0/2705238487 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5d8162850 0x7fa5d8158770 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0007e70 tx=0x7fa5d000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.883+0000 7fa5d57fa640 1 -- 172.21.15.83:0/2705238487 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d0014620 con 0x7fa5d8162850 2024-03-06T07:22:52.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.883+0000 7fa5d57fa640 1 -- 172.21.15.83:0/2705238487 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5d0019070 con 0x7fa5d8162850 2024-03-06T07:22:52.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.883+0000 7fa5dedae640 1 -- 172.21.15.83:0/2705238487 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5d8158cb0 con 0x7fa5d8162850 2024-03-06T07:22:52.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.883+0000 7fa5d57fa640 1 -- 172.21.15.83:0/2705238487 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d0014a20 con 0x7fa5d8162850 2024-03-06T07:22:52.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.883+0000 7fa5dedae640 1 -- 172.21.15.83:0/2705238487 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5d80b4db0 con 0x7fa5d8162850 2024-03-06T07:22:52.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.884+0000 7fa5dedae640 1 -- 172.21.15.83:0/2705238487 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5d804f330 con 0x7fa5d8162850 2024-03-06T07:22:52.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.884+0000 7fa5d57fa640 1 -- 172.21.15.83:0/2705238487 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa5d0014bc0 con 0x7fa5d8162850 2024-03-06T07:22:52.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.884+0000 7fa5d57fa640 1 --2- 172.21.15.83:0/2705238487 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa5c406b500 0x7fa5c406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.885+0000 7fa5d57fa640 1 -- 172.21.15.83:0/2705238487 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5560+0+0 (secure 0 0 0) 0x7fa5d008a990 con 0x7fa5d8162850 2024-03-06T07:22:52.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.885+0000 7fa5dcb4a640 1 --2- 172.21.15.83:0/2705238487 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa5c406b500 0x7fa5c406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.885+0000 7fa5dcb4a640 1 --2- 172.21.15.83:0/2705238487 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa5c406b500 0x7fa5c406d9c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa5c8004ea0 tx=0x7fa5c8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:52.889+0000 7fa5d57fa640 1 -- 172.21.15.83:0/2705238487 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa5d0059930 con 0x7fa5d8162850 2024-03-06T07:22:53.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.062+0000 7fa5dedae640 1 -- 172.21.15.83:0/2705238487 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa5d80d1230 con 0x7fa5d8162850 2024-03-06T07:22:53.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.755+0000 7fa5d57fa640 1 -- 172.21.15.83:0/2705238487 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v25) v1 ==== 169+0+0 (secure 0 0 0) 0x7fa5d0059230 con 0x7fa5d8162850 2024-03-06T07:22:53.758 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'cephfs_data' removed 2024-03-06T07:22:53.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.759+0000 7fa5dedae640 1 -- 172.21.15.83:0/2705238487 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa5c406b500 msgr2=0x7fa5c406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.759+0000 7fa5dedae640 1 --2- 172.21.15.83:0/2705238487 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa5c406b500 0x7fa5c406d9c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa5c8004ea0 tx=0x7fa5c8009040 comp rx=0 tx=0).stop 2024-03-06T07:22:53.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.759+0000 7fa5dedae640 1 -- 172.21.15.83:0/2705238487 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5d8162850 msgr2=0x7fa5d8158770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.759+0000 7fa5dedae640 1 --2- 172.21.15.83:0/2705238487 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5d8162850 0x7fa5d8158770 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0007e70 tx=0x7fa5d000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.760+0000 7fa5dedae640 1 -- 172.21.15.83:0/2705238487 shutdown_connections 2024-03-06T07:22:53.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.760+0000 7fa5dedae640 1 --2- 172.21.15.83:0/2705238487 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa5c406b500 0x7fa5c406d9c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.760+0000 7fa5dedae640 1 --2- 172.21.15.83:0/2705238487 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5d8162850 0x7fa5d8158770 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.760+0000 7fa5dedae640 1 --2- 172.21.15.83:0/2705238487 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5d8162470 0x7fa5d815ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.760+0000 7fa5dedae640 1 --2- 172.21.15.83:0/2705238487 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa5d80d1f90 0x7fa5d81f81d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.760+0000 7fa5dedae640 1 -- 172.21.15.83:0/2705238487 >> 172.21.15.83:0/2705238487 conn(0x7fa5d809a9f0 msgr2=0x7fa5d806f700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:53.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.760+0000 7fa5dedae640 1 -- 172.21.15.83:0/2705238487 shutdown_connections 2024-03-06T07:22:53.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.760+0000 7fa5dedae640 1 -- 172.21.15.83:0/2705238487 wait complete. 2024-03-06T07:22:53.778 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:22:53.778 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:22:53.778 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:22:53.778 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:53.779 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:22:53.779 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:22:53.779 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:22:53.779 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:53.779 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:22:53.779 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:22:53.780 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:22:53.780 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:53.780 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:22:53.780 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:22:53.780 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:22:53.780 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:53.791 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:22:53.792 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:22:53.792 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:22:53.793 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:22:53.794 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:22:53.794 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:22:53.795 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:22:53.795 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:22:53.834 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:22:53.834 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:53.835 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:22:53.835 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:53.837 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:22:53.837 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:53.838 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:22:53.838 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:53.838 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:22:53.915 INFO:tasks.ceph.mds.a.smithi083.stdout:starting mds.a at 2024-03-06T07:22:53.915 INFO:tasks.ceph.mds.c.smithi083.stdout:starting mds.c at 2024-03-06T07:22:53.921 INFO:tasks.ceph.mds.b.smithi183.stdout:starting mds.b at 2024-03-06T07:22:53.921 INFO:tasks.ceph.mds.d.smithi183.stdout:starting mds.d at 2024-03-06T07:22:53.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.945+0000 7f3e58fa0640 1 Processor -- start 2024-03-06T07:22:53.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.945+0000 7f3e58fa0640 1 -- start start 2024-03-06T07:22:53.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.946+0000 7f3e58fa0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f3e54146ac0 0x7f3e540b38f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.946+0000 7f3e58fa0640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f3e5406f500 0x7f3e5406b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.946+0000 7f3e58fa0640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e540b5830 con 0x7f3e5406afb0 2024-03-06T07:22:53.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.946+0000 7f3e58fa0640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e540b5b10 con 0x7f3e54146ac0 2024-03-06T07:22:53.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.946+0000 7f3e58fa0640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e540724b0 con 0x7f3e5406f500 2024-03-06T07:22:53.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.946+0000 7f3e537fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f3e54146ac0 0x7f3e540b38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.946+0000 7f3e537fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f3e54146ac0 0x7f3e540b38f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36728/0 (socket says 172.21.15.83:36728) 2024-03-06T07:22:53.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.946+0000 7f3e537fe640 1 -- 172.21.15.83:0/2202218293 learned_addr learned my addr 172.21.15.83:0/2202218293 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:53.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.946+0000 7f3e53fff640 1 --2- 172.21.15.83:0/2202218293 >> v2:172.21.15.183:3300/0 conn(0x7f3e5406f500 0x7f3e5406b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.946+0000 7f3e537fe640 1 -- 172.21.15.83:0/2202218293 >> v2:172.21.15.183:3300/0 conn(0x7f3e5406f500 msgr2=0x7f3e5406b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:53.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.946+0000 7f3e537fe640 1 --2- 172.21.15.83:0/2202218293 >> v2:172.21.15.183:3300/0 conn(0x7f3e5406f500 0x7f3e5406b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.946+0000 7f3e537fe640 1 -- 172.21.15.83:0/2202218293 >> v1:172.21.15.183:6789/0 conn(0x7f3e5406afb0 legacy=0x7f3e540d0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:53.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.946+0000 7f3e537fe640 1 -- 172.21.15.83:0/2202218293 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e54072790 con 0x7f3e54146ac0 2024-03-06T07:22:53.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.947+0000 7f3e537fe640 1 --2- 172.21.15.83:0/2202218293 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e54146ac0 0x7f3e540b38f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f3e541479e0 tx=0x7f3e4402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=900e60ed7fc3ec76 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:53.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.947+0000 7f3e527fc640 1 -- 172.21.15.83:0/2202218293 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e44035020 con 0x7f3e54146ac0 2024-03-06T07:22:53.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.947+0000 7f3e527fc640 1 -- 172.21.15.83:0/2202218293 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e44031870 con 0x7f3e54146ac0 2024-03-06T07:22:53.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.947+0000 7f3e527fc640 1 -- 172.21.15.83:0/2202218293 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e44031c40 con 0x7f3e54146ac0 2024-03-06T07:22:53.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.948+0000 7f3e58fa0640 1 -- 172.21.15.83:0/2202218293 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e54146ac0 msgr2=0x7f3e540b38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.948+0000 7f3e58fa0640 1 --2- 172.21.15.83:0/2202218293 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e54146ac0 0x7f3e540b38f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f3e541479e0 tx=0x7f3e4402b910 comp rx=0 tx=0).stop 2024-03-06T07:22:53.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.948+0000 7f3e58fa0640 1 -- 172.21.15.83:0/2202218293 shutdown_connections 2024-03-06T07:22:53.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.948+0000 7f3e58fa0640 1 --2- 172.21.15.83:0/2202218293 >> v2:172.21.15.183:3300/0 conn(0x7f3e5406f500 0x7f3e5406b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.948+0000 7f3e58fa0640 1 --2- 172.21.15.83:0/2202218293 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e54146ac0 0x7f3e540b38f0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.948+0000 7f3e58fa0640 1 -- 172.21.15.83:0/2202218293 >> 172.21.15.83:0/2202218293 conn(0x7f3e5406c0c0 msgr2=0x7f3e5406c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:53.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.948+0000 7f3e58fa0640 1 -- 172.21.15.83:0/2202218293 shutdown_connections 2024-03-06T07:22:53.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.948+0000 7f3e58fa0640 1 -- 172.21.15.83:0/2202218293 wait complete. 2024-03-06T07:22:53.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.949+0000 7f3e58fa0640 1 Processor -- start 2024-03-06T07:22:53.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.949+0000 7f3e58fa0640 1 -- start start 2024-03-06T07:22:53.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.950+0000 7f3e58fa0640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3e5406afb0 0x7f3e540b1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.950+0000 7f3e58fa0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e5406f500 0x7f3e5413c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.950+0000 7f3e58fa0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3e54146ac0 0x7f3e541505a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.950+0000 7f3e58fa0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e540720a0 con 0x7f3e5406f500 2024-03-06T07:22:53.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.950+0000 7f3e58fa0640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e54071f20 con 0x7f3e5406afb0 2024-03-06T07:22:53.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.950+0000 7f3e58fa0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e540b3e30 con 0x7f3e54146ac0 2024-03-06T07:22:53.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.950+0000 7f3e52ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e5406f500 0x7f3e5413c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.950+0000 7f3e52ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e5406f500 0x7f3e5413c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36742/0 (socket says 172.21.15.83:36742) 2024-03-06T07:22:53.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.950+0000 7f3e52ffd640 1 -- 172.21.15.83:0/1104671054 learned_addr learned my addr 172.21.15.83:0/1104671054 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:53.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.950+0000 7f3e53fff640 1 --2- 172.21.15.83:0/1104671054 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3e54146ac0 0x7f3e541505a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.950+0000 7f3e537fe640 1 --2- 172.21.15.83:0/1104671054 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3e5406afb0 0x7f3e540b1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.950+0000 7f3e52ffd640 1 -- 172.21.15.83:0/1104671054 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3e54146ac0 msgr2=0x7f3e541505a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.950+0000 7f3e52ffd640 1 --2- 172.21.15.83:0/1104671054 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3e54146ac0 0x7f3e541505a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.950+0000 7f3e52ffd640 1 -- 172.21.15.83:0/1104671054 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3e5406afb0 msgr2=0x7f3e540b1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.950+0000 7f3e52ffd640 1 --2- 172.21.15.83:0/1104671054 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3e5406afb0 0x7f3e540b1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.950+0000 7f3e52ffd640 1 -- 172.21.15.83:0/1104671054 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e540b1e90 con 0x7f3e5406f500 2024-03-06T07:22:53.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.950+0000 7f3e52ffd640 1 --2- 172.21.15.83:0/1104671054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e5406f500 0x7f3e5413c1a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f3e4800b420 tx=0x7f3e4800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:53.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.950+0000 7f3e3bfff640 1 -- 172.21.15.83:0/1104671054 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e48011830 con 0x7f3e5406f500 2024-03-06T07:22:53.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.951+0000 7f3e3bfff640 1 -- 172.21.15.83:0/1104671054 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e48016070 con 0x7f3e5406f500 2024-03-06T07:22:53.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.951+0000 7f3e58fa0640 1 -- 172.21.15.83:0/1104671054 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e540b2170 con 0x7f3e5406f500 2024-03-06T07:22:53.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.951+0000 7f3e3bfff640 1 -- 172.21.15.83:0/1104671054 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e48011c10 con 0x7f3e5406f500 2024-03-06T07:22:53.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.951+0000 7f3e58fa0640 1 -- 172.21.15.83:0/1104671054 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e540b2a10 con 0x7f3e5406f500 2024-03-06T07:22:53.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.952+0000 7f3e58fa0640 1 -- 172.21.15.83:0/1104671054 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e540b2340 con 0x7f3e5406f500 2024-03-06T07:22:53.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.952+0000 7f3e3bfff640 1 -- 172.21.15.83:0/1104671054 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3e4801b460 con 0x7f3e5406f500 2024-03-06T07:22:53.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.952+0000 7f3e3bfff640 1 --2- 172.21.15.83:0/1104671054 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3e4006b4d0 0x7f3e4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.952+0000 7f3e3bfff640 1 -- 172.21.15.83:0/1104671054 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f3e480898b0 con 0x7f3e5406f500 2024-03-06T07:22:53.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.952+0000 7f3e537fe640 1 --2- 172.21.15.83:0/1104671054 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3e4006b4d0 0x7f3e4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.953+0000 7f3e537fe640 1 --2- 172.21.15.83:0/1104671054 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3e4006b4d0 0x7f3e4006d990 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3e44036000 tx=0x7f3e4403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:53.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:53.956+0000 7f3e3bfff640 1 -- 172.21.15.83:0/1104671054 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3e48020020 con 0x7f3e5406f500 2024-03-06T07:22:54.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.141+0000 7f3e58fa0640 1 -- 172.21.15.83:0/1104671054 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f3e540b4700 con 0x7f3e5406f500 2024-03-06T07:22:54.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.772+0000 7f3e3bfff640 1 -- 172.21.15.83:0/1104671054 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v27) v1 ==== 97+0+0 (secure 0 0 0) 0x7f3e540b4700 con 0x7f3e5406f500 2024-03-06T07:22:54.774 INFO:teuthology.orchestra.run.smithi083.stderr: removed all blocklist entries 2024-03-06T07:22:54.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.774+0000 7f3e58fa0640 1 -- 172.21.15.83:0/1104671054 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3e4006b4d0 msgr2=0x7f3e4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.774+0000 7f3e58fa0640 1 --2- 172.21.15.83:0/1104671054 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3e4006b4d0 0x7f3e4006d990 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3e44036000 tx=0x7f3e4403c040 comp rx=0 tx=0).stop 2024-03-06T07:22:54.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.774+0000 7f3e58fa0640 1 -- 172.21.15.83:0/1104671054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e5406f500 msgr2=0x7f3e5413c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.774+0000 7f3e58fa0640 1 --2- 172.21.15.83:0/1104671054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e5406f500 0x7f3e5413c1a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f3e4800b420 tx=0x7f3e4800b8e0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.774+0000 7f3e58fa0640 1 -- 172.21.15.83:0/1104671054 shutdown_connections 2024-03-06T07:22:54.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.774+0000 7f3e58fa0640 1 --2- 172.21.15.83:0/1104671054 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3e4006b4d0 0x7f3e4006d990 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.774+0000 7f3e58fa0640 1 --2- 172.21.15.83:0/1104671054 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3e54146ac0 0x7f3e541505a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.774+0000 7f3e58fa0640 1 --2- 172.21.15.83:0/1104671054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e5406f500 0x7f3e5413c1a0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.774+0000 7f3e58fa0640 1 --2- 172.21.15.83:0/1104671054 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3e5406afb0 0x7f3e540b1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.774+0000 7f3e58fa0640 1 -- 172.21.15.83:0/1104671054 >> 172.21.15.83:0/1104671054 conn(0x7f3e5406c0c0 msgr2=0x7f3e540af760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:54.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.775+0000 7f3e58fa0640 1 -- 172.21.15.83:0/1104671054 shutdown_connections 2024-03-06T07:22:54.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.775+0000 7f3e58fa0640 1 -- 172.21.15.83:0/1104671054 wait complete. 2024-03-06T07:22:54.794 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:22:54.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.893+0000 7fa30ba15640 1 Processor -- start 2024-03-06T07:22:54.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.893+0000 7fa30ba15640 1 -- start start 2024-03-06T07:22:54.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.894+0000 7fa30ba15640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa3040d2710 0x7fa3040b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.894+0000 7fa30ba15640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3040b3430 con 0x7fa30415dfe0 2024-03-06T07:22:54.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.894+0000 7fa30ba15640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3040b3710 con 0x7fa3040d22b0 2024-03-06T07:22:54.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.894+0000 7fa30ba15640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3040b31e0 con 0x7fa3040d2710 2024-03-06T07:22:54.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.894+0000 7fa3097b1640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fa30415dfe0 0x7fa3040cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:47220/0 (socket says 172.21.15.83:47220) 2024-03-06T07:22:54.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.894+0000 7fa3097b1640 1 -- 172.21.15.83:0/3922518593 learned_addr learned my addr 172.21.15.83:0/3922518593 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:54.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.894+0000 7fa2fbfff640 1 -- 172.21.15.83:0/3922518593 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2124343409 0 0) 0x7fa3040b3430 con 0x7fa30415dfe0 2024-03-06T07:22:54.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.894+0000 7fa2fbfff640 1 -- 172.21.15.83:0/3922518593 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2f0003520 con 0x7fa30415dfe0 2024-03-06T07:22:54.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.895+0000 7fa309fb2640 1 --2- 172.21.15.83:0/3922518593 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa3040d2710 0x7fa3040b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.895+0000 7fa2fbfff640 1 -- 172.21.15.83:0/3922518593 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3442440488 0 0) 0x7fa2f0003520 con 0x7fa30415dfe0 2024-03-06T07:22:54.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.895+0000 7fa2fbfff640 1 -- 172.21.15.83:0/3922518593 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa3040d2710 msgr2=0x7fa3040b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:54.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.895+0000 7fa2fbfff640 1 --2- 172.21.15.83:0/3922518593 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa3040d2710 0x7fa3040b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.895+0000 7fa2fbfff640 1 -- 172.21.15.83:0/3922518593 >> v1:172.21.15.183:6789/0 conn(0x7fa3040d22b0 legacy=0x7fa3040b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.895+0000 7fa2fbfff640 1 -- 172.21.15.83:0/3922518593 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3040b39f0 con 0x7fa30415dfe0 2024-03-06T07:22:54.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.895+0000 7fa2fbfff640 1 -- 172.21.15.83:0/3922518593 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7fa300001670 con 0x7fa30415dfe0 2024-03-06T07:22:54.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.895+0000 7fa2fbfff640 1 -- 172.21.15.83:0/3922518593 >> v1:172.21.15.83:6789/0 conn(0x7fa30415dfe0 legacy=0x7fa3040cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.895+0000 7fa2fbfff640 1 --2- 172.21.15.83:0/3922518593 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa2f00039e0 0x7fa2f0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.895+0000 7fa2fbfff640 1 --2- 172.21.15.83:0/3922518593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2f0004800 0x7fa2f0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.895+0000 7fa2fbfff640 1 --2- 172.21.15.83:0/3922518593 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa3040d2710 0x7fa2f0005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.895+0000 7fa2fbfff640 1 -- 172.21.15.83:0/3922518593 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3040b31e0 con 0x7fa2f0004800 2024-03-06T07:22:54.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.895+0000 7fa2fbfff640 1 -- 172.21.15.83:0/3922518593 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f000bb50 con 0x7fa2f00039e0 2024-03-06T07:22:54.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.895+0000 7fa2fbfff640 1 -- 172.21.15.83:0/3922518593 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f000bd50 con 0x7fa3040d2710 2024-03-06T07:22:54.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.895+0000 7fa308fb0640 1 --2- 172.21.15.83:0/3922518593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2f0004800 0x7fa2f0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.895+0000 7fa309fb2640 1 --2- 172.21.15.83:0/3922518593 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa3040d2710 0x7fa2f0005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.895+0000 7fa3097b1640 1 --2- 172.21.15.83:0/3922518593 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa2f00039e0 0x7fa2f0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.895+0000 7fa308fb0640 1 -- 172.21.15.83:0/3922518593 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa3040d2710 msgr2=0x7fa2f0005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:54.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.895+0000 7fa308fb0640 1 --2- 172.21.15.83:0/3922518593 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa3040d2710 0x7fa2f0005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.895+0000 7fa308fb0640 1 -- 172.21.15.83:0/3922518593 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa2f00039e0 msgr2=0x7fa2f0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:54.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.895+0000 7fa308fb0640 1 --2- 172.21.15.83:0/3922518593 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa2f00039e0 0x7fa2f0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.895+0000 7fa308fb0640 1 -- 172.21.15.83:0/3922518593 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f0003520 con 0x7fa2f0004800 2024-03-06T07:22:54.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.896+0000 7fa308fb0640 1 --2- 172.21.15.83:0/3922518593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2f0004800 0x7fa2f0004c80 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4001750 tx=0x7fa2f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec0f2b77ffa6054 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.896+0000 7fa2fbfff640 1 -- 172.21.15.83:0/3922518593 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f4001370 con 0x7fa2f0004800 2024-03-06T07:22:54.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.896+0000 7fa2fbfff640 1 -- 172.21.15.83:0/3922518593 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2f40052d0 con 0x7fa2f0004800 2024-03-06T07:22:54.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.896+0000 7fa2fbfff640 1 -- 172.21.15.83:0/3922518593 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f4003830 con 0x7fa2f0004800 2024-03-06T07:22:54.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.897+0000 7fa30ba15640 1 -- 172.21.15.83:0/3922518593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2f0004800 msgr2=0x7fa2f0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.897+0000 7fa30ba15640 1 --2- 172.21.15.83:0/3922518593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2f0004800 0x7fa2f0004c80 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4001750 tx=0x7fa2f0005830 comp rx=0 tx=0).stop 2024-03-06T07:22:54.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.897+0000 7fa309fb2640 1 -- 172.21.15.83:0/3922518593 reap_dead start 2024-03-06T07:22:54.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.897+0000 7fa30ba15640 1 -- 172.21.15.83:0/3922518593 shutdown_connections 2024-03-06T07:22:54.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.897+0000 7fa30ba15640 1 -- 172.21.15.83:0/3922518593 >> 172.21.15.83:0/3922518593 conn(0x7fa3040d3cd0 msgr2=0x7fa3040d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:54.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.897+0000 7fa30ba15640 1 -- 172.21.15.83:0/3922518593 shutdown_connections 2024-03-06T07:22:54.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.897+0000 7fa30ba15640 1 -- 172.21.15.83:0/3922518593 wait complete. 2024-03-06T07:22:54.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.898+0000 7fa30ba15640 1 Processor -- start 2024-03-06T07:22:54.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.898+0000 7fa30ba15640 1 -- start start 2024-03-06T07:22:54.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.898+0000 7fa30ba15640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa2f0004800 0x7fa3041f7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.898+0000 7fa30ba15640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3040b69d0 0x7fa304158540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.898+0000 7fa30ba15640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa3040cada0 0x7fa3040b20d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.898+0000 7fa30ba15640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3040cfa00 con 0x7fa3040b69d0 2024-03-06T07:22:54.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.898+0000 7fa30ba15640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3040b3d60 con 0x7fa2f0004800 2024-03-06T07:22:54.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.898+0000 7fa30ba15640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3040d1c70 con 0x7fa3040cada0 2024-03-06T07:22:54.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.899+0000 7fa308fb0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3040b69d0 0x7fa304158540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.899+0000 7fa308fb0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3040b69d0 0x7fa304158540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36756/0 (socket says 172.21.15.83:36756) 2024-03-06T07:22:54.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.899+0000 7fa308fb0640 1 -- 172.21.15.83:0/2651993762 learned_addr learned my addr 172.21.15.83:0/2651993762 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:54.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.899+0000 7fa3097b1640 1 --2- 172.21.15.83:0/2651993762 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa2f0004800 0x7fa3041f7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.899+0000 7fa309fb2640 1 --2- 172.21.15.83:0/2651993762 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa3040cada0 0x7fa3040b20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.899+0000 7fa308fb0640 1 -- 172.21.15.83:0/2651993762 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa3040cada0 msgr2=0x7fa3040b20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.899+0000 7fa308fb0640 1 --2- 172.21.15.83:0/2651993762 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa3040cada0 0x7fa3040b20d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.899+0000 7fa308fb0640 1 -- 172.21.15.83:0/2651993762 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa2f0004800 msgr2=0x7fa3041f7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.899+0000 7fa308fb0640 1 --2- 172.21.15.83:0/2651993762 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa2f0004800 0x7fa3041f7a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.899+0000 7fa308fb0640 1 -- 172.21.15.83:0/2651993762 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3041f7f60 con 0x7fa3040b69d0 2024-03-06T07:22:54.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.899+0000 7fa309fb2640 1 --2- 172.21.15.83:0/2651993762 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa3040cada0 0x7fa3040b20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:54.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.899+0000 7fa308fb0640 1 --2- 172.21.15.83:0/2651993762 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3040b69d0 0x7fa304158540 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4008fd0 tx=0x7fa2f4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.899+0000 7fa2f9ffb640 1 -- 172.21.15.83:0/2651993762 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f4041070 con 0x7fa3040b69d0 2024-03-06T07:22:54.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.899+0000 7fa2f9ffb640 1 -- 172.21.15.83:0/2651993762 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2f4005480 con 0x7fa3040b69d0 2024-03-06T07:22:54.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.899+0000 7fa2f9ffb640 1 -- 172.21.15.83:0/2651993762 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f4001370 con 0x7fa3040b69d0 2024-03-06T07:22:54.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.899+0000 7fa30ba15640 1 -- 172.21.15.83:0/2651993762 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3040b77a0 con 0x7fa3040b69d0 2024-03-06T07:22:54.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.899+0000 7fa30ba15640 1 -- 172.21.15.83:0/2651993762 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3040b7a40 con 0x7fa3040b69d0 2024-03-06T07:22:54.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.900+0000 7fa30ba15640 1 -- 172.21.15.83:0/2651993762 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa30404f330 con 0x7fa3040b69d0 2024-03-06T07:22:54.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.901+0000 7fa2f9ffb640 1 -- 172.21.15.83:0/2651993762 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa2f4044070 con 0x7fa3040b69d0 2024-03-06T07:22:54.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.901+0000 7fa2f9ffb640 1 --2- 172.21.15.83:0/2651993762 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa2e806b500 0x7fa2e806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.901+0000 7fa2f9ffb640 1 -- 172.21.15.83:0/2651993762 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa2f40ab8a0 con 0x7fa3040b69d0 2024-03-06T07:22:54.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.901+0000 7fa3097b1640 1 --2- 172.21.15.83:0/2651993762 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa2e806b500 0x7fa2e806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.902+0000 7fa3097b1640 1 --2- 172.21.15.83:0/2651993762 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa2e806b500 0x7fa2e806d9c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa3000011e0 tx=0x7fa300001170 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:54.906+0000 7fa2f9ffb640 1 -- 172.21.15.83:0/2651993762 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa3041f7f60 con 0x7fa3040b69d0 2024-03-06T07:22:55.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.150+0000 7fa30ba15640 1 -- 172.21.15.83:0/2651993762 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fa30415d650 con 0x7fa3040b69d0 2024-03-06T07:22:55.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.151+0000 7fa2f9ffb640 1 -- 172.21.15.83:0/2651993762 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) v1 ==== 80+0+6645 (secure 0 0 0) 0x7fa2f407a540 con 0x7fa3040b69d0 2024-03-06T07:22:55.152 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:22:55.153 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:22:55.153 INFO:teuthology.orchestra.run.smithi083.stdout: "auth_dump": [ 2024-03-06T07:22:55.153 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:22:55.153 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mds.a", 2024-03-06T07:22:55.153 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlVHaUGhAAVRpLsHYTUkaYDmceq5Rlgw==", 2024-03-06T07:22:55.153 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:22:55.153 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:22:55.153 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow *", 2024-03-06T07:22:55.153 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow *", 2024-03-06T07:22:55.153 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:22:55.153 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:22:55.153 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:22:55.154 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:22:55.154 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mds.b", 2024-03-06T07:22:55.154 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlFlrrJhAAQbY6xU910eYRg1zwgpj9vg==", 2024-03-06T07:22:55.154 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:22:55.154 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:22:55.154 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow *", 2024-03-06T07:22:55.154 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow *", 2024-03-06T07:22:55.154 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:22:55.154 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:22:55.154 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:22:55.154 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:22:55.154 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mds.c", 2024-03-06T07:22:55.154 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlX0jxIRAAxYNi39qxIFXazlL2qNzLCg==", 2024-03-06T07:22:55.154 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:22:55.155 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:22:55.155 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow *", 2024-03-06T07:22:55.155 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow *", 2024-03-06T07:22:55.155 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:22:55.155 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:22:55.155 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:22:55.155 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:22:55.155 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mds.d", 2024-03-06T07:22:55.155 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehl3utCLhAAzEjASLnYpPYrJsWOt3WwpQ==", 2024-03-06T07:22:55.155 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:22:55.155 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:22:55.155 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow *", 2024-03-06T07:22:55.155 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow *", 2024-03-06T07:22:55.155 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:22:55.155 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:22:55.156 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:22:55.156 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:22:55.156 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.0", 2024-03-06T07:22:55.156 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBgGehlkezBExAAZOLMH2ua3CTWAL7H6LGeZw==", 2024-03-06T07:22:55.156 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:22:55.156 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:55.156 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:22:55.156 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:22:55.156 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:22:55.156 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:22:55.156 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:22:55.156 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.1", 2024-03-06T07:22:55.156 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBjGehlC04qEhAAqirTFR6y3NOGw6N+iz3/pg==", 2024-03-06T07:22:55.157 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:22:55.157 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:55.157 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:22:55.157 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:22:55.157 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:22:55.157 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:22:55.157 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:22:55.157 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.2", 2024-03-06T07:22:55.157 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBmGehlfuFWEBAAj/vjZXrv1wVNSJ1R74UmnQ==", 2024-03-06T07:22:55.157 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:22:55.157 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:55.157 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:22:55.157 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:22:55.157 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:22:55.157 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:22:55.158 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:22:55.158 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.3", 2024-03-06T07:22:55.158 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBpGehl5+ItDhAA0YNc83MhKcgb3JgRuP87aw==", 2024-03-06T07:22:55.158 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:22:55.158 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:55.158 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:22:55.158 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:22:55.158 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:22:55.158 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:22:55.158 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:22:55.158 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.4", 2024-03-06T07:22:55.158 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBvGehlYa/TFBAAvGcizpRQ21pvgpEGgSET0w==", 2024-03-06T07:22:55.158 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:22:55.158 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:55.159 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:22:55.159 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:22:55.159 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:22:55.159 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:22:55.159 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:22:55.159 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.5", 2024-03-06T07:22:55.159 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQByGehlE80TExAAFcWDgnLtW8OKlyu4B0jnmg==", 2024-03-06T07:22:55.159 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:22:55.159 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:55.159 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:22:55.159 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:22:55.159 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:22:55.159 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:22:55.159 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:22:55.159 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.6", 2024-03-06T07:22:55.160 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQB1Gehla/LbERAAYfAjTKL11VMVnGUGV5GVcg==", 2024-03-06T07:22:55.160 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:22:55.160 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:55.160 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:22:55.160 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:22:55.160 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:22:55.160 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:22:55.160 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:22:55.160 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.7", 2024-03-06T07:22:55.160 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQB4GehlpSJkEBAA17tUgMf94jfqvXlD51P8DQ==", 2024-03-06T07:22:55.160 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:22:55.160 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:55.160 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:22:55.161 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:22:55.161 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:22:55.161 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:22:55.161 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:22:55.161 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.0", 2024-03-06T07:22:55.161 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlRwCIMhAA1DdbdQZPplu7LQOTK3BmSg==", 2024-03-06T07:22:55.161 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:22:55.161 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:22:55.161 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow r", 2024-03-06T07:22:55.161 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow rw", 2024-03-06T07:22:55.161 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow rwx" 2024-03-06T07:22:55.161 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:22:55.161 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:22:55.161 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:22:55.161 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.1", 2024-03-06T07:22:55.162 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBdGehl77spABAAsiEf+tXos3MwFrXToEQ8dQ==", 2024-03-06T07:22:55.162 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:22:55.162 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:22:55.162 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow r", 2024-03-06T07:22:55.162 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow rw", 2024-03-06T07:22:55.162 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow rwx" 2024-03-06T07:22:55.162 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:22:55.162 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:22:55.162 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:22:55.162 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.2", 2024-03-06T07:22:55.162 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehltY5SOBAAI8rcIEVqMZaWQGgNaYKJTA==", 2024-03-06T07:22:55.162 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:22:55.162 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:22:55.162 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow r", 2024-03-06T07:22:55.163 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow rw", 2024-03-06T07:22:55.163 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow rwx" 2024-03-06T07:22:55.163 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:22:55.163 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:22:55.163 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:22:55.163 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.3", 2024-03-06T07:22:55.163 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBdGehl9ZX1BRAAA7fKnRyaQ+zqHoSGs4UvDw==", 2024-03-06T07:22:55.163 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:22:55.163 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:22:55.163 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow r", 2024-03-06T07:22:55.163 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow rw", 2024-03-06T07:22:55.163 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow rwx" 2024-03-06T07:22:55.163 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:22:55.163 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:22:55.164 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:22:55.164 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.admin", 2024-03-06T07:22:55.164 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBbGehl4gqRORAAobqSWoeyHYA/lXkp6/EcIQ==", 2024-03-06T07:22:55.164 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:22:55.164 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow *", 2024-03-06T07:22:55.164 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow *", 2024-03-06T07:22:55.164 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow *", 2024-03-06T07:22:55.164 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:22:55.164 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:22:55.164 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:22:55.164 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:22:55.164 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.bootstrap-mds", 2024-03-06T07:22:55.164 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQB+GehlfqwoCxAAr9cvGa18f/sQcx1alrU2lA==", 2024-03-06T07:22:55.164 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:22:55.164 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile bootstrap-mds" 2024-03-06T07:22:55.165 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:22:55.165 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:22:55.165 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:22:55.165 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.bootstrap-mgr", 2024-03-06T07:22:55.165 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQB+GehlQ7QoCxAAPc/Uo0S0b5cyrk89U7x4Hw==", 2024-03-06T07:22:55.165 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:22:55.165 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile bootstrap-mgr" 2024-03-06T07:22:55.165 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:22:55.165 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:22:55.165 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:22:55.165 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.bootstrap-osd", 2024-03-06T07:22:55.165 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQB+GehlyrooCxAAAgL1zAFE80v9177LR2BTDg==", 2024-03-06T07:22:55.165 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:22:55.165 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile bootstrap-osd" 2024-03-06T07:22:55.166 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:22:55.166 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:22:55.166 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:22:55.166 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.bootstrap-rbd", 2024-03-06T07:22:55.166 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQB+GehlQsEoCxAAQRqFws7fnr4ApeiinllrQw==", 2024-03-06T07:22:55.166 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:22:55.166 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile bootstrap-rbd" 2024-03-06T07:22:55.166 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:22:55.166 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:22:55.166 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:22:55.166 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-03-06T07:22:55.166 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQB+Gehl1scoCxAAEG8Li5rJ2fQXYVS1g/jzlg==", 2024-03-06T07:22:55.166 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:22:55.166 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-03-06T07:22:55.166 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:22:55.167 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:22:55.167 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:22:55.167 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.bootstrap-rgw", 2024-03-06T07:22:55.167 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQB+Gehll84oCxAAnbII0E7gFx899QBrIbn/4Q==", 2024-03-06T07:22:55.167 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:22:55.167 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile bootstrap-rgw" 2024-03-06T07:22:55.167 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:22:55.167 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:22:55.167 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:22:55.167 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mgr.x", 2024-03-06T07:22:55.167 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlfikrFxAAoZkTUtpHlBcUAMpwWABRxg==", 2024-03-06T07:22:55.167 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:22:55.167 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow *", 2024-03-06T07:22:55.167 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile mgr", 2024-03-06T07:22:55.168 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:22:55.168 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:22:55.168 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:22:55.168 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:22:55.168 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mgr.y", 2024-03-06T07:22:55.168 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlvdcdExAAySXlEfJqLnpMySiiYGFkMQ==", 2024-03-06T07:22:55.168 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:22:55.168 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow *", 2024-03-06T07:22:55.168 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile mgr", 2024-03-06T07:22:55.168 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:22:55.168 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:22:55.168 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:22:55.168 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-03-06T07:22:55.168 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:22:55.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.152+0000 7fa30ba15640 1 -- 172.21.15.83:0/2651993762 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa2e806b500 msgr2=0x7fa2e806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.152+0000 7fa30ba15640 1 --2- 172.21.15.83:0/2651993762 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa2e806b500 0x7fa2e806d9c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa3000011e0 tx=0x7fa300001170 comp rx=0 tx=0).stop 2024-03-06T07:22:55.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.152+0000 7fa30ba15640 1 -- 172.21.15.83:0/2651993762 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3040b69d0 msgr2=0x7fa304158540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.152+0000 7fa30ba15640 1 --2- 172.21.15.83:0/2651993762 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3040b69d0 0x7fa304158540 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4008fd0 tx=0x7fa2f4001120 comp rx=0 tx=0).stop 2024-03-06T07:22:55.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.153+0000 7fa30ba15640 1 -- 172.21.15.83:0/2651993762 shutdown_connections 2024-03-06T07:22:55.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.153+0000 7fa30ba15640 1 --2- 172.21.15.83:0/2651993762 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa2e806b500 0x7fa2e806d9c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.153+0000 7fa30ba15640 1 --2- 172.21.15.83:0/2651993762 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa3040cada0 0x7fa3040b20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.153+0000 7fa30ba15640 1 --2- 172.21.15.83:0/2651993762 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3040b69d0 0x7fa304158540 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.153+0000 7fa30ba15640 1 --2- 172.21.15.83:0/2651993762 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa2f0004800 0x7fa3041f7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.153+0000 7fa30ba15640 1 -- 172.21.15.83:0/2651993762 >> 172.21.15.83:0/2651993762 conn(0x7fa3040d3cd0 msgr2=0x7fa30406bbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:55.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.153+0000 7fa30ba15640 1 -- 172.21.15.83:0/2651993762 shutdown_connections 2024-03-06T07:22:55.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.153+0000 7fa30ba15640 1 -- 172.21.15.83:0/2651993762 wait complete. 2024-03-06T07:22:55.171 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-03-06T07:22:55.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.270+0000 7f18abfe2640 1 Processor -- start 2024-03-06T07:22:55.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.270+0000 7f18abfe2640 1 -- start start 2024-03-06T07:22:55.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.271+0000 7f18abfe2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18a40b4460 0x7f18a40d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.271+0000 7f18abfe2640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f18a4161cf0 0x7f18a4162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.271+0000 7f18abfe2640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18a40d22c0 con 0x7f18a4162970 2024-03-06T07:22:55.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.271+0000 7f18abfe2640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18a4161b00 con 0x7f18a4161cf0 2024-03-06T07:22:55.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.271+0000 7f18abfe2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18a40cecc0 con 0x7f18a40b4460 2024-03-06T07:22:55.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.271+0000 7f18a957d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f18a4162970 0x7f18a40b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:47234/0 (socket says 172.21.15.83:47234) 2024-03-06T07:22:55.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.271+0000 7f18a957d640 1 -- 172.21.15.83:0/1674425281 learned_addr learned my addr 172.21.15.83:0/1674425281 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:55.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.271+0000 7f18a9d7e640 1 --2- 172.21.15.83:0/1674425281 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18a40b4460 0x7f18a40d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.271+0000 7f18aa57f640 1 --2- 172.21.15.83:0/1674425281 >> v2:172.21.15.183:3300/0 conn(0x7f18a4161cf0 0x7f18a4162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.271+0000 7f18a8d7c640 1 -- 172.21.15.83:0/1674425281 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 964825589 0 0) 0x7f18a40d22c0 con 0x7f18a4162970 2024-03-06T07:22:55.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.271+0000 7f18a8d7c640 1 -- 172.21.15.83:0/1674425281 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1890003520 con 0x7f18a4162970 2024-03-06T07:22:55.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.271+0000 7f18a8d7c640 1 -- 172.21.15.83:0/1674425281 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3830173336 0 0) 0x7f1890003520 con 0x7f18a4162970 2024-03-06T07:22:55.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.271+0000 7f18a8d7c640 1 -- 172.21.15.83:0/1674425281 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18a40b4460 msgr2=0x7f18a40d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:55.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.271+0000 7f18a8d7c640 1 --2- 172.21.15.83:0/1674425281 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18a40b4460 0x7f18a40d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.271+0000 7f18a8d7c640 1 -- 172.21.15.83:0/1674425281 >> v2:172.21.15.183:3300/0 conn(0x7f18a4161cf0 msgr2=0x7f18a4162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:55.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.271+0000 7f18a8d7c640 1 --2- 172.21.15.83:0/1674425281 >> v2:172.21.15.183:3300/0 conn(0x7f18a4161cf0 0x7f18a4162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.271+0000 7f18a8d7c640 1 -- 172.21.15.83:0/1674425281 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a40d25a0 con 0x7f18a4162970 2024-03-06T07:22:55.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.271+0000 7f18a8d7c640 1 -- 172.21.15.83:0/1674425281 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f1894001670 con 0x7f18a4162970 2024-03-06T07:22:55.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.272+0000 7f18a8d7c640 1 -- 172.21.15.83:0/1674425281 >> v1:172.21.15.83:6789/0 conn(0x7f18a4162970 legacy=0x7f18a40b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.272+0000 7f18a8d7c640 1 --2- 172.21.15.83:0/1674425281 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18900039b0 0x7f1890003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.272+0000 7f18a8d7c640 1 --2- 172.21.15.83:0/1674425281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1890004800 0x7f1890004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.272+0000 7f18a8d7c640 1 --2- 172.21.15.83:0/1674425281 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18a40b4460 0x7f18900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.272+0000 7f18a8d7c640 1 -- 172.21.15.83:0/1674425281 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a4161b00 con 0x7f1890004800 2024-03-06T07:22:55.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.272+0000 7f18a8d7c640 1 -- 172.21.15.83:0/1674425281 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a40cecc0 con 0x7f18900039b0 2024-03-06T07:22:55.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.272+0000 7f18a8d7c640 1 -- 172.21.15.83:0/1674425281 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f189000b9c0 con 0x7f18a40b4460 2024-03-06T07:22:55.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.272+0000 7f18a957d640 1 --2- 172.21.15.83:0/1674425281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1890004800 0x7f1890004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.272+0000 7f18a9d7e640 1 --2- 172.21.15.83:0/1674425281 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18900039b0 0x7f1890003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.272+0000 7f18aa57f640 1 --2- 172.21.15.83:0/1674425281 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18a40b4460 0x7f18900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.272+0000 7f18a957d640 1 -- 172.21.15.83:0/1674425281 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18a40b4460 msgr2=0x7f18900051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:55.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.272+0000 7f18a957d640 1 --2- 172.21.15.83:0/1674425281 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18a40b4460 0x7f18900051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.272+0000 7f18a957d640 1 -- 172.21.15.83:0/1674425281 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18900039b0 msgr2=0x7f1890003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:55.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.272+0000 7f18a957d640 1 --2- 172.21.15.83:0/1674425281 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18900039b0 0x7f1890003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.272+0000 7f18a957d640 1 -- 172.21.15.83:0/1674425281 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1890003520 con 0x7f1890004800 2024-03-06T07:22:55.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.273+0000 7f18a957d640 1 --2- 172.21.15.83:0/1674425281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1890004800 0x7f1890004c80 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f1894001050 tx=0x7f1890005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b75a390d2b492fab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.273+0000 7f18a8d7c640 1 -- 172.21.15.83:0/1674425281 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1894007d70 con 0x7f1890004800 2024-03-06T07:22:55.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.273+0000 7f18a8d7c640 1 -- 172.21.15.83:0/1674425281 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1894030070 con 0x7f1890004800 2024-03-06T07:22:55.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.273+0000 7f18a8d7c640 1 -- 172.21.15.83:0/1674425281 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f189402cd40 con 0x7f1890004800 2024-03-06T07:22:55.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.273+0000 7f18abfe2640 1 -- 172.21.15.83:0/1674425281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1890004800 msgr2=0x7f1890004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.273+0000 7f18abfe2640 1 --2- 172.21.15.83:0/1674425281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1890004800 0x7f1890004c80 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f1894001050 tx=0x7f1890005830 comp rx=0 tx=0).stop 2024-03-06T07:22:55.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.273+0000 7f18aa57f640 1 -- 172.21.15.83:0/1674425281 reap_dead start 2024-03-06T07:22:55.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.273+0000 7f18abfe2640 1 -- 172.21.15.83:0/1674425281 shutdown_connections 2024-03-06T07:22:55.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.273+0000 7f18abfe2640 1 -- 172.21.15.83:0/1674425281 >> 172.21.15.83:0/1674425281 conn(0x7f18a40d3ce0 msgr2=0x7f18a40d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:55.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.273+0000 7f18abfe2640 1 -- 172.21.15.83:0/1674425281 shutdown_connections 2024-03-06T07:22:55.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.273+0000 7f18abfe2640 1 -- 172.21.15.83:0/1674425281 wait complete. 2024-03-06T07:22:55.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.275+0000 7f18abfe2640 1 Processor -- start 2024-03-06T07:22:55.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.275+0000 7f18abfe2640 1 -- start start 2024-03-06T07:22:55.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.275+0000 7f18abfe2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1890004800 0x7f18a40b8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.275+0000 7f18abfe2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18a40cadb0 0x7f18a4155f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.275+0000 7f18abfe2640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18a40b1f30 0x7f18a40b23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.275+0000 7f18abfe2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a40ce950 con 0x7f1890004800 2024-03-06T07:22:55.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.275+0000 7f18abfe2640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a40d2b60 con 0x7f18a40b1f30 2024-03-06T07:22:55.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.275+0000 7f18abfe2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18a40b4840 con 0x7f18a40cadb0 2024-03-06T07:22:55.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.275+0000 7f18a9d7e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1890004800 0x7f18a40b8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.275+0000 7f18a9d7e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1890004800 0x7f18a40b8a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36780/0 (socket says 172.21.15.83:36780) 2024-03-06T07:22:55.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.275+0000 7f18a9d7e640 1 -- 172.21.15.83:0/3464242128 learned_addr learned my addr 172.21.15.83:0/3464242128 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:55.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.276+0000 7f18aa57f640 1 --2- 172.21.15.83:0/3464242128 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18a40b1f30 0x7f18a40b23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.276+0000 7f18a957d640 1 --2- 172.21.15.83:0/3464242128 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18a40cadb0 0x7f18a4155f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.276+0000 7f18a9d7e640 1 -- 172.21.15.83:0/3464242128 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18a40cadb0 msgr2=0x7f18a4155f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.276+0000 7f18a9d7e640 1 --2- 172.21.15.83:0/3464242128 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18a40cadb0 0x7f18a4155f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.276+0000 7f18a9d7e640 1 -- 172.21.15.83:0/3464242128 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18a40b1f30 msgr2=0x7f18a40b23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.276+0000 7f18a9d7e640 1 --2- 172.21.15.83:0/3464242128 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18a40b1f30 0x7f18a40b23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.276+0000 7f18a9d7e640 1 -- 172.21.15.83:0/3464242128 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a40cb520 con 0x7f1890004800 2024-03-06T07:22:55.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.276+0000 7f18a9d7e640 1 --2- 172.21.15.83:0/3464242128 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1890004800 0x7f18a40b8a60 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f18a0011910 tx=0x7f18a0011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.276+0000 7f189a7fc640 1 -- 172.21.15.83:0/3464242128 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a000faf0 con 0x7f1890004800 2024-03-06T07:22:55.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.276+0000 7f189a7fc640 1 -- 172.21.15.83:0/3464242128 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18a000fc90 con 0x7f1890004800 2024-03-06T07:22:55.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.276+0000 7f18abfe2640 1 -- 172.21.15.83:0/3464242128 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18a40cb800 con 0x7f1890004800 2024-03-06T07:22:55.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.276+0000 7f189a7fc640 1 -- 172.21.15.83:0/3464242128 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a0014440 con 0x7f1890004800 2024-03-06T07:22:55.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.276+0000 7f18abfe2640 1 -- 172.21.15.83:0/3464242128 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18a40cbdc0 con 0x7f1890004800 2024-03-06T07:22:55.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.277+0000 7f18abfe2640 1 -- 172.21.15.83:0/3464242128 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18a404f330 con 0x7f1890004800 2024-03-06T07:22:55.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.277+0000 7f189a7fc640 1 -- 172.21.15.83:0/3464242128 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f18a0012040 con 0x7f1890004800 2024-03-06T07:22:55.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.278+0000 7f189a7fc640 1 --2- 172.21.15.83:0/3464242128 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f188806b500 0x7f188806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.278+0000 7f189a7fc640 1 -- 172.21.15.83:0/3464242128 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f18a008a330 con 0x7f1890004800 2024-03-06T07:22:55.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.278+0000 7f18a957d640 1 --2- 172.21.15.83:0/3464242128 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f188806b500 0x7f188806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.278+0000 7f18a957d640 1 --2- 172.21.15.83:0/3464242128 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f188806b500 0x7f188806d9c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1894034000 tx=0x7f1894007000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.282+0000 7f189a7fc640 1 -- 172.21.15.83:0/3464242128 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f18a0019030 con 0x7f1890004800 2024-03-06T07:22:55.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.521+0000 7f18abfe2640 1 -- 172.21.15.83:0/3464242128 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f18a406b180 con 0x7f1890004800 2024-03-06T07:22:55.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.523+0000 7f189a7fc640 1 -- 172.21.15.83:0/3464242128 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) v1 ==== 90+0+0 (secure 0 0 0) 0x7f18a0059590 con 0x7f1890004800 2024-03-06T07:22:55.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.525+0000 7f18abfe2640 1 -- 172.21.15.83:0/3464242128 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f188806b500 msgr2=0x7f188806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.525+0000 7f18abfe2640 1 --2- 172.21.15.83:0/3464242128 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f188806b500 0x7f188806d9c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1894034000 tx=0x7f1894007000 comp rx=0 tx=0).stop 2024-03-06T07:22:55.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.525+0000 7f18abfe2640 1 -- 172.21.15.83:0/3464242128 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1890004800 msgr2=0x7f18a40b8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.525+0000 7f18abfe2640 1 --2- 172.21.15.83:0/3464242128 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1890004800 0x7f18a40b8a60 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f18a0011910 tx=0x7f18a0011dd0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.525+0000 7f18abfe2640 1 -- 172.21.15.83:0/3464242128 shutdown_connections 2024-03-06T07:22:55.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.525+0000 7f18abfe2640 1 --2- 172.21.15.83:0/3464242128 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f188806b500 0x7f188806d9c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.525+0000 7f18abfe2640 1 --2- 172.21.15.83:0/3464242128 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18a40b1f30 0x7f18a40b23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.525+0000 7f18abfe2640 1 --2- 172.21.15.83:0/3464242128 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18a40cadb0 0x7f18a4155f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.525+0000 7f18abfe2640 1 --2- 172.21.15.83:0/3464242128 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1890004800 0x7f18a40b8a60 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.525+0000 7f18abfe2640 1 -- 172.21.15.83:0/3464242128 >> 172.21.15.83:0/3464242128 conn(0x7f18a40d3ce0 msgr2=0x7f18a40d1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:55.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.525+0000 7f18abfe2640 1 -- 172.21.15.83:0/3464242128 shutdown_connections 2024-03-06T07:22:55.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.525+0000 7f18abfe2640 1 -- 172.21.15.83:0/3464242128 wait complete. 2024-03-06T07:22:55.538 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-03-06T07:22:55.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.637+0000 7f7a6ad3a640 1 Processor -- start 2024-03-06T07:22:55.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.637+0000 7f7a6ad3a640 1 -- start start 2024-03-06T07:22:55.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.638+0000 7f7a6ad3a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7a640cc920 0x7f7a6415eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.638+0000 7f7a6ad3a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f7a640b4a50 0x7f7a64150b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.638+0000 7f7a6ad3a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a6406e770 con 0x7f7a6406e000 2024-03-06T07:22:55.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.638+0000 7f7a6ad3a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a6415e660 con 0x7f7a640cc920 2024-03-06T07:22:55.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.638+0000 7f7a6ad3a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a6406e490 con 0x7f7a640b4a50 2024-03-06T07:22:55.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.638+0000 7f7a68ad6640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7a640cc920 0x7f7a6415eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.638+0000 7f7a68ad6640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7a640cc920 0x7f7a6415eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36792/0 (socket says 172.21.15.83:36792) 2024-03-06T07:22:55.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.638+0000 7f7a68ad6640 1 -- 172.21.15.83:0/2675586803 learned_addr learned my addr 172.21.15.83:0/2675586803 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:55.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.638+0000 7f7a63fff640 1 --1- 172.21.15.83:0/2675586803 >> v1:172.21.15.83:6789/0 conn(0x7f7a6406e000 0x7f7a64072070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:47242/0 (socket says 172.21.15.83:47242) 2024-03-06T07:22:55.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.638+0000 7f7a68ad6640 1 -- 172.21.15.83:0/2675586803 >> v2:172.21.15.183:3300/0 conn(0x7f7a640b4a50 msgr2=0x7f7a64150b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:55.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.638+0000 7f7a68ad6640 1 --2- 172.21.15.83:0/2675586803 >> v2:172.21.15.183:3300/0 conn(0x7f7a640b4a50 0x7f7a64150b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.638+0000 7f7a68ad6640 1 -- 172.21.15.83:0/2675586803 >> v1:172.21.15.83:6789/0 conn(0x7f7a6406e000 legacy=0x7f7a64072070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.638+0000 7f7a68ad6640 1 -- 172.21.15.83:0/2675586803 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a6415e010 con 0x7f7a640cc920 2024-03-06T07:22:55.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.638+0000 7f7a637fe640 1 -- 172.21.15.83:0/2675586803 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 991736538 0 0) 0x7f7a6406e770 con 0x7f7a6406e000 2024-03-06T07:22:55.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.639+0000 7f7a68ad6640 1 --2- 172.21.15.83:0/2675586803 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a640cc920 0x7f7a6415eaf0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f7a640cd660 tx=0x7f7a5802df50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e43354ca3b3d9c9a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.639+0000 7f7a637fe640 1 -- 172.21.15.83:0/2675586803 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a58034070 con 0x7f7a640cc920 2024-03-06T07:22:55.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.639+0000 7f7a637fe640 1 -- 172.21.15.83:0/2675586803 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a5802ed00 con 0x7f7a640cc920 2024-03-06T07:22:55.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.639+0000 7f7a637fe640 1 -- 172.21.15.83:0/2675586803 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a58038620 con 0x7f7a640cc920 2024-03-06T07:22:55.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.639+0000 7f7a6ad3a640 1 -- 172.21.15.83:0/2675586803 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a640cc920 msgr2=0x7f7a6415eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.639+0000 7f7a6ad3a640 1 --2- 172.21.15.83:0/2675586803 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a640cc920 0x7f7a6415eaf0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f7a640cd660 tx=0x7f7a5802df50 comp rx=0 tx=0).stop 2024-03-06T07:22:55.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.639+0000 7f7a6ad3a640 1 -- 172.21.15.83:0/2675586803 shutdown_connections 2024-03-06T07:22:55.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.639+0000 7f7a6ad3a640 1 --2- 172.21.15.83:0/2675586803 >> v2:172.21.15.183:3300/0 conn(0x7f7a640b4a50 0x7f7a64150b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.639+0000 7f7a6ad3a640 1 --2- 172.21.15.83:0/2675586803 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a640cc920 0x7f7a6415eaf0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.639+0000 7f7a6ad3a640 1 -- 172.21.15.83:0/2675586803 >> 172.21.15.83:0/2675586803 conn(0x7f7a640b1c80 msgr2=0x7f7a640cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:55.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.639+0000 7f7a6ad3a640 1 -- 172.21.15.83:0/2675586803 shutdown_connections 2024-03-06T07:22:55.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.639+0000 7f7a6ad3a640 1 -- 172.21.15.83:0/2675586803 wait complete. 2024-03-06T07:22:55.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.641+0000 7f7a6ad3a640 1 Processor -- start 2024-03-06T07:22:55.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.641+0000 7f7a6ad3a640 1 -- start start 2024-03-06T07:22:55.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.641+0000 7f7a6ad3a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7a640b4a50 0x7f7a641562c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.641+0000 7f7a6ad3a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7a640cc920 0x7f7a641ef610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.641+0000 7f7a6ad3a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a64159250 0x7f7a640b0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.641+0000 7f7a6ad3a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a640cce80 con 0x7f7a64159250 2024-03-06T07:22:55.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.641+0000 7f7a6ad3a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a640ccd00 con 0x7f7a640cc920 2024-03-06T07:22:55.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.641+0000 7f7a6ad3a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a64070940 con 0x7f7a640b4a50 2024-03-06T07:22:55.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.642+0000 7f7a692d7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a64159250 0x7f7a640b0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.642+0000 7f7a692d7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a64159250 0x7f7a640b0610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36794/0 (socket says 172.21.15.83:36794) 2024-03-06T07:22:55.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.642+0000 7f7a692d7640 1 -- 172.21.15.83:0/68776429 learned_addr learned my addr 172.21.15.83:0/68776429 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:55.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.642+0000 7f7a63fff640 1 --2- 172.21.15.83:0/68776429 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7a640cc920 0x7f7a641ef610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.642+0000 7f7a68ad6640 1 --2- 172.21.15.83:0/68776429 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7a640b4a50 0x7f7a641562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.642+0000 7f7a692d7640 1 -- 172.21.15.83:0/68776429 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7a640b4a50 msgr2=0x7f7a641562c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.642+0000 7f7a692d7640 1 --2- 172.21.15.83:0/68776429 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7a640b4a50 0x7f7a641562c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.642+0000 7f7a692d7640 1 -- 172.21.15.83:0/68776429 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7a640cc920 msgr2=0x7f7a641ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.642+0000 7f7a692d7640 1 --2- 172.21.15.83:0/68776429 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7a640cc920 0x7f7a641ef610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.642+0000 7f7a692d7640 1 -- 172.21.15.83:0/68776429 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a641569c0 con 0x7f7a64159250 2024-03-06T07:22:55.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.642+0000 7f7a692d7640 1 --2- 172.21.15.83:0/68776429 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a64159250 0x7f7a640b0610 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c00b970 tx=0x7f7a5c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.642+0000 7f7a617fa640 1 -- 172.21.15.83:0/68776429 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a5c019070 con 0x7f7a64159250 2024-03-06T07:22:55.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.642+0000 7f7a617fa640 1 -- 172.21.15.83:0/68776429 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a5c00cb60 con 0x7f7a64159250 2024-03-06T07:22:55.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.642+0000 7f7a6ad3a640 1 -- 172.21.15.83:0/68776429 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a64156ca0 con 0x7f7a64159250 2024-03-06T07:22:55.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.642+0000 7f7a617fa640 1 -- 172.21.15.83:0/68776429 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a5c014440 con 0x7f7a64159250 2024-03-06T07:22:55.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.642+0000 7f7a6ad3a640 1 -- 172.21.15.83:0/68776429 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a64157260 con 0x7f7a64159250 2024-03-06T07:22:55.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.643+0000 7f7a6ad3a640 1 -- 172.21.15.83:0/68776429 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a6404f330 con 0x7f7a64159250 2024-03-06T07:22:55.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.644+0000 7f7a617fa640 1 -- 172.21.15.83:0/68776429 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7a5c01c070 con 0x7f7a64159250 2024-03-06T07:22:55.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.644+0000 7f7a617fa640 1 --2- 172.21.15.83:0/68776429 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7a4806b4d0 0x7f7a4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.644+0000 7f7a617fa640 1 -- 172.21.15.83:0/68776429 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7a5c089b90 con 0x7f7a64159250 2024-03-06T07:22:55.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.644+0000 7f7a68ad6640 1 --2- 172.21.15.83:0/68776429 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7a4806b4d0 0x7f7a4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.644+0000 7f7a68ad6640 1 --2- 172.21.15.83:0/68776429 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7a4806b4d0 0x7f7a4806d990 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7a58039000 tx=0x7f7a58034000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.648+0000 7f7a617fa640 1 -- 172.21.15.83:0/68776429 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7a5c058df0 con 0x7f7a64159250 2024-03-06T07:22:55.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.888+0000 7f7a6ad3a640 1 -- 172.21.15.83:0/68776429 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f7a6406ce40 con 0x7f7a64159250 2024-03-06T07:22:55.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.892+0000 7f7a617fa640 1 -- 172.21.15.83:0/68776429 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7f7a5c0586f0 con 0x7f7a64159250 2024-03-06T07:22:55.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.894+0000 7f7a6ad3a640 1 -- 172.21.15.83:0/68776429 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7a4806b4d0 msgr2=0x7f7a4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.894+0000 7f7a6ad3a640 1 --2- 172.21.15.83:0/68776429 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7a4806b4d0 0x7f7a4806d990 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7a58039000 tx=0x7f7a58034000 comp rx=0 tx=0).stop 2024-03-06T07:22:55.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.894+0000 7f7a6ad3a640 1 -- 172.21.15.83:0/68776429 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a64159250 msgr2=0x7f7a640b0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.894+0000 7f7a6ad3a640 1 --2- 172.21.15.83:0/68776429 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a64159250 0x7f7a640b0610 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c00b970 tx=0x7f7a5c00be30 comp rx=0 tx=0).stop 2024-03-06T07:22:55.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.894+0000 7f7a6ad3a640 1 -- 172.21.15.83:0/68776429 shutdown_connections 2024-03-06T07:22:55.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.894+0000 7f7a6ad3a640 1 --2- 172.21.15.83:0/68776429 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7a4806b4d0 0x7f7a4806d990 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.894+0000 7f7a6ad3a640 1 --2- 172.21.15.83:0/68776429 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a64159250 0x7f7a640b0610 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.894+0000 7f7a6ad3a640 1 --2- 172.21.15.83:0/68776429 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7a640cc920 0x7f7a641ef610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.894+0000 7f7a6ad3a640 1 --2- 172.21.15.83:0/68776429 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7a640b4a50 0x7f7a641562c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.894+0000 7f7a6ad3a640 1 -- 172.21.15.83:0/68776429 >> 172.21.15.83:0/68776429 conn(0x7f7a640b1c80 msgr2=0x7f7a640d0120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:55.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.894+0000 7f7a6ad3a640 1 -- 172.21.15.83:0/68776429 shutdown_connections 2024-03-06T07:22:55.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:55.894+0000 7f7a6ad3a640 1 -- 172.21.15.83:0/68776429 wait complete. 2024-03-06T07:22:55.906 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-03-06T07:22:56.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.006+0000 7f542413e640 1 Processor -- start 2024-03-06T07:22:56.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.006+0000 7f542413e640 1 -- start start 2024-03-06T07:22:56.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.007+0000 7f542413e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f541c071e00 0x7f541c0cebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.007+0000 7f542413e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f541c072490 con 0x7f541c0cd7c0 2024-03-06T07:22:56.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.007+0000 7f542413e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f541c0d0090 con 0x7f541c15bbf0 2024-03-06T07:22:56.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.007+0000 7f542413e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f541c072250 con 0x7f541c071e00 2024-03-06T07:22:56.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.007+0000 7f54226db640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f541c0cd7c0 0x7f541c0cdbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:47250/0 (socket says 172.21.15.83:47250) 2024-03-06T07:22:56.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.007+0000 7f54226db640 1 -- 172.21.15.83:0/2500194934 learned_addr learned my addr 172.21.15.83:0/2500194934 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:56.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.007+0000 7f5420ed8640 1 -- 172.21.15.83:0/2500194934 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1938864032 0 0) 0x7f541c072490 con 0x7f541c0cd7c0 2024-03-06T07:22:56.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.007+0000 7f5420ed8640 1 -- 172.21.15.83:0/2500194934 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5408003520 con 0x7f541c0cd7c0 2024-03-06T07:22:56.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.007+0000 7f54216d9640 1 --2- 172.21.15.83:0/2500194934 >> v2:172.21.15.183:3300/0 conn(0x7f541c071e00 0x7f541c0cebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.007+0000 7f5420ed8640 1 -- 172.21.15.83:0/2500194934 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2313885492 0 0) 0x7f5408003520 con 0x7f541c0cd7c0 2024-03-06T07:22:56.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.007+0000 7f5420ed8640 1 -- 172.21.15.83:0/2500194934 >> v2:172.21.15.183:3300/0 conn(0x7f541c071e00 msgr2=0x7f541c0cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:56.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.007+0000 7f5420ed8640 1 --2- 172.21.15.83:0/2500194934 >> v2:172.21.15.183:3300/0 conn(0x7f541c071e00 0x7f541c0cebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.007+0000 7f5420ed8640 1 -- 172.21.15.83:0/2500194934 >> v1:172.21.15.183:6789/0 conn(0x7f541c15bbf0 legacy=0x7f541c15bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.007+0000 7f5420ed8640 1 -- 172.21.15.83:0/2500194934 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f541c0d0370 con 0x7f541c0cd7c0 2024-03-06T07:22:56.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.007+0000 7f5420ed8640 1 -- 172.21.15.83:0/2500194934 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f54140019f0 con 0x7f541c0cd7c0 2024-03-06T07:22:56.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.007+0000 7f5420ed8640 1 -- 172.21.15.83:0/2500194934 >> v1:172.21.15.83:6789/0 conn(0x7f541c0cd7c0 legacy=0x7f541c0cdbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.007+0000 7f5420ed8640 1 --2- 172.21.15.83:0/2500194934 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f54080039b0 0x7f5408003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.007+0000 7f5420ed8640 1 --2- 172.21.15.83:0/2500194934 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5408004800 0x7f5408004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.007+0000 7f5420ed8640 1 --2- 172.21.15.83:0/2500194934 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54080051c0 0x7f540800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.007+0000 7f5420ed8640 1 -- 172.21.15.83:0/2500194934 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f541c072250 con 0x7f54080051c0 2024-03-06T07:22:56.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.008+0000 7f5420ed8640 1 -- 172.21.15.83:0/2500194934 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540800c060 con 0x7f5408004800 2024-03-06T07:22:56.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.008+0000 7f5420ed8640 1 -- 172.21.15.83:0/2500194934 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f540800c260 con 0x7f54080039b0 2024-03-06T07:22:56.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.008+0000 7f54226db640 1 --2- 172.21.15.83:0/2500194934 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54080051c0 0x7f540800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.008+0000 7f5421eda640 1 --2- 172.21.15.83:0/2500194934 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f54080039b0 0x7f5408003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.008+0000 7f54216d9640 1 --2- 172.21.15.83:0/2500194934 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5408004800 0x7f5408004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.008+0000 7f54226db640 1 -- 172.21.15.83:0/2500194934 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f54080039b0 msgr2=0x7f5408003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:56.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.008+0000 7f54226db640 1 --2- 172.21.15.83:0/2500194934 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f54080039b0 0x7f5408003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.008+0000 7f54226db640 1 -- 172.21.15.83:0/2500194934 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5408004800 msgr2=0x7f5408004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:56.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.008+0000 7f54226db640 1 --2- 172.21.15.83:0/2500194934 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5408004800 0x7f5408004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.008+0000 7f54226db640 1 -- 172.21.15.83:0/2500194934 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5408003520 con 0x7f54080051c0 2024-03-06T07:22:56.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.009+0000 7f54226db640 1 -- 172.21.15.83:0/2500194934 reap_dead start 2024-03-06T07:22:56.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.009+0000 7f54226db640 1 --2- 172.21.15.83:0/2500194934 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54080051c0 0x7f540800b9c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f541400a840 tx=0x7f5408005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dbe5f7443e0f7cf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.009+0000 7f5420ed8640 1 -- 172.21.15.83:0/2500194934 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f541400a460 con 0x7f54080051c0 2024-03-06T07:22:56.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.009+0000 7f5420ed8640 1 -- 172.21.15.83:0/2500194934 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5414034650 con 0x7f54080051c0 2024-03-06T07:22:56.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.009+0000 7f5420ed8640 1 -- 172.21.15.83:0/2500194934 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5414034a40 con 0x7f54080051c0 2024-03-06T07:22:56.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.009+0000 7f542413e640 1 -- 172.21.15.83:0/2500194934 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54080051c0 msgr2=0x7f540800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.009+0000 7f542413e640 1 --2- 172.21.15.83:0/2500194934 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54080051c0 0x7f540800b9c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f541400a840 tx=0x7f5408005830 comp rx=0 tx=0).stop 2024-03-06T07:22:56.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.009+0000 7f542413e640 1 -- 172.21.15.83:0/2500194934 shutdown_connections 2024-03-06T07:22:56.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.009+0000 7f542413e640 1 --2- 172.21.15.83:0/2500194934 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54080051c0 0x7f540800b9c0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.009+0000 7f542413e640 1 -- 172.21.15.83:0/2500194934 >> 172.21.15.83:0/2500194934 conn(0x7f541c0b1c80 msgr2=0x7f541c0cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:56.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.009+0000 7f542413e640 1 -- 172.21.15.83:0/2500194934 shutdown_connections 2024-03-06T07:22:56.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.009+0000 7f542413e640 1 -- 172.21.15.83:0/2500194934 wait complete. 2024-03-06T07:22:56.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.011+0000 7f542413e640 1 Processor -- start 2024-03-06T07:22:56.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.011+0000 7f542413e640 1 -- start start 2024-03-06T07:22:56.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.011+0000 7f542413e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54080051c0 0x7f541c1f11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.011+0000 7f542413e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f541c15bfd0 0x7f541c15c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.011+0000 7f542413e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f541c149b00 0x7f541c149f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.011+0000 7f542413e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f541c0b3800 con 0x7f54080051c0 2024-03-06T07:22:56.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.011+0000 7f542413e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f541c0b3680 con 0x7f541c149b00 2024-03-06T07:22:56.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.011+0000 7f542413e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f541c0cebc0 con 0x7f541c15bfd0 2024-03-06T07:22:56.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.011+0000 7f5421eda640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54080051c0 0x7f541c1f11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.011+0000 7f5421eda640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54080051c0 0x7f541c1f11c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36818/0 (socket says 172.21.15.83:36818) 2024-03-06T07:22:56.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.011+0000 7f5421eda640 1 -- 172.21.15.83:0/690158300 learned_addr learned my addr 172.21.15.83:0/690158300 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:56.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.011+0000 7f54226db640 1 --2- 172.21.15.83:0/690158300 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f541c149b00 0x7f541c149f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.011+0000 7f54216d9640 1 --2- 172.21.15.83:0/690158300 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f541c15bfd0 0x7f541c15c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.012+0000 7f5421eda640 1 -- 172.21.15.83:0/690158300 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f541c15bfd0 msgr2=0x7f541c15c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.012+0000 7f5421eda640 1 --2- 172.21.15.83:0/690158300 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f541c15bfd0 0x7f541c15c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.012+0000 7f5421eda640 1 -- 172.21.15.83:0/690158300 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f541c149b00 msgr2=0x7f541c149f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.012+0000 7f5421eda640 1 --2- 172.21.15.83:0/690158300 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f541c149b00 0x7f541c149f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.012+0000 7f5421eda640 1 -- 172.21.15.83:0/690158300 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f541c1f1700 con 0x7f54080051c0 2024-03-06T07:22:56.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.012+0000 7f54216d9640 1 --2- 172.21.15.83:0/690158300 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f541c15bfd0 0x7f541c15c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:56.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.012+0000 7f5421eda640 1 --2- 172.21.15.83:0/690158300 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54080051c0 0x7f541c1f11c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f541800f5f0 tx=0x7f541800fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.012+0000 7f54127fc640 1 -- 172.21.15.83:0/690158300 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54180052e0 con 0x7f54080051c0 2024-03-06T07:22:56.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.012+0000 7f54127fc640 1 -- 172.21.15.83:0/690158300 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5418018070 con 0x7f54080051c0 2024-03-06T07:22:56.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.012+0000 7f54127fc640 1 -- 172.21.15.83:0/690158300 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5418003810 con 0x7f54080051c0 2024-03-06T07:22:56.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.012+0000 7f542413e640 1 -- 172.21.15.83:0/690158300 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f541c139440 con 0x7f54080051c0 2024-03-06T07:22:56.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.012+0000 7f542413e640 1 -- 172.21.15.83:0/690158300 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f541c139a00 con 0x7f54080051c0 2024-03-06T07:22:56.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.013+0000 7f542413e640 1 -- 172.21.15.83:0/690158300 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f541c04f330 con 0x7f54080051c0 2024-03-06T07:22:56.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.013+0000 7f54127fc640 1 -- 172.21.15.83:0/690158300 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f54180039b0 con 0x7f54080051c0 2024-03-06T07:22:56.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.014+0000 7f54127fc640 1 --2- 172.21.15.83:0/690158300 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f540006b4d0 0x7f540006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.014+0000 7f54127fc640 1 -- 172.21.15.83:0/690158300 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5418089bf0 con 0x7f54080051c0 2024-03-06T07:22:56.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.014+0000 7f54216d9640 1 --2- 172.21.15.83:0/690158300 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f540006b4d0 0x7f540006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.014+0000 7f54216d9640 1 --2- 172.21.15.83:0/690158300 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f540006b4d0 0x7f540006d990 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f540c0093d0 tx=0x7f540c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.018+0000 7f54127fc640 1 -- 172.21.15.83:0/690158300 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5418058e50 con 0x7f54080051c0 2024-03-06T07:22:56.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.253+0000 7f542413e640 1 -- 172.21.15.83:0/690158300 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f541c158b20 con 0x7f54080051c0 2024-03-06T07:22:56.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.254+0000 7f54127fc640 1 -- 172.21.15.83:0/690158300 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7f5418058750 con 0x7f54080051c0 2024-03-06T07:22:56.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.256+0000 7f542413e640 1 -- 172.21.15.83:0/690158300 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f540006b4d0 msgr2=0x7f540006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.256+0000 7f542413e640 1 --2- 172.21.15.83:0/690158300 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f540006b4d0 0x7f540006d990 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f540c0093d0 tx=0x7f540c008040 comp rx=0 tx=0).stop 2024-03-06T07:22:56.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.256+0000 7f542413e640 1 -- 172.21.15.83:0/690158300 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54080051c0 msgr2=0x7f541c1f11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.256+0000 7f542413e640 1 --2- 172.21.15.83:0/690158300 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54080051c0 0x7f541c1f11c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f541800f5f0 tx=0x7f541800fab0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.257+0000 7f542413e640 1 -- 172.21.15.83:0/690158300 shutdown_connections 2024-03-06T07:22:56.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.257+0000 7f542413e640 1 --2- 172.21.15.83:0/690158300 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f540006b4d0 0x7f540006d990 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.257+0000 7f542413e640 1 --2- 172.21.15.83:0/690158300 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f541c149b00 0x7f541c149f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.257+0000 7f542413e640 1 --2- 172.21.15.83:0/690158300 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f541c15bfd0 0x7f541c15c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.257+0000 7f542413e640 1 --2- 172.21.15.83:0/690158300 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54080051c0 0x7f541c1f11c0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.257+0000 7f542413e640 1 -- 172.21.15.83:0/690158300 >> 172.21.15.83:0/690158300 conn(0x7f541c0b1c80 msgr2=0x7f541c15a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:56.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.257+0000 7f542413e640 1 -- 172.21.15.83:0/690158300 shutdown_connections 2024-03-06T07:22:56.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.257+0000 7f542413e640 1 -- 172.21.15.83:0/690158300 wait complete. 2024-03-06T07:22:56.269 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-03-06T07:22:56.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.368+0000 7fe1f10af640 1 Processor -- start 2024-03-06T07:22:56.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.368+0000 7fe1f10af640 1 -- start start 2024-03-06T07:22:56.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.368+0000 7fe1f10af640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe1ec06e000 0x7fe1ec15eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.368+0000 7fe1f10af640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1ec06e770 con 0x7fe1ec0cc920 2024-03-06T07:22:56.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.368+0000 7fe1f10af640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1ec06e490 con 0x7fe1ec0b4a50 2024-03-06T07:22:56.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.368+0000 7fe1f10af640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1ec15e660 con 0x7fe1ec06e000 2024-03-06T07:22:56.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.369+0000 7fe1ea59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe1ec06e000 0x7fe1ec15eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.369+0000 7fe1ead9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fe1ec0cc920 0x7fe1ec072070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:47256/0 (socket says 172.21.15.83:47256) 2024-03-06T07:22:56.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.369+0000 7fe1ea59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe1ec06e000 0x7fe1ec15eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36824/0 (socket says 172.21.15.83:36824) 2024-03-06T07:22:56.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.369+0000 7fe1ead9d640 1 -- 172.21.15.83:0/1006673542 learned_addr learned my addr 172.21.15.83:0/1006673542 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:56.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.369+0000 7fe1ea59c640 1 -- 172.21.15.83:0/1006673542 >> v1:172.21.15.183:6789/0 conn(0x7fe1ec0b4a50 legacy=0x7fe1ec150b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:56.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.369+0000 7fe1ea59c640 1 -- 172.21.15.83:0/1006673542 >> v1:172.21.15.83:6789/0 conn(0x7fe1ec0cc920 legacy=0x7fe1ec072070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.369+0000 7fe1ea59c640 1 -- 172.21.15.83:0/1006673542 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1ec15e010 con 0x7fe1ec06e000 2024-03-06T07:22:56.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.369+0000 7fe1e9d9b640 1 -- 172.21.15.83:0/1006673542 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 945175532 0 0) 0x7fe1ec06e770 con 0x7fe1ec0cc920 2024-03-06T07:22:56.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.370+0000 7fe1ea59c640 1 --2- 172.21.15.83:0/1006673542 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1ec06e000 0x7fe1ec15eaf0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec15e7e0 tx=0x7fe1e002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=583bb9869134d57b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.370+0000 7fe1e9d9b640 1 -- 172.21.15.83:0/1006673542 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e0034070 con 0x7fe1ec06e000 2024-03-06T07:22:56.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.370+0000 7fe1e9d9b640 1 -- 172.21.15.83:0/1006673542 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1e0030610 con 0x7fe1ec06e000 2024-03-06T07:22:56.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.370+0000 7fe1e9d9b640 1 -- 172.21.15.83:0/1006673542 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e00309e0 con 0x7fe1ec06e000 2024-03-06T07:22:56.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.370+0000 7fe1f10af640 1 -- 172.21.15.83:0/1006673542 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1ec06e000 msgr2=0x7fe1ec15eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.370+0000 7fe1f10af640 1 --2- 172.21.15.83:0/1006673542 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1ec06e000 0x7fe1ec15eaf0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec15e7e0 tx=0x7fe1e002b960 comp rx=0 tx=0).stop 2024-03-06T07:22:56.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.370+0000 7fe1f10af640 1 -- 172.21.15.83:0/1006673542 shutdown_connections 2024-03-06T07:22:56.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.370+0000 7fe1f10af640 1 --2- 172.21.15.83:0/1006673542 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1ec06e000 0x7fe1ec15eaf0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.370+0000 7fe1f10af640 1 -- 172.21.15.83:0/1006673542 >> 172.21.15.83:0/1006673542 conn(0x7fe1ec0b1c80 msgr2=0x7fe1ec0cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:56.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.370+0000 7fe1f10af640 1 -- 172.21.15.83:0/1006673542 shutdown_connections 2024-03-06T07:22:56.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.370+0000 7fe1f10af640 1 -- 172.21.15.83:0/1006673542 wait complete. 2024-03-06T07:22:56.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.372+0000 7fe1f10af640 1 Processor -- start 2024-03-06T07:22:56.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.372+0000 7fe1f10af640 1 -- start start 2024-03-06T07:22:56.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.372+0000 7fe1f10af640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe1ec0b4a50 0x7fe1ec0af5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.372+0000 7fe1f10af640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1ec0cc920 0x7fe1ec13f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.372+0000 7fe1f10af640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe1ec0b0fa0 0x7fe1ec169340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.372+0000 7fe1f10af640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ec0cce80 con 0x7fe1ec0cc920 2024-03-06T07:22:56.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.372+0000 7fe1f10af640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ec0ccd00 con 0x7fe1ec0b4a50 2024-03-06T07:22:56.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.372+0000 7fe1f10af640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ec070940 con 0x7fe1ec0b0fa0 2024-03-06T07:22:56.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.372+0000 7fe1ea59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1ec0cc920 0x7fe1ec13f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.372+0000 7fe1ea59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1ec0cc920 0x7fe1ec13f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36834/0 (socket says 172.21.15.83:36834) 2024-03-06T07:22:56.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.372+0000 7fe1ea59c640 1 -- 172.21.15.83:0/1153015483 learned_addr learned my addr 172.21.15.83:0/1153015483 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:56.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.373+0000 7fe1ead9d640 1 --2- 172.21.15.83:0/1153015483 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe1ec0b4a50 0x7fe1ec0af5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.373+0000 7fe1eb59e640 1 --2- 172.21.15.83:0/1153015483 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe1ec0b0fa0 0x7fe1ec169340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.373+0000 7fe1ea59c640 1 -- 172.21.15.83:0/1153015483 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe1ec0b0fa0 msgr2=0x7fe1ec169340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.373+0000 7fe1ea59c640 1 --2- 172.21.15.83:0/1153015483 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe1ec0b0fa0 0x7fe1ec169340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.373+0000 7fe1ea59c640 1 -- 172.21.15.83:0/1153015483 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe1ec0b4a50 msgr2=0x7fe1ec0af5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.373+0000 7fe1ea59c640 1 --2- 172.21.15.83:0/1153015483 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe1ec0b4a50 0x7fe1ec0af5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.373+0000 7fe1ea59c640 1 -- 172.21.15.83:0/1153015483 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1ec0afd30 con 0x7fe1ec0cc920 2024-03-06T07:22:56.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.373+0000 7fe1ead9d640 1 --2- 172.21.15.83:0/1153015483 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe1ec0b4a50 0x7fe1ec0af5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:56.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.373+0000 7fe1ea59c640 1 --2- 172.21.15.83:0/1153015483 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1ec0cc920 0x7fe1ec13f120 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe1e0009890 tx=0x7fe1e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.373+0000 7fe1d37fe640 1 -- 172.21.15.83:0/1153015483 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e0034040 con 0x7fe1ec0cc920 2024-03-06T07:22:56.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.373+0000 7fe1d37fe640 1 -- 172.21.15.83:0/1153015483 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1e0039a10 con 0x7fe1ec0cc920 2024-03-06T07:22:56.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.373+0000 7fe1f10af640 1 -- 172.21.15.83:0/1153015483 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1ec0b0010 con 0x7fe1ec0cc920 2024-03-06T07:22:56.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.373+0000 7fe1d37fe640 1 -- 172.21.15.83:0/1153015483 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e0039e00 con 0x7fe1ec0cc920 2024-03-06T07:22:56.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.373+0000 7fe1f10af640 1 -- 172.21.15.83:0/1153015483 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1ec0b05d0 con 0x7fe1ec0cc920 2024-03-06T07:22:56.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.374+0000 7fe1f10af640 1 -- 172.21.15.83:0/1153015483 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1ec04f330 con 0x7fe1ec0cc920 2024-03-06T07:22:56.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.375+0000 7fe1d37fe640 1 -- 172.21.15.83:0/1153015483 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe1e0037070 con 0x7fe1ec0cc920 2024-03-06T07:22:56.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.375+0000 7fe1d37fe640 1 --2- 172.21.15.83:0/1153015483 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe1d806b500 0x7fe1d806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.375+0000 7fe1d37fe640 1 -- 172.21.15.83:0/1153015483 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe1e00abfb0 con 0x7fe1ec0cc920 2024-03-06T07:22:56.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.375+0000 7fe1ead9d640 1 --2- 172.21.15.83:0/1153015483 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe1d806b500 0x7fe1d806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.375+0000 7fe1ead9d640 1 --2- 172.21.15.83:0/1153015483 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe1d806b500 0x7fe1d806d9c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc007fd0 tx=0x7fe1dc007de0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.379+0000 7fe1d37fe640 1 -- 172.21.15.83:0/1153015483 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe1e007b180 con 0x7fe1ec0cc920 2024-03-06T07:22:56.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.615+0000 7fe1f10af640 1 -- 172.21.15.83:0/1153015483 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7fe1ec06d410 con 0x7fe1ec0cc920 2024-03-06T07:22:56.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.619+0000 7fe1d37fe640 1 -- 172.21.15.83:0/1153015483 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7fe1e007aa80 con 0x7fe1ec0cc920 2024-03-06T07:22:56.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.621+0000 7fe1f10af640 1 -- 172.21.15.83:0/1153015483 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe1d806b500 msgr2=0x7fe1d806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.621+0000 7fe1f10af640 1 --2- 172.21.15.83:0/1153015483 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe1d806b500 0x7fe1d806d9c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc007fd0 tx=0x7fe1dc007de0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.621+0000 7fe1f10af640 1 -- 172.21.15.83:0/1153015483 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1ec0cc920 msgr2=0x7fe1ec13f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.621+0000 7fe1f10af640 1 --2- 172.21.15.83:0/1153015483 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1ec0cc920 0x7fe1ec13f120 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe1e0009890 tx=0x7fe1e00046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.621+0000 7fe1f10af640 1 -- 172.21.15.83:0/1153015483 shutdown_connections 2024-03-06T07:22:56.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.621+0000 7fe1f10af640 1 --2- 172.21.15.83:0/1153015483 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe1d806b500 0x7fe1d806d9c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.621+0000 7fe1f10af640 1 --2- 172.21.15.83:0/1153015483 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe1ec0b0fa0 0x7fe1ec169340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.621+0000 7fe1f10af640 1 --2- 172.21.15.83:0/1153015483 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1ec0cc920 0x7fe1ec13f120 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.621+0000 7fe1f10af640 1 --2- 172.21.15.83:0/1153015483 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe1ec0b4a50 0x7fe1ec0af5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.621+0000 7fe1f10af640 1 -- 172.21.15.83:0/1153015483 >> 172.21.15.83:0/1153015483 conn(0x7fe1ec0b1c80 msgr2=0x7fe1ec0b3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:56.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.621+0000 7fe1f10af640 1 -- 172.21.15.83:0/1153015483 shutdown_connections 2024-03-06T07:22:56.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.621+0000 7fe1f10af640 1 -- 172.21.15.83:0/1153015483 wait complete. 2024-03-06T07:22:56.634 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-03-06T07:22:56.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.733+0000 7f53c22af640 1 Processor -- start 2024-03-06T07:22:56.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.733+0000 7f53c22af640 1 -- start start 2024-03-06T07:22:56.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.734+0000 7f53c22af640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f53bc15e000 0x7f53bc0d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.734+0000 7f53c22af640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f53bc0d4810 0x7f53bc0d0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.734+0000 7f53c22af640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53bc0d25e0 con 0x7f53bc0c80e0 2024-03-06T07:22:56.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.734+0000 7f53c22af640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53bc0c7c50 con 0x7f53bc0d4810 2024-03-06T07:22:56.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.734+0000 7f53c22af640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53bc0d2880 con 0x7f53bc15e000 2024-03-06T07:22:56.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.734+0000 7f53bb7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f53bc0d4810 0x7f53bc0d0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.734+0000 7f53bb7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f53bc0d4810 0x7f53bc0d0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35106/0 (socket says 172.21.15.83:35106) 2024-03-06T07:22:56.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.734+0000 7f53bb7fe640 1 -- 172.21.15.83:0/3526974261 learned_addr learned my addr 172.21.15.83:0/3526974261 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:56.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.734+0000 7f53bbfff640 1 --2- 172.21.15.83:0/3526974261 >> v2:172.21.15.183:3300/0 conn(0x7f53bc15e000 0x7f53bc0d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.734+0000 7f53bb7fe640 1 -- 172.21.15.83:0/3526974261 >> v2:172.21.15.183:3300/0 conn(0x7f53bc15e000 msgr2=0x7f53bc0d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:56.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.734+0000 7f53bb7fe640 1 --2- 172.21.15.83:0/3526974261 >> v2:172.21.15.183:3300/0 conn(0x7f53bc15e000 0x7f53bc0d0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.735+0000 7f53bb7fe640 1 -- 172.21.15.83:0/3526974261 >> v1:172.21.15.183:6789/0 conn(0x7f53bc0c80e0 legacy=0x7f53bc0c8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:56.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.735+0000 7f53bb7fe640 1 -- 172.21.15.83:0/3526974261 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53bc0d2b40 con 0x7f53bc0d4810 2024-03-06T07:22:56.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.735+0000 7f53bb7fe640 1 --2- 172.21.15.83:0/3526974261 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53bc0d4810 0x7f53bc0d0bc0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f53bc08c360 tx=0x7f53b002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5cdbecc961f3869 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.735+0000 7f53baffd640 1 -- 172.21.15.83:0/3526974261 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b0034070 con 0x7f53bc0d4810 2024-03-06T07:22:56.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.735+0000 7f53baffd640 1 -- 172.21.15.83:0/3526974261 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53b0030860 con 0x7f53bc0d4810 2024-03-06T07:22:56.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.735+0000 7f53baffd640 1 -- 172.21.15.83:0/3526974261 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b0030c30 con 0x7f53bc0d4810 2024-03-06T07:22:56.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.736+0000 7f53c22af640 1 -- 172.21.15.83:0/3526974261 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53bc0d4810 msgr2=0x7f53bc0d0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.736+0000 7f53c22af640 1 --2- 172.21.15.83:0/3526974261 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53bc0d4810 0x7f53bc0d0bc0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f53bc08c360 tx=0x7f53b002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.736+0000 7f53c22af640 1 -- 172.21.15.83:0/3526974261 shutdown_connections 2024-03-06T07:22:56.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.736+0000 7f53c22af640 1 --2- 172.21.15.83:0/3526974261 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53bc0d4810 0x7f53bc0d0bc0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.736+0000 7f53c22af640 1 --2- 172.21.15.83:0/3526974261 >> v2:172.21.15.183:3300/0 conn(0x7f53bc15e000 0x7f53bc0d0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.736+0000 7f53c22af640 1 -- 172.21.15.83:0/3526974261 >> 172.21.15.83:0/3526974261 conn(0x7f53bc0d3cf0 msgr2=0x7f53bc0d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:56.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.736+0000 7f53c22af640 1 -- 172.21.15.83:0/3526974261 shutdown_connections 2024-03-06T07:22:56.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.736+0000 7f53c22af640 1 -- 172.21.15.83:0/3526974261 wait complete. 2024-03-06T07:22:56.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.737+0000 7f53c22af640 1 Processor -- start 2024-03-06T07:22:56.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.737+0000 7f53c22af640 1 -- start start 2024-03-06T07:22:56.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.738+0000 7f53c22af640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53bc0c80e0 0x7f53bc0a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.738+0000 7f53c22af640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f53bc0d4810 0x7f53bc15be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.738+0000 7f53c22af640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53bc15e000 0x7f53bc15c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.738+0000 7f53c22af640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc15e3e0 con 0x7f53bc0c80e0 2024-03-06T07:22:56.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.738+0000 7f53c22af640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc0d22d0 con 0x7f53bc15e000 2024-03-06T07:22:56.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.738+0000 7f53c22af640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc0cf7d0 con 0x7f53bc0d4810 2024-03-06T07:22:56.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.738+0000 7f53bbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53bc0c80e0 0x7f53bc0a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.738+0000 7f53bbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53bc0c80e0 0x7f53bc0a1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35118/0 (socket says 172.21.15.83:35118) 2024-03-06T07:22:56.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.738+0000 7f53bbfff640 1 -- 172.21.15.83:0/1266324405 learned_addr learned my addr 172.21.15.83:0/1266324405 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:56.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.738+0000 7f53bbfff640 1 -- 172.21.15.83:0/1266324405 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f53bc0d4810 msgr2=0x7f53bc15be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.738+0000 7f53bbfff640 1 --2- 172.21.15.83:0/1266324405 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f53bc0d4810 0x7f53bc15be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.738+0000 7f53bbfff640 1 -- 172.21.15.83:0/1266324405 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53bc15e000 msgr2=0x7f53bc15c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.738+0000 7f53bbfff640 1 --2- 172.21.15.83:0/1266324405 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53bc15e000 0x7f53bc15c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.738+0000 7f53bbfff640 1 -- 172.21.15.83:0/1266324405 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53bc1f5030 con 0x7f53bc0c80e0 2024-03-06T07:22:56.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.738+0000 7f53bbfff640 1 --2- 172.21.15.83:0/1266324405 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53bc0c80e0 0x7f53bc0a1210 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f53ac00d5f0 tx=0x7f53ac00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.739+0000 7f53b8ff9640 1 -- 172.21.15.83:0/1266324405 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ac0125f0 con 0x7f53bc0c80e0 2024-03-06T07:22:56.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.739+0000 7f53b8ff9640 1 -- 172.21.15.83:0/1266324405 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53ac019070 con 0x7f53bc0c80e0 2024-03-06T07:22:56.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.739+0000 7f53b8ff9640 1 -- 172.21.15.83:0/1266324405 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ac0129d0 con 0x7f53bc0c80e0 2024-03-06T07:22:56.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.739+0000 7f53c22af640 1 -- 172.21.15.83:0/1266324405 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53bc1f5310 con 0x7f53bc0c80e0 2024-03-06T07:22:56.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.739+0000 7f53c22af640 1 -- 172.21.15.83:0/1266324405 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53bc0b0c60 con 0x7f53bc0c80e0 2024-03-06T07:22:56.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.740+0000 7f53c22af640 1 -- 172.21.15.83:0/1266324405 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53bc0b1090 con 0x7f53bc0c80e0 2024-03-06T07:22:56.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.740+0000 7f53b8ff9640 1 -- 172.21.15.83:0/1266324405 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f53ac012b70 con 0x7f53bc0c80e0 2024-03-06T07:22:56.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.740+0000 7f53b8ff9640 1 --2- 172.21.15.83:0/1266324405 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f53a806b4d0 0x7f53a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.740+0000 7f53b8ff9640 1 -- 172.21.15.83:0/1266324405 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f53ac08a2c0 con 0x7f53bc0c80e0 2024-03-06T07:22:56.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.740+0000 7f53bb7fe640 1 --2- 172.21.15.83:0/1266324405 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f53a806b4d0 0x7f53a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.741+0000 7f53bb7fe640 1 --2- 172.21.15.83:0/1266324405 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f53a806b4d0 0x7f53a806d990 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f53b0036000 tx=0x7f53b003b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.745+0000 7f53b8ff9640 1 -- 172.21.15.83:0/1266324405 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f53ac059490 con 0x7f53bc0c80e0 2024-03-06T07:22:56.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.985+0000 7f53c22af640 1 -- 172.21.15.83:0/1266324405 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f53bc04f330 con 0x7f53bc0c80e0 2024-03-06T07:22:56.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.989+0000 7f53b8ff9640 1 -- 172.21.15.83:0/1266324405 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) v1 ==== 97+0+0 (secure 0 0 0) 0x7f53ac058d90 con 0x7f53bc0c80e0 2024-03-06T07:22:56.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.991+0000 7f53c22af640 1 -- 172.21.15.83:0/1266324405 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f53a806b4d0 msgr2=0x7f53a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.991+0000 7f53c22af640 1 --2- 172.21.15.83:0/1266324405 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f53a806b4d0 0x7f53a806d990 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f53b0036000 tx=0x7f53b003b040 comp rx=0 tx=0).stop 2024-03-06T07:22:56.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.991+0000 7f53c22af640 1 -- 172.21.15.83:0/1266324405 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53bc0c80e0 msgr2=0x7f53bc0a1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.991+0000 7f53c22af640 1 --2- 172.21.15.83:0/1266324405 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53bc0c80e0 0x7f53bc0a1210 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f53ac00d5f0 tx=0x7f53ac00dab0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.991+0000 7f53c22af640 1 -- 172.21.15.83:0/1266324405 shutdown_connections 2024-03-06T07:22:56.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.991+0000 7f53c22af640 1 --2- 172.21.15.83:0/1266324405 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f53a806b4d0 0x7f53a806d990 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.991+0000 7f53c22af640 1 --2- 172.21.15.83:0/1266324405 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53bc15e000 0x7f53bc15c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.991+0000 7f53c22af640 1 --2- 172.21.15.83:0/1266324405 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f53bc0d4810 0x7f53bc15be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.991+0000 7f53c22af640 1 --2- 172.21.15.83:0/1266324405 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53bc0c80e0 0x7f53bc0a1210 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.991+0000 7f53c22af640 1 -- 172.21.15.83:0/1266324405 >> 172.21.15.83:0/1266324405 conn(0x7f53bc0d3cf0 msgr2=0x7f53bc0c8a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:56.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.991+0000 7f53c22af640 1 -- 172.21.15.83:0/1266324405 shutdown_connections 2024-03-06T07:22:56.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:56.991+0000 7f53c22af640 1 -- 172.21.15.83:0/1266324405 wait complete. 2024-03-06T07:22:57.004 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-03-06T07:22:57.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.104+0000 7faeb2555640 1 Processor -- start 2024-03-06T07:22:57.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.104+0000 7faeb2555640 1 -- start start 2024-03-06T07:22:57.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.105+0000 7faeb2555640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7faeac0b4c00 0x7faeac0b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.105+0000 7faeb2555640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeac15d510 con 0x7faeac0b28a0 2024-03-06T07:22:57.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.105+0000 7faeb2555640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeac15d780 con 0x7faeac15ba50 2024-03-06T07:22:57.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.105+0000 7faeb2555640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faeac0b3420 con 0x7faeac0b4c00 2024-03-06T07:22:57.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.105+0000 7faeab7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7faeac0b28a0 0x7faeac0b2d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:50846/0 (socket says 172.21.15.83:50846) 2024-03-06T07:22:57.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.105+0000 7faeab7fe640 1 -- 172.21.15.83:0/500262471 learned_addr learned my addr 172.21.15.83:0/500262471 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:57.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.105+0000 7faeaaffd640 1 -- 172.21.15.83:0/500262471 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2022051095 0 0) 0x7faeac15d510 con 0x7faeac0b28a0 2024-03-06T07:22:57.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.105+0000 7faeaaffd640 1 -- 172.21.15.83:0/500262471 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae94003520 con 0x7faeac0b28a0 2024-03-06T07:22:57.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.105+0000 7faeabfff640 1 --2- 172.21.15.83:0/500262471 >> v2:172.21.15.183:3300/0 conn(0x7faeac0b4c00 0x7faeac0b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.105+0000 7faeaaffd640 1 -- 172.21.15.83:0/500262471 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1125364810 0 0) 0x7fae94003520 con 0x7faeac0b28a0 2024-03-06T07:22:57.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.105+0000 7faeaaffd640 1 -- 172.21.15.83:0/500262471 >> v2:172.21.15.183:3300/0 conn(0x7faeac0b4c00 msgr2=0x7faeac0b4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:57.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.105+0000 7faeaaffd640 1 --2- 172.21.15.83:0/500262471 >> v2:172.21.15.183:3300/0 conn(0x7faeac0b4c00 0x7faeac0b4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.105+0000 7faeaaffd640 1 -- 172.21.15.83:0/500262471 >> v1:172.21.15.183:6789/0 conn(0x7faeac15ba50 legacy=0x7faeac15bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.105+0000 7faeaaffd640 1 -- 172.21.15.83:0/500262471 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeac15da60 con 0x7faeac0b28a0 2024-03-06T07:22:57.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.105+0000 7faeaaffd640 1 -- 172.21.15.83:0/500262471 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7faea0001670 con 0x7faeac0b28a0 2024-03-06T07:22:57.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.105+0000 7faeaaffd640 1 -- 172.21.15.83:0/500262471 >> v1:172.21.15.83:6789/0 conn(0x7faeac0b28a0 legacy=0x7faeac0b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.105+0000 7faeaaffd640 1 --2- 172.21.15.83:0/500262471 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae940039b0 0x7fae94003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.106+0000 7faeaaffd640 1 --2- 172.21.15.83:0/500262471 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae94004800 0x7fae94004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.106+0000 7faeaaffd640 1 --2- 172.21.15.83:0/500262471 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fae940051c0 0x7fae9400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.106+0000 7faeaaffd640 1 -- 172.21.15.83:0/500262471 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeac0b3420 con 0x7fae940039b0 2024-03-06T07:22:57.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.106+0000 7faeaaffd640 1 -- 172.21.15.83:0/500262471 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9400c070 con 0x7fae94004800 2024-03-06T07:22:57.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.106+0000 7faeaaffd640 1 -- 172.21.15.83:0/500262471 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae9400c270 con 0x7fae940051c0 2024-03-06T07:22:57.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.106+0000 7faeabfff640 1 --2- 172.21.15.83:0/500262471 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae940039b0 0x7fae94003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.106+0000 7faeab7fe640 1 --2- 172.21.15.83:0/500262471 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae94004800 0x7fae94004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.106+0000 7faeb0af2640 1 --2- 172.21.15.83:0/500262471 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fae940051c0 0x7fae9400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.106+0000 7faeabfff640 1 -- 172.21.15.83:0/500262471 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fae940051c0 msgr2=0x7fae9400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:57.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.106+0000 7faeabfff640 1 --2- 172.21.15.83:0/500262471 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fae940051c0 0x7fae9400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.106+0000 7faeabfff640 1 -- 172.21.15.83:0/500262471 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae94004800 msgr2=0x7fae94004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:57.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.106+0000 7faeabfff640 1 --2- 172.21.15.83:0/500262471 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae94004800 0x7fae94004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.106+0000 7faeabfff640 1 -- 172.21.15.83:0/500262471 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae94003520 con 0x7fae940039b0 2024-03-06T07:22:57.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.106+0000 7faeb0af2640 1 -- 172.21.15.83:0/500262471 reap_dead start 2024-03-06T07:22:57.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.106+0000 7faeab7fe640 1 --2- 172.21.15.83:0/500262471 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae94004800 0x7fae94004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:57.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.107+0000 7faeabfff640 1 --2- 172.21.15.83:0/500262471 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae940039b0 0x7fae94003dd0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fae9c009ac0 tx=0x7fae94005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f195da0d8425bd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:57.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.107+0000 7faeaaffd640 1 -- 172.21.15.83:0/500262471 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae9c039070 con 0x7fae940039b0 2024-03-06T07:22:57.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.107+0000 7faeaaffd640 1 -- 172.21.15.83:0/500262471 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae9c02c6b0 con 0x7fae940039b0 2024-03-06T07:22:57.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.107+0000 7faeaaffd640 1 -- 172.21.15.83:0/500262471 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae9c009690 con 0x7fae940039b0 2024-03-06T07:22:57.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.107+0000 7faeb2555640 1 -- 172.21.15.83:0/500262471 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae940039b0 msgr2=0x7fae94003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.107+0000 7faeb2555640 1 --2- 172.21.15.83:0/500262471 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae940039b0 0x7fae94003dd0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fae9c009ac0 tx=0x7fae94005830 comp rx=0 tx=0).stop 2024-03-06T07:22:57.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.107+0000 7faeb2555640 1 -- 172.21.15.83:0/500262471 shutdown_connections 2024-03-06T07:22:57.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.107+0000 7faeb2555640 1 --2- 172.21.15.83:0/500262471 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae940039b0 0x7fae94003dd0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.107+0000 7faeb2555640 1 -- 172.21.15.83:0/500262471 >> 172.21.15.83:0/500262471 conn(0x7faeac09a9f0 msgr2=0x7faeac09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:57.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.107+0000 7faeb2555640 1 -- 172.21.15.83:0/500262471 shutdown_connections 2024-03-06T07:22:57.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.107+0000 7faeb2555640 1 -- 172.21.15.83:0/500262471 wait complete. 2024-03-06T07:22:57.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.108+0000 7faeb2555640 1 Processor -- start 2024-03-06T07:22:57.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.109+0000 7faeb2555640 1 -- start start 2024-03-06T07:22:57.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.109+0000 7faeb2555640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae940039b0 0x7faeac142600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.109+0000 7faeb2555640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faeac1e75d0 0x7faeac1e7a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.109+0000 7faeb2555640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faeac155450 0x7faeac1558d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.109+0000 7faeb2555640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeac0b4fe0 con 0x7fae940039b0 2024-03-06T07:22:57.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.109+0000 7faeb2555640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeac15e0b0 con 0x7faeac1e75d0 2024-03-06T07:22:57.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.109+0000 7faeb2555640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faeac0b5160 con 0x7faeac155450 2024-03-06T07:22:57.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.109+0000 7faeabfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae940039b0 0x7faeac142600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.109+0000 7faeabfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae940039b0 0x7faeac142600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35138/0 (socket says 172.21.15.83:35138) 2024-03-06T07:22:57.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.109+0000 7faeabfff640 1 -- 172.21.15.83:0/3874085966 learned_addr learned my addr 172.21.15.83:0/3874085966 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:57.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.109+0000 7faeab7fe640 1 --2- 172.21.15.83:0/3874085966 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faeac1e75d0 0x7faeac1e7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.109+0000 7faeb0af2640 1 --2- 172.21.15.83:0/3874085966 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faeac155450 0x7faeac1558d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.110+0000 7faeabfff640 1 -- 172.21.15.83:0/3874085966 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faeac155450 msgr2=0x7faeac1558d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.110+0000 7faeabfff640 1 --2- 172.21.15.83:0/3874085966 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faeac155450 0x7faeac1558d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.110+0000 7faeabfff640 1 -- 172.21.15.83:0/3874085966 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faeac1e75d0 msgr2=0x7faeac1e7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.110+0000 7faeabfff640 1 --2- 172.21.15.83:0/3874085966 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faeac1e75d0 0x7faeac1e7a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.110+0000 7faeabfff640 1 -- 172.21.15.83:0/3874085966 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeac152d40 con 0x7fae940039b0 2024-03-06T07:22:57.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.110+0000 7faeabfff640 1 --2- 172.21.15.83:0/3874085966 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae940039b0 0x7faeac142600 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fae9c000c00 tx=0x7fae9c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:57.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.110+0000 7faea8ff9640 1 -- 172.21.15.83:0/3874085966 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae9c042020 con 0x7fae940039b0 2024-03-06T07:22:57.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.110+0000 7faea8ff9640 1 -- 172.21.15.83:0/3874085966 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae9c03a040 con 0x7fae940039b0 2024-03-06T07:22:57.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.110+0000 7faea8ff9640 1 -- 172.21.15.83:0/3874085966 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae9c039070 con 0x7fae940039b0 2024-03-06T07:22:57.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.110+0000 7faeb2555640 1 -- 172.21.15.83:0/3874085966 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeac155e10 con 0x7fae940039b0 2024-03-06T07:22:57.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.110+0000 7faeb2555640 1 -- 172.21.15.83:0/3874085966 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faeac1563a0 con 0x7fae940039b0 2024-03-06T07:22:57.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.111+0000 7faeb2555640 1 -- 172.21.15.83:0/3874085966 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faeac04f330 con 0x7fae940039b0 2024-03-06T07:22:57.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.111+0000 7faea8ff9640 1 -- 172.21.15.83:0/3874085966 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fae9c045020 con 0x7fae940039b0 2024-03-06T07:22:57.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.112+0000 7faea8ff9640 1 --2- 172.21.15.83:0/3874085966 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fae9806b4d0 0x7fae9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.112+0000 7faea8ff9640 1 -- 172.21.15.83:0/3874085966 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fae9c0ab260 con 0x7fae940039b0 2024-03-06T07:22:57.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.112+0000 7faeab7fe640 1 --2- 172.21.15.83:0/3874085966 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fae9806b4d0 0x7fae9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.112+0000 7faeab7fe640 1 --2- 172.21.15.83:0/3874085966 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fae9806b4d0 0x7fae9806d990 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7faeac1f6fe0 tx=0x7faea0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:57.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.116+0000 7faea8ff9640 1 -- 172.21.15.83:0/3874085966 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fae9c04c070 con 0x7fae940039b0 2024-03-06T07:22:57.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.353+0000 7faeb2555640 1 -- 172.21.15.83:0/3874085966 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7faeac1582c0 con 0x7fae940039b0 2024-03-06T07:22:57.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.355+0000 7faea8ff9640 1 -- 172.21.15.83:0/3874085966 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7fae9c07a430 con 0x7fae940039b0 2024-03-06T07:22:57.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.357+0000 7faeb2555640 1 -- 172.21.15.83:0/3874085966 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fae9806b4d0 msgr2=0x7fae9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.357+0000 7faeb2555640 1 --2- 172.21.15.83:0/3874085966 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fae9806b4d0 0x7fae9806d990 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7faeac1f6fe0 tx=0x7faea0009040 comp rx=0 tx=0).stop 2024-03-06T07:22:57.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.357+0000 7faeb2555640 1 -- 172.21.15.83:0/3874085966 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae940039b0 msgr2=0x7faeac142600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.357+0000 7faeb2555640 1 --2- 172.21.15.83:0/3874085966 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae940039b0 0x7faeac142600 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fae9c000c00 tx=0x7fae9c004730 comp rx=0 tx=0).stop 2024-03-06T07:22:57.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.357+0000 7faeb2555640 1 -- 172.21.15.83:0/3874085966 shutdown_connections 2024-03-06T07:22:57.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.357+0000 7faeb2555640 1 --2- 172.21.15.83:0/3874085966 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fae9806b4d0 0x7fae9806d990 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.357+0000 7faeb2555640 1 --2- 172.21.15.83:0/3874085966 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faeac155450 0x7faeac1558d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.357+0000 7faeb2555640 1 --2- 172.21.15.83:0/3874085966 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faeac1e75d0 0x7faeac1e7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.357+0000 7faeb2555640 1 --2- 172.21.15.83:0/3874085966 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae940039b0 0x7faeac142600 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.357+0000 7faeb2555640 1 -- 172.21.15.83:0/3874085966 >> 172.21.15.83:0/3874085966 conn(0x7faeac09a9f0 msgr2=0x7faeac0b2d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:57.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.357+0000 7faeb2555640 1 -- 172.21.15.83:0/3874085966 shutdown_connections 2024-03-06T07:22:57.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.357+0000 7faeb2555640 1 -- 172.21.15.83:0/3874085966 wait complete. 2024-03-06T07:22:57.370 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:57.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.468+0000 7f0d23878640 1 Processor -- start 2024-03-06T07:22:57.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.468+0000 7f0d23878640 1 -- start start 2024-03-06T07:22:57.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.469+0000 7f0d23878640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f0d1c0cd7a0 0x7f0d1c0cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.469+0000 7f0d23878640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d1c0cc980 con 0x7f0d1c1424f0 2024-03-06T07:22:57.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.469+0000 7f0d23878640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d1c0ccc60 con 0x7f0d1c06da40 2024-03-06T07:22:57.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.469+0000 7f0d23878640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d1c0cd220 con 0x7f0d1c0cd7a0 2024-03-06T07:22:57.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.469+0000 7f0d21614640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f0d1c1424f0 0x7f0d1c06d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:50850/0 (socket says 172.21.15.83:50850) 2024-03-06T07:22:57.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.469+0000 7f0d21614640 1 -- 172.21.15.83:0/1389162623 learned_addr learned my addr 172.21.15.83:0/1389162623 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:57.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.469+0000 7f0d13fff640 1 -- 172.21.15.83:0/1389162623 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2223736218 0 0) 0x7f0d1c0cc980 con 0x7f0d1c1424f0 2024-03-06T07:22:57.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.470+0000 7f0d13fff640 1 -- 172.21.15.83:0/1389162623 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d08003520 con 0x7f0d1c1424f0 2024-03-06T07:22:57.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.470+0000 7f0d21e15640 1 --2- 172.21.15.83:0/1389162623 >> v2:172.21.15.183:3300/0 conn(0x7f0d1c0cd7a0 0x7f0d1c0cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.470+0000 7f0d13fff640 1 -- 172.21.15.83:0/1389162623 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2888632031 0 0) 0x7f0d08003520 con 0x7f0d1c1424f0 2024-03-06T07:22:57.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.470+0000 7f0d13fff640 1 -- 172.21.15.83:0/1389162623 >> v2:172.21.15.183:3300/0 conn(0x7f0d1c0cd7a0 msgr2=0x7f0d1c0cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:57.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.470+0000 7f0d13fff640 1 --2- 172.21.15.83:0/1389162623 >> v2:172.21.15.183:3300/0 conn(0x7f0d1c0cd7a0 0x7f0d1c0cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.470+0000 7f0d13fff640 1 -- 172.21.15.83:0/1389162623 >> v1:172.21.15.183:6789/0 conn(0x7f0d1c06da40 legacy=0x7f0d1c071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.470+0000 7f0d13fff640 1 -- 172.21.15.83:0/1389162623 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d1c0ccf40 con 0x7f0d1c1424f0 2024-03-06T07:22:57.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.470+0000 7f0d13fff640 1 -- 172.21.15.83:0/1389162623 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f0d18001670 con 0x7f0d1c1424f0 2024-03-06T07:22:57.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.470+0000 7f0d13fff640 1 -- 172.21.15.83:0/1389162623 >> v1:172.21.15.83:6789/0 conn(0x7f0d1c1424f0 legacy=0x7f0d1c06d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.470+0000 7f0d13fff640 1 --2- 172.21.15.83:0/1389162623 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0d080039b0 0x7f0d08003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.470+0000 7f0d13fff640 1 --2- 172.21.15.83:0/1389162623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0d08004800 0x7f0d08004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.470+0000 7f0d13fff640 1 --2- 172.21.15.83:0/1389162623 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d080051c0 0x7f0d0800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.470+0000 7f0d13fff640 1 -- 172.21.15.83:0/1389162623 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c0cd220 con 0x7f0d08004800 2024-03-06T07:22:57.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.470+0000 7f0d13fff640 1 -- 172.21.15.83:0/1389162623 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0800c060 con 0x7f0d080051c0 2024-03-06T07:22:57.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.470+0000 7f0d13fff640 1 -- 172.21.15.83:0/1389162623 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0800c260 con 0x7f0d080039b0 2024-03-06T07:22:57.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.470+0000 7f0d20e13640 1 --2- 172.21.15.83:0/1389162623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0d08004800 0x7f0d08004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.470+0000 7f0d21e15640 1 --2- 172.21.15.83:0/1389162623 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d080051c0 0x7f0d0800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.470+0000 7f0d21614640 1 --2- 172.21.15.83:0/1389162623 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0d080039b0 0x7f0d08003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.470+0000 7f0d20e13640 1 -- 172.21.15.83:0/1389162623 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0d080039b0 msgr2=0x7f0d08003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:57.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.470+0000 7f0d20e13640 1 --2- 172.21.15.83:0/1389162623 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0d080039b0 0x7f0d08003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.470+0000 7f0d20e13640 1 -- 172.21.15.83:0/1389162623 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d080051c0 msgr2=0x7f0d0800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:57.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.470+0000 7f0d20e13640 1 --2- 172.21.15.83:0/1389162623 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d080051c0 0x7f0d0800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.470+0000 7f0d21e15640 1 -- 172.21.15.83:0/1389162623 reap_dead start 2024-03-06T07:22:57.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.470+0000 7f0d20e13640 1 -- 172.21.15.83:0/1389162623 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d08003520 con 0x7f0d08004800 2024-03-06T07:22:57.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.471+0000 7f0d20e13640 1 --2- 172.21.15.83:0/1389162623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0d08004800 0x7f0d08004c80 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c001750 tx=0x7f0d08005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=248fd3f1ab08811c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:57.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.471+0000 7f0d13fff640 1 -- 172.21.15.83:0/1389162623 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c001370 con 0x7f0d08004800 2024-03-06T07:22:57.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.471+0000 7f0d13fff640 1 -- 172.21.15.83:0/1389162623 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d0c0052d0 con 0x7f0d08004800 2024-03-06T07:22:57.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.471+0000 7f0d13fff640 1 -- 172.21.15.83:0/1389162623 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c003830 con 0x7f0d08004800 2024-03-06T07:22:57.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.472+0000 7f0d23878640 1 -- 172.21.15.83:0/1389162623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0d08004800 msgr2=0x7f0d08004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.472+0000 7f0d23878640 1 --2- 172.21.15.83:0/1389162623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0d08004800 0x7f0d08004c80 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c001750 tx=0x7f0d08005830 comp rx=0 tx=0).stop 2024-03-06T07:22:57.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.472+0000 7f0d23878640 1 -- 172.21.15.83:0/1389162623 shutdown_connections 2024-03-06T07:22:57.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.472+0000 7f0d23878640 1 --2- 172.21.15.83:0/1389162623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0d08004800 0x7f0d08004c80 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.472+0000 7f0d23878640 1 -- 172.21.15.83:0/1389162623 >> 172.21.15.83:0/1389162623 conn(0x7f0d1c0cf8a0 msgr2=0x7f0d1c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:57.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.472+0000 7f0d23878640 1 -- 172.21.15.83:0/1389162623 shutdown_connections 2024-03-06T07:22:57.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.472+0000 7f0d23878640 1 -- 172.21.15.83:0/1389162623 wait complete. 2024-03-06T07:22:57.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.473+0000 7f0d23878640 1 Processor -- start 2024-03-06T07:22:57.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.473+0000 7f0d23878640 1 -- start start 2024-03-06T07:22:57.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.473+0000 7f0d23878640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d08004800 0x7f0d1c1451e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.474+0000 7f0d23878640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0d1c0b3e20 0x7f0d1c1e7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.474+0000 7f0d23878640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0d1c1e7dd0 0x7f0d1c1f7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.474+0000 7f0d23878640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c0cc470 con 0x7f0d1c0b3e20 2024-03-06T07:22:57.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.474+0000 7f0d23878640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c0cc2f0 con 0x7f0d08004800 2024-03-06T07:22:57.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.474+0000 7f0d23878640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c0cc5f0 con 0x7f0d1c1e7dd0 2024-03-06T07:22:57.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.474+0000 7f0d20e13640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0d1c0b3e20 0x7f0d1c1e7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.474+0000 7f0d20e13640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0d1c0b3e20 0x7f0d1c1e7890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35146/0 (socket says 172.21.15.83:35146) 2024-03-06T07:22:57.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.474+0000 7f0d20e13640 1 -- 172.21.15.83:0/3531065180 learned_addr learned my addr 172.21.15.83:0/3531065180 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:57.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.474+0000 7f0d21614640 1 --2- 172.21.15.83:0/3531065180 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d08004800 0x7f0d1c1451e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.474+0000 7f0d20e13640 1 -- 172.21.15.83:0/3531065180 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0d1c1e7dd0 msgr2=0x7f0d1c1f7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.474+0000 7f0d20e13640 1 --2- 172.21.15.83:0/3531065180 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0d1c1e7dd0 0x7f0d1c1f7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.474+0000 7f0d20e13640 1 -- 172.21.15.83:0/3531065180 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d08004800 msgr2=0x7f0d1c1451e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.474+0000 7f0d20e13640 1 --2- 172.21.15.83:0/3531065180 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d08004800 0x7f0d1c1451e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.474+0000 7f0d20e13640 1 -- 172.21.15.83:0/3531065180 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d1c145720 con 0x7f0d1c0b3e20 2024-03-06T07:22:57.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.474+0000 7f0d21614640 1 --2- 172.21.15.83:0/3531065180 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d08004800 0x7f0d1c1451e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:57.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.474+0000 7f0d20e13640 1 --2- 172.21.15.83:0/3531065180 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0d1c0b3e20 0x7f0d1c1e7890 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c000c00 tx=0x7f0d0c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:57.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.474+0000 7f0d11ffb640 1 -- 172.21.15.83:0/3531065180 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c041020 con 0x7f0d1c0b3e20 2024-03-06T07:22:57.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.474+0000 7f0d11ffb640 1 -- 172.21.15.83:0/3531065180 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d0c005480 con 0x7f0d1c0b3e20 2024-03-06T07:22:57.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.474+0000 7f0d23878640 1 -- 172.21.15.83:0/3531065180 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d1c0b2d80 con 0x7f0d1c0b3e20 2024-03-06T07:22:57.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.474+0000 7f0d11ffb640 1 -- 172.21.15.83:0/3531065180 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c007a30 con 0x7f0d1c0b3e20 2024-03-06T07:22:57.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.474+0000 7f0d23878640 1 -- 172.21.15.83:0/3531065180 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d1c0b3340 con 0x7f0d1c0b3e20 2024-03-06T07:22:57.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.475+0000 7f0d23878640 1 -- 172.21.15.83:0/3531065180 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d1c04f330 con 0x7f0d1c0b3e20 2024-03-06T07:22:57.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.476+0000 7f0d11ffb640 1 -- 172.21.15.83:0/3531065180 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0d0c044020 con 0x7f0d1c0b3e20 2024-03-06T07:22:57.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.476+0000 7f0d11ffb640 1 --2- 172.21.15.83:0/3531065180 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0d0006b4d0 0x7f0d0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.476+0000 7f0d11ffb640 1 -- 172.21.15.83:0/3531065180 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0d0c0ab7e0 con 0x7f0d1c0b3e20 2024-03-06T07:22:57.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.476+0000 7f0d21614640 1 --2- 172.21.15.83:0/3531065180 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0d0006b4d0 0x7f0d0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.477+0000 7f0d21614640 1 --2- 172.21.15.83:0/3531065180 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0d0006b4d0 0x7f0d0006d990 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0d18001050 tx=0x7f0d18001d70 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:57.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.480+0000 7f0d11ffb640 1 -- 172.21.15.83:0/3531065180 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0d0c04b030 con 0x7f0d1c0b3e20 2024-03-06T07:22:57.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.651+0000 7f0d23878640 1 -- 172.21.15.83:0/3531065180 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d1c0af810 con 0x7f0d1c0b3e20 2024-03-06T07:22:57.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.652+0000 7f0d11ffb640 1 -- 172.21.15.83:0/3531065180 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+12903 (secure 0 0 0) 0x7f0d1c145720 con 0x7f0d1c0b3e20 2024-03-06T07:22:57.654 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:22:57.654 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":29,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:22:54.780223+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:57.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.654+0000 7f0d23878640 1 -- 172.21.15.83:0/3531065180 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0d0006b4d0 msgr2=0x7f0d0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.654+0000 7f0d23878640 1 --2- 172.21.15.83:0/3531065180 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0d0006b4d0 0x7f0d0006d990 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0d18001050 tx=0x7f0d18001d70 comp rx=0 tx=0).stop 2024-03-06T07:22:57.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.654+0000 7f0d23878640 1 -- 172.21.15.83:0/3531065180 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0d1c0b3e20 msgr2=0x7f0d1c1e7890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.654+0000 7f0d23878640 1 --2- 172.21.15.83:0/3531065180 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0d1c0b3e20 0x7f0d1c1e7890 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c000c00 tx=0x7f0d0c001120 comp rx=0 tx=0).stop 2024-03-06T07:22:57.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.654+0000 7f0d23878640 1 -- 172.21.15.83:0/3531065180 shutdown_connections 2024-03-06T07:22:57.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.654+0000 7f0d23878640 1 --2- 172.21.15.83:0/3531065180 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0d0006b4d0 0x7f0d0006d990 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.654+0000 7f0d23878640 1 --2- 172.21.15.83:0/3531065180 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0d1c1e7dd0 0x7f0d1c1f7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.654+0000 7f0d23878640 1 --2- 172.21.15.83:0/3531065180 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0d1c0b3e20 0x7f0d1c1e7890 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.654+0000 7f0d23878640 1 --2- 172.21.15.83:0/3531065180 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d08004800 0x7f0d1c1451e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.654+0000 7f0d23878640 1 -- 172.21.15.83:0/3531065180 >> 172.21.15.83:0/3531065180 conn(0x7f0d1c0cf8a0 msgr2=0x7f0d1c141820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:57.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.654+0000 7f0d23878640 1 -- 172.21.15.83:0/3531065180 shutdown_connections 2024-03-06T07:22:57.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.654+0000 7f0d23878640 1 -- 172.21.15.83:0/3531065180 wait complete. 2024-03-06T07:22:57.667 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:57.667 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:57.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.768+0000 7f98bc556640 1 Processor -- start 2024-03-06T07:22:57.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.768+0000 7f98bc556640 1 -- start start 2024-03-06T07:22:57.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.769+0000 7f98bc556640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98b40cfa90 0x7f98b40cff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.769+0000 7f98bc556640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98b4158ac0 con 0x7f98b41586e0 2024-03-06T07:22:57.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.769+0000 7f98bc556640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98b415b2a0 con 0x7f98b40b33a0 2024-03-06T07:22:57.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.769+0000 7f98bc556640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98b415b540 con 0x7f98b40cfa90 2024-03-06T07:22:57.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.769+0000 7f98ba2f2640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f98b41586e0 0x7f98b40aea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:50852/0 (socket says 172.21.15.83:50852) 2024-03-06T07:22:57.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.769+0000 7f98ba2f2640 1 -- 172.21.15.83:0/905646667 learned_addr learned my addr 172.21.15.83:0/905646667 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:57.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.769+0000 7f98b9af1640 1 --2- 172.21.15.83:0/905646667 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98b40cfa90 0x7f98b40cff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.769+0000 7f98b92f0640 1 -- 172.21.15.83:0/905646667 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 171769101 0 0) 0x7f98b4158ac0 con 0x7f98b41586e0 2024-03-06T07:22:57.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.769+0000 7f98b92f0640 1 -- 172.21.15.83:0/905646667 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98a0003520 con 0x7f98b41586e0 2024-03-06T07:22:57.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.769+0000 7f98b92f0640 1 -- 172.21.15.83:0/905646667 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2157617905 0 0) 0x7f98a0003520 con 0x7f98b41586e0 2024-03-06T07:22:57.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.769+0000 7f98b92f0640 1 -- 172.21.15.83:0/905646667 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98b40cfa90 msgr2=0x7f98b40cff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:57.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.769+0000 7f98b92f0640 1 --2- 172.21.15.83:0/905646667 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98b40cfa90 0x7f98b40cff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.769+0000 7f98b92f0640 1 -- 172.21.15.83:0/905646667 >> v1:172.21.15.183:6789/0 conn(0x7f98b40b33a0 legacy=0x7f98b40b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.769+0000 7f98b92f0640 1 -- 172.21.15.83:0/905646667 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b415b820 con 0x7f98b41586e0 2024-03-06T07:22:57.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.769+0000 7f98b92f0640 1 -- 172.21.15.83:0/905646667 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f98b0001670 con 0x7f98b41586e0 2024-03-06T07:22:57.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.769+0000 7f98b92f0640 1 -- 172.21.15.83:0/905646667 >> v1:172.21.15.83:6789/0 conn(0x7f98b41586e0 legacy=0x7f98b40aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.769+0000 7f98b92f0640 1 --2- 172.21.15.83:0/905646667 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98b40cfa90 0x7f98a00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.770+0000 7f98b92f0640 1 --2- 172.21.15.83:0/905646667 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98a0003f20 0x7f98a0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.770+0000 7f98b92f0640 1 --2- 172.21.15.83:0/905646667 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98a0004d40 0x7f98a00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.770+0000 7f98b92f0640 1 -- 172.21.15.83:0/905646667 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b415b540 con 0x7f98a0003f20 2024-03-06T07:22:57.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.770+0000 7f98b92f0640 1 -- 172.21.15.83:0/905646667 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a000ba20 con 0x7f98a0004d40 2024-03-06T07:22:57.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.770+0000 7f98b92f0640 1 -- 172.21.15.83:0/905646667 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98a000de30 con 0x7f98b40cfa90 2024-03-06T07:22:57.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.770+0000 7f98b9af1640 1 --2- 172.21.15.83:0/905646667 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98a0003f20 0x7f98a0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.770+0000 7f98ba2f2640 1 --2- 172.21.15.83:0/905646667 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98b40cfa90 0x7f98a00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.770+0000 7f98baaf3640 1 --2- 172.21.15.83:0/905646667 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98a0004d40 0x7f98a00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.770+0000 7f98b9af1640 1 -- 172.21.15.83:0/905646667 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98b40cfa90 msgr2=0x7f98a00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:57.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.770+0000 7f98b9af1640 1 --2- 172.21.15.83:0/905646667 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98b40cfa90 0x7f98a00039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.770+0000 7f98b9af1640 1 -- 172.21.15.83:0/905646667 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98a0004d40 msgr2=0x7f98a00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:57.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.770+0000 7f98b9af1640 1 --2- 172.21.15.83:0/905646667 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98a0004d40 0x7f98a00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.770+0000 7f98b9af1640 1 -- 172.21.15.83:0/905646667 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a0003520 con 0x7f98a0003f20 2024-03-06T07:22:57.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.771+0000 7f98b9af1640 1 --2- 172.21.15.83:0/905646667 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98a0003f20 0x7f98a0004800 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f98a4009ac0 tx=0x7f98a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98625a316097cea9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:57.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.771+0000 7f98b92f0640 1 -- 172.21.15.83:0/905646667 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a4039070 con 0x7f98a0003f20 2024-03-06T07:22:57.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.771+0000 7f98b92f0640 1 -- 172.21.15.83:0/905646667 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98a402c600 con 0x7f98a0003f20 2024-03-06T07:22:57.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.771+0000 7f98b92f0640 1 -- 172.21.15.83:0/905646667 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a4009690 con 0x7f98a0003f20 2024-03-06T07:22:57.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.771+0000 7f98bc556640 1 -- 172.21.15.83:0/905646667 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98a0003f20 msgr2=0x7f98a0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.771+0000 7f98bc556640 1 --2- 172.21.15.83:0/905646667 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98a0003f20 0x7f98a0004800 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f98a4009ac0 tx=0x7f98a0005830 comp rx=0 tx=0).stop 2024-03-06T07:22:57.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.771+0000 7f98baaf3640 1 -- 172.21.15.83:0/905646667 reap_dead start 2024-03-06T07:22:57.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.771+0000 7f98bc556640 1 -- 172.21.15.83:0/905646667 shutdown_connections 2024-03-06T07:22:57.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.771+0000 7f98bc556640 1 -- 172.21.15.83:0/905646667 >> 172.21.15.83:0/905646667 conn(0x7f98b406b190 msgr2=0x7f98b406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:57.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.771+0000 7f98bc556640 1 -- 172.21.15.83:0/905646667 shutdown_connections 2024-03-06T07:22:57.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.771+0000 7f98bc556640 1 -- 172.21.15.83:0/905646667 wait complete. 2024-03-06T07:22:57.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.772+0000 7f98bc556640 1 Processor -- start 2024-03-06T07:22:57.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.773+0000 7f98bc556640 1 -- start start 2024-03-06T07:22:57.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.773+0000 7f98bc556640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98a0003f20 0x7f98b41ef550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.773+0000 7f98bc556640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98b4153e90 0x7f98b40ce080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.773+0000 7f98bc556640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98b406dbd0 0x7f98b4153250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.773+0000 7f98bc556640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b40af150 con 0x7f98a0003f20 2024-03-06T07:22:57.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.773+0000 7f98bc556640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b415bde0 con 0x7f98b406dbd0 2024-03-06T07:22:57.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.773+0000 7f98bc556640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98b4071350 con 0x7f98b4153e90 2024-03-06T07:22:57.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.773+0000 7f98ba2f2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98a0003f20 0x7f98b41ef550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.773+0000 7f98ba2f2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98a0003f20 0x7f98b41ef550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35162/0 (socket says 172.21.15.83:35162) 2024-03-06T07:22:57.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.773+0000 7f98ba2f2640 1 -- 172.21.15.83:0/679381210 learned_addr learned my addr 172.21.15.83:0/679381210 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:57.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.773+0000 7f98b9af1640 1 --2- 172.21.15.83:0/679381210 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98b4153e90 0x7f98b40ce080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.773+0000 7f98baaf3640 1 --2- 172.21.15.83:0/679381210 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98b406dbd0 0x7f98b4153250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.774+0000 7f98ba2f2640 1 -- 172.21.15.83:0/679381210 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98b4153e90 msgr2=0x7f98b40ce080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.774+0000 7f98ba2f2640 1 --2- 172.21.15.83:0/679381210 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98b4153e90 0x7f98b40ce080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.774+0000 7f98ba2f2640 1 -- 172.21.15.83:0/679381210 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98b406dbd0 msgr2=0x7f98b4153250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.774+0000 7f98ba2f2640 1 --2- 172.21.15.83:0/679381210 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98b406dbd0 0x7f98b4153250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.774+0000 7f98ba2f2640 1 -- 172.21.15.83:0/679381210 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b40ce5c0 con 0x7f98a0003f20 2024-03-06T07:22:57.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.774+0000 7f98ba2f2640 1 --2- 172.21.15.83:0/679381210 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98a0003f20 0x7f98b41ef550 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f98b0007290 tx=0x7f98b0007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:57.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.774+0000 7f98aaffd640 1 -- 172.21.15.83:0/679381210 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b001a070 con 0x7f98a0003f20 2024-03-06T07:22:57.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.774+0000 7f98aaffd640 1 -- 172.21.15.83:0/679381210 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98b000f960 con 0x7f98a0003f20 2024-03-06T07:22:57.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.774+0000 7f98bc556640 1 -- 172.21.15.83:0/679381210 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98b41e1210 con 0x7f98a0003f20 2024-03-06T07:22:57.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.774+0000 7f98aaffd640 1 -- 172.21.15.83:0/679381210 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b000fd50 con 0x7f98a0003f20 2024-03-06T07:22:57.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.774+0000 7f98bc556640 1 -- 172.21.15.83:0/679381210 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98b41e14b0 con 0x7f98a0003f20 2024-03-06T07:22:57.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.775+0000 7f98bc556640 1 -- 172.21.15.83:0/679381210 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98b40b2630 con 0x7f98a0003f20 2024-03-06T07:22:57.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.775+0000 7f98aaffd640 1 -- 172.21.15.83:0/679381210 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f98b001d070 con 0x7f98a0003f20 2024-03-06T07:22:57.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.776+0000 7f98aaffd640 1 --2- 172.21.15.83:0/679381210 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f989806b4d0 0x7f989806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.776+0000 7f98aaffd640 1 -- 172.21.15.83:0/679381210 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f98b0089870 con 0x7f98a0003f20 2024-03-06T07:22:57.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.776+0000 7f98b9af1640 1 --2- 172.21.15.83:0/679381210 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f989806b4d0 0x7f989806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.776+0000 7f98b9af1640 1 --2- 172.21.15.83:0/679381210 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f989806b4d0 0x7f989806d990 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f98a4036000 tx=0x7f98a4030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:57.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.780+0000 7f98aaffd640 1 -- 172.21.15.83:0/679381210 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f98b40b2630 con 0x7f98a0003f20 2024-03-06T07:22:57.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.951+0000 7f98bc556640 1 -- 172.21.15.83:0/679381210 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f98b4068ff0 con 0x7f98a0003f20 2024-03-06T07:22:57.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.951+0000 7f98aaffd640 1 -- 172.21.15.83:0/679381210 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7f98b40ce5c0 con 0x7f98a0003f20 2024-03-06T07:22:57.953 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:22:57.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.953+0000 7f98bc556640 1 -- 172.21.15.83:0/679381210 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f989806b4d0 msgr2=0x7f989806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.953+0000 7f98bc556640 1 --2- 172.21.15.83:0/679381210 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f989806b4d0 0x7f989806d990 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f98a4036000 tx=0x7f98a4030000 comp rx=0 tx=0).stop 2024-03-06T07:22:57.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.953+0000 7f98bc556640 1 -- 172.21.15.83:0/679381210 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98a0003f20 msgr2=0x7f98b41ef550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.953+0000 7f98bc556640 1 --2- 172.21.15.83:0/679381210 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98a0003f20 0x7f98b41ef550 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f98b0007290 tx=0x7f98b0007750 comp rx=0 tx=0).stop 2024-03-06T07:22:57.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.953+0000 7f98bc556640 1 -- 172.21.15.83:0/679381210 shutdown_connections 2024-03-06T07:22:57.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.953+0000 7f98bc556640 1 --2- 172.21.15.83:0/679381210 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f989806b4d0 0x7f989806d990 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.953+0000 7f98bc556640 1 --2- 172.21.15.83:0/679381210 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98b406dbd0 0x7f98b4153250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.953+0000 7f98bc556640 1 --2- 172.21.15.83:0/679381210 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98b4153e90 0x7f98b40ce080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.953+0000 7f98bc556640 1 --2- 172.21.15.83:0/679381210 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98a0003f20 0x7f98b41ef550 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.953+0000 7f98bc556640 1 -- 172.21.15.83:0/679381210 >> 172.21.15.83:0/679381210 conn(0x7f98b406b190 msgr2=0x7f98b40ae920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:57.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.953+0000 7f98bc556640 1 -- 172.21.15.83:0/679381210 shutdown_connections 2024-03-06T07:22:57.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:57.953+0000 7f98bc556640 1 -- 172.21.15.83:0/679381210 wait complete. 2024-03-06T07:22:57.966 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:22:58.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.064+0000 7f0854081640 1 Processor -- start 2024-03-06T07:22:58.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.064+0000 7f0854081640 1 -- start start 2024-03-06T07:22:58.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.065+0000 7f0854081640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f084c1424f0 0x7f084c0cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.065+0000 7f0854081640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f084c0cd7a0 0x7f084c15cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.065+0000 7f0854081640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f084c0cc980 con 0x7f084c06d330 2024-03-06T07:22:58.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.065+0000 7f0854081640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f084c0cd220 con 0x7f084c1424f0 2024-03-06T07:22:58.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.065+0000 7f0854081640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f084c0ccc60 con 0x7f084c0cd7a0 2024-03-06T07:22:58.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.065+0000 7f085261e640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f084c06d330 0x7f084c071b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:50868/0 (socket says 172.21.15.83:50868) 2024-03-06T07:22:58.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.065+0000 7f085261e640 1 -- 172.21.15.83:0/4110834735 learned_addr learned my addr 172.21.15.83:0/4110834735 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:58.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.065+0000 7f0850e1b640 1 -- 172.21.15.83:0/4110834735 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1530867952 0 0) 0x7f084c0cc980 con 0x7f084c06d330 2024-03-06T07:22:58.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.065+0000 7f0851e1d640 1 --2- 172.21.15.83:0/4110834735 >> v2:172.21.15.183:3300/0 conn(0x7f084c1424f0 0x7f084c0cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.065+0000 7f0850e1b640 1 -- 172.21.15.83:0/4110834735 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0838003520 con 0x7f084c06d330 2024-03-06T07:22:58.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.065+0000 7f085161c640 1 --2- 172.21.15.83:0/4110834735 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f084c0cd7a0 0x7f084c15cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.065+0000 7f0850e1b640 1 -- 172.21.15.83:0/4110834735 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4009995512 0 0) 0x7f0838003520 con 0x7f084c06d330 2024-03-06T07:22:58.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.065+0000 7f0850e1b640 1 -- 172.21.15.83:0/4110834735 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f084c0cd7a0 msgr2=0x7f084c15cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:58.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.065+0000 7f0850e1b640 1 --2- 172.21.15.83:0/4110834735 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f084c0cd7a0 0x7f084c15cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.065+0000 7f0850e1b640 1 -- 172.21.15.83:0/4110834735 >> v2:172.21.15.183:3300/0 conn(0x7f084c1424f0 msgr2=0x7f084c0cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:58.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.065+0000 7f0850e1b640 1 --2- 172.21.15.83:0/4110834735 >> v2:172.21.15.183:3300/0 conn(0x7f084c1424f0 0x7f084c0cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.065+0000 7f0850e1b640 1 -- 172.21.15.83:0/4110834735 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f084c0ccf40 con 0x7f084c06d330 2024-03-06T07:22:58.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.065+0000 7f0850e1b640 1 -- 172.21.15.83:0/4110834735 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f08440019f0 con 0x7f084c06d330 2024-03-06T07:22:58.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.065+0000 7f0851e1d640 1 --2- 172.21.15.83:0/4110834735 >> v2:172.21.15.183:3300/0 conn(0x7f084c1424f0 0x7f084c0cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:58.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.065+0000 7f0850e1b640 1 -- 172.21.15.83:0/4110834735 >> v1:172.21.15.83:6789/0 conn(0x7f084c06d330 legacy=0x7f084c071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.065+0000 7f0850e1b640 1 --2- 172.21.15.83:0/4110834735 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f08380039b0 0x7f0838003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.065+0000 7f0850e1b640 1 --2- 172.21.15.83:0/4110834735 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0838004800 0x7f0838004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.065+0000 7f0850e1b640 1 --2- 172.21.15.83:0/4110834735 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f084c0cd7a0 0x7f08380051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.065+0000 7f0850e1b640 1 -- 172.21.15.83:0/4110834735 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f084c0cd220 con 0x7f0838004800 2024-03-06T07:22:58.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.065+0000 7f0850e1b640 1 -- 172.21.15.83:0/4110834735 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f084c0ccc60 con 0x7f08380039b0 2024-03-06T07:22:58.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.065+0000 7f0850e1b640 1 -- 172.21.15.83:0/4110834735 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f083800ba70 con 0x7f084c0cd7a0 2024-03-06T07:22:58.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.065+0000 7f085161c640 1 --2- 172.21.15.83:0/4110834735 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0838004800 0x7f0838004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.066+0000 7f0851e1d640 1 --2- 172.21.15.83:0/4110834735 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f08380039b0 0x7f0838003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.066+0000 7f085261e640 1 --2- 172.21.15.83:0/4110834735 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f084c0cd7a0 0x7f08380051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.066+0000 7f085161c640 1 -- 172.21.15.83:0/4110834735 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f084c0cd7a0 msgr2=0x7f08380051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:58.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.066+0000 7f085161c640 1 --2- 172.21.15.83:0/4110834735 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f084c0cd7a0 0x7f08380051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.066+0000 7f085161c640 1 -- 172.21.15.83:0/4110834735 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f08380039b0 msgr2=0x7f0838003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:58.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.066+0000 7f085161c640 1 --2- 172.21.15.83:0/4110834735 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f08380039b0 0x7f0838003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.066+0000 7f085161c640 1 -- 172.21.15.83:0/4110834735 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0838003520 con 0x7f0838004800 2024-03-06T07:22:58.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.066+0000 7f085161c640 1 --2- 172.21.15.83:0/4110834735 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0838004800 0x7f0838004c80 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f083c00ba00 tx=0x7f0838005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4673af6b90360670 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:58.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.067+0000 7f0850e1b640 1 -- 172.21.15.83:0/4110834735 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083c036020 con 0x7f0838004800 2024-03-06T07:22:58.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.067+0000 7f0850e1b640 1 -- 172.21.15.83:0/4110834735 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f083c0325f0 con 0x7f0838004800 2024-03-06T07:22:58.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.067+0000 7f0850e1b640 1 -- 172.21.15.83:0/4110834735 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083c00b600 con 0x7f0838004800 2024-03-06T07:22:58.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.067+0000 7f0854081640 1 -- 172.21.15.83:0/4110834735 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0838004800 msgr2=0x7f0838004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.067+0000 7f0854081640 1 --2- 172.21.15.83:0/4110834735 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0838004800 0x7f0838004c80 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f083c00ba00 tx=0x7f0838005830 comp rx=0 tx=0).stop 2024-03-06T07:22:58.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.067+0000 7f085261e640 1 -- 172.21.15.83:0/4110834735 reap_dead start 2024-03-06T07:22:58.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.067+0000 7f0854081640 1 -- 172.21.15.83:0/4110834735 shutdown_connections 2024-03-06T07:22:58.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.067+0000 7f0854081640 1 -- 172.21.15.83:0/4110834735 >> 172.21.15.83:0/4110834735 conn(0x7f084c0cf8a0 msgr2=0x7f084c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:58.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.067+0000 7f0854081640 1 -- 172.21.15.83:0/4110834735 shutdown_connections 2024-03-06T07:22:58.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.067+0000 7f0854081640 1 -- 172.21.15.83:0/4110834735 wait complete. 2024-03-06T07:22:58.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.068+0000 7f0854081640 1 Processor -- start 2024-03-06T07:22:58.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.068+0000 7f0854081640 1 -- start start 2024-03-06T07:22:58.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.069+0000 7f0854081640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0838004800 0x7f084c13afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.069+0000 7f0854081640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f084c13e700 0x7f084c13caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.069+0000 7f0854081640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f084c13eb60 0x7f084c13cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.069+0000 7f0854081640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f084c15d070 con 0x7f084c13eb60 2024-03-06T07:22:58.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.069+0000 7f0854081640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f084c15cef0 con 0x7f0838004800 2024-03-06T07:22:58.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.069+0000 7f0854081640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f084c15d1f0 con 0x7f084c13e700 2024-03-06T07:22:58.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.069+0000 7f085261e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f084c13eb60 0x7f084c13cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.069+0000 7f085261e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f084c13eb60 0x7f084c13cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35188/0 (socket says 172.21.15.83:35188) 2024-03-06T07:22:58.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.069+0000 7f085261e640 1 -- 172.21.15.83:0/302734605 learned_addr learned my addr 172.21.15.83:0/302734605 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:58.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.069+0000 7f0851e1d640 1 --2- 172.21.15.83:0/302734605 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0838004800 0x7f084c13afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.069+0000 7f085161c640 1 --2- 172.21.15.83:0/302734605 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f084c13e700 0x7f084c13caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.070+0000 7f085261e640 1 -- 172.21.15.83:0/302734605 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f084c13e700 msgr2=0x7f084c13caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.070+0000 7f085261e640 1 --2- 172.21.15.83:0/302734605 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f084c13e700 0x7f084c13caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.070+0000 7f085261e640 1 -- 172.21.15.83:0/302734605 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0838004800 msgr2=0x7f084c13afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.070+0000 7f085261e640 1 --2- 172.21.15.83:0/302734605 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0838004800 0x7f084c13afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.070+0000 7f085261e640 1 -- 172.21.15.83:0/302734605 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f084c13d660 con 0x7f084c13eb60 2024-03-06T07:22:58.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.070+0000 7f085261e640 1 --2- 172.21.15.83:0/302734605 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f084c13eb60 0x7f084c13cfe0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f084c0cd5f0 tx=0x7f0844007b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:58.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.070+0000 7f08427fc640 1 -- 172.21.15.83:0/302734605 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f084400aca0 con 0x7f084c13eb60 2024-03-06T07:22:58.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.070+0000 7f08427fc640 1 -- 172.21.15.83:0/302734605 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0844018070 con 0x7f084c13eb60 2024-03-06T07:22:58.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.070+0000 7f0854081640 1 -- 172.21.15.83:0/302734605 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f084c13d940 con 0x7f084c13eb60 2024-03-06T07:22:58.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.070+0000 7f08427fc640 1 -- 172.21.15.83:0/302734605 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08440135f0 con 0x7f084c13eb60 2024-03-06T07:22:58.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.070+0000 7f0854081640 1 -- 172.21.15.83:0/302734605 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f084c13df00 con 0x7f084c13eb60 2024-03-06T07:22:58.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.071+0000 7f0854081640 1 -- 172.21.15.83:0/302734605 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f084c04f330 con 0x7f084c13eb60 2024-03-06T07:22:58.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.071+0000 7f08427fc640 1 -- 172.21.15.83:0/302734605 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0844013790 con 0x7f084c13eb60 2024-03-06T07:22:58.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.072+0000 7f08427fc640 1 --2- 172.21.15.83:0/302734605 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f083006b500 0x7f083006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.072+0000 7f08427fc640 1 -- 172.21.15.83:0/302734605 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f084408a2c0 con 0x7f084c13eb60 2024-03-06T07:22:58.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.072+0000 7f0851e1d640 1 --2- 172.21.15.83:0/302734605 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f083006b500 0x7f083006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.072+0000 7f0851e1d640 1 --2- 172.21.15.83:0/302734605 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f083006b500 0x7f083006d9c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0848006fd0 tx=0x7f0848008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:58.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.076+0000 7f08427fc640 1 -- 172.21.15.83:0/302734605 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0844058e40 con 0x7f084c13eb60 2024-03-06T07:22:58.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.247+0000 7f0854081640 1 -- 172.21.15.83:0/302734605 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f084c074580 con 0x7f084c13eb60 2024-03-06T07:22:58.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.248+0000 7f08427fc640 1 -- 172.21.15.83:0/302734605 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v29) v1 ==== 84+0+65 (secure 0 0 0) 0x7f084405cee0 con 0x7f084c13eb60 2024-03-06T07:22:58.250 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:22:58.250 INFO:teuthology.orchestra.run.smithi083.stdout:[ 2024-03-06T07:22:58.250 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:22:58.250 INFO:teuthology.orchestra.run.smithi083.stdout: "poolnum": 1, 2024-03-06T07:22:58.250 INFO:teuthology.orchestra.run.smithi083.stdout: "poolname": ".mgr" 2024-03-06T07:22:58.250 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:22:58.250 INFO:teuthology.orchestra.run.smithi083.stdout:] 2024-03-06T07:22:58.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.249+0000 7f0854081640 1 -- 172.21.15.83:0/302734605 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f083006b500 msgr2=0x7f083006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.249+0000 7f0854081640 1 --2- 172.21.15.83:0/302734605 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f083006b500 0x7f083006d9c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0848006fd0 tx=0x7f0848008040 comp rx=0 tx=0).stop 2024-03-06T07:22:58.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.249+0000 7f0854081640 1 -- 172.21.15.83:0/302734605 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f084c13eb60 msgr2=0x7f084c13cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.249+0000 7f0854081640 1 --2- 172.21.15.83:0/302734605 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f084c13eb60 0x7f084c13cfe0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f084c0cd5f0 tx=0x7f0844007b20 comp rx=0 tx=0).stop 2024-03-06T07:22:58.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.249+0000 7f0854081640 1 -- 172.21.15.83:0/302734605 shutdown_connections 2024-03-06T07:22:58.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.249+0000 7f0854081640 1 --2- 172.21.15.83:0/302734605 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f083006b500 0x7f083006d9c0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.249+0000 7f0854081640 1 --2- 172.21.15.83:0/302734605 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f084c13eb60 0x7f084c13cfe0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.249+0000 7f0854081640 1 --2- 172.21.15.83:0/302734605 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f084c13e700 0x7f084c13caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.249+0000 7f0854081640 1 --2- 172.21.15.83:0/302734605 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0838004800 0x7f084c13afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.250+0000 7f0854081640 1 -- 172.21.15.83:0/302734605 >> 172.21.15.83:0/302734605 conn(0x7f084c0cf8a0 msgr2=0x7f084c15ffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:58.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.250+0000 7f0854081640 1 -- 172.21.15.83:0/302734605 shutdown_connections 2024-03-06T07:22:58.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.250+0000 7f0854081640 1 -- 172.21.15.83:0/302734605 wait complete. 2024-03-06T07:22:58.262 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:22:58.262 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:22:58.342 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:22:58.344 DEBUG:teuthology.orchestra.run.smithi183:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:22:58.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.365+0000 7ffb26088640 1 Processor -- start 2024-03-06T07:22:58.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.365+0000 7ffb26088640 1 -- start start 2024-03-06T07:22:58.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.366+0000 7ffb26088640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffb2015f640 0x7ffb2006afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.366+0000 7ffb26088640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ffb200cd140 0x7ffb200cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.366+0000 7ffb26088640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb2015fa20 con 0x7ffb2006b4f0 2024-03-06T07:22:58.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.366+0000 7ffb26088640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb2006b950 con 0x7ffb200cd140 2024-03-06T07:22:58.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.366+0000 7ffb26088640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb2006f380 con 0x7ffb2015f640 2024-03-06T07:22:58.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.366+0000 7ffb1ffff640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ffb200cd140 0x7ffb200cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.366+0000 7ffb1effd640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7ffb2006b4f0 0x7ffb200cca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:35074/0 (socket says 172.21.15.83:35074) 2024-03-06T07:22:58.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.366+0000 7ffb1effd640 1 -- 172.21.15.83:0/2873479695 learned_addr learned my addr 172.21.15.83:0/2873479695 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:22:58.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.366+0000 7ffb1f7fe640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffb2015f640 0x7ffb2006afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.366+0000 7ffb1ffff640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ffb200cd140 0x7ffb200cd560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:37252/0 (socket says 172.21.15.83:37252) 2024-03-06T07:22:58.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.366+0000 7ffb1ffff640 1 -- 172.21.15.83:0/2873479695 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffb2015f640 msgr2=0x7ffb2006afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:58.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.366+0000 7ffb1ffff640 1 --2- 172.21.15.83:0/2873479695 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffb2015f640 0x7ffb2006afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.366+0000 7ffb1ffff640 1 -- 172.21.15.83:0/2873479695 >> v1:172.21.15.183:6789/0 conn(0x7ffb2006b4f0 legacy=0x7ffb200cca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.366+0000 7ffb1ffff640 1 -- 172.21.15.83:0/2873479695 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb200b5fe0 con 0x7ffb200cd140 2024-03-06T07:22:58.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.367+0000 7ffb1ffff640 1 --2- 172.21.15.83:0/2873479695 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffb200cd140 0x7ffb200cd560 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ffb18009d90 tx=0x7ffb1802bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1202763d7737fa67 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:58.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.367+0000 7ffb1e7fc640 1 -- 172.21.15.83:0/2873479695 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb18039070 con 0x7ffb200cd140 2024-03-06T07:22:58.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.367+0000 7ffb1e7fc640 1 -- 172.21.15.83:0/2873479695 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb1802cb30 con 0x7ffb200cd140 2024-03-06T07:22:58.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.367+0000 7ffb1e7fc640 1 -- 172.21.15.83:0/2873479695 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb18034440 con 0x7ffb200cd140 2024-03-06T07:22:58.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.368+0000 7ffb26088640 1 -- 172.21.15.83:0/2873479695 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffb200cd140 msgr2=0x7ffb200cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.368+0000 7ffb26088640 1 --2- 172.21.15.83:0/2873479695 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffb200cd140 0x7ffb200cd560 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ffb18009d90 tx=0x7ffb1802bd80 comp rx=0 tx=0).stop 2024-03-06T07:22:58.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.368+0000 7ffb26088640 1 -- 172.21.15.83:0/2873479695 shutdown_connections 2024-03-06T07:22:58.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.368+0000 7ffb26088640 1 --2- 172.21.15.83:0/2873479695 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffb200cd140 0x7ffb200cd560 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.368+0000 7ffb26088640 1 --2- 172.21.15.83:0/2873479695 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffb2015f640 0x7ffb2006afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.368+0000 7ffb26088640 1 -- 172.21.15.83:0/2873479695 >> 172.21.15.83:0/2873479695 conn(0x7ffb200725f0 msgr2=0x7ffb20072a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:58.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.368+0000 7ffb26088640 1 -- 172.21.15.83:0/2873479695 shutdown_connections 2024-03-06T07:22:58.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.368+0000 7ffb26088640 1 -- 172.21.15.83:0/2873479695 wait complete. 2024-03-06T07:22:58.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.369+0000 7ffb26088640 1 Processor -- start 2024-03-06T07:22:58.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.369+0000 7ffb26088640 1 -- start start 2024-03-06T07:22:58.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.370+0000 7ffb26088640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffb2006b4f0 0x7ffb2014da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.370+0000 7ffb26088640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb200cd140 0x7ffb20159c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.370+0000 7ffb26088640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffb2015f640 0x7ffb2014b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.370+0000 7ffb26088640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb200afa80 con 0x7ffb200cd140 2024-03-06T07:22:58.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.370+0000 7ffb26088640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb20067160 con 0x7ffb2015f640 2024-03-06T07:22:58.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.370+0000 7ffb26088640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb20073320 con 0x7ffb2006b4f0 2024-03-06T07:22:58.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.370+0000 7ffb1effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb200cd140 0x7ffb20159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.370+0000 7ffb1effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb200cd140 0x7ffb20159c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35202/0 (socket says 172.21.15.83:35202) 2024-03-06T07:22:58.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.370+0000 7ffb1effd640 1 -- 172.21.15.83:0/3065062570 learned_addr learned my addr 172.21.15.83:0/3065062570 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:58.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.370+0000 7ffb1ffff640 1 --2- 172.21.15.83:0/3065062570 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffb2015f640 0x7ffb2014b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.370+0000 7ffb1f7fe640 1 --2- 172.21.15.83:0/3065062570 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffb2006b4f0 0x7ffb2014da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.370+0000 7ffb1effd640 1 -- 172.21.15.83:0/3065062570 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffb2006b4f0 msgr2=0x7ffb2014da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.370+0000 7ffb1effd640 1 --2- 172.21.15.83:0/3065062570 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffb2006b4f0 0x7ffb2014da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.371+0000 7ffb1effd640 1 -- 172.21.15.83:0/3065062570 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffb2015f640 msgr2=0x7ffb2014b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.371+0000 7ffb1effd640 1 --2- 172.21.15.83:0/3065062570 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffb2015f640 0x7ffb2014b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.371+0000 7ffb1effd640 1 -- 172.21.15.83:0/3065062570 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb2014c060 con 0x7ffb200cd140 2024-03-06T07:22:58.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.371+0000 7ffb1effd640 1 --2- 172.21.15.83:0/3065062570 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb200cd140 0x7ffb20159c90 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7ffb1400c430 tx=0x7ffb1400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:58.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.371+0000 7ffb07fff640 1 -- 172.21.15.83:0/3065062570 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb14017070 con 0x7ffb200cd140 2024-03-06T07:22:58.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.371+0000 7ffb07fff640 1 -- 172.21.15.83:0/3065062570 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb14005ce0 con 0x7ffb200cd140 2024-03-06T07:22:58.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.371+0000 7ffb26088640 1 -- 172.21.15.83:0/3065062570 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb2014c340 con 0x7ffb200cd140 2024-03-06T07:22:58.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.371+0000 7ffb07fff640 1 -- 172.21.15.83:0/3065062570 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb140072f0 con 0x7ffb200cd140 2024-03-06T07:22:58.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.371+0000 7ffb26088640 1 -- 172.21.15.83:0/3065062570 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb2014c900 con 0x7ffb200cd140 2024-03-06T07:22:58.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.372+0000 7ffb26088640 1 -- 172.21.15.83:0/3065062570 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb2004f330 con 0x7ffb200cd140 2024-03-06T07:22:58.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.372+0000 7ffb07fff640 1 -- 172.21.15.83:0/3065062570 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffb1401a070 con 0x7ffb200cd140 2024-03-06T07:22:58.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.373+0000 7ffb07fff640 1 --2- 172.21.15.83:0/3065062570 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffb0c06b4d0 0x7ffb0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.373+0000 7ffb1f7fe640 1 --2- 172.21.15.83:0/3065062570 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffb0c06b4d0 0x7ffb0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.373+0000 7ffb07fff640 1 -- 172.21.15.83:0/3065062570 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ffb14089210 con 0x7ffb200cd140 2024-03-06T07:22:58.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.373+0000 7ffb1f7fe640 1 --2- 172.21.15.83:0/3065062570 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffb0c06b4d0 0x7ffb0c06d990 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ffb10009a80 tx=0x7ffb10009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:58.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.377+0000 7ffb07fff640 1 -- 172.21.15.83:0/3065062570 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffb14057da0 con 0x7ffb200cd140 2024-03-06T07:22:58.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.549+0000 7ffb26088640 1 -- 172.21.15.83:0/3065062570 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7ffb2015e380 con 0x7ffb200cd140 2024-03-06T07:22:58.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.788+0000 7ffb07fff640 1 -- 172.21.15.83:0/3065062570 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v30) v1 ==== 138+0+0 (secure 0 0 0) 0x7ffb1405be40 con 0x7ffb200cd140 2024-03-06T07:22:58.790 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'cephfs_metadata' created 2024-03-06T07:22:58.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.790+0000 7ffb26088640 1 -- 172.21.15.83:0/3065062570 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffb0c06b4d0 msgr2=0x7ffb0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.790+0000 7ffb26088640 1 --2- 172.21.15.83:0/3065062570 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffb0c06b4d0 0x7ffb0c06d990 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ffb10009a80 tx=0x7ffb10009190 comp rx=0 tx=0).stop 2024-03-06T07:22:58.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.790+0000 7ffb26088640 1 -- 172.21.15.83:0/3065062570 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb200cd140 msgr2=0x7ffb20159c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.790+0000 7ffb26088640 1 --2- 172.21.15.83:0/3065062570 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb200cd140 0x7ffb20159c90 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7ffb1400c430 tx=0x7ffb1400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.790+0000 7ffb26088640 1 -- 172.21.15.83:0/3065062570 shutdown_connections 2024-03-06T07:22:58.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.790+0000 7ffb26088640 1 --2- 172.21.15.83:0/3065062570 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffb0c06b4d0 0x7ffb0c06d990 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.790+0000 7ffb26088640 1 --2- 172.21.15.83:0/3065062570 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffb2015f640 0x7ffb2014b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.790+0000 7ffb26088640 1 --2- 172.21.15.83:0/3065062570 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb200cd140 0x7ffb20159c90 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.790+0000 7ffb26088640 1 --2- 172.21.15.83:0/3065062570 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffb2006b4f0 0x7ffb2014da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.790+0000 7ffb26088640 1 -- 172.21.15.83:0/3065062570 >> 172.21.15.83:0/3065062570 conn(0x7ffb200725f0 msgr2=0x7ffb2006d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:58.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.790+0000 7ffb26088640 1 -- 172.21.15.83:0/3065062570 shutdown_connections 2024-03-06T07:22:58.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.790+0000 7ffb26088640 1 -- 172.21.15.83:0/3065062570 wait complete. 2024-03-06T07:22:58.807 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:22:58.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.912+0000 7fcc567d7640 1 Processor -- start 2024-03-06T07:22:58.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.912+0000 7fcc567d7640 1 -- start start 2024-03-06T07:22:58.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.913+0000 7fcc567d7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcc500cf9a0 0x7fcc500b5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.913+0000 7fcc567d7640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fcc500b55e0 0x7fcc500ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.913+0000 7fcc567d7640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc5006b710 con 0x7fcc50066d90 2024-03-06T07:22:58.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.913+0000 7fcc567d7640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc5006b150 con 0x7fcc500cf9a0 2024-03-06T07:22:58.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.913+0000 7fcc567d7640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc5006b430 con 0x7fcc500b55e0 2024-03-06T07:22:58.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.913+0000 7fcc4ffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcc500cf9a0 0x7fcc500b5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.913+0000 7fcc4ffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcc500cf9a0 0x7fcc500b5ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35208/0 (socket says 172.21.15.83:35208) 2024-03-06T07:22:58.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.913+0000 7fcc4ffff640 1 -- 172.21.15.83:0/3309468896 learned_addr learned my addr 172.21.15.83:0/3309468896 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:58.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.913+0000 7fcc54d74640 1 --2- 172.21.15.83:0/3309468896 >> v2:172.21.15.183:3300/0 conn(0x7fcc500b55e0 0x7fcc500ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.913+0000 7fcc4ffff640 1 -- 172.21.15.83:0/3309468896 >> v2:172.21.15.183:3300/0 conn(0x7fcc500b55e0 msgr2=0x7fcc500ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:58.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.913+0000 7fcc4ffff640 1 --2- 172.21.15.83:0/3309468896 >> v2:172.21.15.183:3300/0 conn(0x7fcc500b55e0 0x7fcc500ce040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.913+0000 7fcc4ffff640 1 -- 172.21.15.83:0/3309468896 >> v1:172.21.15.183:6789/0 conn(0x7fcc50066d90 legacy=0x7fcc500b4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:58.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.913+0000 7fcc4ffff640 1 -- 172.21.15.83:0/3309468896 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc500b0970 con 0x7fcc500cf9a0 2024-03-06T07:22:58.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.914+0000 7fcc4ffff640 1 --2- 172.21.15.83:0/3309468896 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc500cf9a0 0x7fcc500b5ab0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fcc500b3120 tx=0x7fcc4002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a10d646de845c28a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:58.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.914+0000 7fcc4effd640 1 -- 172.21.15.83:0/3309468896 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc40035020 con 0x7fcc500cf9a0 2024-03-06T07:22:58.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.914+0000 7fcc4effd640 1 -- 172.21.15.83:0/3309468896 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc40031870 con 0x7fcc500cf9a0 2024-03-06T07:22:58.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.914+0000 7fcc4effd640 1 -- 172.21.15.83:0/3309468896 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc40031c40 con 0x7fcc500cf9a0 2024-03-06T07:22:58.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.915+0000 7fcc567d7640 1 -- 172.21.15.83:0/3309468896 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc500cf9a0 msgr2=0x7fcc500b5ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.915+0000 7fcc567d7640 1 --2- 172.21.15.83:0/3309468896 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc500cf9a0 0x7fcc500b5ab0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fcc500b3120 tx=0x7fcc4002b910 comp rx=0 tx=0).stop 2024-03-06T07:22:58.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.915+0000 7fcc567d7640 1 -- 172.21.15.83:0/3309468896 shutdown_connections 2024-03-06T07:22:58.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.915+0000 7fcc567d7640 1 --2- 172.21.15.83:0/3309468896 >> v2:172.21.15.183:3300/0 conn(0x7fcc500b55e0 0x7fcc500ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.915+0000 7fcc567d7640 1 --2- 172.21.15.83:0/3309468896 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc500cf9a0 0x7fcc500b5ab0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.915+0000 7fcc567d7640 1 -- 172.21.15.83:0/3309468896 >> 172.21.15.83:0/3309468896 conn(0x7fcc500b1d70 msgr2=0x7fcc500b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:58.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.915+0000 7fcc567d7640 1 -- 172.21.15.83:0/3309468896 shutdown_connections 2024-03-06T07:22:58.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.915+0000 7fcc567d7640 1 -- 172.21.15.83:0/3309468896 wait complete. 2024-03-06T07:22:58.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.916+0000 7fcc567d7640 1 Processor -- start 2024-03-06T07:22:58.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.916+0000 7fcc567d7640 1 -- start start 2024-03-06T07:22:58.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.917+0000 7fcc567d7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcc50066d90 0x7fcc500caac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.917+0000 7fcc567d7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc500b55e0 0x7fcc500d0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.917+0000 7fcc567d7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcc500cf9a0 0x7fcc5015f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.917+0000 7fcc567d7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc500b6170 con 0x7fcc500b55e0 2024-03-06T07:22:58.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.917+0000 7fcc567d7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc500b5ff0 con 0x7fcc50066d90 2024-03-06T07:22:58.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.917+0000 7fcc567d7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc500cfd80 con 0x7fcc500cf9a0 2024-03-06T07:22:58.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.917+0000 7fcc4f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc500b55e0 0x7fcc500d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.917+0000 7fcc4f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc500b55e0 0x7fcc500d0380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35224/0 (socket says 172.21.15.83:35224) 2024-03-06T07:22:58.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.917+0000 7fcc4f7fe640 1 -- 172.21.15.83:0/2216016320 learned_addr learned my addr 172.21.15.83:0/2216016320 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:58.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.917+0000 7fcc4ffff640 1 --2- 172.21.15.83:0/2216016320 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcc50066d90 0x7fcc500caac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.917+0000 7fcc4f7fe640 1 -- 172.21.15.83:0/2216016320 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcc500cf9a0 msgr2=0x7fcc5015f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.917+0000 7fcc4f7fe640 1 --2- 172.21.15.83:0/2216016320 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcc500cf9a0 0x7fcc5015f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.917+0000 7fcc4f7fe640 1 -- 172.21.15.83:0/2216016320 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcc50066d90 msgr2=0x7fcc500caac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.917+0000 7fcc4f7fe640 1 --2- 172.21.15.83:0/2216016320 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcc50066d90 0x7fcc500caac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.917+0000 7fcc4f7fe640 1 -- 172.21.15.83:0/2216016320 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc500cb230 con 0x7fcc500b55e0 2024-03-06T07:22:58.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.917+0000 7fcc4ffff640 1 --2- 172.21.15.83:0/2216016320 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcc50066d90 0x7fcc500caac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:58.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.917+0000 7fcc4f7fe640 1 --2- 172.21.15.83:0/2216016320 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc500b55e0 0x7fcc500d0380 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fcc4400c430 tx=0x7fcc4400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:58.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.917+0000 7fcc4cff9640 1 -- 172.21.15.83:0/2216016320 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc44011790 con 0x7fcc500b55e0 2024-03-06T07:22:58.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.917+0000 7fcc4cff9640 1 -- 172.21.15.83:0/2216016320 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc44017070 con 0x7fcc500b55e0 2024-03-06T07:22:58.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.917+0000 7fcc4cff9640 1 -- 172.21.15.83:0/2216016320 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc44011b70 con 0x7fcc500b55e0 2024-03-06T07:22:58.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.917+0000 7fcc567d7640 1 -- 172.21.15.83:0/2216016320 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc500cb510 con 0x7fcc500b55e0 2024-03-06T07:22:58.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.918+0000 7fcc567d7640 1 -- 172.21.15.83:0/2216016320 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc500cbad0 con 0x7fcc500b55e0 2024-03-06T07:22:58.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.919+0000 7fcc567d7640 1 -- 172.21.15.83:0/2216016320 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc5009e3d0 con 0x7fcc500b55e0 2024-03-06T07:22:58.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.919+0000 7fcc4cff9640 1 -- 172.21.15.83:0/2216016320 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcc44011d10 con 0x7fcc500b55e0 2024-03-06T07:22:58.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.919+0000 7fcc4cff9640 1 --2- 172.21.15.83:0/2216016320 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcc3c06b4d0 0x7fcc3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.919+0000 7fcc4cff9640 1 -- 172.21.15.83:0/2216016320 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fcc44089e70 con 0x7fcc500b55e0 2024-03-06T07:22:58.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.919+0000 7fcc4ffff640 1 --2- 172.21.15.83:0/2216016320 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcc3c06b4d0 0x7fcc3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.920+0000 7fcc4ffff640 1 --2- 172.21.15.83:0/2216016320 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcc3c06b4d0 0x7fcc3c06d990 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fcc40036000 tx=0x7fcc4003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:58.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:58.924+0000 7fcc4cff9640 1 -- 172.21.15.83:0/2216016320 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcc44058ed0 con 0x7fcc500b55e0 2024-03-06T07:22:59.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.098+0000 7fcc567d7640 1 -- 172.21.15.83:0/2216016320 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fcc50068ff0 con 0x7fcc500b55e0 2024-03-06T07:22:59.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.790+0000 7fcc4cff9640 1 -- 172.21.15.83:0/2216016320 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v31) v1 ==== 170+0+0 (secure 0 0 0) 0x7fcc440587d0 con 0x7fcc500b55e0 2024-03-06T07:22:59.792 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'cephfs_data' created 2024-03-06T07:22:59.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.792+0000 7fcc567d7640 1 -- 172.21.15.83:0/2216016320 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcc3c06b4d0 msgr2=0x7fcc3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.792+0000 7fcc567d7640 1 --2- 172.21.15.83:0/2216016320 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcc3c06b4d0 0x7fcc3c06d990 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fcc40036000 tx=0x7fcc4003c040 comp rx=0 tx=0).stop 2024-03-06T07:22:59.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.792+0000 7fcc567d7640 1 -- 172.21.15.83:0/2216016320 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc500b55e0 msgr2=0x7fcc500d0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.792+0000 7fcc567d7640 1 --2- 172.21.15.83:0/2216016320 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc500b55e0 0x7fcc500d0380 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fcc4400c430 tx=0x7fcc4400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.792+0000 7fcc567d7640 1 -- 172.21.15.83:0/2216016320 shutdown_connections 2024-03-06T07:22:59.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.792+0000 7fcc567d7640 1 --2- 172.21.15.83:0/2216016320 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcc3c06b4d0 0x7fcc3c06d990 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.792+0000 7fcc567d7640 1 --2- 172.21.15.83:0/2216016320 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcc500cf9a0 0x7fcc5015f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.792+0000 7fcc567d7640 1 --2- 172.21.15.83:0/2216016320 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc500b55e0 0x7fcc500d0380 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.792+0000 7fcc567d7640 1 --2- 172.21.15.83:0/2216016320 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcc50066d90 0x7fcc500caac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.792+0000 7fcc567d7640 1 -- 172.21.15.83:0/2216016320 >> 172.21.15.83:0/2216016320 conn(0x7fcc500b1d70 msgr2=0x7fcc500af060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:59.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.793+0000 7fcc567d7640 1 -- 172.21.15.83:0/2216016320 shutdown_connections 2024-03-06T07:22:59.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.793+0000 7fcc567d7640 1 -- 172.21.15.83:0/2216016320 wait complete. 2024-03-06T07:22:59.819 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:22:59.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.920+0000 7f8569fa0640 1 Processor -- start 2024-03-06T07:22:59.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.921+0000 7f8569fa0640 1 -- start start 2024-03-06T07:22:59.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.921+0000 7f8569fa0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f8564144240 0x7f85640ceab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.921+0000 7f8569fa0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f85640ceff0 0x7f856406f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.921+0000 7f8569fa0640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f856406fb60 0x7f856406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.921+0000 7f8569fa0640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85640b3960 con 0x7f8564144240 2024-03-06T07:22:59.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.921+0000 7f8569fa0640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f856406ffc0 con 0x7f856406fb60 2024-03-06T07:22:59.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.921+0000 7f8569fa0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85640cfea0 con 0x7f85640ceff0 2024-03-06T07:22:59.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.921+0000 7f8568f9e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f8564144240 0x7f85640ceab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.922+0000 7f8568f9e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f8564144240 0x7f85640ceab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35238/0 (socket says 172.21.15.83:35238) 2024-03-06T07:22:59.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.922+0000 7f8568f9e640 1 -- 172.21.15.83:0/711983858 learned_addr learned my addr 172.21.15.83:0/711983858 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:59.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.922+0000 7f856979f640 1 --2- 172.21.15.83:0/711983858 >> v2:172.21.15.183:3300/0 conn(0x7f856406fb60 0x7f856406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.922+0000 7f8568f9e640 1 -- 172.21.15.83:0/711983858 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f85640ceff0 msgr2=0x7f856406f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:59.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.922+0000 7f8563fff640 1 --2- 172.21.15.83:0/711983858 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f85640ceff0 0x7f856406f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.922+0000 7f8568f9e640 1 --2- 172.21.15.83:0/711983858 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f85640ceff0 0x7f856406f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.922+0000 7f8568f9e640 1 -- 172.21.15.83:0/711983858 >> v2:172.21.15.183:3300/0 conn(0x7f856406fb60 msgr2=0x7f856406dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:59.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.922+0000 7f8568f9e640 1 --2- 172.21.15.83:0/711983858 >> v2:172.21.15.183:3300/0 conn(0x7f856406fb60 0x7f856406dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.922+0000 7f8568f9e640 1 -- 172.21.15.83:0/711983858 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85640cf600 con 0x7f8564144240 2024-03-06T07:22:59.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.923+0000 7f8568f9e640 1 --2- 172.21.15.83:0/711983858 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8564144240 0x7f85640ceab0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f85641554a0 tx=0x7f855802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a020b20a849187fa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:59.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.923+0000 7f85637fe640 1 -- 172.21.15.83:0/711983858 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f855802e070 con 0x7f8564144240 2024-03-06T07:22:59.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.923+0000 7f85637fe640 1 -- 172.21.15.83:0/711983858 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8558037070 con 0x7f8564144240 2024-03-06T07:22:59.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.923+0000 7f85637fe640 1 -- 172.21.15.83:0/711983858 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8558031a00 con 0x7f8564144240 2024-03-06T07:22:59.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.923+0000 7f8569fa0640 1 -- 172.21.15.83:0/711983858 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8564144240 msgr2=0x7f85640ceab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.923+0000 7f8569fa0640 1 --2- 172.21.15.83:0/711983858 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8564144240 0x7f85640ceab0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f85641554a0 tx=0x7f855802b880 comp rx=0 tx=0).stop 2024-03-06T07:22:59.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.923+0000 7f8569fa0640 1 -- 172.21.15.83:0/711983858 shutdown_connections 2024-03-06T07:22:59.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.923+0000 7f8569fa0640 1 --2- 172.21.15.83:0/711983858 >> v2:172.21.15.183:3300/0 conn(0x7f856406fb60 0x7f856406dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.923+0000 7f8569fa0640 1 --2- 172.21.15.83:0/711983858 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f85640ceff0 0x7f856406f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.923+0000 7f8569fa0640 1 --2- 172.21.15.83:0/711983858 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8564144240 0x7f85640ceab0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.923+0000 7f8569fa0640 1 -- 172.21.15.83:0/711983858 >> 172.21.15.83:0/711983858 conn(0x7f85640b2660 msgr2=0x7f85640b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:59.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.923+0000 7f8569fa0640 1 -- 172.21.15.83:0/711983858 shutdown_connections 2024-03-06T07:22:59.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.923+0000 7f8569fa0640 1 -- 172.21.15.83:0/711983858 wait complete. 2024-03-06T07:22:59.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.924+0000 7f8569fa0640 1 Processor -- start 2024-03-06T07:22:59.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.924+0000 7f8569fa0640 1 -- start start 2024-03-06T07:22:59.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.925+0000 7f8569fa0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f856406fb60 0x7f8564159800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.925+0000 7f8569fa0640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f85640ceff0 0x7f85641e73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.925+0000 7f8569fa0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8564144240 0x7f85641e7920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.925+0000 7f8569fa0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f856406d800 con 0x7f856406fb60 2024-03-06T07:22:59.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.925+0000 7f8569fa0640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85640cfbc0 con 0x7f85640ceff0 2024-03-06T07:22:59.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.925+0000 7f8569fa0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8564144620 con 0x7f8564144240 2024-03-06T07:22:59.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.925+0000 7f8568f9e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f856406fb60 0x7f8564159800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.925+0000 7f8568f9e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f856406fb60 0x7f8564159800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35250/0 (socket says 172.21.15.83:35250) 2024-03-06T07:22:59.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.925+0000 7f8568f9e640 1 -- 172.21.15.83:0/4054708670 learned_addr learned my addr 172.21.15.83:0/4054708670 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:22:59.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.926+0000 7f8563fff640 1 --2- 172.21.15.83:0/4054708670 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f85640ceff0 0x7f85641e73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.926+0000 7f8568f9e640 1 -- 172.21.15.83:0/4054708670 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8564144240 msgr2=0x7f85641e7920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.926+0000 7f856979f640 1 --2- 172.21.15.83:0/4054708670 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8564144240 0x7f85641e7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.926+0000 7f8568f9e640 1 --2- 172.21.15.83:0/4054708670 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8564144240 0x7f85641e7920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.926+0000 7f8568f9e640 1 -- 172.21.15.83:0/4054708670 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f85640ceff0 msgr2=0x7f85641e73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.926+0000 7f8568f9e640 1 --2- 172.21.15.83:0/4054708670 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f85640ceff0 0x7f85641e73e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.926+0000 7f8568f9e640 1 -- 172.21.15.83:0/4054708670 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f856415caa0 con 0x7f856406fb60 2024-03-06T07:22:59.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.926+0000 7f8563fff640 1 --2- 172.21.15.83:0/4054708670 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f85640ceff0 0x7f85641e73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:59.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.926+0000 7f8568f9e640 1 --2- 172.21.15.83:0/4054708670 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f856406fb60 0x7f8564159800 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f855802b8c0 tx=0x7f85580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:59.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.926+0000 7f85617fa640 1 -- 172.21.15.83:0/4054708670 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8558032880 con 0x7f856406fb60 2024-03-06T07:22:59.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.926+0000 7f85617fa640 1 -- 172.21.15.83:0/4054708670 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8558032a20 con 0x7f856406fb60 2024-03-06T07:22:59.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.926+0000 7f85617fa640 1 -- 172.21.15.83:0/4054708670 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8558032e10 con 0x7f856406fb60 2024-03-06T07:22:59.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.926+0000 7f8569fa0640 1 -- 172.21.15.83:0/4054708670 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f856412a470 con 0x7f856406fb60 2024-03-06T07:22:59.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.926+0000 7f8569fa0640 1 -- 172.21.15.83:0/4054708670 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f856412aa30 con 0x7f856406fb60 2024-03-06T07:22:59.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.927+0000 7f8569fa0640 1 -- 172.21.15.83:0/4054708670 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8564068f10 con 0x7f856406fb60 2024-03-06T07:22:59.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.927+0000 7f85617fa640 1 -- 172.21.15.83:0/4054708670 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f855804c060 con 0x7f856406fb60 2024-03-06T07:22:59.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.928+0000 7f85617fa640 1 --2- 172.21.15.83:0/4054708670 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f854806b4d0 0x7f854806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.928+0000 7f85617fa640 1 -- 172.21.15.83:0/4054708670 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f85580ac0a0 con 0x7f856406fb60 2024-03-06T07:22:59.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.928+0000 7f8563fff640 1 --2- 172.21.15.83:0/4054708670 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f854806b4d0 0x7f854806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.928+0000 7f8563fff640 1 --2- 172.21.15.83:0/4054708670 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f854806b4d0 0x7f854806d990 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f856412ae30 tx=0x7f8550008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:59.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:22:59.932+0000 7f85617fa640 1 -- 172.21.15.83:0/4054708670 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8558040070 con 0x7f856406fb60 2024-03-06T07:23:00.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.143+0000 7f8569fa0640 1 -- 172.21.15.83:0/4054708670 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f856415dbb0 con 0x7f856406fb60 2024-03-06T07:23:00.814 INFO:teuthology.orchestra.run.smithi083.stderr:new fs with metadata pool 4 and data pool 5 2024-03-06T07:23:00.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.812+0000 7f85617fa640 1 -- 172.21.15.83:0/4054708670 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v11) v1 ==== 172+0+0 (secure 0 0 0) 0x7f855807aef0 con 0x7f856406fb60 2024-03-06T07:23:00.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.827+0000 7f8569fa0640 1 -- 172.21.15.83:0/4054708670 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f854806b4d0 msgr2=0x7f854806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.827+0000 7f8569fa0640 1 --2- 172.21.15.83:0/4054708670 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f854806b4d0 0x7f854806d990 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f856412ae30 tx=0x7f8550008040 comp rx=0 tx=0).stop 2024-03-06T07:23:00.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.827+0000 7f8569fa0640 1 -- 172.21.15.83:0/4054708670 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f856406fb60 msgr2=0x7f8564159800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.827+0000 7f8569fa0640 1 --2- 172.21.15.83:0/4054708670 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f856406fb60 0x7f8564159800 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f855802b8c0 tx=0x7f85580046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.827+0000 7f8569fa0640 1 -- 172.21.15.83:0/4054708670 shutdown_connections 2024-03-06T07:23:00.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.827+0000 7f8569fa0640 1 --2- 172.21.15.83:0/4054708670 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f854806b4d0 0x7f854806d990 secure :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f856412ae30 tx=0x7f8550008040 comp rx=0 tx=0).stop 2024-03-06T07:23:00.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.827+0000 7f8569fa0640 1 --2- 172.21.15.83:0/4054708670 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8564144240 0x7f85641e7920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.827+0000 7f8569fa0640 1 --2- 172.21.15.83:0/4054708670 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f85640ceff0 0x7f85641e73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.827+0000 7f8569fa0640 1 --2- 172.21.15.83:0/4054708670 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f856406fb60 0x7f8564159800 secure :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f855802b8c0 tx=0x7f85580046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.827+0000 7f8569fa0640 1 -- 172.21.15.83:0/4054708670 >> 172.21.15.83:0/4054708670 conn(0x7f85640b2660 msgr2=0x7f856406db70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:00.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.830+0000 7f8569fa0640 1 -- 172.21.15.83:0/4054708670 shutdown_connections 2024-03-06T07:23:00.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.830+0000 7f8569fa0640 1 -- 172.21.15.83:0/4054708670 wait complete. 2024-03-06T07:23:00.846 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:00.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.947+0000 7f5ba9bbc640 1 Processor -- start 2024-03-06T07:23:00.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.947+0000 7f5ba9bbc640 1 -- start start 2024-03-06T07:23:00.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.948+0000 7f5ba9bbc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5ba413dc10 0x7f5ba40b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.948+0000 7f5ba9bbc640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ba415f580 con 0x7f5ba415ef60 2024-03-06T07:23:00.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.948+0000 7f5ba9bbc640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ba415b450 con 0x7f5ba415bf30 2024-03-06T07:23:00.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.948+0000 7f5ba9bbc640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ba40cf3f0 con 0x7f5ba413dc10 2024-03-06T07:23:00.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.948+0000 7f5ba37fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5ba413dc10 0x7f5ba40b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.948+0000 7f5ba37fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5ba413dc10 0x7f5ba40b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35252/0 (socket says 172.21.15.83:35252) 2024-03-06T07:23:00.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.948+0000 7f5ba3fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f5ba415ef60 0x7f5ba40708d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:50876/0 (socket says 172.21.15.83:50876) 2024-03-06T07:23:00.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.948+0000 7f5ba37fe640 1 -- 172.21.15.83:0/2096011713 learned_addr learned my addr 172.21.15.83:0/2096011713 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:00.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.948+0000 7f5ba37fe640 1 -- 172.21.15.83:0/2096011713 >> v1:172.21.15.183:6789/0 conn(0x7f5ba415bf30 legacy=0x7f5ba415e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:00.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.948+0000 7f5ba37fe640 1 -- 172.21.15.83:0/2096011713 >> v1:172.21.15.83:6789/0 conn(0x7f5ba415ef60 legacy=0x7f5ba40708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.948+0000 7f5ba27fc640 1 -- 172.21.15.83:0/2096011713 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 459138314 0 0) 0x7f5ba415f580 con 0x7f5ba415ef60 2024-03-06T07:23:00.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.948+0000 7f5ba37fe640 1 -- 172.21.15.83:0/2096011713 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ba415b730 con 0x7f5ba413dc10 2024-03-06T07:23:00.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.949+0000 7f5ba37fe640 1 --2- 172.21.15.83:0/2096011713 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ba413dc10 0x7f5ba40b2950 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f5b94009870 tx=0x7f5b9402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f99884a231f950a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:00.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.949+0000 7f5ba27fc640 1 -- 172.21.15.83:0/2096011713 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b94035020 con 0x7f5ba413dc10 2024-03-06T07:23:00.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.949+0000 7f5ba27fc640 1 -- 172.21.15.83:0/2096011713 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b94031870 con 0x7f5ba413dc10 2024-03-06T07:23:00.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.949+0000 7f5ba27fc640 1 -- 172.21.15.83:0/2096011713 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b94031c40 con 0x7f5ba413dc10 2024-03-06T07:23:00.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.950+0000 7f5ba9bbc640 1 -- 172.21.15.83:0/2096011713 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ba413dc10 msgr2=0x7f5ba40b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.950+0000 7f5ba9bbc640 1 --2- 172.21.15.83:0/2096011713 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ba413dc10 0x7f5ba40b2950 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f5b94009870 tx=0x7f5b9402b910 comp rx=0 tx=0).stop 2024-03-06T07:23:00.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.950+0000 7f5ba9bbc640 1 -- 172.21.15.83:0/2096011713 shutdown_connections 2024-03-06T07:23:00.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.950+0000 7f5ba9bbc640 1 --2- 172.21.15.83:0/2096011713 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ba413dc10 0x7f5ba40b2950 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.950+0000 7f5ba9bbc640 1 -- 172.21.15.83:0/2096011713 >> 172.21.15.83:0/2096011713 conn(0x7f5ba40cf8a0 msgr2=0x7f5ba40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:00.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.950+0000 7f5ba9bbc640 1 -- 172.21.15.83:0/2096011713 shutdown_connections 2024-03-06T07:23:00.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.950+0000 7f5ba9bbc640 1 -- 172.21.15.83:0/2096011713 wait complete. 2024-03-06T07:23:00.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.951+0000 7f5ba9bbc640 1 Processor -- start 2024-03-06T07:23:00.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.951+0000 7f5ba9bbc640 1 -- start start 2024-03-06T07:23:00.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.952+0000 7f5ba9bbc640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5ba413dc10 0x7f5ba413fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.952+0000 7f5ba9bbc640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ba415bf30 0x7f5ba413ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.952+0000 7f5ba9bbc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ba415ef60 0x7f5ba413bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.952+0000 7f5ba9bbc640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba40b2e90 con 0x7f5ba415ef60 2024-03-06T07:23:00.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.952+0000 7f5ba9bbc640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba415b210 con 0x7f5ba415bf30 2024-03-06T07:23:00.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.952+0000 7f5ba9bbc640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba40b3010 con 0x7f5ba413dc10 2024-03-06T07:23:00.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.952+0000 7f5ba3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ba415ef60 0x7f5ba413bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.952+0000 7f5ba3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ba415ef60 0x7f5ba413bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35266/0 (socket says 172.21.15.83:35266) 2024-03-06T07:23:00.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.952+0000 7f5ba3fff640 1 -- 172.21.15.83:0/2464138712 learned_addr learned my addr 172.21.15.83:0/2464138712 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:00.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.952+0000 7f5ba37fe640 1 --2- 172.21.15.83:0/2464138712 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5ba413dc10 0x7f5ba413fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.952+0000 7f5ba2ffd640 1 --2- 172.21.15.83:0/2464138712 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ba415bf30 0x7f5ba413ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.952+0000 7f5ba3fff640 1 -- 172.21.15.83:0/2464138712 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5ba413dc10 msgr2=0x7f5ba413fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.952+0000 7f5ba3fff640 1 --2- 172.21.15.83:0/2464138712 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5ba413dc10 0x7f5ba413fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.952+0000 7f5ba3fff640 1 -- 172.21.15.83:0/2464138712 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ba415bf30 msgr2=0x7f5ba413ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.952+0000 7f5ba3fff640 1 --2- 172.21.15.83:0/2464138712 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ba415bf30 0x7f5ba413ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.952+0000 7f5ba3fff640 1 -- 172.21.15.83:0/2464138712 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ba41415d0 con 0x7f5ba415ef60 2024-03-06T07:23:00.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.952+0000 7f5ba3fff640 1 --2- 172.21.15.83:0/2464138712 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ba415ef60 0x7f5ba413bff0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c00c730 tx=0x7f5b9c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:00.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.952+0000 7f5b9bfff640 1 -- 172.21.15.83:0/2464138712 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c018070 con 0x7f5ba415ef60 2024-03-06T07:23:00.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.952+0000 7f5b9bfff640 1 -- 172.21.15.83:0/2464138712 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b9c00ce40 con 0x7f5ba415ef60 2024-03-06T07:23:00.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.952+0000 7f5b9bfff640 1 -- 172.21.15.83:0/2464138712 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c00a860 con 0x7f5ba415ef60 2024-03-06T07:23:00.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.952+0000 7f5ba9bbc640 1 -- 172.21.15.83:0/2464138712 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ba4141870 con 0x7f5ba415ef60 2024-03-06T07:23:00.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.952+0000 7f5ba9bbc640 1 -- 172.21.15.83:0/2464138712 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ba4141e30 con 0x7f5ba415ef60 2024-03-06T07:23:00.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.953+0000 7f5ba9bbc640 1 -- 172.21.15.83:0/2464138712 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ba404f330 con 0x7f5ba415ef60 2024-03-06T07:23:00.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.954+0000 7f5b9bfff640 1 -- 172.21.15.83:0/2464138712 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5b9c01b070 con 0x7f5ba415ef60 2024-03-06T07:23:00.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.954+0000 7f5b9bfff640 1 --2- 172.21.15.83:0/2464138712 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5b8c06b4d0 0x7f5b8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.954+0000 7f5b9bfff640 1 -- 172.21.15.83:0/2464138712 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5b9c0896d0 con 0x7f5ba415ef60 2024-03-06T07:23:00.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.954+0000 7f5ba37fe640 1 --2- 172.21.15.83:0/2464138712 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5b8c06b4d0 0x7f5b8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.955+0000 7f5ba37fe640 1 --2- 172.21.15.83:0/2464138712 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5b8c06b4d0 0x7f5b8c06d990 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5b94036000 tx=0x7f5b9403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:00.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:00.959+0000 7f5b9bfff640 1 -- 172.21.15.83:0/2464138712 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5b9c0585e0 con 0x7f5ba415ef60 2024-03-06T07:23:01.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.129+0000 7f5ba9bbc640 1 -- 172.21.15.83:0/2464138712 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ba406eb00 con 0x7f5ba415ef60 2024-03-06T07:23:01.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.131+0000 7f5b9bfff640 1 -- 172.21.15.83:0/2464138712 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+16694 (secure 0 0 0) 0x7f5b9c057ee0 con 0x7f5ba415ef60 2024-03-06T07:23:01.133 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:23:01.133 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":32,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:23:00.145014+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:58.568071+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:59.116376+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:01.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.132+0000 7f5ba9bbc640 1 -- 172.21.15.83:0/2464138712 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5b8c06b4d0 msgr2=0x7f5b8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.132+0000 7f5ba9bbc640 1 --2- 172.21.15.83:0/2464138712 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5b8c06b4d0 0x7f5b8c06d990 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5b94036000 tx=0x7f5b9403c040 comp rx=0 tx=0).stop 2024-03-06T07:23:01.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.132+0000 7f5ba9bbc640 1 -- 172.21.15.83:0/2464138712 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ba415ef60 msgr2=0x7f5ba413bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.132+0000 7f5ba9bbc640 1 --2- 172.21.15.83:0/2464138712 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ba415ef60 0x7f5ba413bff0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c00c730 tx=0x7f5b9c00cbf0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.133+0000 7f5ba9bbc640 1 -- 172.21.15.83:0/2464138712 shutdown_connections 2024-03-06T07:23:01.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.133+0000 7f5ba9bbc640 1 --2- 172.21.15.83:0/2464138712 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5b8c06b4d0 0x7f5b8c06d990 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.133+0000 7f5ba9bbc640 1 --2- 172.21.15.83:0/2464138712 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ba415ef60 0x7f5ba413bff0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.133+0000 7f5ba9bbc640 1 --2- 172.21.15.83:0/2464138712 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ba415bf30 0x7f5ba413ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.133+0000 7f5ba9bbc640 1 --2- 172.21.15.83:0/2464138712 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5ba413dc10 0x7f5ba413fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.133+0000 7f5ba9bbc640 1 -- 172.21.15.83:0/2464138712 >> 172.21.15.83:0/2464138712 conn(0x7f5ba40cf8a0 msgr2=0x7f5ba40af5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:01.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.133+0000 7f5ba9bbc640 1 -- 172.21.15.83:0/2464138712 shutdown_connections 2024-03-06T07:23:01.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.133+0000 7f5ba9bbc640 1 -- 172.21.15.83:0/2464138712 wait complete. 2024-03-06T07:23:01.146 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:01.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.244+0000 7f81c909a640 1 Processor -- start 2024-03-06T07:23:01.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.244+0000 7f81c909a640 1 -- start start 2024-03-06T07:23:01.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.245+0000 7f81c909a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f81c40cf850 0x7f81c40cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.245+0000 7f81c909a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f81c406cf20 0x7f81c406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.245+0000 7f81c909a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81c415e380 con 0x7f81c40d00c0 2024-03-06T07:23:01.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.245+0000 7f81c909a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81c406ca90 con 0x7f81c406cf20 2024-03-06T07:23:01.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.245+0000 7f81c909a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81c40cdc10 con 0x7f81c40cf850 2024-03-06T07:23:01.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.245+0000 7f81c359e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f81c406cf20 0x7f81c406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.245+0000 7f81c259c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f81c40cf850 0x7f81c40cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.245+0000 7f81c359e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f81c406cf20 0x7f81c406d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:37322/0 (socket says 172.21.15.83:37322) 2024-03-06T07:23:01.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.245+0000 7f81c359e640 1 -- 172.21.15.83:0/2575040015 learned_addr learned my addr 172.21.15.83:0/2575040015 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:01.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.246+0000 7f81c1d9b640 1 -- 172.21.15.83:0/2575040015 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1478181568 0 0) 0x7f81c415e380 con 0x7f81c40d00c0 2024-03-06T07:23:01.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.246+0000 7f81c1d9b640 1 -- 172.21.15.83:0/2575040015 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81ac003520 con 0x7f81c40d00c0 2024-03-06T07:23:01.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.246+0000 7f81c259c640 1 -- 172.21.15.83:0/2575040015 >> v2:172.21.15.183:3300/0 conn(0x7f81c406cf20 msgr2=0x7f81c406d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:01.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.246+0000 7f81c259c640 1 --2- 172.21.15.83:0/2575040015 >> v2:172.21.15.183:3300/0 conn(0x7f81c406cf20 0x7f81c406d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.246+0000 7f81c259c640 1 -- 172.21.15.83:0/2575040015 >> v1:172.21.15.183:6789/0 conn(0x7f81c40d00c0 legacy=0x7f81c40d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.246+0000 7f81c259c640 1 -- 172.21.15.83:0/2575040015 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81c406c160 con 0x7f81c40cf850 2024-03-06T07:23:01.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.246+0000 7f81c359e640 1 --2- 172.21.15.83:0/2575040015 >> v2:172.21.15.183:3300/0 conn(0x7f81c406cf20 0x7f81c406d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:01.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.246+0000 7f81c259c640 1 --2- 172.21.15.83:0/2575040015 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f81c40cf850 0x7f81c40cd3c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f81b80097a0 tx=0x7f81b802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6753c223bacd86a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:01.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.247+0000 7f81c1d9b640 1 -- 172.21.15.83:0/2575040015 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b8035020 con 0x7f81c40cf850 2024-03-06T07:23:01.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.247+0000 7f81c1d9b640 1 -- 172.21.15.83:0/2575040015 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81b8031830 con 0x7f81c40cf850 2024-03-06T07:23:01.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.247+0000 7f81c1d9b640 1 -- 172.21.15.83:0/2575040015 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b8031be0 con 0x7f81c40cf850 2024-03-06T07:23:01.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.247+0000 7f81c909a640 1 -- 172.21.15.83:0/2575040015 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f81c40cf850 msgr2=0x7f81c40cd3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.247+0000 7f81c909a640 1 --2- 172.21.15.83:0/2575040015 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f81c40cf850 0x7f81c40cd3c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f81b80097a0 tx=0x7f81b802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.247+0000 7f81c909a640 1 -- 172.21.15.83:0/2575040015 shutdown_connections 2024-03-06T07:23:01.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.247+0000 7f81c909a640 1 --2- 172.21.15.83:0/2575040015 >> v2:172.21.15.183:3300/0 conn(0x7f81c406cf20 0x7f81c406d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.247+0000 7f81c909a640 1 --2- 172.21.15.83:0/2575040015 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f81c40cf850 0x7f81c40cd3c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.247+0000 7f81c909a640 1 -- 172.21.15.83:0/2575040015 >> 172.21.15.83:0/2575040015 conn(0x7f81c40b35a0 msgr2=0x7f81c40b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:01.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.247+0000 7f81c909a640 1 -- 172.21.15.83:0/2575040015 shutdown_connections 2024-03-06T07:23:01.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.247+0000 7f81c909a640 1 -- 172.21.15.83:0/2575040015 wait complete. 2024-03-06T07:23:01.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.249+0000 7f81c909a640 1 Processor -- start 2024-03-06T07:23:01.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.249+0000 7f81c909a640 1 -- start start 2024-03-06T07:23:01.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.249+0000 7f81c909a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f81c406cf20 0x7f81c4158550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.249+0000 7f81c909a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f81c40cf850 0x7f81c40c67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.249+0000 7f81c909a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f81c40d00c0 0x7f81c40b67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.249+0000 7f81c909a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c40cd900 con 0x7f81c40d00c0 2024-03-06T07:23:01.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.249+0000 7f81c909a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c406c720 con 0x7f81c406cf20 2024-03-06T07:23:01.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.249+0000 7f81c909a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81c40b1030 con 0x7f81c40cf850 2024-03-06T07:23:01.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.249+0000 7f81c359e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f81c40d00c0 0x7f81c40b67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.250+0000 7f81c359e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f81c40d00c0 0x7f81c40b67f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35274/0 (socket says 172.21.15.83:35274) 2024-03-06T07:23:01.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.250+0000 7f81c359e640 1 -- 172.21.15.83:0/1199583223 learned_addr learned my addr 172.21.15.83:0/1199583223 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:01.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.250+0000 7f81c2d9d640 1 --2- 172.21.15.83:0/1199583223 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f81c406cf20 0x7f81c4158550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.250+0000 7f81c259c640 1 --2- 172.21.15.83:0/1199583223 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f81c40cf850 0x7f81c40c67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.250+0000 7f81c359e640 1 -- 172.21.15.83:0/1199583223 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f81c40cf850 msgr2=0x7f81c40c67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.250+0000 7f81c359e640 1 --2- 172.21.15.83:0/1199583223 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f81c40cf850 0x7f81c40c67e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.250+0000 7f81c359e640 1 -- 172.21.15.83:0/1199583223 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f81c406cf20 msgr2=0x7f81c4158550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.250+0000 7f81c359e640 1 --2- 172.21.15.83:0/1199583223 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f81c406cf20 0x7f81c4158550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.250+0000 7f81c359e640 1 -- 172.21.15.83:0/1199583223 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81c40b5860 con 0x7f81c40d00c0 2024-03-06T07:23:01.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.250+0000 7f81c359e640 1 --2- 172.21.15.83:0/1199583223 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f81c40d00c0 0x7f81c40b67f0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f81bc00efc0 tx=0x7f81bc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:01.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.250+0000 7f81ab7fe640 1 -- 172.21.15.83:0/1199583223 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81bc00cbe0 con 0x7f81c40d00c0 2024-03-06T07:23:01.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.250+0000 7f81ab7fe640 1 -- 172.21.15.83:0/1199583223 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81bc00cd80 con 0x7f81c40d00c0 2024-03-06T07:23:01.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.250+0000 7f81c909a640 1 -- 172.21.15.83:0/1199583223 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81c40c8d90 con 0x7f81c40d00c0 2024-03-06T07:23:01.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.250+0000 7f81ab7fe640 1 -- 172.21.15.83:0/1199583223 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81bc003bb0 con 0x7f81c40d00c0 2024-03-06T07:23:01.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.250+0000 7f81c909a640 1 -- 172.21.15.83:0/1199583223 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81c40c9350 con 0x7f81c40d00c0 2024-03-06T07:23:01.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.251+0000 7f81c909a640 1 -- 172.21.15.83:0/1199583223 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81c404f330 con 0x7f81c40d00c0 2024-03-06T07:23:01.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.252+0000 7f81ab7fe640 1 -- 172.21.15.83:0/1199583223 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f81bc01b030 con 0x7f81c40d00c0 2024-03-06T07:23:01.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.252+0000 7f81ab7fe640 1 --2- 172.21.15.83:0/1199583223 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f81b006b4d0 0x7f81b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.252+0000 7f81ab7fe640 1 -- 172.21.15.83:0/1199583223 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f81bc08a490 con 0x7f81c40d00c0 2024-03-06T07:23:01.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.252+0000 7f81c2d9d640 1 --2- 172.21.15.83:0/1199583223 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f81b006b4d0 0x7f81b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.252+0000 7f81c2d9d640 1 --2- 172.21.15.83:0/1199583223 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f81b006b4d0 0x7f81b006d990 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f81b4007fd0 tx=0x7f81b4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:01.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.256+0000 7f81ab7fe640 1 -- 172.21.15.83:0/1199583223 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f81bc022020 con 0x7f81c40d00c0 2024-03-06T07:23:01.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.429+0000 7f81c909a640 1 -- 172.21.15.83:0/1199583223 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f81c40b3060 con 0x7f81c40d00c0 2024-03-06T07:23:01.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.430+0000 7f81ab7fe640 1 -- 172.21.15.83:0/1199583223 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+16694 (secure 0 0 0) 0x7f81bc010040 con 0x7f81c40d00c0 2024-03-06T07:23:01.432 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:23:01.432 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":32,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:23:00.145014+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:58.568071+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:59.116376+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:01.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.432+0000 7f81c909a640 1 -- 172.21.15.83:0/1199583223 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f81b006b4d0 msgr2=0x7f81b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.432+0000 7f81c909a640 1 --2- 172.21.15.83:0/1199583223 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f81b006b4d0 0x7f81b006d990 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f81b4007fd0 tx=0x7f81b4009040 comp rx=0 tx=0).stop 2024-03-06T07:23:01.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.432+0000 7f81c909a640 1 -- 172.21.15.83:0/1199583223 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f81c40d00c0 msgr2=0x7f81c40b67f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.432+0000 7f81c909a640 1 --2- 172.21.15.83:0/1199583223 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f81c40d00c0 0x7f81c40b67f0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f81bc00efc0 tx=0x7f81bc00c490 comp rx=0 tx=0).stop 2024-03-06T07:23:01.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.432+0000 7f81c909a640 1 -- 172.21.15.83:0/1199583223 shutdown_connections 2024-03-06T07:23:01.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.432+0000 7f81c909a640 1 --2- 172.21.15.83:0/1199583223 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f81b006b4d0 0x7f81b006d990 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.432+0000 7f81c909a640 1 --2- 172.21.15.83:0/1199583223 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f81c40d00c0 0x7f81c40b67f0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.432+0000 7f81c909a640 1 --2- 172.21.15.83:0/1199583223 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f81c40cf850 0x7f81c40c67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.432+0000 7f81c909a640 1 --2- 172.21.15.83:0/1199583223 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f81c406cf20 0x7f81c4158550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.432+0000 7f81c909a640 1 -- 172.21.15.83:0/1199583223 >> 172.21.15.83:0/1199583223 conn(0x7f81c40b35a0 msgr2=0x7f81c40b0820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:01.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.432+0000 7f81c909a640 1 -- 172.21.15.83:0/1199583223 shutdown_connections 2024-03-06T07:23:01.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.432+0000 7f81c909a640 1 -- 172.21.15.83:0/1199583223 wait complete. 2024-03-06T07:23:01.445 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:23:01.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.546+0000 7f73544fb640 1 Processor -- start 2024-03-06T07:23:01.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.546+0000 7f73544fb640 1 -- start start 2024-03-06T07:23:01.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.547+0000 7f73544fb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f734c1614d0 0x7f734c0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.547+0000 7f73544fb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f734c1618b0 con 0x7f734c0d1910 2024-03-06T07:23:01.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.547+0000 7f73544fb640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f734c0cf400 con 0x7f734c0d2480 2024-03-06T07:23:01.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.547+0000 7f73544fb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f734c0cf6a0 con 0x7f734c1614d0 2024-03-06T07:23:01.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.547+0000 7f7351a96640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f734c0d1910 0x7f734c0d1d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:50886/0 (socket says 172.21.15.83:50886) 2024-03-06T07:23:01.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.547+0000 7f7351a96640 1 -- 172.21.15.83:0/431392172 learned_addr learned my addr 172.21.15.83:0/431392172 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:23:01.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.548+0000 7f7352297640 1 --2- 172.21.15.83:0/431392172 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f734c1614d0 0x7f734c0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.548+0000 7f7351295640 1 -- 172.21.15.83:0/431392172 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2477044426 0 0) 0x7f734c1618b0 con 0x7f734c0d1910 2024-03-06T07:23:01.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.548+0000 7f7351295640 1 -- 172.21.15.83:0/431392172 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7338003520 con 0x7f734c0d1910 2024-03-06T07:23:01.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.548+0000 7f7351295640 1 -- 172.21.15.83:0/431392172 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1003498931 0 0) 0x7f7338003520 con 0x7f734c0d1910 2024-03-06T07:23:01.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.548+0000 7f7352297640 1 -- 172.21.15.83:0/431392172 >> v1:172.21.15.183:6789/0 conn(0x7f734c0d2480 legacy=0x7f734c0b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.548+0000 7f7352297640 1 -- 172.21.15.83:0/431392172 >> v1:172.21.15.83:6789/0 conn(0x7f734c0d1910 legacy=0x7f734c0d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.548+0000 7f7352297640 1 -- 172.21.15.83:0/431392172 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f734c0cf980 con 0x7f734c1614d0 2024-03-06T07:23:01.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.549+0000 7f7352297640 1 --2- 172.21.15.83:0/431392172 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f734c1614d0 0x7f734c0d4860 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7348009dc0 tx=0x7f734802bc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d41b9ee8725052c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:01.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.549+0000 7f7351295640 1 -- 172.21.15.83:0/431392172 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7348039070 con 0x7f734c1614d0 2024-03-06T07:23:01.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.549+0000 7f7351295640 1 -- 172.21.15.83:0/431392172 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f734802c770 con 0x7f734c1614d0 2024-03-06T07:23:01.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.549+0000 7f7351295640 1 -- 172.21.15.83:0/431392172 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f733c001670 con 0x7f734c1614d0 2024-03-06T07:23:01.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.549+0000 7f73544fb640 1 -- 172.21.15.83:0/431392172 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f734c1614d0 msgr2=0x7f734c0d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.549+0000 7f73544fb640 1 --2- 172.21.15.83:0/431392172 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f734c1614d0 0x7f734c0d4860 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7348009dc0 tx=0x7f734802bc40 comp rx=0 tx=0).stop 2024-03-06T07:23:01.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.549+0000 7f73544fb640 1 -- 172.21.15.83:0/431392172 shutdown_connections 2024-03-06T07:23:01.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.549+0000 7f73544fb640 1 --2- 172.21.15.83:0/431392172 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f734c1614d0 0x7f734c0d4860 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.549+0000 7f73544fb640 1 -- 172.21.15.83:0/431392172 >> 172.21.15.83:0/431392172 conn(0x7f734c0d3d40 msgr2=0x7f734c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:01.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.550+0000 7f73544fb640 1 -- 172.21.15.83:0/431392172 shutdown_connections 2024-03-06T07:23:01.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.550+0000 7f73544fb640 1 -- 172.21.15.83:0/431392172 wait complete. 2024-03-06T07:23:01.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.551+0000 7f73544fb640 1 Processor -- start 2024-03-06T07:23:01.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.551+0000 7f73544fb640 1 -- start start 2024-03-06T07:23:01.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.551+0000 7f73544fb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f734c0d1910 0x7f734c0b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.551+0000 7f73544fb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f734c0d2480 0x7f734c15d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.551+0000 7f73544fb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f734c1614d0 0x7f734c0cfea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.551+0000 7f73544fb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f734c0b1540 con 0x7f734c0d1910 2024-03-06T07:23:01.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.551+0000 7f73544fb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f734c0b0910 con 0x7f734c1614d0 2024-03-06T07:23:01.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.551+0000 7f73544fb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f734c0b16c0 con 0x7f734c0d2480 2024-03-06T07:23:01.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.551+0000 7f7352297640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f734c0d1910 0x7f734c0b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.552+0000 7f7352297640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f734c0d1910 0x7f734c0b68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35280/0 (socket says 172.21.15.83:35280) 2024-03-06T07:23:01.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.552+0000 7f7352297640 1 -- 172.21.15.83:0/3923963595 learned_addr learned my addr 172.21.15.83:0/3923963595 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:01.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.552+0000 7f7351a96640 1 --2- 172.21.15.83:0/3923963595 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f734c0d2480 0x7f734c15d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.552+0000 7f7352a98640 1 --2- 172.21.15.83:0/3923963595 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f734c1614d0 0x7f734c0cfea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.552+0000 7f7352297640 1 -- 172.21.15.83:0/3923963595 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f734c0d2480 msgr2=0x7f734c15d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.552+0000 7f7352297640 1 --2- 172.21.15.83:0/3923963595 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f734c0d2480 0x7f734c15d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.552+0000 7f7352297640 1 -- 172.21.15.83:0/3923963595 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f734c1614d0 msgr2=0x7f734c0cfea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.552+0000 7f7352297640 1 --2- 172.21.15.83:0/3923963595 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f734c1614d0 0x7f734c0cfea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.552+0000 7f7352297640 1 -- 172.21.15.83:0/3923963595 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f734c15de00 con 0x7f734c0d1910 2024-03-06T07:23:01.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.552+0000 7f7352a98640 1 --2- 172.21.15.83:0/3923963595 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f734c1614d0 0x7f734c0cfea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:01.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.552+0000 7f7351a96640 1 --2- 172.21.15.83:0/3923963595 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f734c0d2480 0x7f734c15d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:01.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.552+0000 7f7352297640 1 --2- 172.21.15.83:0/3923963595 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f734c0d1910 0x7f734c0b68d0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f7348009ee0 tx=0x7f7348004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:01.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.552+0000 7f7342ffd640 1 -- 172.21.15.83:0/3923963595 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7348044020 con 0x7f734c0d1910 2024-03-06T07:23:01.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.552+0000 7f7342ffd640 1 -- 172.21.15.83:0/3923963595 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7348034a90 con 0x7f734c0d1910 2024-03-06T07:23:01.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.552+0000 7f73544fb640 1 -- 172.21.15.83:0/3923963595 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f734c0b7660 con 0x7f734c0d1910 2024-03-06T07:23:01.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.552+0000 7f7342ffd640 1 -- 172.21.15.83:0/3923963595 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7348039040 con 0x7f734c0d1910 2024-03-06T07:23:01.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.552+0000 7f73544fb640 1 -- 172.21.15.83:0/3923963595 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f734c0b7c20 con 0x7f734c0d1910 2024-03-06T07:23:01.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.553+0000 7f73544fb640 1 -- 172.21.15.83:0/3923963595 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f734c0cc360 con 0x7f734c0d1910 2024-03-06T07:23:01.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.554+0000 7f7342ffd640 1 -- 172.21.15.83:0/3923963595 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7348047020 con 0x7f734c0d1910 2024-03-06T07:23:01.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.554+0000 7f7342ffd640 1 --2- 172.21.15.83:0/3923963595 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f733006b4d0 0x7f733006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.554+0000 7f7342ffd640 1 -- 172.21.15.83:0/3923963595 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f73480ac840 con 0x7f734c0d1910 2024-03-06T07:23:01.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.554+0000 7f7351a96640 1 --2- 172.21.15.83:0/3923963595 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f733006b4d0 0x7f733006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.554+0000 7f7351a96640 1 --2- 172.21.15.83:0/3923963595 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f733006b4d0 0x7f733006d990 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f733c001eb0 tx=0x7f733c001d80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:01.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.558+0000 7f7342ffd640 1 -- 172.21.15.83:0/3923963595 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f734804e030 con 0x7f734c0d1910 2024-03-06T07:23:01.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.766+0000 7f73544fb640 1 -- 172.21.15.83:0/3923963595 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f734c068ff0 con 0x7f734c0d1910 2024-03-06T07:23:01.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.812+0000 7f7342ffd640 1 -- 172.21.15.83:0/3923963595 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v12) v1 ==== 118+0+0 (secure 0 0 0) 0x7f734c15de00 con 0x7f734c0d1910 2024-03-06T07:23:01.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.815+0000 7f73544fb640 1 -- 172.21.15.83:0/3923963595 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f733006b4d0 msgr2=0x7f733006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.815+0000 7f73544fb640 1 --2- 172.21.15.83:0/3923963595 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f733006b4d0 0x7f733006d990 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f733c001eb0 tx=0x7f733c001d80 comp rx=0 tx=0).stop 2024-03-06T07:23:01.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.815+0000 7f73544fb640 1 -- 172.21.15.83:0/3923963595 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f734c0d1910 msgr2=0x7f734c0b68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.815+0000 7f73544fb640 1 --2- 172.21.15.83:0/3923963595 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f734c0d1910 0x7f734c0b68d0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f7348009ee0 tx=0x7f7348004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:01.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.815+0000 7f73544fb640 1 -- 172.21.15.83:0/3923963595 shutdown_connections 2024-03-06T07:23:01.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.815+0000 7f73544fb640 1 --2- 172.21.15.83:0/3923963595 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f733006b4d0 0x7f733006d990 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.815+0000 7f73544fb640 1 --2- 172.21.15.83:0/3923963595 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f734c1614d0 0x7f734c0cfea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.815+0000 7f73544fb640 1 --2- 172.21.15.83:0/3923963595 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f734c0d2480 0x7f734c15d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.815+0000 7f73544fb640 1 --2- 172.21.15.83:0/3923963595 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f734c0d1910 0x7f734c0b68d0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.815+0000 7f73544fb640 1 -- 172.21.15.83:0/3923963595 >> 172.21.15.83:0/3923963595 conn(0x7f734c0d3d40 msgr2=0x7f734c06c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:01.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.815+0000 7f73544fb640 1 -- 172.21.15.83:0/3923963595 shutdown_connections 2024-03-06T07:23:01.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.815+0000 7f73544fb640 1 -- 172.21.15.83:0/3923963595 wait complete. 2024-03-06T07:23:01.828 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:23:01.829 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:23:01.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.931+0000 7f386e793640 1 Processor -- start 2024-03-06T07:23:01.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.931+0000 7f386e793640 1 -- start start 2024-03-06T07:23:01.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.932+0000 7f386e793640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f386815ad90 0x7f38680b3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.932+0000 7f386e793640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f386815b260 con 0x7f3868071a70 2024-03-06T07:23:01.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.932+0000 7f386e793640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f386815b540 con 0x7f38680b4110 2024-03-06T07:23:01.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.932+0000 7f386e793640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f386815c020 con 0x7f386815ad90 2024-03-06T07:23:01.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.932+0000 7f3867fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f3868071a70 0x7f3868071e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:50888/0 (socket says 172.21.15.83:50888) 2024-03-06T07:23:01.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.932+0000 7f3867fff640 1 -- 172.21.15.83:0/1625611890 learned_addr learned my addr 172.21.15.83:0/1625611890 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:23:01.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.933+0000 7f3866ffd640 1 -- 172.21.15.83:0/1625611890 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3935935875 0 0) 0x7f386815b260 con 0x7f3868071a70 2024-03-06T07:23:01.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.933+0000 7f3866ffd640 1 -- 172.21.15.83:0/1625611890 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3850003520 con 0x7f3868071a70 2024-03-06T07:23:01.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.933+0000 7f38677fe640 1 --2- 172.21.15.83:0/1625611890 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f386815ad90 0x7f38680b3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.933+0000 7f3866ffd640 1 -- 172.21.15.83:0/1625611890 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2537468166 0 0) 0x7f3850003520 con 0x7f3868071a70 2024-03-06T07:23:01.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.933+0000 7f3866ffd640 1 -- 172.21.15.83:0/1625611890 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f386815ad90 msgr2=0x7f38680b3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:01.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.933+0000 7f3866ffd640 1 --2- 172.21.15.83:0/1625611890 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f386815ad90 0x7f38680b3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.933+0000 7f3866ffd640 1 -- 172.21.15.83:0/1625611890 >> v1:172.21.15.183:6789/0 conn(0x7f38680b4110 legacy=0x7f386815c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.933+0000 7f3866ffd640 1 -- 172.21.15.83:0/1625611890 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f386815b820 con 0x7f3868071a70 2024-03-06T07:23:01.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.933+0000 7f3866ffd640 1 -- 172.21.15.83:0/1625611890 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f3858001670 con 0x7f3868071a70 2024-03-06T07:23:01.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.933+0000 7f3866ffd640 1 -- 172.21.15.83:0/1625611890 >> v1:172.21.15.83:6789/0 conn(0x7f3868071a70 legacy=0x7f3868071e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.933+0000 7f3866ffd640 1 --2- 172.21.15.83:0/1625611890 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f386815ad90 0x7f38500039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.933+0000 7f3866ffd640 1 --2- 172.21.15.83:0/1625611890 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3850004060 0x7f3850004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.933+0000 7f3866ffd640 1 --2- 172.21.15.83:0/1625611890 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3850004d40 0x7f38500051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.933+0000 7f3866ffd640 1 -- 172.21.15.83:0/1625611890 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f386815c020 con 0x7f3850004d40 2024-03-06T07:23:01.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.933+0000 7f3866ffd640 1 -- 172.21.15.83:0/1625611890 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f385000bca0 con 0x7f3850004060 2024-03-06T07:23:01.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.933+0000 7f3866ffd640 1 -- 172.21.15.83:0/1625611890 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f385000e0b0 con 0x7f386815ad90 2024-03-06T07:23:01.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.933+0000 7f386cd30640 1 --2- 172.21.15.83:0/1625611890 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3850004d40 0x7f38500051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.933+0000 7f3867fff640 1 --2- 172.21.15.83:0/1625611890 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f386815ad90 0x7f38500039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.933+0000 7f38677fe640 1 --2- 172.21.15.83:0/1625611890 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3850004060 0x7f3850004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.933+0000 7f386cd30640 1 -- 172.21.15.83:0/1625611890 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f386815ad90 msgr2=0x7f38500039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:01.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.933+0000 7f386cd30640 1 --2- 172.21.15.83:0/1625611890 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f386815ad90 0x7f38500039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.933+0000 7f386cd30640 1 -- 172.21.15.83:0/1625611890 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3850004060 msgr2=0x7f3850004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:01.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.933+0000 7f386cd30640 1 --2- 172.21.15.83:0/1625611890 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3850004060 0x7f3850004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.933+0000 7f386cd30640 1 -- 172.21.15.83:0/1625611890 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3850003520 con 0x7f3850004d40 2024-03-06T07:23:01.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.934+0000 7f386cd30640 1 --2- 172.21.15.83:0/1625611890 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3850004d40 0x7f38500051c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f3860001e50 tx=0x7f3850005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43b40bbdb40ea952 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:01.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.934+0000 7f3866ffd640 1 -- 172.21.15.83:0/1625611890 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3860001a70 con 0x7f3850004d40 2024-03-06T07:23:01.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.934+0000 7f3866ffd640 1 -- 172.21.15.83:0/1625611890 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3860003b70 con 0x7f3850004d40 2024-03-06T07:23:01.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.934+0000 7f3866ffd640 1 -- 172.21.15.83:0/1625611890 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38600050d0 con 0x7f3850004d40 2024-03-06T07:23:01.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.935+0000 7f386e793640 1 -- 172.21.15.83:0/1625611890 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3850004d40 msgr2=0x7f38500051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.935+0000 7f386e793640 1 --2- 172.21.15.83:0/1625611890 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3850004d40 0x7f38500051c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f3860001e50 tx=0x7f3850005830 comp rx=0 tx=0).stop 2024-03-06T07:23:01.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.935+0000 7f386cd30640 1 -- 172.21.15.83:0/1625611890 reap_dead start 2024-03-06T07:23:01.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.935+0000 7f386e793640 1 -- 172.21.15.83:0/1625611890 shutdown_connections 2024-03-06T07:23:01.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.935+0000 7f386e793640 1 -- 172.21.15.83:0/1625611890 >> 172.21.15.83:0/1625611890 conn(0x7f38680b2660 msgr2=0x7f38680b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:01.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.935+0000 7f386e793640 1 -- 172.21.15.83:0/1625611890 shutdown_connections 2024-03-06T07:23:01.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.935+0000 7f386e793640 1 -- 172.21.15.83:0/1625611890 wait complete. 2024-03-06T07:23:01.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.936+0000 7f386e793640 1 Processor -- start 2024-03-06T07:23:01.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.936+0000 7f386e793640 1 -- start start 2024-03-06T07:23:01.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.937+0000 7f386e793640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3850004d40 0x7f386806e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.937+0000 7f386e793640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f38680aebe0 0x7f38680af060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.937+0000 7f386e793640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f38681f3f10 0x7f38681f4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.937+0000 7f386e793640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38680b46e0 con 0x7f38681f3f10 2024-03-06T07:23:01.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.937+0000 7f386e793640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38680b4560 con 0x7f3850004d40 2024-03-06T07:23:01.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.937+0000 7f386e793640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38680b3bd0 con 0x7f38680aebe0 2024-03-06T07:23:01.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.937+0000 7f386cd30640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f38681f3f10 0x7f38681f4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.937+0000 7f386cd30640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f38681f3f10 0x7f38681f4390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35290/0 (socket says 172.21.15.83:35290) 2024-03-06T07:23:01.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.937+0000 7f386cd30640 1 -- 172.21.15.83:0/706634655 learned_addr learned my addr 172.21.15.83:0/706634655 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:01.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.937+0000 7f3867fff640 1 --2- 172.21.15.83:0/706634655 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3850004d40 0x7f386806e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.937+0000 7f38677fe640 1 --2- 172.21.15.83:0/706634655 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f38680aebe0 0x7f38680af060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.937+0000 7f386cd30640 1 -- 172.21.15.83:0/706634655 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f38680aebe0 msgr2=0x7f38680af060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.937+0000 7f386cd30640 1 --2- 172.21.15.83:0/706634655 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f38680aebe0 0x7f38680af060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.937+0000 7f386cd30640 1 -- 172.21.15.83:0/706634655 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3850004d40 msgr2=0x7f386806e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.937+0000 7f386cd30640 1 --2- 172.21.15.83:0/706634655 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3850004d40 0x7f386806e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.937+0000 7f386cd30640 1 -- 172.21.15.83:0/706634655 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38680af5a0 con 0x7f38681f3f10 2024-03-06T07:23:01.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.937+0000 7f386cd30640 1 --2- 172.21.15.83:0/706634655 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f38681f3f10 0x7f38681f4390 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f3860001f70 tx=0x7f38600014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:01.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.938+0000 7f3864ff9640 1 -- 172.21.15.83:0/706634655 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3860007db0 con 0x7f38681f3f10 2024-03-06T07:23:01.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.938+0000 7f3864ff9640 1 -- 172.21.15.83:0/706634655 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3860003970 con 0x7f38681f3f10 2024-03-06T07:23:01.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.938+0000 7f386e793640 1 -- 172.21.15.83:0/706634655 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38681f4d30 con 0x7f38681f3f10 2024-03-06T07:23:01.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.938+0000 7f3864ff9640 1 -- 172.21.15.83:0/706634655 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3860006d70 con 0x7f38681f3f10 2024-03-06T07:23:01.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.938+0000 7f386e793640 1 -- 172.21.15.83:0/706634655 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38681f7a90 con 0x7f38681f3f10 2024-03-06T07:23:01.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.939+0000 7f386e793640 1 -- 172.21.15.83:0/706634655 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3868073390 con 0x7f38681f3f10 2024-03-06T07:23:01.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.939+0000 7f3864ff9640 1 -- 172.21.15.83:0/706634655 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3860041030 con 0x7f38681f3f10 2024-03-06T07:23:01.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.940+0000 7f3864ff9640 1 --2- 172.21.15.83:0/706634655 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f385406b4d0 0x7f385406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.940+0000 7f3867fff640 1 --2- 172.21.15.83:0/706634655 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f385406b4d0 0x7f385406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.940+0000 7f3864ff9640 1 -- 172.21.15.83:0/706634655 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f38600ab130 con 0x7f38681f3f10 2024-03-06T07:23:01.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.940+0000 7f3867fff640 1 --2- 172.21.15.83:0/706634655 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f385406b4d0 0x7f385406d990 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3858001050 tx=0x7f3858000f80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:01.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:01.944+0000 7f3864ff9640 1 -- 172.21.15.83:0/706634655 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3860033020 con 0x7f38681f3f10 2024-03-06T07:23:02.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.318+0000 7f386e793640 1 -- 172.21.15.83:0/706634655 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f38680d07d0 con 0x7f38681f3f10 2024-03-06T07:23:02.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.822+0000 7f3864ff9640 1 -- 172.21.15.83:0/706634655 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v13) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3860003b40 con 0x7f38681f3f10 2024-03-06T07:23:02.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.824+0000 7f386e793640 1 -- 172.21.15.83:0/706634655 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f385406b4d0 msgr2=0x7f385406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.824+0000 7f386e793640 1 --2- 172.21.15.83:0/706634655 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f385406b4d0 0x7f385406d990 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3858001050 tx=0x7f3858000f80 comp rx=0 tx=0).stop 2024-03-06T07:23:02.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.824+0000 7f386e793640 1 -- 172.21.15.83:0/706634655 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f38681f3f10 msgr2=0x7f38681f4390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.824+0000 7f386e793640 1 --2- 172.21.15.83:0/706634655 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f38681f3f10 0x7f38681f4390 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f3860001f70 tx=0x7f38600014a0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.825+0000 7f386e793640 1 -- 172.21.15.83:0/706634655 shutdown_connections 2024-03-06T07:23:02.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.825+0000 7f386e793640 1 --2- 172.21.15.83:0/706634655 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f385406b4d0 0x7f385406d990 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.825+0000 7f386e793640 1 --2- 172.21.15.83:0/706634655 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f38681f3f10 0x7f38681f4390 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.825+0000 7f386e793640 1 --2- 172.21.15.83:0/706634655 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f38680aebe0 0x7f38680af060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.825+0000 7f386e793640 1 --2- 172.21.15.83:0/706634655 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3850004d40 0x7f386806e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.825+0000 7f386e793640 1 -- 172.21.15.83:0/706634655 >> 172.21.15.83:0/706634655 conn(0x7f38680b2660 msgr2=0x7f38680b07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:02.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.825+0000 7f386e793640 1 -- 172.21.15.83:0/706634655 shutdown_connections 2024-03-06T07:23:02.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.825+0000 7f386e793640 1 -- 172.21.15.83:0/706634655 wait complete. 2024-03-06T07:23:02.841 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:23:02.841 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:02.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.941+0000 7f9be553e640 1 Processor -- start 2024-03-06T07:23:02.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.941+0000 7f9be553e640 1 -- start start 2024-03-06T07:23:02.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.942+0000 7f9be553e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9be00d22b0 0x7f9be00d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.942+0000 7f9be553e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f9be00b41f0 0x7f9be00b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.942+0000 7f9be553e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9be00b3430 con 0x7f9be015dfe0 2024-03-06T07:23:02.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.942+0000 7f9be553e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9be00d2c50 con 0x7f9be00b41f0 2024-03-06T07:23:02.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.942+0000 7f9be553e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9be00b31e0 con 0x7f9be00d22b0 2024-03-06T07:23:02.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.942+0000 7f9bdeffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f9be015dfe0 0x7f9be00cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:50900/0 (socket says 172.21.15.83:50900) 2024-03-06T07:23:02.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.942+0000 7f9bdeffd640 1 -- 172.21.15.83:0/1242735974 learned_addr learned my addr 172.21.15.83:0/1242735974 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:23:02.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.942+0000 7f9bde7fc640 1 -- 172.21.15.83:0/1242735974 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1035713904 0 0) 0x7f9be00b3430 con 0x7f9be015dfe0 2024-03-06T07:23:02.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.942+0000 7f9bde7fc640 1 -- 172.21.15.83:0/1242735974 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bcc003520 con 0x7f9be015dfe0 2024-03-06T07:23:02.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.942+0000 7f9bdf7fe640 1 --2- 172.21.15.83:0/1242735974 >> v2:172.21.15.183:3300/0 conn(0x7f9be00b41f0 0x7f9be00b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.942+0000 7f9bd7fff640 1 --2- 172.21.15.83:0/1242735974 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9be00d22b0 0x7f9be00d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.942+0000 7f9bde7fc640 1 -- 172.21.15.83:0/1242735974 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 88331640 0 0) 0x7f9bcc003520 con 0x7f9be015dfe0 2024-03-06T07:23:02.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.942+0000 7f9bde7fc640 1 -- 172.21.15.83:0/1242735974 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9be00d22b0 msgr2=0x7f9be00d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:02.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.942+0000 7f9bde7fc640 1 --2- 172.21.15.83:0/1242735974 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9be00d22b0 0x7f9be00d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.942+0000 7f9bde7fc640 1 -- 172.21.15.83:0/1242735974 >> v2:172.21.15.183:3300/0 conn(0x7f9be00b41f0 msgr2=0x7f9be00b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:02.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.942+0000 7f9bde7fc640 1 --2- 172.21.15.83:0/1242735974 >> v2:172.21.15.183:3300/0 conn(0x7f9be00b41f0 0x7f9be00b4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.942+0000 7f9bde7fc640 1 -- 172.21.15.83:0/1242735974 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be00b3710 con 0x7f9be015dfe0 2024-03-06T07:23:02.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.942+0000 7f9bde7fc640 1 -- 172.21.15.83:0/1242735974 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f9bd0001670 con 0x7f9be015dfe0 2024-03-06T07:23:02.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.942+0000 7f9bde7fc640 1 -- 172.21.15.83:0/1242735974 >> v1:172.21.15.83:6789/0 conn(0x7f9be015dfe0 legacy=0x7f9be00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.942+0000 7f9bde7fc640 1 --2- 172.21.15.83:0/1242735974 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9be00d22b0 0x7f9bcc0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.942+0000 7f9bde7fc640 1 --2- 172.21.15.83:0/1242735974 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9bcc003f30 0x7f9bcc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.942+0000 7f9bde7fc640 1 --2- 172.21.15.83:0/1242735974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9bcc004d40 0x7f9bcc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.942+0000 7f9bde7fc640 1 -- 172.21.15.83:0/1242735974 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be00b31e0 con 0x7f9bcc004d40 2024-03-06T07:23:02.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.942+0000 7f9bde7fc640 1 -- 172.21.15.83:0/1242735974 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bcc00b9c0 con 0x7f9bcc003f30 2024-03-06T07:23:02.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.942+0000 7f9bde7fc640 1 -- 172.21.15.83:0/1242735974 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bcc00bb40 con 0x7f9be00d22b0 2024-03-06T07:23:02.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.943+0000 7f9bdf7fe640 1 --2- 172.21.15.83:0/1242735974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9bcc004d40 0x7f9bcc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.943+0000 7f9bdeffd640 1 --2- 172.21.15.83:0/1242735974 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9be00d22b0 0x7f9bcc0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.943+0000 7f9bd7fff640 1 --2- 172.21.15.83:0/1242735974 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9bcc003f30 0x7f9bcc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.943+0000 7f9bdf7fe640 1 -- 172.21.15.83:0/1242735974 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9be00d22b0 msgr2=0x7f9bcc0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:02.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.943+0000 7f9bdf7fe640 1 --2- 172.21.15.83:0/1242735974 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9be00d22b0 0x7f9bcc0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.943+0000 7f9bdf7fe640 1 -- 172.21.15.83:0/1242735974 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9bcc003f30 msgr2=0x7f9bcc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:02.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.943+0000 7f9bdf7fe640 1 --2- 172.21.15.83:0/1242735974 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9bcc003f30 0x7f9bcc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.943+0000 7f9bdf7fe640 1 -- 172.21.15.83:0/1242735974 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bcc003520 con 0x7f9bcc004d40 2024-03-06T07:23:02.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.943+0000 7f9bdf7fe640 1 --2- 172.21.15.83:0/1242735974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9bcc004d40 0x7f9bcc0051c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f9bd8009fc0 tx=0x7f9bcc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=934eab10d649888f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:02.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.944+0000 7f9bde7fc640 1 -- 172.21.15.83:0/1242735974 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd8039070 con 0x7f9bcc004d40 2024-03-06T07:23:02.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.944+0000 7f9bde7fc640 1 -- 172.21.15.83:0/1242735974 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bd802cc40 con 0x7f9bcc004d40 2024-03-06T07:23:02.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.944+0000 7f9bde7fc640 1 -- 172.21.15.83:0/1242735974 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd8009c10 con 0x7f9bcc004d40 2024-03-06T07:23:02.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.944+0000 7f9be553e640 1 -- 172.21.15.83:0/1242735974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9bcc004d40 msgr2=0x7f9bcc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.944+0000 7f9be553e640 1 --2- 172.21.15.83:0/1242735974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9bcc004d40 0x7f9bcc0051c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f9bd8009fc0 tx=0x7f9bcc005830 comp rx=0 tx=0).stop 2024-03-06T07:23:02.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.944+0000 7f9bdf7fe640 1 -- 172.21.15.83:0/1242735974 reap_dead start 2024-03-06T07:23:02.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.944+0000 7f9be553e640 1 -- 172.21.15.83:0/1242735974 shutdown_connections 2024-03-06T07:23:02.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.944+0000 7f9be553e640 1 -- 172.21.15.83:0/1242735974 >> 172.21.15.83:0/1242735974 conn(0x7f9be00d3cd0 msgr2=0x7f9be00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:02.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.944+0000 7f9be553e640 1 -- 172.21.15.83:0/1242735974 shutdown_connections 2024-03-06T07:23:02.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.944+0000 7f9be553e640 1 -- 172.21.15.83:0/1242735974 wait complete. 2024-03-06T07:23:02.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.945+0000 7f9be553e640 1 Processor -- start 2024-03-06T07:23:02.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.945+0000 7f9be553e640 1 -- start start 2024-03-06T07:23:02.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.946+0000 7f9be553e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9bcc004d40 0x7f9be00cc6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.946+0000 7f9be553e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9be01f44c0 0x7f9be00a15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.946+0000 7f9be553e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9be01f4c00 0x7f9be00b0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.946+0000 7f9be553e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be00cfa00 con 0x7f9bcc004d40 2024-03-06T07:23:02.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.946+0000 7f9be553e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be00b3d60 con 0x7f9be01f4c00 2024-03-06T07:23:02.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.946+0000 7f9be553e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9be00d1c70 con 0x7f9be01f44c0 2024-03-06T07:23:02.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.946+0000 7f9bdeffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9bcc004d40 0x7f9be00cc6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.946+0000 7f9bdeffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9bcc004d40 0x7f9be00cc6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35318/0 (socket says 172.21.15.83:35318) 2024-03-06T07:23:02.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.946+0000 7f9bdeffd640 1 -- 172.21.15.83:0/3277957255 learned_addr learned my addr 172.21.15.83:0/3277957255 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:02.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.946+0000 7f9bd7fff640 1 --2- 172.21.15.83:0/3277957255 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9be01f44c0 0x7f9be00a15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.946+0000 7f9bdf7fe640 1 --2- 172.21.15.83:0/3277957255 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9be01f4c00 0x7f9be00b0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.947+0000 7f9bdeffd640 1 -- 172.21.15.83:0/3277957255 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9be01f44c0 msgr2=0x7f9be00a15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.947+0000 7f9bdeffd640 1 --2- 172.21.15.83:0/3277957255 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9be01f44c0 0x7f9be00a15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.947+0000 7f9bdeffd640 1 -- 172.21.15.83:0/3277957255 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9be01f4c00 msgr2=0x7f9be00b0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.947+0000 7f9bdeffd640 1 --2- 172.21.15.83:0/3277957255 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9be01f4c00 0x7f9be00b0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.947+0000 7f9bdeffd640 1 -- 172.21.15.83:0/3277957255 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be00a1b20 con 0x7f9bcc004d40 2024-03-06T07:23:02.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.947+0000 7f9bdeffd640 1 --2- 172.21.15.83:0/3277957255 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9bcc004d40 0x7f9be00cc6a0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0007340 tx=0x7f9bd0007740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:02.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.947+0000 7f9bd77fe640 1 -- 172.21.15.83:0/3277957255 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd000f940 con 0x7f9bcc004d40 2024-03-06T07:23:02.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.947+0000 7f9bd77fe640 1 -- 172.21.15.83:0/3277957255 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bd0018070 con 0x7f9bcc004d40 2024-03-06T07:23:02.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.947+0000 7f9be553e640 1 -- 172.21.15.83:0/3277957255 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9be00ccc80 con 0x7f9bcc004d40 2024-03-06T07:23:02.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.947+0000 7f9bd77fe640 1 -- 172.21.15.83:0/3277957255 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd000fd40 con 0x7f9bcc004d40 2024-03-06T07:23:02.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.947+0000 7f9be553e640 1 -- 172.21.15.83:0/3277957255 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9be00cb1b0 con 0x7f9bcc004d40 2024-03-06T07:23:02.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.948+0000 7f9be553e640 1 -- 172.21.15.83:0/3277957255 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9be004f330 con 0x7f9bcc004d40 2024-03-06T07:23:02.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.949+0000 7f9bd77fe640 1 -- 172.21.15.83:0/3277957255 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9bd001d460 con 0x7f9bcc004d40 2024-03-06T07:23:02.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.949+0000 7f9bd77fe640 1 --2- 172.21.15.83:0/3277957255 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9bc006b4d0 0x7f9bc006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.949+0000 7f9bd77fe640 1 -- 172.21.15.83:0/3277957255 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9bd008a1b0 con 0x7f9bcc004d40 2024-03-06T07:23:02.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.949+0000 7f9bd7fff640 1 --2- 172.21.15.83:0/3277957255 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9bc006b4d0 0x7f9bc006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.950+0000 7f9bd7fff640 1 --2- 172.21.15.83:0/3277957255 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9bc006b4d0 0x7f9bc006d990 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9be00d3180 tx=0x7f9bc8009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:02.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:02.957+0000 7f9bd77fe640 1 -- 172.21.15.83:0/3277957255 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9bd00590c0 con 0x7f9bcc004d40 2024-03-06T07:23:03.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.174+0000 7f9be553e640 1 -- 172.21.15.83:0/3277957255 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9be00b1b40 con 0x7f9bcc004d40 2024-03-06T07:23:03.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.174+0000 7f9bd77fe640 1 -- 172.21.15.83:0/3277957255 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f9bd00589c0 con 0x7f9bcc004d40 2024-03-06T07:23:03.177 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:23:03.177 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4799,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/3542419057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3542419057},{"type":"v1","addr":"172.21.15.83:6836","nonce":3542419057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":5023,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1853869906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":1853869906},{"type":"v1","addr":"172.21.15.183:6835","nonce":1853869906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5026,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/3666667479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3666667479},{"type":"v1","addr":"172.21.15.183:6834","nonce":3666667479}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:00.145321+0000","modified":"2024-03-06T07:23:02.319419+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5244":{"gid":5244,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/2668925454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":2668925454},{"type":"v1","addr":"172.21.15.83:6837","nonce":2668925454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5244,"qdb_cluster":[5244]},"id":2}]} 2024-03-06T07:23:03.177 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 13 2024-03-06T07:23:03.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.176+0000 7f9be553e640 1 -- 172.21.15.83:0/3277957255 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9bc006b4d0 msgr2=0x7f9bc006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.176+0000 7f9be553e640 1 --2- 172.21.15.83:0/3277957255 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9bc006b4d0 0x7f9bc006d990 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9be00d3180 tx=0x7f9bc8009190 comp rx=0 tx=0).stop 2024-03-06T07:23:03.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.176+0000 7f9be553e640 1 -- 172.21.15.83:0/3277957255 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9bcc004d40 msgr2=0x7f9be00cc6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.176+0000 7f9be553e640 1 --2- 172.21.15.83:0/3277957255 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9bcc004d40 0x7f9be00cc6a0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0007340 tx=0x7f9bd0007740 comp rx=0 tx=0).stop 2024-03-06T07:23:03.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.176+0000 7f9be553e640 1 -- 172.21.15.83:0/3277957255 shutdown_connections 2024-03-06T07:23:03.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.176+0000 7f9be553e640 1 --2- 172.21.15.83:0/3277957255 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9bc006b4d0 0x7f9bc006d990 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.176+0000 7f9be553e640 1 --2- 172.21.15.83:0/3277957255 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9be01f4c00 0x7f9be00b0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.176+0000 7f9be553e640 1 --2- 172.21.15.83:0/3277957255 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9be01f44c0 0x7f9be00a15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.176+0000 7f9be553e640 1 --2- 172.21.15.83:0/3277957255 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9bcc004d40 0x7f9be00cc6a0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.176+0000 7f9be553e640 1 -- 172.21.15.83:0/3277957255 >> 172.21.15.83:0/3277957255 conn(0x7f9be00d3cd0 msgr2=0x7f9be00b5f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:03.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.177+0000 7f9be553e640 1 -- 172.21.15.83:0/3277957255 shutdown_connections 2024-03-06T07:23:03.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.177+0000 7f9be553e640 1 -- 172.21.15.83:0/3277957255 wait complete. 2024-03-06T07:23:03.190 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:03.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.289+0000 7f746cc81640 1 Processor -- start 2024-03-06T07:23:03.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.289+0000 7f746cc81640 1 -- start start 2024-03-06T07:23:03.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.290+0000 7f746cc81640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f746815dfe0 0x7f74680cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.290+0000 7f746cc81640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f74680cf830 0x7f74680d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.290+0000 7f746cc81640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74680b3430 con 0x7f74680d27f0 2024-03-06T07:23:03.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.290+0000 7f746cc81640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74680b31e0 con 0x7f74680cf830 2024-03-06T07:23:03.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.290+0000 7f746cc81640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74680b3d60 con 0x7f746815dfe0 2024-03-06T07:23:03.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.290+0000 7f745dd9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f74680cf830 0x7f74680d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.290+0000 7f745dd9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f74680cf830 0x7f74680d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35322/0 (socket says 172.21.15.83:35322) 2024-03-06T07:23:03.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.290+0000 7f745dd9b640 1 -- 172.21.15.83:0/1861477428 learned_addr learned my addr 172.21.15.83:0/1861477428 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:03.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.290+0000 7f746659c640 1 --2- 172.21.15.83:0/1861477428 >> v2:172.21.15.183:3300/0 conn(0x7f746815dfe0 0x7f74680cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.290+0000 7f745dd9b640 1 -- 172.21.15.83:0/1861477428 >> v2:172.21.15.183:3300/0 conn(0x7f746815dfe0 msgr2=0x7f74680cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:03.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.290+0000 7f745dd9b640 1 --2- 172.21.15.83:0/1861477428 >> v2:172.21.15.183:3300/0 conn(0x7f746815dfe0 0x7f74680cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.290+0000 7f745dd9b640 1 -- 172.21.15.83:0/1861477428 >> v1:172.21.15.183:6789/0 conn(0x7f74680d27f0 legacy=0x7f74680d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:03.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.290+0000 7f745dd9b640 1 -- 172.21.15.83:0/1861477428 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74680b3710 con 0x7f74680cf830 2024-03-06T07:23:03.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.290+0000 7f746659c640 1 --2- 172.21.15.83:0/1861477428 >> v2:172.21.15.183:3300/0 conn(0x7f746815dfe0 0x7f74680cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:03.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.291+0000 7f745dd9b640 1 --2- 172.21.15.83:0/1861477428 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74680cf830 0x7f74680d22b0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f7450009870 tx=0x7f745002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72f4119f1f93331e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.291+0000 7f7465d9b640 1 -- 172.21.15.83:0/1861477428 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7450034070 con 0x7f74680cf830 2024-03-06T07:23:03.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.291+0000 7f7465d9b640 1 -- 172.21.15.83:0/1861477428 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7450030860 con 0x7f74680cf830 2024-03-06T07:23:03.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.291+0000 7f7465d9b640 1 -- 172.21.15.83:0/1861477428 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7450030c30 con 0x7f74680cf830 2024-03-06T07:23:03.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.292+0000 7f746cc81640 1 -- 172.21.15.83:0/1861477428 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74680cf830 msgr2=0x7f74680d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.292+0000 7f746cc81640 1 --2- 172.21.15.83:0/1861477428 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74680cf830 0x7f74680d22b0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f7450009870 tx=0x7f745002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.292+0000 7f746cc81640 1 -- 172.21.15.83:0/1861477428 shutdown_connections 2024-03-06T07:23:03.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.292+0000 7f746cc81640 1 --2- 172.21.15.83:0/1861477428 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74680cf830 0x7f74680d22b0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.292+0000 7f746cc81640 1 --2- 172.21.15.83:0/1861477428 >> v2:172.21.15.183:3300/0 conn(0x7f746815dfe0 0x7f74680cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.292+0000 7f746cc81640 1 -- 172.21.15.83:0/1861477428 >> 172.21.15.83:0/1861477428 conn(0x7f74680d3cd0 msgr2=0x7f74680d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:03.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.292+0000 7f746cc81640 1 -- 172.21.15.83:0/1861477428 shutdown_connections 2024-03-06T07:23:03.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.292+0000 7f746cc81640 1 -- 172.21.15.83:0/1861477428 wait complete. 2024-03-06T07:23:03.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.294+0000 7f746cc81640 1 Processor -- start 2024-03-06T07:23:03.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.294+0000 7f746cc81640 1 -- start start 2024-03-06T07:23:03.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.295+0000 7f746cc81640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f74680cf830 0x7f74680cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.295+0000 7f746cc81640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74680d27f0 0x7f74680b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.295+0000 7f746cc81640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f746815dfe0 0x7f74680b6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.295+0000 7f746cc81640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74680d1df0 con 0x7f74680d27f0 2024-03-06T07:23:03.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.295+0000 7f746cc81640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74680d1c70 con 0x7f74680cf830 2024-03-06T07:23:03.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.295+0000 7f746cc81640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f746815e3c0 con 0x7f746815dfe0 2024-03-06T07:23:03.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.295+0000 7f745dd9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74680d27f0 0x7f74680b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.295+0000 7f745dd9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74680d27f0 0x7f74680b68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35338/0 (socket says 172.21.15.83:35338) 2024-03-06T07:23:03.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.295+0000 7f745dd9b640 1 -- 172.21.15.83:0/973850581 learned_addr learned my addr 172.21.15.83:0/973850581 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:03.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.295+0000 7f7466d9d640 1 --2- 172.21.15.83:0/973850581 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f746815dfe0 0x7f74680b6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.295+0000 7f746659c640 1 --2- 172.21.15.83:0/973850581 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f74680cf830 0x7f74680cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.296+0000 7f745dd9b640 1 -- 172.21.15.83:0/973850581 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f746815dfe0 msgr2=0x7f74680b6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.296+0000 7f745dd9b640 1 --2- 172.21.15.83:0/973850581 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f746815dfe0 0x7f74680b6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.296+0000 7f745dd9b640 1 -- 172.21.15.83:0/973850581 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f74680cf830 msgr2=0x7f74680cbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.296+0000 7f745dd9b640 1 --2- 172.21.15.83:0/973850581 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f74680cf830 0x7f74680cbdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.296+0000 7f745dd9b640 1 -- 172.21.15.83:0/973850581 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74680cc330 con 0x7f74680d27f0 2024-03-06T07:23:03.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.296+0000 7f7466d9d640 1 --2- 172.21.15.83:0/973850581 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f746815dfe0 0x7f74680b6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:03.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.296+0000 7f746659c640 1 --2- 172.21.15.83:0/973850581 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f74680cf830 0x7f74680cbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:03.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.296+0000 7f745dd9b640 1 --2- 172.21.15.83:0/973850581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74680d27f0 0x7f74680b68a0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f745002b9e0 tx=0x7f74500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.296+0000 7f745f7fe640 1 -- 172.21.15.83:0/973850581 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7450034040 con 0x7f74680d27f0 2024-03-06T07:23:03.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.296+0000 7f745f7fe640 1 -- 172.21.15.83:0/973850581 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7450039d90 con 0x7f74680d27f0 2024-03-06T07:23:03.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.296+0000 7f745f7fe640 1 -- 172.21.15.83:0/973850581 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7450043630 con 0x7f74680d27f0 2024-03-06T07:23:03.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.296+0000 7f746cc81640 1 -- 172.21.15.83:0/973850581 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74680cc5b0 con 0x7f74680d27f0 2024-03-06T07:23:03.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.296+0000 7f746cc81640 1 -- 172.21.15.83:0/973850581 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74680ccb70 con 0x7f74680d27f0 2024-03-06T07:23:03.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.297+0000 7f746cc81640 1 -- 172.21.15.83:0/973850581 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f746804f330 con 0x7f74680d27f0 2024-03-06T07:23:03.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.297+0000 7f745f7fe640 1 -- 172.21.15.83:0/973850581 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f745004a060 con 0x7f74680d27f0 2024-03-06T07:23:03.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.298+0000 7f745f7fe640 1 --2- 172.21.15.83:0/973850581 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f744806b4d0 0x7f744806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.298+0000 7f746659c640 1 --2- 172.21.15.83:0/973850581 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f744806b4d0 0x7f744806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.298+0000 7f745f7fe640 1 -- 172.21.15.83:0/973850581 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f74500ac280 con 0x7f74680d27f0 2024-03-06T07:23:03.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.298+0000 7f746659c640 1 --2- 172.21.15.83:0/973850581 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f744806b4d0 0x7f744806d990 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7458006fd0 tx=0x7f7458008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.307+0000 7f745f7fe640 1 -- 172.21.15.83:0/973850581 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f74680cc330 con 0x7f74680d27f0 2024-03-06T07:23:03.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.484+0000 7f746cc81640 1 -- 172.21.15.83:0/973850581 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f746815d220 con 0x7f74680d27f0 2024-03-06T07:23:03.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.486+0000 7f745f7fe640 1 -- 172.21.15.83:0/973850581 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16681 (secure 0 0 0) 0x7f745003e070 con 0x7f74680d27f0 2024-03-06T07:23:03.488 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:23:03.488 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":33,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:23:01.812855+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:58.568071+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:59.116376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:03.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.487+0000 7f746cc81640 1 -- 172.21.15.83:0/973850581 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f744806b4d0 msgr2=0x7f744806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.487+0000 7f746cc81640 1 --2- 172.21.15.83:0/973850581 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f744806b4d0 0x7f744806d990 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7458006fd0 tx=0x7f7458008040 comp rx=0 tx=0).stop 2024-03-06T07:23:03.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.487+0000 7f746cc81640 1 -- 172.21.15.83:0/973850581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74680d27f0 msgr2=0x7f74680b68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.487+0000 7f746cc81640 1 --2- 172.21.15.83:0/973850581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74680d27f0 0x7f74680b68a0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f745002b9e0 tx=0x7f74500046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.488+0000 7f746cc81640 1 -- 172.21.15.83:0/973850581 shutdown_connections 2024-03-06T07:23:03.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.488+0000 7f746cc81640 1 --2- 172.21.15.83:0/973850581 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f744806b4d0 0x7f744806d990 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.488+0000 7f746cc81640 1 --2- 172.21.15.83:0/973850581 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f746815dfe0 0x7f74680b6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.488+0000 7f746cc81640 1 --2- 172.21.15.83:0/973850581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74680d27f0 0x7f74680b68a0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.488+0000 7f746cc81640 1 --2- 172.21.15.83:0/973850581 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f74680cf830 0x7f74680cbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.488+0000 7f746cc81640 1 -- 172.21.15.83:0/973850581 >> 172.21.15.83:0/973850581 conn(0x7f74680d3cd0 msgr2=0x7f74680b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:03.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.488+0000 7f746cc81640 1 -- 172.21.15.83:0/973850581 shutdown_connections 2024-03-06T07:23:03.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.488+0000 7f746cc81640 1 -- 172.21.15.83:0/973850581 wait complete. 2024-03-06T07:23:03.501 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:23:03.501 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:23:03.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.600+0000 7f0d00113640 1 Processor -- start 2024-03-06T07:23:03.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.600+0000 7f0d00113640 1 -- start start 2024-03-06T07:23:03.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.601+0000 7f0d00113640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0cf815dfe0 0x7f0cf80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.601+0000 7f0d00113640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0cf80cf830 0x7f0cf80d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.601+0000 7f0d00113640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf80b3430 con 0x7f0cf80d27f0 2024-03-06T07:23:03.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.601+0000 7f0d00113640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cf80d2c50 con 0x7f0cf80cf830 2024-03-06T07:23:03.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.601+0000 7f0d00113640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf80b31e0 con 0x7f0cf815dfe0 2024-03-06T07:23:03.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.601+0000 7f0cfd6ae640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0cf80cf830 0x7f0cf80d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.601+0000 7f0cfd6ae640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0cf80cf830 0x7f0cf80d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35352/0 (socket says 172.21.15.83:35352) 2024-03-06T07:23:03.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.601+0000 7f0cfd6ae640 1 -- 172.21.15.83:0/3596242586 learned_addr learned my addr 172.21.15.83:0/3596242586 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:03.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.602+0000 7f0cfdeaf640 1 --2- 172.21.15.83:0/3596242586 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0cf815dfe0 0x7f0cf80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.602+0000 7f0cfd6ae640 1 -- 172.21.15.83:0/3596242586 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0cf815dfe0 msgr2=0x7f0cf80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:03.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.602+0000 7f0cfd6ae640 1 --2- 172.21.15.83:0/3596242586 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0cf815dfe0 0x7f0cf80cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.602+0000 7f0cfd6ae640 1 -- 172.21.15.83:0/3596242586 >> v1:172.21.15.183:6789/0 conn(0x7f0cf80d27f0 legacy=0x7f0cf80d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.602+0000 7f0cfd6ae640 1 -- 172.21.15.83:0/3596242586 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf80b3710 con 0x7f0cf80cf830 2024-03-06T07:23:03.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.602+0000 7f0cfd6ae640 1 --2- 172.21.15.83:0/3596242586 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0cf80cf830 0x7f0cf80d22b0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f0cec009870 tx=0x7f0cec02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5038a7e12e4a02ec server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.603+0000 7f0cfcead640 1 -- 172.21.15.83:0/3596242586 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cec034070 con 0x7f0cf80cf830 2024-03-06T07:23:03.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.603+0000 7f0cfcead640 1 -- 172.21.15.83:0/3596242586 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cec030840 con 0x7f0cf80cf830 2024-03-06T07:23:03.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.603+0000 7f0cfcead640 1 -- 172.21.15.83:0/3596242586 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cec030c10 con 0x7f0cf80cf830 2024-03-06T07:23:03.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.603+0000 7f0d00113640 1 -- 172.21.15.83:0/3596242586 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0cf80cf830 msgr2=0x7f0cf80d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.603+0000 7f0d00113640 1 --2- 172.21.15.83:0/3596242586 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0cf80cf830 0x7f0cf80d22b0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f0cec009870 tx=0x7f0cec02b980 comp rx=0 tx=0).stop 2024-03-06T07:23:03.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.603+0000 7f0d00113640 1 -- 172.21.15.83:0/3596242586 shutdown_connections 2024-03-06T07:23:03.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.603+0000 7f0d00113640 1 --2- 172.21.15.83:0/3596242586 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0cf80cf830 0x7f0cf80d22b0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.603+0000 7f0d00113640 1 --2- 172.21.15.83:0/3596242586 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0cf815dfe0 0x7f0cf80cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.603+0000 7f0d00113640 1 -- 172.21.15.83:0/3596242586 >> 172.21.15.83:0/3596242586 conn(0x7f0cf80d3cd0 msgr2=0x7f0cf80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:03.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.603+0000 7f0d00113640 1 -- 172.21.15.83:0/3596242586 shutdown_connections 2024-03-06T07:23:03.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.603+0000 7f0d00113640 1 -- 172.21.15.83:0/3596242586 wait complete. 2024-03-06T07:23:03.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.604+0000 7f0d00113640 1 Processor -- start 2024-03-06T07:23:03.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.604+0000 7f0d00113640 1 -- start start 2024-03-06T07:23:03.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.605+0000 7f0d00113640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0cf80cf830 0x7f0cf8154270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.605+0000 7f0d00113640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0cf80d27f0 0x7f0cf80cb620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.605+0000 7f0d00113640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0cf815dfe0 0x7f0cf80cbb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.605+0000 7f0d00113640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf80d1c70 con 0x7f0cf815dfe0 2024-03-06T07:23:03.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.605+0000 7f0d00113640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf80b3d60 con 0x7f0cf80d27f0 2024-03-06T07:23:03.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.605+0000 7f0d00113640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf80d1df0 con 0x7f0cf80cf830 2024-03-06T07:23:03.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.605+0000 7f0cfe6b0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0cf815dfe0 0x7f0cf80cbb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.605+0000 7f0cfe6b0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0cf815dfe0 0x7f0cf80cbb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35362/0 (socket says 172.21.15.83:35362) 2024-03-06T07:23:03.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.605+0000 7f0cfe6b0640 1 -- 172.21.15.83:0/3263847137 learned_addr learned my addr 172.21.15.83:0/3263847137 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:03.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.605+0000 7f0cfd6ae640 1 --2- 172.21.15.83:0/3263847137 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0cf80d27f0 0x7f0cf80cb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.605+0000 7f0cfdeaf640 1 --2- 172.21.15.83:0/3263847137 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0cf80cf830 0x7f0cf8154270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.605+0000 7f0cfe6b0640 1 -- 172.21.15.83:0/3263847137 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0cf80cf830 msgr2=0x7f0cf8154270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.605+0000 7f0cfe6b0640 1 --2- 172.21.15.83:0/3263847137 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0cf80cf830 0x7f0cf8154270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.605+0000 7f0cfe6b0640 1 -- 172.21.15.83:0/3263847137 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0cf80d27f0 msgr2=0x7f0cf80cb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.605+0000 7f0cfe6b0640 1 --2- 172.21.15.83:0/3263847137 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0cf80d27f0 0x7f0cf80cb620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.605+0000 7f0cfe6b0640 1 -- 172.21.15.83:0/3263847137 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf80b7c50 con 0x7f0cf815dfe0 2024-03-06T07:23:03.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.605+0000 7f0cfd6ae640 1 --2- 172.21.15.83:0/3263847137 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0cf80d27f0 0x7f0cf80cb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:03.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.605+0000 7f0cfdeaf640 1 --2- 172.21.15.83:0/3263847137 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0cf80cf830 0x7f0cf8154270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:03.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.606+0000 7f0cfe6b0640 1 --2- 172.21.15.83:0/3263847137 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0cf815dfe0 0x7f0cf80cbb60 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8001eb0 tx=0x7f0ce800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.606+0000 7f0cf67fc640 1 -- 172.21.15.83:0/3263847137 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce8017070 con 0x7f0cf815dfe0 2024-03-06T07:23:03.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.606+0000 7f0cf67fc640 1 -- 172.21.15.83:0/3263847137 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce80052d0 con 0x7f0cf815dfe0 2024-03-06T07:23:03.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.606+0000 7f0d00113640 1 -- 172.21.15.83:0/3263847137 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cf80b7f30 con 0x7f0cf815dfe0 2024-03-06T07:23:03.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.606+0000 7f0cf67fc640 1 -- 172.21.15.83:0/3263847137 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce8003810 con 0x7f0cf815dfe0 2024-03-06T07:23:03.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.606+0000 7f0d00113640 1 -- 172.21.15.83:0/3263847137 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cf80b84f0 con 0x7f0cf815dfe0 2024-03-06T07:23:03.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.607+0000 7f0d00113640 1 -- 172.21.15.83:0/3263847137 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cf804f330 con 0x7f0cf815dfe0 2024-03-06T07:23:03.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.607+0000 7f0cf67fc640 1 -- 172.21.15.83:0/3263847137 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0ce801a070 con 0x7f0cf815dfe0 2024-03-06T07:23:03.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.608+0000 7f0cf67fc640 1 --2- 172.21.15.83:0/3263847137 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0ce406b4d0 0x7f0ce406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.608+0000 7f0cf67fc640 1 -- 172.21.15.83:0/3263847137 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0ce8089620 con 0x7f0cf815dfe0 2024-03-06T07:23:03.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.608+0000 7f0cfdeaf640 1 --2- 172.21.15.83:0/3263847137 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0ce406b4d0 0x7f0ce406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.608+0000 7f0cfdeaf640 1 --2- 172.21.15.83:0/3263847137 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0ce406b4d0 0x7f0ce406d990 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0007440 tx=0x7f0cf000b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.612+0000 7f0cf67fc640 1 -- 172.21.15.83:0/3263847137 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0ce8058400 con 0x7f0cf815dfe0 2024-03-06T07:23:03.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.859+0000 7f0d00113640 1 -- 172.21.15.83:0/3263847137 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0cf80ceb50 con 0x7f0ce406b4d0 2024-03-06T07:23:03.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.865+0000 7f0cf67fc640 1 -- 172.21.15.83:0/3263847137 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246633 (secure 0 0 0) 0x7f0cf80ceb50 con 0x7f0ce406b4d0 2024-03-06T07:23:03.870 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:23:03.871 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-06T07:23:03.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.869+0000 7f0d00113640 1 -- 172.21.15.83:0/3263847137 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0ce406b4d0 msgr2=0x7f0ce406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.869+0000 7f0d00113640 1 --2- 172.21.15.83:0/3263847137 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0ce406b4d0 0x7f0ce406d990 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0007440 tx=0x7f0cf000b040 comp rx=0 tx=0).stop 2024-03-06T07:23:03.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.869+0000 7f0d00113640 1 -- 172.21.15.83:0/3263847137 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0cf815dfe0 msgr2=0x7f0cf80cbb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.869+0000 7f0d00113640 1 --2- 172.21.15.83:0/3263847137 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0cf815dfe0 0x7f0cf80cbb60 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8001eb0 tx=0x7f0ce800c890 comp rx=0 tx=0).stop 2024-03-06T07:23:03.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.869+0000 7f0d00113640 1 -- 172.21.15.83:0/3263847137 shutdown_connections 2024-03-06T07:23:03.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.869+0000 7f0d00113640 1 --2- 172.21.15.83:0/3263847137 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0ce406b4d0 0x7f0ce406d990 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.869+0000 7f0d00113640 1 --2- 172.21.15.83:0/3263847137 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0cf815dfe0 0x7f0cf80cbb60 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.869+0000 7f0d00113640 1 --2- 172.21.15.83:0/3263847137 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0cf80d27f0 0x7f0cf80cb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.869+0000 7f0d00113640 1 --2- 172.21.15.83:0/3263847137 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0cf80cf830 0x7f0cf8154270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.869+0000 7f0d00113640 1 -- 172.21.15.83:0/3263847137 >> 172.21.15.83:0/3263847137 conn(0x7f0cf80d3cd0 msgr2=0x7f0cf806cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:03.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.869+0000 7f0d00113640 1 -- 172.21.15.83:0/3263847137 shutdown_connections 2024-03-06T07:23:03.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:03.869+0000 7f0d00113640 1 -- 172.21.15.83:0/3263847137 wait complete. 2024-03-06T07:23:03.882 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":65,"stamp":"2024-03-06T07:23:02.333161+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":35,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220000,"kb_used_data":3832,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754720,"statfs":{"total":773094113280,"available":772868833280,"internally_reserved":0,"allocated":3923968,"data_stored":2227948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12757,"internal_metadata":221236779},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000854"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797969+0000","last_change":"2024-03-06T07:22:59.797969+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797969+0000","last_undegraded":"2024-03-06T07:22:59.797969+0000","last_fullsized":"2024-03-06T07:22:59.797969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.799502+0000","last_change":"2024-03-06T07:22:59.799502+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.799502+0000","last_undegraded":"2024-03-06T07:22:59.799502+0000","last_fullsized":"2024-03-06T07:22:59.799502+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.804718+0000","last_change":"2024-03-06T07:22:59.804718+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.804718+0000","last_undegraded":"2024-03-06T07:22:59.804718+0000","last_fullsized":"2024-03-06T07:22:59.804718+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.800525+0000","last_change":"2024-03-06T07:22:59.800525+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.800525+0000","last_undegraded":"2024-03-06T07:22:59.800525+0000","last_fullsized":"2024-03-06T07:22:59.800525+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.798524+0000","last_change":"2024-03-06T07:22:59.798524+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.798524+0000","last_undegraded":"2024-03-06T07:22:59.798524+0000","last_fullsized":"2024-03-06T07:22:59.798524+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797989+0000","last_change":"2024-03-06T07:22:59.797989+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797989+0000","last_undegraded":"2024-03-06T07:22:59.797989+0000","last_fullsized":"2024-03-06T07:22:59.797989+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.801589+0000","last_change":"2024-03-06T07:22:59.801589+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.801589+0000","last_undegraded":"2024-03-06T07:22:59.801589+0000","last_fullsized":"2024-03-06T07:22:59.801589+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.804790+0000","last_change":"2024-03-06T07:22:59.804790+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.804790+0000","last_undegraded":"2024-03-06T07:22:59.804790+0000","last_fullsized":"2024-03-06T07:22:59.804790+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.799988+0000","last_change":"2024-03-06T07:22:59.799988+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.799988+0000","last_undegraded":"2024-03-06T07:22:59.799988+0000","last_fullsized":"2024-03-06T07:22:59.799988+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.799777+0000","last_change":"2024-03-06T07:22:59.799777+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.799777+0000","last_undegraded":"2024-03-06T07:22:59.799777+0000","last_fullsized":"2024-03-06T07:22:59.799777+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.801040+0000","last_change":"2024-03-06T07:22:59.801040+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.801040+0000","last_undegraded":"2024-03-06T07:22:59.801040+0000","last_fullsized":"2024-03-06T07:22:59.801040+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.803466+0000","last_change":"2024-03-06T07:22:59.803466+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.803466+0000","last_undegraded":"2024-03-06T07:22:59.803466+0000","last_fullsized":"2024-03-06T07:22:59.803466+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.798121+0000","last_change":"2024-03-06T07:22:59.798121+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.798121+0000","last_undegraded":"2024-03-06T07:22:59.798121+0000","last_fullsized":"2024-03-06T07:22:59.798121+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797601+0000","last_change":"2024-03-06T07:22:59.797601+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797601+0000","last_undegraded":"2024-03-06T07:22:59.797601+0000","last_fullsized":"2024-03-06T07:22:59.797601+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.800974+0000","last_change":"2024-03-06T07:22:59.800974+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.800974+0000","last_undegraded":"2024-03-06T07:22:59.800974+0000","last_fullsized":"2024-03-06T07:22:59.800974+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.801086+0000","last_change":"2024-03-06T07:22:59.801086+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.801086+0000","last_undegraded":"2024-03-06T07:22:59.801086+0000","last_fullsized":"2024-03-06T07:22:59.801086+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.802385+0000","last_change":"2024-03-06T07:22:59.802385+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.802385+0000","last_undegraded":"2024-03-06T07:22:59.802385+0000","last_fullsized":"2024-03-06T07:22:59.802385+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.799075+0000","last_change":"2024-03-06T07:22:59.799075+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.799075+0000","last_undegraded":"2024-03-06T07:22:59.799075+0000","last_fullsized":"2024-03-06T07:22:59.799075+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797394+0000","last_change":"2024-03-06T07:22:59.797394+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797394+0000","last_undegraded":"2024-03-06T07:22:59.797394+0000","last_fullsized":"2024-03-06T07:22:59.797394+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.798311+0000","last_change":"2024-03-06T07:22:59.798311+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.798311+0000","last_undegraded":"2024-03-06T07:22:59.798311+0000","last_fullsized":"2024-03-06T07:22:59.798311+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.800053+0000","last_change":"2024-03-06T07:22:59.800053+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.800053+0000","last_undegraded":"2024-03-06T07:22:59.800053+0000","last_fullsized":"2024-03-06T07:22:59.800053+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796916+0000","last_change":"2024-03-06T07:22:59.796916+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796916+0000","last_undegraded":"2024-03-06T07:22:59.796916+0000","last_fullsized":"2024-03-06T07:22:59.796916+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796381+0000","last_change":"2024-03-06T07:22:59.796381+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796381+0000","last_undegraded":"2024-03-06T07:22:59.796381+0000","last_fullsized":"2024-03-06T07:22:59.796381+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.806940+0000","last_change":"2024-03-06T07:22:59.806940+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.806940+0000","last_undegraded":"2024-03-06T07:22:59.806940+0000","last_fullsized":"2024-03-06T07:22:59.806940+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.799846+0000","last_change":"2024-03-06T07:22:59.799846+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.799846+0000","last_undegraded":"2024-03-06T07:22:59.799846+0000","last_fullsized":"2024-03-06T07:22:59.799846+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.798684+0000","last_change":"2024-03-06T07:22:59.798684+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.798684+0000","last_undegraded":"2024-03-06T07:22:59.798684+0000","last_fullsized":"2024-03-06T07:22:59.798684+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.800428+0000","last_change":"2024-03-06T07:22:59.800428+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.800428+0000","last_undegraded":"2024-03-06T07:22:59.800428+0000","last_fullsized":"2024-03-06T07:22:59.800428+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.803078+0000","last_change":"2024-03-06T07:22:59.803078+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.803078+0000","last_undegraded":"2024-03-06T07:22:59.803078+0000","last_fullsized":"2024-03-06T07:22:59.803078+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.799594+0000","last_change":"2024-03-06T07:22:59.799594+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.799594+0000","last_undegraded":"2024-03-06T07:22:59.799594+0000","last_fullsized":"2024-03-06T07:22:59.799594+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.801930+0000","last_change":"2024-03-06T07:22:59.801930+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.801930+0000","last_undegraded":"2024-03-06T07:22:59.801930+0000","last_fullsized":"2024-03-06T07:22:59.801930+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797043+0000","last_change":"2024-03-06T07:22:59.797043+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797043+0000","last_undegraded":"2024-03-06T07:22:59.797043+0000","last_fullsized":"2024-03-06T07:22:59.797043+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.804395+0000","last_change":"2024-03-06T07:22:59.804395+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.804395+0000","last_undegraded":"2024-03-06T07:22:59.804395+0000","last_fullsized":"2024-03-06T07:22:59.804395+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796359+0000","last_change":"2024-03-06T07:22:59.796359+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796359+0000","last_undegraded":"2024-03-06T07:22:59.796359+0000","last_fullsized":"2024-03-06T07:22:59.796359+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.812031+0000","last_change":"2024-03-06T07:22:59.810363+0000","last_active":"2024-03-06T07:22:59.812031+0000","last_peered":"2024-03-06T07:22:59.812031+0000","last_clean":"2024-03-06T07:22:59.812031+0000","last_became_active":"2024-03-06T07:22:59.810032+0000","last_became_peered":"2024-03-06T07:22:59.810032+0000","last_unstale":"2024-03-06T07:22:59.812031+0000","last_undegraded":"2024-03-06T07:22:59.812031+0000","last_fullsized":"2024-03-06T07:22:59.812031+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:36:12.264660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797548+0000","last_change":"2024-03-06T07:22:59.797548+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797548+0000","last_undegraded":"2024-03-06T07:22:59.797548+0000","last_fullsized":"2024-03-06T07:22:59.797548+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.811396+0000","last_change":"2024-03-06T07:22:59.810687+0000","last_active":"2024-03-06T07:22:59.811396+0000","last_peered":"2024-03-06T07:22:59.811396+0000","last_clean":"2024-03-06T07:22:59.811396+0000","last_became_active":"2024-03-06T07:22:59.810405+0000","last_became_peered":"2024-03-06T07:22:59.810405+0000","last_unstale":"2024-03-06T07:22:59.811396+0000","last_undegraded":"2024-03-06T07:22:59.811396+0000","last_fullsized":"2024-03-06T07:22:59.811396+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:24:26.919565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.798640+0000","last_change":"2024-03-06T07:22:59.798640+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.798640+0000","last_undegraded":"2024-03-06T07:22:59.798640+0000","last_fullsized":"2024-03-06T07:22:59.798640+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.807686+0000","last_change":"2024-03-06T07:22:59.807093+0000","last_active":"2024-03-06T07:22:59.807686+0000","last_peered":"2024-03-06T07:22:59.807686+0000","last_clean":"2024-03-06T07:22:59.807686+0000","last_became_active":"2024-03-06T07:22:59.806858+0000","last_became_peered":"2024-03-06T07:22:59.806858+0000","last_unstale":"2024-03-06T07:22:59.807686+0000","last_undegraded":"2024-03-06T07:22:59.807686+0000","last_fullsized":"2024-03-06T07:22:59.807686+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:38:31.114570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.799435+0000","last_change":"2024-03-06T07:22:59.799435+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.799435+0000","last_undegraded":"2024-03-06T07:22:59.799435+0000","last_fullsized":"2024-03-06T07:22:59.799435+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.809136+0000","last_change":"2024-03-06T07:22:59.805788+0000","last_active":"2024-03-06T07:22:59.809136+0000","last_peered":"2024-03-06T07:22:59.809136+0000","last_clean":"2024-03-06T07:22:59.809136+0000","last_became_active":"2024-03-06T07:22:59.805463+0000","last_became_peered":"2024-03-06T07:22:59.805463+0000","last_unstale":"2024-03-06T07:22:59.809136+0000","last_undegraded":"2024-03-06T07:22:59.809136+0000","last_fullsized":"2024-03-06T07:22:59.809136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:14.437437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.801363+0000","last_change":"2024-03-06T07:22:59.801363+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.801363+0000","last_undegraded":"2024-03-06T07:22:59.801363+0000","last_fullsized":"2024-03-06T07:22:59.801363+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.811699+0000","last_change":"2024-03-06T07:22:59.808162+0000","last_active":"2024-03-06T07:22:59.811699+0000","last_peered":"2024-03-06T07:22:59.811699+0000","last_clean":"2024-03-06T07:22:59.811699+0000","last_became_active":"2024-03-06T07:22:59.807875+0000","last_became_peered":"2024-03-06T07:22:59.807875+0000","last_unstale":"2024-03-06T07:22:59.811699+0000","last_undegraded":"2024-03-06T07:22:59.811699+0000","last_fullsized":"2024-03-06T07:22:59.811699+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:53.014712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.802462+0000","last_change":"2024-03-06T07:22:59.802462+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.802462+0000","last_undegraded":"2024-03-06T07:22:59.802462+0000","last_fullsized":"2024-03-06T07:22:59.802462+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.811787+0000","last_change":"2024-03-06T07:22:59.806399+0000","last_active":"2024-03-06T07:22:59.811787+0000","last_peered":"2024-03-06T07:22:59.811787+0000","last_clean":"2024-03-06T07:22:59.811787+0000","last_became_active":"2024-03-06T07:22:59.805990+0000","last_became_peered":"2024-03-06T07:22:59.805990+0000","last_unstale":"2024-03-06T07:22:59.811787+0000","last_undegraded":"2024-03-06T07:22:59.811787+0000","last_fullsized":"2024-03-06T07:22:59.811787+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:07.271082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797495+0000","last_change":"2024-03-06T07:22:59.797495+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797495+0000","last_undegraded":"2024-03-06T07:22:59.797495+0000","last_fullsized":"2024-03-06T07:22:59.797495+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.806357+0000","last_change":"2024-03-06T07:22:59.805251+0000","last_active":"2024-03-06T07:22:59.806357+0000","last_peered":"2024-03-06T07:22:59.806357+0000","last_clean":"2024-03-06T07:22:59.806357+0000","last_became_active":"2024-03-06T07:22:59.804769+0000","last_became_peered":"2024-03-06T07:22:59.804769+0000","last_unstale":"2024-03-06T07:22:59.806357+0000","last_undegraded":"2024-03-06T07:22:59.806357+0000","last_fullsized":"2024-03-06T07:22:59.806357+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:30:21.970435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797342+0000","last_change":"2024-03-06T07:22:59.797342+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797342+0000","last_undegraded":"2024-03-06T07:22:59.797342+0000","last_fullsized":"2024-03-06T07:22:59.797342+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.810668+0000","last_change":"2024-03-06T07:22:59.804481+0000","last_active":"2024-03-06T07:22:59.810668+0000","last_peered":"2024-03-06T07:22:59.810668+0000","last_clean":"2024-03-06T07:22:59.810668+0000","last_became_active":"2024-03-06T07:22:59.803413+0000","last_became_peered":"2024-03-06T07:22:59.803413+0000","last_unstale":"2024-03-06T07:22:59.810668+0000","last_undegraded":"2024-03-06T07:22:59.810668+0000","last_fullsized":"2024-03-06T07:22:59.810668+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:14.437437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.802007+0000","last_change":"2024-03-06T07:22:59.802007+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.802007+0000","last_undegraded":"2024-03-06T07:22:59.802007+0000","last_fullsized":"2024-03-06T07:22:59.802007+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.810221+0000","last_change":"2024-03-06T07:22:59.808659+0000","last_active":"2024-03-06T07:22:59.810221+0000","last_peered":"2024-03-06T07:22:59.810221+0000","last_clean":"2024-03-06T07:22:59.810221+0000","last_became_active":"2024-03-06T07:22:59.808416+0000","last_became_peered":"2024-03-06T07:22:59.808416+0000","last_unstale":"2024-03-06T07:22:59.810221+0000","last_undegraded":"2024-03-06T07:22:59.810221+0000","last_fullsized":"2024-03-06T07:22:59.810221+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:32.447994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.798133+0000","last_change":"2024-03-06T07:22:59.798133+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.798133+0000","last_undegraded":"2024-03-06T07:22:59.798133+0000","last_fullsized":"2024-03-06T07:22:59.798133+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.805530+0000","last_change":"2024-03-06T07:22:59.804638+0000","last_active":"2024-03-06T07:22:59.805530+0000","last_peered":"2024-03-06T07:22:59.805530+0000","last_clean":"2024-03-06T07:22:59.805530+0000","last_became_active":"2024-03-06T07:22:59.804026+0000","last_became_peered":"2024-03-06T07:22:59.804026+0000","last_unstale":"2024-03-06T07:22:59.805530+0000","last_undegraded":"2024-03-06T07:22:59.805530+0000","last_fullsized":"2024-03-06T07:22:59.805530+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:04:10.795313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.798485+0000","last_change":"2024-03-06T07:22:59.798485+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.798485+0000","last_undegraded":"2024-03-06T07:22:59.798485+0000","last_fullsized":"2024-03-06T07:22:59.798485+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.816328+0000","last_change":"2024-03-06T07:22:59.815296+0000","last_active":"2024-03-06T07:22:59.816328+0000","last_peered":"2024-03-06T07:22:59.816328+0000","last_clean":"2024-03-06T07:22:59.816328+0000","last_became_active":"2024-03-06T07:22:59.815033+0000","last_became_peered":"2024-03-06T07:22:59.815033+0000","last_unstale":"2024-03-06T07:22:59.816328+0000","last_undegraded":"2024-03-06T07:22:59.816328+0000","last_fullsized":"2024-03-06T07:22:59.816328+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:29.015317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.808497+0000","last_change":"2024-03-06T07:22:59.808497+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.808497+0000","last_undegraded":"2024-03-06T07:22:59.808497+0000","last_fullsized":"2024-03-06T07:22:59.808497+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.808772+0000","last_change":"2024-03-06T07:22:59.807399+0000","last_active":"2024-03-06T07:22:59.808772+0000","last_peered":"2024-03-06T07:22:59.808772+0000","last_clean":"2024-03-06T07:22:59.808772+0000","last_became_active":"2024-03-06T07:22:59.807150+0000","last_became_peered":"2024-03-06T07:22:59.807150+0000","last_unstale":"2024-03-06T07:22:59.808772+0000","last_undegraded":"2024-03-06T07:22:59.808772+0000","last_fullsized":"2024-03-06T07:22:59.808772+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:19.337313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.798310+0000","last_change":"2024-03-06T07:22:59.798310+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.798310+0000","last_undegraded":"2024-03-06T07:22:59.798310+0000","last_fullsized":"2024-03-06T07:22:59.798310+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.811735+0000","last_change":"2024-03-06T07:22:59.809344+0000","last_active":"2024-03-06T07:22:59.811735+0000","last_peered":"2024-03-06T07:22:59.811735+0000","last_clean":"2024-03-06T07:22:59.811735+0000","last_became_active":"2024-03-06T07:22:59.809076+0000","last_became_peered":"2024-03-06T07:22:59.809076+0000","last_unstale":"2024-03-06T07:22:59.811735+0000","last_undegraded":"2024-03-06T07:22:59.811735+0000","last_fullsized":"2024-03-06T07:22:59.811735+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:12.683272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796615+0000","last_change":"2024-03-06T07:22:59.796615+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796615+0000","last_undegraded":"2024-03-06T07:22:59.796615+0000","last_fullsized":"2024-03-06T07:22:59.796615+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.809522+0000","last_change":"2024-03-06T07:22:59.807007+0000","last_active":"2024-03-06T07:22:59.809522+0000","last_peered":"2024-03-06T07:22:59.809522+0000","last_clean":"2024-03-06T07:22:59.809522+0000","last_became_active":"2024-03-06T07:22:59.806750+0000","last_became_peered":"2024-03-06T07:22:59.806750+0000","last_unstale":"2024-03-06T07:22:59.809522+0000","last_undegraded":"2024-03-06T07:22:59.809522+0000","last_fullsized":"2024-03-06T07:22:59.809522+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:24:26.919565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.804026+0000","last_change":"2024-03-06T07:22:59.804026+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.804026+0000","last_undegraded":"2024-03-06T07:22:59.804026+0000","last_fullsized":"2024-03-06T07:22:59.804026+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.809695+0000","last_change":"2024-03-06T07:22:59.808043+0000","last_active":"2024-03-06T07:22:59.809695+0000","last_peered":"2024-03-06T07:22:59.809695+0000","last_clean":"2024-03-06T07:22:59.809695+0000","last_became_active":"2024-03-06T07:22:59.807794+0000","last_became_peered":"2024-03-06T07:22:59.807794+0000","last_unstale":"2024-03-06T07:22:59.809695+0000","last_undegraded":"2024-03-06T07:22:59.809695+0000","last_fullsized":"2024-03-06T07:22:59.809695+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:08:18.005082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796469+0000","last_change":"2024-03-06T07:22:59.796469+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796469+0000","last_undegraded":"2024-03-06T07:22:59.796469+0000","last_fullsized":"2024-03-06T07:22:59.796469+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.807259+0000","last_change":"2024-03-06T07:22:59.806459+0000","last_active":"2024-03-06T07:22:59.807259+0000","last_peered":"2024-03-06T07:22:59.807259+0000","last_clean":"2024-03-06T07:22:59.807259+0000","last_became_active":"2024-03-06T07:22:59.806194+0000","last_became_peered":"2024-03-06T07:22:59.806194+0000","last_unstale":"2024-03-06T07:22:59.807259+0000","last_undegraded":"2024-03-06T07:22:59.807259+0000","last_fullsized":"2024-03-06T07:22:59.807259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:53.894652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796668+0000","last_change":"2024-03-06T07:22:59.796668+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796668+0000","last_undegraded":"2024-03-06T07:22:59.796668+0000","last_fullsized":"2024-03-06T07:22:59.796668+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.810182+0000","last_change":"2024-03-06T07:22:59.808555+0000","last_active":"2024-03-06T07:22:59.810182+0000","last_peered":"2024-03-06T07:22:59.810182+0000","last_clean":"2024-03-06T07:22:59.810182+0000","last_became_active":"2024-03-06T07:22:59.808284+0000","last_became_peered":"2024-03-06T07:22:59.808284+0000","last_unstale":"2024-03-06T07:22:59.810182+0000","last_undegraded":"2024-03-06T07:22:59.810182+0000","last_fullsized":"2024-03-06T07:22:59.810182+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:30:21.970435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.803648+0000","last_change":"2024-03-06T07:22:59.803648+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.803648+0000","last_undegraded":"2024-03-06T07:22:59.803648+0000","last_fullsized":"2024-03-06T07:22:59.803648+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.805500+0000","last_change":"2024-03-06T07:22:59.802730+0000","last_active":"2024-03-06T07:22:59.805500+0000","last_peered":"2024-03-06T07:22:59.805500+0000","last_clean":"2024-03-06T07:22:59.805500+0000","last_became_active":"2024-03-06T07:22:59.802461+0000","last_became_peered":"2024-03-06T07:22:59.802461+0000","last_unstale":"2024-03-06T07:22:59.805500+0000","last_undegraded":"2024-03-06T07:22:59.805500+0000","last_fullsized":"2024-03-06T07:22:59.805500+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:26:21.615251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.798752+0000","last_change":"2024-03-06T07:22:59.798752+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.798752+0000","last_undegraded":"2024-03-06T07:22:59.798752+0000","last_fullsized":"2024-03-06T07:22:59.798752+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.811762+0000","last_change":"2024-03-06T07:22:59.807016+0000","last_active":"2024-03-06T07:22:59.811762+0000","last_peered":"2024-03-06T07:22:59.811762+0000","last_clean":"2024-03-06T07:22:59.811762+0000","last_became_active":"2024-03-06T07:22:59.806733+0000","last_became_peered":"2024-03-06T07:22:59.806733+0000","last_unstale":"2024-03-06T07:22:59.811762+0000","last_undegraded":"2024-03-06T07:22:59.811762+0000","last_fullsized":"2024-03-06T07:22:59.811762+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:55.870572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797444+0000","last_change":"2024-03-06T07:22:59.797444+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797444+0000","last_undegraded":"2024-03-06T07:22:59.797444+0000","last_fullsized":"2024-03-06T07:22:59.797444+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.811731+0000","last_change":"2024-03-06T07:22:59.808721+0000","last_active":"2024-03-06T07:22:59.811731+0000","last_peered":"2024-03-06T07:22:59.811731+0000","last_clean":"2024-03-06T07:22:59.811731+0000","last_became_active":"2024-03-06T07:22:59.808465+0000","last_became_peered":"2024-03-06T07:22:59.808465+0000","last_unstale":"2024-03-06T07:22:59.811731+0000","last_undegraded":"2024-03-06T07:22:59.811731+0000","last_fullsized":"2024-03-06T07:22:59.811731+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:08.896012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796675+0000","last_change":"2024-03-06T07:22:59.796675+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796675+0000","last_undegraded":"2024-03-06T07:22:59.796675+0000","last_fullsized":"2024-03-06T07:22:59.796675+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.805543+0000","last_change":"2024-03-06T07:22:59.804367+0000","last_active":"2024-03-06T07:22:59.805543+0000","last_peered":"2024-03-06T07:22:59.805543+0000","last_clean":"2024-03-06T07:22:59.805543+0000","last_became_active":"2024-03-06T07:22:59.804115+0000","last_became_peered":"2024-03-06T07:22:59.804115+0000","last_unstale":"2024-03-06T07:22:59.805543+0000","last_undegraded":"2024-03-06T07:22:59.805543+0000","last_fullsized":"2024-03-06T07:22:59.805543+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:44:35.664305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797737+0000","last_change":"2024-03-06T07:22:59.797737+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797737+0000","last_undegraded":"2024-03-06T07:22:59.797737+0000","last_fullsized":"2024-03-06T07:22:59.797737+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.806310+0000","last_change":"2024-03-06T07:22:59.804749+0000","last_active":"2024-03-06T07:22:59.806310+0000","last_peered":"2024-03-06T07:22:59.806310+0000","last_clean":"2024-03-06T07:22:59.806310+0000","last_became_active":"2024-03-06T07:22:59.802839+0000","last_became_peered":"2024-03-06T07:22:59.802839+0000","last_unstale":"2024-03-06T07:22:59.806310+0000","last_undegraded":"2024-03-06T07:22:59.806310+0000","last_fullsized":"2024-03-06T07:22:59.806310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:04:10.795313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796191+0000","last_change":"2024-03-06T07:22:59.796191+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796191+0000","last_undegraded":"2024-03-06T07:22:59.796191+0000","last_fullsized":"2024-03-06T07:22:59.796191+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.813816+0000","last_change":"2024-03-06T07:22:59.810390+0000","last_active":"2024-03-06T07:22:59.813816+0000","last_peered":"2024-03-06T07:22:59.813816+0000","last_clean":"2024-03-06T07:22:59.813816+0000","last_became_active":"2024-03-06T07:22:59.810140+0000","last_became_peered":"2024-03-06T07:22:59.810140+0000","last_unstale":"2024-03-06T07:22:59.813816+0000","last_undegraded":"2024-03-06T07:22:59.813816+0000","last_fullsized":"2024-03-06T07:22:59.813816+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:13.131161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.802019+0000","last_change":"2024-03-06T07:22:59.802019+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.802019+0000","last_undegraded":"2024-03-06T07:22:59.802019+0000","last_fullsized":"2024-03-06T07:22:59.802019+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.814736+0000","last_change":"2024-03-06T07:22:59.812896+0000","last_active":"2024-03-06T07:22:59.814736+0000","last_peered":"2024-03-06T07:22:59.814736+0000","last_clean":"2024-03-06T07:22:59.814736+0000","last_became_active":"2024-03-06T07:22:59.812598+0000","last_became_peered":"2024-03-06T07:22:59.812598+0000","last_unstale":"2024-03-06T07:22:59.814736+0000","last_undegraded":"2024-03-06T07:22:59.814736+0000","last_fullsized":"2024-03-06T07:22:59.814736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:37.715070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797040+0000","last_change":"2024-03-06T07:22:59.797040+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797040+0000","last_undegraded":"2024-03-06T07:22:59.797040+0000","last_fullsized":"2024-03-06T07:22:59.797040+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.813596+0000","last_change":"2024-03-06T07:22:59.811226+0000","last_active":"2024-03-06T07:22:59.813596+0000","last_peered":"2024-03-06T07:22:59.813596+0000","last_clean":"2024-03-06T07:22:59.813596+0000","last_became_active":"2024-03-06T07:22:59.810822+0000","last_became_peered":"2024-03-06T07:22:59.810822+0000","last_unstale":"2024-03-06T07:22:59.813596+0000","last_undegraded":"2024-03-06T07:22:59.813596+0000","last_fullsized":"2024-03-06T07:22:59.813596+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:53.894652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.803232+0000","last_change":"2024-03-06T07:22:59.803232+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.803232+0000","last_undegraded":"2024-03-06T07:22:59.803232+0000","last_fullsized":"2024-03-06T07:22:59.803232+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.805549+0000","last_change":"2024-03-06T07:22:59.804853+0000","last_active":"2024-03-06T07:22:59.805549+0000","last_peered":"2024-03-06T07:22:59.805549+0000","last_clean":"2024-03-06T07:22:59.805549+0000","last_became_active":"2024-03-06T07:22:59.804353+0000","last_became_peered":"2024-03-06T07:22:59.804353+0000","last_unstale":"2024-03-06T07:22:59.805549+0000","last_undegraded":"2024-03-06T07:22:59.805549+0000","last_fullsized":"2024-03-06T07:22:59.805549+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:46:08.564158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796177+0000","last_change":"2024-03-06T07:22:59.796177+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796177+0000","last_undegraded":"2024-03-06T07:22:59.796177+0000","last_fullsized":"2024-03-06T07:22:59.796177+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.804694+0000","last_change":"2024-03-06T07:22:59.803415+0000","last_active":"2024-03-06T07:22:59.804694+0000","last_peered":"2024-03-06T07:22:59.804694+0000","last_clean":"2024-03-06T07:22:59.804694+0000","last_became_active":"2024-03-06T07:22:59.803175+0000","last_became_peered":"2024-03-06T07:22:59.803175+0000","last_unstale":"2024-03-06T07:22:59.804694+0000","last_undegraded":"2024-03-06T07:22:59.804694+0000","last_fullsized":"2024-03-06T07:22:59.804694+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:36:12.264660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796946+0000","last_change":"2024-03-06T07:22:59.796946+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796946+0000","last_undegraded":"2024-03-06T07:22:59.796946+0000","last_fullsized":"2024-03-06T07:22:59.796946+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.811396+0000","last_change":"2024-03-06T07:22:59.810599+0000","last_active":"2024-03-06T07:22:59.811396+0000","last_peered":"2024-03-06T07:22:59.811396+0000","last_clean":"2024-03-06T07:22:59.811396+0000","last_became_active":"2024-03-06T07:22:59.810291+0000","last_became_peered":"2024-03-06T07:22:59.810291+0000","last_unstale":"2024-03-06T07:22:59.811396+0000","last_undegraded":"2024-03-06T07:22:59.811396+0000","last_fullsized":"2024-03-06T07:22:59.811396+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:36:12.264660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796176+0000","last_change":"2024-03-06T07:22:59.796176+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796176+0000","last_undegraded":"2024-03-06T07:22:59.796176+0000","last_fullsized":"2024-03-06T07:22:59.796176+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.816470+0000","last_change":"2024-03-06T07:22:59.812044+0000","last_active":"2024-03-06T07:22:59.816470+0000","last_peered":"2024-03-06T07:22:59.816470+0000","last_clean":"2024-03-06T07:22:59.816470+0000","last_became_active":"2024-03-06T07:22:59.811822+0000","last_became_peered":"2024-03-06T07:22:59.811822+0000","last_unstale":"2024-03-06T07:22:59.816470+0000","last_undegraded":"2024-03-06T07:22:59.816470+0000","last_fullsized":"2024-03-06T07:22:59.816470+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:38:31.114570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.798677+0000","last_change":"2024-03-06T07:22:59.798677+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.798677+0000","last_undegraded":"2024-03-06T07:22:59.798677+0000","last_fullsized":"2024-03-06T07:22:59.798677+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.811751+0000","last_change":"2024-03-06T07:22:59.809833+0000","last_active":"2024-03-06T07:22:59.811751+0000","last_peered":"2024-03-06T07:22:59.811751+0000","last_clean":"2024-03-06T07:22:59.811751+0000","last_became_active":"2024-03-06T07:22:59.809550+0000","last_became_peered":"2024-03-06T07:22:59.809550+0000","last_unstale":"2024-03-06T07:22:59.811751+0000","last_undegraded":"2024-03-06T07:22:59.811751+0000","last_fullsized":"2024-03-06T07:22:59.811751+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:40:47.871999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.799865+0000","last_change":"2024-03-06T07:22:59.799865+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.799865+0000","last_undegraded":"2024-03-06T07:22:59.799865+0000","last_fullsized":"2024-03-06T07:22:59.799865+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.816542+0000","last_change":"2024-03-06T07:22:59.812895+0000","last_active":"2024-03-06T07:22:59.816542+0000","last_peered":"2024-03-06T07:22:59.816542+0000","last_clean":"2024-03-06T07:22:59.816542+0000","last_became_active":"2024-03-06T07:22:59.812605+0000","last_became_peered":"2024-03-06T07:22:59.812605+0000","last_unstale":"2024-03-06T07:22:59.816542+0000","last_undegraded":"2024-03-06T07:22:59.816542+0000","last_fullsized":"2024-03-06T07:22:59.816542+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:33.788501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.800675+0000","last_change":"2024-03-06T07:22:59.800675+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.800675+0000","last_undegraded":"2024-03-06T07:22:59.800675+0000","last_fullsized":"2024-03-06T07:22:59.800675+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.809068+0000","last_change":"2024-03-06T07:22:59.808026+0000","last_active":"2024-03-06T07:22:59.809068+0000","last_peered":"2024-03-06T07:22:59.809068+0000","last_clean":"2024-03-06T07:22:59.809068+0000","last_became_active":"2024-03-06T07:22:59.807197+0000","last_became_peered":"2024-03-06T07:22:59.807197+0000","last_unstale":"2024-03-06T07:22:59.809068+0000","last_undegraded":"2024-03-06T07:22:59.809068+0000","last_fullsized":"2024-03-06T07:22:59.809068+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:46:08.564158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":189,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.805202+0000","last_change":"2024-03-06T07:21:47.553693+0000","last_active":"2024-03-06T07:22:59.805202+0000","last_peered":"2024-03-06T07:22:59.805202+0000","last_clean":"2024-03-06T07:22:59.805202+0000","last_became_active":"2024-03-06T07:21:47.552664+0000","last_became_peered":"2024-03-06T07:21:47.552664+0000","last_unstale":"2024-03-06T07:22:59.805202+0000","last_undegraded":"2024-03-06T07:22:59.805202+0000","last_fullsized":"2024-03-06T07:22:59.805202+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_clean_scrub_stamp":"2024-03-06T07:21:46.362028+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:02.916426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343900,"statfs":{"total":96636764160,"available":96608153600,"internally_reserved":0,"allocated":942080,"data_stored":724403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":127768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":134035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":335872,"data_stored":127768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":127768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574867,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":134035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574868,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":134035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574867,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":929792,"data_stored":718136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:23:03.886 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:23:04.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.021+0000 7efd60de9640 1 Processor -- start 2024-03-06T07:23:04.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.022+0000 7efd60de9640 1 -- start start 2024-03-06T07:23:04.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.022+0000 7efd60de9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efd5c0cfff0 0x7efd5c0d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:04.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.022+0000 7efd60de9640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7efd5c0cc8b0 0x7efd5c0ccd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:04.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.022+0000 7efd60de9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd5c161980 con 0x7efd5c15bb20 2024-03-06T07:23:04.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.022+0000 7efd60de9640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd5c0d2500 con 0x7efd5c0cc8b0 2024-03-06T07:23:04.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.022+0000 7efd60de9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd5c161c60 con 0x7efd5c0cfff0 2024-03-06T07:23:04.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.023+0000 7efd5ad9d640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7efd5c0cc8b0 0x7efd5c0ccd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:04.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.023+0000 7efd5a59c640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7efd5c15bb20 0x7efd5c0d1df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:35148/0 (socket says 172.21.15.83:35148) 2024-03-06T07:23:04.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.023+0000 7efd5a59c640 1 -- 172.21.15.83:0/4039957720 learned_addr learned my addr 172.21.15.83:0/4039957720 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:23:04.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.023+0000 7efd53fff640 1 --2- 172.21.15.83:0/4039957720 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efd5c0cfff0 0x7efd5c0d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:04.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.023+0000 7efd5ad9d640 1 -- 172.21.15.83:0/4039957720 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efd5c0cfff0 msgr2=0x7efd5c0d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:04.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.023+0000 7efd5ad9d640 1 --2- 172.21.15.83:0/4039957720 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efd5c0cfff0 0x7efd5c0d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.023+0000 7efd5ad9d640 1 -- 172.21.15.83:0/4039957720 >> v1:172.21.15.183:6789/0 conn(0x7efd5c15bb20 legacy=0x7efd5c0d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:04.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.023+0000 7efd5ad9d640 1 -- 172.21.15.83:0/4039957720 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd5c161f40 con 0x7efd5c0cc8b0 2024-03-06T07:23:04.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.023+0000 7efd53fff640 1 --2- 172.21.15.83:0/4039957720 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efd5c0cfff0 0x7efd5c0d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:04.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.024+0000 7efd5ad9d640 1 --2- 172.21.15.83:0/4039957720 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efd5c0cc8b0 0x7efd5c0ccd40 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7efd54009d90 tx=0x7efd5402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3eacd6ed0ff26cbe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:04.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.024+0000 7efd59d9b640 1 -- 172.21.15.83:0/4039957720 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd54039070 con 0x7efd5c0cc8b0 2024-03-06T07:23:04.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.024+0000 7efd59d9b640 1 -- 172.21.15.83:0/4039957720 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd5402cc50 con 0x7efd5c0cc8b0 2024-03-06T07:23:04.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.024+0000 7efd59d9b640 1 -- 172.21.15.83:0/4039957720 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd54034440 con 0x7efd5c0cc8b0 2024-03-06T07:23:04.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.025+0000 7efd60de9640 1 -- 172.21.15.83:0/4039957720 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efd5c0cc8b0 msgr2=0x7efd5c0ccd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:04.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.025+0000 7efd60de9640 1 --2- 172.21.15.83:0/4039957720 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efd5c0cc8b0 0x7efd5c0ccd40 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7efd54009d90 tx=0x7efd5402bea0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.025+0000 7efd60de9640 1 -- 172.21.15.83:0/4039957720 shutdown_connections 2024-03-06T07:23:04.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.025+0000 7efd60de9640 1 --2- 172.21.15.83:0/4039957720 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efd5c0cc8b0 0x7efd5c0ccd40 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.025+0000 7efd60de9640 1 --2- 172.21.15.83:0/4039957720 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efd5c0cfff0 0x7efd5c0d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.025+0000 7efd60de9640 1 -- 172.21.15.83:0/4039957720 >> 172.21.15.83:0/4039957720 conn(0x7efd5c09a9f0 msgr2=0x7efd5c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:04.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.025+0000 7efd60de9640 1 -- 172.21.15.83:0/4039957720 shutdown_connections 2024-03-06T07:23:04.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.025+0000 7efd60de9640 1 -- 172.21.15.83:0/4039957720 wait complete. 2024-03-06T07:23:04.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.026+0000 7efd60de9640 1 Processor -- start 2024-03-06T07:23:04.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.026+0000 7efd60de9640 1 -- start start 2024-03-06T07:23:04.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.027+0000 7efd60de9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efd5c0cc8b0 0x7efd5c0c8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:04.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.027+0000 7efd60de9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efd5c0cfff0 0x7efd5c1f36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:04.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.027+0000 7efd60de9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efd5c15bb20 0x7efd5c0c6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:04.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.027+0000 7efd60de9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd5c0d1980 con 0x7efd5c15bb20 2024-03-06T07:23:04.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.027+0000 7efd60de9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd5c162500 con 0x7efd5c0cc8b0 2024-03-06T07:23:04.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.027+0000 7efd60de9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd5c0d1b00 con 0x7efd5c0cfff0 2024-03-06T07:23:04.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.027+0000 7efd5ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efd5c15bb20 0x7efd5c0c6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:04.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.027+0000 7efd5ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efd5c15bb20 0x7efd5c0c6fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35368/0 (socket says 172.21.15.83:35368) 2024-03-06T07:23:04.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.027+0000 7efd5ad9d640 1 -- 172.21.15.83:0/1621043717 learned_addr learned my addr 172.21.15.83:0/1621043717 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:04.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.027+0000 7efd53fff640 1 --2- 172.21.15.83:0/1621043717 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efd5c0cfff0 0x7efd5c1f36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:04.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.027+0000 7efd5a59c640 1 --2- 172.21.15.83:0/1621043717 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efd5c0cc8b0 0x7efd5c0c8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:04.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.027+0000 7efd5ad9d640 1 -- 172.21.15.83:0/1621043717 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efd5c0cfff0 msgr2=0x7efd5c1f36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:04.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.027+0000 7efd5ad9d640 1 --2- 172.21.15.83:0/1621043717 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efd5c0cfff0 0x7efd5c1f36a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.027+0000 7efd5ad9d640 1 -- 172.21.15.83:0/1621043717 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efd5c0cc8b0 msgr2=0x7efd5c0c8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:04.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.027+0000 7efd5ad9d640 1 --2- 172.21.15.83:0/1621043717 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efd5c0cc8b0 0x7efd5c0c8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.027+0000 7efd5ad9d640 1 -- 172.21.15.83:0/1621043717 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd5c1e7d50 con 0x7efd5c15bb20 2024-03-06T07:23:04.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.027+0000 7efd53fff640 1 --2- 172.21.15.83:0/1621043717 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efd5c0cfff0 0x7efd5c1f36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:04.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.027+0000 7efd5ad9d640 1 --2- 172.21.15.83:0/1621043717 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efd5c15bb20 0x7efd5c0c6fb0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7efd54009d90 tx=0x7efd54004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:04.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.028+0000 7efd52ffd640 1 -- 172.21.15.83:0/1621043717 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd54039050 con 0x7efd5c15bb20 2024-03-06T07:23:04.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.028+0000 7efd52ffd640 1 -- 172.21.15.83:0/1621043717 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd54045070 con 0x7efd5c15bb20 2024-03-06T07:23:04.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.028+0000 7efd60de9640 1 -- 172.21.15.83:0/1621043717 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd5c1e8030 con 0x7efd5c15bb20 2024-03-06T07:23:04.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.028+0000 7efd52ffd640 1 -- 172.21.15.83:0/1621043717 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5403e610 con 0x7efd5c15bb20 2024-03-06T07:23:04.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.028+0000 7efd60de9640 1 -- 172.21.15.83:0/1621043717 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd5c1e85f0 con 0x7efd5c15bb20 2024-03-06T07:23:04.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.029+0000 7efd60de9640 1 -- 172.21.15.83:0/1621043717 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd5c04f330 con 0x7efd5c15bb20 2024-03-06T07:23:04.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.029+0000 7efd52ffd640 1 -- 172.21.15.83:0/1621043717 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efd5403e7b0 con 0x7efd5c15bb20 2024-03-06T07:23:04.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.029+0000 7efd52ffd640 1 --2- 172.21.15.83:0/1621043717 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efd3c06b4d0 0x7efd3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:04.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.030+0000 7efd52ffd640 1 -- 172.21.15.83:0/1621043717 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efd540acd70 con 0x7efd5c15bb20 2024-03-06T07:23:04.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.030+0000 7efd5a59c640 1 --2- 172.21.15.83:0/1621043717 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efd3c06b4d0 0x7efd3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:04.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.030+0000 7efd5a59c640 1 --2- 172.21.15.83:0/1621043717 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efd3c06b4d0 0x7efd3c06d990 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7efd4c000f60 tx=0x7efd4c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:04.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.038+0000 7efd52ffd640 1 -- 172.21.15.83:0/1621043717 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efd5407bc00 con 0x7efd5c15bb20 2024-03-06T07:23:04.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.209+0000 7efd60de9640 1 -- 172.21.15.83:0/1621043717 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efd5c0d04f0 con 0x7efd3c06b4d0 2024-03-06T07:23:04.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.214+0000 7efd52ffd640 1 -- 172.21.15.83:0/1621043717 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246633 (secure 0 0 0) 0x7efd5c0d04f0 con 0x7efd3c06b4d0 2024-03-06T07:23:04.219 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:23:04.220 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-06T07:23:04.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.217+0000 7efd60de9640 1 -- 172.21.15.83:0/1621043717 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efd3c06b4d0 msgr2=0x7efd3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:04.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.217+0000 7efd60de9640 1 --2- 172.21.15.83:0/1621043717 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efd3c06b4d0 0x7efd3c06d990 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7efd4c000f60 tx=0x7efd4c009040 comp rx=0 tx=0).stop 2024-03-06T07:23:04.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.217+0000 7efd60de9640 1 -- 172.21.15.83:0/1621043717 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efd5c15bb20 msgr2=0x7efd5c0c6fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:04.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.217+0000 7efd60de9640 1 --2- 172.21.15.83:0/1621043717 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efd5c15bb20 0x7efd5c0c6fb0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7efd54009d90 tx=0x7efd54004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:04.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.217+0000 7efd60de9640 1 -- 172.21.15.83:0/1621043717 shutdown_connections 2024-03-06T07:23:04.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.217+0000 7efd60de9640 1 --2- 172.21.15.83:0/1621043717 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efd3c06b4d0 0x7efd3c06d990 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.217+0000 7efd60de9640 1 --2- 172.21.15.83:0/1621043717 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efd5c15bb20 0x7efd5c0c6fb0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.217+0000 7efd60de9640 1 --2- 172.21.15.83:0/1621043717 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efd5c0cfff0 0x7efd5c1f36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.217+0000 7efd60de9640 1 --2- 172.21.15.83:0/1621043717 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efd5c0cc8b0 0x7efd5c0c8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.218+0000 7efd60de9640 1 -- 172.21.15.83:0/1621043717 >> 172.21.15.83:0/1621043717 conn(0x7efd5c09a9f0 msgr2=0x7efd5c09add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:04.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.218+0000 7efd60de9640 1 -- 172.21.15.83:0/1621043717 shutdown_connections 2024-03-06T07:23:04.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.218+0000 7efd60de9640 1 -- 172.21.15.83:0/1621043717 wait complete. 2024-03-06T07:23:04.230 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":65,"stamp":"2024-03-06T07:23:02.333161+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":35,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220000,"kb_used_data":3832,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754720,"statfs":{"total":773094113280,"available":772868833280,"internally_reserved":0,"allocated":3923968,"data_stored":2227948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12757,"internal_metadata":221236779},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000854"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797969+0000","last_change":"2024-03-06T07:22:59.797969+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797969+0000","last_undegraded":"2024-03-06T07:22:59.797969+0000","last_fullsized":"2024-03-06T07:22:59.797969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.799502+0000","last_change":"2024-03-06T07:22:59.799502+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.799502+0000","last_undegraded":"2024-03-06T07:22:59.799502+0000","last_fullsized":"2024-03-06T07:22:59.799502+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.804718+0000","last_change":"2024-03-06T07:22:59.804718+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.804718+0000","last_undegraded":"2024-03-06T07:22:59.804718+0000","last_fullsized":"2024-03-06T07:22:59.804718+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.800525+0000","last_change":"2024-03-06T07:22:59.800525+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.800525+0000","last_undegraded":"2024-03-06T07:22:59.800525+0000","last_fullsized":"2024-03-06T07:22:59.800525+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.798524+0000","last_change":"2024-03-06T07:22:59.798524+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.798524+0000","last_undegraded":"2024-03-06T07:22:59.798524+0000","last_fullsized":"2024-03-06T07:22:59.798524+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797989+0000","last_change":"2024-03-06T07:22:59.797989+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797989+0000","last_undegraded":"2024-03-06T07:22:59.797989+0000","last_fullsized":"2024-03-06T07:22:59.797989+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.801589+0000","last_change":"2024-03-06T07:22:59.801589+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.801589+0000","last_undegraded":"2024-03-06T07:22:59.801589+0000","last_fullsized":"2024-03-06T07:22:59.801589+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.804790+0000","last_change":"2024-03-06T07:22:59.804790+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.804790+0000","last_undegraded":"2024-03-06T07:22:59.804790+0000","last_fullsized":"2024-03-06T07:22:59.804790+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.799988+0000","last_change":"2024-03-06T07:22:59.799988+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.799988+0000","last_undegraded":"2024-03-06T07:22:59.799988+0000","last_fullsized":"2024-03-06T07:22:59.799988+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.799777+0000","last_change":"2024-03-06T07:22:59.799777+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.799777+0000","last_undegraded":"2024-03-06T07:22:59.799777+0000","last_fullsized":"2024-03-06T07:22:59.799777+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.801040+0000","last_change":"2024-03-06T07:22:59.801040+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.801040+0000","last_undegraded":"2024-03-06T07:22:59.801040+0000","last_fullsized":"2024-03-06T07:22:59.801040+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.803466+0000","last_change":"2024-03-06T07:22:59.803466+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.803466+0000","last_undegraded":"2024-03-06T07:22:59.803466+0000","last_fullsized":"2024-03-06T07:22:59.803466+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.798121+0000","last_change":"2024-03-06T07:22:59.798121+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.798121+0000","last_undegraded":"2024-03-06T07:22:59.798121+0000","last_fullsized":"2024-03-06T07:22:59.798121+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797601+0000","last_change":"2024-03-06T07:22:59.797601+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797601+0000","last_undegraded":"2024-03-06T07:22:59.797601+0000","last_fullsized":"2024-03-06T07:22:59.797601+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.800974+0000","last_change":"2024-03-06T07:22:59.800974+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.800974+0000","last_undegraded":"2024-03-06T07:22:59.800974+0000","last_fullsized":"2024-03-06T07:22:59.800974+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.801086+0000","last_change":"2024-03-06T07:22:59.801086+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.801086+0000","last_undegraded":"2024-03-06T07:22:59.801086+0000","last_fullsized":"2024-03-06T07:22:59.801086+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.802385+0000","last_change":"2024-03-06T07:22:59.802385+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.802385+0000","last_undegraded":"2024-03-06T07:22:59.802385+0000","last_fullsized":"2024-03-06T07:22:59.802385+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.799075+0000","last_change":"2024-03-06T07:22:59.799075+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.799075+0000","last_undegraded":"2024-03-06T07:22:59.799075+0000","last_fullsized":"2024-03-06T07:22:59.799075+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797394+0000","last_change":"2024-03-06T07:22:59.797394+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797394+0000","last_undegraded":"2024-03-06T07:22:59.797394+0000","last_fullsized":"2024-03-06T07:22:59.797394+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.798311+0000","last_change":"2024-03-06T07:22:59.798311+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.798311+0000","last_undegraded":"2024-03-06T07:22:59.798311+0000","last_fullsized":"2024-03-06T07:22:59.798311+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.800053+0000","last_change":"2024-03-06T07:22:59.800053+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.800053+0000","last_undegraded":"2024-03-06T07:22:59.800053+0000","last_fullsized":"2024-03-06T07:22:59.800053+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796916+0000","last_change":"2024-03-06T07:22:59.796916+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796916+0000","last_undegraded":"2024-03-06T07:22:59.796916+0000","last_fullsized":"2024-03-06T07:22:59.796916+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796381+0000","last_change":"2024-03-06T07:22:59.796381+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796381+0000","last_undegraded":"2024-03-06T07:22:59.796381+0000","last_fullsized":"2024-03-06T07:22:59.796381+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.806940+0000","last_change":"2024-03-06T07:22:59.806940+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.806940+0000","last_undegraded":"2024-03-06T07:22:59.806940+0000","last_fullsized":"2024-03-06T07:22:59.806940+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.799846+0000","last_change":"2024-03-06T07:22:59.799846+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.799846+0000","last_undegraded":"2024-03-06T07:22:59.799846+0000","last_fullsized":"2024-03-06T07:22:59.799846+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.798684+0000","last_change":"2024-03-06T07:22:59.798684+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.798684+0000","last_undegraded":"2024-03-06T07:22:59.798684+0000","last_fullsized":"2024-03-06T07:22:59.798684+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.800428+0000","last_change":"2024-03-06T07:22:59.800428+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.800428+0000","last_undegraded":"2024-03-06T07:22:59.800428+0000","last_fullsized":"2024-03-06T07:22:59.800428+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.803078+0000","last_change":"2024-03-06T07:22:59.803078+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.803078+0000","last_undegraded":"2024-03-06T07:22:59.803078+0000","last_fullsized":"2024-03-06T07:22:59.803078+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.799594+0000","last_change":"2024-03-06T07:22:59.799594+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.799594+0000","last_undegraded":"2024-03-06T07:22:59.799594+0000","last_fullsized":"2024-03-06T07:22:59.799594+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.801930+0000","last_change":"2024-03-06T07:22:59.801930+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.801930+0000","last_undegraded":"2024-03-06T07:22:59.801930+0000","last_fullsized":"2024-03-06T07:22:59.801930+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797043+0000","last_change":"2024-03-06T07:22:59.797043+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797043+0000","last_undegraded":"2024-03-06T07:22:59.797043+0000","last_fullsized":"2024-03-06T07:22:59.797043+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.804395+0000","last_change":"2024-03-06T07:22:59.804395+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.804395+0000","last_undegraded":"2024-03-06T07:22:59.804395+0000","last_fullsized":"2024-03-06T07:22:59.804395+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796359+0000","last_change":"2024-03-06T07:22:59.796359+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796359+0000","last_undegraded":"2024-03-06T07:22:59.796359+0000","last_fullsized":"2024-03-06T07:22:59.796359+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.812031+0000","last_change":"2024-03-06T07:22:59.810363+0000","last_active":"2024-03-06T07:22:59.812031+0000","last_peered":"2024-03-06T07:22:59.812031+0000","last_clean":"2024-03-06T07:22:59.812031+0000","last_became_active":"2024-03-06T07:22:59.810032+0000","last_became_peered":"2024-03-06T07:22:59.810032+0000","last_unstale":"2024-03-06T07:22:59.812031+0000","last_undegraded":"2024-03-06T07:22:59.812031+0000","last_fullsized":"2024-03-06T07:22:59.812031+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:36:12.264660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797548+0000","last_change":"2024-03-06T07:22:59.797548+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797548+0000","last_undegraded":"2024-03-06T07:22:59.797548+0000","last_fullsized":"2024-03-06T07:22:59.797548+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.811396+0000","last_change":"2024-03-06T07:22:59.810687+0000","last_active":"2024-03-06T07:22:59.811396+0000","last_peered":"2024-03-06T07:22:59.811396+0000","last_clean":"2024-03-06T07:22:59.811396+0000","last_became_active":"2024-03-06T07:22:59.810405+0000","last_became_peered":"2024-03-06T07:22:59.810405+0000","last_unstale":"2024-03-06T07:22:59.811396+0000","last_undegraded":"2024-03-06T07:22:59.811396+0000","last_fullsized":"2024-03-06T07:22:59.811396+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:24:26.919565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.798640+0000","last_change":"2024-03-06T07:22:59.798640+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.798640+0000","last_undegraded":"2024-03-06T07:22:59.798640+0000","last_fullsized":"2024-03-06T07:22:59.798640+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.807686+0000","last_change":"2024-03-06T07:22:59.807093+0000","last_active":"2024-03-06T07:22:59.807686+0000","last_peered":"2024-03-06T07:22:59.807686+0000","last_clean":"2024-03-06T07:22:59.807686+0000","last_became_active":"2024-03-06T07:22:59.806858+0000","last_became_peered":"2024-03-06T07:22:59.806858+0000","last_unstale":"2024-03-06T07:22:59.807686+0000","last_undegraded":"2024-03-06T07:22:59.807686+0000","last_fullsized":"2024-03-06T07:22:59.807686+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:38:31.114570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.799435+0000","last_change":"2024-03-06T07:22:59.799435+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.799435+0000","last_undegraded":"2024-03-06T07:22:59.799435+0000","last_fullsized":"2024-03-06T07:22:59.799435+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.809136+0000","last_change":"2024-03-06T07:22:59.805788+0000","last_active":"2024-03-06T07:22:59.809136+0000","last_peered":"2024-03-06T07:22:59.809136+0000","last_clean":"2024-03-06T07:22:59.809136+0000","last_became_active":"2024-03-06T07:22:59.805463+0000","last_became_peered":"2024-03-06T07:22:59.805463+0000","last_unstale":"2024-03-06T07:22:59.809136+0000","last_undegraded":"2024-03-06T07:22:59.809136+0000","last_fullsized":"2024-03-06T07:22:59.809136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:14.437437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.801363+0000","last_change":"2024-03-06T07:22:59.801363+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.801363+0000","last_undegraded":"2024-03-06T07:22:59.801363+0000","last_fullsized":"2024-03-06T07:22:59.801363+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.811699+0000","last_change":"2024-03-06T07:22:59.808162+0000","last_active":"2024-03-06T07:22:59.811699+0000","last_peered":"2024-03-06T07:22:59.811699+0000","last_clean":"2024-03-06T07:22:59.811699+0000","last_became_active":"2024-03-06T07:22:59.807875+0000","last_became_peered":"2024-03-06T07:22:59.807875+0000","last_unstale":"2024-03-06T07:22:59.811699+0000","last_undegraded":"2024-03-06T07:22:59.811699+0000","last_fullsized":"2024-03-06T07:22:59.811699+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:53.014712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.802462+0000","last_change":"2024-03-06T07:22:59.802462+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.802462+0000","last_undegraded":"2024-03-06T07:22:59.802462+0000","last_fullsized":"2024-03-06T07:22:59.802462+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.811787+0000","last_change":"2024-03-06T07:22:59.806399+0000","last_active":"2024-03-06T07:22:59.811787+0000","last_peered":"2024-03-06T07:22:59.811787+0000","last_clean":"2024-03-06T07:22:59.811787+0000","last_became_active":"2024-03-06T07:22:59.805990+0000","last_became_peered":"2024-03-06T07:22:59.805990+0000","last_unstale":"2024-03-06T07:22:59.811787+0000","last_undegraded":"2024-03-06T07:22:59.811787+0000","last_fullsized":"2024-03-06T07:22:59.811787+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:07.271082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797495+0000","last_change":"2024-03-06T07:22:59.797495+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797495+0000","last_undegraded":"2024-03-06T07:22:59.797495+0000","last_fullsized":"2024-03-06T07:22:59.797495+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.806357+0000","last_change":"2024-03-06T07:22:59.805251+0000","last_active":"2024-03-06T07:22:59.806357+0000","last_peered":"2024-03-06T07:22:59.806357+0000","last_clean":"2024-03-06T07:22:59.806357+0000","last_became_active":"2024-03-06T07:22:59.804769+0000","last_became_peered":"2024-03-06T07:22:59.804769+0000","last_unstale":"2024-03-06T07:22:59.806357+0000","last_undegraded":"2024-03-06T07:22:59.806357+0000","last_fullsized":"2024-03-06T07:22:59.806357+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:30:21.970435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797342+0000","last_change":"2024-03-06T07:22:59.797342+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797342+0000","last_undegraded":"2024-03-06T07:22:59.797342+0000","last_fullsized":"2024-03-06T07:22:59.797342+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.810668+0000","last_change":"2024-03-06T07:22:59.804481+0000","last_active":"2024-03-06T07:22:59.810668+0000","last_peered":"2024-03-06T07:22:59.810668+0000","last_clean":"2024-03-06T07:22:59.810668+0000","last_became_active":"2024-03-06T07:22:59.803413+0000","last_became_peered":"2024-03-06T07:22:59.803413+0000","last_unstale":"2024-03-06T07:22:59.810668+0000","last_undegraded":"2024-03-06T07:22:59.810668+0000","last_fullsized":"2024-03-06T07:22:59.810668+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:14.437437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.802007+0000","last_change":"2024-03-06T07:22:59.802007+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.802007+0000","last_undegraded":"2024-03-06T07:22:59.802007+0000","last_fullsized":"2024-03-06T07:22:59.802007+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.810221+0000","last_change":"2024-03-06T07:22:59.808659+0000","last_active":"2024-03-06T07:22:59.810221+0000","last_peered":"2024-03-06T07:22:59.810221+0000","last_clean":"2024-03-06T07:22:59.810221+0000","last_became_active":"2024-03-06T07:22:59.808416+0000","last_became_peered":"2024-03-06T07:22:59.808416+0000","last_unstale":"2024-03-06T07:22:59.810221+0000","last_undegraded":"2024-03-06T07:22:59.810221+0000","last_fullsized":"2024-03-06T07:22:59.810221+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:32.447994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.798133+0000","last_change":"2024-03-06T07:22:59.798133+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.798133+0000","last_undegraded":"2024-03-06T07:22:59.798133+0000","last_fullsized":"2024-03-06T07:22:59.798133+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.805530+0000","last_change":"2024-03-06T07:22:59.804638+0000","last_active":"2024-03-06T07:22:59.805530+0000","last_peered":"2024-03-06T07:22:59.805530+0000","last_clean":"2024-03-06T07:22:59.805530+0000","last_became_active":"2024-03-06T07:22:59.804026+0000","last_became_peered":"2024-03-06T07:22:59.804026+0000","last_unstale":"2024-03-06T07:22:59.805530+0000","last_undegraded":"2024-03-06T07:22:59.805530+0000","last_fullsized":"2024-03-06T07:22:59.805530+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:04:10.795313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.798485+0000","last_change":"2024-03-06T07:22:59.798485+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.798485+0000","last_undegraded":"2024-03-06T07:22:59.798485+0000","last_fullsized":"2024-03-06T07:22:59.798485+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.816328+0000","last_change":"2024-03-06T07:22:59.815296+0000","last_active":"2024-03-06T07:22:59.816328+0000","last_peered":"2024-03-06T07:22:59.816328+0000","last_clean":"2024-03-06T07:22:59.816328+0000","last_became_active":"2024-03-06T07:22:59.815033+0000","last_became_peered":"2024-03-06T07:22:59.815033+0000","last_unstale":"2024-03-06T07:22:59.816328+0000","last_undegraded":"2024-03-06T07:22:59.816328+0000","last_fullsized":"2024-03-06T07:22:59.816328+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:29.015317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.808497+0000","last_change":"2024-03-06T07:22:59.808497+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.808497+0000","last_undegraded":"2024-03-06T07:22:59.808497+0000","last_fullsized":"2024-03-06T07:22:59.808497+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.808772+0000","last_change":"2024-03-06T07:22:59.807399+0000","last_active":"2024-03-06T07:22:59.808772+0000","last_peered":"2024-03-06T07:22:59.808772+0000","last_clean":"2024-03-06T07:22:59.808772+0000","last_became_active":"2024-03-06T07:22:59.807150+0000","last_became_peered":"2024-03-06T07:22:59.807150+0000","last_unstale":"2024-03-06T07:22:59.808772+0000","last_undegraded":"2024-03-06T07:22:59.808772+0000","last_fullsized":"2024-03-06T07:22:59.808772+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:19.337313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.798310+0000","last_change":"2024-03-06T07:22:59.798310+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.798310+0000","last_undegraded":"2024-03-06T07:22:59.798310+0000","last_fullsized":"2024-03-06T07:22:59.798310+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.811735+0000","last_change":"2024-03-06T07:22:59.809344+0000","last_active":"2024-03-06T07:22:59.811735+0000","last_peered":"2024-03-06T07:22:59.811735+0000","last_clean":"2024-03-06T07:22:59.811735+0000","last_became_active":"2024-03-06T07:22:59.809076+0000","last_became_peered":"2024-03-06T07:22:59.809076+0000","last_unstale":"2024-03-06T07:22:59.811735+0000","last_undegraded":"2024-03-06T07:22:59.811735+0000","last_fullsized":"2024-03-06T07:22:59.811735+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:12.683272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796615+0000","last_change":"2024-03-06T07:22:59.796615+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796615+0000","last_undegraded":"2024-03-06T07:22:59.796615+0000","last_fullsized":"2024-03-06T07:22:59.796615+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.809522+0000","last_change":"2024-03-06T07:22:59.807007+0000","last_active":"2024-03-06T07:22:59.809522+0000","last_peered":"2024-03-06T07:22:59.809522+0000","last_clean":"2024-03-06T07:22:59.809522+0000","last_became_active":"2024-03-06T07:22:59.806750+0000","last_became_peered":"2024-03-06T07:22:59.806750+0000","last_unstale":"2024-03-06T07:22:59.809522+0000","last_undegraded":"2024-03-06T07:22:59.809522+0000","last_fullsized":"2024-03-06T07:22:59.809522+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:24:26.919565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.804026+0000","last_change":"2024-03-06T07:22:59.804026+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.804026+0000","last_undegraded":"2024-03-06T07:22:59.804026+0000","last_fullsized":"2024-03-06T07:22:59.804026+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.809695+0000","last_change":"2024-03-06T07:22:59.808043+0000","last_active":"2024-03-06T07:22:59.809695+0000","last_peered":"2024-03-06T07:22:59.809695+0000","last_clean":"2024-03-06T07:22:59.809695+0000","last_became_active":"2024-03-06T07:22:59.807794+0000","last_became_peered":"2024-03-06T07:22:59.807794+0000","last_unstale":"2024-03-06T07:22:59.809695+0000","last_undegraded":"2024-03-06T07:22:59.809695+0000","last_fullsized":"2024-03-06T07:22:59.809695+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:08:18.005082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796469+0000","last_change":"2024-03-06T07:22:59.796469+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796469+0000","last_undegraded":"2024-03-06T07:22:59.796469+0000","last_fullsized":"2024-03-06T07:22:59.796469+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.807259+0000","last_change":"2024-03-06T07:22:59.806459+0000","last_active":"2024-03-06T07:22:59.807259+0000","last_peered":"2024-03-06T07:22:59.807259+0000","last_clean":"2024-03-06T07:22:59.807259+0000","last_became_active":"2024-03-06T07:22:59.806194+0000","last_became_peered":"2024-03-06T07:22:59.806194+0000","last_unstale":"2024-03-06T07:22:59.807259+0000","last_undegraded":"2024-03-06T07:22:59.807259+0000","last_fullsized":"2024-03-06T07:22:59.807259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:53.894652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796668+0000","last_change":"2024-03-06T07:22:59.796668+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796668+0000","last_undegraded":"2024-03-06T07:22:59.796668+0000","last_fullsized":"2024-03-06T07:22:59.796668+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.810182+0000","last_change":"2024-03-06T07:22:59.808555+0000","last_active":"2024-03-06T07:22:59.810182+0000","last_peered":"2024-03-06T07:22:59.810182+0000","last_clean":"2024-03-06T07:22:59.810182+0000","last_became_active":"2024-03-06T07:22:59.808284+0000","last_became_peered":"2024-03-06T07:22:59.808284+0000","last_unstale":"2024-03-06T07:22:59.810182+0000","last_undegraded":"2024-03-06T07:22:59.810182+0000","last_fullsized":"2024-03-06T07:22:59.810182+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:30:21.970435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.803648+0000","last_change":"2024-03-06T07:22:59.803648+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.803648+0000","last_undegraded":"2024-03-06T07:22:59.803648+0000","last_fullsized":"2024-03-06T07:22:59.803648+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.805500+0000","last_change":"2024-03-06T07:22:59.802730+0000","last_active":"2024-03-06T07:22:59.805500+0000","last_peered":"2024-03-06T07:22:59.805500+0000","last_clean":"2024-03-06T07:22:59.805500+0000","last_became_active":"2024-03-06T07:22:59.802461+0000","last_became_peered":"2024-03-06T07:22:59.802461+0000","last_unstale":"2024-03-06T07:22:59.805500+0000","last_undegraded":"2024-03-06T07:22:59.805500+0000","last_fullsized":"2024-03-06T07:22:59.805500+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:26:21.615251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.798752+0000","last_change":"2024-03-06T07:22:59.798752+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.798752+0000","last_undegraded":"2024-03-06T07:22:59.798752+0000","last_fullsized":"2024-03-06T07:22:59.798752+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.811762+0000","last_change":"2024-03-06T07:22:59.807016+0000","last_active":"2024-03-06T07:22:59.811762+0000","last_peered":"2024-03-06T07:22:59.811762+0000","last_clean":"2024-03-06T07:22:59.811762+0000","last_became_active":"2024-03-06T07:22:59.806733+0000","last_became_peered":"2024-03-06T07:22:59.806733+0000","last_unstale":"2024-03-06T07:22:59.811762+0000","last_undegraded":"2024-03-06T07:22:59.811762+0000","last_fullsized":"2024-03-06T07:22:59.811762+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:55.870572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797444+0000","last_change":"2024-03-06T07:22:59.797444+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797444+0000","last_undegraded":"2024-03-06T07:22:59.797444+0000","last_fullsized":"2024-03-06T07:22:59.797444+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.811731+0000","last_change":"2024-03-06T07:22:59.808721+0000","last_active":"2024-03-06T07:22:59.811731+0000","last_peered":"2024-03-06T07:22:59.811731+0000","last_clean":"2024-03-06T07:22:59.811731+0000","last_became_active":"2024-03-06T07:22:59.808465+0000","last_became_peered":"2024-03-06T07:22:59.808465+0000","last_unstale":"2024-03-06T07:22:59.811731+0000","last_undegraded":"2024-03-06T07:22:59.811731+0000","last_fullsized":"2024-03-06T07:22:59.811731+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:08.896012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796675+0000","last_change":"2024-03-06T07:22:59.796675+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796675+0000","last_undegraded":"2024-03-06T07:22:59.796675+0000","last_fullsized":"2024-03-06T07:22:59.796675+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.805543+0000","last_change":"2024-03-06T07:22:59.804367+0000","last_active":"2024-03-06T07:22:59.805543+0000","last_peered":"2024-03-06T07:22:59.805543+0000","last_clean":"2024-03-06T07:22:59.805543+0000","last_became_active":"2024-03-06T07:22:59.804115+0000","last_became_peered":"2024-03-06T07:22:59.804115+0000","last_unstale":"2024-03-06T07:22:59.805543+0000","last_undegraded":"2024-03-06T07:22:59.805543+0000","last_fullsized":"2024-03-06T07:22:59.805543+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:44:35.664305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797737+0000","last_change":"2024-03-06T07:22:59.797737+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797737+0000","last_undegraded":"2024-03-06T07:22:59.797737+0000","last_fullsized":"2024-03-06T07:22:59.797737+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.806310+0000","last_change":"2024-03-06T07:22:59.804749+0000","last_active":"2024-03-06T07:22:59.806310+0000","last_peered":"2024-03-06T07:22:59.806310+0000","last_clean":"2024-03-06T07:22:59.806310+0000","last_became_active":"2024-03-06T07:22:59.802839+0000","last_became_peered":"2024-03-06T07:22:59.802839+0000","last_unstale":"2024-03-06T07:22:59.806310+0000","last_undegraded":"2024-03-06T07:22:59.806310+0000","last_fullsized":"2024-03-06T07:22:59.806310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:04:10.795313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796191+0000","last_change":"2024-03-06T07:22:59.796191+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796191+0000","last_undegraded":"2024-03-06T07:22:59.796191+0000","last_fullsized":"2024-03-06T07:22:59.796191+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.813816+0000","last_change":"2024-03-06T07:22:59.810390+0000","last_active":"2024-03-06T07:22:59.813816+0000","last_peered":"2024-03-06T07:22:59.813816+0000","last_clean":"2024-03-06T07:22:59.813816+0000","last_became_active":"2024-03-06T07:22:59.810140+0000","last_became_peered":"2024-03-06T07:22:59.810140+0000","last_unstale":"2024-03-06T07:22:59.813816+0000","last_undegraded":"2024-03-06T07:22:59.813816+0000","last_fullsized":"2024-03-06T07:22:59.813816+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:13.131161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.802019+0000","last_change":"2024-03-06T07:22:59.802019+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.802019+0000","last_undegraded":"2024-03-06T07:22:59.802019+0000","last_fullsized":"2024-03-06T07:22:59.802019+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.814736+0000","last_change":"2024-03-06T07:22:59.812896+0000","last_active":"2024-03-06T07:22:59.814736+0000","last_peered":"2024-03-06T07:22:59.814736+0000","last_clean":"2024-03-06T07:22:59.814736+0000","last_became_active":"2024-03-06T07:22:59.812598+0000","last_became_peered":"2024-03-06T07:22:59.812598+0000","last_unstale":"2024-03-06T07:22:59.814736+0000","last_undegraded":"2024-03-06T07:22:59.814736+0000","last_fullsized":"2024-03-06T07:22:59.814736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:37.715070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797040+0000","last_change":"2024-03-06T07:22:59.797040+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797040+0000","last_undegraded":"2024-03-06T07:22:59.797040+0000","last_fullsized":"2024-03-06T07:22:59.797040+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.813596+0000","last_change":"2024-03-06T07:22:59.811226+0000","last_active":"2024-03-06T07:22:59.813596+0000","last_peered":"2024-03-06T07:22:59.813596+0000","last_clean":"2024-03-06T07:22:59.813596+0000","last_became_active":"2024-03-06T07:22:59.810822+0000","last_became_peered":"2024-03-06T07:22:59.810822+0000","last_unstale":"2024-03-06T07:22:59.813596+0000","last_undegraded":"2024-03-06T07:22:59.813596+0000","last_fullsized":"2024-03-06T07:22:59.813596+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:53.894652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.803232+0000","last_change":"2024-03-06T07:22:59.803232+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.803232+0000","last_undegraded":"2024-03-06T07:22:59.803232+0000","last_fullsized":"2024-03-06T07:22:59.803232+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.805549+0000","last_change":"2024-03-06T07:22:59.804853+0000","last_active":"2024-03-06T07:22:59.805549+0000","last_peered":"2024-03-06T07:22:59.805549+0000","last_clean":"2024-03-06T07:22:59.805549+0000","last_became_active":"2024-03-06T07:22:59.804353+0000","last_became_peered":"2024-03-06T07:22:59.804353+0000","last_unstale":"2024-03-06T07:22:59.805549+0000","last_undegraded":"2024-03-06T07:22:59.805549+0000","last_fullsized":"2024-03-06T07:22:59.805549+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:46:08.564158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796177+0000","last_change":"2024-03-06T07:22:59.796177+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796177+0000","last_undegraded":"2024-03-06T07:22:59.796177+0000","last_fullsized":"2024-03-06T07:22:59.796177+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.804694+0000","last_change":"2024-03-06T07:22:59.803415+0000","last_active":"2024-03-06T07:22:59.804694+0000","last_peered":"2024-03-06T07:22:59.804694+0000","last_clean":"2024-03-06T07:22:59.804694+0000","last_became_active":"2024-03-06T07:22:59.803175+0000","last_became_peered":"2024-03-06T07:22:59.803175+0000","last_unstale":"2024-03-06T07:22:59.804694+0000","last_undegraded":"2024-03-06T07:22:59.804694+0000","last_fullsized":"2024-03-06T07:22:59.804694+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:36:12.264660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796946+0000","last_change":"2024-03-06T07:22:59.796946+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796946+0000","last_undegraded":"2024-03-06T07:22:59.796946+0000","last_fullsized":"2024-03-06T07:22:59.796946+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.811396+0000","last_change":"2024-03-06T07:22:59.810599+0000","last_active":"2024-03-06T07:22:59.811396+0000","last_peered":"2024-03-06T07:22:59.811396+0000","last_clean":"2024-03-06T07:22:59.811396+0000","last_became_active":"2024-03-06T07:22:59.810291+0000","last_became_peered":"2024-03-06T07:22:59.810291+0000","last_unstale":"2024-03-06T07:22:59.811396+0000","last_undegraded":"2024-03-06T07:22:59.811396+0000","last_fullsized":"2024-03-06T07:22:59.811396+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:36:12.264660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796176+0000","last_change":"2024-03-06T07:22:59.796176+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796176+0000","last_undegraded":"2024-03-06T07:22:59.796176+0000","last_fullsized":"2024-03-06T07:22:59.796176+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.816470+0000","last_change":"2024-03-06T07:22:59.812044+0000","last_active":"2024-03-06T07:22:59.816470+0000","last_peered":"2024-03-06T07:22:59.816470+0000","last_clean":"2024-03-06T07:22:59.816470+0000","last_became_active":"2024-03-06T07:22:59.811822+0000","last_became_peered":"2024-03-06T07:22:59.811822+0000","last_unstale":"2024-03-06T07:22:59.816470+0000","last_undegraded":"2024-03-06T07:22:59.816470+0000","last_fullsized":"2024-03-06T07:22:59.816470+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:38:31.114570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.798677+0000","last_change":"2024-03-06T07:22:59.798677+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.798677+0000","last_undegraded":"2024-03-06T07:22:59.798677+0000","last_fullsized":"2024-03-06T07:22:59.798677+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.811751+0000","last_change":"2024-03-06T07:22:59.809833+0000","last_active":"2024-03-06T07:22:59.811751+0000","last_peered":"2024-03-06T07:22:59.811751+0000","last_clean":"2024-03-06T07:22:59.811751+0000","last_became_active":"2024-03-06T07:22:59.809550+0000","last_became_peered":"2024-03-06T07:22:59.809550+0000","last_unstale":"2024-03-06T07:22:59.811751+0000","last_undegraded":"2024-03-06T07:22:59.811751+0000","last_fullsized":"2024-03-06T07:22:59.811751+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:40:47.871999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.799865+0000","last_change":"2024-03-06T07:22:59.799865+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.799865+0000","last_undegraded":"2024-03-06T07:22:59.799865+0000","last_fullsized":"2024-03-06T07:22:59.799865+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.816542+0000","last_change":"2024-03-06T07:22:59.812895+0000","last_active":"2024-03-06T07:22:59.816542+0000","last_peered":"2024-03-06T07:22:59.816542+0000","last_clean":"2024-03-06T07:22:59.816542+0000","last_became_active":"2024-03-06T07:22:59.812605+0000","last_became_peered":"2024-03-06T07:22:59.812605+0000","last_unstale":"2024-03-06T07:22:59.816542+0000","last_undegraded":"2024-03-06T07:22:59.816542+0000","last_fullsized":"2024-03-06T07:22:59.816542+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:33.788501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.800675+0000","last_change":"2024-03-06T07:22:59.800675+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.800675+0000","last_undegraded":"2024-03-06T07:22:59.800675+0000","last_fullsized":"2024-03-06T07:22:59.800675+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.809068+0000","last_change":"2024-03-06T07:22:59.808026+0000","last_active":"2024-03-06T07:22:59.809068+0000","last_peered":"2024-03-06T07:22:59.809068+0000","last_clean":"2024-03-06T07:22:59.809068+0000","last_became_active":"2024-03-06T07:22:59.807197+0000","last_became_peered":"2024-03-06T07:22:59.807197+0000","last_unstale":"2024-03-06T07:22:59.809068+0000","last_undegraded":"2024-03-06T07:22:59.809068+0000","last_fullsized":"2024-03-06T07:22:59.809068+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:46:08.564158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":189,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.805202+0000","last_change":"2024-03-06T07:21:47.553693+0000","last_active":"2024-03-06T07:22:59.805202+0000","last_peered":"2024-03-06T07:22:59.805202+0000","last_clean":"2024-03-06T07:22:59.805202+0000","last_became_active":"2024-03-06T07:21:47.552664+0000","last_became_peered":"2024-03-06T07:21:47.552664+0000","last_unstale":"2024-03-06T07:22:59.805202+0000","last_undegraded":"2024-03-06T07:22:59.805202+0000","last_fullsized":"2024-03-06T07:22:59.805202+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_clean_scrub_stamp":"2024-03-06T07:21:46.362028+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:02.916426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343900,"statfs":{"total":96636764160,"available":96608153600,"internally_reserved":0,"allocated":942080,"data_stored":724403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":127768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":134035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":335872,"data_stored":127768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":127768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574867,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":134035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574868,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":134035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574867,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":929792,"data_stored":718136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:23:04.235 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:23:04.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.371+0000 7ff2bf95e640 1 Processor -- start 2024-03-06T07:23:04.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.371+0000 7ff2bf95e640 1 -- start start 2024-03-06T07:23:04.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.372+0000 7ff2bf95e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff2b80d0a80 0x7ff2b80b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:04.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.372+0000 7ff2bf95e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2b80d25d0 con 0x7ff2b80b4170 2024-03-06T07:23:04.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.372+0000 7ff2bf95e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2b80d2870 con 0x7ff2b80d4760 2024-03-06T07:23:04.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.372+0000 7ff2bf95e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2b80d4bf0 con 0x7ff2b80d0a80 2024-03-06T07:23:04.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.372+0000 7ff2bd6fa640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff2b80b4170 0x7ff2b80d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:50912/0 (socket says 172.21.15.83:50912) 2024-03-06T07:23:04.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.372+0000 7ff2bd6fa640 1 -- 172.21.15.83:0/2982344670 learned_addr learned my addr 172.21.15.83:0/2982344670 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:23:04.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.372+0000 7ff2bdefb640 1 --2- 172.21.15.83:0/2982344670 >> v2:172.21.15.83:3300/0 conn(0x7ff2b80d0a80 0x7ff2b80b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:04.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.373+0000 7ff2affff640 1 -- 172.21.15.83:0/2982344670 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2096073875 0 0) 0x7ff2b80d25d0 con 0x7ff2b80b4170 2024-03-06T07:23:04.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.373+0000 7ff2affff640 1 -- 172.21.15.83:0/2982344670 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2a4003520 con 0x7ff2b80b4170 2024-03-06T07:23:04.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.373+0000 7ff2bdefb640 1 -- 172.21.15.83:0/2982344670 >> v1:172.21.15.183:6789/0 conn(0x7ff2b80d4760 legacy=0x7ff2b80cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:04.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.373+0000 7ff2bdefb640 1 -- 172.21.15.83:0/2982344670 >> v1:172.21.15.83:6789/0 conn(0x7ff2b80b4170 legacy=0x7ff2b80d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:04.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.373+0000 7ff2bdefb640 1 -- 172.21.15.83:0/2982344670 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2b80d2b50 con 0x7ff2b80d0a80 2024-03-06T07:23:04.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.373+0000 7ff2bdefb640 1 --2- 172.21.15.83:0/2982344670 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2b80d0a80 0x7ff2b80b6a90 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7ff2b4009d90 tx=0x7ff2b402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5546bb2cd83b8014 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:04.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.374+0000 7ff2affff640 1 -- 172.21.15.83:0/2982344670 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b4039070 con 0x7ff2b80d0a80 2024-03-06T07:23:04.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.374+0000 7ff2affff640 1 -- 172.21.15.83:0/2982344670 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2b402cc60 con 0x7ff2b80d0a80 2024-03-06T07:23:04.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.374+0000 7ff2affff640 1 -- 172.21.15.83:0/2982344670 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b4034660 con 0x7ff2b80d0a80 2024-03-06T07:23:04.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.374+0000 7ff2bf95e640 1 -- 172.21.15.83:0/2982344670 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2b80d0a80 msgr2=0x7ff2b80b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:04.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.374+0000 7ff2bf95e640 1 --2- 172.21.15.83:0/2982344670 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2b80d0a80 0x7ff2b80b6a90 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7ff2b4009d90 tx=0x7ff2b402beb0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.374+0000 7ff2bf95e640 1 -- 172.21.15.83:0/2982344670 shutdown_connections 2024-03-06T07:23:04.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.374+0000 7ff2bf95e640 1 --2- 172.21.15.83:0/2982344670 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2b80d0a80 0x7ff2b80b6a90 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.374+0000 7ff2bf95e640 1 -- 172.21.15.83:0/2982344670 >> 172.21.15.83:0/2982344670 conn(0x7ff2b80d3c40 msgr2=0x7ff2b80d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:04.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.374+0000 7ff2bf95e640 1 -- 172.21.15.83:0/2982344670 shutdown_connections 2024-03-06T07:23:04.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.374+0000 7ff2bf95e640 1 -- 172.21.15.83:0/2982344670 wait complete. 2024-03-06T07:23:04.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.375+0000 7ff2bf95e640 1 Processor -- start 2024-03-06T07:23:04.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.375+0000 7ff2bf95e640 1 -- start start 2024-03-06T07:23:04.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.376+0000 7ff2bf95e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff2b80b4170 0x7ff2b80caf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:04.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.376+0000 7ff2bf95e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff2b80d0a80 0x7ff2b80b3070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:04.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.376+0000 7ff2bf95e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2b80cc760 0x7ff2b815f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:04.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.376+0000 7ff2bf95e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b80b4dd0 con 0x7ff2b80cc760 2024-03-06T07:23:04.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.376+0000 7ff2bf95e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b80ce9e0 con 0x7ff2b80b4170 2024-03-06T07:23:04.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.376+0000 7ff2bf95e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b80b4f50 con 0x7ff2b80d0a80 2024-03-06T07:23:04.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.376+0000 7ff2bdefb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2b80cc760 0x7ff2b815f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:04.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.376+0000 7ff2bdefb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2b80cc760 0x7ff2b815f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35396/0 (socket says 172.21.15.83:35396) 2024-03-06T07:23:04.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.376+0000 7ff2bdefb640 1 -- 172.21.15.83:0/2014146859 learned_addr learned my addr 172.21.15.83:0/2014146859 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:04.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.376+0000 7ff2bdefb640 1 -- 172.21.15.83:0/2014146859 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff2b80d0a80 msgr2=0x7ff2b80b3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:04.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.376+0000 7ff2bdefb640 1 --2- 172.21.15.83:0/2014146859 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff2b80d0a80 0x7ff2b80b3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.376+0000 7ff2bdefb640 1 -- 172.21.15.83:0/2014146859 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff2b80b4170 msgr2=0x7ff2b80caf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:04.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.376+0000 7ff2bdefb640 1 --2- 172.21.15.83:0/2014146859 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff2b80b4170 0x7ff2b80caf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.376+0000 7ff2bdefb640 1 -- 172.21.15.83:0/2014146859 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2b80cb6c0 con 0x7ff2b80cc760 2024-03-06T07:23:04.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.377+0000 7ff2bdefb640 1 --2- 172.21.15.83:0/2014146859 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2b80cc760 0x7ff2b815f3c0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7ff2b4009d90 tx=0x7ff2b4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:04.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.377+0000 7ff2adffb640 1 -- 172.21.15.83:0/2014146859 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b4044020 con 0x7ff2b80cc760 2024-03-06T07:23:04.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.377+0000 7ff2adffb640 1 -- 172.21.15.83:0/2014146859 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2b403f470 con 0x7ff2b80cc760 2024-03-06T07:23:04.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.377+0000 7ff2adffb640 1 -- 172.21.15.83:0/2014146859 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b4039040 con 0x7ff2b80cc760 2024-03-06T07:23:04.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.377+0000 7ff2bf95e640 1 -- 172.21.15.83:0/2014146859 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2b80cb9a0 con 0x7ff2b80cc760 2024-03-06T07:23:04.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.377+0000 7ff2bf95e640 1 -- 172.21.15.83:0/2014146859 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2b80cbf60 con 0x7ff2b80cc760 2024-03-06T07:23:04.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.378+0000 7ff2bf95e640 1 -- 172.21.15.83:0/2014146859 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2b804f330 con 0x7ff2b80cc760 2024-03-06T07:23:04.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.378+0000 7ff2adffb640 1 -- 172.21.15.83:0/2014146859 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff2b4047020 con 0x7ff2b80cc760 2024-03-06T07:23:04.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.379+0000 7ff2adffb640 1 --2- 172.21.15.83:0/2014146859 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff29c06b4d0 0x7ff29c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:04.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.379+0000 7ff2bd6fa640 1 --2- 172.21.15.83:0/2014146859 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff29c06b4d0 0x7ff29c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:04.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.379+0000 7ff2adffb640 1 -- 172.21.15.83:0/2014146859 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff2b40acd80 con 0x7ff2b80cc760 2024-03-06T07:23:04.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.379+0000 7ff2bd6fa640 1 --2- 172.21.15.83:0/2014146859 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff29c06b4d0 0x7ff29c06d990 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff2b0000f60 tx=0x7ff2b00078f0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:04.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.383+0000 7ff2adffb640 1 -- 172.21.15.83:0/2014146859 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff2b404e030 con 0x7ff2b80cc760 2024-03-06T07:23:04.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.712+0000 7ff2bf95e640 1 -- 172.21.15.83:0/2014146859 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff2b80733b0 con 0x7ff29c06b4d0 2024-03-06T07:23:04.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.718+0000 7ff2adffb640 1 -- 172.21.15.83:0/2014146859 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246633 (secure 0 0 0) 0x7ff2b80733b0 con 0x7ff29c06b4d0 2024-03-06T07:23:04.722 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:23:04.723 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-06T07:23:04.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.721+0000 7ff2bf95e640 1 -- 172.21.15.83:0/2014146859 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff29c06b4d0 msgr2=0x7ff29c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:04.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.721+0000 7ff2bf95e640 1 --2- 172.21.15.83:0/2014146859 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff29c06b4d0 0x7ff29c06d990 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff2b0000f60 tx=0x7ff2b00078f0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.721+0000 7ff2bf95e640 1 -- 172.21.15.83:0/2014146859 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2b80cc760 msgr2=0x7ff2b815f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:04.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.721+0000 7ff2bf95e640 1 --2- 172.21.15.83:0/2014146859 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2b80cc760 0x7ff2b815f3c0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7ff2b4009d90 tx=0x7ff2b4004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:04.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.721+0000 7ff2bf95e640 1 -- 172.21.15.83:0/2014146859 shutdown_connections 2024-03-06T07:23:04.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.721+0000 7ff2bf95e640 1 --2- 172.21.15.83:0/2014146859 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff29c06b4d0 0x7ff29c06d990 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.721+0000 7ff2bf95e640 1 --2- 172.21.15.83:0/2014146859 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2b80cc760 0x7ff2b815f3c0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.721+0000 7ff2bf95e640 1 --2- 172.21.15.83:0/2014146859 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff2b80d0a80 0x7ff2b80b3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.721+0000 7ff2bf95e640 1 --2- 172.21.15.83:0/2014146859 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff2b80b4170 0x7ff2b80caf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.721+0000 7ff2bf95e640 1 -- 172.21.15.83:0/2014146859 >> 172.21.15.83:0/2014146859 conn(0x7ff2b80d3c40 msgr2=0x7ff2b80b8ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:04.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.721+0000 7ff2bf95e640 1 -- 172.21.15.83:0/2014146859 shutdown_connections 2024-03-06T07:23:04.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.721+0000 7ff2bf95e640 1 -- 172.21.15.83:0/2014146859 wait complete. 2024-03-06T07:23:04.737 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":66,"stamp":"2024-03-06T07:23:04.333421+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":35,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220000,"kb_used_data":3832,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754720,"statfs":{"total":773094113280,"available":772868833280,"internally_reserved":0,"allocated":3923968,"data_stored":2227948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12757,"internal_metadata":221236779},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.543580"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797969+0000","last_change":"2024-03-06T07:22:59.797969+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797969+0000","last_undegraded":"2024-03-06T07:22:59.797969+0000","last_fullsized":"2024-03-06T07:22:59.797969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.799502+0000","last_change":"2024-03-06T07:22:59.799502+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.799502+0000","last_undegraded":"2024-03-06T07:22:59.799502+0000","last_fullsized":"2024-03-06T07:22:59.799502+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.804718+0000","last_change":"2024-03-06T07:22:59.804718+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.804718+0000","last_undegraded":"2024-03-06T07:22:59.804718+0000","last_fullsized":"2024-03-06T07:22:59.804718+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.800525+0000","last_change":"2024-03-06T07:22:59.800525+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.800525+0000","last_undegraded":"2024-03-06T07:22:59.800525+0000","last_fullsized":"2024-03-06T07:22:59.800525+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.798524+0000","last_change":"2024-03-06T07:22:59.798524+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.798524+0000","last_undegraded":"2024-03-06T07:22:59.798524+0000","last_fullsized":"2024-03-06T07:22:59.798524+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797989+0000","last_change":"2024-03-06T07:22:59.797989+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797989+0000","last_undegraded":"2024-03-06T07:22:59.797989+0000","last_fullsized":"2024-03-06T07:22:59.797989+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.801589+0000","last_change":"2024-03-06T07:22:59.801589+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.801589+0000","last_undegraded":"2024-03-06T07:22:59.801589+0000","last_fullsized":"2024-03-06T07:22:59.801589+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.804790+0000","last_change":"2024-03-06T07:22:59.804790+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.804790+0000","last_undegraded":"2024-03-06T07:22:59.804790+0000","last_fullsized":"2024-03-06T07:22:59.804790+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.799988+0000","last_change":"2024-03-06T07:22:59.799988+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.799988+0000","last_undegraded":"2024-03-06T07:22:59.799988+0000","last_fullsized":"2024-03-06T07:22:59.799988+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.799777+0000","last_change":"2024-03-06T07:22:59.799777+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.799777+0000","last_undegraded":"2024-03-06T07:22:59.799777+0000","last_fullsized":"2024-03-06T07:22:59.799777+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.801040+0000","last_change":"2024-03-06T07:22:59.801040+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.801040+0000","last_undegraded":"2024-03-06T07:22:59.801040+0000","last_fullsized":"2024-03-06T07:22:59.801040+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.803466+0000","last_change":"2024-03-06T07:22:59.803466+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.803466+0000","last_undegraded":"2024-03-06T07:22:59.803466+0000","last_fullsized":"2024-03-06T07:22:59.803466+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.798121+0000","last_change":"2024-03-06T07:22:59.798121+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.798121+0000","last_undegraded":"2024-03-06T07:22:59.798121+0000","last_fullsized":"2024-03-06T07:22:59.798121+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797601+0000","last_change":"2024-03-06T07:22:59.797601+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797601+0000","last_undegraded":"2024-03-06T07:22:59.797601+0000","last_fullsized":"2024-03-06T07:22:59.797601+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.800974+0000","last_change":"2024-03-06T07:22:59.800974+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.800974+0000","last_undegraded":"2024-03-06T07:22:59.800974+0000","last_fullsized":"2024-03-06T07:22:59.800974+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.801086+0000","last_change":"2024-03-06T07:22:59.801086+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.801086+0000","last_undegraded":"2024-03-06T07:22:59.801086+0000","last_fullsized":"2024-03-06T07:22:59.801086+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.802385+0000","last_change":"2024-03-06T07:22:59.802385+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.802385+0000","last_undegraded":"2024-03-06T07:22:59.802385+0000","last_fullsized":"2024-03-06T07:22:59.802385+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.799075+0000","last_change":"2024-03-06T07:22:59.799075+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.799075+0000","last_undegraded":"2024-03-06T07:22:59.799075+0000","last_fullsized":"2024-03-06T07:22:59.799075+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797394+0000","last_change":"2024-03-06T07:22:59.797394+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797394+0000","last_undegraded":"2024-03-06T07:22:59.797394+0000","last_fullsized":"2024-03-06T07:22:59.797394+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.798311+0000","last_change":"2024-03-06T07:22:59.798311+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.798311+0000","last_undegraded":"2024-03-06T07:22:59.798311+0000","last_fullsized":"2024-03-06T07:22:59.798311+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.800053+0000","last_change":"2024-03-06T07:22:59.800053+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.800053+0000","last_undegraded":"2024-03-06T07:22:59.800053+0000","last_fullsized":"2024-03-06T07:22:59.800053+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796916+0000","last_change":"2024-03-06T07:22:59.796916+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796916+0000","last_undegraded":"2024-03-06T07:22:59.796916+0000","last_fullsized":"2024-03-06T07:22:59.796916+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796381+0000","last_change":"2024-03-06T07:22:59.796381+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796381+0000","last_undegraded":"2024-03-06T07:22:59.796381+0000","last_fullsized":"2024-03-06T07:22:59.796381+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.806940+0000","last_change":"2024-03-06T07:22:59.806940+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.806940+0000","last_undegraded":"2024-03-06T07:22:59.806940+0000","last_fullsized":"2024-03-06T07:22:59.806940+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.799846+0000","last_change":"2024-03-06T07:22:59.799846+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.799846+0000","last_undegraded":"2024-03-06T07:22:59.799846+0000","last_fullsized":"2024-03-06T07:22:59.799846+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.798684+0000","last_change":"2024-03-06T07:22:59.798684+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.798684+0000","last_undegraded":"2024-03-06T07:22:59.798684+0000","last_fullsized":"2024-03-06T07:22:59.798684+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.800428+0000","last_change":"2024-03-06T07:22:59.800428+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.800428+0000","last_undegraded":"2024-03-06T07:22:59.800428+0000","last_fullsized":"2024-03-06T07:22:59.800428+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.803078+0000","last_change":"2024-03-06T07:22:59.803078+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.803078+0000","last_undegraded":"2024-03-06T07:22:59.803078+0000","last_fullsized":"2024-03-06T07:22:59.803078+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.799594+0000","last_change":"2024-03-06T07:22:59.799594+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.799594+0000","last_undegraded":"2024-03-06T07:22:59.799594+0000","last_fullsized":"2024-03-06T07:22:59.799594+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.801930+0000","last_change":"2024-03-06T07:22:59.801930+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.801930+0000","last_undegraded":"2024-03-06T07:22:59.801930+0000","last_fullsized":"2024-03-06T07:22:59.801930+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797043+0000","last_change":"2024-03-06T07:22:59.797043+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797043+0000","last_undegraded":"2024-03-06T07:22:59.797043+0000","last_fullsized":"2024-03-06T07:22:59.797043+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.804395+0000","last_change":"2024-03-06T07:22:59.804395+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.804395+0000","last_undegraded":"2024-03-06T07:22:59.804395+0000","last_fullsized":"2024-03-06T07:22:59.804395+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796359+0000","last_change":"2024-03-06T07:22:59.796359+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796359+0000","last_undegraded":"2024-03-06T07:22:59.796359+0000","last_fullsized":"2024-03-06T07:22:59.796359+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.812031+0000","last_change":"2024-03-06T07:22:59.810363+0000","last_active":"2024-03-06T07:22:59.812031+0000","last_peered":"2024-03-06T07:22:59.812031+0000","last_clean":"2024-03-06T07:22:59.812031+0000","last_became_active":"2024-03-06T07:22:59.810032+0000","last_became_peered":"2024-03-06T07:22:59.810032+0000","last_unstale":"2024-03-06T07:22:59.812031+0000","last_undegraded":"2024-03-06T07:22:59.812031+0000","last_fullsized":"2024-03-06T07:22:59.812031+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:36:12.264660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797548+0000","last_change":"2024-03-06T07:22:59.797548+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797548+0000","last_undegraded":"2024-03-06T07:22:59.797548+0000","last_fullsized":"2024-03-06T07:22:59.797548+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.811396+0000","last_change":"2024-03-06T07:22:59.810687+0000","last_active":"2024-03-06T07:22:59.811396+0000","last_peered":"2024-03-06T07:22:59.811396+0000","last_clean":"2024-03-06T07:22:59.811396+0000","last_became_active":"2024-03-06T07:22:59.810405+0000","last_became_peered":"2024-03-06T07:22:59.810405+0000","last_unstale":"2024-03-06T07:22:59.811396+0000","last_undegraded":"2024-03-06T07:22:59.811396+0000","last_fullsized":"2024-03-06T07:22:59.811396+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:24:26.919565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.798640+0000","last_change":"2024-03-06T07:22:59.798640+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.798640+0000","last_undegraded":"2024-03-06T07:22:59.798640+0000","last_fullsized":"2024-03-06T07:22:59.798640+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.807686+0000","last_change":"2024-03-06T07:22:59.807093+0000","last_active":"2024-03-06T07:22:59.807686+0000","last_peered":"2024-03-06T07:22:59.807686+0000","last_clean":"2024-03-06T07:22:59.807686+0000","last_became_active":"2024-03-06T07:22:59.806858+0000","last_became_peered":"2024-03-06T07:22:59.806858+0000","last_unstale":"2024-03-06T07:22:59.807686+0000","last_undegraded":"2024-03-06T07:22:59.807686+0000","last_fullsized":"2024-03-06T07:22:59.807686+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:38:31.114570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.799435+0000","last_change":"2024-03-06T07:22:59.799435+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.799435+0000","last_undegraded":"2024-03-06T07:22:59.799435+0000","last_fullsized":"2024-03-06T07:22:59.799435+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.809136+0000","last_change":"2024-03-06T07:22:59.805788+0000","last_active":"2024-03-06T07:22:59.809136+0000","last_peered":"2024-03-06T07:22:59.809136+0000","last_clean":"2024-03-06T07:22:59.809136+0000","last_became_active":"2024-03-06T07:22:59.805463+0000","last_became_peered":"2024-03-06T07:22:59.805463+0000","last_unstale":"2024-03-06T07:22:59.809136+0000","last_undegraded":"2024-03-06T07:22:59.809136+0000","last_fullsized":"2024-03-06T07:22:59.809136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:14.437437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.801363+0000","last_change":"2024-03-06T07:22:59.801363+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.801363+0000","last_undegraded":"2024-03-06T07:22:59.801363+0000","last_fullsized":"2024-03-06T07:22:59.801363+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.811699+0000","last_change":"2024-03-06T07:22:59.808162+0000","last_active":"2024-03-06T07:22:59.811699+0000","last_peered":"2024-03-06T07:22:59.811699+0000","last_clean":"2024-03-06T07:22:59.811699+0000","last_became_active":"2024-03-06T07:22:59.807875+0000","last_became_peered":"2024-03-06T07:22:59.807875+0000","last_unstale":"2024-03-06T07:22:59.811699+0000","last_undegraded":"2024-03-06T07:22:59.811699+0000","last_fullsized":"2024-03-06T07:22:59.811699+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:53.014712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.802462+0000","last_change":"2024-03-06T07:22:59.802462+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.802462+0000","last_undegraded":"2024-03-06T07:22:59.802462+0000","last_fullsized":"2024-03-06T07:22:59.802462+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.811787+0000","last_change":"2024-03-06T07:22:59.806399+0000","last_active":"2024-03-06T07:22:59.811787+0000","last_peered":"2024-03-06T07:22:59.811787+0000","last_clean":"2024-03-06T07:22:59.811787+0000","last_became_active":"2024-03-06T07:22:59.805990+0000","last_became_peered":"2024-03-06T07:22:59.805990+0000","last_unstale":"2024-03-06T07:22:59.811787+0000","last_undegraded":"2024-03-06T07:22:59.811787+0000","last_fullsized":"2024-03-06T07:22:59.811787+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:07.271082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797495+0000","last_change":"2024-03-06T07:22:59.797495+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797495+0000","last_undegraded":"2024-03-06T07:22:59.797495+0000","last_fullsized":"2024-03-06T07:22:59.797495+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.806357+0000","last_change":"2024-03-06T07:22:59.805251+0000","last_active":"2024-03-06T07:22:59.806357+0000","last_peered":"2024-03-06T07:22:59.806357+0000","last_clean":"2024-03-06T07:22:59.806357+0000","last_became_active":"2024-03-06T07:22:59.804769+0000","last_became_peered":"2024-03-06T07:22:59.804769+0000","last_unstale":"2024-03-06T07:22:59.806357+0000","last_undegraded":"2024-03-06T07:22:59.806357+0000","last_fullsized":"2024-03-06T07:22:59.806357+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:30:21.970435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797342+0000","last_change":"2024-03-06T07:22:59.797342+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797342+0000","last_undegraded":"2024-03-06T07:22:59.797342+0000","last_fullsized":"2024-03-06T07:22:59.797342+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.810668+0000","last_change":"2024-03-06T07:22:59.804481+0000","last_active":"2024-03-06T07:22:59.810668+0000","last_peered":"2024-03-06T07:22:59.810668+0000","last_clean":"2024-03-06T07:22:59.810668+0000","last_became_active":"2024-03-06T07:22:59.803413+0000","last_became_peered":"2024-03-06T07:22:59.803413+0000","last_unstale":"2024-03-06T07:22:59.810668+0000","last_undegraded":"2024-03-06T07:22:59.810668+0000","last_fullsized":"2024-03-06T07:22:59.810668+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:14.437437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.802007+0000","last_change":"2024-03-06T07:22:59.802007+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.802007+0000","last_undegraded":"2024-03-06T07:22:59.802007+0000","last_fullsized":"2024-03-06T07:22:59.802007+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.810221+0000","last_change":"2024-03-06T07:22:59.808659+0000","last_active":"2024-03-06T07:22:59.810221+0000","last_peered":"2024-03-06T07:22:59.810221+0000","last_clean":"2024-03-06T07:22:59.810221+0000","last_became_active":"2024-03-06T07:22:59.808416+0000","last_became_peered":"2024-03-06T07:22:59.808416+0000","last_unstale":"2024-03-06T07:22:59.810221+0000","last_undegraded":"2024-03-06T07:22:59.810221+0000","last_fullsized":"2024-03-06T07:22:59.810221+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:32.447994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.798133+0000","last_change":"2024-03-06T07:22:59.798133+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.798133+0000","last_undegraded":"2024-03-06T07:22:59.798133+0000","last_fullsized":"2024-03-06T07:22:59.798133+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.805530+0000","last_change":"2024-03-06T07:22:59.804638+0000","last_active":"2024-03-06T07:22:59.805530+0000","last_peered":"2024-03-06T07:22:59.805530+0000","last_clean":"2024-03-06T07:22:59.805530+0000","last_became_active":"2024-03-06T07:22:59.804026+0000","last_became_peered":"2024-03-06T07:22:59.804026+0000","last_unstale":"2024-03-06T07:22:59.805530+0000","last_undegraded":"2024-03-06T07:22:59.805530+0000","last_fullsized":"2024-03-06T07:22:59.805530+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:04:10.795313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.798485+0000","last_change":"2024-03-06T07:22:59.798485+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.798485+0000","last_undegraded":"2024-03-06T07:22:59.798485+0000","last_fullsized":"2024-03-06T07:22:59.798485+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.816328+0000","last_change":"2024-03-06T07:22:59.815296+0000","last_active":"2024-03-06T07:22:59.816328+0000","last_peered":"2024-03-06T07:22:59.816328+0000","last_clean":"2024-03-06T07:22:59.816328+0000","last_became_active":"2024-03-06T07:22:59.815033+0000","last_became_peered":"2024-03-06T07:22:59.815033+0000","last_unstale":"2024-03-06T07:22:59.816328+0000","last_undegraded":"2024-03-06T07:22:59.816328+0000","last_fullsized":"2024-03-06T07:22:59.816328+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:29.015317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.808497+0000","last_change":"2024-03-06T07:22:59.808497+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.808497+0000","last_undegraded":"2024-03-06T07:22:59.808497+0000","last_fullsized":"2024-03-06T07:22:59.808497+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.808772+0000","last_change":"2024-03-06T07:22:59.807399+0000","last_active":"2024-03-06T07:22:59.808772+0000","last_peered":"2024-03-06T07:22:59.808772+0000","last_clean":"2024-03-06T07:22:59.808772+0000","last_became_active":"2024-03-06T07:22:59.807150+0000","last_became_peered":"2024-03-06T07:22:59.807150+0000","last_unstale":"2024-03-06T07:22:59.808772+0000","last_undegraded":"2024-03-06T07:22:59.808772+0000","last_fullsized":"2024-03-06T07:22:59.808772+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:19.337313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.798310+0000","last_change":"2024-03-06T07:22:59.798310+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.798310+0000","last_undegraded":"2024-03-06T07:22:59.798310+0000","last_fullsized":"2024-03-06T07:22:59.798310+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.811735+0000","last_change":"2024-03-06T07:22:59.809344+0000","last_active":"2024-03-06T07:22:59.811735+0000","last_peered":"2024-03-06T07:22:59.811735+0000","last_clean":"2024-03-06T07:22:59.811735+0000","last_became_active":"2024-03-06T07:22:59.809076+0000","last_became_peered":"2024-03-06T07:22:59.809076+0000","last_unstale":"2024-03-06T07:22:59.811735+0000","last_undegraded":"2024-03-06T07:22:59.811735+0000","last_fullsized":"2024-03-06T07:22:59.811735+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:12.683272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796615+0000","last_change":"2024-03-06T07:22:59.796615+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796615+0000","last_undegraded":"2024-03-06T07:22:59.796615+0000","last_fullsized":"2024-03-06T07:22:59.796615+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.809522+0000","last_change":"2024-03-06T07:22:59.807007+0000","last_active":"2024-03-06T07:22:59.809522+0000","last_peered":"2024-03-06T07:22:59.809522+0000","last_clean":"2024-03-06T07:22:59.809522+0000","last_became_active":"2024-03-06T07:22:59.806750+0000","last_became_peered":"2024-03-06T07:22:59.806750+0000","last_unstale":"2024-03-06T07:22:59.809522+0000","last_undegraded":"2024-03-06T07:22:59.809522+0000","last_fullsized":"2024-03-06T07:22:59.809522+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:24:26.919565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.804026+0000","last_change":"2024-03-06T07:22:59.804026+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.804026+0000","last_undegraded":"2024-03-06T07:22:59.804026+0000","last_fullsized":"2024-03-06T07:22:59.804026+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.809695+0000","last_change":"2024-03-06T07:22:59.808043+0000","last_active":"2024-03-06T07:22:59.809695+0000","last_peered":"2024-03-06T07:22:59.809695+0000","last_clean":"2024-03-06T07:22:59.809695+0000","last_became_active":"2024-03-06T07:22:59.807794+0000","last_became_peered":"2024-03-06T07:22:59.807794+0000","last_unstale":"2024-03-06T07:22:59.809695+0000","last_undegraded":"2024-03-06T07:22:59.809695+0000","last_fullsized":"2024-03-06T07:22:59.809695+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:08:18.005082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796469+0000","last_change":"2024-03-06T07:22:59.796469+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796469+0000","last_undegraded":"2024-03-06T07:22:59.796469+0000","last_fullsized":"2024-03-06T07:22:59.796469+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.807259+0000","last_change":"2024-03-06T07:22:59.806459+0000","last_active":"2024-03-06T07:22:59.807259+0000","last_peered":"2024-03-06T07:22:59.807259+0000","last_clean":"2024-03-06T07:22:59.807259+0000","last_became_active":"2024-03-06T07:22:59.806194+0000","last_became_peered":"2024-03-06T07:22:59.806194+0000","last_unstale":"2024-03-06T07:22:59.807259+0000","last_undegraded":"2024-03-06T07:22:59.807259+0000","last_fullsized":"2024-03-06T07:22:59.807259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:53.894652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796668+0000","last_change":"2024-03-06T07:22:59.796668+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796668+0000","last_undegraded":"2024-03-06T07:22:59.796668+0000","last_fullsized":"2024-03-06T07:22:59.796668+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.810182+0000","last_change":"2024-03-06T07:22:59.808555+0000","last_active":"2024-03-06T07:22:59.810182+0000","last_peered":"2024-03-06T07:22:59.810182+0000","last_clean":"2024-03-06T07:22:59.810182+0000","last_became_active":"2024-03-06T07:22:59.808284+0000","last_became_peered":"2024-03-06T07:22:59.808284+0000","last_unstale":"2024-03-06T07:22:59.810182+0000","last_undegraded":"2024-03-06T07:22:59.810182+0000","last_fullsized":"2024-03-06T07:22:59.810182+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:30:21.970435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.803648+0000","last_change":"2024-03-06T07:22:59.803648+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.803648+0000","last_undegraded":"2024-03-06T07:22:59.803648+0000","last_fullsized":"2024-03-06T07:22:59.803648+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.805500+0000","last_change":"2024-03-06T07:22:59.802730+0000","last_active":"2024-03-06T07:22:59.805500+0000","last_peered":"2024-03-06T07:22:59.805500+0000","last_clean":"2024-03-06T07:22:59.805500+0000","last_became_active":"2024-03-06T07:22:59.802461+0000","last_became_peered":"2024-03-06T07:22:59.802461+0000","last_unstale":"2024-03-06T07:22:59.805500+0000","last_undegraded":"2024-03-06T07:22:59.805500+0000","last_fullsized":"2024-03-06T07:22:59.805500+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:26:21.615251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.798752+0000","last_change":"2024-03-06T07:22:59.798752+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.798752+0000","last_undegraded":"2024-03-06T07:22:59.798752+0000","last_fullsized":"2024-03-06T07:22:59.798752+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.811762+0000","last_change":"2024-03-06T07:22:59.807016+0000","last_active":"2024-03-06T07:22:59.811762+0000","last_peered":"2024-03-06T07:22:59.811762+0000","last_clean":"2024-03-06T07:22:59.811762+0000","last_became_active":"2024-03-06T07:22:59.806733+0000","last_became_peered":"2024-03-06T07:22:59.806733+0000","last_unstale":"2024-03-06T07:22:59.811762+0000","last_undegraded":"2024-03-06T07:22:59.811762+0000","last_fullsized":"2024-03-06T07:22:59.811762+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:55.870572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797444+0000","last_change":"2024-03-06T07:22:59.797444+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797444+0000","last_undegraded":"2024-03-06T07:22:59.797444+0000","last_fullsized":"2024-03-06T07:22:59.797444+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.811731+0000","last_change":"2024-03-06T07:22:59.808721+0000","last_active":"2024-03-06T07:22:59.811731+0000","last_peered":"2024-03-06T07:22:59.811731+0000","last_clean":"2024-03-06T07:22:59.811731+0000","last_became_active":"2024-03-06T07:22:59.808465+0000","last_became_peered":"2024-03-06T07:22:59.808465+0000","last_unstale":"2024-03-06T07:22:59.811731+0000","last_undegraded":"2024-03-06T07:22:59.811731+0000","last_fullsized":"2024-03-06T07:22:59.811731+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:08.896012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796675+0000","last_change":"2024-03-06T07:22:59.796675+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796675+0000","last_undegraded":"2024-03-06T07:22:59.796675+0000","last_fullsized":"2024-03-06T07:22:59.796675+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.805543+0000","last_change":"2024-03-06T07:22:59.804367+0000","last_active":"2024-03-06T07:22:59.805543+0000","last_peered":"2024-03-06T07:22:59.805543+0000","last_clean":"2024-03-06T07:22:59.805543+0000","last_became_active":"2024-03-06T07:22:59.804115+0000","last_became_peered":"2024-03-06T07:22:59.804115+0000","last_unstale":"2024-03-06T07:22:59.805543+0000","last_undegraded":"2024-03-06T07:22:59.805543+0000","last_fullsized":"2024-03-06T07:22:59.805543+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:44:35.664305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797737+0000","last_change":"2024-03-06T07:22:59.797737+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797737+0000","last_undegraded":"2024-03-06T07:22:59.797737+0000","last_fullsized":"2024-03-06T07:22:59.797737+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.806310+0000","last_change":"2024-03-06T07:22:59.804749+0000","last_active":"2024-03-06T07:22:59.806310+0000","last_peered":"2024-03-06T07:22:59.806310+0000","last_clean":"2024-03-06T07:22:59.806310+0000","last_became_active":"2024-03-06T07:22:59.802839+0000","last_became_peered":"2024-03-06T07:22:59.802839+0000","last_unstale":"2024-03-06T07:22:59.806310+0000","last_undegraded":"2024-03-06T07:22:59.806310+0000","last_fullsized":"2024-03-06T07:22:59.806310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:04:10.795313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796191+0000","last_change":"2024-03-06T07:22:59.796191+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796191+0000","last_undegraded":"2024-03-06T07:22:59.796191+0000","last_fullsized":"2024-03-06T07:22:59.796191+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.813816+0000","last_change":"2024-03-06T07:22:59.810390+0000","last_active":"2024-03-06T07:22:59.813816+0000","last_peered":"2024-03-06T07:22:59.813816+0000","last_clean":"2024-03-06T07:22:59.813816+0000","last_became_active":"2024-03-06T07:22:59.810140+0000","last_became_peered":"2024-03-06T07:22:59.810140+0000","last_unstale":"2024-03-06T07:22:59.813816+0000","last_undegraded":"2024-03-06T07:22:59.813816+0000","last_fullsized":"2024-03-06T07:22:59.813816+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:13.131161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.802019+0000","last_change":"2024-03-06T07:22:59.802019+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.802019+0000","last_undegraded":"2024-03-06T07:22:59.802019+0000","last_fullsized":"2024-03-06T07:22:59.802019+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.814736+0000","last_change":"2024-03-06T07:22:59.812896+0000","last_active":"2024-03-06T07:22:59.814736+0000","last_peered":"2024-03-06T07:22:59.814736+0000","last_clean":"2024-03-06T07:22:59.814736+0000","last_became_active":"2024-03-06T07:22:59.812598+0000","last_became_peered":"2024-03-06T07:22:59.812598+0000","last_unstale":"2024-03-06T07:22:59.814736+0000","last_undegraded":"2024-03-06T07:22:59.814736+0000","last_fullsized":"2024-03-06T07:22:59.814736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:37.715070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797040+0000","last_change":"2024-03-06T07:22:59.797040+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797040+0000","last_undegraded":"2024-03-06T07:22:59.797040+0000","last_fullsized":"2024-03-06T07:22:59.797040+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.813596+0000","last_change":"2024-03-06T07:22:59.811226+0000","last_active":"2024-03-06T07:22:59.813596+0000","last_peered":"2024-03-06T07:22:59.813596+0000","last_clean":"2024-03-06T07:22:59.813596+0000","last_became_active":"2024-03-06T07:22:59.810822+0000","last_became_peered":"2024-03-06T07:22:59.810822+0000","last_unstale":"2024-03-06T07:22:59.813596+0000","last_undegraded":"2024-03-06T07:22:59.813596+0000","last_fullsized":"2024-03-06T07:22:59.813596+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:53.894652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.803232+0000","last_change":"2024-03-06T07:22:59.803232+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.803232+0000","last_undegraded":"2024-03-06T07:22:59.803232+0000","last_fullsized":"2024-03-06T07:22:59.803232+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.805549+0000","last_change":"2024-03-06T07:22:59.804853+0000","last_active":"2024-03-06T07:22:59.805549+0000","last_peered":"2024-03-06T07:22:59.805549+0000","last_clean":"2024-03-06T07:22:59.805549+0000","last_became_active":"2024-03-06T07:22:59.804353+0000","last_became_peered":"2024-03-06T07:22:59.804353+0000","last_unstale":"2024-03-06T07:22:59.805549+0000","last_undegraded":"2024-03-06T07:22:59.805549+0000","last_fullsized":"2024-03-06T07:22:59.805549+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:46:08.564158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796177+0000","last_change":"2024-03-06T07:22:59.796177+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796177+0000","last_undegraded":"2024-03-06T07:22:59.796177+0000","last_fullsized":"2024-03-06T07:22:59.796177+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.804694+0000","last_change":"2024-03-06T07:22:59.803415+0000","last_active":"2024-03-06T07:22:59.804694+0000","last_peered":"2024-03-06T07:22:59.804694+0000","last_clean":"2024-03-06T07:22:59.804694+0000","last_became_active":"2024-03-06T07:22:59.803175+0000","last_became_peered":"2024-03-06T07:22:59.803175+0000","last_unstale":"2024-03-06T07:22:59.804694+0000","last_undegraded":"2024-03-06T07:22:59.804694+0000","last_fullsized":"2024-03-06T07:22:59.804694+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:36:12.264660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796946+0000","last_change":"2024-03-06T07:22:59.796946+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796946+0000","last_undegraded":"2024-03-06T07:22:59.796946+0000","last_fullsized":"2024-03-06T07:22:59.796946+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.811396+0000","last_change":"2024-03-06T07:22:59.810599+0000","last_active":"2024-03-06T07:22:59.811396+0000","last_peered":"2024-03-06T07:22:59.811396+0000","last_clean":"2024-03-06T07:22:59.811396+0000","last_became_active":"2024-03-06T07:22:59.810291+0000","last_became_peered":"2024-03-06T07:22:59.810291+0000","last_unstale":"2024-03-06T07:22:59.811396+0000","last_undegraded":"2024-03-06T07:22:59.811396+0000","last_fullsized":"2024-03-06T07:22:59.811396+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:36:12.264660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796176+0000","last_change":"2024-03-06T07:22:59.796176+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796176+0000","last_undegraded":"2024-03-06T07:22:59.796176+0000","last_fullsized":"2024-03-06T07:22:59.796176+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.816470+0000","last_change":"2024-03-06T07:22:59.812044+0000","last_active":"2024-03-06T07:22:59.816470+0000","last_peered":"2024-03-06T07:22:59.816470+0000","last_clean":"2024-03-06T07:22:59.816470+0000","last_became_active":"2024-03-06T07:22:59.811822+0000","last_became_peered":"2024-03-06T07:22:59.811822+0000","last_unstale":"2024-03-06T07:22:59.816470+0000","last_undegraded":"2024-03-06T07:22:59.816470+0000","last_fullsized":"2024-03-06T07:22:59.816470+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:38:31.114570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.798677+0000","last_change":"2024-03-06T07:22:59.798677+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.798677+0000","last_undegraded":"2024-03-06T07:22:59.798677+0000","last_fullsized":"2024-03-06T07:22:59.798677+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.811751+0000","last_change":"2024-03-06T07:22:59.809833+0000","last_active":"2024-03-06T07:22:59.811751+0000","last_peered":"2024-03-06T07:22:59.811751+0000","last_clean":"2024-03-06T07:22:59.811751+0000","last_became_active":"2024-03-06T07:22:59.809550+0000","last_became_peered":"2024-03-06T07:22:59.809550+0000","last_unstale":"2024-03-06T07:22:59.811751+0000","last_undegraded":"2024-03-06T07:22:59.811751+0000","last_fullsized":"2024-03-06T07:22:59.811751+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:40:47.871999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.799865+0000","last_change":"2024-03-06T07:22:59.799865+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.799865+0000","last_undegraded":"2024-03-06T07:22:59.799865+0000","last_fullsized":"2024-03-06T07:22:59.799865+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.816542+0000","last_change":"2024-03-06T07:22:59.812895+0000","last_active":"2024-03-06T07:22:59.816542+0000","last_peered":"2024-03-06T07:22:59.816542+0000","last_clean":"2024-03-06T07:22:59.816542+0000","last_became_active":"2024-03-06T07:22:59.812605+0000","last_became_peered":"2024-03-06T07:22:59.812605+0000","last_unstale":"2024-03-06T07:22:59.816542+0000","last_undegraded":"2024-03-06T07:22:59.816542+0000","last_fullsized":"2024-03-06T07:22:59.816542+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:33.788501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.800675+0000","last_change":"2024-03-06T07:22:59.800675+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.800675+0000","last_undegraded":"2024-03-06T07:22:59.800675+0000","last_fullsized":"2024-03-06T07:22:59.800675+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.809068+0000","last_change":"2024-03-06T07:22:59.808026+0000","last_active":"2024-03-06T07:22:59.809068+0000","last_peered":"2024-03-06T07:22:59.809068+0000","last_clean":"2024-03-06T07:22:59.809068+0000","last_became_active":"2024-03-06T07:22:59.807197+0000","last_became_peered":"2024-03-06T07:22:59.807197+0000","last_unstale":"2024-03-06T07:22:59.809068+0000","last_undegraded":"2024-03-06T07:22:59.809068+0000","last_fullsized":"2024-03-06T07:22:59.809068+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:46:08.564158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":189,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.805202+0000","last_change":"2024-03-06T07:21:47.553693+0000","last_active":"2024-03-06T07:22:59.805202+0000","last_peered":"2024-03-06T07:22:59.805202+0000","last_clean":"2024-03-06T07:22:59.805202+0000","last_became_active":"2024-03-06T07:21:47.552664+0000","last_became_peered":"2024-03-06T07:21:47.552664+0000","last_unstale":"2024-03-06T07:22:59.805202+0000","last_undegraded":"2024-03-06T07:22:59.805202+0000","last_fullsized":"2024-03-06T07:22:59.805202+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_clean_scrub_stamp":"2024-03-06T07:21:46.362028+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:02.916426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343900,"statfs":{"total":96636764160,"available":96608153600,"internally_reserved":0,"allocated":942080,"data_stored":724403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":127768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":134035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":335872,"data_stored":127768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":127768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574867,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":134035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574868,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":134035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574867,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":929792,"data_stored":718136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:23:04.741 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3f is not active+clean 2024-03-06T07:23:04.741 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.797969+0000', 'last_change': '2024-03-06T07:22:59.797969+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.797969+0000', 'last_undegraded': '2024-03-06T07:22:59.797969+0000', 'last_fullsized': '2024-03-06T07:22:59.797969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:23:04.741 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3e is not active+clean 2024-03-06T07:23:04.741 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.799502+0000', 'last_change': '2024-03-06T07:22:59.799502+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.799502+0000', 'last_undegraded': '2024-03-06T07:22:59.799502+0000', 'last_fullsized': '2024-03-06T07:22:59.799502+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:04.741 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3d is not active+clean 2024-03-06T07:23:04.741 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.804718+0000', 'last_change': '2024-03-06T07:22:59.804718+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.804718+0000', 'last_undegraded': '2024-03-06T07:22:59.804718+0000', 'last_fullsized': '2024-03-06T07:22:59.804718+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:23:04.741 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3c is not active+clean 2024-03-06T07:23:04.741 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.800525+0000', 'last_change': '2024-03-06T07:22:59.800525+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.800525+0000', 'last_undegraded': '2024-03-06T07:22:59.800525+0000', 'last_fullsized': '2024-03-06T07:22:59.800525+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:23:04.742 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3b is not active+clean 2024-03-06T07:23:04.742 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.798524+0000', 'last_change': '2024-03-06T07:22:59.798524+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.798524+0000', 'last_undegraded': '2024-03-06T07:22:59.798524+0000', 'last_fullsized': '2024-03-06T07:22:59.798524+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:23:04.742 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3a is not active+clean 2024-03-06T07:23:04.742 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.797989+0000', 'last_change': '2024-03-06T07:22:59.797989+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.797989+0000', 'last_undegraded': '2024-03-06T07:22:59.797989+0000', 'last_fullsized': '2024-03-06T07:22:59.797989+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:04.742 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.39 is not active+clean 2024-03-06T07:23:04.742 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.801589+0000', 'last_change': '2024-03-06T07:22:59.801589+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.801589+0000', 'last_undegraded': '2024-03-06T07:22:59.801589+0000', 'last_fullsized': '2024-03-06T07:22:59.801589+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:23:04.742 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.38 is not active+clean 2024-03-06T07:23:04.742 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.804790+0000', 'last_change': '2024-03-06T07:22:59.804790+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.804790+0000', 'last_undegraded': '2024-03-06T07:22:59.804790+0000', 'last_fullsized': '2024-03-06T07:22:59.804790+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:23:04.742 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.37 is not active+clean 2024-03-06T07:23:04.743 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.799988+0000', 'last_change': '2024-03-06T07:22:59.799988+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.799988+0000', 'last_undegraded': '2024-03-06T07:22:59.799988+0000', 'last_fullsized': '2024-03-06T07:22:59.799988+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:23:04.743 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.36 is not active+clean 2024-03-06T07:23:04.743 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.799777+0000', 'last_change': '2024-03-06T07:22:59.799777+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.799777+0000', 'last_undegraded': '2024-03-06T07:22:59.799777+0000', 'last_fullsized': '2024-03-06T07:22:59.799777+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:23:04.743 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.35 is not active+clean 2024-03-06T07:23:04.743 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.801040+0000', 'last_change': '2024-03-06T07:22:59.801040+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.801040+0000', 'last_undegraded': '2024-03-06T07:22:59.801040+0000', 'last_fullsized': '2024-03-06T07:22:59.801040+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:23:04.743 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.34 is not active+clean 2024-03-06T07:23:04.743 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.803466+0000', 'last_change': '2024-03-06T07:22:59.803466+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.803466+0000', 'last_undegraded': '2024-03-06T07:22:59.803466+0000', 'last_fullsized': '2024-03-06T07:22:59.803466+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:23:04.743 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.33 is not active+clean 2024-03-06T07:23:04.743 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.798121+0000', 'last_change': '2024-03-06T07:22:59.798121+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.798121+0000', 'last_undegraded': '2024-03-06T07:22:59.798121+0000', 'last_fullsized': '2024-03-06T07:22:59.798121+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:23:04.743 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.32 is not active+clean 2024-03-06T07:23:04.743 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.797601+0000', 'last_change': '2024-03-06T07:22:59.797601+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.797601+0000', 'last_undegraded': '2024-03-06T07:22:59.797601+0000', 'last_fullsized': '2024-03-06T07:22:59.797601+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:04.744 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.31 is not active+clean 2024-03-06T07:23:04.744 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.800974+0000', 'last_change': '2024-03-06T07:22:59.800974+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.800974+0000', 'last_undegraded': '2024-03-06T07:22:59.800974+0000', 'last_fullsized': '2024-03-06T07:22:59.800974+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:23:04.744 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.30 is not active+clean 2024-03-06T07:23:04.744 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.801086+0000', 'last_change': '2024-03-06T07:22:59.801086+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.801086+0000', 'last_undegraded': '2024-03-06T07:22:59.801086+0000', 'last_fullsized': '2024-03-06T07:22:59.801086+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:23:04.744 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2f is not active+clean 2024-03-06T07:23:04.744 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.802385+0000', 'last_change': '2024-03-06T07:22:59.802385+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.802385+0000', 'last_undegraded': '2024-03-06T07:22:59.802385+0000', 'last_fullsized': '2024-03-06T07:22:59.802385+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:23:04.744 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2e is not active+clean 2024-03-06T07:23:04.744 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.799075+0000', 'last_change': '2024-03-06T07:22:59.799075+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.799075+0000', 'last_undegraded': '2024-03-06T07:22:59.799075+0000', 'last_fullsized': '2024-03-06T07:22:59.799075+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:04.744 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2d is not active+clean 2024-03-06T07:23:04.744 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.797394+0000', 'last_change': '2024-03-06T07:22:59.797394+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.797394+0000', 'last_undegraded': '2024-03-06T07:22:59.797394+0000', 'last_fullsized': '2024-03-06T07:22:59.797394+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:04.744 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2c is not active+clean 2024-03-06T07:23:04.745 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.798311+0000', 'last_change': '2024-03-06T07:22:59.798311+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.798311+0000', 'last_undegraded': '2024-03-06T07:22:59.798311+0000', 'last_fullsized': '2024-03-06T07:22:59.798311+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:23:04.745 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2b is not active+clean 2024-03-06T07:23:04.745 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.800053+0000', 'last_change': '2024-03-06T07:22:59.800053+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.800053+0000', 'last_undegraded': '2024-03-06T07:22:59.800053+0000', 'last_fullsized': '2024-03-06T07:22:59.800053+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:23:04.745 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2a is not active+clean 2024-03-06T07:23:04.745 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.796916+0000', 'last_change': '2024-03-06T07:22:59.796916+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.796916+0000', 'last_undegraded': '2024-03-06T07:22:59.796916+0000', 'last_fullsized': '2024-03-06T07:22:59.796916+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:23:04.745 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.29 is not active+clean 2024-03-06T07:23:04.745 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.796381+0000', 'last_change': '2024-03-06T07:22:59.796381+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.796381+0000', 'last_undegraded': '2024-03-06T07:22:59.796381+0000', 'last_fullsized': '2024-03-06T07:22:59.796381+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:04.745 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.28 is not active+clean 2024-03-06T07:23:04.745 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.806940+0000', 'last_change': '2024-03-06T07:22:59.806940+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.806940+0000', 'last_undegraded': '2024-03-06T07:22:59.806940+0000', 'last_fullsized': '2024-03-06T07:22:59.806940+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:23:04.745 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.27 is not active+clean 2024-03-06T07:23:04.745 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.799846+0000', 'last_change': '2024-03-06T07:22:59.799846+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.799846+0000', 'last_undegraded': '2024-03-06T07:22:59.799846+0000', 'last_fullsized': '2024-03-06T07:22:59.799846+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:23:04.746 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.26 is not active+clean 2024-03-06T07:23:04.746 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.798684+0000', 'last_change': '2024-03-06T07:22:59.798684+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.798684+0000', 'last_undegraded': '2024-03-06T07:22:59.798684+0000', 'last_fullsized': '2024-03-06T07:22:59.798684+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:04.746 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.25 is not active+clean 2024-03-06T07:23:04.746 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.800428+0000', 'last_change': '2024-03-06T07:22:59.800428+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.800428+0000', 'last_undegraded': '2024-03-06T07:22:59.800428+0000', 'last_fullsized': '2024-03-06T07:22:59.800428+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:23:04.746 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.24 is not active+clean 2024-03-06T07:23:04.746 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.803078+0000', 'last_change': '2024-03-06T07:22:59.803078+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.803078+0000', 'last_undegraded': '2024-03-06T07:22:59.803078+0000', 'last_fullsized': '2024-03-06T07:22:59.803078+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:23:04.746 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.23 is not active+clean 2024-03-06T07:23:04.746 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.799594+0000', 'last_change': '2024-03-06T07:22:59.799594+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.799594+0000', 'last_undegraded': '2024-03-06T07:22:59.799594+0000', 'last_fullsized': '2024-03-06T07:22:59.799594+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:23:04.746 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.22 is not active+clean 2024-03-06T07:23:04.746 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.801930+0000', 'last_change': '2024-03-06T07:22:59.801930+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.801930+0000', 'last_undegraded': '2024-03-06T07:22:59.801930+0000', 'last_fullsized': '2024-03-06T07:22:59.801930+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:23:04.747 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.21 is not active+clean 2024-03-06T07:23:04.747 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.797043+0000', 'last_change': '2024-03-06T07:22:59.797043+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.797043+0000', 'last_undegraded': '2024-03-06T07:22:59.797043+0000', 'last_fullsized': '2024-03-06T07:22:59.797043+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:23:04.747 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.20 is not active+clean 2024-03-06T07:23:04.747 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.804395+0000', 'last_change': '2024-03-06T07:22:59.804395+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.804395+0000', 'last_undegraded': '2024-03-06T07:22:59.804395+0000', 'last_fullsized': '2024-03-06T07:22:59.804395+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:23:04.747 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1e is not active+clean 2024-03-06T07:23:04.747 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.796359+0000', 'last_change': '2024-03-06T07:22:59.796359+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.796359+0000', 'last_undegraded': '2024-03-06T07:22:59.796359+0000', 'last_fullsized': '2024-03-06T07:22:59.796359+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:04.747 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1f is not active+clean 2024-03-06T07:23:04.747 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.797548+0000', 'last_change': '2024-03-06T07:22:59.797548+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.797548+0000', 'last_undegraded': '2024-03-06T07:22:59.797548+0000', 'last_fullsized': '2024-03-06T07:22:59.797548+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:23:04.747 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1c is not active+clean 2024-03-06T07:23:04.747 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.798640+0000', 'last_change': '2024-03-06T07:22:59.798640+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.798640+0000', 'last_undegraded': '2024-03-06T07:22:59.798640+0000', 'last_fullsized': '2024-03-06T07:22:59.798640+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:23:04.747 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1d is not active+clean 2024-03-06T07:23:04.748 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.799435+0000', 'last_change': '2024-03-06T07:22:59.799435+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.799435+0000', 'last_undegraded': '2024-03-06T07:22:59.799435+0000', 'last_fullsized': '2024-03-06T07:22:59.799435+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:23:04.748 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1a is not active+clean 2024-03-06T07:23:04.748 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.801363+0000', 'last_change': '2024-03-06T07:22:59.801363+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.801363+0000', 'last_undegraded': '2024-03-06T07:22:59.801363+0000', 'last_fullsized': '2024-03-06T07:22:59.801363+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:23:04.748 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1b is not active+clean 2024-03-06T07:23:04.748 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.802462+0000', 'last_change': '2024-03-06T07:22:59.802462+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.802462+0000', 'last_undegraded': '2024-03-06T07:22:59.802462+0000', 'last_fullsized': '2024-03-06T07:22:59.802462+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:23:04.748 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.18 is not active+clean 2024-03-06T07:23:04.748 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.797495+0000', 'last_change': '2024-03-06T07:22:59.797495+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.797495+0000', 'last_undegraded': '2024-03-06T07:22:59.797495+0000', 'last_fullsized': '2024-03-06T07:22:59.797495+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:23:04.748 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.19 is not active+clean 2024-03-06T07:23:04.748 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.797342+0000', 'last_change': '2024-03-06T07:22:59.797342+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.797342+0000', 'last_undegraded': '2024-03-06T07:22:59.797342+0000', 'last_fullsized': '2024-03-06T07:22:59.797342+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:23:04.748 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.16 is not active+clean 2024-03-06T07:23:04.749 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.802007+0000', 'last_change': '2024-03-06T07:22:59.802007+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.802007+0000', 'last_undegraded': '2024-03-06T07:22:59.802007+0000', 'last_fullsized': '2024-03-06T07:22:59.802007+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:23:04.749 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.17 is not active+clean 2024-03-06T07:23:04.749 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.798133+0000', 'last_change': '2024-03-06T07:22:59.798133+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.798133+0000', 'last_undegraded': '2024-03-06T07:22:59.798133+0000', 'last_fullsized': '2024-03-06T07:22:59.798133+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:04.749 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.14 is not active+clean 2024-03-06T07:23:04.749 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.798485+0000', 'last_change': '2024-03-06T07:22:59.798485+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.798485+0000', 'last_undegraded': '2024-03-06T07:22:59.798485+0000', 'last_fullsized': '2024-03-06T07:22:59.798485+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:04.749 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.15 is not active+clean 2024-03-06T07:23:04.749 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.808497+0000', 'last_change': '2024-03-06T07:22:59.808497+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.808497+0000', 'last_undegraded': '2024-03-06T07:22:59.808497+0000', 'last_fullsized': '2024-03-06T07:22:59.808497+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:23:04.749 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.12 is not active+clean 2024-03-06T07:23:04.749 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.798310+0000', 'last_change': '2024-03-06T07:22:59.798310+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.798310+0000', 'last_undegraded': '2024-03-06T07:22:59.798310+0000', 'last_fullsized': '2024-03-06T07:22:59.798310+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:23:04.749 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.13 is not active+clean 2024-03-06T07:23:04.750 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.796615+0000', 'last_change': '2024-03-06T07:22:59.796615+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.796615+0000', 'last_undegraded': '2024-03-06T07:22:59.796615+0000', 'last_fullsized': '2024-03-06T07:22:59.796615+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:04.750 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.10 is not active+clean 2024-03-06T07:23:04.750 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.804026+0000', 'last_change': '2024-03-06T07:22:59.804026+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.804026+0000', 'last_undegraded': '2024-03-06T07:22:59.804026+0000', 'last_fullsized': '2024-03-06T07:22:59.804026+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:23:04.750 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.11 is not active+clean 2024-03-06T07:23:04.750 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.796469+0000', 'last_change': '2024-03-06T07:22:59.796469+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.796469+0000', 'last_undegraded': '2024-03-06T07:22:59.796469+0000', 'last_fullsized': '2024-03-06T07:22:59.796469+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:23:04.750 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.e is not active+clean 2024-03-06T07:23:04.750 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.796668+0000', 'last_change': '2024-03-06T07:22:59.796668+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.796668+0000', 'last_undegraded': '2024-03-06T07:22:59.796668+0000', 'last_fullsized': '2024-03-06T07:22:59.796668+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:23:04.750 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.f is not active+clean 2024-03-06T07:23:04.750 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.803648+0000', 'last_change': '2024-03-06T07:22:59.803648+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.803648+0000', 'last_undegraded': '2024-03-06T07:22:59.803648+0000', 'last_fullsized': '2024-03-06T07:22:59.803648+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:23:04.750 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.c is not active+clean 2024-03-06T07:23:04.751 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.798752+0000', 'last_change': '2024-03-06T07:22:59.798752+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.798752+0000', 'last_undegraded': '2024-03-06T07:22:59.798752+0000', 'last_fullsized': '2024-03-06T07:22:59.798752+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:23:04.751 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.d is not active+clean 2024-03-06T07:23:04.751 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.797444+0000', 'last_change': '2024-03-06T07:22:59.797444+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.797444+0000', 'last_undegraded': '2024-03-06T07:22:59.797444+0000', 'last_fullsized': '2024-03-06T07:22:59.797444+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:23:04.751 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.a is not active+clean 2024-03-06T07:23:04.751 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.796675+0000', 'last_change': '2024-03-06T07:22:59.796675+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.796675+0000', 'last_undegraded': '2024-03-06T07:22:59.796675+0000', 'last_fullsized': '2024-03-06T07:22:59.796675+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:23:04.751 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.b is not active+clean 2024-03-06T07:23:04.751 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.797737+0000', 'last_change': '2024-03-06T07:22:59.797737+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.797737+0000', 'last_undegraded': '2024-03-06T07:22:59.797737+0000', 'last_fullsized': '2024-03-06T07:22:59.797737+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:23:04.751 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.8 is not active+clean 2024-03-06T07:23:04.751 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.796191+0000', 'last_change': '2024-03-06T07:22:59.796191+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.796191+0000', 'last_undegraded': '2024-03-06T07:22:59.796191+0000', 'last_fullsized': '2024-03-06T07:22:59.796191+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:23:04.751 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.9 is not active+clean 2024-03-06T07:23:04.751 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.802019+0000', 'last_change': '2024-03-06T07:22:59.802019+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.802019+0000', 'last_undegraded': '2024-03-06T07:22:59.802019+0000', 'last_fullsized': '2024-03-06T07:22:59.802019+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:23:04.752 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.6 is not active+clean 2024-03-06T07:23:04.752 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.797040+0000', 'last_change': '2024-03-06T07:22:59.797040+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.797040+0000', 'last_undegraded': '2024-03-06T07:22:59.797040+0000', 'last_fullsized': '2024-03-06T07:22:59.797040+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:23:04.752 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.7 is not active+clean 2024-03-06T07:23:04.752 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.803232+0000', 'last_change': '2024-03-06T07:22:59.803232+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.803232+0000', 'last_undegraded': '2024-03-06T07:22:59.803232+0000', 'last_fullsized': '2024-03-06T07:22:59.803232+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:23:04.752 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.5 is not active+clean 2024-03-06T07:23:04.752 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.796177+0000', 'last_change': '2024-03-06T07:22:59.796177+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.796177+0000', 'last_undegraded': '2024-03-06T07:22:59.796177+0000', 'last_fullsized': '2024-03-06T07:22:59.796177+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:04.752 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2 is not active+clean 2024-03-06T07:23:04.752 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.796946+0000', 'last_change': '2024-03-06T07:22:59.796946+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.796946+0000', 'last_undegraded': '2024-03-06T07:22:59.796946+0000', 'last_fullsized': '2024-03-06T07:22:59.796946+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:23:04.752 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3 is not active+clean 2024-03-06T07:23:04.753 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.796176+0000', 'last_change': '2024-03-06T07:22:59.796176+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.796176+0000', 'last_undegraded': '2024-03-06T07:22:59.796176+0000', 'last_fullsized': '2024-03-06T07:22:59.796176+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:04.753 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.0 is not active+clean 2024-03-06T07:23:04.753 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.798677+0000', 'last_change': '2024-03-06T07:22:59.798677+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.798677+0000', 'last_undegraded': '2024-03-06T07:22:59.798677+0000', 'last_fullsized': '2024-03-06T07:22:59.798677+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:04.753 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1 is not active+clean 2024-03-06T07:23:04.753 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.799865+0000', 'last_change': '2024-03-06T07:22:59.799865+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.799865+0000', 'last_undegraded': '2024-03-06T07:22:59.799865+0000', 'last_fullsized': '2024-03-06T07:22:59.799865+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:23:04.753 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.4 is not active+clean 2024-03-06T07:23:04.753 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:59.800675+0000', 'last_change': '2024-03-06T07:22:59.800675+0000', 'last_active': '2024-03-06T07:22:59.788978+0000', 'last_peered': '2024-03-06T07:22:59.788978+0000', 'last_clean': '2024-03-06T07:22:59.788978+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:59.800675+0000', 'last_undegraded': '2024-03-06T07:22:59.800675+0000', 'last_fullsized': '2024-03-06T07:22:59.800675+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:59.788978+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:23:04.754 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:23:04.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.894+0000 7f44c7ec9640 1 Processor -- start 2024-03-06T07:23:04.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.894+0000 7f44c7ec9640 1 -- start start 2024-03-06T07:23:04.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.895+0000 7f44c7ec9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f44c006b6c0 0x7f44c0168ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:04.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.895+0000 7f44c7ec9640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44c00ce510 con 0x7f44c00b0240 2024-03-06T07:23:04.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.895+0000 7f44c7ec9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44c0161cf0 con 0x7f44c01586c0 2024-03-06T07:23:04.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.895+0000 7f44c7ec9640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44c0158aa0 con 0x7f44c006b6c0 2024-03-06T07:23:04.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.895+0000 7f44c6466640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f44c006b6c0 0x7f44c0168ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:04.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.895+0000 7f44c5464640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f44c00b0240 0x7f44c006afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:50914/0 (socket says 172.21.15.83:50914) 2024-03-06T07:23:04.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.895+0000 7f44c5464640 1 -- 172.21.15.83:0/1118953679 learned_addr learned my addr 172.21.15.83:0/1118953679 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:23:04.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.896+0000 7f44c6466640 1 -- 172.21.15.83:0/1118953679 >> v1:172.21.15.183:6789/0 conn(0x7f44c01586c0 legacy=0x7f44c015efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:04.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.896+0000 7f44c6466640 1 -- 172.21.15.83:0/1118953679 >> v1:172.21.15.83:6789/0 conn(0x7f44c00b0240 legacy=0x7f44c006afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:04.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.896+0000 7f44c4c63640 1 -- 172.21.15.83:0/1118953679 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2646683533 0 0) 0x7f44c00ce510 con 0x7f44c00b0240 2024-03-06T07:23:04.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.896+0000 7f44c6466640 1 -- 172.21.15.83:0/1118953679 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c0161fd0 con 0x7f44c006b6c0 2024-03-06T07:23:04.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.896+0000 7f44c6466640 1 --2- 172.21.15.83:0/1118953679 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44c006b6c0 0x7f44c0168ee0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f44b0009d90 tx=0x7f44b002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aecd01776f19128 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:04.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.896+0000 7f44c4c63640 1 -- 172.21.15.83:0/1118953679 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b0039070 con 0x7f44c006b6c0 2024-03-06T07:23:04.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.896+0000 7f44c4c63640 1 -- 172.21.15.83:0/1118953679 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44b002cc60 con 0x7f44c006b6c0 2024-03-06T07:23:04.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.896+0000 7f44c4c63640 1 -- 172.21.15.83:0/1118953679 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b0034660 con 0x7f44c006b6c0 2024-03-06T07:23:04.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.897+0000 7f44c7ec9640 1 -- 172.21.15.83:0/1118953679 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44c006b6c0 msgr2=0x7f44c0168ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:04.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.897+0000 7f44c7ec9640 1 --2- 172.21.15.83:0/1118953679 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44c006b6c0 0x7f44c0168ee0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f44b0009d90 tx=0x7f44b002beb0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.897+0000 7f44c7ec9640 1 -- 172.21.15.83:0/1118953679 shutdown_connections 2024-03-06T07:23:04.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.897+0000 7f44c7ec9640 1 --2- 172.21.15.83:0/1118953679 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44c006b6c0 0x7f44c0168ee0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.897+0000 7f44c7ec9640 1 -- 172.21.15.83:0/1118953679 >> 172.21.15.83:0/1118953679 conn(0x7f44c00d09d0 msgr2=0x7f44c00d0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:04.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.897+0000 7f44c7ec9640 1 -- 172.21.15.83:0/1118953679 shutdown_connections 2024-03-06T07:23:04.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.897+0000 7f44c7ec9640 1 -- 172.21.15.83:0/1118953679 wait complete. 2024-03-06T07:23:04.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.899+0000 7f44c7ec9640 1 Processor -- start 2024-03-06T07:23:04.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.899+0000 7f44c7ec9640 1 -- start start 2024-03-06T07:23:04.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.899+0000 7f44c7ec9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44c006b6c0 0x7f44c01e9300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:04.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.899+0000 7f44c7ec9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f44c00b0240 0x7f44c0157110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:04.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.899+0000 7f44c7ec9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44c01586c0 0x7f44c01e7960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:04.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.899+0000 7f44c7ec9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c0162870 con 0x7f44c006b6c0 2024-03-06T07:23:04.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.899+0000 7f44c7ec9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c0162590 con 0x7f44c00b0240 2024-03-06T07:23:04.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.899+0000 7f44c7ec9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44c00719e0 con 0x7f44c01586c0 2024-03-06T07:23:04.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.900+0000 7f44c5c65640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44c006b6c0 0x7f44c01e9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:04.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.900+0000 7f44c5c65640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44c006b6c0 0x7f44c01e9300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35424/0 (socket says 172.21.15.83:35424) 2024-03-06T07:23:04.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.900+0000 7f44c5c65640 1 -- 172.21.15.83:0/733719759 learned_addr learned my addr 172.21.15.83:0/733719759 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:04.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.900+0000 7f44c5464640 1 --2- 172.21.15.83:0/733719759 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f44c00b0240 0x7f44c0157110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:04.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.900+0000 7f44c6466640 1 --2- 172.21.15.83:0/733719759 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44c01586c0 0x7f44c01e7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:04.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.900+0000 7f44c5c65640 1 -- 172.21.15.83:0/733719759 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44c01586c0 msgr2=0x7f44c01e7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:04.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.900+0000 7f44c5c65640 1 --2- 172.21.15.83:0/733719759 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44c01586c0 0x7f44c01e7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.900+0000 7f44c5c65640 1 -- 172.21.15.83:0/733719759 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f44c00b0240 msgr2=0x7f44c0157110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:04.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.900+0000 7f44c5c65640 1 --2- 172.21.15.83:0/733719759 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f44c00b0240 0x7f44c0157110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.900+0000 7f44c5c65640 1 -- 172.21.15.83:0/733719759 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c01e8260 con 0x7f44c006b6c0 2024-03-06T07:23:04.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.900+0000 7f44c5c65640 1 --2- 172.21.15.83:0/733719759 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44c006b6c0 0x7f44c01e9300 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f44b800b3f0 tx=0x7f44b800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:04.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.900+0000 7f44be7fc640 1 -- 172.21.15.83:0/733719759 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b8015020 con 0x7f44c006b6c0 2024-03-06T07:23:04.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.900+0000 7f44be7fc640 1 -- 172.21.15.83:0/733719759 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44b8011830 con 0x7f44c006b6c0 2024-03-06T07:23:04.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.900+0000 7f44be7fc640 1 -- 172.21.15.83:0/733719759 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b8011c20 con 0x7f44c006b6c0 2024-03-06T07:23:04.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.900+0000 7f44c7ec9640 1 -- 172.21.15.83:0/733719759 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44c01e8540 con 0x7f44c006b6c0 2024-03-06T07:23:04.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.900+0000 7f44c7ec9640 1 -- 172.21.15.83:0/733719759 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44c01e8b00 con 0x7f44c006b6c0 2024-03-06T07:23:04.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.901+0000 7f44c7ec9640 1 -- 172.21.15.83:0/733719759 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44c004f330 con 0x7f44c006b6c0 2024-03-06T07:23:04.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.902+0000 7f44be7fc640 1 -- 172.21.15.83:0/733719759 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f44b8018020 con 0x7f44c006b6c0 2024-03-06T07:23:04.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.902+0000 7f44be7fc640 1 --2- 172.21.15.83:0/733719759 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f44ac06b4d0 0x7f44ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:04.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.902+0000 7f44be7fc640 1 -- 172.21.15.83:0/733719759 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f44b8089ec0 con 0x7f44c006b6c0 2024-03-06T07:23:04.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.902+0000 7f44c5464640 1 --2- 172.21.15.83:0/733719759 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f44ac06b4d0 0x7f44ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:04.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.903+0000 7f44c5464640 1 --2- 172.21.15.83:0/733719759 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f44ac06b4d0 0x7f44ac06d990 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f44c01f73c0 tx=0x7f44b4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:04.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:04.906+0000 7f44be7fc640 1 -- 172.21.15.83:0/733719759 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f44b801f030 con 0x7f44c006b6c0 2024-03-06T07:23:05.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.152+0000 7f44c7ec9640 1 -- 172.21.15.83:0/733719759 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f44c00d17a0 con 0x7f44c006b6c0 2024-03-06T07:23:05.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.152+0000 7f44be7fc640 1 -- 172.21.15.83:0/733719759 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1194 (secure 0 0 0) 0x7f44b80172c0 con 0x7f44c006b6c0 2024-03-06T07:23:05.154 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:23:05.154 INFO:teuthology.orchestra.run.smithi083.stdout:{"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":90,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709705,"num_in_osds":8,"osd_in_since":1709709696,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":225280000,"bytes_avail":772868833280,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":13,"id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"a","status":"up:active","gid":5244}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-03-06T07:22:54.331751+0000","services":{"mds":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:23:05.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.154+0000 7f44c7ec9640 1 -- 172.21.15.83:0/733719759 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f44ac06b4d0 msgr2=0x7f44ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.154+0000 7f44c7ec9640 1 --2- 172.21.15.83:0/733719759 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f44ac06b4d0 0x7f44ac06d990 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f44c01f73c0 tx=0x7f44b4009040 comp rx=0 tx=0).stop 2024-03-06T07:23:05.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.154+0000 7f44c7ec9640 1 -- 172.21.15.83:0/733719759 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44c006b6c0 msgr2=0x7f44c01e9300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.154+0000 7f44c7ec9640 1 --2- 172.21.15.83:0/733719759 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44c006b6c0 0x7f44c01e9300 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f44b800b3f0 tx=0x7f44b800b8b0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.154+0000 7f44c7ec9640 1 -- 172.21.15.83:0/733719759 shutdown_connections 2024-03-06T07:23:05.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.154+0000 7f44c7ec9640 1 --2- 172.21.15.83:0/733719759 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f44ac06b4d0 0x7f44ac06d990 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.154+0000 7f44c7ec9640 1 --2- 172.21.15.83:0/733719759 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44c01586c0 0x7f44c01e7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.154+0000 7f44c7ec9640 1 --2- 172.21.15.83:0/733719759 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f44c00b0240 0x7f44c0157110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.154+0000 7f44c7ec9640 1 --2- 172.21.15.83:0/733719759 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44c006b6c0 0x7f44c01e9300 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.154+0000 7f44c7ec9640 1 -- 172.21.15.83:0/733719759 >> 172.21.15.83:0/733719759 conn(0x7f44c00d09d0 msgr2=0x7f44c00b0950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:05.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.154+0000 7f44c7ec9640 1 -- 172.21.15.83:0/733719759 shutdown_connections 2024-03-06T07:23:05.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.154+0000 7f44c7ec9640 1 -- 172.21.15.83:0/733719759 wait complete. 2024-03-06T07:23:05.167 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:23:05.167 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:23:05.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.266+0000 7faefe9a0640 1 Processor -- start 2024-03-06T07:23:05.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.267+0000 7faefe9a0640 1 -- start start 2024-03-06T07:23:05.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.267+0000 7faefe9a0640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7faef815dfe0 0x7faef80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.267+0000 7faefe9a0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7faef80cf830 0x7faef80d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.267+0000 7faefe9a0640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faef80b3430 con 0x7faef80d27f0 2024-03-06T07:23:05.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.267+0000 7faefe9a0640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faef80b31e0 con 0x7faef80cf830 2024-03-06T07:23:05.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.267+0000 7faefe9a0640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faef80b3d60 con 0x7faef815dfe0 2024-03-06T07:23:05.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.267+0000 7faef77fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7faef80cf830 0x7faef80d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.267+0000 7faefcf3d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7faef80d27f0 0x7faef80d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:50928/0 (socket says 172.21.15.83:50928) 2024-03-06T07:23:05.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.267+0000 7faefcf3d640 1 -- 172.21.15.83:0/1058894940 learned_addr learned my addr 172.21.15.83:0/1058894940 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:23:05.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.268+0000 7faef7fff640 1 --2- 172.21.15.83:0/1058894940 >> v2:172.21.15.183:3300/0 conn(0x7faef815dfe0 0x7faef80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.268+0000 7faef77fe640 1 -- 172.21.15.83:0/1058894940 >> v2:172.21.15.183:3300/0 conn(0x7faef815dfe0 msgr2=0x7faef80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:05.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.268+0000 7faef77fe640 1 --2- 172.21.15.83:0/1058894940 >> v2:172.21.15.183:3300/0 conn(0x7faef815dfe0 0x7faef80cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.268+0000 7faef77fe640 1 -- 172.21.15.83:0/1058894940 >> v1:172.21.15.83:6789/0 conn(0x7faef80d27f0 legacy=0x7faef80d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.268+0000 7faef77fe640 1 -- 172.21.15.83:0/1058894940 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faef80b3710 con 0x7faef80cf830 2024-03-06T07:23:05.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.268+0000 7faef6ffd640 1 -- 172.21.15.83:0/1058894940 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3643769980 0 0) 0x7faef80b3430 con 0x7faef80d27f0 2024-03-06T07:23:05.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.268+0000 7faef7fff640 1 --2- 172.21.15.83:0/1058894940 >> v2:172.21.15.183:3300/0 conn(0x7faef815dfe0 0x7faef80cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:05.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.268+0000 7faef77fe640 1 --2- 172.21.15.83:0/1058894940 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faef80cf830 0x7faef80d22b0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7faee80097a0 tx=0x7faee802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f579c8c546f4e786 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:05.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.269+0000 7faef6ffd640 1 -- 172.21.15.83:0/1058894940 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee8035020 con 0x7faef80cf830 2024-03-06T07:23:05.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.269+0000 7faef6ffd640 1 -- 172.21.15.83:0/1058894940 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faee8031830 con 0x7faef80cf830 2024-03-06T07:23:05.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.269+0000 7faef6ffd640 1 -- 172.21.15.83:0/1058894940 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee8031c00 con 0x7faef80cf830 2024-03-06T07:23:05.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.269+0000 7faefe9a0640 1 -- 172.21.15.83:0/1058894940 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faef80cf830 msgr2=0x7faef80d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.269+0000 7faefe9a0640 1 --2- 172.21.15.83:0/1058894940 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faef80cf830 0x7faef80d22b0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7faee80097a0 tx=0x7faee802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.269+0000 7faefe9a0640 1 -- 172.21.15.83:0/1058894940 shutdown_connections 2024-03-06T07:23:05.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.269+0000 7faefe9a0640 1 --2- 172.21.15.83:0/1058894940 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faef80cf830 0x7faef80d22b0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.269+0000 7faefe9a0640 1 --2- 172.21.15.83:0/1058894940 >> v2:172.21.15.183:3300/0 conn(0x7faef815dfe0 0x7faef80cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.269+0000 7faefe9a0640 1 -- 172.21.15.83:0/1058894940 >> 172.21.15.83:0/1058894940 conn(0x7faef80d3cd0 msgr2=0x7faef80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:05.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.269+0000 7faefe9a0640 1 -- 172.21.15.83:0/1058894940 shutdown_connections 2024-03-06T07:23:05.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.269+0000 7faefe9a0640 1 -- 172.21.15.83:0/1058894940 wait complete. 2024-03-06T07:23:05.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.270+0000 7faefe9a0640 1 Processor -- start 2024-03-06T07:23:05.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.271+0000 7faefe9a0640 1 -- start start 2024-03-06T07:23:05.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.271+0000 7faefe9a0640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faef80cf830 0x7faef815be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.271+0000 7faefe9a0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faef80d27f0 0x7faef815e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.271+0000 7faefe9a0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faef815dfe0 0x7faef8160c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.271+0000 7faefe9a0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef80d1df0 con 0x7faef80d27f0 2024-03-06T07:23:05.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.271+0000 7faefe9a0640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef80d1c70 con 0x7faef80cf830 2024-03-06T07:23:05.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.271+0000 7faefe9a0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faef815e3c0 con 0x7faef815dfe0 2024-03-06T07:23:05.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.271+0000 7faef77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faef80d27f0 0x7faef815e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.272+0000 7faef77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faef80d27f0 0x7faef815e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35452/0 (socket says 172.21.15.83:35452) 2024-03-06T07:23:05.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.272+0000 7faef77fe640 1 -- 172.21.15.83:0/3813412226 learned_addr learned my addr 172.21.15.83:0/3813412226 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:05.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.272+0000 7faefcf3d640 1 --2- 172.21.15.83:0/3813412226 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faef815dfe0 0x7faef8160c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.272+0000 7faef7fff640 1 --2- 172.21.15.83:0/3813412226 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faef80cf830 0x7faef815be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.272+0000 7faef77fe640 1 -- 172.21.15.83:0/3813412226 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faef815dfe0 msgr2=0x7faef8160c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.272+0000 7faef77fe640 1 --2- 172.21.15.83:0/3813412226 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faef815dfe0 0x7faef8160c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.272+0000 7faef77fe640 1 -- 172.21.15.83:0/3813412226 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faef80cf830 msgr2=0x7faef815be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.272+0000 7faef77fe640 1 --2- 172.21.15.83:0/3813412226 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faef80cf830 0x7faef815be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.272+0000 7faef77fe640 1 -- 172.21.15.83:0/3813412226 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faef815f020 con 0x7faef80d27f0 2024-03-06T07:23:05.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.272+0000 7faefcf3d640 1 --2- 172.21.15.83:0/3813412226 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faef815dfe0 0x7faef8160c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:05.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.272+0000 7faef7fff640 1 --2- 172.21.15.83:0/3813412226 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faef80cf830 0x7faef815be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:05.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.272+0000 7faef77fe640 1 --2- 172.21.15.83:0/3813412226 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faef80d27f0 0x7faef815e6f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7faee8034040 tx=0x7faee80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:05.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.272+0000 7faef4ff9640 1 -- 172.21.15.83:0/3813412226 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee8040070 con 0x7faef80d27f0 2024-03-06T07:23:05.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.272+0000 7faef4ff9640 1 -- 172.21.15.83:0/3813412226 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faee8039a20 con 0x7faef80d27f0 2024-03-06T07:23:05.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.272+0000 7faefe9a0640 1 -- 172.21.15.83:0/3813412226 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faef815f300 con 0x7faef80d27f0 2024-03-06T07:23:05.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.272+0000 7faef4ff9640 1 -- 172.21.15.83:0/3813412226 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee8035040 con 0x7faef80d27f0 2024-03-06T07:23:05.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.272+0000 7faefe9a0640 1 -- 172.21.15.83:0/3813412226 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faef815f8c0 con 0x7faef80d27f0 2024-03-06T07:23:05.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.273+0000 7faefe9a0640 1 -- 172.21.15.83:0/3813412226 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faef804f330 con 0x7faef80d27f0 2024-03-06T07:23:05.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.274+0000 7faef4ff9640 1 -- 172.21.15.83:0/3813412226 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faee8038020 con 0x7faef80d27f0 2024-03-06T07:23:05.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.274+0000 7faef4ff9640 1 --2- 172.21.15.83:0/3813412226 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7faee406b4d0 0x7faee406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.274+0000 7faef4ff9640 1 -- 172.21.15.83:0/3813412226 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faee80abc20 con 0x7faef80d27f0 2024-03-06T07:23:05.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.274+0000 7faef7fff640 1 --2- 172.21.15.83:0/3813412226 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7faee406b4d0 0x7faee406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.274+0000 7faef7fff640 1 --2- 172.21.15.83:0/3813412226 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7faee406b4d0 0x7faee406d990 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7faef80d3050 tx=0x7faeec008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:05.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.278+0000 7faef4ff9640 1 -- 172.21.15.83:0/3813412226 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faee807aaa0 con 0x7faef80d27f0 2024-03-06T07:23:05.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.552+0000 7faefe9a0640 1 -- 172.21.15.83:0/3813412226 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7faef80ceae0 con 0x7faee406b4d0 2024-03-06T07:23:05.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.558+0000 7faef4ff9640 1 -- 172.21.15.83:0/3813412226 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246633 (secure 0 0 0) 0x7faef80ceae0 con 0x7faee406b4d0 2024-03-06T07:23:05.562 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:23:05.563 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-06T07:23:05.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.561+0000 7faefe9a0640 1 -- 172.21.15.83:0/3813412226 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7faee406b4d0 msgr2=0x7faee406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.561+0000 7faefe9a0640 1 --2- 172.21.15.83:0/3813412226 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7faee406b4d0 0x7faee406d990 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7faef80d3050 tx=0x7faeec008040 comp rx=0 tx=0).stop 2024-03-06T07:23:05.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.561+0000 7faefe9a0640 1 -- 172.21.15.83:0/3813412226 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faef80d27f0 msgr2=0x7faef815e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.561+0000 7faefe9a0640 1 --2- 172.21.15.83:0/3813412226 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faef80d27f0 0x7faef815e6f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7faee8034040 tx=0x7faee80046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.561+0000 7faefe9a0640 1 -- 172.21.15.83:0/3813412226 shutdown_connections 2024-03-06T07:23:05.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.561+0000 7faefe9a0640 1 --2- 172.21.15.83:0/3813412226 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7faee406b4d0 0x7faee406d990 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.561+0000 7faefe9a0640 1 --2- 172.21.15.83:0/3813412226 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faef815dfe0 0x7faef8160c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.561+0000 7faefe9a0640 1 --2- 172.21.15.83:0/3813412226 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faef80d27f0 0x7faef815e6f0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.561+0000 7faefe9a0640 1 --2- 172.21.15.83:0/3813412226 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faef80cf830 0x7faef815be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.561+0000 7faefe9a0640 1 -- 172.21.15.83:0/3813412226 >> 172.21.15.83:0/3813412226 conn(0x7faef80d3cd0 msgr2=0x7faef80ccdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:05.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.561+0000 7faefe9a0640 1 -- 172.21.15.83:0/3813412226 shutdown_connections 2024-03-06T07:23:05.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:05.561+0000 7faefe9a0640 1 -- 172.21.15.83:0/3813412226 wait complete. 2024-03-06T07:23:05.574 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":66,"stamp":"2024-03-06T07:23:04.333421+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":35,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220000,"kb_used_data":3832,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754720,"statfs":{"total":773094113280,"available":772868833280,"internally_reserved":0,"allocated":3923968,"data_stored":2227948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12757,"internal_metadata":221236779},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.543580"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797969+0000","last_change":"2024-03-06T07:22:59.797969+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797969+0000","last_undegraded":"2024-03-06T07:22:59.797969+0000","last_fullsized":"2024-03-06T07:22:59.797969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.799502+0000","last_change":"2024-03-06T07:22:59.799502+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.799502+0000","last_undegraded":"2024-03-06T07:22:59.799502+0000","last_fullsized":"2024-03-06T07:22:59.799502+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.804718+0000","last_change":"2024-03-06T07:22:59.804718+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.804718+0000","last_undegraded":"2024-03-06T07:22:59.804718+0000","last_fullsized":"2024-03-06T07:22:59.804718+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.800525+0000","last_change":"2024-03-06T07:22:59.800525+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.800525+0000","last_undegraded":"2024-03-06T07:22:59.800525+0000","last_fullsized":"2024-03-06T07:22:59.800525+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.798524+0000","last_change":"2024-03-06T07:22:59.798524+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.798524+0000","last_undegraded":"2024-03-06T07:22:59.798524+0000","last_fullsized":"2024-03-06T07:22:59.798524+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797989+0000","last_change":"2024-03-06T07:22:59.797989+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797989+0000","last_undegraded":"2024-03-06T07:22:59.797989+0000","last_fullsized":"2024-03-06T07:22:59.797989+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.801589+0000","last_change":"2024-03-06T07:22:59.801589+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.801589+0000","last_undegraded":"2024-03-06T07:22:59.801589+0000","last_fullsized":"2024-03-06T07:22:59.801589+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.804790+0000","last_change":"2024-03-06T07:22:59.804790+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.804790+0000","last_undegraded":"2024-03-06T07:22:59.804790+0000","last_fullsized":"2024-03-06T07:22:59.804790+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.799988+0000","last_change":"2024-03-06T07:22:59.799988+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.799988+0000","last_undegraded":"2024-03-06T07:22:59.799988+0000","last_fullsized":"2024-03-06T07:22:59.799988+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.799777+0000","last_change":"2024-03-06T07:22:59.799777+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.799777+0000","last_undegraded":"2024-03-06T07:22:59.799777+0000","last_fullsized":"2024-03-06T07:22:59.799777+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.801040+0000","last_change":"2024-03-06T07:22:59.801040+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.801040+0000","last_undegraded":"2024-03-06T07:22:59.801040+0000","last_fullsized":"2024-03-06T07:22:59.801040+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.803466+0000","last_change":"2024-03-06T07:22:59.803466+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.803466+0000","last_undegraded":"2024-03-06T07:22:59.803466+0000","last_fullsized":"2024-03-06T07:22:59.803466+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.798121+0000","last_change":"2024-03-06T07:22:59.798121+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.798121+0000","last_undegraded":"2024-03-06T07:22:59.798121+0000","last_fullsized":"2024-03-06T07:22:59.798121+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797601+0000","last_change":"2024-03-06T07:22:59.797601+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797601+0000","last_undegraded":"2024-03-06T07:22:59.797601+0000","last_fullsized":"2024-03-06T07:22:59.797601+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.800974+0000","last_change":"2024-03-06T07:22:59.800974+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.800974+0000","last_undegraded":"2024-03-06T07:22:59.800974+0000","last_fullsized":"2024-03-06T07:22:59.800974+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.801086+0000","last_change":"2024-03-06T07:22:59.801086+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.801086+0000","last_undegraded":"2024-03-06T07:22:59.801086+0000","last_fullsized":"2024-03-06T07:22:59.801086+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.802385+0000","last_change":"2024-03-06T07:22:59.802385+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.802385+0000","last_undegraded":"2024-03-06T07:22:59.802385+0000","last_fullsized":"2024-03-06T07:22:59.802385+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.799075+0000","last_change":"2024-03-06T07:22:59.799075+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.799075+0000","last_undegraded":"2024-03-06T07:22:59.799075+0000","last_fullsized":"2024-03-06T07:22:59.799075+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797394+0000","last_change":"2024-03-06T07:22:59.797394+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797394+0000","last_undegraded":"2024-03-06T07:22:59.797394+0000","last_fullsized":"2024-03-06T07:22:59.797394+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.798311+0000","last_change":"2024-03-06T07:22:59.798311+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.798311+0000","last_undegraded":"2024-03-06T07:22:59.798311+0000","last_fullsized":"2024-03-06T07:22:59.798311+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.800053+0000","last_change":"2024-03-06T07:22:59.800053+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.800053+0000","last_undegraded":"2024-03-06T07:22:59.800053+0000","last_fullsized":"2024-03-06T07:22:59.800053+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796916+0000","last_change":"2024-03-06T07:22:59.796916+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796916+0000","last_undegraded":"2024-03-06T07:22:59.796916+0000","last_fullsized":"2024-03-06T07:22:59.796916+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796381+0000","last_change":"2024-03-06T07:22:59.796381+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796381+0000","last_undegraded":"2024-03-06T07:22:59.796381+0000","last_fullsized":"2024-03-06T07:22:59.796381+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.806940+0000","last_change":"2024-03-06T07:22:59.806940+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.806940+0000","last_undegraded":"2024-03-06T07:22:59.806940+0000","last_fullsized":"2024-03-06T07:22:59.806940+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.799846+0000","last_change":"2024-03-06T07:22:59.799846+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.799846+0000","last_undegraded":"2024-03-06T07:22:59.799846+0000","last_fullsized":"2024-03-06T07:22:59.799846+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.798684+0000","last_change":"2024-03-06T07:22:59.798684+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.798684+0000","last_undegraded":"2024-03-06T07:22:59.798684+0000","last_fullsized":"2024-03-06T07:22:59.798684+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.800428+0000","last_change":"2024-03-06T07:22:59.800428+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.800428+0000","last_undegraded":"2024-03-06T07:22:59.800428+0000","last_fullsized":"2024-03-06T07:22:59.800428+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.803078+0000","last_change":"2024-03-06T07:22:59.803078+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.803078+0000","last_undegraded":"2024-03-06T07:22:59.803078+0000","last_fullsized":"2024-03-06T07:22:59.803078+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.799594+0000","last_change":"2024-03-06T07:22:59.799594+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.799594+0000","last_undegraded":"2024-03-06T07:22:59.799594+0000","last_fullsized":"2024-03-06T07:22:59.799594+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.801930+0000","last_change":"2024-03-06T07:22:59.801930+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.801930+0000","last_undegraded":"2024-03-06T07:22:59.801930+0000","last_fullsized":"2024-03-06T07:22:59.801930+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797043+0000","last_change":"2024-03-06T07:22:59.797043+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797043+0000","last_undegraded":"2024-03-06T07:22:59.797043+0000","last_fullsized":"2024-03-06T07:22:59.797043+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.804395+0000","last_change":"2024-03-06T07:22:59.804395+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.804395+0000","last_undegraded":"2024-03-06T07:22:59.804395+0000","last_fullsized":"2024-03-06T07:22:59.804395+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796359+0000","last_change":"2024-03-06T07:22:59.796359+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796359+0000","last_undegraded":"2024-03-06T07:22:59.796359+0000","last_fullsized":"2024-03-06T07:22:59.796359+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.812031+0000","last_change":"2024-03-06T07:22:59.810363+0000","last_active":"2024-03-06T07:22:59.812031+0000","last_peered":"2024-03-06T07:22:59.812031+0000","last_clean":"2024-03-06T07:22:59.812031+0000","last_became_active":"2024-03-06T07:22:59.810032+0000","last_became_peered":"2024-03-06T07:22:59.810032+0000","last_unstale":"2024-03-06T07:22:59.812031+0000","last_undegraded":"2024-03-06T07:22:59.812031+0000","last_fullsized":"2024-03-06T07:22:59.812031+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:36:12.264660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797548+0000","last_change":"2024-03-06T07:22:59.797548+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797548+0000","last_undegraded":"2024-03-06T07:22:59.797548+0000","last_fullsized":"2024-03-06T07:22:59.797548+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.811396+0000","last_change":"2024-03-06T07:22:59.810687+0000","last_active":"2024-03-06T07:22:59.811396+0000","last_peered":"2024-03-06T07:22:59.811396+0000","last_clean":"2024-03-06T07:22:59.811396+0000","last_became_active":"2024-03-06T07:22:59.810405+0000","last_became_peered":"2024-03-06T07:22:59.810405+0000","last_unstale":"2024-03-06T07:22:59.811396+0000","last_undegraded":"2024-03-06T07:22:59.811396+0000","last_fullsized":"2024-03-06T07:22:59.811396+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:24:26.919565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.798640+0000","last_change":"2024-03-06T07:22:59.798640+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.798640+0000","last_undegraded":"2024-03-06T07:22:59.798640+0000","last_fullsized":"2024-03-06T07:22:59.798640+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.807686+0000","last_change":"2024-03-06T07:22:59.807093+0000","last_active":"2024-03-06T07:22:59.807686+0000","last_peered":"2024-03-06T07:22:59.807686+0000","last_clean":"2024-03-06T07:22:59.807686+0000","last_became_active":"2024-03-06T07:22:59.806858+0000","last_became_peered":"2024-03-06T07:22:59.806858+0000","last_unstale":"2024-03-06T07:22:59.807686+0000","last_undegraded":"2024-03-06T07:22:59.807686+0000","last_fullsized":"2024-03-06T07:22:59.807686+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:38:31.114570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.799435+0000","last_change":"2024-03-06T07:22:59.799435+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.799435+0000","last_undegraded":"2024-03-06T07:22:59.799435+0000","last_fullsized":"2024-03-06T07:22:59.799435+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.809136+0000","last_change":"2024-03-06T07:22:59.805788+0000","last_active":"2024-03-06T07:22:59.809136+0000","last_peered":"2024-03-06T07:22:59.809136+0000","last_clean":"2024-03-06T07:22:59.809136+0000","last_became_active":"2024-03-06T07:22:59.805463+0000","last_became_peered":"2024-03-06T07:22:59.805463+0000","last_unstale":"2024-03-06T07:22:59.809136+0000","last_undegraded":"2024-03-06T07:22:59.809136+0000","last_fullsized":"2024-03-06T07:22:59.809136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:14.437437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.801363+0000","last_change":"2024-03-06T07:22:59.801363+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.801363+0000","last_undegraded":"2024-03-06T07:22:59.801363+0000","last_fullsized":"2024-03-06T07:22:59.801363+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.811699+0000","last_change":"2024-03-06T07:22:59.808162+0000","last_active":"2024-03-06T07:22:59.811699+0000","last_peered":"2024-03-06T07:22:59.811699+0000","last_clean":"2024-03-06T07:22:59.811699+0000","last_became_active":"2024-03-06T07:22:59.807875+0000","last_became_peered":"2024-03-06T07:22:59.807875+0000","last_unstale":"2024-03-06T07:22:59.811699+0000","last_undegraded":"2024-03-06T07:22:59.811699+0000","last_fullsized":"2024-03-06T07:22:59.811699+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:53.014712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.802462+0000","last_change":"2024-03-06T07:22:59.802462+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.802462+0000","last_undegraded":"2024-03-06T07:22:59.802462+0000","last_fullsized":"2024-03-06T07:22:59.802462+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.811787+0000","last_change":"2024-03-06T07:22:59.806399+0000","last_active":"2024-03-06T07:22:59.811787+0000","last_peered":"2024-03-06T07:22:59.811787+0000","last_clean":"2024-03-06T07:22:59.811787+0000","last_became_active":"2024-03-06T07:22:59.805990+0000","last_became_peered":"2024-03-06T07:22:59.805990+0000","last_unstale":"2024-03-06T07:22:59.811787+0000","last_undegraded":"2024-03-06T07:22:59.811787+0000","last_fullsized":"2024-03-06T07:22:59.811787+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:07.271082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797495+0000","last_change":"2024-03-06T07:22:59.797495+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797495+0000","last_undegraded":"2024-03-06T07:22:59.797495+0000","last_fullsized":"2024-03-06T07:22:59.797495+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.806357+0000","last_change":"2024-03-06T07:22:59.805251+0000","last_active":"2024-03-06T07:22:59.806357+0000","last_peered":"2024-03-06T07:22:59.806357+0000","last_clean":"2024-03-06T07:22:59.806357+0000","last_became_active":"2024-03-06T07:22:59.804769+0000","last_became_peered":"2024-03-06T07:22:59.804769+0000","last_unstale":"2024-03-06T07:22:59.806357+0000","last_undegraded":"2024-03-06T07:22:59.806357+0000","last_fullsized":"2024-03-06T07:22:59.806357+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:30:21.970435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797342+0000","last_change":"2024-03-06T07:22:59.797342+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797342+0000","last_undegraded":"2024-03-06T07:22:59.797342+0000","last_fullsized":"2024-03-06T07:22:59.797342+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.810668+0000","last_change":"2024-03-06T07:22:59.804481+0000","last_active":"2024-03-06T07:22:59.810668+0000","last_peered":"2024-03-06T07:22:59.810668+0000","last_clean":"2024-03-06T07:22:59.810668+0000","last_became_active":"2024-03-06T07:22:59.803413+0000","last_became_peered":"2024-03-06T07:22:59.803413+0000","last_unstale":"2024-03-06T07:22:59.810668+0000","last_undegraded":"2024-03-06T07:22:59.810668+0000","last_fullsized":"2024-03-06T07:22:59.810668+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:14.437437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.802007+0000","last_change":"2024-03-06T07:22:59.802007+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.802007+0000","last_undegraded":"2024-03-06T07:22:59.802007+0000","last_fullsized":"2024-03-06T07:22:59.802007+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.810221+0000","last_change":"2024-03-06T07:22:59.808659+0000","last_active":"2024-03-06T07:22:59.810221+0000","last_peered":"2024-03-06T07:22:59.810221+0000","last_clean":"2024-03-06T07:22:59.810221+0000","last_became_active":"2024-03-06T07:22:59.808416+0000","last_became_peered":"2024-03-06T07:22:59.808416+0000","last_unstale":"2024-03-06T07:22:59.810221+0000","last_undegraded":"2024-03-06T07:22:59.810221+0000","last_fullsized":"2024-03-06T07:22:59.810221+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:32.447994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.798133+0000","last_change":"2024-03-06T07:22:59.798133+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.798133+0000","last_undegraded":"2024-03-06T07:22:59.798133+0000","last_fullsized":"2024-03-06T07:22:59.798133+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.805530+0000","last_change":"2024-03-06T07:22:59.804638+0000","last_active":"2024-03-06T07:22:59.805530+0000","last_peered":"2024-03-06T07:22:59.805530+0000","last_clean":"2024-03-06T07:22:59.805530+0000","last_became_active":"2024-03-06T07:22:59.804026+0000","last_became_peered":"2024-03-06T07:22:59.804026+0000","last_unstale":"2024-03-06T07:22:59.805530+0000","last_undegraded":"2024-03-06T07:22:59.805530+0000","last_fullsized":"2024-03-06T07:22:59.805530+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:04:10.795313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.798485+0000","last_change":"2024-03-06T07:22:59.798485+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.798485+0000","last_undegraded":"2024-03-06T07:22:59.798485+0000","last_fullsized":"2024-03-06T07:22:59.798485+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.816328+0000","last_change":"2024-03-06T07:22:59.815296+0000","last_active":"2024-03-06T07:22:59.816328+0000","last_peered":"2024-03-06T07:22:59.816328+0000","last_clean":"2024-03-06T07:22:59.816328+0000","last_became_active":"2024-03-06T07:22:59.815033+0000","last_became_peered":"2024-03-06T07:22:59.815033+0000","last_unstale":"2024-03-06T07:22:59.816328+0000","last_undegraded":"2024-03-06T07:22:59.816328+0000","last_fullsized":"2024-03-06T07:22:59.816328+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:29.015317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.808497+0000","last_change":"2024-03-06T07:22:59.808497+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.808497+0000","last_undegraded":"2024-03-06T07:22:59.808497+0000","last_fullsized":"2024-03-06T07:22:59.808497+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.808772+0000","last_change":"2024-03-06T07:22:59.807399+0000","last_active":"2024-03-06T07:22:59.808772+0000","last_peered":"2024-03-06T07:22:59.808772+0000","last_clean":"2024-03-06T07:22:59.808772+0000","last_became_active":"2024-03-06T07:22:59.807150+0000","last_became_peered":"2024-03-06T07:22:59.807150+0000","last_unstale":"2024-03-06T07:22:59.808772+0000","last_undegraded":"2024-03-06T07:22:59.808772+0000","last_fullsized":"2024-03-06T07:22:59.808772+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:19.337313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.798310+0000","last_change":"2024-03-06T07:22:59.798310+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.798310+0000","last_undegraded":"2024-03-06T07:22:59.798310+0000","last_fullsized":"2024-03-06T07:22:59.798310+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.811735+0000","last_change":"2024-03-06T07:22:59.809344+0000","last_active":"2024-03-06T07:22:59.811735+0000","last_peered":"2024-03-06T07:22:59.811735+0000","last_clean":"2024-03-06T07:22:59.811735+0000","last_became_active":"2024-03-06T07:22:59.809076+0000","last_became_peered":"2024-03-06T07:22:59.809076+0000","last_unstale":"2024-03-06T07:22:59.811735+0000","last_undegraded":"2024-03-06T07:22:59.811735+0000","last_fullsized":"2024-03-06T07:22:59.811735+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:12.683272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796615+0000","last_change":"2024-03-06T07:22:59.796615+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796615+0000","last_undegraded":"2024-03-06T07:22:59.796615+0000","last_fullsized":"2024-03-06T07:22:59.796615+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.809522+0000","last_change":"2024-03-06T07:22:59.807007+0000","last_active":"2024-03-06T07:22:59.809522+0000","last_peered":"2024-03-06T07:22:59.809522+0000","last_clean":"2024-03-06T07:22:59.809522+0000","last_became_active":"2024-03-06T07:22:59.806750+0000","last_became_peered":"2024-03-06T07:22:59.806750+0000","last_unstale":"2024-03-06T07:22:59.809522+0000","last_undegraded":"2024-03-06T07:22:59.809522+0000","last_fullsized":"2024-03-06T07:22:59.809522+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:24:26.919565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.804026+0000","last_change":"2024-03-06T07:22:59.804026+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.804026+0000","last_undegraded":"2024-03-06T07:22:59.804026+0000","last_fullsized":"2024-03-06T07:22:59.804026+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.809695+0000","last_change":"2024-03-06T07:22:59.808043+0000","last_active":"2024-03-06T07:22:59.809695+0000","last_peered":"2024-03-06T07:22:59.809695+0000","last_clean":"2024-03-06T07:22:59.809695+0000","last_became_active":"2024-03-06T07:22:59.807794+0000","last_became_peered":"2024-03-06T07:22:59.807794+0000","last_unstale":"2024-03-06T07:22:59.809695+0000","last_undegraded":"2024-03-06T07:22:59.809695+0000","last_fullsized":"2024-03-06T07:22:59.809695+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:08:18.005082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796469+0000","last_change":"2024-03-06T07:22:59.796469+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796469+0000","last_undegraded":"2024-03-06T07:22:59.796469+0000","last_fullsized":"2024-03-06T07:22:59.796469+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.807259+0000","last_change":"2024-03-06T07:22:59.806459+0000","last_active":"2024-03-06T07:22:59.807259+0000","last_peered":"2024-03-06T07:22:59.807259+0000","last_clean":"2024-03-06T07:22:59.807259+0000","last_became_active":"2024-03-06T07:22:59.806194+0000","last_became_peered":"2024-03-06T07:22:59.806194+0000","last_unstale":"2024-03-06T07:22:59.807259+0000","last_undegraded":"2024-03-06T07:22:59.807259+0000","last_fullsized":"2024-03-06T07:22:59.807259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:53.894652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796668+0000","last_change":"2024-03-06T07:22:59.796668+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796668+0000","last_undegraded":"2024-03-06T07:22:59.796668+0000","last_fullsized":"2024-03-06T07:22:59.796668+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.810182+0000","last_change":"2024-03-06T07:22:59.808555+0000","last_active":"2024-03-06T07:22:59.810182+0000","last_peered":"2024-03-06T07:22:59.810182+0000","last_clean":"2024-03-06T07:22:59.810182+0000","last_became_active":"2024-03-06T07:22:59.808284+0000","last_became_peered":"2024-03-06T07:22:59.808284+0000","last_unstale":"2024-03-06T07:22:59.810182+0000","last_undegraded":"2024-03-06T07:22:59.810182+0000","last_fullsized":"2024-03-06T07:22:59.810182+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:30:21.970435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.803648+0000","last_change":"2024-03-06T07:22:59.803648+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.803648+0000","last_undegraded":"2024-03-06T07:22:59.803648+0000","last_fullsized":"2024-03-06T07:22:59.803648+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.805500+0000","last_change":"2024-03-06T07:22:59.802730+0000","last_active":"2024-03-06T07:22:59.805500+0000","last_peered":"2024-03-06T07:22:59.805500+0000","last_clean":"2024-03-06T07:22:59.805500+0000","last_became_active":"2024-03-06T07:22:59.802461+0000","last_became_peered":"2024-03-06T07:22:59.802461+0000","last_unstale":"2024-03-06T07:22:59.805500+0000","last_undegraded":"2024-03-06T07:22:59.805500+0000","last_fullsized":"2024-03-06T07:22:59.805500+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:26:21.615251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.798752+0000","last_change":"2024-03-06T07:22:59.798752+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.798752+0000","last_undegraded":"2024-03-06T07:22:59.798752+0000","last_fullsized":"2024-03-06T07:22:59.798752+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.811762+0000","last_change":"2024-03-06T07:22:59.807016+0000","last_active":"2024-03-06T07:22:59.811762+0000","last_peered":"2024-03-06T07:22:59.811762+0000","last_clean":"2024-03-06T07:22:59.811762+0000","last_became_active":"2024-03-06T07:22:59.806733+0000","last_became_peered":"2024-03-06T07:22:59.806733+0000","last_unstale":"2024-03-06T07:22:59.811762+0000","last_undegraded":"2024-03-06T07:22:59.811762+0000","last_fullsized":"2024-03-06T07:22:59.811762+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:55.870572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797444+0000","last_change":"2024-03-06T07:22:59.797444+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797444+0000","last_undegraded":"2024-03-06T07:22:59.797444+0000","last_fullsized":"2024-03-06T07:22:59.797444+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.811731+0000","last_change":"2024-03-06T07:22:59.808721+0000","last_active":"2024-03-06T07:22:59.811731+0000","last_peered":"2024-03-06T07:22:59.811731+0000","last_clean":"2024-03-06T07:22:59.811731+0000","last_became_active":"2024-03-06T07:22:59.808465+0000","last_became_peered":"2024-03-06T07:22:59.808465+0000","last_unstale":"2024-03-06T07:22:59.811731+0000","last_undegraded":"2024-03-06T07:22:59.811731+0000","last_fullsized":"2024-03-06T07:22:59.811731+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:08.896012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796675+0000","last_change":"2024-03-06T07:22:59.796675+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796675+0000","last_undegraded":"2024-03-06T07:22:59.796675+0000","last_fullsized":"2024-03-06T07:22:59.796675+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.805543+0000","last_change":"2024-03-06T07:22:59.804367+0000","last_active":"2024-03-06T07:22:59.805543+0000","last_peered":"2024-03-06T07:22:59.805543+0000","last_clean":"2024-03-06T07:22:59.805543+0000","last_became_active":"2024-03-06T07:22:59.804115+0000","last_became_peered":"2024-03-06T07:22:59.804115+0000","last_unstale":"2024-03-06T07:22:59.805543+0000","last_undegraded":"2024-03-06T07:22:59.805543+0000","last_fullsized":"2024-03-06T07:22:59.805543+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:44:35.664305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797737+0000","last_change":"2024-03-06T07:22:59.797737+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797737+0000","last_undegraded":"2024-03-06T07:22:59.797737+0000","last_fullsized":"2024-03-06T07:22:59.797737+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.806310+0000","last_change":"2024-03-06T07:22:59.804749+0000","last_active":"2024-03-06T07:22:59.806310+0000","last_peered":"2024-03-06T07:22:59.806310+0000","last_clean":"2024-03-06T07:22:59.806310+0000","last_became_active":"2024-03-06T07:22:59.802839+0000","last_became_peered":"2024-03-06T07:22:59.802839+0000","last_unstale":"2024-03-06T07:22:59.806310+0000","last_undegraded":"2024-03-06T07:22:59.806310+0000","last_fullsized":"2024-03-06T07:22:59.806310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:04:10.795313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796191+0000","last_change":"2024-03-06T07:22:59.796191+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796191+0000","last_undegraded":"2024-03-06T07:22:59.796191+0000","last_fullsized":"2024-03-06T07:22:59.796191+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.813816+0000","last_change":"2024-03-06T07:22:59.810390+0000","last_active":"2024-03-06T07:22:59.813816+0000","last_peered":"2024-03-06T07:22:59.813816+0000","last_clean":"2024-03-06T07:22:59.813816+0000","last_became_active":"2024-03-06T07:22:59.810140+0000","last_became_peered":"2024-03-06T07:22:59.810140+0000","last_unstale":"2024-03-06T07:22:59.813816+0000","last_undegraded":"2024-03-06T07:22:59.813816+0000","last_fullsized":"2024-03-06T07:22:59.813816+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:13.131161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.802019+0000","last_change":"2024-03-06T07:22:59.802019+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.802019+0000","last_undegraded":"2024-03-06T07:22:59.802019+0000","last_fullsized":"2024-03-06T07:22:59.802019+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.814736+0000","last_change":"2024-03-06T07:22:59.812896+0000","last_active":"2024-03-06T07:22:59.814736+0000","last_peered":"2024-03-06T07:22:59.814736+0000","last_clean":"2024-03-06T07:22:59.814736+0000","last_became_active":"2024-03-06T07:22:59.812598+0000","last_became_peered":"2024-03-06T07:22:59.812598+0000","last_unstale":"2024-03-06T07:22:59.814736+0000","last_undegraded":"2024-03-06T07:22:59.814736+0000","last_fullsized":"2024-03-06T07:22:59.814736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:37.715070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.797040+0000","last_change":"2024-03-06T07:22:59.797040+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.797040+0000","last_undegraded":"2024-03-06T07:22:59.797040+0000","last_fullsized":"2024-03-06T07:22:59.797040+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.813596+0000","last_change":"2024-03-06T07:22:59.811226+0000","last_active":"2024-03-06T07:22:59.813596+0000","last_peered":"2024-03-06T07:22:59.813596+0000","last_clean":"2024-03-06T07:22:59.813596+0000","last_became_active":"2024-03-06T07:22:59.810822+0000","last_became_peered":"2024-03-06T07:22:59.810822+0000","last_unstale":"2024-03-06T07:22:59.813596+0000","last_undegraded":"2024-03-06T07:22:59.813596+0000","last_fullsized":"2024-03-06T07:22:59.813596+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:53.894652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.803232+0000","last_change":"2024-03-06T07:22:59.803232+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.803232+0000","last_undegraded":"2024-03-06T07:22:59.803232+0000","last_fullsized":"2024-03-06T07:22:59.803232+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.805549+0000","last_change":"2024-03-06T07:22:59.804853+0000","last_active":"2024-03-06T07:22:59.805549+0000","last_peered":"2024-03-06T07:22:59.805549+0000","last_clean":"2024-03-06T07:22:59.805549+0000","last_became_active":"2024-03-06T07:22:59.804353+0000","last_became_peered":"2024-03-06T07:22:59.804353+0000","last_unstale":"2024-03-06T07:22:59.805549+0000","last_undegraded":"2024-03-06T07:22:59.805549+0000","last_fullsized":"2024-03-06T07:22:59.805549+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:46:08.564158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796177+0000","last_change":"2024-03-06T07:22:59.796177+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796177+0000","last_undegraded":"2024-03-06T07:22:59.796177+0000","last_fullsized":"2024-03-06T07:22:59.796177+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.804694+0000","last_change":"2024-03-06T07:22:59.803415+0000","last_active":"2024-03-06T07:22:59.804694+0000","last_peered":"2024-03-06T07:22:59.804694+0000","last_clean":"2024-03-06T07:22:59.804694+0000","last_became_active":"2024-03-06T07:22:59.803175+0000","last_became_peered":"2024-03-06T07:22:59.803175+0000","last_unstale":"2024-03-06T07:22:59.804694+0000","last_undegraded":"2024-03-06T07:22:59.804694+0000","last_fullsized":"2024-03-06T07:22:59.804694+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:36:12.264660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796946+0000","last_change":"2024-03-06T07:22:59.796946+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796946+0000","last_undegraded":"2024-03-06T07:22:59.796946+0000","last_fullsized":"2024-03-06T07:22:59.796946+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.811396+0000","last_change":"2024-03-06T07:22:59.810599+0000","last_active":"2024-03-06T07:22:59.811396+0000","last_peered":"2024-03-06T07:22:59.811396+0000","last_clean":"2024-03-06T07:22:59.811396+0000","last_became_active":"2024-03-06T07:22:59.810291+0000","last_became_peered":"2024-03-06T07:22:59.810291+0000","last_unstale":"2024-03-06T07:22:59.811396+0000","last_undegraded":"2024-03-06T07:22:59.811396+0000","last_fullsized":"2024-03-06T07:22:59.811396+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:36:12.264660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.796176+0000","last_change":"2024-03-06T07:22:59.796176+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.796176+0000","last_undegraded":"2024-03-06T07:22:59.796176+0000","last_fullsized":"2024-03-06T07:22:59.796176+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.816470+0000","last_change":"2024-03-06T07:22:59.812044+0000","last_active":"2024-03-06T07:22:59.816470+0000","last_peered":"2024-03-06T07:22:59.816470+0000","last_clean":"2024-03-06T07:22:59.816470+0000","last_became_active":"2024-03-06T07:22:59.811822+0000","last_became_peered":"2024-03-06T07:22:59.811822+0000","last_unstale":"2024-03-06T07:22:59.816470+0000","last_undegraded":"2024-03-06T07:22:59.816470+0000","last_fullsized":"2024-03-06T07:22:59.816470+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:38:31.114570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.798677+0000","last_change":"2024-03-06T07:22:59.798677+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.798677+0000","last_undegraded":"2024-03-06T07:22:59.798677+0000","last_fullsized":"2024-03-06T07:22:59.798677+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.811751+0000","last_change":"2024-03-06T07:22:59.809833+0000","last_active":"2024-03-06T07:22:59.811751+0000","last_peered":"2024-03-06T07:22:59.811751+0000","last_clean":"2024-03-06T07:22:59.811751+0000","last_became_active":"2024-03-06T07:22:59.809550+0000","last_became_peered":"2024-03-06T07:22:59.809550+0000","last_unstale":"2024-03-06T07:22:59.811751+0000","last_undegraded":"2024-03-06T07:22:59.811751+0000","last_fullsized":"2024-03-06T07:22:59.811751+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:40:47.871999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.799865+0000","last_change":"2024-03-06T07:22:59.799865+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.799865+0000","last_undegraded":"2024-03-06T07:22:59.799865+0000","last_fullsized":"2024-03-06T07:22:59.799865+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.816542+0000","last_change":"2024-03-06T07:22:59.812895+0000","last_active":"2024-03-06T07:22:59.816542+0000","last_peered":"2024-03-06T07:22:59.816542+0000","last_clean":"2024-03-06T07:22:59.816542+0000","last_became_active":"2024-03-06T07:22:59.812605+0000","last_became_peered":"2024-03-06T07:22:59.812605+0000","last_unstale":"2024-03-06T07:22:59.816542+0000","last_undegraded":"2024-03-06T07:22:59.816542+0000","last_fullsized":"2024-03-06T07:22:59.816542+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:33.788501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-03-06T07:22:59.800675+0000","last_change":"2024-03-06T07:22:59.800675+0000","last_active":"2024-03-06T07:22:59.788978+0000","last_peered":"2024-03-06T07:22:59.788978+0000","last_clean":"2024-03-06T07:22:59.788978+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:59.800675+0000","last_undegraded":"2024-03-06T07:22:59.800675+0000","last_fullsized":"2024-03-06T07:22:59.800675+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.809068+0000","last_change":"2024-03-06T07:22:59.808026+0000","last_active":"2024-03-06T07:22:59.809068+0000","last_peered":"2024-03-06T07:22:59.809068+0000","last_clean":"2024-03-06T07:22:59.809068+0000","last_became_active":"2024-03-06T07:22:59.807197+0000","last_became_peered":"2024-03-06T07:22:59.807197+0000","last_unstale":"2024-03-06T07:22:59.809068+0000","last_undegraded":"2024-03-06T07:22:59.809068+0000","last_fullsized":"2024-03-06T07:22:59.809068+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:46:08.564158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":189,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-06T07:22:59.805202+0000","last_change":"2024-03-06T07:21:47.553693+0000","last_active":"2024-03-06T07:22:59.805202+0000","last_peered":"2024-03-06T07:22:59.805202+0000","last_clean":"2024-03-06T07:22:59.805202+0000","last_became_active":"2024-03-06T07:21:47.552664+0000","last_became_peered":"2024-03-06T07:21:47.552664+0000","last_unstale":"2024-03-06T07:22:59.805202+0000","last_undegraded":"2024-03-06T07:22:59.805202+0000","last_fullsized":"2024-03-06T07:22:59.805202+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_clean_scrub_stamp":"2024-03-06T07:21:46.362028+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:02.916426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343900,"statfs":{"total":96636764160,"available":96608153600,"internally_reserved":0,"allocated":942080,"data_stored":724403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":127768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":134035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":335872,"data_stored":127768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":127768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574867,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":134035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574868,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":134035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574867,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":929792,"data_stored":718136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:23:05.779 INFO:tasks.ceph.mds.a.smithi083.stderr:2024-03-06T07:23:05.777+0000 7f3401fbd640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:23:08.579 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:23:08.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.700+0000 7fc790c29640 1 Processor -- start 2024-03-06T07:23:08.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.700+0000 7fc790c29640 1 -- start start 2024-03-06T07:23:08.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.701+0000 7fc790c29640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc78c0b4420 0x7fc78c0cdb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.701+0000 7fc790c29640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc78c0d0a70 con 0x7fc78c06afb0 2024-03-06T07:23:08.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.701+0000 7fc790c29640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc78c0d0d10 con 0x7fc78c0ce0d0 2024-03-06T07:23:08.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.701+0000 7fc790c29640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc78c0d0850 con 0x7fc78c0b4420 2024-03-06T07:23:08.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.701+0000 7fc783fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc78c0b4420 0x7fc78c0cdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.701+0000 7fc783fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc78c0b4420 0x7fc78c0cdb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55620/0 (socket says 172.21.15.83:55620) 2024-03-06T07:23:08.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.701+0000 7fc78a59c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fc78c06afb0 0x7fc78c06b390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:37342/0 (socket says 172.21.15.83:37342) 2024-03-06T07:23:08.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.701+0000 7fc783fff640 1 -- 172.21.15.83:0/1122177144 learned_addr learned my addr 172.21.15.83:0/1122177144 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:08.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.701+0000 7fc783fff640 1 -- 172.21.15.83:0/1122177144 >> v1:172.21.15.183:6789/0 conn(0x7fc78c0ce0d0 legacy=0x7fc78c15c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:08.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.701+0000 7fc783fff640 1 -- 172.21.15.83:0/1122177144 >> v1:172.21.15.83:6789/0 conn(0x7fc78c06afb0 legacy=0x7fc78c06b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.701+0000 7fc783fff640 1 -- 172.21.15.83:0/1122177144 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc78c0d0ff0 con 0x7fc78c0b4420 2024-03-06T07:23:08.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.701+0000 7fc789d9b640 1 -- 172.21.15.83:0/1122177144 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3277512065 0 0) 0x7fc78c0d0a70 con 0x7fc78c06afb0 2024-03-06T07:23:08.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.702+0000 7fc783fff640 1 --2- 172.21.15.83:0/1122177144 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc78c0b4420 0x7fc78c0cdb90 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fc774009870 tx=0x7fc77402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdcc2d4c58371415 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:08.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.702+0000 7fc789d9b640 1 -- 172.21.15.83:0/1122177144 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc774035020 con 0x7fc78c0b4420 2024-03-06T07:23:08.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.702+0000 7fc789d9b640 1 -- 172.21.15.83:0/1122177144 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc774031870 con 0x7fc78c0b4420 2024-03-06T07:23:08.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.702+0000 7fc789d9b640 1 -- 172.21.15.83:0/1122177144 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc774031c40 con 0x7fc78c0b4420 2024-03-06T07:23:08.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.703+0000 7fc790c29640 1 -- 172.21.15.83:0/1122177144 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc78c0b4420 msgr2=0x7fc78c0cdb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.703+0000 7fc790c29640 1 --2- 172.21.15.83:0/1122177144 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc78c0b4420 0x7fc78c0cdb90 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fc774009870 tx=0x7fc77402b910 comp rx=0 tx=0).stop 2024-03-06T07:23:08.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.703+0000 7fc790c29640 1 -- 172.21.15.83:0/1122177144 shutdown_connections 2024-03-06T07:23:08.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.703+0000 7fc790c29640 1 --2- 172.21.15.83:0/1122177144 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc78c0b4420 0x7fc78c0cdb90 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.703+0000 7fc790c29640 1 -- 172.21.15.83:0/1122177144 >> 172.21.15.83:0/1122177144 conn(0x7fc78c06c0c0 msgr2=0x7fc78c06c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:08.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.703+0000 7fc790c29640 1 -- 172.21.15.83:0/1122177144 shutdown_connections 2024-03-06T07:23:08.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.703+0000 7fc790c29640 1 -- 172.21.15.83:0/1122177144 wait complete. 2024-03-06T07:23:08.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.704+0000 7fc790c29640 1 Processor -- start 2024-03-06T07:23:08.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.704+0000 7fc790c29640 1 -- start start 2024-03-06T07:23:08.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.705+0000 7fc790c29640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc78c06afb0 0x7fc78c155460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.705+0000 7fc790c29640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc78c0b4420 0x7fc78c1575f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.705+0000 7fc790c29640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc78c0ce0d0 0x7fc78c15b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.705+0000 7fc790c29640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc78c0ceb90 con 0x7fc78c0b4420 2024-03-06T07:23:08.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.705+0000 7fc790c29640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc78c145840 con 0x7fc78c06afb0 2024-03-06T07:23:08.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.705+0000 7fc790c29640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc78c15cba0 con 0x7fc78c0ce0d0 2024-03-06T07:23:08.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.705+0000 7fc783fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc78c0b4420 0x7fc78c1575f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.705+0000 7fc783fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc78c0b4420 0x7fc78c1575f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55632/0 (socket says 172.21.15.83:55632) 2024-03-06T07:23:08.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.705+0000 7fc783fff640 1 -- 172.21.15.83:0/1399245164 learned_addr learned my addr 172.21.15.83:0/1399245164 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:08.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.705+0000 7fc783fff640 1 -- 172.21.15.83:0/1399245164 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc78c0ce0d0 msgr2=0x7fc78c15b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.705+0000 7fc783fff640 1 --2- 172.21.15.83:0/1399245164 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc78c0ce0d0 0x7fc78c15b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.705+0000 7fc783fff640 1 -- 172.21.15.83:0/1399245164 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc78c06afb0 msgr2=0x7fc78c155460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.705+0000 7fc783fff640 1 --2- 172.21.15.83:0/1399245164 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc78c06afb0 0x7fc78c155460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.706+0000 7fc783fff640 1 -- 172.21.15.83:0/1399245164 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc78c1580b0 con 0x7fc78c0b4420 2024-03-06T07:23:08.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.706+0000 7fc783fff640 1 --2- 172.21.15.83:0/1399245164 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc78c0b4420 0x7fc78c1575f0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fc774009870 tx=0x7fc7740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:08.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.706+0000 7fc782ffd640 1 -- 172.21.15.83:0/1399245164 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc77403ad60 con 0x7fc78c0b4420 2024-03-06T07:23:08.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.706+0000 7fc782ffd640 1 -- 172.21.15.83:0/1399245164 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7740443f0 con 0x7fc78c0b4420 2024-03-06T07:23:08.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.706+0000 7fc790c29640 1 -- 172.21.15.83:0/1399245164 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc78c158390 con 0x7fc78c0b4420 2024-03-06T07:23:08.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.706+0000 7fc782ffd640 1 -- 172.21.15.83:0/1399245164 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7740447e0 con 0x7fc78c0b4420 2024-03-06T07:23:08.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.706+0000 7fc790c29640 1 -- 172.21.15.83:0/1399245164 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc78c158950 con 0x7fc78c0b4420 2024-03-06T07:23:08.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.707+0000 7fc790c29640 1 -- 172.21.15.83:0/1399245164 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc78c04f330 con 0x7fc78c0b4420 2024-03-06T07:23:08.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.707+0000 7fc782ffd640 1 -- 172.21.15.83:0/1399245164 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc77404b060 con 0x7fc78c0b4420 2024-03-06T07:23:08.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.708+0000 7fc782ffd640 1 --2- 172.21.15.83:0/1399245164 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc76c06b4d0 0x7fc76c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.708+0000 7fc782ffd640 1 -- 172.21.15.83:0/1399245164 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc7740ac430 con 0x7fc78c0b4420 2024-03-06T07:23:08.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.708+0000 7fc78a59c640 1 --2- 172.21.15.83:0/1399245164 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc76c06b4d0 0x7fc76c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.708+0000 7fc78a59c640 1 --2- 172.21.15.83:0/1399245164 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc76c06b4d0 0x7fc76c06d990 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc77c007fd0 tx=0x7fc77c007920 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:08.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.716+0000 7fc782ffd640 1 -- 172.21.15.83:0/1399245164 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc77403f070 con 0x7fc78c0b4420 2024-03-06T07:23:08.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.895+0000 7fc790c29640 1 -- 172.21.15.83:0/1399245164 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc78c0b5cb0 con 0x7fc76c06b4d0 2024-03-06T07:23:08.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.901+0000 7fc782ffd640 1 -- 172.21.15.83:0/1399245164 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251951 (secure 0 0 0) 0x7fc78c0b5cb0 con 0x7fc76c06b4d0 2024-03-06T07:23:08.905 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:23:08.907 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-06T07:23:08.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.904+0000 7fc790c29640 1 -- 172.21.15.83:0/1399245164 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc76c06b4d0 msgr2=0x7fc76c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.904+0000 7fc790c29640 1 --2- 172.21.15.83:0/1399245164 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc76c06b4d0 0x7fc76c06d990 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc77c007fd0 tx=0x7fc77c007920 comp rx=0 tx=0).stop 2024-03-06T07:23:08.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.904+0000 7fc790c29640 1 -- 172.21.15.83:0/1399245164 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc78c0b4420 msgr2=0x7fc78c1575f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.904+0000 7fc790c29640 1 --2- 172.21.15.83:0/1399245164 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc78c0b4420 0x7fc78c1575f0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fc774009870 tx=0x7fc7740046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.904+0000 7fc790c29640 1 -- 172.21.15.83:0/1399245164 shutdown_connections 2024-03-06T07:23:08.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.904+0000 7fc790c29640 1 --2- 172.21.15.83:0/1399245164 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc76c06b4d0 0x7fc76c06d990 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.904+0000 7fc790c29640 1 --2- 172.21.15.83:0/1399245164 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc78c0ce0d0 0x7fc78c15b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.904+0000 7fc790c29640 1 --2- 172.21.15.83:0/1399245164 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc78c0b4420 0x7fc78c1575f0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.904+0000 7fc790c29640 1 --2- 172.21.15.83:0/1399245164 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc78c06afb0 0x7fc78c155460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.904+0000 7fc790c29640 1 -- 172.21.15.83:0/1399245164 >> 172.21.15.83:0/1399245164 conn(0x7fc78c06c0c0 msgr2=0x7fc78c0b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:08.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.904+0000 7fc790c29640 1 -- 172.21.15.83:0/1399245164 shutdown_connections 2024-03-06T07:23:08.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:08.904+0000 7fc790c29640 1 -- 172.21.15.83:0/1399245164 wait complete. 2024-03-06T07:23:08.917 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":68,"stamp":"2024-03-06T07:23:08.334154+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220300,"kb_used_data":4136,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754420,"statfs":{"total":773094113280,"available":772868526080,"internally_reserved":0,"allocated":4235264,"data_stored":2362436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12757,"internal_metadata":221236779},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001536"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:01.821780+0000","last_change":"2024-03-06T07:23:00.818568+0000","last_active":"2024-03-06T07:23:01.821780+0000","last_peered":"2024-03-06T07:23:01.821780+0000","last_clean":"2024-03-06T07:23:01.821780+0000","last_became_active":"2024-03-06T07:23:00.818251+0000","last_became_peered":"2024-03-06T07:23:00.818251+0000","last_unstale":"2024-03-06T07:23:01.821780+0000","last_undegraded":"2024-03-06T07:23:01.821780+0000","last_fullsized":"2024-03-06T07:23:01.821780+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:59:25.523773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.979220+0000","last_change":"2024-03-06T07:23:00.816872+0000","last_active":"2024-03-06T07:23:02.979220+0000","last_peered":"2024-03-06T07:23:02.979220+0000","last_clean":"2024-03-06T07:23:02.979220+0000","last_became_active":"2024-03-06T07:23:00.816326+0000","last_became_peered":"2024-03-06T07:23:00.816326+0000","last_unstale":"2024-03-06T07:23:02.979220+0000","last_undegraded":"2024-03-06T07:23:02.979220+0000","last_fullsized":"2024-03-06T07:23:02.979220+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:50.287073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.478362+0000","last_change":"2024-03-06T07:23:00.828820+0000","last_active":"2024-03-06T07:23:02.478362+0000","last_peered":"2024-03-06T07:23:02.478362+0000","last_clean":"2024-03-06T07:23:02.478362+0000","last_became_active":"2024-03-06T07:23:00.828239+0000","last_became_peered":"2024-03-06T07:23:00.828239+0000","last_unstale":"2024-03-06T07:23:02.478362+0000","last_undegraded":"2024-03-06T07:23:02.478362+0000","last_fullsized":"2024-03-06T07:23:02.478362+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:41:29.030630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.610106+0000","last_change":"2024-03-06T07:23:00.824104+0000","last_active":"2024-03-06T07:23:02.610106+0000","last_peered":"2024-03-06T07:23:02.610106+0000","last_clean":"2024-03-06T07:23:02.610106+0000","last_became_active":"2024-03-06T07:23:00.823780+0000","last_became_peered":"2024-03-06T07:23:00.823780+0000","last_unstale":"2024-03-06T07:23:02.610106+0000","last_undegraded":"2024-03-06T07:23:02.610106+0000","last_fullsized":"2024-03-06T07:23:02.610106+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:31:29.735408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.978542+0000","last_change":"2024-03-06T07:23:00.814843+0000","last_active":"2024-03-06T07:23:02.978542+0000","last_peered":"2024-03-06T07:23:02.978542+0000","last_clean":"2024-03-06T07:23:02.978542+0000","last_became_active":"2024-03-06T07:23:00.813505+0000","last_became_peered":"2024-03-06T07:23:00.813505+0000","last_unstale":"2024-03-06T07:23:02.978542+0000","last_undegraded":"2024-03-06T07:23:02.978542+0000","last_fullsized":"2024-03-06T07:23:02.978542+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:24:27.923656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.979158+0000","last_change":"2024-03-06T07:23:00.816998+0000","last_active":"2024-03-06T07:23:02.979158+0000","last_peered":"2024-03-06T07:23:02.979158+0000","last_clean":"2024-03-06T07:23:02.979158+0000","last_became_active":"2024-03-06T07:23:00.816668+0000","last_became_peered":"2024-03-06T07:23:00.816668+0000","last_unstale":"2024-03-06T07:23:02.979158+0000","last_undegraded":"2024-03-06T07:23:02.979158+0000","last_fullsized":"2024-03-06T07:23:02.979158+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:09:03.972077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.614714+0000","last_change":"2024-03-06T07:23:00.831244+0000","last_active":"2024-03-06T07:23:02.614714+0000","last_peered":"2024-03-06T07:23:02.614714+0000","last_clean":"2024-03-06T07:23:02.614714+0000","last_became_active":"2024-03-06T07:23:00.830668+0000","last_became_peered":"2024-03-06T07:23:00.830668+0000","last_unstale":"2024-03-06T07:23:02.614714+0000","last_undegraded":"2024-03-06T07:23:02.614714+0000","last_fullsized":"2024-03-06T07:23:02.614714+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:09:03.972077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.614502+0000","last_change":"2024-03-06T07:23:00.820781+0000","last_active":"2024-03-06T07:23:02.614502+0000","last_peered":"2024-03-06T07:23:02.614502+0000","last_clean":"2024-03-06T07:23:02.614502+0000","last_became_active":"2024-03-06T07:23:00.820490+0000","last_became_peered":"2024-03-06T07:23:00.820490+0000","last_unstale":"2024-03-06T07:23:02.614502+0000","last_undegraded":"2024-03-06T07:23:02.614502+0000","last_fullsized":"2024-03-06T07:23:02.614502+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:25:59.927903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:01.822396+0000","last_change":"2024-03-06T07:23:00.828906+0000","last_active":"2024-03-06T07:23:01.822396+0000","last_peered":"2024-03-06T07:23:01.822396+0000","last_clean":"2024-03-06T07:23:01.822396+0000","last_became_active":"2024-03-06T07:23:00.828617+0000","last_became_peered":"2024-03-06T07:23:00.828617+0000","last_unstale":"2024-03-06T07:23:01.822396+0000","last_undegraded":"2024-03-06T07:23:01.822396+0000","last_fullsized":"2024-03-06T07:23:01.822396+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:31:29.735408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.610192+0000","last_change":"2024-03-06T07:23:00.824053+0000","last_active":"2024-03-06T07:23:02.610192+0000","last_peered":"2024-03-06T07:23:02.610192+0000","last_clean":"2024-03-06T07:23:02.610192+0000","last_became_active":"2024-03-06T07:23:00.823748+0000","last_became_peered":"2024-03-06T07:23:00.823748+0000","last_unstale":"2024-03-06T07:23:02.610192+0000","last_undegraded":"2024-03-06T07:23:02.610192+0000","last_fullsized":"2024-03-06T07:23:02.610192+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:46.029517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.614963+0000","last_change":"2024-03-06T07:23:00.826817+0000","last_active":"2024-03-06T07:23:02.614963+0000","last_peered":"2024-03-06T07:23:02.614963+0000","last_clean":"2024-03-06T07:23:02.614963+0000","last_became_active":"2024-03-06T07:23:00.826185+0000","last_became_peered":"2024-03-06T07:23:00.826185+0000","last_unstale":"2024-03-06T07:23:02.614963+0000","last_undegraded":"2024-03-06T07:23:02.614963+0000","last_fullsized":"2024-03-06T07:23:02.614963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:27.157625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.478322+0000","last_change":"2024-03-06T07:23:00.817901+0000","last_active":"2024-03-06T07:23:02.478322+0000","last_peered":"2024-03-06T07:23:02.478322+0000","last_clean":"2024-03-06T07:23:02.478322+0000","last_became_active":"2024-03-06T07:23:00.817616+0000","last_became_peered":"2024-03-06T07:23:00.817616+0000","last_unstale":"2024-03-06T07:23:02.478322+0000","last_undegraded":"2024-03-06T07:23:02.478322+0000","last_fullsized":"2024-03-06T07:23:02.478322+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:36.807300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.979123+0000","last_change":"2024-03-06T07:23:00.822689+0000","last_active":"2024-03-06T07:23:02.979123+0000","last_peered":"2024-03-06T07:23:02.979123+0000","last_clean":"2024-03-06T07:23:02.979123+0000","last_became_active":"2024-03-06T07:23:00.822448+0000","last_became_peered":"2024-03-06T07:23:00.822448+0000","last_unstale":"2024-03-06T07:23:02.979123+0000","last_undegraded":"2024-03-06T07:23:02.979123+0000","last_fullsized":"2024-03-06T07:23:02.979123+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:08:19.009173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.979283+0000","last_change":"2024-03-06T07:23:00.813273+0000","last_active":"2024-03-06T07:23:02.979283+0000","last_peered":"2024-03-06T07:23:02.979283+0000","last_clean":"2024-03-06T07:23:02.979283+0000","last_became_active":"2024-03-06T07:23:00.812788+0000","last_became_peered":"2024-03-06T07:23:00.812788+0000","last_unstale":"2024-03-06T07:23:02.979283+0000","last_undegraded":"2024-03-06T07:23:02.979283+0000","last_fullsized":"2024-03-06T07:23:02.979283+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:42:57.165308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.478205+0000","last_change":"2024-03-06T07:23:00.817237+0000","last_active":"2024-03-06T07:23:02.478205+0000","last_peered":"2024-03-06T07:23:02.478205+0000","last_clean":"2024-03-06T07:23:02.478205+0000","last_became_active":"2024-03-06T07:23:00.816971+0000","last_became_peered":"2024-03-06T07:23:00.816971+0000","last_unstale":"2024-03-06T07:23:02.478205+0000","last_undegraded":"2024-03-06T07:23:02.478205+0000","last_fullsized":"2024-03-06T07:23:02.478205+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:45:54.192806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.614904+0000","last_change":"2024-03-06T07:23:00.828121+0000","last_active":"2024-03-06T07:23:02.614904+0000","last_peered":"2024-03-06T07:23:02.614904+0000","last_clean":"2024-03-06T07:23:02.614904+0000","last_became_active":"2024-03-06T07:23:00.827831+0000","last_became_peered":"2024-03-06T07:23:00.827831+0000","last_unstale":"2024-03-06T07:23:02.614904+0000","last_undegraded":"2024-03-06T07:23:02.614904+0000","last_fullsized":"2024-03-06T07:23:02.614904+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:42:57.165308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.615072+0000","last_change":"2024-03-06T07:23:00.828232+0000","last_active":"2024-03-06T07:23:02.615072+0000","last_peered":"2024-03-06T07:23:02.615072+0000","last_clean":"2024-03-06T07:23:02.615072+0000","last_became_active":"2024-03-06T07:23:00.827964+0000","last_became_peered":"2024-03-06T07:23:00.827964+0000","last_unstale":"2024-03-06T07:23:02.615072+0000","last_undegraded":"2024-03-06T07:23:02.615072+0000","last_fullsized":"2024-03-06T07:23:02.615072+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:45.396857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.979342+0000","last_change":"2024-03-06T07:23:00.821515+0000","last_active":"2024-03-06T07:23:02.979342+0000","last_peered":"2024-03-06T07:23:02.979342+0000","last_clean":"2024-03-06T07:23:02.979342+0000","last_became_active":"2024-03-06T07:23:00.821203+0000","last_became_peered":"2024-03-06T07:23:00.821203+0000","last_unstale":"2024-03-06T07:23:02.979342+0000","last_undegraded":"2024-03-06T07:23:02.979342+0000","last_fullsized":"2024-03-06T07:23:02.979342+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:27.157625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.978803+0000","last_change":"2024-03-06T07:23:00.822065+0000","last_active":"2024-03-06T07:23:02.978803+0000","last_peered":"2024-03-06T07:23:02.978803+0000","last_clean":"2024-03-06T07:23:02.978803+0000","last_became_active":"2024-03-06T07:23:00.821772+0000","last_became_peered":"2024-03-06T07:23:00.821772+0000","last_unstale":"2024-03-06T07:23:02.978803+0000","last_undegraded":"2024-03-06T07:23:02.978803+0000","last_fullsized":"2024-03-06T07:23:02.978803+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:36.807300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.979177+0000","last_change":"2024-03-06T07:23:00.820553+0000","last_active":"2024-03-06T07:23:02.979177+0000","last_peered":"2024-03-06T07:23:02.979177+0000","last_clean":"2024-03-06T07:23:02.979177+0000","last_became_active":"2024-03-06T07:23:00.819892+0000","last_became_peered":"2024-03-06T07:23:00.819892+0000","last_unstale":"2024-03-06T07:23:02.979177+0000","last_undegraded":"2024-03-06T07:23:02.979177+0000","last_fullsized":"2024-03-06T07:23:02.979177+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:33.452085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:01.821797+0000","last_change":"2024-03-06T07:23:00.829406+0000","last_active":"2024-03-06T07:23:01.821797+0000","last_peered":"2024-03-06T07:23:01.821797+0000","last_clean":"2024-03-06T07:23:01.821797+0000","last_became_active":"2024-03-06T07:23:00.829147+0000","last_became_peered":"2024-03-06T07:23:00.829147+0000","last_unstale":"2024-03-06T07:23:01.821797+0000","last_undegraded":"2024-03-06T07:23:01.821797+0000","last_fullsized":"2024-03-06T07:23:01.821797+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:48:26.113468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:01.822205+0000","last_change":"2024-03-06T07:23:00.822911+0000","last_active":"2024-03-06T07:23:01.822205+0000","last_peered":"2024-03-06T07:23:01.822205+0000","last_clean":"2024-03-06T07:23:01.822205+0000","last_became_active":"2024-03-06T07:23:00.822645+0000","last_became_peered":"2024-03-06T07:23:00.822645+0000","last_unstale":"2024-03-06T07:23:01.822205+0000","last_undegraded":"2024-03-06T07:23:01.822205+0000","last_fullsized":"2024-03-06T07:23:01.822205+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:51:48.943812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:01.822082+0000","last_change":"2024-03-06T07:23:00.832295+0000","last_active":"2024-03-06T07:23:01.822082+0000","last_peered":"2024-03-06T07:23:01.822082+0000","last_clean":"2024-03-06T07:23:01.822082+0000","last_became_active":"2024-03-06T07:23:00.832027+0000","last_became_peered":"2024-03-06T07:23:00.832027+0000","last_unstale":"2024-03-06T07:23:01.822082+0000","last_undegraded":"2024-03-06T07:23:01.822082+0000","last_fullsized":"2024-03-06T07:23:01.822082+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:59:25.523773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.478054+0000","last_change":"2024-03-06T07:23:00.816357+0000","last_active":"2024-03-06T07:23:02.478054+0000","last_peered":"2024-03-06T07:23:02.478054+0000","last_clean":"2024-03-06T07:23:02.478054+0000","last_became_active":"2024-03-06T07:23:00.816068+0000","last_became_peered":"2024-03-06T07:23:00.816068+0000","last_unstale":"2024-03-06T07:23:02.478054+0000","last_undegraded":"2024-03-06T07:23:02.478054+0000","last_fullsized":"2024-03-06T07:23:02.478054+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:25:59.927903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.610337+0000","last_change":"2024-03-06T07:23:00.831005+0000","last_active":"2024-03-06T07:23:02.610337+0000","last_peered":"2024-03-06T07:23:02.610337+0000","last_clean":"2024-03-06T07:23:02.610337+0000","last_became_active":"2024-03-06T07:23:00.830750+0000","last_became_peered":"2024-03-06T07:23:00.830750+0000","last_unstale":"2024-03-06T07:23:02.610337+0000","last_undegraded":"2024-03-06T07:23:02.610337+0000","last_fullsized":"2024-03-06T07:23:02.610337+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:25:59.927903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.979464+0000","last_change":"2024-03-06T07:23:00.828464+0000","last_active":"2024-03-06T07:23:02.979464+0000","last_peered":"2024-03-06T07:23:02.979464+0000","last_clean":"2024-03-06T07:23:02.979464+0000","last_became_active":"2024-03-06T07:23:00.828176+0000","last_became_peered":"2024-03-06T07:23:00.828176+0000","last_unstale":"2024-03-06T07:23:02.979464+0000","last_undegraded":"2024-03-06T07:23:02.979464+0000","last_fullsized":"2024-03-06T07:23:02.979464+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:45.396857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.610310+0000","last_change":"2024-03-06T07:23:00.825273+0000","last_active":"2024-03-06T07:23:02.610310+0000","last_peered":"2024-03-06T07:23:02.610310+0000","last_clean":"2024-03-06T07:23:02.610310+0000","last_became_active":"2024-03-06T07:23:00.824809+0000","last_became_peered":"2024-03-06T07:23:00.824809+0000","last_unstale":"2024-03-06T07:23:02.610310+0000","last_undegraded":"2024-03-06T07:23:02.610310+0000","last_fullsized":"2024-03-06T07:23:02.610310+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:07:39.846532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.478378+0000","last_change":"2024-03-06T07:23:00.831065+0000","last_active":"2024-03-06T07:23:02.478378+0000","last_peered":"2024-03-06T07:23:02.478378+0000","last_clean":"2024-03-06T07:23:02.478378+0000","last_became_active":"2024-03-06T07:23:00.830758+0000","last_became_peered":"2024-03-06T07:23:00.830758+0000","last_unstale":"2024-03-06T07:23:02.478378+0000","last_undegraded":"2024-03-06T07:23:02.478378+0000","last_fullsized":"2024-03-06T07:23:02.478378+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:09:03.972077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:01.822314+0000","last_change":"2024-03-06T07:23:00.823499+0000","last_active":"2024-03-06T07:23:01.822314+0000","last_peered":"2024-03-06T07:23:01.822314+0000","last_clean":"2024-03-06T07:23:01.822314+0000","last_became_active":"2024-03-06T07:23:00.823230+0000","last_became_peered":"2024-03-06T07:23:00.823230+0000","last_unstale":"2024-03-06T07:23:01.822314+0000","last_undegraded":"2024-03-06T07:23:01.822314+0000","last_fullsized":"2024-03-06T07:23:01.822314+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:33.703308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.614681+0000","last_change":"2024-03-06T07:23:00.823993+0000","last_active":"2024-03-06T07:23:02.614681+0000","last_peered":"2024-03-06T07:23:02.614681+0000","last_clean":"2024-03-06T07:23:02.614681+0000","last_became_active":"2024-03-06T07:23:00.822296+0000","last_became_peered":"2024-03-06T07:23:00.822296+0000","last_unstale":"2024-03-06T07:23:02.614681+0000","last_undegraded":"2024-03-06T07:23:02.614681+0000","last_fullsized":"2024-03-06T07:23:02.614681+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:36.807300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:01.821481+0000","last_change":"2024-03-06T07:23:00.818077+0000","last_active":"2024-03-06T07:23:01.821481+0000","last_peered":"2024-03-06T07:23:01.821481+0000","last_clean":"2024-03-06T07:23:01.821481+0000","last_became_active":"2024-03-06T07:23:00.817788+0000","last_became_peered":"2024-03-06T07:23:00.817788+0000","last_unstale":"2024-03-06T07:23:01.821481+0000","last_undegraded":"2024-03-06T07:23:01.821481+0000","last_fullsized":"2024-03-06T07:23:01.821481+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:30.019408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.614606+0000","last_change":"2024-03-06T07:23:00.833213+0000","last_active":"2024-03-06T07:23:02.614606+0000","last_peered":"2024-03-06T07:23:02.614606+0000","last_clean":"2024-03-06T07:23:02.614606+0000","last_became_active":"2024-03-06T07:23:00.832946+0000","last_became_peered":"2024-03-06T07:23:00.832946+0000","last_unstale":"2024-03-06T07:23:02.614606+0000","last_undegraded":"2024-03-06T07:23:02.614606+0000","last_fullsized":"2024-03-06T07:23:02.614606+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:17.020204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:01.822309+0000","last_change":"2024-03-06T07:23:00.825247+0000","last_active":"2024-03-06T07:23:01.822309+0000","last_peered":"2024-03-06T07:23:01.822309+0000","last_clean":"2024-03-06T07:23:01.822309+0000","last_became_active":"2024-03-06T07:23:00.824894+0000","last_became_peered":"2024-03-06T07:23:00.824894+0000","last_unstale":"2024-03-06T07:23:01.822309+0000","last_undegraded":"2024-03-06T07:23:01.822309+0000","last_fullsized":"2024-03-06T07:23:01.822309+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:47:35.916346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"32'4","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:01.821883+0000","last_change":"2024-03-06T07:22:59.810363+0000","last_active":"2024-03-06T07:23:01.821883+0000","last_peered":"2024-03-06T07:23:01.821883+0000","last_clean":"2024-03-06T07:23:01.821883+0000","last_became_active":"2024-03-06T07:22:59.810032+0000","last_became_peered":"2024-03-06T07:22:59.810032+0000","last_unstale":"2024-03-06T07:23:01.821883+0000","last_undegraded":"2024-03-06T07:23:01.821883+0000","last_fullsized":"2024-03-06T07:23:01.821883+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:53.894652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:01.821807+0000","last_change":"2024-03-06T07:23:00.828182+0000","last_active":"2024-03-06T07:23:01.821807+0000","last_peered":"2024-03-06T07:23:01.821807+0000","last_clean":"2024-03-06T07:23:01.821807+0000","last_became_active":"2024-03-06T07:23:00.825996+0000","last_became_peered":"2024-03-06T07:23:00.825996+0000","last_unstale":"2024-03-06T07:23:01.821807+0000","last_undegraded":"2024-03-06T07:23:01.821807+0000","last_fullsized":"2024-03-06T07:23:01.821807+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:12:16.981197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:01.822275+0000","last_change":"2024-03-06T07:22:59.810687+0000","last_active":"2024-03-06T07:23:01.822275+0000","last_peered":"2024-03-06T07:23:01.822275+0000","last_clean":"2024-03-06T07:23:01.822275+0000","last_became_active":"2024-03-06T07:22:59.810405+0000","last_became_peered":"2024-03-06T07:22:59.810405+0000","last_unstale":"2024-03-06T07:23:01.822275+0000","last_undegraded":"2024-03-06T07:23:01.822275+0000","last_fullsized":"2024-03-06T07:23:01.822275+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:30:21.970435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:01.821938+0000","last_change":"2024-03-06T07:23:00.827865+0000","last_active":"2024-03-06T07:23:01.821938+0000","last_peered":"2024-03-06T07:23:01.821938+0000","last_clean":"2024-03-06T07:23:01.821938+0000","last_became_active":"2024-03-06T07:23:00.827520+0000","last_became_peered":"2024-03-06T07:23:00.827520+0000","last_unstale":"2024-03-06T07:23:01.821938+0000","last_undegraded":"2024-03-06T07:23:01.821938+0000","last_fullsized":"2024-03-06T07:23:01.821938+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:41:21.249213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.978907+0000","last_change":"2024-03-06T07:22:59.807093+0000","last_active":"2024-03-06T07:23:02.978907+0000","last_peered":"2024-03-06T07:23:02.978907+0000","last_clean":"2024-03-06T07:23:02.978907+0000","last_became_active":"2024-03-06T07:22:59.806858+0000","last_became_peered":"2024-03-06T07:22:59.806858+0000","last_unstale":"2024-03-06T07:23:02.978907+0000","last_undegraded":"2024-03-06T07:23:02.978907+0000","last_fullsized":"2024-03-06T07:23:02.978907+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:12:15.977106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.610297+0000","last_change":"2024-03-06T07:23:00.825308+0000","last_active":"2024-03-06T07:23:02.610297+0000","last_peered":"2024-03-06T07:23:02.610297+0000","last_clean":"2024-03-06T07:23:02.610297+0000","last_became_active":"2024-03-06T07:23:00.824989+0000","last_became_peered":"2024-03-06T07:23:00.824989+0000","last_unstale":"2024-03-06T07:23:02.610297+0000","last_undegraded":"2024-03-06T07:23:02.610297+0000","last_fullsized":"2024-03-06T07:23:02.610297+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:27.933112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.979149+0000","last_change":"2024-03-06T07:22:59.805788+0000","last_active":"2024-03-06T07:23:02.979149+0000","last_peered":"2024-03-06T07:23:02.979149+0000","last_clean":"2024-03-06T07:23:02.979149+0000","last_became_active":"2024-03-06T07:22:59.805463+0000","last_became_peered":"2024-03-06T07:22:59.805463+0000","last_unstale":"2024-03-06T07:23:02.979149+0000","last_undegraded":"2024-03-06T07:23:02.979149+0000","last_fullsized":"2024-03-06T07:23:02.979149+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:55.870572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.614696+0000","last_change":"2024-03-06T07:23:00.827202+0000","last_active":"2024-03-06T07:23:02.614696+0000","last_peered":"2024-03-06T07:23:02.614696+0000","last_clean":"2024-03-06T07:23:02.614696+0000","last_became_active":"2024-03-06T07:23:00.826545+0000","last_became_peered":"2024-03-06T07:23:00.826545+0000","last_unstale":"2024-03-06T07:23:02.614696+0000","last_undegraded":"2024-03-06T07:23:02.614696+0000","last_fullsized":"2024-03-06T07:23:02.614696+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:41:29.030630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:01.822224+0000","last_change":"2024-03-06T07:22:59.808162+0000","last_active":"2024-03-06T07:23:01.822224+0000","last_peered":"2024-03-06T07:23:01.822224+0000","last_clean":"2024-03-06T07:23:01.822224+0000","last_became_active":"2024-03-06T07:22:59.807875+0000","last_became_peered":"2024-03-06T07:22:59.807875+0000","last_unstale":"2024-03-06T07:23:01.822224+0000","last_undegraded":"2024-03-06T07:23:01.822224+0000","last_fullsized":"2024-03-06T07:23:01.822224+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:38:31.114570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.478204+0000","last_change":"2024-03-06T07:23:00.818305+0000","last_active":"2024-03-06T07:23:02.478204+0000","last_peered":"2024-03-06T07:23:02.478204+0000","last_clean":"2024-03-06T07:23:02.478204+0000","last_became_active":"2024-03-06T07:23:00.818040+0000","last_became_peered":"2024-03-06T07:23:00.818040+0000","last_unstale":"2024-03-06T07:23:02.478204+0000","last_undegraded":"2024-03-06T07:23:02.478204+0000","last_fullsized":"2024-03-06T07:23:02.478204+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:17.020204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:01.822164+0000","last_change":"2024-03-06T07:22:59.806399+0000","last_active":"2024-03-06T07:23:01.822164+0000","last_peered":"2024-03-06T07:23:01.822164+0000","last_clean":"2024-03-06T07:23:01.822164+0000","last_became_active":"2024-03-06T07:22:59.805990+0000","last_became_peered":"2024-03-06T07:22:59.805990+0000","last_unstale":"2024-03-06T07:23:01.822164+0000","last_undegraded":"2024-03-06T07:23:01.822164+0000","last_fullsized":"2024-03-06T07:23:01.822164+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:46:08.564158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:01.821612+0000","last_change":"2024-03-06T07:23:00.816452+0000","last_active":"2024-03-06T07:23:01.821612+0000","last_peered":"2024-03-06T07:23:01.821612+0000","last_clean":"2024-03-06T07:23:01.821612+0000","last_became_active":"2024-03-06T07:23:00.816116+0000","last_became_peered":"2024-03-06T07:23:00.816116+0000","last_unstale":"2024-03-06T07:23:01.821612+0000","last_undegraded":"2024-03-06T07:23:01.821612+0000","last_fullsized":"2024-03-06T07:23:01.821612+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:47:35.916346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.978852+0000","last_change":"2024-03-06T07:22:59.805251+0000","last_active":"2024-03-06T07:23:02.978852+0000","last_peered":"2024-03-06T07:23:02.978852+0000","last_clean":"2024-03-06T07:23:02.978852+0000","last_became_active":"2024-03-06T07:22:59.804769+0000","last_became_peered":"2024-03-06T07:22:59.804769+0000","last_unstale":"2024-03-06T07:23:02.978852+0000","last_undegraded":"2024-03-06T07:23:02.978852+0000","last_fullsized":"2024-03-06T07:23:02.978852+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:29.015317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.609936+0000","last_change":"2024-03-06T07:23:00.824529+0000","last_active":"2024-03-06T07:23:02.609936+0000","last_peered":"2024-03-06T07:23:02.609936+0000","last_clean":"2024-03-06T07:23:02.609936+0000","last_became_active":"2024-03-06T07:23:00.824226+0000","last_became_peered":"2024-03-06T07:23:00.824226+0000","last_unstale":"2024-03-06T07:23:02.609936+0000","last_undegraded":"2024-03-06T07:23:02.609936+0000","last_fullsized":"2024-03-06T07:23:02.609936+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:51:48.943812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:01.821721+0000","last_change":"2024-03-06T07:22:59.804481+0000","last_active":"2024-03-06T07:23:01.821721+0000","last_peered":"2024-03-06T07:23:01.821721+0000","last_clean":"2024-03-06T07:23:01.821721+0000","last_became_active":"2024-03-06T07:22:59.803413+0000","last_became_peered":"2024-03-06T07:22:59.803413+0000","last_unstale":"2024-03-06T07:23:01.821721+0000","last_undegraded":"2024-03-06T07:23:01.821721+0000","last_fullsized":"2024-03-06T07:23:01.821721+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:04:10.795313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.478434+0000","last_change":"2024-03-06T07:23:00.817186+0000","last_active":"2024-03-06T07:23:02.478434+0000","last_peered":"2024-03-06T07:23:02.478434+0000","last_clean":"2024-03-06T07:23:02.478434+0000","last_became_active":"2024-03-06T07:23:00.816909+0000","last_became_peered":"2024-03-06T07:23:00.816909+0000","last_unstale":"2024-03-06T07:23:02.478434+0000","last_undegraded":"2024-03-06T07:23:02.478434+0000","last_fullsized":"2024-03-06T07:23:02.478434+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:42:57.165308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.979634+0000","last_change":"2024-03-06T07:22:59.808659+0000","last_active":"2024-03-06T07:23:02.979634+0000","last_peered":"2024-03-06T07:23:02.979634+0000","last_clean":"2024-03-06T07:23:02.979634+0000","last_became_active":"2024-03-06T07:22:59.808416+0000","last_became_peered":"2024-03-06T07:22:59.808416+0000","last_unstale":"2024-03-06T07:23:02.979634+0000","last_undegraded":"2024-03-06T07:23:02.979634+0000","last_fullsized":"2024-03-06T07:23:02.979634+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:36:44.009182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.978886+0000","last_change":"2024-03-06T07:23:00.821608+0000","last_active":"2024-03-06T07:23:02.978886+0000","last_peered":"2024-03-06T07:23:02.978886+0000","last_clean":"2024-03-06T07:23:02.978886+0000","last_became_active":"2024-03-06T07:23:00.821182+0000","last_became_peered":"2024-03-06T07:23:00.821182+0000","last_unstale":"2024-03-06T07:23:02.978886+0000","last_undegraded":"2024-03-06T07:23:02.978886+0000","last_fullsized":"2024-03-06T07:23:02.978886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:41:29.030630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:01.822389+0000","last_change":"2024-03-06T07:22:59.804638+0000","last_active":"2024-03-06T07:23:01.822389+0000","last_peered":"2024-03-06T07:23:01.822389+0000","last_clean":"2024-03-06T07:23:01.822389+0000","last_became_active":"2024-03-06T07:22:59.804026+0000","last_became_peered":"2024-03-06T07:22:59.804026+0000","last_unstale":"2024-03-06T07:23:01.822389+0000","last_undegraded":"2024-03-06T07:23:01.822389+0000","last_fullsized":"2024-03-06T07:23:01.822389+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:19.337313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.979538+0000","last_change":"2024-03-06T07:23:00.822938+0000","last_active":"2024-03-06T07:23:02.979538+0000","last_peered":"2024-03-06T07:23:02.979538+0000","last_clean":"2024-03-06T07:23:02.979538+0000","last_became_active":"2024-03-06T07:23:00.822694+0000","last_became_peered":"2024-03-06T07:23:00.822694+0000","last_unstale":"2024-03-06T07:23:02.979538+0000","last_undegraded":"2024-03-06T07:23:02.979538+0000","last_fullsized":"2024-03-06T07:23:02.979538+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:29.494971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"32'3","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.478495+0000","last_change":"2024-03-06T07:22:59.815296+0000","last_active":"2024-03-06T07:23:02.478495+0000","last_peered":"2024-03-06T07:23:02.478495+0000","last_clean":"2024-03-06T07:23:02.478495+0000","last_became_active":"2024-03-06T07:22:59.815033+0000","last_became_peered":"2024-03-06T07:22:59.815033+0000","last_unstale":"2024-03-06T07:23:02.478495+0000","last_undegraded":"2024-03-06T07:23:02.478495+0000","last_fullsized":"2024-03-06T07:23:02.478495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:59:24.519682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.478400+0000","last_change":"2024-03-06T07:23:00.828559+0000","last_active":"2024-03-06T07:23:02.478400+0000","last_peered":"2024-03-06T07:23:02.478400+0000","last_clean":"2024-03-06T07:23:02.478400+0000","last_became_active":"2024-03-06T07:23:00.827857+0000","last_became_peered":"2024-03-06T07:23:00.827857+0000","last_unstale":"2024-03-06T07:23:02.478400+0000","last_undegraded":"2024-03-06T07:23:02.478400+0000","last_fullsized":"2024-03-06T07:23:02.478400+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:50.287073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"32'6","reported_seq":20,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.979596+0000","last_change":"2024-03-06T07:22:59.807399+0000","last_active":"2024-03-06T07:23:02.979596+0000","last_peered":"2024-03-06T07:23:02.979596+0000","last_clean":"2024-03-06T07:23:02.979596+0000","last_became_active":"2024-03-06T07:22:59.807150+0000","last_became_peered":"2024-03-06T07:22:59.807150+0000","last_unstale":"2024-03-06T07:23:02.979596+0000","last_undegraded":"2024-03-06T07:23:02.979596+0000","last_fullsized":"2024-03-06T07:23:02.979596+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:56:30.621632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.610014+0000","last_change":"2024-03-06T07:23:00.825211+0000","last_active":"2024-03-06T07:23:02.610014+0000","last_peered":"2024-03-06T07:23:02.610014+0000","last_clean":"2024-03-06T07:23:02.610014+0000","last_became_active":"2024-03-06T07:23:00.824901+0000","last_became_peered":"2024-03-06T07:23:00.824901+0000","last_unstale":"2024-03-06T07:23:02.610014+0000","last_undegraded":"2024-03-06T07:23:02.610014+0000","last_fullsized":"2024-03-06T07:23:02.610014+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:33.703308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:01.822365+0000","last_change":"2024-03-06T07:22:59.809344+0000","last_active":"2024-03-06T07:23:01.822365+0000","last_peered":"2024-03-06T07:23:01.822365+0000","last_clean":"2024-03-06T07:23:01.822365+0000","last_became_active":"2024-03-06T07:22:59.809076+0000","last_became_peered":"2024-03-06T07:22:59.809076+0000","last_unstale":"2024-03-06T07:23:01.822365+0000","last_undegraded":"2024-03-06T07:23:01.822365+0000","last_fullsized":"2024-03-06T07:23:01.822365+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:08:18.005082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.979589+0000","last_change":"2024-03-06T07:23:00.818727+0000","last_active":"2024-03-06T07:23:02.979589+0000","last_peered":"2024-03-06T07:23:02.979589+0000","last_clean":"2024-03-06T07:23:02.979589+0000","last_became_active":"2024-03-06T07:23:00.818466+0000","last_became_peered":"2024-03-06T07:23:00.818466+0000","last_unstale":"2024-03-06T07:23:02.979589+0000","last_undegraded":"2024-03-06T07:23:02.979589+0000","last_fullsized":"2024-03-06T07:23:02.979589+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:19:47.710031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.609967+0000","last_change":"2024-03-06T07:22:59.807007+0000","last_active":"2024-03-06T07:23:02.609967+0000","last_peered":"2024-03-06T07:23:02.609967+0000","last_clean":"2024-03-06T07:23:02.609967+0000","last_became_active":"2024-03-06T07:22:59.806750+0000","last_became_peered":"2024-03-06T07:22:59.806750+0000","last_unstale":"2024-03-06T07:23:02.609967+0000","last_undegraded":"2024-03-06T07:23:02.609967+0000","last_fullsized":"2024-03-06T07:23:02.609967+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:08:18.005082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.614930+0000","last_change":"2024-03-06T07:23:00.833298+0000","last_active":"2024-03-06T07:23:02.614930+0000","last_peered":"2024-03-06T07:23:02.614930+0000","last_clean":"2024-03-06T07:23:02.614930+0000","last_became_active":"2024-03-06T07:23:00.833033+0000","last_became_peered":"2024-03-06T07:23:00.833033+0000","last_unstale":"2024-03-06T07:23:02.614930+0000","last_undegraded":"2024-03-06T07:23:02.614930+0000","last_fullsized":"2024-03-06T07:23:02.614930+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:29.494971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.978942+0000","last_change":"2024-03-06T07:22:59.808043+0000","last_active":"2024-03-06T07:23:02.978942+0000","last_peered":"2024-03-06T07:23:02.978942+0000","last_clean":"2024-03-06T07:23:02.978942+0000","last_became_active":"2024-03-06T07:22:59.807794+0000","last_became_peered":"2024-03-06T07:22:59.807794+0000","last_unstale":"2024-03-06T07:23:02.978942+0000","last_undegraded":"2024-03-06T07:23:02.978942+0000","last_fullsized":"2024-03-06T07:23:02.978942+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:47:34.912255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:01.821558+0000","last_change":"2024-03-06T07:23:00.823000+0000","last_active":"2024-03-06T07:23:01.821558+0000","last_peered":"2024-03-06T07:23:01.821558+0000","last_clean":"2024-03-06T07:23:01.821558+0000","last_became_active":"2024-03-06T07:23:00.822748+0000","last_became_peered":"2024-03-06T07:23:00.822748+0000","last_unstale":"2024-03-06T07:23:01.821558+0000","last_undegraded":"2024-03-06T07:23:01.821558+0000","last_fullsized":"2024-03-06T07:23:01.821558+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:56:31.625723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"32'8","reported_seq":22,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.978602+0000","last_change":"2024-03-06T07:22:59.806459+0000","last_active":"2024-03-06T07:23:02.978602+0000","last_peered":"2024-03-06T07:23:02.978602+0000","last_clean":"2024-03-06T07:23:02.978602+0000","last_became_active":"2024-03-06T07:22:59.806194+0000","last_became_peered":"2024-03-06T07:22:59.806194+0000","last_unstale":"2024-03-06T07:23:02.978602+0000","last_undegraded":"2024-03-06T07:23:02.978602+0000","last_fullsized":"2024-03-06T07:23:02.978602+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:37.715070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:01.821771+0000","last_change":"2024-03-06T07:23:00.822497+0000","last_active":"2024-03-06T07:23:01.821771+0000","last_peered":"2024-03-06T07:23:01.821771+0000","last_clean":"2024-03-06T07:23:01.821771+0000","last_became_active":"2024-03-06T07:23:00.822229+0000","last_became_peered":"2024-03-06T07:23:00.822229+0000","last_unstale":"2024-03-06T07:23:01.821771+0000","last_undegraded":"2024-03-06T07:23:01.821771+0000","last_fullsized":"2024-03-06T07:23:01.821771+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:18.050757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.610384+0000","last_change":"2024-03-06T07:22:59.808555+0000","last_active":"2024-03-06T07:23:02.610384+0000","last_peered":"2024-03-06T07:23:02.610384+0000","last_clean":"2024-03-06T07:23:02.610384+0000","last_became_active":"2024-03-06T07:22:59.808284+0000","last_became_peered":"2024-03-06T07:22:59.808284+0000","last_unstale":"2024-03-06T07:23:02.610384+0000","last_undegraded":"2024-03-06T07:23:02.610384+0000","last_fullsized":"2024-03-06T07:23:02.610384+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:33.788501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.478540+0000","last_change":"2024-03-06T07:23:00.827669+0000","last_active":"2024-03-06T07:23:02.478540+0000","last_peered":"2024-03-06T07:23:02.478540+0000","last_clean":"2024-03-06T07:23:02.478540+0000","last_became_active":"2024-03-06T07:23:00.827400+0000","last_became_peered":"2024-03-06T07:23:00.827400+0000","last_unstale":"2024-03-06T07:23:02.478540+0000","last_undegraded":"2024-03-06T07:23:02.478540+0000","last_fullsized":"2024-03-06T07:23:02.478540+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:27.157625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:01.821983+0000","last_change":"2024-03-06T07:22:59.802730+0000","last_active":"2024-03-06T07:23:01.821983+0000","last_peered":"2024-03-06T07:23:01.821983+0000","last_clean":"2024-03-06T07:23:01.821983+0000","last_became_active":"2024-03-06T07:22:59.802461+0000","last_became_peered":"2024-03-06T07:22:59.802461+0000","last_unstale":"2024-03-06T07:23:01.821983+0000","last_undegraded":"2024-03-06T07:23:01.821983+0000","last_fullsized":"2024-03-06T07:23:01.821983+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:36:12.264660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.610142+0000","last_change":"2024-03-06T07:23:00.819667+0000","last_active":"2024-03-06T07:23:02.610142+0000","last_peered":"2024-03-06T07:23:02.610142+0000","last_clean":"2024-03-06T07:23:02.610142+0000","last_became_active":"2024-03-06T07:23:00.819381+0000","last_became_peered":"2024-03-06T07:23:00.819381+0000","last_unstale":"2024-03-06T07:23:02.610142+0000","last_undegraded":"2024-03-06T07:23:02.610142+0000","last_fullsized":"2024-03-06T07:23:02.610142+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:17.020204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:01.822259+0000","last_change":"2024-03-06T07:22:59.807016+0000","last_active":"2024-03-06T07:23:01.822259+0000","last_peered":"2024-03-06T07:23:01.822259+0000","last_clean":"2024-03-06T07:23:01.822259+0000","last_became_active":"2024-03-06T07:22:59.806733+0000","last_became_peered":"2024-03-06T07:22:59.806733+0000","last_unstale":"2024-03-06T07:23:01.822259+0000","last_undegraded":"2024-03-06T07:23:01.822259+0000","last_fullsized":"2024-03-06T07:23:01.822259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:53.894652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.978844+0000","last_change":"2024-03-06T07:23:00.830060+0000","last_active":"2024-03-06T07:23:02.978844+0000","last_peered":"2024-03-06T07:23:02.978844+0000","last_clean":"2024-03-06T07:23:02.978844+0000","last_became_active":"2024-03-06T07:23:00.828964+0000","last_became_peered":"2024-03-06T07:23:00.828964+0000","last_unstale":"2024-03-06T07:23:02.978844+0000","last_undegraded":"2024-03-06T07:23:02.978844+0000","last_fullsized":"2024-03-06T07:23:02.978844+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:54.898743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:01.822068+0000","last_change":"2024-03-06T07:22:59.808721+0000","last_active":"2024-03-06T07:23:01.822068+0000","last_peered":"2024-03-06T07:23:01.822068+0000","last_clean":"2024-03-06T07:23:01.822068+0000","last_became_active":"2024-03-06T07:22:59.808465+0000","last_became_peered":"2024-03-06T07:22:59.808465+0000","last_unstale":"2024-03-06T07:23:01.822068+0000","last_undegraded":"2024-03-06T07:23:01.822068+0000","last_fullsized":"2024-03-06T07:23:01.822068+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:24:26.919565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.978613+0000","last_change":"2024-03-06T07:23:00.820608+0000","last_active":"2024-03-06T07:23:02.978613+0000","last_peered":"2024-03-06T07:23:02.978613+0000","last_clean":"2024-03-06T07:23:02.978613+0000","last_became_active":"2024-03-06T07:23:00.820245+0000","last_became_peered":"2024-03-06T07:23:00.820245+0000","last_unstale":"2024-03-06T07:23:02.978613+0000","last_undegraded":"2024-03-06T07:23:02.978613+0000","last_fullsized":"2024-03-06T07:23:02.978613+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:30:22.974526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:01.822218+0000","last_change":"2024-03-06T07:22:59.804367+0000","last_active":"2024-03-06T07:23:01.822218+0000","last_peered":"2024-03-06T07:23:01.822218+0000","last_clean":"2024-03-06T07:23:01.822218+0000","last_became_active":"2024-03-06T07:22:59.804115+0000","last_became_peered":"2024-03-06T07:22:59.804115+0000","last_unstale":"2024-03-06T07:23:01.822218+0000","last_undegraded":"2024-03-06T07:23:01.822218+0000","last_fullsized":"2024-03-06T07:23:01.822218+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:53.894652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.979292+0000","last_change":"2024-03-06T07:23:00.821915+0000","last_active":"2024-03-06T07:23:02.979292+0000","last_peered":"2024-03-06T07:23:02.979292+0000","last_clean":"2024-03-06T07:23:02.979292+0000","last_became_active":"2024-03-06T07:23:00.821637+0000","last_became_peered":"2024-03-06T07:23:00.821637+0000","last_unstale":"2024-03-06T07:23:02.979292+0000","last_undegraded":"2024-03-06T07:23:02.979292+0000","last_fullsized":"2024-03-06T07:23:02.979292+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:34.792592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:01.821526+0000","last_change":"2024-03-06T07:22:59.804749+0000","last_active":"2024-03-06T07:23:01.821526+0000","last_peered":"2024-03-06T07:23:01.821526+0000","last_clean":"2024-03-06T07:23:01.821526+0000","last_became_active":"2024-03-06T07:22:59.802839+0000","last_became_peered":"2024-03-06T07:22:59.802839+0000","last_unstale":"2024-03-06T07:23:01.821526+0000","last_undegraded":"2024-03-06T07:23:01.821526+0000","last_fullsized":"2024-03-06T07:23:01.821526+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:30:21.970435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.978957+0000","last_change":"2024-03-06T07:23:00.829760+0000","last_active":"2024-03-06T07:23:02.978957+0000","last_peered":"2024-03-06T07:23:02.978957+0000","last_clean":"2024-03-06T07:23:02.978957+0000","last_became_active":"2024-03-06T07:23:00.829500+0000","last_became_peered":"2024-03-06T07:23:00.829500+0000","last_unstale":"2024-03-06T07:23:02.978957+0000","last_undegraded":"2024-03-06T07:23:02.978957+0000","last_fullsized":"2024-03-06T07:23:02.978957+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:20.341404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:01.821637+0000","last_change":"2024-03-06T07:22:59.810390+0000","last_active":"2024-03-06T07:23:01.821637+0000","last_peered":"2024-03-06T07:23:01.821637+0000","last_clean":"2024-03-06T07:23:01.821637+0000","last_became_active":"2024-03-06T07:22:59.810140+0000","last_became_peered":"2024-03-06T07:22:59.810140+0000","last_unstale":"2024-03-06T07:23:01.821637+0000","last_undegraded":"2024-03-06T07:23:01.821637+0000","last_fullsized":"2024-03-06T07:23:01.821637+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:44:35.664305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.614632+0000","last_change":"2024-03-06T07:23:00.831081+0000","last_active":"2024-03-06T07:23:02.614632+0000","last_peered":"2024-03-06T07:23:02.614632+0000","last_clean":"2024-03-06T07:23:02.614632+0000","last_became_active":"2024-03-06T07:23:00.830285+0000","last_became_peered":"2024-03-06T07:23:00.830285+0000","last_unstale":"2024-03-06T07:23:02.614632+0000","last_undegraded":"2024-03-06T07:23:02.614632+0000","last_fullsized":"2024-03-06T07:23:02.614632+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:45:54.192806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.477984+0000","last_change":"2024-03-06T07:22:59.812896+0000","last_active":"2024-03-06T07:23:02.477984+0000","last_peered":"2024-03-06T07:23:02.477984+0000","last_clean":"2024-03-06T07:23:02.477984+0000","last_became_active":"2024-03-06T07:22:59.812598+0000","last_became_peered":"2024-03-06T07:22:59.812598+0000","last_unstale":"2024-03-06T07:23:02.477984+0000","last_undegraded":"2024-03-06T07:23:02.477984+0000","last_fullsized":"2024-03-06T07:23:02.477984+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:56:30.621632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.978846+0000","last_change":"2024-03-06T07:23:00.823809+0000","last_active":"2024-03-06T07:23:02.978846+0000","last_peered":"2024-03-06T07:23:02.978846+0000","last_clean":"2024-03-06T07:23:02.978846+0000","last_became_active":"2024-03-06T07:23:00.823457+0000","last_became_peered":"2024-03-06T07:23:00.823457+0000","last_unstale":"2024-03-06T07:23:02.978846+0000","last_undegraded":"2024-03-06T07:23:02.978846+0000","last_fullsized":"2024-03-06T07:23:02.978846+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:38:32.118661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.610422+0000","last_change":"2024-03-06T07:22:59.811226+0000","last_active":"2024-03-06T07:23:02.610422+0000","last_peered":"2024-03-06T07:23:02.610422+0000","last_clean":"2024-03-06T07:23:02.610422+0000","last_became_active":"2024-03-06T07:22:59.810822+0000","last_became_peered":"2024-03-06T07:22:59.810822+0000","last_unstale":"2024-03-06T07:23:02.610422+0000","last_undegraded":"2024-03-06T07:23:02.610422+0000","last_fullsized":"2024-03-06T07:23:02.610422+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:37.715070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.478627+0000","last_change":"2024-03-06T07:23:00.825206+0000","last_active":"2024-03-06T07:23:02.478627+0000","last_peered":"2024-03-06T07:23:02.478627+0000","last_clean":"2024-03-06T07:23:02.478627+0000","last_became_active":"2024-03-06T07:23:00.824931+0000","last_became_peered":"2024-03-06T07:23:00.824931+0000","last_unstale":"2024-03-06T07:23:02.478627+0000","last_undegraded":"2024-03-06T07:23:02.478627+0000","last_fullsized":"2024-03-06T07:23:02.478627+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:45.396857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:01.822397+0000","last_change":"2024-03-06T07:22:59.804853+0000","last_active":"2024-03-06T07:23:01.822397+0000","last_peered":"2024-03-06T07:23:01.822397+0000","last_clean":"2024-03-06T07:23:01.822397+0000","last_became_active":"2024-03-06T07:22:59.804353+0000","last_became_peered":"2024-03-06T07:22:59.804353+0000","last_unstale":"2024-03-06T07:23:01.822397+0000","last_undegraded":"2024-03-06T07:23:01.822397+0000","last_fullsized":"2024-03-06T07:23:01.822397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:08:18.005082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:01.822288+0000","last_change":"2024-03-06T07:23:00.826325+0000","last_active":"2024-03-06T07:23:01.822288+0000","last_peered":"2024-03-06T07:23:01.822288+0000","last_clean":"2024-03-06T07:23:01.822288+0000","last_became_active":"2024-03-06T07:23:00.826068+0000","last_became_peered":"2024-03-06T07:23:00.826068+0000","last_unstale":"2024-03-06T07:23:01.822288+0000","last_undegraded":"2024-03-06T07:23:01.822288+0000","last_fullsized":"2024-03-06T07:23:01.822288+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:36:45.013273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.610198+0000","last_change":"2024-03-06T07:22:59.803415+0000","last_active":"2024-03-06T07:23:02.610198+0000","last_peered":"2024-03-06T07:23:02.610198+0000","last_clean":"2024-03-06T07:23:02.610198+0000","last_became_active":"2024-03-06T07:22:59.803175+0000","last_became_peered":"2024-03-06T07:22:59.803175+0000","last_unstale":"2024-03-06T07:23:02.610198+0000","last_undegraded":"2024-03-06T07:23:02.610198+0000","last_fullsized":"2024-03-06T07:23:02.610198+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:19.337313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:01.821408+0000","last_change":"2024-03-06T07:23:00.815137+0000","last_active":"2024-03-06T07:23:01.821408+0000","last_peered":"2024-03-06T07:23:01.821408+0000","last_clean":"2024-03-06T07:23:01.821408+0000","last_became_active":"2024-03-06T07:23:00.814800+0000","last_became_peered":"2024-03-06T07:23:00.814800+0000","last_unstale":"2024-03-06T07:23:01.821408+0000","last_undegraded":"2024-03-06T07:23:01.821408+0000","last_fullsized":"2024-03-06T07:23:01.821408+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:38.719161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:01.822356+0000","last_change":"2024-03-06T07:22:59.810599+0000","last_active":"2024-03-06T07:23:01.822356+0000","last_peered":"2024-03-06T07:23:01.822356+0000","last_clean":"2024-03-06T07:23:01.822356+0000","last_became_active":"2024-03-06T07:22:59.810291+0000","last_became_peered":"2024-03-06T07:22:59.810291+0000","last_unstale":"2024-03-06T07:23:01.822356+0000","last_undegraded":"2024-03-06T07:23:01.822356+0000","last_fullsized":"2024-03-06T07:23:01.822356+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:38:31.114570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:01.822229+0000","last_change":"2024-03-06T07:23:00.817687+0000","last_active":"2024-03-06T07:23:01.822229+0000","last_peered":"2024-03-06T07:23:01.822229+0000","last_clean":"2024-03-06T07:23:01.822229+0000","last_became_active":"2024-03-06T07:23:00.817423+0000","last_became_peered":"2024-03-06T07:23:00.817423+0000","last_unstale":"2024-03-06T07:23:01.822229+0000","last_undegraded":"2024-03-06T07:23:01.822229+0000","last_fullsized":"2024-03-06T07:23:01.822229+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:12:16.981197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.610114+0000","last_change":"2024-03-06T07:22:59.812044+0000","last_active":"2024-03-06T07:23:02.610114+0000","last_peered":"2024-03-06T07:23:02.610114+0000","last_clean":"2024-03-06T07:23:02.610114+0000","last_became_active":"2024-03-06T07:22:59.811822+0000","last_became_peered":"2024-03-06T07:22:59.811822+0000","last_unstale":"2024-03-06T07:23:02.610114+0000","last_undegraded":"2024-03-06T07:23:02.610114+0000","last_fullsized":"2024-03-06T07:23:02.610114+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:32.447994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.978591+0000","last_change":"2024-03-06T07:23:00.812684+0000","last_active":"2024-03-06T07:23:02.978591+0000","last_peered":"2024-03-06T07:23:02.978591+0000","last_clean":"2024-03-06T07:23:02.978591+0000","last_became_active":"2024-03-06T07:23:00.812384+0000","last_became_peered":"2024-03-06T07:23:00.812384+0000","last_unstale":"2024-03-06T07:23:02.978591+0000","last_undegraded":"2024-03-06T07:23:02.978591+0000","last_fullsized":"2024-03-06T07:23:02.978591+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:45:54.192806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:01.821741+0000","last_change":"2024-03-06T07:22:59.809833+0000","last_active":"2024-03-06T07:23:01.821741+0000","last_peered":"2024-03-06T07:23:01.821741+0000","last_clean":"2024-03-06T07:23:01.821741+0000","last_became_active":"2024-03-06T07:22:59.809550+0000","last_became_peered":"2024-03-06T07:22:59.809550+0000","last_unstale":"2024-03-06T07:23:01.821741+0000","last_undegraded":"2024-03-06T07:23:01.821741+0000","last_fullsized":"2024-03-06T07:23:01.821741+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:19.337313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:01.821525+0000","last_change":"2024-03-06T07:23:00.821601+0000","last_active":"2024-03-06T07:23:01.821525+0000","last_peered":"2024-03-06T07:23:01.821525+0000","last_clean":"2024-03-06T07:23:01.821525+0000","last_became_active":"2024-03-06T07:23:00.820913+0000","last_became_peered":"2024-03-06T07:23:00.820913+0000","last_unstale":"2024-03-06T07:23:01.821525+0000","last_undegraded":"2024-03-06T07:23:01.821525+0000","last_fullsized":"2024-03-06T07:23:01.821525+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:36:45.013273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.978680+0000","last_change":"2024-03-06T07:22:59.812895+0000","last_active":"2024-03-06T07:23:02.978680+0000","last_peered":"2024-03-06T07:23:02.978680+0000","last_clean":"2024-03-06T07:23:02.978680+0000","last_became_active":"2024-03-06T07:22:59.812605+0000","last_became_peered":"2024-03-06T07:22:59.812605+0000","last_unstale":"2024-03-06T07:23:02.978680+0000","last_undegraded":"2024-03-06T07:23:02.978680+0000","last_fullsized":"2024-03-06T07:23:02.978680+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:59:24.519682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.614865+0000","last_change":"2024-03-06T07:23:00.827003+0000","last_active":"2024-03-06T07:23:02.614865+0000","last_peered":"2024-03-06T07:23:02.614865+0000","last_clean":"2024-03-06T07:23:02.614865+0000","last_became_active":"2024-03-06T07:23:00.826672+0000","last_became_peered":"2024-03-06T07:23:00.826672+0000","last_unstale":"2024-03-06T07:23:02.614865+0000","last_undegraded":"2024-03-06T07:23:02.614865+0000","last_fullsized":"2024-03-06T07:23:02.614865+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:59.788978+0000","last_clean_scrub_stamp":"2024-03-06T07:22:59.788978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:50.287073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:01.821162+0000","last_change":"2024-03-06T07:22:59.808026+0000","last_active":"2024-03-06T07:23:01.821162+0000","last_peered":"2024-03-06T07:23:01.821162+0000","last_clean":"2024-03-06T07:23:01.821162+0000","last_became_active":"2024-03-06T07:22:59.807197+0000","last_became_peered":"2024-03-06T07:22:59.807197+0000","last_unstale":"2024-03-06T07:23:01.821162+0000","last_undegraded":"2024-03-06T07:23:01.821162+0000","last_fullsized":"2024-03-06T07:23:01.821162+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:58.784887+0000","last_clean_scrub_stamp":"2024-03-06T07:22:58.784887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:24:26.919565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":193,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:02.615035+0000","last_change":"2024-03-06T07:21:47.553693+0000","last_active":"2024-03-06T07:23:02.615035+0000","last_peered":"2024-03-06T07:23:02.615035+0000","last_clean":"2024-03-06T07:23:02.615035+0000","last_became_active":"2024-03-06T07:21:47.552664+0000","last_became_peered":"2024-03-06T07:21:47.552664+0000","last_unstale":"2024-03-06T07:23:02.615035+0000","last_undegraded":"2024-03-06T07:23:02.615035+0000","last_fullsized":"2024-03-06T07:23:02.615035+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_clean_scrub_stamp":"2024-03-06T07:21:46.362028+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:02.916426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":970752,"data_stored":737568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":147726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574868,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":147782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":147214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574868,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":148744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574868,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":376832,"data_stored":147180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":148744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":966656,"data_stored":737478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:23:08.920 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:23:08.921 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:23:09.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.065+0000 7f9db38b5640 1 Processor -- start 2024-03-06T07:23:09.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.065+0000 7f9db38b5640 1 -- start start 2024-03-06T07:23:09.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.066+0000 7f9db38b5640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f9dac0cf9a0 0x7f9dac0b5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.066+0000 7f9db38b5640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9dac0b55e0 0x7f9dac0ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.066+0000 7f9db38b5640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dac06b710 con 0x7f9dac066d90 2024-03-06T07:23:09.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.066+0000 7f9db38b5640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dac06b150 con 0x7f9dac0cf9a0 2024-03-06T07:23:09.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.066+0000 7f9db38b5640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dac06b430 con 0x7f9dac0b55e0 2024-03-06T07:23:09.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.066+0000 7f9db0e50640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f9dac066d90 0x7f9dac0b4ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:37356/0 (socket says 172.21.15.83:37356) 2024-03-06T07:23:09.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.066+0000 7f9db0e50640 1 -- 172.21.15.83:0/3369163375 learned_addr learned my addr 172.21.15.83:0/3369163375 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:23:09.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.066+0000 7f9db1651640 1 --2- 172.21.15.83:0/3369163375 >> v2:172.21.15.183:3300/0 conn(0x7f9dac0cf9a0 0x7f9dac0b5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.066+0000 7f9da3fff640 1 -- 172.21.15.83:0/3369163375 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 776612545 0 0) 0x7f9dac06b710 con 0x7f9dac066d90 2024-03-06T07:23:09.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.066+0000 7f9da3fff640 1 -- 172.21.15.83:0/3369163375 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d98003520 con 0x7f9dac066d90 2024-03-06T07:23:09.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.066+0000 7f9db1e52640 1 --2- 172.21.15.83:0/3369163375 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9dac0b55e0 0x7f9dac0ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.066+0000 7f9da3fff640 1 -- 172.21.15.83:0/3369163375 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3058394549 0 0) 0x7f9d98003520 con 0x7f9dac066d90 2024-03-06T07:23:09.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.066+0000 7f9da3fff640 1 -- 172.21.15.83:0/3369163375 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9dac0b55e0 msgr2=0x7f9dac0ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:09.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.066+0000 7f9da3fff640 1 --2- 172.21.15.83:0/3369163375 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9dac0b55e0 0x7f9dac0ce040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.066+0000 7f9da3fff640 1 -- 172.21.15.83:0/3369163375 >> v2:172.21.15.183:3300/0 conn(0x7f9dac0cf9a0 msgr2=0x7f9dac0b5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:09.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.066+0000 7f9da3fff640 1 --2- 172.21.15.83:0/3369163375 >> v2:172.21.15.183:3300/0 conn(0x7f9dac0cf9a0 0x7f9dac0b5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.066+0000 7f9da3fff640 1 -- 172.21.15.83:0/3369163375 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dac0b0970 con 0x7f9dac066d90 2024-03-06T07:23:09.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.066+0000 7f9da3fff640 1 -- 172.21.15.83:0/3369163375 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f9da8001670 con 0x7f9dac066d90 2024-03-06T07:23:09.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.066+0000 7f9da3fff640 1 -- 172.21.15.83:0/3369163375 >> v1:172.21.15.83:6789/0 conn(0x7f9dac066d90 legacy=0x7f9dac0b4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.066+0000 7f9da3fff640 1 --2- 172.21.15.83:0/3369163375 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d980039b0 0x7f9d98003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.066+0000 7f9da3fff640 1 --2- 172.21.15.83:0/3369163375 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9dac0b55e0 0x7f9d98004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.066+0000 7f9da3fff640 1 --2- 172.21.15.83:0/3369163375 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9d98004d40 0x7f9d980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.066+0000 7f9da3fff640 1 -- 172.21.15.83:0/3369163375 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dac06b150 con 0x7f9d980039b0 2024-03-06T07:23:09.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.066+0000 7f9da3fff640 1 -- 172.21.15.83:0/3369163375 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dac06b430 con 0x7f9d98004d40 2024-03-06T07:23:09.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.066+0000 7f9da3fff640 1 -- 172.21.15.83:0/3369163375 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9800b9c0 con 0x7f9dac0b55e0 2024-03-06T07:23:09.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.067+0000 7f9db1651640 1 --2- 172.21.15.83:0/3369163375 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d980039b0 0x7f9d98003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.067+0000 7f9db0e50640 1 --2- 172.21.15.83:0/3369163375 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9dac0b55e0 0x7f9d98004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.067+0000 7f9db1e52640 1 --2- 172.21.15.83:0/3369163375 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9d98004d40 0x7f9d980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.067+0000 7f9db1651640 1 -- 172.21.15.83:0/3369163375 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9dac0b55e0 msgr2=0x7f9d98004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:09.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.067+0000 7f9db1651640 1 --2- 172.21.15.83:0/3369163375 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9dac0b55e0 0x7f9d98004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.067+0000 7f9db1651640 1 -- 172.21.15.83:0/3369163375 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9d98004d40 msgr2=0x7f9d980051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:09.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.067+0000 7f9db1651640 1 --2- 172.21.15.83:0/3369163375 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9d98004d40 0x7f9d980051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.067+0000 7f9db1651640 1 -- 172.21.15.83:0/3369163375 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d98003520 con 0x7f9d980039b0 2024-03-06T07:23:09.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.067+0000 7f9db1e52640 1 --2- 172.21.15.83:0/3369163375 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9d98004d40 0x7f9d980051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:09.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.068+0000 7f9db1651640 1 --2- 172.21.15.83:0/3369163375 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d980039b0 0x7f9d98003dd0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c009af0 tx=0x7f9d98005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb8f43f70734de92 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:09.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.068+0000 7f9da3fff640 1 -- 172.21.15.83:0/3369163375 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d9c039070 con 0x7f9d980039b0 2024-03-06T07:23:09.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.068+0000 7f9da3fff640 1 -- 172.21.15.83:0/3369163375 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d9c02c710 con 0x7f9d980039b0 2024-03-06T07:23:09.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.068+0000 7f9da3fff640 1 -- 172.21.15.83:0/3369163375 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d9c009690 con 0x7f9d980039b0 2024-03-06T07:23:09.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.068+0000 7f9db38b5640 1 -- 172.21.15.83:0/3369163375 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d980039b0 msgr2=0x7f9d98003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.068+0000 7f9db38b5640 1 --2- 172.21.15.83:0/3369163375 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d980039b0 0x7f9d98003dd0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c009af0 tx=0x7f9d98005830 comp rx=0 tx=0).stop 2024-03-06T07:23:09.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.068+0000 7f9db1e52640 1 -- 172.21.15.83:0/3369163375 reap_dead start 2024-03-06T07:23:09.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.068+0000 7f9db38b5640 1 -- 172.21.15.83:0/3369163375 shutdown_connections 2024-03-06T07:23:09.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.068+0000 7f9db38b5640 1 -- 172.21.15.83:0/3369163375 >> 172.21.15.83:0/3369163375 conn(0x7f9dac0b1d70 msgr2=0x7f9dac0b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:09.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.068+0000 7f9db38b5640 1 -- 172.21.15.83:0/3369163375 shutdown_connections 2024-03-06T07:23:09.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.068+0000 7f9db38b5640 1 -- 172.21.15.83:0/3369163375 wait complete. 2024-03-06T07:23:09.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.069+0000 7f9db38b5640 1 Processor -- start 2024-03-06T07:23:09.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.070+0000 7f9db38b5640 1 -- start start 2024-03-06T07:23:09.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.070+0000 7f9db38b5640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9d980039b0 0x7f9dac1565b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.070+0000 7f9db38b5640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9dac1f3f00 0x7f9dac1f7500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.070+0000 7f9db38b5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9dac1f4380 0x7f9dac0d0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.070+0000 7f9db38b5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dac0b6170 con 0x7f9dac1f4380 2024-03-06T07:23:09.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.070+0000 7f9db38b5640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dac0b5ff0 con 0x7f9dac1f3f00 2024-03-06T07:23:09.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.070+0000 7f9db38b5640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dac0cfd80 con 0x7f9d980039b0 2024-03-06T07:23:09.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.070+0000 7f9db1e52640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9dac1f4380 0x7f9dac0d0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.070+0000 7f9db1e52640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9dac1f4380 0x7f9dac0d0100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55654/0 (socket says 172.21.15.83:55654) 2024-03-06T07:23:09.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.070+0000 7f9db1e52640 1 -- 172.21.15.83:0/440971394 learned_addr learned my addr 172.21.15.83:0/440971394 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:09.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.071+0000 7f9db1651640 1 --2- 172.21.15.83:0/440971394 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9d980039b0 0x7f9dac1565b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.071+0000 7f9db0e50640 1 --2- 172.21.15.83:0/440971394 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9dac1f3f00 0x7f9dac1f7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.071+0000 7f9db1e52640 1 -- 172.21.15.83:0/440971394 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9d980039b0 msgr2=0x7f9dac1565b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.071+0000 7f9db1e52640 1 --2- 172.21.15.83:0/440971394 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9d980039b0 0x7f9dac1565b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.071+0000 7f9db1e52640 1 -- 172.21.15.83:0/440971394 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9dac1f3f00 msgr2=0x7f9dac1f7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.071+0000 7f9db1e52640 1 --2- 172.21.15.83:0/440971394 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9dac1f3f00 0x7f9dac1f7500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.071+0000 7f9db1e52640 1 -- 172.21.15.83:0/440971394 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dac1f7f30 con 0x7f9dac1f4380 2024-03-06T07:23:09.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.071+0000 7f9db1651640 1 --2- 172.21.15.83:0/440971394 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9d980039b0 0x7f9dac1565b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:09.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.071+0000 7f9db0e50640 1 --2- 172.21.15.83:0/440971394 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9dac1f3f00 0x7f9dac1f7500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:09.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.071+0000 7f9db1e52640 1 --2- 172.21.15.83:0/440971394 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9dac1f4380 0x7f9dac0d0100 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f9dac071110 tx=0x7f9da40115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:09.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.071+0000 7f9da1ffb640 1 -- 172.21.15.83:0/440971394 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da401a020 con 0x7f9dac1f4380 2024-03-06T07:23:09.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.071+0000 7f9da1ffb640 1 -- 172.21.15.83:0/440971394 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9da4007e60 con 0x7f9dac1f4380 2024-03-06T07:23:09.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.071+0000 7f9db38b5640 1 -- 172.21.15.83:0/440971394 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dac1f8210 con 0x7f9dac1f4380 2024-03-06T07:23:09.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.071+0000 7f9da1ffb640 1 -- 172.21.15.83:0/440971394 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da4016470 con 0x7f9dac1f4380 2024-03-06T07:23:09.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.071+0000 7f9db38b5640 1 -- 172.21.15.83:0/440971394 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dac0c86e0 con 0x7f9dac1f4380 2024-03-06T07:23:09.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.072+0000 7f9db38b5640 1 -- 172.21.15.83:0/440971394 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dac09e3d0 con 0x7f9dac1f4380 2024-03-06T07:23:09.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.072+0000 7f9da1ffb640 1 -- 172.21.15.83:0/440971394 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9da4005d10 con 0x7f9dac1f4380 2024-03-06T07:23:09.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.073+0000 7f9da1ffb640 1 --2- 172.21.15.83:0/440971394 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9d9006b4d0 0x7f9d9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.073+0000 7f9db1651640 1 --2- 172.21.15.83:0/440971394 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9d9006b4d0 0x7f9d9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.073+0000 7f9da1ffb640 1 -- 172.21.15.83:0/440971394 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9da408aa40 con 0x7f9dac1f4380 2024-03-06T07:23:09.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.073+0000 7f9db1651640 1 --2- 172.21.15.83:0/440971394 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9d9006b4d0 0x7f9d9006d990 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c036000 tx=0x7f9d9c030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:09.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.077+0000 7f9da1ffb640 1 -- 172.21.15.83:0/440971394 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9da4024070 con 0x7f9dac1f4380 2024-03-06T07:23:09.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.370+0000 7f9db38b5640 1 -- 172.21.15.83:0/440971394 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f9dac0ca240 con 0x7f9dac1f4380 2024-03-06T07:23:09.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.372+0000 7f9da1ffb640 1 -- 172.21.15.83:0/440971394 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) v1 ==== 190+0+147 (secure 0 0 0) 0x7f9da408ae20 con 0x7f9dac1f4380 2024-03-06T07:23:09.374 INFO:teuthology.orchestra.run.smithi083.stdout:[client.0] 2024-03-06T07:23:09.374 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBcGehlRwCIMhAA1DdbdQZPplu7LQOTK3BmSg== 2024-03-06T07:23:09.374 INFO:teuthology.orchestra.run.smithi083.stdout: caps mds = "allow" 2024-03-06T07:23:09.375 INFO:teuthology.orchestra.run.smithi083.stdout: caps mon = "allow r" 2024-03-06T07:23:09.375 INFO:teuthology.orchestra.run.smithi083.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:23:09.375 INFO:teuthology.orchestra.run.smithi083.stderr:updated caps for client.0 2024-03-06T07:23:09.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.374+0000 7f9db38b5640 1 -- 172.21.15.83:0/440971394 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9d9006b4d0 msgr2=0x7f9d9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.374+0000 7f9db38b5640 1 --2- 172.21.15.83:0/440971394 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9d9006b4d0 0x7f9d9006d990 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c036000 tx=0x7f9d9c030000 comp rx=0 tx=0).stop 2024-03-06T07:23:09.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.374+0000 7f9db38b5640 1 -- 172.21.15.83:0/440971394 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9dac1f4380 msgr2=0x7f9dac0d0100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.374+0000 7f9db38b5640 1 --2- 172.21.15.83:0/440971394 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9dac1f4380 0x7f9dac0d0100 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f9dac071110 tx=0x7f9da40115a0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.374+0000 7f9db38b5640 1 -- 172.21.15.83:0/440971394 shutdown_connections 2024-03-06T07:23:09.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.374+0000 7f9db38b5640 1 --2- 172.21.15.83:0/440971394 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9d9006b4d0 0x7f9d9006d990 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.374+0000 7f9db38b5640 1 --2- 172.21.15.83:0/440971394 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9dac1f4380 0x7f9dac0d0100 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.374+0000 7f9db38b5640 1 --2- 172.21.15.83:0/440971394 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9dac1f3f00 0x7f9dac1f7500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.374+0000 7f9db38b5640 1 --2- 172.21.15.83:0/440971394 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9d980039b0 0x7f9dac1565b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.375+0000 7f9db38b5640 1 -- 172.21.15.83:0/440971394 >> 172.21.15.83:0/440971394 conn(0x7f9dac0b1d70 msgr2=0x7f9dac0b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:09.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.375+0000 7f9db38b5640 1 -- 172.21.15.83:0/440971394 shutdown_connections 2024-03-06T07:23:09.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.375+0000 7f9db38b5640 1 -- 172.21.15.83:0/440971394 wait complete. 2024-03-06T07:23:09.387 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:09.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.486+0000 7f724077a640 1 Processor -- start 2024-03-06T07:23:09.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.486+0000 7f724077a640 1 -- start start 2024-03-06T07:23:09.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.487+0000 7f724077a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f72380d22b0 0x7f72380d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.487+0000 7f724077a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f72380b41f0 0x7f72380b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.487+0000 7f724077a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72380b3430 con 0x7f723815dfe0 2024-03-06T07:23:09.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.487+0000 7f724077a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72380d2c50 con 0x7f72380d22b0 2024-03-06T07:23:09.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.487+0000 7f724077a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72380b31e0 con 0x7f72380b41f0 2024-03-06T07:23:09.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.487+0000 7f723dd15640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f72380d22b0 0x7f72380d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.487+0000 7f723dd15640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f72380d22b0 0x7f72380d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55664/0 (socket says 172.21.15.83:55664) 2024-03-06T07:23:09.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.487+0000 7f723dd15640 1 -- 172.21.15.83:0/3645808679 learned_addr learned my addr 172.21.15.83:0/3645808679 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:09.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.487+0000 7f723ed17640 1 --2- 172.21.15.83:0/3645808679 >> v2:172.21.15.183:3300/0 conn(0x7f72380b41f0 0x7f72380b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.487+0000 7f723dd15640 1 -- 172.21.15.83:0/3645808679 >> v2:172.21.15.183:3300/0 conn(0x7f72380b41f0 msgr2=0x7f72380b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:09.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.487+0000 7f723dd15640 1 --2- 172.21.15.83:0/3645808679 >> v2:172.21.15.183:3300/0 conn(0x7f72380b41f0 0x7f72380b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.487+0000 7f723dd15640 1 -- 172.21.15.83:0/3645808679 >> v1:172.21.15.183:6789/0 conn(0x7f723815dfe0 legacy=0x7f72380cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:09.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.487+0000 7f723dd15640 1 -- 172.21.15.83:0/3645808679 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72380b3710 con 0x7f72380d22b0 2024-03-06T07:23:09.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.487+0000 7f723ed17640 1 --2- 172.21.15.83:0/3645808679 >> v2:172.21.15.183:3300/0 conn(0x7f72380b41f0 0x7f72380b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:09.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.488+0000 7f723dd15640 1 --2- 172.21.15.83:0/3645808679 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72380d22b0 0x7f72380d2710 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f7228009870 tx=0x7f722802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c675464760f2785 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:09.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.488+0000 7f723d514640 1 -- 172.21.15.83:0/3645808679 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7228035020 con 0x7f72380d22b0 2024-03-06T07:23:09.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.488+0000 7f723d514640 1 -- 172.21.15.83:0/3645808679 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7228031870 con 0x7f72380d22b0 2024-03-06T07:23:09.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.488+0000 7f723d514640 1 -- 172.21.15.83:0/3645808679 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7228031c40 con 0x7f72380d22b0 2024-03-06T07:23:09.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.489+0000 7f724077a640 1 -- 172.21.15.83:0/3645808679 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72380d22b0 msgr2=0x7f72380d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.489+0000 7f724077a640 1 --2- 172.21.15.83:0/3645808679 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72380d22b0 0x7f72380d2710 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f7228009870 tx=0x7f722802b910 comp rx=0 tx=0).stop 2024-03-06T07:23:09.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.489+0000 7f724077a640 1 -- 172.21.15.83:0/3645808679 shutdown_connections 2024-03-06T07:23:09.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.489+0000 7f724077a640 1 --2- 172.21.15.83:0/3645808679 >> v2:172.21.15.183:3300/0 conn(0x7f72380b41f0 0x7f72380b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.489+0000 7f724077a640 1 --2- 172.21.15.83:0/3645808679 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72380d22b0 0x7f72380d2710 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.489+0000 7f724077a640 1 -- 172.21.15.83:0/3645808679 >> 172.21.15.83:0/3645808679 conn(0x7f72380d3cd0 msgr2=0x7f72380d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:09.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.489+0000 7f724077a640 1 -- 172.21.15.83:0/3645808679 shutdown_connections 2024-03-06T07:23:09.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.489+0000 7f724077a640 1 -- 172.21.15.83:0/3645808679 wait complete. 2024-03-06T07:23:09.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.490+0000 7f724077a640 1 Processor -- start 2024-03-06T07:23:09.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.490+0000 7f724077a640 1 -- start start 2024-03-06T07:23:09.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.491+0000 7f724077a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72380b41f0 0x7f72380cd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.491+0000 7f724077a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f72380d22b0 0x7f72380b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.491+0000 7f724077a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f723815dfe0 0x7f72380b6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.491+0000 7f724077a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72380cfa00 con 0x7f72380b41f0 2024-03-06T07:23:09.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.491+0000 7f724077a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72380b3d60 con 0x7f723815dfe0 2024-03-06T07:23:09.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.491+0000 7f724077a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72380d1c70 con 0x7f72380d22b0 2024-03-06T07:23:09.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.491+0000 7f723e516640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72380b41f0 0x7f72380cd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.491+0000 7f723e516640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72380b41f0 0x7f72380cd270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55668/0 (socket says 172.21.15.83:55668) 2024-03-06T07:23:09.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.491+0000 7f723e516640 1 -- 172.21.15.83:0/1100372551 learned_addr learned my addr 172.21.15.83:0/1100372551 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:09.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.491+0000 7f723dd15640 1 --2- 172.21.15.83:0/1100372551 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f72380d22b0 0x7f72380b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.491+0000 7f723ed17640 1 --2- 172.21.15.83:0/1100372551 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f723815dfe0 0x7f72380b6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.491+0000 7f723e516640 1 -- 172.21.15.83:0/1100372551 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f72380d22b0 msgr2=0x7f72380b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.491+0000 7f723e516640 1 --2- 172.21.15.83:0/1100372551 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f72380d22b0 0x7f72380b68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.491+0000 7f723e516640 1 -- 172.21.15.83:0/1100372551 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f723815dfe0 msgr2=0x7f72380b6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.491+0000 7f723e516640 1 --2- 172.21.15.83:0/1100372551 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f723815dfe0 0x7f72380b6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.491+0000 7f723e516640 1 -- 172.21.15.83:0/1100372551 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72380cd7b0 con 0x7f72380b41f0 2024-03-06T07:23:09.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.491+0000 7f723ed17640 1 --2- 172.21.15.83:0/1100372551 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f723815dfe0 0x7f72380b6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:09.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.491+0000 7f723e516640 1 --2- 172.21.15.83:0/1100372551 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72380b41f0 0x7f72380cd270 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f723400b3f0 tx=0x7f723400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:09.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.492+0000 7f722effd640 1 -- 172.21.15.83:0/1100372551 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7234015020 con 0x7f72380b41f0 2024-03-06T07:23:09.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.492+0000 7f722effd640 1 -- 172.21.15.83:0/1100372551 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7234011830 con 0x7f72380b41f0 2024-03-06T07:23:09.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.492+0000 7f724077a640 1 -- 172.21.15.83:0/1100372551 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72380cc1d0 con 0x7f72380b41f0 2024-03-06T07:23:09.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.492+0000 7f722effd640 1 -- 172.21.15.83:0/1100372551 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7234011c20 con 0x7f72380b41f0 2024-03-06T07:23:09.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.492+0000 7f724077a640 1 -- 172.21.15.83:0/1100372551 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72380cc790 con 0x7f72380b41f0 2024-03-06T07:23:09.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.493+0000 7f724077a640 1 -- 172.21.15.83:0/1100372551 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f723804f330 con 0x7f72380b41f0 2024-03-06T07:23:09.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.493+0000 7f722effd640 1 -- 172.21.15.83:0/1100372551 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7234018020 con 0x7f72380b41f0 2024-03-06T07:23:09.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.493+0000 7f722effd640 1 --2- 172.21.15.83:0/1100372551 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f721c06b4d0 0x7f721c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.493+0000 7f722effd640 1 -- 172.21.15.83:0/1100372551 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7234089d90 con 0x7f72380b41f0 2024-03-06T07:23:09.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.493+0000 7f723dd15640 1 --2- 172.21.15.83:0/1100372551 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f721c06b4d0 0x7f721c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.494+0000 7f723dd15640 1 --2- 172.21.15.83:0/1100372551 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f721c06b4d0 0x7f721c06d990 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7228036000 tx=0x7f722803c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:09.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.502+0000 7f722effd640 1 -- 172.21.15.83:0/1100372551 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f723401f030 con 0x7f72380b41f0 2024-03-06T07:23:09.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.717+0000 7f724077a640 1 -- 172.21.15.83:0/1100372551 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72380b0af0 con 0x7f72380b41f0 2024-03-06T07:23:09.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.717+0000 7f722effd640 1 -- 172.21.15.83:0/1100372551 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f72340172c0 con 0x7f72380b41f0 2024-03-06T07:23:09.720 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:23:09.720 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4799,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/3542419057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3542419057},{"type":"v1","addr":"172.21.15.83:6836","nonce":3542419057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":5023,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1853869906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":1853869906},{"type":"v1","addr":"172.21.15.183:6835","nonce":1853869906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5026,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/3666667479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3666667479},{"type":"v1","addr":"172.21.15.183:6834","nonce":3666667479}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:00.145321+0000","modified":"2024-03-06T07:23:02.319419+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5244":{"gid":5244,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/2668925454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":2668925454},{"type":"v1","addr":"172.21.15.83:6837","nonce":2668925454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5244,"qdb_cluster":[5244]},"id":2}]} 2024-03-06T07:23:09.720 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 13 2024-03-06T07:23:09.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.719+0000 7f724077a640 1 -- 172.21.15.83:0/1100372551 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f721c06b4d0 msgr2=0x7f721c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.719+0000 7f724077a640 1 --2- 172.21.15.83:0/1100372551 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f721c06b4d0 0x7f721c06d990 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7228036000 tx=0x7f722803c040 comp rx=0 tx=0).stop 2024-03-06T07:23:09.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.719+0000 7f724077a640 1 -- 172.21.15.83:0/1100372551 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72380b41f0 msgr2=0x7f72380cd270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.719+0000 7f724077a640 1 --2- 172.21.15.83:0/1100372551 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72380b41f0 0x7f72380cd270 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f723400b3f0 tx=0x7f723400b8b0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.719+0000 7f724077a640 1 -- 172.21.15.83:0/1100372551 shutdown_connections 2024-03-06T07:23:09.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.719+0000 7f724077a640 1 --2- 172.21.15.83:0/1100372551 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f721c06b4d0 0x7f721c06d990 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.719+0000 7f724077a640 1 --2- 172.21.15.83:0/1100372551 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f723815dfe0 0x7f72380b6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.719+0000 7f724077a640 1 --2- 172.21.15.83:0/1100372551 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f72380d22b0 0x7f72380b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.719+0000 7f724077a640 1 --2- 172.21.15.83:0/1100372551 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72380b41f0 0x7f72380cd270 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.719+0000 7f724077a640 1 -- 172.21.15.83:0/1100372551 >> 172.21.15.83:0/1100372551 conn(0x7f72380d3cd0 msgr2=0x7f72380cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:09.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.719+0000 7f724077a640 1 -- 172.21.15.83:0/1100372551 shutdown_connections 2024-03-06T07:23:09.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:09.719+0000 7f724077a640 1 -- 172.21.15.83:0/1100372551 wait complete. 2024-03-06T07:23:10.135 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 13, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:23:00.145321+0000', 'modified': '2024-03-06T07:23:02.319419+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5244}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5244': {'gid': 5244, 'name': 'a', 'rank': 0, 'incarnation': 11, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.83:6837/2668925454', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 2668925454}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 2668925454}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5244, 'qdb_cluster': [5244]} 2024-03-06T07:23:10.135 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:23:10.135 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:23:10.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.241+0000 7f5f4b884640 1 Processor -- start 2024-03-06T07:23:10.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.241+0000 7f5f4b884640 1 -- start start 2024-03-06T07:23:10.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.242+0000 7f5f4b884640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5f440b4fe0 0x7f5f440b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.242+0000 7f5f4b884640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f4415ba50 0x7f5f4415be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.242+0000 7f5f4b884640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f4415d510 con 0x7f5f440b4c00 2024-03-06T07:23:10.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.242+0000 7f5f4b884640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f4415e0b0 con 0x7f5f440b4fe0 2024-03-06T07:23:10.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.242+0000 7f5f4b884640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4415d780 con 0x7f5f4415ba50 2024-03-06T07:23:10.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.242+0000 7f5f48e1f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5f440b4fe0 0x7f5f440b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.242+0000 7f5f48e1f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5f440b4fe0 0x7f5f440b2fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55682/0 (socket says 172.21.15.83:55682) 2024-03-06T07:23:10.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.242+0000 7f5f48e1f640 1 -- 172.21.15.83:0/2548125108 learned_addr learned my addr 172.21.15.83:0/2548125108 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:10.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.242+0000 7f5f49e21640 1 --2- 172.21.15.83:0/2548125108 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f4415ba50 0x7f5f4415be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.243+0000 7f5f48e1f640 1 -- 172.21.15.83:0/2548125108 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f4415ba50 msgr2=0x7f5f4415be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:10.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.243+0000 7f5f48e1f640 1 --2- 172.21.15.83:0/2548125108 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f4415ba50 0x7f5f4415be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.243+0000 7f5f48e1f640 1 -- 172.21.15.83:0/2548125108 >> v1:172.21.15.183:6789/0 conn(0x7f5f440b4c00 legacy=0x7f5f440b28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:10.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.243+0000 7f5f48e1f640 1 -- 172.21.15.83:0/2548125108 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f4415da60 con 0x7f5f440b4fe0 2024-03-06T07:23:10.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.243+0000 7f5f49e21640 1 --2- 172.21.15.83:0/2548125108 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f4415ba50 0x7f5f4415be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:10.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.244+0000 7f5f48e1f640 1 --2- 172.21.15.83:0/2548125108 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f440b4fe0 0x7f5f440b2fb0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f5f38009870 tx=0x7f5f3802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5805f599fe682d67 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:10.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.244+0000 7f5f37fff640 1 -- 172.21.15.83:0/2548125108 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f38034070 con 0x7f5f440b4fe0 2024-03-06T07:23:10.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.244+0000 7f5f37fff640 1 -- 172.21.15.83:0/2548125108 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f38030840 con 0x7f5f440b4fe0 2024-03-06T07:23:10.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.244+0000 7f5f37fff640 1 -- 172.21.15.83:0/2548125108 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f38030c10 con 0x7f5f440b4fe0 2024-03-06T07:23:10.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.244+0000 7f5f4b884640 1 -- 172.21.15.83:0/2548125108 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f440b4fe0 msgr2=0x7f5f440b2fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.244+0000 7f5f4b884640 1 --2- 172.21.15.83:0/2548125108 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f440b4fe0 0x7f5f440b2fb0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f5f38009870 tx=0x7f5f3802b980 comp rx=0 tx=0).stop 2024-03-06T07:23:10.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.245+0000 7f5f4b884640 1 -- 172.21.15.83:0/2548125108 shutdown_connections 2024-03-06T07:23:10.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.245+0000 7f5f4b884640 1 --2- 172.21.15.83:0/2548125108 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f4415ba50 0x7f5f4415be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.245+0000 7f5f4b884640 1 --2- 172.21.15.83:0/2548125108 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f440b4fe0 0x7f5f440b2fb0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.245+0000 7f5f4b884640 1 -- 172.21.15.83:0/2548125108 >> 172.21.15.83:0/2548125108 conn(0x7f5f4409a9f0 msgr2=0x7f5f4409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:10.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.245+0000 7f5f4b884640 1 -- 172.21.15.83:0/2548125108 shutdown_connections 2024-03-06T07:23:10.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.245+0000 7f5f4b884640 1 -- 172.21.15.83:0/2548125108 wait complete. 2024-03-06T07:23:10.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.246+0000 7f5f4b884640 1 Processor -- start 2024-03-06T07:23:10.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.246+0000 7f5f4b884640 1 -- start start 2024-03-06T07:23:10.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.246+0000 7f5f4b884640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f440b4c00 0x7f5f4414f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.246+0000 7f5f4b884640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f440b4fe0 0x7f5f440b0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.246+0000 7f5f4b884640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f4415ba50 0x7f5f4414e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.246+0000 7f5f4b884640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4415f6d0 con 0x7f5f440b4fe0 2024-03-06T07:23:10.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.246+0000 7f5f4b884640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4415f550 con 0x7f5f440b4c00 2024-03-06T07:23:10.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.246+0000 7f5f4b884640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4415c3d0 con 0x7f5f4415ba50 2024-03-06T07:23:10.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.247+0000 7f5f48e1f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f440b4fe0 0x7f5f440b0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.247+0000 7f5f48e1f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f440b4fe0 0x7f5f440b0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55684/0 (socket says 172.21.15.83:55684) 2024-03-06T07:23:10.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.247+0000 7f5f48e1f640 1 -- 172.21.15.83:0/1385383222 learned_addr learned my addr 172.21.15.83:0/1385383222 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:10.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.247+0000 7f5f49e21640 1 --2- 172.21.15.83:0/1385383222 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f4415ba50 0x7f5f4414e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.247+0000 7f5f49620640 1 --2- 172.21.15.83:0/1385383222 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f440b4c00 0x7f5f4414f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.247+0000 7f5f48e1f640 1 -- 172.21.15.83:0/1385383222 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f4415ba50 msgr2=0x7f5f4414e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.247+0000 7f5f48e1f640 1 --2- 172.21.15.83:0/1385383222 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f4415ba50 0x7f5f4414e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.247+0000 7f5f48e1f640 1 -- 172.21.15.83:0/1385383222 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f440b4c00 msgr2=0x7f5f4414f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.247+0000 7f5f48e1f640 1 --2- 172.21.15.83:0/1385383222 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f440b4c00 0x7f5f4414f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.247+0000 7f5f48e1f640 1 -- 172.21.15.83:0/1385383222 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f4414eb70 con 0x7f5f440b4fe0 2024-03-06T07:23:10.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.247+0000 7f5f49e21640 1 --2- 172.21.15.83:0/1385383222 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f4415ba50 0x7f5f4414e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:10.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.247+0000 7f5f49620640 1 --2- 172.21.15.83:0/1385383222 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f440b4c00 0x7f5f4414f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:10.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.247+0000 7f5f48e1f640 1 --2- 172.21.15.83:0/1385383222 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f440b4fe0 0x7f5f440b0190 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f5f38000c00 tx=0x7f5f38004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:10.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.247+0000 7f5f35ffb640 1 -- 172.21.15.83:0/1385383222 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f38048070 con 0x7f5f440b4fe0 2024-03-06T07:23:10.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.247+0000 7f5f35ffb640 1 -- 172.21.15.83:0/1385383222 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f38030640 con 0x7f5f440b4fe0 2024-03-06T07:23:10.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.247+0000 7f5f4b884640 1 -- 172.21.15.83:0/1385383222 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f4414fbd0 con 0x7f5f440b4fe0 2024-03-06T07:23:10.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.247+0000 7f5f35ffb640 1 -- 172.21.15.83:0/1385383222 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f38034040 con 0x7f5f440b4fe0 2024-03-06T07:23:10.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.247+0000 7f5f4b884640 1 -- 172.21.15.83:0/1385383222 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f441504c0 con 0x7f5f440b4fe0 2024-03-06T07:23:10.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.249+0000 7f5f35ffb640 1 -- 172.21.15.83:0/1385383222 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5f3804b070 con 0x7f5f440b4fe0 2024-03-06T07:23:10.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.249+0000 7f5f35ffb640 1 --2- 172.21.15.83:0/1385383222 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5f2806b500 0x7f5f2806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.249+0000 7f5f35ffb640 1 -- 172.21.15.83:0/1385383222 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5f380ac260 con 0x7f5f440b4fe0 2024-03-06T07:23:10.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.249+0000 7f5f49620640 1 --2- 172.21.15.83:0/1385383222 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5f2806b500 0x7f5f2806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.250+0000 7f5f49620640 1 --2- 172.21.15.83:0/1385383222 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5f2806b500 0x7f5f2806d9c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5f40003eb0 tx=0x7f5f40009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:10.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.250+0000 7f5f4e46c740 1 Processor -- start 2024-03-06T07:23:10.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.250+0000 7f5f4e46c740 1 -- start start 2024-03-06T07:23:10.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.251+0000 7f5f4e46c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55869dff5870 0x55869dff5c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.251+0000 7f5f4e46c740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55869dff98c0 0x55869dffb360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.251+0000 7f5f4e46c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55869e025600 0x55869e025a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.251+0000 7f5f4e46c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x55869df5f6b0 con 0x55869e025600 2024-03-06T07:23:10.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.251+0000 7f5f4e46c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55869df08a60 con 0x55869dff5870 2024-03-06T07:23:10.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.251+0000 7f5f4e46c740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x55869df5b750 con 0x55869dff98c0 2024-03-06T07:23:10.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.251+0000 7f5f49e21640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55869e025600 0x55869e025a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.251+0000 7f5f49e21640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55869e025600 0x55869e025a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55698/0 (socket says 172.21.15.83:55698) 2024-03-06T07:23:10.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.251+0000 7f5f49e21640 1 -- 172.21.15.83:0/1065069115 learned_addr learned my addr 172.21.15.83:0/1065069115 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:10.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.251+0000 7f5f49620640 1 --2- 172.21.15.83:0/1065069115 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55869dff5870 0x55869dff5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.251+0000 7f5f49e21640 1 -- 172.21.15.83:0/1065069115 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55869dff98c0 msgr2=0x55869dffb360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:10.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.251+0000 7f5f48e1f640 1 --2- 172.21.15.83:0/1065069115 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55869dff98c0 0x55869dffb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.251+0000 7f5f49e21640 1 --2- 172.21.15.83:0/1065069115 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55869dff98c0 0x55869dffb360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.251+0000 7f5f49e21640 1 -- 172.21.15.83:0/1065069115 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55869dff5870 msgr2=0x55869dff5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:10.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.251+0000 7f5f49e21640 1 --2- 172.21.15.83:0/1065069115 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55869dff5870 0x55869dff5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.251+0000 7f5f49e21640 1 -- 172.21.15.83:0/1065069115 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55869df26f30 con 0x55869e025600 2024-03-06T07:23:10.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.251+0000 7f5f49620640 1 --2- 172.21.15.83:0/1065069115 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55869dff5870 0x55869dff5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:10.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.251+0000 7f5f48e1f640 1 --2- 172.21.15.83:0/1065069115 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55869dff98c0 0x55869dffb360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:10.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.252+0000 7f5f49e21640 1 --2- 172.21.15.83:0/1065069115 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55869e025600 0x55869e025a80 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c00d370 tx=0x7f5f3c00d830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c61b79fd536a513 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:10.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.252+0000 7f5f2effd640 1 -- 172.21.15.83:0/1065069115 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f3c00f920 con 0x55869e025600 2024-03-06T07:23:10.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.252+0000 7f5f2effd640 1 -- 172.21.15.83:0/1065069115 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f3c00fac0 con 0x55869e025600 2024-03-06T07:23:10.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.252+0000 7f5f2effd640 1 -- 172.21.15.83:0/1065069115 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f3c018480 con 0x55869e025600 2024-03-06T07:23:10.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.252+0000 7f5f4e46c740 1 -- 172.21.15.83:0/1065069115 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55869e025600 msgr2=0x55869e025a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.252+0000 7f5f4e46c740 1 --2- 172.21.15.83:0/1065069115 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55869e025600 0x55869e025a80 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c00d370 tx=0x7f5f3c00d830 comp rx=0 tx=0).stop 2024-03-06T07:23:10.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.252+0000 7f5f4e46c740 1 -- 172.21.15.83:0/1065069115 shutdown_connections 2024-03-06T07:23:10.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.252+0000 7f5f4e46c740 1 --2- 172.21.15.83:0/1065069115 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55869e025600 0x55869e025a80 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.252+0000 7f5f4e46c740 1 --2- 172.21.15.83:0/1065069115 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55869dff98c0 0x55869dffb360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.252+0000 7f5f4e46c740 1 --2- 172.21.15.83:0/1065069115 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55869dff5870 0x55869dff5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.252+0000 7f5f4e46c740 1 -- 172.21.15.83:0/1065069115 >> 172.21.15.83:0/1065069115 conn(0x55869dff78f0 msgr2=0x55869dff91b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:10.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.252+0000 7f5f4e46c740 1 -- 172.21.15.83:0/1065069115 shutdown_connections 2024-03-06T07:23:10.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.252+0000 7f5f4e46c740 1 -- 172.21.15.83:0/1065069115 wait complete. 2024-03-06T07:23:10.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.253+0000 7f5f4e46c740 1 -- start start 2024-03-06T07:23:10.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.253+0000 7f5f4e46c740 1 Processor -- start 2024-03-06T07:23:10.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.254+0000 7f5f4e46c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55869e025600 0x55869e01f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.254+0000 7f5f4e46c740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55869e020520 0x55869e03ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.254+0000 7f5f4e46c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55869e01ffd0 0x55869e027bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.254+0000 7f5f4e46c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x55869dff9d50 con 0x55869e01ffd0 2024-03-06T07:23:10.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.254+0000 7f5f4e46c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55869df1bea0 con 0x55869e025600 2024-03-06T07:23:10.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.254+0000 7f5f4e46c740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x55869dffa030 con 0x55869e020520 2024-03-06T07:23:10.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.254+0000 7f5f4e46c740 10 client.0 _ll_register_callbacks cb 0x55869df1b870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:23:10.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.254+0000 7f5f49e21640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55869e01ffd0 0x55869e027bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.254+0000 7f5f49e21640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55869e01ffd0 0x55869e027bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55702/0 (socket says 172.21.15.83:55702) 2024-03-06T07:23:10.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.254+0000 7f5f49e21640 1 -- 172.21.15.83:0/2718546717 learned_addr learned my addr 172.21.15.83:0/2718546717 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:10.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.254+0000 7f5f49620640 1 --2- 172.21.15.83:0/2718546717 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55869e025600 0x55869e01f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.254+0000 7f5f48e1f640 1 --2- 172.21.15.83:0/2718546717 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55869e020520 0x55869e03ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.254+0000 7f5f49e21640 1 -- 172.21.15.83:0/2718546717 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55869e020520 msgr2=0x55869e03ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:10.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.254+0000 7f5f49e21640 1 --2- 172.21.15.83:0/2718546717 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55869e020520 0x55869e03ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.254+0000 7f5f49e21640 1 -- 172.21.15.83:0/2718546717 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55869e025600 msgr2=0x55869e01f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:10.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.254+0000 7f5f49e21640 1 --2- 172.21.15.83:0/2718546717 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55869e025600 0x55869e01f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.254+0000 7f5f49e21640 1 -- 172.21.15.83:0/2718546717 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55869df1c270 con 0x55869e01ffd0 2024-03-06T07:23:10.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.255+0000 7f5f49e21640 1 --2- 172.21.15.83:0/2718546717 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55869e01ffd0 0x55869e027bb0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c00dd30 tx=0x7f5f3c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9634505eb0423e14 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:10.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.255+0000 7f5f23fff640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-06T07:23:10.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.255+0000 7f5f23fff640 1 -- 172.21.15.83:0/2718546717 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f3c018890 con 0x55869e01ffd0 2024-03-06T07:23:10.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.255+0000 7f5f23fff640 1 -- 172.21.15.83:0/2718546717 <== mon.0 v2:172.21.15.83:3300/0 2 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5f3c018a30 con 0x55869e01ffd0 2024-03-06T07:23:10.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.255+0000 7f5f4e46c740 1 -- 172.21.15.83:0/2718546717 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5f3c00fac0 con 0x55869e01ffd0 2024-03-06T07:23:10.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.255+0000 7f5f23fff640 1 -- 172.21.15.83:0/2718546717 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=13) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5f3c030050 con 0x55869e01ffd0 2024-03-06T07:23:10.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.255+0000 7f5f4e46c740 10 client.5445 fetch_fsmap learned FSMap version 13 2024-03-06T07:23:10.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.255+0000 7f5f4e46c740 1 -- 172.21.15.83:0/2718546717 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=13,monmap=2+}) v3 -- 0x55869dc742d0 con 0x55869e01ffd0 2024-03-06T07:23:10.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.255+0000 7f5f23fff640 1 -- 172.21.15.83:0/2718546717 <== mon.0 v2:172.21.15.83:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f3c00fac0 con 0x55869e01ffd0 2024-03-06T07:23:10.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.255+0000 7f5f23fff640 1 -- 172.21.15.83:0/2718546717 <== mon.0 v2:172.21.15.83:3300/0 5 ==== fsmap(e 13) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f5f3c033020 con 0x55869e01ffd0 2024-03-06T07:23:10.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.255+0000 7f5f4e46c740 10 client.5445 fetch_fsmap finished waiting for FSMap version 13 2024-03-06T07:23:10.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.255+0000 7f5f4e46c740 10 client.5445 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:23:10.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.255+0000 7f5f4e46c740 20 client.5445 populate_metadata read hostname 'smithi083' 2024-03-06T07:23:10.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.255+0000 7f5f4e46c740 1 --2- 172.21.15.83:0/2718546717 >> [v2:172.21.15.83:6835/2668925454,v1:172.21.15.83:6837/2668925454] conn(0x55869e027380 0x55869e05f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.255+0000 7f5f4e46c740 4 client.5445 mds_command: new command op to 5244 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:23:10.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.255+0000 7f5f4e46c740 1 -- 172.21.15.83:0/2718546717 --> [v2:172.21.15.83:6835/2668925454,v1:172.21.15.83:6837/2668925454] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55869df6cbf0 con 0x55869e027380 2024-03-06T07:23:10.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.255+0000 7f5f49620640 1 --2- 172.21.15.83:0/2718546717 >> [v2:172.21.15.83:6835/2668925454,v1:172.21.15.83:6837/2668925454] conn(0x55869e027380 0x55869e05f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.256+0000 7f5f23fff640 10 client.5445 ms_handle_connect on v2:172.21.15.83:6835/2668925454 2024-03-06T07:23:10.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.256+0000 7f5f49620640 1 --2- 172.21.15.83:0/2718546717 >> [v2:172.21.15.83:6835/2668925454,v1:172.21.15.83:6837/2668925454] conn(0x55869e027380 0x55869e05f7a0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c7ae9ee213e0fe7 server_cookie=29227a5508918910 in_seq=0 out_seq=0 2024-03-06T07:23:10.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.256+0000 7f5f23fff640 1 -- 172.21.15.83:0/2718546717 <== mds.0 v2:172.21.15.83:6835/2668925454 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55869df6cbf0 con 0x55869e027380 2024-03-06T07:23:10.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.256+0000 7f5f23fff640 10 client.5445 handle_command_reply: tid=0 2024-03-06T07:23:10.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.256+0000 7f5f4e46c740 1 client.5445 shutdown 2024-03-06T07:23:10.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.256+0000 7f5f4e46c740 1 -- 172.21.15.83:0/2718546717 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55869e01ffd0 msgr2=0x55869e027bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.256+0000 7f5f4e46c740 1 --2- 172.21.15.83:0/2718546717 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55869e01ffd0 0x55869e027bb0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c00dd30 tx=0x7f5f3c004ab0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.256+0000 7f5f4e46c740 1 -- 172.21.15.83:0/2718546717 shutdown_connections 2024-03-06T07:23:10.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.256+0000 7f5f4e46c740 1 --2- 172.21.15.83:0/2718546717 >> [v2:172.21.15.83:6835/2668925454,v1:172.21.15.83:6837/2668925454] conn(0x55869e027380 0x55869e05f7a0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.256+0000 7f5f4e46c740 1 --2- 172.21.15.83:0/2718546717 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55869e01ffd0 0x55869e027bb0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.256+0000 7f5f4e46c740 1 --2- 172.21.15.83:0/2718546717 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55869e020520 0x55869e03ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.256+0000 7f5f23fff640 0 client.5445 ms_handle_reset on v2:172.21.15.83:6835/2668925454 2024-03-06T07:23:10.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.256+0000 7f5f4e46c740 1 --2- 172.21.15.83:0/2718546717 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55869e025600 0x55869e01f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.256+0000 7f5f4e46c740 1 -- 172.21.15.83:0/2718546717 >> 172.21.15.83:0/2718546717 conn(0x55869dff5870 msgr2=0x55869e0235c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:10.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.257+0000 7f5f4e46c740 1 -- 172.21.15.83:0/2718546717 shutdown_connections 2024-03-06T07:23:10.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.257+0000 7f5f4e46c740 1 -- 172.21.15.83:0/2718546717 wait complete. 2024-03-06T07:23:10.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.257+0000 7f5f4e46c740 20 client.5445 trim_cache size 0 max 16384 2024-03-06T07:23:10.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.273+0000 7f5f4e46c740 1 Processor -- start 2024-03-06T07:23:10.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.273+0000 7f5f4e46c740 1 -- start start 2024-03-06T07:23:10.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.274+0000 7f5f4e46c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55869e01ffd0 0x55869dff9080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.274+0000 7f5f4e46c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55869e020520 0x55869dff4700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.274+0000 7f5f4e46c740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55869e025600 0x55869e045c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.274+0000 7f5f4e46c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f441ff460 con 0x55869e020520 2024-03-06T07:23:10.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.274+0000 7f5f4e46c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f441f7620 con 0x55869e01ffd0 2024-03-06T07:23:10.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.274+0000 7f5f4e46c740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f441ff0a0 con 0x55869e025600 2024-03-06T07:23:10.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.274+0000 7f5f48e1f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55869e020520 0x55869dff4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.274+0000 7f5f48e1f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55869e020520 0x55869dff4700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55712/0 (socket says 172.21.15.83:55712) 2024-03-06T07:23:10.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.274+0000 7f5f48e1f640 1 -- 172.21.15.83:0/3199373054 learned_addr learned my addr 172.21.15.83:0/3199373054 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:10.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.274+0000 7f5f49e21640 1 --2- 172.21.15.83:0/3199373054 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55869e025600 0x55869e045c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.274+0000 7f5f49620640 1 --2- 172.21.15.83:0/3199373054 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55869e01ffd0 0x55869dff9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.274+0000 7f5f48e1f640 1 -- 172.21.15.83:0/3199373054 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55869e025600 msgr2=0x55869e045c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:10.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.274+0000 7f5f48e1f640 1 --2- 172.21.15.83:0/3199373054 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55869e025600 0x55869e045c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.274+0000 7f5f48e1f640 1 -- 172.21.15.83:0/3199373054 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55869e01ffd0 msgr2=0x55869dff9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:10.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.274+0000 7f5f48e1f640 1 --2- 172.21.15.83:0/3199373054 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55869e01ffd0 0x55869dff9080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.275+0000 7f5f48e1f640 1 -- 172.21.15.83:0/3199373054 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55869dd3e6b0 con 0x55869e020520 2024-03-06T07:23:10.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.275+0000 7f5f48e1f640 1 --2- 172.21.15.83:0/3199373054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55869e020520 0x55869dff4700 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f5f38076ed0 tx=0x7f5f380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcd39eac5f61f8d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:10.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.275+0000 7f5f2e7fc640 1 -- 172.21.15.83:0/3199373054 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f38083180 con 0x55869e020520 2024-03-06T07:23:10.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.275+0000 7f5f2e7fc640 1 -- 172.21.15.83:0/3199373054 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f380a5070 con 0x55869e020520 2024-03-06T07:23:10.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.275+0000 7f5f2e7fc640 1 -- 172.21.15.83:0/3199373054 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f380091c0 con 0x55869e020520 2024-03-06T07:23:10.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.275+0000 7f5f4e46c740 1 -- 172.21.15.83:0/3199373054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55869e020520 msgr2=0x55869dff4700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.276+0000 7f5f4e46c740 1 --2- 172.21.15.83:0/3199373054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55869e020520 0x55869dff4700 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f5f38076ed0 tx=0x7f5f380046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.276+0000 7f5f4e46c740 1 -- 172.21.15.83:0/3199373054 shutdown_connections 2024-03-06T07:23:10.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.276+0000 7f5f4e46c740 1 --2- 172.21.15.83:0/3199373054 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55869e025600 0x55869e045c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.276+0000 7f5f4e46c740 1 --2- 172.21.15.83:0/3199373054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55869e020520 0x55869dff4700 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.276+0000 7f5f4e46c740 1 --2- 172.21.15.83:0/3199373054 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55869e01ffd0 0x55869dff9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.276+0000 7f5f4e46c740 1 -- 172.21.15.83:0/3199373054 >> 172.21.15.83:0/3199373054 conn(0x55869dff5870 msgr2=0x55869e01f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:10.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.276+0000 7f5f4e46c740 1 -- 172.21.15.83:0/3199373054 shutdown_connections 2024-03-06T07:23:10.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.276+0000 7f5f4e46c740 1 -- 172.21.15.83:0/3199373054 wait complete. 2024-03-06T07:23:10.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.276+0000 7f5f4e46c740 1 -- start start 2024-03-06T07:23:10.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.276+0000 7f5f4e46c740 1 Processor -- start 2024-03-06T07:23:10.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.277+0000 7f5f4e46c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55869e01ffd0 0x55869e0afba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.277+0000 7f5f4e46c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55869e020520 0x55869e047470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.277+0000 7f5f4e46c740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55869e025600 0x55869e0479b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.277+0000 7f5f4e46c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x55869e03efc0 con 0x55869e01ffd0 2024-03-06T07:23:10.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.277+0000 7f5f4e46c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55869e044cd0 con 0x55869e020520 2024-03-06T07:23:10.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.277+0000 7f5f4e46c740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x55869e047000 con 0x55869e025600 2024-03-06T07:23:10.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.277+0000 7f5f4e46c740 10 client.0 _ll_register_callbacks cb 0x7f5f3c007ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:23:10.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.277+0000 7f5f49620640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55869e01ffd0 0x55869e0afba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.277+0000 7f5f49620640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55869e01ffd0 0x55869e0afba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55720/0 (socket says 172.21.15.83:55720) 2024-03-06T07:23:10.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.277+0000 7f5f49620640 1 -- 172.21.15.83:0/1527511415 learned_addr learned my addr 172.21.15.83:0/1527511415 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:10.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.277+0000 7f5f48e1f640 1 --2- 172.21.15.83:0/1527511415 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55869e020520 0x55869e047470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.277+0000 7f5f49e21640 1 --2- 172.21.15.83:0/1527511415 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55869e025600 0x55869e0479b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.277+0000 7f5f49620640 1 -- 172.21.15.83:0/1527511415 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55869e025600 msgr2=0x55869e0479b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:10.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.277+0000 7f5f49620640 1 --2- 172.21.15.83:0/1527511415 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55869e025600 0x55869e0479b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.277+0000 7f5f49620640 1 -- 172.21.15.83:0/1527511415 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55869e020520 msgr2=0x55869e047470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:10.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.277+0000 7f5f49620640 1 --2- 172.21.15.83:0/1527511415 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55869e020520 0x55869e047470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.277+0000 7f5f49620640 1 -- 172.21.15.83:0/1527511415 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55869e024360 con 0x55869e01ffd0 2024-03-06T07:23:10.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.278+0000 7f5f49e21640 1 --2- 172.21.15.83:0/1527511415 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55869e025600 0x55869e0479b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:10.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.278+0000 7f5f49620640 1 --2- 172.21.15.83:0/1527511415 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55869e01ffd0 0x55869e0afba0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f5f4000d520 tx=0x7f5f400036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c92067df38888241 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:10.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.278+0000 7f5f23fff640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-06T07:23:10.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.278+0000 7f5f23fff640 1 -- 172.21.15.83:0/1527511415 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f40030640 con 0x55869e01ffd0 2024-03-06T07:23:10.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.278+0000 7f5f23fff640 1 -- 172.21.15.83:0/1527511415 <== mon.0 v2:172.21.15.83:3300/0 2 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5f400307e0 con 0x55869e01ffd0 2024-03-06T07:23:10.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.278+0000 7f5f4e46c740 1 -- 172.21.15.83:0/1527511415 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5f380a5070 con 0x55869e01ffd0 2024-03-06T07:23:10.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.278+0000 7f5f23fff640 1 -- 172.21.15.83:0/1527511415 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=13) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5f40038050 con 0x55869e01ffd0 2024-03-06T07:23:10.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.278+0000 7f5f4e46c740 10 client.5451 fetch_fsmap learned FSMap version 13 2024-03-06T07:23:10.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.278+0000 7f5f4e46c740 1 -- 172.21.15.83:0/1527511415 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=13,monmap=2+}) v3 -- 0x55869e024100 con 0x55869e01ffd0 2024-03-06T07:23:10.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.278+0000 7f5f23fff640 1 -- 172.21.15.83:0/1527511415 <== mon.0 v2:172.21.15.83:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f380a5070 con 0x55869e01ffd0 2024-03-06T07:23:10.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.278+0000 7f5f23fff640 1 -- 172.21.15.83:0/1527511415 <== mon.0 v2:172.21.15.83:3300/0 5 ==== fsmap(e 13) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f5f4002e020 con 0x55869e01ffd0 2024-03-06T07:23:10.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.278+0000 7f5f4e46c740 10 client.5451 fetch_fsmap finished waiting for FSMap version 13 2024-03-06T07:23:10.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.278+0000 7f5f4e46c740 10 client.5451 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:23:10.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.278+0000 7f5f4e46c740 20 client.5451 populate_metadata read hostname 'smithi083' 2024-03-06T07:23:10.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.278+0000 7f5f4e46c740 1 --2- 172.21.15.83:0/1527511415 >> [v2:172.21.15.83:6835/2668925454,v1:172.21.15.83:6837/2668925454] conn(0x55869e023bb0 0x55869e01c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.278+0000 7f5f4e46c740 4 client.5451 mds_command: new command op to 5244 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:23:10.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.278+0000 7f5f4e46c740 1 -- 172.21.15.83:0/1527511415 --> [v2:172.21.15.83:6835/2668925454,v1:172.21.15.83:6837/2668925454] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55869dff8050 con 0x55869e023bb0 2024-03-06T07:23:10.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.278+0000 7f5f48e1f640 1 --2- 172.21.15.83:0/1527511415 >> [v2:172.21.15.83:6835/2668925454,v1:172.21.15.83:6837/2668925454] conn(0x55869e023bb0 0x55869e01c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.279+0000 7f5f23fff640 10 client.5451 ms_handle_connect on v2:172.21.15.83:6835/2668925454 2024-03-06T07:23:10.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.279+0000 7f5f48e1f640 1 --2- 172.21.15.83:0/1527511415 >> [v2:172.21.15.83:6835/2668925454,v1:172.21.15.83:6837/2668925454] conn(0x55869e023bb0 0x55869e01c8a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61ff63d303e980c2 server_cookie=dafa5d76ea365a81 in_seq=0 out_seq=0 2024-03-06T07:23:10.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.279+0000 7f5f23fff640 1 -- 172.21.15.83:0/1527511415 <== mds.0 v2:172.21.15.83:6835/2668925454 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55869dff8050 con 0x55869e023bb0 2024-03-06T07:23:10.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.279+0000 7f5f23fff640 10 client.5451 handle_command_reply: tid=0 2024-03-06T07:23:10.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.279+0000 7f5f4e46c740 1 client.5451 shutdown 2024-03-06T07:23:10.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.279+0000 7f5f4e46c740 1 -- 172.21.15.83:0/1527511415 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55869e01ffd0 msgr2=0x55869e0afba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.279+0000 7f5f4e46c740 1 --2- 172.21.15.83:0/1527511415 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55869e01ffd0 0x55869e0afba0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f5f4000d520 tx=0x7f5f400036f0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.279+0000 7f5f4e46c740 1 -- 172.21.15.83:0/1527511415 shutdown_connections 2024-03-06T07:23:10.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.279+0000 7f5f4e46c740 1 --2- 172.21.15.83:0/1527511415 >> [v2:172.21.15.83:6835/2668925454,v1:172.21.15.83:6837/2668925454] conn(0x55869e023bb0 0x55869e01c8a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.279+0000 7f5f4e46c740 1 --2- 172.21.15.83:0/1527511415 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55869e025600 0x55869e0479b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.279+0000 7f5f23fff640 0 client.5451 ms_handle_reset on v2:172.21.15.83:6835/2668925454 2024-03-06T07:23:10.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.279+0000 7f5f4e46c740 1 --2- 172.21.15.83:0/1527511415 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55869e020520 0x55869e047470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.279+0000 7f5f4e46c740 1 --2- 172.21.15.83:0/1527511415 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55869e01ffd0 0x55869e0afba0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.279+0000 7f5f4e46c740 1 -- 172.21.15.83:0/1527511415 >> 172.21.15.83:0/1527511415 conn(0x55869dff5870 msgr2=0x55869dffbf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:10.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.279+0000 7f5f4e46c740 1 -- 172.21.15.83:0/1527511415 shutdown_connections 2024-03-06T07:23:10.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.279+0000 7f5f4e46c740 1 -- 172.21.15.83:0/1527511415 wait complete. 2024-03-06T07:23:10.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.279+0000 7f5f4e46c740 20 client.5451 trim_cache size 0 max 16384 2024-03-06T07:23:10.287 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:23:10.287 INFO:teuthology.orchestra.run.smithi083.stdout: "cluster_fsid": "039877dd-d2ce-45d2-8e77-ac5eb7e1d55a", 2024-03-06T07:23:10.288 INFO:teuthology.orchestra.run.smithi083.stdout: "whoami": 0, 2024-03-06T07:23:10.288 INFO:teuthology.orchestra.run.smithi083.stdout: "id": 5244, 2024-03-06T07:23:10.288 INFO:teuthology.orchestra.run.smithi083.stdout: "want_state": "up:active", 2024-03-06T07:23:10.288 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "up:active", 2024-03-06T07:23:10.288 INFO:teuthology.orchestra.run.smithi083.stdout: "fs_name": "cephfs", 2024-03-06T07:23:10.288 INFO:teuthology.orchestra.run.smithi083.stdout: "rank_uptime": 9.4668458219999998, 2024-03-06T07:23:10.288 INFO:teuthology.orchestra.run.smithi083.stdout: "mdsmap_epoch": 13, 2024-03-06T07:23:10.288 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_epoch": 33, 2024-03-06T07:23:10.288 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_epoch_barrier": 33, 2024-03-06T07:23:10.288 INFO:teuthology.orchestra.run.smithi083.stdout: "uptime": 16.363649871 2024-03-06T07:23:10.288 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:23:10.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.280+0000 7f5f4b884640 1 -- 172.21.15.83:0/1385383222 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5f2806b500 msgr2=0x7f5f2806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.280+0000 7f5f4b884640 1 --2- 172.21.15.83:0/1385383222 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5f2806b500 0x7f5f2806d9c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5f40003eb0 tx=0x7f5f40009040 comp rx=0 tx=0).stop 2024-03-06T07:23:10.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.280+0000 7f5f4b884640 1 -- 172.21.15.83:0/1385383222 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f440b4fe0 msgr2=0x7f5f440b0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.280+0000 7f5f4b884640 1 --2- 172.21.15.83:0/1385383222 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f440b4fe0 0x7f5f440b0190 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f5f38000c00 tx=0x7f5f38004730 comp rx=0 tx=0).stop 2024-03-06T07:23:10.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.280+0000 7f5f4b884640 1 -- 172.21.15.83:0/1385383222 shutdown_connections 2024-03-06T07:23:10.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.280+0000 7f5f4b884640 1 --2- 172.21.15.83:0/1385383222 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5f2806b500 0x7f5f2806d9c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.280+0000 7f5f4b884640 1 --2- 172.21.15.83:0/1385383222 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f4415ba50 0x7f5f4414e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.280+0000 7f5f4b884640 1 --2- 172.21.15.83:0/1385383222 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f440b4fe0 0x7f5f440b0190 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.280+0000 7f5f4b884640 1 --2- 172.21.15.83:0/1385383222 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f440b4c00 0x7f5f4414f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.280+0000 7f5f4b884640 1 -- 172.21.15.83:0/1385383222 >> 172.21.15.83:0/1385383222 conn(0x7f5f4409a9f0 msgr2=0x7f5f4406fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:10.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.280+0000 7f5f4b884640 1 -- 172.21.15.83:0/1385383222 shutdown_connections 2024-03-06T07:23:10.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.280+0000 7f5f4b884640 1 -- 172.21.15.83:0/1385383222 wait complete. 2024-03-06T07:23:10.292 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:23:10.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.392+0000 7f7d886af640 1 Processor -- start 2024-03-06T07:23:10.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.392+0000 7f7d886af640 1 -- start start 2024-03-06T07:23:10.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.393+0000 7f7d886af640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7d8015bb20 0x7f7d800d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.393+0000 7f7d886af640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f7d800cfff0 0x7f7d800d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.393+0000 7f7d886af640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d80161980 con 0x7f7d800ce4b0 2024-03-06T07:23:10.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.393+0000 7f7d886af640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d80161c60 con 0x7f7d8015bb20 2024-03-06T07:23:10.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.393+0000 7f7d886af640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d80161f40 con 0x7f7d800cfff0 2024-03-06T07:23:10.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.393+0000 7f7d8644b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7d8015bb20 0x7f7d800d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.394+0000 7f7d86c4c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f7d800ce4b0 0x7f7d800ce8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:37368/0 (socket says 172.21.15.83:37368) 2024-03-06T07:23:10.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.394+0000 7f7d8644b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7d8015bb20 0x7f7d800d1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55734/0 (socket says 172.21.15.83:55734) 2024-03-06T07:23:10.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.394+0000 7f7d86c4c640 1 -- 172.21.15.83:0/1996863770 learned_addr learned my addr 172.21.15.83:0/1996863770 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:23:10.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.394+0000 7f7d85c4a640 1 --2- 172.21.15.83:0/1996863770 >> v2:172.21.15.183:3300/0 conn(0x7f7d800cfff0 0x7f7d800d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.394+0000 7f7d8644b640 1 -- 172.21.15.83:0/1996863770 >> v2:172.21.15.183:3300/0 conn(0x7f7d800cfff0 msgr2=0x7f7d800d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:10.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.394+0000 7f7d8644b640 1 --2- 172.21.15.83:0/1996863770 >> v2:172.21.15.183:3300/0 conn(0x7f7d800cfff0 0x7f7d800d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.394+0000 7f7d8644b640 1 -- 172.21.15.83:0/1996863770 >> v1:172.21.15.83:6789/0 conn(0x7f7d800ce4b0 legacy=0x7f7d800ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.394+0000 7f7d8644b640 1 -- 172.21.15.83:0/1996863770 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d80162220 con 0x7f7d8015bb20 2024-03-06T07:23:10.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.395+0000 7f7d8644b640 1 --2- 172.21.15.83:0/1996863770 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d8015bb20 0x7f7d800d1df0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f7d80071290 tx=0x7f7d7c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fc0ff031049fabe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:10.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.395+0000 7f7d85449640 1 -- 172.21.15.83:0/1996863770 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7c035020 con 0x7f7d8015bb20 2024-03-06T07:23:10.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.395+0000 7f7d85449640 1 -- 172.21.15.83:0/1996863770 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d7c031830 con 0x7f7d8015bb20 2024-03-06T07:23:10.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.395+0000 7f7d85449640 1 -- 172.21.15.83:0/1996863770 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7c031c00 con 0x7f7d8015bb20 2024-03-06T07:23:10.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.395+0000 7f7d886af640 1 -- 172.21.15.83:0/1996863770 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d8015bb20 msgr2=0x7f7d800d1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.395+0000 7f7d886af640 1 --2- 172.21.15.83:0/1996863770 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d8015bb20 0x7f7d800d1df0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f7d80071290 tx=0x7f7d7c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.395+0000 7f7d886af640 1 -- 172.21.15.83:0/1996863770 shutdown_connections 2024-03-06T07:23:10.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.395+0000 7f7d886af640 1 --2- 172.21.15.83:0/1996863770 >> v2:172.21.15.183:3300/0 conn(0x7f7d800cfff0 0x7f7d800d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.395+0000 7f7d886af640 1 --2- 172.21.15.83:0/1996863770 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d8015bb20 0x7f7d800d1df0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.395+0000 7f7d886af640 1 -- 172.21.15.83:0/1996863770 >> 172.21.15.83:0/1996863770 conn(0x7f7d8009a9f0 msgr2=0x7f7d8009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:10.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.395+0000 7f7d886af640 1 -- 172.21.15.83:0/1996863770 shutdown_connections 2024-03-06T07:23:10.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.395+0000 7f7d886af640 1 -- 172.21.15.83:0/1996863770 wait complete. 2024-03-06T07:23:10.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.397+0000 7f7d886af640 1 Processor -- start 2024-03-06T07:23:10.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.397+0000 7f7d886af640 1 -- start start 2024-03-06T07:23:10.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.397+0000 7f7d886af640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d800ce4b0 0x7f7d800c7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.397+0000 7f7d886af640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d800cfff0 0x7f7d800b5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.397+0000 7f7d886af640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d8015bb20 0x7f7d800b5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.397+0000 7f7d886af640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d800d24b0 con 0x7f7d800cfff0 2024-03-06T07:23:10.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.397+0000 7f7d886af640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d800d2330 con 0x7f7d800ce4b0 2024-03-06T07:23:10.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.397+0000 7f7d886af640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d800d1980 con 0x7f7d8015bb20 2024-03-06T07:23:10.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.397+0000 7f7d85c4a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d800cfff0 0x7f7d800b5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.397+0000 7f7d85c4a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d800cfff0 0x7f7d800b5ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55748/0 (socket says 172.21.15.83:55748) 2024-03-06T07:23:10.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.397+0000 7f7d85c4a640 1 -- 172.21.15.83:0/1716329244 learned_addr learned my addr 172.21.15.83:0/1716329244 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:10.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.398+0000 7f7d86c4c640 1 --2- 172.21.15.83:0/1716329244 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d8015bb20 0x7f7d800b5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.398+0000 7f7d8644b640 1 --2- 172.21.15.83:0/1716329244 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d800ce4b0 0x7f7d800c7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.398+0000 7f7d85c4a640 1 -- 172.21.15.83:0/1716329244 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d8015bb20 msgr2=0x7f7d800b5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.398+0000 7f7d85c4a640 1 --2- 172.21.15.83:0/1716329244 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d8015bb20 0x7f7d800b5ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.398+0000 7f7d85c4a640 1 -- 172.21.15.83:0/1716329244 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d800ce4b0 msgr2=0x7f7d800c7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.398+0000 7f7d85c4a640 1 --2- 172.21.15.83:0/1716329244 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d800ce4b0 0x7f7d800c7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.398+0000 7f7d85c4a640 1 -- 172.21.15.83:0/1716329244 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d800b4a10 con 0x7f7d800cfff0 2024-03-06T07:23:10.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.398+0000 7f7d85c4a640 1 --2- 172.21.15.83:0/1716329244 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d800cfff0 0x7f7d800b5ab0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f7d7000d800 tx=0x7f7d7000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:10.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.398+0000 7f7d76ffd640 1 -- 172.21.15.83:0/1716329244 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7000b660 con 0x7f7d800cfff0 2024-03-06T07:23:10.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.398+0000 7f7d76ffd640 1 -- 172.21.15.83:0/1716329244 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d70019070 con 0x7f7d800cfff0 2024-03-06T07:23:10.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.398+0000 7f7d886af640 1 -- 172.21.15.83:0/1716329244 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d800b4cf0 con 0x7f7d800cfff0 2024-03-06T07:23:10.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.398+0000 7f7d76ffd640 1 -- 172.21.15.83:0/1716329244 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7000ba20 con 0x7f7d800cfff0 2024-03-06T07:23:10.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.398+0000 7f7d886af640 1 -- 172.21.15.83:0/1716329244 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d800b52b0 con 0x7f7d800cfff0 2024-03-06T07:23:10.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.399+0000 7f7d886af640 1 -- 172.21.15.83:0/1716329244 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d8004f330 con 0x7f7d800cfff0 2024-03-06T07:23:10.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.400+0000 7f7d76ffd640 1 -- 172.21.15.83:0/1716329244 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7d7000bbc0 con 0x7f7d800cfff0 2024-03-06T07:23:10.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.400+0000 7f7d76ffd640 1 --2- 172.21.15.83:0/1716329244 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7d6406b4d0 0x7f7d6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.400+0000 7f7d76ffd640 1 -- 172.21.15.83:0/1716329244 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7d7008a630 con 0x7f7d800cfff0 2024-03-06T07:23:10.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.400+0000 7f7d8644b640 1 --2- 172.21.15.83:0/1716329244 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7d6406b4d0 0x7f7d6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.401+0000 7f7d8644b640 1 --2- 172.21.15.83:0/1716329244 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7d6406b4d0 0x7f7d6406d990 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c036000 tx=0x7f7d7c03c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:10.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.405+0000 7f7d76ffd640 1 -- 172.21.15.83:0/1716329244 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7d700594b0 con 0x7f7d800cfff0 2024-03-06T07:23:10.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.669+0000 7f7d886af640 1 -- 172.21.15.83:0/1716329244 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f7d8006f250 con 0x7f7d800cfff0 2024-03-06T07:23:10.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.669+0000 7f7d76ffd640 1 -- 172.21.15.83:0/1716329244 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v13) v1 ==== 53+0+73 (secure 0 0 0) 0x7f7d7005de20 con 0x7f7d800cfff0 2024-03-06T07:23:10.671 INFO:teuthology.orchestra.run.smithi083.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:23:10.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.671+0000 7f7d886af640 1 -- 172.21.15.83:0/1716329244 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7d6406b4d0 msgr2=0x7f7d6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.671+0000 7f7d886af640 1 --2- 172.21.15.83:0/1716329244 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7d6406b4d0 0x7f7d6406d990 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c036000 tx=0x7f7d7c03c040 comp rx=0 tx=0).stop 2024-03-06T07:23:10.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.671+0000 7f7d886af640 1 -- 172.21.15.83:0/1716329244 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d800cfff0 msgr2=0x7f7d800b5ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.671+0000 7f7d886af640 1 --2- 172.21.15.83:0/1716329244 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d800cfff0 0x7f7d800b5ab0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f7d7000d800 tx=0x7f7d7000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.671+0000 7f7d886af640 1 -- 172.21.15.83:0/1716329244 shutdown_connections 2024-03-06T07:23:10.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.671+0000 7f7d886af640 1 --2- 172.21.15.83:0/1716329244 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7d6406b4d0 0x7f7d6406d990 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.671+0000 7f7d886af640 1 --2- 172.21.15.83:0/1716329244 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d8015bb20 0x7f7d800b5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.671+0000 7f7d886af640 1 --2- 172.21.15.83:0/1716329244 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d800cfff0 0x7f7d800b5ab0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.671+0000 7f7d886af640 1 --2- 172.21.15.83:0/1716329244 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d800ce4b0 0x7f7d800c7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.671+0000 7f7d886af640 1 -- 172.21.15.83:0/1716329244 >> 172.21.15.83:0/1716329244 conn(0x7f7d8009a9f0 msgr2=0x7f7d800ce890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:10.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.671+0000 7f7d886af640 1 -- 172.21.15.83:0/1716329244 shutdown_connections 2024-03-06T07:23:10.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:10.671+0000 7f7d886af640 1 -- 172.21.15.83:0/1716329244 wait complete. 2024-03-06T07:23:10.684 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:23:10.684 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:23:10.684 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi083.front.sepia.ceph.com 2024-03-06T07:23:10.684 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:23:10.684 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:10.684 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:23:10.684 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:23:10.684 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:23:10.685 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:23:10.685 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:10.685 DEBUG:teuthology.orchestra.run.smithi083:> ip addr 2024-03-06T07:23:10.700 INFO:teuthology.orchestra.run.smithi083.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:23:10.701 INFO:teuthology.orchestra.run.smithi083.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:23:10.701 INFO:teuthology.orchestra.run.smithi083.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:23:10.701 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:10.701 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 ::1/128 scope host 2024-03-06T07:23:10.701 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:10.701 INFO:teuthology.orchestra.run.smithi083.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:23:10.701 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:d8:da:1e brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:10.701 INFO:teuthology.orchestra.run.smithi083.stdout: altname enp7s0f0 2024-03-06T07:23:10.701 INFO:teuthology.orchestra.run.smithi083.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:23:10.701 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:d8:da:1f brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:10.701 INFO:teuthology.orchestra.run.smithi083.stdout: altname enp7s0f1 2024-03-06T07:23:10.702 INFO:teuthology.orchestra.run.smithi083.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:23:10.702 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:8f:ce:90 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:10.702 INFO:teuthology.orchestra.run.smithi083.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:23:10.702 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:8f:ce:91 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:10.702 INFO:teuthology.orchestra.run.smithi083.stdout: inet 172.21.15.83/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:23:10.702 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft 42665sec preferred_lft 42665sec 2024-03-06T07:23:10.702 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::ec4:7aff:fe8f:ce91/64 scope link 2024-03-06T07:23:10.702 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:10.702 INFO:teuthology.orchestra.run.smithi083.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:23:10.702 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 1e:29:46:c9:e7:c8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:10.703 INFO:teuthology.orchestra.run.smithi083.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:23:10.703 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:10.703 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::60c5:d7ff:fe70:2b1c/64 scope link 2024-03-06T07:23:10.703 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:10.703 INFO:teuthology.orchestra.run.smithi083.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:23:10.703 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 1e:29:46:c9:e7:c8 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:10.703 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::1c29:46ff:fec9:e7c8/64 scope link 2024-03-06T07:23:10.703 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:10.703 INFO:teuthology.orchestra.run.smithi083.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:23:10.703 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 3e:00:2f:23:d5:45 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:10.703 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::3c00:2fff:fe23:d545/64 scope link 2024-03-06T07:23:10.703 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:10.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:10.704 DEBUG:teuthology.orchestra.run.smithi083:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:23:10.765 INFO:teuthology.orchestra.run.smithi083.stdout:1 2024-03-06T07:23:10.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:10.767 DEBUG:teuthology.orchestra.run.smithi083:> route 2024-03-06T07:23:10.822 INFO:teuthology.orchestra.run.smithi083.stdout:Kernel IP routing table 2024-03-06T07:23:10.823 INFO:teuthology.orchestra.run.smithi083.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:23:10.823 INFO:teuthology.orchestra.run.smithi083.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:23:10.823 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:23:10.823 INFO:teuthology.orchestra.run.smithi083.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:23:10.823 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:10.823 DEBUG:teuthology.orchestra.run.smithi083:> (cd / && exec bash -c ' 2024-03-06T07:23:10.823 DEBUG:teuthology.orchestra.run.smithi083:> set -e 2024-03-06T07:23:10.823 DEBUG:teuthology.orchestra.run.smithi083:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:23:10.823 DEBUG:teuthology.orchestra.run.smithi083:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:23:10.824 DEBUG:teuthology.orchestra.run.smithi083:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:23:10.824 DEBUG:teuthology.orchestra.run.smithi083:> ') 2024-03-06T07:23:10.921 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:10.922 DEBUG:teuthology.orchestra.run.smithi083:> ip netns list 2024-03-06T07:23:10.936 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:23:10.936 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:23:10.937 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:23:10.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:10.937 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip link set brx.0 up 2024-03-06T07:23:11.006 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:23:11.006 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:23:11.006 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:11.060 INFO:teuthology.orchestra.run.smithi083.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:23:11.060 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:23:11.060 DEBUG:teuthology.orchestra.run.smithi083:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:11.114 DEBUG:teuthology.orchestra.run.smithi083:> sudo modprobe fuse 2024-03-06T07:23:11.181 DEBUG:teuthology.orchestra.run.smithi083:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:23:11.237 INFO:teuthology.orchestra.run.smithi083.stdout:/proc 2024-03-06T07:23:11.237 INFO:teuthology.orchestra.run.smithi083.stdout:/sys 2024-03-06T07:23:11.237 INFO:teuthology.orchestra.run.smithi083.stdout:/dev 2024-03-06T07:23:11.237 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/security 2024-03-06T07:23:11.237 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/shm 2024-03-06T07:23:11.237 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/pts 2024-03-06T07:23:11.237 INFO:teuthology.orchestra.run.smithi083.stdout:/run 2024-03-06T07:23:11.237 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/cgroup 2024-03-06T07:23:11.237 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/pstore 2024-03-06T07:23:11.237 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/bpf 2024-03-06T07:23:11.238 INFO:teuthology.orchestra.run.smithi083.stdout:/ 2024-03-06T07:23:11.238 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/selinux 2024-03-06T07:23:11.238 INFO:teuthology.orchestra.run.smithi083.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:23:11.238 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/mqueue 2024-03-06T07:23:11.238 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/hugepages 2024-03-06T07:23:11.238 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/debug 2024-03-06T07:23:11.238 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/tracing 2024-03-06T07:23:11.238 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:23:11.238 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/fuse/connections 2024-03-06T07:23:11.238 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/config 2024-03-06T07:23:11.238 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:23:11.238 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:23:11.238 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:23:11.239 INFO:teuthology.orchestra.run.smithi083.stdout:/run/user/1000 2024-03-06T07:23:11.239 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph 2024-03-06T07:23:11.239 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:23:11.239 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:23:11.239 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:23:11.239 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:23:11.239 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns 2024-03-06T07:23:11.239 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:11.239 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:11.239 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:11.239 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:11.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:11.240 DEBUG:teuthology.orchestra.run.smithi083:> ls /sys/fs/fuse/connections 2024-03-06T07:23:11.292 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:23:11.292 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:23:11.335 DEBUG:teuthology.orchestra.run.smithi083:> sudo modprobe fuse 2024-03-06T07:23:11.364 DEBUG:teuthology.orchestra.run.smithi083:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:23:11.420 INFO:teuthology.orchestra.run.smithi083.stdout:/proc 2024-03-06T07:23:11.420 INFO:teuthology.orchestra.run.smithi083.stdout:/sys 2024-03-06T07:23:11.420 INFO:teuthology.orchestra.run.smithi083.stdout:/dev 2024-03-06T07:23:11.420 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/security 2024-03-06T07:23:11.420 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/shm 2024-03-06T07:23:11.420 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/pts 2024-03-06T07:23:11.420 INFO:teuthology.orchestra.run.smithi083.stdout:/run 2024-03-06T07:23:11.420 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/cgroup 2024-03-06T07:23:11.421 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/pstore 2024-03-06T07:23:11.421 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/bpf 2024-03-06T07:23:11.421 INFO:teuthology.orchestra.run.smithi083.stdout:/ 2024-03-06T07:23:11.421 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/selinux 2024-03-06T07:23:11.421 INFO:teuthology.orchestra.run.smithi083.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:23:11.421 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/mqueue 2024-03-06T07:23:11.421 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/hugepages 2024-03-06T07:23:11.421 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/debug 2024-03-06T07:23:11.421 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/tracing 2024-03-06T07:23:11.421 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:23:11.421 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/fuse/connections 2024-03-06T07:23:11.421 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/config 2024-03-06T07:23:11.421 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:23:11.421 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:23:11.422 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:23:11.422 INFO:teuthology.orchestra.run.smithi083.stdout:/run/user/1000 2024-03-06T07:23:11.422 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph 2024-03-06T07:23:11.422 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:23:11.422 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:23:11.422 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:23:11.422 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:23:11.422 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns 2024-03-06T07:23:11.422 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:11.422 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:11.422 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:11.422 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:11.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:11.423 DEBUG:teuthology.orchestra.run.smithi083:> ls /sys/fs/fuse/connections 2024-03-06T07:23:11.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:2024-03-06T07:23:11.423+0000 7fd5ff83e480 -1 init, newargv = 0x55e7f0047750 newargc=15 2024-03-06T07:23:11.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:ceph-fuse[52896]: starting ceph client 2024-03-06T07:23:11.429 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:ceph-fuse[52896]: starting fuse 2024-03-06T07:23:11.442 INFO:teuthology.orchestra.run.smithi083.stdout:41 2024-03-06T07:23:11.443 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-06T07:23:11.443 DEBUG:teuthology.orchestra.run.smithi083:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:23:11.443 DEBUG:teuthology.orchestra.run.smithi083:> import glob 2024-03-06T07:23:11.443 DEBUG:teuthology.orchestra.run.smithi083:> import re 2024-03-06T07:23:11.443 DEBUG:teuthology.orchestra.run.smithi083:> import os 2024-03-06T07:23:11.443 DEBUG:teuthology.orchestra.run.smithi083:> import subprocess 2024-03-06T07:23:11.443 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:23:11.443 DEBUG:teuthology.orchestra.run.smithi083:> def _find_admin_socket(client_name): 2024-03-06T07:23:11.443 DEBUG:teuthology.orchestra.run.smithi083:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:23:11.443 DEBUG:teuthology.orchestra.run.smithi083:> files = glob.glob(asok_path) 2024-03-06T07:23:11.443 DEBUG:teuthology.orchestra.run.smithi083:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:23:11.444 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:23:11.444 DEBUG:teuthology.orchestra.run.smithi083:> # Given a non-glob path, it better be there 2024-03-06T07:23:11.444 DEBUG:teuthology.orchestra.run.smithi083:> if "*" not in asok_path: 2024-03-06T07:23:11.444 DEBUG:teuthology.orchestra.run.smithi083:> assert(len(files) == 1) 2024-03-06T07:23:11.444 DEBUG:teuthology.orchestra.run.smithi083:> return files[0] 2024-03-06T07:23:11.444 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:23:11.444 DEBUG:teuthology.orchestra.run.smithi083:> for f in files: 2024-03-06T07:23:11.444 DEBUG:teuthology.orchestra.run.smithi083:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:23:11.444 DEBUG:teuthology.orchestra.run.smithi083:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:23:11.444 DEBUG:teuthology.orchestra.run.smithi083:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:23:11.444 DEBUG:teuthology.orchestra.run.smithi083:> contents = proc_f.read() 2024-03-06T07:23:11.444 DEBUG:teuthology.orchestra.run.smithi083:> if mountpoint in contents: 2024-03-06T07:23:11.444 DEBUG:teuthology.orchestra.run.smithi083:> return f 2024-03-06T07:23:11.445 DEBUG:teuthology.orchestra.run.smithi083:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:23:11.445 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:23:11.445 DEBUG:teuthology.orchestra.run.smithi083:> print(_find_admin_socket("client.0")) 2024-03-06T07:23:11.445 DEBUG:teuthology.orchestra.run.smithi083:> ' 2024-03-06T07:23:11.562 INFO:teuthology.orchestra.run.smithi083.stdout:/var/run/ceph/ceph-client.0.52896.asok 2024-03-06T07:23:11.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:23:11.573 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.52896.asok 2024-03-06T07:23:11.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:11.573 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.52896.asok status 2024-03-06T07:23:11.709 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:23:11.710 INFO:teuthology.orchestra.run.smithi083.stdout: "metadata": { 2024-03-06T07:23:11.710 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:23:11.710 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:23:11.710 INFO:teuthology.orchestra.run.smithi083.stdout: "entity_id": "0", 2024-03-06T07:23:11.710 INFO:teuthology.orchestra.run.smithi083.stdout: "hostname": "smithi083", 2024-03-06T07:23:11.710 INFO:teuthology.orchestra.run.smithi083.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:23:11.710 INFO:teuthology.orchestra.run.smithi083.stdout: "pid": "52896", 2024-03-06T07:23:11.710 INFO:teuthology.orchestra.run.smithi083.stdout: "root": "/" 2024-03-06T07:23:11.710 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:23:11.710 INFO:teuthology.orchestra.run.smithi083.stdout: "dentry_count": 0, 2024-03-06T07:23:11.710 INFO:teuthology.orchestra.run.smithi083.stdout: "dentry_pinned_count": 0, 2024-03-06T07:23:11.711 INFO:teuthology.orchestra.run.smithi083.stdout: "id": 5466, 2024-03-06T07:23:11.711 INFO:teuthology.orchestra.run.smithi083.stdout: "inst": { 2024-03-06T07:23:11.711 INFO:teuthology.orchestra.run.smithi083.stdout: "name": { 2024-03-06T07:23:11.711 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "client", 2024-03-06T07:23:11.711 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 5466 2024-03-06T07:23:11.711 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:23:11.711 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": { 2024-03-06T07:23:11.711 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-03-06T07:23:11.711 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:23:11.711 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 3303732289 2024-03-06T07:23:11.711 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:23:11.711 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:23:11.711 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": { 2024-03-06T07:23:11.712 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-03-06T07:23:11.712 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:23:11.712 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 3303732289 2024-03-06T07:23:11.712 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:23:11.712 INFO:teuthology.orchestra.run.smithi083.stdout: "inst_str": "client.5466 192.168.0.1:0/3303732289", 2024-03-06T07:23:11.712 INFO:teuthology.orchestra.run.smithi083.stdout: "addr_str": "192.168.0.1:0/3303732289", 2024-03-06T07:23:11.712 INFO:teuthology.orchestra.run.smithi083.stdout: "inode_count": 1, 2024-03-06T07:23:11.712 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_epoch": 13, 2024-03-06T07:23:11.712 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_epoch": 33, 2024-03-06T07:23:11.712 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:23:11.712 INFO:teuthology.orchestra.run.smithi083.stdout: "blocklisted": false, 2024-03-06T07:23:11.712 INFO:teuthology.orchestra.run.smithi083.stdout: "fs_name": "cephfs" 2024-03-06T07:23:11.712 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:23:11.718 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:11.718 DEBUG:teuthology.orchestra.run.smithi083:> stat --file-system '--printf=%T 2024-03-06T07:23:11.718 DEBUG:teuthology.orchestra.run.smithi083:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:11.733 INFO:teuthology.orchestra.run.smithi083.stdout:fuseblk 2024-03-06T07:23:11.733 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:11.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:11.734 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:11.801 DEBUG:teuthology.orchestra.run.smithi083:> stdin-killer -- python3 -c ' 2024-03-06T07:23:11.801 DEBUG:teuthology.orchestra.run.smithi083:> import os 2024-03-06T07:23:11.801 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:23:11.801 DEBUG:teuthology.orchestra.run.smithi083:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:23:11.801 DEBUG:teuthology.orchestra.run.smithi083:> subdir = "delete_me" 2024-03-06T07:23:11.802 DEBUG:teuthology.orchestra.run.smithi083:> size = 1024 2024-03-06T07:23:11.802 DEBUG:teuthology.orchestra.run.smithi083:> file_count = 1000 2024-03-06T07:23:11.802 DEBUG:teuthology.orchestra.run.smithi083:> os.mkdir(os.path.join(mountpoint, subdir)) 2024-03-06T07:23:11.802 DEBUG:teuthology.orchestra.run.smithi083:> for i in range(0, file_count): 2024-03-06T07:23:11.802 DEBUG:teuthology.orchestra.run.smithi083:> filename = "{0}_{1}.bin".format(i, size) 2024-03-06T07:23:11.802 DEBUG:teuthology.orchestra.run.smithi083:> with open(os.path.join(mountpoint, subdir, filename), '"'"'w'"'"') as f: 2024-03-06T07:23:11.802 DEBUG:teuthology.orchestra.run.smithi083:> f.write(size * '"'"'x'"'"') 2024-03-06T07:23:11.802 DEBUG:teuthology.orchestra.run.smithi083:> ' 2024-03-06T07:23:13.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:23:13.679 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:13.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.817+0000 7fe2ba26b640 1 Processor -- start 2024-03-06T07:23:13.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.817+0000 7fe2ba26b640 1 -- start start 2024-03-06T07:23:13.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.818+0000 7fe2ba26b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe2b415dfe0 0x7fe2b40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.818+0000 7fe2ba26b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe2b40d29c0 0x7fe2b40b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.818+0000 7fe2ba26b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2b40b3430 con 0x7fe2b40cf830 2024-03-06T07:23:13.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.818+0000 7fe2b8808640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe2b40d29c0 0x7fe2b40b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.818+0000 7fe2ba26b640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2b40b31e0 con 0x7fe2b40d29c0 2024-03-06T07:23:13.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.818+0000 7fe2ba26b640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b40b3d60 con 0x7fe2b415dfe0 2024-03-06T07:23:13.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.818+0000 7fe2b8808640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe2b40d29c0 0x7fe2b40b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55758/0 (socket says 172.21.15.83:55758) 2024-03-06T07:23:13.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.818+0000 7fe2b8808640 1 -- 172.21.15.83:0/1865523434 learned_addr learned my addr 172.21.15.83:0/1865523434 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:13.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.818+0000 7fe2b3fff640 1 --2- 172.21.15.83:0/1865523434 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe2b415dfe0 0x7fe2b40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.818+0000 7fe2b8808640 1 -- 172.21.15.83:0/1865523434 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe2b415dfe0 msgr2=0x7fe2b40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:13.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.818+0000 7fe2b8808640 1 --2- 172.21.15.83:0/1865523434 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe2b415dfe0 0x7fe2b40cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.818+0000 7fe2b8808640 1 -- 172.21.15.83:0/1865523434 >> v1:172.21.15.183:6789/0 conn(0x7fe2b40cf830 legacy=0x7fe2b40d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.818+0000 7fe2b8808640 1 -- 172.21.15.83:0/1865523434 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b40b3710 con 0x7fe2b40d29c0 2024-03-06T07:23:13.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.819+0000 7fe2b8808640 1 --2- 172.21.15.83:0/1865523434 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe2b40d29c0 0x7fe2b40b41f0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac009e50 tx=0x7fe2ac02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f85078f9752e7577 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:13.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.819+0000 7fe2b2ffd640 1 -- 172.21.15.83:0/1865523434 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ac02cb00 con 0x7fe2b40d29c0 2024-03-06T07:23:13.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.819+0000 7fe2b2ffd640 1 -- 172.21.15.83:0/1865523434 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2ac02cca0 con 0x7fe2b40d29c0 2024-03-06T07:23:13.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.819+0000 7fe2b2ffd640 1 -- 172.21.15.83:0/1865523434 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ac034610 con 0x7fe2b40d29c0 2024-03-06T07:23:13.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.820+0000 7fe2ba26b640 1 -- 172.21.15.83:0/1865523434 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe2b40d29c0 msgr2=0x7fe2b40b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.820+0000 7fe2ba26b640 1 --2- 172.21.15.83:0/1865523434 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe2b40d29c0 0x7fe2b40b41f0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac009e50 tx=0x7fe2ac02bf60 comp rx=0 tx=0).stop 2024-03-06T07:23:13.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.820+0000 7fe2ba26b640 1 -- 172.21.15.83:0/1865523434 shutdown_connections 2024-03-06T07:23:13.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.820+0000 7fe2ba26b640 1 --2- 172.21.15.83:0/1865523434 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe2b40d29c0 0x7fe2b40b41f0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.820+0000 7fe2ba26b640 1 --2- 172.21.15.83:0/1865523434 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe2b415dfe0 0x7fe2b40cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.820+0000 7fe2ba26b640 1 -- 172.21.15.83:0/1865523434 >> 172.21.15.83:0/1865523434 conn(0x7fe2b40d3cd0 msgr2=0x7fe2b40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:13.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.820+0000 7fe2ba26b640 1 -- 172.21.15.83:0/1865523434 shutdown_connections 2024-03-06T07:23:13.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.820+0000 7fe2ba26b640 1 -- 172.21.15.83:0/1865523434 wait complete. 2024-03-06T07:23:13.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.822+0000 7fe2ba26b640 1 Processor -- start 2024-03-06T07:23:13.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.822+0000 7fe2ba26b640 1 -- start start 2024-03-06T07:23:13.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.823+0000 7fe2ba26b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe2b40cf830 0x7fe2b40b9420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.823+0000 7fe2ba26b640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe2b40d29c0 0x7fe2b40cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.823+0000 7fe2ba26b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe2b415dfe0 0x7fe2b40d4320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.823+0000 7fe2ba26b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b40d1df0 con 0x7fe2b415dfe0 2024-03-06T07:23:13.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.823+0000 7fe2ba26b640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b40d1c70 con 0x7fe2b40d29c0 2024-03-06T07:23:13.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.823+0000 7fe2ba26b640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b415e3c0 con 0x7fe2b40cf830 2024-03-06T07:23:13.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.823+0000 7fe2b8808640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe2b415dfe0 0x7fe2b40d4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.823+0000 7fe2b8808640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe2b415dfe0 0x7fe2b40d4320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55768/0 (socket says 172.21.15.83:55768) 2024-03-06T07:23:13.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.823+0000 7fe2b8808640 1 -- 172.21.15.83:0/3568133181 learned_addr learned my addr 172.21.15.83:0/3568133181 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:13.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.823+0000 7fe2b3fff640 1 --2- 172.21.15.83:0/3568133181 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe2b40cf830 0x7fe2b40b9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.823+0000 7fe2b37fe640 1 --2- 172.21.15.83:0/3568133181 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe2b40d29c0 0x7fe2b40cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.823+0000 7fe2b8808640 1 -- 172.21.15.83:0/3568133181 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe2b40cf830 msgr2=0x7fe2b40b9420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.823+0000 7fe2b8808640 1 --2- 172.21.15.83:0/3568133181 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe2b40cf830 0x7fe2b40b9420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.823+0000 7fe2b8808640 1 -- 172.21.15.83:0/3568133181 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe2b40d29c0 msgr2=0x7fe2b40cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.823+0000 7fe2b8808640 1 --2- 172.21.15.83:0/3568133181 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe2b40d29c0 0x7fe2b40cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.823+0000 7fe2b8808640 1 -- 172.21.15.83:0/3568133181 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b40cc950 con 0x7fe2b415dfe0 2024-03-06T07:23:13.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.823+0000 7fe2b8808640 1 --2- 172.21.15.83:0/3568133181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe2b415dfe0 0x7fe2b40d4320 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac036040 tx=0x7fe2ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:13.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.823+0000 7fe2b0ff9640 1 -- 172.21.15.83:0/3568133181 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ac043020 con 0x7fe2b415dfe0 2024-03-06T07:23:13.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.823+0000 7fe2b0ff9640 1 -- 172.21.15.83:0/3568133181 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2ac02ce50 con 0x7fe2b415dfe0 2024-03-06T07:23:13.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.823+0000 7fe2ba26b640 1 -- 172.21.15.83:0/3568133181 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2b41f7940 con 0x7fe2b415dfe0 2024-03-06T07:23:13.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.823+0000 7fe2b0ff9640 1 -- 172.21.15.83:0/3568133181 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ac009510 con 0x7fe2b415dfe0 2024-03-06T07:23:13.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.823+0000 7fe2ba26b640 1 -- 172.21.15.83:0/3568133181 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2b41f81e0 con 0x7fe2b415dfe0 2024-03-06T07:23:13.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.825+0000 7fe2ba26b640 1 -- 172.21.15.83:0/3568133181 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2b404f330 con 0x7fe2b415dfe0 2024-03-06T07:23:13.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.825+0000 7fe2b0ff9640 1 -- 172.21.15.83:0/3568133181 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe2ac046020 con 0x7fe2b415dfe0 2024-03-06T07:23:13.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.825+0000 7fe2b0ff9640 1 --2- 172.21.15.83:0/3568133181 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe2a006b4d0 0x7fe2a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.826+0000 7fe2b3fff640 1 --2- 172.21.15.83:0/3568133181 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe2a006b4d0 0x7fe2a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.826+0000 7fe2b0ff9640 1 -- 172.21.15.83:0/3568133181 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe2ac0ac6d0 con 0x7fe2b415dfe0 2024-03-06T07:23:13.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.826+0000 7fe2b3fff640 1 --2- 172.21.15.83:0/3568133181 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe2a006b4d0 0x7fe2a006d990 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe2a4009aa0 tx=0x7fe2a4009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:13.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:13.830+0000 7fe2b0ff9640 1 -- 172.21.15.83:0/3568133181 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe2ac04d030 con 0x7fe2b415dfe0 2024-03-06T07:23:14.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.097+0000 7fe2ba26b640 1 -- 172.21.15.83:0/3568133181 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe2b40d0550 con 0x7fe2b415dfe0 2024-03-06T07:23:14.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.097+0000 7fe2b0ff9640 1 -- 172.21.15.83:0/3568133181 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+4982 (secure 0 0 0) 0x7fe2ac07af80 con 0x7fe2b415dfe0 2024-03-06T07:23:14.101 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:23:14.101 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4799,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/3542419057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3542419057},{"type":"v1","addr":"172.21.15.83:6836","nonce":3542419057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":5023,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1853869906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":1853869906},{"type":"v1","addr":"172.21.15.183:6835","nonce":1853869906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5026,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/3666667479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3666667479},{"type":"v1","addr":"172.21.15.183:6834","nonce":3666667479}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:00.145321+0000","modified":"2024-03-06T07:23:02.319419+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5244":{"gid":5244,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/2668925454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":2668925454},{"type":"v1","addr":"172.21.15.83:6837","nonce":2668925454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5244,"qdb_cluster":[5244]},"id":2}]} 2024-03-06T07:23:14.101 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 13 2024-03-06T07:23:14.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.099+0000 7fe2ba26b640 1 -- 172.21.15.83:0/3568133181 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe2a006b4d0 msgr2=0x7fe2a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.099+0000 7fe2ba26b640 1 --2- 172.21.15.83:0/3568133181 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe2a006b4d0 0x7fe2a006d990 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe2a4009aa0 tx=0x7fe2a4009190 comp rx=0 tx=0).stop 2024-03-06T07:23:14.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.099+0000 7fe2ba26b640 1 -- 172.21.15.83:0/3568133181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe2b415dfe0 msgr2=0x7fe2b40d4320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.099+0000 7fe2ba26b640 1 --2- 172.21.15.83:0/3568133181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe2b415dfe0 0x7fe2b40d4320 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac036040 tx=0x7fe2ac004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:14.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.100+0000 7fe2ba26b640 1 -- 172.21.15.83:0/3568133181 shutdown_connections 2024-03-06T07:23:14.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.100+0000 7fe2ba26b640 1 --2- 172.21.15.83:0/3568133181 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe2a006b4d0 0x7fe2a006d990 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.100+0000 7fe2ba26b640 1 --2- 172.21.15.83:0/3568133181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe2b415dfe0 0x7fe2b40d4320 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.100+0000 7fe2ba26b640 1 --2- 172.21.15.83:0/3568133181 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe2b40d29c0 0x7fe2b40cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.100+0000 7fe2ba26b640 1 --2- 172.21.15.83:0/3568133181 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe2b40cf830 0x7fe2b40b9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.100+0000 7fe2ba26b640 1 -- 172.21.15.83:0/3568133181 >> 172.21.15.83:0/3568133181 conn(0x7fe2b40d3cd0 msgr2=0x7fe2b406c370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:14.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.100+0000 7fe2ba26b640 1 -- 172.21.15.83:0/3568133181 shutdown_connections 2024-03-06T07:23:14.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.100+0000 7fe2ba26b640 1 -- 172.21.15.83:0/3568133181 wait complete. 2024-03-06T07:23:14.117 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:23:14.375 INFO:teuthology.orchestra.run.smithi083.stdout:{"message":"","return_code":0} 2024-03-06T07:23:14.375 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:23:14.376 DEBUG:teuthology.orchestra.run.smithi083:> stdin-killer -- python3 -c ' 2024-03-06T07:23:14.376 DEBUG:teuthology.orchestra.run.smithi083:> import os 2024-03-06T07:23:14.376 DEBUG:teuthology.orchestra.run.smithi083:> import stat 2024-03-06T07:23:14.376 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:23:14.376 DEBUG:teuthology.orchestra.run.smithi083:> print(os.stat("/home/ubuntu/cephtest/mnt.0/delete_me").st_ino) 2024-03-06T07:23:14.376 DEBUG:teuthology.orchestra.run.smithi083:> ' 2024-03-06T07:23:14.444 INFO:teuthology.orchestra.run.smithi083.stdout:1099511627776 2024-03-06T07:23:14.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:23:14.453 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 10000000000.00000000 2024-03-06T07:23:14.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.532+0000 7f066143bf40 1 Processor -- start 2024-03-06T07:23:14.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.532+0000 7f066143bf40 1 -- start start 2024-03-06T07:23:14.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.532+0000 7f066143bf40 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06480030f0 0x555a73415d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.532+0000 7f066143bf40 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x555a73413450 0x555a734149e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.532+0000 7f066143bf40 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a734129b0 0x555a73414f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.532+0000 7f066143bf40 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a732e5b70 con 0x7f06480030f0 2024-03-06T07:23:14.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.532+0000 7f066143bf40 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a732e9ad0 con 0x555a734129b0 2024-03-06T07:23:14.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.532+0000 7f066143bf40 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x555a73387700 con 0x555a73413450 2024-03-06T07:23:14.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.532+0000 7f065e726640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06480030f0 0x555a73415d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.532+0000 7f065e726640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06480030f0 0x555a73415d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55780/0 (socket says 172.21.15.83:55780) 2024-03-06T07:23:14.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.532+0000 7f065e726640 1 -- 172.21.15.83:0/3235638870 learned_addr learned my addr 172.21.15.83:0/3235638870 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:14.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.533+0000 7f065ef27640 1 --2- 172.21.15.83:0/3235638870 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a734129b0 0x555a73414f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.533+0000 7f065df25640 1 --2- 172.21.15.83:0/3235638870 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x555a73413450 0x555a734149e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.533+0000 7f065e726640 1 -- 172.21.15.83:0/3235638870 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x555a73413450 msgr2=0x555a734149e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:14.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.533+0000 7f065e726640 1 --2- 172.21.15.83:0/3235638870 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x555a73413450 0x555a734149e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.533+0000 7f065e726640 1 -- 172.21.15.83:0/3235638870 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a734129b0 msgr2=0x555a73414f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:14.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.533+0000 7f065e726640 1 --2- 172.21.15.83:0/3235638870 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a734129b0 0x555a73414f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.533+0000 7f065e726640 1 -- 172.21.15.83:0/3235638870 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555a7336b5e0 con 0x7f06480030f0 2024-03-06T07:23:14.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.533+0000 7f065e726640 1 --2- 172.21.15.83:0/3235638870 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06480030f0 0x555a73415d30 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f065800d720 tx=0x7f065800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc89a7f570c102a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:14.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.533+0000 7f065cf23640 1 -- 172.21.15.83:0/3235638870 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0658013070 con 0x7f06480030f0 2024-03-06T07:23:14.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.533+0000 7f065cf23640 1 -- 172.21.15.83:0/3235638870 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f065800b850 con 0x7f06480030f0 2024-03-06T07:23:14.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.533+0000 7f065cf23640 1 -- 172.21.15.83:0/3235638870 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f065800bc40 con 0x7f06480030f0 2024-03-06T07:23:14.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.533+0000 7f066143bf40 1 -- 172.21.15.83:0/3235638870 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06480030f0 msgr2=0x555a73415d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.533+0000 7f066143bf40 1 --2- 172.21.15.83:0/3235638870 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06480030f0 0x555a73415d30 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f065800d720 tx=0x7f065800dbe0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.533+0000 7f066143bf40 1 -- 172.21.15.83:0/3235638870 shutdown_connections 2024-03-06T07:23:14.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.533+0000 7f066143bf40 1 --2- 172.21.15.83:0/3235638870 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a734129b0 0x555a73414f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.533+0000 7f066143bf40 1 --2- 172.21.15.83:0/3235638870 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x555a73413450 0x555a734149e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.533+0000 7f066143bf40 1 --2- 172.21.15.83:0/3235638870 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06480030f0 0x555a73415d30 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.533+0000 7f066143bf40 1 -- 172.21.15.83:0/3235638870 >> 172.21.15.83:0/3235638870 conn(0x555a7333d8b0 msgr2=0x555a7333cb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:14.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.534+0000 7f066143bf40 1 -- 172.21.15.83:0/3235638870 shutdown_connections 2024-03-06T07:23:14.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.534+0000 7f066143bf40 1 -- 172.21.15.83:0/3235638870 wait complete. 2024-03-06T07:23:14.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.534+0000 7f066143bf40 1 Processor -- start 2024-03-06T07:23:14.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.534+0000 7f066143bf40 1 -- start start 2024-03-06T07:23:14.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.535+0000 7f066143bf40 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555a734129b0 0x555a7340e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.535+0000 7f066143bf40 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a73413450 0x555a733f5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.535+0000 7f066143bf40 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f06480030f0 0x555a7333cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.535+0000 7f066143bf40 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a73416a80 con 0x555a734129b0 2024-03-06T07:23:14.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.535+0000 7f066143bf40 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a73416270 con 0x555a73413450 2024-03-06T07:23:14.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.535+0000 7f066143bf40 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x555a73416eb0 con 0x7f06480030f0 2024-03-06T07:23:14.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.535+0000 7f065e726640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555a734129b0 0x555a7340e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.535+0000 7f065e726640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555a734129b0 0x555a7340e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55786/0 (socket says 172.21.15.83:55786) 2024-03-06T07:23:14.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.535+0000 7f065e726640 1 -- 172.21.15.83:0/1302034946 learned_addr learned my addr 172.21.15.83:0/1302034946 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:14.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.535+0000 7f065df25640 1 --2- 172.21.15.83:0/1302034946 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a73413450 0x555a733f5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.535+0000 7f065ef27640 1 --2- 172.21.15.83:0/1302034946 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f06480030f0 0x555a7333cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.535+0000 7f065e726640 1 -- 172.21.15.83:0/1302034946 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f06480030f0 msgr2=0x555a7333cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.535+0000 7f065e726640 1 --2- 172.21.15.83:0/1302034946 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f06480030f0 0x555a7333cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.535+0000 7f065e726640 1 -- 172.21.15.83:0/1302034946 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a73413450 msgr2=0x555a733f5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.535+0000 7f065e726640 1 --2- 172.21.15.83:0/1302034946 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555a73413450 0x555a733f5740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.535+0000 7f065e726640 1 -- 172.21.15.83:0/1302034946 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555a7340ef30 con 0x555a734129b0 2024-03-06T07:23:14.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.535+0000 7f065e726640 1 --2- 172.21.15.83:0/1302034946 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555a734129b0 0x555a7340e9f0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f065800b6c0 tx=0x555a73341c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:14.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.535+0000 7f064f7fe640 1 -- 172.21.15.83:0/1302034946 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0658028070 con 0x555a734129b0 2024-03-06T07:23:14.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.536+0000 7f064f7fe640 1 -- 172.21.15.83:0/1302034946 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0658018cb0 con 0x555a734129b0 2024-03-06T07:23:14.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.536+0000 7f064f7fe640 1 -- 172.21.15.83:0/1302034946 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0658013050 con 0x555a734129b0 2024-03-06T07:23:14.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.536+0000 7f066143bf40 1 -- 172.21.15.83:0/1302034946 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x555a733f5c80 con 0x555a734129b0 2024-03-06T07:23:14.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.536+0000 7f066143bf40 1 -- 172.21.15.83:0/1302034946 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555a733f6220 con 0x555a734129b0 2024-03-06T07:23:14.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.537+0000 7f064f7fe640 1 -- 172.21.15.83:0/1302034946 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f065802b070 con 0x555a734129b0 2024-03-06T07:23:14.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.538+0000 7f064f7fe640 1 --2- 172.21.15.83:0/1302034946 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0640069eb0 0x7f064006c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.538+0000 7f064f7fe640 1 -- 172.21.15.83:0/1302034946 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f065808aab0 con 0x555a734129b0 2024-03-06T07:23:14.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.538+0000 7f065df25640 1 --2- 172.21.15.83:0/1302034946 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0640069eb0 0x7f064006c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.538+0000 7f066143bf40 1 --2- 172.21.15.83:0/1302034946 >> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] conn(0x555a733fb2f0 0x555a733fb750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.538+0000 7f066143bf40 1 -- 172.21.15.83:0/1302034946 --> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] -- osd_op(unknown.0.0:1 4.10 4:0cf6ad0f:::10000000000.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33) v9 -- 0x555a733fbc90 con 0x555a733fb2f0 2024-03-06T07:23:14.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.538+0000 7f065ef27640 1 --2- 172.21.15.83:0/1302034946 >> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] conn(0x555a733fb2f0 0x555a733fb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.538+0000 7f065df25640 1 --2- 172.21.15.83:0/1302034946 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0640069eb0 0x7f064006c370 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0654009810 tx=0x7f065404c070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:14.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.538+0000 7f065ef27640 1 --2- 172.21.15.83:0/1302034946 >> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] conn(0x555a733fb2f0 0x555a733fb750 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:14.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.539+0000 7f065ef27640 1 -- 172.21.15.83:0/1302034946 <== osd.3 v2:172.21.15.83:6815/2656563756 1 ==== osd_op_reply(1 10000000000.00000000 [stat out=16b] v0'0 uv10 ondisk = 0) v8 ==== 164+0+16 (crc 0 0 0) 0x7f0650012040 con 0x555a733fb2f0 2024-03-06T07:23:14.548 INFO:teuthology.orchestra.run.smithi083.stdout:cephfs_metadata/10000000000.00000000 mtime 2024-03-06T07:23:14.000000+0000, size 0 2024-03-06T07:23:14.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.539+0000 7f066143bf40 1 -- 172.21.15.83:0/1302034946 >> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] conn(0x555a733fb2f0 msgr2=0x555a733fb750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.539+0000 7f066143bf40 1 --2- 172.21.15.83:0/1302034946 >> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] conn(0x555a733fb2f0 0x555a733fb750 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.539+0000 7f066143bf40 1 -- 172.21.15.83:0/1302034946 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0640069eb0 msgr2=0x7f064006c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.539+0000 7f066143bf40 1 --2- 172.21.15.83:0/1302034946 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0640069eb0 0x7f064006c370 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0654009810 tx=0x7f065404c070 comp rx=0 tx=0).stop 2024-03-06T07:23:14.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.539+0000 7f066143bf40 1 -- 172.21.15.83:0/1302034946 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555a734129b0 msgr2=0x555a7340e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.539+0000 7f066143bf40 1 --2- 172.21.15.83:0/1302034946 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555a734129b0 0x555a7340e9f0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f065800b6c0 tx=0x555a73341c00 comp rx=0 tx=0).stop 2024-03-06T07:23:14.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.539+0000 7f065ef27640 1 -- 172.21.15.83:0/1302034946 reap_dead start 2024-03-06T07:23:14.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.539+0000 7f066143bf40 1 -- 172.21.15.83:0/1302034946 shutdown_connections 2024-03-06T07:23:14.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.539+0000 7f066143bf40 1 -- 172.21.15.83:0/1302034946 >> 172.21.15.83:0/1302034946 conn(0x555a7333d8b0 msgr2=0x555a734147a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:14.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.539+0000 7f066143bf40 1 -- 172.21.15.83:0/1302034946 shutdown_connections 2024-03-06T07:23:14.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:14.539+0000 7f066143bf40 1 -- 172.21.15.83:0/1302034946 wait complete. 2024-03-06T07:23:14.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:14.550 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -rf delete_me) 2024-03-06T07:23:16.905 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:17.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.023+0000 7f311f98c640 1 Processor -- start 2024-03-06T07:23:17.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.024+0000 7f311f98c640 1 -- start start 2024-03-06T07:23:17.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.024+0000 7f311f98c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f31180d22b0 0x7f31180d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.024+0000 7f311f98c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31180b3430 con 0x7f31180b41f0 2024-03-06T07:23:17.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.024+0000 7f311f98c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31180b3710 con 0x7f311815dfe0 2024-03-06T07:23:17.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.024+0000 7f311f98c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31180d2c50 con 0x7f31180d22b0 2024-03-06T07:23:17.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.024+0000 7f311df29640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f31180b41f0 0x7f31180b4650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:44612/0 (socket says 172.21.15.83:44612) 2024-03-06T07:23:17.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.024+0000 7f311df29640 1 -- 172.21.15.83:0/822885295 learned_addr learned my addr 172.21.15.83:0/822885295 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:23:17.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.025+0000 7f311cf27640 1 --2- 172.21.15.83:0/822885295 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f31180d22b0 0x7f31180d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.025+0000 7f310ffff640 1 -- 172.21.15.83:0/822885295 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2632224904 0 0) 0x7f31180b3430 con 0x7f31180b41f0 2024-03-06T07:23:17.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.025+0000 7f310ffff640 1 -- 172.21.15.83:0/822885295 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3104003520 con 0x7f31180b41f0 2024-03-06T07:23:17.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.025+0000 7f310ffff640 1 -- 172.21.15.83:0/822885295 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2547541205 0 0) 0x7f31180b3710 con 0x7f311815dfe0 2024-03-06T07:23:17.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.025+0000 7f310ffff640 1 -- 172.21.15.83:0/822885295 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31180b3430 con 0x7f311815dfe0 2024-03-06T07:23:17.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.025+0000 7f310ffff640 1 -- 172.21.15.83:0/822885295 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2860532264 0 0) 0x7f3104003520 con 0x7f31180b41f0 2024-03-06T07:23:17.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.025+0000 7f310ffff640 1 -- 172.21.15.83:0/822885295 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f31180d22b0 msgr2=0x7f31180d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:17.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.025+0000 7f310ffff640 1 --2- 172.21.15.83:0/822885295 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f31180d22b0 0x7f31180d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.025+0000 7f310ffff640 1 -- 172.21.15.83:0/822885295 >> v1:172.21.15.183:6789/0 conn(0x7f311815dfe0 legacy=0x7f31180cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.025+0000 7f310ffff640 1 -- 172.21.15.83:0/822885295 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31180b39f0 con 0x7f31180b41f0 2024-03-06T07:23:17.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.025+0000 7f310ffff640 1 -- 172.21.15.83:0/822885295 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f3114001670 con 0x7f31180b41f0 2024-03-06T07:23:17.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.025+0000 7f310ffff640 1 -- 172.21.15.83:0/822885295 >> v1:172.21.15.83:6789/0 conn(0x7f31180b41f0 legacy=0x7f31180b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.025+0000 7f310ffff640 1 --2- 172.21.15.83:0/822885295 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f31180d22b0 0x7f3104003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.025+0000 7f310ffff640 1 --2- 172.21.15.83:0/822885295 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f31040040d0 0x7f3104004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.025+0000 7f310ffff640 1 --2- 172.21.15.83:0/822885295 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3104004d40 0x7f31040051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.025+0000 7f310ffff640 1 -- 172.21.15.83:0/822885295 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f310400ba70 con 0x7f31040040d0 2024-03-06T07:23:17.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.025+0000 7f310ffff640 1 -- 172.21.15.83:0/822885295 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f310400bc20 con 0x7f3104004d40 2024-03-06T07:23:17.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.025+0000 7f310ffff640 1 -- 172.21.15.83:0/822885295 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f310400be20 con 0x7f31180d22b0 2024-03-06T07:23:17.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.025+0000 7f311cf27640 1 --2- 172.21.15.83:0/822885295 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f31040040d0 0x7f3104004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.025+0000 7f311d728640 1 --2- 172.21.15.83:0/822885295 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f31180d22b0 0x7f3104003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.025+0000 7f311df29640 1 --2- 172.21.15.83:0/822885295 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3104004d40 0x7f31040051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.026+0000 7f311cf27640 1 -- 172.21.15.83:0/822885295 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f31180d22b0 msgr2=0x7f3104003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:17.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.026+0000 7f311cf27640 1 --2- 172.21.15.83:0/822885295 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f31180d22b0 0x7f3104003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.026+0000 7f311cf27640 1 -- 172.21.15.83:0/822885295 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3104004d40 msgr2=0x7f31040051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:17.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.026+0000 7f311cf27640 1 --2- 172.21.15.83:0/822885295 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3104004d40 0x7f31040051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.026+0000 7f311cf27640 1 -- 172.21.15.83:0/822885295 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3104003520 con 0x7f31040040d0 2024-03-06T07:23:17.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.027+0000 7f311cf27640 1 --2- 172.21.15.83:0/822885295 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f31040040d0 0x7f3104004800 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f3108009ac0 tx=0x7f3104005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d22ca61399fcad1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.027+0000 7f310ffff640 1 -- 172.21.15.83:0/822885295 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3108039070 con 0x7f31040040d0 2024-03-06T07:23:17.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.027+0000 7f310ffff640 1 -- 172.21.15.83:0/822885295 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f310802c6b0 con 0x7f31040040d0 2024-03-06T07:23:17.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.027+0000 7f310ffff640 1 -- 172.21.15.83:0/822885295 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3108009690 con 0x7f31040040d0 2024-03-06T07:23:17.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.027+0000 7f311f98c640 1 -- 172.21.15.83:0/822885295 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f31040040d0 msgr2=0x7f3104004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.027+0000 7f311f98c640 1 --2- 172.21.15.83:0/822885295 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f31040040d0 0x7f3104004800 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f3108009ac0 tx=0x7f3104005830 comp rx=0 tx=0).stop 2024-03-06T07:23:17.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.027+0000 7f311df29640 1 -- 172.21.15.83:0/822885295 reap_dead start 2024-03-06T07:23:17.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.027+0000 7f311f98c640 1 -- 172.21.15.83:0/822885295 shutdown_connections 2024-03-06T07:23:17.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.027+0000 7f311f98c640 1 -- 172.21.15.83:0/822885295 >> 172.21.15.83:0/822885295 conn(0x7f31180d3cd0 msgr2=0x7f31180d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:17.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.027+0000 7f311f98c640 1 -- 172.21.15.83:0/822885295 shutdown_connections 2024-03-06T07:23:17.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.027+0000 7f311f98c640 1 -- 172.21.15.83:0/822885295 wait complete. 2024-03-06T07:23:17.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.029+0000 7f311f98c640 1 Processor -- start 2024-03-06T07:23:17.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.029+0000 7f311f98c640 1 -- start start 2024-03-06T07:23:17.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.029+0000 7f311f98c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f31040040d0 0x7f311814f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.029+0000 7f311f98c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f31180b9690 0x7f31180cc800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.029+0000 7f311f98c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f31180a0240 0x7f31180a06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.029+0000 7f311f98c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31180b3d60 con 0x7f31040040d0 2024-03-06T07:23:17.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.029+0000 7f311f98c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31180b31e0 con 0x7f31180b9690 2024-03-06T07:23:17.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.029+0000 7f311f98c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31180d1c70 con 0x7f31180a0240 2024-03-06T07:23:17.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.029+0000 7f311d728640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f31040040d0 0x7f311814f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.029+0000 7f311d728640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f31040040d0 0x7f311814f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56658/0 (socket says 172.21.15.83:56658) 2024-03-06T07:23:17.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.029+0000 7f311d728640 1 -- 172.21.15.83:0/3759568176 learned_addr learned my addr 172.21.15.83:0/3759568176 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:17.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.029+0000 7f311df29640 1 --2- 172.21.15.83:0/3759568176 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f31180a0240 0x7f31180a06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.029+0000 7f311cf27640 1 --2- 172.21.15.83:0/3759568176 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f31180b9690 0x7f31180cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.030+0000 7f311d728640 1 -- 172.21.15.83:0/3759568176 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f31180a0240 msgr2=0x7f31180a06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.030+0000 7f311d728640 1 --2- 172.21.15.83:0/3759568176 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f31180a0240 0x7f31180a06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.030+0000 7f311d728640 1 -- 172.21.15.83:0/3759568176 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f31180b9690 msgr2=0x7f31180cc800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.030+0000 7f311d728640 1 --2- 172.21.15.83:0/3759568176 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f31180b9690 0x7f31180cc800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.030+0000 7f311d728640 1 -- 172.21.15.83:0/3759568176 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31180b9b10 con 0x7f31040040d0 2024-03-06T07:23:17.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.030+0000 7f311cf27640 1 --2- 172.21.15.83:0/3759568176 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f31180b9690 0x7f31180cc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:17.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.030+0000 7f311df29640 1 --2- 172.21.15.83:0/3759568176 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f31180a0240 0x7f31180a06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:17.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.030+0000 7f311d728640 1 --2- 172.21.15.83:0/3759568176 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f31040040d0 0x7f311814f950 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f311000fb10 tx=0x7f31100021e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.030+0000 7f310dffb640 1 -- 172.21.15.83:0/3759568176 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3110003b80 con 0x7f31040040d0 2024-03-06T07:23:17.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.030+0000 7f310dffb640 1 -- 172.21.15.83:0/3759568176 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3110018070 con 0x7f31040040d0 2024-03-06T07:23:17.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.030+0000 7f310dffb640 1 -- 172.21.15.83:0/3759568176 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311000d880 con 0x7f31040040d0 2024-03-06T07:23:17.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.030+0000 7f311f98c640 1 -- 172.21.15.83:0/3759568176 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31180cd050 con 0x7f31040040d0 2024-03-06T07:23:17.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.030+0000 7f311f98c640 1 -- 172.21.15.83:0/3759568176 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31180cd610 con 0x7f31040040d0 2024-03-06T07:23:17.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.031+0000 7f311f98c640 1 -- 172.21.15.83:0/3759568176 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f311804f330 con 0x7f31040040d0 2024-03-06T07:23:17.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.031+0000 7f310dffb640 1 -- 172.21.15.83:0/3759568176 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f311000da20 con 0x7f31040040d0 2024-03-06T07:23:17.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.032+0000 7f310dffb640 1 --2- 172.21.15.83:0/3759568176 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f30fc06b500 0x7f30fc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.032+0000 7f310dffb640 1 -- 172.21.15.83:0/3759568176 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f311008a430 con 0x7f31040040d0 2024-03-06T07:23:17.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.032+0000 7f311cf27640 1 --2- 172.21.15.83:0/3759568176 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f30fc06b500 0x7f30fc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.032+0000 7f311cf27640 1 --2- 172.21.15.83:0/3759568176 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f30fc06b500 0x7f30fc06d9c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3108036000 tx=0x7f3108030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.039+0000 7f310dffb640 1 -- 172.21.15.83:0/3759568176 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f31100592b0 con 0x7f31040040d0 2024-03-06T07:23:17.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.255+0000 7f311f98c640 1 -- 172.21.15.83:0/3759568176 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31180b2ce0 con 0x7f31040040d0 2024-03-06T07:23:17.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.255+0000 7f310dffb640 1 -- 172.21.15.83:0/3759568176 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f3110058bb0 con 0x7f31040040d0 2024-03-06T07:23:17.257 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:23:17.257 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4799,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/3542419057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3542419057},{"type":"v1","addr":"172.21.15.83:6836","nonce":3542419057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":5023,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1853869906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":1853869906},{"type":"v1","addr":"172.21.15.183:6835","nonce":1853869906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5026,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/3666667479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3666667479},{"type":"v1","addr":"172.21.15.183:6834","nonce":3666667479}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:00.145321+0000","modified":"2024-03-06T07:23:02.319419+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5244":{"gid":5244,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/2668925454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":2668925454},{"type":"v1","addr":"172.21.15.83:6837","nonce":2668925454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5244,"qdb_cluster":[5244]},"id":2}]} 2024-03-06T07:23:17.258 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 13 2024-03-06T07:23:17.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.257+0000 7f311f98c640 1 -- 172.21.15.83:0/3759568176 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f30fc06b500 msgr2=0x7f30fc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.257+0000 7f311f98c640 1 --2- 172.21.15.83:0/3759568176 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f30fc06b500 0x7f30fc06d9c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3108036000 tx=0x7f3108030000 comp rx=0 tx=0).stop 2024-03-06T07:23:17.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.257+0000 7f311f98c640 1 -- 172.21.15.83:0/3759568176 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f31040040d0 msgr2=0x7f311814f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.257+0000 7f311f98c640 1 --2- 172.21.15.83:0/3759568176 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f31040040d0 0x7f311814f950 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f311000fb10 tx=0x7f31100021e0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.257+0000 7f311f98c640 1 -- 172.21.15.83:0/3759568176 shutdown_connections 2024-03-06T07:23:17.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.257+0000 7f311f98c640 1 --2- 172.21.15.83:0/3759568176 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f30fc06b500 0x7f30fc06d9c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.257+0000 7f311f98c640 1 --2- 172.21.15.83:0/3759568176 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f31180a0240 0x7f31180a06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.257+0000 7f311f98c640 1 --2- 172.21.15.83:0/3759568176 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f31180b9690 0x7f31180cc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.257+0000 7f311f98c640 1 --2- 172.21.15.83:0/3759568176 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f31040040d0 0x7f311814f950 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.257+0000 7f311f98c640 1 -- 172.21.15.83:0/3759568176 >> 172.21.15.83:0/3759568176 conn(0x7f31180d3cd0 msgr2=0x7f31181603d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:17.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.257+0000 7f311f98c640 1 -- 172.21.15.83:0/3759568176 shutdown_connections 2024-03-06T07:23:17.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.257+0000 7f311f98c640 1 -- 172.21.15.83:0/3759568176 wait complete. 2024-03-06T07:23:17.270 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:23:17.538 INFO:teuthology.orchestra.run.smithi083.stdout:{"message":"","return_code":0} 2024-03-06T07:23:17.538 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:23:17.539 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:17.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.639+0000 7f018ea6f640 1 Processor -- start 2024-03-06T07:23:17.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.639+0000 7f018ea6f640 1 -- start start 2024-03-06T07:23:17.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.639+0000 7f018ea6f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f018815bdd0 0x7f01880c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.639+0000 7f018ea6f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01880cebe0 con 0x7f018815fa10 2024-03-06T07:23:17.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.639+0000 7f018ea6f640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01880ceec0 con 0x7f01880d16e0 2024-03-06T07:23:17.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.639+0000 7f018ea6f640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01880ce980 con 0x7f018815bdd0 2024-03-06T07:23:17.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.640+0000 7f018c80b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f018815bdd0 0x7f01880c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.640+0000 7f0187fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f018815fa10 0x7f01880ce270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:44616/0 (socket says 172.21.15.83:44616) 2024-03-06T07:23:17.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.640+0000 7f0187fff640 1 -- 172.21.15.83:0/3858617113 learned_addr learned my addr 172.21.15.83:0/3858617113 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:23:17.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.640+0000 7f018c80b640 1 -- 172.21.15.83:0/3858617113 >> v1:172.21.15.183:6789/0 conn(0x7f01880d16e0 legacy=0x7f01880d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:17.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.640+0000 7f018c80b640 1 -- 172.21.15.83:0/3858617113 >> v1:172.21.15.83:6789/0 conn(0x7f018815fa10 legacy=0x7f01880ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.640+0000 7f01877fe640 1 -- 172.21.15.83:0/3858617113 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1906759674 0 0) 0x7f01880cebe0 con 0x7f018815fa10 2024-03-06T07:23:17.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.640+0000 7f018c80b640 1 -- 172.21.15.83:0/3858617113 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01880cf1a0 con 0x7f018815bdd0 2024-03-06T07:23:17.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.641+0000 7f018c80b640 1 --2- 172.21.15.83:0/3858617113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f018815bdd0 0x7f01880c5610 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f017c0097a0 tx=0x7f017c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d603750921e6db9d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.641+0000 7f01877fe640 1 -- 172.21.15.83:0/3858617113 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f017c0315a0 con 0x7f018815bdd0 2024-03-06T07:23:17.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.641+0000 7f01877fe640 1 -- 172.21.15.83:0/3858617113 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f017c037070 con 0x7f018815bdd0 2024-03-06T07:23:17.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.641+0000 7f01877fe640 1 -- 172.21.15.83:0/3858617113 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f017c031960 con 0x7f018815bdd0 2024-03-06T07:23:17.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.641+0000 7f018ea6f640 1 -- 172.21.15.83:0/3858617113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f018815bdd0 msgr2=0x7f01880c5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.641+0000 7f018ea6f640 1 --2- 172.21.15.83:0/3858617113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f018815bdd0 0x7f01880c5610 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f017c0097a0 tx=0x7f017c02b840 comp rx=0 tx=0).stop 2024-03-06T07:23:17.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.641+0000 7f018ea6f640 1 -- 172.21.15.83:0/3858617113 shutdown_connections 2024-03-06T07:23:17.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.641+0000 7f018ea6f640 1 --2- 172.21.15.83:0/3858617113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f018815bdd0 0x7f01880c5610 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.641+0000 7f018ea6f640 1 -- 172.21.15.83:0/3858617113 >> 172.21.15.83:0/3858617113 conn(0x7f018809a9f0 msgr2=0x7f018809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:17.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.641+0000 7f018ea6f640 1 -- 172.21.15.83:0/3858617113 shutdown_connections 2024-03-06T07:23:17.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.641+0000 7f018ea6f640 1 -- 172.21.15.83:0/3858617113 wait complete. 2024-03-06T07:23:17.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.643+0000 7f018ea6f640 1 Processor -- start 2024-03-06T07:23:17.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.643+0000 7f018ea6f640 1 -- start start 2024-03-06T07:23:17.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.643+0000 7f018ea6f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f01880d16e0 0x7f018815d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.644+0000 7f018ea6f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f018815bdd0 0x7f0188154450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.644+0000 7f018ea6f640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f018815fa10 0x7f0188153cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.644+0000 7f018ea6f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01880cc5d0 con 0x7f018815bdd0 2024-03-06T07:23:17.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.644+0000 7f018ea6f640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01880c5b50 con 0x7f018815fa10 2024-03-06T07:23:17.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.644+0000 7f018ea6f640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f018815c1b0 con 0x7f01880d16e0 2024-03-06T07:23:17.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.644+0000 7f0187fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f018815bdd0 0x7f0188154450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.644+0000 7f0187fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f018815bdd0 0x7f0188154450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56672/0 (socket says 172.21.15.83:56672) 2024-03-06T07:23:17.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.644+0000 7f0187fff640 1 -- 172.21.15.83:0/116378 learned_addr learned my addr 172.21.15.83:0/116378 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:17.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.644+0000 7f018d00c640 1 --2- 172.21.15.83:0/116378 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f018815fa10 0x7f0188153cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.644+0000 7f018c80b640 1 --2- 172.21.15.83:0/116378 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f01880d16e0 0x7f018815d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.644+0000 7f0187fff640 1 -- 172.21.15.83:0/116378 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f01880d16e0 msgr2=0x7f018815d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.644+0000 7f0187fff640 1 --2- 172.21.15.83:0/116378 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f01880d16e0 0x7f018815d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.644+0000 7f0187fff640 1 -- 172.21.15.83:0/116378 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f018815fa10 msgr2=0x7f0188153cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.644+0000 7f0187fff640 1 --2- 172.21.15.83:0/116378 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f018815fa10 0x7f0188153cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.644+0000 7f0187fff640 1 -- 172.21.15.83:0/116378 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0188154200 con 0x7f018815bdd0 2024-03-06T07:23:17.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.644+0000 7f018c80b640 1 --2- 172.21.15.83:0/116378 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f01880d16e0 0x7f018815d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:17.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.644+0000 7f0187fff640 1 --2- 172.21.15.83:0/116378 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f018815bdd0 0x7f0188154450 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f017400c3e0 tx=0x7f017400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.644+0000 7f01857fa640 1 -- 172.21.15.83:0/116378 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0174017070 con 0x7f018815bdd0 2024-03-06T07:23:17.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.644+0000 7f01857fa640 1 -- 172.21.15.83:0/116378 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0174005ce0 con 0x7f018815bdd0 2024-03-06T07:23:17.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.644+0000 7f018ea6f640 1 -- 172.21.15.83:0/116378 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0188154b50 con 0x7f018815bdd0 2024-03-06T07:23:17.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.644+0000 7f01857fa640 1 -- 172.21.15.83:0/116378 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0174007310 con 0x7f018815bdd0 2024-03-06T07:23:17.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.644+0000 7f018ea6f640 1 -- 172.21.15.83:0/116378 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0188155110 con 0x7f018815bdd0 2024-03-06T07:23:17.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.645+0000 7f018ea6f640 1 -- 172.21.15.83:0/116378 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f018804f330 con 0x7f018815bdd0 2024-03-06T07:23:17.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.646+0000 7f01857fa640 1 -- 172.21.15.83:0/116378 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f017401a070 con 0x7f018815bdd0 2024-03-06T07:23:17.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.646+0000 7f01857fa640 1 --2- 172.21.15.83:0/116378 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f016c06b500 0x7f016c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.646+0000 7f01857fa640 1 -- 172.21.15.83:0/116378 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0174089580 con 0x7f018815bdd0 2024-03-06T07:23:17.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.646+0000 7f018c80b640 1 --2- 172.21.15.83:0/116378 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f016c06b500 0x7f016c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.647+0000 7f018c80b640 1 --2- 172.21.15.83:0/116378 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f016c06b500 0x7f016c06d9c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f017c036000 tx=0x7f017c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.650+0000 7f01857fa640 1 -- 172.21.15.83:0/116378 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0174058400 con 0x7f018815bdd0 2024-03-06T07:23:17.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.868+0000 7f018ea6f640 1 -- 172.21.15.83:0/116378 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f018815b560 con 0x7f018815bdd0 2024-03-06T07:23:17.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.869+0000 7f01857fa640 1 -- 172.21.15.83:0/116378 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f0174057d00 con 0x7f018815bdd0 2024-03-06T07:23:17.871 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:23:17.872 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4799,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/3542419057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3542419057},{"type":"v1","addr":"172.21.15.83:6836","nonce":3542419057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":5023,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1853869906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":1853869906},{"type":"v1","addr":"172.21.15.183:6835","nonce":1853869906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5026,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/3666667479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3666667479},{"type":"v1","addr":"172.21.15.183:6834","nonce":3666667479}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:00.145321+0000","modified":"2024-03-06T07:23:02.319419+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5244":{"gid":5244,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/2668925454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":2668925454},{"type":"v1","addr":"172.21.15.83:6837","nonce":2668925454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5244,"qdb_cluster":[5244]},"id":2}]} 2024-03-06T07:23:17.872 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 13 2024-03-06T07:23:17.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.873+0000 7f018ea6f640 1 -- 172.21.15.83:0/116378 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f016c06b500 msgr2=0x7f016c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.873+0000 7f018ea6f640 1 --2- 172.21.15.83:0/116378 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f016c06b500 0x7f016c06d9c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f017c036000 tx=0x7f017c008030 comp rx=0 tx=0).stop 2024-03-06T07:23:17.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.873+0000 7f018ea6f640 1 -- 172.21.15.83:0/116378 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f018815bdd0 msgr2=0x7f0188154450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.873+0000 7f018ea6f640 1 --2- 172.21.15.83:0/116378 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f018815bdd0 0x7f0188154450 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f017400c3e0 tx=0x7f017400c8a0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.873+0000 7f018ea6f640 1 -- 172.21.15.83:0/116378 shutdown_connections 2024-03-06T07:23:17.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.873+0000 7f018ea6f640 1 --2- 172.21.15.83:0/116378 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f016c06b500 0x7f016c06d9c0 secure :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f017c036000 tx=0x7f017c008030 comp rx=0 tx=0).stop 2024-03-06T07:23:17.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.873+0000 7f018ea6f640 1 --2- 172.21.15.83:0/116378 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f018815fa10 0x7f0188153cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.873+0000 7f018ea6f640 1 --2- 172.21.15.83:0/116378 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f018815bdd0 0x7f0188154450 secure :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f017400c3e0 tx=0x7f017400c8a0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.873+0000 7f018ea6f640 1 --2- 172.21.15.83:0/116378 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f01880d16e0 0x7f018815d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.873+0000 7f018ea6f640 1 -- 172.21.15.83:0/116378 >> 172.21.15.83:0/116378 conn(0x7f018809a9f0 msgr2=0x7f018806eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:17.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.874+0000 7f018ea6f640 1 -- 172.21.15.83:0/116378 shutdown_connections 2024-03-06T07:23:17.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:17.874+0000 7f018ea6f640 1 -- 172.21.15.83:0/116378 wait complete. 2024-03-06T07:23:17.893 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:23:18.021 INFO:teuthology.orchestra.run.smithi083.stdout:{"mds_cache":{"strays_created":1001}} 2024-03-06T07:23:18.021 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1001 } } 2024-03-06T07:23:18.021 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:18.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.122+0000 7f0bfed1e640 1 Processor -- start 2024-03-06T07:23:18.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.122+0000 7f0bfed1e640 1 -- start start 2024-03-06T07:23:18.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.123+0000 7f0bfed1e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f0bf81424f0 0x7f0bf80cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.123+0000 7f0bfed1e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0bf80cd7a0 0x7f0bf815cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.123+0000 7f0bfed1e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0bf806d330 0x7f0bf806d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.123+0000 7f0bfed1e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bf80cd220 con 0x7f0bf806d330 2024-03-06T07:23:18.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.123+0000 7f0bfed1e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bf80cc980 con 0x7f0bf81424f0 2024-03-06T07:23:18.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.123+0000 7f0bfed1e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf80ccc60 con 0x7f0bf80cd7a0 2024-03-06T07:23:18.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.123+0000 7f0bfd2bb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0bf806d330 0x7f0bf806d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.123+0000 7f0bfd2bb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0bf806d330 0x7f0bf806d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56678/0 (socket says 172.21.15.83:56678) 2024-03-06T07:23:18.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.123+0000 7f0bfd2bb640 1 -- 172.21.15.83:0/2879216072 learned_addr learned my addr 172.21.15.83:0/2879216072 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:18.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.123+0000 7f0bfcaba640 1 --2- 172.21.15.83:0/2879216072 >> v2:172.21.15.183:3300/0 conn(0x7f0bf81424f0 0x7f0bf80cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.123+0000 7f0bf7fff640 1 --2- 172.21.15.83:0/2879216072 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0bf80cd7a0 0x7f0bf815cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.123+0000 7f0bfd2bb640 1 -- 172.21.15.83:0/2879216072 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0bf80cd7a0 msgr2=0x7f0bf815cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:18.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.123+0000 7f0bfd2bb640 1 --2- 172.21.15.83:0/2879216072 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0bf80cd7a0 0x7f0bf815cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.123+0000 7f0bfd2bb640 1 -- 172.21.15.83:0/2879216072 >> v2:172.21.15.183:3300/0 conn(0x7f0bf81424f0 msgr2=0x7f0bf80cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:18.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.123+0000 7f0bfd2bb640 1 --2- 172.21.15.83:0/2879216072 >> v2:172.21.15.183:3300/0 conn(0x7f0bf81424f0 0x7f0bf80cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.123+0000 7f0bfd2bb640 1 -- 172.21.15.83:0/2879216072 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf80ccf40 con 0x7f0bf806d330 2024-03-06T07:23:18.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.123+0000 7f0bfcaba640 1 --2- 172.21.15.83:0/2879216072 >> v2:172.21.15.183:3300/0 conn(0x7f0bf81424f0 0x7f0bf80cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:18.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.124+0000 7f0bfd2bb640 1 --2- 172.21.15.83:0/2879216072 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0bf806d330 0x7f0bf806d7c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f0bf806c6d0 tx=0x7f0be802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee31591e9ae224c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:18.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.124+0000 7f0bf77fe640 1 -- 172.21.15.83:0/2879216072 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be8035020 con 0x7f0bf806d330 2024-03-06T07:23:18.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.124+0000 7f0bf77fe640 1 -- 172.21.15.83:0/2879216072 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0be8031870 con 0x7f0bf806d330 2024-03-06T07:23:18.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.124+0000 7f0bf77fe640 1 -- 172.21.15.83:0/2879216072 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be8031c40 con 0x7f0bf806d330 2024-03-06T07:23:18.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.124+0000 7f0bfed1e640 1 -- 172.21.15.83:0/2879216072 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0bf806d330 msgr2=0x7f0bf806d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.124+0000 7f0bfed1e640 1 --2- 172.21.15.83:0/2879216072 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0bf806d330 0x7f0bf806d7c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f0bf806c6d0 tx=0x7f0be802b910 comp rx=0 tx=0).stop 2024-03-06T07:23:18.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.124+0000 7f0bfed1e640 1 -- 172.21.15.83:0/2879216072 shutdown_connections 2024-03-06T07:23:18.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.124+0000 7f0bfed1e640 1 --2- 172.21.15.83:0/2879216072 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0bf806d330 0x7f0bf806d7c0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.124+0000 7f0bfed1e640 1 --2- 172.21.15.83:0/2879216072 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0bf80cd7a0 0x7f0bf815cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.124+0000 7f0bfed1e640 1 --2- 172.21.15.83:0/2879216072 >> v2:172.21.15.183:3300/0 conn(0x7f0bf81424f0 0x7f0bf80cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.124+0000 7f0bfed1e640 1 -- 172.21.15.83:0/2879216072 >> 172.21.15.83:0/2879216072 conn(0x7f0bf80cf8a0 msgr2=0x7f0bf80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:18.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.124+0000 7f0bfed1e640 1 -- 172.21.15.83:0/2879216072 shutdown_connections 2024-03-06T07:23:18.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.124+0000 7f0bfed1e640 1 -- 172.21.15.83:0/2879216072 wait complete. 2024-03-06T07:23:18.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.126+0000 7f0bfed1e640 1 Processor -- start 2024-03-06T07:23:18.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.126+0000 7f0bfed1e640 1 -- start start 2024-03-06T07:23:18.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.126+0000 7f0bfed1e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0bf806d330 0x7f0bf815dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.126+0000 7f0bfed1e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0bf80cd7a0 0x7f0bf80ac0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.126+0000 7f0bfed1e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0bf81424f0 0x7f0bf813e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.126+0000 7f0bfed1e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf8071ce0 con 0x7f0bf81424f0 2024-03-06T07:23:18.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.126+0000 7f0bfed1e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf8071b60 con 0x7f0bf80cd7a0 2024-03-06T07:23:18.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.126+0000 7f0bfed1e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf8071e60 con 0x7f0bf806d330 2024-03-06T07:23:18.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.127+0000 7f0bfd2bb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0bf81424f0 0x7f0bf813e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.127+0000 7f0bfd2bb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0bf81424f0 0x7f0bf813e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56684/0 (socket says 172.21.15.83:56684) 2024-03-06T07:23:18.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.127+0000 7f0bfd2bb640 1 -- 172.21.15.83:0/1568214323 learned_addr learned my addr 172.21.15.83:0/1568214323 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:18.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.127+0000 7f0bfcaba640 1 --2- 172.21.15.83:0/1568214323 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0bf806d330 0x7f0bf815dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.127+0000 7f0bf7fff640 1 --2- 172.21.15.83:0/1568214323 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0bf80cd7a0 0x7f0bf80ac0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.127+0000 7f0bfd2bb640 1 -- 172.21.15.83:0/1568214323 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0bf806d330 msgr2=0x7f0bf815dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.127+0000 7f0bfd2bb640 1 --2- 172.21.15.83:0/1568214323 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0bf806d330 0x7f0bf815dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.127+0000 7f0bfd2bb640 1 -- 172.21.15.83:0/1568214323 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0bf80cd7a0 msgr2=0x7f0bf80ac0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.127+0000 7f0bfd2bb640 1 --2- 172.21.15.83:0/1568214323 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0bf80cd7a0 0x7f0bf80ac0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.127+0000 7f0bfd2bb640 1 -- 172.21.15.83:0/1568214323 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf815e5b0 con 0x7f0bf81424f0 2024-03-06T07:23:18.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.127+0000 7f0bfd2bb640 1 --2- 172.21.15.83:0/1568214323 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0bf81424f0 0x7f0bf813e2b0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f0be802b950 tx=0x7f0be80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:18.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.127+0000 7f0bf57fa640 1 -- 172.21.15.83:0/1568214323 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be803ad60 con 0x7f0bf81424f0 2024-03-06T07:23:18.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.127+0000 7f0bf57fa640 1 -- 172.21.15.83:0/1568214323 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0be80443f0 con 0x7f0bf81424f0 2024-03-06T07:23:18.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.127+0000 7f0bfed1e640 1 -- 172.21.15.83:0/1568214323 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bf815e890 con 0x7f0bf81424f0 2024-03-06T07:23:18.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.127+0000 7f0bf57fa640 1 -- 172.21.15.83:0/1568214323 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be80447e0 con 0x7f0bf81424f0 2024-03-06T07:23:18.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.127+0000 7f0bfed1e640 1 -- 172.21.15.83:0/1568214323 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bf80a9680 con 0x7f0bf81424f0 2024-03-06T07:23:18.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.128+0000 7f0bfed1e640 1 -- 172.21.15.83:0/1568214323 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bf804f330 con 0x7f0bf81424f0 2024-03-06T07:23:18.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.128+0000 7f0bf57fa640 1 -- 172.21.15.83:0/1568214323 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0be8038020 con 0x7f0bf81424f0 2024-03-06T07:23:18.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.129+0000 7f0bf57fa640 1 --2- 172.21.15.83:0/1568214323 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0be406b4d0 0x7f0be406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.129+0000 7f0bfcaba640 1 --2- 172.21.15.83:0/1568214323 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0be406b4d0 0x7f0be406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.129+0000 7f0bf57fa640 1 -- 172.21.15.83:0/1568214323 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0be80ac370 con 0x7f0bf81424f0 2024-03-06T07:23:18.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.129+0000 7f0bfcaba640 1 --2- 172.21.15.83:0/1568214323 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0be406b4d0 0x7f0be406d990 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0bf0009fd0 tx=0x7f0bf00097f0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:18.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.134+0000 7f0bf57fa640 1 -- 172.21.15.83:0/1568214323 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0be803f070 con 0x7f0bf81424f0 2024-03-06T07:23:18.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.406+0000 7f0bfed1e640 1 -- 172.21.15.83:0/1568214323 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0bf80d0620 con 0x7f0bf81424f0 2024-03-06T07:23:18.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.407+0000 7f0bf57fa640 1 -- 172.21.15.83:0/1568214323 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f0be807ab10 con 0x7f0bf81424f0 2024-03-06T07:23:18.409 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:23:18.409 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4799,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/3542419057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3542419057},{"type":"v1","addr":"172.21.15.83:6836","nonce":3542419057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":5023,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1853869906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":1853869906},{"type":"v1","addr":"172.21.15.183:6835","nonce":1853869906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5026,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/3666667479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3666667479},{"type":"v1","addr":"172.21.15.183:6834","nonce":3666667479}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:00.145321+0000","modified":"2024-03-06T07:23:02.319419+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5244":{"gid":5244,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/2668925454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":2668925454},{"type":"v1","addr":"172.21.15.83:6837","nonce":2668925454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5244,"qdb_cluster":[5244]},"id":2}]} 2024-03-06T07:23:18.409 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 13 2024-03-06T07:23:18.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.408+0000 7f0bfed1e640 1 -- 172.21.15.83:0/1568214323 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0be406b4d0 msgr2=0x7f0be406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.408+0000 7f0bfed1e640 1 --2- 172.21.15.83:0/1568214323 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0be406b4d0 0x7f0be406d990 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0bf0009fd0 tx=0x7f0bf00097f0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.408+0000 7f0bfed1e640 1 -- 172.21.15.83:0/1568214323 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0bf81424f0 msgr2=0x7f0bf813e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.408+0000 7f0bfed1e640 1 --2- 172.21.15.83:0/1568214323 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0bf81424f0 0x7f0bf813e2b0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f0be802b950 tx=0x7f0be80046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.409+0000 7f0bfed1e640 1 -- 172.21.15.83:0/1568214323 shutdown_connections 2024-03-06T07:23:18.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.409+0000 7f0bfed1e640 1 --2- 172.21.15.83:0/1568214323 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0be406b4d0 0x7f0be406d990 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.409+0000 7f0bfed1e640 1 --2- 172.21.15.83:0/1568214323 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0bf81424f0 0x7f0bf813e2b0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.409+0000 7f0bfed1e640 1 --2- 172.21.15.83:0/1568214323 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0bf80cd7a0 0x7f0bf80ac0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.409+0000 7f0bfed1e640 1 --2- 172.21.15.83:0/1568214323 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0bf806d330 0x7f0bf815dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.409+0000 7f0bfed1e640 1 -- 172.21.15.83:0/1568214323 >> 172.21.15.83:0/1568214323 conn(0x7f0bf80cf8a0 msgr2=0x7f0bf815b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:18.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.409+0000 7f0bfed1e640 1 -- 172.21.15.83:0/1568214323 shutdown_connections 2024-03-06T07:23:18.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:18.409+0000 7f0bfed1e640 1 -- 172.21.15.83:0/1568214323 wait complete. 2024-03-06T07:23:18.422 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:23:18.538 INFO:teuthology.orchestra.run.smithi083.stdout:{"mds_cache":{"strays_enqueued":427}} 2024-03-06T07:23:18.538 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 427 } } 2024-03-06T07:23:18.538 DEBUG:tasks.ceph_test_case:wait_until_equal: 427 != 1001, waiting (timeout=600)... 2024-03-06T07:23:23.540 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:23.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.643+0000 7f3b34b5c640 1 Processor -- start 2024-03-06T07:23:23.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.643+0000 7f3b34b5c640 1 -- start start 2024-03-06T07:23:23.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.644+0000 7f3b34b5c640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f3b300d22b0 0x7f3b300d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.644+0000 7f3b34b5c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3b300b41f0 0x7f3b300b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.644+0000 7f3b34b5c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b300b3430 con 0x7f3b3015dfe0 2024-03-06T07:23:23.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.644+0000 7f3b34b5c640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b300d2c50 con 0x7f3b300d22b0 2024-03-06T07:23:23.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.644+0000 7f3b34b5c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b300b31e0 con 0x7f3b300b41f0 2024-03-06T07:23:23.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.644+0000 7f3b2e59c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f3b3015dfe0 0x7f3b300cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:44632/0 (socket says 172.21.15.83:44632) 2024-03-06T07:23:23.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.644+0000 7f3b2e59c640 1 -- 172.21.15.83:0/3953595925 learned_addr learned my addr 172.21.15.83:0/3953595925 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:23:23.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.645+0000 7f3b27fff640 1 --2- 172.21.15.83:0/3953595925 >> v2:172.21.15.183:3300/0 conn(0x7f3b300d22b0 0x7f3b300d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.645+0000 7f3b2ed9d640 1 --2- 172.21.15.83:0/3953595925 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3b300b41f0 0x7f3b300b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.645+0000 7f3b2dd9b640 1 -- 172.21.15.83:0/3953595925 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 29045713 0 0) 0x7f3b300b3430 con 0x7f3b3015dfe0 2024-03-06T07:23:23.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.645+0000 7f3b2dd9b640 1 -- 172.21.15.83:0/3953595925 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b1c003520 con 0x7f3b3015dfe0 2024-03-06T07:23:23.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.645+0000 7f3b2dd9b640 1 -- 172.21.15.83:0/3953595925 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1884031461 0 0) 0x7f3b1c003520 con 0x7f3b3015dfe0 2024-03-06T07:23:23.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.645+0000 7f3b2dd9b640 1 -- 172.21.15.83:0/3953595925 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3b300b41f0 msgr2=0x7f3b300b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:23.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.645+0000 7f3b2dd9b640 1 --2- 172.21.15.83:0/3953595925 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3b300b41f0 0x7f3b300b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.645+0000 7f3b2dd9b640 1 -- 172.21.15.83:0/3953595925 >> v2:172.21.15.183:3300/0 conn(0x7f3b300d22b0 msgr2=0x7f3b300d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:23.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.645+0000 7f3b2dd9b640 1 --2- 172.21.15.83:0/3953595925 >> v2:172.21.15.183:3300/0 conn(0x7f3b300d22b0 0x7f3b300d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.645+0000 7f3b2dd9b640 1 -- 172.21.15.83:0/3953595925 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b300b3710 con 0x7f3b3015dfe0 2024-03-06T07:23:23.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.645+0000 7f3b2dd9b640 1 -- 172.21.15.83:0/3953595925 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f3b20001670 con 0x7f3b3015dfe0 2024-03-06T07:23:23.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.645+0000 7f3b2dd9b640 1 -- 172.21.15.83:0/3953595925 >> v1:172.21.15.83:6789/0 conn(0x7f3b3015dfe0 legacy=0x7f3b300cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.645+0000 7f3b2dd9b640 1 --2- 172.21.15.83:0/3953595925 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3b1c0039b0 0x7f3b1c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.645+0000 7f3b2dd9b640 1 --2- 172.21.15.83:0/3953595925 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b1c004800 0x7f3b1c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.645+0000 7f3b2dd9b640 1 --2- 172.21.15.83:0/3953595925 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3b300b41f0 0x7f3b1c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.645+0000 7f3b2dd9b640 1 -- 172.21.15.83:0/3953595925 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b300b31e0 con 0x7f3b1c004800 2024-03-06T07:23:23.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.645+0000 7f3b2dd9b640 1 -- 172.21.15.83:0/3953595925 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1c00b9c0 con 0x7f3b1c0039b0 2024-03-06T07:23:23.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.645+0000 7f3b2dd9b640 1 -- 172.21.15.83:0/3953595925 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1c00bb40 con 0x7f3b300b41f0 2024-03-06T07:23:23.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.645+0000 7f3b27fff640 1 --2- 172.21.15.83:0/3953595925 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b1c004800 0x7f3b1c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.645+0000 7f3b2ed9d640 1 --2- 172.21.15.83:0/3953595925 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3b300b41f0 0x7f3b1c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.645+0000 7f3b2e59c640 1 --2- 172.21.15.83:0/3953595925 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3b1c0039b0 0x7f3b1c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.645+0000 7f3b27fff640 1 -- 172.21.15.83:0/3953595925 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3b300b41f0 msgr2=0x7f3b1c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:23.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.645+0000 7f3b27fff640 1 --2- 172.21.15.83:0/3953595925 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3b300b41f0 0x7f3b1c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.645+0000 7f3b27fff640 1 -- 172.21.15.83:0/3953595925 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3b1c0039b0 msgr2=0x7f3b1c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:23.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.645+0000 7f3b27fff640 1 --2- 172.21.15.83:0/3953595925 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3b1c0039b0 0x7f3b1c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.645+0000 7f3b27fff640 1 -- 172.21.15.83:0/3953595925 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b1c003520 con 0x7f3b1c004800 2024-03-06T07:23:23.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.646+0000 7f3b27fff640 1 --2- 172.21.15.83:0/3953595925 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b1c004800 0x7f3b1c004c80 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f3b18009ac0 tx=0x7f3b1c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3d4d73165a74e76 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.646+0000 7f3b2dd9b640 1 -- 172.21.15.83:0/3953595925 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b18039070 con 0x7f3b1c004800 2024-03-06T07:23:23.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.646+0000 7f3b2dd9b640 1 -- 172.21.15.83:0/3953595925 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b1802c710 con 0x7f3b1c004800 2024-03-06T07:23:23.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.646+0000 7f3b2dd9b640 1 -- 172.21.15.83:0/3953595925 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b18009690 con 0x7f3b1c004800 2024-03-06T07:23:23.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.647+0000 7f3b34b5c640 1 -- 172.21.15.83:0/3953595925 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b1c004800 msgr2=0x7f3b1c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.647+0000 7f3b34b5c640 1 --2- 172.21.15.83:0/3953595925 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b1c004800 0x7f3b1c004c80 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f3b18009ac0 tx=0x7f3b1c005830 comp rx=0 tx=0).stop 2024-03-06T07:23:23.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.647+0000 7f3b2ed9d640 1 -- 172.21.15.83:0/3953595925 reap_dead start 2024-03-06T07:23:23.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.647+0000 7f3b34b5c640 1 -- 172.21.15.83:0/3953595925 shutdown_connections 2024-03-06T07:23:23.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.647+0000 7f3b34b5c640 1 -- 172.21.15.83:0/3953595925 >> 172.21.15.83:0/3953595925 conn(0x7f3b300d3cd0 msgr2=0x7f3b300d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:23.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.647+0000 7f3b34b5c640 1 -- 172.21.15.83:0/3953595925 shutdown_connections 2024-03-06T07:23:23.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.647+0000 7f3b34b5c640 1 -- 172.21.15.83:0/3953595925 wait complete. 2024-03-06T07:23:23.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.648+0000 7f3b34b5c640 1 Processor -- start 2024-03-06T07:23:23.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.648+0000 7f3b34b5c640 1 -- start start 2024-03-06T07:23:23.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.649+0000 7f3b34b5c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b1c004800 0x7f3b30160ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.649+0000 7f3b34b5c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3b3015e3c0 0x7f3b300d01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.649+0000 7f3b34b5c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3b3015abb0 0x7f3b3015b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.649+0000 7f3b34b5c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b300cfa00 con 0x7f3b1c004800 2024-03-06T07:23:23.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.649+0000 7f3b34b5c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b300b3d60 con 0x7f3b3015e3c0 2024-03-06T07:23:23.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.649+0000 7f3b34b5c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b300d1c70 con 0x7f3b3015abb0 2024-03-06T07:23:23.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.649+0000 7f3b2e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b1c004800 0x7f3b30160ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.649+0000 7f3b2e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b1c004800 0x7f3b30160ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56702/0 (socket says 172.21.15.83:56702) 2024-03-06T07:23:23.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.649+0000 7f3b2e59c640 1 -- 172.21.15.83:0/1393524189 learned_addr learned my addr 172.21.15.83:0/1393524189 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:23.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.649+0000 7f3b2ed9d640 1 --2- 172.21.15.83:0/1393524189 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3b3015abb0 0x7f3b3015b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.649+0000 7f3b27fff640 1 --2- 172.21.15.83:0/1393524189 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3b3015e3c0 0x7f3b300d01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.649+0000 7f3b2e59c640 1 -- 172.21.15.83:0/1393524189 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3b3015abb0 msgr2=0x7f3b3015b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.649+0000 7f3b2e59c640 1 --2- 172.21.15.83:0/1393524189 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3b3015abb0 0x7f3b3015b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.649+0000 7f3b2e59c640 1 -- 172.21.15.83:0/1393524189 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3b3015e3c0 msgr2=0x7f3b300d01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.649+0000 7f3b2e59c640 1 --2- 172.21.15.83:0/1393524189 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3b3015e3c0 0x7f3b300d01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.649+0000 7f3b2e59c640 1 -- 172.21.15.83:0/1393524189 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b3015b5a0 con 0x7f3b1c004800 2024-03-06T07:23:23.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.649+0000 7f3b2e59c640 1 --2- 172.21.15.83:0/1393524189 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b1c004800 0x7f3b30160ad0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f3b20007370 tx=0x7f3b20007830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.650+0000 7f3b26ffd640 1 -- 172.21.15.83:0/1393524189 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b20007e10 con 0x7f3b1c004800 2024-03-06T07:23:23.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.650+0000 7f3b26ffd640 1 -- 172.21.15.83:0/1393524189 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b20018070 con 0x7f3b1c004800 2024-03-06T07:23:23.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.650+0000 7f3b34b5c640 1 -- 172.21.15.83:0/1393524189 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b3015eba0 con 0x7f3b1c004800 2024-03-06T07:23:23.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.650+0000 7f3b26ffd640 1 -- 172.21.15.83:0/1393524189 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2000fc00 con 0x7f3b1c004800 2024-03-06T07:23:23.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.650+0000 7f3b34b5c640 1 -- 172.21.15.83:0/1393524189 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b3015f160 con 0x7f3b1c004800 2024-03-06T07:23:23.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.651+0000 7f3b34b5c640 1 -- 172.21.15.83:0/1393524189 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b3004f330 con 0x7f3b1c004800 2024-03-06T07:23:23.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.651+0000 7f3b26ffd640 1 -- 172.21.15.83:0/1393524189 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3b2001d460 con 0x7f3b1c004800 2024-03-06T07:23:23.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.651+0000 7f3b26ffd640 1 --2- 172.21.15.83:0/1393524189 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3b1006b4d0 0x7f3b1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.652+0000 7f3b26ffd640 1 -- 172.21.15.83:0/1393524189 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3b2008a110 con 0x7f3b1c004800 2024-03-06T07:23:23.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.652+0000 7f3b27fff640 1 --2- 172.21.15.83:0/1393524189 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3b1006b4d0 0x7f3b1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.652+0000 7f3b27fff640 1 --2- 172.21.15.83:0/1393524189 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3b1006b4d0 0x7f3b1006d990 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3b18036000 tx=0x7f3b18030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.656+0000 7f3b26ffd640 1 -- 172.21.15.83:0/1393524189 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3b20058940 con 0x7f3b1c004800 2024-03-06T07:23:23.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.871+0000 7f3b34b5c640 1 -- 172.21.15.83:0/1393524189 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b301e6b80 con 0x7f3b1c004800 2024-03-06T07:23:23.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.872+0000 7f3b26ffd640 1 -- 172.21.15.83:0/1393524189 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f3b2005c9e0 con 0x7f3b1c004800 2024-03-06T07:23:23.874 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:23:23.874 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4799,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/3542419057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3542419057},{"type":"v1","addr":"172.21.15.83:6836","nonce":3542419057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":5023,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1853869906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":1853869906},{"type":"v1","addr":"172.21.15.183:6835","nonce":1853869906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5026,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/3666667479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3666667479},{"type":"v1","addr":"172.21.15.183:6834","nonce":3666667479}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:00.145321+0000","modified":"2024-03-06T07:23:02.319419+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5244":{"gid":5244,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/2668925454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":2668925454},{"type":"v1","addr":"172.21.15.83:6837","nonce":2668925454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5244,"qdb_cluster":[5244]},"id":2}]} 2024-03-06T07:23:23.874 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 13 2024-03-06T07:23:23.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.874+0000 7f3b34b5c640 1 -- 172.21.15.83:0/1393524189 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3b1006b4d0 msgr2=0x7f3b1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.874+0000 7f3b34b5c640 1 --2- 172.21.15.83:0/1393524189 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3b1006b4d0 0x7f3b1006d990 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3b18036000 tx=0x7f3b18030000 comp rx=0 tx=0).stop 2024-03-06T07:23:23.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.874+0000 7f3b34b5c640 1 -- 172.21.15.83:0/1393524189 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b1c004800 msgr2=0x7f3b30160ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.874+0000 7f3b34b5c640 1 --2- 172.21.15.83:0/1393524189 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b1c004800 0x7f3b30160ad0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f3b20007370 tx=0x7f3b20007830 comp rx=0 tx=0).stop 2024-03-06T07:23:23.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.874+0000 7f3b34b5c640 1 -- 172.21.15.83:0/1393524189 shutdown_connections 2024-03-06T07:23:23.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.874+0000 7f3b34b5c640 1 --2- 172.21.15.83:0/1393524189 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3b1006b4d0 0x7f3b1006d990 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.874+0000 7f3b34b5c640 1 --2- 172.21.15.83:0/1393524189 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3b3015abb0 0x7f3b3015b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.874+0000 7f3b34b5c640 1 --2- 172.21.15.83:0/1393524189 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3b3015e3c0 0x7f3b300d01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.874+0000 7f3b34b5c640 1 --2- 172.21.15.83:0/1393524189 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b1c004800 0x7f3b30160ad0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.874+0000 7f3b34b5c640 1 -- 172.21.15.83:0/1393524189 >> 172.21.15.83:0/1393524189 conn(0x7f3b300d3cd0 msgr2=0x7f3b3015d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:23.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.874+0000 7f3b34b5c640 1 -- 172.21.15.83:0/1393524189 shutdown_connections 2024-03-06T07:23:23.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:23.874+0000 7f3b34b5c640 1 -- 172.21.15.83:0/1393524189 wait complete. 2024-03-06T07:23:23.890 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:23:24.057 INFO:teuthology.orchestra.run.smithi083.stdout:{"mds_cache":{"strays_enqueued":1001}} 2024-03-06T07:23:24.057 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1001 } } 2024-03-06T07:23:24.057 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:24.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.188+0000 7f2ba8ac9640 1 Processor -- start 2024-03-06T07:23:24.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.188+0000 7f2ba8ac9640 1 -- start start 2024-03-06T07:23:24.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.189+0000 7f2ba8ac9640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f2ba40b4170 0x7f2ba40b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.189+0000 7f2ba8ac9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2ba40d4760 0x7f2ba40d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.189+0000 7f2ba8ac9640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ba40d25d0 con 0x7f2ba40b4dd0 2024-03-06T07:23:24.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.189+0000 7f2ba8ac9640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ba40d4bf0 con 0x7f2ba40d4760 2024-03-06T07:23:24.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.189+0000 7f2ba8ac9640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ba40ce9e0 con 0x7f2ba40b4170 2024-03-06T07:23:24.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.189+0000 7f2b9bfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2ba40d4760 0x7f2ba40d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.189+0000 7f2b9bfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2ba40d4760 0x7f2ba40d0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56714/0 (socket says 172.21.15.83:56714) 2024-03-06T07:23:24.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.189+0000 7f2b9bfff640 1 -- 172.21.15.83:0/2346526103 learned_addr learned my addr 172.21.15.83:0/2346526103 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:24.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.189+0000 7f2ba2d9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f2ba40b4dd0 0x7f2ba40cee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:44648/0 (socket says 172.21.15.83:44648) 2024-03-06T07:23:24.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.189+0000 7f2ba259c640 1 --2- 172.21.15.83:0/2346526103 >> v2:172.21.15.183:3300/0 conn(0x7f2ba40b4170 0x7f2ba40b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.189+0000 7f2b9bfff640 1 -- 172.21.15.83:0/2346526103 >> v2:172.21.15.183:3300/0 conn(0x7f2ba40b4170 msgr2=0x7f2ba40b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:24.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.189+0000 7f2b9bfff640 1 --2- 172.21.15.83:0/2346526103 >> v2:172.21.15.183:3300/0 conn(0x7f2ba40b4170 0x7f2ba40b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.189+0000 7f2b9bfff640 1 -- 172.21.15.83:0/2346526103 >> v1:172.21.15.83:6789/0 conn(0x7f2ba40b4dd0 legacy=0x7f2ba40cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.189+0000 7f2ba1d9b640 1 -- 172.21.15.83:0/2346526103 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3663238580 0 0) 0x7f2ba40d25d0 con 0x7f2ba40b4dd0 2024-03-06T07:23:24.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.189+0000 7f2b9bfff640 1 -- 172.21.15.83:0/2346526103 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba40d2870 con 0x7f2ba40d4760 2024-03-06T07:23:24.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.189+0000 7f2ba259c640 1 --2- 172.21.15.83:0/2346526103 >> v2:172.21.15.183:3300/0 conn(0x7f2ba40b4170 0x7f2ba40b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:24.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.190+0000 7f2b9bfff640 1 --2- 172.21.15.83:0/2346526103 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ba40d4760 0x7f2ba40d0370 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f2ba406e030 tx=0x7f2b8c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8f914d64dac425f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.190+0000 7f2ba1d9b640 1 -- 172.21.15.83:0/2346526103 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b8c034070 con 0x7f2ba40d4760 2024-03-06T07:23:24.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.190+0000 7f2ba1d9b640 1 -- 172.21.15.83:0/2346526103 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b8c030860 con 0x7f2ba40d4760 2024-03-06T07:23:24.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.190+0000 7f2ba1d9b640 1 -- 172.21.15.83:0/2346526103 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b8c030c30 con 0x7f2ba40d4760 2024-03-06T07:23:24.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.190+0000 7f2ba8ac9640 1 -- 172.21.15.83:0/2346526103 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ba40d4760 msgr2=0x7f2ba40d0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.190+0000 7f2ba8ac9640 1 --2- 172.21.15.83:0/2346526103 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ba40d4760 0x7f2ba40d0370 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f2ba406e030 tx=0x7f2b8c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.191+0000 7f2ba8ac9640 1 -- 172.21.15.83:0/2346526103 shutdown_connections 2024-03-06T07:23:24.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.191+0000 7f2ba8ac9640 1 --2- 172.21.15.83:0/2346526103 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ba40d4760 0x7f2ba40d0370 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.191+0000 7f2ba8ac9640 1 --2- 172.21.15.83:0/2346526103 >> v2:172.21.15.183:3300/0 conn(0x7f2ba40b4170 0x7f2ba40b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.191+0000 7f2ba8ac9640 1 -- 172.21.15.83:0/2346526103 >> 172.21.15.83:0/2346526103 conn(0x7f2ba40d3c40 msgr2=0x7f2ba40d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:24.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.191+0000 7f2ba8ac9640 1 -- 172.21.15.83:0/2346526103 shutdown_connections 2024-03-06T07:23:24.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.191+0000 7f2ba8ac9640 1 -- 172.21.15.83:0/2346526103 wait complete. 2024-03-06T07:23:24.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.192+0000 7f2ba8ac9640 1 Processor -- start 2024-03-06T07:23:24.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.192+0000 7f2ba8ac9640 1 -- start start 2024-03-06T07:23:24.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.193+0000 7f2ba8ac9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2ba40b4170 0x7f2ba415cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.193+0000 7f2ba8ac9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2ba40b4dd0 0x7f2ba41f39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.193+0000 7f2ba8ac9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ba40c7f40 0x7f2ba40b88f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.193+0000 7f2ba8ac9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba40b47d0 con 0x7f2ba40c7f40 2024-03-06T07:23:24.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.193+0000 7f2ba8ac9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba40b4550 con 0x7f2ba40b4170 2024-03-06T07:23:24.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.193+0000 7f2ba8ac9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba40b4950 con 0x7f2ba40b4dd0 2024-03-06T07:23:24.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.193+0000 7f2ba2d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ba40c7f40 0x7f2ba40b88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.193+0000 7f2ba2d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ba40c7f40 0x7f2ba40b88f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56720/0 (socket says 172.21.15.83:56720) 2024-03-06T07:23:24.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.193+0000 7f2ba2d9d640 1 -- 172.21.15.83:0/1160323015 learned_addr learned my addr 172.21.15.83:0/1160323015 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:24.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.193+0000 7f2ba259c640 1 --2- 172.21.15.83:0/1160323015 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2ba40b4170 0x7f2ba415cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.193+0000 7f2b9bfff640 1 --2- 172.21.15.83:0/1160323015 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2ba40b4dd0 0x7f2ba41f39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.193+0000 7f2ba2d9d640 1 -- 172.21.15.83:0/1160323015 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2ba40b4dd0 msgr2=0x7f2ba41f39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.193+0000 7f2ba2d9d640 1 --2- 172.21.15.83:0/1160323015 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2ba40b4dd0 0x7f2ba41f39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.193+0000 7f2ba2d9d640 1 -- 172.21.15.83:0/1160323015 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2ba40b4170 msgr2=0x7f2ba415cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.193+0000 7f2ba2d9d640 1 --2- 172.21.15.83:0/1160323015 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2ba40b4170 0x7f2ba415cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.193+0000 7f2ba2d9d640 1 -- 172.21.15.83:0/1160323015 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba41f4280 con 0x7f2ba40c7f40 2024-03-06T07:23:24.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.193+0000 7f2ba2d9d640 1 --2- 172.21.15.83:0/1160323015 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ba40c7f40 0x7f2ba40b88f0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c00c7a0 tx=0x7f2b9c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.193+0000 7f2b9affd640 1 -- 172.21.15.83:0/1160323015 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b9c018070 con 0x7f2ba40c7f40 2024-03-06T07:23:24.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.193+0000 7f2b9affd640 1 -- 172.21.15.83:0/1160323015 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b9c00a5e0 con 0x7f2ba40c7f40 2024-03-06T07:23:24.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.193+0000 7f2ba8ac9640 1 -- 172.21.15.83:0/1160323015 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ba41f4560 con 0x7f2ba40c7f40 2024-03-06T07:23:24.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.193+0000 7f2b9affd640 1 -- 172.21.15.83:0/1160323015 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b9c00aa50 con 0x7f2ba40c7f40 2024-03-06T07:23:24.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.193+0000 7f2ba8ac9640 1 -- 172.21.15.83:0/1160323015 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ba40c7a20 con 0x7f2ba40c7f40 2024-03-06T07:23:24.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.194+0000 7f2ba8ac9640 1 -- 172.21.15.83:0/1160323015 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ba404f330 con 0x7f2ba40c7f40 2024-03-06T07:23:24.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.195+0000 7f2b9affd640 1 -- 172.21.15.83:0/1160323015 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2b9c01b070 con 0x7f2ba40c7f40 2024-03-06T07:23:24.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.195+0000 7f2b9affd640 1 --2- 172.21.15.83:0/1160323015 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2b8406b500 0x7f2b8406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.195+0000 7f2b9affd640 1 -- 172.21.15.83:0/1160323015 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2b9c089720 con 0x7f2ba40c7f40 2024-03-06T07:23:24.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.195+0000 7f2ba259c640 1 --2- 172.21.15.83:0/1160323015 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2b8406b500 0x7f2b8406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.196+0000 7f2ba259c640 1 --2- 172.21.15.83:0/1160323015 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2b8406b500 0x7f2b8406d9c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2b9400a990 tx=0x7f2b94008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.200+0000 7f2b9affd640 1 -- 172.21.15.83:0/1160323015 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2b9c058630 con 0x7f2ba40c7f40 2024-03-06T07:23:24.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.409+0000 7f2ba8ac9640 1 -- 172.21.15.83:0/1160323015 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ba40733b0 con 0x7f2ba40c7f40 2024-03-06T07:23:24.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.409+0000 7f2b9affd640 1 -- 172.21.15.83:0/1160323015 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f2b9c057f30 con 0x7f2ba40c7f40 2024-03-06T07:23:24.411 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:23:24.412 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4799,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/3542419057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3542419057},{"type":"v1","addr":"172.21.15.83:6836","nonce":3542419057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":5023,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1853869906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":1853869906},{"type":"v1","addr":"172.21.15.183:6835","nonce":1853869906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5026,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/3666667479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3666667479},{"type":"v1","addr":"172.21.15.183:6834","nonce":3666667479}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:00.145321+0000","modified":"2024-03-06T07:23:02.319419+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5244":{"gid":5244,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/2668925454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":2668925454},{"type":"v1","addr":"172.21.15.83:6837","nonce":2668925454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5244,"qdb_cluster":[5244]},"id":2}]} 2024-03-06T07:23:24.412 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 13 2024-03-06T07:23:24.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.412+0000 7f2ba8ac9640 1 -- 172.21.15.83:0/1160323015 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2b8406b500 msgr2=0x7f2b8406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.412+0000 7f2ba8ac9640 1 --2- 172.21.15.83:0/1160323015 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2b8406b500 0x7f2b8406d9c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2b9400a990 tx=0x7f2b94008040 comp rx=0 tx=0).stop 2024-03-06T07:23:24.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.412+0000 7f2ba8ac9640 1 -- 172.21.15.83:0/1160323015 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ba40c7f40 msgr2=0x7f2ba40b88f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.412+0000 7f2ba8ac9640 1 --2- 172.21.15.83:0/1160323015 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ba40c7f40 0x7f2ba40b88f0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c00c7a0 tx=0x7f2b9c00cc60 comp rx=0 tx=0).stop 2024-03-06T07:23:24.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.412+0000 7f2ba8ac9640 1 -- 172.21.15.83:0/1160323015 shutdown_connections 2024-03-06T07:23:24.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.412+0000 7f2ba8ac9640 1 --2- 172.21.15.83:0/1160323015 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2b8406b500 0x7f2b8406d9c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.412+0000 7f2ba8ac9640 1 --2- 172.21.15.83:0/1160323015 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ba40c7f40 0x7f2ba40b88f0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.412+0000 7f2ba8ac9640 1 --2- 172.21.15.83:0/1160323015 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2ba40b4dd0 0x7f2ba41f39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.412+0000 7f2ba8ac9640 1 --2- 172.21.15.83:0/1160323015 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2ba40b4170 0x7f2ba415cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.412+0000 7f2ba8ac9640 1 -- 172.21.15.83:0/1160323015 >> 172.21.15.83:0/1160323015 conn(0x7f2ba40d3c40 msgr2=0x7f2ba406cfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:24.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.412+0000 7f2ba8ac9640 1 -- 172.21.15.83:0/1160323015 shutdown_connections 2024-03-06T07:23:24.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.412+0000 7f2ba8ac9640 1 -- 172.21.15.83:0/1160323015 wait complete. 2024-03-06T07:23:24.428 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump purge_queue pq_executed 2024-03-06T07:23:24.587 INFO:teuthology.orchestra.run.smithi083.stdout:{"purge_queue":{"pq_executed":1001}} 2024-03-06T07:23:24.587 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1001 } } 2024-03-06T07:23:24.587 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 10000000000.00000000 2024-03-06T07:23:24.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.626+0000 7f7d3be45f40 1 Processor -- start 2024-03-06T07:23:24.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.626+0000 7f7d3be45f40 1 -- start start 2024-03-06T07:23:24.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.627+0000 7f7d3be45f40 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561e305b2d80 0x561e3063ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.627+0000 7f7d3be45f40 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x561e3063cf30 0x561e3063b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.627+0000 7f7d3be45f40 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561e30566e20 0x561e305974e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.627+0000 7f7d3be45f40 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e305b0700 con 0x561e30566e20 2024-03-06T07:23:24.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.627+0000 7f7d3be45f40 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e3050eb70 con 0x561e305b2d80 2024-03-06T07:23:24.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.627+0000 7f7d3be45f40 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x561e305b01c0 con 0x561e3063cf30 2024-03-06T07:23:24.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.627+0000 7f7d399f0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561e30566e20 0x561e305974e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.627+0000 7f7d399f0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561e30566e20 0x561e305974e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56730/0 (socket says 172.21.15.83:56730) 2024-03-06T07:23:24.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.627+0000 7f7d399f0640 1 -- 172.21.15.83:0/4107422830 learned_addr learned my addr 172.21.15.83:0/4107422830 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:24.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.627+0000 7f7d399f0640 1 -- 172.21.15.83:0/4107422830 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x561e3063cf30 msgr2=0x561e3063b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:24.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.627+0000 7f7d399f0640 1 --2- 172.21.15.83:0/4107422830 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x561e3063cf30 0x561e3063b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.627+0000 7f7d399f0640 1 -- 172.21.15.83:0/4107422830 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561e305b2d80 msgr2=0x561e3063ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:24.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.627+0000 7f7d399f0640 1 --2- 172.21.15.83:0/4107422830 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561e305b2d80 0x561e3063ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.627+0000 7f7d399f0640 1 -- 172.21.15.83:0/4107422830 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561e305aea10 con 0x561e30566e20 2024-03-06T07:23:24.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.627+0000 7f7d399f0640 1 --2- 172.21.15.83:0/4107422830 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561e30566e20 0x561e305974e0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f7d3400a8e0 tx=0x7f7d34005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53e076561b71937e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.628+0000 7f7d2b7fe640 1 -- 172.21.15.83:0/4107422830 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3400d9c0 con 0x561e30566e20 2024-03-06T07:23:24.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.628+0000 7f7d2b7fe640 1 -- 172.21.15.83:0/4107422830 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d3400db60 con 0x561e30566e20 2024-03-06T07:23:24.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.628+0000 7f7d2b7fe640 1 -- 172.21.15.83:0/4107422830 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d34012450 con 0x561e30566e20 2024-03-06T07:23:24.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.628+0000 7f7d3be45f40 1 -- 172.21.15.83:0/4107422830 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561e30566e20 msgr2=0x561e305974e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.628+0000 7f7d3be45f40 1 --2- 172.21.15.83:0/4107422830 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561e30566e20 0x561e305974e0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f7d3400a8e0 tx=0x7f7d34005d80 comp rx=0 tx=0).stop 2024-03-06T07:23:24.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.628+0000 7f7d3be45f40 1 -- 172.21.15.83:0/4107422830 shutdown_connections 2024-03-06T07:23:24.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.628+0000 7f7d3be45f40 1 --2- 172.21.15.83:0/4107422830 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561e30566e20 0x561e305974e0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.628+0000 7f7d3be45f40 1 --2- 172.21.15.83:0/4107422830 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x561e3063cf30 0x561e3063b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.628+0000 7f7d3be45f40 1 --2- 172.21.15.83:0/4107422830 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561e305b2d80 0x561e3063ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.628+0000 7f7d3be45f40 1 -- 172.21.15.83:0/4107422830 >> 172.21.15.83:0/4107422830 conn(0x561e305668b0 msgr2=0x561e3063d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:24.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.628+0000 7f7d3be45f40 1 -- 172.21.15.83:0/4107422830 shutdown_connections 2024-03-06T07:23:24.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.628+0000 7f7d3be45f40 1 -- 172.21.15.83:0/4107422830 wait complete. 2024-03-06T07:23:24.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.629+0000 7f7d3be45f40 1 Processor -- start 2024-03-06T07:23:24.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.629+0000 7f7d3be45f40 1 -- start start 2024-03-06T07:23:24.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.630+0000 7f7d3be45f40 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561e30566e20 0x561e30628350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.630+0000 7f7d3be45f40 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x561e305b2d80 0x561e305ad300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.630+0000 7f7d3be45f40 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561e3063cf30 0x561e3063acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.630+0000 7f7d3be45f40 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e305aecb0 con 0x561e3063cf30 2024-03-06T07:23:24.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.630+0000 7f7d3be45f40 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e30593dd0 con 0x561e30566e20 2024-03-06T07:23:24.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.630+0000 7f7d3be45f40 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x561e30638860 con 0x561e305b2d80 2024-03-06T07:23:24.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.630+0000 7f7d399f0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561e3063cf30 0x561e3063acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.630+0000 7f7d399f0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561e3063cf30 0x561e3063acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56734/0 (socket says 172.21.15.83:56734) 2024-03-06T07:23:24.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.630+0000 7f7d399f0640 1 -- 172.21.15.83:0/1640033342 learned_addr learned my addr 172.21.15.83:0/1640033342 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:24.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.630+0000 7f7d399f0640 1 -- 172.21.15.83:0/1640033342 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x561e305b2d80 msgr2=0x561e305ad300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.630+0000 7f7d389ee640 1 --2- 172.21.15.83:0/1640033342 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x561e305b2d80 0x561e305ad300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.630+0000 7f7d399f0640 1 --2- 172.21.15.83:0/1640033342 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x561e305b2d80 0x561e305ad300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.630+0000 7f7d391ef640 1 --2- 172.21.15.83:0/1640033342 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561e30566e20 0x561e30628350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.630+0000 7f7d399f0640 1 -- 172.21.15.83:0/1640033342 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561e30566e20 msgr2=0x561e30628350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.630+0000 7f7d399f0640 1 --2- 172.21.15.83:0/1640033342 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561e30566e20 0x561e30628350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.630+0000 7f7d399f0640 1 -- 172.21.15.83:0/1640033342 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561e30569260 con 0x561e3063cf30 2024-03-06T07:23:24.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.630+0000 7f7d399f0640 1 --2- 172.21.15.83:0/1640033342 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561e3063cf30 0x561e3063acd0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f7d34005dc0 tx=0x7f7d340033a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.630+0000 7f7d2a7fc640 1 -- 172.21.15.83:0/1640033342 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3400d9c0 con 0x561e3063cf30 2024-03-06T07:23:24.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.630+0000 7f7d2a7fc640 1 -- 172.21.15.83:0/1640033342 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d3401d660 con 0x561e3063cf30 2024-03-06T07:23:24.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.630+0000 7f7d2a7fc640 1 -- 172.21.15.83:0/1640033342 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3401da50 con 0x561e3063cf30 2024-03-06T07:23:24.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.630+0000 7f7d3be45f40 1 -- 172.21.15.83:0/1640033342 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x561e30569540 con 0x561e3063cf30 2024-03-06T07:23:24.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.630+0000 7f7d3be45f40 1 -- 172.21.15.83:0/1640033342 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561e305ad840 con 0x561e3063cf30 2024-03-06T07:23:24.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:23:24.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.632+0000 7f7d2a7fc640 1 -- 172.21.15.83:0/1640033342 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7d3400dd10 con 0x561e3063cf30 2024-03-06T07:23:24.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.632+0000 7f7d2a7fc640 1 --2- 172.21.15.83:0/1640033342 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7d1c069eb0 0x7f7d1c06c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.632+0000 7f7d2a7fc640 1 -- 172.21.15.83:0/1640033342 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7d3408b610 con 0x561e3063cf30 2024-03-06T07:23:24.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.632+0000 7f7d391ef640 1 --2- 172.21.15.83:0/1640033342 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7d1c069eb0 0x7f7d1c06c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.632+0000 7f7d3be45f40 1 --2- 172.21.15.83:0/1640033342 >> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] conn(0x561e306d8ed0 0x561e306d92f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.632+0000 7f7d3be45f40 1 -- 172.21.15.83:0/1640033342 --> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] -- osd_op(unknown.0.0:1 4.10 4:0cf6ad0f:::10000000000.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33) v9 -- 0x561e306d9830 con 0x561e306d8ed0 2024-03-06T07:23:24.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.632+0000 7f7d389ee640 1 --2- 172.21.15.83:0/1640033342 >> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] conn(0x561e306d8ed0 0x561e306d92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.632+0000 7f7d391ef640 1 --2- 172.21.15.83:0/1640033342 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7d1c069eb0 0x7f7d1c06c370 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c001cf0 tx=0x7f7d2c001310 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.633+0000 7f7d389ee640 1 --2- 172.21.15.83:0/1640033342 >> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] conn(0x561e306d8ed0 0x561e306d92f0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.633+0000 7f7d389ee640 1 -- 172.21.15.83:0/1640033342 <== osd.3 v2:172.21.15.83:6815/2656563756 1 ==== osd_op_reply(1 10000000000.00000000 [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 164+0+0 (crc 0 0 0) 0x7f7d3005d040 con 0x561e306d8ed0 2024-03-06T07:23:24.640 INFO:teuthology.orchestra.run.smithi083.stderr: error stat-ing cephfs_metadata/10000000000.00000000: (2) No such file or directory 2024-03-06T07:23:24.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.633+0000 7f7d3be45f40 1 -- 172.21.15.83:0/1640033342 >> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] conn(0x561e306d8ed0 msgr2=0x561e306d92f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.633+0000 7f7d3be45f40 1 --2- 172.21.15.83:0/1640033342 >> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] conn(0x561e306d8ed0 0x561e306d92f0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.633+0000 7f7d3be45f40 1 -- 172.21.15.83:0/1640033342 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7d1c069eb0 msgr2=0x7f7d1c06c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.633+0000 7f7d3be45f40 1 --2- 172.21.15.83:0/1640033342 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7d1c069eb0 0x7f7d1c06c370 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c001cf0 tx=0x7f7d2c001310 comp rx=0 tx=0).stop 2024-03-06T07:23:24.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.633+0000 7f7d3be45f40 1 -- 172.21.15.83:0/1640033342 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561e3063cf30 msgr2=0x561e3063acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.633+0000 7f7d3be45f40 1 --2- 172.21.15.83:0/1640033342 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561e3063cf30 0x561e3063acd0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f7d34005dc0 tx=0x7f7d340033a0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.633+0000 7f7d399f0640 1 -- 172.21.15.83:0/1640033342 reap_dead start 2024-03-06T07:23:24.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.633+0000 7f7d3be45f40 1 -- 172.21.15.83:0/1640033342 shutdown_connections 2024-03-06T07:23:24.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.633+0000 7f7d3be45f40 1 -- 172.21.15.83:0/1640033342 >> 172.21.15.83:0/1640033342 conn(0x561e305668b0 msgr2=0x561e305aeee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:24.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.633+0000 7f7d3be45f40 1 -- 172.21.15.83:0/1640033342 shutdown_connections 2024-03-06T07:23:24.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.633+0000 7f7d3be45f40 1 -- 172.21.15.83:0/1640033342 wait complete. 2024-03-06T07:23:24.642 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2024-03-06T07:23:24.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.781+0000 7f9f9046b640 1 Processor -- start 2024-03-06T07:23:24.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.781+0000 7f9f9046b640 1 -- start start 2024-03-06T07:23:24.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.782+0000 7f9f9046b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9f880b4460 0x7f9f880d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.782+0000 7f9f9046b640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f9f88162970 0x7f9f880b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.782+0000 7f9f9046b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9f88161cf0 0x7f9f88162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.782+0000 7f9f9046b640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f88161b00 con 0x7f9f880b4460 2024-03-06T07:23:24.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.782+0000 7f9f9046b640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f880cecc0 con 0x7f9f88162970 2024-03-06T07:23:24.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.782+0000 7f9f9046b640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f880d22c0 con 0x7f9f88161cf0 2024-03-06T07:23:24.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.782+0000 7f9f8e207640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9f880b4460 0x7f9f880d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.782+0000 7f9f8e207640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9f880b4460 0x7f9f880d4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56750/0 (socket says 172.21.15.83:56750) 2024-03-06T07:23:24.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.782+0000 7f9f8e207640 1 -- 172.21.15.83:0/513183211 learned_addr learned my addr 172.21.15.83:0/513183211 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:24.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.782+0000 7f9f8ea08640 1 --2- 172.21.15.83:0/513183211 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9f88161cf0 0x7f9f88162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.782+0000 7f9f8e207640 1 -- 172.21.15.83:0/513183211 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9f88161cf0 msgr2=0x7f9f88162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:24.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.782+0000 7f9f8e207640 1 --2- 172.21.15.83:0/513183211 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9f88161cf0 0x7f9f88162180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.782+0000 7f9f8e207640 1 -- 172.21.15.83:0/513183211 >> v2:172.21.15.183:3300/0 conn(0x7f9f88162970 msgr2=0x7f9f880b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:24.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.782+0000 7f9f8da06640 1 --2- 172.21.15.83:0/513183211 >> v2:172.21.15.183:3300/0 conn(0x7f9f88162970 0x7f9f880b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.782+0000 7f9f8e207640 1 --2- 172.21.15.83:0/513183211 >> v2:172.21.15.183:3300/0 conn(0x7f9f88162970 0x7f9f880b3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.782+0000 7f9f8e207640 1 -- 172.21.15.83:0/513183211 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f880d25a0 con 0x7f9f880b4460 2024-03-06T07:23:24.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.783+0000 7f9f8e207640 1 --2- 172.21.15.83:0/513183211 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f880b4460 0x7f9f880d4800 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f9f880d17e0 tx=0x7f9f7c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53d603e73fddf403 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.783+0000 7f9f8d205640 1 -- 172.21.15.83:0/513183211 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f7c035020 con 0x7f9f880b4460 2024-03-06T07:23:24.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.783+0000 7f9f8d205640 1 -- 172.21.15.83:0/513183211 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f7c031870 con 0x7f9f880b4460 2024-03-06T07:23:24.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.783+0000 7f9f8d205640 1 -- 172.21.15.83:0/513183211 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f7c031c40 con 0x7f9f880b4460 2024-03-06T07:23:24.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.784+0000 7f9f9046b640 1 -- 172.21.15.83:0/513183211 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f880b4460 msgr2=0x7f9f880d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.784+0000 7f9f9046b640 1 --2- 172.21.15.83:0/513183211 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f880b4460 0x7f9f880d4800 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f9f880d17e0 tx=0x7f9f7c02b910 comp rx=0 tx=0).stop 2024-03-06T07:23:24.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.784+0000 7f9f9046b640 1 -- 172.21.15.83:0/513183211 shutdown_connections 2024-03-06T07:23:24.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.784+0000 7f9f9046b640 1 --2- 172.21.15.83:0/513183211 >> v2:172.21.15.183:3300/0 conn(0x7f9f88162970 0x7f9f880b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.784+0000 7f9f9046b640 1 --2- 172.21.15.83:0/513183211 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9f88161cf0 0x7f9f88162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.784+0000 7f9f9046b640 1 --2- 172.21.15.83:0/513183211 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f880b4460 0x7f9f880d4800 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.784+0000 7f9f9046b640 1 -- 172.21.15.83:0/513183211 >> 172.21.15.83:0/513183211 conn(0x7f9f880d3ce0 msgr2=0x7f9f880d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:24.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.784+0000 7f9f9046b640 1 -- 172.21.15.83:0/513183211 shutdown_connections 2024-03-06T07:23:24.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.784+0000 7f9f9046b640 1 -- 172.21.15.83:0/513183211 wait complete. 2024-03-06T07:23:24.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.785+0000 7f9f9046b640 1 Processor -- start 2024-03-06T07:23:24.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.785+0000 7f9f9046b640 1 -- start start 2024-03-06T07:23:24.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.787+0000 7f9f9046b640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9f880b4460 0x7f9f88155f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.787+0000 7f9f9046b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f88161cf0 0x7f9f880c9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.787+0000 7f9f9046b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9f88162970 0x7f9f880b6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.787+0000 7f9f9046b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f880ce950 con 0x7f9f88161cf0 2024-03-06T07:23:24.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.787+0000 7f9f9046b640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f880d2b60 con 0x7f9f880b4460 2024-03-06T07:23:24.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.787+0000 7f9f9046b640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f880b3ac0 con 0x7f9f88162970 2024-03-06T07:23:24.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.787+0000 7f9f8da06640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f88161cf0 0x7f9f880c9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.787+0000 7f9f8da06640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f88161cf0 0x7f9f880c9bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56764/0 (socket says 172.21.15.83:56764) 2024-03-06T07:23:24.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.787+0000 7f9f8da06640 1 -- 172.21.15.83:0/1486678927 learned_addr learned my addr 172.21.15.83:0/1486678927 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:24.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.787+0000 7f9f8e207640 1 --2- 172.21.15.83:0/1486678927 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9f880b4460 0x7f9f88155f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.787+0000 7f9f8ea08640 1 --2- 172.21.15.83:0/1486678927 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9f88162970 0x7f9f880b6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.787+0000 7f9f8da06640 1 -- 172.21.15.83:0/1486678927 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9f88162970 msgr2=0x7f9f880b6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.787+0000 7f9f8da06640 1 --2- 172.21.15.83:0/1486678927 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9f88162970 0x7f9f880b6900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.787+0000 7f9f8da06640 1 -- 172.21.15.83:0/1486678927 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9f880b4460 msgr2=0x7f9f88155f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.787+0000 7f9f8da06640 1 --2- 172.21.15.83:0/1486678927 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9f880b4460 0x7f9f88155f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.787+0000 7f9f8da06640 1 -- 172.21.15.83:0/1486678927 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f880cb700 con 0x7f9f88161cf0 2024-03-06T07:23:24.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.787+0000 7f9f8da06640 1 --2- 172.21.15.83:0/1486678927 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f88161cf0 0x7f9f880c9bb0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f9f8400c7f0 tx=0x7f9f8400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.787+0000 7f9f7affd640 1 -- 172.21.15.83:0/1486678927 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f84015070 con 0x7f9f88161cf0 2024-03-06T07:23:24.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.788+0000 7f9f7affd640 1 -- 172.21.15.83:0/1486678927 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f84004220 con 0x7f9f88161cf0 2024-03-06T07:23:24.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.788+0000 7f9f9046b640 1 -- 172.21.15.83:0/1486678927 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f880b8b10 con 0x7f9f88161cf0 2024-03-06T07:23:24.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.788+0000 7f9f7affd640 1 -- 172.21.15.83:0/1486678927 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f84007620 con 0x7f9f88161cf0 2024-03-06T07:23:24.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.788+0000 7f9f9046b640 1 -- 172.21.15.83:0/1486678927 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f880c78b0 con 0x7f9f88161cf0 2024-03-06T07:23:24.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.789+0000 7f9f9046b640 1 -- 172.21.15.83:0/1486678927 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f8808da10 con 0x7f9f88161cf0 2024-03-06T07:23:24.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.789+0000 7f9f7affd640 1 -- 172.21.15.83:0/1486678927 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9f840079d0 con 0x7f9f88161cf0 2024-03-06T07:23:24.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.789+0000 7f9f7affd640 1 --2- 172.21.15.83:0/1486678927 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9f6c06b4d0 0x7f9f6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.790+0000 7f9f7affd640 1 -- 172.21.15.83:0/1486678927 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9f8408a3f0 con 0x7f9f88161cf0 2024-03-06T07:23:24.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.790+0000 7f9f8e207640 1 --2- 172.21.15.83:0/1486678927 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9f6c06b4d0 0x7f9f6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.790+0000 7f9f8e207640 1 --2- 172.21.15.83:0/1486678927 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9f6c06b4d0 0x7f9f6c06d990 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c036000 tx=0x7f9f7c03c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:24.794+0000 7f9f7affd640 1 -- 172.21.15.83:0/1486678927 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9f8808da10 con 0x7f9f88161cf0 2024-03-06T07:23:25.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.042+0000 7f9f9046b640 1 -- 172.21.15.83:0/1486678927 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7f9f8804f330 con 0x7f9f88161cf0 2024-03-06T07:23:25.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.042+0000 7f9f7affd640 1 -- 172.21.15.83:0/1486678927 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1656 (secure 0 0 0) 0x7f9f880cb700 con 0x7f9f88161cf0 2024-03-06T07:23:25.044 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:23:25.044 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:23:25.044 INFO:teuthology.orchestra.run.smithi083.stdout: "stats": { 2024-03-06T07:23:25.044 INFO:teuthology.orchestra.run.smithi083.stdout: "total_bytes": 773094113280, 2024-03-06T07:23:25.044 INFO:teuthology.orchestra.run.smithi083.stdout: "total_avail_bytes": 772818747392, 2024-03-06T07:23:25.044 INFO:teuthology.orchestra.run.smithi083.stdout: "total_used_bytes": 275365888, 2024-03-06T07:23:25.044 INFO:teuthology.orchestra.run.smithi083.stdout: "total_used_raw_bytes": 275365888, 2024-03-06T07:23:25.045 INFO:teuthology.orchestra.run.smithi083.stdout: "total_used_raw_ratio": 0.00035618676338344812, 2024-03-06T07:23:25.045 INFO:teuthology.orchestra.run.smithi083.stdout: "num_osds": 8, 2024-03-06T07:23:25.045 INFO:teuthology.orchestra.run.smithi083.stdout: "num_per_pool_osds": 8, 2024-03-06T07:23:25.045 INFO:teuthology.orchestra.run.smithi083.stdout: "num_per_pool_omap_osds": 8 2024-03-06T07:23:25.045 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:23:25.045 INFO:teuthology.orchestra.run.smithi083.stdout: "stats_by_class": { 2024-03-06T07:23:25.045 INFO:teuthology.orchestra.run.smithi083.stdout: "ssd": { 2024-03-06T07:23:25.045 INFO:teuthology.orchestra.run.smithi083.stdout: "total_bytes": 773094113280, 2024-03-06T07:23:25.045 INFO:teuthology.orchestra.run.smithi083.stdout: "total_avail_bytes": 772818747392, 2024-03-06T07:23:25.045 INFO:teuthology.orchestra.run.smithi083.stdout: "total_used_bytes": 275365888, 2024-03-06T07:23:25.045 INFO:teuthology.orchestra.run.smithi083.stdout: "total_used_raw_bytes": 275365888, 2024-03-06T07:23:25.045 INFO:teuthology.orchestra.run.smithi083.stdout: "total_used_raw_ratio": 0.00035618676338344812 2024-03-06T07:23:25.045 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:23:25.045 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:23:25.046 INFO:teuthology.orchestra.run.smithi083.stdout: "pools": [ 2024-03-06T07:23:25.046 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:23:25.046 INFO:teuthology.orchestra.run.smithi083.stdout: "name": ".mgr", 2024-03-06T07:23:25.046 INFO:teuthology.orchestra.run.smithi083.stdout: "id": 1, 2024-03-06T07:23:25.046 INFO:teuthology.orchestra.run.smithi083.stdout: "stats": { 2024-03-06T07:23:25.046 INFO:teuthology.orchestra.run.smithi083.stdout: "stored": 590368, 2024-03-06T07:23:25.046 INFO:teuthology.orchestra.run.smithi083.stdout: "objects": 2, 2024-03-06T07:23:25.046 INFO:teuthology.orchestra.run.smithi083.stdout: "kb_used": 1160, 2024-03-06T07:23:25.046 INFO:teuthology.orchestra.run.smithi083.stdout: "bytes_used": 1187840, 2024-03-06T07:23:25.046 INFO:teuthology.orchestra.run.smithi083.stdout: "percent_used": 1.618046326257172e-06, 2024-03-06T07:23:25.046 INFO:teuthology.orchestra.run.smithi083.stdout: "max_avail": 367059369984 2024-03-06T07:23:25.046 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:23:25.046 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:23:25.046 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:23:25.047 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "cephfs_metadata", 2024-03-06T07:23:25.047 INFO:teuthology.orchestra.run.smithi083.stdout: "id": 4, 2024-03-06T07:23:25.047 INFO:teuthology.orchestra.run.smithi083.stdout: "stats": { 2024-03-06T07:23:25.047 INFO:teuthology.orchestra.run.smithi083.stdout: "stored": 3677172, 2024-03-06T07:23:25.047 INFO:teuthology.orchestra.run.smithi083.stdout: "objects": 23, 2024-03-06T07:23:25.047 INFO:teuthology.orchestra.run.smithi083.stdout: "kb_used": 7240, 2024-03-06T07:23:25.047 INFO:teuthology.orchestra.run.smithi083.stdout: "bytes_used": 7413760, 2024-03-06T07:23:25.047 INFO:teuthology.orchestra.run.smithi083.stdout: "percent_used": 1.009875450108666e-05, 2024-03-06T07:23:25.047 INFO:teuthology.orchestra.run.smithi083.stdout: "max_avail": 367059369984 2024-03-06T07:23:25.047 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:23:25.047 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:23:25.047 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:23:25.047 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "cephfs_data", 2024-03-06T07:23:25.048 INFO:teuthology.orchestra.run.smithi083.stdout: "id": 5, 2024-03-06T07:23:25.048 INFO:teuthology.orchestra.run.smithi083.stdout: "stats": { 2024-03-06T07:23:25.048 INFO:teuthology.orchestra.run.smithi083.stdout: "stored": 0, 2024-03-06T07:23:25.048 INFO:teuthology.orchestra.run.smithi083.stdout: "objects": 0, 2024-03-06T07:23:25.048 INFO:teuthology.orchestra.run.smithi083.stdout: "kb_used": 0, 2024-03-06T07:23:25.048 INFO:teuthology.orchestra.run.smithi083.stdout: "bytes_used": 0, 2024-03-06T07:23:25.048 INFO:teuthology.orchestra.run.smithi083.stdout: "percent_used": 0, 2024-03-06T07:23:25.048 INFO:teuthology.orchestra.run.smithi083.stdout: "max_avail": 367059369984 2024-03-06T07:23:25.048 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:23:25.048 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:23:25.048 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-03-06T07:23:25.048 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:23:25.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.044+0000 7f9f9046b640 1 -- 172.21.15.83:0/1486678927 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9f6c06b4d0 msgr2=0x7f9f6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.044+0000 7f9f9046b640 1 --2- 172.21.15.83:0/1486678927 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9f6c06b4d0 0x7f9f6c06d990 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c036000 tx=0x7f9f7c03c040 comp rx=0 tx=0).stop 2024-03-06T07:23:25.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.044+0000 7f9f9046b640 1 -- 172.21.15.83:0/1486678927 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f88161cf0 msgr2=0x7f9f880c9bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.044+0000 7f9f9046b640 1 --2- 172.21.15.83:0/1486678927 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f88161cf0 0x7f9f880c9bb0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f9f8400c7f0 tx=0x7f9f8400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.044+0000 7f9f9046b640 1 -- 172.21.15.83:0/1486678927 shutdown_connections 2024-03-06T07:23:25.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.044+0000 7f9f9046b640 1 --2- 172.21.15.83:0/1486678927 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9f6c06b4d0 0x7f9f6c06d990 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.044+0000 7f9f9046b640 1 --2- 172.21.15.83:0/1486678927 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9f88162970 0x7f9f880b6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.044+0000 7f9f9046b640 1 --2- 172.21.15.83:0/1486678927 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f88161cf0 0x7f9f880c9bb0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.044+0000 7f9f9046b640 1 --2- 172.21.15.83:0/1486678927 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9f880b4460 0x7f9f88155f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.044+0000 7f9f9046b640 1 -- 172.21.15.83:0/1486678927 >> 172.21.15.83:0/1486678927 conn(0x7f9f880d3ce0 msgr2=0x7f9f8815fb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:25.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.044+0000 7f9f9046b640 1 -- 172.21.15.83:0/1486678927 shutdown_connections 2024-03-06T07:23:25.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.044+0000 7f9f9046b640 1 -- 172.21.15.83:0/1486678927 wait complete. 2024-03-06T07:23:25.056 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-03-06T07:23:25.057 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:23:25.057 DEBUG:teuthology.orchestra.run.smithi083:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:25.101 DEBUG:teuthology.orchestra.run.smithi183:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:25.153 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:23:25.154 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:23:25.154 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:25.170 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:25.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.309+0000 7fe8517ac640 1 Processor -- start 2024-03-06T07:23:25.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.309+0000 7fe8517ac640 1 -- start start 2024-03-06T07:23:25.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.310+0000 7fe8517ac640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe84c06c5c0 0x7fe84c06b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.310+0000 7fe8517ac640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fe84c06bc00 0x7fe84c06d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.310+0000 7fe8517ac640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe84c0cc090 con 0x7fe84c15bd10 2024-03-06T07:23:25.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.310+0000 7fe8517ac640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe84c0ae170 con 0x7fe84c06bc00 2024-03-06T07:23:25.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.310+0000 7fe8517ac640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe84c0cc330 con 0x7fe84c06c5c0 2024-03-06T07:23:25.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.311+0000 7fe84a7fc640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe84c06c5c0 0x7fe84c06b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.311+0000 7fe84b7fe640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fe84c06bc00 0x7fe84c06d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.311+0000 7fe84a7fc640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe84c06c5c0 0x7fe84c06b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:50322/0 (socket says 172.21.15.83:50322) 2024-03-06T07:23:25.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.311+0000 7fe84a7fc640 1 -- 172.21.15.83:0/2710978568 learned_addr learned my addr 172.21.15.83:0/2710978568 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:25.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.311+0000 7fe84affd640 1 --1- 172.21.15.83:0/2710978568 >> v1:172.21.15.183:6789/0 conn(0x7fe84c15bd10 0x7fe84c06afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:55708/0 (socket says 172.21.15.83:55708) 2024-03-06T07:23:25.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.311+0000 7fe84b7fe640 1 -- 172.21.15.83:0/2710978568 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe84c06c5c0 msgr2=0x7fe84c06b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:25.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.311+0000 7fe84b7fe640 1 --2- 172.21.15.83:0/2710978568 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe84c06c5c0 0x7fe84c06b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.311+0000 7fe84b7fe640 1 -- 172.21.15.83:0/2710978568 >> v1:172.21.15.183:6789/0 conn(0x7fe84c15bd10 legacy=0x7fe84c06afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.311+0000 7fe84b7fe640 1 -- 172.21.15.83:0/2710978568 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe84c0cc610 con 0x7fe84c06bc00 2024-03-06T07:23:25.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.312+0000 7fe84b7fe640 1 --2- 172.21.15.83:0/2710978568 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe84c06bc00 0x7fe84c06d0d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe844009d90 tx=0x7fe84402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1b22e240b20c79 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.312+0000 7fe849ffb640 1 -- 172.21.15.83:0/2710978568 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe844039070 con 0x7fe84c06bc00 2024-03-06T07:23:25.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.312+0000 7fe849ffb640 1 -- 172.21.15.83:0/2710978568 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe84402cc50 con 0x7fe84c06bc00 2024-03-06T07:23:25.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.312+0000 7fe849ffb640 1 -- 172.21.15.83:0/2710978568 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe844034440 con 0x7fe84c06bc00 2024-03-06T07:23:25.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.313+0000 7fe8517ac640 1 -- 172.21.15.83:0/2710978568 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe84c06bc00 msgr2=0x7fe84c06d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.313+0000 7fe8517ac640 1 --2- 172.21.15.83:0/2710978568 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe84c06bc00 0x7fe84c06d0d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe844009d90 tx=0x7fe84402bea0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.313+0000 7fe8517ac640 1 -- 172.21.15.83:0/2710978568 shutdown_connections 2024-03-06T07:23:25.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.313+0000 7fe8517ac640 1 --2- 172.21.15.83:0/2710978568 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe84c06bc00 0x7fe84c06d0d0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.313+0000 7fe8517ac640 1 --2- 172.21.15.83:0/2710978568 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe84c06c5c0 0x7fe84c06b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.313+0000 7fe8517ac640 1 -- 172.21.15.83:0/2710978568 >> 172.21.15.83:0/2710978568 conn(0x7fe84c0d00e0 msgr2=0x7fe84c0d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:25.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.313+0000 7fe8517ac640 1 -- 172.21.15.83:0/2710978568 shutdown_connections 2024-03-06T07:23:25.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.313+0000 7fe8517ac640 1 -- 172.21.15.83:0/2710978568 wait complete. 2024-03-06T07:23:25.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.314+0000 7fe8517ac640 1 Processor -- start 2024-03-06T07:23:25.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.314+0000 7fe8517ac640 1 -- start start 2024-03-06T07:23:25.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.315+0000 7fe8517ac640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe84c06bc00 0x7fe84c155210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.315+0000 7fe8517ac640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe84c06c5c0 0x7fe84c1f4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.315+0000 7fe8517ac640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe84c15bd10 0x7fe84c06da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.315+0000 7fe8517ac640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe84c15c270 con 0x7fe84c06c5c0 2024-03-06T07:23:25.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.315+0000 7fe8517ac640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe84c15c0f0 con 0x7fe84c15bd10 2024-03-06T07:23:25.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.315+0000 7fe8517ac640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe84c06d610 con 0x7fe84c06bc00 2024-03-06T07:23:25.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.315+0000 7fe84a7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe84c06c5c0 0x7fe84c1f4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.315+0000 7fe84a7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe84c06c5c0 0x7fe84c1f4f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56774/0 (socket says 172.21.15.83:56774) 2024-03-06T07:23:25.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.315+0000 7fe84a7fc640 1 -- 172.21.15.83:0/3777413612 learned_addr learned my addr 172.21.15.83:0/3777413612 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:25.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.315+0000 7fe84affd640 1 --2- 172.21.15.83:0/3777413612 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe84c06bc00 0x7fe84c155210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.315+0000 7fe84b7fe640 1 --2- 172.21.15.83:0/3777413612 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe84c15bd10 0x7fe84c06da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.315+0000 7fe84a7fc640 1 -- 172.21.15.83:0/3777413612 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe84c06bc00 msgr2=0x7fe84c155210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.315+0000 7fe84a7fc640 1 --2- 172.21.15.83:0/3777413612 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe84c06bc00 0x7fe84c155210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.315+0000 7fe84a7fc640 1 -- 172.21.15.83:0/3777413612 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe84c15bd10 msgr2=0x7fe84c06da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.315+0000 7fe84a7fc640 1 --2- 172.21.15.83:0/3777413612 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe84c15bd10 0x7fe84c06da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.315+0000 7fe84a7fc640 1 -- 172.21.15.83:0/3777413612 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe84c1f56a0 con 0x7fe84c06c5c0 2024-03-06T07:23:25.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.315+0000 7fe84a7fc640 1 --2- 172.21.15.83:0/3777413612 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe84c06c5c0 0x7fe84c1f4f70 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fe84000e8b0 tx=0x7fe84000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.315+0000 7fe8337fe640 1 -- 172.21.15.83:0/3777413612 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe840017070 con 0x7fe84c06c5c0 2024-03-06T07:23:25.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.316+0000 7fe8337fe640 1 -- 172.21.15.83:0/3777413612 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe84000c940 con 0x7fe84c06c5c0 2024-03-06T07:23:25.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.316+0000 7fe8517ac640 1 -- 172.21.15.83:0/3777413612 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe84c1f5980 con 0x7fe84c06c5c0 2024-03-06T07:23:25.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.316+0000 7fe8337fe640 1 -- 172.21.15.83:0/3777413612 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe84000cd10 con 0x7fe84c06c5c0 2024-03-06T07:23:25.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.316+0000 7fe8517ac640 1 -- 172.21.15.83:0/3777413612 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe84c0c49b0 con 0x7fe84c06c5c0 2024-03-06T07:23:25.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.317+0000 7fe8517ac640 1 -- 172.21.15.83:0/3777413612 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe84c0c47a0 con 0x7fe84c06c5c0 2024-03-06T07:23:25.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.317+0000 7fe8337fe640 1 -- 172.21.15.83:0/3777413612 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe84001a070 con 0x7fe84c06c5c0 2024-03-06T07:23:25.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.317+0000 7fe8337fe640 1 --2- 172.21.15.83:0/3777413612 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe83806b500 0x7fe83806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.317+0000 7fe8337fe640 1 -- 172.21.15.83:0/3777413612 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe84008a370 con 0x7fe84c06c5c0 2024-03-06T07:23:25.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.318+0000 7fe84affd640 1 --2- 172.21.15.83:0/3777413612 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe83806b500 0x7fe83806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.318+0000 7fe84affd640 1 --2- 172.21.15.83:0/3777413612 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe83806b500 0x7fe83806d9c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe83c007fd0 tx=0x7fe83c007e80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.322+0000 7fe8337fe640 1 -- 172.21.15.83:0/3777413612 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe840058c30 con 0x7fe84c06c5c0 2024-03-06T07:23:25.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.543+0000 7fe8517ac640 1 -- 172.21.15.83:0/3777413612 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe84c04f330 con 0x7fe84c06c5c0 2024-03-06T07:23:25.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.544+0000 7fe8337fe640 1 -- 172.21.15.83:0/3777413612 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16681 (secure 0 0 0) 0x7fe84005ccd0 con 0x7fe84c06c5c0 2024-03-06T07:23:25.546 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:23:25.546 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":33,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:23:01.812855+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:58.568071+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:59.116376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:25.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.546+0000 7fe8517ac640 1 -- 172.21.15.83:0/3777413612 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe83806b500 msgr2=0x7fe83806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.546+0000 7fe8517ac640 1 --2- 172.21.15.83:0/3777413612 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe83806b500 0x7fe83806d9c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe83c007fd0 tx=0x7fe83c007e80 comp rx=0 tx=0).stop 2024-03-06T07:23:25.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.546+0000 7fe8517ac640 1 -- 172.21.15.83:0/3777413612 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe84c06c5c0 msgr2=0x7fe84c1f4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.546+0000 7fe8517ac640 1 --2- 172.21.15.83:0/3777413612 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe84c06c5c0 0x7fe84c1f4f70 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fe84000e8b0 tx=0x7fe84000ed70 comp rx=0 tx=0).stop 2024-03-06T07:23:25.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.546+0000 7fe8517ac640 1 -- 172.21.15.83:0/3777413612 shutdown_connections 2024-03-06T07:23:25.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.546+0000 7fe8517ac640 1 --2- 172.21.15.83:0/3777413612 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe83806b500 0x7fe83806d9c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.546+0000 7fe8517ac640 1 --2- 172.21.15.83:0/3777413612 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe84c15bd10 0x7fe84c06da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.546+0000 7fe8517ac640 1 --2- 172.21.15.83:0/3777413612 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe84c06c5c0 0x7fe84c1f4f70 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.546+0000 7fe8517ac640 1 --2- 172.21.15.83:0/3777413612 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe84c06bc00 0x7fe84c155210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.546+0000 7fe8517ac640 1 -- 172.21.15.83:0/3777413612 >> 172.21.15.83:0/3777413612 conn(0x7fe84c0d00e0 msgr2=0x7fe84c15a310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:25.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.546+0000 7fe8517ac640 1 -- 172.21.15.83:0/3777413612 shutdown_connections 2024-03-06T07:23:25.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.546+0000 7fe8517ac640 1 -- 172.21.15.83:0/3777413612 wait complete. 2024-03-06T07:23:25.563 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:22:58.568071+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:22:59.116376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:25.564 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:25.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.666+0000 7f0b41c37640 1 Processor -- start 2024-03-06T07:23:25.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.666+0000 7f0b41c37640 1 -- start start 2024-03-06T07:23:25.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.668+0000 7f0b41c37640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0b3c06b600 0x7f0b3c0b58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.668+0000 7f0b41c37640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f0b3c15ecb0 0x7f0b3c157e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.668+0000 7f0b41c37640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b3c0b3a00 con 0x7f0b3c06b220 2024-03-06T07:23:25.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.668+0000 7f0b41c37640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b3c0b3ce0 con 0x7f0b3c06b600 2024-03-06T07:23:25.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.668+0000 7f0b41c37640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b3c15e4b0 con 0x7f0b3c15ecb0 2024-03-06T07:23:25.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.668+0000 7f0b32ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0b3c06b600 0x7f0b3c0b58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.668+0000 7f0b32ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0b3c06b600 0x7f0b3c0b58f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56782/0 (socket says 172.21.15.83:56782) 2024-03-06T07:23:25.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.668+0000 7f0b32ffd640 1 -- 172.21.15.83:0/2827997395 learned_addr learned my addr 172.21.15.83:0/2827997395 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:25.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.668+0000 7f0b3bfff640 1 --2- 172.21.15.83:0/2827997395 >> v2:172.21.15.183:3300/0 conn(0x7f0b3c15ecb0 0x7f0b3c157e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.668+0000 7f0b32ffd640 1 -- 172.21.15.83:0/2827997395 >> v2:172.21.15.183:3300/0 conn(0x7f0b3c15ecb0 msgr2=0x7f0b3c157e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:25.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.668+0000 7f0b32ffd640 1 --2- 172.21.15.83:0/2827997395 >> v2:172.21.15.183:3300/0 conn(0x7f0b3c15ecb0 0x7f0b3c157e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.668+0000 7f0b32ffd640 1 -- 172.21.15.83:0/2827997395 >> v1:172.21.15.183:6789/0 conn(0x7f0b3c06b220 legacy=0x7f0b3c072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.668+0000 7f0b32ffd640 1 -- 172.21.15.83:0/2827997395 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b3c15e790 con 0x7f0b3c06b600 2024-03-06T07:23:25.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.669+0000 7f0b32ffd640 1 --2- 172.21.15.83:0/2827997395 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b3c06b600 0x7f0b3c0b58f0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c0b50e0 tx=0x7f0b2402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f12151fac098ed08 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.669+0000 7f0b3affd640 1 -- 172.21.15.83:0/2827997395 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b24035020 con 0x7f0b3c06b600 2024-03-06T07:23:25.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.669+0000 7f0b3affd640 1 -- 172.21.15.83:0/2827997395 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b24031870 con 0x7f0b3c06b600 2024-03-06T07:23:25.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.669+0000 7f0b3affd640 1 -- 172.21.15.83:0/2827997395 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b24031c40 con 0x7f0b3c06b600 2024-03-06T07:23:25.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.670+0000 7f0b41c37640 1 -- 172.21.15.83:0/2827997395 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b3c06b600 msgr2=0x7f0b3c0b58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.670+0000 7f0b41c37640 1 --2- 172.21.15.83:0/2827997395 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b3c06b600 0x7f0b3c0b58f0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c0b50e0 tx=0x7f0b2402b910 comp rx=0 tx=0).stop 2024-03-06T07:23:25.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.670+0000 7f0b41c37640 1 -- 172.21.15.83:0/2827997395 shutdown_connections 2024-03-06T07:23:25.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.670+0000 7f0b41c37640 1 --2- 172.21.15.83:0/2827997395 >> v2:172.21.15.183:3300/0 conn(0x7f0b3c15ecb0 0x7f0b3c157e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.670+0000 7f0b41c37640 1 --2- 172.21.15.83:0/2827997395 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b3c06b600 0x7f0b3c0b58f0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.670+0000 7f0b41c37640 1 -- 172.21.15.83:0/2827997395 >> 172.21.15.83:0/2827997395 conn(0x7f0b3c06c0c0 msgr2=0x7f0b3c06c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:25.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.670+0000 7f0b41c37640 1 -- 172.21.15.83:0/2827997395 shutdown_connections 2024-03-06T07:23:25.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.670+0000 7f0b41c37640 1 -- 172.21.15.83:0/2827997395 wait complete. 2024-03-06T07:23:25.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.671+0000 7f0b41c37640 1 Processor -- start 2024-03-06T07:23:25.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.671+0000 7f0b41c37640 1 -- start start 2024-03-06T07:23:25.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.672+0000 7f0b41c37640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b3c06b220 0x7f0b3c14e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.672+0000 7f0b41c37640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b3c06b600 0x7f0b3c0aeb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.672+0000 7f0b41c37640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b3c15ecb0 0x7f0b3c1f6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.672+0000 7f0b41c37640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3c15f2c0 con 0x7f0b3c06b220 2024-03-06T07:23:25.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.672+0000 7f0b41c37640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3c15f140 con 0x7f0b3c06b600 2024-03-06T07:23:25.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.672+0000 7f0b41c37640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3c158390 con 0x7f0b3c15ecb0 2024-03-06T07:23:25.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.672+0000 7f0b3b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b3c06b220 0x7f0b3c14e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.672+0000 7f0b3b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b3c06b220 0x7f0b3c14e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56788/0 (socket says 172.21.15.83:56788) 2024-03-06T07:23:25.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.672+0000 7f0b3b7fe640 1 -- 172.21.15.83:0/3326180905 learned_addr learned my addr 172.21.15.83:0/3326180905 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:25.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.672+0000 7f0b32ffd640 1 --2- 172.21.15.83:0/3326180905 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b3c06b600 0x7f0b3c0aeb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.672+0000 7f0b3bfff640 1 --2- 172.21.15.83:0/3326180905 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b3c15ecb0 0x7f0b3c1f6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.672+0000 7f0b3b7fe640 1 -- 172.21.15.83:0/3326180905 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b3c15ecb0 msgr2=0x7f0b3c1f6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.672+0000 7f0b3b7fe640 1 --2- 172.21.15.83:0/3326180905 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b3c15ecb0 0x7f0b3c1f6cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.672+0000 7f0b3b7fe640 1 -- 172.21.15.83:0/3326180905 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b3c06b600 msgr2=0x7f0b3c0aeb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.672+0000 7f0b3b7fe640 1 --2- 172.21.15.83:0/3326180905 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b3c06b600 0x7f0b3c0aeb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.672+0000 7f0b3b7fe640 1 -- 172.21.15.83:0/3326180905 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b3c0af0b0 con 0x7f0b3c06b220 2024-03-06T07:23:25.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.672+0000 7f0b3b7fe640 1 --2- 172.21.15.83:0/3326180905 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b3c06b220 0x7f0b3c14e570 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c00c430 tx=0x7f0b2c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.673+0000 7f0b38ff9640 1 -- 172.21.15.83:0/3326180905 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c0052d0 con 0x7f0b3c06b220 2024-03-06T07:23:25.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.673+0000 7f0b38ff9640 1 -- 172.21.15.83:0/3326180905 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b2c005490 con 0x7f0b3c06b220 2024-03-06T07:23:25.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.673+0000 7f0b38ff9640 1 -- 172.21.15.83:0/3326180905 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c007260 con 0x7f0b3c06b220 2024-03-06T07:23:25.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.673+0000 7f0b41c37640 1 -- 172.21.15.83:0/3326180905 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b3c14ebe0 con 0x7f0b3c06b220 2024-03-06T07:23:25.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.673+0000 7f0b41c37640 1 -- 172.21.15.83:0/3326180905 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b3c14f1a0 con 0x7f0b3c06b220 2024-03-06T07:23:25.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.674+0000 7f0b41c37640 1 -- 172.21.15.83:0/3326180905 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b3c0733b0 con 0x7f0b3c06b220 2024-03-06T07:23:25.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.674+0000 7f0b38ff9640 1 -- 172.21.15.83:0/3326180905 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0b2c007400 con 0x7f0b3c06b220 2024-03-06T07:23:25.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.674+0000 7f0b38ff9640 1 --2- 172.21.15.83:0/3326180905 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0b1c06b500 0x7f0b1c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.675+0000 7f0b32ffd640 1 --2- 172.21.15.83:0/3326180905 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0b1c06b500 0x7f0b1c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.675+0000 7f0b38ff9640 1 -- 172.21.15.83:0/3326180905 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0b2c089640 con 0x7f0b3c06b220 2024-03-06T07:23:25.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.675+0000 7f0b32ffd640 1 --2- 172.21.15.83:0/3326180905 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0b1c06b500 0x7f0b1c06d9c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0b24036000 tx=0x7f0b2403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.679+0000 7f0b38ff9640 1 -- 172.21.15.83:0/3326180905 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0b2c057e70 con 0x7f0b3c06b220 2024-03-06T07:23:25.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.934+0000 7f0b41c37640 1 -- 172.21.15.83:0/3326180905 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0b3c068ff0 con 0x7f0b3c06b220 2024-03-06T07:23:25.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.934+0000 7f0b38ff9640 1 -- 172.21.15.83:0/3326180905 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v33) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0b2c05bf10 con 0x7f0b3c06b220 2024-03-06T07:23:25.936 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:23:25.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.936+0000 7f0b41c37640 1 -- 172.21.15.83:0/3326180905 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0b1c06b500 msgr2=0x7f0b1c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.936+0000 7f0b41c37640 1 --2- 172.21.15.83:0/3326180905 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0b1c06b500 0x7f0b1c06d9c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0b24036000 tx=0x7f0b2403c040 comp rx=0 tx=0).stop 2024-03-06T07:23:25.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.936+0000 7f0b41c37640 1 -- 172.21.15.83:0/3326180905 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b3c06b220 msgr2=0x7f0b3c14e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.936+0000 7f0b41c37640 1 --2- 172.21.15.83:0/3326180905 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b3c06b220 0x7f0b3c14e570 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c00c430 tx=0x7f0b2c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.936+0000 7f0b41c37640 1 -- 172.21.15.83:0/3326180905 shutdown_connections 2024-03-06T07:23:25.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.936+0000 7f0b41c37640 1 --2- 172.21.15.83:0/3326180905 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0b1c06b500 0x7f0b1c06d9c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.936+0000 7f0b41c37640 1 --2- 172.21.15.83:0/3326180905 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b3c15ecb0 0x7f0b3c1f6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.936+0000 7f0b41c37640 1 --2- 172.21.15.83:0/3326180905 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b3c06b600 0x7f0b3c0aeb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.936+0000 7f0b41c37640 1 --2- 172.21.15.83:0/3326180905 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b3c06b220 0x7f0b3c14e570 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.936+0000 7f0b41c37640 1 -- 172.21.15.83:0/3326180905 >> 172.21.15.83:0/3326180905 conn(0x7f0b3c06c0c0 msgr2=0x7f0b3c0b5770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:25.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.936+0000 7f0b41c37640 1 -- 172.21.15.83:0/3326180905 shutdown_connections 2024-03-06T07:23:25.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:25.936+0000 7f0b41c37640 1 -- 172.21.15.83:0/3326180905 wait complete. 2024-03-06T07:23:25.952 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:23:26.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.053+0000 7feabedc7640 1 Processor -- start 2024-03-06T07:23:26.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.053+0000 7feabedc7640 1 -- start start 2024-03-06T07:23:26.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.055+0000 7feabedc7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7feab80cf0d0 0x7feab806d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.055+0000 7feabedc7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feab80b3320 con 0x7feab815a220 2024-03-06T07:23:26.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.055+0000 7feabedc7640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feab80728d0 con 0x7feab80719e0 2024-03-06T07:23:26.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.055+0000 7feabedc7640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feab80b3590 con 0x7feab80cf0d0 2024-03-06T07:23:26.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.055+0000 7feabd364640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7feab80cf0d0 0x7feab806d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.055+0000 7feabd364640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7feab80cf0d0 0x7feab806d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56804/0 (socket says 172.21.15.83:56804) 2024-03-06T07:23:26.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.055+0000 7feabd364640 1 -- 172.21.15.83:0/3885673551 learned_addr learned my addr 172.21.15.83:0/3885673551 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:26.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.055+0000 7feabd364640 1 -- 172.21.15.83:0/3885673551 >> v1:172.21.15.183:6789/0 conn(0x7feab80719e0 legacy=0x7feab806c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.055+0000 7feabd364640 1 -- 172.21.15.83:0/3885673551 >> v1:172.21.15.83:6789/0 conn(0x7feab815a220 legacy=0x7feab80712d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.055+0000 7feab77fe640 1 -- 172.21.15.83:0/3885673551 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4143520609 0 0) 0x7feab80b3320 con 0x7feab815a220 2024-03-06T07:23:26.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.055+0000 7feabd364640 1 -- 172.21.15.83:0/3885673551 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab80cf520 con 0x7feab80cf0d0 2024-03-06T07:23:26.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.056+0000 7feabd364640 1 --2- 172.21.15.83:0/3885673551 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feab80cf0d0 0x7feab806d100 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7feab0009e50 tx=0x7feab002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c066cb06379e1794 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.056+0000 7feab77fe640 1 -- 172.21.15.83:0/3885673551 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab002cb20 con 0x7feab80cf0d0 2024-03-06T07:23:26.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.056+0000 7feab77fe640 1 -- 172.21.15.83:0/3885673551 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feab002ccc0 con 0x7feab80cf0d0 2024-03-06T07:23:26.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.056+0000 7feab77fe640 1 -- 172.21.15.83:0/3885673551 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab0034610 con 0x7feab80cf0d0 2024-03-06T07:23:26.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.057+0000 7feabedc7640 1 -- 172.21.15.83:0/3885673551 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feab80cf0d0 msgr2=0x7feab806d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.057+0000 7feabedc7640 1 --2- 172.21.15.83:0/3885673551 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feab80cf0d0 0x7feab806d100 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7feab0009e50 tx=0x7feab002bf80 comp rx=0 tx=0).stop 2024-03-06T07:23:26.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.057+0000 7feabedc7640 1 -- 172.21.15.83:0/3885673551 shutdown_connections 2024-03-06T07:23:26.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.057+0000 7feabedc7640 1 --2- 172.21.15.83:0/3885673551 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feab80cf0d0 0x7feab806d100 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.057+0000 7feabedc7640 1 -- 172.21.15.83:0/3885673551 >> 172.21.15.83:0/3885673551 conn(0x7feab806b190 msgr2=0x7feab806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:26.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.057+0000 7feabedc7640 1 -- 172.21.15.83:0/3885673551 shutdown_connections 2024-03-06T07:23:26.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.057+0000 7feabedc7640 1 -- 172.21.15.83:0/3885673551 wait complete. 2024-03-06T07:23:26.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.058+0000 7feabedc7640 1 Processor -- start 2024-03-06T07:23:26.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.058+0000 7feabedc7640 1 -- start start 2024-03-06T07:23:26.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.059+0000 7feabedc7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feab80719e0 0x7feab81508e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.059+0000 7feabedc7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feab80cf0d0 0x7feab8151070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.059+0000 7feabedc7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feab815a220 0x7feab81515b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.059+0000 7feabedc7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab8072530 con 0x7feab80cf0d0 2024-03-06T07:23:26.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.059+0000 7feabedc7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab8072030 con 0x7feab815a220 2024-03-06T07:23:26.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.059+0000 7feabedc7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feab80726b0 con 0x7feab80719e0 2024-03-06T07:23:26.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.059+0000 7feab7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feab80cf0d0 0x7feab8151070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.059+0000 7feab7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feab80cf0d0 0x7feab8151070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56810/0 (socket says 172.21.15.83:56810) 2024-03-06T07:23:26.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.059+0000 7feab7fff640 1 -- 172.21.15.83:0/3013416271 learned_addr learned my addr 172.21.15.83:0/3013416271 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:26.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.059+0000 7feabd364640 1 --2- 172.21.15.83:0/3013416271 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feab815a220 0x7feab81515b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.059+0000 7feabcb63640 1 --2- 172.21.15.83:0/3013416271 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feab80719e0 0x7feab81508e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.059+0000 7feab7fff640 1 -- 172.21.15.83:0/3013416271 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feab80719e0 msgr2=0x7feab81508e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.059+0000 7feab7fff640 1 --2- 172.21.15.83:0/3013416271 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feab80719e0 0x7feab81508e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.059+0000 7feab7fff640 1 -- 172.21.15.83:0/3013416271 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feab815a220 msgr2=0x7feab81515b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.059+0000 7feab7fff640 1 --2- 172.21.15.83:0/3013416271 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feab815a220 0x7feab81515b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.059+0000 7feab7fff640 1 -- 172.21.15.83:0/3013416271 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab8150e20 con 0x7feab80cf0d0 2024-03-06T07:23:26.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.059+0000 7feab7fff640 1 --2- 172.21.15.83:0/3013416271 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feab80cf0d0 0x7feab8151070 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7feaa000c480 tx=0x7feaa000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.060+0000 7feab57fa640 1 -- 172.21.15.83:0/3013416271 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa00052d0 con 0x7feab80cf0d0 2024-03-06T07:23:26.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.060+0000 7feab57fa640 1 -- 172.21.15.83:0/3013416271 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaa0005470 con 0x7feab80cf0d0 2024-03-06T07:23:26.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.060+0000 7feabedc7640 1 -- 172.21.15.83:0/3013416271 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feab815d250 con 0x7feab80cf0d0 2024-03-06T07:23:26.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.060+0000 7feab57fa640 1 -- 172.21.15.83:0/3013416271 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa0012870 con 0x7feab80cf0d0 2024-03-06T07:23:26.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.060+0000 7feabedc7640 1 -- 172.21.15.83:0/3013416271 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feab815a7a0 con 0x7feab80cf0d0 2024-03-06T07:23:26.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.061+0000 7feabedc7640 1 -- 172.21.15.83:0/3013416271 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feab815d770 con 0x7feab80cf0d0 2024-03-06T07:23:26.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.061+0000 7feab57fa640 1 -- 172.21.15.83:0/3013416271 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feaa0012a10 con 0x7feab80cf0d0 2024-03-06T07:23:26.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.061+0000 7feab57fa640 1 --2- 172.21.15.83:0/3013416271 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fea9806b4d0 0x7fea9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.061+0000 7feab57fa640 1 -- 172.21.15.83:0/3013416271 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feaa0089cc0 con 0x7feab80cf0d0 2024-03-06T07:23:26.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.062+0000 7feabcb63640 1 --2- 172.21.15.83:0/3013416271 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fea9806b4d0 0x7fea9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.062+0000 7feabcb63640 1 --2- 172.21.15.83:0/3013416271 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fea9806b4d0 0x7fea9806d990 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7feab8152890 tx=0x7feaa8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.066+0000 7feab57fa640 1 -- 172.21.15.83:0/3013416271 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feab815d770 con 0x7feab80cf0d0 2024-03-06T07:23:26.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.242+0000 7feabedc7640 1 -- 172.21.15.83:0/3013416271 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7feab80732b0 con 0x7feab80cf0d0 2024-03-06T07:23:26.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.243+0000 7feab57fa640 1 -- 172.21.15.83:0/3013416271 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v33) v1 ==== 104+0+11 (secure 0 0 0) 0x7feaa0058580 con 0x7feab80cf0d0 2024-03-06T07:23:26.245 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-03-06T07:23:26.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.245+0000 7feabedc7640 1 -- 172.21.15.83:0/3013416271 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fea9806b4d0 msgr2=0x7fea9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.245+0000 7feabedc7640 1 --2- 172.21.15.83:0/3013416271 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fea9806b4d0 0x7fea9806d990 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7feab8152890 tx=0x7feaa8009040 comp rx=0 tx=0).stop 2024-03-06T07:23:26.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.245+0000 7feabedc7640 1 -- 172.21.15.83:0/3013416271 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feab80cf0d0 msgr2=0x7feab8151070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.245+0000 7feabedc7640 1 --2- 172.21.15.83:0/3013416271 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feab80cf0d0 0x7feab8151070 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7feaa000c480 tx=0x7feaa000c940 comp rx=0 tx=0).stop 2024-03-06T07:23:26.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.245+0000 7feabedc7640 1 -- 172.21.15.83:0/3013416271 shutdown_connections 2024-03-06T07:23:26.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.245+0000 7feabedc7640 1 --2- 172.21.15.83:0/3013416271 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fea9806b4d0 0x7fea9806d990 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.245+0000 7feabedc7640 1 --2- 172.21.15.83:0/3013416271 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feab815a220 0x7feab81515b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.245+0000 7feabedc7640 1 --2- 172.21.15.83:0/3013416271 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feab80cf0d0 0x7feab8151070 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.245+0000 7feabedc7640 1 --2- 172.21.15.83:0/3013416271 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feab80719e0 0x7feab81508e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.245+0000 7feabedc7640 1 -- 172.21.15.83:0/3013416271 >> 172.21.15.83:0/3013416271 conn(0x7feab806b190 msgr2=0x7feab80712d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:26.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.245+0000 7feabedc7640 1 -- 172.21.15.83:0/3013416271 shutdown_connections 2024-03-06T07:23:26.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.245+0000 7feabedc7640 1 -- 172.21.15.83:0/3013416271 wait complete. 2024-03-06T07:23:26.257 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:23:26.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.360+0000 7ffa33379640 1 Processor -- start 2024-03-06T07:23:26.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.360+0000 7ffa33379640 1 -- start start 2024-03-06T07:23:26.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.362+0000 7ffa33379640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffa2c06fd10 0x7ffa2c0cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.362+0000 7ffa33379640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa2c0cd4b0 con 0x7ffa2c15c050 2024-03-06T07:23:26.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.362+0000 7ffa33379640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa2c0cd790 con 0x7ffa2c0d0af0 2024-03-06T07:23:26.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.362+0000 7ffa33379640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c0cda70 con 0x7ffa2c06fd10 2024-03-06T07:23:26.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.362+0000 7ffa31115640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ffa2c15c050 0x7ffa2c06f600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:44664/0 (socket says 172.21.15.83:44664) 2024-03-06T07:23:26.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.362+0000 7ffa31115640 1 -- 172.21.15.83:0/3372796143 learned_addr learned my addr 172.21.15.83:0/3372796143 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:23:26.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.362+0000 7ffa30914640 1 --2- 172.21.15.83:0/3372796143 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffa2c06fd10 0x7ffa2c0cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.362+0000 7ffa23fff640 1 -- 172.21.15.83:0/3372796143 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 880428986 0 0) 0x7ffa2c0cd4b0 con 0x7ffa2c15c050 2024-03-06T07:23:26.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.362+0000 7ffa23fff640 1 -- 172.21.15.83:0/3372796143 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa18003520 con 0x7ffa2c15c050 2024-03-06T07:23:26.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.362+0000 7ffa23fff640 1 -- 172.21.15.83:0/3372796143 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2973761288 0 0) 0x7ffa2c0cd790 con 0x7ffa2c0d0af0 2024-03-06T07:23:26.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.362+0000 7ffa23fff640 1 -- 172.21.15.83:0/3372796143 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa2c0cd4b0 con 0x7ffa2c0d0af0 2024-03-06T07:23:26.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.362+0000 7ffa30914640 1 -- 172.21.15.83:0/3372796143 >> v1:172.21.15.183:6789/0 conn(0x7ffa2c0d0af0 legacy=0x7ffa2c0ccca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.362+0000 7ffa30914640 1 -- 172.21.15.83:0/3372796143 >> v1:172.21.15.83:6789/0 conn(0x7ffa2c15c050 legacy=0x7ffa2c06f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.362+0000 7ffa23fff640 1 -- 172.21.15.83:0/3372796143 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3364986793 0 0) 0x7ffa18003520 con 0x7ffa2c15c050 2024-03-06T07:23:26.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.362+0000 7ffa30914640 1 -- 172.21.15.83:0/3372796143 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa2c0cdd50 con 0x7ffa2c06fd10 2024-03-06T07:23:26.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.363+0000 7ffa30914640 1 --2- 172.21.15.83:0/3372796143 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffa2c06fd10 0x7ffa2c0cc760 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c0097d0 tx=0x7ffa1c02b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bad5aedbda41ec6e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.363+0000 7ffa23fff640 1 -- 172.21.15.83:0/3372796143 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c037070 con 0x7ffa2c06fd10 2024-03-06T07:23:26.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.363+0000 7ffa23fff640 1 -- 172.21.15.83:0/3372796143 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa1c030510 con 0x7ffa2c06fd10 2024-03-06T07:23:26.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.363+0000 7ffa23fff640 1 -- 172.21.15.83:0/3372796143 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa28001670 con 0x7ffa2c06fd10 2024-03-06T07:23:26.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.364+0000 7ffa33379640 1 -- 172.21.15.83:0/3372796143 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffa2c06fd10 msgr2=0x7ffa2c0cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.364+0000 7ffa33379640 1 --2- 172.21.15.83:0/3372796143 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffa2c06fd10 0x7ffa2c0cc760 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c0097d0 tx=0x7ffa1c02b770 comp rx=0 tx=0).stop 2024-03-06T07:23:26.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.364+0000 7ffa33379640 1 -- 172.21.15.83:0/3372796143 shutdown_connections 2024-03-06T07:23:26.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.364+0000 7ffa33379640 1 --2- 172.21.15.83:0/3372796143 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffa2c06fd10 0x7ffa2c0cc760 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.364+0000 7ffa33379640 1 -- 172.21.15.83:0/3372796143 >> 172.21.15.83:0/3372796143 conn(0x7ffa2c0b3800 msgr2=0x7ffa2c0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:26.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.364+0000 7ffa33379640 1 -- 172.21.15.83:0/3372796143 shutdown_connections 2024-03-06T07:23:26.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.364+0000 7ffa33379640 1 -- 172.21.15.83:0/3372796143 wait complete. 2024-03-06T07:23:26.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.365+0000 7ffa33379640 1 Processor -- start 2024-03-06T07:23:26.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.365+0000 7ffa33379640 1 -- start start 2024-03-06T07:23:26.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.366+0000 7ffa33379640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffa2c06fd10 0x7ffa2c0b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.366+0000 7ffa33379640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa2c0d0af0 0x7ffa2c13c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.366+0000 7ffa33379640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffa2c15c050 0x7ffa2c155f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.366+0000 7ffa33379640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c166e50 con 0x7ffa2c0d0af0 2024-03-06T07:23:26.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.366+0000 7ffa33379640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c166cd0 con 0x7ffa2c15c050 2024-03-06T07:23:26.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.366+0000 7ffa33379640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c166fd0 con 0x7ffa2c06fd10 2024-03-06T07:23:26.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.366+0000 7ffa30914640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa2c0d0af0 0x7ffa2c13c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.366+0000 7ffa30914640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa2c0d0af0 0x7ffa2c13c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56824/0 (socket says 172.21.15.83:56824) 2024-03-06T07:23:26.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.366+0000 7ffa30914640 1 -- 172.21.15.83:0/1709500999 learned_addr learned my addr 172.21.15.83:0/1709500999 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:26.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.366+0000 7ffa31115640 1 --2- 172.21.15.83:0/1709500999 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffa2c06fd10 0x7ffa2c0b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.366+0000 7ffa31916640 1 --2- 172.21.15.83:0/1709500999 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffa2c15c050 0x7ffa2c155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.366+0000 7ffa30914640 1 -- 172.21.15.83:0/1709500999 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffa2c06fd10 msgr2=0x7ffa2c0b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.366+0000 7ffa30914640 1 --2- 172.21.15.83:0/1709500999 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffa2c06fd10 0x7ffa2c0b65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.366+0000 7ffa30914640 1 -- 172.21.15.83:0/1709500999 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffa2c15c050 msgr2=0x7ffa2c155f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.366+0000 7ffa30914640 1 --2- 172.21.15.83:0/1709500999 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffa2c15c050 0x7ffa2c155f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.366+0000 7ffa30914640 1 -- 172.21.15.83:0/1709500999 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa2c0b52d0 con 0x7ffa2c0d0af0 2024-03-06T07:23:26.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.366+0000 7ffa31916640 1 --2- 172.21.15.83:0/1709500999 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffa2c15c050 0x7ffa2c155f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:26.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.366+0000 7ffa31115640 1 --2- 172.21.15.83:0/1709500999 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffa2c06fd10 0x7ffa2c0b65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:26.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.366+0000 7ffa30914640 1 --2- 172.21.15.83:0/1709500999 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa2c0d0af0 0x7ffa2c13c4a0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c0097d0 tx=0x7ffa1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.367+0000 7ffa21ffb640 1 -- 172.21.15.83:0/1709500999 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c037040 con 0x7ffa2c0d0af0 2024-03-06T07:23:26.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.367+0000 7ffa21ffb640 1 -- 172.21.15.83:0/1709500999 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa1c004250 con 0x7ffa2c0d0af0 2024-03-06T07:23:26.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.367+0000 7ffa21ffb640 1 -- 172.21.15.83:0/1709500999 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c006e10 con 0x7ffa2c0d0af0 2024-03-06T07:23:26.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.367+0000 7ffa33379640 1 -- 172.21.15.83:0/1709500999 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa2c0b5550 con 0x7ffa2c0d0af0 2024-03-06T07:23:26.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.367+0000 7ffa33379640 1 -- 172.21.15.83:0/1709500999 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa2c0b5b10 con 0x7ffa2c0d0af0 2024-03-06T07:23:26.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.368+0000 7ffa33379640 1 -- 172.21.15.83:0/1709500999 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa2c151760 con 0x7ffa2c0d0af0 2024-03-06T07:23:26.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.368+0000 7ffa21ffb640 1 -- 172.21.15.83:0/1709500999 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffa1c03a070 con 0x7ffa2c0d0af0 2024-03-06T07:23:26.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.369+0000 7ffa21ffb640 1 --2- 172.21.15.83:0/1709500999 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffa1006b4d0 0x7ffa1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.369+0000 7ffa21ffb640 1 -- 172.21.15.83:0/1709500999 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffa1c0abad0 con 0x7ffa2c0d0af0 2024-03-06T07:23:26.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.369+0000 7ffa31115640 1 --2- 172.21.15.83:0/1709500999 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffa1006b4d0 0x7ffa1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.369+0000 7ffa31115640 1 --2- 172.21.15.83:0/1709500999 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffa1006b4d0 0x7ffa1006d990 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ffa28000fd0 tx=0x7ffa28009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.373+0000 7ffa21ffb640 1 -- 172.21.15.83:0/1709500999 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffa2c151760 con 0x7ffa2c0d0af0 2024-03-06T07:23:26.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.608+0000 7ffa33379640 1 -- 172.21.15.83:0/1709500999 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ffa2c068ff0 con 0x7ffa2c0d0af0 2024-03-06T07:23:26.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.609+0000 7ffa21ffb640 1 -- 172.21.15.83:0/1709500999 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v33) v1 ==== 100+0+11 (secure 0 0 0) 0x7ffa2c0b52d0 con 0x7ffa2c0d0af0 2024-03-06T07:23:26.611 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 64 2024-03-06T07:23:26.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.610+0000 7ffa33379640 1 -- 172.21.15.83:0/1709500999 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffa1006b4d0 msgr2=0x7ffa1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.610+0000 7ffa33379640 1 --2- 172.21.15.83:0/1709500999 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffa1006b4d0 0x7ffa1006d990 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ffa28000fd0 tx=0x7ffa28009040 comp rx=0 tx=0).stop 2024-03-06T07:23:26.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.610+0000 7ffa33379640 1 -- 172.21.15.83:0/1709500999 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa2c0d0af0 msgr2=0x7ffa2c13c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.610+0000 7ffa33379640 1 --2- 172.21.15.83:0/1709500999 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa2c0d0af0 0x7ffa2c13c4a0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c0097d0 tx=0x7ffa1c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.611+0000 7ffa33379640 1 -- 172.21.15.83:0/1709500999 shutdown_connections 2024-03-06T07:23:26.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.611+0000 7ffa33379640 1 --2- 172.21.15.83:0/1709500999 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffa1006b4d0 0x7ffa1006d990 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.611+0000 7ffa33379640 1 --2- 172.21.15.83:0/1709500999 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffa2c15c050 0x7ffa2c155f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.611+0000 7ffa33379640 1 --2- 172.21.15.83:0/1709500999 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa2c0d0af0 0x7ffa2c13c4a0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.611+0000 7ffa33379640 1 --2- 172.21.15.83:0/1709500999 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffa2c06fd10 0x7ffa2c0b65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.611+0000 7ffa33379640 1 -- 172.21.15.83:0/1709500999 >> 172.21.15.83:0/1709500999 conn(0x7ffa2c0b3800 msgr2=0x7ffa2c0b29b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:26.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.611+0000 7ffa33379640 1 -- 172.21.15.83:0/1709500999 shutdown_connections 2024-03-06T07:23:26.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.611+0000 7ffa33379640 1 -- 172.21.15.83:0/1709500999 wait complete. 2024-03-06T07:23:26.623 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:23:26.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.727+0000 7f0746535640 1 Processor -- start 2024-03-06T07:23:26.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.727+0000 7f0746535640 1 -- start start 2024-03-06T07:23:26.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.728+0000 7f0746535640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f07400b4460 0x7f07400d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.728+0000 7f0746535640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f0740162970 0x7f07400b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.728+0000 7f0746535640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07400d22c0 con 0x7f0740161cf0 2024-03-06T07:23:26.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.728+0000 7f0746535640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0740161b00 con 0x7f07400b4460 2024-03-06T07:23:26.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.728+0000 7f0746535640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07400cecc0 con 0x7f0740162970 2024-03-06T07:23:26.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.728+0000 7f073ffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f07400b4460 0x7f07400d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.728+0000 7f073ffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f07400b4460 0x7f07400d4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57248/0 (socket says 172.21.15.83:57248) 2024-03-06T07:23:26.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.728+0000 7f073ffff640 1 -- 172.21.15.83:0/4156020201 learned_addr learned my addr 172.21.15.83:0/4156020201 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:26.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.728+0000 7f073f7fe640 1 --2- 172.21.15.83:0/4156020201 >> v2:172.21.15.183:3300/0 conn(0x7f0740162970 0x7f07400b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.728+0000 7f073ffff640 1 -- 172.21.15.83:0/4156020201 >> v2:172.21.15.183:3300/0 conn(0x7f0740162970 msgr2=0x7f07400b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:26.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.728+0000 7f073ffff640 1 --2- 172.21.15.83:0/4156020201 >> v2:172.21.15.183:3300/0 conn(0x7f0740162970 0x7f07400b3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.728+0000 7f073ffff640 1 -- 172.21.15.83:0/4156020201 >> v1:172.21.15.183:6789/0 conn(0x7f0740161cf0 legacy=0x7f07400cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.728+0000 7f073ffff640 1 -- 172.21.15.83:0/4156020201 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07400d25a0 con 0x7f07400b4460 2024-03-06T07:23:26.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.729+0000 7f073ffff640 1 --2- 172.21.15.83:0/4156020201 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f07400b4460 0x7f07400d4800 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f07400d17e0 tx=0x7f073002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=672ee9ddd53da49c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.729+0000 7f073effd640 1 -- 172.21.15.83:0/4156020201 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0730034070 con 0x7f07400b4460 2024-03-06T07:23:26.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.729+0000 7f073effd640 1 -- 172.21.15.83:0/4156020201 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0730030860 con 0x7f07400b4460 2024-03-06T07:23:26.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.729+0000 7f073effd640 1 -- 172.21.15.83:0/4156020201 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0730030c30 con 0x7f07400b4460 2024-03-06T07:23:26.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.730+0000 7f0746535640 1 -- 172.21.15.83:0/4156020201 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f07400b4460 msgr2=0x7f07400d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.730+0000 7f0746535640 1 --2- 172.21.15.83:0/4156020201 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f07400b4460 0x7f07400d4800 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f07400d17e0 tx=0x7f073002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.730+0000 7f0746535640 1 -- 172.21.15.83:0/4156020201 shutdown_connections 2024-03-06T07:23:26.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.730+0000 7f0746535640 1 --2- 172.21.15.83:0/4156020201 >> v2:172.21.15.183:3300/0 conn(0x7f0740162970 0x7f07400b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.730+0000 7f0746535640 1 --2- 172.21.15.83:0/4156020201 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f07400b4460 0x7f07400d4800 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.730+0000 7f0746535640 1 -- 172.21.15.83:0/4156020201 >> 172.21.15.83:0/4156020201 conn(0x7f07400d3ce0 msgr2=0x7f07400d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:26.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.730+0000 7f0746535640 1 -- 172.21.15.83:0/4156020201 shutdown_connections 2024-03-06T07:23:26.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.730+0000 7f0746535640 1 -- 172.21.15.83:0/4156020201 wait complete. 2024-03-06T07:23:26.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.731+0000 7f0746535640 1 Processor -- start 2024-03-06T07:23:26.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.731+0000 7f0746535640 1 -- start start 2024-03-06T07:23:26.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.732+0000 7f0746535640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f07400b4460 0x7f0740156550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.732+0000 7f0746535640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0740161cf0 0x7f07400cd500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.732+0000 7f0746535640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0740162970 0x7f074015eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.732+0000 7f0746535640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07400ce950 con 0x7f0740161cf0 2024-03-06T07:23:26.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.732+0000 7f0746535640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07400d2b60 con 0x7f0740162970 2024-03-06T07:23:26.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.732+0000 7f0746535640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07400b3ac0 con 0x7f07400b4460 2024-03-06T07:23:26.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.732+0000 7f073f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0740161cf0 0x7f07400cd500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.732+0000 7f073f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0740161cf0 0x7f07400cd500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57252/0 (socket says 172.21.15.83:57252) 2024-03-06T07:23:26.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.732+0000 7f073f7fe640 1 -- 172.21.15.83:0/6980335 learned_addr learned my addr 172.21.15.83:0/6980335 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:26.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.732+0000 7f0744ad2640 1 --2- 172.21.15.83:0/6980335 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0740162970 0x7f074015eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.732+0000 7f073ffff640 1 --2- 172.21.15.83:0/6980335 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f07400b4460 0x7f0740156550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.732+0000 7f073f7fe640 1 -- 172.21.15.83:0/6980335 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f07400b4460 msgr2=0x7f0740156550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.732+0000 7f073f7fe640 1 --2- 172.21.15.83:0/6980335 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f07400b4460 0x7f0740156550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.732+0000 7f073f7fe640 1 -- 172.21.15.83:0/6980335 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0740162970 msgr2=0x7f074015eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.732+0000 7f073f7fe640 1 --2- 172.21.15.83:0/6980335 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0740162970 0x7f074015eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.732+0000 7f073f7fe640 1 -- 172.21.15.83:0/6980335 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07400cc460 con 0x7f0740161cf0 2024-03-06T07:23:26.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.733+0000 7f073f7fe640 1 --2- 172.21.15.83:0/6980335 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0740161cf0 0x7f07400cd500 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f073400e9b0 tx=0x7f073400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.733+0000 7f073cff9640 1 -- 172.21.15.83:0/6980335 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0734017070 con 0x7f0740161cf0 2024-03-06T07:23:26.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.733+0000 7f073cff9640 1 -- 172.21.15.83:0/6980335 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f073400c9a0 con 0x7f0740161cf0 2024-03-06T07:23:26.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.733+0000 7f0746535640 1 -- 172.21.15.83:0/6980335 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07400cc740 con 0x7f0740161cf0 2024-03-06T07:23:26.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.733+0000 7f073cff9640 1 -- 172.21.15.83:0/6980335 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f073400cd70 con 0x7f0740161cf0 2024-03-06T07:23:26.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.733+0000 7f0746535640 1 -- 172.21.15.83:0/6980335 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07400ccd00 con 0x7f0740161cf0 2024-03-06T07:23:26.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.734+0000 7f0746535640 1 -- 172.21.15.83:0/6980335 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f074004f330 con 0x7f0740161cf0 2024-03-06T07:23:26.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.734+0000 7f073cff9640 1 -- 172.21.15.83:0/6980335 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f073401a070 con 0x7f0740161cf0 2024-03-06T07:23:26.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.735+0000 7f073cff9640 1 --2- 172.21.15.83:0/6980335 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f072c06b500 0x7f072c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.735+0000 7f073cff9640 1 -- 172.21.15.83:0/6980335 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f073408a3c0 con 0x7f0740161cf0 2024-03-06T07:23:26.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.735+0000 7f073ffff640 1 --2- 172.21.15.83:0/6980335 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f072c06b500 0x7f072c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.735+0000 7f073ffff640 1 --2- 172.21.15.83:0/6980335 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f072c06b500 0x7f072c06d9c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f0730036000 tx=0x7f073003b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:26.739+0000 7f073cff9640 1 -- 172.21.15.83:0/6980335 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0734058c80 con 0x7f0740161cf0 2024-03-06T07:23:27.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.012+0000 7f0746535640 1 -- 172.21.15.83:0/6980335 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f074006b180 con 0x7f0740161cf0 2024-03-06T07:23:27.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.012+0000 7f073cff9640 1 -- 172.21.15.83:0/6980335 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v33) v1 ==== 80+0+0 (secure 0 0 0) 0x7f073405cd20 con 0x7f0740161cf0 2024-03-06T07:23:27.014 INFO:teuthology.orchestra.run.smithi083.stderr:listed 0 entries 2024-03-06T07:23:27.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.015+0000 7f0746535640 1 -- 172.21.15.83:0/6980335 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f072c06b500 msgr2=0x7f072c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.015+0000 7f0746535640 1 --2- 172.21.15.83:0/6980335 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f072c06b500 0x7f072c06d9c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f0730036000 tx=0x7f073003b040 comp rx=0 tx=0).stop 2024-03-06T07:23:27.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.015+0000 7f0746535640 1 -- 172.21.15.83:0/6980335 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0740161cf0 msgr2=0x7f07400cd500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.015+0000 7f0746535640 1 --2- 172.21.15.83:0/6980335 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0740161cf0 0x7f07400cd500 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f073400e9b0 tx=0x7f073400ee70 comp rx=0 tx=0).stop 2024-03-06T07:23:27.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.015+0000 7f0746535640 1 -- 172.21.15.83:0/6980335 shutdown_connections 2024-03-06T07:23:27.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.015+0000 7f0746535640 1 --2- 172.21.15.83:0/6980335 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f072c06b500 0x7f072c06d9c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.015+0000 7f0746535640 1 --2- 172.21.15.83:0/6980335 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0740162970 0x7f074015eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.015+0000 7f0746535640 1 --2- 172.21.15.83:0/6980335 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0740161cf0 0x7f07400cd500 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.015+0000 7f0746535640 1 --2- 172.21.15.83:0/6980335 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f07400b4460 0x7f0740156550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.015+0000 7f0746535640 1 -- 172.21.15.83:0/6980335 >> 172.21.15.83:0/6980335 conn(0x7f07400d3ce0 msgr2=0x7f07400d0500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:27.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.015+0000 7f0746535640 1 -- 172.21.15.83:0/6980335 shutdown_connections 2024-03-06T07:23:27.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.015+0000 7f0746535640 1 -- 172.21.15.83:0/6980335 wait complete. 2024-03-06T07:23:27.027 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi083.front.sepia.ceph.com... 2024-03-06T07:23:27.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:27.027 DEBUG:teuthology.orchestra.run.smithi083:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:27.052 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:ceph-fuse[52896]: fuse finished with error 0 and tester_r 0 2024-03-06T07:23:27.054 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi083.front.sepia.ceph.com 2024-03-06T07:23:27.054 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:27.054 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:23:27.222 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:23:27.222 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:27.238 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=287 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:23:27.238 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi183.front.sepia.ceph.com 2024-03-06T07:23:27.238 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:27.238 DEBUG:teuthology.orchestra.run.smithi183:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:23:27.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:23:27.293 INFO:teuthology.orchestra.run.smithi183.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:23:27.293 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:23:27.293 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:27.308 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1257 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:23:27.308 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi083.front.sepia.ceph.com 2024-03-06T07:23:27.309 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:27.309 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:23:27.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:23:27.364 INFO:teuthology.orchestra.run.smithi083.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:23:27.364 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:23:27.364 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:27.379 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=287 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:23:27.379 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi183.front.sepia.ceph.com 2024-03-06T07:23:27.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:27.379 DEBUG:teuthology.orchestra.run.smithi183:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:23:27.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:23:27.434 INFO:teuthology.orchestra.run.smithi183.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:23:27.435 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:27.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.536+0000 7fe029a88640 1 Processor -- start 2024-03-06T07:23:27.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.536+0000 7fe029a88640 1 -- start start 2024-03-06T07:23:27.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.536+0000 7fe029a88640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fe02415dfe0 0x7fe0240cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.536+0000 7fe029a88640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe0240d29c0 0x7fe0240b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.536+0000 7fe029a88640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0240b3430 con 0x7fe0240cf830 2024-03-06T07:23:27.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.536+0000 7fe029a88640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0240b31e0 con 0x7fe02415dfe0 2024-03-06T07:23:27.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.537+0000 7fe029a88640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0240b3d60 con 0x7fe0240d29c0 2024-03-06T07:23:27.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.537+0000 7fe0237fe640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fe02415dfe0 0x7fe0240cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.537+0000 7fe023fff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe0240d29c0 0x7fe0240b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.537+0000 7fe022ffd640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fe0240cf830 0x7fe0240d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:49092/0 (socket says 172.21.15.83:49092) 2024-03-06T07:23:27.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.537+0000 7fe022ffd640 1 -- 172.21.15.83:0/2889453090 learned_addr learned my addr 172.21.15.83:0/2889453090 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:23:27.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.537+0000 7fe023fff640 1 --2- 172.21.15.83:0/2889453090 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe0240d29c0 0x7fe0240b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:57076/0 (socket says 172.21.15.83:57076) 2024-03-06T07:23:27.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.537+0000 7fe0237fe640 1 --2- 172.21.15.83:0/2889453090 >> v2:172.21.15.183:3300/0 conn(0x7fe02415dfe0 0x7fe0240cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:44670/0 (socket says 172.21.15.83:44670) 2024-03-06T07:23:27.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.537+0000 7fe023fff640 1 -- 172.21.15.83:0/2889453090 >> v2:172.21.15.183:3300/0 conn(0x7fe02415dfe0 msgr2=0x7fe0240cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:27.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.537+0000 7fe023fff640 1 --2- 172.21.15.83:0/2889453090 >> v2:172.21.15.183:3300/0 conn(0x7fe02415dfe0 0x7fe0240cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.537+0000 7fe023fff640 1 -- 172.21.15.83:0/2889453090 >> v1:172.21.15.183:6789/0 conn(0x7fe0240cf830 legacy=0x7fe0240d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.537+0000 7fe023fff640 1 -- 172.21.15.83:0/2889453090 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0240b3710 con 0x7fe0240d29c0 2024-03-06T07:23:27.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.537+0000 7fe0227fc640 1 -- 172.21.15.83:0/2889453090 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 211841085 0 0) 0x7fe0240b3430 con 0x7fe0240cf830 2024-03-06T07:23:27.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.537+0000 7fe0237fe640 1 --2- 172.21.15.83:0/2889453090 >> v2:172.21.15.183:3300/0 conn(0x7fe02415dfe0 0x7fe0240cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:27.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.538+0000 7fe023fff640 1 --2- 172.21.15.83:0/2889453090 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe0240d29c0 0x7fe0240b41f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe01c009d90 tx=0x7fe01c02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a0bffffab7962f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.538+0000 7fe0227fc640 1 -- 172.21.15.83:0/2889453090 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe01c039070 con 0x7fe0240d29c0 2024-03-06T07:23:27.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.538+0000 7fe0227fc640 1 -- 172.21.15.83:0/2889453090 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe01c02cc50 con 0x7fe0240d29c0 2024-03-06T07:23:27.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.538+0000 7fe0227fc640 1 -- 172.21.15.83:0/2889453090 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe01c034440 con 0x7fe0240d29c0 2024-03-06T07:23:27.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.539+0000 7fe029a88640 1 -- 172.21.15.83:0/2889453090 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe0240d29c0 msgr2=0x7fe0240b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.539+0000 7fe029a88640 1 --2- 172.21.15.83:0/2889453090 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe0240d29c0 0x7fe0240b41f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe01c009d90 tx=0x7fe01c02bea0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.539+0000 7fe029a88640 1 -- 172.21.15.83:0/2889453090 shutdown_connections 2024-03-06T07:23:27.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.539+0000 7fe029a88640 1 --2- 172.21.15.83:0/2889453090 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe0240d29c0 0x7fe0240b41f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.539+0000 7fe029a88640 1 --2- 172.21.15.83:0/2889453090 >> v2:172.21.15.183:3300/0 conn(0x7fe02415dfe0 0x7fe0240cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.539+0000 7fe029a88640 1 -- 172.21.15.83:0/2889453090 >> 172.21.15.83:0/2889453090 conn(0x7fe0240d3cd0 msgr2=0x7fe0240d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:27.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.539+0000 7fe029a88640 1 -- 172.21.15.83:0/2889453090 shutdown_connections 2024-03-06T07:23:27.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.539+0000 7fe029a88640 1 -- 172.21.15.83:0/2889453090 wait complete. 2024-03-06T07:23:27.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.540+0000 7fe029a88640 1 Processor -- start 2024-03-06T07:23:27.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.540+0000 7fe029a88640 1 -- start start 2024-03-06T07:23:27.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.541+0000 7fe029a88640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe0240cf830 0x7fe024160980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.541+0000 7fe029a88640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0240d29c0 0x7fe0240b8370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.541+0000 7fe029a88640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe02415dfe0 0x7fe0240b88b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.541+0000 7fe029a88640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0240d1df0 con 0x7fe0240d29c0 2024-03-06T07:23:27.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.541+0000 7fe029a88640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0240d1c70 con 0x7fe02415dfe0 2024-03-06T07:23:27.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.541+0000 7fe029a88640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe02415e3c0 con 0x7fe0240cf830 2024-03-06T07:23:27.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.541+0000 7fe022ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0240d29c0 0x7fe0240b8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.541+0000 7fe022ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0240d29c0 0x7fe0240b8370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57258/0 (socket says 172.21.15.83:57258) 2024-03-06T07:23:27.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.541+0000 7fe022ffd640 1 -- 172.21.15.83:0/3180060615 learned_addr learned my addr 172.21.15.83:0/3180060615 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:27.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.541+0000 7fe023fff640 1 --2- 172.21.15.83:0/3180060615 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe02415dfe0 0x7fe0240b88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.541+0000 7fe0237fe640 1 --2- 172.21.15.83:0/3180060615 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe0240cf830 0x7fe024160980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.541+0000 7fe022ffd640 1 -- 172.21.15.83:0/3180060615 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe0240cf830 msgr2=0x7fe024160980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.541+0000 7fe022ffd640 1 --2- 172.21.15.83:0/3180060615 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe0240cf830 0x7fe024160980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.541+0000 7fe022ffd640 1 -- 172.21.15.83:0/3180060615 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe02415dfe0 msgr2=0x7fe0240b88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.541+0000 7fe022ffd640 1 --2- 172.21.15.83:0/3180060615 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe02415dfe0 0x7fe0240b88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.541+0000 7fe022ffd640 1 -- 172.21.15.83:0/3180060615 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe024160ec0 con 0x7fe0240d29c0 2024-03-06T07:23:27.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.541+0000 7fe022ffd640 1 --2- 172.21.15.83:0/3180060615 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0240d29c0 0x7fe0240b8370 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fe01800c4b0 tx=0x7fe01800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.541+0000 7fe00bfff640 1 -- 172.21.15.83:0/3180060615 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe018018070 con 0x7fe0240d29c0 2024-03-06T07:23:27.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.542+0000 7fe00bfff640 1 -- 172.21.15.83:0/3180060615 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe018007500 con 0x7fe0240d29c0 2024-03-06T07:23:27.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.542+0000 7fe029a88640 1 -- 172.21.15.83:0/3180060615 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0240caa90 con 0x7fe0240d29c0 2024-03-06T07:23:27.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.542+0000 7fe00bfff640 1 -- 172.21.15.83:0/3180060615 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe018004a20 con 0x7fe0240d29c0 2024-03-06T07:23:27.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.542+0000 7fe029a88640 1 -- 172.21.15.83:0/3180060615 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0240b7590 con 0x7fe0240d29c0 2024-03-06T07:23:27.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.543+0000 7fe029a88640 1 -- 172.21.15.83:0/3180060615 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0240d1a90 con 0x7fe0240d29c0 2024-03-06T07:23:27.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.543+0000 7fe00bfff640 1 -- 172.21.15.83:0/3180060615 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe01801b070 con 0x7fe0240d29c0 2024-03-06T07:23:27.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.544+0000 7fe00bfff640 1 --2- 172.21.15.83:0/3180060615 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe01006b4d0 0x7fe01006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.544+0000 7fe00bfff640 1 -- 172.21.15.83:0/3180060615 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe018089480 con 0x7fe0240d29c0 2024-03-06T07:23:27.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.544+0000 7fe0237fe640 1 --2- 172.21.15.83:0/3180060615 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe01006b4d0 0x7fe01006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.544+0000 7fe0237fe640 1 --2- 172.21.15.83:0/3180060615 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe01006b4d0 0x7fe01006d990 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe014009560 tx=0x7fe014009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.552+0000 7fe00bfff640 1 -- 172.21.15.83:0/3180060615 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe0240d1a90 con 0x7fe0240d29c0 2024-03-06T07:23:27.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.761+0000 7fe029a88640 1 -- 172.21.15.83:0/3180060615 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0241610f0 con 0x7fe0240d29c0 2024-03-06T07:23:27.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.762+0000 7fe00bfff640 1 -- 172.21.15.83:0/3180060615 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+4982 (secure 0 0 0) 0x7fe018057d70 con 0x7fe0240d29c0 2024-03-06T07:23:27.764 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:23:27.764 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4799,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/3542419057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3542419057},{"type":"v1","addr":"172.21.15.83:6836","nonce":3542419057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":5023,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1853869906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":1853869906},{"type":"v1","addr":"172.21.15.183:6835","nonce":1853869906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5026,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/3666667479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3666667479},{"type":"v1","addr":"172.21.15.183:6834","nonce":3666667479}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:00.145321+0000","modified":"2024-03-06T07:23:02.319419+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5244":{"gid":5244,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/2668925454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":2668925454},{"type":"v1","addr":"172.21.15.83:6837","nonce":2668925454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5244,"qdb_cluster":[5244]},"id":2}]} 2024-03-06T07:23:27.764 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 13 2024-03-06T07:23:27.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.763+0000 7fe029a88640 1 -- 172.21.15.83:0/3180060615 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe01006b4d0 msgr2=0x7fe01006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.763+0000 7fe029a88640 1 --2- 172.21.15.83:0/3180060615 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe01006b4d0 0x7fe01006d990 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe014009560 tx=0x7fe014009190 comp rx=0 tx=0).stop 2024-03-06T07:23:27.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.763+0000 7fe029a88640 1 -- 172.21.15.83:0/3180060615 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0240d29c0 msgr2=0x7fe0240b8370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.763+0000 7fe029a88640 1 --2- 172.21.15.83:0/3180060615 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0240d29c0 0x7fe0240b8370 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fe01800c4b0 tx=0x7fe01800c970 comp rx=0 tx=0).stop 2024-03-06T07:23:27.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.764+0000 7fe029a88640 1 -- 172.21.15.83:0/3180060615 shutdown_connections 2024-03-06T07:23:27.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.764+0000 7fe029a88640 1 --2- 172.21.15.83:0/3180060615 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe01006b4d0 0x7fe01006d990 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.764+0000 7fe029a88640 1 --2- 172.21.15.83:0/3180060615 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe02415dfe0 0x7fe0240b88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.764+0000 7fe029a88640 1 --2- 172.21.15.83:0/3180060615 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0240d29c0 0x7fe0240b8370 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.764+0000 7fe029a88640 1 --2- 172.21.15.83:0/3180060615 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe0240cf830 0x7fe024160980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.764+0000 7fe029a88640 1 -- 172.21.15.83:0/3180060615 >> 172.21.15.83:0/3180060615 conn(0x7fe0240d3cd0 msgr2=0x7fe02406cb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:27.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.764+0000 7fe029a88640 1 -- 172.21.15.83:0/3180060615 shutdown_connections 2024-03-06T07:23:27.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.764+0000 7fe029a88640 1 -- 172.21.15.83:0/3180060615 wait complete. 2024-03-06T07:23:27.777 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:27.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.877+0000 7ff7b1f02640 1 Processor -- start 2024-03-06T07:23:27.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.877+0000 7ff7b1f02640 1 -- start start 2024-03-06T07:23:27.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.878+0000 7ff7b1f02640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ff7ac15dfe0 0x7ff7ac0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.878+0000 7ff7b1f02640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff7ac0cf830 0x7ff7ac0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.878+0000 7ff7b1f02640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7ac0b3430 con 0x7ff7ac0d27f0 2024-03-06T07:23:27.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.878+0000 7ff7b1f02640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7ac0d2c50 con 0x7ff7ac15dfe0 2024-03-06T07:23:27.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.878+0000 7ff7b1f02640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7ac0b31e0 con 0x7ff7ac0cf830 2024-03-06T07:23:27.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.878+0000 7ff7ab7fe640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ff7ac15dfe0 0x7ff7ac0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.878+0000 7ff7aaffd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff7ac0cf830 0x7ff7ac0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.878+0000 7ff7abfff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7ff7ac0d27f0 0x7ff7ac0d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:49094/0 (socket says 172.21.15.83:49094) 2024-03-06T07:23:27.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.878+0000 7ff7abfff640 1 -- 172.21.15.83:0/4202488973 learned_addr learned my addr 172.21.15.83:0/4202488973 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:23:27.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.878+0000 7ff7aaffd640 1 -- 172.21.15.83:0/4202488973 >> v2:172.21.15.183:3300/0 conn(0x7ff7ac15dfe0 msgr2=0x7ff7ac0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:27.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.878+0000 7ff7aaffd640 1 --2- 172.21.15.83:0/4202488973 >> v2:172.21.15.183:3300/0 conn(0x7ff7ac15dfe0 0x7ff7ac0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.878+0000 7ff7aaffd640 1 -- 172.21.15.83:0/4202488973 >> v1:172.21.15.183:6789/0 conn(0x7ff7ac0d27f0 legacy=0x7ff7ac0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.878+0000 7ff7aaffd640 1 -- 172.21.15.83:0/4202488973 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7ac0b3710 con 0x7ff7ac0cf830 2024-03-06T07:23:27.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.878+0000 7ff7ab7fe640 1 --2- 172.21.15.83:0/4202488973 >> v2:172.21.15.183:3300/0 conn(0x7ff7ac15dfe0 0x7ff7ac0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:27.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.879+0000 7ff7aaffd640 1 --2- 172.21.15.83:0/4202488973 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff7ac0cf830 0x7ff7ac0d22b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff7a00097a0 tx=0x7ff7a002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b2b9c076b73b0a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.880+0000 7ff7aa7fc640 1 -- 172.21.15.83:0/4202488973 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a0035020 con 0x7ff7ac0cf830 2024-03-06T07:23:27.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.880+0000 7ff7aa7fc640 1 -- 172.21.15.83:0/4202488973 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7a0031830 con 0x7ff7ac0cf830 2024-03-06T07:23:27.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.880+0000 7ff7aa7fc640 1 -- 172.21.15.83:0/4202488973 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a0031c00 con 0x7ff7ac0cf830 2024-03-06T07:23:27.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.880+0000 7ff7b1f02640 1 -- 172.21.15.83:0/4202488973 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff7ac0cf830 msgr2=0x7ff7ac0d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.880+0000 7ff7b1f02640 1 --2- 172.21.15.83:0/4202488973 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff7ac0cf830 0x7ff7ac0d22b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff7a00097a0 tx=0x7ff7a002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.880+0000 7ff7b1f02640 1 -- 172.21.15.83:0/4202488973 shutdown_connections 2024-03-06T07:23:27.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.880+0000 7ff7b1f02640 1 --2- 172.21.15.83:0/4202488973 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff7ac0cf830 0x7ff7ac0d22b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.880+0000 7ff7b1f02640 1 --2- 172.21.15.83:0/4202488973 >> v2:172.21.15.183:3300/0 conn(0x7ff7ac15dfe0 0x7ff7ac0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.880+0000 7ff7b1f02640 1 -- 172.21.15.83:0/4202488973 >> 172.21.15.83:0/4202488973 conn(0x7ff7ac0d3cd0 msgr2=0x7ff7ac0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:27.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.881+0000 7ff7b1f02640 1 -- 172.21.15.83:0/4202488973 shutdown_connections 2024-03-06T07:23:27.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.881+0000 7ff7b1f02640 1 -- 172.21.15.83:0/4202488973 wait complete. 2024-03-06T07:23:27.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.882+0000 7ff7b1f02640 1 Processor -- start 2024-03-06T07:23:27.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.882+0000 7ff7b1f02640 1 -- start start 2024-03-06T07:23:27.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.882+0000 7ff7b1f02640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff7ac0cf830 0x7ff7ac0b79d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.882+0000 7ff7b1f02640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7ac0d27f0 0x7ff7ac1532f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.882+0000 7ff7b1f02640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7ac15dfe0 0x7ff7ac15e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.882+0000 7ff7b1f02640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7ac0d1c70 con 0x7ff7ac0d27f0 2024-03-06T07:23:27.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.882+0000 7ff7b1f02640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7ac0b3d60 con 0x7ff7ac15dfe0 2024-03-06T07:23:27.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.882+0000 7ff7b1f02640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7ac0d1df0 con 0x7ff7ac0cf830 2024-03-06T07:23:27.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.883+0000 7ff7aaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7ac0d27f0 0x7ff7ac1532f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.883+0000 7ff7aaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7ac0d27f0 0x7ff7ac1532f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57262/0 (socket says 172.21.15.83:57262) 2024-03-06T07:23:27.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.883+0000 7ff7aaffd640 1 -- 172.21.15.83:0/2476222062 learned_addr learned my addr 172.21.15.83:0/2476222062 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:27.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.883+0000 7ff7ab7fe640 1 --2- 172.21.15.83:0/2476222062 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff7ac0cf830 0x7ff7ac0b79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.883+0000 7ff7abfff640 1 --2- 172.21.15.83:0/2476222062 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7ac15dfe0 0x7ff7ac15e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.883+0000 7ff7aaffd640 1 -- 172.21.15.83:0/2476222062 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff7ac0cf830 msgr2=0x7ff7ac0b79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.883+0000 7ff7aaffd640 1 --2- 172.21.15.83:0/2476222062 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff7ac0cf830 0x7ff7ac0b79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.883+0000 7ff7aaffd640 1 -- 172.21.15.83:0/2476222062 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7ac15dfe0 msgr2=0x7ff7ac15e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.883+0000 7ff7aaffd640 1 --2- 172.21.15.83:0/2476222062 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7ac15dfe0 0x7ff7ac15e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.883+0000 7ff7aaffd640 1 -- 172.21.15.83:0/2476222062 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7ac153830 con 0x7ff7ac0d27f0 2024-03-06T07:23:27.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.883+0000 7ff7aaffd640 1 --2- 172.21.15.83:0/2476222062 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7ac0d27f0 0x7ff7ac1532f0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ff7a002bee0 tx=0x7ff7a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.883+0000 7ff793fff640 1 -- 172.21.15.83:0/2476222062 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a0035040 con 0x7ff7ac0d27f0 2024-03-06T07:23:27.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.883+0000 7ff793fff640 1 -- 172.21.15.83:0/2476222062 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7a003b920 con 0x7ff7ac0d27f0 2024-03-06T07:23:27.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.883+0000 7ff793fff640 1 -- 172.21.15.83:0/2476222062 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a003bd10 con 0x7ff7ac0d27f0 2024-03-06T07:23:27.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.883+0000 7ff7b1f02640 1 -- 172.21.15.83:0/2476222062 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7ac15e990 con 0x7ff7ac0d27f0 2024-03-06T07:23:27.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.883+0000 7ff7b1f02640 1 -- 172.21.15.83:0/2476222062 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7ac15ef10 con 0x7ff7ac0d27f0 2024-03-06T07:23:27.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.884+0000 7ff7b1f02640 1 -- 172.21.15.83:0/2476222062 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7ac04f330 con 0x7ff7ac0d27f0 2024-03-06T07:23:27.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.885+0000 7ff793fff640 1 -- 172.21.15.83:0/2476222062 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff7a0038020 con 0x7ff7ac0d27f0 2024-03-06T07:23:27.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.885+0000 7ff793fff640 1 --2- 172.21.15.83:0/2476222062 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff79806b4d0 0x7ff79806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.885+0000 7ff7ab7fe640 1 --2- 172.21.15.83:0/2476222062 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff79806b4d0 0x7ff79806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.885+0000 7ff793fff640 1 -- 172.21.15.83:0/2476222062 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff7a00ac1f0 con 0x7ff7ac0d27f0 2024-03-06T07:23:27.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.886+0000 7ff7ab7fe640 1 --2- 172.21.15.83:0/2476222062 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff79806b4d0 0x7ff79806d990 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff79c009560 tx=0x7ff79c009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:27.890+0000 7ff793fff640 1 -- 172.21.15.83:0/2476222062 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff7ac153830 con 0x7ff7ac0d27f0 2024-03-06T07:23:28.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.064+0000 7ff7b1f02640 1 -- 172.21.15.83:0/2476222062 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff7ac0b1110 con 0x7ff7ac0d27f0 2024-03-06T07:23:28.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.065+0000 7ff793fff640 1 -- 172.21.15.83:0/2476222062 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16681 (secure 0 0 0) 0x7ff7a0031800 con 0x7ff7ac0d27f0 2024-03-06T07:23:28.067 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:23:28.067 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":33,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:23:01.812855+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:58.568071+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:59.116376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:28.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.067+0000 7ff7b1f02640 1 -- 172.21.15.83:0/2476222062 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff79806b4d0 msgr2=0x7ff79806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.067+0000 7ff7b1f02640 1 --2- 172.21.15.83:0/2476222062 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff79806b4d0 0x7ff79806d990 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff79c009560 tx=0x7ff79c009190 comp rx=0 tx=0).stop 2024-03-06T07:23:28.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.067+0000 7ff7b1f02640 1 -- 172.21.15.83:0/2476222062 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7ac0d27f0 msgr2=0x7ff7ac1532f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.067+0000 7ff7b1f02640 1 --2- 172.21.15.83:0/2476222062 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7ac0d27f0 0x7ff7ac1532f0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ff7a002bee0 tx=0x7ff7a00046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.067+0000 7ff7b1f02640 1 -- 172.21.15.83:0/2476222062 shutdown_connections 2024-03-06T07:23:28.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.067+0000 7ff7b1f02640 1 --2- 172.21.15.83:0/2476222062 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff79806b4d0 0x7ff79806d990 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.067+0000 7ff7b1f02640 1 --2- 172.21.15.83:0/2476222062 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7ac15dfe0 0x7ff7ac15e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.067+0000 7ff7b1f02640 1 --2- 172.21.15.83:0/2476222062 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7ac0d27f0 0x7ff7ac1532f0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.067+0000 7ff7b1f02640 1 --2- 172.21.15.83:0/2476222062 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff7ac0cf830 0x7ff7ac0b79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.067+0000 7ff7b1f02640 1 -- 172.21.15.83:0/2476222062 >> 172.21.15.83:0/2476222062 conn(0x7ff7ac0d3cd0 msgr2=0x7ff7ac15d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:28.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.067+0000 7ff7b1f02640 1 -- 172.21.15.83:0/2476222062 shutdown_connections 2024-03-06T07:23:28.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.067+0000 7ff7b1f02640 1 -- 172.21.15.83:0/2476222062 wait complete. 2024-03-06T07:23:28.080 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:22:58.568071+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:22:59.116376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:28.081 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:28.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.183+0000 7f4ad9e2a640 1 Processor -- start 2024-03-06T07:23:28.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.183+0000 7f4ad9e2a640 1 -- start start 2024-03-06T07:23:28.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.184+0000 7f4ad9e2a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f4ad41614d0 0x7f4ad40d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.184+0000 7f4ad9e2a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4ad40d24b0 0x7f4ad40b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.184+0000 7f4ad9e2a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ad41618b0 con 0x7f4ad40d1910 2024-03-06T07:23:28.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.184+0000 7f4ad9e2a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ad40cf400 con 0x7f4ad40d24b0 2024-03-06T07:23:28.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.184+0000 7f4ad9e2a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ad40cf6a0 con 0x7f4ad41614d0 2024-03-06T07:23:28.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.184+0000 7f4ad3fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4ad40d24b0 0x7f4ad40b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.184+0000 7f4ad2ffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f4ad40d1910 0x7f4ad40d1da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49010/0 (socket says 172.21.15.83:49010) 2024-03-06T07:23:28.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.184+0000 7f4ad3fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4ad40d24b0 0x7f4ad40b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57264/0 (socket says 172.21.15.83:57264) 2024-03-06T07:23:28.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.184+0000 7f4ad2ffd640 1 -- 172.21.15.83:0/1495797081 learned_addr learned my addr 172.21.15.83:0/1495797081 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:23:28.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.184+0000 7f4ad37fe640 1 --2- 172.21.15.83:0/1495797081 >> v2:172.21.15.183:3300/0 conn(0x7f4ad41614d0 0x7f4ad40d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.185+0000 7f4ad3fff640 1 -- 172.21.15.83:0/1495797081 >> v2:172.21.15.183:3300/0 conn(0x7f4ad41614d0 msgr2=0x7f4ad40d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:28.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.185+0000 7f4ad3fff640 1 --2- 172.21.15.83:0/1495797081 >> v2:172.21.15.183:3300/0 conn(0x7f4ad41614d0 0x7f4ad40d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.185+0000 7f4ad3fff640 1 -- 172.21.15.83:0/1495797081 >> v1:172.21.15.83:6789/0 conn(0x7f4ad40d1910 legacy=0x7f4ad40d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.185+0000 7f4ad3fff640 1 -- 172.21.15.83:0/1495797081 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad40cf980 con 0x7f4ad40d24b0 2024-03-06T07:23:28.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.185+0000 7f4ad27fc640 1 -- 172.21.15.83:0/1495797081 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1612497428 0 0) 0x7f4ad41618b0 con 0x7f4ad40d1910 2024-03-06T07:23:28.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.186+0000 7f4ad3fff640 1 --2- 172.21.15.83:0/1495797081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ad40d24b0 0x7f4ad40b0e30 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f4ad408c470 tx=0x7f4acc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3a6970ced76ba59 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:28.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.186+0000 7f4ad27fc640 1 -- 172.21.15.83:0/1495797081 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4acc039070 con 0x7f4ad40d24b0 2024-03-06T07:23:28.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.186+0000 7f4ad27fc640 1 -- 172.21.15.83:0/1495797081 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4acc02cc80 con 0x7f4ad40d24b0 2024-03-06T07:23:28.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.186+0000 7f4ad27fc640 1 -- 172.21.15.83:0/1495797081 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4acc034660 con 0x7f4ad40d24b0 2024-03-06T07:23:28.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.186+0000 7f4ad9e2a640 1 -- 172.21.15.83:0/1495797081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ad40d24b0 msgr2=0x7f4ad40b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.186+0000 7f4ad9e2a640 1 --2- 172.21.15.83:0/1495797081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ad40d24b0 0x7f4ad40b0e30 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f4ad408c470 tx=0x7f4acc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.186+0000 7f4ad9e2a640 1 -- 172.21.15.83:0/1495797081 shutdown_connections 2024-03-06T07:23:28.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.186+0000 7f4ad9e2a640 1 --2- 172.21.15.83:0/1495797081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ad40d24b0 0x7f4ad40b0e30 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.186+0000 7f4ad9e2a640 1 --2- 172.21.15.83:0/1495797081 >> v2:172.21.15.183:3300/0 conn(0x7f4ad41614d0 0x7f4ad40d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.186+0000 7f4ad9e2a640 1 -- 172.21.15.83:0/1495797081 >> 172.21.15.83:0/1495797081 conn(0x7f4ad40d3d40 msgr2=0x7f4ad40d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:28.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.186+0000 7f4ad9e2a640 1 -- 172.21.15.83:0/1495797081 shutdown_connections 2024-03-06T07:23:28.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.186+0000 7f4ad9e2a640 1 -- 172.21.15.83:0/1495797081 wait complete. 2024-03-06T07:23:28.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.187+0000 7f4ad9e2a640 1 Processor -- start 2024-03-06T07:23:28.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.188+0000 7f4ad9e2a640 1 -- start start 2024-03-06T07:23:28.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.188+0000 7f4ad9e2a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4ad40d1910 0x7f4ad414e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.188+0000 7f4ad9e2a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4ad40d24b0 0x7f4ad40cd890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.188+0000 7f4ad9e2a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ad41614d0 0x7f4ad40c8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.188+0000 7f4ad9e2a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad40b1370 con 0x7f4ad41614d0 2024-03-06T07:23:28.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.188+0000 7f4ad9e2a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad40b0910 con 0x7f4ad40d24b0 2024-03-06T07:23:28.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.188+0000 7f4ad9e2a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad40b14f0 con 0x7f4ad40d1910 2024-03-06T07:23:28.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.188+0000 7f4ad3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ad41614d0 0x7f4ad40c8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.188+0000 7f4ad3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ad41614d0 0x7f4ad40c8280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57266/0 (socket says 172.21.15.83:57266) 2024-03-06T07:23:28.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.188+0000 7f4ad3fff640 1 -- 172.21.15.83:0/33794247 learned_addr learned my addr 172.21.15.83:0/33794247 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:28.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.189+0000 7f4ad2ffd640 1 --2- 172.21.15.83:0/33794247 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4ad40d24b0 0x7f4ad40cd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.189+0000 7f4ad37fe640 1 --2- 172.21.15.83:0/33794247 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4ad40d1910 0x7f4ad414e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.189+0000 7f4ad3fff640 1 -- 172.21.15.83:0/33794247 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4ad40d1910 msgr2=0x7f4ad414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.189+0000 7f4ad3fff640 1 --2- 172.21.15.83:0/33794247 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4ad40d1910 0x7f4ad414e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.189+0000 7f4ad3fff640 1 -- 172.21.15.83:0/33794247 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4ad40d24b0 msgr2=0x7f4ad40cd890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.189+0000 7f4ad3fff640 1 --2- 172.21.15.83:0/33794247 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4ad40d24b0 0x7f4ad40cd890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.189+0000 7f4ad3fff640 1 -- 172.21.15.83:0/33794247 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad40cc7f0 con 0x7f4ad41614d0 2024-03-06T07:23:28.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.189+0000 7f4ad3fff640 1 --2- 172.21.15.83:0/33794247 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ad41614d0 0x7f4ad40c8280 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f4acc037040 tx=0x7f4acc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:28.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.189+0000 7f4abbfff640 1 -- 172.21.15.83:0/33794247 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4acc044020 con 0x7f4ad41614d0 2024-03-06T07:23:28.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.189+0000 7f4abbfff640 1 -- 172.21.15.83:0/33794247 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4acc03a040 con 0x7f4ad41614d0 2024-03-06T07:23:28.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.189+0000 7f4ad9e2a640 1 -- 172.21.15.83:0/33794247 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ad40ccad0 con 0x7f4ad41614d0 2024-03-06T07:23:28.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.189+0000 7f4abbfff640 1 -- 172.21.15.83:0/33794247 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4acc039050 con 0x7f4ad41614d0 2024-03-06T07:23:28.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.189+0000 7f4ad9e2a640 1 -- 172.21.15.83:0/33794247 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ad40cd090 con 0x7f4ad41614d0 2024-03-06T07:23:28.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.190+0000 7f4abbfff640 1 -- 172.21.15.83:0/33794247 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4acc003b90 con 0x7f4ad41614d0 2024-03-06T07:23:28.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.190+0000 7f4ad9e2a640 1 -- 172.21.15.83:0/33794247 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ad4068ff0 con 0x7f4ad41614d0 2024-03-06T07:23:28.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.191+0000 7f4abbfff640 1 --2- 172.21.15.83:0/33794247 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4ac006b4d0 0x7f4ac006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.191+0000 7f4abbfff640 1 -- 172.21.15.83:0/33794247 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4acc0ac3b0 con 0x7f4ad41614d0 2024-03-06T07:23:28.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.191+0000 7f4ad37fe640 1 --2- 172.21.15.83:0/33794247 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4ac006b4d0 0x7f4ac006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.191+0000 7f4ad37fe640 1 --2- 172.21.15.83:0/33794247 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4ac006b4d0 0x7f4ac006d990 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4ac4009aa0 tx=0x7f4ac4009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:28.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.195+0000 7f4abbfff640 1 -- 172.21.15.83:0/33794247 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4acc07ac70 con 0x7f4ad41614d0 2024-03-06T07:23:28.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.388+0000 7f4ad9e2a640 1 -- 172.21.15.83:0/33794247 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4ad40cddd0 con 0x7f4ad41614d0 2024-03-06T07:23:28.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.388+0000 7f4abbfff640 1 -- 172.21.15.83:0/33794247 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v33) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4acc07ed10 con 0x7f4ad41614d0 2024-03-06T07:23:28.390 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:23:28.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.391+0000 7f4ad9e2a640 1 -- 172.21.15.83:0/33794247 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4ac006b4d0 msgr2=0x7f4ac006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.391+0000 7f4ad9e2a640 1 --2- 172.21.15.83:0/33794247 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4ac006b4d0 0x7f4ac006d990 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4ac4009aa0 tx=0x7f4ac4009190 comp rx=0 tx=0).stop 2024-03-06T07:23:28.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.391+0000 7f4ad9e2a640 1 -- 172.21.15.83:0/33794247 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ad41614d0 msgr2=0x7f4ad40c8280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.391+0000 7f4ad9e2a640 1 --2- 172.21.15.83:0/33794247 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ad41614d0 0x7f4ad40c8280 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f4acc037040 tx=0x7f4acc004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:28.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.391+0000 7f4ad9e2a640 1 -- 172.21.15.83:0/33794247 shutdown_connections 2024-03-06T07:23:28.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.391+0000 7f4ad9e2a640 1 --2- 172.21.15.83:0/33794247 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4ac006b4d0 0x7f4ac006d990 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.391+0000 7f4ad9e2a640 1 --2- 172.21.15.83:0/33794247 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ad41614d0 0x7f4ad40c8280 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.391+0000 7f4ad9e2a640 1 --2- 172.21.15.83:0/33794247 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4ad40d24b0 0x7f4ad40cd890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.391+0000 7f4ad9e2a640 1 --2- 172.21.15.83:0/33794247 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4ad40d1910 0x7f4ad414e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.391+0000 7f4ad9e2a640 1 -- 172.21.15.83:0/33794247 >> 172.21.15.83:0/33794247 conn(0x7f4ad40d3d40 msgr2=0x7f4ad406b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:28.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.391+0000 7f4ad9e2a640 1 -- 172.21.15.83:0/33794247 shutdown_connections 2024-03-06T07:23:28.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.391+0000 7f4ad9e2a640 1 -- 172.21.15.83:0/33794247 wait complete. 2024-03-06T07:23:28.405 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:23:28.405 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:23:28.408 DEBUG:teuthology.orchestra.run.smithi183:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:23:28.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.508+0000 7fb47aaf4640 1 Processor -- start 2024-03-06T07:23:28.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.508+0000 7fb47aaf4640 1 -- start start 2024-03-06T07:23:28.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.509+0000 7fb47aaf4640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fb47415bb40 0x7fb4740b4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.509+0000 7fb47aaf4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb47415f7d0 0x7fb47406d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.509+0000 7fb47aaf4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb47406d5c0 0x7fb4740cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.509+0000 7fb47aaf4640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb47406c910 con 0x7fb47406d5c0 2024-03-06T07:23:28.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.509+0000 7fb47aaf4640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb47415ff90 con 0x7fb47415bb40 2024-03-06T07:23:28.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.509+0000 7fb47aaf4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb474160270 con 0x7fb47415f7d0 2024-03-06T07:23:28.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.509+0000 7fb479091640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb47406d5c0 0x7fb4740cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.509+0000 7fb479091640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb47406d5c0 0x7fb4740cfa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57272/0 (socket says 172.21.15.83:57272) 2024-03-06T07:23:28.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.509+0000 7fb479091640 1 -- 172.21.15.83:0/993446826 learned_addr learned my addr 172.21.15.83:0/993446826 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:28.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.509+0000 7fb473fff640 1 --2- 172.21.15.83:0/993446826 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb47415f7d0 0x7fb47406d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.509+0000 7fb478890640 1 --2- 172.21.15.83:0/993446826 >> v2:172.21.15.183:3300/0 conn(0x7fb47415bb40 0x7fb4740b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.509+0000 7fb479091640 1 -- 172.21.15.83:0/993446826 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb47415f7d0 msgr2=0x7fb47406d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:28.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.509+0000 7fb479091640 1 --2- 172.21.15.83:0/993446826 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb47415f7d0 0x7fb47406d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.509+0000 7fb479091640 1 -- 172.21.15.83:0/993446826 >> v2:172.21.15.183:3300/0 conn(0x7fb47415bb40 msgr2=0x7fb4740b4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:28.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.509+0000 7fb479091640 1 --2- 172.21.15.83:0/993446826 >> v2:172.21.15.183:3300/0 conn(0x7fb47415bb40 0x7fb4740b4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.509+0000 7fb479091640 1 -- 172.21.15.83:0/993446826 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb474160550 con 0x7fb47406d5c0 2024-03-06T07:23:28.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.509+0000 7fb473fff640 1 --2- 172.21.15.83:0/993446826 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb47415f7d0 0x7fb47406d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:28.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.510+0000 7fb479091640 1 --2- 172.21.15.83:0/993446826 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb47406d5c0 0x7fb4740cfa90 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fb4740b4f60 tx=0x7fb46c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49c1b97b73ec6ac server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:28.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.510+0000 7fb4737fe640 1 -- 172.21.15.83:0/993446826 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46c02ca70 con 0x7fb47406d5c0 2024-03-06T07:23:28.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.510+0000 7fb4737fe640 1 -- 172.21.15.83:0/993446826 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb46c02cc10 con 0x7fb47406d5c0 2024-03-06T07:23:28.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.510+0000 7fb4737fe640 1 -- 172.21.15.83:0/993446826 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46c0343f0 con 0x7fb47406d5c0 2024-03-06T07:23:28.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.510+0000 7fb47aaf4640 1 -- 172.21.15.83:0/993446826 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb47406d5c0 msgr2=0x7fb4740cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.510+0000 7fb47aaf4640 1 --2- 172.21.15.83:0/993446826 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb47406d5c0 0x7fb4740cfa90 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fb4740b4f60 tx=0x7fb46c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.511+0000 7fb47aaf4640 1 -- 172.21.15.83:0/993446826 shutdown_connections 2024-03-06T07:23:28.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.511+0000 7fb47aaf4640 1 --2- 172.21.15.83:0/993446826 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb47406d5c0 0x7fb4740cfa90 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.511+0000 7fb47aaf4640 1 --2- 172.21.15.83:0/993446826 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb47415f7d0 0x7fb47406d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.511+0000 7fb47aaf4640 1 --2- 172.21.15.83:0/993446826 >> v2:172.21.15.183:3300/0 conn(0x7fb47415bb40 0x7fb4740b4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.511+0000 7fb47aaf4640 1 -- 172.21.15.83:0/993446826 >> 172.21.15.83:0/993446826 conn(0x7fb47406b190 msgr2=0x7fb47406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:28.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.511+0000 7fb47aaf4640 1 -- 172.21.15.83:0/993446826 shutdown_connections 2024-03-06T07:23:28.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.511+0000 7fb47aaf4640 1 -- 172.21.15.83:0/993446826 wait complete. 2024-03-06T07:23:28.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.512+0000 7fb47aaf4640 1 Processor -- start 2024-03-06T07:23:28.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.512+0000 7fb47aaf4640 1 -- start start 2024-03-06T07:23:28.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.513+0000 7fb47aaf4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb47406d5c0 0x7fb474141990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.513+0000 7fb47aaf4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb47415bb40 0x7fb47415aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.513+0000 7fb47aaf4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb47415f7d0 0x7fb47413f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.513+0000 7fb47aaf4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47415bf20 con 0x7fb47406d5c0 2024-03-06T07:23:28.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.513+0000 7fb47aaf4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47415fc30 con 0x7fb47415f7d0 2024-03-06T07:23:28.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.513+0000 7fb47aaf4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb47415c0a0 con 0x7fb47415bb40 2024-03-06T07:23:28.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.513+0000 7fb478890640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb47406d5c0 0x7fb474141990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.513+0000 7fb478890640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb47406d5c0 0x7fb474141990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57288/0 (socket says 172.21.15.83:57288) 2024-03-06T07:23:28.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.513+0000 7fb478890640 1 -- 172.21.15.83:0/3718807257 learned_addr learned my addr 172.21.15.83:0/3718807257 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:28.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.513+0000 7fb479091640 1 --2- 172.21.15.83:0/3718807257 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb47415f7d0 0x7fb47413f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.513+0000 7fb473fff640 1 --2- 172.21.15.83:0/3718807257 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb47415bb40 0x7fb47415aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.513+0000 7fb478890640 1 -- 172.21.15.83:0/3718807257 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb47415bb40 msgr2=0x7fb47415aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.513+0000 7fb478890640 1 --2- 172.21.15.83:0/3718807257 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb47415bb40 0x7fb47415aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.513+0000 7fb478890640 1 -- 172.21.15.83:0/3718807257 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb47415f7d0 msgr2=0x7fb47413f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.513+0000 7fb478890640 1 --2- 172.21.15.83:0/3718807257 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb47415f7d0 0x7fb47413f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.513+0000 7fb478890640 1 -- 172.21.15.83:0/3718807257 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb47413ff60 con 0x7fb47406d5c0 2024-03-06T07:23:28.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.513+0000 7fb473fff640 1 --2- 172.21.15.83:0/3718807257 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb47415bb40 0x7fb47415aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:28.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.513+0000 7fb478890640 1 --2- 172.21.15.83:0/3718807257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb47406d5c0 0x7fb474141990 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fb46800d620 tx=0x7fb46800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:28.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.513+0000 7fb4717fa640 1 -- 172.21.15.83:0/3718807257 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46800b620 con 0x7fb47406d5c0 2024-03-06T07:23:28.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.513+0000 7fb4717fa640 1 -- 172.21.15.83:0/3718807257 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb468019070 con 0x7fb47406d5c0 2024-03-06T07:23:28.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.513+0000 7fb47aaf4640 1 -- 172.21.15.83:0/3718807257 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb474140240 con 0x7fb47406d5c0 2024-03-06T07:23:28.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.513+0000 7fb4717fa640 1 -- 172.21.15.83:0/3718807257 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46800ba00 con 0x7fb47406d5c0 2024-03-06T07:23:28.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.513+0000 7fb47aaf4640 1 -- 172.21.15.83:0/3718807257 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb474140800 con 0x7fb47406d5c0 2024-03-06T07:23:28.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.514+0000 7fb47aaf4640 1 -- 172.21.15.83:0/3718807257 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4740732b0 con 0x7fb47406d5c0 2024-03-06T07:23:28.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.515+0000 7fb4717fa640 1 -- 172.21.15.83:0/3718807257 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb46800bba0 con 0x7fb47406d5c0 2024-03-06T07:23:28.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.515+0000 7fb4717fa640 1 --2- 172.21.15.83:0/3718807257 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb45806b4d0 0x7fb45806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.516+0000 7fb473fff640 1 --2- 172.21.15.83:0/3718807257 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb45806b4d0 0x7fb45806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.516+0000 7fb4717fa640 1 -- 172.21.15.83:0/3718807257 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb46808a4e0 con 0x7fb47406d5c0 2024-03-06T07:23:28.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.516+0000 7fb473fff640 1 --2- 172.21.15.83:0/3718807257 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb45806b4d0 0x7fb45806d990 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb474145530 tx=0x7fb460008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:28.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.520+0000 7fb4717fa640 1 -- 172.21.15.83:0/3718807257 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb4680593f0 con 0x7fb47406d5c0 2024-03-06T07:23:28.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.694+0000 7fb47aaf4640 1 -- 172.21.15.83:0/3718807257 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb474068ff0 con 0x7fb47406d5c0 2024-03-06T07:23:28.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.694+0000 7fb4717fa640 1 -- 172.21.15.83:0/3718807257 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v33) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb468058cf0 con 0x7fb47406d5c0 2024-03-06T07:23:28.696 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-03-06T07:23:28.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.696+0000 7fb47aaf4640 1 -- 172.21.15.83:0/3718807257 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb45806b4d0 msgr2=0x7fb45806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.696+0000 7fb47aaf4640 1 --2- 172.21.15.83:0/3718807257 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb45806b4d0 0x7fb45806d990 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb474145530 tx=0x7fb460008040 comp rx=0 tx=0).stop 2024-03-06T07:23:28.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.696+0000 7fb47aaf4640 1 -- 172.21.15.83:0/3718807257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb47406d5c0 msgr2=0x7fb474141990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.696+0000 7fb47aaf4640 1 --2- 172.21.15.83:0/3718807257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb47406d5c0 0x7fb474141990 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fb46800d620 tx=0x7fb46800dae0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.696+0000 7fb47aaf4640 1 -- 172.21.15.83:0/3718807257 shutdown_connections 2024-03-06T07:23:28.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.696+0000 7fb47aaf4640 1 --2- 172.21.15.83:0/3718807257 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb45806b4d0 0x7fb45806d990 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.696+0000 7fb47aaf4640 1 --2- 172.21.15.83:0/3718807257 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb47415f7d0 0x7fb47413f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.696+0000 7fb47aaf4640 1 --2- 172.21.15.83:0/3718807257 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb47415bb40 0x7fb47415aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.696+0000 7fb47aaf4640 1 --2- 172.21.15.83:0/3718807257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb47406d5c0 0x7fb474141990 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.696+0000 7fb47aaf4640 1 -- 172.21.15.83:0/3718807257 >> 172.21.15.83:0/3718807257 conn(0x7fb47406b190 msgr2=0x7fb4740b2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:28.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.696+0000 7fb47aaf4640 1 -- 172.21.15.83:0/3718807257 shutdown_connections 2024-03-06T07:23:28.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.696+0000 7fb47aaf4640 1 -- 172.21.15.83:0/3718807257 wait complete. 2024-03-06T07:23:28.709 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:23:28.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.810+0000 7fdc10f55640 1 Processor -- start 2024-03-06T07:23:28.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.811+0000 7fdc10f55640 1 -- start start 2024-03-06T07:23:28.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.812+0000 7fdc10f55640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdc0c0d4860 0x7fdc0c0d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.812+0000 7fdc10f55640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fdc0c0b0e30 0x7fdc0c0b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.812+0000 7fdc10f55640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc0c1618b0 con 0x7fdc0c1614d0 2024-03-06T07:23:28.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.812+0000 7fdc10f55640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc0c0d4cc0 con 0x7fdc0c0b0e30 2024-03-06T07:23:28.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.812+0000 7fdc10f55640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c0cf400 con 0x7fdc0c0d4860 2024-03-06T07:23:28.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.812+0000 7fdc0a59c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fdc0c1614d0 0x7fdc0c0d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49012/0 (socket says 172.21.15.83:49012) 2024-03-06T07:23:28.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.812+0000 7fdc0a59c640 1 -- 172.21.15.83:0/1105040329 learned_addr learned my addr 172.21.15.83:0/1105040329 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:23:28.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.812+0000 7fdc0ad9d640 1 --2- 172.21.15.83:0/1105040329 >> v2:172.21.15.183:3300/0 conn(0x7fdc0c0b0e30 0x7fdc0c0b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.812+0000 7fdc09d9b640 1 --2- 172.21.15.83:0/1105040329 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdc0c0d4860 0x7fdc0c0d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.812+0000 7fdc0959a640 1 -- 172.21.15.83:0/1105040329 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1321828419 0 0) 0x7fdc0c1618b0 con 0x7fdc0c1614d0 2024-03-06T07:23:28.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.812+0000 7fdc0959a640 1 -- 172.21.15.83:0/1105040329 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbf4003520 con 0x7fdc0c1614d0 2024-03-06T07:23:28.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.813+0000 7fdc0959a640 1 -- 172.21.15.83:0/1105040329 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3812178167 0 0) 0x7fdbf4003520 con 0x7fdc0c1614d0 2024-03-06T07:23:28.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.813+0000 7fdc0959a640 1 -- 172.21.15.83:0/1105040329 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdc0c0d4860 msgr2=0x7fdc0c0d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:28.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.813+0000 7fdc0959a640 1 --2- 172.21.15.83:0/1105040329 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdc0c0d4860 0x7fdc0c0d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.813+0000 7fdc0959a640 1 -- 172.21.15.83:0/1105040329 >> v2:172.21.15.183:3300/0 conn(0x7fdc0c0b0e30 msgr2=0x7fdc0c0b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:28.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.813+0000 7fdc0959a640 1 --2- 172.21.15.83:0/1105040329 >> v2:172.21.15.183:3300/0 conn(0x7fdc0c0b0e30 0x7fdc0c0b12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.813+0000 7fdc0959a640 1 -- 172.21.15.83:0/1105040329 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc0c0cf6a0 con 0x7fdc0c1614d0 2024-03-06T07:23:28.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.813+0000 7fdc0959a640 1 -- 172.21.15.83:0/1105040329 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7fdc040019f0 con 0x7fdc0c1614d0 2024-03-06T07:23:28.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.813+0000 7fdc0959a640 1 -- 172.21.15.83:0/1105040329 >> v1:172.21.15.83:6789/0 conn(0x7fdc0c1614d0 legacy=0x7fdc0c0d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.813+0000 7fdc0959a640 1 --2- 172.21.15.83:0/1105040329 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbf40039b0 0x7fdbf4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.813+0000 7fdc0959a640 1 --2- 172.21.15.83:0/1105040329 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdbf4004800 0x7fdbf4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.813+0000 7fdc0959a640 1 --2- 172.21.15.83:0/1105040329 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdc0c0d4860 0x7fdbf40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.813+0000 7fdc0959a640 1 -- 172.21.15.83:0/1105040329 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c0d4cc0 con 0x7fdbf40039b0 2024-03-06T07:23:28.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.813+0000 7fdc0959a640 1 -- 172.21.15.83:0/1105040329 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c0cf400 con 0x7fdbf4004800 2024-03-06T07:23:28.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.813+0000 7fdc0959a640 1 -- 172.21.15.83:0/1105040329 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf400b9c0 con 0x7fdc0c0d4860 2024-03-06T07:23:28.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.813+0000 7fdc0a59c640 1 --2- 172.21.15.83:0/1105040329 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbf40039b0 0x7fdbf4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.813+0000 7fdc09d9b640 1 --2- 172.21.15.83:0/1105040329 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdbf4004800 0x7fdbf4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.813+0000 7fdc0ad9d640 1 --2- 172.21.15.83:0/1105040329 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdc0c0d4860 0x7fdbf40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.813+0000 7fdc0a59c640 1 -- 172.21.15.83:0/1105040329 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdc0c0d4860 msgr2=0x7fdbf40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:28.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.813+0000 7fdc0a59c640 1 --2- 172.21.15.83:0/1105040329 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdc0c0d4860 0x7fdbf40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.813+0000 7fdc0a59c640 1 -- 172.21.15.83:0/1105040329 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdbf4004800 msgr2=0x7fdbf4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:28.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.813+0000 7fdc0a59c640 1 --2- 172.21.15.83:0/1105040329 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdbf4004800 0x7fdbf4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.813+0000 7fdc0a59c640 1 -- 172.21.15.83:0/1105040329 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf4003520 con 0x7fdbf40039b0 2024-03-06T07:23:28.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.813+0000 7fdc0ad9d640 1 --2- 172.21.15.83:0/1105040329 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdc0c0d4860 0x7fdbf40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:28.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.814+0000 7fdc0a59c640 1 --2- 172.21.15.83:0/1105040329 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbf40039b0 0x7fdbf4003dd0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fdc040013d0 tx=0x7fdbf4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f9f33651b543b1e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:28.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.814+0000 7fdc0959a640 1 -- 172.21.15.83:0/1105040329 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0400a4e0 con 0x7fdbf40039b0 2024-03-06T07:23:28.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.815+0000 7fdc0959a640 1 -- 172.21.15.83:0/1105040329 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc04034650 con 0x7fdbf40039b0 2024-03-06T07:23:28.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.815+0000 7fdc0959a640 1 -- 172.21.15.83:0/1105040329 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc04034a40 con 0x7fdbf40039b0 2024-03-06T07:23:28.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.815+0000 7fdc10f55640 1 -- 172.21.15.83:0/1105040329 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbf40039b0 msgr2=0x7fdbf4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.815+0000 7fdc10f55640 1 --2- 172.21.15.83:0/1105040329 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbf40039b0 0x7fdbf4003dd0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fdc040013d0 tx=0x7fdbf4005830 comp rx=0 tx=0).stop 2024-03-06T07:23:28.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.815+0000 7fdc0ad9d640 1 -- 172.21.15.83:0/1105040329 reap_dead start 2024-03-06T07:23:28.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.815+0000 7fdc10f55640 1 -- 172.21.15.83:0/1105040329 shutdown_connections 2024-03-06T07:23:28.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.815+0000 7fdc10f55640 1 -- 172.21.15.83:0/1105040329 >> 172.21.15.83:0/1105040329 conn(0x7fdc0c0d3d40 msgr2=0x7fdc0c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:28.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.815+0000 7fdc10f55640 1 -- 172.21.15.83:0/1105040329 shutdown_connections 2024-03-06T07:23:28.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.815+0000 7fdc10f55640 1 -- 172.21.15.83:0/1105040329 wait complete. 2024-03-06T07:23:28.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.817+0000 7fdc10f55640 1 Processor -- start 2024-03-06T07:23:28.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.817+0000 7fdc10f55640 1 -- start start 2024-03-06T07:23:28.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.818+0000 7fdc10f55640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdbf40039b0 0x7fdc0c0cb270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.818+0000 7fdc10f55640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc0c0cca10 0x7fdc0c1f7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.818+0000 7fdc10f55640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdc0c14d890 0x7fdc0c1f3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.818+0000 7fdc10f55640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c0b0910 con 0x7fdc0c0cca10 2024-03-06T07:23:28.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.818+0000 7fdc10f55640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c0cfc60 con 0x7fdbf40039b0 2024-03-06T07:23:28.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.818+0000 7fdc10f55640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c0d2020 con 0x7fdc0c14d890 2024-03-06T07:23:28.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.818+0000 7fdc09d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc0c0cca10 0x7fdc0c1f7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.818+0000 7fdc09d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc0c0cca10 0x7fdc0c1f7c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57298/0 (socket says 172.21.15.83:57298) 2024-03-06T07:23:28.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.818+0000 7fdc09d9b640 1 -- 172.21.15.83:0/2837555781 learned_addr learned my addr 172.21.15.83:0/2837555781 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:28.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.819+0000 7fdc0a59c640 1 --2- 172.21.15.83:0/2837555781 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdbf40039b0 0x7fdc0c0cb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.819+0000 7fdc0ad9d640 1 --2- 172.21.15.83:0/2837555781 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdc0c14d890 0x7fdc0c1f3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.819+0000 7fdc09d9b640 1 -- 172.21.15.83:0/2837555781 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdc0c14d890 msgr2=0x7fdc0c1f3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.819+0000 7fdc09d9b640 1 --2- 172.21.15.83:0/2837555781 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdc0c14d890 0x7fdc0c1f3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.819+0000 7fdc09d9b640 1 -- 172.21.15.83:0/2837555781 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdbf40039b0 msgr2=0x7fdc0c0cb270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.819+0000 7fdc09d9b640 1 --2- 172.21.15.83:0/2837555781 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdbf40039b0 0x7fdc0c0cb270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.819+0000 7fdc09d9b640 1 -- 172.21.15.83:0/2837555781 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc0c0cb970 con 0x7fdc0c0cca10 2024-03-06T07:23:28.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.819+0000 7fdc09d9b640 1 --2- 172.21.15.83:0/2837555781 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc0c0cca10 0x7fdc0c1f7c90 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fdc000118e0 tx=0x7fdc00011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:28.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.819+0000 7fdbfaffd640 1 -- 172.21.15.83:0/2837555781 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0000faf0 con 0x7fdc0c0cca10 2024-03-06T07:23:28.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.819+0000 7fdbfaffd640 1 -- 172.21.15.83:0/2837555781 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc0000fc90 con 0x7fdc0c0cca10 2024-03-06T07:23:28.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.819+0000 7fdc10f55640 1 -- 172.21.15.83:0/2837555781 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc0c0cbc50 con 0x7fdc0c0cca10 2024-03-06T07:23:28.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.819+0000 7fdbfaffd640 1 -- 172.21.15.83:0/2837555781 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc00014440 con 0x7fdc0c0cca10 2024-03-06T07:23:28.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.819+0000 7fdc10f55640 1 -- 172.21.15.83:0/2837555781 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc0c0cc210 con 0x7fdc0c0cca10 2024-03-06T07:23:28.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.820+0000 7fdc10f55640 1 -- 172.21.15.83:0/2837555781 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc0c068ff0 con 0x7fdc0c0cca10 2024-03-06T07:23:28.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.820+0000 7fdbfaffd640 1 -- 172.21.15.83:0/2837555781 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdc00012040 con 0x7fdc0c0cca10 2024-03-06T07:23:28.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.821+0000 7fdbfaffd640 1 --2- 172.21.15.83:0/2837555781 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdbec06b4d0 0x7fdbec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.821+0000 7fdc0a59c640 1 --2- 172.21.15.83:0/2837555781 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdbec06b4d0 0x7fdbec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.821+0000 7fdbfaffd640 1 -- 172.21.15.83:0/2837555781 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdc0008a810 con 0x7fdc0c0cca10 2024-03-06T07:23:28.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.821+0000 7fdc0a59c640 1 --2- 172.21.15.83:0/2837555781 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdbec06b4d0 0x7fdbec06d990 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fdc04034000 tx=0x7fdc04009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:28.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:28.825+0000 7fdbfaffd640 1 -- 172.21.15.83:0/2837555781 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdc00019030 con 0x7fdc0c0cca10 2024-03-06T07:23:29.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.003+0000 7fdc10f55640 1 -- 172.21.15.83:0/2837555781 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdc0c0d3440 con 0x7fdc0c0cca10 2024-03-06T07:23:29.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.004+0000 7fdbfaffd640 1 -- 172.21.15.83:0/2837555781 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v33) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdc00059040 con 0x7fdc0c0cca10 2024-03-06T07:23:29.006 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 64 2024-03-06T07:23:29.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.005+0000 7fdc10f55640 1 -- 172.21.15.83:0/2837555781 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdbec06b4d0 msgr2=0x7fdbec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.005+0000 7fdc10f55640 1 --2- 172.21.15.83:0/2837555781 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdbec06b4d0 0x7fdbec06d990 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fdc04034000 tx=0x7fdc04009030 comp rx=0 tx=0).stop 2024-03-06T07:23:29.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.005+0000 7fdc10f55640 1 -- 172.21.15.83:0/2837555781 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc0c0cca10 msgr2=0x7fdc0c1f7c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.005+0000 7fdc10f55640 1 --2- 172.21.15.83:0/2837555781 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc0c0cca10 0x7fdc0c1f7c90 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fdc000118e0 tx=0x7fdc00011da0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.005+0000 7fdc10f55640 1 -- 172.21.15.83:0/2837555781 shutdown_connections 2024-03-06T07:23:29.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.005+0000 7fdc10f55640 1 --2- 172.21.15.83:0/2837555781 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdbec06b4d0 0x7fdbec06d990 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.006+0000 7fdc10f55640 1 --2- 172.21.15.83:0/2837555781 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdc0c14d890 0x7fdc0c1f3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.006+0000 7fdc10f55640 1 --2- 172.21.15.83:0/2837555781 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc0c0cca10 0x7fdc0c1f7c90 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.006+0000 7fdc10f55640 1 --2- 172.21.15.83:0/2837555781 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdbf40039b0 0x7fdc0c0cb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.006+0000 7fdc10f55640 1 -- 172.21.15.83:0/2837555781 >> 172.21.15.83:0/2837555781 conn(0x7fdc0c0d3d40 msgr2=0x7fdc0c06d070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:29.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.006+0000 7fdc10f55640 1 -- 172.21.15.83:0/2837555781 shutdown_connections 2024-03-06T07:23:29.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.006+0000 7fdc10f55640 1 -- 172.21.15.83:0/2837555781 wait complete. 2024-03-06T07:23:29.018 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:29.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.141+0000 7f14c7a37640 1 Processor -- start 2024-03-06T07:23:29.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.141+0000 7f14c7a37640 1 -- start start 2024-03-06T07:23:29.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.141+0000 7f14c7a37640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f14c015bdd0 0x7f14c00c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.141+0000 7f14c7a37640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14c00cebe0 con 0x7f14c015fa10 2024-03-06T07:23:29.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.141+0000 7f14c7a37640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14c00ceec0 con 0x7f14c00d16e0 2024-03-06T07:23:29.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.141+0000 7f14c7a37640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14c00ce980 con 0x7f14c015bdd0 2024-03-06T07:23:29.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.142+0000 7f14c57d3640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f14c015bdd0 0x7f14c00c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.142+0000 7f14c4fd2640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f14c015fa10 0x7f14c00ce270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49028/0 (socket says 172.21.15.83:49028) 2024-03-06T07:23:29.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.142+0000 7f14c4fd2640 1 -- 172.21.15.83:0/3889344867 learned_addr learned my addr 172.21.15.83:0/3889344867 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:23:29.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.142+0000 7f14c57d3640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f14c015bdd0 0x7f14c00c5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57308/0 (socket says 172.21.15.83:57308) 2024-03-06T07:23:29.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.142+0000 7f14c57d3640 1 -- 172.21.15.83:0/3889344867 >> v1:172.21.15.183:6789/0 conn(0x7f14c00d16e0 legacy=0x7f14c00d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:29.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.142+0000 7f14c57d3640 1 -- 172.21.15.83:0/3889344867 >> v1:172.21.15.83:6789/0 conn(0x7f14c015fa10 legacy=0x7f14c00ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.142+0000 7f14c57d3640 1 -- 172.21.15.83:0/3889344867 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c00cf1a0 con 0x7f14c015bdd0 2024-03-06T07:23:29.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.143+0000 7f14c57d3640 1 --2- 172.21.15.83:0/3889344867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14c015bdd0 0x7f14c00c5610 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f14bc0097a0 tx=0x7f14bc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f147d92e02eb1aed server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.143+0000 7f14b7fff640 1 -- 172.21.15.83:0/3889344867 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14bc0315a0 con 0x7f14c015bdd0 2024-03-06T07:23:29.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.143+0000 7f14b7fff640 1 -- 172.21.15.83:0/3889344867 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14bc037070 con 0x7f14c015bdd0 2024-03-06T07:23:29.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.143+0000 7f14b7fff640 1 -- 172.21.15.83:0/3889344867 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14bc031960 con 0x7f14c015bdd0 2024-03-06T07:23:29.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.143+0000 7f14c7a37640 1 -- 172.21.15.83:0/3889344867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14c015bdd0 msgr2=0x7f14c00c5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.143+0000 7f14c7a37640 1 --2- 172.21.15.83:0/3889344867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14c015bdd0 0x7f14c00c5610 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f14bc0097a0 tx=0x7f14bc02b840 comp rx=0 tx=0).stop 2024-03-06T07:23:29.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.143+0000 7f14c7a37640 1 -- 172.21.15.83:0/3889344867 shutdown_connections 2024-03-06T07:23:29.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.143+0000 7f14c7a37640 1 --2- 172.21.15.83:0/3889344867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14c015bdd0 0x7f14c00c5610 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.143+0000 7f14c7a37640 1 -- 172.21.15.83:0/3889344867 >> 172.21.15.83:0/3889344867 conn(0x7f14c009a9f0 msgr2=0x7f14c009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:29.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.143+0000 7f14c7a37640 1 -- 172.21.15.83:0/3889344867 shutdown_connections 2024-03-06T07:23:29.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.144+0000 7f14c7a37640 1 -- 172.21.15.83:0/3889344867 wait complete. 2024-03-06T07:23:29.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.145+0000 7f14c7a37640 1 Processor -- start 2024-03-06T07:23:29.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.145+0000 7f14c7a37640 1 -- start start 2024-03-06T07:23:29.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.145+0000 7f14c7a37640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14c00d16e0 0x7f14c015c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.145+0000 7f14c7a37640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f14c015bdd0 0x7f14c015a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.145+0000 7f14c7a37640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14c015fa10 0x7f14c00c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.145+0000 7f14c7a37640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c00cc5d0 con 0x7f14c015fa10 2024-03-06T07:23:29.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.145+0000 7f14c7a37640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c00c5b50 con 0x7f14c00d16e0 2024-03-06T07:23:29.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.145+0000 7f14c7a37640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14c015c1b0 con 0x7f14c015bdd0 2024-03-06T07:23:29.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.145+0000 7f14c5fd4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14c015fa10 0x7f14c00c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.145+0000 7f14c5fd4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14c015fa10 0x7f14c00c8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57320/0 (socket says 172.21.15.83:57320) 2024-03-06T07:23:29.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.145+0000 7f14c5fd4640 1 -- 172.21.15.83:0/2293290070 learned_addr learned my addr 172.21.15.83:0/2293290070 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:29.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.146+0000 7f14c57d3640 1 --2- 172.21.15.83:0/2293290070 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14c00d16e0 0x7f14c015c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.146+0000 7f14c4fd2640 1 --2- 172.21.15.83:0/2293290070 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f14c015bdd0 0x7f14c015a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.146+0000 7f14c5fd4640 1 -- 172.21.15.83:0/2293290070 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f14c015bdd0 msgr2=0x7f14c015a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.146+0000 7f14c5fd4640 1 --2- 172.21.15.83:0/2293290070 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f14c015bdd0 0x7f14c015a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.146+0000 7f14c5fd4640 1 -- 172.21.15.83:0/2293290070 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14c00d16e0 msgr2=0x7f14c015c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.146+0000 7f14c5fd4640 1 --2- 172.21.15.83:0/2293290070 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14c00d16e0 0x7f14c015c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.146+0000 7f14c5fd4640 1 -- 172.21.15.83:0/2293290070 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c00c90d0 con 0x7f14c015fa10 2024-03-06T07:23:29.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.146+0000 7f14c57d3640 1 --2- 172.21.15.83:0/2293290070 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14c00d16e0 0x7f14c015c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:29.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.146+0000 7f14c5fd4640 1 --2- 172.21.15.83:0/2293290070 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14c015fa10 0x7f14c00c8b90 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f14b800c780 tx=0x7f14b800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.146+0000 7f14b5ffb640 1 -- 172.21.15.83:0/2293290070 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b8017070 con 0x7f14c015fa10 2024-03-06T07:23:29.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.146+0000 7f14b5ffb640 1 -- 172.21.15.83:0/2293290070 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14b8003da0 con 0x7f14c015fa10 2024-03-06T07:23:29.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.146+0000 7f14c7a37640 1 -- 172.21.15.83:0/2293290070 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14c015dba0 con 0x7f14c015fa10 2024-03-06T07:23:29.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.146+0000 7f14b5ffb640 1 -- 172.21.15.83:0/2293290070 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b80052b0 con 0x7f14c015fa10 2024-03-06T07:23:29.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.146+0000 7f14c7a37640 1 -- 172.21.15.83:0/2293290070 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14c015ede0 con 0x7f14c015fa10 2024-03-06T07:23:29.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.147+0000 7f14c7a37640 1 -- 172.21.15.83:0/2293290070 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14c004f330 con 0x7f14c015fa10 2024-03-06T07:23:29.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.148+0000 7f14b5ffb640 1 -- 172.21.15.83:0/2293290070 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f14b801a070 con 0x7f14c015fa10 2024-03-06T07:23:29.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.148+0000 7f14b5ffb640 1 --2- 172.21.15.83:0/2293290070 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f14a406b500 0x7f14a406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.148+0000 7f14b5ffb640 1 -- 172.21.15.83:0/2293290070 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f14b8089880 con 0x7f14c015fa10 2024-03-06T07:23:29.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.148+0000 7f14c57d3640 1 --2- 172.21.15.83:0/2293290070 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f14a406b500 0x7f14a406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.149+0000 7f14c57d3640 1 --2- 172.21.15.83:0/2293290070 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f14a406b500 0x7f14a406d9c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f14bc036000 tx=0x7f14bc03d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.153+0000 7f14b5ffb640 1 -- 172.21.15.83:0/2293290070 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f14b80586e0 con 0x7f14c015fa10 2024-03-06T07:23:29.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.364+0000 7f14c7a37640 1 -- 172.21.15.83:0/2293290070 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14c00b7a40 con 0x7f14c015fa10 2024-03-06T07:23:29.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.364+0000 7f14b5ffb640 1 -- 172.21.15.83:0/2293290070 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f14b8089030 con 0x7f14c015fa10 2024-03-06T07:23:29.366 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:23:29.367 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4799,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/3542419057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3542419057},{"type":"v1","addr":"172.21.15.83:6836","nonce":3542419057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":5023,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1853869906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":1853869906},{"type":"v1","addr":"172.21.15.183:6835","nonce":1853869906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5026,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/3666667479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3666667479},{"type":"v1","addr":"172.21.15.183:6834","nonce":3666667479}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:00.145321+0000","modified":"2024-03-06T07:23:02.319419+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5244":{"gid":5244,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/2668925454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":2668925454},{"type":"v1","addr":"172.21.15.83:6837","nonce":2668925454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5244,"qdb_cluster":[5244]},"id":2}]} 2024-03-06T07:23:29.367 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 13 2024-03-06T07:23:29.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.366+0000 7f14c7a37640 1 -- 172.21.15.83:0/2293290070 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f14a406b500 msgr2=0x7f14a406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.366+0000 7f14c7a37640 1 --2- 172.21.15.83:0/2293290070 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f14a406b500 0x7f14a406d9c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f14bc036000 tx=0x7f14bc03d040 comp rx=0 tx=0).stop 2024-03-06T07:23:29.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.366+0000 7f14c7a37640 1 -- 172.21.15.83:0/2293290070 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14c015fa10 msgr2=0x7f14c00c8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.366+0000 7f14c7a37640 1 --2- 172.21.15.83:0/2293290070 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14c015fa10 0x7f14c00c8b90 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f14b800c780 tx=0x7f14b800cc40 comp rx=0 tx=0).stop 2024-03-06T07:23:29.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.366+0000 7f14c7a37640 1 -- 172.21.15.83:0/2293290070 shutdown_connections 2024-03-06T07:23:29.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.366+0000 7f14c7a37640 1 --2- 172.21.15.83:0/2293290070 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f14a406b500 0x7f14a406d9c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.366+0000 7f14c7a37640 1 --2- 172.21.15.83:0/2293290070 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14c015fa10 0x7f14c00c8b90 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.366+0000 7f14c7a37640 1 --2- 172.21.15.83:0/2293290070 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f14c015bdd0 0x7f14c015a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.366+0000 7f14c7a37640 1 --2- 172.21.15.83:0/2293290070 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14c00d16e0 0x7f14c015c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.366+0000 7f14c7a37640 1 -- 172.21.15.83:0/2293290070 >> 172.21.15.83:0/2293290070 conn(0x7f14c009a9f0 msgr2=0x7f14c00cce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:29.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.367+0000 7f14c7a37640 1 -- 172.21.15.83:0/2293290070 shutdown_connections 2024-03-06T07:23:29.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.367+0000 7f14c7a37640 1 -- 172.21.15.83:0/2293290070 wait complete. 2024-03-06T07:23:29.379 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:29.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.478+0000 7fa5ecf3a640 1 Processor -- start 2024-03-06T07:23:29.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.478+0000 7fa5ecf3a640 1 -- start start 2024-03-06T07:23:29.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.479+0000 7fa5ecf3a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa5e815dfe0 0x7fa5e80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.479+0000 7fa5ecf3a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fa5e80d29c0 0x7fa5e80b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.479+0000 7fa5ecf3a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5e80b3430 con 0x7fa5e80cf830 2024-03-06T07:23:29.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.479+0000 7fa5ecf3a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5e80b31e0 con 0x7fa5e815dfe0 2024-03-06T07:23:29.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.479+0000 7fa5ecf3a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5e80b3d60 con 0x7fa5e80d29c0 2024-03-06T07:23:29.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.479+0000 7fa5e659c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa5e815dfe0 0x7fa5e80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.479+0000 7fa5e5d9b640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fa5e80cf830 0x7fa5e80d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49036/0 (socket says 172.21.15.83:49036) 2024-03-06T07:23:29.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.479+0000 7fa5e5d9b640 1 -- 172.21.15.83:0/2636756482 learned_addr learned my addr 172.21.15.83:0/2636756482 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:23:29.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.480+0000 7fa5e6d9d640 1 --2- 172.21.15.83:0/2636756482 >> v2:172.21.15.183:3300/0 conn(0x7fa5e80d29c0 0x7fa5e80b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.480+0000 7fa5e659c640 1 -- 172.21.15.83:0/2636756482 >> v2:172.21.15.183:3300/0 conn(0x7fa5e80d29c0 msgr2=0x7fa5e80b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:29.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.480+0000 7fa5e659c640 1 --2- 172.21.15.83:0/2636756482 >> v2:172.21.15.183:3300/0 conn(0x7fa5e80d29c0 0x7fa5e80b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.480+0000 7fa5e659c640 1 -- 172.21.15.83:0/2636756482 >> v1:172.21.15.83:6789/0 conn(0x7fa5e80cf830 legacy=0x7fa5e80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.480+0000 7fa5e659c640 1 -- 172.21.15.83:0/2636756482 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e80b3710 con 0x7fa5e815dfe0 2024-03-06T07:23:29.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.480+0000 7fa5e559a640 1 -- 172.21.15.83:0/2636756482 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2687657768 0 0) 0x7fa5e80b3430 con 0x7fa5e80cf830 2024-03-06T07:23:29.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.480+0000 7fa5e6d9d640 1 --2- 172.21.15.83:0/2636756482 >> v2:172.21.15.183:3300/0 conn(0x7fa5e80d29c0 0x7fa5e80b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:29.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.480+0000 7fa5e659c640 1 --2- 172.21.15.83:0/2636756482 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5e815dfe0 0x7fa5e80cf2f0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fa5e80b3cd0 tx=0x7fa5e002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a54ac18188a3eaac server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.481+0000 7fa5e559a640 1 -- 172.21.15.83:0/2636756482 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e0039070 con 0x7fa5e815dfe0 2024-03-06T07:23:29.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.481+0000 7fa5e559a640 1 -- 172.21.15.83:0/2636756482 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5e002cc80 con 0x7fa5e815dfe0 2024-03-06T07:23:29.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.481+0000 7fa5e559a640 1 -- 172.21.15.83:0/2636756482 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e0034660 con 0x7fa5e815dfe0 2024-03-06T07:23:29.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.481+0000 7fa5ecf3a640 1 -- 172.21.15.83:0/2636756482 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5e815dfe0 msgr2=0x7fa5e80cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.481+0000 7fa5ecf3a640 1 --2- 172.21.15.83:0/2636756482 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5e815dfe0 0x7fa5e80cf2f0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fa5e80b3cd0 tx=0x7fa5e002bed0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.481+0000 7fa5ecf3a640 1 -- 172.21.15.83:0/2636756482 shutdown_connections 2024-03-06T07:23:29.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.481+0000 7fa5ecf3a640 1 --2- 172.21.15.83:0/2636756482 >> v2:172.21.15.183:3300/0 conn(0x7fa5e80d29c0 0x7fa5e80b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.481+0000 7fa5ecf3a640 1 --2- 172.21.15.83:0/2636756482 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5e815dfe0 0x7fa5e80cf2f0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.481+0000 7fa5ecf3a640 1 -- 172.21.15.83:0/2636756482 >> 172.21.15.83:0/2636756482 conn(0x7fa5e80d3cd0 msgr2=0x7fa5e80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:29.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.481+0000 7fa5ecf3a640 1 -- 172.21.15.83:0/2636756482 shutdown_connections 2024-03-06T07:23:29.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.481+0000 7fa5ecf3a640 1 -- 172.21.15.83:0/2636756482 wait complete. 2024-03-06T07:23:29.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.483+0000 7fa5ecf3a640 1 Processor -- start 2024-03-06T07:23:29.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.483+0000 7fa5ecf3a640 1 -- start start 2024-03-06T07:23:29.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.483+0000 7fa5ecf3a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5e80cf830 0x7fa5e81f4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.483+0000 7fa5ecf3a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa5e80d29c0 0x7fa5e80cd430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.483+0000 7fa5ecf3a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5e815dfe0 0x7fa5e814b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.483+0000 7fa5ecf3a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e80d1df0 con 0x7fa5e815dfe0 2024-03-06T07:23:29.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.483+0000 7fa5ecf3a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e80d1c70 con 0x7fa5e80cf830 2024-03-06T07:23:29.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.483+0000 7fa5ecf3a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e815e3c0 con 0x7fa5e80d29c0 2024-03-06T07:23:29.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.483+0000 7fa5e6d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5e815dfe0 0x7fa5e814b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.483+0000 7fa5e6d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5e815dfe0 0x7fa5e814b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57348/0 (socket says 172.21.15.83:57348) 2024-03-06T07:23:29.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.483+0000 7fa5e6d9d640 1 -- 172.21.15.83:0/1187510830 learned_addr learned my addr 172.21.15.83:0/1187510830 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:29.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.484+0000 7fa5e5d9b640 1 --2- 172.21.15.83:0/1187510830 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa5e80d29c0 0x7fa5e80cd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.484+0000 7fa5e659c640 1 --2- 172.21.15.83:0/1187510830 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5e80cf830 0x7fa5e81f4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.484+0000 7fa5e6d9d640 1 -- 172.21.15.83:0/1187510830 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa5e80d29c0 msgr2=0x7fa5e80cd430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.484+0000 7fa5e6d9d640 1 --2- 172.21.15.83:0/1187510830 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa5e80d29c0 0x7fa5e80cd430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.484+0000 7fa5e6d9d640 1 -- 172.21.15.83:0/1187510830 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5e80cf830 msgr2=0x7fa5e81f4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.484+0000 7fa5e6d9d640 1 --2- 172.21.15.83:0/1187510830 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5e80cf830 0x7fa5e81f4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.484+0000 7fa5e6d9d640 1 -- 172.21.15.83:0/1187510830 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e814e1d0 con 0x7fa5e815dfe0 2024-03-06T07:23:29.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.484+0000 7fa5e659c640 1 --2- 172.21.15.83:0/1187510830 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5e80cf830 0x7fa5e81f4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:29.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.484+0000 7fa5e6d9d640 1 --2- 172.21.15.83:0/1187510830 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5e815dfe0 0x7fa5e814b330 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fa5d800c7f0 tx=0x7fa5d800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.484+0000 7fa5deffd640 1 -- 172.21.15.83:0/1187510830 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d8015070 con 0x7fa5e815dfe0 2024-03-06T07:23:29.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.484+0000 7fa5deffd640 1 -- 172.21.15.83:0/1187510830 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5d8007820 con 0x7fa5e815dfe0 2024-03-06T07:23:29.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.484+0000 7fa5ecf3a640 1 -- 172.21.15.83:0/1187510830 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5e814e4b0 con 0x7fa5e815dfe0 2024-03-06T07:23:29.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.484+0000 7fa5deffd640 1 -- 172.21.15.83:0/1187510830 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d8007c50 con 0x7fa5e815dfe0 2024-03-06T07:23:29.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.484+0000 7fa5ecf3a640 1 -- 172.21.15.83:0/1187510830 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5e814ea70 con 0x7fa5e815dfe0 2024-03-06T07:23:29.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.485+0000 7fa5ecf3a640 1 -- 172.21.15.83:0/1187510830 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5e80d3160 con 0x7fa5e815dfe0 2024-03-06T07:23:29.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.486+0000 7fa5deffd640 1 -- 172.21.15.83:0/1187510830 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa5d8018070 con 0x7fa5e815dfe0 2024-03-06T07:23:29.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.486+0000 7fa5deffd640 1 --2- 172.21.15.83:0/1187510830 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa5c806b4d0 0x7fa5c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.486+0000 7fa5deffd640 1 -- 172.21.15.83:0/1187510830 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa5d808a0d0 con 0x7fa5e815dfe0 2024-03-06T07:23:29.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.486+0000 7fa5e659c640 1 --2- 172.21.15.83:0/1187510830 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa5c806b4d0 0x7fa5c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.487+0000 7fa5e659c640 1 --2- 172.21.15.83:0/1187510830 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa5c806b4d0 0x7fa5c806d990 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa5e0036000 tx=0x7fa5e0030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.491+0000 7fa5deffd640 1 -- 172.21.15.83:0/1187510830 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa5e80d3160 con 0x7fa5e815dfe0 2024-03-06T07:23:29.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.670+0000 7fa5ecf3a640 1 -- 172.21.15.83:0/1187510830 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5e80d00c0 con 0x7fa5e815dfe0 2024-03-06T07:23:29.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.671+0000 7fa5deffd640 1 -- 172.21.15.83:0/1187510830 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16681 (secure 0 0 0) 0x7fa5d8058fe0 con 0x7fa5e815dfe0 2024-03-06T07:23:29.673 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:23:29.674 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":33,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:23:01.812855+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:58.568071+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:59.116376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:29.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.673+0000 7fa5ecf3a640 1 -- 172.21.15.83:0/1187510830 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa5c806b4d0 msgr2=0x7fa5c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.673+0000 7fa5ecf3a640 1 --2- 172.21.15.83:0/1187510830 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa5c806b4d0 0x7fa5c806d990 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa5e0036000 tx=0x7fa5e0030000 comp rx=0 tx=0).stop 2024-03-06T07:23:29.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.673+0000 7fa5ecf3a640 1 -- 172.21.15.83:0/1187510830 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5e815dfe0 msgr2=0x7fa5e814b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.673+0000 7fa5ecf3a640 1 --2- 172.21.15.83:0/1187510830 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5e815dfe0 0x7fa5e814b330 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fa5d800c7f0 tx=0x7fa5d800ccb0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.673+0000 7fa5ecf3a640 1 -- 172.21.15.83:0/1187510830 shutdown_connections 2024-03-06T07:23:29.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.673+0000 7fa5ecf3a640 1 --2- 172.21.15.83:0/1187510830 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa5c806b4d0 0x7fa5c806d990 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.673+0000 7fa5ecf3a640 1 --2- 172.21.15.83:0/1187510830 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5e815dfe0 0x7fa5e814b330 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.673+0000 7fa5ecf3a640 1 --2- 172.21.15.83:0/1187510830 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa5e80d29c0 0x7fa5e80cd430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.673+0000 7fa5ecf3a640 1 --2- 172.21.15.83:0/1187510830 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5e80cf830 0x7fa5e81f4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.673+0000 7fa5ecf3a640 1 -- 172.21.15.83:0/1187510830 >> 172.21.15.83:0/1187510830 conn(0x7fa5e80d3cd0 msgr2=0x7fa5e80d05e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:29.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.673+0000 7fa5ecf3a640 1 -- 172.21.15.83:0/1187510830 shutdown_connections 2024-03-06T07:23:29.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.673+0000 7fa5ecf3a640 1 -- 172.21.15.83:0/1187510830 wait complete. 2024-03-06T07:23:29.686 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:23:29.687 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:29.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.786+0000 7f758fa8f640 1 Processor -- start 2024-03-06T07:23:29.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.786+0000 7f758fa8f640 1 -- start start 2024-03-06T07:23:29.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.787+0000 7f758fa8f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f75880b4170 0x7f75880b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.787+0000 7f758fa8f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f75880d4760 0x7f75880d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.787+0000 7f758fa8f640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f75880b4dd0 0x7f75880d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.787+0000 7f758fa8f640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75880d4bf0 con 0x7f75880b4170 2024-03-06T07:23:29.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.787+0000 7f758fa8f640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75880ce9e0 con 0x7f75880b4dd0 2024-03-06T07:23:29.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.787+0000 7f758fa8f640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75880d25d0 con 0x7f75880d4760 2024-03-06T07:23:29.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.787+0000 7f758d82b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f75880b4170 0x7f75880b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.787+0000 7f758d82b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f75880b4170 0x7f75880b6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57364/0 (socket says 172.21.15.83:57364) 2024-03-06T07:23:29.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.787+0000 7f758d82b640 1 -- 172.21.15.83:0/3325041456 learned_addr learned my addr 172.21.15.83:0/3325041456 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:29.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.787+0000 7f758d02a640 1 --2- 172.21.15.83:0/3325041456 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f75880d4760 0x7f75880d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.787+0000 7f758e02c640 1 --2- 172.21.15.83:0/3325041456 >> v2:172.21.15.183:3300/0 conn(0x7f75880b4dd0 0x7f75880d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.788+0000 7f758d82b640 1 -- 172.21.15.83:0/3325041456 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f75880d4760 msgr2=0x7f75880d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:29.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.788+0000 7f758d82b640 1 --2- 172.21.15.83:0/3325041456 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f75880d4760 0x7f75880d0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.788+0000 7f758d82b640 1 -- 172.21.15.83:0/3325041456 >> v2:172.21.15.183:3300/0 conn(0x7f75880b4dd0 msgr2=0x7f75880d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:29.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.788+0000 7f758d82b640 1 --2- 172.21.15.83:0/3325041456 >> v2:172.21.15.183:3300/0 conn(0x7f75880b4dd0 0x7f75880d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.788+0000 7f758d82b640 1 -- 172.21.15.83:0/3325041456 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75880d2870 con 0x7f75880b4170 2024-03-06T07:23:29.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.788+0000 7f758d82b640 1 --2- 172.21.15.83:0/3325041456 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f75880b4170 0x7f75880b6a90 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f75880cffb0 tx=0x7f758002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76984cf6b76a664b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.788+0000 7f758c829640 1 -- 172.21.15.83:0/3325041456 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f758002ca70 con 0x7f75880b4170 2024-03-06T07:23:29.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.788+0000 7f758c829640 1 -- 172.21.15.83:0/3325041456 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f758002cc10 con 0x7f75880b4170 2024-03-06T07:23:29.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.788+0000 7f758c829640 1 -- 172.21.15.83:0/3325041456 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75800343f0 con 0x7f75880b4170 2024-03-06T07:23:29.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.789+0000 7f758fa8f640 1 -- 172.21.15.83:0/3325041456 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f75880b4170 msgr2=0x7f75880b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.789+0000 7f758fa8f640 1 --2- 172.21.15.83:0/3325041456 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f75880b4170 0x7f75880b6a90 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f75880cffb0 tx=0x7f758002bed0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.789+0000 7f758fa8f640 1 -- 172.21.15.83:0/3325041456 shutdown_connections 2024-03-06T07:23:29.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.789+0000 7f758fa8f640 1 --2- 172.21.15.83:0/3325041456 >> v2:172.21.15.183:3300/0 conn(0x7f75880b4dd0 0x7f75880d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.789+0000 7f758fa8f640 1 --2- 172.21.15.83:0/3325041456 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f75880d4760 0x7f75880d0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.789+0000 7f758fa8f640 1 --2- 172.21.15.83:0/3325041456 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f75880b4170 0x7f75880b6a90 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.789+0000 7f758fa8f640 1 -- 172.21.15.83:0/3325041456 >> 172.21.15.83:0/3325041456 conn(0x7f75880d3c40 msgr2=0x7f75880d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:29.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.789+0000 7f758fa8f640 1 -- 172.21.15.83:0/3325041456 shutdown_connections 2024-03-06T07:23:29.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.789+0000 7f758fa8f640 1 -- 172.21.15.83:0/3325041456 wait complete. 2024-03-06T07:23:29.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.790+0000 7f758fa8f640 1 Processor -- start 2024-03-06T07:23:29.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.790+0000 7f758fa8f640 1 -- start start 2024-03-06T07:23:29.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.791+0000 7f758fa8f640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f75880b4170 0x7f75880a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.791+0000 7f758fa8f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f75880b4dd0 0x7f758809e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.791+0000 7f758fa8f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7588160430 0x7f75880ccee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.791+0000 7f758fa8f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75880b47d0 con 0x7f7588160430 2024-03-06T07:23:29.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.791+0000 7f758fa8f640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75880b4550 con 0x7f75880b4170 2024-03-06T07:23:29.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.791+0000 7f758fa8f640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75880b4950 con 0x7f75880b4dd0 2024-03-06T07:23:29.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.791+0000 7f758e02c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7588160430 0x7f75880ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.791+0000 7f758e02c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7588160430 0x7f75880ccee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57374/0 (socket says 172.21.15.83:57374) 2024-03-06T07:23:29.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.791+0000 7f758e02c640 1 -- 172.21.15.83:0/2286396111 learned_addr learned my addr 172.21.15.83:0/2286396111 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:29.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.791+0000 7f758e02c640 1 -- 172.21.15.83:0/2286396111 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f75880b4dd0 msgr2=0x7f758809e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.791+0000 7f758e02c640 1 --2- 172.21.15.83:0/2286396111 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f75880b4dd0 0x7f758809e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.791+0000 7f758e02c640 1 -- 172.21.15.83:0/2286396111 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f75880b4170 msgr2=0x7f75880a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.791+0000 7f758e02c640 1 --2- 172.21.15.83:0/2286396111 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f75880b4170 0x7f75880a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.791+0000 7f758e02c640 1 -- 172.21.15.83:0/2286396111 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f758809f6c0 con 0x7f7588160430 2024-03-06T07:23:29.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.792+0000 7f758e02c640 1 --2- 172.21.15.83:0/2286396111 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7588160430 0x7f75880ccee0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f758400d830 tx=0x7f758400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.792+0000 7f757e7fc640 1 -- 172.21.15.83:0/2286396111 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7584018020 con 0x7f7588160430 2024-03-06T07:23:29.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.792+0000 7f757e7fc640 1 -- 172.21.15.83:0/2286396111 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7584009660 con 0x7f7588160430 2024-03-06T07:23:29.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.792+0000 7f757e7fc640 1 -- 172.21.15.83:0/2286396111 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7584009a30 con 0x7f7588160430 2024-03-06T07:23:29.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.792+0000 7f758fa8f640 1 -- 172.21.15.83:0/2286396111 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f758809eb40 con 0x7f7588160430 2024-03-06T07:23:29.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.792+0000 7f758fa8f640 1 -- 172.21.15.83:0/2286396111 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f758809f100 con 0x7f7588160430 2024-03-06T07:23:29.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.793+0000 7f758fa8f640 1 -- 172.21.15.83:0/2286396111 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f758804f330 con 0x7f7588160430 2024-03-06T07:23:29.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.793+0000 7f757e7fc640 1 -- 172.21.15.83:0/2286396111 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f758401b020 con 0x7f7588160430 2024-03-06T07:23:29.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.794+0000 7f757e7fc640 1 --2- 172.21.15.83:0/2286396111 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f756c06b4d0 0x7f756c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.794+0000 7f757e7fc640 1 -- 172.21.15.83:0/2286396111 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f758408a8f0 con 0x7f7588160430 2024-03-06T07:23:29.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.794+0000 7f758d82b640 1 --2- 172.21.15.83:0/2286396111 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f756c06b4d0 0x7f756c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.794+0000 7f758d82b640 1 --2- 172.21.15.83:0/2286396111 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f756c06b4d0 0x7f756c06d990 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7580035000 tx=0x7f758002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:29.798+0000 7f757e7fc640 1 -- 172.21.15.83:0/2286396111 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7584022020 con 0x7f7588160430 2024-03-06T07:23:30.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.014+0000 7f758fa8f640 1 -- 172.21.15.83:0/2286396111 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75880b39c0 con 0x7f7588160430 2024-03-06T07:23:30.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.015+0000 7f757e7fc640 1 -- 172.21.15.83:0/2286396111 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f7584011340 con 0x7f7588160430 2024-03-06T07:23:30.017 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:23:30.017 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4799,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/3542419057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3542419057},{"type":"v1","addr":"172.21.15.83:6836","nonce":3542419057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":5023,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1853869906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":1853869906},{"type":"v1","addr":"172.21.15.183:6835","nonce":1853869906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5026,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/3666667479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3666667479},{"type":"v1","addr":"172.21.15.183:6834","nonce":3666667479}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:00.145321+0000","modified":"2024-03-06T07:23:02.319419+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5244":{"gid":5244,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/2668925454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":2668925454},{"type":"v1","addr":"172.21.15.83:6837","nonce":2668925454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5244,"qdb_cluster":[5244]},"id":2}]} 2024-03-06T07:23:30.017 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 13 2024-03-06T07:23:30.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.017+0000 7f758fa8f640 1 -- 172.21.15.83:0/2286396111 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f756c06b4d0 msgr2=0x7f756c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.017+0000 7f758fa8f640 1 --2- 172.21.15.83:0/2286396111 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f756c06b4d0 0x7f756c06d990 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7580035000 tx=0x7f758002e000 comp rx=0 tx=0).stop 2024-03-06T07:23:30.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.017+0000 7f758fa8f640 1 -- 172.21.15.83:0/2286396111 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7588160430 msgr2=0x7f75880ccee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.017+0000 7f758fa8f640 1 --2- 172.21.15.83:0/2286396111 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7588160430 0x7f75880ccee0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f758400d830 tx=0x7f758400dcf0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.017+0000 7f758fa8f640 1 -- 172.21.15.83:0/2286396111 shutdown_connections 2024-03-06T07:23:30.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.017+0000 7f758fa8f640 1 --2- 172.21.15.83:0/2286396111 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f756c06b4d0 0x7f756c06d990 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.017+0000 7f758fa8f640 1 --2- 172.21.15.83:0/2286396111 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7588160430 0x7f75880ccee0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.017+0000 7f758fa8f640 1 --2- 172.21.15.83:0/2286396111 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f75880b4dd0 0x7f758809e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.017+0000 7f758fa8f640 1 --2- 172.21.15.83:0/2286396111 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f75880b4170 0x7f75880a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.017+0000 7f758fa8f640 1 -- 172.21.15.83:0/2286396111 >> 172.21.15.83:0/2286396111 conn(0x7f75880d3c40 msgr2=0x7f75880b7130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:30.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.017+0000 7f758fa8f640 1 -- 172.21.15.83:0/2286396111 shutdown_connections 2024-03-06T07:23:30.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.017+0000 7f758fa8f640 1 -- 172.21.15.83:0/2286396111 wait complete. 2024-03-06T07:23:30.030 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:30.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.130+0000 7f6c63979640 1 Processor -- start 2024-03-06T07:23:30.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.130+0000 7f6c63979640 1 -- start start 2024-03-06T07:23:30.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.131+0000 7f6c63979640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f6c5c0b4170 0x7f6c5c0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.131+0000 7f6c63979640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6c5c0d0a80 0x7f6c5c0cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.131+0000 7f6c63979640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c5c0d25d0 con 0x7f6c5c0d4760 2024-03-06T07:23:30.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.131+0000 7f6c63979640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c5c0d4bf0 con 0x7f6c5c0b4170 2024-03-06T07:23:30.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.131+0000 7f6c63979640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5c0ce9e0 con 0x7f6c5c0d0a80 2024-03-06T07:23:30.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.131+0000 7f6c61715640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f6c5c0b4170 0x7f6c5c0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.131+0000 7f6c61f16640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6c5c0d0a80 0x7f6c5c0cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.131+0000 7f6c61715640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f6c5c0b4170 0x7f6c5c0b6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:44812/0 (socket says 172.21.15.83:44812) 2024-03-06T07:23:30.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.131+0000 7f6c61715640 1 -- 172.21.15.83:0/3761624396 learned_addr learned my addr 172.21.15.83:0/3761624396 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:30.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.131+0000 7f6c60f14640 1 --1- 172.21.15.83:0/3761624396 >> v1:172.21.15.183:6789/0 conn(0x7f6c5c0d4760 0x7f6c5c0d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:49116/0 (socket says 172.21.15.83:49116) 2024-03-06T07:23:30.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.132+0000 7f6c61f16640 1 -- 172.21.15.83:0/3761624396 >> v2:172.21.15.183:3300/0 conn(0x7f6c5c0b4170 msgr2=0x7f6c5c0b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:30.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.132+0000 7f6c61f16640 1 --2- 172.21.15.83:0/3761624396 >> v2:172.21.15.183:3300/0 conn(0x7f6c5c0b4170 0x7f6c5c0b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.132+0000 7f6c61f16640 1 -- 172.21.15.83:0/3761624396 >> v1:172.21.15.183:6789/0 conn(0x7f6c5c0d4760 legacy=0x7f6c5c0d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.132+0000 7f6c61f16640 1 -- 172.21.15.83:0/3761624396 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c5c0d2870 con 0x7f6c5c0d0a80 2024-03-06T07:23:30.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.132+0000 7f6c61715640 1 --2- 172.21.15.83:0/3761624396 >> v2:172.21.15.183:3300/0 conn(0x7f6c5c0b4170 0x7f6c5c0b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:30.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.132+0000 7f6c61f16640 1 --2- 172.21.15.83:0/3761624396 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6c5c0d0a80 0x7f6c5c0cee70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6c540097a0 tx=0x7f6c5402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d8f1533c3361d01 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:30.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.133+0000 7f6c53fff640 1 -- 172.21.15.83:0/3761624396 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c54035020 con 0x7f6c5c0d0a80 2024-03-06T07:23:30.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.133+0000 7f6c53fff640 1 -- 172.21.15.83:0/3761624396 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c54031830 con 0x7f6c5c0d0a80 2024-03-06T07:23:30.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.133+0000 7f6c53fff640 1 -- 172.21.15.83:0/3761624396 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c54031c00 con 0x7f6c5c0d0a80 2024-03-06T07:23:30.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.133+0000 7f6c63979640 1 -- 172.21.15.83:0/3761624396 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6c5c0d0a80 msgr2=0x7f6c5c0cee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.133+0000 7f6c63979640 1 --2- 172.21.15.83:0/3761624396 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6c5c0d0a80 0x7f6c5c0cee70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6c540097a0 tx=0x7f6c5402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.133+0000 7f6c63979640 1 -- 172.21.15.83:0/3761624396 shutdown_connections 2024-03-06T07:23:30.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.133+0000 7f6c63979640 1 --2- 172.21.15.83:0/3761624396 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6c5c0d0a80 0x7f6c5c0cee70 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.133+0000 7f6c63979640 1 --2- 172.21.15.83:0/3761624396 >> v2:172.21.15.183:3300/0 conn(0x7f6c5c0b4170 0x7f6c5c0b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.133+0000 7f6c63979640 1 -- 172.21.15.83:0/3761624396 >> 172.21.15.83:0/3761624396 conn(0x7f6c5c0d3c40 msgr2=0x7f6c5c0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:30.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.133+0000 7f6c63979640 1 -- 172.21.15.83:0/3761624396 shutdown_connections 2024-03-06T07:23:30.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.133+0000 7f6c63979640 1 -- 172.21.15.83:0/3761624396 wait complete. 2024-03-06T07:23:30.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.135+0000 7f6c63979640 1 Processor -- start 2024-03-06T07:23:30.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.135+0000 7f6c63979640 1 -- start start 2024-03-06T07:23:30.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.136+0000 7f6c63979640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6c5c0b4170 0x7f6c5c156cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.136+0000 7f6c63979640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c5c0d0a80 0x7f6c5c15eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.136+0000 7f6c63979640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c5c15f050 0x7f6c5c0b0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.136+0000 7f6c63979640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5c0b4dd0 con 0x7f6c5c0d0a80 2024-03-06T07:23:30.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.136+0000 7f6c63979640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5c0b4550 con 0x7f6c5c15f050 2024-03-06T07:23:30.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.136+0000 7f6c63979640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5c0b4f50 con 0x7f6c5c0b4170 2024-03-06T07:23:30.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.136+0000 7f6c60f14640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c5c0d0a80 0x7f6c5c15eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.136+0000 7f6c60f14640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c5c0d0a80 0x7f6c5c15eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57376/0 (socket says 172.21.15.83:57376) 2024-03-06T07:23:30.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.137+0000 7f6c60f14640 1 -- 172.21.15.83:0/917061432 learned_addr learned my addr 172.21.15.83:0/917061432 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:30.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.137+0000 7f6c61f16640 1 --2- 172.21.15.83:0/917061432 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c5c15f050 0x7f6c5c0b0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.137+0000 7f6c61715640 1 --2- 172.21.15.83:0/917061432 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6c5c0b4170 0x7f6c5c156cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.137+0000 7f6c61f16640 1 -- 172.21.15.83:0/917061432 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6c5c0b4170 msgr2=0x7f6c5c156cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.137+0000 7f6c61f16640 1 --2- 172.21.15.83:0/917061432 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6c5c0b4170 0x7f6c5c156cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.137+0000 7f6c61f16640 1 -- 172.21.15.83:0/917061432 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c5c0d0a80 msgr2=0x7f6c5c15eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.137+0000 7f6c61f16640 1 --2- 172.21.15.83:0/917061432 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c5c0d0a80 0x7f6c5c15eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.137+0000 7f6c61f16640 1 -- 172.21.15.83:0/917061432 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c5c0b2f40 con 0x7f6c5c15f050 2024-03-06T07:23:30.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.137+0000 7f6c60f14640 1 --2- 172.21.15.83:0/917061432 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c5c0d0a80 0x7f6c5c15eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:30.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.137+0000 7f6c61f16640 1 --2- 172.21.15.83:0/917061432 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c5c15f050 0x7f6c5c0b0b30 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6c54033000 tx=0x7f6c540046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:30.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.137+0000 7f6c51ffb640 1 -- 172.21.15.83:0/917061432 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c54035040 con 0x7f6c5c15f050 2024-03-06T07:23:30.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.137+0000 7f6c51ffb640 1 -- 172.21.15.83:0/917061432 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c5403b840 con 0x7f6c5c15f050 2024-03-06T07:23:30.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.137+0000 7f6c51ffb640 1 -- 172.21.15.83:0/917061432 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c5403bc10 con 0x7f6c5c15f050 2024-03-06T07:23:30.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.137+0000 7f6c63979640 1 -- 172.21.15.83:0/917061432 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c5c0b3220 con 0x7f6c5c15f050 2024-03-06T07:23:30.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.138+0000 7f6c63979640 1 -- 172.21.15.83:0/917061432 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c5c0b37e0 con 0x7f6c5c15f050 2024-03-06T07:23:30.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.138+0000 7f6c63979640 1 -- 172.21.15.83:0/917061432 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c5c04f330 con 0x7f6c5c15f050 2024-03-06T07:23:30.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.139+0000 7f6c51ffb640 1 -- 172.21.15.83:0/917061432 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6c54038020 con 0x7f6c5c15f050 2024-03-06T07:23:30.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.139+0000 7f6c51ffb640 1 --2- 172.21.15.83:0/917061432 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6c4006b4d0 0x7f6c4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.139+0000 7f6c51ffb640 1 -- 172.21.15.83:0/917061432 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6c540ac190 con 0x7f6c5c15f050 2024-03-06T07:23:30.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.139+0000 7f6c61715640 1 --2- 172.21.15.83:0/917061432 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6c4006b4d0 0x7f6c4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.140+0000 7f6c61715640 1 --2- 172.21.15.83:0/917061432 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6c4006b4d0 0x7f6c4006d990 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c15fe10 tx=0x7f6c5800b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:30.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.144+0000 7f6c51ffb640 1 -- 172.21.15.83:0/917061432 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6c54031800 con 0x7f6c5c15f050 2024-03-06T07:23:30.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.359+0000 7f6c63979640 1 -- 172.21.15.83:0/917061432 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c5c0b3bf0 con 0x7f6c5c15f050 2024-03-06T07:23:30.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.360+0000 7f6c51ffb640 1 -- 172.21.15.83:0/917061432 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f6c5407a9d0 con 0x7f6c5c15f050 2024-03-06T07:23:30.361 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:23:30.362 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4799,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/3542419057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3542419057},{"type":"v1","addr":"172.21.15.83:6836","nonce":3542419057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":5023,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1853869906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":1853869906},{"type":"v1","addr":"172.21.15.183:6835","nonce":1853869906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5026,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/3666667479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3666667479},{"type":"v1","addr":"172.21.15.183:6834","nonce":3666667479}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:00.145321+0000","modified":"2024-03-06T07:23:02.319419+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5244":{"gid":5244,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/2668925454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":2668925454},{"type":"v1","addr":"172.21.15.83:6837","nonce":2668925454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5244,"qdb_cluster":[5244]},"id":2}]} 2024-03-06T07:23:30.362 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 13 2024-03-06T07:23:30.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.362+0000 7f6c63979640 1 -- 172.21.15.83:0/917061432 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6c4006b4d0 msgr2=0x7f6c4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.362+0000 7f6c63979640 1 --2- 172.21.15.83:0/917061432 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6c4006b4d0 0x7f6c4006d990 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c15fe10 tx=0x7f6c5800b040 comp rx=0 tx=0).stop 2024-03-06T07:23:30.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.362+0000 7f6c63979640 1 -- 172.21.15.83:0/917061432 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c5c15f050 msgr2=0x7f6c5c0b0b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.362+0000 7f6c63979640 1 --2- 172.21.15.83:0/917061432 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c5c15f050 0x7f6c5c0b0b30 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6c54033000 tx=0x7f6c540046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.362+0000 7f6c63979640 1 -- 172.21.15.83:0/917061432 shutdown_connections 2024-03-06T07:23:30.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.362+0000 7f6c63979640 1 --2- 172.21.15.83:0/917061432 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6c4006b4d0 0x7f6c4006d990 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.362+0000 7f6c63979640 1 --2- 172.21.15.83:0/917061432 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c5c15f050 0x7f6c5c0b0b30 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.362+0000 7f6c63979640 1 --2- 172.21.15.83:0/917061432 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c5c0d0a80 0x7f6c5c15eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.362+0000 7f6c63979640 1 --2- 172.21.15.83:0/917061432 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6c5c0b4170 0x7f6c5c156cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.362+0000 7f6c63979640 1 -- 172.21.15.83:0/917061432 >> 172.21.15.83:0/917061432 conn(0x7f6c5c0d3c40 msgr2=0x7f6c5c06b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:30.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.362+0000 7f6c63979640 1 -- 172.21.15.83:0/917061432 shutdown_connections 2024-03-06T07:23:30.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.362+0000 7f6c63979640 1 -- 172.21.15.83:0/917061432 wait complete. 2024-03-06T07:23:30.375 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:30.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.474+0000 7fb9478c2640 1 Processor -- start 2024-03-06T07:23:30.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.474+0000 7fb9478c2640 1 -- start start 2024-03-06T07:23:30.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.475+0000 7fb9478c2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb94015bb20 0x7fb9400d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.475+0000 7fb9478c2640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fb9400cfff0 0x7fb9400d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.476+0000 7fb9478c2640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb940161980 con 0x7fb9400ce4b0 2024-03-06T07:23:30.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.476+0000 7fb9478c2640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb940161c60 con 0x7fb94015bb20 2024-03-06T07:23:30.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.476+0000 7fb9478c2640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb940161f40 con 0x7fb9400cfff0 2024-03-06T07:23:30.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.476+0000 7fb94565e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb94015bb20 0x7fb9400d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.476+0000 7fb94565e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb94015bb20 0x7fb9400d1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57382/0 (socket says 172.21.15.83:57382) 2024-03-06T07:23:30.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.476+0000 7fb94565e640 1 -- 172.21.15.83:0/521179368 learned_addr learned my addr 172.21.15.83:0/521179368 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:30.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.476+0000 7fb944e5d640 1 --2- 172.21.15.83:0/521179368 >> v2:172.21.15.183:3300/0 conn(0x7fb9400cfff0 0x7fb9400d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.476+0000 7fb94565e640 1 -- 172.21.15.83:0/521179368 >> v2:172.21.15.183:3300/0 conn(0x7fb9400cfff0 msgr2=0x7fb9400d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:30.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.476+0000 7fb94565e640 1 --2- 172.21.15.83:0/521179368 >> v2:172.21.15.183:3300/0 conn(0x7fb9400cfff0 0x7fb9400d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.476+0000 7fb94565e640 1 -- 172.21.15.83:0/521179368 >> v1:172.21.15.183:6789/0 conn(0x7fb9400ce4b0 legacy=0x7fb9400ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.476+0000 7fb94565e640 1 -- 172.21.15.83:0/521179368 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb940162220 con 0x7fb94015bb20 2024-03-06T07:23:30.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.477+0000 7fb94565e640 1 --2- 172.21.15.83:0/521179368 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb94015bb20 0x7fb9400d1df0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fb940071290 tx=0x7fb93c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bde399fadb98c386 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:30.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.477+0000 7fb937fff640 1 -- 172.21.15.83:0/521179368 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93c034070 con 0x7fb94015bb20 2024-03-06T07:23:30.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.477+0000 7fb937fff640 1 -- 172.21.15.83:0/521179368 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb93c030860 con 0x7fb94015bb20 2024-03-06T07:23:30.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.477+0000 7fb937fff640 1 -- 172.21.15.83:0/521179368 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93c030c30 con 0x7fb94015bb20 2024-03-06T07:23:30.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.477+0000 7fb9478c2640 1 -- 172.21.15.83:0/521179368 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb94015bb20 msgr2=0x7fb9400d1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.477+0000 7fb9478c2640 1 --2- 172.21.15.83:0/521179368 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb94015bb20 0x7fb9400d1df0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fb940071290 tx=0x7fb93c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.477+0000 7fb9478c2640 1 -- 172.21.15.83:0/521179368 shutdown_connections 2024-03-06T07:23:30.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.477+0000 7fb9478c2640 1 --2- 172.21.15.83:0/521179368 >> v2:172.21.15.183:3300/0 conn(0x7fb9400cfff0 0x7fb9400d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.477+0000 7fb9478c2640 1 --2- 172.21.15.83:0/521179368 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb94015bb20 0x7fb9400d1df0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.477+0000 7fb9478c2640 1 -- 172.21.15.83:0/521179368 >> 172.21.15.83:0/521179368 conn(0x7fb94009a9f0 msgr2=0x7fb94009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:30.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.478+0000 7fb9478c2640 1 -- 172.21.15.83:0/521179368 shutdown_connections 2024-03-06T07:23:30.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.478+0000 7fb9478c2640 1 -- 172.21.15.83:0/521179368 wait complete. 2024-03-06T07:23:30.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.479+0000 7fb9478c2640 1 Processor -- start 2024-03-06T07:23:30.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.479+0000 7fb9478c2640 1 -- start start 2024-03-06T07:23:30.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.480+0000 7fb9478c2640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9400ce4b0 0x7fb940158470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.480+0000 7fb9478c2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb9400cfff0 0x7fb940159690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.480+0000 7fb9478c2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb94015bb20 0x7fb9400cd900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.480+0000 7fb9478c2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9400d24b0 con 0x7fb94015bb20 2024-03-06T07:23:30.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.480+0000 7fb9478c2640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9400d2330 con 0x7fb9400ce4b0 2024-03-06T07:23:30.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.480+0000 7fb9478c2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9400d1980 con 0x7fb9400cfff0 2024-03-06T07:23:30.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.480+0000 7fb945e5f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb94015bb20 0x7fb9400cd900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.480+0000 7fb945e5f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb94015bb20 0x7fb9400cd900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57390/0 (socket says 172.21.15.83:57390) 2024-03-06T07:23:30.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.480+0000 7fb945e5f640 1 -- 172.21.15.83:0/3127111158 learned_addr learned my addr 172.21.15.83:0/3127111158 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:30.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.480+0000 7fb94565e640 1 --2- 172.21.15.83:0/3127111158 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9400ce4b0 0x7fb940158470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.480+0000 7fb944e5d640 1 --2- 172.21.15.83:0/3127111158 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb9400cfff0 0x7fb940159690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.480+0000 7fb945e5f640 1 -- 172.21.15.83:0/3127111158 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb9400cfff0 msgr2=0x7fb940159690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.480+0000 7fb945e5f640 1 --2- 172.21.15.83:0/3127111158 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb9400cfff0 0x7fb940159690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.480+0000 7fb945e5f640 1 -- 172.21.15.83:0/3127111158 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9400ce4b0 msgr2=0x7fb940158470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.480+0000 7fb945e5f640 1 --2- 172.21.15.83:0/3127111158 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9400ce4b0 0x7fb940158470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.480+0000 7fb945e5f640 1 -- 172.21.15.83:0/3127111158 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb94006dd40 con 0x7fb94015bb20 2024-03-06T07:23:30.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.480+0000 7fb945e5f640 1 --2- 172.21.15.83:0/3127111158 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb94015bb20 0x7fb9400cd900 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fb93800c7d0 tx=0x7fb93800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:30.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.481+0000 7fb935ffb640 1 -- 172.21.15.83:0/3127111158 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb938003b70 con 0x7fb94015bb20 2024-03-06T07:23:30.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.481+0000 7fb935ffb640 1 -- 172.21.15.83:0/3127111158 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb938003d30 con 0x7fb94015bb20 2024-03-06T07:23:30.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.481+0000 7fb9478c2640 1 -- 172.21.15.83:0/3127111158 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb94006e020 con 0x7fb94015bb20 2024-03-06T07:23:30.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.481+0000 7fb935ffb640 1 -- 172.21.15.83:0/3127111158 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93800aa90 con 0x7fb94015bb20 2024-03-06T07:23:30.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.481+0000 7fb9478c2640 1 -- 172.21.15.83:0/3127111158 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9401589b0 con 0x7fb94015bb20 2024-03-06T07:23:30.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.482+0000 7fb9478c2640 1 -- 172.21.15.83:0/3127111158 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb940157a90 con 0x7fb94015bb20 2024-03-06T07:23:30.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.482+0000 7fb935ffb640 1 -- 172.21.15.83:0/3127111158 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb93800ac30 con 0x7fb94015bb20 2024-03-06T07:23:30.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.482+0000 7fb935ffb640 1 --2- 172.21.15.83:0/3127111158 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb92406b4d0 0x7fb92406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.483+0000 7fb935ffb640 1 -- 172.21.15.83:0/3127111158 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb93808a130 con 0x7fb94015bb20 2024-03-06T07:23:30.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.483+0000 7fb94565e640 1 --2- 172.21.15.83:0/3127111158 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb92406b4d0 0x7fb92406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.483+0000 7fb94565e640 1 --2- 172.21.15.83:0/3127111158 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb92406b4d0 0x7fb92406d990 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb93c036000 tx=0x7fb93c03b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:30.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.487+0000 7fb935ffb640 1 -- 172.21.15.83:0/3127111158 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb940157a90 con 0x7fb94015bb20 2024-03-06T07:23:30.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.665+0000 7fb9478c2640 1 -- 172.21.15.83:0/3127111158 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb94006e1f0 con 0x7fb94015bb20 2024-03-06T07:23:30.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.666+0000 7fb935ffb640 1 -- 172.21.15.83:0/3127111158 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16681 (secure 0 0 0) 0x7fb938058fb0 con 0x7fb94015bb20 2024-03-06T07:23:30.668 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:23:30.668 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":33,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:23:01.812855+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:58.568071+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:59.116376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:30.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.668+0000 7fb9478c2640 1 -- 172.21.15.83:0/3127111158 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb92406b4d0 msgr2=0x7fb92406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.668+0000 7fb9478c2640 1 --2- 172.21.15.83:0/3127111158 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb92406b4d0 0x7fb92406d990 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb93c036000 tx=0x7fb93c03b040 comp rx=0 tx=0).stop 2024-03-06T07:23:30.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.668+0000 7fb9478c2640 1 -- 172.21.15.83:0/3127111158 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb94015bb20 msgr2=0x7fb9400cd900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.668+0000 7fb9478c2640 1 --2- 172.21.15.83:0/3127111158 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb94015bb20 0x7fb9400cd900 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fb93800c7d0 tx=0x7fb93800cc90 comp rx=0 tx=0).stop 2024-03-06T07:23:30.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.668+0000 7fb9478c2640 1 -- 172.21.15.83:0/3127111158 shutdown_connections 2024-03-06T07:23:30.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.668+0000 7fb9478c2640 1 --2- 172.21.15.83:0/3127111158 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb92406b4d0 0x7fb92406d990 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.668+0000 7fb9478c2640 1 --2- 172.21.15.83:0/3127111158 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb94015bb20 0x7fb9400cd900 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.668+0000 7fb9478c2640 1 --2- 172.21.15.83:0/3127111158 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb9400cfff0 0x7fb940159690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.668+0000 7fb9478c2640 1 --2- 172.21.15.83:0/3127111158 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9400ce4b0 0x7fb940158470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.668+0000 7fb9478c2640 1 -- 172.21.15.83:0/3127111158 >> 172.21.15.83:0/3127111158 conn(0x7fb94009a9f0 msgr2=0x7fb94006f170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:30.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.668+0000 7fb9478c2640 1 -- 172.21.15.83:0/3127111158 shutdown_connections 2024-03-06T07:23:30.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.668+0000 7fb9478c2640 1 -- 172.21.15.83:0/3127111158 wait complete. 2024-03-06T07:23:30.681 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:23:30.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.781+0000 7f0de2bac640 1 Processor -- start 2024-03-06T07:23:30.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.781+0000 7f0de2bac640 1 -- start start 2024-03-06T07:23:30.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.782+0000 7f0de2bac640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f0ddc162970 0x7f0ddc0d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.782+0000 7f0de2bac640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ddc0d22c0 con 0x7f0ddc0b4460 2024-03-06T07:23:30.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.782+0000 7f0de2bac640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ddc0d25a0 con 0x7f0ddc161cf0 2024-03-06T07:23:30.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.782+0000 7f0de2bac640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ddc161b00 con 0x7f0ddc162970 2024-03-06T07:23:30.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.782+0000 7f0de0948640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f0ddc0b4460 0x7f0ddc0b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49038/0 (socket says 172.21.15.83:49038) 2024-03-06T07:23:30.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.782+0000 7f0de0948640 1 -- 172.21.15.83:0/637564322 learned_addr learned my addr 172.21.15.83:0/637564322 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:23:30.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.783+0000 7f0ddbfff640 1 --2- 172.21.15.83:0/637564322 >> v2:172.21.15.183:3300/0 conn(0x7f0ddc162970 0x7f0ddc0d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.783+0000 7f0ddb7fe640 1 -- 172.21.15.83:0/637564322 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4142020939 0 0) 0x7f0ddc0d22c0 con 0x7f0ddc0b4460 2024-03-06T07:23:30.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.783+0000 7f0ddb7fe640 1 -- 172.21.15.83:0/637564322 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dcc003520 con 0x7f0ddc0b4460 2024-03-06T07:23:30.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.783+0000 7f0ddb7fe640 1 -- 172.21.15.83:0/637564322 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3854013448 0 0) 0x7f0dcc003520 con 0x7f0ddc0b4460 2024-03-06T07:23:30.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.783+0000 7f0ddb7fe640 1 -- 172.21.15.83:0/637564322 >> v2:172.21.15.183:3300/0 conn(0x7f0ddc162970 msgr2=0x7f0ddc0d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:30.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.783+0000 7f0ddb7fe640 1 --2- 172.21.15.83:0/637564322 >> v2:172.21.15.183:3300/0 conn(0x7f0ddc162970 0x7f0ddc0d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.783+0000 7f0ddb7fe640 1 -- 172.21.15.83:0/637564322 >> v1:172.21.15.183:6789/0 conn(0x7f0ddc161cf0 legacy=0x7f0ddc0cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.783+0000 7f0ddb7fe640 1 -- 172.21.15.83:0/637564322 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ddc0d2880 con 0x7f0ddc0b4460 2024-03-06T07:23:30.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.783+0000 7f0ddb7fe640 1 -- 172.21.15.83:0/637564322 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f0dd0001670 con 0x7f0ddc0b4460 2024-03-06T07:23:30.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.783+0000 7f0ddb7fe640 1 -- 172.21.15.83:0/637564322 >> v1:172.21.15.83:6789/0 conn(0x7f0ddc0b4460 legacy=0x7f0ddc0b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.783+0000 7f0ddb7fe640 1 --2- 172.21.15.83:0/637564322 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0dcc0039b0 0x7f0dcc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.783+0000 7f0ddb7fe640 1 --2- 172.21.15.83:0/637564322 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0dcc0043b0 0x7f0dcc004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.783+0000 7f0ddb7fe640 1 --2- 172.21.15.83:0/637564322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0dcc005080 0x7f0dcc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.783+0000 7f0ddb7fe640 1 -- 172.21.15.83:0/637564322 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ddc161b00 con 0x7f0dcc005080 2024-03-06T07:23:30.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.783+0000 7f0ddb7fe640 1 -- 172.21.15.83:0/637564322 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dcc00bfe0 con 0x7f0dcc0039b0 2024-03-06T07:23:30.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.783+0000 7f0ddb7fe640 1 -- 172.21.15.83:0/637564322 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dcc00c1e0 con 0x7f0dcc0043b0 2024-03-06T07:23:30.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.783+0000 7f0de1149640 1 --2- 172.21.15.83:0/637564322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0dcc005080 0x7f0dcc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.783+0000 7f0de0948640 1 --2- 172.21.15.83:0/637564322 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0dcc0039b0 0x7f0dcc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.783+0000 7f0ddbfff640 1 --2- 172.21.15.83:0/637564322 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0dcc0043b0 0x7f0dcc004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.783+0000 7f0de1149640 1 -- 172.21.15.83:0/637564322 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0dcc0043b0 msgr2=0x7f0dcc004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:30.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.783+0000 7f0de1149640 1 --2- 172.21.15.83:0/637564322 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0dcc0043b0 0x7f0dcc004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.783+0000 7f0de1149640 1 -- 172.21.15.83:0/637564322 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0dcc0039b0 msgr2=0x7f0dcc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:30.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.783+0000 7f0de1149640 1 --2- 172.21.15.83:0/637564322 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0dcc0039b0 0x7f0dcc003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.784+0000 7f0de1149640 1 -- 172.21.15.83:0/637564322 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dcc003520 con 0x7f0dcc005080 2024-03-06T07:23:30.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.784+0000 7f0de1149640 1 -- 172.21.15.83:0/637564322 reap_dead start 2024-03-06T07:23:30.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.784+0000 7f0de1149640 1 --2- 172.21.15.83:0/637564322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0dcc005080 0x7f0dcc00b9c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4001f60 tx=0x7f0dcc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf10351c85df7f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:30.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.784+0000 7f0ddb7fe640 1 -- 172.21.15.83:0/637564322 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dd4001b30 con 0x7f0dcc005080 2024-03-06T07:23:30.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.784+0000 7f0ddb7fe640 1 -- 172.21.15.83:0/637564322 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dd4003b90 con 0x7f0dcc005080 2024-03-06T07:23:30.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.784+0000 7f0ddb7fe640 1 -- 172.21.15.83:0/637564322 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dd4005070 con 0x7f0dcc005080 2024-03-06T07:23:30.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.785+0000 7f0de2bac640 1 -- 172.21.15.83:0/637564322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0dcc005080 msgr2=0x7f0dcc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.785+0000 7f0de2bac640 1 --2- 172.21.15.83:0/637564322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0dcc005080 0x7f0dcc00b9c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4001f60 tx=0x7f0dcc005830 comp rx=0 tx=0).stop 2024-03-06T07:23:30.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.785+0000 7f0de2bac640 1 -- 172.21.15.83:0/637564322 shutdown_connections 2024-03-06T07:23:30.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.785+0000 7f0de2bac640 1 --2- 172.21.15.83:0/637564322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0dcc005080 0x7f0dcc00b9c0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.785+0000 7f0de2bac640 1 -- 172.21.15.83:0/637564322 >> 172.21.15.83:0/637564322 conn(0x7f0ddc0d3ce0 msgr2=0x7f0ddc0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:30.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.785+0000 7f0de2bac640 1 -- 172.21.15.83:0/637564322 shutdown_connections 2024-03-06T07:23:30.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.785+0000 7f0de2bac640 1 -- 172.21.15.83:0/637564322 wait complete. 2024-03-06T07:23:30.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.786+0000 7f0de2bac640 1 Processor -- start 2024-03-06T07:23:30.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.786+0000 7f0de2bac640 1 -- start start 2024-03-06T07:23:30.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.787+0000 7f0de2bac640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0dcc005080 0x7f0ddc0d1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.787+0000 7f0de2bac640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0ddc0cc6d0 0x7f0ddc0ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.787+0000 7f0de2bac640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ddc158e10 0x7f0ddc159290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.787+0000 7f0de2bac640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ddc0ce950 con 0x7f0ddc158e10 2024-03-06T07:23:30.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.787+0000 7f0de2bac640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ddc0cecc0 con 0x7f0ddc0cc6d0 2024-03-06T07:23:30.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.787+0000 7f0de2bac640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ddc0b4840 con 0x7f0dcc005080 2024-03-06T07:23:30.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.787+0000 7f0de1149640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ddc158e10 0x7f0ddc159290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.787+0000 7f0de1149640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ddc158e10 0x7f0ddc159290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57406/0 (socket says 172.21.15.83:57406) 2024-03-06T07:23:30.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.787+0000 7f0de1149640 1 -- 172.21.15.83:0/406438325 learned_addr learned my addr 172.21.15.83:0/406438325 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:30.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.787+0000 7f0de1149640 1 -- 172.21.15.83:0/406438325 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0dcc005080 msgr2=0x7f0ddc0d1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.787+0000 7f0de1149640 1 --2- 172.21.15.83:0/406438325 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0dcc005080 0x7f0ddc0d1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.787+0000 7f0de1149640 1 -- 172.21.15.83:0/406438325 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0ddc0cc6d0 msgr2=0x7f0ddc0ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.787+0000 7f0de1149640 1 --2- 172.21.15.83:0/406438325 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0ddc0cc6d0 0x7f0ddc0ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.787+0000 7f0de1149640 1 -- 172.21.15.83:0/406438325 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ddc0cd170 con 0x7f0ddc158e10 2024-03-06T07:23:30.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.787+0000 7f0de1149640 1 --2- 172.21.15.83:0/406438325 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ddc158e10 0x7f0ddc159290 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4002050 tx=0x7f0dd40015b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:30.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.788+0000 7f0dd97fa640 1 -- 172.21.15.83:0/406438325 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dd4046070 con 0x7f0ddc158e10 2024-03-06T07:23:30.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.788+0000 7f0dd97fa640 1 -- 172.21.15.83:0/406438325 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dd4006b50 con 0x7f0ddc158e10 2024-03-06T07:23:30.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.788+0000 7f0dd97fa640 1 -- 172.21.15.83:0/406438325 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dd402d4f0 con 0x7f0ddc158e10 2024-03-06T07:23:30.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.788+0000 7f0de2bac640 1 -- 172.21.15.83:0/406438325 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ddc0cd450 con 0x7f0ddc158e10 2024-03-06T07:23:30.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.788+0000 7f0de2bac640 1 -- 172.21.15.83:0/406438325 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ddc0cda10 con 0x7f0ddc158e10 2024-03-06T07:23:30.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.789+0000 7f0de2bac640 1 -- 172.21.15.83:0/406438325 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ddc04f330 con 0x7f0ddc158e10 2024-03-06T07:23:30.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.789+0000 7f0dd97fa640 1 -- 172.21.15.83:0/406438325 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0dd4049070 con 0x7f0ddc158e10 2024-03-06T07:23:30.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.790+0000 7f0dd97fa640 1 --2- 172.21.15.83:0/406438325 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0dc006b4d0 0x7f0dc006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.790+0000 7f0dd97fa640 1 -- 172.21.15.83:0/406438325 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0dd40ab790 con 0x7f0ddc158e10 2024-03-06T07:23:30.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.790+0000 7f0de0948640 1 --2- 172.21.15.83:0/406438325 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0dc006b4d0 0x7f0dc006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.790+0000 7f0de0948640 1 --2- 172.21.15.83:0/406438325 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0dc006b4d0 0x7f0dc006d990 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0dd0001ba0 tx=0x7f0dd0001b10 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:30.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:30.794+0000 7f0dd97fa640 1 -- 172.21.15.83:0/406438325 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0dd4079fc0 con 0x7f0ddc158e10 2024-03-06T07:23:31.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.019+0000 7f0de2bac640 1 -- 172.21.15.83:0/406438325 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f0ddc06b180 con 0x7f0ddc158e10 2024-03-06T07:23:31.022 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-03-06T07:23:31.020+0000 7fb0dae11640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:23:31.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.023+0000 7f0dd97fa640 1 -- 172.21.15.83:0/406438325 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v14) v1 ==== 161+0+0 (secure 0 0 0) 0x7f0dd407e060 con 0x7f0ddc158e10 2024-03-06T07:23:31.025 INFO:teuthology.orchestra.run.smithi083.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:23:31.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.026+0000 7f0de2bac640 1 -- 172.21.15.83:0/406438325 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0dc006b4d0 msgr2=0x7f0dc006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.026+0000 7f0de2bac640 1 --2- 172.21.15.83:0/406438325 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0dc006b4d0 0x7f0dc006d990 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0dd0001ba0 tx=0x7f0dd0001b10 comp rx=0 tx=0).stop 2024-03-06T07:23:31.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.026+0000 7f0de2bac640 1 -- 172.21.15.83:0/406438325 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ddc158e10 msgr2=0x7f0ddc159290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.026+0000 7f0de2bac640 1 --2- 172.21.15.83:0/406438325 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ddc158e10 0x7f0ddc159290 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4002050 tx=0x7f0dd40015b0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.026+0000 7f0de2bac640 1 -- 172.21.15.83:0/406438325 shutdown_connections 2024-03-06T07:23:31.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.026+0000 7f0de2bac640 1 --2- 172.21.15.83:0/406438325 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0dc006b4d0 0x7f0dc006d990 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.026+0000 7f0de2bac640 1 --2- 172.21.15.83:0/406438325 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ddc158e10 0x7f0ddc159290 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.026+0000 7f0de2bac640 1 --2- 172.21.15.83:0/406438325 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0ddc0cc6d0 0x7f0ddc0ccb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.026+0000 7f0de2bac640 1 --2- 172.21.15.83:0/406438325 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0dcc005080 0x7f0ddc0d1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.026+0000 7f0de2bac640 1 -- 172.21.15.83:0/406438325 >> 172.21.15.83:0/406438325 conn(0x7f0ddc0d3ce0 msgr2=0x7f0ddc06ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:31.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.026+0000 7f0de2bac640 1 -- 172.21.15.83:0/406438325 shutdown_connections 2024-03-06T07:23:31.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.026+0000 7f0de2bac640 1 -- 172.21.15.83:0/406438325 wait complete. 2024-03-06T07:23:31.041 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:23:31.067 INFO:tasks.ceph.mds.a.smithi083.stdout:starting mds.a at 2024-03-06T07:23:31.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.142+0000 7f6c80513640 1 Processor -- start 2024-03-06T07:23:31.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.142+0000 7f6c80513640 1 -- start start 2024-03-06T07:23:31.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.143+0000 7f6c80513640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6c7815a220 0x7f6c780cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.143+0000 7f6c80513640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f6c780712d0 0x7f6c78071730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.143+0000 7f6c80513640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c780b3320 con 0x7f6c7815de60 2024-03-06T07:23:31.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.143+0000 7f6c80513640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c780cf610 con 0x7f6c7815a220 2024-03-06T07:23:31.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.143+0000 7f6c80513640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c78071c70 con 0x7f6c780712d0 2024-03-06T07:23:31.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.143+0000 7f6c7e2af640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6c7815a220 0x7f6c780cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.143+0000 7f6c7e2af640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6c7815a220 0x7f6c780cf0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57430/0 (socket says 172.21.15.83:57430) 2024-03-06T07:23:31.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.143+0000 7f6c7e2af640 1 -- 172.21.15.83:0/1671591354 learned_addr learned my addr 172.21.15.83:0/1671591354 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:31.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.143+0000 7f6c7daae640 1 --2- 172.21.15.83:0/1671591354 >> v2:172.21.15.183:3300/0 conn(0x7f6c780712d0 0x7f6c78071730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.143+0000 7f6c7e2af640 1 -- 172.21.15.83:0/1671591354 >> v2:172.21.15.183:3300/0 conn(0x7f6c780712d0 msgr2=0x7f6c78071730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:31.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.143+0000 7f6c7e2af640 1 --2- 172.21.15.83:0/1671591354 >> v2:172.21.15.183:3300/0 conn(0x7f6c780712d0 0x7f6c78071730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.143+0000 7f6c7e2af640 1 -- 172.21.15.83:0/1671591354 >> v1:172.21.15.183:6789/0 conn(0x7f6c7815de60 legacy=0x7f6c7815e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.143+0000 7f6c7e2af640 1 -- 172.21.15.83:0/1671591354 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c780728d0 con 0x7f6c7815a220 2024-03-06T07:23:31.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.144+0000 7f6c7e2af640 1 --2- 172.21.15.83:0/1671591354 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c7815a220 0x7f6c780cf0d0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f6c780d0980 tx=0x7f6c7402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8e5e55e16d63205 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:31.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.144+0000 7f6c7d2ad640 1 -- 172.21.15.83:0/1671591354 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c74035020 con 0x7f6c7815a220 2024-03-06T07:23:31.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.144+0000 7f6c7d2ad640 1 -- 172.21.15.83:0/1671591354 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c74031870 con 0x7f6c7815a220 2024-03-06T07:23:31.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.144+0000 7f6c7d2ad640 1 -- 172.21.15.83:0/1671591354 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c74031c40 con 0x7f6c7815a220 2024-03-06T07:23:31.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.144+0000 7f6c80513640 1 -- 172.21.15.83:0/1671591354 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c7815a220 msgr2=0x7f6c780cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.145+0000 7f6c80513640 1 --2- 172.21.15.83:0/1671591354 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c7815a220 0x7f6c780cf0d0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f6c780d0980 tx=0x7f6c7402b910 comp rx=0 tx=0).stop 2024-03-06T07:23:31.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.145+0000 7f6c80513640 1 -- 172.21.15.83:0/1671591354 shutdown_connections 2024-03-06T07:23:31.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.145+0000 7f6c80513640 1 --2- 172.21.15.83:0/1671591354 >> v2:172.21.15.183:3300/0 conn(0x7f6c780712d0 0x7f6c78071730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.145+0000 7f6c80513640 1 --2- 172.21.15.83:0/1671591354 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c7815a220 0x7f6c780cf0d0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.145+0000 7f6c80513640 1 -- 172.21.15.83:0/1671591354 >> 172.21.15.83:0/1671591354 conn(0x7f6c7806b190 msgr2=0x7f6c7806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:31.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.145+0000 7f6c80513640 1 -- 172.21.15.83:0/1671591354 shutdown_connections 2024-03-06T07:23:31.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.145+0000 7f6c80513640 1 -- 172.21.15.83:0/1671591354 wait complete. 2024-03-06T07:23:31.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.146+0000 7f6c80513640 1 Processor -- start 2024-03-06T07:23:31.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.146+0000 7f6c80513640 1 -- start start 2024-03-06T07:23:31.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.146+0000 7f6c80513640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6c780712d0 0x7f6c78156e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.146+0000 7f6c80513640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c7815a220 0x7f6c78159400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.146+0000 7f6c80513640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c7815de60 0x7f6c78146150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.146+0000 7f6c80513640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c78072030 con 0x7f6c7815a220 2024-03-06T07:23:31.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.146+0000 7f6c80513640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c78072bb0 con 0x7f6c7815de60 2024-03-06T07:23:31.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.146+0000 7f6c80513640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c78072530 con 0x7f6c780712d0 2024-03-06T07:23:31.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.147+0000 7f6c7daae640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c7815a220 0x7f6c78159400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.147+0000 7f6c7daae640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c7815a220 0x7f6c78159400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57434/0 (socket says 172.21.15.83:57434) 2024-03-06T07:23:31.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.147+0000 7f6c7daae640 1 -- 172.21.15.83:0/2778642161 learned_addr learned my addr 172.21.15.83:0/2778642161 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:31.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.147+0000 7f6c7eab0640 1 --2- 172.21.15.83:0/2778642161 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c7815de60 0x7f6c78146150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.147+0000 7f6c7e2af640 1 --2- 172.21.15.83:0/2778642161 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6c780712d0 0x7f6c78156e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.147+0000 7f6c7daae640 1 -- 172.21.15.83:0/2778642161 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6c780712d0 msgr2=0x7f6c78156e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.147+0000 7f6c7daae640 1 --2- 172.21.15.83:0/2778642161 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6c780712d0 0x7f6c78156e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.147+0000 7f6c7daae640 1 -- 172.21.15.83:0/2778642161 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c7815de60 msgr2=0x7f6c78146150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.147+0000 7f6c7daae640 1 --2- 172.21.15.83:0/2778642161 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c7815de60 0x7f6c78146150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.147+0000 7f6c7daae640 1 -- 172.21.15.83:0/2778642161 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c78159940 con 0x7f6c7815a220 2024-03-06T07:23:31.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.147+0000 7f6c7daae640 1 --2- 172.21.15.83:0/2778642161 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c7815a220 0x7f6c78159400 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f6c6800d830 tx=0x7f6c6800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:31.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.147+0000 7f6c6effd640 1 -- 172.21.15.83:0/2778642161 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c6800b6c0 con 0x7f6c7815a220 2024-03-06T07:23:31.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.147+0000 7f6c6effd640 1 -- 172.21.15.83:0/2778642161 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c68019070 con 0x7f6c7815a220 2024-03-06T07:23:31.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.147+0000 7f6c80513640 1 -- 172.21.15.83:0/2778642161 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c781574c0 con 0x7f6c7815a220 2024-03-06T07:23:31.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.147+0000 7f6c6effd640 1 -- 172.21.15.83:0/2778642161 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c6800bac0 con 0x7f6c7815a220 2024-03-06T07:23:31.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.147+0000 7f6c80513640 1 -- 172.21.15.83:0/2778642161 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c78157a80 con 0x7f6c7815a220 2024-03-06T07:23:31.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.148+0000 7f6c80513640 1 -- 172.21.15.83:0/2778642161 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c780732b0 con 0x7f6c7815a220 2024-03-06T07:23:31.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.149+0000 7f6c6effd640 1 -- 172.21.15.83:0/2778642161 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6c6801e460 con 0x7f6c7815a220 2024-03-06T07:23:31.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.149+0000 7f6c6effd640 1 --2- 172.21.15.83:0/2778642161 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6c5c06b4d0 0x7f6c5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.149+0000 7f6c7e2af640 1 --2- 172.21.15.83:0/2778642161 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6c5c06b4d0 0x7f6c5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.149+0000 7f6c6effd640 1 -- 172.21.15.83:0/2778642161 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f6c6808a900 con 0x7f6c7815a220 2024-03-06T07:23:31.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.150+0000 7f6c7e2af640 1 --2- 172.21.15.83:0/2778642161 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6c5c06b4d0 0x7f6c5c06d990 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6c74036000 tx=0x7f6c74008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:31.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.154+0000 7f6c6effd640 1 -- 172.21.15.83:0/2778642161 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6c680597b0 con 0x7f6c7815a220 2024-03-06T07:23:31.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:31.361+0000 7f6c80513640 1 -- 172.21.15.83:0/2778642161 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f6c78068ff0 con 0x7f6c7815a220 2024-03-06T07:23:32.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.024+0000 7f6c6effd640 1 -- 172.21.15.83:0/2778642161 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v15) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6c680590b0 con 0x7f6c7815a220 2024-03-06T07:23:32.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.027+0000 7f6c80513640 1 -- 172.21.15.83:0/2778642161 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6c5c06b4d0 msgr2=0x7f6c5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.027+0000 7f6c80513640 1 --2- 172.21.15.83:0/2778642161 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6c5c06b4d0 0x7f6c5c06d990 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6c74036000 tx=0x7f6c74008030 comp rx=0 tx=0).stop 2024-03-06T07:23:32.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.027+0000 7f6c80513640 1 -- 172.21.15.83:0/2778642161 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c7815a220 msgr2=0x7f6c78159400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.027+0000 7f6c80513640 1 --2- 172.21.15.83:0/2778642161 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c7815a220 0x7f6c78159400 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f6c6800d830 tx=0x7f6c6800dcf0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.027+0000 7f6c80513640 1 -- 172.21.15.83:0/2778642161 shutdown_connections 2024-03-06T07:23:32.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.027+0000 7f6c80513640 1 --2- 172.21.15.83:0/2778642161 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6c5c06b4d0 0x7f6c5c06d990 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.027+0000 7f6c80513640 1 --2- 172.21.15.83:0/2778642161 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c7815de60 0x7f6c78146150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.027+0000 7f6c80513640 1 --2- 172.21.15.83:0/2778642161 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c7815a220 0x7f6c78159400 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.027+0000 7f6c80513640 1 --2- 172.21.15.83:0/2778642161 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6c780712d0 0x7f6c78156e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.027+0000 7f6c80513640 1 -- 172.21.15.83:0/2778642161 >> 172.21.15.83:0/2778642161 conn(0x7f6c7806b190 msgr2=0x7f6c780cfa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:32.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.027+0000 7f6c80513640 1 -- 172.21.15.83:0/2778642161 shutdown_connections 2024-03-06T07:23:32.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.027+0000 7f6c80513640 1 -- 172.21.15.83:0/2778642161 wait complete. 2024-03-06T07:23:32.043 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:23:32.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.193+0000 7f3a80bd7640 1 Processor -- start 2024-03-06T07:23:32.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.193+0000 7f3a80bd7640 1 -- start start 2024-03-06T07:23:32.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.194+0000 7f3a80bd7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f3a7c15ac00 0x7f3a7c06c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.194+0000 7f3a80bd7640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f3a7c0708b0 0x7f3a7c070d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.194+0000 7f3a80bd7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a7c15afe0 con 0x7f3a7c15e7a0 2024-03-06T07:23:32.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.194+0000 7f3a80bd7640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a7c071250 con 0x7f3a7c15ac00 2024-03-06T07:23:32.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.194+0000 7f3a80bd7640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a7c0cdc10 con 0x7f3a7c0708b0 2024-03-06T07:23:32.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.194+0000 7f3a7a59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f3a7c15ac00 0x7f3a7c06c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.194+0000 7f3a7ad9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f3a7c15e7a0 0x7f3a7c15ec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49046/0 (socket says 172.21.15.83:49046) 2024-03-06T07:23:32.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.194+0000 7f3a7ad9d640 1 -- 172.21.15.83:0/2051564628 learned_addr learned my addr 172.21.15.83:0/2051564628 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:23:32.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.194+0000 7f3a7959a640 1 -- 172.21.15.83:0/2051564628 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 651395078 0 0) 0x7f3a7c15afe0 con 0x7f3a7c15e7a0 2024-03-06T07:23:32.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.194+0000 7f3a7959a640 1 -- 172.21.15.83:0/2051564628 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a68003520 con 0x7f3a7c15e7a0 2024-03-06T07:23:32.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.194+0000 7f3a7a59c640 1 -- 172.21.15.83:0/2051564628 >> v2:172.21.15.183:3300/0 conn(0x7f3a7c0708b0 msgr2=0x7f3a7c070d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:32.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.194+0000 7f3a7a59c640 1 --2- 172.21.15.83:0/2051564628 >> v2:172.21.15.183:3300/0 conn(0x7f3a7c0708b0 0x7f3a7c070d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.194+0000 7f3a7a59c640 1 -- 172.21.15.83:0/2051564628 >> v1:172.21.15.83:6789/0 conn(0x7f3a7c15e7a0 legacy=0x7f3a7c15ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.194+0000 7f3a7a59c640 1 -- 172.21.15.83:0/2051564628 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a7c0cd000 con 0x7f3a7c15ac00 2024-03-06T07:23:32.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.195+0000 7f3a7a59c640 1 --2- 172.21.15.83:0/2051564628 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a7c15ac00 0x7f3a7c06c910 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c074f80 tx=0x7f3a7402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c79a789dbe86787 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.195+0000 7f3a7959a640 1 -- 172.21.15.83:0/2051564628 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a74039070 con 0x7f3a7c15ac00 2024-03-06T07:23:32.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.195+0000 7f3a7959a640 1 -- 172.21.15.83:0/2051564628 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a7402cbf0 con 0x7f3a7c15ac00 2024-03-06T07:23:32.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.195+0000 7f3a7959a640 1 -- 172.21.15.83:0/2051564628 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a74034440 con 0x7f3a7c15ac00 2024-03-06T07:23:32.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.196+0000 7f3a80bd7640 1 -- 172.21.15.83:0/2051564628 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a7c15ac00 msgr2=0x7f3a7c06c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.196+0000 7f3a80bd7640 1 --2- 172.21.15.83:0/2051564628 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a7c15ac00 0x7f3a7c06c910 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c074f80 tx=0x7f3a7402be40 comp rx=0 tx=0).stop 2024-03-06T07:23:32.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.196+0000 7f3a80bd7640 1 -- 172.21.15.83:0/2051564628 shutdown_connections 2024-03-06T07:23:32.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.196+0000 7f3a80bd7640 1 --2- 172.21.15.83:0/2051564628 >> v2:172.21.15.183:3300/0 conn(0x7f3a7c0708b0 0x7f3a7c070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.196+0000 7f3a80bd7640 1 --2- 172.21.15.83:0/2051564628 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a7c15ac00 0x7f3a7c06c910 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.196+0000 7f3a80bd7640 1 -- 172.21.15.83:0/2051564628 >> 172.21.15.83:0/2051564628 conn(0x7f3a7c0b1650 msgr2=0x7f3a7c0b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:32.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.196+0000 7f3a80bd7640 1 -- 172.21.15.83:0/2051564628 shutdown_connections 2024-03-06T07:23:32.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.196+0000 7f3a80bd7640 1 -- 172.21.15.83:0/2051564628 wait complete. 2024-03-06T07:23:32.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.197+0000 7f3a80bd7640 1 Processor -- start 2024-03-06T07:23:32.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.197+0000 7f3a80bd7640 1 -- start start 2024-03-06T07:23:32.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.198+0000 7f3a80bd7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a7c0708b0 0x7f3a7c151640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.198+0000 7f3a80bd7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a7c15ac00 0x7f3a7c1f2790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.198+0000 7f3a80bd7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a7c15e7a0 0x7f3a7c1575a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.198+0000 7f3a80bd7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c0cd8a0 con 0x7f3a7c15e7a0 2024-03-06T07:23:32.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.198+0000 7f3a80bd7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c0cd5c0 con 0x7f3a7c15ac00 2024-03-06T07:23:32.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.198+0000 7f3a80bd7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c06ce50 con 0x7f3a7c0708b0 2024-03-06T07:23:32.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.198+0000 7f3a7ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a7c15e7a0 0x7f3a7c1575a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.198+0000 7f3a7ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a7c15e7a0 0x7f3a7c1575a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57456/0 (socket says 172.21.15.83:57456) 2024-03-06T07:23:32.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.198+0000 7f3a7ad9d640 1 -- 172.21.15.83:0/76195310 learned_addr learned my addr 172.21.15.83:0/76195310 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:32.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.198+0000 7f3a7ad9d640 1 -- 172.21.15.83:0/76195310 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a7c0708b0 msgr2=0x7f3a7c151640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.198+0000 7f3a7ad9d640 1 --2- 172.21.15.83:0/76195310 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a7c0708b0 0x7f3a7c151640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.198+0000 7f3a7ad9d640 1 -- 172.21.15.83:0/76195310 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a7c15ac00 msgr2=0x7f3a7c1f2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.198+0000 7f3a7ad9d640 1 --2- 172.21.15.83:0/76195310 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a7c15ac00 0x7f3a7c1f2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.198+0000 7f3a7ad9d640 1 -- 172.21.15.83:0/76195310 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a7c151b80 con 0x7f3a7c15e7a0 2024-03-06T07:23:32.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.198+0000 7f3a7ad9d640 1 --2- 172.21.15.83:0/76195310 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a7c15e7a0 0x7f3a7c1575a0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c00c3c0 tx=0x7f3a6c001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.199+0000 7f3a72ffd640 1 -- 172.21.15.83:0/76195310 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c017070 con 0x7f3a7c15e7a0 2024-03-06T07:23:32.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.199+0000 7f3a72ffd640 1 -- 172.21.15.83:0/76195310 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a6c005ce0 con 0x7f3a7c15e7a0 2024-03-06T07:23:32.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.199+0000 7f3a72ffd640 1 -- 172.21.15.83:0/76195310 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c007310 con 0x7f3a7c15e7a0 2024-03-06T07:23:32.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.199+0000 7f3a80bd7640 1 -- 172.21.15.83:0/76195310 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a7c153e40 con 0x7f3a7c15e7a0 2024-03-06T07:23:32.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.199+0000 7f3a80bd7640 1 -- 172.21.15.83:0/76195310 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a7c154400 con 0x7f3a7c15e7a0 2024-03-06T07:23:32.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.200+0000 7f3a80bd7640 1 -- 172.21.15.83:0/76195310 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a7c0687d0 con 0x7f3a7c15e7a0 2024-03-06T07:23:32.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.200+0000 7f3a72ffd640 1 -- 172.21.15.83:0/76195310 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3a6c01a070 con 0x7f3a7c15e7a0 2024-03-06T07:23:32.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.200+0000 7f3a72ffd640 1 --2- 172.21.15.83:0/76195310 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3a5c06b4d0 0x7f3a5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.200+0000 7f3a72ffd640 1 -- 172.21.15.83:0/76195310 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f3a6c0895a0 con 0x7f3a7c15e7a0 2024-03-06T07:23:32.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.200+0000 7f3a7a59c640 1 --2- 172.21.15.83:0/76195310 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3a5c06b4d0 0x7f3a5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.201+0000 7f3a7a59c640 1 --2- 172.21.15.83:0/76195310 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3a5c06b4d0 0x7f3a5c06d990 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f3a74036000 tx=0x7f3a7402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.205+0000 7f3a72ffd640 1 -- 172.21.15.83:0/76195310 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3a6c058450 con 0x7f3a7c15e7a0 2024-03-06T07:23:32.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:32.377+0000 7f3a80bd7640 1 -- 172.21.15.83:0/76195310 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3a7c04f330 con 0x7f3a7c15e7a0 2024-03-06T07:23:33.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.031+0000 7f3a72ffd640 1 -- 172.21.15.83:0/76195310 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v35) v1 ==== 181+0+0 (secure 0 0 0) 0x7f3a6c057d50 con 0x7f3a7c15e7a0 2024-03-06T07:23:33.033 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:23:33.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.034+0000 7f3a80bd7640 1 -- 172.21.15.83:0/76195310 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3a5c06b4d0 msgr2=0x7f3a5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.034+0000 7f3a80bd7640 1 --2- 172.21.15.83:0/76195310 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3a5c06b4d0 0x7f3a5c06d990 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f3a74036000 tx=0x7f3a7402e000 comp rx=0 tx=0).stop 2024-03-06T07:23:33.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.034+0000 7f3a80bd7640 1 -- 172.21.15.83:0/76195310 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a7c15e7a0 msgr2=0x7f3a7c1575a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.034+0000 7f3a80bd7640 1 --2- 172.21.15.83:0/76195310 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a7c15e7a0 0x7f3a7c1575a0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c00c3c0 tx=0x7f3a6c001e70 comp rx=0 tx=0).stop 2024-03-06T07:23:33.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.035+0000 7f3a80bd7640 1 -- 172.21.15.83:0/76195310 shutdown_connections 2024-03-06T07:23:33.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.035+0000 7f3a80bd7640 1 --2- 172.21.15.83:0/76195310 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3a5c06b4d0 0x7f3a5c06d990 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.035+0000 7f3a80bd7640 1 --2- 172.21.15.83:0/76195310 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a7c15e7a0 0x7f3a7c1575a0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.035+0000 7f3a80bd7640 1 --2- 172.21.15.83:0/76195310 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a7c15ac00 0x7f3a7c1f2790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.035+0000 7f3a80bd7640 1 --2- 172.21.15.83:0/76195310 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a7c0708b0 0x7f3a7c151640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.035+0000 7f3a80bd7640 1 -- 172.21.15.83:0/76195310 >> 172.21.15.83:0/76195310 conn(0x7f3a7c0b1650 msgr2=0x7f3a7c0af5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:33.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.035+0000 7f3a80bd7640 1 -- 172.21.15.83:0/76195310 shutdown_connections 2024-03-06T07:23:33.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.035+0000 7f3a80bd7640 1 -- 172.21.15.83:0/76195310 wait complete. 2024-03-06T07:23:33.050 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:23:33.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.154+0000 7f4c14039640 1 Processor -- start 2024-03-06T07:23:33.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.155+0000 7f4c14039640 1 -- start start 2024-03-06T07:23:33.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.155+0000 7f4c14039640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4c0c15ac00 0x7f4c0c06c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.155+0000 7f4c14039640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f4c0c070d10 0x7f4c0c15eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.155+0000 7f4c14039640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c0c0711c0 con 0x7f4c0c0708b0 2024-03-06T07:23:33.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.155+0000 7f4c14039640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c0c0cdc10 con 0x7f4c0c15ac00 2024-03-06T07:23:33.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.155+0000 7f4c14039640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c0c15afe0 con 0x7f4c0c070d10 2024-03-06T07:23:33.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.156+0000 7f4c11dd5640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4c0c15ac00 0x7f4c0c06c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.156+0000 7f4c11dd5640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4c0c15ac00 0x7f4c0c06c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57472/0 (socket says 172.21.15.83:57472) 2024-03-06T07:23:33.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.156+0000 7f4c11dd5640 1 -- 172.21.15.83:0/630939384 learned_addr learned my addr 172.21.15.83:0/630939384 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:33.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.156+0000 7f4c125d6640 1 --2- 172.21.15.83:0/630939384 >> v2:172.21.15.183:3300/0 conn(0x7f4c0c070d10 0x7f4c0c15eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.156+0000 7f4c11dd5640 1 -- 172.21.15.83:0/630939384 >> v2:172.21.15.183:3300/0 conn(0x7f4c0c070d10 msgr2=0x7f4c0c15eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:33.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.156+0000 7f4c11dd5640 1 --2- 172.21.15.83:0/630939384 >> v2:172.21.15.183:3300/0 conn(0x7f4c0c070d10 0x7f4c0c15eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.156+0000 7f4c11dd5640 1 -- 172.21.15.83:0/630939384 >> v1:172.21.15.183:6789/0 conn(0x7f4c0c0708b0 legacy=0x7f4c0c15e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:33.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.156+0000 7f4c11dd5640 1 -- 172.21.15.83:0/630939384 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c0c0cd000 con 0x7f4c0c15ac00 2024-03-06T07:23:33.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.157+0000 7f4c11dd5640 1 --2- 172.21.15.83:0/630939384 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c0c15ac00 0x7f4c0c06c910 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c0cc0a0 tx=0x7f4bfc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6d7cc9e68199dc5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.157+0000 7f4c10dd3640 1 -- 172.21.15.83:0/630939384 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc02e070 con 0x7f4c0c15ac00 2024-03-06T07:23:33.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.157+0000 7f4c10dd3640 1 -- 172.21.15.83:0/630939384 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bfc037070 con 0x7f4c0c15ac00 2024-03-06T07:23:33.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.157+0000 7f4c10dd3640 1 -- 172.21.15.83:0/630939384 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc031a00 con 0x7f4c0c15ac00 2024-03-06T07:23:33.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.157+0000 7f4c14039640 1 -- 172.21.15.83:0/630939384 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c0c15ac00 msgr2=0x7f4c0c06c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.157+0000 7f4c14039640 1 --2- 172.21.15.83:0/630939384 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c0c15ac00 0x7f4c0c06c910 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c0cc0a0 tx=0x7f4bfc02b880 comp rx=0 tx=0).stop 2024-03-06T07:23:33.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.157+0000 7f4c14039640 1 -- 172.21.15.83:0/630939384 shutdown_connections 2024-03-06T07:23:33.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.157+0000 7f4c14039640 1 --2- 172.21.15.83:0/630939384 >> v2:172.21.15.183:3300/0 conn(0x7f4c0c070d10 0x7f4c0c15eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.157+0000 7f4c14039640 1 --2- 172.21.15.83:0/630939384 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c0c15ac00 0x7f4c0c06c910 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.157+0000 7f4c14039640 1 -- 172.21.15.83:0/630939384 >> 172.21.15.83:0/630939384 conn(0x7f4c0c0b1650 msgr2=0x7f4c0c0b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:33.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.157+0000 7f4c14039640 1 -- 172.21.15.83:0/630939384 shutdown_connections 2024-03-06T07:23:33.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.157+0000 7f4c14039640 1 -- 172.21.15.83:0/630939384 wait complete. 2024-03-06T07:23:33.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.158+0000 7f4c14039640 1 Processor -- start 2024-03-06T07:23:33.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.159+0000 7f4c14039640 1 -- start start 2024-03-06T07:23:33.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.159+0000 7f4c14039640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c0c0708b0 0x7f4c0c14b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.159+0000 7f4c14039640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4c0c070d10 0x7f4c0c13ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.159+0000 7f4c14039640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4c0c15ac00 0x7f4c0c15d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.159+0000 7f4c14039640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0c0cd8a0 con 0x7f4c0c0708b0 2024-03-06T07:23:33.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.159+0000 7f4c14039640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0c0cd5c0 con 0x7f4c0c15ac00 2024-03-06T07:23:33.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.159+0000 7f4c14039640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0c06ce50 con 0x7f4c0c070d10 2024-03-06T07:23:33.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.159+0000 7f4c11dd5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c0c0708b0 0x7f4c0c14b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.159+0000 7f4c11dd5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c0c0708b0 0x7f4c0c14b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57476/0 (socket says 172.21.15.83:57476) 2024-03-06T07:23:33.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.159+0000 7f4c11dd5640 1 -- 172.21.15.83:0/4293300555 learned_addr learned my addr 172.21.15.83:0/4293300555 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:33.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.160+0000 7f4c11dd5640 1 -- 172.21.15.83:0/4293300555 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4c0c070d10 msgr2=0x7f4c0c13ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.160+0000 7f4c11dd5640 1 --2- 172.21.15.83:0/4293300555 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4c0c070d10 0x7f4c0c13ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.160+0000 7f4c11dd5640 1 -- 172.21.15.83:0/4293300555 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4c0c15ac00 msgr2=0x7f4c0c15d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.160+0000 7f4c11dd5640 1 --2- 172.21.15.83:0/4293300555 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4c0c15ac00 0x7f4c0c15d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.160+0000 7f4c11dd5640 1 -- 172.21.15.83:0/4293300555 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c0c13f160 con 0x7f4c0c0708b0 2024-03-06T07:23:33.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.160+0000 7f4c11dd5640 1 --2- 172.21.15.83:0/4293300555 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c0c0708b0 0x7f4c0c14b0a0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc02b8c0 tx=0x7f4bfc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.160+0000 7f4c027fc640 1 -- 172.21.15.83:0/4293300555 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc032880 con 0x7f4c0c0708b0 2024-03-06T07:23:33.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.160+0000 7f4c027fc640 1 -- 172.21.15.83:0/4293300555 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bfc032a40 con 0x7f4c0c0708b0 2024-03-06T07:23:33.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.160+0000 7f4c027fc640 1 -- 172.21.15.83:0/4293300555 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc032e30 con 0x7f4c0c0708b0 2024-03-06T07:23:33.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.160+0000 7f4c14039640 1 -- 172.21.15.83:0/4293300555 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c0c1519d0 con 0x7f4c0c0708b0 2024-03-06T07:23:33.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.160+0000 7f4c14039640 1 -- 172.21.15.83:0/4293300555 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c0c151f90 con 0x7f4c0c0708b0 2024-03-06T07:23:33.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.161+0000 7f4c14039640 1 -- 172.21.15.83:0/4293300555 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c0c0687d0 con 0x7f4c0c0708b0 2024-03-06T07:23:33.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.161+0000 7f4c027fc640 1 -- 172.21.15.83:0/4293300555 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4bfc04c060 con 0x7f4c0c0708b0 2024-03-06T07:23:33.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.162+0000 7f4c027fc640 1 --2- 172.21.15.83:0/4293300555 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4bf006b4d0 0x7f4bf006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.162+0000 7f4c027fc640 1 -- 172.21.15.83:0/4293300555 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5357+0+0 (secure 0 0 0) 0x7f4bfc0abbe0 con 0x7f4c0c0708b0 2024-03-06T07:23:33.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.162+0000 7f4c115d4640 1 --2- 172.21.15.83:0/4293300555 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4bf006b4d0 0x7f4bf006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.162+0000 7f4c115d4640 1 --2- 172.21.15.83:0/4293300555 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4bf006b4d0 0x7f4bf006d990 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c14bb20 tx=0x7f4c080079b0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.166+0000 7f4c027fc640 1 -- 172.21.15.83:0/4293300555 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4c0c13f160 con 0x7f4c0c0708b0 2024-03-06T07:23:33.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:33.339+0000 7f4c14039640 1 -- 172.21.15.83:0/4293300555 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4c0c072600 con 0x7f4c0c0708b0 2024-03-06T07:23:34.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.036+0000 7f4c027fc640 1 -- 172.21.15.83:0/4293300555 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v36) v1 ==== 169+0+0 (secure 0 0 0) 0x7f4bfc040070 con 0x7f4c0c0708b0 2024-03-06T07:23:34.038 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'cephfs_data' removed 2024-03-06T07:23:34.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.040+0000 7f4c14039640 1 -- 172.21.15.83:0/4293300555 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4bf006b4d0 msgr2=0x7f4bf006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.040+0000 7f4c14039640 1 --2- 172.21.15.83:0/4293300555 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4bf006b4d0 0x7f4bf006d990 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c14bb20 tx=0x7f4c080079b0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.040+0000 7f4c14039640 1 -- 172.21.15.83:0/4293300555 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c0c0708b0 msgr2=0x7f4c0c14b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.040+0000 7f4c14039640 1 --2- 172.21.15.83:0/4293300555 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c0c0708b0 0x7f4c0c14b0a0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc02b8c0 tx=0x7f4bfc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.040+0000 7f4c14039640 1 -- 172.21.15.83:0/4293300555 shutdown_connections 2024-03-06T07:23:34.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.040+0000 7f4c14039640 1 --2- 172.21.15.83:0/4293300555 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4bf006b4d0 0x7f4bf006d990 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.040+0000 7f4c14039640 1 --2- 172.21.15.83:0/4293300555 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4c0c15ac00 0x7f4c0c15d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.040+0000 7f4c14039640 1 --2- 172.21.15.83:0/4293300555 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4c0c070d10 0x7f4c0c13ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.040+0000 7f4c14039640 1 --2- 172.21.15.83:0/4293300555 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c0c0708b0 0x7f4c0c14b0a0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.040+0000 7f4c14039640 1 -- 172.21.15.83:0/4293300555 >> 172.21.15.83:0/4293300555 conn(0x7f4c0c0b1650 msgr2=0x7f4c0c15efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:34.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.040+0000 7f4c14039640 1 -- 172.21.15.83:0/4293300555 shutdown_connections 2024-03-06T07:23:34.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.040+0000 7f4c14039640 1 -- 172.21.15.83:0/4293300555 wait complete. 2024-03-06T07:23:34.056 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:23:34.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.162+0000 7fcf2a716640 1 Processor -- start 2024-03-06T07:23:34.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.162+0000 7fcf2a716640 1 -- start start 2024-03-06T07:23:34.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.163+0000 7fcf2a716640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcf240719e0 0x7fcf240cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.163+0000 7fcf2a716640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fcf2406c9f0 0x7fcf2406ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.163+0000 7fcf2a716640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf240b3320 con 0x7fcf2415a220 2024-03-06T07:23:34.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.163+0000 7fcf2a716640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf240cf610 con 0x7fcf2406c9f0 2024-03-06T07:23:34.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.163+0000 7fcf2a716640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf240728d0 con 0x7fcf240719e0 2024-03-06T07:23:34.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.163+0000 7fcf28cb3640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fcf2406c9f0 0x7fcf2406ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.163+0000 7fcf237fe640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcf240719e0 0x7fcf240cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.163+0000 7fcf28cb3640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fcf2406c9f0 0x7fcf2406ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:44952/0 (socket says 172.21.15.83:44952) 2024-03-06T07:23:34.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.163+0000 7fcf23fff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fcf2415a220 0x7fcf240712d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:49148/0 (socket says 172.21.15.83:49148) 2024-03-06T07:23:34.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.163+0000 7fcf28cb3640 1 -- 172.21.15.83:0/1655395862 learned_addr learned my addr 172.21.15.83:0/1655395862 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:34.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.164+0000 7fcf237fe640 1 -- 172.21.15.83:0/1655395862 >> v2:172.21.15.183:3300/0 conn(0x7fcf2406c9f0 msgr2=0x7fcf2406ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:34.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.164+0000 7fcf237fe640 1 --2- 172.21.15.83:0/1655395862 >> v2:172.21.15.183:3300/0 conn(0x7fcf2406c9f0 0x7fcf2406ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.164+0000 7fcf237fe640 1 -- 172.21.15.83:0/1655395862 >> v1:172.21.15.183:6789/0 conn(0x7fcf2415a220 legacy=0x7fcf240712d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.164+0000 7fcf237fe640 1 -- 172.21.15.83:0/1655395862 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf240b3590 con 0x7fcf240719e0 2024-03-06T07:23:34.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.164+0000 7fcf28cb3640 1 --2- 172.21.15.83:0/1655395862 >> v2:172.21.15.183:3300/0 conn(0x7fcf2406c9f0 0x7fcf2406ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:34.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.164+0000 7fcf237fe640 1 --2- 172.21.15.83:0/1655395862 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcf240719e0 0x7fcf240cf0d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fcf180097a0 tx=0x7fcf1802c950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f2de97b01bfd030 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:34.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.165+0000 7fcf22ffd640 1 -- 172.21.15.83:0/1655395862 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf18035070 con 0x7fcf240719e0 2024-03-06T07:23:34.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.165+0000 7fcf22ffd640 1 -- 172.21.15.83:0/1655395862 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf18031610 con 0x7fcf240719e0 2024-03-06T07:23:34.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.165+0000 7fcf22ffd640 1 -- 172.21.15.83:0/1655395862 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf180319e0 con 0x7fcf240719e0 2024-03-06T07:23:34.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.165+0000 7fcf2a716640 1 -- 172.21.15.83:0/1655395862 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcf240719e0 msgr2=0x7fcf240cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.165+0000 7fcf2a716640 1 --2- 172.21.15.83:0/1655395862 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcf240719e0 0x7fcf240cf0d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fcf180097a0 tx=0x7fcf1802c950 comp rx=0 tx=0).stop 2024-03-06T07:23:34.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.165+0000 7fcf2a716640 1 -- 172.21.15.83:0/1655395862 shutdown_connections 2024-03-06T07:23:34.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.165+0000 7fcf2a716640 1 --2- 172.21.15.83:0/1655395862 >> v2:172.21.15.183:3300/0 conn(0x7fcf2406c9f0 0x7fcf2406ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.165+0000 7fcf2a716640 1 --2- 172.21.15.83:0/1655395862 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcf240719e0 0x7fcf240cf0d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.165+0000 7fcf2a716640 1 -- 172.21.15.83:0/1655395862 >> 172.21.15.83:0/1655395862 conn(0x7fcf2406b190 msgr2=0x7fcf2406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:34.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.165+0000 7fcf2a716640 1 -- 172.21.15.83:0/1655395862 shutdown_connections 2024-03-06T07:23:34.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.165+0000 7fcf2a716640 1 -- 172.21.15.83:0/1655395862 wait complete. 2024-03-06T07:23:34.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.166+0000 7fcf2a716640 1 Processor -- start 2024-03-06T07:23:34.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.167+0000 7fcf2a716640 1 -- start start 2024-03-06T07:23:34.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.167+0000 7fcf2a716640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcf2406c9f0 0x7fcf24147320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.167+0000 7fcf2a716640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf240719e0 0x7fcf2415c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.167+0000 7fcf2a716640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcf2415a220 0x7fcf2415cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.167+0000 7fcf2a716640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf24072530 con 0x7fcf240719e0 2024-03-06T07:23:34.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.167+0000 7fcf2a716640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf24072030 con 0x7fcf2415a220 2024-03-06T07:23:34.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.167+0000 7fcf2a716640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf240726b0 con 0x7fcf2406c9f0 2024-03-06T07:23:34.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.167+0000 7fcf237fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf240719e0 0x7fcf2415c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.167+0000 7fcf237fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf240719e0 0x7fcf2415c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57480/0 (socket says 172.21.15.83:57480) 2024-03-06T07:23:34.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.167+0000 7fcf237fe640 1 -- 172.21.15.83:0/2337235848 learned_addr learned my addr 172.21.15.83:0/2337235848 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:34.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.167+0000 7fcf23fff640 1 --2- 172.21.15.83:0/2337235848 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcf2406c9f0 0x7fcf24147320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.167+0000 7fcf28cb3640 1 --2- 172.21.15.83:0/2337235848 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcf2415a220 0x7fcf2415cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.168+0000 7fcf237fe640 1 -- 172.21.15.83:0/2337235848 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcf2406c9f0 msgr2=0x7fcf24147320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.168+0000 7fcf237fe640 1 --2- 172.21.15.83:0/2337235848 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcf2406c9f0 0x7fcf24147320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.168+0000 7fcf237fe640 1 -- 172.21.15.83:0/2337235848 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcf2415a220 msgr2=0x7fcf2415cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.168+0000 7fcf237fe640 1 --2- 172.21.15.83:0/2337235848 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcf2415a220 0x7fcf2415cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.168+0000 7fcf237fe640 1 -- 172.21.15.83:0/2337235848 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf24147860 con 0x7fcf240719e0 2024-03-06T07:23:34.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.168+0000 7fcf237fe640 1 --2- 172.21.15.83:0/2337235848 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf240719e0 0x7fcf2415c610 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fcf18000c00 tx=0x7fcf18004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:34.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.168+0000 7fcf20ff9640 1 -- 172.21.15.83:0/2337235848 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf18035040 con 0x7fcf240719e0 2024-03-06T07:23:34.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.168+0000 7fcf20ff9640 1 -- 172.21.15.83:0/2337235848 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf1803aa90 con 0x7fcf240719e0 2024-03-06T07:23:34.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.168+0000 7fcf20ff9640 1 -- 172.21.15.83:0/2337235848 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf18044410 con 0x7fcf240719e0 2024-03-06T07:23:34.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.168+0000 7fcf2a716640 1 -- 172.21.15.83:0/2337235848 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf24156220 con 0x7fcf240719e0 2024-03-06T07:23:34.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.168+0000 7fcf2a716640 1 -- 172.21.15.83:0/2337235848 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf241567e0 con 0x7fcf240719e0 2024-03-06T07:23:34.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.169+0000 7fcf2a716640 1 -- 172.21.15.83:0/2337235848 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf240732b0 con 0x7fcf240719e0 2024-03-06T07:23:34.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.169+0000 7fcf20ff9640 1 -- 172.21.15.83:0/2337235848 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcf1804b060 con 0x7fcf240719e0 2024-03-06T07:23:34.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.170+0000 7fcf20ff9640 1 --2- 172.21.15.83:0/2337235848 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcf1006b500 0x7fcf1006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.170+0000 7fcf20ff9640 1 -- 172.21.15.83:0/2337235848 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fcf180ac8c0 con 0x7fcf240719e0 2024-03-06T07:23:34.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.170+0000 7fcf23fff640 1 --2- 172.21.15.83:0/2337235848 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcf1006b500 0x7fcf1006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.170+0000 7fcf23fff640 1 --2- 172.21.15.83:0/2337235848 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcf1006b500 0x7fcf1006d9c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fcf14007fd0 tx=0x7fcf14007e80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:34.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.174+0000 7fcf20ff9640 1 -- 172.21.15.83:0/2337235848 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcf24147860 con 0x7fcf240719e0 2024-03-06T07:23:34.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.351+0000 7fcf2a716640 1 -- 172.21.15.83:0/2337235848 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fcf2406b8a0 con 0x7fcf240719e0 2024-03-06T07:23:34.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.354+0000 7fcf20ff9640 1 -- 172.21.15.83:0/2337235848 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7fcf1807b3f0 con 0x7fcf240719e0 2024-03-06T07:23:34.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.356+0000 7fcf2a716640 1 -- 172.21.15.83:0/2337235848 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcf1006b500 msgr2=0x7fcf1006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.356+0000 7fcf2a716640 1 --2- 172.21.15.83:0/2337235848 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcf1006b500 0x7fcf1006d9c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fcf14007fd0 tx=0x7fcf14007e80 comp rx=0 tx=0).stop 2024-03-06T07:23:34.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.356+0000 7fcf2a716640 1 -- 172.21.15.83:0/2337235848 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf240719e0 msgr2=0x7fcf2415c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.356+0000 7fcf2a716640 1 --2- 172.21.15.83:0/2337235848 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf240719e0 0x7fcf2415c610 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fcf18000c00 tx=0x7fcf18004730 comp rx=0 tx=0).stop 2024-03-06T07:23:34.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.356+0000 7fcf2a716640 1 -- 172.21.15.83:0/2337235848 shutdown_connections 2024-03-06T07:23:34.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.356+0000 7fcf2a716640 1 --2- 172.21.15.83:0/2337235848 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcf1006b500 0x7fcf1006d9c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.356+0000 7fcf2a716640 1 --2- 172.21.15.83:0/2337235848 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcf2415a220 0x7fcf2415cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.356+0000 7fcf2a716640 1 --2- 172.21.15.83:0/2337235848 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf240719e0 0x7fcf2415c610 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.356+0000 7fcf2a716640 1 --2- 172.21.15.83:0/2337235848 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcf2406c9f0 0x7fcf24147320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.356+0000 7fcf2a716640 1 -- 172.21.15.83:0/2337235848 >> 172.21.15.83:0/2337235848 conn(0x7fcf2406b190 msgr2=0x7fcf240712d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:34.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.357+0000 7fcf2a716640 1 -- 172.21.15.83:0/2337235848 shutdown_connections 2024-03-06T07:23:34.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.357+0000 7fcf2a716640 1 -- 172.21.15.83:0/2337235848 wait complete. 2024-03-06T07:23:34.369 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dir_deletion' 2024-03-06T07:23:34.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.468+0000 7fba4d9f6640 1 Processor -- start 2024-03-06T07:23:34.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.468+0000 7fba4d9f6640 1 -- start start 2024-03-06T07:23:34.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.469+0000 7fba4d9f6640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fba481625a0 0x7fba480cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.469+0000 7fba4d9f6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba480ceac0 con 0x7fba480d4920 2024-03-06T07:23:34.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.469+0000 7fba4d9f6640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba480d2dc0 con 0x7fba481621c0 2024-03-06T07:23:34.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.469+0000 7fba4d9f6640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba480cf690 con 0x7fba481625a0 2024-03-06T07:23:34.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.469+0000 7fba3e7fc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fba480d4920 0x7fba4815dd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49050/0 (socket says 172.21.15.83:49050) 2024-03-06T07:23:34.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.469+0000 7fba3e7fc640 1 -- 172.21.15.83:0/3060698550 learned_addr learned my addr 172.21.15.83:0/3060698550 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:23:34.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.469+0000 7fba477fe640 1 --2- 172.21.15.83:0/3060698550 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fba481625a0 0x7fba480cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.469+0000 7fba467fc640 1 -- 172.21.15.83:0/3060698550 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1748527061 0 0) 0x7fba480ceac0 con 0x7fba480d4920 2024-03-06T07:23:34.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.469+0000 7fba467fc640 1 -- 172.21.15.83:0/3060698550 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba34003520 con 0x7fba480d4920 2024-03-06T07:23:34.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.469+0000 7fba467fc640 1 -- 172.21.15.83:0/3060698550 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1818546602 0 0) 0x7fba34003520 con 0x7fba480d4920 2024-03-06T07:23:34.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.469+0000 7fba467fc640 1 -- 172.21.15.83:0/3060698550 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fba481625a0 msgr2=0x7fba480cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:34.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.469+0000 7fba467fc640 1 --2- 172.21.15.83:0/3060698550 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fba481625a0 0x7fba480cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.469+0000 7fba467fc640 1 -- 172.21.15.83:0/3060698550 >> v1:172.21.15.183:6789/0 conn(0x7fba481621c0 legacy=0x7fba480d26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.469+0000 7fba467fc640 1 -- 172.21.15.83:0/3060698550 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba480ced60 con 0x7fba480d4920 2024-03-06T07:23:34.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.469+0000 7fba467fc640 1 -- 172.21.15.83:0/3060698550 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7fba30001670 con 0x7fba480d4920 2024-03-06T07:23:34.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.469+0000 7fba467fc640 1 -- 172.21.15.83:0/3060698550 >> v1:172.21.15.83:6789/0 conn(0x7fba480d4920 legacy=0x7fba4815dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.469+0000 7fba467fc640 1 --2- 172.21.15.83:0/3060698550 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fba340039e0 0x7fba34003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.470+0000 7fba467fc640 1 --2- 172.21.15.83:0/3060698550 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fba481625a0 0x7fba34004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.470+0000 7fba467fc640 1 --2- 172.21.15.83:0/3060698550 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba34004d40 0x7fba340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.470+0000 7fba467fc640 1 -- 172.21.15.83:0/3060698550 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba480cf690 con 0x7fba34004d40 2024-03-06T07:23:34.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.470+0000 7fba467fc640 1 -- 172.21.15.83:0/3060698550 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3400b9c0 con 0x7fba340039e0 2024-03-06T07:23:34.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.470+0000 7fba467fc640 1 -- 172.21.15.83:0/3060698550 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba3400bbc0 con 0x7fba481625a0 2024-03-06T07:23:34.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.470+0000 7fba477fe640 1 --2- 172.21.15.83:0/3060698550 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba34004d40 0x7fba340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.470+0000 7fba3e7fc640 1 --2- 172.21.15.83:0/3060698550 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fba481625a0 0x7fba34004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.470+0000 7fba46ffd640 1 --2- 172.21.15.83:0/3060698550 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fba340039e0 0x7fba34003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.470+0000 7fba477fe640 1 -- 172.21.15.83:0/3060698550 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fba481625a0 msgr2=0x7fba34004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:34.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.470+0000 7fba477fe640 1 --2- 172.21.15.83:0/3060698550 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fba481625a0 0x7fba34004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.470+0000 7fba477fe640 1 -- 172.21.15.83:0/3060698550 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fba340039e0 msgr2=0x7fba34003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:34.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.470+0000 7fba477fe640 1 --2- 172.21.15.83:0/3060698550 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fba340039e0 0x7fba34003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.470+0000 7fba477fe640 1 -- 172.21.15.83:0/3060698550 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba34003520 con 0x7fba34004d40 2024-03-06T07:23:34.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.471+0000 7fba477fe640 1 --2- 172.21.15.83:0/3060698550 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba34004d40 0x7fba340051c0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fba40009fc0 tx=0x7fba34005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c8f932819cf56b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:34.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.471+0000 7fba467fc640 1 -- 172.21.15.83:0/3060698550 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba40039070 con 0x7fba34004d40 2024-03-06T07:23:34.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.471+0000 7fba467fc640 1 -- 172.21.15.83:0/3060698550 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba4002cc20 con 0x7fba34004d40 2024-03-06T07:23:34.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.471+0000 7fba467fc640 1 -- 172.21.15.83:0/3060698550 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba40009c10 con 0x7fba34004d40 2024-03-06T07:23:34.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.471+0000 7fba4d9f6640 1 -- 172.21.15.83:0/3060698550 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba34004d40 msgr2=0x7fba340051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.471+0000 7fba4d9f6640 1 --2- 172.21.15.83:0/3060698550 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba34004d40 0x7fba340051c0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fba40009fc0 tx=0x7fba34005830 comp rx=0 tx=0).stop 2024-03-06T07:23:34.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.471+0000 7fba477fe640 1 -- 172.21.15.83:0/3060698550 reap_dead start 2024-03-06T07:23:34.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.471+0000 7fba4d9f6640 1 -- 172.21.15.83:0/3060698550 shutdown_connections 2024-03-06T07:23:34.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.471+0000 7fba4d9f6640 1 -- 172.21.15.83:0/3060698550 >> 172.21.15.83:0/3060698550 conn(0x7fba480d3d10 msgr2=0x7fba480d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:34.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.471+0000 7fba4d9f6640 1 -- 172.21.15.83:0/3060698550 shutdown_connections 2024-03-06T07:23:34.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.471+0000 7fba4d9f6640 1 -- 172.21.15.83:0/3060698550 wait complete. 2024-03-06T07:23:34.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.472+0000 7fba4d9f6640 1 Processor -- start 2024-03-06T07:23:34.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.473+0000 7fba4d9f6640 1 -- start start 2024-03-06T07:23:34.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.473+0000 7fba4d9f6640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fba480cc600 0x7fba480c9a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.473+0000 7fba4d9f6640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fba481f4c10 0x7fba480c9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.473+0000 7fba4d9f6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba480cc9e0 0x7fba481e7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.473+0000 7fba4d9f6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba480cfb20 con 0x7fba480cc9e0 2024-03-06T07:23:34.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.473+0000 7fba4d9f6640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba480cf320 con 0x7fba481f4c10 2024-03-06T07:23:34.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.473+0000 7fba4d9f6640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba480cfca0 con 0x7fba480cc600 2024-03-06T07:23:34.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.474+0000 7fba477fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba480cc9e0 0x7fba481e7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.474+0000 7fba477fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba480cc9e0 0x7fba481e7700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57502/0 (socket says 172.21.15.83:57502) 2024-03-06T07:23:34.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.474+0000 7fba477fe640 1 -- 172.21.15.83:0/1509600419 learned_addr learned my addr 172.21.15.83:0/1509600419 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:34.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.474+0000 7fba46ffd640 1 --2- 172.21.15.83:0/1509600419 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fba480cc600 0x7fba480c9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.474+0000 7fba3e7fc640 1 --2- 172.21.15.83:0/1509600419 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fba481f4c10 0x7fba480c9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.474+0000 7fba477fe640 1 -- 172.21.15.83:0/1509600419 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fba480cc600 msgr2=0x7fba480c9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.474+0000 7fba477fe640 1 --2- 172.21.15.83:0/1509600419 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fba480cc600 0x7fba480c9a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.474+0000 7fba477fe640 1 -- 172.21.15.83:0/1509600419 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fba481f4c10 msgr2=0x7fba480c9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.474+0000 7fba477fe640 1 --2- 172.21.15.83:0/1509600419 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fba481f4c10 0x7fba480c9f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.474+0000 7fba477fe640 1 -- 172.21.15.83:0/1509600419 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba481f3eb0 con 0x7fba480cc9e0 2024-03-06T07:23:34.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.474+0000 7fba3e7fc640 1 --2- 172.21.15.83:0/1509600419 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fba481f4c10 0x7fba480c9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:34.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.474+0000 7fba477fe640 1 --2- 172.21.15.83:0/1509600419 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba480cc9e0 0x7fba481e7700 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fba40002230 tx=0x7fba40009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:34.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.474+0000 7fba44ff9640 1 -- 172.21.15.83:0/1509600419 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba40039030 con 0x7fba480cc9e0 2024-03-06T07:23:34.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.474+0000 7fba44ff9640 1 -- 172.21.15.83:0/1509600419 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba40043070 con 0x7fba480cc9e0 2024-03-06T07:23:34.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.474+0000 7fba4d9f6640 1 -- 172.21.15.83:0/1509600419 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba481e8430 con 0x7fba480cc9e0 2024-03-06T07:23:34.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.474+0000 7fba44ff9640 1 -- 172.21.15.83:0/1509600419 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba40009c10 con 0x7fba480cc9e0 2024-03-06T07:23:34.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.474+0000 7fba4d9f6640 1 -- 172.21.15.83:0/1509600419 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba481f4410 con 0x7fba480cc9e0 2024-03-06T07:23:34.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.475+0000 7fba4d9f6640 1 -- 172.21.15.83:0/1509600419 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba4804f330 con 0x7fba480cc9e0 2024-03-06T07:23:34.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.476+0000 7fba44ff9640 1 -- 172.21.15.83:0/1509600419 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fba40005d10 con 0x7fba480cc9e0 2024-03-06T07:23:34.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.476+0000 7fba44ff9640 1 --2- 172.21.15.83:0/1509600419 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fba2806b4d0 0x7fba2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.476+0000 7fba44ff9640 1 -- 172.21.15.83:0/1509600419 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fba400ac070 con 0x7fba480cc9e0 2024-03-06T07:23:34.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.476+0000 7fba46ffd640 1 --2- 172.21.15.83:0/1509600419 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fba2806b4d0 0x7fba2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.476+0000 7fba46ffd640 1 --2- 172.21.15.83:0/1509600419 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fba2806b4d0 0x7fba2806d990 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fba38000c00 tx=0x7fba38009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:34.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.481+0000 7fba44ff9640 1 -- 172.21.15.83:0/1509600419 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fba481f3eb0 con 0x7fba480cc9e0 2024-03-06T07:23:34.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:34.719+0000 7fba4d9f6640 1 -- 172.21.15.83:0/1509600419 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]} v 0) v1 -- 0x7fba480b50b0 con 0x7fba480cc9e0 2024-03-06T07:23:35.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.041+0000 7fba44ff9640 1 -- 172.21.15.83:0/1509600419 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]}]=0 v113) v1 ==== 132+0+0 (secure 0 0 0) 0x7fba4007aba0 con 0x7fba480cc9e0 2024-03-06T07:23:35.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.044+0000 7fba4d9f6640 1 -- 172.21.15.83:0/1509600419 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fba2806b4d0 msgr2=0x7fba2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.044+0000 7fba4d9f6640 1 --2- 172.21.15.83:0/1509600419 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fba2806b4d0 0x7fba2806d990 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fba38000c00 tx=0x7fba38009040 comp rx=0 tx=0).stop 2024-03-06T07:23:35.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.044+0000 7fba4d9f6640 1 -- 172.21.15.83:0/1509600419 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba480cc9e0 msgr2=0x7fba481e7700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.044+0000 7fba4d9f6640 1 --2- 172.21.15.83:0/1509600419 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba480cc9e0 0x7fba481e7700 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fba40002230 tx=0x7fba40009750 comp rx=0 tx=0).stop 2024-03-06T07:23:35.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.044+0000 7fba4d9f6640 1 -- 172.21.15.83:0/1509600419 shutdown_connections 2024-03-06T07:23:35.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.044+0000 7fba4d9f6640 1 --2- 172.21.15.83:0/1509600419 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fba2806b4d0 0x7fba2806d990 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.044+0000 7fba4d9f6640 1 --2- 172.21.15.83:0/1509600419 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba480cc9e0 0x7fba481e7700 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.044+0000 7fba4d9f6640 1 --2- 172.21.15.83:0/1509600419 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fba481f4c10 0x7fba480c9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.044+0000 7fba4d9f6640 1 --2- 172.21.15.83:0/1509600419 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fba480cc600 0x7fba480c9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.044+0000 7fba4d9f6640 1 -- 172.21.15.83:0/1509600419 >> 172.21.15.83:0/1509600419 conn(0x7fba480d3d10 msgr2=0x7fba480b7830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:35.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.044+0000 7fba4d9f6640 1 -- 172.21.15.83:0/1509600419 shutdown_connections 2024-03-06T07:23:35.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.044+0000 7fba4d9f6640 1 -- 172.21.15.83:0/1509600419 wait complete. 2024-03-06T07:23:35.059 INFO:tasks.cephfs_test_runner:That when deleting a bunch of dentries and the containing ... ok 2024-03-06T07:23:35.060 INFO:tasks.cephfs_test_runner:Starting test: test_dirfrag_limit (tasks.cephfs.test_strays.TestStrays) That the directory fragment size cannot exceed mds_bal_fragment_size_max (using a limit of 50 in all configurations). 2024-03-06T07:23:35.060 INFO:tasks.cephfs_test_runner:test_dirfrag_limit (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:23:35.060 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:35.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.163+0000 7f197aad4640 1 Processor -- start 2024-03-06T07:23:35.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.164+0000 7f197aad4640 1 -- start start 2024-03-06T07:23:35.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.164+0000 7f197aad4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f19741424f0 0x7f19740cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.164+0000 7f197aad4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f19740cd7a0 0x7f197415cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.164+0000 7f197aad4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19740cc980 con 0x7f197406d330 2024-03-06T07:23:35.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.164+0000 7f197aad4640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19740cd220 con 0x7f19740cd7a0 2024-03-06T07:23:35.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.164+0000 7f197aad4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19740ccc60 con 0x7f19741424f0 2024-03-06T07:23:35.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.165+0000 7f1973fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f19740cd7a0 0x7f197415cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.165+0000 7f1973fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f19740cd7a0 0x7f197415cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57506/0 (socket says 172.21.15.83:57506) 2024-03-06T07:23:35.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.165+0000 7f1973fff640 1 -- 172.21.15.83:0/2338063271 learned_addr learned my addr 172.21.15.83:0/2338063271 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:35.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.165+0000 7f1973fff640 1 -- 172.21.15.83:0/2338063271 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f19741424f0 msgr2=0x7f19740cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:35.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.165+0000 7f1973fff640 1 --2- 172.21.15.83:0/2338063271 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f19741424f0 0x7f19740cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.165+0000 7f1973fff640 1 -- 172.21.15.83:0/2338063271 >> v1:172.21.15.183:6789/0 conn(0x7f197406d330 legacy=0x7f1974071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:35.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.165+0000 7f1973fff640 1 -- 172.21.15.83:0/2338063271 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19740ccf40 con 0x7f19740cd7a0 2024-03-06T07:23:35.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.166+0000 7f1973fff640 1 --2- 172.21.15.83:0/2338063271 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f19740cd7a0 0x7f197415cef0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f1960009870 tx=0x7f196002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=326929e062d4b367 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:35.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.166+0000 7f19737fe640 1 -- 172.21.15.83:0/2338063271 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1960035020 con 0x7f19740cd7a0 2024-03-06T07:23:35.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.166+0000 7f19737fe640 1 -- 172.21.15.83:0/2338063271 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1960031640 con 0x7f19740cd7a0 2024-03-06T07:23:35.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.166+0000 7f19737fe640 1 -- 172.21.15.83:0/2338063271 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f196002d040 con 0x7f19740cd7a0 2024-03-06T07:23:35.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.166+0000 7f197aad4640 1 -- 172.21.15.83:0/2338063271 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f19740cd7a0 msgr2=0x7f197415cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.166+0000 7f197aad4640 1 --2- 172.21.15.83:0/2338063271 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f19740cd7a0 0x7f197415cef0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f1960009870 tx=0x7f196002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.166+0000 7f197aad4640 1 -- 172.21.15.83:0/2338063271 shutdown_connections 2024-03-06T07:23:35.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.166+0000 7f197aad4640 1 --2- 172.21.15.83:0/2338063271 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f19740cd7a0 0x7f197415cef0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.166+0000 7f197aad4640 1 --2- 172.21.15.83:0/2338063271 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f19741424f0 0x7f19740cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.166+0000 7f197aad4640 1 -- 172.21.15.83:0/2338063271 >> 172.21.15.83:0/2338063271 conn(0x7f19740cf8a0 msgr2=0x7f19740cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:35.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.166+0000 7f197aad4640 1 -- 172.21.15.83:0/2338063271 shutdown_connections 2024-03-06T07:23:35.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.166+0000 7f197aad4640 1 -- 172.21.15.83:0/2338063271 wait complete. 2024-03-06T07:23:35.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.167+0000 7f197aad4640 1 Processor -- start 2024-03-06T07:23:35.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.167+0000 7f197aad4640 1 -- start start 2024-03-06T07:23:35.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.168+0000 7f197aad4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f197406d330 0x7f19740ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.168+0000 7f197aad4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f19740cd7a0 0x7f19741449c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.168+0000 7f197aad4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f19741424f0 0x7f19741f79d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.168+0000 7f197aad4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f197406d940 con 0x7f19740cd7a0 2024-03-06T07:23:35.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.168+0000 7f197aad4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f197406d7c0 con 0x7f197406d330 2024-03-06T07:23:35.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.168+0000 7f197aad4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f197406dac0 con 0x7f19741424f0 2024-03-06T07:23:35.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.168+0000 7f1973fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f19740cd7a0 0x7f19741449c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.168+0000 7f1973fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f19740cd7a0 0x7f19741449c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57516/0 (socket says 172.21.15.83:57516) 2024-03-06T07:23:35.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.168+0000 7f1973fff640 1 -- 172.21.15.83:0/1705147934 learned_addr learned my addr 172.21.15.83:0/1705147934 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:35.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.169+0000 7f1979071640 1 --2- 172.21.15.83:0/1705147934 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f19741424f0 0x7f19741f79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.169+0000 7f1973fff640 1 -- 172.21.15.83:0/1705147934 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f19741424f0 msgr2=0x7f19741f79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.169+0000 7f1973fff640 1 --2- 172.21.15.83:0/1705147934 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f19741424f0 0x7f19741f79d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.169+0000 7f1978870640 1 --2- 172.21.15.83:0/1705147934 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f197406d330 0x7f19740ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.169+0000 7f1973fff640 1 -- 172.21.15.83:0/1705147934 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f197406d330 msgr2=0x7f19740ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.169+0000 7f1973fff640 1 --2- 172.21.15.83:0/1705147934 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f197406d330 0x7f19740ad860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.169+0000 7f1973fff640 1 -- 172.21.15.83:0/1705147934 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19740b4a70 con 0x7f19740cd7a0 2024-03-06T07:23:35.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.169+0000 7f1973fff640 1 --2- 172.21.15.83:0/1705147934 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f19740cd7a0 0x7f19741449c0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f196002b930 tx=0x7f19600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:35.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.169+0000 7f19717fa640 1 -- 172.21.15.83:0/1705147934 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f196003ac60 con 0x7f19740cd7a0 2024-03-06T07:23:35.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.169+0000 7f19717fa640 1 -- 172.21.15.83:0/1705147934 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f196003ae20 con 0x7f19740cd7a0 2024-03-06T07:23:35.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.169+0000 7f197aad4640 1 -- 172.21.15.83:0/1705147934 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19740adf60 con 0x7f19740cd7a0 2024-03-06T07:23:35.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.169+0000 7f19717fa640 1 -- 172.21.15.83:0/1705147934 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1960044630 con 0x7f19740cd7a0 2024-03-06T07:23:35.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.169+0000 7f197aad4640 1 -- 172.21.15.83:0/1705147934 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19740ae800 con 0x7f19740cd7a0 2024-03-06T07:23:35.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.170+0000 7f197aad4640 1 -- 172.21.15.83:0/1705147934 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f197404f330 con 0x7f19740cd7a0 2024-03-06T07:23:35.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.170+0000 7f19717fa640 1 -- 172.21.15.83:0/1705147934 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f196004b060 con 0x7f19740cd7a0 2024-03-06T07:23:35.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.171+0000 7f19717fa640 1 --2- 172.21.15.83:0/1705147934 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f195806b4d0 0x7f195806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.171+0000 7f19717fa640 1 -- 172.21.15.83:0/1705147934 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f19600abd70 con 0x7f19740cd7a0 2024-03-06T07:23:35.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.171+0000 7f1978870640 1 --2- 172.21.15.83:0/1705147934 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f195806b4d0 0x7f195806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.171+0000 7f1978870640 1 --2- 172.21.15.83:0/1705147934 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f195806b4d0 0x7f195806d990 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f19741458d0 tx=0x7f1968008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:35.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.175+0000 7f19717fa640 1 -- 172.21.15.83:0/1705147934 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f196003f070 con 0x7f19740cd7a0 2024-03-06T07:23:35.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.347+0000 7f197aad4640 1 -- 172.21.15.83:0/1705147934 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19740ae2f0 con 0x7f19740cd7a0 2024-03-06T07:23:35.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.348+0000 7f19717fa640 1 -- 172.21.15.83:0/1705147934 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+13032 (secure 0 0 0) 0x7f196002f300 con 0x7f19740cd7a0 2024-03-06T07:23:35.349 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:23:35.350 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":36,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:23:34.032669+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.83:6837/2668925454":"2024-03-07T07:23:31.021205+0000","172.21.15.83:6835/2668925454":"2024-03-07T07:23:31.021205+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:35.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.350+0000 7f197aad4640 1 -- 172.21.15.83:0/1705147934 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f195806b4d0 msgr2=0x7f195806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.350+0000 7f197aad4640 1 --2- 172.21.15.83:0/1705147934 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f195806b4d0 0x7f195806d990 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f19741458d0 tx=0x7f1968008040 comp rx=0 tx=0).stop 2024-03-06T07:23:35.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.350+0000 7f197aad4640 1 -- 172.21.15.83:0/1705147934 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f19740cd7a0 msgr2=0x7f19741449c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.350+0000 7f197aad4640 1 --2- 172.21.15.83:0/1705147934 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f19740cd7a0 0x7f19741449c0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f196002b930 tx=0x7f19600046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.350+0000 7f197aad4640 1 -- 172.21.15.83:0/1705147934 shutdown_connections 2024-03-06T07:23:35.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.350+0000 7f197aad4640 1 --2- 172.21.15.83:0/1705147934 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f195806b4d0 0x7f195806d990 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.350+0000 7f197aad4640 1 --2- 172.21.15.83:0/1705147934 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f19741424f0 0x7f19741f79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.350+0000 7f197aad4640 1 --2- 172.21.15.83:0/1705147934 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f19740cd7a0 0x7f19741449c0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.350+0000 7f197aad4640 1 --2- 172.21.15.83:0/1705147934 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f197406d330 0x7f19740ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.350+0000 7f197aad4640 1 -- 172.21.15.83:0/1705147934 >> 172.21.15.83:0/1705147934 conn(0x7f19740cf8a0 msgr2=0x7f197415edb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:35.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.350+0000 7f197aad4640 1 -- 172.21.15.83:0/1705147934 shutdown_connections 2024-03-06T07:23:35.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.350+0000 7f197aad4640 1 -- 172.21.15.83:0/1705147934 wait complete. 2024-03-06T07:23:35.363 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:35.363 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:35.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.465+0000 7f93629c7640 1 Processor -- start 2024-03-06T07:23:35.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.465+0000 7f93629c7640 1 -- start start 2024-03-06T07:23:35.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.465+0000 7f93629c7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f935c0719e0 0x7f935c0cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.465+0000 7f93629c7640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f935c06c9f0 0x7f935c06ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.465+0000 7f93629c7640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f935c0b3320 con 0x7f935c15a220 2024-03-06T07:23:35.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.465+0000 7f93629c7640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f935c0cf610 con 0x7f935c06c9f0 2024-03-06T07:23:35.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.465+0000 7f93629c7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f935c0728d0 con 0x7f935c0719e0 2024-03-06T07:23:35.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.466+0000 7f9360f64640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f935c06c9f0 0x7f935c06ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.466+0000 7f935b7fe640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f935c0719e0 0x7f935c0cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.466+0000 7f9360f64640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f935c06c9f0 0x7f935c06ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:44996/0 (socket says 172.21.15.83:44996) 2024-03-06T07:23:35.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.466+0000 7f9360f64640 1 -- 172.21.15.83:0/1574320291 learned_addr learned my addr 172.21.15.83:0/1574320291 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:35.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.466+0000 7f9360f64640 1 -- 172.21.15.83:0/1574320291 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f935c0719e0 msgr2=0x7f935c0cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:35.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.466+0000 7f9360f64640 1 --2- 172.21.15.83:0/1574320291 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f935c0719e0 0x7f935c0cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.466+0000 7f9360f64640 1 -- 172.21.15.83:0/1574320291 >> v1:172.21.15.183:6789/0 conn(0x7f935c15a220 legacy=0x7f935c0712d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.466+0000 7f935affd640 1 -- 172.21.15.83:0/1574320291 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1523150050 0 0) 0x7f935c0b3320 con 0x7f935c15a220 2024-03-06T07:23:35.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.466+0000 7f9360f64640 1 -- 172.21.15.83:0/1574320291 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f935c0b3590 con 0x7f935c06c9f0 2024-03-06T07:23:35.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.466+0000 7f935b7fe640 1 --2- 172.21.15.83:0/1574320291 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f935c0719e0 0x7f935c0cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:35.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.467+0000 7f9360f64640 1 --2- 172.21.15.83:0/1574320291 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f935c06c9f0 0x7f935c06ce80 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f935400af60 tx=0x7f935402cfe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e3b61d0c038aa7d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:35.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.467+0000 7f935affd640 1 -- 172.21.15.83:0/1574320291 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f935402d980 con 0x7f935c06c9f0 2024-03-06T07:23:35.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.467+0000 7f935affd640 1 -- 172.21.15.83:0/1574320291 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f935402db20 con 0x7f935c06c9f0 2024-03-06T07:23:35.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.467+0000 7f935affd640 1 -- 172.21.15.83:0/1574320291 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9354008500 con 0x7f935c06c9f0 2024-03-06T07:23:35.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.468+0000 7f93629c7640 1 -- 172.21.15.83:0/1574320291 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f935c06c9f0 msgr2=0x7f935c06ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.468+0000 7f93629c7640 1 --2- 172.21.15.83:0/1574320291 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f935c06c9f0 0x7f935c06ce80 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f935400af60 tx=0x7f935402cfe0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.468+0000 7f93629c7640 1 -- 172.21.15.83:0/1574320291 shutdown_connections 2024-03-06T07:23:35.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.468+0000 7f93629c7640 1 --2- 172.21.15.83:0/1574320291 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f935c06c9f0 0x7f935c06ce80 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.468+0000 7f93629c7640 1 --2- 172.21.15.83:0/1574320291 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f935c0719e0 0x7f935c0cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.468+0000 7f93629c7640 1 -- 172.21.15.83:0/1574320291 >> 172.21.15.83:0/1574320291 conn(0x7f935c06b190 msgr2=0x7f935c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:35.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.468+0000 7f93629c7640 1 -- 172.21.15.83:0/1574320291 shutdown_connections 2024-03-06T07:23:35.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.468+0000 7f93629c7640 1 -- 172.21.15.83:0/1574320291 wait complete. 2024-03-06T07:23:35.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.469+0000 7f93629c7640 1 Processor -- start 2024-03-06T07:23:35.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.469+0000 7f93629c7640 1 -- start start 2024-03-06T07:23:35.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.470+0000 7f93629c7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f935c06c9f0 0x7f935c0a9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.470+0000 7f93629c7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f935c0719e0 0x7f935c0aa970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.470+0000 7f93629c7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f935c15a220 0x7f935c149560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.470+0000 7f93629c7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f935c072530 con 0x7f935c0719e0 2024-03-06T07:23:35.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.470+0000 7f93629c7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f935c072030 con 0x7f935c15a220 2024-03-06T07:23:35.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.470+0000 7f93629c7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f935c0726b0 con 0x7f935c06c9f0 2024-03-06T07:23:35.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.470+0000 7f935b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f935c0719e0 0x7f935c0aa970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.470+0000 7f935b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f935c0719e0 0x7f935c0aa970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57520/0 (socket says 172.21.15.83:57520) 2024-03-06T07:23:35.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.470+0000 7f935b7fe640 1 -- 172.21.15.83:0/923409843 learned_addr learned my addr 172.21.15.83:0/923409843 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:35.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.470+0000 7f9360f64640 1 --2- 172.21.15.83:0/923409843 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f935c15a220 0x7f935c149560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.470+0000 7f935bfff640 1 --2- 172.21.15.83:0/923409843 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f935c06c9f0 0x7f935c0a9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.470+0000 7f935b7fe640 1 -- 172.21.15.83:0/923409843 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f935c06c9f0 msgr2=0x7f935c0a9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.470+0000 7f935b7fe640 1 --2- 172.21.15.83:0/923409843 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f935c06c9f0 0x7f935c0a9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.470+0000 7f935b7fe640 1 -- 172.21.15.83:0/923409843 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f935c15a220 msgr2=0x7f935c149560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.470+0000 7f935b7fe640 1 --2- 172.21.15.83:0/923409843 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f935c15a220 0x7f935c149560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.470+0000 7f935b7fe640 1 -- 172.21.15.83:0/923409843 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f935c0aa520 con 0x7f935c0719e0 2024-03-06T07:23:35.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.470+0000 7f9360f64640 1 --2- 172.21.15.83:0/923409843 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f935c15a220 0x7f935c149560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:35.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.470+0000 7f935bfff640 1 --2- 172.21.15.83:0/923409843 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f935c06c9f0 0x7f935c0a9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:35.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.470+0000 7f935b7fe640 1 --2- 172.21.15.83:0/923409843 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f935c0719e0 0x7f935c0aa970 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f935000ebe0 tx=0x7f935000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:35.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.471+0000 7f9358ff9640 1 -- 172.21.15.83:0/923409843 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9350017070 con 0x7f935c0719e0 2024-03-06T07:23:35.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.471+0000 7f9358ff9640 1 -- 172.21.15.83:0/923409843 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f935000cbd0 con 0x7f935c0719e0 2024-03-06T07:23:35.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.471+0000 7f93629c7640 1 -- 172.21.15.83:0/923409843 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f935c147510 con 0x7f935c0719e0 2024-03-06T07:23:35.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.471+0000 7f9358ff9640 1 -- 172.21.15.83:0/923409843 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9350013470 con 0x7f935c0719e0 2024-03-06T07:23:35.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.471+0000 7f93629c7640 1 -- 172.21.15.83:0/923409843 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f935c147ad0 con 0x7f935c0719e0 2024-03-06T07:23:35.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.472+0000 7f93629c7640 1 -- 172.21.15.83:0/923409843 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f935c0732b0 con 0x7f935c0719e0 2024-03-06T07:23:35.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.472+0000 7f9358ff9640 1 -- 172.21.15.83:0/923409843 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f935001a070 con 0x7f935c0719e0 2024-03-06T07:23:35.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.472+0000 7f9358ff9640 1 --2- 172.21.15.83:0/923409843 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f934806b4d0 0x7f934806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.473+0000 7f935bfff640 1 --2- 172.21.15.83:0/923409843 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f934806b4d0 0x7f934806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.473+0000 7f9358ff9640 1 -- 172.21.15.83:0/923409843 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f9350089e10 con 0x7f935c0719e0 2024-03-06T07:23:35.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.473+0000 7f935bfff640 1 --2- 172.21.15.83:0/923409843 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f934806b4d0 0x7f934806d990 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f934c000f60 tx=0x7f934c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:35.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.477+0000 7f9358ff9640 1 -- 172.21.15.83:0/923409843 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9350058f90 con 0x7f935c0719e0 2024-03-06T07:23:35.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.646+0000 7f93629c7640 1 -- 172.21.15.83:0/923409843 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f935c068ff0 con 0x7f935c0719e0 2024-03-06T07:23:35.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.647+0000 7f9358ff9640 1 -- 172.21.15.83:0/923409843 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v36) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9350058890 con 0x7f935c0719e0 2024-03-06T07:23:35.649 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:23:35.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.648+0000 7f93629c7640 1 -- 172.21.15.83:0/923409843 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f934806b4d0 msgr2=0x7f934806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.648+0000 7f93629c7640 1 --2- 172.21.15.83:0/923409843 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f934806b4d0 0x7f934806d990 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f934c000f60 tx=0x7f934c009040 comp rx=0 tx=0).stop 2024-03-06T07:23:35.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.649+0000 7f93629c7640 1 -- 172.21.15.83:0/923409843 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f935c0719e0 msgr2=0x7f935c0aa970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.649+0000 7f93629c7640 1 --2- 172.21.15.83:0/923409843 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f935c0719e0 0x7f935c0aa970 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f935000ebe0 tx=0x7f935000c520 comp rx=0 tx=0).stop 2024-03-06T07:23:35.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.649+0000 7f93629c7640 1 -- 172.21.15.83:0/923409843 shutdown_connections 2024-03-06T07:23:35.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.649+0000 7f93629c7640 1 --2- 172.21.15.83:0/923409843 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f934806b4d0 0x7f934806d990 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.649+0000 7f93629c7640 1 --2- 172.21.15.83:0/923409843 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f935c15a220 0x7f935c149560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.649+0000 7f93629c7640 1 --2- 172.21.15.83:0/923409843 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f935c0719e0 0x7f935c0aa970 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.649+0000 7f93629c7640 1 --2- 172.21.15.83:0/923409843 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f935c06c9f0 0x7f935c0a9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.649+0000 7f93629c7640 1 -- 172.21.15.83:0/923409843 >> 172.21.15.83:0/923409843 conn(0x7f935c06b190 msgr2=0x7f935c0cbaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:35.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.649+0000 7f93629c7640 1 -- 172.21.15.83:0/923409843 shutdown_connections 2024-03-06T07:23:35.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.649+0000 7f93629c7640 1 -- 172.21.15.83:0/923409843 wait complete. 2024-03-06T07:23:35.662 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit' 2024-03-06T07:23:35.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.765+0000 7f4ab2193640 1 Processor -- start 2024-03-06T07:23:35.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.766+0000 7f4ab2193640 1 -- start start 2024-03-06T07:23:35.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.766+0000 7f4ab2193640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f4aac0d2380 0x7f4aac0d27e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.766+0000 7f4ab2193640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4aac0cee00 0x7f4aac0cf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.766+0000 7f4ab2193640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aac0b3fc0 con 0x7f4aac15e0b0 2024-03-06T07:23:35.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.766+0000 7f4ab2193640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4aac0ce990 con 0x7f4aac0cee00 2024-03-06T07:23:35.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.766+0000 7f4ab2193640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4aac0d2d20 con 0x7f4aac0d2380 2024-03-06T07:23:35.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.766+0000 7f4aabfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4aac0cee00 0x7f4aac0cf2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.766+0000 7f4aab7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f4aac15e0b0 0x7f4aac0b38b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49060/0 (socket says 172.21.15.83:49060) 2024-03-06T07:23:35.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.766+0000 7f4aab7fe640 1 -- 172.21.15.83:0/3994557758 learned_addr learned my addr 172.21.15.83:0/3994557758 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:23:35.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.767+0000 7f4aa2ffd640 1 --2- 172.21.15.83:0/3994557758 >> v2:172.21.15.183:3300/0 conn(0x7f4aac0d2380 0x7f4aac0d27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.767+0000 7f4aabfff640 1 -- 172.21.15.83:0/3994557758 >> v2:172.21.15.183:3300/0 conn(0x7f4aac0d2380 msgr2=0x7f4aac0d27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:35.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.767+0000 7f4aabfff640 1 --2- 172.21.15.83:0/3994557758 >> v2:172.21.15.183:3300/0 conn(0x7f4aac0d2380 0x7f4aac0d27e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.767+0000 7f4aaaffd640 1 -- 172.21.15.83:0/3994557758 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2680956317 0 0) 0x7f4aac0b3fc0 con 0x7f4aac15e0b0 2024-03-06T07:23:35.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.767+0000 7f4aabfff640 1 -- 172.21.15.83:0/3994557758 >> v1:172.21.15.83:6789/0 conn(0x7f4aac15e0b0 legacy=0x7f4aac0b38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.767+0000 7f4aabfff640 1 -- 172.21.15.83:0/3994557758 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aac0b4290 con 0x7f4aac0cee00 2024-03-06T07:23:35.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.767+0000 7f4aabfff640 1 --2- 172.21.15.83:0/3994557758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4aac0cee00 0x7f4aac0cf2b0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f4aac08dbc0 tx=0x7f4aa402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75fd21b1236b116f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:35.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.768+0000 7f4aaaffd640 1 -- 172.21.15.83:0/3994557758 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa4039070 con 0x7f4aac0cee00 2024-03-06T07:23:35.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.768+0000 7f4aaaffd640 1 -- 172.21.15.83:0/3994557758 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aa402cc80 con 0x7f4aac0cee00 2024-03-06T07:23:35.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.768+0000 7f4aaaffd640 1 -- 172.21.15.83:0/3994557758 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa4034660 con 0x7f4aac0cee00 2024-03-06T07:23:35.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.768+0000 7f4ab2193640 1 -- 172.21.15.83:0/3994557758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4aac0cee00 msgr2=0x7f4aac0cf2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.768+0000 7f4ab2193640 1 --2- 172.21.15.83:0/3994557758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4aac0cee00 0x7f4aac0cf2b0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f4aac08dbc0 tx=0x7f4aa402bed0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.768+0000 7f4ab2193640 1 -- 172.21.15.83:0/3994557758 shutdown_connections 2024-03-06T07:23:35.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.768+0000 7f4ab2193640 1 --2- 172.21.15.83:0/3994557758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4aac0cee00 0x7f4aac0cf2b0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.768+0000 7f4ab2193640 1 --2- 172.21.15.83:0/3994557758 >> v2:172.21.15.183:3300/0 conn(0x7f4aac0d2380 0x7f4aac0d27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.768+0000 7f4ab2193640 1 -- 172.21.15.83:0/3994557758 >> 172.21.15.83:0/3994557758 conn(0x7f4aac08d6f0 msgr2=0x7f4aac09d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:35.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.768+0000 7f4ab2193640 1 -- 172.21.15.83:0/3994557758 shutdown_connections 2024-03-06T07:23:35.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.768+0000 7f4ab2193640 1 -- 172.21.15.83:0/3994557758 wait complete. 2024-03-06T07:23:35.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.769+0000 7f4ab2193640 1 Processor -- start 2024-03-06T07:23:35.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.770+0000 7f4ab2193640 1 -- start start 2024-03-06T07:23:35.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.770+0000 7f4ab2193640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4aac0cee00 0x7f4aac0b2970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.770+0000 7f4ab2193640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4aac0d2380 0x7f4aac152f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.770+0000 7f4ab2193640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4aac15e0b0 0x7f4aac0b69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.770+0000 7f4ab2193640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aac0b4b30 con 0x7f4aac15e0b0 2024-03-06T07:23:35.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.770+0000 7f4ab2193640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aac0b4850 con 0x7f4aac0d2380 2024-03-06T07:23:35.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.770+0000 7f4ab2193640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aac15e490 con 0x7f4aac0cee00 2024-03-06T07:23:35.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.770+0000 7f4aabfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4aac15e0b0 0x7f4aac0b69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.770+0000 7f4aabfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4aac15e0b0 0x7f4aac0b69c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57536/0 (socket says 172.21.15.83:57536) 2024-03-06T07:23:35.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.770+0000 7f4aabfff640 1 -- 172.21.15.83:0/963363370 learned_addr learned my addr 172.21.15.83:0/963363370 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:35.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.771+0000 7f4aa2ffd640 1 --2- 172.21.15.83:0/963363370 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4aac0d2380 0x7f4aac152f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.771+0000 7f4aab7fe640 1 --2- 172.21.15.83:0/963363370 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4aac0cee00 0x7f4aac0b2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.771+0000 7f4aabfff640 1 -- 172.21.15.83:0/963363370 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4aac0cee00 msgr2=0x7f4aac0b2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.771+0000 7f4aabfff640 1 --2- 172.21.15.83:0/963363370 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4aac0cee00 0x7f4aac0b2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.771+0000 7f4aabfff640 1 -- 172.21.15.83:0/963363370 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4aac0d2380 msgr2=0x7f4aac152f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.771+0000 7f4aabfff640 1 --2- 172.21.15.83:0/963363370 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4aac0d2380 0x7f4aac152f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.771+0000 7f4aabfff640 1 -- 172.21.15.83:0/963363370 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aac1539c0 con 0x7f4aac15e0b0 2024-03-06T07:23:35.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.771+0000 7f4aa2ffd640 1 --2- 172.21.15.83:0/963363370 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4aac0d2380 0x7f4aac152f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:35.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.771+0000 7f4aab7fe640 1 --2- 172.21.15.83:0/963363370 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4aac0cee00 0x7f4aac0b2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:35.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.771+0000 7f4aabfff640 1 --2- 172.21.15.83:0/963363370 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4aac15e0b0 0x7f4aac0b69c0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f4aa4009e80 tx=0x7f4aa4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:35.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.771+0000 7f4aa8ff9640 1 -- 172.21.15.83:0/963363370 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa4039050 con 0x7f4aac15e0b0 2024-03-06T07:23:35.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.771+0000 7f4aa8ff9640 1 -- 172.21.15.83:0/963363370 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aa4045070 con 0x7f4aac15e0b0 2024-03-06T07:23:35.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.771+0000 7f4ab2193640 1 -- 172.21.15.83:0/963363370 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aac0b7c90 con 0x7f4aac15e0b0 2024-03-06T07:23:35.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.771+0000 7f4aa8ff9640 1 -- 172.21.15.83:0/963363370 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa403e830 con 0x7f4aac15e0b0 2024-03-06T07:23:35.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.771+0000 7f4ab2193640 1 -- 172.21.15.83:0/963363370 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aac160600 con 0x7f4aac15e0b0 2024-03-06T07:23:35.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.772+0000 7f4ab2193640 1 -- 172.21.15.83:0/963363370 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4aac04f330 con 0x7f4aac15e0b0 2024-03-06T07:23:35.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.772+0000 7f4aa8ff9640 1 -- 172.21.15.83:0/963363370 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4aa403e9d0 con 0x7f4aac15e0b0 2024-03-06T07:23:35.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.773+0000 7f4aa8ff9640 1 --2- 172.21.15.83:0/963363370 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4a8c06b4d0 0x7f4a8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.773+0000 7f4aab7fe640 1 --2- 172.21.15.83:0/963363370 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4a8c06b4d0 0x7f4a8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.773+0000 7f4aa8ff9640 1 -- 172.21.15.83:0/963363370 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4aa40acf60 con 0x7f4aac15e0b0 2024-03-06T07:23:35.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.773+0000 7f4aab7fe640 1 --2- 172.21.15.83:0/963363370 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4a8c06b4d0 0x7f4a8c06d990 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c000f60 tx=0x7f4a9c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:35.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:35.777+0000 7f4aa8ff9640 1 -- 172.21.15.83:0/963363370 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4aa407c170 con 0x7f4aac15e0b0 2024-03-06T07:23:36.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.011+0000 7f4ab2193640 1 -- 172.21.15.83:0/963363370 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]} v 0) v1 -- 0x7f4aac0d2160 con 0x7f4aac15e0b0 2024-03-06T07:23:36.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.043+0000 7f4aa8ff9640 1 -- 172.21.15.83:0/963363370 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]}]=0 v114) v1 ==== 136+0+0 (secure 0 0 0) 0x7f4aa407ba70 con 0x7f4aac15e0b0 2024-03-06T07:23:36.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.045+0000 7f4ab2193640 1 -- 172.21.15.83:0/963363370 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4a8c06b4d0 msgr2=0x7f4a8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.045+0000 7f4ab2193640 1 --2- 172.21.15.83:0/963363370 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4a8c06b4d0 0x7f4a8c06d990 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c000f60 tx=0x7f4a9c009040 comp rx=0 tx=0).stop 2024-03-06T07:23:36.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.045+0000 7f4ab2193640 1 -- 172.21.15.83:0/963363370 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4aac15e0b0 msgr2=0x7f4aac0b69c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.045+0000 7f4ab2193640 1 --2- 172.21.15.83:0/963363370 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4aac15e0b0 0x7f4aac0b69c0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f4aa4009e80 tx=0x7f4aa4004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:36.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.045+0000 7f4ab2193640 1 -- 172.21.15.83:0/963363370 shutdown_connections 2024-03-06T07:23:36.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.045+0000 7f4ab2193640 1 --2- 172.21.15.83:0/963363370 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4a8c06b4d0 0x7f4a8c06d990 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.045+0000 7f4ab2193640 1 --2- 172.21.15.83:0/963363370 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4aac15e0b0 0x7f4aac0b69c0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.045+0000 7f4ab2193640 1 --2- 172.21.15.83:0/963363370 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4aac0d2380 0x7f4aac152f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.045+0000 7f4ab2193640 1 --2- 172.21.15.83:0/963363370 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4aac0cee00 0x7f4aac0b2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.045+0000 7f4ab2193640 1 -- 172.21.15.83:0/963363370 >> 172.21.15.83:0/963363370 conn(0x7f4aac08d6f0 msgr2=0x7f4aac0b5890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:36.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.045+0000 7f4ab2193640 1 -- 172.21.15.83:0/963363370 shutdown_connections 2024-03-06T07:23:36.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.045+0000 7f4ab2193640 1 -- 172.21.15.83:0/963363370 wait complete. 2024-03-06T07:23:36.058 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:23:36.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.160+0000 7fd85473e640 1 Processor -- start 2024-03-06T07:23:36.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.160+0000 7fd85473e640 1 -- start start 2024-03-06T07:23:36.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.161+0000 7fd85473e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd84c1586e0 0x7fd84c0aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.161+0000 7fd85473e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fd84c0aef80 0x7fd84c0cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.161+0000 7fd85473e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd84c0cffd0 0x7fd84c06da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.161+0000 7fd85473e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd84c0d03f0 con 0x7fd84c0cffd0 2024-03-06T07:23:36.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.161+0000 7fd85473e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd84c158ac0 con 0x7fd84c0aef80 2024-03-06T07:23:36.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.161+0000 7fd85473e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd84c15b2a0 con 0x7fd84c1586e0 2024-03-06T07:23:36.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.161+0000 7fd852cdb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd84c0cffd0 0x7fd84c06da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.161+0000 7fd852cdb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd84c0cffd0 0x7fd84c06da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57552/0 (socket says 172.21.15.83:57552) 2024-03-06T07:23:36.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.161+0000 7fd852cdb640 1 -- 172.21.15.83:0/3316739283 learned_addr learned my addr 172.21.15.83:0/3316739283 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:36.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.161+0000 7fd852cdb640 1 -- 172.21.15.83:0/3316739283 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd84c1586e0 msgr2=0x7fd84c0aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:36.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.161+0000 7fd852cdb640 1 --2- 172.21.15.83:0/3316739283 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd84c1586e0 0x7fd84c0aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.161+0000 7fd852cdb640 1 -- 172.21.15.83:0/3316739283 >> v2:172.21.15.183:3300/0 conn(0x7fd84c0aef80 msgr2=0x7fd84c0cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:36.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.161+0000 7fd851cd9640 1 --2- 172.21.15.83:0/3316739283 >> v2:172.21.15.183:3300/0 conn(0x7fd84c0aef80 0x7fd84c0cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.161+0000 7fd852cdb640 1 --2- 172.21.15.83:0/3316739283 >> v2:172.21.15.183:3300/0 conn(0x7fd84c0aef80 0x7fd84c0cfa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.161+0000 7fd852cdb640 1 -- 172.21.15.83:0/3316739283 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd84c15b540 con 0x7fd84c0cffd0 2024-03-06T07:23:36.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.162+0000 7fd852cdb640 1 --2- 172.21.15.83:0/3316739283 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd84c0cffd0 0x7fd84c06da20 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fd84c0d0610 tx=0x7fd84402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2791b5de002fd80f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:36.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.162+0000 7fd8514d8640 1 -- 172.21.15.83:0/3316739283 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd84402ca70 con 0x7fd84c0cffd0 2024-03-06T07:23:36.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.162+0000 7fd8514d8640 1 -- 172.21.15.83:0/3316739283 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd84402cc10 con 0x7fd84c0cffd0 2024-03-06T07:23:36.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.162+0000 7fd8514d8640 1 -- 172.21.15.83:0/3316739283 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8440343f0 con 0x7fd84c0cffd0 2024-03-06T07:23:36.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.162+0000 7fd85473e640 1 -- 172.21.15.83:0/3316739283 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd84c0cffd0 msgr2=0x7fd84c06da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.162+0000 7fd85473e640 1 --2- 172.21.15.83:0/3316739283 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd84c0cffd0 0x7fd84c06da20 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fd84c0d0610 tx=0x7fd84402bed0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.163+0000 7fd85473e640 1 -- 172.21.15.83:0/3316739283 shutdown_connections 2024-03-06T07:23:36.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.163+0000 7fd85473e640 1 --2- 172.21.15.83:0/3316739283 >> v2:172.21.15.183:3300/0 conn(0x7fd84c0aef80 0x7fd84c0cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.163+0000 7fd85473e640 1 --2- 172.21.15.83:0/3316739283 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd84c0cffd0 0x7fd84c06da20 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.163+0000 7fd85473e640 1 --2- 172.21.15.83:0/3316739283 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd84c1586e0 0x7fd84c0aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.163+0000 7fd85473e640 1 -- 172.21.15.83:0/3316739283 >> 172.21.15.83:0/3316739283 conn(0x7fd84c06b190 msgr2=0x7fd84c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:36.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.163+0000 7fd85473e640 1 -- 172.21.15.83:0/3316739283 shutdown_connections 2024-03-06T07:23:36.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.163+0000 7fd85473e640 1 -- 172.21.15.83:0/3316739283 wait complete. 2024-03-06T07:23:36.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.164+0000 7fd85473e640 1 Processor -- start 2024-03-06T07:23:36.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.164+0000 7fd85473e640 1 -- start start 2024-03-06T07:23:36.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.164+0000 7fd85473e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd84c0aef80 0x7fd84c145490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.164+0000 7fd85473e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd84c0cffd0 0x7fd84c148720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.164+0000 7fd85473e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd84c1586e0 0x7fd84c06c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.164+0000 7fd85473e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd84c15bde0 con 0x7fd84c1586e0 2024-03-06T07:23:36.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.164+0000 7fd85473e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd84c15bb00 con 0x7fd84c0aef80 2024-03-06T07:23:36.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.164+0000 7fd85473e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd84c071350 con 0x7fd84c0cffd0 2024-03-06T07:23:36.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.164+0000 7fd852cdb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd84c1586e0 0x7fd84c06c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.164+0000 7fd852cdb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd84c1586e0 0x7fd84c06c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57556/0 (socket says 172.21.15.83:57556) 2024-03-06T07:23:36.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.164+0000 7fd852cdb640 1 -- 172.21.15.83:0/4057154895 learned_addr learned my addr 172.21.15.83:0/4057154895 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:36.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.165+0000 7fd851cd9640 1 --2- 172.21.15.83:0/4057154895 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd84c0cffd0 0x7fd84c148720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.165+0000 7fd8524da640 1 --2- 172.21.15.83:0/4057154895 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd84c0aef80 0x7fd84c145490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.165+0000 7fd852cdb640 1 -- 172.21.15.83:0/4057154895 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd84c0cffd0 msgr2=0x7fd84c148720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.165+0000 7fd852cdb640 1 --2- 172.21.15.83:0/4057154895 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd84c0cffd0 0x7fd84c148720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.165+0000 7fd852cdb640 1 -- 172.21.15.83:0/4057154895 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd84c0aef80 msgr2=0x7fd84c145490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.165+0000 7fd852cdb640 1 --2- 172.21.15.83:0/4057154895 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd84c0aef80 0x7fd84c145490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.165+0000 7fd852cdb640 1 -- 172.21.15.83:0/4057154895 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd84c1459d0 con 0x7fd84c1586e0 2024-03-06T07:23:36.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.165+0000 7fd851cd9640 1 --2- 172.21.15.83:0/4057154895 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd84c0cffd0 0x7fd84c148720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:36.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.165+0000 7fd852cdb640 1 --2- 172.21.15.83:0/4057154895 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd84c1586e0 0x7fd84c06c910 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fd844036040 tx=0x7fd844004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:36.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.165+0000 7fd842ffd640 1 -- 172.21.15.83:0/4057154895 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd844043020 con 0x7fd84c1586e0 2024-03-06T07:23:36.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.165+0000 7fd842ffd640 1 -- 172.21.15.83:0/4057154895 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd84403e450 con 0x7fd84c1586e0 2024-03-06T07:23:36.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.165+0000 7fd85473e640 1 -- 172.21.15.83:0/4057154895 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd84c0cde70 con 0x7fd84c1586e0 2024-03-06T07:23:36.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.165+0000 7fd842ffd640 1 -- 172.21.15.83:0/4057154895 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd84403e820 con 0x7fd84c1586e0 2024-03-06T07:23:36.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.165+0000 7fd85473e640 1 -- 172.21.15.83:0/4057154895 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd84c0ce430 con 0x7fd84c1586e0 2024-03-06T07:23:36.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.166+0000 7fd85473e640 1 -- 172.21.15.83:0/4057154895 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd84c0732b0 con 0x7fd84c1586e0 2024-03-06T07:23:36.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.166+0000 7fd842ffd640 1 -- 172.21.15.83:0/4057154895 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd844046020 con 0x7fd84c1586e0 2024-03-06T07:23:36.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.167+0000 7fd842ffd640 1 --2- 172.21.15.83:0/4057154895 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd83006b4d0 0x7fd83006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.167+0000 7fd842ffd640 1 -- 172.21.15.83:0/4057154895 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd8440acaa0 con 0x7fd84c1586e0 2024-03-06T07:23:36.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.167+0000 7fd8524da640 1 --2- 172.21.15.83:0/4057154895 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd83006b4d0 0x7fd83006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.167+0000 7fd8524da640 1 --2- 172.21.15.83:0/4057154895 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd83006b4d0 0x7fd83006d990 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd8480098e0 tx=0x7fd848009060 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:36.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.171+0000 7fd842ffd640 1 -- 172.21.15.83:0/4057154895 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd84404d020 con 0x7fd84c1586e0 2024-03-06T07:23:36.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.341+0000 7fd85473e640 1 -- 172.21.15.83:0/4057154895 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fd84c148c60 con 0x7fd84c1586e0 2024-03-06T07:23:36.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.344+0000 7fd842ffd640 1 -- 172.21.15.83:0/4057154895 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd84407bc20 con 0x7fd84c1586e0 2024-03-06T07:23:36.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.346+0000 7fd85473e640 1 -- 172.21.15.83:0/4057154895 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd83006b4d0 msgr2=0x7fd83006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.346+0000 7fd85473e640 1 --2- 172.21.15.83:0/4057154895 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd83006b4d0 0x7fd83006d990 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd8480098e0 tx=0x7fd848009060 comp rx=0 tx=0).stop 2024-03-06T07:23:36.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.346+0000 7fd85473e640 1 -- 172.21.15.83:0/4057154895 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd84c1586e0 msgr2=0x7fd84c06c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.346+0000 7fd85473e640 1 --2- 172.21.15.83:0/4057154895 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd84c1586e0 0x7fd84c06c910 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fd844036040 tx=0x7fd844004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:36.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.346+0000 7fd85473e640 1 -- 172.21.15.83:0/4057154895 shutdown_connections 2024-03-06T07:23:36.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.346+0000 7fd85473e640 1 --2- 172.21.15.83:0/4057154895 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd83006b4d0 0x7fd83006d990 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.346+0000 7fd85473e640 1 --2- 172.21.15.83:0/4057154895 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd84c1586e0 0x7fd84c06c910 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.346+0000 7fd85473e640 1 --2- 172.21.15.83:0/4057154895 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd84c0cffd0 0x7fd84c148720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.346+0000 7fd85473e640 1 --2- 172.21.15.83:0/4057154895 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd84c0aef80 0x7fd84c145490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.346+0000 7fd85473e640 1 -- 172.21.15.83:0/4057154895 >> 172.21.15.83:0/4057154895 conn(0x7fd84c06b190 msgr2=0x7fd84c0cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:36.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.346+0000 7fd85473e640 1 -- 172.21.15.83:0/4057154895 shutdown_connections 2024-03-06T07:23:36.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.346+0000 7fd85473e640 1 -- 172.21.15.83:0/4057154895 wait complete. 2024-03-06T07:23:36.359 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:23:36.359 DEBUG:teuthology.orchestra.run.smithi083:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:36.397 DEBUG:teuthology.orchestra.run.smithi183:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:36.437 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:23:36.437 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:23:36.437 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:36.453 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1257 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:23:36.453 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:23:36.453 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:36.493 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=287 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:23:36.493 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:23:36.493 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:36.508 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1257 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:23:36.508 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:23:36.508 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:36.548 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=287 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:23:36.549 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:23:36.549 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:36.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.646+0000 7fafb8f3e640 1 Processor -- start 2024-03-06T07:23:36.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.646+0000 7fafb8f3e640 1 -- start start 2024-03-06T07:23:36.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.647+0000 7fafb8f3e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fafb413dc10 0x7fafb40b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.647+0000 7fafb8f3e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafb415f580 con 0x7fafb415bf30 2024-03-06T07:23:36.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.647+0000 7fafb8f3e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafb415b450 con 0x7fafb415ef60 2024-03-06T07:23:36.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.647+0000 7fafb8f3e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafb40cf3f0 con 0x7fafb413dc10 2024-03-06T07:23:36.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.647+0000 7fafb1d9b640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fafb415bf30 0x7fafb415e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:42948/0 (socket says 172.21.15.83:42948) 2024-03-06T07:23:36.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.647+0000 7fafb1d9b640 1 -- 172.21.15.83:0/2298851934 learned_addr learned my addr 172.21.15.83:0/2298851934 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:23:36.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.648+0000 7fafb259c640 1 --2- 172.21.15.83:0/2298851934 >> v2:172.21.15.183:3300/0 conn(0x7fafb413dc10 0x7fafb40b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.648+0000 7fafb159a640 1 -- 172.21.15.83:0/2298851934 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1964555070 0 0) 0x7fafb415f580 con 0x7fafb415bf30 2024-03-06T07:23:36.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.648+0000 7fafb159a640 1 -- 172.21.15.83:0/2298851934 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf9c003520 con 0x7fafb415bf30 2024-03-06T07:23:36.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.648+0000 7fafb259c640 1 -- 172.21.15.83:0/2298851934 >> v1:172.21.15.183:6789/0 conn(0x7fafb415ef60 legacy=0x7fafb40708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.648+0000 7fafb259c640 1 -- 172.21.15.83:0/2298851934 >> v1:172.21.15.83:6789/0 conn(0x7fafb415bf30 legacy=0x7fafb415e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.648+0000 7fafb259c640 1 -- 172.21.15.83:0/2298851934 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb415b730 con 0x7fafb413dc10 2024-03-06T07:23:36.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.649+0000 7fafb259c640 1 --2- 172.21.15.83:0/2298851934 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fafb413dc10 0x7fafb40b2950 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fafa4009d90 tx=0x7fafa402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8158889db182663f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:36.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.649+0000 7fafb159a640 1 -- 172.21.15.83:0/2298851934 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa4039070 con 0x7fafb413dc10 2024-03-06T07:23:36.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.649+0000 7fafb159a640 1 -- 172.21.15.83:0/2298851934 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafa402cc60 con 0x7fafb413dc10 2024-03-06T07:23:36.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.649+0000 7fafb159a640 1 -- 172.21.15.83:0/2298851934 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa4034640 con 0x7fafb413dc10 2024-03-06T07:23:36.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.649+0000 7fafb8f3e640 1 -- 172.21.15.83:0/2298851934 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fafb413dc10 msgr2=0x7fafb40b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.649+0000 7fafb8f3e640 1 --2- 172.21.15.83:0/2298851934 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fafb413dc10 0x7fafb40b2950 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fafa4009d90 tx=0x7fafa402beb0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.649+0000 7fafb8f3e640 1 -- 172.21.15.83:0/2298851934 shutdown_connections 2024-03-06T07:23:36.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.649+0000 7fafb8f3e640 1 --2- 172.21.15.83:0/2298851934 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fafb413dc10 0x7fafb40b2950 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.649+0000 7fafb8f3e640 1 -- 172.21.15.83:0/2298851934 >> 172.21.15.83:0/2298851934 conn(0x7fafb40cf8a0 msgr2=0x7fafb40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:36.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.649+0000 7fafb8f3e640 1 -- 172.21.15.83:0/2298851934 shutdown_connections 2024-03-06T07:23:36.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.649+0000 7fafb8f3e640 1 -- 172.21.15.83:0/2298851934 wait complete. 2024-03-06T07:23:36.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.651+0000 7fafb8f3e640 1 Processor -- start 2024-03-06T07:23:36.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.651+0000 7fafb8f3e640 1 -- start start 2024-03-06T07:23:36.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.651+0000 7fafb8f3e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fafb413dc10 0x7fafb413bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.651+0000 7fafb8f3e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fafb415bf30 0x7fafb4158700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.651+0000 7fafb8f3e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fafb415ef60 0x7fafb413c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.651+0000 7fafb8f3e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb40b2e90 con 0x7fafb413dc10 2024-03-06T07:23:36.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.651+0000 7fafb8f3e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb415b210 con 0x7fafb415bf30 2024-03-06T07:23:36.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.651+0000 7fafb8f3e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb40b3010 con 0x7fafb415ef60 2024-03-06T07:23:36.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.652+0000 7fafb259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fafb413dc10 0x7fafb413bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.652+0000 7fafb259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fafb413dc10 0x7fafb413bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48122/0 (socket says 172.21.15.83:48122) 2024-03-06T07:23:36.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.652+0000 7fafb259c640 1 -- 172.21.15.83:0/3774276856 learned_addr learned my addr 172.21.15.83:0/3774276856 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:36.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.652+0000 7fafb1d9b640 1 --2- 172.21.15.83:0/3774276856 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fafb415bf30 0x7fafb4158700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.652+0000 7fafb2d9d640 1 --2- 172.21.15.83:0/3774276856 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fafb415ef60 0x7fafb413c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.652+0000 7fafb259c640 1 -- 172.21.15.83:0/3774276856 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fafb415ef60 msgr2=0x7fafb413c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.652+0000 7fafb259c640 1 --2- 172.21.15.83:0/3774276856 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fafb415ef60 0x7fafb413c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.652+0000 7fafb259c640 1 -- 172.21.15.83:0/3774276856 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fafb415bf30 msgr2=0x7fafb4158700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.652+0000 7fafb259c640 1 --2- 172.21.15.83:0/3774276856 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fafb415bf30 0x7fafb4158700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.652+0000 7fafb259c640 1 -- 172.21.15.83:0/3774276856 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb4156a40 con 0x7fafb413dc10 2024-03-06T07:23:36.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.652+0000 7fafb2d9d640 1 --2- 172.21.15.83:0/3774276856 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fafb415ef60 0x7fafb413c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:36.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.652+0000 7fafb259c640 1 --2- 172.21.15.83:0/3774276856 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fafb413dc10 0x7fafb413bbf0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fafa4037040 tx=0x7fafa4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:36.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.652+0000 7fafa2ffd640 1 -- 172.21.15.83:0/3774276856 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa4039050 con 0x7fafb413dc10 2024-03-06T07:23:36.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.652+0000 7fafa2ffd640 1 -- 172.21.15.83:0/3774276856 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafa403a040 con 0x7fafb413dc10 2024-03-06T07:23:36.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.652+0000 7fafb8f3e640 1 -- 172.21.15.83:0/3774276856 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafb4156d20 con 0x7fafb413dc10 2024-03-06T07:23:36.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.652+0000 7fafa2ffd640 1 -- 172.21.15.83:0/3774276856 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa4009450 con 0x7fafb413dc10 2024-03-06T07:23:36.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.652+0000 7fafb8f3e640 1 -- 172.21.15.83:0/3774276856 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafb41f1fe0 con 0x7fafb413dc10 2024-03-06T07:23:36.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.653+0000 7fafb8f3e640 1 -- 172.21.15.83:0/3774276856 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafb41f21e0 con 0x7fafb413dc10 2024-03-06T07:23:36.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.654+0000 7fafa2ffd640 1 -- 172.21.15.83:0/3774276856 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fafa4047020 con 0x7fafb413dc10 2024-03-06T07:23:36.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.654+0000 7fafa2ffd640 1 --2- 172.21.15.83:0/3774276856 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7faf9406b4d0 0x7faf9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.654+0000 7fafa2ffd640 1 -- 172.21.15.83:0/3774276856 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fafa40ac170 con 0x7fafb413dc10 2024-03-06T07:23:36.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.654+0000 7fafb1d9b640 1 --2- 172.21.15.83:0/3774276856 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7faf9406b4d0 0x7faf9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.654+0000 7fafb1d9b640 1 --2- 172.21.15.83:0/3774276856 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7faf9406b4d0 0x7faf9406d990 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fafb40b2c00 tx=0x7fafa8007d30 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:36.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.658+0000 7fafa2ffd640 1 -- 172.21.15.83:0/3774276856 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fafb41f21e0 con 0x7fafb413dc10 2024-03-06T07:23:36.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.862+0000 7fafb8f3e640 1 -- 172.21.15.83:0/3774276856 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fafb406eb00 con 0x7fafb413dc10 2024-03-06T07:23:36.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.863+0000 7fafa2ffd640 1 -- 172.21.15.83:0/3774276856 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+3704 (secure 0 0 0) 0x7fafa404e020 con 0x7fafb413dc10 2024-03-06T07:23:36.865 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:23:36.865 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":15,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4799,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/3542419057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3542419057},{"type":"v1","addr":"172.21.15.83:6836","nonce":3542419057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":5023,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1853869906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":1853869906},{"type":"v1","addr":"172.21.15.183:6835","nonce":1853869906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5026,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/3666667479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3666667479},{"type":"v1","addr":"172.21.15.183:6834","nonce":3666667479}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1704606453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1704606453},{"type":"v1","addr":"172.21.15.83:6837","nonce":1704606453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":15}],"filesystems":[]} 2024-03-06T07:23:36.865 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 15 2024-03-06T07:23:36.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.865+0000 7fafb8f3e640 1 -- 172.21.15.83:0/3774276856 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7faf9406b4d0 msgr2=0x7faf9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.865+0000 7fafb8f3e640 1 --2- 172.21.15.83:0/3774276856 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7faf9406b4d0 0x7faf9406d990 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fafb40b2c00 tx=0x7fafa8007d30 comp rx=0 tx=0).stop 2024-03-06T07:23:36.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.865+0000 7fafb8f3e640 1 -- 172.21.15.83:0/3774276856 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fafb413dc10 msgr2=0x7fafb413bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.865+0000 7fafb8f3e640 1 --2- 172.21.15.83:0/3774276856 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fafb413dc10 0x7fafb413bbf0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fafa4037040 tx=0x7fafa4004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:36.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.865+0000 7fafb8f3e640 1 -- 172.21.15.83:0/3774276856 shutdown_connections 2024-03-06T07:23:36.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.865+0000 7fafb8f3e640 1 --2- 172.21.15.83:0/3774276856 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7faf9406b4d0 0x7faf9406d990 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.865+0000 7fafb8f3e640 1 --2- 172.21.15.83:0/3774276856 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fafb415ef60 0x7fafb413c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.865+0000 7fafb8f3e640 1 --2- 172.21.15.83:0/3774276856 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fafb415bf30 0x7fafb4158700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.865+0000 7fafb8f3e640 1 --2- 172.21.15.83:0/3774276856 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fafb413dc10 0x7fafb413bbf0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.865+0000 7fafb8f3e640 1 -- 172.21.15.83:0/3774276856 >> 172.21.15.83:0/3774276856 conn(0x7fafb40cf8a0 msgr2=0x7fafb415cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:36.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.865+0000 7fafb8f3e640 1 -- 172.21.15.83:0/3774276856 shutdown_connections 2024-03-06T07:23:36.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:36.865+0000 7fafb8f3e640 1 -- 172.21.15.83:0/3774276856 wait complete. 2024-03-06T07:23:36.878 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:23:36.878 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:23:36.878 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:23:36.878 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:36.879 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:23:36.879 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:23:36.879 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:23:36.879 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:36.879 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:23:36.879 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:23:36.879 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:23:36.880 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:36.880 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:23:36.880 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:23:36.880 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:23:36.880 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:36.889 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:23:36.889 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:23:36.890 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:23:36.890 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:23:36.891 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:23:36.891 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:23:36.892 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:23:36.892 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:23:36.931 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:23:36.931 DEBUG:teuthology.parallel:result is None 2024-03-06T07:23:36.932 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:23:36.932 DEBUG:teuthology.parallel:result is None 2024-03-06T07:23:36.936 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:23:36.936 DEBUG:teuthology.parallel:result is None 2024-03-06T07:23:36.936 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:23:36.936 DEBUG:teuthology.parallel:result is None 2024-03-06T07:23:36.937 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:23:37.008 INFO:tasks.ceph.mds.a.smithi083.stdout:starting mds.a at 2024-03-06T07:23:37.008 INFO:tasks.ceph.mds.c.smithi083.stdout:starting mds.c at 2024-03-06T07:23:37.011 INFO:tasks.ceph.mds.b.smithi183.stdout:starting mds.b at 2024-03-06T07:23:37.011 INFO:tasks.ceph.mds.d.smithi183.stdout:starting mds.d at 2024-03-06T07:23:37.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.042+0000 7f209d319640 1 Processor -- start 2024-03-06T07:23:37.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.042+0000 7f209d319640 1 -- start start 2024-03-06T07:23:37.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.043+0000 7f209d319640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f209815e000 0x7f20980d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.043+0000 7f209d319640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f20980cf7d0 0x7f20980d0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.043+0000 7f209d319640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20980d25e0 con 0x7f20980d4810 2024-03-06T07:23:37.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.043+0000 7f209d319640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20980c7c50 con 0x7f20980cf7d0 2024-03-06T07:23:37.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.043+0000 7f209d319640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20980d2880 con 0x7f209815e000 2024-03-06T07:23:37.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.043+0000 7f2096ffd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f209815e000 0x7f20980d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.043+0000 7f20977fe640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f20980cf7d0 0x7f20980d0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.043+0000 7f2096ffd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f209815e000 0x7f20980d0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:59916/0 (socket says 172.21.15.83:59916) 2024-03-06T07:23:37.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.043+0000 7f2096ffd640 1 -- 172.21.15.83:0/3254189258 learned_addr learned my addr 172.21.15.83:0/3254189258 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:37.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.043+0000 7f20967fc640 1 --1- 172.21.15.83:0/3254189258 >> v1:172.21.15.183:6789/0 conn(0x7f20980d4810 0x7f20980c80e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:42144/0 (socket says 172.21.15.83:42144) 2024-03-06T07:23:37.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.043+0000 7f2096ffd640 1 -- 172.21.15.83:0/3254189258 >> v2:172.21.15.183:3300/0 conn(0x7f20980cf7d0 msgr2=0x7f20980d0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:37.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.043+0000 7f2096ffd640 1 --2- 172.21.15.83:0/3254189258 >> v2:172.21.15.183:3300/0 conn(0x7f20980cf7d0 0x7f20980d0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.043+0000 7f2096ffd640 1 -- 172.21.15.83:0/3254189258 >> v1:172.21.15.183:6789/0 conn(0x7f20980d4810 legacy=0x7f20980c80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.043+0000 7f2096ffd640 1 -- 172.21.15.83:0/3254189258 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20980d2b40 con 0x7f209815e000 2024-03-06T07:23:37.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.044+0000 7f2096ffd640 1 --2- 172.21.15.83:0/3254189258 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f209815e000 0x7f20980d0680 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f208c009870 tx=0x7f208c02ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f2e21f9deb9845f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:37.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.046+0000 7f2095ffb640 1 -- 172.21.15.83:0/3254189258 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f208c02f840 con 0x7f209815e000 2024-03-06T07:23:37.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.046+0000 7f2095ffb640 1 -- 172.21.15.83:0/3254189258 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f208c02f9e0 con 0x7f209815e000 2024-03-06T07:23:37.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.046+0000 7f2095ffb640 1 -- 172.21.15.83:0/3254189258 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f208c02fdb0 con 0x7f209815e000 2024-03-06T07:23:37.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.047+0000 7f209d319640 1 -- 172.21.15.83:0/3254189258 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f209815e000 msgr2=0x7f20980d0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.047+0000 7f209d319640 1 --2- 172.21.15.83:0/3254189258 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f209815e000 0x7f20980d0680 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f208c009870 tx=0x7f208c02ba20 comp rx=0 tx=0).stop 2024-03-06T07:23:37.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.047+0000 7f209d319640 1 -- 172.21.15.83:0/3254189258 shutdown_connections 2024-03-06T07:23:37.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.047+0000 7f209d319640 1 --2- 172.21.15.83:0/3254189258 >> v2:172.21.15.183:3300/0 conn(0x7f20980cf7d0 0x7f20980d0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.047+0000 7f209d319640 1 --2- 172.21.15.83:0/3254189258 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f209815e000 0x7f20980d0680 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.047+0000 7f209d319640 1 -- 172.21.15.83:0/3254189258 >> 172.21.15.83:0/3254189258 conn(0x7f20980d3cf0 msgr2=0x7f20980d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:37.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.047+0000 7f209d319640 1 -- 172.21.15.83:0/3254189258 shutdown_connections 2024-03-06T07:23:37.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.047+0000 7f209d319640 1 -- 172.21.15.83:0/3254189258 wait complete. 2024-03-06T07:23:37.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.048+0000 7f209d319640 1 Processor -- start 2024-03-06T07:23:37.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.049+0000 7f209d319640 1 -- start start 2024-03-06T07:23:37.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.049+0000 7f209d319640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f20980cf7d0 0x7f209809e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.049+0000 7f209d319640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f20980d4810 0x7f20981f4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.049+0000 7f209d319640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f209815e000 0x7f209815cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.049+0000 7f209d319640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f209815e3e0 con 0x7f20980d4810 2024-03-06T07:23:37.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.049+0000 7f209d319640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20980d22d0 con 0x7f209815e000 2024-03-06T07:23:37.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.049+0000 7f209d319640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20980c87f0 con 0x7f20980cf7d0 2024-03-06T07:23:37.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.049+0000 7f2096ffd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f20980cf7d0 0x7f209809e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.049+0000 7f20977fe640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f209815e000 0x7f209815cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.049+0000 7f2096ffd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f20980cf7d0 0x7f209809e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:59928/0 (socket says 172.21.15.83:59928) 2024-03-06T07:23:37.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.049+0000 7f2096ffd640 1 -- 172.21.15.83:0/864824058 learned_addr learned my addr 172.21.15.83:0/864824058 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:37.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.049+0000 7f20967fc640 1 --2- 172.21.15.83:0/864824058 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f20980d4810 0x7f20981f4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.050+0000 7f20977fe640 1 -- 172.21.15.83:0/864824058 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f20980cf7d0 msgr2=0x7f209809e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.050+0000 7f20977fe640 1 --2- 172.21.15.83:0/864824058 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f20980cf7d0 0x7f209809e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.050+0000 7f20977fe640 1 -- 172.21.15.83:0/864824058 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f20980d4810 msgr2=0x7f20981f4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.050+0000 7f20977fe640 1 --2- 172.21.15.83:0/864824058 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f20980d4810 0x7f20981f4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.050+0000 7f20977fe640 1 -- 172.21.15.83:0/864824058 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f209815d0e0 con 0x7f209815e000 2024-03-06T07:23:37.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.050+0000 7f2096ffd640 1 --2- 172.21.15.83:0/864824058 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f20980cf7d0 0x7f209809e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:37.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.050+0000 7f20977fe640 1 --2- 172.21.15.83:0/864824058 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f209815e000 0x7f209815cba0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f209000efc0 tx=0x7f209000c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:37.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.050+0000 7f207f7fe640 1 -- 172.21.15.83:0/864824058 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2090018020 con 0x7f209815e000 2024-03-06T07:23:37.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.050+0000 7f209d319640 1 -- 172.21.15.83:0/864824058 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20980a0080 con 0x7f209815e000 2024-03-06T07:23:37.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.050+0000 7f207f7fe640 1 -- 172.21.15.83:0/864824058 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2090014480 con 0x7f209815e000 2024-03-06T07:23:37.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.050+0000 7f207f7fe640 1 -- 172.21.15.83:0/864824058 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2090014870 con 0x7f209815e000 2024-03-06T07:23:37.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.050+0000 7f209d319640 1 -- 172.21.15.83:0/864824058 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20980a0640 con 0x7f209815e000 2024-03-06T07:23:37.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.052+0000 7f209d319640 1 -- 172.21.15.83:0/864824058 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f209804f330 con 0x7f209815e000 2024-03-06T07:23:37.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.052+0000 7f207f7fe640 1 -- 172.21.15.83:0/864824058 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2090005d10 con 0x7f209815e000 2024-03-06T07:23:37.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.052+0000 7f207f7fe640 1 --2- 172.21.15.83:0/864824058 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f208406b4d0 0x7f208406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.052+0000 7f2096ffd640 1 --2- 172.21.15.83:0/864824058 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f208406b4d0 0x7f208406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.053+0000 7f207f7fe640 1 -- 172.21.15.83:0/864824058 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f20900895b0 con 0x7f209815e000 2024-03-06T07:23:37.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.053+0000 7f2096ffd640 1 --2- 172.21.15.83:0/864824058 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f208406b4d0 0x7f208406d990 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f208c009870 tx=0x7f208c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:37.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.058+0000 7f207f7fe640 1 -- 172.21.15.83:0/864824058 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2090022030 con 0x7f209815e000 2024-03-06T07:23:37.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:37.241+0000 7f209d319640 1 -- 172.21.15.83:0/864824058 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f20980d0180 con 0x7f209815e000 2024-03-06T07:23:38.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.066+0000 7f207f7fe640 1 -- 172.21.15.83:0/864824058 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v41) v1 ==== 97+0+0 (secure 0 0 0) 0x7f2090010040 con 0x7f209815e000 2024-03-06T07:23:38.068 INFO:teuthology.orchestra.run.smithi083.stderr: removed all blocklist entries 2024-03-06T07:23:38.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.069+0000 7f209d319640 1 -- 172.21.15.83:0/864824058 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f208406b4d0 msgr2=0x7f208406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.069+0000 7f209d319640 1 --2- 172.21.15.83:0/864824058 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f208406b4d0 0x7f208406d990 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f208c009870 tx=0x7f208c008030 comp rx=0 tx=0).stop 2024-03-06T07:23:38.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.069+0000 7f209d319640 1 -- 172.21.15.83:0/864824058 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f209815e000 msgr2=0x7f209815cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.069+0000 7f209d319640 1 --2- 172.21.15.83:0/864824058 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f209815e000 0x7f209815cba0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f209000efc0 tx=0x7f209000c490 comp rx=0 tx=0).stop 2024-03-06T07:23:38.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.069+0000 7f209d319640 1 -- 172.21.15.83:0/864824058 shutdown_connections 2024-03-06T07:23:38.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.069+0000 7f209d319640 1 --2- 172.21.15.83:0/864824058 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f208406b4d0 0x7f208406d990 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.069+0000 7f209d319640 1 --2- 172.21.15.83:0/864824058 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f209815e000 0x7f209815cba0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.069+0000 7f209d319640 1 --2- 172.21.15.83:0/864824058 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f20980d4810 0x7f20981f4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.069+0000 7f209d319640 1 --2- 172.21.15.83:0/864824058 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f20980cf7d0 0x7f209809e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.069+0000 7f209d319640 1 -- 172.21.15.83:0/864824058 >> 172.21.15.83:0/864824058 conn(0x7f20980d3cf0 msgr2=0x7f20980d0980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:38.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.069+0000 7f209d319640 1 -- 172.21.15.83:0/864824058 shutdown_connections 2024-03-06T07:23:38.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.069+0000 7f209d319640 1 -- 172.21.15.83:0/864824058 wait complete. 2024-03-06T07:23:38.083 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:23:38.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.186+0000 7f0db99c7640 1 Processor -- start 2024-03-06T07:23:38.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.186+0000 7f0db99c7640 1 -- start start 2024-03-06T07:23:38.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.187+0000 7f0db99c7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0db40b4170 0x7f0db40b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.187+0000 7f0db99c7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db40d25d0 con 0x7f0db40d0a80 2024-03-06T07:23:38.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.187+0000 7f0db99c7640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db40d2870 con 0x7f0db40d4760 2024-03-06T07:23:38.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.187+0000 7f0db99c7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0db40d4bf0 con 0x7f0db40b4170 2024-03-06T07:23:38.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.187+0000 7f0db37fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f0db40d0a80 0x7f0db40cee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:42962/0 (socket says 172.21.15.83:42962) 2024-03-06T07:23:38.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.187+0000 7f0db37fe640 1 -- 172.21.15.83:0/3612478526 learned_addr learned my addr 172.21.15.83:0/3612478526 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:23:38.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.187+0000 7f0db2ffd640 1 --2- 172.21.15.83:0/3612478526 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0db40b4170 0x7f0db40b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.187+0000 7f0db1ffb640 1 -- 172.21.15.83:0/3612478526 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1511270638 0 0) 0x7f0db40d25d0 con 0x7f0db40d0a80 2024-03-06T07:23:38.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.188+0000 7f0db1ffb640 1 -- 172.21.15.83:0/3612478526 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d9c003520 con 0x7f0db40d0a80 2024-03-06T07:23:38.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.188+0000 7f0db1ffb640 1 -- 172.21.15.83:0/3612478526 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2086222751 0 0) 0x7f0db40d2870 con 0x7f0db40d4760 2024-03-06T07:23:38.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.188+0000 7f0db1ffb640 1 -- 172.21.15.83:0/3612478526 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0db40d25d0 con 0x7f0db40d4760 2024-03-06T07:23:38.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.188+0000 7f0db2ffd640 1 -- 172.21.15.83:0/3612478526 >> v1:172.21.15.183:6789/0 conn(0x7f0db40d4760 legacy=0x7f0db40d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.188+0000 7f0db2ffd640 1 -- 172.21.15.83:0/3612478526 >> v1:172.21.15.83:6789/0 conn(0x7f0db40d0a80 legacy=0x7f0db40cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.188+0000 7f0db2ffd640 1 -- 172.21.15.83:0/3612478526 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db40d2b50 con 0x7f0db40b4170 2024-03-06T07:23:38.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.188+0000 7f0db2ffd640 1 --2- 172.21.15.83:0/3612478526 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0db40b4170 0x7f0db40b6a90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0da4009dc0 tx=0x7f0da402bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31727f796254bba4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:38.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.189+0000 7f0db1ffb640 1 -- 172.21.15.83:0/3612478526 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da4039070 con 0x7f0db40b4170 2024-03-06T07:23:38.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.189+0000 7f0db1ffb640 1 -- 172.21.15.83:0/3612478526 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0da402ccd0 con 0x7f0db40b4170 2024-03-06T07:23:38.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.189+0000 7f0db1ffb640 1 -- 172.21.15.83:0/3612478526 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da4034640 con 0x7f0db40b4170 2024-03-06T07:23:38.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.189+0000 7f0db99c7640 1 -- 172.21.15.83:0/3612478526 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0db40b4170 msgr2=0x7f0db40b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.189+0000 7f0db99c7640 1 --2- 172.21.15.83:0/3612478526 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0db40b4170 0x7f0db40b6a90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0da4009dc0 tx=0x7f0da402bf20 comp rx=0 tx=0).stop 2024-03-06T07:23:38.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.189+0000 7f0db99c7640 1 -- 172.21.15.83:0/3612478526 shutdown_connections 2024-03-06T07:23:38.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.189+0000 7f0db99c7640 1 --2- 172.21.15.83:0/3612478526 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0db40b4170 0x7f0db40b6a90 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.189+0000 7f0db99c7640 1 -- 172.21.15.83:0/3612478526 >> 172.21.15.83:0/3612478526 conn(0x7f0db40d3c40 msgr2=0x7f0db40d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:38.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.189+0000 7f0db99c7640 1 -- 172.21.15.83:0/3612478526 shutdown_connections 2024-03-06T07:23:38.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.189+0000 7f0db99c7640 1 -- 172.21.15.83:0/3612478526 wait complete. 2024-03-06T07:23:38.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.190+0000 7f0db99c7640 1 Processor -- start 2024-03-06T07:23:38.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.190+0000 7f0db99c7640 1 -- start start 2024-03-06T07:23:38.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.191+0000 7f0db99c7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0db40b4170 0x7f0db415dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.191+0000 7f0db99c7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0db40d0a80 0x7f0db4156d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.191+0000 7f0db99c7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0db40cb6b0 0x7f0db40cd160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.191+0000 7f0db99c7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db40b4dd0 con 0x7f0db40b4170 2024-03-06T07:23:38.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.191+0000 7f0db99c7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db40ce9e0 con 0x7f0db40d0a80 2024-03-06T07:23:38.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.191+0000 7f0db99c7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0db40b4f50 con 0x7f0db40cb6b0 2024-03-06T07:23:38.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.191+0000 7f0db2ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0db40b4170 0x7f0db415dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.191+0000 7f0db2ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0db40b4170 0x7f0db415dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48166/0 (socket says 172.21.15.83:48166) 2024-03-06T07:23:38.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.191+0000 7f0db2ffd640 1 -- 172.21.15.83:0/2682658429 learned_addr learned my addr 172.21.15.83:0/2682658429 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:38.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.191+0000 7f0db37fe640 1 --2- 172.21.15.83:0/2682658429 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0db40cb6b0 0x7f0db40cd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.191+0000 7f0db27fc640 1 --2- 172.21.15.83:0/2682658429 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0db40d0a80 0x7f0db4156d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.191+0000 7f0db2ffd640 1 -- 172.21.15.83:0/2682658429 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0db40cb6b0 msgr2=0x7f0db40cd160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.191+0000 7f0db2ffd640 1 --2- 172.21.15.83:0/2682658429 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0db40cb6b0 0x7f0db40cd160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.191+0000 7f0db2ffd640 1 -- 172.21.15.83:0/2682658429 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0db40d0a80 msgr2=0x7f0db4156d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.191+0000 7f0db2ffd640 1 --2- 172.21.15.83:0/2682658429 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0db40d0a80 0x7f0db4156d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.191+0000 7f0db2ffd640 1 -- 172.21.15.83:0/2682658429 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db40b82c0 con 0x7f0db40b4170 2024-03-06T07:23:38.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.191+0000 7f0db37fe640 1 --2- 172.21.15.83:0/2682658429 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0db40cb6b0 0x7f0db40cd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:38.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.192+0000 7f0db2ffd640 1 --2- 172.21.15.83:0/2682658429 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0db40b4170 0x7f0db415dfe0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f0da4009ee0 tx=0x7f0da4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:38.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.192+0000 7f0d9bfff640 1 -- 172.21.15.83:0/2682658429 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da4042020 con 0x7f0db40b4170 2024-03-06T07:23:38.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.192+0000 7f0d9bfff640 1 -- 172.21.15.83:0/2682658429 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0da403a040 con 0x7f0db40b4170 2024-03-06T07:23:38.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.192+0000 7f0db99c7640 1 -- 172.21.15.83:0/2682658429 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0db40b85a0 con 0x7f0db40b4170 2024-03-06T07:23:38.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.192+0000 7f0d9bfff640 1 -- 172.21.15.83:0/2682658429 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da4039070 con 0x7f0db40b4170 2024-03-06T07:23:38.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.192+0000 7f0db99c7640 1 -- 172.21.15.83:0/2682658429 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0db40b8b60 con 0x7f0db40b4170 2024-03-06T07:23:38.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.193+0000 7f0db99c7640 1 -- 172.21.15.83:0/2682658429 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0db404f330 con 0x7f0db40b4170 2024-03-06T07:23:38.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.193+0000 7f0d9bfff640 1 -- 172.21.15.83:0/2682658429 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0da4045020 con 0x7f0db40b4170 2024-03-06T07:23:38.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.194+0000 7f0d9bfff640 1 --2- 172.21.15.83:0/2682658429 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0da006b500 0x7f0da006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.194+0000 7f0db27fc640 1 --2- 172.21.15.83:0/2682658429 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0da006b500 0x7f0da006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.194+0000 7f0d9bfff640 1 -- 172.21.15.83:0/2682658429 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0da40ac140 con 0x7f0db40b4170 2024-03-06T07:23:38.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.194+0000 7f0db27fc640 1 --2- 172.21.15.83:0/2682658429 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0da006b500 0x7f0da006d9c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f0db40ccf50 tx=0x7f0da8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:38.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.198+0000 7f0d9bfff640 1 -- 172.21.15.83:0/2682658429 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0da404c030 con 0x7f0db40b4170 2024-03-06T07:23:38.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.435+0000 7f0db99c7640 1 -- 172.21.15.83:0/2682658429 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f0db40b8f80 con 0x7f0db40b4170 2024-03-06T07:23:38.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.436+0000 7f0d9bfff640 1 -- 172.21.15.83:0/2682658429 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v11) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f0da407b320 con 0x7f0db40b4170 2024-03-06T07:23:38.437 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:23:38.438 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:23:38.438 INFO:teuthology.orchestra.run.smithi083.stdout: "auth_dump": [ 2024-03-06T07:23:38.438 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:23:38.438 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mds.a", 2024-03-06T07:23:38.438 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlVHaUGhAAVRpLsHYTUkaYDmceq5Rlgw==", 2024-03-06T07:23:38.438 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:23:38.438 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:23:38.438 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow *", 2024-03-06T07:23:38.438 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow *", 2024-03-06T07:23:38.438 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:23:38.438 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:23:38.438 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:23:38.438 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:23:38.439 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mds.b", 2024-03-06T07:23:38.439 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlFlrrJhAAQbY6xU910eYRg1zwgpj9vg==", 2024-03-06T07:23:38.439 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:23:38.439 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:23:38.439 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow *", 2024-03-06T07:23:38.439 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow *", 2024-03-06T07:23:38.439 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:23:38.439 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:23:38.439 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:23:38.439 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:23:38.439 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mds.c", 2024-03-06T07:23:38.439 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlX0jxIRAAxYNi39qxIFXazlL2qNzLCg==", 2024-03-06T07:23:38.439 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:23:38.439 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:23:38.440 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow *", 2024-03-06T07:23:38.440 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow *", 2024-03-06T07:23:38.440 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:23:38.440 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:23:38.440 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:23:38.440 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:23:38.440 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mds.d", 2024-03-06T07:23:38.440 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehl3utCLhAAzEjASLnYpPYrJsWOt3WwpQ==", 2024-03-06T07:23:38.440 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:23:38.440 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:23:38.440 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow *", 2024-03-06T07:23:38.440 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow *", 2024-03-06T07:23:38.440 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:23:38.441 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:23:38.441 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:23:38.441 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:23:38.441 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.0", 2024-03-06T07:23:38.441 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBgGehlkezBExAAZOLMH2ua3CTWAL7H6LGeZw==", 2024-03-06T07:23:38.441 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:23:38.441 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:38.441 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:23:38.441 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:23:38.441 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:23:38.441 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:23:38.441 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:23:38.441 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.1", 2024-03-06T07:23:38.441 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBjGehlC04qEhAAqirTFR6y3NOGw6N+iz3/pg==", 2024-03-06T07:23:38.442 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:23:38.442 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:38.442 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:23:38.442 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:23:38.442 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:23:38.442 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:23:38.442 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:23:38.442 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.2", 2024-03-06T07:23:38.442 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBmGehlfuFWEBAAj/vjZXrv1wVNSJ1R74UmnQ==", 2024-03-06T07:23:38.442 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:23:38.442 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:38.442 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:23:38.442 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:23:38.442 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:23:38.443 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:23:38.443 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:23:38.443 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.3", 2024-03-06T07:23:38.443 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBpGehl5+ItDhAA0YNc83MhKcgb3JgRuP87aw==", 2024-03-06T07:23:38.443 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:23:38.443 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:38.443 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:23:38.443 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:23:38.443 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:23:38.443 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:23:38.443 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:23:38.443 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.4", 2024-03-06T07:23:38.443 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBvGehlYa/TFBAAvGcizpRQ21pvgpEGgSET0w==", 2024-03-06T07:23:38.443 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:23:38.444 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:38.444 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:23:38.444 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:23:38.444 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:23:38.444 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:23:38.444 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:23:38.444 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.5", 2024-03-06T07:23:38.444 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQByGehlE80TExAAFcWDgnLtW8OKlyu4B0jnmg==", 2024-03-06T07:23:38.444 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:23:38.444 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:38.444 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:23:38.444 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:23:38.444 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:23:38.444 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:23:38.445 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:23:38.445 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.6", 2024-03-06T07:23:38.445 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQB1Gehla/LbERAAYfAjTKL11VMVnGUGV5GVcg==", 2024-03-06T07:23:38.445 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:23:38.445 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:38.445 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:23:38.445 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:23:38.445 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:23:38.445 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:23:38.445 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:23:38.445 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.7", 2024-03-06T07:23:38.445 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQB4GehlpSJkEBAA17tUgMf94jfqvXlD51P8DQ==", 2024-03-06T07:23:38.445 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:23:38.445 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:38.446 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:23:38.446 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:23:38.446 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:23:38.446 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:23:38.446 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:23:38.446 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.0", 2024-03-06T07:23:38.446 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlRwCIMhAA1DdbdQZPplu7LQOTK3BmSg==", 2024-03-06T07:23:38.446 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:23:38.446 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:23:38.446 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow r", 2024-03-06T07:23:38.446 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:23:38.446 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:23:38.446 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:23:38.447 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:23:38.447 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.1", 2024-03-06T07:23:38.447 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBdGehl77spABAAsiEf+tXos3MwFrXToEQ8dQ==", 2024-03-06T07:23:38.447 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:23:38.447 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:23:38.447 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow r", 2024-03-06T07:23:38.447 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow rw", 2024-03-06T07:23:38.447 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow rwx" 2024-03-06T07:23:38.447 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:23:38.447 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:23:38.447 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:23:38.447 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.2", 2024-03-06T07:23:38.447 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehltY5SOBAAI8rcIEVqMZaWQGgNaYKJTA==", 2024-03-06T07:23:38.447 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:23:38.447 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:23:38.448 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow r", 2024-03-06T07:23:38.448 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow rw", 2024-03-06T07:23:38.448 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow rwx" 2024-03-06T07:23:38.448 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:23:38.448 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:23:38.448 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:23:38.448 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.3", 2024-03-06T07:23:38.448 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBdGehl9ZX1BRAAA7fKnRyaQ+zqHoSGs4UvDw==", 2024-03-06T07:23:38.448 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:23:38.448 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:23:38.448 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow r", 2024-03-06T07:23:38.448 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow rw", 2024-03-06T07:23:38.448 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow rwx" 2024-03-06T07:23:38.449 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:23:38.449 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:23:38.449 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:23:38.449 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.admin", 2024-03-06T07:23:38.449 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBbGehl4gqRORAAobqSWoeyHYA/lXkp6/EcIQ==", 2024-03-06T07:23:38.449 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:23:38.449 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow *", 2024-03-06T07:23:38.449 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow *", 2024-03-06T07:23:38.449 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow *", 2024-03-06T07:23:38.449 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:23:38.449 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:23:38.449 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:23:38.449 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:23:38.449 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mgr.x", 2024-03-06T07:23:38.450 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlfikrFxAAoZkTUtpHlBcUAMpwWABRxg==", 2024-03-06T07:23:38.450 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:23:38.450 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow *", 2024-03-06T07:23:38.450 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile mgr", 2024-03-06T07:23:38.450 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:23:38.450 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:23:38.450 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:23:38.450 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:23:38.450 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mgr.y", 2024-03-06T07:23:38.450 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlvdcdExAAySXlEfJqLnpMySiiYGFkMQ==", 2024-03-06T07:23:38.450 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:23:38.450 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow *", 2024-03-06T07:23:38.450 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile mgr", 2024-03-06T07:23:38.451 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:23:38.451 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:23:38.451 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:23:38.451 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-03-06T07:23:38.451 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:23:38.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.437+0000 7f0db99c7640 1 -- 172.21.15.83:0/2682658429 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0da006b500 msgr2=0x7f0da006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.437+0000 7f0db99c7640 1 --2- 172.21.15.83:0/2682658429 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0da006b500 0x7f0da006d9c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f0db40ccf50 tx=0x7f0da8009040 comp rx=0 tx=0).stop 2024-03-06T07:23:38.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.437+0000 7f0db99c7640 1 -- 172.21.15.83:0/2682658429 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0db40b4170 msgr2=0x7f0db415dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.437+0000 7f0db99c7640 1 --2- 172.21.15.83:0/2682658429 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0db40b4170 0x7f0db415dfe0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f0da4009ee0 tx=0x7f0da4004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:38.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.437+0000 7f0db99c7640 1 -- 172.21.15.83:0/2682658429 shutdown_connections 2024-03-06T07:23:38.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.437+0000 7f0db99c7640 1 --2- 172.21.15.83:0/2682658429 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0da006b500 0x7f0da006d9c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.437+0000 7f0db99c7640 1 --2- 172.21.15.83:0/2682658429 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0db40cb6b0 0x7f0db40cd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.437+0000 7f0db99c7640 1 --2- 172.21.15.83:0/2682658429 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0db40d0a80 0x7f0db4156d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.437+0000 7f0db99c7640 1 --2- 172.21.15.83:0/2682658429 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0db40b4170 0x7f0db415dfe0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.437+0000 7f0db99c7640 1 -- 172.21.15.83:0/2682658429 >> 172.21.15.83:0/2682658429 conn(0x7f0db40d3c40 msgr2=0x7f0db40b1750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:38.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.438+0000 7f0db99c7640 1 -- 172.21.15.83:0/2682658429 shutdown_connections 2024-03-06T07:23:38.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.438+0000 7f0db99c7640 1 -- 172.21.15.83:0/2682658429 wait complete. 2024-03-06T07:23:38.454 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:38.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.551+0000 7ff933895640 1 Processor -- start 2024-03-06T07:23:38.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.552+0000 7ff933895640 1 -- start start 2024-03-06T07:23:38.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.552+0000 7ff933895640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff92c0d22b0 0x7ff92c0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.552+0000 7ff933895640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ff92c0b41f0 0x7ff92c0b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.552+0000 7ff933895640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff92c0b3430 con 0x7ff92c15dfe0 2024-03-06T07:23:38.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.552+0000 7ff933895640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff92c0d2c50 con 0x7ff92c0b41f0 2024-03-06T07:23:38.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.552+0000 7ff933895640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff92c0b31e0 con 0x7ff92c0d22b0 2024-03-06T07:23:38.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.553+0000 7ff931631640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff92c15dfe0 0x7ff92c0cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:42978/0 (socket says 172.21.15.83:42978) 2024-03-06T07:23:38.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.553+0000 7ff931631640 1 -- 172.21.15.83:0/758165506 learned_addr learned my addr 172.21.15.83:0/758165506 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:23:38.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.553+0000 7ff930e30640 1 --2- 172.21.15.83:0/758165506 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff92c0d22b0 0x7ff92c0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.553+0000 7ff931e32640 1 --2- 172.21.15.83:0/758165506 >> v2:172.21.15.183:3300/0 conn(0x7ff92c0b41f0 0x7ff92c0b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.553+0000 7ff923fff640 1 -- 172.21.15.83:0/758165506 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3061305411 0 0) 0x7ff92c0b3430 con 0x7ff92c15dfe0 2024-03-06T07:23:38.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.553+0000 7ff923fff640 1 -- 172.21.15.83:0/758165506 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff918003520 con 0x7ff92c15dfe0 2024-03-06T07:23:38.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.553+0000 7ff923fff640 1 -- 172.21.15.83:0/758165506 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3969942835 0 0) 0x7ff918003520 con 0x7ff92c15dfe0 2024-03-06T07:23:38.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.553+0000 7ff923fff640 1 -- 172.21.15.83:0/758165506 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff92c0d22b0 msgr2=0x7ff92c0d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:38.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.553+0000 7ff923fff640 1 --2- 172.21.15.83:0/758165506 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff92c0d22b0 0x7ff92c0d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.553+0000 7ff923fff640 1 -- 172.21.15.83:0/758165506 >> v2:172.21.15.183:3300/0 conn(0x7ff92c0b41f0 msgr2=0x7ff92c0b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:38.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.553+0000 7ff923fff640 1 --2- 172.21.15.83:0/758165506 >> v2:172.21.15.183:3300/0 conn(0x7ff92c0b41f0 0x7ff92c0b4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.553+0000 7ff923fff640 1 -- 172.21.15.83:0/758165506 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff92c0b3710 con 0x7ff92c15dfe0 2024-03-06T07:23:38.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.553+0000 7ff923fff640 1 -- 172.21.15.83:0/758165506 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7ff91c001670 con 0x7ff92c15dfe0 2024-03-06T07:23:38.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.553+0000 7ff930e30640 1 --2- 172.21.15.83:0/758165506 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff92c0d22b0 0x7ff92c0d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:38.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.553+0000 7ff923fff640 1 -- 172.21.15.83:0/758165506 >> v1:172.21.15.83:6789/0 conn(0x7ff92c15dfe0 legacy=0x7ff92c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.553+0000 7ff923fff640 1 --2- 172.21.15.83:0/758165506 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9180039b0 0x7ff918003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.553+0000 7ff923fff640 1 --2- 172.21.15.83:0/758165506 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff92c0d22b0 0x7ff918004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.553+0000 7ff923fff640 1 --2- 172.21.15.83:0/758165506 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff918004d40 0x7ff9180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.553+0000 7ff923fff640 1 -- 172.21.15.83:0/758165506 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff92c0b31e0 con 0x7ff9180039b0 2024-03-06T07:23:38.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.553+0000 7ff923fff640 1 -- 172.21.15.83:0/758165506 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff91800b9c0 con 0x7ff918004d40 2024-03-06T07:23:38.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.553+0000 7ff923fff640 1 -- 172.21.15.83:0/758165506 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff91800bb40 con 0x7ff92c0d22b0 2024-03-06T07:23:38.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.553+0000 7ff931631640 1 --2- 172.21.15.83:0/758165506 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9180039b0 0x7ff918003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.553+0000 7ff930e30640 1 --2- 172.21.15.83:0/758165506 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff92c0d22b0 0x7ff918004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.553+0000 7ff931e32640 1 --2- 172.21.15.83:0/758165506 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff918004d40 0x7ff9180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.553+0000 7ff931631640 1 -- 172.21.15.83:0/758165506 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff92c0d22b0 msgr2=0x7ff918004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:38.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.553+0000 7ff931631640 1 --2- 172.21.15.83:0/758165506 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff92c0d22b0 0x7ff918004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.553+0000 7ff931631640 1 -- 172.21.15.83:0/758165506 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff918004d40 msgr2=0x7ff9180051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:38.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.553+0000 7ff931631640 1 --2- 172.21.15.83:0/758165506 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff918004d40 0x7ff9180051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.554+0000 7ff931631640 1 -- 172.21.15.83:0/758165506 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff918003520 con 0x7ff9180039b0 2024-03-06T07:23:38.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.554+0000 7ff931e32640 1 --2- 172.21.15.83:0/758165506 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff918004d40 0x7ff9180051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:38.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.554+0000 7ff931631640 1 --2- 172.21.15.83:0/758165506 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9180039b0 0x7ff918003dd0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7ff91c00c290 tx=0x7ff918005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f735ae41edda2a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:38.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.554+0000 7ff923fff640 1 -- 172.21.15.83:0/758165506 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91c007d50 con 0x7ff9180039b0 2024-03-06T07:23:38.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.554+0000 7ff923fff640 1 -- 172.21.15.83:0/758165506 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff91c030070 con 0x7ff9180039b0 2024-03-06T07:23:38.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.554+0000 7ff923fff640 1 -- 172.21.15.83:0/758165506 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91c02cd80 con 0x7ff9180039b0 2024-03-06T07:23:38.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.555+0000 7ff933895640 1 -- 172.21.15.83:0/758165506 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9180039b0 msgr2=0x7ff918003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.555+0000 7ff933895640 1 --2- 172.21.15.83:0/758165506 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9180039b0 0x7ff918003dd0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7ff91c00c290 tx=0x7ff918005830 comp rx=0 tx=0).stop 2024-03-06T07:23:38.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.555+0000 7ff931e32640 1 -- 172.21.15.83:0/758165506 reap_dead start 2024-03-06T07:23:38.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.555+0000 7ff933895640 1 -- 172.21.15.83:0/758165506 shutdown_connections 2024-03-06T07:23:38.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.555+0000 7ff933895640 1 -- 172.21.15.83:0/758165506 >> 172.21.15.83:0/758165506 conn(0x7ff92c0d3cd0 msgr2=0x7ff92c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:38.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.555+0000 7ff933895640 1 -- 172.21.15.83:0/758165506 shutdown_connections 2024-03-06T07:23:38.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.555+0000 7ff933895640 1 -- 172.21.15.83:0/758165506 wait complete. 2024-03-06T07:23:38.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.556+0000 7ff933895640 1 Processor -- start 2024-03-06T07:23:38.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.556+0000 7ff933895640 1 -- start start 2024-03-06T07:23:38.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.557+0000 7ff933895640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9180039b0 0x7ff92c1f3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.557+0000 7ff933895640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff92c0cd260 0x7ff92c1f3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.557+0000 7ff933895640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff92c1f4890 0x7ff92c0b8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.557+0000 7ff933895640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff92c0cfa00 con 0x7ff92c0cd260 2024-03-06T07:23:38.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.557+0000 7ff933895640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff92c0b3d60 con 0x7ff92c1f4890 2024-03-06T07:23:38.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.557+0000 7ff933895640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff92c0d1c70 con 0x7ff9180039b0 2024-03-06T07:23:38.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.557+0000 7ff930e30640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff92c0cd260 0x7ff92c1f3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.557+0000 7ff930e30640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff92c0cd260 0x7ff92c1f3fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48184/0 (socket says 172.21.15.83:48184) 2024-03-06T07:23:38.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.557+0000 7ff930e30640 1 -- 172.21.15.83:0/271939848 learned_addr learned my addr 172.21.15.83:0/271939848 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:38.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.557+0000 7ff930e30640 1 -- 172.21.15.83:0/271939848 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9180039b0 msgr2=0x7ff92c1f3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.557+0000 7ff930e30640 1 --2- 172.21.15.83:0/271939848 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9180039b0 0x7ff92c1f3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.557+0000 7ff930e30640 1 -- 172.21.15.83:0/271939848 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff92c1f4890 msgr2=0x7ff92c0b8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.557+0000 7ff930e30640 1 --2- 172.21.15.83:0/271939848 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff92c1f4890 0x7ff92c0b8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.557+0000 7ff930e30640 1 -- 172.21.15.83:0/271939848 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff92c0b6a00 con 0x7ff92c0cd260 2024-03-06T07:23:38.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.557+0000 7ff930e30640 1 --2- 172.21.15.83:0/271939848 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff92c0cd260 0x7ff92c1f3fc0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7ff928011660 tx=0x7ff928011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:38.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.557+0000 7ff921ffb640 1 -- 172.21.15.83:0/271939848 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff92800f8e0 con 0x7ff92c0cd260 2024-03-06T07:23:38.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.558+0000 7ff921ffb640 1 -- 172.21.15.83:0/271939848 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff92800fa80 con 0x7ff92c0cd260 2024-03-06T07:23:38.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.558+0000 7ff933895640 1 -- 172.21.15.83:0/271939848 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff92c0b6ce0 con 0x7ff92c0cd260 2024-03-06T07:23:38.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.558+0000 7ff921ffb640 1 -- 172.21.15.83:0/271939848 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff92800fe30 con 0x7ff92c0cd260 2024-03-06T07:23:38.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.558+0000 7ff933895640 1 -- 172.21.15.83:0/271939848 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff92c0b72a0 con 0x7ff92c0cd260 2024-03-06T07:23:38.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.559+0000 7ff933895640 1 -- 172.21.15.83:0/271939848 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff92c04f330 con 0x7ff92c0cd260 2024-03-06T07:23:38.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.559+0000 7ff921ffb640 1 -- 172.21.15.83:0/271939848 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff928015460 con 0x7ff92c0cd260 2024-03-06T07:23:38.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.559+0000 7ff921ffb640 1 --2- 172.21.15.83:0/271939848 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff91006b500 0x7ff91006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.560+0000 7ff921ffb640 1 -- 172.21.15.83:0/271939848 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff92808a610 con 0x7ff92c0cd260 2024-03-06T07:23:38.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.560+0000 7ff931631640 1 --2- 172.21.15.83:0/271939848 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff91006b500 0x7ff91006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.560+0000 7ff931631640 1 --2- 172.21.15.83:0/271939848 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff91006b500 0x7ff91006d9c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff91c034000 tx=0x7ff91c039040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:38.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.564+0000 7ff921ffb640 1 -- 172.21.15.83:0/271939848 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff928019030 con 0x7ff92c0cd260 2024-03-06T07:23:38.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.735+0000 7ff933895640 1 -- 172.21.15.83:0/271939848 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff92c0d41f0 con 0x7ff92c0cd260 2024-03-06T07:23:38.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.735+0000 7ff921ffb640 1 -- 172.21.15.83:0/271939848 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+12903 (secure 0 0 0) 0x7ff928059220 con 0x7ff92c0cd260 2024-03-06T07:23:38.737 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:23:38.737 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":41,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:23:38.059150+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:38.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.737+0000 7ff933895640 1 -- 172.21.15.83:0/271939848 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff91006b500 msgr2=0x7ff91006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.737+0000 7ff933895640 1 --2- 172.21.15.83:0/271939848 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff91006b500 0x7ff91006d9c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff91c034000 tx=0x7ff91c039040 comp rx=0 tx=0).stop 2024-03-06T07:23:38.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.737+0000 7ff933895640 1 -- 172.21.15.83:0/271939848 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff92c0cd260 msgr2=0x7ff92c1f3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.737+0000 7ff933895640 1 --2- 172.21.15.83:0/271939848 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff92c0cd260 0x7ff92c1f3fc0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7ff928011660 tx=0x7ff928011b20 comp rx=0 tx=0).stop 2024-03-06T07:23:38.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.737+0000 7ff933895640 1 -- 172.21.15.83:0/271939848 shutdown_connections 2024-03-06T07:23:38.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.738+0000 7ff933895640 1 --2- 172.21.15.83:0/271939848 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff91006b500 0x7ff91006d9c0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.738+0000 7ff933895640 1 --2- 172.21.15.83:0/271939848 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff92c1f4890 0x7ff92c0b8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.738+0000 7ff933895640 1 --2- 172.21.15.83:0/271939848 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff92c0cd260 0x7ff92c1f3fc0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.738+0000 7ff933895640 1 --2- 172.21.15.83:0/271939848 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9180039b0 0x7ff92c1f3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.738+0000 7ff933895640 1 -- 172.21.15.83:0/271939848 >> 172.21.15.83:0/271939848 conn(0x7ff92c0d3cd0 msgr2=0x7ff92c06ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:38.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.738+0000 7ff933895640 1 -- 172.21.15.83:0/271939848 shutdown_connections 2024-03-06T07:23:38.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.738+0000 7ff933895640 1 -- 172.21.15.83:0/271939848 wait complete. 2024-03-06T07:23:38.750 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:38.751 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:38.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.851+0000 7fdd8942a640 1 Processor -- start 2024-03-06T07:23:38.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.851+0000 7fdd8942a640 1 -- start start 2024-03-06T07:23:38.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.852+0000 7fdd8942a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fdd8415bb40 0x7fdd840b4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.852+0000 7fdd8942a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fdd8415f7d0 0x7fdd8406d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.852+0000 7fdd8942a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd8415ff90 con 0x7fdd8406d5c0 2024-03-06T07:23:38.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.852+0000 7fdd8942a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd8406c910 con 0x7fdd8415bb40 2024-03-06T07:23:38.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.852+0000 7fdd8942a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd84160270 con 0x7fdd8415f7d0 2024-03-06T07:23:38.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.852+0000 7fdd82ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fdd8415bb40 0x7fdd840b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.852+0000 7fdd82ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fdd8415bb40 0x7fdd840b4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48190/0 (socket says 172.21.15.83:48190) 2024-03-06T07:23:38.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.852+0000 7fdd82ffd640 1 -- 172.21.15.83:0/1197909504 learned_addr learned my addr 172.21.15.83:0/1197909504 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:38.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.852+0000 7fdd827fc640 1 --2- 172.21.15.83:0/1197909504 >> v2:172.21.15.183:3300/0 conn(0x7fdd8415f7d0 0x7fdd8406d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.853+0000 7fdd82ffd640 1 -- 172.21.15.83:0/1197909504 >> v2:172.21.15.183:3300/0 conn(0x7fdd8415f7d0 msgr2=0x7fdd8406d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:38.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.853+0000 7fdd82ffd640 1 --2- 172.21.15.83:0/1197909504 >> v2:172.21.15.183:3300/0 conn(0x7fdd8415f7d0 0x7fdd8406d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.853+0000 7fdd82ffd640 1 -- 172.21.15.83:0/1197909504 >> v1:172.21.15.183:6789/0 conn(0x7fdd8406d5c0 legacy=0x7fdd840cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:38.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.853+0000 7fdd82ffd640 1 -- 172.21.15.83:0/1197909504 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd84160550 con 0x7fdd8415bb40 2024-03-06T07:23:38.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.853+0000 7fdd82ffd640 1 --2- 172.21.15.83:0/1197909504 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd8415bb40 0x7fdd840b4970 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fdd8406bfe0 tx=0x7fdd7402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f77240c0fe2d4abe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:38.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.853+0000 7fdd81ffb640 1 -- 172.21.15.83:0/1197909504 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd7402ca80 con 0x7fdd8415bb40 2024-03-06T07:23:38.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.853+0000 7fdd81ffb640 1 -- 172.21.15.83:0/1197909504 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd7402cc20 con 0x7fdd8415bb40 2024-03-06T07:23:38.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.853+0000 7fdd81ffb640 1 -- 172.21.15.83:0/1197909504 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd740343f0 con 0x7fdd8415bb40 2024-03-06T07:23:38.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.854+0000 7fdd8942a640 1 -- 172.21.15.83:0/1197909504 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd8415bb40 msgr2=0x7fdd840b4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.854+0000 7fdd8942a640 1 --2- 172.21.15.83:0/1197909504 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd8415bb40 0x7fdd840b4970 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fdd8406bfe0 tx=0x7fdd7402bee0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.854+0000 7fdd8942a640 1 -- 172.21.15.83:0/1197909504 shutdown_connections 2024-03-06T07:23:38.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.854+0000 7fdd8942a640 1 --2- 172.21.15.83:0/1197909504 >> v2:172.21.15.183:3300/0 conn(0x7fdd8415f7d0 0x7fdd8406d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.854+0000 7fdd8942a640 1 --2- 172.21.15.83:0/1197909504 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd8415bb40 0x7fdd840b4970 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.854+0000 7fdd8942a640 1 -- 172.21.15.83:0/1197909504 >> 172.21.15.83:0/1197909504 conn(0x7fdd8406b190 msgr2=0x7fdd8406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:38.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.854+0000 7fdd8942a640 1 -- 172.21.15.83:0/1197909504 shutdown_connections 2024-03-06T07:23:38.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.854+0000 7fdd8942a640 1 -- 172.21.15.83:0/1197909504 wait complete. 2024-03-06T07:23:38.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.855+0000 7fdd8942a640 1 Processor -- start 2024-03-06T07:23:38.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.855+0000 7fdd8942a640 1 -- start start 2024-03-06T07:23:38.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.856+0000 7fdd8942a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd8406d5c0 0x7fdd84146430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.856+0000 7fdd8942a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd8415bb40 0x7fdd8415ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.856+0000 7fdd8942a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd8415f7d0 0x7fdd840b0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.856+0000 7fdd8942a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8415bf20 con 0x7fdd8415bb40 2024-03-06T07:23:38.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.856+0000 7fdd8942a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8415fc30 con 0x7fdd8415f7d0 2024-03-06T07:23:38.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.856+0000 7fdd8942a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8415c0a0 con 0x7fdd8406d5c0 2024-03-06T07:23:38.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.856+0000 7fdd827fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd8415bb40 0x7fdd8415ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.856+0000 7fdd827fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd8415bb40 0x7fdd8415ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48192/0 (socket says 172.21.15.83:48192) 2024-03-06T07:23:38.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.856+0000 7fdd827fc640 1 -- 172.21.15.83:0/2093382364 learned_addr learned my addr 172.21.15.83:0/2093382364 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:38.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.856+0000 7fdd827fc640 1 -- 172.21.15.83:0/2093382364 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd8406d5c0 msgr2=0x7fdd84146430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.856+0000 7fdd827fc640 1 --2- 172.21.15.83:0/2093382364 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd8406d5c0 0x7fdd84146430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.856+0000 7fdd827fc640 1 -- 172.21.15.83:0/2093382364 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd8415f7d0 msgr2=0x7fdd840b0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.856+0000 7fdd837fe640 1 --2- 172.21.15.83:0/2093382364 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd8415f7d0 0x7fdd840b0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.856+0000 7fdd827fc640 1 --2- 172.21.15.83:0/2093382364 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd8415f7d0 0x7fdd840b0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.856+0000 7fdd827fc640 1 -- 172.21.15.83:0/2093382364 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd8415ecf0 con 0x7fdd8415bb40 2024-03-06T07:23:38.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.856+0000 7fdd837fe640 1 --2- 172.21.15.83:0/2093382364 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd8415f7d0 0x7fdd840b0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:38.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.857+0000 7fdd827fc640 1 --2- 172.21.15.83:0/2093382364 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd8415bb40 0x7fdd8415ce60 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fdd7800d5f0 tx=0x7fdd7800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:38.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.857+0000 7fdd6b7fe640 1 -- 172.21.15.83:0/2093382364 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd780125f0 con 0x7fdd8415bb40 2024-03-06T07:23:38.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.857+0000 7fdd6b7fe640 1 -- 172.21.15.83:0/2093382364 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd78019070 con 0x7fdd8415bb40 2024-03-06T07:23:38.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.857+0000 7fdd6b7fe640 1 -- 172.21.15.83:0/2093382364 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd780129d0 con 0x7fdd8415bb40 2024-03-06T07:23:38.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.857+0000 7fdd8942a640 1 -- 172.21.15.83:0/2093382364 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd8415c380 con 0x7fdd8415bb40 2024-03-06T07:23:38.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.857+0000 7fdd8942a640 1 -- 172.21.15.83:0/2093382364 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd840b1200 con 0x7fdd8415bb40 2024-03-06T07:23:38.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.858+0000 7fdd8942a640 1 -- 172.21.15.83:0/2093382364 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd840732b0 con 0x7fdd8415bb40 2024-03-06T07:23:38.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.858+0000 7fdd6b7fe640 1 -- 172.21.15.83:0/2093382364 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdd78012b70 con 0x7fdd8415bb40 2024-03-06T07:23:38.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.859+0000 7fdd6b7fe640 1 --2- 172.21.15.83:0/2093382364 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdd7006b4d0 0x7fdd7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.859+0000 7fdd6b7fe640 1 -- 172.21.15.83:0/2093382364 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fdd7808a230 con 0x7fdd8415bb40 2024-03-06T07:23:38.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.859+0000 7fdd82ffd640 1 --2- 172.21.15.83:0/2093382364 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdd7006b4d0 0x7fdd7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.859+0000 7fdd82ffd640 1 --2- 172.21.15.83:0/2093382364 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdd7006b4d0 0x7fdd7006d990 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdd74035000 tx=0x7fdd7402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:38.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:38.863+0000 7fdd6b7fe640 1 -- 172.21.15.83:0/2093382364 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdd78059490 con 0x7fdd8415bb40 2024-03-06T07:23:39.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.036+0000 7fdd8942a640 1 -- 172.21.15.83:0/2093382364 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdd84068ff0 con 0x7fdd8415bb40 2024-03-06T07:23:39.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.036+0000 7fdd6b7fe640 1 -- 172.21.15.83:0/2093382364 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdd78058d90 con 0x7fdd8415bb40 2024-03-06T07:23:39.038 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:23:39.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.038+0000 7fdd8942a640 1 -- 172.21.15.83:0/2093382364 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdd7006b4d0 msgr2=0x7fdd7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.038+0000 7fdd8942a640 1 --2- 172.21.15.83:0/2093382364 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdd7006b4d0 0x7fdd7006d990 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdd74035000 tx=0x7fdd7402e000 comp rx=0 tx=0).stop 2024-03-06T07:23:39.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.038+0000 7fdd8942a640 1 -- 172.21.15.83:0/2093382364 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd8415bb40 msgr2=0x7fdd8415ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.038+0000 7fdd8942a640 1 --2- 172.21.15.83:0/2093382364 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd8415bb40 0x7fdd8415ce60 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fdd7800d5f0 tx=0x7fdd7800dab0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.038+0000 7fdd8942a640 1 -- 172.21.15.83:0/2093382364 shutdown_connections 2024-03-06T07:23:39.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.038+0000 7fdd8942a640 1 --2- 172.21.15.83:0/2093382364 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdd7006b4d0 0x7fdd7006d990 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.038+0000 7fdd8942a640 1 --2- 172.21.15.83:0/2093382364 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd8415f7d0 0x7fdd840b0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.038+0000 7fdd8942a640 1 --2- 172.21.15.83:0/2093382364 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd8415bb40 0x7fdd8415ce60 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.038+0000 7fdd8942a640 1 --2- 172.21.15.83:0/2093382364 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd8406d5c0 0x7fdd84146430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.038+0000 7fdd8942a640 1 -- 172.21.15.83:0/2093382364 >> 172.21.15.83:0/2093382364 conn(0x7fdd8406b190 msgr2=0x7fdd840af4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:39.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.038+0000 7fdd8942a640 1 -- 172.21.15.83:0/2093382364 shutdown_connections 2024-03-06T07:23:39.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.038+0000 7fdd8942a640 1 -- 172.21.15.83:0/2093382364 wait complete. 2024-03-06T07:23:39.051 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:23:39.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.149+0000 7f1f77c37640 1 Processor -- start 2024-03-06T07:23:39.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.149+0000 7f1f77c37640 1 -- start start 2024-03-06T07:23:39.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.150+0000 7f1f77c37640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1f7013dc10 0x7f1f700b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.150+0000 7f1f77c37640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f7015f580 con 0x7f1f7015bf30 2024-03-06T07:23:39.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.150+0000 7f1f77c37640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f7015b450 con 0x7f1f7015ef60 2024-03-06T07:23:39.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.150+0000 7f1f77c37640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f700cf3f0 con 0x7f1f7013dc10 2024-03-06T07:23:39.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.150+0000 7f1f751d2640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f1f7015bf30 0x7f1f7015e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:42982/0 (socket says 172.21.15.83:42982) 2024-03-06T07:23:39.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.150+0000 7f1f751d2640 1 -- 172.21.15.83:0/23032601 learned_addr learned my addr 172.21.15.83:0/23032601 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:23:39.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.150+0000 7f1f749d1640 1 -- 172.21.15.83:0/23032601 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 319459512 0 0) 0x7f1f7015f580 con 0x7f1f7015bf30 2024-03-06T07:23:39.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.150+0000 7f1f749d1640 1 -- 172.21.15.83:0/23032601 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f5c003520 con 0x7f1f7015bf30 2024-03-06T07:23:39.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.150+0000 7f1f759d3640 1 --2- 172.21.15.83:0/23032601 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1f7013dc10 0x7f1f700b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.150+0000 7f1f749d1640 1 -- 172.21.15.83:0/23032601 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 420448503 0 0) 0x7f1f5c003520 con 0x7f1f7015bf30 2024-03-06T07:23:39.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.150+0000 7f1f749d1640 1 -- 172.21.15.83:0/23032601 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1f7013dc10 msgr2=0x7f1f700b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:39.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.150+0000 7f1f749d1640 1 --2- 172.21.15.83:0/23032601 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1f7013dc10 0x7f1f700b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.150+0000 7f1f749d1640 1 -- 172.21.15.83:0/23032601 >> v1:172.21.15.183:6789/0 conn(0x7f1f7015ef60 legacy=0x7f1f700708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.150+0000 7f1f749d1640 1 -- 172.21.15.83:0/23032601 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f7015b730 con 0x7f1f7015bf30 2024-03-06T07:23:39.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.150+0000 7f1f749d1640 1 -- 172.21.15.83:0/23032601 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f1f60001670 con 0x7f1f7015bf30 2024-03-06T07:23:39.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.150+0000 7f1f749d1640 1 -- 172.21.15.83:0/23032601 >> v1:172.21.15.83:6789/0 conn(0x7f1f7015bf30 legacy=0x7f1f7015e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.150+0000 7f1f749d1640 1 --2- 172.21.15.83:0/23032601 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f5c0039e0 0x7f1f5c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.150+0000 7f1f749d1640 1 --2- 172.21.15.83:0/23032601 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f5c004800 0x7f1f5c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.150+0000 7f1f749d1640 1 --2- 172.21.15.83:0/23032601 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1f7013dc10 0x7f1f5c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.150+0000 7f1f749d1640 1 -- 172.21.15.83:0/23032601 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f700cf3f0 con 0x7f1f5c004800 2024-03-06T07:23:39.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.150+0000 7f1f749d1640 1 -- 172.21.15.83:0/23032601 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f5c00baf0 con 0x7f1f5c0039e0 2024-03-06T07:23:39.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.150+0000 7f1f749d1640 1 -- 172.21.15.83:0/23032601 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f5c00bcf0 con 0x7f1f7013dc10 2024-03-06T07:23:39.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.151+0000 7f1f751d2640 1 --2- 172.21.15.83:0/23032601 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f5c004800 0x7f1f5c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.151+0000 7f1f759d3640 1 --2- 172.21.15.83:0/23032601 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f5c0039e0 0x7f1f5c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.151+0000 7f1f761d4640 1 --2- 172.21.15.83:0/23032601 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1f7013dc10 0x7f1f5c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.151+0000 7f1f751d2640 1 -- 172.21.15.83:0/23032601 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1f7013dc10 msgr2=0x7f1f5c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:39.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.151+0000 7f1f751d2640 1 --2- 172.21.15.83:0/23032601 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1f7013dc10 0x7f1f5c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.151+0000 7f1f751d2640 1 -- 172.21.15.83:0/23032601 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f5c0039e0 msgr2=0x7f1f5c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:39.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.151+0000 7f1f751d2640 1 --2- 172.21.15.83:0/23032601 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f5c0039e0 0x7f1f5c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.151+0000 7f1f751d2640 1 -- 172.21.15.83:0/23032601 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f5c003520 con 0x7f1f5c004800 2024-03-06T07:23:39.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.151+0000 7f1f759d3640 1 --2- 172.21.15.83:0/23032601 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f5c0039e0 0x7f1f5c003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:39.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.151+0000 7f1f751d2640 1 --2- 172.21.15.83:0/23032601 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f5c004800 0x7f1f5c004c80 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f1f6000c1c0 tx=0x7f1f5c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30c17aa099ae4934 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:39.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.152+0000 7f1f749d1640 1 -- 172.21.15.83:0/23032601 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f60007cf0 con 0x7f1f5c004800 2024-03-06T07:23:39.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.152+0000 7f1f749d1640 1 -- 172.21.15.83:0/23032601 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f60034070 con 0x7f1f5c004800 2024-03-06T07:23:39.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.152+0000 7f1f749d1640 1 -- 172.21.15.83:0/23032601 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f60005b10 con 0x7f1f5c004800 2024-03-06T07:23:39.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.152+0000 7f1f77c37640 1 -- 172.21.15.83:0/23032601 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f5c004800 msgr2=0x7f1f5c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.152+0000 7f1f77c37640 1 --2- 172.21.15.83:0/23032601 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f5c004800 0x7f1f5c004c80 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f1f6000c1c0 tx=0x7f1f5c005830 comp rx=0 tx=0).stop 2024-03-06T07:23:39.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.152+0000 7f1f761d4640 1 -- 172.21.15.83:0/23032601 reap_dead start 2024-03-06T07:23:39.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.152+0000 7f1f77c37640 1 -- 172.21.15.83:0/23032601 shutdown_connections 2024-03-06T07:23:39.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.152+0000 7f1f77c37640 1 -- 172.21.15.83:0/23032601 >> 172.21.15.83:0/23032601 conn(0x7f1f700cf8a0 msgr2=0x7f1f700cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:39.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.152+0000 7f1f77c37640 1 -- 172.21.15.83:0/23032601 shutdown_connections 2024-03-06T07:23:39.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.152+0000 7f1f77c37640 1 -- 172.21.15.83:0/23032601 wait complete. 2024-03-06T07:23:39.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.153+0000 7f1f77c37640 1 Processor -- start 2024-03-06T07:23:39.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.153+0000 7f1f77c37640 1 -- start start 2024-03-06T07:23:39.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.154+0000 7f1f77c37640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f5c004800 0x7f1f70132030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.154+0000 7f1f77c37640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1f7015c400 0x7f1f7015c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.154+0000 7f1f77c37640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f7006c800 0x7f1f7006ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.154+0000 7f1f77c37640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f70071ff0 con 0x7f1f7006c800 2024-03-06T07:23:39.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.154+0000 7f1f77c37640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7015b210 con 0x7f1f5c004800 2024-03-06T07:23:39.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.154+0000 7f1f77c37640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f70072170 con 0x7f1f7015c400 2024-03-06T07:23:39.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.154+0000 7f1f761d4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f7006c800 0x7f1f7006ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.154+0000 7f1f761d4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f7006c800 0x7f1f7006ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48218/0 (socket says 172.21.15.83:48218) 2024-03-06T07:23:39.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.154+0000 7f1f761d4640 1 -- 172.21.15.83:0/3310025170 learned_addr learned my addr 172.21.15.83:0/3310025170 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:39.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.154+0000 7f1f761d4640 1 -- 172.21.15.83:0/3310025170 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1f7015c400 msgr2=0x7f1f7015c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.154+0000 7f1f761d4640 1 --2- 172.21.15.83:0/3310025170 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1f7015c400 0x7f1f7015c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.154+0000 7f1f761d4640 1 -- 172.21.15.83:0/3310025170 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f5c004800 msgr2=0x7f1f70132030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.154+0000 7f1f761d4640 1 --2- 172.21.15.83:0/3310025170 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f5c004800 0x7f1f70132030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.155+0000 7f1f761d4640 1 -- 172.21.15.83:0/3310025170 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f7006d450 con 0x7f1f7006c800 2024-03-06T07:23:39.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.155+0000 7f1f761d4640 1 --2- 172.21.15.83:0/3310025170 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f7006c800 0x7f1f7006ccb0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f1f7015c390 tx=0x7f1f6800eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:39.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.155+0000 7f1f667fc640 1 -- 172.21.15.83:0/3310025170 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f68003970 con 0x7f1f7006c800 2024-03-06T07:23:39.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.155+0000 7f1f667fc640 1 -- 172.21.15.83:0/3310025170 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f68018070 con 0x7f1f7006c800 2024-03-06T07:23:39.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.155+0000 7f1f77c37640 1 -- 172.21.15.83:0/3310025170 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f7006d730 con 0x7f1f7006c800 2024-03-06T07:23:39.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.155+0000 7f1f667fc640 1 -- 172.21.15.83:0/3310025170 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f68003d50 con 0x7f1f7006c800 2024-03-06T07:23:39.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.155+0000 7f1f77c37640 1 -- 172.21.15.83:0/3310025170 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f7006dcf0 con 0x7f1f7006c800 2024-03-06T07:23:39.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.156+0000 7f1f77c37640 1 -- 172.21.15.83:0/3310025170 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f7015cdc0 con 0x7f1f7006c800 2024-03-06T07:23:39.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.156+0000 7f1f667fc640 1 -- 172.21.15.83:0/3310025170 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1f680050d0 con 0x7f1f7006c800 2024-03-06T07:23:39.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.157+0000 7f1f667fc640 1 --2- 172.21.15.83:0/3310025170 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1f5406b500 0x7f1f5406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.157+0000 7f1f667fc640 1 -- 172.21.15.83:0/3310025170 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1f68089b20 con 0x7f1f7006c800 2024-03-06T07:23:39.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.157+0000 7f1f759d3640 1 --2- 172.21.15.83:0/3310025170 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1f5406b500 0x7f1f5406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.157+0000 7f1f759d3640 1 --2- 172.21.15.83:0/3310025170 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1f5406b500 0x7f1f5406d9c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1f70159240 tx=0x7f1f6c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:39.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.161+0000 7f1f667fc640 1 -- 172.21.15.83:0/3310025170 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1f68058cf0 con 0x7f1f7006c800 2024-03-06T07:23:39.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.337+0000 7f1f77c37640 1 -- 172.21.15.83:0/3310025170 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f1f7004f330 con 0x7f1f7006c800 2024-03-06T07:23:39.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.337+0000 7f1f667fc640 1 -- 172.21.15.83:0/3310025170 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v41) v1 ==== 84+0+65 (secure 0 0 0) 0x7f1f680585f0 con 0x7f1f7006c800 2024-03-06T07:23:39.339 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:23:39.339 INFO:teuthology.orchestra.run.smithi083.stdout:[ 2024-03-06T07:23:39.339 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:23:39.339 INFO:teuthology.orchestra.run.smithi083.stdout: "poolnum": 1, 2024-03-06T07:23:39.339 INFO:teuthology.orchestra.run.smithi083.stdout: "poolname": ".mgr" 2024-03-06T07:23:39.339 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:23:39.339 INFO:teuthology.orchestra.run.smithi083.stdout:] 2024-03-06T07:23:39.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.339+0000 7f1f77c37640 1 -- 172.21.15.83:0/3310025170 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1f5406b500 msgr2=0x7f1f5406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.339+0000 7f1f77c37640 1 --2- 172.21.15.83:0/3310025170 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1f5406b500 0x7f1f5406d9c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1f70159240 tx=0x7f1f6c008040 comp rx=0 tx=0).stop 2024-03-06T07:23:39.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.339+0000 7f1f77c37640 1 -- 172.21.15.83:0/3310025170 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f7006c800 msgr2=0x7f1f7006ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.339+0000 7f1f77c37640 1 --2- 172.21.15.83:0/3310025170 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f7006c800 0x7f1f7006ccb0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f1f7015c390 tx=0x7f1f6800eb90 comp rx=0 tx=0).stop 2024-03-06T07:23:39.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.339+0000 7f1f77c37640 1 -- 172.21.15.83:0/3310025170 shutdown_connections 2024-03-06T07:23:39.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.339+0000 7f1f77c37640 1 --2- 172.21.15.83:0/3310025170 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1f5406b500 0x7f1f5406d9c0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.339+0000 7f1f77c37640 1 --2- 172.21.15.83:0/3310025170 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f7006c800 0x7f1f7006ccb0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.339+0000 7f1f77c37640 1 --2- 172.21.15.83:0/3310025170 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1f7015c400 0x7f1f7015c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.339+0000 7f1f77c37640 1 --2- 172.21.15.83:0/3310025170 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f5c004800 0x7f1f70132030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.339+0000 7f1f77c37640 1 -- 172.21.15.83:0/3310025170 >> 172.21.15.83:0/3310025170 conn(0x7f1f700cf8a0 msgr2=0x7f1f700b0830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:39.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.339+0000 7f1f77c37640 1 -- 172.21.15.83:0/3310025170 shutdown_connections 2024-03-06T07:23:39.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.339+0000 7f1f77c37640 1 -- 172.21.15.83:0/3310025170 wait complete. 2024-03-06T07:23:39.352 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:23:39.352 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:23:39.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.454+0000 7ff7a2815640 1 Processor -- start 2024-03-06T07:23:39.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.454+0000 7ff7a2815640 1 -- start start 2024-03-06T07:23:39.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.455+0000 7ff7a2815640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff79c15dfa0 0x7ff79c0d18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.455+0000 7ff7a2815640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ff79c0d29f0 0x7ff79c0cb190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.455+0000 7ff7a2815640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff79c15e380 con 0x7ff79c0d4630 2024-03-06T07:23:39.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.455+0000 7ff7a2815640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff79c0d2100 con 0x7ff79c15dfa0 2024-03-06T07:23:39.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.455+0000 7ff7a2815640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff79c0d1e30 con 0x7ff79c0d29f0 2024-03-06T07:23:39.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.455+0000 7ff79bfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff79c15dfa0 0x7ff79c0d18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.455+0000 7ff79b7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff79c0d4630 0x7ff79c0d22e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:42998/0 (socket says 172.21.15.83:42998) 2024-03-06T07:23:39.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.455+0000 7ff79b7fe640 1 -- 172.21.15.83:0/1641820330 learned_addr learned my addr 172.21.15.83:0/1641820330 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:23:39.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.456+0000 7ff7a0db2640 1 --2- 172.21.15.83:0/1641820330 >> v2:172.21.15.183:3300/0 conn(0x7ff79c0d29f0 0x7ff79c0cb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.456+0000 7ff79bfff640 1 -- 172.21.15.83:0/1641820330 >> v2:172.21.15.183:3300/0 conn(0x7ff79c0d29f0 msgr2=0x7ff79c0cb190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:39.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.456+0000 7ff79bfff640 1 --2- 172.21.15.83:0/1641820330 >> v2:172.21.15.183:3300/0 conn(0x7ff79c0d29f0 0x7ff79c0cb190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.456+0000 7ff79bfff640 1 -- 172.21.15.83:0/1641820330 >> v1:172.21.15.83:6789/0 conn(0x7ff79c0d4630 legacy=0x7ff79c0d22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.456+0000 7ff79bfff640 1 -- 172.21.15.83:0/1641820330 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff79c0cf470 con 0x7ff79c15dfa0 2024-03-06T07:23:39.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.456+0000 7ff79affd640 1 -- 172.21.15.83:0/1641820330 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2900709687 0 0) 0x7ff79c15e380 con 0x7ff79c0d4630 2024-03-06T07:23:39.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.456+0000 7ff7a0db2640 1 --2- 172.21.15.83:0/1641820330 >> v2:172.21.15.183:3300/0 conn(0x7ff79c0d29f0 0x7ff79c0cb190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:39.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.456+0000 7ff79bfff640 1 --2- 172.21.15.83:0/1641820330 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff79c15dfa0 0x7ff79c0d18f0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7ff79c0ce580 tx=0x7ff78c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dd493d9071b7a09 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:39.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.457+0000 7ff79affd640 1 -- 172.21.15.83:0/1641820330 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff78c0315a0 con 0x7ff79c15dfa0 2024-03-06T07:23:39.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.457+0000 7ff79affd640 1 -- 172.21.15.83:0/1641820330 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff78c037070 con 0x7ff79c15dfa0 2024-03-06T07:23:39.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.457+0000 7ff79affd640 1 -- 172.21.15.83:0/1641820330 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff78c031960 con 0x7ff79c15dfa0 2024-03-06T07:23:39.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.457+0000 7ff7a2815640 1 -- 172.21.15.83:0/1641820330 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff79c15dfa0 msgr2=0x7ff79c0d18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.457+0000 7ff7a2815640 1 --2- 172.21.15.83:0/1641820330 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff79c15dfa0 0x7ff79c0d18f0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7ff79c0ce580 tx=0x7ff78c02b840 comp rx=0 tx=0).stop 2024-03-06T07:23:39.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.457+0000 7ff7a2815640 1 -- 172.21.15.83:0/1641820330 shutdown_connections 2024-03-06T07:23:39.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.457+0000 7ff7a2815640 1 --2- 172.21.15.83:0/1641820330 >> v2:172.21.15.183:3300/0 conn(0x7ff79c0d29f0 0x7ff79c0cb190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.457+0000 7ff7a2815640 1 --2- 172.21.15.83:0/1641820330 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff79c15dfa0 0x7ff79c0d18f0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.457+0000 7ff7a2815640 1 -- 172.21.15.83:0/1641820330 >> 172.21.15.83:0/1641820330 conn(0x7ff79c09d000 msgr2=0x7ff79c0d3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:39.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.457+0000 7ff7a2815640 1 -- 172.21.15.83:0/1641820330 shutdown_connections 2024-03-06T07:23:39.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.457+0000 7ff7a2815640 1 -- 172.21.15.83:0/1641820330 wait complete. 2024-03-06T07:23:39.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.458+0000 7ff7a2815640 1 Processor -- start 2024-03-06T07:23:39.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.459+0000 7ff7a2815640 1 -- start start 2024-03-06T07:23:39.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.459+0000 7ff7a2815640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff79c0d29f0 0x7ff79c15bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.459+0000 7ff7a2815640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff79c0d4630 0x7ff79c09da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.459+0000 7ff7a2815640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff79c15dfa0 0x7ff79c09f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.459+0000 7ff7a2815640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c0cb6d0 con 0x7ff79c15dfa0 2024-03-06T07:23:39.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.459+0000 7ff7a2815640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c0cfa30 con 0x7ff79c0d4630 2024-03-06T07:23:39.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.459+0000 7ff7a2815640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c0cb850 con 0x7ff79c0d29f0 2024-03-06T07:23:39.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.459+0000 7ff7a0db2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff79c15dfa0 0x7ff79c09f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.459+0000 7ff7a0db2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff79c15dfa0 0x7ff79c09f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48246/0 (socket says 172.21.15.83:48246) 2024-03-06T07:23:39.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.459+0000 7ff7a0db2640 1 -- 172.21.15.83:0/1461148857 learned_addr learned my addr 172.21.15.83:0/1461148857 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:39.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.459+0000 7ff79bfff640 1 --2- 172.21.15.83:0/1461148857 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff79c0d29f0 0x7ff79c15bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.460+0000 7ff79b7fe640 1 --2- 172.21.15.83:0/1461148857 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff79c0d4630 0x7ff79c09da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.460+0000 7ff7a0db2640 1 -- 172.21.15.83:0/1461148857 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff79c0d29f0 msgr2=0x7ff79c15bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.460+0000 7ff7a0db2640 1 --2- 172.21.15.83:0/1461148857 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff79c0d29f0 0x7ff79c15bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.460+0000 7ff7a0db2640 1 -- 172.21.15.83:0/1461148857 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff79c0d4630 msgr2=0x7ff79c09da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.460+0000 7ff7a0db2640 1 --2- 172.21.15.83:0/1461148857 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff79c0d4630 0x7ff79c09da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.460+0000 7ff7a0db2640 1 -- 172.21.15.83:0/1461148857 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff79c0a03b0 con 0x7ff79c15dfa0 2024-03-06T07:23:39.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.460+0000 7ff79bfff640 1 --2- 172.21.15.83:0/1461148857 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff79c0d29f0 0x7ff79c15bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:39.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.460+0000 7ff7a0db2640 1 --2- 172.21.15.83:0/1461148857 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff79c15dfa0 0x7ff79c09f6e0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7ff79400ce80 tx=0x7ff79400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:39.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.460+0000 7ff798ff9640 1 -- 172.21.15.83:0/1461148857 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff79401a070 con 0x7ff79c15dfa0 2024-03-06T07:23:39.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.460+0000 7ff798ff9640 1 -- 172.21.15.83:0/1461148857 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff79400d880 con 0x7ff79c15dfa0 2024-03-06T07:23:39.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.460+0000 7ff798ff9640 1 -- 172.21.15.83:0/1461148857 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff79400dc70 con 0x7ff79c15dfa0 2024-03-06T07:23:39.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.460+0000 7ff7a2815640 1 -- 172.21.15.83:0/1461148857 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff79c0a0690 con 0x7ff79c15dfa0 2024-03-06T07:23:39.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.460+0000 7ff7a2815640 1 -- 172.21.15.83:0/1461148857 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff79c0c7e90 con 0x7ff79c15dfa0 2024-03-06T07:23:39.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.461+0000 7ff7a2815640 1 -- 172.21.15.83:0/1461148857 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff79c04f330 con 0x7ff79c15dfa0 2024-03-06T07:23:39.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.461+0000 7ff798ff9640 1 -- 172.21.15.83:0/1461148857 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff79401d070 con 0x7ff79c15dfa0 2024-03-06T07:23:39.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.462+0000 7ff798ff9640 1 --2- 172.21.15.83:0/1461148857 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff78806b4d0 0x7ff78806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.462+0000 7ff798ff9640 1 -- 172.21.15.83:0/1461148857 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff79408a5a0 con 0x7ff79c15dfa0 2024-03-06T07:23:39.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.462+0000 7ff79bfff640 1 --2- 172.21.15.83:0/1461148857 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff78806b4d0 0x7ff78806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.462+0000 7ff79bfff640 1 --2- 172.21.15.83:0/1461148857 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff78806b4d0 0x7ff78806d990 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff78c036000 tx=0x7ff78c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:39.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.466+0000 7ff798ff9640 1 -- 172.21.15.83:0/1461148857 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff7940591b0 con 0x7ff79c15dfa0 2024-03-06T07:23:39.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:39.637+0000 7ff7a2815640 1 -- 172.21.15.83:0/1461148857 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7ff79c0747a0 con 0x7ff79c15dfa0 2024-03-06T07:23:40.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.074+0000 7ff798ff9640 1 -- 172.21.15.83:0/1461148857 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v42) v1 ==== 138+0+0 (secure 0 0 0) 0x7ff79405d250 con 0x7ff79c15dfa0 2024-03-06T07:23:40.077 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'cephfs_metadata' created 2024-03-06T07:23:40.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.076+0000 7ff7a2815640 1 -- 172.21.15.83:0/1461148857 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff78806b4d0 msgr2=0x7ff78806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:40.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.076+0000 7ff7a2815640 1 --2- 172.21.15.83:0/1461148857 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff78806b4d0 0x7ff78806d990 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff78c036000 tx=0x7ff78c008030 comp rx=0 tx=0).stop 2024-03-06T07:23:40.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.076+0000 7ff7a2815640 1 -- 172.21.15.83:0/1461148857 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff79c15dfa0 msgr2=0x7ff79c09f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:40.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.076+0000 7ff7a2815640 1 --2- 172.21.15.83:0/1461148857 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff79c15dfa0 0x7ff79c09f6e0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7ff79400ce80 tx=0x7ff79400a710 comp rx=0 tx=0).stop 2024-03-06T07:23:40.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.077+0000 7ff7a2815640 1 -- 172.21.15.83:0/1461148857 shutdown_connections 2024-03-06T07:23:40.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.077+0000 7ff7a2815640 1 --2- 172.21.15.83:0/1461148857 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff78806b4d0 0x7ff78806d990 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.077+0000 7ff7a2815640 1 --2- 172.21.15.83:0/1461148857 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff79c15dfa0 0x7ff79c09f6e0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.077+0000 7ff7a2815640 1 --2- 172.21.15.83:0/1461148857 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff79c0d4630 0x7ff79c09da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.077+0000 7ff7a2815640 1 --2- 172.21.15.83:0/1461148857 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff79c0d29f0 0x7ff79c15bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.077+0000 7ff7a2815640 1 -- 172.21.15.83:0/1461148857 >> 172.21.15.83:0/1461148857 conn(0x7ff79c09d000 msgr2=0x7ff79c0cd410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:40.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.077+0000 7ff7a2815640 1 -- 172.21.15.83:0/1461148857 shutdown_connections 2024-03-06T07:23:40.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.077+0000 7ff7a2815640 1 -- 172.21.15.83:0/1461148857 wait complete. 2024-03-06T07:23:40.093 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:23:40.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.198+0000 7fbc0c71e640 1 Processor -- start 2024-03-06T07:23:40.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.198+0000 7fbc0c71e640 1 -- start start 2024-03-06T07:23:40.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.199+0000 7fbc0c71e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fbc041586e0 0x7fbc040aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.199+0000 7fbc0c71e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbc040aef80 0x7fbc040cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.199+0000 7fbc0c71e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbc040cffd0 0x7fbc0406da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.199+0000 7fbc0c71e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc040d03f0 con 0x7fbc040cffd0 2024-03-06T07:23:40.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.199+0000 7fbc0c71e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc04158ac0 con 0x7fbc041586e0 2024-03-06T07:23:40.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.199+0000 7fbc0c71e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0415b2a0 con 0x7fbc040aef80 2024-03-06T07:23:40.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.199+0000 7fbc0acbb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbc040cffd0 0x7fbc0406da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.199+0000 7fbc0acbb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbc040cffd0 0x7fbc0406da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48260/0 (socket says 172.21.15.83:48260) 2024-03-06T07:23:40.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.199+0000 7fbc0acbb640 1 -- 172.21.15.83:0/1985635812 learned_addr learned my addr 172.21.15.83:0/1985635812 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:40.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.199+0000 7fbc09cb9640 1 --2- 172.21.15.83:0/1985635812 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbc040aef80 0x7fbc040cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.199+0000 7fbc0a4ba640 1 --2- 172.21.15.83:0/1985635812 >> v2:172.21.15.183:3300/0 conn(0x7fbc041586e0 0x7fbc040aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.199+0000 7fbc0acbb640 1 -- 172.21.15.83:0/1985635812 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbc040aef80 msgr2=0x7fbc040cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:40.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.199+0000 7fbc0acbb640 1 --2- 172.21.15.83:0/1985635812 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbc040aef80 0x7fbc040cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.199+0000 7fbc0acbb640 1 -- 172.21.15.83:0/1985635812 >> v2:172.21.15.183:3300/0 conn(0x7fbc041586e0 msgr2=0x7fbc040aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:40.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.199+0000 7fbc0acbb640 1 --2- 172.21.15.83:0/1985635812 >> v2:172.21.15.183:3300/0 conn(0x7fbc041586e0 0x7fbc040aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.199+0000 7fbc0acbb640 1 -- 172.21.15.83:0/1985635812 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc0415b540 con 0x7fbc040cffd0 2024-03-06T07:23:40.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.199+0000 7fbc0a4ba640 1 --2- 172.21.15.83:0/1985635812 >> v2:172.21.15.183:3300/0 conn(0x7fbc041586e0 0x7fbc040aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:40.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.200+0000 7fbc0acbb640 1 --2- 172.21.15.83:0/1985635812 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc040cffd0 0x7fbc0406da20 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fbc04070d10 tx=0x7fbbfc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f129ad8e0a02c07 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:40.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.200+0000 7fbc094b8640 1 -- 172.21.15.83:0/1985635812 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc02ca70 con 0x7fbc040cffd0 2024-03-06T07:23:40.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.200+0000 7fbc094b8640 1 -- 172.21.15.83:0/1985635812 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbfc02cc10 con 0x7fbc040cffd0 2024-03-06T07:23:40.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.200+0000 7fbc094b8640 1 -- 172.21.15.83:0/1985635812 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc0343f0 con 0x7fbc040cffd0 2024-03-06T07:23:40.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.201+0000 7fbc0c71e640 1 -- 172.21.15.83:0/1985635812 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc040cffd0 msgr2=0x7fbc0406da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:40.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.201+0000 7fbc0c71e640 1 --2- 172.21.15.83:0/1985635812 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc040cffd0 0x7fbc0406da20 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fbc04070d10 tx=0x7fbbfc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.201+0000 7fbc0c71e640 1 -- 172.21.15.83:0/1985635812 shutdown_connections 2024-03-06T07:23:40.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.201+0000 7fbc0c71e640 1 --2- 172.21.15.83:0/1985635812 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc040cffd0 0x7fbc0406da20 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.201+0000 7fbc0c71e640 1 --2- 172.21.15.83:0/1985635812 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbc040aef80 0x7fbc040cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.201+0000 7fbc0c71e640 1 --2- 172.21.15.83:0/1985635812 >> v2:172.21.15.183:3300/0 conn(0x7fbc041586e0 0x7fbc040aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.201+0000 7fbc0c71e640 1 -- 172.21.15.83:0/1985635812 >> 172.21.15.83:0/1985635812 conn(0x7fbc0406b190 msgr2=0x7fbc0406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:40.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.201+0000 7fbc0c71e640 1 -- 172.21.15.83:0/1985635812 shutdown_connections 2024-03-06T07:23:40.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.201+0000 7fbc0c71e640 1 -- 172.21.15.83:0/1985635812 wait complete. 2024-03-06T07:23:40.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.202+0000 7fbc0c71e640 1 Processor -- start 2024-03-06T07:23:40.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.202+0000 7fbc0c71e640 1 -- start start 2024-03-06T07:23:40.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.203+0000 7fbc0c71e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbc040aef80 0x7fbc0414f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.203+0000 7fbc0c71e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbc040cffd0 0x7fbc04150010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.203+0000 7fbc0c71e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc041586e0 0x7fbc04150550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.203+0000 7fbc0c71e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0415bde0 con 0x7fbc041586e0 2024-03-06T07:23:40.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.203+0000 7fbc0c71e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0415bb00 con 0x7fbc040aef80 2024-03-06T07:23:40.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.203+0000 7fbc0c71e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc04071350 con 0x7fbc040cffd0 2024-03-06T07:23:40.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.203+0000 7fbc0acbb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc041586e0 0x7fbc04150550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.203+0000 7fbc0acbb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc041586e0 0x7fbc04150550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48270/0 (socket says 172.21.15.83:48270) 2024-03-06T07:23:40.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.203+0000 7fbc0acbb640 1 -- 172.21.15.83:0/2098189849 learned_addr learned my addr 172.21.15.83:0/2098189849 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:40.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.203+0000 7fbc0a4ba640 1 --2- 172.21.15.83:0/2098189849 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbc040aef80 0x7fbc0414f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.203+0000 7fbc09cb9640 1 --2- 172.21.15.83:0/2098189849 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbc040cffd0 0x7fbc04150010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.203+0000 7fbc0acbb640 1 -- 172.21.15.83:0/2098189849 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbc040cffd0 msgr2=0x7fbc04150010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:40.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.203+0000 7fbc0acbb640 1 --2- 172.21.15.83:0/2098189849 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbc040cffd0 0x7fbc04150010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.203+0000 7fbc0acbb640 1 -- 172.21.15.83:0/2098189849 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbc040aef80 msgr2=0x7fbc0414f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:40.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.203+0000 7fbc0acbb640 1 --2- 172.21.15.83:0/2098189849 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbc040aef80 0x7fbc0414f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.203+0000 7fbc0acbb640 1 -- 172.21.15.83:0/2098189849 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc0406cac0 con 0x7fbc041586e0 2024-03-06T07:23:40.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.203+0000 7fbc0a4ba640 1 --2- 172.21.15.83:0/2098189849 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbc040aef80 0x7fbc0414f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:40.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.203+0000 7fbc0acbb640 1 --2- 172.21.15.83:0/2098189849 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc041586e0 0x7fbc04150550 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc036040 tx=0x7fbbfc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:40.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.204+0000 7fbbfaffd640 1 -- 172.21.15.83:0/2098189849 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc043020 con 0x7fbc041586e0 2024-03-06T07:23:40.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.204+0000 7fbbfaffd640 1 -- 172.21.15.83:0/2098189849 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbfc03e450 con 0x7fbc041586e0 2024-03-06T07:23:40.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.204+0000 7fbc0c71e640 1 -- 172.21.15.83:0/2098189849 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc04150b80 con 0x7fbc041586e0 2024-03-06T07:23:40.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.204+0000 7fbbfaffd640 1 -- 172.21.15.83:0/2098189849 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc009510 con 0x7fbc041586e0 2024-03-06T07:23:40.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.204+0000 7fbc0c71e640 1 -- 172.21.15.83:0/2098189849 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc0406d260 con 0x7fbc041586e0 2024-03-06T07:23:40.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.205+0000 7fbc0c71e640 1 -- 172.21.15.83:0/2098189849 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc040732b0 con 0x7fbc041586e0 2024-03-06T07:23:40.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.205+0000 7fbbfaffd640 1 -- 172.21.15.83:0/2098189849 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbbfc046020 con 0x7fbc041586e0 2024-03-06T07:23:40.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.205+0000 7fbbfaffd640 1 --2- 172.21.15.83:0/2098189849 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbbe806b4d0 0x7fbbe806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.205+0000 7fbbfaffd640 1 -- 172.21.15.83:0/2098189849 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fbbfc0ac7e0 con 0x7fbc041586e0 2024-03-06T07:23:40.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.205+0000 7fbc0a4ba640 1 --2- 172.21.15.83:0/2098189849 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbbe806b4d0 0x7fbbe806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.206+0000 7fbc0a4ba640 1 --2- 172.21.15.83:0/2098189849 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbbe806b4d0 0x7fbbe806d990 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fbc041f7440 tx=0x7fbc00008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:40.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.210+0000 7fbbfaffd640 1 -- 172.21.15.83:0/2098189849 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbbfc04d030 con 0x7fbc041586e0 2024-03-06T07:23:40.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:40.384+0000 7fbc0c71e640 1 -- 172.21.15.83:0/2098189849 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fbc04068ff0 con 0x7fbc041586e0 2024-03-06T07:23:41.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.079+0000 7fbbfaffd640 1 -- 172.21.15.83:0/2098189849 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v43) v1 ==== 170+0+0 (secure 0 0 0) 0x7fbbfc07b8d0 con 0x7fbc041586e0 2024-03-06T07:23:41.081 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'cephfs_data' created 2024-03-06T07:23:41.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.082+0000 7fbc0c71e640 1 -- 172.21.15.83:0/2098189849 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbbe806b4d0 msgr2=0x7fbbe806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:41.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.082+0000 7fbc0c71e640 1 --2- 172.21.15.83:0/2098189849 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbbe806b4d0 0x7fbbe806d990 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fbc041f7440 tx=0x7fbc00008040 comp rx=0 tx=0).stop 2024-03-06T07:23:41.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.082+0000 7fbc0c71e640 1 -- 172.21.15.83:0/2098189849 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc041586e0 msgr2=0x7fbc04150550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:41.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.082+0000 7fbc0c71e640 1 --2- 172.21.15.83:0/2098189849 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc041586e0 0x7fbc04150550 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc036040 tx=0x7fbbfc004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:41.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.082+0000 7fbc0c71e640 1 -- 172.21.15.83:0/2098189849 shutdown_connections 2024-03-06T07:23:41.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.082+0000 7fbc0c71e640 1 --2- 172.21.15.83:0/2098189849 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbbe806b4d0 0x7fbbe806d990 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.082+0000 7fbc0c71e640 1 --2- 172.21.15.83:0/2098189849 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc041586e0 0x7fbc04150550 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.082+0000 7fbc0c71e640 1 --2- 172.21.15.83:0/2098189849 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbc040cffd0 0x7fbc04150010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.082+0000 7fbc0c71e640 1 --2- 172.21.15.83:0/2098189849 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbc040aef80 0x7fbc0414f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.082+0000 7fbc0c71e640 1 -- 172.21.15.83:0/2098189849 >> 172.21.15.83:0/2098189849 conn(0x7fbc0406b190 msgr2=0x7fbc0406b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:41.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.082+0000 7fbc0c71e640 1 -- 172.21.15.83:0/2098189849 shutdown_connections 2024-03-06T07:23:41.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.082+0000 7fbc0c71e640 1 -- 172.21.15.83:0/2098189849 wait complete. 2024-03-06T07:23:41.104 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:23:41.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.209+0000 7f0597f02640 1 Processor -- start 2024-03-06T07:23:41.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.209+0000 7f0597f02640 1 -- start start 2024-03-06T07:23:41.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.210+0000 7f0597f02640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f05900b5f80 0x7f05900b55e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:41.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.210+0000 7f0597f02640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0590066d00 0x7f0590167d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:41.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.210+0000 7f0597f02640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f059006b050 con 0x7f059015e270 2024-03-06T07:23:41.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.210+0000 7f0597f02640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f059015e650 con 0x7f0590066d00 2024-03-06T07:23:41.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.210+0000 7f0597f02640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05900afdf0 con 0x7f05900b5f80 2024-03-06T07:23:41.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.210+0000 7f059649f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0590066d00 0x7f0590167d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:41.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.210+0000 7f059649f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0590066d00 0x7f0590167d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48286/0 (socket says 172.21.15.83:48286) 2024-03-06T07:23:41.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.210+0000 7f059649f640 1 -- 172.21.15.83:0/3493090376 learned_addr learned my addr 172.21.15.83:0/3493090376 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:41.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.210+0000 7f059649f640 1 -- 172.21.15.83:0/3493090376 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f05900b5f80 msgr2=0x7f05900b55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:41.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.210+0000 7f059649f640 1 --2- 172.21.15.83:0/3493090376 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f05900b5f80 0x7f05900b55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.210+0000 7f059649f640 1 -- 172.21.15.83:0/3493090376 >> v1:172.21.15.183:6789/0 conn(0x7f059015e270 legacy=0x7f05900b4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:41.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.210+0000 7f059649f640 1 -- 172.21.15.83:0/3493090376 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f059006b330 con 0x7f0590066d00 2024-03-06T07:23:41.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.211+0000 7f059649f640 1 --2- 172.21.15.83:0/3493090376 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0590066d00 0x7f0590167d20 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f0588009e50 tx=0x7f058802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a904f5fee5535346 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:41.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.211+0000 7f0594c9c640 1 -- 172.21.15.83:0/3493090376 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058802ca70 con 0x7f0590066d00 2024-03-06T07:23:41.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.211+0000 7f0594c9c640 1 -- 172.21.15.83:0/3493090376 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f058802cc10 con 0x7f0590066d00 2024-03-06T07:23:41.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.211+0000 7f0594c9c640 1 -- 172.21.15.83:0/3493090376 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05880343f0 con 0x7f0590066d00 2024-03-06T07:23:41.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.211+0000 7f0597f02640 1 -- 172.21.15.83:0/3493090376 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0590066d00 msgr2=0x7f0590167d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:41.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.211+0000 7f0597f02640 1 --2- 172.21.15.83:0/3493090376 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0590066d00 0x7f0590167d20 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f0588009e50 tx=0x7f058802bed0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.211+0000 7f0597f02640 1 -- 172.21.15.83:0/3493090376 shutdown_connections 2024-03-06T07:23:41.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.211+0000 7f0597f02640 1 --2- 172.21.15.83:0/3493090376 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0590066d00 0x7f0590167d20 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.211+0000 7f0597f02640 1 --2- 172.21.15.83:0/3493090376 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f05900b5f80 0x7f05900b55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.211+0000 7f0597f02640 1 -- 172.21.15.83:0/3493090376 >> 172.21.15.83:0/3493090376 conn(0x7f05900725d0 msgr2=0x7f05900729e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:41.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.212+0000 7f0597f02640 1 -- 172.21.15.83:0/3493090376 shutdown_connections 2024-03-06T07:23:41.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.212+0000 7f0597f02640 1 -- 172.21.15.83:0/3493090376 wait complete. 2024-03-06T07:23:41.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.213+0000 7f0597f02640 1 Processor -- start 2024-03-06T07:23:41.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.213+0000 7f0597f02640 1 -- start start 2024-03-06T07:23:41.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.213+0000 7f0597f02640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0590066d00 0x7f059009d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:41.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.213+0000 7f0597f02640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f05900b5f80 0x7f05900cc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:41.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.213+0000 7f0597f02640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f059015e270 0x7f05900cc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:41.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.213+0000 7f0597f02640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05900b0c70 con 0x7f0590066d00 2024-03-06T07:23:41.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.213+0000 7f0597f02640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05900b0990 con 0x7f059015e270 2024-03-06T07:23:41.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.213+0000 7f0597f02640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05900b1100 con 0x7f05900b5f80 2024-03-06T07:23:41.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.213+0000 7f0595c9e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0590066d00 0x7f059009d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:41.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.213+0000 7f0595c9e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0590066d00 0x7f059009d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48294/0 (socket says 172.21.15.83:48294) 2024-03-06T07:23:41.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.213+0000 7f0595c9e640 1 -- 172.21.15.83:0/4130007886 learned_addr learned my addr 172.21.15.83:0/4130007886 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:41.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.214+0000 7f059649f640 1 --2- 172.21.15.83:0/4130007886 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f059015e270 0x7f05900cc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:41.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.214+0000 7f059549d640 1 --2- 172.21.15.83:0/4130007886 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f05900b5f80 0x7f05900cc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:41.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.214+0000 7f0595c9e640 1 -- 172.21.15.83:0/4130007886 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f05900b5f80 msgr2=0x7f05900cc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:41.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.214+0000 7f0595c9e640 1 --2- 172.21.15.83:0/4130007886 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f05900b5f80 0x7f05900cc1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.214+0000 7f0595c9e640 1 -- 172.21.15.83:0/4130007886 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f059015e270 msgr2=0x7f05900cc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:41.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.214+0000 7f0595c9e640 1 --2- 172.21.15.83:0/4130007886 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f059015e270 0x7f05900cc6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.214+0000 7f0595c9e640 1 -- 172.21.15.83:0/4130007886 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05900c9c90 con 0x7f0590066d00 2024-03-06T07:23:41.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.214+0000 7f0595c9e640 1 --2- 172.21.15.83:0/4130007886 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0590066d00 0x7f059009d860 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f058c00c440 tx=0x7f058c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:41.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.214+0000 7f05867fc640 1 -- 172.21.15.83:0/4130007886 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058c017070 con 0x7f0590066d00 2024-03-06T07:23:41.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.214+0000 7f05867fc640 1 -- 172.21.15.83:0/4130007886 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f058c0035f0 con 0x7f0590066d00 2024-03-06T07:23:41.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.214+0000 7f0597f02640 1 -- 172.21.15.83:0/4130007886 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05900c9f70 con 0x7f0590066d00 2024-03-06T07:23:41.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.214+0000 7f05867fc640 1 -- 172.21.15.83:0/4130007886 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058c0039e0 con 0x7f0590066d00 2024-03-06T07:23:41.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.214+0000 7f0597f02640 1 -- 172.21.15.83:0/4130007886 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f059015c290 con 0x7f0590066d00 2024-03-06T07:23:41.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.215+0000 7f0597f02640 1 -- 172.21.15.83:0/4130007886 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f059004f330 con 0x7f0590066d00 2024-03-06T07:23:41.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.215+0000 7f05867fc640 1 -- 172.21.15.83:0/4130007886 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f058c01a070 con 0x7f0590066d00 2024-03-06T07:23:41.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.216+0000 7f05867fc640 1 --2- 172.21.15.83:0/4130007886 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f057406b4d0 0x7f057406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:41.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.216+0000 7f05867fc640 1 -- 172.21.15.83:0/4130007886 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f058c089490 con 0x7f0590066d00 2024-03-06T07:23:41.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.216+0000 7f059549d640 1 --2- 172.21.15.83:0/4130007886 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f057406b4d0 0x7f057406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:41.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.216+0000 7f059549d640 1 --2- 172.21.15.83:0/4130007886 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f057406b4d0 0x7f057406d990 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f05900c6c50 tx=0x7f0580008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:41.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.220+0000 7f05867fc640 1 -- 172.21.15.83:0/4130007886 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f058c058400 con 0x7f0590066d00 2024-03-06T07:23:41.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:41.428+0000 7f0597f02640 1 -- 172.21.15.83:0/4130007886 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f059009e1b0 con 0x7f0590066d00 2024-03-06T07:23:42.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.081+0000 7f05867fc640 1 -- 172.21.15.83:0/4130007886 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v17) v1 ==== 172+0+0 (secure 0 0 0) 0x7f058c0010c0 con 0x7f0590066d00 2024-03-06T07:23:42.083 INFO:teuthology.orchestra.run.smithi083.stderr:new fs with metadata pool 6 and data pool 7 2024-03-06T07:23:42.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.087+0000 7f0597f02640 1 -- 172.21.15.83:0/4130007886 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f057406b4d0 msgr2=0x7f057406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.087+0000 7f0597f02640 1 --2- 172.21.15.83:0/4130007886 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f057406b4d0 0x7f057406d990 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f05900c6c50 tx=0x7f0580008040 comp rx=0 tx=0).stop 2024-03-06T07:23:42.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.087+0000 7f0597f02640 1 -- 172.21.15.83:0/4130007886 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0590066d00 msgr2=0x7f059009d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.087+0000 7f0597f02640 1 --2- 172.21.15.83:0/4130007886 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0590066d00 0x7f059009d860 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f058c00c440 tx=0x7f058c00c900 comp rx=0 tx=0).stop 2024-03-06T07:23:42.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.088+0000 7f0597f02640 1 -- 172.21.15.83:0/4130007886 shutdown_connections 2024-03-06T07:23:42.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.088+0000 7f0597f02640 1 --2- 172.21.15.83:0/4130007886 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f057406b4d0 0x7f057406d990 secure :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f05900c6c50 tx=0x7f0580008040 comp rx=0 tx=0).stop 2024-03-06T07:23:42.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.088+0000 7f0597f02640 1 --2- 172.21.15.83:0/4130007886 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f059015e270 0x7f05900cc6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.088+0000 7f0597f02640 1 --2- 172.21.15.83:0/4130007886 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f05900b5f80 0x7f05900cc1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.088+0000 7f0597f02640 1 --2- 172.21.15.83:0/4130007886 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0590066d00 0x7f059009d860 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.088+0000 7f0597f02640 1 -- 172.21.15.83:0/4130007886 >> 172.21.15.83:0/4130007886 conn(0x7f05900725d0 msgr2=0x7f05900ad610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:42.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.098+0000 7f0597f02640 1 -- 172.21.15.83:0/4130007886 shutdown_connections 2024-03-06T07:23:42.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.098+0000 7f0597f02640 1 -- 172.21.15.83:0/4130007886 wait complete. 2024-03-06T07:23:42.123 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:42.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.225+0000 7ff14ed55640 1 Processor -- start 2024-03-06T07:23:42.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.225+0000 7ff14ed55640 1 -- start start 2024-03-06T07:23:42.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.226+0000 7ff14ed55640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff14813dc10 0x7ff1480b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.226+0000 7ff14ed55640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ff14815ef60 0x7ff1480b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.226+0000 7ff14ed55640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14815f580 con 0x7ff14815bf30 2024-03-06T07:23:42.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.226+0000 7ff14ed55640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1480cf3f0 con 0x7ff14815ef60 2024-03-06T07:23:42.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.226+0000 7ff14ed55640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff14815b210 con 0x7ff14813dc10 2024-03-06T07:23:42.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.226+0000 7ff147fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff14815bf30 0x7ff14815e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43008/0 (socket says 172.21.15.83:43008) 2024-03-06T07:23:42.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.226+0000 7ff147fff640 1 -- 172.21.15.83:0/2503150400 learned_addr learned my addr 172.21.15.83:0/2503150400 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:23:42.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.226+0000 7ff1477fe640 1 -- 172.21.15.83:0/2503150400 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1557879324 0 0) 0x7ff14815f580 con 0x7ff14815bf30 2024-03-06T07:23:42.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.226+0000 7ff1477fe640 1 -- 172.21.15.83:0/2503150400 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff138003520 con 0x7ff14815bf30 2024-03-06T07:23:42.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.226+0000 7ff14d2f2640 1 --2- 172.21.15.83:0/2503150400 >> v2:172.21.15.183:3300/0 conn(0x7ff14815ef60 0x7ff1480b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.226+0000 7ff14caf1640 1 --2- 172.21.15.83:0/2503150400 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff14813dc10 0x7ff1480b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.227+0000 7ff1477fe640 1 -- 172.21.15.83:0/2503150400 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 748978431 0 0) 0x7ff138003520 con 0x7ff14815bf30 2024-03-06T07:23:42.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.227+0000 7ff1477fe640 1 -- 172.21.15.83:0/2503150400 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff14813dc10 msgr2=0x7ff1480b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:42.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.227+0000 7ff1477fe640 1 --2- 172.21.15.83:0/2503150400 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff14813dc10 0x7ff1480b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.227+0000 7ff1477fe640 1 -- 172.21.15.83:0/2503150400 >> v2:172.21.15.183:3300/0 conn(0x7ff14815ef60 msgr2=0x7ff1480b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:42.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.227+0000 7ff1477fe640 1 --2- 172.21.15.83:0/2503150400 >> v2:172.21.15.183:3300/0 conn(0x7ff14815ef60 0x7ff1480b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.227+0000 7ff1477fe640 1 -- 172.21.15.83:0/2503150400 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14815b450 con 0x7ff14815bf30 2024-03-06T07:23:42.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.227+0000 7ff1477fe640 1 -- 172.21.15.83:0/2503150400 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7ff134001670 con 0x7ff14815bf30 2024-03-06T07:23:42.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.227+0000 7ff1477fe640 1 -- 172.21.15.83:0/2503150400 >> v1:172.21.15.83:6789/0 conn(0x7ff14815bf30 legacy=0x7ff14815e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.227+0000 7ff1477fe640 1 --2- 172.21.15.83:0/2503150400 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff14813dc10 0x7ff1380039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.227+0000 7ff1477fe640 1 --2- 172.21.15.83:0/2503150400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff138003f30 0x7ff138004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.227+0000 7ff1477fe640 1 --2- 172.21.15.83:0/2503150400 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff138004d40 0x7ff1380051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.227+0000 7ff1477fe640 1 -- 172.21.15.83:0/2503150400 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1480cf3f0 con 0x7ff138003f30 2024-03-06T07:23:42.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.227+0000 7ff1477fe640 1 -- 172.21.15.83:0/2503150400 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14815b210 con 0x7ff138004d40 2024-03-06T07:23:42.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.227+0000 7ff1477fe640 1 -- 172.21.15.83:0/2503150400 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff13800dcd0 con 0x7ff14813dc10 2024-03-06T07:23:42.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.227+0000 7ff147fff640 1 --2- 172.21.15.83:0/2503150400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff138003f30 0x7ff138004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.227+0000 7ff14caf1640 1 --2- 172.21.15.83:0/2503150400 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff14813dc10 0x7ff1380039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.227+0000 7ff14d2f2640 1 --2- 172.21.15.83:0/2503150400 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff138004d40 0x7ff1380051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.227+0000 7ff147fff640 1 -- 172.21.15.83:0/2503150400 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff14813dc10 msgr2=0x7ff1380039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:42.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.227+0000 7ff147fff640 1 --2- 172.21.15.83:0/2503150400 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff14813dc10 0x7ff1380039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.227+0000 7ff147fff640 1 -- 172.21.15.83:0/2503150400 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff138004d40 msgr2=0x7ff1380051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:42.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.227+0000 7ff147fff640 1 --2- 172.21.15.83:0/2503150400 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff138004d40 0x7ff1380051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.227+0000 7ff147fff640 1 -- 172.21.15.83:0/2503150400 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff138003520 con 0x7ff138003f30 2024-03-06T07:23:42.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.227+0000 7ff14d2f2640 1 --2- 172.21.15.83:0/2503150400 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff138004d40 0x7ff1380051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:42.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.228+0000 7ff147fff640 1 --2- 172.21.15.83:0/2503150400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff138003f30 0x7ff138004800 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7ff13400c210 tx=0x7ff138005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1097f48e0e11adb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:42.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.228+0000 7ff1477fe640 1 -- 172.21.15.83:0/2503150400 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff134007d50 con 0x7ff138003f30 2024-03-06T07:23:42.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.228+0000 7ff1477fe640 1 -- 172.21.15.83:0/2503150400 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff134030070 con 0x7ff138003f30 2024-03-06T07:23:42.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.228+0000 7ff1477fe640 1 -- 172.21.15.83:0/2503150400 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13402cd20 con 0x7ff138003f30 2024-03-06T07:23:42.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.228+0000 7ff14ed55640 1 -- 172.21.15.83:0/2503150400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff138003f30 msgr2=0x7ff138004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.228+0000 7ff14ed55640 1 --2- 172.21.15.83:0/2503150400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff138003f30 0x7ff138004800 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7ff13400c210 tx=0x7ff138005830 comp rx=0 tx=0).stop 2024-03-06T07:23:42.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.228+0000 7ff14d2f2640 1 -- 172.21.15.83:0/2503150400 reap_dead start 2024-03-06T07:23:42.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.228+0000 7ff14ed55640 1 -- 172.21.15.83:0/2503150400 shutdown_connections 2024-03-06T07:23:42.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.228+0000 7ff14ed55640 1 -- 172.21.15.83:0/2503150400 >> 172.21.15.83:0/2503150400 conn(0x7ff1480cf8a0 msgr2=0x7ff1480cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:42.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.228+0000 7ff14ed55640 1 -- 172.21.15.83:0/2503150400 shutdown_connections 2024-03-06T07:23:42.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.229+0000 7ff14ed55640 1 -- 172.21.15.83:0/2503150400 wait complete. 2024-03-06T07:23:42.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.230+0000 7ff14ed55640 1 Processor -- start 2024-03-06T07:23:42.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.230+0000 7ff14ed55640 1 -- start start 2024-03-06T07:23:42.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.230+0000 7ff14ed55640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff138003f30 0x7ff1481365b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.230+0000 7ff14ed55640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff14813e170 0x7ff14813e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.230+0000 7ff14ed55640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1480ad860 0x7ff1480adce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.230+0000 7ff14ed55640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1480b2950 con 0x7ff14813e170 2024-03-06T07:23:42.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.230+0000 7ff14ed55640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14813dff0 con 0x7ff1480ad860 2024-03-06T07:23:42.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.230+0000 7ff14ed55640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1480b2ad0 con 0x7ff138003f30 2024-03-06T07:23:42.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.231+0000 7ff147fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff14813e170 0x7ff14813e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.231+0000 7ff147fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff14813e170 0x7ff14813e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48316/0 (socket says 172.21.15.83:48316) 2024-03-06T07:23:42.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.231+0000 7ff147fff640 1 -- 172.21.15.83:0/392920270 learned_addr learned my addr 172.21.15.83:0/392920270 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:42.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.231+0000 7ff14caf1640 1 --2- 172.21.15.83:0/392920270 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff138003f30 0x7ff1481365b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.231+0000 7ff14d2f2640 1 --2- 172.21.15.83:0/392920270 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1480ad860 0x7ff1480adce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.231+0000 7ff147fff640 1 -- 172.21.15.83:0/392920270 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff138003f30 msgr2=0x7ff1481365b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.231+0000 7ff147fff640 1 --2- 172.21.15.83:0/392920270 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff138003f30 0x7ff1481365b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.231+0000 7ff147fff640 1 -- 172.21.15.83:0/392920270 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1480ad860 msgr2=0x7ff1480adce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.231+0000 7ff147fff640 1 --2- 172.21.15.83:0/392920270 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1480ad860 0x7ff1480adce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.231+0000 7ff147fff640 1 -- 172.21.15.83:0/392920270 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1480ae590 con 0x7ff14813e170 2024-03-06T07:23:42.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.231+0000 7ff147fff640 1 --2- 172.21.15.83:0/392920270 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff14813e170 0x7ff14813e5f0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7ff13400c330 tx=0x7ff14815b620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:42.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.231+0000 7ff1457fa640 1 -- 172.21.15.83:0/392920270 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff134045070 con 0x7ff14813e170 2024-03-06T07:23:42.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.231+0000 7ff1457fa640 1 -- 172.21.15.83:0/392920270 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff134004bf0 con 0x7ff14813e170 2024-03-06T07:23:42.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.231+0000 7ff14ed55640 1 -- 172.21.15.83:0/392920270 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1480ae870 con 0x7ff14813e170 2024-03-06T07:23:42.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.231+0000 7ff1457fa640 1 -- 172.21.15.83:0/392920270 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff134001cb0 con 0x7ff14813e170 2024-03-06T07:23:42.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.231+0000 7ff14ed55640 1 -- 172.21.15.83:0/392920270 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff14806cd30 con 0x7ff14813e170 2024-03-06T07:23:42.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.232+0000 7ff14ed55640 1 -- 172.21.15.83:0/392920270 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1480aea40 con 0x7ff14813e170 2024-03-06T07:23:42.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.233+0000 7ff1457fa640 1 -- 172.21.15.83:0/392920270 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff134007070 con 0x7ff14813e170 2024-03-06T07:23:42.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.233+0000 7ff1457fa640 1 --2- 172.21.15.83:0/392920270 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff12c06b4d0 0x7ff12c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.233+0000 7ff1457fa640 1 -- 172.21.15.83:0/392920270 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff1340ab240 con 0x7ff14813e170 2024-03-06T07:23:42.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.233+0000 7ff14caf1640 1 --2- 172.21.15.83:0/392920270 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff12c06b4d0 0x7ff12c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.233+0000 7ff14caf1640 1 --2- 172.21.15.83:0/392920270 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff12c06b4d0 0x7ff12c06d990 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff13c009a20 tx=0x7ff13c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:42.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.237+0000 7ff1457fa640 1 -- 172.21.15.83:0/392920270 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff13407a230 con 0x7ff14813e170 2024-03-06T07:23:42.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.410+0000 7ff14ed55640 1 -- 172.21.15.83:0/392920270 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff14804f330 con 0x7ff14813e170 2024-03-06T07:23:42.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.412+0000 7ff1457fa640 1 -- 172.21.15.83:0/392920270 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+16638 (secure 0 0 0) 0x7ff134079b30 con 0x7ff14813e170 2024-03-06T07:23:42.414 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:23:42.414 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":44,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:23:41.429827+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:39.656617+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:40.402765+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:42.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.413+0000 7ff14ed55640 1 -- 172.21.15.83:0/392920270 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff12c06b4d0 msgr2=0x7ff12c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.413+0000 7ff14ed55640 1 --2- 172.21.15.83:0/392920270 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff12c06b4d0 0x7ff12c06d990 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff13c009a20 tx=0x7ff13c008040 comp rx=0 tx=0).stop 2024-03-06T07:23:42.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.413+0000 7ff14ed55640 1 -- 172.21.15.83:0/392920270 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff14813e170 msgr2=0x7ff14813e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.413+0000 7ff14ed55640 1 --2- 172.21.15.83:0/392920270 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff14813e170 0x7ff14813e5f0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7ff13400c330 tx=0x7ff14815b620 comp rx=0 tx=0).stop 2024-03-06T07:23:42.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.414+0000 7ff14ed55640 1 -- 172.21.15.83:0/392920270 shutdown_connections 2024-03-06T07:23:42.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.414+0000 7ff14ed55640 1 --2- 172.21.15.83:0/392920270 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff12c06b4d0 0x7ff12c06d990 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.414+0000 7ff14ed55640 1 --2- 172.21.15.83:0/392920270 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1480ad860 0x7ff1480adce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.414+0000 7ff14ed55640 1 --2- 172.21.15.83:0/392920270 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff14813e170 0x7ff14813e5f0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.414+0000 7ff14ed55640 1 --2- 172.21.15.83:0/392920270 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff138003f30 0x7ff1481365b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.414+0000 7ff14ed55640 1 -- 172.21.15.83:0/392920270 >> 172.21.15.83:0/392920270 conn(0x7ff1480cf8a0 msgr2=0x7ff1480cd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:42.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.414+0000 7ff14ed55640 1 -- 172.21.15.83:0/392920270 shutdown_connections 2024-03-06T07:23:42.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.414+0000 7ff14ed55640 1 -- 172.21.15.83:0/392920270 wait complete. 2024-03-06T07:23:42.427 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:42.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.525+0000 7f706ac9d640 1 Processor -- start 2024-03-06T07:23:42.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.525+0000 7f706ac9d640 1 -- start start 2024-03-06T07:23:42.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.526+0000 7f706ac9d640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f706413dc10 0x7f70640b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.526+0000 7f706ac9d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f706415bf30 0x7f70640b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.526+0000 7f706ac9d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f706415f580 con 0x7f706415e850 2024-03-06T07:23:42.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.526+0000 7f706ac9d640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70640cf3f0 con 0x7f706413dc10 2024-03-06T07:23:42.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.526+0000 7f706ac9d640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f706415b210 con 0x7f706415bf30 2024-03-06T07:23:42.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.526+0000 7f7063fff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f706415bf30 0x7f70640b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.526+0000 7f7068a39640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f706413dc10 0x7f70640b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.526+0000 7f7063fff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f706415bf30 0x7f70640b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:60074/0 (socket says 172.21.15.83:60074) 2024-03-06T07:23:42.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.526+0000 7f7068a39640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f706413dc10 0x7f70640b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:41176/0 (socket says 172.21.15.83:41176) 2024-03-06T07:23:42.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.526+0000 7f706923a640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f706415e850 0x7f706415eca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:42190/0 (socket says 172.21.15.83:42190) 2024-03-06T07:23:42.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.526+0000 7f7063fff640 1 -- 172.21.15.83:0/2940870379 learned_addr learned my addr 172.21.15.83:0/2940870379 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:42.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.527+0000 7f7068a39640 1 -- 172.21.15.83:0/2940870379 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f706415bf30 msgr2=0x7f70640b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:42.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.527+0000 7f7068a39640 1 --2- 172.21.15.83:0/2940870379 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f706415bf30 0x7f70640b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.527+0000 7f7068a39640 1 -- 172.21.15.83:0/2940870379 >> v1:172.21.15.183:6789/0 conn(0x7f706415e850 legacy=0x7f706415eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.527+0000 7f7068a39640 1 -- 172.21.15.83:0/2940870379 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f706415b450 con 0x7f706413dc10 2024-03-06T07:23:42.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.527+0000 7f70637fe640 1 -- 172.21.15.83:0/2940870379 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3673333551 0 0) 0x7f706415f580 con 0x7f706415e850 2024-03-06T07:23:42.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.527+0000 7f7063fff640 1 --2- 172.21.15.83:0/2940870379 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f706415bf30 0x7f70640b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:42.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.528+0000 7f7068a39640 1 --2- 172.21.15.83:0/2940870379 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f706413dc10 0x7f70640b2950 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f70540097a0 tx=0x7f705402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f348bda88c2f6f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:42.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.528+0000 7f70637fe640 1 -- 172.21.15.83:0/2940870379 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70540315a0 con 0x7f706413dc10 2024-03-06T07:23:42.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.528+0000 7f70637fe640 1 -- 172.21.15.83:0/2940870379 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7054037070 con 0x7f706413dc10 2024-03-06T07:23:42.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.528+0000 7f70637fe640 1 -- 172.21.15.83:0/2940870379 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7054031960 con 0x7f706413dc10 2024-03-06T07:23:42.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.528+0000 7f706ac9d640 1 -- 172.21.15.83:0/2940870379 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f706413dc10 msgr2=0x7f70640b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.528+0000 7f706ac9d640 1 --2- 172.21.15.83:0/2940870379 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f706413dc10 0x7f70640b2950 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f70540097a0 tx=0x7f705402b840 comp rx=0 tx=0).stop 2024-03-06T07:23:42.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.528+0000 7f706ac9d640 1 -- 172.21.15.83:0/2940870379 shutdown_connections 2024-03-06T07:23:42.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.528+0000 7f706ac9d640 1 --2- 172.21.15.83:0/2940870379 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f706415bf30 0x7f70640b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.528+0000 7f706ac9d640 1 --2- 172.21.15.83:0/2940870379 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f706413dc10 0x7f70640b2950 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.528+0000 7f706ac9d640 1 -- 172.21.15.83:0/2940870379 >> 172.21.15.83:0/2940870379 conn(0x7f70640cf8a0 msgr2=0x7f70640cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:42.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.528+0000 7f706ac9d640 1 -- 172.21.15.83:0/2940870379 shutdown_connections 2024-03-06T07:23:42.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.528+0000 7f706ac9d640 1 -- 172.21.15.83:0/2940870379 wait complete. 2024-03-06T07:23:42.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.529+0000 7f706ac9d640 1 Processor -- start 2024-03-06T07:23:42.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.530+0000 7f706ac9d640 1 -- start start 2024-03-06T07:23:42.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.530+0000 7f706ac9d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f706413dc10 0x7f70641585d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.530+0000 7f706ac9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f706415bf30 0x7f706413c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.530+0000 7f706ac9d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f706415e850 0x7f706412ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.530+0000 7f706ac9d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70640cd7a0 con 0x7f706415bf30 2024-03-06T07:23:42.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.530+0000 7f706ac9d640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f706413dff0 con 0x7f706415e850 2024-03-06T07:23:42.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.530+0000 7f706ac9d640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70640cd920 con 0x7f706413dc10 2024-03-06T07:23:42.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.531+0000 7f7063fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f706415bf30 0x7f706413c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.531+0000 7f7063fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f706415bf30 0x7f706413c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48324/0 (socket says 172.21.15.83:48324) 2024-03-06T07:23:42.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.531+0000 7f7063fff640 1 -- 172.21.15.83:0/2032520127 learned_addr learned my addr 172.21.15.83:0/2032520127 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:42.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.531+0000 7f7068a39640 1 --2- 172.21.15.83:0/2032520127 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f706413dc10 0x7f70641585d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.531+0000 7f706923a640 1 --2- 172.21.15.83:0/2032520127 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f706415e850 0x7f706412ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.531+0000 7f7063fff640 1 -- 172.21.15.83:0/2032520127 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f706413dc10 msgr2=0x7f70641585d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.531+0000 7f7063fff640 1 --2- 172.21.15.83:0/2032520127 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f706413dc10 0x7f70641585d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.531+0000 7f7063fff640 1 -- 172.21.15.83:0/2032520127 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f706415e850 msgr2=0x7f706412ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.531+0000 7f7063fff640 1 --2- 172.21.15.83:0/2032520127 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f706415e850 0x7f706412ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.531+0000 7f7063fff640 1 -- 172.21.15.83:0/2032520127 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f706406d9f0 con 0x7f706415bf30 2024-03-06T07:23:42.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.531+0000 7f7063fff640 1 --2- 172.21.15.83:0/2032520127 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f706415bf30 0x7f706413c400 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f7058010f10 tx=0x7f705800e730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:42.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.531+0000 7f70617fa640 1 -- 172.21.15.83:0/2032520127 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f705801e070 con 0x7f706415bf30 2024-03-06T07:23:42.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.531+0000 7f70617fa640 1 -- 172.21.15.83:0/2032520127 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7058011860 con 0x7f706415bf30 2024-03-06T07:23:42.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.531+0000 7f706ac9d640 1 -- 172.21.15.83:0/2032520127 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f706406dcd0 con 0x7f706415bf30 2024-03-06T07:23:42.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.531+0000 7f70617fa640 1 -- 172.21.15.83:0/2032520127 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7058011c10 con 0x7f706415bf30 2024-03-06T07:23:42.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.531+0000 7f706ac9d640 1 -- 172.21.15.83:0/2032520127 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f706406e290 con 0x7f706415bf30 2024-03-06T07:23:42.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.532+0000 7f706ac9d640 1 -- 172.21.15.83:0/2032520127 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f706404f330 con 0x7f706415bf30 2024-03-06T07:23:42.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.533+0000 7f70617fa640 1 -- 172.21.15.83:0/2032520127 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7058021070 con 0x7f706415bf30 2024-03-06T07:23:42.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.533+0000 7f70617fa640 1 --2- 172.21.15.83:0/2032520127 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f705006b4d0 0x7f705006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.533+0000 7f7068a39640 1 --2- 172.21.15.83:0/2032520127 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f705006b4d0 0x7f705006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.533+0000 7f70617fa640 1 -- 172.21.15.83:0/2032520127 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f705808cc70 con 0x7f706415bf30 2024-03-06T07:23:42.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.534+0000 7f7068a39640 1 --2- 172.21.15.83:0/2032520127 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f705006b4d0 0x7f705006d990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7054036000 tx=0x7f7054008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:42.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.538+0000 7f70617fa640 1 -- 172.21.15.83:0/2032520127 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f705805b4a0 con 0x7f706415bf30 2024-03-06T07:23:42.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.710+0000 7f706ac9d640 1 -- 172.21.15.83:0/2032520127 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f706406eb00 con 0x7f706415bf30 2024-03-06T07:23:42.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.711+0000 7f70617fa640 1 -- 172.21.15.83:0/2032520127 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+16638 (secure 0 0 0) 0x7f705805f540 con 0x7f706415bf30 2024-03-06T07:23:42.713 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:23:42.714 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":44,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:23:41.429827+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:39.656617+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:40.402765+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:42.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.714+0000 7f706ac9d640 1 -- 172.21.15.83:0/2032520127 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f705006b4d0 msgr2=0x7f705006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.714+0000 7f706ac9d640 1 --2- 172.21.15.83:0/2032520127 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f705006b4d0 0x7f705006d990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7054036000 tx=0x7f7054008030 comp rx=0 tx=0).stop 2024-03-06T07:23:42.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.714+0000 7f706ac9d640 1 -- 172.21.15.83:0/2032520127 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f706415bf30 msgr2=0x7f706413c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.714+0000 7f706ac9d640 1 --2- 172.21.15.83:0/2032520127 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f706415bf30 0x7f706413c400 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f7058010f10 tx=0x7f705800e730 comp rx=0 tx=0).stop 2024-03-06T07:23:42.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.714+0000 7f706ac9d640 1 -- 172.21.15.83:0/2032520127 shutdown_connections 2024-03-06T07:23:42.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.714+0000 7f706ac9d640 1 --2- 172.21.15.83:0/2032520127 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f705006b4d0 0x7f705006d990 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.714+0000 7f706ac9d640 1 --2- 172.21.15.83:0/2032520127 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f706415e850 0x7f706412ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.714+0000 7f706ac9d640 1 --2- 172.21.15.83:0/2032520127 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f706415bf30 0x7f706413c400 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.714+0000 7f706ac9d640 1 --2- 172.21.15.83:0/2032520127 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f706413dc10 0x7f70641585d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.714+0000 7f706ac9d640 1 -- 172.21.15.83:0/2032520127 >> 172.21.15.83:0/2032520127 conn(0x7f70640cf8a0 msgr2=0x7f70640cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:42.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.714+0000 7f706ac9d640 1 -- 172.21.15.83:0/2032520127 shutdown_connections 2024-03-06T07:23:42.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.714+0000 7f706ac9d640 1 -- 172.21.15.83:0/2032520127 wait complete. 2024-03-06T07:23:42.727 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:23:42.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.828+0000 7fcac7d16640 1 Processor -- start 2024-03-06T07:23:42.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.828+0000 7fcac7d16640 1 -- start start 2024-03-06T07:23:42.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.829+0000 7fcac7d16640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fcac015f4f0 0x7fcac0066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.829+0000 7fcac7d16640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcac00b0db0 0x7fcac00ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.829+0000 7fcac7d16640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcac006b050 con 0x7fcac00ce580 2024-03-06T07:23:42.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.829+0000 7fcac7d16640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcac006b2c0 con 0x7fcac00b0db0 2024-03-06T07:23:42.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.829+0000 7fcac7d16640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcac00b0890 con 0x7fcac015f4f0 2024-03-06T07:23:42.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.829+0000 7fcac52b1640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcac00b0db0 0x7fcac00ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.829+0000 7fcac62b3640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fcac00ce580 0x7fcac0072550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43018/0 (socket says 172.21.15.83:43018) 2024-03-06T07:23:42.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.829+0000 7fcac62b3640 1 -- 172.21.15.83:0/1949911181 learned_addr learned my addr 172.21.15.83:0/1949911181 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:23:42.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.829+0000 7fcac5ab2640 1 --2- 172.21.15.83:0/1949911181 >> v2:172.21.15.183:3300/0 conn(0x7fcac015f4f0 0x7fcac0066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.829+0000 7fcac52b1640 1 -- 172.21.15.83:0/1949911181 >> v2:172.21.15.183:3300/0 conn(0x7fcac015f4f0 msgr2=0x7fcac0066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:42.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.829+0000 7fcac52b1640 1 --2- 172.21.15.83:0/1949911181 >> v2:172.21.15.183:3300/0 conn(0x7fcac015f4f0 0x7fcac0066d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.829+0000 7fcac52b1640 1 -- 172.21.15.83:0/1949911181 >> v1:172.21.15.83:6789/0 conn(0x7fcac00ce580 legacy=0x7fcac0072550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.829+0000 7fcac52b1640 1 -- 172.21.15.83:0/1949911181 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac00672e0 con 0x7fcac00b0db0 2024-03-06T07:23:42.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.830+0000 7fcac52b1640 1 --2- 172.21.15.83:0/1949911181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcac00b0db0 0x7fcac00ce040 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fcac00707a0 tx=0x7fcab002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b58278de3b939331 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:42.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.830+0000 7fcac4ab0640 1 -- 172.21.15.83:0/1949911181 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab0035020 con 0x7fcac00b0db0 2024-03-06T07:23:42.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.830+0000 7fcac4ab0640 1 -- 172.21.15.83:0/1949911181 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcab0031830 con 0x7fcac00b0db0 2024-03-06T07:23:42.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.830+0000 7fcac4ab0640 1 -- 172.21.15.83:0/1949911181 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab0031c00 con 0x7fcac00b0db0 2024-03-06T07:23:42.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.830+0000 7fcac7d16640 1 -- 172.21.15.83:0/1949911181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcac00b0db0 msgr2=0x7fcac00ce040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.830+0000 7fcac7d16640 1 --2- 172.21.15.83:0/1949911181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcac00b0db0 0x7fcac00ce040 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fcac00707a0 tx=0x7fcab002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.831+0000 7fcac7d16640 1 -- 172.21.15.83:0/1949911181 shutdown_connections 2024-03-06T07:23:42.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.831+0000 7fcac7d16640 1 --2- 172.21.15.83:0/1949911181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcac00b0db0 0x7fcac00ce040 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.831+0000 7fcac7d16640 1 --2- 172.21.15.83:0/1949911181 >> v2:172.21.15.183:3300/0 conn(0x7fcac015f4f0 0x7fcac0066d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.831+0000 7fcac7d16640 1 -- 172.21.15.83:0/1949911181 >> 172.21.15.83:0/1949911181 conn(0x7fcac00b1d70 msgr2=0x7fcac00b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:42.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.831+0000 7fcac7d16640 1 -- 172.21.15.83:0/1949911181 shutdown_connections 2024-03-06T07:23:42.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.831+0000 7fcac7d16640 1 -- 172.21.15.83:0/1949911181 wait complete. 2024-03-06T07:23:42.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.832+0000 7fcac7d16640 1 Processor -- start 2024-03-06T07:23:42.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.832+0000 7fcac7d16640 1 -- start start 2024-03-06T07:23:42.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.832+0000 7fcac7d16640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcac00b0db0 0x7fcac01f7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.832+0000 7fcac7d16640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcac00ce580 0x7fcac015c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.832+0000 7fcac7d16640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcac015f4f0 0x7fcac00c9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.832+0000 7fcac7d16640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac015f8d0 con 0x7fcac00ce580 2024-03-06T07:23:42.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.832+0000 7fcac7d16640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac006b5a0 con 0x7fcac00b0db0 2024-03-06T07:23:42.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.832+0000 7fcac7d16640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcac015b7d0 con 0x7fcac015f4f0 2024-03-06T07:23:42.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.833+0000 7fcac52b1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcac00ce580 0x7fcac015c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.833+0000 7fcac52b1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcac00ce580 0x7fcac015c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48342/0 (socket says 172.21.15.83:48342) 2024-03-06T07:23:42.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.833+0000 7fcac52b1640 1 -- 172.21.15.83:0/3516656570 learned_addr learned my addr 172.21.15.83:0/3516656570 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:42.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.833+0000 7fcac5ab2640 1 --2- 172.21.15.83:0/3516656570 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcac00b0db0 0x7fcac01f7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.833+0000 7fcac62b3640 1 --2- 172.21.15.83:0/3516656570 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcac015f4f0 0x7fcac00c9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.833+0000 7fcac52b1640 1 -- 172.21.15.83:0/3516656570 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcac015f4f0 msgr2=0x7fcac00c9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.833+0000 7fcac52b1640 1 --2- 172.21.15.83:0/3516656570 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcac015f4f0 0x7fcac00c9ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.833+0000 7fcac52b1640 1 -- 172.21.15.83:0/3516656570 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcac00b0db0 msgr2=0x7fcac01f7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.833+0000 7fcac52b1640 1 --2- 172.21.15.83:0/3516656570 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcac00b0db0 0x7fcac01f7590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.833+0000 7fcac52b1640 1 -- 172.21.15.83:0/3516656570 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac015abc0 con 0x7fcac00ce580 2024-03-06T07:23:42.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.833+0000 7fcac52b1640 1 --2- 172.21.15.83:0/3516656570 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcac00ce580 0x7fcac015c0f0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fcab0009890 tx=0x7fcab00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:42.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.833+0000 7fcab67fc640 1 -- 172.21.15.83:0/3516656570 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab0041070 con 0x7fcac00ce580 2024-03-06T07:23:42.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.833+0000 7fcab67fc640 1 -- 172.21.15.83:0/3516656570 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcab003aa30 con 0x7fcac00ce580 2024-03-06T07:23:42.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.833+0000 7fcac7d16640 1 -- 172.21.15.83:0/3516656570 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcac015aea0 con 0x7fcac00ce580 2024-03-06T07:23:42.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.833+0000 7fcab67fc640 1 -- 172.21.15.83:0/3516656570 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab0035040 con 0x7fcac00ce580 2024-03-06T07:23:42.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.833+0000 7fcac7d16640 1 -- 172.21.15.83:0/3516656570 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcac015b460 con 0x7fcac00ce580 2024-03-06T07:23:42.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.834+0000 7fcac7d16640 1 -- 172.21.15.83:0/3516656570 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcac009e3d0 con 0x7fcac00ce580 2024-03-06T07:23:42.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.835+0000 7fcab67fc640 1 -- 172.21.15.83:0/3516656570 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcab0038020 con 0x7fcac00ce580 2024-03-06T07:23:42.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.835+0000 7fcab67fc640 1 --2- 172.21.15.83:0/3516656570 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcaa406b4d0 0x7fcaa406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.835+0000 7fcab67fc640 1 -- 172.21.15.83:0/3516656570 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcab00abc20 con 0x7fcac00ce580 2024-03-06T07:23:42.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.835+0000 7fcac5ab2640 1 --2- 172.21.15.83:0/3516656570 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcaa406b4d0 0x7fcaa406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.835+0000 7fcac5ab2640 1 --2- 172.21.15.83:0/3516656570 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcaa406b4d0 0x7fcaa406d990 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fcabc006fd0 tx=0x7fcabc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:42.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:42.839+0000 7fcab67fc640 1 -- 172.21.15.83:0/3516656570 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcab007aaa0 con 0x7fcac00ce580 2024-03-06T07:23:43.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.045+0000 7fcac7d16640 1 -- 172.21.15.83:0/3516656570 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fcac015c7d0 con 0x7fcac00ce580 2024-03-06T07:23:43.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.098+0000 7fcab67fc640 1 -- 172.21.15.83:0/3516656570 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) v1 ==== 118+0+0 (secure 0 0 0) 0x7fcab007a3a0 con 0x7fcac00ce580 2024-03-06T07:23:43.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.101+0000 7fcac7d16640 1 -- 172.21.15.83:0/3516656570 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcaa406b4d0 msgr2=0x7fcaa406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:43.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.101+0000 7fcac7d16640 1 --2- 172.21.15.83:0/3516656570 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcaa406b4d0 0x7fcaa406d990 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fcabc006fd0 tx=0x7fcabc008040 comp rx=0 tx=0).stop 2024-03-06T07:23:43.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.101+0000 7fcac7d16640 1 -- 172.21.15.83:0/3516656570 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcac00ce580 msgr2=0x7fcac015c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:43.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.101+0000 7fcac7d16640 1 --2- 172.21.15.83:0/3516656570 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcac00ce580 0x7fcac015c0f0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fcab0009890 tx=0x7fcab00046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.101+0000 7fcac7d16640 1 -- 172.21.15.83:0/3516656570 shutdown_connections 2024-03-06T07:23:43.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.101+0000 7fcac7d16640 1 --2- 172.21.15.83:0/3516656570 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcaa406b4d0 0x7fcaa406d990 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.101+0000 7fcac7d16640 1 --2- 172.21.15.83:0/3516656570 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcac015f4f0 0x7fcac00c9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.101+0000 7fcac7d16640 1 --2- 172.21.15.83:0/3516656570 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcac00ce580 0x7fcac015c0f0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.101+0000 7fcac7d16640 1 --2- 172.21.15.83:0/3516656570 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcac00b0db0 0x7fcac01f7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.101+0000 7fcac7d16640 1 -- 172.21.15.83:0/3516656570 >> 172.21.15.83:0/3516656570 conn(0x7fcac00b1d70 msgr2=0x7fcac015daf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:43.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.102+0000 7fcac7d16640 1 -- 172.21.15.83:0/3516656570 shutdown_connections 2024-03-06T07:23:43.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.102+0000 7fcac7d16640 1 -- 172.21.15.83:0/3516656570 wait complete. 2024-03-06T07:23:43.114 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:23:43.114 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:23:43.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.217+0000 7f764a265640 1 Processor -- start 2024-03-06T07:23:43.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.217+0000 7f764a265640 1 -- start start 2024-03-06T07:23:43.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.218+0000 7f764a265640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f764415c050 0x7f764406f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:43.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.218+0000 7f764a265640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f76440d0af0 0x7f76440cce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:43.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.218+0000 7f764a265640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76440cd4b0 con 0x7f764406fb40 2024-03-06T07:23:43.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.218+0000 7f764a265640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f764406ff80 con 0x7f76440d0af0 2024-03-06T07:23:43.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.218+0000 7f764a265640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76440cd790 con 0x7f764415c050 2024-03-06T07:23:43.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.218+0000 7f7643fff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f764415c050 0x7f764406f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:43.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.218+0000 7f7648802640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f76440d0af0 0x7f76440cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:43.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.218+0000 7f7643fff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f764415c050 0x7f764406f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:60102/0 (socket says 172.21.15.83:60102) 2024-03-06T07:23:43.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.218+0000 7f7648802640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f76440d0af0 0x7f76440cce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:41206/0 (socket says 172.21.15.83:41206) 2024-03-06T07:23:43.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.218+0000 7f7643fff640 1 -- 172.21.15.83:0/935389153 learned_addr learned my addr 172.21.15.83:0/935389153 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:43.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.218+0000 7f76437fe640 1 --1- 172.21.15.83:0/935389153 >> v1:172.21.15.183:6789/0 conn(0x7f764406fb40 0x7f76440cc760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:42206/0 (socket says 172.21.15.83:42206) 2024-03-06T07:23:43.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.218+0000 7f7643fff640 1 -- 172.21.15.83:0/935389153 >> v2:172.21.15.183:3300/0 conn(0x7f76440d0af0 msgr2=0x7f76440cce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:43.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.218+0000 7f7643fff640 1 --2- 172.21.15.83:0/935389153 >> v2:172.21.15.183:3300/0 conn(0x7f76440d0af0 0x7f76440cce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.218+0000 7f7643fff640 1 -- 172.21.15.83:0/935389153 >> v1:172.21.15.183:6789/0 conn(0x7f764406fb40 legacy=0x7f76440cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:43.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.218+0000 7f7643fff640 1 -- 172.21.15.83:0/935389153 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76440cda70 con 0x7f764415c050 2024-03-06T07:23:43.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.219+0000 7f7648802640 1 --2- 172.21.15.83:0/935389153 >> v2:172.21.15.183:3300/0 conn(0x7f76440d0af0 0x7f76440cce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:43.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.219+0000 7f7643fff640 1 --2- 172.21.15.83:0/935389153 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f764415c050 0x7f764406f600 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7634009870 tx=0x7f763402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=555b2323cf54f28f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:43.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.219+0000 7f7642ffd640 1 -- 172.21.15.83:0/935389153 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7634035020 con 0x7f764415c050 2024-03-06T07:23:43.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.219+0000 7f7642ffd640 1 -- 172.21.15.83:0/935389153 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7634031640 con 0x7f764415c050 2024-03-06T07:23:43.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.219+0000 7f7642ffd640 1 -- 172.21.15.83:0/935389153 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f763402d040 con 0x7f764415c050 2024-03-06T07:23:43.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.220+0000 7f764a265640 1 -- 172.21.15.83:0/935389153 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f764415c050 msgr2=0x7f764406f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:43.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.220+0000 7f764a265640 1 --2- 172.21.15.83:0/935389153 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f764415c050 0x7f764406f600 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7634009870 tx=0x7f763402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.220+0000 7f764a265640 1 -- 172.21.15.83:0/935389153 shutdown_connections 2024-03-06T07:23:43.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.220+0000 7f764a265640 1 --2- 172.21.15.83:0/935389153 >> v2:172.21.15.183:3300/0 conn(0x7f76440d0af0 0x7f76440cce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.220+0000 7f764a265640 1 --2- 172.21.15.83:0/935389153 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f764415c050 0x7f764406f600 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.220+0000 7f764a265640 1 -- 172.21.15.83:0/935389153 >> 172.21.15.83:0/935389153 conn(0x7f76440b3800 msgr2=0x7f76440b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:43.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.220+0000 7f764a265640 1 -- 172.21.15.83:0/935389153 shutdown_connections 2024-03-06T07:23:43.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.220+0000 7f764a265640 1 -- 172.21.15.83:0/935389153 wait complete. 2024-03-06T07:23:43.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.221+0000 7f764a265640 1 Processor -- start 2024-03-06T07:23:43.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.221+0000 7f764a265640 1 -- start start 2024-03-06T07:23:43.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.222+0000 7f764a265640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f764406fb40 0x7f76441496f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:43.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.222+0000 7f764a265640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f76440d0af0 0x7f76440b48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:43.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.222+0000 7f764a265640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f764415c050 0x7f764409dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:43.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.222+0000 7f764a265640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7644165c30 con 0x7f764415c050 2024-03-06T07:23:43.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.222+0000 7f764a265640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7644165ab0 con 0x7f764406fb40 2024-03-06T07:23:43.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.222+0000 7f764a265640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7644165db0 con 0x7f76440d0af0 2024-03-06T07:23:43.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.222+0000 7f7648802640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f764415c050 0x7f764409dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:43.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.222+0000 7f7648802640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f764415c050 0x7f764409dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48356/0 (socket says 172.21.15.83:48356) 2024-03-06T07:23:43.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.222+0000 7f7648802640 1 -- 172.21.15.83:0/3763498292 learned_addr learned my addr 172.21.15.83:0/3763498292 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:43.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.222+0000 7f76437fe640 1 --2- 172.21.15.83:0/3763498292 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f76440d0af0 0x7f76440b48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:43.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.222+0000 7f7643fff640 1 --2- 172.21.15.83:0/3763498292 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f764406fb40 0x7f76441496f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:43.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.223+0000 7f7648802640 1 -- 172.21.15.83:0/3763498292 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f76440d0af0 msgr2=0x7f76440b48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:43.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.223+0000 7f7648802640 1 --2- 172.21.15.83:0/3763498292 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f76440d0af0 0x7f76440b48f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.223+0000 7f7648802640 1 -- 172.21.15.83:0/3763498292 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f764406fb40 msgr2=0x7f76441496f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:43.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.223+0000 7f7648802640 1 --2- 172.21.15.83:0/3763498292 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f764406fb40 0x7f76441496f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.223+0000 7f7648802640 1 -- 172.21.15.83:0/3763498292 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76440b6290 con 0x7f764415c050 2024-03-06T07:23:43.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.223+0000 7f76437fe640 1 --2- 172.21.15.83:0/3763498292 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f76440d0af0 0x7f76440b48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:43.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.223+0000 7f7643fff640 1 --2- 172.21.15.83:0/3763498292 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f764406fb40 0x7f76441496f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:43.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.223+0000 7f7648802640 1 --2- 172.21.15.83:0/3763498292 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f764415c050 0x7f764409dc10 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f763c00ef90 tx=0x7f763c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:43.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.223+0000 7f7640ff9640 1 -- 172.21.15.83:0/3763498292 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f763c018020 con 0x7f764415c050 2024-03-06T07:23:43.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.223+0000 7f7640ff9640 1 -- 172.21.15.83:0/3763498292 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f763c014480 con 0x7f764415c050 2024-03-06T07:23:43.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.223+0000 7f764a265640 1 -- 172.21.15.83:0/3763498292 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76440b51f0 con 0x7f764415c050 2024-03-06T07:23:43.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.223+0000 7f7640ff9640 1 -- 172.21.15.83:0/3763498292 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f763c014850 con 0x7f764415c050 2024-03-06T07:23:43.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.223+0000 7f764a265640 1 -- 172.21.15.83:0/3763498292 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76440b57b0 con 0x7f764415c050 2024-03-06T07:23:43.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.224+0000 7f764a265640 1 -- 172.21.15.83:0/3763498292 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f764409fe60 con 0x7f764415c050 2024-03-06T07:23:43.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.225+0000 7f7640ff9640 1 -- 172.21.15.83:0/3763498292 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f763c005d10 con 0x7f764415c050 2024-03-06T07:23:43.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.225+0000 7f7640ff9640 1 --2- 172.21.15.83:0/3763498292 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f763006b4d0 0x7f763006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:43.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.225+0000 7f7640ff9640 1 -- 172.21.15.83:0/3763498292 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f763c08ae90 con 0x7f764415c050 2024-03-06T07:23:43.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.225+0000 7f7643fff640 1 --2- 172.21.15.83:0/3763498292 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f763006b4d0 0x7f763006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:43.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.226+0000 7f7643fff640 1 --2- 172.21.15.83:0/3763498292 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f763006b4d0 0x7f763006d990 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f7634036000 tx=0x7f763403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:43.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.230+0000 7f7640ff9640 1 -- 172.21.15.83:0/3763498292 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f763c022030 con 0x7f764415c050 2024-03-06T07:23:43.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:43.444+0000 7f764a265640 1 -- 172.21.15.83:0/3763498292 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f76440b5b50 con 0x7f764415c050 2024-03-06T07:23:44.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.103+0000 7f7640ff9640 1 -- 172.21.15.83:0/3763498292 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v19) v1 ==== 122+0+0 (secure 0 0 0) 0x7f763c010040 con 0x7f764415c050 2024-03-06T07:23:44.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.105+0000 7f764a265640 1 -- 172.21.15.83:0/3763498292 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f763006b4d0 msgr2=0x7f763006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.105+0000 7f764a265640 1 --2- 172.21.15.83:0/3763498292 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f763006b4d0 0x7f763006d990 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f7634036000 tx=0x7f763403c040 comp rx=0 tx=0).stop 2024-03-06T07:23:44.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.106+0000 7f764a265640 1 -- 172.21.15.83:0/3763498292 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f764415c050 msgr2=0x7f764409dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.106+0000 7f764a265640 1 --2- 172.21.15.83:0/3763498292 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f764415c050 0x7f764409dc10 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f763c00ef90 tx=0x7f763c00c490 comp rx=0 tx=0).stop 2024-03-06T07:23:44.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.106+0000 7f764a265640 1 -- 172.21.15.83:0/3763498292 shutdown_connections 2024-03-06T07:23:44.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.106+0000 7f764a265640 1 --2- 172.21.15.83:0/3763498292 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f763006b4d0 0x7f763006d990 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.106+0000 7f764a265640 1 --2- 172.21.15.83:0/3763498292 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f764415c050 0x7f764409dc10 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.106+0000 7f764a265640 1 --2- 172.21.15.83:0/3763498292 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f76440d0af0 0x7f76440b48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.106+0000 7f764a265640 1 --2- 172.21.15.83:0/3763498292 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f764406fb40 0x7f76441496f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.106+0000 7f764a265640 1 -- 172.21.15.83:0/3763498292 >> 172.21.15.83:0/3763498292 conn(0x7f76440b3800 msgr2=0x7f76440b1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:44.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.106+0000 7f764a265640 1 -- 172.21.15.83:0/3763498292 shutdown_connections 2024-03-06T07:23:44.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.106+0000 7f764a265640 1 -- 172.21.15.83:0/3763498292 wait complete. 2024-03-06T07:23:44.118 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:23:44.118 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:44.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.218+0000 7fcd699b6640 1 Processor -- start 2024-03-06T07:23:44.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.219+0000 7fcd699b6640 1 -- start start 2024-03-06T07:23:44.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.219+0000 7fcd699b6640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fcd640d4760 0x7fcd640b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.219+0000 7fcd699b6640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcd640d0a80 0x7fcd640cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.219+0000 7fcd699b6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd640d25d0 con 0x7fcd640b4170 2024-03-06T07:23:44.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.219+0000 7fcd699b6640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd640d4bf0 con 0x7fcd640d0a80 2024-03-06T07:23:44.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.219+0000 7fcd699b6640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd640ce9e0 con 0x7fcd640d4760 2024-03-06T07:23:44.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.220+0000 7fcd637fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcd640d0a80 0x7fcd640cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.220+0000 7fcd62ffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fcd640b4170 0x7fcd640d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43022/0 (socket says 172.21.15.83:43022) 2024-03-06T07:23:44.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.220+0000 7fcd62ffd640 1 -- 172.21.15.83:0/2229202505 learned_addr learned my addr 172.21.15.83:0/2229202505 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:23:44.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.220+0000 7fcd627fc640 1 --2- 172.21.15.83:0/2229202505 >> v2:172.21.15.183:3300/0 conn(0x7fcd640d4760 0x7fcd640b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.220+0000 7fcd637fe640 1 -- 172.21.15.83:0/2229202505 >> v2:172.21.15.183:3300/0 conn(0x7fcd640d4760 msgr2=0x7fcd640b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:44.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.220+0000 7fcd637fe640 1 --2- 172.21.15.83:0/2229202505 >> v2:172.21.15.183:3300/0 conn(0x7fcd640d4760 0x7fcd640b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.220+0000 7fcd637fe640 1 -- 172.21.15.83:0/2229202505 >> v1:172.21.15.83:6789/0 conn(0x7fcd640b4170 legacy=0x7fcd640d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.220+0000 7fcd61ffb640 1 -- 172.21.15.83:0/2229202505 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1433337573 0 0) 0x7fcd640d25d0 con 0x7fcd640b4170 2024-03-06T07:23:44.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.220+0000 7fcd637fe640 1 -- 172.21.15.83:0/2229202505 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd640d2870 con 0x7fcd640d0a80 2024-03-06T07:23:44.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.221+0000 7fcd637fe640 1 --2- 172.21.15.83:0/2229202505 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd640d0a80 0x7fcd640cee70 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fcd64067920 tx=0x7fcd5402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=909934513210f247 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:44.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.221+0000 7fcd61ffb640 1 -- 172.21.15.83:0/2229202505 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd54039070 con 0x7fcd640d0a80 2024-03-06T07:23:44.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.221+0000 7fcd61ffb640 1 -- 172.21.15.83:0/2229202505 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd5402cc80 con 0x7fcd640d0a80 2024-03-06T07:23:44.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.221+0000 7fcd61ffb640 1 -- 172.21.15.83:0/2229202505 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd54034660 con 0x7fcd640d0a80 2024-03-06T07:23:44.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.221+0000 7fcd699b6640 1 -- 172.21.15.83:0/2229202505 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd640d0a80 msgr2=0x7fcd640cee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.221+0000 7fcd699b6640 1 --2- 172.21.15.83:0/2229202505 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd640d0a80 0x7fcd640cee70 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fcd64067920 tx=0x7fcd5402bed0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.222+0000 7fcd699b6640 1 -- 172.21.15.83:0/2229202505 shutdown_connections 2024-03-06T07:23:44.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.222+0000 7fcd699b6640 1 --2- 172.21.15.83:0/2229202505 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd640d0a80 0x7fcd640cee70 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.222+0000 7fcd699b6640 1 --2- 172.21.15.83:0/2229202505 >> v2:172.21.15.183:3300/0 conn(0x7fcd640d4760 0x7fcd640b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.222+0000 7fcd699b6640 1 -- 172.21.15.83:0/2229202505 >> 172.21.15.83:0/2229202505 conn(0x7fcd640d3c40 msgr2=0x7fcd640d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:44.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.222+0000 7fcd699b6640 1 -- 172.21.15.83:0/2229202505 shutdown_connections 2024-03-06T07:23:44.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.222+0000 7fcd699b6640 1 -- 172.21.15.83:0/2229202505 wait complete. 2024-03-06T07:23:44.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.223+0000 7fcd699b6640 1 Processor -- start 2024-03-06T07:23:44.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.223+0000 7fcd699b6640 1 -- start start 2024-03-06T07:23:44.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.224+0000 7fcd699b6640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcd640b4170 0x7fcd641f7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.224+0000 7fcd699b6640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcd640d0a80 0x7fcd640c9e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.224+0000 7fcd699b6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd640cc370 0x7fcd6414dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.224+0000 7fcd699b6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd640b4dd0 con 0x7fcd640cc370 2024-03-06T07:23:44.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.224+0000 7fcd699b6640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd640b4550 con 0x7fcd640b4170 2024-03-06T07:23:44.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.224+0000 7fcd699b6640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd640b4f50 con 0x7fcd640d0a80 2024-03-06T07:23:44.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.224+0000 7fcd637fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd640cc370 0x7fcd6414dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.224+0000 7fcd637fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd640cc370 0x7fcd6414dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48378/0 (socket says 172.21.15.83:48378) 2024-03-06T07:23:44.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.224+0000 7fcd637fe640 1 -- 172.21.15.83:0/3783188022 learned_addr learned my addr 172.21.15.83:0/3783188022 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:44.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.224+0000 7fcd627fc640 1 --2- 172.21.15.83:0/3783188022 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcd640d0a80 0x7fcd640c9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.224+0000 7fcd62ffd640 1 --2- 172.21.15.83:0/3783188022 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcd640b4170 0x7fcd641f7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.224+0000 7fcd637fe640 1 -- 172.21.15.83:0/3783188022 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcd640d0a80 msgr2=0x7fcd640c9e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.224+0000 7fcd637fe640 1 --2- 172.21.15.83:0/3783188022 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcd640d0a80 0x7fcd640c9e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.224+0000 7fcd637fe640 1 -- 172.21.15.83:0/3783188022 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcd640b4170 msgr2=0x7fcd641f7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.224+0000 7fcd637fe640 1 --2- 172.21.15.83:0/3783188022 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcd640b4170 0x7fcd641f7db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.224+0000 7fcd637fe640 1 -- 172.21.15.83:0/3783188022 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd641f82f0 con 0x7fcd640cc370 2024-03-06T07:23:44.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.224+0000 7fcd637fe640 1 --2- 172.21.15.83:0/3783188022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd640cc370 0x7fcd6414dc50 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fcd54037040 tx=0x7fcd54004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:44.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.224+0000 7fcd4bfff640 1 -- 172.21.15.83:0/3783188022 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd54039050 con 0x7fcd640cc370 2024-03-06T07:23:44.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.224+0000 7fcd4bfff640 1 -- 172.21.15.83:0/3783188022 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd54045070 con 0x7fcd640cc370 2024-03-06T07:23:44.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.224+0000 7fcd699b6640 1 -- 172.21.15.83:0/3783188022 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd640cb2d0 con 0x7fcd640cc370 2024-03-06T07:23:44.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.224+0000 7fcd4bfff640 1 -- 172.21.15.83:0/3783188022 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd5403e830 con 0x7fcd640cc370 2024-03-06T07:23:44.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.224+0000 7fcd699b6640 1 -- 172.21.15.83:0/3783188022 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd640cb890 con 0x7fcd640cc370 2024-03-06T07:23:44.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.226+0000 7fcd699b6640 1 -- 172.21.15.83:0/3783188022 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd6404f330 con 0x7fcd640cc370 2024-03-06T07:23:44.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.226+0000 7fcd4bfff640 1 -- 172.21.15.83:0/3783188022 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcd5403e9d0 con 0x7fcd640cc370 2024-03-06T07:23:44.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.226+0000 7fcd4bfff640 1 --2- 172.21.15.83:0/3783188022 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcd5006b4d0 0x7fcd5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.227+0000 7fcd62ffd640 1 --2- 172.21.15.83:0/3783188022 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcd5006b4d0 0x7fcd5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.227+0000 7fcd4bfff640 1 -- 172.21.15.83:0/3783188022 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcd540acf10 con 0x7fcd640cc370 2024-03-06T07:23:44.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.227+0000 7fcd62ffd640 1 --2- 172.21.15.83:0/3783188022 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcd5006b4d0 0x7fcd5006d990 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c000f60 tx=0x7fcd5c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:44.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.231+0000 7fcd4bfff640 1 -- 172.21.15.83:0/3783188022 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcd641f82f0 con 0x7fcd640cc370 2024-03-06T07:23:44.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.541+0000 7fcd699b6640 1 -- 172.21.15.83:0/3783188022 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd640733b0 con 0x7fcd640cc370 2024-03-06T07:23:44.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.542+0000 7fcd4bfff640 1 -- 172.21.15.83:0/3783188022 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+4980 (secure 0 0 0) 0x7fcd5407bf40 con 0x7fcd640cc370 2024-03-06T07:23:44.544 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:23:44.544 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":19,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5395,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/2270670914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2270670914},{"type":"v1","addr":"172.21.15.83:6837","nonce":2270670914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5398,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/2359880029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2359880029},{"type":"v1","addr":"172.21.15.183:6834","nonce":2359880029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5401,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3099809273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":3099809273},{"type":"v1","addr":"172.21.15.183:6835","nonce":3099809273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:41.430159+0000","modified":"2024-03-06T07:23:43.445803+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5706},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5706":{"gid":5706,"name":"a","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6835/534553538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":534553538},{"type":"v1","addr":"172.21.15.83:6835","nonce":534553538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5706,"qdb_cluster":[5706]},"id":3}]} 2024-03-06T07:23:44.544 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 19 2024-03-06T07:23:44.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.544+0000 7fcd699b6640 1 -- 172.21.15.83:0/3783188022 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcd5006b4d0 msgr2=0x7fcd5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.544+0000 7fcd699b6640 1 --2- 172.21.15.83:0/3783188022 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcd5006b4d0 0x7fcd5006d990 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c000f60 tx=0x7fcd5c009040 comp rx=0 tx=0).stop 2024-03-06T07:23:44.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.544+0000 7fcd699b6640 1 -- 172.21.15.83:0/3783188022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd640cc370 msgr2=0x7fcd6414dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.544+0000 7fcd699b6640 1 --2- 172.21.15.83:0/3783188022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd640cc370 0x7fcd6414dc50 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fcd54037040 tx=0x7fcd54004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:44.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.544+0000 7fcd699b6640 1 -- 172.21.15.83:0/3783188022 shutdown_connections 2024-03-06T07:23:44.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.544+0000 7fcd699b6640 1 --2- 172.21.15.83:0/3783188022 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcd5006b4d0 0x7fcd5006d990 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.544+0000 7fcd699b6640 1 --2- 172.21.15.83:0/3783188022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd640cc370 0x7fcd6414dc50 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.544+0000 7fcd699b6640 1 --2- 172.21.15.83:0/3783188022 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcd640d0a80 0x7fcd640c9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.544+0000 7fcd699b6640 1 --2- 172.21.15.83:0/3783188022 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcd640b4170 0x7fcd641f7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.544+0000 7fcd699b6640 1 -- 172.21.15.83:0/3783188022 >> 172.21.15.83:0/3783188022 conn(0x7fcd640d3c40 msgr2=0x7fcd640b8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:44.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.544+0000 7fcd699b6640 1 -- 172.21.15.83:0/3783188022 shutdown_connections 2024-03-06T07:23:44.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.544+0000 7fcd699b6640 1 -- 172.21.15.83:0/3783188022 wait complete. 2024-03-06T07:23:44.557 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:44.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.656+0000 7f8b9ec88640 1 Processor -- start 2024-03-06T07:23:44.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.656+0000 7f8b9ec88640 1 -- start start 2024-03-06T07:23:44.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.657+0000 7f8b9ec88640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f8b980d22b0 0x7f8b980d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.657+0000 7f8b9ec88640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f8b980b41f0 0x7f8b980b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.657+0000 7f8b9ec88640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b980b3430 con 0x7f8b9815dfe0 2024-03-06T07:23:44.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.657+0000 7f8b9ec88640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b980d2c50 con 0x7f8b980b41f0 2024-03-06T07:23:44.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.657+0000 7f8b9ec88640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b980b31e0 con 0x7f8b980d22b0 2024-03-06T07:23:44.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.657+0000 7f8b9d225640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f8b980b41f0 0x7f8b980b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.657+0000 7f8b9d225640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f8b980b41f0 0x7f8b980b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48390/0 (socket says 172.21.15.83:48390) 2024-03-06T07:23:44.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.657+0000 7f8b9d225640 1 -- 172.21.15.83:0/347368409 learned_addr learned my addr 172.21.15.83:0/347368409 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:44.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.657+0000 7f8b9d225640 1 -- 172.21.15.83:0/347368409 >> v2:172.21.15.183:3300/0 conn(0x7f8b980d22b0 msgr2=0x7f8b980d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:44.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.657+0000 7f8b9d225640 1 --2- 172.21.15.83:0/347368409 >> v2:172.21.15.183:3300/0 conn(0x7f8b980d22b0 0x7f8b980d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.657+0000 7f8b9d225640 1 -- 172.21.15.83:0/347368409 >> v1:172.21.15.183:6789/0 conn(0x7f8b9815dfe0 legacy=0x7f8b980cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:44.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.657+0000 7f8b9d225640 1 -- 172.21.15.83:0/347368409 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b980b3710 con 0x7f8b980b41f0 2024-03-06T07:23:44.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.658+0000 7f8b9d225640 1 --2- 172.21.15.83:0/347368409 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8b980b41f0 0x7f8b980b4680 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f8b80009870 tx=0x7f8b8002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba58a7bdbb798342 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:44.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.658+0000 7f8b977fe640 1 -- 172.21.15.83:0/347368409 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b80035020 con 0x7f8b980b41f0 2024-03-06T07:23:44.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.658+0000 7f8b977fe640 1 -- 172.21.15.83:0/347368409 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b80031870 con 0x7f8b980b41f0 2024-03-06T07:23:44.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.658+0000 7f8b977fe640 1 -- 172.21.15.83:0/347368409 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b80031c40 con 0x7f8b980b41f0 2024-03-06T07:23:44.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.659+0000 7f8b9ec88640 1 -- 172.21.15.83:0/347368409 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8b980b41f0 msgr2=0x7f8b980b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.659+0000 7f8b9ec88640 1 --2- 172.21.15.83:0/347368409 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8b980b41f0 0x7f8b980b4680 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f8b80009870 tx=0x7f8b8002b910 comp rx=0 tx=0).stop 2024-03-06T07:23:44.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.659+0000 7f8b9ec88640 1 -- 172.21.15.83:0/347368409 shutdown_connections 2024-03-06T07:23:44.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.659+0000 7f8b9ec88640 1 --2- 172.21.15.83:0/347368409 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8b980b41f0 0x7f8b980b4680 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.659+0000 7f8b9ec88640 1 --2- 172.21.15.83:0/347368409 >> v2:172.21.15.183:3300/0 conn(0x7f8b980d22b0 0x7f8b980d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.659+0000 7f8b9ec88640 1 -- 172.21.15.83:0/347368409 >> 172.21.15.83:0/347368409 conn(0x7f8b980d3cd0 msgr2=0x7f8b980d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:44.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.659+0000 7f8b9ec88640 1 -- 172.21.15.83:0/347368409 shutdown_connections 2024-03-06T07:23:44.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.659+0000 7f8b9ec88640 1 -- 172.21.15.83:0/347368409 wait complete. 2024-03-06T07:23:44.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.660+0000 7f8b9ec88640 1 Processor -- start 2024-03-06T07:23:44.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.660+0000 7f8b9ec88640 1 -- start start 2024-03-06T07:23:44.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.661+0000 7f8b9ec88640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8b980b41f0 0x7f8b9815e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.661+0000 7f8b9ec88640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8b980d22b0 0x7f8b980b7a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.661+0000 7f8b9ec88640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8b9815dfe0 0x7f8b98150000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.661+0000 7f8b9ec88640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b980cfa00 con 0x7f8b9815dfe0 2024-03-06T07:23:44.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.661+0000 7f8b9ec88640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b980b3d60 con 0x7f8b980d22b0 2024-03-06T07:23:44.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.661+0000 7f8b9ec88640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b980d1c70 con 0x7f8b980b41f0 2024-03-06T07:23:44.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.661+0000 7f8b9d225640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8b9815dfe0 0x7f8b98150000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.661+0000 7f8b9d225640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8b9815dfe0 0x7f8b98150000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48398/0 (socket says 172.21.15.83:48398) 2024-03-06T07:23:44.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.661+0000 7f8b9d225640 1 -- 172.21.15.83:0/2344215569 learned_addr learned my addr 172.21.15.83:0/2344215569 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:44.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.661+0000 7f8b97fff640 1 --2- 172.21.15.83:0/2344215569 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8b980d22b0 0x7f8b980b7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.661+0000 7f8b9ca24640 1 --2- 172.21.15.83:0/2344215569 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8b980b41f0 0x7f8b9815e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.661+0000 7f8b9d225640 1 -- 172.21.15.83:0/2344215569 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8b980b41f0 msgr2=0x7f8b9815e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.661+0000 7f8b9d225640 1 --2- 172.21.15.83:0/2344215569 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8b980b41f0 0x7f8b9815e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.661+0000 7f8b9d225640 1 -- 172.21.15.83:0/2344215569 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8b980d22b0 msgr2=0x7f8b980b7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.661+0000 7f8b9d225640 1 --2- 172.21.15.83:0/2344215569 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8b980d22b0 0x7f8b980b7a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.661+0000 7f8b9d225640 1 -- 172.21.15.83:0/2344215569 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b9815eaa0 con 0x7f8b9815dfe0 2024-03-06T07:23:44.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.661+0000 7f8b9ca24640 1 --2- 172.21.15.83:0/2344215569 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8b980b41f0 0x7f8b9815e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:44.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.661+0000 7f8b9d225640 1 --2- 172.21.15.83:0/2344215569 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8b9815dfe0 0x7f8b98150000 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f8b8002b950 tx=0x7f8b800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:44.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.661+0000 7f8b957fa640 1 -- 172.21.15.83:0/2344215569 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b800443f0 con 0x7f8b9815dfe0 2024-03-06T07:23:44.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.661+0000 7f8b957fa640 1 -- 172.21.15.83:0/2344215569 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b800445b0 con 0x7f8b9815dfe0 2024-03-06T07:23:44.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.661+0000 7f8b9ec88640 1 -- 172.21.15.83:0/2344215569 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b9815ed80 con 0x7f8b9815dfe0 2024-03-06T07:23:44.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.661+0000 7f8b957fa640 1 -- 172.21.15.83:0/2344215569 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b800449a0 con 0x7f8b9815dfe0 2024-03-06T07:23:44.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.662+0000 7f8b9ec88640 1 -- 172.21.15.83:0/2344215569 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b9815f340 con 0x7f8b9815dfe0 2024-03-06T07:23:44.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.663+0000 7f8b9ec88640 1 -- 172.21.15.83:0/2344215569 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b9804f330 con 0x7f8b9815dfe0 2024-03-06T07:23:44.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.663+0000 7f8b957fa640 1 -- 172.21.15.83:0/2344215569 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8b8004b060 con 0x7f8b9815dfe0 2024-03-06T07:23:44.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.663+0000 7f8b957fa640 1 --2- 172.21.15.83:0/2344215569 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8b7806b500 0x7f8b7806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.663+0000 7f8b957fa640 1 -- 172.21.15.83:0/2344215569 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8b800ac3d0 con 0x7f8b9815dfe0 2024-03-06T07:23:44.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.663+0000 7f8b9ca24640 1 --2- 172.21.15.83:0/2344215569 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8b7806b500 0x7f8b7806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.664+0000 7f8b9ca24640 1 --2- 172.21.15.83:0/2344215569 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8b7806b500 0x7f8b7806d9c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f8b90007eb0 tx=0x7f8b900073c0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:44.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.667+0000 7f8b957fa640 1 -- 172.21.15.83:0/2344215569 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8b8003f070 con 0x7f8b9815dfe0 2024-03-06T07:23:44.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.845+0000 7f8b9ec88640 1 -- 172.21.15.83:0/2344215569 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8b980d4940 con 0x7f8b9815dfe0 2024-03-06T07:23:44.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.846+0000 7f8b957fa640 1 -- 172.21.15.83:0/2344215569 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+16625 (secure 0 0 0) 0x7f8b8007b250 con 0x7f8b9815dfe0 2024-03-06T07:23:44.848 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:23:44.848 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":45,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:23:43.092917+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:39.656617+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:40.402765+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:44.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.848+0000 7f8b9ec88640 1 -- 172.21.15.83:0/2344215569 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8b7806b500 msgr2=0x7f8b7806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.848+0000 7f8b9ec88640 1 --2- 172.21.15.83:0/2344215569 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8b7806b500 0x7f8b7806d9c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f8b90007eb0 tx=0x7f8b900073c0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.848+0000 7f8b9ec88640 1 -- 172.21.15.83:0/2344215569 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8b9815dfe0 msgr2=0x7f8b98150000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.848+0000 7f8b9ec88640 1 --2- 172.21.15.83:0/2344215569 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8b9815dfe0 0x7f8b98150000 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f8b8002b950 tx=0x7f8b800046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.848+0000 7f8b9ec88640 1 -- 172.21.15.83:0/2344215569 shutdown_connections 2024-03-06T07:23:44.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.848+0000 7f8b9ec88640 1 --2- 172.21.15.83:0/2344215569 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8b7806b500 0x7f8b7806d9c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.848+0000 7f8b9ec88640 1 --2- 172.21.15.83:0/2344215569 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8b9815dfe0 0x7f8b98150000 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.848+0000 7f8b9ec88640 1 --2- 172.21.15.83:0/2344215569 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8b980d22b0 0x7f8b980b7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.848+0000 7f8b9ec88640 1 --2- 172.21.15.83:0/2344215569 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8b980b41f0 0x7f8b9815e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.848+0000 7f8b9ec88640 1 -- 172.21.15.83:0/2344215569 >> 172.21.15.83:0/2344215569 conn(0x7f8b980d3cd0 msgr2=0x7f8b9815d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:44.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.848+0000 7f8b9ec88640 1 -- 172.21.15.83:0/2344215569 shutdown_connections 2024-03-06T07:23:44.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.848+0000 7f8b9ec88640 1 -- 172.21.15.83:0/2344215569 wait complete. 2024-03-06T07:23:44.861 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:23:44.861 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:23:44.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.959+0000 7f261439a640 1 Processor -- start 2024-03-06T07:23:44.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.959+0000 7f261439a640 1 -- start start 2024-03-06T07:23:44.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.960+0000 7f261439a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f260c15bb20 0x7f260c0d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.960+0000 7f261439a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f260c0cfff0 0x7f260c0d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.960+0000 7f261439a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f260c161980 con 0x7f260c0ce4b0 2024-03-06T07:23:44.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.960+0000 7f261439a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f260c161c60 con 0x7f260c15bb20 2024-03-06T07:23:44.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.960+0000 7f261439a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f260c161f40 con 0x7f260c0cfff0 2024-03-06T07:23:44.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.961+0000 7f2612136640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f260c15bb20 0x7f260c0d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.961+0000 7f2612136640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f260c15bb20 0x7f260c0d1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48410/0 (socket says 172.21.15.83:48410) 2024-03-06T07:23:44.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.961+0000 7f2612136640 1 -- 172.21.15.83:0/3170367152 learned_addr learned my addr 172.21.15.83:0/3170367152 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:44.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.961+0000 7f2611935640 1 --2- 172.21.15.83:0/3170367152 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f260c0cfff0 0x7f260c0d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.961+0000 7f2612136640 1 -- 172.21.15.83:0/3170367152 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f260c0cfff0 msgr2=0x7f260c0d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:44.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.961+0000 7f2612136640 1 --2- 172.21.15.83:0/3170367152 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f260c0cfff0 0x7f260c0d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.961+0000 7f2612136640 1 -- 172.21.15.83:0/3170367152 >> v1:172.21.15.183:6789/0 conn(0x7f260c0ce4b0 legacy=0x7f260c0ce940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:44.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.961+0000 7f2612136640 1 -- 172.21.15.83:0/3170367152 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f260c162220 con 0x7f260c15bb20 2024-03-06T07:23:44.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.962+0000 7f2612136640 1 --2- 172.21.15.83:0/3170367152 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f260c15bb20 0x7f260c0d1df0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f2604009e50 tx=0x7f260402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c78792300d09e08 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:44.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.962+0000 7f2611134640 1 -- 172.21.15.83:0/3170367152 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f260402ca70 con 0x7f260c15bb20 2024-03-06T07:23:44.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.962+0000 7f2611134640 1 -- 172.21.15.83:0/3170367152 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f260402cc10 con 0x7f260c15bb20 2024-03-06T07:23:44.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.962+0000 7f2611134640 1 -- 172.21.15.83:0/3170367152 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26040343f0 con 0x7f260c15bb20 2024-03-06T07:23:44.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.963+0000 7f261439a640 1 -- 172.21.15.83:0/3170367152 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f260c15bb20 msgr2=0x7f260c0d1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.963+0000 7f261439a640 1 --2- 172.21.15.83:0/3170367152 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f260c15bb20 0x7f260c0d1df0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f2604009e50 tx=0x7f260402bed0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.963+0000 7f261439a640 1 -- 172.21.15.83:0/3170367152 shutdown_connections 2024-03-06T07:23:44.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.963+0000 7f261439a640 1 --2- 172.21.15.83:0/3170367152 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f260c0cfff0 0x7f260c0d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.963+0000 7f261439a640 1 --2- 172.21.15.83:0/3170367152 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f260c15bb20 0x7f260c0d1df0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.963+0000 7f261439a640 1 -- 172.21.15.83:0/3170367152 >> 172.21.15.83:0/3170367152 conn(0x7f260c09a9f0 msgr2=0x7f260c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:44.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.963+0000 7f261439a640 1 -- 172.21.15.83:0/3170367152 shutdown_connections 2024-03-06T07:23:44.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.963+0000 7f261439a640 1 -- 172.21.15.83:0/3170367152 wait complete. 2024-03-06T07:23:44.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.964+0000 7f261439a640 1 Processor -- start 2024-03-06T07:23:44.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.964+0000 7f261439a640 1 -- start start 2024-03-06T07:23:44.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.965+0000 7f261439a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f260c0ce4b0 0x7f260c146e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.965+0000 7f261439a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f260c0cfff0 0x7f260c0b2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.965+0000 7f261439a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f260c15bb20 0x7f260c0b2560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.965+0000 7f261439a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f260c0d24b0 con 0x7f260c0cfff0 2024-03-06T07:23:44.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.965+0000 7f261439a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f260c0d2330 con 0x7f260c0ce4b0 2024-03-06T07:23:44.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.965+0000 7f261439a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f260c0d1980 con 0x7f260c15bb20 2024-03-06T07:23:44.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.965+0000 7f2611935640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f260c0cfff0 0x7f260c0b2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.965+0000 7f2611935640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f260c0cfff0 0x7f260c0b2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48414/0 (socket says 172.21.15.83:48414) 2024-03-06T07:23:44.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.965+0000 7f2611935640 1 -- 172.21.15.83:0/3339390911 learned_addr learned my addr 172.21.15.83:0/3339390911 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:44.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.966+0000 7f2612937640 1 --2- 172.21.15.83:0/3339390911 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f260c15bb20 0x7f260c0b2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.966+0000 7f2612136640 1 --2- 172.21.15.83:0/3339390911 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f260c0ce4b0 0x7f260c146e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.966+0000 7f2611935640 1 -- 172.21.15.83:0/3339390911 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f260c15bb20 msgr2=0x7f260c0b2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.966+0000 7f2611935640 1 --2- 172.21.15.83:0/3339390911 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f260c15bb20 0x7f260c0b2560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.966+0000 7f2611935640 1 -- 172.21.15.83:0/3339390911 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f260c0ce4b0 msgr2=0x7f260c146e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.966+0000 7f2611935640 1 --2- 172.21.15.83:0/3339390911 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f260c0ce4b0 0x7f260c146e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.966+0000 7f2611935640 1 -- 172.21.15.83:0/3339390911 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f260c15c1f0 con 0x7f260c0cfff0 2024-03-06T07:23:44.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.966+0000 7f2611935640 1 --2- 172.21.15.83:0/3339390911 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f260c0cfff0 0x7f260c0b2020 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f25fc00d800 tx=0x7f25fc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:44.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.966+0000 7f2602ffd640 1 -- 172.21.15.83:0/3339390911 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25fc00b6c0 con 0x7f260c0cfff0 2024-03-06T07:23:44.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.966+0000 7f2602ffd640 1 -- 172.21.15.83:0/3339390911 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25fc019070 con 0x7f260c0cfff0 2024-03-06T07:23:44.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.966+0000 7f2602ffd640 1 -- 172.21.15.83:0/3339390911 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25fc00ba80 con 0x7f260c0cfff0 2024-03-06T07:23:44.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.966+0000 7f261439a640 1 -- 172.21.15.83:0/3339390911 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f260c15c4d0 con 0x7f260c0cfff0 2024-03-06T07:23:44.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.966+0000 7f261439a640 1 -- 172.21.15.83:0/3339390911 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f260c15ca90 con 0x7f260c0cfff0 2024-03-06T07:23:44.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.967+0000 7f261439a640 1 -- 172.21.15.83:0/3339390911 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f260c04f330 con 0x7f260c0cfff0 2024-03-06T07:23:44.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.967+0000 7f2602ffd640 1 -- 172.21.15.83:0/3339390911 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f25fc00bc20 con 0x7f260c0cfff0 2024-03-06T07:23:44.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.968+0000 7f2602ffd640 1 --2- 172.21.15.83:0/3339390911 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f25f006b4d0 0x7f25f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.968+0000 7f2602ffd640 1 -- 172.21.15.83:0/3339390911 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f25fc08a650 con 0x7f260c0cfff0 2024-03-06T07:23:44.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.968+0000 7f2612136640 1 --2- 172.21.15.83:0/3339390911 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f25f006b4d0 0x7f25f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.968+0000 7f2612136640 1 --2- 172.21.15.83:0/3339390911 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f25f006b4d0 0x7f25f006d990 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f2604035000 tx=0x7f2604030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:44.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:44.974+0000 7f2602ffd640 1 -- 172.21.15.83:0/3339390911 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f25fc059560 con 0x7f260c0cfff0 2024-03-06T07:23:45.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.158+0000 7f261439a640 1 -- 172.21.15.83:0/3339390911 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f260c0b0a10 con 0x7f25f006b4d0 2024-03-06T07:23:45.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.164+0000 7f2602ffd640 1 -- 172.21.15.83:0/3339390911 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+239846 (secure 0 0 0) 0x7f260c0b0a10 con 0x7f25f006b4d0 2024-03-06T07:23:45.169 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:23:45.170 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-06T07:23:45.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.167+0000 7f261439a640 1 -- 172.21.15.83:0/3339390911 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f25f006b4d0 msgr2=0x7f25f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.167+0000 7f261439a640 1 --2- 172.21.15.83:0/3339390911 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f25f006b4d0 0x7f25f006d990 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f2604035000 tx=0x7f2604030000 comp rx=0 tx=0).stop 2024-03-06T07:23:45.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.167+0000 7f261439a640 1 -- 172.21.15.83:0/3339390911 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f260c0cfff0 msgr2=0x7f260c0b2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.167+0000 7f261439a640 1 --2- 172.21.15.83:0/3339390911 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f260c0cfff0 0x7f260c0b2020 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f25fc00d800 tx=0x7f25fc00dcc0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.167+0000 7f261439a640 1 -- 172.21.15.83:0/3339390911 shutdown_connections 2024-03-06T07:23:45.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.167+0000 7f261439a640 1 --2- 172.21.15.83:0/3339390911 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f25f006b4d0 0x7f25f006d990 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.167+0000 7f261439a640 1 --2- 172.21.15.83:0/3339390911 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f260c15bb20 0x7f260c0b2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.167+0000 7f261439a640 1 --2- 172.21.15.83:0/3339390911 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f260c0cfff0 0x7f260c0b2020 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.167+0000 7f261439a640 1 --2- 172.21.15.83:0/3339390911 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f260c0ce4b0 0x7f260c146e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.167+0000 7f261439a640 1 -- 172.21.15.83:0/3339390911 >> 172.21.15.83:0/3339390911 conn(0x7f260c09a9f0 msgr2=0x7f260c070360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:45.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.167+0000 7f261439a640 1 -- 172.21.15.83:0/3339390911 shutdown_connections 2024-03-06T07:23:45.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.167+0000 7f261439a640 1 -- 172.21.15.83:0/3339390911 wait complete. 2024-03-06T07:23:45.180 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":98,"stamp":"2024-03-06T07:23:44.340205+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":254376,"kb_used_data":4928,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754720344,"statfs":{"total":773094113280,"available":772833632256,"internally_reserved":0,"allocated":5046272,"data_stored":2716034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12757,"internal_metadata":255315499},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.261865"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.082391+0000","last_change":"2024-03-06T07:23:40.082391+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.082391+0000","last_undegraded":"2024-03-06T07:23:40.082391+0000","last_fullsized":"2024-03-06T07:23:40.082391+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.084350+0000","last_change":"2024-03-06T07:23:40.084350+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.084350+0000","last_undegraded":"2024-03-06T07:23:40.084350+0000","last_fullsized":"2024-03-06T07:23:40.084350+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.081457+0000","last_change":"2024-03-06T07:23:40.081457+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.081457+0000","last_undegraded":"2024-03-06T07:23:40.081457+0000","last_fullsized":"2024-03-06T07:23:40.081457+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.086423+0000","last_change":"2024-03-06T07:23:40.086423+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.086423+0000","last_undegraded":"2024-03-06T07:23:40.086423+0000","last_fullsized":"2024-03-06T07:23:40.086423+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.085503+0000","last_change":"2024-03-06T07:23:40.085503+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.085503+0000","last_undegraded":"2024-03-06T07:23:40.085503+0000","last_fullsized":"2024-03-06T07:23:40.085503+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.084363+0000","last_change":"2024-03-06T07:23:40.084363+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.084363+0000","last_undegraded":"2024-03-06T07:23:40.084363+0000","last_fullsized":"2024-03-06T07:23:40.084363+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.079858+0000","last_change":"2024-03-06T07:23:40.079858+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.079858+0000","last_undegraded":"2024-03-06T07:23:40.079858+0000","last_fullsized":"2024-03-06T07:23:40.079858+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.096035+0000","last_change":"2024-03-06T07:23:40.096035+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.096035+0000","last_undegraded":"2024-03-06T07:23:40.096035+0000","last_fullsized":"2024-03-06T07:23:40.096035+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.083857+0000","last_change":"2024-03-06T07:23:40.083857+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.083857+0000","last_undegraded":"2024-03-06T07:23:40.083857+0000","last_fullsized":"2024-03-06T07:23:40.083857+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.090531+0000","last_change":"2024-03-06T07:23:40.090531+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.090531+0000","last_undegraded":"2024-03-06T07:23:40.090531+0000","last_fullsized":"2024-03-06T07:23:40.090531+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.086933+0000","last_change":"2024-03-06T07:23:40.086933+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.086933+0000","last_undegraded":"2024-03-06T07:23:40.086933+0000","last_fullsized":"2024-03-06T07:23:40.086933+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.087670+0000","last_change":"2024-03-06T07:23:40.087670+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.087670+0000","last_undegraded":"2024-03-06T07:23:40.087670+0000","last_fullsized":"2024-03-06T07:23:40.087670+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.085110+0000","last_change":"2024-03-06T07:23:40.085110+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.085110+0000","last_undegraded":"2024-03-06T07:23:40.085110+0000","last_fullsized":"2024-03-06T07:23:40.085110+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.086411+0000","last_change":"2024-03-06T07:23:40.086411+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.086411+0000","last_undegraded":"2024-03-06T07:23:40.086411+0000","last_fullsized":"2024-03-06T07:23:40.086411+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.082306+0000","last_change":"2024-03-06T07:23:40.082306+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.082306+0000","last_undegraded":"2024-03-06T07:23:40.082306+0000","last_fullsized":"2024-03-06T07:23:40.082306+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.095608+0000","last_change":"2024-03-06T07:23:40.095608+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.095608+0000","last_undegraded":"2024-03-06T07:23:40.095608+0000","last_fullsized":"2024-03-06T07:23:40.095608+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.087683+0000","last_change":"2024-03-06T07:23:40.087683+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.087683+0000","last_undegraded":"2024-03-06T07:23:40.087683+0000","last_fullsized":"2024-03-06T07:23:40.087683+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.083800+0000","last_change":"2024-03-06T07:23:40.083800+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.083800+0000","last_undegraded":"2024-03-06T07:23:40.083800+0000","last_fullsized":"2024-03-06T07:23:40.083800+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.079827+0000","last_change":"2024-03-06T07:23:40.079827+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.079827+0000","last_undegraded":"2024-03-06T07:23:40.079827+0000","last_fullsized":"2024-03-06T07:23:40.079827+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.082418+0000","last_change":"2024-03-06T07:23:40.082418+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.082418+0000","last_undegraded":"2024-03-06T07:23:40.082418+0000","last_fullsized":"2024-03-06T07:23:40.082418+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.082263+0000","last_change":"2024-03-06T07:23:40.082263+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.082263+0000","last_undegraded":"2024-03-06T07:23:40.082263+0000","last_fullsized":"2024-03-06T07:23:40.082263+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.079841+0000","last_change":"2024-03-06T07:23:40.079841+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.079841+0000","last_undegraded":"2024-03-06T07:23:40.079841+0000","last_fullsized":"2024-03-06T07:23:40.079841+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.090538+0000","last_change":"2024-03-06T07:23:40.090538+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.090538+0000","last_undegraded":"2024-03-06T07:23:40.090538+0000","last_fullsized":"2024-03-06T07:23:40.090538+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.086423+0000","last_change":"2024-03-06T07:23:40.086423+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.086423+0000","last_undegraded":"2024-03-06T07:23:40.086423+0000","last_fullsized":"2024-03-06T07:23:40.086423+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.082211+0000","last_change":"2024-03-06T07:23:40.082211+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.082211+0000","last_undegraded":"2024-03-06T07:23:40.082211+0000","last_fullsized":"2024-03-06T07:23:40.082211+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.086363+0000","last_change":"2024-03-06T07:23:40.086363+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.086363+0000","last_undegraded":"2024-03-06T07:23:40.086363+0000","last_fullsized":"2024-03-06T07:23:40.086363+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.081368+0000","last_change":"2024-03-06T07:23:40.081368+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.081368+0000","last_undegraded":"2024-03-06T07:23:40.081368+0000","last_fullsized":"2024-03-06T07:23:40.081368+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.086404+0000","last_change":"2024-03-06T07:23:40.086404+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.086404+0000","last_undegraded":"2024-03-06T07:23:40.086404+0000","last_fullsized":"2024-03-06T07:23:40.086404+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.083857+0000","last_change":"2024-03-06T07:23:40.083857+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.083857+0000","last_undegraded":"2024-03-06T07:23:40.083857+0000","last_fullsized":"2024-03-06T07:23:40.083857+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.081341+0000","last_change":"2024-03-06T07:23:40.081341+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.081341+0000","last_undegraded":"2024-03-06T07:23:40.081341+0000","last_fullsized":"2024-03-06T07:23:40.081341+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.090464+0000","last_change":"2024-03-06T07:23:40.090464+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.090464+0000","last_undegraded":"2024-03-06T07:23:40.090464+0000","last_fullsized":"2024-03-06T07:23:40.090464+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.080272+0000","last_change":"2024-03-06T07:23:40.080272+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.080272+0000","last_undegraded":"2024-03-06T07:23:40.080272+0000","last_fullsized":"2024-03-06T07:23:40.080272+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":208,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-03-06T07:23:40.088817+0000","last_change":"2024-03-06T07:21:47.553693+0000","last_active":"2024-03-06T07:23:40.088817+0000","last_peered":"2024-03-06T07:23:40.088817+0000","last_clean":"2024-03-06T07:23:40.088817+0000","last_became_active":"2024-03-06T07:21:47.552664+0000","last_became_peered":"2024-03-06T07:21:47.552664+0000","last_unstale":"2024-03-06T07:23:40.088817+0000","last_undegraded":"2024-03-06T07:23:40.088817+0000","last_fullsized":"2024-03-06T07:23:40.088817+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_clean_scrub_stamp":"2024-03-06T07:21:46.362028+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:02.916426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574876,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32228,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339612,"statfs":{"total":96636764160,"available":96603762688,"internally_reserved":0,"allocated":1073152,"data_stored":780826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31648,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340192,"statfs":{"total":96636764160,"available":96604356608,"internally_reserved":0,"allocated":479232,"data_stored":190458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31664,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340176,"statfs":{"total":96636764160,"available":96604340224,"internally_reserved":0,"allocated":491520,"data_stored":196275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31648,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340192,"statfs":{"total":96636764160,"available":96604356608,"internally_reserved":0,"allocated":479232,"data_stored":190458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574875,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31648,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340192,"statfs":{"total":96636764160,"available":96604356608,"internally_reserved":0,"allocated":479232,"data_stored":190458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31648,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340192,"statfs":{"total":96636764160,"available":96604356608,"internally_reserved":0,"allocated":479232,"data_stored":190458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31652,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340188,"statfs":{"total":96636764160,"available":96604352512,"internally_reserved":0,"allocated":479232,"data_stored":190458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32240,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339600,"statfs":{"total":96636764160,"available":96603750400,"internally_reserved":0,"allocated":1085440,"data_stored":786643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:23:45.184 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:23:45.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.319+0000 7f4cb1e0e640 1 Processor -- start 2024-03-06T07:23:45.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.319+0000 7f4cb1e0e640 1 -- start start 2024-03-06T07:23:45.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.320+0000 7f4cb1e0e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4cac15dfe0 0x7f4cac0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.320+0000 7f4cb1e0e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cac0b3430 con 0x7f4cac0d29c0 2024-03-06T07:23:45.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.320+0000 7f4cb1e0e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cac0b3710 con 0x7f4cac0cf830 2024-03-06T07:23:45.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.320+0000 7f4cb1e0e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cac0b31e0 con 0x7f4cac15dfe0 2024-03-06T07:23:45.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.320+0000 7f4cab7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4cac15dfe0 0x7f4cac0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.320+0000 7f4cabfff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f4cac0d29c0 0x7f4cac0b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43036/0 (socket says 172.21.15.83:43036) 2024-03-06T07:23:45.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.320+0000 7f4cabfff640 1 -- 172.21.15.83:0/3595226343 learned_addr learned my addr 172.21.15.83:0/3595226343 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:23:45.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.320+0000 7f4cab7fe640 1 -- 172.21.15.83:0/3595226343 >> v1:172.21.15.183:6789/0 conn(0x7f4cac0cf830 legacy=0x7f4cac0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:45.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.320+0000 7f4cab7fe640 1 -- 172.21.15.83:0/3595226343 >> v1:172.21.15.83:6789/0 conn(0x7f4cac0d29c0 legacy=0x7f4cac0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.320+0000 7f4cab7fe640 1 -- 172.21.15.83:0/3595226343 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cac0b39f0 con 0x7f4cac15dfe0 2024-03-06T07:23:45.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.321+0000 7f4cab7fe640 1 --2- 172.21.15.83:0/3595226343 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cac15dfe0 0x7f4cac0cf2f0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f4cac0b3360 tx=0x7f4c9c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=473ae3023930f43d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:45.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.321+0000 7f4caa7fc640 1 -- 172.21.15.83:0/3595226343 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c035020 con 0x7f4cac15dfe0 2024-03-06T07:23:45.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.321+0000 7f4caa7fc640 1 -- 172.21.15.83:0/3595226343 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c9c031830 con 0x7f4cac15dfe0 2024-03-06T07:23:45.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.321+0000 7f4caa7fc640 1 -- 172.21.15.83:0/3595226343 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c031c00 con 0x7f4cac15dfe0 2024-03-06T07:23:45.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.322+0000 7f4cb1e0e640 1 -- 172.21.15.83:0/3595226343 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cac15dfe0 msgr2=0x7f4cac0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.322+0000 7f4cb1e0e640 1 --2- 172.21.15.83:0/3595226343 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cac15dfe0 0x7f4cac0cf2f0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f4cac0b3360 tx=0x7f4c9c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.322+0000 7f4cb1e0e640 1 -- 172.21.15.83:0/3595226343 shutdown_connections 2024-03-06T07:23:45.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.322+0000 7f4cb1e0e640 1 --2- 172.21.15.83:0/3595226343 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cac15dfe0 0x7f4cac0cf2f0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.322+0000 7f4cb1e0e640 1 -- 172.21.15.83:0/3595226343 >> 172.21.15.83:0/3595226343 conn(0x7f4cac0d3cd0 msgr2=0x7f4cac0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:45.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.322+0000 7f4cb1e0e640 1 -- 172.21.15.83:0/3595226343 shutdown_connections 2024-03-06T07:23:45.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.322+0000 7f4cb1e0e640 1 -- 172.21.15.83:0/3595226343 wait complete. 2024-03-06T07:23:45.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.323+0000 7f4cb1e0e640 1 Processor -- start 2024-03-06T07:23:45.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.323+0000 7f4cb1e0e640 1 -- start start 2024-03-06T07:23:45.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.324+0000 7f4cb1e0e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cac0cf830 0x7f4cac0b87e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.324+0000 7f4cb1e0e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4cac0d29c0 0x7f4cac0b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.324+0000 7f4cb1e0e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4cac15dfe0 0x7f4cac0b6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.324+0000 7f4cb1e0e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cac15e3c0 con 0x7f4cac0cf830 2024-03-06T07:23:45.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.324+0000 7f4cb1e0e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cac0b3d60 con 0x7f4cac0d29c0 2024-03-06T07:23:45.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.324+0000 7f4cb1e0e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cac15e540 con 0x7f4cac15dfe0 2024-03-06T07:23:45.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.324+0000 7f4cab7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cac0cf830 0x7f4cac0b87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.324+0000 7f4cab7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cac0cf830 0x7f4cac0b87e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48434/0 (socket says 172.21.15.83:48434) 2024-03-06T07:23:45.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.324+0000 7f4cab7fe640 1 -- 172.21.15.83:0/2002541842 learned_addr learned my addr 172.21.15.83:0/2002541842 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:45.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.324+0000 7f4cab7fe640 1 -- 172.21.15.83:0/2002541842 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4cac15dfe0 msgr2=0x7f4cac0b6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.324+0000 7f4cab7fe640 1 --2- 172.21.15.83:0/2002541842 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4cac15dfe0 0x7f4cac0b6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.324+0000 7f4cab7fe640 1 -- 172.21.15.83:0/2002541842 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4cac0d29c0 msgr2=0x7f4cac0b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.324+0000 7f4cab7fe640 1 --2- 172.21.15.83:0/2002541842 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4cac0d29c0 0x7f4cac0b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.324+0000 7f4cab7fe640 1 -- 172.21.15.83:0/2002541842 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cac0b7590 con 0x7f4cac0cf830 2024-03-06T07:23:45.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.324+0000 7f4cab7fe640 1 --2- 172.21.15.83:0/2002541842 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cac0cf830 0x7f4cac0b87e0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c000c00 tx=0x7f4c9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:45.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.324+0000 7f4c93fff640 1 -- 172.21.15.83:0/2002541842 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c03aa70 con 0x7f4cac0cf830 2024-03-06T07:23:45.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.325+0000 7f4cb1e0e640 1 -- 172.21.15.83:0/2002541842 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cac0b7870 con 0x7f4cac0cf830 2024-03-06T07:23:45.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.325+0000 7f4c93fff640 1 -- 172.21.15.83:0/2002541842 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c9c03ac30 con 0x7f4cac0cf830 2024-03-06T07:23:45.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.325+0000 7f4c93fff640 1 -- 172.21.15.83:0/2002541842 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c0443f0 con 0x7f4cac0cf830 2024-03-06T07:23:45.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.325+0000 7f4cb1e0e640 1 -- 172.21.15.83:0/2002541842 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cac0b7e30 con 0x7f4cac0cf830 2024-03-06T07:23:45.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.326+0000 7f4cb1e0e640 1 -- 172.21.15.83:0/2002541842 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cac15e750 con 0x7f4cac0cf830 2024-03-06T07:23:45.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.326+0000 7f4c93fff640 1 -- 172.21.15.83:0/2002541842 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4c9c04b060 con 0x7f4cac0cf830 2024-03-06T07:23:45.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.326+0000 7f4c93fff640 1 --2- 172.21.15.83:0/2002541842 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4c9806b4d0 0x7f4c9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.327+0000 7f4c93fff640 1 -- 172.21.15.83:0/2002541842 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4c9c0ac010 con 0x7f4cac0cf830 2024-03-06T07:23:45.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.327+0000 7f4caaffd640 1 --2- 172.21.15.83:0/2002541842 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4c9806b4d0 0x7f4c9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.327+0000 7f4caaffd640 1 --2- 172.21.15.83:0/2002541842 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4c9806b4d0 0x7f4c9806d990 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f4cac15c820 tx=0x7f4ca0007990 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:45.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.335+0000 7f4c93fff640 1 -- 172.21.15.83:0/2002541842 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4cac15e750 con 0x7f4cac0cf830 2024-03-06T07:23:45.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.506+0000 7f4cb1e0e640 1 -- 172.21.15.83:0/2002541842 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4cac04f330 con 0x7f4c9806b4d0 2024-03-06T07:23:45.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.517+0000 7f4c93fff640 1 -- 172.21.15.83:0/2002541842 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+239846 (secure 0 0 0) 0x7f4cac04f330 con 0x7f4c9806b4d0 2024-03-06T07:23:45.521 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:23:45.522 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-06T07:23:45.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.520+0000 7f4cb1e0e640 1 -- 172.21.15.83:0/2002541842 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4c9806b4d0 msgr2=0x7f4c9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.520+0000 7f4cb1e0e640 1 --2- 172.21.15.83:0/2002541842 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4c9806b4d0 0x7f4c9806d990 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f4cac15c820 tx=0x7f4ca0007990 comp rx=0 tx=0).stop 2024-03-06T07:23:45.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.520+0000 7f4cb1e0e640 1 -- 172.21.15.83:0/2002541842 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cac0cf830 msgr2=0x7f4cac0b87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.520+0000 7f4cb1e0e640 1 --2- 172.21.15.83:0/2002541842 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cac0cf830 0x7f4cac0b87e0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c000c00 tx=0x7f4c9c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.520+0000 7f4cb1e0e640 1 -- 172.21.15.83:0/2002541842 shutdown_connections 2024-03-06T07:23:45.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.520+0000 7f4cb1e0e640 1 --2- 172.21.15.83:0/2002541842 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4c9806b4d0 0x7f4c9806d990 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.520+0000 7f4cb1e0e640 1 --2- 172.21.15.83:0/2002541842 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4cac15dfe0 0x7f4cac0b6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.520+0000 7f4cb1e0e640 1 --2- 172.21.15.83:0/2002541842 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4cac0d29c0 0x7f4cac0b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.520+0000 7f4cb1e0e640 1 --2- 172.21.15.83:0/2002541842 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cac0cf830 0x7f4cac0b87e0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.520+0000 7f4cb1e0e640 1 -- 172.21.15.83:0/2002541842 >> 172.21.15.83:0/2002541842 conn(0x7f4cac0d3cd0 msgr2=0x7f4cac0b6110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:45.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.520+0000 7f4cb1e0e640 1 -- 172.21.15.83:0/2002541842 shutdown_connections 2024-03-06T07:23:45.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.520+0000 7f4cb1e0e640 1 -- 172.21.15.83:0/2002541842 wait complete. 2024-03-06T07:23:45.533 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":98,"stamp":"2024-03-06T07:23:44.340205+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":254376,"kb_used_data":4928,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754720344,"statfs":{"total":773094113280,"available":772833632256,"internally_reserved":0,"allocated":5046272,"data_stored":2716034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12757,"internal_metadata":255315499},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.261865"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.082391+0000","last_change":"2024-03-06T07:23:40.082391+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.082391+0000","last_undegraded":"2024-03-06T07:23:40.082391+0000","last_fullsized":"2024-03-06T07:23:40.082391+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.084350+0000","last_change":"2024-03-06T07:23:40.084350+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.084350+0000","last_undegraded":"2024-03-06T07:23:40.084350+0000","last_fullsized":"2024-03-06T07:23:40.084350+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.081457+0000","last_change":"2024-03-06T07:23:40.081457+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.081457+0000","last_undegraded":"2024-03-06T07:23:40.081457+0000","last_fullsized":"2024-03-06T07:23:40.081457+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.086423+0000","last_change":"2024-03-06T07:23:40.086423+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.086423+0000","last_undegraded":"2024-03-06T07:23:40.086423+0000","last_fullsized":"2024-03-06T07:23:40.086423+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.085503+0000","last_change":"2024-03-06T07:23:40.085503+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.085503+0000","last_undegraded":"2024-03-06T07:23:40.085503+0000","last_fullsized":"2024-03-06T07:23:40.085503+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.084363+0000","last_change":"2024-03-06T07:23:40.084363+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.084363+0000","last_undegraded":"2024-03-06T07:23:40.084363+0000","last_fullsized":"2024-03-06T07:23:40.084363+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.079858+0000","last_change":"2024-03-06T07:23:40.079858+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.079858+0000","last_undegraded":"2024-03-06T07:23:40.079858+0000","last_fullsized":"2024-03-06T07:23:40.079858+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.096035+0000","last_change":"2024-03-06T07:23:40.096035+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.096035+0000","last_undegraded":"2024-03-06T07:23:40.096035+0000","last_fullsized":"2024-03-06T07:23:40.096035+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.083857+0000","last_change":"2024-03-06T07:23:40.083857+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.083857+0000","last_undegraded":"2024-03-06T07:23:40.083857+0000","last_fullsized":"2024-03-06T07:23:40.083857+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.090531+0000","last_change":"2024-03-06T07:23:40.090531+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.090531+0000","last_undegraded":"2024-03-06T07:23:40.090531+0000","last_fullsized":"2024-03-06T07:23:40.090531+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.086933+0000","last_change":"2024-03-06T07:23:40.086933+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.086933+0000","last_undegraded":"2024-03-06T07:23:40.086933+0000","last_fullsized":"2024-03-06T07:23:40.086933+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.087670+0000","last_change":"2024-03-06T07:23:40.087670+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.087670+0000","last_undegraded":"2024-03-06T07:23:40.087670+0000","last_fullsized":"2024-03-06T07:23:40.087670+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.085110+0000","last_change":"2024-03-06T07:23:40.085110+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.085110+0000","last_undegraded":"2024-03-06T07:23:40.085110+0000","last_fullsized":"2024-03-06T07:23:40.085110+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.086411+0000","last_change":"2024-03-06T07:23:40.086411+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.086411+0000","last_undegraded":"2024-03-06T07:23:40.086411+0000","last_fullsized":"2024-03-06T07:23:40.086411+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.082306+0000","last_change":"2024-03-06T07:23:40.082306+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.082306+0000","last_undegraded":"2024-03-06T07:23:40.082306+0000","last_fullsized":"2024-03-06T07:23:40.082306+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.095608+0000","last_change":"2024-03-06T07:23:40.095608+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.095608+0000","last_undegraded":"2024-03-06T07:23:40.095608+0000","last_fullsized":"2024-03-06T07:23:40.095608+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.087683+0000","last_change":"2024-03-06T07:23:40.087683+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.087683+0000","last_undegraded":"2024-03-06T07:23:40.087683+0000","last_fullsized":"2024-03-06T07:23:40.087683+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.083800+0000","last_change":"2024-03-06T07:23:40.083800+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.083800+0000","last_undegraded":"2024-03-06T07:23:40.083800+0000","last_fullsized":"2024-03-06T07:23:40.083800+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.079827+0000","last_change":"2024-03-06T07:23:40.079827+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.079827+0000","last_undegraded":"2024-03-06T07:23:40.079827+0000","last_fullsized":"2024-03-06T07:23:40.079827+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.082418+0000","last_change":"2024-03-06T07:23:40.082418+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.082418+0000","last_undegraded":"2024-03-06T07:23:40.082418+0000","last_fullsized":"2024-03-06T07:23:40.082418+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.082263+0000","last_change":"2024-03-06T07:23:40.082263+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.082263+0000","last_undegraded":"2024-03-06T07:23:40.082263+0000","last_fullsized":"2024-03-06T07:23:40.082263+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.079841+0000","last_change":"2024-03-06T07:23:40.079841+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.079841+0000","last_undegraded":"2024-03-06T07:23:40.079841+0000","last_fullsized":"2024-03-06T07:23:40.079841+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.090538+0000","last_change":"2024-03-06T07:23:40.090538+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.090538+0000","last_undegraded":"2024-03-06T07:23:40.090538+0000","last_fullsized":"2024-03-06T07:23:40.090538+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.086423+0000","last_change":"2024-03-06T07:23:40.086423+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.086423+0000","last_undegraded":"2024-03-06T07:23:40.086423+0000","last_fullsized":"2024-03-06T07:23:40.086423+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.082211+0000","last_change":"2024-03-06T07:23:40.082211+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.082211+0000","last_undegraded":"2024-03-06T07:23:40.082211+0000","last_fullsized":"2024-03-06T07:23:40.082211+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.086363+0000","last_change":"2024-03-06T07:23:40.086363+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.086363+0000","last_undegraded":"2024-03-06T07:23:40.086363+0000","last_fullsized":"2024-03-06T07:23:40.086363+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.081368+0000","last_change":"2024-03-06T07:23:40.081368+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.081368+0000","last_undegraded":"2024-03-06T07:23:40.081368+0000","last_fullsized":"2024-03-06T07:23:40.081368+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.086404+0000","last_change":"2024-03-06T07:23:40.086404+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.086404+0000","last_undegraded":"2024-03-06T07:23:40.086404+0000","last_fullsized":"2024-03-06T07:23:40.086404+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.083857+0000","last_change":"2024-03-06T07:23:40.083857+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.083857+0000","last_undegraded":"2024-03-06T07:23:40.083857+0000","last_fullsized":"2024-03-06T07:23:40.083857+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.081341+0000","last_change":"2024-03-06T07:23:40.081341+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.081341+0000","last_undegraded":"2024-03-06T07:23:40.081341+0000","last_fullsized":"2024-03-06T07:23:40.081341+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.090464+0000","last_change":"2024-03-06T07:23:40.090464+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.090464+0000","last_undegraded":"2024-03-06T07:23:40.090464+0000","last_fullsized":"2024-03-06T07:23:40.090464+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.080272+0000","last_change":"2024-03-06T07:23:40.080272+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.080272+0000","last_undegraded":"2024-03-06T07:23:40.080272+0000","last_fullsized":"2024-03-06T07:23:40.080272+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":208,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-03-06T07:23:40.088817+0000","last_change":"2024-03-06T07:21:47.553693+0000","last_active":"2024-03-06T07:23:40.088817+0000","last_peered":"2024-03-06T07:23:40.088817+0000","last_clean":"2024-03-06T07:23:40.088817+0000","last_became_active":"2024-03-06T07:21:47.552664+0000","last_became_peered":"2024-03-06T07:21:47.552664+0000","last_unstale":"2024-03-06T07:23:40.088817+0000","last_undegraded":"2024-03-06T07:23:40.088817+0000","last_fullsized":"2024-03-06T07:23:40.088817+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_clean_scrub_stamp":"2024-03-06T07:21:46.362028+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:02.916426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574876,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32228,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339612,"statfs":{"total":96636764160,"available":96603762688,"internally_reserved":0,"allocated":1073152,"data_stored":780826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31648,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340192,"statfs":{"total":96636764160,"available":96604356608,"internally_reserved":0,"allocated":479232,"data_stored":190458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31664,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340176,"statfs":{"total":96636764160,"available":96604340224,"internally_reserved":0,"allocated":491520,"data_stored":196275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31648,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340192,"statfs":{"total":96636764160,"available":96604356608,"internally_reserved":0,"allocated":479232,"data_stored":190458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574875,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31648,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340192,"statfs":{"total":96636764160,"available":96604356608,"internally_reserved":0,"allocated":479232,"data_stored":190458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31648,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340192,"statfs":{"total":96636764160,"available":96604356608,"internally_reserved":0,"allocated":479232,"data_stored":190458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31652,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340188,"statfs":{"total":96636764160,"available":96604352512,"internally_reserved":0,"allocated":479232,"data_stored":190458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32240,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339600,"statfs":{"total":96636764160,"available":96603750400,"internally_reserved":0,"allocated":1085440,"data_stored":786643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:23:45.536 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:23:45.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.671+0000 7f98dbaeb640 1 Processor -- start 2024-03-06T07:23:45.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.672+0000 7f98dbaeb640 1 -- start start 2024-03-06T07:23:45.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.672+0000 7f98dbaeb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98d40d2710 0x7f98d40b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.672+0000 7f98dbaeb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d40b3430 con 0x7f98d415dfe0 2024-03-06T07:23:45.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.672+0000 7f98dbaeb640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d40b3710 con 0x7f98d40d22b0 2024-03-06T07:23:45.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.672+0000 7f98dbaeb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98d40b31e0 con 0x7f98d40d2710 2024-03-06T07:23:45.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.672+0000 7f98d9887640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f98d415dfe0 0x7f98d40cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43050/0 (socket says 172.21.15.83:43050) 2024-03-06T07:23:45.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.672+0000 7f98d9887640 1 -- 172.21.15.83:0/2309432610 learned_addr learned my addr 172.21.15.83:0/2309432610 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:23:45.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.673+0000 7f98da088640 1 --2- 172.21.15.83:0/2309432610 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98d40d2710 0x7f98d40b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.673+0000 7f98d8885640 1 -- 172.21.15.83:0/2309432610 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 891732540 0 0) 0x7f98d40b3430 con 0x7f98d415dfe0 2024-03-06T07:23:45.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.673+0000 7f98d8885640 1 -- 172.21.15.83:0/2309432610 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98c0003520 con 0x7f98d415dfe0 2024-03-06T07:23:45.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.673+0000 7f98d8885640 1 -- 172.21.15.83:0/2309432610 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 198043418 0 0) 0x7f98c0003520 con 0x7f98d415dfe0 2024-03-06T07:23:45.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.673+0000 7f98d8885640 1 -- 172.21.15.83:0/2309432610 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98d40d2710 msgr2=0x7f98d40b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:45.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.673+0000 7f98d8885640 1 --2- 172.21.15.83:0/2309432610 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98d40d2710 0x7f98d40b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.673+0000 7f98d8885640 1 -- 172.21.15.83:0/2309432610 >> v1:172.21.15.183:6789/0 conn(0x7f98d40d22b0 legacy=0x7f98d40b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.673+0000 7f98d8885640 1 -- 172.21.15.83:0/2309432610 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d40b39f0 con 0x7f98d415dfe0 2024-03-06T07:23:45.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.673+0000 7f98d8885640 1 -- 172.21.15.83:0/2309432610 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f98d0001670 con 0x7f98d415dfe0 2024-03-06T07:23:45.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.673+0000 7f98d8885640 1 -- 172.21.15.83:0/2309432610 >> v1:172.21.15.83:6789/0 conn(0x7f98d415dfe0 legacy=0x7f98d40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.673+0000 7f98d8885640 1 --2- 172.21.15.83:0/2309432610 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98d40d2710 0x7f98c0003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.673+0000 7f98d8885640 1 --2- 172.21.15.83:0/2309432610 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98c0003f80 0x7f98c0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.673+0000 7f98d8885640 1 --2- 172.21.15.83:0/2309432610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98c0004d40 0x7f98c00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.673+0000 7f98d8885640 1 -- 172.21.15.83:0/2309432610 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d40b31e0 con 0x7f98c0004d40 2024-03-06T07:23:45.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.673+0000 7f98d8885640 1 -- 172.21.15.83:0/2309432610 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c000bad0 con 0x7f98c0003f80 2024-03-06T07:23:45.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.673+0000 7f98d8885640 1 -- 172.21.15.83:0/2309432610 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98c000bcd0 con 0x7f98d40d2710 2024-03-06T07:23:45.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.673+0000 7f98d9887640 1 --2- 172.21.15.83:0/2309432610 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98d40d2710 0x7f98c0003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.673+0000 7f98da088640 1 --2- 172.21.15.83:0/2309432610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98c0004d40 0x7f98c00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.673+0000 7f98d9086640 1 --2- 172.21.15.83:0/2309432610 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98c0003f80 0x7f98c0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.673+0000 7f98da088640 1 -- 172.21.15.83:0/2309432610 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98d40d2710 msgr2=0x7f98c0003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:45.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.673+0000 7f98da088640 1 --2- 172.21.15.83:0/2309432610 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98d40d2710 0x7f98c0003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.673+0000 7f98da088640 1 -- 172.21.15.83:0/2309432610 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98c0003f80 msgr2=0x7f98c0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:45.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.673+0000 7f98da088640 1 --2- 172.21.15.83:0/2309432610 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98c0003f80 0x7f98c0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.673+0000 7f98da088640 1 -- 172.21.15.83:0/2309432610 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c0003520 con 0x7f98c0004d40 2024-03-06T07:23:45.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.674+0000 7f98da088640 1 --2- 172.21.15.83:0/2309432610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98c0004d40 0x7f98c00051c0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f98cc009f90 tx=0x7f98c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3131d02ab5eef07 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:45.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.674+0000 7f98d8885640 1 -- 172.21.15.83:0/2309432610 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98cc039070 con 0x7f98c0004d40 2024-03-06T07:23:45.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.674+0000 7f98d8885640 1 -- 172.21.15.83:0/2309432610 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98cc02cbd0 con 0x7f98c0004d40 2024-03-06T07:23:45.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.674+0000 7f98d8885640 1 -- 172.21.15.83:0/2309432610 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98cc009c10 con 0x7f98c0004d40 2024-03-06T07:23:45.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.675+0000 7f98dbaeb640 1 -- 172.21.15.83:0/2309432610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98c0004d40 msgr2=0x7f98c00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.675+0000 7f98dbaeb640 1 --2- 172.21.15.83:0/2309432610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98c0004d40 0x7f98c00051c0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f98cc009f90 tx=0x7f98c0005830 comp rx=0 tx=0).stop 2024-03-06T07:23:45.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.675+0000 7f98da088640 1 -- 172.21.15.83:0/2309432610 reap_dead start 2024-03-06T07:23:45.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.675+0000 7f98dbaeb640 1 -- 172.21.15.83:0/2309432610 shutdown_connections 2024-03-06T07:23:45.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.675+0000 7f98dbaeb640 1 -- 172.21.15.83:0/2309432610 >> 172.21.15.83:0/2309432610 conn(0x7f98d40d3cd0 msgr2=0x7f98d40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:45.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.675+0000 7f98dbaeb640 1 -- 172.21.15.83:0/2309432610 shutdown_connections 2024-03-06T07:23:45.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.675+0000 7f98dbaeb640 1 -- 172.21.15.83:0/2309432610 wait complete. 2024-03-06T07:23:45.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.676+0000 7f98dbaeb640 1 Processor -- start 2024-03-06T07:23:45.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.676+0000 7f98dbaeb640 1 -- start start 2024-03-06T07:23:45.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.677+0000 7f98dbaeb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98d40b2480 0x7f98d415e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.677+0000 7f98dbaeb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98d415e900 0x7f98d41543c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.677+0000 7f98dbaeb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98d40b2860 0x7f98d4154900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.677+0000 7f98dbaeb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d40cfa00 con 0x7f98d40b2860 2024-03-06T07:23:45.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.677+0000 7f98dbaeb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d40b3d60 con 0x7f98d40b2480 2024-03-06T07:23:45.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.677+0000 7f98dbaeb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98d40d1c70 con 0x7f98d415e900 2024-03-06T07:23:45.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.677+0000 7f98da088640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98d40b2860 0x7f98d4154900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.677+0000 7f98da088640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98d40b2860 0x7f98d4154900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48446/0 (socket says 172.21.15.83:48446) 2024-03-06T07:23:45.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.677+0000 7f98da088640 1 -- 172.21.15.83:0/2265827124 learned_addr learned my addr 172.21.15.83:0/2265827124 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:45.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.677+0000 7f98da088640 1 -- 172.21.15.83:0/2265827124 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98d415e900 msgr2=0x7f98d41543c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.677+0000 7f98d9887640 1 --2- 172.21.15.83:0/2265827124 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98d40b2480 0x7f98d415e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.677+0000 7f98da088640 1 --2- 172.21.15.83:0/2265827124 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98d415e900 0x7f98d41543c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.677+0000 7f98da088640 1 -- 172.21.15.83:0/2265827124 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98d40b2480 msgr2=0x7f98d415e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.677+0000 7f98da088640 1 --2- 172.21.15.83:0/2265827124 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98d40b2480 0x7f98d415e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.677+0000 7f98da088640 1 -- 172.21.15.83:0/2265827124 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d415ee40 con 0x7f98d40b2860 2024-03-06T07:23:45.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.677+0000 7f98da088640 1 --2- 172.21.15.83:0/2265827124 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98d40b2860 0x7f98d4154900 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f98cc009f90 tx=0x7f98cc009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:45.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.678+0000 7f98ca7fc640 1 -- 172.21.15.83:0/2265827124 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98cc045070 con 0x7f98d40b2860 2024-03-06T07:23:45.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.678+0000 7f98ca7fc640 1 -- 172.21.15.83:0/2265827124 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98cc03a040 con 0x7f98d40b2860 2024-03-06T07:23:45.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.678+0000 7f98dbaeb640 1 -- 172.21.15.83:0/2265827124 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98d415f120 con 0x7f98d40b2860 2024-03-06T07:23:45.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.678+0000 7f98ca7fc640 1 -- 172.21.15.83:0/2265827124 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98cc039070 con 0x7f98d40b2860 2024-03-06T07:23:45.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.678+0000 7f98dbaeb640 1 -- 172.21.15.83:0/2265827124 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98d415f6e0 con 0x7f98d40b2860 2024-03-06T07:23:45.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.679+0000 7f98dbaeb640 1 -- 172.21.15.83:0/2265827124 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98d404f330 con 0x7f98d40b2860 2024-03-06T07:23:45.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.679+0000 7f98ca7fc640 1 -- 172.21.15.83:0/2265827124 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f98cc003bd0 con 0x7f98d40b2860 2024-03-06T07:23:45.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.680+0000 7f98ca7fc640 1 --2- 172.21.15.83:0/2265827124 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f98b806b4d0 0x7f98b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.680+0000 7f98ca7fc640 1 -- 172.21.15.83:0/2265827124 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f98cc0aba90 con 0x7f98d40b2860 2024-03-06T07:23:45.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.680+0000 7f98d9887640 1 --2- 172.21.15.83:0/2265827124 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f98b806b4d0 0x7f98b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.680+0000 7f98d9887640 1 --2- 172.21.15.83:0/2265827124 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f98b806b4d0 0x7f98b806d990 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f98d0001050 tx=0x7f98d0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:45.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.684+0000 7f98ca7fc640 1 -- 172.21.15.83:0/2265827124 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f98cc07a350 con 0x7f98d40b2860 2024-03-06T07:23:45.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.863+0000 7f98dbaeb640 1 -- 172.21.15.83:0/2265827124 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f98d40747a0 con 0x7f98b806b4d0 2024-03-06T07:23:45.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.869+0000 7f98ca7fc640 1 -- 172.21.15.83:0/2265827124 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+239846 (secure 0 0 0) 0x7f98d40747a0 con 0x7f98b806b4d0 2024-03-06T07:23:45.873 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:23:45.874 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-06T07:23:45.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.871+0000 7f98dbaeb640 1 -- 172.21.15.83:0/2265827124 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f98b806b4d0 msgr2=0x7f98b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.871+0000 7f98dbaeb640 1 --2- 172.21.15.83:0/2265827124 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f98b806b4d0 0x7f98b806d990 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f98d0001050 tx=0x7f98d0009040 comp rx=0 tx=0).stop 2024-03-06T07:23:45.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.871+0000 7f98dbaeb640 1 -- 172.21.15.83:0/2265827124 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98d40b2860 msgr2=0x7f98d4154900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.871+0000 7f98dbaeb640 1 --2- 172.21.15.83:0/2265827124 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98d40b2860 0x7f98d4154900 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f98cc009f90 tx=0x7f98cc009750 comp rx=0 tx=0).stop 2024-03-06T07:23:45.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.871+0000 7f98dbaeb640 1 -- 172.21.15.83:0/2265827124 shutdown_connections 2024-03-06T07:23:45.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.871+0000 7f98dbaeb640 1 --2- 172.21.15.83:0/2265827124 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f98b806b4d0 0x7f98b806d990 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.871+0000 7f98dbaeb640 1 --2- 172.21.15.83:0/2265827124 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98d40b2860 0x7f98d4154900 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.871+0000 7f98dbaeb640 1 --2- 172.21.15.83:0/2265827124 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98d415e900 0x7f98d41543c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.871+0000 7f98dbaeb640 1 --2- 172.21.15.83:0/2265827124 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98d40b2480 0x7f98d415e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.871+0000 7f98dbaeb640 1 -- 172.21.15.83:0/2265827124 >> 172.21.15.83:0/2265827124 conn(0x7f98d40d3cd0 msgr2=0x7f98d406cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:45.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.871+0000 7f98dbaeb640 1 -- 172.21.15.83:0/2265827124 shutdown_connections 2024-03-06T07:23:45.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:45.871+0000 7f98dbaeb640 1 -- 172.21.15.83:0/2265827124 wait complete. 2024-03-06T07:23:45.884 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":98,"stamp":"2024-03-06T07:23:44.340205+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":254376,"kb_used_data":4928,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754720344,"statfs":{"total":773094113280,"available":772833632256,"internally_reserved":0,"allocated":5046272,"data_stored":2716034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12757,"internal_metadata":255315499},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.261865"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.082391+0000","last_change":"2024-03-06T07:23:40.082391+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.082391+0000","last_undegraded":"2024-03-06T07:23:40.082391+0000","last_fullsized":"2024-03-06T07:23:40.082391+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.084350+0000","last_change":"2024-03-06T07:23:40.084350+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.084350+0000","last_undegraded":"2024-03-06T07:23:40.084350+0000","last_fullsized":"2024-03-06T07:23:40.084350+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.081457+0000","last_change":"2024-03-06T07:23:40.081457+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.081457+0000","last_undegraded":"2024-03-06T07:23:40.081457+0000","last_fullsized":"2024-03-06T07:23:40.081457+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.086423+0000","last_change":"2024-03-06T07:23:40.086423+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.086423+0000","last_undegraded":"2024-03-06T07:23:40.086423+0000","last_fullsized":"2024-03-06T07:23:40.086423+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.085503+0000","last_change":"2024-03-06T07:23:40.085503+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.085503+0000","last_undegraded":"2024-03-06T07:23:40.085503+0000","last_fullsized":"2024-03-06T07:23:40.085503+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.084363+0000","last_change":"2024-03-06T07:23:40.084363+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.084363+0000","last_undegraded":"2024-03-06T07:23:40.084363+0000","last_fullsized":"2024-03-06T07:23:40.084363+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.079858+0000","last_change":"2024-03-06T07:23:40.079858+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.079858+0000","last_undegraded":"2024-03-06T07:23:40.079858+0000","last_fullsized":"2024-03-06T07:23:40.079858+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.096035+0000","last_change":"2024-03-06T07:23:40.096035+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.096035+0000","last_undegraded":"2024-03-06T07:23:40.096035+0000","last_fullsized":"2024-03-06T07:23:40.096035+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.083857+0000","last_change":"2024-03-06T07:23:40.083857+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.083857+0000","last_undegraded":"2024-03-06T07:23:40.083857+0000","last_fullsized":"2024-03-06T07:23:40.083857+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.090531+0000","last_change":"2024-03-06T07:23:40.090531+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.090531+0000","last_undegraded":"2024-03-06T07:23:40.090531+0000","last_fullsized":"2024-03-06T07:23:40.090531+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.086933+0000","last_change":"2024-03-06T07:23:40.086933+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.086933+0000","last_undegraded":"2024-03-06T07:23:40.086933+0000","last_fullsized":"2024-03-06T07:23:40.086933+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.087670+0000","last_change":"2024-03-06T07:23:40.087670+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.087670+0000","last_undegraded":"2024-03-06T07:23:40.087670+0000","last_fullsized":"2024-03-06T07:23:40.087670+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.085110+0000","last_change":"2024-03-06T07:23:40.085110+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.085110+0000","last_undegraded":"2024-03-06T07:23:40.085110+0000","last_fullsized":"2024-03-06T07:23:40.085110+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.086411+0000","last_change":"2024-03-06T07:23:40.086411+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.086411+0000","last_undegraded":"2024-03-06T07:23:40.086411+0000","last_fullsized":"2024-03-06T07:23:40.086411+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.082306+0000","last_change":"2024-03-06T07:23:40.082306+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.082306+0000","last_undegraded":"2024-03-06T07:23:40.082306+0000","last_fullsized":"2024-03-06T07:23:40.082306+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.095608+0000","last_change":"2024-03-06T07:23:40.095608+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.095608+0000","last_undegraded":"2024-03-06T07:23:40.095608+0000","last_fullsized":"2024-03-06T07:23:40.095608+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.087683+0000","last_change":"2024-03-06T07:23:40.087683+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.087683+0000","last_undegraded":"2024-03-06T07:23:40.087683+0000","last_fullsized":"2024-03-06T07:23:40.087683+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.083800+0000","last_change":"2024-03-06T07:23:40.083800+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.083800+0000","last_undegraded":"2024-03-06T07:23:40.083800+0000","last_fullsized":"2024-03-06T07:23:40.083800+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.079827+0000","last_change":"2024-03-06T07:23:40.079827+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.079827+0000","last_undegraded":"2024-03-06T07:23:40.079827+0000","last_fullsized":"2024-03-06T07:23:40.079827+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.082418+0000","last_change":"2024-03-06T07:23:40.082418+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.082418+0000","last_undegraded":"2024-03-06T07:23:40.082418+0000","last_fullsized":"2024-03-06T07:23:40.082418+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.082263+0000","last_change":"2024-03-06T07:23:40.082263+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.082263+0000","last_undegraded":"2024-03-06T07:23:40.082263+0000","last_fullsized":"2024-03-06T07:23:40.082263+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.079841+0000","last_change":"2024-03-06T07:23:40.079841+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.079841+0000","last_undegraded":"2024-03-06T07:23:40.079841+0000","last_fullsized":"2024-03-06T07:23:40.079841+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.090538+0000","last_change":"2024-03-06T07:23:40.090538+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.090538+0000","last_undegraded":"2024-03-06T07:23:40.090538+0000","last_fullsized":"2024-03-06T07:23:40.090538+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.086423+0000","last_change":"2024-03-06T07:23:40.086423+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.086423+0000","last_undegraded":"2024-03-06T07:23:40.086423+0000","last_fullsized":"2024-03-06T07:23:40.086423+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.082211+0000","last_change":"2024-03-06T07:23:40.082211+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.082211+0000","last_undegraded":"2024-03-06T07:23:40.082211+0000","last_fullsized":"2024-03-06T07:23:40.082211+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.086363+0000","last_change":"2024-03-06T07:23:40.086363+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.086363+0000","last_undegraded":"2024-03-06T07:23:40.086363+0000","last_fullsized":"2024-03-06T07:23:40.086363+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.081368+0000","last_change":"2024-03-06T07:23:40.081368+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.081368+0000","last_undegraded":"2024-03-06T07:23:40.081368+0000","last_fullsized":"2024-03-06T07:23:40.081368+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.086404+0000","last_change":"2024-03-06T07:23:40.086404+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.086404+0000","last_undegraded":"2024-03-06T07:23:40.086404+0000","last_fullsized":"2024-03-06T07:23:40.086404+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.083857+0000","last_change":"2024-03-06T07:23:40.083857+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.083857+0000","last_undegraded":"2024-03-06T07:23:40.083857+0000","last_fullsized":"2024-03-06T07:23:40.083857+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.081341+0000","last_change":"2024-03-06T07:23:40.081341+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.081341+0000","last_undegraded":"2024-03-06T07:23:40.081341+0000","last_fullsized":"2024-03-06T07:23:40.081341+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.090464+0000","last_change":"2024-03-06T07:23:40.090464+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.090464+0000","last_undegraded":"2024-03-06T07:23:40.090464+0000","last_fullsized":"2024-03-06T07:23:40.090464+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:41.075905+0000","last_change":"2024-03-06T07:23:41.075905+0000","last_active":"2024-03-06T07:23:41.075905+0000","last_peered":"2024-03-06T07:23:41.075905+0000","last_clean":"2024-03-06T07:23:41.075905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:41.075905+0000","last_undegraded":"2024-03-06T07:23:41.075905+0000","last_fullsized":"2024-03-06T07:23:41.075905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-03-06T07:23:40.080272+0000","last_change":"2024-03-06T07:23:40.080272+0000","last_active":"2024-03-06T07:23:40.069167+0000","last_peered":"2024-03-06T07:23:40.069167+0000","last_clean":"2024-03-06T07:23:40.069167+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:40.080272+0000","last_undegraded":"2024-03-06T07:23:40.080272+0000","last_fullsized":"2024-03-06T07:23:40.080272+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":208,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-03-06T07:23:40.088817+0000","last_change":"2024-03-06T07:21:47.553693+0000","last_active":"2024-03-06T07:23:40.088817+0000","last_peered":"2024-03-06T07:23:40.088817+0000","last_clean":"2024-03-06T07:23:40.088817+0000","last_became_active":"2024-03-06T07:21:47.552664+0000","last_became_peered":"2024-03-06T07:21:47.552664+0000","last_unstale":"2024-03-06T07:23:40.088817+0000","last_undegraded":"2024-03-06T07:23:40.088817+0000","last_fullsized":"2024-03-06T07:23:40.088817+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_clean_scrub_stamp":"2024-03-06T07:21:46.362028+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:02.916426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574876,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32228,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339612,"statfs":{"total":96636764160,"available":96603762688,"internally_reserved":0,"allocated":1073152,"data_stored":780826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31648,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340192,"statfs":{"total":96636764160,"available":96604356608,"internally_reserved":0,"allocated":479232,"data_stored":190458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31664,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340176,"statfs":{"total":96636764160,"available":96604340224,"internally_reserved":0,"allocated":491520,"data_stored":196275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31648,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340192,"statfs":{"total":96636764160,"available":96604356608,"internally_reserved":0,"allocated":479232,"data_stored":190458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574875,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31648,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340192,"statfs":{"total":96636764160,"available":96604356608,"internally_reserved":0,"allocated":479232,"data_stored":190458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31648,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340192,"statfs":{"total":96636764160,"available":96604356608,"internally_reserved":0,"allocated":479232,"data_stored":190458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31652,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340188,"statfs":{"total":96636764160,"available":96604352512,"internally_reserved":0,"allocated":479232,"data_stored":190458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32240,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339600,"statfs":{"total":96636764160,"available":96603750400,"internally_reserved":0,"allocated":1085440,"data_stored":786643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:23:45.889 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3f is not active+clean 2024-03-06T07:23:45.889 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.889 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3e is not active+clean 2024-03-06T07:23:45.889 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.890 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3d is not active+clean 2024-03-06T07:23:45.890 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.890 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3c is not active+clean 2024-03-06T07:23:45.890 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.890 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3b is not active+clean 2024-03-06T07:23:45.890 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.890 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3a is not active+clean 2024-03-06T07:23:45.890 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.890 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.39 is not active+clean 2024-03-06T07:23:45.890 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.891 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.38 is not active+clean 2024-03-06T07:23:45.891 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.891 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.37 is not active+clean 2024-03-06T07:23:45.891 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.891 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.36 is not active+clean 2024-03-06T07:23:45.891 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.891 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.35 is not active+clean 2024-03-06T07:23:45.891 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.891 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.34 is not active+clean 2024-03-06T07:23:45.891 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.891 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.33 is not active+clean 2024-03-06T07:23:45.891 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.892 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.32 is not active+clean 2024-03-06T07:23:45.892 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.892 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.31 is not active+clean 2024-03-06T07:23:45.892 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.892 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.30 is not active+clean 2024-03-06T07:23:45.892 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.892 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2f is not active+clean 2024-03-06T07:23:45.892 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.892 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2e is not active+clean 2024-03-06T07:23:45.892 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.893 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2d is not active+clean 2024-03-06T07:23:45.893 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.893 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2c is not active+clean 2024-03-06T07:23:45.893 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.893 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2b is not active+clean 2024-03-06T07:23:45.893 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.893 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2a is not active+clean 2024-03-06T07:23:45.893 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.893 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.29 is not active+clean 2024-03-06T07:23:45.893 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.894 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.28 is not active+clean 2024-03-06T07:23:45.894 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.894 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.27 is not active+clean 2024-03-06T07:23:45.894 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.894 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.26 is not active+clean 2024-03-06T07:23:45.894 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.894 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.25 is not active+clean 2024-03-06T07:23:45.894 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.895 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.24 is not active+clean 2024-03-06T07:23:45.895 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.895 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.23 is not active+clean 2024-03-06T07:23:45.895 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.895 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.22 is not active+clean 2024-03-06T07:23:45.895 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.895 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.21 is not active+clean 2024-03-06T07:23:45.895 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.895 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.20 is not active+clean 2024-03-06T07:23:45.895 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.895 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1e is not active+clean 2024-03-06T07:23:45.896 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.896 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1f is not active+clean 2024-03-06T07:23:45.896 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 42, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:40.082391+0000', 'last_change': '2024-03-06T07:23:40.082391+0000', 'last_active': '2024-03-06T07:23:40.069167+0000', 'last_peered': '2024-03-06T07:23:40.069167+0000', 'last_clean': '2024-03-06T07:23:40.069167+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:40.082391+0000', 'last_undegraded': '2024-03-06T07:23:40.082391+0000', 'last_fullsized': '2024-03-06T07:23:40.082391+0000', 'mapping_epoch': 42, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 42, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:45.896 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1f is not active+clean 2024-03-06T07:23:45.896 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.897 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1e is not active+clean 2024-03-06T07:23:45.897 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 42, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:40.084350+0000', 'last_change': '2024-03-06T07:23:40.084350+0000', 'last_active': '2024-03-06T07:23:40.069167+0000', 'last_peered': '2024-03-06T07:23:40.069167+0000', 'last_clean': '2024-03-06T07:23:40.069167+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:40.084350+0000', 'last_undegraded': '2024-03-06T07:23:40.084350+0000', 'last_fullsized': '2024-03-06T07:23:40.084350+0000', 'mapping_epoch': 42, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 42, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:23:45.897 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1c is not active+clean 2024-03-06T07:23:45.897 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.897 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1d is not active+clean 2024-03-06T07:23:45.897 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 42, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:40.081457+0000', 'last_change': '2024-03-06T07:23:40.081457+0000', 'last_active': '2024-03-06T07:23:40.069167+0000', 'last_peered': '2024-03-06T07:23:40.069167+0000', 'last_clean': '2024-03-06T07:23:40.069167+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:40.081457+0000', 'last_undegraded': '2024-03-06T07:23:40.081457+0000', 'last_fullsized': '2024-03-06T07:23:40.081457+0000', 'mapping_epoch': 42, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 42, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:23:45.897 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1d is not active+clean 2024-03-06T07:23:45.897 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.897 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1c is not active+clean 2024-03-06T07:23:45.898 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 42, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:40.086423+0000', 'last_change': '2024-03-06T07:23:40.086423+0000', 'last_active': '2024-03-06T07:23:40.069167+0000', 'last_peered': '2024-03-06T07:23:40.069167+0000', 'last_clean': '2024-03-06T07:23:40.069167+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:40.086423+0000', 'last_undegraded': '2024-03-06T07:23:40.086423+0000', 'last_fullsized': '2024-03-06T07:23:40.086423+0000', 'mapping_epoch': 42, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 42, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:23:45.898 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1a is not active+clean 2024-03-06T07:23:45.898 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.898 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1b is not active+clean 2024-03-06T07:23:45.898 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 42, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:40.085503+0000', 'last_change': '2024-03-06T07:23:40.085503+0000', 'last_active': '2024-03-06T07:23:40.069167+0000', 'last_peered': '2024-03-06T07:23:40.069167+0000', 'last_clean': '2024-03-06T07:23:40.069167+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:40.085503+0000', 'last_undegraded': '2024-03-06T07:23:40.085503+0000', 'last_fullsized': '2024-03-06T07:23:40.085503+0000', 'mapping_epoch': 42, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 42, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:45.898 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1b is not active+clean 2024-03-06T07:23:45.898 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.898 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1a is not active+clean 2024-03-06T07:23:45.898 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 42, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:40.084363+0000', 'last_change': '2024-03-06T07:23:40.084363+0000', 'last_active': '2024-03-06T07:23:40.069167+0000', 'last_peered': '2024-03-06T07:23:40.069167+0000', 'last_clean': '2024-03-06T07:23:40.069167+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:40.084363+0000', 'last_undegraded': '2024-03-06T07:23:40.084363+0000', 'last_fullsized': '2024-03-06T07:23:40.084363+0000', 'mapping_epoch': 42, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 42, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:23:45.899 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.18 is not active+clean 2024-03-06T07:23:45.899 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.899 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.19 is not active+clean 2024-03-06T07:23:45.899 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 42, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:40.079858+0000', 'last_change': '2024-03-06T07:23:40.079858+0000', 'last_active': '2024-03-06T07:23:40.069167+0000', 'last_peered': '2024-03-06T07:23:40.069167+0000', 'last_clean': '2024-03-06T07:23:40.069167+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:40.079858+0000', 'last_undegraded': '2024-03-06T07:23:40.079858+0000', 'last_fullsized': '2024-03-06T07:23:40.079858+0000', 'mapping_epoch': 42, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 42, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:23:45.899 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.19 is not active+clean 2024-03-06T07:23:45.899 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.899 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.18 is not active+clean 2024-03-06T07:23:45.899 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 42, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:40.096035+0000', 'last_change': '2024-03-06T07:23:40.096035+0000', 'last_active': '2024-03-06T07:23:40.069167+0000', 'last_peered': '2024-03-06T07:23:40.069167+0000', 'last_clean': '2024-03-06T07:23:40.069167+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:40.096035+0000', 'last_undegraded': '2024-03-06T07:23:40.096035+0000', 'last_fullsized': '2024-03-06T07:23:40.096035+0000', 'mapping_epoch': 42, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 42, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:45.899 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.16 is not active+clean 2024-03-06T07:23:45.899 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.900 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.17 is not active+clean 2024-03-06T07:23:45.900 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 42, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:40.083857+0000', 'last_change': '2024-03-06T07:23:40.083857+0000', 'last_active': '2024-03-06T07:23:40.069167+0000', 'last_peered': '2024-03-06T07:23:40.069167+0000', 'last_clean': '2024-03-06T07:23:40.069167+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:40.083857+0000', 'last_undegraded': '2024-03-06T07:23:40.083857+0000', 'last_fullsized': '2024-03-06T07:23:40.083857+0000', 'mapping_epoch': 42, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 42, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:23:45.900 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.17 is not active+clean 2024-03-06T07:23:45.900 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.900 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.16 is not active+clean 2024-03-06T07:23:45.900 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 42, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:40.090531+0000', 'last_change': '2024-03-06T07:23:40.090531+0000', 'last_active': '2024-03-06T07:23:40.069167+0000', 'last_peered': '2024-03-06T07:23:40.069167+0000', 'last_clean': '2024-03-06T07:23:40.069167+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:40.090531+0000', 'last_undegraded': '2024-03-06T07:23:40.090531+0000', 'last_fullsized': '2024-03-06T07:23:40.090531+0000', 'mapping_epoch': 42, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 42, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:45.900 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.14 is not active+clean 2024-03-06T07:23:45.900 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.900 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.15 is not active+clean 2024-03-06T07:23:45.901 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 42, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:40.086933+0000', 'last_change': '2024-03-06T07:23:40.086933+0000', 'last_active': '2024-03-06T07:23:40.069167+0000', 'last_peered': '2024-03-06T07:23:40.069167+0000', 'last_clean': '2024-03-06T07:23:40.069167+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:40.086933+0000', 'last_undegraded': '2024-03-06T07:23:40.086933+0000', 'last_fullsized': '2024-03-06T07:23:40.086933+0000', 'mapping_epoch': 42, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 42, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:23:45.901 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.15 is not active+clean 2024-03-06T07:23:45.901 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.901 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.14 is not active+clean 2024-03-06T07:23:45.901 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 42, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:40.087670+0000', 'last_change': '2024-03-06T07:23:40.087670+0000', 'last_active': '2024-03-06T07:23:40.069167+0000', 'last_peered': '2024-03-06T07:23:40.069167+0000', 'last_clean': '2024-03-06T07:23:40.069167+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:40.087670+0000', 'last_undegraded': '2024-03-06T07:23:40.087670+0000', 'last_fullsized': '2024-03-06T07:23:40.087670+0000', 'mapping_epoch': 42, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 42, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:23:45.901 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.12 is not active+clean 2024-03-06T07:23:45.901 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.901 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.13 is not active+clean 2024-03-06T07:23:45.901 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 42, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:40.085110+0000', 'last_change': '2024-03-06T07:23:40.085110+0000', 'last_active': '2024-03-06T07:23:40.069167+0000', 'last_peered': '2024-03-06T07:23:40.069167+0000', 'last_clean': '2024-03-06T07:23:40.069167+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:40.085110+0000', 'last_undegraded': '2024-03-06T07:23:40.085110+0000', 'last_fullsized': '2024-03-06T07:23:40.085110+0000', 'mapping_epoch': 42, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 42, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:45.902 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.13 is not active+clean 2024-03-06T07:23:45.902 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.902 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.12 is not active+clean 2024-03-06T07:23:45.902 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 42, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:40.086411+0000', 'last_change': '2024-03-06T07:23:40.086411+0000', 'last_active': '2024-03-06T07:23:40.069167+0000', 'last_peered': '2024-03-06T07:23:40.069167+0000', 'last_clean': '2024-03-06T07:23:40.069167+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:40.086411+0000', 'last_undegraded': '2024-03-06T07:23:40.086411+0000', 'last_fullsized': '2024-03-06T07:23:40.086411+0000', 'mapping_epoch': 42, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 42, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:23:45.902 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.10 is not active+clean 2024-03-06T07:23:45.902 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.902 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.11 is not active+clean 2024-03-06T07:23:45.902 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 42, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:40.082306+0000', 'last_change': '2024-03-06T07:23:40.082306+0000', 'last_active': '2024-03-06T07:23:40.069167+0000', 'last_peered': '2024-03-06T07:23:40.069167+0000', 'last_clean': '2024-03-06T07:23:40.069167+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:40.082306+0000', 'last_undegraded': '2024-03-06T07:23:40.082306+0000', 'last_fullsized': '2024-03-06T07:23:40.082306+0000', 'mapping_epoch': 42, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 42, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:45.902 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.11 is not active+clean 2024-03-06T07:23:45.903 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.903 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.10 is not active+clean 2024-03-06T07:23:45.903 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 42, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:40.095608+0000', 'last_change': '2024-03-06T07:23:40.095608+0000', 'last_active': '2024-03-06T07:23:40.069167+0000', 'last_peered': '2024-03-06T07:23:40.069167+0000', 'last_clean': '2024-03-06T07:23:40.069167+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:40.095608+0000', 'last_undegraded': '2024-03-06T07:23:40.095608+0000', 'last_fullsized': '2024-03-06T07:23:40.095608+0000', 'mapping_epoch': 42, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 42, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:45.903 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.e is not active+clean 2024-03-06T07:23:45.903 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.903 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.f is not active+clean 2024-03-06T07:23:45.903 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 42, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:40.087683+0000', 'last_change': '2024-03-06T07:23:40.087683+0000', 'last_active': '2024-03-06T07:23:40.069167+0000', 'last_peered': '2024-03-06T07:23:40.069167+0000', 'last_clean': '2024-03-06T07:23:40.069167+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:40.087683+0000', 'last_undegraded': '2024-03-06T07:23:40.087683+0000', 'last_fullsized': '2024-03-06T07:23:40.087683+0000', 'mapping_epoch': 42, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 42, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:23:45.903 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.f is not active+clean 2024-03-06T07:23:45.903 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.904 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.e is not active+clean 2024-03-06T07:23:45.904 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 42, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:40.083800+0000', 'last_change': '2024-03-06T07:23:40.083800+0000', 'last_active': '2024-03-06T07:23:40.069167+0000', 'last_peered': '2024-03-06T07:23:40.069167+0000', 'last_clean': '2024-03-06T07:23:40.069167+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:40.083800+0000', 'last_undegraded': '2024-03-06T07:23:40.083800+0000', 'last_fullsized': '2024-03-06T07:23:40.083800+0000', 'mapping_epoch': 42, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 42, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:23:45.904 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.c is not active+clean 2024-03-06T07:23:45.904 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.904 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.d is not active+clean 2024-03-06T07:23:45.904 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 42, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:40.079827+0000', 'last_change': '2024-03-06T07:23:40.079827+0000', 'last_active': '2024-03-06T07:23:40.069167+0000', 'last_peered': '2024-03-06T07:23:40.069167+0000', 'last_clean': '2024-03-06T07:23:40.069167+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:40.079827+0000', 'last_undegraded': '2024-03-06T07:23:40.079827+0000', 'last_fullsized': '2024-03-06T07:23:40.079827+0000', 'mapping_epoch': 42, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 42, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:23:45.904 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.d is not active+clean 2024-03-06T07:23:45.904 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.904 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.c is not active+clean 2024-03-06T07:23:45.905 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 42, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:40.082418+0000', 'last_change': '2024-03-06T07:23:40.082418+0000', 'last_active': '2024-03-06T07:23:40.069167+0000', 'last_peered': '2024-03-06T07:23:40.069167+0000', 'last_clean': '2024-03-06T07:23:40.069167+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:40.082418+0000', 'last_undegraded': '2024-03-06T07:23:40.082418+0000', 'last_fullsized': '2024-03-06T07:23:40.082418+0000', 'mapping_epoch': 42, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 42, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:45.905 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.a is not active+clean 2024-03-06T07:23:45.905 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.905 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.b is not active+clean 2024-03-06T07:23:45.905 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 42, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:40.082263+0000', 'last_change': '2024-03-06T07:23:40.082263+0000', 'last_active': '2024-03-06T07:23:40.069167+0000', 'last_peered': '2024-03-06T07:23:40.069167+0000', 'last_clean': '2024-03-06T07:23:40.069167+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:40.082263+0000', 'last_undegraded': '2024-03-06T07:23:40.082263+0000', 'last_fullsized': '2024-03-06T07:23:40.082263+0000', 'mapping_epoch': 42, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 42, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:45.905 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.b is not active+clean 2024-03-06T07:23:45.905 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.905 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.a is not active+clean 2024-03-06T07:23:45.905 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 42, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:40.079841+0000', 'last_change': '2024-03-06T07:23:40.079841+0000', 'last_active': '2024-03-06T07:23:40.069167+0000', 'last_peered': '2024-03-06T07:23:40.069167+0000', 'last_clean': '2024-03-06T07:23:40.069167+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:40.079841+0000', 'last_undegraded': '2024-03-06T07:23:40.079841+0000', 'last_fullsized': '2024-03-06T07:23:40.079841+0000', 'mapping_epoch': 42, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 42, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:23:45.905 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.8 is not active+clean 2024-03-06T07:23:45.906 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.906 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.9 is not active+clean 2024-03-06T07:23:45.906 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 42, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:40.090538+0000', 'last_change': '2024-03-06T07:23:40.090538+0000', 'last_active': '2024-03-06T07:23:40.069167+0000', 'last_peered': '2024-03-06T07:23:40.069167+0000', 'last_clean': '2024-03-06T07:23:40.069167+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:40.090538+0000', 'last_undegraded': '2024-03-06T07:23:40.090538+0000', 'last_fullsized': '2024-03-06T07:23:40.090538+0000', 'mapping_epoch': 42, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 42, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:45.906 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.9 is not active+clean 2024-03-06T07:23:45.906 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.906 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.8 is not active+clean 2024-03-06T07:23:45.906 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 42, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:40.086423+0000', 'last_change': '2024-03-06T07:23:40.086423+0000', 'last_active': '2024-03-06T07:23:40.069167+0000', 'last_peered': '2024-03-06T07:23:40.069167+0000', 'last_clean': '2024-03-06T07:23:40.069167+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:40.086423+0000', 'last_undegraded': '2024-03-06T07:23:40.086423+0000', 'last_fullsized': '2024-03-06T07:23:40.086423+0000', 'mapping_epoch': 42, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 42, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:23:45.906 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.7 is not active+clean 2024-03-06T07:23:45.906 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.907 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.6 is not active+clean 2024-03-06T07:23:45.907 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 42, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:40.082211+0000', 'last_change': '2024-03-06T07:23:40.082211+0000', 'last_active': '2024-03-06T07:23:40.069167+0000', 'last_peered': '2024-03-06T07:23:40.069167+0000', 'last_clean': '2024-03-06T07:23:40.069167+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:40.082211+0000', 'last_undegraded': '2024-03-06T07:23:40.082211+0000', 'last_fullsized': '2024-03-06T07:23:40.082211+0000', 'mapping_epoch': 42, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 42, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:45.907 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.4 is not active+clean 2024-03-06T07:23:45.907 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.907 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.5 is not active+clean 2024-03-06T07:23:45.907 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 42, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:40.086363+0000', 'last_change': '2024-03-06T07:23:40.086363+0000', 'last_active': '2024-03-06T07:23:40.069167+0000', 'last_peered': '2024-03-06T07:23:40.069167+0000', 'last_clean': '2024-03-06T07:23:40.069167+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:40.086363+0000', 'last_undegraded': '2024-03-06T07:23:40.086363+0000', 'last_fullsized': '2024-03-06T07:23:40.086363+0000', 'mapping_epoch': 42, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 42, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:23:45.907 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.5 is not active+clean 2024-03-06T07:23:45.907 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.907 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.4 is not active+clean 2024-03-06T07:23:45.907 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 42, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:40.081368+0000', 'last_change': '2024-03-06T07:23:40.081368+0000', 'last_active': '2024-03-06T07:23:40.069167+0000', 'last_peered': '2024-03-06T07:23:40.069167+0000', 'last_clean': '2024-03-06T07:23:40.069167+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:40.081368+0000', 'last_undegraded': '2024-03-06T07:23:40.081368+0000', 'last_fullsized': '2024-03-06T07:23:40.081368+0000', 'mapping_epoch': 42, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 42, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:23:45.907 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2 is not active+clean 2024-03-06T07:23:45.908 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.908 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3 is not active+clean 2024-03-06T07:23:45.908 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 42, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:40.086404+0000', 'last_change': '2024-03-06T07:23:40.086404+0000', 'last_active': '2024-03-06T07:23:40.069167+0000', 'last_peered': '2024-03-06T07:23:40.069167+0000', 'last_clean': '2024-03-06T07:23:40.069167+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:40.086404+0000', 'last_undegraded': '2024-03-06T07:23:40.086404+0000', 'last_fullsized': '2024-03-06T07:23:40.086404+0000', 'mapping_epoch': 42, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 42, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:23:45.908 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3 is not active+clean 2024-03-06T07:23:45.908 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.908 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2 is not active+clean 2024-03-06T07:23:45.908 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 42, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:40.083857+0000', 'last_change': '2024-03-06T07:23:40.083857+0000', 'last_active': '2024-03-06T07:23:40.069167+0000', 'last_peered': '2024-03-06T07:23:40.069167+0000', 'last_clean': '2024-03-06T07:23:40.069167+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:40.083857+0000', 'last_undegraded': '2024-03-06T07:23:40.083857+0000', 'last_fullsized': '2024-03-06T07:23:40.083857+0000', 'mapping_epoch': 42, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 42, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:23:45.908 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.0 is not active+clean 2024-03-06T07:23:45.909 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.909 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1 is not active+clean 2024-03-06T07:23:45.909 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 42, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:40.081341+0000', 'last_change': '2024-03-06T07:23:40.081341+0000', 'last_active': '2024-03-06T07:23:40.069167+0000', 'last_peered': '2024-03-06T07:23:40.069167+0000', 'last_clean': '2024-03-06T07:23:40.069167+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:40.081341+0000', 'last_undegraded': '2024-03-06T07:23:40.081341+0000', 'last_fullsized': '2024-03-06T07:23:40.081341+0000', 'mapping_epoch': 42, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 42, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:23:45.909 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1 is not active+clean 2024-03-06T07:23:45.909 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.909 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.0 is not active+clean 2024-03-06T07:23:45.909 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 42, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:40.090464+0000', 'last_change': '2024-03-06T07:23:40.090464+0000', 'last_active': '2024-03-06T07:23:40.069167+0000', 'last_peered': '2024-03-06T07:23:40.069167+0000', 'last_clean': '2024-03-06T07:23:40.069167+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:40.090464+0000', 'last_undegraded': '2024-03-06T07:23:40.090464+0000', 'last_fullsized': '2024-03-06T07:23:40.090464+0000', 'mapping_epoch': 42, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 42, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:45.909 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.6 is not active+clean 2024-03-06T07:23:45.909 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:23:41.075905+0000', 'last_change': '2024-03-06T07:23:41.075905+0000', 'last_active': '2024-03-06T07:23:41.075905+0000', 'last_peered': '2024-03-06T07:23:41.075905+0000', 'last_clean': '2024-03-06T07:23:41.075905+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:41.075905+0000', 'last_undegraded': '2024-03-06T07:23:41.075905+0000', 'last_fullsized': '2024-03-06T07:23:41.075905+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:41.075905+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:23:45.910 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.7 is not active+clean 2024-03-06T07:23:45.910 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 42, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:40.080272+0000', 'last_change': '2024-03-06T07:23:40.080272+0000', 'last_active': '2024-03-06T07:23:40.069167+0000', 'last_peered': '2024-03-06T07:23:40.069167+0000', 'last_clean': '2024-03-06T07:23:40.069167+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:40.080272+0000', 'last_undegraded': '2024-03-06T07:23:40.080272+0000', 'last_fullsized': '2024-03-06T07:23:40.080272+0000', 'mapping_epoch': 42, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 42, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:40.069167+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:23:45.910 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:23:46.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.022+0000 7fd102d1e640 1 Processor -- start 2024-03-06T07:23:46.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.022+0000 7fd102d1e640 1 -- start start 2024-03-06T07:23:46.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.022+0000 7fd102d1e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fd0fc0b4460 0x7fd0fc0d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.022+0000 7fd102d1e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0fc161cf0 0x7fd0fc162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.022+0000 7fd102d1e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0fc0d22c0 con 0x7fd0fc162970 2024-03-06T07:23:46.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.022+0000 7fd102d1e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0fc161b00 con 0x7fd0fc0b4460 2024-03-06T07:23:46.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.022+0000 7fd102d1e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0fc0cecc0 con 0x7fd0fc161cf0 2024-03-06T07:23:46.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.023+0000 7fd1012bb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0fc161cf0 0x7fd0fc162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.023+0000 7fd100aba640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fd0fc0b4460 0x7fd0fc0d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.023+0000 7fd1012bb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0fc161cf0 0x7fd0fc162180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:60194/0 (socket says 172.21.15.83:60194) 2024-03-06T07:23:46.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.023+0000 7fd0fbfff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fd0fc162970 0x7fd0fc0b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:42258/0 (socket says 172.21.15.83:42258) 2024-03-06T07:23:46.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.023+0000 7fd1012bb640 1 -- 172.21.15.83:0/2496109577 learned_addr learned my addr 172.21.15.83:0/2496109577 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:46.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.023+0000 7fd1012bb640 1 -- 172.21.15.83:0/2496109577 >> v2:172.21.15.183:3300/0 conn(0x7fd0fc0b4460 msgr2=0x7fd0fc0d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:46.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.023+0000 7fd1012bb640 1 --2- 172.21.15.83:0/2496109577 >> v2:172.21.15.183:3300/0 conn(0x7fd0fc0b4460 0x7fd0fc0d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.023+0000 7fd1012bb640 1 -- 172.21.15.83:0/2496109577 >> v1:172.21.15.183:6789/0 conn(0x7fd0fc162970 legacy=0x7fd0fc0b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:46.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.023+0000 7fd1012bb640 1 -- 172.21.15.83:0/2496109577 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0fc0d25a0 con 0x7fd0fc161cf0 2024-03-06T07:23:46.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.023+0000 7fd100aba640 1 --2- 172.21.15.83:0/2496109577 >> v2:172.21.15.183:3300/0 conn(0x7fd0fc0b4460 0x7fd0fc0d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:46.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.024+0000 7fd1012bb640 1 --2- 172.21.15.83:0/2496109577 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0fc161cf0 0x7fd0fc162180 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec009870 tx=0x7fd0ec02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66f50073220eac18 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:46.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.024+0000 7fd0fb7fe640 1 -- 172.21.15.83:0/2496109577 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0ec034070 con 0x7fd0fc161cf0 2024-03-06T07:23:46.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.024+0000 7fd0fb7fe640 1 -- 172.21.15.83:0/2496109577 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0ec030840 con 0x7fd0fc161cf0 2024-03-06T07:23:46.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.024+0000 7fd0fb7fe640 1 -- 172.21.15.83:0/2496109577 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0ec030c10 con 0x7fd0fc161cf0 2024-03-06T07:23:46.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.025+0000 7fd102d1e640 1 -- 172.21.15.83:0/2496109577 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0fc161cf0 msgr2=0x7fd0fc162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:46.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.025+0000 7fd102d1e640 1 --2- 172.21.15.83:0/2496109577 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0fc161cf0 0x7fd0fc162180 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec009870 tx=0x7fd0ec02b980 comp rx=0 tx=0).stop 2024-03-06T07:23:46.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.025+0000 7fd102d1e640 1 -- 172.21.15.83:0/2496109577 shutdown_connections 2024-03-06T07:23:46.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.025+0000 7fd102d1e640 1 --2- 172.21.15.83:0/2496109577 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0fc161cf0 0x7fd0fc162180 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.025+0000 7fd102d1e640 1 --2- 172.21.15.83:0/2496109577 >> v2:172.21.15.183:3300/0 conn(0x7fd0fc0b4460 0x7fd0fc0d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.025+0000 7fd102d1e640 1 -- 172.21.15.83:0/2496109577 >> 172.21.15.83:0/2496109577 conn(0x7fd0fc0d3ce0 msgr2=0x7fd0fc0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:46.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.025+0000 7fd102d1e640 1 -- 172.21.15.83:0/2496109577 shutdown_connections 2024-03-06T07:23:46.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.025+0000 7fd102d1e640 1 -- 172.21.15.83:0/2496109577 wait complete. 2024-03-06T07:23:46.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.027+0000 7fd102d1e640 1 Processor -- start 2024-03-06T07:23:46.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.027+0000 7fd102d1e640 1 -- start start 2024-03-06T07:23:46.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.028+0000 7fd102d1e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd0fc0b4460 0x7fd0fc1e8500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.028+0000 7fd102d1e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0fc161cf0 0x7fd0fc0cd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.028+0000 7fd102d1e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0fc162970 0x7fd0fc0cb390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.028+0000 7fd102d1e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0fc0ce950 con 0x7fd0fc162970 2024-03-06T07:23:46.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.028+0000 7fd102d1e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0fc0d2b60 con 0x7fd0fc0b4460 2024-03-06T07:23:46.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.028+0000 7fd102d1e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0fc0b4840 con 0x7fd0fc161cf0 2024-03-06T07:23:46.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.028+0000 7fd1012bb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0fc162970 0x7fd0fc0cb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.028+0000 7fd1012bb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0fc162970 0x7fd0fc0cb390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48462/0 (socket says 172.21.15.83:48462) 2024-03-06T07:23:46.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.028+0000 7fd1012bb640 1 -- 172.21.15.83:0/1876805891 learned_addr learned my addr 172.21.15.83:0/1876805891 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:46.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.029+0000 7fd100aba640 1 --2- 172.21.15.83:0/1876805891 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd0fc0b4460 0x7fd0fc1e8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.029+0000 7fd0fbfff640 1 --2- 172.21.15.83:0/1876805891 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0fc161cf0 0x7fd0fc0cd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.029+0000 7fd1012bb640 1 -- 172.21.15.83:0/1876805891 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0fc161cf0 msgr2=0x7fd0fc0cd0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:46.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.029+0000 7fd1012bb640 1 --2- 172.21.15.83:0/1876805891 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0fc161cf0 0x7fd0fc0cd0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.029+0000 7fd1012bb640 1 -- 172.21.15.83:0/1876805891 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd0fc0b4460 msgr2=0x7fd0fc1e8500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:46.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.029+0000 7fd1012bb640 1 --2- 172.21.15.83:0/1876805891 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd0fc0b4460 0x7fd0fc1e8500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.029+0000 7fd1012bb640 1 -- 172.21.15.83:0/1876805891 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0fc1e8a40 con 0x7fd0fc162970 2024-03-06T07:23:46.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.029+0000 7fd1012bb640 1 --2- 172.21.15.83:0/1876805891 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0fc162970 0x7fd0fc0cb390 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec02b9c0 tx=0x7fd0ec004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:46.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.029+0000 7fd0f97fa640 1 -- 172.21.15.83:0/1876805891 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0ec048070 con 0x7fd0fc162970 2024-03-06T07:23:46.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.029+0000 7fd0f97fa640 1 -- 172.21.15.83:0/1876805891 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0ec030640 con 0x7fd0fc162970 2024-03-06T07:23:46.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.029+0000 7fd102d1e640 1 -- 172.21.15.83:0/1876805891 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0fc0b6aa0 con 0x7fd0fc162970 2024-03-06T07:23:46.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.029+0000 7fd0f97fa640 1 -- 172.21.15.83:0/1876805891 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0ec034040 con 0x7fd0fc162970 2024-03-06T07:23:46.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.029+0000 7fd102d1e640 1 -- 172.21.15.83:0/1876805891 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0fc0b7060 con 0x7fd0fc162970 2024-03-06T07:23:46.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.030+0000 7fd102d1e640 1 -- 172.21.15.83:0/1876805891 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0fc0b9be0 con 0x7fd0fc162970 2024-03-06T07:23:46.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.031+0000 7fd0f97fa640 1 -- 172.21.15.83:0/1876805891 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd0ec04b070 con 0x7fd0fc162970 2024-03-06T07:23:46.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.031+0000 7fd0f97fa640 1 --2- 172.21.15.83:0/1876805891 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd0e806b4d0 0x7fd0e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.031+0000 7fd100aba640 1 --2- 172.21.15.83:0/1876805891 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd0e806b4d0 0x7fd0e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.031+0000 7fd0f97fa640 1 -- 172.21.15.83:0/1876805891 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd0ec0ac100 con 0x7fd0fc162970 2024-03-06T07:23:46.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.032+0000 7fd100aba640 1 --2- 172.21.15.83:0/1876805891 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd0e806b4d0 0x7fd0e806d990 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4009bd0 tx=0x7fd0f400b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:46.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.036+0000 7fd0f97fa640 1 -- 172.21.15.83:0/1876805891 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd0fc0b9be0 con 0x7fd0fc162970 2024-03-06T07:23:46.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.282+0000 7fd102d1e640 1 -- 172.21.15.83:0/1876805891 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fd0fc04f330 con 0x7fd0fc162970 2024-03-06T07:23:46.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.282+0000 7fd0f97fa640 1 -- 172.21.15.83:0/1876805891 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1270 (secure 0 0 0) 0x7fd0ec07aae0 con 0x7fd0fc162970 2024-03-06T07:23:46.285 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:23:46.285 INFO:teuthology.orchestra.run.smithi083.stdout:{"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":132,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":45,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709705,"num_in_osds":8,"osd_in_since":1709709696,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":64},{"state_name":"creating+peering","count":32},{"state_name":"active+clean","count":1}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":260481024,"bytes_avail":772833632256,"bytes_total":773094113280,"unknown_pgs_ratio":0.65979379415512085,"inactive_pgs_ratio":0.32989689707756042},"fsmap":{"epoch":19,"id":3,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":3,"rank":0,"name":"a","status":"up:active","gid":5706}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-03-06T07:22:54.331751+0000","services":{"mds":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:23:46.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.285+0000 7fd102d1e640 1 -- 172.21.15.83:0/1876805891 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd0e806b4d0 msgr2=0x7fd0e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:46.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.285+0000 7fd102d1e640 1 --2- 172.21.15.83:0/1876805891 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd0e806b4d0 0x7fd0e806d990 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4009bd0 tx=0x7fd0f400b040 comp rx=0 tx=0).stop 2024-03-06T07:23:46.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.285+0000 7fd102d1e640 1 -- 172.21.15.83:0/1876805891 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0fc162970 msgr2=0x7fd0fc0cb390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:46.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.285+0000 7fd102d1e640 1 --2- 172.21.15.83:0/1876805891 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0fc162970 0x7fd0fc0cb390 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec02b9c0 tx=0x7fd0ec004730 comp rx=0 tx=0).stop 2024-03-06T07:23:46.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.285+0000 7fd102d1e640 1 -- 172.21.15.83:0/1876805891 shutdown_connections 2024-03-06T07:23:46.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.285+0000 7fd102d1e640 1 --2- 172.21.15.83:0/1876805891 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd0e806b4d0 0x7fd0e806d990 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.285+0000 7fd102d1e640 1 --2- 172.21.15.83:0/1876805891 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0fc162970 0x7fd0fc0cb390 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.285+0000 7fd102d1e640 1 --2- 172.21.15.83:0/1876805891 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0fc161cf0 0x7fd0fc0cd0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.285+0000 7fd102d1e640 1 --2- 172.21.15.83:0/1876805891 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd0fc0b4460 0x7fd0fc1e8500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.285+0000 7fd102d1e640 1 -- 172.21.15.83:0/1876805891 >> 172.21.15.83:0/1876805891 conn(0x7fd0fc0d3ce0 msgr2=0x7fd0fc0d0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:46.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.285+0000 7fd102d1e640 1 -- 172.21.15.83:0/1876805891 shutdown_connections 2024-03-06T07:23:46.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.285+0000 7fd102d1e640 1 -- 172.21.15.83:0/1876805891 wait complete. 2024-03-06T07:23:46.298 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:23:46.298 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:23:46.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.398+0000 7f4f5c73e640 1 Processor -- start 2024-03-06T07:23:46.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.398+0000 7f4f5c73e640 1 -- start start 2024-03-06T07:23:46.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.399+0000 7f4f5c73e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f4f5415dfe0 0x7f4f540cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.399+0000 7f4f5c73e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4f540cf830 0x7f4f540d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.399+0000 7f4f5c73e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4f540d27f0 0x7f4f540d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.399+0000 7f4f5c73e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f540b31e0 con 0x7f4f540cf830 2024-03-06T07:23:46.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.399+0000 7f4f5c73e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f540b3d60 con 0x7f4f5415dfe0 2024-03-06T07:23:46.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.399+0000 7f4f5c73e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f540b3430 con 0x7f4f540d27f0 2024-03-06T07:23:46.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.399+0000 7f4f59cd9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4f540cf830 0x7f4f540d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.399+0000 7f4f59cd9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4f540cf830 0x7f4f540d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48478/0 (socket says 172.21.15.83:48478) 2024-03-06T07:23:46.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.399+0000 7f4f59cd9640 1 -- 172.21.15.83:0/2749137236 learned_addr learned my addr 172.21.15.83:0/2749137236 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:46.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.399+0000 7f4f5acdb640 1 --2- 172.21.15.83:0/2749137236 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4f540d27f0 0x7f4f540d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.399+0000 7f4f5a4da640 1 --2- 172.21.15.83:0/2749137236 >> v2:172.21.15.183:3300/0 conn(0x7f4f5415dfe0 0x7f4f540cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.399+0000 7f4f59cd9640 1 -- 172.21.15.83:0/2749137236 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4f540d27f0 msgr2=0x7f4f540d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:46.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.399+0000 7f4f59cd9640 1 --2- 172.21.15.83:0/2749137236 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4f540d27f0 0x7f4f540d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.399+0000 7f4f59cd9640 1 -- 172.21.15.83:0/2749137236 >> v2:172.21.15.183:3300/0 conn(0x7f4f5415dfe0 msgr2=0x7f4f540cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:46.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.399+0000 7f4f59cd9640 1 --2- 172.21.15.83:0/2749137236 >> v2:172.21.15.183:3300/0 conn(0x7f4f5415dfe0 0x7f4f540cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.399+0000 7f4f59cd9640 1 -- 172.21.15.83:0/2749137236 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f540b3710 con 0x7f4f540cf830 2024-03-06T07:23:46.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.400+0000 7f4f59cd9640 1 --2- 172.21.15.83:0/2749137236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f540cf830 0x7f4f540d22b0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f4f54067990 tx=0x7f4f5002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca70f59bd258f197 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:46.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.400+0000 7f4f594d8640 1 -- 172.21.15.83:0/2749137236 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f50035020 con 0x7f4f540cf830 2024-03-06T07:23:46.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.400+0000 7f4f594d8640 1 -- 172.21.15.83:0/2749137236 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f50031870 con 0x7f4f540cf830 2024-03-06T07:23:46.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.400+0000 7f4f594d8640 1 -- 172.21.15.83:0/2749137236 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f50031c40 con 0x7f4f540cf830 2024-03-06T07:23:46.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.401+0000 7f4f5c73e640 1 -- 172.21.15.83:0/2749137236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f540cf830 msgr2=0x7f4f540d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:46.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.401+0000 7f4f5c73e640 1 --2- 172.21.15.83:0/2749137236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f540cf830 0x7f4f540d22b0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f4f54067990 tx=0x7f4f5002b910 comp rx=0 tx=0).stop 2024-03-06T07:23:46.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.401+0000 7f4f5c73e640 1 -- 172.21.15.83:0/2749137236 shutdown_connections 2024-03-06T07:23:46.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.401+0000 7f4f5c73e640 1 --2- 172.21.15.83:0/2749137236 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4f540d27f0 0x7f4f540d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.401+0000 7f4f5c73e640 1 --2- 172.21.15.83:0/2749137236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f540cf830 0x7f4f540d22b0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.401+0000 7f4f5c73e640 1 --2- 172.21.15.83:0/2749137236 >> v2:172.21.15.183:3300/0 conn(0x7f4f5415dfe0 0x7f4f540cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.401+0000 7f4f5c73e640 1 -- 172.21.15.83:0/2749137236 >> 172.21.15.83:0/2749137236 conn(0x7f4f540d3cd0 msgr2=0x7f4f540d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:46.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.401+0000 7f4f5c73e640 1 -- 172.21.15.83:0/2749137236 shutdown_connections 2024-03-06T07:23:46.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.401+0000 7f4f5c73e640 1 -- 172.21.15.83:0/2749137236 wait complete. 2024-03-06T07:23:46.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.402+0000 7f4f5c73e640 1 Processor -- start 2024-03-06T07:23:46.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.402+0000 7f4f5c73e640 1 -- start start 2024-03-06T07:23:46.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.403+0000 7f4f5c73e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f540cf830 0x7f4f540b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.403+0000 7f4f5c73e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4f540d27f0 0x7f4f5415be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.403+0000 7f4f5c73e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4f5415dfe0 0x7f4f540b80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.403+0000 7f4f5c73e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f540d1df0 con 0x7f4f540cf830 2024-03-06T07:23:46.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.403+0000 7f4f5c73e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f540d1c70 con 0x7f4f540d27f0 2024-03-06T07:23:46.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.403+0000 7f4f5c73e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5415e3c0 con 0x7f4f5415dfe0 2024-03-06T07:23:46.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.403+0000 7f4f5a4da640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f540cf830 0x7f4f540b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.403+0000 7f4f5a4da640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f540cf830 0x7f4f540b68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48490/0 (socket says 172.21.15.83:48490) 2024-03-06T07:23:46.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.403+0000 7f4f5a4da640 1 -- 172.21.15.83:0/3124443664 learned_addr learned my addr 172.21.15.83:0/3124443664 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:46.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.403+0000 7f4f59cd9640 1 --2- 172.21.15.83:0/3124443664 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4f540d27f0 0x7f4f5415be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.403+0000 7f4f5acdb640 1 --2- 172.21.15.83:0/3124443664 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4f5415dfe0 0x7f4f540b80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.403+0000 7f4f5a4da640 1 -- 172.21.15.83:0/3124443664 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4f5415dfe0 msgr2=0x7f4f540b80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:46.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.403+0000 7f4f5a4da640 1 --2- 172.21.15.83:0/3124443664 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4f5415dfe0 0x7f4f540b80f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.403+0000 7f4f5a4da640 1 -- 172.21.15.83:0/3124443664 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4f540d27f0 msgr2=0x7f4f5415be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:46.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.403+0000 7f4f5a4da640 1 --2- 172.21.15.83:0/3124443664 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4f540d27f0 0x7f4f5415be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.403+0000 7f4f5a4da640 1 -- 172.21.15.83:0/3124443664 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f540a1b70 con 0x7f4f540cf830 2024-03-06T07:23:46.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.403+0000 7f4f59cd9640 1 --2- 172.21.15.83:0/3124443664 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4f540d27f0 0x7f4f5415be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:46.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.403+0000 7f4f5a4da640 1 --2- 172.21.15.83:0/3124443664 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f540cf830 0x7f4f540b68d0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c007e40 tx=0x7f4f4c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:46.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.403+0000 7f4f46ffd640 1 -- 172.21.15.83:0/3124443664 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4c014620 con 0x7f4f540cf830 2024-03-06T07:23:46.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.403+0000 7f4f46ffd640 1 -- 172.21.15.83:0/3124443664 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f4c019070 con 0x7f4f540cf830 2024-03-06T07:23:46.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.403+0000 7f4f46ffd640 1 -- 172.21.15.83:0/3124443664 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4c014a00 con 0x7f4f540cf830 2024-03-06T07:23:46.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.403+0000 7f4f5c73e640 1 -- 172.21.15.83:0/3124443664 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f540b6fd0 con 0x7f4f540cf830 2024-03-06T07:23:46.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.403+0000 7f4f5c73e640 1 -- 172.21.15.83:0/3124443664 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f540b7590 con 0x7f4f540cf830 2024-03-06T07:23:46.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.404+0000 7f4f5c73e640 1 -- 172.21.15.83:0/3124443664 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f5404f330 con 0x7f4f540cf830 2024-03-06T07:23:46.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.405+0000 7f4f46ffd640 1 -- 172.21.15.83:0/3124443664 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4f4c014ba0 con 0x7f4f540cf830 2024-03-06T07:23:46.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.405+0000 7f4f46ffd640 1 --2- 172.21.15.83:0/3124443664 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4f3806b4d0 0x7f4f3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.405+0000 7f4f46ffd640 1 -- 172.21.15.83:0/3124443664 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4f4c08aaa0 con 0x7f4f540cf830 2024-03-06T07:23:46.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.405+0000 7f4f59cd9640 1 --2- 172.21.15.83:0/3124443664 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4f3806b4d0 0x7f4f3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.406+0000 7f4f59cd9640 1 --2- 172.21.15.83:0/3124443664 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4f3806b4d0 0x7f4f3806d990 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4f50036000 tx=0x7f4f5003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:46.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.410+0000 7f4f46ffd640 1 -- 172.21.15.83:0/3124443664 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4f4c059920 con 0x7f4f540cf830 2024-03-06T07:23:46.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.584+0000 7f4f5c73e640 1 -- 172.21.15.83:0/3124443664 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4f540ce9b0 con 0x7f4f3806b4d0 2024-03-06T07:23:46.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.590+0000 7f4f46ffd640 1 -- 172.21.15.83:0/3124443664 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251958 (secure 0 0 0) 0x7f4f540ce9b0 con 0x7f4f3806b4d0 2024-03-06T07:23:46.595 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:23:46.596 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-06T07:23:46.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.593+0000 7f4f5c73e640 1 -- 172.21.15.83:0/3124443664 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4f3806b4d0 msgr2=0x7f4f3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:46.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.593+0000 7f4f5c73e640 1 --2- 172.21.15.83:0/3124443664 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4f3806b4d0 0x7f4f3806d990 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4f50036000 tx=0x7f4f5003c040 comp rx=0 tx=0).stop 2024-03-06T07:23:46.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.593+0000 7f4f5c73e640 1 -- 172.21.15.83:0/3124443664 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f540cf830 msgr2=0x7f4f540b68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:46.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.593+0000 7f4f5c73e640 1 --2- 172.21.15.83:0/3124443664 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f540cf830 0x7f4f540b68d0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c007e40 tx=0x7f4f4c00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.593+0000 7f4f5c73e640 1 -- 172.21.15.83:0/3124443664 shutdown_connections 2024-03-06T07:23:46.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.593+0000 7f4f5c73e640 1 --2- 172.21.15.83:0/3124443664 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4f3806b4d0 0x7f4f3806d990 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.593+0000 7f4f5c73e640 1 --2- 172.21.15.83:0/3124443664 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4f5415dfe0 0x7f4f540b80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.593+0000 7f4f5c73e640 1 --2- 172.21.15.83:0/3124443664 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4f540d27f0 0x7f4f5415be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.593+0000 7f4f5c73e640 1 --2- 172.21.15.83:0/3124443664 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f540cf830 0x7f4f540b68d0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.593+0000 7f4f5c73e640 1 -- 172.21.15.83:0/3124443664 >> 172.21.15.83:0/3124443664 conn(0x7f4f540d3cd0 msgr2=0x7f4f5406bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:46.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.593+0000 7f4f5c73e640 1 -- 172.21.15.83:0/3124443664 shutdown_connections 2024-03-06T07:23:46.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:46.593+0000 7f4f5c73e640 1 -- 172.21.15.83:0/3124443664 wait complete. 2024-03-06T07:23:46.606 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":99,"stamp":"2024-03-06T07:23:46.340489+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":156,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254848,"kb_used_data":5352,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754719872,"statfs":{"total":773094113280,"available":772833148928,"internally_reserved":0,"allocated":5480448,"data_stored":2910492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12757,"internal_metadata":255315499},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000991"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.749408+0000","last_change":"2024-03-06T07:23:42.101550+0000","last_active":"2024-03-06T07:23:43.749408+0000","last_peered":"2024-03-06T07:23:43.749408+0000","last_clean":"2024-03-06T07:23:43.749408+0000","last_became_active":"2024-03-06T07:23:42.101282+0000","last_became_peered":"2024-03-06T07:23:42.101282+0000","last_unstale":"2024-03-06T07:23:43.749408+0000","last_undegraded":"2024-03-06T07:23:43.749408+0000","last_fullsized":"2024-03-06T07:23:43.749408+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:44.930081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748570+0000","last_change":"2024-03-06T07:23:42.115833+0000","last_active":"2024-03-06T07:23:43.748570+0000","last_peered":"2024-03-06T07:23:43.748570+0000","last_clean":"2024-03-06T07:23:43.748570+0000","last_became_active":"2024-03-06T07:23:42.115559+0000","last_became_peered":"2024-03-06T07:23:42.115559+0000","last_unstale":"2024-03-06T07:23:43.748570+0000","last_undegraded":"2024-03-06T07:23:43.748570+0000","last_fullsized":"2024-03-06T07:23:43.748570+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:39:58.306094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.102686+0000","last_change":"2024-03-06T07:23:42.094590+0000","last_active":"2024-03-06T07:23:43.102686+0000","last_peered":"2024-03-06T07:23:43.102686+0000","last_clean":"2024-03-06T07:23:43.102686+0000","last_became_active":"2024-03-06T07:23:42.094264+0000","last_became_peered":"2024-03-06T07:23:42.094264+0000","last_unstale":"2024-03-06T07:23:43.102686+0000","last_undegraded":"2024-03-06T07:23:43.102686+0000","last_fullsized":"2024-03-06T07:23:43.102686+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:41.381834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748540+0000","last_change":"2024-03-06T07:23:42.108875+0000","last_active":"2024-03-06T07:23:43.748540+0000","last_peered":"2024-03-06T07:23:43.748540+0000","last_clean":"2024-03-06T07:23:43.748540+0000","last_became_active":"2024-03-06T07:23:42.108613+0000","last_became_peered":"2024-03-06T07:23:42.108613+0000","last_unstale":"2024-03-06T07:23:43.748540+0000","last_undegraded":"2024-03-06T07:23:43.748540+0000","last_fullsized":"2024-03-06T07:23:43.748540+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:45:58.613060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748294+0000","last_change":"2024-03-06T07:23:42.116009+0000","last_active":"2024-03-06T07:23:43.748294+0000","last_peered":"2024-03-06T07:23:43.748294+0000","last_clean":"2024-03-06T07:23:43.748294+0000","last_became_active":"2024-03-06T07:23:42.115609+0000","last_became_peered":"2024-03-06T07:23:42.115609+0000","last_unstale":"2024-03-06T07:23:43.748294+0000","last_undegraded":"2024-03-06T07:23:43.748294+0000","last_fullsized":"2024-03-06T07:23:43.748294+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:12.037110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748771+0000","last_change":"2024-03-06T07:23:42.104726+0000","last_active":"2024-03-06T07:23:43.748771+0000","last_peered":"2024-03-06T07:23:43.748771+0000","last_clean":"2024-03-06T07:23:43.748771+0000","last_became_active":"2024-03-06T07:23:42.104233+0000","last_became_peered":"2024-03-06T07:23:42.104233+0000","last_unstale":"2024-03-06T07:23:43.748771+0000","last_undegraded":"2024-03-06T07:23:43.748771+0000","last_fullsized":"2024-03-06T07:23:43.748771+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:54:40.118136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.749066+0000","last_change":"2024-03-06T07:23:42.097305+0000","last_active":"2024-03-06T07:23:43.749066+0000","last_peered":"2024-03-06T07:23:43.749066+0000","last_clean":"2024-03-06T07:23:43.749066+0000","last_became_active":"2024-03-06T07:23:42.097033+0000","last_became_peered":"2024-03-06T07:23:42.097033+0000","last_unstale":"2024-03-06T07:23:43.749066+0000","last_undegraded":"2024-03-06T07:23:43.749066+0000","last_fullsized":"2024-03-06T07:23:43.749066+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:47:34.961757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.747864+0000","last_change":"2024-03-06T07:23:42.113045+0000","last_active":"2024-03-06T07:23:43.747864+0000","last_peered":"2024-03-06T07:23:43.747864+0000","last_clean":"2024-03-06T07:23:43.747864+0000","last_became_active":"2024-03-06T07:23:42.112672+0000","last_became_peered":"2024-03-06T07:23:42.112672+0000","last_unstale":"2024-03-06T07:23:43.747864+0000","last_undegraded":"2024-03-06T07:23:43.747864+0000","last_fullsized":"2024-03-06T07:23:43.747864+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:18.102969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748981+0000","last_change":"2024-03-06T07:23:42.113581+0000","last_active":"2024-03-06T07:23:43.748981+0000","last_peered":"2024-03-06T07:23:43.748981+0000","last_clean":"2024-03-06T07:23:43.748981+0000","last_became_active":"2024-03-06T07:23:42.113268+0000","last_became_peered":"2024-03-06T07:23:42.113268+0000","last_unstale":"2024-03-06T07:23:43.748981+0000","last_undegraded":"2024-03-06T07:23:43.748981+0000","last_fullsized":"2024-03-06T07:23:43.748981+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:04:27.854692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.749447+0000","last_change":"2024-03-06T07:23:42.113788+0000","last_active":"2024-03-06T07:23:43.749447+0000","last_peered":"2024-03-06T07:23:43.749447+0000","last_clean":"2024-03-06T07:23:43.749447+0000","last_became_active":"2024-03-06T07:23:42.113259+0000","last_became_peered":"2024-03-06T07:23:42.113259+0000","last_unstale":"2024-03-06T07:23:43.749447+0000","last_undegraded":"2024-03-06T07:23:43.749447+0000","last_fullsized":"2024-03-06T07:23:43.749447+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:25.231002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748400+0000","last_change":"2024-03-06T07:23:42.113821+0000","last_active":"2024-03-06T07:23:43.748400+0000","last_peered":"2024-03-06T07:23:43.748400+0000","last_clean":"2024-03-06T07:23:43.748400+0000","last_became_active":"2024-03-06T07:23:42.113442+0000","last_became_peered":"2024-03-06T07:23:42.113442+0000","last_unstale":"2024-03-06T07:23:43.748400+0000","last_undegraded":"2024-03-06T07:23:43.748400+0000","last_fullsized":"2024-03-06T07:23:43.748400+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:34:33.726443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748252+0000","last_change":"2024-03-06T07:23:42.102990+0000","last_active":"2024-03-06T07:23:43.748252+0000","last_peered":"2024-03-06T07:23:43.748252+0000","last_clean":"2024-03-06T07:23:43.748252+0000","last_became_active":"2024-03-06T07:23:42.101295+0000","last_became_peered":"2024-03-06T07:23:42.101295+0000","last_unstale":"2024-03-06T07:23:43.748252+0000","last_undegraded":"2024-03-06T07:23:43.748252+0000","last_fullsized":"2024-03-06T07:23:43.748252+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:54.936995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.101971+0000","last_change":"2024-03-06T07:23:42.108304+0000","last_active":"2024-03-06T07:23:43.101971+0000","last_peered":"2024-03-06T07:23:43.101971+0000","last_clean":"2024-03-06T07:23:43.101971+0000","last_became_active":"2024-03-06T07:23:42.107999+0000","last_became_peered":"2024-03-06T07:23:42.107999+0000","last_unstale":"2024-03-06T07:23:43.101971+0000","last_undegraded":"2024-03-06T07:23:43.101971+0000","last_fullsized":"2024-03-06T07:23:43.101971+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:44:26.683785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748044+0000","last_change":"2024-03-06T07:23:42.103822+0000","last_active":"2024-03-06T07:23:43.748044+0000","last_peered":"2024-03-06T07:23:43.748044+0000","last_clean":"2024-03-06T07:23:43.748044+0000","last_became_active":"2024-03-06T07:23:42.103493+0000","last_became_peered":"2024-03-06T07:23:42.103493+0000","last_unstale":"2024-03-06T07:23:43.748044+0000","last_undegraded":"2024-03-06T07:23:43.748044+0000","last_fullsized":"2024-03-06T07:23:43.748044+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:41.381834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748718+0000","last_change":"2024-03-06T07:23:42.106167+0000","last_active":"2024-03-06T07:23:43.748718+0000","last_peered":"2024-03-06T07:23:43.748718+0000","last_clean":"2024-03-06T07:23:43.748718+0000","last_became_active":"2024-03-06T07:23:42.105894+0000","last_became_peered":"2024-03-06T07:23:42.105894+0000","last_unstale":"2024-03-06T07:23:43.748718+0000","last_undegraded":"2024-03-06T07:23:43.748718+0000","last_fullsized":"2024-03-06T07:23:43.748718+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:17:46.084647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.102843+0000","last_change":"2024-03-06T07:23:42.106702+0000","last_active":"2024-03-06T07:23:43.102843+0000","last_peered":"2024-03-06T07:23:43.102843+0000","last_clean":"2024-03-06T07:23:43.102843+0000","last_became_active":"2024-03-06T07:23:42.106435+0000","last_became_peered":"2024-03-06T07:23:42.106435+0000","last_unstale":"2024-03-06T07:23:43.102843+0000","last_undegraded":"2024-03-06T07:23:43.102843+0000","last_fullsized":"2024-03-06T07:23:43.102843+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:14.675455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.103160+0000","last_change":"2024-03-06T07:23:42.099748+0000","last_active":"2024-03-06T07:23:43.103160+0000","last_peered":"2024-03-06T07:23:43.103160+0000","last_clean":"2024-03-06T07:23:43.103160+0000","last_became_active":"2024-03-06T07:23:42.099485+0000","last_became_peered":"2024-03-06T07:23:42.099485+0000","last_unstale":"2024-03-06T07:23:43.103160+0000","last_undegraded":"2024-03-06T07:23:43.103160+0000","last_fullsized":"2024-03-06T07:23:43.103160+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:59:10.744765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.103107+0000","last_change":"2024-03-06T07:23:42.114674+0000","last_active":"2024-03-06T07:23:43.103107+0000","last_peered":"2024-03-06T07:23:43.103107+0000","last_clean":"2024-03-06T07:23:43.103107+0000","last_became_active":"2024-03-06T07:23:42.114375+0000","last_became_peered":"2024-03-06T07:23:42.114375+0000","last_unstale":"2024-03-06T07:23:43.103107+0000","last_undegraded":"2024-03-06T07:23:43.103107+0000","last_fullsized":"2024-03-06T07:23:43.103107+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:49:52.564351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.103332+0000","last_change":"2024-03-06T07:23:42.105556+0000","last_active":"2024-03-06T07:23:43.103332+0000","last_peered":"2024-03-06T07:23:43.103332+0000","last_clean":"2024-03-06T07:23:43.103332+0000","last_became_active":"2024-03-06T07:23:42.105235+0000","last_became_peered":"2024-03-06T07:23:42.105235+0000","last_unstale":"2024-03-06T07:23:43.103332+0000","last_undegraded":"2024-03-06T07:23:43.103332+0000","last_fullsized":"2024-03-06T07:23:43.103332+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:26.696304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.102242+0000","last_change":"2024-03-06T07:23:42.107082+0000","last_active":"2024-03-06T07:23:43.102242+0000","last_peered":"2024-03-06T07:23:43.102242+0000","last_clean":"2024-03-06T07:23:43.102242+0000","last_became_active":"2024-03-06T07:23:42.106795+0000","last_became_peered":"2024-03-06T07:23:42.106795+0000","last_unstale":"2024-03-06T07:23:43.102242+0000","last_undegraded":"2024-03-06T07:23:43.102242+0000","last_fullsized":"2024-03-06T07:23:43.102242+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:28.996958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748539+0000","last_change":"2024-03-06T07:23:42.105880+0000","last_active":"2024-03-06T07:23:43.748539+0000","last_peered":"2024-03-06T07:23:43.748539+0000","last_clean":"2024-03-06T07:23:43.748539+0000","last_became_active":"2024-03-06T07:23:42.105503+0000","last_became_peered":"2024-03-06T07:23:42.105503+0000","last_unstale":"2024-03-06T07:23:43.748539+0000","last_undegraded":"2024-03-06T07:23:43.748539+0000","last_fullsized":"2024-03-06T07:23:43.748539+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:07.956361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748182+0000","last_change":"2024-03-06T07:23:42.110836+0000","last_active":"2024-03-06T07:23:43.748182+0000","last_peered":"2024-03-06T07:23:43.748182+0000","last_clean":"2024-03-06T07:23:43.748182+0000","last_became_active":"2024-03-06T07:23:42.110557+0000","last_became_peered":"2024-03-06T07:23:42.110557+0000","last_unstale":"2024-03-06T07:23:43.748182+0000","last_undegraded":"2024-03-06T07:23:43.748182+0000","last_fullsized":"2024-03-06T07:23:43.748182+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:58.160523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.103735+0000","last_change":"2024-03-06T07:23:42.098866+0000","last_active":"2024-03-06T07:23:43.103735+0000","last_peered":"2024-03-06T07:23:43.103735+0000","last_clean":"2024-03-06T07:23:43.103735+0000","last_became_active":"2024-03-06T07:23:42.097444+0000","last_became_peered":"2024-03-06T07:23:42.097444+0000","last_unstale":"2024-03-06T07:23:43.103735+0000","last_undegraded":"2024-03-06T07:23:43.103735+0000","last_fullsized":"2024-03-06T07:23:43.103735+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:04:47.346959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.102977+0000","last_change":"2024-03-06T07:23:42.095878+0000","last_active":"2024-03-06T07:23:43.102977+0000","last_peered":"2024-03-06T07:23:43.102977+0000","last_clean":"2024-03-06T07:23:43.102977+0000","last_became_active":"2024-03-06T07:23:42.095597+0000","last_became_peered":"2024-03-06T07:23:42.095597+0000","last_unstale":"2024-03-06T07:23:43.102977+0000","last_undegraded":"2024-03-06T07:23:43.102977+0000","last_fullsized":"2024-03-06T07:23:43.102977+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:35.837317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.103480+0000","last_change":"2024-03-06T07:23:42.102398+0000","last_active":"2024-03-06T07:23:43.103480+0000","last_peered":"2024-03-06T07:23:43.103480+0000","last_clean":"2024-03-06T07:23:43.103480+0000","last_became_active":"2024-03-06T07:23:42.099793+0000","last_became_peered":"2024-03-06T07:23:42.099793+0000","last_unstale":"2024-03-06T07:23:43.103480+0000","last_undegraded":"2024-03-06T07:23:43.103480+0000","last_fullsized":"2024-03-06T07:23:43.103480+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:01:39.323391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.101896+0000","last_change":"2024-03-06T07:23:42.109996+0000","last_active":"2024-03-06T07:23:43.101896+0000","last_peered":"2024-03-06T07:23:43.101896+0000","last_clean":"2024-03-06T07:23:43.101896+0000","last_became_active":"2024-03-06T07:23:42.109714+0000","last_became_peered":"2024-03-06T07:23:42.109714+0000","last_unstale":"2024-03-06T07:23:43.101896+0000","last_undegraded":"2024-03-06T07:23:43.101896+0000","last_fullsized":"2024-03-06T07:23:43.101896+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:38.452235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.102840+0000","last_change":"2024-03-06T07:23:42.099076+0000","last_active":"2024-03-06T07:23:43.102840+0000","last_peered":"2024-03-06T07:23:43.102840+0000","last_clean":"2024-03-06T07:23:43.102840+0000","last_became_active":"2024-03-06T07:23:42.098775+0000","last_became_peered":"2024-03-06T07:23:42.098775+0000","last_unstale":"2024-03-06T07:23:43.102840+0000","last_undegraded":"2024-03-06T07:23:43.102840+0000","last_fullsized":"2024-03-06T07:23:43.102840+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:58.160523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748495+0000","last_change":"2024-03-06T07:23:42.103218+0000","last_active":"2024-03-06T07:23:43.748495+0000","last_peered":"2024-03-06T07:23:43.748495+0000","last_clean":"2024-03-06T07:23:43.748495+0000","last_became_active":"2024-03-06T07:23:42.102222+0000","last_became_peered":"2024-03-06T07:23:42.102222+0000","last_unstale":"2024-03-06T07:23:43.748495+0000","last_undegraded":"2024-03-06T07:23:43.748495+0000","last_fullsized":"2024-03-06T07:23:43.748495+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:51.678792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.102736+0000","last_change":"2024-03-06T07:23:42.100064+0000","last_active":"2024-03-06T07:23:43.102736+0000","last_peered":"2024-03-06T07:23:43.102736+0000","last_clean":"2024-03-06T07:23:43.102736+0000","last_became_active":"2024-03-06T07:23:42.099719+0000","last_became_peered":"2024-03-06T07:23:42.099719+0000","last_unstale":"2024-03-06T07:23:43.102736+0000","last_undegraded":"2024-03-06T07:23:43.102736+0000","last_fullsized":"2024-03-06T07:23:43.102736+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:04.000572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.103024+0000","last_change":"2024-03-06T07:23:42.100949+0000","last_active":"2024-03-06T07:23:43.103024+0000","last_peered":"2024-03-06T07:23:43.103024+0000","last_clean":"2024-03-06T07:23:43.103024+0000","last_became_active":"2024-03-06T07:23:42.100593+0000","last_became_peered":"2024-03-06T07:23:42.100593+0000","last_unstale":"2024-03-06T07:23:43.103024+0000","last_undegraded":"2024-03-06T07:23:43.103024+0000","last_fullsized":"2024-03-06T07:23:43.103024+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:34:33.726443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748842+0000","last_change":"2024-03-06T07:23:42.104004+0000","last_active":"2024-03-06T07:23:43.748842+0000","last_peered":"2024-03-06T07:23:43.748842+0000","last_clean":"2024-03-06T07:23:43.748842+0000","last_became_active":"2024-03-06T07:23:42.103629+0000","last_became_peered":"2024-03-06T07:23:42.103629+0000","last_unstale":"2024-03-06T07:23:43.748842+0000","last_undegraded":"2024-03-06T07:23:43.748842+0000","last_fullsized":"2024-03-06T07:23:43.748842+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:42:14.658158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.101940+0000","last_change":"2024-03-06T07:23:42.099172+0000","last_active":"2024-03-06T07:23:43.101940+0000","last_peered":"2024-03-06T07:23:43.101940+0000","last_clean":"2024-03-06T07:23:43.101940+0000","last_became_active":"2024-03-06T07:23:42.098888+0000","last_became_peered":"2024-03-06T07:23:42.098888+0000","last_unstale":"2024-03-06T07:23:43.101940+0000","last_undegraded":"2024-03-06T07:23:43.101940+0000","last_fullsized":"2024-03-06T07:23:43.101940+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:08.444552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748780+0000","last_change":"2024-03-06T07:23:42.111685+0000","last_active":"2024-03-06T07:23:43.748780+0000","last_peered":"2024-03-06T07:23:43.748780+0000","last_clean":"2024-03-06T07:23:43.748780+0000","last_became_active":"2024-03-06T07:23:42.110757+0000","last_became_peered":"2024-03-06T07:23:42.110757+0000","last_unstale":"2024-03-06T07:23:43.748780+0000","last_undegraded":"2024-03-06T07:23:43.748780+0000","last_fullsized":"2024-03-06T07:23:43.748780+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:01:39.323391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"44'4","reported_seq":18,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748124+0000","last_change":"2024-03-06T07:23:41.102362+0000","last_active":"2024-03-06T07:23:43.748124+0000","last_peered":"2024-03-06T07:23:43.748124+0000","last_clean":"2024-03-06T07:23:43.748124+0000","last_became_active":"2024-03-06T07:23:41.102058+0000","last_became_peered":"2024-03-06T07:23:41.102058+0000","last_unstale":"2024-03-06T07:23:43.748124+0000","last_undegraded":"2024-03-06T07:23:43.748124+0000","last_fullsized":"2024-03-06T07:23:43.748124+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:24.224264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.103214+0000","last_change":"2024-03-06T07:23:42.090875+0000","last_active":"2024-03-06T07:23:43.103214+0000","last_peered":"2024-03-06T07:23:43.103214+0000","last_clean":"2024-03-06T07:23:43.103214+0000","last_became_active":"2024-03-06T07:23:42.090581+0000","last_became_peered":"2024-03-06T07:23:42.090581+0000","last_unstale":"2024-03-06T07:23:43.103214+0000","last_undegraded":"2024-03-06T07:23:43.103214+0000","last_fullsized":"2024-03-06T07:23:43.103214+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:32:21.061612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.103170+0000","last_change":"2024-03-06T07:23:41.102284+0000","last_active":"2024-03-06T07:23:43.103170+0000","last_peered":"2024-03-06T07:23:43.103170+0000","last_clean":"2024-03-06T07:23:43.103170+0000","last_became_active":"2024-03-06T07:23:41.102011+0000","last_became_peered":"2024-03-06T07:23:41.102011+0000","last_unstale":"2024-03-06T07:23:43.103170+0000","last_undegraded":"2024-03-06T07:23:43.103170+0000","last_fullsized":"2024-03-06T07:23:43.103170+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:02.993834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748584+0000","last_change":"2024-03-06T07:23:42.107954+0000","last_active":"2024-03-06T07:23:43.748584+0000","last_peered":"2024-03-06T07:23:43.748584+0000","last_clean":"2024-03-06T07:23:43.748584+0000","last_became_active":"2024-03-06T07:23:42.107689+0000","last_became_peered":"2024-03-06T07:23:42.107689+0000","last_unstale":"2024-03-06T07:23:43.748584+0000","last_undegraded":"2024-03-06T07:23:43.748584+0000","last_fullsized":"2024-03-06T07:23:43.748584+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:47:34.961757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748453+0000","last_change":"2024-03-06T07:23:41.101648+0000","last_active":"2024-03-06T07:23:43.748453+0000","last_peered":"2024-03-06T07:23:43.748453+0000","last_clean":"2024-03-06T07:23:43.748453+0000","last_became_active":"2024-03-06T07:23:41.101345+0000","last_became_peered":"2024-03-06T07:23:41.101345+0000","last_unstale":"2024-03-06T07:23:43.748453+0000","last_undegraded":"2024-03-06T07:23:43.748453+0000","last_fullsized":"2024-03-06T07:23:43.748453+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:34.830579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748403+0000","last_change":"2024-03-06T07:23:42.100513+0000","last_active":"2024-03-06T07:23:43.748403+0000","last_peered":"2024-03-06T07:23:43.748403+0000","last_clean":"2024-03-06T07:23:43.748403+0000","last_became_active":"2024-03-06T07:23:42.100233+0000","last_became_peered":"2024-03-06T07:23:42.100233+0000","last_unstale":"2024-03-06T07:23:43.748403+0000","last_undegraded":"2024-03-06T07:23:43.748403+0000","last_fullsized":"2024-03-06T07:23:43.748403+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:58.192125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.749120+0000","last_change":"2024-03-06T07:23:41.100199+0000","last_active":"2024-03-06T07:23:43.749120+0000","last_peered":"2024-03-06T07:23:43.749120+0000","last_clean":"2024-03-06T07:23:43.749120+0000","last_became_active":"2024-03-06T07:23:41.099943+0000","last_became_peered":"2024-03-06T07:23:41.099943+0000","last_unstale":"2024-03-06T07:23:43.749120+0000","last_undegraded":"2024-03-06T07:23:43.749120+0000","last_fullsized":"2024-03-06T07:23:43.749120+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:49:51.557613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.102646+0000","last_change":"2024-03-06T07:23:42.090685+0000","last_active":"2024-03-06T07:23:43.102646+0000","last_peered":"2024-03-06T07:23:43.102646+0000","last_clean":"2024-03-06T07:23:43.102646+0000","last_became_active":"2024-03-06T07:23:42.090184+0000","last_became_peered":"2024-03-06T07:23:42.090184+0000","last_unstale":"2024-03-06T07:23:43.102646+0000","last_undegraded":"2024-03-06T07:23:43.102646+0000","last_fullsized":"2024-03-06T07:23:43.102646+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:31:33.587498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748595+0000","last_change":"2024-03-06T07:23:41.096923+0000","last_active":"2024-03-06T07:23:43.748595+0000","last_peered":"2024-03-06T07:23:43.748595+0000","last_clean":"2024-03-06T07:23:43.748595+0000","last_became_active":"2024-03-06T07:23:41.096352+0000","last_became_peered":"2024-03-06T07:23:41.096352+0000","last_unstale":"2024-03-06T07:23:43.748595+0000","last_undegraded":"2024-03-06T07:23:43.748595+0000","last_fullsized":"2024-03-06T07:23:43.748595+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:57:45.844000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748855+0000","last_change":"2024-03-06T07:23:42.115530+0000","last_active":"2024-03-06T07:23:43.748855+0000","last_peered":"2024-03-06T07:23:43.748855+0000","last_clean":"2024-03-06T07:23:43.748855+0000","last_became_active":"2024-03-06T07:23:42.115252+0000","last_became_peered":"2024-03-06T07:23:42.115252+0000","last_unstale":"2024-03-06T07:23:43.748855+0000","last_undegraded":"2024-03-06T07:23:43.748855+0000","last_fullsized":"2024-03-06T07:23:43.748855+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:59:10.744765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.102718+0000","last_change":"2024-03-06T07:23:41.106716+0000","last_active":"2024-03-06T07:23:43.102718+0000","last_peered":"2024-03-06T07:23:43.102718+0000","last_clean":"2024-03-06T07:23:43.102718+0000","last_became_active":"2024-03-06T07:23:41.106375+0000","last_became_peered":"2024-03-06T07:23:41.106375+0000","last_unstale":"2024-03-06T07:23:43.102718+0000","last_undegraded":"2024-03-06T07:23:43.102718+0000","last_fullsized":"2024-03-06T07:23:43.102718+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:34.830579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748015+0000","last_change":"2024-03-06T07:23:42.112489+0000","last_active":"2024-03-06T07:23:43.748015+0000","last_peered":"2024-03-06T07:23:43.748015+0000","last_clean":"2024-03-06T07:23:43.748015+0000","last_became_active":"2024-03-06T07:23:42.111597+0000","last_became_peered":"2024-03-06T07:23:42.111597+0000","last_unstale":"2024-03-06T07:23:43.748015+0000","last_undegraded":"2024-03-06T07:23:43.748015+0000","last_fullsized":"2024-03-06T07:23:43.748015+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:31:33.587498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748017+0000","last_change":"2024-03-06T07:23:41.099133+0000","last_active":"2024-03-06T07:23:43.748017+0000","last_peered":"2024-03-06T07:23:43.748017+0000","last_clean":"2024-03-06T07:23:43.748017+0000","last_became_active":"2024-03-06T07:23:41.098838+0000","last_became_peered":"2024-03-06T07:23:41.098838+0000","last_unstale":"2024-03-06T07:23:43.748017+0000","last_undegraded":"2024-03-06T07:23:43.748017+0000","last_fullsized":"2024-03-06T07:23:43.748017+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:57.153785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.103521+0000","last_change":"2024-03-06T07:23:42.099119+0000","last_active":"2024-03-06T07:23:43.103521+0000","last_peered":"2024-03-06T07:23:43.103521+0000","last_clean":"2024-03-06T07:23:43.103521+0000","last_became_active":"2024-03-06T07:23:42.097984+0000","last_became_peered":"2024-03-06T07:23:42.097984+0000","last_unstale":"2024-03-06T07:23:43.103521+0000","last_undegraded":"2024-03-06T07:23:43.103521+0000","last_fullsized":"2024-03-06T07:23:43.103521+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:33:48.156540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748064+0000","last_change":"2024-03-06T07:23:41.095707+0000","last_active":"2024-03-06T07:23:43.748064+0000","last_peered":"2024-03-06T07:23:43.748064+0000","last_clean":"2024-03-06T07:23:43.748064+0000","last_became_active":"2024-03-06T07:23:41.095430+0000","last_became_peered":"2024-03-06T07:23:41.095430+0000","last_unstale":"2024-03-06T07:23:43.748064+0000","last_undegraded":"2024-03-06T07:23:43.748064+0000","last_fullsized":"2024-03-06T07:23:43.748064+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:44:25.677046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.103089+0000","last_change":"2024-03-06T07:23:42.114516+0000","last_active":"2024-03-06T07:23:43.103089+0000","last_peered":"2024-03-06T07:23:43.103089+0000","last_clean":"2024-03-06T07:23:43.103089+0000","last_became_active":"2024-03-06T07:23:42.114282+0000","last_became_peered":"2024-03-06T07:23:42.114282+0000","last_unstale":"2024-03-06T07:23:43.103089+0000","last_undegraded":"2024-03-06T07:23:43.103089+0000","last_fullsized":"2024-03-06T07:23:43.103089+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:45:58.613060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.103083+0000","last_change":"2024-03-06T07:23:41.100731+0000","last_active":"2024-03-06T07:23:43.103083+0000","last_peered":"2024-03-06T07:23:43.103083+0000","last_clean":"2024-03-06T07:23:43.103083+0000","last_became_active":"2024-03-06T07:23:41.100431+0000","last_became_peered":"2024-03-06T07:23:41.100431+0000","last_unstale":"2024-03-06T07:23:43.103083+0000","last_undegraded":"2024-03-06T07:23:43.103083+0000","last_fullsized":"2024-03-06T07:23:43.103083+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:49.562199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.749423+0000","last_change":"2024-03-06T07:23:42.099151+0000","last_active":"2024-03-06T07:23:43.749423+0000","last_peered":"2024-03-06T07:23:43.749423+0000","last_clean":"2024-03-06T07:23:43.749423+0000","last_became_active":"2024-03-06T07:23:42.098887+0000","last_became_peered":"2024-03-06T07:23:42.098887+0000","last_unstale":"2024-03-06T07:23:43.749423+0000","last_undegraded":"2024-03-06T07:23:43.749423+0000","last_fullsized":"2024-03-06T07:23:43.749423+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:22.697354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.749619+0000","last_change":"2024-03-06T07:23:41.098118+0000","last_active":"2024-03-06T07:23:43.749619+0000","last_peered":"2024-03-06T07:23:43.749619+0000","last_clean":"2024-03-06T07:23:43.749619+0000","last_became_active":"2024-03-06T07:23:41.097846+0000","last_became_peered":"2024-03-06T07:23:41.097846+0000","last_unstale":"2024-03-06T07:23:43.749619+0000","last_undegraded":"2024-03-06T07:23:43.749619+0000","last_fullsized":"2024-03-06T07:23:43.749619+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:29:44.897241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748322+0000","last_change":"2024-03-06T07:23:42.106067+0000","last_active":"2024-03-06T07:23:43.748322+0000","last_peered":"2024-03-06T07:23:43.748322+0000","last_clean":"2024-03-06T07:23:43.748322+0000","last_became_active":"2024-03-06T07:23:42.105700+0000","last_became_peered":"2024-03-06T07:23:42.105700+0000","last_unstale":"2024-03-06T07:23:43.748322+0000","last_undegraded":"2024-03-06T07:23:43.748322+0000","last_fullsized":"2024-03-06T07:23:43.748322+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:36:39.813573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"44'3","reported_seq":17,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.749370+0000","last_change":"2024-03-06T07:23:41.102169+0000","last_active":"2024-03-06T07:23:43.749370+0000","last_peered":"2024-03-06T07:23:43.749370+0000","last_clean":"2024-03-06T07:23:43.749370+0000","last_became_active":"2024-03-06T07:23:41.101887+0000","last_became_peered":"2024-03-06T07:23:41.101887+0000","last_unstale":"2024-03-06T07:23:43.749370+0000","last_undegraded":"2024-03-06T07:23:43.749370+0000","last_fullsized":"2024-03-06T07:23:43.749370+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:04:26.847954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.749268+0000","last_change":"2024-03-06T07:23:42.100912+0000","last_active":"2024-03-06T07:23:43.749268+0000","last_peered":"2024-03-06T07:23:43.749268+0000","last_clean":"2024-03-06T07:23:43.749268+0000","last_became_active":"2024-03-06T07:23:42.100585+0000","last_became_peered":"2024-03-06T07:23:42.100585+0000","last_unstale":"2024-03-06T07:23:43.749268+0000","last_undegraded":"2024-03-06T07:23:43.749268+0000","last_fullsized":"2024-03-06T07:23:43.749268+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:31.905624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"44'6","reported_seq":20,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.102192+0000","last_change":"2024-03-06T07:23:41.099116+0000","last_active":"2024-03-06T07:23:43.102192+0000","last_peered":"2024-03-06T07:23:43.102192+0000","last_clean":"2024-03-06T07:23:43.102192+0000","last_became_active":"2024-03-06T07:23:41.098778+0000","last_became_peered":"2024-03-06T07:23:41.098778+0000","last_unstale":"2024-03-06T07:23:43.102192+0000","last_undegraded":"2024-03-06T07:23:43.102192+0000","last_fullsized":"2024-03-06T07:23:43.102192+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:20.126721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.102798+0000","last_change":"2024-03-06T07:23:42.105793+0000","last_active":"2024-03-06T07:23:43.102798+0000","last_peered":"2024-03-06T07:23:43.102798+0000","last_clean":"2024-03-06T07:23:43.102798+0000","last_became_active":"2024-03-06T07:23:42.105536+0000","last_became_peered":"2024-03-06T07:23:42.105536+0000","last_unstale":"2024-03-06T07:23:43.102798+0000","last_undegraded":"2024-03-06T07:23:43.102798+0000","last_fullsized":"2024-03-06T07:23:43.102798+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:14:24.834163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748688+0000","last_change":"2024-03-06T07:23:41.096739+0000","last_active":"2024-03-06T07:23:43.748688+0000","last_peered":"2024-03-06T07:23:43.748688+0000","last_clean":"2024-03-06T07:23:43.748688+0000","last_became_active":"2024-03-06T07:23:41.096035+0000","last_became_peered":"2024-03-06T07:23:41.096035+0000","last_unstale":"2024-03-06T07:23:43.748688+0000","last_undegraded":"2024-03-06T07:23:43.748688+0000","last_fullsized":"2024-03-06T07:23:43.748688+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:53.930256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748996+0000","last_change":"2024-03-06T07:23:42.112674+0000","last_active":"2024-03-06T07:23:43.748996+0000","last_peered":"2024-03-06T07:23:43.748996+0000","last_clean":"2024-03-06T07:23:43.748996+0000","last_became_active":"2024-03-06T07:23:42.112325+0000","last_became_peered":"2024-03-06T07:23:42.112325+0000","last_unstale":"2024-03-06T07:23:43.748996+0000","last_undegraded":"2024-03-06T07:23:43.748996+0000","last_fullsized":"2024-03-06T07:23:43.748996+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:35:56.747430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.749066+0000","last_change":"2024-03-06T07:23:41.097571+0000","last_active":"2024-03-06T07:23:43.749066+0000","last_peered":"2024-03-06T07:23:43.749066+0000","last_clean":"2024-03-06T07:23:43.749066+0000","last_became_active":"2024-03-06T07:23:41.097282+0000","last_became_peered":"2024-03-06T07:23:41.097282+0000","last_unstale":"2024-03-06T07:23:43.749066+0000","last_undegraded":"2024-03-06T07:23:43.749066+0000","last_fullsized":"2024-03-06T07:23:43.749066+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:00.236121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.747967+0000","last_change":"2024-03-06T07:23:42.103662+0000","last_active":"2024-03-06T07:23:43.747967+0000","last_peered":"2024-03-06T07:23:43.747967+0000","last_clean":"2024-03-06T07:23:43.747967+0000","last_became_active":"2024-03-06T07:23:42.103324+0000","last_became_peered":"2024-03-06T07:23:42.103324+0000","last_unstale":"2024-03-06T07:23:43.747967+0000","last_undegraded":"2024-03-06T07:23:43.747967+0000","last_fullsized":"2024-03-06T07:23:43.747967+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:13:30.550460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748882+0000","last_change":"2024-03-06T07:23:41.094487+0000","last_active":"2024-03-06T07:23:43.748882+0000","last_peered":"2024-03-06T07:23:43.748882+0000","last_clean":"2024-03-06T07:23:43.748882+0000","last_became_active":"2024-03-06T07:23:41.094218+0000","last_became_peered":"2024-03-06T07:23:41.094218+0000","last_unstale":"2024-03-06T07:23:43.748882+0000","last_undegraded":"2024-03-06T07:23:43.748882+0000","last_fullsized":"2024-03-06T07:23:43.748882+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:35:55.740692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.749149+0000","last_change":"2024-03-06T07:23:42.108171+0000","last_active":"2024-03-06T07:23:43.749149+0000","last_peered":"2024-03-06T07:23:43.749149+0000","last_clean":"2024-03-06T07:23:43.749149+0000","last_became_active":"2024-03-06T07:23:42.107811+0000","last_became_peered":"2024-03-06T07:23:42.107811+0000","last_unstale":"2024-03-06T07:23:43.749149+0000","last_undegraded":"2024-03-06T07:23:43.749149+0000","last_fullsized":"2024-03-06T07:23:43.749149+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:35.437029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"44'8","reported_seq":22,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748158+0000","last_change":"2024-03-06T07:23:41.097316+0000","last_active":"2024-03-06T07:23:43.748158+0000","last_peered":"2024-03-06T07:23:43.748158+0000","last_clean":"2024-03-06T07:23:43.748158+0000","last_became_active":"2024-03-06T07:23:41.097051+0000","last_became_peered":"2024-03-06T07:23:41.097051+0000","last_unstale":"2024-03-06T07:23:43.748158+0000","last_undegraded":"2024-03-06T07:23:43.748158+0000","last_fullsized":"2024-03-06T07:23:43.748158+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:09.775160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.102064+0000","last_change":"2024-03-06T07:23:42.100837+0000","last_active":"2024-03-06T07:23:43.102064+0000","last_peered":"2024-03-06T07:23:43.102064+0000","last_clean":"2024-03-06T07:23:43.102064+0000","last_became_active":"2024-03-06T07:23:42.100538+0000","last_became_peered":"2024-03-06T07:23:42.100538+0000","last_unstale":"2024-03-06T07:23:43.102064+0000","last_undegraded":"2024-03-06T07:23:43.102064+0000","last_fullsized":"2024-03-06T07:23:43.102064+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:10.781898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.102114+0000","last_change":"2024-03-06T07:23:41.090841+0000","last_active":"2024-03-06T07:23:43.102114+0000","last_peered":"2024-03-06T07:23:43.102114+0000","last_clean":"2024-03-06T07:23:43.102114+0000","last_became_active":"2024-03-06T07:23:41.090530+0000","last_became_peered":"2024-03-06T07:23:41.090530+0000","last_unstale":"2024-03-06T07:23:43.102114+0000","last_undegraded":"2024-03-06T07:23:43.102114+0000","last_fullsized":"2024-03-06T07:23:43.102114+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:08.213301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.749115+0000","last_change":"2024-03-06T07:23:42.114734+0000","last_active":"2024-03-06T07:23:43.749115+0000","last_peered":"2024-03-06T07:23:43.749115+0000","last_clean":"2024-03-06T07:23:43.749115+0000","last_became_active":"2024-03-06T07:23:42.114436+0000","last_became_peered":"2024-03-06T07:23:42.114436+0000","last_unstale":"2024-03-06T07:23:43.749115+0000","last_undegraded":"2024-03-06T07:23:43.749115+0000","last_fullsized":"2024-03-06T07:23:43.749115+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:01.242859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.103237+0000","last_change":"2024-03-06T07:23:41.098489+0000","last_active":"2024-03-06T07:23:43.103237+0000","last_peered":"2024-03-06T07:23:43.103237+0000","last_clean":"2024-03-06T07:23:43.103237+0000","last_became_active":"2024-03-06T07:23:41.098182+0000","last_became_peered":"2024-03-06T07:23:41.098182+0000","last_unstale":"2024-03-06T07:23:43.103237+0000","last_undegraded":"2024-03-06T07:23:43.103237+0000","last_fullsized":"2024-03-06T07:23:43.103237+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:14:23.827424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.103012+0000","last_change":"2024-03-06T07:23:42.108706+0000","last_active":"2024-03-06T07:23:43.103012+0000","last_peered":"2024-03-06T07:23:43.103012+0000","last_clean":"2024-03-06T07:23:43.103012+0000","last_became_active":"2024-03-06T07:23:42.108373+0000","last_became_peered":"2024-03-06T07:23:42.108373+0000","last_unstale":"2024-03-06T07:23:43.103012+0000","last_undegraded":"2024-03-06T07:23:43.103012+0000","last_fullsized":"2024-03-06T07:23:43.103012+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:50.568937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748306+0000","last_change":"2024-03-06T07:23:41.100506+0000","last_active":"2024-03-06T07:23:43.748306+0000","last_peered":"2024-03-06T07:23:43.748306+0000","last_clean":"2024-03-06T07:23:43.748306+0000","last_became_active":"2024-03-06T07:23:41.099926+0000","last_became_peered":"2024-03-06T07:23:41.099926+0000","last_unstale":"2024-03-06T07:23:43.748306+0000","last_undegraded":"2024-03-06T07:23:43.748306+0000","last_fullsized":"2024-03-06T07:23:43.748306+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:45:57.606322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.103086+0000","last_change":"2024-03-06T07:23:42.100948+0000","last_active":"2024-03-06T07:23:43.103086+0000","last_peered":"2024-03-06T07:23:43.103086+0000","last_clean":"2024-03-06T07:23:43.103086+0000","last_became_active":"2024-03-06T07:23:42.100226+0000","last_became_peered":"2024-03-06T07:23:42.100226+0000","last_unstale":"2024-03-06T07:23:43.103086+0000","last_undegraded":"2024-03-06T07:23:43.103086+0000","last_fullsized":"2024-03-06T07:23:43.103086+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:10:31.867655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748369+0000","last_change":"2024-03-06T07:23:41.102819+0000","last_active":"2024-03-06T07:23:43.748369+0000","last_peered":"2024-03-06T07:23:43.748369+0000","last_clean":"2024-03-06T07:23:43.748369+0000","last_became_active":"2024-03-06T07:23:41.102522+0000","last_became_peered":"2024-03-06T07:23:41.102522+0000","last_unstale":"2024-03-06T07:23:43.748369+0000","last_undegraded":"2024-03-06T07:23:43.748369+0000","last_fullsized":"2024-03-06T07:23:43.748369+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:00.236121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748906+0000","last_change":"2024-03-06T07:23:42.109387+0000","last_active":"2024-03-06T07:23:43.748906+0000","last_peered":"2024-03-06T07:23:43.748906+0000","last_clean":"2024-03-06T07:23:43.748906+0000","last_became_active":"2024-03-06T07:23:42.109118+0000","last_became_peered":"2024-03-06T07:23:42.109118+0000","last_unstale":"2024-03-06T07:23:43.748906+0000","last_undegraded":"2024-03-06T07:23:43.748906+0000","last_fullsized":"2024-03-06T07:23:43.748906+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:25:38.596240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748839+0000","last_change":"2024-03-06T07:23:41.095585+0000","last_active":"2024-03-06T07:23:43.748839+0000","last_peered":"2024-03-06T07:23:43.748839+0000","last_clean":"2024-03-06T07:23:43.748839+0000","last_became_active":"2024-03-06T07:23:41.095341+0000","last_became_peered":"2024-03-06T07:23:41.095341+0000","last_unstale":"2024-03-06T07:23:43.748839+0000","last_undegraded":"2024-03-06T07:23:43.748839+0000","last_fullsized":"2024-03-06T07:23:43.748839+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:11.030371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.747920+0000","last_change":"2024-03-06T07:23:42.099069+0000","last_active":"2024-03-06T07:23:43.747920+0000","last_peered":"2024-03-06T07:23:43.747920+0000","last_clean":"2024-03-06T07:23:43.747920+0000","last_became_active":"2024-03-06T07:23:42.098793+0000","last_became_peered":"2024-03-06T07:23:42.098793+0000","last_unstale":"2024-03-06T07:23:43.747920+0000","last_undegraded":"2024-03-06T07:23:43.747920+0000","last_fullsized":"2024-03-06T07:23:43.747920+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:26.696304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748041+0000","last_change":"2024-03-06T07:23:41.099634+0000","last_active":"2024-03-06T07:23:43.748041+0000","last_peered":"2024-03-06T07:23:43.748041+0000","last_clean":"2024-03-06T07:23:43.748041+0000","last_became_active":"2024-03-06T07:23:41.099364+0000","last_became_peered":"2024-03-06T07:23:41.099364+0000","last_unstale":"2024-03-06T07:23:43.748041+0000","last_undegraded":"2024-03-06T07:23:43.748041+0000","last_fullsized":"2024-03-06T07:23:43.748041+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:34:32.719704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.101880+0000","last_change":"2024-03-06T07:23:42.095201+0000","last_active":"2024-03-06T07:23:43.101880+0000","last_peered":"2024-03-06T07:23:43.101880+0000","last_clean":"2024-03-06T07:23:43.101880+0000","last_became_active":"2024-03-06T07:23:42.093370+0000","last_became_peered":"2024-03-06T07:23:42.093370+0000","last_unstale":"2024-03-06T07:23:43.101880+0000","last_undegraded":"2024-03-06T07:23:43.101880+0000","last_fullsized":"2024-03-06T07:23:43.101880+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:31.574000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.749695+0000","last_change":"2024-03-06T07:23:41.096450+0000","last_active":"2024-03-06T07:23:43.749695+0000","last_peered":"2024-03-06T07:23:43.749695+0000","last_clean":"2024-03-06T07:23:43.749695+0000","last_became_active":"2024-03-06T07:23:41.096210+0000","last_became_peered":"2024-03-06T07:23:41.096210+0000","last_unstale":"2024-03-06T07:23:43.749695+0000","last_undegraded":"2024-03-06T07:23:43.749695+0000","last_fullsized":"2024-03-06T07:23:43.749695+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:07.437814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.749651+0000","last_change":"2024-03-06T07:23:42.104521+0000","last_active":"2024-03-06T07:23:43.749651+0000","last_peered":"2024-03-06T07:23:43.749651+0000","last_clean":"2024-03-06T07:23:43.749651+0000","last_became_active":"2024-03-06T07:23:42.104220+0000","last_became_peered":"2024-03-06T07:23:42.104220+0000","last_unstale":"2024-03-06T07:23:43.749651+0000","last_undegraded":"2024-03-06T07:23:43.749651+0000","last_fullsized":"2024-03-06T07:23:43.749651+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:57:46.850738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748263+0000","last_change":"2024-03-06T07:23:41.096092+0000","last_active":"2024-03-06T07:23:43.748263+0000","last_peered":"2024-03-06T07:23:43.748263+0000","last_clean":"2024-03-06T07:23:43.748263+0000","last_became_active":"2024-03-06T07:23:41.094371+0000","last_became_peered":"2024-03-06T07:23:41.094371+0000","last_unstale":"2024-03-06T07:23:43.748263+0000","last_undegraded":"2024-03-06T07:23:43.748263+0000","last_fullsized":"2024-03-06T07:23:43.748263+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:45:57.606322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.103337+0000","last_change":"2024-03-06T07:23:42.108169+0000","last_active":"2024-03-06T07:23:43.103337+0000","last_peered":"2024-03-06T07:23:43.103337+0000","last_clean":"2024-03-06T07:23:43.103337+0000","last_became_active":"2024-03-06T07:23:42.107066+0000","last_became_peered":"2024-03-06T07:23:42.107066+0000","last_unstale":"2024-03-06T07:23:43.103337+0000","last_undegraded":"2024-03-06T07:23:43.103337+0000","last_fullsized":"2024-03-06T07:23:43.103337+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:58.192125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"45'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.749128+0000","last_change":"2024-03-06T07:23:41.100601+0000","last_active":"2024-03-06T07:23:43.749128+0000","last_peered":"2024-03-06T07:23:43.749128+0000","last_clean":"2024-03-06T07:23:43.749128+0000","last_became_active":"2024-03-06T07:23:41.100365+0000","last_became_peered":"2024-03-06T07:23:41.100365+0000","last_unstale":"2024-03-06T07:23:43.749128+0000","last_undegraded":"2024-03-06T07:23:43.749128+0000","last_fullsized":"2024-03-06T07:23:43.749128+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:29:23.109319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748750+0000","last_change":"2024-03-06T07:23:42.107366+0000","last_active":"2024-03-06T07:23:43.748750+0000","last_peered":"2024-03-06T07:23:43.748750+0000","last_clean":"2024-03-06T07:23:43.748750+0000","last_became_active":"2024-03-06T07:23:42.107088+0000","last_became_peered":"2024-03-06T07:23:42.107088+0000","last_unstale":"2024-03-06T07:23:43.748750+0000","last_undegraded":"2024-03-06T07:23:43.748750+0000","last_fullsized":"2024-03-06T07:23:43.748750+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:49:52.564351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.749379+0000","last_change":"2024-03-06T07:23:41.102841+0000","last_active":"2024-03-06T07:23:43.749379+0000","last_peered":"2024-03-06T07:23:43.749379+0000","last_clean":"2024-03-06T07:23:43.749379+0000","last_became_active":"2024-03-06T07:23:41.102564+0000","last_became_peered":"2024-03-06T07:23:41.102564+0000","last_unstale":"2024-03-06T07:23:43.749379+0000","last_undegraded":"2024-03-06T07:23:43.749379+0000","last_fullsized":"2024-03-06T07:23:43.749379+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:35:55.740692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.749304+0000","last_change":"2024-03-06T07:23:42.102019+0000","last_active":"2024-03-06T07:23:43.749304+0000","last_peered":"2024-03-06T07:23:43.749304+0000","last_clean":"2024-03-06T07:23:43.749304+0000","last_became_active":"2024-03-06T07:23:42.101745+0000","last_became_peered":"2024-03-06T07:23:42.101745+0000","last_unstale":"2024-03-06T07:23:43.749304+0000","last_undegraded":"2024-03-06T07:23:43.749304+0000","last_fullsized":"2024-03-06T07:23:43.749304+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:29.723189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.747979+0000","last_change":"2024-03-06T07:23:41.106360+0000","last_active":"2024-03-06T07:23:43.747979+0000","last_peered":"2024-03-06T07:23:43.747979+0000","last_clean":"2024-03-06T07:23:43.747979+0000","last_became_active":"2024-03-06T07:23:41.106134+0000","last_became_peered":"2024-03-06T07:23:41.106134+0000","last_unstale":"2024-03-06T07:23:43.747979+0000","last_undegraded":"2024-03-06T07:23:43.747979+0000","last_fullsized":"2024-03-06T07:23:43.747979+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:13.668717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.749048+0000","last_change":"2024-03-06T07:23:42.104488+0000","last_active":"2024-03-06T07:23:43.749048+0000","last_peered":"2024-03-06T07:23:43.749048+0000","last_clean":"2024-03-06T07:23:43.749048+0000","last_became_active":"2024-03-06T07:23:42.103898+0000","last_became_peered":"2024-03-06T07:23:42.103898+0000","last_unstale":"2024-03-06T07:23:43.749048+0000","last_undegraded":"2024-03-06T07:23:43.749048+0000","last_fullsized":"2024-03-06T07:23:43.749048+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:40.424045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.749203+0000","last_change":"2024-03-06T07:23:41.099446+0000","last_active":"2024-03-06T07:23:43.749203+0000","last_peered":"2024-03-06T07:23:43.749203+0000","last_clean":"2024-03-06T07:23:43.749203+0000","last_became_active":"2024-03-06T07:23:41.099181+0000","last_became_peered":"2024-03-06T07:23:41.099181+0000","last_unstale":"2024-03-06T07:23:43.749203+0000","last_undegraded":"2024-03-06T07:23:43.749203+0000","last_fullsized":"2024-03-06T07:23:43.749203+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:59:09.738026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748556+0000","last_change":"2024-03-06T07:23:42.109438+0000","last_active":"2024-03-06T07:23:43.748556+0000","last_peered":"2024-03-06T07:23:43.748556+0000","last_clean":"2024-03-06T07:23:43.748556+0000","last_became_active":"2024-03-06T07:23:42.109158+0000","last_became_peered":"2024-03-06T07:23:42.109158+0000","last_unstale":"2024-03-06T07:23:43.748556+0000","last_undegraded":"2024-03-06T07:23:43.748556+0000","last_fullsized":"2024-03-06T07:23:43.748556+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:37.631094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.102832+0000","last_change":"2024-03-06T07:23:41.099994+0000","last_active":"2024-03-06T07:23:43.102832+0000","last_peered":"2024-03-06T07:23:43.102832+0000","last_clean":"2024-03-06T07:23:43.102832+0000","last_became_active":"2024-03-06T07:23:41.099731+0000","last_became_peered":"2024-03-06T07:23:41.099731+0000","last_unstale":"2024-03-06T07:23:43.102832+0000","last_undegraded":"2024-03-06T07:23:43.102832+0000","last_fullsized":"2024-03-06T07:23:43.102832+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:13.668717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748071+0000","last_change":"2024-03-06T07:23:42.102864+0000","last_active":"2024-03-06T07:23:43.748071+0000","last_peered":"2024-03-06T07:23:43.748071+0000","last_clean":"2024-03-06T07:23:43.748071+0000","last_became_active":"2024-03-06T07:23:42.102594+0000","last_became_peered":"2024-03-06T07:23:42.102594+0000","last_unstale":"2024-03-06T07:23:43.748071+0000","last_undegraded":"2024-03-06T07:23:43.748071+0000","last_fullsized":"2024-03-06T07:23:43.748071+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:57:46.850738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748569+0000","last_change":"2024-03-06T07:23:41.099078+0000","last_active":"2024-03-06T07:23:43.748569+0000","last_peered":"2024-03-06T07:23:43.748569+0000","last_clean":"2024-03-06T07:23:43.748569+0000","last_became_active":"2024-03-06T07:23:41.098812+0000","last_became_peered":"2024-03-06T07:23:41.098812+0000","last_unstale":"2024-03-06T07:23:43.748569+0000","last_undegraded":"2024-03-06T07:23:43.748569+0000","last_fullsized":"2024-03-06T07:23:43.748569+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:14:23.827424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.102416+0000","last_change":"2024-03-06T07:23:42.104651+0000","last_active":"2024-03-06T07:23:43.102416+0000","last_peered":"2024-03-06T07:23:43.102416+0000","last_clean":"2024-03-06T07:23:43.102416+0000","last_became_active":"2024-03-06T07:23:42.104347+0000","last_became_peered":"2024-03-06T07:23:42.104347+0000","last_unstale":"2024-03-06T07:23:43.102416+0000","last_undegraded":"2024-03-06T07:23:43.102416+0000","last_fullsized":"2024-03-06T07:23:43.102416+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:18.102969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748255+0000","last_change":"2024-03-06T07:23:41.098205+0000","last_active":"2024-03-06T07:23:43.748255+0000","last_peered":"2024-03-06T07:23:43.748255+0000","last_clean":"2024-03-06T07:23:43.748255+0000","last_became_active":"2024-03-06T07:23:41.097800+0000","last_became_peered":"2024-03-06T07:23:41.097800+0000","last_unstale":"2024-03-06T07:23:43.748255+0000","last_undegraded":"2024-03-06T07:23:43.748255+0000","last_fullsized":"2024-03-06T07:23:43.748255+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:27.990220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.749574+0000","last_change":"2024-03-06T07:23:42.108189+0000","last_active":"2024-03-06T07:23:43.749574+0000","last_peered":"2024-03-06T07:23:43.749574+0000","last_clean":"2024-03-06T07:23:43.749574+0000","last_became_active":"2024-03-06T07:23:42.107945+0000","last_became_peered":"2024-03-06T07:23:42.107945+0000","last_unstale":"2024-03-06T07:23:43.749574+0000","last_undegraded":"2024-03-06T07:23:43.749574+0000","last_fullsized":"2024-03-06T07:23:43.749574+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:29:24.116057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748396+0000","last_change":"2024-03-06T07:23:41.101372+0000","last_active":"2024-03-06T07:23:43.748396+0000","last_peered":"2024-03-06T07:23:43.748396+0000","last_clean":"2024-03-06T07:23:43.748396+0000","last_became_active":"2024-03-06T07:23:41.100976+0000","last_became_peered":"2024-03-06T07:23:41.100976+0000","last_unstale":"2024-03-06T07:23:43.748396+0000","last_undegraded":"2024-03-06T07:23:43.748396+0000","last_fullsized":"2024-03-06T07:23:43.748396+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:49:51.557613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":214,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748370+0000","last_change":"2024-03-06T07:21:47.553693+0000","last_active":"2024-03-06T07:23:43.748370+0000","last_peered":"2024-03-06T07:23:43.748370+0000","last_clean":"2024-03-06T07:23:43.748370+0000","last_became_active":"2024-03-06T07:21:47.552664+0000","last_became_peered":"2024-03-06T07:21:47.552664+0000","last_unstale":"2024-03-06T07:23:43.748370+0000","last_undegraded":"2024-03-06T07:23:43.748370+0000","last_fullsized":"2024-03-06T07:23:43.748370+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_clean_scrub_stamp":"2024-03-06T07:21:46.362028+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:02.916426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574877,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32288,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339552,"statfs":{"total":96636764160,"available":96603701248,"internally_reserved":0,"allocated":1126400,"data_stored":806075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574875,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31712,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340128,"statfs":{"total":96636764160,"available":96604291072,"internally_reserved":0,"allocated":536576,"data_stored":216289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574876,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31708,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340132,"statfs":{"total":96636764160,"available":96604295168,"internally_reserved":0,"allocated":532480,"data_stored":215687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574877,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31716,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340124,"statfs":{"total":96636764160,"available":96604286976,"internally_reserved":0,"allocated":540672,"data_stored":217229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574876,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31716,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340124,"statfs":{"total":96636764160,"available":96604286976,"internally_reserved":0,"allocated":540672,"data_stored":216255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574876,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31720,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340120,"statfs":{"total":96636764160,"available":96604282880,"internally_reserved":0,"allocated":544768,"data_stored":217251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574877,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31708,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340132,"statfs":{"total":96636764160,"available":96604295168,"internally_reserved":0,"allocated":532480,"data_stored":215687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574876,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32280,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339560,"statfs":{"total":96636764160,"available":96603709440,"internally_reserved":0,"allocated":1126400,"data_stored":806019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:23:47.352 INFO:tasks.ceph.mds.a.smithi083.stderr:2024-03-06T07:23:47.350+0000 7f60fa2fd640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:23:49.610 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:23:49.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.710+0000 7f335ffc9640 1 Processor -- start 2024-03-06T07:23:49.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.710+0000 7f335ffc9640 1 -- start start 2024-03-06T07:23:49.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.711+0000 7f335ffc9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f335815bdd0 0x7f33580c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.711+0000 7f335ffc9640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f33580d16e0 0x7f33580d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.711+0000 7f335ffc9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33580cebe0 con 0x7f335815fa10 2024-03-06T07:23:49.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.711+0000 7f335ffc9640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33580ce980 con 0x7f335815bdd0 2024-03-06T07:23:49.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.711+0000 7f335ffc9640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33580ceec0 con 0x7f33580d16e0 2024-03-06T07:23:49.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.711+0000 7f335dd65640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f335815bdd0 0x7f33580c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.711+0000 7f335dd65640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f335815bdd0 0x7f33580c5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:44660/0 (socket says 172.21.15.83:44660) 2024-03-06T07:23:49.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.711+0000 7f335dd65640 1 -- 172.21.15.83:0/4196078212 learned_addr learned my addr 172.21.15.83:0/4196078212 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:49.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.712+0000 7f335e566640 1 --2- 172.21.15.83:0/4196078212 >> v2:172.21.15.183:3300/0 conn(0x7f33580d16e0 0x7f33580d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.712+0000 7f335dd65640 1 -- 172.21.15.83:0/4196078212 >> v2:172.21.15.183:3300/0 conn(0x7f33580d16e0 msgr2=0x7f33580d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:49.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.712+0000 7f335dd65640 1 --2- 172.21.15.83:0/4196078212 >> v2:172.21.15.183:3300/0 conn(0x7f33580d16e0 0x7f33580d1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.712+0000 7f335dd65640 1 -- 172.21.15.83:0/4196078212 >> v1:172.21.15.183:6789/0 conn(0x7f335815fa10 legacy=0x7f33580ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:49.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.712+0000 7f335dd65640 1 -- 172.21.15.83:0/4196078212 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33580cf1a0 con 0x7f335815bdd0 2024-03-06T07:23:49.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.713+0000 7f335dd65640 1 --2- 172.21.15.83:0/4196078212 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f335815bdd0 0x7f33580c5610 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f33580c5520 tx=0x7f335402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11d8a70506182a67 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.713+0000 7f335cd63640 1 -- 172.21.15.83:0/4196078212 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3354035020 con 0x7f335815bdd0 2024-03-06T07:23:49.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.713+0000 7f335cd63640 1 -- 172.21.15.83:0/4196078212 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3354031870 con 0x7f335815bdd0 2024-03-06T07:23:49.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.713+0000 7f335cd63640 1 -- 172.21.15.83:0/4196078212 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3354031c40 con 0x7f335815bdd0 2024-03-06T07:23:49.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.713+0000 7f335ffc9640 1 -- 172.21.15.83:0/4196078212 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f335815bdd0 msgr2=0x7f33580c5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.713+0000 7f335ffc9640 1 --2- 172.21.15.83:0/4196078212 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f335815bdd0 0x7f33580c5610 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f33580c5520 tx=0x7f335402b910 comp rx=0 tx=0).stop 2024-03-06T07:23:49.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.713+0000 7f335ffc9640 1 -- 172.21.15.83:0/4196078212 shutdown_connections 2024-03-06T07:23:49.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.713+0000 7f335ffc9640 1 --2- 172.21.15.83:0/4196078212 >> v2:172.21.15.183:3300/0 conn(0x7f33580d16e0 0x7f33580d1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.713+0000 7f335ffc9640 1 --2- 172.21.15.83:0/4196078212 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f335815bdd0 0x7f33580c5610 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.713+0000 7f335ffc9640 1 -- 172.21.15.83:0/4196078212 >> 172.21.15.83:0/4196078212 conn(0x7f335809a9f0 msgr2=0x7f335809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:49.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.713+0000 7f335ffc9640 1 -- 172.21.15.83:0/4196078212 shutdown_connections 2024-03-06T07:23:49.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.713+0000 7f335ffc9640 1 -- 172.21.15.83:0/4196078212 wait complete. 2024-03-06T07:23:49.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.715+0000 7f335ffc9640 1 Processor -- start 2024-03-06T07:23:49.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.715+0000 7f335ffc9640 1 -- start start 2024-03-06T07:23:49.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.716+0000 7f335ffc9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33580d16e0 0x7f33580c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.716+0000 7f335ffc9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f335815bdd0 0x7f3358155320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.716+0000 7f335ffc9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f335815fa10 0x7f33580b6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.716+0000 7f335ffc9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33580cc5d0 con 0x7f33580d16e0 2024-03-06T07:23:49.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.716+0000 7f335ffc9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33580c5b50 con 0x7f335815fa10 2024-03-06T07:23:49.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.716+0000 7f335ffc9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f335815c1b0 con 0x7f335815bdd0 2024-03-06T07:23:49.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.716+0000 7f335dd65640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33580d16e0 0x7f33580c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.716+0000 7f335dd65640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33580d16e0 0x7f33580c8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:44676/0 (socket says 172.21.15.83:44676) 2024-03-06T07:23:49.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.716+0000 7f335dd65640 1 -- 172.21.15.83:0/2137643718 learned_addr learned my addr 172.21.15.83:0/2137643718 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:49.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.717+0000 7f335e566640 1 --2- 172.21.15.83:0/2137643718 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f335815fa10 0x7f33580b6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.717+0000 7f335d564640 1 --2- 172.21.15.83:0/2137643718 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f335815bdd0 0x7f3358155320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.717+0000 7f335dd65640 1 -- 172.21.15.83:0/2137643718 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f335815bdd0 msgr2=0x7f3358155320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.717+0000 7f335dd65640 1 --2- 172.21.15.83:0/2137643718 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f335815bdd0 0x7f3358155320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.717+0000 7f335dd65640 1 -- 172.21.15.83:0/2137643718 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f335815fa10 msgr2=0x7f33580b6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.717+0000 7f335dd65640 1 --2- 172.21.15.83:0/2137643718 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f335815fa10 0x7f33580b6a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.717+0000 7f335dd65640 1 -- 172.21.15.83:0/2137643718 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33580b5680 con 0x7f33580d16e0 2024-03-06T07:23:49.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.717+0000 7f335dd65640 1 --2- 172.21.15.83:0/2137643718 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33580d16e0 0x7f33580c8b90 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f3354000c00 tx=0x7f33540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.717+0000 7f334e7fc640 1 -- 172.21.15.83:0/2137643718 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3354035040 con 0x7f33580d16e0 2024-03-06T07:23:49.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.717+0000 7f334e7fc640 1 -- 172.21.15.83:0/2137643718 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3354030d00 con 0x7f33580d16e0 2024-03-06T07:23:49.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.717+0000 7f335ffc9640 1 -- 172.21.15.83:0/2137643718 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33580b5960 con 0x7f33580d16e0 2024-03-06T07:23:49.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.717+0000 7f334e7fc640 1 -- 172.21.15.83:0/2137643718 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33540456a0 con 0x7f33580d16e0 2024-03-06T07:23:49.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.717+0000 7f335ffc9640 1 -- 172.21.15.83:0/2137643718 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33580b5f20 con 0x7f33580d16e0 2024-03-06T07:23:49.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.718+0000 7f335ffc9640 1 -- 172.21.15.83:0/2137643718 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3358160760 con 0x7f33580d16e0 2024-03-06T07:23:49.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.719+0000 7f334e7fc640 1 -- 172.21.15.83:0/2137643718 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3354038020 con 0x7f33580d16e0 2024-03-06T07:23:49.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.719+0000 7f334e7fc640 1 --2- 172.21.15.83:0/2137643718 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f333c06b4d0 0x7f333c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.719+0000 7f335d564640 1 --2- 172.21.15.83:0/2137643718 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f333c06b4d0 0x7f333c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.719+0000 7f334e7fc640 1 -- 172.21.15.83:0/2137643718 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f33540ac490 con 0x7f33580d16e0 2024-03-06T07:23:49.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.720+0000 7f335d564640 1 --2- 172.21.15.83:0/2137643718 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f333c06b4d0 0x7f333c06d990 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f33580cb9d0 tx=0x7f3348009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.724+0000 7f334e7fc640 1 -- 172.21.15.83:0/2137643718 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3358160760 con 0x7f33580d16e0 2024-03-06T07:23:49.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.910+0000 7f335ffc9640 1 -- 172.21.15.83:0/2137643718 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f335804f330 con 0x7f333c06b4d0 2024-03-06T07:23:49.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.916+0000 7f334e7fc640 1 -- 172.21.15.83:0/2137643718 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251959 (secure 0 0 0) 0x7f335804f330 con 0x7f333c06b4d0 2024-03-06T07:23:49.921 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:23:49.923 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-06T07:23:49.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.919+0000 7f335ffc9640 1 -- 172.21.15.83:0/2137643718 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f333c06b4d0 msgr2=0x7f333c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.919+0000 7f335ffc9640 1 --2- 172.21.15.83:0/2137643718 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f333c06b4d0 0x7f333c06d990 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f33580cb9d0 tx=0x7f3348009040 comp rx=0 tx=0).stop 2024-03-06T07:23:49.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.919+0000 7f335ffc9640 1 -- 172.21.15.83:0/2137643718 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33580d16e0 msgr2=0x7f33580c8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.919+0000 7f335ffc9640 1 --2- 172.21.15.83:0/2137643718 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33580d16e0 0x7f33580c8b90 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f3354000c00 tx=0x7f33540046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.919+0000 7f335ffc9640 1 -- 172.21.15.83:0/2137643718 shutdown_connections 2024-03-06T07:23:49.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.919+0000 7f335ffc9640 1 --2- 172.21.15.83:0/2137643718 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f333c06b4d0 0x7f333c06d990 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.919+0000 7f335ffc9640 1 --2- 172.21.15.83:0/2137643718 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f335815fa10 0x7f33580b6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.919+0000 7f335ffc9640 1 --2- 172.21.15.83:0/2137643718 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f335815bdd0 0x7f3358155320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.919+0000 7f335ffc9640 1 --2- 172.21.15.83:0/2137643718 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33580d16e0 0x7f33580c8b90 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.919+0000 7f335ffc9640 1 -- 172.21.15.83:0/2137643718 >> 172.21.15.83:0/2137643718 conn(0x7f335809a9f0 msgr2=0x7f33580ccd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:49.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.919+0000 7f335ffc9640 1 -- 172.21.15.83:0/2137643718 shutdown_connections 2024-03-06T07:23:49.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:49.919+0000 7f335ffc9640 1 -- 172.21.15.83:0/2137643718 wait complete. 2024-03-06T07:23:49.932 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":100,"stamp":"2024-03-06T07:23:48.340907+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":156,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254848,"kb_used_data":5352,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754719872,"statfs":{"total":773094113280,"available":772833148928,"internally_reserved":0,"allocated":5480448,"data_stored":2910492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12757,"internal_metadata":255315499},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.259486"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.749408+0000","last_change":"2024-03-06T07:23:42.101550+0000","last_active":"2024-03-06T07:23:43.749408+0000","last_peered":"2024-03-06T07:23:43.749408+0000","last_clean":"2024-03-06T07:23:43.749408+0000","last_became_active":"2024-03-06T07:23:42.101282+0000","last_became_peered":"2024-03-06T07:23:42.101282+0000","last_unstale":"2024-03-06T07:23:43.749408+0000","last_undegraded":"2024-03-06T07:23:43.749408+0000","last_fullsized":"2024-03-06T07:23:43.749408+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:44.930081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748570+0000","last_change":"2024-03-06T07:23:42.115833+0000","last_active":"2024-03-06T07:23:43.748570+0000","last_peered":"2024-03-06T07:23:43.748570+0000","last_clean":"2024-03-06T07:23:43.748570+0000","last_became_active":"2024-03-06T07:23:42.115559+0000","last_became_peered":"2024-03-06T07:23:42.115559+0000","last_unstale":"2024-03-06T07:23:43.748570+0000","last_undegraded":"2024-03-06T07:23:43.748570+0000","last_fullsized":"2024-03-06T07:23:43.748570+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:39:58.306094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.102686+0000","last_change":"2024-03-06T07:23:42.094590+0000","last_active":"2024-03-06T07:23:43.102686+0000","last_peered":"2024-03-06T07:23:43.102686+0000","last_clean":"2024-03-06T07:23:43.102686+0000","last_became_active":"2024-03-06T07:23:42.094264+0000","last_became_peered":"2024-03-06T07:23:42.094264+0000","last_unstale":"2024-03-06T07:23:43.102686+0000","last_undegraded":"2024-03-06T07:23:43.102686+0000","last_fullsized":"2024-03-06T07:23:43.102686+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:41.381834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748540+0000","last_change":"2024-03-06T07:23:42.108875+0000","last_active":"2024-03-06T07:23:43.748540+0000","last_peered":"2024-03-06T07:23:43.748540+0000","last_clean":"2024-03-06T07:23:43.748540+0000","last_became_active":"2024-03-06T07:23:42.108613+0000","last_became_peered":"2024-03-06T07:23:42.108613+0000","last_unstale":"2024-03-06T07:23:43.748540+0000","last_undegraded":"2024-03-06T07:23:43.748540+0000","last_fullsized":"2024-03-06T07:23:43.748540+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:45:58.613060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748294+0000","last_change":"2024-03-06T07:23:42.116009+0000","last_active":"2024-03-06T07:23:43.748294+0000","last_peered":"2024-03-06T07:23:43.748294+0000","last_clean":"2024-03-06T07:23:43.748294+0000","last_became_active":"2024-03-06T07:23:42.115609+0000","last_became_peered":"2024-03-06T07:23:42.115609+0000","last_unstale":"2024-03-06T07:23:43.748294+0000","last_undegraded":"2024-03-06T07:23:43.748294+0000","last_fullsized":"2024-03-06T07:23:43.748294+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:12.037110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748771+0000","last_change":"2024-03-06T07:23:42.104726+0000","last_active":"2024-03-06T07:23:43.748771+0000","last_peered":"2024-03-06T07:23:43.748771+0000","last_clean":"2024-03-06T07:23:43.748771+0000","last_became_active":"2024-03-06T07:23:42.104233+0000","last_became_peered":"2024-03-06T07:23:42.104233+0000","last_unstale":"2024-03-06T07:23:43.748771+0000","last_undegraded":"2024-03-06T07:23:43.748771+0000","last_fullsized":"2024-03-06T07:23:43.748771+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:54:40.118136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.749066+0000","last_change":"2024-03-06T07:23:42.097305+0000","last_active":"2024-03-06T07:23:43.749066+0000","last_peered":"2024-03-06T07:23:43.749066+0000","last_clean":"2024-03-06T07:23:43.749066+0000","last_became_active":"2024-03-06T07:23:42.097033+0000","last_became_peered":"2024-03-06T07:23:42.097033+0000","last_unstale":"2024-03-06T07:23:43.749066+0000","last_undegraded":"2024-03-06T07:23:43.749066+0000","last_fullsized":"2024-03-06T07:23:43.749066+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:47:34.961757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.747864+0000","last_change":"2024-03-06T07:23:42.113045+0000","last_active":"2024-03-06T07:23:43.747864+0000","last_peered":"2024-03-06T07:23:43.747864+0000","last_clean":"2024-03-06T07:23:43.747864+0000","last_became_active":"2024-03-06T07:23:42.112672+0000","last_became_peered":"2024-03-06T07:23:42.112672+0000","last_unstale":"2024-03-06T07:23:43.747864+0000","last_undegraded":"2024-03-06T07:23:43.747864+0000","last_fullsized":"2024-03-06T07:23:43.747864+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:18.102969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748981+0000","last_change":"2024-03-06T07:23:42.113581+0000","last_active":"2024-03-06T07:23:43.748981+0000","last_peered":"2024-03-06T07:23:43.748981+0000","last_clean":"2024-03-06T07:23:43.748981+0000","last_became_active":"2024-03-06T07:23:42.113268+0000","last_became_peered":"2024-03-06T07:23:42.113268+0000","last_unstale":"2024-03-06T07:23:43.748981+0000","last_undegraded":"2024-03-06T07:23:43.748981+0000","last_fullsized":"2024-03-06T07:23:43.748981+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:04:27.854692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.749447+0000","last_change":"2024-03-06T07:23:42.113788+0000","last_active":"2024-03-06T07:23:43.749447+0000","last_peered":"2024-03-06T07:23:43.749447+0000","last_clean":"2024-03-06T07:23:43.749447+0000","last_became_active":"2024-03-06T07:23:42.113259+0000","last_became_peered":"2024-03-06T07:23:42.113259+0000","last_unstale":"2024-03-06T07:23:43.749447+0000","last_undegraded":"2024-03-06T07:23:43.749447+0000","last_fullsized":"2024-03-06T07:23:43.749447+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:25.231002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748400+0000","last_change":"2024-03-06T07:23:42.113821+0000","last_active":"2024-03-06T07:23:43.748400+0000","last_peered":"2024-03-06T07:23:43.748400+0000","last_clean":"2024-03-06T07:23:43.748400+0000","last_became_active":"2024-03-06T07:23:42.113442+0000","last_became_peered":"2024-03-06T07:23:42.113442+0000","last_unstale":"2024-03-06T07:23:43.748400+0000","last_undegraded":"2024-03-06T07:23:43.748400+0000","last_fullsized":"2024-03-06T07:23:43.748400+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:34:33.726443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748252+0000","last_change":"2024-03-06T07:23:42.102990+0000","last_active":"2024-03-06T07:23:43.748252+0000","last_peered":"2024-03-06T07:23:43.748252+0000","last_clean":"2024-03-06T07:23:43.748252+0000","last_became_active":"2024-03-06T07:23:42.101295+0000","last_became_peered":"2024-03-06T07:23:42.101295+0000","last_unstale":"2024-03-06T07:23:43.748252+0000","last_undegraded":"2024-03-06T07:23:43.748252+0000","last_fullsized":"2024-03-06T07:23:43.748252+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:54.936995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.101971+0000","last_change":"2024-03-06T07:23:42.108304+0000","last_active":"2024-03-06T07:23:43.101971+0000","last_peered":"2024-03-06T07:23:43.101971+0000","last_clean":"2024-03-06T07:23:43.101971+0000","last_became_active":"2024-03-06T07:23:42.107999+0000","last_became_peered":"2024-03-06T07:23:42.107999+0000","last_unstale":"2024-03-06T07:23:43.101971+0000","last_undegraded":"2024-03-06T07:23:43.101971+0000","last_fullsized":"2024-03-06T07:23:43.101971+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:44:26.683785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748044+0000","last_change":"2024-03-06T07:23:42.103822+0000","last_active":"2024-03-06T07:23:43.748044+0000","last_peered":"2024-03-06T07:23:43.748044+0000","last_clean":"2024-03-06T07:23:43.748044+0000","last_became_active":"2024-03-06T07:23:42.103493+0000","last_became_peered":"2024-03-06T07:23:42.103493+0000","last_unstale":"2024-03-06T07:23:43.748044+0000","last_undegraded":"2024-03-06T07:23:43.748044+0000","last_fullsized":"2024-03-06T07:23:43.748044+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:41.381834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748718+0000","last_change":"2024-03-06T07:23:42.106167+0000","last_active":"2024-03-06T07:23:43.748718+0000","last_peered":"2024-03-06T07:23:43.748718+0000","last_clean":"2024-03-06T07:23:43.748718+0000","last_became_active":"2024-03-06T07:23:42.105894+0000","last_became_peered":"2024-03-06T07:23:42.105894+0000","last_unstale":"2024-03-06T07:23:43.748718+0000","last_undegraded":"2024-03-06T07:23:43.748718+0000","last_fullsized":"2024-03-06T07:23:43.748718+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:17:46.084647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.102843+0000","last_change":"2024-03-06T07:23:42.106702+0000","last_active":"2024-03-06T07:23:43.102843+0000","last_peered":"2024-03-06T07:23:43.102843+0000","last_clean":"2024-03-06T07:23:43.102843+0000","last_became_active":"2024-03-06T07:23:42.106435+0000","last_became_peered":"2024-03-06T07:23:42.106435+0000","last_unstale":"2024-03-06T07:23:43.102843+0000","last_undegraded":"2024-03-06T07:23:43.102843+0000","last_fullsized":"2024-03-06T07:23:43.102843+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:14.675455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.103160+0000","last_change":"2024-03-06T07:23:42.099748+0000","last_active":"2024-03-06T07:23:43.103160+0000","last_peered":"2024-03-06T07:23:43.103160+0000","last_clean":"2024-03-06T07:23:43.103160+0000","last_became_active":"2024-03-06T07:23:42.099485+0000","last_became_peered":"2024-03-06T07:23:42.099485+0000","last_unstale":"2024-03-06T07:23:43.103160+0000","last_undegraded":"2024-03-06T07:23:43.103160+0000","last_fullsized":"2024-03-06T07:23:43.103160+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:59:10.744765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.103107+0000","last_change":"2024-03-06T07:23:42.114674+0000","last_active":"2024-03-06T07:23:43.103107+0000","last_peered":"2024-03-06T07:23:43.103107+0000","last_clean":"2024-03-06T07:23:43.103107+0000","last_became_active":"2024-03-06T07:23:42.114375+0000","last_became_peered":"2024-03-06T07:23:42.114375+0000","last_unstale":"2024-03-06T07:23:43.103107+0000","last_undegraded":"2024-03-06T07:23:43.103107+0000","last_fullsized":"2024-03-06T07:23:43.103107+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:49:52.564351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.103332+0000","last_change":"2024-03-06T07:23:42.105556+0000","last_active":"2024-03-06T07:23:43.103332+0000","last_peered":"2024-03-06T07:23:43.103332+0000","last_clean":"2024-03-06T07:23:43.103332+0000","last_became_active":"2024-03-06T07:23:42.105235+0000","last_became_peered":"2024-03-06T07:23:42.105235+0000","last_unstale":"2024-03-06T07:23:43.103332+0000","last_undegraded":"2024-03-06T07:23:43.103332+0000","last_fullsized":"2024-03-06T07:23:43.103332+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:26.696304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.102242+0000","last_change":"2024-03-06T07:23:42.107082+0000","last_active":"2024-03-06T07:23:43.102242+0000","last_peered":"2024-03-06T07:23:43.102242+0000","last_clean":"2024-03-06T07:23:43.102242+0000","last_became_active":"2024-03-06T07:23:42.106795+0000","last_became_peered":"2024-03-06T07:23:42.106795+0000","last_unstale":"2024-03-06T07:23:43.102242+0000","last_undegraded":"2024-03-06T07:23:43.102242+0000","last_fullsized":"2024-03-06T07:23:43.102242+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:28.996958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748539+0000","last_change":"2024-03-06T07:23:42.105880+0000","last_active":"2024-03-06T07:23:43.748539+0000","last_peered":"2024-03-06T07:23:43.748539+0000","last_clean":"2024-03-06T07:23:43.748539+0000","last_became_active":"2024-03-06T07:23:42.105503+0000","last_became_peered":"2024-03-06T07:23:42.105503+0000","last_unstale":"2024-03-06T07:23:43.748539+0000","last_undegraded":"2024-03-06T07:23:43.748539+0000","last_fullsized":"2024-03-06T07:23:43.748539+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:07.956361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748182+0000","last_change":"2024-03-06T07:23:42.110836+0000","last_active":"2024-03-06T07:23:43.748182+0000","last_peered":"2024-03-06T07:23:43.748182+0000","last_clean":"2024-03-06T07:23:43.748182+0000","last_became_active":"2024-03-06T07:23:42.110557+0000","last_became_peered":"2024-03-06T07:23:42.110557+0000","last_unstale":"2024-03-06T07:23:43.748182+0000","last_undegraded":"2024-03-06T07:23:43.748182+0000","last_fullsized":"2024-03-06T07:23:43.748182+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:58.160523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.103735+0000","last_change":"2024-03-06T07:23:42.098866+0000","last_active":"2024-03-06T07:23:43.103735+0000","last_peered":"2024-03-06T07:23:43.103735+0000","last_clean":"2024-03-06T07:23:43.103735+0000","last_became_active":"2024-03-06T07:23:42.097444+0000","last_became_peered":"2024-03-06T07:23:42.097444+0000","last_unstale":"2024-03-06T07:23:43.103735+0000","last_undegraded":"2024-03-06T07:23:43.103735+0000","last_fullsized":"2024-03-06T07:23:43.103735+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:04:47.346959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.102977+0000","last_change":"2024-03-06T07:23:42.095878+0000","last_active":"2024-03-06T07:23:43.102977+0000","last_peered":"2024-03-06T07:23:43.102977+0000","last_clean":"2024-03-06T07:23:43.102977+0000","last_became_active":"2024-03-06T07:23:42.095597+0000","last_became_peered":"2024-03-06T07:23:42.095597+0000","last_unstale":"2024-03-06T07:23:43.102977+0000","last_undegraded":"2024-03-06T07:23:43.102977+0000","last_fullsized":"2024-03-06T07:23:43.102977+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:35.837317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.103480+0000","last_change":"2024-03-06T07:23:42.102398+0000","last_active":"2024-03-06T07:23:43.103480+0000","last_peered":"2024-03-06T07:23:43.103480+0000","last_clean":"2024-03-06T07:23:43.103480+0000","last_became_active":"2024-03-06T07:23:42.099793+0000","last_became_peered":"2024-03-06T07:23:42.099793+0000","last_unstale":"2024-03-06T07:23:43.103480+0000","last_undegraded":"2024-03-06T07:23:43.103480+0000","last_fullsized":"2024-03-06T07:23:43.103480+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:01:39.323391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.101896+0000","last_change":"2024-03-06T07:23:42.109996+0000","last_active":"2024-03-06T07:23:43.101896+0000","last_peered":"2024-03-06T07:23:43.101896+0000","last_clean":"2024-03-06T07:23:43.101896+0000","last_became_active":"2024-03-06T07:23:42.109714+0000","last_became_peered":"2024-03-06T07:23:42.109714+0000","last_unstale":"2024-03-06T07:23:43.101896+0000","last_undegraded":"2024-03-06T07:23:43.101896+0000","last_fullsized":"2024-03-06T07:23:43.101896+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:38.452235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.102840+0000","last_change":"2024-03-06T07:23:42.099076+0000","last_active":"2024-03-06T07:23:43.102840+0000","last_peered":"2024-03-06T07:23:43.102840+0000","last_clean":"2024-03-06T07:23:43.102840+0000","last_became_active":"2024-03-06T07:23:42.098775+0000","last_became_peered":"2024-03-06T07:23:42.098775+0000","last_unstale":"2024-03-06T07:23:43.102840+0000","last_undegraded":"2024-03-06T07:23:43.102840+0000","last_fullsized":"2024-03-06T07:23:43.102840+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:58.160523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748495+0000","last_change":"2024-03-06T07:23:42.103218+0000","last_active":"2024-03-06T07:23:43.748495+0000","last_peered":"2024-03-06T07:23:43.748495+0000","last_clean":"2024-03-06T07:23:43.748495+0000","last_became_active":"2024-03-06T07:23:42.102222+0000","last_became_peered":"2024-03-06T07:23:42.102222+0000","last_unstale":"2024-03-06T07:23:43.748495+0000","last_undegraded":"2024-03-06T07:23:43.748495+0000","last_fullsized":"2024-03-06T07:23:43.748495+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:51.678792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.102736+0000","last_change":"2024-03-06T07:23:42.100064+0000","last_active":"2024-03-06T07:23:43.102736+0000","last_peered":"2024-03-06T07:23:43.102736+0000","last_clean":"2024-03-06T07:23:43.102736+0000","last_became_active":"2024-03-06T07:23:42.099719+0000","last_became_peered":"2024-03-06T07:23:42.099719+0000","last_unstale":"2024-03-06T07:23:43.102736+0000","last_undegraded":"2024-03-06T07:23:43.102736+0000","last_fullsized":"2024-03-06T07:23:43.102736+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:04.000572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.103024+0000","last_change":"2024-03-06T07:23:42.100949+0000","last_active":"2024-03-06T07:23:43.103024+0000","last_peered":"2024-03-06T07:23:43.103024+0000","last_clean":"2024-03-06T07:23:43.103024+0000","last_became_active":"2024-03-06T07:23:42.100593+0000","last_became_peered":"2024-03-06T07:23:42.100593+0000","last_unstale":"2024-03-06T07:23:43.103024+0000","last_undegraded":"2024-03-06T07:23:43.103024+0000","last_fullsized":"2024-03-06T07:23:43.103024+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:34:33.726443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748842+0000","last_change":"2024-03-06T07:23:42.104004+0000","last_active":"2024-03-06T07:23:43.748842+0000","last_peered":"2024-03-06T07:23:43.748842+0000","last_clean":"2024-03-06T07:23:43.748842+0000","last_became_active":"2024-03-06T07:23:42.103629+0000","last_became_peered":"2024-03-06T07:23:42.103629+0000","last_unstale":"2024-03-06T07:23:43.748842+0000","last_undegraded":"2024-03-06T07:23:43.748842+0000","last_fullsized":"2024-03-06T07:23:43.748842+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:42:14.658158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.101940+0000","last_change":"2024-03-06T07:23:42.099172+0000","last_active":"2024-03-06T07:23:43.101940+0000","last_peered":"2024-03-06T07:23:43.101940+0000","last_clean":"2024-03-06T07:23:43.101940+0000","last_became_active":"2024-03-06T07:23:42.098888+0000","last_became_peered":"2024-03-06T07:23:42.098888+0000","last_unstale":"2024-03-06T07:23:43.101940+0000","last_undegraded":"2024-03-06T07:23:43.101940+0000","last_fullsized":"2024-03-06T07:23:43.101940+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:08.444552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748780+0000","last_change":"2024-03-06T07:23:42.111685+0000","last_active":"2024-03-06T07:23:43.748780+0000","last_peered":"2024-03-06T07:23:43.748780+0000","last_clean":"2024-03-06T07:23:43.748780+0000","last_became_active":"2024-03-06T07:23:42.110757+0000","last_became_peered":"2024-03-06T07:23:42.110757+0000","last_unstale":"2024-03-06T07:23:43.748780+0000","last_undegraded":"2024-03-06T07:23:43.748780+0000","last_fullsized":"2024-03-06T07:23:43.748780+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:01:39.323391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"44'4","reported_seq":18,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748124+0000","last_change":"2024-03-06T07:23:41.102362+0000","last_active":"2024-03-06T07:23:43.748124+0000","last_peered":"2024-03-06T07:23:43.748124+0000","last_clean":"2024-03-06T07:23:43.748124+0000","last_became_active":"2024-03-06T07:23:41.102058+0000","last_became_peered":"2024-03-06T07:23:41.102058+0000","last_unstale":"2024-03-06T07:23:43.748124+0000","last_undegraded":"2024-03-06T07:23:43.748124+0000","last_fullsized":"2024-03-06T07:23:43.748124+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:24.224264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.103214+0000","last_change":"2024-03-06T07:23:42.090875+0000","last_active":"2024-03-06T07:23:43.103214+0000","last_peered":"2024-03-06T07:23:43.103214+0000","last_clean":"2024-03-06T07:23:43.103214+0000","last_became_active":"2024-03-06T07:23:42.090581+0000","last_became_peered":"2024-03-06T07:23:42.090581+0000","last_unstale":"2024-03-06T07:23:43.103214+0000","last_undegraded":"2024-03-06T07:23:43.103214+0000","last_fullsized":"2024-03-06T07:23:43.103214+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:32:21.061612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.103170+0000","last_change":"2024-03-06T07:23:41.102284+0000","last_active":"2024-03-06T07:23:43.103170+0000","last_peered":"2024-03-06T07:23:43.103170+0000","last_clean":"2024-03-06T07:23:43.103170+0000","last_became_active":"2024-03-06T07:23:41.102011+0000","last_became_peered":"2024-03-06T07:23:41.102011+0000","last_unstale":"2024-03-06T07:23:43.103170+0000","last_undegraded":"2024-03-06T07:23:43.103170+0000","last_fullsized":"2024-03-06T07:23:43.103170+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:02.993834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748584+0000","last_change":"2024-03-06T07:23:42.107954+0000","last_active":"2024-03-06T07:23:43.748584+0000","last_peered":"2024-03-06T07:23:43.748584+0000","last_clean":"2024-03-06T07:23:43.748584+0000","last_became_active":"2024-03-06T07:23:42.107689+0000","last_became_peered":"2024-03-06T07:23:42.107689+0000","last_unstale":"2024-03-06T07:23:43.748584+0000","last_undegraded":"2024-03-06T07:23:43.748584+0000","last_fullsized":"2024-03-06T07:23:43.748584+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:47:34.961757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748453+0000","last_change":"2024-03-06T07:23:41.101648+0000","last_active":"2024-03-06T07:23:43.748453+0000","last_peered":"2024-03-06T07:23:43.748453+0000","last_clean":"2024-03-06T07:23:43.748453+0000","last_became_active":"2024-03-06T07:23:41.101345+0000","last_became_peered":"2024-03-06T07:23:41.101345+0000","last_unstale":"2024-03-06T07:23:43.748453+0000","last_undegraded":"2024-03-06T07:23:43.748453+0000","last_fullsized":"2024-03-06T07:23:43.748453+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:34.830579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748403+0000","last_change":"2024-03-06T07:23:42.100513+0000","last_active":"2024-03-06T07:23:43.748403+0000","last_peered":"2024-03-06T07:23:43.748403+0000","last_clean":"2024-03-06T07:23:43.748403+0000","last_became_active":"2024-03-06T07:23:42.100233+0000","last_became_peered":"2024-03-06T07:23:42.100233+0000","last_unstale":"2024-03-06T07:23:43.748403+0000","last_undegraded":"2024-03-06T07:23:43.748403+0000","last_fullsized":"2024-03-06T07:23:43.748403+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:58.192125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.749120+0000","last_change":"2024-03-06T07:23:41.100199+0000","last_active":"2024-03-06T07:23:43.749120+0000","last_peered":"2024-03-06T07:23:43.749120+0000","last_clean":"2024-03-06T07:23:43.749120+0000","last_became_active":"2024-03-06T07:23:41.099943+0000","last_became_peered":"2024-03-06T07:23:41.099943+0000","last_unstale":"2024-03-06T07:23:43.749120+0000","last_undegraded":"2024-03-06T07:23:43.749120+0000","last_fullsized":"2024-03-06T07:23:43.749120+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:49:51.557613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.102646+0000","last_change":"2024-03-06T07:23:42.090685+0000","last_active":"2024-03-06T07:23:43.102646+0000","last_peered":"2024-03-06T07:23:43.102646+0000","last_clean":"2024-03-06T07:23:43.102646+0000","last_became_active":"2024-03-06T07:23:42.090184+0000","last_became_peered":"2024-03-06T07:23:42.090184+0000","last_unstale":"2024-03-06T07:23:43.102646+0000","last_undegraded":"2024-03-06T07:23:43.102646+0000","last_fullsized":"2024-03-06T07:23:43.102646+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:31:33.587498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748595+0000","last_change":"2024-03-06T07:23:41.096923+0000","last_active":"2024-03-06T07:23:43.748595+0000","last_peered":"2024-03-06T07:23:43.748595+0000","last_clean":"2024-03-06T07:23:43.748595+0000","last_became_active":"2024-03-06T07:23:41.096352+0000","last_became_peered":"2024-03-06T07:23:41.096352+0000","last_unstale":"2024-03-06T07:23:43.748595+0000","last_undegraded":"2024-03-06T07:23:43.748595+0000","last_fullsized":"2024-03-06T07:23:43.748595+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:57:45.844000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748855+0000","last_change":"2024-03-06T07:23:42.115530+0000","last_active":"2024-03-06T07:23:43.748855+0000","last_peered":"2024-03-06T07:23:43.748855+0000","last_clean":"2024-03-06T07:23:43.748855+0000","last_became_active":"2024-03-06T07:23:42.115252+0000","last_became_peered":"2024-03-06T07:23:42.115252+0000","last_unstale":"2024-03-06T07:23:43.748855+0000","last_undegraded":"2024-03-06T07:23:43.748855+0000","last_fullsized":"2024-03-06T07:23:43.748855+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:59:10.744765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.102718+0000","last_change":"2024-03-06T07:23:41.106716+0000","last_active":"2024-03-06T07:23:43.102718+0000","last_peered":"2024-03-06T07:23:43.102718+0000","last_clean":"2024-03-06T07:23:43.102718+0000","last_became_active":"2024-03-06T07:23:41.106375+0000","last_became_peered":"2024-03-06T07:23:41.106375+0000","last_unstale":"2024-03-06T07:23:43.102718+0000","last_undegraded":"2024-03-06T07:23:43.102718+0000","last_fullsized":"2024-03-06T07:23:43.102718+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:34.830579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748015+0000","last_change":"2024-03-06T07:23:42.112489+0000","last_active":"2024-03-06T07:23:43.748015+0000","last_peered":"2024-03-06T07:23:43.748015+0000","last_clean":"2024-03-06T07:23:43.748015+0000","last_became_active":"2024-03-06T07:23:42.111597+0000","last_became_peered":"2024-03-06T07:23:42.111597+0000","last_unstale":"2024-03-06T07:23:43.748015+0000","last_undegraded":"2024-03-06T07:23:43.748015+0000","last_fullsized":"2024-03-06T07:23:43.748015+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:31:33.587498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748017+0000","last_change":"2024-03-06T07:23:41.099133+0000","last_active":"2024-03-06T07:23:43.748017+0000","last_peered":"2024-03-06T07:23:43.748017+0000","last_clean":"2024-03-06T07:23:43.748017+0000","last_became_active":"2024-03-06T07:23:41.098838+0000","last_became_peered":"2024-03-06T07:23:41.098838+0000","last_unstale":"2024-03-06T07:23:43.748017+0000","last_undegraded":"2024-03-06T07:23:43.748017+0000","last_fullsized":"2024-03-06T07:23:43.748017+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:57.153785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.103521+0000","last_change":"2024-03-06T07:23:42.099119+0000","last_active":"2024-03-06T07:23:43.103521+0000","last_peered":"2024-03-06T07:23:43.103521+0000","last_clean":"2024-03-06T07:23:43.103521+0000","last_became_active":"2024-03-06T07:23:42.097984+0000","last_became_peered":"2024-03-06T07:23:42.097984+0000","last_unstale":"2024-03-06T07:23:43.103521+0000","last_undegraded":"2024-03-06T07:23:43.103521+0000","last_fullsized":"2024-03-06T07:23:43.103521+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:33:48.156540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748064+0000","last_change":"2024-03-06T07:23:41.095707+0000","last_active":"2024-03-06T07:23:43.748064+0000","last_peered":"2024-03-06T07:23:43.748064+0000","last_clean":"2024-03-06T07:23:43.748064+0000","last_became_active":"2024-03-06T07:23:41.095430+0000","last_became_peered":"2024-03-06T07:23:41.095430+0000","last_unstale":"2024-03-06T07:23:43.748064+0000","last_undegraded":"2024-03-06T07:23:43.748064+0000","last_fullsized":"2024-03-06T07:23:43.748064+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:44:25.677046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.103089+0000","last_change":"2024-03-06T07:23:42.114516+0000","last_active":"2024-03-06T07:23:43.103089+0000","last_peered":"2024-03-06T07:23:43.103089+0000","last_clean":"2024-03-06T07:23:43.103089+0000","last_became_active":"2024-03-06T07:23:42.114282+0000","last_became_peered":"2024-03-06T07:23:42.114282+0000","last_unstale":"2024-03-06T07:23:43.103089+0000","last_undegraded":"2024-03-06T07:23:43.103089+0000","last_fullsized":"2024-03-06T07:23:43.103089+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:45:58.613060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.103083+0000","last_change":"2024-03-06T07:23:41.100731+0000","last_active":"2024-03-06T07:23:43.103083+0000","last_peered":"2024-03-06T07:23:43.103083+0000","last_clean":"2024-03-06T07:23:43.103083+0000","last_became_active":"2024-03-06T07:23:41.100431+0000","last_became_peered":"2024-03-06T07:23:41.100431+0000","last_unstale":"2024-03-06T07:23:43.103083+0000","last_undegraded":"2024-03-06T07:23:43.103083+0000","last_fullsized":"2024-03-06T07:23:43.103083+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:49.562199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.749423+0000","last_change":"2024-03-06T07:23:42.099151+0000","last_active":"2024-03-06T07:23:43.749423+0000","last_peered":"2024-03-06T07:23:43.749423+0000","last_clean":"2024-03-06T07:23:43.749423+0000","last_became_active":"2024-03-06T07:23:42.098887+0000","last_became_peered":"2024-03-06T07:23:42.098887+0000","last_unstale":"2024-03-06T07:23:43.749423+0000","last_undegraded":"2024-03-06T07:23:43.749423+0000","last_fullsized":"2024-03-06T07:23:43.749423+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:22.697354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.749619+0000","last_change":"2024-03-06T07:23:41.098118+0000","last_active":"2024-03-06T07:23:43.749619+0000","last_peered":"2024-03-06T07:23:43.749619+0000","last_clean":"2024-03-06T07:23:43.749619+0000","last_became_active":"2024-03-06T07:23:41.097846+0000","last_became_peered":"2024-03-06T07:23:41.097846+0000","last_unstale":"2024-03-06T07:23:43.749619+0000","last_undegraded":"2024-03-06T07:23:43.749619+0000","last_fullsized":"2024-03-06T07:23:43.749619+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:29:44.897241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748322+0000","last_change":"2024-03-06T07:23:42.106067+0000","last_active":"2024-03-06T07:23:43.748322+0000","last_peered":"2024-03-06T07:23:43.748322+0000","last_clean":"2024-03-06T07:23:43.748322+0000","last_became_active":"2024-03-06T07:23:42.105700+0000","last_became_peered":"2024-03-06T07:23:42.105700+0000","last_unstale":"2024-03-06T07:23:43.748322+0000","last_undegraded":"2024-03-06T07:23:43.748322+0000","last_fullsized":"2024-03-06T07:23:43.748322+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:36:39.813573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"44'3","reported_seq":17,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.749370+0000","last_change":"2024-03-06T07:23:41.102169+0000","last_active":"2024-03-06T07:23:43.749370+0000","last_peered":"2024-03-06T07:23:43.749370+0000","last_clean":"2024-03-06T07:23:43.749370+0000","last_became_active":"2024-03-06T07:23:41.101887+0000","last_became_peered":"2024-03-06T07:23:41.101887+0000","last_unstale":"2024-03-06T07:23:43.749370+0000","last_undegraded":"2024-03-06T07:23:43.749370+0000","last_fullsized":"2024-03-06T07:23:43.749370+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:04:26.847954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.749268+0000","last_change":"2024-03-06T07:23:42.100912+0000","last_active":"2024-03-06T07:23:43.749268+0000","last_peered":"2024-03-06T07:23:43.749268+0000","last_clean":"2024-03-06T07:23:43.749268+0000","last_became_active":"2024-03-06T07:23:42.100585+0000","last_became_peered":"2024-03-06T07:23:42.100585+0000","last_unstale":"2024-03-06T07:23:43.749268+0000","last_undegraded":"2024-03-06T07:23:43.749268+0000","last_fullsized":"2024-03-06T07:23:43.749268+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:31.905624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"44'6","reported_seq":20,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.102192+0000","last_change":"2024-03-06T07:23:41.099116+0000","last_active":"2024-03-06T07:23:43.102192+0000","last_peered":"2024-03-06T07:23:43.102192+0000","last_clean":"2024-03-06T07:23:43.102192+0000","last_became_active":"2024-03-06T07:23:41.098778+0000","last_became_peered":"2024-03-06T07:23:41.098778+0000","last_unstale":"2024-03-06T07:23:43.102192+0000","last_undegraded":"2024-03-06T07:23:43.102192+0000","last_fullsized":"2024-03-06T07:23:43.102192+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:20.126721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.102798+0000","last_change":"2024-03-06T07:23:42.105793+0000","last_active":"2024-03-06T07:23:43.102798+0000","last_peered":"2024-03-06T07:23:43.102798+0000","last_clean":"2024-03-06T07:23:43.102798+0000","last_became_active":"2024-03-06T07:23:42.105536+0000","last_became_peered":"2024-03-06T07:23:42.105536+0000","last_unstale":"2024-03-06T07:23:43.102798+0000","last_undegraded":"2024-03-06T07:23:43.102798+0000","last_fullsized":"2024-03-06T07:23:43.102798+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:14:24.834163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748688+0000","last_change":"2024-03-06T07:23:41.096739+0000","last_active":"2024-03-06T07:23:43.748688+0000","last_peered":"2024-03-06T07:23:43.748688+0000","last_clean":"2024-03-06T07:23:43.748688+0000","last_became_active":"2024-03-06T07:23:41.096035+0000","last_became_peered":"2024-03-06T07:23:41.096035+0000","last_unstale":"2024-03-06T07:23:43.748688+0000","last_undegraded":"2024-03-06T07:23:43.748688+0000","last_fullsized":"2024-03-06T07:23:43.748688+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:53.930256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748996+0000","last_change":"2024-03-06T07:23:42.112674+0000","last_active":"2024-03-06T07:23:43.748996+0000","last_peered":"2024-03-06T07:23:43.748996+0000","last_clean":"2024-03-06T07:23:43.748996+0000","last_became_active":"2024-03-06T07:23:42.112325+0000","last_became_peered":"2024-03-06T07:23:42.112325+0000","last_unstale":"2024-03-06T07:23:43.748996+0000","last_undegraded":"2024-03-06T07:23:43.748996+0000","last_fullsized":"2024-03-06T07:23:43.748996+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:35:56.747430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.749066+0000","last_change":"2024-03-06T07:23:41.097571+0000","last_active":"2024-03-06T07:23:43.749066+0000","last_peered":"2024-03-06T07:23:43.749066+0000","last_clean":"2024-03-06T07:23:43.749066+0000","last_became_active":"2024-03-06T07:23:41.097282+0000","last_became_peered":"2024-03-06T07:23:41.097282+0000","last_unstale":"2024-03-06T07:23:43.749066+0000","last_undegraded":"2024-03-06T07:23:43.749066+0000","last_fullsized":"2024-03-06T07:23:43.749066+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:00.236121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.747967+0000","last_change":"2024-03-06T07:23:42.103662+0000","last_active":"2024-03-06T07:23:43.747967+0000","last_peered":"2024-03-06T07:23:43.747967+0000","last_clean":"2024-03-06T07:23:43.747967+0000","last_became_active":"2024-03-06T07:23:42.103324+0000","last_became_peered":"2024-03-06T07:23:42.103324+0000","last_unstale":"2024-03-06T07:23:43.747967+0000","last_undegraded":"2024-03-06T07:23:43.747967+0000","last_fullsized":"2024-03-06T07:23:43.747967+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:13:30.550460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748882+0000","last_change":"2024-03-06T07:23:41.094487+0000","last_active":"2024-03-06T07:23:43.748882+0000","last_peered":"2024-03-06T07:23:43.748882+0000","last_clean":"2024-03-06T07:23:43.748882+0000","last_became_active":"2024-03-06T07:23:41.094218+0000","last_became_peered":"2024-03-06T07:23:41.094218+0000","last_unstale":"2024-03-06T07:23:43.748882+0000","last_undegraded":"2024-03-06T07:23:43.748882+0000","last_fullsized":"2024-03-06T07:23:43.748882+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:35:55.740692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.749149+0000","last_change":"2024-03-06T07:23:42.108171+0000","last_active":"2024-03-06T07:23:43.749149+0000","last_peered":"2024-03-06T07:23:43.749149+0000","last_clean":"2024-03-06T07:23:43.749149+0000","last_became_active":"2024-03-06T07:23:42.107811+0000","last_became_peered":"2024-03-06T07:23:42.107811+0000","last_unstale":"2024-03-06T07:23:43.749149+0000","last_undegraded":"2024-03-06T07:23:43.749149+0000","last_fullsized":"2024-03-06T07:23:43.749149+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:35.437029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"44'8","reported_seq":22,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748158+0000","last_change":"2024-03-06T07:23:41.097316+0000","last_active":"2024-03-06T07:23:43.748158+0000","last_peered":"2024-03-06T07:23:43.748158+0000","last_clean":"2024-03-06T07:23:43.748158+0000","last_became_active":"2024-03-06T07:23:41.097051+0000","last_became_peered":"2024-03-06T07:23:41.097051+0000","last_unstale":"2024-03-06T07:23:43.748158+0000","last_undegraded":"2024-03-06T07:23:43.748158+0000","last_fullsized":"2024-03-06T07:23:43.748158+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:09.775160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.102064+0000","last_change":"2024-03-06T07:23:42.100837+0000","last_active":"2024-03-06T07:23:43.102064+0000","last_peered":"2024-03-06T07:23:43.102064+0000","last_clean":"2024-03-06T07:23:43.102064+0000","last_became_active":"2024-03-06T07:23:42.100538+0000","last_became_peered":"2024-03-06T07:23:42.100538+0000","last_unstale":"2024-03-06T07:23:43.102064+0000","last_undegraded":"2024-03-06T07:23:43.102064+0000","last_fullsized":"2024-03-06T07:23:43.102064+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:10.781898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.102114+0000","last_change":"2024-03-06T07:23:41.090841+0000","last_active":"2024-03-06T07:23:43.102114+0000","last_peered":"2024-03-06T07:23:43.102114+0000","last_clean":"2024-03-06T07:23:43.102114+0000","last_became_active":"2024-03-06T07:23:41.090530+0000","last_became_peered":"2024-03-06T07:23:41.090530+0000","last_unstale":"2024-03-06T07:23:43.102114+0000","last_undegraded":"2024-03-06T07:23:43.102114+0000","last_fullsized":"2024-03-06T07:23:43.102114+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:08.213301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.749115+0000","last_change":"2024-03-06T07:23:42.114734+0000","last_active":"2024-03-06T07:23:43.749115+0000","last_peered":"2024-03-06T07:23:43.749115+0000","last_clean":"2024-03-06T07:23:43.749115+0000","last_became_active":"2024-03-06T07:23:42.114436+0000","last_became_peered":"2024-03-06T07:23:42.114436+0000","last_unstale":"2024-03-06T07:23:43.749115+0000","last_undegraded":"2024-03-06T07:23:43.749115+0000","last_fullsized":"2024-03-06T07:23:43.749115+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:01.242859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.103237+0000","last_change":"2024-03-06T07:23:41.098489+0000","last_active":"2024-03-06T07:23:43.103237+0000","last_peered":"2024-03-06T07:23:43.103237+0000","last_clean":"2024-03-06T07:23:43.103237+0000","last_became_active":"2024-03-06T07:23:41.098182+0000","last_became_peered":"2024-03-06T07:23:41.098182+0000","last_unstale":"2024-03-06T07:23:43.103237+0000","last_undegraded":"2024-03-06T07:23:43.103237+0000","last_fullsized":"2024-03-06T07:23:43.103237+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:14:23.827424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.103012+0000","last_change":"2024-03-06T07:23:42.108706+0000","last_active":"2024-03-06T07:23:43.103012+0000","last_peered":"2024-03-06T07:23:43.103012+0000","last_clean":"2024-03-06T07:23:43.103012+0000","last_became_active":"2024-03-06T07:23:42.108373+0000","last_became_peered":"2024-03-06T07:23:42.108373+0000","last_unstale":"2024-03-06T07:23:43.103012+0000","last_undegraded":"2024-03-06T07:23:43.103012+0000","last_fullsized":"2024-03-06T07:23:43.103012+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:50.568937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748306+0000","last_change":"2024-03-06T07:23:41.100506+0000","last_active":"2024-03-06T07:23:43.748306+0000","last_peered":"2024-03-06T07:23:43.748306+0000","last_clean":"2024-03-06T07:23:43.748306+0000","last_became_active":"2024-03-06T07:23:41.099926+0000","last_became_peered":"2024-03-06T07:23:41.099926+0000","last_unstale":"2024-03-06T07:23:43.748306+0000","last_undegraded":"2024-03-06T07:23:43.748306+0000","last_fullsized":"2024-03-06T07:23:43.748306+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:45:57.606322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.103086+0000","last_change":"2024-03-06T07:23:42.100948+0000","last_active":"2024-03-06T07:23:43.103086+0000","last_peered":"2024-03-06T07:23:43.103086+0000","last_clean":"2024-03-06T07:23:43.103086+0000","last_became_active":"2024-03-06T07:23:42.100226+0000","last_became_peered":"2024-03-06T07:23:42.100226+0000","last_unstale":"2024-03-06T07:23:43.103086+0000","last_undegraded":"2024-03-06T07:23:43.103086+0000","last_fullsized":"2024-03-06T07:23:43.103086+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:10:31.867655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748369+0000","last_change":"2024-03-06T07:23:41.102819+0000","last_active":"2024-03-06T07:23:43.748369+0000","last_peered":"2024-03-06T07:23:43.748369+0000","last_clean":"2024-03-06T07:23:43.748369+0000","last_became_active":"2024-03-06T07:23:41.102522+0000","last_became_peered":"2024-03-06T07:23:41.102522+0000","last_unstale":"2024-03-06T07:23:43.748369+0000","last_undegraded":"2024-03-06T07:23:43.748369+0000","last_fullsized":"2024-03-06T07:23:43.748369+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:00.236121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748906+0000","last_change":"2024-03-06T07:23:42.109387+0000","last_active":"2024-03-06T07:23:43.748906+0000","last_peered":"2024-03-06T07:23:43.748906+0000","last_clean":"2024-03-06T07:23:43.748906+0000","last_became_active":"2024-03-06T07:23:42.109118+0000","last_became_peered":"2024-03-06T07:23:42.109118+0000","last_unstale":"2024-03-06T07:23:43.748906+0000","last_undegraded":"2024-03-06T07:23:43.748906+0000","last_fullsized":"2024-03-06T07:23:43.748906+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:25:38.596240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748839+0000","last_change":"2024-03-06T07:23:41.095585+0000","last_active":"2024-03-06T07:23:43.748839+0000","last_peered":"2024-03-06T07:23:43.748839+0000","last_clean":"2024-03-06T07:23:43.748839+0000","last_became_active":"2024-03-06T07:23:41.095341+0000","last_became_peered":"2024-03-06T07:23:41.095341+0000","last_unstale":"2024-03-06T07:23:43.748839+0000","last_undegraded":"2024-03-06T07:23:43.748839+0000","last_fullsized":"2024-03-06T07:23:43.748839+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:11.030371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.747920+0000","last_change":"2024-03-06T07:23:42.099069+0000","last_active":"2024-03-06T07:23:43.747920+0000","last_peered":"2024-03-06T07:23:43.747920+0000","last_clean":"2024-03-06T07:23:43.747920+0000","last_became_active":"2024-03-06T07:23:42.098793+0000","last_became_peered":"2024-03-06T07:23:42.098793+0000","last_unstale":"2024-03-06T07:23:43.747920+0000","last_undegraded":"2024-03-06T07:23:43.747920+0000","last_fullsized":"2024-03-06T07:23:43.747920+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:26.696304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748041+0000","last_change":"2024-03-06T07:23:41.099634+0000","last_active":"2024-03-06T07:23:43.748041+0000","last_peered":"2024-03-06T07:23:43.748041+0000","last_clean":"2024-03-06T07:23:43.748041+0000","last_became_active":"2024-03-06T07:23:41.099364+0000","last_became_peered":"2024-03-06T07:23:41.099364+0000","last_unstale":"2024-03-06T07:23:43.748041+0000","last_undegraded":"2024-03-06T07:23:43.748041+0000","last_fullsized":"2024-03-06T07:23:43.748041+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:34:32.719704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.101880+0000","last_change":"2024-03-06T07:23:42.095201+0000","last_active":"2024-03-06T07:23:43.101880+0000","last_peered":"2024-03-06T07:23:43.101880+0000","last_clean":"2024-03-06T07:23:43.101880+0000","last_became_active":"2024-03-06T07:23:42.093370+0000","last_became_peered":"2024-03-06T07:23:42.093370+0000","last_unstale":"2024-03-06T07:23:43.101880+0000","last_undegraded":"2024-03-06T07:23:43.101880+0000","last_fullsized":"2024-03-06T07:23:43.101880+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:31.574000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.749695+0000","last_change":"2024-03-06T07:23:41.096450+0000","last_active":"2024-03-06T07:23:43.749695+0000","last_peered":"2024-03-06T07:23:43.749695+0000","last_clean":"2024-03-06T07:23:43.749695+0000","last_became_active":"2024-03-06T07:23:41.096210+0000","last_became_peered":"2024-03-06T07:23:41.096210+0000","last_unstale":"2024-03-06T07:23:43.749695+0000","last_undegraded":"2024-03-06T07:23:43.749695+0000","last_fullsized":"2024-03-06T07:23:43.749695+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:07.437814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.749651+0000","last_change":"2024-03-06T07:23:42.104521+0000","last_active":"2024-03-06T07:23:43.749651+0000","last_peered":"2024-03-06T07:23:43.749651+0000","last_clean":"2024-03-06T07:23:43.749651+0000","last_became_active":"2024-03-06T07:23:42.104220+0000","last_became_peered":"2024-03-06T07:23:42.104220+0000","last_unstale":"2024-03-06T07:23:43.749651+0000","last_undegraded":"2024-03-06T07:23:43.749651+0000","last_fullsized":"2024-03-06T07:23:43.749651+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:57:46.850738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748263+0000","last_change":"2024-03-06T07:23:41.096092+0000","last_active":"2024-03-06T07:23:43.748263+0000","last_peered":"2024-03-06T07:23:43.748263+0000","last_clean":"2024-03-06T07:23:43.748263+0000","last_became_active":"2024-03-06T07:23:41.094371+0000","last_became_peered":"2024-03-06T07:23:41.094371+0000","last_unstale":"2024-03-06T07:23:43.748263+0000","last_undegraded":"2024-03-06T07:23:43.748263+0000","last_fullsized":"2024-03-06T07:23:43.748263+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:45:57.606322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.103337+0000","last_change":"2024-03-06T07:23:42.108169+0000","last_active":"2024-03-06T07:23:43.103337+0000","last_peered":"2024-03-06T07:23:43.103337+0000","last_clean":"2024-03-06T07:23:43.103337+0000","last_became_active":"2024-03-06T07:23:42.107066+0000","last_became_peered":"2024-03-06T07:23:42.107066+0000","last_unstale":"2024-03-06T07:23:43.103337+0000","last_undegraded":"2024-03-06T07:23:43.103337+0000","last_fullsized":"2024-03-06T07:23:43.103337+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:58.192125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"45'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.749128+0000","last_change":"2024-03-06T07:23:41.100601+0000","last_active":"2024-03-06T07:23:43.749128+0000","last_peered":"2024-03-06T07:23:43.749128+0000","last_clean":"2024-03-06T07:23:43.749128+0000","last_became_active":"2024-03-06T07:23:41.100365+0000","last_became_peered":"2024-03-06T07:23:41.100365+0000","last_unstale":"2024-03-06T07:23:43.749128+0000","last_undegraded":"2024-03-06T07:23:43.749128+0000","last_fullsized":"2024-03-06T07:23:43.749128+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:29:23.109319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748750+0000","last_change":"2024-03-06T07:23:42.107366+0000","last_active":"2024-03-06T07:23:43.748750+0000","last_peered":"2024-03-06T07:23:43.748750+0000","last_clean":"2024-03-06T07:23:43.748750+0000","last_became_active":"2024-03-06T07:23:42.107088+0000","last_became_peered":"2024-03-06T07:23:42.107088+0000","last_unstale":"2024-03-06T07:23:43.748750+0000","last_undegraded":"2024-03-06T07:23:43.748750+0000","last_fullsized":"2024-03-06T07:23:43.748750+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:49:52.564351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.749379+0000","last_change":"2024-03-06T07:23:41.102841+0000","last_active":"2024-03-06T07:23:43.749379+0000","last_peered":"2024-03-06T07:23:43.749379+0000","last_clean":"2024-03-06T07:23:43.749379+0000","last_became_active":"2024-03-06T07:23:41.102564+0000","last_became_peered":"2024-03-06T07:23:41.102564+0000","last_unstale":"2024-03-06T07:23:43.749379+0000","last_undegraded":"2024-03-06T07:23:43.749379+0000","last_fullsized":"2024-03-06T07:23:43.749379+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:35:55.740692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.749304+0000","last_change":"2024-03-06T07:23:42.102019+0000","last_active":"2024-03-06T07:23:43.749304+0000","last_peered":"2024-03-06T07:23:43.749304+0000","last_clean":"2024-03-06T07:23:43.749304+0000","last_became_active":"2024-03-06T07:23:42.101745+0000","last_became_peered":"2024-03-06T07:23:42.101745+0000","last_unstale":"2024-03-06T07:23:43.749304+0000","last_undegraded":"2024-03-06T07:23:43.749304+0000","last_fullsized":"2024-03-06T07:23:43.749304+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:29.723189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.747979+0000","last_change":"2024-03-06T07:23:41.106360+0000","last_active":"2024-03-06T07:23:43.747979+0000","last_peered":"2024-03-06T07:23:43.747979+0000","last_clean":"2024-03-06T07:23:43.747979+0000","last_became_active":"2024-03-06T07:23:41.106134+0000","last_became_peered":"2024-03-06T07:23:41.106134+0000","last_unstale":"2024-03-06T07:23:43.747979+0000","last_undegraded":"2024-03-06T07:23:43.747979+0000","last_fullsized":"2024-03-06T07:23:43.747979+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:13.668717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.749048+0000","last_change":"2024-03-06T07:23:42.104488+0000","last_active":"2024-03-06T07:23:43.749048+0000","last_peered":"2024-03-06T07:23:43.749048+0000","last_clean":"2024-03-06T07:23:43.749048+0000","last_became_active":"2024-03-06T07:23:42.103898+0000","last_became_peered":"2024-03-06T07:23:42.103898+0000","last_unstale":"2024-03-06T07:23:43.749048+0000","last_undegraded":"2024-03-06T07:23:43.749048+0000","last_fullsized":"2024-03-06T07:23:43.749048+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:40.424045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.749203+0000","last_change":"2024-03-06T07:23:41.099446+0000","last_active":"2024-03-06T07:23:43.749203+0000","last_peered":"2024-03-06T07:23:43.749203+0000","last_clean":"2024-03-06T07:23:43.749203+0000","last_became_active":"2024-03-06T07:23:41.099181+0000","last_became_peered":"2024-03-06T07:23:41.099181+0000","last_unstale":"2024-03-06T07:23:43.749203+0000","last_undegraded":"2024-03-06T07:23:43.749203+0000","last_fullsized":"2024-03-06T07:23:43.749203+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:59:09.738026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748556+0000","last_change":"2024-03-06T07:23:42.109438+0000","last_active":"2024-03-06T07:23:43.748556+0000","last_peered":"2024-03-06T07:23:43.748556+0000","last_clean":"2024-03-06T07:23:43.748556+0000","last_became_active":"2024-03-06T07:23:42.109158+0000","last_became_peered":"2024-03-06T07:23:42.109158+0000","last_unstale":"2024-03-06T07:23:43.748556+0000","last_undegraded":"2024-03-06T07:23:43.748556+0000","last_fullsized":"2024-03-06T07:23:43.748556+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:37.631094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.102832+0000","last_change":"2024-03-06T07:23:41.099994+0000","last_active":"2024-03-06T07:23:43.102832+0000","last_peered":"2024-03-06T07:23:43.102832+0000","last_clean":"2024-03-06T07:23:43.102832+0000","last_became_active":"2024-03-06T07:23:41.099731+0000","last_became_peered":"2024-03-06T07:23:41.099731+0000","last_unstale":"2024-03-06T07:23:43.102832+0000","last_undegraded":"2024-03-06T07:23:43.102832+0000","last_fullsized":"2024-03-06T07:23:43.102832+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:13.668717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748071+0000","last_change":"2024-03-06T07:23:42.102864+0000","last_active":"2024-03-06T07:23:43.748071+0000","last_peered":"2024-03-06T07:23:43.748071+0000","last_clean":"2024-03-06T07:23:43.748071+0000","last_became_active":"2024-03-06T07:23:42.102594+0000","last_became_peered":"2024-03-06T07:23:42.102594+0000","last_unstale":"2024-03-06T07:23:43.748071+0000","last_undegraded":"2024-03-06T07:23:43.748071+0000","last_fullsized":"2024-03-06T07:23:43.748071+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:57:46.850738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748569+0000","last_change":"2024-03-06T07:23:41.099078+0000","last_active":"2024-03-06T07:23:43.748569+0000","last_peered":"2024-03-06T07:23:43.748569+0000","last_clean":"2024-03-06T07:23:43.748569+0000","last_became_active":"2024-03-06T07:23:41.098812+0000","last_became_peered":"2024-03-06T07:23:41.098812+0000","last_unstale":"2024-03-06T07:23:43.748569+0000","last_undegraded":"2024-03-06T07:23:43.748569+0000","last_fullsized":"2024-03-06T07:23:43.748569+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:14:23.827424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.102416+0000","last_change":"2024-03-06T07:23:42.104651+0000","last_active":"2024-03-06T07:23:43.102416+0000","last_peered":"2024-03-06T07:23:43.102416+0000","last_clean":"2024-03-06T07:23:43.102416+0000","last_became_active":"2024-03-06T07:23:42.104347+0000","last_became_peered":"2024-03-06T07:23:42.104347+0000","last_unstale":"2024-03-06T07:23:43.102416+0000","last_undegraded":"2024-03-06T07:23:43.102416+0000","last_fullsized":"2024-03-06T07:23:43.102416+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:18.102969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748255+0000","last_change":"2024-03-06T07:23:41.098205+0000","last_active":"2024-03-06T07:23:43.748255+0000","last_peered":"2024-03-06T07:23:43.748255+0000","last_clean":"2024-03-06T07:23:43.748255+0000","last_became_active":"2024-03-06T07:23:41.097800+0000","last_became_peered":"2024-03-06T07:23:41.097800+0000","last_unstale":"2024-03-06T07:23:43.748255+0000","last_undegraded":"2024-03-06T07:23:43.748255+0000","last_fullsized":"2024-03-06T07:23:43.748255+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:27.990220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.749574+0000","last_change":"2024-03-06T07:23:42.108189+0000","last_active":"2024-03-06T07:23:43.749574+0000","last_peered":"2024-03-06T07:23:43.749574+0000","last_clean":"2024-03-06T07:23:43.749574+0000","last_became_active":"2024-03-06T07:23:42.107945+0000","last_became_peered":"2024-03-06T07:23:42.107945+0000","last_unstale":"2024-03-06T07:23:43.749574+0000","last_undegraded":"2024-03-06T07:23:43.749574+0000","last_fullsized":"2024-03-06T07:23:43.749574+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:41.075905+0000","last_clean_scrub_stamp":"2024-03-06T07:23:41.075905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:29:24.116057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748396+0000","last_change":"2024-03-06T07:23:41.101372+0000","last_active":"2024-03-06T07:23:43.748396+0000","last_peered":"2024-03-06T07:23:43.748396+0000","last_clean":"2024-03-06T07:23:43.748396+0000","last_became_active":"2024-03-06T07:23:41.100976+0000","last_became_peered":"2024-03-06T07:23:41.100976+0000","last_unstale":"2024-03-06T07:23:43.748396+0000","last_undegraded":"2024-03-06T07:23:43.748396+0000","last_fullsized":"2024-03-06T07:23:43.748396+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:40.069167+0000","last_clean_scrub_stamp":"2024-03-06T07:23:40.069167+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:49:51.557613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":214,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:43.748370+0000","last_change":"2024-03-06T07:21:47.553693+0000","last_active":"2024-03-06T07:23:43.748370+0000","last_peered":"2024-03-06T07:23:43.748370+0000","last_clean":"2024-03-06T07:23:43.748370+0000","last_became_active":"2024-03-06T07:21:47.552664+0000","last_became_peered":"2024-03-06T07:21:47.552664+0000","last_unstale":"2024-03-06T07:23:43.748370+0000","last_undegraded":"2024-03-06T07:23:43.748370+0000","last_fullsized":"2024-03-06T07:23:43.748370+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_clean_scrub_stamp":"2024-03-06T07:21:46.362028+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:02.916426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574877,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32288,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339552,"statfs":{"total":96636764160,"available":96603701248,"internally_reserved":0,"allocated":1126400,"data_stored":806075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574875,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31712,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340128,"statfs":{"total":96636764160,"available":96604291072,"internally_reserved":0,"allocated":536576,"data_stored":216289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574876,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31708,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340132,"statfs":{"total":96636764160,"available":96604295168,"internally_reserved":0,"allocated":532480,"data_stored":215687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574877,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31716,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340124,"statfs":{"total":96636764160,"available":96604286976,"internally_reserved":0,"allocated":540672,"data_stored":217229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574876,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31716,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340124,"statfs":{"total":96636764160,"available":96604286976,"internally_reserved":0,"allocated":540672,"data_stored":216255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574876,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31720,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340120,"statfs":{"total":96636764160,"available":96604282880,"internally_reserved":0,"allocated":544768,"data_stored":217251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574877,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31708,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340132,"statfs":{"total":96636764160,"available":96604295168,"internally_reserved":0,"allocated":532480,"data_stored":215687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574876,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32280,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339560,"statfs":{"total":96636764160,"available":96603709440,"internally_reserved":0,"allocated":1126400,"data_stored":806019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:23:49.935 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:23:49.936 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:23:50.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.078+0000 7fe1e05d0640 1 Processor -- start 2024-03-06T07:23:50.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.078+0000 7fe1e05d0640 1 -- start start 2024-03-06T07:23:50.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.079+0000 7fe1e05d0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe1d8146b80 0x7fe1d8147890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.079+0000 7fe1e05d0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe1d8146f60 0x7fe1d8070e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.079+0000 7fe1e05d0640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d80b5080 con 0x7fe1d80b4870 2024-03-06T07:23:50.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.079+0000 7fe1e05d0640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1d80ad900 con 0x7fe1d8146b80 2024-03-06T07:23:50.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.079+0000 7fe1e05d0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d80adbe0 con 0x7fe1d8146f60 2024-03-06T07:23:50.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.079+0000 7fe1de36c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe1d8146b80 0x7fe1d8147890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.079+0000 7fe1deb6d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fe1d80b4870 0x7fe1d806e2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:53952/0 (socket says 172.21.15.83:53952) 2024-03-06T07:23:50.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.079+0000 7fe1deb6d640 1 -- 172.21.15.83:0/1827338513 learned_addr learned my addr 172.21.15.83:0/1827338513 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:23:50.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.079+0000 7fe1ddb6b640 1 --2- 172.21.15.83:0/1827338513 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe1d8146f60 0x7fe1d8070e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.079+0000 7fe1de36c640 1 -- 172.21.15.83:0/1827338513 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe1d8146f60 msgr2=0x7fe1d8070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:50.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.079+0000 7fe1de36c640 1 --2- 172.21.15.83:0/1827338513 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe1d8146f60 0x7fe1d8070e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.079+0000 7fe1de36c640 1 -- 172.21.15.83:0/1827338513 >> v1:172.21.15.83:6789/0 conn(0x7fe1d80b4870 legacy=0x7fe1d806e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.079+0000 7fe1dd36a640 1 -- 172.21.15.83:0/1827338513 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3355780215 0 0) 0x7fe1d80b5080 con 0x7fe1d80b4870 2024-03-06T07:23:50.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.079+0000 7fe1de36c640 1 -- 172.21.15.83:0/1827338513 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d80adec0 con 0x7fe1d8146b80 2024-03-06T07:23:50.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.079+0000 7fe1ddb6b640 1 --2- 172.21.15.83:0/1827338513 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe1d8146f60 0x7fe1d8070e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:50.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.080+0000 7fe1de36c640 1 --2- 172.21.15.83:0/1827338513 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1d8146b80 0x7fe1d8147890 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fe1d40097a0 tx=0x7fe1d402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81b44c02c10590cf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:50.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.080+0000 7fe1dd36a640 1 -- 172.21.15.83:0/1827338513 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d4035020 con 0x7fe1d8146b80 2024-03-06T07:23:50.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.080+0000 7fe1dd36a640 1 -- 172.21.15.83:0/1827338513 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1d4031830 con 0x7fe1d8146b80 2024-03-06T07:23:50.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.080+0000 7fe1dd36a640 1 -- 172.21.15.83:0/1827338513 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d4031c00 con 0x7fe1d8146b80 2024-03-06T07:23:50.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.081+0000 7fe1e05d0640 1 -- 172.21.15.83:0/1827338513 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1d8146b80 msgr2=0x7fe1d8147890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.081+0000 7fe1e05d0640 1 --2- 172.21.15.83:0/1827338513 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1d8146b80 0x7fe1d8147890 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fe1d40097a0 tx=0x7fe1d402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.081+0000 7fe1e05d0640 1 -- 172.21.15.83:0/1827338513 shutdown_connections 2024-03-06T07:23:50.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.081+0000 7fe1e05d0640 1 --2- 172.21.15.83:0/1827338513 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe1d8146f60 0x7fe1d8070e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.081+0000 7fe1e05d0640 1 --2- 172.21.15.83:0/1827338513 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1d8146b80 0x7fe1d8147890 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.081+0000 7fe1e05d0640 1 -- 172.21.15.83:0/1827338513 >> 172.21.15.83:0/1827338513 conn(0x7fe1d806b830 msgr2=0x7fe1d80b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:50.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.081+0000 7fe1e05d0640 1 -- 172.21.15.83:0/1827338513 shutdown_connections 2024-03-06T07:23:50.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.081+0000 7fe1e05d0640 1 -- 172.21.15.83:0/1827338513 wait complete. 2024-03-06T07:23:50.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.082+0000 7fe1e05d0640 1 Processor -- start 2024-03-06T07:23:50.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.082+0000 7fe1e05d0640 1 -- start start 2024-03-06T07:23:50.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.082+0000 7fe1e05d0640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe1d80b4870 0x7fe1d815d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.082+0000 7fe1e05d0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe1d8146b80 0x7fe1d812fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.082+0000 7fe1e05d0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1d8160d80 0x7fe1d8161230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.083+0000 7fe1e05d0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d80b4e30 con 0x7fe1d8160d80 2024-03-06T07:23:50.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.083+0000 7fe1e05d0640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d80b4cb0 con 0x7fe1d80b4870 2024-03-06T07:23:50.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.083+0000 7fe1e05d0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d806e9d0 con 0x7fe1d8146b80 2024-03-06T07:23:50.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.083+0000 7fe1deb6d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1d8160d80 0x7fe1d8161230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.083+0000 7fe1deb6d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1d8160d80 0x7fe1d8161230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:44682/0 (socket says 172.21.15.83:44682) 2024-03-06T07:23:50.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.083+0000 7fe1deb6d640 1 -- 172.21.15.83:0/2928195469 learned_addr learned my addr 172.21.15.83:0/2928195469 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:50.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.083+0000 7fe1de36c640 1 --2- 172.21.15.83:0/2928195469 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe1d80b4870 0x7fe1d815d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.083+0000 7fe1deb6d640 1 -- 172.21.15.83:0/2928195469 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe1d8146b80 msgr2=0x7fe1d812fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.083+0000 7fe1ddb6b640 1 --2- 172.21.15.83:0/2928195469 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe1d8146b80 0x7fe1d812fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.083+0000 7fe1deb6d640 1 --2- 172.21.15.83:0/2928195469 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe1d8146b80 0x7fe1d812fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.083+0000 7fe1deb6d640 1 -- 172.21.15.83:0/2928195469 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe1d80b4870 msgr2=0x7fe1d815d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.083+0000 7fe1deb6d640 1 --2- 172.21.15.83:0/2928195469 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe1d80b4870 0x7fe1d815d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.083+0000 7fe1deb6d640 1 -- 172.21.15.83:0/2928195469 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d815dd70 con 0x7fe1d8160d80 2024-03-06T07:23:50.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.083+0000 7fe1deb6d640 1 --2- 172.21.15.83:0/2928195469 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1d8160d80 0x7fe1d8161230 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fe1d000c7a0 tx=0x7fe1d000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:50.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.083+0000 7fe1ceffd640 1 -- 172.21.15.83:0/2928195469 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d0018070 con 0x7fe1d8160d80 2024-03-06T07:23:50.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.083+0000 7fe1ceffd640 1 -- 172.21.15.83:0/2928195469 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1d000a640 con 0x7fe1d8160d80 2024-03-06T07:23:50.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.083+0000 7fe1e05d0640 1 -- 172.21.15.83:0/2928195469 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1d81448c0 con 0x7fe1d8160d80 2024-03-06T07:23:50.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.083+0000 7fe1ceffd640 1 -- 172.21.15.83:0/2928195469 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d000aa70 con 0x7fe1d8160d80 2024-03-06T07:23:50.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.083+0000 7fe1e05d0640 1 -- 172.21.15.83:0/2928195469 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1d8144e80 con 0x7fe1d8160d80 2024-03-06T07:23:50.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.085+0000 7fe1e05d0640 1 -- 172.21.15.83:0/2928195469 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1d806e0b0 con 0x7fe1d8160d80 2024-03-06T07:23:50.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.085+0000 7fe1ceffd640 1 -- 172.21.15.83:0/2928195469 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe1d001b070 con 0x7fe1d8160d80 2024-03-06T07:23:50.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.085+0000 7fe1ceffd640 1 --2- 172.21.15.83:0/2928195469 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe1bc06b4d0 0x7fe1bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.085+0000 7fe1ceffd640 1 -- 172.21.15.83:0/2928195469 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe1d00897b0 con 0x7fe1d8160d80 2024-03-06T07:23:50.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.085+0000 7fe1de36c640 1 --2- 172.21.15.83:0/2928195469 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe1bc06b4d0 0x7fe1bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.085+0000 7fe1de36c640 1 --2- 172.21.15.83:0/2928195469 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe1bc06b4d0 0x7fe1bc06d990 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe1d4036000 tx=0x7fe1d4008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:50.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.089+0000 7fe1ceffd640 1 -- 172.21.15.83:0/2928195469 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe1d815dd70 con 0x7fe1d8160d80 2024-03-06T07:23:50.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.343+0000 7fe1e05d0640 1 -- 172.21.15.83:0/2928195469 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fe1d8068ff0 con 0x7fe1d8160d80 2024-03-06T07:23:50.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.345+0000 7fe1ceffd640 1 -- 172.21.15.83:0/2928195469 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v12) v1 ==== 190+0+147 (secure 0 0 0) 0x7fe1d0058630 con 0x7fe1d8160d80 2024-03-06T07:23:50.347 INFO:teuthology.orchestra.run.smithi083.stdout:[client.0] 2024-03-06T07:23:50.347 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBcGehlRwCIMhAA1DdbdQZPplu7LQOTK3BmSg== 2024-03-06T07:23:50.347 INFO:teuthology.orchestra.run.smithi083.stdout: caps mds = "allow" 2024-03-06T07:23:50.347 INFO:teuthology.orchestra.run.smithi083.stdout: caps mon = "allow r" 2024-03-06T07:23:50.347 INFO:teuthology.orchestra.run.smithi083.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:23:50.348 INFO:teuthology.orchestra.run.smithi083.stderr:updated caps for client.0 2024-03-06T07:23:50.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.347+0000 7fe1e05d0640 1 -- 172.21.15.83:0/2928195469 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe1bc06b4d0 msgr2=0x7fe1bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.347+0000 7fe1e05d0640 1 --2- 172.21.15.83:0/2928195469 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe1bc06b4d0 0x7fe1bc06d990 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe1d4036000 tx=0x7fe1d4008030 comp rx=0 tx=0).stop 2024-03-06T07:23:50.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.347+0000 7fe1e05d0640 1 -- 172.21.15.83:0/2928195469 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1d8160d80 msgr2=0x7fe1d8161230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.347+0000 7fe1e05d0640 1 --2- 172.21.15.83:0/2928195469 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1d8160d80 0x7fe1d8161230 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fe1d000c7a0 tx=0x7fe1d000cc60 comp rx=0 tx=0).stop 2024-03-06T07:23:50.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.347+0000 7fe1e05d0640 1 -- 172.21.15.83:0/2928195469 shutdown_connections 2024-03-06T07:23:50.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.347+0000 7fe1e05d0640 1 --2- 172.21.15.83:0/2928195469 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe1bc06b4d0 0x7fe1bc06d990 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.347+0000 7fe1e05d0640 1 --2- 172.21.15.83:0/2928195469 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1d8160d80 0x7fe1d8161230 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.347+0000 7fe1e05d0640 1 --2- 172.21.15.83:0/2928195469 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe1d8146b80 0x7fe1d812fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.347+0000 7fe1e05d0640 1 --2- 172.21.15.83:0/2928195469 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe1d80b4870 0x7fe1d815d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.347+0000 7fe1e05d0640 1 -- 172.21.15.83:0/2928195469 >> 172.21.15.83:0/2928195469 conn(0x7fe1d806b830 msgr2=0x7fe1d80af4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:50.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.347+0000 7fe1e05d0640 1 -- 172.21.15.83:0/2928195469 shutdown_connections 2024-03-06T07:23:50.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.347+0000 7fe1e05d0640 1 -- 172.21.15.83:0/2928195469 wait complete. 2024-03-06T07:23:50.361 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:50.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.460+0000 7f82618e0640 1 Processor -- start 2024-03-06T07:23:50.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.460+0000 7f82618e0640 1 -- start start 2024-03-06T07:23:50.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.461+0000 7f82618e0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f825c0d16e0 0x7f825c0d1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.461+0000 7f82618e0640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f825c0cebe0 con 0x7f825c15fa10 2024-03-06T07:23:50.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.461+0000 7f82618e0640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f825c0ceec0 con 0x7f825c15bdd0 2024-03-06T07:23:50.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.461+0000 7f82618e0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f825c0ce980 con 0x7f825c0d16e0 2024-03-06T07:23:50.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.461+0000 7f825a7fc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f825c15fa10 0x7f825c0ce270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:53962/0 (socket says 172.21.15.83:53962) 2024-03-06T07:23:50.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.461+0000 7f825a7fc640 1 -- 172.21.15.83:0/542659580 learned_addr learned my addr 172.21.15.83:0/542659580 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:23:50.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.461+0000 7f825b7fe640 1 --2- 172.21.15.83:0/542659580 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f825c0d16e0 0x7f825c0d1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.461+0000 7f8259ffb640 1 -- 172.21.15.83:0/542659580 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4117666395 0 0) 0x7f825c0cebe0 con 0x7f825c15fa10 2024-03-06T07:23:50.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.461+0000 7f8259ffb640 1 -- 172.21.15.83:0/542659580 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8244003520 con 0x7f825c15fa10 2024-03-06T07:23:50.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.462+0000 7f8259ffb640 1 -- 172.21.15.83:0/542659580 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 440255959 0 0) 0x7f8244003520 con 0x7f825c15fa10 2024-03-06T07:23:50.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.462+0000 7f8259ffb640 1 -- 172.21.15.83:0/542659580 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f825c0d16e0 msgr2=0x7f825c0d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:50.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.462+0000 7f8259ffb640 1 --2- 172.21.15.83:0/542659580 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f825c0d16e0 0x7f825c0d1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.462+0000 7f8259ffb640 1 -- 172.21.15.83:0/542659580 >> v1:172.21.15.183:6789/0 conn(0x7f825c15bdd0 legacy=0x7f825c0c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.462+0000 7f8259ffb640 1 -- 172.21.15.83:0/542659580 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f825c0cf1a0 con 0x7f825c15fa10 2024-03-06T07:23:50.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.462+0000 7f8259ffb640 1 -- 172.21.15.83:0/542659580 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f8250001670 con 0x7f825c15fa10 2024-03-06T07:23:50.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.462+0000 7f8259ffb640 1 -- 172.21.15.83:0/542659580 >> v1:172.21.15.83:6789/0 conn(0x7f825c15fa10 legacy=0x7f825c0ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.462+0000 7f8259ffb640 1 --2- 172.21.15.83:0/542659580 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f825c0d16e0 0x7f82440039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.462+0000 7f8259ffb640 1 --2- 172.21.15.83:0/542659580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8244003f20 0x7f8244004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.462+0000 7f8259ffb640 1 --2- 172.21.15.83:0/542659580 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8244004d40 0x7f82440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.462+0000 7f8259ffb640 1 -- 172.21.15.83:0/542659580 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f825c0ce980 con 0x7f8244003f20 2024-03-06T07:23:50.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.462+0000 7f8259ffb640 1 -- 172.21.15.83:0/542659580 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f824400bad0 con 0x7f8244004d40 2024-03-06T07:23:50.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.462+0000 7f8259ffb640 1 -- 172.21.15.83:0/542659580 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f824400bcd0 con 0x7f825c0d16e0 2024-03-06T07:23:50.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.462+0000 7f825a7fc640 1 --2- 172.21.15.83:0/542659580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8244003f20 0x7f8244004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.462+0000 7f825affd640 1 --2- 172.21.15.83:0/542659580 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f825c0d16e0 0x7f82440039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.462+0000 7f825b7fe640 1 --2- 172.21.15.83:0/542659580 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8244004d40 0x7f82440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.462+0000 7f825a7fc640 1 -- 172.21.15.83:0/542659580 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f825c0d16e0 msgr2=0x7f82440039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:50.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.462+0000 7f825a7fc640 1 --2- 172.21.15.83:0/542659580 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f825c0d16e0 0x7f82440039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.462+0000 7f825a7fc640 1 -- 172.21.15.83:0/542659580 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8244004d40 msgr2=0x7f82440051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:50.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.462+0000 7f825a7fc640 1 --2- 172.21.15.83:0/542659580 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8244004d40 0x7f82440051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.462+0000 7f825a7fc640 1 -- 172.21.15.83:0/542659580 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8244003520 con 0x7f8244003f20 2024-03-06T07:23:50.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.463+0000 7f825a7fc640 1 --2- 172.21.15.83:0/542659580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8244003f20 0x7f8244004800 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f8250007fd0 tx=0x7f8244005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e655988a61d0744a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:50.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.463+0000 7f8259ffb640 1 -- 172.21.15.83:0/542659580 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8250007d50 con 0x7f8244003f20 2024-03-06T07:23:50.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.463+0000 7f8259ffb640 1 -- 172.21.15.83:0/542659580 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8250030070 con 0x7f8244003f20 2024-03-06T07:23:50.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.463+0000 7f8259ffb640 1 -- 172.21.15.83:0/542659580 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8250005d70 con 0x7f8244003f20 2024-03-06T07:23:50.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.463+0000 7f82618e0640 1 -- 172.21.15.83:0/542659580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8244003f20 msgr2=0x7f8244004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.463+0000 7f82618e0640 1 --2- 172.21.15.83:0/542659580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8244003f20 0x7f8244004800 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f8250007fd0 tx=0x7f8244005830 comp rx=0 tx=0).stop 2024-03-06T07:23:50.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.463+0000 7f825b7fe640 1 -- 172.21.15.83:0/542659580 reap_dead start 2024-03-06T07:23:50.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.464+0000 7f82618e0640 1 -- 172.21.15.83:0/542659580 shutdown_connections 2024-03-06T07:23:50.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.464+0000 7f82618e0640 1 -- 172.21.15.83:0/542659580 >> 172.21.15.83:0/542659580 conn(0x7f825c09a9f0 msgr2=0x7f825c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:50.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.464+0000 7f82618e0640 1 -- 172.21.15.83:0/542659580 shutdown_connections 2024-03-06T07:23:50.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.464+0000 7f82618e0640 1 -- 172.21.15.83:0/542659580 wait complete. 2024-03-06T07:23:50.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.465+0000 7f82618e0640 1 Processor -- start 2024-03-06T07:23:50.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.465+0000 7f82618e0640 1 -- start start 2024-03-06T07:23:50.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.466+0000 7f82618e0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8244003f20 0x7f825c158ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.466+0000 7f82618e0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f825c159310 0x7f825c1ef7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.466+0000 7f82618e0640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f825c1efd10 0x7f825c1f0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.466+0000 7f82618e0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f825c15c1b0 con 0x7f8244003f20 2024-03-06T07:23:50.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.466+0000 7f82618e0640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f825c0cc5d0 con 0x7f825c1efd10 2024-03-06T07:23:50.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.466+0000 7f82618e0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f825c0d1b10 con 0x7f825c159310 2024-03-06T07:23:50.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.466+0000 7f825affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8244003f20 0x7f825c158ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.466+0000 7f825affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8244003f20 0x7f825c158ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:44694/0 (socket says 172.21.15.83:44694) 2024-03-06T07:23:50.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.466+0000 7f825affd640 1 -- 172.21.15.83:0/2645846287 learned_addr learned my addr 172.21.15.83:0/2645846287 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:50.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.466+0000 7f825a7fc640 1 --2- 172.21.15.83:0/2645846287 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f825c159310 0x7f825c1ef7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.466+0000 7f825b7fe640 1 --2- 172.21.15.83:0/2645846287 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f825c1efd10 0x7f825c1f0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.466+0000 7f825affd640 1 -- 172.21.15.83:0/2645846287 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f825c159310 msgr2=0x7f825c1ef7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.466+0000 7f825affd640 1 --2- 172.21.15.83:0/2645846287 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f825c159310 0x7f825c1ef7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.466+0000 7f825affd640 1 -- 172.21.15.83:0/2645846287 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f825c1efd10 msgr2=0x7f825c1f0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.466+0000 7f825affd640 1 --2- 172.21.15.83:0/2645846287 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f825c1efd10 0x7f825c1f0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.466+0000 7f825affd640 1 -- 172.21.15.83:0/2645846287 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f825c15dff0 con 0x7f8244003f20 2024-03-06T07:23:50.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.466+0000 7f825affd640 1 --2- 172.21.15.83:0/2645846287 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8244003f20 0x7f825c158ca0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f825400fb60 tx=0x7f825400d520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:50.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.466+0000 7f8243fff640 1 -- 172.21.15.83:0/2645846287 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825400dc30 con 0x7f8244003f20 2024-03-06T07:23:50.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.466+0000 7f8243fff640 1 -- 172.21.15.83:0/2645846287 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8254018070 con 0x7f8244003f20 2024-03-06T07:23:50.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.466+0000 7f82618e0640 1 -- 172.21.15.83:0/2645846287 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f825c15d230 con 0x7f8244003f20 2024-03-06T07:23:50.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.466+0000 7f8243fff640 1 -- 172.21.15.83:0/2645846287 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8254003970 con 0x7f8244003f20 2024-03-06T07:23:50.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.466+0000 7f82618e0640 1 -- 172.21.15.83:0/2645846287 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f825c15d7f0 con 0x7f8244003f20 2024-03-06T07:23:50.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.468+0000 7f82618e0640 1 -- 172.21.15.83:0/2645846287 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f825c04f330 con 0x7f8244003f20 2024-03-06T07:23:50.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.468+0000 7f8243fff640 1 -- 172.21.15.83:0/2645846287 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8254003b30 con 0x7f8244003f20 2024-03-06T07:23:50.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.468+0000 7f8243fff640 1 --2- 172.21.15.83:0/2645846287 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f824806b500 0x7f824806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.468+0000 7f825a7fc640 1 --2- 172.21.15.83:0/2645846287 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f824806b500 0x7f824806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.468+0000 7f8243fff640 1 -- 172.21.15.83:0/2645846287 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f825408a490 con 0x7f8244003f20 2024-03-06T07:23:50.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.469+0000 7f825a7fc640 1 --2- 172.21.15.83:0/2645846287 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f824806b500 0x7f824806d9c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8250034000 tx=0x7f8250009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:50.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.472+0000 7f8243fff640 1 -- 172.21.15.83:0/2645846287 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8254003e20 con 0x7f8244003f20 2024-03-06T07:23:50.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.688+0000 7f82618e0640 1 -- 172.21.15.83:0/2645846287 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f825c0b7a40 con 0x7f8244003f20 2024-03-06T07:23:50.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.688+0000 7f8243fff640 1 -- 172.21.15.83:0/2645846287 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f82540593a0 con 0x7f8244003f20 2024-03-06T07:23:50.691 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:23:50.691 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":19,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5395,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/2270670914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2270670914},{"type":"v1","addr":"172.21.15.83:6837","nonce":2270670914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5398,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/2359880029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2359880029},{"type":"v1","addr":"172.21.15.183:6834","nonce":2359880029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5401,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3099809273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":3099809273},{"type":"v1","addr":"172.21.15.183:6835","nonce":3099809273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:41.430159+0000","modified":"2024-03-06T07:23:43.445803+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5706},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5706":{"gid":5706,"name":"a","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6835/534553538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":534553538},{"type":"v1","addr":"172.21.15.83:6835","nonce":534553538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5706,"qdb_cluster":[5706]},"id":3}]} 2024-03-06T07:23:50.691 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 19 2024-03-06T07:23:50.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.691+0000 7f82618e0640 1 -- 172.21.15.83:0/2645846287 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f824806b500 msgr2=0x7f824806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.691+0000 7f82618e0640 1 --2- 172.21.15.83:0/2645846287 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f824806b500 0x7f824806d9c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8250034000 tx=0x7f8250009030 comp rx=0 tx=0).stop 2024-03-06T07:23:50.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.691+0000 7f82618e0640 1 -- 172.21.15.83:0/2645846287 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8244003f20 msgr2=0x7f825c158ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.691+0000 7f82618e0640 1 --2- 172.21.15.83:0/2645846287 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8244003f20 0x7f825c158ca0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f825400fb60 tx=0x7f825400d520 comp rx=0 tx=0).stop 2024-03-06T07:23:50.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.691+0000 7f82618e0640 1 -- 172.21.15.83:0/2645846287 shutdown_connections 2024-03-06T07:23:50.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.691+0000 7f82618e0640 1 --2- 172.21.15.83:0/2645846287 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f824806b500 0x7f824806d9c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.691+0000 7f82618e0640 1 --2- 172.21.15.83:0/2645846287 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f825c1efd10 0x7f825c1f0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.691+0000 7f82618e0640 1 --2- 172.21.15.83:0/2645846287 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f825c159310 0x7f825c1ef7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.691+0000 7f82618e0640 1 --2- 172.21.15.83:0/2645846287 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8244003f20 0x7f825c158ca0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.691+0000 7f82618e0640 1 -- 172.21.15.83:0/2645846287 >> 172.21.15.83:0/2645846287 conn(0x7f825c09a9f0 msgr2=0x7f825c0ccd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:50.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.691+0000 7f82618e0640 1 -- 172.21.15.83:0/2645846287 shutdown_connections 2024-03-06T07:23:50.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:50.691+0000 7f82618e0640 1 -- 172.21.15.83:0/2645846287 wait complete. 2024-03-06T07:23:51.108 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 19, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:23:41.430159+0000', 'modified': '2024-03-06T07:23:43.445803+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5706}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5706': {'gid': 5706, 'name': 'a', 'rank': 0, 'incarnation': 17, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.83:6835/534553538', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 534553538}, {'type': 'v1', 'addr': '172.21.15.83:6835', 'nonce': 534553538}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5706, 'qdb_cluster': [5706]} 2024-03-06T07:23:51.108 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:23:51.109 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:23:51.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.219+0000 7fdb45555640 1 Processor -- start 2024-03-06T07:23:51.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.219+0000 7fdb45555640 1 -- start start 2024-03-06T07:23:51.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.219+0000 7fdb45555640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdb4015e000 0x7fdb400d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.219+0000 7fdb45555640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fdb400cf7d0 0x7fdb400d0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.219+0000 7fdb45555640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb400d25e0 con 0x7fdb400d4810 2024-03-06T07:23:51.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.219+0000 7fdb45555640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb400c7c50 con 0x7fdb400cf7d0 2024-03-06T07:23:51.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.219+0000 7fdb45555640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb400d2880 con 0x7fdb4015e000 2024-03-06T07:23:51.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.220+0000 7fdb367fc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fdb400d4810 0x7fdb400c80e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:53972/0 (socket says 172.21.15.83:53972) 2024-03-06T07:23:51.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.220+0000 7fdb367fc640 1 -- 172.21.15.83:0/4157308163 learned_addr learned my addr 172.21.15.83:0/4157308163 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:23:51.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.220+0000 7fdb3f7fe640 1 --2- 172.21.15.83:0/4157308163 >> v2:172.21.15.183:3300/0 conn(0x7fdb400cf7d0 0x7fdb400d0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.220+0000 7fdb3effd640 1 --2- 172.21.15.83:0/4157308163 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdb4015e000 0x7fdb400d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.220+0000 7fdb3e7fc640 1 -- 172.21.15.83:0/4157308163 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3052508961 0 0) 0x7fdb400d25e0 con 0x7fdb400d4810 2024-03-06T07:23:51.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.220+0000 7fdb3e7fc640 1 -- 172.21.15.83:0/4157308163 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb2c003520 con 0x7fdb400d4810 2024-03-06T07:23:51.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.220+0000 7fdb3e7fc640 1 -- 172.21.15.83:0/4157308163 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1374538907 0 0) 0x7fdb2c003520 con 0x7fdb400d4810 2024-03-06T07:23:51.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.220+0000 7fdb3e7fc640 1 -- 172.21.15.83:0/4157308163 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdb4015e000 msgr2=0x7fdb400d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:51.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.220+0000 7fdb3e7fc640 1 --2- 172.21.15.83:0/4157308163 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdb4015e000 0x7fdb400d0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.220+0000 7fdb3e7fc640 1 -- 172.21.15.83:0/4157308163 >> v2:172.21.15.183:3300/0 conn(0x7fdb400cf7d0 msgr2=0x7fdb400d0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:51.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.220+0000 7fdb3e7fc640 1 --2- 172.21.15.83:0/4157308163 >> v2:172.21.15.183:3300/0 conn(0x7fdb400cf7d0 0x7fdb400d0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.220+0000 7fdb3e7fc640 1 -- 172.21.15.83:0/4157308163 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb400d2b40 con 0x7fdb400d4810 2024-03-06T07:23:51.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.220+0000 7fdb3e7fc640 1 -- 172.21.15.83:0/4157308163 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7fdb28001670 con 0x7fdb400d4810 2024-03-06T07:23:51.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.220+0000 7fdb3e7fc640 1 -- 172.21.15.83:0/4157308163 >> v1:172.21.15.83:6789/0 conn(0x7fdb400d4810 legacy=0x7fdb400c80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.220+0000 7fdb3e7fc640 1 --2- 172.21.15.83:0/4157308163 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdb4015e000 0x7fdb2c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.220+0000 7fdb3e7fc640 1 --2- 172.21.15.83:0/4157308163 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdb2c003f30 0x7fdb2c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.220+0000 7fdb3e7fc640 1 --2- 172.21.15.83:0/4157308163 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb2c004d40 0x7fdb2c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.220+0000 7fdb3e7fc640 1 -- 172.21.15.83:0/4157308163 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb400c7c50 con 0x7fdb2c004d40 2024-03-06T07:23:51.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.220+0000 7fdb3e7fc640 1 -- 172.21.15.83:0/4157308163 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb400d2880 con 0x7fdb2c003f30 2024-03-06T07:23:51.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.220+0000 7fdb3e7fc640 1 -- 172.21.15.83:0/4157308163 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2c00b9f0 con 0x7fdb4015e000 2024-03-06T07:23:51.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.220+0000 7fdb3f7fe640 1 --2- 172.21.15.83:0/4157308163 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb2c004d40 0x7fdb2c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.220+0000 7fdb3effd640 1 --2- 172.21.15.83:0/4157308163 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdb4015e000 0x7fdb2c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.220+0000 7fdb367fc640 1 --2- 172.21.15.83:0/4157308163 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdb2c003f30 0x7fdb2c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.221+0000 7fdb3f7fe640 1 -- 172.21.15.83:0/4157308163 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdb4015e000 msgr2=0x7fdb2c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:51.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.221+0000 7fdb3f7fe640 1 --2- 172.21.15.83:0/4157308163 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdb4015e000 0x7fdb2c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.221+0000 7fdb3f7fe640 1 -- 172.21.15.83:0/4157308163 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdb2c003f30 msgr2=0x7fdb2c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:51.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.221+0000 7fdb3f7fe640 1 --2- 172.21.15.83:0/4157308163 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdb2c003f30 0x7fdb2c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.221+0000 7fdb3f7fe640 1 -- 172.21.15.83:0/4157308163 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb2c003520 con 0x7fdb2c004d40 2024-03-06T07:23:51.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.222+0000 7fdb3f7fe640 1 --2- 172.21.15.83:0/4157308163 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb2c004d40 0x7fdb2c0051c0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fdb38009fc0 tx=0x7fdb2c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=926a2e46a864e97 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:51.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.222+0000 7fdb3e7fc640 1 -- 172.21.15.83:0/4157308163 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb38039070 con 0x7fdb2c004d40 2024-03-06T07:23:51.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.222+0000 7fdb3e7fc640 1 -- 172.21.15.83:0/4157308163 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb3802cc60 con 0x7fdb2c004d40 2024-03-06T07:23:51.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.222+0000 7fdb3e7fc640 1 -- 172.21.15.83:0/4157308163 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb38009c10 con 0x7fdb2c004d40 2024-03-06T07:23:51.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.222+0000 7fdb45555640 1 -- 172.21.15.83:0/4157308163 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb2c004d40 msgr2=0x7fdb2c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.222+0000 7fdb45555640 1 --2- 172.21.15.83:0/4157308163 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb2c004d40 0x7fdb2c0051c0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fdb38009fc0 tx=0x7fdb2c005830 comp rx=0 tx=0).stop 2024-03-06T07:23:51.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.222+0000 7fdb3f7fe640 1 -- 172.21.15.83:0/4157308163 reap_dead start 2024-03-06T07:23:51.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.222+0000 7fdb45555640 1 -- 172.21.15.83:0/4157308163 shutdown_connections 2024-03-06T07:23:51.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.222+0000 7fdb45555640 1 -- 172.21.15.83:0/4157308163 >> 172.21.15.83:0/4157308163 conn(0x7fdb400d3cf0 msgr2=0x7fdb400d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:51.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.223+0000 7fdb45555640 1 -- 172.21.15.83:0/4157308163 shutdown_connections 2024-03-06T07:23:51.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.223+0000 7fdb45555640 1 -- 172.21.15.83:0/4157308163 wait complete. 2024-03-06T07:23:51.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.224+0000 7fdb45555640 1 Processor -- start 2024-03-06T07:23:51.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.224+0000 7fdb45555640 1 -- start start 2024-03-06T07:23:51.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.224+0000 7fdb45555640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdb40160910 0x7fdb40160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.224+0000 7fdb45555640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb4009f9e0 0x7fdb401f6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.225+0000 7fdb45555640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdb401f73f0 0x7fdb401f2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.225+0000 7fdb45555640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4015e3e0 con 0x7fdb4009f9e0 2024-03-06T07:23:51.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.225+0000 7fdb45555640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb400d22d0 con 0x7fdb401f73f0 2024-03-06T07:23:51.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.225+0000 7fdb45555640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb400d0680 con 0x7fdb40160910 2024-03-06T07:23:51.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.225+0000 7fdb367fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb4009f9e0 0x7fdb401f6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.225+0000 7fdb367fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb4009f9e0 0x7fdb401f6eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:44702/0 (socket says 172.21.15.83:44702) 2024-03-06T07:23:51.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.225+0000 7fdb367fc640 1 -- 172.21.15.83:0/1456894698 learned_addr learned my addr 172.21.15.83:0/1456894698 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:51.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.225+0000 7fdb3effd640 1 --2- 172.21.15.83:0/1456894698 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdb40160910 0x7fdb40160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.225+0000 7fdb3f7fe640 1 --2- 172.21.15.83:0/1456894698 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdb401f73f0 0x7fdb401f2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.225+0000 7fdb367fc640 1 -- 172.21.15.83:0/1456894698 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdb40160910 msgr2=0x7fdb40160cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.225+0000 7fdb367fc640 1 --2- 172.21.15.83:0/1456894698 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdb40160910 0x7fdb40160cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.225+0000 7fdb367fc640 1 -- 172.21.15.83:0/1456894698 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdb401f73f0 msgr2=0x7fdb401f2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.225+0000 7fdb367fc640 1 --2- 172.21.15.83:0/1456894698 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdb401f73f0 0x7fdb401f2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.225+0000 7fdb367fc640 1 -- 172.21.15.83:0/1456894698 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb400b3330 con 0x7fdb4009f9e0 2024-03-06T07:23:51.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.225+0000 7fdb3effd640 1 --2- 172.21.15.83:0/1456894698 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdb40160910 0x7fdb40160cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:51.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.225+0000 7fdb3f7fe640 1 --2- 172.21.15.83:0/1456894698 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdb401f73f0 0x7fdb401f2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:51.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.225+0000 7fdb367fc640 1 --2- 172.21.15.83:0/1456894698 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb4009f9e0 0x7fdb401f6eb0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fdb280072c0 tx=0x7fdb28007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:51.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.225+0000 7fdb37fff640 1 -- 172.21.15.83:0/1456894698 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb2800f9c0 con 0x7fdb4009f9e0 2024-03-06T07:23:51.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.225+0000 7fdb37fff640 1 -- 172.21.15.83:0/1456894698 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb28018070 con 0x7fdb4009f9e0 2024-03-06T07:23:51.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.225+0000 7fdb37fff640 1 -- 172.21.15.83:0/1456894698 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb2800fde0 con 0x7fdb4009f9e0 2024-03-06T07:23:51.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.225+0000 7fdb45555640 1 -- 172.21.15.83:0/1456894698 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb401f7a50 con 0x7fdb4009f9e0 2024-03-06T07:23:51.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.225+0000 7fdb45555640 1 -- 172.21.15.83:0/1456894698 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb400b3880 con 0x7fdb4009f9e0 2024-03-06T07:23:51.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.227+0000 7fdb37fff640 1 -- 172.21.15.83:0/1456894698 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdb2801d460 con 0x7fdb4009f9e0 2024-03-06T07:23:51.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.227+0000 7fdb37fff640 1 --2- 172.21.15.83:0/1456894698 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdb2006b500 0x7fdb2006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.227+0000 7fdb37fff640 1 -- 172.21.15.83:0/1456894698 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdb2808a390 con 0x7fdb4009f9e0 2024-03-06T07:23:51.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.227+0000 7fdb3effd640 1 --2- 172.21.15.83:0/1456894698 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdb2006b500 0x7fdb2006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.228+0000 7fdb3effd640 1 --2- 172.21.15.83:0/1456894698 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdb2006b500 0x7fdb2006d9c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fdb30009ab0 tx=0x7fdb30008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:51.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.228+0000 7fdb4820f740 1 Processor -- start 2024-03-06T07:23:51.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.229+0000 7fdb4820f740 1 -- start start 2024-03-06T07:23:51.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.230+0000 7fdb4820f740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56453d76e870 0x56453d76ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.230+0000 7fdb4820f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56453d7728c0 0x56453d774360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.230+0000 7fdb4820f740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56453d79e600 0x56453d79ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.230+0000 7fdb4820f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x56453d6d86b0 con 0x56453d7728c0 2024-03-06T07:23:51.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.230+0000 7fdb4820f740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x56453d681a60 con 0x56453d76e870 2024-03-06T07:23:51.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.230+0000 7fdb4820f740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x56453d6d4750 con 0x56453d79e600 2024-03-06T07:23:51.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.230+0000 7fdb367fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56453d7728c0 0x56453d774360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.230+0000 7fdb367fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56453d7728c0 0x56453d774360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:44704/0 (socket says 172.21.15.83:44704) 2024-03-06T07:23:51.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.230+0000 7fdb367fc640 1 -- 172.21.15.83:0/277085764 learned_addr learned my addr 172.21.15.83:0/277085764 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:51.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.230+0000 7fdb3f7fe640 1 --2- 172.21.15.83:0/277085764 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56453d79e600 0x56453d79ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.230+0000 7fdb3effd640 1 --2- 172.21.15.83:0/277085764 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56453d76e870 0x56453d76ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.230+0000 7fdb367fc640 1 -- 172.21.15.83:0/277085764 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56453d79e600 msgr2=0x56453d79ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:51.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.230+0000 7fdb367fc640 1 --2- 172.21.15.83:0/277085764 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56453d79e600 0x56453d79ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.230+0000 7fdb367fc640 1 -- 172.21.15.83:0/277085764 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56453d76e870 msgr2=0x56453d76ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:51.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.230+0000 7fdb367fc640 1 --2- 172.21.15.83:0/277085764 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56453d76e870 0x56453d76ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.230+0000 7fdb367fc640 1 -- 172.21.15.83:0/277085764 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56453d69ff30 con 0x56453d7728c0 2024-03-06T07:23:51.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.230+0000 7fdb367fc640 1 --2- 172.21.15.83:0/277085764 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56453d7728c0 0x56453d774360 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fdb28088800 tx=0x7fdb280585a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fd40d0075f3c86a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:51.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.231+0000 7fdb34ff9640 1 -- 172.21.15.83:0/277085764 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb28061220 con 0x56453d7728c0 2024-03-06T07:23:51.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.231+0000 7fdb34ff9640 1 -- 172.21.15.83:0/277085764 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb28056e10 con 0x56453d7728c0 2024-03-06T07:23:51.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.231+0000 7fdb34ff9640 1 -- 172.21.15.83:0/277085764 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb28055180 con 0x56453d7728c0 2024-03-06T07:23:51.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.231+0000 7fdb4820f740 1 -- 172.21.15.83:0/277085764 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56453d7728c0 msgr2=0x56453d774360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.231+0000 7fdb4820f740 1 --2- 172.21.15.83:0/277085764 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56453d7728c0 0x56453d774360 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fdb28088800 tx=0x7fdb280585a0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.231+0000 7fdb4820f740 1 -- 172.21.15.83:0/277085764 shutdown_connections 2024-03-06T07:23:51.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.231+0000 7fdb4820f740 1 --2- 172.21.15.83:0/277085764 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56453d79e600 0x56453d79ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.231+0000 7fdb4820f740 1 --2- 172.21.15.83:0/277085764 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56453d7728c0 0x56453d774360 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.231+0000 7fdb4820f740 1 --2- 172.21.15.83:0/277085764 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56453d76e870 0x56453d76ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.231+0000 7fdb4820f740 1 -- 172.21.15.83:0/277085764 >> 172.21.15.83:0/277085764 conn(0x56453d7708f0 msgr2=0x56453d7721b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:51.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.232+0000 7fdb4820f740 1 -- 172.21.15.83:0/277085764 shutdown_connections 2024-03-06T07:23:51.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.232+0000 7fdb4820f740 1 -- 172.21.15.83:0/277085764 wait complete. 2024-03-06T07:23:51.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.232+0000 7fdb4820f740 1 -- start start 2024-03-06T07:23:51.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.232+0000 7fdb4820f740 1 Processor -- start 2024-03-06T07:23:51.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.233+0000 7fdb4820f740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56453d79e600 0x56453d798710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.233+0000 7fdb4820f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56453d7b7160 0x56453d7a1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.233+0000 7fdb4820f740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56453d7b7540 0x56453d7a1e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.233+0000 7fdb4820f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x56453d772d50 con 0x56453d7b7160 2024-03-06T07:23:51.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.233+0000 7fdb4820f740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x56453d694ea0 con 0x56453d7b7540 2024-03-06T07:23:51.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.233+0000 7fdb4820f740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x56453d773030 con 0x56453d79e600 2024-03-06T07:23:51.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.233+0000 7fdb4820f740 10 client.0 _ll_register_callbacks cb 0x56453d694870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:23:51.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.233+0000 7fdb367fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56453d7b7160 0x56453d7a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.233+0000 7fdb367fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56453d7b7160 0x56453d7a1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:44706/0 (socket says 172.21.15.83:44706) 2024-03-06T07:23:51.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.233+0000 7fdb367fc640 1 -- 172.21.15.83:0/2587396383 learned_addr learned my addr 172.21.15.83:0/2587396383 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:51.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.233+0000 7fdb3f7fe640 1 --2- 172.21.15.83:0/2587396383 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56453d7b7540 0x56453d7a1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.233+0000 7fdb3effd640 1 --2- 172.21.15.83:0/2587396383 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56453d79e600 0x56453d798710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.233+0000 7fdb367fc640 1 -- 172.21.15.83:0/2587396383 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56453d79e600 msgr2=0x56453d798710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:51.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.233+0000 7fdb367fc640 1 --2- 172.21.15.83:0/2587396383 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56453d79e600 0x56453d798710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.233+0000 7fdb367fc640 1 -- 172.21.15.83:0/2587396383 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56453d7b7540 msgr2=0x56453d7a1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:51.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.233+0000 7fdb367fc640 1 --2- 172.21.15.83:0/2587396383 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56453d7b7540 0x56453d7a1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.233+0000 7fdb367fc640 1 -- 172.21.15.83:0/2587396383 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56453d695270 con 0x56453d7b7160 2024-03-06T07:23:51.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.233+0000 7fdb3f7fe640 1 --2- 172.21.15.83:0/2587396383 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56453d7b7540 0x56453d7a1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:51.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.233+0000 7fdb367fc640 1 --2- 172.21.15.83:0/2587396383 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56453d7b7160 0x56453d7a1920 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fdb28061190 tx=0x7fdb28015040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36c03e0c33531093 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:51.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.233+0000 7fdb1dffb640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-06T07:23:51.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.234+0000 7fdb1dffb640 1 -- 172.21.15.83:0/2587396383 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb2805dd80 con 0x56453d7b7160 2024-03-06T07:23:51.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.234+0000 7fdb1dffb640 1 -- 172.21.15.83:0/2587396383 <== mon.0 v2:172.21.15.83:3300/0 2 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdb28057f60 con 0x56453d7b7160 2024-03-06T07:23:51.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.234+0000 7fdb4820f740 1 -- 172.21.15.83:0/2587396383 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdb28056e10 con 0x56453d7b7160 2024-03-06T07:23:51.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.234+0000 7fdb1dffb640 1 -- 172.21.15.83:0/2587396383 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdb28076050 con 0x56453d7b7160 2024-03-06T07:23:51.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.234+0000 7fdb4820f740 10 client.5862 fetch_fsmap learned FSMap version 19 2024-03-06T07:23:51.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.234+0000 7fdb4820f740 1 -- 172.21.15.83:0/2587396383 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) v3 -- 0x56453d3ed2d0 con 0x56453d7b7160 2024-03-06T07:23:51.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.234+0000 7fdb1dffb640 1 -- 172.21.15.83:0/2587396383 <== mon.0 v2:172.21.15.83:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb28056e10 con 0x56453d7b7160 2024-03-06T07:23:51.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.234+0000 7fdb1dffb640 1 -- 172.21.15.83:0/2587396383 <== mon.0 v2:172.21.15.83:3300/0 5 ==== fsmap(e 19) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fdb28088ce0 con 0x56453d7b7160 2024-03-06T07:23:51.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.234+0000 7fdb4820f740 10 client.5862 fetch_fsmap finished waiting for FSMap version 19 2024-03-06T07:23:51.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.234+0000 7fdb4820f740 10 client.5862 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:23:51.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.234+0000 7fdb4820f740 20 client.5862 populate_metadata read hostname 'smithi083' 2024-03-06T07:23:51.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.234+0000 7fdb4820f740 1 --2- 172.21.15.83:0/2587396383 >> [v2:172.21.15.83:6834/534553538,v1:172.21.15.83:6835/534553538] conn(0x56453d7b5710 0x56453d7b5af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.234+0000 7fdb4820f740 4 client.5862 mds_command: new command op to 5706 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:23:51.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.234+0000 7fdb4820f740 1 -- 172.21.15.83:0/2587396383 --> [v2:172.21.15.83:6834/534553538,v1:172.21.15.83:6835/534553538] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56453d6e5bf0 con 0x56453d7b5710 2024-03-06T07:23:51.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.234+0000 7fdb3effd640 1 --2- 172.21.15.83:0/2587396383 >> [v2:172.21.15.83:6834/534553538,v1:172.21.15.83:6835/534553538] conn(0x56453d7b5710 0x56453d7b5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.234+0000 7fdb1dffb640 10 client.5862 ms_handle_connect on v2:172.21.15.83:6834/534553538 2024-03-06T07:23:51.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.235+0000 7fdb3effd640 1 --2- 172.21.15.83:0/2587396383 >> [v2:172.21.15.83:6834/534553538,v1:172.21.15.83:6835/534553538] conn(0x56453d7b5710 0x56453d7b5af0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ded7f2df6bb8082d server_cookie=4873ac5aa2a37293 in_seq=0 out_seq=0 2024-03-06T07:23:51.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.235+0000 7fdb1dffb640 1 -- 172.21.15.83:0/2587396383 <== mds.0 v2:172.21.15.83:6834/534553538 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56453d6e5bf0 con 0x56453d7b5710 2024-03-06T07:23:51.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.235+0000 7fdb1dffb640 10 client.5862 handle_command_reply: tid=0 2024-03-06T07:23:51.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.235+0000 7fdb4820f740 1 client.5862 shutdown 2024-03-06T07:23:51.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.235+0000 7fdb4820f740 1 -- 172.21.15.83:0/2587396383 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56453d7b7160 msgr2=0x56453d7a1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.235+0000 7fdb4820f740 1 --2- 172.21.15.83:0/2587396383 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56453d7b7160 0x56453d7a1920 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fdb28061190 tx=0x7fdb28015040 comp rx=0 tx=0).stop 2024-03-06T07:23:51.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.235+0000 7fdb4820f740 1 -- 172.21.15.83:0/2587396383 shutdown_connections 2024-03-06T07:23:51.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.235+0000 7fdb4820f740 1 --2- 172.21.15.83:0/2587396383 >> [v2:172.21.15.83:6834/534553538,v1:172.21.15.83:6835/534553538] conn(0x56453d7b5710 0x56453d7b5af0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.235+0000 7fdb4820f740 1 --2- 172.21.15.83:0/2587396383 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56453d7b7540 0x56453d7a1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.235+0000 7fdb4820f740 1 --2- 172.21.15.83:0/2587396383 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56453d7b7160 0x56453d7a1920 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.235+0000 7fdb4820f740 1 --2- 172.21.15.83:0/2587396383 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56453d79e600 0x56453d798710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.235+0000 7fdb4820f740 1 -- 172.21.15.83:0/2587396383 >> 172.21.15.83:0/2587396383 conn(0x56453d76e870 msgr2=0x56453d76d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:51.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.235+0000 7fdb1dffb640 0 client.5862 ms_handle_reset on v2:172.21.15.83:6834/534553538 2024-03-06T07:23:51.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.235+0000 7fdb4820f740 1 -- 172.21.15.83:0/2587396383 shutdown_connections 2024-03-06T07:23:51.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.235+0000 7fdb4820f740 1 -- 172.21.15.83:0/2587396383 wait complete. 2024-03-06T07:23:51.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.236+0000 7fdb4820f740 20 client.5862 trim_cache size 0 max 16384 2024-03-06T07:23:51.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.252+0000 7fdb4820f740 1 Processor -- start 2024-03-06T07:23:51.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.253+0000 7fdb4820f740 1 -- start start 2024-03-06T07:23:51.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.253+0000 7fdb4820f740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56453d79e600 0x56453d79ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.253+0000 7fdb4820f740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56453d7b7160 0x56453d7b5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.253+0000 7fdb4820f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56453d7b7540 0x56453d79fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.253+0000 7fdb4820f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb401ffd10 con 0x56453d7b7540 2024-03-06T07:23:51.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.253+0000 7fdb4820f740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb401ff240 con 0x56453d7b7160 2024-03-06T07:23:51.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.253+0000 7fdb4820f740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb401ff710 con 0x56453d79e600 2024-03-06T07:23:51.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.253+0000 7fdb3f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56453d7b7540 0x56453d79fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.253+0000 7fdb3f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56453d7b7540 0x56453d79fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:44722/0 (socket says 172.21.15.83:44722) 2024-03-06T07:23:51.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.253+0000 7fdb3f7fe640 1 -- 172.21.15.83:0/2864187726 learned_addr learned my addr 172.21.15.83:0/2864187726 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:51.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.254+0000 7fdb3effd640 1 --2- 172.21.15.83:0/2864187726 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56453d79e600 0x56453d79ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.254+0000 7fdb367fc640 1 --2- 172.21.15.83:0/2864187726 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56453d7b7160 0x56453d7b5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.254+0000 7fdb3f7fe640 1 -- 172.21.15.83:0/2864187726 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56453d79e600 msgr2=0x56453d79ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:51.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.254+0000 7fdb3f7fe640 1 --2- 172.21.15.83:0/2864187726 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56453d79e600 0x56453d79ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.254+0000 7fdb3f7fe640 1 -- 172.21.15.83:0/2864187726 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56453d7b7160 msgr2=0x56453d7b5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:51.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.254+0000 7fdb3f7fe640 1 --2- 172.21.15.83:0/2864187726 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56453d7b7160 0x56453d7b5dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.254+0000 7fdb3f7fe640 1 -- 172.21.15.83:0/2864187726 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56453d799410 con 0x56453d7b7540 2024-03-06T07:23:51.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.254+0000 7fdb3f7fe640 1 --2- 172.21.15.83:0/2864187726 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56453d7b7540 0x56453d79fc60 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fdb38007100 tx=0x7fdb3802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c4535fcb6a28132 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:51.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.254+0000 7fdb1ffff640 1 -- 172.21.15.83:0/2864187726 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb38039040 con 0x56453d7b7540 2024-03-06T07:23:51.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.254+0000 7fdb1ffff640 1 -- 172.21.15.83:0/2864187726 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb3802cc60 con 0x56453d7b7540 2024-03-06T07:23:51.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.254+0000 7fdb1ffff640 1 -- 172.21.15.83:0/2864187726 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb38003bb0 con 0x56453d7b7540 2024-03-06T07:23:51.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.254+0000 7fdb4820f740 1 -- 172.21.15.83:0/2864187726 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56453d7b7540 msgr2=0x56453d79fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.254+0000 7fdb4820f740 1 --2- 172.21.15.83:0/2864187726 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56453d7b7540 0x56453d79fc60 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fdb38007100 tx=0x7fdb3802e070 comp rx=0 tx=0).stop 2024-03-06T07:23:51.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.254+0000 7fdb4820f740 1 -- 172.21.15.83:0/2864187726 shutdown_connections 2024-03-06T07:23:51.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.254+0000 7fdb4820f740 1 --2- 172.21.15.83:0/2864187726 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56453d7b7540 0x56453d79fc60 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.254+0000 7fdb4820f740 1 --2- 172.21.15.83:0/2864187726 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56453d7b7160 0x56453d7b5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.255+0000 7fdb4820f740 1 --2- 172.21.15.83:0/2864187726 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56453d79e600 0x56453d79ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.255+0000 7fdb4820f740 1 -- 172.21.15.83:0/2864187726 >> 172.21.15.83:0/2864187726 conn(0x56453d76e870 msgr2=0x56453d79b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:51.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.255+0000 7fdb4820f740 1 -- 172.21.15.83:0/2864187726 shutdown_connections 2024-03-06T07:23:51.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.255+0000 7fdb4820f740 1 -- 172.21.15.83:0/2864187726 wait complete. 2024-03-06T07:23:51.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.255+0000 7fdb4820f740 1 -- start start 2024-03-06T07:23:51.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.255+0000 7fdb4820f740 1 Processor -- start 2024-03-06T07:23:51.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.256+0000 7fdb4820f740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56453d79e600 0x56453d7b4370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.256+0000 7fdb4820f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56453d7b7160 0x56453d7731b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.256+0000 7fdb4820f740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56453d7b7540 0x56453d79fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.256+0000 7fdb4820f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x56453d772d50 con 0x56453d7b7160 2024-03-06T07:23:51.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.256+0000 7fdb4820f740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x56453d773030 con 0x56453d79e600 2024-03-06T07:23:51.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.256+0000 7fdb4820f740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x56453d694ea0 con 0x56453d7b7540 2024-03-06T07:23:51.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.256+0000 7fdb4820f740 10 client.0 _ll_register_callbacks cb 0x7fdb28058470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:23:51.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.256+0000 7fdb367fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56453d7b7160 0x56453d7731b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.256+0000 7fdb367fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56453d7b7160 0x56453d7731b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:44738/0 (socket says 172.21.15.83:44738) 2024-03-06T07:23:51.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.256+0000 7fdb367fc640 1 -- 172.21.15.83:0/423504651 learned_addr learned my addr 172.21.15.83:0/423504651 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:51.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.256+0000 7fdb3f7fe640 1 --2- 172.21.15.83:0/423504651 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56453d7b7540 0x56453d79fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.256+0000 7fdb3effd640 1 --2- 172.21.15.83:0/423504651 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56453d79e600 0x56453d7b4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.256+0000 7fdb367fc640 1 -- 172.21.15.83:0/423504651 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56453d7b7540 msgr2=0x56453d79fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:51.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.256+0000 7fdb367fc640 1 --2- 172.21.15.83:0/423504651 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56453d7b7540 0x56453d79fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.256+0000 7fdb367fc640 1 -- 172.21.15.83:0/423504651 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56453d79e600 msgr2=0x56453d7b4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:51.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.256+0000 7fdb367fc640 1 --2- 172.21.15.83:0/423504651 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56453d79e600 0x56453d7b4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.256+0000 7fdb367fc640 1 -- 172.21.15.83:0/423504651 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56453d7b6700 con 0x56453d7b7160 2024-03-06T07:23:51.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.256+0000 7fdb367fc640 1 --2- 172.21.15.83:0/423504651 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56453d7b7160 0x56453d7731b0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fdb28061160 tx=0x7fdb28054ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7555f5b36795a0fd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:51.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.256+0000 7fdb1dffb640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-06T07:23:51.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.256+0000 7fdb1dffb640 1 -- 172.21.15.83:0/423504651 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb2807c020 con 0x56453d7b7160 2024-03-06T07:23:51.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.257+0000 7fdb1dffb640 1 -- 172.21.15.83:0/423504651 <== mon.0 v2:172.21.15.83:3300/0 2 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdb28054860 con 0x56453d7b7160 2024-03-06T07:23:51.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.257+0000 7fdb4820f740 1 -- 172.21.15.83:0/423504651 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdb3802cc60 con 0x56453d7b7160 2024-03-06T07:23:51.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.257+0000 7fdb1dffb640 1 -- 172.21.15.83:0/423504651 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdb280588e0 con 0x56453d7b7160 2024-03-06T07:23:51.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.257+0000 7fdb4820f740 10 client.5868 fetch_fsmap learned FSMap version 19 2024-03-06T07:23:51.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.257+0000 7fdb4820f740 1 -- 172.21.15.83:0/423504651 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) v3 -- 0x56453d4b76b0 con 0x56453d7b7160 2024-03-06T07:23:51.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.257+0000 7fdb1dffb640 1 -- 172.21.15.83:0/423504651 <== mon.0 v2:172.21.15.83:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb3802cc60 con 0x56453d7b7160 2024-03-06T07:23:51.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.257+0000 7fdb1dffb640 1 -- 172.21.15.83:0/423504651 <== mon.0 v2:172.21.15.83:3300/0 5 ==== fsmap(e 19) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fdb28088ce0 con 0x56453d7b7160 2024-03-06T07:23:51.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.257+0000 7fdb4820f740 10 client.5868 fetch_fsmap finished waiting for FSMap version 19 2024-03-06T07:23:51.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.257+0000 7fdb4820f740 10 client.5868 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:23:51.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.257+0000 7fdb4820f740 20 client.5868 populate_metadata read hostname 'smithi083' 2024-03-06T07:23:51.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.257+0000 7fdb4820f740 1 --2- 172.21.15.83:0/423504651 >> [v2:172.21.15.83:6834/534553538,v1:172.21.15.83:6835/534553538] conn(0x56453d79d470 0x56453d7a0110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.257+0000 7fdb4820f740 4 client.5868 mds_command: new command op to 5706 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:23:51.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.257+0000 7fdb4820f740 1 -- 172.21.15.83:0/423504651 --> [v2:172.21.15.83:6834/534553538,v1:172.21.15.83:6835/534553538] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56453d6dff40 con 0x56453d79d470 2024-03-06T07:23:51.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.257+0000 7fdb3effd640 1 --2- 172.21.15.83:0/423504651 >> [v2:172.21.15.83:6834/534553538,v1:172.21.15.83:6835/534553538] conn(0x56453d79d470 0x56453d7a0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.257+0000 7fdb1dffb640 10 client.5868 ms_handle_connect on v2:172.21.15.83:6834/534553538 2024-03-06T07:23:51.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.258+0000 7fdb3effd640 1 --2- 172.21.15.83:0/423504651 >> [v2:172.21.15.83:6834/534553538,v1:172.21.15.83:6835/534553538] conn(0x56453d79d470 0x56453d7a0110 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ddb283a2eef9ea03 server_cookie=7eb978e10a7a1feb in_seq=0 out_seq=0 2024-03-06T07:23:51.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.258+0000 7fdb1dffb640 1 -- 172.21.15.83:0/423504651 <== mds.0 v2:172.21.15.83:6834/534553538 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x56453d6dff40 con 0x56453d79d470 2024-03-06T07:23:51.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.258+0000 7fdb1dffb640 10 client.5868 handle_command_reply: tid=0 2024-03-06T07:23:51.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.258+0000 7fdb4820f740 1 client.5868 shutdown 2024-03-06T07:23:51.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.258+0000 7fdb4820f740 1 -- 172.21.15.83:0/423504651 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56453d7b7160 msgr2=0x56453d7731b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.258+0000 7fdb4820f740 1 --2- 172.21.15.83:0/423504651 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56453d7b7160 0x56453d7731b0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fdb28061160 tx=0x7fdb28054ee0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.258+0000 7fdb4820f740 1 -- 172.21.15.83:0/423504651 shutdown_connections 2024-03-06T07:23:51.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.258+0000 7fdb4820f740 1 --2- 172.21.15.83:0/423504651 >> [v2:172.21.15.83:6834/534553538,v1:172.21.15.83:6835/534553538] conn(0x56453d79d470 0x56453d7a0110 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.258+0000 7fdb4820f740 1 --2- 172.21.15.83:0/423504651 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56453d7b7540 0x56453d79fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.258+0000 7fdb1dffb640 0 client.5868 ms_handle_reset on v2:172.21.15.83:6834/534553538 2024-03-06T07:23:51.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.258+0000 7fdb4820f740 1 --2- 172.21.15.83:0/423504651 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56453d7b7160 0x56453d7731b0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.258+0000 7fdb4820f740 1 --2- 172.21.15.83:0/423504651 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56453d79e600 0x56453d7b4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.258+0000 7fdb4820f740 1 -- 172.21.15.83:0/423504651 >> 172.21.15.83:0/423504651 conn(0x56453d76e870 msgr2=0x56453d774fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:51.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.258+0000 7fdb4820f740 1 -- 172.21.15.83:0/423504651 shutdown_connections 2024-03-06T07:23:51.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.258+0000 7fdb4820f740 1 -- 172.21.15.83:0/423504651 wait complete. 2024-03-06T07:23:51.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.258+0000 7fdb4820f740 20 client.5868 trim_cache size 0 max 16384 2024-03-06T07:23:51.266 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:23:51.266 INFO:teuthology.orchestra.run.smithi083.stdout: "cluster_fsid": "039877dd-d2ce-45d2-8e77-ac5eb7e1d55a", 2024-03-06T07:23:51.267 INFO:teuthology.orchestra.run.smithi083.stdout: "whoami": 0, 2024-03-06T07:23:51.267 INFO:teuthology.orchestra.run.smithi083.stdout: "id": 5706, 2024-03-06T07:23:51.267 INFO:teuthology.orchestra.run.smithi083.stdout: "want_state": "up:active", 2024-03-06T07:23:51.267 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "up:active", 2024-03-06T07:23:51.267 INFO:teuthology.orchestra.run.smithi083.stdout: "fs_name": "cephfs", 2024-03-06T07:23:51.267 INFO:teuthology.orchestra.run.smithi083.stdout: "rank_uptime": 9.1761346580000005, 2024-03-06T07:23:51.267 INFO:teuthology.orchestra.run.smithi083.stdout: "mdsmap_epoch": 19, 2024-03-06T07:23:51.267 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_epoch": 45, 2024-03-06T07:23:51.267 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_epoch_barrier": 45, 2024-03-06T07:23:51.267 INFO:teuthology.orchestra.run.smithi083.stdout: "uptime": 14.250830957 2024-03-06T07:23:51.267 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:23:51.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.259+0000 7fdb45555640 1 -- 172.21.15.83:0/1456894698 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdb2006b500 msgr2=0x7fdb2006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.259+0000 7fdb45555640 1 --2- 172.21.15.83:0/1456894698 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdb2006b500 0x7fdb2006d9c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fdb30009ab0 tx=0x7fdb30008040 comp rx=0 tx=0).stop 2024-03-06T07:23:51.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.259+0000 7fdb45555640 1 -- 172.21.15.83:0/1456894698 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb4009f9e0 msgr2=0x7fdb401f6eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.259+0000 7fdb45555640 1 --2- 172.21.15.83:0/1456894698 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb4009f9e0 0x7fdb401f6eb0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fdb280072c0 tx=0x7fdb28007780 comp rx=0 tx=0).stop 2024-03-06T07:23:51.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.259+0000 7fdb45555640 1 -- 172.21.15.83:0/1456894698 shutdown_connections 2024-03-06T07:23:51.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.259+0000 7fdb45555640 1 --2- 172.21.15.83:0/1456894698 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdb2006b500 0x7fdb2006d9c0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.259+0000 7fdb45555640 1 --2- 172.21.15.83:0/1456894698 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdb401f73f0 0x7fdb401f2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.259+0000 7fdb45555640 1 --2- 172.21.15.83:0/1456894698 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb4009f9e0 0x7fdb401f6eb0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.259+0000 7fdb45555640 1 --2- 172.21.15.83:0/1456894698 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdb40160910 0x7fdb40160cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.259+0000 7fdb45555640 1 -- 172.21.15.83:0/1456894698 >> 172.21.15.83:0/1456894698 conn(0x7fdb400d3cf0 msgr2=0x7fdb400c8270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:51.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.259+0000 7fdb45555640 1 -- 172.21.15.83:0/1456894698 shutdown_connections 2024-03-06T07:23:51.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.259+0000 7fdb45555640 1 -- 172.21.15.83:0/1456894698 wait complete. 2024-03-06T07:23:51.271 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:23:51.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.371+0000 7ffb257a0640 1 Processor -- start 2024-03-06T07:23:51.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.371+0000 7ffb257a0640 1 -- start start 2024-03-06T07:23:51.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.372+0000 7ffb257a0640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ffb2015dd30 0x7ffb200b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.372+0000 7ffb257a0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ffb2006afb0 0x7ffb2006b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.372+0000 7ffb257a0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffb2006f710 0x7ffb2006fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.372+0000 7ffb257a0640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb200b4360 con 0x7ffb2006afb0 2024-03-06T07:23:51.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.372+0000 7ffb257a0640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb2006d670 con 0x7ffb2015dd30 2024-03-06T07:23:51.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.372+0000 7ffb257a0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb2006d950 con 0x7ffb2006f710 2024-03-06T07:23:51.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.372+0000 7ffb1f7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ffb2006afb0 0x7ffb2006b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.372+0000 7ffb1f7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ffb2006afb0 0x7ffb2006b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:44754/0 (socket says 172.21.15.83:44754) 2024-03-06T07:23:51.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.372+0000 7ffb1f7fe640 1 -- 172.21.15.83:0/1334865712 learned_addr learned my addr 172.21.15.83:0/1334865712 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:51.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.372+0000 7ffb24f9f640 1 --2- 172.21.15.83:0/1334865712 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffb2006f710 0x7ffb2006fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.372+0000 7ffb1ffff640 1 --2- 172.21.15.83:0/1334865712 >> v2:172.21.15.183:3300/0 conn(0x7ffb2015dd30 0x7ffb200b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.372+0000 7ffb1f7fe640 1 -- 172.21.15.83:0/1334865712 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffb2006f710 msgr2=0x7ffb2006fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:51.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.372+0000 7ffb1f7fe640 1 --2- 172.21.15.83:0/1334865712 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffb2006f710 0x7ffb2006fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.372+0000 7ffb1f7fe640 1 -- 172.21.15.83:0/1334865712 >> v2:172.21.15.183:3300/0 conn(0x7ffb2015dd30 msgr2=0x7ffb200b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:51.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.372+0000 7ffb1f7fe640 1 --2- 172.21.15.83:0/1334865712 >> v2:172.21.15.183:3300/0 conn(0x7ffb2015dd30 0x7ffb200b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.372+0000 7ffb1f7fe640 1 -- 172.21.15.83:0/1334865712 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb2006dc30 con 0x7ffb2006afb0 2024-03-06T07:23:51.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.372+0000 7ffb1ffff640 1 --2- 172.21.15.83:0/1334865712 >> v2:172.21.15.183:3300/0 conn(0x7ffb2015dd30 0x7ffb200b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:51.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.373+0000 7ffb1f7fe640 1 --2- 172.21.15.83:0/1334865712 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb2006afb0 0x7ffb2006b440 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7ffb14009870 tx=0x7ffb1402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7778ac65c91bca47 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:51.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.373+0000 7ffb1effd640 1 -- 172.21.15.83:0/1334865712 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb14035020 con 0x7ffb2006afb0 2024-03-06T07:23:51.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.373+0000 7ffb1effd640 1 -- 172.21.15.83:0/1334865712 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb14031640 con 0x7ffb2006afb0 2024-03-06T07:23:51.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.373+0000 7ffb1effd640 1 -- 172.21.15.83:0/1334865712 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb1402d040 con 0x7ffb2006afb0 2024-03-06T07:23:51.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.374+0000 7ffb257a0640 1 -- 172.21.15.83:0/1334865712 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb2006afb0 msgr2=0x7ffb2006b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.374+0000 7ffb257a0640 1 --2- 172.21.15.83:0/1334865712 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb2006afb0 0x7ffb2006b440 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7ffb14009870 tx=0x7ffb1402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.374+0000 7ffb257a0640 1 -- 172.21.15.83:0/1334865712 shutdown_connections 2024-03-06T07:23:51.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.374+0000 7ffb257a0640 1 --2- 172.21.15.83:0/1334865712 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffb2006f710 0x7ffb2006fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.374+0000 7ffb257a0640 1 --2- 172.21.15.83:0/1334865712 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb2006afb0 0x7ffb2006b440 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.374+0000 7ffb257a0640 1 --2- 172.21.15.83:0/1334865712 >> v2:172.21.15.183:3300/0 conn(0x7ffb2015dd30 0x7ffb200b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.374+0000 7ffb257a0640 1 -- 172.21.15.83:0/1334865712 >> 172.21.15.83:0/1334865712 conn(0x7ffb200d0860 msgr2=0x7ffb200d0c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:51.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.374+0000 7ffb257a0640 1 -- 172.21.15.83:0/1334865712 shutdown_connections 2024-03-06T07:23:51.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.374+0000 7ffb257a0640 1 -- 172.21.15.83:0/1334865712 wait complete. 2024-03-06T07:23:51.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.375+0000 7ffb257a0640 1 Processor -- start 2024-03-06T07:23:51.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.375+0000 7ffb257a0640 1 -- start start 2024-03-06T07:23:51.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.376+0000 7ffb257a0640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffb2006afb0 0x7ffb20141470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.376+0000 7ffb257a0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb2006f710 0x7ffb2013b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.376+0000 7ffb257a0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffb2015dd30 0x7ffb201e6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.376+0000 7ffb257a0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb200700e0 con 0x7ffb2006f710 2024-03-06T07:23:51.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.376+0000 7ffb257a0640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb2006e1f0 con 0x7ffb2006afb0 2024-03-06T07:23:51.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.376+0000 7ffb257a0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb2006e710 con 0x7ffb2015dd30 2024-03-06T07:23:51.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.376+0000 7ffb1f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb2006f710 0x7ffb2013b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.376+0000 7ffb1f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb2006f710 0x7ffb2013b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:44758/0 (socket says 172.21.15.83:44758) 2024-03-06T07:23:51.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.376+0000 7ffb1f7fe640 1 -- 172.21.15.83:0/292157565 learned_addr learned my addr 172.21.15.83:0/292157565 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:51.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.376+0000 7ffb1f7fe640 1 -- 172.21.15.83:0/292157565 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffb2015dd30 msgr2=0x7ffb201e6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.376+0000 7ffb1f7fe640 1 --2- 172.21.15.83:0/292157565 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffb2015dd30 0x7ffb201e6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.376+0000 7ffb1f7fe640 1 -- 172.21.15.83:0/292157565 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffb2006afb0 msgr2=0x7ffb20141470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.376+0000 7ffb1f7fe640 1 --2- 172.21.15.83:0/292157565 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffb2006afb0 0x7ffb20141470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.376+0000 7ffb1f7fe640 1 -- 172.21.15.83:0/292157565 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb201e7280 con 0x7ffb2006f710 2024-03-06T07:23:51.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.377+0000 7ffb1f7fe640 1 --2- 172.21.15.83:0/292157565 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb2006f710 0x7ffb2013b2c0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7ffb14000c00 tx=0x7ffb140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:51.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.377+0000 7ffb1cff9640 1 -- 172.21.15.83:0/292157565 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb14040070 con 0x7ffb2006f710 2024-03-06T07:23:51.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.377+0000 7ffb1cff9640 1 -- 172.21.15.83:0/292157565 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb140399f0 con 0x7ffb2006f710 2024-03-06T07:23:51.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.377+0000 7ffb257a0640 1 -- 172.21.15.83:0/292157565 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb20140990 con 0x7ffb2006f710 2024-03-06T07:23:51.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.377+0000 7ffb1cff9640 1 -- 172.21.15.83:0/292157565 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb14035040 con 0x7ffb2006f710 2024-03-06T07:23:51.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.377+0000 7ffb257a0640 1 -- 172.21.15.83:0/292157565 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb20140f50 con 0x7ffb2006f710 2024-03-06T07:23:51.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.378+0000 7ffb257a0640 1 -- 172.21.15.83:0/292157565 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb20068f80 con 0x7ffb2006f710 2024-03-06T07:23:51.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.378+0000 7ffb1cff9640 1 -- 172.21.15.83:0/292157565 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffb1404b060 con 0x7ffb2006f710 2024-03-06T07:23:51.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.378+0000 7ffb1cff9640 1 --2- 172.21.15.83:0/292157565 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffb0c06b4d0 0x7ffb0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.379+0000 7ffb1cff9640 1 -- 172.21.15.83:0/292157565 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffb140abad0 con 0x7ffb2006f710 2024-03-06T07:23:51.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.379+0000 7ffb1ffff640 1 --2- 172.21.15.83:0/292157565 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffb0c06b4d0 0x7ffb0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.379+0000 7ffb1ffff640 1 --2- 172.21.15.83:0/292157565 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffb0c06b4d0 0x7ffb0c06d990 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ffb10006b10 tx=0x7ffb10008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:51.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.383+0000 7ffb1cff9640 1 -- 172.21.15.83:0/292157565 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffb1407a9e0 con 0x7ffb2006f710 2024-03-06T07:23:51.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.597+0000 7ffb257a0640 1 -- 172.21.15.83:0/292157565 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ffb201614b0 con 0x7ffb2006f710 2024-03-06T07:23:51.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.597+0000 7ffb1cff9640 1 -- 172.21.15.83:0/292157565 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v19) v1 ==== 53+0+73 (secure 0 0 0) 0x7ffb1407a2e0 con 0x7ffb2006f710 2024-03-06T07:23:51.599 INFO:teuthology.orchestra.run.smithi083.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:23:51.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.599+0000 7ffb257a0640 1 -- 172.21.15.83:0/292157565 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffb0c06b4d0 msgr2=0x7ffb0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.599+0000 7ffb257a0640 1 --2- 172.21.15.83:0/292157565 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffb0c06b4d0 0x7ffb0c06d990 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ffb10006b10 tx=0x7ffb10008040 comp rx=0 tx=0).stop 2024-03-06T07:23:51.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.599+0000 7ffb257a0640 1 -- 172.21.15.83:0/292157565 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb2006f710 msgr2=0x7ffb2013b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.599+0000 7ffb257a0640 1 --2- 172.21.15.83:0/292157565 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb2006f710 0x7ffb2013b2c0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7ffb14000c00 tx=0x7ffb140046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.599+0000 7ffb257a0640 1 -- 172.21.15.83:0/292157565 shutdown_connections 2024-03-06T07:23:51.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.599+0000 7ffb257a0640 1 --2- 172.21.15.83:0/292157565 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffb0c06b4d0 0x7ffb0c06d990 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.599+0000 7ffb257a0640 1 --2- 172.21.15.83:0/292157565 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffb2015dd30 0x7ffb201e6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.599+0000 7ffb257a0640 1 --2- 172.21.15.83:0/292157565 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb2006f710 0x7ffb2013b2c0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.599+0000 7ffb257a0640 1 --2- 172.21.15.83:0/292157565 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffb2006afb0 0x7ffb20141470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.599+0000 7ffb257a0640 1 -- 172.21.15.83:0/292157565 >> 172.21.15.83:0/292157565 conn(0x7ffb200d0860 msgr2=0x7ffb200ae270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:51.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.599+0000 7ffb257a0640 1 -- 172.21.15.83:0/292157565 shutdown_connections 2024-03-06T07:23:51.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:51.600+0000 7ffb257a0640 1 -- 172.21.15.83:0/292157565 wait complete. 2024-03-06T07:23:51.613 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:23:51.613 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:23:51.613 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi083.front.sepia.ceph.com 2024-03-06T07:23:51.613 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:23:51.613 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:51.613 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:23:51.613 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:23:51.613 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:23:51.613 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:23:51.613 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:51.613 DEBUG:teuthology.orchestra.run.smithi083:> ip addr 2024-03-06T07:23:51.628 INFO:teuthology.orchestra.run.smithi083.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:23:51.628 INFO:teuthology.orchestra.run.smithi083.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:23:51.628 INFO:teuthology.orchestra.run.smithi083.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:23:51.628 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:51.628 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 ::1/128 scope host 2024-03-06T07:23:51.628 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:51.628 INFO:teuthology.orchestra.run.smithi083.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:23:51.628 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:d8:da:1e brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:51.628 INFO:teuthology.orchestra.run.smithi083.stdout: altname enp7s0f0 2024-03-06T07:23:51.629 INFO:teuthology.orchestra.run.smithi083.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:23:51.629 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:d8:da:1f brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:51.629 INFO:teuthology.orchestra.run.smithi083.stdout: altname enp7s0f1 2024-03-06T07:23:51.629 INFO:teuthology.orchestra.run.smithi083.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:23:51.629 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:8f:ce:90 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:51.629 INFO:teuthology.orchestra.run.smithi083.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:23:51.629 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:8f:ce:91 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:51.629 INFO:teuthology.orchestra.run.smithi083.stdout: inet 172.21.15.83/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:23:51.629 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft 42624sec preferred_lft 42624sec 2024-03-06T07:23:51.629 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::ec4:7aff:fe8f:ce91/64 scope link 2024-03-06T07:23:51.629 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:51.629 INFO:teuthology.orchestra.run.smithi083.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:23:51.629 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 1e:29:46:c9:e7:c8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:51.629 INFO:teuthology.orchestra.run.smithi083.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:23:51.630 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:51.630 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::60c5:d7ff:fe70:2b1c/64 scope link 2024-03-06T07:23:51.630 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:51.630 INFO:teuthology.orchestra.run.smithi083.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:23:51.630 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 1e:29:46:c9:e7:c8 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:51.630 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::1c29:46ff:fec9:e7c8/64 scope link 2024-03-06T07:23:51.630 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:51.630 INFO:teuthology.orchestra.run.smithi083.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:23:51.630 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 3e:00:2f:23:d5:45 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:51.630 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::3c00:2fff:fe23:d545/64 scope link 2024-03-06T07:23:51.630 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:51.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:51.631 DEBUG:teuthology.orchestra.run.smithi083:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:23:51.693 INFO:teuthology.orchestra.run.smithi083.stdout:1 2024-03-06T07:23:51.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:51.695 DEBUG:teuthology.orchestra.run.smithi083:> route 2024-03-06T07:23:51.751 INFO:teuthology.orchestra.run.smithi083.stdout:Kernel IP routing table 2024-03-06T07:23:51.751 INFO:teuthology.orchestra.run.smithi083.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:23:51.751 INFO:teuthology.orchestra.run.smithi083.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:23:51.751 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:23:51.751 INFO:teuthology.orchestra.run.smithi083.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:23:51.751 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:51.752 DEBUG:teuthology.orchestra.run.smithi083:> (cd / && exec bash -c ' 2024-03-06T07:23:51.752 DEBUG:teuthology.orchestra.run.smithi083:> set -e 2024-03-06T07:23:51.752 DEBUG:teuthology.orchestra.run.smithi083:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:23:51.752 DEBUG:teuthology.orchestra.run.smithi083:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:23:51.752 DEBUG:teuthology.orchestra.run.smithi083:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:23:51.752 DEBUG:teuthology.orchestra.run.smithi083:> ') 2024-03-06T07:23:51.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:51.846 DEBUG:teuthology.orchestra.run.smithi083:> ip netns list 2024-03-06T07:23:51.861 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:23:51.861 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:23:51.862 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:23:51.862 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:51.862 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip link set brx.0 up 2024-03-06T07:23:51.929 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:23:51.929 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:23:51.929 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:51.983 INFO:teuthology.orchestra.run.smithi083.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:23:51.983 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:23:51.984 DEBUG:teuthology.orchestra.run.smithi083:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:52.038 DEBUG:teuthology.orchestra.run.smithi083:> sudo modprobe fuse 2024-03-06T07:23:52.106 DEBUG:teuthology.orchestra.run.smithi083:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:23:52.161 INFO:teuthology.orchestra.run.smithi083.stdout:/proc 2024-03-06T07:23:52.161 INFO:teuthology.orchestra.run.smithi083.stdout:/sys 2024-03-06T07:23:52.161 INFO:teuthology.orchestra.run.smithi083.stdout:/dev 2024-03-06T07:23:52.161 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/security 2024-03-06T07:23:52.161 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/shm 2024-03-06T07:23:52.161 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/pts 2024-03-06T07:23:52.161 INFO:teuthology.orchestra.run.smithi083.stdout:/run 2024-03-06T07:23:52.162 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/cgroup 2024-03-06T07:23:52.162 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/pstore 2024-03-06T07:23:52.162 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/bpf 2024-03-06T07:23:52.162 INFO:teuthology.orchestra.run.smithi083.stdout:/ 2024-03-06T07:23:52.162 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/selinux 2024-03-06T07:23:52.163 INFO:teuthology.orchestra.run.smithi083.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:23:52.163 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/mqueue 2024-03-06T07:23:52.163 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/hugepages 2024-03-06T07:23:52.163 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/debug 2024-03-06T07:23:52.163 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/tracing 2024-03-06T07:23:52.163 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:23:52.163 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/fuse/connections 2024-03-06T07:23:52.163 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/config 2024-03-06T07:23:52.163 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:23:52.163 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:23:52.163 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:23:52.164 INFO:teuthology.orchestra.run.smithi083.stdout:/run/user/1000 2024-03-06T07:23:52.164 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph 2024-03-06T07:23:52.164 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:23:52.164 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:23:52.164 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:23:52.164 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:23:52.164 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns 2024-03-06T07:23:52.164 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:52.164 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:52.164 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:52.164 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:52.165 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:52.165 DEBUG:teuthology.orchestra.run.smithi083:> ls /sys/fs/fuse/connections 2024-03-06T07:23:52.216 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:23:52.216 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:23:52.259 DEBUG:teuthology.orchestra.run.smithi083:> sudo modprobe fuse 2024-03-06T07:23:52.287 DEBUG:teuthology.orchestra.run.smithi083:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:23:52.343 INFO:teuthology.orchestra.run.smithi083.stdout:/proc 2024-03-06T07:23:52.343 INFO:teuthology.orchestra.run.smithi083.stdout:/sys 2024-03-06T07:23:52.343 INFO:teuthology.orchestra.run.smithi083.stdout:/dev 2024-03-06T07:23:52.343 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/security 2024-03-06T07:23:52.343 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/shm 2024-03-06T07:23:52.343 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/pts 2024-03-06T07:23:52.343 INFO:teuthology.orchestra.run.smithi083.stdout:/run 2024-03-06T07:23:52.344 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/cgroup 2024-03-06T07:23:52.344 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/pstore 2024-03-06T07:23:52.344 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/bpf 2024-03-06T07:23:52.344 INFO:teuthology.orchestra.run.smithi083.stdout:/ 2024-03-06T07:23:52.344 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/selinux 2024-03-06T07:23:52.344 INFO:teuthology.orchestra.run.smithi083.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:23:52.344 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/mqueue 2024-03-06T07:23:52.344 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/hugepages 2024-03-06T07:23:52.344 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/debug 2024-03-06T07:23:52.344 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/tracing 2024-03-06T07:23:52.344 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:23:52.344 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/fuse/connections 2024-03-06T07:23:52.344 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/config 2024-03-06T07:23:52.344 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:23:52.345 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:23:52.345 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:23:52.345 INFO:teuthology.orchestra.run.smithi083.stdout:/run/user/1000 2024-03-06T07:23:52.345 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph 2024-03-06T07:23:52.345 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:23:52.345 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:23:52.345 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:23:52.345 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:23:52.345 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns 2024-03-06T07:23:52.345 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:52.345 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:52.345 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:52.345 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:52.346 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:52.346 DEBUG:teuthology.orchestra.run.smithi083:> ls /sys/fs/fuse/connections 2024-03-06T07:23:52.346 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:2024-03-06T07:23:52.342+0000 7fbfdde3c480 -1 init, newargv = 0x55bb34559600 newargc=15 2024-03-06T07:23:52.347 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:ceph-fuse[56225]: starting ceph client 2024-03-06T07:23:52.349 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:ceph-fuse[56225]: starting fuse 2024-03-06T07:23:52.361 INFO:teuthology.orchestra.run.smithi083.stdout:41 2024-03-06T07:23:52.361 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-06T07:23:52.361 DEBUG:teuthology.orchestra.run.smithi083:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:23:52.361 DEBUG:teuthology.orchestra.run.smithi083:> import glob 2024-03-06T07:23:52.361 DEBUG:teuthology.orchestra.run.smithi083:> import re 2024-03-06T07:23:52.362 DEBUG:teuthology.orchestra.run.smithi083:> import os 2024-03-06T07:23:52.362 DEBUG:teuthology.orchestra.run.smithi083:> import subprocess 2024-03-06T07:23:52.362 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:23:52.362 DEBUG:teuthology.orchestra.run.smithi083:> def _find_admin_socket(client_name): 2024-03-06T07:23:52.362 DEBUG:teuthology.orchestra.run.smithi083:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:23:52.362 DEBUG:teuthology.orchestra.run.smithi083:> files = glob.glob(asok_path) 2024-03-06T07:23:52.362 DEBUG:teuthology.orchestra.run.smithi083:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:23:52.362 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:23:52.362 DEBUG:teuthology.orchestra.run.smithi083:> # Given a non-glob path, it better be there 2024-03-06T07:23:52.362 DEBUG:teuthology.orchestra.run.smithi083:> if "*" not in asok_path: 2024-03-06T07:23:52.362 DEBUG:teuthology.orchestra.run.smithi083:> assert(len(files) == 1) 2024-03-06T07:23:52.362 DEBUG:teuthology.orchestra.run.smithi083:> return files[0] 2024-03-06T07:23:52.362 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:23:52.362 DEBUG:teuthology.orchestra.run.smithi083:> for f in files: 2024-03-06T07:23:52.363 DEBUG:teuthology.orchestra.run.smithi083:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:23:52.363 DEBUG:teuthology.orchestra.run.smithi083:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:23:52.363 DEBUG:teuthology.orchestra.run.smithi083:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:23:52.363 DEBUG:teuthology.orchestra.run.smithi083:> contents = proc_f.read() 2024-03-06T07:23:52.363 DEBUG:teuthology.orchestra.run.smithi083:> if mountpoint in contents: 2024-03-06T07:23:52.363 DEBUG:teuthology.orchestra.run.smithi083:> return f 2024-03-06T07:23:52.363 DEBUG:teuthology.orchestra.run.smithi083:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:23:52.363 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:23:52.363 DEBUG:teuthology.orchestra.run.smithi083:> print(_find_admin_socket("client.0")) 2024-03-06T07:23:52.363 DEBUG:teuthology.orchestra.run.smithi083:> ' 2024-03-06T07:23:52.481 INFO:teuthology.orchestra.run.smithi083.stdout:/var/run/ceph/ceph-client.0.56225.asok 2024-03-06T07:23:52.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:23:52.492 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.56225.asok 2024-03-06T07:23:52.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:52.492 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.56225.asok status 2024-03-06T07:23:52.629 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:23:52.629 INFO:teuthology.orchestra.run.smithi083.stdout: "metadata": { 2024-03-06T07:23:52.629 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:23:52.629 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:23:52.630 INFO:teuthology.orchestra.run.smithi083.stdout: "entity_id": "0", 2024-03-06T07:23:52.630 INFO:teuthology.orchestra.run.smithi083.stdout: "hostname": "smithi083", 2024-03-06T07:23:52.630 INFO:teuthology.orchestra.run.smithi083.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:23:52.630 INFO:teuthology.orchestra.run.smithi083.stdout: "pid": "56225", 2024-03-06T07:23:52.630 INFO:teuthology.orchestra.run.smithi083.stdout: "root": "/" 2024-03-06T07:23:52.630 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:23:52.630 INFO:teuthology.orchestra.run.smithi083.stdout: "dentry_count": 0, 2024-03-06T07:23:52.630 INFO:teuthology.orchestra.run.smithi083.stdout: "dentry_pinned_count": 0, 2024-03-06T07:23:52.630 INFO:teuthology.orchestra.run.smithi083.stdout: "id": 5880, 2024-03-06T07:23:52.630 INFO:teuthology.orchestra.run.smithi083.stdout: "inst": { 2024-03-06T07:23:52.630 INFO:teuthology.orchestra.run.smithi083.stdout: "name": { 2024-03-06T07:23:52.630 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "client", 2024-03-06T07:23:52.630 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 5880 2024-03-06T07:23:52.630 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:23:52.631 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": { 2024-03-06T07:23:52.631 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-03-06T07:23:52.631 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:23:52.631 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 2000630073 2024-03-06T07:23:52.631 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:23:52.631 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:23:52.631 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": { 2024-03-06T07:23:52.631 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-03-06T07:23:52.631 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:23:52.631 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 2000630073 2024-03-06T07:23:52.631 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:23:52.631 INFO:teuthology.orchestra.run.smithi083.stdout: "inst_str": "client.5880 192.168.0.1:0/2000630073", 2024-03-06T07:23:52.631 INFO:teuthology.orchestra.run.smithi083.stdout: "addr_str": "192.168.0.1:0/2000630073", 2024-03-06T07:23:52.631 INFO:teuthology.orchestra.run.smithi083.stdout: "inode_count": 1, 2024-03-06T07:23:52.632 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_epoch": 19, 2024-03-06T07:23:52.632 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_epoch": 45, 2024-03-06T07:23:52.632 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:23:52.632 INFO:teuthology.orchestra.run.smithi083.stdout: "blocklisted": false, 2024-03-06T07:23:52.632 INFO:teuthology.orchestra.run.smithi083.stdout: "fs_name": "cephfs" 2024-03-06T07:23:52.632 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:23:52.637 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:52.638 DEBUG:teuthology.orchestra.run.smithi083:> stat --file-system '--printf=%T 2024-03-06T07:23:52.638 DEBUG:teuthology.orchestra.run.smithi083:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:52.652 INFO:teuthology.orchestra.run.smithi083.stdout:fuseblk 2024-03-06T07:23:52.652 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:52.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:52.653 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:52.720 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_fragment_size_max 50 2024-03-06T07:23:52.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.863+0000 7f46c36df640 1 Processor -- start 2024-03-06T07:23:52.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.863+0000 7f46c36df640 1 -- start start 2024-03-06T07:23:52.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.864+0000 7f46c36df640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f46bc0d4860 0x7f46bc0d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.864+0000 7f46c36df640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46bc1618b0 con 0x7f46bc1614d0 2024-03-06T07:23:52.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.864+0000 7f46c36df640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46bc0cf400 con 0x7f46bc0b0e30 2024-03-06T07:23:52.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.864+0000 7f46c36df640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46bc0d4cf0 con 0x7f46bc0d4860 2024-03-06T07:23:52.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.864+0000 7f46c147b640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f46bc1614d0 0x7f46bc0d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:53974/0 (socket says 172.21.15.83:53974) 2024-03-06T07:23:52.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.864+0000 7f46c147b640 1 -- 172.21.15.83:0/4160193734 learned_addr learned my addr 172.21.15.83:0/4160193734 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:23:52.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.865+0000 7f46c0c7a640 1 --2- 172.21.15.83:0/4160193734 >> v2:172.21.15.183:3300/0 conn(0x7f46bc0d4860 0x7f46bc0d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.865+0000 7f46b3fff640 1 -- 172.21.15.83:0/4160193734 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 183613608 0 0) 0x7f46bc1618b0 con 0x7f46bc1614d0 2024-03-06T07:23:52.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.865+0000 7f46b3fff640 1 -- 172.21.15.83:0/4160193734 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46a8003520 con 0x7f46bc1614d0 2024-03-06T07:23:52.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.865+0000 7f46b3fff640 1 -- 172.21.15.83:0/4160193734 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1159237800 0 0) 0x7f46a8003520 con 0x7f46bc1614d0 2024-03-06T07:23:52.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.865+0000 7f46b3fff640 1 -- 172.21.15.83:0/4160193734 >> v2:172.21.15.183:3300/0 conn(0x7f46bc0d4860 msgr2=0x7f46bc0d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:52.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.865+0000 7f46b3fff640 1 --2- 172.21.15.83:0/4160193734 >> v2:172.21.15.183:3300/0 conn(0x7f46bc0d4860 0x7f46bc0d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.865+0000 7f46b3fff640 1 -- 172.21.15.83:0/4160193734 >> v1:172.21.15.183:6789/0 conn(0x7f46bc0b0e30 legacy=0x7f46bc0b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.865+0000 7f46b3fff640 1 -- 172.21.15.83:0/4160193734 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46bc0cf6a0 con 0x7f46bc1614d0 2024-03-06T07:23:52.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.865+0000 7f46b3fff640 1 -- 172.21.15.83:0/4160193734 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f46b8001690 con 0x7f46bc1614d0 2024-03-06T07:23:52.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.865+0000 7f46b3fff640 1 -- 172.21.15.83:0/4160193734 >> v1:172.21.15.83:6789/0 conn(0x7f46bc1614d0 legacy=0x7f46bc0d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.865+0000 7f46b3fff640 1 --2- 172.21.15.83:0/4160193734 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46a80039b0 0x7f46a8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.865+0000 7f46b3fff640 1 --2- 172.21.15.83:0/4160193734 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46a80043b0 0x7f46a8004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.865+0000 7f46b3fff640 1 --2- 172.21.15.83:0/4160193734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46a8005080 0x7f46a800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.865+0000 7f46b3fff640 1 -- 172.21.15.83:0/4160193734 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46bc0d4cf0 con 0x7f46a8005080 2024-03-06T07:23:52.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.865+0000 7f46b3fff640 1 -- 172.21.15.83:0/4160193734 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a800bfe0 con 0x7f46a80043b0 2024-03-06T07:23:52.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.865+0000 7f46b3fff640 1 -- 172.21.15.83:0/4160193734 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46a800c1e0 con 0x7f46a80039b0 2024-03-06T07:23:52.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.865+0000 7f46c0c7a640 1 --2- 172.21.15.83:0/4160193734 >> v2:172.21.15.183:3300/0 conn(0x7f46bc0d4860 0x7f46bc0d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:52.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.865+0000 7f46c147b640 1 --2- 172.21.15.83:0/4160193734 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46a80039b0 0x7f46a8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.865+0000 7f46c1c7c640 1 --2- 172.21.15.83:0/4160193734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46a8005080 0x7f46a800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.865+0000 7f46c0c7a640 1 --2- 172.21.15.83:0/4160193734 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46a80043b0 0x7f46a8004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.865+0000 7f46c1c7c640 1 -- 172.21.15.83:0/4160193734 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46a80039b0 msgr2=0x7f46a8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:52.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.865+0000 7f46c1c7c640 1 --2- 172.21.15.83:0/4160193734 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46a80039b0 0x7f46a8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.865+0000 7f46c1c7c640 1 -- 172.21.15.83:0/4160193734 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46a80043b0 msgr2=0x7f46a8004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:52.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.865+0000 7f46c1c7c640 1 --2- 172.21.15.83:0/4160193734 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46a80043b0 0x7f46a8004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.865+0000 7f46c1c7c640 1 -- 172.21.15.83:0/4160193734 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a8003520 con 0x7f46a8005080 2024-03-06T07:23:52.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.866+0000 7f46c1c7c640 1 -- 172.21.15.83:0/4160193734 reap_dead start 2024-03-06T07:23:52.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.866+0000 7f46c1c7c640 1 --2- 172.21.15.83:0/4160193734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46a8005080 0x7f46a800b9c0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f46b4002010 tx=0x7f46a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=596c6374503abf80 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:52.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.866+0000 7f46b3fff640 1 -- 172.21.15.83:0/4160193734 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b4001be0 con 0x7f46a8005080 2024-03-06T07:23:52.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.866+0000 7f46b3fff640 1 -- 172.21.15.83:0/4160193734 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46b4003b70 con 0x7f46a8005080 2024-03-06T07:23:52.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.866+0000 7f46b3fff640 1 -- 172.21.15.83:0/4160193734 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b4005110 con 0x7f46a8005080 2024-03-06T07:23:52.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.867+0000 7f46c36df640 1 -- 172.21.15.83:0/4160193734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46a8005080 msgr2=0x7f46a800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.867+0000 7f46c36df640 1 --2- 172.21.15.83:0/4160193734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46a8005080 0x7f46a800b9c0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f46b4002010 tx=0x7f46a8005830 comp rx=0 tx=0).stop 2024-03-06T07:23:52.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.867+0000 7f46c36df640 1 -- 172.21.15.83:0/4160193734 shutdown_connections 2024-03-06T07:23:52.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.867+0000 7f46c36df640 1 --2- 172.21.15.83:0/4160193734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46a8005080 0x7f46a800b9c0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.867+0000 7f46c36df640 1 -- 172.21.15.83:0/4160193734 >> 172.21.15.83:0/4160193734 conn(0x7f46bc0d3d40 msgr2=0x7f46bc0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:52.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.867+0000 7f46c36df640 1 -- 172.21.15.83:0/4160193734 shutdown_connections 2024-03-06T07:23:52.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.867+0000 7f46c36df640 1 -- 172.21.15.83:0/4160193734 wait complete. 2024-03-06T07:23:52.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.868+0000 7f46c36df640 1 Processor -- start 2024-03-06T07:23:52.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.868+0000 7f46c36df640 1 -- start start 2024-03-06T07:23:52.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.869+0000 7f46c36df640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46a8005080 0x7f46bc1f35d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.869+0000 7f46c36df640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46bc0b61f0 0x7f46bc0cbf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.869+0000 7f46c36df640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46bc15d610 0x7f46bc15da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.869+0000 7f46c36df640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46bc0b0910 con 0x7f46bc0b61f0 2024-03-06T07:23:52.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.869+0000 7f46c36df640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46bc0cfc60 con 0x7f46bc15d610 2024-03-06T07:23:52.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.869+0000 7f46c36df640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46bc0d1910 con 0x7f46a8005080 2024-03-06T07:23:52.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.869+0000 7f46c0c7a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46bc0b61f0 0x7f46bc0cbf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.869+0000 7f46c0c7a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46bc0b61f0 0x7f46bc0cbf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:44772/0 (socket says 172.21.15.83:44772) 2024-03-06T07:23:52.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.869+0000 7f46c0c7a640 1 -- 172.21.15.83:0/1611486919 learned_addr learned my addr 172.21.15.83:0/1611486919 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:23:52.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.869+0000 7f46c147b640 1 --2- 172.21.15.83:0/1611486919 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46a8005080 0x7f46bc1f35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.869+0000 7f46c1c7c640 1 --2- 172.21.15.83:0/1611486919 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46bc15d610 0x7f46bc15da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.869+0000 7f46c0c7a640 1 -- 172.21.15.83:0/1611486919 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46a8005080 msgr2=0x7f46bc1f35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.869+0000 7f46c0c7a640 1 --2- 172.21.15.83:0/1611486919 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46a8005080 0x7f46bc1f35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.869+0000 7f46c0c7a640 1 -- 172.21.15.83:0/1611486919 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46bc15d610 msgr2=0x7f46bc15da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.869+0000 7f46c0c7a640 1 --2- 172.21.15.83:0/1611486919 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46bc15d610 0x7f46bc15da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.869+0000 7f46c0c7a640 1 -- 172.21.15.83:0/1611486919 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46bc0cc8b0 con 0x7f46bc0b61f0 2024-03-06T07:23:52.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.869+0000 7f46c147b640 1 --2- 172.21.15.83:0/1611486919 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46a8005080 0x7f46bc1f35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:52.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.869+0000 7f46c1c7c640 1 --2- 172.21.15.83:0/1611486919 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46bc15d610 0x7f46bc15da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:52.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.869+0000 7f46c0c7a640 1 --2- 172.21.15.83:0/1611486919 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46bc0b61f0 0x7f46bc0cbf40 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f46ac011af0 tx=0x7f46ac00f240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:52.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.869+0000 7f46b1ffb640 1 -- 172.21.15.83:0/1611486919 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46ac00fc80 con 0x7f46bc0b61f0 2024-03-06T07:23:52.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.869+0000 7f46b1ffb640 1 -- 172.21.15.83:0/1611486919 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46ac00fe20 con 0x7f46bc0b61f0 2024-03-06T07:23:52.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.869+0000 7f46c36df640 1 -- 172.21.15.83:0/1611486919 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46bc0ccb90 con 0x7f46bc0b61f0 2024-03-06T07:23:52.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.869+0000 7f46b1ffb640 1 -- 172.21.15.83:0/1611486919 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46ac004260 con 0x7f46bc0b61f0 2024-03-06T07:23:52.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.870+0000 7f46c36df640 1 -- 172.21.15.83:0/1611486919 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46bc0cd150 con 0x7f46bc0b61f0 2024-03-06T07:23:52.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.871+0000 7f46c36df640 1 -- 172.21.15.83:0/1611486919 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46bc068ff0 con 0x7f46bc0b61f0 2024-03-06T07:23:52.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.871+0000 7f46b1ffb640 1 -- 172.21.15.83:0/1611486919 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f46ac012040 con 0x7f46bc0b61f0 2024-03-06T07:23:52.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.871+0000 7f46b1ffb640 1 --2- 172.21.15.83:0/1611486919 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f46a006b4d0 0x7f46a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.871+0000 7f46b1ffb640 1 -- 172.21.15.83:0/1611486919 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f46ac08a780 con 0x7f46bc0b61f0 2024-03-06T07:23:52.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.872+0000 7f46c147b640 1 --2- 172.21.15.83:0/1611486919 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f46a006b4d0 0x7f46a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.872+0000 7f46c147b640 1 --2- 172.21.15.83:0/1611486919 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f46a006b4d0 0x7f46a006d990 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f46bc161a80 tx=0x7f46b8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:52.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:52.876+0000 7f46b1ffb640 1 -- 172.21.15.83:0/1611486919 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f46ac019030 con 0x7f46bc0b61f0 2024-03-06T07:23:53.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:53.081+0000 7f46c36df640 1 -- 172.21.15.83:0/1611486919 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) v1 -- 0x7f46bc15f620 con 0x7f46bc0b61f0 2024-03-06T07:23:53.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:53.084+0000 7f46b1ffb640 1 -- 172.21.15.83:0/1611486919 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v13)=0 v13) v1 ==== 124+0+0 (secure 0 0 0) 0x7f46ac059600 con 0x7f46bc0b61f0 2024-03-06T07:23:53.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:53.086+0000 7f46c36df640 1 -- 172.21.15.83:0/1611486919 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f46a006b4d0 msgr2=0x7f46a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:53.086+0000 7f46c36df640 1 --2- 172.21.15.83:0/1611486919 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f46a006b4d0 0x7f46a006d990 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f46bc161a80 tx=0x7f46b8009040 comp rx=0 tx=0).stop 2024-03-06T07:23:53.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:53.086+0000 7f46c36df640 1 -- 172.21.15.83:0/1611486919 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46bc0b61f0 msgr2=0x7f46bc0cbf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:53.086+0000 7f46c36df640 1 --2- 172.21.15.83:0/1611486919 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46bc0b61f0 0x7f46bc0cbf40 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f46ac011af0 tx=0x7f46ac00f240 comp rx=0 tx=0).stop 2024-03-06T07:23:53.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:53.086+0000 7f46c36df640 1 -- 172.21.15.83:0/1611486919 shutdown_connections 2024-03-06T07:23:53.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:53.086+0000 7f46c36df640 1 --2- 172.21.15.83:0/1611486919 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f46a006b4d0 0x7f46a006d990 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:53.086+0000 7f46c36df640 1 --2- 172.21.15.83:0/1611486919 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46bc15d610 0x7f46bc15da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:53.086+0000 7f46c36df640 1 --2- 172.21.15.83:0/1611486919 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46bc0b61f0 0x7f46bc0cbf40 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:53.086+0000 7f46c36df640 1 --2- 172.21.15.83:0/1611486919 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46a8005080 0x7f46bc1f35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:53.086+0000 7f46c36df640 1 -- 172.21.15.83:0/1611486919 >> 172.21.15.83:0/1611486919 conn(0x7f46bc0d3d40 msgr2=0x7f46bc06b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:53.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:53.086+0000 7f46c36df640 1 -- 172.21.15.83:0/1611486919 shutdown_connections 2024-03-06T07:23:53.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:23:53.086+0000 7f46c36df640 1 -- 172.21.15.83:0/1611486919 wait complete. 2024-03-06T07:23:58.477 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:23:58.479 DEBUG:teuthology.orchestra.run.smithi183:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:24:03.100 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:24:03.100 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:03.116 DEBUG:teuthology.orchestra.run.smithi083:> stdin-killer -- python3 -c ' 2024-03-06T07:24:03.116 DEBUG:teuthology.orchestra.run.smithi083:> import os 2024-03-06T07:24:03.116 DEBUG:teuthology.orchestra.run.smithi083:> import uuid 2024-03-06T07:24:03.116 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:24:03.116 DEBUG:teuthology.orchestra.run.smithi083:> n = 51 2024-03-06T07:24:03.116 DEBUG:teuthology.orchestra.run.smithi083:> create_hard_links = False 2024-03-06T07:24:03.116 DEBUG:teuthology.orchestra.run.smithi083:> if 0 > 0: 2024-03-06T07:24:03.116 DEBUG:teuthology.orchestra.run.smithi083:> create_hard_links = True 2024-03-06T07:24:03.116 DEBUG:teuthology.orchestra.run.smithi083:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:24:03.116 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:24:03.116 DEBUG:teuthology.orchestra.run.smithi083:> dpath = os.path.dirname(path) 2024-03-06T07:24:03.116 DEBUG:teuthology.orchestra.run.smithi083:> fnameprefix = os.path.basename(path) 2024-03-06T07:24:03.116 DEBUG:teuthology.orchestra.run.smithi083:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:24:03.117 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:24:03.117 DEBUG:teuthology.orchestra.run.smithi083:> try: 2024-03-06T07:24:03.117 DEBUG:teuthology.orchestra.run.smithi083:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:24:03.117 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:24:03.117 DEBUG:teuthology.orchestra.run.smithi083:> for i in range(n): 2024-03-06T07:24:03.117 DEBUG:teuthology.orchestra.run.smithi083:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:24:03.117 DEBUG:teuthology.orchestra.run.smithi083:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:24:03.117 DEBUG:teuthology.orchestra.run.smithi083:> f.write(f"{i}") 2024-03-06T07:24:03.117 DEBUG:teuthology.orchestra.run.smithi083:> if False: 2024-03-06T07:24:03.117 DEBUG:teuthology.orchestra.run.smithi083:> f.flush() 2024-03-06T07:24:03.117 DEBUG:teuthology.orchestra.run.smithi083:> os.fsync(f.fileno()) 2024-03-06T07:24:03.117 DEBUG:teuthology.orchestra.run.smithi083:> if False: 2024-03-06T07:24:03.117 DEBUG:teuthology.orchestra.run.smithi083:> os.unlink(fpath) 2024-03-06T07:24:03.117 DEBUG:teuthology.orchestra.run.smithi083:> if False: 2024-03-06T07:24:03.117 DEBUG:teuthology.orchestra.run.smithi083:> os.fsync(dirfd) 2024-03-06T07:24:03.117 DEBUG:teuthology.orchestra.run.smithi083:> if create_hard_links: 2024-03-06T07:24:03.118 DEBUG:teuthology.orchestra.run.smithi083:> for j in range(0): 2024-03-06T07:24:03.118 DEBUG:teuthology.orchestra.run.smithi083:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:24:03.118 DEBUG:teuthology.orchestra.run.smithi083:> if True: 2024-03-06T07:24:03.118 DEBUG:teuthology.orchestra.run.smithi083:> os.fsync(dirfd) 2024-03-06T07:24:03.118 DEBUG:teuthology.orchestra.run.smithi083:> finally: 2024-03-06T07:24:03.118 DEBUG:teuthology.orchestra.run.smithi083:> os.close(dirfd) 2024-03-06T07:24:03.118 DEBUG:teuthology.orchestra.run.smithi083:> ' 2024-03-06T07:24:03.313 INFO:teuthology.orchestra.run.smithi083.stderr:Traceback (most recent call last): 2024-03-06T07:24:03.313 INFO:teuthology.orchestra.run.smithi083.stderr: File "", line 20, in 2024-03-06T07:24:03.314 INFO:teuthology.orchestra.run.smithi083.stderr:OSError: [Errno 28] No space left on device: '/home/ubuntu/cephtest/mnt.0/subdir/file_50' 2024-03-06T07:24:03.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03 stdin-killer INFO: command exited with status 1: exiting normally with same code! 2024-03-06T07:24:03.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:24:03.323 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:24:03.324 DEBUG:teuthology.orchestra.run.smithi083:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:24:03.406 DEBUG:teuthology.orchestra.run.smithi183:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:24:03.457 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:24:03.457 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:24:03.457 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:03.473 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:03.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.612+0000 7fd1b7116640 1 Processor -- start 2024-03-06T07:24:03.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.613+0000 7fd1b7116640 1 -- start start 2024-03-06T07:24:03.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.613+0000 7fd1b7116640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fd1b00d22b0 0x7fd1b00d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.613+0000 7fd1b7116640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1b00b3430 con 0x7fd1b00b41f0 2024-03-06T07:24:03.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.613+0000 7fd1b7116640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1b00b3710 con 0x7fd1b015dfe0 2024-03-06T07:24:03.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.613+0000 7fd1b7116640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1b00d2c50 con 0x7fd1b00d22b0 2024-03-06T07:24:03.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.613+0000 7fd1b56b3640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fd1b00b41f0 0x7fd1b00b4680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:35396/0 (socket says 172.21.15.83:35396) 2024-03-06T07:24:03.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.613+0000 7fd1b56b3640 1 -- 172.21.15.83:0/2415133585 learned_addr learned my addr 172.21.15.83:0/2415133585 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:24:03.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.614+0000 7fd1affff640 1 --2- 172.21.15.83:0/2415133585 >> v2:172.21.15.183:3300/0 conn(0x7fd1b00d22b0 0x7fd1b00d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.614+0000 7fd1af7fe640 1 -- 172.21.15.83:0/2415133585 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2902992197 0 0) 0x7fd1b00b3430 con 0x7fd1b00b41f0 2024-03-06T07:24:03.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.614+0000 7fd1af7fe640 1 -- 172.21.15.83:0/2415133585 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1a0003520 con 0x7fd1b00b41f0 2024-03-06T07:24:03.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.614+0000 7fd1affff640 1 -- 172.21.15.83:0/2415133585 >> v1:172.21.15.183:6789/0 conn(0x7fd1b015dfe0 legacy=0x7fd1b00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.614+0000 7fd1af7fe640 1 -- 172.21.15.83:0/2415133585 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3696545477 0 0) 0x7fd1a0003520 con 0x7fd1b00b41f0 2024-03-06T07:24:03.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.614+0000 7fd1affff640 1 -- 172.21.15.83:0/2415133585 >> v1:172.21.15.83:6789/0 conn(0x7fd1b00b41f0 legacy=0x7fd1b00b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.614+0000 7fd1affff640 1 -- 172.21.15.83:0/2415133585 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b00b39f0 con 0x7fd1b00d22b0 2024-03-06T07:24:03.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.615+0000 7fd1affff640 1 --2- 172.21.15.83:0/2415133585 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd1b00d22b0 0x7fd1b00d2710 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd19c0097d0 tx=0x7fd19c02b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=316bab62e1d3e99c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:03.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.615+0000 7fd1af7fe640 1 -- 172.21.15.83:0/2415133585 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19c038070 con 0x7fd1b00d22b0 2024-03-06T07:24:03.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.615+0000 7fd1af7fe640 1 -- 172.21.15.83:0/2415133585 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd19c031500 con 0x7fd1b00d22b0 2024-03-06T07:24:03.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.615+0000 7fd1af7fe640 1 -- 172.21.15.83:0/2415133585 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a80019f0 con 0x7fd1b00d22b0 2024-03-06T07:24:03.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.615+0000 7fd1b7116640 1 -- 172.21.15.83:0/2415133585 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd1b00d22b0 msgr2=0x7fd1b00d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.615+0000 7fd1b7116640 1 --2- 172.21.15.83:0/2415133585 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd1b00d22b0 0x7fd1b00d2710 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd19c0097d0 tx=0x7fd19c02b820 comp rx=0 tx=0).stop 2024-03-06T07:24:03.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.615+0000 7fd1b7116640 1 -- 172.21.15.83:0/2415133585 shutdown_connections 2024-03-06T07:24:03.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.615+0000 7fd1b7116640 1 --2- 172.21.15.83:0/2415133585 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd1b00d22b0 0x7fd1b00d2710 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.615+0000 7fd1b7116640 1 -- 172.21.15.83:0/2415133585 >> 172.21.15.83:0/2415133585 conn(0x7fd1b00d3cd0 msgr2=0x7fd1b00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:03.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.616+0000 7fd1b7116640 1 -- 172.21.15.83:0/2415133585 shutdown_connections 2024-03-06T07:24:03.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.616+0000 7fd1b7116640 1 -- 172.21.15.83:0/2415133585 wait complete. 2024-03-06T07:24:03.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.617+0000 7fd1b7116640 1 Processor -- start 2024-03-06T07:24:03.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.617+0000 7fd1b7116640 1 -- start start 2024-03-06T07:24:03.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.618+0000 7fd1b7116640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd1b00b41f0 0x7fd1b00cc150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.618+0000 7fd1b7116640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1b00d22b0 0x7fd1b01f3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.618+0000 7fd1b7116640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd1b015dfe0 0x7fd1b00cd9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.618+0000 7fd1b7116640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b00b3d60 con 0x7fd1b00d22b0 2024-03-06T07:24:03.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.618+0000 7fd1b7116640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b00b31e0 con 0x7fd1b015dfe0 2024-03-06T07:24:03.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.618+0000 7fd1b7116640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b00d1c70 con 0x7fd1b00b41f0 2024-03-06T07:24:03.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.618+0000 7fd1affff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1b00d22b0 0x7fd1b01f3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.618+0000 7fd1affff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1b00d22b0 0x7fd1b01f3ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47732/0 (socket says 172.21.15.83:47732) 2024-03-06T07:24:03.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.618+0000 7fd1affff640 1 -- 172.21.15.83:0/224065734 learned_addr learned my addr 172.21.15.83:0/224065734 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:03.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.618+0000 7fd1b56b3640 1 --2- 172.21.15.83:0/224065734 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd1b015dfe0 0x7fd1b00cd9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.618+0000 7fd1b4eb2640 1 --2- 172.21.15.83:0/224065734 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd1b00b41f0 0x7fd1b00cc150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.618+0000 7fd1affff640 1 -- 172.21.15.83:0/224065734 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd1b00b41f0 msgr2=0x7fd1b00cc150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.618+0000 7fd1affff640 1 --2- 172.21.15.83:0/224065734 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd1b00b41f0 0x7fd1b00cc150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.618+0000 7fd1affff640 1 -- 172.21.15.83:0/224065734 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd1b015dfe0 msgr2=0x7fd1b00cd9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.618+0000 7fd1affff640 1 --2- 172.21.15.83:0/224065734 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd1b015dfe0 0x7fd1b00cd9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.618+0000 7fd1affff640 1 -- 172.21.15.83:0/224065734 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b015fc20 con 0x7fd1b00d22b0 2024-03-06T07:24:03.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.618+0000 7fd1b4eb2640 1 --2- 172.21.15.83:0/224065734 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd1b00b41f0 0x7fd1b00cc150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:03.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.618+0000 7fd1affff640 1 --2- 172.21.15.83:0/224065734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1b00d22b0 0x7fd1b01f3ea0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fd19c000c00 tx=0x7fd19c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:03.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.618+0000 7fd1ad7fa640 1 -- 172.21.15.83:0/224065734 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19c043070 con 0x7fd1b00d22b0 2024-03-06T07:24:03.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.618+0000 7fd1ad7fa640 1 -- 172.21.15.83:0/224065734 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd19c03f870 con 0x7fd1b00d22b0 2024-03-06T07:24:03.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.618+0000 7fd1b7116640 1 -- 172.21.15.83:0/224065734 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1b015ff00 con 0x7fd1b00d22b0 2024-03-06T07:24:03.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.618+0000 7fd1ad7fa640 1 -- 172.21.15.83:0/224065734 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19c038040 con 0x7fd1b00d22b0 2024-03-06T07:24:03.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.618+0000 7fd1b7116640 1 -- 172.21.15.83:0/224065734 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1b01604c0 con 0x7fd1b00d22b0 2024-03-06T07:24:03.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.619+0000 7fd1b7116640 1 -- 172.21.15.83:0/224065734 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1b004f330 con 0x7fd1b00d22b0 2024-03-06T07:24:03.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.620+0000 7fd1ad7fa640 1 -- 172.21.15.83:0/224065734 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd19c046070 con 0x7fd1b00d22b0 2024-03-06T07:24:03.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.620+0000 7fd1ad7fa640 1 --2- 172.21.15.83:0/224065734 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd19406b4d0 0x7fd19406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.620+0000 7fd1ad7fa640 1 -- 172.21.15.83:0/224065734 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd19c0abc70 con 0x7fd1b00d22b0 2024-03-06T07:24:03.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.620+0000 7fd1b4eb2640 1 --2- 172.21.15.83:0/224065734 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd19406b4d0 0x7fd19406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.621+0000 7fd1b4eb2640 1 --2- 172.21.15.83:0/224065734 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd19406b4d0 0x7fd19406d990 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4007fd0 tx=0x7fd1a4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:03.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.625+0000 7fd1ad7fa640 1 -- 172.21.15.83:0/224065734 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd19c030070 con 0x7fd1b00d22b0 2024-03-06T07:24:03.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.816+0000 7fd1b7116640 1 -- 172.21.15.83:0/224065734 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd1b00b09c0 con 0x7fd1b00d22b0 2024-03-06T07:24:03.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.817+0000 7fd1ad7fa640 1 -- 172.21.15.83:0/224065734 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+16625 (secure 0 0 0) 0x7fd19c07a4b0 con 0x7fd1b00d22b0 2024-03-06T07:24:03.819 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:24:03.819 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":45,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:23:43.092917+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:39.656617+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:40.402765+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:03.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.819+0000 7fd1b7116640 1 -- 172.21.15.83:0/224065734 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd19406b4d0 msgr2=0x7fd19406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.819+0000 7fd1b7116640 1 --2- 172.21.15.83:0/224065734 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd19406b4d0 0x7fd19406d990 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4007fd0 tx=0x7fd1a4009040 comp rx=0 tx=0).stop 2024-03-06T07:24:03.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.819+0000 7fd1b7116640 1 -- 172.21.15.83:0/224065734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1b00d22b0 msgr2=0x7fd1b01f3ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.819+0000 7fd1b7116640 1 --2- 172.21.15.83:0/224065734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1b00d22b0 0x7fd1b01f3ea0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fd19c000c00 tx=0x7fd19c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.819+0000 7fd1b7116640 1 -- 172.21.15.83:0/224065734 shutdown_connections 2024-03-06T07:24:03.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.819+0000 7fd1b7116640 1 --2- 172.21.15.83:0/224065734 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd19406b4d0 0x7fd19406d990 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.819+0000 7fd1b7116640 1 --2- 172.21.15.83:0/224065734 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd1b015dfe0 0x7fd1b00cd9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.819+0000 7fd1b7116640 1 --2- 172.21.15.83:0/224065734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1b00d22b0 0x7fd1b01f3ea0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.819+0000 7fd1b7116640 1 --2- 172.21.15.83:0/224065734 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd1b00b41f0 0x7fd1b00cc150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.819+0000 7fd1b7116640 1 -- 172.21.15.83:0/224065734 >> 172.21.15.83:0/224065734 conn(0x7fd1b00d3cd0 msgr2=0x7fd1b006ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:03.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.819+0000 7fd1b7116640 1 -- 172.21.15.83:0/224065734 shutdown_connections 2024-03-06T07:24:03.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.819+0000 7fd1b7116640 1 -- 172.21.15.83:0/224065734 wait complete. 2024-03-06T07:24:03.831 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:23:39.656617+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:23:40.402765+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:03.832 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:03.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.937+0000 7f7f1478a640 1 Processor -- start 2024-03-06T07:24:03.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.937+0000 7f7f1478a640 1 -- start start 2024-03-06T07:24:03.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.938+0000 7f7f1478a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7f0c0d4860 0x7f7f0c0d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.938+0000 7f7f1478a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7f0c0b0e30 0x7f7f0c0b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.938+0000 7f7f1478a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f0c1618b0 con 0x7f7f0c1614d0 2024-03-06T07:24:03.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.938+0000 7f7f1478a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f0c0d4cf0 con 0x7f7f0c0d4860 2024-03-06T07:24:03.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.938+0000 7f7f1478a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0c0cf400 con 0x7f7f0c0b0e30 2024-03-06T07:24:03.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.938+0000 7f7f11d25640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7f0c0d4860 0x7f7f0c0d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.938+0000 7f7f11d25640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7f0c0d4860 0x7f7f0c0d2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47742/0 (socket says 172.21.15.83:47742) 2024-03-06T07:24:03.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.938+0000 7f7f11d25640 1 -- 172.21.15.83:0/2584898647 learned_addr learned my addr 172.21.15.83:0/2584898647 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:03.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.938+0000 7f7f12d27640 1 --2- 172.21.15.83:0/2584898647 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7f0c0b0e30 0x7f7f0c0b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.938+0000 7f7f11d25640 1 -- 172.21.15.83:0/2584898647 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7f0c0b0e30 msgr2=0x7f7f0c0b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:03.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.938+0000 7f7f11d25640 1 --2- 172.21.15.83:0/2584898647 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7f0c0b0e30 0x7f7f0c0b12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.938+0000 7f7f11d25640 1 -- 172.21.15.83:0/2584898647 >> v1:172.21.15.183:6789/0 conn(0x7f7f0c1614d0 legacy=0x7f7f0c0d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.938+0000 7f7f11d25640 1 -- 172.21.15.83:0/2584898647 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f0c0cf6a0 con 0x7f7f0c0d4860 2024-03-06T07:24:03.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.939+0000 7f7f11d25640 1 --2- 172.21.15.83:0/2584898647 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f0c0d4860 0x7f7f0c0d2020 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f7efc009870 tx=0x7f7efc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc0a75cbb55145a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:03.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.939+0000 7f7f11524640 1 -- 172.21.15.83:0/2584898647 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7efc034070 con 0x7f7f0c0d4860 2024-03-06T07:24:03.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.939+0000 7f7f11524640 1 -- 172.21.15.83:0/2584898647 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7efc030840 con 0x7f7f0c0d4860 2024-03-06T07:24:03.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.939+0000 7f7f11524640 1 -- 172.21.15.83:0/2584898647 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7efc030c10 con 0x7f7f0c0d4860 2024-03-06T07:24:03.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.939+0000 7f7f1478a640 1 -- 172.21.15.83:0/2584898647 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f0c0d4860 msgr2=0x7f7f0c0d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.939+0000 7f7f1478a640 1 --2- 172.21.15.83:0/2584898647 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f0c0d4860 0x7f7f0c0d2020 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f7efc009870 tx=0x7f7efc02b980 comp rx=0 tx=0).stop 2024-03-06T07:24:03.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.940+0000 7f7f1478a640 1 -- 172.21.15.83:0/2584898647 shutdown_connections 2024-03-06T07:24:03.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.940+0000 7f7f1478a640 1 --2- 172.21.15.83:0/2584898647 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7f0c0b0e30 0x7f7f0c0b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.940+0000 7f7f1478a640 1 --2- 172.21.15.83:0/2584898647 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f0c0d4860 0x7f7f0c0d2020 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.940+0000 7f7f1478a640 1 -- 172.21.15.83:0/2584898647 >> 172.21.15.83:0/2584898647 conn(0x7f7f0c0d3d40 msgr2=0x7f7f0c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:03.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.940+0000 7f7f1478a640 1 -- 172.21.15.83:0/2584898647 shutdown_connections 2024-03-06T07:24:03.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.940+0000 7f7f1478a640 1 -- 172.21.15.83:0/2584898647 wait complete. 2024-03-06T07:24:03.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.941+0000 7f7f1478a640 1 Processor -- start 2024-03-06T07:24:03.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.941+0000 7f7f1478a640 1 -- start start 2024-03-06T07:24:03.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.941+0000 7f7f1478a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f0c0b0e30 0x7f7f0c0cd650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.941+0000 7f7f1478a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7f0c0d4860 0x7f7f0c151140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.941+0000 7f7f1478a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7f0c1614d0 0x7f7f0c0cfea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.941+0000 7f7f1478a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0c0b0910 con 0x7f7f0c0b0e30 2024-03-06T07:24:03.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.941+0000 7f7f1478a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0c0cfc60 con 0x7f7f0c1614d0 2024-03-06T07:24:03.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.941+0000 7f7f1478a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0c0d2560 con 0x7f7f0c0d4860 2024-03-06T07:24:03.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.942+0000 7f7f12526640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f0c0b0e30 0x7f7f0c0cd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.942+0000 7f7f12526640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f0c0b0e30 0x7f7f0c0cd650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47748/0 (socket says 172.21.15.83:47748) 2024-03-06T07:24:03.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.942+0000 7f7f12526640 1 -- 172.21.15.83:0/1591552734 learned_addr learned my addr 172.21.15.83:0/1591552734 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:03.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.942+0000 7f7f11d25640 1 --2- 172.21.15.83:0/1591552734 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7f0c0d4860 0x7f7f0c151140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.942+0000 7f7f12d27640 1 --2- 172.21.15.83:0/1591552734 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7f0c1614d0 0x7f7f0c0cfea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.942+0000 7f7f12526640 1 -- 172.21.15.83:0/1591552734 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7f0c0d4860 msgr2=0x7f7f0c151140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.942+0000 7f7f12526640 1 --2- 172.21.15.83:0/1591552734 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7f0c0d4860 0x7f7f0c151140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.942+0000 7f7f12526640 1 -- 172.21.15.83:0/1591552734 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7f0c1614d0 msgr2=0x7f7f0c0cfea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.942+0000 7f7f12526640 1 --2- 172.21.15.83:0/1591552734 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7f0c1614d0 0x7f7f0c0cfea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.942+0000 7f7f12526640 1 -- 172.21.15.83:0/1591552734 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f0c0b9770 con 0x7f7f0c0b0e30 2024-03-06T07:24:03.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.942+0000 7f7f12526640 1 --2- 172.21.15.83:0/1591552734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f0c0b0e30 0x7f7f0c0cd650 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f7f0800c3c0 tx=0x7f7f0800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:03.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.942+0000 7f7f02ffd640 1 -- 172.21.15.83:0/1591552734 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f080052d0 con 0x7f7f0c0b0e30 2024-03-06T07:24:03.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.942+0000 7f7f02ffd640 1 -- 172.21.15.83:0/1591552734 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f08005490 con 0x7f7f0c0b0e30 2024-03-06T07:24:03.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.942+0000 7f7f02ffd640 1 -- 172.21.15.83:0/1591552734 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f08007260 con 0x7f7f0c0b0e30 2024-03-06T07:24:03.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.942+0000 7f7f1478a640 1 -- 172.21.15.83:0/1591552734 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f0c0b8ed0 con 0x7f7f0c0b0e30 2024-03-06T07:24:03.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.942+0000 7f7f1478a640 1 -- 172.21.15.83:0/1591552734 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f0c0b9490 con 0x7f7f0c0b0e30 2024-03-06T07:24:03.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.943+0000 7f7f1478a640 1 -- 172.21.15.83:0/1591552734 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f0c0cb040 con 0x7f7f0c0b0e30 2024-03-06T07:24:03.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.944+0000 7f7f02ffd640 1 -- 172.21.15.83:0/1591552734 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7f08007400 con 0x7f7f0c0b0e30 2024-03-06T07:24:03.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.944+0000 7f7f02ffd640 1 --2- 172.21.15.83:0/1591552734 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7ef006b4d0 0x7f7ef006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.944+0000 7f7f02ffd640 1 -- 172.21.15.83:0/1591552734 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7f080895f0 con 0x7f7f0c0b0e30 2024-03-06T07:24:03.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.944+0000 7f7f11d25640 1 --2- 172.21.15.83:0/1591552734 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7ef006b4d0 0x7f7ef006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.945+0000 7f7f11d25640 1 --2- 172.21.15.83:0/1591552734 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7ef006b4d0 0x7f7ef006d990 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f7efc036000 tx=0x7f7efc008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:03.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:03.948+0000 7f7f02ffd640 1 -- 172.21.15.83:0/1591552734 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7f08057e20 con 0x7f7f0c0b0e30 2024-03-06T07:24:04.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.121+0000 7f7f1478a640 1 -- 172.21.15.83:0/1591552734 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7f0c068ff0 con 0x7f7f0c0b0e30 2024-03-06T07:24:04.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.122+0000 7f7f02ffd640 1 -- 172.21.15.83:0/1591552734 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v45) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7f0805bec0 con 0x7f7f0c0b0e30 2024-03-06T07:24:04.124 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:24:04.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.124+0000 7f7f1478a640 1 -- 172.21.15.83:0/1591552734 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7ef006b4d0 msgr2=0x7f7ef006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.124+0000 7f7f1478a640 1 --2- 172.21.15.83:0/1591552734 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7ef006b4d0 0x7f7ef006d990 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f7efc036000 tx=0x7f7efc008030 comp rx=0 tx=0).stop 2024-03-06T07:24:04.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.124+0000 7f7f1478a640 1 -- 172.21.15.83:0/1591552734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f0c0b0e30 msgr2=0x7f7f0c0cd650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.124+0000 7f7f1478a640 1 --2- 172.21.15.83:0/1591552734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f0c0b0e30 0x7f7f0c0cd650 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f7f0800c3c0 tx=0x7f7f0800c880 comp rx=0 tx=0).stop 2024-03-06T07:24:04.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.124+0000 7f7f1478a640 1 -- 172.21.15.83:0/1591552734 shutdown_connections 2024-03-06T07:24:04.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.124+0000 7f7f1478a640 1 --2- 172.21.15.83:0/1591552734 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7ef006b4d0 0x7f7ef006d990 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.124+0000 7f7f1478a640 1 --2- 172.21.15.83:0/1591552734 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7f0c1614d0 0x7f7f0c0cfea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.124+0000 7f7f1478a640 1 --2- 172.21.15.83:0/1591552734 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7f0c0d4860 0x7f7f0c151140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.124+0000 7f7f1478a640 1 --2- 172.21.15.83:0/1591552734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f0c0b0e30 0x7f7f0c0cd650 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.124+0000 7f7f1478a640 1 -- 172.21.15.83:0/1591552734 >> 172.21.15.83:0/1591552734 conn(0x7f7f0c0d3d40 msgr2=0x7f7f0c0b12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:04.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.124+0000 7f7f1478a640 1 -- 172.21.15.83:0/1591552734 shutdown_connections 2024-03-06T07:24:04.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.124+0000 7f7f1478a640 1 -- 172.21.15.83:0/1591552734 wait complete. 2024-03-06T07:24:04.137 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:24:04.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.238+0000 7f61eb288640 1 Processor -- start 2024-03-06T07:24:04.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.239+0000 7f61eb288640 1 -- start start 2024-03-06T07:24:04.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.239+0000 7f61eb288640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f61e415c050 0x7f61e406f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.239+0000 7f61eb288640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f61e406fb40 0x7f61e40cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.239+0000 7f61eb288640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61e40cd4b0 con 0x7f61e40ccca0 2024-03-06T07:24:04.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.239+0000 7f61eb288640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61e406ffb0 con 0x7f61e406fb40 2024-03-06T07:24:04.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.239+0000 7f61eb288640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61e40cd790 con 0x7f61e415c050 2024-03-06T07:24:04.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.240+0000 7f61e8823640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f61e406fb40 0x7f61e40cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.240+0000 7f61e9825640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f61e40ccca0 0x7f61e40cfae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:35412/0 (socket says 172.21.15.83:35412) 2024-03-06T07:24:04.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.240+0000 7f61e9825640 1 -- 172.21.15.83:0/2955834985 learned_addr learned my addr 172.21.15.83:0/2955834985 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:24:04.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.240+0000 7f61e9024640 1 --2- 172.21.15.83:0/2955834985 >> v2:172.21.15.183:3300/0 conn(0x7f61e415c050 0x7f61e406f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.240+0000 7f61e8823640 1 -- 172.21.15.83:0/2955834985 >> v2:172.21.15.183:3300/0 conn(0x7f61e415c050 msgr2=0x7f61e406f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:04.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.240+0000 7f61e8823640 1 --2- 172.21.15.83:0/2955834985 >> v2:172.21.15.183:3300/0 conn(0x7f61e415c050 0x7f61e406f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.240+0000 7f61e8823640 1 -- 172.21.15.83:0/2955834985 >> v1:172.21.15.83:6789/0 conn(0x7f61e40ccca0 legacy=0x7f61e40cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.240+0000 7f61dbfff640 1 -- 172.21.15.83:0/2955834985 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2978099886 0 0) 0x7f61e40cd4b0 con 0x7f61e40ccca0 2024-03-06T07:24:04.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.240+0000 7f61e8823640 1 -- 172.21.15.83:0/2955834985 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e40cda70 con 0x7f61e406fb40 2024-03-06T07:24:04.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.241+0000 7f61e8823640 1 --2- 172.21.15.83:0/2955834985 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61e406fb40 0x7f61e40cc760 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f61e4070ef0 tx=0x7f61d402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb90bf04a99a4891 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.241+0000 7f61dbfff640 1 -- 172.21.15.83:0/2955834985 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d40315a0 con 0x7f61e406fb40 2024-03-06T07:24:04.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.241+0000 7f61dbfff640 1 -- 172.21.15.83:0/2955834985 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61d4037070 con 0x7f61e406fb40 2024-03-06T07:24:04.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.241+0000 7f61dbfff640 1 -- 172.21.15.83:0/2955834985 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d4031960 con 0x7f61e406fb40 2024-03-06T07:24:04.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.241+0000 7f61eb288640 1 -- 172.21.15.83:0/2955834985 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61e406fb40 msgr2=0x7f61e40cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.241+0000 7f61eb288640 1 --2- 172.21.15.83:0/2955834985 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61e406fb40 0x7f61e40cc760 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f61e4070ef0 tx=0x7f61d402b840 comp rx=0 tx=0).stop 2024-03-06T07:24:04.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.241+0000 7f61eb288640 1 -- 172.21.15.83:0/2955834985 shutdown_connections 2024-03-06T07:24:04.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.241+0000 7f61eb288640 1 --2- 172.21.15.83:0/2955834985 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61e406fb40 0x7f61e40cc760 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.241+0000 7f61eb288640 1 --2- 172.21.15.83:0/2955834985 >> v2:172.21.15.183:3300/0 conn(0x7f61e415c050 0x7f61e406f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.241+0000 7f61eb288640 1 -- 172.21.15.83:0/2955834985 >> 172.21.15.83:0/2955834985 conn(0x7f61e40b3800 msgr2=0x7f61e40b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:04.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.241+0000 7f61eb288640 1 -- 172.21.15.83:0/2955834985 shutdown_connections 2024-03-06T07:24:04.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.241+0000 7f61eb288640 1 -- 172.21.15.83:0/2955834985 wait complete. 2024-03-06T07:24:04.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.243+0000 7f61eb288640 1 Processor -- start 2024-03-06T07:24:04.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.243+0000 7f61eb288640 1 -- start start 2024-03-06T07:24:04.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.243+0000 7f61eb288640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f61e406fb40 0x7f61e40ca5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.243+0000 7f61eb288640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f61e40ccca0 0x7f61e41518f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.243+0000 7f61eb288640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61e415c050 0x7f61e409f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.243+0000 7f61eb288640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e40d01f0 con 0x7f61e415c050 2024-03-06T07:24:04.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.243+0000 7f61eb288640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e40cd130 con 0x7f61e406fb40 2024-03-06T07:24:04.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.243+0000 7f61eb288640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61e40d0370 con 0x7f61e40ccca0 2024-03-06T07:24:04.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.243+0000 7f61e9825640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61e415c050 0x7f61e409f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.243+0000 7f61e9825640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61e415c050 0x7f61e409f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47760/0 (socket says 172.21.15.83:47760) 2024-03-06T07:24:04.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.243+0000 7f61e9825640 1 -- 172.21.15.83:0/4143484324 learned_addr learned my addr 172.21.15.83:0/4143484324 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:04.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.243+0000 7f61e8823640 1 --2- 172.21.15.83:0/4143484324 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f61e40ccca0 0x7f61e41518f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.243+0000 7f61e9024640 1 --2- 172.21.15.83:0/4143484324 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f61e406fb40 0x7f61e40ca5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.244+0000 7f61e9825640 1 -- 172.21.15.83:0/4143484324 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f61e40ccca0 msgr2=0x7f61e41518f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.244+0000 7f61e9825640 1 --2- 172.21.15.83:0/4143484324 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f61e40ccca0 0x7f61e41518f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.244+0000 7f61e9825640 1 -- 172.21.15.83:0/4143484324 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f61e406fb40 msgr2=0x7f61e40ca5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.244+0000 7f61e9825640 1 --2- 172.21.15.83:0/4143484324 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f61e406fb40 0x7f61e40ca5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.244+0000 7f61e9825640 1 -- 172.21.15.83:0/4143484324 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e4151e30 con 0x7f61e415c050 2024-03-06T07:24:04.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.244+0000 7f61e9825640 1 --2- 172.21.15.83:0/4143484324 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61e415c050 0x7f61e409f230 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f61dc00c7a0 tx=0x7f61dc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.244+0000 7f61d9ffb640 1 -- 172.21.15.83:0/4143484324 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61dc018070 con 0x7f61e415c050 2024-03-06T07:24:04.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.244+0000 7f61d9ffb640 1 -- 172.21.15.83:0/4143484324 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61dc00a5e0 con 0x7f61e415c050 2024-03-06T07:24:04.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.244+0000 7f61eb288640 1 -- 172.21.15.83:0/4143484324 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61e40cabf0 con 0x7f61e415c050 2024-03-06T07:24:04.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.244+0000 7f61d9ffb640 1 -- 172.21.15.83:0/4143484324 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61dc00aa50 con 0x7f61e415c050 2024-03-06T07:24:04.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.244+0000 7f61eb288640 1 -- 172.21.15.83:0/4143484324 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61e40cb1b0 con 0x7f61e415c050 2024-03-06T07:24:04.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.245+0000 7f61eb288640 1 -- 172.21.15.83:0/4143484324 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61e409fe60 con 0x7f61e415c050 2024-03-06T07:24:04.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.245+0000 7f61d9ffb640 1 -- 172.21.15.83:0/4143484324 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f61dc01b070 con 0x7f61e415c050 2024-03-06T07:24:04.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.246+0000 7f61d9ffb640 1 --2- 172.21.15.83:0/4143484324 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f61c806b4d0 0x7f61c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.246+0000 7f61e9024640 1 --2- 172.21.15.83:0/4143484324 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f61c806b4d0 0x7f61c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.246+0000 7f61d9ffb640 1 -- 172.21.15.83:0/4143484324 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f61dc089610 con 0x7f61e415c050 2024-03-06T07:24:04.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.246+0000 7f61e9024640 1 --2- 172.21.15.83:0/4143484324 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f61c806b4d0 0x7f61c806d990 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f61e0006fd0 tx=0x7f61e0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.254+0000 7f61d9ffb640 1 -- 172.21.15.83:0/4143484324 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f61e4151e30 con 0x7f61e415c050 2024-03-06T07:24:04.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.427+0000 7f61eb288640 1 -- 172.21.15.83:0/4143484324 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f61e40b10e0 con 0x7f61e415c050 2024-03-06T07:24:04.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.427+0000 7f61d9ffb640 1 -- 172.21.15.83:0/4143484324 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v45) v1 ==== 104+0+11 (secure 0 0 0) 0x7f61dc057eb0 con 0x7f61e415c050 2024-03-06T07:24:04.429 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-03-06T07:24:04.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.429+0000 7f61eb288640 1 -- 172.21.15.83:0/4143484324 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f61c806b4d0 msgr2=0x7f61c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.429+0000 7f61eb288640 1 --2- 172.21.15.83:0/4143484324 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f61c806b4d0 0x7f61c806d990 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f61e0006fd0 tx=0x7f61e0008040 comp rx=0 tx=0).stop 2024-03-06T07:24:04.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.429+0000 7f61eb288640 1 -- 172.21.15.83:0/4143484324 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61e415c050 msgr2=0x7f61e409f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.429+0000 7f61eb288640 1 --2- 172.21.15.83:0/4143484324 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61e415c050 0x7f61e409f230 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f61dc00c7a0 tx=0x7f61dc00cc60 comp rx=0 tx=0).stop 2024-03-06T07:24:04.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.429+0000 7f61eb288640 1 -- 172.21.15.83:0/4143484324 shutdown_connections 2024-03-06T07:24:04.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.429+0000 7f61eb288640 1 --2- 172.21.15.83:0/4143484324 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f61c806b4d0 0x7f61c806d990 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.429+0000 7f61eb288640 1 --2- 172.21.15.83:0/4143484324 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f61e415c050 0x7f61e409f230 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.429+0000 7f61eb288640 1 --2- 172.21.15.83:0/4143484324 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f61e40ccca0 0x7f61e41518f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.429+0000 7f61eb288640 1 --2- 172.21.15.83:0/4143484324 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f61e406fb40 0x7f61e40ca5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.429+0000 7f61eb288640 1 -- 172.21.15.83:0/4143484324 >> 172.21.15.83:0/4143484324 conn(0x7f61e40b3800 msgr2=0x7f61e40aee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:04.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.429+0000 7f61eb288640 1 -- 172.21.15.83:0/4143484324 shutdown_connections 2024-03-06T07:24:04.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.429+0000 7f61eb288640 1 -- 172.21.15.83:0/4143484324 wait complete. 2024-03-06T07:24:04.442 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:24:04.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.547+0000 7f6663df7640 1 Processor -- start 2024-03-06T07:24:04.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.547+0000 7f6663df7640 1 -- start start 2024-03-06T07:24:04.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.549+0000 7f6663df7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f665c1586e0 0x7f665c0aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.549+0000 7f6663df7640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f665c0d01a0 0x7f665c0b33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.549+0000 7f6663df7640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f665c158ac0 con 0x7f665c0aef80 2024-03-06T07:24:04.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.549+0000 7f6663df7640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f665c15b2a0 con 0x7f665c1586e0 2024-03-06T07:24:04.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.549+0000 7f6663df7640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f665c15b540 con 0x7f665c0d01a0 2024-03-06T07:24:04.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.549+0000 7f6661b93640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f665c1586e0 0x7f665c0aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.549+0000 7f6661b93640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f665c1586e0 0x7f665c0aea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47772/0 (socket says 172.21.15.83:47772) 2024-03-06T07:24:04.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.549+0000 7f6661b93640 1 -- 172.21.15.83:0/3230207307 learned_addr learned my addr 172.21.15.83:0/3230207307 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:04.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.549+0000 7f6662394640 1 --2- 172.21.15.83:0/3230207307 >> v2:172.21.15.183:3300/0 conn(0x7f665c0d01a0 0x7f665c0b33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.549+0000 7f6661b93640 1 -- 172.21.15.83:0/3230207307 >> v2:172.21.15.183:3300/0 conn(0x7f665c0d01a0 msgr2=0x7f665c0b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:04.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.549+0000 7f6661b93640 1 --2- 172.21.15.83:0/3230207307 >> v2:172.21.15.183:3300/0 conn(0x7f665c0d01a0 0x7f665c0b33a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.549+0000 7f6661b93640 1 -- 172.21.15.83:0/3230207307 >> v1:172.21.15.183:6789/0 conn(0x7f665c0aef80 legacy=0x7f665c0cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:04.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.549+0000 7f6661b93640 1 -- 172.21.15.83:0/3230207307 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f665c15b820 con 0x7f665c1586e0 2024-03-06T07:24:04.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.550+0000 7f6661b93640 1 --2- 172.21.15.83:0/3230207307 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f665c1586e0 0x7f665c0aea40 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f665c0cc350 tx=0x7f664c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2c84e140ddd81f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.550+0000 7f6660b91640 1 -- 172.21.15.83:0/3230207307 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f664c02e070 con 0x7f665c1586e0 2024-03-06T07:24:04.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.550+0000 7f6660b91640 1 -- 172.21.15.83:0/3230207307 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f664c037070 con 0x7f665c1586e0 2024-03-06T07:24:04.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.550+0000 7f6660b91640 1 -- 172.21.15.83:0/3230207307 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f664c031a00 con 0x7f665c1586e0 2024-03-06T07:24:04.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.551+0000 7f6663df7640 1 -- 172.21.15.83:0/3230207307 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f665c1586e0 msgr2=0x7f665c0aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.551+0000 7f6663df7640 1 --2- 172.21.15.83:0/3230207307 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f665c1586e0 0x7f665c0aea40 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f665c0cc350 tx=0x7f664c02b880 comp rx=0 tx=0).stop 2024-03-06T07:24:04.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.551+0000 7f6663df7640 1 -- 172.21.15.83:0/3230207307 shutdown_connections 2024-03-06T07:24:04.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.551+0000 7f6663df7640 1 --2- 172.21.15.83:0/3230207307 >> v2:172.21.15.183:3300/0 conn(0x7f665c0d01a0 0x7f665c0b33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.551+0000 7f6663df7640 1 --2- 172.21.15.83:0/3230207307 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f665c1586e0 0x7f665c0aea40 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.551+0000 7f6663df7640 1 -- 172.21.15.83:0/3230207307 >> 172.21.15.83:0/3230207307 conn(0x7f665c06b190 msgr2=0x7f665c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:04.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.551+0000 7f6663df7640 1 -- 172.21.15.83:0/3230207307 shutdown_connections 2024-03-06T07:24:04.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.551+0000 7f6663df7640 1 -- 172.21.15.83:0/3230207307 wait complete. 2024-03-06T07:24:04.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.552+0000 7f6663df7640 1 Processor -- start 2024-03-06T07:24:04.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.552+0000 7f6663df7640 1 -- start start 2024-03-06T07:24:04.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.552+0000 7f6663df7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f665c0aef80 0x7f665c06d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.553+0000 7f6663df7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f665c0d01a0 0x7f665c1f1e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.553+0000 7f6663df7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f665c1586e0 0x7f665c1f5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.553+0000 7f6663df7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f665c071350 con 0x7f665c0aef80 2024-03-06T07:24:04.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.553+0000 7f6663df7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f665c15bde0 con 0x7f665c1586e0 2024-03-06T07:24:04.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.553+0000 7f6663df7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f665c0714d0 con 0x7f665c0d01a0 2024-03-06T07:24:04.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.553+0000 7f6661b93640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f665c0aef80 0x7f665c06d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.553+0000 7f6661b93640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f665c0aef80 0x7f665c06d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47782/0 (socket says 172.21.15.83:47782) 2024-03-06T07:24:04.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.553+0000 7f6661b93640 1 -- 172.21.15.83:0/2972141205 learned_addr learned my addr 172.21.15.83:0/2972141205 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:04.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.553+0000 7f6661b93640 1 -- 172.21.15.83:0/2972141205 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f665c0d01a0 msgr2=0x7f665c1f1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.553+0000 7f6661392640 1 --2- 172.21.15.83:0/2972141205 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f665c0d01a0 0x7f665c1f1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.553+0000 7f6662394640 1 --2- 172.21.15.83:0/2972141205 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f665c1586e0 0x7f665c1f5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.553+0000 7f6661b93640 1 --2- 172.21.15.83:0/2972141205 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f665c0d01a0 0x7f665c1f1e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.553+0000 7f6661b93640 1 -- 172.21.15.83:0/2972141205 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f665c1586e0 msgr2=0x7f665c1f5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.553+0000 7f6661b93640 1 --2- 172.21.15.83:0/2972141205 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f665c1586e0 0x7f665c1f5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.553+0000 7f6661b93640 1 -- 172.21.15.83:0/2972141205 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f665c1f2690 con 0x7f665c0aef80 2024-03-06T07:24:04.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.553+0000 7f6661b93640 1 --2- 172.21.15.83:0/2972141205 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f665c0aef80 0x7f665c06d0e0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f664c000c00 tx=0x7f664c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.553+0000 7f66527fc640 1 -- 172.21.15.83:0/2972141205 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f664c032880 con 0x7f665c0aef80 2024-03-06T07:24:04.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.553+0000 7f66527fc640 1 -- 172.21.15.83:0/2972141205 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f664c032a20 con 0x7f665c0aef80 2024-03-06T07:24:04.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.553+0000 7f6663df7640 1 -- 172.21.15.83:0/2972141205 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f665c1f2970 con 0x7f665c0aef80 2024-03-06T07:24:04.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.553+0000 7f66527fc640 1 -- 172.21.15.83:0/2972141205 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f664c032e10 con 0x7f665c0aef80 2024-03-06T07:24:04.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.553+0000 7f6663df7640 1 -- 172.21.15.83:0/2972141205 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f665c1f2c50 con 0x7f665c0aef80 2024-03-06T07:24:04.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.554+0000 7f6663df7640 1 -- 172.21.15.83:0/2972141205 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f665c158220 con 0x7f665c0aef80 2024-03-06T07:24:04.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.555+0000 7f66527fc640 1 -- 172.21.15.83:0/2972141205 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f664c04c060 con 0x7f665c0aef80 2024-03-06T07:24:04.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.555+0000 7f66527fc640 1 --2- 172.21.15.83:0/2972141205 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f664006b4d0 0x7f664006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.555+0000 7f6661392640 1 --2- 172.21.15.83:0/2972141205 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f664006b4d0 0x7f664006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.555+0000 7f66527fc640 1 -- 172.21.15.83:0/2972141205 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f664c0abaf0 con 0x7f665c0aef80 2024-03-06T07:24:04.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.556+0000 7f6661392640 1 --2- 172.21.15.83:0/2972141205 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f664006b4d0 0x7f664006d990 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f665c157450 tx=0x7f6658009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.560+0000 7f66527fc640 1 -- 172.21.15.83:0/2972141205 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f664c040070 con 0x7f665c0aef80 2024-03-06T07:24:04.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.803+0000 7f6663df7640 1 -- 172.21.15.83:0/2972141205 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f665c0732b0 con 0x7f665c0aef80 2024-03-06T07:24:04.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.804+0000 7f66527fc640 1 -- 172.21.15.83:0/2972141205 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v45) v1 ==== 100+0+11 (secure 0 0 0) 0x7f664c07a320 con 0x7f665c0aef80 2024-03-06T07:24:04.806 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 64 2024-03-06T07:24:04.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.806+0000 7f6663df7640 1 -- 172.21.15.83:0/2972141205 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f664006b4d0 msgr2=0x7f664006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.806+0000 7f6663df7640 1 --2- 172.21.15.83:0/2972141205 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f664006b4d0 0x7f664006d990 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f665c157450 tx=0x7f6658009040 comp rx=0 tx=0).stop 2024-03-06T07:24:04.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.806+0000 7f6663df7640 1 -- 172.21.15.83:0/2972141205 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f665c0aef80 msgr2=0x7f665c06d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.806+0000 7f6663df7640 1 --2- 172.21.15.83:0/2972141205 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f665c0aef80 0x7f665c06d0e0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f664c000c00 tx=0x7f664c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.806+0000 7f6663df7640 1 -- 172.21.15.83:0/2972141205 shutdown_connections 2024-03-06T07:24:04.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.806+0000 7f6663df7640 1 --2- 172.21.15.83:0/2972141205 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f664006b4d0 0x7f664006d990 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.806+0000 7f6663df7640 1 --2- 172.21.15.83:0/2972141205 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f665c1586e0 0x7f665c1f5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.806+0000 7f6663df7640 1 --2- 172.21.15.83:0/2972141205 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f665c0d01a0 0x7f665c1f1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.806+0000 7f6663df7640 1 --2- 172.21.15.83:0/2972141205 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f665c0aef80 0x7f665c06d0e0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.806+0000 7f6663df7640 1 -- 172.21.15.83:0/2972141205 >> 172.21.15.83:0/2972141205 conn(0x7f665c06b190 msgr2=0x7f665c159870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:04.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.806+0000 7f6663df7640 1 -- 172.21.15.83:0/2972141205 shutdown_connections 2024-03-06T07:24:04.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.806+0000 7f6663df7640 1 -- 172.21.15.83:0/2972141205 wait complete. 2024-03-06T07:24:04.819 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:24:04.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.918+0000 7fe10a3ac640 1 Processor -- start 2024-03-06T07:24:04.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.918+0000 7fe10a3ac640 1 -- start start 2024-03-06T07:24:04.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.918+0000 7fe10a3ac640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fe1041614a0 0x7fe10406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.918+0000 7fe10a3ac640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1040af4b0 con 0x7fe1040b1a20 2024-03-06T07:24:04.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.918+0000 7fe10a3ac640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1040af790 con 0x7fe1040b0270 2024-03-06T07:24:04.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.918+0000 7fe10a3ac640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1040afa70 con 0x7fe1041614a0 2024-03-06T07:24:04.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.919+0000 7fe108949640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fe1040b1a20 0x7fe1040b5510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:35418/0 (socket says 172.21.15.83:35418) 2024-03-06T07:24:04.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.919+0000 7fe108949640 1 -- 172.21.15.83:0/4030277140 learned_addr learned my addr 172.21.15.83:0/4030277140 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:24:04.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.919+0000 7fe103fff640 1 --2- 172.21.15.83:0/4030277140 >> v2:172.21.15.183:3300/0 conn(0x7fe1041614a0 0x7fe10406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.919+0000 7fe102ffd640 1 -- 172.21.15.83:0/4030277140 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 867317513 0 0) 0x7fe1040af4b0 con 0x7fe1040b1a20 2024-03-06T07:24:04.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.919+0000 7fe102ffd640 1 -- 172.21.15.83:0/4030277140 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0ec003520 con 0x7fe1040b1a20 2024-03-06T07:24:04.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.919+0000 7fe103fff640 1 -- 172.21.15.83:0/4030277140 >> v1:172.21.15.183:6789/0 conn(0x7fe1040b0270 legacy=0x7fe1040b1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.919+0000 7fe103fff640 1 -- 172.21.15.83:0/4030277140 >> v1:172.21.15.83:6789/0 conn(0x7fe1040b1a20 legacy=0x7fe1040b5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.919+0000 7fe103fff640 1 -- 172.21.15.83:0/4030277140 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1040afd50 con 0x7fe1041614a0 2024-03-06T07:24:04.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.920+0000 7fe103fff640 1 --2- 172.21.15.83:0/4030277140 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe1041614a0 0x7fe10406d3c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe0f40097a0 tx=0x7fe0f402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc6b576d04c76eae server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.920+0000 7fe102ffd640 1 -- 172.21.15.83:0/4030277140 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f4035020 con 0x7fe1041614a0 2024-03-06T07:24:04.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.920+0000 7fe102ffd640 1 -- 172.21.15.83:0/4030277140 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0f4031830 con 0x7fe1041614a0 2024-03-06T07:24:04.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.920+0000 7fe102ffd640 1 -- 172.21.15.83:0/4030277140 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f4031be0 con 0x7fe1041614a0 2024-03-06T07:24:04.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.920+0000 7fe10a3ac640 1 -- 172.21.15.83:0/4030277140 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe1041614a0 msgr2=0x7fe10406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.920+0000 7fe10a3ac640 1 --2- 172.21.15.83:0/4030277140 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe1041614a0 0x7fe10406d3c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe0f40097a0 tx=0x7fe0f402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.921+0000 7fe10a3ac640 1 -- 172.21.15.83:0/4030277140 shutdown_connections 2024-03-06T07:24:04.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.921+0000 7fe10a3ac640 1 --2- 172.21.15.83:0/4030277140 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe1041614a0 0x7fe10406d3c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.921+0000 7fe10a3ac640 1 -- 172.21.15.83:0/4030277140 >> 172.21.15.83:0/4030277140 conn(0x7fe10409a9f0 msgr2=0x7fe10409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:04.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.921+0000 7fe10a3ac640 1 -- 172.21.15.83:0/4030277140 shutdown_connections 2024-03-06T07:24:04.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.921+0000 7fe10a3ac640 1 -- 172.21.15.83:0/4030277140 wait complete. 2024-03-06T07:24:04.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.922+0000 7fe10a3ac640 1 Processor -- start 2024-03-06T07:24:04.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.922+0000 7fe10a3ac640 1 -- start start 2024-03-06T07:24:04.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.923+0000 7fe10a3ac640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1040b0270 0x7fe104148810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.923+0000 7fe10a3ac640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe1040b1a20 0x7fe1040b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.923+0000 7fe10a3ac640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe1041614a0 0x7fe1040b2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.923+0000 7fe10a3ac640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1040b1e00 con 0x7fe1040b0270 2024-03-06T07:24:04.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.923+0000 7fe10a3ac640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1040b06e0 con 0x7fe1041614a0 2024-03-06T07:24:04.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.923+0000 7fe10a3ac640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1040b1f80 con 0x7fe1040b1a20 2024-03-06T07:24:04.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.923+0000 7fe103fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1040b0270 0x7fe104148810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.923+0000 7fe103fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1040b0270 0x7fe104148810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47798/0 (socket says 172.21.15.83:47798) 2024-03-06T07:24:04.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.923+0000 7fe103fff640 1 -- 172.21.15.83:0/1370440473 learned_addr learned my addr 172.21.15.83:0/1370440473 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:04.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.923+0000 7fe108949640 1 --2- 172.21.15.83:0/1370440473 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe1041614a0 0x7fe1040b2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.923+0000 7fe1037fe640 1 --2- 172.21.15.83:0/1370440473 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe1040b1a20 0x7fe1040b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.923+0000 7fe103fff640 1 -- 172.21.15.83:0/1370440473 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe1040b1a20 msgr2=0x7fe1040b2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.923+0000 7fe103fff640 1 --2- 172.21.15.83:0/1370440473 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe1040b1a20 0x7fe1040b2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.923+0000 7fe103fff640 1 -- 172.21.15.83:0/1370440473 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe1041614a0 msgr2=0x7fe1040b2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.923+0000 7fe103fff640 1 --2- 172.21.15.83:0/1370440473 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe1041614a0 0x7fe1040b2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.923+0000 7fe103fff640 1 -- 172.21.15.83:0/1370440473 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe10409f190 con 0x7fe1040b0270 2024-03-06T07:24:04.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.923+0000 7fe103fff640 1 --2- 172.21.15.83:0/1370440473 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1040b0270 0x7fe104148810 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fe0f4000c00 tx=0x7fe0f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.923+0000 7fe100ff9640 1 -- 172.21.15.83:0/1370440473 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f4035070 con 0x7fe1040b0270 2024-03-06T07:24:04.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.923+0000 7fe100ff9640 1 -- 172.21.15.83:0/1370440473 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0f4004230 con 0x7fe1040b0270 2024-03-06T07:24:04.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.923+0000 7fe100ff9640 1 -- 172.21.15.83:0/1370440473 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f403a630 con 0x7fe1040b0270 2024-03-06T07:24:04.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.923+0000 7fe10a3ac640 1 -- 172.21.15.83:0/1370440473 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe10409f470 con 0x7fe1040b0270 2024-03-06T07:24:04.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.923+0000 7fe10a3ac640 1 -- 172.21.15.83:0/1370440473 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1040cfa50 con 0x7fe1040b0270 2024-03-06T07:24:04.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.924+0000 7fe10a3ac640 1 -- 172.21.15.83:0/1370440473 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe10404f330 con 0x7fe1040b0270 2024-03-06T07:24:04.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.925+0000 7fe100ff9640 1 -- 172.21.15.83:0/1370440473 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe0f4038020 con 0x7fe1040b0270 2024-03-06T07:24:04.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.925+0000 7fe100ff9640 1 --2- 172.21.15.83:0/1370440473 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe0f006b4d0 0x7fe0f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.925+0000 7fe100ff9640 1 -- 172.21.15.83:0/1370440473 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe0f40abe30 con 0x7fe1040b0270 2024-03-06T07:24:04.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.925+0000 7fe1037fe640 1 --2- 172.21.15.83:0/1370440473 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe0f006b4d0 0x7fe0f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.926+0000 7fe1037fe640 1 --2- 172.21.15.83:0/1370440473 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe0f006b4d0 0x7fe0f006d990 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fe0f8007fd0 tx=0x7fe0f8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:04.930+0000 7fe100ff9640 1 -- 172.21.15.83:0/1370440473 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe0f4040070 con 0x7fe1040b0270 2024-03-06T07:24:05.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.123+0000 7fe10a3ac640 1 -- 172.21.15.83:0/1370440473 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe10406e8e0 con 0x7fe1040b0270 2024-03-06T07:24:05.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.123+0000 7fe100ff9640 1 -- 172.21.15.83:0/1370440473 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v45) v1 ==== 80+0+0 (secure 0 0 0) 0x7fe0f407ad40 con 0x7fe1040b0270 2024-03-06T07:24:05.125 INFO:teuthology.orchestra.run.smithi083.stderr:listed 0 entries 2024-03-06T07:24:05.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.126+0000 7fe10a3ac640 1 -- 172.21.15.83:0/1370440473 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe0f006b4d0 msgr2=0x7fe0f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.126+0000 7fe10a3ac640 1 --2- 172.21.15.83:0/1370440473 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe0f006b4d0 0x7fe0f006d990 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fe0f8007fd0 tx=0x7fe0f8009040 comp rx=0 tx=0).stop 2024-03-06T07:24:05.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.126+0000 7fe10a3ac640 1 -- 172.21.15.83:0/1370440473 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1040b0270 msgr2=0x7fe104148810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.126+0000 7fe10a3ac640 1 --2- 172.21.15.83:0/1370440473 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1040b0270 0x7fe104148810 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fe0f4000c00 tx=0x7fe0f40046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.126+0000 7fe10a3ac640 1 -- 172.21.15.83:0/1370440473 shutdown_connections 2024-03-06T07:24:05.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.126+0000 7fe10a3ac640 1 --2- 172.21.15.83:0/1370440473 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe0f006b4d0 0x7fe0f006d990 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.126+0000 7fe10a3ac640 1 --2- 172.21.15.83:0/1370440473 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe1041614a0 0x7fe1040b2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.126+0000 7fe10a3ac640 1 --2- 172.21.15.83:0/1370440473 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe1040b1a20 0x7fe1040b2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.126+0000 7fe10a3ac640 1 --2- 172.21.15.83:0/1370440473 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1040b0270 0x7fe104148810 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.126+0000 7fe10a3ac640 1 -- 172.21.15.83:0/1370440473 >> 172.21.15.83:0/1370440473 conn(0x7fe10409a9f0 msgr2=0x7fe1040b5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:05.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.126+0000 7fe10a3ac640 1 -- 172.21.15.83:0/1370440473 shutdown_connections 2024-03-06T07:24:05.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.126+0000 7fe10a3ac640 1 -- 172.21.15.83:0/1370440473 wait complete. 2024-03-06T07:24:05.142 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi083.front.sepia.ceph.com... 2024-03-06T07:24:05.142 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:05.142 DEBUG:teuthology.orchestra.run.smithi083:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:05.167 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:ceph-fuse[56225]: fuse finished with error 0 and tester_r 0 2024-03-06T07:24:05.169 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi083.front.sepia.ceph.com 2024-03-06T07:24:05.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:05.170 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:24:05.340 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:24:05.340 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:05.358 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=287 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:24:05.358 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi183.front.sepia.ceph.com 2024-03-06T07:24:05.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:05.359 DEBUG:teuthology.orchestra.run.smithi183:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:24:05.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:24:05.418 INFO:teuthology.orchestra.run.smithi183.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:24:05.418 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:24:05.418 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:05.433 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1257 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:24:05.434 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi083.front.sepia.ceph.com 2024-03-06T07:24:05.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:05.434 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:24:05.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:24:05.488 INFO:teuthology.orchestra.run.smithi083.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:24:05.489 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:24:05.489 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:05.506 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=287 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:24:05.506 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi183.front.sepia.ceph.com 2024-03-06T07:24:05.507 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:05.507 DEBUG:teuthology.orchestra.run.smithi183:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:24:05.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:24:05.565 INFO:teuthology.orchestra.run.smithi183.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:24:05.565 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:05.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.668+0000 7f34e3f5c640 1 Processor -- start 2024-03-06T07:24:05.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.668+0000 7f34e3f5c640 1 -- start start 2024-03-06T07:24:05.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.669+0000 7f34e3f5c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f34dc15dfe0 0x7f34dc0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.669+0000 7f34e3f5c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f34dc0cf830 0x7f34dc0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.669+0000 7f34e3f5c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34dc0b3430 con 0x7f34dc0d27f0 2024-03-06T07:24:05.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.669+0000 7f34e3f5c640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34dc0d2c50 con 0x7f34dc15dfe0 2024-03-06T07:24:05.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.669+0000 7f34e3f5c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34dc0b31e0 con 0x7f34dc0cf830 2024-03-06T07:24:05.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.669+0000 7f34e1cf8640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f34dc15dfe0 0x7f34dc0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.669+0000 7f34e24f9640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f34dc0d27f0 0x7f34dc0d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:35430/0 (socket says 172.21.15.83:35430) 2024-03-06T07:24:05.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.669+0000 7f34e1cf8640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f34dc15dfe0 0x7f34dc0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47814/0 (socket says 172.21.15.83:47814) 2024-03-06T07:24:05.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.669+0000 7f34e24f9640 1 -- 172.21.15.83:0/312722325 learned_addr learned my addr 172.21.15.83:0/312722325 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:24:05.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.669+0000 7f34e14f7640 1 --2- 172.21.15.83:0/312722325 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f34dc0cf830 0x7f34dc0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.669+0000 7f34e1cf8640 1 -- 172.21.15.83:0/312722325 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f34dc0cf830 msgr2=0x7f34dc0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:05.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.669+0000 7f34e1cf8640 1 --2- 172.21.15.83:0/312722325 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f34dc0cf830 0x7f34dc0d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.669+0000 7f34e1cf8640 1 -- 172.21.15.83:0/312722325 >> v1:172.21.15.83:6789/0 conn(0x7f34dc0d27f0 legacy=0x7f34dc0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.669+0000 7f34e1cf8640 1 -- 172.21.15.83:0/312722325 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34dc0b3710 con 0x7f34dc15dfe0 2024-03-06T07:24:05.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.670+0000 7f34e1cf8640 1 --2- 172.21.15.83:0/312722325 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34dc15dfe0 0x7f34dc0cf2f0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f34d80097a0 tx=0x7f34d802b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3520787d790bc692 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:05.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.670+0000 7f34e0cf6640 1 -- 172.21.15.83:0/312722325 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d8037070 con 0x7f34dc15dfe0 2024-03-06T07:24:05.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.670+0000 7f34e0cf6640 1 -- 172.21.15.83:0/312722325 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34d8030710 con 0x7f34dc15dfe0 2024-03-06T07:24:05.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.670+0000 7f34e0cf6640 1 -- 172.21.15.83:0/312722325 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d8030ae0 con 0x7f34dc15dfe0 2024-03-06T07:24:05.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.671+0000 7f34e3f5c640 1 -- 172.21.15.83:0/312722325 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34dc15dfe0 msgr2=0x7f34dc0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.671+0000 7f34e3f5c640 1 --2- 172.21.15.83:0/312722325 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34dc15dfe0 0x7f34dc0cf2f0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f34d80097a0 tx=0x7f34d802b710 comp rx=0 tx=0).stop 2024-03-06T07:24:05.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.671+0000 7f34e3f5c640 1 -- 172.21.15.83:0/312722325 shutdown_connections 2024-03-06T07:24:05.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.671+0000 7f34e3f5c640 1 --2- 172.21.15.83:0/312722325 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f34dc0cf830 0x7f34dc0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.671+0000 7f34e3f5c640 1 --2- 172.21.15.83:0/312722325 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34dc15dfe0 0x7f34dc0cf2f0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.671+0000 7f34e3f5c640 1 -- 172.21.15.83:0/312722325 >> 172.21.15.83:0/312722325 conn(0x7f34dc0d3cd0 msgr2=0x7f34dc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:05.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.671+0000 7f34e3f5c640 1 -- 172.21.15.83:0/312722325 shutdown_connections 2024-03-06T07:24:05.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.671+0000 7f34e3f5c640 1 -- 172.21.15.83:0/312722325 wait complete. 2024-03-06T07:24:05.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.672+0000 7f34e3f5c640 1 Processor -- start 2024-03-06T07:24:05.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.672+0000 7f34e3f5c640 1 -- start start 2024-03-06T07:24:05.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.673+0000 7f34e3f5c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f34dc0cf830 0x7f34dc0cda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.673+0000 7f34e3f5c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34dc0d27f0 0x7f34dc0b6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.673+0000 7f34e3f5c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f34dc15dfe0 0x7f34dc1f73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.673+0000 7f34e3f5c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34dc0d1c70 con 0x7f34dc0d27f0 2024-03-06T07:24:05.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.673+0000 7f34e3f5c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34dc0b3d60 con 0x7f34dc15dfe0 2024-03-06T07:24:05.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.673+0000 7f34e3f5c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34dc0d1df0 con 0x7f34dc0cf830 2024-03-06T07:24:05.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.673+0000 7f34e14f7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34dc0d27f0 0x7f34dc0b6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.673+0000 7f34e14f7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34dc0d27f0 0x7f34dc0b6a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47826/0 (socket says 172.21.15.83:47826) 2024-03-06T07:24:05.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.673+0000 7f34e14f7640 1 -- 172.21.15.83:0/1348607385 learned_addr learned my addr 172.21.15.83:0/1348607385 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:05.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.673+0000 7f34e14f7640 1 -- 172.21.15.83:0/1348607385 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f34dc0cf830 msgr2=0x7f34dc0cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.673+0000 7f34e14f7640 1 --2- 172.21.15.83:0/1348607385 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f34dc0cf830 0x7f34dc0cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.673+0000 7f34e14f7640 1 -- 172.21.15.83:0/1348607385 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f34dc15dfe0 msgr2=0x7f34dc1f73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.673+0000 7f34e14f7640 1 --2- 172.21.15.83:0/1348607385 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f34dc15dfe0 0x7f34dc1f73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.673+0000 7f34e14f7640 1 -- 172.21.15.83:0/1348607385 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34dc0b9ba0 con 0x7f34dc0d27f0 2024-03-06T07:24:05.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.673+0000 7f34e14f7640 1 --2- 172.21.15.83:0/1348607385 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34dc0d27f0 0x7f34dc0b6a30 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f34cc00d7d0 tx=0x7f34cc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:05.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.673+0000 7f34d27fc640 1 -- 172.21.15.83:0/1348607385 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34cc00b6c0 con 0x7f34dc0d27f0 2024-03-06T07:24:05.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.673+0000 7f34d27fc640 1 -- 172.21.15.83:0/1348607385 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34cc019070 con 0x7f34dc0d27f0 2024-03-06T07:24:05.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.673+0000 7f34e3f5c640 1 -- 172.21.15.83:0/1348607385 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34dc15f1c0 con 0x7f34dc0d27f0 2024-03-06T07:24:05.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.673+0000 7f34d27fc640 1 -- 172.21.15.83:0/1348607385 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34cc00bac0 con 0x7f34dc0d27f0 2024-03-06T07:24:05.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.674+0000 7f34e3f5c640 1 -- 172.21.15.83:0/1348607385 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34dc0b98c0 con 0x7f34dc0d27f0 2024-03-06T07:24:05.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.674+0000 7f34e3f5c640 1 -- 172.21.15.83:0/1348607385 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34dc0cf5f0 con 0x7f34dc0d27f0 2024-03-06T07:24:05.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.675+0000 7f34d27fc640 1 -- 172.21.15.83:0/1348607385 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f34cc00bc60 con 0x7f34dc0d27f0 2024-03-06T07:24:05.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.675+0000 7f34d27fc640 1 --2- 172.21.15.83:0/1348607385 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f34c006b4d0 0x7f34c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.675+0000 7f34d27fc640 1 -- 172.21.15.83:0/1348607385 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f34cc08a650 con 0x7f34dc0d27f0 2024-03-06T07:24:05.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.675+0000 7f34e1cf8640 1 --2- 172.21.15.83:0/1348607385 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f34c006b4d0 0x7f34c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.676+0000 7f34e1cf8640 1 --2- 172.21.15.83:0/1348607385 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f34c006b4d0 0x7f34c006d990 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f34d8034000 tx=0x7f34d802f000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:05.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.684+0000 7f34d27fc640 1 -- 172.21.15.83:0/1348607385 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f34dc0cf5f0 con 0x7f34dc0d27f0 2024-03-06T07:24:05.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.895+0000 7f34e3f5c640 1 -- 172.21.15.83:0/1348607385 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34dc04f330 con 0x7f34dc0d27f0 2024-03-06T07:24:05.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.896+0000 7f34d27fc640 1 -- 172.21.15.83:0/1348607385 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f34cc01c360 con 0x7f34dc0d27f0 2024-03-06T07:24:05.898 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:24:05.898 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":19,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5395,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/2270670914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2270670914},{"type":"v1","addr":"172.21.15.83:6837","nonce":2270670914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5398,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/2359880029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2359880029},{"type":"v1","addr":"172.21.15.183:6834","nonce":2359880029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5401,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3099809273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":3099809273},{"type":"v1","addr":"172.21.15.183:6835","nonce":3099809273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:41.430159+0000","modified":"2024-03-06T07:23:43.445803+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5706},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5706":{"gid":5706,"name":"a","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6835/534553538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":534553538},{"type":"v1","addr":"172.21.15.83:6835","nonce":534553538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5706,"qdb_cluster":[5706]},"id":3}]} 2024-03-06T07:24:05.898 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 19 2024-03-06T07:24:05.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.897+0000 7f34e3f5c640 1 -- 172.21.15.83:0/1348607385 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f34c006b4d0 msgr2=0x7f34c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.897+0000 7f34e3f5c640 1 --2- 172.21.15.83:0/1348607385 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f34c006b4d0 0x7f34c006d990 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f34d8034000 tx=0x7f34d802f000 comp rx=0 tx=0).stop 2024-03-06T07:24:05.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.897+0000 7f34e3f5c640 1 -- 172.21.15.83:0/1348607385 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34dc0d27f0 msgr2=0x7f34dc0b6a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.897+0000 7f34e3f5c640 1 --2- 172.21.15.83:0/1348607385 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34dc0d27f0 0x7f34dc0b6a30 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f34cc00d7d0 tx=0x7f34cc00dc90 comp rx=0 tx=0).stop 2024-03-06T07:24:05.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.898+0000 7f34e3f5c640 1 -- 172.21.15.83:0/1348607385 shutdown_connections 2024-03-06T07:24:05.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.898+0000 7f34e3f5c640 1 --2- 172.21.15.83:0/1348607385 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f34c006b4d0 0x7f34c006d990 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.898+0000 7f34e3f5c640 1 --2- 172.21.15.83:0/1348607385 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f34dc15dfe0 0x7f34dc1f73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.898+0000 7f34e3f5c640 1 --2- 172.21.15.83:0/1348607385 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34dc0d27f0 0x7f34dc0b6a30 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.898+0000 7f34e3f5c640 1 --2- 172.21.15.83:0/1348607385 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f34dc0cf830 0x7f34dc0cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.898+0000 7f34e3f5c640 1 -- 172.21.15.83:0/1348607385 >> 172.21.15.83:0/1348607385 conn(0x7f34dc0d3cd0 msgr2=0x7f34dc0cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:05.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.898+0000 7f34e3f5c640 1 -- 172.21.15.83:0/1348607385 shutdown_connections 2024-03-06T07:24:05.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:05.898+0000 7f34e3f5c640 1 -- 172.21.15.83:0/1348607385 wait complete. 2024-03-06T07:24:05.910 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:06.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.012+0000 7fd90709a640 1 Processor -- start 2024-03-06T07:24:06.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.013+0000 7fd90709a640 1 -- start start 2024-03-06T07:24:06.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.013+0000 7fd90709a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fd90015dfe0 0x7fd9000cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.013+0000 7fd90709a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9000cf830 0x7fd9000d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.013+0000 7fd90709a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9000b3430 con 0x7fd9000d27f0 2024-03-06T07:24:06.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.013+0000 7fd90709a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9000d2c50 con 0x7fd90015dfe0 2024-03-06T07:24:06.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.013+0000 7fd90709a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9000b31e0 con 0x7fd9000cf830 2024-03-06T07:24:06.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.014+0000 7fd904e36640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fd90015dfe0 0x7fd9000cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.014+0000 7fd8f7fff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9000cf830 0x7fd9000d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.014+0000 7fd8f7fff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9000cf830 0x7fd9000d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:36638/0 (socket says 172.21.15.83:36638) 2024-03-06T07:24:06.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.014+0000 7fd904e36640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fd90015dfe0 0x7fd9000cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:58946/0 (socket says 172.21.15.83:58946) 2024-03-06T07:24:06.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.014+0000 7fd8f7fff640 1 -- 172.21.15.83:0/1578072229 learned_addr learned my addr 172.21.15.83:0/1578072229 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:06.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.014+0000 7fd905637640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fd9000d27f0 0x7fd9000d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:54296/0 (socket says 172.21.15.83:54296) 2024-03-06T07:24:06.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.014+0000 7fd8f7fff640 1 -- 172.21.15.83:0/1578072229 >> v2:172.21.15.183:3300/0 conn(0x7fd90015dfe0 msgr2=0x7fd9000cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:06.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.014+0000 7fd8f7fff640 1 --2- 172.21.15.83:0/1578072229 >> v2:172.21.15.183:3300/0 conn(0x7fd90015dfe0 0x7fd9000cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.014+0000 7fd8f7fff640 1 -- 172.21.15.83:0/1578072229 >> v1:172.21.15.183:6789/0 conn(0x7fd9000d27f0 legacy=0x7fd9000d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.014+0000 7fd8f7fff640 1 -- 172.21.15.83:0/1578072229 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9000b3710 con 0x7fd9000cf830 2024-03-06T07:24:06.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.014+0000 7fd904e36640 1 --2- 172.21.15.83:0/1578072229 >> v2:172.21.15.183:3300/0 conn(0x7fd90015dfe0 0x7fd9000cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:06.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.015+0000 7fd8f7fff640 1 --2- 172.21.15.83:0/1578072229 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9000cf830 0x7fd9000d22b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd8f0009870 tx=0x7fd8f002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0ef9dfd631f7886 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.015+0000 7fd8f77fe640 1 -- 172.21.15.83:0/1578072229 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f0034070 con 0x7fd9000cf830 2024-03-06T07:24:06.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.015+0000 7fd8f77fe640 1 -- 172.21.15.83:0/1578072229 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8f0030840 con 0x7fd9000cf830 2024-03-06T07:24:06.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.015+0000 7fd8f77fe640 1 -- 172.21.15.83:0/1578072229 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f0030c10 con 0x7fd9000cf830 2024-03-06T07:24:06.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.015+0000 7fd90709a640 1 -- 172.21.15.83:0/1578072229 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9000cf830 msgr2=0x7fd9000d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.015+0000 7fd90709a640 1 --2- 172.21.15.83:0/1578072229 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9000cf830 0x7fd9000d22b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd8f0009870 tx=0x7fd8f002b980 comp rx=0 tx=0).stop 2024-03-06T07:24:06.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.016+0000 7fd90709a640 1 -- 172.21.15.83:0/1578072229 shutdown_connections 2024-03-06T07:24:06.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.016+0000 7fd90709a640 1 --2- 172.21.15.83:0/1578072229 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9000cf830 0x7fd9000d22b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.016+0000 7fd90709a640 1 --2- 172.21.15.83:0/1578072229 >> v2:172.21.15.183:3300/0 conn(0x7fd90015dfe0 0x7fd9000cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.016+0000 7fd90709a640 1 -- 172.21.15.83:0/1578072229 >> 172.21.15.83:0/1578072229 conn(0x7fd9000d3cd0 msgr2=0x7fd9000d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:06.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.016+0000 7fd90709a640 1 -- 172.21.15.83:0/1578072229 shutdown_connections 2024-03-06T07:24:06.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.016+0000 7fd90709a640 1 -- 172.21.15.83:0/1578072229 wait complete. 2024-03-06T07:24:06.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.017+0000 7fd90709a640 1 Processor -- start 2024-03-06T07:24:06.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.017+0000 7fd90709a640 1 -- start start 2024-03-06T07:24:06.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.018+0000 7fd90709a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9000cf830 0x7fd9000ccc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.018+0000 7fd90709a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9000d27f0 0x7fd9000cd190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.018+0000 7fd90709a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd90015dfe0 0x7fd90015f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.018+0000 7fd90709a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9000d1c70 con 0x7fd90015dfe0 2024-03-06T07:24:06.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.018+0000 7fd90709a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9000b3d60 con 0x7fd9000cf830 2024-03-06T07:24:06.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.018+0000 7fd90709a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9000d1df0 con 0x7fd9000d27f0 2024-03-06T07:24:06.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.018+0000 7fd905637640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd90015dfe0 0x7fd90015f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.018+0000 7fd905637640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd90015dfe0 0x7fd90015f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47832/0 (socket says 172.21.15.83:47832) 2024-03-06T07:24:06.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.018+0000 7fd905637640 1 -- 172.21.15.83:0/3187995396 learned_addr learned my addr 172.21.15.83:0/3187995396 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:06.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.018+0000 7fd8f7fff640 1 --2- 172.21.15.83:0/3187995396 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9000d27f0 0x7fd9000cd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.018+0000 7fd904e36640 1 --2- 172.21.15.83:0/3187995396 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9000cf830 0x7fd9000ccc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.018+0000 7fd905637640 1 -- 172.21.15.83:0/3187995396 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9000d27f0 msgr2=0x7fd9000cd190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.018+0000 7fd905637640 1 --2- 172.21.15.83:0/3187995396 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9000d27f0 0x7fd9000cd190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.018+0000 7fd905637640 1 -- 172.21.15.83:0/3187995396 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9000cf830 msgr2=0x7fd9000ccc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.018+0000 7fd905637640 1 --2- 172.21.15.83:0/3187995396 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9000cf830 0x7fd9000ccc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.018+0000 7fd905637640 1 -- 172.21.15.83:0/3187995396 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9000caf40 con 0x7fd90015dfe0 2024-03-06T07:24:06.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.018+0000 7fd905637640 1 --2- 172.21.15.83:0/3187995396 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd90015dfe0 0x7fd90015f870 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fd8f800c780 tx=0x7fd8f800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.018+0000 7fd8f57fa640 1 -- 172.21.15.83:0/3187995396 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f8017070 con 0x7fd90015dfe0 2024-03-06T07:24:06.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.018+0000 7fd8f57fa640 1 -- 172.21.15.83:0/3187995396 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8f800a640 con 0x7fd90015dfe0 2024-03-06T07:24:06.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.018+0000 7fd90709a640 1 -- 172.21.15.83:0/3187995396 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9000cb1e0 con 0x7fd90015dfe0 2024-03-06T07:24:06.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.019+0000 7fd8f57fa640 1 -- 172.21.15.83:0/3187995396 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f800aa30 con 0x7fd90015dfe0 2024-03-06T07:24:06.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.019+0000 7fd90709a640 1 -- 172.21.15.83:0/3187995396 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9000cd8b0 con 0x7fd90015dfe0 2024-03-06T07:24:06.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.020+0000 7fd90709a640 1 -- 172.21.15.83:0/3187995396 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd90004f330 con 0x7fd90015dfe0 2024-03-06T07:24:06.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.020+0000 7fd8f57fa640 1 -- 172.21.15.83:0/3187995396 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd8f801a070 con 0x7fd90015dfe0 2024-03-06T07:24:06.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.020+0000 7fd8f57fa640 1 --2- 172.21.15.83:0/3187995396 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd8ec06b4d0 0x7fd8ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.020+0000 7fd904e36640 1 --2- 172.21.15.83:0/3187995396 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd8ec06b4d0 0x7fd8ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.021+0000 7fd8f57fa640 1 -- 172.21.15.83:0/3187995396 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd8f80896f0 con 0x7fd90015dfe0 2024-03-06T07:24:06.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.021+0000 7fd904e36640 1 --2- 172.21.15.83:0/3187995396 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd8ec06b4d0 0x7fd8ec06d990 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc009560 tx=0x7fd8fc009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.024+0000 7fd8f57fa640 1 -- 172.21.15.83:0/3187995396 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd8f8057f30 con 0x7fd90015dfe0 2024-03-06T07:24:06.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.300+0000 7fd90709a640 1 -- 172.21.15.83:0/3187995396 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9000d0bb0 con 0x7fd90015dfe0 2024-03-06T07:24:06.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.301+0000 7fd8f57fa640 1 -- 172.21.15.83:0/3187995396 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+16625 (secure 0 0 0) 0x7fd8f805bfd0 con 0x7fd90015dfe0 2024-03-06T07:24:06.303 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:24:06.303 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":45,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:23:43.092917+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:39.656617+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:40.402765+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:06.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.303+0000 7fd90709a640 1 -- 172.21.15.83:0/3187995396 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd8ec06b4d0 msgr2=0x7fd8ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.303+0000 7fd90709a640 1 --2- 172.21.15.83:0/3187995396 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd8ec06b4d0 0x7fd8ec06d990 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc009560 tx=0x7fd8fc009190 comp rx=0 tx=0).stop 2024-03-06T07:24:06.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.303+0000 7fd90709a640 1 -- 172.21.15.83:0/3187995396 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd90015dfe0 msgr2=0x7fd90015f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.303+0000 7fd90709a640 1 --2- 172.21.15.83:0/3187995396 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd90015dfe0 0x7fd90015f870 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fd8f800c780 tx=0x7fd8f800cc40 comp rx=0 tx=0).stop 2024-03-06T07:24:06.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.303+0000 7fd90709a640 1 -- 172.21.15.83:0/3187995396 shutdown_connections 2024-03-06T07:24:06.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.303+0000 7fd90709a640 1 --2- 172.21.15.83:0/3187995396 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd8ec06b4d0 0x7fd8ec06d990 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.303+0000 7fd90709a640 1 --2- 172.21.15.83:0/3187995396 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd90015dfe0 0x7fd90015f870 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.303+0000 7fd90709a640 1 --2- 172.21.15.83:0/3187995396 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9000d27f0 0x7fd9000cd190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.303+0000 7fd90709a640 1 --2- 172.21.15.83:0/3187995396 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9000cf830 0x7fd9000ccc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.303+0000 7fd90709a640 1 -- 172.21.15.83:0/3187995396 >> 172.21.15.83:0/3187995396 conn(0x7fd9000d3cd0 msgr2=0x7fd9000cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:06.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.303+0000 7fd90709a640 1 -- 172.21.15.83:0/3187995396 shutdown_connections 2024-03-06T07:24:06.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.303+0000 7fd90709a640 1 -- 172.21.15.83:0/3187995396 wait complete. 2024-03-06T07:24:06.316 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:23:39.656617+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:23:40.402765+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:06.316 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:06.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.419+0000 7f2b38215640 1 Processor -- start 2024-03-06T07:24:06.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.419+0000 7f2b38215640 1 -- start start 2024-03-06T07:24:06.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.420+0000 7f2b38215640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2b300b2590 0x7f2b30066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.420+0000 7f2b38215640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b3006bc10 con 0x7f2b3015c1a0 2024-03-06T07:24:06.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.420+0000 7f2b38215640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b3006bef0 con 0x7f2b3015bdc0 2024-03-06T07:24:06.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.420+0000 7f2b38215640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b30067240 con 0x7f2b300b2590 2024-03-06T07:24:06.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.420+0000 7f2b357b0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2b300b2590 0x7f2b30066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.420+0000 7f2b367b2640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f2b3015c1a0 0x7f2b3015cf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:35432/0 (socket says 172.21.15.83:35432) 2024-03-06T07:24:06.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.420+0000 7f2b367b2640 1 -- 172.21.15.83:0/2763940608 learned_addr learned my addr 172.21.15.83:0/2763940608 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:24:06.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.421+0000 7f2b357b0640 1 -- 172.21.15.83:0/2763940608 >> v1:172.21.15.183:6789/0 conn(0x7f2b3015bdc0 legacy=0x7f2b300b1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:06.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.421+0000 7f2b357b0640 1 -- 172.21.15.83:0/2763940608 >> v1:172.21.15.83:6789/0 conn(0x7f2b3015c1a0 legacy=0x7f2b3015cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.421+0000 7f2b357b0640 1 -- 172.21.15.83:0/2763940608 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b3015c730 con 0x7f2b300b2590 2024-03-06T07:24:06.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.421+0000 7f2b34faf640 1 -- 172.21.15.83:0/2763940608 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2391452588 0 0) 0x7f2b3006bc10 con 0x7f2b3015c1a0 2024-03-06T07:24:06.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.421+0000 7f2b357b0640 1 --2- 172.21.15.83:0/2763940608 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b300b2590 0x7f2b30066d00 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f2b200097a0 tx=0x7f2b2002b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5bad1520121357c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.422+0000 7f2b34faf640 1 -- 172.21.15.83:0/2763940608 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b2002f810 con 0x7f2b300b2590 2024-03-06T07:24:06.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.422+0000 7f2b34faf640 1 -- 172.21.15.83:0/2763940608 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b2002f9b0 con 0x7f2b300b2590 2024-03-06T07:24:06.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.422+0000 7f2b34faf640 1 -- 172.21.15.83:0/2763940608 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b2002fd80 con 0x7f2b300b2590 2024-03-06T07:24:06.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.422+0000 7f2b38215640 1 -- 172.21.15.83:0/2763940608 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b300b2590 msgr2=0x7f2b30066d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.422+0000 7f2b38215640 1 --2- 172.21.15.83:0/2763940608 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b300b2590 0x7f2b30066d00 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f2b200097a0 tx=0x7f2b2002b9f0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.422+0000 7f2b38215640 1 -- 172.21.15.83:0/2763940608 shutdown_connections 2024-03-06T07:24:06.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.422+0000 7f2b38215640 1 --2- 172.21.15.83:0/2763940608 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b300b2590 0x7f2b30066d00 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.422+0000 7f2b38215640 1 -- 172.21.15.83:0/2763940608 >> 172.21.15.83:0/2763940608 conn(0x7f2b300b4fe0 msgr2=0x7f2b300b56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:06.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.422+0000 7f2b38215640 1 -- 172.21.15.83:0/2763940608 shutdown_connections 2024-03-06T07:24:06.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.422+0000 7f2b38215640 1 -- 172.21.15.83:0/2763940608 wait complete. 2024-03-06T07:24:06.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.423+0000 7f2b38215640 1 Processor -- start 2024-03-06T07:24:06.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.423+0000 7f2b38215640 1 -- start start 2024-03-06T07:24:06.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.424+0000 7f2b38215640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b300b2590 0x7f2b30156090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.424+0000 7f2b38215640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2b3015bdc0 0x7f2b30154c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.424+0000 7f2b38215640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b3015c1a0 0x7f2b301508c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.424+0000 7f2b38215640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3006b8a0 con 0x7f2b300b2590 2024-03-06T07:24:06.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.424+0000 7f2b38215640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3006c1d0 con 0x7f2b3015c1a0 2024-03-06T07:24:06.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.424+0000 7f2b38215640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b300b5f70 con 0x7f2b3015bdc0 2024-03-06T07:24:06.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.424+0000 7f2b35fb1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b300b2590 0x7f2b30156090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.424+0000 7f2b35fb1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b300b2590 0x7f2b30156090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47864/0 (socket says 172.21.15.83:47864) 2024-03-06T07:24:06.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.424+0000 7f2b35fb1640 1 -- 172.21.15.83:0/123054090 learned_addr learned my addr 172.21.15.83:0/123054090 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:06.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.424+0000 7f2b35fb1640 1 -- 172.21.15.83:0/123054090 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2b3015bdc0 msgr2=0x7f2b30154c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.424+0000 7f2b357b0640 1 --2- 172.21.15.83:0/123054090 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2b3015bdc0 0x7f2b30154c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.424+0000 7f2b367b2640 1 --2- 172.21.15.83:0/123054090 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b3015c1a0 0x7f2b301508c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.425+0000 7f2b35fb1640 1 --2- 172.21.15.83:0/123054090 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2b3015bdc0 0x7f2b30154c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.425+0000 7f2b35fb1640 1 -- 172.21.15.83:0/123054090 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b3015c1a0 msgr2=0x7f2b301508c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.425+0000 7f2b35fb1640 1 --2- 172.21.15.83:0/123054090 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b3015c1a0 0x7f2b301508c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.425+0000 7f2b35fb1640 1 -- 172.21.15.83:0/123054090 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b30155470 con 0x7f2b300b2590 2024-03-06T07:24:06.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.425+0000 7f2b35fb1640 1 --2- 172.21.15.83:0/123054090 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b300b2590 0x7f2b30156090 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c00c450 tx=0x7f2b2c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.425+0000 7f2b267fc640 1 -- 172.21.15.83:0/123054090 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b2c0052d0 con 0x7f2b300b2590 2024-03-06T07:24:06.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.425+0000 7f2b267fc640 1 -- 172.21.15.83:0/123054090 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b2c005470 con 0x7f2b300b2590 2024-03-06T07:24:06.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.425+0000 7f2b38215640 1 -- 172.21.15.83:0/123054090 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b300ad300 con 0x7f2b300b2590 2024-03-06T07:24:06.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.425+0000 7f2b267fc640 1 -- 172.21.15.83:0/123054090 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b2c012870 con 0x7f2b300b2590 2024-03-06T07:24:06.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.425+0000 7f2b38215640 1 -- 172.21.15.83:0/123054090 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b300ad8c0 con 0x7f2b300b2590 2024-03-06T07:24:06.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.426+0000 7f2b38215640 1 -- 172.21.15.83:0/123054090 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b30156740 con 0x7f2b300b2590 2024-03-06T07:24:06.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.426+0000 7f2b267fc640 1 -- 172.21.15.83:0/123054090 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2b2c012a10 con 0x7f2b300b2590 2024-03-06T07:24:06.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.427+0000 7f2b267fc640 1 --2- 172.21.15.83:0/123054090 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2b1406b4d0 0x7f2b1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.427+0000 7f2b357b0640 1 --2- 172.21.15.83:0/123054090 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2b1406b4d0 0x7f2b1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.427+0000 7f2b267fc640 1 -- 172.21.15.83:0/123054090 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2b2c089d80 con 0x7f2b300b2590 2024-03-06T07:24:06.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.427+0000 7f2b357b0640 1 --2- 172.21.15.83:0/123054090 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2b1406b4d0 0x7f2b1406d990 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f2b20036000 tx=0x7f2b20039040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.431+0000 7f2b267fc640 1 -- 172.21.15.83:0/123054090 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2b2c058c90 con 0x7f2b300b2590 2024-03-06T07:24:06.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.673+0000 7f2b38215640 1 -- 172.21.15.83:0/123054090 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2b30068ff0 con 0x7f2b300b2590 2024-03-06T07:24:06.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.674+0000 7f2b267fc640 1 -- 172.21.15.83:0/123054090 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v45) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2b2c058590 con 0x7f2b300b2590 2024-03-06T07:24:06.676 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:24:06.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.676+0000 7f2b38215640 1 -- 172.21.15.83:0/123054090 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2b1406b4d0 msgr2=0x7f2b1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.676+0000 7f2b38215640 1 --2- 172.21.15.83:0/123054090 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2b1406b4d0 0x7f2b1406d990 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f2b20036000 tx=0x7f2b20039040 comp rx=0 tx=0).stop 2024-03-06T07:24:06.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.676+0000 7f2b38215640 1 -- 172.21.15.83:0/123054090 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b300b2590 msgr2=0x7f2b30156090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.676+0000 7f2b38215640 1 --2- 172.21.15.83:0/123054090 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b300b2590 0x7f2b30156090 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c00c450 tx=0x7f2b2c00c910 comp rx=0 tx=0).stop 2024-03-06T07:24:06.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.676+0000 7f2b38215640 1 -- 172.21.15.83:0/123054090 shutdown_connections 2024-03-06T07:24:06.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.676+0000 7f2b38215640 1 --2- 172.21.15.83:0/123054090 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2b1406b4d0 0x7f2b1406d990 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.676+0000 7f2b38215640 1 --2- 172.21.15.83:0/123054090 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b3015c1a0 0x7f2b301508c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.676+0000 7f2b38215640 1 --2- 172.21.15.83:0/123054090 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2b3015bdc0 0x7f2b30154c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.676+0000 7f2b38215640 1 --2- 172.21.15.83:0/123054090 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b300b2590 0x7f2b30156090 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.676+0000 7f2b38215640 1 -- 172.21.15.83:0/123054090 >> 172.21.15.83:0/123054090 conn(0x7f2b300b4fe0 msgr2=0x7f2b3015a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:06.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.676+0000 7f2b38215640 1 -- 172.21.15.83:0/123054090 shutdown_connections 2024-03-06T07:24:06.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.676+0000 7f2b38215640 1 -- 172.21.15.83:0/123054090 wait complete. 2024-03-06T07:24:06.692 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:24:06.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.794+0000 7f3aafc0f640 1 Processor -- start 2024-03-06T07:24:06.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.794+0000 7f3aafc0f640 1 -- start start 2024-03-06T07:24:06.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.795+0000 7f3aafc0f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3aa81614d0 0x7f3aa80d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.795+0000 7f3aafc0f640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f3aa80d1910 0x7f3aa80d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.795+0000 7f3aafc0f640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aa80d2740 con 0x7f3aa80d22b0 2024-03-06T07:24:06.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.795+0000 7f3aafc0f640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3aa81618b0 con 0x7f3aa80d1910 2024-03-06T07:24:06.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.795+0000 7f3aafc0f640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa80cf400 con 0x7f3aa81614d0 2024-03-06T07:24:06.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.795+0000 7f3aad1aa640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f3aa80d1910 0x7f3aa80d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.795+0000 7f3aae1ac640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f3aa80d22b0 0x7f3aa80d06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:55982/0 (socket says 172.21.15.83:55982) 2024-03-06T07:24:06.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.795+0000 7f3aad1aa640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f3aa80d1910 0x7f3aa80d1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:52684/0 (socket says 172.21.15.83:52684) 2024-03-06T07:24:06.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.795+0000 7f3aae1ac640 1 -- 172.21.15.83:0/2389976601 learned_addr learned my addr 172.21.15.83:0/2389976601 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:24:06.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.795+0000 7f3aad9ab640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3aa81614d0 0x7f3aa80d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.796+0000 7f3aad1aa640 1 -- 172.21.15.83:0/2389976601 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3aa81614d0 msgr2=0x7f3aa80d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:06.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.796+0000 7f3aad1aa640 1 --2- 172.21.15.83:0/2389976601 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3aa81614d0 0x7f3aa80d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.796+0000 7f3aad1aa640 1 -- 172.21.15.83:0/2389976601 >> v1:172.21.15.183:6789/0 conn(0x7f3aa80d22b0 legacy=0x7f3aa80d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.796+0000 7f3aad1aa640 1 -- 172.21.15.83:0/2389976601 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa80cf6a0 con 0x7f3aa80d1910 2024-03-06T07:24:06.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.796+0000 7f3aad9ab640 1 --2- 172.21.15.83:0/2389976601 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3aa81614d0 0x7f3aa80d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:06.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.796+0000 7f3aad1aa640 1 --2- 172.21.15.83:0/2389976601 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3aa80d1910 0x7f3aa80d1d70 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3a980097a0 tx=0x7f3a9802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0f351e829432e5a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.797+0000 7f3aac9a9640 1 -- 172.21.15.83:0/2389976601 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a98035020 con 0x7f3aa80d1910 2024-03-06T07:24:06.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.797+0000 7f3aac9a9640 1 -- 172.21.15.83:0/2389976601 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a98031830 con 0x7f3aa80d1910 2024-03-06T07:24:06.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.797+0000 7f3aac9a9640 1 -- 172.21.15.83:0/2389976601 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a98031c00 con 0x7f3aa80d1910 2024-03-06T07:24:06.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.797+0000 7f3aafc0f640 1 -- 172.21.15.83:0/2389976601 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3aa80d1910 msgr2=0x7f3aa80d1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.797+0000 7f3aafc0f640 1 --2- 172.21.15.83:0/2389976601 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3aa80d1910 0x7f3aa80d1d70 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3a980097a0 tx=0x7f3a9802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.797+0000 7f3aafc0f640 1 -- 172.21.15.83:0/2389976601 shutdown_connections 2024-03-06T07:24:06.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.797+0000 7f3aafc0f640 1 --2- 172.21.15.83:0/2389976601 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3aa80d1910 0x7f3aa80d1d70 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.797+0000 7f3aafc0f640 1 --2- 172.21.15.83:0/2389976601 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3aa81614d0 0x7f3aa80d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.797+0000 7f3aafc0f640 1 -- 172.21.15.83:0/2389976601 >> 172.21.15.83:0/2389976601 conn(0x7f3aa80d3d40 msgr2=0x7f3aa80d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:06.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.797+0000 7f3aafc0f640 1 -- 172.21.15.83:0/2389976601 shutdown_connections 2024-03-06T07:24:06.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.797+0000 7f3aafc0f640 1 -- 172.21.15.83:0/2389976601 wait complete. 2024-03-06T07:24:06.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.798+0000 7f3aafc0f640 1 Processor -- start 2024-03-06T07:24:06.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.798+0000 7f3aafc0f640 1 -- start start 2024-03-06T07:24:06.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.799+0000 7f3aafc0f640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3aa80d1910 0x7f3aa815c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.799+0000 7f3aafc0f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3aa80d22b0 0x7f3aa80cd420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.799+0000 7f3aafc0f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3aa81614d0 0x7f3aa81602b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.799+0000 7f3aafc0f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa80b0910 con 0x7f3aa81614d0 2024-03-06T07:24:06.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.799+0000 7f3aafc0f640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa80cfc60 con 0x7f3aa80d1910 2024-03-06T07:24:06.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.799+0000 7f3aafc0f640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa80b1e40 con 0x7f3aa80d22b0 2024-03-06T07:24:06.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.799+0000 7f3aae1ac640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3aa81614d0 0x7f3aa81602b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.799+0000 7f3aae1ac640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3aa81614d0 0x7f3aa81602b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33306/0 (socket says 172.21.15.83:33306) 2024-03-06T07:24:06.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.799+0000 7f3aae1ac640 1 -- 172.21.15.83:0/2418033796 learned_addr learned my addr 172.21.15.83:0/2418033796 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:06.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.799+0000 7f3aad9ab640 1 --2- 172.21.15.83:0/2418033796 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3aa80d1910 0x7f3aa815c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.799+0000 7f3aad1aa640 1 --2- 172.21.15.83:0/2418033796 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3aa80d22b0 0x7f3aa80cd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.800+0000 7f3aae1ac640 1 -- 172.21.15.83:0/2418033796 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3aa80d22b0 msgr2=0x7f3aa80cd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.800+0000 7f3aae1ac640 1 --2- 172.21.15.83:0/2418033796 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3aa80d22b0 0x7f3aa80cd420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.800+0000 7f3aae1ac640 1 -- 172.21.15.83:0/2418033796 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3aa80d1910 msgr2=0x7f3aa815c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.800+0000 7f3aae1ac640 1 --2- 172.21.15.83:0/2418033796 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3aa80d1910 0x7f3aa815c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.800+0000 7f3aae1ac640 1 -- 172.21.15.83:0/2418033796 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa80b7b30 con 0x7f3aa81614d0 2024-03-06T07:24:06.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.800+0000 7f3aae1ac640 1 --2- 172.21.15.83:0/2418033796 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3aa81614d0 0x7f3aa81602b0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f3aa000cbf0 tx=0x7f3aa000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.800+0000 7f3a9e7fc640 1 -- 172.21.15.83:0/2418033796 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa0017070 con 0x7f3aa81614d0 2024-03-06T07:24:06.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.800+0000 7f3a9e7fc640 1 -- 172.21.15.83:0/2418033796 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aa000ad40 con 0x7f3aa81614d0 2024-03-06T07:24:06.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.800+0000 7f3aafc0f640 1 -- 172.21.15.83:0/2418033796 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3aa815c880 con 0x7f3aa81614d0 2024-03-06T07:24:06.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.800+0000 7f3a9e7fc640 1 -- 172.21.15.83:0/2418033796 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa0012600 con 0x7f3aa81614d0 2024-03-06T07:24:06.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.800+0000 7f3aafc0f640 1 -- 172.21.15.83:0/2418033796 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3aa80b80b0 con 0x7f3aa81614d0 2024-03-06T07:24:06.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.801+0000 7f3aafc0f640 1 -- 172.21.15.83:0/2418033796 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3aa8068ff0 con 0x7f3aa81614d0 2024-03-06T07:24:06.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.801+0000 7f3a9e7fc640 1 -- 172.21.15.83:0/2418033796 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3aa001a070 con 0x7f3aa81614d0 2024-03-06T07:24:06.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.802+0000 7f3a9e7fc640 1 --2- 172.21.15.83:0/2418033796 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3a8c06b4d0 0x7f3a8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.802+0000 7f3a9e7fc640 1 -- 172.21.15.83:0/2418033796 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3aa0089e20 con 0x7f3aa81614d0 2024-03-06T07:24:06.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.802+0000 7f3aad9ab640 1 --2- 172.21.15.83:0/2418033796 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3a8c06b4d0 0x7f3a8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.802+0000 7f3aad9ab640 1 --2- 172.21.15.83:0/2418033796 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3a8c06b4d0 0x7f3a8c06d990 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f3aa80ca4a0 tx=0x7f3aa4009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.806+0000 7f3a9e7fc640 1 -- 172.21.15.83:0/2418033796 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3aa80d2740 con 0x7f3aa81614d0 2024-03-06T07:24:06.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.983+0000 7f3aafc0f640 1 -- 172.21.15.83:0/2418033796 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3aa80a1c10 con 0x7f3aa81614d0 2024-03-06T07:24:06.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.983+0000 7f3a9e7fc640 1 -- 172.21.15.83:0/2418033796 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v45) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3aa00586e0 con 0x7f3aa81614d0 2024-03-06T07:24:06.985 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-03-06T07:24:06.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.985+0000 7f3aafc0f640 1 -- 172.21.15.83:0/2418033796 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3a8c06b4d0 msgr2=0x7f3a8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.985+0000 7f3aafc0f640 1 --2- 172.21.15.83:0/2418033796 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3a8c06b4d0 0x7f3a8c06d990 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f3aa80ca4a0 tx=0x7f3aa4009190 comp rx=0 tx=0).stop 2024-03-06T07:24:06.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.985+0000 7f3aafc0f640 1 -- 172.21.15.83:0/2418033796 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3aa81614d0 msgr2=0x7f3aa81602b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.985+0000 7f3aafc0f640 1 --2- 172.21.15.83:0/2418033796 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3aa81614d0 0x7f3aa81602b0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f3aa000cbf0 tx=0x7f3aa000a430 comp rx=0 tx=0).stop 2024-03-06T07:24:06.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.985+0000 7f3aafc0f640 1 -- 172.21.15.83:0/2418033796 shutdown_connections 2024-03-06T07:24:06.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.985+0000 7f3aafc0f640 1 --2- 172.21.15.83:0/2418033796 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3a8c06b4d0 0x7f3a8c06d990 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.985+0000 7f3aafc0f640 1 --2- 172.21.15.83:0/2418033796 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3aa81614d0 0x7f3aa81602b0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.985+0000 7f3aafc0f640 1 --2- 172.21.15.83:0/2418033796 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3aa80d22b0 0x7f3aa80cd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.985+0000 7f3aafc0f640 1 --2- 172.21.15.83:0/2418033796 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3aa80d1910 0x7f3aa815c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.985+0000 7f3aafc0f640 1 -- 172.21.15.83:0/2418033796 >> 172.21.15.83:0/2418033796 conn(0x7f3aa80d3d40 msgr2=0x7f3aa815fad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:06.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.985+0000 7f3aafc0f640 1 -- 172.21.15.83:0/2418033796 shutdown_connections 2024-03-06T07:24:06.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:06.985+0000 7f3aafc0f640 1 -- 172.21.15.83:0/2418033796 wait complete. 2024-03-06T07:24:07.001 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:24:07.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.105+0000 7fdd6a288640 1 Processor -- start 2024-03-06T07:24:07.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.105+0000 7fdd6a288640 1 -- start start 2024-03-06T07:24:07.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.106+0000 7fdd6a288640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd6415dd00 0x7fdd6406e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.106+0000 7fdd6a288640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fdd6406b6c0 0x7fdd640b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.106+0000 7fdd6a288640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd64161bd0 con 0x7fdd6415e0e0 2024-03-06T07:24:07.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.106+0000 7fdd6a288640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd64070220 con 0x7fdd6406b6c0 2024-03-06T07:24:07.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.106+0000 7fdd6a288640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd64161e90 con 0x7fdd6415dd00 2024-03-06T07:24:07.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.106+0000 7fdd68825640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fdd6406b6c0 0x7fdd640b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.106+0000 7fdd68825640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fdd6406b6c0 0x7fdd640b42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33310/0 (socket says 172.21.15.83:33310) 2024-03-06T07:24:07.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.106+0000 7fdd68825640 1 -- 172.21.15.83:0/1766674083 learned_addr learned my addr 172.21.15.83:0/1766674083 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:07.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.106+0000 7fdd63fff640 1 --2- 172.21.15.83:0/1766674083 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd6415dd00 0x7fdd6406e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.106+0000 7fdd68825640 1 -- 172.21.15.83:0/1766674083 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd6415dd00 msgr2=0x7fdd6406e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:07.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.106+0000 7fdd68825640 1 --2- 172.21.15.83:0/1766674083 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd6415dd00 0x7fdd6406e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.106+0000 7fdd68825640 1 -- 172.21.15.83:0/1766674083 >> v1:172.21.15.183:6789/0 conn(0x7fdd6415e0e0 legacy=0x7fdd6406afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.106+0000 7fdd68825640 1 -- 172.21.15.83:0/1766674083 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd64162170 con 0x7fdd6406b6c0 2024-03-06T07:24:07.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.107+0000 7fdd68825640 1 --2- 172.21.15.83:0/1766674083 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd6406b6c0 0x7fdd640b42d0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c009e50 tx=0x7fdd5c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0231c6341af8f11 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:07.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.108+0000 7fdd62ffd640 1 -- 172.21.15.83:0/1766674083 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5c02cb00 con 0x7fdd6406b6c0 2024-03-06T07:24:07.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.108+0000 7fdd62ffd640 1 -- 172.21.15.83:0/1766674083 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd5c02cca0 con 0x7fdd6406b6c0 2024-03-06T07:24:07.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.108+0000 7fdd62ffd640 1 -- 172.21.15.83:0/1766674083 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5c034610 con 0x7fdd6406b6c0 2024-03-06T07:24:07.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.108+0000 7fdd6a288640 1 -- 172.21.15.83:0/1766674083 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd6406b6c0 msgr2=0x7fdd640b42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.108+0000 7fdd6a288640 1 --2- 172.21.15.83:0/1766674083 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd6406b6c0 0x7fdd640b42d0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c009e50 tx=0x7fdd5c02bf60 comp rx=0 tx=0).stop 2024-03-06T07:24:07.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.108+0000 7fdd6a288640 1 -- 172.21.15.83:0/1766674083 shutdown_connections 2024-03-06T07:24:07.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.108+0000 7fdd6a288640 1 --2- 172.21.15.83:0/1766674083 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd6406b6c0 0x7fdd640b42d0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.108+0000 7fdd6a288640 1 --2- 172.21.15.83:0/1766674083 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd6415dd00 0x7fdd6406e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.108+0000 7fdd6a288640 1 -- 172.21.15.83:0/1766674083 >> 172.21.15.83:0/1766674083 conn(0x7fdd640b2660 msgr2=0x7fdd640b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:07.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.108+0000 7fdd6a288640 1 -- 172.21.15.83:0/1766674083 shutdown_connections 2024-03-06T07:24:07.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.108+0000 7fdd6a288640 1 -- 172.21.15.83:0/1766674083 wait complete. 2024-03-06T07:24:07.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.109+0000 7fdd6a288640 1 Processor -- start 2024-03-06T07:24:07.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.110+0000 7fdd6a288640 1 -- start start 2024-03-06T07:24:07.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.110+0000 7fdd6a288640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd6406b6c0 0x7fdd64148620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.110+0000 7fdd6a288640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd6415dd00 0x7fdd6415acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.110+0000 7fdd6a288640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd641f42c0 0x7fdd6415fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.110+0000 7fdd6a288640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd640b0180 con 0x7fdd641f42c0 2024-03-06T07:24:07.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.110+0000 7fdd6a288640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd640b0000 con 0x7fdd6406b6c0 2024-03-06T07:24:07.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.110+0000 7fdd6a288640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd640b0300 con 0x7fdd6415dd00 2024-03-06T07:24:07.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.110+0000 7fdd68825640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd641f42c0 0x7fdd6415fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.110+0000 7fdd68825640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd641f42c0 0x7fdd6415fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33312/0 (socket says 172.21.15.83:33312) 2024-03-06T07:24:07.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.110+0000 7fdd68825640 1 -- 172.21.15.83:0/2930778405 learned_addr learned my addr 172.21.15.83:0/2930778405 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:07.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.111+0000 7fdd637fe640 1 --2- 172.21.15.83:0/2930778405 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd6415dd00 0x7fdd6415acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.111+0000 7fdd63fff640 1 --2- 172.21.15.83:0/2930778405 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd6406b6c0 0x7fdd64148620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.111+0000 7fdd68825640 1 -- 172.21.15.83:0/2930778405 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd6415dd00 msgr2=0x7fdd6415acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.111+0000 7fdd68825640 1 --2- 172.21.15.83:0/2930778405 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd6415dd00 0x7fdd6415acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.111+0000 7fdd68825640 1 -- 172.21.15.83:0/2930778405 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd6406b6c0 msgr2=0x7fdd64148620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.111+0000 7fdd68825640 1 --2- 172.21.15.83:0/2930778405 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd6406b6c0 0x7fdd64148620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.111+0000 7fdd68825640 1 -- 172.21.15.83:0/2930778405 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd641f4880 con 0x7fdd641f42c0 2024-03-06T07:24:07.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.111+0000 7fdd68825640 1 --2- 172.21.15.83:0/2930778405 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd641f42c0 0x7fdd6415fe10 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c036040 tx=0x7fdd5c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:07.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.111+0000 7fdd60ff9640 1 -- 172.21.15.83:0/2930778405 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5c043020 con 0x7fdd641f42c0 2024-03-06T07:24:07.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.111+0000 7fdd60ff9640 1 -- 172.21.15.83:0/2930778405 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd5c02ce50 con 0x7fdd641f42c0 2024-03-06T07:24:07.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.111+0000 7fdd6a288640 1 -- 172.21.15.83:0/2930778405 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd641f4b60 con 0x7fdd641f42c0 2024-03-06T07:24:07.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.111+0000 7fdd60ff9640 1 -- 172.21.15.83:0/2930778405 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5c009510 con 0x7fdd641f42c0 2024-03-06T07:24:07.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.111+0000 7fdd6a288640 1 -- 172.21.15.83:0/2930778405 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd641f5120 con 0x7fdd641f42c0 2024-03-06T07:24:07.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.112+0000 7fdd6a288640 1 -- 172.21.15.83:0/2930778405 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd64073390 con 0x7fdd641f42c0 2024-03-06T07:24:07.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.113+0000 7fdd60ff9640 1 -- 172.21.15.83:0/2930778405 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdd5c046020 con 0x7fdd641f42c0 2024-03-06T07:24:07.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.113+0000 7fdd60ff9640 1 --2- 172.21.15.83:0/2930778405 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdd5006b4d0 0x7fdd5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.113+0000 7fdd60ff9640 1 -- 172.21.15.83:0/2930778405 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdd5c0ac630 con 0x7fdd641f42c0 2024-03-06T07:24:07.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.113+0000 7fdd63fff640 1 --2- 172.21.15.83:0/2930778405 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdd5006b4d0 0x7fdd5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.113+0000 7fdd63fff640 1 --2- 172.21.15.83:0/2930778405 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdd5006b4d0 0x7fdd5006d990 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fdd54009b40 tx=0x7fdd54009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:07.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.117+0000 7fdd60ff9640 1 -- 172.21.15.83:0/2930778405 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdd5c04d030 con 0x7fdd641f42c0 2024-03-06T07:24:07.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.294+0000 7fdd6a288640 1 -- 172.21.15.83:0/2930778405 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdd64068ff0 con 0x7fdd641f42c0 2024-03-06T07:24:07.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.294+0000 7fdd60ff9640 1 -- 172.21.15.83:0/2930778405 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v45) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdd5c07fe20 con 0x7fdd641f42c0 2024-03-06T07:24:07.296 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 64 2024-03-06T07:24:07.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.296+0000 7fdd6a288640 1 -- 172.21.15.83:0/2930778405 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdd5006b4d0 msgr2=0x7fdd5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.296+0000 7fdd6a288640 1 --2- 172.21.15.83:0/2930778405 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdd5006b4d0 0x7fdd5006d990 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fdd54009b40 tx=0x7fdd54009190 comp rx=0 tx=0).stop 2024-03-06T07:24:07.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.296+0000 7fdd6a288640 1 -- 172.21.15.83:0/2930778405 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd641f42c0 msgr2=0x7fdd6415fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.296+0000 7fdd6a288640 1 --2- 172.21.15.83:0/2930778405 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd641f42c0 0x7fdd6415fe10 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c036040 tx=0x7fdd5c004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:07.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.296+0000 7fdd6a288640 1 -- 172.21.15.83:0/2930778405 shutdown_connections 2024-03-06T07:24:07.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.296+0000 7fdd6a288640 1 --2- 172.21.15.83:0/2930778405 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdd5006b4d0 0x7fdd5006d990 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.296+0000 7fdd6a288640 1 --2- 172.21.15.83:0/2930778405 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdd641f42c0 0x7fdd6415fe10 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.296+0000 7fdd6a288640 1 --2- 172.21.15.83:0/2930778405 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd6415dd00 0x7fdd6415acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.296+0000 7fdd6a288640 1 --2- 172.21.15.83:0/2930778405 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd6406b6c0 0x7fdd64148620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.296+0000 7fdd6a288640 1 -- 172.21.15.83:0/2930778405 >> 172.21.15.83:0/2930778405 conn(0x7fdd640b2660 msgr2=0x7fdd64071e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:07.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.296+0000 7fdd6a288640 1 -- 172.21.15.83:0/2930778405 shutdown_connections 2024-03-06T07:24:07.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.296+0000 7fdd6a288640 1 -- 172.21.15.83:0/2930778405 wait complete. 2024-03-06T07:24:07.312 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:07.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.412+0000 7fd3fd51e640 1 Processor -- start 2024-03-06T07:24:07.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.412+0000 7fd3fd51e640 1 -- start start 2024-03-06T07:24:07.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.413+0000 7fd3fd51e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd3f80b3060 0x7fd3f815e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.413+0000 7fd3fd51e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fd3f815bf30 0x7fd3f815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.413+0000 7fd3fd51e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3f815f2d0 con 0x7fd3f813dc10 2024-03-06T07:24:07.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.413+0000 7fd3fd51e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3f80cf3f0 con 0x7fd3f80b3060 2024-03-06T07:24:07.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.413+0000 7fd3fd51e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3f815b210 con 0x7fd3f815bf30 2024-03-06T07:24:07.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.413+0000 7fd3effff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd3f80b3060 0x7fd3f815e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.413+0000 7fd3effff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd3f80b3060 0x7fd3f815e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33322/0 (socket says 172.21.15.83:33322) 2024-03-06T07:24:07.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.413+0000 7fd3effff640 1 -- 172.21.15.83:0/2744543887 learned_addr learned my addr 172.21.15.83:0/2744543887 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:07.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.413+0000 7fd3f77fe640 1 --2- 172.21.15.83:0/2744543887 >> v2:172.21.15.183:3300/0 conn(0x7fd3f815bf30 0x7fd3f815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.413+0000 7fd3effff640 1 -- 172.21.15.83:0/2744543887 >> v2:172.21.15.183:3300/0 conn(0x7fd3f815bf30 msgr2=0x7fd3f815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:07.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.413+0000 7fd3effff640 1 --2- 172.21.15.83:0/2744543887 >> v2:172.21.15.183:3300/0 conn(0x7fd3f815bf30 0x7fd3f815ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.413+0000 7fd3effff640 1 -- 172.21.15.83:0/2744543887 >> v1:172.21.15.183:6789/0 conn(0x7fd3f813dc10 legacy=0x7fd3f80b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:07.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.413+0000 7fd3effff640 1 -- 172.21.15.83:0/2744543887 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f815f580 con 0x7fd3f80b3060 2024-03-06T07:24:07.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.413+0000 7fd3f77fe640 1 --2- 172.21.15.83:0/2744543887 >> v2:172.21.15.183:3300/0 conn(0x7fd3f815bf30 0x7fd3f815ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:07.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.414+0000 7fd3effff640 1 --2- 172.21.15.83:0/2744543887 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd3f80b3060 0x7fd3f815e850 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fd3f80b4e00 tx=0x7fd3e002e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31eba5ae83e6e672 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:07.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.414+0000 7fd3f67fc640 1 -- 172.21.15.83:0/2744543887 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e003b070 con 0x7fd3f80b3060 2024-03-06T07:24:07.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.414+0000 7fd3f67fc640 1 -- 172.21.15.83:0/2744543887 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3e002ee00 con 0x7fd3f80b3060 2024-03-06T07:24:07.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.414+0000 7fd3f67fc640 1 -- 172.21.15.83:0/2744543887 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e0036660 con 0x7fd3f80b3060 2024-03-06T07:24:07.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.414+0000 7fd3fd51e640 1 -- 172.21.15.83:0/2744543887 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd3f80b3060 msgr2=0x7fd3f815e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.414+0000 7fd3fd51e640 1 --2- 172.21.15.83:0/2744543887 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd3f80b3060 0x7fd3f815e850 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fd3f80b4e00 tx=0x7fd3e002e050 comp rx=0 tx=0).stop 2024-03-06T07:24:07.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.415+0000 7fd3fd51e640 1 -- 172.21.15.83:0/2744543887 shutdown_connections 2024-03-06T07:24:07.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.415+0000 7fd3fd51e640 1 --2- 172.21.15.83:0/2744543887 >> v2:172.21.15.183:3300/0 conn(0x7fd3f815bf30 0x7fd3f815ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.415+0000 7fd3fd51e640 1 --2- 172.21.15.83:0/2744543887 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd3f80b3060 0x7fd3f815e850 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.415+0000 7fd3fd51e640 1 -- 172.21.15.83:0/2744543887 >> 172.21.15.83:0/2744543887 conn(0x7fd3f80cf8a0 msgr2=0x7fd3f80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:07.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.415+0000 7fd3fd51e640 1 -- 172.21.15.83:0/2744543887 shutdown_connections 2024-03-06T07:24:07.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.415+0000 7fd3fd51e640 1 -- 172.21.15.83:0/2744543887 wait complete. 2024-03-06T07:24:07.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.416+0000 7fd3fd51e640 1 Processor -- start 2024-03-06T07:24:07.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.416+0000 7fd3fd51e640 1 -- start start 2024-03-06T07:24:07.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.416+0000 7fd3fd51e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd3f80b3060 0x7fd3f806b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.416+0000 7fd3fd51e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd3f813dc10 0x7fd3f8154e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.416+0000 7fd3fd51e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd3f815bf30 0x7fd3f8137dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.416+0000 7fd3fd51e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f813dff0 con 0x7fd3f815bf30 2024-03-06T07:24:07.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.416+0000 7fd3fd51e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f815b730 con 0x7fd3f80b3060 2024-03-06T07:24:07.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.416+0000 7fd3fd51e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f80cd7a0 con 0x7fd3f813dc10 2024-03-06T07:24:07.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.417+0000 7fd3f77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd3f815bf30 0x7fd3f8137dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.417+0000 7fd3f77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd3f815bf30 0x7fd3f8137dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33336/0 (socket says 172.21.15.83:33336) 2024-03-06T07:24:07.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.417+0000 7fd3f77fe640 1 -- 172.21.15.83:0/3706141489 learned_addr learned my addr 172.21.15.83:0/3706141489 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:07.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.417+0000 7fd3f6ffd640 1 --2- 172.21.15.83:0/3706141489 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd3f80b3060 0x7fd3f806b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.417+0000 7fd3effff640 1 --2- 172.21.15.83:0/3706141489 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd3f813dc10 0x7fd3f8154e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.417+0000 7fd3f77fe640 1 -- 172.21.15.83:0/3706141489 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd3f813dc10 msgr2=0x7fd3f8154e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.417+0000 7fd3f77fe640 1 --2- 172.21.15.83:0/3706141489 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd3f813dc10 0x7fd3f8154e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.417+0000 7fd3f77fe640 1 -- 172.21.15.83:0/3706141489 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd3f80b3060 msgr2=0x7fd3f806b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.417+0000 7fd3f77fe640 1 --2- 172.21.15.83:0/3706141489 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd3f80b3060 0x7fd3f806b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.417+0000 7fd3f77fe640 1 -- 172.21.15.83:0/3706141489 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f806bda0 con 0x7fd3f815bf30 2024-03-06T07:24:07.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.417+0000 7fd3effff640 1 --2- 172.21.15.83:0/3706141489 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd3f813dc10 0x7fd3f8154e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:07.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.417+0000 7fd3f77fe640 1 --2- 172.21.15.83:0/3706141489 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd3f815bf30 0x7fd3f8137dc0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fd3f000cd50 tx=0x7fd3f000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:07.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.417+0000 7fd3ef7fe640 1 -- 172.21.15.83:0/3706141489 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f000f680 con 0x7fd3f815bf30 2024-03-06T07:24:07.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.417+0000 7fd3ef7fe640 1 -- 172.21.15.83:0/3706141489 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3f000f840 con 0x7fd3f815bf30 2024-03-06T07:24:07.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.417+0000 7fd3fd51e640 1 -- 172.21.15.83:0/3706141489 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3f8152e30 con 0x7fd3f815bf30 2024-03-06T07:24:07.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.417+0000 7fd3ef7fe640 1 -- 172.21.15.83:0/3706141489 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f000fc30 con 0x7fd3f815bf30 2024-03-06T07:24:07.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.417+0000 7fd3fd51e640 1 -- 172.21.15.83:0/3706141489 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3f81533f0 con 0x7fd3f815bf30 2024-03-06T07:24:07.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.418+0000 7fd3fd51e640 1 -- 172.21.15.83:0/3706141489 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3f80682b0 con 0x7fd3f815bf30 2024-03-06T07:24:07.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.419+0000 7fd3ef7fe640 1 -- 172.21.15.83:0/3706141489 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd3f00213f0 con 0x7fd3f815bf30 2024-03-06T07:24:07.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.419+0000 7fd3ef7fe640 1 --2- 172.21.15.83:0/3706141489 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd3d806b4d0 0x7fd3d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.419+0000 7fd3ef7fe640 1 -- 172.21.15.83:0/3706141489 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd3f008ae60 con 0x7fd3f815bf30 2024-03-06T07:24:07.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.419+0000 7fd3f6ffd640 1 --2- 172.21.15.83:0/3706141489 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd3d806b4d0 0x7fd3d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.420+0000 7fd3f6ffd640 1 --2- 172.21.15.83:0/3706141489 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd3d806b4d0 0x7fd3d806d990 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fd3e8004f60 tx=0x7fd3e8007e30 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:07.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.424+0000 7fd3ef7fe640 1 -- 172.21.15.83:0/3706141489 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd3f80682b0 con 0x7fd3f815bf30 2024-03-06T07:24:07.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.760+0000 7fd3fd51e640 1 -- 172.21.15.83:0/3706141489 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3f804f330 con 0x7fd3f815bf30 2024-03-06T07:24:07.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.761+0000 7fd3ef7fe640 1 -- 172.21.15.83:0/3706141489 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+4980 (secure 0 0 0) 0x7fd3f0014030 con 0x7fd3f815bf30 2024-03-06T07:24:07.763 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:24:07.763 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":19,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5395,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/2270670914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2270670914},{"type":"v1","addr":"172.21.15.83:6837","nonce":2270670914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5398,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/2359880029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2359880029},{"type":"v1","addr":"172.21.15.183:6834","nonce":2359880029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5401,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3099809273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":3099809273},{"type":"v1","addr":"172.21.15.183:6835","nonce":3099809273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:41.430159+0000","modified":"2024-03-06T07:23:43.445803+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5706},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5706":{"gid":5706,"name":"a","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6835/534553538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":534553538},{"type":"v1","addr":"172.21.15.83:6835","nonce":534553538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5706,"qdb_cluster":[5706]},"id":3}]} 2024-03-06T07:24:07.763 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 19 2024-03-06T07:24:07.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.763+0000 7fd3fd51e640 1 -- 172.21.15.83:0/3706141489 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd3d806b4d0 msgr2=0x7fd3d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.763+0000 7fd3fd51e640 1 --2- 172.21.15.83:0/3706141489 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd3d806b4d0 0x7fd3d806d990 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fd3e8004f60 tx=0x7fd3e8007e30 comp rx=0 tx=0).stop 2024-03-06T07:24:07.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.763+0000 7fd3fd51e640 1 -- 172.21.15.83:0/3706141489 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd3f815bf30 msgr2=0x7fd3f8137dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.763+0000 7fd3fd51e640 1 --2- 172.21.15.83:0/3706141489 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd3f815bf30 0x7fd3f8137dc0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fd3f000cd50 tx=0x7fd3f000a650 comp rx=0 tx=0).stop 2024-03-06T07:24:07.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.763+0000 7fd3fd51e640 1 -- 172.21.15.83:0/3706141489 shutdown_connections 2024-03-06T07:24:07.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.763+0000 7fd3fd51e640 1 --2- 172.21.15.83:0/3706141489 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd3d806b4d0 0x7fd3d806d990 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.763+0000 7fd3fd51e640 1 --2- 172.21.15.83:0/3706141489 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd3f815bf30 0x7fd3f8137dc0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.763+0000 7fd3fd51e640 1 --2- 172.21.15.83:0/3706141489 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd3f813dc10 0x7fd3f8154e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.763+0000 7fd3fd51e640 1 --2- 172.21.15.83:0/3706141489 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd3f80b3060 0x7fd3f806b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.763+0000 7fd3fd51e640 1 -- 172.21.15.83:0/3706141489 >> 172.21.15.83:0/3706141489 conn(0x7fd3f80cf8a0 msgr2=0x7fd3f813afa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:07.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.763+0000 7fd3fd51e640 1 -- 172.21.15.83:0/3706141489 shutdown_connections 2024-03-06T07:24:07.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.763+0000 7fd3fd51e640 1 -- 172.21.15.83:0/3706141489 wait complete. 2024-03-06T07:24:07.779 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:07.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.878+0000 7f3fd47e2640 1 Processor -- start 2024-03-06T07:24:07.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.878+0000 7f3fd47e2640 1 -- start start 2024-03-06T07:24:07.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.879+0000 7f3fd47e2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3fcc15bb20 0x7f3fcc0d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.879+0000 7f3fd47e2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f3fcc0cfff0 0x7f3fcc0d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.879+0000 7f3fd47e2640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fcc161980 con 0x7f3fcc0ce4b0 2024-03-06T07:24:07.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.879+0000 7f3fd47e2640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fcc161c60 con 0x7f3fcc0cfff0 2024-03-06T07:24:07.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.879+0000 7f3fd47e2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fcc161f40 con 0x7f3fcc15bb20 2024-03-06T07:24:07.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.879+0000 7f3fd2d7f640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f3fcc0ce4b0 0x7f3fcc0ce940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48482/0 (socket says 172.21.15.83:48482) 2024-03-06T07:24:07.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.879+0000 7f3fd2d7f640 1 -- 172.21.15.83:0/1722055511 learned_addr learned my addr 172.21.15.83:0/1722055511 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:24:07.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.879+0000 7f3fd1d7d640 1 --2- 172.21.15.83:0/1722055511 >> v2:172.21.15.83:3300/0 conn(0x7f3fcc0cfff0 0x7f3fcc0d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.879+0000 7f3fd257e640 1 --2- 172.21.15.83:0/1722055511 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3fcc15bb20 0x7f3fcc0d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.879+0000 7f3fd157c640 1 -- 172.21.15.83:0/1722055511 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 914014189 0 0) 0x7f3fcc161980 con 0x7f3fcc0ce4b0 2024-03-06T07:24:07.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.879+0000 7f3fd157c640 1 -- 172.21.15.83:0/1722055511 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fb8003520 con 0x7f3fcc0ce4b0 2024-03-06T07:24:07.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.879+0000 7f3fd1d7d640 1 -- 172.21.15.83:0/1722055511 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3fcc15bb20 msgr2=0x7f3fcc0d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:07.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.879+0000 7f3fd1d7d640 1 --2- 172.21.15.83:0/1722055511 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3fcc15bb20 0x7f3fcc0d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.879+0000 7f3fd1d7d640 1 -- 172.21.15.83:0/1722055511 >> v1:172.21.15.83:6789/0 conn(0x7f3fcc0ce4b0 legacy=0x7f3fcc0ce940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.879+0000 7f3fd1d7d640 1 -- 172.21.15.83:0/1722055511 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fcc162220 con 0x7f3fcc0cfff0 2024-03-06T07:24:07.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.880+0000 7f3fd1d7d640 1 --2- 172.21.15.83:0/1722055511 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3fcc0cfff0 0x7f3fcc0d0470 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f3fc80097a0 tx=0x7f3fc802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc048550e98a6e96 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:07.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.880+0000 7f3fd157c640 1 -- 172.21.15.83:0/1722055511 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc8035020 con 0x7f3fcc0cfff0 2024-03-06T07:24:07.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.880+0000 7f3fd157c640 1 -- 172.21.15.83:0/1722055511 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fc8031830 con 0x7f3fcc0cfff0 2024-03-06T07:24:07.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.880+0000 7f3fd157c640 1 -- 172.21.15.83:0/1722055511 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc8031c00 con 0x7f3fcc0cfff0 2024-03-06T07:24:07.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.881+0000 7f3fd47e2640 1 -- 172.21.15.83:0/1722055511 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3fcc0cfff0 msgr2=0x7f3fcc0d0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.881+0000 7f3fd47e2640 1 --2- 172.21.15.83:0/1722055511 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3fcc0cfff0 0x7f3fcc0d0470 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f3fc80097a0 tx=0x7f3fc802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.881+0000 7f3fd47e2640 1 -- 172.21.15.83:0/1722055511 shutdown_connections 2024-03-06T07:24:07.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.881+0000 7f3fd47e2640 1 --2- 172.21.15.83:0/1722055511 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3fcc0cfff0 0x7f3fcc0d0470 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.881+0000 7f3fd47e2640 1 --2- 172.21.15.83:0/1722055511 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3fcc15bb20 0x7f3fcc0d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.881+0000 7f3fd47e2640 1 -- 172.21.15.83:0/1722055511 >> 172.21.15.83:0/1722055511 conn(0x7f3fcc09a9f0 msgr2=0x7f3fcc09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:07.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.881+0000 7f3fd47e2640 1 -- 172.21.15.83:0/1722055511 shutdown_connections 2024-03-06T07:24:07.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.881+0000 7f3fd47e2640 1 -- 172.21.15.83:0/1722055511 wait complete. 2024-03-06T07:24:07.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.882+0000 7f3fd47e2640 1 Processor -- start 2024-03-06T07:24:07.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.882+0000 7f3fd47e2640 1 -- start start 2024-03-06T07:24:07.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.883+0000 7f3fd47e2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3fcc0ce4b0 0x7f3fcc154910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.883+0000 7f3fd47e2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3fcc0cfff0 0x7f3fcc160eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.883+0000 7f3fd47e2640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3fcc15bb20 0x7f3fcc06fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.883+0000 7f3fd47e2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fcc0d24b0 con 0x7f3fcc0cfff0 2024-03-06T07:24:07.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.883+0000 7f3fd47e2640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fcc0d2330 con 0x7f3fcc15bb20 2024-03-06T07:24:07.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.883+0000 7f3fd47e2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fcc0d1980 con 0x7f3fcc0ce4b0 2024-03-06T07:24:07.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.883+0000 7f3fd1d7d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3fcc0cfff0 0x7f3fcc160eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.883+0000 7f3fd1d7d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3fcc0cfff0 0x7f3fcc160eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33348/0 (socket says 172.21.15.83:33348) 2024-03-06T07:24:07.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.883+0000 7f3fd1d7d640 1 -- 172.21.15.83:0/1433594961 learned_addr learned my addr 172.21.15.83:0/1433594961 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:07.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.883+0000 7f3fd257e640 1 --2- 172.21.15.83:0/1433594961 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3fcc0ce4b0 0x7f3fcc154910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.883+0000 7f3fd2d7f640 1 --2- 172.21.15.83:0/1433594961 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3fcc15bb20 0x7f3fcc06fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.883+0000 7f3fd1d7d640 1 -- 172.21.15.83:0/1433594961 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3fcc0ce4b0 msgr2=0x7f3fcc154910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.883+0000 7f3fd1d7d640 1 --2- 172.21.15.83:0/1433594961 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3fcc0ce4b0 0x7f3fcc154910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.883+0000 7f3fd1d7d640 1 -- 172.21.15.83:0/1433594961 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3fcc15bb20 msgr2=0x7f3fcc06fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.883+0000 7f3fd1d7d640 1 --2- 172.21.15.83:0/1433594961 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3fcc15bb20 0x7f3fcc06fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.883+0000 7f3fd1d7d640 1 -- 172.21.15.83:0/1433594961 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fcc0c5920 con 0x7f3fcc0cfff0 2024-03-06T07:24:07.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.883+0000 7f3fd2d7f640 1 --2- 172.21.15.83:0/1433594961 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3fcc15bb20 0x7f3fcc06fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:07.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.884+0000 7f3fd1d7d640 1 --2- 172.21.15.83:0/1433594961 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3fcc0cfff0 0x7f3fcc160eb0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f3fc802bee0 tx=0x7f3fc80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:07.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.884+0000 7f3fc2ffd640 1 -- 172.21.15.83:0/1433594961 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc8035040 con 0x7f3fcc0cfff0 2024-03-06T07:24:07.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.884+0000 7f3fc2ffd640 1 -- 172.21.15.83:0/1433594961 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fc803b920 con 0x7f3fcc0cfff0 2024-03-06T07:24:07.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.884+0000 7f3fd47e2640 1 -- 172.21.15.83:0/1433594961 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fcc0c5c00 con 0x7f3fcc0cfff0 2024-03-06T07:24:07.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.884+0000 7f3fc2ffd640 1 -- 172.21.15.83:0/1433594961 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc803bd10 con 0x7f3fcc0cfff0 2024-03-06T07:24:07.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.884+0000 7f3fd47e2640 1 -- 172.21.15.83:0/1433594961 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fcc0c61c0 con 0x7f3fcc0cfff0 2024-03-06T07:24:07.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.885+0000 7f3fd47e2640 1 -- 172.21.15.83:0/1433594961 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fcc04f330 con 0x7f3fcc0cfff0 2024-03-06T07:24:07.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.885+0000 7f3fc2ffd640 1 -- 172.21.15.83:0/1433594961 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3fc8038020 con 0x7f3fcc0cfff0 2024-03-06T07:24:07.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.886+0000 7f3fc2ffd640 1 --2- 172.21.15.83:0/1433594961 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3fb006b4d0 0x7f3fb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.886+0000 7f3fd257e640 1 --2- 172.21.15.83:0/1433594961 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3fb006b4d0 0x7f3fb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.886+0000 7f3fc2ffd640 1 -- 172.21.15.83:0/1433594961 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3fc80ac290 con 0x7f3fcc0cfff0 2024-03-06T07:24:07.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.886+0000 7f3fd257e640 1 --2- 172.21.15.83:0/1433594961 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3fb006b4d0 0x7f3fb006d990 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc009a30 tx=0x7f3fbc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:07.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:07.890+0000 7f3fc2ffd640 1 -- 172.21.15.83:0/1433594961 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3fc807b1a0 con 0x7f3fcc0cfff0 2024-03-06T07:24:08.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.063+0000 7f3fd47e2640 1 -- 172.21.15.83:0/1433594961 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3fcc0c65a0 con 0x7f3fcc0cfff0 2024-03-06T07:24:08.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.064+0000 7f3fc2ffd640 1 -- 172.21.15.83:0/1433594961 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+16625 (secure 0 0 0) 0x7f3fc80a9110 con 0x7f3fcc0cfff0 2024-03-06T07:24:08.066 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:24:08.066 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":45,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:23:43.092917+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:39.656617+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:40.402765+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:08.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.066+0000 7f3fd47e2640 1 -- 172.21.15.83:0/1433594961 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3fb006b4d0 msgr2=0x7f3fb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.066+0000 7f3fd47e2640 1 --2- 172.21.15.83:0/1433594961 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3fb006b4d0 0x7f3fb006d990 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc009a30 tx=0x7f3fbc008040 comp rx=0 tx=0).stop 2024-03-06T07:24:08.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.066+0000 7f3fd47e2640 1 -- 172.21.15.83:0/1433594961 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3fcc0cfff0 msgr2=0x7f3fcc160eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.066+0000 7f3fd47e2640 1 --2- 172.21.15.83:0/1433594961 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3fcc0cfff0 0x7f3fcc160eb0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f3fc802bee0 tx=0x7f3fc80046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.066+0000 7f3fd47e2640 1 -- 172.21.15.83:0/1433594961 shutdown_connections 2024-03-06T07:24:08.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.066+0000 7f3fd47e2640 1 --2- 172.21.15.83:0/1433594961 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3fb006b4d0 0x7f3fb006d990 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.066+0000 7f3fd47e2640 1 --2- 172.21.15.83:0/1433594961 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3fcc15bb20 0x7f3fcc06fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.066+0000 7f3fd47e2640 1 --2- 172.21.15.83:0/1433594961 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3fcc0cfff0 0x7f3fcc160eb0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.066+0000 7f3fd47e2640 1 --2- 172.21.15.83:0/1433594961 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3fcc0ce4b0 0x7f3fcc154910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.066+0000 7f3fd47e2640 1 -- 172.21.15.83:0/1433594961 >> 172.21.15.83:0/1433594961 conn(0x7f3fcc09a9f0 msgr2=0x7f3fcc0b09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:08.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.066+0000 7f3fd47e2640 1 -- 172.21.15.83:0/1433594961 shutdown_connections 2024-03-06T07:24:08.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.066+0000 7f3fd47e2640 1 -- 172.21.15.83:0/1433594961 wait complete. 2024-03-06T07:24:08.080 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:24:08.080 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:08.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.179+0000 7f0c5dc64640 1 Processor -- start 2024-03-06T07:24:08.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.179+0000 7f0c5dc64640 1 -- start start 2024-03-06T07:24:08.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.180+0000 7f0c5dc64640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0c580d22b0 0x7f0c580d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.180+0000 7f0c5dc64640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0c580b41f0 0x7f0c580b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.180+0000 7f0c5dc64640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c580b3430 con 0x7f0c5815dfe0 2024-03-06T07:24:08.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.180+0000 7f0c5dc64640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c580d2c50 con 0x7f0c580d22b0 2024-03-06T07:24:08.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.180+0000 7f0c5dc64640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c580b31e0 con 0x7f0c580b41f0 2024-03-06T07:24:08.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.180+0000 7f0c56ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0c580d22b0 0x7f0c580d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.180+0000 7f0c56ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0c580d22b0 0x7f0c580d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33356/0 (socket says 172.21.15.83:33356) 2024-03-06T07:24:08.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.180+0000 7f0c56ffd640 1 -- 172.21.15.83:0/864369548 learned_addr learned my addr 172.21.15.83:0/864369548 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:08.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.181+0000 7f0c57fff640 1 --2- 172.21.15.83:0/864369548 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0c580b41f0 0x7f0c580b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.181+0000 7f0c56ffd640 1 -- 172.21.15.83:0/864369548 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0c580b41f0 msgr2=0x7f0c580b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:08.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.181+0000 7f0c56ffd640 1 --2- 172.21.15.83:0/864369548 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0c580b41f0 0x7f0c580b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.181+0000 7f0c56ffd640 1 -- 172.21.15.83:0/864369548 >> v1:172.21.15.183:6789/0 conn(0x7f0c5815dfe0 legacy=0x7f0c580cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:08.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.181+0000 7f0c56ffd640 1 -- 172.21.15.83:0/864369548 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c580b3710 con 0x7f0c580d22b0 2024-03-06T07:24:08.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.181+0000 7f0c56ffd640 1 --2- 172.21.15.83:0/864369548 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c580d22b0 0x7f0c580d2710 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c009870 tx=0x7f0c4c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa7a2191467da2f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.182+0000 7f0c567fc640 1 -- 172.21.15.83:0/864369548 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c4c035020 con 0x7f0c580d22b0 2024-03-06T07:24:08.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.182+0000 7f0c567fc640 1 -- 172.21.15.83:0/864369548 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c4c031640 con 0x7f0c580d22b0 2024-03-06T07:24:08.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.182+0000 7f0c567fc640 1 -- 172.21.15.83:0/864369548 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c4c02d040 con 0x7f0c580d22b0 2024-03-06T07:24:08.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.182+0000 7f0c5dc64640 1 -- 172.21.15.83:0/864369548 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c580d22b0 msgr2=0x7f0c580d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.182+0000 7f0c5dc64640 1 --2- 172.21.15.83:0/864369548 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c580d22b0 0x7f0c580d2710 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c009870 tx=0x7f0c4c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.182+0000 7f0c5dc64640 1 -- 172.21.15.83:0/864369548 shutdown_connections 2024-03-06T07:24:08.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.182+0000 7f0c5dc64640 1 --2- 172.21.15.83:0/864369548 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0c580b41f0 0x7f0c580b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.182+0000 7f0c5dc64640 1 --2- 172.21.15.83:0/864369548 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c580d22b0 0x7f0c580d2710 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.182+0000 7f0c5dc64640 1 -- 172.21.15.83:0/864369548 >> 172.21.15.83:0/864369548 conn(0x7f0c580d3cd0 msgr2=0x7f0c580d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:08.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.182+0000 7f0c5dc64640 1 -- 172.21.15.83:0/864369548 shutdown_connections 2024-03-06T07:24:08.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.182+0000 7f0c5dc64640 1 -- 172.21.15.83:0/864369548 wait complete. 2024-03-06T07:24:08.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.184+0000 7f0c5dc64640 1 Processor -- start 2024-03-06T07:24:08.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.184+0000 7f0c5dc64640 1 -- start start 2024-03-06T07:24:08.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.185+0000 7f0c5dc64640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0c580b41f0 0x7f0c5815ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.185+0000 7f0c5dc64640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c580d22b0 0x7f0c58147ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.185+0000 7f0c5dc64640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c5815dfe0 0x7f0c5814a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.185+0000 7f0c5dc64640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c580cfa00 con 0x7f0c5815dfe0 2024-03-06T07:24:08.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.185+0000 7f0c5dc64640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c580b3d60 con 0x7f0c580d22b0 2024-03-06T07:24:08.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.185+0000 7f0c5dc64640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c580d1c70 con 0x7f0c580b41f0 2024-03-06T07:24:08.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.185+0000 7f0c57fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c5815dfe0 0x7f0c5814a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.185+0000 7f0c57fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c5815dfe0 0x7f0c5814a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33358/0 (socket says 172.21.15.83:33358) 2024-03-06T07:24:08.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.185+0000 7f0c57fff640 1 -- 172.21.15.83:0/1132430860 learned_addr learned my addr 172.21.15.83:0/1132430860 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:08.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.185+0000 7f0c577fe640 1 --2- 172.21.15.83:0/1132430860 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0c580b41f0 0x7f0c5815ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.185+0000 7f0c56ffd640 1 --2- 172.21.15.83:0/1132430860 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c580d22b0 0x7f0c58147ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.186+0000 7f0c57fff640 1 -- 172.21.15.83:0/1132430860 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0c580b41f0 msgr2=0x7f0c5815ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.186+0000 7f0c57fff640 1 --2- 172.21.15.83:0/1132430860 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0c580b41f0 0x7f0c5815ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.186+0000 7f0c57fff640 1 -- 172.21.15.83:0/1132430860 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c580d22b0 msgr2=0x7f0c58147ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.186+0000 7f0c57fff640 1 --2- 172.21.15.83:0/1132430860 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c580d22b0 0x7f0c58147ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.186+0000 7f0c57fff640 1 -- 172.21.15.83:0/1132430860 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c58148430 con 0x7f0c5815dfe0 2024-03-06T07:24:08.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.186+0000 7f0c57fff640 1 --2- 172.21.15.83:0/1132430860 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c5815dfe0 0x7f0c5814a110 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f0c50007e70 tx=0x7f0c5000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.186+0000 7f0c3ffff640 1 -- 172.21.15.83:0/1132430860 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c50014620 con 0x7f0c5815dfe0 2024-03-06T07:24:08.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.186+0000 7f0c3ffff640 1 -- 172.21.15.83:0/1132430860 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c50019070 con 0x7f0c5815dfe0 2024-03-06T07:24:08.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.186+0000 7f0c5dc64640 1 -- 172.21.15.83:0/1132430860 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c580b9b90 con 0x7f0c5815dfe0 2024-03-06T07:24:08.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.186+0000 7f0c3ffff640 1 -- 172.21.15.83:0/1132430860 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c50014a20 con 0x7f0c5815dfe0 2024-03-06T07:24:08.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.186+0000 7f0c5dc64640 1 -- 172.21.15.83:0/1132430860 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c580b6c20 con 0x7f0c5815dfe0 2024-03-06T07:24:08.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.187+0000 7f0c5dc64640 1 -- 172.21.15.83:0/1132430860 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c5804f330 con 0x7f0c5815dfe0 2024-03-06T07:24:08.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.187+0000 7f0c3ffff640 1 -- 172.21.15.83:0/1132430860 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0c50014bc0 con 0x7f0c5815dfe0 2024-03-06T07:24:08.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.188+0000 7f0c3ffff640 1 --2- 172.21.15.83:0/1132430860 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0c4406b4d0 0x7f0c4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.188+0000 7f0c3ffff640 1 -- 172.21.15.83:0/1132430860 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0c5008a9c0 con 0x7f0c5815dfe0 2024-03-06T07:24:08.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.188+0000 7f0c577fe640 1 --2- 172.21.15.83:0/1132430860 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0c4406b4d0 0x7f0c4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.188+0000 7f0c577fe640 1 --2- 172.21.15.83:0/1132430860 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0c4406b4d0 0x7f0c4406d990 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f0c48006fd0 tx=0x7f0c48009060 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.192+0000 7f0c3ffff640 1 -- 172.21.15.83:0/1132430860 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0c5008ce20 con 0x7f0c5815dfe0 2024-03-06T07:24:08.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.409+0000 7f0c5dc64640 1 -- 172.21.15.83:0/1132430860 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c580cea80 con 0x7f0c5815dfe0 2024-03-06T07:24:08.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.410+0000 7f0c3ffff640 1 -- 172.21.15.83:0/1132430860 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f0c5005f070 con 0x7f0c5815dfe0 2024-03-06T07:24:08.412 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:24:08.412 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":19,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5395,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/2270670914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2270670914},{"type":"v1","addr":"172.21.15.83:6837","nonce":2270670914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5398,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/2359880029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2359880029},{"type":"v1","addr":"172.21.15.183:6834","nonce":2359880029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5401,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3099809273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":3099809273},{"type":"v1","addr":"172.21.15.183:6835","nonce":3099809273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:41.430159+0000","modified":"2024-03-06T07:23:43.445803+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5706},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5706":{"gid":5706,"name":"a","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6835/534553538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":534553538},{"type":"v1","addr":"172.21.15.83:6835","nonce":534553538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5706,"qdb_cluster":[5706]},"id":3}]} 2024-03-06T07:24:08.412 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 19 2024-03-06T07:24:08.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.412+0000 7f0c5dc64640 1 -- 172.21.15.83:0/1132430860 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0c4406b4d0 msgr2=0x7f0c4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.412+0000 7f0c5dc64640 1 --2- 172.21.15.83:0/1132430860 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0c4406b4d0 0x7f0c4406d990 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f0c48006fd0 tx=0x7f0c48009060 comp rx=0 tx=0).stop 2024-03-06T07:24:08.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.412+0000 7f0c5dc64640 1 -- 172.21.15.83:0/1132430860 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c5815dfe0 msgr2=0x7f0c5814a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.412+0000 7f0c5dc64640 1 --2- 172.21.15.83:0/1132430860 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c5815dfe0 0x7f0c5814a110 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f0c50007e70 tx=0x7f0c5000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.412+0000 7f0c5dc64640 1 -- 172.21.15.83:0/1132430860 shutdown_connections 2024-03-06T07:24:08.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.412+0000 7f0c5dc64640 1 --2- 172.21.15.83:0/1132430860 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0c4406b4d0 0x7f0c4406d990 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.412+0000 7f0c5dc64640 1 --2- 172.21.15.83:0/1132430860 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c5815dfe0 0x7f0c5814a110 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.412+0000 7f0c5dc64640 1 --2- 172.21.15.83:0/1132430860 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c580d22b0 0x7f0c58147ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.412+0000 7f0c5dc64640 1 --2- 172.21.15.83:0/1132430860 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0c580b41f0 0x7f0c5815ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.412+0000 7f0c5dc64640 1 -- 172.21.15.83:0/1132430860 >> 172.21.15.83:0/1132430860 conn(0x7f0c580d3cd0 msgr2=0x7f0c580b4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:08.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.412+0000 7f0c5dc64640 1 -- 172.21.15.83:0/1132430860 shutdown_connections 2024-03-06T07:24:08.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.412+0000 7f0c5dc64640 1 -- 172.21.15.83:0/1132430860 wait complete. 2024-03-06T07:24:08.425 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:08.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.524+0000 7f1d832c9640 1 Processor -- start 2024-03-06T07:24:08.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.524+0000 7f1d832c9640 1 -- start start 2024-03-06T07:24:08.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.525+0000 7f1d832c9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1d7c15dfe0 0x7f1d7c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.525+0000 7f1d832c9640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f1d7c0cf830 0x7f1d7c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.525+0000 7f1d832c9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d7c0b3430 con 0x7f1d7c0d27f0 2024-03-06T07:24:08.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.525+0000 7f1d832c9640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d7c0b31e0 con 0x7f1d7c15dfe0 2024-03-06T07:24:08.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.525+0000 7f1d832c9640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d7c0b3d60 con 0x7f1d7c0cf830 2024-03-06T07:24:08.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.525+0000 7f1d81065640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1d7c15dfe0 0x7f1d7c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.525+0000 7f1d81065640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1d7c15dfe0 0x7f1d7c0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33374/0 (socket says 172.21.15.83:33374) 2024-03-06T07:24:08.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.525+0000 7f1d81065640 1 -- 172.21.15.83:0/2329750858 learned_addr learned my addr 172.21.15.83:0/2329750858 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:08.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.525+0000 7f1d80864640 1 --2- 172.21.15.83:0/2329750858 >> v2:172.21.15.183:3300/0 conn(0x7f1d7c0cf830 0x7f1d7c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.525+0000 7f1d81065640 1 -- 172.21.15.83:0/2329750858 >> v2:172.21.15.183:3300/0 conn(0x7f1d7c0cf830 msgr2=0x7f1d7c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:08.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.525+0000 7f1d81065640 1 --2- 172.21.15.83:0/2329750858 >> v2:172.21.15.183:3300/0 conn(0x7f1d7c0cf830 0x7f1d7c0d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.525+0000 7f1d81065640 1 -- 172.21.15.83:0/2329750858 >> v1:172.21.15.183:6789/0 conn(0x7f1d7c0d27f0 legacy=0x7f1d7c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:08.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.525+0000 7f1d81065640 1 -- 172.21.15.83:0/2329750858 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d7c0b3710 con 0x7f1d7c15dfe0 2024-03-06T07:24:08.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.526+0000 7f1d81065640 1 --2- 172.21.15.83:0/2329750858 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d7c15dfe0 0x7f1d7c0cf2f0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f1d78009870 tx=0x7f1d7802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aca29af71c4feeab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.526+0000 7f1d73fff640 1 -- 172.21.15.83:0/2329750858 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d78035020 con 0x7f1d7c15dfe0 2024-03-06T07:24:08.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.526+0000 7f1d73fff640 1 -- 172.21.15.83:0/2329750858 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d78031870 con 0x7f1d7c15dfe0 2024-03-06T07:24:08.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.526+0000 7f1d73fff640 1 -- 172.21.15.83:0/2329750858 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d78031c40 con 0x7f1d7c15dfe0 2024-03-06T07:24:08.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.527+0000 7f1d832c9640 1 -- 172.21.15.83:0/2329750858 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d7c15dfe0 msgr2=0x7f1d7c0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.527+0000 7f1d832c9640 1 --2- 172.21.15.83:0/2329750858 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d7c15dfe0 0x7f1d7c0cf2f0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f1d78009870 tx=0x7f1d7802b910 comp rx=0 tx=0).stop 2024-03-06T07:24:08.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.527+0000 7f1d832c9640 1 -- 172.21.15.83:0/2329750858 shutdown_connections 2024-03-06T07:24:08.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.527+0000 7f1d832c9640 1 --2- 172.21.15.83:0/2329750858 >> v2:172.21.15.183:3300/0 conn(0x7f1d7c0cf830 0x7f1d7c0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.527+0000 7f1d832c9640 1 --2- 172.21.15.83:0/2329750858 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d7c15dfe0 0x7f1d7c0cf2f0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.527+0000 7f1d832c9640 1 -- 172.21.15.83:0/2329750858 >> 172.21.15.83:0/2329750858 conn(0x7f1d7c0d3cd0 msgr2=0x7f1d7c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:08.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.527+0000 7f1d832c9640 1 -- 172.21.15.83:0/2329750858 shutdown_connections 2024-03-06T07:24:08.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.527+0000 7f1d832c9640 1 -- 172.21.15.83:0/2329750858 wait complete. 2024-03-06T07:24:08.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.528+0000 7f1d832c9640 1 Processor -- start 2024-03-06T07:24:08.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.528+0000 7f1d832c9640 1 -- start start 2024-03-06T07:24:08.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.529+0000 7f1d832c9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1d7c0cf830 0x7f1d7c1610a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.529+0000 7f1d832c9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1d7c0d27f0 0x7f1d7c149f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.529+0000 7f1d832c9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d7c15dfe0 0x7f1d7c14a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.529+0000 7f1d832c9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c0d1df0 con 0x7f1d7c15dfe0 2024-03-06T07:24:08.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.529+0000 7f1d832c9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c0d1c70 con 0x7f1d7c0d27f0 2024-03-06T07:24:08.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.529+0000 7f1d832c9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c15e3c0 con 0x7f1d7c0cf830 2024-03-06T07:24:08.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.529+0000 7f1d81866640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d7c15dfe0 0x7f1d7c14a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.529+0000 7f1d81866640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d7c15dfe0 0x7f1d7c14a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33384/0 (socket says 172.21.15.83:33384) 2024-03-06T07:24:08.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.529+0000 7f1d81866640 1 -- 172.21.15.83:0/867600250 learned_addr learned my addr 172.21.15.83:0/867600250 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:08.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.529+0000 7f1d81866640 1 -- 172.21.15.83:0/867600250 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1d7c0cf830 msgr2=0x7f1d7c1610a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.529+0000 7f1d81866640 1 --2- 172.21.15.83:0/867600250 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1d7c0cf830 0x7f1d7c1610a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.529+0000 7f1d81866640 1 -- 172.21.15.83:0/867600250 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1d7c0d27f0 msgr2=0x7f1d7c149f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.529+0000 7f1d81866640 1 --2- 172.21.15.83:0/867600250 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1d7c0d27f0 0x7f1d7c149f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.529+0000 7f1d81866640 1 -- 172.21.15.83:0/867600250 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d7c0ccbd0 con 0x7f1d7c15dfe0 2024-03-06T07:24:08.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.529+0000 7f1d81866640 1 --2- 172.21.15.83:0/867600250 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d7c15dfe0 0x7f1d7c14a480 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f1d7400c7b0 tx=0x7f1d7400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.530+0000 7f1d71ffb640 1 -- 172.21.15.83:0/867600250 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d74004b60 con 0x7f1d7c15dfe0 2024-03-06T07:24:08.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.530+0000 7f1d71ffb640 1 -- 172.21.15.83:0/867600250 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d74004d00 con 0x7f1d7c15dfe0 2024-03-06T07:24:08.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.530+0000 7f1d832c9640 1 -- 172.21.15.83:0/867600250 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d7c0cceb0 con 0x7f1d7c15dfe0 2024-03-06T07:24:08.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.530+0000 7f1d71ffb640 1 -- 172.21.15.83:0/867600250 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d7400aa90 con 0x7f1d7c15dfe0 2024-03-06T07:24:08.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.530+0000 7f1d832c9640 1 -- 172.21.15.83:0/867600250 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d7c0cd470 con 0x7f1d7c15dfe0 2024-03-06T07:24:08.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.531+0000 7f1d832c9640 1 -- 172.21.15.83:0/867600250 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d7c04f330 con 0x7f1d7c15dfe0 2024-03-06T07:24:08.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.531+0000 7f1d71ffb640 1 -- 172.21.15.83:0/867600250 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1d7400ac30 con 0x7f1d7c15dfe0 2024-03-06T07:24:08.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.532+0000 7f1d71ffb640 1 --2- 172.21.15.83:0/867600250 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1d6006b4d0 0x7f1d6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.532+0000 7f1d81065640 1 --2- 172.21.15.83:0/867600250 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1d6006b4d0 0x7f1d6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.532+0000 7f1d71ffb640 1 -- 172.21.15.83:0/867600250 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1d7408a440 con 0x7f1d7c15dfe0 2024-03-06T07:24:08.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.532+0000 7f1d81065640 1 --2- 172.21.15.83:0/867600250 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1d6006b4d0 0x7f1d6006d990 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1d78036000 tx=0x7f1d78030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.536+0000 7f1d71ffb640 1 -- 172.21.15.83:0/867600250 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1d74020070 con 0x7f1d7c15dfe0 2024-03-06T07:24:08.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.743+0000 7f1d832c9640 1 -- 172.21.15.83:0/867600250 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d7c1615e0 con 0x7f1d7c15dfe0 2024-03-06T07:24:08.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.744+0000 7f1d71ffb640 1 -- 172.21.15.83:0/867600250 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f1d74059350 con 0x7f1d7c15dfe0 2024-03-06T07:24:08.746 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:24:08.746 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":19,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5395,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/2270670914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2270670914},{"type":"v1","addr":"172.21.15.83:6837","nonce":2270670914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5398,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/2359880029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2359880029},{"type":"v1","addr":"172.21.15.183:6834","nonce":2359880029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5401,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3099809273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":3099809273},{"type":"v1","addr":"172.21.15.183:6835","nonce":3099809273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:41.430159+0000","modified":"2024-03-06T07:23:43.445803+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5706},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5706":{"gid":5706,"name":"a","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6835/534553538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":534553538},{"type":"v1","addr":"172.21.15.83:6835","nonce":534553538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5706,"qdb_cluster":[5706]},"id":3}]} 2024-03-06T07:24:08.746 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 19 2024-03-06T07:24:08.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.745+0000 7f1d832c9640 1 -- 172.21.15.83:0/867600250 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1d6006b4d0 msgr2=0x7f1d6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.745+0000 7f1d832c9640 1 --2- 172.21.15.83:0/867600250 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1d6006b4d0 0x7f1d6006d990 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1d78036000 tx=0x7f1d78030000 comp rx=0 tx=0).stop 2024-03-06T07:24:08.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.745+0000 7f1d832c9640 1 -- 172.21.15.83:0/867600250 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d7c15dfe0 msgr2=0x7f1d7c14a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.745+0000 7f1d832c9640 1 --2- 172.21.15.83:0/867600250 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d7c15dfe0 0x7f1d7c14a480 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f1d7400c7b0 tx=0x7f1d7400cc70 comp rx=0 tx=0).stop 2024-03-06T07:24:08.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.745+0000 7f1d832c9640 1 -- 172.21.15.83:0/867600250 shutdown_connections 2024-03-06T07:24:08.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.745+0000 7f1d832c9640 1 --2- 172.21.15.83:0/867600250 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1d6006b4d0 0x7f1d6006d990 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.746+0000 7f1d832c9640 1 --2- 172.21.15.83:0/867600250 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d7c15dfe0 0x7f1d7c14a480 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.746+0000 7f1d832c9640 1 --2- 172.21.15.83:0/867600250 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1d7c0d27f0 0x7f1d7c149f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.746+0000 7f1d832c9640 1 --2- 172.21.15.83:0/867600250 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1d7c0cf830 0x7f1d7c1610a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.746+0000 7f1d832c9640 1 -- 172.21.15.83:0/867600250 >> 172.21.15.83:0/867600250 conn(0x7f1d7c0d3cd0 msgr2=0x7f1d7c0cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:08.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.746+0000 7f1d832c9640 1 -- 172.21.15.83:0/867600250 shutdown_connections 2024-03-06T07:24:08.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.746+0000 7f1d832c9640 1 -- 172.21.15.83:0/867600250 wait complete. 2024-03-06T07:24:08.758 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:08.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.858+0000 7fee11f1e640 1 Processor -- start 2024-03-06T07:24:08.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.858+0000 7fee11f1e640 1 -- start start 2024-03-06T07:24:08.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.858+0000 7fee11f1e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fee0c0cfff0 0x7fee0c0d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.858+0000 7fee11f1e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fee0c0cc8b0 0x7fee0c0ccd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.858+0000 7fee11f1e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0c161980 con 0x7fee0c15bb20 2024-03-06T07:24:08.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.858+0000 7fee11f1e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee0c0d2500 con 0x7fee0c0cfff0 2024-03-06T07:24:08.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.859+0000 7fee11f1e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee0c161c60 con 0x7fee0c0cc8b0 2024-03-06T07:24:08.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.859+0000 7fee0b7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fee0c15bb20 0x7fee0c0d1df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48492/0 (socket says 172.21.15.83:48492) 2024-03-06T07:24:08.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.859+0000 7fee0b7fe640 1 -- 172.21.15.83:0/3010312586 learned_addr learned my addr 172.21.15.83:0/3010312586 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:24:08.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.859+0000 7fee0bfff640 1 --2- 172.21.15.83:0/3010312586 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fee0c0cc8b0 0x7fee0c0ccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.859+0000 7fee0a7fc640 1 -- 172.21.15.83:0/3010312586 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2490663040 0 0) 0x7fee0c161980 con 0x7fee0c15bb20 2024-03-06T07:24:08.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.859+0000 7fee0affd640 1 --2- 172.21.15.83:0/3010312586 >> v2:172.21.15.183:3300/0 conn(0x7fee0c0cfff0 0x7fee0c0d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.859+0000 7fee0a7fc640 1 -- 172.21.15.83:0/3010312586 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedf4003520 con 0x7fee0c15bb20 2024-03-06T07:24:08.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.859+0000 7fee0a7fc640 1 -- 172.21.15.83:0/3010312586 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 183679884 0 0) 0x7fedf4003520 con 0x7fee0c15bb20 2024-03-06T07:24:08.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.859+0000 7fee0a7fc640 1 -- 172.21.15.83:0/3010312586 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fee0c0cc8b0 msgr2=0x7fee0c0ccd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:08.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.859+0000 7fee0a7fc640 1 --2- 172.21.15.83:0/3010312586 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fee0c0cc8b0 0x7fee0c0ccd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.859+0000 7fee0a7fc640 1 -- 172.21.15.83:0/3010312586 >> v2:172.21.15.183:3300/0 conn(0x7fee0c0cfff0 msgr2=0x7fee0c0d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:08.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.859+0000 7fee0a7fc640 1 --2- 172.21.15.83:0/3010312586 >> v2:172.21.15.183:3300/0 conn(0x7fee0c0cfff0 0x7fee0c0d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.859+0000 7fee0a7fc640 1 -- 172.21.15.83:0/3010312586 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee0c161f40 con 0x7fee0c15bb20 2024-03-06T07:24:08.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.859+0000 7fee0a7fc640 1 -- 172.21.15.83:0/3010312586 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7fedfc001670 con 0x7fee0c15bb20 2024-03-06T07:24:08.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.859+0000 7fee0a7fc640 1 -- 172.21.15.83:0/3010312586 >> v1:172.21.15.83:6789/0 conn(0x7fee0c15bb20 legacy=0x7fee0c0d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.859+0000 7fee0a7fc640 1 --2- 172.21.15.83:0/3010312586 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fedf40039b0 0x7fedf4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.859+0000 7fee0a7fc640 1 --2- 172.21.15.83:0/3010312586 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fee0c0cc8b0 0x7fedf4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.859+0000 7fee0a7fc640 1 --2- 172.21.15.83:0/3010312586 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedf4004d40 0x7fedf40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.859+0000 7fee0a7fc640 1 -- 172.21.15.83:0/3010312586 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0c0d2500 con 0x7fedf4004d40 2024-03-06T07:24:08.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.859+0000 7fee0a7fc640 1 -- 172.21.15.83:0/3010312586 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0c161c60 con 0x7fedf40039b0 2024-03-06T07:24:08.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.859+0000 7fee0a7fc640 1 -- 172.21.15.83:0/3010312586 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedf400b9c0 con 0x7fee0c0cc8b0 2024-03-06T07:24:08.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.859+0000 7fee0bfff640 1 --2- 172.21.15.83:0/3010312586 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedf4004d40 0x7fedf40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.859+0000 7fee0b7fe640 1 --2- 172.21.15.83:0/3010312586 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fedf40039b0 0x7fedf4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.859+0000 7fee0affd640 1 --2- 172.21.15.83:0/3010312586 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fee0c0cc8b0 0x7fedf4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.860+0000 7fee0bfff640 1 -- 172.21.15.83:0/3010312586 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fee0c0cc8b0 msgr2=0x7fedf4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:08.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.860+0000 7fee0bfff640 1 --2- 172.21.15.83:0/3010312586 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fee0c0cc8b0 0x7fedf4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.860+0000 7fee0bfff640 1 -- 172.21.15.83:0/3010312586 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fedf40039b0 msgr2=0x7fedf4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:08.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.860+0000 7fee0bfff640 1 --2- 172.21.15.83:0/3010312586 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fedf40039b0 0x7fedf4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.860+0000 7fee0bfff640 1 -- 172.21.15.83:0/3010312586 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf4003520 con 0x7fedf4004d40 2024-03-06T07:24:08.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.860+0000 7fee0bfff640 1 --2- 172.21.15.83:0/3010312586 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedf4004d40 0x7fedf40051c0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fee04009fc0 tx=0x7fedf4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b18ca31e4a2557d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.861+0000 7fee0a7fc640 1 -- 172.21.15.83:0/3010312586 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee04039070 con 0x7fedf4004d40 2024-03-06T07:24:08.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.861+0000 7fee0a7fc640 1 -- 172.21.15.83:0/3010312586 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee0402cc20 con 0x7fedf4004d40 2024-03-06T07:24:08.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.861+0000 7fee0a7fc640 1 -- 172.21.15.83:0/3010312586 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee04009c10 con 0x7fedf4004d40 2024-03-06T07:24:08.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.861+0000 7fee11f1e640 1 -- 172.21.15.83:0/3010312586 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedf4004d40 msgr2=0x7fedf40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.861+0000 7fee11f1e640 1 --2- 172.21.15.83:0/3010312586 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedf4004d40 0x7fedf40051c0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fee04009fc0 tx=0x7fedf4005830 comp rx=0 tx=0).stop 2024-03-06T07:24:08.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.861+0000 7fee0bfff640 1 -- 172.21.15.83:0/3010312586 reap_dead start 2024-03-06T07:24:08.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.861+0000 7fee11f1e640 1 -- 172.21.15.83:0/3010312586 shutdown_connections 2024-03-06T07:24:08.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.861+0000 7fee11f1e640 1 -- 172.21.15.83:0/3010312586 >> 172.21.15.83:0/3010312586 conn(0x7fee0c09a9f0 msgr2=0x7fee0c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:08.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.861+0000 7fee11f1e640 1 -- 172.21.15.83:0/3010312586 shutdown_connections 2024-03-06T07:24:08.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.861+0000 7fee11f1e640 1 -- 172.21.15.83:0/3010312586 wait complete. 2024-03-06T07:24:08.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.862+0000 7fee11f1e640 1 Processor -- start 2024-03-06T07:24:08.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.863+0000 7fee11f1e640 1 -- start start 2024-03-06T07:24:08.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.863+0000 7fee11f1e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fee0c0c80d0 0x7fee0c0b6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.863+0000 7fee11f1e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee0c154900 0x7fee0c0b4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.863+0000 7fee11f1e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fee0c0b4c40 0x7fee0c158cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.863+0000 7fee11f1e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0c0d1980 con 0x7fee0c154900 2024-03-06T07:24:08.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.863+0000 7fee11f1e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0c162500 con 0x7fee0c0c80d0 2024-03-06T07:24:08.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.863+0000 7fee11f1e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee0c0d1b00 con 0x7fee0c0b4c40 2024-03-06T07:24:08.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.863+0000 7fee0affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee0c154900 0x7fee0c0b4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.863+0000 7fee0affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee0c154900 0x7fee0c0b4700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33394/0 (socket says 172.21.15.83:33394) 2024-03-06T07:24:08.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.863+0000 7fee0affd640 1 -- 172.21.15.83:0/2926179511 learned_addr learned my addr 172.21.15.83:0/2926179511 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:08.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.863+0000 7fee0bfff640 1 --2- 172.21.15.83:0/2926179511 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fee0c0b4c40 0x7fee0c158cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.863+0000 7fee0b7fe640 1 --2- 172.21.15.83:0/2926179511 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fee0c0c80d0 0x7fee0c0b6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.864+0000 7fee0affd640 1 -- 172.21.15.83:0/2926179511 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fee0c0b4c40 msgr2=0x7fee0c158cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.864+0000 7fee0affd640 1 --2- 172.21.15.83:0/2926179511 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fee0c0b4c40 0x7fee0c158cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.864+0000 7fee0affd640 1 -- 172.21.15.83:0/2926179511 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fee0c0c80d0 msgr2=0x7fee0c0b6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.864+0000 7fee0affd640 1 --2- 172.21.15.83:0/2926179511 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fee0c0c80d0 0x7fee0c0b6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.864+0000 7fee0affd640 1 -- 172.21.15.83:0/2926179511 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee0c0c84b0 con 0x7fee0c154900 2024-03-06T07:24:08.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.864+0000 7fee0affd640 1 --2- 172.21.15.83:0/2926179511 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee0c154900 0x7fee0c0b4700 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fee00011aa0 tx=0x7fee00011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.864+0000 7fedf3fff640 1 -- 172.21.15.83:0/2926179511 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0000fb20 con 0x7fee0c154900 2024-03-06T07:24:08.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.864+0000 7fedf3fff640 1 -- 172.21.15.83:0/2926179511 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee0000fcc0 con 0x7fee0c154900 2024-03-06T07:24:08.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.864+0000 7fee11f1e640 1 -- 172.21.15.83:0/2926179511 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee0c0b6e40 con 0x7fee0c154900 2024-03-06T07:24:08.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.864+0000 7fedf3fff640 1 -- 172.21.15.83:0/2926179511 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee00014660 con 0x7fee0c154900 2024-03-06T07:24:08.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.864+0000 7fee11f1e640 1 -- 172.21.15.83:0/2926179511 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee0c0b7400 con 0x7fee0c154900 2024-03-06T07:24:08.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.865+0000 7fee11f1e640 1 -- 172.21.15.83:0/2926179511 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee0c04f330 con 0x7fee0c154900 2024-03-06T07:24:08.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.865+0000 7fedf3fff640 1 -- 172.21.15.83:0/2926179511 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fee00012040 con 0x7fee0c154900 2024-03-06T07:24:08.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.866+0000 7fedf3fff640 1 --2- 172.21.15.83:0/2926179511 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fedf806b4d0 0x7fedf806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.866+0000 7fedf3fff640 1 -- 172.21.15.83:0/2926179511 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fee0008a980 con 0x7fee0c154900 2024-03-06T07:24:08.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.866+0000 7fee0b7fe640 1 --2- 172.21.15.83:0/2926179511 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fedf806b4d0 0x7fedf806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.866+0000 7fee0b7fe640 1 --2- 172.21.15.83:0/2926179511 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fedf806b4d0 0x7fedf806d990 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fedfc004840 tx=0x7fedfc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:08.874+0000 7fedf3fff640 1 -- 172.21.15.83:0/2926179511 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fee00019030 con 0x7fee0c154900 2024-03-06T07:24:09.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.121+0000 7fee11f1e640 1 -- 172.21.15.83:0/2926179511 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee0c0c5cc0 con 0x7fee0c154900 2024-03-06T07:24:09.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.122+0000 7fedf3fff640 1 -- 172.21.15.83:0/2926179511 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+16625 (secure 0 0 0) 0x7fee00059770 con 0x7fee0c154900 2024-03-06T07:24:09.124 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:24:09.125 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":45,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:23:43.092917+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:39.656617+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:40.402765+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:09.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.124+0000 7fee11f1e640 1 -- 172.21.15.83:0/2926179511 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fedf806b4d0 msgr2=0x7fedf806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.124+0000 7fee11f1e640 1 --2- 172.21.15.83:0/2926179511 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fedf806b4d0 0x7fedf806d990 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fedfc004840 tx=0x7fedfc009040 comp rx=0 tx=0).stop 2024-03-06T07:24:09.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.124+0000 7fee11f1e640 1 -- 172.21.15.83:0/2926179511 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee0c154900 msgr2=0x7fee0c0b4700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.124+0000 7fee11f1e640 1 --2- 172.21.15.83:0/2926179511 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee0c154900 0x7fee0c0b4700 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fee00011aa0 tx=0x7fee00011f60 comp rx=0 tx=0).stop 2024-03-06T07:24:09.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.124+0000 7fee11f1e640 1 -- 172.21.15.83:0/2926179511 shutdown_connections 2024-03-06T07:24:09.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.124+0000 7fee11f1e640 1 --2- 172.21.15.83:0/2926179511 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fedf806b4d0 0x7fedf806d990 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.124+0000 7fee11f1e640 1 --2- 172.21.15.83:0/2926179511 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fee0c0b4c40 0x7fee0c158cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.124+0000 7fee11f1e640 1 --2- 172.21.15.83:0/2926179511 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee0c154900 0x7fee0c0b4700 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.124+0000 7fee11f1e640 1 --2- 172.21.15.83:0/2926179511 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fee0c0c80d0 0x7fee0c0b6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.124+0000 7fee11f1e640 1 -- 172.21.15.83:0/2926179511 >> 172.21.15.83:0/2926179511 conn(0x7fee0c09a9f0 msgr2=0x7fee0c0703f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:09.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.124+0000 7fee11f1e640 1 -- 172.21.15.83:0/2926179511 shutdown_connections 2024-03-06T07:24:09.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.124+0000 7fee11f1e640 1 -- 172.21.15.83:0/2926179511 wait complete. 2024-03-06T07:24:09.137 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:24:09.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.238+0000 7fe13c93e640 1 Processor -- start 2024-03-06T07:24:09.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.238+0000 7fe13c93e640 1 -- start start 2024-03-06T07:24:09.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.239+0000 7fe13c93e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe138162970 0x7fe1380d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.239+0000 7fe13c93e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1380d22c0 con 0x7fe138161cf0 2024-03-06T07:24:09.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.239+0000 7fe13c93e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1380d25a0 con 0x7fe1380b4460 2024-03-06T07:24:09.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.239+0000 7fe13c93e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe138161b00 con 0x7fe138162970 2024-03-06T07:24:09.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.240+0000 7fe136d9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fe138161cf0 0x7fe1380cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48506/0 (socket says 172.21.15.83:48506) 2024-03-06T07:24:09.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.240+0000 7fe136d9d640 1 -- 172.21.15.83:0/3676873802 learned_addr learned my addr 172.21.15.83:0/3676873802 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:24:09.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.240+0000 7fe135d9b640 1 --2- 172.21.15.83:0/3676873802 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe138162970 0x7fe1380d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.240+0000 7fe13559a640 1 -- 172.21.15.83:0/3676873802 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1638632014 0 0) 0x7fe1380d22c0 con 0x7fe138161cf0 2024-03-06T07:24:09.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.240+0000 7fe13559a640 1 -- 172.21.15.83:0/3676873802 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe120003520 con 0x7fe138161cf0 2024-03-06T07:24:09.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.240+0000 7fe13559a640 1 -- 172.21.15.83:0/3676873802 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3865200018 0 0) 0x7fe1380d25a0 con 0x7fe1380b4460 2024-03-06T07:24:09.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.240+0000 7fe13559a640 1 -- 172.21.15.83:0/3676873802 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1380d22c0 con 0x7fe1380b4460 2024-03-06T07:24:09.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.240+0000 7fe13559a640 1 -- 172.21.15.83:0/3676873802 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 382206204 0 0) 0x7fe120003520 con 0x7fe138161cf0 2024-03-06T07:24:09.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.240+0000 7fe13559a640 1 -- 172.21.15.83:0/3676873802 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe138162970 msgr2=0x7fe1380d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:09.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.240+0000 7fe13559a640 1 --2- 172.21.15.83:0/3676873802 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe138162970 0x7fe1380d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.240+0000 7fe13559a640 1 -- 172.21.15.83:0/3676873802 >> v1:172.21.15.183:6789/0 conn(0x7fe1380b4460 legacy=0x7fe1380b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.240+0000 7fe13559a640 1 -- 172.21.15.83:0/3676873802 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1380d2880 con 0x7fe138161cf0 2024-03-06T07:24:09.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.240+0000 7fe13559a640 1 -- 172.21.15.83:0/3676873802 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7fe1300019f0 con 0x7fe138161cf0 2024-03-06T07:24:09.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.240+0000 7fe135d9b640 1 --2- 172.21.15.83:0/3676873802 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe138162970 0x7fe1380d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:09.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.240+0000 7fe13559a640 1 -- 172.21.15.83:0/3676873802 >> v1:172.21.15.83:6789/0 conn(0x7fe138161cf0 legacy=0x7fe1380cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.240+0000 7fe13559a640 1 --2- 172.21.15.83:0/3676873802 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe120003b70 0x7fe120003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.240+0000 7fe13559a640 1 --2- 172.21.15.83:0/3676873802 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe120004800 0x7fe120004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.240+0000 7fe13559a640 1 --2- 172.21.15.83:0/3676873802 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe138162970 0x7fe1200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.240+0000 7fe13559a640 1 -- 172.21.15.83:0/3676873802 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe138161b00 con 0x7fe120004800 2024-03-06T07:24:09.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.240+0000 7fe13559a640 1 -- 172.21.15.83:0/3676873802 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe12000b9f0 con 0x7fe120003b70 2024-03-06T07:24:09.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.240+0000 7fe13559a640 1 -- 172.21.15.83:0/3676873802 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe12000bbf0 con 0x7fe138162970 2024-03-06T07:24:09.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.240+0000 7fe135d9b640 1 --2- 172.21.15.83:0/3676873802 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe120004800 0x7fe120004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.240+0000 7fe136d9d640 1 --2- 172.21.15.83:0/3676873802 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe138162970 0x7fe1200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.240+0000 7fe13659c640 1 --2- 172.21.15.83:0/3676873802 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe120003b70 0x7fe120003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.241+0000 7fe135d9b640 1 -- 172.21.15.83:0/3676873802 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe138162970 msgr2=0x7fe1200051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:09.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.241+0000 7fe135d9b640 1 --2- 172.21.15.83:0/3676873802 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe138162970 0x7fe1200051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.241+0000 7fe135d9b640 1 -- 172.21.15.83:0/3676873802 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe120003b70 msgr2=0x7fe120003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:09.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.241+0000 7fe135d9b640 1 --2- 172.21.15.83:0/3676873802 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe120003b70 0x7fe120003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.241+0000 7fe135d9b640 1 -- 172.21.15.83:0/3676873802 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe120003520 con 0x7fe120004800 2024-03-06T07:24:09.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.241+0000 7fe13659c640 1 --2- 172.21.15.83:0/3676873802 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe120003b70 0x7fe120003f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:09.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.241+0000 7fe135d9b640 1 --2- 172.21.15.83:0/3676873802 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe120004800 0x7fe120004c80 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fe138162db0 tx=0x7fe120005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98a27d90c53be706 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:09.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.241+0000 7fe13559a640 1 -- 172.21.15.83:0/3676873802 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe128039070 con 0x7fe120004800 2024-03-06T07:24:09.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.241+0000 7fe13559a640 1 -- 172.21.15.83:0/3676873802 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe12802c890 con 0x7fe120004800 2024-03-06T07:24:09.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.241+0000 7fe13559a640 1 -- 172.21.15.83:0/3676873802 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe128009860 con 0x7fe120004800 2024-03-06T07:24:09.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.242+0000 7fe13c93e640 1 -- 172.21.15.83:0/3676873802 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe120004800 msgr2=0x7fe120004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.242+0000 7fe13c93e640 1 --2- 172.21.15.83:0/3676873802 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe120004800 0x7fe120004c80 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fe138162db0 tx=0x7fe120005830 comp rx=0 tx=0).stop 2024-03-06T07:24:09.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.242+0000 7fe136d9d640 1 -- 172.21.15.83:0/3676873802 reap_dead start 2024-03-06T07:24:09.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.242+0000 7fe13c93e640 1 -- 172.21.15.83:0/3676873802 shutdown_connections 2024-03-06T07:24:09.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.242+0000 7fe13c93e640 1 -- 172.21.15.83:0/3676873802 >> 172.21.15.83:0/3676873802 conn(0x7fe1380d3ce0 msgr2=0x7fe1380d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:09.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.242+0000 7fe13c93e640 1 -- 172.21.15.83:0/3676873802 shutdown_connections 2024-03-06T07:24:09.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.242+0000 7fe13c93e640 1 -- 172.21.15.83:0/3676873802 wait complete. 2024-03-06T07:24:09.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.243+0000 7fe13c93e640 1 Processor -- start 2024-03-06T07:24:09.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.243+0000 7fe13c93e640 1 -- start start 2024-03-06T07:24:09.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.244+0000 7fe13c93e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe120004800 0x7fe1380cc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.244+0000 7fe13c93e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1380b6900 0x7fe138152af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.244+0000 7fe13c93e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe1381f3cb0 0x7fe1381f4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.244+0000 7fe13c93e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1380ce950 con 0x7fe1380b6900 2024-03-06T07:24:09.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.244+0000 7fe13c93e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1380cecc0 con 0x7fe1381f3cb0 2024-03-06T07:24:09.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.244+0000 7fe13c93e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1380b4840 con 0x7fe120004800 2024-03-06T07:24:09.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.244+0000 7fe135d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1380b6900 0x7fe138152af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.244+0000 7fe135d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1380b6900 0x7fe138152af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33410/0 (socket says 172.21.15.83:33410) 2024-03-06T07:24:09.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.244+0000 7fe135d9b640 1 -- 172.21.15.83:0/1210903423 learned_addr learned my addr 172.21.15.83:0/1210903423 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:09.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.244+0000 7fe13659c640 1 --2- 172.21.15.83:0/1210903423 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe120004800 0x7fe1380cc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.244+0000 7fe136d9d640 1 --2- 172.21.15.83:0/1210903423 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe1381f3cb0 0x7fe1381f4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.244+0000 7fe135d9b640 1 -- 172.21.15.83:0/1210903423 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe120004800 msgr2=0x7fe1380cc3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.245+0000 7fe135d9b640 1 --2- 172.21.15.83:0/1210903423 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe120004800 0x7fe1380cc3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.245+0000 7fe135d9b640 1 -- 172.21.15.83:0/1210903423 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe1381f3cb0 msgr2=0x7fe1381f4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.245+0000 7fe135d9b640 1 --2- 172.21.15.83:0/1210903423 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe1381f3cb0 0x7fe1381f4130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.245+0000 7fe135d9b640 1 -- 172.21.15.83:0/1210903423 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1381f7310 con 0x7fe1380b6900 2024-03-06T07:24:09.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.245+0000 7fe13659c640 1 --2- 172.21.15.83:0/1210903423 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe120004800 0x7fe1380cc3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:09.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.245+0000 7fe135d9b640 1 --2- 172.21.15.83:0/1210903423 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1380b6900 0x7fe138152af0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fe12802c440 tx=0x7fe128004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:09.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.245+0000 7fe126ffd640 1 -- 172.21.15.83:0/1210903423 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe128039050 con 0x7fe1380b6900 2024-03-06T07:24:09.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.245+0000 7fe126ffd640 1 -- 172.21.15.83:0/1210903423 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe128045070 con 0x7fe1380b6900 2024-03-06T07:24:09.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.245+0000 7fe13c93e640 1 -- 172.21.15.83:0/1210903423 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1380b8cd0 con 0x7fe1380b6900 2024-03-06T07:24:09.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.245+0000 7fe126ffd640 1 -- 172.21.15.83:0/1210903423 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe128009860 con 0x7fe1380b6900 2024-03-06T07:24:09.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.245+0000 7fe13c93e640 1 -- 172.21.15.83:0/1210903423 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1380b7180 con 0x7fe1380b6900 2024-03-06T07:24:09.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.246+0000 7fe13c93e640 1 -- 172.21.15.83:0/1210903423 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe13804f330 con 0x7fe1380b6900 2024-03-06T07:24:09.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.246+0000 7fe126ffd640 1 -- 172.21.15.83:0/1210903423 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe12804a460 con 0x7fe1380b6900 2024-03-06T07:24:09.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.247+0000 7fe126ffd640 1 --2- 172.21.15.83:0/1210903423 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe11806b4d0 0x7fe11806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.247+0000 7fe13659c640 1 --2- 172.21.15.83:0/1210903423 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe11806b4d0 0x7fe11806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.247+0000 7fe126ffd640 1 -- 172.21.15.83:0/1210903423 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe1280ac310 con 0x7fe1380b6900 2024-03-06T07:24:09.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.247+0000 7fe13659c640 1 --2- 172.21.15.83:0/1210903423 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe11806b4d0 0x7fe11806d990 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe12c000f60 tx=0x7fe12c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:09.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.251+0000 7fe126ffd640 1 -- 172.21.15.83:0/1210903423 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe1380b8cd0 con 0x7fe1380b6900 2024-03-06T07:24:09.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.460+0000 7fe13c93e640 1 -- 172.21.15.83:0/1210903423 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fe13806b180 con 0x7fe1380b6900 2024-03-06T07:24:09.463 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-03-06T07:24:09.461+0000 7fb0dae11640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:24:09.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.463+0000 7fe126ffd640 1 -- 172.21.15.83:0/1210903423 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v20) v1 ==== 161+0+0 (secure 0 0 0) 0x7fe12807b220 con 0x7fe1380b6900 2024-03-06T07:24:09.464 INFO:teuthology.orchestra.run.smithi083.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:24:09.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.465+0000 7fe13c93e640 1 -- 172.21.15.83:0/1210903423 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe11806b4d0 msgr2=0x7fe11806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.465+0000 7fe13c93e640 1 --2- 172.21.15.83:0/1210903423 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe11806b4d0 0x7fe11806d990 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe12c000f60 tx=0x7fe12c009040 comp rx=0 tx=0).stop 2024-03-06T07:24:09.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.465+0000 7fe13c93e640 1 -- 172.21.15.83:0/1210903423 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1380b6900 msgr2=0x7fe138152af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.465+0000 7fe13c93e640 1 --2- 172.21.15.83:0/1210903423 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1380b6900 0x7fe138152af0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fe12802c440 tx=0x7fe128004730 comp rx=0 tx=0).stop 2024-03-06T07:24:09.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.466+0000 7fe13c93e640 1 -- 172.21.15.83:0/1210903423 shutdown_connections 2024-03-06T07:24:09.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.466+0000 7fe13c93e640 1 --2- 172.21.15.83:0/1210903423 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe11806b4d0 0x7fe11806d990 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.466+0000 7fe13c93e640 1 --2- 172.21.15.83:0/1210903423 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe1381f3cb0 0x7fe1381f4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.466+0000 7fe13c93e640 1 --2- 172.21.15.83:0/1210903423 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1380b6900 0x7fe138152af0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.466+0000 7fe13c93e640 1 --2- 172.21.15.83:0/1210903423 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe120004800 0x7fe1380cc3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.466+0000 7fe13c93e640 1 -- 172.21.15.83:0/1210903423 >> 172.21.15.83:0/1210903423 conn(0x7fe1380d3ce0 msgr2=0x7fe1380d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:09.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.466+0000 7fe13c93e640 1 -- 172.21.15.83:0/1210903423 shutdown_connections 2024-03-06T07:24:09.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.466+0000 7fe13c93e640 1 -- 172.21.15.83:0/1210903423 wait complete. 2024-03-06T07:24:09.479 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:24:09.505 INFO:tasks.ceph.mds.a.smithi083.stdout:starting mds.a at 2024-03-06T07:24:09.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.582+0000 7fe0e0015640 1 Processor -- start 2024-03-06T07:24:09.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.582+0000 7fe0e0015640 1 -- start start 2024-03-06T07:24:09.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.583+0000 7fe0e0015640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fe0d80b4970 0x7fe0d806d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.583+0000 7fe0e0015640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe0d806d5c0 0x7fe0d80cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.583+0000 7fe0e0015640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0d815ff90 con 0x7fe0d815bb40 2024-03-06T07:24:09.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.583+0000 7fe0e0015640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0d80b4dd0 con 0x7fe0d806d5c0 2024-03-06T07:24:09.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.583+0000 7fe0e0015640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0d806c910 con 0x7fe0d80b4970 2024-03-06T07:24:09.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.583+0000 7fe0de5b2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe0d806d5c0 0x7fe0d80cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.583+0000 7fe0dddb1640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fe0d815bb40 0x7fe0d815f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48526/0 (socket says 172.21.15.83:48526) 2024-03-06T07:24:09.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.583+0000 7fe0dddb1640 1 -- 172.21.15.83:0/12794822 learned_addr learned my addr 172.21.15.83:0/12794822 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:24:09.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.583+0000 7fe0dd5b0640 1 --2- 172.21.15.83:0/12794822 >> v2:172.21.15.183:3300/0 conn(0x7fe0d80b4970 0x7fe0d806d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.583+0000 7fe0de5b2640 1 -- 172.21.15.83:0/12794822 >> v2:172.21.15.183:3300/0 conn(0x7fe0d80b4970 msgr2=0x7fe0d806d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:09.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.583+0000 7fe0de5b2640 1 --2- 172.21.15.83:0/12794822 >> v2:172.21.15.183:3300/0 conn(0x7fe0d80b4970 0x7fe0d806d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.583+0000 7fe0de5b2640 1 -- 172.21.15.83:0/12794822 >> v1:172.21.15.83:6789/0 conn(0x7fe0d815bb40 legacy=0x7fe0d815f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.583+0000 7fe0de5b2640 1 -- 172.21.15.83:0/12794822 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d8160270 con 0x7fe0d806d5c0 2024-03-06T07:24:09.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.584+0000 7fe0de5b2640 1 --2- 172.21.15.83:0/12794822 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0d806d5c0 0x7fe0d80cfa90 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fe0d806bfe0 tx=0x7fe0d002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=537458ae2904087f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:09.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.584+0000 7fe0dcdaf640 1 -- 172.21.15.83:0/12794822 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d0039070 con 0x7fe0d806d5c0 2024-03-06T07:24:09.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.584+0000 7fe0dcdaf640 1 -- 172.21.15.83:0/12794822 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0d002cc80 con 0x7fe0d806d5c0 2024-03-06T07:24:09.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.584+0000 7fe0dcdaf640 1 -- 172.21.15.83:0/12794822 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d0034660 con 0x7fe0d806d5c0 2024-03-06T07:24:09.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.584+0000 7fe0e0015640 1 -- 172.21.15.83:0/12794822 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0d806d5c0 msgr2=0x7fe0d80cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.584+0000 7fe0e0015640 1 --2- 172.21.15.83:0/12794822 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0d806d5c0 0x7fe0d80cfa90 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fe0d806bfe0 tx=0x7fe0d002bed0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.585+0000 7fe0e0015640 1 -- 172.21.15.83:0/12794822 shutdown_connections 2024-03-06T07:24:09.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.585+0000 7fe0e0015640 1 --2- 172.21.15.83:0/12794822 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0d806d5c0 0x7fe0d80cfa90 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.585+0000 7fe0e0015640 1 --2- 172.21.15.83:0/12794822 >> v2:172.21.15.183:3300/0 conn(0x7fe0d80b4970 0x7fe0d806d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.585+0000 7fe0e0015640 1 -- 172.21.15.83:0/12794822 >> 172.21.15.83:0/12794822 conn(0x7fe0d806b190 msgr2=0x7fe0d806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:09.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.585+0000 7fe0e0015640 1 -- 172.21.15.83:0/12794822 shutdown_connections 2024-03-06T07:24:09.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.585+0000 7fe0e0015640 1 -- 172.21.15.83:0/12794822 wait complete. 2024-03-06T07:24:09.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.586+0000 7fe0e0015640 1 Processor -- start 2024-03-06T07:24:09.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.586+0000 7fe0e0015640 1 -- start start 2024-03-06T07:24:09.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.586+0000 7fe0e0015640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe0d806d5c0 0x7fe0d8147320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.586+0000 7fe0e0015640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe0d80b4970 0x7fe0d8156c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.586+0000 7fe0e0015640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0d815bb40 0x7fe0d8157140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.586+0000 7fe0e0015640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d815bf20 con 0x7fe0d815bb40 2024-03-06T07:24:09.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.586+0000 7fe0e0015640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d806cb60 con 0x7fe0d80b4970 2024-03-06T07:24:09.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.586+0000 7fe0e0015640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d815c0a0 con 0x7fe0d806d5c0 2024-03-06T07:24:09.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.587+0000 7fe0de5b2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0d815bb40 0x7fe0d8157140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.587+0000 7fe0de5b2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0d815bb40 0x7fe0d8157140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33456/0 (socket says 172.21.15.83:33456) 2024-03-06T07:24:09.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.587+0000 7fe0de5b2640 1 -- 172.21.15.83:0/93812574 learned_addr learned my addr 172.21.15.83:0/93812574 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:09.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.587+0000 7fe0dddb1640 1 --2- 172.21.15.83:0/93812574 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe0d806d5c0 0x7fe0d8147320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.587+0000 7fe0dd5b0640 1 --2- 172.21.15.83:0/93812574 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe0d80b4970 0x7fe0d8156c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.587+0000 7fe0de5b2640 1 -- 172.21.15.83:0/93812574 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe0d806d5c0 msgr2=0x7fe0d8147320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.587+0000 7fe0de5b2640 1 --2- 172.21.15.83:0/93812574 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe0d806d5c0 0x7fe0d8147320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.587+0000 7fe0de5b2640 1 -- 172.21.15.83:0/93812574 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe0d80b4970 msgr2=0x7fe0d8156c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.587+0000 7fe0de5b2640 1 --2- 172.21.15.83:0/93812574 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe0d80b4970 0x7fe0d8156c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.587+0000 7fe0de5b2640 1 -- 172.21.15.83:0/93812574 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d815c4a0 con 0x7fe0d815bb40 2024-03-06T07:24:09.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.587+0000 7fe0de5b2640 1 --2- 172.21.15.83:0/93812574 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0d815bb40 0x7fe0d8157140 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fe0d0009e80 tx=0x7fe0d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:09.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.587+0000 7fe0ce7fc640 1 -- 172.21.15.83:0/93812574 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d0039050 con 0x7fe0d815bb40 2024-03-06T07:24:09.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.587+0000 7fe0ce7fc640 1 -- 172.21.15.83:0/93812574 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0d0045070 con 0x7fe0d815bb40 2024-03-06T07:24:09.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.587+0000 7fe0e0015640 1 -- 172.21.15.83:0/93812574 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0d815c780 con 0x7fe0d815bb40 2024-03-06T07:24:09.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.587+0000 7fe0ce7fc640 1 -- 172.21.15.83:0/93812574 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d003e830 con 0x7fe0d815bb40 2024-03-06T07:24:09.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.587+0000 7fe0e0015640 1 -- 172.21.15.83:0/93812574 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0d815cd40 con 0x7fe0d815bb40 2024-03-06T07:24:09.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.588+0000 7fe0e0015640 1 -- 172.21.15.83:0/93812574 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0d80732b0 con 0x7fe0d815bb40 2024-03-06T07:24:09.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.589+0000 7fe0ce7fc640 1 -- 172.21.15.83:0/93812574 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe0d003e9d0 con 0x7fe0d815bb40 2024-03-06T07:24:09.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.589+0000 7fe0ce7fc640 1 --2- 172.21.15.83:0/93812574 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe0bc06b4d0 0x7fe0bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.589+0000 7fe0ce7fc640 1 -- 172.21.15.83:0/93812574 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fe0d00ad2c0 con 0x7fe0d815bb40 2024-03-06T07:24:09.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.589+0000 7fe0dddb1640 1 --2- 172.21.15.83:0/93812574 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe0bc06b4d0 0x7fe0bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.589+0000 7fe0dddb1640 1 --2- 172.21.15.83:0/93812574 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe0bc06b4d0 0x7fe0bc06d990 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4000f60 tx=0x7fe0d4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:09.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.593+0000 7fe0ce7fc640 1 -- 172.21.15.83:0/93812574 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe0d007c170 con 0x7fe0d815bb40 2024-03-06T07:24:09.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:09.802+0000 7fe0e0015640 1 -- 172.21.15.83:0/93812574 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe0d8068ff0 con 0x7fe0d815bb40 2024-03-06T07:24:10.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.464+0000 7fe0ce7fc640 1 -- 172.21.15.83:0/93812574 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v21) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe0d007ba70 con 0x7fe0d815bb40 2024-03-06T07:24:10.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.467+0000 7fe0e0015640 1 -- 172.21.15.83:0/93812574 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe0bc06b4d0 msgr2=0x7fe0bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:10.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.467+0000 7fe0e0015640 1 --2- 172.21.15.83:0/93812574 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe0bc06b4d0 0x7fe0bc06d990 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4000f60 tx=0x7fe0d4009040 comp rx=0 tx=0).stop 2024-03-06T07:24:10.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.467+0000 7fe0e0015640 1 -- 172.21.15.83:0/93812574 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0d815bb40 msgr2=0x7fe0d8157140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:10.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.467+0000 7fe0e0015640 1 --2- 172.21.15.83:0/93812574 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0d815bb40 0x7fe0d8157140 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fe0d0009e80 tx=0x7fe0d0004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:10.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.468+0000 7fe0e0015640 1 -- 172.21.15.83:0/93812574 shutdown_connections 2024-03-06T07:24:10.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.468+0000 7fe0e0015640 1 --2- 172.21.15.83:0/93812574 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe0bc06b4d0 0x7fe0bc06d990 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.468+0000 7fe0e0015640 1 --2- 172.21.15.83:0/93812574 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0d815bb40 0x7fe0d8157140 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.468+0000 7fe0e0015640 1 --2- 172.21.15.83:0/93812574 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe0d80b4970 0x7fe0d8156c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.468+0000 7fe0e0015640 1 --2- 172.21.15.83:0/93812574 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe0d806d5c0 0x7fe0d8147320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.468+0000 7fe0e0015640 1 -- 172.21.15.83:0/93812574 >> 172.21.15.83:0/93812574 conn(0x7fe0d806b190 msgr2=0x7fe0d8159130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:10.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.468+0000 7fe0e0015640 1 -- 172.21.15.83:0/93812574 shutdown_connections 2024-03-06T07:24:10.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.468+0000 7fe0e0015640 1 -- 172.21.15.83:0/93812574 wait complete. 2024-03-06T07:24:10.483 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:24:10.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.629+0000 7f5dd2adf640 1 Processor -- start 2024-03-06T07:24:10.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.629+0000 7f5dd2adf640 1 -- start start 2024-03-06T07:24:10.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.630+0000 7f5dd2adf640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5dcc147720 0x7f5dcc0b3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:10.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.630+0000 7f5dd2adf640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f5dcc0ad940 0x7f5dcc0ae750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:10.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.630+0000 7f5dd2adf640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dcc06d090 con 0x7f5dcc071d20 2024-03-06T07:24:10.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.630+0000 7f5dd2adf640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dcc06d370 con 0x7f5dcc147720 2024-03-06T07:24:10.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.630+0000 7f5dd2adf640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dcc06d650 con 0x7f5dcc0ad940 2024-03-06T07:24:10.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.630+0000 7f5dd087b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5dcc147720 0x7f5dcc0b3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:10.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.630+0000 7f5dd087b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5dcc147720 0x7f5dcc0b3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33470/0 (socket says 172.21.15.83:33470) 2024-03-06T07:24:10.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.630+0000 7f5dd087b640 1 -- 172.21.15.83:0/3391608948 learned_addr learned my addr 172.21.15.83:0/3391608948 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:10.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.630+0000 7f5dcbfff640 1 --2- 172.21.15.83:0/3391608948 >> v2:172.21.15.183:3300/0 conn(0x7f5dcc0ad940 0x7f5dcc0ae750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:10.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.630+0000 7f5dd087b640 1 -- 172.21.15.83:0/3391608948 >> v2:172.21.15.183:3300/0 conn(0x7f5dcc0ad940 msgr2=0x7f5dcc0ae750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:10.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.630+0000 7f5dd087b640 1 --2- 172.21.15.83:0/3391608948 >> v2:172.21.15.183:3300/0 conn(0x7f5dcc0ad940 0x7f5dcc0ae750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.630+0000 7f5dd087b640 1 -- 172.21.15.83:0/3391608948 >> v1:172.21.15.183:6789/0 conn(0x7f5dcc071d20 legacy=0x7f5dcc072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:10.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.630+0000 7f5dd087b640 1 -- 172.21.15.83:0/3391608948 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dcc06d930 con 0x7f5dcc147720 2024-03-06T07:24:10.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.631+0000 7f5dd087b640 1 --2- 172.21.15.83:0/3391608948 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5dcc147720 0x7f5dcc0b3710 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc074af0 tx=0x7f5dc002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2394552130a67d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:10.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.631+0000 7f5dcb7fe640 1 -- 172.21.15.83:0/3391608948 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc0034070 con 0x7f5dcc147720 2024-03-06T07:24:10.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.631+0000 7f5dcb7fe640 1 -- 172.21.15.83:0/3391608948 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dc0030860 con 0x7f5dcc147720 2024-03-06T07:24:10.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.631+0000 7f5dcb7fe640 1 -- 172.21.15.83:0/3391608948 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc0030c30 con 0x7f5dcc147720 2024-03-06T07:24:10.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.631+0000 7f5dd2adf640 1 -- 172.21.15.83:0/3391608948 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5dcc147720 msgr2=0x7f5dcc0b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:10.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.631+0000 7f5dd2adf640 1 --2- 172.21.15.83:0/3391608948 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5dcc147720 0x7f5dcc0b3710 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc074af0 tx=0x7f5dc002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.632+0000 7f5dd2adf640 1 -- 172.21.15.83:0/3391608948 shutdown_connections 2024-03-06T07:24:10.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.632+0000 7f5dd2adf640 1 --2- 172.21.15.83:0/3391608948 >> v2:172.21.15.183:3300/0 conn(0x7f5dcc0ad940 0x7f5dcc0ae750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.632+0000 7f5dd2adf640 1 --2- 172.21.15.83:0/3391608948 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5dcc147720 0x7f5dcc0b3710 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.632+0000 7f5dd2adf640 1 -- 172.21.15.83:0/3391608948 >> 172.21.15.83:0/3391608948 conn(0x7f5dcc0b1a90 msgr2=0x7f5dcc0b1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:10.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.632+0000 7f5dd2adf640 1 -- 172.21.15.83:0/3391608948 shutdown_connections 2024-03-06T07:24:10.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.632+0000 7f5dd2adf640 1 -- 172.21.15.83:0/3391608948 wait complete. 2024-03-06T07:24:10.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.633+0000 7f5dd2adf640 1 Processor -- start 2024-03-06T07:24:10.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.633+0000 7f5dd2adf640 1 -- start start 2024-03-06T07:24:10.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.634+0000 7f5dd2adf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5dcc071d20 0x7f5dcc0c4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:10.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.634+0000 7f5dd2adf640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5dcc0ad940 0x7f5dcc15d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:10.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.634+0000 7f5dd2adf640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5dcc147720 0x7f5dcc143030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:10.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.634+0000 7f5dd2adf640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dcc0b3c50 con 0x7f5dcc071d20 2024-03-06T07:24:10.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.634+0000 7f5dd2adf640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dcc147b00 con 0x7f5dcc147720 2024-03-06T07:24:10.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.634+0000 7f5dd2adf640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dcc0b3dd0 con 0x7f5dcc0ad940 2024-03-06T07:24:10.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.634+0000 7f5dd087b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5dcc071d20 0x7f5dcc0c4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:10.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.634+0000 7f5dd087b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5dcc071d20 0x7f5dcc0c4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33478/0 (socket says 172.21.15.83:33478) 2024-03-06T07:24:10.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.634+0000 7f5dd087b640 1 -- 172.21.15.83:0/3915748626 learned_addr learned my addr 172.21.15.83:0/3915748626 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:10.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.634+0000 7f5dd087b640 1 -- 172.21.15.83:0/3915748626 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5dcc0ad940 msgr2=0x7f5dcc15d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:10.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.634+0000 7f5dd087b640 1 --2- 172.21.15.83:0/3915748626 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5dcc0ad940 0x7f5dcc15d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.634+0000 7f5dd087b640 1 -- 172.21.15.83:0/3915748626 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5dcc147720 msgr2=0x7f5dcc143030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:10.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.634+0000 7f5dd087b640 1 --2- 172.21.15.83:0/3915748626 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5dcc147720 0x7f5dcc143030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.634+0000 7f5dd087b640 1 -- 172.21.15.83:0/3915748626 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dcc15deb0 con 0x7f5dcc071d20 2024-03-06T07:24:10.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.634+0000 7f5dd087b640 1 --2- 172.21.15.83:0/3915748626 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5dcc071d20 0x7f5dcc0c4d00 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f5dc002b9e0 tx=0x7f5dc00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:10.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.634+0000 7f5dc97fa640 1 -- 172.21.15.83:0/3915748626 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc0034040 con 0x7f5dcc071d20 2024-03-06T07:24:10.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.634+0000 7f5dc97fa640 1 -- 172.21.15.83:0/3915748626 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dc0039d90 con 0x7f5dcc071d20 2024-03-06T07:24:10.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.634+0000 7f5dd2adf640 1 -- 172.21.15.83:0/3915748626 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dcc13df60 con 0x7f5dcc071d20 2024-03-06T07:24:10.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.634+0000 7f5dc97fa640 1 -- 172.21.15.83:0/3915748626 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc0043610 con 0x7f5dcc071d20 2024-03-06T07:24:10.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.634+0000 7f5dd2adf640 1 -- 172.21.15.83:0/3915748626 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dcc13e520 con 0x7f5dcc071d20 2024-03-06T07:24:10.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.635+0000 7f5dd2adf640 1 -- 172.21.15.83:0/3915748626 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dcc04f330 con 0x7f5dcc071d20 2024-03-06T07:24:10.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.636+0000 7f5dc97fa640 1 -- 172.21.15.83:0/3915748626 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5dc004a060 con 0x7f5dcc071d20 2024-03-06T07:24:10.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.636+0000 7f5dc97fa640 1 --2- 172.21.15.83:0/3915748626 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5db006b500 0x7f5db006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:10.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.636+0000 7f5dc97fa640 1 -- 172.21.15.83:0/3915748626 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f5dc00ac300 con 0x7f5dcc071d20 2024-03-06T07:24:10.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.636+0000 7f5dcbfff640 1 --2- 172.21.15.83:0/3915748626 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5db006b500 0x7f5db006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:10.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.636+0000 7f5dcbfff640 1 --2- 172.21.15.83:0/3915748626 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5db006b500 0x7f5db006d9c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc143de0 tx=0x7f5db8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:10.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.640+0000 7f5dc97fa640 1 -- 172.21.15.83:0/3915748626 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5dcc15deb0 con 0x7f5dcc071d20 2024-03-06T07:24:10.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:10.810+0000 7f5dd2adf640 1 -- 172.21.15.83:0/3915748626 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5dcc072ce0 con 0x7f5dcc071d20 2024-03-06T07:24:11.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.466+0000 7f5dc97fa640 1 -- 172.21.15.83:0/3915748626 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v47) v1 ==== 181+0+0 (secure 0 0 0) 0x7f5dc003e070 con 0x7f5dcc071d20 2024-03-06T07:24:11.467 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:24:11.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.468+0000 7f5dd2adf640 1 -- 172.21.15.83:0/3915748626 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5db006b500 msgr2=0x7f5db006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.468+0000 7f5dd2adf640 1 --2- 172.21.15.83:0/3915748626 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5db006b500 0x7f5db006d9c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc143de0 tx=0x7f5db8008040 comp rx=0 tx=0).stop 2024-03-06T07:24:11.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.468+0000 7f5dd2adf640 1 -- 172.21.15.83:0/3915748626 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5dcc071d20 msgr2=0x7f5dcc0c4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.468+0000 7f5dd2adf640 1 --2- 172.21.15.83:0/3915748626 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5dcc071d20 0x7f5dcc0c4d00 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f5dc002b9e0 tx=0x7f5dc00046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.468+0000 7f5dd2adf640 1 -- 172.21.15.83:0/3915748626 shutdown_connections 2024-03-06T07:24:11.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.468+0000 7f5dd2adf640 1 --2- 172.21.15.83:0/3915748626 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5db006b500 0x7f5db006d9c0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.468+0000 7f5dd2adf640 1 --2- 172.21.15.83:0/3915748626 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5dcc147720 0x7f5dcc143030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.468+0000 7f5dd2adf640 1 --2- 172.21.15.83:0/3915748626 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5dcc0ad940 0x7f5dcc15d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.468+0000 7f5dd2adf640 1 --2- 172.21.15.83:0/3915748626 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5dcc071d20 0x7f5dcc0c4d00 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.468+0000 7f5dd2adf640 1 -- 172.21.15.83:0/3915748626 >> 172.21.15.83:0/3915748626 conn(0x7f5dcc0b1a90 msgr2=0x7f5dcc146930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:11.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.468+0000 7f5dd2adf640 1 -- 172.21.15.83:0/3915748626 shutdown_connections 2024-03-06T07:24:11.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.468+0000 7f5dd2adf640 1 -- 172.21.15.83:0/3915748626 wait complete. 2024-03-06T07:24:11.483 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:24:11.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.587+0000 7fcc7522a640 1 Processor -- start 2024-03-06T07:24:11.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.587+0000 7fcc7522a640 1 -- start start 2024-03-06T07:24:11.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.588+0000 7fcc7522a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcc7015ac00 0x7fcc7006c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.588+0000 7fcc7522a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fcc70070d10 0x7fcc7015eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.588+0000 7fcc7522a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc700711c0 con 0x7fcc700708b0 2024-03-06T07:24:11.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.588+0000 7fcc7522a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc700cdc10 con 0x7fcc7015ac00 2024-03-06T07:24:11.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.588+0000 7fcc7522a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc7015afe0 con 0x7fcc70070d10 2024-03-06T07:24:11.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.588+0000 7fcc6ed9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcc7015ac00 0x7fcc7006c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.588+0000 7fcc6e59c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fcc700708b0 0x7fcc7015e7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48542/0 (socket says 172.21.15.83:48542) 2024-03-06T07:24:11.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.588+0000 7fcc6e59c640 1 -- 172.21.15.83:0/1017806236 learned_addr learned my addr 172.21.15.83:0/1017806236 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:24:11.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.588+0000 7fcc6ed9d640 1 -- 172.21.15.83:0/1017806236 >> v2:172.21.15.183:3300/0 conn(0x7fcc70070d10 msgr2=0x7fcc7015eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:11.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.588+0000 7fcc6f59e640 1 --2- 172.21.15.83:0/1017806236 >> v2:172.21.15.183:3300/0 conn(0x7fcc70070d10 0x7fcc7015eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.588+0000 7fcc6dd9b640 1 -- 172.21.15.83:0/1017806236 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 412717083 0 0) 0x7fcc700711c0 con 0x7fcc700708b0 2024-03-06T07:24:11.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.588+0000 7fcc6ed9d640 1 --2- 172.21.15.83:0/1017806236 >> v2:172.21.15.183:3300/0 conn(0x7fcc70070d10 0x7fcc7015eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.588+0000 7fcc6ed9d640 1 -- 172.21.15.83:0/1017806236 >> v1:172.21.15.83:6789/0 conn(0x7fcc700708b0 legacy=0x7fcc7015e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.588+0000 7fcc6ed9d640 1 -- 172.21.15.83:0/1017806236 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc700cd000 con 0x7fcc7015ac00 2024-03-06T07:24:11.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.588+0000 7fcc6f59e640 1 --2- 172.21.15.83:0/1017806236 >> v2:172.21.15.183:3300/0 conn(0x7fcc70070d10 0x7fcc7015eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:11.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.589+0000 7fcc6ed9d640 1 --2- 172.21.15.83:0/1017806236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc7015ac00 0x7fcc7006c910 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fcc700cc0a0 tx=0x7fcc6002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=408ddfb0427b01b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.589+0000 7fcc6dd9b640 1 -- 172.21.15.83:0/1017806236 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc600315a0 con 0x7fcc7015ac00 2024-03-06T07:24:11.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.589+0000 7fcc6dd9b640 1 -- 172.21.15.83:0/1017806236 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc60037070 con 0x7fcc7015ac00 2024-03-06T07:24:11.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.589+0000 7fcc6dd9b640 1 -- 172.21.15.83:0/1017806236 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc60031960 con 0x7fcc7015ac00 2024-03-06T07:24:11.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.589+0000 7fcc7522a640 1 -- 172.21.15.83:0/1017806236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc7015ac00 msgr2=0x7fcc7006c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.589+0000 7fcc7522a640 1 --2- 172.21.15.83:0/1017806236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc7015ac00 0x7fcc7006c910 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fcc700cc0a0 tx=0x7fcc6002b840 comp rx=0 tx=0).stop 2024-03-06T07:24:11.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.589+0000 7fcc7522a640 1 -- 172.21.15.83:0/1017806236 shutdown_connections 2024-03-06T07:24:11.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.589+0000 7fcc7522a640 1 --2- 172.21.15.83:0/1017806236 >> v2:172.21.15.183:3300/0 conn(0x7fcc70070d10 0x7fcc7015eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.589+0000 7fcc7522a640 1 --2- 172.21.15.83:0/1017806236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc7015ac00 0x7fcc7006c910 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.589+0000 7fcc7522a640 1 -- 172.21.15.83:0/1017806236 >> 172.21.15.83:0/1017806236 conn(0x7fcc700b1650 msgr2=0x7fcc700b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:11.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.590+0000 7fcc7522a640 1 -- 172.21.15.83:0/1017806236 shutdown_connections 2024-03-06T07:24:11.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.590+0000 7fcc7522a640 1 -- 172.21.15.83:0/1017806236 wait complete. 2024-03-06T07:24:11.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.591+0000 7fcc7522a640 1 Processor -- start 2024-03-06T07:24:11.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.591+0000 7fcc7522a640 1 -- start start 2024-03-06T07:24:11.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.592+0000 7fcc7522a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcc700708b0 0x7fcc7014e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.592+0000 7fcc7522a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc70070d10 0x7fcc70158080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.592+0000 7fcc7522a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcc7015ac00 0x7fcc7015d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.592+0000 7fcc7522a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc700cd8a0 con 0x7fcc70070d10 2024-03-06T07:24:11.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.592+0000 7fcc7522a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc700cd5c0 con 0x7fcc7015ac00 2024-03-06T07:24:11.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.592+0000 7fcc7522a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7006ce50 con 0x7fcc700708b0 2024-03-06T07:24:11.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.592+0000 7fcc6e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc70070d10 0x7fcc70158080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.592+0000 7fcc6e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc70070d10 0x7fcc70158080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33502/0 (socket says 172.21.15.83:33502) 2024-03-06T07:24:11.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.592+0000 7fcc6e59c640 1 -- 172.21.15.83:0/248801043 learned_addr learned my addr 172.21.15.83:0/248801043 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:11.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.592+0000 7fcc6f59e640 1 --2- 172.21.15.83:0/248801043 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcc7015ac00 0x7fcc7015d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.592+0000 7fcc6e59c640 1 -- 172.21.15.83:0/248801043 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcc700708b0 msgr2=0x7fcc7014e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.592+0000 7fcc6ed9d640 1 --2- 172.21.15.83:0/248801043 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcc700708b0 0x7fcc7014e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.592+0000 7fcc6e59c640 1 --2- 172.21.15.83:0/248801043 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcc700708b0 0x7fcc7014e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.592+0000 7fcc6e59c640 1 -- 172.21.15.83:0/248801043 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcc7015ac00 msgr2=0x7fcc7015d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.592+0000 7fcc6e59c640 1 --2- 172.21.15.83:0/248801043 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcc7015ac00 0x7fcc7015d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.592+0000 7fcc6e59c640 1 -- 172.21.15.83:0/248801043 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc7014ea30 con 0x7fcc70070d10 2024-03-06T07:24:11.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.592+0000 7fcc6ed9d640 1 --2- 172.21.15.83:0/248801043 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcc700708b0 0x7fcc7014e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:11.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.592+0000 7fcc6f59e640 1 --2- 172.21.15.83:0/248801043 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcc7015ac00 0x7fcc7015d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:11.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.592+0000 7fcc6e59c640 1 --2- 172.21.15.83:0/248801043 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc70070d10 0x7fcc70158080 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fcc6400c4b0 tx=0x7fcc6400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.592+0000 7fcc577fe640 1 -- 172.21.15.83:0/248801043 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc64018070 con 0x7fcc70070d10 2024-03-06T07:24:11.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.592+0000 7fcc577fe640 1 -- 172.21.15.83:0/248801043 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc64007500 con 0x7fcc70070d10 2024-03-06T07:24:11.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.592+0000 7fcc7522a640 1 -- 172.21.15.83:0/248801043 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc7014f400 con 0x7fcc70070d10 2024-03-06T07:24:11.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.592+0000 7fcc577fe640 1 -- 172.21.15.83:0/248801043 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc64004a20 con 0x7fcc70070d10 2024-03-06T07:24:11.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.592+0000 7fcc7522a640 1 -- 172.21.15.83:0/248801043 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc7014f9c0 con 0x7fcc70070d10 2024-03-06T07:24:11.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.593+0000 7fcc7522a640 1 -- 172.21.15.83:0/248801043 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc700687d0 con 0x7fcc70070d10 2024-03-06T07:24:11.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.594+0000 7fcc577fe640 1 -- 172.21.15.83:0/248801043 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcc6401b070 con 0x7fcc70070d10 2024-03-06T07:24:11.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.594+0000 7fcc577fe640 1 --2- 172.21.15.83:0/248801043 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcc5c06b4d0 0x7fcc5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.594+0000 7fcc577fe640 1 -- 172.21.15.83:0/248801043 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5357+0+0 (secure 0 0 0) 0x7fcc64089310 con 0x7fcc70070d10 2024-03-06T07:24:11.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.594+0000 7fcc6ed9d640 1 --2- 172.21.15.83:0/248801043 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcc5c06b4d0 0x7fcc5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.595+0000 7fcc6ed9d640 1 --2- 172.21.15.83:0/248801043 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcc5c06b4d0 0x7fcc5c06d990 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fcc60036000 tx=0x7fcc60008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.599+0000 7fcc577fe640 1 -- 172.21.15.83:0/248801043 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcc7014ea30 con 0x7fcc70070d10 2024-03-06T07:24:11.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:11.770+0000 7fcc7522a640 1 -- 172.21.15.83:0/248801043 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fcc7004f330 con 0x7fcc70070d10 2024-03-06T07:24:12.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.467+0000 7fcc577fe640 1 -- 172.21.15.83:0/248801043 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v48) v1 ==== 169+0+0 (secure 0 0 0) 0x7fcc64058440 con 0x7fcc70070d10 2024-03-06T07:24:12.469 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'cephfs_data' removed 2024-03-06T07:24:12.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.470+0000 7fcc7522a640 1 -- 172.21.15.83:0/248801043 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcc5c06b4d0 msgr2=0x7fcc5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.470+0000 7fcc7522a640 1 --2- 172.21.15.83:0/248801043 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcc5c06b4d0 0x7fcc5c06d990 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fcc60036000 tx=0x7fcc60008030 comp rx=0 tx=0).stop 2024-03-06T07:24:12.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.470+0000 7fcc7522a640 1 -- 172.21.15.83:0/248801043 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc70070d10 msgr2=0x7fcc70158080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.470+0000 7fcc7522a640 1 --2- 172.21.15.83:0/248801043 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc70070d10 0x7fcc70158080 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fcc6400c4b0 tx=0x7fcc6400c970 comp rx=0 tx=0).stop 2024-03-06T07:24:12.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.470+0000 7fcc7522a640 1 -- 172.21.15.83:0/248801043 shutdown_connections 2024-03-06T07:24:12.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.470+0000 7fcc7522a640 1 --2- 172.21.15.83:0/248801043 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcc5c06b4d0 0x7fcc5c06d990 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.470+0000 7fcc7522a640 1 --2- 172.21.15.83:0/248801043 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcc7015ac00 0x7fcc7015d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.470+0000 7fcc7522a640 1 --2- 172.21.15.83:0/248801043 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc70070d10 0x7fcc70158080 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.470+0000 7fcc7522a640 1 --2- 172.21.15.83:0/248801043 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcc700708b0 0x7fcc7014e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.470+0000 7fcc7522a640 1 -- 172.21.15.83:0/248801043 >> 172.21.15.83:0/248801043 conn(0x7fcc700b1650 msgr2=0x7fcc700b1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:12.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.470+0000 7fcc7522a640 1 -- 172.21.15.83:0/248801043 shutdown_connections 2024-03-06T07:24:12.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.471+0000 7fcc7522a640 1 -- 172.21.15.83:0/248801043 wait complete. 2024-03-06T07:24:12.486 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_fragment_size_max 2024-03-06T07:24:12.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.587+0000 7ffb002fa640 1 Processor -- start 2024-03-06T07:24:12.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.588+0000 7ffb002fa640 1 -- start start 2024-03-06T07:24:12.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.588+0000 7ffb002fa640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffaf81614d0 0x7ffaf80d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.588+0000 7ffb002fa640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ffaf80d2480 0x7ffaf80b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.588+0000 7ffb002fa640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf81618b0 con 0x7ffaf80d1910 2024-03-06T07:24:12.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.588+0000 7ffb002fa640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaf80cf400 con 0x7ffaf80d2480 2024-03-06T07:24:12.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.588+0000 7ffb002fa640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf80cf6a0 con 0x7ffaf81614d0 2024-03-06T07:24:12.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.589+0000 7ffafe897640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ffaf80d2480 0x7ffaf80b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.589+0000 7ffafd895640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ffaf80d1910 0x7ffaf80d1d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48544/0 (socket says 172.21.15.83:48544) 2024-03-06T07:24:12.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.589+0000 7ffafe897640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ffaf80d2480 0x7ffaf80b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33510/0 (socket says 172.21.15.83:33510) 2024-03-06T07:24:12.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.589+0000 7ffafd895640 1 -- 172.21.15.83:0/1479418653 learned_addr learned my addr 172.21.15.83:0/1479418653 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:24:12.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.589+0000 7ffafe897640 1 -- 172.21.15.83:0/1479418653 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffaf81614d0 msgr2=0x7ffaf80d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:12.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.589+0000 7ffafe096640 1 --2- 172.21.15.83:0/1479418653 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffaf81614d0 0x7ffaf80d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.589+0000 7ffafe897640 1 --2- 172.21.15.83:0/1479418653 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffaf81614d0 0x7ffaf80d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.589+0000 7ffafe897640 1 -- 172.21.15.83:0/1479418653 >> v1:172.21.15.83:6789/0 conn(0x7ffaf80d1910 legacy=0x7ffaf80d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.589+0000 7ffafe897640 1 -- 172.21.15.83:0/1479418653 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf80cf980 con 0x7ffaf80d2480 2024-03-06T07:24:12.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.589+0000 7ffafd094640 1 -- 172.21.15.83:0/1479418653 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 966242998 0 0) 0x7ffaf81618b0 con 0x7ffaf80d1910 2024-03-06T07:24:12.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.590+0000 7ffafe897640 1 --2- 172.21.15.83:0/1479418653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffaf80d2480 0x7ffaf80b0e30 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7ffaf0009d90 tx=0x7ffaf002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71bcb6e6f226d0af server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:12.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.590+0000 7ffafd094640 1 -- 172.21.15.83:0/1479418653 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf0039070 con 0x7ffaf80d2480 2024-03-06T07:24:12.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.590+0000 7ffafd094640 1 -- 172.21.15.83:0/1479418653 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaf002cc50 con 0x7ffaf80d2480 2024-03-06T07:24:12.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.590+0000 7ffafd094640 1 -- 172.21.15.83:0/1479418653 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf0034440 con 0x7ffaf80d2480 2024-03-06T07:24:12.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.590+0000 7ffb002fa640 1 -- 172.21.15.83:0/1479418653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffaf80d2480 msgr2=0x7ffaf80b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.590+0000 7ffb002fa640 1 --2- 172.21.15.83:0/1479418653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffaf80d2480 0x7ffaf80b0e30 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7ffaf0009d90 tx=0x7ffaf002bea0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.590+0000 7ffb002fa640 1 -- 172.21.15.83:0/1479418653 shutdown_connections 2024-03-06T07:24:12.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.590+0000 7ffb002fa640 1 --2- 172.21.15.83:0/1479418653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffaf80d2480 0x7ffaf80b0e30 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.590+0000 7ffb002fa640 1 --2- 172.21.15.83:0/1479418653 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffaf81614d0 0x7ffaf80d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.590+0000 7ffb002fa640 1 -- 172.21.15.83:0/1479418653 >> 172.21.15.83:0/1479418653 conn(0x7ffaf80d3d40 msgr2=0x7ffaf80d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:12.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.590+0000 7ffb002fa640 1 -- 172.21.15.83:0/1479418653 shutdown_connections 2024-03-06T07:24:12.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.590+0000 7ffb002fa640 1 -- 172.21.15.83:0/1479418653 wait complete. 2024-03-06T07:24:12.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.591+0000 7ffb002fa640 1 Processor -- start 2024-03-06T07:24:12.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.592+0000 7ffb002fa640 1 -- start start 2024-03-06T07:24:12.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.592+0000 7ffb002fa640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffaf80d1910 0x7ffaf8159950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.592+0000 7ffb002fa640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffaf80d2480 0x7ffaf80b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.592+0000 7ffb002fa640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffaf81614d0 0x7ffaf815d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.592+0000 7ffb002fa640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf80b1370 con 0x7ffaf81614d0 2024-03-06T07:24:12.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.592+0000 7ffb002fa640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf80b0910 con 0x7ffaf80d1910 2024-03-06T07:24:12.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.592+0000 7ffb002fa640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf80b14f0 con 0x7ffaf80d2480 2024-03-06T07:24:12.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.592+0000 7ffafe897640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffaf81614d0 0x7ffaf815d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.592+0000 7ffafe897640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffaf81614d0 0x7ffaf815d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33526/0 (socket says 172.21.15.83:33526) 2024-03-06T07:24:12.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.592+0000 7ffafe897640 1 -- 172.21.15.83:0/3114273096 learned_addr learned my addr 172.21.15.83:0/3114273096 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:12.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.593+0000 7ffafe897640 1 -- 172.21.15.83:0/3114273096 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffaf80d2480 msgr2=0x7ffaf80b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.593+0000 7ffafe897640 1 --2- 172.21.15.83:0/3114273096 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffaf80d2480 0x7ffaf80b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.593+0000 7ffafe897640 1 -- 172.21.15.83:0/3114273096 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffaf80d1910 msgr2=0x7ffaf8159950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.593+0000 7ffafe897640 1 --2- 172.21.15.83:0/3114273096 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffaf80d1910 0x7ffaf8159950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.593+0000 7ffafe897640 1 -- 172.21.15.83:0/3114273096 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf80b7390 con 0x7ffaf81614d0 2024-03-06T07:24:12.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.593+0000 7ffafe897640 1 --2- 172.21.15.83:0/3114273096 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffaf81614d0 0x7ffaf815d330 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7ffaf0009d90 tx=0x7ffaf0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:12.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.593+0000 7ffaeeffd640 1 -- 172.21.15.83:0/3114273096 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf0039050 con 0x7ffaf81614d0 2024-03-06T07:24:12.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.593+0000 7ffaeeffd640 1 -- 172.21.15.83:0/3114273096 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaf0045070 con 0x7ffaf81614d0 2024-03-06T07:24:12.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.593+0000 7ffb002fa640 1 -- 172.21.15.83:0/3114273096 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaf80b8d70 con 0x7ffaf81614d0 2024-03-06T07:24:12.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.593+0000 7ffaeeffd640 1 -- 172.21.15.83:0/3114273096 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf003e610 con 0x7ffaf81614d0 2024-03-06T07:24:12.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.593+0000 7ffb002fa640 1 -- 172.21.15.83:0/3114273096 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaf80b7670 con 0x7ffaf81614d0 2024-03-06T07:24:12.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.594+0000 7ffb002fa640 1 -- 172.21.15.83:0/3114273096 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaf806c160 con 0x7ffaf81614d0 2024-03-06T07:24:12.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.594+0000 7ffaeeffd640 1 -- 172.21.15.83:0/3114273096 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffaf003e7b0 con 0x7ffaf81614d0 2024-03-06T07:24:12.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.595+0000 7ffaeeffd640 1 --2- 172.21.15.83:0/3114273096 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffadc06b4d0 0x7ffadc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.595+0000 7ffaeeffd640 1 -- 172.21.15.83:0/3114273096 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ffaf00acb20 con 0x7ffaf81614d0 2024-03-06T07:24:12.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.595+0000 7ffafe096640 1 --2- 172.21.15.83:0/3114273096 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffadc06b4d0 0x7ffadc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.595+0000 7ffafe096640 1 --2- 172.21.15.83:0/3114273096 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffadc06b4d0 0x7ffadc06d990 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7ffae800a810 tx=0x7ffae8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:12.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.599+0000 7ffaeeffd640 1 -- 172.21.15.83:0/3114273096 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffaf0032e20 con 0x7ffaf81614d0 2024-03-06T07:24:12.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.774+0000 7ffb002fa640 1 -- 172.21.15.83:0/3114273096 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"} v 0) v1 -- 0x7ffaf80ce9f0 con 0x7ffaf81614d0 2024-03-06T07:24:12.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.777+0000 7ffaeeffd640 1 -- 172.21.15.83:0/3114273096 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"}]=0 v14) v1 ==== 108+0+0 (secure 0 0 0) 0x7ffaf007bca0 con 0x7ffaf81614d0 2024-03-06T07:24:12.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.779+0000 7ffb002fa640 1 -- 172.21.15.83:0/3114273096 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffadc06b4d0 msgr2=0x7ffadc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.779+0000 7ffb002fa640 1 --2- 172.21.15.83:0/3114273096 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffadc06b4d0 0x7ffadc06d990 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7ffae800a810 tx=0x7ffae8008040 comp rx=0 tx=0).stop 2024-03-06T07:24:12.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.779+0000 7ffb002fa640 1 -- 172.21.15.83:0/3114273096 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffaf81614d0 msgr2=0x7ffaf815d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.779+0000 7ffb002fa640 1 --2- 172.21.15.83:0/3114273096 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffaf81614d0 0x7ffaf815d330 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7ffaf0009d90 tx=0x7ffaf0004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:12.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.779+0000 7ffb002fa640 1 -- 172.21.15.83:0/3114273096 shutdown_connections 2024-03-06T07:24:12.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.779+0000 7ffb002fa640 1 --2- 172.21.15.83:0/3114273096 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffadc06b4d0 0x7ffadc06d990 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.779+0000 7ffb002fa640 1 --2- 172.21.15.83:0/3114273096 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffaf81614d0 0x7ffaf815d330 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.779+0000 7ffb002fa640 1 --2- 172.21.15.83:0/3114273096 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffaf80d2480 0x7ffaf80b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.779+0000 7ffb002fa640 1 --2- 172.21.15.83:0/3114273096 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffaf80d1910 0x7ffaf8159950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.779+0000 7ffb002fa640 1 -- 172.21.15.83:0/3114273096 >> 172.21.15.83:0/3114273096 conn(0x7ffaf80d3d40 msgr2=0x7ffaf80d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:12.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.779+0000 7ffb002fa640 1 -- 172.21.15.83:0/3114273096 shutdown_connections 2024-03-06T07:24:12.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.779+0000 7ffb002fa640 1 -- 172.21.15.83:0/3114273096 wait complete. 2024-03-06T07:24:12.792 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:24:12.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.892+0000 7ffaccb3e640 1 Processor -- start 2024-03-06T07:24:12.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.892+0000 7ffaccb3e640 1 -- start start 2024-03-06T07:24:12.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.893+0000 7ffaccb3e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffac80b4970 0x7ffac806d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.893+0000 7ffaccb3e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ffac806d5c0 0x7ffac80cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.893+0000 7ffaccb3e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac815ff90 con 0x7ffac815bb40 2024-03-06T07:24:12.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.893+0000 7ffaccb3e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffac80b4dd0 con 0x7ffac806d5c0 2024-03-06T07:24:12.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.893+0000 7ffaccb3e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffac806c910 con 0x7ffac80b4970 2024-03-06T07:24:12.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.894+0000 7ffac5d9b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffac80b4970 0x7ffac806d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.894+0000 7ffac5d9b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffac80b4970 0x7ffac806d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:57950/0 (socket says 172.21.15.83:57950) 2024-03-06T07:24:12.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.894+0000 7ffac5d9b640 1 -- 172.21.15.83:0/880226187 learned_addr learned my addr 172.21.15.83:0/880226187 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:12.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.894+0000 7ffac6d9d640 1 --2- 172.21.15.83:0/880226187 >> v2:172.21.15.183:3300/0 conn(0x7ffac806d5c0 0x7ffac80cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.894+0000 7ffac559a640 1 -- 172.21.15.83:0/880226187 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2085118685 0 0) 0x7ffac815ff90 con 0x7ffac815bb40 2024-03-06T07:24:12.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.894+0000 7ffac559a640 1 -- 172.21.15.83:0/880226187 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffab0003520 con 0x7ffac815bb40 2024-03-06T07:24:12.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.894+0000 7ffac5d9b640 1 -- 172.21.15.83:0/880226187 >> v2:172.21.15.183:3300/0 conn(0x7ffac806d5c0 msgr2=0x7ffac80cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:12.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.894+0000 7ffac5d9b640 1 --2- 172.21.15.83:0/880226187 >> v2:172.21.15.183:3300/0 conn(0x7ffac806d5c0 0x7ffac80cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.894+0000 7ffac5d9b640 1 -- 172.21.15.83:0/880226187 >> v1:172.21.15.183:6789/0 conn(0x7ffac815bb40 legacy=0x7ffac815f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.894+0000 7ffac5d9b640 1 -- 172.21.15.83:0/880226187 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac8160270 con 0x7ffac80b4970 2024-03-06T07:24:12.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.894+0000 7ffac6d9d640 1 --2- 172.21.15.83:0/880226187 >> v2:172.21.15.183:3300/0 conn(0x7ffac806d5c0 0x7ffac80cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:12.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.895+0000 7ffac5d9b640 1 --2- 172.21.15.83:0/880226187 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffac80b4970 0x7ffac806d080 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ffabc009870 tx=0x7ffabc02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=274dcd022c65f431 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:12.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.895+0000 7ffac559a640 1 -- 172.21.15.83:0/880226187 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffabc034070 con 0x7ffac80b4970 2024-03-06T07:24:12.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.895+0000 7ffac559a640 1 -- 172.21.15.83:0/880226187 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffabc030840 con 0x7ffac80b4970 2024-03-06T07:24:12.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.895+0000 7ffac559a640 1 -- 172.21.15.83:0/880226187 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffabc030c10 con 0x7ffac80b4970 2024-03-06T07:24:12.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.895+0000 7ffaccb3e640 1 -- 172.21.15.83:0/880226187 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffac80b4970 msgr2=0x7ffac806d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.895+0000 7ffaccb3e640 1 --2- 172.21.15.83:0/880226187 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffac80b4970 0x7ffac806d080 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ffabc009870 tx=0x7ffabc02b980 comp rx=0 tx=0).stop 2024-03-06T07:24:12.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.895+0000 7ffaccb3e640 1 -- 172.21.15.83:0/880226187 shutdown_connections 2024-03-06T07:24:12.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.895+0000 7ffaccb3e640 1 --2- 172.21.15.83:0/880226187 >> v2:172.21.15.183:3300/0 conn(0x7ffac806d5c0 0x7ffac80cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.895+0000 7ffaccb3e640 1 --2- 172.21.15.83:0/880226187 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffac80b4970 0x7ffac806d080 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.895+0000 7ffaccb3e640 1 -- 172.21.15.83:0/880226187 >> 172.21.15.83:0/880226187 conn(0x7ffac806b190 msgr2=0x7ffac806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:12.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.896+0000 7ffaccb3e640 1 -- 172.21.15.83:0/880226187 shutdown_connections 2024-03-06T07:24:12.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.896+0000 7ffaccb3e640 1 -- 172.21.15.83:0/880226187 wait complete. 2024-03-06T07:24:12.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.897+0000 7ffaccb3e640 1 Processor -- start 2024-03-06T07:24:12.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.897+0000 7ffaccb3e640 1 -- start start 2024-03-06T07:24:12.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.897+0000 7ffaccb3e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffac806d5c0 0x7ffac81f5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.897+0000 7ffaccb3e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffac80b4970 0x7ffac8149f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.898+0000 7ffaccb3e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffac815bb40 0x7ffac81512e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.898+0000 7ffaccb3e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac815bf20 con 0x7ffac815bb40 2024-03-06T07:24:12.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.898+0000 7ffaccb3e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac806cb60 con 0x7ffac80b4970 2024-03-06T07:24:12.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.898+0000 7ffaccb3e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffac815c0a0 con 0x7ffac806d5c0 2024-03-06T07:24:12.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.898+0000 7ffac6d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffac815bb40 0x7ffac81512e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.898+0000 7ffac6d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffac815bb40 0x7ffac81512e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33538/0 (socket says 172.21.15.83:33538) 2024-03-06T07:24:12.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.898+0000 7ffac6d9d640 1 -- 172.21.15.83:0/260768777 learned_addr learned my addr 172.21.15.83:0/260768777 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:12.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.898+0000 7ffac659c640 1 --2- 172.21.15.83:0/260768777 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffac806d5c0 0x7ffac81f5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.898+0000 7ffac5d9b640 1 --2- 172.21.15.83:0/260768777 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffac80b4970 0x7ffac8149f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.898+0000 7ffac6d9d640 1 -- 172.21.15.83:0/260768777 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffac806d5c0 msgr2=0x7ffac81f5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.898+0000 7ffac6d9d640 1 --2- 172.21.15.83:0/260768777 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffac806d5c0 0x7ffac81f5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.898+0000 7ffac6d9d640 1 -- 172.21.15.83:0/260768777 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffac80b4970 msgr2=0x7ffac8149f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.898+0000 7ffac6d9d640 1 --2- 172.21.15.83:0/260768777 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffac80b4970 0x7ffac8149f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.898+0000 7ffac6d9d640 1 -- 172.21.15.83:0/260768777 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac8151820 con 0x7ffac815bb40 2024-03-06T07:24:12.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.898+0000 7ffac659c640 1 --2- 172.21.15.83:0/260768777 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffac806d5c0 0x7ffac81f5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:12.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.898+0000 7ffac6d9d640 1 --2- 172.21.15.83:0/260768777 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffac815bb40 0x7ffac81512e0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7ffac000e930 tx=0x7ffac000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:12.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.898+0000 7ffab6ffd640 1 -- 172.21.15.83:0/260768777 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac0017070 con 0x7ffac815bb40 2024-03-06T07:24:12.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.898+0000 7ffab6ffd640 1 -- 172.21.15.83:0/260768777 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffac000ca00 con 0x7ffac815bb40 2024-03-06T07:24:12.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.898+0000 7ffab6ffd640 1 -- 172.21.15.83:0/260768777 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac000cdd0 con 0x7ffac815bb40 2024-03-06T07:24:12.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.898+0000 7ffaccb3e640 1 -- 172.21.15.83:0/260768777 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffac81f5bb0 con 0x7ffac815bb40 2024-03-06T07:24:12.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.898+0000 7ffaccb3e640 1 -- 172.21.15.83:0/260768777 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffac81f6170 con 0x7ffac815bb40 2024-03-06T07:24:12.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.899+0000 7ffaccb3e640 1 -- 172.21.15.83:0/260768777 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffac81f63b0 con 0x7ffac815bb40 2024-03-06T07:24:12.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.900+0000 7ffab6ffd640 1 -- 172.21.15.83:0/260768777 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffac0013470 con 0x7ffac815bb40 2024-03-06T07:24:12.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.900+0000 7ffab6ffd640 1 --2- 172.21.15.83:0/260768777 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffaa806b4d0 0x7ffaa806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.900+0000 7ffac659c640 1 --2- 172.21.15.83:0/260768777 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffaa806b4d0 0x7ffaa806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.900+0000 7ffab6ffd640 1 -- 172.21.15.83:0/260768777 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ffac0089ad0 con 0x7ffac815bb40 2024-03-06T07:24:12.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.901+0000 7ffac659c640 1 --2- 172.21.15.83:0/260768777 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffaa806b4d0 0x7ffaa806d990 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ffab8001d70 tx=0x7ffab8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:12.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:12.904+0000 7ffab6ffd640 1 -- 172.21.15.83:0/260768777 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffac81f63b0 con 0x7ffac815bb40 2024-03-06T07:24:13.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.075+0000 7ffaccb3e640 1 -- 172.21.15.83:0/260768777 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7ffac80732b0 con 0x7ffac815bb40 2024-03-06T07:24:13.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.078+0000 7ffab6ffd640 1 -- 172.21.15.83:0/260768777 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v15) v1 ==== 104+0+0 (secure 0 0 0) 0x7ffac0058c90 con 0x7ffac815bb40 2024-03-06T07:24:13.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.080+0000 7ffaccb3e640 1 -- 172.21.15.83:0/260768777 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffaa806b4d0 msgr2=0x7ffaa806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.080+0000 7ffaccb3e640 1 --2- 172.21.15.83:0/260768777 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffaa806b4d0 0x7ffaa806d990 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ffab8001d70 tx=0x7ffab8009040 comp rx=0 tx=0).stop 2024-03-06T07:24:13.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.080+0000 7ffaccb3e640 1 -- 172.21.15.83:0/260768777 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffac815bb40 msgr2=0x7ffac81512e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.080+0000 7ffaccb3e640 1 --2- 172.21.15.83:0/260768777 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffac815bb40 0x7ffac81512e0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7ffac000e930 tx=0x7ffac000edf0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.081+0000 7ffaccb3e640 1 -- 172.21.15.83:0/260768777 shutdown_connections 2024-03-06T07:24:13.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.081+0000 7ffaccb3e640 1 --2- 172.21.15.83:0/260768777 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffaa806b4d0 0x7ffaa806d990 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.081+0000 7ffaccb3e640 1 --2- 172.21.15.83:0/260768777 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffac815bb40 0x7ffac81512e0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.081+0000 7ffaccb3e640 1 --2- 172.21.15.83:0/260768777 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffac80b4970 0x7ffac8149f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.081+0000 7ffaccb3e640 1 --2- 172.21.15.83:0/260768777 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffac806d5c0 0x7ffac81f5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.081+0000 7ffaccb3e640 1 -- 172.21.15.83:0/260768777 >> 172.21.15.83:0/260768777 conn(0x7ffac806b190 msgr2=0x7ffac81547a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:13.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.081+0000 7ffaccb3e640 1 -- 172.21.15.83:0/260768777 shutdown_connections 2024-03-06T07:24:13.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.081+0000 7ffaccb3e640 1 -- 172.21.15.83:0/260768777 wait complete. 2024-03-06T07:24:13.093 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit' 2024-03-06T07:24:13.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.196+0000 7f5019f9a640 1 Processor -- start 2024-03-06T07:24:13.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.196+0000 7f5019f9a640 1 -- start start 2024-03-06T07:24:13.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.196+0000 7f5019f9a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f50141424f0 0x7f50140cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.196+0000 7f5019f9a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f501406da40 0x7f501415cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.196+0000 7f5019f9a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50140cc980 con 0x7f50140cd7a0 2024-03-06T07:24:13.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.196+0000 7f5019f9a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50140cd220 con 0x7f501406da40 2024-03-06T07:24:13.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.196+0000 7f5019f9a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50140ccc60 con 0x7f50141424f0 2024-03-06T07:24:13.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.197+0000 7f5013fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f501406da40 0x7f501415cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.197+0000 7f5012ffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f50140cd7a0 0x7f501406d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48546/0 (socket says 172.21.15.83:48546) 2024-03-06T07:24:13.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.197+0000 7f5012ffd640 1 -- 172.21.15.83:0/2571265587 learned_addr learned my addr 172.21.15.83:0/2571265587 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:24:13.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.197+0000 7f50137fe640 1 --2- 172.21.15.83:0/2571265587 >> v2:172.21.15.183:3300/0 conn(0x7f50141424f0 0x7f50140cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.197+0000 7f50127fc640 1 -- 172.21.15.83:0/2571265587 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3300236903 0 0) 0x7f50140cc980 con 0x7f50140cd7a0 2024-03-06T07:24:13.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.197+0000 7f50127fc640 1 -- 172.21.15.83:0/2571265587 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ffc003520 con 0x7f50140cd7a0 2024-03-06T07:24:13.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.197+0000 7f5013fff640 1 -- 172.21.15.83:0/2571265587 >> v2:172.21.15.183:3300/0 conn(0x7f50141424f0 msgr2=0x7f50140cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:13.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.197+0000 7f5013fff640 1 --2- 172.21.15.83:0/2571265587 >> v2:172.21.15.183:3300/0 conn(0x7f50141424f0 0x7f50140cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.197+0000 7f5013fff640 1 -- 172.21.15.83:0/2571265587 >> v1:172.21.15.83:6789/0 conn(0x7f50140cd7a0 legacy=0x7f501406d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.197+0000 7f5013fff640 1 -- 172.21.15.83:0/2571265587 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50140ccf40 con 0x7f501406da40 2024-03-06T07:24:13.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.198+0000 7f5013fff640 1 --2- 172.21.15.83:0/2571265587 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f501406da40 0x7f501415cef0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f501406c6d0 tx=0x7f500c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=165b1fda17a41b99 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.198+0000 7f50127fc640 1 -- 172.21.15.83:0/2571265587 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f500c039070 con 0x7f501406da40 2024-03-06T07:24:13.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.198+0000 7f50127fc640 1 -- 172.21.15.83:0/2571265587 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f500c02cc80 con 0x7f501406da40 2024-03-06T07:24:13.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.198+0000 7f50127fc640 1 -- 172.21.15.83:0/2571265587 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f500c034640 con 0x7f501406da40 2024-03-06T07:24:13.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.198+0000 7f5019f9a640 1 -- 172.21.15.83:0/2571265587 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f501406da40 msgr2=0x7f501415cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.198+0000 7f5019f9a640 1 --2- 172.21.15.83:0/2571265587 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f501406da40 0x7f501415cef0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f501406c6d0 tx=0x7f500c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.198+0000 7f5019f9a640 1 -- 172.21.15.83:0/2571265587 shutdown_connections 2024-03-06T07:24:13.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.198+0000 7f5019f9a640 1 --2- 172.21.15.83:0/2571265587 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f501406da40 0x7f501415cef0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.198+0000 7f5019f9a640 1 --2- 172.21.15.83:0/2571265587 >> v2:172.21.15.183:3300/0 conn(0x7f50141424f0 0x7f50140cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.198+0000 7f5019f9a640 1 -- 172.21.15.83:0/2571265587 >> 172.21.15.83:0/2571265587 conn(0x7f50140cf8a0 msgr2=0x7f50140cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:13.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.198+0000 7f5019f9a640 1 -- 172.21.15.83:0/2571265587 shutdown_connections 2024-03-06T07:24:13.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.198+0000 7f5019f9a640 1 -- 172.21.15.83:0/2571265587 wait complete. 2024-03-06T07:24:13.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.200+0000 7f5019f9a640 1 Processor -- start 2024-03-06T07:24:13.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.200+0000 7f5019f9a640 1 -- start start 2024-03-06T07:24:13.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.200+0000 7f5019f9a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f501406da40 0x7f5014159ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.200+0000 7f5019f9a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50140cd7a0 0x7f5014157270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.200+0000 7f5019f9a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f50141424f0 0x7f50140ac7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.200+0000 7f5019f9a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5014071ce0 con 0x7f50140cd7a0 2024-03-06T07:24:13.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.200+0000 7f5019f9a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5014071b60 con 0x7f50141424f0 2024-03-06T07:24:13.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.200+0000 7f5019f9a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5014071e60 con 0x7f501406da40 2024-03-06T07:24:13.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.201+0000 7f5012ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50140cd7a0 0x7f5014157270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.201+0000 7f5012ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50140cd7a0 0x7f5014157270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33546/0 (socket says 172.21.15.83:33546) 2024-03-06T07:24:13.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.201+0000 7f5012ffd640 1 -- 172.21.15.83:0/3823603889 learned_addr learned my addr 172.21.15.83:0/3823603889 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:13.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.201+0000 7f50137fe640 1 --2- 172.21.15.83:0/3823603889 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f501406da40 0x7f5014159ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.201+0000 7f5013fff640 1 --2- 172.21.15.83:0/3823603889 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f50141424f0 0x7f50140ac7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.201+0000 7f5012ffd640 1 -- 172.21.15.83:0/3823603889 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f501406da40 msgr2=0x7f5014159ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.201+0000 7f5012ffd640 1 --2- 172.21.15.83:0/3823603889 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f501406da40 0x7f5014159ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.201+0000 7f5012ffd640 1 -- 172.21.15.83:0/3823603889 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f50141424f0 msgr2=0x7f50140ac7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.201+0000 7f5012ffd640 1 --2- 172.21.15.83:0/3823603889 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f50141424f0 0x7f50140ac7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.201+0000 7f5012ffd640 1 -- 172.21.15.83:0/3823603889 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50140aa3b0 con 0x7f50140cd7a0 2024-03-06T07:24:13.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.201+0000 7f5012ffd640 1 --2- 172.21.15.83:0/3823603889 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50140cd7a0 0x7f5014157270 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f500800c430 tx=0x7f500800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.201+0000 7f4ffbfff640 1 -- 172.21.15.83:0/3823603889 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5008017070 con 0x7f50140cd7a0 2024-03-06T07:24:13.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.201+0000 7f4ffbfff640 1 -- 172.21.15.83:0/3823603889 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5008005ce0 con 0x7f50140cd7a0 2024-03-06T07:24:13.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.201+0000 7f5019f9a640 1 -- 172.21.15.83:0/3823603889 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50140aa690 con 0x7f50140cd7a0 2024-03-06T07:24:13.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.201+0000 7f4ffbfff640 1 -- 172.21.15.83:0/3823603889 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50080072f0 con 0x7f50140cd7a0 2024-03-06T07:24:13.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.201+0000 7f5019f9a640 1 -- 172.21.15.83:0/3823603889 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50140aac50 con 0x7f50140cd7a0 2024-03-06T07:24:13.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.202+0000 7f5019f9a640 1 -- 172.21.15.83:0/3823603889 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f501404f330 con 0x7f50140cd7a0 2024-03-06T07:24:13.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.203+0000 7f4ffbfff640 1 -- 172.21.15.83:0/3823603889 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f500801a070 con 0x7f50140cd7a0 2024-03-06T07:24:13.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.203+0000 7f4ffbfff640 1 --2- 172.21.15.83:0/3823603889 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f500006b4d0 0x7f500006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.203+0000 7f4ffbfff640 1 -- 172.21.15.83:0/3823603889 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f50080893e0 con 0x7f50140cd7a0 2024-03-06T07:24:13.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.203+0000 7f50137fe640 1 --2- 172.21.15.83:0/3823603889 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f500006b4d0 0x7f500006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.204+0000 7f50137fe640 1 --2- 172.21.15.83:0/3823603889 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f500006b4d0 0x7f500006d990 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f5004006fd0 tx=0x7f5004008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.207+0000 7f4ffbfff640 1 -- 172.21.15.83:0/3823603889 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5008058560 con 0x7f50140cd7a0 2024-03-06T07:24:13.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.446+0000 7f5019f9a640 1 -- 172.21.15.83:0/3823603889 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]} v 0) v1 -- 0x7f50140d0620 con 0x7f50140cd7a0 2024-03-06T07:24:13.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.468+0000 7f4ffbfff640 1 -- 172.21.15.83:0/3823603889 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]}]=0 v144) v1 ==== 133+0+0 (secure 0 0 0) 0x7f5008057e60 con 0x7f50140cd7a0 2024-03-06T07:24:13.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.470+0000 7f5019f9a640 1 -- 172.21.15.83:0/3823603889 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f500006b4d0 msgr2=0x7f500006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.470+0000 7f5019f9a640 1 --2- 172.21.15.83:0/3823603889 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f500006b4d0 0x7f500006d990 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f5004006fd0 tx=0x7f5004008040 comp rx=0 tx=0).stop 2024-03-06T07:24:13.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.470+0000 7f5019f9a640 1 -- 172.21.15.83:0/3823603889 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50140cd7a0 msgr2=0x7f5014157270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.470+0000 7f5019f9a640 1 --2- 172.21.15.83:0/3823603889 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50140cd7a0 0x7f5014157270 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f500800c430 tx=0x7f500800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.470+0000 7f5019f9a640 1 -- 172.21.15.83:0/3823603889 shutdown_connections 2024-03-06T07:24:13.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.470+0000 7f5019f9a640 1 --2- 172.21.15.83:0/3823603889 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f500006b4d0 0x7f500006d990 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.470+0000 7f5019f9a640 1 --2- 172.21.15.83:0/3823603889 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f50141424f0 0x7f50140ac7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.470+0000 7f5019f9a640 1 --2- 172.21.15.83:0/3823603889 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50140cd7a0 0x7f5014157270 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.470+0000 7f5019f9a640 1 --2- 172.21.15.83:0/3823603889 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f501406da40 0x7f5014159ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.470+0000 7f5019f9a640 1 -- 172.21.15.83:0/3823603889 >> 172.21.15.83:0/3823603889 conn(0x7f50140cf8a0 msgr2=0x7f50140ce5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:13.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.470+0000 7f5019f9a640 1 -- 172.21.15.83:0/3823603889 shutdown_connections 2024-03-06T07:24:13.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.470+0000 7f5019f9a640 1 -- 172.21.15.83:0/3823603889 wait complete. 2024-03-06T07:24:13.483 INFO:tasks.cephfs_test_runner:That the directory fragment size cannot exceed mds_bal_fragment_size_max (using a limit of 50 in all configurations). ... ok 2024-03-06T07:24:13.483 INFO:tasks.cephfs_test_runner:Starting test: test_dirfrag_limit_fragmented (tasks.cephfs.test_strays.TestStrays) That fragmentation (forced) will allow more entries to be created. 2024-03-06T07:24:13.483 INFO:tasks.cephfs_test_runner:test_dirfrag_limit_fragmented (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:24:13.484 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:13.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.582+0000 7fb8d9b56640 1 Processor -- start 2024-03-06T07:24:13.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.582+0000 7fb8d9b56640 1 -- start start 2024-03-06T07:24:13.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.583+0000 7fb8d9b56640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb8d41515e0 0x7fb8d40cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.583+0000 7fb8d9b56640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb8d415b430 0x7fb8d4071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.583+0000 7fb8d9b56640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8d415b8b0 con 0x7fb8d40cd7a0 2024-03-06T07:24:13.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.583+0000 7fb8d9b56640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8d4070a70 con 0x7fb8d415b430 2024-03-06T07:24:13.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.583+0000 7fb8d9b56640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d4070d50 con 0x7fb8d41515e0 2024-03-06T07:24:13.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.583+0000 7fb8d2ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb8d415b430 0x7fb8d4071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.583+0000 7fb8d2ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb8d415b430 0x7fb8d4071830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33548/0 (socket says 172.21.15.83:33548) 2024-03-06T07:24:13.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.583+0000 7fb8d2ffd640 1 -- 172.21.15.83:0/2743404056 learned_addr learned my addr 172.21.15.83:0/2743404056 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:13.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.583+0000 7fb8d37fe640 1 --2- 172.21.15.83:0/2743404056 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb8d41515e0 0x7fb8d40cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.583+0000 7fb8d2ffd640 1 -- 172.21.15.83:0/2743404056 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb8d41515e0 msgr2=0x7fb8d40cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:13.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.583+0000 7fb8d2ffd640 1 --2- 172.21.15.83:0/2743404056 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb8d41515e0 0x7fb8d40cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.583+0000 7fb8d2ffd640 1 -- 172.21.15.83:0/2743404056 >> v1:172.21.15.183:6789/0 conn(0x7fb8d40cd7a0 legacy=0x7fb8d40cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.583+0000 7fb8d2ffd640 1 -- 172.21.15.83:0/2743404056 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8d4071030 con 0x7fb8d415b430 2024-03-06T07:24:13.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.584+0000 7fb8d2ffd640 1 --2- 172.21.15.83:0/2743404056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb8d415b430 0x7fb8d4071830 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fb8c8009870 tx=0x7fb8c802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e11b3b2801284a9a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.584+0000 7fb8d27fc640 1 -- 172.21.15.83:0/2743404056 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c8034070 con 0x7fb8d415b430 2024-03-06T07:24:13.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.584+0000 7fb8d27fc640 1 -- 172.21.15.83:0/2743404056 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8c8030840 con 0x7fb8d415b430 2024-03-06T07:24:13.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.584+0000 7fb8d27fc640 1 -- 172.21.15.83:0/2743404056 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c8030c10 con 0x7fb8d415b430 2024-03-06T07:24:13.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.584+0000 7fb8d9b56640 1 -- 172.21.15.83:0/2743404056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb8d415b430 msgr2=0x7fb8d4071830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.584+0000 7fb8d9b56640 1 --2- 172.21.15.83:0/2743404056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb8d415b430 0x7fb8d4071830 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fb8c8009870 tx=0x7fb8c802b980 comp rx=0 tx=0).stop 2024-03-06T07:24:13.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.584+0000 7fb8d9b56640 1 -- 172.21.15.83:0/2743404056 shutdown_connections 2024-03-06T07:24:13.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.584+0000 7fb8d9b56640 1 --2- 172.21.15.83:0/2743404056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb8d415b430 0x7fb8d4071830 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.584+0000 7fb8d9b56640 1 --2- 172.21.15.83:0/2743404056 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb8d41515e0 0x7fb8d40cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.584+0000 7fb8d9b56640 1 -- 172.21.15.83:0/2743404056 >> 172.21.15.83:0/2743404056 conn(0x7fb8d40cf8a0 msgr2=0x7fb8d40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:13.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.584+0000 7fb8d9b56640 1 -- 172.21.15.83:0/2743404056 shutdown_connections 2024-03-06T07:24:13.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.584+0000 7fb8d9b56640 1 -- 172.21.15.83:0/2743404056 wait complete. 2024-03-06T07:24:13.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.585+0000 7fb8d9b56640 1 Processor -- start 2024-03-06T07:24:13.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.586+0000 7fb8d9b56640 1 -- start start 2024-03-06T07:24:13.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.586+0000 7fb8d9b56640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb8d40cd7a0 0x7fb8d414a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.586+0000 7fb8d9b56640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb8d41515e0 0x7fb8d415da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.586+0000 7fb8d9b56640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb8d415b430 0x7fb8d41f5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.586+0000 7fb8d9b56640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d4071ef0 con 0x7fb8d41515e0 2024-03-06T07:24:13.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.586+0000 7fb8d9b56640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d4071d70 con 0x7fb8d40cd7a0 2024-03-06T07:24:13.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.586+0000 7fb8d9b56640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d40cc3a0 con 0x7fb8d415b430 2024-03-06T07:24:13.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.586+0000 7fb8d2ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb8d41515e0 0x7fb8d415da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.586+0000 7fb8d2ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb8d41515e0 0x7fb8d415da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33558/0 (socket says 172.21.15.83:33558) 2024-03-06T07:24:13.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.587+0000 7fb8d2ffd640 1 -- 172.21.15.83:0/1347015854 learned_addr learned my addr 172.21.15.83:0/1347015854 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:13.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.587+0000 7fb8d37fe640 1 --2- 172.21.15.83:0/1347015854 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb8d40cd7a0 0x7fb8d414a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.587+0000 7fb8d3fff640 1 --2- 172.21.15.83:0/1347015854 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb8d415b430 0x7fb8d41f5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.587+0000 7fb8d2ffd640 1 -- 172.21.15.83:0/1347015854 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb8d415b430 msgr2=0x7fb8d41f5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.587+0000 7fb8d2ffd640 1 --2- 172.21.15.83:0/1347015854 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb8d415b430 0x7fb8d41f5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.587+0000 7fb8d2ffd640 1 -- 172.21.15.83:0/1347015854 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb8d40cd7a0 msgr2=0x7fb8d414a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.587+0000 7fb8d2ffd640 1 --2- 172.21.15.83:0/1347015854 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb8d40cd7a0 0x7fb8d414a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.587+0000 7fb8d2ffd640 1 -- 172.21.15.83:0/1347015854 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8d40c8e00 con 0x7fb8d41515e0 2024-03-06T07:24:13.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.587+0000 7fb8d3fff640 1 --2- 172.21.15.83:0/1347015854 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb8d415b430 0x7fb8d41f5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:13.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.587+0000 7fb8d37fe640 1 --2- 172.21.15.83:0/1347015854 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb8d40cd7a0 0x7fb8d414a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:13.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.587+0000 7fb8d2ffd640 1 --2- 172.21.15.83:0/1347015854 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb8d41515e0 0x7fb8d415da50 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fb8c802b9c0 tx=0x7fb8c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.587+0000 7fb8bbfff640 1 -- 172.21.15.83:0/1347015854 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c8034040 con 0x7fb8d41515e0 2024-03-06T07:24:13.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.587+0000 7fb8bbfff640 1 -- 172.21.15.83:0/1347015854 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8c8039d90 con 0x7fb8d41515e0 2024-03-06T07:24:13.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.587+0000 7fb8d9b56640 1 -- 172.21.15.83:0/1347015854 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8d40c90e0 con 0x7fb8d41515e0 2024-03-06T07:24:13.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.587+0000 7fb8bbfff640 1 -- 172.21.15.83:0/1347015854 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c8043610 con 0x7fb8d41515e0 2024-03-06T07:24:13.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.587+0000 7fb8d9b56640 1 -- 172.21.15.83:0/1347015854 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8d40c96a0 con 0x7fb8d41515e0 2024-03-06T07:24:13.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.588+0000 7fb8d9b56640 1 -- 172.21.15.83:0/1347015854 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8d4068840 con 0x7fb8d41515e0 2024-03-06T07:24:13.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.589+0000 7fb8bbfff640 1 -- 172.21.15.83:0/1347015854 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb8c804a060 con 0x7fb8d41515e0 2024-03-06T07:24:13.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.589+0000 7fb8bbfff640 1 --2- 172.21.15.83:0/1347015854 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb8c006b500 0x7fb8c006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.589+0000 7fb8bbfff640 1 -- 172.21.15.83:0/1347015854 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fb8c80abe40 con 0x7fb8d41515e0 2024-03-06T07:24:13.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.589+0000 7fb8d37fe640 1 --2- 172.21.15.83:0/1347015854 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb8c006b500 0x7fb8c006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.589+0000 7fb8d37fe640 1 --2- 172.21.15.83:0/1347015854 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb8c006b500 0x7fb8c006d9c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb8c4006fd0 tx=0x7fb8c4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.593+0000 7fb8bbfff640 1 -- 172.21.15.83:0/1347015854 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb8c803e070 con 0x7fb8d41515e0 2024-03-06T07:24:13.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.767+0000 7fb8d9b56640 1 -- 172.21.15.83:0/1347015854 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8d404f330 con 0x7fb8d41515e0 2024-03-06T07:24:13.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.768+0000 7fb8bbfff640 1 -- 172.21.15.83:0/1347015854 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) v1 ==== 74+0+13030 (secure 0 0 0) 0x7fb8c802d330 con 0x7fb8d41515e0 2024-03-06T07:24:13.769 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:24:13.770 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":48,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:24:12.466851+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.83:6835/534553538":"2024-03-07T07:24:09.462329+0000","172.21.15.83:6834/534553538":"2024-03-07T07:24:09.462329+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:13.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.769+0000 7fb8d9b56640 1 -- 172.21.15.83:0/1347015854 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb8c006b500 msgr2=0x7fb8c006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.769+0000 7fb8d9b56640 1 --2- 172.21.15.83:0/1347015854 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb8c006b500 0x7fb8c006d9c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb8c4006fd0 tx=0x7fb8c4008040 comp rx=0 tx=0).stop 2024-03-06T07:24:13.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.769+0000 7fb8d9b56640 1 -- 172.21.15.83:0/1347015854 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb8d41515e0 msgr2=0x7fb8d415da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.769+0000 7fb8d9b56640 1 --2- 172.21.15.83:0/1347015854 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb8d41515e0 0x7fb8d415da50 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fb8c802b9c0 tx=0x7fb8c80046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.769+0000 7fb8d9b56640 1 -- 172.21.15.83:0/1347015854 shutdown_connections 2024-03-06T07:24:13.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.770+0000 7fb8d9b56640 1 --2- 172.21.15.83:0/1347015854 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb8c006b500 0x7fb8c006d9c0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.770+0000 7fb8d9b56640 1 --2- 172.21.15.83:0/1347015854 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb8d415b430 0x7fb8d41f5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.770+0000 7fb8d9b56640 1 --2- 172.21.15.83:0/1347015854 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb8d41515e0 0x7fb8d415da50 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.770+0000 7fb8d9b56640 1 --2- 172.21.15.83:0/1347015854 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb8d40cd7a0 0x7fb8d414a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.770+0000 7fb8d9b56640 1 -- 172.21.15.83:0/1347015854 >> 172.21.15.83:0/1347015854 conn(0x7fb8d40cf8a0 msgr2=0x7fb8d414fac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:13.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.770+0000 7fb8d9b56640 1 -- 172.21.15.83:0/1347015854 shutdown_connections 2024-03-06T07:24:13.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.770+0000 7fb8d9b56640 1 -- 172.21.15.83:0/1347015854 wait complete. 2024-03-06T07:24:13.782 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:13.783 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:13.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.884+0000 7fd6fa837640 1 Processor -- start 2024-03-06T07:24:13.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.884+0000 7fd6fa837640 1 -- start start 2024-03-06T07:24:13.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.884+0000 7fd6fa837640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fd6f41586e0 0x7fd6f40aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.884+0000 7fd6fa837640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd6f40aef80 0x7fd6f40cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.884+0000 7fd6fa837640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6f40d03f0 con 0x7fd6f40cffd0 2024-03-06T07:24:13.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.884+0000 7fd6fa837640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6f4158ac0 con 0x7fd6f41586e0 2024-03-06T07:24:13.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.884+0000 7fd6fa837640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f415b2a0 con 0x7fd6f40aef80 2024-03-06T07:24:13.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.885+0000 7fd6f3fff640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fd6f41586e0 0x7fd6f40aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.885+0000 7fd6f8dd4640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fd6f40cffd0 0x7fd6f406da20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:56080/0 (socket says 172.21.15.83:56080) 2024-03-06T07:24:13.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.885+0000 7fd6f37fe640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd6f40aef80 0x7fd6f40cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.885+0000 7fd6f8dd4640 1 -- 172.21.15.83:0/3752276677 learned_addr learned my addr 172.21.15.83:0/3752276677 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:24:13.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.885+0000 7fd6f37fe640 1 -- 172.21.15.83:0/3752276677 >> v2:172.21.15.183:3300/0 conn(0x7fd6f41586e0 msgr2=0x7fd6f40aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:13.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.885+0000 7fd6f37fe640 1 --2- 172.21.15.83:0/3752276677 >> v2:172.21.15.183:3300/0 conn(0x7fd6f41586e0 0x7fd6f40aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.885+0000 7fd6f37fe640 1 -- 172.21.15.83:0/3752276677 >> v1:172.21.15.183:6789/0 conn(0x7fd6f40cffd0 legacy=0x7fd6f406da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.885+0000 7fd6f37fe640 1 -- 172.21.15.83:0/3752276677 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f415b540 con 0x7fd6f40aef80 2024-03-06T07:24:13.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.885+0000 7fd6f3fff640 1 --2- 172.21.15.83:0/3752276677 >> v2:172.21.15.183:3300/0 conn(0x7fd6f41586e0 0x7fd6f40aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:13.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.886+0000 7fd6f37fe640 1 --2- 172.21.15.83:0/3752276677 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd6f40aef80 0x7fd6f40cfa90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd6e80097a0 tx=0x7fd6e802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33e7b59b92e9c9fd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.886+0000 7fd6f2ffd640 1 -- 172.21.15.83:0/3752276677 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e8035020 con 0x7fd6f40aef80 2024-03-06T07:24:13.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.886+0000 7fd6f2ffd640 1 -- 172.21.15.83:0/3752276677 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6e8031830 con 0x7fd6f40aef80 2024-03-06T07:24:13.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.886+0000 7fd6f2ffd640 1 -- 172.21.15.83:0/3752276677 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e8031c00 con 0x7fd6f40aef80 2024-03-06T07:24:13.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.887+0000 7fd6fa837640 1 -- 172.21.15.83:0/3752276677 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd6f40aef80 msgr2=0x7fd6f40cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.887+0000 7fd6fa837640 1 --2- 172.21.15.83:0/3752276677 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd6f40aef80 0x7fd6f40cfa90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd6e80097a0 tx=0x7fd6e802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.887+0000 7fd6fa837640 1 -- 172.21.15.83:0/3752276677 shutdown_connections 2024-03-06T07:24:13.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.887+0000 7fd6fa837640 1 --2- 172.21.15.83:0/3752276677 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd6f40aef80 0x7fd6f40cfa90 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.887+0000 7fd6fa837640 1 --2- 172.21.15.83:0/3752276677 >> v2:172.21.15.183:3300/0 conn(0x7fd6f41586e0 0x7fd6f40aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.887+0000 7fd6fa837640 1 -- 172.21.15.83:0/3752276677 >> 172.21.15.83:0/3752276677 conn(0x7fd6f406b190 msgr2=0x7fd6f406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:13.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.887+0000 7fd6fa837640 1 -- 172.21.15.83:0/3752276677 shutdown_connections 2024-03-06T07:24:13.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.887+0000 7fd6fa837640 1 -- 172.21.15.83:0/3752276677 wait complete. 2024-03-06T07:24:13.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.888+0000 7fd6fa837640 1 Processor -- start 2024-03-06T07:24:13.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.888+0000 7fd6fa837640 1 -- start start 2024-03-06T07:24:13.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.889+0000 7fd6fa837640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd6f40aef80 0x7fd6f4144b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.889+0000 7fd6fa837640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd6f40cffd0 0x7fd6f41450a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.889+0000 7fd6fa837640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6f41586e0 0x7fd6f41455e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.889+0000 7fd6fa837640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f415bde0 con 0x7fd6f41586e0 2024-03-06T07:24:13.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.889+0000 7fd6fa837640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f415bb00 con 0x7fd6f40aef80 2024-03-06T07:24:13.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.889+0000 7fd6fa837640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f4071350 con 0x7fd6f40cffd0 2024-03-06T07:24:13.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.889+0000 7fd6f8dd4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6f41586e0 0x7fd6f41455e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.889+0000 7fd6f8dd4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6f41586e0 0x7fd6f41455e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33572/0 (socket says 172.21.15.83:33572) 2024-03-06T07:24:13.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.889+0000 7fd6f8dd4640 1 -- 172.21.15.83:0/753691159 learned_addr learned my addr 172.21.15.83:0/753691159 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:13.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.889+0000 7fd6f37fe640 1 --2- 172.21.15.83:0/753691159 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd6f40cffd0 0x7fd6f41450a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.889+0000 7fd6f3fff640 1 --2- 172.21.15.83:0/753691159 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd6f40aef80 0x7fd6f4144b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.889+0000 7fd6f8dd4640 1 -- 172.21.15.83:0/753691159 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd6f40cffd0 msgr2=0x7fd6f41450a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.889+0000 7fd6f8dd4640 1 --2- 172.21.15.83:0/753691159 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd6f40cffd0 0x7fd6f41450a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.889+0000 7fd6f8dd4640 1 -- 172.21.15.83:0/753691159 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd6f40aef80 msgr2=0x7fd6f4144b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.889+0000 7fd6f8dd4640 1 --2- 172.21.15.83:0/753691159 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd6f40aef80 0x7fd6f4144b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.889+0000 7fd6f8dd4640 1 -- 172.21.15.83:0/753691159 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f4143b00 con 0x7fd6f41586e0 2024-03-06T07:24:13.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.889+0000 7fd6f8dd4640 1 --2- 172.21.15.83:0/753691159 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6f41586e0 0x7fd6f41455e0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fd6ec00cbc0 tx=0x7fd6ec00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.889+0000 7fd6f0ff9640 1 -- 172.21.15.83:0/753691159 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ec017070 con 0x7fd6f41586e0 2024-03-06T07:24:13.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.889+0000 7fd6f0ff9640 1 -- 172.21.15.83:0/753691159 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6ec00ad40 con 0x7fd6f41586e0 2024-03-06T07:24:13.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.889+0000 7fd6fa837640 1 -- 172.21.15.83:0/753691159 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6f4143da0 con 0x7fd6f41586e0 2024-03-06T07:24:13.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.889+0000 7fd6f0ff9640 1 -- 172.21.15.83:0/753691159 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ec012600 con 0x7fd6f41586e0 2024-03-06T07:24:13.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.889+0000 7fd6fa837640 1 -- 172.21.15.83:0/753691159 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6f4144360 con 0x7fd6f41586e0 2024-03-06T07:24:13.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.890+0000 7fd6fa837640 1 -- 172.21.15.83:0/753691159 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6f4156ae0 con 0x7fd6f41586e0 2024-03-06T07:24:13.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.891+0000 7fd6f0ff9640 1 -- 172.21.15.83:0/753691159 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd6ec01a070 con 0x7fd6f41586e0 2024-03-06T07:24:13.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.891+0000 7fd6f0ff9640 1 --2- 172.21.15.83:0/753691159 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd6e006b4d0 0x7fd6e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.891+0000 7fd6f0ff9640 1 -- 172.21.15.83:0/753691159 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd6ec089ab0 con 0x7fd6f41586e0 2024-03-06T07:24:13.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.891+0000 7fd6f3fff640 1 --2- 172.21.15.83:0/753691159 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd6e006b4d0 0x7fd6e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.892+0000 7fd6f3fff640 1 --2- 172.21.15.83:0/753691159 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd6e006b4d0 0x7fd6e006d990 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4009560 tx=0x7fd6e4009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:13.896+0000 7fd6f0ff9640 1 -- 172.21.15.83:0/753691159 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd6f4156ae0 con 0x7fd6f41586e0 2024-03-06T07:24:14.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.066+0000 7fd6fa837640 1 -- 172.21.15.83:0/753691159 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd6f4068ff0 con 0x7fd6f41586e0 2024-03-06T07:24:14.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.067+0000 7fd6f0ff9640 1 -- 172.21.15.83:0/753691159 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v48) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd6ec058670 con 0x7fd6f41586e0 2024-03-06T07:24:14.069 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:24:14.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.068+0000 7fd6fa837640 1 -- 172.21.15.83:0/753691159 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd6e006b4d0 msgr2=0x7fd6e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.068+0000 7fd6fa837640 1 --2- 172.21.15.83:0/753691159 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd6e006b4d0 0x7fd6e006d990 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4009560 tx=0x7fd6e4009190 comp rx=0 tx=0).stop 2024-03-06T07:24:14.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.068+0000 7fd6fa837640 1 -- 172.21.15.83:0/753691159 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6f41586e0 msgr2=0x7fd6f41455e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.068+0000 7fd6fa837640 1 --2- 172.21.15.83:0/753691159 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6f41586e0 0x7fd6f41455e0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fd6ec00cbc0 tx=0x7fd6ec00a430 comp rx=0 tx=0).stop 2024-03-06T07:24:14.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.068+0000 7fd6fa837640 1 -- 172.21.15.83:0/753691159 shutdown_connections 2024-03-06T07:24:14.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.069+0000 7fd6fa837640 1 --2- 172.21.15.83:0/753691159 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd6e006b4d0 0x7fd6e006d990 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.069+0000 7fd6fa837640 1 --2- 172.21.15.83:0/753691159 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6f41586e0 0x7fd6f41455e0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.069+0000 7fd6fa837640 1 --2- 172.21.15.83:0/753691159 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd6f40cffd0 0x7fd6f41450a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.069+0000 7fd6fa837640 1 --2- 172.21.15.83:0/753691159 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd6f40aef80 0x7fd6f4144b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.069+0000 7fd6fa837640 1 -- 172.21.15.83:0/753691159 >> 172.21.15.83:0/753691159 conn(0x7fd6f406b190 msgr2=0x7fd6f4156ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:14.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.069+0000 7fd6fa837640 1 -- 172.21.15.83:0/753691159 shutdown_connections 2024-03-06T07:24:14.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.069+0000 7fd6fa837640 1 -- 172.21.15.83:0/753691159 wait complete. 2024-03-06T07:24:14.081 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented' 2024-03-06T07:24:14.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.181+0000 7f9c129e9640 1 Processor -- start 2024-03-06T07:24:14.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.181+0000 7f9c129e9640 1 -- start start 2024-03-06T07:24:14.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.181+0000 7f9c129e9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c0c1515e0 0x7f9c0c0cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.181+0000 7f9c129e9640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f9c0c0cd7a0 0x7f9c0c0cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.181+0000 7f9c129e9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c0c071f40 con 0x7f9c0c15b430 2024-03-06T07:24:14.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.181+0000 7f9c129e9640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c0c15b8b0 con 0x7f9c0c0cd7a0 2024-03-06T07:24:14.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.181+0000 7f9c129e9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0c070a70 con 0x7f9c0c1515e0 2024-03-06T07:24:14.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.182+0000 7f9c10f86640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f9c0c0cd7a0 0x7f9c0c0cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.182+0000 7f9c0bfff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c0c1515e0 0x7f9c0c0cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.182+0000 7f9c0b7fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f9c0c15b430 0x7f9c0c071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:56094/0 (socket says 172.21.15.83:56094) 2024-03-06T07:24:14.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.182+0000 7f9c0b7fe640 1 -- 172.21.15.83:0/1953288218 learned_addr learned my addr 172.21.15.83:0/1953288218 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:24:14.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.182+0000 7f9c10f86640 1 -- 172.21.15.83:0/1953288218 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c0c1515e0 msgr2=0x7f9c0c0cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:14.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.182+0000 7f9c10f86640 1 --2- 172.21.15.83:0/1953288218 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c0c1515e0 0x7f9c0c0cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.182+0000 7f9c10f86640 1 -- 172.21.15.83:0/1953288218 >> v1:172.21.15.183:6789/0 conn(0x7f9c0c15b430 legacy=0x7f9c0c071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.182+0000 7f9c10f86640 1 -- 172.21.15.83:0/1953288218 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c0c070d50 con 0x7f9c0c0cd7a0 2024-03-06T07:24:14.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.183+0000 7f9c10f86640 1 --2- 172.21.15.83:0/1953288218 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c0c0cd7a0 0x7f9c0c0cdbd0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9c04009d90 tx=0x7f9c0402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=acbbae7b788c6092 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.183+0000 7f9c0affd640 1 -- 172.21.15.83:0/1953288218 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c04039070 con 0x7f9c0c0cd7a0 2024-03-06T07:24:14.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.183+0000 7f9c0affd640 1 -- 172.21.15.83:0/1953288218 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c0402cc50 con 0x7f9c0c0cd7a0 2024-03-06T07:24:14.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.183+0000 7f9c0affd640 1 -- 172.21.15.83:0/1953288218 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c04034440 con 0x7f9c0c0cd7a0 2024-03-06T07:24:14.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.184+0000 7f9c129e9640 1 -- 172.21.15.83:0/1953288218 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c0c0cd7a0 msgr2=0x7f9c0c0cdbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.184+0000 7f9c129e9640 1 --2- 172.21.15.83:0/1953288218 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c0c0cd7a0 0x7f9c0c0cdbd0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9c04009d90 tx=0x7f9c0402bea0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.184+0000 7f9c129e9640 1 -- 172.21.15.83:0/1953288218 shutdown_connections 2024-03-06T07:24:14.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.184+0000 7f9c129e9640 1 --2- 172.21.15.83:0/1953288218 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c0c0cd7a0 0x7f9c0c0cdbd0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.184+0000 7f9c129e9640 1 --2- 172.21.15.83:0/1953288218 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c0c1515e0 0x7f9c0c0cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.184+0000 7f9c129e9640 1 -- 172.21.15.83:0/1953288218 >> 172.21.15.83:0/1953288218 conn(0x7f9c0c0cf8a0 msgr2=0x7f9c0c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:14.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.184+0000 7f9c129e9640 1 -- 172.21.15.83:0/1953288218 shutdown_connections 2024-03-06T07:24:14.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.184+0000 7f9c129e9640 1 -- 172.21.15.83:0/1953288218 wait complete. 2024-03-06T07:24:14.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.185+0000 7f9c129e9640 1 Processor -- start 2024-03-06T07:24:14.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.185+0000 7f9c129e9640 1 -- start start 2024-03-06T07:24:14.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.186+0000 7f9c129e9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c0c0cd7a0 0x7f9c0c15ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.186+0000 7f9c129e9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c0c1515e0 0x7f9c0c15bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.186+0000 7f9c129e9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c0c15b430 0x7f9c0c14bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.186+0000 7f9c129e9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0c0ce110 con 0x7f9c0c15b430 2024-03-06T07:24:14.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.186+0000 7f9c129e9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0c071310 con 0x7f9c0c0cd7a0 2024-03-06T07:24:14.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.186+0000 7f9c129e9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0c0cc3a0 con 0x7f9c0c1515e0 2024-03-06T07:24:14.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.186+0000 7f9c10f86640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c0c15b430 0x7f9c0c14bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.186+0000 7f9c10f86640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c0c15b430 0x7f9c0c14bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33578/0 (socket says 172.21.15.83:33578) 2024-03-06T07:24:14.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.186+0000 7f9c10f86640 1 -- 172.21.15.83:0/3668040533 learned_addr learned my addr 172.21.15.83:0/3668040533 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:14.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.186+0000 7f9c0bfff640 1 --2- 172.21.15.83:0/3668040533 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c0c0cd7a0 0x7f9c0c15ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.186+0000 7f9c0b7fe640 1 --2- 172.21.15.83:0/3668040533 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c0c1515e0 0x7f9c0c15bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.186+0000 7f9c10f86640 1 -- 172.21.15.83:0/3668040533 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c0c1515e0 msgr2=0x7f9c0c15bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.186+0000 7f9c10f86640 1 --2- 172.21.15.83:0/3668040533 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c0c1515e0 0x7f9c0c15bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.186+0000 7f9c10f86640 1 -- 172.21.15.83:0/3668040533 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c0c0cd7a0 msgr2=0x7f9c0c15ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.186+0000 7f9c10f86640 1 --2- 172.21.15.83:0/3668040533 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c0c0cd7a0 0x7f9c0c15ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.186+0000 7f9c10f86640 1 -- 172.21.15.83:0/3668040533 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c0c15c5a0 con 0x7f9c0c15b430 2024-03-06T07:24:14.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.187+0000 7f9c10f86640 1 --2- 172.21.15.83:0/3668040533 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c0c15b430 0x7f9c0c14bad0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f9c04009d90 tx=0x7f9c04004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.187+0000 7f9c08ff9640 1 -- 172.21.15.83:0/3668040533 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c04039050 con 0x7f9c0c15b430 2024-03-06T07:24:14.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.187+0000 7f9c08ff9640 1 -- 172.21.15.83:0/3668040533 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c04045070 con 0x7f9c0c15b430 2024-03-06T07:24:14.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.187+0000 7f9c129e9640 1 -- 172.21.15.83:0/3668040533 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c0c15c880 con 0x7f9c0c15b430 2024-03-06T07:24:14.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.187+0000 7f9c08ff9640 1 -- 172.21.15.83:0/3668040533 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c04009450 con 0x7f9c0c15b430 2024-03-06T07:24:14.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.187+0000 7f9c129e9640 1 -- 172.21.15.83:0/3668040533 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c0c1e40a0 con 0x7f9c0c15b430 2024-03-06T07:24:14.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.188+0000 7f9c129e9640 1 -- 172.21.15.83:0/3668040533 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c0c068840 con 0x7f9c0c15b430 2024-03-06T07:24:14.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.188+0000 7f9c08ff9640 1 -- 172.21.15.83:0/3668040533 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9c0403e610 con 0x7f9c0c15b430 2024-03-06T07:24:14.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.189+0000 7f9c08ff9640 1 --2- 172.21.15.83:0/3668040533 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9bf806b4d0 0x7f9bf806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.189+0000 7f9c08ff9640 1 -- 172.21.15.83:0/3668040533 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f9c040ac810 con 0x7f9c0c15b430 2024-03-06T07:24:14.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.189+0000 7f9c0bfff640 1 --2- 172.21.15.83:0/3668040533 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9bf806b4d0 0x7f9bf806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.189+0000 7f9c0bfff640 1 --2- 172.21.15.83:0/3668040533 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9bf806b4d0 0x7f9bf806d990 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc00ac40 tx=0x7f9bfc00a2a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.193+0000 7f9c08ff9640 1 -- 172.21.15.83:0/3668040533 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9c0407b3d0 con 0x7f9c0c15b430 2024-03-06T07:24:14.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.430+0000 7f9c129e9640 1 -- 172.21.15.83:0/3668040533 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]} v 0) v1 -- 0x7f9c0c04f330 con 0x7f9c0c15b430 2024-03-06T07:24:14.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.473+0000 7f9c08ff9640 1 -- 172.21.15.83:0/3668040533 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]}]=0 v145) v1 ==== 147+0+0 (secure 0 0 0) 0x7f9c0407f470 con 0x7f9c0c15b430 2024-03-06T07:24:14.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.475+0000 7f9c129e9640 1 -- 172.21.15.83:0/3668040533 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9bf806b4d0 msgr2=0x7f9bf806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.475+0000 7f9c129e9640 1 --2- 172.21.15.83:0/3668040533 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9bf806b4d0 0x7f9bf806d990 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc00ac40 tx=0x7f9bfc00a2a0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.475+0000 7f9c129e9640 1 -- 172.21.15.83:0/3668040533 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c0c15b430 msgr2=0x7f9c0c14bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.475+0000 7f9c129e9640 1 --2- 172.21.15.83:0/3668040533 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c0c15b430 0x7f9c0c14bad0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f9c04009d90 tx=0x7f9c04004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:14.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.475+0000 7f9c129e9640 1 -- 172.21.15.83:0/3668040533 shutdown_connections 2024-03-06T07:24:14.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.475+0000 7f9c129e9640 1 --2- 172.21.15.83:0/3668040533 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9bf806b4d0 0x7f9bf806d990 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.475+0000 7f9c129e9640 1 --2- 172.21.15.83:0/3668040533 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c0c15b430 0x7f9c0c14bad0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.475+0000 7f9c129e9640 1 --2- 172.21.15.83:0/3668040533 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c0c1515e0 0x7f9c0c15bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.475+0000 7f9c129e9640 1 --2- 172.21.15.83:0/3668040533 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c0c0cd7a0 0x7f9c0c15ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.475+0000 7f9c129e9640 1 -- 172.21.15.83:0/3668040533 >> 172.21.15.83:0/3668040533 conn(0x7f9c0c0cf8a0 msgr2=0x7f9c0c06afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:14.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.475+0000 7f9c129e9640 1 -- 172.21.15.83:0/3668040533 shutdown_connections 2024-03-06T07:24:14.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.475+0000 7f9c129e9640 1 -- 172.21.15.83:0/3668040533 wait complete. 2024-03-06T07:24:14.488 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:24:14.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.589+0000 7fb65644e640 1 Processor -- start 2024-03-06T07:24:14.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.589+0000 7fb65644e640 1 -- start start 2024-03-06T07:24:14.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.590+0000 7fb65644e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb65015def0 0x7fb6500b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.590+0000 7fb65644e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb6500cea50 0x7fb6500ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.590+0000 7fb65644e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb65006d030 con 0x7fb65006cab0 2024-03-06T07:24:14.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.590+0000 7fb65644e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6500cf1c0 con 0x7fb6500cea50 2024-03-06T07:24:14.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.590+0000 7fb65644e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6500cc760 con 0x7fb65015def0 2024-03-06T07:24:14.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.590+0000 7fb6549eb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb6500cea50 0x7fb6500ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.590+0000 7fb64f7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fb65006cab0 0x7fb6500ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48552/0 (socket says 172.21.15.83:48552) 2024-03-06T07:24:14.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.590+0000 7fb64f7fe640 1 -- 172.21.15.83:0/2673230113 learned_addr learned my addr 172.21.15.83:0/2673230113 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:24:14.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.591+0000 7fb6549eb640 1 -- 172.21.15.83:0/2673230113 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb65015def0 msgr2=0x7fb6500b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:14.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.591+0000 7fb6549eb640 1 --2- 172.21.15.83:0/2673230113 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb65015def0 0x7fb6500b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.591+0000 7fb6549eb640 1 -- 172.21.15.83:0/2673230113 >> v1:172.21.15.83:6789/0 conn(0x7fb65006cab0 legacy=0x7fb6500ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.591+0000 7fb6549eb640 1 -- 172.21.15.83:0/2673230113 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb65006d310 con 0x7fb6500cea50 2024-03-06T07:24:14.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.591+0000 7fb64effd640 1 -- 172.21.15.83:0/2673230113 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2673093581 0 0) 0x7fb65006d030 con 0x7fb65006cab0 2024-03-06T07:24:14.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.591+0000 7fb6549eb640 1 --2- 172.21.15.83:0/2673230113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb6500cea50 0x7fb6500ccbf0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fb648009d90 tx=0x7fb64802bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af749ce36400c72 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.591+0000 7fb64effd640 1 -- 172.21.15.83:0/2673230113 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb648039070 con 0x7fb6500cea50 2024-03-06T07:24:14.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.591+0000 7fb64effd640 1 -- 172.21.15.83:0/2673230113 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb64802cce0 con 0x7fb6500cea50 2024-03-06T07:24:14.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.592+0000 7fb64effd640 1 -- 172.21.15.83:0/2673230113 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb648034660 con 0x7fb6500cea50 2024-03-06T07:24:14.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.592+0000 7fb65644e640 1 -- 172.21.15.83:0/2673230113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb6500cea50 msgr2=0x7fb6500ccbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.592+0000 7fb65644e640 1 --2- 172.21.15.83:0/2673230113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb6500cea50 0x7fb6500ccbf0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fb648009d90 tx=0x7fb64802bf30 comp rx=0 tx=0).stop 2024-03-06T07:24:14.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.592+0000 7fb65644e640 1 -- 172.21.15.83:0/2673230113 shutdown_connections 2024-03-06T07:24:14.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.592+0000 7fb65644e640 1 --2- 172.21.15.83:0/2673230113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb6500cea50 0x7fb6500ccbf0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.592+0000 7fb65644e640 1 --2- 172.21.15.83:0/2673230113 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb65015def0 0x7fb6500b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.592+0000 7fb65644e640 1 -- 172.21.15.83:0/2673230113 >> 172.21.15.83:0/2673230113 conn(0x7fb6500b3690 msgr2=0x7fb6500b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:14.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.592+0000 7fb65644e640 1 -- 172.21.15.83:0/2673230113 shutdown_connections 2024-03-06T07:24:14.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.592+0000 7fb65644e640 1 -- 172.21.15.83:0/2673230113 wait complete. 2024-03-06T07:24:14.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.593+0000 7fb65644e640 1 Processor -- start 2024-03-06T07:24:14.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.593+0000 7fb65644e640 1 -- start start 2024-03-06T07:24:14.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.594+0000 7fb65644e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb65006cab0 0x7fb6500cb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.594+0000 7fb65644e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb6500cea50 0x7fb6500c6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.594+0000 7fb65644e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb65015def0 0x7fb6500c5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.594+0000 7fb65644e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65015e2d0 con 0x7fb6500cea50 2024-03-06T07:24:14.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.594+0000 7fb65644e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6500cc3f0 con 0x7fb65015def0 2024-03-06T07:24:14.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.594+0000 7fb65644e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb65015e450 con 0x7fb65006cab0 2024-03-06T07:24:14.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.594+0000 7fb64f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb6500cea50 0x7fb6500c6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.594+0000 7fb64f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb6500cea50 0x7fb6500c6da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33594/0 (socket says 172.21.15.83:33594) 2024-03-06T07:24:14.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.594+0000 7fb64f7fe640 1 -- 172.21.15.83:0/4222765718 learned_addr learned my addr 172.21.15.83:0/4222765718 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:14.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.594+0000 7fb64ffff640 1 --2- 172.21.15.83:0/4222765718 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb65006cab0 0x7fb6500cb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.594+0000 7fb64f7fe640 1 -- 172.21.15.83:0/4222765718 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb65006cab0 msgr2=0x7fb6500cb5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.594+0000 7fb64f7fe640 1 --2- 172.21.15.83:0/4222765718 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb65006cab0 0x7fb6500cb5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.594+0000 7fb6549eb640 1 --2- 172.21.15.83:0/4222765718 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb65015def0 0x7fb6500c5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.594+0000 7fb64f7fe640 1 -- 172.21.15.83:0/4222765718 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb65015def0 msgr2=0x7fb6500c5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.594+0000 7fb64f7fe640 1 --2- 172.21.15.83:0/4222765718 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb65015def0 0x7fb6500c5540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.594+0000 7fb64f7fe640 1 -- 172.21.15.83:0/4222765718 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6500c5a80 con 0x7fb6500cea50 2024-03-06T07:24:14.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.595+0000 7fb64ffff640 1 --2- 172.21.15.83:0/4222765718 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb65006cab0 0x7fb6500cb5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:14.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.595+0000 7fb64f7fe640 1 --2- 172.21.15.83:0/4222765718 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb6500cea50 0x7fb6500c6da0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fb64400c4a0 tx=0x7fb64400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.595+0000 7fb64cff9640 1 -- 172.21.15.83:0/4222765718 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb644018070 con 0x7fb6500cea50 2024-03-06T07:24:14.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.595+0000 7fb64cff9640 1 -- 172.21.15.83:0/4222765718 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6440074e0 con 0x7fb6500cea50 2024-03-06T07:24:14.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.595+0000 7fb65644e640 1 -- 172.21.15.83:0/4222765718 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6500c5d00 con 0x7fb6500cea50 2024-03-06T07:24:14.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.595+0000 7fb64cff9640 1 -- 172.21.15.83:0/4222765718 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb644004a20 con 0x7fb6500cea50 2024-03-06T07:24:14.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.595+0000 7fb65644e640 1 -- 172.21.15.83:0/4222765718 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6500c62c0 con 0x7fb6500cea50 2024-03-06T07:24:14.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.596+0000 7fb65644e640 1 -- 172.21.15.83:0/4222765718 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb65009fcf0 con 0x7fb6500cea50 2024-03-06T07:24:14.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.596+0000 7fb64cff9640 1 -- 172.21.15.83:0/4222765718 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb64401b070 con 0x7fb6500cea50 2024-03-06T07:24:14.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.597+0000 7fb64cff9640 1 --2- 172.21.15.83:0/4222765718 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb63c06b4d0 0x7fb63c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.597+0000 7fb64cff9640 1 -- 172.21.15.83:0/4222765718 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fb644089230 con 0x7fb6500cea50 2024-03-06T07:24:14.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.597+0000 7fb64ffff640 1 --2- 172.21.15.83:0/4222765718 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb63c06b4d0 0x7fb63c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.597+0000 7fb64ffff640 1 --2- 172.21.15.83:0/4222765718 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb63c06b4d0 0x7fb63c06d990 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb6500d2690 tx=0x7fb640008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.601+0000 7fb64cff9640 1 -- 172.21.15.83:0/4222765718 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb6500c5a80 con 0x7fb6500cea50 2024-03-06T07:24:14.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.771+0000 7fb65644e640 1 -- 172.21.15.83:0/4222765718 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fb650068ff0 con 0x7fb6500cea50 2024-03-06T07:24:14.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.774+0000 7fb64cff9640 1 -- 172.21.15.83:0/4222765718 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb6440583b0 con 0x7fb6500cea50 2024-03-06T07:24:14.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.776+0000 7fb65644e640 1 -- 172.21.15.83:0/4222765718 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb63c06b4d0 msgr2=0x7fb63c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.776+0000 7fb65644e640 1 --2- 172.21.15.83:0/4222765718 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb63c06b4d0 0x7fb63c06d990 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb6500d2690 tx=0x7fb640008040 comp rx=0 tx=0).stop 2024-03-06T07:24:14.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.776+0000 7fb65644e640 1 -- 172.21.15.83:0/4222765718 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb6500cea50 msgr2=0x7fb6500c6da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.776+0000 7fb65644e640 1 --2- 172.21.15.83:0/4222765718 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb6500cea50 0x7fb6500c6da0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fb64400c4a0 tx=0x7fb64400c960 comp rx=0 tx=0).stop 2024-03-06T07:24:14.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.776+0000 7fb65644e640 1 -- 172.21.15.83:0/4222765718 shutdown_connections 2024-03-06T07:24:14.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.776+0000 7fb65644e640 1 --2- 172.21.15.83:0/4222765718 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb63c06b4d0 0x7fb63c06d990 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.776+0000 7fb65644e640 1 --2- 172.21.15.83:0/4222765718 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb65015def0 0x7fb6500c5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.776+0000 7fb65644e640 1 --2- 172.21.15.83:0/4222765718 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb6500cea50 0x7fb6500c6da0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.776+0000 7fb65644e640 1 --2- 172.21.15.83:0/4222765718 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb65006cab0 0x7fb6500cb5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.776+0000 7fb65644e640 1 -- 172.21.15.83:0/4222765718 >> 172.21.15.83:0/4222765718 conn(0x7fb6500b3690 msgr2=0x7fb6501605e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:14.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.776+0000 7fb65644e640 1 -- 172.21.15.83:0/4222765718 shutdown_connections 2024-03-06T07:24:14.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:14.776+0000 7fb65644e640 1 -- 172.21.15.83:0/4222765718 wait complete. 2024-03-06T07:24:14.789 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:24:14.789 DEBUG:teuthology.orchestra.run.smithi083:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:24:14.830 DEBUG:teuthology.orchestra.run.smithi183:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:24:14.878 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:24:14.878 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:24:14.878 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:14.894 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1257 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:24:14.894 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:24:14.894 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:14.911 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=287 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:24:14.911 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:24:14.911 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:14.948 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1257 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:24:14.948 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:24:14.948 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:14.969 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=287 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:24:14.969 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:24:14.969 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:15.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.086+0000 7fd9d7df7640 1 Processor -- start 2024-03-06T07:24:15.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.086+0000 7fd9d7df7640 1 -- start start 2024-03-06T07:24:15.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.087+0000 7fd9d7df7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9d013dc10 0x7fd9d00b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.087+0000 7fd9d7df7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd9d015bf30 0x7fd9d00b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.087+0000 7fd9d7df7640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d015f580 con 0x7fd9d015e850 2024-03-06T07:24:15.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.087+0000 7fd9d7df7640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9d00cf3f0 con 0x7fd9d015bf30 2024-03-06T07:24:15.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.087+0000 7fd9d7df7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d015b210 con 0x7fd9d013dc10 2024-03-06T07:24:15.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.087+0000 7fd9d5392640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd9d015bf30 0x7fd9d00b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.087+0000 7fd9d5392640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd9d015bf30 0x7fd9d00b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33598/0 (socket says 172.21.15.83:33598) 2024-03-06T07:24:15.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.087+0000 7fd9d5392640 1 -- 172.21.15.83:0/2442028501 learned_addr learned my addr 172.21.15.83:0/2442028501 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:15.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.087+0000 7fd9d5b93640 1 --2- 172.21.15.83:0/2442028501 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9d013dc10 0x7fd9d00b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.087+0000 7fd9d5392640 1 -- 172.21.15.83:0/2442028501 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9d013dc10 msgr2=0x7fd9d00b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:15.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.087+0000 7fd9d5392640 1 --2- 172.21.15.83:0/2442028501 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9d013dc10 0x7fd9d00b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.087+0000 7fd9d5392640 1 -- 172.21.15.83:0/2442028501 >> v1:172.21.15.183:6789/0 conn(0x7fd9d015e850 legacy=0x7fd9d015eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.087+0000 7fd9d5392640 1 -- 172.21.15.83:0/2442028501 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d015b450 con 0x7fd9d015bf30 2024-03-06T07:24:15.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.088+0000 7fd9d5392640 1 --2- 172.21.15.83:0/2442028501 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9d015bf30 0x7fd9d00b2e90 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fd9c0009870 tx=0x7fd9c002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=434ecac8c68c8318 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:15.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.088+0000 7fd9d4b91640 1 -- 172.21.15.83:0/2442028501 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c0035020 con 0x7fd9d015bf30 2024-03-06T07:24:15.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.088+0000 7fd9d4b91640 1 -- 172.21.15.83:0/2442028501 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9c0031640 con 0x7fd9d015bf30 2024-03-06T07:24:15.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.088+0000 7fd9d4b91640 1 -- 172.21.15.83:0/2442028501 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c002d040 con 0x7fd9d015bf30 2024-03-06T07:24:15.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.089+0000 7fd9d7df7640 1 -- 172.21.15.83:0/2442028501 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9d015bf30 msgr2=0x7fd9d00b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.089+0000 7fd9d7df7640 1 --2- 172.21.15.83:0/2442028501 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9d015bf30 0x7fd9d00b2e90 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fd9c0009870 tx=0x7fd9c002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.089+0000 7fd9d7df7640 1 -- 172.21.15.83:0/2442028501 shutdown_connections 2024-03-06T07:24:15.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.089+0000 7fd9d7df7640 1 --2- 172.21.15.83:0/2442028501 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9d015bf30 0x7fd9d00b2e90 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.089+0000 7fd9d7df7640 1 --2- 172.21.15.83:0/2442028501 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9d013dc10 0x7fd9d00b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.089+0000 7fd9d7df7640 1 -- 172.21.15.83:0/2442028501 >> 172.21.15.83:0/2442028501 conn(0x7fd9d00cf8a0 msgr2=0x7fd9d00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:15.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.089+0000 7fd9d7df7640 1 -- 172.21.15.83:0/2442028501 shutdown_connections 2024-03-06T07:24:15.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.089+0000 7fd9d7df7640 1 -- 172.21.15.83:0/2442028501 wait complete. 2024-03-06T07:24:15.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.090+0000 7fd9d7df7640 1 Processor -- start 2024-03-06T07:24:15.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.090+0000 7fd9d7df7640 1 -- start start 2024-03-06T07:24:15.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.091+0000 7fd9d7df7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9d013dc10 0x7fd9d01e6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.091+0000 7fd9d7df7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9d015bf30 0x7fd9d01536b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.091+0000 7fd9d7df7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9d015e850 0x7fd9d0136e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.091+0000 7fd9d7df7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d00708d0 con 0x7fd9d015e850 2024-03-06T07:24:15.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.091+0000 7fd9d7df7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d013dff0 con 0x7fd9d013dc10 2024-03-06T07:24:15.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.091+0000 7fd9d7df7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d0070a50 con 0x7fd9d015bf30 2024-03-06T07:24:15.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.091+0000 7fd9d6394640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9d015e850 0x7fd9d0136e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.091+0000 7fd9d6394640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9d015e850 0x7fd9d0136e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33608/0 (socket says 172.21.15.83:33608) 2024-03-06T07:24:15.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.091+0000 7fd9d6394640 1 -- 172.21.15.83:0/2234978698 learned_addr learned my addr 172.21.15.83:0/2234978698 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:15.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.091+0000 7fd9d5392640 1 --2- 172.21.15.83:0/2234978698 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9d015bf30 0x7fd9d01536b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.091+0000 7fd9d6394640 1 -- 172.21.15.83:0/2234978698 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9d015bf30 msgr2=0x7fd9d01536b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.091+0000 7fd9d6394640 1 --2- 172.21.15.83:0/2234978698 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9d015bf30 0x7fd9d01536b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.091+0000 7fd9d5b93640 1 --2- 172.21.15.83:0/2234978698 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9d013dc10 0x7fd9d01e6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.091+0000 7fd9d6394640 1 -- 172.21.15.83:0/2234978698 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9d013dc10 msgr2=0x7fd9d01e6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.091+0000 7fd9d6394640 1 --2- 172.21.15.83:0/2234978698 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9d013dc10 0x7fd9d01e6950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.091+0000 7fd9d6394640 1 -- 172.21.15.83:0/2234978698 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d0137bf0 con 0x7fd9d015e850 2024-03-06T07:24:15.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.091+0000 7fd9d5392640 1 --2- 172.21.15.83:0/2234978698 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9d015bf30 0x7fd9d01536b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:15.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.091+0000 7fd9d6394640 1 --2- 172.21.15.83:0/2234978698 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9d015e850 0x7fd9d0136e80 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fd9c800c6b0 tx=0x7fd9c800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:15.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.091+0000 7fd9c67fc640 1 -- 172.21.15.83:0/2234978698 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c8003b70 con 0x7fd9d015e850 2024-03-06T07:24:15.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.091+0000 7fd9c67fc640 1 -- 172.21.15.83:0/2234978698 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9c8003d30 con 0x7fd9d015e850 2024-03-06T07:24:15.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.091+0000 7fd9c67fc640 1 -- 172.21.15.83:0/2234978698 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c8005310 con 0x7fd9d015e850 2024-03-06T07:24:15.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.092+0000 7fd9d7df7640 1 -- 172.21.15.83:0/2234978698 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9d0154150 con 0x7fd9d015e850 2024-03-06T07:24:15.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.092+0000 7fd9d7df7640 1 -- 172.21.15.83:0/2234978698 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9d0137eb0 con 0x7fd9d015e850 2024-03-06T07:24:15.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.092+0000 7fd9d7df7640 1 -- 172.21.15.83:0/2234978698 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9d004f330 con 0x7fd9d015e850 2024-03-06T07:24:15.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.093+0000 7fd9c67fc640 1 -- 172.21.15.83:0/2234978698 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd9c80054b0 con 0x7fd9d015e850 2024-03-06T07:24:15.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.093+0000 7fd9c67fc640 1 --2- 172.21.15.83:0/2234978698 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd9b406b500 0x7fd9b406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.093+0000 7fd9c67fc640 1 -- 172.21.15.83:0/2234978698 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd9c8089d20 con 0x7fd9d015e850 2024-03-06T07:24:15.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.093+0000 7fd9d5b93640 1 --2- 172.21.15.83:0/2234978698 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd9b406b500 0x7fd9b406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.094+0000 7fd9d5b93640 1 --2- 172.21.15.83:0/2234978698 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd9b406b500 0x7fd9b406d9c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc006fd0 tx=0x7fd9cc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:15.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.098+0000 7fd9c67fc640 1 -- 172.21.15.83:0/2234978698 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd9c8058ea0 con 0x7fd9d015e850 2024-03-06T07:24:15.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.304+0000 7fd9d7df7640 1 -- 172.21.15.83:0/2234978698 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9d00b3c60 con 0x7fd9d015e850 2024-03-06T07:24:15.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.304+0000 7fd9c67fc640 1 -- 172.21.15.83:0/2234978698 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+3705 (secure 0 0 0) 0x7fd9c80587a0 con 0x7fd9d015e850 2024-03-06T07:24:15.306 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:24:15.306 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":21,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5395,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/2270670914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2270670914},{"type":"v1","addr":"172.21.15.83:6837","nonce":2270670914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5398,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/2359880029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2359880029},{"type":"v1","addr":"172.21.15.183:6834","nonce":2359880029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5401,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3099809273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":3099809273},{"type":"v1","addr":"172.21.15.183:6835","nonce":3099809273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5994,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/4014774862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":4014774862},{"type":"v1","addr":"172.21.15.83:6835","nonce":4014774862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[]} 2024-03-06T07:24:15.307 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 21 2024-03-06T07:24:15.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.306+0000 7fd9d7df7640 1 -- 172.21.15.83:0/2234978698 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd9b406b500 msgr2=0x7fd9b406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.306+0000 7fd9d7df7640 1 --2- 172.21.15.83:0/2234978698 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd9b406b500 0x7fd9b406d9c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc006fd0 tx=0x7fd9cc008040 comp rx=0 tx=0).stop 2024-03-06T07:24:15.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.306+0000 7fd9d7df7640 1 -- 172.21.15.83:0/2234978698 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9d015e850 msgr2=0x7fd9d0136e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.306+0000 7fd9d7df7640 1 --2- 172.21.15.83:0/2234978698 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9d015e850 0x7fd9d0136e80 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fd9c800c6b0 tx=0x7fd9c800cb70 comp rx=0 tx=0).stop 2024-03-06T07:24:15.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.306+0000 7fd9d7df7640 1 -- 172.21.15.83:0/2234978698 shutdown_connections 2024-03-06T07:24:15.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.306+0000 7fd9d7df7640 1 --2- 172.21.15.83:0/2234978698 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd9b406b500 0x7fd9b406d9c0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.306+0000 7fd9d7df7640 1 --2- 172.21.15.83:0/2234978698 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9d015e850 0x7fd9d0136e80 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.306+0000 7fd9d7df7640 1 --2- 172.21.15.83:0/2234978698 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9d015bf30 0x7fd9d01536b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.306+0000 7fd9d7df7640 1 --2- 172.21.15.83:0/2234978698 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9d013dc10 0x7fd9d01e6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.306+0000 7fd9d7df7640 1 -- 172.21.15.83:0/2234978698 >> 172.21.15.83:0/2234978698 conn(0x7fd9d00cf8a0 msgr2=0x7fd9d00affd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:15.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.306+0000 7fd9d7df7640 1 -- 172.21.15.83:0/2234978698 shutdown_connections 2024-03-06T07:24:15.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.306+0000 7fd9d7df7640 1 -- 172.21.15.83:0/2234978698 wait complete. 2024-03-06T07:24:15.319 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:24:15.319 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:24:15.320 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:24:15.320 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:15.320 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:24:15.320 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:24:15.321 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:24:15.321 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:15.321 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:24:15.321 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:24:15.321 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:24:15.321 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:15.321 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:24:15.321 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:24:15.322 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:24:15.322 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:15.330 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:24:15.330 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:24:15.331 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:24:15.331 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:24:15.333 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:24:15.333 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:24:15.334 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:24:15.334 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:24:15.372 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:24:15.372 DEBUG:teuthology.parallel:result is None 2024-03-06T07:24:15.373 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:24:15.373 DEBUG:teuthology.parallel:result is None 2024-03-06T07:24:15.376 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:24:15.376 DEBUG:teuthology.parallel:result is None 2024-03-06T07:24:15.376 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:24:15.377 DEBUG:teuthology.parallel:result is None 2024-03-06T07:24:15.377 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:24:15.448 INFO:tasks.ceph.mds.a.smithi083.stdout:starting mds.a at 2024-03-06T07:24:15.448 INFO:tasks.ceph.mds.c.smithi083.stdout:starting mds.c at 2024-03-06T07:24:15.457 INFO:tasks.ceph.mds.d.smithi183.stdout:starting mds.d at 2024-03-06T07:24:15.458 INFO:tasks.ceph.mds.b.smithi183.stdout:starting mds.b at 2024-03-06T07:24:15.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.480+0000 7f39b431e640 1 Processor -- start 2024-03-06T07:24:15.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.480+0000 7f39b431e640 1 -- start start 2024-03-06T07:24:15.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.481+0000 7f39b431e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f39ac15e000 0x7f39ac0d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.481+0000 7f39b431e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f39ac0d4810 0x7f39ac0d0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.481+0000 7f39b431e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39ac0d25e0 con 0x7f39ac0c80e0 2024-03-06T07:24:15.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.481+0000 7f39b431e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39ac0c7c50 con 0x7f39ac0d4810 2024-03-06T07:24:15.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.481+0000 7f39b431e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39ac0d2880 con 0x7f39ac15e000 2024-03-06T07:24:15.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.482+0000 7f39b18b9640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f39ac0d4810 0x7f39ac0d0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.482+0000 7f39b20ba640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f39ac15e000 0x7f39ac0d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.482+0000 7f39b18b9640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f39ac0d4810 0x7f39ac0d0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:52998/0 (socket says 172.21.15.83:52998) 2024-03-06T07:24:15.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.482+0000 7f39b18b9640 1 -- 172.21.15.83:0/1077536508 learned_addr learned my addr 172.21.15.83:0/1077536508 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:15.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.482+0000 7f39b18b9640 1 -- 172.21.15.83:0/1077536508 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f39ac15e000 msgr2=0x7f39ac0d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:15.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.482+0000 7f39b18b9640 1 --2- 172.21.15.83:0/1077536508 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f39ac15e000 0x7f39ac0d0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.482+0000 7f39b18b9640 1 -- 172.21.15.83:0/1077536508 >> v1:172.21.15.183:6789/0 conn(0x7f39ac0c80e0 legacy=0x7f39ac0c8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.482+0000 7f39b18b9640 1 -- 172.21.15.83:0/1077536508 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39ac0d2b40 con 0x7f39ac0d4810 2024-03-06T07:24:15.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.482+0000 7f39b20ba640 1 --2- 172.21.15.83:0/1077536508 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f39ac15e000 0x7f39ac0d0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:15.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.483+0000 7f39b18b9640 1 --2- 172.21.15.83:0/1077536508 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f39ac0d4810 0x7f39ac0d0bc0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f399c009870 tx=0x7f399c02ba20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ba7195c40486aba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:15.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.483+0000 7f39b10b8640 1 -- 172.21.15.83:0/1077536508 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399c02f840 con 0x7f39ac0d4810 2024-03-06T07:24:15.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.483+0000 7f39b10b8640 1 -- 172.21.15.83:0/1077536508 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f399c02f9e0 con 0x7f39ac0d4810 2024-03-06T07:24:15.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.483+0000 7f39b10b8640 1 -- 172.21.15.83:0/1077536508 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399c02fdb0 con 0x7f39ac0d4810 2024-03-06T07:24:15.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.484+0000 7f39b431e640 1 -- 172.21.15.83:0/1077536508 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f39ac0d4810 msgr2=0x7f39ac0d0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.484+0000 7f39b431e640 1 --2- 172.21.15.83:0/1077536508 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f39ac0d4810 0x7f39ac0d0bc0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f399c009870 tx=0x7f399c02ba20 comp rx=0 tx=0).stop 2024-03-06T07:24:15.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.484+0000 7f39b431e640 1 -- 172.21.15.83:0/1077536508 shutdown_connections 2024-03-06T07:24:15.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.484+0000 7f39b431e640 1 --2- 172.21.15.83:0/1077536508 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f39ac0d4810 0x7f39ac0d0bc0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.484+0000 7f39b431e640 1 --2- 172.21.15.83:0/1077536508 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f39ac15e000 0x7f39ac0d0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.484+0000 7f39b431e640 1 -- 172.21.15.83:0/1077536508 >> 172.21.15.83:0/1077536508 conn(0x7f39ac0d3cf0 msgr2=0x7f39ac0d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:15.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.484+0000 7f39b431e640 1 -- 172.21.15.83:0/1077536508 shutdown_connections 2024-03-06T07:24:15.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.484+0000 7f39b431e640 1 -- 172.21.15.83:0/1077536508 wait complete. 2024-03-06T07:24:15.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.485+0000 7f39b431e640 1 Processor -- start 2024-03-06T07:24:15.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.485+0000 7f39b431e640 1 -- start start 2024-03-06T07:24:15.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.486+0000 7f39b431e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f39ac0c80e0 0x7f39ac0cc920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.486+0000 7f39b431e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f39ac0d4810 0x7f39ac153110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.486+0000 7f39b431e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f39ac15e000 0x7f39ac09da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.486+0000 7f39b431e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39ac15e3e0 con 0x7f39ac15e000 2024-03-06T07:24:15.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.486+0000 7f39b431e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39ac0d22d0 con 0x7f39ac0c80e0 2024-03-06T07:24:15.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.486+0000 7f39b431e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39ac0cf7d0 con 0x7f39ac0d4810 2024-03-06T07:24:15.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.486+0000 7f39b28bb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f39ac15e000 0x7f39ac09da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.486+0000 7f39b28bb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f39ac15e000 0x7f39ac09da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33658/0 (socket says 172.21.15.83:33658) 2024-03-06T07:24:15.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.486+0000 7f39b28bb640 1 -- 172.21.15.83:0/2083426715 learned_addr learned my addr 172.21.15.83:0/2083426715 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:15.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.486+0000 7f39b18b9640 1 --2- 172.21.15.83:0/2083426715 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f39ac0d4810 0x7f39ac153110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.486+0000 7f39b20ba640 1 --2- 172.21.15.83:0/2083426715 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f39ac0c80e0 0x7f39ac0cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.487+0000 7f39b28bb640 1 -- 172.21.15.83:0/2083426715 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f39ac0d4810 msgr2=0x7f39ac153110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.487+0000 7f39b28bb640 1 --2- 172.21.15.83:0/2083426715 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f39ac0d4810 0x7f39ac153110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.487+0000 7f39b28bb640 1 -- 172.21.15.83:0/2083426715 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f39ac0c80e0 msgr2=0x7f39ac0cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.487+0000 7f39b28bb640 1 --2- 172.21.15.83:0/2083426715 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f39ac0c80e0 0x7f39ac0cc920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.487+0000 7f39b28bb640 1 -- 172.21.15.83:0/2083426715 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39ac0ccf70 con 0x7f39ac15e000 2024-03-06T07:24:15.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.487+0000 7f39b28bb640 1 --2- 172.21.15.83:0/2083426715 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f39ac15e000 0x7f39ac09da90 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f39a400c7d0 tx=0x7f39a400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:15.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.487+0000 7f39a2ffd640 1 -- 172.21.15.83:0/2083426715 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a4003b70 con 0x7f39ac15e000 2024-03-06T07:24:15.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.487+0000 7f39a2ffd640 1 -- 172.21.15.83:0/2083426715 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39a4003d30 con 0x7f39ac15e000 2024-03-06T07:24:15.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.487+0000 7f39b431e640 1 -- 172.21.15.83:0/2083426715 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39ac0cd250 con 0x7f39ac15e000 2024-03-06T07:24:15.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.487+0000 7f39a2ffd640 1 -- 172.21.15.83:0/2083426715 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a400aa90 con 0x7f39ac15e000 2024-03-06T07:24:15.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.487+0000 7f39b431e640 1 -- 172.21.15.83:0/2083426715 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39ac0cd810 con 0x7f39ac15e000 2024-03-06T07:24:15.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.488+0000 7f39b431e640 1 -- 172.21.15.83:0/2083426715 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39ac04f330 con 0x7f39ac15e000 2024-03-06T07:24:15.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.489+0000 7f39a2ffd640 1 -- 172.21.15.83:0/2083426715 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f39a400ac30 con 0x7f39ac15e000 2024-03-06T07:24:15.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.489+0000 7f39a2ffd640 1 --2- 172.21.15.83:0/2083426715 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f399006b500 0x7f399006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.489+0000 7f39b20ba640 1 --2- 172.21.15.83:0/2083426715 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f399006b500 0x7f399006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.489+0000 7f39a2ffd640 1 -- 172.21.15.83:0/2083426715 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f39a4089cd0 con 0x7f39ac15e000 2024-03-06T07:24:15.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.490+0000 7f39b20ba640 1 --2- 172.21.15.83:0/2083426715 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f399006b500 0x7f399006d9c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f39a8009560 tx=0x7f39a8009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:15.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.497+0000 7f39a2ffd640 1 -- 172.21.15.83:0/2083426715 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f39a4058800 con 0x7f39ac15e000 2024-03-06T07:24:15.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:15.680+0000 7f39b431e640 1 -- 172.21.15.83:0/2083426715 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f39ac0c9860 con 0x7f39ac15e000 2024-03-06T07:24:16.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.491+0000 7f39a2ffd640 1 -- 172.21.15.83:0/2083426715 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v53) v1 ==== 97+0+0 (secure 0 0 0) 0x7f39a405c8a0 con 0x7f39ac15e000 2024-03-06T07:24:16.494 INFO:teuthology.orchestra.run.smithi083.stderr: removed all blocklist entries 2024-03-06T07:24:16.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.494+0000 7f39b431e640 1 -- 172.21.15.83:0/2083426715 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f399006b500 msgr2=0x7f399006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.494+0000 7f39b431e640 1 --2- 172.21.15.83:0/2083426715 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f399006b500 0x7f399006d9c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f39a8009560 tx=0x7f39a8009190 comp rx=0 tx=0).stop 2024-03-06T07:24:16.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.494+0000 7f39b431e640 1 -- 172.21.15.83:0/2083426715 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f39ac15e000 msgr2=0x7f39ac09da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.494+0000 7f39b431e640 1 --2- 172.21.15.83:0/2083426715 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f39ac15e000 0x7f39ac09da90 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f39a400c7d0 tx=0x7f39a400cc90 comp rx=0 tx=0).stop 2024-03-06T07:24:16.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.494+0000 7f39b431e640 1 -- 172.21.15.83:0/2083426715 shutdown_connections 2024-03-06T07:24:16.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.494+0000 7f39b431e640 1 --2- 172.21.15.83:0/2083426715 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f399006b500 0x7f399006d9c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.494+0000 7f39b431e640 1 --2- 172.21.15.83:0/2083426715 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f39ac15e000 0x7f39ac09da90 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.494+0000 7f39b431e640 1 --2- 172.21.15.83:0/2083426715 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f39ac0d4810 0x7f39ac153110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.494+0000 7f39b431e640 1 --2- 172.21.15.83:0/2083426715 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f39ac0c80e0 0x7f39ac0cc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.494+0000 7f39b431e640 1 -- 172.21.15.83:0/2083426715 >> 172.21.15.83:0/2083426715 conn(0x7f39ac0d3cf0 msgr2=0x7f39ac0c8850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:16.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.494+0000 7f39b431e640 1 -- 172.21.15.83:0/2083426715 shutdown_connections 2024-03-06T07:24:16.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.494+0000 7f39b431e640 1 -- 172.21.15.83:0/2083426715 wait complete. 2024-03-06T07:24:16.509 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:24:16.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.611+0000 7f51b3dd6640 1 Processor -- start 2024-03-06T07:24:16.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.612+0000 7f51b3dd6640 1 -- start start 2024-03-06T07:24:16.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.612+0000 7f51b3dd6640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f51ac13dc10 0x7f51ac0b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.612+0000 7f51b3dd6640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f51ac15ef60 0x7f51ac0b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.612+0000 7f51b3dd6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51ac15f580 con 0x7f51ac15bf30 2024-03-06T07:24:16.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.612+0000 7f51b3dd6640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51ac0cf3f0 con 0x7f51ac15ef60 2024-03-06T07:24:16.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.612+0000 7f51b3dd6640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51ac15b210 con 0x7f51ac13dc10 2024-03-06T07:24:16.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.613+0000 7f51b2373640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f51ac15ef60 0x7f51ac0b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.613+0000 7f51b1371640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f51ac15bf30 0x7f51ac15e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39300/0 (socket says 172.21.15.83:39300) 2024-03-06T07:24:16.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.613+0000 7f51b1371640 1 -- 172.21.15.83:0/1390147067 learned_addr learned my addr 172.21.15.83:0/1390147067 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:24:16.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.613+0000 7f51b1b72640 1 --2- 172.21.15.83:0/1390147067 >> v2:172.21.15.183:3300/0 conn(0x7f51ac13dc10 0x7f51ac0b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.613+0000 7f51b2373640 1 -- 172.21.15.83:0/1390147067 >> v2:172.21.15.183:3300/0 conn(0x7f51ac13dc10 msgr2=0x7f51ac0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:16.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.613+0000 7f51b0b70640 1 -- 172.21.15.83:0/1390147067 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2647598543 0 0) 0x7f51ac15f580 con 0x7f51ac15bf30 2024-03-06T07:24:16.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.613+0000 7f51b2373640 1 --2- 172.21.15.83:0/1390147067 >> v2:172.21.15.183:3300/0 conn(0x7f51ac13dc10 0x7f51ac0b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.613+0000 7f51b2373640 1 -- 172.21.15.83:0/1390147067 >> v1:172.21.15.83:6789/0 conn(0x7f51ac15bf30 legacy=0x7f51ac15e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.613+0000 7f51b2373640 1 -- 172.21.15.83:0/1390147067 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51ac15b450 con 0x7f51ac15ef60 2024-03-06T07:24:16.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.613+0000 7f51b1b72640 1 --2- 172.21.15.83:0/1390147067 >> v2:172.21.15.183:3300/0 conn(0x7f51ac13dc10 0x7f51ac0b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:16.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.614+0000 7f51b2373640 1 --2- 172.21.15.83:0/1390147067 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51ac15ef60 0x7f51ac0b2e90 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f51ac0731a0 tx=0x7f51a802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5830eb6886c6ba1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:16.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.614+0000 7f51b0b70640 1 -- 172.21.15.83:0/1390147067 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51a8039070 con 0x7f51ac15ef60 2024-03-06T07:24:16.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.614+0000 7f51b0b70640 1 -- 172.21.15.83:0/1390147067 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51a802cbf0 con 0x7f51ac15ef60 2024-03-06T07:24:16.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.614+0000 7f51b0b70640 1 -- 172.21.15.83:0/1390147067 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51a8034440 con 0x7f51ac15ef60 2024-03-06T07:24:16.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.614+0000 7f51b3dd6640 1 -- 172.21.15.83:0/1390147067 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51ac15ef60 msgr2=0x7f51ac0b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.614+0000 7f51b3dd6640 1 --2- 172.21.15.83:0/1390147067 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51ac15ef60 0x7f51ac0b2e90 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f51ac0731a0 tx=0x7f51a802be40 comp rx=0 tx=0).stop 2024-03-06T07:24:16.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.614+0000 7f51b3dd6640 1 -- 172.21.15.83:0/1390147067 shutdown_connections 2024-03-06T07:24:16.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.614+0000 7f51b3dd6640 1 --2- 172.21.15.83:0/1390147067 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51ac15ef60 0x7f51ac0b2e90 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.614+0000 7f51b3dd6640 1 --2- 172.21.15.83:0/1390147067 >> v2:172.21.15.183:3300/0 conn(0x7f51ac13dc10 0x7f51ac0b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.614+0000 7f51b3dd6640 1 -- 172.21.15.83:0/1390147067 >> 172.21.15.83:0/1390147067 conn(0x7f51ac0cf8a0 msgr2=0x7f51ac0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:16.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.614+0000 7f51b3dd6640 1 -- 172.21.15.83:0/1390147067 shutdown_connections 2024-03-06T07:24:16.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.614+0000 7f51b3dd6640 1 -- 172.21.15.83:0/1390147067 wait complete. 2024-03-06T07:24:16.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.615+0000 7f51b3dd6640 1 Processor -- start 2024-03-06T07:24:16.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.616+0000 7f51b3dd6640 1 -- start start 2024-03-06T07:24:16.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.616+0000 7f51b3dd6640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f51ac13dc10 0x7f51ac132fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.616+0000 7f51b3dd6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51ac15bf30 0x7f51ac15d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.616+0000 7f51b3dd6640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f51ac15ef60 0x7f51ac159d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.616+0000 7f51b3dd6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51ac0708d0 con 0x7f51ac15bf30 2024-03-06T07:24:16.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.616+0000 7f51b3dd6640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51ac13dff0 con 0x7f51ac13dc10 2024-03-06T07:24:16.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.616+0000 7f51b3dd6640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51ac070a50 con 0x7f51ac15ef60 2024-03-06T07:24:16.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.616+0000 7f51b1371640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51ac15bf30 0x7f51ac15d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.616+0000 7f51b1371640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51ac15bf30 0x7f51ac15d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40578/0 (socket says 172.21.15.83:40578) 2024-03-06T07:24:16.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.616+0000 7f51b1371640 1 -- 172.21.15.83:0/1510561740 learned_addr learned my addr 172.21.15.83:0/1510561740 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:16.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.617+0000 7f51b1371640 1 -- 172.21.15.83:0/1510561740 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f51ac15ef60 msgr2=0x7f51ac159d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.617+0000 7f51b2373640 1 --2- 172.21.15.83:0/1510561740 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f51ac15ef60 0x7f51ac159d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.617+0000 7f51b1b72640 1 --2- 172.21.15.83:0/1510561740 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f51ac13dc10 0x7f51ac132fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.617+0000 7f51b1371640 1 --2- 172.21.15.83:0/1510561740 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f51ac15ef60 0x7f51ac159d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.617+0000 7f51b1371640 1 -- 172.21.15.83:0/1510561740 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f51ac13dc10 msgr2=0x7f51ac132fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.617+0000 7f51b1371640 1 --2- 172.21.15.83:0/1510561740 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f51ac13dc10 0x7f51ac132fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.617+0000 7f51b1371640 1 -- 172.21.15.83:0/1510561740 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51ac1416b0 con 0x7f51ac15bf30 2024-03-06T07:24:16.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.617+0000 7f51b2373640 1 --2- 172.21.15.83:0/1510561740 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f51ac15ef60 0x7f51ac159d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:16.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.617+0000 7f51b1b72640 1 --2- 172.21.15.83:0/1510561740 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f51ac13dc10 0x7f51ac132fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:16.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.617+0000 7f51b1371640 1 --2- 172.21.15.83:0/1510561740 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51ac15bf30 0x7f51ac15d480 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f519c00c4d0 tx=0x7f519c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:16.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.617+0000 7f51a27fc640 1 -- 172.21.15.83:0/1510561740 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f519c018070 con 0x7f51ac15bf30 2024-03-06T07:24:16.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.617+0000 7f51a27fc640 1 -- 172.21.15.83:0/1510561740 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f519c007500 con 0x7f51ac15bf30 2024-03-06T07:24:16.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.617+0000 7f51b3dd6640 1 -- 172.21.15.83:0/1510561740 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51ac1f35a0 con 0x7f51ac15bf30 2024-03-06T07:24:16.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.617+0000 7f51a27fc640 1 -- 172.21.15.83:0/1510561740 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f519c004a20 con 0x7f51ac15bf30 2024-03-06T07:24:16.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.617+0000 7f51b3dd6640 1 -- 172.21.15.83:0/1510561740 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51ac141950 con 0x7f51ac15bf30 2024-03-06T07:24:16.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.618+0000 7f51b3dd6640 1 -- 172.21.15.83:0/1510561740 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51ac04f330 con 0x7f51ac15bf30 2024-03-06T07:24:16.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.618+0000 7f51a27fc640 1 -- 172.21.15.83:0/1510561740 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f519c01b070 con 0x7f51ac15bf30 2024-03-06T07:24:16.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.619+0000 7f51a27fc640 1 --2- 172.21.15.83:0/1510561740 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f519006b4d0 0x7f519006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.619+0000 7f51a27fc640 1 -- 172.21.15.83:0/1510561740 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f519c0891b0 con 0x7f51ac15bf30 2024-03-06T07:24:16.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.619+0000 7f51b1b72640 1 --2- 172.21.15.83:0/1510561740 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f519006b4d0 0x7f519006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.619+0000 7f51b1b72640 1 --2- 172.21.15.83:0/1510561740 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f519006b4d0 0x7f519006d990 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f51a4006fd0 tx=0x7f51a4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:16.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.623+0000 7f51a27fc640 1 -- 172.21.15.83:0/1510561740 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f519c058380 con 0x7f51ac15bf30 2024-03-06T07:24:16.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.858+0000 7f51b3dd6640 1 -- 172.21.15.83:0/1510561740 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f51ac06eb00 con 0x7f51ac15bf30 2024-03-06T07:24:16.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.858+0000 7f51a27fc640 1 -- 172.21.15.83:0/1510561740 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v12) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f519c057c80 con 0x7f51ac15bf30 2024-03-06T07:24:16.860 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:24:16.860 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:24:16.861 INFO:teuthology.orchestra.run.smithi083.stdout: "auth_dump": [ 2024-03-06T07:24:16.861 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:24:16.861 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mds.a", 2024-03-06T07:24:16.861 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlVHaUGhAAVRpLsHYTUkaYDmceq5Rlgw==", 2024-03-06T07:24:16.861 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:24:16.861 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:24:16.861 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow *", 2024-03-06T07:24:16.861 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow *", 2024-03-06T07:24:16.861 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:24:16.861 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:24:16.861 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:24:16.862 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:24:16.862 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mds.b", 2024-03-06T07:24:16.862 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlFlrrJhAAQbY6xU910eYRg1zwgpj9vg==", 2024-03-06T07:24:16.862 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:24:16.862 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:24:16.862 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow *", 2024-03-06T07:24:16.862 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow *", 2024-03-06T07:24:16.862 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:24:16.862 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:24:16.862 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:24:16.862 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:24:16.862 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mds.c", 2024-03-06T07:24:16.862 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlX0jxIRAAxYNi39qxIFXazlL2qNzLCg==", 2024-03-06T07:24:16.863 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:24:16.863 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:24:16.863 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow *", 2024-03-06T07:24:16.863 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow *", 2024-03-06T07:24:16.863 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:24:16.863 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:24:16.863 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:24:16.863 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:24:16.863 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mds.d", 2024-03-06T07:24:16.863 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehl3utCLhAAzEjASLnYpPYrJsWOt3WwpQ==", 2024-03-06T07:24:16.863 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:24:16.864 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:24:16.864 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow *", 2024-03-06T07:24:16.864 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow *", 2024-03-06T07:24:16.864 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:24:16.864 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:24:16.864 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:24:16.864 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:24:16.864 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.0", 2024-03-06T07:24:16.864 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBgGehlkezBExAAZOLMH2ua3CTWAL7H6LGeZw==", 2024-03-06T07:24:16.864 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:24:16.864 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:16.865 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:24:16.865 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:24:16.865 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:24:16.865 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:24:16.865 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:24:16.865 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.1", 2024-03-06T07:24:16.865 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBjGehlC04qEhAAqirTFR6y3NOGw6N+iz3/pg==", 2024-03-06T07:24:16.865 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:24:16.865 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:16.865 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:24:16.865 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:24:16.866 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:24:16.866 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:24:16.866 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:24:16.866 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.2", 2024-03-06T07:24:16.866 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBmGehlfuFWEBAAj/vjZXrv1wVNSJ1R74UmnQ==", 2024-03-06T07:24:16.866 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:24:16.866 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:16.866 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:24:16.866 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:24:16.866 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:24:16.866 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:24:16.866 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:24:16.866 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.3", 2024-03-06T07:24:16.867 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBpGehl5+ItDhAA0YNc83MhKcgb3JgRuP87aw==", 2024-03-06T07:24:16.867 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:24:16.867 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:16.867 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:24:16.867 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:24:16.867 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:24:16.867 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:24:16.867 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:24:16.867 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.4", 2024-03-06T07:24:16.867 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBvGehlYa/TFBAAvGcizpRQ21pvgpEGgSET0w==", 2024-03-06T07:24:16.867 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:24:16.867 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:16.868 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:24:16.868 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:24:16.868 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:24:16.868 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:24:16.868 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:24:16.868 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.5", 2024-03-06T07:24:16.868 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQByGehlE80TExAAFcWDgnLtW8OKlyu4B0jnmg==", 2024-03-06T07:24:16.868 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:24:16.868 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:16.868 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:24:16.868 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:24:16.869 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:24:16.869 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:24:16.869 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:24:16.869 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.6", 2024-03-06T07:24:16.869 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQB1Gehla/LbERAAYfAjTKL11VMVnGUGV5GVcg==", 2024-03-06T07:24:16.869 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:24:16.869 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:16.869 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:24:16.869 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:24:16.869 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:24:16.869 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:24:16.869 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:24:16.870 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.7", 2024-03-06T07:24:16.870 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQB4GehlpSJkEBAA17tUgMf94jfqvXlD51P8DQ==", 2024-03-06T07:24:16.870 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:24:16.870 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:16.870 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:24:16.870 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:24:16.870 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:24:16.870 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:24:16.870 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:24:16.870 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.0", 2024-03-06T07:24:16.870 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlRwCIMhAA1DdbdQZPplu7LQOTK3BmSg==", 2024-03-06T07:24:16.870 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:24:16.870 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:24:16.871 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow r", 2024-03-06T07:24:16.871 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:24:16.871 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:24:16.871 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:24:16.871 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:24:16.871 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.1", 2024-03-06T07:24:16.871 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBdGehl77spABAAsiEf+tXos3MwFrXToEQ8dQ==", 2024-03-06T07:24:16.871 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:24:16.871 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:24:16.871 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow r", 2024-03-06T07:24:16.871 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow rw", 2024-03-06T07:24:16.871 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow rwx" 2024-03-06T07:24:16.872 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:24:16.872 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:24:16.872 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:24:16.872 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.2", 2024-03-06T07:24:16.872 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehltY5SOBAAI8rcIEVqMZaWQGgNaYKJTA==", 2024-03-06T07:24:16.872 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:24:16.872 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:24:16.872 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow r", 2024-03-06T07:24:16.872 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow rw", 2024-03-06T07:24:16.872 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow rwx" 2024-03-06T07:24:16.872 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:24:16.872 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:24:16.873 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:24:16.873 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.3", 2024-03-06T07:24:16.873 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBdGehl9ZX1BRAAA7fKnRyaQ+zqHoSGs4UvDw==", 2024-03-06T07:24:16.873 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:24:16.873 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:24:16.873 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow r", 2024-03-06T07:24:16.873 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow rw", 2024-03-06T07:24:16.873 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow rwx" 2024-03-06T07:24:16.873 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:24:16.873 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:24:16.873 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:24:16.873 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.admin", 2024-03-06T07:24:16.873 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBbGehl4gqRORAAobqSWoeyHYA/lXkp6/EcIQ==", 2024-03-06T07:24:16.874 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:24:16.874 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow *", 2024-03-06T07:24:16.874 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow *", 2024-03-06T07:24:16.874 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow *", 2024-03-06T07:24:16.874 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:24:16.874 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:24:16.874 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:24:16.874 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:24:16.874 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mgr.x", 2024-03-06T07:24:16.874 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlfikrFxAAoZkTUtpHlBcUAMpwWABRxg==", 2024-03-06T07:24:16.874 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:24:16.874 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow *", 2024-03-06T07:24:16.875 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile mgr", 2024-03-06T07:24:16.875 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:24:16.875 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:24:16.875 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:24:16.875 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:24:16.875 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mgr.y", 2024-03-06T07:24:16.875 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlvdcdExAAySXlEfJqLnpMySiiYGFkMQ==", 2024-03-06T07:24:16.875 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:24:16.875 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow *", 2024-03-06T07:24:16.875 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile mgr", 2024-03-06T07:24:16.875 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:24:16.875 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:24:16.876 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:24:16.876 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-03-06T07:24:16.876 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:24:16.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.860+0000 7f51b3dd6640 1 -- 172.21.15.83:0/1510561740 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f519006b4d0 msgr2=0x7f519006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.860+0000 7f51b3dd6640 1 --2- 172.21.15.83:0/1510561740 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f519006b4d0 0x7f519006d990 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f51a4006fd0 tx=0x7f51a4008040 comp rx=0 tx=0).stop 2024-03-06T07:24:16.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.860+0000 7f51b3dd6640 1 -- 172.21.15.83:0/1510561740 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51ac15bf30 msgr2=0x7f51ac15d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.860+0000 7f51b3dd6640 1 --2- 172.21.15.83:0/1510561740 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51ac15bf30 0x7f51ac15d480 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f519c00c4d0 tx=0x7f519c00c990 comp rx=0 tx=0).stop 2024-03-06T07:24:16.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.860+0000 7f51b3dd6640 1 -- 172.21.15.83:0/1510561740 shutdown_connections 2024-03-06T07:24:16.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.860+0000 7f51b3dd6640 1 --2- 172.21.15.83:0/1510561740 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f519006b4d0 0x7f519006d990 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.860+0000 7f51b3dd6640 1 --2- 172.21.15.83:0/1510561740 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f51ac15ef60 0x7f51ac159d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.860+0000 7f51b3dd6640 1 --2- 172.21.15.83:0/1510561740 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51ac15bf30 0x7f51ac15d480 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.860+0000 7f51b3dd6640 1 --2- 172.21.15.83:0/1510561740 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f51ac13dc10 0x7f51ac132fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.860+0000 7f51b3dd6640 1 -- 172.21.15.83:0/1510561740 >> 172.21.15.83:0/1510561740 conn(0x7f51ac0cf8a0 msgr2=0x7f51ac0b0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:16.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.860+0000 7f51b3dd6640 1 -- 172.21.15.83:0/1510561740 shutdown_connections 2024-03-06T07:24:16.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.860+0000 7f51b3dd6640 1 -- 172.21.15.83:0/1510561740 wait complete. 2024-03-06T07:24:16.879 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:16.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.978+0000 7f7f1595e640 1 Processor -- start 2024-03-06T07:24:16.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.978+0000 7f7f1595e640 1 -- start start 2024-03-06T07:24:16.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.978+0000 7f7f1595e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7f100d22b0 0x7f7f100d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.978+0000 7f7f1595e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f7f100b41f0 0x7f7f100b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.978+0000 7f7f1595e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f100b3430 con 0x7f7f1015dfe0 2024-03-06T07:24:16.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.978+0000 7f7f1595e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f100d2c50 con 0x7f7f100d22b0 2024-03-06T07:24:16.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.978+0000 7f7f1595e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f100b31e0 con 0x7f7f100b41f0 2024-03-06T07:24:16.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.979+0000 7f7f0e7fc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7f100d22b0 0x7f7f100d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.979+0000 7f7f0e7fc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7f100d22b0 0x7f7f100d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40594/0 (socket says 172.21.15.83:40594) 2024-03-06T07:24:16.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.979+0000 7f7f0e7fc640 1 -- 172.21.15.83:0/631005197 learned_addr learned my addr 172.21.15.83:0/631005197 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:16.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.979+0000 7f7f0f7fe640 1 --2- 172.21.15.83:0/631005197 >> v2:172.21.15.183:3300/0 conn(0x7f7f100b41f0 0x7f7f100b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.979+0000 7f7f0e7fc640 1 -- 172.21.15.83:0/631005197 >> v2:172.21.15.183:3300/0 conn(0x7f7f100b41f0 msgr2=0x7f7f100b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:16.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.979+0000 7f7f0e7fc640 1 --2- 172.21.15.83:0/631005197 >> v2:172.21.15.183:3300/0 conn(0x7f7f100b41f0 0x7f7f100b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.979+0000 7f7f0e7fc640 1 -- 172.21.15.83:0/631005197 >> v1:172.21.15.183:6789/0 conn(0x7f7f1015dfe0 legacy=0x7f7f100cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.979+0000 7f7f0e7fc640 1 -- 172.21.15.83:0/631005197 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f100b3710 con 0x7f7f100d22b0 2024-03-06T07:24:16.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.980+0000 7f7f0e7fc640 1 --2- 172.21.15.83:0/631005197 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f100d22b0 0x7f7f100d2710 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f7f04009870 tx=0x7f7f0402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=493e236ff81cf3f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:16.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.980+0000 7f7f0dffb640 1 -- 172.21.15.83:0/631005197 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f04034070 con 0x7f7f100d22b0 2024-03-06T07:24:16.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.980+0000 7f7f0dffb640 1 -- 172.21.15.83:0/631005197 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f04030860 con 0x7f7f100d22b0 2024-03-06T07:24:16.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.980+0000 7f7f0dffb640 1 -- 172.21.15.83:0/631005197 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f04030c30 con 0x7f7f100d22b0 2024-03-06T07:24:16.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.980+0000 7f7f1595e640 1 -- 172.21.15.83:0/631005197 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f100d22b0 msgr2=0x7f7f100d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.980+0000 7f7f1595e640 1 --2- 172.21.15.83:0/631005197 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f100d22b0 0x7f7f100d2710 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f7f04009870 tx=0x7f7f0402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.980+0000 7f7f1595e640 1 -- 172.21.15.83:0/631005197 shutdown_connections 2024-03-06T07:24:16.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.980+0000 7f7f1595e640 1 --2- 172.21.15.83:0/631005197 >> v2:172.21.15.183:3300/0 conn(0x7f7f100b41f0 0x7f7f100b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.980+0000 7f7f1595e640 1 --2- 172.21.15.83:0/631005197 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f100d22b0 0x7f7f100d2710 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.980+0000 7f7f1595e640 1 -- 172.21.15.83:0/631005197 >> 172.21.15.83:0/631005197 conn(0x7f7f100d3cd0 msgr2=0x7f7f100d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:16.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.980+0000 7f7f1595e640 1 -- 172.21.15.83:0/631005197 shutdown_connections 2024-03-06T07:24:16.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.980+0000 7f7f1595e640 1 -- 172.21.15.83:0/631005197 wait complete. 2024-03-06T07:24:16.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.981+0000 7f7f1595e640 1 Processor -- start 2024-03-06T07:24:16.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.982+0000 7f7f1595e640 1 -- start start 2024-03-06T07:24:16.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.982+0000 7f7f1595e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f100b41f0 0x7f7f101599b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.982+0000 7f7f1595e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7f100d22b0 0x7f7f101f29f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.982+0000 7f7f1595e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7f1015dfe0 0x7f7f101f2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.982+0000 7f7f1595e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f100cfa00 con 0x7f7f100b41f0 2024-03-06T07:24:16.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.982+0000 7f7f1595e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f100b3d60 con 0x7f7f1015dfe0 2024-03-06T07:24:16.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.982+0000 7f7f1595e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f100d1c70 con 0x7f7f100d22b0 2024-03-06T07:24:16.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.982+0000 7f7f0effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f100b41f0 0x7f7f101599b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.982+0000 7f7f0effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f100b41f0 0x7f7f101599b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40600/0 (socket says 172.21.15.83:40600) 2024-03-06T07:24:16.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.982+0000 7f7f0effd640 1 -- 172.21.15.83:0/2447423316 learned_addr learned my addr 172.21.15.83:0/2447423316 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:16.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.983+0000 7f7f0e7fc640 1 --2- 172.21.15.83:0/2447423316 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7f100d22b0 0x7f7f101f29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.983+0000 7f7f0f7fe640 1 --2- 172.21.15.83:0/2447423316 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7f1015dfe0 0x7f7f101f2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.983+0000 7f7f0effd640 1 -- 172.21.15.83:0/2447423316 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7f100d22b0 msgr2=0x7f7f101f29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.983+0000 7f7f0effd640 1 --2- 172.21.15.83:0/2447423316 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7f100d22b0 0x7f7f101f29f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.983+0000 7f7f0effd640 1 -- 172.21.15.83:0/2447423316 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7f1015dfe0 msgr2=0x7f7f101f2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.983+0000 7f7f0effd640 1 --2- 172.21.15.83:0/2447423316 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7f1015dfe0 0x7f7f101f2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.983+0000 7f7f0effd640 1 -- 172.21.15.83:0/2447423316 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f1015a2f0 con 0x7f7f100b41f0 2024-03-06T07:24:16.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.983+0000 7f7f0e7fc640 1 --2- 172.21.15.83:0/2447423316 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7f100d22b0 0x7f7f101f29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:16.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.983+0000 7f7f0effd640 1 --2- 172.21.15.83:0/2447423316 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f100b41f0 0x7f7f101599b0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f7f0000c730 tx=0x7f7f0000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:16.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.983+0000 7f7ef7fff640 1 -- 172.21.15.83:0/2447423316 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f00016020 con 0x7f7f100b41f0 2024-03-06T07:24:16.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.983+0000 7f7ef7fff640 1 -- 172.21.15.83:0/2447423316 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f00006d50 con 0x7f7f100b41f0 2024-03-06T07:24:16.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.983+0000 7f7ef7fff640 1 -- 172.21.15.83:0/2447423316 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f00003bb0 con 0x7f7f100b41f0 2024-03-06T07:24:16.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.983+0000 7f7f1595e640 1 -- 172.21.15.83:0/2447423316 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f101581a0 con 0x7f7f100b41f0 2024-03-06T07:24:16.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.983+0000 7f7f1595e640 1 -- 172.21.15.83:0/2447423316 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f1015a590 con 0x7f7f100b41f0 2024-03-06T07:24:16.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.984+0000 7f7f1595e640 1 -- 172.21.15.83:0/2447423316 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f1004f330 con 0x7f7f100b41f0 2024-03-06T07:24:16.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.985+0000 7f7ef7fff640 1 -- 172.21.15.83:0/2447423316 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7f0000a850 con 0x7f7f100b41f0 2024-03-06T07:24:16.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.985+0000 7f7ef7fff640 1 --2- 172.21.15.83:0/2447423316 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7efc06b4d0 0x7f7efc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.985+0000 7f7ef7fff640 1 -- 172.21.15.83:0/2447423316 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7f00089ad0 con 0x7f7f100b41f0 2024-03-06T07:24:16.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.985+0000 7f7f0e7fc640 1 --2- 172.21.15.83:0/2447423316 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7efc06b4d0 0x7f7efc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.985+0000 7f7f0e7fc640 1 --2- 172.21.15.83:0/2447423316 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7efc06b4d0 0x7f7efc06d990 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7f04036000 tx=0x7f7f0403b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:16.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:16.989+0000 7f7ef7fff640 1 -- 172.21.15.83:0/2447423316 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7f00020020 con 0x7f7f100b41f0 2024-03-06T07:24:17.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.160+0000 7f7f1595e640 1 -- 172.21.15.83:0/2447423316 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7f1015d3c0 con 0x7f7f100b41f0 2024-03-06T07:24:17.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.161+0000 7f7ef7fff640 1 -- 172.21.15.83:0/2447423316 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) v1 ==== 74+0+12903 (secure 0 0 0) 0x7f7f00019310 con 0x7f7f100b41f0 2024-03-06T07:24:17.163 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:24:17.163 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":53,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:24:16.489462+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:17.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.163+0000 7f7f1595e640 1 -- 172.21.15.83:0/2447423316 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7efc06b4d0 msgr2=0x7f7efc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:17.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.163+0000 7f7f1595e640 1 --2- 172.21.15.83:0/2447423316 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7efc06b4d0 0x7f7efc06d990 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7f04036000 tx=0x7f7f0403b040 comp rx=0 tx=0).stop 2024-03-06T07:24:17.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.163+0000 7f7f1595e640 1 -- 172.21.15.83:0/2447423316 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f100b41f0 msgr2=0x7f7f101599b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:17.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.163+0000 7f7f1595e640 1 --2- 172.21.15.83:0/2447423316 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f100b41f0 0x7f7f101599b0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f7f0000c730 tx=0x7f7f0000cbf0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.163+0000 7f7f1595e640 1 -- 172.21.15.83:0/2447423316 shutdown_connections 2024-03-06T07:24:17.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.163+0000 7f7f1595e640 1 --2- 172.21.15.83:0/2447423316 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7efc06b4d0 0x7f7efc06d990 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.163+0000 7f7f1595e640 1 --2- 172.21.15.83:0/2447423316 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7f1015dfe0 0x7f7f101f2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.163+0000 7f7f1595e640 1 --2- 172.21.15.83:0/2447423316 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7f100d22b0 0x7f7f101f29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.163+0000 7f7f1595e640 1 --2- 172.21.15.83:0/2447423316 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f100b41f0 0x7f7f101599b0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.163+0000 7f7f1595e640 1 -- 172.21.15.83:0/2447423316 >> 172.21.15.83:0/2447423316 conn(0x7f7f100d3cd0 msgr2=0x7f7f100d0970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:17.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.163+0000 7f7f1595e640 1 -- 172.21.15.83:0/2447423316 shutdown_connections 2024-03-06T07:24:17.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.163+0000 7f7f1595e640 1 -- 172.21.15.83:0/2447423316 wait complete. 2024-03-06T07:24:17.176 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:17.177 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:17.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.278+0000 7f579f913640 1 Processor -- start 2024-03-06T07:24:17.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.279+0000 7f579f913640 1 -- start start 2024-03-06T07:24:17.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.279+0000 7f579f913640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f579815a220 0x7f57980cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.279+0000 7f579f913640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f57980712d0 0x7f5798071730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.279+0000 7f579f913640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57980b3320 con 0x7f579815de60 2024-03-06T07:24:17.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.279+0000 7f579f913640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57980cf610 con 0x7f579815a220 2024-03-06T07:24:17.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.279+0000 7f579f913640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5798071c70 con 0x7f57980712d0 2024-03-06T07:24:17.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.280+0000 7f579d6af640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f579815a220 0x7f57980cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.280+0000 7f579deb0640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f579815de60 0x7f579815e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39304/0 (socket says 172.21.15.83:39304) 2024-03-06T07:24:17.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.280+0000 7f579deb0640 1 -- 172.21.15.83:0/1272269464 learned_addr learned my addr 172.21.15.83:0/1272269464 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:24:17.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.280+0000 7f579d6af640 1 -- 172.21.15.83:0/1272269464 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f57980712d0 msgr2=0x7f5798071730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:17.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.280+0000 7f579d6af640 1 --2- 172.21.15.83:0/1272269464 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f57980712d0 0x7f5798071730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.280+0000 7f579d6af640 1 -- 172.21.15.83:0/1272269464 >> v1:172.21.15.83:6789/0 conn(0x7f579815de60 legacy=0x7f579815e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:17.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.280+0000 7f578bfff640 1 -- 172.21.15.83:0/1272269464 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2865593658 0 0) 0x7f57980b3320 con 0x7f579815de60 2024-03-06T07:24:17.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.280+0000 7f579d6af640 1 -- 172.21.15.83:0/1272269464 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57980728d0 con 0x7f579815a220 2024-03-06T07:24:17.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.281+0000 7f579d6af640 1 --2- 172.21.15.83:0/1272269464 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f579815a220 0x7f57980cf0d0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f578c0097a0 tx=0x7f578c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6432950158e0444f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:17.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.281+0000 7f578bfff640 1 -- 172.21.15.83:0/1272269464 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f578c0315a0 con 0x7f579815a220 2024-03-06T07:24:17.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.281+0000 7f578bfff640 1 -- 172.21.15.83:0/1272269464 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f578c037070 con 0x7f579815a220 2024-03-06T07:24:17.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.281+0000 7f578bfff640 1 -- 172.21.15.83:0/1272269464 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f578c031960 con 0x7f579815a220 2024-03-06T07:24:17.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.281+0000 7f579f913640 1 -- 172.21.15.83:0/1272269464 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f579815a220 msgr2=0x7f57980cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:17.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.281+0000 7f579f913640 1 --2- 172.21.15.83:0/1272269464 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f579815a220 0x7f57980cf0d0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f578c0097a0 tx=0x7f578c02b840 comp rx=0 tx=0).stop 2024-03-06T07:24:17.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.281+0000 7f579f913640 1 -- 172.21.15.83:0/1272269464 shutdown_connections 2024-03-06T07:24:17.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.281+0000 7f579f913640 1 --2- 172.21.15.83:0/1272269464 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f57980712d0 0x7f5798071730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.281+0000 7f579f913640 1 --2- 172.21.15.83:0/1272269464 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f579815a220 0x7f57980cf0d0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.281+0000 7f579f913640 1 -- 172.21.15.83:0/1272269464 >> 172.21.15.83:0/1272269464 conn(0x7f579806b190 msgr2=0x7f579806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:17.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.281+0000 7f579f913640 1 -- 172.21.15.83:0/1272269464 shutdown_connections 2024-03-06T07:24:17.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.281+0000 7f579f913640 1 -- 172.21.15.83:0/1272269464 wait complete. 2024-03-06T07:24:17.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.283+0000 7f579f913640 1 Processor -- start 2024-03-06T07:24:17.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.283+0000 7f579f913640 1 -- start start 2024-03-06T07:24:17.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.283+0000 7f579f913640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57980712d0 0x7f579814c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.283+0000 7f579f913640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f579815a220 0x7f579814dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.283+0000 7f579f913640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f579815de60 0x7f579815c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.283+0000 7f579f913640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5798072030 con 0x7f57980712d0 2024-03-06T07:24:17.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.283+0000 7f579f913640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5798072bb0 con 0x7f579815de60 2024-03-06T07:24:17.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.283+0000 7f579f913640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5798072530 con 0x7f579815a220 2024-03-06T07:24:17.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.283+0000 7f579d6af640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57980712d0 0x7f579814c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.283+0000 7f579d6af640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57980712d0 0x7f579814c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40620/0 (socket says 172.21.15.83:40620) 2024-03-06T07:24:17.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.283+0000 7f579d6af640 1 -- 172.21.15.83:0/3620974654 learned_addr learned my addr 172.21.15.83:0/3620974654 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:17.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.284+0000 7f579ceae640 1 --2- 172.21.15.83:0/3620974654 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f579815a220 0x7f579814dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.284+0000 7f579deb0640 1 --2- 172.21.15.83:0/3620974654 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f579815de60 0x7f579815c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.284+0000 7f579d6af640 1 -- 172.21.15.83:0/3620974654 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f579815a220 msgr2=0x7f579814dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:17.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.284+0000 7f579d6af640 1 --2- 172.21.15.83:0/3620974654 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f579815a220 0x7f579814dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.284+0000 7f579d6af640 1 -- 172.21.15.83:0/3620974654 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f579815de60 msgr2=0x7f579815c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:17.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.284+0000 7f579d6af640 1 --2- 172.21.15.83:0/3620974654 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f579815de60 0x7f579815c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.284+0000 7f579d6af640 1 -- 172.21.15.83:0/3620974654 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f579814cb80 con 0x7f57980712d0 2024-03-06T07:24:17.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.284+0000 7f579d6af640 1 --2- 172.21.15.83:0/3620974654 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57980712d0 0x7f579814c5a0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f578c000c00 tx=0x7f578c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:17.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.284+0000 7f5789ffb640 1 -- 172.21.15.83:0/3620974654 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f578c004230 con 0x7f57980712d0 2024-03-06T07:24:17.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.284+0000 7f5789ffb640 1 -- 172.21.15.83:0/3620974654 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f578c0043d0 con 0x7f57980712d0 2024-03-06T07:24:17.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.284+0000 7f5789ffb640 1 -- 172.21.15.83:0/3620974654 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f578c032880 con 0x7f57980712d0 2024-03-06T07:24:17.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.284+0000 7f579f913640 1 -- 172.21.15.83:0/3620974654 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f579814ce60 con 0x7f57980712d0 2024-03-06T07:24:17.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.284+0000 7f579f913640 1 -- 172.21.15.83:0/3620974654 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f579814d420 con 0x7f57980712d0 2024-03-06T07:24:17.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.285+0000 7f579f913640 1 -- 172.21.15.83:0/3620974654 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57980732b0 con 0x7f57980712d0 2024-03-06T07:24:17.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.285+0000 7f5789ffb640 1 -- 172.21.15.83:0/3620974654 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f578c032a20 con 0x7f57980712d0 2024-03-06T07:24:17.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.286+0000 7f5789ffb640 1 --2- 172.21.15.83:0/3620974654 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f577c06b4d0 0x7f577c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.286+0000 7f579ceae640 1 --2- 172.21.15.83:0/3620974654 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f577c06b4d0 0x7f577c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.286+0000 7f5789ffb640 1 -- 172.21.15.83:0/3620974654 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f578c02f030 con 0x7f57980712d0 2024-03-06T07:24:17.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.286+0000 7f579ceae640 1 --2- 172.21.15.83:0/3620974654 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f577c06b4d0 0x7f577c06d990 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f5798167280 tx=0x7f5794008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:17.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.290+0000 7f5789ffb640 1 -- 172.21.15.83:0/3620974654 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f578c041070 con 0x7f57980712d0 2024-03-06T07:24:17.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.460+0000 7f579f913640 1 -- 172.21.15.83:0/3620974654 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5798070ba0 con 0x7f57980712d0 2024-03-06T07:24:17.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.461+0000 7f5789ffb640 1 -- 172.21.15.83:0/3620974654 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v53) v1 ==== 93+0+10 (secure 0 0 0) 0x7f578c07b3b0 con 0x7f57980712d0 2024-03-06T07:24:17.463 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:24:17.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.463+0000 7f579f913640 1 -- 172.21.15.83:0/3620974654 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f577c06b4d0 msgr2=0x7f577c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:17.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.463+0000 7f579f913640 1 --2- 172.21.15.83:0/3620974654 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f577c06b4d0 0x7f577c06d990 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f5798167280 tx=0x7f5794008040 comp rx=0 tx=0).stop 2024-03-06T07:24:17.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.463+0000 7f579f913640 1 -- 172.21.15.83:0/3620974654 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57980712d0 msgr2=0x7f579814c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:17.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.463+0000 7f579f913640 1 --2- 172.21.15.83:0/3620974654 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57980712d0 0x7f579814c5a0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f578c000c00 tx=0x7f578c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.463+0000 7f579f913640 1 -- 172.21.15.83:0/3620974654 shutdown_connections 2024-03-06T07:24:17.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.463+0000 7f579f913640 1 --2- 172.21.15.83:0/3620974654 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f577c06b4d0 0x7f577c06d990 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.463+0000 7f579f913640 1 --2- 172.21.15.83:0/3620974654 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f579815de60 0x7f579815c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.463+0000 7f579f913640 1 --2- 172.21.15.83:0/3620974654 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f579815a220 0x7f579814dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.463+0000 7f579f913640 1 --2- 172.21.15.83:0/3620974654 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57980712d0 0x7f579814c5a0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.463+0000 7f579f913640 1 -- 172.21.15.83:0/3620974654 >> 172.21.15.83:0/3620974654 conn(0x7f579806b190 msgr2=0x7f57980cd430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:17.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.463+0000 7f579f913640 1 -- 172.21.15.83:0/3620974654 shutdown_connections 2024-03-06T07:24:17.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.463+0000 7f579f913640 1 -- 172.21.15.83:0/3620974654 wait complete. 2024-03-06T07:24:17.475 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:24:17.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.573+0000 7efc790b5640 1 Processor -- start 2024-03-06T07:24:17.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.573+0000 7efc790b5640 1 -- start start 2024-03-06T07:24:17.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.574+0000 7efc790b5640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7efc7415dfe0 0x7efc740cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.574+0000 7efc790b5640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efc740d29c0 0x7efc740b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.574+0000 7efc790b5640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc740b3430 con 0x7efc740cf830 2024-03-06T07:24:17.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.574+0000 7efc790b5640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc740b31e0 con 0x7efc7415dfe0 2024-03-06T07:24:17.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.574+0000 7efc790b5640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc740b3d60 con 0x7efc740d29c0 2024-03-06T07:24:17.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.574+0000 7efc72d9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7efc7415dfe0 0x7efc740cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.574+0000 7efc72d9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7efc7415dfe0 0x7efc740cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40634/0 (socket says 172.21.15.83:40634) 2024-03-06T07:24:17.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.574+0000 7efc72d9d640 1 -- 172.21.15.83:0/3365829939 learned_addr learned my addr 172.21.15.83:0/3365829939 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:17.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.574+0000 7efc7359e640 1 --2- 172.21.15.83:0/3365829939 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efc740d29c0 0x7efc740b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.575+0000 7efc72d9d640 1 -- 172.21.15.83:0/3365829939 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efc740d29c0 msgr2=0x7efc740b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:17.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.575+0000 7efc72d9d640 1 --2- 172.21.15.83:0/3365829939 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efc740d29c0 0x7efc740b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.575+0000 7efc72d9d640 1 -- 172.21.15.83:0/3365829939 >> v1:172.21.15.183:6789/0 conn(0x7efc740cf830 legacy=0x7efc740d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:17.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.575+0000 7efc72d9d640 1 -- 172.21.15.83:0/3365829939 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc740b3710 con 0x7efc7415dfe0 2024-03-06T07:24:17.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.575+0000 7efc72d9d640 1 --2- 172.21.15.83:0/3365829939 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc7415dfe0 0x7efc740cf2f0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7efc68009870 tx=0x7efc6802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93fc7e397b964c76 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:17.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.575+0000 7efc71d9b640 1 -- 172.21.15.83:0/3365829939 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc68034070 con 0x7efc7415dfe0 2024-03-06T07:24:17.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.576+0000 7efc71d9b640 1 -- 172.21.15.83:0/3365829939 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc68030840 con 0x7efc7415dfe0 2024-03-06T07:24:17.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.576+0000 7efc71d9b640 1 -- 172.21.15.83:0/3365829939 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc68030c10 con 0x7efc7415dfe0 2024-03-06T07:24:17.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.576+0000 7efc790b5640 1 -- 172.21.15.83:0/3365829939 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc7415dfe0 msgr2=0x7efc740cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:17.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.576+0000 7efc790b5640 1 --2- 172.21.15.83:0/3365829939 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc7415dfe0 0x7efc740cf2f0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7efc68009870 tx=0x7efc6802b980 comp rx=0 tx=0).stop 2024-03-06T07:24:17.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.576+0000 7efc790b5640 1 -- 172.21.15.83:0/3365829939 shutdown_connections 2024-03-06T07:24:17.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.576+0000 7efc790b5640 1 --2- 172.21.15.83:0/3365829939 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efc740d29c0 0x7efc740b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.576+0000 7efc790b5640 1 --2- 172.21.15.83:0/3365829939 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc7415dfe0 0x7efc740cf2f0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.576+0000 7efc790b5640 1 -- 172.21.15.83:0/3365829939 >> 172.21.15.83:0/3365829939 conn(0x7efc740d3cd0 msgr2=0x7efc740d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:17.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.576+0000 7efc790b5640 1 -- 172.21.15.83:0/3365829939 shutdown_connections 2024-03-06T07:24:17.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.576+0000 7efc790b5640 1 -- 172.21.15.83:0/3365829939 wait complete. 2024-03-06T07:24:17.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.578+0000 7efc790b5640 1 Processor -- start 2024-03-06T07:24:17.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.578+0000 7efc790b5640 1 -- start start 2024-03-06T07:24:17.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.578+0000 7efc790b5640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efc740cf830 0x7efc7415ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.578+0000 7efc790b5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc740d29c0 0x7efc740ca160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.578+0000 7efc790b5640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efc7415dfe0 0x7efc740d02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.578+0000 7efc790b5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc740d1df0 con 0x7efc740d29c0 2024-03-06T07:24:17.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.578+0000 7efc790b5640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc740d1c70 con 0x7efc740cf830 2024-03-06T07:24:17.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.578+0000 7efc790b5640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc7415e3c0 con 0x7efc7415dfe0 2024-03-06T07:24:17.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.578+0000 7efc7259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc740d29c0 0x7efc740ca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.578+0000 7efc7259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc740d29c0 0x7efc740ca160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40640/0 (socket says 172.21.15.83:40640) 2024-03-06T07:24:17.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.578+0000 7efc7259c640 1 -- 172.21.15.83:0/2529100517 learned_addr learned my addr 172.21.15.83:0/2529100517 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:17.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.579+0000 7efc7359e640 1 --2- 172.21.15.83:0/2529100517 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efc7415dfe0 0x7efc740d02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.579+0000 7efc72d9d640 1 --2- 172.21.15.83:0/2529100517 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efc740cf830 0x7efc7415ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.579+0000 7efc7259c640 1 -- 172.21.15.83:0/2529100517 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efc7415dfe0 msgr2=0x7efc740d02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:17.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.579+0000 7efc7259c640 1 --2- 172.21.15.83:0/2529100517 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efc7415dfe0 0x7efc740d02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.579+0000 7efc7259c640 1 -- 172.21.15.83:0/2529100517 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efc740cf830 msgr2=0x7efc7415ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:17.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.579+0000 7efc7259c640 1 --2- 172.21.15.83:0/2529100517 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efc740cf830 0x7efc7415ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.579+0000 7efc7259c640 1 -- 172.21.15.83:0/2529100517 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc7415b3d0 con 0x7efc740d29c0 2024-03-06T07:24:17.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.579+0000 7efc7359e640 1 --2- 172.21.15.83:0/2529100517 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efc7415dfe0 0x7efc740d02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:17.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.579+0000 7efc72d9d640 1 --2- 172.21.15.83:0/2529100517 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efc740cf830 0x7efc7415ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:17.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.579+0000 7efc7259c640 1 --2- 172.21.15.83:0/2529100517 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc740d29c0 0x7efc740ca160 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7efc6400c440 tx=0x7efc64001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:17.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.579+0000 7efc5b7fe640 1 -- 172.21.15.83:0/2529100517 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc640052d0 con 0x7efc740d29c0 2024-03-06T07:24:17.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.579+0000 7efc5b7fe640 1 -- 172.21.15.83:0/2529100517 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc64005470 con 0x7efc740d29c0 2024-03-06T07:24:17.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.579+0000 7efc790b5640 1 -- 172.21.15.83:0/2529100517 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc740caf80 con 0x7efc740d29c0 2024-03-06T07:24:17.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.579+0000 7efc5b7fe640 1 -- 172.21.15.83:0/2529100517 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc64007260 con 0x7efc740d29c0 2024-03-06T07:24:17.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.579+0000 7efc790b5640 1 -- 172.21.15.83:0/2529100517 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc740cb540 con 0x7efc740d29c0 2024-03-06T07:24:17.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.580+0000 7efc790b5640 1 -- 172.21.15.83:0/2529100517 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc74068090 con 0x7efc740d29c0 2024-03-06T07:24:17.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.580+0000 7efc5b7fe640 1 -- 172.21.15.83:0/2529100517 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efc64007400 con 0x7efc740d29c0 2024-03-06T07:24:17.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.581+0000 7efc5b7fe640 1 --2- 172.21.15.83:0/2529100517 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efc6006b4d0 0x7efc6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.581+0000 7efc5b7fe640 1 -- 172.21.15.83:0/2529100517 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4857+0+0 (secure 0 0 0) 0x7efc64089360 con 0x7efc740d29c0 2024-03-06T07:24:17.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.581+0000 7efc72d9d640 1 --2- 172.21.15.83:0/2529100517 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efc6006b4d0 0x7efc6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.581+0000 7efc72d9d640 1 --2- 172.21.15.83:0/2529100517 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efc6006b4d0 0x7efc6006d990 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7efc68036000 tx=0x7efc6803b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:17.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.585+0000 7efc5b7fe640 1 -- 172.21.15.83:0/2529100517 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efc74068090 con 0x7efc740d29c0 2024-03-06T07:24:17.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.758+0000 7efc790b5640 1 -- 172.21.15.83:0/2529100517 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7efc7404f330 con 0x7efc740d29c0 2024-03-06T07:24:17.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.758+0000 7efc5b7fe640 1 -- 172.21.15.83:0/2529100517 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v53) v1 ==== 84+0+65 (secure 0 0 0) 0x7efc640585c0 con 0x7efc740d29c0 2024-03-06T07:24:17.760 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:24:17.760 INFO:teuthology.orchestra.run.smithi083.stdout:[ 2024-03-06T07:24:17.760 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:24:17.760 INFO:teuthology.orchestra.run.smithi083.stdout: "poolnum": 1, 2024-03-06T07:24:17.760 INFO:teuthology.orchestra.run.smithi083.stdout: "poolname": ".mgr" 2024-03-06T07:24:17.760 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:24:17.761 INFO:teuthology.orchestra.run.smithi083.stdout:] 2024-03-06T07:24:17.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.760+0000 7efc790b5640 1 -- 172.21.15.83:0/2529100517 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efc6006b4d0 msgr2=0x7efc6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:17.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.760+0000 7efc790b5640 1 --2- 172.21.15.83:0/2529100517 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efc6006b4d0 0x7efc6006d990 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7efc68036000 tx=0x7efc6803b040 comp rx=0 tx=0).stop 2024-03-06T07:24:17.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.760+0000 7efc790b5640 1 -- 172.21.15.83:0/2529100517 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc740d29c0 msgr2=0x7efc740ca160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:17.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.760+0000 7efc790b5640 1 --2- 172.21.15.83:0/2529100517 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc740d29c0 0x7efc740ca160 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7efc6400c440 tx=0x7efc64001e70 comp rx=0 tx=0).stop 2024-03-06T07:24:17.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.760+0000 7efc790b5640 1 -- 172.21.15.83:0/2529100517 shutdown_connections 2024-03-06T07:24:17.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.760+0000 7efc790b5640 1 --2- 172.21.15.83:0/2529100517 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efc6006b4d0 0x7efc6006d990 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.760+0000 7efc790b5640 1 --2- 172.21.15.83:0/2529100517 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efc7415dfe0 0x7efc740d02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.760+0000 7efc790b5640 1 --2- 172.21.15.83:0/2529100517 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc740d29c0 0x7efc740ca160 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.760+0000 7efc790b5640 1 --2- 172.21.15.83:0/2529100517 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efc740cf830 0x7efc7415ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.760+0000 7efc790b5640 1 -- 172.21.15.83:0/2529100517 >> 172.21.15.83:0/2529100517 conn(0x7efc740d3cd0 msgr2=0x7efc7406b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:17.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.760+0000 7efc790b5640 1 -- 172.21.15.83:0/2529100517 shutdown_connections 2024-03-06T07:24:17.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.760+0000 7efc790b5640 1 -- 172.21.15.83:0/2529100517 wait complete. 2024-03-06T07:24:17.772 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:24:17.773 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:24:17.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.875+0000 7fd73ed7a640 1 Processor -- start 2024-03-06T07:24:17.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.875+0000 7fd73ed7a640 1 -- start start 2024-03-06T07:24:17.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.876+0000 7fd73ed7a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd73815ac00 0x7fd73806c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.876+0000 7fd73ed7a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd738070d10 0x7fd73815eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.876+0000 7fd73ed7a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd73815afe0 con 0x7fd7380708b0 2024-03-06T07:24:17.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.876+0000 7fd73ed7a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7380cdc10 con 0x7fd73815ac00 2024-03-06T07:24:17.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.876+0000 7fd73ed7a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7380cd000 con 0x7fd738070d10 2024-03-06T07:24:17.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.876+0000 7fd73cb16640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd73815ac00 0x7fd73806c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.876+0000 7fd73cb16640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd73815ac00 0x7fd73806c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40646/0 (socket says 172.21.15.83:40646) 2024-03-06T07:24:17.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.876+0000 7fd73cb16640 1 -- 172.21.15.83:0/428246555 learned_addr learned my addr 172.21.15.83:0/428246555 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:17.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.876+0000 7fd73cb16640 1 -- 172.21.15.83:0/428246555 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd738070d10 msgr2=0x7fd73815eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:17.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.876+0000 7fd73d317640 1 --2- 172.21.15.83:0/428246555 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd738070d10 0x7fd73815eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.876+0000 7fd73cb16640 1 --2- 172.21.15.83:0/428246555 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd738070d10 0x7fd73815eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.876+0000 7fd73cb16640 1 -- 172.21.15.83:0/428246555 >> v1:172.21.15.183:6789/0 conn(0x7fd7380708b0 legacy=0x7fd73815e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:17.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.876+0000 7fd73cb16640 1 -- 172.21.15.83:0/428246555 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7380cd2e0 con 0x7fd73815ac00 2024-03-06T07:24:17.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.877+0000 7fd73cb16640 1 --2- 172.21.15.83:0/428246555 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd73815ac00 0x7fd73806c910 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fd73000d150 tx=0x7fd73002e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7b08d875178bdd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:17.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.877+0000 7fd7377fe640 1 -- 172.21.15.83:0/428246555 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd73002e9e0 con 0x7fd73815ac00 2024-03-06T07:24:17.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.877+0000 7fd7377fe640 1 -- 172.21.15.83:0/428246555 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd73002eb80 con 0x7fd73815ac00 2024-03-06T07:24:17.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.877+0000 7fd7377fe640 1 -- 172.21.15.83:0/428246555 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd730009600 con 0x7fd73815ac00 2024-03-06T07:24:17.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.877+0000 7fd73ed7a640 1 -- 172.21.15.83:0/428246555 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd73815ac00 msgr2=0x7fd73806c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:17.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.877+0000 7fd73ed7a640 1 --2- 172.21.15.83:0/428246555 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd73815ac00 0x7fd73806c910 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fd73000d150 tx=0x7fd73002e040 comp rx=0 tx=0).stop 2024-03-06T07:24:17.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.877+0000 7fd73ed7a640 1 -- 172.21.15.83:0/428246555 shutdown_connections 2024-03-06T07:24:17.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.877+0000 7fd73ed7a640 1 --2- 172.21.15.83:0/428246555 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd738070d10 0x7fd73815eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.877+0000 7fd73ed7a640 1 --2- 172.21.15.83:0/428246555 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd73815ac00 0x7fd73806c910 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.877+0000 7fd73ed7a640 1 -- 172.21.15.83:0/428246555 >> 172.21.15.83:0/428246555 conn(0x7fd7380b1650 msgr2=0x7fd7380b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:17.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.878+0000 7fd73ed7a640 1 -- 172.21.15.83:0/428246555 shutdown_connections 2024-03-06T07:24:17.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.878+0000 7fd73ed7a640 1 -- 172.21.15.83:0/428246555 wait complete. 2024-03-06T07:24:17.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.879+0000 7fd73ed7a640 1 Processor -- start 2024-03-06T07:24:17.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.879+0000 7fd73ed7a640 1 -- start start 2024-03-06T07:24:17.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.879+0000 7fd73ed7a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd7380708b0 0x7fd73814caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.879+0000 7fd73ed7a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd738070d10 0x7fd7380af5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.879+0000 7fd73ed7a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd73815ac00 0x7fd7381eed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.879+0000 7fd73ed7a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd738071190 con 0x7fd7380708b0 2024-03-06T07:24:17.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.879+0000 7fd73ed7a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7380cd8a0 con 0x7fd73815ac00 2024-03-06T07:24:17.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.879+0000 7fd73ed7a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd73806ce50 con 0x7fd738070d10 2024-03-06T07:24:17.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.880+0000 7fd73cb16640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd7380708b0 0x7fd73814caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.880+0000 7fd73cb16640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd7380708b0 0x7fd73814caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40652/0 (socket says 172.21.15.83:40652) 2024-03-06T07:24:17.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.880+0000 7fd73cb16640 1 -- 172.21.15.83:0/3561553048 learned_addr learned my addr 172.21.15.83:0/3561553048 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:17.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.880+0000 7fd737fff640 1 --2- 172.21.15.83:0/3561553048 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd738070d10 0x7fd7380af5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.880+0000 7fd73d317640 1 --2- 172.21.15.83:0/3561553048 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd73815ac00 0x7fd7381eed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.880+0000 7fd73cb16640 1 -- 172.21.15.83:0/3561553048 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd738070d10 msgr2=0x7fd7380af5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:17.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.880+0000 7fd73cb16640 1 --2- 172.21.15.83:0/3561553048 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd738070d10 0x7fd7380af5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.880+0000 7fd73cb16640 1 -- 172.21.15.83:0/3561553048 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd73815ac00 msgr2=0x7fd7381eed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:17.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.880+0000 7fd73cb16640 1 --2- 172.21.15.83:0/3561553048 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd73815ac00 0x7fd7381eed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.880+0000 7fd73cb16640 1 -- 172.21.15.83:0/3561553048 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd73814d080 con 0x7fd7380708b0 2024-03-06T07:24:17.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.880+0000 7fd73cb16640 1 --2- 172.21.15.83:0/3561553048 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd7380708b0 0x7fd73814caa0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fd730038040 tx=0x7fd730004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:17.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.880+0000 7fd7357fa640 1 -- 172.21.15.83:0/3561553048 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd730045020 con 0x7fd7380708b0 2024-03-06T07:24:17.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.880+0000 7fd7357fa640 1 -- 172.21.15.83:0/3561553048 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd730040660 con 0x7fd7380708b0 2024-03-06T07:24:17.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.880+0000 7fd7357fa640 1 -- 172.21.15.83:0/3561553048 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd73000b720 con 0x7fd7380708b0 2024-03-06T07:24:17.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.880+0000 7fd73ed7a640 1 -- 172.21.15.83:0/3561553048 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd73814d360 con 0x7fd7380708b0 2024-03-06T07:24:17.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.880+0000 7fd73ed7a640 1 -- 172.21.15.83:0/3561553048 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd73814d920 con 0x7fd7380708b0 2024-03-06T07:24:17.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.881+0000 7fd73ed7a640 1 -- 172.21.15.83:0/3561553048 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7380687d0 con 0x7fd7380708b0 2024-03-06T07:24:17.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.882+0000 7fd7357fa640 1 -- 172.21.15.83:0/3561553048 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd730048020 con 0x7fd7380708b0 2024-03-06T07:24:17.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.882+0000 7fd7357fa640 1 --2- 172.21.15.83:0/3561553048 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd72406b4d0 0x7fd72406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.882+0000 7fd7357fa640 1 -- 172.21.15.83:0/3561553048 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd7300aeb60 con 0x7fd7380708b0 2024-03-06T07:24:17.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.882+0000 7fd737fff640 1 --2- 172.21.15.83:0/3561553048 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd72406b4d0 0x7fd72406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.882+0000 7fd737fff640 1 --2- 172.21.15.83:0/3561553048 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd72406b4d0 0x7fd72406d990 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fd7380cbb00 tx=0x7fd72c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:17.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:17.886+0000 7fd7357fa640 1 -- 172.21.15.83:0/3561553048 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd73004f030 con 0x7fd7380708b0 2024-03-06T07:24:18.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.055+0000 7fd73ed7a640 1 -- 172.21.15.83:0/3561553048 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fd738156160 con 0x7fd7380708b0 2024-03-06T07:24:18.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.500+0000 7fd7357fa640 1 -- 172.21.15.83:0/3561553048 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v54) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd73007d6f0 con 0x7fd7380708b0 2024-03-06T07:24:18.502 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'cephfs_metadata' created 2024-03-06T07:24:18.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.503+0000 7fd73ed7a640 1 -- 172.21.15.83:0/3561553048 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd72406b4d0 msgr2=0x7fd72406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:18.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.503+0000 7fd73ed7a640 1 --2- 172.21.15.83:0/3561553048 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd72406b4d0 0x7fd72406d990 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fd7380cbb00 tx=0x7fd72c009040 comp rx=0 tx=0).stop 2024-03-06T07:24:18.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.503+0000 7fd73ed7a640 1 -- 172.21.15.83:0/3561553048 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd7380708b0 msgr2=0x7fd73814caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:18.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.503+0000 7fd73ed7a640 1 --2- 172.21.15.83:0/3561553048 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd7380708b0 0x7fd73814caa0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fd730038040 tx=0x7fd730004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:18.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.503+0000 7fd73ed7a640 1 -- 172.21.15.83:0/3561553048 shutdown_connections 2024-03-06T07:24:18.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.503+0000 7fd73ed7a640 1 --2- 172.21.15.83:0/3561553048 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd72406b4d0 0x7fd72406d990 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.503+0000 7fd73ed7a640 1 --2- 172.21.15.83:0/3561553048 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd73815ac00 0x7fd7381eed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.503+0000 7fd73ed7a640 1 --2- 172.21.15.83:0/3561553048 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd738070d10 0x7fd7380af5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.503+0000 7fd73ed7a640 1 --2- 172.21.15.83:0/3561553048 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd7380708b0 0x7fd73814caa0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.503+0000 7fd73ed7a640 1 -- 172.21.15.83:0/3561553048 >> 172.21.15.83:0/3561553048 conn(0x7fd7380b1650 msgr2=0x7fd7381551c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:18.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.503+0000 7fd73ed7a640 1 -- 172.21.15.83:0/3561553048 shutdown_connections 2024-03-06T07:24:18.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.503+0000 7fd73ed7a640 1 -- 172.21.15.83:0/3561553048 wait complete. 2024-03-06T07:24:18.520 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:24:18.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.627+0000 7f367d088640 1 Processor -- start 2024-03-06T07:24:18.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.628+0000 7f367d088640 1 -- start start 2024-03-06T07:24:18.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.628+0000 7f367d088640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f36780cfa90 0x7f36780cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:18.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.628+0000 7f367d088640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f36780b33a0 0x7f36780b37c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:18.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.628+0000 7f367d088640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3678158ac0 con 0x7f36781586e0 2024-03-06T07:24:18.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.628+0000 7f367d088640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f367815b2a0 con 0x7f36780cfa90 2024-03-06T07:24:18.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.628+0000 7f367d088640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f367815b540 con 0x7f36780b33a0 2024-03-06T07:24:18.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.629+0000 7f367759e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f36780b33a0 0x7f36780b37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:18.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.629+0000 7f366ffff640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f36780cfa90 0x7f36780cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:18.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.629+0000 7f3676d9d640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f36781586e0 0x7f36780aea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:59090/0 (socket says 172.21.15.83:59090) 2024-03-06T07:24:18.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.629+0000 7f367759e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f36780b33a0 0x7f36780b37c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:36622/0 (socket says 172.21.15.83:36622) 2024-03-06T07:24:18.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.629+0000 7f366ffff640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f36780cfa90 0x7f36780cff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:60840/0 (socket says 172.21.15.83:60840) 2024-03-06T07:24:18.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.629+0000 7f3676d9d640 1 -- 172.21.15.83:0/3384621626 learned_addr learned my addr 172.21.15.83:0/3384621626 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:24:18.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.629+0000 7f367759e640 1 -- 172.21.15.83:0/3384621626 >> v2:172.21.15.183:3300/0 conn(0x7f36780cfa90 msgr2=0x7f36780cff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:18.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.629+0000 7f367759e640 1 --2- 172.21.15.83:0/3384621626 >> v2:172.21.15.183:3300/0 conn(0x7f36780cfa90 0x7f36780cff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.629+0000 7f367759e640 1 -- 172.21.15.83:0/3384621626 >> v1:172.21.15.183:6789/0 conn(0x7f36781586e0 legacy=0x7f36780aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:18.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.629+0000 7f367759e640 1 -- 172.21.15.83:0/3384621626 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f367815b820 con 0x7f36780b33a0 2024-03-06T07:24:18.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.629+0000 7f366ffff640 1 --2- 172.21.15.83:0/3384621626 >> v2:172.21.15.183:3300/0 conn(0x7f36780cfa90 0x7f36780cff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:18.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.630+0000 7f367759e640 1 --2- 172.21.15.83:0/3384621626 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f36780b33a0 0x7f36780b37c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3670009d90 tx=0x7f367002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7900ce1463eedc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:18.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.630+0000 7f367659c640 1 -- 172.21.15.83:0/3384621626 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3670039070 con 0x7f36780b33a0 2024-03-06T07:24:18.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.630+0000 7f367659c640 1 -- 172.21.15.83:0/3384621626 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f367002cc50 con 0x7f36780b33a0 2024-03-06T07:24:18.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.630+0000 7f367659c640 1 -- 172.21.15.83:0/3384621626 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3670034440 con 0x7f36780b33a0 2024-03-06T07:24:18.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.630+0000 7f367d088640 1 -- 172.21.15.83:0/3384621626 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f36780b33a0 msgr2=0x7f36780b37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:18.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.630+0000 7f367d088640 1 --2- 172.21.15.83:0/3384621626 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f36780b33a0 0x7f36780b37c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3670009d90 tx=0x7f367002bea0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.631+0000 7f367d088640 1 -- 172.21.15.83:0/3384621626 shutdown_connections 2024-03-06T07:24:18.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.631+0000 7f367d088640 1 --2- 172.21.15.83:0/3384621626 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f36780b33a0 0x7f36780b37c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.631+0000 7f367d088640 1 --2- 172.21.15.83:0/3384621626 >> v2:172.21.15.183:3300/0 conn(0x7f36780cfa90 0x7f36780cff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.631+0000 7f367d088640 1 -- 172.21.15.83:0/3384621626 >> 172.21.15.83:0/3384621626 conn(0x7f367806b190 msgr2=0x7f367806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:18.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.631+0000 7f367d088640 1 -- 172.21.15.83:0/3384621626 shutdown_connections 2024-03-06T07:24:18.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.631+0000 7f367d088640 1 -- 172.21.15.83:0/3384621626 wait complete. 2024-03-06T07:24:18.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.632+0000 7f367d088640 1 Processor -- start 2024-03-06T07:24:18.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.632+0000 7f367d088640 1 -- start start 2024-03-06T07:24:18.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.633+0000 7f367d088640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f36780b33a0 0x7f367814ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:18.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.633+0000 7f367d088640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f36780cfa90 0x7f36780a9db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:18.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.633+0000 7f367d088640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f36781586e0 0x7f36781557d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:18.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.633+0000 7f367d088640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36780af150 con 0x7f36780b33a0 2024-03-06T07:24:18.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.633+0000 7f367d088640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367815bde0 con 0x7f36781586e0 2024-03-06T07:24:18.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.633+0000 7f367d088640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3678071350 con 0x7f36780cfa90 2024-03-06T07:24:18.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.633+0000 7f3676d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f36780b33a0 0x7f367814ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:18.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.633+0000 7f3676d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f36780b33a0 0x7f367814ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40656/0 (socket says 172.21.15.83:40656) 2024-03-06T07:24:18.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.633+0000 7f3676d9d640 1 -- 172.21.15.83:0/3577609036 learned_addr learned my addr 172.21.15.83:0/3577609036 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:18.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.633+0000 7f367759e640 1 --2- 172.21.15.83:0/3577609036 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f36781586e0 0x7f36781557d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:18.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.633+0000 7f366ffff640 1 --2- 172.21.15.83:0/3577609036 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f36780cfa90 0x7f36780a9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:18.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.633+0000 7f3676d9d640 1 -- 172.21.15.83:0/3577609036 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f36780cfa90 msgr2=0x7f36780a9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:18.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.633+0000 7f3676d9d640 1 --2- 172.21.15.83:0/3577609036 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f36780cfa90 0x7f36780a9db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.633+0000 7f3676d9d640 1 -- 172.21.15.83:0/3577609036 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f36781586e0 msgr2=0x7f36781557d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:18.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.633+0000 7f3676d9d640 1 --2- 172.21.15.83:0/3577609036 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f36781586e0 0x7f36781557d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.633+0000 7f3676d9d640 1 -- 172.21.15.83:0/3577609036 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36780a95d0 con 0x7f36780b33a0 2024-03-06T07:24:18.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.633+0000 7f3676d9d640 1 --2- 172.21.15.83:0/3577609036 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f36780b33a0 0x7f367814ad60 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f366800c3c0 tx=0x7f366800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:18.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.633+0000 7f366f7fe640 1 -- 172.21.15.83:0/3577609036 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3668017070 con 0x7f36780b33a0 2024-03-06T07:24:18.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.633+0000 7f366f7fe640 1 -- 172.21.15.83:0/3577609036 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3668005ce0 con 0x7f36780b33a0 2024-03-06T07:24:18.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.633+0000 7f366f7fe640 1 -- 172.21.15.83:0/3577609036 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36680072f0 con 0x7f36780b33a0 2024-03-06T07:24:18.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.633+0000 7f367d088640 1 -- 172.21.15.83:0/3577609036 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3678156290 con 0x7f36780b33a0 2024-03-06T07:24:18.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.633+0000 7f367d088640 1 -- 172.21.15.83:0/3577609036 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36780a9890 con 0x7f36780b33a0 2024-03-06T07:24:18.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.634+0000 7f367d088640 1 -- 172.21.15.83:0/3577609036 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36780732b0 con 0x7f36780b33a0 2024-03-06T07:24:18.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.635+0000 7f366f7fe640 1 -- 172.21.15.83:0/3577609036 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f366801a070 con 0x7f36780b33a0 2024-03-06T07:24:18.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.635+0000 7f366f7fe640 1 --2- 172.21.15.83:0/3577609036 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f365806b4d0 0x7f365806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:18.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.635+0000 7f366ffff640 1 --2- 172.21.15.83:0/3577609036 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f365806b4d0 0x7f365806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:18.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.635+0000 7f366f7fe640 1 -- 172.21.15.83:0/3577609036 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f36680893c0 con 0x7f36780b33a0 2024-03-06T07:24:18.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.636+0000 7f366ffff640 1 --2- 172.21.15.83:0/3577609036 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f365806b4d0 0x7f365806d990 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f3660009560 tx=0x7f3660009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:18.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.640+0000 7f366f7fe640 1 -- 172.21.15.83:0/3577609036 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3668057d50 con 0x7f36780b33a0 2024-03-06T07:24:18.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:18.812+0000 7f367d088640 1 -- 172.21.15.83:0/3577609036 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f3678068ff0 con 0x7f36780b33a0 2024-03-06T07:24:19.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.504+0000 7f366f7fe640 1 -- 172.21.15.83:0/3577609036 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v55) v1 ==== 170+0+0 (secure 0 0 0) 0x7f366805bdf0 con 0x7f36780b33a0 2024-03-06T07:24:19.506 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'cephfs_data' created 2024-03-06T07:24:19.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.506+0000 7f367d088640 1 -- 172.21.15.83:0/3577609036 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f365806b4d0 msgr2=0x7f365806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.506+0000 7f367d088640 1 --2- 172.21.15.83:0/3577609036 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f365806b4d0 0x7f365806d990 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f3660009560 tx=0x7f3660009190 comp rx=0 tx=0).stop 2024-03-06T07:24:19.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.506+0000 7f367d088640 1 -- 172.21.15.83:0/3577609036 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f36780b33a0 msgr2=0x7f367814ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.506+0000 7f367d088640 1 --2- 172.21.15.83:0/3577609036 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f36780b33a0 0x7f367814ad60 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f366800c3c0 tx=0x7f366800c880 comp rx=0 tx=0).stop 2024-03-06T07:24:19.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.506+0000 7f367d088640 1 -- 172.21.15.83:0/3577609036 shutdown_connections 2024-03-06T07:24:19.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.506+0000 7f367d088640 1 --2- 172.21.15.83:0/3577609036 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f365806b4d0 0x7f365806d990 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.506+0000 7f367d088640 1 --2- 172.21.15.83:0/3577609036 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f36781586e0 0x7f36781557d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.506+0000 7f367d088640 1 --2- 172.21.15.83:0/3577609036 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f36780cfa90 0x7f36780a9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.506+0000 7f367d088640 1 --2- 172.21.15.83:0/3577609036 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f36780b33a0 0x7f367814ad60 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.506+0000 7f367d088640 1 -- 172.21.15.83:0/3577609036 >> 172.21.15.83:0/3577609036 conn(0x7f367806b190 msgr2=0x7f36780cbbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:19.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.506+0000 7f367d088640 1 -- 172.21.15.83:0/3577609036 shutdown_connections 2024-03-06T07:24:19.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.507+0000 7f367d088640 1 -- 172.21.15.83:0/3577609036 wait complete. 2024-03-06T07:24:19.530 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:24:19.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.632+0000 7ff19efd7640 1 Processor -- start 2024-03-06T07:24:19.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.632+0000 7ff19efd7640 1 -- start start 2024-03-06T07:24:19.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.633+0000 7ff19efd7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff198158800 0x7ff19815b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.633+0000 7ff19efd7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff19815e8a0 con 0x7ff1980b3cd0 2024-03-06T07:24:19.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.633+0000 7ff19efd7640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff19815eb80 con 0x7ff198158420 2024-03-06T07:24:19.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.633+0000 7ff19efd7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff19815e650 con 0x7ff198158800 2024-03-06T07:24:19.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.633+0000 7ff19d574640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff1980b3cd0 0x7ff19815b8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39318/0 (socket says 172.21.15.83:39318) 2024-03-06T07:24:19.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.633+0000 7ff19d574640 1 -- 172.21.15.83:0/555246822 learned_addr learned my addr 172.21.15.83:0/555246822 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:24:19.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.633+0000 7ff197fff640 1 --2- 172.21.15.83:0/555246822 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff198158800 0x7ff19815b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.633+0000 7ff1977fe640 1 -- 172.21.15.83:0/555246822 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 936059701 0 0) 0x7ff19815e8a0 con 0x7ff1980b3cd0 2024-03-06T07:24:19.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.633+0000 7ff1977fe640 1 -- 172.21.15.83:0/555246822 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff180003520 con 0x7ff1980b3cd0 2024-03-06T07:24:19.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.634+0000 7ff1977fe640 1 -- 172.21.15.83:0/555246822 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2098916989 0 0) 0x7ff180003520 con 0x7ff1980b3cd0 2024-03-06T07:24:19.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.634+0000 7ff1977fe640 1 -- 172.21.15.83:0/555246822 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff198158800 msgr2=0x7ff19815b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:19.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.634+0000 7ff1977fe640 1 --2- 172.21.15.83:0/555246822 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff198158800 0x7ff19815b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.634+0000 7ff1977fe640 1 -- 172.21.15.83:0/555246822 >> v1:172.21.15.183:6789/0 conn(0x7ff198158420 legacy=0x7ff1980b35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.634+0000 7ff1977fe640 1 -- 172.21.15.83:0/555246822 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff19815ee60 con 0x7ff1980b3cd0 2024-03-06T07:24:19.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.634+0000 7ff1977fe640 1 -- 172.21.15.83:0/555246822 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7ff1900019f0 con 0x7ff1980b3cd0 2024-03-06T07:24:19.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.634+0000 7ff1977fe640 1 -- 172.21.15.83:0/555246822 >> v1:172.21.15.83:6789/0 conn(0x7ff1980b3cd0 legacy=0x7ff19815b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.634+0000 7ff1977fe640 1 --2- 172.21.15.83:0/555246822 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1800039e0 0x7ff180003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.634+0000 7ff1977fe640 1 --2- 172.21.15.83:0/555246822 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff180004390 0x7ff180004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.634+0000 7ff1977fe640 1 --2- 172.21.15.83:0/555246822 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff180005100 0x7ff18000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.634+0000 7ff1977fe640 1 -- 172.21.15.83:0/555246822 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff19815e650 con 0x7ff1800039e0 2024-03-06T07:24:19.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.634+0000 7ff1977fe640 1 -- 172.21.15.83:0/555246822 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18000bf00 con 0x7ff180004390 2024-03-06T07:24:19.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.634+0000 7ff1977fe640 1 -- 172.21.15.83:0/555246822 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff18000c080 con 0x7ff180005100 2024-03-06T07:24:19.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.634+0000 7ff19cd73640 1 --2- 172.21.15.83:0/555246822 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1800039e0 0x7ff180003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.634+0000 7ff197fff640 1 --2- 172.21.15.83:0/555246822 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff180004390 0x7ff180004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.634+0000 7ff19d574640 1 --2- 172.21.15.83:0/555246822 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff180005100 0x7ff18000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.634+0000 7ff19cd73640 1 -- 172.21.15.83:0/555246822 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff180005100 msgr2=0x7ff18000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:19.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.634+0000 7ff19cd73640 1 --2- 172.21.15.83:0/555246822 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff180005100 0x7ff18000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.634+0000 7ff19cd73640 1 -- 172.21.15.83:0/555246822 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff180004390 msgr2=0x7ff180004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:19.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.634+0000 7ff19cd73640 1 --2- 172.21.15.83:0/555246822 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff180004390 0x7ff180004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.634+0000 7ff19cd73640 1 -- 172.21.15.83:0/555246822 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff180003520 con 0x7ff1800039e0 2024-03-06T07:24:19.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.634+0000 7ff197fff640 1 --2- 172.21.15.83:0/555246822 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff180004390 0x7ff180004860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:19.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.635+0000 7ff19cd73640 1 --2- 172.21.15.83:0/555246822 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1800039e0 0x7ff180003dc0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7ff188001860 tx=0x7ff180005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c882a9ded2d97c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:19.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.635+0000 7ff1977fe640 1 -- 172.21.15.83:0/555246822 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff18802f050 con 0x7ff1800039e0 2024-03-06T07:24:19.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.635+0000 7ff1977fe640 1 -- 172.21.15.83:0/555246822 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff188033650 con 0x7ff1800039e0 2024-03-06T07:24:19.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.635+0000 7ff1977fe640 1 -- 172.21.15.83:0/555246822 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff188001430 con 0x7ff1800039e0 2024-03-06T07:24:19.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.635+0000 7ff19efd7640 1 -- 172.21.15.83:0/555246822 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1800039e0 msgr2=0x7ff180003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.635+0000 7ff19efd7640 1 --2- 172.21.15.83:0/555246822 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1800039e0 0x7ff180003dc0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7ff188001860 tx=0x7ff180005830 comp rx=0 tx=0).stop 2024-03-06T07:24:19.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.635+0000 7ff19d574640 1 -- 172.21.15.83:0/555246822 reap_dead start 2024-03-06T07:24:19.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.635+0000 7ff19efd7640 1 -- 172.21.15.83:0/555246822 shutdown_connections 2024-03-06T07:24:19.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.635+0000 7ff19efd7640 1 -- 172.21.15.83:0/555246822 >> 172.21.15.83:0/555246822 conn(0x7ff19806afb0 msgr2=0x7ff19806b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:19.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.635+0000 7ff19efd7640 1 -- 172.21.15.83:0/555246822 shutdown_connections 2024-03-06T07:24:19.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.635+0000 7ff19efd7640 1 -- 172.21.15.83:0/555246822 wait complete. 2024-03-06T07:24:19.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.637+0000 7ff19efd7640 1 Processor -- start 2024-03-06T07:24:19.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.637+0000 7ff19efd7640 1 -- start start 2024-03-06T07:24:19.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.637+0000 7ff19efd7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1800039e0 0x7ff1980ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.637+0000 7ff19efd7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff1981f4ce0 0x7ff1981e4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.637+0000 7ff19efd7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1981f5140 0x7ff1981e5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.637+0000 7ff19efd7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff198072bb0 con 0x7ff1800039e0 2024-03-06T07:24:19.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.637+0000 7ff19efd7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff198072a30 con 0x7ff1981f5140 2024-03-06T07:24:19.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.637+0000 7ff19efd7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff19815b370 con 0x7ff1981f4ce0 2024-03-06T07:24:19.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.638+0000 7ff19cd73640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1800039e0 0x7ff1980ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.638+0000 7ff19cd73640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1800039e0 0x7ff1980ccae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40676/0 (socket says 172.21.15.83:40676) 2024-03-06T07:24:19.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.638+0000 7ff19cd73640 1 -- 172.21.15.83:0/3367817653 learned_addr learned my addr 172.21.15.83:0/3367817653 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:19.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.638+0000 7ff197fff640 1 --2- 172.21.15.83:0/3367817653 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff1981f4ce0 0x7ff1981e4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.638+0000 7ff19d574640 1 --2- 172.21.15.83:0/3367817653 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1981f5140 0x7ff1981e5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.638+0000 7ff19cd73640 1 -- 172.21.15.83:0/3367817653 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff1981f4ce0 msgr2=0x7ff1981e4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.638+0000 7ff19cd73640 1 --2- 172.21.15.83:0/3367817653 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff1981f4ce0 0x7ff1981e4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.638+0000 7ff19cd73640 1 -- 172.21.15.83:0/3367817653 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1981f5140 msgr2=0x7ff1981e5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.638+0000 7ff19cd73640 1 --2- 172.21.15.83:0/3367817653 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1981f5140 0x7ff1981e5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.638+0000 7ff19cd73640 1 -- 172.21.15.83:0/3367817653 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1981e4610 con 0x7ff1800039e0 2024-03-06T07:24:19.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.638+0000 7ff19d574640 1 --2- 172.21.15.83:0/3367817653 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1981f5140 0x7ff1981e5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:19.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.638+0000 7ff19cd73640 1 --2- 172.21.15.83:0/3367817653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1800039e0 0x7ff1980ccae0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7ff18802bdf0 tx=0x7ff188001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:19.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.638+0000 7ff1957fa640 1 -- 172.21.15.83:0/3367817653 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1880035f0 con 0x7ff1800039e0 2024-03-06T07:24:19.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.638+0000 7ff1957fa640 1 -- 172.21.15.83:0/3367817653 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff188003790 con 0x7ff1800039e0 2024-03-06T07:24:19.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.638+0000 7ff19efd7640 1 -- 172.21.15.83:0/3367817653 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1980cfce0 con 0x7ff1800039e0 2024-03-06T07:24:19.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.638+0000 7ff1957fa640 1 -- 172.21.15.83:0/3367817653 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff188003b40 con 0x7ff1800039e0 2024-03-06T07:24:19.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.638+0000 7ff19efd7640 1 -- 172.21.15.83:0/3367817653 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff198152e00 con 0x7ff1800039e0 2024-03-06T07:24:19.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.639+0000 7ff19efd7640 1 -- 172.21.15.83:0/3367817653 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff19804f330 con 0x7ff1800039e0 2024-03-06T07:24:19.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.640+0000 7ff1957fa640 1 -- 172.21.15.83:0/3367817653 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff188003ce0 con 0x7ff1800039e0 2024-03-06T07:24:19.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.640+0000 7ff1957fa640 1 --2- 172.21.15.83:0/3367817653 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff18406b4d0 0x7ff18406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.640+0000 7ff1957fa640 1 -- 172.21.15.83:0/3367817653 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5609+0+0 (secure 0 0 0) 0x7ff1880ac1d0 con 0x7ff1800039e0 2024-03-06T07:24:19.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.640+0000 7ff197fff640 1 --2- 172.21.15.83:0/3367817653 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff18406b4d0 0x7ff18406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.640+0000 7ff197fff640 1 --2- 172.21.15.83:0/3367817653 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff18406b4d0 0x7ff18406d990 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff1981f49d0 tx=0x7ff18c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:19.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.645+0000 7ff1957fa640 1 -- 172.21.15.83:0/3367817653 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff18804a060 con 0x7ff1800039e0 2024-03-06T07:24:19.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:19.850+0000 7ff19efd7640 1 -- 172.21.15.83:0/3367817653 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7ff1980b2a30 con 0x7ff1800039e0 2024-03-06T07:24:20.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.528+0000 7ff1957fa640 1 -- 172.21.15.83:0/3367817653 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v23) v1 ==== 172+0+0 (secure 0 0 0) 0x7ff188041030 con 0x7ff1800039e0 2024-03-06T07:24:20.530 INFO:teuthology.orchestra.run.smithi083.stderr:new fs with metadata pool 8 and data pool 9 2024-03-06T07:24:20.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.537+0000 7ff19efd7640 1 -- 172.21.15.83:0/3367817653 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff18406b4d0 msgr2=0x7ff18406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.537+0000 7ff19efd7640 1 --2- 172.21.15.83:0/3367817653 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff18406b4d0 0x7ff18406d990 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff1981f49d0 tx=0x7ff18c008040 comp rx=0 tx=0).stop 2024-03-06T07:24:20.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.537+0000 7ff19efd7640 1 -- 172.21.15.83:0/3367817653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1800039e0 msgr2=0x7ff1980ccae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.537+0000 7ff19efd7640 1 --2- 172.21.15.83:0/3367817653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1800039e0 0x7ff1980ccae0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7ff18802bdf0 tx=0x7ff188001230 comp rx=0 tx=0).stop 2024-03-06T07:24:20.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.539+0000 7ff19efd7640 1 -- 172.21.15.83:0/3367817653 shutdown_connections 2024-03-06T07:24:20.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.539+0000 7ff19efd7640 1 --2- 172.21.15.83:0/3367817653 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff18406b4d0 0x7ff18406d990 secure :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff1981f49d0 tx=0x7ff18c008040 comp rx=0 tx=0).stop 2024-03-06T07:24:20.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.539+0000 7ff19efd7640 1 --2- 172.21.15.83:0/3367817653 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1981f5140 0x7ff1981e5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.539+0000 7ff19efd7640 1 --2- 172.21.15.83:0/3367817653 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff1981f4ce0 0x7ff1981e4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.539+0000 7ff19efd7640 1 --2- 172.21.15.83:0/3367817653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1800039e0 0x7ff1980ccae0 secure :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7ff18802bdf0 tx=0x7ff188001230 comp rx=0 tx=0).stop 2024-03-06T07:24:20.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.539+0000 7ff19efd7640 1 -- 172.21.15.83:0/3367817653 >> 172.21.15.83:0/3367817653 conn(0x7ff19806afb0 msgr2=0x7ff1980af6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:20.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.545+0000 7ff19efd7640 1 -- 172.21.15.83:0/3367817653 shutdown_connections 2024-03-06T07:24:20.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.546+0000 7ff19efd7640 1 -- 172.21.15.83:0/3367817653 wait complete. 2024-03-06T07:24:20.561 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:20.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.660+0000 7f8746b5e640 1 Processor -- start 2024-03-06T07:24:20.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.660+0000 7f8746b5e640 1 -- start start 2024-03-06T07:24:20.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.661+0000 7f8746b5e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f874013dc10 0x7f87400b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.661+0000 7f8746b5e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f874015ef60 0x7f87400b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.661+0000 7f8746b5e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f874015f580 con 0x7f874015bf30 2024-03-06T07:24:20.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.661+0000 7f8746b5e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87400cf3f0 con 0x7f874015ef60 2024-03-06T07:24:20.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.661+0000 7f8746b5e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f874015b210 con 0x7f874013dc10 2024-03-06T07:24:20.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.661+0000 7f87450fb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f874015ef60 0x7f87400b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.661+0000 7f87450fb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f874015ef60 0x7f87400b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40686/0 (socket says 172.21.15.83:40686) 2024-03-06T07:24:20.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.661+0000 7f87450fb640 1 -- 172.21.15.83:0/728850897 learned_addr learned my addr 172.21.15.83:0/728850897 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:20.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.661+0000 7f87448fa640 1 --2- 172.21.15.83:0/728850897 >> v2:172.21.15.183:3300/0 conn(0x7f874013dc10 0x7f87400b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.661+0000 7f87450fb640 1 -- 172.21.15.83:0/728850897 >> v2:172.21.15.183:3300/0 conn(0x7f874013dc10 msgr2=0x7f87400b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:20.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.661+0000 7f87450fb640 1 --2- 172.21.15.83:0/728850897 >> v2:172.21.15.183:3300/0 conn(0x7f874013dc10 0x7f87400b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.661+0000 7f87450fb640 1 -- 172.21.15.83:0/728850897 >> v1:172.21.15.183:6789/0 conn(0x7f874015bf30 legacy=0x7f874015e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.661+0000 7f87450fb640 1 -- 172.21.15.83:0/728850897 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f874015b450 con 0x7f874015ef60 2024-03-06T07:24:20.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.662+0000 7f87450fb640 1 --2- 172.21.15.83:0/728850897 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f874015ef60 0x7f87400b2e90 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f87400731a0 tx=0x7f873802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98f07588c1a97d06 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:20.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.662+0000 7f873f7fe640 1 -- 172.21.15.83:0/728850897 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873802ca80 con 0x7f874015ef60 2024-03-06T07:24:20.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.662+0000 7f873f7fe640 1 -- 172.21.15.83:0/728850897 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f873802cc20 con 0x7f874015ef60 2024-03-06T07:24:20.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.662+0000 7f873f7fe640 1 -- 172.21.15.83:0/728850897 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87380343f0 con 0x7f874015ef60 2024-03-06T07:24:20.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.663+0000 7f8746b5e640 1 -- 172.21.15.83:0/728850897 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f874015ef60 msgr2=0x7f87400b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.663+0000 7f8746b5e640 1 --2- 172.21.15.83:0/728850897 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f874015ef60 0x7f87400b2e90 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f87400731a0 tx=0x7f873802bee0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.663+0000 7f8746b5e640 1 -- 172.21.15.83:0/728850897 shutdown_connections 2024-03-06T07:24:20.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.663+0000 7f8746b5e640 1 --2- 172.21.15.83:0/728850897 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f874015ef60 0x7f87400b2e90 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.663+0000 7f8746b5e640 1 --2- 172.21.15.83:0/728850897 >> v2:172.21.15.183:3300/0 conn(0x7f874013dc10 0x7f87400b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.663+0000 7f8746b5e640 1 -- 172.21.15.83:0/728850897 >> 172.21.15.83:0/728850897 conn(0x7f87400cf8a0 msgr2=0x7f87400cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:20.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.663+0000 7f8746b5e640 1 -- 172.21.15.83:0/728850897 shutdown_connections 2024-03-06T07:24:20.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.663+0000 7f8746b5e640 1 -- 172.21.15.83:0/728850897 wait complete. 2024-03-06T07:24:20.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.664+0000 7f8746b5e640 1 Processor -- start 2024-03-06T07:24:20.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.664+0000 7f8746b5e640 1 -- start start 2024-03-06T07:24:20.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.665+0000 7f8746b5e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f874013dc10 0x7f8740151570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.665+0000 7f8746b5e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f874015bf30 0x7f8740140340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.665+0000 7f8746b5e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f874015ef60 0x7f8740140880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.665+0000 7f8746b5e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87400708d0 con 0x7f874015ef60 2024-03-06T07:24:20.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.665+0000 7f8746b5e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f874013dff0 con 0x7f874013dc10 2024-03-06T07:24:20.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.665+0000 7f8746b5e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8740070a50 con 0x7f874015bf30 2024-03-06T07:24:20.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.665+0000 7f87450fb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f874015ef60 0x7f8740140880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.665+0000 7f87450fb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f874015ef60 0x7f8740140880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40702/0 (socket says 172.21.15.83:40702) 2024-03-06T07:24:20.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.665+0000 7f87450fb640 1 -- 172.21.15.83:0/2847975733 learned_addr learned my addr 172.21.15.83:0/2847975733 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:20.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.665+0000 7f873ffff640 1 --2- 172.21.15.83:0/2847975733 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f874015bf30 0x7f8740140340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.665+0000 7f87448fa640 1 --2- 172.21.15.83:0/2847975733 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f874013dc10 0x7f8740151570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.665+0000 7f87450fb640 1 -- 172.21.15.83:0/2847975733 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f874015bf30 msgr2=0x7f8740140340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.665+0000 7f87450fb640 1 --2- 172.21.15.83:0/2847975733 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f874015bf30 0x7f8740140340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.665+0000 7f87450fb640 1 -- 172.21.15.83:0/2847975733 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f874013dc10 msgr2=0x7f8740151570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.665+0000 7f87450fb640 1 --2- 172.21.15.83:0/2847975733 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f874013dc10 0x7f8740151570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.665+0000 7f87450fb640 1 -- 172.21.15.83:0/2847975733 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8740141760 con 0x7f874015ef60 2024-03-06T07:24:20.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.665+0000 7f873ffff640 1 --2- 172.21.15.83:0/2847975733 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f874015bf30 0x7f8740140340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:20.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.666+0000 7f87450fb640 1 --2- 172.21.15.83:0/2847975733 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f874015ef60 0x7f8740140880 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f8738009f40 tx=0x7f8738004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:20.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.666+0000 7f873d7fa640 1 -- 172.21.15.83:0/2847975733 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8738043020 con 0x7f874015ef60 2024-03-06T07:24:20.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.666+0000 7f873d7fa640 1 -- 172.21.15.83:0/2847975733 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f873803e450 con 0x7f874015ef60 2024-03-06T07:24:20.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.666+0000 7f8746b5e640 1 -- 172.21.15.83:0/2847975733 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8740141a40 con 0x7f874015ef60 2024-03-06T07:24:20.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.666+0000 7f873d7fa640 1 -- 172.21.15.83:0/2847975733 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8738009510 con 0x7f874015ef60 2024-03-06T07:24:20.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.666+0000 7f8746b5e640 1 -- 172.21.15.83:0/2847975733 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8740151ab0 con 0x7f874015ef60 2024-03-06T07:24:20.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.667+0000 7f8746b5e640 1 -- 172.21.15.83:0/2847975733 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f874004f330 con 0x7f874015ef60 2024-03-06T07:24:20.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.667+0000 7f873d7fa640 1 -- 172.21.15.83:0/2847975733 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8738046020 con 0x7f874015ef60 2024-03-06T07:24:20.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.668+0000 7f873d7fa640 1 --2- 172.21.15.83:0/2847975733 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f872c06b4d0 0x7f872c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.668+0000 7f87448fa640 1 --2- 172.21.15.83:0/2847975733 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f872c06b4d0 0x7f872c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.668+0000 7f873d7fa640 1 -- 172.21.15.83:0/2847975733 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f87380ac9f0 con 0x7f874015ef60 2024-03-06T07:24:20.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.668+0000 7f87448fa640 1 --2- 172.21.15.83:0/2847975733 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f872c06b4d0 0x7f872c06d990 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f8730006fd0 tx=0x7f8730008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:20.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.672+0000 7f873d7fa640 1 -- 172.21.15.83:0/2847975733 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f873804d030 con 0x7f874015ef60 2024-03-06T07:24:20.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.846+0000 7f8746b5e640 1 -- 172.21.15.83:0/2847975733 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f874006eb00 con 0x7f874015ef60 2024-03-06T07:24:20.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.847+0000 7f873d7fa640 1 -- 172.21.15.83:0/2847975733 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+16690 (secure 0 0 0) 0x7f873807b900 con 0x7f874015ef60 2024-03-06T07:24:20.849 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:24:20.849 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":56,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:24:19.851842+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:18.074366+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:18.831101+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:20.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.849+0000 7f8746b5e640 1 -- 172.21.15.83:0/2847975733 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f872c06b4d0 msgr2=0x7f872c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.849+0000 7f8746b5e640 1 --2- 172.21.15.83:0/2847975733 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f872c06b4d0 0x7f872c06d990 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f8730006fd0 tx=0x7f8730008040 comp rx=0 tx=0).stop 2024-03-06T07:24:20.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.849+0000 7f8746b5e640 1 -- 172.21.15.83:0/2847975733 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f874015ef60 msgr2=0x7f8740140880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.849+0000 7f8746b5e640 1 --2- 172.21.15.83:0/2847975733 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f874015ef60 0x7f8740140880 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f8738009f40 tx=0x7f8738004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:20.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.850+0000 7f8746b5e640 1 -- 172.21.15.83:0/2847975733 shutdown_connections 2024-03-06T07:24:20.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.850+0000 7f8746b5e640 1 --2- 172.21.15.83:0/2847975733 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f872c06b4d0 0x7f872c06d990 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.850+0000 7f8746b5e640 1 --2- 172.21.15.83:0/2847975733 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f874015ef60 0x7f8740140880 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.850+0000 7f8746b5e640 1 --2- 172.21.15.83:0/2847975733 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f874015bf30 0x7f8740140340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.850+0000 7f8746b5e640 1 --2- 172.21.15.83:0/2847975733 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f874013dc10 0x7f8740151570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.850+0000 7f8746b5e640 1 -- 172.21.15.83:0/2847975733 >> 172.21.15.83:0/2847975733 conn(0x7f87400cf8a0 msgr2=0x7f87400afff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:20.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.850+0000 7f8746b5e640 1 -- 172.21.15.83:0/2847975733 shutdown_connections 2024-03-06T07:24:20.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.850+0000 7f8746b5e640 1 -- 172.21.15.83:0/2847975733 wait complete. 2024-03-06T07:24:20.862 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:20.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.960+0000 7f3ce9cc9640 1 Processor -- start 2024-03-06T07:24:20.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.961+0000 7f3ce9cc9640 1 -- start start 2024-03-06T07:24:20.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.961+0000 7f3ce9cc9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3ce415fa10 0x7f3ce40ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.961+0000 7f3ce9cc9640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f3ce40d16e0 0x7f3ce40d1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.961+0000 7f3ce9cc9640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ce40cebe0 con 0x7f3ce415bdd0 2024-03-06T07:24:20.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.961+0000 7f3ce9cc9640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ce40ceec0 con 0x7f3ce40d16e0 2024-03-06T07:24:20.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.961+0000 7f3ce9cc9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce40cf1a0 con 0x7f3ce415fa10 2024-03-06T07:24:20.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.961+0000 7f3ce37fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f3ce415bdd0 0x7f3ce40c5610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39320/0 (socket says 172.21.15.83:39320) 2024-03-06T07:24:20.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.961+0000 7f3ce37fe640 1 -- 172.21.15.83:0/2415063812 learned_addr learned my addr 172.21.15.83:0/2415063812 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:24:20.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.962+0000 7f3ce2ffd640 1 --2- 172.21.15.83:0/2415063812 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3ce415fa10 0x7f3ce40ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.962+0000 7f3ce3fff640 1 --2- 172.21.15.83:0/2415063812 >> v2:172.21.15.183:3300/0 conn(0x7f3ce40d16e0 0x7f3ce40d1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.962+0000 7f3ce27fc640 1 -- 172.21.15.83:0/2415063812 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 656195997 0 0) 0x7f3ce40cebe0 con 0x7f3ce415bdd0 2024-03-06T07:24:20.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.962+0000 7f3ce27fc640 1 -- 172.21.15.83:0/2415063812 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ccc003520 con 0x7f3ce415bdd0 2024-03-06T07:24:20.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.962+0000 7f3ce27fc640 1 -- 172.21.15.83:0/2415063812 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4258570145 0 0) 0x7f3ccc003520 con 0x7f3ce415bdd0 2024-03-06T07:24:20.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.962+0000 7f3ce27fc640 1 -- 172.21.15.83:0/2415063812 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3ce415fa10 msgr2=0x7f3ce40ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:20.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.962+0000 7f3ce27fc640 1 --2- 172.21.15.83:0/2415063812 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3ce415fa10 0x7f3ce40ce270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.962+0000 7f3ce27fc640 1 -- 172.21.15.83:0/2415063812 >> v2:172.21.15.183:3300/0 conn(0x7f3ce40d16e0 msgr2=0x7f3ce40d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:20.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.962+0000 7f3ce27fc640 1 --2- 172.21.15.83:0/2415063812 >> v2:172.21.15.183:3300/0 conn(0x7f3ce40d16e0 0x7f3ce40d1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.962+0000 7f3ce27fc640 1 -- 172.21.15.83:0/2415063812 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce40cc8e0 con 0x7f3ce415bdd0 2024-03-06T07:24:20.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.962+0000 7f3ce27fc640 1 -- 172.21.15.83:0/2415063812 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f3cd4001670 con 0x7f3ce415bdd0 2024-03-06T07:24:20.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.962+0000 7f3ce2ffd640 1 --2- 172.21.15.83:0/2415063812 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3ce415fa10 0x7f3ce40ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:20.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.962+0000 7f3ce27fc640 1 -- 172.21.15.83:0/2415063812 >> v1:172.21.15.83:6789/0 conn(0x7f3ce415bdd0 legacy=0x7f3ce40c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.962+0000 7f3ce27fc640 1 --2- 172.21.15.83:0/2415063812 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ccc0039b0 0x7f3ccc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.962+0000 7f3ce27fc640 1 --2- 172.21.15.83:0/2415063812 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3ce415fa10 0x7f3ccc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.962+0000 7f3ce27fc640 1 --2- 172.21.15.83:0/2415063812 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3ccc004d40 0x7f3ccc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.962+0000 7f3ce27fc640 1 -- 172.21.15.83:0/2415063812 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce40ceec0 con 0x7f3ccc0039b0 2024-03-06T07:24:20.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.962+0000 7f3ce27fc640 1 -- 172.21.15.83:0/2415063812 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce40cf1a0 con 0x7f3ccc004d40 2024-03-06T07:24:20.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.962+0000 7f3ce27fc640 1 -- 172.21.15.83:0/2415063812 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ccc00b9c0 con 0x7f3ce415fa10 2024-03-06T07:24:20.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.962+0000 7f3ce37fe640 1 --2- 172.21.15.83:0/2415063812 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ccc0039b0 0x7f3ccc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.962+0000 7f3ce2ffd640 1 --2- 172.21.15.83:0/2415063812 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3ce415fa10 0x7f3ccc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.962+0000 7f3ce3fff640 1 --2- 172.21.15.83:0/2415063812 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3ccc004d40 0x7f3ccc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.962+0000 7f3ce37fe640 1 -- 172.21.15.83:0/2415063812 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3ce415fa10 msgr2=0x7f3ccc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:20.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.962+0000 7f3ce37fe640 1 --2- 172.21.15.83:0/2415063812 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3ce415fa10 0x7f3ccc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.962+0000 7f3ce37fe640 1 -- 172.21.15.83:0/2415063812 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3ccc004d40 msgr2=0x7f3ccc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:20.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.962+0000 7f3ce37fe640 1 --2- 172.21.15.83:0/2415063812 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3ccc004d40 0x7f3ccc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.962+0000 7f3ce37fe640 1 -- 172.21.15.83:0/2415063812 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ccc003520 con 0x7f3ccc0039b0 2024-03-06T07:24:20.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.962+0000 7f3ce3fff640 1 --2- 172.21.15.83:0/2415063812 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3ccc004d40 0x7f3ccc0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:20.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.963+0000 7f3ce37fe640 1 --2- 172.21.15.83:0/2415063812 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ccc0039b0 0x7f3ccc003dd0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f3cd400c210 tx=0x7f3ccc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfe221e4ceb960ec server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:20.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.963+0000 7f3ce27fc640 1 -- 172.21.15.83:0/2415063812 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd4007d50 con 0x7f3ccc0039b0 2024-03-06T07:24:20.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.963+0000 7f3ce27fc640 1 -- 172.21.15.83:0/2415063812 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cd4030070 con 0x7f3ccc0039b0 2024-03-06T07:24:20.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.963+0000 7f3ce27fc640 1 -- 172.21.15.83:0/2415063812 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd402cd80 con 0x7f3ccc0039b0 2024-03-06T07:24:20.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.964+0000 7f3ce9cc9640 1 -- 172.21.15.83:0/2415063812 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ccc0039b0 msgr2=0x7f3ccc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.964+0000 7f3ce9cc9640 1 --2- 172.21.15.83:0/2415063812 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ccc0039b0 0x7f3ccc003dd0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f3cd400c210 tx=0x7f3ccc005830 comp rx=0 tx=0).stop 2024-03-06T07:24:20.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.964+0000 7f3ce3fff640 1 -- 172.21.15.83:0/2415063812 reap_dead start 2024-03-06T07:24:20.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.964+0000 7f3ce9cc9640 1 -- 172.21.15.83:0/2415063812 shutdown_connections 2024-03-06T07:24:20.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.964+0000 7f3ce9cc9640 1 -- 172.21.15.83:0/2415063812 >> 172.21.15.83:0/2415063812 conn(0x7f3ce409a9f0 msgr2=0x7f3ce409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:20.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.964+0000 7f3ce9cc9640 1 -- 172.21.15.83:0/2415063812 shutdown_connections 2024-03-06T07:24:20.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.964+0000 7f3ce9cc9640 1 -- 172.21.15.83:0/2415063812 wait complete. 2024-03-06T07:24:20.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.965+0000 7f3ce9cc9640 1 Processor -- start 2024-03-06T07:24:20.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.965+0000 7f3ce9cc9640 1 -- start start 2024-03-06T07:24:20.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.966+0000 7f3ce9cc9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ccc0039b0 0x7f3ce409ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.966+0000 7f3ce9cc9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3ce40cb810 0x7f3ce4150e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.966+0000 7f3ce9cc9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3ce40c98f0 0x7f3ce40c9d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.966+0000 7f3ce9cc9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce415c1b0 con 0x7f3ccc0039b0 2024-03-06T07:24:20.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.966+0000 7f3ce9cc9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce40cc5d0 con 0x7f3ce40cb810 2024-03-06T07:24:20.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.966+0000 7f3ce9cc9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce40ce270 con 0x7f3ce40c98f0 2024-03-06T07:24:20.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.966+0000 7f3ce37fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ccc0039b0 0x7f3ce409ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.966+0000 7f3ce37fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ccc0039b0 0x7f3ce409ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40716/0 (socket says 172.21.15.83:40716) 2024-03-06T07:24:20.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.966+0000 7f3ce37fe640 1 -- 172.21.15.83:0/3266342359 learned_addr learned my addr 172.21.15.83:0/3266342359 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:20.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.966+0000 7f3ce37fe640 1 -- 172.21.15.83:0/3266342359 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3ce40c98f0 msgr2=0x7f3ce40c9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.966+0000 7f3ce3fff640 1 --2- 172.21.15.83:0/3266342359 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3ce40c98f0 0x7f3ce40c9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.966+0000 7f3ce2ffd640 1 --2- 172.21.15.83:0/3266342359 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3ce40cb810 0x7f3ce4150e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.966+0000 7f3ce37fe640 1 --2- 172.21.15.83:0/3266342359 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3ce40c98f0 0x7f3ce40c9d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.966+0000 7f3ce37fe640 1 -- 172.21.15.83:0/3266342359 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3ce40cb810 msgr2=0x7f3ce4150e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.966+0000 7f3ce37fe640 1 --2- 172.21.15.83:0/3266342359 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3ce40cb810 0x7f3ce4150e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.966+0000 7f3ce37fe640 1 -- 172.21.15.83:0/3266342359 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce40c9100 con 0x7f3ccc0039b0 2024-03-06T07:24:20.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.966+0000 7f3ce3fff640 1 --2- 172.21.15.83:0/3266342359 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3ce40c98f0 0x7f3ce40c9d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:20.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.966+0000 7f3ce2ffd640 1 --2- 172.21.15.83:0/3266342359 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3ce40cb810 0x7f3ce4150e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:20.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.966+0000 7f3ce37fe640 1 --2- 172.21.15.83:0/3266342359 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ccc0039b0 0x7f3ce409ba00 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f3cd400c330 tx=0x7f3cd4001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:20.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.967+0000 7f3ccbfff640 1 -- 172.21.15.83:0/3266342359 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd4045070 con 0x7f3ccc0039b0 2024-03-06T07:24:20.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.967+0000 7f3ccbfff640 1 -- 172.21.15.83:0/3266342359 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cd4004bf0 con 0x7f3ccc0039b0 2024-03-06T07:24:20.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.967+0000 7f3ce9cc9640 1 -- 172.21.15.83:0/3266342359 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ce40cbc90 con 0x7f3ccc0039b0 2024-03-06T07:24:20.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.967+0000 7f3ccbfff640 1 -- 172.21.15.83:0/3266342359 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd4001c90 con 0x7f3ccc0039b0 2024-03-06T07:24:20.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.967+0000 7f3ce9cc9640 1 -- 172.21.15.83:0/3266342359 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ce40c93a0 con 0x7f3ccc0039b0 2024-03-06T07:24:20.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.968+0000 7f3ce9cc9640 1 -- 172.21.15.83:0/3266342359 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ce404f330 con 0x7f3ccc0039b0 2024-03-06T07:24:20.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.968+0000 7f3ccbfff640 1 -- 172.21.15.83:0/3266342359 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3cd4007070 con 0x7f3ccc0039b0 2024-03-06T07:24:20.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.968+0000 7f3ccbfff640 1 --2- 172.21.15.83:0/3266342359 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3cd006b4d0 0x7f3cd006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.969+0000 7f3ccbfff640 1 -- 172.21.15.83:0/3266342359 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3cd40ab2b0 con 0x7f3ccc0039b0 2024-03-06T07:24:20.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.969+0000 7f3ce2ffd640 1 --2- 172.21.15.83:0/3266342359 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3cd006b4d0 0x7f3cd006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.969+0000 7f3ce2ffd640 1 --2- 172.21.15.83:0/3266342359 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3cd006b4d0 0x7f3cd006d990 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3ce40cadf0 tx=0x7f3cd8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:20.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:20.973+0000 7f3ccbfff640 1 -- 172.21.15.83:0/3266342359 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3cd4079bc0 con 0x7f3ccc0039b0 2024-03-06T07:24:21.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.144+0000 7f3ce9cc9640 1 -- 172.21.15.83:0/3266342359 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3ce40cbed0 con 0x7f3ccc0039b0 2024-03-06T07:24:21.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.145+0000 7f3ccbfff640 1 -- 172.21.15.83:0/3266342359 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+16690 (secure 0 0 0) 0x7f3cd407dc60 con 0x7f3ccc0039b0 2024-03-06T07:24:21.147 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:24:21.147 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":56,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:24:19.851842+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:18.074366+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:18.831101+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:21.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.147+0000 7f3ce9cc9640 1 -- 172.21.15.83:0/3266342359 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3cd006b4d0 msgr2=0x7f3cd006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.147+0000 7f3ce9cc9640 1 --2- 172.21.15.83:0/3266342359 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3cd006b4d0 0x7f3cd006d990 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3ce40cadf0 tx=0x7f3cd8008040 comp rx=0 tx=0).stop 2024-03-06T07:24:21.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.147+0000 7f3ce9cc9640 1 -- 172.21.15.83:0/3266342359 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ccc0039b0 msgr2=0x7f3ce409ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.147+0000 7f3ce9cc9640 1 --2- 172.21.15.83:0/3266342359 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ccc0039b0 0x7f3ce409ba00 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f3cd400c330 tx=0x7f3cd4001a40 comp rx=0 tx=0).stop 2024-03-06T07:24:21.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.147+0000 7f3ce9cc9640 1 -- 172.21.15.83:0/3266342359 shutdown_connections 2024-03-06T07:24:21.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.147+0000 7f3ce9cc9640 1 --2- 172.21.15.83:0/3266342359 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3cd006b4d0 0x7f3cd006d990 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.147+0000 7f3ce9cc9640 1 --2- 172.21.15.83:0/3266342359 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3ce40c98f0 0x7f3ce40c9d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.147+0000 7f3ce9cc9640 1 --2- 172.21.15.83:0/3266342359 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3ce40cb810 0x7f3ce4150e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.147+0000 7f3ce9cc9640 1 --2- 172.21.15.83:0/3266342359 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ccc0039b0 0x7f3ce409ba00 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.147+0000 7f3ce9cc9640 1 -- 172.21.15.83:0/3266342359 >> 172.21.15.83:0/3266342359 conn(0x7f3ce409a9f0 msgr2=0x7f3ce40b66e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:21.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.147+0000 7f3ce9cc9640 1 -- 172.21.15.83:0/3266342359 shutdown_connections 2024-03-06T07:24:21.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.147+0000 7f3ce9cc9640 1 -- 172.21.15.83:0/3266342359 wait complete. 2024-03-06T07:24:21.160 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:24:21.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.261+0000 7fd44789d640 1 Processor -- start 2024-03-06T07:24:21.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.261+0000 7fd44789d640 1 -- start start 2024-03-06T07:24:21.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.262+0000 7fd44789d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd440146f60 0x7fd440147890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.262+0000 7fd44789d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4400712b0 con 0x7fd440146b80 2024-03-06T07:24:21.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.262+0000 7fd44789d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4400b5080 con 0x7fd440070e20 2024-03-06T07:24:21.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.262+0000 7fd44789d640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4400ad900 con 0x7fd440146f60 2024-03-06T07:24:21.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.262+0000 7fd444e38640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd440146f60 0x7fd440147890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.262+0000 7fd445639640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fd440146b80 0x7fd4400b4870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39330/0 (socket says 172.21.15.83:39330) 2024-03-06T07:24:21.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.262+0000 7fd445639640 1 -- 172.21.15.83:0/1595404442 learned_addr learned my addr 172.21.15.83:0/1595404442 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:24:21.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.262+0000 7fd444e38640 1 -- 172.21.15.83:0/1595404442 >> v1:172.21.15.183:6789/0 conn(0x7fd440070e20 legacy=0x7fd44006e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:21.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.262+0000 7fd444e38640 1 -- 172.21.15.83:0/1595404442 >> v1:172.21.15.83:6789/0 conn(0x7fd440146b80 legacy=0x7fd4400b4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.262+0000 7fd444e38640 1 -- 172.21.15.83:0/1595404442 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4400adbe0 con 0x7fd440146f60 2024-03-06T07:24:21.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.263+0000 7fd444e38640 1 --2- 172.21.15.83:0/1595404442 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd440146f60 0x7fd440147890 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fd4300097a0 tx=0x7fd43002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef71659e3dcb016a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:21.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.263+0000 7fd437fff640 1 -- 172.21.15.83:0/1595404442 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd430035020 con 0x7fd440146f60 2024-03-06T07:24:21.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.263+0000 7fd437fff640 1 -- 172.21.15.83:0/1595404442 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd430031830 con 0x7fd440146f60 2024-03-06T07:24:21.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.263+0000 7fd437fff640 1 -- 172.21.15.83:0/1595404442 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd430031c00 con 0x7fd440146f60 2024-03-06T07:24:21.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.263+0000 7fd44789d640 1 -- 172.21.15.83:0/1595404442 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd440146f60 msgr2=0x7fd440147890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.263+0000 7fd44789d640 1 --2- 172.21.15.83:0/1595404442 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd440146f60 0x7fd440147890 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fd4300097a0 tx=0x7fd43002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.264+0000 7fd44789d640 1 -- 172.21.15.83:0/1595404442 shutdown_connections 2024-03-06T07:24:21.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.264+0000 7fd44789d640 1 --2- 172.21.15.83:0/1595404442 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd440146f60 0x7fd440147890 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.264+0000 7fd44789d640 1 -- 172.21.15.83:0/1595404442 >> 172.21.15.83:0/1595404442 conn(0x7fd44006b830 msgr2=0x7fd4400b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:21.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.264+0000 7fd44789d640 1 -- 172.21.15.83:0/1595404442 shutdown_connections 2024-03-06T07:24:21.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.264+0000 7fd44789d640 1 -- 172.21.15.83:0/1595404442 wait complete. 2024-03-06T07:24:21.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.265+0000 7fd44789d640 1 Processor -- start 2024-03-06T07:24:21.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.265+0000 7fd44789d640 1 -- start start 2024-03-06T07:24:21.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.265+0000 7fd44789d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd440070e20 0x7fd4401e6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.265+0000 7fd44789d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd440146b80 0x7fd4401f3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.265+0000 7fd44789d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd4401e8240 0x7fd440145840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.265+0000 7fd44789d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd44006e9d0 con 0x7fd440146b80 2024-03-06T07:24:21.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.265+0000 7fd44789d640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4400b5360 con 0x7fd440070e20 2024-03-06T07:24:21.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.265+0000 7fd44789d640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd44006eb50 con 0x7fd4401e8240 2024-03-06T07:24:21.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.266+0000 7fd444e38640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd440146b80 0x7fd4401f3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.266+0000 7fd444e38640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd440146b80 0x7fd4401f3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40732/0 (socket says 172.21.15.83:40732) 2024-03-06T07:24:21.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.266+0000 7fd444e38640 1 -- 172.21.15.83:0/4069836428 learned_addr learned my addr 172.21.15.83:0/4069836428 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:21.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.266+0000 7fd445e3a640 1 --2- 172.21.15.83:0/4069836428 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd4401e8240 0x7fd440145840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.266+0000 7fd445639640 1 --2- 172.21.15.83:0/4069836428 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd440070e20 0x7fd4401e6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.266+0000 7fd444e38640 1 -- 172.21.15.83:0/4069836428 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd4401e8240 msgr2=0x7fd440145840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.266+0000 7fd444e38640 1 --2- 172.21.15.83:0/4069836428 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd4401e8240 0x7fd440145840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.266+0000 7fd444e38640 1 -- 172.21.15.83:0/4069836428 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd440070e20 msgr2=0x7fd4401e6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.266+0000 7fd444e38640 1 --2- 172.21.15.83:0/4069836428 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd440070e20 0x7fd4401e6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.266+0000 7fd444e38640 1 -- 172.21.15.83:0/4069836428 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4401e6cb0 con 0x7fd440146b80 2024-03-06T07:24:21.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.266+0000 7fd445639640 1 --2- 172.21.15.83:0/4069836428 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd440070e20 0x7fd4401e6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:21.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.266+0000 7fd444e38640 1 --2- 172.21.15.83:0/4069836428 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd440146b80 0x7fd4401f3c40 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fd430000c00 tx=0x7fd4300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:21.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.266+0000 7fd435ffb640 1 -- 172.21.15.83:0/4069836428 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd43003aa70 con 0x7fd440146b80 2024-03-06T07:24:21.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.266+0000 7fd435ffb640 1 -- 172.21.15.83:0/4069836428 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd43003ac10 con 0x7fd440146b80 2024-03-06T07:24:21.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.266+0000 7fd44789d640 1 -- 172.21.15.83:0/4069836428 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4401e6f50 con 0x7fd440146b80 2024-03-06T07:24:21.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.266+0000 7fd435ffb640 1 -- 172.21.15.83:0/4069836428 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4300443f0 con 0x7fd440146b80 2024-03-06T07:24:21.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.266+0000 7fd44789d640 1 -- 172.21.15.83:0/4069836428 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4401e7510 con 0x7fd440146b80 2024-03-06T07:24:21.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.267+0000 7fd44789d640 1 -- 172.21.15.83:0/4069836428 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd44006e0b0 con 0x7fd440146b80 2024-03-06T07:24:21.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.268+0000 7fd435ffb640 1 -- 172.21.15.83:0/4069836428 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd43004b060 con 0x7fd440146b80 2024-03-06T07:24:21.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.268+0000 7fd435ffb640 1 --2- 172.21.15.83:0/4069836428 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd42406b4d0 0x7fd42406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.268+0000 7fd445639640 1 --2- 172.21.15.83:0/4069836428 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd42406b4d0 0x7fd42406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.268+0000 7fd435ffb640 1 -- 172.21.15.83:0/4069836428 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd4300abfa0 con 0x7fd440146b80 2024-03-06T07:24:21.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.269+0000 7fd445639640 1 --2- 172.21.15.83:0/4069836428 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd42406b4d0 0x7fd42406d990 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd43c000f60 tx=0x7fd43c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:21.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.273+0000 7fd435ffb640 1 -- 172.21.15.83:0/4069836428 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd43003f070 con 0x7fd440146b80 2024-03-06T07:24:21.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.482+0000 7fd44789d640 1 -- 172.21.15.83:0/4069836428 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fd4400af1c0 con 0x7fd440146b80 2024-03-06T07:24:21.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.531+0000 7fd435ffb640 1 -- 172.21.15.83:0/4069836428 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v24) v1 ==== 118+0+0 (secure 0 0 0) 0x7fd43007ad90 con 0x7fd440146b80 2024-03-06T07:24:21.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.533+0000 7fd44789d640 1 -- 172.21.15.83:0/4069836428 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd42406b4d0 msgr2=0x7fd42406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.533+0000 7fd44789d640 1 --2- 172.21.15.83:0/4069836428 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd42406b4d0 0x7fd42406d990 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd43c000f60 tx=0x7fd43c009040 comp rx=0 tx=0).stop 2024-03-06T07:24:21.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.533+0000 7fd44789d640 1 -- 172.21.15.83:0/4069836428 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd440146b80 msgr2=0x7fd4401f3c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.533+0000 7fd44789d640 1 --2- 172.21.15.83:0/4069836428 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd440146b80 0x7fd4401f3c40 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fd430000c00 tx=0x7fd4300046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.533+0000 7fd44789d640 1 -- 172.21.15.83:0/4069836428 shutdown_connections 2024-03-06T07:24:21.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.533+0000 7fd44789d640 1 --2- 172.21.15.83:0/4069836428 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd42406b4d0 0x7fd42406d990 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.533+0000 7fd44789d640 1 --2- 172.21.15.83:0/4069836428 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd4401e8240 0x7fd440145840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.533+0000 7fd44789d640 1 --2- 172.21.15.83:0/4069836428 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd440146b80 0x7fd4401f3c40 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.533+0000 7fd44789d640 1 --2- 172.21.15.83:0/4069836428 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd440070e20 0x7fd4401e6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.533+0000 7fd44789d640 1 -- 172.21.15.83:0/4069836428 >> 172.21.15.83:0/4069836428 conn(0x7fd44006b830 msgr2=0x7fd440144ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:21.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.533+0000 7fd44789d640 1 -- 172.21.15.83:0/4069836428 shutdown_connections 2024-03-06T07:24:21.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.533+0000 7fd44789d640 1 -- 172.21.15.83:0/4069836428 wait complete. 2024-03-06T07:24:21.546 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:24:21.547 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:24:21.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.650+0000 7f73b4a81640 1 Processor -- start 2024-03-06T07:24:21.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.650+0000 7f73b4a81640 1 -- start start 2024-03-06T07:24:21.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.650+0000 7f73b4a81640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f73b01614d0 0x7f73b00d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.650+0000 7f73b4a81640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f73b00d24b0 0x7f73b00b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.650+0000 7f73b4a81640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73b01618b0 con 0x7f73b00d1910 2024-03-06T07:24:21.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.650+0000 7f73b4a81640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73b00cf400 con 0x7f73b01614d0 2024-03-06T07:24:21.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.650+0000 7f73b4a81640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73b00cf6a0 con 0x7f73b00d24b0 2024-03-06T07:24:21.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.651+0000 7f73ae59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f73b01614d0 0x7f73b00d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.651+0000 7f73a5d9b640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f73b00d1910 0x7f73b00d1da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39332/0 (socket says 172.21.15.83:39332) 2024-03-06T07:24:21.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.651+0000 7f73a5d9b640 1 -- 172.21.15.83:0/907384807 learned_addr learned my addr 172.21.15.83:0/907384807 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:24:21.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.651+0000 7f73aed9d640 1 --2- 172.21.15.83:0/907384807 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f73b00d24b0 0x7f73b00b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.651+0000 7f73ae59c640 1 -- 172.21.15.83:0/907384807 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f73b00d24b0 msgr2=0x7f73b00b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:21.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.651+0000 7f73ae59c640 1 --2- 172.21.15.83:0/907384807 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f73b00d24b0 0x7f73b00b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.651+0000 7f73ae59c640 1 -- 172.21.15.83:0/907384807 >> v1:172.21.15.83:6789/0 conn(0x7f73b00d1910 legacy=0x7f73b00d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.651+0000 7f73ae59c640 1 -- 172.21.15.83:0/907384807 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b00cf980 con 0x7f73b01614d0 2024-03-06T07:24:21.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.651+0000 7f73add9b640 1 -- 172.21.15.83:0/907384807 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 849540706 0 0) 0x7f73b01618b0 con 0x7f73b00d1910 2024-03-06T07:24:21.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.652+0000 7f73ae59c640 1 --2- 172.21.15.83:0/907384807 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73b01614d0 0x7f73b00d4860 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f73a00097a0 tx=0x7f73a002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=767752d3cde22f0b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:21.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.652+0000 7f73add9b640 1 -- 172.21.15.83:0/907384807 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a0035020 con 0x7f73b01614d0 2024-03-06T07:24:21.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.652+0000 7f73add9b640 1 -- 172.21.15.83:0/907384807 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73a0031830 con 0x7f73b01614d0 2024-03-06T07:24:21.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.652+0000 7f73add9b640 1 -- 172.21.15.83:0/907384807 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a0031c00 con 0x7f73b01614d0 2024-03-06T07:24:21.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.653+0000 7f73b4a81640 1 -- 172.21.15.83:0/907384807 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73b01614d0 msgr2=0x7f73b00d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.653+0000 7f73b4a81640 1 --2- 172.21.15.83:0/907384807 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73b01614d0 0x7f73b00d4860 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f73a00097a0 tx=0x7f73a002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.653+0000 7f73b4a81640 1 -- 172.21.15.83:0/907384807 shutdown_connections 2024-03-06T07:24:21.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.653+0000 7f73b4a81640 1 --2- 172.21.15.83:0/907384807 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f73b00d24b0 0x7f73b00b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.653+0000 7f73b4a81640 1 --2- 172.21.15.83:0/907384807 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73b01614d0 0x7f73b00d4860 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.653+0000 7f73b4a81640 1 -- 172.21.15.83:0/907384807 >> 172.21.15.83:0/907384807 conn(0x7f73b00d3d40 msgr2=0x7f73b00d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:21.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.653+0000 7f73b4a81640 1 -- 172.21.15.83:0/907384807 shutdown_connections 2024-03-06T07:24:21.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.653+0000 7f73b4a81640 1 -- 172.21.15.83:0/907384807 wait complete. 2024-03-06T07:24:21.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.654+0000 7f73b4a81640 1 Processor -- start 2024-03-06T07:24:21.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.654+0000 7f73b4a81640 1 -- start start 2024-03-06T07:24:21.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.655+0000 7f73b4a81640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f73b00d1910 0x7f73b0157310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.655+0000 7f73b4a81640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73b00d24b0 0x7f73b00b7e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.655+0000 7f73b4a81640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73b01614d0 0x7f73b006c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.655+0000 7f73b4a81640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b00b1370 con 0x7f73b01614d0 2024-03-06T07:24:21.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.655+0000 7f73b4a81640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b00b0910 con 0x7f73b00d24b0 2024-03-06T07:24:21.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.655+0000 7f73b4a81640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73b00b14f0 con 0x7f73b00d1910 2024-03-06T07:24:21.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.655+0000 7f73aed9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73b01614d0 0x7f73b006c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.655+0000 7f73aed9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73b01614d0 0x7f73b006c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40758/0 (socket says 172.21.15.83:40758) 2024-03-06T07:24:21.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.655+0000 7f73aed9d640 1 -- 172.21.15.83:0/2503522891 learned_addr learned my addr 172.21.15.83:0/2503522891 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:21.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.655+0000 7f73ae59c640 1 --2- 172.21.15.83:0/2503522891 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f73b00d1910 0x7f73b0157310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.655+0000 7f73a5d9b640 1 --2- 172.21.15.83:0/2503522891 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73b00d24b0 0x7f73b00b7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.655+0000 7f73aed9d640 1 -- 172.21.15.83:0/2503522891 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f73b00d1910 msgr2=0x7f73b0157310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.655+0000 7f73aed9d640 1 --2- 172.21.15.83:0/2503522891 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f73b00d1910 0x7f73b0157310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.655+0000 7f73aed9d640 1 -- 172.21.15.83:0/2503522891 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73b00d24b0 msgr2=0x7f73b00b7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.655+0000 7f73aed9d640 1 --2- 172.21.15.83:0/2503522891 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73b00d24b0 0x7f73b00b7e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.655+0000 7f73aed9d640 1 -- 172.21.15.83:0/2503522891 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b00cd310 con 0x7f73b01614d0 2024-03-06T07:24:21.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.655+0000 7f73aed9d640 1 --2- 172.21.15.83:0/2503522891 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73b01614d0 0x7f73b006c6d0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f73a8007e40 tx=0x7f73a800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:21.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.656+0000 7f73a77fe640 1 -- 172.21.15.83:0/2503522891 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a8014620 con 0x7f73b01614d0 2024-03-06T07:24:21.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.656+0000 7f73a77fe640 1 -- 172.21.15.83:0/2503522891 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73a8019070 con 0x7f73b01614d0 2024-03-06T07:24:21.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.656+0000 7f73b4a81640 1 -- 172.21.15.83:0/2503522891 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73b01f7ba0 con 0x7f73b01614d0 2024-03-06T07:24:21.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.656+0000 7f73a77fe640 1 -- 172.21.15.83:0/2503522891 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a8014a20 con 0x7f73b01614d0 2024-03-06T07:24:21.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.656+0000 7f73b4a81640 1 -- 172.21.15.83:0/2503522891 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73b00cb220 con 0x7f73b01614d0 2024-03-06T07:24:21.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.657+0000 7f73a77fe640 1 -- 172.21.15.83:0/2503522891 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f73a8014bc0 con 0x7f73b01614d0 2024-03-06T07:24:21.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.657+0000 7f73b4a81640 1 -- 172.21.15.83:0/2503522891 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73b0068ff0 con 0x7f73b01614d0 2024-03-06T07:24:21.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.658+0000 7f73a77fe640 1 --2- 172.21.15.83:0/2503522891 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f739006b4d0 0x7f739006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.658+0000 7f73ae59c640 1 --2- 172.21.15.83:0/2503522891 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f739006b4d0 0x7f739006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.658+0000 7f73a77fe640 1 -- 172.21.15.83:0/2503522891 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f73a808aa00 con 0x7f73b01614d0 2024-03-06T07:24:21.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.658+0000 7f73ae59c640 1 --2- 172.21.15.83:0/2503522891 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f739006b4d0 0x7f739006d990 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f73a0036000 tx=0x7f73a003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:21.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.662+0000 7f73a77fe640 1 -- 172.21.15.83:0/2503522891 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f73b00cd310 con 0x7f73b01614d0 2024-03-06T07:24:21.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:21.878+0000 7f73b4a81640 1 -- 172.21.15.83:0/2503522891 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f73b00cb6b0 con 0x7f73b01614d0 2024-03-06T07:24:22.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.537+0000 7f73a77fe640 1 -- 172.21.15.83:0/2503522891 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v25) v1 ==== 122+0+0 (secure 0 0 0) 0x7f73a8059910 con 0x7f73b01614d0 2024-03-06T07:24:22.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.539+0000 7f73b4a81640 1 -- 172.21.15.83:0/2503522891 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f739006b4d0 msgr2=0x7f739006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.539+0000 7f73b4a81640 1 --2- 172.21.15.83:0/2503522891 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f739006b4d0 0x7f739006d990 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f73a0036000 tx=0x7f73a003c040 comp rx=0 tx=0).stop 2024-03-06T07:24:22.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.539+0000 7f73b4a81640 1 -- 172.21.15.83:0/2503522891 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73b01614d0 msgr2=0x7f73b006c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.539+0000 7f73b4a81640 1 --2- 172.21.15.83:0/2503522891 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73b01614d0 0x7f73b006c6d0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f73a8007e40 tx=0x7f73a800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.539+0000 7f73b4a81640 1 -- 172.21.15.83:0/2503522891 shutdown_connections 2024-03-06T07:24:22.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.539+0000 7f73b4a81640 1 --2- 172.21.15.83:0/2503522891 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f739006b4d0 0x7f739006d990 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.539+0000 7f73b4a81640 1 --2- 172.21.15.83:0/2503522891 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73b01614d0 0x7f73b006c6d0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.539+0000 7f73b4a81640 1 --2- 172.21.15.83:0/2503522891 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73b00d24b0 0x7f73b00b7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.539+0000 7f73b4a81640 1 --2- 172.21.15.83:0/2503522891 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f73b00d1910 0x7f73b0157310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.539+0000 7f73b4a81640 1 -- 172.21.15.83:0/2503522891 >> 172.21.15.83:0/2503522891 conn(0x7f73b00d3d40 msgr2=0x7f73b00cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:22.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.539+0000 7f73b4a81640 1 -- 172.21.15.83:0/2503522891 shutdown_connections 2024-03-06T07:24:22.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.539+0000 7f73b4a81640 1 -- 172.21.15.83:0/2503522891 wait complete. 2024-03-06T07:24:22.552 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:24:22.552 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:22.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.651+0000 7f4b3355e640 1 Processor -- start 2024-03-06T07:24:22.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.652+0000 7f4b3355e640 1 -- start start 2024-03-06T07:24:22.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.652+0000 7f4b3355e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4b2c0d22b0 0x7f4b2c0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.652+0000 7f4b3355e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b2c0b3430 con 0x7f4b2c0b41f0 2024-03-06T07:24:22.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.652+0000 7f4b3355e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b2c0b3710 con 0x7f4b2c15dfe0 2024-03-06T07:24:22.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.652+0000 7f4b3355e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b2c0d2c50 con 0x7f4b2c0d22b0 2024-03-06T07:24:22.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.652+0000 7f4b30af9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4b2c0d22b0 0x7f4b2c0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.653+0000 7f4b31afb640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f4b2c0b41f0 0x7f4b2c0b4680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39342/0 (socket says 172.21.15.83:39342) 2024-03-06T07:24:22.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.653+0000 7f4b31afb640 1 -- 172.21.15.83:0/1955992653 learned_addr learned my addr 172.21.15.83:0/1955992653 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:24:22.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.653+0000 7f4b30af9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4b2c0d22b0 0x7f4b2c0d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40766/0 (socket says 172.21.15.83:40766) 2024-03-06T07:24:22.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.653+0000 7f4b30af9640 1 -- 172.21.15.83:0/1955992653 >> v1:172.21.15.183:6789/0 conn(0x7f4b2c15dfe0 legacy=0x7f4b2c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:22.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.653+0000 7f4b30af9640 1 -- 172.21.15.83:0/1955992653 >> v1:172.21.15.83:6789/0 conn(0x7f4b2c0b41f0 legacy=0x7f4b2c0b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.653+0000 7f4b23fff640 1 -- 172.21.15.83:0/1955992653 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2426917876 0 0) 0x7f4b2c0b3430 con 0x7f4b2c0b41f0 2024-03-06T07:24:22.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.653+0000 7f4b30af9640 1 -- 172.21.15.83:0/1955992653 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b2c0b39f0 con 0x7f4b2c0d22b0 2024-03-06T07:24:22.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.654+0000 7f4b30af9640 1 --2- 172.21.15.83:0/1955992653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b2c0d22b0 0x7f4b2c0d2710 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c0097d0 tx=0x7f4b1c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e246db88aad60932 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:22.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.654+0000 7f4b23fff640 1 -- 172.21.15.83:0/1955992653 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b1c034070 con 0x7f4b2c0d22b0 2024-03-06T07:24:22.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.654+0000 7f4b23fff640 1 -- 172.21.15.83:0/1955992653 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b1c030610 con 0x7f4b2c0d22b0 2024-03-06T07:24:22.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.654+0000 7f4b23fff640 1 -- 172.21.15.83:0/1955992653 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b1c0309e0 con 0x7f4b2c0d22b0 2024-03-06T07:24:22.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.654+0000 7f4b3355e640 1 -- 172.21.15.83:0/1955992653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b2c0d22b0 msgr2=0x7f4b2c0d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.654+0000 7f4b3355e640 1 --2- 172.21.15.83:0/1955992653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b2c0d22b0 0x7f4b2c0d2710 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c0097d0 tx=0x7f4b1c02b960 comp rx=0 tx=0).stop 2024-03-06T07:24:22.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.654+0000 7f4b3355e640 1 -- 172.21.15.83:0/1955992653 shutdown_connections 2024-03-06T07:24:22.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.654+0000 7f4b3355e640 1 --2- 172.21.15.83:0/1955992653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b2c0d22b0 0x7f4b2c0d2710 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.654+0000 7f4b3355e640 1 -- 172.21.15.83:0/1955992653 >> 172.21.15.83:0/1955992653 conn(0x7f4b2c0d3cd0 msgr2=0x7f4b2c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:22.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.655+0000 7f4b3355e640 1 -- 172.21.15.83:0/1955992653 shutdown_connections 2024-03-06T07:24:22.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.655+0000 7f4b3355e640 1 -- 172.21.15.83:0/1955992653 wait complete. 2024-03-06T07:24:22.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.656+0000 7f4b3355e640 1 Processor -- start 2024-03-06T07:24:22.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.656+0000 7f4b3355e640 1 -- start start 2024-03-06T07:24:22.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.656+0000 7f4b3355e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b2c0b41f0 0x7f4b2c0b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.656+0000 7f4b3355e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4b2c0d22b0 0x7f4b2c1510b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.656+0000 7f4b3355e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4b2c15dfe0 0x7f4b2c0b8500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.656+0000 7f4b3355e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b2c0b3d60 con 0x7f4b2c0b41f0 2024-03-06T07:24:22.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.656+0000 7f4b3355e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b2c0b31e0 con 0x7f4b2c15dfe0 2024-03-06T07:24:22.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.656+0000 7f4b3355e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b2c0d1c70 con 0x7f4b2c0d22b0 2024-03-06T07:24:22.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.656+0000 7f4b312fa640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b2c0b41f0 0x7f4b2c0b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.656+0000 7f4b312fa640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b2c0b41f0 0x7f4b2c0b68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40770/0 (socket says 172.21.15.83:40770) 2024-03-06T07:24:22.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.656+0000 7f4b312fa640 1 -- 172.21.15.83:0/960146375 learned_addr learned my addr 172.21.15.83:0/960146375 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:22.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.657+0000 7f4b31afb640 1 --2- 172.21.15.83:0/960146375 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4b2c15dfe0 0x7f4b2c0b8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.657+0000 7f4b30af9640 1 --2- 172.21.15.83:0/960146375 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4b2c0d22b0 0x7f4b2c1510b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.657+0000 7f4b312fa640 1 -- 172.21.15.83:0/960146375 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4b2c0d22b0 msgr2=0x7f4b2c1510b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.657+0000 7f4b312fa640 1 --2- 172.21.15.83:0/960146375 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4b2c0d22b0 0x7f4b2c1510b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.657+0000 7f4b312fa640 1 -- 172.21.15.83:0/960146375 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4b2c15dfe0 msgr2=0x7f4b2c0b8500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.657+0000 7f4b312fa640 1 --2- 172.21.15.83:0/960146375 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4b2c15dfe0 0x7f4b2c0b8500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.657+0000 7f4b312fa640 1 -- 172.21.15.83:0/960146375 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b2c0b9380 con 0x7f4b2c0b41f0 2024-03-06T07:24:22.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.657+0000 7f4b312fa640 1 --2- 172.21.15.83:0/960146375 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b2c0b41f0 0x7f4b2c0b68d0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f4b2800c400 tx=0x7f4b2800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:22.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.657+0000 7f4b21ffb640 1 -- 172.21.15.83:0/960146375 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b28017070 con 0x7f4b2c0b41f0 2024-03-06T07:24:22.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.657+0000 7f4b21ffb640 1 -- 172.21.15.83:0/960146375 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b280052d0 con 0x7f4b2c0b41f0 2024-03-06T07:24:22.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.657+0000 7f4b21ffb640 1 -- 172.21.15.83:0/960146375 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b28003830 con 0x7f4b2c0b41f0 2024-03-06T07:24:22.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.657+0000 7f4b3355e640 1 -- 172.21.15.83:0/960146375 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b2c0b9660 con 0x7f4b2c0b41f0 2024-03-06T07:24:22.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.657+0000 7f4b3355e640 1 -- 172.21.15.83:0/960146375 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b2c0b6e10 con 0x7f4b2c0b41f0 2024-03-06T07:24:22.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.658+0000 7f4b3355e640 1 -- 172.21.15.83:0/960146375 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b2c04f330 con 0x7f4b2c0b41f0 2024-03-06T07:24:22.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.659+0000 7f4b21ffb640 1 -- 172.21.15.83:0/960146375 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4b2801a070 con 0x7f4b2c0b41f0 2024-03-06T07:24:22.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.659+0000 7f4b21ffb640 1 --2- 172.21.15.83:0/960146375 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4b1006b4d0 0x7f4b1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.659+0000 7f4b21ffb640 1 -- 172.21.15.83:0/960146375 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4b280894b0 con 0x7f4b2c0b41f0 2024-03-06T07:24:22.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.659+0000 7f4b30af9640 1 --2- 172.21.15.83:0/960146375 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4b1006b4d0 0x7f4b1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.660+0000 7f4b30af9640 1 --2- 172.21.15.83:0/960146375 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4b1006b4d0 0x7f4b1006d990 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c036000 tx=0x7f4b1c03b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:22.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.664+0000 7f4b21ffb640 1 -- 172.21.15.83:0/960146375 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4b28057ce0 con 0x7f4b2c0b41f0 2024-03-06T07:24:22.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.878+0000 7f4b3355e640 1 -- 172.21.15.83:0/960146375 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b2c0cb370 con 0x7f4b2c0b41f0 2024-03-06T07:24:22.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.879+0000 7f4b21ffb640 1 -- 172.21.15.83:0/960146375 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f4b2805bd80 con 0x7f4b2c0b41f0 2024-03-06T07:24:22.881 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:24:22.881 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":25,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5665,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/3334609507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3334609507},{"type":"v1","addr":"172.21.15.183:6834","nonce":3334609507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5668,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/691189112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":691189112},{"type":"v1","addr":"172.21.15.183:6835","nonce":691189112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6081,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/3952571284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":3952571284},{"type":"v1","addr":"172.21.15.83:6836","nonce":3952571284}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:19.852157+0000","modified":"2024-03-06T07:24:21.879875+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6084},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6084":{"gid":6084,"name":"a","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/3237506441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3237506441},{"type":"v1","addr":"172.21.15.83:6837","nonce":3237506441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6084,"qdb_cluster":[6084]},"id":4}]} 2024-03-06T07:24:22.881 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 25 2024-03-06T07:24:22.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.881+0000 7f4b3355e640 1 -- 172.21.15.83:0/960146375 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4b1006b4d0 msgr2=0x7f4b1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.881+0000 7f4b3355e640 1 --2- 172.21.15.83:0/960146375 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4b1006b4d0 0x7f4b1006d990 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c036000 tx=0x7f4b1c03b040 comp rx=0 tx=0).stop 2024-03-06T07:24:22.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.881+0000 7f4b3355e640 1 -- 172.21.15.83:0/960146375 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b2c0b41f0 msgr2=0x7f4b2c0b68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.881+0000 7f4b3355e640 1 --2- 172.21.15.83:0/960146375 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b2c0b41f0 0x7f4b2c0b68d0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f4b2800c400 tx=0x7f4b2800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.881+0000 7f4b3355e640 1 -- 172.21.15.83:0/960146375 shutdown_connections 2024-03-06T07:24:22.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.881+0000 7f4b3355e640 1 --2- 172.21.15.83:0/960146375 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4b1006b4d0 0x7f4b1006d990 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.881+0000 7f4b3355e640 1 --2- 172.21.15.83:0/960146375 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4b2c15dfe0 0x7f4b2c0b8500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.881+0000 7f4b3355e640 1 --2- 172.21.15.83:0/960146375 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4b2c0d22b0 0x7f4b2c1510b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.881+0000 7f4b3355e640 1 --2- 172.21.15.83:0/960146375 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b2c0b41f0 0x7f4b2c0b68d0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.881+0000 7f4b3355e640 1 -- 172.21.15.83:0/960146375 >> 172.21.15.83:0/960146375 conn(0x7f4b2c0d3cd0 msgr2=0x7f4b2c0d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:22.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.881+0000 7f4b3355e640 1 -- 172.21.15.83:0/960146375 shutdown_connections 2024-03-06T07:24:22.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.881+0000 7f4b3355e640 1 -- 172.21.15.83:0/960146375 wait complete. 2024-03-06T07:24:22.894 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:22.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.996+0000 7fc296281640 1 Processor -- start 2024-03-06T07:24:22.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.996+0000 7fc296281640 1 -- start start 2024-03-06T07:24:23.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.997+0000 7fc296281640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fc29015dfe0 0x7fc2900cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.997+0000 7fc296281640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc2900cf830 0x7fc2900d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.997+0000 7fc296281640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2900b3430 con 0x7fc2900d27f0 2024-03-06T07:24:23.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.997+0000 7fc296281640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2900b31e0 con 0x7fc2900cf830 2024-03-06T07:24:23.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.997+0000 7fc296281640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2900b3d60 con 0x7fc29015dfe0 2024-03-06T07:24:23.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.997+0000 7fc28f7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc2900cf830 0x7fc2900d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.997+0000 7fc28f7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc2900cf830 0x7fc2900d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40774/0 (socket says 172.21.15.83:40774) 2024-03-06T07:24:23.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.997+0000 7fc28f7fe640 1 -- 172.21.15.83:0/3855175191 learned_addr learned my addr 172.21.15.83:0/3855175191 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:23.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.997+0000 7fc28ffff640 1 --2- 172.21.15.83:0/3855175191 >> v2:172.21.15.183:3300/0 conn(0x7fc29015dfe0 0x7fc2900cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.997+0000 7fc28f7fe640 1 -- 172.21.15.83:0/3855175191 >> v2:172.21.15.183:3300/0 conn(0x7fc29015dfe0 msgr2=0x7fc2900cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:23.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.997+0000 7fc28f7fe640 1 --2- 172.21.15.83:0/3855175191 >> v2:172.21.15.183:3300/0 conn(0x7fc29015dfe0 0x7fc2900cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.997+0000 7fc28f7fe640 1 -- 172.21.15.83:0/3855175191 >> v1:172.21.15.83:6789/0 conn(0x7fc2900d27f0 legacy=0x7fc2900d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.997+0000 7fc28effd640 1 -- 172.21.15.83:0/3855175191 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 211289530 0 0) 0x7fc2900b3430 con 0x7fc2900d27f0 2024-03-06T07:24:23.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.997+0000 7fc28f7fe640 1 -- 172.21.15.83:0/3855175191 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2900b3710 con 0x7fc2900cf830 2024-03-06T07:24:23.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.998+0000 7fc28f7fe640 1 --2- 172.21.15.83:0/3855175191 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2900cf830 0x7fc2900d22b0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fc280009870 tx=0x7fc28002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c07fe73e98512983 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:23.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.998+0000 7fc28effd640 1 -- 172.21.15.83:0/3855175191 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc280034070 con 0x7fc2900cf830 2024-03-06T07:24:23.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.998+0000 7fc28effd640 1 -- 172.21.15.83:0/3855175191 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc280030860 con 0x7fc2900cf830 2024-03-06T07:24:23.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.998+0000 7fc28effd640 1 -- 172.21.15.83:0/3855175191 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc280030c30 con 0x7fc2900cf830 2024-03-06T07:24:23.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.999+0000 7fc296281640 1 -- 172.21.15.83:0/3855175191 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2900cf830 msgr2=0x7fc2900d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.999+0000 7fc296281640 1 --2- 172.21.15.83:0/3855175191 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2900cf830 0x7fc2900d22b0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fc280009870 tx=0x7fc28002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.999+0000 7fc296281640 1 -- 172.21.15.83:0/3855175191 shutdown_connections 2024-03-06T07:24:23.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.999+0000 7fc296281640 1 --2- 172.21.15.83:0/3855175191 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2900cf830 0x7fc2900d22b0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.999+0000 7fc296281640 1 --2- 172.21.15.83:0/3855175191 >> v2:172.21.15.183:3300/0 conn(0x7fc29015dfe0 0x7fc2900cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.999+0000 7fc296281640 1 -- 172.21.15.83:0/3855175191 >> 172.21.15.83:0/3855175191 conn(0x7fc2900d3cd0 msgr2=0x7fc2900d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:23.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.999+0000 7fc296281640 1 -- 172.21.15.83:0/3855175191 shutdown_connections 2024-03-06T07:24:23.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:22.999+0000 7fc296281640 1 -- 172.21.15.83:0/3855175191 wait complete. 2024-03-06T07:24:23.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.000+0000 7fc296281640 1 Processor -- start 2024-03-06T07:24:23.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.000+0000 7fc296281640 1 -- start start 2024-03-06T07:24:23.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.001+0000 7fc296281640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc2900cf830 0x7fc290161000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.001+0000 7fc296281640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc2900d27f0 0x7fc2900b8b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.001+0000 7fc296281640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc29015dfe0 0x7fc2900b9050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.001+0000 7fc296281640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2900d1df0 con 0x7fc29015dfe0 2024-03-06T07:24:23.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.001+0000 7fc296281640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2900d1c70 con 0x7fc2900cf830 2024-03-06T07:24:23.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.001+0000 7fc296281640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc29015e3c0 con 0x7fc2900d27f0 2024-03-06T07:24:23.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.001+0000 7fc29481e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc29015dfe0 0x7fc2900b9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.001+0000 7fc29481e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc29015dfe0 0x7fc2900b9050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40782/0 (socket says 172.21.15.83:40782) 2024-03-06T07:24:23.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.001+0000 7fc29481e640 1 -- 172.21.15.83:0/577697131 learned_addr learned my addr 172.21.15.83:0/577697131 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:23.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.001+0000 7fc28ffff640 1 --2- 172.21.15.83:0/577697131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc2900cf830 0x7fc290161000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.001+0000 7fc28f7fe640 1 --2- 172.21.15.83:0/577697131 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc2900d27f0 0x7fc2900b8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.001+0000 7fc29481e640 1 -- 172.21.15.83:0/577697131 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc2900d27f0 msgr2=0x7fc2900b8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.001+0000 7fc29481e640 1 --2- 172.21.15.83:0/577697131 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc2900d27f0 0x7fc2900b8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.001+0000 7fc29481e640 1 -- 172.21.15.83:0/577697131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc2900cf830 msgr2=0x7fc290161000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.001+0000 7fc29481e640 1 --2- 172.21.15.83:0/577697131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc2900cf830 0x7fc290161000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.001+0000 7fc29481e640 1 -- 172.21.15.83:0/577697131 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc29009f550 con 0x7fc29015dfe0 2024-03-06T07:24:23.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.001+0000 7fc28f7fe640 1 --2- 172.21.15.83:0/577697131 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc2900d27f0 0x7fc2900b8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:23.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.001+0000 7fc28ffff640 1 --2- 172.21.15.83:0/577697131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc2900cf830 0x7fc290161000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:23.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.001+0000 7fc29481e640 1 --2- 172.21.15.83:0/577697131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc29015dfe0 0x7fc2900b9050 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fc28800c760 tx=0x7fc28800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:23.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.001+0000 7fc28cff9640 1 -- 172.21.15.83:0/577697131 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc288003da0 con 0x7fc29015dfe0 2024-03-06T07:24:23.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.002+0000 7fc28cff9640 1 -- 172.21.15.83:0/577697131 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc288005070 con 0x7fc29015dfe0 2024-03-06T07:24:23.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.002+0000 7fc296281640 1 -- 172.21.15.83:0/577697131 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc29009f830 con 0x7fc29015dfe0 2024-03-06T07:24:23.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.002+0000 7fc28cff9640 1 -- 172.21.15.83:0/577697131 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc288005460 con 0x7fc29015dfe0 2024-03-06T07:24:23.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.002+0000 7fc296281640 1 -- 172.21.15.83:0/577697131 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc29009fdf0 con 0x7fc29015dfe0 2024-03-06T07:24:23.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.002+0000 7fc296281640 1 -- 172.21.15.83:0/577697131 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2900cf5f0 con 0x7fc29015dfe0 2024-03-06T07:24:23.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.003+0000 7fc28cff9640 1 -- 172.21.15.83:0/577697131 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc288005600 con 0x7fc29015dfe0 2024-03-06T07:24:23.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.004+0000 7fc28cff9640 1 --2- 172.21.15.83:0/577697131 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc27c06b4d0 0x7fc27c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.004+0000 7fc28ffff640 1 --2- 172.21.15.83:0/577697131 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc27c06b4d0 0x7fc27c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.004+0000 7fc28cff9640 1 -- 172.21.15.83:0/577697131 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc28808a1b0 con 0x7fc29015dfe0 2024-03-06T07:24:23.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.004+0000 7fc28ffff640 1 --2- 172.21.15.83:0/577697131 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc27c06b4d0 0x7fc27c06d990 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc284008890 tx=0x7fc284008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:23.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.008+0000 7fc28cff9640 1 -- 172.21.15.83:0/577697131 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc2900cf5f0 con 0x7fc29015dfe0 2024-03-06T07:24:23.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.236+0000 7fc296281640 1 -- 172.21.15.83:0/577697131 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc29009fff0 con 0x7fc29015dfe0 2024-03-06T07:24:23.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.237+0000 7fc28cff9640 1 -- 172.21.15.83:0/577697131 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+16677 (secure 0 0 0) 0x7fc288058fa0 con 0x7fc29015dfe0 2024-03-06T07:24:23.240 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:24:23.240 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":57,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:24:21.525937+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:18.074366+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:18.831101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:23.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.240+0000 7fc296281640 1 -- 172.21.15.83:0/577697131 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc27c06b4d0 msgr2=0x7fc27c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.240+0000 7fc296281640 1 --2- 172.21.15.83:0/577697131 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc27c06b4d0 0x7fc27c06d990 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc284008890 tx=0x7fc284008040 comp rx=0 tx=0).stop 2024-03-06T07:24:23.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.240+0000 7fc296281640 1 -- 172.21.15.83:0/577697131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc29015dfe0 msgr2=0x7fc2900b9050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.240+0000 7fc296281640 1 --2- 172.21.15.83:0/577697131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc29015dfe0 0x7fc2900b9050 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fc28800c760 tx=0x7fc28800cc20 comp rx=0 tx=0).stop 2024-03-06T07:24:23.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.240+0000 7fc296281640 1 -- 172.21.15.83:0/577697131 shutdown_connections 2024-03-06T07:24:23.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.240+0000 7fc296281640 1 --2- 172.21.15.83:0/577697131 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc27c06b4d0 0x7fc27c06d990 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.240+0000 7fc296281640 1 --2- 172.21.15.83:0/577697131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc29015dfe0 0x7fc2900b9050 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.240+0000 7fc296281640 1 --2- 172.21.15.83:0/577697131 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc2900d27f0 0x7fc2900b8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.240+0000 7fc296281640 1 --2- 172.21.15.83:0/577697131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc2900cf830 0x7fc290161000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.240+0000 7fc296281640 1 -- 172.21.15.83:0/577697131 >> 172.21.15.83:0/577697131 conn(0x7fc2900d3cd0 msgr2=0x7fc29006c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:23.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.240+0000 7fc296281640 1 -- 172.21.15.83:0/577697131 shutdown_connections 2024-03-06T07:24:23.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.240+0000 7fc296281640 1 -- 172.21.15.83:0/577697131 wait complete. 2024-03-06T07:24:23.253 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:24:23.254 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:23.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.355+0000 7f9ba8f55640 1 Processor -- start 2024-03-06T07:24:23.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.355+0000 7f9ba8f55640 1 -- start start 2024-03-06T07:24:23.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.356+0000 7f9ba8f55640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ba40d2710 0x7f9ba40b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.356+0000 7f9ba8f55640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ba40b3430 con 0x7f9ba40d22b0 2024-03-06T07:24:23.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.356+0000 7f9ba8f55640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ba40b3710 con 0x7f9ba415dfe0 2024-03-06T07:24:23.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.356+0000 7f9ba8f55640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba40b31e0 con 0x7f9ba40d2710 2024-03-06T07:24:23.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.356+0000 7f9ba1d9b640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f9ba40d22b0 0x7f9ba40b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39364/0 (socket says 172.21.15.83:39364) 2024-03-06T07:24:23.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.356+0000 7f9ba1d9b640 1 -- 172.21.15.83:0/3528388377 learned_addr learned my addr 172.21.15.83:0/3528388377 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:24:23.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.356+0000 7f9ba2d9d640 1 --2- 172.21.15.83:0/3528388377 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ba40d2710 0x7f9ba40b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.357+0000 7f9ba159a640 1 -- 172.21.15.83:0/3528388377 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2820818766 0 0) 0x7f9ba40b3430 con 0x7f9ba40d22b0 2024-03-06T07:24:23.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.357+0000 7f9ba159a640 1 -- 172.21.15.83:0/3528388377 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b8c003520 con 0x7f9ba40d22b0 2024-03-06T07:24:23.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.357+0000 7f9ba159a640 1 -- 172.21.15.83:0/3528388377 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 671475702 0 0) 0x7f9ba40b3710 con 0x7f9ba415dfe0 2024-03-06T07:24:23.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.357+0000 7f9ba2d9d640 1 -- 172.21.15.83:0/3528388377 >> v1:172.21.15.183:6789/0 conn(0x7f9ba415dfe0 legacy=0x7f9ba40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.357+0000 7f9ba2d9d640 1 -- 172.21.15.83:0/3528388377 >> v1:172.21.15.83:6789/0 conn(0x7f9ba40d22b0 legacy=0x7f9ba40b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.357+0000 7f9ba2d9d640 1 -- 172.21.15.83:0/3528388377 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ba40b39f0 con 0x7f9ba40d2710 2024-03-06T07:24:23.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.358+0000 7f9ba2d9d640 1 --2- 172.21.15.83:0/3528388377 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ba40d2710 0x7f9ba40b4900 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9ba40b3360 tx=0x7f9b9c02bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0b32de8e44c94db server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:23.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.358+0000 7f9ba159a640 1 -- 172.21.15.83:0/3528388377 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c039070 con 0x7f9ba40d2710 2024-03-06T07:24:23.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.358+0000 7f9ba159a640 1 -- 172.21.15.83:0/3528388377 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b9c02ccd0 con 0x7f9ba40d2710 2024-03-06T07:24:23.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.358+0000 7f9ba159a640 1 -- 172.21.15.83:0/3528388377 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c034640 con 0x7f9ba40d2710 2024-03-06T07:24:23.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.358+0000 7f9ba8f55640 1 -- 172.21.15.83:0/3528388377 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ba40d2710 msgr2=0x7f9ba40b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.358+0000 7f9ba8f55640 1 --2- 172.21.15.83:0/3528388377 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ba40d2710 0x7f9ba40b4900 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9ba40b3360 tx=0x7f9b9c02bf20 comp rx=0 tx=0).stop 2024-03-06T07:24:23.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.358+0000 7f9ba8f55640 1 -- 172.21.15.83:0/3528388377 shutdown_connections 2024-03-06T07:24:23.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.358+0000 7f9ba8f55640 1 --2- 172.21.15.83:0/3528388377 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ba40d2710 0x7f9ba40b4900 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.359+0000 7f9ba8f55640 1 -- 172.21.15.83:0/3528388377 >> 172.21.15.83:0/3528388377 conn(0x7f9ba40d3cd0 msgr2=0x7f9ba40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:23.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.359+0000 7f9ba8f55640 1 -- 172.21.15.83:0/3528388377 shutdown_connections 2024-03-06T07:24:23.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.359+0000 7f9ba8f55640 1 -- 172.21.15.83:0/3528388377 wait complete. 2024-03-06T07:24:23.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.360+0000 7f9ba8f55640 1 Processor -- start 2024-03-06T07:24:23.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.360+0000 7f9ba8f55640 1 -- start start 2024-03-06T07:24:23.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.360+0000 7f9ba8f55640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ba40d22b0 0x7f9ba40b8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.360+0000 7f9ba8f55640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ba40d2710 0x7f9ba4160f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.361+0000 7f9ba8f55640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9ba415dfe0 0x7f9ba41f7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.361+0000 7f9ba8f55640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba40cfa00 con 0x7f9ba40d22b0 2024-03-06T07:24:23.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.361+0000 7f9ba8f55640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba40b3d60 con 0x7f9ba415dfe0 2024-03-06T07:24:23.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.361+0000 7f9ba8f55640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba40d1c70 con 0x7f9ba40d2710 2024-03-06T07:24:23.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.361+0000 7f9ba259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ba40d22b0 0x7f9ba40b8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.361+0000 7f9ba259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ba40d22b0 0x7f9ba40b8a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40788/0 (socket says 172.21.15.83:40788) 2024-03-06T07:24:23.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.361+0000 7f9ba259c640 1 -- 172.21.15.83:0/2979532178 learned_addr learned my addr 172.21.15.83:0/2979532178 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:23.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.361+0000 7f9ba1d9b640 1 --2- 172.21.15.83:0/2979532178 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ba40d2710 0x7f9ba4160f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.361+0000 7f9ba2d9d640 1 --2- 172.21.15.83:0/2979532178 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9ba415dfe0 0x7f9ba41f7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.361+0000 7f9ba259c640 1 -- 172.21.15.83:0/2979532178 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ba40d2710 msgr2=0x7f9ba4160f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.361+0000 7f9ba259c640 1 --2- 172.21.15.83:0/2979532178 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ba40d2710 0x7f9ba4160f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.361+0000 7f9ba259c640 1 -- 172.21.15.83:0/2979532178 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9ba415dfe0 msgr2=0x7f9ba41f7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.361+0000 7f9ba259c640 1 --2- 172.21.15.83:0/2979532178 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9ba415dfe0 0x7f9ba41f7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.361+0000 7f9ba259c640 1 -- 172.21.15.83:0/2979532178 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ba40b9600 con 0x7f9ba40d22b0 2024-03-06T07:24:23.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.361+0000 7f9ba259c640 1 --2- 172.21.15.83:0/2979532178 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ba40d22b0 0x7f9ba40b8a10 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f9b9400c3c0 tx=0x7f9b9400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:23.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.361+0000 7f9b92ffd640 1 -- 172.21.15.83:0/2979532178 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b94017070 con 0x7f9ba40d22b0 2024-03-06T07:24:23.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.361+0000 7f9b92ffd640 1 -- 172.21.15.83:0/2979532178 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b940035f0 con 0x7f9ba40d22b0 2024-03-06T07:24:23.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.361+0000 7f9ba8f55640 1 -- 172.21.15.83:0/2979532178 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ba40cb5e0 con 0x7f9ba40d22b0 2024-03-06T07:24:23.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.361+0000 7f9b92ffd640 1 -- 172.21.15.83:0/2979532178 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b940039e0 con 0x7f9ba40d22b0 2024-03-06T07:24:23.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.361+0000 7f9ba8f55640 1 -- 172.21.15.83:0/2979532178 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ba40b8f50 con 0x7f9ba40d22b0 2024-03-06T07:24:23.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.362+0000 7f9ba8f55640 1 -- 172.21.15.83:0/2979532178 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ba408d3c0 con 0x7f9ba40d22b0 2024-03-06T07:24:23.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.363+0000 7f9b92ffd640 1 -- 172.21.15.83:0/2979532178 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9b9401a070 con 0x7f9ba40d22b0 2024-03-06T07:24:23.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.363+0000 7f9b92ffd640 1 --2- 172.21.15.83:0/2979532178 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9b8406b4d0 0x7f9b8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.363+0000 7f9ba1d9b640 1 --2- 172.21.15.83:0/2979532178 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9b8406b4d0 0x7f9b8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.363+0000 7f9b92ffd640 1 -- 172.21.15.83:0/2979532178 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9b94089580 con 0x7f9ba40d22b0 2024-03-06T07:24:23.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.363+0000 7f9ba1d9b640 1 --2- 172.21.15.83:0/2979532178 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9b8406b4d0 0x7f9b8406d990 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9ba4159a90 tx=0x7f9b98009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:23.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.372+0000 7f9b92ffd640 1 -- 172.21.15.83:0/2979532178 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9ba408d3c0 con 0x7f9ba40d22b0 2024-03-06T07:24:23.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.542+0000 7f9ba8f55640 1 -- 172.21.15.83:0/2979532178 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9ba415db20 con 0x7f9b8406b4d0 2024-03-06T07:24:23.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.553+0000 7f9b92ffd640 1 -- 172.21.15.83:0/2979532178 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246639 (secure 0 0 0) 0x7f9ba415db20 con 0x7f9b8406b4d0 2024-03-06T07:24:23.558 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:24:23.560 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-06T07:24:23.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.556+0000 7f9ba8f55640 1 -- 172.21.15.83:0/2979532178 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9b8406b4d0 msgr2=0x7f9b8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.556+0000 7f9ba8f55640 1 --2- 172.21.15.83:0/2979532178 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9b8406b4d0 0x7f9b8406d990 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9ba4159a90 tx=0x7f9b98009040 comp rx=0 tx=0).stop 2024-03-06T07:24:23.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.556+0000 7f9ba8f55640 1 -- 172.21.15.83:0/2979532178 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ba40d22b0 msgr2=0x7f9ba40b8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.556+0000 7f9ba8f55640 1 --2- 172.21.15.83:0/2979532178 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ba40d22b0 0x7f9ba40b8a10 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f9b9400c3c0 tx=0x7f9b9400c880 comp rx=0 tx=0).stop 2024-03-06T07:24:23.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.556+0000 7f9ba8f55640 1 -- 172.21.15.83:0/2979532178 shutdown_connections 2024-03-06T07:24:23.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.557+0000 7f9ba8f55640 1 --2- 172.21.15.83:0/2979532178 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9b8406b4d0 0x7f9b8406d990 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.557+0000 7f9ba8f55640 1 --2- 172.21.15.83:0/2979532178 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9ba415dfe0 0x7f9ba41f7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.557+0000 7f9ba8f55640 1 --2- 172.21.15.83:0/2979532178 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ba40d2710 0x7f9ba4160f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.557+0000 7f9ba8f55640 1 --2- 172.21.15.83:0/2979532178 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ba40d22b0 0x7f9ba40b8a10 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.557+0000 7f9ba8f55640 1 -- 172.21.15.83:0/2979532178 >> 172.21.15.83:0/2979532178 conn(0x7f9ba40d3cd0 msgr2=0x7f9ba406d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:23.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.557+0000 7f9ba8f55640 1 -- 172.21.15.83:0/2979532178 shutdown_connections 2024-03-06T07:24:23.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.557+0000 7f9ba8f55640 1 -- 172.21.15.83:0/2979532178 wait complete. 2024-03-06T07:24:23.569 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2024-03-06T07:24:22.347170+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":49,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":255764,"kb_used_data":6288,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754718956,"statfs":{"total":773094113280,"available":772832210944,"internally_reserved":0,"allocated":6438912,"data_stored":3336594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12757,"internal_metadata":255315499},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000578"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.514792+0000","last_change":"2024-03-06T07:24:19.514792+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.514792+0000","last_undegraded":"2024-03-06T07:24:19.514792+0000","last_fullsized":"2024-03-06T07:24:19.514792+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.524462+0000","last_change":"2024-03-06T07:24:19.524462+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.524462+0000","last_undegraded":"2024-03-06T07:24:19.524462+0000","last_fullsized":"2024-03-06T07:24:19.524462+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.513384+0000","last_change":"2024-03-06T07:24:19.513384+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.513384+0000","last_undegraded":"2024-03-06T07:24:19.513384+0000","last_fullsized":"2024-03-06T07:24:19.513384+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.510475+0000","last_change":"2024-03-06T07:24:19.510475+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.510475+0000","last_undegraded":"2024-03-06T07:24:19.510475+0000","last_fullsized":"2024-03-06T07:24:19.510475+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.518129+0000","last_change":"2024-03-06T07:24:19.518129+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.518129+0000","last_undegraded":"2024-03-06T07:24:19.518129+0000","last_fullsized":"2024-03-06T07:24:19.518129+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.518791+0000","last_change":"2024-03-06T07:24:19.518791+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.518791+0000","last_undegraded":"2024-03-06T07:24:19.518791+0000","last_fullsized":"2024-03-06T07:24:19.518791+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.512145+0000","last_change":"2024-03-06T07:24:19.512145+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.512145+0000","last_undegraded":"2024-03-06T07:24:19.512145+0000","last_fullsized":"2024-03-06T07:24:19.512145+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.523059+0000","last_change":"2024-03-06T07:24:19.523059+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.523059+0000","last_undegraded":"2024-03-06T07:24:19.523059+0000","last_fullsized":"2024-03-06T07:24:19.523059+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.510901+0000","last_change":"2024-03-06T07:24:19.510901+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.510901+0000","last_undegraded":"2024-03-06T07:24:19.510901+0000","last_fullsized":"2024-03-06T07:24:19.510901+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.515316+0000","last_change":"2024-03-06T07:24:19.515316+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.515316+0000","last_undegraded":"2024-03-06T07:24:19.515316+0000","last_fullsized":"2024-03-06T07:24:19.515316+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.516335+0000","last_change":"2024-03-06T07:24:19.516335+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.516335+0000","last_undegraded":"2024-03-06T07:24:19.516335+0000","last_fullsized":"2024-03-06T07:24:19.516335+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.510027+0000","last_change":"2024-03-06T07:24:19.510027+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.510027+0000","last_undegraded":"2024-03-06T07:24:19.510027+0000","last_fullsized":"2024-03-06T07:24:19.510027+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.515035+0000","last_change":"2024-03-06T07:24:19.515035+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.515035+0000","last_undegraded":"2024-03-06T07:24:19.515035+0000","last_fullsized":"2024-03-06T07:24:19.515035+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.512059+0000","last_change":"2024-03-06T07:24:19.512059+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.512059+0000","last_undegraded":"2024-03-06T07:24:19.512059+0000","last_fullsized":"2024-03-06T07:24:19.512059+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.511672+0000","last_change":"2024-03-06T07:24:19.511672+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.511672+0000","last_undegraded":"2024-03-06T07:24:19.511672+0000","last_fullsized":"2024-03-06T07:24:19.511672+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.519288+0000","last_change":"2024-03-06T07:24:19.519288+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.519288+0000","last_undegraded":"2024-03-06T07:24:19.519288+0000","last_fullsized":"2024-03-06T07:24:19.519288+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.522695+0000","last_change":"2024-03-06T07:24:19.522695+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.522695+0000","last_undegraded":"2024-03-06T07:24:19.522695+0000","last_fullsized":"2024-03-06T07:24:19.522695+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508661+0000","last_change":"2024-03-06T07:24:19.508661+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508661+0000","last_undegraded":"2024-03-06T07:24:19.508661+0000","last_fullsized":"2024-03-06T07:24:19.508661+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.514373+0000","last_change":"2024-03-06T07:24:19.514373+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.514373+0000","last_undegraded":"2024-03-06T07:24:19.514373+0000","last_fullsized":"2024-03-06T07:24:19.514373+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.522340+0000","last_change":"2024-03-06T07:24:19.522340+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.522340+0000","last_undegraded":"2024-03-06T07:24:19.522340+0000","last_fullsized":"2024-03-06T07:24:19.522340+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.513618+0000","last_change":"2024-03-06T07:24:19.513618+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.513618+0000","last_undegraded":"2024-03-06T07:24:19.513618+0000","last_fullsized":"2024-03-06T07:24:19.513618+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.521571+0000","last_change":"2024-03-06T07:24:19.521571+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.521571+0000","last_undegraded":"2024-03-06T07:24:19.521571+0000","last_fullsized":"2024-03-06T07:24:19.521571+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.509993+0000","last_change":"2024-03-06T07:24:19.509993+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.509993+0000","last_undegraded":"2024-03-06T07:24:19.509993+0000","last_fullsized":"2024-03-06T07:24:19.509993+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.514668+0000","last_change":"2024-03-06T07:24:19.514668+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.514668+0000","last_undegraded":"2024-03-06T07:24:19.514668+0000","last_fullsized":"2024-03-06T07:24:19.514668+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.514380+0000","last_change":"2024-03-06T07:24:19.514380+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.514380+0000","last_undegraded":"2024-03-06T07:24:19.514380+0000","last_fullsized":"2024-03-06T07:24:19.514380+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.509876+0000","last_change":"2024-03-06T07:24:19.509876+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.509876+0000","last_undegraded":"2024-03-06T07:24:19.509876+0000","last_fullsized":"2024-03-06T07:24:19.509876+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.520572+0000","last_change":"2024-03-06T07:24:19.520572+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.520572+0000","last_undegraded":"2024-03-06T07:24:19.520572+0000","last_fullsized":"2024-03-06T07:24:19.520572+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.515332+0000","last_change":"2024-03-06T07:24:19.515332+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.515332+0000","last_undegraded":"2024-03-06T07:24:19.515332+0000","last_fullsized":"2024-03-06T07:24:19.515332+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.525727+0000","last_change":"2024-03-06T07:24:19.525727+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.525727+0000","last_undegraded":"2024-03-06T07:24:19.525727+0000","last_fullsized":"2024-03-06T07:24:19.525727+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508941+0000","last_change":"2024-03-06T07:24:19.508941+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508941+0000","last_undegraded":"2024-03-06T07:24:19.508941+0000","last_fullsized":"2024-03-06T07:24:19.508941+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.513795+0000","last_change":"2024-03-06T07:24:19.513795+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.513795+0000","last_undegraded":"2024-03-06T07:24:19.513795+0000","last_fullsized":"2024-03-06T07:24:19.513795+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.518693+0000","last_change":"2024-03-06T07:24:19.518693+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.518693+0000","last_undegraded":"2024-03-06T07:24:19.518693+0000","last_fullsized":"2024-03-06T07:24:19.518693+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.518564+0000","last_change":"2024-03-06T07:24:19.518564+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.518564+0000","last_undegraded":"2024-03-06T07:24:19.518564+0000","last_fullsized":"2024-03-06T07:24:19.518564+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.525975+0000","last_change":"2024-03-06T07:24:19.523037+0000","last_active":"2024-03-06T07:24:19.525975+0000","last_peered":"2024-03-06T07:24:19.525975+0000","last_clean":"2024-03-06T07:24:19.525975+0000","last_became_active":"2024-03-06T07:24:19.522767+0000","last_became_peered":"2024-03-06T07:24:19.522767+0000","last_unstale":"2024-03-06T07:24:19.525975+0000","last_undegraded":"2024-03-06T07:24:19.525975+0000","last_fullsized":"2024-03-06T07:24:19.525975+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:42:52.076503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.515662+0000","last_change":"2024-03-06T07:24:19.515662+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.515662+0000","last_undegraded":"2024-03-06T07:24:19.515662+0000","last_fullsized":"2024-03-06T07:24:19.515662+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.517806+0000","last_change":"2024-03-06T07:24:19.516320+0000","last_active":"2024-03-06T07:24:19.517806+0000","last_peered":"2024-03-06T07:24:19.517806+0000","last_clean":"2024-03-06T07:24:19.517806+0000","last_became_active":"2024-03-06T07:24:19.516067+0000","last_became_peered":"2024-03-06T07:24:19.516067+0000","last_unstale":"2024-03-06T07:24:19.517806+0000","last_undegraded":"2024-03-06T07:24:19.517806+0000","last_fullsized":"2024-03-06T07:24:19.517806+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:44:06.313408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.512557+0000","last_change":"2024-03-06T07:24:19.512557+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.512557+0000","last_undegraded":"2024-03-06T07:24:19.512557+0000","last_fullsized":"2024-03-06T07:24:19.512557+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.530956+0000","last_change":"2024-03-06T07:24:19.528754+0000","last_active":"2024-03-06T07:24:19.530956+0000","last_peered":"2024-03-06T07:24:19.530956+0000","last_clean":"2024-03-06T07:24:19.530956+0000","last_became_active":"2024-03-06T07:24:19.528479+0000","last_became_peered":"2024-03-06T07:24:19.528479+0000","last_unstale":"2024-03-06T07:24:19.530956+0000","last_undegraded":"2024-03-06T07:24:19.530956+0000","last_fullsized":"2024-03-06T07:24:19.530956+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:17.842390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.513655+0000","last_change":"2024-03-06T07:24:19.513655+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.513655+0000","last_undegraded":"2024-03-06T07:24:19.513655+0000","last_fullsized":"2024-03-06T07:24:19.513655+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.524698+0000","last_change":"2024-03-06T07:24:19.520169+0000","last_active":"2024-03-06T07:24:19.524698+0000","last_peered":"2024-03-06T07:24:19.524698+0000","last_clean":"2024-03-06T07:24:19.524698+0000","last_became_active":"2024-03-06T07:24:19.519886+0000","last_became_peered":"2024-03-06T07:24:19.519886+0000","last_unstale":"2024-03-06T07:24:19.524698+0000","last_undegraded":"2024-03-06T07:24:19.524698+0000","last_fullsized":"2024-03-06T07:24:19.524698+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:38.333961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.520743+0000","last_change":"2024-03-06T07:24:19.520743+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.520743+0000","last_undegraded":"2024-03-06T07:24:19.520743+0000","last_fullsized":"2024-03-06T07:24:19.520743+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.525372+0000","last_change":"2024-03-06T07:24:19.524214+0000","last_active":"2024-03-06T07:24:19.525372+0000","last_peered":"2024-03-06T07:24:19.525372+0000","last_clean":"2024-03-06T07:24:19.525372+0000","last_became_active":"2024-03-06T07:24:19.523589+0000","last_became_peered":"2024-03-06T07:24:19.523589+0000","last_unstale":"2024-03-06T07:24:19.525372+0000","last_undegraded":"2024-03-06T07:24:19.525372+0000","last_fullsized":"2024-03-06T07:24:19.525372+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:00.115699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.521187+0000","last_change":"2024-03-06T07:24:19.521187+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.521187+0000","last_undegraded":"2024-03-06T07:24:19.521187+0000","last_fullsized":"2024-03-06T07:24:19.521187+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.524818+0000","last_change":"2024-03-06T07:24:19.521026+0000","last_active":"2024-03-06T07:24:19.524818+0000","last_peered":"2024-03-06T07:24:19.524818+0000","last_clean":"2024-03-06T07:24:19.524818+0000","last_became_active":"2024-03-06T07:24:19.520754+0000","last_became_peered":"2024-03-06T07:24:19.520754+0000","last_unstale":"2024-03-06T07:24:19.524818+0000","last_undegraded":"2024-03-06T07:24:19.524818+0000","last_fullsized":"2024-03-06T07:24:19.524818+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:23.502992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.514185+0000","last_change":"2024-03-06T07:24:19.514185+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.514185+0000","last_undegraded":"2024-03-06T07:24:19.514185+0000","last_fullsized":"2024-03-06T07:24:19.514185+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.521399+0000","last_change":"2024-03-06T07:24:19.516459+0000","last_active":"2024-03-06T07:24:19.521399+0000","last_peered":"2024-03-06T07:24:19.521399+0000","last_clean":"2024-03-06T07:24:19.521399+0000","last_became_active":"2024-03-06T07:24:19.516181+0000","last_became_peered":"2024-03-06T07:24:19.516181+0000","last_unstale":"2024-03-06T07:24:19.521399+0000","last_undegraded":"2024-03-06T07:24:19.521399+0000","last_fullsized":"2024-03-06T07:24:19.521399+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:08.665102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.520302+0000","last_change":"2024-03-06T07:24:19.520302+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.520302+0000","last_undegraded":"2024-03-06T07:24:19.520302+0000","last_fullsized":"2024-03-06T07:24:19.520302+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.524364+0000","last_change":"2024-03-06T07:24:19.518659+0000","last_active":"2024-03-06T07:24:19.524364+0000","last_peered":"2024-03-06T07:24:19.524364+0000","last_clean":"2024-03-06T07:24:19.524364+0000","last_became_active":"2024-03-06T07:24:19.517834+0000","last_became_peered":"2024-03-06T07:24:19.517834+0000","last_unstale":"2024-03-06T07:24:19.524364+0000","last_undegraded":"2024-03-06T07:24:19.524364+0000","last_fullsized":"2024-03-06T07:24:19.524364+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:09.323969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.511187+0000","last_change":"2024-03-06T07:24:19.511187+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.511187+0000","last_undegraded":"2024-03-06T07:24:19.511187+0000","last_fullsized":"2024-03-06T07:24:19.511187+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.529051+0000","last_change":"2024-03-06T07:24:19.528439+0000","last_active":"2024-03-06T07:24:19.529051+0000","last_peered":"2024-03-06T07:24:19.529051+0000","last_clean":"2024-03-06T07:24:19.529051+0000","last_became_active":"2024-03-06T07:24:19.528141+0000","last_became_peered":"2024-03-06T07:24:19.528141+0000","last_unstale":"2024-03-06T07:24:19.529051+0000","last_undegraded":"2024-03-06T07:24:19.529051+0000","last_fullsized":"2024-03-06T07:24:19.529051+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:24:36.321071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508569+0000","last_change":"2024-03-06T07:24:19.508569+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508569+0000","last_undegraded":"2024-03-06T07:24:19.508569+0000","last_fullsized":"2024-03-06T07:24:19.508569+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.526823+0000","last_change":"2024-03-06T07:24:19.519750+0000","last_active":"2024-03-06T07:24:19.526823+0000","last_peered":"2024-03-06T07:24:19.526823+0000","last_clean":"2024-03-06T07:24:19.526823+0000","last_became_active":"2024-03-06T07:24:19.519503+0000","last_became_peered":"2024-03-06T07:24:19.519503+0000","last_unstale":"2024-03-06T07:24:19.526823+0000","last_undegraded":"2024-03-06T07:24:19.526823+0000","last_fullsized":"2024-03-06T07:24:19.526823+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:00:04.275748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.514274+0000","last_change":"2024-03-06T07:24:19.514274+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.514274+0000","last_undegraded":"2024-03-06T07:24:19.514274+0000","last_fullsized":"2024-03-06T07:24:19.514274+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.528856+0000","last_change":"2024-03-06T07:24:19.528356+0000","last_active":"2024-03-06T07:24:19.528856+0000","last_peered":"2024-03-06T07:24:19.528856+0000","last_clean":"2024-03-06T07:24:19.528856+0000","last_became_active":"2024-03-06T07:24:19.527954+0000","last_became_peered":"2024-03-06T07:24:19.527954+0000","last_unstale":"2024-03-06T07:24:19.528856+0000","last_undegraded":"2024-03-06T07:24:19.528856+0000","last_fullsized":"2024-03-06T07:24:19.528856+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:22.158292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.512986+0000","last_change":"2024-03-06T07:24:19.512986+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.512986+0000","last_undegraded":"2024-03-06T07:24:19.512986+0000","last_fullsized":"2024-03-06T07:24:19.512986+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.525242+0000","last_change":"2024-03-06T07:24:19.523322+0000","last_active":"2024-03-06T07:24:19.525242+0000","last_peered":"2024-03-06T07:24:19.525242+0000","last_clean":"2024-03-06T07:24:19.525242+0000","last_became_active":"2024-03-06T07:24:19.523034+0000","last_became_peered":"2024-03-06T07:24:19.523034+0000","last_unstale":"2024-03-06T07:24:19.525242+0000","last_undegraded":"2024-03-06T07:24:19.525242+0000","last_fullsized":"2024-03-06T07:24:19.525242+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:43:07.141534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508951+0000","last_change":"2024-03-06T07:24:19.508951+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508951+0000","last_undegraded":"2024-03-06T07:24:19.508951+0000","last_fullsized":"2024-03-06T07:24:19.508951+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.530266+0000","last_change":"2024-03-06T07:24:19.526669+0000","last_active":"2024-03-06T07:24:19.530266+0000","last_peered":"2024-03-06T07:24:19.530266+0000","last_clean":"2024-03-06T07:24:19.530266+0000","last_became_active":"2024-03-06T07:24:19.526276+0000","last_became_peered":"2024-03-06T07:24:19.526276+0000","last_unstale":"2024-03-06T07:24:19.530266+0000","last_undegraded":"2024-03-06T07:24:19.530266+0000","last_fullsized":"2024-03-06T07:24:19.530266+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:32:50.322407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.510711+0000","last_change":"2024-03-06T07:24:19.510711+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.510711+0000","last_undegraded":"2024-03-06T07:24:19.510711+0000","last_fullsized":"2024-03-06T07:24:19.510711+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.520976+0000","last_change":"2024-03-06T07:24:19.520321+0000","last_active":"2024-03-06T07:24:19.520976+0000","last_peered":"2024-03-06T07:24:19.520976+0000","last_clean":"2024-03-06T07:24:19.520976+0000","last_became_active":"2024-03-06T07:24:19.520045+0000","last_became_peered":"2024-03-06T07:24:19.520045+0000","last_unstale":"2024-03-06T07:24:19.520976+0000","last_undegraded":"2024-03-06T07:24:19.520976+0000","last_fullsized":"2024-03-06T07:24:19.520976+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:01.534402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.511628+0000","last_change":"2024-03-06T07:24:19.511628+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.511628+0000","last_undegraded":"2024-03-06T07:24:19.511628+0000","last_fullsized":"2024-03-06T07:24:19.511628+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.524716+0000","last_change":"2024-03-06T07:24:19.520152+0000","last_active":"2024-03-06T07:24:19.524716+0000","last_peered":"2024-03-06T07:24:19.524716+0000","last_clean":"2024-03-06T07:24:19.524716+0000","last_became_active":"2024-03-06T07:24:19.519855+0000","last_became_peered":"2024-03-06T07:24:19.519855+0000","last_unstale":"2024-03-06T07:24:19.524716+0000","last_undegraded":"2024-03-06T07:24:19.524716+0000","last_fullsized":"2024-03-06T07:24:19.524716+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:20.649547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.509270+0000","last_change":"2024-03-06T07:24:19.509270+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.509270+0000","last_undegraded":"2024-03-06T07:24:19.509270+0000","last_fullsized":"2024-03-06T07:24:19.509270+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.525829+0000","last_change":"2024-03-06T07:24:19.525401+0000","last_active":"2024-03-06T07:24:19.525829+0000","last_peered":"2024-03-06T07:24:19.525829+0000","last_clean":"2024-03-06T07:24:19.525829+0000","last_became_active":"2024-03-06T07:24:19.525064+0000","last_became_peered":"2024-03-06T07:24:19.525064+0000","last_unstale":"2024-03-06T07:24:19.525829+0000","last_undegraded":"2024-03-06T07:24:19.525829+0000","last_fullsized":"2024-03-06T07:24:19.525829+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:38.333961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508079+0000","last_change":"2024-03-06T07:24:19.508079+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508079+0000","last_undegraded":"2024-03-06T07:24:19.508079+0000","last_fullsized":"2024-03-06T07:24:19.508079+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.527834+0000","last_change":"2024-03-06T07:24:19.521974+0000","last_active":"2024-03-06T07:24:19.527834+0000","last_peered":"2024-03-06T07:24:19.527834+0000","last_clean":"2024-03-06T07:24:19.527834+0000","last_became_active":"2024-03-06T07:24:19.521736+0000","last_became_peered":"2024-03-06T07:24:19.521736+0000","last_unstale":"2024-03-06T07:24:19.527834+0000","last_undegraded":"2024-03-06T07:24:19.527834+0000","last_fullsized":"2024-03-06T07:24:19.527834+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:20:20.174477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508869+0000","last_change":"2024-03-06T07:24:19.508869+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508869+0000","last_undegraded":"2024-03-06T07:24:19.508869+0000","last_fullsized":"2024-03-06T07:24:19.508869+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.525378+0000","last_change":"2024-03-06T07:24:19.523871+0000","last_active":"2024-03-06T07:24:19.525378+0000","last_peered":"2024-03-06T07:24:19.525378+0000","last_clean":"2024-03-06T07:24:19.525378+0000","last_became_active":"2024-03-06T07:24:19.523587+0000","last_became_peered":"2024-03-06T07:24:19.523587+0000","last_unstale":"2024-03-06T07:24:19.525378+0000","last_undegraded":"2024-03-06T07:24:19.525378+0000","last_fullsized":"2024-03-06T07:24:19.525378+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:16.014585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.520446+0000","last_change":"2024-03-06T07:24:19.520446+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.520446+0000","last_undegraded":"2024-03-06T07:24:19.520446+0000","last_fullsized":"2024-03-06T07:24:19.520446+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.524688+0000","last_change":"2024-03-06T07:24:19.520212+0000","last_active":"2024-03-06T07:24:19.524688+0000","last_peered":"2024-03-06T07:24:19.524688+0000","last_clean":"2024-03-06T07:24:19.524688+0000","last_became_active":"2024-03-06T07:24:19.519913+0000","last_became_peered":"2024-03-06T07:24:19.519913+0000","last_unstale":"2024-03-06T07:24:19.524688+0000","last_undegraded":"2024-03-06T07:24:19.524688+0000","last_fullsized":"2024-03-06T07:24:19.524688+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:22.158292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508125+0000","last_change":"2024-03-06T07:24:19.508125+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508125+0000","last_undegraded":"2024-03-06T07:24:19.508125+0000","last_fullsized":"2024-03-06T07:24:19.508125+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.524670+0000","last_change":"2024-03-06T07:24:19.520035+0000","last_active":"2024-03-06T07:24:19.524670+0000","last_peered":"2024-03-06T07:24:19.524670+0000","last_clean":"2024-03-06T07:24:19.524670+0000","last_became_active":"2024-03-06T07:24:19.519761+0000","last_became_peered":"2024-03-06T07:24:19.519761+0000","last_unstale":"2024-03-06T07:24:19.524670+0000","last_undegraded":"2024-03-06T07:24:19.524670+0000","last_fullsized":"2024-03-06T07:24:19.524670+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:45.374706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508234+0000","last_change":"2024-03-06T07:24:19.508234+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508234+0000","last_undegraded":"2024-03-06T07:24:19.508234+0000","last_fullsized":"2024-03-06T07:24:19.508234+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.517678+0000","last_change":"2024-03-06T07:24:19.515544+0000","last_active":"2024-03-06T07:24:19.517678+0000","last_peered":"2024-03-06T07:24:19.517678+0000","last_clean":"2024-03-06T07:24:19.517678+0000","last_became_active":"2024-03-06T07:24:19.515266+0000","last_became_peered":"2024-03-06T07:24:19.515266+0000","last_unstale":"2024-03-06T07:24:19.517678+0000","last_undegraded":"2024-03-06T07:24:19.517678+0000","last_fullsized":"2024-03-06T07:24:19.517678+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:25:58.500915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.512821+0000","last_change":"2024-03-06T07:24:19.512821+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.512821+0000","last_undegraded":"2024-03-06T07:24:19.512821+0000","last_fullsized":"2024-03-06T07:24:19.512821+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.527496+0000","last_change":"2024-03-06T07:24:19.521024+0000","last_active":"2024-03-06T07:24:19.527496+0000","last_peered":"2024-03-06T07:24:19.527496+0000","last_clean":"2024-03-06T07:24:19.527496+0000","last_became_active":"2024-03-06T07:24:19.520754+0000","last_became_peered":"2024-03-06T07:24:19.520754+0000","last_unstale":"2024-03-06T07:24:19.527496+0000","last_undegraded":"2024-03-06T07:24:19.527496+0000","last_fullsized":"2024-03-06T07:24:19.527496+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:38.661204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.512496+0000","last_change":"2024-03-06T07:24:19.512496+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.512496+0000","last_undegraded":"2024-03-06T07:24:19.512496+0000","last_fullsized":"2024-03-06T07:24:19.512496+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.524611+0000","last_change":"2024-03-06T07:24:19.519653+0000","last_active":"2024-03-06T07:24:19.524611+0000","last_peered":"2024-03-06T07:24:19.524611+0000","last_clean":"2024-03-06T07:24:19.524611+0000","last_became_active":"2024-03-06T07:24:19.519374+0000","last_became_peered":"2024-03-06T07:24:19.519374+0000","last_unstale":"2024-03-06T07:24:19.524611+0000","last_undegraded":"2024-03-06T07:24:19.524611+0000","last_fullsized":"2024-03-06T07:24:19.524611+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:02:35.379525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508772+0000","last_change":"2024-03-06T07:24:19.508772+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508772+0000","last_undegraded":"2024-03-06T07:24:19.508772+0000","last_fullsized":"2024-03-06T07:24:19.508772+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.527636+0000","last_change":"2024-03-06T07:24:19.527152+0000","last_active":"2024-03-06T07:24:19.527636+0000","last_peered":"2024-03-06T07:24:19.527636+0000","last_clean":"2024-03-06T07:24:19.527636+0000","last_became_active":"2024-03-06T07:24:19.526879+0000","last_became_peered":"2024-03-06T07:24:19.526879+0000","last_unstale":"2024-03-06T07:24:19.527636+0000","last_undegraded":"2024-03-06T07:24:19.527636+0000","last_fullsized":"2024-03-06T07:24:19.527636+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:01.534402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508386+0000","last_change":"2024-03-06T07:24:19.508386+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508386+0000","last_undegraded":"2024-03-06T07:24:19.508386+0000","last_fullsized":"2024-03-06T07:24:19.508386+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.524915+0000","last_change":"2024-03-06T07:24:19.520942+0000","last_active":"2024-03-06T07:24:19.524915+0000","last_peered":"2024-03-06T07:24:19.524915+0000","last_clean":"2024-03-06T07:24:19.524915+0000","last_became_active":"2024-03-06T07:24:19.520672+0000","last_became_peered":"2024-03-06T07:24:19.520672+0000","last_unstale":"2024-03-06T07:24:19.524915+0000","last_undegraded":"2024-03-06T07:24:19.524915+0000","last_fullsized":"2024-03-06T07:24:19.524915+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:15.049439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.513801+0000","last_change":"2024-03-06T07:24:19.513801+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.513801+0000","last_undegraded":"2024-03-06T07:24:19.513801+0000","last_fullsized":"2024-03-06T07:24:19.513801+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.524240+0000","last_change":"2024-03-06T07:24:19.522226+0000","last_active":"2024-03-06T07:24:19.524240+0000","last_peered":"2024-03-06T07:24:19.524240+0000","last_clean":"2024-03-06T07:24:19.524240+0000","last_became_active":"2024-03-06T07:24:19.521882+0000","last_became_peered":"2024-03-06T07:24:19.521882+0000","last_unstale":"2024-03-06T07:24:19.524240+0000","last_undegraded":"2024-03-06T07:24:19.524240+0000","last_fullsized":"2024-03-06T07:24:19.524240+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:24.269083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.513191+0000","last_change":"2024-03-06T07:24:19.513191+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.513191+0000","last_undegraded":"2024-03-06T07:24:19.513191+0000","last_fullsized":"2024-03-06T07:24:19.513191+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.521727+0000","last_change":"2024-03-06T07:24:19.516436+0000","last_active":"2024-03-06T07:24:19.521727+0000","last_peered":"2024-03-06T07:24:19.521727+0000","last_clean":"2024-03-06T07:24:19.521727+0000","last_became_active":"2024-03-06T07:24:19.515998+0000","last_became_peered":"2024-03-06T07:24:19.515998+0000","last_unstale":"2024-03-06T07:24:19.521727+0000","last_undegraded":"2024-03-06T07:24:19.521727+0000","last_fullsized":"2024-03-06T07:24:19.521727+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:09.161121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.519430+0000","last_change":"2024-03-06T07:24:19.519430+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.519430+0000","last_undegraded":"2024-03-06T07:24:19.519430+0000","last_fullsized":"2024-03-06T07:24:19.519430+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.527547+0000","last_change":"2024-03-06T07:24:19.521792+0000","last_active":"2024-03-06T07:24:19.527547+0000","last_peered":"2024-03-06T07:24:19.527547+0000","last_clean":"2024-03-06T07:24:19.527547+0000","last_became_active":"2024-03-06T07:24:19.521565+0000","last_became_peered":"2024-03-06T07:24:19.521565+0000","last_unstale":"2024-03-06T07:24:19.527547+0000","last_undegraded":"2024-03-06T07:24:19.527547+0000","last_fullsized":"2024-03-06T07:24:19.527547+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:02.649347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508067+0000","last_change":"2024-03-06T07:24:19.508067+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508067+0000","last_undegraded":"2024-03-06T07:24:19.508067+0000","last_fullsized":"2024-03-06T07:24:19.508067+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.525392+0000","last_change":"2024-03-06T07:24:19.524110+0000","last_active":"2024-03-06T07:24:19.525392+0000","last_peered":"2024-03-06T07:24:19.525392+0000","last_clean":"2024-03-06T07:24:19.525392+0000","last_became_active":"2024-03-06T07:24:19.523812+0000","last_became_peered":"2024-03-06T07:24:19.523812+0000","last_unstale":"2024-03-06T07:24:19.525392+0000","last_undegraded":"2024-03-06T07:24:19.525392+0000","last_fullsized":"2024-03-06T07:24:19.525392+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:22:22.348426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.513750+0000","last_change":"2024-03-06T07:24:19.513750+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.513750+0000","last_undegraded":"2024-03-06T07:24:19.513750+0000","last_fullsized":"2024-03-06T07:24:19.513750+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.527467+0000","last_change":"2024-03-06T07:24:19.520220+0000","last_active":"2024-03-06T07:24:19.527467+0000","last_peered":"2024-03-06T07:24:19.527467+0000","last_clean":"2024-03-06T07:24:19.527467+0000","last_became_active":"2024-03-06T07:24:19.519962+0000","last_became_peered":"2024-03-06T07:24:19.519962+0000","last_unstale":"2024-03-06T07:24:19.527467+0000","last_undegraded":"2024-03-06T07:24:19.527467+0000","last_fullsized":"2024-03-06T07:24:19.527467+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:36:34.165775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.518980+0000","last_change":"2024-03-06T07:24:19.518980+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.518980+0000","last_undegraded":"2024-03-06T07:24:19.518980+0000","last_fullsized":"2024-03-06T07:24:19.518980+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.521731+0000","last_change":"2024-03-06T07:24:19.516940+0000","last_active":"2024-03-06T07:24:19.521731+0000","last_peered":"2024-03-06T07:24:19.521731+0000","last_clean":"2024-03-06T07:24:19.521731+0000","last_became_active":"2024-03-06T07:24:19.516691+0000","last_became_peered":"2024-03-06T07:24:19.516691+0000","last_unstale":"2024-03-06T07:24:19.521731+0000","last_undegraded":"2024-03-06T07:24:19.521731+0000","last_fullsized":"2024-03-06T07:24:19.521731+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:19.361223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.519861+0000","last_change":"2024-03-06T07:24:19.519861+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.519861+0000","last_undegraded":"2024-03-06T07:24:19.519861+0000","last_fullsized":"2024-03-06T07:24:19.519861+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.524300+0000","last_change":"2024-03-06T07:24:19.522294+0000","last_active":"2024-03-06T07:24:19.524300+0000","last_peered":"2024-03-06T07:24:19.524300+0000","last_clean":"2024-03-06T07:24:19.524300+0000","last_became_active":"2024-03-06T07:24:19.521845+0000","last_became_peered":"2024-03-06T07:24:19.521845+0000","last_unstale":"2024-03-06T07:24:19.524300+0000","last_undegraded":"2024-03-06T07:24:19.524300+0000","last_fullsized":"2024-03-06T07:24:19.524300+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:00:32.355340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":232,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.516321+0000","last_change":"2024-03-06T07:21:47.553693+0000","last_active":"2024-03-06T07:24:19.516321+0000","last_peered":"2024-03-06T07:24:19.516321+0000","last_clean":"2024-03-06T07:24:19.516321+0000","last_became_active":"2024-03-06T07:21:47.552664+0000","last_became_peered":"2024-03-06T07:21:47.552664+0000","last_unstale":"2024-03-06T07:24:19.516321+0000","last_undegraded":"2024-03-06T07:24:19.516321+0000","last_fullsized":"2024-03-06T07:24:19.516321+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_clean_scrub_stamp":"2024-03-06T07:21:46.362028+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:02.916426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32412,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339428,"statfs":{"total":96636764160,"available":96603574272,"internally_reserved":0,"allocated":1253376,"data_stored":861417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574882,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31828,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340012,"statfs":{"total":96636764160,"available":96604172288,"internally_reserved":0,"allocated":659456,"data_stored":271049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31832,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340008,"statfs":{"total":96636764160,"available":96604168192,"internally_reserved":0,"allocated":659456,"data_stored":271049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574884,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31812,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340028,"statfs":{"total":96636764160,"available":96604188672,"internally_reserved":0,"allocated":647168,"data_stored":264782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574883,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31816,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340024,"statfs":{"total":96636764160,"available":96604184576,"internally_reserved":0,"allocated":647168,"data_stored":264782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31832,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340008,"statfs":{"total":96636764160,"available":96604168192,"internally_reserved":0,"allocated":659456,"data_stored":271049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31828,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340012,"statfs":{"total":96636764160,"available":96604172288,"internally_reserved":0,"allocated":659456,"data_stored":271049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574883,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32404,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339436,"statfs":{"total":96636764160,"available":96603582464,"internally_reserved":0,"allocated":1253376,"data_stored":861417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:23.574 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:23.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.709+0000 7f9da70af640 1 Processor -- start 2024-03-06T07:24:23.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.709+0000 7f9da70af640 1 -- start start 2024-03-06T07:24:23.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.710+0000 7f9da70af640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f9da015dfe0 0x7f9da00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.710+0000 7f9da70af640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9da00d29c0 0x7f9da00b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.710+0000 7f9da70af640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9da00b3430 con 0x7f9da00cf830 2024-03-06T07:24:23.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.710+0000 7f9da70af640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9da00b31e0 con 0x7f9da015dfe0 2024-03-06T07:24:23.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.710+0000 7f9da70af640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9da00b3d60 con 0x7f9da00d29c0 2024-03-06T07:24:23.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.710+0000 7f9d9ffff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f9da00cf830 0x7f9da00d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39378/0 (socket says 172.21.15.83:39378) 2024-03-06T07:24:23.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.710+0000 7f9d9ffff640 1 -- 172.21.15.83:0/640817751 learned_addr learned my addr 172.21.15.83:0/640817751 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:24:23.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.710+0000 7f9da564c640 1 --2- 172.21.15.83:0/640817751 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9da00d29c0 0x7f9da00b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.710+0000 7f9da4e4b640 1 --2- 172.21.15.83:0/640817751 >> v2:172.21.15.183:3300/0 conn(0x7f9da015dfe0 0x7f9da00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.710+0000 7f9d9f7fe640 1 -- 172.21.15.83:0/640817751 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2755375050 0 0) 0x7f9da00b3430 con 0x7f9da00cf830 2024-03-06T07:24:23.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.710+0000 7f9d9f7fe640 1 -- 172.21.15.83:0/640817751 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d90003520 con 0x7f9da00cf830 2024-03-06T07:24:23.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.711+0000 7f9d9f7fe640 1 -- 172.21.15.83:0/640817751 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4060864435 0 0) 0x7f9d90003520 con 0x7f9da00cf830 2024-03-06T07:24:23.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.711+0000 7f9da564c640 1 -- 172.21.15.83:0/640817751 >> v2:172.21.15.183:3300/0 conn(0x7f9da015dfe0 msgr2=0x7f9da00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:23.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.711+0000 7f9da564c640 1 --2- 172.21.15.83:0/640817751 >> v2:172.21.15.183:3300/0 conn(0x7f9da015dfe0 0x7f9da00cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.711+0000 7f9da564c640 1 -- 172.21.15.83:0/640817751 >> v1:172.21.15.83:6789/0 conn(0x7f9da00cf830 legacy=0x7f9da00d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.711+0000 7f9da564c640 1 -- 172.21.15.83:0/640817751 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da00b3710 con 0x7f9da00d29c0 2024-03-06T07:24:23.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.711+0000 7f9da4e4b640 1 --2- 172.21.15.83:0/640817751 >> v2:172.21.15.183:3300/0 conn(0x7f9da015dfe0 0x7f9da00cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:23.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.711+0000 7f9da564c640 1 --2- 172.21.15.83:0/640817751 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9da00d29c0 0x7f9da00b41f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9d98009d90 tx=0x7f9d9802bd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9ed28c0fa6c4ce0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:23.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.712+0000 7f9d9f7fe640 1 -- 172.21.15.83:0/640817751 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d98039070 con 0x7f9da00d29c0 2024-03-06T07:24:23.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.712+0000 7f9d9f7fe640 1 -- 172.21.15.83:0/640817751 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d9802c8a0 con 0x7f9da00d29c0 2024-03-06T07:24:23.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.712+0000 7f9d9f7fe640 1 -- 172.21.15.83:0/640817751 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c001670 con 0x7f9da00d29c0 2024-03-06T07:24:23.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.712+0000 7f9da70af640 1 -- 172.21.15.83:0/640817751 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9da00d29c0 msgr2=0x7f9da00b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.712+0000 7f9da70af640 1 --2- 172.21.15.83:0/640817751 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9da00d29c0 0x7f9da00b41f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9d98009d90 tx=0x7f9d9802bd70 comp rx=0 tx=0).stop 2024-03-06T07:24:23.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.712+0000 7f9da70af640 1 -- 172.21.15.83:0/640817751 shutdown_connections 2024-03-06T07:24:23.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.712+0000 7f9da70af640 1 --2- 172.21.15.83:0/640817751 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9da00d29c0 0x7f9da00b41f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.712+0000 7f9da70af640 1 --2- 172.21.15.83:0/640817751 >> v2:172.21.15.183:3300/0 conn(0x7f9da015dfe0 0x7f9da00cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.712+0000 7f9da70af640 1 -- 172.21.15.83:0/640817751 >> 172.21.15.83:0/640817751 conn(0x7f9da00d3cd0 msgr2=0x7f9da00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:23.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.712+0000 7f9da70af640 1 -- 172.21.15.83:0/640817751 shutdown_connections 2024-03-06T07:24:23.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.712+0000 7f9da70af640 1 -- 172.21.15.83:0/640817751 wait complete. 2024-03-06T07:24:23.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.713+0000 7f9da70af640 1 Processor -- start 2024-03-06T07:24:23.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.713+0000 7f9da70af640 1 -- start start 2024-03-06T07:24:23.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.714+0000 7f9da70af640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9da00cf830 0x7f9da015e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.714+0000 7f9da70af640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9da00d29c0 0x7f9da01521c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.714+0000 7f9da70af640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9da015dfe0 0x7f9da00b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.714+0000 7f9da70af640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da00d1df0 con 0x7f9da00cf830 2024-03-06T07:24:23.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.714+0000 7f9da70af640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da00d1c70 con 0x7f9da00d29c0 2024-03-06T07:24:23.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.714+0000 7f9da70af640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9da015e3c0 con 0x7f9da015dfe0 2024-03-06T07:24:23.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.714+0000 7f9da4e4b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9da00cf830 0x7f9da015e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.714+0000 7f9da4e4b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9da00cf830 0x7f9da015e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40802/0 (socket says 172.21.15.83:40802) 2024-03-06T07:24:23.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.714+0000 7f9da4e4b640 1 -- 172.21.15.83:0/4082351464 learned_addr learned my addr 172.21.15.83:0/4082351464 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:23.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.714+0000 7f9da564c640 1 --2- 172.21.15.83:0/4082351464 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9da015dfe0 0x7f9da00b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.714+0000 7f9d9ffff640 1 --2- 172.21.15.83:0/4082351464 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9da00d29c0 0x7f9da01521c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.714+0000 7f9da4e4b640 1 -- 172.21.15.83:0/4082351464 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9da015dfe0 msgr2=0x7f9da00b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.714+0000 7f9da4e4b640 1 --2- 172.21.15.83:0/4082351464 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9da015dfe0 0x7f9da00b68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.714+0000 7f9da4e4b640 1 -- 172.21.15.83:0/4082351464 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9da00d29c0 msgr2=0x7f9da01521c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.714+0000 7f9da4e4b640 1 --2- 172.21.15.83:0/4082351464 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9da00d29c0 0x7f9da01521c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.714+0000 7f9da4e4b640 1 -- 172.21.15.83:0/4082351464 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da0152700 con 0x7f9da00cf830 2024-03-06T07:24:23.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.714+0000 7f9d9ffff640 1 --2- 172.21.15.83:0/4082351464 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9da00d29c0 0x7f9da01521c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:23.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.714+0000 7f9da564c640 1 --2- 172.21.15.83:0/4082351464 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9da015dfe0 0x7f9da00b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:23.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.715+0000 7f9da4e4b640 1 --2- 172.21.15.83:0/4082351464 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9da00cf830 0x7f9da015e750 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f9d9400e960 tx=0x7f9d9400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:23.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.715+0000 7f9d9d7fa640 1 -- 172.21.15.83:0/4082351464 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d94017070 con 0x7f9da00cf830 2024-03-06T07:24:23.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.715+0000 7f9da70af640 1 -- 172.21.15.83:0/4082351464 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9da015ed30 con 0x7f9da00cf830 2024-03-06T07:24:23.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.715+0000 7f9d9d7fa640 1 -- 172.21.15.83:0/4082351464 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d9400ca00 con 0x7f9da00cf830 2024-03-06T07:24:23.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.715+0000 7f9d9d7fa640 1 -- 172.21.15.83:0/4082351464 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d9400cdd0 con 0x7f9da00cf830 2024-03-06T07:24:23.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.715+0000 7f9da70af640 1 -- 172.21.15.83:0/4082351464 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9da015f2f0 con 0x7f9da00cf830 2024-03-06T07:24:23.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.716+0000 7f9da70af640 1 -- 172.21.15.83:0/4082351464 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9da00d3740 con 0x7f9da00cf830 2024-03-06T07:24:23.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.716+0000 7f9d9d7fa640 1 -- 172.21.15.83:0/4082351464 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9d9401a070 con 0x7f9da00cf830 2024-03-06T07:24:23.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.717+0000 7f9d9d7fa640 1 --2- 172.21.15.83:0/4082351464 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9d8406b4d0 0x7f9d8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.717+0000 7f9d9ffff640 1 --2- 172.21.15.83:0/4082351464 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9d8406b4d0 0x7f9d8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.717+0000 7f9d9d7fa640 1 -- 172.21.15.83:0/4082351464 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9d9408a140 con 0x7f9da00cf830 2024-03-06T07:24:23.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.717+0000 7f9d9ffff640 1 --2- 172.21.15.83:0/4082351464 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9d8406b4d0 0x7f9d8406d990 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f9da00b53e0 tx=0x7f9d8c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:23.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.721+0000 7f9d9d7fa640 1 -- 172.21.15.83:0/4082351464 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9da00d3740 con 0x7f9da00cf830 2024-03-06T07:24:23.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.896+0000 7f9da70af640 1 -- 172.21.15.83:0/4082351464 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9da004f330 con 0x7f9d8406b4d0 2024-03-06T07:24:23.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.902+0000 7f9d9d7fa640 1 -- 172.21.15.83:0/4082351464 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246639 (secure 0 0 0) 0x7f9da004f330 con 0x7f9d8406b4d0 2024-03-06T07:24:23.907 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:24:23.908 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-06T07:24:23.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.905+0000 7f9da70af640 1 -- 172.21.15.83:0/4082351464 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9d8406b4d0 msgr2=0x7f9d8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.905+0000 7f9da70af640 1 --2- 172.21.15.83:0/4082351464 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9d8406b4d0 0x7f9d8406d990 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f9da00b53e0 tx=0x7f9d8c009040 comp rx=0 tx=0).stop 2024-03-06T07:24:23.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.905+0000 7f9da70af640 1 -- 172.21.15.83:0/4082351464 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9da00cf830 msgr2=0x7f9da015e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.905+0000 7f9da70af640 1 --2- 172.21.15.83:0/4082351464 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9da00cf830 0x7f9da015e750 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f9d9400e960 tx=0x7f9d9400ee20 comp rx=0 tx=0).stop 2024-03-06T07:24:23.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.905+0000 7f9da70af640 1 -- 172.21.15.83:0/4082351464 shutdown_connections 2024-03-06T07:24:23.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.905+0000 7f9da70af640 1 --2- 172.21.15.83:0/4082351464 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9d8406b4d0 0x7f9d8406d990 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.905+0000 7f9da70af640 1 --2- 172.21.15.83:0/4082351464 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9da015dfe0 0x7f9da00b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.905+0000 7f9da70af640 1 --2- 172.21.15.83:0/4082351464 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9da00d29c0 0x7f9da01521c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.905+0000 7f9da70af640 1 --2- 172.21.15.83:0/4082351464 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9da00cf830 0x7f9da015e750 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.905+0000 7f9da70af640 1 -- 172.21.15.83:0/4082351464 >> 172.21.15.83:0/4082351464 conn(0x7f9da00d3cd0 msgr2=0x7f9da00d22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:23.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.905+0000 7f9da70af640 1 -- 172.21.15.83:0/4082351464 shutdown_connections 2024-03-06T07:24:23.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:23.905+0000 7f9da70af640 1 -- 172.21.15.83:0/4082351464 wait complete. 2024-03-06T07:24:23.919 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2024-03-06T07:24:22.347170+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":49,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":255764,"kb_used_data":6288,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754718956,"statfs":{"total":773094113280,"available":772832210944,"internally_reserved":0,"allocated":6438912,"data_stored":3336594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12757,"internal_metadata":255315499},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000578"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.514792+0000","last_change":"2024-03-06T07:24:19.514792+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.514792+0000","last_undegraded":"2024-03-06T07:24:19.514792+0000","last_fullsized":"2024-03-06T07:24:19.514792+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.524462+0000","last_change":"2024-03-06T07:24:19.524462+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.524462+0000","last_undegraded":"2024-03-06T07:24:19.524462+0000","last_fullsized":"2024-03-06T07:24:19.524462+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.513384+0000","last_change":"2024-03-06T07:24:19.513384+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.513384+0000","last_undegraded":"2024-03-06T07:24:19.513384+0000","last_fullsized":"2024-03-06T07:24:19.513384+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.510475+0000","last_change":"2024-03-06T07:24:19.510475+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.510475+0000","last_undegraded":"2024-03-06T07:24:19.510475+0000","last_fullsized":"2024-03-06T07:24:19.510475+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.518129+0000","last_change":"2024-03-06T07:24:19.518129+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.518129+0000","last_undegraded":"2024-03-06T07:24:19.518129+0000","last_fullsized":"2024-03-06T07:24:19.518129+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.518791+0000","last_change":"2024-03-06T07:24:19.518791+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.518791+0000","last_undegraded":"2024-03-06T07:24:19.518791+0000","last_fullsized":"2024-03-06T07:24:19.518791+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.512145+0000","last_change":"2024-03-06T07:24:19.512145+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.512145+0000","last_undegraded":"2024-03-06T07:24:19.512145+0000","last_fullsized":"2024-03-06T07:24:19.512145+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.523059+0000","last_change":"2024-03-06T07:24:19.523059+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.523059+0000","last_undegraded":"2024-03-06T07:24:19.523059+0000","last_fullsized":"2024-03-06T07:24:19.523059+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.510901+0000","last_change":"2024-03-06T07:24:19.510901+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.510901+0000","last_undegraded":"2024-03-06T07:24:19.510901+0000","last_fullsized":"2024-03-06T07:24:19.510901+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.515316+0000","last_change":"2024-03-06T07:24:19.515316+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.515316+0000","last_undegraded":"2024-03-06T07:24:19.515316+0000","last_fullsized":"2024-03-06T07:24:19.515316+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.516335+0000","last_change":"2024-03-06T07:24:19.516335+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.516335+0000","last_undegraded":"2024-03-06T07:24:19.516335+0000","last_fullsized":"2024-03-06T07:24:19.516335+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.510027+0000","last_change":"2024-03-06T07:24:19.510027+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.510027+0000","last_undegraded":"2024-03-06T07:24:19.510027+0000","last_fullsized":"2024-03-06T07:24:19.510027+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.515035+0000","last_change":"2024-03-06T07:24:19.515035+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.515035+0000","last_undegraded":"2024-03-06T07:24:19.515035+0000","last_fullsized":"2024-03-06T07:24:19.515035+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.512059+0000","last_change":"2024-03-06T07:24:19.512059+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.512059+0000","last_undegraded":"2024-03-06T07:24:19.512059+0000","last_fullsized":"2024-03-06T07:24:19.512059+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.511672+0000","last_change":"2024-03-06T07:24:19.511672+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.511672+0000","last_undegraded":"2024-03-06T07:24:19.511672+0000","last_fullsized":"2024-03-06T07:24:19.511672+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.519288+0000","last_change":"2024-03-06T07:24:19.519288+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.519288+0000","last_undegraded":"2024-03-06T07:24:19.519288+0000","last_fullsized":"2024-03-06T07:24:19.519288+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.522695+0000","last_change":"2024-03-06T07:24:19.522695+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.522695+0000","last_undegraded":"2024-03-06T07:24:19.522695+0000","last_fullsized":"2024-03-06T07:24:19.522695+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508661+0000","last_change":"2024-03-06T07:24:19.508661+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508661+0000","last_undegraded":"2024-03-06T07:24:19.508661+0000","last_fullsized":"2024-03-06T07:24:19.508661+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.514373+0000","last_change":"2024-03-06T07:24:19.514373+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.514373+0000","last_undegraded":"2024-03-06T07:24:19.514373+0000","last_fullsized":"2024-03-06T07:24:19.514373+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.522340+0000","last_change":"2024-03-06T07:24:19.522340+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.522340+0000","last_undegraded":"2024-03-06T07:24:19.522340+0000","last_fullsized":"2024-03-06T07:24:19.522340+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.513618+0000","last_change":"2024-03-06T07:24:19.513618+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.513618+0000","last_undegraded":"2024-03-06T07:24:19.513618+0000","last_fullsized":"2024-03-06T07:24:19.513618+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.521571+0000","last_change":"2024-03-06T07:24:19.521571+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.521571+0000","last_undegraded":"2024-03-06T07:24:19.521571+0000","last_fullsized":"2024-03-06T07:24:19.521571+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.509993+0000","last_change":"2024-03-06T07:24:19.509993+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.509993+0000","last_undegraded":"2024-03-06T07:24:19.509993+0000","last_fullsized":"2024-03-06T07:24:19.509993+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.514668+0000","last_change":"2024-03-06T07:24:19.514668+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.514668+0000","last_undegraded":"2024-03-06T07:24:19.514668+0000","last_fullsized":"2024-03-06T07:24:19.514668+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.514380+0000","last_change":"2024-03-06T07:24:19.514380+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.514380+0000","last_undegraded":"2024-03-06T07:24:19.514380+0000","last_fullsized":"2024-03-06T07:24:19.514380+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.509876+0000","last_change":"2024-03-06T07:24:19.509876+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.509876+0000","last_undegraded":"2024-03-06T07:24:19.509876+0000","last_fullsized":"2024-03-06T07:24:19.509876+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.520572+0000","last_change":"2024-03-06T07:24:19.520572+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.520572+0000","last_undegraded":"2024-03-06T07:24:19.520572+0000","last_fullsized":"2024-03-06T07:24:19.520572+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.515332+0000","last_change":"2024-03-06T07:24:19.515332+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.515332+0000","last_undegraded":"2024-03-06T07:24:19.515332+0000","last_fullsized":"2024-03-06T07:24:19.515332+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.525727+0000","last_change":"2024-03-06T07:24:19.525727+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.525727+0000","last_undegraded":"2024-03-06T07:24:19.525727+0000","last_fullsized":"2024-03-06T07:24:19.525727+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508941+0000","last_change":"2024-03-06T07:24:19.508941+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508941+0000","last_undegraded":"2024-03-06T07:24:19.508941+0000","last_fullsized":"2024-03-06T07:24:19.508941+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.513795+0000","last_change":"2024-03-06T07:24:19.513795+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.513795+0000","last_undegraded":"2024-03-06T07:24:19.513795+0000","last_fullsized":"2024-03-06T07:24:19.513795+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.518693+0000","last_change":"2024-03-06T07:24:19.518693+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.518693+0000","last_undegraded":"2024-03-06T07:24:19.518693+0000","last_fullsized":"2024-03-06T07:24:19.518693+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.518564+0000","last_change":"2024-03-06T07:24:19.518564+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.518564+0000","last_undegraded":"2024-03-06T07:24:19.518564+0000","last_fullsized":"2024-03-06T07:24:19.518564+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.525975+0000","last_change":"2024-03-06T07:24:19.523037+0000","last_active":"2024-03-06T07:24:19.525975+0000","last_peered":"2024-03-06T07:24:19.525975+0000","last_clean":"2024-03-06T07:24:19.525975+0000","last_became_active":"2024-03-06T07:24:19.522767+0000","last_became_peered":"2024-03-06T07:24:19.522767+0000","last_unstale":"2024-03-06T07:24:19.525975+0000","last_undegraded":"2024-03-06T07:24:19.525975+0000","last_fullsized":"2024-03-06T07:24:19.525975+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:42:52.076503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.515662+0000","last_change":"2024-03-06T07:24:19.515662+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.515662+0000","last_undegraded":"2024-03-06T07:24:19.515662+0000","last_fullsized":"2024-03-06T07:24:19.515662+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.517806+0000","last_change":"2024-03-06T07:24:19.516320+0000","last_active":"2024-03-06T07:24:19.517806+0000","last_peered":"2024-03-06T07:24:19.517806+0000","last_clean":"2024-03-06T07:24:19.517806+0000","last_became_active":"2024-03-06T07:24:19.516067+0000","last_became_peered":"2024-03-06T07:24:19.516067+0000","last_unstale":"2024-03-06T07:24:19.517806+0000","last_undegraded":"2024-03-06T07:24:19.517806+0000","last_fullsized":"2024-03-06T07:24:19.517806+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:44:06.313408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.512557+0000","last_change":"2024-03-06T07:24:19.512557+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.512557+0000","last_undegraded":"2024-03-06T07:24:19.512557+0000","last_fullsized":"2024-03-06T07:24:19.512557+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.530956+0000","last_change":"2024-03-06T07:24:19.528754+0000","last_active":"2024-03-06T07:24:19.530956+0000","last_peered":"2024-03-06T07:24:19.530956+0000","last_clean":"2024-03-06T07:24:19.530956+0000","last_became_active":"2024-03-06T07:24:19.528479+0000","last_became_peered":"2024-03-06T07:24:19.528479+0000","last_unstale":"2024-03-06T07:24:19.530956+0000","last_undegraded":"2024-03-06T07:24:19.530956+0000","last_fullsized":"2024-03-06T07:24:19.530956+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:17.842390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.513655+0000","last_change":"2024-03-06T07:24:19.513655+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.513655+0000","last_undegraded":"2024-03-06T07:24:19.513655+0000","last_fullsized":"2024-03-06T07:24:19.513655+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.524698+0000","last_change":"2024-03-06T07:24:19.520169+0000","last_active":"2024-03-06T07:24:19.524698+0000","last_peered":"2024-03-06T07:24:19.524698+0000","last_clean":"2024-03-06T07:24:19.524698+0000","last_became_active":"2024-03-06T07:24:19.519886+0000","last_became_peered":"2024-03-06T07:24:19.519886+0000","last_unstale":"2024-03-06T07:24:19.524698+0000","last_undegraded":"2024-03-06T07:24:19.524698+0000","last_fullsized":"2024-03-06T07:24:19.524698+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:38.333961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.520743+0000","last_change":"2024-03-06T07:24:19.520743+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.520743+0000","last_undegraded":"2024-03-06T07:24:19.520743+0000","last_fullsized":"2024-03-06T07:24:19.520743+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.525372+0000","last_change":"2024-03-06T07:24:19.524214+0000","last_active":"2024-03-06T07:24:19.525372+0000","last_peered":"2024-03-06T07:24:19.525372+0000","last_clean":"2024-03-06T07:24:19.525372+0000","last_became_active":"2024-03-06T07:24:19.523589+0000","last_became_peered":"2024-03-06T07:24:19.523589+0000","last_unstale":"2024-03-06T07:24:19.525372+0000","last_undegraded":"2024-03-06T07:24:19.525372+0000","last_fullsized":"2024-03-06T07:24:19.525372+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:00.115699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.521187+0000","last_change":"2024-03-06T07:24:19.521187+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.521187+0000","last_undegraded":"2024-03-06T07:24:19.521187+0000","last_fullsized":"2024-03-06T07:24:19.521187+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.524818+0000","last_change":"2024-03-06T07:24:19.521026+0000","last_active":"2024-03-06T07:24:19.524818+0000","last_peered":"2024-03-06T07:24:19.524818+0000","last_clean":"2024-03-06T07:24:19.524818+0000","last_became_active":"2024-03-06T07:24:19.520754+0000","last_became_peered":"2024-03-06T07:24:19.520754+0000","last_unstale":"2024-03-06T07:24:19.524818+0000","last_undegraded":"2024-03-06T07:24:19.524818+0000","last_fullsized":"2024-03-06T07:24:19.524818+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:23.502992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.514185+0000","last_change":"2024-03-06T07:24:19.514185+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.514185+0000","last_undegraded":"2024-03-06T07:24:19.514185+0000","last_fullsized":"2024-03-06T07:24:19.514185+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.521399+0000","last_change":"2024-03-06T07:24:19.516459+0000","last_active":"2024-03-06T07:24:19.521399+0000","last_peered":"2024-03-06T07:24:19.521399+0000","last_clean":"2024-03-06T07:24:19.521399+0000","last_became_active":"2024-03-06T07:24:19.516181+0000","last_became_peered":"2024-03-06T07:24:19.516181+0000","last_unstale":"2024-03-06T07:24:19.521399+0000","last_undegraded":"2024-03-06T07:24:19.521399+0000","last_fullsized":"2024-03-06T07:24:19.521399+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:08.665102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.520302+0000","last_change":"2024-03-06T07:24:19.520302+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.520302+0000","last_undegraded":"2024-03-06T07:24:19.520302+0000","last_fullsized":"2024-03-06T07:24:19.520302+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.524364+0000","last_change":"2024-03-06T07:24:19.518659+0000","last_active":"2024-03-06T07:24:19.524364+0000","last_peered":"2024-03-06T07:24:19.524364+0000","last_clean":"2024-03-06T07:24:19.524364+0000","last_became_active":"2024-03-06T07:24:19.517834+0000","last_became_peered":"2024-03-06T07:24:19.517834+0000","last_unstale":"2024-03-06T07:24:19.524364+0000","last_undegraded":"2024-03-06T07:24:19.524364+0000","last_fullsized":"2024-03-06T07:24:19.524364+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:09.323969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.511187+0000","last_change":"2024-03-06T07:24:19.511187+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.511187+0000","last_undegraded":"2024-03-06T07:24:19.511187+0000","last_fullsized":"2024-03-06T07:24:19.511187+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.529051+0000","last_change":"2024-03-06T07:24:19.528439+0000","last_active":"2024-03-06T07:24:19.529051+0000","last_peered":"2024-03-06T07:24:19.529051+0000","last_clean":"2024-03-06T07:24:19.529051+0000","last_became_active":"2024-03-06T07:24:19.528141+0000","last_became_peered":"2024-03-06T07:24:19.528141+0000","last_unstale":"2024-03-06T07:24:19.529051+0000","last_undegraded":"2024-03-06T07:24:19.529051+0000","last_fullsized":"2024-03-06T07:24:19.529051+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:24:36.321071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508569+0000","last_change":"2024-03-06T07:24:19.508569+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508569+0000","last_undegraded":"2024-03-06T07:24:19.508569+0000","last_fullsized":"2024-03-06T07:24:19.508569+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.526823+0000","last_change":"2024-03-06T07:24:19.519750+0000","last_active":"2024-03-06T07:24:19.526823+0000","last_peered":"2024-03-06T07:24:19.526823+0000","last_clean":"2024-03-06T07:24:19.526823+0000","last_became_active":"2024-03-06T07:24:19.519503+0000","last_became_peered":"2024-03-06T07:24:19.519503+0000","last_unstale":"2024-03-06T07:24:19.526823+0000","last_undegraded":"2024-03-06T07:24:19.526823+0000","last_fullsized":"2024-03-06T07:24:19.526823+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:00:04.275748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.514274+0000","last_change":"2024-03-06T07:24:19.514274+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.514274+0000","last_undegraded":"2024-03-06T07:24:19.514274+0000","last_fullsized":"2024-03-06T07:24:19.514274+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.528856+0000","last_change":"2024-03-06T07:24:19.528356+0000","last_active":"2024-03-06T07:24:19.528856+0000","last_peered":"2024-03-06T07:24:19.528856+0000","last_clean":"2024-03-06T07:24:19.528856+0000","last_became_active":"2024-03-06T07:24:19.527954+0000","last_became_peered":"2024-03-06T07:24:19.527954+0000","last_unstale":"2024-03-06T07:24:19.528856+0000","last_undegraded":"2024-03-06T07:24:19.528856+0000","last_fullsized":"2024-03-06T07:24:19.528856+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:22.158292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.512986+0000","last_change":"2024-03-06T07:24:19.512986+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.512986+0000","last_undegraded":"2024-03-06T07:24:19.512986+0000","last_fullsized":"2024-03-06T07:24:19.512986+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.525242+0000","last_change":"2024-03-06T07:24:19.523322+0000","last_active":"2024-03-06T07:24:19.525242+0000","last_peered":"2024-03-06T07:24:19.525242+0000","last_clean":"2024-03-06T07:24:19.525242+0000","last_became_active":"2024-03-06T07:24:19.523034+0000","last_became_peered":"2024-03-06T07:24:19.523034+0000","last_unstale":"2024-03-06T07:24:19.525242+0000","last_undegraded":"2024-03-06T07:24:19.525242+0000","last_fullsized":"2024-03-06T07:24:19.525242+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:43:07.141534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508951+0000","last_change":"2024-03-06T07:24:19.508951+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508951+0000","last_undegraded":"2024-03-06T07:24:19.508951+0000","last_fullsized":"2024-03-06T07:24:19.508951+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.530266+0000","last_change":"2024-03-06T07:24:19.526669+0000","last_active":"2024-03-06T07:24:19.530266+0000","last_peered":"2024-03-06T07:24:19.530266+0000","last_clean":"2024-03-06T07:24:19.530266+0000","last_became_active":"2024-03-06T07:24:19.526276+0000","last_became_peered":"2024-03-06T07:24:19.526276+0000","last_unstale":"2024-03-06T07:24:19.530266+0000","last_undegraded":"2024-03-06T07:24:19.530266+0000","last_fullsized":"2024-03-06T07:24:19.530266+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:32:50.322407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.510711+0000","last_change":"2024-03-06T07:24:19.510711+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.510711+0000","last_undegraded":"2024-03-06T07:24:19.510711+0000","last_fullsized":"2024-03-06T07:24:19.510711+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.520976+0000","last_change":"2024-03-06T07:24:19.520321+0000","last_active":"2024-03-06T07:24:19.520976+0000","last_peered":"2024-03-06T07:24:19.520976+0000","last_clean":"2024-03-06T07:24:19.520976+0000","last_became_active":"2024-03-06T07:24:19.520045+0000","last_became_peered":"2024-03-06T07:24:19.520045+0000","last_unstale":"2024-03-06T07:24:19.520976+0000","last_undegraded":"2024-03-06T07:24:19.520976+0000","last_fullsized":"2024-03-06T07:24:19.520976+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:01.534402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.511628+0000","last_change":"2024-03-06T07:24:19.511628+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.511628+0000","last_undegraded":"2024-03-06T07:24:19.511628+0000","last_fullsized":"2024-03-06T07:24:19.511628+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.524716+0000","last_change":"2024-03-06T07:24:19.520152+0000","last_active":"2024-03-06T07:24:19.524716+0000","last_peered":"2024-03-06T07:24:19.524716+0000","last_clean":"2024-03-06T07:24:19.524716+0000","last_became_active":"2024-03-06T07:24:19.519855+0000","last_became_peered":"2024-03-06T07:24:19.519855+0000","last_unstale":"2024-03-06T07:24:19.524716+0000","last_undegraded":"2024-03-06T07:24:19.524716+0000","last_fullsized":"2024-03-06T07:24:19.524716+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:20.649547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.509270+0000","last_change":"2024-03-06T07:24:19.509270+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.509270+0000","last_undegraded":"2024-03-06T07:24:19.509270+0000","last_fullsized":"2024-03-06T07:24:19.509270+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.525829+0000","last_change":"2024-03-06T07:24:19.525401+0000","last_active":"2024-03-06T07:24:19.525829+0000","last_peered":"2024-03-06T07:24:19.525829+0000","last_clean":"2024-03-06T07:24:19.525829+0000","last_became_active":"2024-03-06T07:24:19.525064+0000","last_became_peered":"2024-03-06T07:24:19.525064+0000","last_unstale":"2024-03-06T07:24:19.525829+0000","last_undegraded":"2024-03-06T07:24:19.525829+0000","last_fullsized":"2024-03-06T07:24:19.525829+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:38.333961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508079+0000","last_change":"2024-03-06T07:24:19.508079+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508079+0000","last_undegraded":"2024-03-06T07:24:19.508079+0000","last_fullsized":"2024-03-06T07:24:19.508079+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.527834+0000","last_change":"2024-03-06T07:24:19.521974+0000","last_active":"2024-03-06T07:24:19.527834+0000","last_peered":"2024-03-06T07:24:19.527834+0000","last_clean":"2024-03-06T07:24:19.527834+0000","last_became_active":"2024-03-06T07:24:19.521736+0000","last_became_peered":"2024-03-06T07:24:19.521736+0000","last_unstale":"2024-03-06T07:24:19.527834+0000","last_undegraded":"2024-03-06T07:24:19.527834+0000","last_fullsized":"2024-03-06T07:24:19.527834+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:20:20.174477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508869+0000","last_change":"2024-03-06T07:24:19.508869+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508869+0000","last_undegraded":"2024-03-06T07:24:19.508869+0000","last_fullsized":"2024-03-06T07:24:19.508869+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.525378+0000","last_change":"2024-03-06T07:24:19.523871+0000","last_active":"2024-03-06T07:24:19.525378+0000","last_peered":"2024-03-06T07:24:19.525378+0000","last_clean":"2024-03-06T07:24:19.525378+0000","last_became_active":"2024-03-06T07:24:19.523587+0000","last_became_peered":"2024-03-06T07:24:19.523587+0000","last_unstale":"2024-03-06T07:24:19.525378+0000","last_undegraded":"2024-03-06T07:24:19.525378+0000","last_fullsized":"2024-03-06T07:24:19.525378+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:16.014585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.520446+0000","last_change":"2024-03-06T07:24:19.520446+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.520446+0000","last_undegraded":"2024-03-06T07:24:19.520446+0000","last_fullsized":"2024-03-06T07:24:19.520446+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.524688+0000","last_change":"2024-03-06T07:24:19.520212+0000","last_active":"2024-03-06T07:24:19.524688+0000","last_peered":"2024-03-06T07:24:19.524688+0000","last_clean":"2024-03-06T07:24:19.524688+0000","last_became_active":"2024-03-06T07:24:19.519913+0000","last_became_peered":"2024-03-06T07:24:19.519913+0000","last_unstale":"2024-03-06T07:24:19.524688+0000","last_undegraded":"2024-03-06T07:24:19.524688+0000","last_fullsized":"2024-03-06T07:24:19.524688+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:22.158292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508125+0000","last_change":"2024-03-06T07:24:19.508125+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508125+0000","last_undegraded":"2024-03-06T07:24:19.508125+0000","last_fullsized":"2024-03-06T07:24:19.508125+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.524670+0000","last_change":"2024-03-06T07:24:19.520035+0000","last_active":"2024-03-06T07:24:19.524670+0000","last_peered":"2024-03-06T07:24:19.524670+0000","last_clean":"2024-03-06T07:24:19.524670+0000","last_became_active":"2024-03-06T07:24:19.519761+0000","last_became_peered":"2024-03-06T07:24:19.519761+0000","last_unstale":"2024-03-06T07:24:19.524670+0000","last_undegraded":"2024-03-06T07:24:19.524670+0000","last_fullsized":"2024-03-06T07:24:19.524670+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:45.374706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508234+0000","last_change":"2024-03-06T07:24:19.508234+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508234+0000","last_undegraded":"2024-03-06T07:24:19.508234+0000","last_fullsized":"2024-03-06T07:24:19.508234+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.517678+0000","last_change":"2024-03-06T07:24:19.515544+0000","last_active":"2024-03-06T07:24:19.517678+0000","last_peered":"2024-03-06T07:24:19.517678+0000","last_clean":"2024-03-06T07:24:19.517678+0000","last_became_active":"2024-03-06T07:24:19.515266+0000","last_became_peered":"2024-03-06T07:24:19.515266+0000","last_unstale":"2024-03-06T07:24:19.517678+0000","last_undegraded":"2024-03-06T07:24:19.517678+0000","last_fullsized":"2024-03-06T07:24:19.517678+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:25:58.500915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.512821+0000","last_change":"2024-03-06T07:24:19.512821+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.512821+0000","last_undegraded":"2024-03-06T07:24:19.512821+0000","last_fullsized":"2024-03-06T07:24:19.512821+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.527496+0000","last_change":"2024-03-06T07:24:19.521024+0000","last_active":"2024-03-06T07:24:19.527496+0000","last_peered":"2024-03-06T07:24:19.527496+0000","last_clean":"2024-03-06T07:24:19.527496+0000","last_became_active":"2024-03-06T07:24:19.520754+0000","last_became_peered":"2024-03-06T07:24:19.520754+0000","last_unstale":"2024-03-06T07:24:19.527496+0000","last_undegraded":"2024-03-06T07:24:19.527496+0000","last_fullsized":"2024-03-06T07:24:19.527496+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:38.661204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.512496+0000","last_change":"2024-03-06T07:24:19.512496+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.512496+0000","last_undegraded":"2024-03-06T07:24:19.512496+0000","last_fullsized":"2024-03-06T07:24:19.512496+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.524611+0000","last_change":"2024-03-06T07:24:19.519653+0000","last_active":"2024-03-06T07:24:19.524611+0000","last_peered":"2024-03-06T07:24:19.524611+0000","last_clean":"2024-03-06T07:24:19.524611+0000","last_became_active":"2024-03-06T07:24:19.519374+0000","last_became_peered":"2024-03-06T07:24:19.519374+0000","last_unstale":"2024-03-06T07:24:19.524611+0000","last_undegraded":"2024-03-06T07:24:19.524611+0000","last_fullsized":"2024-03-06T07:24:19.524611+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:02:35.379525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508772+0000","last_change":"2024-03-06T07:24:19.508772+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508772+0000","last_undegraded":"2024-03-06T07:24:19.508772+0000","last_fullsized":"2024-03-06T07:24:19.508772+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.527636+0000","last_change":"2024-03-06T07:24:19.527152+0000","last_active":"2024-03-06T07:24:19.527636+0000","last_peered":"2024-03-06T07:24:19.527636+0000","last_clean":"2024-03-06T07:24:19.527636+0000","last_became_active":"2024-03-06T07:24:19.526879+0000","last_became_peered":"2024-03-06T07:24:19.526879+0000","last_unstale":"2024-03-06T07:24:19.527636+0000","last_undegraded":"2024-03-06T07:24:19.527636+0000","last_fullsized":"2024-03-06T07:24:19.527636+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:01.534402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508386+0000","last_change":"2024-03-06T07:24:19.508386+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508386+0000","last_undegraded":"2024-03-06T07:24:19.508386+0000","last_fullsized":"2024-03-06T07:24:19.508386+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.524915+0000","last_change":"2024-03-06T07:24:19.520942+0000","last_active":"2024-03-06T07:24:19.524915+0000","last_peered":"2024-03-06T07:24:19.524915+0000","last_clean":"2024-03-06T07:24:19.524915+0000","last_became_active":"2024-03-06T07:24:19.520672+0000","last_became_peered":"2024-03-06T07:24:19.520672+0000","last_unstale":"2024-03-06T07:24:19.524915+0000","last_undegraded":"2024-03-06T07:24:19.524915+0000","last_fullsized":"2024-03-06T07:24:19.524915+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:15.049439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.513801+0000","last_change":"2024-03-06T07:24:19.513801+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.513801+0000","last_undegraded":"2024-03-06T07:24:19.513801+0000","last_fullsized":"2024-03-06T07:24:19.513801+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.524240+0000","last_change":"2024-03-06T07:24:19.522226+0000","last_active":"2024-03-06T07:24:19.524240+0000","last_peered":"2024-03-06T07:24:19.524240+0000","last_clean":"2024-03-06T07:24:19.524240+0000","last_became_active":"2024-03-06T07:24:19.521882+0000","last_became_peered":"2024-03-06T07:24:19.521882+0000","last_unstale":"2024-03-06T07:24:19.524240+0000","last_undegraded":"2024-03-06T07:24:19.524240+0000","last_fullsized":"2024-03-06T07:24:19.524240+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:24.269083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.513191+0000","last_change":"2024-03-06T07:24:19.513191+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.513191+0000","last_undegraded":"2024-03-06T07:24:19.513191+0000","last_fullsized":"2024-03-06T07:24:19.513191+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.521727+0000","last_change":"2024-03-06T07:24:19.516436+0000","last_active":"2024-03-06T07:24:19.521727+0000","last_peered":"2024-03-06T07:24:19.521727+0000","last_clean":"2024-03-06T07:24:19.521727+0000","last_became_active":"2024-03-06T07:24:19.515998+0000","last_became_peered":"2024-03-06T07:24:19.515998+0000","last_unstale":"2024-03-06T07:24:19.521727+0000","last_undegraded":"2024-03-06T07:24:19.521727+0000","last_fullsized":"2024-03-06T07:24:19.521727+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:09.161121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.519430+0000","last_change":"2024-03-06T07:24:19.519430+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.519430+0000","last_undegraded":"2024-03-06T07:24:19.519430+0000","last_fullsized":"2024-03-06T07:24:19.519430+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.527547+0000","last_change":"2024-03-06T07:24:19.521792+0000","last_active":"2024-03-06T07:24:19.527547+0000","last_peered":"2024-03-06T07:24:19.527547+0000","last_clean":"2024-03-06T07:24:19.527547+0000","last_became_active":"2024-03-06T07:24:19.521565+0000","last_became_peered":"2024-03-06T07:24:19.521565+0000","last_unstale":"2024-03-06T07:24:19.527547+0000","last_undegraded":"2024-03-06T07:24:19.527547+0000","last_fullsized":"2024-03-06T07:24:19.527547+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:02.649347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508067+0000","last_change":"2024-03-06T07:24:19.508067+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508067+0000","last_undegraded":"2024-03-06T07:24:19.508067+0000","last_fullsized":"2024-03-06T07:24:19.508067+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.525392+0000","last_change":"2024-03-06T07:24:19.524110+0000","last_active":"2024-03-06T07:24:19.525392+0000","last_peered":"2024-03-06T07:24:19.525392+0000","last_clean":"2024-03-06T07:24:19.525392+0000","last_became_active":"2024-03-06T07:24:19.523812+0000","last_became_peered":"2024-03-06T07:24:19.523812+0000","last_unstale":"2024-03-06T07:24:19.525392+0000","last_undegraded":"2024-03-06T07:24:19.525392+0000","last_fullsized":"2024-03-06T07:24:19.525392+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:22:22.348426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.513750+0000","last_change":"2024-03-06T07:24:19.513750+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.513750+0000","last_undegraded":"2024-03-06T07:24:19.513750+0000","last_fullsized":"2024-03-06T07:24:19.513750+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.527467+0000","last_change":"2024-03-06T07:24:19.520220+0000","last_active":"2024-03-06T07:24:19.527467+0000","last_peered":"2024-03-06T07:24:19.527467+0000","last_clean":"2024-03-06T07:24:19.527467+0000","last_became_active":"2024-03-06T07:24:19.519962+0000","last_became_peered":"2024-03-06T07:24:19.519962+0000","last_unstale":"2024-03-06T07:24:19.527467+0000","last_undegraded":"2024-03-06T07:24:19.527467+0000","last_fullsized":"2024-03-06T07:24:19.527467+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:36:34.165775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.518980+0000","last_change":"2024-03-06T07:24:19.518980+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.518980+0000","last_undegraded":"2024-03-06T07:24:19.518980+0000","last_fullsized":"2024-03-06T07:24:19.518980+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.521731+0000","last_change":"2024-03-06T07:24:19.516940+0000","last_active":"2024-03-06T07:24:19.521731+0000","last_peered":"2024-03-06T07:24:19.521731+0000","last_clean":"2024-03-06T07:24:19.521731+0000","last_became_active":"2024-03-06T07:24:19.516691+0000","last_became_peered":"2024-03-06T07:24:19.516691+0000","last_unstale":"2024-03-06T07:24:19.521731+0000","last_undegraded":"2024-03-06T07:24:19.521731+0000","last_fullsized":"2024-03-06T07:24:19.521731+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:19.361223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.519861+0000","last_change":"2024-03-06T07:24:19.519861+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.519861+0000","last_undegraded":"2024-03-06T07:24:19.519861+0000","last_fullsized":"2024-03-06T07:24:19.519861+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.524300+0000","last_change":"2024-03-06T07:24:19.522294+0000","last_active":"2024-03-06T07:24:19.524300+0000","last_peered":"2024-03-06T07:24:19.524300+0000","last_clean":"2024-03-06T07:24:19.524300+0000","last_became_active":"2024-03-06T07:24:19.521845+0000","last_became_peered":"2024-03-06T07:24:19.521845+0000","last_unstale":"2024-03-06T07:24:19.524300+0000","last_undegraded":"2024-03-06T07:24:19.524300+0000","last_fullsized":"2024-03-06T07:24:19.524300+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:00:32.355340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":232,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.516321+0000","last_change":"2024-03-06T07:21:47.553693+0000","last_active":"2024-03-06T07:24:19.516321+0000","last_peered":"2024-03-06T07:24:19.516321+0000","last_clean":"2024-03-06T07:24:19.516321+0000","last_became_active":"2024-03-06T07:21:47.552664+0000","last_became_peered":"2024-03-06T07:21:47.552664+0000","last_unstale":"2024-03-06T07:24:19.516321+0000","last_undegraded":"2024-03-06T07:24:19.516321+0000","last_fullsized":"2024-03-06T07:24:19.516321+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_clean_scrub_stamp":"2024-03-06T07:21:46.362028+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:02.916426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32412,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339428,"statfs":{"total":96636764160,"available":96603574272,"internally_reserved":0,"allocated":1253376,"data_stored":861417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574882,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31828,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340012,"statfs":{"total":96636764160,"available":96604172288,"internally_reserved":0,"allocated":659456,"data_stored":271049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31832,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340008,"statfs":{"total":96636764160,"available":96604168192,"internally_reserved":0,"allocated":659456,"data_stored":271049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574884,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31812,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340028,"statfs":{"total":96636764160,"available":96604188672,"internally_reserved":0,"allocated":647168,"data_stored":264782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574883,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31816,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340024,"statfs":{"total":96636764160,"available":96604184576,"internally_reserved":0,"allocated":647168,"data_stored":264782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31832,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340008,"statfs":{"total":96636764160,"available":96604168192,"internally_reserved":0,"allocated":659456,"data_stored":271049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31828,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340012,"statfs":{"total":96636764160,"available":96604172288,"internally_reserved":0,"allocated":659456,"data_stored":271049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574883,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32404,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339436,"statfs":{"total":96636764160,"available":96603582464,"internally_reserved":0,"allocated":1253376,"data_stored":861417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:23.922 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:24.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.058+0000 7f1646f1e640 1 Processor -- start 2024-03-06T07:24:24.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.058+0000 7f1646f1e640 1 -- start start 2024-03-06T07:24:24.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.059+0000 7f1646f1e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f16400d22b0 0x7f16400d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:24.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.059+0000 7f1646f1e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16400b41f0 0x7f16400b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:24.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.059+0000 7f1646f1e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16400b3430 con 0x7f164015dfe0 2024-03-06T07:24:24.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.059+0000 7f1646f1e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16400d2c50 con 0x7f16400d22b0 2024-03-06T07:24:24.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.059+0000 7f1646f1e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16400b31e0 con 0x7f16400b41f0 2024-03-06T07:24:24.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.059+0000 7f16454bb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16400b41f0 0x7f16400b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:24.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.059+0000 7f163ffff640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f16400d22b0 0x7f16400d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:24.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.059+0000 7f16454bb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16400b41f0 0x7f16400b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:36764/0 (socket says 172.21.15.83:36764) 2024-03-06T07:24:24.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.059+0000 7f163ffff640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f16400d22b0 0x7f16400d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:60978/0 (socket says 172.21.15.83:60978) 2024-03-06T07:24:24.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.059+0000 7f16454bb640 1 -- 172.21.15.83:0/743840950 learned_addr learned my addr 172.21.15.83:0/743840950 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:24.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.059+0000 7f16454bb640 1 -- 172.21.15.83:0/743840950 >> v2:172.21.15.183:3300/0 conn(0x7f16400d22b0 msgr2=0x7f16400d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:24.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.059+0000 7f16454bb640 1 --2- 172.21.15.83:0/743840950 >> v2:172.21.15.183:3300/0 conn(0x7f16400d22b0 0x7f16400d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.059+0000 7f163f7fe640 1 -- 172.21.15.83:0/743840950 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1577259814 0 0) 0x7f16400b3430 con 0x7f164015dfe0 2024-03-06T07:24:24.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.060+0000 7f16454bb640 1 -- 172.21.15.83:0/743840950 >> v1:172.21.15.183:6789/0 conn(0x7f164015dfe0 legacy=0x7f16400cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:24.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.060+0000 7f16454bb640 1 -- 172.21.15.83:0/743840950 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16400b3710 con 0x7f16400b41f0 2024-03-06T07:24:24.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.061+0000 7f16454bb640 1 --2- 172.21.15.83:0/743840950 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16400b41f0 0x7f16400b4680 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1638009e50 tx=0x7f163802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=868b9189cfd3c3cc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:24.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.061+0000 7f163f7fe640 1 -- 172.21.15.83:0/743840950 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f163802cb00 con 0x7f16400b41f0 2024-03-06T07:24:24.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.061+0000 7f163f7fe640 1 -- 172.21.15.83:0/743840950 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f163802cca0 con 0x7f16400b41f0 2024-03-06T07:24:24.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.061+0000 7f163f7fe640 1 -- 172.21.15.83:0/743840950 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1638034610 con 0x7f16400b41f0 2024-03-06T07:24:24.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.061+0000 7f1646f1e640 1 -- 172.21.15.83:0/743840950 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16400b41f0 msgr2=0x7f16400b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:24.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.061+0000 7f1646f1e640 1 --2- 172.21.15.83:0/743840950 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16400b41f0 0x7f16400b4680 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1638009e50 tx=0x7f163802bf60 comp rx=0 tx=0).stop 2024-03-06T07:24:24.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.061+0000 7f1646f1e640 1 -- 172.21.15.83:0/743840950 shutdown_connections 2024-03-06T07:24:24.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.061+0000 7f1646f1e640 1 --2- 172.21.15.83:0/743840950 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16400b41f0 0x7f16400b4680 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.061+0000 7f1646f1e640 1 --2- 172.21.15.83:0/743840950 >> v2:172.21.15.183:3300/0 conn(0x7f16400d22b0 0x7f16400d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.061+0000 7f1646f1e640 1 -- 172.21.15.83:0/743840950 >> 172.21.15.83:0/743840950 conn(0x7f16400d3cd0 msgr2=0x7f16400d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:24.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.061+0000 7f1646f1e640 1 -- 172.21.15.83:0/743840950 shutdown_connections 2024-03-06T07:24:24.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.061+0000 7f1646f1e640 1 -- 172.21.15.83:0/743840950 wait complete. 2024-03-06T07:24:24.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.063+0000 7f1646f1e640 1 Processor -- start 2024-03-06T07:24:24.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.063+0000 7f1646f1e640 1 -- start start 2024-03-06T07:24:24.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.064+0000 7f1646f1e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16400b41f0 0x7f16400b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:24.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.064+0000 7f1646f1e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16400d22b0 0x7f16401588d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:24.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.064+0000 7f1646f1e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f164015dfe0 0x7f16401603d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:24.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.064+0000 7f1646f1e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16400cfa00 con 0x7f164015dfe0 2024-03-06T07:24:24.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.064+0000 7f1646f1e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16400b3d60 con 0x7f16400b41f0 2024-03-06T07:24:24.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.064+0000 7f1646f1e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16400d1c70 con 0x7f16400d22b0 2024-03-06T07:24:24.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.064+0000 7f16454bb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f164015dfe0 0x7f16401603d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:24.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.064+0000 7f16454bb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f164015dfe0 0x7f16401603d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40810/0 (socket says 172.21.15.83:40810) 2024-03-06T07:24:24.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.064+0000 7f16454bb640 1 -- 172.21.15.83:0/3884669222 learned_addr learned my addr 172.21.15.83:0/3884669222 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:24.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.065+0000 7f163ffff640 1 --2- 172.21.15.83:0/3884669222 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16400d22b0 0x7f16401588d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:24.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.065+0000 7f1644cba640 1 --2- 172.21.15.83:0/3884669222 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16400b41f0 0x7f16400b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:24.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.065+0000 7f16454bb640 1 -- 172.21.15.83:0/3884669222 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16400d22b0 msgr2=0x7f16401588d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:24.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.065+0000 7f16454bb640 1 --2- 172.21.15.83:0/3884669222 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16400d22b0 0x7f16401588d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.065+0000 7f16454bb640 1 -- 172.21.15.83:0/3884669222 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16400b41f0 msgr2=0x7f16400b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:24.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.065+0000 7f16454bb640 1 --2- 172.21.15.83:0/3884669222 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16400b41f0 0x7f16400b68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.065+0000 7f16454bb640 1 -- 172.21.15.83:0/3884669222 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1640158e10 con 0x7f164015dfe0 2024-03-06T07:24:24.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.065+0000 7f16454bb640 1 --2- 172.21.15.83:0/3884669222 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f164015dfe0 0x7f16401603d0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f1638009e50 tx=0x7f1638004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:24.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.065+0000 7f163d7fa640 1 -- 172.21.15.83:0/3884669222 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1638043020 con 0x7f164015dfe0 2024-03-06T07:24:24.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.065+0000 7f163d7fa640 1 -- 172.21.15.83:0/3884669222 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f163802ce50 con 0x7f164015dfe0 2024-03-06T07:24:24.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.065+0000 7f163d7fa640 1 -- 172.21.15.83:0/3884669222 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1638009510 con 0x7f164015dfe0 2024-03-06T07:24:24.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.065+0000 7f1646f1e640 1 -- 172.21.15.83:0/3884669222 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16400b71b0 con 0x7f164015dfe0 2024-03-06T07:24:24.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.065+0000 7f1646f1e640 1 -- 172.21.15.83:0/3884669222 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16401590e0 con 0x7f164015dfe0 2024-03-06T07:24:24.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.066+0000 7f1646f1e640 1 -- 172.21.15.83:0/3884669222 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f164004f330 con 0x7f164015dfe0 2024-03-06T07:24:24.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.067+0000 7f163d7fa640 1 -- 172.21.15.83:0/3884669222 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1638046020 con 0x7f164015dfe0 2024-03-06T07:24:24.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.067+0000 7f163d7fa640 1 --2- 172.21.15.83:0/3884669222 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f162406b4d0 0x7f162406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:24.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.067+0000 7f163d7fa640 1 -- 172.21.15.83:0/3884669222 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f16380ac610 con 0x7f164015dfe0 2024-03-06T07:24:24.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.067+0000 7f1644cba640 1 --2- 172.21.15.83:0/3884669222 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f162406b4d0 0x7f162406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:24.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.067+0000 7f1644cba640 1 --2- 172.21.15.83:0/3884669222 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f162406b4d0 0x7f162406d990 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f1634000f60 tx=0x7f1634009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:24.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.076+0000 7f163d7fa640 1 -- 172.21.15.83:0/3884669222 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f163804d030 con 0x7f164015dfe0 2024-03-06T07:24:24.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.250+0000 7f1646f1e640 1 -- 172.21.15.83:0/3884669222 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f16400d0e70 con 0x7f162406b4d0 2024-03-06T07:24:24.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.256+0000 7f163d7fa640 1 -- 172.21.15.83:0/3884669222 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246639 (secure 0 0 0) 0x7f16400d0e70 con 0x7f162406b4d0 2024-03-06T07:24:24.260 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:24:24.261 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-06T07:24:24.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.259+0000 7f1646f1e640 1 -- 172.21.15.83:0/3884669222 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f162406b4d0 msgr2=0x7f162406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:24.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.259+0000 7f1646f1e640 1 --2- 172.21.15.83:0/3884669222 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f162406b4d0 0x7f162406d990 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f1634000f60 tx=0x7f1634009040 comp rx=0 tx=0).stop 2024-03-06T07:24:24.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.259+0000 7f1646f1e640 1 -- 172.21.15.83:0/3884669222 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f164015dfe0 msgr2=0x7f16401603d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:24.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.259+0000 7f1646f1e640 1 --2- 172.21.15.83:0/3884669222 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f164015dfe0 0x7f16401603d0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f1638009e50 tx=0x7f1638004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:24.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.260+0000 7f1646f1e640 1 -- 172.21.15.83:0/3884669222 shutdown_connections 2024-03-06T07:24:24.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.260+0000 7f1646f1e640 1 --2- 172.21.15.83:0/3884669222 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f162406b4d0 0x7f162406d990 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.260+0000 7f1646f1e640 1 --2- 172.21.15.83:0/3884669222 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f164015dfe0 0x7f16401603d0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.260+0000 7f1646f1e640 1 --2- 172.21.15.83:0/3884669222 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16400d22b0 0x7f16401588d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.260+0000 7f1646f1e640 1 --2- 172.21.15.83:0/3884669222 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16400b41f0 0x7f16400b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.260+0000 7f1646f1e640 1 -- 172.21.15.83:0/3884669222 >> 172.21.15.83:0/3884669222 conn(0x7f16400d3cd0 msgr2=0x7f16400d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:24.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.260+0000 7f1646f1e640 1 -- 172.21.15.83:0/3884669222 shutdown_connections 2024-03-06T07:24:24.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.260+0000 7f1646f1e640 1 -- 172.21.15.83:0/3884669222 wait complete. 2024-03-06T07:24:24.272 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2024-03-06T07:24:22.347170+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":49,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":255764,"kb_used_data":6288,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754718956,"statfs":{"total":773094113280,"available":772832210944,"internally_reserved":0,"allocated":6438912,"data_stored":3336594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12757,"internal_metadata":255315499},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000578"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.514792+0000","last_change":"2024-03-06T07:24:19.514792+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.514792+0000","last_undegraded":"2024-03-06T07:24:19.514792+0000","last_fullsized":"2024-03-06T07:24:19.514792+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.524462+0000","last_change":"2024-03-06T07:24:19.524462+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.524462+0000","last_undegraded":"2024-03-06T07:24:19.524462+0000","last_fullsized":"2024-03-06T07:24:19.524462+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.513384+0000","last_change":"2024-03-06T07:24:19.513384+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.513384+0000","last_undegraded":"2024-03-06T07:24:19.513384+0000","last_fullsized":"2024-03-06T07:24:19.513384+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.510475+0000","last_change":"2024-03-06T07:24:19.510475+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.510475+0000","last_undegraded":"2024-03-06T07:24:19.510475+0000","last_fullsized":"2024-03-06T07:24:19.510475+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.518129+0000","last_change":"2024-03-06T07:24:19.518129+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.518129+0000","last_undegraded":"2024-03-06T07:24:19.518129+0000","last_fullsized":"2024-03-06T07:24:19.518129+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.518791+0000","last_change":"2024-03-06T07:24:19.518791+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.518791+0000","last_undegraded":"2024-03-06T07:24:19.518791+0000","last_fullsized":"2024-03-06T07:24:19.518791+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.512145+0000","last_change":"2024-03-06T07:24:19.512145+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.512145+0000","last_undegraded":"2024-03-06T07:24:19.512145+0000","last_fullsized":"2024-03-06T07:24:19.512145+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.523059+0000","last_change":"2024-03-06T07:24:19.523059+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.523059+0000","last_undegraded":"2024-03-06T07:24:19.523059+0000","last_fullsized":"2024-03-06T07:24:19.523059+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.510901+0000","last_change":"2024-03-06T07:24:19.510901+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.510901+0000","last_undegraded":"2024-03-06T07:24:19.510901+0000","last_fullsized":"2024-03-06T07:24:19.510901+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.515316+0000","last_change":"2024-03-06T07:24:19.515316+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.515316+0000","last_undegraded":"2024-03-06T07:24:19.515316+0000","last_fullsized":"2024-03-06T07:24:19.515316+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.516335+0000","last_change":"2024-03-06T07:24:19.516335+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.516335+0000","last_undegraded":"2024-03-06T07:24:19.516335+0000","last_fullsized":"2024-03-06T07:24:19.516335+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.510027+0000","last_change":"2024-03-06T07:24:19.510027+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.510027+0000","last_undegraded":"2024-03-06T07:24:19.510027+0000","last_fullsized":"2024-03-06T07:24:19.510027+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.515035+0000","last_change":"2024-03-06T07:24:19.515035+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.515035+0000","last_undegraded":"2024-03-06T07:24:19.515035+0000","last_fullsized":"2024-03-06T07:24:19.515035+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.512059+0000","last_change":"2024-03-06T07:24:19.512059+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.512059+0000","last_undegraded":"2024-03-06T07:24:19.512059+0000","last_fullsized":"2024-03-06T07:24:19.512059+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.511672+0000","last_change":"2024-03-06T07:24:19.511672+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.511672+0000","last_undegraded":"2024-03-06T07:24:19.511672+0000","last_fullsized":"2024-03-06T07:24:19.511672+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.519288+0000","last_change":"2024-03-06T07:24:19.519288+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.519288+0000","last_undegraded":"2024-03-06T07:24:19.519288+0000","last_fullsized":"2024-03-06T07:24:19.519288+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.522695+0000","last_change":"2024-03-06T07:24:19.522695+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.522695+0000","last_undegraded":"2024-03-06T07:24:19.522695+0000","last_fullsized":"2024-03-06T07:24:19.522695+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508661+0000","last_change":"2024-03-06T07:24:19.508661+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508661+0000","last_undegraded":"2024-03-06T07:24:19.508661+0000","last_fullsized":"2024-03-06T07:24:19.508661+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.514373+0000","last_change":"2024-03-06T07:24:19.514373+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.514373+0000","last_undegraded":"2024-03-06T07:24:19.514373+0000","last_fullsized":"2024-03-06T07:24:19.514373+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.522340+0000","last_change":"2024-03-06T07:24:19.522340+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.522340+0000","last_undegraded":"2024-03-06T07:24:19.522340+0000","last_fullsized":"2024-03-06T07:24:19.522340+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.513618+0000","last_change":"2024-03-06T07:24:19.513618+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.513618+0000","last_undegraded":"2024-03-06T07:24:19.513618+0000","last_fullsized":"2024-03-06T07:24:19.513618+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.521571+0000","last_change":"2024-03-06T07:24:19.521571+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.521571+0000","last_undegraded":"2024-03-06T07:24:19.521571+0000","last_fullsized":"2024-03-06T07:24:19.521571+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.509993+0000","last_change":"2024-03-06T07:24:19.509993+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.509993+0000","last_undegraded":"2024-03-06T07:24:19.509993+0000","last_fullsized":"2024-03-06T07:24:19.509993+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.514668+0000","last_change":"2024-03-06T07:24:19.514668+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.514668+0000","last_undegraded":"2024-03-06T07:24:19.514668+0000","last_fullsized":"2024-03-06T07:24:19.514668+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.514380+0000","last_change":"2024-03-06T07:24:19.514380+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.514380+0000","last_undegraded":"2024-03-06T07:24:19.514380+0000","last_fullsized":"2024-03-06T07:24:19.514380+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.509876+0000","last_change":"2024-03-06T07:24:19.509876+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.509876+0000","last_undegraded":"2024-03-06T07:24:19.509876+0000","last_fullsized":"2024-03-06T07:24:19.509876+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.520572+0000","last_change":"2024-03-06T07:24:19.520572+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.520572+0000","last_undegraded":"2024-03-06T07:24:19.520572+0000","last_fullsized":"2024-03-06T07:24:19.520572+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.515332+0000","last_change":"2024-03-06T07:24:19.515332+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.515332+0000","last_undegraded":"2024-03-06T07:24:19.515332+0000","last_fullsized":"2024-03-06T07:24:19.515332+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.525727+0000","last_change":"2024-03-06T07:24:19.525727+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.525727+0000","last_undegraded":"2024-03-06T07:24:19.525727+0000","last_fullsized":"2024-03-06T07:24:19.525727+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508941+0000","last_change":"2024-03-06T07:24:19.508941+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508941+0000","last_undegraded":"2024-03-06T07:24:19.508941+0000","last_fullsized":"2024-03-06T07:24:19.508941+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.513795+0000","last_change":"2024-03-06T07:24:19.513795+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.513795+0000","last_undegraded":"2024-03-06T07:24:19.513795+0000","last_fullsized":"2024-03-06T07:24:19.513795+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.518693+0000","last_change":"2024-03-06T07:24:19.518693+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.518693+0000","last_undegraded":"2024-03-06T07:24:19.518693+0000","last_fullsized":"2024-03-06T07:24:19.518693+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.518564+0000","last_change":"2024-03-06T07:24:19.518564+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.518564+0000","last_undegraded":"2024-03-06T07:24:19.518564+0000","last_fullsized":"2024-03-06T07:24:19.518564+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.525975+0000","last_change":"2024-03-06T07:24:19.523037+0000","last_active":"2024-03-06T07:24:19.525975+0000","last_peered":"2024-03-06T07:24:19.525975+0000","last_clean":"2024-03-06T07:24:19.525975+0000","last_became_active":"2024-03-06T07:24:19.522767+0000","last_became_peered":"2024-03-06T07:24:19.522767+0000","last_unstale":"2024-03-06T07:24:19.525975+0000","last_undegraded":"2024-03-06T07:24:19.525975+0000","last_fullsized":"2024-03-06T07:24:19.525975+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:42:52.076503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.515662+0000","last_change":"2024-03-06T07:24:19.515662+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.515662+0000","last_undegraded":"2024-03-06T07:24:19.515662+0000","last_fullsized":"2024-03-06T07:24:19.515662+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.517806+0000","last_change":"2024-03-06T07:24:19.516320+0000","last_active":"2024-03-06T07:24:19.517806+0000","last_peered":"2024-03-06T07:24:19.517806+0000","last_clean":"2024-03-06T07:24:19.517806+0000","last_became_active":"2024-03-06T07:24:19.516067+0000","last_became_peered":"2024-03-06T07:24:19.516067+0000","last_unstale":"2024-03-06T07:24:19.517806+0000","last_undegraded":"2024-03-06T07:24:19.517806+0000","last_fullsized":"2024-03-06T07:24:19.517806+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:44:06.313408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.512557+0000","last_change":"2024-03-06T07:24:19.512557+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.512557+0000","last_undegraded":"2024-03-06T07:24:19.512557+0000","last_fullsized":"2024-03-06T07:24:19.512557+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.530956+0000","last_change":"2024-03-06T07:24:19.528754+0000","last_active":"2024-03-06T07:24:19.530956+0000","last_peered":"2024-03-06T07:24:19.530956+0000","last_clean":"2024-03-06T07:24:19.530956+0000","last_became_active":"2024-03-06T07:24:19.528479+0000","last_became_peered":"2024-03-06T07:24:19.528479+0000","last_unstale":"2024-03-06T07:24:19.530956+0000","last_undegraded":"2024-03-06T07:24:19.530956+0000","last_fullsized":"2024-03-06T07:24:19.530956+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:17.842390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.513655+0000","last_change":"2024-03-06T07:24:19.513655+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.513655+0000","last_undegraded":"2024-03-06T07:24:19.513655+0000","last_fullsized":"2024-03-06T07:24:19.513655+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.524698+0000","last_change":"2024-03-06T07:24:19.520169+0000","last_active":"2024-03-06T07:24:19.524698+0000","last_peered":"2024-03-06T07:24:19.524698+0000","last_clean":"2024-03-06T07:24:19.524698+0000","last_became_active":"2024-03-06T07:24:19.519886+0000","last_became_peered":"2024-03-06T07:24:19.519886+0000","last_unstale":"2024-03-06T07:24:19.524698+0000","last_undegraded":"2024-03-06T07:24:19.524698+0000","last_fullsized":"2024-03-06T07:24:19.524698+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:38.333961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.520743+0000","last_change":"2024-03-06T07:24:19.520743+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.520743+0000","last_undegraded":"2024-03-06T07:24:19.520743+0000","last_fullsized":"2024-03-06T07:24:19.520743+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.525372+0000","last_change":"2024-03-06T07:24:19.524214+0000","last_active":"2024-03-06T07:24:19.525372+0000","last_peered":"2024-03-06T07:24:19.525372+0000","last_clean":"2024-03-06T07:24:19.525372+0000","last_became_active":"2024-03-06T07:24:19.523589+0000","last_became_peered":"2024-03-06T07:24:19.523589+0000","last_unstale":"2024-03-06T07:24:19.525372+0000","last_undegraded":"2024-03-06T07:24:19.525372+0000","last_fullsized":"2024-03-06T07:24:19.525372+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:00.115699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.521187+0000","last_change":"2024-03-06T07:24:19.521187+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.521187+0000","last_undegraded":"2024-03-06T07:24:19.521187+0000","last_fullsized":"2024-03-06T07:24:19.521187+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.524818+0000","last_change":"2024-03-06T07:24:19.521026+0000","last_active":"2024-03-06T07:24:19.524818+0000","last_peered":"2024-03-06T07:24:19.524818+0000","last_clean":"2024-03-06T07:24:19.524818+0000","last_became_active":"2024-03-06T07:24:19.520754+0000","last_became_peered":"2024-03-06T07:24:19.520754+0000","last_unstale":"2024-03-06T07:24:19.524818+0000","last_undegraded":"2024-03-06T07:24:19.524818+0000","last_fullsized":"2024-03-06T07:24:19.524818+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:23.502992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.514185+0000","last_change":"2024-03-06T07:24:19.514185+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.514185+0000","last_undegraded":"2024-03-06T07:24:19.514185+0000","last_fullsized":"2024-03-06T07:24:19.514185+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.521399+0000","last_change":"2024-03-06T07:24:19.516459+0000","last_active":"2024-03-06T07:24:19.521399+0000","last_peered":"2024-03-06T07:24:19.521399+0000","last_clean":"2024-03-06T07:24:19.521399+0000","last_became_active":"2024-03-06T07:24:19.516181+0000","last_became_peered":"2024-03-06T07:24:19.516181+0000","last_unstale":"2024-03-06T07:24:19.521399+0000","last_undegraded":"2024-03-06T07:24:19.521399+0000","last_fullsized":"2024-03-06T07:24:19.521399+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:08.665102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.520302+0000","last_change":"2024-03-06T07:24:19.520302+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.520302+0000","last_undegraded":"2024-03-06T07:24:19.520302+0000","last_fullsized":"2024-03-06T07:24:19.520302+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.524364+0000","last_change":"2024-03-06T07:24:19.518659+0000","last_active":"2024-03-06T07:24:19.524364+0000","last_peered":"2024-03-06T07:24:19.524364+0000","last_clean":"2024-03-06T07:24:19.524364+0000","last_became_active":"2024-03-06T07:24:19.517834+0000","last_became_peered":"2024-03-06T07:24:19.517834+0000","last_unstale":"2024-03-06T07:24:19.524364+0000","last_undegraded":"2024-03-06T07:24:19.524364+0000","last_fullsized":"2024-03-06T07:24:19.524364+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:09.323969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.511187+0000","last_change":"2024-03-06T07:24:19.511187+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.511187+0000","last_undegraded":"2024-03-06T07:24:19.511187+0000","last_fullsized":"2024-03-06T07:24:19.511187+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.529051+0000","last_change":"2024-03-06T07:24:19.528439+0000","last_active":"2024-03-06T07:24:19.529051+0000","last_peered":"2024-03-06T07:24:19.529051+0000","last_clean":"2024-03-06T07:24:19.529051+0000","last_became_active":"2024-03-06T07:24:19.528141+0000","last_became_peered":"2024-03-06T07:24:19.528141+0000","last_unstale":"2024-03-06T07:24:19.529051+0000","last_undegraded":"2024-03-06T07:24:19.529051+0000","last_fullsized":"2024-03-06T07:24:19.529051+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:24:36.321071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508569+0000","last_change":"2024-03-06T07:24:19.508569+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508569+0000","last_undegraded":"2024-03-06T07:24:19.508569+0000","last_fullsized":"2024-03-06T07:24:19.508569+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.526823+0000","last_change":"2024-03-06T07:24:19.519750+0000","last_active":"2024-03-06T07:24:19.526823+0000","last_peered":"2024-03-06T07:24:19.526823+0000","last_clean":"2024-03-06T07:24:19.526823+0000","last_became_active":"2024-03-06T07:24:19.519503+0000","last_became_peered":"2024-03-06T07:24:19.519503+0000","last_unstale":"2024-03-06T07:24:19.526823+0000","last_undegraded":"2024-03-06T07:24:19.526823+0000","last_fullsized":"2024-03-06T07:24:19.526823+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:00:04.275748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.514274+0000","last_change":"2024-03-06T07:24:19.514274+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.514274+0000","last_undegraded":"2024-03-06T07:24:19.514274+0000","last_fullsized":"2024-03-06T07:24:19.514274+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.528856+0000","last_change":"2024-03-06T07:24:19.528356+0000","last_active":"2024-03-06T07:24:19.528856+0000","last_peered":"2024-03-06T07:24:19.528856+0000","last_clean":"2024-03-06T07:24:19.528856+0000","last_became_active":"2024-03-06T07:24:19.527954+0000","last_became_peered":"2024-03-06T07:24:19.527954+0000","last_unstale":"2024-03-06T07:24:19.528856+0000","last_undegraded":"2024-03-06T07:24:19.528856+0000","last_fullsized":"2024-03-06T07:24:19.528856+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:22.158292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.512986+0000","last_change":"2024-03-06T07:24:19.512986+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.512986+0000","last_undegraded":"2024-03-06T07:24:19.512986+0000","last_fullsized":"2024-03-06T07:24:19.512986+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.525242+0000","last_change":"2024-03-06T07:24:19.523322+0000","last_active":"2024-03-06T07:24:19.525242+0000","last_peered":"2024-03-06T07:24:19.525242+0000","last_clean":"2024-03-06T07:24:19.525242+0000","last_became_active":"2024-03-06T07:24:19.523034+0000","last_became_peered":"2024-03-06T07:24:19.523034+0000","last_unstale":"2024-03-06T07:24:19.525242+0000","last_undegraded":"2024-03-06T07:24:19.525242+0000","last_fullsized":"2024-03-06T07:24:19.525242+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:43:07.141534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508951+0000","last_change":"2024-03-06T07:24:19.508951+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508951+0000","last_undegraded":"2024-03-06T07:24:19.508951+0000","last_fullsized":"2024-03-06T07:24:19.508951+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.530266+0000","last_change":"2024-03-06T07:24:19.526669+0000","last_active":"2024-03-06T07:24:19.530266+0000","last_peered":"2024-03-06T07:24:19.530266+0000","last_clean":"2024-03-06T07:24:19.530266+0000","last_became_active":"2024-03-06T07:24:19.526276+0000","last_became_peered":"2024-03-06T07:24:19.526276+0000","last_unstale":"2024-03-06T07:24:19.530266+0000","last_undegraded":"2024-03-06T07:24:19.530266+0000","last_fullsized":"2024-03-06T07:24:19.530266+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:32:50.322407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.510711+0000","last_change":"2024-03-06T07:24:19.510711+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.510711+0000","last_undegraded":"2024-03-06T07:24:19.510711+0000","last_fullsized":"2024-03-06T07:24:19.510711+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.520976+0000","last_change":"2024-03-06T07:24:19.520321+0000","last_active":"2024-03-06T07:24:19.520976+0000","last_peered":"2024-03-06T07:24:19.520976+0000","last_clean":"2024-03-06T07:24:19.520976+0000","last_became_active":"2024-03-06T07:24:19.520045+0000","last_became_peered":"2024-03-06T07:24:19.520045+0000","last_unstale":"2024-03-06T07:24:19.520976+0000","last_undegraded":"2024-03-06T07:24:19.520976+0000","last_fullsized":"2024-03-06T07:24:19.520976+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:01.534402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.511628+0000","last_change":"2024-03-06T07:24:19.511628+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.511628+0000","last_undegraded":"2024-03-06T07:24:19.511628+0000","last_fullsized":"2024-03-06T07:24:19.511628+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.524716+0000","last_change":"2024-03-06T07:24:19.520152+0000","last_active":"2024-03-06T07:24:19.524716+0000","last_peered":"2024-03-06T07:24:19.524716+0000","last_clean":"2024-03-06T07:24:19.524716+0000","last_became_active":"2024-03-06T07:24:19.519855+0000","last_became_peered":"2024-03-06T07:24:19.519855+0000","last_unstale":"2024-03-06T07:24:19.524716+0000","last_undegraded":"2024-03-06T07:24:19.524716+0000","last_fullsized":"2024-03-06T07:24:19.524716+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:20.649547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.509270+0000","last_change":"2024-03-06T07:24:19.509270+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.509270+0000","last_undegraded":"2024-03-06T07:24:19.509270+0000","last_fullsized":"2024-03-06T07:24:19.509270+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.525829+0000","last_change":"2024-03-06T07:24:19.525401+0000","last_active":"2024-03-06T07:24:19.525829+0000","last_peered":"2024-03-06T07:24:19.525829+0000","last_clean":"2024-03-06T07:24:19.525829+0000","last_became_active":"2024-03-06T07:24:19.525064+0000","last_became_peered":"2024-03-06T07:24:19.525064+0000","last_unstale":"2024-03-06T07:24:19.525829+0000","last_undegraded":"2024-03-06T07:24:19.525829+0000","last_fullsized":"2024-03-06T07:24:19.525829+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:38.333961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508079+0000","last_change":"2024-03-06T07:24:19.508079+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508079+0000","last_undegraded":"2024-03-06T07:24:19.508079+0000","last_fullsized":"2024-03-06T07:24:19.508079+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.527834+0000","last_change":"2024-03-06T07:24:19.521974+0000","last_active":"2024-03-06T07:24:19.527834+0000","last_peered":"2024-03-06T07:24:19.527834+0000","last_clean":"2024-03-06T07:24:19.527834+0000","last_became_active":"2024-03-06T07:24:19.521736+0000","last_became_peered":"2024-03-06T07:24:19.521736+0000","last_unstale":"2024-03-06T07:24:19.527834+0000","last_undegraded":"2024-03-06T07:24:19.527834+0000","last_fullsized":"2024-03-06T07:24:19.527834+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:20:20.174477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508869+0000","last_change":"2024-03-06T07:24:19.508869+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508869+0000","last_undegraded":"2024-03-06T07:24:19.508869+0000","last_fullsized":"2024-03-06T07:24:19.508869+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.525378+0000","last_change":"2024-03-06T07:24:19.523871+0000","last_active":"2024-03-06T07:24:19.525378+0000","last_peered":"2024-03-06T07:24:19.525378+0000","last_clean":"2024-03-06T07:24:19.525378+0000","last_became_active":"2024-03-06T07:24:19.523587+0000","last_became_peered":"2024-03-06T07:24:19.523587+0000","last_unstale":"2024-03-06T07:24:19.525378+0000","last_undegraded":"2024-03-06T07:24:19.525378+0000","last_fullsized":"2024-03-06T07:24:19.525378+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:16.014585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.520446+0000","last_change":"2024-03-06T07:24:19.520446+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.520446+0000","last_undegraded":"2024-03-06T07:24:19.520446+0000","last_fullsized":"2024-03-06T07:24:19.520446+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.524688+0000","last_change":"2024-03-06T07:24:19.520212+0000","last_active":"2024-03-06T07:24:19.524688+0000","last_peered":"2024-03-06T07:24:19.524688+0000","last_clean":"2024-03-06T07:24:19.524688+0000","last_became_active":"2024-03-06T07:24:19.519913+0000","last_became_peered":"2024-03-06T07:24:19.519913+0000","last_unstale":"2024-03-06T07:24:19.524688+0000","last_undegraded":"2024-03-06T07:24:19.524688+0000","last_fullsized":"2024-03-06T07:24:19.524688+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:22.158292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508125+0000","last_change":"2024-03-06T07:24:19.508125+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508125+0000","last_undegraded":"2024-03-06T07:24:19.508125+0000","last_fullsized":"2024-03-06T07:24:19.508125+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.524670+0000","last_change":"2024-03-06T07:24:19.520035+0000","last_active":"2024-03-06T07:24:19.524670+0000","last_peered":"2024-03-06T07:24:19.524670+0000","last_clean":"2024-03-06T07:24:19.524670+0000","last_became_active":"2024-03-06T07:24:19.519761+0000","last_became_peered":"2024-03-06T07:24:19.519761+0000","last_unstale":"2024-03-06T07:24:19.524670+0000","last_undegraded":"2024-03-06T07:24:19.524670+0000","last_fullsized":"2024-03-06T07:24:19.524670+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:45.374706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508234+0000","last_change":"2024-03-06T07:24:19.508234+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508234+0000","last_undegraded":"2024-03-06T07:24:19.508234+0000","last_fullsized":"2024-03-06T07:24:19.508234+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.517678+0000","last_change":"2024-03-06T07:24:19.515544+0000","last_active":"2024-03-06T07:24:19.517678+0000","last_peered":"2024-03-06T07:24:19.517678+0000","last_clean":"2024-03-06T07:24:19.517678+0000","last_became_active":"2024-03-06T07:24:19.515266+0000","last_became_peered":"2024-03-06T07:24:19.515266+0000","last_unstale":"2024-03-06T07:24:19.517678+0000","last_undegraded":"2024-03-06T07:24:19.517678+0000","last_fullsized":"2024-03-06T07:24:19.517678+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:25:58.500915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.512821+0000","last_change":"2024-03-06T07:24:19.512821+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.512821+0000","last_undegraded":"2024-03-06T07:24:19.512821+0000","last_fullsized":"2024-03-06T07:24:19.512821+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.527496+0000","last_change":"2024-03-06T07:24:19.521024+0000","last_active":"2024-03-06T07:24:19.527496+0000","last_peered":"2024-03-06T07:24:19.527496+0000","last_clean":"2024-03-06T07:24:19.527496+0000","last_became_active":"2024-03-06T07:24:19.520754+0000","last_became_peered":"2024-03-06T07:24:19.520754+0000","last_unstale":"2024-03-06T07:24:19.527496+0000","last_undegraded":"2024-03-06T07:24:19.527496+0000","last_fullsized":"2024-03-06T07:24:19.527496+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:38.661204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.512496+0000","last_change":"2024-03-06T07:24:19.512496+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.512496+0000","last_undegraded":"2024-03-06T07:24:19.512496+0000","last_fullsized":"2024-03-06T07:24:19.512496+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.524611+0000","last_change":"2024-03-06T07:24:19.519653+0000","last_active":"2024-03-06T07:24:19.524611+0000","last_peered":"2024-03-06T07:24:19.524611+0000","last_clean":"2024-03-06T07:24:19.524611+0000","last_became_active":"2024-03-06T07:24:19.519374+0000","last_became_peered":"2024-03-06T07:24:19.519374+0000","last_unstale":"2024-03-06T07:24:19.524611+0000","last_undegraded":"2024-03-06T07:24:19.524611+0000","last_fullsized":"2024-03-06T07:24:19.524611+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:02:35.379525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508772+0000","last_change":"2024-03-06T07:24:19.508772+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508772+0000","last_undegraded":"2024-03-06T07:24:19.508772+0000","last_fullsized":"2024-03-06T07:24:19.508772+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.527636+0000","last_change":"2024-03-06T07:24:19.527152+0000","last_active":"2024-03-06T07:24:19.527636+0000","last_peered":"2024-03-06T07:24:19.527636+0000","last_clean":"2024-03-06T07:24:19.527636+0000","last_became_active":"2024-03-06T07:24:19.526879+0000","last_became_peered":"2024-03-06T07:24:19.526879+0000","last_unstale":"2024-03-06T07:24:19.527636+0000","last_undegraded":"2024-03-06T07:24:19.527636+0000","last_fullsized":"2024-03-06T07:24:19.527636+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:01.534402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508386+0000","last_change":"2024-03-06T07:24:19.508386+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508386+0000","last_undegraded":"2024-03-06T07:24:19.508386+0000","last_fullsized":"2024-03-06T07:24:19.508386+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.524915+0000","last_change":"2024-03-06T07:24:19.520942+0000","last_active":"2024-03-06T07:24:19.524915+0000","last_peered":"2024-03-06T07:24:19.524915+0000","last_clean":"2024-03-06T07:24:19.524915+0000","last_became_active":"2024-03-06T07:24:19.520672+0000","last_became_peered":"2024-03-06T07:24:19.520672+0000","last_unstale":"2024-03-06T07:24:19.524915+0000","last_undegraded":"2024-03-06T07:24:19.524915+0000","last_fullsized":"2024-03-06T07:24:19.524915+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:15.049439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.513801+0000","last_change":"2024-03-06T07:24:19.513801+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.513801+0000","last_undegraded":"2024-03-06T07:24:19.513801+0000","last_fullsized":"2024-03-06T07:24:19.513801+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.524240+0000","last_change":"2024-03-06T07:24:19.522226+0000","last_active":"2024-03-06T07:24:19.524240+0000","last_peered":"2024-03-06T07:24:19.524240+0000","last_clean":"2024-03-06T07:24:19.524240+0000","last_became_active":"2024-03-06T07:24:19.521882+0000","last_became_peered":"2024-03-06T07:24:19.521882+0000","last_unstale":"2024-03-06T07:24:19.524240+0000","last_undegraded":"2024-03-06T07:24:19.524240+0000","last_fullsized":"2024-03-06T07:24:19.524240+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:24.269083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.513191+0000","last_change":"2024-03-06T07:24:19.513191+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.513191+0000","last_undegraded":"2024-03-06T07:24:19.513191+0000","last_fullsized":"2024-03-06T07:24:19.513191+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.521727+0000","last_change":"2024-03-06T07:24:19.516436+0000","last_active":"2024-03-06T07:24:19.521727+0000","last_peered":"2024-03-06T07:24:19.521727+0000","last_clean":"2024-03-06T07:24:19.521727+0000","last_became_active":"2024-03-06T07:24:19.515998+0000","last_became_peered":"2024-03-06T07:24:19.515998+0000","last_unstale":"2024-03-06T07:24:19.521727+0000","last_undegraded":"2024-03-06T07:24:19.521727+0000","last_fullsized":"2024-03-06T07:24:19.521727+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:09.161121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.519430+0000","last_change":"2024-03-06T07:24:19.519430+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.519430+0000","last_undegraded":"2024-03-06T07:24:19.519430+0000","last_fullsized":"2024-03-06T07:24:19.519430+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.527547+0000","last_change":"2024-03-06T07:24:19.521792+0000","last_active":"2024-03-06T07:24:19.527547+0000","last_peered":"2024-03-06T07:24:19.527547+0000","last_clean":"2024-03-06T07:24:19.527547+0000","last_became_active":"2024-03-06T07:24:19.521565+0000","last_became_peered":"2024-03-06T07:24:19.521565+0000","last_unstale":"2024-03-06T07:24:19.527547+0000","last_undegraded":"2024-03-06T07:24:19.527547+0000","last_fullsized":"2024-03-06T07:24:19.527547+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:02.649347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508067+0000","last_change":"2024-03-06T07:24:19.508067+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508067+0000","last_undegraded":"2024-03-06T07:24:19.508067+0000","last_fullsized":"2024-03-06T07:24:19.508067+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.525392+0000","last_change":"2024-03-06T07:24:19.524110+0000","last_active":"2024-03-06T07:24:19.525392+0000","last_peered":"2024-03-06T07:24:19.525392+0000","last_clean":"2024-03-06T07:24:19.525392+0000","last_became_active":"2024-03-06T07:24:19.523812+0000","last_became_peered":"2024-03-06T07:24:19.523812+0000","last_unstale":"2024-03-06T07:24:19.525392+0000","last_undegraded":"2024-03-06T07:24:19.525392+0000","last_fullsized":"2024-03-06T07:24:19.525392+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:22:22.348426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.513750+0000","last_change":"2024-03-06T07:24:19.513750+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.513750+0000","last_undegraded":"2024-03-06T07:24:19.513750+0000","last_fullsized":"2024-03-06T07:24:19.513750+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.527467+0000","last_change":"2024-03-06T07:24:19.520220+0000","last_active":"2024-03-06T07:24:19.527467+0000","last_peered":"2024-03-06T07:24:19.527467+0000","last_clean":"2024-03-06T07:24:19.527467+0000","last_became_active":"2024-03-06T07:24:19.519962+0000","last_became_peered":"2024-03-06T07:24:19.519962+0000","last_unstale":"2024-03-06T07:24:19.527467+0000","last_undegraded":"2024-03-06T07:24:19.527467+0000","last_fullsized":"2024-03-06T07:24:19.527467+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:36:34.165775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.518980+0000","last_change":"2024-03-06T07:24:19.518980+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.518980+0000","last_undegraded":"2024-03-06T07:24:19.518980+0000","last_fullsized":"2024-03-06T07:24:19.518980+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.521731+0000","last_change":"2024-03-06T07:24:19.516940+0000","last_active":"2024-03-06T07:24:19.521731+0000","last_peered":"2024-03-06T07:24:19.521731+0000","last_clean":"2024-03-06T07:24:19.521731+0000","last_became_active":"2024-03-06T07:24:19.516691+0000","last_became_peered":"2024-03-06T07:24:19.516691+0000","last_unstale":"2024-03-06T07:24:19.521731+0000","last_undegraded":"2024-03-06T07:24:19.521731+0000","last_fullsized":"2024-03-06T07:24:19.521731+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:19.361223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.519861+0000","last_change":"2024-03-06T07:24:19.519861+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.519861+0000","last_undegraded":"2024-03-06T07:24:19.519861+0000","last_fullsized":"2024-03-06T07:24:19.519861+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.524300+0000","last_change":"2024-03-06T07:24:19.522294+0000","last_active":"2024-03-06T07:24:19.524300+0000","last_peered":"2024-03-06T07:24:19.524300+0000","last_clean":"2024-03-06T07:24:19.524300+0000","last_became_active":"2024-03-06T07:24:19.521845+0000","last_became_peered":"2024-03-06T07:24:19.521845+0000","last_unstale":"2024-03-06T07:24:19.524300+0000","last_undegraded":"2024-03-06T07:24:19.524300+0000","last_fullsized":"2024-03-06T07:24:19.524300+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:00:32.355340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":232,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.516321+0000","last_change":"2024-03-06T07:21:47.553693+0000","last_active":"2024-03-06T07:24:19.516321+0000","last_peered":"2024-03-06T07:24:19.516321+0000","last_clean":"2024-03-06T07:24:19.516321+0000","last_became_active":"2024-03-06T07:21:47.552664+0000","last_became_peered":"2024-03-06T07:21:47.552664+0000","last_unstale":"2024-03-06T07:24:19.516321+0000","last_undegraded":"2024-03-06T07:24:19.516321+0000","last_fullsized":"2024-03-06T07:24:19.516321+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_clean_scrub_stamp":"2024-03-06T07:21:46.362028+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:02.916426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32412,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339428,"statfs":{"total":96636764160,"available":96603574272,"internally_reserved":0,"allocated":1253376,"data_stored":861417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574882,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31828,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340012,"statfs":{"total":96636764160,"available":96604172288,"internally_reserved":0,"allocated":659456,"data_stored":271049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31832,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340008,"statfs":{"total":96636764160,"available":96604168192,"internally_reserved":0,"allocated":659456,"data_stored":271049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574884,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31812,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340028,"statfs":{"total":96636764160,"available":96604188672,"internally_reserved":0,"allocated":647168,"data_stored":264782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574883,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31816,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340024,"statfs":{"total":96636764160,"available":96604184576,"internally_reserved":0,"allocated":647168,"data_stored":264782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31832,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340008,"statfs":{"total":96636764160,"available":96604168192,"internally_reserved":0,"allocated":659456,"data_stored":271049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31828,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340012,"statfs":{"total":96636764160,"available":96604172288,"internally_reserved":0,"allocated":659456,"data_stored":271049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574883,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32404,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339436,"statfs":{"total":96636764160,"available":96603582464,"internally_reserved":0,"allocated":1253376,"data_stored":861417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:24.276 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3f is not active+clean 2024-03-06T07:24:24.276 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.514792+0000', 'last_change': '2024-03-06T07:24:19.514792+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.514792+0000', 'last_undegraded': '2024-03-06T07:24:19.514792+0000', 'last_fullsized': '2024-03-06T07:24:19.514792+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:24:24.276 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3e is not active+clean 2024-03-06T07:24:24.276 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.524462+0000', 'last_change': '2024-03-06T07:24:19.524462+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.524462+0000', 'last_undegraded': '2024-03-06T07:24:19.524462+0000', 'last_fullsized': '2024-03-06T07:24:19.524462+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:24.276 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3d is not active+clean 2024-03-06T07:24:24.276 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.513384+0000', 'last_change': '2024-03-06T07:24:19.513384+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.513384+0000', 'last_undegraded': '2024-03-06T07:24:19.513384+0000', 'last_fullsized': '2024-03-06T07:24:19.513384+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:24.277 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3c is not active+clean 2024-03-06T07:24:24.277 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.510475+0000', 'last_change': '2024-03-06T07:24:19.510475+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.510475+0000', 'last_undegraded': '2024-03-06T07:24:19.510475+0000', 'last_fullsized': '2024-03-06T07:24:19.510475+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:24:24.277 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3b is not active+clean 2024-03-06T07:24:24.277 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.518129+0000', 'last_change': '2024-03-06T07:24:19.518129+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.518129+0000', 'last_undegraded': '2024-03-06T07:24:19.518129+0000', 'last_fullsized': '2024-03-06T07:24:19.518129+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:24:24.277 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3a is not active+clean 2024-03-06T07:24:24.277 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.518791+0000', 'last_change': '2024-03-06T07:24:19.518791+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.518791+0000', 'last_undegraded': '2024-03-06T07:24:19.518791+0000', 'last_fullsized': '2024-03-06T07:24:19.518791+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:24:24.277 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.39 is not active+clean 2024-03-06T07:24:24.277 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.512145+0000', 'last_change': '2024-03-06T07:24:19.512145+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.512145+0000', 'last_undegraded': '2024-03-06T07:24:19.512145+0000', 'last_fullsized': '2024-03-06T07:24:19.512145+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:24:24.277 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.38 is not active+clean 2024-03-06T07:24:24.277 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.523059+0000', 'last_change': '2024-03-06T07:24:19.523059+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.523059+0000', 'last_undegraded': '2024-03-06T07:24:19.523059+0000', 'last_fullsized': '2024-03-06T07:24:19.523059+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:24.277 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.37 is not active+clean 2024-03-06T07:24:24.277 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.510901+0000', 'last_change': '2024-03-06T07:24:19.510901+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.510901+0000', 'last_undegraded': '2024-03-06T07:24:19.510901+0000', 'last_fullsized': '2024-03-06T07:24:19.510901+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:24:24.278 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.36 is not active+clean 2024-03-06T07:24:24.278 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.515316+0000', 'last_change': '2024-03-06T07:24:19.515316+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.515316+0000', 'last_undegraded': '2024-03-06T07:24:19.515316+0000', 'last_fullsized': '2024-03-06T07:24:19.515316+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:24:24.278 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.35 is not active+clean 2024-03-06T07:24:24.278 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.516335+0000', 'last_change': '2024-03-06T07:24:19.516335+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.516335+0000', 'last_undegraded': '2024-03-06T07:24:19.516335+0000', 'last_fullsized': '2024-03-06T07:24:19.516335+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:24:24.278 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.34 is not active+clean 2024-03-06T07:24:24.278 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.510027+0000', 'last_change': '2024-03-06T07:24:19.510027+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.510027+0000', 'last_undegraded': '2024-03-06T07:24:19.510027+0000', 'last_fullsized': '2024-03-06T07:24:19.510027+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:24:24.278 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.33 is not active+clean 2024-03-06T07:24:24.278 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.515035+0000', 'last_change': '2024-03-06T07:24:19.515035+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.515035+0000', 'last_undegraded': '2024-03-06T07:24:19.515035+0000', 'last_fullsized': '2024-03-06T07:24:19.515035+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:24:24.278 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.32 is not active+clean 2024-03-06T07:24:24.278 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.512059+0000', 'last_change': '2024-03-06T07:24:19.512059+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.512059+0000', 'last_undegraded': '2024-03-06T07:24:19.512059+0000', 'last_fullsized': '2024-03-06T07:24:19.512059+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:24:24.279 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.31 is not active+clean 2024-03-06T07:24:24.279 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.511672+0000', 'last_change': '2024-03-06T07:24:19.511672+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.511672+0000', 'last_undegraded': '2024-03-06T07:24:19.511672+0000', 'last_fullsized': '2024-03-06T07:24:19.511672+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:24:24.279 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.30 is not active+clean 2024-03-06T07:24:24.279 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.519288+0000', 'last_change': '2024-03-06T07:24:19.519288+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.519288+0000', 'last_undegraded': '2024-03-06T07:24:19.519288+0000', 'last_fullsized': '2024-03-06T07:24:19.519288+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:24:24.279 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2f is not active+clean 2024-03-06T07:24:24.279 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.522695+0000', 'last_change': '2024-03-06T07:24:19.522695+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.522695+0000', 'last_undegraded': '2024-03-06T07:24:19.522695+0000', 'last_fullsized': '2024-03-06T07:24:19.522695+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:24.279 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2e is not active+clean 2024-03-06T07:24:24.279 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.508661+0000', 'last_change': '2024-03-06T07:24:19.508661+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.508661+0000', 'last_undegraded': '2024-03-06T07:24:19.508661+0000', 'last_fullsized': '2024-03-06T07:24:19.508661+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:24.279 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2d is not active+clean 2024-03-06T07:24:24.279 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.514373+0000', 'last_change': '2024-03-06T07:24:19.514373+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.514373+0000', 'last_undegraded': '2024-03-06T07:24:19.514373+0000', 'last_fullsized': '2024-03-06T07:24:19.514373+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:24:24.279 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2c is not active+clean 2024-03-06T07:24:24.280 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.522340+0000', 'last_change': '2024-03-06T07:24:19.522340+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.522340+0000', 'last_undegraded': '2024-03-06T07:24:19.522340+0000', 'last_fullsized': '2024-03-06T07:24:19.522340+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:24.280 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2b is not active+clean 2024-03-06T07:24:24.280 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.513618+0000', 'last_change': '2024-03-06T07:24:19.513618+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.513618+0000', 'last_undegraded': '2024-03-06T07:24:19.513618+0000', 'last_fullsized': '2024-03-06T07:24:19.513618+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:24.280 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2a is not active+clean 2024-03-06T07:24:24.280 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.521571+0000', 'last_change': '2024-03-06T07:24:19.521571+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.521571+0000', 'last_undegraded': '2024-03-06T07:24:19.521571+0000', 'last_fullsized': '2024-03-06T07:24:19.521571+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:24.280 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.29 is not active+clean 2024-03-06T07:24:24.280 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.509993+0000', 'last_change': '2024-03-06T07:24:19.509993+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.509993+0000', 'last_undegraded': '2024-03-06T07:24:19.509993+0000', 'last_fullsized': '2024-03-06T07:24:19.509993+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:24.280 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.28 is not active+clean 2024-03-06T07:24:24.280 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.514668+0000', 'last_change': '2024-03-06T07:24:19.514668+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.514668+0000', 'last_undegraded': '2024-03-06T07:24:19.514668+0000', 'last_fullsized': '2024-03-06T07:24:19.514668+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:24:24.280 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.27 is not active+clean 2024-03-06T07:24:24.280 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.514380+0000', 'last_change': '2024-03-06T07:24:19.514380+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.514380+0000', 'last_undegraded': '2024-03-06T07:24:19.514380+0000', 'last_fullsized': '2024-03-06T07:24:19.514380+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:24:24.283 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.26 is not active+clean 2024-03-06T07:24:24.283 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.509876+0000', 'last_change': '2024-03-06T07:24:19.509876+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.509876+0000', 'last_undegraded': '2024-03-06T07:24:19.509876+0000', 'last_fullsized': '2024-03-06T07:24:19.509876+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:24:24.283 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.25 is not active+clean 2024-03-06T07:24:24.283 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.520572+0000', 'last_change': '2024-03-06T07:24:19.520572+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.520572+0000', 'last_undegraded': '2024-03-06T07:24:19.520572+0000', 'last_fullsized': '2024-03-06T07:24:19.520572+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:24:24.284 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.24 is not active+clean 2024-03-06T07:24:24.284 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.515332+0000', 'last_change': '2024-03-06T07:24:19.515332+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.515332+0000', 'last_undegraded': '2024-03-06T07:24:19.515332+0000', 'last_fullsized': '2024-03-06T07:24:19.515332+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:24:24.284 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.23 is not active+clean 2024-03-06T07:24:24.284 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.525727+0000', 'last_change': '2024-03-06T07:24:19.525727+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.525727+0000', 'last_undegraded': '2024-03-06T07:24:19.525727+0000', 'last_fullsized': '2024-03-06T07:24:19.525727+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:24.284 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.22 is not active+clean 2024-03-06T07:24:24.284 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.508941+0000', 'last_change': '2024-03-06T07:24:19.508941+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.508941+0000', 'last_undegraded': '2024-03-06T07:24:19.508941+0000', 'last_fullsized': '2024-03-06T07:24:19.508941+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:24.284 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.21 is not active+clean 2024-03-06T07:24:24.284 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.513795+0000', 'last_change': '2024-03-06T07:24:19.513795+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.513795+0000', 'last_undegraded': '2024-03-06T07:24:19.513795+0000', 'last_fullsized': '2024-03-06T07:24:19.513795+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:24:24.284 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.20 is not active+clean 2024-03-06T07:24:24.284 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.518693+0000', 'last_change': '2024-03-06T07:24:19.518693+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.518693+0000', 'last_undegraded': '2024-03-06T07:24:19.518693+0000', 'last_fullsized': '2024-03-06T07:24:19.518693+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:24:24.285 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1e is not active+clean 2024-03-06T07:24:24.285 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.518564+0000', 'last_change': '2024-03-06T07:24:19.518564+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.518564+0000', 'last_undegraded': '2024-03-06T07:24:19.518564+0000', 'last_fullsized': '2024-03-06T07:24:19.518564+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:24:24.285 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1f is not active+clean 2024-03-06T07:24:24.285 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.515662+0000', 'last_change': '2024-03-06T07:24:19.515662+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.515662+0000', 'last_undegraded': '2024-03-06T07:24:19.515662+0000', 'last_fullsized': '2024-03-06T07:24:19.515662+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:24:24.285 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1c is not active+clean 2024-03-06T07:24:24.285 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.512557+0000', 'last_change': '2024-03-06T07:24:19.512557+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.512557+0000', 'last_undegraded': '2024-03-06T07:24:19.512557+0000', 'last_fullsized': '2024-03-06T07:24:19.512557+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:24:24.285 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1d is not active+clean 2024-03-06T07:24:24.285 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.513655+0000', 'last_change': '2024-03-06T07:24:19.513655+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.513655+0000', 'last_undegraded': '2024-03-06T07:24:19.513655+0000', 'last_fullsized': '2024-03-06T07:24:19.513655+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:24:24.285 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1a is not active+clean 2024-03-06T07:24:24.285 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.520743+0000', 'last_change': '2024-03-06T07:24:19.520743+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.520743+0000', 'last_undegraded': '2024-03-06T07:24:19.520743+0000', 'last_fullsized': '2024-03-06T07:24:19.520743+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:24.285 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1b is not active+clean 2024-03-06T07:24:24.285 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.521187+0000', 'last_change': '2024-03-06T07:24:19.521187+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.521187+0000', 'last_undegraded': '2024-03-06T07:24:19.521187+0000', 'last_fullsized': '2024-03-06T07:24:19.521187+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:24.286 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.18 is not active+clean 2024-03-06T07:24:24.286 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.514185+0000', 'last_change': '2024-03-06T07:24:19.514185+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.514185+0000', 'last_undegraded': '2024-03-06T07:24:19.514185+0000', 'last_fullsized': '2024-03-06T07:24:19.514185+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:24:24.286 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.19 is not active+clean 2024-03-06T07:24:24.286 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.520302+0000', 'last_change': '2024-03-06T07:24:19.520302+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.520302+0000', 'last_undegraded': '2024-03-06T07:24:19.520302+0000', 'last_fullsized': '2024-03-06T07:24:19.520302+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:24.286 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.16 is not active+clean 2024-03-06T07:24:24.286 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.511187+0000', 'last_change': '2024-03-06T07:24:19.511187+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.511187+0000', 'last_undegraded': '2024-03-06T07:24:19.511187+0000', 'last_fullsized': '2024-03-06T07:24:19.511187+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:24:24.286 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.17 is not active+clean 2024-03-06T07:24:24.286 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.508569+0000', 'last_change': '2024-03-06T07:24:19.508569+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.508569+0000', 'last_undegraded': '2024-03-06T07:24:19.508569+0000', 'last_fullsized': '2024-03-06T07:24:19.508569+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:24.286 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.14 is not active+clean 2024-03-06T07:24:24.286 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.514274+0000', 'last_change': '2024-03-06T07:24:19.514274+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.514274+0000', 'last_undegraded': '2024-03-06T07:24:19.514274+0000', 'last_fullsized': '2024-03-06T07:24:19.514274+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:24:24.286 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.15 is not active+clean 2024-03-06T07:24:24.287 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.512986+0000', 'last_change': '2024-03-06T07:24:19.512986+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.512986+0000', 'last_undegraded': '2024-03-06T07:24:19.512986+0000', 'last_fullsized': '2024-03-06T07:24:19.512986+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:24.287 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.12 is not active+clean 2024-03-06T07:24:24.287 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.508951+0000', 'last_change': '2024-03-06T07:24:19.508951+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.508951+0000', 'last_undegraded': '2024-03-06T07:24:19.508951+0000', 'last_fullsized': '2024-03-06T07:24:19.508951+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:24:24.287 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.13 is not active+clean 2024-03-06T07:24:24.287 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.510711+0000', 'last_change': '2024-03-06T07:24:19.510711+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.510711+0000', 'last_undegraded': '2024-03-06T07:24:19.510711+0000', 'last_fullsized': '2024-03-06T07:24:19.510711+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:24.287 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.10 is not active+clean 2024-03-06T07:24:24.287 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.511628+0000', 'last_change': '2024-03-06T07:24:19.511628+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.511628+0000', 'last_undegraded': '2024-03-06T07:24:19.511628+0000', 'last_fullsized': '2024-03-06T07:24:19.511628+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:24:24.287 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.11 is not active+clean 2024-03-06T07:24:24.287 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.509270+0000', 'last_change': '2024-03-06T07:24:19.509270+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.509270+0000', 'last_undegraded': '2024-03-06T07:24:19.509270+0000', 'last_fullsized': '2024-03-06T07:24:19.509270+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:24.287 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.e is not active+clean 2024-03-06T07:24:24.288 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.508079+0000', 'last_change': '2024-03-06T07:24:19.508079+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.508079+0000', 'last_undegraded': '2024-03-06T07:24:19.508079+0000', 'last_fullsized': '2024-03-06T07:24:19.508079+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:24.288 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.f is not active+clean 2024-03-06T07:24:24.288 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.508869+0000', 'last_change': '2024-03-06T07:24:19.508869+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.508869+0000', 'last_undegraded': '2024-03-06T07:24:19.508869+0000', 'last_fullsized': '2024-03-06T07:24:19.508869+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:24:24.288 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.c is not active+clean 2024-03-06T07:24:24.288 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.520446+0000', 'last_change': '2024-03-06T07:24:19.520446+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.520446+0000', 'last_undegraded': '2024-03-06T07:24:19.520446+0000', 'last_fullsized': '2024-03-06T07:24:19.520446+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:24.288 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.d is not active+clean 2024-03-06T07:24:24.288 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.508125+0000', 'last_change': '2024-03-06T07:24:19.508125+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.508125+0000', 'last_undegraded': '2024-03-06T07:24:19.508125+0000', 'last_fullsized': '2024-03-06T07:24:19.508125+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:24.288 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.a is not active+clean 2024-03-06T07:24:24.288 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.508234+0000', 'last_change': '2024-03-06T07:24:19.508234+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.508234+0000', 'last_undegraded': '2024-03-06T07:24:19.508234+0000', 'last_fullsized': '2024-03-06T07:24:19.508234+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:24:24.288 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.b is not active+clean 2024-03-06T07:24:24.289 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.512821+0000', 'last_change': '2024-03-06T07:24:19.512821+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.512821+0000', 'last_undegraded': '2024-03-06T07:24:19.512821+0000', 'last_fullsized': '2024-03-06T07:24:19.512821+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:24:24.289 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.9 is not active+clean 2024-03-06T07:24:24.289 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.512496+0000', 'last_change': '2024-03-06T07:24:19.512496+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.512496+0000', 'last_undegraded': '2024-03-06T07:24:19.512496+0000', 'last_fullsized': '2024-03-06T07:24:19.512496+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:24:24.289 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.6 is not active+clean 2024-03-06T07:24:24.289 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.508772+0000', 'last_change': '2024-03-06T07:24:19.508772+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.508772+0000', 'last_undegraded': '2024-03-06T07:24:19.508772+0000', 'last_fullsized': '2024-03-06T07:24:19.508772+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:24:24.289 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.7 is not active+clean 2024-03-06T07:24:24.289 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.508386+0000', 'last_change': '2024-03-06T07:24:19.508386+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.508386+0000', 'last_undegraded': '2024-03-06T07:24:19.508386+0000', 'last_fullsized': '2024-03-06T07:24:19.508386+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:24:24.289 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.4 is not active+clean 2024-03-06T07:24:24.289 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.513801+0000', 'last_change': '2024-03-06T07:24:19.513801+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.513801+0000', 'last_undegraded': '2024-03-06T07:24:19.513801+0000', 'last_fullsized': '2024-03-06T07:24:19.513801+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:24:24.289 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.5 is not active+clean 2024-03-06T07:24:24.289 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.513191+0000', 'last_change': '2024-03-06T07:24:19.513191+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.513191+0000', 'last_undegraded': '2024-03-06T07:24:19.513191+0000', 'last_fullsized': '2024-03-06T07:24:19.513191+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:24:24.290 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2 is not active+clean 2024-03-06T07:24:24.290 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.519430+0000', 'last_change': '2024-03-06T07:24:19.519430+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.519430+0000', 'last_undegraded': '2024-03-06T07:24:19.519430+0000', 'last_fullsized': '2024-03-06T07:24:19.519430+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:24.290 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3 is not active+clean 2024-03-06T07:24:24.290 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.508067+0000', 'last_change': '2024-03-06T07:24:19.508067+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.508067+0000', 'last_undegraded': '2024-03-06T07:24:19.508067+0000', 'last_fullsized': '2024-03-06T07:24:19.508067+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:24.290 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.0 is not active+clean 2024-03-06T07:24:24.290 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.513750+0000', 'last_change': '2024-03-06T07:24:19.513750+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.513750+0000', 'last_undegraded': '2024-03-06T07:24:19.513750+0000', 'last_fullsized': '2024-03-06T07:24:19.513750+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:24:24.290 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1 is not active+clean 2024-03-06T07:24:24.290 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.518980+0000', 'last_change': '2024-03-06T07:24:19.518980+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.518980+0000', 'last_undegraded': '2024-03-06T07:24:19.518980+0000', 'last_fullsized': '2024-03-06T07:24:19.518980+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:24.290 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.8 is not active+clean 2024-03-06T07:24:24.290 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:19.519861+0000', 'last_change': '2024-03-06T07:24:19.519861+0000', 'last_active': '2024-03-06T07:24:19.501657+0000', 'last_peered': '2024-03-06T07:24:19.501657+0000', 'last_clean': '2024-03-06T07:24:19.501657+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:19.519861+0000', 'last_undegraded': '2024-03-06T07:24:19.519861+0000', 'last_fullsized': '2024-03-06T07:24:19.519861+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:19.501657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:24.291 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:24:24.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.410+0000 7fd2fa247640 1 Processor -- start 2024-03-06T07:24:24.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.410+0000 7fd2fa247640 1 -- start start 2024-03-06T07:24:24.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.411+0000 7fd2fa247640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fd2f4162970 0x7fd2f40d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:24.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.411+0000 7fd2fa247640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd2f4161cf0 0x7fd2f4162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:24.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.411+0000 7fd2fa247640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2f40d22c0 con 0x7fd2f40b4460 2024-03-06T07:24:24.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.411+0000 7fd2fa247640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2f4161b00 con 0x7fd2f4162970 2024-03-06T07:24:24.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.411+0000 7fd2fa247640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f40cecc0 con 0x7fd2f4161cf0 2024-03-06T07:24:24.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.411+0000 7fd2f3fff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd2f4161cf0 0x7fd2f4162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:24.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.411+0000 7fd2f2ffd640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fd2f4162970 0x7fd2f40d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:24.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.411+0000 7fd2f2ffd640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fd2f4162970 0x7fd2f40d4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:60992/0 (socket says 172.21.15.83:60992) 2024-03-06T07:24:24.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.411+0000 7fd2f2ffd640 1 -- 172.21.15.83:0/4055949248 learned_addr learned my addr 172.21.15.83:0/4055949248 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:24.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.411+0000 7fd2f3fff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd2f4161cf0 0x7fd2f4162180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:36786/0 (socket says 172.21.15.83:36786) 2024-03-06T07:24:24.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.411+0000 7fd2f37fe640 1 --1- 172.21.15.83:0/4055949248 >> v1:172.21.15.183:6789/0 conn(0x7fd2f40b4460 0x7fd2f40b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:59144/0 (socket says 172.21.15.83:59144) 2024-03-06T07:24:24.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.411+0000 7fd2f2ffd640 1 -- 172.21.15.83:0/4055949248 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd2f4161cf0 msgr2=0x7fd2f4162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:24.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.411+0000 7fd2f2ffd640 1 --2- 172.21.15.83:0/4055949248 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd2f4161cf0 0x7fd2f4162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.411+0000 7fd2f2ffd640 1 -- 172.21.15.83:0/4055949248 >> v1:172.21.15.183:6789/0 conn(0x7fd2f40b4460 legacy=0x7fd2f40b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:24.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.411+0000 7fd2f2ffd640 1 -- 172.21.15.83:0/4055949248 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f40d25a0 con 0x7fd2f4162970 2024-03-06T07:24:24.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.411+0000 7fd2f3fff640 1 --2- 172.21.15.83:0/4055949248 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd2f4161cf0 0x7fd2f4162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:24.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.412+0000 7fd2f2ffd640 1 --2- 172.21.15.83:0/4055949248 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd2f4162970 0x7fd2f40d4800 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8009870 tx=0x7fd2e802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=664a7ea0b0566d10 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:24.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.412+0000 7fd2f27fc640 1 -- 172.21.15.83:0/4055949248 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e8034070 con 0x7fd2f4162970 2024-03-06T07:24:24.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.412+0000 7fd2f27fc640 1 -- 172.21.15.83:0/4055949248 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2e8030840 con 0x7fd2f4162970 2024-03-06T07:24:24.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.412+0000 7fd2f27fc640 1 -- 172.21.15.83:0/4055949248 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e8030c10 con 0x7fd2f4162970 2024-03-06T07:24:24.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.413+0000 7fd2fa247640 1 -- 172.21.15.83:0/4055949248 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd2f4162970 msgr2=0x7fd2f40d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:24.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.413+0000 7fd2fa247640 1 --2- 172.21.15.83:0/4055949248 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd2f4162970 0x7fd2f40d4800 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8009870 tx=0x7fd2e802b980 comp rx=0 tx=0).stop 2024-03-06T07:24:24.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.413+0000 7fd2fa247640 1 -- 172.21.15.83:0/4055949248 shutdown_connections 2024-03-06T07:24:24.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.413+0000 7fd2fa247640 1 --2- 172.21.15.83:0/4055949248 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd2f4161cf0 0x7fd2f4162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.413+0000 7fd2fa247640 1 --2- 172.21.15.83:0/4055949248 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd2f4162970 0x7fd2f40d4800 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.413+0000 7fd2fa247640 1 -- 172.21.15.83:0/4055949248 >> 172.21.15.83:0/4055949248 conn(0x7fd2f40d3ce0 msgr2=0x7fd2f40d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:24.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.413+0000 7fd2fa247640 1 -- 172.21.15.83:0/4055949248 shutdown_connections 2024-03-06T07:24:24.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.413+0000 7fd2fa247640 1 -- 172.21.15.83:0/4055949248 wait complete. 2024-03-06T07:24:24.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.414+0000 7fd2fa247640 1 Processor -- start 2024-03-06T07:24:24.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.414+0000 7fd2fa247640 1 -- start start 2024-03-06T07:24:24.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.415+0000 7fd2fa247640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd2f40b4460 0x7fd2f40b8b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:24.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.415+0000 7fd2fa247640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd2f4161cf0 0x7fd2f41565c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:24.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.415+0000 7fd2fa247640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd2f4162970 0x7fd2f41e8140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:24.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.415+0000 7fd2fa247640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f40ce950 con 0x7fd2f4161cf0 2024-03-06T07:24:24.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.415+0000 7fd2fa247640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f40d2b60 con 0x7fd2f40b4460 2024-03-06T07:24:24.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.415+0000 7fd2fa247640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f40b4840 con 0x7fd2f4162970 2024-03-06T07:24:24.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.415+0000 7fd2f2ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd2f4161cf0 0x7fd2f41565c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:24.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.415+0000 7fd2f2ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd2f4161cf0 0x7fd2f41565c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40818/0 (socket says 172.21.15.83:40818) 2024-03-06T07:24:24.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.415+0000 7fd2f2ffd640 1 -- 172.21.15.83:0/2883262034 learned_addr learned my addr 172.21.15.83:0/2883262034 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:24.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.415+0000 7fd2f3fff640 1 --2- 172.21.15.83:0/2883262034 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd2f4162970 0x7fd2f41e8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:24.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.415+0000 7fd2f37fe640 1 --2- 172.21.15.83:0/2883262034 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd2f40b4460 0x7fd2f40b8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:24.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.415+0000 7fd2f2ffd640 1 -- 172.21.15.83:0/2883262034 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd2f4162970 msgr2=0x7fd2f41e8140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:24.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.415+0000 7fd2f2ffd640 1 --2- 172.21.15.83:0/2883262034 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd2f4162970 0x7fd2f41e8140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.415+0000 7fd2f2ffd640 1 -- 172.21.15.83:0/2883262034 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd2f40b4460 msgr2=0x7fd2f40b8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:24.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.415+0000 7fd2f2ffd640 1 --2- 172.21.15.83:0/2883262034 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd2f40b4460 0x7fd2f40b8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.415+0000 7fd2f2ffd640 1 -- 172.21.15.83:0/2883262034 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f40c81f0 con 0x7fd2f4161cf0 2024-03-06T07:24:24.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.415+0000 7fd2f3fff640 1 --2- 172.21.15.83:0/2883262034 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd2f4162970 0x7fd2f41e8140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:24.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.415+0000 7fd2f37fe640 1 --2- 172.21.15.83:0/2883262034 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd2f40b4460 0x7fd2f40b8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:24.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.415+0000 7fd2f2ffd640 1 --2- 172.21.15.83:0/2883262034 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd2f4161cf0 0x7fd2f41565c0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fd2e802b9c0 tx=0x7fd2e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:24.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.416+0000 7fd2dbfff640 1 -- 172.21.15.83:0/2883262034 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e8034040 con 0x7fd2f4161cf0 2024-03-06T07:24:24.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.416+0000 7fd2dbfff640 1 -- 172.21.15.83:0/2883262034 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2e8030640 con 0x7fd2f4161cf0 2024-03-06T07:24:24.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.416+0000 7fd2fa247640 1 -- 172.21.15.83:0/2883262034 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2f40c84b0 con 0x7fd2f4161cf0 2024-03-06T07:24:24.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.416+0000 7fd2dbfff640 1 -- 172.21.15.83:0/2883262034 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e80423f0 con 0x7fd2f4161cf0 2024-03-06T07:24:24.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.416+0000 7fd2fa247640 1 -- 172.21.15.83:0/2883262034 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2f40c8a70 con 0x7fd2f4161cf0 2024-03-06T07:24:24.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.417+0000 7fd2fa247640 1 -- 172.21.15.83:0/2883262034 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2f40d4880 con 0x7fd2f4161cf0 2024-03-06T07:24:24.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.417+0000 7fd2dbfff640 1 -- 172.21.15.83:0/2883262034 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd2e8037070 con 0x7fd2f4161cf0 2024-03-06T07:24:24.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.417+0000 7fd2dbfff640 1 --2- 172.21.15.83:0/2883262034 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd2e006b4d0 0x7fd2e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:24.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.418+0000 7fd2dbfff640 1 -- 172.21.15.83:0/2883262034 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd2e80ac3f0 con 0x7fd2f4161cf0 2024-03-06T07:24:24.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.418+0000 7fd2f37fe640 1 --2- 172.21.15.83:0/2883262034 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd2e006b4d0 0x7fd2e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:24.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.418+0000 7fd2f37fe640 1 --2- 172.21.15.83:0/2883262034 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd2e006b4d0 0x7fd2e006d990 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd2f40ca9f0 tx=0x7fd2e4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:24.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.422+0000 7fd2dbfff640 1 -- 172.21.15.83:0/2883262034 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd2f40d4880 con 0x7fd2f4161cf0 2024-03-06T07:24:24.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.686+0000 7fd2fa247640 1 -- 172.21.15.83:0/2883262034 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fd2f404f330 con 0x7fd2f4161cf0 2024-03-06T07:24:24.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.686+0000 7fd2dbfff640 1 -- 172.21.15.83:0/2883262034 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1453 (secure 0 0 0) 0x7fd2e807b390 con 0x7fd2f4161cf0 2024-03-06T07:24:24.689 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:24:24.689 INFO:teuthology.orchestra.run.smithi083.stdout:{"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":170,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":57,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709705,"num_in_osds":8,"osd_in_since":1709709696,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":261902336,"bytes_avail":772832210944,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":25,"id":4,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":4,"rank":0,"name":"a","status":"up:active","gid":6084}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-03-06T07:24:10.345019+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:24:24.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.688+0000 7fd2fa247640 1 -- 172.21.15.83:0/2883262034 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd2e006b4d0 msgr2=0x7fd2e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:24.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.688+0000 7fd2fa247640 1 --2- 172.21.15.83:0/2883262034 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd2e006b4d0 0x7fd2e006d990 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd2f40ca9f0 tx=0x7fd2e4009040 comp rx=0 tx=0).stop 2024-03-06T07:24:24.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.688+0000 7fd2fa247640 1 -- 172.21.15.83:0/2883262034 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd2f4161cf0 msgr2=0x7fd2f41565c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:24.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.688+0000 7fd2fa247640 1 --2- 172.21.15.83:0/2883262034 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd2f4161cf0 0x7fd2f41565c0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fd2e802b9c0 tx=0x7fd2e80046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.688+0000 7fd2fa247640 1 -- 172.21.15.83:0/2883262034 shutdown_connections 2024-03-06T07:24:24.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.688+0000 7fd2fa247640 1 --2- 172.21.15.83:0/2883262034 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd2e006b4d0 0x7fd2e006d990 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.688+0000 7fd2fa247640 1 --2- 172.21.15.83:0/2883262034 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd2f4162970 0x7fd2f41e8140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.688+0000 7fd2fa247640 1 --2- 172.21.15.83:0/2883262034 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd2f4161cf0 0x7fd2f41565c0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.688+0000 7fd2fa247640 1 --2- 172.21.15.83:0/2883262034 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd2f40b4460 0x7fd2f40b8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.688+0000 7fd2fa247640 1 -- 172.21.15.83:0/2883262034 >> 172.21.15.83:0/2883262034 conn(0x7fd2f40d3ce0 msgr2=0x7fd2f40d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:24.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.689+0000 7fd2fa247640 1 -- 172.21.15.83:0/2883262034 shutdown_connections 2024-03-06T07:24:24.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.689+0000 7fd2fa247640 1 -- 172.21.15.83:0/2883262034 wait complete. 2024-03-06T07:24:24.701 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:24:24.702 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:24.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.804+0000 7fe14da6d640 1 Processor -- start 2024-03-06T07:24:24.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.805+0000 7fe14da6d640 1 -- start start 2024-03-06T07:24:24.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.805+0000 7fe14da6d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe14815bdd0 0x7fe1480c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:24.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.805+0000 7fe14da6d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe1480d16e0 0x7fe1480d1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:24.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.805+0000 7fe14da6d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1480cebe0 con 0x7fe14815fa10 2024-03-06T07:24:24.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.805+0000 7fe14da6d640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1480ce980 con 0x7fe1480d16e0 2024-03-06T07:24:24.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.805+0000 7fe14da6d640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1480ceec0 con 0x7fe14815bdd0 2024-03-06T07:24:24.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.805+0000 7fe147fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe1480d16e0 0x7fe1480d1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:24.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.805+0000 7fe147fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe1480d16e0 0x7fe1480d1b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40832/0 (socket says 172.21.15.83:40832) 2024-03-06T07:24:24.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.805+0000 7fe147fff640 1 -- 172.21.15.83:0/2856188970 learned_addr learned my addr 172.21.15.83:0/2856188970 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:24.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.806+0000 7fe1477fe640 1 --2- 172.21.15.83:0/2856188970 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe14815bdd0 0x7fe1480c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:24.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.806+0000 7fe147fff640 1 -- 172.21.15.83:0/2856188970 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe14815bdd0 msgr2=0x7fe1480c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:24.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.806+0000 7fe147fff640 1 --2- 172.21.15.83:0/2856188970 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe14815bdd0 0x7fe1480c5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.806+0000 7fe147fff640 1 -- 172.21.15.83:0/2856188970 >> v1:172.21.15.183:6789/0 conn(0x7fe14815fa10 legacy=0x7fe1480ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:24.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.806+0000 7fe147fff640 1 -- 172.21.15.83:0/2856188970 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1480cf1a0 con 0x7fe1480d16e0 2024-03-06T07:24:24.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.806+0000 7fe147fff640 1 --2- 172.21.15.83:0/2856188970 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1480d16e0 0x7fe1480d1b30 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fe138009870 tx=0x7fe13802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3073e93dd1dcdaec server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:24.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.807+0000 7fe1467fc640 1 -- 172.21.15.83:0/2856188970 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1380317c0 con 0x7fe1480d16e0 2024-03-06T07:24:24.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.807+0000 7fe1467fc640 1 -- 172.21.15.83:0/2856188970 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe138037070 con 0x7fe1480d16e0 2024-03-06T07:24:24.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.807+0000 7fe1467fc640 1 -- 172.21.15.83:0/2856188970 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe138031b80 con 0x7fe1480d16e0 2024-03-06T07:24:24.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.807+0000 7fe14da6d640 1 -- 172.21.15.83:0/2856188970 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1480d16e0 msgr2=0x7fe1480d1b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:24.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.807+0000 7fe14da6d640 1 --2- 172.21.15.83:0/2856188970 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1480d16e0 0x7fe1480d1b30 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fe138009870 tx=0x7fe13802b860 comp rx=0 tx=0).stop 2024-03-06T07:24:24.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.807+0000 7fe14da6d640 1 -- 172.21.15.83:0/2856188970 shutdown_connections 2024-03-06T07:24:24.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.807+0000 7fe14da6d640 1 --2- 172.21.15.83:0/2856188970 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1480d16e0 0x7fe1480d1b30 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.807+0000 7fe14da6d640 1 --2- 172.21.15.83:0/2856188970 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe14815bdd0 0x7fe1480c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.807+0000 7fe14da6d640 1 -- 172.21.15.83:0/2856188970 >> 172.21.15.83:0/2856188970 conn(0x7fe14809a9f0 msgr2=0x7fe14809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:24.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.807+0000 7fe14da6d640 1 -- 172.21.15.83:0/2856188970 shutdown_connections 2024-03-06T07:24:24.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.807+0000 7fe14da6d640 1 -- 172.21.15.83:0/2856188970 wait complete. 2024-03-06T07:24:24.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.809+0000 7fe14da6d640 1 Processor -- start 2024-03-06T07:24:24.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.809+0000 7fe14da6d640 1 -- start start 2024-03-06T07:24:24.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.810+0000 7fe14da6d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1480d16e0 0x7fe1480b5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:24.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.810+0000 7fe14da6d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe14815bdd0 0x7fe1481f1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:24.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.810+0000 7fe14da6d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe14815fa10 0x7fe1481e79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:24.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.810+0000 7fe14da6d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1480d2070 con 0x7fe1480d16e0 2024-03-06T07:24:24.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.810+0000 7fe14da6d640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1480c5b50 con 0x7fe14815bdd0 2024-03-06T07:24:24.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.810+0000 7fe14da6d640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1480cc5d0 con 0x7fe14815fa10 2024-03-06T07:24:24.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.810+0000 7fe1477fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1480d16e0 0x7fe1480b5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:24.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.811+0000 7fe1477fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1480d16e0 0x7fe1480b5490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40840/0 (socket says 172.21.15.83:40840) 2024-03-06T07:24:24.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.811+0000 7fe1477fe640 1 -- 172.21.15.83:0/3420233348 learned_addr learned my addr 172.21.15.83:0/3420233348 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:24.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.811+0000 7fe147fff640 1 --2- 172.21.15.83:0/3420233348 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe14815fa10 0x7fe1481e79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:24.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.811+0000 7fe146ffd640 1 --2- 172.21.15.83:0/3420233348 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe14815bdd0 0x7fe1481f1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:24.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.811+0000 7fe1477fe640 1 -- 172.21.15.83:0/3420233348 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe14815fa10 msgr2=0x7fe1481e79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:24.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.811+0000 7fe1477fe640 1 --2- 172.21.15.83:0/3420233348 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe14815fa10 0x7fe1481e79c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.811+0000 7fe1477fe640 1 -- 172.21.15.83:0/3420233348 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe14815bdd0 msgr2=0x7fe1481f1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:24.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.811+0000 7fe1477fe640 1 --2- 172.21.15.83:0/3420233348 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe14815bdd0 0x7fe1481f1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.811+0000 7fe1477fe640 1 -- 172.21.15.83:0/3420233348 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1481e6940 con 0x7fe1480d16e0 2024-03-06T07:24:24.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.811+0000 7fe147fff640 1 --2- 172.21.15.83:0/3420233348 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe14815fa10 0x7fe1481e79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:24.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.811+0000 7fe1477fe640 1 --2- 172.21.15.83:0/3420233348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1480d16e0 0x7fe1480b5490 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fe140007e90 tx=0x7fe14000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:24.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.811+0000 7fe13ffff640 1 -- 172.21.15.83:0/3420233348 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe140014620 con 0x7fe1480d16e0 2024-03-06T07:24:24.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.811+0000 7fe13ffff640 1 -- 172.21.15.83:0/3420233348 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe140019070 con 0x7fe1480d16e0 2024-03-06T07:24:24.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.811+0000 7fe14da6d640 1 -- 172.21.15.83:0/3420233348 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1481e6c00 con 0x7fe1480d16e0 2024-03-06T07:24:24.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.811+0000 7fe13ffff640 1 -- 172.21.15.83:0/3420233348 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe140014a20 con 0x7fe1480d16e0 2024-03-06T07:24:24.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.811+0000 7fe14da6d640 1 -- 172.21.15.83:0/3420233348 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1481e71c0 con 0x7fe1480d16e0 2024-03-06T07:24:24.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.812+0000 7fe14da6d640 1 -- 172.21.15.83:0/3420233348 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe14815f3f0 con 0x7fe1480d16e0 2024-03-06T07:24:24.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.813+0000 7fe13ffff640 1 -- 172.21.15.83:0/3420233348 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe140014bc0 con 0x7fe1480d16e0 2024-03-06T07:24:24.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.813+0000 7fe13ffff640 1 --2- 172.21.15.83:0/3420233348 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe12806b4d0 0x7fe12806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:24.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.813+0000 7fe13ffff640 1 -- 172.21.15.83:0/3420233348 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe14008aac0 con 0x7fe1480d16e0 2024-03-06T07:24:24.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.813+0000 7fe146ffd640 1 --2- 172.21.15.83:0/3420233348 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe12806b4d0 0x7fe12806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:24.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.814+0000 7fe146ffd640 1 --2- 172.21.15.83:0/3420233348 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe12806b4d0 0x7fe12806d990 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe130006fd0 tx=0x7fe130008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:24.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.817+0000 7fe13ffff640 1 -- 172.21.15.83:0/3420233348 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe14815f3f0 con 0x7fe1480d16e0 2024-03-06T07:24:24.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.992+0000 7fe14da6d640 1 -- 172.21.15.83:0/3420233348 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe14804f330 con 0x7fe12806b4d0 2024-03-06T07:24:24.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:24.998+0000 7fe13ffff640 1 -- 172.21.15.83:0/3420233348 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246639 (secure 0 0 0) 0x7fe14804f330 con 0x7fe12806b4d0 2024-03-06T07:24:25.002 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:24:25.003 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-06T07:24:25.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:25.001+0000 7fe14da6d640 1 -- 172.21.15.83:0/3420233348 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe12806b4d0 msgr2=0x7fe12806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:25.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:25.001+0000 7fe14da6d640 1 --2- 172.21.15.83:0/3420233348 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe12806b4d0 0x7fe12806d990 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe130006fd0 tx=0x7fe130008040 comp rx=0 tx=0).stop 2024-03-06T07:24:25.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:25.001+0000 7fe14da6d640 1 -- 172.21.15.83:0/3420233348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1480d16e0 msgr2=0x7fe1480b5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:25.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:25.001+0000 7fe14da6d640 1 --2- 172.21.15.83:0/3420233348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1480d16e0 0x7fe1480b5490 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fe140007e90 tx=0x7fe14000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:25.001+0000 7fe14da6d640 1 -- 172.21.15.83:0/3420233348 shutdown_connections 2024-03-06T07:24:25.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:25.001+0000 7fe14da6d640 1 --2- 172.21.15.83:0/3420233348 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe12806b4d0 0x7fe12806d990 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:25.001+0000 7fe14da6d640 1 --2- 172.21.15.83:0/3420233348 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe14815fa10 0x7fe1481e79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:25.001+0000 7fe14da6d640 1 --2- 172.21.15.83:0/3420233348 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe14815bdd0 0x7fe1481f1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:25.001+0000 7fe14da6d640 1 --2- 172.21.15.83:0/3420233348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1480d16e0 0x7fe1480b5490 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:25.001+0000 7fe14da6d640 1 -- 172.21.15.83:0/3420233348 >> 172.21.15.83:0/3420233348 conn(0x7fe14809a9f0 msgr2=0x7fe1480c69c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:25.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:25.001+0000 7fe14da6d640 1 -- 172.21.15.83:0/3420233348 shutdown_connections 2024-03-06T07:24:25.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:25.001+0000 7fe14da6d640 1 -- 172.21.15.83:0/3420233348 wait complete. 2024-03-06T07:24:25.014 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":130,"stamp":"2024-03-06T07:24:24.347476+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":49,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":255764,"kb_used_data":6288,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754718956,"statfs":{"total":773094113280,"available":772832210944,"internally_reserved":0,"allocated":6438912,"data_stored":3336594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12757,"internal_metadata":255315499},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.843089"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.514792+0000","last_change":"2024-03-06T07:24:19.514792+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.514792+0000","last_undegraded":"2024-03-06T07:24:19.514792+0000","last_fullsized":"2024-03-06T07:24:19.514792+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.524462+0000","last_change":"2024-03-06T07:24:19.524462+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.524462+0000","last_undegraded":"2024-03-06T07:24:19.524462+0000","last_fullsized":"2024-03-06T07:24:19.524462+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.513384+0000","last_change":"2024-03-06T07:24:19.513384+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.513384+0000","last_undegraded":"2024-03-06T07:24:19.513384+0000","last_fullsized":"2024-03-06T07:24:19.513384+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.510475+0000","last_change":"2024-03-06T07:24:19.510475+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.510475+0000","last_undegraded":"2024-03-06T07:24:19.510475+0000","last_fullsized":"2024-03-06T07:24:19.510475+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.518129+0000","last_change":"2024-03-06T07:24:19.518129+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.518129+0000","last_undegraded":"2024-03-06T07:24:19.518129+0000","last_fullsized":"2024-03-06T07:24:19.518129+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.518791+0000","last_change":"2024-03-06T07:24:19.518791+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.518791+0000","last_undegraded":"2024-03-06T07:24:19.518791+0000","last_fullsized":"2024-03-06T07:24:19.518791+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.512145+0000","last_change":"2024-03-06T07:24:19.512145+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.512145+0000","last_undegraded":"2024-03-06T07:24:19.512145+0000","last_fullsized":"2024-03-06T07:24:19.512145+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.523059+0000","last_change":"2024-03-06T07:24:19.523059+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.523059+0000","last_undegraded":"2024-03-06T07:24:19.523059+0000","last_fullsized":"2024-03-06T07:24:19.523059+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.510901+0000","last_change":"2024-03-06T07:24:19.510901+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.510901+0000","last_undegraded":"2024-03-06T07:24:19.510901+0000","last_fullsized":"2024-03-06T07:24:19.510901+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.515316+0000","last_change":"2024-03-06T07:24:19.515316+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.515316+0000","last_undegraded":"2024-03-06T07:24:19.515316+0000","last_fullsized":"2024-03-06T07:24:19.515316+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.516335+0000","last_change":"2024-03-06T07:24:19.516335+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.516335+0000","last_undegraded":"2024-03-06T07:24:19.516335+0000","last_fullsized":"2024-03-06T07:24:19.516335+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.510027+0000","last_change":"2024-03-06T07:24:19.510027+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.510027+0000","last_undegraded":"2024-03-06T07:24:19.510027+0000","last_fullsized":"2024-03-06T07:24:19.510027+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.515035+0000","last_change":"2024-03-06T07:24:19.515035+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.515035+0000","last_undegraded":"2024-03-06T07:24:19.515035+0000","last_fullsized":"2024-03-06T07:24:19.515035+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.512059+0000","last_change":"2024-03-06T07:24:19.512059+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.512059+0000","last_undegraded":"2024-03-06T07:24:19.512059+0000","last_fullsized":"2024-03-06T07:24:19.512059+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.511672+0000","last_change":"2024-03-06T07:24:19.511672+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.511672+0000","last_undegraded":"2024-03-06T07:24:19.511672+0000","last_fullsized":"2024-03-06T07:24:19.511672+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.519288+0000","last_change":"2024-03-06T07:24:19.519288+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.519288+0000","last_undegraded":"2024-03-06T07:24:19.519288+0000","last_fullsized":"2024-03-06T07:24:19.519288+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.522695+0000","last_change":"2024-03-06T07:24:19.522695+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.522695+0000","last_undegraded":"2024-03-06T07:24:19.522695+0000","last_fullsized":"2024-03-06T07:24:19.522695+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508661+0000","last_change":"2024-03-06T07:24:19.508661+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508661+0000","last_undegraded":"2024-03-06T07:24:19.508661+0000","last_fullsized":"2024-03-06T07:24:19.508661+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.514373+0000","last_change":"2024-03-06T07:24:19.514373+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.514373+0000","last_undegraded":"2024-03-06T07:24:19.514373+0000","last_fullsized":"2024-03-06T07:24:19.514373+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.522340+0000","last_change":"2024-03-06T07:24:19.522340+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.522340+0000","last_undegraded":"2024-03-06T07:24:19.522340+0000","last_fullsized":"2024-03-06T07:24:19.522340+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.513618+0000","last_change":"2024-03-06T07:24:19.513618+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.513618+0000","last_undegraded":"2024-03-06T07:24:19.513618+0000","last_fullsized":"2024-03-06T07:24:19.513618+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.521571+0000","last_change":"2024-03-06T07:24:19.521571+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.521571+0000","last_undegraded":"2024-03-06T07:24:19.521571+0000","last_fullsized":"2024-03-06T07:24:19.521571+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.509993+0000","last_change":"2024-03-06T07:24:19.509993+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.509993+0000","last_undegraded":"2024-03-06T07:24:19.509993+0000","last_fullsized":"2024-03-06T07:24:19.509993+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.514668+0000","last_change":"2024-03-06T07:24:19.514668+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.514668+0000","last_undegraded":"2024-03-06T07:24:19.514668+0000","last_fullsized":"2024-03-06T07:24:19.514668+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.514380+0000","last_change":"2024-03-06T07:24:19.514380+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.514380+0000","last_undegraded":"2024-03-06T07:24:19.514380+0000","last_fullsized":"2024-03-06T07:24:19.514380+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.509876+0000","last_change":"2024-03-06T07:24:19.509876+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.509876+0000","last_undegraded":"2024-03-06T07:24:19.509876+0000","last_fullsized":"2024-03-06T07:24:19.509876+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.520572+0000","last_change":"2024-03-06T07:24:19.520572+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.520572+0000","last_undegraded":"2024-03-06T07:24:19.520572+0000","last_fullsized":"2024-03-06T07:24:19.520572+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.515332+0000","last_change":"2024-03-06T07:24:19.515332+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.515332+0000","last_undegraded":"2024-03-06T07:24:19.515332+0000","last_fullsized":"2024-03-06T07:24:19.515332+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.525727+0000","last_change":"2024-03-06T07:24:19.525727+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.525727+0000","last_undegraded":"2024-03-06T07:24:19.525727+0000","last_fullsized":"2024-03-06T07:24:19.525727+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508941+0000","last_change":"2024-03-06T07:24:19.508941+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508941+0000","last_undegraded":"2024-03-06T07:24:19.508941+0000","last_fullsized":"2024-03-06T07:24:19.508941+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.513795+0000","last_change":"2024-03-06T07:24:19.513795+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.513795+0000","last_undegraded":"2024-03-06T07:24:19.513795+0000","last_fullsized":"2024-03-06T07:24:19.513795+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.518693+0000","last_change":"2024-03-06T07:24:19.518693+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.518693+0000","last_undegraded":"2024-03-06T07:24:19.518693+0000","last_fullsized":"2024-03-06T07:24:19.518693+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.518564+0000","last_change":"2024-03-06T07:24:19.518564+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.518564+0000","last_undegraded":"2024-03-06T07:24:19.518564+0000","last_fullsized":"2024-03-06T07:24:19.518564+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.525975+0000","last_change":"2024-03-06T07:24:19.523037+0000","last_active":"2024-03-06T07:24:19.525975+0000","last_peered":"2024-03-06T07:24:19.525975+0000","last_clean":"2024-03-06T07:24:19.525975+0000","last_became_active":"2024-03-06T07:24:19.522767+0000","last_became_peered":"2024-03-06T07:24:19.522767+0000","last_unstale":"2024-03-06T07:24:19.525975+0000","last_undegraded":"2024-03-06T07:24:19.525975+0000","last_fullsized":"2024-03-06T07:24:19.525975+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:42:52.076503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.515662+0000","last_change":"2024-03-06T07:24:19.515662+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.515662+0000","last_undegraded":"2024-03-06T07:24:19.515662+0000","last_fullsized":"2024-03-06T07:24:19.515662+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.517806+0000","last_change":"2024-03-06T07:24:19.516320+0000","last_active":"2024-03-06T07:24:19.517806+0000","last_peered":"2024-03-06T07:24:19.517806+0000","last_clean":"2024-03-06T07:24:19.517806+0000","last_became_active":"2024-03-06T07:24:19.516067+0000","last_became_peered":"2024-03-06T07:24:19.516067+0000","last_unstale":"2024-03-06T07:24:19.517806+0000","last_undegraded":"2024-03-06T07:24:19.517806+0000","last_fullsized":"2024-03-06T07:24:19.517806+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:44:06.313408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.512557+0000","last_change":"2024-03-06T07:24:19.512557+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.512557+0000","last_undegraded":"2024-03-06T07:24:19.512557+0000","last_fullsized":"2024-03-06T07:24:19.512557+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.530956+0000","last_change":"2024-03-06T07:24:19.528754+0000","last_active":"2024-03-06T07:24:19.530956+0000","last_peered":"2024-03-06T07:24:19.530956+0000","last_clean":"2024-03-06T07:24:19.530956+0000","last_became_active":"2024-03-06T07:24:19.528479+0000","last_became_peered":"2024-03-06T07:24:19.528479+0000","last_unstale":"2024-03-06T07:24:19.530956+0000","last_undegraded":"2024-03-06T07:24:19.530956+0000","last_fullsized":"2024-03-06T07:24:19.530956+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:17.842390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.513655+0000","last_change":"2024-03-06T07:24:19.513655+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.513655+0000","last_undegraded":"2024-03-06T07:24:19.513655+0000","last_fullsized":"2024-03-06T07:24:19.513655+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.524698+0000","last_change":"2024-03-06T07:24:19.520169+0000","last_active":"2024-03-06T07:24:19.524698+0000","last_peered":"2024-03-06T07:24:19.524698+0000","last_clean":"2024-03-06T07:24:19.524698+0000","last_became_active":"2024-03-06T07:24:19.519886+0000","last_became_peered":"2024-03-06T07:24:19.519886+0000","last_unstale":"2024-03-06T07:24:19.524698+0000","last_undegraded":"2024-03-06T07:24:19.524698+0000","last_fullsized":"2024-03-06T07:24:19.524698+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:38.333961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.520743+0000","last_change":"2024-03-06T07:24:19.520743+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.520743+0000","last_undegraded":"2024-03-06T07:24:19.520743+0000","last_fullsized":"2024-03-06T07:24:19.520743+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.525372+0000","last_change":"2024-03-06T07:24:19.524214+0000","last_active":"2024-03-06T07:24:19.525372+0000","last_peered":"2024-03-06T07:24:19.525372+0000","last_clean":"2024-03-06T07:24:19.525372+0000","last_became_active":"2024-03-06T07:24:19.523589+0000","last_became_peered":"2024-03-06T07:24:19.523589+0000","last_unstale":"2024-03-06T07:24:19.525372+0000","last_undegraded":"2024-03-06T07:24:19.525372+0000","last_fullsized":"2024-03-06T07:24:19.525372+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:00.115699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.521187+0000","last_change":"2024-03-06T07:24:19.521187+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.521187+0000","last_undegraded":"2024-03-06T07:24:19.521187+0000","last_fullsized":"2024-03-06T07:24:19.521187+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.524818+0000","last_change":"2024-03-06T07:24:19.521026+0000","last_active":"2024-03-06T07:24:19.524818+0000","last_peered":"2024-03-06T07:24:19.524818+0000","last_clean":"2024-03-06T07:24:19.524818+0000","last_became_active":"2024-03-06T07:24:19.520754+0000","last_became_peered":"2024-03-06T07:24:19.520754+0000","last_unstale":"2024-03-06T07:24:19.524818+0000","last_undegraded":"2024-03-06T07:24:19.524818+0000","last_fullsized":"2024-03-06T07:24:19.524818+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:23.502992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.514185+0000","last_change":"2024-03-06T07:24:19.514185+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.514185+0000","last_undegraded":"2024-03-06T07:24:19.514185+0000","last_fullsized":"2024-03-06T07:24:19.514185+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.521399+0000","last_change":"2024-03-06T07:24:19.516459+0000","last_active":"2024-03-06T07:24:19.521399+0000","last_peered":"2024-03-06T07:24:19.521399+0000","last_clean":"2024-03-06T07:24:19.521399+0000","last_became_active":"2024-03-06T07:24:19.516181+0000","last_became_peered":"2024-03-06T07:24:19.516181+0000","last_unstale":"2024-03-06T07:24:19.521399+0000","last_undegraded":"2024-03-06T07:24:19.521399+0000","last_fullsized":"2024-03-06T07:24:19.521399+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:08.665102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.520302+0000","last_change":"2024-03-06T07:24:19.520302+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.520302+0000","last_undegraded":"2024-03-06T07:24:19.520302+0000","last_fullsized":"2024-03-06T07:24:19.520302+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.524364+0000","last_change":"2024-03-06T07:24:19.518659+0000","last_active":"2024-03-06T07:24:19.524364+0000","last_peered":"2024-03-06T07:24:19.524364+0000","last_clean":"2024-03-06T07:24:19.524364+0000","last_became_active":"2024-03-06T07:24:19.517834+0000","last_became_peered":"2024-03-06T07:24:19.517834+0000","last_unstale":"2024-03-06T07:24:19.524364+0000","last_undegraded":"2024-03-06T07:24:19.524364+0000","last_fullsized":"2024-03-06T07:24:19.524364+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:09.323969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.511187+0000","last_change":"2024-03-06T07:24:19.511187+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.511187+0000","last_undegraded":"2024-03-06T07:24:19.511187+0000","last_fullsized":"2024-03-06T07:24:19.511187+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.529051+0000","last_change":"2024-03-06T07:24:19.528439+0000","last_active":"2024-03-06T07:24:19.529051+0000","last_peered":"2024-03-06T07:24:19.529051+0000","last_clean":"2024-03-06T07:24:19.529051+0000","last_became_active":"2024-03-06T07:24:19.528141+0000","last_became_peered":"2024-03-06T07:24:19.528141+0000","last_unstale":"2024-03-06T07:24:19.529051+0000","last_undegraded":"2024-03-06T07:24:19.529051+0000","last_fullsized":"2024-03-06T07:24:19.529051+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:24:36.321071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508569+0000","last_change":"2024-03-06T07:24:19.508569+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508569+0000","last_undegraded":"2024-03-06T07:24:19.508569+0000","last_fullsized":"2024-03-06T07:24:19.508569+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.526823+0000","last_change":"2024-03-06T07:24:19.519750+0000","last_active":"2024-03-06T07:24:19.526823+0000","last_peered":"2024-03-06T07:24:19.526823+0000","last_clean":"2024-03-06T07:24:19.526823+0000","last_became_active":"2024-03-06T07:24:19.519503+0000","last_became_peered":"2024-03-06T07:24:19.519503+0000","last_unstale":"2024-03-06T07:24:19.526823+0000","last_undegraded":"2024-03-06T07:24:19.526823+0000","last_fullsized":"2024-03-06T07:24:19.526823+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:00:04.275748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.514274+0000","last_change":"2024-03-06T07:24:19.514274+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.514274+0000","last_undegraded":"2024-03-06T07:24:19.514274+0000","last_fullsized":"2024-03-06T07:24:19.514274+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.528856+0000","last_change":"2024-03-06T07:24:19.528356+0000","last_active":"2024-03-06T07:24:19.528856+0000","last_peered":"2024-03-06T07:24:19.528856+0000","last_clean":"2024-03-06T07:24:19.528856+0000","last_became_active":"2024-03-06T07:24:19.527954+0000","last_became_peered":"2024-03-06T07:24:19.527954+0000","last_unstale":"2024-03-06T07:24:19.528856+0000","last_undegraded":"2024-03-06T07:24:19.528856+0000","last_fullsized":"2024-03-06T07:24:19.528856+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:22.158292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.512986+0000","last_change":"2024-03-06T07:24:19.512986+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.512986+0000","last_undegraded":"2024-03-06T07:24:19.512986+0000","last_fullsized":"2024-03-06T07:24:19.512986+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.525242+0000","last_change":"2024-03-06T07:24:19.523322+0000","last_active":"2024-03-06T07:24:19.525242+0000","last_peered":"2024-03-06T07:24:19.525242+0000","last_clean":"2024-03-06T07:24:19.525242+0000","last_became_active":"2024-03-06T07:24:19.523034+0000","last_became_peered":"2024-03-06T07:24:19.523034+0000","last_unstale":"2024-03-06T07:24:19.525242+0000","last_undegraded":"2024-03-06T07:24:19.525242+0000","last_fullsized":"2024-03-06T07:24:19.525242+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:43:07.141534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508951+0000","last_change":"2024-03-06T07:24:19.508951+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508951+0000","last_undegraded":"2024-03-06T07:24:19.508951+0000","last_fullsized":"2024-03-06T07:24:19.508951+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.530266+0000","last_change":"2024-03-06T07:24:19.526669+0000","last_active":"2024-03-06T07:24:19.530266+0000","last_peered":"2024-03-06T07:24:19.530266+0000","last_clean":"2024-03-06T07:24:19.530266+0000","last_became_active":"2024-03-06T07:24:19.526276+0000","last_became_peered":"2024-03-06T07:24:19.526276+0000","last_unstale":"2024-03-06T07:24:19.530266+0000","last_undegraded":"2024-03-06T07:24:19.530266+0000","last_fullsized":"2024-03-06T07:24:19.530266+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:32:50.322407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.510711+0000","last_change":"2024-03-06T07:24:19.510711+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.510711+0000","last_undegraded":"2024-03-06T07:24:19.510711+0000","last_fullsized":"2024-03-06T07:24:19.510711+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.520976+0000","last_change":"2024-03-06T07:24:19.520321+0000","last_active":"2024-03-06T07:24:19.520976+0000","last_peered":"2024-03-06T07:24:19.520976+0000","last_clean":"2024-03-06T07:24:19.520976+0000","last_became_active":"2024-03-06T07:24:19.520045+0000","last_became_peered":"2024-03-06T07:24:19.520045+0000","last_unstale":"2024-03-06T07:24:19.520976+0000","last_undegraded":"2024-03-06T07:24:19.520976+0000","last_fullsized":"2024-03-06T07:24:19.520976+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:01.534402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.511628+0000","last_change":"2024-03-06T07:24:19.511628+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.511628+0000","last_undegraded":"2024-03-06T07:24:19.511628+0000","last_fullsized":"2024-03-06T07:24:19.511628+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.524716+0000","last_change":"2024-03-06T07:24:19.520152+0000","last_active":"2024-03-06T07:24:19.524716+0000","last_peered":"2024-03-06T07:24:19.524716+0000","last_clean":"2024-03-06T07:24:19.524716+0000","last_became_active":"2024-03-06T07:24:19.519855+0000","last_became_peered":"2024-03-06T07:24:19.519855+0000","last_unstale":"2024-03-06T07:24:19.524716+0000","last_undegraded":"2024-03-06T07:24:19.524716+0000","last_fullsized":"2024-03-06T07:24:19.524716+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:20.649547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.509270+0000","last_change":"2024-03-06T07:24:19.509270+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.509270+0000","last_undegraded":"2024-03-06T07:24:19.509270+0000","last_fullsized":"2024-03-06T07:24:19.509270+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.525829+0000","last_change":"2024-03-06T07:24:19.525401+0000","last_active":"2024-03-06T07:24:19.525829+0000","last_peered":"2024-03-06T07:24:19.525829+0000","last_clean":"2024-03-06T07:24:19.525829+0000","last_became_active":"2024-03-06T07:24:19.525064+0000","last_became_peered":"2024-03-06T07:24:19.525064+0000","last_unstale":"2024-03-06T07:24:19.525829+0000","last_undegraded":"2024-03-06T07:24:19.525829+0000","last_fullsized":"2024-03-06T07:24:19.525829+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:38.333961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508079+0000","last_change":"2024-03-06T07:24:19.508079+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508079+0000","last_undegraded":"2024-03-06T07:24:19.508079+0000","last_fullsized":"2024-03-06T07:24:19.508079+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.527834+0000","last_change":"2024-03-06T07:24:19.521974+0000","last_active":"2024-03-06T07:24:19.527834+0000","last_peered":"2024-03-06T07:24:19.527834+0000","last_clean":"2024-03-06T07:24:19.527834+0000","last_became_active":"2024-03-06T07:24:19.521736+0000","last_became_peered":"2024-03-06T07:24:19.521736+0000","last_unstale":"2024-03-06T07:24:19.527834+0000","last_undegraded":"2024-03-06T07:24:19.527834+0000","last_fullsized":"2024-03-06T07:24:19.527834+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:20:20.174477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508869+0000","last_change":"2024-03-06T07:24:19.508869+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508869+0000","last_undegraded":"2024-03-06T07:24:19.508869+0000","last_fullsized":"2024-03-06T07:24:19.508869+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.525378+0000","last_change":"2024-03-06T07:24:19.523871+0000","last_active":"2024-03-06T07:24:19.525378+0000","last_peered":"2024-03-06T07:24:19.525378+0000","last_clean":"2024-03-06T07:24:19.525378+0000","last_became_active":"2024-03-06T07:24:19.523587+0000","last_became_peered":"2024-03-06T07:24:19.523587+0000","last_unstale":"2024-03-06T07:24:19.525378+0000","last_undegraded":"2024-03-06T07:24:19.525378+0000","last_fullsized":"2024-03-06T07:24:19.525378+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:16.014585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.520446+0000","last_change":"2024-03-06T07:24:19.520446+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.520446+0000","last_undegraded":"2024-03-06T07:24:19.520446+0000","last_fullsized":"2024-03-06T07:24:19.520446+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.524688+0000","last_change":"2024-03-06T07:24:19.520212+0000","last_active":"2024-03-06T07:24:19.524688+0000","last_peered":"2024-03-06T07:24:19.524688+0000","last_clean":"2024-03-06T07:24:19.524688+0000","last_became_active":"2024-03-06T07:24:19.519913+0000","last_became_peered":"2024-03-06T07:24:19.519913+0000","last_unstale":"2024-03-06T07:24:19.524688+0000","last_undegraded":"2024-03-06T07:24:19.524688+0000","last_fullsized":"2024-03-06T07:24:19.524688+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:22.158292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508125+0000","last_change":"2024-03-06T07:24:19.508125+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508125+0000","last_undegraded":"2024-03-06T07:24:19.508125+0000","last_fullsized":"2024-03-06T07:24:19.508125+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.524670+0000","last_change":"2024-03-06T07:24:19.520035+0000","last_active":"2024-03-06T07:24:19.524670+0000","last_peered":"2024-03-06T07:24:19.524670+0000","last_clean":"2024-03-06T07:24:19.524670+0000","last_became_active":"2024-03-06T07:24:19.519761+0000","last_became_peered":"2024-03-06T07:24:19.519761+0000","last_unstale":"2024-03-06T07:24:19.524670+0000","last_undegraded":"2024-03-06T07:24:19.524670+0000","last_fullsized":"2024-03-06T07:24:19.524670+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:45.374706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508234+0000","last_change":"2024-03-06T07:24:19.508234+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508234+0000","last_undegraded":"2024-03-06T07:24:19.508234+0000","last_fullsized":"2024-03-06T07:24:19.508234+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.517678+0000","last_change":"2024-03-06T07:24:19.515544+0000","last_active":"2024-03-06T07:24:19.517678+0000","last_peered":"2024-03-06T07:24:19.517678+0000","last_clean":"2024-03-06T07:24:19.517678+0000","last_became_active":"2024-03-06T07:24:19.515266+0000","last_became_peered":"2024-03-06T07:24:19.515266+0000","last_unstale":"2024-03-06T07:24:19.517678+0000","last_undegraded":"2024-03-06T07:24:19.517678+0000","last_fullsized":"2024-03-06T07:24:19.517678+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:25:58.500915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.512821+0000","last_change":"2024-03-06T07:24:19.512821+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.512821+0000","last_undegraded":"2024-03-06T07:24:19.512821+0000","last_fullsized":"2024-03-06T07:24:19.512821+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.527496+0000","last_change":"2024-03-06T07:24:19.521024+0000","last_active":"2024-03-06T07:24:19.527496+0000","last_peered":"2024-03-06T07:24:19.527496+0000","last_clean":"2024-03-06T07:24:19.527496+0000","last_became_active":"2024-03-06T07:24:19.520754+0000","last_became_peered":"2024-03-06T07:24:19.520754+0000","last_unstale":"2024-03-06T07:24:19.527496+0000","last_undegraded":"2024-03-06T07:24:19.527496+0000","last_fullsized":"2024-03-06T07:24:19.527496+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:38.661204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.512496+0000","last_change":"2024-03-06T07:24:19.512496+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.512496+0000","last_undegraded":"2024-03-06T07:24:19.512496+0000","last_fullsized":"2024-03-06T07:24:19.512496+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.524611+0000","last_change":"2024-03-06T07:24:19.519653+0000","last_active":"2024-03-06T07:24:19.524611+0000","last_peered":"2024-03-06T07:24:19.524611+0000","last_clean":"2024-03-06T07:24:19.524611+0000","last_became_active":"2024-03-06T07:24:19.519374+0000","last_became_peered":"2024-03-06T07:24:19.519374+0000","last_unstale":"2024-03-06T07:24:19.524611+0000","last_undegraded":"2024-03-06T07:24:19.524611+0000","last_fullsized":"2024-03-06T07:24:19.524611+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:02:35.379525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508772+0000","last_change":"2024-03-06T07:24:19.508772+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508772+0000","last_undegraded":"2024-03-06T07:24:19.508772+0000","last_fullsized":"2024-03-06T07:24:19.508772+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.527636+0000","last_change":"2024-03-06T07:24:19.527152+0000","last_active":"2024-03-06T07:24:19.527636+0000","last_peered":"2024-03-06T07:24:19.527636+0000","last_clean":"2024-03-06T07:24:19.527636+0000","last_became_active":"2024-03-06T07:24:19.526879+0000","last_became_peered":"2024-03-06T07:24:19.526879+0000","last_unstale":"2024-03-06T07:24:19.527636+0000","last_undegraded":"2024-03-06T07:24:19.527636+0000","last_fullsized":"2024-03-06T07:24:19.527636+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:01.534402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508386+0000","last_change":"2024-03-06T07:24:19.508386+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508386+0000","last_undegraded":"2024-03-06T07:24:19.508386+0000","last_fullsized":"2024-03-06T07:24:19.508386+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.524915+0000","last_change":"2024-03-06T07:24:19.520942+0000","last_active":"2024-03-06T07:24:19.524915+0000","last_peered":"2024-03-06T07:24:19.524915+0000","last_clean":"2024-03-06T07:24:19.524915+0000","last_became_active":"2024-03-06T07:24:19.520672+0000","last_became_peered":"2024-03-06T07:24:19.520672+0000","last_unstale":"2024-03-06T07:24:19.524915+0000","last_undegraded":"2024-03-06T07:24:19.524915+0000","last_fullsized":"2024-03-06T07:24:19.524915+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:15.049439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.513801+0000","last_change":"2024-03-06T07:24:19.513801+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.513801+0000","last_undegraded":"2024-03-06T07:24:19.513801+0000","last_fullsized":"2024-03-06T07:24:19.513801+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.524240+0000","last_change":"2024-03-06T07:24:19.522226+0000","last_active":"2024-03-06T07:24:19.524240+0000","last_peered":"2024-03-06T07:24:19.524240+0000","last_clean":"2024-03-06T07:24:19.524240+0000","last_became_active":"2024-03-06T07:24:19.521882+0000","last_became_peered":"2024-03-06T07:24:19.521882+0000","last_unstale":"2024-03-06T07:24:19.524240+0000","last_undegraded":"2024-03-06T07:24:19.524240+0000","last_fullsized":"2024-03-06T07:24:19.524240+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:24.269083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.513191+0000","last_change":"2024-03-06T07:24:19.513191+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.513191+0000","last_undegraded":"2024-03-06T07:24:19.513191+0000","last_fullsized":"2024-03-06T07:24:19.513191+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.521727+0000","last_change":"2024-03-06T07:24:19.516436+0000","last_active":"2024-03-06T07:24:19.521727+0000","last_peered":"2024-03-06T07:24:19.521727+0000","last_clean":"2024-03-06T07:24:19.521727+0000","last_became_active":"2024-03-06T07:24:19.515998+0000","last_became_peered":"2024-03-06T07:24:19.515998+0000","last_unstale":"2024-03-06T07:24:19.521727+0000","last_undegraded":"2024-03-06T07:24:19.521727+0000","last_fullsized":"2024-03-06T07:24:19.521727+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:09.161121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.519430+0000","last_change":"2024-03-06T07:24:19.519430+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.519430+0000","last_undegraded":"2024-03-06T07:24:19.519430+0000","last_fullsized":"2024-03-06T07:24:19.519430+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.527547+0000","last_change":"2024-03-06T07:24:19.521792+0000","last_active":"2024-03-06T07:24:19.527547+0000","last_peered":"2024-03-06T07:24:19.527547+0000","last_clean":"2024-03-06T07:24:19.527547+0000","last_became_active":"2024-03-06T07:24:19.521565+0000","last_became_peered":"2024-03-06T07:24:19.521565+0000","last_unstale":"2024-03-06T07:24:19.527547+0000","last_undegraded":"2024-03-06T07:24:19.527547+0000","last_fullsized":"2024-03-06T07:24:19.527547+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:02.649347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.508067+0000","last_change":"2024-03-06T07:24:19.508067+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.508067+0000","last_undegraded":"2024-03-06T07:24:19.508067+0000","last_fullsized":"2024-03-06T07:24:19.508067+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.525392+0000","last_change":"2024-03-06T07:24:19.524110+0000","last_active":"2024-03-06T07:24:19.525392+0000","last_peered":"2024-03-06T07:24:19.525392+0000","last_clean":"2024-03-06T07:24:19.525392+0000","last_became_active":"2024-03-06T07:24:19.523812+0000","last_became_peered":"2024-03-06T07:24:19.523812+0000","last_unstale":"2024-03-06T07:24:19.525392+0000","last_undegraded":"2024-03-06T07:24:19.525392+0000","last_fullsized":"2024-03-06T07:24:19.525392+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:22:22.348426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.513750+0000","last_change":"2024-03-06T07:24:19.513750+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.513750+0000","last_undegraded":"2024-03-06T07:24:19.513750+0000","last_fullsized":"2024-03-06T07:24:19.513750+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.527467+0000","last_change":"2024-03-06T07:24:19.520220+0000","last_active":"2024-03-06T07:24:19.527467+0000","last_peered":"2024-03-06T07:24:19.527467+0000","last_clean":"2024-03-06T07:24:19.527467+0000","last_became_active":"2024-03-06T07:24:19.519962+0000","last_became_peered":"2024-03-06T07:24:19.519962+0000","last_unstale":"2024-03-06T07:24:19.527467+0000","last_undegraded":"2024-03-06T07:24:19.527467+0000","last_fullsized":"2024-03-06T07:24:19.527467+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:36:34.165775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.518980+0000","last_change":"2024-03-06T07:24:19.518980+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.518980+0000","last_undegraded":"2024-03-06T07:24:19.518980+0000","last_fullsized":"2024-03-06T07:24:19.518980+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.521731+0000","last_change":"2024-03-06T07:24:19.516940+0000","last_active":"2024-03-06T07:24:19.521731+0000","last_peered":"2024-03-06T07:24:19.521731+0000","last_clean":"2024-03-06T07:24:19.521731+0000","last_became_active":"2024-03-06T07:24:19.516691+0000","last_became_peered":"2024-03-06T07:24:19.516691+0000","last_unstale":"2024-03-06T07:24:19.521731+0000","last_undegraded":"2024-03-06T07:24:19.521731+0000","last_fullsized":"2024-03-06T07:24:19.521731+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:19.361223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-03-06T07:24:19.519861+0000","last_change":"2024-03-06T07:24:19.519861+0000","last_active":"2024-03-06T07:24:19.501657+0000","last_peered":"2024-03-06T07:24:19.501657+0000","last_clean":"2024-03-06T07:24:19.501657+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:19.519861+0000","last_undegraded":"2024-03-06T07:24:19.519861+0000","last_fullsized":"2024-03-06T07:24:19.519861+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.524300+0000","last_change":"2024-03-06T07:24:19.522294+0000","last_active":"2024-03-06T07:24:19.524300+0000","last_peered":"2024-03-06T07:24:19.524300+0000","last_clean":"2024-03-06T07:24:19.524300+0000","last_became_active":"2024-03-06T07:24:19.521845+0000","last_became_peered":"2024-03-06T07:24:19.521845+0000","last_unstale":"2024-03-06T07:24:19.524300+0000","last_undegraded":"2024-03-06T07:24:19.524300+0000","last_fullsized":"2024-03-06T07:24:19.524300+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:00:32.355340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":232,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:24:19.516321+0000","last_change":"2024-03-06T07:21:47.553693+0000","last_active":"2024-03-06T07:24:19.516321+0000","last_peered":"2024-03-06T07:24:19.516321+0000","last_clean":"2024-03-06T07:24:19.516321+0000","last_became_active":"2024-03-06T07:21:47.552664+0000","last_became_peered":"2024-03-06T07:21:47.552664+0000","last_unstale":"2024-03-06T07:24:19.516321+0000","last_undegraded":"2024-03-06T07:24:19.516321+0000","last_fullsized":"2024-03-06T07:24:19.516321+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_clean_scrub_stamp":"2024-03-06T07:21:46.362028+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:02.916426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32412,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339428,"statfs":{"total":96636764160,"available":96603574272,"internally_reserved":0,"allocated":1253376,"data_stored":861417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574882,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31828,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340012,"statfs":{"total":96636764160,"available":96604172288,"internally_reserved":0,"allocated":659456,"data_stored":271049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31832,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340008,"statfs":{"total":96636764160,"available":96604168192,"internally_reserved":0,"allocated":659456,"data_stored":271049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574884,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31812,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340028,"statfs":{"total":96636764160,"available":96604188672,"internally_reserved":0,"allocated":647168,"data_stored":264782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574883,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31816,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340024,"statfs":{"total":96636764160,"available":96604184576,"internally_reserved":0,"allocated":647168,"data_stored":264782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31832,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340008,"statfs":{"total":96636764160,"available":96604168192,"internally_reserved":0,"allocated":659456,"data_stored":271049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31828,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340012,"statfs":{"total":96636764160,"available":96604172288,"internally_reserved":0,"allocated":659456,"data_stored":271049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574883,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32404,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339436,"statfs":{"total":96636764160,"available":96603582464,"internally_reserved":0,"allocated":1253376,"data_stored":861417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:25.781 INFO:tasks.ceph.mds.a.smithi083.stderr:2024-03-06T07:24:25.780+0000 7f728d9ea640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:24:28.018 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:28.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.121+0000 7f08bddeb640 1 Processor -- start 2024-03-06T07:24:28.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.121+0000 7f08bddeb640 1 -- start start 2024-03-06T07:24:28.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.122+0000 7f08bddeb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f08b815dfe0 0x7f08b80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.122+0000 7f08bddeb640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f08b80cf830 0x7f08b80d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.122+0000 7f08bddeb640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08b80b3430 con 0x7f08b80d27f0 2024-03-06T07:24:28.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.122+0000 7f08bddeb640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08b80d2c50 con 0x7f08b80cf830 2024-03-06T07:24:28.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.122+0000 7f08bddeb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08b80b31e0 con 0x7f08b815dfe0 2024-03-06T07:24:28.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.123+0000 7f08b6ffd640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f08b80cf830 0x7f08b80d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.123+0000 7f08b77fe640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f08b815dfe0 0x7f08b80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.123+0000 7f08b6ffd640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f08b80cf830 0x7f08b80d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:45002/0 (socket says 172.21.15.83:45002) 2024-03-06T07:24:28.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.123+0000 7f08b77fe640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f08b815dfe0 0x7f08b80cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:53096/0 (socket says 172.21.15.83:53096) 2024-03-06T07:24:28.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.123+0000 7f08b6ffd640 1 -- 172.21.15.83:0/1653987435 learned_addr learned my addr 172.21.15.83:0/1653987435 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:28.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.123+0000 7f08b7fff640 1 --1- 172.21.15.83:0/1653987435 >> v1:172.21.15.183:6789/0 conn(0x7f08b80d27f0 0x7f08b80d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:58422/0 (socket says 172.21.15.83:58422) 2024-03-06T07:24:28.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.123+0000 7f08b77fe640 1 -- 172.21.15.83:0/1653987435 >> v2:172.21.15.183:3300/0 conn(0x7f08b80cf830 msgr2=0x7f08b80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:28.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.123+0000 7f08b77fe640 1 --2- 172.21.15.83:0/1653987435 >> v2:172.21.15.183:3300/0 conn(0x7f08b80cf830 0x7f08b80d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.123+0000 7f08b77fe640 1 -- 172.21.15.83:0/1653987435 >> v1:172.21.15.183:6789/0 conn(0x7f08b80d27f0 legacy=0x7f08b80d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.123+0000 7f08b77fe640 1 -- 172.21.15.83:0/1653987435 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b80b3710 con 0x7f08b815dfe0 2024-03-06T07:24:28.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.123+0000 7f08b67fc640 1 -- 172.21.15.83:0/1653987435 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 459973265 0 0) 0x7f08b80b3430 con 0x7f08b80d27f0 2024-03-06T07:24:28.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.123+0000 7f08b6ffd640 1 --2- 172.21.15.83:0/1653987435 >> v2:172.21.15.183:3300/0 conn(0x7f08b80cf830 0x7f08b80d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:28.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.124+0000 7f08b77fe640 1 --2- 172.21.15.83:0/1653987435 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f08b815dfe0 0x7f08b80cf2f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f08a80097a0 tx=0x7f08a802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eee9ce4cb42a8ed0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:28.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.124+0000 7f08b67fc640 1 -- 172.21.15.83:0/1653987435 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08a8035020 con 0x7f08b815dfe0 2024-03-06T07:24:28.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.124+0000 7f08b67fc640 1 -- 172.21.15.83:0/1653987435 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08a8031830 con 0x7f08b815dfe0 2024-03-06T07:24:28.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.124+0000 7f08b67fc640 1 -- 172.21.15.83:0/1653987435 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08a8031c00 con 0x7f08b815dfe0 2024-03-06T07:24:28.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.124+0000 7f08bddeb640 1 -- 172.21.15.83:0/1653987435 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f08b815dfe0 msgr2=0x7f08b80cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.124+0000 7f08bddeb640 1 --2- 172.21.15.83:0/1653987435 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f08b815dfe0 0x7f08b80cf2f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f08a80097a0 tx=0x7f08a802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.125+0000 7f08bddeb640 1 -- 172.21.15.83:0/1653987435 shutdown_connections 2024-03-06T07:24:28.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.125+0000 7f08bddeb640 1 --2- 172.21.15.83:0/1653987435 >> v2:172.21.15.183:3300/0 conn(0x7f08b80cf830 0x7f08b80d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.125+0000 7f08bddeb640 1 --2- 172.21.15.83:0/1653987435 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f08b815dfe0 0x7f08b80cf2f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.125+0000 7f08bddeb640 1 -- 172.21.15.83:0/1653987435 >> 172.21.15.83:0/1653987435 conn(0x7f08b80d3cd0 msgr2=0x7f08b80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:28.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.125+0000 7f08bddeb640 1 -- 172.21.15.83:0/1653987435 shutdown_connections 2024-03-06T07:24:28.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.125+0000 7f08bddeb640 1 -- 172.21.15.83:0/1653987435 wait complete. 2024-03-06T07:24:28.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.126+0000 7f08bddeb640 1 Processor -- start 2024-03-06T07:24:28.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.126+0000 7f08bddeb640 1 -- start start 2024-03-06T07:24:28.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.126+0000 7f08bddeb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f08b80cf830 0x7f08b80a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.126+0000 7f08bddeb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f08b80d27f0 0x7f08b80b8720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.127+0000 7f08bddeb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08b815dfe0 0x7f08b81603b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.127+0000 7f08bddeb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b80d1c70 con 0x7f08b815dfe0 2024-03-06T07:24:28.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.127+0000 7f08bddeb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b80b3d60 con 0x7f08b80d27f0 2024-03-06T07:24:28.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.127+0000 7f08bddeb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08b80d1df0 con 0x7f08b80cf830 2024-03-06T07:24:28.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.127+0000 7f08b7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08b815dfe0 0x7f08b81603b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.127+0000 7f08b7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08b815dfe0 0x7f08b81603b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:42966/0 (socket says 172.21.15.83:42966) 2024-03-06T07:24:28.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.127+0000 7f08b7fff640 1 -- 172.21.15.83:0/564329474 learned_addr learned my addr 172.21.15.83:0/564329474 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:28.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.127+0000 7f08b77fe640 1 --2- 172.21.15.83:0/564329474 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f08b80cf830 0x7f08b80a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.127+0000 7f08b6ffd640 1 --2- 172.21.15.83:0/564329474 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f08b80d27f0 0x7f08b80b8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.127+0000 7f08b7fff640 1 -- 172.21.15.83:0/564329474 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f08b80cf830 msgr2=0x7f08b80a1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.127+0000 7f08b7fff640 1 --2- 172.21.15.83:0/564329474 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f08b80cf830 0x7f08b80a1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.127+0000 7f08b7fff640 1 -- 172.21.15.83:0/564329474 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f08b80d27f0 msgr2=0x7f08b80b8720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.127+0000 7f08b7fff640 1 --2- 172.21.15.83:0/564329474 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f08b80d27f0 0x7f08b80b8720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.127+0000 7f08b7fff640 1 -- 172.21.15.83:0/564329474 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b80cbcb0 con 0x7f08b815dfe0 2024-03-06T07:24:28.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.127+0000 7f08b7fff640 1 --2- 172.21.15.83:0/564329474 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08b815dfe0 0x7f08b81603b0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f08b000c7a0 tx=0x7f08b000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:28.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.127+0000 7f089ffff640 1 -- 172.21.15.83:0/564329474 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b0018070 con 0x7f08b815dfe0 2024-03-06T07:24:28.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.127+0000 7f089ffff640 1 -- 172.21.15.83:0/564329474 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08b000a640 con 0x7f08b815dfe0 2024-03-06T07:24:28.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.127+0000 7f08bddeb640 1 -- 172.21.15.83:0/564329474 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08b80cbf90 con 0x7f08b815dfe0 2024-03-06T07:24:28.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.127+0000 7f089ffff640 1 -- 172.21.15.83:0/564329474 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b000aab0 con 0x7f08b815dfe0 2024-03-06T07:24:28.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.127+0000 7f08bddeb640 1 -- 172.21.15.83:0/564329474 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08b80cc550 con 0x7f08b815dfe0 2024-03-06T07:24:28.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.129+0000 7f08bddeb640 1 -- 172.21.15.83:0/564329474 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08b808d3c0 con 0x7f08b815dfe0 2024-03-06T07:24:28.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.129+0000 7f089ffff640 1 -- 172.21.15.83:0/564329474 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f08b001b070 con 0x7f08b815dfe0 2024-03-06T07:24:28.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.129+0000 7f089ffff640 1 --2- 172.21.15.83:0/564329474 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f08a406b4d0 0x7f08a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.129+0000 7f089ffff640 1 -- 172.21.15.83:0/564329474 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f08b00897a0 con 0x7f08b815dfe0 2024-03-06T07:24:28.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.129+0000 7f08b77fe640 1 --2- 172.21.15.83:0/564329474 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f08a406b4d0 0x7f08a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.130+0000 7f08b77fe640 1 --2- 172.21.15.83:0/564329474 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f08a406b4d0 0x7f08a406d990 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f08a8036000 tx=0x7f08a8008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:28.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.137+0000 7f089ffff640 1 -- 172.21.15.83:0/564329474 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f08b808d3c0 con 0x7f08b815dfe0 2024-03-06T07:24:28.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.311+0000 7f08bddeb640 1 -- 172.21.15.83:0/564329474 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f08b804f330 con 0x7f08a406b4d0 2024-03-06T07:24:28.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.317+0000 7f089ffff640 1 -- 172.21.15.83:0/564329474 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251955 (secure 0 0 0) 0x7f08b804f330 con 0x7f08a406b4d0 2024-03-06T07:24:28.321 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:24:28.322 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-06T07:24:28.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.320+0000 7f08bddeb640 1 -- 172.21.15.83:0/564329474 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f08a406b4d0 msgr2=0x7f08a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.320+0000 7f08bddeb640 1 --2- 172.21.15.83:0/564329474 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f08a406b4d0 0x7f08a406d990 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f08a8036000 tx=0x7f08a8008030 comp rx=0 tx=0).stop 2024-03-06T07:24:28.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.320+0000 7f08bddeb640 1 -- 172.21.15.83:0/564329474 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08b815dfe0 msgr2=0x7f08b81603b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.320+0000 7f08bddeb640 1 --2- 172.21.15.83:0/564329474 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08b815dfe0 0x7f08b81603b0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f08b000c7a0 tx=0x7f08b000cc60 comp rx=0 tx=0).stop 2024-03-06T07:24:28.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.320+0000 7f08bddeb640 1 -- 172.21.15.83:0/564329474 shutdown_connections 2024-03-06T07:24:28.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.320+0000 7f08bddeb640 1 --2- 172.21.15.83:0/564329474 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f08a406b4d0 0x7f08a406d990 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.320+0000 7f08bddeb640 1 --2- 172.21.15.83:0/564329474 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08b815dfe0 0x7f08b81603b0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.320+0000 7f08bddeb640 1 --2- 172.21.15.83:0/564329474 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f08b80d27f0 0x7f08b80b8720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.320+0000 7f08bddeb640 1 --2- 172.21.15.83:0/564329474 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f08b80cf830 0x7f08b80a1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.320+0000 7f08bddeb640 1 -- 172.21.15.83:0/564329474 >> 172.21.15.83:0/564329474 conn(0x7f08b80d3cd0 msgr2=0x7f08b806d100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:28.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.320+0000 7f08bddeb640 1 -- 172.21.15.83:0/564329474 shutdown_connections 2024-03-06T07:24:28.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.320+0000 7f08bddeb640 1 -- 172.21.15.83:0/564329474 wait complete. 2024-03-06T07:24:28.332 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":131,"stamp":"2024-03-06T07:24:26.347786+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":256044,"kb_used_data":6568,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754718676,"statfs":{"total":773094113280,"available":772831924224,"internally_reserved":0,"allocated":6725632,"data_stored":3458548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12757,"internal_metadata":255315499},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.839935"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:22.424109+0000","last_change":"2024-03-06T07:24:20.547061+0000","last_active":"2024-03-06T07:24:22.424109+0000","last_peered":"2024-03-06T07:24:22.424109+0000","last_clean":"2024-03-06T07:24:22.424109+0000","last_became_active":"2024-03-06T07:24:20.546791+0000","last_became_peered":"2024-03-06T07:24:20.546791+0000","last_unstale":"2024-03-06T07:24:22.424109+0000","last_undegraded":"2024-03-06T07:24:22.424109+0000","last_fullsized":"2024-03-06T07:24:22.424109+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:00:33.362746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.704787+0000","last_change":"2024-03-06T07:24:20.538312+0000","last_active":"2024-03-06T07:24:21.704787+0000","last_peered":"2024-03-06T07:24:21.704787+0000","last_clean":"2024-03-06T07:24:21.704787+0000","last_became_active":"2024-03-06T07:24:20.537455+0000","last_became_peered":"2024-03-06T07:24:20.537455+0000","last_unstale":"2024-03-06T07:24:21.704787+0000","last_undegraded":"2024-03-06T07:24:21.704787+0000","last_fullsized":"2024-03-06T07:24:21.704787+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:44.313052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:22.424951+0000","last_change":"2024-03-06T07:24:20.541352+0000","last_active":"2024-03-06T07:24:22.424951+0000","last_peered":"2024-03-06T07:24:22.424951+0000","last_clean":"2024-03-06T07:24:22.424951+0000","last_became_active":"2024-03-06T07:24:20.540735+0000","last_became_peered":"2024-03-06T07:24:20.540735+0000","last_unstale":"2024-03-06T07:24:22.424951+0000","last_undegraded":"2024-03-06T07:24:22.424951+0000","last_fullsized":"2024-03-06T07:24:22.424951+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:19:28.418585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:22.425236+0000","last_change":"2024-03-06T07:24:20.537314+0000","last_active":"2024-03-06T07:24:22.425236+0000","last_peered":"2024-03-06T07:24:22.425236+0000","last_clean":"2024-03-06T07:24:22.425236+0000","last_became_active":"2024-03-06T07:24:20.537015+0000","last_became_peered":"2024-03-06T07:24:20.537015+0000","last_unstale":"2024-03-06T07:24:22.425236+0000","last_undegraded":"2024-03-06T07:24:22.425236+0000","last_fullsized":"2024-03-06T07:24:22.425236+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:19:28.418585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:22.423927+0000","last_change":"2024-03-06T07:24:20.542965+0000","last_active":"2024-03-06T07:24:22.423927+0000","last_peered":"2024-03-06T07:24:22.423927+0000","last_clean":"2024-03-06T07:24:22.423927+0000","last_became_active":"2024-03-06T07:24:20.542676+0000","last_became_peered":"2024-03-06T07:24:20.542676+0000","last_unstale":"2024-03-06T07:24:22.423927+0000","last_undegraded":"2024-03-06T07:24:22.423927+0000","last_fullsized":"2024-03-06T07:24:22.423927+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:39.668611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.536360+0000","last_change":"2024-03-06T07:24:20.543866+0000","last_active":"2024-03-06T07:24:21.536360+0000","last_peered":"2024-03-06T07:24:21.536360+0000","last_clean":"2024-03-06T07:24:21.536360+0000","last_became_active":"2024-03-06T07:24:20.543587+0000","last_became_peered":"2024-03-06T07:24:20.543587+0000","last_unstale":"2024-03-06T07:24:21.536360+0000","last_undegraded":"2024-03-06T07:24:21.536360+0000","last_fullsized":"2024-03-06T07:24:21.536360+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:33.862949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.530654+0000","last_change":"2024-03-06T07:24:20.533073+0000","last_active":"2024-03-06T07:24:21.530654+0000","last_peered":"2024-03-06T07:24:21.530654+0000","last_clean":"2024-03-06T07:24:21.530654+0000","last_became_active":"2024-03-06T07:24:20.532821+0000","last_became_peered":"2024-03-06T07:24:20.532821+0000","last_unstale":"2024-03-06T07:24:21.530654+0000","last_undegraded":"2024-03-06T07:24:21.530654+0000","last_fullsized":"2024-03-06T07:24:21.530654+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:39.341368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.704314+0000","last_change":"2024-03-06T07:24:20.536571+0000","last_active":"2024-03-06T07:24:21.704314+0000","last_peered":"2024-03-06T07:24:21.704314+0000","last_clean":"2024-03-06T07:24:21.704314+0000","last_became_active":"2024-03-06T07:24:20.536313+0000","last_became_peered":"2024-03-06T07:24:20.536313+0000","last_unstale":"2024-03-06T07:24:21.704314+0000","last_undegraded":"2024-03-06T07:24:21.704314+0000","last_fullsized":"2024-03-06T07:24:21.704314+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:26.535463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.530837+0000","last_change":"2024-03-06T07:24:20.532368+0000","last_active":"2024-03-06T07:24:21.530837+0000","last_peered":"2024-03-06T07:24:21.530837+0000","last_clean":"2024-03-06T07:24:21.530837+0000","last_became_active":"2024-03-06T07:24:20.532094+0000","last_became_peered":"2024-03-06T07:24:20.532094+0000","last_unstale":"2024-03-06T07:24:21.530837+0000","last_undegraded":"2024-03-06T07:24:21.530837+0000","last_fullsized":"2024-03-06T07:24:21.530837+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:24:37.328478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.529200+0000","last_change":"2024-03-06T07:24:20.535751+0000","last_active":"2024-03-06T07:24:21.529200+0000","last_peered":"2024-03-06T07:24:21.529200+0000","last_clean":"2024-03-06T07:24:21.529200+0000","last_became_active":"2024-03-06T07:24:20.535486+0000","last_became_peered":"2024-03-06T07:24:20.535486+0000","last_unstale":"2024-03-06T07:24:21.529200+0000","last_undegraded":"2024-03-06T07:24:21.529200+0000","last_fullsized":"2024-03-06T07:24:21.529200+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:46.382113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.529117+0000","last_change":"2024-03-06T07:24:20.531062+0000","last_active":"2024-03-06T07:24:21.529117+0000","last_peered":"2024-03-06T07:24:21.529117+0000","last_clean":"2024-03-06T07:24:21.529117+0000","last_became_active":"2024-03-06T07:24:20.530800+0000","last_became_peered":"2024-03-06T07:24:20.530800+0000","last_unstale":"2024-03-06T07:24:21.529117+0000","last_undegraded":"2024-03-06T07:24:21.529117+0000","last_fullsized":"2024-03-06T07:24:21.529117+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:18.239325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:22.425313+0000","last_change":"2024-03-06T07:24:20.537974+0000","last_active":"2024-03-06T07:24:22.425313+0000","last_peered":"2024-03-06T07:24:22.425313+0000","last_clean":"2024-03-06T07:24:22.425313+0000","last_became_active":"2024-03-06T07:24:20.537635+0000","last_became_peered":"2024-03-06T07:24:20.537635+0000","last_unstale":"2024-03-06T07:24:22.425313+0000","last_undegraded":"2024-03-06T07:24:22.425313+0000","last_fullsized":"2024-03-06T07:24:22.425313+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:17:33.713316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.530057+0000","last_change":"2024-03-06T07:24:20.535020+0000","last_active":"2024-03-06T07:24:21.530057+0000","last_peered":"2024-03-06T07:24:21.530057+0000","last_clean":"2024-03-06T07:24:21.530057+0000","last_became_active":"2024-03-06T07:24:20.534479+0000","last_became_peered":"2024-03-06T07:24:20.534479+0000","last_unstale":"2024-03-06T07:24:21.530057+0000","last_undegraded":"2024-03-06T07:24:21.530057+0000","last_fullsized":"2024-03-06T07:24:21.530057+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:01.123105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.530811+0000","last_change":"2024-03-06T07:24:20.535710+0000","last_active":"2024-03-06T07:24:21.530811+0000","last_peered":"2024-03-06T07:24:21.530811+0000","last_clean":"2024-03-06T07:24:21.530811+0000","last_became_active":"2024-03-06T07:24:20.535417+0000","last_became_peered":"2024-03-06T07:24:20.535417+0000","last_unstale":"2024-03-06T07:24:21.530811+0000","last_undegraded":"2024-03-06T07:24:21.530811+0000","last_fullsized":"2024-03-06T07:24:21.530811+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:23.165698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.530510+0000","last_change":"2024-03-06T07:24:20.532409+0000","last_active":"2024-03-06T07:24:21.530510+0000","last_peered":"2024-03-06T07:24:21.530510+0000","last_clean":"2024-03-06T07:24:21.530510+0000","last_became_active":"2024-03-06T07:24:20.532137+0000","last_became_peered":"2024-03-06T07:24:20.532137+0000","last_unstale":"2024-03-06T07:24:21.530510+0000","last_undegraded":"2024-03-06T07:24:21.530510+0000","last_fullsized":"2024-03-06T07:24:21.530510+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:21.656954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.536307+0000","last_change":"2024-03-06T07:24:20.528603+0000","last_active":"2024-03-06T07:24:21.536307+0000","last_peered":"2024-03-06T07:24:21.536307+0000","last_clean":"2024-03-06T07:24:21.536307+0000","last_became_active":"2024-03-06T07:24:20.527899+0000","last_became_peered":"2024-03-06T07:24:20.527899+0000","last_unstale":"2024-03-06T07:24:21.536307+0000","last_undegraded":"2024-03-06T07:24:21.536307+0000","last_fullsized":"2024-03-06T07:24:21.536307+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:10.247145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.704821+0000","last_change":"2024-03-06T07:24:20.530534+0000","last_active":"2024-03-06T07:24:21.704821+0000","last_peered":"2024-03-06T07:24:21.704821+0000","last_clean":"2024-03-06T07:24:21.704821+0000","last_became_active":"2024-03-06T07:24:20.530280+0000","last_became_peered":"2024-03-06T07:24:20.530280+0000","last_unstale":"2024-03-06T07:24:21.704821+0000","last_undegraded":"2024-03-06T07:24:21.704821+0000","last_fullsized":"2024-03-06T07:24:21.704821+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:14:42.771422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:22.424807+0000","last_change":"2024-03-06T07:24:20.535422+0000","last_active":"2024-03-06T07:24:22.424807+0000","last_peered":"2024-03-06T07:24:22.424807+0000","last_clean":"2024-03-06T07:24:22.424807+0000","last_became_active":"2024-03-06T07:24:20.535152+0000","last_became_peered":"2024-03-06T07:24:20.535152+0000","last_unstale":"2024-03-06T07:24:22.424807+0000","last_undegraded":"2024-03-06T07:24:22.424807+0000","last_fullsized":"2024-03-06T07:24:22.424807+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:48:29.376354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:22.424019+0000","last_change":"2024-03-06T07:24:20.547011+0000","last_active":"2024-03-06T07:24:22.424019+0000","last_peered":"2024-03-06T07:24:22.424019+0000","last_clean":"2024-03-06T07:24:22.424019+0000","last_became_active":"2024-03-06T07:24:20.546738+0000","last_became_peered":"2024-03-06T07:24:20.546738+0000","last_unstale":"2024-03-06T07:24:22.424019+0000","last_undegraded":"2024-03-06T07:24:22.424019+0000","last_fullsized":"2024-03-06T07:24:22.424019+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:02.541809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.704530+0000","last_change":"2024-03-06T07:24:20.536774+0000","last_active":"2024-03-06T07:24:21.704530+0000","last_peered":"2024-03-06T07:24:21.704530+0000","last_clean":"2024-03-06T07:24:21.704530+0000","last_became_active":"2024-03-06T07:24:20.536517+0000","last_became_peered":"2024-03-06T07:24:20.536517+0000","last_unstale":"2024-03-06T07:24:21.704530+0000","last_undegraded":"2024-03-06T07:24:21.704530+0000","last_fullsized":"2024-03-06T07:24:21.704530+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:27:44.362264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:22.425061+0000","last_change":"2024-03-06T07:24:20.541133+0000","last_active":"2024-03-06T07:24:22.425061+0000","last_peered":"2024-03-06T07:24:22.425061+0000","last_clean":"2024-03-06T07:24:22.425061+0000","last_became_active":"2024-03-06T07:24:20.540835+0000","last_became_peered":"2024-03-06T07:24:20.540835+0000","last_unstale":"2024-03-06T07:24:22.425061+0000","last_undegraded":"2024-03-06T07:24:22.425061+0000","last_fullsized":"2024-03-06T07:24:22.425061+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:00:12.129984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.704676+0000","last_change":"2024-03-06T07:24:20.539003+0000","last_active":"2024-03-06T07:24:21.704676+0000","last_peered":"2024-03-06T07:24:21.704676+0000","last_clean":"2024-03-06T07:24:21.704676+0000","last_became_active":"2024-03-06T07:24:20.537487+0000","last_became_peered":"2024-03-06T07:24:20.537487+0000","last_unstale":"2024-03-06T07:24:21.704676+0000","last_undegraded":"2024-03-06T07:24:21.704676+0000","last_fullsized":"2024-03-06T07:24:21.704676+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:45.229237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:22.424617+0000","last_change":"2024-03-06T07:24:20.545986+0000","last_active":"2024-03-06T07:24:22.424617+0000","last_peered":"2024-03-06T07:24:22.424617+0000","last_clean":"2024-03-06T07:24:22.424617+0000","last_became_active":"2024-03-06T07:24:20.545696+0000","last_became_peered":"2024-03-06T07:24:20.545696+0000","last_unstale":"2024-03-06T07:24:22.424617+0000","last_undegraded":"2024-03-06T07:24:22.424617+0000","last_fullsized":"2024-03-06T07:24:22.424617+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:59:03.575875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.529790+0000","last_change":"2024-03-06T07:24:20.534605+0000","last_active":"2024-03-06T07:24:21.529790+0000","last_peered":"2024-03-06T07:24:21.529790+0000","last_clean":"2024-03-06T07:24:21.529790+0000","last_became_active":"2024-03-06T07:24:20.534306+0000","last_became_peered":"2024-03-06T07:24:20.534306+0000","last_unstale":"2024-03-06T07:24:21.529790+0000","last_undegraded":"2024-03-06T07:24:21.529790+0000","last_fullsized":"2024-03-06T07:24:21.529790+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:43:08.148941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:22.424154+0000","last_change":"2024-03-06T07:24:20.543558+0000","last_active":"2024-03-06T07:24:22.424154+0000","last_peered":"2024-03-06T07:24:22.424154+0000","last_clean":"2024-03-06T07:24:22.424154+0000","last_became_active":"2024-03-06T07:24:20.543313+0000","last_became_peered":"2024-03-06T07:24:20.543313+0000","last_unstale":"2024-03-06T07:24:22.424154+0000","last_undegraded":"2024-03-06T07:24:22.424154+0000","last_fullsized":"2024-03-06T07:24:22.424154+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:50.462861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:22.424717+0000","last_change":"2024-03-06T07:24:20.536320+0000","last_active":"2024-03-06T07:24:22.424717+0000","last_peered":"2024-03-06T07:24:22.424717+0000","last_clean":"2024-03-06T07:24:22.424717+0000","last_became_active":"2024-03-06T07:24:20.535999+0000","last_became_peered":"2024-03-06T07:24:20.535999+0000","last_unstale":"2024-03-06T07:24:22.424717+0000","last_undegraded":"2024-03-06T07:24:22.424717+0000","last_fullsized":"2024-03-06T07:24:22.424717+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:04:09.600928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.536629+0000","last_change":"2024-03-06T07:24:20.528827+0000","last_active":"2024-03-06T07:24:21.536629+0000","last_peered":"2024-03-06T07:24:21.536629+0000","last_clean":"2024-03-06T07:24:21.536629+0000","last_became_active":"2024-03-06T07:24:20.528501+0000","last_became_peered":"2024-03-06T07:24:20.528501+0000","last_unstale":"2024-03-06T07:24:21.536629+0000","last_undegraded":"2024-03-06T07:24:21.536629+0000","last_fullsized":"2024-03-06T07:24:21.536629+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:59:03.575875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:22.423981+0000","last_change":"2024-03-06T07:24:20.537762+0000","last_active":"2024-03-06T07:24:22.423981+0000","last_peered":"2024-03-06T07:24:22.423981+0000","last_clean":"2024-03-06T07:24:22.423981+0000","last_became_active":"2024-03-06T07:24:20.537455+0000","last_became_peered":"2024-03-06T07:24:20.537455+0000","last_unstale":"2024-03-06T07:24:22.423981+0000","last_undegraded":"2024-03-06T07:24:22.423981+0000","last_fullsized":"2024-03-06T07:24:22.423981+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:03.656754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.704666+0000","last_change":"2024-03-06T07:24:20.540937+0000","last_active":"2024-03-06T07:24:21.704666+0000","last_peered":"2024-03-06T07:24:21.704666+0000","last_clean":"2024-03-06T07:24:21.704666+0000","last_became_active":"2024-03-06T07:24:20.540440+0000","last_became_peered":"2024-03-06T07:24:20.540440+0000","last_unstale":"2024-03-06T07:24:21.704666+0000","last_undegraded":"2024-03-06T07:24:21.704666+0000","last_fullsized":"2024-03-06T07:24:21.704666+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:52:36.135158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:22.425105+0000","last_change":"2024-03-06T07:24:20.538857+0000","last_active":"2024-03-06T07:24:22.425105+0000","last_peered":"2024-03-06T07:24:22.425105+0000","last_clean":"2024-03-06T07:24:22.425105+0000","last_became_active":"2024-03-06T07:24:20.538564+0000","last_became_peered":"2024-03-06T07:24:20.538564+0000","last_unstale":"2024-03-06T07:24:22.425105+0000","last_undegraded":"2024-03-06T07:24:22.425105+0000","last_fullsized":"2024-03-06T07:24:22.425105+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:44.651979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:22.424032+0000","last_change":"2024-03-06T07:24:20.534928+0000","last_active":"2024-03-06T07:24:22.424032+0000","last_peered":"2024-03-06T07:24:22.424032+0000","last_clean":"2024-03-06T07:24:22.424032+0000","last_became_active":"2024-03-06T07:24:20.534651+0000","last_became_peered":"2024-03-06T07:24:20.534651+0000","last_unstale":"2024-03-06T07:24:22.424032+0000","last_undegraded":"2024-03-06T07:24:22.424032+0000","last_fullsized":"2024-03-06T07:24:22.424032+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:25.276490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.536383+0000","last_change":"2024-03-06T07:24:20.528859+0000","last_active":"2024-03-06T07:24:21.536383+0000","last_peered":"2024-03-06T07:24:21.536383+0000","last_clean":"2024-03-06T07:24:21.536383+0000","last_became_active":"2024-03-06T07:24:20.528331+0000","last_became_peered":"2024-03-06T07:24:20.528331+0000","last_unstale":"2024-03-06T07:24:21.536383+0000","last_undegraded":"2024-03-06T07:24:21.536383+0000","last_fullsized":"2024-03-06T07:24:21.536383+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:09:14.101321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.536677+0000","last_change":"2024-03-06T07:24:20.530380+0000","last_active":"2024-03-06T07:24:21.536677+0000","last_peered":"2024-03-06T07:24:21.536677+0000","last_clean":"2024-03-06T07:24:21.536677+0000","last_became_active":"2024-03-06T07:24:20.530114+0000","last_became_peered":"2024-03-06T07:24:20.530114+0000","last_unstale":"2024-03-06T07:24:21.536677+0000","last_undegraded":"2024-03-06T07:24:21.536677+0000","last_fullsized":"2024-03-06T07:24:21.536677+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:48:29.376354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"56'4","reported_seq":18,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.536789+0000","last_change":"2024-03-06T07:24:19.523037+0000","last_active":"2024-03-06T07:24:21.536789+0000","last_peered":"2024-03-06T07:24:21.536789+0000","last_clean":"2024-03-06T07:24:21.536789+0000","last_became_active":"2024-03-06T07:24:19.522767+0000","last_became_peered":"2024-03-06T07:24:19.522767+0000","last_unstale":"2024-03-06T07:24:21.536789+0000","last_undegraded":"2024-03-06T07:24:21.536789+0000","last_fullsized":"2024-03-06T07:24:21.536789+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:08.665102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.529275+0000","last_change":"2024-03-06T07:24:20.534855+0000","last_active":"2024-03-06T07:24:21.529275+0000","last_peered":"2024-03-06T07:24:21.529275+0000","last_clean":"2024-03-06T07:24:21.529275+0000","last_became_active":"2024-03-06T07:24:20.534506+0000","last_became_peered":"2024-03-06T07:24:20.534506+0000","last_unstale":"2024-03-06T07:24:21.529275+0000","last_undegraded":"2024-03-06T07:24:21.529275+0000","last_fullsized":"2024-03-06T07:24:21.529275+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:16.056846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.704784+0000","last_change":"2024-03-06T07:24:19.516320+0000","last_active":"2024-03-06T07:24:21.704784+0000","last_peered":"2024-03-06T07:24:21.704784+0000","last_clean":"2024-03-06T07:24:21.704784+0000","last_became_active":"2024-03-06T07:24:19.516067+0000","last_became_peered":"2024-03-06T07:24:19.516067+0000","last_unstale":"2024-03-06T07:24:21.704784+0000","last_undegraded":"2024-03-06T07:24:21.704784+0000","last_fullsized":"2024-03-06T07:24:21.704784+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:50:28.234386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.536595+0000","last_change":"2024-03-06T07:24:20.533171+0000","last_active":"2024-03-06T07:24:21.536595+0000","last_peered":"2024-03-06T07:24:21.536595+0000","last_clean":"2024-03-06T07:24:21.536595+0000","last_became_active":"2024-03-06T07:24:20.532908+0000","last_became_peered":"2024-03-06T07:24:20.532908+0000","last_unstale":"2024-03-06T07:24:21.536595+0000","last_undegraded":"2024-03-06T07:24:21.536595+0000","last_fullsized":"2024-03-06T07:24:21.536595+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:02.495918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.536758+0000","last_change":"2024-03-06T07:24:19.528754+0000","last_active":"2024-03-06T07:24:21.536758+0000","last_peered":"2024-03-06T07:24:21.536758+0000","last_clean":"2024-03-06T07:24:21.536758+0000","last_became_active":"2024-03-06T07:24:19.528479+0000","last_became_peered":"2024-03-06T07:24:19.528479+0000","last_unstale":"2024-03-06T07:24:21.536758+0000","last_undegraded":"2024-03-06T07:24:21.536758+0000","last_fullsized":"2024-03-06T07:24:21.536758+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:22.158292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.536672+0000","last_change":"2024-03-06T07:24:20.538577+0000","last_active":"2024-03-06T07:24:21.536672+0000","last_peered":"2024-03-06T07:24:21.536672+0000","last_clean":"2024-03-06T07:24:21.536672+0000","last_became_active":"2024-03-06T07:24:20.537907+0000","last_became_peered":"2024-03-06T07:24:20.537907+0000","last_unstale":"2024-03-06T07:24:21.536672+0000","last_undegraded":"2024-03-06T07:24:21.536672+0000","last_fullsized":"2024-03-06T07:24:21.536672+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:04:09.600928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:22.425324+0000","last_change":"2024-03-06T07:24:19.520169+0000","last_active":"2024-03-06T07:24:22.425324+0000","last_peered":"2024-03-06T07:24:22.425324+0000","last_clean":"2024-03-06T07:24:22.425324+0000","last_became_active":"2024-03-06T07:24:19.519886+0000","last_became_peered":"2024-03-06T07:24:19.519886+0000","last_unstale":"2024-03-06T07:24:22.425324+0000","last_undegraded":"2024-03-06T07:24:22.425324+0000","last_fullsized":"2024-03-06T07:24:22.425324+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:19.361223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.704655+0000","last_change":"2024-03-06T07:24:20.537135+0000","last_active":"2024-03-06T07:24:21.704655+0000","last_peered":"2024-03-06T07:24:21.704655+0000","last_clean":"2024-03-06T07:24:21.704655+0000","last_became_active":"2024-03-06T07:24:20.536891+0000","last_became_peered":"2024-03-06T07:24:20.536891+0000","last_unstale":"2024-03-06T07:24:21.704655+0000","last_undegraded":"2024-03-06T07:24:21.704655+0000","last_fullsized":"2024-03-06T07:24:21.704655+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:03:02.147211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.530552+0000","last_change":"2024-03-06T07:24:19.524214+0000","last_active":"2024-03-06T07:24:21.530552+0000","last_peered":"2024-03-06T07:24:21.530552+0000","last_clean":"2024-03-06T07:24:21.530552+0000","last_became_active":"2024-03-06T07:24:19.523589+0000","last_became_peered":"2024-03-06T07:24:19.523589+0000","last_unstale":"2024-03-06T07:24:21.530552+0000","last_undegraded":"2024-03-06T07:24:21.530552+0000","last_fullsized":"2024-03-06T07:24:21.530552+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:14:07.968805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.704482+0000","last_change":"2024-03-06T07:24:20.540682+0000","last_active":"2024-03-06T07:24:21.704482+0000","last_peered":"2024-03-06T07:24:21.704482+0000","last_clean":"2024-03-06T07:24:21.704482+0000","last_became_active":"2024-03-06T07:24:20.540077+0000","last_became_peered":"2024-03-06T07:24:20.540077+0000","last_unstale":"2024-03-06T07:24:21.704482+0000","last_undegraded":"2024-03-06T07:24:21.704482+0000","last_fullsized":"2024-03-06T07:24:21.704482+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:32:48.316020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.536368+0000","last_change":"2024-03-06T07:24:19.521026+0000","last_active":"2024-03-06T07:24:21.536368+0000","last_peered":"2024-03-06T07:24:21.536368+0000","last_clean":"2024-03-06T07:24:21.536368+0000","last_became_active":"2024-03-06T07:24:19.520754+0000","last_became_peered":"2024-03-06T07:24:19.520754+0000","last_unstale":"2024-03-06T07:24:21.536368+0000","last_undegraded":"2024-03-06T07:24:21.536368+0000","last_fullsized":"2024-03-06T07:24:21.536368+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:02:35.379525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.529942+0000","last_change":"2024-03-06T07:24:20.530700+0000","last_active":"2024-03-06T07:24:21.529942+0000","last_peered":"2024-03-06T07:24:21.529942+0000","last_clean":"2024-03-06T07:24:21.529942+0000","last_became_active":"2024-03-06T07:24:20.530430+0000","last_became_peered":"2024-03-06T07:24:20.530430+0000","last_unstale":"2024-03-06T07:24:21.529942+0000","last_undegraded":"2024-03-06T07:24:21.529942+0000","last_fullsized":"2024-03-06T07:24:21.529942+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:22:23.355833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:22.424254+0000","last_change":"2024-03-06T07:24:19.516459+0000","last_active":"2024-03-06T07:24:22.424254+0000","last_peered":"2024-03-06T07:24:22.424254+0000","last_clean":"2024-03-06T07:24:22.424254+0000","last_became_active":"2024-03-06T07:24:19.516181+0000","last_became_peered":"2024-03-06T07:24:19.516181+0000","last_unstale":"2024-03-06T07:24:22.424254+0000","last_undegraded":"2024-03-06T07:24:22.424254+0000","last_fullsized":"2024-03-06T07:24:22.424254+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:25:58.500915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.704691+0000","last_change":"2024-03-06T07:24:20.533586+0000","last_active":"2024-03-06T07:24:21.704691+0000","last_peered":"2024-03-06T07:24:21.704691+0000","last_clean":"2024-03-06T07:24:21.704691+0000","last_became_active":"2024-03-06T07:24:20.532778+0000","last_became_peered":"2024-03-06T07:24:20.532778+0000","last_unstale":"2024-03-06T07:24:21.704691+0000","last_undegraded":"2024-03-06T07:24:21.704691+0000","last_fullsized":"2024-03-06T07:24:21.704691+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:42:17.057413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.530355+0000","last_change":"2024-03-06T07:24:19.518659+0000","last_active":"2024-03-06T07:24:21.530355+0000","last_peered":"2024-03-06T07:24:21.530355+0000","last_clean":"2024-03-06T07:24:21.530355+0000","last_became_active":"2024-03-06T07:24:19.517834+0000","last_became_peered":"2024-03-06T07:24:19.517834+0000","last_unstale":"2024-03-06T07:24:21.530355+0000","last_undegraded":"2024-03-06T07:24:21.530355+0000","last_fullsized":"2024-03-06T07:24:21.530355+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:17.231918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.530988+0000","last_change":"2024-03-06T07:24:20.531681+0000","last_active":"2024-03-06T07:24:21.530988+0000","last_peered":"2024-03-06T07:24:21.530988+0000","last_clean":"2024-03-06T07:24:21.530988+0000","last_became_active":"2024-03-06T07:24:20.531397+0000","last_became_peered":"2024-03-06T07:24:20.531397+0000","last_unstale":"2024-03-06T07:24:21.530988+0000","last_undegraded":"2024-03-06T07:24:21.530988+0000","last_fullsized":"2024-03-06T07:24:21.530988+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:25:59.508321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:22.424880+0000","last_change":"2024-03-06T07:24:19.528439+0000","last_active":"2024-03-06T07:24:22.424880+0000","last_peered":"2024-03-06T07:24:22.424880+0000","last_clean":"2024-03-06T07:24:22.424880+0000","last_became_active":"2024-03-06T07:24:19.528141+0000","last_became_peered":"2024-03-06T07:24:19.528141+0000","last_unstale":"2024-03-06T07:24:22.424880+0000","last_undegraded":"2024-03-06T07:24:22.424880+0000","last_fullsized":"2024-03-06T07:24:22.424880+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:25:58.500915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:22.424873+0000","last_change":"2024-03-06T07:24:20.541419+0000","last_active":"2024-03-06T07:24:22.424873+0000","last_peered":"2024-03-06T07:24:22.424873+0000","last_clean":"2024-03-06T07:24:22.424873+0000","last_became_active":"2024-03-06T07:24:20.540578+0000","last_became_peered":"2024-03-06T07:24:20.540578+0000","last_unstale":"2024-03-06T07:24:22.424873+0000","last_undegraded":"2024-03-06T07:24:22.424873+0000","last_fullsized":"2024-03-06T07:24:22.424873+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:14.609650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.704867+0000","last_change":"2024-03-06T07:24:19.519750+0000","last_active":"2024-03-06T07:24:21.704867+0000","last_peered":"2024-03-06T07:24:21.704867+0000","last_clean":"2024-03-06T07:24:21.704867+0000","last_became_active":"2024-03-06T07:24:19.519503+0000","last_became_peered":"2024-03-06T07:24:19.519503+0000","last_unstale":"2024-03-06T07:24:21.704867+0000","last_undegraded":"2024-03-06T07:24:21.704867+0000","last_fullsized":"2024-03-06T07:24:21.704867+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:32.855542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.529018+0000","last_change":"2024-03-06T07:24:20.536834+0000","last_active":"2024-03-06T07:24:21.529018+0000","last_peered":"2024-03-06T07:24:21.529018+0000","last_clean":"2024-03-06T07:24:21.529018+0000","last_became_active":"2024-03-06T07:24:20.536570+0000","last_became_peered":"2024-03-06T07:24:20.536570+0000","last_unstale":"2024-03-06T07:24:21.529018+0000","last_undegraded":"2024-03-06T07:24:21.529018+0000","last_fullsized":"2024-03-06T07:24:21.529018+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:48.134627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"56'3","reported_seq":17,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:22.425226+0000","last_change":"2024-03-06T07:24:19.528356+0000","last_active":"2024-03-06T07:24:22.425226+0000","last_peered":"2024-03-06T07:24:22.425226+0000","last_clean":"2024-03-06T07:24:22.425226+0000","last_became_active":"2024-03-06T07:24:19.527954+0000","last_became_peered":"2024-03-06T07:24:19.527954+0000","last_unstale":"2024-03-06T07:24:22.425226+0000","last_undegraded":"2024-03-06T07:24:22.425226+0000","last_fullsized":"2024-03-06T07:24:22.425226+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:19.361223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:22.425129+0000","last_change":"2024-03-06T07:24:20.539846+0000","last_active":"2024-03-06T07:24:22.425129+0000","last_peered":"2024-03-06T07:24:22.425129+0000","last_clean":"2024-03-06T07:24:22.425129+0000","last_became_active":"2024-03-06T07:24:20.539587+0000","last_became_peered":"2024-03-06T07:24:20.539587+0000","last_unstale":"2024-03-06T07:24:22.425129+0000","last_undegraded":"2024-03-06T07:24:22.425129+0000","last_fullsized":"2024-03-06T07:24:22.425129+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:24.380167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"56'6","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.531015+0000","last_change":"2024-03-06T07:24:19.523322+0000","last_active":"2024-03-06T07:24:21.531015+0000","last_peered":"2024-03-06T07:24:21.531015+0000","last_clean":"2024-03-06T07:24:21.531015+0000","last_became_active":"2024-03-06T07:24:19.523034+0000","last_became_peered":"2024-03-06T07:24:19.523034+0000","last_unstale":"2024-03-06T07:24:21.531015+0000","last_undegraded":"2024-03-06T07:24:21.531015+0000","last_fullsized":"2024-03-06T07:24:21.531015+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:45.374706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:22.424171+0000","last_change":"2024-03-06T07:24:20.535700+0000","last_active":"2024-03-06T07:24:22.424171+0000","last_peered":"2024-03-06T07:24:22.424171+0000","last_clean":"2024-03-06T07:24:22.424171+0000","last_became_active":"2024-03-06T07:24:20.534992+0000","last_became_peered":"2024-03-06T07:24:20.534992+0000","last_unstale":"2024-03-06T07:24:22.424171+0000","last_undegraded":"2024-03-06T07:24:22.424171+0000","last_fullsized":"2024-03-06T07:24:22.424171+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:02.495918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.704480+0000","last_change":"2024-03-06T07:24:19.526669+0000","last_active":"2024-03-06T07:24:21.704480+0000","last_peered":"2024-03-06T07:24:21.704480+0000","last_clean":"2024-03-06T07:24:21.704480+0000","last_became_active":"2024-03-06T07:24:19.526276+0000","last_became_peered":"2024-03-06T07:24:19.526276+0000","last_unstale":"2024-03-06T07:24:21.704480+0000","last_undegraded":"2024-03-06T07:24:21.704480+0000","last_fullsized":"2024-03-06T07:24:21.704480+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:44.208278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:22.424650+0000","last_change":"2024-03-06T07:24:20.535765+0000","last_active":"2024-03-06T07:24:22.424650+0000","last_peered":"2024-03-06T07:24:22.424650+0000","last_clean":"2024-03-06T07:24:22.424650+0000","last_became_active":"2024-03-06T07:24:20.535454+0000","last_became_peered":"2024-03-06T07:24:20.535454+0000","last_unstale":"2024-03-06T07:24:22.424650+0000","last_undegraded":"2024-03-06T07:24:22.424650+0000","last_fullsized":"2024-03-06T07:24:22.424650+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:04:09.600928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.529799+0000","last_change":"2024-03-06T07:24:19.520321+0000","last_active":"2024-03-06T07:24:21.529799+0000","last_peered":"2024-03-06T07:24:21.529799+0000","last_clean":"2024-03-06T07:24:21.529799+0000","last_became_active":"2024-03-06T07:24:19.520045+0000","last_became_peered":"2024-03-06T07:24:19.520045+0000","last_unstale":"2024-03-06T07:24:21.529799+0000","last_undegraded":"2024-03-06T07:24:21.529799+0000","last_fullsized":"2024-03-06T07:24:21.529799+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:35.724439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.530435+0000","last_change":"2024-03-06T07:24:20.534693+0000","last_active":"2024-03-06T07:24:21.530435+0000","last_peered":"2024-03-06T07:24:21.530435+0000","last_clean":"2024-03-06T07:24:21.530435+0000","last_became_active":"2024-03-06T07:24:20.534415+0000","last_became_peered":"2024-03-06T07:24:20.534415+0000","last_unstale":"2024-03-06T07:24:21.530435+0000","last_undegraded":"2024-03-06T07:24:21.530435+0000","last_fullsized":"2024-03-06T07:24:21.530435+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:02:36.386932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:22.424303+0000","last_change":"2024-03-06T07:24:19.520152+0000","last_active":"2024-03-06T07:24:22.424303+0000","last_peered":"2024-03-06T07:24:22.424303+0000","last_clean":"2024-03-06T07:24:22.424303+0000","last_became_active":"2024-03-06T07:24:19.519855+0000","last_became_peered":"2024-03-06T07:24:19.519855+0000","last_unstale":"2024-03-06T07:24:22.424303+0000","last_undegraded":"2024-03-06T07:24:22.424303+0000","last_fullsized":"2024-03-06T07:24:22.424303+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:44:06.313408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:22.425108+0000","last_change":"2024-03-06T07:24:20.546169+0000","last_active":"2024-03-06T07:24:22.425108+0000","last_peered":"2024-03-06T07:24:22.425108+0000","last_clean":"2024-03-06T07:24:22.425108+0000","last_became_active":"2024-03-06T07:24:20.545915+0000","last_became_peered":"2024-03-06T07:24:20.545915+0000","last_unstale":"2024-03-06T07:24:22.425108+0000","last_undegraded":"2024-03-06T07:24:22.425108+0000","last_fullsized":"2024-03-06T07:24:22.425108+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:35:27.766704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"56'8","reported_seq":22,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:22.424166+0000","last_change":"2024-03-06T07:24:19.525401+0000","last_active":"2024-03-06T07:24:22.424166+0000","last_peered":"2024-03-06T07:24:22.424166+0000","last_clean":"2024-03-06T07:24:22.424166+0000","last_became_active":"2024-03-06T07:24:19.525064+0000","last_became_peered":"2024-03-06T07:24:19.525064+0000","last_unstale":"2024-03-06T07:24:22.424166+0000","last_undegraded":"2024-03-06T07:24:22.424166+0000","last_fullsized":"2024-03-06T07:24:22.424166+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:44:06.313408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:22.424992+0000","last_change":"2024-03-06T07:24:20.538917+0000","last_active":"2024-03-06T07:24:22.424992+0000","last_peered":"2024-03-06T07:24:22.424992+0000","last_clean":"2024-03-06T07:24:22.424992+0000","last_became_active":"2024-03-06T07:24:20.538616+0000","last_became_peered":"2024-03-06T07:24:20.538616+0000","last_unstale":"2024-03-06T07:24:22.424992+0000","last_undegraded":"2024-03-06T07:24:22.424992+0000","last_fullsized":"2024-03-06T07:24:22.424992+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:17:33.713316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.704915+0000","last_change":"2024-03-06T07:24:19.521974+0000","last_active":"2024-03-06T07:24:21.704915+0000","last_peered":"2024-03-06T07:24:21.704915+0000","last_clean":"2024-03-06T07:24:21.704915+0000","last_became_active":"2024-03-06T07:24:19.521736+0000","last_became_peered":"2024-03-06T07:24:19.521736+0000","last_unstale":"2024-03-06T07:24:21.704915+0000","last_undegraded":"2024-03-06T07:24:21.704915+0000","last_fullsized":"2024-03-06T07:24:21.704915+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:09:13.093914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:22.425045+0000","last_change":"2024-03-06T07:24:20.535419+0000","last_active":"2024-03-06T07:24:22.425045+0000","last_peered":"2024-03-06T07:24:22.425045+0000","last_clean":"2024-03-06T07:24:22.425045+0000","last_became_active":"2024-03-06T07:24:20.535152+0000","last_became_peered":"2024-03-06T07:24:20.535152+0000","last_unstale":"2024-03-06T07:24:22.425045+0000","last_undegraded":"2024-03-06T07:24:22.425045+0000","last_fullsized":"2024-03-06T07:24:22.425045+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:59:00.131064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.536758+0000","last_change":"2024-03-06T07:24:19.523871+0000","last_active":"2024-03-06T07:24:21.536758+0000","last_peered":"2024-03-06T07:24:21.536758+0000","last_clean":"2024-03-06T07:24:21.536758+0000","last_became_active":"2024-03-06T07:24:19.523587+0000","last_became_peered":"2024-03-06T07:24:19.523587+0000","last_unstale":"2024-03-06T07:24:21.536758+0000","last_undegraded":"2024-03-06T07:24:21.536758+0000","last_fullsized":"2024-03-06T07:24:21.536758+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:20.649547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.704685+0000","last_change":"2024-03-06T07:24:20.540178+0000","last_active":"2024-03-06T07:24:21.704685+0000","last_peered":"2024-03-06T07:24:21.704685+0000","last_clean":"2024-03-06T07:24:21.704685+0000","last_became_active":"2024-03-06T07:24:20.539909+0000","last_became_peered":"2024-03-06T07:24:20.539909+0000","last_unstale":"2024-03-06T07:24:21.704685+0000","last_undegraded":"2024-03-06T07:24:21.704685+0000","last_fullsized":"2024-03-06T07:24:21.704685+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:47.928688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:22.424816+0000","last_change":"2024-03-06T07:24:19.520212+0000","last_active":"2024-03-06T07:24:22.424816+0000","last_peered":"2024-03-06T07:24:22.424816+0000","last_clean":"2024-03-06T07:24:22.424816+0000","last_became_active":"2024-03-06T07:24:19.519913+0000","last_became_peered":"2024-03-06T07:24:19.519913+0000","last_unstale":"2024-03-06T07:24:22.424816+0000","last_undegraded":"2024-03-06T07:24:22.424816+0000","last_fullsized":"2024-03-06T07:24:22.424816+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:20:20.174477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:22.425196+0000","last_change":"2024-03-06T07:24:20.541159+0000","last_active":"2024-03-06T07:24:22.425196+0000","last_peered":"2024-03-06T07:24:22.425196+0000","last_clean":"2024-03-06T07:24:22.425196+0000","last_became_active":"2024-03-06T07:24:20.540832+0000","last_became_peered":"2024-03-06T07:24:20.540832+0000","last_unstale":"2024-03-06T07:24:22.425196+0000","last_undegraded":"2024-03-06T07:24:22.425196+0000","last_fullsized":"2024-03-06T07:24:22.425196+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:02.300856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.536792+0000","last_change":"2024-03-06T07:24:19.520035+0000","last_active":"2024-03-06T07:24:21.536792+0000","last_peered":"2024-03-06T07:24:21.536792+0000","last_clean":"2024-03-06T07:24:21.536792+0000","last_became_active":"2024-03-06T07:24:19.519761+0000","last_became_peered":"2024-03-06T07:24:19.519761+0000","last_unstale":"2024-03-06T07:24:21.536792+0000","last_undegraded":"2024-03-06T07:24:21.536792+0000","last_fullsized":"2024-03-06T07:24:21.536792+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:09.161121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:22.424427+0000","last_change":"2024-03-06T07:24:20.534605+0000","last_active":"2024-03-06T07:24:22.424427+0000","last_peered":"2024-03-06T07:24:22.424427+0000","last_clean":"2024-03-06T07:24:22.424427+0000","last_became_active":"2024-03-06T07:24:20.534345+0000","last_became_peered":"2024-03-06T07:24:20.534345+0000","last_unstale":"2024-03-06T07:24:22.424427+0000","last_undegraded":"2024-03-06T07:24:22.424427+0000","last_fullsized":"2024-03-06T07:24:22.424427+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:59:03.575875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.704625+0000","last_change":"2024-03-06T07:24:19.515544+0000","last_active":"2024-03-06T07:24:21.704625+0000","last_peered":"2024-03-06T07:24:21.704625+0000","last_clean":"2024-03-06T07:24:21.704625+0000","last_became_active":"2024-03-06T07:24:19.515266+0000","last_became_peered":"2024-03-06T07:24:19.515266+0000","last_unstale":"2024-03-06T07:24:21.704625+0000","last_undegraded":"2024-03-06T07:24:21.704625+0000","last_fullsized":"2024-03-06T07:24:21.704625+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:09.239738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.536569+0000","last_change":"2024-03-06T07:24:20.527697+0000","last_active":"2024-03-06T07:24:21.536569+0000","last_peered":"2024-03-06T07:24:21.536569+0000","last_clean":"2024-03-06T07:24:21.536569+0000","last_became_active":"2024-03-06T07:24:20.527434+0000","last_became_peered":"2024-03-06T07:24:20.527434+0000","last_unstale":"2024-03-06T07:24:21.536569+0000","last_undegraded":"2024-03-06T07:24:21.536569+0000","last_fullsized":"2024-03-06T07:24:21.536569+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:51:10.863241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.528937+0000","last_change":"2024-03-06T07:24:19.521024+0000","last_active":"2024-03-06T07:24:21.528937+0000","last_peered":"2024-03-06T07:24:21.528937+0000","last_clean":"2024-03-06T07:24:21.528937+0000","last_became_active":"2024-03-06T07:24:19.520754+0000","last_became_peered":"2024-03-06T07:24:19.520754+0000","last_unstale":"2024-03-06T07:24:21.528937+0000","last_undegraded":"2024-03-06T07:24:21.528937+0000","last_fullsized":"2024-03-06T07:24:21.528937+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:00:32.355340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.536344+0000","last_change":"2024-03-06T07:24:20.531103+0000","last_active":"2024-03-06T07:24:21.536344+0000","last_peered":"2024-03-06T07:24:21.536344+0000","last_clean":"2024-03-06T07:24:21.536344+0000","last_became_active":"2024-03-06T07:24:20.530839+0000","last_became_peered":"2024-03-06T07:24:20.530839+0000","last_unstale":"2024-03-06T07:24:21.536344+0000","last_undegraded":"2024-03-06T07:24:21.536344+0000","last_fullsized":"2024-03-06T07:24:21.536344+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:50:29.241792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:22.424122+0000","last_change":"2024-03-06T07:24:19.519653+0000","last_active":"2024-03-06T07:24:22.424122+0000","last_peered":"2024-03-06T07:24:22.424122+0000","last_clean":"2024-03-06T07:24:22.424122+0000","last_became_active":"2024-03-06T07:24:19.519374+0000","last_became_peered":"2024-03-06T07:24:19.519374+0000","last_unstale":"2024-03-06T07:24:22.424122+0000","last_undegraded":"2024-03-06T07:24:22.424122+0000","last_fullsized":"2024-03-06T07:24:22.424122+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:24:36.321071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:22.424872+0000","last_change":"2024-03-06T07:24:20.537837+0000","last_active":"2024-03-06T07:24:22.424872+0000","last_peered":"2024-03-06T07:24:22.424872+0000","last_clean":"2024-03-06T07:24:22.424872+0000","last_became_active":"2024-03-06T07:24:20.537456+0000","last_became_peered":"2024-03-06T07:24:20.537456+0000","last_unstale":"2024-03-06T07:24:22.424872+0000","last_undegraded":"2024-03-06T07:24:22.424872+0000","last_fullsized":"2024-03-06T07:24:22.424872+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:48:29.376354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.529291+0000","last_change":"2024-03-06T07:24:19.527152+0000","last_active":"2024-03-06T07:24:21.529291+0000","last_peered":"2024-03-06T07:24:21.529291+0000","last_clean":"2024-03-06T07:24:21.529291+0000","last_became_active":"2024-03-06T07:24:19.526879+0000","last_became_peered":"2024-03-06T07:24:19.526879+0000","last_unstale":"2024-03-06T07:24:21.529291+0000","last_undegraded":"2024-03-06T07:24:21.529291+0000","last_fullsized":"2024-03-06T07:24:21.529291+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:04:29.097137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:22.425075+0000","last_change":"2024-03-06T07:24:20.537184+0000","last_active":"2024-03-06T07:24:22.425075+0000","last_peered":"2024-03-06T07:24:22.425075+0000","last_clean":"2024-03-06T07:24:22.425075+0000","last_became_active":"2024-03-06T07:24:20.536912+0000","last_became_peered":"2024-03-06T07:24:20.536912+0000","last_unstale":"2024-03-06T07:24:22.425075+0000","last_undegraded":"2024-03-06T07:24:22.425075+0000","last_fullsized":"2024-03-06T07:24:22.425075+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:14.609650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"57'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.536887+0000","last_change":"2024-03-06T07:24:19.520942+0000","last_active":"2024-03-06T07:24:21.536887+0000","last_peered":"2024-03-06T07:24:21.536887+0000","last_clean":"2024-03-06T07:24:21.536887+0000","last_became_active":"2024-03-06T07:24:19.520672+0000","last_became_peered":"2024-03-06T07:24:19.520672+0000","last_unstale":"2024-03-06T07:24:21.536887+0000","last_undegraded":"2024-03-06T07:24:21.536887+0000","last_fullsized":"2024-03-06T07:24:21.536887+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:38.333961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.529096+0000","last_change":"2024-03-06T07:24:20.536201+0000","last_active":"2024-03-06T07:24:21.529096+0000","last_peered":"2024-03-06T07:24:21.529096+0000","last_clean":"2024-03-06T07:24:21.529096+0000","last_became_active":"2024-03-06T07:24:20.535930+0000","last_became_peered":"2024-03-06T07:24:20.535930+0000","last_unstale":"2024-03-06T07:24:21.529096+0000","last_undegraded":"2024-03-06T07:24:21.529096+0000","last_fullsized":"2024-03-06T07:24:21.529096+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:14:08.976212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.530030+0000","last_change":"2024-03-06T07:24:19.522226+0000","last_active":"2024-03-06T07:24:21.530030+0000","last_peered":"2024-03-06T07:24:21.530030+0000","last_clean":"2024-03-06T07:24:21.530030+0000","last_became_active":"2024-03-06T07:24:19.521882+0000","last_became_peered":"2024-03-06T07:24:19.521882+0000","last_unstale":"2024-03-06T07:24:21.530030+0000","last_undegraded":"2024-03-06T07:24:21.530030+0000","last_fullsized":"2024-03-06T07:24:21.530030+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:04:29.097137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.536765+0000","last_change":"2024-03-06T07:24:20.538822+0000","last_active":"2024-03-06T07:24:21.536765+0000","last_peered":"2024-03-06T07:24:21.536765+0000","last_clean":"2024-03-06T07:24:21.536765+0000","last_became_active":"2024-03-06T07:24:20.538272+0000","last_became_peered":"2024-03-06T07:24:20.538272+0000","last_unstale":"2024-03-06T07:24:21.536765+0000","last_undegraded":"2024-03-06T07:24:21.536765+0000","last_fullsized":"2024-03-06T07:24:21.536765+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:59:00.131064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:22.425407+0000","last_change":"2024-03-06T07:24:19.516436+0000","last_active":"2024-03-06T07:24:22.425407+0000","last_peered":"2024-03-06T07:24:22.425407+0000","last_clean":"2024-03-06T07:24:22.425407+0000","last_became_active":"2024-03-06T07:24:19.515998+0000","last_became_peered":"2024-03-06T07:24:19.515998+0000","last_unstale":"2024-03-06T07:24:22.425407+0000","last_undegraded":"2024-03-06T07:24:22.425407+0000","last_fullsized":"2024-03-06T07:24:22.425407+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:00:04.275748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.704811+0000","last_change":"2024-03-06T07:24:20.535877+0000","last_active":"2024-03-06T07:24:21.704811+0000","last_peered":"2024-03-06T07:24:21.704811+0000","last_clean":"2024-03-06T07:24:21.704811+0000","last_became_active":"2024-03-06T07:24:20.535637+0000","last_became_peered":"2024-03-06T07:24:20.535637+0000","last_unstale":"2024-03-06T07:24:21.704811+0000","last_undegraded":"2024-03-06T07:24:21.704811+0000","last_fullsized":"2024-03-06T07:24:21.704811+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:56.725801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.528988+0000","last_change":"2024-03-06T07:24:19.521792+0000","last_active":"2024-03-06T07:24:21.528988+0000","last_peered":"2024-03-06T07:24:21.528988+0000","last_clean":"2024-03-06T07:24:21.528988+0000","last_became_active":"2024-03-06T07:24:19.521565+0000","last_became_peered":"2024-03-06T07:24:19.521565+0000","last_unstale":"2024-03-06T07:24:21.528988+0000","last_undegraded":"2024-03-06T07:24:21.528988+0000","last_fullsized":"2024-03-06T07:24:21.528988+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:24.269083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:22.424820+0000","last_change":"2024-03-06T07:24:20.547927+0000","last_active":"2024-03-06T07:24:22.424820+0000","last_peered":"2024-03-06T07:24:22.424820+0000","last_clean":"2024-03-06T07:24:22.424820+0000","last_became_active":"2024-03-06T07:24:20.547663+0000","last_became_peered":"2024-03-06T07:24:20.547663+0000","last_unstale":"2024-03-06T07:24:22.424820+0000","last_undegraded":"2024-03-06T07:24:22.424820+0000","last_fullsized":"2024-03-06T07:24:22.424820+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:59:00.131064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.530753+0000","last_change":"2024-03-06T07:24:19.524110+0000","last_active":"2024-03-06T07:24:21.530753+0000","last_peered":"2024-03-06T07:24:21.530753+0000","last_clean":"2024-03-06T07:24:21.530753+0000","last_became_active":"2024-03-06T07:24:19.523812+0000","last_became_peered":"2024-03-06T07:24:19.523812+0000","last_unstale":"2024-03-06T07:24:21.530753+0000","last_undegraded":"2024-03-06T07:24:21.530753+0000","last_fullsized":"2024-03-06T07:24:21.530753+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:15.049439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.528850+0000","last_change":"2024-03-06T07:24:20.532528+0000","last_active":"2024-03-06T07:24:21.528850+0000","last_peered":"2024-03-06T07:24:21.528850+0000","last_clean":"2024-03-06T07:24:21.528850+0000","last_became_active":"2024-03-06T07:24:20.532272+0000","last_became_peered":"2024-03-06T07:24:20.532272+0000","last_unstale":"2024-03-06T07:24:21.528850+0000","last_undegraded":"2024-03-06T07:24:21.528850+0000","last_fullsized":"2024-03-06T07:24:21.528850+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:01.123105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.528879+0000","last_change":"2024-03-06T07:24:19.520220+0000","last_active":"2024-03-06T07:24:21.528879+0000","last_peered":"2024-03-06T07:24:21.528879+0000","last_clean":"2024-03-06T07:24:21.528879+0000","last_became_active":"2024-03-06T07:24:19.519962+0000","last_became_peered":"2024-03-06T07:24:19.519962+0000","last_unstale":"2024-03-06T07:24:21.528879+0000","last_undegraded":"2024-03-06T07:24:21.528879+0000","last_fullsized":"2024-03-06T07:24:21.528879+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:49.455455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.704690+0000","last_change":"2024-03-06T07:24:20.531174+0000","last_active":"2024-03-06T07:24:21.704690+0000","last_peered":"2024-03-06T07:24:21.704690+0000","last_clean":"2024-03-06T07:24:21.704690+0000","last_became_active":"2024-03-06T07:24:20.530903+0000","last_became_peered":"2024-03-06T07:24:20.530903+0000","last_unstale":"2024-03-06T07:24:21.704690+0000","last_undegraded":"2024-03-06T07:24:21.704690+0000","last_fullsized":"2024-03-06T07:24:21.704690+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:48:37.815385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.704525+0000","last_change":"2024-03-06T07:24:19.516940+0000","last_active":"2024-03-06T07:24:21.704525+0000","last_peered":"2024-03-06T07:24:21.704525+0000","last_clean":"2024-03-06T07:24:21.704525+0000","last_became_active":"2024-03-06T07:24:19.516691+0000","last_became_peered":"2024-03-06T07:24:19.516691+0000","last_unstale":"2024-03-06T07:24:21.704525+0000","last_undegraded":"2024-03-06T07:24:21.704525+0000","last_fullsized":"2024-03-06T07:24:21.704525+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:36:37.404646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.704439+0000","last_change":"2024-03-06T07:24:20.528793+0000","last_active":"2024-03-06T07:24:21.704439+0000","last_peered":"2024-03-06T07:24:21.704439+0000","last_clean":"2024-03-06T07:24:21.704439+0000","last_became_active":"2024-03-06T07:24:20.528524+0000","last_became_peered":"2024-03-06T07:24:20.528524+0000","last_unstale":"2024-03-06T07:24:21.704439+0000","last_undegraded":"2024-03-06T07:24:21.704439+0000","last_fullsized":"2024-03-06T07:24:21.704439+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:19.501657+0000","last_clean_scrub_stamp":"2024-03-06T07:24:19.501657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:22.140546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:21.529866+0000","last_change":"2024-03-06T07:24:19.522294+0000","last_active":"2024-03-06T07:24:21.529866+0000","last_peered":"2024-03-06T07:24:21.529866+0000","last_clean":"2024-03-06T07:24:21.529866+0000","last_became_active":"2024-03-06T07:24:19.521845+0000","last_became_peered":"2024-03-06T07:24:19.521845+0000","last_unstale":"2024-03-06T07:24:21.529866+0000","last_undegraded":"2024-03-06T07:24:21.529866+0000","last_fullsized":"2024-03-06T07:24:21.529866+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:18.494250+0000","last_clean_scrub_stamp":"2024-03-06T07:24:18.494250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:49.455455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":236,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:22.424235+0000","last_change":"2024-03-06T07:21:47.553693+0000","last_active":"2024-03-06T07:24:22.424235+0000","last_peered":"2024-03-06T07:24:22.424235+0000","last_clean":"2024-03-06T07:24:22.424235+0000","last_became_active":"2024-03-06T07:21:47.552664+0000","last_became_peered":"2024-03-06T07:21:47.552664+0000","last_unstale":"2024-03-06T07:24:22.424235+0000","last_undegraded":"2024-03-06T07:24:22.424235+0000","last_fullsized":"2024-03-06T07:24:22.424235+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_clean_scrub_stamp":"2024-03-06T07:21:46.362028+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:02.916426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574885,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32440,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339400,"statfs":{"total":96636764160,"available":96603545600,"internally_reserved":0,"allocated":1286144,"data_stored":874652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574883,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31852,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339988,"statfs":{"total":96636764160,"available":96604147712,"internally_reserved":0,"allocated":684032,"data_stored":284124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574884,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31860,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339980,"statfs":{"total":96636764160,"available":96604139520,"internally_reserved":0,"allocated":692224,"data_stored":284776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574885,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31864,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339976,"statfs":{"total":96636764160,"available":96604135424,"internally_reserved":0,"allocated":696320,"data_stored":285736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574884,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31860,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339980,"statfs":{"total":96636764160,"available":96604139520,"internally_reserved":0,"allocated":692224,"data_stored":284180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574884,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31860,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339980,"statfs":{"total":96636764160,"available":96604139520,"internally_reserved":0,"allocated":692224,"data_stored":284180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574885,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31856,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339984,"statfs":{"total":96636764160,"available":96604143616,"internally_reserved":0,"allocated":688128,"data_stored":284214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574884,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32452,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339388,"statfs":{"total":96636764160,"available":96603533312,"internally_reserved":0,"allocated":1294336,"data_stored":876686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:28.336 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:24:28.337 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:24:28.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.481+0000 7f942ca88640 1 Processor -- start 2024-03-06T07:24:28.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.481+0000 7f942ca88640 1 -- start start 2024-03-06T07:24:28.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.482+0000 7f942ca88640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f94281614d0 0x7f94280d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.482+0000 7f942ca88640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f94280d1910 0x7f94280d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.482+0000 7f942ca88640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f94280d22b0 0x7f94280d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.482+0000 7f942ca88640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94280d2740 con 0x7f94280d22b0 2024-03-06T07:24:28.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.482+0000 7f942ca88640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94281618b0 con 0x7f94281614d0 2024-03-06T07:24:28.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.482+0000 7f942ca88640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94280cf400 con 0x7f94280d1910 2024-03-06T07:24:28.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.482+0000 7f9426d9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f94280d22b0 0x7f94280d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.482+0000 7f9426d9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f94280d22b0 0x7f94280d06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:42974/0 (socket says 172.21.15.83:42974) 2024-03-06T07:24:28.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.482+0000 7f9426d9d640 1 -- 172.21.15.83:0/2807838997 learned_addr learned my addr 172.21.15.83:0/2807838997 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:28.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.482+0000 7f9425d9b640 1 --2- 172.21.15.83:0/2807838997 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f94280d1910 0x7f94280d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.482+0000 7f942659c640 1 --2- 172.21.15.83:0/2807838997 >> v2:172.21.15.183:3300/0 conn(0x7f94281614d0 0x7f94280d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.482+0000 7f9426d9d640 1 -- 172.21.15.83:0/2807838997 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f94280d1910 msgr2=0x7f94280d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:28.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.482+0000 7f9426d9d640 1 --2- 172.21.15.83:0/2807838997 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f94280d1910 0x7f94280d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.482+0000 7f9426d9d640 1 -- 172.21.15.83:0/2807838997 >> v2:172.21.15.183:3300/0 conn(0x7f94281614d0 msgr2=0x7f94280d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:28.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.482+0000 7f9426d9d640 1 --2- 172.21.15.83:0/2807838997 >> v2:172.21.15.183:3300/0 conn(0x7f94281614d0 0x7f94280d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.482+0000 7f9426d9d640 1 -- 172.21.15.83:0/2807838997 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94280cf6a0 con 0x7f94280d22b0 2024-03-06T07:24:28.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.482+0000 7f9425d9b640 1 --2- 172.21.15.83:0/2807838997 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f94280d1910 0x7f94280d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:28.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.482+0000 7f942659c640 1 --2- 172.21.15.83:0/2807838997 >> v2:172.21.15.183:3300/0 conn(0x7f94281614d0 0x7f94280d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:28.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.483+0000 7f9426d9d640 1 --2- 172.21.15.83:0/2807838997 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94280d22b0 0x7f94280d06a0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f942808c470 tx=0x7f942002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e69061ba87c26e0c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:28.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.483+0000 7f942559a640 1 -- 172.21.15.83:0/2807838997 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942002ca70 con 0x7f94280d22b0 2024-03-06T07:24:28.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.483+0000 7f942559a640 1 -- 172.21.15.83:0/2807838997 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f942002cc10 con 0x7f94280d22b0 2024-03-06T07:24:28.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.483+0000 7f942559a640 1 -- 172.21.15.83:0/2807838997 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94200343f0 con 0x7f94280d22b0 2024-03-06T07:24:28.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.484+0000 7f942ca88640 1 -- 172.21.15.83:0/2807838997 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94280d22b0 msgr2=0x7f94280d06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.484+0000 7f942ca88640 1 --2- 172.21.15.83:0/2807838997 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94280d22b0 0x7f94280d06a0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f942808c470 tx=0x7f942002bed0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.484+0000 7f942ca88640 1 -- 172.21.15.83:0/2807838997 shutdown_connections 2024-03-06T07:24:28.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.484+0000 7f942ca88640 1 --2- 172.21.15.83:0/2807838997 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94280d22b0 0x7f94280d06a0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.484+0000 7f942ca88640 1 --2- 172.21.15.83:0/2807838997 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f94280d1910 0x7f94280d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.484+0000 7f942ca88640 1 --2- 172.21.15.83:0/2807838997 >> v2:172.21.15.183:3300/0 conn(0x7f94281614d0 0x7f94280d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.484+0000 7f942ca88640 1 -- 172.21.15.83:0/2807838997 >> 172.21.15.83:0/2807838997 conn(0x7f94280d3d40 msgr2=0x7f94280d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:28.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.484+0000 7f942ca88640 1 -- 172.21.15.83:0/2807838997 shutdown_connections 2024-03-06T07:24:28.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.484+0000 7f942ca88640 1 -- 172.21.15.83:0/2807838997 wait complete. 2024-03-06T07:24:28.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.485+0000 7f942ca88640 1 Processor -- start 2024-03-06T07:24:28.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.485+0000 7f942ca88640 1 -- start start 2024-03-06T07:24:28.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.486+0000 7f942ca88640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94280d1910 0x7f942815c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.486+0000 7f942ca88640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f94280d22b0 0x7f94281531d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.486+0000 7f942ca88640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f94281614d0 0x7f94280b6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.486+0000 7f942ca88640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94280b0910 con 0x7f94280d1910 2024-03-06T07:24:28.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.486+0000 7f942ca88640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94280cfc60 con 0x7f94280d22b0 2024-03-06T07:24:28.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.486+0000 7f942ca88640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94280d0be0 con 0x7f94281614d0 2024-03-06T07:24:28.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.486+0000 7f942659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94280d1910 0x7f942815c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.486+0000 7f942659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94280d1910 0x7f942815c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:42982/0 (socket says 172.21.15.83:42982) 2024-03-06T07:24:28.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.486+0000 7f942659c640 1 -- 172.21.15.83:0/3577458874 learned_addr learned my addr 172.21.15.83:0/3577458874 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:28.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.486+0000 7f9426d9d640 1 --2- 172.21.15.83:0/3577458874 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f94281614d0 0x7f94280b6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.486+0000 7f9425d9b640 1 --2- 172.21.15.83:0/3577458874 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f94280d22b0 0x7f94281531d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.486+0000 7f942659c640 1 -- 172.21.15.83:0/3577458874 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f94281614d0 msgr2=0x7f94280b6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.486+0000 7f942659c640 1 --2- 172.21.15.83:0/3577458874 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f94281614d0 0x7f94280b6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.486+0000 7f942659c640 1 -- 172.21.15.83:0/3577458874 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f94280d22b0 msgr2=0x7f94281531d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.486+0000 7f942659c640 1 --2- 172.21.15.83:0/3577458874 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f94280d22b0 0x7f94281531d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.486+0000 7f942659c640 1 -- 172.21.15.83:0/3577458874 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9428153710 con 0x7f94280d1910 2024-03-06T07:24:28.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.486+0000 7f942659c640 1 --2- 172.21.15.83:0/3577458874 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94280d1910 0x7f942815c140 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f941800c7f0 tx=0x7f941800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:28.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.487+0000 7f9416ffd640 1 -- 172.21.15.83:0/3577458874 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9418015070 con 0x7f94280d1910 2024-03-06T07:24:28.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.487+0000 7f9416ffd640 1 -- 172.21.15.83:0/3577458874 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9418007820 con 0x7f94280d1910 2024-03-06T07:24:28.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.487+0000 7f942ca88640 1 -- 172.21.15.83:0/3577458874 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94280b7180 con 0x7f94280d1910 2024-03-06T07:24:28.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.487+0000 7f9416ffd640 1 -- 172.21.15.83:0/3577458874 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9418007bd0 con 0x7f94280d1910 2024-03-06T07:24:28.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.487+0000 7f942ca88640 1 -- 172.21.15.83:0/3577458874 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94280b7740 con 0x7f94280d1910 2024-03-06T07:24:28.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.488+0000 7f942ca88640 1 -- 172.21.15.83:0/3577458874 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94280cef30 con 0x7f94280d1910 2024-03-06T07:24:28.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.488+0000 7f9416ffd640 1 -- 172.21.15.83:0/3577458874 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9418018070 con 0x7f94280d1910 2024-03-06T07:24:28.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.489+0000 7f9416ffd640 1 --2- 172.21.15.83:0/3577458874 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f940806b4d0 0x7f940806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.489+0000 7f9416ffd640 1 -- 172.21.15.83:0/3577458874 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f941808a2c0 con 0x7f94280d1910 2024-03-06T07:24:28.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.489+0000 7f9425d9b640 1 --2- 172.21.15.83:0/3577458874 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f940806b4d0 0x7f940806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.489+0000 7f9425d9b640 1 --2- 172.21.15.83:0/3577458874 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f940806b4d0 0x7f940806d990 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f94280ce1d0 tx=0x7f941c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:28.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.493+0000 7f9416ffd640 1 -- 172.21.15.83:0/3577458874 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9428153710 con 0x7f94280d1910 2024-03-06T07:24:28.657 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:24:28.659 DEBUG:teuthology.orchestra.run.smithi183:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:24:28.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.809+0000 7f942ca88640 1 -- 172.21.15.83:0/3577458874 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f9428068ff0 con 0x7f94280d1910 2024-03-06T07:24:28.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.811+0000 7f9416ffd640 1 -- 172.21.15.83:0/3577458874 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 190+0+147 (secure 0 0 0) 0x7f9418059140 con 0x7f94280d1910 2024-03-06T07:24:28.812 INFO:teuthology.orchestra.run.smithi083.stdout:[client.0] 2024-03-06T07:24:28.813 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBcGehlRwCIMhAA1DdbdQZPplu7LQOTK3BmSg== 2024-03-06T07:24:28.813 INFO:teuthology.orchestra.run.smithi083.stdout: caps mds = "allow" 2024-03-06T07:24:28.813 INFO:teuthology.orchestra.run.smithi083.stdout: caps mon = "allow r" 2024-03-06T07:24:28.813 INFO:teuthology.orchestra.run.smithi083.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:24:28.813 INFO:teuthology.orchestra.run.smithi083.stderr:updated caps for client.0 2024-03-06T07:24:28.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.813+0000 7f942ca88640 1 -- 172.21.15.83:0/3577458874 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f940806b4d0 msgr2=0x7f940806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.813+0000 7f942ca88640 1 --2- 172.21.15.83:0/3577458874 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f940806b4d0 0x7f940806d990 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f94280ce1d0 tx=0x7f941c008040 comp rx=0 tx=0).stop 2024-03-06T07:24:28.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.813+0000 7f942ca88640 1 -- 172.21.15.83:0/3577458874 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94280d1910 msgr2=0x7f942815c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.813+0000 7f942ca88640 1 --2- 172.21.15.83:0/3577458874 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94280d1910 0x7f942815c140 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f941800c7f0 tx=0x7f941800ccb0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.813+0000 7f942ca88640 1 -- 172.21.15.83:0/3577458874 shutdown_connections 2024-03-06T07:24:28.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.813+0000 7f942ca88640 1 --2- 172.21.15.83:0/3577458874 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f940806b4d0 0x7f940806d990 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.813+0000 7f942ca88640 1 --2- 172.21.15.83:0/3577458874 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f94281614d0 0x7f94280b6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.813+0000 7f942ca88640 1 --2- 172.21.15.83:0/3577458874 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f94280d22b0 0x7f94281531d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.813+0000 7f942ca88640 1 --2- 172.21.15.83:0/3577458874 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94280d1910 0x7f942815c140 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.813+0000 7f942ca88640 1 -- 172.21.15.83:0/3577458874 >> 172.21.15.83:0/3577458874 conn(0x7f94280d3d40 msgr2=0x7f94280b1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:28.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.813+0000 7f942ca88640 1 -- 172.21.15.83:0/3577458874 shutdown_connections 2024-03-06T07:24:28.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.813+0000 7f942ca88640 1 -- 172.21.15.83:0/3577458874 wait complete. 2024-03-06T07:24:28.826 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:28.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.929+0000 7fc95777a640 1 Processor -- start 2024-03-06T07:24:28.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.929+0000 7fc95777a640 1 -- start start 2024-03-06T07:24:28.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.930+0000 7fc95777a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc95015dfe0 0x7fc9500cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.930+0000 7fc95777a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc9500d29c0 0x7fc9500b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.930+0000 7fc95777a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9500b3430 con 0x7fc9500cf830 2024-03-06T07:24:28.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.930+0000 7fc95777a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9500b31e0 con 0x7fc95015dfe0 2024-03-06T07:24:28.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.930+0000 7fc95777a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9500b3d60 con 0x7fc9500d29c0 2024-03-06T07:24:28.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.930+0000 7fc955516640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc95015dfe0 0x7fc9500cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.930+0000 7fc955516640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc95015dfe0 0x7fc9500cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:42992/0 (socket says 172.21.15.83:42992) 2024-03-06T07:24:28.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.930+0000 7fc955516640 1 -- 172.21.15.83:0/3059721025 learned_addr learned my addr 172.21.15.83:0/3059721025 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:28.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.930+0000 7fc955d17640 1 --2- 172.21.15.83:0/3059721025 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc9500d29c0 0x7fc9500b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.930+0000 7fc955516640 1 -- 172.21.15.83:0/3059721025 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc9500d29c0 msgr2=0x7fc9500b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:28.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.930+0000 7fc955516640 1 --2- 172.21.15.83:0/3059721025 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc9500d29c0 0x7fc9500b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.930+0000 7fc955516640 1 -- 172.21.15.83:0/3059721025 >> v1:172.21.15.183:6789/0 conn(0x7fc9500cf830 legacy=0x7fc9500d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:28.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.930+0000 7fc955516640 1 -- 172.21.15.83:0/3059721025 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9500b3710 con 0x7fc95015dfe0 2024-03-06T07:24:28.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.931+0000 7fc955516640 1 --2- 172.21.15.83:0/3059721025 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc95015dfe0 0x7fc9500cf2f0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fc94c009870 tx=0x7fc94c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=604e0ab7c60caa92 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:28.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.931+0000 7fc947fff640 1 -- 172.21.15.83:0/3059721025 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc94c034070 con 0x7fc95015dfe0 2024-03-06T07:24:28.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.931+0000 7fc947fff640 1 -- 172.21.15.83:0/3059721025 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc94c030840 con 0x7fc95015dfe0 2024-03-06T07:24:28.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.931+0000 7fc947fff640 1 -- 172.21.15.83:0/3059721025 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc94c030c10 con 0x7fc95015dfe0 2024-03-06T07:24:28.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.931+0000 7fc95777a640 1 -- 172.21.15.83:0/3059721025 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc95015dfe0 msgr2=0x7fc9500cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.931+0000 7fc95777a640 1 --2- 172.21.15.83:0/3059721025 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc95015dfe0 0x7fc9500cf2f0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fc94c009870 tx=0x7fc94c02b980 comp rx=0 tx=0).stop 2024-03-06T07:24:28.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.932+0000 7fc95777a640 1 -- 172.21.15.83:0/3059721025 shutdown_connections 2024-03-06T07:24:28.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.932+0000 7fc95777a640 1 --2- 172.21.15.83:0/3059721025 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc9500d29c0 0x7fc9500b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.932+0000 7fc95777a640 1 --2- 172.21.15.83:0/3059721025 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc95015dfe0 0x7fc9500cf2f0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.932+0000 7fc95777a640 1 -- 172.21.15.83:0/3059721025 >> 172.21.15.83:0/3059721025 conn(0x7fc9500d3cd0 msgr2=0x7fc9500d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:28.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.932+0000 7fc95777a640 1 -- 172.21.15.83:0/3059721025 shutdown_connections 2024-03-06T07:24:28.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.932+0000 7fc95777a640 1 -- 172.21.15.83:0/3059721025 wait complete. 2024-03-06T07:24:28.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.933+0000 7fc95777a640 1 Processor -- start 2024-03-06T07:24:28.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.933+0000 7fc95777a640 1 -- start start 2024-03-06T07:24:28.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.933+0000 7fc95777a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc9500cf830 0x7fc9500ccb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.933+0000 7fc95777a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9500d29c0 0x7fc9500b1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.933+0000 7fc95777a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc95015dfe0 0x7fc95015b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.933+0000 7fc95777a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9500d1df0 con 0x7fc9500d29c0 2024-03-06T07:24:28.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.933+0000 7fc95777a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9500d1c70 con 0x7fc95015dfe0 2024-03-06T07:24:28.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.933+0000 7fc95777a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc95015e3c0 con 0x7fc9500cf830 2024-03-06T07:24:28.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.934+0000 7fc954d15640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9500d29c0 0x7fc9500b1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.934+0000 7fc954d15640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9500d29c0 0x7fc9500b1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:42998/0 (socket says 172.21.15.83:42998) 2024-03-06T07:24:28.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.934+0000 7fc954d15640 1 -- 172.21.15.83:0/1883772095 learned_addr learned my addr 172.21.15.83:0/1883772095 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:28.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.934+0000 7fc955516640 1 --2- 172.21.15.83:0/1883772095 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc9500cf830 0x7fc9500ccb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.934+0000 7fc955d17640 1 --2- 172.21.15.83:0/1883772095 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc95015dfe0 0x7fc95015b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.934+0000 7fc954d15640 1 -- 172.21.15.83:0/1883772095 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc9500cf830 msgr2=0x7fc9500ccb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.934+0000 7fc954d15640 1 --2- 172.21.15.83:0/1883772095 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc9500cf830 0x7fc9500ccb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.934+0000 7fc954d15640 1 -- 172.21.15.83:0/1883772095 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc95015dfe0 msgr2=0x7fc95015b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.934+0000 7fc954d15640 1 --2- 172.21.15.83:0/1883772095 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc95015dfe0 0x7fc95015b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.934+0000 7fc954d15640 1 -- 172.21.15.83:0/1883772095 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9500cbad0 con 0x7fc9500d29c0 2024-03-06T07:24:28.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.934+0000 7fc954d15640 1 --2- 172.21.15.83:0/1883772095 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9500d29c0 0x7fc9500b1e30 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fc94000c470 tx=0x7fc940001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:28.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.934+0000 7fc945ffb640 1 -- 172.21.15.83:0/1883772095 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9400052d0 con 0x7fc9500d29c0 2024-03-06T07:24:28.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.934+0000 7fc945ffb640 1 -- 172.21.15.83:0/1883772095 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc940005470 con 0x7fc9500d29c0 2024-03-06T07:24:28.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.934+0000 7fc95777a640 1 -- 172.21.15.83:0/1883772095 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9500cbdb0 con 0x7fc9500d29c0 2024-03-06T07:24:28.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.934+0000 7fc945ffb640 1 -- 172.21.15.83:0/1883772095 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc940007260 con 0x7fc9500d29c0 2024-03-06T07:24:28.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.934+0000 7fc95777a640 1 -- 172.21.15.83:0/1883772095 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9500cc370 con 0x7fc9500d29c0 2024-03-06T07:24:28.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.935+0000 7fc95777a640 1 -- 172.21.15.83:0/1883772095 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc95004f330 con 0x7fc9500d29c0 2024-03-06T07:24:28.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.936+0000 7fc945ffb640 1 -- 172.21.15.83:0/1883772095 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc940007400 con 0x7fc9500d29c0 2024-03-06T07:24:28.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.936+0000 7fc945ffb640 1 --2- 172.21.15.83:0/1883772095 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc93406b4d0 0x7fc93406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.937+0000 7fc955516640 1 --2- 172.21.15.83:0/1883772095 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc93406b4d0 0x7fc93406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.937+0000 7fc945ffb640 1 -- 172.21.15.83:0/1883772095 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc940089660 con 0x7fc9500d29c0 2024-03-06T07:24:28.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.937+0000 7fc955516640 1 --2- 172.21.15.83:0/1883772095 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc93406b4d0 0x7fc93406d990 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fc94c036000 tx=0x7fc94c03b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:28.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:28.941+0000 7fc945ffb640 1 -- 172.21.15.83:0/1883772095 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc940057e90 con 0x7fc9500d29c0 2024-03-06T07:24:29.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.155+0000 7fc95777a640 1 -- 172.21.15.83:0/1883772095 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9500b2ba0 con 0x7fc9500d29c0 2024-03-06T07:24:29.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.155+0000 7fc945ffb640 1 -- 172.21.15.83:0/1883772095 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4980 (secure 0 0 0) 0x7fc94005bf30 con 0x7fc9500d29c0 2024-03-06T07:24:29.157 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:24:29.157 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":25,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5665,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/3334609507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3334609507},{"type":"v1","addr":"172.21.15.183:6834","nonce":3334609507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5668,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/691189112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":691189112},{"type":"v1","addr":"172.21.15.183:6835","nonce":691189112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6081,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/3952571284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":3952571284},{"type":"v1","addr":"172.21.15.83:6836","nonce":3952571284}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:19.852157+0000","modified":"2024-03-06T07:24:21.879875+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6084},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6084":{"gid":6084,"name":"a","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/3237506441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3237506441},{"type":"v1","addr":"172.21.15.83:6837","nonce":3237506441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6084,"qdb_cluster":[6084]},"id":4}]} 2024-03-06T07:24:29.157 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 25 2024-03-06T07:24:29.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.158+0000 7fc95777a640 1 -- 172.21.15.83:0/1883772095 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc93406b4d0 msgr2=0x7fc93406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.158+0000 7fc95777a640 1 --2- 172.21.15.83:0/1883772095 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc93406b4d0 0x7fc93406d990 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fc94c036000 tx=0x7fc94c03b040 comp rx=0 tx=0).stop 2024-03-06T07:24:29.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.158+0000 7fc95777a640 1 -- 172.21.15.83:0/1883772095 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9500d29c0 msgr2=0x7fc9500b1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.158+0000 7fc95777a640 1 --2- 172.21.15.83:0/1883772095 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9500d29c0 0x7fc9500b1e30 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fc94000c470 tx=0x7fc940001e70 comp rx=0 tx=0).stop 2024-03-06T07:24:29.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.158+0000 7fc95777a640 1 -- 172.21.15.83:0/1883772095 shutdown_connections 2024-03-06T07:24:29.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.158+0000 7fc95777a640 1 --2- 172.21.15.83:0/1883772095 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc93406b4d0 0x7fc93406d990 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.158+0000 7fc95777a640 1 --2- 172.21.15.83:0/1883772095 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc95015dfe0 0x7fc95015b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.158+0000 7fc95777a640 1 --2- 172.21.15.83:0/1883772095 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc9500d29c0 0x7fc9500b1e30 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.158+0000 7fc95777a640 1 --2- 172.21.15.83:0/1883772095 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc9500cf830 0x7fc9500ccb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.158+0000 7fc95777a640 1 -- 172.21.15.83:0/1883772095 >> 172.21.15.83:0/1883772095 conn(0x7fc9500d3cd0 msgr2=0x7fc95006b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:29.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.158+0000 7fc95777a640 1 -- 172.21.15.83:0/1883772095 shutdown_connections 2024-03-06T07:24:29.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.158+0000 7fc95777a640 1 -- 172.21.15.83:0/1883772095 wait complete. 2024-03-06T07:24:29.575 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:24:19.852157+0000', 'modified': '2024-03-06T07:24:21.879875+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6084}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6084': {'gid': 6084, 'name': 'a', 'rank': 0, 'incarnation': 23, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.83:6837/3237506441', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3237506441}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 3237506441}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6084, 'qdb_cluster': [6084]} 2024-03-06T07:24:29.575 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:24:29.575 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:24:29.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.677+0000 7f74b7c8d640 1 Processor -- start 2024-03-06T07:24:29.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.677+0000 7f74b7c8d640 1 -- start start 2024-03-06T07:24:29.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.678+0000 7f74b7c8d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f74b00d0d90 0x7f74b00ce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.678+0000 7f74b7c8d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b00d25e0 con 0x7f74b015e000 2024-03-06T07:24:29.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.678+0000 7f74b7c8d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b00d2880 con 0x7f74b00c90f0 2024-03-06T07:24:29.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.678+0000 7f74b7c8d640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74b00c7c50 con 0x7f74b00d0d90 2024-03-06T07:24:29.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.678+0000 7f74b5228640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f74b00d0d90 0x7f74b00ce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.678+0000 7f74b5228640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f74b00d0d90 0x7f74b00ce7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43006/0 (socket says 172.21.15.83:43006) 2024-03-06T07:24:29.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.678+0000 7f74b5228640 1 -- 172.21.15.83:0/1671028568 learned_addr learned my addr 172.21.15.83:0/1671028568 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:29.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.678+0000 7f74b5a29640 1 --1- 172.21.15.83:0/1671028568 >> v1:172.21.15.83:6789/0 conn(0x7f74b015e000 0x7f74b00d0680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:58002/0 (socket says 172.21.15.83:58002) 2024-03-06T07:24:29.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.678+0000 7f74b5228640 1 -- 172.21.15.83:0/1671028568 >> v1:172.21.15.183:6789/0 conn(0x7f74b00c90f0 legacy=0x7f74b00ced00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:29.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.678+0000 7f74b5228640 1 -- 172.21.15.83:0/1671028568 >> v1:172.21.15.83:6789/0 conn(0x7f74b015e000 legacy=0x7f74b00d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.678+0000 7f74b4a27640 1 -- 172.21.15.83:0/1671028568 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2925949061 0 0) 0x7f74b00d25e0 con 0x7f74b015e000 2024-03-06T07:24:29.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.678+0000 7f74b5228640 1 -- 172.21.15.83:0/1671028568 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b00d2b40 con 0x7f74b00d0d90 2024-03-06T07:24:29.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.680+0000 7f74b5228640 1 --2- 172.21.15.83:0/1671028568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74b00d0d90 0x7f74b00ce7c0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f74a0009870 tx=0x7f74a002ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12eb610eeab51065 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:29.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.680+0000 7f74b4a27640 1 -- 172.21.15.83:0/1671028568 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a002f840 con 0x7f74b00d0d90 2024-03-06T07:24:29.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.680+0000 7f74b4a27640 1 -- 172.21.15.83:0/1671028568 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74a002f9e0 con 0x7f74b00d0d90 2024-03-06T07:24:29.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.680+0000 7f74b4a27640 1 -- 172.21.15.83:0/1671028568 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a002fdb0 con 0x7f74b00d0d90 2024-03-06T07:24:29.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.680+0000 7f74b7c8d640 1 -- 172.21.15.83:0/1671028568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74b00d0d90 msgr2=0x7f74b00ce7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.680+0000 7f74b7c8d640 1 --2- 172.21.15.83:0/1671028568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74b00d0d90 0x7f74b00ce7c0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f74a0009870 tx=0x7f74a002ba20 comp rx=0 tx=0).stop 2024-03-06T07:24:29.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.680+0000 7f74b7c8d640 1 -- 172.21.15.83:0/1671028568 shutdown_connections 2024-03-06T07:24:29.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.680+0000 7f74b7c8d640 1 --2- 172.21.15.83:0/1671028568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74b00d0d90 0x7f74b00ce7c0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.680+0000 7f74b7c8d640 1 -- 172.21.15.83:0/1671028568 >> 172.21.15.83:0/1671028568 conn(0x7f74b00d3cf0 msgr2=0x7f74b00d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:29.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.680+0000 7f74b7c8d640 1 -- 172.21.15.83:0/1671028568 shutdown_connections 2024-03-06T07:24:29.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.680+0000 7f74b7c8d640 1 -- 172.21.15.83:0/1671028568 wait complete. 2024-03-06T07:24:29.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.681+0000 7f74b7c8d640 1 Processor -- start 2024-03-06T07:24:29.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.682+0000 7f74b7c8d640 1 -- start start 2024-03-06T07:24:29.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.682+0000 7f74b7c8d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74b00c90f0 0x7f74b0157350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.682+0000 7f74b7c8d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f74b00d0d90 0x7f74b00b84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.682+0000 7f74b7c8d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f74b015e000 0x7f74b00b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.682+0000 7f74b7c8d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b015e3e0 con 0x7f74b00c90f0 2024-03-06T07:24:29.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.682+0000 7f74b7c8d640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b00d22d0 con 0x7f74b00d0d90 2024-03-06T07:24:29.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.682+0000 7f74b7c8d640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74b00d4810 con 0x7f74b015e000 2024-03-06T07:24:29.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.682+0000 7f74b5a29640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74b00c90f0 0x7f74b0157350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.682+0000 7f74b5a29640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74b00c90f0 0x7f74b0157350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43008/0 (socket says 172.21.15.83:43008) 2024-03-06T07:24:29.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.682+0000 7f74b5a29640 1 -- 172.21.15.83:0/2119809162 learned_addr learned my addr 172.21.15.83:0/2119809162 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:29.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.683+0000 7f74b5228640 1 --2- 172.21.15.83:0/2119809162 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f74b00d0d90 0x7f74b00b84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.683+0000 7f74b622a640 1 --2- 172.21.15.83:0/2119809162 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f74b015e000 0x7f74b00b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.683+0000 7f74b5a29640 1 -- 172.21.15.83:0/2119809162 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f74b015e000 msgr2=0x7f74b00b8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.683+0000 7f74b5a29640 1 --2- 172.21.15.83:0/2119809162 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f74b015e000 0x7f74b00b8a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.683+0000 7f74b5a29640 1 -- 172.21.15.83:0/2119809162 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f74b00d0d90 msgr2=0x7f74b00b84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.683+0000 7f74b5a29640 1 --2- 172.21.15.83:0/2119809162 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f74b00d0d90 0x7f74b00b84e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.683+0000 7f74b5a29640 1 -- 172.21.15.83:0/2119809162 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b0157890 con 0x7f74b00c90f0 2024-03-06T07:24:29.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.683+0000 7f74b5228640 1 --2- 172.21.15.83:0/2119809162 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f74b00d0d90 0x7f74b00b84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:29.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.683+0000 7f74b5a29640 1 --2- 172.21.15.83:0/2119809162 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74b00c90f0 0x7f74b0157350 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f74ac00c460 tx=0x7f74ac00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:29.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.683+0000 7f74a67fc640 1 -- 172.21.15.83:0/2119809162 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ac018070 con 0x7f74b00c90f0 2024-03-06T07:24:29.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.683+0000 7f74a67fc640 1 -- 172.21.15.83:0/2119809162 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74ac007500 con 0x7f74b00c90f0 2024-03-06T07:24:29.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.683+0000 7f74b7c8d640 1 -- 172.21.15.83:0/2119809162 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74b00a09e0 con 0x7f74b00c90f0 2024-03-06T07:24:29.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.683+0000 7f74a67fc640 1 -- 172.21.15.83:0/2119809162 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ac004a20 con 0x7f74b00c90f0 2024-03-06T07:24:29.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.683+0000 7f74b7c8d640 1 -- 172.21.15.83:0/2119809162 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74b00a0fa0 con 0x7f74b00c90f0 2024-03-06T07:24:29.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.684+0000 7f74a67fc640 1 -- 172.21.15.83:0/2119809162 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f74ac01b070 con 0x7f74b00c90f0 2024-03-06T07:24:29.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.685+0000 7f74a67fc640 1 --2- 172.21.15.83:0/2119809162 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f749406b500 0x7f749406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.685+0000 7f74a67fc640 1 -- 172.21.15.83:0/2119809162 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f74ac0893f0 con 0x7f74b00c90f0 2024-03-06T07:24:29.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.685+0000 7f74b5228640 1 --2- 172.21.15.83:0/2119809162 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f749406b500 0x7f749406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.685+0000 7f74b5228640 1 --2- 172.21.15.83:0/2119809162 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f749406b500 0x7f749406d9c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f74a0036000 tx=0x7f74a0039040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:29.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.686+0000 7f74ba870740 1 Processor -- start 2024-03-06T07:24:29.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.686+0000 7f74ba870740 1 -- start start 2024-03-06T07:24:29.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.687+0000 7f74ba870740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f4b0ac0870 0x55f4b0ac0c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.688+0000 7f74ba870740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4b0ac48c0 0x55f4b0ac6360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.688+0000 7f74ba870740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55f4b0af0600 0x55f4b0af0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.688+0000 7f74ba870740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4b0a2a6b0 con 0x55f4b0ac48c0 2024-03-06T07:24:29.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.688+0000 7f74ba870740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4b09d3a60 con 0x55f4b0ac0870 2024-03-06T07:24:29.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.688+0000 7f74ba870740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f4b0a26750 con 0x55f4b0af0600 2024-03-06T07:24:29.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.688+0000 7f74b5228640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4b0ac48c0 0x55f4b0ac6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.688+0000 7f74b5228640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4b0ac48c0 0x55f4b0ac6360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43020/0 (socket says 172.21.15.83:43020) 2024-03-06T07:24:29.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.688+0000 7f74b5228640 1 -- 172.21.15.83:0/3035101545 learned_addr learned my addr 172.21.15.83:0/3035101545 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:29.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.688+0000 7f74b5a29640 1 --2- 172.21.15.83:0/3035101545 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f4b0ac0870 0x55f4b0ac0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.688+0000 7f74b5228640 1 -- 172.21.15.83:0/3035101545 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55f4b0af0600 msgr2=0x55f4b0af0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:29.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.688+0000 7f74b622a640 1 --2- 172.21.15.83:0/3035101545 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55f4b0af0600 0x55f4b0af0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.688+0000 7f74b5228640 1 --2- 172.21.15.83:0/3035101545 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55f4b0af0600 0x55f4b0af0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.688+0000 7f74b5228640 1 -- 172.21.15.83:0/3035101545 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f4b0ac0870 msgr2=0x55f4b0ac0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:29.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.688+0000 7f74b5228640 1 --2- 172.21.15.83:0/3035101545 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f4b0ac0870 0x55f4b0ac0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.688+0000 7f74b5228640 1 -- 172.21.15.83:0/3035101545 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f4b09f1f30 con 0x55f4b0ac48c0 2024-03-06T07:24:29.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.688+0000 7f74b5a29640 1 --2- 172.21.15.83:0/3035101545 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f4b0ac0870 0x55f4b0ac0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:29.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.688+0000 7f74b5228640 1 --2- 172.21.15.83:0/3035101545 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4b0ac48c0 0x55f4b0ac6360 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f74a0009960 tx=0x7f74a0030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b50c87b0bf119311 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:29.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.688+0000 7f74937fe640 1 -- 172.21.15.83:0/3035101545 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a0042c70 con 0x55f4b0ac48c0 2024-03-06T07:24:29.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.688+0000 7f74937fe640 1 -- 172.21.15.83:0/3035101545 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74a0051070 con 0x55f4b0ac48c0 2024-03-06T07:24:29.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.688+0000 7f74937fe640 1 -- 172.21.15.83:0/3035101545 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a0037600 con 0x55f4b0ac48c0 2024-03-06T07:24:29.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.689+0000 7f74ba870740 1 -- 172.21.15.83:0/3035101545 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4b0ac48c0 msgr2=0x55f4b0ac6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.689+0000 7f74ba870740 1 --2- 172.21.15.83:0/3035101545 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4b0ac48c0 0x55f4b0ac6360 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f74a0009960 tx=0x7f74a0030000 comp rx=0 tx=0).stop 2024-03-06T07:24:29.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.689+0000 7f74ba870740 1 -- 172.21.15.83:0/3035101545 shutdown_connections 2024-03-06T07:24:29.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.689+0000 7f74ba870740 1 --2- 172.21.15.83:0/3035101545 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55f4b0af0600 0x55f4b0af0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.689+0000 7f74ba870740 1 --2- 172.21.15.83:0/3035101545 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4b0ac48c0 0x55f4b0ac6360 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.689+0000 7f74ba870740 1 --2- 172.21.15.83:0/3035101545 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f4b0ac0870 0x55f4b0ac0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.689+0000 7f74ba870740 1 -- 172.21.15.83:0/3035101545 >> 172.21.15.83:0/3035101545 conn(0x55f4b0ac28f0 msgr2=0x55f4b0ac41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:29.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.689+0000 7f74ba870740 1 -- 172.21.15.83:0/3035101545 shutdown_connections 2024-03-06T07:24:29.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.689+0000 7f74ba870740 1 -- 172.21.15.83:0/3035101545 wait complete. 2024-03-06T07:24:29.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.690+0000 7f74ba870740 1 -- start start 2024-03-06T07:24:29.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.690+0000 7f74ba870740 1 Processor -- start 2024-03-06T07:24:29.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.691+0000 7f74ba870740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f4b0af0600 0x55f4b0aebac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.691+0000 7f74ba870740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55f4b0b062e0 0x55f4b0b066c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.691+0000 7f74ba870740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4b0af3e00 0x55f4b0af4280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.691+0000 7f74ba870740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4b0ac4d50 con 0x55f4b0af3e00 2024-03-06T07:24:29.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.691+0000 7f74ba870740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4b09e6ea0 con 0x55f4b0af0600 2024-03-06T07:24:29.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.691+0000 7f74ba870740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f4b0ac5030 con 0x55f4b0b062e0 2024-03-06T07:24:29.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.691+0000 7f74ba870740 10 client.0 _ll_register_callbacks cb 0x55f4b09e6870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:24:29.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.691+0000 7f74b622a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4b0af3e00 0x55f4b0af4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.691+0000 7f74b622a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4b0af3e00 0x55f4b0af4280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43036/0 (socket says 172.21.15.83:43036) 2024-03-06T07:24:29.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.691+0000 7f74b622a640 1 -- 172.21.15.83:0/1429549674 learned_addr learned my addr 172.21.15.83:0/1429549674 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:29.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.691+0000 7f74b5228640 1 --2- 172.21.15.83:0/1429549674 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55f4b0b062e0 0x55f4b0b066c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.691+0000 7f74b5a29640 1 --2- 172.21.15.83:0/1429549674 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f4b0af0600 0x55f4b0aebac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.691+0000 7f74b622a640 1 -- 172.21.15.83:0/1429549674 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55f4b0b062e0 msgr2=0x55f4b0b066c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:29.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.691+0000 7f74b622a640 1 --2- 172.21.15.83:0/1429549674 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55f4b0b062e0 0x55f4b0b066c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.691+0000 7f74b622a640 1 -- 172.21.15.83:0/1429549674 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f4b0af0600 msgr2=0x55f4b0aebac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:29.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.691+0000 7f74b622a640 1 --2- 172.21.15.83:0/1429549674 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f4b0af0600 0x55f4b0aebac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.691+0000 7f74b622a640 1 -- 172.21.15.83:0/1429549674 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f4b09e7270 con 0x55f4b0af3e00 2024-03-06T07:24:29.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.691+0000 7f74b5228640 1 --2- 172.21.15.83:0/1429549674 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55f4b0b062e0 0x55f4b0b066c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:29.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.691+0000 7f74b622a640 1 --2- 172.21.15.83:0/1429549674 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4b0af3e00 0x55f4b0af4280 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f74a800e2e0 tx=0x7f74a800e7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=487b3234fe4ef344 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:29.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.691+0000 7f7490ff9640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-06T07:24:29.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.692+0000 7f7490ff9640 1 -- 172.21.15.83:0/1429549674 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a8003d80 con 0x55f4b0af3e00 2024-03-06T07:24:29.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.692+0000 7f7490ff9640 1 -- 172.21.15.83:0/1429549674 <== mon.0 v2:172.21.15.83:3300/0 2 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f74a80050d0 con 0x55f4b0af3e00 2024-03-06T07:24:29.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.692+0000 7f74ba870740 1 -- 172.21.15.83:0/1429549674 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f74a0051070 con 0x55f4b0af3e00 2024-03-06T07:24:29.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.692+0000 7f7490ff9640 1 -- 172.21.15.83:0/1429549674 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f74a801f050 con 0x55f4b0af3e00 2024-03-06T07:24:29.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.692+0000 7f74ba870740 10 client.6243 fetch_fsmap learned FSMap version 25 2024-03-06T07:24:29.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.692+0000 7f74ba870740 1 -- 172.21.15.83:0/1429549674 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x55f4b073f2d0 con 0x55f4b0af3e00 2024-03-06T07:24:29.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.692+0000 7f7490ff9640 1 -- 172.21.15.83:0/1429549674 <== mon.0 v2:172.21.15.83:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74a0051070 con 0x55f4b0af3e00 2024-03-06T07:24:29.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.692+0000 7f7490ff9640 1 -- 172.21.15.83:0/1429549674 <== mon.0 v2:172.21.15.83:3300/0 5 ==== fsmap(e 25) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f74a8022020 con 0x55f4b0af3e00 2024-03-06T07:24:29.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.692+0000 7f74ba870740 10 client.6243 fetch_fsmap finished waiting for FSMap version 25 2024-03-06T07:24:29.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.692+0000 7f74ba870740 10 client.6243 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:24:29.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.692+0000 7f74ba870740 20 client.6243 populate_metadata read hostname 'smithi083' 2024-03-06T07:24:29.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.692+0000 7f74ba870740 1 --2- 172.21.15.83:0/1429549674 >> [v2:172.21.15.83:6834/3237506441,v1:172.21.15.83:6837/3237506441] conn(0x55f4b0af26f0 0x55f4b0b2aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.692+0000 7f74ba870740 4 client.6243 mds_command: new command op to 6084 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:24:29.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.692+0000 7f74ba870740 1 -- 172.21.15.83:0/1429549674 --> [v2:172.21.15.83:6834/3237506441,v1:172.21.15.83:6837/3237506441] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f4b0a37bf0 con 0x55f4b0af26f0 2024-03-06T07:24:29.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.692+0000 7f74b5a29640 1 --2- 172.21.15.83:0/1429549674 >> [v2:172.21.15.83:6834/3237506441,v1:172.21.15.83:6837/3237506441] conn(0x55f4b0af26f0 0x55f4b0b2aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.692+0000 7f7490ff9640 10 client.6243 ms_handle_connect on v2:172.21.15.83:6834/3237506441 2024-03-06T07:24:29.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.693+0000 7f74b5a29640 1 --2- 172.21.15.83:0/1429549674 >> [v2:172.21.15.83:6834/3237506441,v1:172.21.15.83:6837/3237506441] conn(0x55f4b0af26f0 0x55f4b0b2aa60 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6be94f6e95183860 server_cookie=e40c891455c7d2f in_seq=0 out_seq=0 2024-03-06T07:24:29.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.693+0000 7f7490ff9640 1 -- 172.21.15.83:0/1429549674 <== mds.0 v2:172.21.15.83:6834/3237506441 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f4b0a37bf0 con 0x55f4b0af26f0 2024-03-06T07:24:29.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.693+0000 7f7490ff9640 10 client.6243 handle_command_reply: tid=0 2024-03-06T07:24:29.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.693+0000 7f74ba870740 1 client.6243 shutdown 2024-03-06T07:24:29.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.693+0000 7f74ba870740 1 -- 172.21.15.83:0/1429549674 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4b0af3e00 msgr2=0x55f4b0af4280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.693+0000 7f74ba870740 1 --2- 172.21.15.83:0/1429549674 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4b0af3e00 0x55f4b0af4280 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f74a800e2e0 tx=0x7f74a800e7a0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.693+0000 7f74ba870740 1 -- 172.21.15.83:0/1429549674 shutdown_connections 2024-03-06T07:24:29.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.693+0000 7f74ba870740 1 --2- 172.21.15.83:0/1429549674 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4b0af3e00 0x55f4b0af4280 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.693+0000 7f74ba870740 1 --2- 172.21.15.83:0/1429549674 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55f4b0b062e0 0x55f4b0b066c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.693+0000 7f74ba870740 1 --2- 172.21.15.83:0/1429549674 >> [v2:172.21.15.83:6834/3237506441,v1:172.21.15.83:6837/3237506441] conn(0x55f4b0af26f0 0x55f4b0b2aa60 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.693+0000 7f74ba870740 1 --2- 172.21.15.83:0/1429549674 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f4b0af0600 0x55f4b0aebac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.693+0000 7f7490ff9640 0 client.6243 ms_handle_reset on v2:172.21.15.83:6834/3237506441 2024-03-06T07:24:29.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.693+0000 7f74ba870740 1 -- 172.21.15.83:0/1429549674 >> 172.21.15.83:0/1429549674 conn(0x55f4b0ac0870 msgr2=0x55f4b0abf510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:29.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.694+0000 7f74ba870740 1 -- 172.21.15.83:0/1429549674 shutdown_connections 2024-03-06T07:24:29.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.694+0000 7f74ba870740 1 -- 172.21.15.83:0/1429549674 wait complete. 2024-03-06T07:24:29.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.694+0000 7f74ba870740 20 client.6243 trim_cache size 0 max 16384 2024-03-06T07:24:29.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.725+0000 7f74ba870740 1 Processor -- start 2024-03-06T07:24:29.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.725+0000 7f74ba870740 1 -- start start 2024-03-06T07:24:29.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.727+0000 7f74ba870740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f4b0af0600 0x55f4b0abf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.727+0000 7f74ba870740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4b0af3e00 0x55f4b0abfa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.727+0000 7f74ba870740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55f4b0b062e0 0x55f4b0b04fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.727+0000 7f74ba870740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b0200160 con 0x55f4b0af3e00 2024-03-06T07:24:29.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.727+0000 7f74ba870740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b01ff300 con 0x55f4b0af0600 2024-03-06T07:24:29.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.727+0000 7f74ba870740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74b00b3960 con 0x55f4b0b062e0 2024-03-06T07:24:29.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.727+0000 7f74b5228640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4b0af3e00 0x55f4b0abfa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.727+0000 7f74b5228640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4b0af3e00 0x55f4b0abfa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43044/0 (socket says 172.21.15.83:43044) 2024-03-06T07:24:29.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.727+0000 7f74b5228640 1 -- 172.21.15.83:0/560427516 learned_addr learned my addr 172.21.15.83:0/560427516 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:29.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.727+0000 7f74b5228640 1 -- 172.21.15.83:0/560427516 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55f4b0b062e0 msgr2=0x55f4b0b04fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:29.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.727+0000 7f74b5228640 1 --2- 172.21.15.83:0/560427516 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55f4b0b062e0 0x55f4b0b04fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.727+0000 7f74b5228640 1 -- 172.21.15.83:0/560427516 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f4b0af0600 msgr2=0x55f4b0abf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:29.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.727+0000 7f74b5228640 1 --2- 172.21.15.83:0/560427516 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f4b0af0600 0x55f4b0abf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.727+0000 7f74b5228640 1 -- 172.21.15.83:0/560427516 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f4b08096b0 con 0x55f4b0af3e00 2024-03-06T07:24:29.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.727+0000 7f74b5228640 1 --2- 172.21.15.83:0/560427516 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4b0af3e00 0x55f4b0abfa50 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f74a002ba60 tx=0x7f74a0009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f3c971ec8da56b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:29.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.727+0000 7f7492ffd640 1 -- 172.21.15.83:0/560427516 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a00429b0 con 0x55f4b0af3e00 2024-03-06T07:24:29.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.727+0000 7f7492ffd640 1 -- 172.21.15.83:0/560427516 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74a0051040 con 0x55f4b0af3e00 2024-03-06T07:24:29.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.727+0000 7f7492ffd640 1 -- 172.21.15.83:0/560427516 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a0042d50 con 0x55f4b0af3e00 2024-03-06T07:24:29.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.728+0000 7f74ba870740 1 -- 172.21.15.83:0/560427516 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4b0af3e00 msgr2=0x55f4b0abfa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.728+0000 7f74ba870740 1 --2- 172.21.15.83:0/560427516 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4b0af3e00 0x55f4b0abfa50 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f74a002ba60 tx=0x7f74a0009240 comp rx=0 tx=0).stop 2024-03-06T07:24:29.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.728+0000 7f74ba870740 1 -- 172.21.15.83:0/560427516 shutdown_connections 2024-03-06T07:24:29.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.728+0000 7f74ba870740 1 --2- 172.21.15.83:0/560427516 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55f4b0b062e0 0x55f4b0b04fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.728+0000 7f74ba870740 1 --2- 172.21.15.83:0/560427516 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4b0af3e00 0x55f4b0abfa50 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.728+0000 7f74ba870740 1 --2- 172.21.15.83:0/560427516 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f4b0af0600 0x55f4b0abf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.728+0000 7f74ba870740 1 -- 172.21.15.83:0/560427516 >> 172.21.15.83:0/560427516 conn(0x55f4b0ac0870 msgr2=0x55f4b0aeb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:29.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.728+0000 7f74ba870740 1 -- 172.21.15.83:0/560427516 shutdown_connections 2024-03-06T07:24:29.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.728+0000 7f74ba870740 1 -- 172.21.15.83:0/560427516 wait complete. 2024-03-06T07:24:29.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.729+0000 7f74ba870740 1 -- start start 2024-03-06T07:24:29.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.729+0000 7f74ba870740 1 Processor -- start 2024-03-06T07:24:29.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.730+0000 7f74ba870740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4b0af0600 0x55f4b0ac4f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.730+0000 7f74ba870740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55f4b0af3e00 0x55f4b0b0be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.730+0000 7f74ba870740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f4b0b062e0 0x55f4b0b0c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.730+0000 7f74ba870740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4b0b07610 con 0x55f4b0af0600 2024-03-06T07:24:29.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.730+0000 7f74ba870740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b00b3f60 con 0x55f4b0b062e0 2024-03-06T07:24:29.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.730+0000 7f74ba870740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f4b0af2140 con 0x55f4b0af3e00 2024-03-06T07:24:29.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.730+0000 7f74ba870740 10 client.0 _ll_register_callbacks cb 0x7f74a8001f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:24:29.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.730+0000 7f74b5a29640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4b0af0600 0x55f4b0ac4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.730+0000 7f74b5a29640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4b0af0600 0x55f4b0ac4f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43056/0 (socket says 172.21.15.83:43056) 2024-03-06T07:24:29.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.730+0000 7f74b5a29640 1 -- 172.21.15.83:0/643069064 learned_addr learned my addr 172.21.15.83:0/643069064 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:29.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.730+0000 7f74b5228640 1 --2- 172.21.15.83:0/643069064 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55f4b0af3e00 0x55f4b0b0be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.730+0000 7f74b622a640 1 --2- 172.21.15.83:0/643069064 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f4b0b062e0 0x55f4b0b0c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.731+0000 7f74b5a29640 1 -- 172.21.15.83:0/643069064 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55f4b0af3e00 msgr2=0x55f4b0b0be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:29.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.731+0000 7f74b5a29640 1 --2- 172.21.15.83:0/643069064 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55f4b0af3e00 0x55f4b0b0be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.731+0000 7f74b5a29640 1 -- 172.21.15.83:0/643069064 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f4b0b062e0 msgr2=0x55f4b0b0c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:29.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.731+0000 7f74b5a29640 1 --2- 172.21.15.83:0/643069064 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f4b0b062e0 0x55f4b0b0c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.731+0000 7f74b5a29640 1 -- 172.21.15.83:0/643069064 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f4b0af14d0 con 0x55f4b0af0600 2024-03-06T07:24:29.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.731+0000 7f74b622a640 1 --2- 172.21.15.83:0/643069064 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f4b0b062e0 0x55f4b0b0c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:29.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.731+0000 7f74b5a29640 1 --2- 172.21.15.83:0/643069064 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4b0af0600 0x55f4b0ac4f60 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f74ac057870 tx=0x7f74ac000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d49ea44ca3f20e32 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:29.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.731+0000 7f7490ff9640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-06T07:24:29.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.731+0000 7f7490ff9640 1 -- 172.21.15.83:0/643069064 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ac05bcc0 con 0x55f4b0af0600 2024-03-06T07:24:29.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.731+0000 7f7490ff9640 1 -- 172.21.15.83:0/643069064 <== mon.0 v2:172.21.15.83:3300/0 2 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f74ac0541e0 con 0x55f4b0af0600 2024-03-06T07:24:29.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.731+0000 7f74ba870740 1 -- 172.21.15.83:0/643069064 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f74a0051040 con 0x55f4b0af0600 2024-03-06T07:24:29.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.731+0000 7f7490ff9640 1 -- 172.21.15.83:0/643069064 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f74ac060280 con 0x55f4b0af0600 2024-03-06T07:24:29.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.731+0000 7f74ba870740 10 client.6249 fetch_fsmap learned FSMap version 25 2024-03-06T07:24:29.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.731+0000 7f74ba870740 1 -- 172.21.15.83:0/643069064 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x55f4b0ac54f0 con 0x55f4b0af0600 2024-03-06T07:24:29.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.731+0000 7f7490ff9640 1 -- 172.21.15.83:0/643069064 <== mon.0 v2:172.21.15.83:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74a0051040 con 0x55f4b0af0600 2024-03-06T07:24:29.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.731+0000 7f7490ff9640 1 -- 172.21.15.83:0/643069064 <== mon.0 v2:172.21.15.83:3300/0 5 ==== fsmap(e 25) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f74ac0890b0 con 0x55f4b0af0600 2024-03-06T07:24:29.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.731+0000 7f74ba870740 10 client.6249 fetch_fsmap finished waiting for FSMap version 25 2024-03-06T07:24:29.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.731+0000 7f74ba870740 10 client.6249 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:24:29.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.731+0000 7f74ba870740 20 client.6249 populate_metadata read hostname 'smithi083' 2024-03-06T07:24:29.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.731+0000 7f74ba870740 1 --2- 172.21.15.83:0/643069064 >> [v2:172.21.15.83:6834/3237506441,v1:172.21.15.83:6837/3237506441] conn(0x55f4b0aec190 0x55f4b0aec570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.731+0000 7f74ba870740 4 client.6249 mds_command: new command op to 6084 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:24:29.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.731+0000 7f74ba870740 1 -- 172.21.15.83:0/643069064 --> [v2:172.21.15.83:6834/3237506441,v1:172.21.15.83:6837/3237506441] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f4b0a31f40 con 0x55f4b0aec190 2024-03-06T07:24:29.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.732+0000 7f74b5228640 1 --2- 172.21.15.83:0/643069064 >> [v2:172.21.15.83:6834/3237506441,v1:172.21.15.83:6837/3237506441] conn(0x55f4b0aec190 0x55f4b0aec570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.732+0000 7f7490ff9640 10 client.6249 ms_handle_connect on v2:172.21.15.83:6834/3237506441 2024-03-06T07:24:29.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.732+0000 7f74b5228640 1 --2- 172.21.15.83:0/643069064 >> [v2:172.21.15.83:6834/3237506441,v1:172.21.15.83:6837/3237506441] conn(0x55f4b0aec190 0x55f4b0aec570 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10adaa8c3079433a server_cookie=96734f9bff143cf6 in_seq=0 out_seq=0 2024-03-06T07:24:29.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.732+0000 7f7490ff9640 1 -- 172.21.15.83:0/643069064 <== mds.0 v2:172.21.15.83:6834/3237506441 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x55f4b0a31f40 con 0x55f4b0aec190 2024-03-06T07:24:29.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.732+0000 7f7490ff9640 10 client.6249 handle_command_reply: tid=0 2024-03-06T07:24:29.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.732+0000 7f74ba870740 1 client.6249 shutdown 2024-03-06T07:24:29.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.732+0000 7f74ba870740 1 -- 172.21.15.83:0/643069064 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4b0af0600 msgr2=0x55f4b0ac4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.732+0000 7f74ba870740 1 --2- 172.21.15.83:0/643069064 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4b0af0600 0x55f4b0ac4f60 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f74ac057870 tx=0x7f74ac000ed0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.732+0000 7f74ba870740 1 -- 172.21.15.83:0/643069064 shutdown_connections 2024-03-06T07:24:29.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.732+0000 7f74ba870740 1 --2- 172.21.15.83:0/643069064 >> [v2:172.21.15.83:6834/3237506441,v1:172.21.15.83:6837/3237506441] conn(0x55f4b0aec190 0x55f4b0aec570 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.732+0000 7f74ba870740 1 --2- 172.21.15.83:0/643069064 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f4b0b062e0 0x55f4b0b0c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.732+0000 7f7490ff9640 0 client.6249 ms_handle_reset on v2:172.21.15.83:6834/3237506441 2024-03-06T07:24:29.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.732+0000 7f74ba870740 1 --2- 172.21.15.83:0/643069064 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55f4b0af3e00 0x55f4b0b0be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.732+0000 7f74ba870740 1 --2- 172.21.15.83:0/643069064 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4b0af0600 0x55f4b0ac4f60 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.732+0000 7f74ba870740 1 -- 172.21.15.83:0/643069064 >> 172.21.15.83:0/643069064 conn(0x55f4b0ac0870 msgr2=0x55f4b0af41e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:29.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.732+0000 7f74ba870740 1 -- 172.21.15.83:0/643069064 shutdown_connections 2024-03-06T07:24:29.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.732+0000 7f74ba870740 1 -- 172.21.15.83:0/643069064 wait complete. 2024-03-06T07:24:29.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.732+0000 7f74ba870740 20 client.6249 trim_cache size 0 max 16384 2024-03-06T07:24:29.740 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:24:29.741 INFO:teuthology.orchestra.run.smithi083.stdout: "cluster_fsid": "039877dd-d2ce-45d2-8e77-ac5eb7e1d55a", 2024-03-06T07:24:29.741 INFO:teuthology.orchestra.run.smithi083.stdout: "whoami": 0, 2024-03-06T07:24:29.741 INFO:teuthology.orchestra.run.smithi083.stdout: "id": 6084, 2024-03-06T07:24:29.741 INFO:teuthology.orchestra.run.smithi083.stdout: "want_state": "up:active", 2024-03-06T07:24:29.741 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "up:active", 2024-03-06T07:24:29.741 INFO:teuthology.orchestra.run.smithi083.stdout: "fs_name": "cephfs", 2024-03-06T07:24:29.741 INFO:teuthology.orchestra.run.smithi083.stdout: "rank_uptime": 9.2063882750000001, 2024-03-06T07:24:29.741 INFO:teuthology.orchestra.run.smithi083.stdout: "mdsmap_epoch": 25, 2024-03-06T07:24:29.741 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_epoch": 57, 2024-03-06T07:24:29.741 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_epoch_barrier": 57, 2024-03-06T07:24:29.741 INFO:teuthology.orchestra.run.smithi083.stdout: "uptime": 14.28448114 2024-03-06T07:24:29.741 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:24:29.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.733+0000 7f74b7c8d640 1 -- 172.21.15.83:0/2119809162 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f749406b500 msgr2=0x7f749406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.733+0000 7f74b7c8d640 1 --2- 172.21.15.83:0/2119809162 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f749406b500 0x7f749406d9c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f74a0036000 tx=0x7f74a0039040 comp rx=0 tx=0).stop 2024-03-06T07:24:29.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.733+0000 7f74b7c8d640 1 -- 172.21.15.83:0/2119809162 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74b00c90f0 msgr2=0x7f74b0157350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.733+0000 7f74b7c8d640 1 --2- 172.21.15.83:0/2119809162 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74b00c90f0 0x7f74b0157350 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f74ac00c460 tx=0x7f74ac00c920 comp rx=0 tx=0).stop 2024-03-06T07:24:29.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.733+0000 7f74b7c8d640 1 -- 172.21.15.83:0/2119809162 shutdown_connections 2024-03-06T07:24:29.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.733+0000 7f74b7c8d640 1 --2- 172.21.15.83:0/2119809162 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f749406b500 0x7f749406d9c0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.733+0000 7f74b7c8d640 1 --2- 172.21.15.83:0/2119809162 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f74b015e000 0x7f74b00b8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.733+0000 7f74b7c8d640 1 --2- 172.21.15.83:0/2119809162 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f74b00d0d90 0x7f74b00b84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.733+0000 7f74b7c8d640 1 --2- 172.21.15.83:0/2119809162 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74b00c90f0 0x7f74b0157350 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.733+0000 7f74b7c8d640 1 -- 172.21.15.83:0/2119809162 >> 172.21.15.83:0/2119809162 conn(0x7f74b00d3cf0 msgr2=0x7f74b00b6910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:29.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.733+0000 7f74b7c8d640 1 -- 172.21.15.83:0/2119809162 shutdown_connections 2024-03-06T07:24:29.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.733+0000 7f74b7c8d640 1 -- 172.21.15.83:0/2119809162 wait complete. 2024-03-06T07:24:29.746 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:24:29.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.849+0000 7f50c2e78640 1 Processor -- start 2024-03-06T07:24:29.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.849+0000 7f50c2e78640 1 -- start start 2024-03-06T07:24:29.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.850+0000 7f50c2e78640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f50bc15dfe0 0x7f50bc0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.850+0000 7f50c2e78640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50bc0b3430 con 0x7f50bc0d29c0 2024-03-06T07:24:29.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.850+0000 7f50c2e78640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50bc0b3710 con 0x7f50bc0cf830 2024-03-06T07:24:29.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.850+0000 7f50c2e78640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50bc0b31e0 con 0x7f50bc15dfe0 2024-03-06T07:24:29.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.850+0000 7f50c1415640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f50bc0d29c0 0x7f50bc0b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:58018/0 (socket says 172.21.15.83:58018) 2024-03-06T07:24:29.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.850+0000 7f50c1415640 1 -- 172.21.15.83:0/2921600593 learned_addr learned my addr 172.21.15.83:0/2921600593 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:24:29.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.850+0000 7f50bb7fe640 1 -- 172.21.15.83:0/2921600593 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2784735704 0 0) 0x7f50bc0b3430 con 0x7f50bc0d29c0 2024-03-06T07:24:29.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.850+0000 7f50c0c14640 1 --2- 172.21.15.83:0/2921600593 >> v2:172.21.15.183:3300/0 conn(0x7f50bc15dfe0 0x7f50bc0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.850+0000 7f50bb7fe640 1 -- 172.21.15.83:0/2921600593 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50ac003520 con 0x7f50bc0d29c0 2024-03-06T07:24:29.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.851+0000 7f50bb7fe640 1 -- 172.21.15.83:0/2921600593 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1057532461 0 0) 0x7f50ac003520 con 0x7f50bc0d29c0 2024-03-06T07:24:29.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.851+0000 7f50bb7fe640 1 -- 172.21.15.83:0/2921600593 >> v2:172.21.15.183:3300/0 conn(0x7f50bc15dfe0 msgr2=0x7f50bc0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:29.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.851+0000 7f50bb7fe640 1 --2- 172.21.15.83:0/2921600593 >> v2:172.21.15.183:3300/0 conn(0x7f50bc15dfe0 0x7f50bc0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.851+0000 7f50bb7fe640 1 -- 172.21.15.83:0/2921600593 >> v1:172.21.15.183:6789/0 conn(0x7f50bc0cf830 legacy=0x7f50bc0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.851+0000 7f50bb7fe640 1 -- 172.21.15.83:0/2921600593 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50bc0b39f0 con 0x7f50bc0d29c0 2024-03-06T07:24:29.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.851+0000 7f50bb7fe640 1 -- 172.21.15.83:0/2921600593 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f50b40019f0 con 0x7f50bc0d29c0 2024-03-06T07:24:29.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.851+0000 7f50bb7fe640 1 -- 172.21.15.83:0/2921600593 >> v1:172.21.15.83:6789/0 conn(0x7f50bc0d29c0 legacy=0x7f50bc0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.851+0000 7f50bb7fe640 1 --2- 172.21.15.83:0/2921600593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50ac0039b0 0x7f50ac003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.851+0000 7f50bb7fe640 1 --2- 172.21.15.83:0/2921600593 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f50ac004800 0x7f50ac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.851+0000 7f50bb7fe640 1 --2- 172.21.15.83:0/2921600593 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f50ac0051c0 0x7f50ac00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.851+0000 7f50bb7fe640 1 -- 172.21.15.83:0/2921600593 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50bc0b31e0 con 0x7f50ac0039b0 2024-03-06T07:24:29.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.851+0000 7f50bb7fe640 1 -- 172.21.15.83:0/2921600593 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50ac00c090 con 0x7f50ac004800 2024-03-06T07:24:29.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.851+0000 7f50bb7fe640 1 -- 172.21.15.83:0/2921600593 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50ac00c290 con 0x7f50ac0051c0 2024-03-06T07:24:29.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.851+0000 7f50c0c14640 1 --2- 172.21.15.83:0/2921600593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50ac0039b0 0x7f50ac003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.851+0000 7f50bbfff640 1 --2- 172.21.15.83:0/2921600593 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f50ac004800 0x7f50ac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.851+0000 7f50c1415640 1 --2- 172.21.15.83:0/2921600593 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f50ac0051c0 0x7f50ac00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.851+0000 7f50c0c14640 1 -- 172.21.15.83:0/2921600593 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f50ac0051c0 msgr2=0x7f50ac00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:29.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.851+0000 7f50c0c14640 1 --2- 172.21.15.83:0/2921600593 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f50ac0051c0 0x7f50ac00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.851+0000 7f50c0c14640 1 -- 172.21.15.83:0/2921600593 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f50ac004800 msgr2=0x7f50ac004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:29.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.851+0000 7f50c0c14640 1 --2- 172.21.15.83:0/2921600593 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f50ac004800 0x7f50ac004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.851+0000 7f50c0c14640 1 -- 172.21.15.83:0/2921600593 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50ac003520 con 0x7f50ac0039b0 2024-03-06T07:24:29.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.851+0000 7f50c1415640 1 --2- 172.21.15.83:0/2921600593 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f50ac0051c0 0x7f50ac00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:29.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.851+0000 7f50c1415640 1 -- 172.21.15.83:0/2921600593 reap_dead start 2024-03-06T07:24:29.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.852+0000 7f50c0c14640 1 --2- 172.21.15.83:0/2921600593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50ac0039b0 0x7f50ac003df0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f50b0009ac0 tx=0x7f50ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2fda8cceab624fd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:29.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.852+0000 7f50bb7fe640 1 -- 172.21.15.83:0/2921600593 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b0039070 con 0x7f50ac0039b0 2024-03-06T07:24:29.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.852+0000 7f50bb7fe640 1 -- 172.21.15.83:0/2921600593 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50b002c6b0 con 0x7f50ac0039b0 2024-03-06T07:24:29.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.852+0000 7f50bb7fe640 1 -- 172.21.15.83:0/2921600593 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b0009690 con 0x7f50ac0039b0 2024-03-06T07:24:29.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.852+0000 7f50c2e78640 1 -- 172.21.15.83:0/2921600593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50ac0039b0 msgr2=0x7f50ac003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.852+0000 7f50c2e78640 1 --2- 172.21.15.83:0/2921600593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50ac0039b0 0x7f50ac003df0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f50b0009ac0 tx=0x7f50ac005830 comp rx=0 tx=0).stop 2024-03-06T07:24:29.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.853+0000 7f50c2e78640 1 -- 172.21.15.83:0/2921600593 shutdown_connections 2024-03-06T07:24:29.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.853+0000 7f50c2e78640 1 --2- 172.21.15.83:0/2921600593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50ac0039b0 0x7f50ac003df0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.853+0000 7f50c2e78640 1 -- 172.21.15.83:0/2921600593 >> 172.21.15.83:0/2921600593 conn(0x7f50bc0d3cd0 msgr2=0x7f50bc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:29.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.853+0000 7f50c2e78640 1 -- 172.21.15.83:0/2921600593 shutdown_connections 2024-03-06T07:24:29.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.853+0000 7f50c2e78640 1 -- 172.21.15.83:0/2921600593 wait complete. 2024-03-06T07:24:29.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.854+0000 7f50c2e78640 1 Processor -- start 2024-03-06T07:24:29.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.854+0000 7f50c2e78640 1 -- start start 2024-03-06T07:24:29.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.855+0000 7f50c2e78640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50ac0039b0 0x7f50bc153030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.855+0000 7f50c2e78640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f50bc0b83f0 0x7f50bc0b8870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.855+0000 7f50c2e78640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f50bc160b50 0x7f50bc160fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.855+0000 7f50c2e78640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50bc15e3c0 con 0x7f50ac0039b0 2024-03-06T07:24:29.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.855+0000 7f50c2e78640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50bc0b3d60 con 0x7f50bc160b50 2024-03-06T07:24:29.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.855+0000 7f50c2e78640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50bc15e540 con 0x7f50bc0b83f0 2024-03-06T07:24:29.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.855+0000 7f50c0c14640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50ac0039b0 0x7f50bc153030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.855+0000 7f50c0c14640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50ac0039b0 0x7f50bc153030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43068/0 (socket says 172.21.15.83:43068) 2024-03-06T07:24:29.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.855+0000 7f50c0c14640 1 -- 172.21.15.83:0/53675755 learned_addr learned my addr 172.21.15.83:0/53675755 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:29.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.855+0000 7f50bbfff640 1 --2- 172.21.15.83:0/53675755 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f50bc0b83f0 0x7f50bc0b8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.855+0000 7f50c1415640 1 --2- 172.21.15.83:0/53675755 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f50bc160b50 0x7f50bc160fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.855+0000 7f50c0c14640 1 -- 172.21.15.83:0/53675755 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f50bc0b83f0 msgr2=0x7f50bc0b8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.855+0000 7f50c0c14640 1 --2- 172.21.15.83:0/53675755 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f50bc0b83f0 0x7f50bc0b8870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.855+0000 7f50c0c14640 1 -- 172.21.15.83:0/53675755 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f50bc160b50 msgr2=0x7f50bc160fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.855+0000 7f50c0c14640 1 --2- 172.21.15.83:0/53675755 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f50bc160b50 0x7f50bc160fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.855+0000 7f50c0c14640 1 -- 172.21.15.83:0/53675755 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50bc0ccde0 con 0x7f50ac0039b0 2024-03-06T07:24:29.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.855+0000 7f50c0c14640 1 --2- 172.21.15.83:0/53675755 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50ac0039b0 0x7f50bc153030 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f50b0000c00 tx=0x7f50b0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:29.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.855+0000 7f50b97fa640 1 -- 172.21.15.83:0/53675755 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b0042020 con 0x7f50ac0039b0 2024-03-06T07:24:29.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.855+0000 7f50b97fa640 1 -- 172.21.15.83:0/53675755 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50b003a040 con 0x7f50ac0039b0 2024-03-06T07:24:29.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.855+0000 7f50c2e78640 1 -- 172.21.15.83:0/53675755 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50bc0cd0c0 con 0x7f50ac0039b0 2024-03-06T07:24:29.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.855+0000 7f50b97fa640 1 -- 172.21.15.83:0/53675755 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b0039070 con 0x7f50ac0039b0 2024-03-06T07:24:29.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.855+0000 7f50c2e78640 1 -- 172.21.15.83:0/53675755 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50bc0cd680 con 0x7f50ac0039b0 2024-03-06T07:24:29.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.856+0000 7f50c2e78640 1 -- 172.21.15.83:0/53675755 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50bc04f330 con 0x7f50ac0039b0 2024-03-06T07:24:29.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.857+0000 7f50b97fa640 1 -- 172.21.15.83:0/53675755 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f50b0045020 con 0x7f50ac0039b0 2024-03-06T07:24:29.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.857+0000 7f50b97fa640 1 --2- 172.21.15.83:0/53675755 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f50a006b4d0 0x7f50a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.858+0000 7f50b97fa640 1 -- 172.21.15.83:0/53675755 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f50b00ab5b0 con 0x7f50ac0039b0 2024-03-06T07:24:29.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.858+0000 7f50bbfff640 1 --2- 172.21.15.83:0/53675755 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f50a006b4d0 0x7f50a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.858+0000 7f50bbfff640 1 --2- 172.21.15.83:0/53675755 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f50a006b4d0 0x7f50a006d990 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f50a8000c00 tx=0x7f50a8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:29.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:29.866+0000 7f50b97fa640 1 -- 172.21.15.83:0/53675755 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f50b004c070 con 0x7f50ac0039b0 2024-03-06T07:24:30.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:30.081+0000 7f50c2e78640 1 -- 172.21.15.83:0/53675755 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f50bc0cf580 con 0x7f50ac0039b0 2024-03-06T07:24:30.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:30.081+0000 7f50b97fa640 1 -- 172.21.15.83:0/53675755 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) v1 ==== 53+0+73 (secure 0 0 0) 0x7f50b007a430 con 0x7f50ac0039b0 2024-03-06T07:24:30.083 INFO:teuthology.orchestra.run.smithi083.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:24:30.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:30.083+0000 7f50c2e78640 1 -- 172.21.15.83:0/53675755 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f50a006b4d0 msgr2=0x7f50a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:30.083+0000 7f50c2e78640 1 --2- 172.21.15.83:0/53675755 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f50a006b4d0 0x7f50a006d990 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f50a8000c00 tx=0x7f50a8009040 comp rx=0 tx=0).stop 2024-03-06T07:24:30.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:30.083+0000 7f50c2e78640 1 -- 172.21.15.83:0/53675755 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50ac0039b0 msgr2=0x7f50bc153030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:30.083+0000 7f50c2e78640 1 --2- 172.21.15.83:0/53675755 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50ac0039b0 0x7f50bc153030 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f50b0000c00 tx=0x7f50b0004730 comp rx=0 tx=0).stop 2024-03-06T07:24:30.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:30.083+0000 7f50c2e78640 1 -- 172.21.15.83:0/53675755 shutdown_connections 2024-03-06T07:24:30.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:30.083+0000 7f50c2e78640 1 --2- 172.21.15.83:0/53675755 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f50a006b4d0 0x7f50a006d990 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:30.083+0000 7f50c2e78640 1 --2- 172.21.15.83:0/53675755 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f50bc160b50 0x7f50bc160fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:30.083+0000 7f50c2e78640 1 --2- 172.21.15.83:0/53675755 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f50bc0b83f0 0x7f50bc0b8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:30.083+0000 7f50c2e78640 1 --2- 172.21.15.83:0/53675755 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50ac0039b0 0x7f50bc153030 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:30.083+0000 7f50c2e78640 1 -- 172.21.15.83:0/53675755 >> 172.21.15.83:0/53675755 conn(0x7f50bc0d3cd0 msgr2=0x7f50bc0b4040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:30.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:30.083+0000 7f50c2e78640 1 -- 172.21.15.83:0/53675755 shutdown_connections 2024-03-06T07:24:30.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:30.083+0000 7f50c2e78640 1 -- 172.21.15.83:0/53675755 wait complete. 2024-03-06T07:24:30.095 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:24:30.095 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:24:30.096 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi083.front.sepia.ceph.com 2024-03-06T07:24:30.096 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:24:30.096 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:30.096 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:24:30.096 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:24:30.096 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:24:30.096 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:24:30.096 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:30.096 DEBUG:teuthology.orchestra.run.smithi083:> ip addr 2024-03-06T07:24:30.111 INFO:teuthology.orchestra.run.smithi083.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:24:30.111 INFO:teuthology.orchestra.run.smithi083.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:24:30.111 INFO:teuthology.orchestra.run.smithi083.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:24:30.111 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:30.112 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 ::1/128 scope host 2024-03-06T07:24:30.112 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:30.112 INFO:teuthology.orchestra.run.smithi083.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:24:30.112 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:d8:da:1e brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:30.112 INFO:teuthology.orchestra.run.smithi083.stdout: altname enp7s0f0 2024-03-06T07:24:30.112 INFO:teuthology.orchestra.run.smithi083.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:24:30.112 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:d8:da:1f brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:30.112 INFO:teuthology.orchestra.run.smithi083.stdout: altname enp7s0f1 2024-03-06T07:24:30.112 INFO:teuthology.orchestra.run.smithi083.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:24:30.112 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:8f:ce:90 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:30.112 INFO:teuthology.orchestra.run.smithi083.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:24:30.112 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:8f:ce:91 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:30.112 INFO:teuthology.orchestra.run.smithi083.stdout: inet 172.21.15.83/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:24:30.113 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft 42586sec preferred_lft 42586sec 2024-03-06T07:24:30.113 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::ec4:7aff:fe8f:ce91/64 scope link 2024-03-06T07:24:30.113 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:30.113 INFO:teuthology.orchestra.run.smithi083.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:24:30.113 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 1e:29:46:c9:e7:c8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:30.113 INFO:teuthology.orchestra.run.smithi083.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:24:30.113 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:30.113 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::60c5:d7ff:fe70:2b1c/64 scope link 2024-03-06T07:24:30.113 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:30.113 INFO:teuthology.orchestra.run.smithi083.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:24:30.113 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 1e:29:46:c9:e7:c8 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:30.113 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::1c29:46ff:fec9:e7c8/64 scope link 2024-03-06T07:24:30.113 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:30.113 INFO:teuthology.orchestra.run.smithi083.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:24:30.113 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 3e:00:2f:23:d5:45 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:30.113 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::3c00:2fff:fe23:d545/64 scope link 2024-03-06T07:24:30.114 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:30.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:30.114 DEBUG:teuthology.orchestra.run.smithi083:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:24:30.177 INFO:teuthology.orchestra.run.smithi083.stdout:1 2024-03-06T07:24:30.178 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:30.178 DEBUG:teuthology.orchestra.run.smithi083:> route 2024-03-06T07:24:30.233 INFO:teuthology.orchestra.run.smithi083.stdout:Kernel IP routing table 2024-03-06T07:24:30.233 INFO:teuthology.orchestra.run.smithi083.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:24:30.234 INFO:teuthology.orchestra.run.smithi083.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:24:30.234 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:24:30.234 INFO:teuthology.orchestra.run.smithi083.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:24:30.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:30.234 DEBUG:teuthology.orchestra.run.smithi083:> (cd / && exec bash -c ' 2024-03-06T07:24:30.234 DEBUG:teuthology.orchestra.run.smithi083:> set -e 2024-03-06T07:24:30.234 DEBUG:teuthology.orchestra.run.smithi083:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:24:30.234 DEBUG:teuthology.orchestra.run.smithi083:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:24:30.234 DEBUG:teuthology.orchestra.run.smithi083:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:24:30.234 DEBUG:teuthology.orchestra.run.smithi083:> ') 2024-03-06T07:24:30.329 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:30.329 DEBUG:teuthology.orchestra.run.smithi083:> ip netns list 2024-03-06T07:24:30.344 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:24:30.344 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:24:30.344 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:24:30.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:30.344 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip link set brx.0 up 2024-03-06T07:24:30.416 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:24:30.416 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:24:30.416 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:30.471 INFO:teuthology.orchestra.run.smithi083.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:24:30.471 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:24:30.471 DEBUG:teuthology.orchestra.run.smithi083:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:30.526 DEBUG:teuthology.orchestra.run.smithi083:> sudo modprobe fuse 2024-03-06T07:24:30.594 DEBUG:teuthology.orchestra.run.smithi083:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:24:30.650 INFO:teuthology.orchestra.run.smithi083.stdout:/proc 2024-03-06T07:24:30.650 INFO:teuthology.orchestra.run.smithi083.stdout:/sys 2024-03-06T07:24:30.650 INFO:teuthology.orchestra.run.smithi083.stdout:/dev 2024-03-06T07:24:30.650 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/security 2024-03-06T07:24:30.650 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/shm 2024-03-06T07:24:30.650 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/pts 2024-03-06T07:24:30.650 INFO:teuthology.orchestra.run.smithi083.stdout:/run 2024-03-06T07:24:30.650 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/cgroup 2024-03-06T07:24:30.651 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/pstore 2024-03-06T07:24:30.651 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/bpf 2024-03-06T07:24:30.651 INFO:teuthology.orchestra.run.smithi083.stdout:/ 2024-03-06T07:24:30.651 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/selinux 2024-03-06T07:24:30.651 INFO:teuthology.orchestra.run.smithi083.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:24:30.651 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/mqueue 2024-03-06T07:24:30.651 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/hugepages 2024-03-06T07:24:30.651 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/debug 2024-03-06T07:24:30.651 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/tracing 2024-03-06T07:24:30.651 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:24:30.651 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/fuse/connections 2024-03-06T07:24:30.651 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/config 2024-03-06T07:24:30.651 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:24:30.652 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:24:30.652 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:24:30.652 INFO:teuthology.orchestra.run.smithi083.stdout:/run/user/1000 2024-03-06T07:24:30.652 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph 2024-03-06T07:24:30.652 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:24:30.652 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:24:30.652 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:24:30.652 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:24:30.652 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns 2024-03-06T07:24:30.652 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:30.652 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:30.652 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:30.653 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:30.653 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:30.654 DEBUG:teuthology.orchestra.run.smithi083:> ls /sys/fs/fuse/connections 2024-03-06T07:24:30.707 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:24:30.708 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:24:30.750 DEBUG:teuthology.orchestra.run.smithi083:> sudo modprobe fuse 2024-03-06T07:24:30.777 DEBUG:teuthology.orchestra.run.smithi083:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:24:30.834 INFO:teuthology.orchestra.run.smithi083.stdout:/proc 2024-03-06T07:24:30.834 INFO:teuthology.orchestra.run.smithi083.stdout:/sys 2024-03-06T07:24:30.834 INFO:teuthology.orchestra.run.smithi083.stdout:/dev 2024-03-06T07:24:30.834 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/security 2024-03-06T07:24:30.834 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/shm 2024-03-06T07:24:30.834 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/pts 2024-03-06T07:24:30.834 INFO:teuthology.orchestra.run.smithi083.stdout:/run 2024-03-06T07:24:30.835 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/cgroup 2024-03-06T07:24:30.835 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/pstore 2024-03-06T07:24:30.835 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/bpf 2024-03-06T07:24:30.835 INFO:teuthology.orchestra.run.smithi083.stdout:/ 2024-03-06T07:24:30.835 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/selinux 2024-03-06T07:24:30.835 INFO:teuthology.orchestra.run.smithi083.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:24:30.835 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/mqueue 2024-03-06T07:24:30.835 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/hugepages 2024-03-06T07:24:30.835 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/debug 2024-03-06T07:24:30.835 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/tracing 2024-03-06T07:24:30.835 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:24:30.835 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/fuse/connections 2024-03-06T07:24:30.835 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/config 2024-03-06T07:24:30.835 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:24:30.836 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:24:30.836 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:24:30.836 INFO:teuthology.orchestra.run.smithi083.stdout:/run/user/1000 2024-03-06T07:24:30.836 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph 2024-03-06T07:24:30.836 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:24:30.836 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:24:30.836 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:24:30.836 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:24:30.836 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns 2024-03-06T07:24:30.836 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:30.836 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:30.836 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:30.836 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:30.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:30.837 DEBUG:teuthology.orchestra.run.smithi083:> ls /sys/fs/fuse/connections 2024-03-06T07:24:30.838 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:2024-03-06T07:24:30.834+0000 7f61c8c43480 -1 init, newargv = 0x5577baf7b7e0 newargc=15 2024-03-06T07:24:30.838 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:ceph-fuse[59150]: starting ceph client 2024-03-06T07:24:30.841 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:ceph-fuse[59150]: starting fuse 2024-03-06T07:24:30.852 INFO:teuthology.orchestra.run.smithi083.stdout:41 2024-03-06T07:24:30.853 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-06T07:24:30.853 DEBUG:teuthology.orchestra.run.smithi083:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:24:30.853 DEBUG:teuthology.orchestra.run.smithi083:> import glob 2024-03-06T07:24:30.853 DEBUG:teuthology.orchestra.run.smithi083:> import re 2024-03-06T07:24:30.853 DEBUG:teuthology.orchestra.run.smithi083:> import os 2024-03-06T07:24:30.853 DEBUG:teuthology.orchestra.run.smithi083:> import subprocess 2024-03-06T07:24:30.853 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:24:30.853 DEBUG:teuthology.orchestra.run.smithi083:> def _find_admin_socket(client_name): 2024-03-06T07:24:30.853 DEBUG:teuthology.orchestra.run.smithi083:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:24:30.853 DEBUG:teuthology.orchestra.run.smithi083:> files = glob.glob(asok_path) 2024-03-06T07:24:30.853 DEBUG:teuthology.orchestra.run.smithi083:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:24:30.853 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:24:30.854 DEBUG:teuthology.orchestra.run.smithi083:> # Given a non-glob path, it better be there 2024-03-06T07:24:30.854 DEBUG:teuthology.orchestra.run.smithi083:> if "*" not in asok_path: 2024-03-06T07:24:30.854 DEBUG:teuthology.orchestra.run.smithi083:> assert(len(files) == 1) 2024-03-06T07:24:30.854 DEBUG:teuthology.orchestra.run.smithi083:> return files[0] 2024-03-06T07:24:30.854 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:24:30.854 DEBUG:teuthology.orchestra.run.smithi083:> for f in files: 2024-03-06T07:24:30.854 DEBUG:teuthology.orchestra.run.smithi083:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:24:30.854 DEBUG:teuthology.orchestra.run.smithi083:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:24:30.854 DEBUG:teuthology.orchestra.run.smithi083:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:24:30.854 DEBUG:teuthology.orchestra.run.smithi083:> contents = proc_f.read() 2024-03-06T07:24:30.854 DEBUG:teuthology.orchestra.run.smithi083:> if mountpoint in contents: 2024-03-06T07:24:30.854 DEBUG:teuthology.orchestra.run.smithi083:> return f 2024-03-06T07:24:30.854 DEBUG:teuthology.orchestra.run.smithi083:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:24:30.854 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:24:30.854 DEBUG:teuthology.orchestra.run.smithi083:> print(_find_admin_socket("client.0")) 2024-03-06T07:24:30.854 DEBUG:teuthology.orchestra.run.smithi083:> ' 2024-03-06T07:24:30.972 INFO:teuthology.orchestra.run.smithi083.stdout:/var/run/ceph/ceph-client.0.59150.asok 2024-03-06T07:24:30.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:24:30.983 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.59150.asok 2024-03-06T07:24:30.983 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:30.983 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.59150.asok status 2024-03-06T07:24:31.121 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:24:31.121 INFO:teuthology.orchestra.run.smithi083.stdout: "metadata": { 2024-03-06T07:24:31.121 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:24:31.121 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:24:31.121 INFO:teuthology.orchestra.run.smithi083.stdout: "entity_id": "0", 2024-03-06T07:24:31.121 INFO:teuthology.orchestra.run.smithi083.stdout: "hostname": "smithi083", 2024-03-06T07:24:31.121 INFO:teuthology.orchestra.run.smithi083.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:24:31.121 INFO:teuthology.orchestra.run.smithi083.stdout: "pid": "59150", 2024-03-06T07:24:31.121 INFO:teuthology.orchestra.run.smithi083.stdout: "root": "/" 2024-03-06T07:24:31.121 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:24:31.121 INFO:teuthology.orchestra.run.smithi083.stdout: "dentry_count": 0, 2024-03-06T07:24:31.122 INFO:teuthology.orchestra.run.smithi083.stdout: "dentry_pinned_count": 0, 2024-03-06T07:24:31.122 INFO:teuthology.orchestra.run.smithi083.stdout: "id": 6264, 2024-03-06T07:24:31.122 INFO:teuthology.orchestra.run.smithi083.stdout: "inst": { 2024-03-06T07:24:31.122 INFO:teuthology.orchestra.run.smithi083.stdout: "name": { 2024-03-06T07:24:31.122 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "client", 2024-03-06T07:24:31.122 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 6264 2024-03-06T07:24:31.122 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:24:31.122 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": { 2024-03-06T07:24:31.122 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-03-06T07:24:31.122 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:24:31.122 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 136119941 2024-03-06T07:24:31.122 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:24:31.122 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:24:31.122 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": { 2024-03-06T07:24:31.123 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-03-06T07:24:31.123 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:24:31.123 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 136119941 2024-03-06T07:24:31.123 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:24:31.123 INFO:teuthology.orchestra.run.smithi083.stdout: "inst_str": "client.6264 192.168.0.1:0/136119941", 2024-03-06T07:24:31.123 INFO:teuthology.orchestra.run.smithi083.stdout: "addr_str": "192.168.0.1:0/136119941", 2024-03-06T07:24:31.123 INFO:teuthology.orchestra.run.smithi083.stdout: "inode_count": 1, 2024-03-06T07:24:31.123 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_epoch": 25, 2024-03-06T07:24:31.123 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_epoch": 57, 2024-03-06T07:24:31.123 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:24:31.123 INFO:teuthology.orchestra.run.smithi083.stdout: "blocklisted": false, 2024-03-06T07:24:31.123 INFO:teuthology.orchestra.run.smithi083.stdout: "fs_name": "cephfs" 2024-03-06T07:24:31.123 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:24:31.130 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:31.130 DEBUG:teuthology.orchestra.run.smithi083:> stat --file-system '--printf=%T 2024-03-06T07:24:31.130 DEBUG:teuthology.orchestra.run.smithi083:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:31.145 INFO:teuthology.orchestra.run.smithi083.stdout:fuseblk 2024-03-06T07:24:31.145 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:31.145 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:31.145 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:31.211 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_fragment_size_max 50 2024-03-06T07:24:31.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.354+0000 7fbc1604e640 1 Processor -- start 2024-03-06T07:24:31.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.354+0000 7fbc1604e640 1 -- start start 2024-03-06T07:24:31.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.355+0000 7fbc1604e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fbc1006b6c0 0x7fbc1015e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.355+0000 7fbc1604e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbc1015ec40 0x7fbc10071ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.355+0000 7fbc1604e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc100cea90 con 0x7fbc1015bbb0 2024-03-06T07:24:31.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.355+0000 7fbc1604e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc10161970 con 0x7fbc1006b6c0 2024-03-06T07:24:31.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.355+0000 7fbc1604e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc101624f0 con 0x7fbc1015ec40 2024-03-06T07:24:31.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.355+0000 7fbc06ffd640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fbc1006b6c0 0x7fbc1015e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.355+0000 7fbc0ffff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbc1015ec40 0x7fbc10071ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.355+0000 7fbc0f7fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fbc1015bbb0 0x7fbc1006afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:58446/0 (socket says 172.21.15.83:58446) 2024-03-06T07:24:31.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.355+0000 7fbc0f7fe640 1 -- 172.21.15.83:0/2028548652 learned_addr learned my addr 172.21.15.83:0/2028548652 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:24:31.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.356+0000 7fbc0effd640 1 -- 172.21.15.83:0/2028548652 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3419370985 0 0) 0x7fbc00001110 con 0x7fbc1015bbb0 2024-03-06T07:24:31.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.356+0000 7fbc0effd640 1 -- 172.21.15.83:0/2028548652 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbfc003520 con 0x7fbc1015bbb0 2024-03-06T07:24:31.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.356+0000 7fbc0ffff640 1 -- 172.21.15.83:0/2028548652 >> v2:172.21.15.183:3300/0 conn(0x7fbc1006b6c0 msgr2=0x7fbc1015e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:31.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.356+0000 7fbc0ffff640 1 --2- 172.21.15.83:0/2028548652 >> v2:172.21.15.183:3300/0 conn(0x7fbc1006b6c0 0x7fbc1015e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.356+0000 7fbc0ffff640 1 -- 172.21.15.83:0/2028548652 >> v1:172.21.15.183:6789/0 conn(0x7fbc1015bbb0 legacy=0x7fbc1006afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.356+0000 7fbc0ffff640 1 -- 172.21.15.83:0/2028548652 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc10161bc0 con 0x7fbc1015ec40 2024-03-06T07:24:31.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.356+0000 7fbc06ffd640 1 --2- 172.21.15.83:0/2028548652 >> v2:172.21.15.183:3300/0 conn(0x7fbc1006b6c0 0x7fbc1015e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:31.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.357+0000 7fbc0ffff640 1 --2- 172.21.15.83:0/2028548652 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbc1015ec40 0x7fbc10071ca0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fbc08009d90 tx=0x7fbc0802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2302b685022f78a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:31.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.357+0000 7fbc0effd640 1 -- 172.21.15.83:0/2028548652 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc08039070 con 0x7fbc1015ec40 2024-03-06T07:24:31.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.357+0000 7fbc0effd640 1 -- 172.21.15.83:0/2028548652 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc0802cc50 con 0x7fbc1015ec40 2024-03-06T07:24:31.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.357+0000 7fbc0effd640 1 -- 172.21.15.83:0/2028548652 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc08034440 con 0x7fbc1015ec40 2024-03-06T07:24:31.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.357+0000 7fbc1604e640 1 -- 172.21.15.83:0/2028548652 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbc1015ec40 msgr2=0x7fbc10071ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.357+0000 7fbc1604e640 1 --2- 172.21.15.83:0/2028548652 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbc1015ec40 0x7fbc10071ca0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fbc08009d90 tx=0x7fbc0802bea0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.357+0000 7fbc1604e640 1 -- 172.21.15.83:0/2028548652 shutdown_connections 2024-03-06T07:24:31.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.357+0000 7fbc1604e640 1 --2- 172.21.15.83:0/2028548652 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbc1015ec40 0x7fbc10071ca0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.357+0000 7fbc1604e640 1 --2- 172.21.15.83:0/2028548652 >> v2:172.21.15.183:3300/0 conn(0x7fbc1006b6c0 0x7fbc1015e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.357+0000 7fbc1604e640 1 -- 172.21.15.83:0/2028548652 >> 172.21.15.83:0/2028548652 conn(0x7fbc100b2660 msgr2=0x7fbc100b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:31.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.357+0000 7fbc1604e640 1 -- 172.21.15.83:0/2028548652 shutdown_connections 2024-03-06T07:24:31.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.357+0000 7fbc1604e640 1 -- 172.21.15.83:0/2028548652 wait complete. 2024-03-06T07:24:31.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.358+0000 7fbc1604e640 1 Processor -- start 2024-03-06T07:24:31.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.359+0000 7fbc1604e640 1 -- start start 2024-03-06T07:24:31.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.359+0000 7fbc1604e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbc1006b6c0 0x7fbc101f3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.359+0000 7fbc1604e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbc1015ec40 0x7fbc1015f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.359+0000 7fbc1604e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc1015a020 0x7fbc1015bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.359+0000 7fbc1604e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc1015f0f0 con 0x7fbc1015a020 2024-03-06T07:24:31.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.359+0000 7fbc1604e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc10162180 con 0x7fbc1015ec40 2024-03-06T07:24:31.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.359+0000 7fbc1604e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc100721e0 con 0x7fbc1006b6c0 2024-03-06T07:24:31.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.359+0000 7fbc0ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc1015a020 0x7fbc1015bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.359+0000 7fbc0ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc1015a020 0x7fbc1015bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43074/0 (socket says 172.21.15.83:43074) 2024-03-06T07:24:31.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.359+0000 7fbc0ffff640 1 -- 172.21.15.83:0/465328286 learned_addr learned my addr 172.21.15.83:0/465328286 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:31.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.360+0000 7fbc0f7fe640 1 --2- 172.21.15.83:0/465328286 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbc1006b6c0 0x7fbc101f3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.360+0000 7fbc06ffd640 1 --2- 172.21.15.83:0/465328286 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbc1015ec40 0x7fbc1015f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.360+0000 7fbc0ffff640 1 -- 172.21.15.83:0/465328286 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbc1006b6c0 msgr2=0x7fbc101f3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.360+0000 7fbc0ffff640 1 --2- 172.21.15.83:0/465328286 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbc1006b6c0 0x7fbc101f3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.360+0000 7fbc0ffff640 1 -- 172.21.15.83:0/465328286 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbc1015ec40 msgr2=0x7fbc1015f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.360+0000 7fbc0ffff640 1 --2- 172.21.15.83:0/465328286 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbc1015ec40 0x7fbc1015f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.360+0000 7fbc0ffff640 1 -- 172.21.15.83:0/465328286 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc1015a5c0 con 0x7fbc1015a020 2024-03-06T07:24:31.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.360+0000 7fbc0ffff640 1 --2- 172.21.15.83:0/465328286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc1015a020 0x7fbc1015bfa0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fbc08009d90 tx=0x7fbc08004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:31.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.360+0000 7fbc0cff9640 1 -- 172.21.15.83:0/465328286 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc08039050 con 0x7fbc1015a020 2024-03-06T07:24:31.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.360+0000 7fbc0cff9640 1 -- 172.21.15.83:0/465328286 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc08045070 con 0x7fbc1015a020 2024-03-06T07:24:31.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.360+0000 7fbc1604e640 1 -- 172.21.15.83:0/465328286 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc1015a8a0 con 0x7fbc1015a020 2024-03-06T07:24:31.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.360+0000 7fbc0cff9640 1 -- 172.21.15.83:0/465328286 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc08009450 con 0x7fbc1015a020 2024-03-06T07:24:31.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.360+0000 7fbc1604e640 1 -- 172.21.15.83:0/465328286 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc1015ae60 con 0x7fbc1015a020 2024-03-06T07:24:31.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.361+0000 7fbc1604e640 1 -- 172.21.15.83:0/465328286 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc100733b0 con 0x7fbc1015a020 2024-03-06T07:24:31.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.362+0000 7fbc0cff9640 1 -- 172.21.15.83:0/465328286 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbc0803e610 con 0x7fbc1015a020 2024-03-06T07:24:31.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.362+0000 7fbc0cff9640 1 --2- 172.21.15.83:0/465328286 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbbf006b4d0 0x7fbbf006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.362+0000 7fbc0cff9640 1 -- 172.21.15.83:0/465328286 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbc080acca0 con 0x7fbc1015a020 2024-03-06T07:24:31.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.362+0000 7fbc0f7fe640 1 --2- 172.21.15.83:0/465328286 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbbf006b4d0 0x7fbbf006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.363+0000 7fbc0f7fe640 1 --2- 172.21.15.83:0/465328286 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbbf006b4d0 0x7fbbf006d990 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fbc00005880 tx=0x7fbc00009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:31.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.371+0000 7fbc0cff9640 1 -- 172.21.15.83:0/465328286 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbc0807bbe0 con 0x7fbc1015a020 2024-03-06T07:24:31.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.549+0000 7fbc1604e640 1 -- 172.21.15.83:0/465328286 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) v1 -- 0x7fbc10068ff0 con 0x7fbc1015a020 2024-03-06T07:24:31.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.552+0000 7fbc0cff9640 1 -- 172.21.15.83:0/465328286 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v17)=0 v17) v1 ==== 124+0+0 (secure 0 0 0) 0x7fbc0807b4e0 con 0x7fbc1015a020 2024-03-06T07:24:31.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.554+0000 7fbc1604e640 1 -- 172.21.15.83:0/465328286 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbbf006b4d0 msgr2=0x7fbbf006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.554+0000 7fbc1604e640 1 --2- 172.21.15.83:0/465328286 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbbf006b4d0 0x7fbbf006d990 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fbc00005880 tx=0x7fbc00009040 comp rx=0 tx=0).stop 2024-03-06T07:24:31.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.554+0000 7fbc1604e640 1 -- 172.21.15.83:0/465328286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc1015a020 msgr2=0x7fbc1015bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.554+0000 7fbc1604e640 1 --2- 172.21.15.83:0/465328286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc1015a020 0x7fbc1015bfa0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fbc08009d90 tx=0x7fbc08004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:31.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.554+0000 7fbc1604e640 1 -- 172.21.15.83:0/465328286 shutdown_connections 2024-03-06T07:24:31.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.554+0000 7fbc1604e640 1 --2- 172.21.15.83:0/465328286 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbbf006b4d0 0x7fbbf006d990 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.554+0000 7fbc1604e640 1 --2- 172.21.15.83:0/465328286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc1015a020 0x7fbc1015bfa0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.554+0000 7fbc1604e640 1 --2- 172.21.15.83:0/465328286 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbc1015ec40 0x7fbc1015f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.554+0000 7fbc1604e640 1 --2- 172.21.15.83:0/465328286 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbc1006b6c0 0x7fbc101f3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.554+0000 7fbc1604e640 1 -- 172.21.15.83:0/465328286 >> 172.21.15.83:0/465328286 conn(0x7fbc100b2660 msgr2=0x7fbc100ca800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:31.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.554+0000 7fbc1604e640 1 -- 172.21.15.83:0/465328286 shutdown_connections 2024-03-06T07:24:31.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.554+0000 7fbc1604e640 1 -- 172.21.15.83:0/465328286 wait complete. 2024-03-06T07:24:31.567 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_merge_size 1 2024-03-06T07:24:31.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.667+0000 7febea165640 1 Processor -- start 2024-03-06T07:24:31.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.667+0000 7febea165640 1 -- start start 2024-03-06T07:24:31.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.668+0000 7febea165640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7febe41614d0 0x7febe40d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.668+0000 7febea165640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7febe40d1910 0x7febe40d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.668+0000 7febea165640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7febe40d22b0 0x7febe40d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.668+0000 7febea165640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febe40d2740 con 0x7febe40d22b0 2024-03-06T07:24:31.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.668+0000 7febea165640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febe41618b0 con 0x7febe40d1910 2024-03-06T07:24:31.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.668+0000 7febea165640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febe40cf400 con 0x7febe41614d0 2024-03-06T07:24:31.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.668+0000 7febe3fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7febe40d22b0 0x7febe40d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.668+0000 7febe3fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7febe40d22b0 0x7febe40d06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43086/0 (socket says 172.21.15.83:43086) 2024-03-06T07:24:31.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.668+0000 7febe3fff640 1 -- 172.21.15.83:0/583210121 learned_addr learned my addr 172.21.15.83:0/583210121 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:31.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.668+0000 7febe37fe640 1 --2- 172.21.15.83:0/583210121 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7febe41614d0 0x7febe40d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.668+0000 7febdbfff640 1 --2- 172.21.15.83:0/583210121 >> v2:172.21.15.183:3300/0 conn(0x7febe40d1910 0x7febe40d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.669+0000 7febe3fff640 1 -- 172.21.15.83:0/583210121 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7febe41614d0 msgr2=0x7febe40d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:31.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.669+0000 7febe3fff640 1 --2- 172.21.15.83:0/583210121 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7febe41614d0 0x7febe40d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.669+0000 7febe3fff640 1 -- 172.21.15.83:0/583210121 >> v2:172.21.15.183:3300/0 conn(0x7febe40d1910 msgr2=0x7febe40d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:31.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.669+0000 7febe3fff640 1 --2- 172.21.15.83:0/583210121 >> v2:172.21.15.183:3300/0 conn(0x7febe40d1910 0x7febe40d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.669+0000 7febe3fff640 1 -- 172.21.15.83:0/583210121 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe40cf6a0 con 0x7febe40d22b0 2024-03-06T07:24:31.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.669+0000 7febe37fe640 1 --2- 172.21.15.83:0/583210121 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7febe41614d0 0x7febe40d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:31.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.669+0000 7febdbfff640 1 --2- 172.21.15.83:0/583210121 >> v2:172.21.15.183:3300/0 conn(0x7febe40d1910 0x7febe40d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:31.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.669+0000 7febe3fff640 1 --2- 172.21.15.83:0/583210121 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7febe40d22b0 0x7febe40d06a0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7febe408c4f0 tx=0x7febdc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=517b806f406d88e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:31.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.669+0000 7febe2ffd640 1 -- 172.21.15.83:0/583210121 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febdc02ca70 con 0x7febe40d22b0 2024-03-06T07:24:31.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.669+0000 7febe2ffd640 1 -- 172.21.15.83:0/583210121 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febdc02cc10 con 0x7febe40d22b0 2024-03-06T07:24:31.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.669+0000 7febe2ffd640 1 -- 172.21.15.83:0/583210121 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febdc0343f0 con 0x7febe40d22b0 2024-03-06T07:24:31.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.670+0000 7febea165640 1 -- 172.21.15.83:0/583210121 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7febe40d22b0 msgr2=0x7febe40d06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.670+0000 7febea165640 1 --2- 172.21.15.83:0/583210121 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7febe40d22b0 0x7febe40d06a0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7febe408c4f0 tx=0x7febdc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.670+0000 7febea165640 1 -- 172.21.15.83:0/583210121 shutdown_connections 2024-03-06T07:24:31.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.670+0000 7febea165640 1 --2- 172.21.15.83:0/583210121 >> v2:172.21.15.183:3300/0 conn(0x7febe40d1910 0x7febe40d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.670+0000 7febea165640 1 --2- 172.21.15.83:0/583210121 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7febe40d22b0 0x7febe40d06a0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.670+0000 7febea165640 1 --2- 172.21.15.83:0/583210121 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7febe41614d0 0x7febe40d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.670+0000 7febea165640 1 -- 172.21.15.83:0/583210121 >> 172.21.15.83:0/583210121 conn(0x7febe40d3d40 msgr2=0x7febe40d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:31.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.670+0000 7febea165640 1 -- 172.21.15.83:0/583210121 shutdown_connections 2024-03-06T07:24:31.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.670+0000 7febea165640 1 -- 172.21.15.83:0/583210121 wait complete. 2024-03-06T07:24:31.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.671+0000 7febea165640 1 Processor -- start 2024-03-06T07:24:31.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.671+0000 7febea165640 1 -- start start 2024-03-06T07:24:31.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.672+0000 7febea165640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7febe40d1910 0x7febe40a0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.672+0000 7febea165640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7febe40d22b0 0x7febe40cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.672+0000 7febea165640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7febe41614d0 0x7febe40cb350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.672+0000 7febea165640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe40b0910 con 0x7febe40d22b0 2024-03-06T07:24:31.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.672+0000 7febea165640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe40cfc60 con 0x7febe41614d0 2024-03-06T07:24:31.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.672+0000 7febea165640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febe40d0be0 con 0x7febe40d1910 2024-03-06T07:24:31.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.672+0000 7febdbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7febe40d22b0 0x7febe40cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.672+0000 7febdbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7febe40d22b0 0x7febe40cae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43100/0 (socket says 172.21.15.83:43100) 2024-03-06T07:24:31.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.672+0000 7febdbfff640 1 -- 172.21.15.83:0/3500406924 learned_addr learned my addr 172.21.15.83:0/3500406924 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:31.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.672+0000 7febe37fe640 1 --2- 172.21.15.83:0/3500406924 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7febe40d1910 0x7febe40a0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.672+0000 7febe3fff640 1 --2- 172.21.15.83:0/3500406924 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7febe41614d0 0x7febe40cb350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.672+0000 7febdbfff640 1 -- 172.21.15.83:0/3500406924 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7febe40d1910 msgr2=0x7febe40a0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.673+0000 7febdbfff640 1 --2- 172.21.15.83:0/3500406924 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7febe40d1910 0x7febe40a0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.673+0000 7febdbfff640 1 -- 172.21.15.83:0/3500406924 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7febe41614d0 msgr2=0x7febe40cb350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.673+0000 7febdbfff640 1 --2- 172.21.15.83:0/3500406924 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7febe41614d0 0x7febe40cb350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.673+0000 7febdbfff640 1 -- 172.21.15.83:0/3500406924 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe40b77e0 con 0x7febe40d22b0 2024-03-06T07:24:31.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.673+0000 7febdbfff640 1 --2- 172.21.15.83:0/3500406924 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7febe40d22b0 0x7febe40cae10 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7febcc00e810 tx=0x7febcc00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:31.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.673+0000 7febe0ff9640 1 -- 172.21.15.83:0/3500406924 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febcc00f040 con 0x7febe40d22b0 2024-03-06T07:24:31.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.673+0000 7febe0ff9640 1 -- 172.21.15.83:0/3500406924 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febcc004030 con 0x7febe40d22b0 2024-03-06T07:24:31.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.673+0000 7febea165640 1 -- 172.21.15.83:0/3500406924 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febe40b7a80 con 0x7febe40d22b0 2024-03-06T07:24:31.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.673+0000 7febe0ff9640 1 -- 172.21.15.83:0/3500406924 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febcc004400 con 0x7febe40d22b0 2024-03-06T07:24:31.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.673+0000 7febea165640 1 -- 172.21.15.83:0/3500406924 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febe40b8040 con 0x7febe40d22b0 2024-03-06T07:24:31.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.674+0000 7febea165640 1 -- 172.21.15.83:0/3500406924 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febe40d2070 con 0x7febe40d22b0 2024-03-06T07:24:31.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.675+0000 7febe0ff9640 1 -- 172.21.15.83:0/3500406924 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7febcc00c6d0 con 0x7febe40d22b0 2024-03-06T07:24:31.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.675+0000 7febe0ff9640 1 --2- 172.21.15.83:0/3500406924 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7febc406b4d0 0x7febc406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.675+0000 7febe0ff9640 1 -- 172.21.15.83:0/3500406924 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7febcc089f00 con 0x7febe40d22b0 2024-03-06T07:24:31.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.675+0000 7febe37fe640 1 --2- 172.21.15.83:0/3500406924 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7febc406b4d0 0x7febc406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.675+0000 7febe37fe640 1 --2- 172.21.15.83:0/3500406924 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7febc406b4d0 0x7febc406d990 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7febd4006fd0 tx=0x7febd4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:31.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:31.679+0000 7febe0ff9640 1 -- 172.21.15.83:0/3500406924 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7febe40d2070 con 0x7febe40d22b0 2024-03-06T07:24:32.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:32.010+0000 7febea165640 1 -- 172.21.15.83:0/3500406924 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command([{prefix=config set, name=mds_bal_merge_size}] v 0) v1 -- 0x7febe40a1c90 con 0x7febe40d22b0 2024-03-06T07:24:32.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:32.013+0000 7febe0ff9640 1 -- 172.21.15.83:0/3500406924 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_merge_size}]=0 v18)=0 v18) v1 ==== 116+0+0 (secure 0 0 0) 0x7febcc017020 con 0x7febe40d22b0 2024-03-06T07:24:32.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:32.016+0000 7febea165640 1 -- 172.21.15.83:0/3500406924 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7febc406b4d0 msgr2=0x7febc406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:32.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:32.016+0000 7febea165640 1 --2- 172.21.15.83:0/3500406924 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7febc406b4d0 0x7febc406d990 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7febd4006fd0 tx=0x7febd4008040 comp rx=0 tx=0).stop 2024-03-06T07:24:32.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:32.016+0000 7febea165640 1 -- 172.21.15.83:0/3500406924 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7febe40d22b0 msgr2=0x7febe40cae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:32.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:32.016+0000 7febea165640 1 --2- 172.21.15.83:0/3500406924 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7febe40d22b0 0x7febe40cae10 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7febcc00e810 tx=0x7febcc00ecd0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:32.016+0000 7febea165640 1 -- 172.21.15.83:0/3500406924 shutdown_connections 2024-03-06T07:24:32.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:32.016+0000 7febea165640 1 --2- 172.21.15.83:0/3500406924 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7febc406b4d0 0x7febc406d990 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:32.016+0000 7febea165640 1 --2- 172.21.15.83:0/3500406924 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7febe41614d0 0x7febe40cb350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:32.016+0000 7febea165640 1 --2- 172.21.15.83:0/3500406924 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7febe40d22b0 0x7febe40cae10 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:32.016+0000 7febea165640 1 --2- 172.21.15.83:0/3500406924 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7febe40d1910 0x7febe40a0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:32.016+0000 7febea165640 1 -- 172.21.15.83:0/3500406924 >> 172.21.15.83:0/3500406924 conn(0x7febe40d3d40 msgr2=0x7febe40b15b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:32.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:32.016+0000 7febea165640 1 -- 172.21.15.83:0/3500406924 shutdown_connections 2024-03-06T07:24:32.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:32.016+0000 7febea165640 1 -- 172.21.15.83:0/3500406924 wait complete. 2024-03-06T07:24:42.029 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:24:42.029 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:42.045 DEBUG:teuthology.orchestra.run.smithi083:> stdin-killer -- python3 -c ' 2024-03-06T07:24:42.045 DEBUG:teuthology.orchestra.run.smithi083:> import os 2024-03-06T07:24:42.045 DEBUG:teuthology.orchestra.run.smithi083:> import uuid 2024-03-06T07:24:42.045 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:24:42.045 DEBUG:teuthology.orchestra.run.smithi083:> n = 50 2024-03-06T07:24:42.045 DEBUG:teuthology.orchestra.run.smithi083:> create_hard_links = False 2024-03-06T07:24:42.045 DEBUG:teuthology.orchestra.run.smithi083:> if 0 > 0: 2024-03-06T07:24:42.045 DEBUG:teuthology.orchestra.run.smithi083:> create_hard_links = True 2024-03-06T07:24:42.045 DEBUG:teuthology.orchestra.run.smithi083:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:24:42.045 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:24:42.045 DEBUG:teuthology.orchestra.run.smithi083:> dpath = os.path.dirname(path) 2024-03-06T07:24:42.045 DEBUG:teuthology.orchestra.run.smithi083:> fnameprefix = os.path.basename(path) 2024-03-06T07:24:42.045 DEBUG:teuthology.orchestra.run.smithi083:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:24:42.045 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:24:42.046 DEBUG:teuthology.orchestra.run.smithi083:> try: 2024-03-06T07:24:42.046 DEBUG:teuthology.orchestra.run.smithi083:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:24:42.046 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:24:42.046 DEBUG:teuthology.orchestra.run.smithi083:> for i in range(n): 2024-03-06T07:24:42.046 DEBUG:teuthology.orchestra.run.smithi083:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:24:42.046 DEBUG:teuthology.orchestra.run.smithi083:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:24:42.046 DEBUG:teuthology.orchestra.run.smithi083:> f.write(f"{i}") 2024-03-06T07:24:42.046 DEBUG:teuthology.orchestra.run.smithi083:> if False: 2024-03-06T07:24:42.046 DEBUG:teuthology.orchestra.run.smithi083:> f.flush() 2024-03-06T07:24:42.046 DEBUG:teuthology.orchestra.run.smithi083:> os.fsync(f.fileno()) 2024-03-06T07:24:42.046 DEBUG:teuthology.orchestra.run.smithi083:> if False: 2024-03-06T07:24:42.046 DEBUG:teuthology.orchestra.run.smithi083:> os.unlink(fpath) 2024-03-06T07:24:42.046 DEBUG:teuthology.orchestra.run.smithi083:> if False: 2024-03-06T07:24:42.046 DEBUG:teuthology.orchestra.run.smithi083:> os.fsync(dirfd) 2024-03-06T07:24:42.046 DEBUG:teuthology.orchestra.run.smithi083:> if create_hard_links: 2024-03-06T07:24:42.046 DEBUG:teuthology.orchestra.run.smithi083:> for j in range(0): 2024-03-06T07:24:42.047 DEBUG:teuthology.orchestra.run.smithi083:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:24:42.047 DEBUG:teuthology.orchestra.run.smithi083:> if True: 2024-03-06T07:24:42.047 DEBUG:teuthology.orchestra.run.smithi083:> os.fsync(dirfd) 2024-03-06T07:24:42.047 DEBUG:teuthology.orchestra.run.smithi083:> finally: 2024-03-06T07:24:42.047 DEBUG:teuthology.orchestra.run.smithi083:> os.close(dirfd) 2024-03-06T07:24:42.047 DEBUG:teuthology.orchestra.run.smithi083:> ' 2024-03-06T07:24:42.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:24:42.262 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:24:42.262 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:42.317 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:42.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.456+0000 7f0c2fc0f640 1 Processor -- start 2024-03-06T07:24:42.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.456+0000 7f0c2fc0f640 1 -- start start 2024-03-06T07:24:42.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.457+0000 7f0c2fc0f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0c280b4170 0x7f0c280b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.457+0000 7f0c2fc0f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0c280d4760 0x7f0c280d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.457+0000 7f0c2fc0f640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f0c280b4dd0 0x7f0c280d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.457+0000 7f0c2fc0f640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c280d4bf0 con 0x7f0c280d4760 2024-03-06T07:24:42.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.457+0000 7f0c2fc0f640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c280ce9e0 con 0x7f0c280b4dd0 2024-03-06T07:24:42.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.457+0000 7f0c2fc0f640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c280d25d0 con 0x7f0c280b4170 2024-03-06T07:24:42.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.457+0000 7f0c2d1aa640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0c280d4760 0x7f0c280d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.457+0000 7f0c2d1aa640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0c280d4760 0x7f0c280d0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34410/0 (socket says 172.21.15.83:34410) 2024-03-06T07:24:42.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.457+0000 7f0c2d1aa640 1 -- 172.21.15.83:0/3311553146 learned_addr learned my addr 172.21.15.83:0/3311553146 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:42.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.457+0000 7f0c2d9ab640 1 --2- 172.21.15.83:0/3311553146 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0c280b4170 0x7f0c280b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.457+0000 7f0c2e1ac640 1 --2- 172.21.15.83:0/3311553146 >> v2:172.21.15.183:3300/0 conn(0x7f0c280b4dd0 0x7f0c280d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.457+0000 7f0c2d1aa640 1 -- 172.21.15.83:0/3311553146 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0c280b4170 msgr2=0x7f0c280b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:42.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.457+0000 7f0c2d1aa640 1 --2- 172.21.15.83:0/3311553146 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0c280b4170 0x7f0c280b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.457+0000 7f0c2d1aa640 1 -- 172.21.15.83:0/3311553146 >> v2:172.21.15.183:3300/0 conn(0x7f0c280b4dd0 msgr2=0x7f0c280d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:42.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.458+0000 7f0c2d1aa640 1 --2- 172.21.15.83:0/3311553146 >> v2:172.21.15.183:3300/0 conn(0x7f0c280b4dd0 0x7f0c280d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.458+0000 7f0c2d1aa640 1 -- 172.21.15.83:0/3311553146 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c280d2870 con 0x7f0c280d4760 2024-03-06T07:24:42.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.458+0000 7f0c2e1ac640 1 --2- 172.21.15.83:0/3311553146 >> v2:172.21.15.183:3300/0 conn(0x7f0c280b4dd0 0x7f0c280d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:42.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.458+0000 7f0c2d1aa640 1 --2- 172.21.15.83:0/3311553146 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c280d4760 0x7f0c280d0370 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f0c280cffb0 tx=0x7f0c1802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c7607467fd882e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:42.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.458+0000 7f0c2c9a9640 1 -- 172.21.15.83:0/3311553146 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c18035020 con 0x7f0c280d4760 2024-03-06T07:24:42.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.458+0000 7f0c2c9a9640 1 -- 172.21.15.83:0/3311553146 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c18031870 con 0x7f0c280d4760 2024-03-06T07:24:42.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.458+0000 7f0c2c9a9640 1 -- 172.21.15.83:0/3311553146 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c18031c40 con 0x7f0c280d4760 2024-03-06T07:24:42.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.459+0000 7f0c2fc0f640 1 -- 172.21.15.83:0/3311553146 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c280d4760 msgr2=0x7f0c280d0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.459+0000 7f0c2fc0f640 1 --2- 172.21.15.83:0/3311553146 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c280d4760 0x7f0c280d0370 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f0c280cffb0 tx=0x7f0c1802b910 comp rx=0 tx=0).stop 2024-03-06T07:24:42.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.459+0000 7f0c2fc0f640 1 -- 172.21.15.83:0/3311553146 shutdown_connections 2024-03-06T07:24:42.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.459+0000 7f0c2fc0f640 1 --2- 172.21.15.83:0/3311553146 >> v2:172.21.15.183:3300/0 conn(0x7f0c280b4dd0 0x7f0c280d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.459+0000 7f0c2fc0f640 1 --2- 172.21.15.83:0/3311553146 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c280d4760 0x7f0c280d0370 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.459+0000 7f0c2fc0f640 1 --2- 172.21.15.83:0/3311553146 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0c280b4170 0x7f0c280b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.459+0000 7f0c2fc0f640 1 -- 172.21.15.83:0/3311553146 >> 172.21.15.83:0/3311553146 conn(0x7f0c280d3c40 msgr2=0x7f0c280d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:42.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.459+0000 7f0c2fc0f640 1 -- 172.21.15.83:0/3311553146 shutdown_connections 2024-03-06T07:24:42.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.459+0000 7f0c2fc0f640 1 -- 172.21.15.83:0/3311553146 wait complete. 2024-03-06T07:24:42.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.460+0000 7f0c2fc0f640 1 Processor -- start 2024-03-06T07:24:42.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.460+0000 7f0c2fc0f640 1 -- start start 2024-03-06T07:24:42.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.461+0000 7f0c2fc0f640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c280b4170 0x7f0c280cad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.461+0000 7f0c2fc0f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0c280b4dd0 0x7f0c2814f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.461+0000 7f0c2fc0f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c281586a0 0x7f0c280c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.461+0000 7f0c2fc0f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c280b47d0 con 0x7f0c281586a0 2024-03-06T07:24:42.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.461+0000 7f0c2fc0f640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c280b4550 con 0x7f0c280b4170 2024-03-06T07:24:42.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.461+0000 7f0c2fc0f640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c280b4950 con 0x7f0c280b4dd0 2024-03-06T07:24:42.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.461+0000 7f0c2e1ac640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c281586a0 0x7f0c280c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.461+0000 7f0c2e1ac640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c281586a0 0x7f0c280c9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34424/0 (socket says 172.21.15.83:34424) 2024-03-06T07:24:42.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.461+0000 7f0c2e1ac640 1 -- 172.21.15.83:0/3306472346 learned_addr learned my addr 172.21.15.83:0/3306472346 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:42.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.461+0000 7f0c2d9ab640 1 --2- 172.21.15.83:0/3306472346 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c280b4170 0x7f0c280cad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.461+0000 7f0c2d1aa640 1 --2- 172.21.15.83:0/3306472346 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0c280b4dd0 0x7f0c2814f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.461+0000 7f0c2e1ac640 1 -- 172.21.15.83:0/3306472346 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0c280b4dd0 msgr2=0x7f0c2814f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.461+0000 7f0c2e1ac640 1 --2- 172.21.15.83:0/3306472346 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0c280b4dd0 0x7f0c2814f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.461+0000 7f0c2e1ac640 1 -- 172.21.15.83:0/3306472346 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c280b4170 msgr2=0x7f0c280cad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.461+0000 7f0c2e1ac640 1 --2- 172.21.15.83:0/3306472346 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c280b4170 0x7f0c280cad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.461+0000 7f0c2e1ac640 1 -- 172.21.15.83:0/3306472346 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c280c8ef0 con 0x7f0c281586a0 2024-03-06T07:24:42.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.461+0000 7f0c2d1aa640 1 --2- 172.21.15.83:0/3306472346 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0c280b4dd0 0x7f0c2814f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:42.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.461+0000 7f0c2e1ac640 1 --2- 172.21.15.83:0/3306472346 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c281586a0 0x7f0c280c9cb0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f0c2400cf20 tx=0x7f0c2400a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:42.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.462+0000 7f0c1e7fc640 1 -- 172.21.15.83:0/3306472346 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2401a070 con 0x7f0c281586a0 2024-03-06T07:24:42.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.462+0000 7f0c1e7fc640 1 -- 172.21.15.83:0/3306472346 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c2400d840 con 0x7f0c281586a0 2024-03-06T07:24:42.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.462+0000 7f0c1e7fc640 1 -- 172.21.15.83:0/3306472346 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2400dc30 con 0x7f0c281586a0 2024-03-06T07:24:42.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.462+0000 7f0c2fc0f640 1 -- 172.21.15.83:0/3306472346 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c280c91d0 con 0x7f0c281586a0 2024-03-06T07:24:42.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.462+0000 7f0c2fc0f640 1 -- 172.21.15.83:0/3306472346 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c280c9790 con 0x7f0c281586a0 2024-03-06T07:24:42.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.463+0000 7f0c2fc0f640 1 -- 172.21.15.83:0/3306472346 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c2804f330 con 0x7f0c281586a0 2024-03-06T07:24:42.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.463+0000 7f0c1e7fc640 1 -- 172.21.15.83:0/3306472346 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0c2401d070 con 0x7f0c281586a0 2024-03-06T07:24:42.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.464+0000 7f0c1e7fc640 1 --2- 172.21.15.83:0/3306472346 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0c0c06b4d0 0x7f0c0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.464+0000 7f0c2d9ab640 1 --2- 172.21.15.83:0/3306472346 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0c0c06b4d0 0x7f0c0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.464+0000 7f0c1e7fc640 1 -- 172.21.15.83:0/3306472346 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0c2408aac0 con 0x7f0c281586a0 2024-03-06T07:24:42.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.464+0000 7f0c2d9ab640 1 --2- 172.21.15.83:0/3306472346 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0c0c06b4d0 0x7f0c0c06d990 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0c20006fd0 tx=0x7f0c20008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:42.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.468+0000 7f0c1e7fc640 1 -- 172.21.15.83:0/3306472346 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0c24059940 con 0x7f0c281586a0 2024-03-06T07:24:42.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.649+0000 7f0c2fc0f640 1 -- 172.21.15.83:0/3306472346 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0c280733b0 con 0x7f0c281586a0 2024-03-06T07:24:42.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.650+0000 7f0c1e7fc640 1 -- 172.21.15.83:0/3306472346 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+16677 (secure 0 0 0) 0x7f0c24059240 con 0x7f0c281586a0 2024-03-06T07:24:42.652 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:24:42.652 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":57,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:24:21.525937+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:18.074366+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:18.831101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:42.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.653+0000 7f0c2fc0f640 1 -- 172.21.15.83:0/3306472346 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0c0c06b4d0 msgr2=0x7f0c0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.653+0000 7f0c2fc0f640 1 --2- 172.21.15.83:0/3306472346 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0c0c06b4d0 0x7f0c0c06d990 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0c20006fd0 tx=0x7f0c20008040 comp rx=0 tx=0).stop 2024-03-06T07:24:42.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.653+0000 7f0c2fc0f640 1 -- 172.21.15.83:0/3306472346 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c281586a0 msgr2=0x7f0c280c9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.653+0000 7f0c2fc0f640 1 --2- 172.21.15.83:0/3306472346 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c281586a0 0x7f0c280c9cb0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f0c2400cf20 tx=0x7f0c2400a7d0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.653+0000 7f0c2fc0f640 1 -- 172.21.15.83:0/3306472346 shutdown_connections 2024-03-06T07:24:42.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.653+0000 7f0c2fc0f640 1 --2- 172.21.15.83:0/3306472346 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0c0c06b4d0 0x7f0c0c06d990 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.653+0000 7f0c2fc0f640 1 --2- 172.21.15.83:0/3306472346 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c281586a0 0x7f0c280c9cb0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.653+0000 7f0c2fc0f640 1 --2- 172.21.15.83:0/3306472346 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0c280b4dd0 0x7f0c2814f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.653+0000 7f0c2fc0f640 1 --2- 172.21.15.83:0/3306472346 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c280b4170 0x7f0c280cad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.653+0000 7f0c2fc0f640 1 -- 172.21.15.83:0/3306472346 >> 172.21.15.83:0/3306472346 conn(0x7f0c280d3c40 msgr2=0x7f0c280b6d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:42.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.653+0000 7f0c2fc0f640 1 -- 172.21.15.83:0/3306472346 shutdown_connections 2024-03-06T07:24:42.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.653+0000 7f0c2fc0f640 1 -- 172.21.15.83:0/3306472346 wait complete. 2024-03-06T07:24:42.665 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:24:18.074366+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:24:18.831101+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:42.666 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:42.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.767+0000 7f9dec113640 1 Processor -- start 2024-03-06T07:24:42.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.767+0000 7f9dec113640 1 -- start start 2024-03-06T07:24:42.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.768+0000 7f9dec113640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9de41614d0 0x7f9de40d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.768+0000 7f9dec113640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9de40d1910 0x7f9de40d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.768+0000 7f9dec113640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9de40d2740 con 0x7f9de40d22b0 2024-03-06T07:24:42.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.768+0000 7f9dec113640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9de41618b0 con 0x7f9de41614d0 2024-03-06T07:24:42.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.768+0000 7f9dec113640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9de40cf400 con 0x7f9de40d1910 2024-03-06T07:24:42.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.768+0000 7f9de9eaf640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9de41614d0 0x7f9de40d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.768+0000 7f9de9eaf640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9de41614d0 0x7f9de40d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34436/0 (socket says 172.21.15.83:34436) 2024-03-06T07:24:42.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.768+0000 7f9de9eaf640 1 -- 172.21.15.83:0/3803537057 learned_addr learned my addr 172.21.15.83:0/3803537057 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:42.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.768+0000 7f9de96ae640 1 --2- 172.21.15.83:0/3803537057 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9de40d1910 0x7f9de40d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.768+0000 7f9de9eaf640 1 -- 172.21.15.83:0/3803537057 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9de40d1910 msgr2=0x7f9de40d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:42.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.768+0000 7f9de9eaf640 1 --2- 172.21.15.83:0/3803537057 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9de40d1910 0x7f9de40d1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.768+0000 7f9de9eaf640 1 -- 172.21.15.83:0/3803537057 >> v1:172.21.15.183:6789/0 conn(0x7f9de40d22b0 legacy=0x7f9de40d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:42.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.768+0000 7f9de9eaf640 1 -- 172.21.15.83:0/3803537057 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de40cf6a0 con 0x7f9de41614d0 2024-03-06T07:24:42.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.769+0000 7f9de9eaf640 1 --2- 172.21.15.83:0/3803537057 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9de41614d0 0x7f9de40d4860 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f9dd4009870 tx=0x7f9dd402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=999a6262e6fb82b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:42.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.769+0000 7f9de8ead640 1 -- 172.21.15.83:0/3803537057 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd4035020 con 0x7f9de41614d0 2024-03-06T07:24:42.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.769+0000 7f9de8ead640 1 -- 172.21.15.83:0/3803537057 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dd4031640 con 0x7f9de41614d0 2024-03-06T07:24:42.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.769+0000 7f9de8ead640 1 -- 172.21.15.83:0/3803537057 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd402d040 con 0x7f9de41614d0 2024-03-06T07:24:42.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.770+0000 7f9dec113640 1 -- 172.21.15.83:0/3803537057 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9de41614d0 msgr2=0x7f9de40d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.770+0000 7f9dec113640 1 --2- 172.21.15.83:0/3803537057 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9de41614d0 0x7f9de40d4860 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f9dd4009870 tx=0x7f9dd402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.770+0000 7f9dec113640 1 -- 172.21.15.83:0/3803537057 shutdown_connections 2024-03-06T07:24:42.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.770+0000 7f9dec113640 1 --2- 172.21.15.83:0/3803537057 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9de40d1910 0x7f9de40d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.770+0000 7f9dec113640 1 --2- 172.21.15.83:0/3803537057 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9de41614d0 0x7f9de40d4860 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.770+0000 7f9dec113640 1 -- 172.21.15.83:0/3803537057 >> 172.21.15.83:0/3803537057 conn(0x7f9de40d3d40 msgr2=0x7f9de40d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:42.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.770+0000 7f9dec113640 1 -- 172.21.15.83:0/3803537057 shutdown_connections 2024-03-06T07:24:42.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.770+0000 7f9dec113640 1 -- 172.21.15.83:0/3803537057 wait complete. 2024-03-06T07:24:42.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.771+0000 7f9dec113640 1 Processor -- start 2024-03-06T07:24:42.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.771+0000 7f9dec113640 1 -- start start 2024-03-06T07:24:42.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.772+0000 7f9dec113640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9de40d1910 0x7f9de40cd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.772+0000 7f9dec113640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9de40d22b0 0x7f9de4155280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.772+0000 7f9dec113640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9de41614d0 0x7f9de40b4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.772+0000 7f9dec113640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de40b0910 con 0x7f9de40d22b0 2024-03-06T07:24:42.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.772+0000 7f9dec113640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de40cfc60 con 0x7f9de41614d0 2024-03-06T07:24:42.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.772+0000 7f9dec113640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9de40b1e40 con 0x7f9de40d1910 2024-03-06T07:24:42.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.772+0000 7f9de96ae640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9de40d22b0 0x7f9de4155280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.772+0000 7f9de96ae640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9de40d22b0 0x7f9de4155280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34450/0 (socket says 172.21.15.83:34450) 2024-03-06T07:24:42.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.772+0000 7f9de96ae640 1 -- 172.21.15.83:0/2509157465 learned_addr learned my addr 172.21.15.83:0/2509157465 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:42.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.772+0000 7f9de9eaf640 1 --2- 172.21.15.83:0/2509157465 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9de40d1910 0x7f9de40cd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.772+0000 7f9dea6b0640 1 --2- 172.21.15.83:0/2509157465 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9de41614d0 0x7f9de40b4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.772+0000 7f9de96ae640 1 -- 172.21.15.83:0/2509157465 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9de40d1910 msgr2=0x7f9de40cd6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.772+0000 7f9de96ae640 1 --2- 172.21.15.83:0/2509157465 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9de40d1910 0x7f9de40cd6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.772+0000 7f9de96ae640 1 -- 172.21.15.83:0/2509157465 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9de41614d0 msgr2=0x7f9de40b4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.772+0000 7f9de96ae640 1 --2- 172.21.15.83:0/2509157465 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9de41614d0 0x7f9de40b4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.772+0000 7f9de96ae640 1 -- 172.21.15.83:0/2509157465 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de40c7d00 con 0x7f9de40d22b0 2024-03-06T07:24:42.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.772+0000 7f9de9eaf640 1 --2- 172.21.15.83:0/2509157465 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9de40d1910 0x7f9de40cd6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:42.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.772+0000 7f9de96ae640 1 --2- 172.21.15.83:0/2509157465 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9de40d22b0 0x7f9de4155280 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f9de000c7f0 tx=0x7f9de000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:42.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.772+0000 7f9dda7fc640 1 -- 172.21.15.83:0/2509157465 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de0015070 con 0x7f9de40d22b0 2024-03-06T07:24:42.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.772+0000 7f9dda7fc640 1 -- 172.21.15.83:0/2509157465 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9de0004240 con 0x7f9de40d22b0 2024-03-06T07:24:42.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.772+0000 7f9dec113640 1 -- 172.21.15.83:0/2509157465 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9de40c7fe0 con 0x7f9de40d22b0 2024-03-06T07:24:42.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.772+0000 7f9dda7fc640 1 -- 172.21.15.83:0/2509157465 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de0007620 con 0x7f9de40d22b0 2024-03-06T07:24:42.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.772+0000 7f9dec113640 1 -- 172.21.15.83:0/2509157465 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9de40c85a0 con 0x7f9de40d22b0 2024-03-06T07:24:42.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.773+0000 7f9dec113640 1 -- 172.21.15.83:0/2509157465 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9de4068ff0 con 0x7f9de40d22b0 2024-03-06T07:24:42.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.774+0000 7f9dda7fc640 1 -- 172.21.15.83:0/2509157465 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9de00079d0 con 0x7f9de40d22b0 2024-03-06T07:24:42.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.774+0000 7f9dda7fc640 1 --2- 172.21.15.83:0/2509157465 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9dc806b4d0 0x7f9dc806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.774+0000 7f9dda7fc640 1 -- 172.21.15.83:0/2509157465 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9de008a680 con 0x7f9de40d22b0 2024-03-06T07:24:42.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.774+0000 7f9de9eaf640 1 --2- 172.21.15.83:0/2509157465 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9dc806b4d0 0x7f9dc806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.775+0000 7f9de9eaf640 1 --2- 172.21.15.83:0/2509157465 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9dc806b4d0 0x7f9dc806d990 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f9dd4036000 tx=0x7f9dd403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:42.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.778+0000 7f9dda7fc640 1 -- 172.21.15.83:0/2509157465 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9de0059500 con 0x7f9de40d22b0 2024-03-06T07:24:42.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.955+0000 7f9dec113640 1 -- 172.21.15.83:0/2509157465 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9de40c7260 con 0x7f9de40d22b0 2024-03-06T07:24:42.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.956+0000 7f9dda7fc640 1 -- 172.21.15.83:0/2509157465 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v57) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9de0058e00 con 0x7f9de40d22b0 2024-03-06T07:24:42.958 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:24:42.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.957+0000 7f9dec113640 1 -- 172.21.15.83:0/2509157465 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9dc806b4d0 msgr2=0x7f9dc806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.957+0000 7f9dec113640 1 --2- 172.21.15.83:0/2509157465 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9dc806b4d0 0x7f9dc806d990 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f9dd4036000 tx=0x7f9dd403c040 comp rx=0 tx=0).stop 2024-03-06T07:24:42.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.957+0000 7f9dec113640 1 -- 172.21.15.83:0/2509157465 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9de40d22b0 msgr2=0x7f9de4155280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.957+0000 7f9dec113640 1 --2- 172.21.15.83:0/2509157465 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9de40d22b0 0x7f9de4155280 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f9de000c7f0 tx=0x7f9de000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.957+0000 7f9dec113640 1 -- 172.21.15.83:0/2509157465 shutdown_connections 2024-03-06T07:24:42.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.957+0000 7f9dec113640 1 --2- 172.21.15.83:0/2509157465 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9dc806b4d0 0x7f9dc806d990 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.957+0000 7f9dec113640 1 --2- 172.21.15.83:0/2509157465 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9de41614d0 0x7f9de40b4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.957+0000 7f9dec113640 1 --2- 172.21.15.83:0/2509157465 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9de40d22b0 0x7f9de4155280 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.957+0000 7f9dec113640 1 --2- 172.21.15.83:0/2509157465 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9de40d1910 0x7f9de40cd6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.958+0000 7f9dec113640 1 -- 172.21.15.83:0/2509157465 >> 172.21.15.83:0/2509157465 conn(0x7f9de40d3d40 msgr2=0x7f9de406c8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:42.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.958+0000 7f9dec113640 1 -- 172.21.15.83:0/2509157465 shutdown_connections 2024-03-06T07:24:42.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:42.958+0000 7f9dec113640 1 -- 172.21.15.83:0/2509157465 wait complete. 2024-03-06T07:24:42.970 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:24:43.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.076+0000 7f8a2a10f640 1 Processor -- start 2024-03-06T07:24:43.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.076+0000 7f8a2a10f640 1 -- start start 2024-03-06T07:24:43.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.076+0000 7f8a2a10f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8a240b1e40 0x7f8a24168b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.076+0000 7f8a2a10f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a240d2730 con 0x7f8a240d4860 2024-03-06T07:24:43.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.076+0000 7f8a2a10f640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a241618b0 con 0x7f8a241614d0 2024-03-06T07:24:43.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.076+0000 7f8a2a10f640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a240d4cf0 con 0x7f8a240b1e40 2024-03-06T07:24:43.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.077+0000 7f8a22ffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f8a240d4860 0x7f8a240d2020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57990/0 (socket says 172.21.15.83:57990) 2024-03-06T07:24:43.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.077+0000 7f8a22ffd640 1 -- 172.21.15.83:0/2955752710 learned_addr learned my addr 172.21.15.83:0/2955752710 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:24:43.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.077+0000 7f8a23fff640 1 --2- 172.21.15.83:0/2955752710 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8a240b1e40 0x7f8a24168b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.077+0000 7f8a227fc640 1 -- 172.21.15.83:0/2955752710 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2770639215 0 0) 0x7f8a240d2730 con 0x7f8a240d4860 2024-03-06T07:24:43.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.077+0000 7f8a227fc640 1 -- 172.21.15.83:0/2955752710 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a0c003520 con 0x7f8a240d4860 2024-03-06T07:24:43.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.077+0000 7f8a227fc640 1 -- 172.21.15.83:0/2955752710 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 520316280 0 0) 0x7f8a241618b0 con 0x7f8a241614d0 2024-03-06T07:24:43.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.077+0000 7f8a227fc640 1 -- 172.21.15.83:0/2955752710 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a240d2730 con 0x7f8a241614d0 2024-03-06T07:24:43.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.077+0000 7f8a227fc640 1 -- 172.21.15.83:0/2955752710 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4131602175 0 0) 0x7f8a0c003520 con 0x7f8a240d4860 2024-03-06T07:24:43.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.077+0000 7f8a227fc640 1 -- 172.21.15.83:0/2955752710 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8a240b1e40 msgr2=0x7f8a24168b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:43.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.077+0000 7f8a227fc640 1 --2- 172.21.15.83:0/2955752710 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8a240b1e40 0x7f8a24168b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.077+0000 7f8a227fc640 1 -- 172.21.15.83:0/2955752710 >> v1:172.21.15.183:6789/0 conn(0x7f8a241614d0 legacy=0x7f8a240d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.077+0000 7f8a227fc640 1 -- 172.21.15.83:0/2955752710 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a240cf400 con 0x7f8a240d4860 2024-03-06T07:24:43.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.077+0000 7f8a227fc640 1 -- 172.21.15.83:0/2955752710 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f8a18001670 con 0x7f8a240d4860 2024-03-06T07:24:43.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.077+0000 7f8a227fc640 1 -- 172.21.15.83:0/2955752710 >> v1:172.21.15.83:6789/0 conn(0x7f8a240d4860 legacy=0x7f8a240d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.077+0000 7f8a227fc640 1 --2- 172.21.15.83:0/2955752710 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a0c003b70 0x7f8a0c003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.077+0000 7f8a227fc640 1 --2- 172.21.15.83:0/2955752710 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8a240b1e40 0x7f8a0c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.077+0000 7f8a227fc640 1 --2- 172.21.15.83:0/2955752710 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8a0c004d40 0x7f8a0c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.077+0000 7f8a227fc640 1 -- 172.21.15.83:0/2955752710 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a240d4cf0 con 0x7f8a0c003b70 2024-03-06T07:24:43.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.077+0000 7f8a227fc640 1 -- 172.21.15.83:0/2955752710 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0c00baa0 con 0x7f8a0c004d40 2024-03-06T07:24:43.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.077+0000 7f8a227fc640 1 -- 172.21.15.83:0/2955752710 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0c00bca0 con 0x7f8a240b1e40 2024-03-06T07:24:43.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.077+0000 7f8a237fe640 1 --2- 172.21.15.83:0/2955752710 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a0c003b70 0x7f8a0c003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.078+0000 7f8a22ffd640 1 --2- 172.21.15.83:0/2955752710 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8a240b1e40 0x7f8a0c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.078+0000 7f8a23fff640 1 --2- 172.21.15.83:0/2955752710 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8a0c004d40 0x7f8a0c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.078+0000 7f8a237fe640 1 -- 172.21.15.83:0/2955752710 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8a240b1e40 msgr2=0x7f8a0c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:43.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.078+0000 7f8a237fe640 1 --2- 172.21.15.83:0/2955752710 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8a240b1e40 0x7f8a0c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.078+0000 7f8a237fe640 1 -- 172.21.15.83:0/2955752710 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8a0c004d40 msgr2=0x7f8a0c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:43.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.078+0000 7f8a237fe640 1 --2- 172.21.15.83:0/2955752710 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8a0c004d40 0x7f8a0c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.078+0000 7f8a237fe640 1 -- 172.21.15.83:0/2955752710 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a0c003520 con 0x7f8a0c003b70 2024-03-06T07:24:43.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.078+0000 7f8a237fe640 1 --2- 172.21.15.83:0/2955752710 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a0c003b70 0x7f8a0c003f90 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f8a140018a0 tx=0x7f8a0c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec195b7eb929f261 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.079+0000 7f8a227fc640 1 -- 172.21.15.83:0/2955752710 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a14037070 con 0x7f8a0c003b70 2024-03-06T07:24:43.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.079+0000 7f8a227fc640 1 -- 172.21.15.83:0/2955752710 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a140052d0 con 0x7f8a0c003b70 2024-03-06T07:24:43.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.079+0000 7f8a227fc640 1 -- 172.21.15.83:0/2955752710 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a140014f0 con 0x7f8a0c003b70 2024-03-06T07:24:43.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.079+0000 7f8a2a10f640 1 -- 172.21.15.83:0/2955752710 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a0c003b70 msgr2=0x7f8a0c003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.079+0000 7f8a2a10f640 1 --2- 172.21.15.83:0/2955752710 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a0c003b70 0x7f8a0c003f90 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f8a140018a0 tx=0x7f8a0c005830 comp rx=0 tx=0).stop 2024-03-06T07:24:43.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.079+0000 7f8a23fff640 1 -- 172.21.15.83:0/2955752710 reap_dead start 2024-03-06T07:24:43.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.079+0000 7f8a2a10f640 1 -- 172.21.15.83:0/2955752710 shutdown_connections 2024-03-06T07:24:43.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.079+0000 7f8a2a10f640 1 -- 172.21.15.83:0/2955752710 >> 172.21.15.83:0/2955752710 conn(0x7f8a240d3d40 msgr2=0x7f8a240d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:43.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.079+0000 7f8a2a10f640 1 -- 172.21.15.83:0/2955752710 shutdown_connections 2024-03-06T07:24:43.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.079+0000 7f8a2a10f640 1 -- 172.21.15.83:0/2955752710 wait complete. 2024-03-06T07:24:43.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.080+0000 7f8a2a10f640 1 Processor -- start 2024-03-06T07:24:43.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.080+0000 7f8a2a10f640 1 -- start start 2024-03-06T07:24:43.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.081+0000 7f8a2a10f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8a0c003b70 0x7f8a2415da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.081+0000 7f8a2a10f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a240ca2b0 0x7f8a2415aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.081+0000 7f8a2a10f640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8a2414ae00 0x7f8a2414ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.081+0000 7f8a2a10f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a240cfc60 con 0x7f8a240ca2b0 2024-03-06T07:24:43.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.081+0000 7f8a2a10f640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a240cf980 con 0x7f8a2414ae00 2024-03-06T07:24:43.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.081+0000 7f8a2a10f640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a240b0910 con 0x7f8a0c003b70 2024-03-06T07:24:43.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.081+0000 7f8a22ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a240ca2b0 0x7f8a2415aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.081+0000 7f8a22ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a240ca2b0 0x7f8a2415aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34478/0 (socket says 172.21.15.83:34478) 2024-03-06T07:24:43.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.081+0000 7f8a22ffd640 1 -- 172.21.15.83:0/1028978301 learned_addr learned my addr 172.21.15.83:0/1028978301 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:43.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.081+0000 7f8a237fe640 1 --2- 172.21.15.83:0/1028978301 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8a0c003b70 0x7f8a2415da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.081+0000 7f8a23fff640 1 --2- 172.21.15.83:0/1028978301 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8a2414ae00 0x7f8a2414ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.081+0000 7f8a22ffd640 1 -- 172.21.15.83:0/1028978301 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8a0c003b70 msgr2=0x7f8a2415da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.081+0000 7f8a22ffd640 1 --2- 172.21.15.83:0/1028978301 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8a0c003b70 0x7f8a2415da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.081+0000 7f8a22ffd640 1 -- 172.21.15.83:0/1028978301 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8a2414ae00 msgr2=0x7f8a2414ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.081+0000 7f8a22ffd640 1 --2- 172.21.15.83:0/1028978301 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8a2414ae00 0x7f8a2414ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.082+0000 7f8a22ffd640 1 -- 172.21.15.83:0/1028978301 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a240ca770 con 0x7f8a240ca2b0 2024-03-06T07:24:43.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.082+0000 7f8a22ffd640 1 --2- 172.21.15.83:0/1028978301 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a240ca2b0 0x7f8a2415aa80 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f8a1800e730 tx=0x7f8a1800ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.082+0000 7f8a0bfff640 1 -- 172.21.15.83:0/1028978301 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1800c660 con 0x7f8a240ca2b0 2024-03-06T07:24:43.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.082+0000 7f8a0bfff640 1 -- 172.21.15.83:0/1028978301 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a18018070 con 0x7f8a240ca2b0 2024-03-06T07:24:43.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.082+0000 7f8a2a10f640 1 -- 172.21.15.83:0/1028978301 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a240b7ab0 con 0x7f8a240ca2b0 2024-03-06T07:24:43.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.082+0000 7f8a0bfff640 1 -- 172.21.15.83:0/1028978301 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1800ca40 con 0x7f8a240ca2b0 2024-03-06T07:24:43.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.082+0000 7f8a2a10f640 1 -- 172.21.15.83:0/1028978301 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a240b8070 con 0x7f8a240ca2b0 2024-03-06T07:24:43.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.083+0000 7f8a2a10f640 1 -- 172.21.15.83:0/1028978301 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a240d24d0 con 0x7f8a240ca2b0 2024-03-06T07:24:43.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.083+0000 7f8a0bfff640 1 -- 172.21.15.83:0/1028978301 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8a1800cbe0 con 0x7f8a240ca2b0 2024-03-06T07:24:43.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.084+0000 7f8a0bfff640 1 --2- 172.21.15.83:0/1028978301 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8a1006b500 0x7f8a1006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.084+0000 7f8a0bfff640 1 -- 172.21.15.83:0/1028978301 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8a18089e30 con 0x7f8a240ca2b0 2024-03-06T07:24:43.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.084+0000 7f8a237fe640 1 --2- 172.21.15.83:0/1028978301 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8a1006b500 0x7f8a1006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.084+0000 7f8a237fe640 1 --2- 172.21.15.83:0/1028978301 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8a1006b500 0x7f8a1006d9c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f8a14034000 tx=0x7f8a14031000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.088+0000 7f8a0bfff640 1 -- 172.21.15.83:0/1028978301 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8a180586f0 con 0x7f8a240ca2b0 2024-03-06T07:24:43.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.264+0000 7f8a2a10f640 1 -- 172.21.15.83:0/1028978301 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8a24068ff0 con 0x7f8a240ca2b0 2024-03-06T07:24:43.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.264+0000 7f8a0bfff640 1 -- 172.21.15.83:0/1028978301 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v57) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8a1805c790 con 0x7f8a240ca2b0 2024-03-06T07:24:43.266 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-03-06T07:24:43.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.266+0000 7f8a2a10f640 1 -- 172.21.15.83:0/1028978301 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8a1006b500 msgr2=0x7f8a1006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.266+0000 7f8a2a10f640 1 --2- 172.21.15.83:0/1028978301 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8a1006b500 0x7f8a1006d9c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f8a14034000 tx=0x7f8a14031000 comp rx=0 tx=0).stop 2024-03-06T07:24:43.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.266+0000 7f8a2a10f640 1 -- 172.21.15.83:0/1028978301 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a240ca2b0 msgr2=0x7f8a2415aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.266+0000 7f8a2a10f640 1 --2- 172.21.15.83:0/1028978301 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a240ca2b0 0x7f8a2415aa80 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f8a1800e730 tx=0x7f8a1800ebf0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.266+0000 7f8a2a10f640 1 -- 172.21.15.83:0/1028978301 shutdown_connections 2024-03-06T07:24:43.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.266+0000 7f8a2a10f640 1 --2- 172.21.15.83:0/1028978301 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8a1006b500 0x7f8a1006d9c0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.266+0000 7f8a2a10f640 1 --2- 172.21.15.83:0/1028978301 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8a2414ae00 0x7f8a2414ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.266+0000 7f8a2a10f640 1 --2- 172.21.15.83:0/1028978301 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a240ca2b0 0x7f8a2415aa80 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.266+0000 7f8a2a10f640 1 --2- 172.21.15.83:0/1028978301 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8a0c003b70 0x7f8a2415da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.266+0000 7f8a2a10f640 1 -- 172.21.15.83:0/1028978301 >> 172.21.15.83:0/1028978301 conn(0x7f8a240d3d40 msgr2=0x7f8a2406d020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:43.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.266+0000 7f8a2a10f640 1 -- 172.21.15.83:0/1028978301 shutdown_connections 2024-03-06T07:24:43.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.266+0000 7f8a2a10f640 1 -- 172.21.15.83:0/1028978301 wait complete. 2024-03-06T07:24:43.279 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:24:43.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.389+0000 7f3645822640 1 Processor -- start 2024-03-06T07:24:43.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.389+0000 7f3645822640 1 -- start start 2024-03-06T07:24:43.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.390+0000 7f3645822640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f36400b4970 0x7f364006d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.390+0000 7f3645822640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f364006d5c0 0x7f36400cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.390+0000 7f3645822640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f364015ff90 con 0x7f364015bb40 2024-03-06T07:24:43.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.390+0000 7f3645822640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36400b4dd0 con 0x7f364006d5c0 2024-03-06T07:24:43.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.390+0000 7f3645822640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f364006c910 con 0x7f36400b4970 2024-03-06T07:24:43.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.390+0000 7f363f7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f364006d5c0 0x7f36400cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.390+0000 7f363f7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f364006d5c0 0x7f36400cfa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34486/0 (socket says 172.21.15.83:34486) 2024-03-06T07:24:43.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.390+0000 7f363effd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f364015bb40 0x7f364015f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57994/0 (socket says 172.21.15.83:57994) 2024-03-06T07:24:43.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.390+0000 7f363f7fe640 1 -- 172.21.15.83:0/2066865481 learned_addr learned my addr 172.21.15.83:0/2066865481 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:43.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.391+0000 7f363e7fc640 1 --2- 172.21.15.83:0/2066865481 >> v2:172.21.15.183:3300/0 conn(0x7f36400b4970 0x7f364006d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.391+0000 7f363f7fe640 1 -- 172.21.15.83:0/2066865481 >> v2:172.21.15.183:3300/0 conn(0x7f36400b4970 msgr2=0x7f364006d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:43.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.391+0000 7f363f7fe640 1 --2- 172.21.15.83:0/2066865481 >> v2:172.21.15.183:3300/0 conn(0x7f36400b4970 0x7f364006d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.391+0000 7f363f7fe640 1 -- 172.21.15.83:0/2066865481 >> v1:172.21.15.83:6789/0 conn(0x7f364015bb40 legacy=0x7f364015f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.391+0000 7f363f7fe640 1 -- 172.21.15.83:0/2066865481 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3640160270 con 0x7f364006d5c0 2024-03-06T07:24:43.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.391+0000 7f363dffb640 1 -- 172.21.15.83:0/2066865481 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 426455443 0 0) 0x7f364015ff90 con 0x7f364015bb40 2024-03-06T07:24:43.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.391+0000 7f363f7fe640 1 --2- 172.21.15.83:0/2066865481 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f364006d5c0 0x7f36400cfa90 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f364006bfe0 tx=0x7f363002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eec267197ef7a0ea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.392+0000 7f363dffb640 1 -- 172.21.15.83:0/2066865481 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3630034070 con 0x7f364006d5c0 2024-03-06T07:24:43.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.392+0000 7f363dffb640 1 -- 172.21.15.83:0/2066865481 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3630030860 con 0x7f364006d5c0 2024-03-06T07:24:43.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.392+0000 7f363dffb640 1 -- 172.21.15.83:0/2066865481 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3630030c30 con 0x7f364006d5c0 2024-03-06T07:24:43.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.392+0000 7f3645822640 1 -- 172.21.15.83:0/2066865481 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f364006d5c0 msgr2=0x7f36400cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.392+0000 7f3645822640 1 --2- 172.21.15.83:0/2066865481 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f364006d5c0 0x7f36400cfa90 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f364006bfe0 tx=0x7f363002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.392+0000 7f3645822640 1 -- 172.21.15.83:0/2066865481 shutdown_connections 2024-03-06T07:24:43.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.392+0000 7f3645822640 1 --2- 172.21.15.83:0/2066865481 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f364006d5c0 0x7f36400cfa90 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.392+0000 7f3645822640 1 --2- 172.21.15.83:0/2066865481 >> v2:172.21.15.183:3300/0 conn(0x7f36400b4970 0x7f364006d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.392+0000 7f3645822640 1 -- 172.21.15.83:0/2066865481 >> 172.21.15.83:0/2066865481 conn(0x7f364006b190 msgr2=0x7f364006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:43.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.392+0000 7f3645822640 1 -- 172.21.15.83:0/2066865481 shutdown_connections 2024-03-06T07:24:43.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.392+0000 7f3645822640 1 -- 172.21.15.83:0/2066865481 wait complete. 2024-03-06T07:24:43.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.393+0000 7f3645822640 1 Processor -- start 2024-03-06T07:24:43.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.394+0000 7f3645822640 1 -- start start 2024-03-06T07:24:43.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.394+0000 7f3645822640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f364006d5c0 0x7f364014d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.394+0000 7f3645822640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f36400b4970 0x7f36401f2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.394+0000 7f3645822640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f364015bb40 0x7f36400b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.394+0000 7f3645822640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364015bf20 con 0x7f36400b4970 2024-03-06T07:24:43.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.394+0000 7f3645822640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364006cb60 con 0x7f364006d5c0 2024-03-06T07:24:43.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.394+0000 7f3645822640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f364015c0a0 con 0x7f364015bb40 2024-03-06T07:24:43.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.394+0000 7f363e7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f36400b4970 0x7f36401f2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.394+0000 7f363e7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f36400b4970 0x7f36401f2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34488/0 (socket says 172.21.15.83:34488) 2024-03-06T07:24:43.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.394+0000 7f363e7fc640 1 -- 172.21.15.83:0/415724240 learned_addr learned my addr 172.21.15.83:0/415724240 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:43.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.394+0000 7f363effd640 1 --2- 172.21.15.83:0/415724240 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f364006d5c0 0x7f364014d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.395+0000 7f363f7fe640 1 --2- 172.21.15.83:0/415724240 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f364015bb40 0x7f36400b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.395+0000 7f363e7fc640 1 -- 172.21.15.83:0/415724240 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f364015bb40 msgr2=0x7f36400b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.395+0000 7f363e7fc640 1 --2- 172.21.15.83:0/415724240 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f364015bb40 0x7f36400b12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.395+0000 7f363e7fc640 1 -- 172.21.15.83:0/415724240 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f364006d5c0 msgr2=0x7f364014d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.395+0000 7f363e7fc640 1 --2- 172.21.15.83:0/415724240 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f364006d5c0 0x7f364014d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.395+0000 7f363e7fc640 1 -- 172.21.15.83:0/415724240 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f364015c380 con 0x7f36400b4970 2024-03-06T07:24:43.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.395+0000 7f363f7fe640 1 --2- 172.21.15.83:0/415724240 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f364015bb40 0x7f36400b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:43.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.395+0000 7f363e7fc640 1 --2- 172.21.15.83:0/415724240 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f36400b4970 0x7f36401f2960 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f362800d800 tx=0x7f362800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.395+0000 7f3637fff640 1 -- 172.21.15.83:0/415724240 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362800b6c0 con 0x7f36400b4970 2024-03-06T07:24:43.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.395+0000 7f3637fff640 1 -- 172.21.15.83:0/415724240 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3628019070 con 0x7f36400b4970 2024-03-06T07:24:43.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.395+0000 7f3645822640 1 -- 172.21.15.83:0/415724240 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f364015c680 con 0x7f36400b4970 2024-03-06T07:24:43.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.395+0000 7f3637fff640 1 -- 172.21.15.83:0/415724240 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362800ba80 con 0x7f36400b4970 2024-03-06T07:24:43.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.395+0000 7f3645822640 1 -- 172.21.15.83:0/415724240 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f364015cec0 con 0x7f36400b4970 2024-03-06T07:24:43.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.396+0000 7f3645822640 1 -- 172.21.15.83:0/415724240 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36400732b0 con 0x7f36400b4970 2024-03-06T07:24:43.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.397+0000 7f3637fff640 1 -- 172.21.15.83:0/415724240 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f362800bc20 con 0x7f36400b4970 2024-03-06T07:24:43.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.397+0000 7f3637fff640 1 --2- 172.21.15.83:0/415724240 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f362006b4d0 0x7f362006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.398+0000 7f3637fff640 1 -- 172.21.15.83:0/415724240 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f362808a580 con 0x7f36400b4970 2024-03-06T07:24:43.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.398+0000 7f363effd640 1 --2- 172.21.15.83:0/415724240 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f362006b4d0 0x7f362006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.398+0000 7f363effd640 1 --2- 172.21.15.83:0/415724240 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f362006b4d0 0x7f362006d990 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f36380012e0 tx=0x7f3638001630 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.402+0000 7f3637fff640 1 -- 172.21.15.83:0/415724240 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f364015c680 con 0x7f36400b4970 2024-03-06T07:24:43.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.583+0000 7f3645822640 1 -- 172.21.15.83:0/415724240 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3640068ff0 con 0x7f36400b4970 2024-03-06T07:24:43.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.583+0000 7f3637fff640 1 -- 172.21.15.83:0/415724240 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v57) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3628059490 con 0x7f36400b4970 2024-03-06T07:24:43.585 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 64 2024-03-06T07:24:43.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.585+0000 7f3645822640 1 -- 172.21.15.83:0/415724240 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f362006b4d0 msgr2=0x7f362006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.585+0000 7f3645822640 1 --2- 172.21.15.83:0/415724240 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f362006b4d0 0x7f362006d990 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f36380012e0 tx=0x7f3638001630 comp rx=0 tx=0).stop 2024-03-06T07:24:43.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.585+0000 7f3645822640 1 -- 172.21.15.83:0/415724240 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f36400b4970 msgr2=0x7f36401f2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.585+0000 7f3645822640 1 --2- 172.21.15.83:0/415724240 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f36400b4970 0x7f36401f2960 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f362800d800 tx=0x7f362800dcc0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.585+0000 7f3645822640 1 -- 172.21.15.83:0/415724240 shutdown_connections 2024-03-06T07:24:43.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.585+0000 7f3645822640 1 --2- 172.21.15.83:0/415724240 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f362006b4d0 0x7f362006d990 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.585+0000 7f3645822640 1 --2- 172.21.15.83:0/415724240 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f364015bb40 0x7f36400b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.585+0000 7f3645822640 1 --2- 172.21.15.83:0/415724240 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f36400b4970 0x7f36401f2960 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.585+0000 7f3645822640 1 --2- 172.21.15.83:0/415724240 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f364006d5c0 0x7f364014d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.585+0000 7f3645822640 1 -- 172.21.15.83:0/415724240 >> 172.21.15.83:0/415724240 conn(0x7f364006b190 msgr2=0x7f364015a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:43.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.585+0000 7f3645822640 1 -- 172.21.15.83:0/415724240 shutdown_connections 2024-03-06T07:24:43.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.585+0000 7f3645822640 1 -- 172.21.15.83:0/415724240 wait complete. 2024-03-06T07:24:43.597 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:24:43.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.697+0000 7f0b2a29a640 1 Processor -- start 2024-03-06T07:24:43.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.697+0000 7f0b2a29a640 1 -- start start 2024-03-06T07:24:43.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.698+0000 7f0b2a29a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b240b4460 0x7f0b240d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.698+0000 7f0b2a29a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f0b24162970 0x7f0b240b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.699+0000 7f0b2a29a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b240d22c0 con 0x7f0b24161cf0 2024-03-06T07:24:43.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.699+0000 7f0b2a29a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b24161b00 con 0x7f0b24162970 2024-03-06T07:24:43.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.699+0000 7f0b2a29a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b240cecc0 con 0x7f0b240b4460 2024-03-06T07:24:43.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.699+0000 7f0b237fe640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f0b24162970 0x7f0b240b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.699+0000 7f0b23fff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b240b4460 0x7f0b240d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.699+0000 7f0b23fff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b240b4460 0x7f0b240d4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:36832/0 (socket says 172.21.15.83:36832) 2024-03-06T07:24:43.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.699+0000 7f0b23fff640 1 -- 172.21.15.83:0/2937154953 learned_addr learned my addr 172.21.15.83:0/2937154953 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:43.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.699+0000 7f0b237fe640 1 -- 172.21.15.83:0/2937154953 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b240b4460 msgr2=0x7f0b240d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:43.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.699+0000 7f0b237fe640 1 --2- 172.21.15.83:0/2937154953 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b240b4460 0x7f0b240d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.699+0000 7f0b237fe640 1 -- 172.21.15.83:0/2937154953 >> v1:172.21.15.183:6789/0 conn(0x7f0b24161cf0 legacy=0x7f0b240cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.699+0000 7f0b237fe640 1 -- 172.21.15.83:0/2937154953 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b240d25a0 con 0x7f0b24162970 2024-03-06T07:24:43.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.700+0000 7f0b237fe640 1 --2- 172.21.15.83:0/2937154953 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b24162970 0x7f0b240b3580 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0b180097a0 tx=0x7f0b1802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d95b7462a101bd10 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.700+0000 7f0b22ffd640 1 -- 172.21.15.83:0/2937154953 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b18035020 con 0x7f0b24162970 2024-03-06T07:24:43.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.700+0000 7f0b22ffd640 1 -- 172.21.15.83:0/2937154953 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b18031830 con 0x7f0b24162970 2024-03-06T07:24:43.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.700+0000 7f0b22ffd640 1 -- 172.21.15.83:0/2937154953 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b18031c00 con 0x7f0b24162970 2024-03-06T07:24:43.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.701+0000 7f0b2a29a640 1 -- 172.21.15.83:0/2937154953 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b24162970 msgr2=0x7f0b240b3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.701+0000 7f0b2a29a640 1 --2- 172.21.15.83:0/2937154953 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b24162970 0x7f0b240b3580 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0b180097a0 tx=0x7f0b1802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.701+0000 7f0b2a29a640 1 -- 172.21.15.83:0/2937154953 shutdown_connections 2024-03-06T07:24:43.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.701+0000 7f0b2a29a640 1 --2- 172.21.15.83:0/2937154953 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b24162970 0x7f0b240b3580 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.701+0000 7f0b2a29a640 1 --2- 172.21.15.83:0/2937154953 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b240b4460 0x7f0b240d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.701+0000 7f0b2a29a640 1 -- 172.21.15.83:0/2937154953 >> 172.21.15.83:0/2937154953 conn(0x7f0b240d3ce0 msgr2=0x7f0b240d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:43.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.701+0000 7f0b2a29a640 1 -- 172.21.15.83:0/2937154953 shutdown_connections 2024-03-06T07:24:43.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.701+0000 7f0b2a29a640 1 -- 172.21.15.83:0/2937154953 wait complete. 2024-03-06T07:24:43.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.703+0000 7f0b2a29a640 1 Processor -- start 2024-03-06T07:24:43.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.703+0000 7f0b2a29a640 1 -- start start 2024-03-06T07:24:43.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.704+0000 7f0b2a29a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b240b4460 0x7f0b240cc030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.704+0000 7f0b2a29a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b24161cf0 0x7f0b241574d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.704+0000 7f0b2a29a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b24162970 0x7f0b2415c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.704+0000 7f0b2a29a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b240ce950 con 0x7f0b240b4460 2024-03-06T07:24:43.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.704+0000 7f0b2a29a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b240d2b60 con 0x7f0b24161cf0 2024-03-06T07:24:43.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.704+0000 7f0b2a29a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b240b3ac0 con 0x7f0b24162970 2024-03-06T07:24:43.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.704+0000 7f0b23fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b240b4460 0x7f0b240cc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.704+0000 7f0b23fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b240b4460 0x7f0b240cc030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34504/0 (socket says 172.21.15.83:34504) 2024-03-06T07:24:43.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.704+0000 7f0b23fff640 1 -- 172.21.15.83:0/2763047242 learned_addr learned my addr 172.21.15.83:0/2763047242 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:43.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.705+0000 7f0b237fe640 1 --2- 172.21.15.83:0/2763047242 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b24161cf0 0x7f0b241574d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.705+0000 7f0b28837640 1 --2- 172.21.15.83:0/2763047242 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b24162970 0x7f0b2415c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.705+0000 7f0b23fff640 1 -- 172.21.15.83:0/2763047242 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b24162970 msgr2=0x7f0b2415c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.705+0000 7f0b23fff640 1 --2- 172.21.15.83:0/2763047242 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b24162970 0x7f0b2415c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.705+0000 7f0b23fff640 1 -- 172.21.15.83:0/2763047242 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b24161cf0 msgr2=0x7f0b241574d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.705+0000 7f0b23fff640 1 --2- 172.21.15.83:0/2763047242 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b24161cf0 0x7f0b241574d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.705+0000 7f0b23fff640 1 -- 172.21.15.83:0/2763047242 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b2415b640 con 0x7f0b240b4460 2024-03-06T07:24:43.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.705+0000 7f0b23fff640 1 --2- 172.21.15.83:0/2763047242 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b240b4460 0x7f0b240cc030 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f0b1400e840 tx=0x7f0b1400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.705+0000 7f0b20ff9640 1 -- 172.21.15.83:0/2763047242 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b14017070 con 0x7f0b240b4460 2024-03-06T07:24:43.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.705+0000 7f0b20ff9640 1 -- 172.21.15.83:0/2763047242 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b1400c8e0 con 0x7f0b240b4460 2024-03-06T07:24:43.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.705+0000 7f0b2a29a640 1 -- 172.21.15.83:0/2763047242 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b2415b920 con 0x7f0b240b4460 2024-03-06T07:24:43.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.705+0000 7f0b20ff9640 1 -- 172.21.15.83:0/2763047242 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1400ccd0 con 0x7f0b240b4460 2024-03-06T07:24:43.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.705+0000 7f0b2a29a640 1 -- 172.21.15.83:0/2763047242 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b2415bee0 con 0x7f0b240b4460 2024-03-06T07:24:43.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.706+0000 7f0b2a29a640 1 -- 172.21.15.83:0/2763047242 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b2404f330 con 0x7f0b240b4460 2024-03-06T07:24:43.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.707+0000 7f0b20ff9640 1 -- 172.21.15.83:0/2763047242 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0b1401a070 con 0x7f0b240b4460 2024-03-06T07:24:43.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.707+0000 7f0b20ff9640 1 --2- 172.21.15.83:0/2763047242 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0b1006b4d0 0x7f0b1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.707+0000 7f0b20ff9640 1 -- 172.21.15.83:0/2763047242 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0b1408a3f0 con 0x7f0b240b4460 2024-03-06T07:24:43.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.707+0000 7f0b237fe640 1 --2- 172.21.15.83:0/2763047242 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0b1006b4d0 0x7f0b1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.708+0000 7f0b237fe640 1 --2- 172.21.15.83:0/2763047242 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0b1006b4d0 0x7f0b1006d990 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0b18036000 tx=0x7f0b18008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.711+0000 7f0b20ff9640 1 -- 172.21.15.83:0/2763047242 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0b14058c20 con 0x7f0b240b4460 2024-03-06T07:24:43.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.898+0000 7f0b2a29a640 1 -- 172.21.15.83:0/2763047242 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f0b2406b180 con 0x7f0b240b4460 2024-03-06T07:24:43.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.899+0000 7f0b20ff9640 1 -- 172.21.15.83:0/2763047242 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v57) v1 ==== 80+0+0 (secure 0 0 0) 0x7f0b1405ccc0 con 0x7f0b240b4460 2024-03-06T07:24:43.901 INFO:teuthology.orchestra.run.smithi083.stderr:listed 0 entries 2024-03-06T07:24:43.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.901+0000 7f0b2a29a640 1 -- 172.21.15.83:0/2763047242 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0b1006b4d0 msgr2=0x7f0b1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.901+0000 7f0b2a29a640 1 --2- 172.21.15.83:0/2763047242 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0b1006b4d0 0x7f0b1006d990 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0b18036000 tx=0x7f0b18008030 comp rx=0 tx=0).stop 2024-03-06T07:24:43.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.901+0000 7f0b2a29a640 1 -- 172.21.15.83:0/2763047242 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b240b4460 msgr2=0x7f0b240cc030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.901+0000 7f0b2a29a640 1 --2- 172.21.15.83:0/2763047242 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b240b4460 0x7f0b240cc030 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f0b1400e840 tx=0x7f0b1400ed00 comp rx=0 tx=0).stop 2024-03-06T07:24:43.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.901+0000 7f0b2a29a640 1 -- 172.21.15.83:0/2763047242 shutdown_connections 2024-03-06T07:24:43.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.901+0000 7f0b2a29a640 1 --2- 172.21.15.83:0/2763047242 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0b1006b4d0 0x7f0b1006d990 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.901+0000 7f0b2a29a640 1 --2- 172.21.15.83:0/2763047242 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b24162970 0x7f0b2415c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.901+0000 7f0b2a29a640 1 --2- 172.21.15.83:0/2763047242 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b24161cf0 0x7f0b241574d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.901+0000 7f0b2a29a640 1 --2- 172.21.15.83:0/2763047242 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b240b4460 0x7f0b240cc030 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.901+0000 7f0b2a29a640 1 -- 172.21.15.83:0/2763047242 >> 172.21.15.83:0/2763047242 conn(0x7f0b240d3ce0 msgr2=0x7f0b2406cd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:43.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.901+0000 7f0b2a29a640 1 -- 172.21.15.83:0/2763047242 shutdown_connections 2024-03-06T07:24:43.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:43.901+0000 7f0b2a29a640 1 -- 172.21.15.83:0/2763047242 wait complete. 2024-03-06T07:24:43.913 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:24:43.913 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:43.929 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:44.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.069+0000 7fa1a9e39640 1 Processor -- start 2024-03-06T07:24:44.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.069+0000 7fa1a9e39640 1 -- start start 2024-03-06T07:24:44.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.069+0000 7fa1a9e39640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa1a415dfe0 0x7fa1a40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.070+0000 7fa1a9e39640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fa1a40cf830 0x7fa1a40d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.070+0000 7fa1a9e39640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1a40b3430 con 0x7fa1a40d27f0 2024-03-06T07:24:44.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.070+0000 7fa1a9e39640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1a40b31e0 con 0x7fa1a415dfe0 2024-03-06T07:24:44.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.070+0000 7fa1a9e39640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1a40b3d60 con 0x7fa1a40cf830 2024-03-06T07:24:44.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.070+0000 7fa1a37fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa1a415dfe0 0x7fa1a40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.070+0000 7fa1a37fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa1a415dfe0 0x7fa1a40cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34508/0 (socket says 172.21.15.83:34508) 2024-03-06T07:24:44.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.070+0000 7fa1a37fe640 1 -- 172.21.15.83:0/2883082866 learned_addr learned my addr 172.21.15.83:0/2883082866 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:44.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.070+0000 7fa19bfff640 1 --2- 172.21.15.83:0/2883082866 >> v2:172.21.15.183:3300/0 conn(0x7fa1a40cf830 0x7fa1a40d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.070+0000 7fa1a37fe640 1 -- 172.21.15.83:0/2883082866 >> v2:172.21.15.183:3300/0 conn(0x7fa1a40cf830 msgr2=0x7fa1a40d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:44.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.070+0000 7fa1a37fe640 1 --2- 172.21.15.83:0/2883082866 >> v2:172.21.15.183:3300/0 conn(0x7fa1a40cf830 0x7fa1a40d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.070+0000 7fa1a37fe640 1 -- 172.21.15.83:0/2883082866 >> v1:172.21.15.183:6789/0 conn(0x7fa1a40d27f0 legacy=0x7fa1a40d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:44.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.070+0000 7fa1a37fe640 1 -- 172.21.15.83:0/2883082866 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a40b3710 con 0x7fa1a415dfe0 2024-03-06T07:24:44.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.071+0000 7fa1a37fe640 1 --2- 172.21.15.83:0/2883082866 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1a415dfe0 0x7fa1a40cf2f0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fa194009870 tx=0x7fa19402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=965bedb4013837ce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.071+0000 7fa1a2ffd640 1 -- 172.21.15.83:0/2883082866 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa194034070 con 0x7fa1a415dfe0 2024-03-06T07:24:44.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.071+0000 7fa1a2ffd640 1 -- 172.21.15.83:0/2883082866 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa194030860 con 0x7fa1a415dfe0 2024-03-06T07:24:44.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.071+0000 7fa1a2ffd640 1 -- 172.21.15.83:0/2883082866 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa194030c30 con 0x7fa1a415dfe0 2024-03-06T07:24:44.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.071+0000 7fa1a9e39640 1 -- 172.21.15.83:0/2883082866 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1a415dfe0 msgr2=0x7fa1a40cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.071+0000 7fa1a9e39640 1 --2- 172.21.15.83:0/2883082866 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1a415dfe0 0x7fa1a40cf2f0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fa194009870 tx=0x7fa19402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.072+0000 7fa1a9e39640 1 -- 172.21.15.83:0/2883082866 shutdown_connections 2024-03-06T07:24:44.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.072+0000 7fa1a9e39640 1 --2- 172.21.15.83:0/2883082866 >> v2:172.21.15.183:3300/0 conn(0x7fa1a40cf830 0x7fa1a40d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.072+0000 7fa1a9e39640 1 --2- 172.21.15.83:0/2883082866 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1a415dfe0 0x7fa1a40cf2f0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.072+0000 7fa1a9e39640 1 -- 172.21.15.83:0/2883082866 >> 172.21.15.83:0/2883082866 conn(0x7fa1a40d3cd0 msgr2=0x7fa1a40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:44.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.072+0000 7fa1a9e39640 1 -- 172.21.15.83:0/2883082866 shutdown_connections 2024-03-06T07:24:44.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.072+0000 7fa1a9e39640 1 -- 172.21.15.83:0/2883082866 wait complete. 2024-03-06T07:24:44.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.073+0000 7fa1a9e39640 1 Processor -- start 2024-03-06T07:24:44.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.073+0000 7fa1a9e39640 1 -- start start 2024-03-06T07:24:44.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.073+0000 7fa1a9e39640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1a40cf830 0x7fa1a415c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.073+0000 7fa1a9e39640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa1a40d27f0 0x7fa1a40cd310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.073+0000 7fa1a9e39640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa1a415dfe0 0x7fa1a40cd850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.073+0000 7fa1a9e39640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a40d1df0 con 0x7fa1a40cf830 2024-03-06T07:24:44.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.073+0000 7fa1a9e39640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a40d1c70 con 0x7fa1a415dfe0 2024-03-06T07:24:44.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.073+0000 7fa1a9e39640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a415e3c0 con 0x7fa1a40d27f0 2024-03-06T07:24:44.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.073+0000 7fa1a37fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1a40cf830 0x7fa1a415c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.073+0000 7fa1a37fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1a40cf830 0x7fa1a415c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34516/0 (socket says 172.21.15.83:34516) 2024-03-06T07:24:44.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.074+0000 7fa1a37fe640 1 -- 172.21.15.83:0/1266312177 learned_addr learned my addr 172.21.15.83:0/1266312177 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:44.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.074+0000 7fa19bfff640 1 --2- 172.21.15.83:0/1266312177 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa1a40d27f0 0x7fa1a40cd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.074+0000 7fa1a3fff640 1 --2- 172.21.15.83:0/1266312177 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa1a415dfe0 0x7fa1a40cd850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.074+0000 7fa1a37fe640 1 -- 172.21.15.83:0/1266312177 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa1a40d27f0 msgr2=0x7fa1a40cd310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.074+0000 7fa1a37fe640 1 --2- 172.21.15.83:0/1266312177 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa1a40d27f0 0x7fa1a40cd310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.074+0000 7fa1a37fe640 1 -- 172.21.15.83:0/1266312177 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa1a415dfe0 msgr2=0x7fa1a40cd850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.074+0000 7fa1a37fe640 1 --2- 172.21.15.83:0/1266312177 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa1a415dfe0 0x7fa1a40cd850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.074+0000 7fa1a37fe640 1 -- 172.21.15.83:0/1266312177 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a415fa70 con 0x7fa1a40cf830 2024-03-06T07:24:44.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.074+0000 7fa1a3fff640 1 --2- 172.21.15.83:0/1266312177 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa1a415dfe0 0x7fa1a40cd850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:44.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.074+0000 7fa1a37fe640 1 --2- 172.21.15.83:0/1266312177 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1a40cf830 0x7fa1a415c6b0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fa194000c00 tx=0x7fa194004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.074+0000 7fa1a0ff9640 1 -- 172.21.15.83:0/1266312177 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa194048070 con 0x7fa1a40cf830 2024-03-06T07:24:44.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.074+0000 7fa1a0ff9640 1 -- 172.21.15.83:0/1266312177 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa194030660 con 0x7fa1a40cf830 2024-03-06T07:24:44.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.074+0000 7fa1a0ff9640 1 -- 172.21.15.83:0/1266312177 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa194034040 con 0x7fa1a40cf830 2024-03-06T07:24:44.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.074+0000 7fa1a9e39640 1 -- 172.21.15.83:0/1266312177 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1a415fd50 con 0x7fa1a40cf830 2024-03-06T07:24:44.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.074+0000 7fa1a9e39640 1 -- 172.21.15.83:0/1266312177 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1a4160310 con 0x7fa1a40cf830 2024-03-06T07:24:44.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.075+0000 7fa1a9e39640 1 -- 172.21.15.83:0/1266312177 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1a404f330 con 0x7fa1a40cf830 2024-03-06T07:24:44.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.076+0000 7fa1a0ff9640 1 -- 172.21.15.83:0/1266312177 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa194037070 con 0x7fa1a40cf830 2024-03-06T07:24:44.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.076+0000 7fa1a0ff9640 1 --2- 172.21.15.83:0/1266312177 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa18406b4d0 0x7fa18406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.076+0000 7fa19bfff640 1 --2- 172.21.15.83:0/1266312177 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa18406b4d0 0x7fa18406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.076+0000 7fa1a0ff9640 1 -- 172.21.15.83:0/1266312177 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa1940abf20 con 0x7fa1a40cf830 2024-03-06T07:24:44.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.077+0000 7fa19bfff640 1 --2- 172.21.15.83:0/1266312177 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa18406b4d0 0x7fa18406d990 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fa1a40cac10 tx=0x7fa18c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.081+0000 7fa1a0ff9640 1 -- 172.21.15.83:0/1266312177 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa19407a900 con 0x7fa1a40cf830 2024-03-06T07:24:44.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.258+0000 7fa1a9e39640 1 -- 172.21.15.83:0/1266312177 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1a40b1010 con 0x7fa1a40cf830 2024-03-06T07:24:44.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.259+0000 7fa1a0ff9640 1 -- 172.21.15.83:0/1266312177 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+16677 (secure 0 0 0) 0x7fa19407e9a0 con 0x7fa1a40cf830 2024-03-06T07:24:44.261 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:24:44.261 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":57,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:24:21.525937+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:18.074366+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:18.831101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:44.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.261+0000 7fa1a9e39640 1 -- 172.21.15.83:0/1266312177 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa18406b4d0 msgr2=0x7fa18406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.261+0000 7fa1a9e39640 1 --2- 172.21.15.83:0/1266312177 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa18406b4d0 0x7fa18406d990 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fa1a40cac10 tx=0x7fa18c008040 comp rx=0 tx=0).stop 2024-03-06T07:24:44.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.261+0000 7fa1a9e39640 1 -- 172.21.15.83:0/1266312177 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1a40cf830 msgr2=0x7fa1a415c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.261+0000 7fa1a9e39640 1 --2- 172.21.15.83:0/1266312177 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1a40cf830 0x7fa1a415c6b0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fa194000c00 tx=0x7fa194004730 comp rx=0 tx=0).stop 2024-03-06T07:24:44.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.261+0000 7fa1a9e39640 1 -- 172.21.15.83:0/1266312177 shutdown_connections 2024-03-06T07:24:44.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.261+0000 7fa1a9e39640 1 --2- 172.21.15.83:0/1266312177 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa18406b4d0 0x7fa18406d990 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.261+0000 7fa1a9e39640 1 --2- 172.21.15.83:0/1266312177 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa1a415dfe0 0x7fa1a40cd850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.261+0000 7fa1a9e39640 1 --2- 172.21.15.83:0/1266312177 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa1a40d27f0 0x7fa1a40cd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.261+0000 7fa1a9e39640 1 --2- 172.21.15.83:0/1266312177 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1a40cf830 0x7fa1a415c6b0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.261+0000 7fa1a9e39640 1 -- 172.21.15.83:0/1266312177 >> 172.21.15.83:0/1266312177 conn(0x7fa1a40d3cd0 msgr2=0x7fa1a406c860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:44.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.261+0000 7fa1a9e39640 1 -- 172.21.15.83:0/1266312177 shutdown_connections 2024-03-06T07:24:44.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.261+0000 7fa1a9e39640 1 -- 172.21.15.83:0/1266312177 wait complete. 2024-03-06T07:24:44.274 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:24:18.074366+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:24:18.831101+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:44.274 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:44.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.376+0000 7f8b0f26b640 1 Processor -- start 2024-03-06T07:24:44.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.376+0000 7f8b0f26b640 1 -- start start 2024-03-06T07:24:44.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.377+0000 7f8b0f26b640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f8b081614d0 0x7f8b080d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.377+0000 7f8b0f26b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f8b080d24b0 0x7f8b080b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.377+0000 7f8b0f26b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b081618b0 con 0x7f8b080d1910 2024-03-06T07:24:44.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.377+0000 7f8b0f26b640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b080cf400 con 0x7f8b080d24b0 2024-03-06T07:24:44.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.377+0000 7f8b0f26b640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b080cf6a0 con 0x7f8b081614d0 2024-03-06T07:24:44.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.377+0000 7f8b0d808640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f8b080d24b0 0x7f8b080b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.377+0000 7f8b0c806640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f8b080d1910 0x7f8b080d1da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:58008/0 (socket says 172.21.15.83:58008) 2024-03-06T07:24:44.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.377+0000 7f8b0c806640 1 -- 172.21.15.83:0/1186276709 learned_addr learned my addr 172.21.15.83:0/1186276709 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:24:44.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.377+0000 7f8b0d007640 1 --2- 172.21.15.83:0/1186276709 >> v2:172.21.15.183:3300/0 conn(0x7f8b081614d0 0x7f8b080d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.377+0000 7f8b0d808640 1 -- 172.21.15.83:0/1186276709 >> v2:172.21.15.183:3300/0 conn(0x7f8b081614d0 msgr2=0x7f8b080d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:44.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.377+0000 7f8b0d808640 1 --2- 172.21.15.83:0/1186276709 >> v2:172.21.15.183:3300/0 conn(0x7f8b081614d0 0x7f8b080d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.377+0000 7f8b0d808640 1 -- 172.21.15.83:0/1186276709 >> v1:172.21.15.83:6789/0 conn(0x7f8b080d1910 legacy=0x7f8b080d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.377+0000 7f8b0d808640 1 -- 172.21.15.83:0/1186276709 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b080cf980 con 0x7f8b080d24b0 2024-03-06T07:24:44.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.377+0000 7f8b0d007640 1 --2- 172.21.15.83:0/1186276709 >> v2:172.21.15.183:3300/0 conn(0x7f8b081614d0 0x7f8b080d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:44.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.378+0000 7f8b0d808640 1 --2- 172.21.15.83:0/1186276709 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8b080d24b0 0x7f8b080b0e30 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f8b0808c470 tx=0x7f8b0402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81a468e91b37dee0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.378+0000 7f8afbfff640 1 -- 172.21.15.83:0/1186276709 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b04035020 con 0x7f8b080d24b0 2024-03-06T07:24:44.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.378+0000 7f8afbfff640 1 -- 172.21.15.83:0/1186276709 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b04031830 con 0x7f8b080d24b0 2024-03-06T07:24:44.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.378+0000 7f8afbfff640 1 -- 172.21.15.83:0/1186276709 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b04031c00 con 0x7f8b080d24b0 2024-03-06T07:24:44.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.379+0000 7f8b0f26b640 1 -- 172.21.15.83:0/1186276709 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8b080d24b0 msgr2=0x7f8b080b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.379+0000 7f8b0f26b640 1 --2- 172.21.15.83:0/1186276709 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8b080d24b0 0x7f8b080b0e30 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f8b0808c470 tx=0x7f8b0402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.379+0000 7f8b0f26b640 1 -- 172.21.15.83:0/1186276709 shutdown_connections 2024-03-06T07:24:44.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.379+0000 7f8b0f26b640 1 --2- 172.21.15.83:0/1186276709 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8b080d24b0 0x7f8b080b0e30 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.379+0000 7f8b0f26b640 1 --2- 172.21.15.83:0/1186276709 >> v2:172.21.15.183:3300/0 conn(0x7f8b081614d0 0x7f8b080d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.379+0000 7f8b0f26b640 1 -- 172.21.15.83:0/1186276709 >> 172.21.15.83:0/1186276709 conn(0x7f8b080d3d40 msgr2=0x7f8b080d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:44.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.379+0000 7f8b0f26b640 1 -- 172.21.15.83:0/1186276709 shutdown_connections 2024-03-06T07:24:44.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.379+0000 7f8b0f26b640 1 -- 172.21.15.83:0/1186276709 wait complete. 2024-03-06T07:24:44.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.380+0000 7f8b0f26b640 1 Processor -- start 2024-03-06T07:24:44.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.380+0000 7f8b0f26b640 1 -- start start 2024-03-06T07:24:44.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.381+0000 7f8b0f26b640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8b080d1910 0x7f8b080c83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.381+0000 7f8b0f26b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8b080d24b0 0x7f8b080c9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.381+0000 7f8b0f26b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8b081614d0 0x7f8b081f4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.381+0000 7f8b0f26b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b080b1370 con 0x7f8b080d24b0 2024-03-06T07:24:44.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.381+0000 7f8b0f26b640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b080b0910 con 0x7f8b080d1910 2024-03-06T07:24:44.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.381+0000 7f8b0f26b640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b080b14f0 con 0x7f8b081614d0 2024-03-06T07:24:44.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.381+0000 7f8b0c806640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8b080d24b0 0x7f8b080c9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.381+0000 7f8b0c806640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8b080d24b0 0x7f8b080c9840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34532/0 (socket says 172.21.15.83:34532) 2024-03-06T07:24:44.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.381+0000 7f8b0c806640 1 -- 172.21.15.83:0/725708097 learned_addr learned my addr 172.21.15.83:0/725708097 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:44.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.381+0000 7f8b0d808640 1 --2- 172.21.15.83:0/725708097 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8b081614d0 0x7f8b081f4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.381+0000 7f8b0d007640 1 --2- 172.21.15.83:0/725708097 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8b080d1910 0x7f8b080c83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.381+0000 7f8b0c806640 1 -- 172.21.15.83:0/725708097 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8b081614d0 msgr2=0x7f8b081f4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.381+0000 7f8b0c806640 1 --2- 172.21.15.83:0/725708097 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8b081614d0 0x7f8b081f4e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.381+0000 7f8b0c806640 1 -- 172.21.15.83:0/725708097 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8b080d1910 msgr2=0x7f8b080c83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.381+0000 7f8b0c806640 1 --2- 172.21.15.83:0/725708097 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8b080d1910 0x7f8b080c83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.381+0000 7f8b0c806640 1 -- 172.21.15.83:0/725708097 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b080b3b00 con 0x7f8b080d24b0 2024-03-06T07:24:44.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.381+0000 7f8b0d007640 1 --2- 172.21.15.83:0/725708097 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8b080d1910 0x7f8b080c83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:44.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.381+0000 7f8b0c806640 1 --2- 172.21.15.83:0/725708097 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8b080d24b0 0x7f8b080c9840 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f8afc00c430 tx=0x7f8afc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.382+0000 7f8af9ffb640 1 -- 172.21.15.83:0/725708097 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8afc017070 con 0x7f8b080d24b0 2024-03-06T07:24:44.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.382+0000 7f8af9ffb640 1 -- 172.21.15.83:0/725708097 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8afc005ce0 con 0x7f8b080d24b0 2024-03-06T07:24:44.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.382+0000 7f8af9ffb640 1 -- 172.21.15.83:0/725708097 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8afc0072f0 con 0x7f8b080d24b0 2024-03-06T07:24:44.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.382+0000 7f8b0f26b640 1 -- 172.21.15.83:0/725708097 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b080b3de0 con 0x7f8b080d24b0 2024-03-06T07:24:44.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.382+0000 7f8b0f26b640 1 -- 172.21.15.83:0/725708097 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b080b43a0 con 0x7f8b080d24b0 2024-03-06T07:24:44.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.383+0000 7f8af9ffb640 1 -- 172.21.15.83:0/725708097 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8afc01a070 con 0x7f8b080d24b0 2024-03-06T07:24:44.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.383+0000 7f8b0f26b640 1 -- 172.21.15.83:0/725708097 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b08068ff0 con 0x7f8b080d24b0 2024-03-06T07:24:44.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.383+0000 7f8af9ffb640 1 --2- 172.21.15.83:0/725708097 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8aec06b500 0x7f8aec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.383+0000 7f8af9ffb640 1 -- 172.21.15.83:0/725708097 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8afc089620 con 0x7f8b080d24b0 2024-03-06T07:24:44.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.383+0000 7f8b0d007640 1 --2- 172.21.15.83:0/725708097 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8aec06b500 0x7f8aec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.384+0000 7f8b0d007640 1 --2- 172.21.15.83:0/725708097 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8aec06b500 0x7f8aec06d9c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f8b00007f40 tx=0x7f8b00007490 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.388+0000 7f8af9ffb640 1 -- 172.21.15.83:0/725708097 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8afc0584a0 con 0x7f8b080d24b0 2024-03-06T07:24:44.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.591+0000 7f8b0f26b640 1 -- 172.21.15.83:0/725708097 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8b080d0950 con 0x7f8b080d24b0 2024-03-06T07:24:44.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.591+0000 7f8af9ffb640 1 -- 172.21.15.83:0/725708097 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v57) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8afc057da0 con 0x7f8b080d24b0 2024-03-06T07:24:44.593 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:24:44.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.593+0000 7f8b0f26b640 1 -- 172.21.15.83:0/725708097 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8aec06b500 msgr2=0x7f8aec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.593+0000 7f8b0f26b640 1 --2- 172.21.15.83:0/725708097 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8aec06b500 0x7f8aec06d9c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f8b00007f40 tx=0x7f8b00007490 comp rx=0 tx=0).stop 2024-03-06T07:24:44.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.593+0000 7f8b0f26b640 1 -- 172.21.15.83:0/725708097 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8b080d24b0 msgr2=0x7f8b080c9840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.593+0000 7f8b0f26b640 1 --2- 172.21.15.83:0/725708097 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8b080d24b0 0x7f8b080c9840 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f8afc00c430 tx=0x7f8afc00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.593+0000 7f8b0f26b640 1 -- 172.21.15.83:0/725708097 shutdown_connections 2024-03-06T07:24:44.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.593+0000 7f8b0f26b640 1 --2- 172.21.15.83:0/725708097 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8aec06b500 0x7f8aec06d9c0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.593+0000 7f8b0f26b640 1 --2- 172.21.15.83:0/725708097 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8b081614d0 0x7f8b081f4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.593+0000 7f8b0f26b640 1 --2- 172.21.15.83:0/725708097 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8b080d24b0 0x7f8b080c9840 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.593+0000 7f8b0f26b640 1 --2- 172.21.15.83:0/725708097 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8b080d1910 0x7f8b080c83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.593+0000 7f8b0f26b640 1 -- 172.21.15.83:0/725708097 >> 172.21.15.83:0/725708097 conn(0x7f8b080d3d40 msgr2=0x7f8b0806c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:44.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.594+0000 7f8b0f26b640 1 -- 172.21.15.83:0/725708097 shutdown_connections 2024-03-06T07:24:44.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.594+0000 7f8b0f26b640 1 -- 172.21.15.83:0/725708097 wait complete. 2024-03-06T07:24:44.609 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:24:44.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.711+0000 7fede2f8a640 1 Processor -- start 2024-03-06T07:24:44.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.711+0000 7fede2f8a640 1 -- start start 2024-03-06T07:24:44.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.712+0000 7fede2f8a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7feddc1614d0 0x7feddc0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.712+0000 7fede2f8a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feddc1618b0 con 0x7feddc0d24b0 2024-03-06T07:24:44.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.712+0000 7fede2f8a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feddc0cf400 con 0x7feddc0d1910 2024-03-06T07:24:44.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.712+0000 7fede2f8a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feddc0cf6a0 con 0x7feddc1614d0 2024-03-06T07:24:44.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.712+0000 7fede0d26640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7feddc1614d0 0x7feddc0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.712+0000 7fede1527640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7feddc0d24b0 0x7feddc0b0e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:58016/0 (socket says 172.21.15.83:58016) 2024-03-06T07:24:44.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.712+0000 7fede1527640 1 -- 172.21.15.83:0/2965240010 learned_addr learned my addr 172.21.15.83:0/2965240010 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:24:44.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.712+0000 7fede0d26640 1 -- 172.21.15.83:0/2965240010 >> v1:172.21.15.183:6789/0 conn(0x7feddc0d1910 legacy=0x7feddc0d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:44.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.712+0000 7fede0d26640 1 -- 172.21.15.83:0/2965240010 >> v1:172.21.15.83:6789/0 conn(0x7feddc0d24b0 legacy=0x7feddc0b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.712+0000 7fede0d26640 1 -- 172.21.15.83:0/2965240010 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feddc0cf980 con 0x7feddc1614d0 2024-03-06T07:24:44.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.712+0000 7feddb7fe640 1 -- 172.21.15.83:0/2965240010 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2249493118 0 0) 0x7feddc1618b0 con 0x7feddc0d24b0 2024-03-06T07:24:44.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.713+0000 7fede0d26640 1 --2- 172.21.15.83:0/2965240010 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feddc1614d0 0x7feddc0d4860 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7feddc08c470 tx=0x7fedd402be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b93cdce5b0a457af server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.713+0000 7feddb7fe640 1 -- 172.21.15.83:0/2965240010 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd4039070 con 0x7feddc1614d0 2024-03-06T07:24:44.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.713+0000 7feddb7fe640 1 -- 172.21.15.83:0/2965240010 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd402cc40 con 0x7feddc1614d0 2024-03-06T07:24:44.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.713+0000 7feddb7fe640 1 -- 172.21.15.83:0/2965240010 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd4034440 con 0x7feddc1614d0 2024-03-06T07:24:44.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.714+0000 7fede2f8a640 1 -- 172.21.15.83:0/2965240010 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feddc1614d0 msgr2=0x7feddc0d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.714+0000 7fede2f8a640 1 --2- 172.21.15.83:0/2965240010 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feddc1614d0 0x7feddc0d4860 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7feddc08c470 tx=0x7fedd402be90 comp rx=0 tx=0).stop 2024-03-06T07:24:44.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.714+0000 7fede2f8a640 1 -- 172.21.15.83:0/2965240010 shutdown_connections 2024-03-06T07:24:44.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.714+0000 7fede2f8a640 1 --2- 172.21.15.83:0/2965240010 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feddc1614d0 0x7feddc0d4860 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.714+0000 7fede2f8a640 1 -- 172.21.15.83:0/2965240010 >> 172.21.15.83:0/2965240010 conn(0x7feddc0d3d40 msgr2=0x7feddc0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:44.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.714+0000 7fede2f8a640 1 -- 172.21.15.83:0/2965240010 shutdown_connections 2024-03-06T07:24:44.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.714+0000 7fede2f8a640 1 -- 172.21.15.83:0/2965240010 wait complete. 2024-03-06T07:24:44.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.715+0000 7fede2f8a640 1 Processor -- start 2024-03-06T07:24:44.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.715+0000 7fede2f8a640 1 -- start start 2024-03-06T07:24:44.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.716+0000 7fede2f8a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feddc0d1910 0x7feddc15f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.716+0000 7fede2f8a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feddc0d24b0 0x7feddc0c9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.716+0000 7fede2f8a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feddc1614d0 0x7feddc0ca110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.716+0000 7fede2f8a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feddc0b1540 con 0x7feddc1614d0 2024-03-06T07:24:44.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.716+0000 7fede2f8a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feddc0b0910 con 0x7feddc0d1910 2024-03-06T07:24:44.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.716+0000 7fede2f8a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feddc0b16c0 con 0x7feddc0d24b0 2024-03-06T07:24:44.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.716+0000 7fede1527640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feddc1614d0 0x7feddc0ca110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.716+0000 7fede1527640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feddc1614d0 0x7feddc0ca110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34538/0 (socket says 172.21.15.83:34538) 2024-03-06T07:24:44.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.716+0000 7fede1527640 1 -- 172.21.15.83:0/3453200126 learned_addr learned my addr 172.21.15.83:0/3453200126 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:44.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.716+0000 7fede1527640 1 -- 172.21.15.83:0/3453200126 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feddc0d24b0 msgr2=0x7feddc0c9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.716+0000 7feddbfff640 1 --2- 172.21.15.83:0/3453200126 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feddc0d24b0 0x7feddc0c9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.716+0000 7fede0d26640 1 --2- 172.21.15.83:0/3453200126 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feddc0d1910 0x7feddc15f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.716+0000 7fede1527640 1 --2- 172.21.15.83:0/3453200126 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feddc0d24b0 0x7feddc0c9bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.716+0000 7fede1527640 1 -- 172.21.15.83:0/3453200126 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feddc0d1910 msgr2=0x7feddc15f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.716+0000 7fede1527640 1 --2- 172.21.15.83:0/3453200126 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feddc0d1910 0x7feddc15f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.716+0000 7fede1527640 1 -- 172.21.15.83:0/3453200126 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feddc0cdcb0 con 0x7feddc1614d0 2024-03-06T07:24:44.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.716+0000 7fede0d26640 1 --2- 172.21.15.83:0/3453200126 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feddc0d1910 0x7feddc15f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:44.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.716+0000 7fede1527640 1 --2- 172.21.15.83:0/3453200126 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feddc1614d0 0x7feddc0ca110 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fedcc00c480 tx=0x7fedcc00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.717+0000 7fedd97fa640 1 -- 172.21.15.83:0/3453200126 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedcc018070 con 0x7feddc1614d0 2024-03-06T07:24:44.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.717+0000 7fedd97fa640 1 -- 172.21.15.83:0/3453200126 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedcc007500 con 0x7feddc1614d0 2024-03-06T07:24:44.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.717+0000 7fede2f8a640 1 -- 172.21.15.83:0/3453200126 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feddc0cdf90 con 0x7feddc1614d0 2024-03-06T07:24:44.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.717+0000 7fedd97fa640 1 -- 172.21.15.83:0/3453200126 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedcc004a20 con 0x7feddc1614d0 2024-03-06T07:24:44.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.717+0000 7fede2f8a640 1 -- 172.21.15.83:0/3453200126 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feddc0cbfb0 con 0x7feddc1614d0 2024-03-06T07:24:44.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.718+0000 7fede2f8a640 1 -- 172.21.15.83:0/3453200126 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feddc068ff0 con 0x7feddc1614d0 2024-03-06T07:24:44.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.718+0000 7fedd97fa640 1 -- 172.21.15.83:0/3453200126 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fedcc01b070 con 0x7feddc1614d0 2024-03-06T07:24:44.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.719+0000 7fedd97fa640 1 --2- 172.21.15.83:0/3453200126 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fedbc06b4d0 0x7fedbc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.719+0000 7fedd97fa640 1 -- 172.21.15.83:0/3453200126 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fedcc089490 con 0x7feddc1614d0 2024-03-06T07:24:44.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.719+0000 7fede0d26640 1 --2- 172.21.15.83:0/3453200126 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fedbc06b4d0 0x7fedbc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.719+0000 7fede0d26640 1 --2- 172.21.15.83:0/3453200126 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fedbc06b4d0 0x7fedbc06d990 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fedd4036000 tx=0x7fedd4030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.723+0000 7fedd97fa640 1 -- 172.21.15.83:0/3453200126 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fedcc0583a0 con 0x7feddc1614d0 2024-03-06T07:24:44.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.900+0000 7fede2f8a640 1 -- 172.21.15.83:0/3453200126 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7feddc0d0950 con 0x7feddc1614d0 2024-03-06T07:24:44.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.900+0000 7fedd97fa640 1 -- 172.21.15.83:0/3453200126 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v57) v1 ==== 104+0+11 (secure 0 0 0) 0x7fedcc057ca0 con 0x7feddc1614d0 2024-03-06T07:24:44.903 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-03-06T07:24:44.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.902+0000 7fede2f8a640 1 -- 172.21.15.83:0/3453200126 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fedbc06b4d0 msgr2=0x7fedbc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.902+0000 7fede2f8a640 1 --2- 172.21.15.83:0/3453200126 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fedbc06b4d0 0x7fedbc06d990 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fedd4036000 tx=0x7fedd4030000 comp rx=0 tx=0).stop 2024-03-06T07:24:44.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.902+0000 7fede2f8a640 1 -- 172.21.15.83:0/3453200126 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feddc1614d0 msgr2=0x7feddc0ca110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.902+0000 7fede2f8a640 1 --2- 172.21.15.83:0/3453200126 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feddc1614d0 0x7feddc0ca110 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fedcc00c480 tx=0x7fedcc00c940 comp rx=0 tx=0).stop 2024-03-06T07:24:44.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.903+0000 7fede2f8a640 1 -- 172.21.15.83:0/3453200126 shutdown_connections 2024-03-06T07:24:44.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.903+0000 7fede2f8a640 1 --2- 172.21.15.83:0/3453200126 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fedbc06b4d0 0x7fedbc06d990 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.903+0000 7fede2f8a640 1 --2- 172.21.15.83:0/3453200126 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feddc1614d0 0x7feddc0ca110 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.903+0000 7fede2f8a640 1 --2- 172.21.15.83:0/3453200126 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feddc0d24b0 0x7feddc0c9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.903+0000 7fede2f8a640 1 --2- 172.21.15.83:0/3453200126 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feddc0d1910 0x7feddc15f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.903+0000 7fede2f8a640 1 -- 172.21.15.83:0/3453200126 >> 172.21.15.83:0/3453200126 conn(0x7feddc0d3d40 msgr2=0x7feddc0b35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:44.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.903+0000 7fede2f8a640 1 -- 172.21.15.83:0/3453200126 shutdown_connections 2024-03-06T07:24:44.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:44.903+0000 7fede2f8a640 1 -- 172.21.15.83:0/3453200126 wait complete. 2024-03-06T07:24:44.916 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:24:45.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.020+0000 7fddeed56640 1 Processor -- start 2024-03-06T07:24:45.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.020+0000 7fddeed56640 1 -- start start 2024-03-06T07:24:45.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.020+0000 7fddeed56640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fdde806fd10 0x7fdde80cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.020+0000 7fddeed56640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdde80d0af0 0x7fdde80ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.020+0000 7fddeed56640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdde80cd1e0 con 0x7fdde815c050 2024-03-06T07:24:45.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.020+0000 7fddeed56640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdde80cd4b0 con 0x7fdde806fd10 2024-03-06T07:24:45.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.020+0000 7fddeed56640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdde80cd790 con 0x7fdde80d0af0 2024-03-06T07:24:45.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.021+0000 7fdded2f3640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdde80d0af0 0x7fdde80ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.021+0000 7fdde7fff640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fdde806fd10 0x7fdde80cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.021+0000 7fdded2f3640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdde80d0af0 0x7fdde80ccca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:36864/0 (socket says 172.21.15.83:36864) 2024-03-06T07:24:45.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.021+0000 7fdded2f3640 1 -- 172.21.15.83:0/3521784149 learned_addr learned my addr 172.21.15.83:0/3521784149 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:45.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.021+0000 7fdded2f3640 1 -- 172.21.15.83:0/3521784149 >> v2:172.21.15.183:3300/0 conn(0x7fdde806fd10 msgr2=0x7fdde80cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:45.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.021+0000 7fdded2f3640 1 --2- 172.21.15.83:0/3521784149 >> v2:172.21.15.183:3300/0 conn(0x7fdde806fd10 0x7fdde80cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.021+0000 7fdded2f3640 1 -- 172.21.15.83:0/3521784149 >> v1:172.21.15.183:6789/0 conn(0x7fdde815c050 legacy=0x7fdde806f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.021+0000 7fdded2f3640 1 -- 172.21.15.83:0/3521784149 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde80cda70 con 0x7fdde80d0af0 2024-03-06T07:24:45.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.021+0000 7fdde7fff640 1 --2- 172.21.15.83:0/3521784149 >> v2:172.21.15.183:3300/0 conn(0x7fdde806fd10 0x7fdde80cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:45.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.022+0000 7fdded2f3640 1 --2- 172.21.15.83:0/3521784149 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdde80d0af0 0x7fdde80ccca0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fdde0009e50 tx=0x7fdde002bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b65e90119b1161d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:45.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.022+0000 7fdde77fe640 1 -- 172.21.15.83:0/3521784149 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde002ca70 con 0x7fdde80d0af0 2024-03-06T07:24:45.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.022+0000 7fdde77fe640 1 -- 172.21.15.83:0/3521784149 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdde002cc10 con 0x7fdde80d0af0 2024-03-06T07:24:45.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.022+0000 7fdde77fe640 1 -- 172.21.15.83:0/3521784149 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde00343f0 con 0x7fdde80d0af0 2024-03-06T07:24:45.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.023+0000 7fddeed56640 1 -- 172.21.15.83:0/3521784149 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdde80d0af0 msgr2=0x7fdde80ccca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.023+0000 7fddeed56640 1 --2- 172.21.15.83:0/3521784149 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdde80d0af0 0x7fdde80ccca0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fdde0009e50 tx=0x7fdde002bed0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.023+0000 7fddeed56640 1 -- 172.21.15.83:0/3521784149 shutdown_connections 2024-03-06T07:24:45.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.023+0000 7fddeed56640 1 --2- 172.21.15.83:0/3521784149 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdde80d0af0 0x7fdde80ccca0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.023+0000 7fddeed56640 1 --2- 172.21.15.83:0/3521784149 >> v2:172.21.15.183:3300/0 conn(0x7fdde806fd10 0x7fdde80cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.023+0000 7fddeed56640 1 -- 172.21.15.83:0/3521784149 >> 172.21.15.83:0/3521784149 conn(0x7fdde80b3800 msgr2=0x7fdde80b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:45.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.023+0000 7fddeed56640 1 -- 172.21.15.83:0/3521784149 shutdown_connections 2024-03-06T07:24:45.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.023+0000 7fddeed56640 1 -- 172.21.15.83:0/3521784149 wait complete. 2024-03-06T07:24:45.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.024+0000 7fddeed56640 1 Processor -- start 2024-03-06T07:24:45.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.024+0000 7fddeed56640 1 -- start start 2024-03-06T07:24:45.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.025+0000 7fddeed56640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdde806fd10 0x7fdde81589f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.025+0000 7fddeed56640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdde80d0af0 0x7fdde809baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.025+0000 7fddeed56640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdde815c050 0x7fdde81576d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.025+0000 7fddeed56640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde8165c30 con 0x7fdde80d0af0 2024-03-06T07:24:45.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.025+0000 7fddeed56640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde8165ab0 con 0x7fdde815c050 2024-03-06T07:24:45.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.025+0000 7fddeed56640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdde8165db0 con 0x7fdde806fd10 2024-03-06T07:24:45.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.025+0000 7fdde7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdde80d0af0 0x7fdde809baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.025+0000 7fdde7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdde80d0af0 0x7fdde809baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34554/0 (socket says 172.21.15.83:34554) 2024-03-06T07:24:45.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.025+0000 7fdde7fff640 1 -- 172.21.15.83:0/2876962241 learned_addr learned my addr 172.21.15.83:0/2876962241 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:45.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.025+0000 7fdded2f3640 1 --2- 172.21.15.83:0/2876962241 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdde815c050 0x7fdde81576d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.025+0000 7fddecaf2640 1 --2- 172.21.15.83:0/2876962241 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdde806fd10 0x7fdde81589f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.025+0000 7fdde7fff640 1 -- 172.21.15.83:0/2876962241 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdde806fd10 msgr2=0x7fdde81589f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.025+0000 7fdde7fff640 1 --2- 172.21.15.83:0/2876962241 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdde806fd10 0x7fdde81589f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.025+0000 7fdde7fff640 1 -- 172.21.15.83:0/2876962241 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdde815c050 msgr2=0x7fdde81576d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.025+0000 7fdde7fff640 1 --2- 172.21.15.83:0/2876962241 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdde815c050 0x7fdde81576d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.025+0000 7fdde7fff640 1 -- 172.21.15.83:0/2876962241 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde80b75f0 con 0x7fdde80d0af0 2024-03-06T07:24:45.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.025+0000 7fddecaf2640 1 --2- 172.21.15.83:0/2876962241 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdde806fd10 0x7fdde81589f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:45.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.026+0000 7fdde7fff640 1 --2- 172.21.15.83:0/2876962241 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdde80d0af0 0x7fdde809baf0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fddd400e960 tx=0x7fddd400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:45.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.026+0000 7fdde57fa640 1 -- 172.21.15.83:0/2876962241 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd4017070 con 0x7fdde80d0af0 2024-03-06T07:24:45.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.026+0000 7fdde57fa640 1 -- 172.21.15.83:0/2876962241 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddd400ca00 con 0x7fdde80d0af0 2024-03-06T07:24:45.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.026+0000 7fdde57fa640 1 -- 172.21.15.83:0/2876962241 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd400cdd0 con 0x7fdde80d0af0 2024-03-06T07:24:45.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.026+0000 7fddeed56640 1 -- 172.21.15.83:0/2876962241 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdde80b78d0 con 0x7fdde80d0af0 2024-03-06T07:24:45.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.026+0000 7fddeed56640 1 -- 172.21.15.83:0/2876962241 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdde81581f0 con 0x7fdde80d0af0 2024-03-06T07:24:45.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.027+0000 7fddeed56640 1 -- 172.21.15.83:0/2876962241 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdde809fe60 con 0x7fdde80d0af0 2024-03-06T07:24:45.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.027+0000 7fdde57fa640 1 -- 172.21.15.83:0/2876962241 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fddd401a070 con 0x7fdde80d0af0 2024-03-06T07:24:45.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.028+0000 7fdde57fa640 1 --2- 172.21.15.83:0/2876962241 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fddcc06b4d0 0x7fddcc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.028+0000 7fdde57fa640 1 -- 172.21.15.83:0/2876962241 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fddd408a330 con 0x7fdde80d0af0 2024-03-06T07:24:45.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.028+0000 7fddecaf2640 1 --2- 172.21.15.83:0/2876962241 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fddcc06b4d0 0x7fddcc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.028+0000 7fddecaf2640 1 --2- 172.21.15.83:0/2876962241 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fddcc06b4d0 0x7fddcc06d990 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fdddc001700 tx=0x7fdddc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:45.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.032+0000 7fdde57fa640 1 -- 172.21.15.83:0/2876962241 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fddd4058bf0 con 0x7fdde80d0af0 2024-03-06T07:24:45.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.205+0000 7fddeed56640 1 -- 172.21.15.83:0/2876962241 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdde8068ff0 con 0x7fdde80d0af0 2024-03-06T07:24:45.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.205+0000 7fdde57fa640 1 -- 172.21.15.83:0/2876962241 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v57) v1 ==== 100+0+11 (secure 0 0 0) 0x7fddd405cc90 con 0x7fdde80d0af0 2024-03-06T07:24:45.207 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 64 2024-03-06T07:24:45.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.207+0000 7fddeed56640 1 -- 172.21.15.83:0/2876962241 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fddcc06b4d0 msgr2=0x7fddcc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.207+0000 7fddeed56640 1 --2- 172.21.15.83:0/2876962241 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fddcc06b4d0 0x7fddcc06d990 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fdddc001700 tx=0x7fdddc009040 comp rx=0 tx=0).stop 2024-03-06T07:24:45.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.207+0000 7fddeed56640 1 -- 172.21.15.83:0/2876962241 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdde80d0af0 msgr2=0x7fdde809baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.207+0000 7fddeed56640 1 --2- 172.21.15.83:0/2876962241 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdde80d0af0 0x7fdde809baf0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fddd400e960 tx=0x7fddd400ee20 comp rx=0 tx=0).stop 2024-03-06T07:24:45.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.207+0000 7fddeed56640 1 -- 172.21.15.83:0/2876962241 shutdown_connections 2024-03-06T07:24:45.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.207+0000 7fddeed56640 1 --2- 172.21.15.83:0/2876962241 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fddcc06b4d0 0x7fddcc06d990 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.207+0000 7fddeed56640 1 --2- 172.21.15.83:0/2876962241 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdde815c050 0x7fdde81576d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.207+0000 7fddeed56640 1 --2- 172.21.15.83:0/2876962241 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdde80d0af0 0x7fdde809baf0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.207+0000 7fddeed56640 1 --2- 172.21.15.83:0/2876962241 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdde806fd10 0x7fdde81589f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.207+0000 7fddeed56640 1 -- 172.21.15.83:0/2876962241 >> 172.21.15.83:0/2876962241 conn(0x7fdde80b3800 msgr2=0x7fdde80aeb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:45.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.207+0000 7fddeed56640 1 -- 172.21.15.83:0/2876962241 shutdown_connections 2024-03-06T07:24:45.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.207+0000 7fddeed56640 1 -- 172.21.15.83:0/2876962241 wait complete. 2024-03-06T07:24:45.219 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:24:45.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.320+0000 7f3a59dbc640 1 Processor -- start 2024-03-06T07:24:45.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.320+0000 7f3a59dbc640 1 -- start start 2024-03-06T07:24:45.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.320+0000 7f3a59dbc640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f3a5415e000 0x7f3a540d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.320+0000 7f3a59dbc640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a540d4810 0x7f3a540d0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.321+0000 7f3a59dbc640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a540d25e0 con 0x7f3a540c80e0 2024-03-06T07:24:45.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.321+0000 7f3a59dbc640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a540c7c50 con 0x7f3a5415e000 2024-03-06T07:24:45.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.321+0000 7f3a59dbc640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a540d2880 con 0x7f3a540d4810 2024-03-06T07:24:45.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.321+0000 7f3a537fe640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f3a5415e000 0x7f3a540d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.321+0000 7f3a52ffd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a540d4810 0x7f3a540d0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.321+0000 7f3a537fe640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f3a5415e000 0x7f3a540d0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:58972/0 (socket says 172.21.15.83:58972) 2024-03-06T07:24:45.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.321+0000 7f3a52ffd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a540d4810 0x7f3a540d0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:36886/0 (socket says 172.21.15.83:36886) 2024-03-06T07:24:45.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.321+0000 7f3a53fff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f3a540c80e0 0x7f3a540c8570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:47086/0 (socket says 172.21.15.83:47086) 2024-03-06T07:24:45.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.321+0000 7f3a537fe640 1 -- 172.21.15.83:0/941198239 learned_addr learned my addr 172.21.15.83:0/941198239 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:45.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.321+0000 7f3a52ffd640 1 -- 172.21.15.83:0/941198239 >> v2:172.21.15.183:3300/0 conn(0x7f3a5415e000 msgr2=0x7f3a540d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:45.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.321+0000 7f3a52ffd640 1 --2- 172.21.15.83:0/941198239 >> v2:172.21.15.183:3300/0 conn(0x7f3a5415e000 0x7f3a540d0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.321+0000 7f3a52ffd640 1 -- 172.21.15.83:0/941198239 >> v1:172.21.15.183:6789/0 conn(0x7f3a540c80e0 legacy=0x7f3a540c8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.321+0000 7f3a52ffd640 1 -- 172.21.15.83:0/941198239 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a540d2b40 con 0x7f3a540d4810 2024-03-06T07:24:45.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.321+0000 7f3a537fe640 1 --2- 172.21.15.83:0/941198239 >> v2:172.21.15.183:3300/0 conn(0x7f3a5415e000 0x7f3a540d0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:45.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.322+0000 7f3a52ffd640 1 --2- 172.21.15.83:0/941198239 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a540d4810 0x7f3a540d0bc0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3a480097a0 tx=0x7f3a4802b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77e91d9273864717 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:45.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.322+0000 7f3a527fc640 1 -- 172.21.15.83:0/941198239 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a48034070 con 0x7f3a540d4810 2024-03-06T07:24:45.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.322+0000 7f3a527fc640 1 -- 172.21.15.83:0/941198239 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a48030610 con 0x7f3a540d4810 2024-03-06T07:24:45.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.322+0000 7f3a527fc640 1 -- 172.21.15.83:0/941198239 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a480309e0 con 0x7f3a540d4810 2024-03-06T07:24:45.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.323+0000 7f3a59dbc640 1 -- 172.21.15.83:0/941198239 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a540d4810 msgr2=0x7f3a540d0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.323+0000 7f3a59dbc640 1 --2- 172.21.15.83:0/941198239 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a540d4810 0x7f3a540d0bc0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3a480097a0 tx=0x7f3a4802b960 comp rx=0 tx=0).stop 2024-03-06T07:24:45.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.323+0000 7f3a59dbc640 1 -- 172.21.15.83:0/941198239 shutdown_connections 2024-03-06T07:24:45.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.323+0000 7f3a59dbc640 1 --2- 172.21.15.83:0/941198239 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a540d4810 0x7f3a540d0bc0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.323+0000 7f3a59dbc640 1 --2- 172.21.15.83:0/941198239 >> v2:172.21.15.183:3300/0 conn(0x7f3a5415e000 0x7f3a540d0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.323+0000 7f3a59dbc640 1 -- 172.21.15.83:0/941198239 >> 172.21.15.83:0/941198239 conn(0x7f3a540d3cf0 msgr2=0x7f3a540d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:45.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.323+0000 7f3a59dbc640 1 -- 172.21.15.83:0/941198239 shutdown_connections 2024-03-06T07:24:45.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.323+0000 7f3a59dbc640 1 -- 172.21.15.83:0/941198239 wait complete. 2024-03-06T07:24:45.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.324+0000 7f3a59dbc640 1 Processor -- start 2024-03-06T07:24:45.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.324+0000 7f3a59dbc640 1 -- start start 2024-03-06T07:24:45.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.325+0000 7f3a59dbc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a540c80e0 0x7f3a541f4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.325+0000 7f3a59dbc640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a540d4810 0x7f3a5415bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.325+0000 7f3a59dbc640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a5415e000 0x7f3a5409ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.325+0000 7f3a59dbc640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5415e3e0 con 0x7f3a540c80e0 2024-03-06T07:24:45.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.325+0000 7f3a59dbc640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a540d22d0 con 0x7f3a5415e000 2024-03-06T07:24:45.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.325+0000 7f3a59dbc640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a540cf7d0 con 0x7f3a540d4810 2024-03-06T07:24:45.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.325+0000 7f3a537fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a540c80e0 0x7f3a541f4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.325+0000 7f3a537fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a540c80e0 0x7f3a541f4b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34558/0 (socket says 172.21.15.83:34558) 2024-03-06T07:24:45.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.325+0000 7f3a537fe640 1 -- 172.21.15.83:0/1253473565 learned_addr learned my addr 172.21.15.83:0/1253473565 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:45.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.325+0000 7f3a52ffd640 1 --2- 172.21.15.83:0/1253473565 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a540d4810 0x7f3a5415bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.325+0000 7f3a53fff640 1 --2- 172.21.15.83:0/1253473565 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a5415e000 0x7f3a5409ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.326+0000 7f3a537fe640 1 -- 172.21.15.83:0/1253473565 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a540d4810 msgr2=0x7f3a5415bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.326+0000 7f3a537fe640 1 --2- 172.21.15.83:0/1253473565 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a540d4810 0x7f3a5415bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.326+0000 7f3a537fe640 1 -- 172.21.15.83:0/1253473565 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a5415e000 msgr2=0x7f3a5409ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.326+0000 7f3a537fe640 1 --2- 172.21.15.83:0/1253473565 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a5415e000 0x7f3a5409ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.326+0000 7f3a537fe640 1 -- 172.21.15.83:0/1253473565 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a541f50b0 con 0x7f3a540c80e0 2024-03-06T07:24:45.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.326+0000 7f3a53fff640 1 --2- 172.21.15.83:0/1253473565 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a5415e000 0x7f3a5409ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:45.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.326+0000 7f3a537fe640 1 --2- 172.21.15.83:0/1253473565 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a540c80e0 0x7f3a541f4b70 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f3a4400e9b0 tx=0x7f3a4400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:45.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.326+0000 7f3a3bfff640 1 -- 172.21.15.83:0/1253473565 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a44017070 con 0x7f3a540c80e0 2024-03-06T07:24:45.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.326+0000 7f3a3bfff640 1 -- 172.21.15.83:0/1253473565 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a4400c9a0 con 0x7f3a540c80e0 2024-03-06T07:24:45.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.326+0000 7f3a59dbc640 1 -- 172.21.15.83:0/1253473565 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a5409e310 con 0x7f3a540c80e0 2024-03-06T07:24:45.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.326+0000 7f3a3bfff640 1 -- 172.21.15.83:0/1253473565 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a4400cd50 con 0x7f3a540c80e0 2024-03-06T07:24:45.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.326+0000 7f3a59dbc640 1 -- 172.21.15.83:0/1253473565 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a5409e5d0 con 0x7f3a540c80e0 2024-03-06T07:24:45.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.327+0000 7f3a59dbc640 1 -- 172.21.15.83:0/1253473565 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a540c91f0 con 0x7f3a540c80e0 2024-03-06T07:24:45.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.327+0000 7f3a3bfff640 1 -- 172.21.15.83:0/1253473565 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3a4401a070 con 0x7f3a540c80e0 2024-03-06T07:24:45.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.328+0000 7f3a3bfff640 1 --2- 172.21.15.83:0/1253473565 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3a4006b4d0 0x7f3a4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.328+0000 7f3a3bfff640 1 -- 172.21.15.83:0/1253473565 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3a4408a0b0 con 0x7f3a540c80e0 2024-03-06T07:24:45.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.328+0000 7f3a52ffd640 1 --2- 172.21.15.83:0/1253473565 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3a4006b4d0 0x7f3a4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.328+0000 7f3a52ffd640 1 --2- 172.21.15.83:0/1253473565 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3a4006b4d0 0x7f3a4006d990 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f3a48036000 tx=0x7f3a4803b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:45.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.336+0000 7f3a3bfff640 1 -- 172.21.15.83:0/1253473565 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3a540c91f0 con 0x7f3a540c80e0 2024-03-06T07:24:45.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.548+0000 7f3a59dbc640 1 -- 172.21.15.83:0/1253473565 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f3a541f76f0 con 0x7f3a540c80e0 2024-03-06T07:24:45.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.549+0000 7f3a3bfff640 1 -- 172.21.15.83:0/1253473565 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v57) v1 ==== 80+0+0 (secure 0 0 0) 0x7f3a44058860 con 0x7f3a540c80e0 2024-03-06T07:24:45.551 INFO:teuthology.orchestra.run.smithi083.stderr:listed 0 entries 2024-03-06T07:24:45.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.551+0000 7f3a59dbc640 1 -- 172.21.15.83:0/1253473565 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3a4006b4d0 msgr2=0x7f3a4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.551+0000 7f3a59dbc640 1 --2- 172.21.15.83:0/1253473565 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3a4006b4d0 0x7f3a4006d990 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f3a48036000 tx=0x7f3a4803b040 comp rx=0 tx=0).stop 2024-03-06T07:24:45.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.551+0000 7f3a59dbc640 1 -- 172.21.15.83:0/1253473565 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a540c80e0 msgr2=0x7f3a541f4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.551+0000 7f3a59dbc640 1 --2- 172.21.15.83:0/1253473565 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a540c80e0 0x7f3a541f4b70 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f3a4400e9b0 tx=0x7f3a4400ee70 comp rx=0 tx=0).stop 2024-03-06T07:24:45.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.551+0000 7f3a59dbc640 1 -- 172.21.15.83:0/1253473565 shutdown_connections 2024-03-06T07:24:45.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.551+0000 7f3a59dbc640 1 --2- 172.21.15.83:0/1253473565 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3a4006b4d0 0x7f3a4006d990 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.551+0000 7f3a59dbc640 1 --2- 172.21.15.83:0/1253473565 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a5415e000 0x7f3a5409ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.551+0000 7f3a59dbc640 1 --2- 172.21.15.83:0/1253473565 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a540d4810 0x7f3a5415bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.551+0000 7f3a59dbc640 1 --2- 172.21.15.83:0/1253473565 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a540c80e0 0x7f3a541f4b70 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.551+0000 7f3a59dbc640 1 -- 172.21.15.83:0/1253473565 >> 172.21.15.83:0/1253473565 conn(0x7f3a540d3cf0 msgr2=0x7f3a5415d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:45.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.551+0000 7f3a59dbc640 1 -- 172.21.15.83:0/1253473565 shutdown_connections 2024-03-06T07:24:45.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:45.551+0000 7f3a59dbc640 1 -- 172.21.15.83:0/1253473565 wait complete. 2024-03-06T07:24:45.564 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi083.front.sepia.ceph.com... 2024-03-06T07:24:45.564 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:45.564 DEBUG:teuthology.orchestra.run.smithi083:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:45.589 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:ceph-fuse[59150]: fuse finished with error 0 and tester_r 0 2024-03-06T07:24:45.590 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:51.692 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi083.front.sepia.ceph.com 2024-03-06T07:24:51.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:51.692 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:24:51.708 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:51.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.847+0000 7f443acd4640 1 Processor -- start 2024-03-06T07:24:51.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.847+0000 7f443acd4640 1 -- start start 2024-03-06T07:24:51.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.848+0000 7f443acd4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f44340d22b0 0x7f44340d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.848+0000 7f443acd4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44340b3430 con 0x7f443415dfe0 2024-03-06T07:24:51.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.848+0000 7f443acd4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44340b3710 con 0x7f44340b41f0 2024-03-06T07:24:51.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.848+0000 7f443acd4640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44340d2c50 con 0x7f44340d22b0 2024-03-06T07:24:51.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.848+0000 7f4433fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f44340d22b0 0x7f44340d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.848+0000 7f4438a70640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f443415dfe0 0x7f44340cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:33922/0 (socket says 172.21.15.83:33922) 2024-03-06T07:24:51.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.848+0000 7f4438a70640 1 -- 172.21.15.83:0/2309441027 learned_addr learned my addr 172.21.15.83:0/2309441027 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:24:51.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.849+0000 7f4433fff640 1 -- 172.21.15.83:0/2309441027 >> v1:172.21.15.183:6789/0 conn(0x7f44340b41f0 legacy=0x7f44340b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:51.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.849+0000 7f4433fff640 1 -- 172.21.15.83:0/2309441027 >> v1:172.21.15.83:6789/0 conn(0x7f443415dfe0 legacy=0x7f44340cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.849+0000 7f4433fff640 1 -- 172.21.15.83:0/2309441027 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44340b39f0 con 0x7f44340d22b0 2024-03-06T07:24:51.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.850+0000 7f4433fff640 1 --2- 172.21.15.83:0/2309441027 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44340d22b0 0x7f44340d2710 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f44200097d0 tx=0x7f442002b6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2915c80a05ac2ee server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.850+0000 7f44337fe640 1 -- 172.21.15.83:0/2309441027 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4420037070 con 0x7f44340d22b0 2024-03-06T07:24:51.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.850+0000 7f44337fe640 1 -- 172.21.15.83:0/2309441027 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4420030710 con 0x7f44340d22b0 2024-03-06T07:24:51.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.850+0000 7f44337fe640 1 -- 172.21.15.83:0/2309441027 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4420030ae0 con 0x7f44340d22b0 2024-03-06T07:24:51.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.850+0000 7f443acd4640 1 -- 172.21.15.83:0/2309441027 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44340d22b0 msgr2=0x7f44340d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.850+0000 7f443acd4640 1 --2- 172.21.15.83:0/2309441027 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44340d22b0 0x7f44340d2710 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f44200097d0 tx=0x7f442002b6f0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.851+0000 7f443acd4640 1 -- 172.21.15.83:0/2309441027 shutdown_connections 2024-03-06T07:24:51.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.851+0000 7f443acd4640 1 --2- 172.21.15.83:0/2309441027 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44340d22b0 0x7f44340d2710 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.851+0000 7f443acd4640 1 -- 172.21.15.83:0/2309441027 >> 172.21.15.83:0/2309441027 conn(0x7f44340d3cd0 msgr2=0x7f44340d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:51.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.851+0000 7f443acd4640 1 -- 172.21.15.83:0/2309441027 shutdown_connections 2024-03-06T07:24:51.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.851+0000 7f443acd4640 1 -- 172.21.15.83:0/2309441027 wait complete. 2024-03-06T07:24:51.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.852+0000 7f443acd4640 1 Processor -- start 2024-03-06T07:24:51.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.852+0000 7f443acd4640 1 -- start start 2024-03-06T07:24:51.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.853+0000 7f443acd4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f44340b41f0 0x7f443414d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.853+0000 7f443acd4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44340d22b0 0x7f44340cd920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.853+0000 7f443acd4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f443415dfe0 0x7f44340b23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.853+0000 7f443acd4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44340b3d60 con 0x7f443415dfe0 2024-03-06T07:24:51.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.853+0000 7f443acd4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44340b31e0 con 0x7f44340b41f0 2024-03-06T07:24:51.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.853+0000 7f443acd4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44340d1c70 con 0x7f44340d22b0 2024-03-06T07:24:51.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.853+0000 7f4439271640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f443415dfe0 0x7f44340b23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.853+0000 7f4439271640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f443415dfe0 0x7f44340b23a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49300/0 (socket says 172.21.15.83:49300) 2024-03-06T07:24:51.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.853+0000 7f4439271640 1 -- 172.21.15.83:0/3786296065 learned_addr learned my addr 172.21.15.83:0/3786296065 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:51.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.853+0000 7f4439271640 1 -- 172.21.15.83:0/3786296065 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44340d22b0 msgr2=0x7f44340cd920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.853+0000 7f4433fff640 1 --2- 172.21.15.83:0/3786296065 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44340d22b0 0x7f44340cd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.853+0000 7f4439271640 1 --2- 172.21.15.83:0/3786296065 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44340d22b0 0x7f44340cd920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.853+0000 7f4439271640 1 -- 172.21.15.83:0/3786296065 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f44340b41f0 msgr2=0x7f443414d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.853+0000 7f4439271640 1 --2- 172.21.15.83:0/3786296065 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f44340b41f0 0x7f443414d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.853+0000 7f4439271640 1 -- 172.21.15.83:0/3786296065 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44340cc2d0 con 0x7f443415dfe0 2024-03-06T07:24:51.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.853+0000 7f4439271640 1 --2- 172.21.15.83:0/3786296065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f443415dfe0 0x7f44340b23a0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f442c00c780 tx=0x7f442c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.853+0000 7f44317fa640 1 -- 172.21.15.83:0/3786296065 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f442c017070 con 0x7f443415dfe0 2024-03-06T07:24:51.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.853+0000 7f44317fa640 1 -- 172.21.15.83:0/3786296065 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f442c003da0 con 0x7f443415dfe0 2024-03-06T07:24:51.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.853+0000 7f443acd4640 1 -- 172.21.15.83:0/3786296065 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44340cc5b0 con 0x7f443415dfe0 2024-03-06T07:24:51.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.853+0000 7f44317fa640 1 -- 172.21.15.83:0/3786296065 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f442c0052b0 con 0x7f443415dfe0 2024-03-06T07:24:51.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.853+0000 7f443acd4640 1 -- 172.21.15.83:0/3786296065 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44340ccb70 con 0x7f443415dfe0 2024-03-06T07:24:51.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.854+0000 7f443acd4640 1 -- 172.21.15.83:0/3786296065 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f443404f330 con 0x7f443415dfe0 2024-03-06T07:24:51.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.855+0000 7f44317fa640 1 -- 172.21.15.83:0/3786296065 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f442c01a070 con 0x7f443415dfe0 2024-03-06T07:24:51.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.855+0000 7f44317fa640 1 --2- 172.21.15.83:0/3786296065 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f441806b4d0 0x7f441806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.855+0000 7f4438a70640 1 --2- 172.21.15.83:0/3786296065 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f441806b4d0 0x7f441806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.855+0000 7f44317fa640 1 -- 172.21.15.83:0/3786296065 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f442c089840 con 0x7f443415dfe0 2024-03-06T07:24:51.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.856+0000 7f4438a70640 1 --2- 172.21.15.83:0/3786296065 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f441806b4d0 0x7f441806d990 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f4428007fd0 tx=0x7f44280079a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:51.860+0000 7f44317fa640 1 -- 172.21.15.83:0/3786296065 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f442c0586c0 con 0x7f443415dfe0 2024-03-06T07:24:52.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.075+0000 7f443acd4640 1 -- 172.21.15.83:0/3786296065 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44340ce920 con 0x7f443415dfe0 2024-03-06T07:24:52.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.075+0000 7f44317fa640 1 -- 172.21.15.83:0/3786296065 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f442c057fc0 con 0x7f443415dfe0 2024-03-06T07:24:52.077 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:24:52.077 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":25,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5665,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/3334609507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3334609507},{"type":"v1","addr":"172.21.15.183:6834","nonce":3334609507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5668,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/691189112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":691189112},{"type":"v1","addr":"172.21.15.183:6835","nonce":691189112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6081,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/3952571284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":3952571284},{"type":"v1","addr":"172.21.15.83:6836","nonce":3952571284}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:19.852157+0000","modified":"2024-03-06T07:24:21.879875+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6084},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6084":{"gid":6084,"name":"a","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/3237506441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3237506441},{"type":"v1","addr":"172.21.15.83:6837","nonce":3237506441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6084,"qdb_cluster":[6084]},"id":4}]} 2024-03-06T07:24:52.078 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 25 2024-03-06T07:24:52.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.077+0000 7f443acd4640 1 -- 172.21.15.83:0/3786296065 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f441806b4d0 msgr2=0x7f441806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.077+0000 7f443acd4640 1 --2- 172.21.15.83:0/3786296065 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f441806b4d0 0x7f441806d990 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f4428007fd0 tx=0x7f44280079a0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.077+0000 7f443acd4640 1 -- 172.21.15.83:0/3786296065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f443415dfe0 msgr2=0x7f44340b23a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.077+0000 7f443acd4640 1 --2- 172.21.15.83:0/3786296065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f443415dfe0 0x7f44340b23a0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f442c00c780 tx=0x7f442c00cc40 comp rx=0 tx=0).stop 2024-03-06T07:24:52.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.078+0000 7f443acd4640 1 -- 172.21.15.83:0/3786296065 shutdown_connections 2024-03-06T07:24:52.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.078+0000 7f443acd4640 1 --2- 172.21.15.83:0/3786296065 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f441806b4d0 0x7f441806d990 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.078+0000 7f443acd4640 1 --2- 172.21.15.83:0/3786296065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f443415dfe0 0x7f44340b23a0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.078+0000 7f443acd4640 1 --2- 172.21.15.83:0/3786296065 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44340d22b0 0x7f44340cd920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.078+0000 7f443acd4640 1 --2- 172.21.15.83:0/3786296065 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f44340b41f0 0x7f443414d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.078+0000 7f443acd4640 1 -- 172.21.15.83:0/3786296065 >> 172.21.15.83:0/3786296065 conn(0x7f44340d3cd0 msgr2=0x7f44340cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:52.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.078+0000 7f443acd4640 1 -- 172.21.15.83:0/3786296065 shutdown_connections 2024-03-06T07:24:52.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.078+0000 7f443acd4640 1 -- 172.21.15.83:0/3786296065 wait complete. 2024-03-06T07:24:52.091 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json dirfrag split /subdir 0/0 1 2024-03-06T07:24:52.220 DEBUG:tasks.cephfs.filesystem:_json_asok output empty 2024-03-06T07:24:52.220 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:52.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.319+0000 7fa611f5e640 1 Processor -- start 2024-03-06T07:24:52.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.319+0000 7fa611f5e640 1 -- start start 2024-03-06T07:24:52.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.320+0000 7fa611f5e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fa60c0b3060 0x7fa60c15e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.320+0000 7fa611f5e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa60c15bf30 0x7fa60c15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.320+0000 7fa611f5e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa60c15f2d0 con 0x7fa60c13dc10 2024-03-06T07:24:52.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.320+0000 7fa611f5e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa60c0cf3f0 con 0x7fa60c15bf30 2024-03-06T07:24:52.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.320+0000 7fa611f5e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa60c15b210 con 0x7fa60c0b3060 2024-03-06T07:24:52.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.320+0000 7fa60bfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa60c15bf30 0x7fa60c15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.320+0000 7fa60b7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fa60c13dc10 0x7fa60c0b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:33932/0 (socket says 172.21.15.83:33932) 2024-03-06T07:24:52.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.320+0000 7fa60b7fe640 1 -- 172.21.15.83:0/4097009437 learned_addr learned my addr 172.21.15.83:0/4097009437 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:24:52.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.320+0000 7fa603fff640 1 --2- 172.21.15.83:0/4097009437 >> v2:172.21.15.183:3300/0 conn(0x7fa60c0b3060 0x7fa60c15e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.320+0000 7fa60bfff640 1 -- 172.21.15.83:0/4097009437 >> v2:172.21.15.183:3300/0 conn(0x7fa60c0b3060 msgr2=0x7fa60c15e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:52.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.320+0000 7fa60bfff640 1 --2- 172.21.15.83:0/4097009437 >> v2:172.21.15.183:3300/0 conn(0x7fa60c0b3060 0x7fa60c15e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.320+0000 7fa60bfff640 1 -- 172.21.15.83:0/4097009437 >> v1:172.21.15.83:6789/0 conn(0x7fa60c13dc10 legacy=0x7fa60c0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.320+0000 7fa60affd640 1 -- 172.21.15.83:0/4097009437 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2468613114 0 0) 0x7fa60c15f2d0 con 0x7fa60c13dc10 2024-03-06T07:24:52.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.321+0000 7fa60bfff640 1 -- 172.21.15.83:0/4097009437 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa60c15f580 con 0x7fa60c15bf30 2024-03-06T07:24:52.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.321+0000 7fa603fff640 1 --2- 172.21.15.83:0/4097009437 >> v2:172.21.15.183:3300/0 conn(0x7fa60c0b3060 0x7fa60c15e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:52.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.321+0000 7fa60bfff640 1 --2- 172.21.15.83:0/4097009437 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa60c15bf30 0x7fa60c15ed90 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fa60c068700 tx=0x7fa60402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9a6c8a7e2e7ba5e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:52.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.321+0000 7fa60affd640 1 -- 172.21.15.83:0/4097009437 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa604039070 con 0x7fa60c15bf30 2024-03-06T07:24:52.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.321+0000 7fa60affd640 1 -- 172.21.15.83:0/4097009437 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa60402cbf0 con 0x7fa60c15bf30 2024-03-06T07:24:52.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.321+0000 7fa60affd640 1 -- 172.21.15.83:0/4097009437 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa604034440 con 0x7fa60c15bf30 2024-03-06T07:24:52.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.322+0000 7fa611f5e640 1 -- 172.21.15.83:0/4097009437 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa60c15bf30 msgr2=0x7fa60c15ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.322+0000 7fa611f5e640 1 --2- 172.21.15.83:0/4097009437 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa60c15bf30 0x7fa60c15ed90 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fa60c068700 tx=0x7fa60402be40 comp rx=0 tx=0).stop 2024-03-06T07:24:52.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.322+0000 7fa611f5e640 1 -- 172.21.15.83:0/4097009437 shutdown_connections 2024-03-06T07:24:52.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.322+0000 7fa611f5e640 1 --2- 172.21.15.83:0/4097009437 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa60c15bf30 0x7fa60c15ed90 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.322+0000 7fa611f5e640 1 --2- 172.21.15.83:0/4097009437 >> v2:172.21.15.183:3300/0 conn(0x7fa60c0b3060 0x7fa60c15e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.322+0000 7fa611f5e640 1 -- 172.21.15.83:0/4097009437 >> 172.21.15.83:0/4097009437 conn(0x7fa60c0cf8a0 msgr2=0x7fa60c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:52.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.322+0000 7fa611f5e640 1 -- 172.21.15.83:0/4097009437 shutdown_connections 2024-03-06T07:24:52.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.322+0000 7fa611f5e640 1 -- 172.21.15.83:0/4097009437 wait complete. 2024-03-06T07:24:52.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.323+0000 7fa611f5e640 1 Processor -- start 2024-03-06T07:24:52.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.323+0000 7fa611f5e640 1 -- start start 2024-03-06T07:24:52.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.324+0000 7fa611f5e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa60c0b3060 0x7fa60c1f6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.324+0000 7fa611f5e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa60c13dc10 0x7fa60c0cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.324+0000 7fa611f5e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa60c15bf30 0x7fa60c0cdce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.324+0000 7fa611f5e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa60c13dff0 con 0x7fa60c0b3060 2024-03-06T07:24:52.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.324+0000 7fa611f5e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa60c15b730 con 0x7fa60c15bf30 2024-03-06T07:24:52.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.324+0000 7fa611f5e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa60c0708d0 con 0x7fa60c13dc10 2024-03-06T07:24:52.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.324+0000 7fa60b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa60c0b3060 0x7fa60c1f6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.324+0000 7fa60b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa60c0b3060 0x7fa60c1f6a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49314/0 (socket says 172.21.15.83:49314) 2024-03-06T07:24:52.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.324+0000 7fa60b7fe640 1 -- 172.21.15.83:0/1413163211 learned_addr learned my addr 172.21.15.83:0/1413163211 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:52.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.324+0000 7fa603fff640 1 --2- 172.21.15.83:0/1413163211 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa60c13dc10 0x7fa60c0cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.324+0000 7fa60bfff640 1 --2- 172.21.15.83:0/1413163211 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa60c15bf30 0x7fa60c0cdce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.324+0000 7fa60b7fe640 1 -- 172.21.15.83:0/1413163211 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa60c13dc10 msgr2=0x7fa60c0cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.324+0000 7fa60b7fe640 1 --2- 172.21.15.83:0/1413163211 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa60c13dc10 0x7fa60c0cd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.324+0000 7fa60b7fe640 1 -- 172.21.15.83:0/1413163211 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa60c15bf30 msgr2=0x7fa60c0cdce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.324+0000 7fa60b7fe640 1 --2- 172.21.15.83:0/1413163211 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa60c15bf30 0x7fa60c0cdce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.324+0000 7fa60b7fe640 1 -- 172.21.15.83:0/1413163211 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa60c140820 con 0x7fa60c0b3060 2024-03-06T07:24:52.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.324+0000 7fa603fff640 1 --2- 172.21.15.83:0/1413163211 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa60c13dc10 0x7fa60c0cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:52.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.324+0000 7fa60bfff640 1 --2- 172.21.15.83:0/1413163211 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa60c15bf30 0x7fa60c0cdce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:52.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.325+0000 7fa60b7fe640 1 --2- 172.21.15.83:0/1413163211 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa60c0b3060 0x7fa60c1f6a00 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc00c4d0 tx=0x7fa5fc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:52.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.325+0000 7fa608ff9640 1 -- 172.21.15.83:0/1413163211 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5fc018070 con 0x7fa60c0b3060 2024-03-06T07:24:52.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.325+0000 7fa608ff9640 1 -- 172.21.15.83:0/1413163211 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5fc007500 con 0x7fa60c0b3060 2024-03-06T07:24:52.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.325+0000 7fa611f5e640 1 -- 172.21.15.83:0/1413163211 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa60c140b00 con 0x7fa60c0b3060 2024-03-06T07:24:52.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.325+0000 7fa608ff9640 1 -- 172.21.15.83:0/1413163211 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5fc004a20 con 0x7fa60c0b3060 2024-03-06T07:24:52.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.325+0000 7fa611f5e640 1 -- 172.21.15.83:0/1413163211 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa60c141750 con 0x7fa60c0b3060 2024-03-06T07:24:52.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.326+0000 7fa611f5e640 1 -- 172.21.15.83:0/1413163211 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa60c04f330 con 0x7fa60c0b3060 2024-03-06T07:24:52.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.326+0000 7fa608ff9640 1 -- 172.21.15.83:0/1413163211 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa5fc01b070 con 0x7fa60c0b3060 2024-03-06T07:24:52.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.327+0000 7fa608ff9640 1 --2- 172.21.15.83:0/1413163211 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa5ec06b4d0 0x7fa5ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.327+0000 7fa608ff9640 1 -- 172.21.15.83:0/1413163211 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa5fc0894d0 con 0x7fa60c0b3060 2024-03-06T07:24:52.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.327+0000 7fa603fff640 1 --2- 172.21.15.83:0/1413163211 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa5ec06b4d0 0x7fa5ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.327+0000 7fa603fff640 1 --2- 172.21.15.83:0/1413163211 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa5ec06b4d0 0x7fa5ec06d990 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa5f4006fd0 tx=0x7fa5f4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:52.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.331+0000 7fa608ff9640 1 -- 172.21.15.83:0/1413163211 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa5fc0583e0 con 0x7fa60c0b3060 2024-03-06T07:24:52.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.556+0000 7fa611f5e640 1 -- 172.21.15.83:0/1413163211 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa60c06eb00 con 0x7fa60c0b3060 2024-03-06T07:24:52.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.556+0000 7fa608ff9640 1 -- 172.21.15.83:0/1413163211 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4980 (secure 0 0 0) 0x7fa5fc087e20 con 0x7fa60c0b3060 2024-03-06T07:24:52.558 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:24:52.559 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":25,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5665,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/3334609507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3334609507},{"type":"v1","addr":"172.21.15.183:6834","nonce":3334609507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5668,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/691189112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":691189112},{"type":"v1","addr":"172.21.15.183:6835","nonce":691189112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6081,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/3952571284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":3952571284},{"type":"v1","addr":"172.21.15.83:6836","nonce":3952571284}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:19.852157+0000","modified":"2024-03-06T07:24:21.879875+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6084},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6084":{"gid":6084,"name":"a","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/3237506441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3237506441},{"type":"v1","addr":"172.21.15.83:6837","nonce":3237506441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6084,"qdb_cluster":[6084]},"id":4}]} 2024-03-06T07:24:52.559 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 25 2024-03-06T07:24:52.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.559+0000 7fa611f5e640 1 -- 172.21.15.83:0/1413163211 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa5ec06b4d0 msgr2=0x7fa5ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.559+0000 7fa611f5e640 1 --2- 172.21.15.83:0/1413163211 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa5ec06b4d0 0x7fa5ec06d990 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa5f4006fd0 tx=0x7fa5f4008040 comp rx=0 tx=0).stop 2024-03-06T07:24:52.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.559+0000 7fa611f5e640 1 -- 172.21.15.83:0/1413163211 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa60c0b3060 msgr2=0x7fa60c1f6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.559+0000 7fa611f5e640 1 --2- 172.21.15.83:0/1413163211 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa60c0b3060 0x7fa60c1f6a00 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc00c4d0 tx=0x7fa5fc00c990 comp rx=0 tx=0).stop 2024-03-06T07:24:52.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.559+0000 7fa611f5e640 1 -- 172.21.15.83:0/1413163211 shutdown_connections 2024-03-06T07:24:52.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.559+0000 7fa611f5e640 1 --2- 172.21.15.83:0/1413163211 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa5ec06b4d0 0x7fa5ec06d990 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.559+0000 7fa611f5e640 1 --2- 172.21.15.83:0/1413163211 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa60c15bf30 0x7fa60c0cdce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.559+0000 7fa611f5e640 1 --2- 172.21.15.83:0/1413163211 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa60c13dc10 0x7fa60c0cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.559+0000 7fa611f5e640 1 --2- 172.21.15.83:0/1413163211 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa60c0b3060 0x7fa60c1f6a00 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.559+0000 7fa611f5e640 1 -- 172.21.15.83:0/1413163211 >> 172.21.15.83:0/1413163211 conn(0x7fa60c0cf8a0 msgr2=0x7fa60c0b0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:52.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.559+0000 7fa611f5e640 1 -- 172.21.15.83:0/1413163211 shutdown_connections 2024-03-06T07:24:52.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.559+0000 7fa611f5e640 1 -- 172.21.15.83:0/1413163211 wait complete. 2024-03-06T07:24:52.572 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:24:52.704 INFO:teuthology.orchestra.run.smithi083.stdout:{"message":"","return_code":0} 2024-03-06T07:24:52.704 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:24:52.704 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:24:52.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.803+0000 7f7b8a30c640 1 Processor -- start 2024-03-06T07:24:52.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.804+0000 7f7b8a30c640 1 -- start start 2024-03-06T07:24:52.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.804+0000 7f7b8a30c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7b8415cc60 0x7f7b8406afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.804+0000 7f7b8a30c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7b8406b4f0 0x7f7b840d0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.804+0000 7f7b8a30c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b8415d0a0 con 0x7f7b841597c0 2024-03-06T07:24:52.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.804+0000 7f7b8a30c640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b84159ba0 con 0x7f7b8406b4f0 2024-03-06T07:24:52.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.804+0000 7f7b8a30c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8415c7d0 con 0x7f7b8415cc60 2024-03-06T07:24:52.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.804+0000 7f7b888a9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7b8406b4f0 0x7f7b840d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.805+0000 7f7b888a9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7b8406b4f0 0x7f7b840d0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49330/0 (socket says 172.21.15.83:49330) 2024-03-06T07:24:52.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.805+0000 7f7b888a9640 1 -- 172.21.15.83:0/3412417481 learned_addr learned my addr 172.21.15.83:0/3412417481 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:52.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.805+0000 7f7b888a9640 1 -- 172.21.15.83:0/3412417481 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7b8415cc60 msgr2=0x7f7b8406afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:52.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.805+0000 7f7b837fe640 1 --2- 172.21.15.83:0/3412417481 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7b8415cc60 0x7f7b8406afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.805+0000 7f7b888a9640 1 --2- 172.21.15.83:0/3412417481 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7b8415cc60 0x7f7b8406afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.805+0000 7f7b888a9640 1 -- 172.21.15.83:0/3412417481 >> v1:172.21.15.183:6789/0 conn(0x7f7b841597c0 legacy=0x7f7b8406f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:52.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.805+0000 7f7b888a9640 1 -- 172.21.15.83:0/3412417481 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b840cd440 con 0x7f7b8406b4f0 2024-03-06T07:24:52.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.805+0000 7f7b837fe640 1 --2- 172.21.15.83:0/3412417481 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7b8415cc60 0x7f7b8406afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:52.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.806+0000 7f7b888a9640 1 --2- 172.21.15.83:0/3412417481 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b8406b4f0 0x7f7b840d0590 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c009e50 tx=0x7f7b7c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=955310772b724dc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:52.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.806+0000 7f7b82ffd640 1 -- 172.21.15.83:0/3412417481 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c02ca70 con 0x7f7b8406b4f0 2024-03-06T07:24:52.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.806+0000 7f7b82ffd640 1 -- 172.21.15.83:0/3412417481 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b7c02cc10 con 0x7f7b8406b4f0 2024-03-06T07:24:52.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.806+0000 7f7b82ffd640 1 -- 172.21.15.83:0/3412417481 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c0343f0 con 0x7f7b8406b4f0 2024-03-06T07:24:52.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.806+0000 7f7b8a30c640 1 -- 172.21.15.83:0/3412417481 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b8406b4f0 msgr2=0x7f7b840d0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.806+0000 7f7b8a30c640 1 --2- 172.21.15.83:0/3412417481 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b8406b4f0 0x7f7b840d0590 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c009e50 tx=0x7f7b7c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.806+0000 7f7b8a30c640 1 -- 172.21.15.83:0/3412417481 shutdown_connections 2024-03-06T07:24:52.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.806+0000 7f7b8a30c640 1 --2- 172.21.15.83:0/3412417481 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b8406b4f0 0x7f7b840d0590 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.806+0000 7f7b8a30c640 1 --2- 172.21.15.83:0/3412417481 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7b8415cc60 0x7f7b8406afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.806+0000 7f7b8a30c640 1 -- 172.21.15.83:0/3412417481 >> 172.21.15.83:0/3412417481 conn(0x7f7b8406c0c0 msgr2=0x7f7b8406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:52.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.806+0000 7f7b8a30c640 1 -- 172.21.15.83:0/3412417481 shutdown_connections 2024-03-06T07:24:52.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.806+0000 7f7b8a30c640 1 -- 172.21.15.83:0/3412417481 wait complete. 2024-03-06T07:24:52.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.807+0000 7f7b8a30c640 1 Processor -- start 2024-03-06T07:24:52.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.808+0000 7f7b8a30c640 1 -- start start 2024-03-06T07:24:52.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.808+0000 7f7b8a30c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7b8406b4f0 0x7f7b840b2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.808+0000 7f7b8a30c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7b841597c0 0x7f7b8414a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.808+0000 7f7b8a30c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b8415cc60 0x7f7b8415ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.808+0000 7f7b8a30c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8415c150 con 0x7f7b8415cc60 2024-03-06T07:24:52.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.808+0000 7f7b8a30c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8415c460 con 0x7f7b8406b4f0 2024-03-06T07:24:52.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.808+0000 7f7b8a30c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b840cd0c0 con 0x7f7b841597c0 2024-03-06T07:24:52.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.808+0000 7f7b888a9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b8415cc60 0x7f7b8415ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.808+0000 7f7b888a9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b8415cc60 0x7f7b8415ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49338/0 (socket says 172.21.15.83:49338) 2024-03-06T07:24:52.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.808+0000 7f7b888a9640 1 -- 172.21.15.83:0/1848744718 learned_addr learned my addr 172.21.15.83:0/1848744718 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:52.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.809+0000 7f7b837fe640 1 --2- 172.21.15.83:0/1848744718 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7b841597c0 0x7f7b8414a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.809+0000 7f7b83fff640 1 --2- 172.21.15.83:0/1848744718 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7b8406b4f0 0x7f7b840b2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.809+0000 7f7b888a9640 1 -- 172.21.15.83:0/1848744718 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7b841597c0 msgr2=0x7f7b8414a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.809+0000 7f7b888a9640 1 --2- 172.21.15.83:0/1848744718 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7b841597c0 0x7f7b8414a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.809+0000 7f7b888a9640 1 -- 172.21.15.83:0/1848744718 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7b8406b4f0 msgr2=0x7f7b840b2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.809+0000 7f7b888a9640 1 --2- 172.21.15.83:0/1848744718 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7b8406b4f0 0x7f7b840b2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.809+0000 7f7b888a9640 1 -- 172.21.15.83:0/1848744718 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b841f7e50 con 0x7f7b8415cc60 2024-03-06T07:24:52.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.809+0000 7f7b888a9640 1 --2- 172.21.15.83:0/1848744718 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b8415cc60 0x7f7b8415ad80 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c009e50 tx=0x7f7b7c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:52.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.809+0000 7f7b80ff9640 1 -- 172.21.15.83:0/1848744718 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c043020 con 0x7f7b8415cc60 2024-03-06T07:24:52.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.809+0000 7f7b80ff9640 1 -- 172.21.15.83:0/1848744718 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b7c03e450 con 0x7f7b8415cc60 2024-03-06T07:24:52.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.809+0000 7f7b80ff9640 1 -- 172.21.15.83:0/1848744718 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c009510 con 0x7f7b8415cc60 2024-03-06T07:24:52.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.809+0000 7f7b8a30c640 1 -- 172.21.15.83:0/1848744718 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b840b16d0 con 0x7f7b8415cc60 2024-03-06T07:24:52.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.809+0000 7f7b8a30c640 1 -- 172.21.15.83:0/1848744718 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b840b1c90 con 0x7f7b8415cc60 2024-03-06T07:24:52.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.810+0000 7f7b8a30c640 1 -- 172.21.15.83:0/1848744718 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b8404f330 con 0x7f7b8415cc60 2024-03-06T07:24:52.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.810+0000 7f7b80ff9640 1 -- 172.21.15.83:0/1848744718 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7b7c046020 con 0x7f7b8415cc60 2024-03-06T07:24:52.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.811+0000 7f7b80ff9640 1 --2- 172.21.15.83:0/1848744718 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7b7006b4d0 0x7f7b7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.811+0000 7f7b80ff9640 1 -- 172.21.15.83:0/1848744718 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7b7c0aca50 con 0x7f7b8415cc60 2024-03-06T07:24:52.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.811+0000 7f7b83fff640 1 --2- 172.21.15.83:0/1848744718 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7b7006b4d0 0x7f7b7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.811+0000 7f7b83fff640 1 --2- 172.21.15.83:0/1848744718 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7b7006b4d0 0x7f7b7006d990 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f7b74004ea0 tx=0x7f7b74009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:52.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:52.819+0000 7f7b80ff9640 1 -- 172.21.15.83:0/1848744718 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7b7c04d030 con 0x7f7b8415cc60 2024-03-06T07:24:53.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:53.033+0000 7f7b8a30c640 1 -- 172.21.15.83:0/1848744718 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f7b8406c640 con 0x7f7b8415cc60 2024-03-06T07:24:53.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:53.034+0000 7f7b80ff9640 1 -- 172.21.15.83:0/1848744718 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) v1 ==== 53+0+73 (secure 0 0 0) 0x7f7b8415d0a0 con 0x7f7b8415cc60 2024-03-06T07:24:53.036 INFO:teuthology.orchestra.run.smithi083.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:24:53.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:53.035+0000 7f7b8a30c640 1 -- 172.21.15.83:0/1848744718 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7b7006b4d0 msgr2=0x7f7b7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:53.035+0000 7f7b8a30c640 1 --2- 172.21.15.83:0/1848744718 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7b7006b4d0 0x7f7b7006d990 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f7b74004ea0 tx=0x7f7b74009040 comp rx=0 tx=0).stop 2024-03-06T07:24:53.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:53.035+0000 7f7b8a30c640 1 -- 172.21.15.83:0/1848744718 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b8415cc60 msgr2=0x7f7b8415ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:53.035+0000 7f7b8a30c640 1 --2- 172.21.15.83:0/1848744718 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b8415cc60 0x7f7b8415ad80 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c009e50 tx=0x7f7b7c004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:53.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:53.035+0000 7f7b8a30c640 1 -- 172.21.15.83:0/1848744718 shutdown_connections 2024-03-06T07:24:53.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:53.035+0000 7f7b8a30c640 1 --2- 172.21.15.83:0/1848744718 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7b7006b4d0 0x7f7b7006d990 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:53.035+0000 7f7b8a30c640 1 --2- 172.21.15.83:0/1848744718 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b8415cc60 0x7f7b8415ad80 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:53.035+0000 7f7b8a30c640 1 --2- 172.21.15.83:0/1848744718 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7b841597c0 0x7f7b8414a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:53.035+0000 7f7b8a30c640 1 --2- 172.21.15.83:0/1848744718 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7b8406b4f0 0x7f7b840b2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:53.035+0000 7f7b8a30c640 1 -- 172.21.15.83:0/1848744718 >> 172.21.15.83:0/1848744718 conn(0x7f7b8406c0c0 msgr2=0x7f7b8415d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:53.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:53.036+0000 7f7b8a30c640 1 -- 172.21.15.83:0/1848744718 shutdown_connections 2024-03-06T07:24:53.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:53.036+0000 7f7b8a30c640 1 -- 172.21.15.83:0/1848744718 wait complete. 2024-03-06T07:24:53.048 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:24:53.048 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:24:53.049 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi083.front.sepia.ceph.com 2024-03-06T07:24:53.049 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:24:53.049 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:53.049 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:24:53.049 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:24:53.049 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:24:53.049 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:24:53.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:53.049 DEBUG:teuthology.orchestra.run.smithi083:> ip addr 2024-03-06T07:24:53.065 INFO:teuthology.orchestra.run.smithi083.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:24:53.065 INFO:teuthology.orchestra.run.smithi083.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:24:53.065 INFO:teuthology.orchestra.run.smithi083.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:24:53.065 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:53.065 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 ::1/128 scope host 2024-03-06T07:24:53.065 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:53.065 INFO:teuthology.orchestra.run.smithi083.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:24:53.065 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:d8:da:1e brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:53.065 INFO:teuthology.orchestra.run.smithi083.stdout: altname enp7s0f0 2024-03-06T07:24:53.065 INFO:teuthology.orchestra.run.smithi083.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:24:53.065 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:d8:da:1f brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:53.066 INFO:teuthology.orchestra.run.smithi083.stdout: altname enp7s0f1 2024-03-06T07:24:53.066 INFO:teuthology.orchestra.run.smithi083.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:24:53.066 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:8f:ce:90 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:53.066 INFO:teuthology.orchestra.run.smithi083.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:24:53.066 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:8f:ce:91 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:53.066 INFO:teuthology.orchestra.run.smithi083.stdout: inet 172.21.15.83/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:24:53.066 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft 42563sec preferred_lft 42563sec 2024-03-06T07:24:53.066 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::ec4:7aff:fe8f:ce91/64 scope link 2024-03-06T07:24:53.066 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:53.066 INFO:teuthology.orchestra.run.smithi083.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:24:53.066 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 1e:29:46:c9:e7:c8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:53.066 INFO:teuthology.orchestra.run.smithi083.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:24:53.066 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:53.066 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::60c5:d7ff:fe70:2b1c/64 scope link 2024-03-06T07:24:53.066 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:53.067 INFO:teuthology.orchestra.run.smithi083.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:24:53.067 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 1e:29:46:c9:e7:c8 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:53.067 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::1c29:46ff:fec9:e7c8/64 scope link 2024-03-06T07:24:53.067 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:53.067 INFO:teuthology.orchestra.run.smithi083.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:24:53.067 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 3e:00:2f:23:d5:45 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:53.067 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::3c00:2fff:fe23:d545/64 scope link 2024-03-06T07:24:53.067 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:53.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:53.068 DEBUG:teuthology.orchestra.run.smithi083:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:24:53.131 INFO:teuthology.orchestra.run.smithi083.stdout:1 2024-03-06T07:24:53.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:53.133 DEBUG:teuthology.orchestra.run.smithi083:> route 2024-03-06T07:24:53.188 INFO:teuthology.orchestra.run.smithi083.stdout:Kernel IP routing table 2024-03-06T07:24:53.189 INFO:teuthology.orchestra.run.smithi083.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:24:53.189 INFO:teuthology.orchestra.run.smithi083.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:24:53.189 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:24:53.189 INFO:teuthology.orchestra.run.smithi083.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:24:53.189 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:53.189 DEBUG:teuthology.orchestra.run.smithi083:> (cd / && exec bash -c ' 2024-03-06T07:24:53.189 DEBUG:teuthology.orchestra.run.smithi083:> set -e 2024-03-06T07:24:53.189 DEBUG:teuthology.orchestra.run.smithi083:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:24:53.189 DEBUG:teuthology.orchestra.run.smithi083:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:24:53.189 DEBUG:teuthology.orchestra.run.smithi083:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:24:53.190 DEBUG:teuthology.orchestra.run.smithi083:> ') 2024-03-06T07:24:53.284 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:53.285 DEBUG:teuthology.orchestra.run.smithi083:> ip netns list 2024-03-06T07:24:53.299 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:24:53.299 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:24:53.299 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:24:53.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:53.300 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip link set brx.0 up 2024-03-06T07:24:53.367 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:24:53.367 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:24:53.367 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:53.422 INFO:teuthology.orchestra.run.smithi083.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:24:53.423 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:24:53.423 DEBUG:teuthology.orchestra.run.smithi083:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:53.478 DEBUG:teuthology.orchestra.run.smithi083:> sudo modprobe fuse 2024-03-06T07:24:53.544 DEBUG:teuthology.orchestra.run.smithi083:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:24:53.600 INFO:teuthology.orchestra.run.smithi083.stdout:/proc 2024-03-06T07:24:53.600 INFO:teuthology.orchestra.run.smithi083.stdout:/sys 2024-03-06T07:24:53.600 INFO:teuthology.orchestra.run.smithi083.stdout:/dev 2024-03-06T07:24:53.600 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/security 2024-03-06T07:24:53.600 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/shm 2024-03-06T07:24:53.600 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/pts 2024-03-06T07:24:53.600 INFO:teuthology.orchestra.run.smithi083.stdout:/run 2024-03-06T07:24:53.600 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/cgroup 2024-03-06T07:24:53.600 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/pstore 2024-03-06T07:24:53.601 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/bpf 2024-03-06T07:24:53.601 INFO:teuthology.orchestra.run.smithi083.stdout:/ 2024-03-06T07:24:53.601 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/selinux 2024-03-06T07:24:53.601 INFO:teuthology.orchestra.run.smithi083.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:24:53.601 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/mqueue 2024-03-06T07:24:53.601 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/hugepages 2024-03-06T07:24:53.601 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/debug 2024-03-06T07:24:53.601 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/tracing 2024-03-06T07:24:53.601 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:24:53.601 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/fuse/connections 2024-03-06T07:24:53.601 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/config 2024-03-06T07:24:53.601 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:24:53.601 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:24:53.602 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:24:53.602 INFO:teuthology.orchestra.run.smithi083.stdout:/run/user/1000 2024-03-06T07:24:53.602 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph 2024-03-06T07:24:53.602 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:24:53.602 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:24:53.602 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:24:53.602 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:24:53.602 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns 2024-03-06T07:24:53.602 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:53.602 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:53.602 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:53.602 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:53.603 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:53.603 DEBUG:teuthology.orchestra.run.smithi083:> ls /sys/fs/fuse/connections 2024-03-06T07:24:53.655 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:24:53.655 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:24:53.698 DEBUG:teuthology.orchestra.run.smithi083:> sudo modprobe fuse 2024-03-06T07:24:53.729 DEBUG:teuthology.orchestra.run.smithi083:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:24:53.785 INFO:teuthology.orchestra.run.smithi083.stdout:/proc 2024-03-06T07:24:53.786 INFO:teuthology.orchestra.run.smithi083.stdout:/sys 2024-03-06T07:24:53.786 INFO:teuthology.orchestra.run.smithi083.stdout:/dev 2024-03-06T07:24:53.786 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/security 2024-03-06T07:24:53.786 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/shm 2024-03-06T07:24:53.786 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/pts 2024-03-06T07:24:53.786 INFO:teuthology.orchestra.run.smithi083.stdout:/run 2024-03-06T07:24:53.786 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/cgroup 2024-03-06T07:24:53.786 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/pstore 2024-03-06T07:24:53.786 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/bpf 2024-03-06T07:24:53.786 INFO:teuthology.orchestra.run.smithi083.stdout:/ 2024-03-06T07:24:53.786 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/selinux 2024-03-06T07:24:53.786 INFO:teuthology.orchestra.run.smithi083.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:24:53.786 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/mqueue 2024-03-06T07:24:53.787 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/hugepages 2024-03-06T07:24:53.787 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/debug 2024-03-06T07:24:53.787 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/tracing 2024-03-06T07:24:53.787 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:24:53.787 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/fuse/connections 2024-03-06T07:24:53.787 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/config 2024-03-06T07:24:53.787 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:24:53.787 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:24:53.787 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:24:53.787 INFO:teuthology.orchestra.run.smithi083.stdout:/run/user/1000 2024-03-06T07:24:53.787 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph 2024-03-06T07:24:53.787 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:24:53.787 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:24:53.787 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:24:53.788 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:24:53.788 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns 2024-03-06T07:24:53.788 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:53.788 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:53.788 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:53.788 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:53.788 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:53.788 DEBUG:teuthology.orchestra.run.smithi083:> ls /sys/fs/fuse/connections 2024-03-06T07:24:53.799 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:2024-03-06T07:24:53.798+0000 7f8382a3a480 -1 init, newargv = 0x5582bc60f790 newargc=15 2024-03-06T07:24:53.800 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:ceph-fuse[60304]: starting ceph client 2024-03-06T07:24:53.804 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:ceph-fuse[60304]: starting fuse 2024-03-06T07:24:53.815 INFO:teuthology.orchestra.run.smithi083.stdout:41 2024-03-06T07:24:53.816 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-06T07:24:53.816 DEBUG:teuthology.orchestra.run.smithi083:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:24:53.816 DEBUG:teuthology.orchestra.run.smithi083:> import glob 2024-03-06T07:24:53.816 DEBUG:teuthology.orchestra.run.smithi083:> import re 2024-03-06T07:24:53.816 DEBUG:teuthology.orchestra.run.smithi083:> import os 2024-03-06T07:24:53.816 DEBUG:teuthology.orchestra.run.smithi083:> import subprocess 2024-03-06T07:24:53.816 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:24:53.816 DEBUG:teuthology.orchestra.run.smithi083:> def _find_admin_socket(client_name): 2024-03-06T07:24:53.816 DEBUG:teuthology.orchestra.run.smithi083:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:24:53.816 DEBUG:teuthology.orchestra.run.smithi083:> files = glob.glob(asok_path) 2024-03-06T07:24:53.816 DEBUG:teuthology.orchestra.run.smithi083:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:24:53.816 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:24:53.817 DEBUG:teuthology.orchestra.run.smithi083:> # Given a non-glob path, it better be there 2024-03-06T07:24:53.817 DEBUG:teuthology.orchestra.run.smithi083:> if "*" not in asok_path: 2024-03-06T07:24:53.817 DEBUG:teuthology.orchestra.run.smithi083:> assert(len(files) == 1) 2024-03-06T07:24:53.817 DEBUG:teuthology.orchestra.run.smithi083:> return files[0] 2024-03-06T07:24:53.817 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:24:53.817 DEBUG:teuthology.orchestra.run.smithi083:> for f in files: 2024-03-06T07:24:53.817 DEBUG:teuthology.orchestra.run.smithi083:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:24:53.817 DEBUG:teuthology.orchestra.run.smithi083:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:24:53.817 DEBUG:teuthology.orchestra.run.smithi083:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:24:53.817 DEBUG:teuthology.orchestra.run.smithi083:> contents = proc_f.read() 2024-03-06T07:24:53.817 DEBUG:teuthology.orchestra.run.smithi083:> if mountpoint in contents: 2024-03-06T07:24:53.817 DEBUG:teuthology.orchestra.run.smithi083:> return f 2024-03-06T07:24:53.817 DEBUG:teuthology.orchestra.run.smithi083:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:24:53.817 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:24:53.817 DEBUG:teuthology.orchestra.run.smithi083:> print(_find_admin_socket("client.0")) 2024-03-06T07:24:53.818 DEBUG:teuthology.orchestra.run.smithi083:> ' 2024-03-06T07:24:53.940 INFO:teuthology.orchestra.run.smithi083.stdout:/var/run/ceph/ceph-client.0.60304.asok 2024-03-06T07:24:53.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:24:53.950 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.60304.asok 2024-03-06T07:24:53.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:53.951 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.60304.asok status 2024-03-06T07:24:54.087 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:24:54.087 INFO:teuthology.orchestra.run.smithi083.stdout: "metadata": { 2024-03-06T07:24:54.088 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:24:54.088 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:24:54.088 INFO:teuthology.orchestra.run.smithi083.stdout: "entity_id": "0", 2024-03-06T07:24:54.088 INFO:teuthology.orchestra.run.smithi083.stdout: "hostname": "smithi083", 2024-03-06T07:24:54.088 INFO:teuthology.orchestra.run.smithi083.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:24:54.088 INFO:teuthology.orchestra.run.smithi083.stdout: "pid": "60304", 2024-03-06T07:24:54.088 INFO:teuthology.orchestra.run.smithi083.stdout: "root": "/" 2024-03-06T07:24:54.088 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:24:54.088 INFO:teuthology.orchestra.run.smithi083.stdout: "dentry_count": 0, 2024-03-06T07:24:54.088 INFO:teuthology.orchestra.run.smithi083.stdout: "dentry_pinned_count": 0, 2024-03-06T07:24:54.088 INFO:teuthology.orchestra.run.smithi083.stdout: "id": 6360, 2024-03-06T07:24:54.088 INFO:teuthology.orchestra.run.smithi083.stdout: "inst": { 2024-03-06T07:24:54.088 INFO:teuthology.orchestra.run.smithi083.stdout: "name": { 2024-03-06T07:24:54.088 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "client", 2024-03-06T07:24:54.089 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 6360 2024-03-06T07:24:54.089 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:24:54.089 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": { 2024-03-06T07:24:54.089 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-03-06T07:24:54.089 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:24:54.089 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 3371188852 2024-03-06T07:24:54.089 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:24:54.089 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:24:54.089 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": { 2024-03-06T07:24:54.089 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-03-06T07:24:54.089 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:24:54.089 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 3371188852 2024-03-06T07:24:54.089 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:24:54.089 INFO:teuthology.orchestra.run.smithi083.stdout: "inst_str": "client.6360 192.168.0.1:0/3371188852", 2024-03-06T07:24:54.090 INFO:teuthology.orchestra.run.smithi083.stdout: "addr_str": "192.168.0.1:0/3371188852", 2024-03-06T07:24:54.090 INFO:teuthology.orchestra.run.smithi083.stdout: "inode_count": 1, 2024-03-06T07:24:54.090 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_epoch": 25, 2024-03-06T07:24:54.090 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_epoch": 57, 2024-03-06T07:24:54.090 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:24:54.090 INFO:teuthology.orchestra.run.smithi083.stdout: "blocklisted": false, 2024-03-06T07:24:54.090 INFO:teuthology.orchestra.run.smithi083.stdout: "fs_name": "cephfs" 2024-03-06T07:24:54.090 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:24:54.096 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:54.096 DEBUG:teuthology.orchestra.run.smithi083:> stat --file-system '--printf=%T 2024-03-06T07:24:54.096 DEBUG:teuthology.orchestra.run.smithi083:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:54.110 INFO:teuthology.orchestra.run.smithi083.stdout:fuseblk 2024-03-06T07:24:54.111 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:54.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:54.111 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:54.176 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:24:54.176 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:54.231 DEBUG:teuthology.orchestra.run.smithi083:> stdin-killer -- python3 -c ' 2024-03-06T07:24:54.231 DEBUG:teuthology.orchestra.run.smithi083:> import os 2024-03-06T07:24:54.232 DEBUG:teuthology.orchestra.run.smithi083:> import uuid 2024-03-06T07:24:54.232 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:24:54.232 DEBUG:teuthology.orchestra.run.smithi083:> n = 75 2024-03-06T07:24:54.232 DEBUG:teuthology.orchestra.run.smithi083:> create_hard_links = False 2024-03-06T07:24:54.232 DEBUG:teuthology.orchestra.run.smithi083:> if 0 > 0: 2024-03-06T07:24:54.232 DEBUG:teuthology.orchestra.run.smithi083:> create_hard_links = True 2024-03-06T07:24:54.232 DEBUG:teuthology.orchestra.run.smithi083:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:24:54.232 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:24:54.232 DEBUG:teuthology.orchestra.run.smithi083:> dpath = os.path.dirname(path) 2024-03-06T07:24:54.232 DEBUG:teuthology.orchestra.run.smithi083:> fnameprefix = os.path.basename(path) 2024-03-06T07:24:54.232 DEBUG:teuthology.orchestra.run.smithi083:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:24:54.232 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:24:54.232 DEBUG:teuthology.orchestra.run.smithi083:> try: 2024-03-06T07:24:54.232 DEBUG:teuthology.orchestra.run.smithi083:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:24:54.232 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:24:54.233 DEBUG:teuthology.orchestra.run.smithi083:> for i in range(n): 2024-03-06T07:24:54.233 DEBUG:teuthology.orchestra.run.smithi083:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:24:54.233 DEBUG:teuthology.orchestra.run.smithi083:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:24:54.233 DEBUG:teuthology.orchestra.run.smithi083:> f.write(f"{i}") 2024-03-06T07:24:54.233 DEBUG:teuthology.orchestra.run.smithi083:> if False: 2024-03-06T07:24:54.233 DEBUG:teuthology.orchestra.run.smithi083:> f.flush() 2024-03-06T07:24:54.233 DEBUG:teuthology.orchestra.run.smithi083:> os.fsync(f.fileno()) 2024-03-06T07:24:54.233 DEBUG:teuthology.orchestra.run.smithi083:> if False: 2024-03-06T07:24:54.233 DEBUG:teuthology.orchestra.run.smithi083:> os.unlink(fpath) 2024-03-06T07:24:54.233 DEBUG:teuthology.orchestra.run.smithi083:> if False: 2024-03-06T07:24:54.233 DEBUG:teuthology.orchestra.run.smithi083:> os.fsync(dirfd) 2024-03-06T07:24:54.233 DEBUG:teuthology.orchestra.run.smithi083:> if create_hard_links: 2024-03-06T07:24:54.233 DEBUG:teuthology.orchestra.run.smithi083:> for j in range(0): 2024-03-06T07:24:54.233 DEBUG:teuthology.orchestra.run.smithi083:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:24:54.233 DEBUG:teuthology.orchestra.run.smithi083:> if True: 2024-03-06T07:24:54.233 DEBUG:teuthology.orchestra.run.smithi083:> os.fsync(dirfd) 2024-03-06T07:24:54.234 DEBUG:teuthology.orchestra.run.smithi083:> finally: 2024-03-06T07:24:54.234 DEBUG:teuthology.orchestra.run.smithi083:> os.close(dirfd) 2024-03-06T07:24:54.234 DEBUG:teuthology.orchestra.run.smithi083:> ' 2024-03-06T07:24:54.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:24:54.688 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:24:54.689 DEBUG:teuthology.orchestra.run.smithi083:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:24:54.777 DEBUG:teuthology.orchestra.run.smithi183:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:24:54.821 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:24:54.821 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:24:54.821 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:54.836 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:54.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.979+0000 7f86527e9640 1 Processor -- start 2024-03-06T07:24:54.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.979+0000 7f86527e9640 1 -- start start 2024-03-06T07:24:54.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.980+0000 7f86527e9640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f864c15bdd0 0x7f864c0c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.980+0000 7f86527e9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f864c15fa10 0x7f864c0ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.980+0000 7f86527e9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f864c0d16e0 0x7f864c0d1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.980+0000 7f86527e9640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f864c0cebe0 con 0x7f864c15fa10 2024-03-06T07:24:54.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.980+0000 7f86527e9640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f864c0ceec0 con 0x7f864c15bdd0 2024-03-06T07:24:54.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.980+0000 7f86527e9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f864c0cf1a0 con 0x7f864c0d16e0 2024-03-06T07:24:54.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.980+0000 7f864b7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f864c15fa10 0x7f864c0ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.980+0000 7f864b7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f864c15fa10 0x7f864c0ce270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49346/0 (socket says 172.21.15.83:49346) 2024-03-06T07:24:54.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.980+0000 7f864b7fe640 1 -- 172.21.15.83:0/3494689438 learned_addr learned my addr 172.21.15.83:0/3494689438 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:54.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.981+0000 7f864bfff640 1 --2- 172.21.15.83:0/3494689438 >> v2:172.21.15.183:3300/0 conn(0x7f864c15bdd0 0x7f864c0c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.981+0000 7f8650d86640 1 --2- 172.21.15.83:0/3494689438 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f864c0d16e0 0x7f864c0d1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.981+0000 7f864b7fe640 1 -- 172.21.15.83:0/3494689438 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f864c0d16e0 msgr2=0x7f864c0d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:54.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.981+0000 7f864b7fe640 1 --2- 172.21.15.83:0/3494689438 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f864c0d16e0 0x7f864c0d1b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.981+0000 7f864b7fe640 1 -- 172.21.15.83:0/3494689438 >> v2:172.21.15.183:3300/0 conn(0x7f864c15bdd0 msgr2=0x7f864c0c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:54.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.981+0000 7f864b7fe640 1 --2- 172.21.15.83:0/3494689438 >> v2:172.21.15.183:3300/0 conn(0x7f864c15bdd0 0x7f864c0c5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.981+0000 7f864b7fe640 1 -- 172.21.15.83:0/3494689438 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f864c0cc8e0 con 0x7f864c15fa10 2024-03-06T07:24:54.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.981+0000 7f864b7fe640 1 --2- 172.21.15.83:0/3494689438 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f864c15fa10 0x7f864c0ce270 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f864c06e3c0 tx=0x7f863c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d81b830d0c06f6eb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:54.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.981+0000 7f864affd640 1 -- 172.21.15.83:0/3494689438 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f863c02e070 con 0x7f864c15fa10 2024-03-06T07:24:54.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.981+0000 7f864affd640 1 -- 172.21.15.83:0/3494689438 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f863c037070 con 0x7f864c15fa10 2024-03-06T07:24:54.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.981+0000 7f864affd640 1 -- 172.21.15.83:0/3494689438 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f863c031a00 con 0x7f864c15fa10 2024-03-06T07:24:54.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.982+0000 7f86527e9640 1 -- 172.21.15.83:0/3494689438 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f864c15fa10 msgr2=0x7f864c0ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.982+0000 7f86527e9640 1 --2- 172.21.15.83:0/3494689438 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f864c15fa10 0x7f864c0ce270 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f864c06e3c0 tx=0x7f863c02b880 comp rx=0 tx=0).stop 2024-03-06T07:24:54.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.982+0000 7f86527e9640 1 -- 172.21.15.83:0/3494689438 shutdown_connections 2024-03-06T07:24:54.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.982+0000 7f86527e9640 1 --2- 172.21.15.83:0/3494689438 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f864c0d16e0 0x7f864c0d1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.982+0000 7f86527e9640 1 --2- 172.21.15.83:0/3494689438 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f864c15fa10 0x7f864c0ce270 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.982+0000 7f86527e9640 1 --2- 172.21.15.83:0/3494689438 >> v2:172.21.15.183:3300/0 conn(0x7f864c15bdd0 0x7f864c0c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.982+0000 7f86527e9640 1 -- 172.21.15.83:0/3494689438 >> 172.21.15.83:0/3494689438 conn(0x7f864c09a9f0 msgr2=0x7f864c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:54.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.982+0000 7f86527e9640 1 -- 172.21.15.83:0/3494689438 shutdown_connections 2024-03-06T07:24:54.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.982+0000 7f86527e9640 1 -- 172.21.15.83:0/3494689438 wait complete. 2024-03-06T07:24:54.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.983+0000 7f86527e9640 1 Processor -- start 2024-03-06T07:24:54.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.983+0000 7f86527e9640 1 -- start start 2024-03-06T07:24:54.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.984+0000 7f86527e9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f864c0d16e0 0x7f864c150ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.984+0000 7f86527e9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f864c15bdd0 0x7f864c09ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.984+0000 7f86527e9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f864c15fa10 0x7f864c09bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.984+0000 7f86527e9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864c0cc5d0 con 0x7f864c0d16e0 2024-03-06T07:24:54.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.984+0000 7f86527e9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864c0d2050 con 0x7f864c15bdd0 2024-03-06T07:24:54.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.984+0000 7f86527e9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f864c15c1b0 con 0x7f864c15fa10 2024-03-06T07:24:54.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.984+0000 7f864bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f864c0d16e0 0x7f864c150ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.984+0000 7f864bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f864c0d16e0 0x7f864c150ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49354/0 (socket says 172.21.15.83:49354) 2024-03-06T07:24:54.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.984+0000 7f864bfff640 1 -- 172.21.15.83:0/939502116 learned_addr learned my addr 172.21.15.83:0/939502116 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:54.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.984+0000 7f864bfff640 1 -- 172.21.15.83:0/939502116 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f864c15fa10 msgr2=0x7f864c09bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.984+0000 7f864bfff640 1 --2- 172.21.15.83:0/939502116 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f864c15fa10 0x7f864c09bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.984+0000 7f864bfff640 1 -- 172.21.15.83:0/939502116 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f864c15bdd0 msgr2=0x7f864c09ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.984+0000 7f864bfff640 1 --2- 172.21.15.83:0/939502116 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f864c15bdd0 0x7f864c09ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.984+0000 7f864bfff640 1 -- 172.21.15.83:0/939502116 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f864c0caa10 con 0x7f864c0d16e0 2024-03-06T07:24:54.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.985+0000 7f864bfff640 1 --2- 172.21.15.83:0/939502116 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f864c0d16e0 0x7f864c150ee0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f864000c7f0 tx=0x7f864000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:54.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.985+0000 7f8648ff9640 1 -- 172.21.15.83:0/939502116 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8640015070 con 0x7f864c0d16e0 2024-03-06T07:24:54.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.985+0000 7f8648ff9640 1 -- 172.21.15.83:0/939502116 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8640004220 con 0x7f864c0d16e0 2024-03-06T07:24:54.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.985+0000 7f86527e9640 1 -- 172.21.15.83:0/939502116 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f864c0cacf0 con 0x7f864c0d16e0 2024-03-06T07:24:54.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.985+0000 7f8648ff9640 1 -- 172.21.15.83:0/939502116 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8640007620 con 0x7f864c0d16e0 2024-03-06T07:24:54.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.985+0000 7f86527e9640 1 -- 172.21.15.83:0/939502116 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f864c0cb2b0 con 0x7f864c0d16e0 2024-03-06T07:24:54.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.986+0000 7f86527e9640 1 -- 172.21.15.83:0/939502116 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f864c04f330 con 0x7f864c0d16e0 2024-03-06T07:24:54.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.986+0000 7f8648ff9640 1 -- 172.21.15.83:0/939502116 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f86400079d0 con 0x7f864c0d16e0 2024-03-06T07:24:54.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.986+0000 7f8648ff9640 1 --2- 172.21.15.83:0/939502116 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f863806b4d0 0x7f863806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.987+0000 7f8648ff9640 1 -- 172.21.15.83:0/939502116 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f864008a4b0 con 0x7f864c0d16e0 2024-03-06T07:24:54.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.987+0000 7f864b7fe640 1 --2- 172.21.15.83:0/939502116 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f863806b4d0 0x7f863806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.987+0000 7f864b7fe640 1 --2- 172.21.15.83:0/939502116 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f863806b4d0 0x7f863806d990 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f863c036000 tx=0x7f863c030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:54.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:54.991+0000 7f8648ff9640 1 -- 172.21.15.83:0/939502116 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f86400593c0 con 0x7f864c0d16e0 2024-03-06T07:24:55.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.165+0000 7f86527e9640 1 -- 172.21.15.83:0/939502116 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f864c0b7a40 con 0x7f864c0d16e0 2024-03-06T07:24:55.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.166+0000 7f8648ff9640 1 -- 172.21.15.83:0/939502116 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+16677 (secure 0 0 0) 0x7f8640058cc0 con 0x7f864c0d16e0 2024-03-06T07:24:55.168 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:24:55.169 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":57,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:24:21.525937+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:18.074366+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:18.831101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:55.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.168+0000 7f86527e9640 1 -- 172.21.15.83:0/939502116 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f863806b4d0 msgr2=0x7f863806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.168+0000 7f86527e9640 1 --2- 172.21.15.83:0/939502116 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f863806b4d0 0x7f863806d990 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f863c036000 tx=0x7f863c030000 comp rx=0 tx=0).stop 2024-03-06T07:24:55.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.168+0000 7f86527e9640 1 -- 172.21.15.83:0/939502116 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f864c0d16e0 msgr2=0x7f864c150ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.168+0000 7f86527e9640 1 --2- 172.21.15.83:0/939502116 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f864c0d16e0 0x7f864c150ee0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f864000c7f0 tx=0x7f864000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.168+0000 7f86527e9640 1 -- 172.21.15.83:0/939502116 shutdown_connections 2024-03-06T07:24:55.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.168+0000 7f86527e9640 1 --2- 172.21.15.83:0/939502116 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f863806b4d0 0x7f863806d990 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.168+0000 7f86527e9640 1 --2- 172.21.15.83:0/939502116 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f864c15fa10 0x7f864c09bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.168+0000 7f86527e9640 1 --2- 172.21.15.83:0/939502116 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f864c15bdd0 0x7f864c09ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.168+0000 7f86527e9640 1 --2- 172.21.15.83:0/939502116 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f864c0d16e0 0x7f864c150ee0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.168+0000 7f86527e9640 1 -- 172.21.15.83:0/939502116 >> 172.21.15.83:0/939502116 conn(0x7f864c09a9f0 msgr2=0x7f864c09add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:55.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.169+0000 7f86527e9640 1 -- 172.21.15.83:0/939502116 shutdown_connections 2024-03-06T07:24:55.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.169+0000 7f86527e9640 1 -- 172.21.15.83:0/939502116 wait complete. 2024-03-06T07:24:55.181 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:24:18.074366+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:24:18.831101+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:55.182 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:55.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.288+0000 7f5dc75f7640 1 Processor -- start 2024-03-06T07:24:55.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.288+0000 7f5dc75f7640 1 -- start start 2024-03-06T07:24:55.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.289+0000 7f5dc75f7640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f5dc01614d0 0x7f5dc00d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.289+0000 7f5dc75f7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5dc00d1910 0x7f5dc00d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.289+0000 7f5dc75f7640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc00d2740 con 0x7f5dc00d22b0 2024-03-06T07:24:55.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.289+0000 7f5dc75f7640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dc01618b0 con 0x7f5dc01614d0 2024-03-06T07:24:55.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.289+0000 7f5dc75f7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc00cf400 con 0x7f5dc00d1910 2024-03-06T07:24:55.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.289+0000 7f5dc4b92640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5dc00d1910 0x7f5dc00d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.289+0000 7f5dc5393640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f5dc01614d0 0x7f5dc00d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.289+0000 7f5dc5b94640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f5dc00d22b0 0x7f5dc00d06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:52214/0 (socket says 172.21.15.83:52214) 2024-03-06T07:24:55.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.289+0000 7f5dc5393640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f5dc01614d0 0x7f5dc00d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:60112/0 (socket says 172.21.15.83:60112) 2024-03-06T07:24:55.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.289+0000 7f5dc5b94640 1 -- 172.21.15.83:0/1029495898 learned_addr learned my addr 172.21.15.83:0/1029495898 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:24:55.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.289+0000 7f5dc4b92640 1 --2- 172.21.15.83:0/1029495898 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5dc00d1910 0x7f5dc00d1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:57682/0 (socket says 172.21.15.83:57682) 2024-03-06T07:24:55.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.289+0000 7f5dc5393640 1 -- 172.21.15.83:0/1029495898 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5dc00d1910 msgr2=0x7f5dc00d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:55.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.289+0000 7f5dc5393640 1 --2- 172.21.15.83:0/1029495898 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5dc00d1910 0x7f5dc00d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.289+0000 7f5dc5393640 1 -- 172.21.15.83:0/1029495898 >> v1:172.21.15.183:6789/0 conn(0x7f5dc00d22b0 legacy=0x7f5dc00d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.289+0000 7f5db7fff640 1 -- 172.21.15.83:0/1029495898 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 416119144 0 0) 0x7f5db8001490 con 0x7f5dc00d22b0 2024-03-06T07:24:55.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.289+0000 7f5dc5393640 1 -- 172.21.15.83:0/1029495898 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc00cf6a0 con 0x7f5dc01614d0 2024-03-06T07:24:55.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.289+0000 7f5dc4b92640 1 --2- 172.21.15.83:0/1029495898 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5dc00d1910 0x7f5dc00d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:55.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.291+0000 7f5dc5393640 1 --2- 172.21.15.83:0/1029495898 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5dc01614d0 0x7f5dc00d4860 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc0097a0 tx=0x7f5dbc02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1fb8f410d085f22 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:55.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.291+0000 7f5db7fff640 1 -- 172.21.15.83:0/1029495898 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc035020 con 0x7f5dc01614d0 2024-03-06T07:24:55.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.291+0000 7f5db7fff640 1 -- 172.21.15.83:0/1029495898 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dbc031830 con 0x7f5dc01614d0 2024-03-06T07:24:55.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.291+0000 7f5db7fff640 1 -- 172.21.15.83:0/1029495898 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc031c00 con 0x7f5dc01614d0 2024-03-06T07:24:55.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.291+0000 7f5dc75f7640 1 -- 172.21.15.83:0/1029495898 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5dc01614d0 msgr2=0x7f5dc00d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.291+0000 7f5dc75f7640 1 --2- 172.21.15.83:0/1029495898 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5dc01614d0 0x7f5dc00d4860 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc0097a0 tx=0x7f5dbc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.291+0000 7f5dc75f7640 1 -- 172.21.15.83:0/1029495898 shutdown_connections 2024-03-06T07:24:55.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.291+0000 7f5dc75f7640 1 --2- 172.21.15.83:0/1029495898 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5dc00d1910 0x7f5dc00d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.291+0000 7f5dc75f7640 1 --2- 172.21.15.83:0/1029495898 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5dc01614d0 0x7f5dc00d4860 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.291+0000 7f5dc75f7640 1 -- 172.21.15.83:0/1029495898 >> 172.21.15.83:0/1029495898 conn(0x7f5dc00d3d40 msgr2=0x7f5dc00d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:55.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.291+0000 7f5dc75f7640 1 -- 172.21.15.83:0/1029495898 shutdown_connections 2024-03-06T07:24:55.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.291+0000 7f5dc75f7640 1 -- 172.21.15.83:0/1029495898 wait complete. 2024-03-06T07:24:55.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.292+0000 7f5dc75f7640 1 Processor -- start 2024-03-06T07:24:55.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.292+0000 7f5dc75f7640 1 -- start start 2024-03-06T07:24:55.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.293+0000 7f5dc75f7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5dc00d1910 0x7f5dc00b6b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.293+0000 7f5dc75f7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5dc00d22b0 0x7f5dc0150950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.293+0000 7f5dc75f7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5dc01614d0 0x7f5dc0150e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.293+0000 7f5dc75f7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc00b0910 con 0x7f5dc00d22b0 2024-03-06T07:24:55.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.293+0000 7f5dc75f7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc00cfc60 con 0x7f5dc00d1910 2024-03-06T07:24:55.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.293+0000 7f5dc75f7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc00b1e40 con 0x7f5dc01614d0 2024-03-06T07:24:55.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.293+0000 7f5dc4b92640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5dc00d22b0 0x7f5dc0150950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.293+0000 7f5dc4b92640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5dc00d22b0 0x7f5dc0150950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49370/0 (socket says 172.21.15.83:49370) 2024-03-06T07:24:55.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.293+0000 7f5dc4b92640 1 -- 172.21.15.83:0/2519262749 learned_addr learned my addr 172.21.15.83:0/2519262749 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:55.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.293+0000 7f5dc5b94640 1 --2- 172.21.15.83:0/2519262749 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5dc01614d0 0x7f5dc0150e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.293+0000 7f5dc5393640 1 --2- 172.21.15.83:0/2519262749 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5dc00d1910 0x7f5dc00b6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.294+0000 7f5dc4b92640 1 -- 172.21.15.83:0/2519262749 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5dc01614d0 msgr2=0x7f5dc0150e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.294+0000 7f5dc4b92640 1 --2- 172.21.15.83:0/2519262749 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5dc01614d0 0x7f5dc0150e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.294+0000 7f5dc4b92640 1 -- 172.21.15.83:0/2519262749 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5dc00d1910 msgr2=0x7f5dc00b6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.294+0000 7f5dc4b92640 1 --2- 172.21.15.83:0/2519262749 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5dc00d1910 0x7f5dc00b6b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.294+0000 7f5dc4b92640 1 -- 172.21.15.83:0/2519262749 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc00b9700 con 0x7f5dc00d22b0 2024-03-06T07:24:55.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.294+0000 7f5dc5393640 1 --2- 172.21.15.83:0/2519262749 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5dc00d1910 0x7f5dc00b6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:55.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.294+0000 7f5dc4b92640 1 --2- 172.21.15.83:0/2519262749 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5dc00d22b0 0x7f5dc0150950 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f5db000ebe0 tx=0x7f5db000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:55.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.294+0000 7f5db5ffb640 1 -- 172.21.15.83:0/2519262749 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db0017070 con 0x7f5dc00d22b0 2024-03-06T07:24:55.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.294+0000 7f5db5ffb640 1 -- 172.21.15.83:0/2519262749 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5db000cc30 con 0x7f5dc00d22b0 2024-03-06T07:24:55.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.294+0000 7f5dc75f7640 1 -- 172.21.15.83:0/2519262749 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dc00b99e0 con 0x7f5dc00d22b0 2024-03-06T07:24:55.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.294+0000 7f5db5ffb640 1 -- 172.21.15.83:0/2519262749 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db0013470 con 0x7f5dc00d22b0 2024-03-06T07:24:55.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.294+0000 7f5dc75f7640 1 -- 172.21.15.83:0/2519262749 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dc00cc140 con 0x7f5dc00d22b0 2024-03-06T07:24:55.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.295+0000 7f5dc75f7640 1 -- 172.21.15.83:0/2519262749 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dc0068ff0 con 0x7f5dc00d22b0 2024-03-06T07:24:55.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.296+0000 7f5db5ffb640 1 -- 172.21.15.83:0/2519262749 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5db001a070 con 0x7f5dc00d22b0 2024-03-06T07:24:55.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.296+0000 7f5db5ffb640 1 --2- 172.21.15.83:0/2519262749 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5da406b4d0 0x7f5da406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.297+0000 7f5db5ffb640 1 -- 172.21.15.83:0/2519262749 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5db008a450 con 0x7f5dc00d22b0 2024-03-06T07:24:55.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.297+0000 7f5dc5393640 1 --2- 172.21.15.83:0/2519262749 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5da406b4d0 0x7f5da406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.297+0000 7f5dc5393640 1 --2- 172.21.15.83:0/2519262749 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5da406b4d0 0x7f5da406d990 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc036000 tx=0x7f5dbc03c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:55.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.305+0000 7f5db5ffb640 1 -- 172.21.15.83:0/2519262749 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5db0058c80 con 0x7f5dc00d22b0 2024-03-06T07:24:55.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.479+0000 7f5dc75f7640 1 -- 172.21.15.83:0/2519262749 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5dc006a090 con 0x7f5dc00d22b0 2024-03-06T07:24:55.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.480+0000 7f5db5ffb640 1 -- 172.21.15.83:0/2519262749 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v57) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5db005cd20 con 0x7f5dc00d22b0 2024-03-06T07:24:55.482 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:24:55.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.482+0000 7f5dc75f7640 1 -- 172.21.15.83:0/2519262749 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5da406b4d0 msgr2=0x7f5da406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.482+0000 7f5dc75f7640 1 --2- 172.21.15.83:0/2519262749 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5da406b4d0 0x7f5da406d990 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc036000 tx=0x7f5dbc03c040 comp rx=0 tx=0).stop 2024-03-06T07:24:55.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.482+0000 7f5dc75f7640 1 -- 172.21.15.83:0/2519262749 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5dc00d22b0 msgr2=0x7f5dc0150950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.482+0000 7f5dc75f7640 1 --2- 172.21.15.83:0/2519262749 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5dc00d22b0 0x7f5dc0150950 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f5db000ebe0 tx=0x7f5db000c520 comp rx=0 tx=0).stop 2024-03-06T07:24:55.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.482+0000 7f5dc75f7640 1 -- 172.21.15.83:0/2519262749 shutdown_connections 2024-03-06T07:24:55.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.482+0000 7f5dc75f7640 1 --2- 172.21.15.83:0/2519262749 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5da406b4d0 0x7f5da406d990 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.482+0000 7f5dc75f7640 1 --2- 172.21.15.83:0/2519262749 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5dc01614d0 0x7f5dc0150e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.482+0000 7f5dc75f7640 1 --2- 172.21.15.83:0/2519262749 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5dc00d22b0 0x7f5dc0150950 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.482+0000 7f5dc75f7640 1 --2- 172.21.15.83:0/2519262749 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5dc00d1910 0x7f5dc00b6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.482+0000 7f5dc75f7640 1 -- 172.21.15.83:0/2519262749 >> 172.21.15.83:0/2519262749 conn(0x7f5dc00d3d40 msgr2=0x7f5dc00cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:55.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.482+0000 7f5dc75f7640 1 -- 172.21.15.83:0/2519262749 shutdown_connections 2024-03-06T07:24:55.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.482+0000 7f5dc75f7640 1 -- 172.21.15.83:0/2519262749 wait complete. 2024-03-06T07:24:55.498 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:24:55.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.599+0000 7f873f73e640 1 Processor -- start 2024-03-06T07:24:55.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.600+0000 7f873f73e640 1 -- start start 2024-03-06T07:24:55.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.600+0000 7f873f73e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f87380d10c0 0x7f87380d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.600+0000 7f873f73e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f87380d14a0 0x7f873815e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.600+0000 7f873f73e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f873815eb00 0x7f87380cae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.601+0000 7f873f73e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f873815c050 con 0x7f87380d10c0 2024-03-06T07:24:55.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.601+0000 7f873f73e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87380d0060 con 0x7f873815eb00 2024-03-06T07:24:55.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.601+0000 7f873f73e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87380cfa50 con 0x7f87380d14a0 2024-03-06T07:24:55.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.601+0000 7f873d4da640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f87380d10c0 0x7f87380d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.601+0000 7f873d4da640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f87380d10c0 0x7f87380d04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49384/0 (socket says 172.21.15.83:49384) 2024-03-06T07:24:55.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.601+0000 7f873d4da640 1 -- 172.21.15.83:0/1560949903 learned_addr learned my addr 172.21.15.83:0/1560949903 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:55.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.601+0000 7f873d4da640 1 -- 172.21.15.83:0/1560949903 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f87380d14a0 msgr2=0x7f873815e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:55.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.601+0000 7f873ccd9640 1 --2- 172.21.15.83:0/1560949903 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f87380d14a0 0x7f873815e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.601+0000 7f873dcdb640 1 --2- 172.21.15.83:0/1560949903 >> v2:172.21.15.183:3300/0 conn(0x7f873815eb00 0x7f87380cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.601+0000 7f873d4da640 1 --2- 172.21.15.83:0/1560949903 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f87380d14a0 0x7f873815e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.601+0000 7f873d4da640 1 -- 172.21.15.83:0/1560949903 >> v2:172.21.15.183:3300/0 conn(0x7f873815eb00 msgr2=0x7f87380cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:55.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.601+0000 7f873d4da640 1 --2- 172.21.15.83:0/1560949903 >> v2:172.21.15.183:3300/0 conn(0x7f873815eb00 0x7f87380cae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.601+0000 7f873d4da640 1 -- 172.21.15.83:0/1560949903 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87380d0a30 con 0x7f87380d10c0 2024-03-06T07:24:55.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.602+0000 7f873d4da640 1 --2- 172.21.15.83:0/1560949903 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87380d10c0 0x7f87380d04f0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f8734009870 tx=0x7f873402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99353be3870b22f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:55.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.602+0000 7f872ffff640 1 -- 172.21.15.83:0/1560949903 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8734035020 con 0x7f87380d10c0 2024-03-06T07:24:55.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.602+0000 7f872ffff640 1 -- 172.21.15.83:0/1560949903 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8734031640 con 0x7f87380d10c0 2024-03-06T07:24:55.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.602+0000 7f872ffff640 1 -- 172.21.15.83:0/1560949903 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873402d040 con 0x7f87380d10c0 2024-03-06T07:24:55.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.602+0000 7f873f73e640 1 -- 172.21.15.83:0/1560949903 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87380d10c0 msgr2=0x7f87380d04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.602+0000 7f873f73e640 1 --2- 172.21.15.83:0/1560949903 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87380d10c0 0x7f87380d04f0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f8734009870 tx=0x7f873402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.602+0000 7f873f73e640 1 -- 172.21.15.83:0/1560949903 shutdown_connections 2024-03-06T07:24:55.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.602+0000 7f873f73e640 1 --2- 172.21.15.83:0/1560949903 >> v2:172.21.15.183:3300/0 conn(0x7f873815eb00 0x7f87380cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.602+0000 7f873f73e640 1 --2- 172.21.15.83:0/1560949903 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f87380d14a0 0x7f873815e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.602+0000 7f873f73e640 1 --2- 172.21.15.83:0/1560949903 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87380d10c0 0x7f87380d04f0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.602+0000 7f873f73e640 1 -- 172.21.15.83:0/1560949903 >> 172.21.15.83:0/1560949903 conn(0x7f87380b3800 msgr2=0x7f87380b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:55.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.603+0000 7f873f73e640 1 -- 172.21.15.83:0/1560949903 shutdown_connections 2024-03-06T07:24:55.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.603+0000 7f873f73e640 1 -- 172.21.15.83:0/1560949903 wait complete. 2024-03-06T07:24:55.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.604+0000 7f873f73e640 1 Processor -- start 2024-03-06T07:24:55.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.604+0000 7f873f73e640 1 -- start start 2024-03-06T07:24:55.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.605+0000 7f873f73e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87380d10c0 0x7f8738152b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.605+0000 7f873f73e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f87380d14a0 0x7f87380b5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.606+0000 7f873f73e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f873815eb00 0x7f87380b6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.606+0000 7f873f73e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873815f440 con 0x7f87380d10c0 2024-03-06T07:24:55.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.606+0000 7f873f73e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873815f160 con 0x7f87380d14a0 2024-03-06T07:24:55.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.606+0000 7f873f73e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87380cfcf0 con 0x7f873815eb00 2024-03-06T07:24:55.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.606+0000 7f873d4da640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87380d10c0 0x7f8738152b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.606+0000 7f873d4da640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87380d10c0 0x7f8738152b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49388/0 (socket says 172.21.15.83:49388) 2024-03-06T07:24:55.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.606+0000 7f873d4da640 1 -- 172.21.15.83:0/2404762713 learned_addr learned my addr 172.21.15.83:0/2404762713 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:55.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.606+0000 7f873d4da640 1 -- 172.21.15.83:0/2404762713 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f873815eb00 msgr2=0x7f87380b6280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.606+0000 7f873d4da640 1 --2- 172.21.15.83:0/2404762713 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f873815eb00 0x7f87380b6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.606+0000 7f873ccd9640 1 --2- 172.21.15.83:0/2404762713 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f87380d14a0 0x7f87380b5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.606+0000 7f873d4da640 1 -- 172.21.15.83:0/2404762713 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f87380d14a0 msgr2=0x7f87380b5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.606+0000 7f873d4da640 1 --2- 172.21.15.83:0/2404762713 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f87380d14a0 0x7f87380b5300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.606+0000 7f873d4da640 1 -- 172.21.15.83:0/2404762713 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8738153050 con 0x7f87380d10c0 2024-03-06T07:24:55.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.606+0000 7f873d4da640 1 --2- 172.21.15.83:0/2404762713 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87380d10c0 0x7f8738152b10 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f8734000c00 tx=0x7f87340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:55.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.606+0000 7f872dffb640 1 -- 172.21.15.83:0/2404762713 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8734035040 con 0x7f87380d10c0 2024-03-06T07:24:55.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.606+0000 7f872dffb640 1 -- 172.21.15.83:0/2404762713 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8734039c10 con 0x7f87380d10c0 2024-03-06T07:24:55.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.606+0000 7f873f73e640 1 -- 172.21.15.83:0/2404762713 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87380b5b50 con 0x7f87380d10c0 2024-03-06T07:24:55.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.606+0000 7f872dffb640 1 -- 172.21.15.83:0/2404762713 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8734045480 con 0x7f87380d10c0 2024-03-06T07:24:55.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.607+0000 7f873f73e640 1 -- 172.21.15.83:0/2404762713 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87380cc180 con 0x7f87380d10c0 2024-03-06T07:24:55.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.608+0000 7f873f73e640 1 -- 172.21.15.83:0/2404762713 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f873809fe60 con 0x7f87380d10c0 2024-03-06T07:24:55.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.608+0000 7f872dffb640 1 -- 172.21.15.83:0/2404762713 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8734038020 con 0x7f87380d10c0 2024-03-06T07:24:55.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.609+0000 7f872dffb640 1 --2- 172.21.15.83:0/2404762713 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f871c06b500 0x7f871c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.609+0000 7f873ccd9640 1 --2- 172.21.15.83:0/2404762713 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f871c06b500 0x7f871c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.609+0000 7f872dffb640 1 -- 172.21.15.83:0/2404762713 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f87340ac100 con 0x7f87380d10c0 2024-03-06T07:24:55.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.609+0000 7f873ccd9640 1 --2- 172.21.15.83:0/2404762713 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f871c06b500 0x7f871c06d9c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f87380ce2b0 tx=0x7f8728008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:55.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.617+0000 7f872dffb640 1 -- 172.21.15.83:0/2404762713 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f873402f300 con 0x7f87380d10c0 2024-03-06T07:24:55.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.793+0000 7f873f73e640 1 -- 172.21.15.83:0/2404762713 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8738068ff0 con 0x7f87380d10c0 2024-03-06T07:24:55.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.794+0000 7f872dffb640 1 -- 172.21.15.83:0/2404762713 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v57) v1 ==== 104+0+11 (secure 0 0 0) 0x7f873407b010 con 0x7f87380d10c0 2024-03-06T07:24:55.796 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-03-06T07:24:55.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.796+0000 7f873f73e640 1 -- 172.21.15.83:0/2404762713 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f871c06b500 msgr2=0x7f871c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.796+0000 7f873f73e640 1 --2- 172.21.15.83:0/2404762713 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f871c06b500 0x7f871c06d9c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f87380ce2b0 tx=0x7f8728008040 comp rx=0 tx=0).stop 2024-03-06T07:24:55.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.796+0000 7f873f73e640 1 -- 172.21.15.83:0/2404762713 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87380d10c0 msgr2=0x7f8738152b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.796+0000 7f873f73e640 1 --2- 172.21.15.83:0/2404762713 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87380d10c0 0x7f8738152b10 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f8734000c00 tx=0x7f87340046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.796+0000 7f873f73e640 1 -- 172.21.15.83:0/2404762713 shutdown_connections 2024-03-06T07:24:55.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.796+0000 7f873f73e640 1 --2- 172.21.15.83:0/2404762713 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f871c06b500 0x7f871c06d9c0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.796+0000 7f873f73e640 1 --2- 172.21.15.83:0/2404762713 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f873815eb00 0x7f87380b6280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.796+0000 7f873f73e640 1 --2- 172.21.15.83:0/2404762713 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f87380d14a0 0x7f87380b5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.796+0000 7f873f73e640 1 --2- 172.21.15.83:0/2404762713 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87380d10c0 0x7f8738152b10 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.796+0000 7f873f73e640 1 -- 172.21.15.83:0/2404762713 >> 172.21.15.83:0/2404762713 conn(0x7f87380b3800 msgr2=0x7f87380b0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:55.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.796+0000 7f873f73e640 1 -- 172.21.15.83:0/2404762713 shutdown_connections 2024-03-06T07:24:55.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.796+0000 7f873f73e640 1 -- 172.21.15.83:0/2404762713 wait complete. 2024-03-06T07:24:55.809 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:24:55.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.909+0000 7fcdee0ed640 1 Processor -- start 2024-03-06T07:24:55.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.909+0000 7fcdee0ed640 1 -- start start 2024-03-06T07:24:55.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.910+0000 7fcdee0ed640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcde80d4860 0x7fcde80d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.910+0000 7fcdee0ed640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcde81618b0 con 0x7fcde81614d0 2024-03-06T07:24:55.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.910+0000 7fcdee0ed640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcde80cf400 con 0x7fcde80b0e30 2024-03-06T07:24:55.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.910+0000 7fcdee0ed640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcde80d4cf0 con 0x7fcde80d4860 2024-03-06T07:24:55.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.910+0000 7fcde6ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcde80d4860 0x7fcde80d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.910+0000 7fcde6ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcde80d4860 0x7fcde80d2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49398/0 (socket says 172.21.15.83:49398) 2024-03-06T07:24:55.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.910+0000 7fcde6ffd640 1 -- 172.21.15.83:0/3944222802 learned_addr learned my addr 172.21.15.83:0/3944222802 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:55.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.910+0000 7fcde77fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fcde81614d0 0x7fcde80d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:33934/0 (socket says 172.21.15.83:33934) 2024-03-06T07:24:55.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.911+0000 7fcde6ffd640 1 -- 172.21.15.83:0/3944222802 >> v1:172.21.15.183:6789/0 conn(0x7fcde80b0e30 legacy=0x7fcde80b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:55.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.911+0000 7fcde6ffd640 1 -- 172.21.15.83:0/3944222802 >> v1:172.21.15.83:6789/0 conn(0x7fcde81614d0 legacy=0x7fcde80d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.911+0000 7fcde67fc640 1 -- 172.21.15.83:0/3944222802 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1328354236 0 0) 0x7fcde81618b0 con 0x7fcde81614d0 2024-03-06T07:24:55.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.911+0000 7fcde6ffd640 1 -- 172.21.15.83:0/3944222802 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde80cf6a0 con 0x7fcde80d4860 2024-03-06T07:24:55.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.911+0000 7fcde6ffd640 1 --2- 172.21.15.83:0/3944222802 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcde80d4860 0x7fcde80d2020 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8009870 tx=0x7fcdd802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbb88a73d234c2fd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:55.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.912+0000 7fcde67fc640 1 -- 172.21.15.83:0/3944222802 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd8034070 con 0x7fcde80d4860 2024-03-06T07:24:55.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.912+0000 7fcde67fc640 1 -- 172.21.15.83:0/3944222802 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdd8030860 con 0x7fcde80d4860 2024-03-06T07:24:55.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.912+0000 7fcde67fc640 1 -- 172.21.15.83:0/3944222802 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd8030c30 con 0x7fcde80d4860 2024-03-06T07:24:55.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.912+0000 7fcdee0ed640 1 -- 172.21.15.83:0/3944222802 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcde80d4860 msgr2=0x7fcde80d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.912+0000 7fcdee0ed640 1 --2- 172.21.15.83:0/3944222802 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcde80d4860 0x7fcde80d2020 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8009870 tx=0x7fcdd802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.912+0000 7fcdee0ed640 1 -- 172.21.15.83:0/3944222802 shutdown_connections 2024-03-06T07:24:55.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.912+0000 7fcdee0ed640 1 --2- 172.21.15.83:0/3944222802 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcde80d4860 0x7fcde80d2020 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.912+0000 7fcdee0ed640 1 -- 172.21.15.83:0/3944222802 >> 172.21.15.83:0/3944222802 conn(0x7fcde80d3d40 msgr2=0x7fcde80d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:55.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.912+0000 7fcdee0ed640 1 -- 172.21.15.83:0/3944222802 shutdown_connections 2024-03-06T07:24:55.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.912+0000 7fcdee0ed640 1 -- 172.21.15.83:0/3944222802 wait complete. 2024-03-06T07:24:55.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.913+0000 7fcdee0ed640 1 Processor -- start 2024-03-06T07:24:55.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.913+0000 7fcdee0ed640 1 -- start start 2024-03-06T07:24:55.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.915+0000 7fcdee0ed640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcde80b0e30 0x7fcde8157410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.915+0000 7fcdee0ed640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcde80d4860 0x7fcde81f3b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.915+0000 7fcdee0ed640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcde81614d0 0x7fcde80cd180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.915+0000 7fcdee0ed640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde80b0910 con 0x7fcde80b0e30 2024-03-06T07:24:55.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.915+0000 7fcdee0ed640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde80cfc60 con 0x7fcde80d4860 2024-03-06T07:24:55.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.915+0000 7fcdee0ed640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcde80d2560 con 0x7fcde81614d0 2024-03-06T07:24:55.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.915+0000 7fcde77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcde80b0e30 0x7fcde8157410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.915+0000 7fcde77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcde80b0e30 0x7fcde8157410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49400/0 (socket says 172.21.15.83:49400) 2024-03-06T07:24:55.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.915+0000 7fcde77fe640 1 -- 172.21.15.83:0/148958426 learned_addr learned my addr 172.21.15.83:0/148958426 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:55.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.915+0000 7fcde77fe640 1 -- 172.21.15.83:0/148958426 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcde81614d0 msgr2=0x7fcde80cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.915+0000 7fcde6ffd640 1 --2- 172.21.15.83:0/148958426 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcde80d4860 0x7fcde81f3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.915+0000 7fcde7fff640 1 --2- 172.21.15.83:0/148958426 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcde81614d0 0x7fcde80cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.915+0000 7fcde77fe640 1 --2- 172.21.15.83:0/148958426 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcde81614d0 0x7fcde80cd180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.915+0000 7fcde77fe640 1 -- 172.21.15.83:0/148958426 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcde80d4860 msgr2=0x7fcde81f3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.915+0000 7fcde77fe640 1 --2- 172.21.15.83:0/148958426 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcde80d4860 0x7fcde81f3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.915+0000 7fcde77fe640 1 -- 172.21.15.83:0/148958426 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde81f4580 con 0x7fcde80b0e30 2024-03-06T07:24:55.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.915+0000 7fcde6ffd640 1 --2- 172.21.15.83:0/148958426 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcde80d4860 0x7fcde81f3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:55.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.915+0000 7fcde77fe640 1 --2- 172.21.15.83:0/148958426 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcde80b0e30 0x7fcde8157410 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fcddc00c440 tx=0x7fcddc00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:55.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.915+0000 7fcdcffff640 1 -- 172.21.15.83:0/148958426 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcddc018070 con 0x7fcde80b0e30 2024-03-06T07:24:55.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.915+0000 7fcdcffff640 1 -- 172.21.15.83:0/148958426 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcddc0074a0 con 0x7fcde80b0e30 2024-03-06T07:24:55.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.915+0000 7fcdcffff640 1 -- 172.21.15.83:0/148958426 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcddc004a20 con 0x7fcde80b0e30 2024-03-06T07:24:55.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.915+0000 7fcdee0ed640 1 -- 172.21.15.83:0/148958426 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcde81f4860 con 0x7fcde80b0e30 2024-03-06T07:24:55.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.915+0000 7fcdee0ed640 1 -- 172.21.15.83:0/148958426 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcde81f4d10 con 0x7fcde80b0e30 2024-03-06T07:24:55.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.916+0000 7fcdee0ed640 1 -- 172.21.15.83:0/148958426 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcde8068ff0 con 0x7fcde80b0e30 2024-03-06T07:24:55.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.917+0000 7fcdcffff640 1 -- 172.21.15.83:0/148958426 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcddc01b070 con 0x7fcde80b0e30 2024-03-06T07:24:55.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.917+0000 7fcdcffff640 1 --2- 172.21.15.83:0/148958426 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcdd406b500 0x7fcdd406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.917+0000 7fcdcffff640 1 -- 172.21.15.83:0/148958426 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcddc089390 con 0x7fcde80b0e30 2024-03-06T07:24:55.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.917+0000 7fcde6ffd640 1 --2- 172.21.15.83:0/148958426 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcdd406b500 0x7fcdd406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.918+0000 7fcde6ffd640 1 --2- 172.21.15.83:0/148958426 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcdd406b500 0x7fcdd406d9c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8036000 tx=0x7fcdd803b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:55.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:55.922+0000 7fcdcffff640 1 -- 172.21.15.83:0/148958426 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcddc007650 con 0x7fcde80b0e30 2024-03-06T07:24:56.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.185+0000 7fcdee0ed640 1 -- 172.21.15.83:0/148958426 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcde80670d0 con 0x7fcde80b0e30 2024-03-06T07:24:56.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.185+0000 7fcdcffff640 1 -- 172.21.15.83:0/148958426 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v57) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcddc0582a0 con 0x7fcde80b0e30 2024-03-06T07:24:56.187 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 64 2024-03-06T07:24:56.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.187+0000 7fcdee0ed640 1 -- 172.21.15.83:0/148958426 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcdd406b500 msgr2=0x7fcdd406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.187+0000 7fcdee0ed640 1 --2- 172.21.15.83:0/148958426 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcdd406b500 0x7fcdd406d9c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8036000 tx=0x7fcdd803b040 comp rx=0 tx=0).stop 2024-03-06T07:24:56.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.187+0000 7fcdee0ed640 1 -- 172.21.15.83:0/148958426 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcde80b0e30 msgr2=0x7fcde8157410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.187+0000 7fcdee0ed640 1 --2- 172.21.15.83:0/148958426 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcde80b0e30 0x7fcde8157410 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fcddc00c440 tx=0x7fcddc00c900 comp rx=0 tx=0).stop 2024-03-06T07:24:56.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.187+0000 7fcdee0ed640 1 -- 172.21.15.83:0/148958426 shutdown_connections 2024-03-06T07:24:56.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.187+0000 7fcdee0ed640 1 --2- 172.21.15.83:0/148958426 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcdd406b500 0x7fcdd406d9c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.187+0000 7fcdee0ed640 1 --2- 172.21.15.83:0/148958426 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcde81614d0 0x7fcde80cd180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.187+0000 7fcdee0ed640 1 --2- 172.21.15.83:0/148958426 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcde80d4860 0x7fcde81f3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.187+0000 7fcdee0ed640 1 --2- 172.21.15.83:0/148958426 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcde80b0e30 0x7fcde8157410 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.187+0000 7fcdee0ed640 1 -- 172.21.15.83:0/148958426 >> 172.21.15.83:0/148958426 conn(0x7fcde80d3d40 msgr2=0x7fcde80ce9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:56.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.187+0000 7fcdee0ed640 1 -- 172.21.15.83:0/148958426 shutdown_connections 2024-03-06T07:24:56.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.187+0000 7fcdee0ed640 1 -- 172.21.15.83:0/148958426 wait complete. 2024-03-06T07:24:56.200 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:24:56.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.300+0000 7fc1cc113640 1 Processor -- start 2024-03-06T07:24:56.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.300+0000 7fc1cc113640 1 -- start start 2024-03-06T07:24:56.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.301+0000 7fc1cc113640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fc1c415e000 0x7fc1c40d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.301+0000 7fc1cc113640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc1c40d4810 0x7fc1c40d0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.301+0000 7fc1cc113640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc1c40c80e0 0x7fc1c40c8570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.301+0000 7fc1cc113640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1c40c7c50 con 0x7fc1c40d4810 2024-03-06T07:24:56.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.301+0000 7fc1cc113640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1c40d25e0 con 0x7fc1c415e000 2024-03-06T07:24:56.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.301+0000 7fc1cc113640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c40d2880 con 0x7fc1c40c80e0 2024-03-06T07:24:56.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.301+0000 7fc1c96ae640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc1c40d4810 0x7fc1c40d0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.301+0000 7fc1c96ae640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc1c40d4810 0x7fc1c40d0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49416/0 (socket says 172.21.15.83:49416) 2024-03-06T07:24:56.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.301+0000 7fc1c96ae640 1 -- 172.21.15.83:0/2693259099 learned_addr learned my addr 172.21.15.83:0/2693259099 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:56.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.302+0000 7fc1ca6b0640 1 --2- 172.21.15.83:0/2693259099 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc1c40c80e0 0x7fc1c40c8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.302+0000 7fc1c9eaf640 1 --2- 172.21.15.83:0/2693259099 >> v2:172.21.15.183:3300/0 conn(0x7fc1c415e000 0x7fc1c40d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.302+0000 7fc1c96ae640 1 -- 172.21.15.83:0/2693259099 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc1c40c80e0 msgr2=0x7fc1c40c8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:56.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.302+0000 7fc1c96ae640 1 --2- 172.21.15.83:0/2693259099 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc1c40c80e0 0x7fc1c40c8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.302+0000 7fc1c96ae640 1 -- 172.21.15.83:0/2693259099 >> v2:172.21.15.183:3300/0 conn(0x7fc1c415e000 msgr2=0x7fc1c40d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:56.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.302+0000 7fc1c96ae640 1 --2- 172.21.15.83:0/2693259099 >> v2:172.21.15.183:3300/0 conn(0x7fc1c415e000 0x7fc1c40d0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.302+0000 7fc1c96ae640 1 -- 172.21.15.83:0/2693259099 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c40d2b40 con 0x7fc1c40d4810 2024-03-06T07:24:56.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.302+0000 7fc1c96ae640 1 --2- 172.21.15.83:0/2693259099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc1c40d4810 0x7fc1c40d0bc0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fc1c408c360 tx=0x7fc1b402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbc19556d5648486 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:56.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.303+0000 7fc1c8ead640 1 -- 172.21.15.83:0/2693259099 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b4034070 con 0x7fc1c40d4810 2024-03-06T07:24:56.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.303+0000 7fc1c8ead640 1 -- 172.21.15.83:0/2693259099 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b4030860 con 0x7fc1c40d4810 2024-03-06T07:24:56.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.303+0000 7fc1c8ead640 1 -- 172.21.15.83:0/2693259099 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b4030c30 con 0x7fc1c40d4810 2024-03-06T07:24:56.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.303+0000 7fc1cc113640 1 -- 172.21.15.83:0/2693259099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc1c40d4810 msgr2=0x7fc1c40d0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.303+0000 7fc1cc113640 1 --2- 172.21.15.83:0/2693259099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc1c40d4810 0x7fc1c40d0bc0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fc1c408c360 tx=0x7fc1b402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.303+0000 7fc1cc113640 1 -- 172.21.15.83:0/2693259099 shutdown_connections 2024-03-06T07:24:56.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.303+0000 7fc1cc113640 1 --2- 172.21.15.83:0/2693259099 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc1c40c80e0 0x7fc1c40c8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.303+0000 7fc1cc113640 1 --2- 172.21.15.83:0/2693259099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc1c40d4810 0x7fc1c40d0bc0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.303+0000 7fc1cc113640 1 --2- 172.21.15.83:0/2693259099 >> v2:172.21.15.183:3300/0 conn(0x7fc1c415e000 0x7fc1c40d0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.303+0000 7fc1cc113640 1 -- 172.21.15.83:0/2693259099 >> 172.21.15.83:0/2693259099 conn(0x7fc1c40d3cf0 msgr2=0x7fc1c40d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:56.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.303+0000 7fc1cc113640 1 -- 172.21.15.83:0/2693259099 shutdown_connections 2024-03-06T07:24:56.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.303+0000 7fc1cc113640 1 -- 172.21.15.83:0/2693259099 wait complete. 2024-03-06T07:24:56.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.305+0000 7fc1cc113640 1 Processor -- start 2024-03-06T07:24:56.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.305+0000 7fc1cc113640 1 -- start start 2024-03-06T07:24:56.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.306+0000 7fc1cc113640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc1c40c80e0 0x7fc1c40a10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.306+0000 7fc1cc113640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc1c40d4810 0x7fc1c40a1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.306+0000 7fc1cc113640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc1c415e000 0x7fc1c4153210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.306+0000 7fc1cc113640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c415e3e0 con 0x7fc1c40c80e0 2024-03-06T07:24:56.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.306+0000 7fc1cc113640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c40d22d0 con 0x7fc1c415e000 2024-03-06T07:24:56.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.306+0000 7fc1cc113640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c40c8ab0 con 0x7fc1c40d4810 2024-03-06T07:24:56.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.306+0000 7fc1c9eaf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc1c40c80e0 0x7fc1c40a10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.306+0000 7fc1c9eaf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc1c40c80e0 0x7fc1c40a10f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49422/0 (socket says 172.21.15.83:49422) 2024-03-06T07:24:56.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.306+0000 7fc1c9eaf640 1 -- 172.21.15.83:0/2024951131 learned_addr learned my addr 172.21.15.83:0/2024951131 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:56.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.306+0000 7fc1ca6b0640 1 --2- 172.21.15.83:0/2024951131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc1c415e000 0x7fc1c4153210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.306+0000 7fc1c96ae640 1 --2- 172.21.15.83:0/2024951131 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc1c40d4810 0x7fc1c40a1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.306+0000 7fc1c9eaf640 1 -- 172.21.15.83:0/2024951131 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc1c40d4810 msgr2=0x7fc1c40a1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.306+0000 7fc1c9eaf640 1 --2- 172.21.15.83:0/2024951131 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc1c40d4810 0x7fc1c40a1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.306+0000 7fc1c9eaf640 1 -- 172.21.15.83:0/2024951131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc1c415e000 msgr2=0x7fc1c4153210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.306+0000 7fc1c9eaf640 1 --2- 172.21.15.83:0/2024951131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc1c415e000 0x7fc1c4153210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.306+0000 7fc1c9eaf640 1 -- 172.21.15.83:0/2024951131 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c40cd670 con 0x7fc1c40c80e0 2024-03-06T07:24:56.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.307+0000 7fc1c9eaf640 1 --2- 172.21.15.83:0/2024951131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc1c40c80e0 0x7fc1c40a10f0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fc1c000d5f0 tx=0x7fc1c000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:56.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.307+0000 7fc1ba7fc640 1 -- 172.21.15.83:0/2024951131 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c00125f0 con 0x7fc1c40c80e0 2024-03-06T07:24:56.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.307+0000 7fc1ba7fc640 1 -- 172.21.15.83:0/2024951131 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1c0019070 con 0x7fc1c40c80e0 2024-03-06T07:24:56.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.307+0000 7fc1ba7fc640 1 -- 172.21.15.83:0/2024951131 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c00129d0 con 0x7fc1c40c80e0 2024-03-06T07:24:56.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.307+0000 7fc1cc113640 1 -- 172.21.15.83:0/2024951131 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1c4151180 con 0x7fc1c40c80e0 2024-03-06T07:24:56.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.307+0000 7fc1cc113640 1 -- 172.21.15.83:0/2024951131 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1c4151740 con 0x7fc1c40c80e0 2024-03-06T07:24:56.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.308+0000 7fc1cc113640 1 -- 172.21.15.83:0/2024951131 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1c404f330 con 0x7fc1c40c80e0 2024-03-06T07:24:56.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.308+0000 7fc1ba7fc640 1 -- 172.21.15.83:0/2024951131 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc1c0012b70 con 0x7fc1c40c80e0 2024-03-06T07:24:56.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.309+0000 7fc1ba7fc640 1 --2- 172.21.15.83:0/2024951131 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc1a806b4d0 0x7fc1a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.309+0000 7fc1ba7fc640 1 -- 172.21.15.83:0/2024951131 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc1c008a580 con 0x7fc1c40c80e0 2024-03-06T07:24:56.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.309+0000 7fc1c96ae640 1 --2- 172.21.15.83:0/2024951131 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc1a806b4d0 0x7fc1a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.309+0000 7fc1c96ae640 1 --2- 172.21.15.83:0/2024951131 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc1a806b4d0 0x7fc1a806d990 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fc1b4036000 tx=0x7fc1b403b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:56.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.313+0000 7fc1ba7fc640 1 -- 172.21.15.83:0/2024951131 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc1c0059400 con 0x7fc1c40c80e0 2024-03-06T07:24:56.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.499+0000 7fc1cc113640 1 -- 172.21.15.83:0/2024951131 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc1c40cf840 con 0x7fc1c40c80e0 2024-03-06T07:24:56.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.499+0000 7fc1ba7fc640 1 -- 172.21.15.83:0/2024951131 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v57) v1 ==== 80+0+0 (secure 0 0 0) 0x7fc1c0058d00 con 0x7fc1c40c80e0 2024-03-06T07:24:56.501 INFO:teuthology.orchestra.run.smithi083.stderr:listed 0 entries 2024-03-06T07:24:56.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.501+0000 7fc1cc113640 1 -- 172.21.15.83:0/2024951131 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc1a806b4d0 msgr2=0x7fc1a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.501+0000 7fc1cc113640 1 --2- 172.21.15.83:0/2024951131 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc1a806b4d0 0x7fc1a806d990 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fc1b4036000 tx=0x7fc1b403b040 comp rx=0 tx=0).stop 2024-03-06T07:24:56.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.501+0000 7fc1cc113640 1 -- 172.21.15.83:0/2024951131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc1c40c80e0 msgr2=0x7fc1c40a10f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.501+0000 7fc1cc113640 1 --2- 172.21.15.83:0/2024951131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc1c40c80e0 0x7fc1c40a10f0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fc1c000d5f0 tx=0x7fc1c000dab0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.501+0000 7fc1cc113640 1 -- 172.21.15.83:0/2024951131 shutdown_connections 2024-03-06T07:24:56.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.501+0000 7fc1cc113640 1 --2- 172.21.15.83:0/2024951131 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc1a806b4d0 0x7fc1a806d990 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.501+0000 7fc1cc113640 1 --2- 172.21.15.83:0/2024951131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc1c415e000 0x7fc1c4153210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.501+0000 7fc1cc113640 1 --2- 172.21.15.83:0/2024951131 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc1c40d4810 0x7fc1c40a1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.501+0000 7fc1cc113640 1 --2- 172.21.15.83:0/2024951131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc1c40c80e0 0x7fc1c40a10f0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.501+0000 7fc1cc113640 1 -- 172.21.15.83:0/2024951131 >> 172.21.15.83:0/2024951131 conn(0x7fc1c40d3cf0 msgr2=0x7fc1c40ce740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:56.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.501+0000 7fc1cc113640 1 -- 172.21.15.83:0/2024951131 shutdown_connections 2024-03-06T07:24:56.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:56.501+0000 7fc1cc113640 1 -- 172.21.15.83:0/2024951131 wait complete. 2024-03-06T07:24:56.513 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi083.front.sepia.ceph.com... 2024-03-06T07:24:56.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:56.514 DEBUG:teuthology.orchestra.run.smithi083:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:56.538 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:ceph-fuse[60304]: fuse finished with error 0 and tester_r 0 2024-03-06T07:24:56.540 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi083.front.sepia.ceph.com 2024-03-06T07:24:56.540 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:56.540 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:24:56.716 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:24:56.716 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:56.734 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=287 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:24:56.734 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi183.front.sepia.ceph.com 2024-03-06T07:24:56.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:56.734 DEBUG:teuthology.orchestra.run.smithi183:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:24:56.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:24:56.791 INFO:teuthology.orchestra.run.smithi183.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:24:56.791 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:24:56.792 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:56.808 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1257 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:24:56.808 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi083.front.sepia.ceph.com 2024-03-06T07:24:56.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:56.808 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:24:56.863 INFO:teuthology.orchestra.run.smithi083.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:24:56.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:24:56.864 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:24:56.864 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:56.881 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=287 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:24:56.881 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi183.front.sepia.ceph.com 2024-03-06T07:24:56.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:56.881 DEBUG:teuthology.orchestra.run.smithi183:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:24:56.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:24:56.938 INFO:teuthology.orchestra.run.smithi183.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:24:56.938 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:57.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.040+0000 7f325b565640 1 Processor -- start 2024-03-06T07:24:57.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.040+0000 7f325b565640 1 -- start start 2024-03-06T07:24:57.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.041+0000 7f325b565640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f32540d22b0 0x7f32540d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.041+0000 7f325b565640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f32540b41f0 0x7f32540b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.041+0000 7f325b565640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32540b3430 con 0x7f325415dfe0 2024-03-06T07:24:57.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.041+0000 7f325b565640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32540d2c50 con 0x7f32540d22b0 2024-03-06T07:24:57.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.041+0000 7f325b565640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32540b31e0 con 0x7f32540b41f0 2024-03-06T07:24:57.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.041+0000 7f3258b00640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f32540d22b0 0x7f32540d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.041+0000 7f3259301640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f325415dfe0 0x7f32540cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49714/0 (socket says 172.21.15.83:49714) 2024-03-06T07:24:57.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.041+0000 7f3258b00640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f32540d22b0 0x7f32540d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41540/0 (socket says 172.21.15.83:41540) 2024-03-06T07:24:57.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.041+0000 7f3259301640 1 -- 172.21.15.83:0/2463826916 learned_addr learned my addr 172.21.15.83:0/2463826916 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:24:57.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.042+0000 7f3258b00640 1 -- 172.21.15.83:0/2463826916 >> v2:172.21.15.183:3300/0 conn(0x7f32540b41f0 msgr2=0x7f32540b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:57.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.042+0000 7f3259b02640 1 --2- 172.21.15.83:0/2463826916 >> v2:172.21.15.183:3300/0 conn(0x7f32540b41f0 0x7f32540b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.042+0000 7f3258b00640 1 --2- 172.21.15.83:0/2463826916 >> v2:172.21.15.183:3300/0 conn(0x7f32540b41f0 0x7f32540b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.042+0000 7f324bfff640 1 -- 172.21.15.83:0/2463826916 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2256931563 0 0) 0x7f32540b3430 con 0x7f325415dfe0 2024-03-06T07:24:57.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.042+0000 7f3258b00640 1 -- 172.21.15.83:0/2463826916 >> v1:172.21.15.83:6789/0 conn(0x7f325415dfe0 legacy=0x7f32540cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.042+0000 7f3258b00640 1 -- 172.21.15.83:0/2463826916 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32540b3710 con 0x7f32540d22b0 2024-03-06T07:24:57.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.042+0000 7f3258b00640 1 --2- 172.21.15.83:0/2463826916 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f32540d22b0 0x7f32540d2710 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f32440097a0 tx=0x7f324402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b20a985be309e269 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.043+0000 7f324bfff640 1 -- 172.21.15.83:0/2463826916 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3244035020 con 0x7f32540d22b0 2024-03-06T07:24:57.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.043+0000 7f324bfff640 1 -- 172.21.15.83:0/2463826916 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3244031830 con 0x7f32540d22b0 2024-03-06T07:24:57.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.043+0000 7f324bfff640 1 -- 172.21.15.83:0/2463826916 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3244031c00 con 0x7f32540d22b0 2024-03-06T07:24:57.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.043+0000 7f325b565640 1 -- 172.21.15.83:0/2463826916 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f32540d22b0 msgr2=0x7f32540d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.043+0000 7f325b565640 1 --2- 172.21.15.83:0/2463826916 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f32540d22b0 0x7f32540d2710 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f32440097a0 tx=0x7f324402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.043+0000 7f325b565640 1 -- 172.21.15.83:0/2463826916 shutdown_connections 2024-03-06T07:24:57.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.043+0000 7f325b565640 1 --2- 172.21.15.83:0/2463826916 >> v2:172.21.15.183:3300/0 conn(0x7f32540b41f0 0x7f32540b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.043+0000 7f325b565640 1 --2- 172.21.15.83:0/2463826916 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f32540d22b0 0x7f32540d2710 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.043+0000 7f325b565640 1 -- 172.21.15.83:0/2463826916 >> 172.21.15.83:0/2463826916 conn(0x7f32540d3cd0 msgr2=0x7f32540d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:57.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.043+0000 7f325b565640 1 -- 172.21.15.83:0/2463826916 shutdown_connections 2024-03-06T07:24:57.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.043+0000 7f325b565640 1 -- 172.21.15.83:0/2463826916 wait complete. 2024-03-06T07:24:57.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.044+0000 7f325b565640 1 Processor -- start 2024-03-06T07:24:57.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.044+0000 7f325b565640 1 -- start start 2024-03-06T07:24:57.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.045+0000 7f325b565640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f32540b41f0 0x7f32540d0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.045+0000 7f325b565640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f32540d22b0 0x7f325415bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.045+0000 7f325b565640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f325415dfe0 0x7f325415c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.045+0000 7f325b565640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32540cfa00 con 0x7f32540d22b0 2024-03-06T07:24:57.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.045+0000 7f325b565640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32540b3d60 con 0x7f325415dfe0 2024-03-06T07:24:57.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.045+0000 7f325b565640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32540d1c70 con 0x7f32540b41f0 2024-03-06T07:24:57.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.045+0000 7f3258b00640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f32540d22b0 0x7f325415bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.045+0000 7f3258b00640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f32540d22b0 0x7f325415bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41554/0 (socket says 172.21.15.83:41554) 2024-03-06T07:24:57.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.045+0000 7f3258b00640 1 -- 172.21.15.83:0/398926064 learned_addr learned my addr 172.21.15.83:0/398926064 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:57.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.045+0000 7f3258b00640 1 -- 172.21.15.83:0/398926064 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f32540b41f0 msgr2=0x7f32540d0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.045+0000 7f3258b00640 1 --2- 172.21.15.83:0/398926064 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f32540b41f0 0x7f32540d0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.045+0000 7f3258b00640 1 -- 172.21.15.83:0/398926064 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f325415dfe0 msgr2=0x7f325415c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.045+0000 7f3259b02640 1 --2- 172.21.15.83:0/398926064 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f325415dfe0 0x7f325415c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.045+0000 7f3258b00640 1 --2- 172.21.15.83:0/398926064 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f325415dfe0 0x7f325415c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.046+0000 7f3258b00640 1 -- 172.21.15.83:0/398926064 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3254160240 con 0x7f32540d22b0 2024-03-06T07:24:57.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.046+0000 7f3258b00640 1 --2- 172.21.15.83:0/398926064 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f32540d22b0 0x7f325415bde0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f3244000c00 tx=0x7f32440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.046+0000 7f3249ffb640 1 -- 172.21.15.83:0/398926064 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3244035050 con 0x7f32540d22b0 2024-03-06T07:24:57.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.046+0000 7f3249ffb640 1 -- 172.21.15.83:0/398926064 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3244004230 con 0x7f32540d22b0 2024-03-06T07:24:57.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.046+0000 7f325b565640 1 -- 172.21.15.83:0/398926064 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32541f7c00 con 0x7f32540d22b0 2024-03-06T07:24:57.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.046+0000 7f3249ffb640 1 -- 172.21.15.83:0/398926064 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f324403b800 con 0x7f32540d22b0 2024-03-06T07:24:57.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.046+0000 7f325b565640 1 -- 172.21.15.83:0/398926064 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32541f3eb0 con 0x7f32540d22b0 2024-03-06T07:24:57.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.047+0000 7f325b565640 1 -- 172.21.15.83:0/398926064 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3254161670 con 0x7f32540d22b0 2024-03-06T07:24:57.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.047+0000 7f3249ffb640 1 -- 172.21.15.83:0/398926064 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3244038020 con 0x7f32540d22b0 2024-03-06T07:24:57.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.048+0000 7f3249ffb640 1 --2- 172.21.15.83:0/398926064 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f323806b4d0 0x7f323806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.048+0000 7f3249ffb640 1 -- 172.21.15.83:0/398926064 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f32440abf80 con 0x7f32540d22b0 2024-03-06T07:24:57.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.048+0000 7f3259301640 1 --2- 172.21.15.83:0/398926064 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f323806b4d0 0x7f323806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.048+0000 7f3259301640 1 --2- 172.21.15.83:0/398926064 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f323806b4d0 0x7f323806d990 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f3250007fd0 tx=0x7f3250007960 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.056+0000 7f3249ffb640 1 -- 172.21.15.83:0/398926064 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3254161670 con 0x7f32540d22b0 2024-03-06T07:24:57.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.283+0000 7f325b565640 1 -- 172.21.15.83:0/398926064 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f325404f330 con 0x7f32540d22b0 2024-03-06T07:24:57.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.283+0000 7f3249ffb640 1 -- 172.21.15.83:0/398926064 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f3254160240 con 0x7f32540d22b0 2024-03-06T07:24:57.286 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:24:57.286 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":25,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5665,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/3334609507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3334609507},{"type":"v1","addr":"172.21.15.183:6834","nonce":3334609507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5668,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/691189112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":691189112},{"type":"v1","addr":"172.21.15.183:6835","nonce":691189112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6081,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/3952571284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":3952571284},{"type":"v1","addr":"172.21.15.83:6836","nonce":3952571284}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:19.852157+0000","modified":"2024-03-06T07:24:21.879875+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6084},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6084":{"gid":6084,"name":"a","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/3237506441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3237506441},{"type":"v1","addr":"172.21.15.83:6837","nonce":3237506441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6084,"qdb_cluster":[6084]},"id":4}]} 2024-03-06T07:24:57.286 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 25 2024-03-06T07:24:57.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.286+0000 7f325b565640 1 -- 172.21.15.83:0/398926064 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f323806b4d0 msgr2=0x7f323806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.286+0000 7f325b565640 1 --2- 172.21.15.83:0/398926064 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f323806b4d0 0x7f323806d990 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f3250007fd0 tx=0x7f3250007960 comp rx=0 tx=0).stop 2024-03-06T07:24:57.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.286+0000 7f325b565640 1 -- 172.21.15.83:0/398926064 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f32540d22b0 msgr2=0x7f325415bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.286+0000 7f325b565640 1 --2- 172.21.15.83:0/398926064 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f32540d22b0 0x7f325415bde0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f3244000c00 tx=0x7f32440046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.286+0000 7f325b565640 1 -- 172.21.15.83:0/398926064 shutdown_connections 2024-03-06T07:24:57.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.286+0000 7f325b565640 1 --2- 172.21.15.83:0/398926064 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f323806b4d0 0x7f323806d990 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.286+0000 7f325b565640 1 --2- 172.21.15.83:0/398926064 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f325415dfe0 0x7f325415c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.286+0000 7f325b565640 1 --2- 172.21.15.83:0/398926064 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f32540d22b0 0x7f325415bde0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.286+0000 7f325b565640 1 --2- 172.21.15.83:0/398926064 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f32540b41f0 0x7f32540d0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.286+0000 7f325b565640 1 -- 172.21.15.83:0/398926064 >> 172.21.15.83:0/398926064 conn(0x7f32540d3cd0 msgr2=0x7f325406ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:57.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.287+0000 7f325b565640 1 -- 172.21.15.83:0/398926064 shutdown_connections 2024-03-06T07:24:57.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.287+0000 7f325b565640 1 -- 172.21.15.83:0/398926064 wait complete. 2024-03-06T07:24:57.300 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:57.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.400+0000 7f04527bc640 1 Processor -- start 2024-03-06T07:24:57.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.400+0000 7f04527bc640 1 -- start start 2024-03-06T07:24:57.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.400+0000 7f04527bc640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f044c158600 0x7f044c072730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.400+0000 7f04527bc640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f044c162030 0x7f044c1624c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.400+0000 7f04527bc640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f044c15eb40 con 0x7f044c15e3a0 2024-03-06T07:24:57.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.400+0000 7f04527bc640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f044c161ba0 con 0x7f044c162030 2024-03-06T07:24:57.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.400+0000 7f04527bc640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f044c15ee20 con 0x7f044c158600 2024-03-06T07:24:57.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.401+0000 7f044b7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f044c15e3a0 0x7f044c06d970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49728/0 (socket says 172.21.15.83:49728) 2024-03-06T07:24:57.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.401+0000 7f044b7fe640 1 -- 172.21.15.83:0/2429218743 learned_addr learned my addr 172.21.15.83:0/2429218743 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:24:57.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.401+0000 7f0450d59640 1 --2- 172.21.15.83:0/2429218743 >> v2:172.21.15.183:3300/0 conn(0x7f044c162030 0x7f044c1624c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.401+0000 7f044bfff640 1 --2- 172.21.15.83:0/2429218743 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f044c158600 0x7f044c072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.401+0000 7f044affd640 1 -- 172.21.15.83:0/2429218743 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 438822116 0 0) 0x7f044c15eb40 con 0x7f044c15e3a0 2024-03-06T07:24:57.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.401+0000 7f044affd640 1 -- 172.21.15.83:0/2429218743 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0434003520 con 0x7f044c15e3a0 2024-03-06T07:24:57.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.401+0000 7f044affd640 1 -- 172.21.15.83:0/2429218743 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2375519251 0 0) 0x7f0434003520 con 0x7f044c15e3a0 2024-03-06T07:24:57.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.401+0000 7f044affd640 1 -- 172.21.15.83:0/2429218743 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f044c158600 msgr2=0x7f044c072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:57.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.401+0000 7f044affd640 1 --2- 172.21.15.83:0/2429218743 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f044c158600 0x7f044c072730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.401+0000 7f044affd640 1 -- 172.21.15.83:0/2429218743 >> v2:172.21.15.183:3300/0 conn(0x7f044c162030 msgr2=0x7f044c1624c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:57.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.401+0000 7f044affd640 1 --2- 172.21.15.83:0/2429218743 >> v2:172.21.15.183:3300/0 conn(0x7f044c162030 0x7f044c1624c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.401+0000 7f044affd640 1 -- 172.21.15.83:0/2429218743 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f044c15f100 con 0x7f044c15e3a0 2024-03-06T07:24:57.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.401+0000 7f044affd640 1 -- 172.21.15.83:0/2429218743 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f0440001670 con 0x7f044c15e3a0 2024-03-06T07:24:57.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.401+0000 7f044affd640 1 -- 172.21.15.83:0/2429218743 >> v1:172.21.15.83:6789/0 conn(0x7f044c15e3a0 legacy=0x7f044c06d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.401+0000 7f044affd640 1 --2- 172.21.15.83:0/2429218743 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f04340039b0 0x7f0434003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.401+0000 7f044affd640 1 --2- 172.21.15.83:0/2429218743 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0434004800 0x7f0434004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.401+0000 7f044affd640 1 --2- 172.21.15.83:0/2429218743 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f044c158600 0x7f04340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.401+0000 7f044affd640 1 -- 172.21.15.83:0/2429218743 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f044c161ba0 con 0x7f04340039b0 2024-03-06T07:24:57.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.401+0000 7f044affd640 1 -- 172.21.15.83:0/2429218743 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f044c15ee20 con 0x7f0434004800 2024-03-06T07:24:57.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.401+0000 7f044affd640 1 -- 172.21.15.83:0/2429218743 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f043400b9c0 con 0x7f044c158600 2024-03-06T07:24:57.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.401+0000 7f044bfff640 1 --2- 172.21.15.83:0/2429218743 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f04340039b0 0x7f0434003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.401+0000 7f044b7fe640 1 --2- 172.21.15.83:0/2429218743 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0434004800 0x7f0434004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.401+0000 7f0450d59640 1 --2- 172.21.15.83:0/2429218743 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f044c158600 0x7f04340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.402+0000 7f044bfff640 1 -- 172.21.15.83:0/2429218743 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f044c158600 msgr2=0x7f04340051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:57.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.402+0000 7f044bfff640 1 --2- 172.21.15.83:0/2429218743 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f044c158600 0x7f04340051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.402+0000 7f044bfff640 1 -- 172.21.15.83:0/2429218743 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0434004800 msgr2=0x7f0434004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:57.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.402+0000 7f044bfff640 1 --2- 172.21.15.83:0/2429218743 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0434004800 0x7f0434004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.402+0000 7f044bfff640 1 -- 172.21.15.83:0/2429218743 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0434003520 con 0x7f04340039b0 2024-03-06T07:24:57.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.402+0000 7f0450d59640 1 --2- 172.21.15.83:0/2429218743 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f044c158600 0x7f04340051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:57.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.402+0000 7f044bfff640 1 --2- 172.21.15.83:0/2429218743 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f04340039b0 0x7f0434003dd0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f043c009af0 tx=0x7f0434005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1a0a34eac03b74b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.402+0000 7f044affd640 1 -- 172.21.15.83:0/2429218743 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043c039070 con 0x7f04340039b0 2024-03-06T07:24:57.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.403+0000 7f044affd640 1 -- 172.21.15.83:0/2429218743 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f043c02c6b0 con 0x7f04340039b0 2024-03-06T07:24:57.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.403+0000 7f044affd640 1 -- 172.21.15.83:0/2429218743 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043c009690 con 0x7f04340039b0 2024-03-06T07:24:57.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.403+0000 7f04527bc640 1 -- 172.21.15.83:0/2429218743 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f04340039b0 msgr2=0x7f0434003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.403+0000 7f04527bc640 1 --2- 172.21.15.83:0/2429218743 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f04340039b0 0x7f0434003dd0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f043c009af0 tx=0x7f0434005830 comp rx=0 tx=0).stop 2024-03-06T07:24:57.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.403+0000 7f0450d59640 1 -- 172.21.15.83:0/2429218743 reap_dead start 2024-03-06T07:24:57.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.403+0000 7f04527bc640 1 -- 172.21.15.83:0/2429218743 shutdown_connections 2024-03-06T07:24:57.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.403+0000 7f04527bc640 1 -- 172.21.15.83:0/2429218743 >> 172.21.15.83:0/2429218743 conn(0x7f044c06c0c0 msgr2=0x7f044c06c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:57.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.403+0000 7f04527bc640 1 -- 172.21.15.83:0/2429218743 shutdown_connections 2024-03-06T07:24:57.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.403+0000 7f04527bc640 1 -- 172.21.15.83:0/2429218743 wait complete. 2024-03-06T07:24:57.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.404+0000 7f04527bc640 1 Processor -- start 2024-03-06T07:24:57.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.404+0000 7f04527bc640 1 -- start start 2024-03-06T07:24:57.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.405+0000 7f04527bc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f04340039b0 0x7f044c1f7500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.405+0000 7f04527bc640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f044c15da20 0x7f044c0cf020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.405+0000 7f04527bc640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f044c0b3220 0x7f044c0b36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.405+0000 7f04527bc640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f044c06e080 con 0x7f04340039b0 2024-03-06T07:24:57.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.405+0000 7f04527bc640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f044c072730 con 0x7f044c15da20 2024-03-06T07:24:57.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.405+0000 7f04527bc640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f044c1590a0 con 0x7f044c0b3220 2024-03-06T07:24:57.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.405+0000 7f044bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f04340039b0 0x7f044c1f7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.405+0000 7f044bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f04340039b0 0x7f044c1f7500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41582/0 (socket says 172.21.15.83:41582) 2024-03-06T07:24:57.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.405+0000 7f044bfff640 1 -- 172.21.15.83:0/1401552606 learned_addr learned my addr 172.21.15.83:0/1401552606 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:57.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.405+0000 7f044b7fe640 1 --2- 172.21.15.83:0/1401552606 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f044c15da20 0x7f044c0cf020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.405+0000 7f0450d59640 1 --2- 172.21.15.83:0/1401552606 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f044c0b3220 0x7f044c0b36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.405+0000 7f044bfff640 1 -- 172.21.15.83:0/1401552606 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f044c0b3220 msgr2=0x7f044c0b36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.405+0000 7f044bfff640 1 --2- 172.21.15.83:0/1401552606 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f044c0b3220 0x7f044c0b36a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.405+0000 7f044bfff640 1 -- 172.21.15.83:0/1401552606 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f044c15da20 msgr2=0x7f044c0cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.405+0000 7f044bfff640 1 --2- 172.21.15.83:0/1401552606 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f044c15da20 0x7f044c0cf020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.405+0000 7f044bfff640 1 -- 172.21.15.83:0/1401552606 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f044c1f7a40 con 0x7f04340039b0 2024-03-06T07:24:57.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.406+0000 7f044b7fe640 1 --2- 172.21.15.83:0/1401552606 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f044c15da20 0x7f044c0cf020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:57.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.406+0000 7f0450d59640 1 --2- 172.21.15.83:0/1401552606 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f044c0b3220 0x7f044c0b36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:57.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.406+0000 7f044bfff640 1 --2- 172.21.15.83:0/1401552606 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f04340039b0 0x7f044c1f7500 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f043c037040 tx=0x7f043c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.406+0000 7f0448ff9640 1 -- 172.21.15.83:0/1401552606 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043c039050 con 0x7f04340039b0 2024-03-06T07:24:57.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.406+0000 7f0448ff9640 1 -- 172.21.15.83:0/1401552606 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f043c045070 con 0x7f04340039b0 2024-03-06T07:24:57.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.406+0000 7f04527bc640 1 -- 172.21.15.83:0/1401552606 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f044c1f7d10 con 0x7f04340039b0 2024-03-06T07:24:57.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.406+0000 7f0448ff9640 1 -- 172.21.15.83:0/1401552606 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043c003a50 con 0x7f04340039b0 2024-03-06T07:24:57.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.406+0000 7f04527bc640 1 -- 172.21.15.83:0/1401552606 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f044c1f82d0 con 0x7f04340039b0 2024-03-06T07:24:57.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.407+0000 7f04527bc640 1 -- 172.21.15.83:0/1401552606 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f044c04f330 con 0x7f04340039b0 2024-03-06T07:24:57.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.407+0000 7f0448ff9640 1 -- 172.21.15.83:0/1401552606 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f043c003bf0 con 0x7f04340039b0 2024-03-06T07:24:57.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.408+0000 7f0448ff9640 1 --2- 172.21.15.83:0/1401552606 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f043806b4d0 0x7f043806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.408+0000 7f0448ff9640 1 -- 172.21.15.83:0/1401552606 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f043c0abe70 con 0x7f04340039b0 2024-03-06T07:24:57.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.408+0000 7f044b7fe640 1 --2- 172.21.15.83:0/1401552606 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f043806b4d0 0x7f043806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.408+0000 7f044b7fe640 1 --2- 172.21.15.83:0/1401552606 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f043806b4d0 0x7f043806d990 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f044c14b2a0 tx=0x7f0440009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.413+0000 7f0448ff9640 1 -- 172.21.15.83:0/1401552606 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f043c04e020 con 0x7f04340039b0 2024-03-06T07:24:57.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.637+0000 7f04527bc640 1 -- 172.21.15.83:0/1401552606 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f044c0d06f0 con 0x7f04340039b0 2024-03-06T07:24:57.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.638+0000 7f0448ff9640 1 -- 172.21.15.83:0/1401552606 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+16677 (secure 0 0 0) 0x7f043c07adb0 con 0x7f04340039b0 2024-03-06T07:24:57.640 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:24:57.640 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":57,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:24:21.525937+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:18.074366+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:18.831101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:57.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.641+0000 7f04527bc640 1 -- 172.21.15.83:0/1401552606 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f043806b4d0 msgr2=0x7f043806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.641+0000 7f04527bc640 1 --2- 172.21.15.83:0/1401552606 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f043806b4d0 0x7f043806d990 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f044c14b2a0 tx=0x7f0440009040 comp rx=0 tx=0).stop 2024-03-06T07:24:57.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.641+0000 7f04527bc640 1 -- 172.21.15.83:0/1401552606 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f04340039b0 msgr2=0x7f044c1f7500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.641+0000 7f04527bc640 1 --2- 172.21.15.83:0/1401552606 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f04340039b0 0x7f044c1f7500 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f043c037040 tx=0x7f043c004730 comp rx=0 tx=0).stop 2024-03-06T07:24:57.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.641+0000 7f04527bc640 1 -- 172.21.15.83:0/1401552606 shutdown_connections 2024-03-06T07:24:57.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.641+0000 7f04527bc640 1 --2- 172.21.15.83:0/1401552606 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f043806b4d0 0x7f043806d990 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.641+0000 7f04527bc640 1 --2- 172.21.15.83:0/1401552606 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f044c0b3220 0x7f044c0b36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.641+0000 7f04527bc640 1 --2- 172.21.15.83:0/1401552606 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f044c15da20 0x7f044c0cf020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.641+0000 7f04527bc640 1 --2- 172.21.15.83:0/1401552606 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f04340039b0 0x7f044c1f7500 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.641+0000 7f04527bc640 1 -- 172.21.15.83:0/1401552606 >> 172.21.15.83:0/1401552606 conn(0x7f044c06c0c0 msgr2=0x7f044c156380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:57.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.641+0000 7f04527bc640 1 -- 172.21.15.83:0/1401552606 shutdown_connections 2024-03-06T07:24:57.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.641+0000 7f04527bc640 1 -- 172.21.15.83:0/1401552606 wait complete. 2024-03-06T07:24:57.654 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:24:18.074366+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:24:18.831101+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:57.654 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:57.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.757+0000 7f1a0dc29640 1 Processor -- start 2024-03-06T07:24:57.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.757+0000 7f1a0dc29640 1 -- start start 2024-03-06T07:24:57.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.757+0000 7f1a0dc29640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f1a081614d0 0x7f1a080d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.757+0000 7f1a0dc29640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a081618b0 con 0x7f1a080d1910 2024-03-06T07:24:57.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.757+0000 7f1a0dc29640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a080cf400 con 0x7f1a080d24b0 2024-03-06T07:24:57.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.757+0000 7f1a0dc29640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a080cf6a0 con 0x7f1a081614d0 2024-03-06T07:24:57.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.758+0000 7f1a06ffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f1a080d1910 0x7f1a080d1da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49742/0 (socket says 172.21.15.83:49742) 2024-03-06T07:24:57.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.758+0000 7f1a06ffd640 1 -- 172.21.15.83:0/3122579627 learned_addr learned my addr 172.21.15.83:0/3122579627 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:24:57.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.758+0000 7f1a077fe640 1 --2- 172.21.15.83:0/3122579627 >> v2:172.21.15.183:3300/0 conn(0x7f1a081614d0 0x7f1a080d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.758+0000 7f1a067fc640 1 -- 172.21.15.83:0/3122579627 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 992146478 0 0) 0x7f1a081618b0 con 0x7f1a080d1910 2024-03-06T07:24:57.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.758+0000 7f1a067fc640 1 -- 172.21.15.83:0/3122579627 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19f0003520 con 0x7f1a080d1910 2024-03-06T07:24:57.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.758+0000 7f1a067fc640 1 -- 172.21.15.83:0/3122579627 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 390016470 0 0) 0x7f19f0003520 con 0x7f1a080d1910 2024-03-06T07:24:57.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.758+0000 7f1a067fc640 1 -- 172.21.15.83:0/3122579627 >> v2:172.21.15.183:3300/0 conn(0x7f1a081614d0 msgr2=0x7f1a080d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:57.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.758+0000 7f1a067fc640 1 --2- 172.21.15.83:0/3122579627 >> v2:172.21.15.183:3300/0 conn(0x7f1a081614d0 0x7f1a080d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.758+0000 7f1a067fc640 1 -- 172.21.15.83:0/3122579627 >> v1:172.21.15.183:6789/0 conn(0x7f1a080d24b0 legacy=0x7f1a080b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.758+0000 7f1a067fc640 1 -- 172.21.15.83:0/3122579627 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a080cf980 con 0x7f1a080d1910 2024-03-06T07:24:57.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.758+0000 7f1a067fc640 1 -- 172.21.15.83:0/3122579627 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f19f8001670 con 0x7f1a080d1910 2024-03-06T07:24:57.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.758+0000 7f1a077fe640 1 --2- 172.21.15.83:0/3122579627 >> v2:172.21.15.183:3300/0 conn(0x7f1a081614d0 0x7f1a080d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:57.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.758+0000 7f1a067fc640 1 -- 172.21.15.83:0/3122579627 >> v1:172.21.15.83:6789/0 conn(0x7f1a080d1910 legacy=0x7f1a080d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.758+0000 7f1a067fc640 1 --2- 172.21.15.83:0/3122579627 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f19f00039b0 0x7f19f0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.758+0000 7f1a067fc640 1 --2- 172.21.15.83:0/3122579627 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f19f0004800 0x7f19f0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.758+0000 7f1a067fc640 1 --2- 172.21.15.83:0/3122579627 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f19f00051c0 0x7f19f000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.758+0000 7f1a067fc640 1 -- 172.21.15.83:0/3122579627 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a080cf6a0 con 0x7f19f00039b0 2024-03-06T07:24:57.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.758+0000 7f1a067fc640 1 -- 172.21.15.83:0/3122579627 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f000c060 con 0x7f19f0004800 2024-03-06T07:24:57.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.758+0000 7f1a067fc640 1 -- 172.21.15.83:0/3122579627 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19f000c260 con 0x7f19f00051c0 2024-03-06T07:24:57.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.758+0000 7f1a077fe640 1 --2- 172.21.15.83:0/3122579627 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f19f00039b0 0x7f19f0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.758+0000 7f1a06ffd640 1 --2- 172.21.15.83:0/3122579627 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f19f0004800 0x7f19f0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.758+0000 7f1a07fff640 1 --2- 172.21.15.83:0/3122579627 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f19f00051c0 0x7f19f000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.759+0000 7f1a077fe640 1 -- 172.21.15.83:0/3122579627 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f19f00051c0 msgr2=0x7f19f000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:57.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.759+0000 7f1a077fe640 1 --2- 172.21.15.83:0/3122579627 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f19f00051c0 0x7f19f000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.759+0000 7f1a077fe640 1 -- 172.21.15.83:0/3122579627 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f19f0004800 msgr2=0x7f19f0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:57.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.759+0000 7f1a077fe640 1 --2- 172.21.15.83:0/3122579627 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f19f0004800 0x7f19f0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.759+0000 7f1a077fe640 1 -- 172.21.15.83:0/3122579627 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f0003520 con 0x7f19f00039b0 2024-03-06T07:24:57.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.759+0000 7f1a07fff640 1 --2- 172.21.15.83:0/3122579627 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f19f00051c0 0x7f19f000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:57.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.759+0000 7f1a07fff640 1 -- 172.21.15.83:0/3122579627 reap_dead start 2024-03-06T07:24:57.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.759+0000 7f1a077fe640 1 --2- 172.21.15.83:0/3122579627 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f19f00039b0 0x7f19f0003dd0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f19fc009c90 tx=0x7f19f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c37a2c2905ed17da server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.759+0000 7f1a067fc640 1 -- 172.21.15.83:0/3122579627 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19fc039070 con 0x7f19f00039b0 2024-03-06T07:24:57.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.759+0000 7f1a067fc640 1 -- 172.21.15.83:0/3122579627 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19fc02c8f0 con 0x7f19f00039b0 2024-03-06T07:24:57.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.759+0000 7f1a067fc640 1 -- 172.21.15.83:0/3122579627 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19fc009860 con 0x7f19f00039b0 2024-03-06T07:24:57.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.760+0000 7f1a0dc29640 1 -- 172.21.15.83:0/3122579627 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f19f00039b0 msgr2=0x7f19f0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.760+0000 7f1a0dc29640 1 --2- 172.21.15.83:0/3122579627 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f19f00039b0 0x7f19f0003dd0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f19fc009c90 tx=0x7f19f0005830 comp rx=0 tx=0).stop 2024-03-06T07:24:57.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.760+0000 7f1a0dc29640 1 -- 172.21.15.83:0/3122579627 shutdown_connections 2024-03-06T07:24:57.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.760+0000 7f1a0dc29640 1 --2- 172.21.15.83:0/3122579627 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f19f00039b0 0x7f19f0003dd0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.760+0000 7f1a0dc29640 1 -- 172.21.15.83:0/3122579627 >> 172.21.15.83:0/3122579627 conn(0x7f1a080d3d40 msgr2=0x7f1a080d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:57.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.760+0000 7f1a0dc29640 1 -- 172.21.15.83:0/3122579627 shutdown_connections 2024-03-06T07:24:57.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.760+0000 7f1a0dc29640 1 -- 172.21.15.83:0/3122579627 wait complete. 2024-03-06T07:24:57.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.761+0000 7f1a0dc29640 1 Processor -- start 2024-03-06T07:24:57.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.761+0000 7f1a0dc29640 1 -- start start 2024-03-06T07:24:57.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.762+0000 7f1a0dc29640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f19f00039b0 0x7f1a080b9160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.762+0000 7f1a0dc29640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a0815fcb0 0x7f1a080c9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.762+0000 7f1a0dc29640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1a0815f120 0x7f1a0815f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.762+0000 7f1a0dc29640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a080d4860 con 0x7f1a0815fcb0 2024-03-06T07:24:57.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.762+0000 7f1a0dc29640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a080b0910 con 0x7f19f00039b0 2024-03-06T07:24:57.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.762+0000 7f1a0dc29640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a080d49e0 con 0x7f1a0815f120 2024-03-06T07:24:57.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.762+0000 7f1a06ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a0815fcb0 0x7f1a080c9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.762+0000 7f1a06ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a0815fcb0 0x7f1a080c9b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41602/0 (socket says 172.21.15.83:41602) 2024-03-06T07:24:57.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.762+0000 7f1a06ffd640 1 -- 172.21.15.83:0/1728513551 learned_addr learned my addr 172.21.15.83:0/1728513551 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:57.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.762+0000 7f1a077fe640 1 --2- 172.21.15.83:0/1728513551 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f19f00039b0 0x7f1a080b9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.762+0000 7f1a07fff640 1 --2- 172.21.15.83:0/1728513551 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1a0815f120 0x7f1a0815f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.762+0000 7f1a06ffd640 1 -- 172.21.15.83:0/1728513551 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1a0815f120 msgr2=0x7f1a0815f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.762+0000 7f1a06ffd640 1 --2- 172.21.15.83:0/1728513551 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1a0815f120 0x7f1a0815f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.762+0000 7f1a06ffd640 1 -- 172.21.15.83:0/1728513551 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f19f00039b0 msgr2=0x7f1a080b9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.762+0000 7f1a06ffd640 1 --2- 172.21.15.83:0/1728513551 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f19f00039b0 0x7f1a080b9160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.762+0000 7f1a06ffd640 1 -- 172.21.15.83:0/1728513551 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a080b96a0 con 0x7f1a0815fcb0 2024-03-06T07:24:57.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.762+0000 7f1a077fe640 1 --2- 172.21.15.83:0/1728513551 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f19f00039b0 0x7f1a080b9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:57.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.762+0000 7f1a07fff640 1 --2- 172.21.15.83:0/1728513551 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1a0815f120 0x7f1a0815f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:57.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.763+0000 7f1a06ffd640 1 --2- 172.21.15.83:0/1728513551 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a0815fcb0 0x7f1a080c9b70 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f19f80072c0 tx=0x7f19f8007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.763+0000 7f19effff640 1 -- 172.21.15.83:0/1728513551 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f800f960 con 0x7f1a0815fcb0 2024-03-06T07:24:57.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.763+0000 7f19effff640 1 -- 172.21.15.83:0/1728513551 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19f8018070 con 0x7f1a0815fcb0 2024-03-06T07:24:57.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.763+0000 7f1a0dc29640 1 -- 172.21.15.83:0/1728513551 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a080b8260 con 0x7f1a0815fcb0 2024-03-06T07:24:57.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.763+0000 7f19effff640 1 -- 172.21.15.83:0/1728513551 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f800fd40 con 0x7f1a0815fcb0 2024-03-06T07:24:57.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.763+0000 7f1a0dc29640 1 -- 172.21.15.83:0/1728513551 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a080b7780 con 0x7f1a0815fcb0 2024-03-06T07:24:57.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.764+0000 7f1a0dc29640 1 -- 172.21.15.83:0/1728513551 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a0806d0b0 con 0x7f1a0815fcb0 2024-03-06T07:24:57.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.764+0000 7f19effff640 1 -- 172.21.15.83:0/1728513551 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f19f801d460 con 0x7f1a0815fcb0 2024-03-06T07:24:57.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.765+0000 7f19effff640 1 --2- 172.21.15.83:0/1728513551 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f19f406b4d0 0x7f19f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.765+0000 7f1a077fe640 1 --2- 172.21.15.83:0/1728513551 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f19f406b4d0 0x7f19f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.765+0000 7f19effff640 1 -- 172.21.15.83:0/1728513551 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f19f808a380 con 0x7f1a0815fcb0 2024-03-06T07:24:57.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.765+0000 7f1a077fe640 1 --2- 172.21.15.83:0/1728513551 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f19f406b4d0 0x7f19f406d990 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f19fc036000 tx=0x7f19fc030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.773+0000 7f19effff640 1 -- 172.21.15.83:0/1728513551 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1a0806d0b0 con 0x7f1a0815fcb0 2024-03-06T07:24:57.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.950+0000 7f1a0dc29640 1 -- 172.21.15.83:0/1728513551 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1a08068ff0 con 0x7f1a0815fcb0 2024-03-06T07:24:57.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.951+0000 7f19effff640 1 -- 172.21.15.83:0/1728513551 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v57) v1 ==== 93+0+10 (secure 0 0 0) 0x7f19f8059200 con 0x7f1a0815fcb0 2024-03-06T07:24:57.953 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:24:57.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.953+0000 7f1a0dc29640 1 -- 172.21.15.83:0/1728513551 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f19f406b4d0 msgr2=0x7f19f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.953+0000 7f1a0dc29640 1 --2- 172.21.15.83:0/1728513551 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f19f406b4d0 0x7f19f406d990 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f19fc036000 tx=0x7f19fc030000 comp rx=0 tx=0).stop 2024-03-06T07:24:57.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.953+0000 7f1a0dc29640 1 -- 172.21.15.83:0/1728513551 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a0815fcb0 msgr2=0x7f1a080c9b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.953+0000 7f1a0dc29640 1 --2- 172.21.15.83:0/1728513551 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a0815fcb0 0x7f1a080c9b70 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f19f80072c0 tx=0x7f19f8007780 comp rx=0 tx=0).stop 2024-03-06T07:24:57.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.953+0000 7f1a0dc29640 1 -- 172.21.15.83:0/1728513551 shutdown_connections 2024-03-06T07:24:57.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.953+0000 7f1a0dc29640 1 --2- 172.21.15.83:0/1728513551 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f19f406b4d0 0x7f19f406d990 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.953+0000 7f1a0dc29640 1 --2- 172.21.15.83:0/1728513551 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1a0815f120 0x7f1a0815f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.953+0000 7f1a0dc29640 1 --2- 172.21.15.83:0/1728513551 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a0815fcb0 0x7f1a080c9b70 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.953+0000 7f1a0dc29640 1 --2- 172.21.15.83:0/1728513551 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f19f00039b0 0x7f1a080b9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.953+0000 7f1a0dc29640 1 -- 172.21.15.83:0/1728513551 >> 172.21.15.83:0/1728513551 conn(0x7f1a080d3d40 msgr2=0x7f1a0806c2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:57.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.953+0000 7f1a0dc29640 1 -- 172.21.15.83:0/1728513551 shutdown_connections 2024-03-06T07:24:57.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:57.953+0000 7f1a0dc29640 1 -- 172.21.15.83:0/1728513551 wait complete. 2024-03-06T07:24:57.966 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:24:58.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.072+0000 7fa39c64e640 1 Processor -- start 2024-03-06T07:24:58.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.072+0000 7fa39c64e640 1 -- start start 2024-03-06T07:24:58.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.073+0000 7fa39c64e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fa3940d10c0 0x7fa3940d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.073+0000 7fa39c64e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa3940d14a0 0x7fa39415e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.073+0000 7fa39c64e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3940cfa50 con 0x7fa39415eb00 2024-03-06T07:24:58.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.073+0000 7fa39c64e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa39415c050 con 0x7fa3940d10c0 2024-03-06T07:24:58.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.073+0000 7fa39c64e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3940d0060 con 0x7fa3940d14a0 2024-03-06T07:24:58.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.073+0000 7fa39a3ea640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fa3940d10c0 0x7fa3940d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.073+0000 7fa399be9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa3940d14a0 0x7fa39415e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.073+0000 7fa39a3ea640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fa3940d10c0 0x7fa3940d04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:37252/0 (socket says 172.21.15.83:37252) 2024-03-06T07:24:58.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.073+0000 7fa39a3ea640 1 -- 172.21.15.83:0/3770095729 learned_addr learned my addr 172.21.15.83:0/3770095729 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:58.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.073+0000 7fa399be9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa3940d14a0 0x7fa39415e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:56248/0 (socket says 172.21.15.83:56248) 2024-03-06T07:24:58.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.073+0000 7fa39abeb640 1 --1- 172.21.15.83:0/3770095729 >> v1:172.21.15.183:6789/0 conn(0x7fa39415eb00 0x7fa3940cae80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:50912/0 (socket says 172.21.15.83:50912) 2024-03-06T07:24:58.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.073+0000 7fa39a3ea640 1 -- 172.21.15.83:0/3770095729 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa3940d14a0 msgr2=0x7fa39415e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:58.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.073+0000 7fa39a3ea640 1 --2- 172.21.15.83:0/3770095729 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa3940d14a0 0x7fa39415e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.073+0000 7fa39a3ea640 1 -- 172.21.15.83:0/3770095729 >> v1:172.21.15.183:6789/0 conn(0x7fa39415eb00 legacy=0x7fa3940cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.073+0000 7fa39a3ea640 1 -- 172.21.15.83:0/3770095729 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3940d0a30 con 0x7fa3940d10c0 2024-03-06T07:24:58.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.073+0000 7fa399be9640 1 --2- 172.21.15.83:0/3770095729 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa3940d14a0 0x7fa39415e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:58.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.074+0000 7fa39a3ea640 1 --2- 172.21.15.83:0/3770095729 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa3940d10c0 0x7fa3940d04f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa3940d01e0 tx=0x7fa38402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13b0c9aea84eedca server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:58.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.074+0000 7fa3993e8640 1 -- 172.21.15.83:0/3770095729 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa384034070 con 0x7fa3940d10c0 2024-03-06T07:24:58.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.074+0000 7fa3993e8640 1 -- 172.21.15.83:0/3770095729 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa384030610 con 0x7fa3940d10c0 2024-03-06T07:24:58.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.074+0000 7fa3993e8640 1 -- 172.21.15.83:0/3770095729 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3840309e0 con 0x7fa3940d10c0 2024-03-06T07:24:58.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.075+0000 7fa39c64e640 1 -- 172.21.15.83:0/3770095729 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa3940d10c0 msgr2=0x7fa3940d04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.075+0000 7fa39c64e640 1 --2- 172.21.15.83:0/3770095729 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa3940d10c0 0x7fa3940d04f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa3940d01e0 tx=0x7fa38402b960 comp rx=0 tx=0).stop 2024-03-06T07:24:58.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.075+0000 7fa39c64e640 1 -- 172.21.15.83:0/3770095729 shutdown_connections 2024-03-06T07:24:58.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.075+0000 7fa39c64e640 1 --2- 172.21.15.83:0/3770095729 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa3940d14a0 0x7fa39415e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.075+0000 7fa39c64e640 1 --2- 172.21.15.83:0/3770095729 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa3940d10c0 0x7fa3940d04f0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.075+0000 7fa39c64e640 1 -- 172.21.15.83:0/3770095729 >> 172.21.15.83:0/3770095729 conn(0x7fa3940b3800 msgr2=0x7fa3940b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:58.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.075+0000 7fa39c64e640 1 -- 172.21.15.83:0/3770095729 shutdown_connections 2024-03-06T07:24:58.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.075+0000 7fa39c64e640 1 -- 172.21.15.83:0/3770095729 wait complete. 2024-03-06T07:24:58.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.076+0000 7fa39c64e640 1 Processor -- start 2024-03-06T07:24:58.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.076+0000 7fa39c64e640 1 -- start start 2024-03-06T07:24:58.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.077+0000 7fa39c64e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3940d10c0 0x7fa39415b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.077+0000 7fa39c64e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa3940d14a0 0x7fa3940cea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.077+0000 7fa39c64e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa39415eb00 0x7fa394160410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.077+0000 7fa39c64e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa39415f440 con 0x7fa3940d10c0 2024-03-06T07:24:58.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.077+0000 7fa39c64e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa39415f160 con 0x7fa3940d14a0 2024-03-06T07:24:58.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.077+0000 7fa39c64e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3940cfcf0 con 0x7fa39415eb00 2024-03-06T07:24:58.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.077+0000 7fa39a3ea640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3940d10c0 0x7fa39415b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.077+0000 7fa39a3ea640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3940d10c0 0x7fa39415b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41604/0 (socket says 172.21.15.83:41604) 2024-03-06T07:24:58.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.077+0000 7fa39a3ea640 1 -- 172.21.15.83:0/3614395030 learned_addr learned my addr 172.21.15.83:0/3614395030 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:58.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.077+0000 7fa39abeb640 1 --2- 172.21.15.83:0/3614395030 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa39415eb00 0x7fa394160410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.077+0000 7fa399be9640 1 --2- 172.21.15.83:0/3614395030 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa3940d14a0 0x7fa3940cea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.078+0000 7fa39a3ea640 1 -- 172.21.15.83:0/3614395030 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa39415eb00 msgr2=0x7fa394160410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.078+0000 7fa39a3ea640 1 --2- 172.21.15.83:0/3614395030 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa39415eb00 0x7fa394160410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.078+0000 7fa39a3ea640 1 -- 172.21.15.83:0/3614395030 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa3940d14a0 msgr2=0x7fa3940cea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.078+0000 7fa39a3ea640 1 --2- 172.21.15.83:0/3614395030 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa3940d14a0 0x7fa3940cea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.078+0000 7fa39a3ea640 1 -- 172.21.15.83:0/3614395030 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3940cefd0 con 0x7fa3940d10c0 2024-03-06T07:24:58.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.078+0000 7fa399be9640 1 --2- 172.21.15.83:0/3614395030 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa3940d14a0 0x7fa3940cea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:58.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.078+0000 7fa39abeb640 1 --2- 172.21.15.83:0/3614395030 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa39415eb00 0x7fa394160410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:58.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.078+0000 7fa39a3ea640 1 --2- 172.21.15.83:0/3614395030 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3940d10c0 0x7fa39415b750 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fa3840097c0 tx=0x7fa3840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:58.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.078+0000 7fa38affd640 1 -- 172.21.15.83:0/3614395030 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa384034040 con 0x7fa3940d10c0 2024-03-06T07:24:58.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.078+0000 7fa38affd640 1 -- 172.21.15.83:0/3614395030 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa384039a90 con 0x7fa3940d10c0 2024-03-06T07:24:58.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.078+0000 7fa39c64e640 1 -- 172.21.15.83:0/3614395030 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3940b4e70 con 0x7fa3940d10c0 2024-03-06T07:24:58.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.078+0000 7fa38affd640 1 -- 172.21.15.83:0/3614395030 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3840433f0 con 0x7fa3940d10c0 2024-03-06T07:24:58.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.078+0000 7fa39c64e640 1 -- 172.21.15.83:0/3614395030 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3940b5430 con 0x7fa3940d10c0 2024-03-06T07:24:58.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.079+0000 7fa39c64e640 1 -- 172.21.15.83:0/3614395030 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa39409fe60 con 0x7fa3940d10c0 2024-03-06T07:24:58.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.079+0000 7fa38affd640 1 -- 172.21.15.83:0/3614395030 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa38404a060 con 0x7fa3940d10c0 2024-03-06T07:24:58.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.080+0000 7fa38affd640 1 --2- 172.21.15.83:0/3614395030 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa37806b500 0x7fa37806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.080+0000 7fa38affd640 1 -- 172.21.15.83:0/3614395030 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa3840abe20 con 0x7fa3940d10c0 2024-03-06T07:24:58.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.080+0000 7fa399be9640 1 --2- 172.21.15.83:0/3614395030 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa37806b500 0x7fa37806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.080+0000 7fa399be9640 1 --2- 172.21.15.83:0/3614395030 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa37806b500 0x7fa37806d9c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fa394152ff0 tx=0x7fa390008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:58.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.084+0000 7fa38affd640 1 -- 172.21.15.83:0/3614395030 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa3940cefd0 con 0x7fa3940d10c0 2024-03-06T07:24:58.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.260+0000 7fa39c64e640 1 -- 172.21.15.83:0/3614395030 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa39414fe60 con 0x7fa3940d10c0 2024-03-06T07:24:58.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.260+0000 7fa38affd640 1 -- 172.21.15.83:0/3614395030 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v57) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa38403e070 con 0x7fa3940d10c0 2024-03-06T07:24:58.262 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-03-06T07:24:58.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.262+0000 7fa39c64e640 1 -- 172.21.15.83:0/3614395030 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa37806b500 msgr2=0x7fa37806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.262+0000 7fa39c64e640 1 --2- 172.21.15.83:0/3614395030 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa37806b500 0x7fa37806d9c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fa394152ff0 tx=0x7fa390008040 comp rx=0 tx=0).stop 2024-03-06T07:24:58.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.262+0000 7fa39c64e640 1 -- 172.21.15.83:0/3614395030 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3940d10c0 msgr2=0x7fa39415b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.262+0000 7fa39c64e640 1 --2- 172.21.15.83:0/3614395030 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3940d10c0 0x7fa39415b750 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fa3840097c0 tx=0x7fa3840046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.262+0000 7fa39c64e640 1 -- 172.21.15.83:0/3614395030 shutdown_connections 2024-03-06T07:24:58.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.262+0000 7fa39c64e640 1 --2- 172.21.15.83:0/3614395030 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa37806b500 0x7fa37806d9c0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.262+0000 7fa39c64e640 1 --2- 172.21.15.83:0/3614395030 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa39415eb00 0x7fa394160410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.262+0000 7fa39c64e640 1 --2- 172.21.15.83:0/3614395030 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa3940d14a0 0x7fa3940cea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.262+0000 7fa39c64e640 1 --2- 172.21.15.83:0/3614395030 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3940d10c0 0x7fa39415b750 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.262+0000 7fa39c64e640 1 -- 172.21.15.83:0/3614395030 >> 172.21.15.83:0/3614395030 conn(0x7fa3940b3800 msgr2=0x7fa39415f760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:58.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.262+0000 7fa39c64e640 1 -- 172.21.15.83:0/3614395030 shutdown_connections 2024-03-06T07:24:58.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.262+0000 7fa39c64e640 1 -- 172.21.15.83:0/3614395030 wait complete. 2024-03-06T07:24:58.275 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:24:58.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.378+0000 7f6885f0c640 1 Processor -- start 2024-03-06T07:24:58.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.378+0000 7f6885f0c640 1 -- start start 2024-03-06T07:24:58.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.379+0000 7f6885f0c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f68800cfa90 0x7f68800cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.379+0000 7f6885f0c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f68800b33a0 0x7f68800b37c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.379+0000 7f6885f0c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6880158ac0 con 0x7f68801586e0 2024-03-06T07:24:58.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.379+0000 7f6885f0c640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f688015b2a0 con 0x7f68800cfa90 2024-03-06T07:24:58.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.379+0000 7f6885f0c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f688015b540 con 0x7f68800b33a0 2024-03-06T07:24:58.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.379+0000 7f687effd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f68800cfa90 0x7f68800cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.379+0000 7f687f7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f68801586e0 0x7f68800aea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49750/0 (socket says 172.21.15.83:49750) 2024-03-06T07:24:58.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.379+0000 7f687f7fe640 1 -- 172.21.15.83:0/1077031456 learned_addr learned my addr 172.21.15.83:0/1077031456 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:24:58.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.379+0000 7f687ffff640 1 --2- 172.21.15.83:0/1077031456 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f68800b33a0 0x7f68800b37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.379+0000 7f687effd640 1 -- 172.21.15.83:0/1077031456 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f68800b33a0 msgr2=0x7f68800b37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:58.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.379+0000 7f687effd640 1 --2- 172.21.15.83:0/1077031456 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f68800b33a0 0x7f68800b37c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.379+0000 7f687effd640 1 -- 172.21.15.83:0/1077031456 >> v1:172.21.15.83:6789/0 conn(0x7f68801586e0 legacy=0x7f68800aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.379+0000 7f687e7fc640 1 -- 172.21.15.83:0/1077031456 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 59465712 0 0) 0x7f6880158ac0 con 0x7f68801586e0 2024-03-06T07:24:58.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.379+0000 7f687effd640 1 -- 172.21.15.83:0/1077031456 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f688015b820 con 0x7f68800cfa90 2024-03-06T07:24:58.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.380+0000 7f687effd640 1 --2- 172.21.15.83:0/1077031456 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68800cfa90 0x7f68800cff40 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f68700097a0 tx=0x7f687002dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=600f38f00e8af027 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:58.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.380+0000 7f687e7fc640 1 -- 172.21.15.83:0/1077031456 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687002f8a0 con 0x7f68800cfa90 2024-03-06T07:24:58.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.380+0000 7f687e7fc640 1 -- 172.21.15.83:0/1077031456 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f687002fa40 con 0x7f68800cfa90 2024-03-06T07:24:58.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.380+0000 7f687e7fc640 1 -- 172.21.15.83:0/1077031456 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687002fe10 con 0x7f68800cfa90 2024-03-06T07:24:58.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.381+0000 7f6885f0c640 1 -- 172.21.15.83:0/1077031456 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68800cfa90 msgr2=0x7f68800cff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.381+0000 7f6885f0c640 1 --2- 172.21.15.83:0/1077031456 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68800cfa90 0x7f68800cff40 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f68700097a0 tx=0x7f687002dae0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.381+0000 7f6885f0c640 1 -- 172.21.15.83:0/1077031456 shutdown_connections 2024-03-06T07:24:58.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.381+0000 7f6885f0c640 1 --2- 172.21.15.83:0/1077031456 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f68800b33a0 0x7f68800b37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.381+0000 7f6885f0c640 1 --2- 172.21.15.83:0/1077031456 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68800cfa90 0x7f68800cff40 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.381+0000 7f6885f0c640 1 -- 172.21.15.83:0/1077031456 >> 172.21.15.83:0/1077031456 conn(0x7f688006b190 msgr2=0x7f688006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:58.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.381+0000 7f6885f0c640 1 -- 172.21.15.83:0/1077031456 shutdown_connections 2024-03-06T07:24:58.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.381+0000 7f6885f0c640 1 -- 172.21.15.83:0/1077031456 wait complete. 2024-03-06T07:24:58.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.382+0000 7f6885f0c640 1 Processor -- start 2024-03-06T07:24:58.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.382+0000 7f6885f0c640 1 -- start start 2024-03-06T07:24:58.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.383+0000 7f6885f0c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68800b33a0 0x7f688006cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.383+0000 7f6885f0c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f68800cfa90 0x7f68801e5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.383+0000 7f6885f0c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f68801586e0 0x7f68801e5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.383+0000 7f6885f0c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68800af150 con 0x7f68800b33a0 2024-03-06T07:24:58.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.383+0000 7f6885f0c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f688015bde0 con 0x7f68800cfa90 2024-03-06T07:24:58.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.383+0000 7f6885f0c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6880071350 con 0x7f68801586e0 2024-03-06T07:24:58.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.383+0000 7f687f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68800b33a0 0x7f688006cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.383+0000 7f687f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68800b33a0 0x7f688006cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41624/0 (socket says 172.21.15.83:41624) 2024-03-06T07:24:58.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.383+0000 7f687f7fe640 1 -- 172.21.15.83:0/1603292493 learned_addr learned my addr 172.21.15.83:0/1603292493 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:58.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.383+0000 7f687f7fe640 1 -- 172.21.15.83:0/1603292493 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f68801586e0 msgr2=0x7f68801e5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.383+0000 7f687ffff640 1 --2- 172.21.15.83:0/1603292493 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f68801586e0 0x7f68801e5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.383+0000 7f687effd640 1 --2- 172.21.15.83:0/1603292493 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f68800cfa90 0x7f68801e5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.383+0000 7f687f7fe640 1 --2- 172.21.15.83:0/1603292493 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f68801586e0 0x7f68801e5ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.383+0000 7f687f7fe640 1 -- 172.21.15.83:0/1603292493 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f68800cfa90 msgr2=0x7f68801e5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.383+0000 7f687f7fe640 1 --2- 172.21.15.83:0/1603292493 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f68800cfa90 0x7f68801e5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.383+0000 7f687f7fe640 1 -- 172.21.15.83:0/1603292493 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f688006d7a0 con 0x7f68800b33a0 2024-03-06T07:24:58.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.383+0000 7f687effd640 1 --2- 172.21.15.83:0/1603292493 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f68800cfa90 0x7f68801e5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:58.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.383+0000 7f687f7fe640 1 --2- 172.21.15.83:0/1603292493 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68800b33a0 0x7f688006cd60 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f686800c430 tx=0x7f686800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:58.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.384+0000 7f6877fff640 1 -- 172.21.15.83:0/1603292493 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68680074e0 con 0x7f68800b33a0 2024-03-06T07:24:58.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.384+0000 7f6877fff640 1 -- 172.21.15.83:0/1603292493 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68680047e0 con 0x7f68800b33a0 2024-03-06T07:24:58.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.384+0000 7f6885f0c640 1 -- 172.21.15.83:0/1603292493 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f688006da80 con 0x7f68800b33a0 2024-03-06T07:24:58.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.384+0000 7f6877fff640 1 -- 172.21.15.83:0/1603292493 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6868004bd0 con 0x7f68800b33a0 2024-03-06T07:24:58.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.384+0000 7f6885f0c640 1 -- 172.21.15.83:0/1603292493 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f688006e040 con 0x7f68800b33a0 2024-03-06T07:24:58.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.384+0000 7f6885f0c640 1 -- 172.21.15.83:0/1603292493 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68800681e0 con 0x7f68800b33a0 2024-03-06T07:24:58.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.385+0000 7f6877fff640 1 -- 172.21.15.83:0/1603292493 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6868011070 con 0x7f68800b33a0 2024-03-06T07:24:58.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.385+0000 7f6877fff640 1 --2- 172.21.15.83:0/1603292493 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f686006b4d0 0x7f686006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.385+0000 7f6877fff640 1 -- 172.21.15.83:0/1603292493 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6868089f50 con 0x7f68800b33a0 2024-03-06T07:24:58.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.386+0000 7f687effd640 1 --2- 172.21.15.83:0/1603292493 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f686006b4d0 0x7f686006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.386+0000 7f687effd640 1 --2- 172.21.15.83:0/1603292493 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f686006b4d0 0x7f686006d990 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f6870038000 tx=0x7f6870008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:58.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.390+0000 7f6877fff640 1 -- 172.21.15.83:0/1603292493 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f68800681e0 con 0x7f68800b33a0 2024-03-06T07:24:58.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.567+0000 7f6885f0c640 1 -- 172.21.15.83:0/1603292493 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6880068ff0 con 0x7f68800b33a0 2024-03-06T07:24:58.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.567+0000 7f6877fff640 1 -- 172.21.15.83:0/1603292493 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v57) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6868058d40 con 0x7f68800b33a0 2024-03-06T07:24:58.569 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 64 2024-03-06T07:24:58.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.569+0000 7f6885f0c640 1 -- 172.21.15.83:0/1603292493 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f686006b4d0 msgr2=0x7f686006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.569+0000 7f6885f0c640 1 --2- 172.21.15.83:0/1603292493 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f686006b4d0 0x7f686006d990 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f6870038000 tx=0x7f6870008030 comp rx=0 tx=0).stop 2024-03-06T07:24:58.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.569+0000 7f6885f0c640 1 -- 172.21.15.83:0/1603292493 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68800b33a0 msgr2=0x7f688006cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.569+0000 7f6885f0c640 1 --2- 172.21.15.83:0/1603292493 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68800b33a0 0x7f688006cd60 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f686800c430 tx=0x7f686800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.569+0000 7f6885f0c640 1 -- 172.21.15.83:0/1603292493 shutdown_connections 2024-03-06T07:24:58.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.569+0000 7f6885f0c640 1 --2- 172.21.15.83:0/1603292493 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f686006b4d0 0x7f686006d990 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.569+0000 7f6885f0c640 1 --2- 172.21.15.83:0/1603292493 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f68801586e0 0x7f68801e5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.569+0000 7f6885f0c640 1 --2- 172.21.15.83:0/1603292493 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f68800cfa90 0x7f68801e5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.569+0000 7f6885f0c640 1 --2- 172.21.15.83:0/1603292493 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68800b33a0 0x7f688006cd60 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.569+0000 7f6885f0c640 1 -- 172.21.15.83:0/1603292493 >> 172.21.15.83:0/1603292493 conn(0x7f688006b190 msgr2=0x7f688006b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:58.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.569+0000 7f6885f0c640 1 -- 172.21.15.83:0/1603292493 shutdown_connections 2024-03-06T07:24:58.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.569+0000 7f6885f0c640 1 -- 172.21.15.83:0/1603292493 wait complete. 2024-03-06T07:24:58.582 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:58.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.681+0000 7f31247ae640 1 Processor -- start 2024-03-06T07:24:58.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.681+0000 7f31247ae640 1 -- start start 2024-03-06T07:24:58.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.682+0000 7f31247ae640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f311c15dfe0 0x7f311c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.682+0000 7f31247ae640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f311c0b3430 con 0x7f311c0d29c0 2024-03-06T07:24:58.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.682+0000 7f31247ae640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f311c0b3710 con 0x7f311c0cf830 2024-03-06T07:24:58.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.682+0000 7f31247ae640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f311c0b31e0 con 0x7f311c15dfe0 2024-03-06T07:24:58.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.682+0000 7f3122d4b640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f311c0d29c0 0x7f311c0b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49766/0 (socket says 172.21.15.83:49766) 2024-03-06T07:24:58.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.683+0000 7f3122d4b640 1 -- 172.21.15.83:0/4053070197 learned_addr learned my addr 172.21.15.83:0/4053070197 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:24:58.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.683+0000 7f312254a640 1 --2- 172.21.15.83:0/4053070197 >> v2:172.21.15.183:3300/0 conn(0x7f311c15dfe0 0x7f311c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.683+0000 7f3121548640 1 -- 172.21.15.83:0/4053070197 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 64171352 0 0) 0x7f311c0b3430 con 0x7f311c0d29c0 2024-03-06T07:24:58.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.683+0000 7f3121548640 1 -- 172.21.15.83:0/4053070197 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3108003520 con 0x7f311c0d29c0 2024-03-06T07:24:58.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.683+0000 7f3121548640 1 -- 172.21.15.83:0/4053070197 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2606331681 0 0) 0x7f311c0b3710 con 0x7f311c0cf830 2024-03-06T07:24:58.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.683+0000 7f3121548640 1 -- 172.21.15.83:0/4053070197 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f311c0b3430 con 0x7f311c0cf830 2024-03-06T07:24:58.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.683+0000 7f312254a640 1 -- 172.21.15.83:0/4053070197 >> v1:172.21.15.183:6789/0 conn(0x7f311c0cf830 legacy=0x7f311c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.683+0000 7f312254a640 1 -- 172.21.15.83:0/4053070197 >> v1:172.21.15.83:6789/0 conn(0x7f311c0d29c0 legacy=0x7f311c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.683+0000 7f312254a640 1 -- 172.21.15.83:0/4053070197 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f311c0b39f0 con 0x7f311c15dfe0 2024-03-06T07:24:58.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.684+0000 7f312254a640 1 --2- 172.21.15.83:0/4053070197 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f311c15dfe0 0x7f311c0cf2f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f311c0b3360 tx=0x7f311802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac72882d74cd63bb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:58.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.684+0000 7f3121548640 1 -- 172.21.15.83:0/4053070197 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3118034070 con 0x7f311c15dfe0 2024-03-06T07:24:58.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.684+0000 7f3121548640 1 -- 172.21.15.83:0/4053070197 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3118030610 con 0x7f311c15dfe0 2024-03-06T07:24:58.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.684+0000 7f3121548640 1 -- 172.21.15.83:0/4053070197 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31180309c0 con 0x7f311c15dfe0 2024-03-06T07:24:58.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.684+0000 7f31247ae640 1 -- 172.21.15.83:0/4053070197 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f311c15dfe0 msgr2=0x7f311c0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.684+0000 7f31247ae640 1 --2- 172.21.15.83:0/4053070197 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f311c15dfe0 0x7f311c0cf2f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f311c0b3360 tx=0x7f311802b960 comp rx=0 tx=0).stop 2024-03-06T07:24:58.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.684+0000 7f31247ae640 1 -- 172.21.15.83:0/4053070197 shutdown_connections 2024-03-06T07:24:58.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.684+0000 7f31247ae640 1 --2- 172.21.15.83:0/4053070197 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f311c15dfe0 0x7f311c0cf2f0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.685+0000 7f31247ae640 1 -- 172.21.15.83:0/4053070197 >> 172.21.15.83:0/4053070197 conn(0x7f311c0d3cd0 msgr2=0x7f311c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:58.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.685+0000 7f31247ae640 1 -- 172.21.15.83:0/4053070197 shutdown_connections 2024-03-06T07:24:58.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.685+0000 7f31247ae640 1 -- 172.21.15.83:0/4053070197 wait complete. 2024-03-06T07:24:58.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.686+0000 7f31247ae640 1 Processor -- start 2024-03-06T07:24:58.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.686+0000 7f31247ae640 1 -- start start 2024-03-06T07:24:58.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.686+0000 7f31247ae640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f311c0cf830 0x7f311c0b9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.686+0000 7f31247ae640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f311c0d29c0 0x7f311c0cb790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.686+0000 7f31247ae640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f311c15dfe0 0x7f311c0b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.686+0000 7f31247ae640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311c15e3c0 con 0x7f311c0d29c0 2024-03-06T07:24:58.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.686+0000 7f31247ae640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311c0b3d60 con 0x7f311c0cf830 2024-03-06T07:24:58.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.686+0000 7f31247ae640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f311c15e540 con 0x7f311c15dfe0 2024-03-06T07:24:58.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.687+0000 7f3121d49640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f311c0d29c0 0x7f311c0cb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.687+0000 7f3121d49640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f311c0d29c0 0x7f311c0cb790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41636/0 (socket says 172.21.15.83:41636) 2024-03-06T07:24:58.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.687+0000 7f3121d49640 1 -- 172.21.15.83:0/3900809197 learned_addr learned my addr 172.21.15.83:0/3900809197 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:58.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.687+0000 7f312254a640 1 --2- 172.21.15.83:0/3900809197 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f311c0cf830 0x7f311c0b9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.687+0000 7f3122d4b640 1 --2- 172.21.15.83:0/3900809197 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f311c15dfe0 0x7f311c0b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.687+0000 7f3121d49640 1 -- 172.21.15.83:0/3900809197 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f311c15dfe0 msgr2=0x7f311c0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.687+0000 7f3121d49640 1 --2- 172.21.15.83:0/3900809197 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f311c15dfe0 0x7f311c0b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.687+0000 7f3121d49640 1 -- 172.21.15.83:0/3900809197 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f311c0cf830 msgr2=0x7f311c0b9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.687+0000 7f3121d49640 1 --2- 172.21.15.83:0/3900809197 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f311c0cf830 0x7f311c0b9830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.687+0000 7f3121d49640 1 -- 172.21.15.83:0/3900809197 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f311c0b6a00 con 0x7f311c0d29c0 2024-03-06T07:24:58.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.687+0000 7f3122d4b640 1 --2- 172.21.15.83:0/3900809197 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f311c15dfe0 0x7f311c0b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:58.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.687+0000 7f312254a640 1 --2- 172.21.15.83:0/3900809197 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f311c0cf830 0x7f311c0b9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:58.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.687+0000 7f3121d49640 1 --2- 172.21.15.83:0/3900809197 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f311c0d29c0 0x7f311c0cb790 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f310c00c400 tx=0x7f310c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:58.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.687+0000 7f3112ffd640 1 -- 172.21.15.83:0/3900809197 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310c017070 con 0x7f311c0d29c0 2024-03-06T07:24:58.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.687+0000 7f3112ffd640 1 -- 172.21.15.83:0/3900809197 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f310c0052d0 con 0x7f311c0d29c0 2024-03-06T07:24:58.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.687+0000 7f3112ffd640 1 -- 172.21.15.83:0/3900809197 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310c003810 con 0x7f311c0d29c0 2024-03-06T07:24:58.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.687+0000 7f31247ae640 1 -- 172.21.15.83:0/3900809197 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f311c0b6ce0 con 0x7f311c0d29c0 2024-03-06T07:24:58.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.687+0000 7f31247ae640 1 -- 172.21.15.83:0/3900809197 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f311c0b72a0 con 0x7f311c0d29c0 2024-03-06T07:24:58.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.688+0000 7f31247ae640 1 -- 172.21.15.83:0/3900809197 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f311c04f330 con 0x7f311c0d29c0 2024-03-06T07:24:58.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.689+0000 7f3112ffd640 1 -- 172.21.15.83:0/3900809197 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f310c01a070 con 0x7f311c0d29c0 2024-03-06T07:24:58.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.689+0000 7f3112ffd640 1 --2- 172.21.15.83:0/3900809197 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f310006b4d0 0x7f310006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.689+0000 7f3112ffd640 1 -- 172.21.15.83:0/3900809197 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f310c089590 con 0x7f311c0d29c0 2024-03-06T07:24:58.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.689+0000 7f312254a640 1 --2- 172.21.15.83:0/3900809197 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f310006b4d0 0x7f310006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.689+0000 7f312254a640 1 --2- 172.21.15.83:0/3900809197 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f310006b4d0 0x7f310006d990 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f3118036000 tx=0x7f311803b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:58.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.697+0000 7f3112ffd640 1 -- 172.21.15.83:0/3900809197 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f310c0584a0 con 0x7f311c0d29c0 2024-03-06T07:24:58.737 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:24:58.739 DEBUG:teuthology.orchestra.run.smithi183:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:24:58.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.934+0000 7f31247ae640 1 -- 172.21.15.83:0/3900809197 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f311c0d2350 con 0x7f311c0d29c0 2024-03-06T07:24:58.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.935+0000 7f3112ffd640 1 -- 172.21.15.83:0/3900809197 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f310c057da0 con 0x7f311c0d29c0 2024-03-06T07:24:58.937 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:24:58.937 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":25,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5665,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/3334609507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3334609507},{"type":"v1","addr":"172.21.15.183:6834","nonce":3334609507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5668,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/691189112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":691189112},{"type":"v1","addr":"172.21.15.183:6835","nonce":691189112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6081,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/3952571284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":3952571284},{"type":"v1","addr":"172.21.15.83:6836","nonce":3952571284}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:19.852157+0000","modified":"2024-03-06T07:24:21.879875+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6084},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6084":{"gid":6084,"name":"a","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/3237506441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3237506441},{"type":"v1","addr":"172.21.15.83:6837","nonce":3237506441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6084,"qdb_cluster":[6084]},"id":4}]} 2024-03-06T07:24:58.938 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 25 2024-03-06T07:24:58.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.937+0000 7f31247ae640 1 -- 172.21.15.83:0/3900809197 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f310006b4d0 msgr2=0x7f310006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.937+0000 7f31247ae640 1 --2- 172.21.15.83:0/3900809197 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f310006b4d0 0x7f310006d990 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f3118036000 tx=0x7f311803b040 comp rx=0 tx=0).stop 2024-03-06T07:24:58.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.937+0000 7f31247ae640 1 -- 172.21.15.83:0/3900809197 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f311c0d29c0 msgr2=0x7f311c0cb790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.937+0000 7f31247ae640 1 --2- 172.21.15.83:0/3900809197 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f311c0d29c0 0x7f311c0cb790 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f310c00c400 tx=0x7f310c00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.938+0000 7f31247ae640 1 -- 172.21.15.83:0/3900809197 shutdown_connections 2024-03-06T07:24:58.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.938+0000 7f31247ae640 1 --2- 172.21.15.83:0/3900809197 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f310006b4d0 0x7f310006d990 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.938+0000 7f31247ae640 1 --2- 172.21.15.83:0/3900809197 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f311c15dfe0 0x7f311c0b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.938+0000 7f31247ae640 1 --2- 172.21.15.83:0/3900809197 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f311c0d29c0 0x7f311c0cb790 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.938+0000 7f31247ae640 1 --2- 172.21.15.83:0/3900809197 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f311c0cf830 0x7f311c0b9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.938+0000 7f31247ae640 1 -- 172.21.15.83:0/3900809197 >> 172.21.15.83:0/3900809197 conn(0x7f311c0d3cd0 msgr2=0x7f311c0cec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:58.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.938+0000 7f31247ae640 1 -- 172.21.15.83:0/3900809197 shutdown_connections 2024-03-06T07:24:58.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:58.938+0000 7f31247ae640 1 -- 172.21.15.83:0/3900809197 wait complete. 2024-03-06T07:24:58.950 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:59.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.086+0000 7f8329804640 1 Processor -- start 2024-03-06T07:24:59.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.086+0000 7f8329804640 1 -- start start 2024-03-06T07:24:59.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.087+0000 7f8329804640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f832415dec0 0x7f832406fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.087+0000 7f8329804640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f832406c7b0 0x7f83240d17a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.087+0000 7f8329804640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f832406cf70 con 0x7f83240d1ce0 2024-03-06T07:24:59.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.087+0000 7f8329804640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f832415e2a0 con 0x7f832406c7b0 2024-03-06T07:24:59.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.087+0000 7f8329804640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f832406d250 con 0x7f832415dec0 2024-03-06T07:24:59.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.087+0000 7f831a7fc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f832406c7b0 0x7f83240d17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.087+0000 7f83237fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f83240d1ce0 0x7f8324161940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49772/0 (socket says 172.21.15.83:49772) 2024-03-06T07:24:59.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.087+0000 7f83237fe640 1 -- 172.21.15.83:0/3029137238 learned_addr learned my addr 172.21.15.83:0/3029137238 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:24:59.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.087+0000 7f8322ffd640 1 --2- 172.21.15.83:0/3029137238 >> v2:172.21.15.183:3300/0 conn(0x7f832415dec0 0x7f832406fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.087+0000 7f831a7fc640 1 -- 172.21.15.83:0/3029137238 >> v2:172.21.15.183:3300/0 conn(0x7f832415dec0 msgr2=0x7f832406fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:59.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.087+0000 7f831a7fc640 1 --2- 172.21.15.83:0/3029137238 >> v2:172.21.15.183:3300/0 conn(0x7f832415dec0 0x7f832406fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.088+0000 7f831a7fc640 1 -- 172.21.15.83:0/3029137238 >> v1:172.21.15.83:6789/0 conn(0x7f83240d1ce0 legacy=0x7f8324161940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.088+0000 7f831a7fc640 1 -- 172.21.15.83:0/3029137238 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f832406d530 con 0x7f832406c7b0 2024-03-06T07:24:59.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.088+0000 7f831a7fc640 1 --2- 172.21.15.83:0/3029137238 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f832406c7b0 0x7f83240d17a0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f8324068860 tx=0x7f830c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e600336d16636291 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.088+0000 7f83227fc640 1 -- 172.21.15.83:0/3029137238 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f830c035020 con 0x7f832406c7b0 2024-03-06T07:24:59.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.088+0000 7f83227fc640 1 -- 172.21.15.83:0/3029137238 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f830c031830 con 0x7f832406c7b0 2024-03-06T07:24:59.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.088+0000 7f83227fc640 1 -- 172.21.15.83:0/3029137238 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f830c031c00 con 0x7f832406c7b0 2024-03-06T07:24:59.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.089+0000 7f8329804640 1 -- 172.21.15.83:0/3029137238 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f832406c7b0 msgr2=0x7f83240d17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.089+0000 7f8329804640 1 --2- 172.21.15.83:0/3029137238 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f832406c7b0 0x7f83240d17a0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f8324068860 tx=0x7f830c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.089+0000 7f8329804640 1 -- 172.21.15.83:0/3029137238 shutdown_connections 2024-03-06T07:24:59.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.089+0000 7f8329804640 1 --2- 172.21.15.83:0/3029137238 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f832406c7b0 0x7f83240d17a0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.089+0000 7f8329804640 1 --2- 172.21.15.83:0/3029137238 >> v2:172.21.15.183:3300/0 conn(0x7f832415dec0 0x7f832406fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.089+0000 7f8329804640 1 -- 172.21.15.83:0/3029137238 >> 172.21.15.83:0/3029137238 conn(0x7f832409a9f0 msgr2=0x7f832409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:59.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.089+0000 7f8329804640 1 -- 172.21.15.83:0/3029137238 shutdown_connections 2024-03-06T07:24:59.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.089+0000 7f8329804640 1 -- 172.21.15.83:0/3029137238 wait complete. 2024-03-06T07:24:59.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.091+0000 7f8329804640 1 Processor -- start 2024-03-06T07:24:59.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.091+0000 7f8329804640 1 -- start start 2024-03-06T07:24:59.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.092+0000 7f8329804640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f832406c7b0 0x7f832409b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.092+0000 7f8329804640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f83240d1ce0 0x7f832409cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.092+0000 7f8329804640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f832415dec0 0x7f8324156de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.092+0000 7f8329804640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83240cfc60 con 0x7f832415dec0 2024-03-06T07:24:59.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.092+0000 7f8329804640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f832406cc40 con 0x7f832406c7b0 2024-03-06T07:24:59.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.092+0000 7f8329804640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83240cfde0 con 0x7f83240d1ce0 2024-03-06T07:24:59.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.092+0000 7f83237fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f832415dec0 0x7f8324156de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.092+0000 7f83237fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f832415dec0 0x7f8324156de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41656/0 (socket says 172.21.15.83:41656) 2024-03-06T07:24:59.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.092+0000 7f83237fe640 1 -- 172.21.15.83:0/2916032000 learned_addr learned my addr 172.21.15.83:0/2916032000 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:59.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.093+0000 7f8322ffd640 1 --2- 172.21.15.83:0/2916032000 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f832406c7b0 0x7f832409b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.093+0000 7f831a7fc640 1 --2- 172.21.15.83:0/2916032000 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f83240d1ce0 0x7f832409cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.093+0000 7f83237fe640 1 -- 172.21.15.83:0/2916032000 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f83240d1ce0 msgr2=0x7f832409cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.093+0000 7f83237fe640 1 --2- 172.21.15.83:0/2916032000 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f83240d1ce0 0x7f832409cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.093+0000 7f83237fe640 1 -- 172.21.15.83:0/2916032000 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f832406c7b0 msgr2=0x7f832409b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.093+0000 7f83237fe640 1 --2- 172.21.15.83:0/2916032000 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f832406c7b0 0x7f832409b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.093+0000 7f83237fe640 1 -- 172.21.15.83:0/2916032000 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8324157810 con 0x7f832415dec0 2024-03-06T07:24:59.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.093+0000 7f83237fe640 1 --2- 172.21.15.83:0/2916032000 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f832415dec0 0x7f8324156de0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f831c00cbc0 tx=0x7f831c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.093+0000 7f8320ff9640 1 -- 172.21.15.83:0/2916032000 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f831c017070 con 0x7f832415dec0 2024-03-06T07:24:59.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.093+0000 7f8320ff9640 1 -- 172.21.15.83:0/2916032000 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f831c00ace0 con 0x7f832415dec0 2024-03-06T07:24:59.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.093+0000 7f8329804640 1 -- 172.21.15.83:0/2916032000 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f832409be70 con 0x7f832415dec0 2024-03-06T07:24:59.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.093+0000 7f8320ff9640 1 -- 172.21.15.83:0/2916032000 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f831c012600 con 0x7f832415dec0 2024-03-06T07:24:59.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.093+0000 7f8329804640 1 -- 172.21.15.83:0/2916032000 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f832409c430 con 0x7f832415dec0 2024-03-06T07:24:59.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.094+0000 7f8329804640 1 -- 172.21.15.83:0/2916032000 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f832409c670 con 0x7f832415dec0 2024-03-06T07:24:59.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.094+0000 7f8320ff9640 1 -- 172.21.15.83:0/2916032000 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f831c01a070 con 0x7f832415dec0 2024-03-06T07:24:59.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.095+0000 7f8320ff9640 1 --2- 172.21.15.83:0/2916032000 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f830806b4d0 0x7f830806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.095+0000 7f8322ffd640 1 --2- 172.21.15.83:0/2916032000 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f830806b4d0 0x7f830806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.095+0000 7f8320ff9640 1 -- 172.21.15.83:0/2916032000 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f831c089de0 con 0x7f832415dec0 2024-03-06T07:24:59.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.095+0000 7f8322ffd640 1 --2- 172.21.15.83:0/2916032000 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f830806b4d0 0x7f830806d990 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f8314006fd0 tx=0x7f8314008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.099+0000 7f8320ff9640 1 -- 172.21.15.83:0/2916032000 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f831c058610 con 0x7f832415dec0 2024-03-06T07:24:59.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.293+0000 7f8329804640 1 -- 172.21.15.83:0/2916032000 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8324068ff0 con 0x7f832415dec0 2024-03-06T07:24:59.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.294+0000 7f8320ff9640 1 -- 172.21.15.83:0/2916032000 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+16677 (secure 0 0 0) 0x7f831c05c6b0 con 0x7f832415dec0 2024-03-06T07:24:59.297 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:24:59.297 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":57,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:24:21.525937+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:18.074366+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:18.831101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:59.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.297+0000 7f8329804640 1 -- 172.21.15.83:0/2916032000 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f830806b4d0 msgr2=0x7f830806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.297+0000 7f8329804640 1 --2- 172.21.15.83:0/2916032000 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f830806b4d0 0x7f830806d990 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f8314006fd0 tx=0x7f8314008040 comp rx=0 tx=0).stop 2024-03-06T07:24:59.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.297+0000 7f8329804640 1 -- 172.21.15.83:0/2916032000 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f832415dec0 msgr2=0x7f8324156de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.297+0000 7f8329804640 1 --2- 172.21.15.83:0/2916032000 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f832415dec0 0x7f8324156de0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f831c00cbc0 tx=0x7f831c00a430 comp rx=0 tx=0).stop 2024-03-06T07:24:59.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.297+0000 7f8329804640 1 -- 172.21.15.83:0/2916032000 shutdown_connections 2024-03-06T07:24:59.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.297+0000 7f8329804640 1 --2- 172.21.15.83:0/2916032000 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f830806b4d0 0x7f830806d990 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.297+0000 7f8329804640 1 --2- 172.21.15.83:0/2916032000 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f832415dec0 0x7f8324156de0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.297+0000 7f8329804640 1 --2- 172.21.15.83:0/2916032000 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f83240d1ce0 0x7f832409cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.297+0000 7f8329804640 1 --2- 172.21.15.83:0/2916032000 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f832406c7b0 0x7f832409b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.297+0000 7f8329804640 1 -- 172.21.15.83:0/2916032000 >> 172.21.15.83:0/2916032000 conn(0x7f832409a9f0 msgr2=0x7f83240b23a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:59.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.297+0000 7f8329804640 1 -- 172.21.15.83:0/2916032000 shutdown_connections 2024-03-06T07:24:59.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.297+0000 7f8329804640 1 -- 172.21.15.83:0/2916032000 wait complete. 2024-03-06T07:24:59.310 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:24:59.311 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:59.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.408+0000 7fbcb8440640 1 Processor -- start 2024-03-06T07:24:59.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.408+0000 7fbcb8440640 1 -- start start 2024-03-06T07:24:59.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.409+0000 7fbcb8440640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbcb015dfe0 0x7fbcb00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.409+0000 7fbcb8440640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fbcb00cf830 0x7fbcb00d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.409+0000 7fbcb8440640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcb00b3430 con 0x7fbcb00d27f0 2024-03-06T07:24:59.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.409+0000 7fbcb8440640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcb00b31e0 con 0x7fbcb015dfe0 2024-03-06T07:24:59.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.409+0000 7fbcb8440640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcb00b3d60 con 0x7fbcb00cf830 2024-03-06T07:24:59.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.410+0000 7fbcb61dc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbcb015dfe0 0x7fbcb00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.410+0000 7fbcb61dc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbcb015dfe0 0x7fbcb00cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41660/0 (socket says 172.21.15.83:41660) 2024-03-06T07:24:59.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.410+0000 7fbcb61dc640 1 -- 172.21.15.83:0/904591962 learned_addr learned my addr 172.21.15.83:0/904591962 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:59.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.410+0000 7fbcb59db640 1 --2- 172.21.15.83:0/904591962 >> v2:172.21.15.183:3300/0 conn(0x7fbcb00cf830 0x7fbcb00d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.410+0000 7fbcb61dc640 1 -- 172.21.15.83:0/904591962 >> v2:172.21.15.183:3300/0 conn(0x7fbcb00cf830 msgr2=0x7fbcb00d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:59.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.410+0000 7fbcb61dc640 1 --2- 172.21.15.83:0/904591962 >> v2:172.21.15.183:3300/0 conn(0x7fbcb00cf830 0x7fbcb00d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.410+0000 7fbcb61dc640 1 -- 172.21.15.83:0/904591962 >> v1:172.21.15.183:6789/0 conn(0x7fbcb00d27f0 legacy=0x7fbcb00d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.410+0000 7fbcb61dc640 1 -- 172.21.15.83:0/904591962 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcb00b3710 con 0x7fbcb015dfe0 2024-03-06T07:24:59.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.411+0000 7fbcb61dc640 1 --2- 172.21.15.83:0/904591962 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbcb015dfe0 0x7fbcb00cf2f0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fbcac009870 tx=0x7fbcac02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfc339f3f51a0222 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.411+0000 7fbcb51da640 1 -- 172.21.15.83:0/904591962 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcac034070 con 0x7fbcb015dfe0 2024-03-06T07:24:59.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.411+0000 7fbcb51da640 1 -- 172.21.15.83:0/904591962 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcac030860 con 0x7fbcb015dfe0 2024-03-06T07:24:59.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.411+0000 7fbcb51da640 1 -- 172.21.15.83:0/904591962 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcac030c30 con 0x7fbcb015dfe0 2024-03-06T07:24:59.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.411+0000 7fbcb8440640 1 -- 172.21.15.83:0/904591962 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbcb015dfe0 msgr2=0x7fbcb00cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.411+0000 7fbcb8440640 1 --2- 172.21.15.83:0/904591962 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbcb015dfe0 0x7fbcb00cf2f0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fbcac009870 tx=0x7fbcac02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.411+0000 7fbcb8440640 1 -- 172.21.15.83:0/904591962 shutdown_connections 2024-03-06T07:24:59.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.411+0000 7fbcb8440640 1 --2- 172.21.15.83:0/904591962 >> v2:172.21.15.183:3300/0 conn(0x7fbcb00cf830 0x7fbcb00d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.411+0000 7fbcb8440640 1 --2- 172.21.15.83:0/904591962 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbcb015dfe0 0x7fbcb00cf2f0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.411+0000 7fbcb8440640 1 -- 172.21.15.83:0/904591962 >> 172.21.15.83:0/904591962 conn(0x7fbcb00d3cd0 msgr2=0x7fbcb00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:59.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.411+0000 7fbcb8440640 1 -- 172.21.15.83:0/904591962 shutdown_connections 2024-03-06T07:24:59.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.412+0000 7fbcb8440640 1 -- 172.21.15.83:0/904591962 wait complete. 2024-03-06T07:24:59.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.413+0000 7fbcb8440640 1 Processor -- start 2024-03-06T07:24:59.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.413+0000 7fbcb8440640 1 -- start start 2024-03-06T07:24:59.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.413+0000 7fbcb8440640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbcb00cf830 0x7fbcb015c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.413+0000 7fbcb8440640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbcb00d27f0 0x7fbcb00cd310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.413+0000 7fbcb8440640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbcb015dfe0 0x7fbcb00cd850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.413+0000 7fbcb8440640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb00d1df0 con 0x7fbcb015dfe0 2024-03-06T07:24:59.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.413+0000 7fbcb8440640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb00d1c70 con 0x7fbcb00cf830 2024-03-06T07:24:59.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.413+0000 7fbcb8440640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb015e3c0 con 0x7fbcb00d27f0 2024-03-06T07:24:59.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.413+0000 7fbcb69dd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbcb015dfe0 0x7fbcb00cd850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.414+0000 7fbcb69dd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbcb015dfe0 0x7fbcb00cd850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41674/0 (socket says 172.21.15.83:41674) 2024-03-06T07:24:59.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.414+0000 7fbcb69dd640 1 -- 172.21.15.83:0/2317517257 learned_addr learned my addr 172.21.15.83:0/2317517257 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:59.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.414+0000 7fbcb61dc640 1 --2- 172.21.15.83:0/2317517257 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbcb00cf830 0x7fbcb015c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.414+0000 7fbcb59db640 1 --2- 172.21.15.83:0/2317517257 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbcb00d27f0 0x7fbcb00cd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.414+0000 7fbcb69dd640 1 -- 172.21.15.83:0/2317517257 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbcb00d27f0 msgr2=0x7fbcb00cd310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.414+0000 7fbcb69dd640 1 --2- 172.21.15.83:0/2317517257 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbcb00d27f0 0x7fbcb00cd310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.414+0000 7fbcb69dd640 1 -- 172.21.15.83:0/2317517257 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbcb00cf830 msgr2=0x7fbcb015c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.414+0000 7fbcb69dd640 1 --2- 172.21.15.83:0/2317517257 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbcb00cf830 0x7fbcb015c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.414+0000 7fbcb69dd640 1 -- 172.21.15.83:0/2317517257 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcb015fa70 con 0x7fbcb015dfe0 2024-03-06T07:24:59.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.414+0000 7fbcb69dd640 1 --2- 172.21.15.83:0/2317517257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbcb015dfe0 0x7fbcb00cd850 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fbca800c7a0 tx=0x7fbca800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.414+0000 7fbca6ffd640 1 -- 172.21.15.83:0/2317517257 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca8003b70 con 0x7fbcb015dfe0 2024-03-06T07:24:59.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.414+0000 7fbca6ffd640 1 -- 172.21.15.83:0/2317517257 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbca8003d30 con 0x7fbcb015dfe0 2024-03-06T07:24:59.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.414+0000 7fbca6ffd640 1 -- 172.21.15.83:0/2317517257 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca800aa90 con 0x7fbcb015dfe0 2024-03-06T07:24:59.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.414+0000 7fbcb8440640 1 -- 172.21.15.83:0/2317517257 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcb015fd50 con 0x7fbcb015dfe0 2024-03-06T07:24:59.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.414+0000 7fbcb8440640 1 -- 172.21.15.83:0/2317517257 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcb0160310 con 0x7fbcb015dfe0 2024-03-06T07:24:59.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.415+0000 7fbcb8440640 1 -- 172.21.15.83:0/2317517257 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcb0160740 con 0x7fbcb015dfe0 2024-03-06T07:24:59.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.416+0000 7fbca6ffd640 1 -- 172.21.15.83:0/2317517257 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbca800ac30 con 0x7fbcb015dfe0 2024-03-06T07:24:59.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.416+0000 7fbca6ffd640 1 --2- 172.21.15.83:0/2317517257 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbc9406b4d0 0x7fbc9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.416+0000 7fbcb61dc640 1 --2- 172.21.15.83:0/2317517257 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbc9406b4d0 0x7fbc9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.416+0000 7fbca6ffd640 1 -- 172.21.15.83:0/2317517257 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbca8089f70 con 0x7fbcb015dfe0 2024-03-06T07:24:59.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.417+0000 7fbcb61dc640 1 --2- 172.21.15.83:0/2317517257 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbc9406b4d0 0x7fbc9406d990 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fbcac036000 tx=0x7fbcac008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.420+0000 7fbca6ffd640 1 -- 172.21.15.83:0/2317517257 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbca8058830 con 0x7fbcb015dfe0 2024-03-06T07:24:59.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.731+0000 7fbcb8440640 1 -- 172.21.15.83:0/2317517257 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbcb004f330 con 0x7fbcb015dfe0 2024-03-06T07:24:59.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.731+0000 7fbca6ffd640 1 -- 172.21.15.83:0/2317517257 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4980 (secure 0 0 0) 0x7fbca805c8d0 con 0x7fbcb015dfe0 2024-03-06T07:24:59.734 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:24:59.734 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":25,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5665,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/3334609507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3334609507},{"type":"v1","addr":"172.21.15.183:6834","nonce":3334609507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5668,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/691189112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":691189112},{"type":"v1","addr":"172.21.15.183:6835","nonce":691189112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6081,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/3952571284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":3952571284},{"type":"v1","addr":"172.21.15.83:6836","nonce":3952571284}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:19.852157+0000","modified":"2024-03-06T07:24:21.879875+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6084},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6084":{"gid":6084,"name":"a","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/3237506441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3237506441},{"type":"v1","addr":"172.21.15.83:6837","nonce":3237506441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6084,"qdb_cluster":[6084]},"id":4}]} 2024-03-06T07:24:59.734 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 25 2024-03-06T07:24:59.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.734+0000 7fbcb8440640 1 -- 172.21.15.83:0/2317517257 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbc9406b4d0 msgr2=0x7fbc9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.734+0000 7fbcb8440640 1 --2- 172.21.15.83:0/2317517257 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbc9406b4d0 0x7fbc9406d990 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fbcac036000 tx=0x7fbcac008030 comp rx=0 tx=0).stop 2024-03-06T07:24:59.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.734+0000 7fbcb8440640 1 -- 172.21.15.83:0/2317517257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbcb015dfe0 msgr2=0x7fbcb00cd850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.734+0000 7fbcb8440640 1 --2- 172.21.15.83:0/2317517257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbcb015dfe0 0x7fbcb00cd850 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fbca800c7a0 tx=0x7fbca800cc60 comp rx=0 tx=0).stop 2024-03-06T07:24:59.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.734+0000 7fbcb8440640 1 -- 172.21.15.83:0/2317517257 shutdown_connections 2024-03-06T07:24:59.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.734+0000 7fbcb8440640 1 --2- 172.21.15.83:0/2317517257 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbc9406b4d0 0x7fbc9406d990 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.734+0000 7fbcb8440640 1 --2- 172.21.15.83:0/2317517257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbcb015dfe0 0x7fbcb00cd850 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.734+0000 7fbcb8440640 1 --2- 172.21.15.83:0/2317517257 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbcb00d27f0 0x7fbcb00cd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.734+0000 7fbcb8440640 1 --2- 172.21.15.83:0/2317517257 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbcb00cf830 0x7fbcb015c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.734+0000 7fbcb8440640 1 -- 172.21.15.83:0/2317517257 >> 172.21.15.83:0/2317517257 conn(0x7fbcb00d3cd0 msgr2=0x7fbcb006c860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:59.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.734+0000 7fbcb8440640 1 -- 172.21.15.83:0/2317517257 shutdown_connections 2024-03-06T07:24:59.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.734+0000 7fbcb8440640 1 -- 172.21.15.83:0/2317517257 wait complete. 2024-03-06T07:24:59.747 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:59.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.845+0000 7f0dfb42a640 1 Processor -- start 2024-03-06T07:24:59.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.845+0000 7f0dfb42a640 1 -- start start 2024-03-06T07:24:59.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.846+0000 7f0dfb42a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0df40d22b0 0x7f0df40d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.846+0000 7f0dfb42a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0df40b41f0 0x7f0df40b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.846+0000 7f0dfb42a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0df40b3430 con 0x7f0df415dfe0 2024-03-06T07:24:59.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.846+0000 7f0dfb42a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0df40d2c50 con 0x7f0df40b41f0 2024-03-06T07:24:59.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.846+0000 7f0dfb42a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0df40b31e0 con 0x7f0df40d22b0 2024-03-06T07:24:59.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.846+0000 7f0df99c7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0df40b41f0 0x7f0df40b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.846+0000 7f0df99c7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0df40b41f0 0x7f0df40b4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41676/0 (socket says 172.21.15.83:41676) 2024-03-06T07:24:59.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.846+0000 7f0df99c7640 1 -- 172.21.15.83:0/2584798809 learned_addr learned my addr 172.21.15.83:0/2584798809 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:59.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.846+0000 7f0df89c5640 1 --2- 172.21.15.83:0/2584798809 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0df40d22b0 0x7f0df40d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.846+0000 7f0df99c7640 1 -- 172.21.15.83:0/2584798809 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0df40d22b0 msgr2=0x7f0df40d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:59.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.846+0000 7f0df99c7640 1 --2- 172.21.15.83:0/2584798809 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0df40d22b0 0x7f0df40d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.846+0000 7f0df99c7640 1 -- 172.21.15.83:0/2584798809 >> v1:172.21.15.183:6789/0 conn(0x7f0df415dfe0 legacy=0x7f0df40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.846+0000 7f0df99c7640 1 -- 172.21.15.83:0/2584798809 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df40b3710 con 0x7f0df40b41f0 2024-03-06T07:24:59.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.847+0000 7f0df99c7640 1 --2- 172.21.15.83:0/2584798809 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0df40b41f0 0x7f0df40b4650 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f0dec009e50 tx=0x7f0dec02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d166980007735748 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.847+0000 7f0debfff640 1 -- 172.21.15.83:0/2584798809 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dec02cb00 con 0x7f0df40b41f0 2024-03-06T07:24:59.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.847+0000 7f0debfff640 1 -- 172.21.15.83:0/2584798809 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dec02cca0 con 0x7f0df40b41f0 2024-03-06T07:24:59.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.847+0000 7f0debfff640 1 -- 172.21.15.83:0/2584798809 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dec034610 con 0x7f0df40b41f0 2024-03-06T07:24:59.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.848+0000 7f0dfb42a640 1 -- 172.21.15.83:0/2584798809 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0df40b41f0 msgr2=0x7f0df40b4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.848+0000 7f0dfb42a640 1 --2- 172.21.15.83:0/2584798809 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0df40b41f0 0x7f0df40b4650 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f0dec009e50 tx=0x7f0dec02bf60 comp rx=0 tx=0).stop 2024-03-06T07:24:59.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.848+0000 7f0dfb42a640 1 -- 172.21.15.83:0/2584798809 shutdown_connections 2024-03-06T07:24:59.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.848+0000 7f0dfb42a640 1 --2- 172.21.15.83:0/2584798809 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0df40b41f0 0x7f0df40b4650 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.848+0000 7f0dfb42a640 1 --2- 172.21.15.83:0/2584798809 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0df40d22b0 0x7f0df40d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.848+0000 7f0dfb42a640 1 -- 172.21.15.83:0/2584798809 >> 172.21.15.83:0/2584798809 conn(0x7f0df40d3cd0 msgr2=0x7f0df40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:59.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.848+0000 7f0dfb42a640 1 -- 172.21.15.83:0/2584798809 shutdown_connections 2024-03-06T07:24:59.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.848+0000 7f0dfb42a640 1 -- 172.21.15.83:0/2584798809 wait complete. 2024-03-06T07:24:59.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.850+0000 7f0dfb42a640 1 Processor -- start 2024-03-06T07:24:59.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.850+0000 7f0dfb42a640 1 -- start start 2024-03-06T07:24:59.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.851+0000 7f0dfb42a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0df40b41f0 0x7f0df40cb860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.851+0000 7f0dfb42a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0df40d22b0 0x7f0df415c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.851+0000 7f0dfb42a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0df415dfe0 0x7f0df41f3820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.851+0000 7f0dfb42a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df40cfa00 con 0x7f0df40b41f0 2024-03-06T07:24:59.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.851+0000 7f0dfb42a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df40b3d60 con 0x7f0df40d22b0 2024-03-06T07:24:59.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.851+0000 7f0dfb42a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0df40d1c70 con 0x7f0df415dfe0 2024-03-06T07:24:59.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.851+0000 7f0df91c6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0df40b41f0 0x7f0df40cb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.851+0000 7f0df91c6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0df40b41f0 0x7f0df40cb860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41682/0 (socket says 172.21.15.83:41682) 2024-03-06T07:24:59.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.851+0000 7f0df91c6640 1 -- 172.21.15.83:0/3376111892 learned_addr learned my addr 172.21.15.83:0/3376111892 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:24:59.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.851+0000 7f0df99c7640 1 --2- 172.21.15.83:0/3376111892 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0df415dfe0 0x7f0df41f3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.851+0000 7f0df89c5640 1 --2- 172.21.15.83:0/3376111892 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0df40d22b0 0x7f0df415c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.851+0000 7f0df91c6640 1 -- 172.21.15.83:0/3376111892 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0df415dfe0 msgr2=0x7f0df41f3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.851+0000 7f0df91c6640 1 --2- 172.21.15.83:0/3376111892 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0df415dfe0 0x7f0df41f3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.851+0000 7f0df91c6640 1 -- 172.21.15.83:0/3376111892 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0df40d22b0 msgr2=0x7f0df415c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.851+0000 7f0df91c6640 1 --2- 172.21.15.83:0/3376111892 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0df40d22b0 0x7f0df415c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.851+0000 7f0df91c6640 1 -- 172.21.15.83:0/3376111892 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df40cada0 con 0x7f0df40b41f0 2024-03-06T07:24:59.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.851+0000 7f0df99c7640 1 --2- 172.21.15.83:0/3376111892 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0df415dfe0 0x7f0df41f3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:59.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.851+0000 7f0df89c5640 1 --2- 172.21.15.83:0/3376111892 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0df40d22b0 0x7f0df415c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:59.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.851+0000 7f0df91c6640 1 --2- 172.21.15.83:0/3376111892 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0df40b41f0 0x7f0df40cb860 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f0df000c3f0 tx=0x7f0df000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.852+0000 7f0de9ffb640 1 -- 172.21.15.83:0/3376111892 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df00052d0 con 0x7f0df40b41f0 2024-03-06T07:24:59.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.852+0000 7f0de9ffb640 1 -- 172.21.15.83:0/3376111892 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0df0005490 con 0x7f0df40b41f0 2024-03-06T07:24:59.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.852+0000 7f0dfb42a640 1 -- 172.21.15.83:0/3376111892 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0df41f43b0 con 0x7f0df40b41f0 2024-03-06T07:24:59.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.852+0000 7f0de9ffb640 1 -- 172.21.15.83:0/3376111892 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df0007260 con 0x7f0df40b41f0 2024-03-06T07:24:59.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.852+0000 7f0dfb42a640 1 -- 172.21.15.83:0/3376111892 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0df40cb340 con 0x7f0df40b41f0 2024-03-06T07:24:59.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.853+0000 7f0dfb42a640 1 -- 172.21.15.83:0/3376111892 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0df40cb540 con 0x7f0df40b41f0 2024-03-06T07:24:59.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.853+0000 7f0de9ffb640 1 -- 172.21.15.83:0/3376111892 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0df0007400 con 0x7f0df40b41f0 2024-03-06T07:24:59.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.854+0000 7f0de9ffb640 1 --2- 172.21.15.83:0/3376111892 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0dd806b4d0 0x7f0dd806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.854+0000 7f0df89c5640 1 --2- 172.21.15.83:0/3376111892 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0dd806b4d0 0x7f0dd806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.854+0000 7f0de9ffb640 1 -- 172.21.15.83:0/3376111892 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0df0089630 con 0x7f0df40b41f0 2024-03-06T07:24:59.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.854+0000 7f0df89c5640 1 --2- 172.21.15.83:0/3376111892 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0dd806b4d0 0x7f0dd806d990 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f0df4161820 tx=0x7f0de4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:24:59.858+0000 7f0de9ffb640 1 -- 172.21.15.83:0/3376111892 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0df40cb540 con 0x7f0df40b41f0 2024-03-06T07:25:00.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.194+0000 7f0dfb42a640 1 -- 172.21.15.83:0/3376111892 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0df404f330 con 0x7f0df40b41f0 2024-03-06T07:25:00.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.195+0000 7f0de9ffb640 1 -- 172.21.15.83:0/3376111892 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f0df005ce20 con 0x7f0df40b41f0 2024-03-06T07:25:00.198 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:25:00.198 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":25,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5665,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/3334609507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3334609507},{"type":"v1","addr":"172.21.15.183:6834","nonce":3334609507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5668,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/691189112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":691189112},{"type":"v1","addr":"172.21.15.183:6835","nonce":691189112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6081,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/3952571284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":3952571284},{"type":"v1","addr":"172.21.15.83:6836","nonce":3952571284}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:19.852157+0000","modified":"2024-03-06T07:24:21.879875+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6084},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6084":{"gid":6084,"name":"a","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/3237506441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3237506441},{"type":"v1","addr":"172.21.15.83:6837","nonce":3237506441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6084,"qdb_cluster":[6084]},"id":4}]} 2024-03-06T07:25:00.198 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 25 2024-03-06T07:25:00.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.197+0000 7f0dfb42a640 1 -- 172.21.15.83:0/3376111892 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0dd806b4d0 msgr2=0x7f0dd806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.197+0000 7f0dfb42a640 1 --2- 172.21.15.83:0/3376111892 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0dd806b4d0 0x7f0dd806d990 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f0df4161820 tx=0x7f0de4008040 comp rx=0 tx=0).stop 2024-03-06T07:25:00.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.197+0000 7f0dfb42a640 1 -- 172.21.15.83:0/3376111892 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0df40b41f0 msgr2=0x7f0df40cb860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.197+0000 7f0dfb42a640 1 --2- 172.21.15.83:0/3376111892 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0df40b41f0 0x7f0df40cb860 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f0df000c3f0 tx=0x7f0df000c8b0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.197+0000 7f0dfb42a640 1 -- 172.21.15.83:0/3376111892 shutdown_connections 2024-03-06T07:25:00.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.197+0000 7f0dfb42a640 1 --2- 172.21.15.83:0/3376111892 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0dd806b4d0 0x7f0dd806d990 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.197+0000 7f0dfb42a640 1 --2- 172.21.15.83:0/3376111892 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0df415dfe0 0x7f0df41f3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.197+0000 7f0dfb42a640 1 --2- 172.21.15.83:0/3376111892 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0df40d22b0 0x7f0df415c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.197+0000 7f0dfb42a640 1 --2- 172.21.15.83:0/3376111892 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0df40b41f0 0x7f0df40cb860 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.197+0000 7f0dfb42a640 1 -- 172.21.15.83:0/3376111892 >> 172.21.15.83:0/3376111892 conn(0x7f0df40d3cd0 msgr2=0x7f0df406cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:00.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.197+0000 7f0dfb42a640 1 -- 172.21.15.83:0/3376111892 shutdown_connections 2024-03-06T07:25:00.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.197+0000 7f0dfb42a640 1 -- 172.21.15.83:0/3376111892 wait complete. 2024-03-06T07:25:00.213 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:00.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.323+0000 7f13a89eb640 1 Processor -- start 2024-03-06T07:25:00.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.323+0000 7f13a89eb640 1 -- start start 2024-03-06T07:25:00.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.324+0000 7f13a89eb640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f13a40cf470 0x7f13a40cfb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.324+0000 7f13a89eb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13a415b160 con 0x7f13a406bb70 2024-03-06T07:25:00.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.324+0000 7f13a89eb640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13a40b16f0 con 0x7f13a40730d0 2024-03-06T07:25:00.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.324+0000 7f13a89eb640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13a40b4f70 con 0x7f13a40cf470 2024-03-06T07:25:00.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.324+0000 7f13a2d9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f13a406bb70 0x7f13a40cc9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49774/0 (socket says 172.21.15.83:49774) 2024-03-06T07:25:00.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.324+0000 7f13a2d9d640 1 -- 172.21.15.83:0/3055553204 learned_addr learned my addr 172.21.15.83:0/3055553204 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:25:00.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.325+0000 7f13a259c640 1 --2- 172.21.15.83:0/3055553204 >> v2:172.21.15.183:3300/0 conn(0x7f13a40cf470 0x7f13a40cfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.325+0000 7f13a1d9b640 1 -- 172.21.15.83:0/3055553204 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1609742027 0 0) 0x7f13a415b160 con 0x7f13a406bb70 2024-03-06T07:25:00.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.325+0000 7f13a1d9b640 1 -- 172.21.15.83:0/3055553204 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1390003520 con 0x7f13a406bb70 2024-03-06T07:25:00.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.325+0000 7f13a1d9b640 1 -- 172.21.15.83:0/3055553204 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2862529537 0 0) 0x7f13a40b16f0 con 0x7f13a40730d0 2024-03-06T07:25:00.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.325+0000 7f13a1d9b640 1 -- 172.21.15.83:0/3055553204 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13a415b160 con 0x7f13a40730d0 2024-03-06T07:25:00.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.325+0000 7f13a1d9b640 1 -- 172.21.15.83:0/3055553204 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3091895407 0 0) 0x7f1390003520 con 0x7f13a406bb70 2024-03-06T07:25:00.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.325+0000 7f13a1d9b640 1 -- 172.21.15.83:0/3055553204 >> v2:172.21.15.183:3300/0 conn(0x7f13a40cf470 msgr2=0x7f13a40cfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:00.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.325+0000 7f13a1d9b640 1 --2- 172.21.15.83:0/3055553204 >> v2:172.21.15.183:3300/0 conn(0x7f13a40cf470 0x7f13a40cfb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.325+0000 7f13a1d9b640 1 -- 172.21.15.83:0/3055553204 >> v1:172.21.15.183:6789/0 conn(0x7f13a40730d0 legacy=0x7f13a40b59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.325+0000 7f13a1d9b640 1 -- 172.21.15.83:0/3055553204 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a40b19d0 con 0x7f13a406bb70 2024-03-06T07:25:00.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.325+0000 7f13a1d9b640 1 -- 172.21.15.83:0/3055553204 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f139c0019f0 con 0x7f13a406bb70 2024-03-06T07:25:00.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.325+0000 7f13a1d9b640 1 -- 172.21.15.83:0/3055553204 >> v1:172.21.15.83:6789/0 conn(0x7f13a406bb70 legacy=0x7f13a40cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.325+0000 7f13a1d9b640 1 --2- 172.21.15.83:0/3055553204 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1390003b40 0x7f1390003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.325+0000 7f13a1d9b640 1 --2- 172.21.15.83:0/3055553204 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1390004800 0x7f1390004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.325+0000 7f13a1d9b640 1 --2- 172.21.15.83:0/3055553204 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f13900051c0 0x7f139000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.325+0000 7f13a1d9b640 1 -- 172.21.15.83:0/3055553204 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a40b4f70 con 0x7f1390003b40 2024-03-06T07:25:00.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.325+0000 7f13a1d9b640 1 -- 172.21.15.83:0/3055553204 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f139000c040 con 0x7f1390004800 2024-03-06T07:25:00.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.325+0000 7f13a1d9b640 1 -- 172.21.15.83:0/3055553204 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f139000c240 con 0x7f13900051c0 2024-03-06T07:25:00.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.325+0000 7f13a259c640 1 --2- 172.21.15.83:0/3055553204 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1390003b40 0x7f1390003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.325+0000 7f139bfff640 1 --2- 172.21.15.83:0/3055553204 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1390004800 0x7f1390004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.325+0000 7f13a2d9d640 1 --2- 172.21.15.83:0/3055553204 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f13900051c0 0x7f139000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.325+0000 7f13a259c640 1 -- 172.21.15.83:0/3055553204 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f13900051c0 msgr2=0x7f139000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:00.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.325+0000 7f13a259c640 1 --2- 172.21.15.83:0/3055553204 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f13900051c0 0x7f139000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.325+0000 7f13a259c640 1 -- 172.21.15.83:0/3055553204 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1390004800 msgr2=0x7f1390004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:00.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.325+0000 7f13a259c640 1 --2- 172.21.15.83:0/3055553204 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1390004800 0x7f1390004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.325+0000 7f13a259c640 1 -- 172.21.15.83:0/3055553204 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1390003520 con 0x7f1390003b40 2024-03-06T07:25:00.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.325+0000 7f13a2d9d640 1 --2- 172.21.15.83:0/3055553204 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f13900051c0 0x7f139000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:00.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.325+0000 7f13a2d9d640 1 -- 172.21.15.83:0/3055553204 reap_dead start 2024-03-06T07:25:00.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.326+0000 7f13a259c640 1 --2- 172.21.15.83:0/3055553204 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1390003b40 0x7f1390003f40 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f1394009ac0 tx=0x7f1390005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2239e0a682b0ca09 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:00.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.326+0000 7f13a1d9b640 1 -- 172.21.15.83:0/3055553204 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1394039070 con 0x7f1390003b40 2024-03-06T07:25:00.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.326+0000 7f13a1d9b640 1 -- 172.21.15.83:0/3055553204 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f139402c6b0 con 0x7f1390003b40 2024-03-06T07:25:00.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.326+0000 7f13a1d9b640 1 -- 172.21.15.83:0/3055553204 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1394009690 con 0x7f1390003b40 2024-03-06T07:25:00.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.327+0000 7f13a89eb640 1 -- 172.21.15.83:0/3055553204 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1390003b40 msgr2=0x7f1390003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.327+0000 7f13a89eb640 1 --2- 172.21.15.83:0/3055553204 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1390003b40 0x7f1390003f40 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f1394009ac0 tx=0x7f1390005830 comp rx=0 tx=0).stop 2024-03-06T07:25:00.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.327+0000 7f13a89eb640 1 -- 172.21.15.83:0/3055553204 shutdown_connections 2024-03-06T07:25:00.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.327+0000 7f13a89eb640 1 --2- 172.21.15.83:0/3055553204 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1390003b40 0x7f1390003f40 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.327+0000 7f13a89eb640 1 -- 172.21.15.83:0/3055553204 >> 172.21.15.83:0/3055553204 conn(0x7f13a40724d0 msgr2=0x7f13a40728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:00.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.327+0000 7f13a89eb640 1 -- 172.21.15.83:0/3055553204 shutdown_connections 2024-03-06T07:25:00.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.327+0000 7f13a89eb640 1 -- 172.21.15.83:0/3055553204 wait complete. 2024-03-06T07:25:00.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.328+0000 7f13a89eb640 1 Processor -- start 2024-03-06T07:25:00.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.328+0000 7f13a89eb640 1 -- start start 2024-03-06T07:25:00.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.329+0000 7f13a89eb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1390003b40 0x7f13a415b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.329+0000 7f13a89eb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f13a4157370 0x7f13a41f4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.329+0000 7f13a89eb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13a41f4c80 0x7f13a406afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.329+0000 7f13a89eb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a40b1330 con 0x7f13a41f4c80 2024-03-06T07:25:00.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.329+0000 7f13a89eb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a40b54a0 con 0x7f1390003b40 2024-03-06T07:25:00.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.329+0000 7f13a89eb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13a40b14b0 con 0x7f13a4157370 2024-03-06T07:25:00.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.329+0000 7f13a2d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13a41f4c80 0x7f13a406afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.329+0000 7f13a2d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13a41f4c80 0x7f13a406afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41700/0 (socket says 172.21.15.83:41700) 2024-03-06T07:25:00.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.329+0000 7f13a2d9d640 1 -- 172.21.15.83:0/2004267879 learned_addr learned my addr 172.21.15.83:0/2004267879 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:00.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.329+0000 7f139bfff640 1 --2- 172.21.15.83:0/2004267879 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f13a4157370 0x7f13a41f4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.329+0000 7f13a259c640 1 --2- 172.21.15.83:0/2004267879 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1390003b40 0x7f13a415b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.329+0000 7f13a2d9d640 1 -- 172.21.15.83:0/2004267879 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f13a4157370 msgr2=0x7f13a41f4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.329+0000 7f13a2d9d640 1 --2- 172.21.15.83:0/2004267879 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f13a4157370 0x7f13a41f4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.329+0000 7f13a2d9d640 1 -- 172.21.15.83:0/2004267879 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1390003b40 msgr2=0x7f13a415b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.329+0000 7f13a2d9d640 1 --2- 172.21.15.83:0/2004267879 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1390003b40 0x7f13a415b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.329+0000 7f13a2d9d640 1 -- 172.21.15.83:0/2004267879 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a4148420 con 0x7f13a41f4c80 2024-03-06T07:25:00.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.330+0000 7f13a2d9d640 1 --2- 172.21.15.83:0/2004267879 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13a41f4c80 0x7f13a406afb0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f139c000f80 tx=0x7f139c00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:00.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.330+0000 7f139affd640 1 -- 172.21.15.83:0/2004267879 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f139c00eb90 con 0x7f13a41f4c80 2024-03-06T07:25:00.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.330+0000 7f139affd640 1 -- 172.21.15.83:0/2004267879 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f139c018070 con 0x7f13a41f4c80 2024-03-06T07:25:00.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.330+0000 7f13a89eb640 1 -- 172.21.15.83:0/2004267879 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13a4148700 con 0x7f13a41f4c80 2024-03-06T07:25:00.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.330+0000 7f139affd640 1 -- 172.21.15.83:0/2004267879 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f139c013420 con 0x7f13a41f4c80 2024-03-06T07:25:00.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.330+0000 7f13a89eb640 1 -- 172.21.15.83:0/2004267879 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13a4148cc0 con 0x7f13a41f4c80 2024-03-06T07:25:00.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.331+0000 7f13a89eb640 1 -- 172.21.15.83:0/2004267879 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13a404f330 con 0x7f13a41f4c80 2024-03-06T07:25:00.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.331+0000 7f139affd640 1 -- 172.21.15.83:0/2004267879 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f139c0135c0 con 0x7f13a41f4c80 2024-03-06T07:25:00.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.332+0000 7f139affd640 1 --2- 172.21.15.83:0/2004267879 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f138406b4d0 0x7f138406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.332+0000 7f139affd640 1 -- 172.21.15.83:0/2004267879 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f139c08a490 con 0x7f13a41f4c80 2024-03-06T07:25:00.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.332+0000 7f13a259c640 1 --2- 172.21.15.83:0/2004267879 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f138406b4d0 0x7f138406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.332+0000 7f13a259c640 1 --2- 172.21.15.83:0/2004267879 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f138406b4d0 0x7f138406d990 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f1394036000 tx=0x7f1394030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:00.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.336+0000 7f139affd640 1 -- 172.21.15.83:0/2004267879 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f139c0593a0 con 0x7f13a41f4c80 2024-03-06T07:25:00.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.525+0000 7f13a89eb640 1 -- 172.21.15.83:0/2004267879 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f13a40713c0 con 0x7f13a41f4c80 2024-03-06T07:25:00.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.526+0000 7f139affd640 1 -- 172.21.15.83:0/2004267879 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+16677 (secure 0 0 0) 0x7f139c058ca0 con 0x7f13a41f4c80 2024-03-06T07:25:00.528 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:25:00.528 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":57,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:24:21.525937+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:18.074366+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:18.831101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:00.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.528+0000 7f13a89eb640 1 -- 172.21.15.83:0/2004267879 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f138406b4d0 msgr2=0x7f138406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.528+0000 7f13a89eb640 1 --2- 172.21.15.83:0/2004267879 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f138406b4d0 0x7f138406d990 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f1394036000 tx=0x7f1394030000 comp rx=0 tx=0).stop 2024-03-06T07:25:00.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.528+0000 7f13a89eb640 1 -- 172.21.15.83:0/2004267879 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13a41f4c80 msgr2=0x7f13a406afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.528+0000 7f13a89eb640 1 --2- 172.21.15.83:0/2004267879 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13a41f4c80 0x7f13a406afb0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f139c000f80 tx=0x7f139c00e530 comp rx=0 tx=0).stop 2024-03-06T07:25:00.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.528+0000 7f13a89eb640 1 -- 172.21.15.83:0/2004267879 shutdown_connections 2024-03-06T07:25:00.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.528+0000 7f13a89eb640 1 --2- 172.21.15.83:0/2004267879 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f138406b4d0 0x7f138406d990 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.528+0000 7f13a89eb640 1 --2- 172.21.15.83:0/2004267879 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13a41f4c80 0x7f13a406afb0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.528+0000 7f13a89eb640 1 --2- 172.21.15.83:0/2004267879 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f13a4157370 0x7f13a41f4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.528+0000 7f13a89eb640 1 --2- 172.21.15.83:0/2004267879 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1390003b40 0x7f13a415b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.528+0000 7f13a89eb640 1 -- 172.21.15.83:0/2004267879 >> 172.21.15.83:0/2004267879 conn(0x7f13a40724d0 msgr2=0x7f13a40cd250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:00.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.528+0000 7f13a89eb640 1 -- 172.21.15.83:0/2004267879 shutdown_connections 2024-03-06T07:25:00.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.528+0000 7f13a89eb640 1 -- 172.21.15.83:0/2004267879 wait complete. 2024-03-06T07:25:00.543 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:25:00.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.643+0000 7fadfe9f6640 1 Processor -- start 2024-03-06T07:25:00.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.643+0000 7fadfe9f6640 1 -- start start 2024-03-06T07:25:00.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.644+0000 7fadfe9f6640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fadf80b4c00 0x7fadf80b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.644+0000 7fadfe9f6640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fadf80b28a0 0x7fadf80b2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.644+0000 7fadfe9f6640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadf815d510 con 0x7fadf815e540 2024-03-06T07:25:00.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.644+0000 7fadfe9f6640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadf815e0b0 con 0x7fadf80b28a0 2024-03-06T07:25:00.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.644+0000 7fadfe9f6640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadf815d780 con 0x7fadf80b4c00 2024-03-06T07:25:00.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.645+0000 7fadf77fe640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fadf80b28a0 0x7fadf80b2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.645+0000 7fadf7fff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fadf80b4c00 0x7fadf80b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.645+0000 7fadfcf93640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fadf815e540 0x7fadf815e980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:50952/0 (socket says 172.21.15.83:50952) 2024-03-06T07:25:00.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.645+0000 7fadfcf93640 1 -- 172.21.15.83:0/3317002862 learned_addr learned my addr 172.21.15.83:0/3317002862 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:25:00.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.645+0000 7fadf7fff640 1 -- 172.21.15.83:0/3317002862 >> v2:172.21.15.183:3300/0 conn(0x7fadf80b28a0 msgr2=0x7fadf80b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:00.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.645+0000 7fadf7fff640 1 --2- 172.21.15.83:0/3317002862 >> v2:172.21.15.183:3300/0 conn(0x7fadf80b28a0 0x7fadf80b2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.645+0000 7fadf7fff640 1 -- 172.21.15.83:0/3317002862 >> v1:172.21.15.183:6789/0 conn(0x7fadf815e540 legacy=0x7fadf815e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.645+0000 7fadf7fff640 1 -- 172.21.15.83:0/3317002862 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf815da60 con 0x7fadf80b4c00 2024-03-06T07:25:00.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.645+0000 7fadf77fe640 1 --2- 172.21.15.83:0/3317002862 >> v2:172.21.15.183:3300/0 conn(0x7fadf80b28a0 0x7fadf80b2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:00.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.646+0000 7fadf7fff640 1 --2- 172.21.15.83:0/3317002862 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fadf80b4c00 0x7fadf80b4fe0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fade80097a0 tx=0x7fade802b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63dcdf4b641d9b5b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:00.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.646+0000 7fadf6ffd640 1 -- 172.21.15.83:0/3317002862 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade8034070 con 0x7fadf80b4c00 2024-03-06T07:25:00.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.646+0000 7fadf6ffd640 1 -- 172.21.15.83:0/3317002862 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fade8030610 con 0x7fadf80b4c00 2024-03-06T07:25:00.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.646+0000 7fadf6ffd640 1 -- 172.21.15.83:0/3317002862 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade80309e0 con 0x7fadf80b4c00 2024-03-06T07:25:00.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.647+0000 7fadfe9f6640 1 -- 172.21.15.83:0/3317002862 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fadf80b4c00 msgr2=0x7fadf80b4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.647+0000 7fadfe9f6640 1 --2- 172.21.15.83:0/3317002862 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fadf80b4c00 0x7fadf80b4fe0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fade80097a0 tx=0x7fade802b960 comp rx=0 tx=0).stop 2024-03-06T07:25:00.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.647+0000 7fadfe9f6640 1 -- 172.21.15.83:0/3317002862 shutdown_connections 2024-03-06T07:25:00.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.647+0000 7fadfe9f6640 1 --2- 172.21.15.83:0/3317002862 >> v2:172.21.15.183:3300/0 conn(0x7fadf80b28a0 0x7fadf80b2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.647+0000 7fadfe9f6640 1 --2- 172.21.15.83:0/3317002862 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fadf80b4c00 0x7fadf80b4fe0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.647+0000 7fadfe9f6640 1 -- 172.21.15.83:0/3317002862 >> 172.21.15.83:0/3317002862 conn(0x7fadf809a9f0 msgr2=0x7fadf809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:00.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.647+0000 7fadfe9f6640 1 -- 172.21.15.83:0/3317002862 shutdown_connections 2024-03-06T07:25:00.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.647+0000 7fadfe9f6640 1 -- 172.21.15.83:0/3317002862 wait complete. 2024-03-06T07:25:00.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.648+0000 7fadfe9f6640 1 Processor -- start 2024-03-06T07:25:00.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.648+0000 7fadfe9f6640 1 -- start start 2024-03-06T07:25:00.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.649+0000 7fadfe9f6640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fadf80b28a0 0x7fadf81f0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.649+0000 7fadfe9f6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fadf80b4c00 0x7fadf809ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.649+0000 7fadfe9f6640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fadf815e540 0x7fadf809f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.649+0000 7fadfe9f6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf80b33a0 con 0x7fadf80b4c00 2024-03-06T07:25:00.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.649+0000 7fadfe9f6640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf80b3220 con 0x7fadf80b28a0 2024-03-06T07:25:00.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.649+0000 7fadfe9f6640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadf815f090 con 0x7fadf815e540 2024-03-06T07:25:00.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.649+0000 7fadf77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fadf80b4c00 0x7fadf809ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.649+0000 7fadf77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fadf80b4c00 0x7fadf809ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41716/0 (socket says 172.21.15.83:41716) 2024-03-06T07:25:00.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.649+0000 7fadf77fe640 1 -- 172.21.15.83:0/4191241257 learned_addr learned my addr 172.21.15.83:0/4191241257 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:00.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.649+0000 7fadfcf93640 1 --2- 172.21.15.83:0/4191241257 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fadf815e540 0x7fadf809f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.649+0000 7fadf7fff640 1 --2- 172.21.15.83:0/4191241257 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fadf80b28a0 0x7fadf81f0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.649+0000 7fadf77fe640 1 -- 172.21.15.83:0/4191241257 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fadf815e540 msgr2=0x7fadf809f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.649+0000 7fadf77fe640 1 --2- 172.21.15.83:0/4191241257 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fadf815e540 0x7fadf809f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.649+0000 7fadf77fe640 1 -- 172.21.15.83:0/4191241257 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fadf80b28a0 msgr2=0x7fadf81f0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.649+0000 7fadf77fe640 1 --2- 172.21.15.83:0/4191241257 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fadf80b28a0 0x7fadf81f0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.649+0000 7fadf77fe640 1 -- 172.21.15.83:0/4191241257 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf80b0ea0 con 0x7fadf80b4c00 2024-03-06T07:25:00.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.649+0000 7fadf7fff640 1 --2- 172.21.15.83:0/4191241257 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fadf80b28a0 0x7fadf81f0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:00.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.649+0000 7fadf77fe640 1 --2- 172.21.15.83:0/4191241257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fadf80b4c00 0x7fadf809ed10 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fadec00ebe0 tx=0x7fadec00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:00.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.649+0000 7fadf4ff9640 1 -- 172.21.15.83:0/4191241257 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadec017070 con 0x7fadf80b4c00 2024-03-06T07:25:00.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.650+0000 7fadf4ff9640 1 -- 172.21.15.83:0/4191241257 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadec00cc30 con 0x7fadf80b4c00 2024-03-06T07:25:00.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.650+0000 7fadfe9f6640 1 -- 172.21.15.83:0/4191241257 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadf80b1180 con 0x7fadf80b4c00 2024-03-06T07:25:00.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.650+0000 7fadf4ff9640 1 -- 172.21.15.83:0/4191241257 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadec013470 con 0x7fadf80b4c00 2024-03-06T07:25:00.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.650+0000 7fadfe9f6640 1 -- 172.21.15.83:0/4191241257 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadf80b1740 con 0x7fadf80b4c00 2024-03-06T07:25:00.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.651+0000 7fadfe9f6640 1 -- 172.21.15.83:0/4191241257 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadf804f330 con 0x7fadf80b4c00 2024-03-06T07:25:00.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.651+0000 7fadf4ff9640 1 -- 172.21.15.83:0/4191241257 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fadec01a070 con 0x7fadf80b4c00 2024-03-06T07:25:00.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.651+0000 7fadf4ff9640 1 --2- 172.21.15.83:0/4191241257 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fade406b4d0 0x7fade406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.651+0000 7fadf4ff9640 1 -- 172.21.15.83:0/4191241257 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fadec08a450 con 0x7fadf80b4c00 2024-03-06T07:25:00.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.651+0000 7fadf7fff640 1 --2- 172.21.15.83:0/4191241257 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fade406b4d0 0x7fade406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.652+0000 7fadf7fff640 1 --2- 172.21.15.83:0/4191241257 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fade406b4d0 0x7fade406d990 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fade8036000 tx=0x7fade803b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:00.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.656+0000 7fadf4ff9640 1 -- 172.21.15.83:0/4191241257 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fadec058c80 con 0x7fadf80b4c00 2024-03-06T07:25:00.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.868+0000 7fadfe9f6640 1 -- 172.21.15.83:0/4191241257 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fadf806f000 con 0x7fadf80b4c00 2024-03-06T07:25:00.871 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-03-06T07:25:00.869+0000 7fb0dae11640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:25:00.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.870+0000 7fadf4ff9640 1 -- 172.21.15.83:0/4191241257 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v26) v1 ==== 161+0+0 (secure 0 0 0) 0x7fadec05cd20 con 0x7fadf80b4c00 2024-03-06T07:25:00.872 INFO:teuthology.orchestra.run.smithi083.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:25:00.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.873+0000 7fadfe9f6640 1 -- 172.21.15.83:0/4191241257 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fade406b4d0 msgr2=0x7fade406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.873+0000 7fadfe9f6640 1 --2- 172.21.15.83:0/4191241257 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fade406b4d0 0x7fade406d990 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fade8036000 tx=0x7fade803b040 comp rx=0 tx=0).stop 2024-03-06T07:25:00.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.873+0000 7fadfe9f6640 1 -- 172.21.15.83:0/4191241257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fadf80b4c00 msgr2=0x7fadf809ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.873+0000 7fadfe9f6640 1 --2- 172.21.15.83:0/4191241257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fadf80b4c00 0x7fadf809ed10 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fadec00ebe0 tx=0x7fadec00c520 comp rx=0 tx=0).stop 2024-03-06T07:25:00.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.873+0000 7fadfe9f6640 1 -- 172.21.15.83:0/4191241257 shutdown_connections 2024-03-06T07:25:00.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.873+0000 7fadfe9f6640 1 --2- 172.21.15.83:0/4191241257 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fade406b4d0 0x7fade406d990 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.873+0000 7fadfe9f6640 1 --2- 172.21.15.83:0/4191241257 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fadf815e540 0x7fadf809f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.873+0000 7fadfe9f6640 1 --2- 172.21.15.83:0/4191241257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fadf80b4c00 0x7fadf809ed10 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.873+0000 7fadfe9f6640 1 --2- 172.21.15.83:0/4191241257 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fadf80b28a0 0x7fadf81f0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.873+0000 7fadfe9f6640 1 -- 172.21.15.83:0/4191241257 >> 172.21.15.83:0/4191241257 conn(0x7fadf809a9f0 msgr2=0x7fadf815aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:00.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.873+0000 7fadfe9f6640 1 -- 172.21.15.83:0/4191241257 shutdown_connections 2024-03-06T07:25:00.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.873+0000 7fadfe9f6640 1 -- 172.21.15.83:0/4191241257 wait complete. 2024-03-06T07:25:00.887 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:25:00.916 INFO:tasks.ceph.mds.a.smithi083.stdout:starting mds.a at 2024-03-06T07:25:00.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.989+0000 7f2c2bcf4640 1 Processor -- start 2024-03-06T07:25:00.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.989+0000 7f2c2bcf4640 1 -- start start 2024-03-06T07:25:00.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.990+0000 7f2c2bcf4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2c241614d0 0x7f2c240d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.990+0000 7f2c2bcf4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2c240d1910 0x7f2c240d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.990+0000 7f2c2bcf4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c240d2740 con 0x7f2c240d22b0 2024-03-06T07:25:00.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.990+0000 7f2c2bcf4640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c241618b0 con 0x7f2c240d1910 2024-03-06T07:25:00.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.990+0000 7f2c2bcf4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c240cf400 con 0x7f2c241614d0 2024-03-06T07:25:00.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.990+0000 7f2c2928f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2c240d1910 0x7f2c240d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.990+0000 7f2c2a291640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f2c240d22b0 0x7f2c240d06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49786/0 (socket says 172.21.15.83:49786) 2024-03-06T07:25:00.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.990+0000 7f2c2a291640 1 -- 172.21.15.83:0/2832117447 learned_addr learned my addr 172.21.15.83:0/2832117447 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:25:00.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.990+0000 7f2c29a90640 1 --2- 172.21.15.83:0/2832117447 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2c241614d0 0x7f2c240d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.990+0000 7f2c2928f640 1 -- 172.21.15.83:0/2832117447 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2c241614d0 msgr2=0x7f2c240d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:00.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.990+0000 7f2c2928f640 1 --2- 172.21.15.83:0/2832117447 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2c241614d0 0x7f2c240d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.990+0000 7f2c2928f640 1 -- 172.21.15.83:0/2832117447 >> v1:172.21.15.83:6789/0 conn(0x7f2c240d22b0 legacy=0x7f2c240d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.990+0000 7f2c2928f640 1 -- 172.21.15.83:0/2832117447 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c240cf6a0 con 0x7f2c240d1910 2024-03-06T07:25:00.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.990+0000 7f2c28a8e640 1 -- 172.21.15.83:0/2832117447 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 291142227 0 0) 0x7f2c1c001490 con 0x7f2c240d22b0 2024-03-06T07:25:00.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.991+0000 7f2c2928f640 1 --2- 172.21.15.83:0/2832117447 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c240d1910 0x7f2c240d1d70 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f2c140097a0 tx=0x7f2c1402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b78b83d472cca263 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:00.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.991+0000 7f2c28a8e640 1 -- 172.21.15.83:0/2832117447 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c14035020 con 0x7f2c240d1910 2024-03-06T07:25:00.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.991+0000 7f2c28a8e640 1 -- 172.21.15.83:0/2832117447 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c14031830 con 0x7f2c240d1910 2024-03-06T07:25:00.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.991+0000 7f2c28a8e640 1 -- 172.21.15.83:0/2832117447 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c14031c00 con 0x7f2c240d1910 2024-03-06T07:25:00.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.992+0000 7f2c2bcf4640 1 -- 172.21.15.83:0/2832117447 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c240d1910 msgr2=0x7f2c240d1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.992+0000 7f2c2bcf4640 1 --2- 172.21.15.83:0/2832117447 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c240d1910 0x7f2c240d1d70 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f2c140097a0 tx=0x7f2c1402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.992+0000 7f2c2bcf4640 1 -- 172.21.15.83:0/2832117447 shutdown_connections 2024-03-06T07:25:00.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.992+0000 7f2c2bcf4640 1 --2- 172.21.15.83:0/2832117447 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c240d1910 0x7f2c240d1d70 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.992+0000 7f2c2bcf4640 1 --2- 172.21.15.83:0/2832117447 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2c241614d0 0x7f2c240d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.992+0000 7f2c2bcf4640 1 -- 172.21.15.83:0/2832117447 >> 172.21.15.83:0/2832117447 conn(0x7f2c240d3d40 msgr2=0x7f2c240d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:00.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.992+0000 7f2c2bcf4640 1 -- 172.21.15.83:0/2832117447 shutdown_connections 2024-03-06T07:25:00.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.992+0000 7f2c2bcf4640 1 -- 172.21.15.83:0/2832117447 wait complete. 2024-03-06T07:25:00.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.993+0000 7f2c2bcf4640 1 Processor -- start 2024-03-06T07:25:00.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.993+0000 7f2c2bcf4640 1 -- start start 2024-03-06T07:25:00.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.994+0000 7f2c2bcf4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2c240d1910 0x7f2c241532f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.994+0000 7f2c2bcf4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c240d22b0 0x7f2c240ca2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.994+0000 7f2c2bcf4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c241614d0 0x7f2c240b84d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.994+0000 7f2c2bcf4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c240b0910 con 0x7f2c240d22b0 2024-03-06T07:25:00.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.994+0000 7f2c2bcf4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c240cfc60 con 0x7f2c241614d0 2024-03-06T07:25:00.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.994+0000 7f2c2bcf4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c240b1e40 con 0x7f2c240d1910 2024-03-06T07:25:00.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.994+0000 7f2c2928f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c240d22b0 0x7f2c240ca2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.994+0000 7f2c2928f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c240d22b0 0x7f2c240ca2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41760/0 (socket says 172.21.15.83:41760) 2024-03-06T07:25:00.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.994+0000 7f2c2928f640 1 -- 172.21.15.83:0/587308913 learned_addr learned my addr 172.21.15.83:0/587308913 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:00.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.994+0000 7f2c2928f640 1 -- 172.21.15.83:0/587308913 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2c240d1910 msgr2=0x7f2c241532f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.994+0000 7f2c2928f640 1 --2- 172.21.15.83:0/587308913 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2c240d1910 0x7f2c241532f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.994+0000 7f2c2928f640 1 -- 172.21.15.83:0/587308913 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c241614d0 msgr2=0x7f2c240b84d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.994+0000 7f2c2928f640 1 --2- 172.21.15.83:0/587308913 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c241614d0 0x7f2c240b84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.994+0000 7f2c2928f640 1 -- 172.21.15.83:0/587308913 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c240b6a70 con 0x7f2c240d22b0 2024-03-06T07:25:00.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.994+0000 7f2c2928f640 1 --2- 172.21.15.83:0/587308913 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c240d22b0 0x7f2c240ca2e0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f2c14009890 tx=0x7f2c140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:00.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.994+0000 7f2c1a7fc640 1 -- 172.21.15.83:0/587308913 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c14041070 con 0x7f2c240d22b0 2024-03-06T07:25:00.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.994+0000 7f2c1a7fc640 1 -- 172.21.15.83:0/587308913 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c1403b920 con 0x7f2c240d22b0 2024-03-06T07:25:00.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.994+0000 7f2c2bcf4640 1 -- 172.21.15.83:0/587308913 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c240b6d50 con 0x7f2c240d22b0 2024-03-06T07:25:00.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.994+0000 7f2c1a7fc640 1 -- 172.21.15.83:0/587308913 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c14035040 con 0x7f2c240d22b0 2024-03-06T07:25:00.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.994+0000 7f2c2bcf4640 1 -- 172.21.15.83:0/587308913 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c240b7310 con 0x7f2c240d22b0 2024-03-06T07:25:00.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.995+0000 7f2c2bcf4640 1 -- 172.21.15.83:0/587308913 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c24068ff0 con 0x7f2c240d22b0 2024-03-06T07:25:00.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.996+0000 7f2c1a7fc640 1 -- 172.21.15.83:0/587308913 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2c1404c060 con 0x7f2c240d22b0 2024-03-06T07:25:00.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.996+0000 7f2c1a7fc640 1 --2- 172.21.15.83:0/587308913 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2c0806b4d0 0x7f2c0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.996+0000 7f2c29a90640 1 --2- 172.21.15.83:0/587308913 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2c0806b4d0 0x7f2c0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.996+0000 7f2c1a7fc640 1 -- 172.21.15.83:0/587308913 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f2c140abc60 con 0x7f2c240d22b0 2024-03-06T07:25:00.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:00.997+0000 7f2c29a90640 1 --2- 172.21.15.83:0/587308913 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2c0806b4d0 0x7f2c0806d990 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f2c20006fd0 tx=0x7f2c20008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:01.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:01.000+0000 7f2c1a7fc640 1 -- 172.21.15.83:0/587308913 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2c1407aa90 con 0x7f2c240d22b0 2024-03-06T07:25:01.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:01.213+0000 7f2c2bcf4640 1 -- 172.21.15.83:0/587308913 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f2c240a1c10 con 0x7f2c240d22b0 2024-03-06T07:25:01.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:01.871+0000 7f2c1a7fc640 1 -- 172.21.15.83:0/587308913 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v27) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2c1407a390 con 0x7f2c240d22b0 2024-03-06T07:25:01.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:01.874+0000 7f2c2bcf4640 1 -- 172.21.15.83:0/587308913 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2c0806b4d0 msgr2=0x7f2c0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:01.874+0000 7f2c2bcf4640 1 --2- 172.21.15.83:0/587308913 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2c0806b4d0 0x7f2c0806d990 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f2c20006fd0 tx=0x7f2c20008040 comp rx=0 tx=0).stop 2024-03-06T07:25:01.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:01.874+0000 7f2c2bcf4640 1 -- 172.21.15.83:0/587308913 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c240d22b0 msgr2=0x7f2c240ca2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:01.874+0000 7f2c2bcf4640 1 --2- 172.21.15.83:0/587308913 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c240d22b0 0x7f2c240ca2e0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f2c14009890 tx=0x7f2c140046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:01.874+0000 7f2c2bcf4640 1 -- 172.21.15.83:0/587308913 shutdown_connections 2024-03-06T07:25:01.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:01.874+0000 7f2c2bcf4640 1 --2- 172.21.15.83:0/587308913 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2c0806b4d0 0x7f2c0806d990 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:01.874+0000 7f2c2bcf4640 1 --2- 172.21.15.83:0/587308913 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c241614d0 0x7f2c240b84d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:01.874+0000 7f2c2bcf4640 1 --2- 172.21.15.83:0/587308913 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c240d22b0 0x7f2c240ca2e0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:01.874+0000 7f2c2bcf4640 1 --2- 172.21.15.83:0/587308913 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2c240d1910 0x7f2c241532f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:01.874+0000 7f2c2bcf4640 1 -- 172.21.15.83:0/587308913 >> 172.21.15.83:0/587308913 conn(0x7f2c240d3d40 msgr2=0x7f2c2406bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:01.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:01.874+0000 7f2c2bcf4640 1 -- 172.21.15.83:0/587308913 shutdown_connections 2024-03-06T07:25:01.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:01.874+0000 7f2c2bcf4640 1 -- 172.21.15.83:0/587308913 wait complete. 2024-03-06T07:25:01.888 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:25:02.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.034+0000 7f1a06afb640 1 Processor -- start 2024-03-06T07:25:02.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.034+0000 7f1a06afb640 1 -- start start 2024-03-06T07:25:02.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.034+0000 7f1a06afb640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f1a00144690 0x7f1a000d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.034+0000 7f1a06afb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1a000cdb30 0x7f1a000ced40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.034+0000 7f1a06afb640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a0006d760 con 0x7f1a000b3a40 2024-03-06T07:25:02.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.034+0000 7f1a06afb640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a000d0830 con 0x7f1a00144690 2024-03-06T07:25:02.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.034+0000 7f1a06afb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a00144a70 con 0x7f1a000cdb30 2024-03-06T07:25:02.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.035+0000 7f1a05098640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1a000cdb30 0x7f1a000ced40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.035+0000 7f1a04897640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f1a00144690 0x7f1a000d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.035+0000 7f1a05098640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1a000cdb30 0x7f1a000ced40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:56402/0 (socket says 172.21.15.83:56402) 2024-03-06T07:25:02.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.035+0000 7f1a05098640 1 -- 172.21.15.83:0/899904623 learned_addr learned my addr 172.21.15.83:0/899904623 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:02.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.035+0000 7f1a05098640 1 -- 172.21.15.83:0/899904623 >> v2:172.21.15.183:3300/0 conn(0x7f1a00144690 msgr2=0x7f1a000d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:02.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.035+0000 7f1a05098640 1 --2- 172.21.15.83:0/899904623 >> v2:172.21.15.183:3300/0 conn(0x7f1a00144690 0x7f1a000d02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.035+0000 7f1a05098640 1 -- 172.21.15.83:0/899904623 >> v1:172.21.15.183:6789/0 conn(0x7f1a000b3a40 legacy=0x7f1a000cd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.035+0000 7f1a05098640 1 -- 172.21.15.83:0/899904623 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a0015f660 con 0x7f1a000cdb30 2024-03-06T07:25:02.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.035+0000 7f1a04897640 1 --2- 172.21.15.83:0/899904623 >> v2:172.21.15.183:3300/0 conn(0x7f1a00144690 0x7f1a000d02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:02.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.036+0000 7f1a05098640 1 --2- 172.21.15.83:0/899904623 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1a000cdb30 0x7f1a000ced40 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f19f8009e50 tx=0x7f19f802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10ee7776ec10c06b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:02.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.036+0000 7f19ff7fe640 1 -- 172.21.15.83:0/899904623 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f802cb00 con 0x7f1a000cdb30 2024-03-06T07:25:02.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.036+0000 7f19ff7fe640 1 -- 172.21.15.83:0/899904623 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19f802cca0 con 0x7f1a000cdb30 2024-03-06T07:25:02.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.036+0000 7f19ff7fe640 1 -- 172.21.15.83:0/899904623 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f8034610 con 0x7f1a000cdb30 2024-03-06T07:25:02.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.037+0000 7f1a06afb640 1 -- 172.21.15.83:0/899904623 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1a000cdb30 msgr2=0x7f1a000ced40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.037+0000 7f1a06afb640 1 --2- 172.21.15.83:0/899904623 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1a000cdb30 0x7f1a000ced40 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f19f8009e50 tx=0x7f19f802bf60 comp rx=0 tx=0).stop 2024-03-06T07:25:02.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.037+0000 7f1a06afb640 1 -- 172.21.15.83:0/899904623 shutdown_connections 2024-03-06T07:25:02.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.037+0000 7f1a06afb640 1 --2- 172.21.15.83:0/899904623 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1a000cdb30 0x7f1a000ced40 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.037+0000 7f1a06afb640 1 --2- 172.21.15.83:0/899904623 >> v2:172.21.15.183:3300/0 conn(0x7f1a00144690 0x7f1a000d02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.037+0000 7f1a06afb640 1 -- 172.21.15.83:0/899904623 >> 172.21.15.83:0/899904623 conn(0x7f1a000d09b0 msgr2=0x7f1a000d0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:02.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.037+0000 7f1a06afb640 1 -- 172.21.15.83:0/899904623 shutdown_connections 2024-03-06T07:25:02.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.037+0000 7f1a06afb640 1 -- 172.21.15.83:0/899904623 wait complete. 2024-03-06T07:25:02.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.038+0000 7f1a06afb640 1 Processor -- start 2024-03-06T07:25:02.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.038+0000 7f1a06afb640 1 -- start start 2024-03-06T07:25:02.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.039+0000 7f1a06afb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a000b3a40 0x7f1a0006b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.039+0000 7f1a06afb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1a000cdb30 0x7f1a0013f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.039+0000 7f1a06afb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a00144690 0x7f1a000a9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.039+0000 7f1a06afb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0015f280 con 0x7f1a000b3a40 2024-03-06T07:25:02.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.039+0000 7f1a06afb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0006dd00 con 0x7f1a00144690 2024-03-06T07:25:02.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.039+0000 7f1a06afb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0015f400 con 0x7f1a000cdb30 2024-03-06T07:25:02.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.039+0000 7f1a04897640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a000b3a40 0x7f1a0006b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.039+0000 7f1a04897640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a000b3a40 0x7f1a0006b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41772/0 (socket says 172.21.15.83:41772) 2024-03-06T07:25:02.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.039+0000 7f1a04897640 1 -- 172.21.15.83:0/681778080 learned_addr learned my addr 172.21.15.83:0/681778080 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:02.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.039+0000 7f19fffff640 1 --2- 172.21.15.83:0/681778080 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1a000cdb30 0x7f1a0013f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.039+0000 7f1a05098640 1 --2- 172.21.15.83:0/681778080 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a00144690 0x7f1a000a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.039+0000 7f1a04897640 1 -- 172.21.15.83:0/681778080 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1a000cdb30 msgr2=0x7f1a0013f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.039+0000 7f1a04897640 1 --2- 172.21.15.83:0/681778080 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1a000cdb30 0x7f1a0013f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.039+0000 7f1a04897640 1 -- 172.21.15.83:0/681778080 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a00144690 msgr2=0x7f1a000a9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.039+0000 7f1a04897640 1 --2- 172.21.15.83:0/681778080 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a00144690 0x7f1a000a9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.039+0000 7f1a04897640 1 -- 172.21.15.83:0/681778080 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a0006ba20 con 0x7f1a000b3a40 2024-03-06T07:25:02.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.039+0000 7f19fffff640 1 --2- 172.21.15.83:0/681778080 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1a000cdb30 0x7f1a0013f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:02.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.039+0000 7f1a05098640 1 --2- 172.21.15.83:0/681778080 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a00144690 0x7f1a000a9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:02.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.039+0000 7f1a04897640 1 --2- 172.21.15.83:0/681778080 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a000b3a40 0x7f1a0006b4e0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f19f400ec10 tx=0x7f19f400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:02.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.040+0000 7f19fd7fa640 1 -- 172.21.15.83:0/681778080 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f4017070 con 0x7f1a000b3a40 2024-03-06T07:25:02.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.040+0000 7f19fd7fa640 1 -- 172.21.15.83:0/681778080 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19f400cc30 con 0x7f1a000b3a40 2024-03-06T07:25:02.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.040+0000 7f1a06afb640 1 -- 172.21.15.83:0/681778080 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a0013e5f0 con 0x7f1a000b3a40 2024-03-06T07:25:02.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.040+0000 7f19fd7fa640 1 -- 172.21.15.83:0/681778080 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f4013470 con 0x7f1a000b3a40 2024-03-06T07:25:02.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.040+0000 7f1a06afb640 1 -- 172.21.15.83:0/681778080 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a0013ebb0 con 0x7f1a000b3a40 2024-03-06T07:25:02.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.040+0000 7f1a06afb640 1 -- 172.21.15.83:0/681778080 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a0004f330 con 0x7f1a000b3a40 2024-03-06T07:25:02.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.041+0000 7f19fd7fa640 1 -- 172.21.15.83:0/681778080 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f19f401a070 con 0x7f1a000b3a40 2024-03-06T07:25:02.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.041+0000 7f19fd7fa640 1 --2- 172.21.15.83:0/681778080 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f19e406b4d0 0x7f19e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.041+0000 7f19fd7fa640 1 -- 172.21.15.83:0/681778080 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f19f408a550 con 0x7f1a000b3a40 2024-03-06T07:25:02.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.041+0000 7f19fffff640 1 --2- 172.21.15.83:0/681778080 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f19e406b4d0 0x7f19e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.042+0000 7f19fffff640 1 --2- 172.21.15.83:0/681778080 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f19e406b4d0 0x7f19e406d990 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f1a0013d380 tx=0x7f19ec009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:02.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.046+0000 7f19fd7fa640 1 -- 172.21.15.83:0/681778080 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1a0006ba20 con 0x7f1a000b3a40 2024-03-06T07:25:02.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:02.219+0000 7f1a06afb640 1 -- 172.21.15.83:0/681778080 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1a000d16b0 con 0x7f1a000b3a40 2024-03-06T07:25:03.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.078+0000 7f19fd7fa640 1 -- 172.21.15.83:0/681778080 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v59) v1 ==== 181+0+0 (secure 0 0 0) 0x7f19f4059370 con 0x7f1a000b3a40 2024-03-06T07:25:03.080 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:25:03.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.081+0000 7f1a06afb640 1 -- 172.21.15.83:0/681778080 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f19e406b4d0 msgr2=0x7f19e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.081+0000 7f1a06afb640 1 --2- 172.21.15.83:0/681778080 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f19e406b4d0 0x7f19e406d990 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f1a0013d380 tx=0x7f19ec009040 comp rx=0 tx=0).stop 2024-03-06T07:25:03.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.081+0000 7f1a06afb640 1 -- 172.21.15.83:0/681778080 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a000b3a40 msgr2=0x7f1a0006b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.081+0000 7f1a06afb640 1 --2- 172.21.15.83:0/681778080 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a000b3a40 0x7f1a0006b4e0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f19f400ec10 tx=0x7f19f400c520 comp rx=0 tx=0).stop 2024-03-06T07:25:03.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.082+0000 7f1a06afb640 1 -- 172.21.15.83:0/681778080 shutdown_connections 2024-03-06T07:25:03.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.082+0000 7f1a06afb640 1 --2- 172.21.15.83:0/681778080 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f19e406b4d0 0x7f19e406d990 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.082+0000 7f1a06afb640 1 --2- 172.21.15.83:0/681778080 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a00144690 0x7f1a000a9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.082+0000 7f1a06afb640 1 --2- 172.21.15.83:0/681778080 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1a000cdb30 0x7f1a0013f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.082+0000 7f1a06afb640 1 --2- 172.21.15.83:0/681778080 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a000b3a40 0x7f1a0006b4e0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.082+0000 7f1a06afb640 1 -- 172.21.15.83:0/681778080 >> 172.21.15.83:0/681778080 conn(0x7f1a000d09b0 msgr2=0x7f1a0015e1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:03.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.082+0000 7f1a06afb640 1 -- 172.21.15.83:0/681778080 shutdown_connections 2024-03-06T07:25:03.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.082+0000 7f1a06afb640 1 -- 172.21.15.83:0/681778080 wait complete. 2024-03-06T07:25:03.098 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:25:03.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.204+0000 7f59c78c9640 1 Processor -- start 2024-03-06T07:25:03.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.204+0000 7f59c78c9640 1 -- start start 2024-03-06T07:25:03.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.205+0000 7f59c78c9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f59c00b5ca0 0x7f59c00b11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:03.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.205+0000 7f59c78c9640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c0068b00 con 0x7f59c015f9d0 2024-03-06T07:25:03.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.205+0000 7f59c78c9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c015e4e0 con 0x7f59c006be80 2024-03-06T07:25:03.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.205+0000 7f59c78c9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59c015edf0 con 0x7f59c00b5ca0 2024-03-06T07:25:03.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.205+0000 7f59c5e66640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f59c015f9d0 0x7f59c015fe60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49790/0 (socket says 172.21.15.83:49790) 2024-03-06T07:25:03.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.205+0000 7f59c5e66640 1 -- 172.21.15.83:0/2575754345 learned_addr learned my addr 172.21.15.83:0/2575754345 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:25:03.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.205+0000 7f59bffff640 1 -- 172.21.15.83:0/2575754345 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4252180791 0 0) 0x7f59c0068b00 con 0x7f59c015f9d0 2024-03-06T07:25:03.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.205+0000 7f59bffff640 1 -- 172.21.15.83:0/2575754345 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59a8003520 con 0x7f59c015f9d0 2024-03-06T07:25:03.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.205+0000 7f59c4e64640 1 --2- 172.21.15.83:0/2575754345 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f59c00b5ca0 0x7f59c00b11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:03.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.205+0000 7f59bffff640 1 -- 172.21.15.83:0/2575754345 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2350112188 0 0) 0x7f59a8003520 con 0x7f59c015f9d0 2024-03-06T07:25:03.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.206+0000 7f59bffff640 1 -- 172.21.15.83:0/2575754345 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f59c00b5ca0 msgr2=0x7f59c00b11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:03.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.206+0000 7f59bffff640 1 --2- 172.21.15.83:0/2575754345 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f59c00b5ca0 0x7f59c00b11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.206+0000 7f59bffff640 1 -- 172.21.15.83:0/2575754345 >> v1:172.21.15.183:6789/0 conn(0x7f59c006be80 legacy=0x7f59c006c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.206+0000 7f59bffff640 1 -- 172.21.15.83:0/2575754345 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c00b1900 con 0x7f59c015f9d0 2024-03-06T07:25:03.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.206+0000 7f59bffff640 1 -- 172.21.15.83:0/2575754345 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f59b8001670 con 0x7f59c015f9d0 2024-03-06T07:25:03.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.206+0000 7f59bffff640 1 -- 172.21.15.83:0/2575754345 >> v1:172.21.15.83:6789/0 conn(0x7f59c015f9d0 legacy=0x7f59c015fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.206+0000 7f59bffff640 1 --2- 172.21.15.83:0/2575754345 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f59c00b5ca0 0x7f59a80039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:03.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.206+0000 7f59bffff640 1 --2- 172.21.15.83:0/2575754345 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f59a8003f60 0x7f59a8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:03.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.206+0000 7f59bffff640 1 --2- 172.21.15.83:0/2575754345 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59a8004d40 0x7f59a80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:03.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.206+0000 7f59bffff640 1 -- 172.21.15.83:0/2575754345 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c015edf0 con 0x7f59a8004d40 2024-03-06T07:25:03.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.206+0000 7f59bffff640 1 -- 172.21.15.83:0/2575754345 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a800bb50 con 0x7f59a8003f60 2024-03-06T07:25:03.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.206+0000 7f59bffff640 1 -- 172.21.15.83:0/2575754345 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59a800bd50 con 0x7f59c00b5ca0 2024-03-06T07:25:03.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.206+0000 7f59c5e66640 1 --2- 172.21.15.83:0/2575754345 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59a8004d40 0x7f59a80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:03.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.206+0000 7f59c5665640 1 --2- 172.21.15.83:0/2575754345 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f59c00b5ca0 0x7f59a80039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:03.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.206+0000 7f59c4e64640 1 --2- 172.21.15.83:0/2575754345 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f59a8003f60 0x7f59a8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:03.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.206+0000 7f59c5e66640 1 -- 172.21.15.83:0/2575754345 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f59c00b5ca0 msgr2=0x7f59a80039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:03.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.206+0000 7f59c5e66640 1 --2- 172.21.15.83:0/2575754345 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f59c00b5ca0 0x7f59a80039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.206+0000 7f59c5e66640 1 -- 172.21.15.83:0/2575754345 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f59a8003f60 msgr2=0x7f59a8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:03.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.206+0000 7f59c5e66640 1 --2- 172.21.15.83:0/2575754345 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f59a8003f60 0x7f59a8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.206+0000 7f59c5e66640 1 -- 172.21.15.83:0/2575754345 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a8003520 con 0x7f59a8004d40 2024-03-06T07:25:03.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.206+0000 7f59c4e64640 1 --2- 172.21.15.83:0/2575754345 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f59a8003f60 0x7f59a8004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:03.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.207+0000 7f59c5e66640 1 --2- 172.21.15.83:0/2575754345 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59a8004d40 0x7f59a80051c0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f59b800c240 tx=0x7f59a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=634ee08c91b67c39 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:03.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.207+0000 7f59bffff640 1 -- 172.21.15.83:0/2575754345 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b8007d10 con 0x7f59a8004d40 2024-03-06T07:25:03.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.207+0000 7f59bffff640 1 -- 172.21.15.83:0/2575754345 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59b8030070 con 0x7f59a8004d40 2024-03-06T07:25:03.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.207+0000 7f59bffff640 1 -- 172.21.15.83:0/2575754345 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b802cd80 con 0x7f59a8004d40 2024-03-06T07:25:03.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.207+0000 7f59c78c9640 1 -- 172.21.15.83:0/2575754345 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59a8004d40 msgr2=0x7f59a80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.207+0000 7f59c78c9640 1 --2- 172.21.15.83:0/2575754345 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59a8004d40 0x7f59a80051c0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f59b800c240 tx=0x7f59a8005830 comp rx=0 tx=0).stop 2024-03-06T07:25:03.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.207+0000 7f59c5e66640 1 -- 172.21.15.83:0/2575754345 reap_dead start 2024-03-06T07:25:03.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.207+0000 7f59c78c9640 1 -- 172.21.15.83:0/2575754345 shutdown_connections 2024-03-06T07:25:03.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.207+0000 7f59c78c9640 1 -- 172.21.15.83:0/2575754345 >> 172.21.15.83:0/2575754345 conn(0x7f59c00b4ed0 msgr2=0x7f59c00b5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:03.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.207+0000 7f59c78c9640 1 -- 172.21.15.83:0/2575754345 shutdown_connections 2024-03-06T07:25:03.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.207+0000 7f59c78c9640 1 -- 172.21.15.83:0/2575754345 wait complete. 2024-03-06T07:25:03.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.209+0000 7f59c78c9640 1 Processor -- start 2024-03-06T07:25:03.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.209+0000 7f59c78c9640 1 -- start start 2024-03-06T07:25:03.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.209+0000 7f59c78c9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f59c014a4b0 0x7f59c00ad050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:03.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.209+0000 7f59c78c9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59c00aee50 0x7f59c015cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:03.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.209+0000 7f59c78c9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f59c015d520 0x7f59c00c9de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:03.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.209+0000 7f59c78c9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c015ea80 con 0x7f59c00aee50 2024-03-06T07:25:03.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.209+0000 7f59c78c9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c01618a0 con 0x7f59c014a4b0 2024-03-06T07:25:03.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.209+0000 7f59c78c9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59c015f280 con 0x7f59c015d520 2024-03-06T07:25:03.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.209+0000 7f59c4e64640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59c00aee50 0x7f59c015cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:03.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.209+0000 7f59c4e64640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59c00aee50 0x7f59c015cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41802/0 (socket says 172.21.15.83:41802) 2024-03-06T07:25:03.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.209+0000 7f59c4e64640 1 -- 172.21.15.83:0/1035443197 learned_addr learned my addr 172.21.15.83:0/1035443197 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:03.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.210+0000 7f59c5665640 1 --2- 172.21.15.83:0/1035443197 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f59c014a4b0 0x7f59c00ad050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:03.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.210+0000 7f59c5e66640 1 --2- 172.21.15.83:0/1035443197 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f59c015d520 0x7f59c00c9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:03.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.210+0000 7f59c4e64640 1 -- 172.21.15.83:0/1035443197 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f59c015d520 msgr2=0x7f59c00c9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.210+0000 7f59c4e64640 1 --2- 172.21.15.83:0/1035443197 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f59c015d520 0x7f59c00c9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.210+0000 7f59c4e64640 1 -- 172.21.15.83:0/1035443197 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f59c014a4b0 msgr2=0x7f59c00ad050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.210+0000 7f59c4e64640 1 --2- 172.21.15.83:0/1035443197 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f59c014a4b0 0x7f59c00ad050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.210+0000 7f59c4e64640 1 -- 172.21.15.83:0/1035443197 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c00ad750 con 0x7f59c00aee50 2024-03-06T07:25:03.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.210+0000 7f59c4e64640 1 --2- 172.21.15.83:0/1035443197 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59c00aee50 0x7f59c015cfe0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f59b40115b0 tx=0x7f59b4011a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:03.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.210+0000 7f59bdffb640 1 -- 172.21.15.83:0/1035443197 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b4015870 con 0x7f59c00aee50 2024-03-06T07:25:03.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.210+0000 7f59bdffb640 1 -- 172.21.15.83:0/1035443197 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59b4015a10 con 0x7f59c00aee50 2024-03-06T07:25:03.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.210+0000 7f59c78c9640 1 -- 172.21.15.83:0/1035443197 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59c00ada30 con 0x7f59c00aee50 2024-03-06T07:25:03.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.210+0000 7f59bdffb640 1 -- 172.21.15.83:0/1035443197 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b4015de0 con 0x7f59c00aee50 2024-03-06T07:25:03.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.210+0000 7f59c78c9640 1 -- 172.21.15.83:0/1035443197 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59c00adff0 con 0x7f59c00aee50 2024-03-06T07:25:03.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.211+0000 7f59c78c9640 1 -- 172.21.15.83:0/1035443197 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59c004f330 con 0x7f59c00aee50 2024-03-06T07:25:03.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.212+0000 7f59bdffb640 1 -- 172.21.15.83:0/1035443197 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f59b400f480 con 0x7f59c00aee50 2024-03-06T07:25:03.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.212+0000 7f59bdffb640 1 --2- 172.21.15.83:0/1035443197 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f59ac06b4d0 0x7f59ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:03.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.212+0000 7f59bdffb640 1 -- 172.21.15.83:0/1035443197 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5357+0+0 (secure 0 0 0) 0x7f59b408a720 con 0x7f59c00aee50 2024-03-06T07:25:03.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.212+0000 7f59c5665640 1 --2- 172.21.15.83:0/1035443197 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f59ac06b4d0 0x7f59ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:03.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.213+0000 7f59c5665640 1 --2- 172.21.15.83:0/1035443197 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f59ac06b4d0 0x7f59ac06d990 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f59b0001ee0 tx=0x7f59b0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:03.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.216+0000 7f59bdffb640 1 -- 172.21.15.83:0/1035443197 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f59b4019030 con 0x7f59c00aee50 2024-03-06T07:25:03.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:03.387+0000 7f59c78c9640 1 -- 172.21.15.83:0/1035443197 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f59c015d900 con 0x7f59c00aee50 2024-03-06T07:25:04.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.079+0000 7f59bdffb640 1 -- 172.21.15.83:0/1035443197 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v60) v1 ==== 169+0+0 (secure 0 0 0) 0x7f59b4059140 con 0x7f59c00aee50 2024-03-06T07:25:04.081 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'cephfs_data' removed 2024-03-06T07:25:04.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.083+0000 7f59c78c9640 1 -- 172.21.15.83:0/1035443197 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f59ac06b4d0 msgr2=0x7f59ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.083+0000 7f59c78c9640 1 --2- 172.21.15.83:0/1035443197 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f59ac06b4d0 0x7f59ac06d990 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f59b0001ee0 tx=0x7f59b0009040 comp rx=0 tx=0).stop 2024-03-06T07:25:04.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.083+0000 7f59c78c9640 1 -- 172.21.15.83:0/1035443197 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59c00aee50 msgr2=0x7f59c015cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.083+0000 7f59c78c9640 1 --2- 172.21.15.83:0/1035443197 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59c00aee50 0x7f59c015cfe0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f59b40115b0 tx=0x7f59b4011a70 comp rx=0 tx=0).stop 2024-03-06T07:25:04.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.083+0000 7f59c78c9640 1 -- 172.21.15.83:0/1035443197 shutdown_connections 2024-03-06T07:25:04.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.083+0000 7f59c78c9640 1 --2- 172.21.15.83:0/1035443197 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f59ac06b4d0 0x7f59ac06d990 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.083+0000 7f59c78c9640 1 --2- 172.21.15.83:0/1035443197 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f59c015d520 0x7f59c00c9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.083+0000 7f59c78c9640 1 --2- 172.21.15.83:0/1035443197 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59c00aee50 0x7f59c015cfe0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.083+0000 7f59c78c9640 1 --2- 172.21.15.83:0/1035443197 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f59c014a4b0 0x7f59c00ad050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.083+0000 7f59c78c9640 1 -- 172.21.15.83:0/1035443197 >> 172.21.15.83:0/1035443197 conn(0x7f59c00b4ed0 msgr2=0x7f59c006d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:04.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.084+0000 7f59c78c9640 1 -- 172.21.15.83:0/1035443197 shutdown_connections 2024-03-06T07:25:04.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.084+0000 7f59c78c9640 1 -- 172.21.15.83:0/1035443197 wait complete. 2024-03-06T07:25:04.101 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_fragment_size_max 2024-03-06T07:25:04.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.205+0000 7fabef38c640 1 Processor -- start 2024-03-06T07:25:04.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.205+0000 7fabef38c640 1 -- start start 2024-03-06T07:25:04.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.206+0000 7fabef38c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fabe815f4f0 0x7fabe8066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.206+0000 7fabef38c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fabe80ce750 0x7fabe80b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.206+0000 7fabef38c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabe806b050 con 0x7fabe80b0db0 2024-03-06T07:25:04.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.206+0000 7fabef38c640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabe806b2c0 con 0x7fabe80ce750 2024-03-06T07:25:04.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.206+0000 7fabef38c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabe80b0890 con 0x7fabe815f4f0 2024-03-06T07:25:04.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.206+0000 7fabed929640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fabe80ce750 0x7fabe80b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.206+0000 7fabed929640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fabe80ce750 0x7fabe80b4ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41810/0 (socket says 172.21.15.83:41810) 2024-03-06T07:25:04.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.206+0000 7fabed929640 1 -- 172.21.15.83:0/3445319905 learned_addr learned my addr 172.21.15.83:0/3445319905 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:04.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.206+0000 7fabed929640 1 -- 172.21.15.83:0/3445319905 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fabe815f4f0 msgr2=0x7fabe8066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:04.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.206+0000 7fabed128640 1 --2- 172.21.15.83:0/3445319905 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fabe815f4f0 0x7fabe8066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.206+0000 7fabed929640 1 --2- 172.21.15.83:0/3445319905 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fabe815f4f0 0x7fabe8066d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.206+0000 7fabed929640 1 -- 172.21.15.83:0/3445319905 >> v1:172.21.15.183:6789/0 conn(0x7fabe80b0db0 legacy=0x7fabe80ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:04.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.206+0000 7fabed929640 1 -- 172.21.15.83:0/3445319905 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe80672e0 con 0x7fabe80ce750 2024-03-06T07:25:04.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.207+0000 7fabed929640 1 --2- 172.21.15.83:0/3445319905 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fabe80ce750 0x7fabe80b4ed0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fabe0009e50 tx=0x7fabe002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c043559c88492a71 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:04.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.207+0000 7fabdffff640 1 -- 172.21.15.83:0/3445319905 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe002ca70 con 0x7fabe80ce750 2024-03-06T07:25:04.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.207+0000 7fabdffff640 1 -- 172.21.15.83:0/3445319905 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabe002cc10 con 0x7fabe80ce750 2024-03-06T07:25:04.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.207+0000 7fabdffff640 1 -- 172.21.15.83:0/3445319905 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe00343f0 con 0x7fabe80ce750 2024-03-06T07:25:04.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.208+0000 7fabef38c640 1 -- 172.21.15.83:0/3445319905 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fabe80ce750 msgr2=0x7fabe80b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.208+0000 7fabef38c640 1 --2- 172.21.15.83:0/3445319905 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fabe80ce750 0x7fabe80b4ed0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fabe0009e50 tx=0x7fabe002bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.208+0000 7fabef38c640 1 -- 172.21.15.83:0/3445319905 shutdown_connections 2024-03-06T07:25:04.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.208+0000 7fabef38c640 1 --2- 172.21.15.83:0/3445319905 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fabe80ce750 0x7fabe80b4ed0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.208+0000 7fabef38c640 1 --2- 172.21.15.83:0/3445319905 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fabe815f4f0 0x7fabe8066d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.208+0000 7fabef38c640 1 -- 172.21.15.83:0/3445319905 >> 172.21.15.83:0/3445319905 conn(0x7fabe80b1d70 msgr2=0x7fabe80b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:04.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.208+0000 7fabef38c640 1 -- 172.21.15.83:0/3445319905 shutdown_connections 2024-03-06T07:25:04.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.208+0000 7fabef38c640 1 -- 172.21.15.83:0/3445319905 wait complete. 2024-03-06T07:25:04.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.209+0000 7fabef38c640 1 Processor -- start 2024-03-06T07:25:04.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.209+0000 7fabef38c640 1 -- start start 2024-03-06T07:25:04.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.210+0000 7fabef38c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fabe80b0db0 0x7fabe8158160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.210+0000 7fabef38c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fabe80ce750 0x7fabe8152c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.210+0000 7fabef38c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fabe815f4f0 0x7fabe80cd480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.210+0000 7fabef38c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe815f8d0 con 0x7fabe815f4f0 2024-03-06T07:25:04.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.210+0000 7fabef38c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe80cebe0 con 0x7fabe80b0db0 2024-03-06T07:25:04.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.210+0000 7fabef38c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabe81f6030 con 0x7fabe80ce750 2024-03-06T07:25:04.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.210+0000 7fabed929640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fabe815f4f0 0x7fabe80cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.210+0000 7fabed929640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fabe815f4f0 0x7fabe80cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41818/0 (socket says 172.21.15.83:41818) 2024-03-06T07:25:04.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.210+0000 7fabed929640 1 -- 172.21.15.83:0/1322141911 learned_addr learned my addr 172.21.15.83:0/1322141911 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:04.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.210+0000 7fabed929640 1 -- 172.21.15.83:0/1322141911 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fabe80ce750 msgr2=0x7fabe8152c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.210+0000 7fabed128640 1 --2- 172.21.15.83:0/1322141911 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fabe80b0db0 0x7fabe8158160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.210+0000 7fabec927640 1 --2- 172.21.15.83:0/1322141911 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fabe80ce750 0x7fabe8152c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.210+0000 7fabed929640 1 --2- 172.21.15.83:0/1322141911 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fabe80ce750 0x7fabe8152c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.210+0000 7fabed929640 1 -- 172.21.15.83:0/1322141911 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fabe80b0db0 msgr2=0x7fabe8158160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.210+0000 7fabed929640 1 --2- 172.21.15.83:0/1322141911 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fabe80b0db0 0x7fabe8158160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.210+0000 7fabed929640 1 -- 172.21.15.83:0/1322141911 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe81f5420 con 0x7fabe815f4f0 2024-03-06T07:25:04.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.210+0000 7fabed128640 1 --2- 172.21.15.83:0/1322141911 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fabe80b0db0 0x7fabe8158160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:04.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.210+0000 7fabec927640 1 --2- 172.21.15.83:0/1322141911 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fabe80ce750 0x7fabe8152c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:04.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.210+0000 7fabed929640 1 --2- 172.21.15.83:0/1322141911 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fabe815f4f0 0x7fabe80cd480 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fabe0036040 tx=0x7fabe0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:04.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.210+0000 7fabddffb640 1 -- 172.21.15.83:0/1322141911 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe0043020 con 0x7fabe815f4f0 2024-03-06T07:25:04.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.210+0000 7fabddffb640 1 -- 172.21.15.83:0/1322141911 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabe003e450 con 0x7fabe815f4f0 2024-03-06T07:25:04.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.211+0000 7fabef38c640 1 -- 172.21.15.83:0/1322141911 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabe81f5700 con 0x7fabe815f4f0 2024-03-06T07:25:04.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.211+0000 7fabddffb640 1 -- 172.21.15.83:0/1322141911 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe0009510 con 0x7fabe815f4f0 2024-03-06T07:25:04.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.211+0000 7fabef38c640 1 -- 172.21.15.83:0/1322141911 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabe81f5cc0 con 0x7fabe815f4f0 2024-03-06T07:25:04.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.212+0000 7fabef38c640 1 -- 172.21.15.83:0/1322141911 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabe809e3d0 con 0x7fabe815f4f0 2024-03-06T07:25:04.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.212+0000 7fabddffb640 1 -- 172.21.15.83:0/1322141911 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fabe0046020 con 0x7fabe815f4f0 2024-03-06T07:25:04.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.212+0000 7fabddffb640 1 --2- 172.21.15.83:0/1322141911 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fabcc06b500 0x7fabcc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.212+0000 7fabddffb640 1 -- 172.21.15.83:0/1322141911 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fabe00ac780 con 0x7fabe815f4f0 2024-03-06T07:25:04.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.212+0000 7fabed128640 1 --2- 172.21.15.83:0/1322141911 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fabcc06b500 0x7fabcc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.213+0000 7fabed128640 1 --2- 172.21.15.83:0/1322141911 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fabcc06b500 0x7fabcc06d9c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fabe400b820 tx=0x7fabe4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:04.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.217+0000 7fabddffb640 1 -- 172.21.15.83:0/1322141911 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fabe004d030 con 0x7fabe815f4f0 2024-03-06T07:25:04.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.401+0000 7fabef38c640 1 -- 172.21.15.83:0/1322141911 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"} v 0) v1 -- 0x7fabe806cd40 con 0x7fabe815f4f0 2024-03-06T07:25:04.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.404+0000 7fabddffb640 1 -- 172.21.15.83:0/1322141911 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"}]=0 v19) v1 ==== 108+0+0 (secure 0 0 0) 0x7fabe007b900 con 0x7fabe815f4f0 2024-03-06T07:25:04.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.406+0000 7fabef38c640 1 -- 172.21.15.83:0/1322141911 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fabcc06b500 msgr2=0x7fabcc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.406+0000 7fabef38c640 1 --2- 172.21.15.83:0/1322141911 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fabcc06b500 0x7fabcc06d9c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fabe400b820 tx=0x7fabe4008040 comp rx=0 tx=0).stop 2024-03-06T07:25:04.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.406+0000 7fabef38c640 1 -- 172.21.15.83:0/1322141911 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fabe815f4f0 msgr2=0x7fabe80cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.406+0000 7fabef38c640 1 --2- 172.21.15.83:0/1322141911 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fabe815f4f0 0x7fabe80cd480 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fabe0036040 tx=0x7fabe0004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:04.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.406+0000 7fabef38c640 1 -- 172.21.15.83:0/1322141911 shutdown_connections 2024-03-06T07:25:04.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.406+0000 7fabef38c640 1 --2- 172.21.15.83:0/1322141911 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fabcc06b500 0x7fabcc06d9c0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.406+0000 7fabef38c640 1 --2- 172.21.15.83:0/1322141911 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fabe815f4f0 0x7fabe80cd480 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.406+0000 7fabef38c640 1 --2- 172.21.15.83:0/1322141911 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fabe80ce750 0x7fabe8152c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.406+0000 7fabef38c640 1 --2- 172.21.15.83:0/1322141911 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fabe80b0db0 0x7fabe8158160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.406+0000 7fabef38c640 1 -- 172.21.15.83:0/1322141911 >> 172.21.15.83:0/1322141911 conn(0x7fabe80b1d70 msgr2=0x7fabe80b56c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:04.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.406+0000 7fabef38c640 1 -- 172.21.15.83:0/1322141911 shutdown_connections 2024-03-06T07:25:04.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.406+0000 7fabef38c640 1 -- 172.21.15.83:0/1322141911 wait complete. 2024-03-06T07:25:04.418 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_merge_size 2024-03-06T07:25:04.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.518+0000 7fb777f3a640 1 Processor -- start 2024-03-06T07:25:04.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.518+0000 7fb777f3a640 1 -- start start 2024-03-06T07:25:04.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.519+0000 7fb777f3a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb77015fc30 0x7fb7700b47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.519+0000 7fb777f3a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fb7700b4d10 0x7fb7700cf200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.519+0000 7fb777f3a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7700d47f0 con 0x7fb7700d06e0 2024-03-06T07:25:04.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.519+0000 7fb777f3a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7700ced70 con 0x7fb7700b4d10 2024-03-06T07:25:04.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.519+0000 7fb777f3a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb770160070 con 0x7fb77015fc30 2024-03-06T07:25:04.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.519+0000 7fb775cd6640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fb7700d06e0 0x7fb7700d0ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49796/0 (socket says 172.21.15.83:49796) 2024-03-06T07:25:04.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.519+0000 7fb775cd6640 1 -- 172.21.15.83:0/3360846791 learned_addr learned my addr 172.21.15.83:0/3360846791 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:25:04.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.520+0000 7fb774cd4640 1 -- 172.21.15.83:0/3360846791 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2800382294 0 0) 0x7fb7700d47f0 con 0x7fb7700d06e0 2024-03-06T07:25:04.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.520+0000 7fb774cd4640 1 -- 172.21.15.83:0/3360846791 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb758003520 con 0x7fb7700d06e0 2024-03-06T07:25:04.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.520+0000 7fb7764d7640 1 --2- 172.21.15.83:0/3360846791 >> v2:172.21.15.183:3300/0 conn(0x7fb7700b4d10 0x7fb7700cf200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.520+0000 7fb7754d5640 1 --2- 172.21.15.83:0/3360846791 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb77015fc30 0x7fb7700b47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.520+0000 7fb774cd4640 1 -- 172.21.15.83:0/3360846791 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 737752762 0 0) 0x7fb758003520 con 0x7fb7700d06e0 2024-03-06T07:25:04.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.520+0000 7fb774cd4640 1 -- 172.21.15.83:0/3360846791 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb77015fc30 msgr2=0x7fb7700b47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:04.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.520+0000 7fb774cd4640 1 --2- 172.21.15.83:0/3360846791 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb77015fc30 0x7fb7700b47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.520+0000 7fb774cd4640 1 -- 172.21.15.83:0/3360846791 >> v2:172.21.15.183:3300/0 conn(0x7fb7700b4d10 msgr2=0x7fb7700cf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:04.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.520+0000 7fb774cd4640 1 --2- 172.21.15.83:0/3360846791 >> v2:172.21.15.183:3300/0 conn(0x7fb7700b4d10 0x7fb7700cf200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.520+0000 7fb774cd4640 1 -- 172.21.15.83:0/3360846791 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7700d1ab0 con 0x7fb7700d06e0 2024-03-06T07:25:04.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.520+0000 7fb774cd4640 1 -- 172.21.15.83:0/3360846791 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7fb768001670 con 0x7fb7700d06e0 2024-03-06T07:25:04.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.520+0000 7fb774cd4640 1 -- 172.21.15.83:0/3360846791 >> v1:172.21.15.83:6789/0 conn(0x7fb7700d06e0 legacy=0x7fb7700d0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.520+0000 7fb774cd4640 1 --2- 172.21.15.83:0/3360846791 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb77015fc30 0x7fb7580039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.520+0000 7fb774cd4640 1 --2- 172.21.15.83:0/3360846791 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb758003f30 0x7fb758004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.520+0000 7fb774cd4640 1 --2- 172.21.15.83:0/3360846791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb758004d40 0x7fb7580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.520+0000 7fb774cd4640 1 -- 172.21.15.83:0/3360846791 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7700ced70 con 0x7fb758004d40 2024-03-06T07:25:04.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.520+0000 7fb774cd4640 1 -- 172.21.15.83:0/3360846791 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb770160070 con 0x7fb758003f30 2024-03-06T07:25:04.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.520+0000 7fb7764d7640 1 --2- 172.21.15.83:0/3360846791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb758004d40 0x7fb7580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.520+0000 7fb774cd4640 1 -- 172.21.15.83:0/3360846791 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb75800b9c0 con 0x7fb77015fc30 2024-03-06T07:25:04.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.520+0000 7fb775cd6640 1 --2- 172.21.15.83:0/3360846791 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb77015fc30 0x7fb7580039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.520+0000 7fb7754d5640 1 --2- 172.21.15.83:0/3360846791 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb758003f30 0x7fb758004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.520+0000 7fb7764d7640 1 -- 172.21.15.83:0/3360846791 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb77015fc30 msgr2=0x7fb7580039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:04.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.520+0000 7fb7764d7640 1 --2- 172.21.15.83:0/3360846791 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb77015fc30 0x7fb7580039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.520+0000 7fb7764d7640 1 -- 172.21.15.83:0/3360846791 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb758003f30 msgr2=0x7fb758004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:04.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.520+0000 7fb7764d7640 1 --2- 172.21.15.83:0/3360846791 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb758003f30 0x7fb758004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.520+0000 7fb7764d7640 1 -- 172.21.15.83:0/3360846791 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb758003520 con 0x7fb758004d40 2024-03-06T07:25:04.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.521+0000 7fb7764d7640 1 --2- 172.21.15.83:0/3360846791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb758004d40 0x7fb7580051c0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fb760009fc0 tx=0x7fb758005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e2f6afd9d755873 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:04.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.521+0000 7fb774cd4640 1 -- 172.21.15.83:0/3360846791 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb760039070 con 0x7fb758004d40 2024-03-06T07:25:04.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.521+0000 7fb774cd4640 1 -- 172.21.15.83:0/3360846791 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb76002cc40 con 0x7fb758004d40 2024-03-06T07:25:04.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.521+0000 7fb774cd4640 1 -- 172.21.15.83:0/3360846791 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb760009c10 con 0x7fb758004d40 2024-03-06T07:25:04.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.522+0000 7fb777f3a640 1 -- 172.21.15.83:0/3360846791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb758004d40 msgr2=0x7fb7580051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.522+0000 7fb777f3a640 1 --2- 172.21.15.83:0/3360846791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb758004d40 0x7fb7580051c0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fb760009fc0 tx=0x7fb758005830 comp rx=0 tx=0).stop 2024-03-06T07:25:04.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.522+0000 7fb7764d7640 1 -- 172.21.15.83:0/3360846791 reap_dead start 2024-03-06T07:25:04.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.522+0000 7fb777f3a640 1 -- 172.21.15.83:0/3360846791 shutdown_connections 2024-03-06T07:25:04.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.522+0000 7fb777f3a640 1 -- 172.21.15.83:0/3360846791 >> 172.21.15.83:0/3360846791 conn(0x7fb7700d3cd0 msgr2=0x7fb7700d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:04.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.522+0000 7fb777f3a640 1 -- 172.21.15.83:0/3360846791 shutdown_connections 2024-03-06T07:25:04.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.522+0000 7fb777f3a640 1 -- 172.21.15.83:0/3360846791 wait complete. 2024-03-06T07:25:04.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.523+0000 7fb777f3a640 1 Processor -- start 2024-03-06T07:25:04.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.523+0000 7fb777f3a640 1 -- start start 2024-03-06T07:25:04.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.524+0000 7fb777f3a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb7701f3ba0 0x7fb7701f3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.524+0000 7fb777f3a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7701f83f0 0x7fb7700b2910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.524+0000 7fb777f3a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb7701f8850 0x7fb7700b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.524+0000 7fb777f3a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7700ce990 con 0x7fb7701f83f0 2024-03-06T07:25:04.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.524+0000 7fb777f3a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7700d4ab0 con 0x7fb7701f3ba0 2024-03-06T07:25:04.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.524+0000 7fb777f3a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7700b44d0 con 0x7fb7701f8850 2024-03-06T07:25:04.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.524+0000 7fb7754d5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7701f83f0 0x7fb7700b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.524+0000 7fb7754d5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7701f83f0 0x7fb7700b2910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41830/0 (socket says 172.21.15.83:41830) 2024-03-06T07:25:04.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.524+0000 7fb7754d5640 1 -- 172.21.15.83:0/2450471254 learned_addr learned my addr 172.21.15.83:0/2450471254 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:04.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.524+0000 7fb7754d5640 1 -- 172.21.15.83:0/2450471254 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb7701f8850 msgr2=0x7fb7700b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.524+0000 7fb7754d5640 1 --2- 172.21.15.83:0/2450471254 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb7701f8850 0x7fb7700b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.524+0000 7fb7754d5640 1 -- 172.21.15.83:0/2450471254 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb7701f3ba0 msgr2=0x7fb7701f3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.524+0000 7fb7754d5640 1 --2- 172.21.15.83:0/2450471254 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb7701f3ba0 0x7fb7701f3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.524+0000 7fb7754d5640 1 -- 172.21.15.83:0/2450471254 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7700ca2a0 con 0x7fb7701f83f0 2024-03-06T07:25:04.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.524+0000 7fb7754d5640 1 --2- 172.21.15.83:0/2450471254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7701f83f0 0x7fb7700b2910 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fb7640118e0 tx=0x7fb764011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:04.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.524+0000 7fb76e7fc640 1 -- 172.21.15.83:0/2450471254 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb76400fa70 con 0x7fb7701f83f0 2024-03-06T07:25:04.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.524+0000 7fb76e7fc640 1 -- 172.21.15.83:0/2450471254 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb76400fc10 con 0x7fb7701f83f0 2024-03-06T07:25:04.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.524+0000 7fb76e7fc640 1 -- 172.21.15.83:0/2450471254 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb764014440 con 0x7fb7701f83f0 2024-03-06T07:25:04.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.524+0000 7fb777f3a640 1 -- 172.21.15.83:0/2450471254 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb770149cd0 con 0x7fb7701f83f0 2024-03-06T07:25:04.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.524+0000 7fb777f3a640 1 -- 172.21.15.83:0/2450471254 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb77014a290 con 0x7fb7701f83f0 2024-03-06T07:25:04.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.525+0000 7fb777f3a640 1 -- 172.21.15.83:0/2450471254 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb770149b00 con 0x7fb7701f83f0 2024-03-06T07:25:04.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.526+0000 7fb76e7fc640 1 -- 172.21.15.83:0/2450471254 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb764012040 con 0x7fb7701f83f0 2024-03-06T07:25:04.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.526+0000 7fb76e7fc640 1 --2- 172.21.15.83:0/2450471254 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb75c06b4d0 0x7fb75c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.526+0000 7fb775cd6640 1 --2- 172.21.15.83:0/2450471254 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb75c06b4d0 0x7fb75c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.526+0000 7fb76e7fc640 1 -- 172.21.15.83:0/2450471254 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fb76408a570 con 0x7fb7701f83f0 2024-03-06T07:25:04.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.527+0000 7fb775cd6640 1 --2- 172.21.15.83:0/2450471254 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb75c06b4d0 0x7fb75c06d990 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fb768001eb0 tx=0x7fb7680047e0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:04.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.531+0000 7fb76e7fc640 1 -- 172.21.15.83:0/2450471254 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb764019030 con 0x7fb7701f83f0 2024-03-06T07:25:04.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.700+0000 7fb777f3a640 1 -- 172.21.15.83:0/2450471254 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_merge_size"} v 0) v1 -- 0x7fb7700a1c80 con 0x7fb7701f83f0 2024-03-06T07:25:04.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.704+0000 7fb76e7fc640 1 -- 172.21.15.83:0/2450471254 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_merge_size"}]=0 v20) v1 ==== 101+0+0 (secure 0 0 0) 0x7fb764059130 con 0x7fb7701f83f0 2024-03-06T07:25:04.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.706+0000 7fb777f3a640 1 -- 172.21.15.83:0/2450471254 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb75c06b4d0 msgr2=0x7fb75c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.706+0000 7fb777f3a640 1 --2- 172.21.15.83:0/2450471254 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb75c06b4d0 0x7fb75c06d990 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fb768001eb0 tx=0x7fb7680047e0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.706+0000 7fb777f3a640 1 -- 172.21.15.83:0/2450471254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7701f83f0 msgr2=0x7fb7700b2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.706+0000 7fb777f3a640 1 --2- 172.21.15.83:0/2450471254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7701f83f0 0x7fb7700b2910 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fb7640118e0 tx=0x7fb764011da0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.706+0000 7fb777f3a640 1 -- 172.21.15.83:0/2450471254 shutdown_connections 2024-03-06T07:25:04.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.706+0000 7fb777f3a640 1 --2- 172.21.15.83:0/2450471254 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb75c06b4d0 0x7fb75c06d990 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.706+0000 7fb777f3a640 1 --2- 172.21.15.83:0/2450471254 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb7701f8850 0x7fb7700b2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.706+0000 7fb777f3a640 1 --2- 172.21.15.83:0/2450471254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7701f83f0 0x7fb7700b2910 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.706+0000 7fb777f3a640 1 --2- 172.21.15.83:0/2450471254 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb7701f3ba0 0x7fb7701f3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.706+0000 7fb777f3a640 1 -- 172.21.15.83:0/2450471254 >> 172.21.15.83:0/2450471254 conn(0x7fb7700d3cd0 msgr2=0x7fb770161330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:04.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.706+0000 7fb777f3a640 1 -- 172.21.15.83:0/2450471254 shutdown_connections 2024-03-06T07:25:04.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.706+0000 7fb777f3a640 1 -- 172.21.15.83:0/2450471254 wait complete. 2024-03-06T07:25:04.719 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:25:04.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.819+0000 7f1020f56640 1 Processor -- start 2024-03-06T07:25:04.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.819+0000 7f1020f56640 1 -- start start 2024-03-06T07:25:04.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.820+0000 7f1020f56640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f101c1586e0 0x7f101c0aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.820+0000 7f1020f56640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f101c0aef80 0x7f101c0cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.820+0000 7f1020f56640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f101c0cffd0 0x7f101c06da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.820+0000 7f1020f56640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f101c0d03f0 con 0x7f101c0cffd0 2024-03-06T07:25:04.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.820+0000 7f1020f56640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f101c158ac0 con 0x7f101c0aef80 2024-03-06T07:25:04.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.820+0000 7f1020f56640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f101c15b2a0 con 0x7f101c1586e0 2024-03-06T07:25:04.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.820+0000 7f101ad9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f101c0cffd0 0x7f101c06da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.820+0000 7f101ad9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f101c0cffd0 0x7f101c06da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41834/0 (socket says 172.21.15.83:41834) 2024-03-06T07:25:04.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.820+0000 7f101ad9d640 1 -- 172.21.15.83:0/2652928425 learned_addr learned my addr 172.21.15.83:0/2652928425 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:04.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.820+0000 7f1013fff640 1 --2- 172.21.15.83:0/2652928425 >> v2:172.21.15.183:3300/0 conn(0x7f101c0aef80 0x7f101c0cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.820+0000 7f101a59c640 1 --2- 172.21.15.83:0/2652928425 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f101c1586e0 0x7f101c0aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.820+0000 7f101ad9d640 1 -- 172.21.15.83:0/2652928425 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f101c1586e0 msgr2=0x7f101c0aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:04.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.820+0000 7f101ad9d640 1 --2- 172.21.15.83:0/2652928425 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f101c1586e0 0x7f101c0aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.820+0000 7f101ad9d640 1 -- 172.21.15.83:0/2652928425 >> v2:172.21.15.183:3300/0 conn(0x7f101c0aef80 msgr2=0x7f101c0cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:04.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.820+0000 7f101ad9d640 1 --2- 172.21.15.83:0/2652928425 >> v2:172.21.15.183:3300/0 conn(0x7f101c0aef80 0x7f101c0cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.820+0000 7f101ad9d640 1 -- 172.21.15.83:0/2652928425 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f101c15b540 con 0x7f101c0cffd0 2024-03-06T07:25:04.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.821+0000 7f101ad9d640 1 --2- 172.21.15.83:0/2652928425 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f101c0cffd0 0x7f101c06da20 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f101c0d0610 tx=0x7f101402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba2aafa7a83c8ab7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:04.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.821+0000 7f1019d9b640 1 -- 172.21.15.83:0/2652928425 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f101402ca70 con 0x7f101c0cffd0 2024-03-06T07:25:04.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.821+0000 7f1019d9b640 1 -- 172.21.15.83:0/2652928425 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f101402cc10 con 0x7f101c0cffd0 2024-03-06T07:25:04.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.821+0000 7f1019d9b640 1 -- 172.21.15.83:0/2652928425 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10140343f0 con 0x7f101c0cffd0 2024-03-06T07:25:04.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.821+0000 7f1020f56640 1 -- 172.21.15.83:0/2652928425 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f101c0cffd0 msgr2=0x7f101c06da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.821+0000 7f1020f56640 1 --2- 172.21.15.83:0/2652928425 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f101c0cffd0 0x7f101c06da20 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f101c0d0610 tx=0x7f101402bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.821+0000 7f1020f56640 1 -- 172.21.15.83:0/2652928425 shutdown_connections 2024-03-06T07:25:04.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.821+0000 7f1020f56640 1 --2- 172.21.15.83:0/2652928425 >> v2:172.21.15.183:3300/0 conn(0x7f101c0aef80 0x7f101c0cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.821+0000 7f1020f56640 1 --2- 172.21.15.83:0/2652928425 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f101c0cffd0 0x7f101c06da20 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.821+0000 7f1020f56640 1 --2- 172.21.15.83:0/2652928425 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f101c1586e0 0x7f101c0aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.821+0000 7f1020f56640 1 -- 172.21.15.83:0/2652928425 >> 172.21.15.83:0/2652928425 conn(0x7f101c06b190 msgr2=0x7f101c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:04.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.821+0000 7f1020f56640 1 -- 172.21.15.83:0/2652928425 shutdown_connections 2024-03-06T07:25:04.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.822+0000 7f1020f56640 1 -- 172.21.15.83:0/2652928425 wait complete. 2024-03-06T07:25:04.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.823+0000 7f1020f56640 1 Processor -- start 2024-03-06T07:25:04.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.823+0000 7f1020f56640 1 -- start start 2024-03-06T07:25:04.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.823+0000 7f1020f56640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f101c0aef80 0x7f101c1f2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.823+0000 7f1020f56640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f101c0cffd0 0x7f101c158f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.823+0000 7f1020f56640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f101c1586e0 0x7f101c156990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.823+0000 7f1020f56640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f101c15bde0 con 0x7f101c0aef80 2024-03-06T07:25:04.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.823+0000 7f1020f56640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f101c15bb00 con 0x7f101c0cffd0 2024-03-06T07:25:04.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.823+0000 7f1020f56640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f101c071350 con 0x7f101c1586e0 2024-03-06T07:25:04.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.824+0000 7f101a59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f101c0aef80 0x7f101c1f2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.824+0000 7f101a59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f101c0aef80 0x7f101c1f2c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41836/0 (socket says 172.21.15.83:41836) 2024-03-06T07:25:04.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.824+0000 7f101a59c640 1 -- 172.21.15.83:0/2531942400 learned_addr learned my addr 172.21.15.83:0/2531942400 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:04.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.824+0000 7f1013fff640 1 --2- 172.21.15.83:0/2531942400 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f101c0cffd0 0x7f101c158f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.824+0000 7f101ad9d640 1 --2- 172.21.15.83:0/2531942400 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f101c1586e0 0x7f101c156990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.824+0000 7f101a59c640 1 -- 172.21.15.83:0/2531942400 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f101c1586e0 msgr2=0x7f101c156990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.824+0000 7f101a59c640 1 --2- 172.21.15.83:0/2531942400 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f101c1586e0 0x7f101c156990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.824+0000 7f101a59c640 1 -- 172.21.15.83:0/2531942400 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f101c0cffd0 msgr2=0x7f101c158f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.824+0000 7f101a59c640 1 --2- 172.21.15.83:0/2531942400 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f101c0cffd0 0x7f101c158f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.824+0000 7f101a59c640 1 -- 172.21.15.83:0/2531942400 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f101c1f3300 con 0x7f101c0aef80 2024-03-06T07:25:04.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.824+0000 7f1013fff640 1 --2- 172.21.15.83:0/2531942400 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f101c0cffd0 0x7f101c158f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:04.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.824+0000 7f101ad9d640 1 --2- 172.21.15.83:0/2531942400 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f101c1586e0 0x7f101c156990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:04.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.824+0000 7f101a59c640 1 --2- 172.21.15.83:0/2531942400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f101c0aef80 0x7f101c1f2c70 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f100c00d620 tx=0x7f100c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:04.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.824+0000 7f1012ffd640 1 -- 172.21.15.83:0/2531942400 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f100c00b620 con 0x7f101c0aef80 2024-03-06T07:25:04.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.824+0000 7f1012ffd640 1 -- 172.21.15.83:0/2531942400 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f100c019070 con 0x7f101c0aef80 2024-03-06T07:25:04.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.824+0000 7f1020f56640 1 -- 172.21.15.83:0/2531942400 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f101c1f35e0 con 0x7f101c0aef80 2024-03-06T07:25:04.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.824+0000 7f1012ffd640 1 -- 172.21.15.83:0/2531942400 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f100c00ba00 con 0x7f101c0aef80 2024-03-06T07:25:04.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.824+0000 7f1020f56640 1 -- 172.21.15.83:0/2531942400 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f101c1594b0 con 0x7f101c0aef80 2024-03-06T07:25:04.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.825+0000 7f1020f56640 1 -- 172.21.15.83:0/2531942400 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f101c0732b0 con 0x7f101c0aef80 2024-03-06T07:25:04.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.826+0000 7f1012ffd640 1 -- 172.21.15.83:0/2531942400 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f100c00bba0 con 0x7f101c0aef80 2024-03-06T07:25:04.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.826+0000 7f1012ffd640 1 --2- 172.21.15.83:0/2531942400 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0ffc06b4d0 0x7f0ffc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.826+0000 7f1012ffd640 1 -- 172.21.15.83:0/2531942400 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f100c08a2e0 con 0x7f101c0aef80 2024-03-06T07:25:04.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.826+0000 7f1013fff640 1 --2- 172.21.15.83:0/2531942400 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0ffc06b4d0 0x7f0ffc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.826+0000 7f1013fff640 1 --2- 172.21.15.83:0/2531942400 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0ffc06b4d0 0x7f0ffc06d990 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f101c153a80 tx=0x7f1004008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:04.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:04.830+0000 7f1012ffd640 1 -- 172.21.15.83:0/2531942400 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f100c0594f0 con 0x7f101c0aef80 2024-03-06T07:25:05.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.001+0000 7f1020f56640 1 -- 172.21.15.83:0/2531942400 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f101c0aec60 con 0x7f101c0aef80 2024-03-06T07:25:05.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.005+0000 7f1012ffd640 1 -- 172.21.15.83:0/2531942400 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v21) v1 ==== 104+0+0 (secure 0 0 0) 0x7f100c058df0 con 0x7f101c0aef80 2024-03-06T07:25:05.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.007+0000 7f1020f56640 1 -- 172.21.15.83:0/2531942400 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0ffc06b4d0 msgr2=0x7f0ffc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:05.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.007+0000 7f1020f56640 1 --2- 172.21.15.83:0/2531942400 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0ffc06b4d0 0x7f0ffc06d990 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f101c153a80 tx=0x7f1004008040 comp rx=0 tx=0).stop 2024-03-06T07:25:05.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.007+0000 7f1020f56640 1 -- 172.21.15.83:0/2531942400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f101c0aef80 msgr2=0x7f101c1f2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:05.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.007+0000 7f1020f56640 1 --2- 172.21.15.83:0/2531942400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f101c0aef80 0x7f101c1f2c70 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f100c00d620 tx=0x7f100c00dae0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.007+0000 7f1020f56640 1 -- 172.21.15.83:0/2531942400 shutdown_connections 2024-03-06T07:25:05.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.007+0000 7f1020f56640 1 --2- 172.21.15.83:0/2531942400 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0ffc06b4d0 0x7f0ffc06d990 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.007+0000 7f1020f56640 1 --2- 172.21.15.83:0/2531942400 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f101c1586e0 0x7f101c156990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.007+0000 7f1020f56640 1 --2- 172.21.15.83:0/2531942400 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f101c0cffd0 0x7f101c158f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.007+0000 7f1020f56640 1 --2- 172.21.15.83:0/2531942400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f101c0aef80 0x7f101c1f2c70 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.007+0000 7f1020f56640 1 -- 172.21.15.83:0/2531942400 >> 172.21.15.83:0/2531942400 conn(0x7f101c06b190 msgr2=0x7f101c0afff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:05.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.007+0000 7f1020f56640 1 -- 172.21.15.83:0/2531942400 shutdown_connections 2024-03-06T07:25:05.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.007+0000 7f1020f56640 1 -- 172.21.15.83:0/2531942400 wait complete. 2024-03-06T07:25:05.020 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented' 2024-03-06T07:25:05.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.120+0000 7fc64109a640 1 Processor -- start 2024-03-06T07:25:05.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.120+0000 7fc64109a640 1 -- start start 2024-03-06T07:25:05.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.121+0000 7fc64109a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fc63c0d00c0 0x7fc63c0d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.121+0000 7fc64109a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc63c06cf20 0x7fc63c06d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.121+0000 7fc64109a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc63c0d09e0 con 0x7fc63c0cf850 2024-03-06T07:25:05.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.121+0000 7fc64109a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc63c06ca90 con 0x7fc63c06cf20 2024-03-06T07:25:05.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.121+0000 7fc64109a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc63c15e380 con 0x7fc63c0d00c0 2024-03-06T07:25:05.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.121+0000 7fc63b59e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc63c06cf20 0x7fc63c06d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:05.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.121+0000 7fc63b59e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc63c06cf20 0x7fc63c06d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41840/0 (socket says 172.21.15.83:41840) 2024-03-06T07:25:05.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.121+0000 7fc63b59e640 1 -- 172.21.15.83:0/167687051 learned_addr learned my addr 172.21.15.83:0/167687051 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:05.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.121+0000 7fc63ad9d640 1 --2- 172.21.15.83:0/167687051 >> v2:172.21.15.183:3300/0 conn(0x7fc63c0d00c0 0x7fc63c0d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:05.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.121+0000 7fc63b59e640 1 -- 172.21.15.83:0/167687051 >> v2:172.21.15.183:3300/0 conn(0x7fc63c0d00c0 msgr2=0x7fc63c0d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:05.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.121+0000 7fc63b59e640 1 --2- 172.21.15.83:0/167687051 >> v2:172.21.15.183:3300/0 conn(0x7fc63c0d00c0 0x7fc63c0d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.121+0000 7fc63b59e640 1 -- 172.21.15.83:0/167687051 >> v1:172.21.15.183:6789/0 conn(0x7fc63c0cf850 legacy=0x7fc63c0cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:05.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.121+0000 7fc63b59e640 1 -- 172.21.15.83:0/167687051 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc63c0cdc10 con 0x7fc63c06cf20 2024-03-06T07:25:05.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.122+0000 7fc63b59e640 1 --2- 172.21.15.83:0/167687051 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc63c06cf20 0x7fc63c06d3b0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fc63c0d1a40 tx=0x7fc63402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e481000e1054ea92 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:05.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.122+0000 7fc639d9b640 1 -- 172.21.15.83:0/167687051 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc63402cb20 con 0x7fc63c06cf20 2024-03-06T07:25:05.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.122+0000 7fc639d9b640 1 -- 172.21.15.83:0/167687051 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc63402ccc0 con 0x7fc63c06cf20 2024-03-06T07:25:05.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.122+0000 7fc639d9b640 1 -- 172.21.15.83:0/167687051 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc634034610 con 0x7fc63c06cf20 2024-03-06T07:25:05.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.123+0000 7fc64109a640 1 -- 172.21.15.83:0/167687051 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc63c06cf20 msgr2=0x7fc63c06d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:05.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.123+0000 7fc64109a640 1 --2- 172.21.15.83:0/167687051 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc63c06cf20 0x7fc63c06d3b0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fc63c0d1a40 tx=0x7fc63402bf80 comp rx=0 tx=0).stop 2024-03-06T07:25:05.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.123+0000 7fc64109a640 1 -- 172.21.15.83:0/167687051 shutdown_connections 2024-03-06T07:25:05.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.123+0000 7fc64109a640 1 --2- 172.21.15.83:0/167687051 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc63c06cf20 0x7fc63c06d3b0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.123+0000 7fc64109a640 1 --2- 172.21.15.83:0/167687051 >> v2:172.21.15.183:3300/0 conn(0x7fc63c0d00c0 0x7fc63c0d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.123+0000 7fc64109a640 1 -- 172.21.15.83:0/167687051 >> 172.21.15.83:0/167687051 conn(0x7fc63c0b35a0 msgr2=0x7fc63c0b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:05.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.123+0000 7fc64109a640 1 -- 172.21.15.83:0/167687051 shutdown_connections 2024-03-06T07:25:05.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.123+0000 7fc64109a640 1 -- 172.21.15.83:0/167687051 wait complete. 2024-03-06T07:25:05.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.124+0000 7fc64109a640 1 Processor -- start 2024-03-06T07:25:05.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.124+0000 7fc64109a640 1 -- start start 2024-03-06T07:25:05.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.125+0000 7fc64109a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc63c06cf20 0x7fc63c1600e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.125+0000 7fc64109a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc63c0cf850 0x7fc63c0b5e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.125+0000 7fc64109a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc63c0d00c0 0x7fc63c0b63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.125+0000 7fc64109a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc63c06c720 con 0x7fc63c0cf850 2024-03-06T07:25:05.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.125+0000 7fc64109a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc63c06c440 con 0x7fc63c06cf20 2024-03-06T07:25:05.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.125+0000 7fc64109a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc63c0b1030 con 0x7fc63c0d00c0 2024-03-06T07:25:05.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.125+0000 7fc63a59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc63c0cf850 0x7fc63c0b5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:05.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.125+0000 7fc63a59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc63c0cf850 0x7fc63c0b5e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41856/0 (socket says 172.21.15.83:41856) 2024-03-06T07:25:05.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.125+0000 7fc63a59c640 1 -- 172.21.15.83:0/2791319294 learned_addr learned my addr 172.21.15.83:0/2791319294 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:05.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.125+0000 7fc63b59e640 1 --2- 172.21.15.83:0/2791319294 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc63c0d00c0 0x7fc63c0b63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:05.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.125+0000 7fc63ad9d640 1 --2- 172.21.15.83:0/2791319294 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc63c06cf20 0x7fc63c1600e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:05.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.125+0000 7fc63a59c640 1 -- 172.21.15.83:0/2791319294 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc63c0d00c0 msgr2=0x7fc63c0b63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:05.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.125+0000 7fc63a59c640 1 --2- 172.21.15.83:0/2791319294 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc63c0d00c0 0x7fc63c0b63c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.125+0000 7fc63a59c640 1 -- 172.21.15.83:0/2791319294 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc63c06cf20 msgr2=0x7fc63c1600e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:05.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.125+0000 7fc63a59c640 1 --2- 172.21.15.83:0/2791319294 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc63c06cf20 0x7fc63c1600e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.125+0000 7fc63a59c640 1 -- 172.21.15.83:0/2791319294 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc63c0c7e10 con 0x7fc63c0cf850 2024-03-06T07:25:05.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.125+0000 7fc63ad9d640 1 --2- 172.21.15.83:0/2791319294 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc63c06cf20 0x7fc63c1600e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:05.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.126+0000 7fc63a59c640 1 --2- 172.21.15.83:0/2791319294 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc63c0cf850 0x7fc63c0b5e80 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fc63000c400 tx=0x7fc63000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:05.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.126+0000 7fc6237fe640 1 -- 172.21.15.83:0/2791319294 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6300052d0 con 0x7fc63c0cf850 2024-03-06T07:25:05.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.126+0000 7fc6237fe640 1 -- 172.21.15.83:0/2791319294 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc630005490 con 0x7fc63c0cf850 2024-03-06T07:25:05.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.126+0000 7fc6237fe640 1 -- 172.21.15.83:0/2791319294 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc630007260 con 0x7fc63c0cf850 2024-03-06T07:25:05.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.126+0000 7fc64109a640 1 -- 172.21.15.83:0/2791319294 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc63c0b6a60 con 0x7fc63c0cf850 2024-03-06T07:25:05.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.126+0000 7fc64109a640 1 -- 172.21.15.83:0/2791319294 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc63c15f8e0 con 0x7fc63c0cf850 2024-03-06T07:25:05.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.127+0000 7fc64109a640 1 -- 172.21.15.83:0/2791319294 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc63c04f330 con 0x7fc63c0cf850 2024-03-06T07:25:05.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.127+0000 7fc6237fe640 1 -- 172.21.15.83:0/2791319294 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc630007400 con 0x7fc63c0cf850 2024-03-06T07:25:05.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.128+0000 7fc6237fe640 1 --2- 172.21.15.83:0/2791319294 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc62806b4d0 0x7fc62806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.128+0000 7fc6237fe640 1 -- 172.21.15.83:0/2791319294 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc630089430 con 0x7fc63c0cf850 2024-03-06T07:25:05.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.128+0000 7fc63ad9d640 1 --2- 172.21.15.83:0/2791319294 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc62806b4d0 0x7fc62806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:05.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.128+0000 7fc63ad9d640 1 --2- 172.21.15.83:0/2791319294 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc62806b4d0 0x7fc62806d990 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fc62c009ac0 tx=0x7fc62c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:05.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.132+0000 7fc6237fe640 1 -- 172.21.15.83:0/2791319294 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc6300585b0 con 0x7fc63c0cf850 2024-03-06T07:25:05.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:05.368+0000 7fc64109a640 1 -- 172.21.15.83:0/2791319294 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]} v 0) v1 -- 0x7fc63c0cc2e0 con 0x7fc63c0cf850 2024-03-06T07:25:06.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.094+0000 7fc6237fe640 1 -- 172.21.15.83:0/2791319294 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]}]=0 v186) v1 ==== 144+0+0 (secure 0 0 0) 0x7fc630057eb0 con 0x7fc63c0cf850 2024-03-06T07:25:06.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.098+0000 7fc64109a640 1 -- 172.21.15.83:0/2791319294 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc62806b4d0 msgr2=0x7fc62806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.098+0000 7fc64109a640 1 --2- 172.21.15.83:0/2791319294 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc62806b4d0 0x7fc62806d990 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fc62c009ac0 tx=0x7fc62c008040 comp rx=0 tx=0).stop 2024-03-06T07:25:06.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.098+0000 7fc64109a640 1 -- 172.21.15.83:0/2791319294 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc63c0cf850 msgr2=0x7fc63c0b5e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.098+0000 7fc64109a640 1 --2- 172.21.15.83:0/2791319294 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc63c0cf850 0x7fc63c0b5e80 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fc63000c400 tx=0x7fc63000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.098+0000 7fc64109a640 1 -- 172.21.15.83:0/2791319294 shutdown_connections 2024-03-06T07:25:06.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.098+0000 7fc64109a640 1 --2- 172.21.15.83:0/2791319294 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc62806b4d0 0x7fc62806d990 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.098+0000 7fc64109a640 1 --2- 172.21.15.83:0/2791319294 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc63c0d00c0 0x7fc63c0b63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.098+0000 7fc64109a640 1 --2- 172.21.15.83:0/2791319294 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc63c0cf850 0x7fc63c0b5e80 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.098+0000 7fc64109a640 1 --2- 172.21.15.83:0/2791319294 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc63c06cf20 0x7fc63c1600e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.098+0000 7fc64109a640 1 -- 172.21.15.83:0/2791319294 >> 172.21.15.83:0/2791319294 conn(0x7fc63c0b35a0 msgr2=0x7fc63c0b3980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:06.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.098+0000 7fc64109a640 1 -- 172.21.15.83:0/2791319294 shutdown_connections 2024-03-06T07:25:06.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.098+0000 7fc64109a640 1 -- 172.21.15.83:0/2791319294 wait complete. 2024-03-06T07:25:06.113 INFO:tasks.cephfs_test_runner:That fragmentation (forced) will allow more entries to be created. ... ok 2024-03-06T07:25:06.113 INFO:tasks.cephfs_test_runner:Starting test: test_dirfrag_limit_strays (tasks.cephfs.test_strays.TestStrays) That unlinking fails when the stray directory fragment becomes too 2024-03-06T07:25:06.114 INFO:tasks.cephfs_test_runner:test_dirfrag_limit_strays (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:25:06.114 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:06.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.219+0000 7f5be3e47640 1 Processor -- start 2024-03-06T07:25:06.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.219+0000 7f5be3e47640 1 -- start start 2024-03-06T07:25:06.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.220+0000 7f5be3e47640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5bdc0b3060 0x7f5bdc15e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.220+0000 7f5be3e47640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f5bdc15bf30 0x7f5bdc15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.220+0000 7f5be3e47640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bdc15f2d0 con 0x7f5bdc13dc10 2024-03-06T07:25:06.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.220+0000 7f5be3e47640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bdc0cf3f0 con 0x7f5bdc0b3060 2024-03-06T07:25:06.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.220+0000 7f5be3e47640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bdc15b210 con 0x7f5bdc15bf30 2024-03-06T07:25:06.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.220+0000 7f5be13e2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5bdc0b3060 0x7f5bdc15e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.220+0000 7f5be13e2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5bdc0b3060 0x7f5bdc15e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41860/0 (socket says 172.21.15.83:41860) 2024-03-06T07:25:06.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.220+0000 7f5be13e2640 1 -- 172.21.15.83:0/1190457348 learned_addr learned my addr 172.21.15.83:0/1190457348 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:06.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.220+0000 7f5be13e2640 1 -- 172.21.15.83:0/1190457348 >> v2:172.21.15.183:3300/0 conn(0x7f5bdc15bf30 msgr2=0x7f5bdc15ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:06.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.220+0000 7f5be23e4640 1 --2- 172.21.15.83:0/1190457348 >> v2:172.21.15.183:3300/0 conn(0x7f5bdc15bf30 0x7f5bdc15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.220+0000 7f5be13e2640 1 --2- 172.21.15.83:0/1190457348 >> v2:172.21.15.183:3300/0 conn(0x7f5bdc15bf30 0x7f5bdc15ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.220+0000 7f5be13e2640 1 -- 172.21.15.83:0/1190457348 >> v1:172.21.15.183:6789/0 conn(0x7f5bdc13dc10 legacy=0x7f5bdc0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:06.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.220+0000 7f5be13e2640 1 -- 172.21.15.83:0/1190457348 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bdc15f580 con 0x7f5bdc0b3060 2024-03-06T07:25:06.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.220+0000 7f5be23e4640 1 --2- 172.21.15.83:0/1190457348 >> v2:172.21.15.183:3300/0 conn(0x7f5bdc15bf30 0x7f5bdc15ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:06.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.221+0000 7f5be13e2640 1 --2- 172.21.15.83:0/1190457348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5bdc0b3060 0x7f5bdc15e850 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc0b4e00 tx=0x7f5bd802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa7972598ca2ad6e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.221+0000 7f5be0be1640 1 -- 172.21.15.83:0/1190457348 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd802e070 con 0x7f5bdc0b3060 2024-03-06T07:25:06.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.221+0000 7f5be0be1640 1 -- 172.21.15.83:0/1190457348 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bd8037070 con 0x7f5bdc0b3060 2024-03-06T07:25:06.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.221+0000 7f5be0be1640 1 -- 172.21.15.83:0/1190457348 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd8031a00 con 0x7f5bdc0b3060 2024-03-06T07:25:06.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.222+0000 7f5be3e47640 1 -- 172.21.15.83:0/1190457348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5bdc0b3060 msgr2=0x7f5bdc15e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.222+0000 7f5be3e47640 1 --2- 172.21.15.83:0/1190457348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5bdc0b3060 0x7f5bdc15e850 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc0b4e00 tx=0x7f5bd802b880 comp rx=0 tx=0).stop 2024-03-06T07:25:06.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.222+0000 7f5be3e47640 1 -- 172.21.15.83:0/1190457348 shutdown_connections 2024-03-06T07:25:06.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.222+0000 7f5be3e47640 1 --2- 172.21.15.83:0/1190457348 >> v2:172.21.15.183:3300/0 conn(0x7f5bdc15bf30 0x7f5bdc15ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.222+0000 7f5be3e47640 1 --2- 172.21.15.83:0/1190457348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5bdc0b3060 0x7f5bdc15e850 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.222+0000 7f5be3e47640 1 -- 172.21.15.83:0/1190457348 >> 172.21.15.83:0/1190457348 conn(0x7f5bdc0cf8a0 msgr2=0x7f5bdc0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:06.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.222+0000 7f5be3e47640 1 -- 172.21.15.83:0/1190457348 shutdown_connections 2024-03-06T07:25:06.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.222+0000 7f5be3e47640 1 -- 172.21.15.83:0/1190457348 wait complete. 2024-03-06T07:25:06.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.223+0000 7f5be3e47640 1 Processor -- start 2024-03-06T07:25:06.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.223+0000 7f5be3e47640 1 -- start start 2024-03-06T07:25:06.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.224+0000 7f5be3e47640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5bdc0b3060 0x7f5bdc1f5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.224+0000 7f5be3e47640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5bdc13dc10 0x7f5bdc144170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.224+0000 7f5be3e47640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5bdc15bf30 0x7f5bdc0b0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.224+0000 7f5be3e47640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bdc13dff0 con 0x7f5bdc0b3060 2024-03-06T07:25:06.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.224+0000 7f5be3e47640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bdc15b730 con 0x7f5bdc13dc10 2024-03-06T07:25:06.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.224+0000 7f5be3e47640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bdc0708d0 con 0x7f5bdc15bf30 2024-03-06T07:25:06.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.224+0000 7f5be1be3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5bdc0b3060 0x7f5bdc1f5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.224+0000 7f5be1be3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5bdc0b3060 0x7f5bdc1f5a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41874/0 (socket says 172.21.15.83:41874) 2024-03-06T07:25:06.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.224+0000 7f5be1be3640 1 -- 172.21.15.83:0/1419282610 learned_addr learned my addr 172.21.15.83:0/1419282610 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:06.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.224+0000 7f5be1be3640 1 -- 172.21.15.83:0/1419282610 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5bdc15bf30 msgr2=0x7f5bdc0b0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.224+0000 7f5be1be3640 1 --2- 172.21.15.83:0/1419282610 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5bdc15bf30 0x7f5bdc0b0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.224+0000 7f5be1be3640 1 -- 172.21.15.83:0/1419282610 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5bdc13dc10 msgr2=0x7f5bdc144170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.224+0000 7f5be1be3640 1 --2- 172.21.15.83:0/1419282610 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5bdc13dc10 0x7f5bdc144170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.224+0000 7f5be1be3640 1 -- 172.21.15.83:0/1419282610 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bdc13d570 con 0x7f5bdc0b3060 2024-03-06T07:25:06.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.224+0000 7f5be1be3640 1 --2- 172.21.15.83:0/1419282610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5bdc0b3060 0x7f5bdc1f5a40 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f5bd000c430 tx=0x7f5bd000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.224+0000 7f5bce7fc640 1 -- 172.21.15.83:0/1419282610 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd0011790 con 0x7f5bdc0b3060 2024-03-06T07:25:06.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.224+0000 7f5bce7fc640 1 -- 172.21.15.83:0/1419282610 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bd0017070 con 0x7f5bdc0b3060 2024-03-06T07:25:06.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.224+0000 7f5bce7fc640 1 -- 172.21.15.83:0/1419282610 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd0011bb0 con 0x7f5bdc0b3060 2024-03-06T07:25:06.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.224+0000 7f5be3e47640 1 -- 172.21.15.83:0/1419282610 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bdc153670 con 0x7f5bdc0b3060 2024-03-06T07:25:06.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.224+0000 7f5be3e47640 1 -- 172.21.15.83:0/1419282610 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bdc153f10 con 0x7f5bdc0b3060 2024-03-06T07:25:06.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.225+0000 7f5be3e47640 1 -- 172.21.15.83:0/1419282610 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bdc04f330 con 0x7f5bdc0b3060 2024-03-06T07:25:06.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.226+0000 7f5bce7fc640 1 -- 172.21.15.83:0/1419282610 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5bd001c460 con 0x7f5bdc0b3060 2024-03-06T07:25:06.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.226+0000 7f5bce7fc640 1 --2- 172.21.15.83:0/1419282610 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5bc006b4d0 0x7f5bc006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.226+0000 7f5bce7fc640 1 -- 172.21.15.83:0/1419282610 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5bd0089de0 con 0x7f5bdc0b3060 2024-03-06T07:25:06.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.226+0000 7f5be13e2640 1 --2- 172.21.15.83:0/1419282610 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5bc006b4d0 0x7f5bc006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.227+0000 7f5be13e2640 1 --2- 172.21.15.83:0/1419282610 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5bc006b4d0 0x7f5bc006d990 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f5bd8036000 tx=0x7f5bd8030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.231+0000 7f5bce7fc640 1 -- 172.21.15.83:0/1419282610 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5bd0058f60 con 0x7f5bdc0b3060 2024-03-06T07:25:06.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.402+0000 7f5be3e47640 1 -- 172.21.15.83:0/1419282610 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5bdc06eb00 con 0x7f5bdc0b3060 2024-03-06T07:25:06.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.403+0000 7f5bce7fc640 1 -- 172.21.15.83:0/1419282610 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v60) v1 ==== 74+0+13032 (secure 0 0 0) 0x7f5bd0058860 con 0x7f5bdc0b3060 2024-03-06T07:25:06.405 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:25:06.405 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":60,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:25:04.078997+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.83:6837/3237506441":"2024-03-07T07:25:00.870034+0000","172.21.15.83:6834/3237506441":"2024-03-07T07:25:00.870034+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:06.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.405+0000 7f5be3e47640 1 -- 172.21.15.83:0/1419282610 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5bc006b4d0 msgr2=0x7f5bc006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.405+0000 7f5be3e47640 1 --2- 172.21.15.83:0/1419282610 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5bc006b4d0 0x7f5bc006d990 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f5bd8036000 tx=0x7f5bd8030000 comp rx=0 tx=0).stop 2024-03-06T07:25:06.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.405+0000 7f5be3e47640 1 -- 172.21.15.83:0/1419282610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5bdc0b3060 msgr2=0x7f5bdc1f5a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.405+0000 7f5be3e47640 1 --2- 172.21.15.83:0/1419282610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5bdc0b3060 0x7f5bdc1f5a40 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f5bd000c430 tx=0x7f5bd000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.405+0000 7f5be3e47640 1 -- 172.21.15.83:0/1419282610 shutdown_connections 2024-03-06T07:25:06.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.405+0000 7f5be3e47640 1 --2- 172.21.15.83:0/1419282610 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5bc006b4d0 0x7f5bc006d990 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.405+0000 7f5be3e47640 1 --2- 172.21.15.83:0/1419282610 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5bdc15bf30 0x7f5bdc0b0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.405+0000 7f5be3e47640 1 --2- 172.21.15.83:0/1419282610 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5bdc13dc10 0x7f5bdc144170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.405+0000 7f5be3e47640 1 --2- 172.21.15.83:0/1419282610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5bdc0b3060 0x7f5bdc1f5a40 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.405+0000 7f5be3e47640 1 -- 172.21.15.83:0/1419282610 >> 172.21.15.83:0/1419282610 conn(0x7f5bdc0cf8a0 msgr2=0x7f5bdc15c390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:06.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.405+0000 7f5be3e47640 1 -- 172.21.15.83:0/1419282610 shutdown_connections 2024-03-06T07:25:06.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.405+0000 7f5be3e47640 1 -- 172.21.15.83:0/1419282610 wait complete. 2024-03-06T07:25:06.418 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:25:06.418 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:25:06.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.518+0000 7fae40c40640 1 Processor -- start 2024-03-06T07:25:06.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.518+0000 7fae40c40640 1 -- start start 2024-03-06T07:25:06.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.519+0000 7fae40c40640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fae3c1586e0 0x7fae3c0aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.519+0000 7fae40c40640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fae3c0aef80 0x7fae3c0cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.519+0000 7fae40c40640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fae3c0cffd0 0x7fae3c06da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.519+0000 7fae40c40640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae3c158ac0 con 0x7fae3c1586e0 2024-03-06T07:25:06.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.519+0000 7fae40c40640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae3c15b2a0 con 0x7fae3c0aef80 2024-03-06T07:25:06.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.519+0000 7fae40c40640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae3c15b540 con 0x7fae3c0cffd0 2024-03-06T07:25:06.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.520+0000 7fae3a59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fae3c1586e0 0x7fae3c0aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.520+0000 7fae3a59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fae3c1586e0 0x7fae3c0aea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41884/0 (socket says 172.21.15.83:41884) 2024-03-06T07:25:06.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.520+0000 7fae3a59c640 1 -- 172.21.15.83:0/515261159 learned_addr learned my addr 172.21.15.83:0/515261159 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:06.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.520+0000 7fae3ad9d640 1 --2- 172.21.15.83:0/515261159 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fae3c0cffd0 0x7fae3c06da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.520+0000 7fae39d9b640 1 --2- 172.21.15.83:0/515261159 >> v2:172.21.15.183:3300/0 conn(0x7fae3c0aef80 0x7fae3c0cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.520+0000 7fae3a59c640 1 -- 172.21.15.83:0/515261159 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fae3c0cffd0 msgr2=0x7fae3c06da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:06.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.520+0000 7fae3a59c640 1 --2- 172.21.15.83:0/515261159 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fae3c0cffd0 0x7fae3c06da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.520+0000 7fae3a59c640 1 -- 172.21.15.83:0/515261159 >> v2:172.21.15.183:3300/0 conn(0x7fae3c0aef80 msgr2=0x7fae3c0cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:06.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.520+0000 7fae3a59c640 1 --2- 172.21.15.83:0/515261159 >> v2:172.21.15.183:3300/0 conn(0x7fae3c0aef80 0x7fae3c0cfa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.520+0000 7fae3a59c640 1 -- 172.21.15.83:0/515261159 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae3c15b820 con 0x7fae3c1586e0 2024-03-06T07:25:06.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.521+0000 7fae3a59c640 1 --2- 172.21.15.83:0/515261159 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae3c1586e0 0x7fae3c0aea40 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fae3c072e30 tx=0x7fae2c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eee260e2e507f1ca server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.521+0000 7fae3959a640 1 -- 172.21.15.83:0/515261159 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae2c02ca70 con 0x7fae3c1586e0 2024-03-06T07:25:06.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.521+0000 7fae3959a640 1 -- 172.21.15.83:0/515261159 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae2c02cc10 con 0x7fae3c1586e0 2024-03-06T07:25:06.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.521+0000 7fae3959a640 1 -- 172.21.15.83:0/515261159 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae2c0343f0 con 0x7fae3c1586e0 2024-03-06T07:25:06.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.521+0000 7fae40c40640 1 -- 172.21.15.83:0/515261159 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae3c1586e0 msgr2=0x7fae3c0aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.521+0000 7fae40c40640 1 --2- 172.21.15.83:0/515261159 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae3c1586e0 0x7fae3c0aea40 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fae3c072e30 tx=0x7fae2c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.521+0000 7fae40c40640 1 -- 172.21.15.83:0/515261159 shutdown_connections 2024-03-06T07:25:06.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.521+0000 7fae40c40640 1 --2- 172.21.15.83:0/515261159 >> v2:172.21.15.183:3300/0 conn(0x7fae3c0aef80 0x7fae3c0cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.521+0000 7fae40c40640 1 --2- 172.21.15.83:0/515261159 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fae3c0cffd0 0x7fae3c06da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.521+0000 7fae40c40640 1 --2- 172.21.15.83:0/515261159 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae3c1586e0 0x7fae3c0aea40 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.521+0000 7fae40c40640 1 -- 172.21.15.83:0/515261159 >> 172.21.15.83:0/515261159 conn(0x7fae3c06b190 msgr2=0x7fae3c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:06.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.521+0000 7fae40c40640 1 -- 172.21.15.83:0/515261159 shutdown_connections 2024-03-06T07:25:06.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.521+0000 7fae40c40640 1 -- 172.21.15.83:0/515261159 wait complete. 2024-03-06T07:25:06.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.523+0000 7fae40c40640 1 Processor -- start 2024-03-06T07:25:06.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.523+0000 7fae40c40640 1 -- start start 2024-03-06T07:25:06.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.523+0000 7fae40c40640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fae3c0aef80 0x7fae3c157ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.523+0000 7fae40c40640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae3c0cffd0 0x7fae3c14f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.523+0000 7fae40c40640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae3c1586e0 0x7fae3c148200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.523+0000 7fae40c40640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae3c071350 con 0x7fae3c0cffd0 2024-03-06T07:25:06.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.523+0000 7fae40c40640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae3c15bde0 con 0x7fae3c1586e0 2024-03-06T07:25:06.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.523+0000 7fae40c40640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae3c0714d0 con 0x7fae3c0aef80 2024-03-06T07:25:06.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.524+0000 7fae39d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae3c0cffd0 0x7fae3c14f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.524+0000 7fae39d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae3c0cffd0 0x7fae3c14f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41890/0 (socket says 172.21.15.83:41890) 2024-03-06T07:25:06.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.524+0000 7fae39d9b640 1 -- 172.21.15.83:0/3231378833 learned_addr learned my addr 172.21.15.83:0/3231378833 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:06.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.524+0000 7fae3a59c640 1 --2- 172.21.15.83:0/3231378833 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fae3c0aef80 0x7fae3c157ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.524+0000 7fae3ad9d640 1 --2- 172.21.15.83:0/3231378833 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae3c1586e0 0x7fae3c148200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.524+0000 7fae39d9b640 1 -- 172.21.15.83:0/3231378833 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fae3c0aef80 msgr2=0x7fae3c157ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.524+0000 7fae39d9b640 1 --2- 172.21.15.83:0/3231378833 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fae3c0aef80 0x7fae3c157ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.524+0000 7fae39d9b640 1 -- 172.21.15.83:0/3231378833 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae3c1586e0 msgr2=0x7fae3c148200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.524+0000 7fae39d9b640 1 --2- 172.21.15.83:0/3231378833 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae3c1586e0 0x7fae3c148200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.524+0000 7fae39d9b640 1 -- 172.21.15.83:0/3231378833 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae3c14fd50 con 0x7fae3c0cffd0 2024-03-06T07:25:06.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.524+0000 7fae3ad9d640 1 --2- 172.21.15.83:0/3231378833 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae3c1586e0 0x7fae3c148200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:06.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.524+0000 7fae3a59c640 1 --2- 172.21.15.83:0/3231378833 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fae3c0aef80 0x7fae3c157ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:06.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.524+0000 7fae39d9b640 1 --2- 172.21.15.83:0/3231378833 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae3c0cffd0 0x7fae3c14f5e0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fae3000c7f0 tx=0x7fae3000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.524+0000 7fae2affd640 1 -- 172.21.15.83:0/3231378833 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae30015070 con 0x7fae3c0cffd0 2024-03-06T07:25:06.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.524+0000 7fae2affd640 1 -- 172.21.15.83:0/3231378833 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae30004220 con 0x7fae3c0cffd0 2024-03-06T07:25:06.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.524+0000 7fae40c40640 1 -- 172.21.15.83:0/3231378833 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae3c150030 con 0x7fae3c0cffd0 2024-03-06T07:25:06.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.524+0000 7fae2affd640 1 -- 172.21.15.83:0/3231378833 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae30007620 con 0x7fae3c0cffd0 2024-03-06T07:25:06.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.524+0000 7fae40c40640 1 -- 172.21.15.83:0/3231378833 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae3c1505f0 con 0x7fae3c0cffd0 2024-03-06T07:25:06.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.525+0000 7fae40c40640 1 -- 172.21.15.83:0/3231378833 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae3c0732b0 con 0x7fae3c0cffd0 2024-03-06T07:25:06.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.526+0000 7fae2affd640 1 -- 172.21.15.83:0/3231378833 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fae300079d0 con 0x7fae3c0cffd0 2024-03-06T07:25:06.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.526+0000 7fae2affd640 1 --2- 172.21.15.83:0/3231378833 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fae1c06b500 0x7fae1c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.526+0000 7fae2affd640 1 -- 172.21.15.83:0/3231378833 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fae30089f40 con 0x7fae3c0cffd0 2024-03-06T07:25:06.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.526+0000 7fae3a59c640 1 --2- 172.21.15.83:0/3231378833 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fae1c06b500 0x7fae1c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.527+0000 7fae3a59c640 1 --2- 172.21.15.83:0/3231378833 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fae1c06b500 0x7fae1c06d9c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fae2c035000 tx=0x7fae2c030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.530+0000 7fae2affd640 1 -- 172.21.15.83:0/3231378833 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fae300590c0 con 0x7fae3c0cffd0 2024-03-06T07:25:06.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.699+0000 7fae40c40640 1 -- 172.21.15.83:0/3231378833 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fae3c0cfdb0 con 0x7fae3c0cffd0 2024-03-06T07:25:06.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.700+0000 7fae2affd640 1 -- 172.21.15.83:0/3231378833 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v60) v1 ==== 93+0+10 (secure 0 0 0) 0x7fae300589c0 con 0x7fae3c0cffd0 2024-03-06T07:25:06.702 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:25:06.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.702+0000 7fae40c40640 1 -- 172.21.15.83:0/3231378833 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fae1c06b500 msgr2=0x7fae1c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.702+0000 7fae40c40640 1 --2- 172.21.15.83:0/3231378833 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fae1c06b500 0x7fae1c06d9c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fae2c035000 tx=0x7fae2c030000 comp rx=0 tx=0).stop 2024-03-06T07:25:06.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.702+0000 7fae40c40640 1 -- 172.21.15.83:0/3231378833 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae3c0cffd0 msgr2=0x7fae3c14f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.702+0000 7fae40c40640 1 --2- 172.21.15.83:0/3231378833 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae3c0cffd0 0x7fae3c14f5e0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fae3000c7f0 tx=0x7fae3000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.702+0000 7fae40c40640 1 -- 172.21.15.83:0/3231378833 shutdown_connections 2024-03-06T07:25:06.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.702+0000 7fae40c40640 1 --2- 172.21.15.83:0/3231378833 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fae1c06b500 0x7fae1c06d9c0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.702+0000 7fae40c40640 1 --2- 172.21.15.83:0/3231378833 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae3c1586e0 0x7fae3c148200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.702+0000 7fae40c40640 1 --2- 172.21.15.83:0/3231378833 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae3c0cffd0 0x7fae3c14f5e0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.702+0000 7fae40c40640 1 --2- 172.21.15.83:0/3231378833 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fae3c0aef80 0x7fae3c157ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.702+0000 7fae40c40640 1 -- 172.21.15.83:0/3231378833 >> 172.21.15.83:0/3231378833 conn(0x7fae3c06b190 msgr2=0x7fae3c06b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:06.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.702+0000 7fae40c40640 1 -- 172.21.15.83:0/3231378833 shutdown_connections 2024-03-06T07:25:06.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.702+0000 7fae40c40640 1 -- 172.21.15.83:0/3231378833 wait complete. 2024-03-06T07:25:06.715 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays' 2024-03-06T07:25:06.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.813+0000 7f8e744eb640 1 Processor -- start 2024-03-06T07:25:06.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.813+0000 7f8e744eb640 1 -- start start 2024-03-06T07:25:06.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.814+0000 7f8e744eb640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f8e6c0cce40 0x7f8e6c15b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.814+0000 7f8e744eb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e6c071f40 con 0x7f8e6c0cd7a0 2024-03-06T07:25:06.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.814+0000 7f8e744eb640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e6c070a70 con 0x7f8e6c1515e0 2024-03-06T07:25:06.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.814+0000 7f8e744eb640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e6c0cd2f0 con 0x7f8e6c0cce40 2024-03-06T07:25:06.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.814+0000 7f8e72a88640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f8e6c0cd7a0 0x7f8e6c0cdb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48720/0 (socket says 172.21.15.83:48720) 2024-03-06T07:25:06.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.814+0000 7f8e72a88640 1 -- 172.21.15.83:0/2107674654 learned_addr learned my addr 172.21.15.83:0/2107674654 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:25:06.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.814+0000 7f8e71285640 1 -- 172.21.15.83:0/2107674654 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 883645303 0 0) 0x7f8e6c071f40 con 0x7f8e6c0cd7a0 2024-03-06T07:25:06.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.814+0000 7f8e71a86640 1 --2- 172.21.15.83:0/2107674654 >> v2:172.21.15.183:3300/0 conn(0x7f8e6c0cce40 0x7f8e6c15b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.814+0000 7f8e71285640 1 -- 172.21.15.83:0/2107674654 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e58003520 con 0x7f8e6c0cd7a0 2024-03-06T07:25:06.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.814+0000 7f8e71285640 1 -- 172.21.15.83:0/2107674654 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 12814157 0 0) 0x7f8e58003520 con 0x7f8e6c0cd7a0 2024-03-06T07:25:06.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.814+0000 7f8e71285640 1 -- 172.21.15.83:0/2107674654 >> v2:172.21.15.183:3300/0 conn(0x7f8e6c0cce40 msgr2=0x7f8e6c15b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:06.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.814+0000 7f8e71285640 1 --2- 172.21.15.83:0/2107674654 >> v2:172.21.15.183:3300/0 conn(0x7f8e6c0cce40 0x7f8e6c15b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.814+0000 7f8e71285640 1 -- 172.21.15.83:0/2107674654 >> v1:172.21.15.183:6789/0 conn(0x7f8e6c1515e0 legacy=0x7f8e6c071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.815+0000 7f8e71285640 1 -- 172.21.15.83:0/2107674654 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e6c070d50 con 0x7f8e6c0cd7a0 2024-03-06T07:25:06.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.815+0000 7f8e71285640 1 -- 172.21.15.83:0/2107674654 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f8e64001a10 con 0x7f8e6c0cd7a0 2024-03-06T07:25:06.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.815+0000 7f8e71285640 1 -- 172.21.15.83:0/2107674654 >> v1:172.21.15.83:6789/0 conn(0x7f8e6c0cd7a0 legacy=0x7f8e6c0cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.815+0000 7f8e71285640 1 --2- 172.21.15.83:0/2107674654 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e58003b80 0x7f8e58003fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.815+0000 7f8e71285640 1 --2- 172.21.15.83:0/2107674654 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8e58008830 0x7f8e5800acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.815+0000 7f8e71285640 1 --2- 172.21.15.83:0/2107674654 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8e5800b4c0 0x7f8e5800d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.815+0000 7f8e71285640 1 -- 172.21.15.83:0/2107674654 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6c0cd2f0 con 0x7f8e58003b80 2024-03-06T07:25:06.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.815+0000 7f8e71285640 1 -- 172.21.15.83:0/2107674654 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5800e260 con 0x7f8e5800b4c0 2024-03-06T07:25:06.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.815+0000 7f8e71285640 1 -- 172.21.15.83:0/2107674654 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5800e460 con 0x7f8e58008830 2024-03-06T07:25:06.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.815+0000 7f8e72287640 1 --2- 172.21.15.83:0/2107674654 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e58003b80 0x7f8e58003fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.815+0000 7f8e72a88640 1 --2- 172.21.15.83:0/2107674654 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8e5800b4c0 0x7f8e5800d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.815+0000 7f8e71a86640 1 --2- 172.21.15.83:0/2107674654 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8e58008830 0x7f8e5800acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.815+0000 7f8e72287640 1 -- 172.21.15.83:0/2107674654 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8e58008830 msgr2=0x7f8e5800acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:06.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.815+0000 7f8e72287640 1 --2- 172.21.15.83:0/2107674654 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8e58008830 0x7f8e5800acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.815+0000 7f8e72287640 1 -- 172.21.15.83:0/2107674654 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8e5800b4c0 msgr2=0x7f8e5800d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:06.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.815+0000 7f8e72287640 1 --2- 172.21.15.83:0/2107674654 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8e5800b4c0 0x7f8e5800d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.815+0000 7f8e72287640 1 -- 172.21.15.83:0/2107674654 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e58003520 con 0x7f8e58003b80 2024-03-06T07:25:06.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.815+0000 7f8e72a88640 1 --2- 172.21.15.83:0/2107674654 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8e5800b4c0 0x7f8e5800d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:06.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.815+0000 7f8e72a88640 1 -- 172.21.15.83:0/2107674654 reap_dead start 2024-03-06T07:25:06.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.816+0000 7f8e72287640 1 --2- 172.21.15.83:0/2107674654 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e58003b80 0x7f8e58003fa0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f8e68001750 tx=0x7f8e58003810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a547821ede43ef0a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.816+0000 7f8e71285640 1 -- 172.21.15.83:0/2107674654 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e68001370 con 0x7f8e58003b80 2024-03-06T07:25:06.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.816+0000 7f8e71285640 1 -- 172.21.15.83:0/2107674654 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e680052d0 con 0x7f8e58003b80 2024-03-06T07:25:06.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.816+0000 7f8e71285640 1 -- 172.21.15.83:0/2107674654 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e68003830 con 0x7f8e58003b80 2024-03-06T07:25:06.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.816+0000 7f8e744eb640 1 -- 172.21.15.83:0/2107674654 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e58003b80 msgr2=0x7f8e58003fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.816+0000 7f8e744eb640 1 --2- 172.21.15.83:0/2107674654 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e58003b80 0x7f8e58003fa0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f8e68001750 tx=0x7f8e58003810 comp rx=0 tx=0).stop 2024-03-06T07:25:06.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.816+0000 7f8e744eb640 1 -- 172.21.15.83:0/2107674654 shutdown_connections 2024-03-06T07:25:06.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.816+0000 7f8e744eb640 1 --2- 172.21.15.83:0/2107674654 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e58003b80 0x7f8e58003fa0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.816+0000 7f8e744eb640 1 -- 172.21.15.83:0/2107674654 >> 172.21.15.83:0/2107674654 conn(0x7f8e6c0cf8a0 msgr2=0x7f8e6c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:06.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.816+0000 7f8e744eb640 1 -- 172.21.15.83:0/2107674654 shutdown_connections 2024-03-06T07:25:06.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.816+0000 7f8e744eb640 1 -- 172.21.15.83:0/2107674654 wait complete. 2024-03-06T07:25:06.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.818+0000 7f8e744eb640 1 Processor -- start 2024-03-06T07:25:06.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.818+0000 7f8e744eb640 1 -- start start 2024-03-06T07:25:06.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.818+0000 7f8e744eb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e58003b80 0x7f8e6c14b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.818+0000 7f8e744eb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8e6c0cdb80 0x7f8e6c0ce000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.818+0000 7f8e744eb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8e6c15bb00 0x7f8e6c15bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.818+0000 7f8e744eb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6c15b430 con 0x7f8e58003b80 2024-03-06T07:25:06.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.818+0000 7f8e744eb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6c071310 con 0x7f8e6c0cdb80 2024-03-06T07:25:06.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.818+0000 7f8e744eb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6c15b5b0 con 0x7f8e6c15bb00 2024-03-06T07:25:06.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.818+0000 7f8e72287640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e58003b80 0x7f8e6c14b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.818+0000 7f8e72287640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e58003b80 0x7f8e6c14b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35376/0 (socket says 172.21.15.83:35376) 2024-03-06T07:25:06.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.818+0000 7f8e72287640 1 -- 172.21.15.83:0/943820500 learned_addr learned my addr 172.21.15.83:0/943820500 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:06.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.819+0000 7f8e71a86640 1 --2- 172.21.15.83:0/943820500 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8e6c0cdb80 0x7f8e6c0ce000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.819+0000 7f8e72287640 1 -- 172.21.15.83:0/943820500 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8e6c15bb00 msgr2=0x7f8e6c15bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.819+0000 7f8e72a88640 1 --2- 172.21.15.83:0/943820500 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8e6c15bb00 0x7f8e6c15bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.819+0000 7f8e72287640 1 --2- 172.21.15.83:0/943820500 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8e6c15bb00 0x7f8e6c15bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.819+0000 7f8e72287640 1 -- 172.21.15.83:0/943820500 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8e6c0cdb80 msgr2=0x7f8e6c0ce000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.819+0000 7f8e72287640 1 --2- 172.21.15.83:0/943820500 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8e6c0cdb80 0x7f8e6c0ce000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.819+0000 7f8e72287640 1 -- 172.21.15.83:0/943820500 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e6c15d0f0 con 0x7f8e58003b80 2024-03-06T07:25:06.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.819+0000 7f8e72287640 1 --2- 172.21.15.83:0/943820500 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e58003b80 0x7f8e6c14b780 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f8e68000c00 tx=0x7f8e68001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.819+0000 7f8e62ffd640 1 -- 172.21.15.83:0/943820500 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e68041020 con 0x7f8e58003b80 2024-03-06T07:25:06.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.819+0000 7f8e62ffd640 1 -- 172.21.15.83:0/943820500 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e68005480 con 0x7f8e58003b80 2024-03-06T07:25:06.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.819+0000 7f8e744eb640 1 -- 172.21.15.83:0/943820500 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e6c14bcc0 con 0x7f8e58003b80 2024-03-06T07:25:06.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.819+0000 7f8e62ffd640 1 -- 172.21.15.83:0/943820500 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e68007a30 con 0x7f8e58003b80 2024-03-06T07:25:06.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.819+0000 7f8e744eb640 1 -- 172.21.15.83:0/943820500 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e6c15cb30 con 0x7f8e58003b80 2024-03-06T07:25:06.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.820+0000 7f8e744eb640 1 -- 172.21.15.83:0/943820500 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e6c068840 con 0x7f8e58003b80 2024-03-06T07:25:06.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.820+0000 7f8e62ffd640 1 -- 172.21.15.83:0/943820500 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8e68044020 con 0x7f8e58003b80 2024-03-06T07:25:06.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.821+0000 7f8e62ffd640 1 --2- 172.21.15.83:0/943820500 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8e5006b500 0x7f8e5006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.821+0000 7f8e71a86640 1 --2- 172.21.15.83:0/943820500 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8e5006b500 0x7f8e5006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.821+0000 7f8e62ffd640 1 -- 172.21.15.83:0/943820500 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8e680ab970 con 0x7f8e58003b80 2024-03-06T07:25:06.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.821+0000 7f8e71a86640 1 --2- 172.21.15.83:0/943820500 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8e5006b500 0x7f8e5006d9c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c1f2e00 tx=0x7f8e5c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:06.825+0000 7f8e62ffd640 1 -- 172.21.15.83:0/943820500 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8e6804b030 con 0x7f8e58003b80 2024-03-06T07:25:07.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.061+0000 7f8e744eb640 1 -- 172.21.15.83:0/943820500 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]} v 0) v1 -- 0x7f8e6c04f330 con 0x7f8e58003b80 2024-03-06T07:25:07.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.100+0000 7f8e62ffd640 1 -- 172.21.15.83:0/943820500 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]}]=0 v187) v1 ==== 143+0+0 (secure 0 0 0) 0x7f8e680303c0 con 0x7f8e58003b80 2024-03-06T07:25:07.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.102+0000 7f8e744eb640 1 -- 172.21.15.83:0/943820500 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8e5006b500 msgr2=0x7f8e5006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.102+0000 7f8e744eb640 1 --2- 172.21.15.83:0/943820500 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8e5006b500 0x7f8e5006d9c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c1f2e00 tx=0x7f8e5c008040 comp rx=0 tx=0).stop 2024-03-06T07:25:07.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.102+0000 7f8e744eb640 1 -- 172.21.15.83:0/943820500 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e58003b80 msgr2=0x7f8e6c14b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.102+0000 7f8e744eb640 1 --2- 172.21.15.83:0/943820500 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e58003b80 0x7f8e6c14b780 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f8e68000c00 tx=0x7f8e68001120 comp rx=0 tx=0).stop 2024-03-06T07:25:07.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.102+0000 7f8e744eb640 1 -- 172.21.15.83:0/943820500 shutdown_connections 2024-03-06T07:25:07.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.102+0000 7f8e744eb640 1 --2- 172.21.15.83:0/943820500 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8e5006b500 0x7f8e5006d9c0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.102+0000 7f8e744eb640 1 --2- 172.21.15.83:0/943820500 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8e6c15bb00 0x7f8e6c15bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.102+0000 7f8e744eb640 1 --2- 172.21.15.83:0/943820500 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8e6c0cdb80 0x7f8e6c0ce000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.102+0000 7f8e744eb640 1 --2- 172.21.15.83:0/943820500 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e58003b80 0x7f8e6c14b780 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.102+0000 7f8e744eb640 1 -- 172.21.15.83:0/943820500 >> 172.21.15.83:0/943820500 conn(0x7f8e6c0cf8a0 msgr2=0x7f8e6c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:07.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.102+0000 7f8e744eb640 1 -- 172.21.15.83:0/943820500 shutdown_connections 2024-03-06T07:25:07.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.102+0000 7f8e744eb640 1 -- 172.21.15.83:0/943820500 wait complete. 2024-03-06T07:25:07.114 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:25:07.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.215+0000 7fe162829640 1 Processor -- start 2024-03-06T07:25:07.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.215+0000 7fe162829640 1 -- start start 2024-03-06T07:25:07.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.216+0000 7fe162829640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fe15c0cfa90 0x7fe15c0cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.216+0000 7fe162829640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe15c0b33a0 0x7fe15c0b37c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.216+0000 7fe162829640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe15c158ac0 con 0x7fe15c1586e0 2024-03-06T07:25:07.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.216+0000 7fe162829640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe15c15b2a0 con 0x7fe15c0cfa90 2024-03-06T07:25:07.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.216+0000 7fe162829640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c15b540 con 0x7fe15c0b33a0 2024-03-06T07:25:07.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.216+0000 7fe15bfff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fe15c1586e0 0x7fe15c0aea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48726/0 (socket says 172.21.15.83:48726) 2024-03-06T07:25:07.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.216+0000 7fe15bfff640 1 -- 172.21.15.83:0/1274646867 learned_addr learned my addr 172.21.15.83:0/1274646867 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:25:07.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.216+0000 7fe15b7fe640 1 --2- 172.21.15.83:0/1274646867 >> v2:172.21.15.183:3300/0 conn(0x7fe15c0cfa90 0x7fe15c0cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.216+0000 7fe160dc6640 1 --2- 172.21.15.83:0/1274646867 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe15c0b33a0 0x7fe15c0b37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.216+0000 7fe15affd640 1 -- 172.21.15.83:0/1274646867 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2135465590 0 0) 0x7fe15c158ac0 con 0x7fe15c1586e0 2024-03-06T07:25:07.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.216+0000 7fe15affd640 1 -- 172.21.15.83:0/1274646867 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe144003520 con 0x7fe15c1586e0 2024-03-06T07:25:07.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.216+0000 7fe15b7fe640 1 -- 172.21.15.83:0/1274646867 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe15c0b33a0 msgr2=0x7fe15c0b37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:07.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.216+0000 7fe15b7fe640 1 --2- 172.21.15.83:0/1274646867 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe15c0b33a0 0x7fe15c0b37c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.216+0000 7fe15b7fe640 1 -- 172.21.15.83:0/1274646867 >> v1:172.21.15.83:6789/0 conn(0x7fe15c1586e0 legacy=0x7fe15c0aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.216+0000 7fe15affd640 1 -- 172.21.15.83:0/1274646867 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2888023010 0 0) 0x7fe144003520 con 0x7fe15c1586e0 2024-03-06T07:25:07.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.216+0000 7fe15b7fe640 1 -- 172.21.15.83:0/1274646867 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe15c15b820 con 0x7fe15c0cfa90 2024-03-06T07:25:07.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.216+0000 7fe160dc6640 1 --2- 172.21.15.83:0/1274646867 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe15c0b33a0 0x7fe15c0b37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:07.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.217+0000 7fe15b7fe640 1 --2- 172.21.15.83:0/1274646867 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe15c0cfa90 0x7fe15c0cff40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fe1500097a0 tx=0x7fe15002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=337eea03547027a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:07.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.217+0000 7fe15affd640 1 -- 172.21.15.83:0/1274646867 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe150037070 con 0x7fe15c0cfa90 2024-03-06T07:25:07.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.217+0000 7fe15affd640 1 -- 172.21.15.83:0/1274646867 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe150030510 con 0x7fe15c0cfa90 2024-03-06T07:25:07.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.217+0000 7fe15affd640 1 -- 172.21.15.83:0/1274646867 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe14c001670 con 0x7fe15c0cfa90 2024-03-06T07:25:07.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.218+0000 7fe162829640 1 -- 172.21.15.83:0/1274646867 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe15c0cfa90 msgr2=0x7fe15c0cff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.218+0000 7fe162829640 1 --2- 172.21.15.83:0/1274646867 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe15c0cfa90 0x7fe15c0cff40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fe1500097a0 tx=0x7fe15002b790 comp rx=0 tx=0).stop 2024-03-06T07:25:07.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.218+0000 7fe162829640 1 -- 172.21.15.83:0/1274646867 shutdown_connections 2024-03-06T07:25:07.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.218+0000 7fe162829640 1 --2- 172.21.15.83:0/1274646867 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe15c0b33a0 0x7fe15c0b37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.218+0000 7fe162829640 1 --2- 172.21.15.83:0/1274646867 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe15c0cfa90 0x7fe15c0cff40 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.218+0000 7fe162829640 1 -- 172.21.15.83:0/1274646867 >> 172.21.15.83:0/1274646867 conn(0x7fe15c06b190 msgr2=0x7fe15c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:07.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.218+0000 7fe162829640 1 -- 172.21.15.83:0/1274646867 shutdown_connections 2024-03-06T07:25:07.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.218+0000 7fe162829640 1 -- 172.21.15.83:0/1274646867 wait complete. 2024-03-06T07:25:07.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.219+0000 7fe162829640 1 Processor -- start 2024-03-06T07:25:07.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.219+0000 7fe162829640 1 -- start start 2024-03-06T07:25:07.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.220+0000 7fe162829640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe15c0b33a0 0x7fe15c0cddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.220+0000 7fe162829640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe15c0cfa90 0x7fe15c0ac870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.220+0000 7fe162829640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe15c1586e0 0x7fe15c0acdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.220+0000 7fe162829640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c0af150 con 0x7fe15c1586e0 2024-03-06T07:25:07.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.220+0000 7fe162829640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c15bde0 con 0x7fe15c0b33a0 2024-03-06T07:25:07.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.220+0000 7fe162829640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c071350 con 0x7fe15c0cfa90 2024-03-06T07:25:07.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.220+0000 7fe160dc6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe15c1586e0 0x7fe15c0acdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.220+0000 7fe160dc6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe15c1586e0 0x7fe15c0acdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35390/0 (socket says 172.21.15.83:35390) 2024-03-06T07:25:07.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.220+0000 7fe160dc6640 1 -- 172.21.15.83:0/4170697168 learned_addr learned my addr 172.21.15.83:0/4170697168 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:07.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.220+0000 7fe15bfff640 1 --2- 172.21.15.83:0/4170697168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe15c0b33a0 0x7fe15c0cddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.220+0000 7fe15b7fe640 1 --2- 172.21.15.83:0/4170697168 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe15c0cfa90 0x7fe15c0ac870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.220+0000 7fe160dc6640 1 -- 172.21.15.83:0/4170697168 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe15c0cfa90 msgr2=0x7fe15c0ac870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.220+0000 7fe160dc6640 1 --2- 172.21.15.83:0/4170697168 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe15c0cfa90 0x7fe15c0ac870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.220+0000 7fe160dc6640 1 -- 172.21.15.83:0/4170697168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe15c0b33a0 msgr2=0x7fe15c0cddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.220+0000 7fe160dc6640 1 --2- 172.21.15.83:0/4170697168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe15c0b33a0 0x7fe15c0cddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.220+0000 7fe160dc6640 1 -- 172.21.15.83:0/4170697168 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe15c0ab120 con 0x7fe15c1586e0 2024-03-06T07:25:07.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.221+0000 7fe160dc6640 1 --2- 172.21.15.83:0/4170697168 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe15c1586e0 0x7fe15c0acdb0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fe154009f00 tx=0x7fe15400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:07.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.221+0000 7fe158ff9640 1 -- 172.21.15.83:0/4170697168 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe154018030 con 0x7fe15c1586e0 2024-03-06T07:25:07.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.221+0000 7fe158ff9640 1 -- 172.21.15.83:0/4170697168 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe15400ce50 con 0x7fe15c1586e0 2024-03-06T07:25:07.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.221+0000 7fe158ff9640 1 -- 172.21.15.83:0/4170697168 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe154003bb0 con 0x7fe15c1586e0 2024-03-06T07:25:07.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.221+0000 7fe162829640 1 -- 172.21.15.83:0/4170697168 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe15c0ab400 con 0x7fe15c1586e0 2024-03-06T07:25:07.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.221+0000 7fe162829640 1 -- 172.21.15.83:0/4170697168 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe15c0ab9c0 con 0x7fe15c1586e0 2024-03-06T07:25:07.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.222+0000 7fe162829640 1 -- 172.21.15.83:0/4170697168 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe15c0732b0 con 0x7fe15c1586e0 2024-03-06T07:25:07.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.222+0000 7fe158ff9640 1 -- 172.21.15.83:0/4170697168 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe15401b070 con 0x7fe15c1586e0 2024-03-06T07:25:07.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.223+0000 7fe158ff9640 1 --2- 172.21.15.83:0/4170697168 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe14806b4d0 0x7fe14806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.223+0000 7fe158ff9640 1 -- 172.21.15.83:0/4170697168 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe15408a7c0 con 0x7fe15c1586e0 2024-03-06T07:25:07.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.223+0000 7fe15bfff640 1 --2- 172.21.15.83:0/4170697168 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe14806b4d0 0x7fe14806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.223+0000 7fe15bfff640 1 --2- 172.21.15.83:0/4170697168 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe14806b4d0 0x7fe14806d990 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fe14c001320 tx=0x7fe14c0012b0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:07.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.227+0000 7fe158ff9640 1 -- 172.21.15.83:0/4170697168 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe154010040 con 0x7fe15c1586e0 2024-03-06T07:25:07.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.399+0000 7fe162829640 1 -- 172.21.15.83:0/4170697168 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fe15c15a6f0 con 0x7fe15c1586e0 2024-03-06T07:25:07.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.403+0000 7fe158ff9640 1 -- 172.21.15.83:0/4170697168 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe154059380 con 0x7fe15c1586e0 2024-03-06T07:25:07.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.405+0000 7fe162829640 1 -- 172.21.15.83:0/4170697168 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe14806b4d0 msgr2=0x7fe14806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.405+0000 7fe162829640 1 --2- 172.21.15.83:0/4170697168 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe14806b4d0 0x7fe14806d990 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fe14c001320 tx=0x7fe14c0012b0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.405+0000 7fe162829640 1 -- 172.21.15.83:0/4170697168 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe15c1586e0 msgr2=0x7fe15c0acdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.405+0000 7fe162829640 1 --2- 172.21.15.83:0/4170697168 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe15c1586e0 0x7fe15c0acdb0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fe154009f00 tx=0x7fe15400c490 comp rx=0 tx=0).stop 2024-03-06T07:25:07.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.405+0000 7fe162829640 1 -- 172.21.15.83:0/4170697168 shutdown_connections 2024-03-06T07:25:07.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.405+0000 7fe162829640 1 --2- 172.21.15.83:0/4170697168 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe14806b4d0 0x7fe14806d990 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.405+0000 7fe162829640 1 --2- 172.21.15.83:0/4170697168 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe15c1586e0 0x7fe15c0acdb0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.405+0000 7fe162829640 1 --2- 172.21.15.83:0/4170697168 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe15c0cfa90 0x7fe15c0ac870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.405+0000 7fe162829640 1 --2- 172.21.15.83:0/4170697168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe15c0b33a0 0x7fe15c0cddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.405+0000 7fe162829640 1 -- 172.21.15.83:0/4170697168 >> 172.21.15.83:0/4170697168 conn(0x7fe15c06b190 msgr2=0x7fe15c157580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:07.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.405+0000 7fe162829640 1 -- 172.21.15.83:0/4170697168 shutdown_connections 2024-03-06T07:25:07.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.405+0000 7fe162829640 1 -- 172.21.15.83:0/4170697168 wait complete. 2024-03-06T07:25:07.417 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:25:07.418 DEBUG:teuthology.orchestra.run.smithi083:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:07.454 DEBUG:teuthology.orchestra.run.smithi183:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:07.500 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:25:07.500 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:25:07.500 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:07.515 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1257 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:25:07.515 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:25:07.515 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:07.532 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=287 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:25:07.532 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:25:07.532 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:07.570 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1257 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:25:07.571 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:25:07.571 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:07.588 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=287 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:25:07.589 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:25:07.589 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:07.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.708+0000 7f5a463c7640 1 Processor -- start 2024-03-06T07:25:07.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.708+0000 7f5a463c7640 1 -- start start 2024-03-06T07:25:07.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.709+0000 7f5a463c7640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f5a4015dfe0 0x7f5a400cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.709+0000 7f5a463c7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5a400d29c0 0x7f5a400b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.709+0000 7f5a463c7640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a400b3430 con 0x7f5a400cf830 2024-03-06T07:25:07.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.709+0000 7f5a463c7640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a400b31e0 con 0x7f5a4015dfe0 2024-03-06T07:25:07.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.709+0000 7f5a463c7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a400b3d60 con 0x7f5a400d29c0 2024-03-06T07:25:07.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.710+0000 7f5a44964640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5a400d29c0 0x7f5a400b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.710+0000 7f5a3ffff640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f5a4015dfe0 0x7f5a400cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.710+0000 7f5a44964640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5a400d29c0 0x7f5a400b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:57410/0 (socket says 172.21.15.83:57410) 2024-03-06T07:25:07.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.710+0000 7f5a3f7fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f5a400cf830 0x7f5a400d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:57854/0 (socket says 172.21.15.83:57854) 2024-03-06T07:25:07.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.710+0000 7f5a44964640 1 -- 172.21.15.83:0/1903184433 learned_addr learned my addr 172.21.15.83:0/1903184433 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:07.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.710+0000 7f5a3ffff640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f5a4015dfe0 0x7f5a400cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:38082/0 (socket says 172.21.15.83:38082) 2024-03-06T07:25:07.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.710+0000 7f5a44964640 1 -- 172.21.15.83:0/1903184433 >> v2:172.21.15.183:3300/0 conn(0x7f5a4015dfe0 msgr2=0x7f5a400cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:07.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.710+0000 7f5a44964640 1 --2- 172.21.15.83:0/1903184433 >> v2:172.21.15.183:3300/0 conn(0x7f5a4015dfe0 0x7f5a400cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.710+0000 7f5a44964640 1 -- 172.21.15.83:0/1903184433 >> v1:172.21.15.183:6789/0 conn(0x7f5a400cf830 legacy=0x7f5a400d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.710+0000 7f5a44964640 1 -- 172.21.15.83:0/1903184433 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a400b3710 con 0x7f5a400d29c0 2024-03-06T07:25:07.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.711+0000 7f5a44964640 1 --2- 172.21.15.83:0/1903184433 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5a400d29c0 0x7f5a400b41f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5a38009e50 tx=0x7f5a3802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=910c4e31b9944f22 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:07.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.711+0000 7f5a3effd640 1 -- 172.21.15.83:0/1903184433 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3802cb00 con 0x7f5a400d29c0 2024-03-06T07:25:07.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.711+0000 7f5a3effd640 1 -- 172.21.15.83:0/1903184433 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a3802cca0 con 0x7f5a400d29c0 2024-03-06T07:25:07.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.711+0000 7f5a3effd640 1 -- 172.21.15.83:0/1903184433 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a38034610 con 0x7f5a400d29c0 2024-03-06T07:25:07.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.711+0000 7f5a463c7640 1 -- 172.21.15.83:0/1903184433 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5a400d29c0 msgr2=0x7f5a400b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.711+0000 7f5a463c7640 1 --2- 172.21.15.83:0/1903184433 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5a400d29c0 0x7f5a400b41f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5a38009e50 tx=0x7f5a3802bf60 comp rx=0 tx=0).stop 2024-03-06T07:25:07.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.711+0000 7f5a463c7640 1 -- 172.21.15.83:0/1903184433 shutdown_connections 2024-03-06T07:25:07.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.711+0000 7f5a463c7640 1 --2- 172.21.15.83:0/1903184433 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5a400d29c0 0x7f5a400b41f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.711+0000 7f5a463c7640 1 --2- 172.21.15.83:0/1903184433 >> v2:172.21.15.183:3300/0 conn(0x7f5a4015dfe0 0x7f5a400cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.711+0000 7f5a463c7640 1 -- 172.21.15.83:0/1903184433 >> 172.21.15.83:0/1903184433 conn(0x7f5a400d3cd0 msgr2=0x7f5a400d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:07.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.711+0000 7f5a463c7640 1 -- 172.21.15.83:0/1903184433 shutdown_connections 2024-03-06T07:25:07.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.711+0000 7f5a463c7640 1 -- 172.21.15.83:0/1903184433 wait complete. 2024-03-06T07:25:07.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.713+0000 7f5a463c7640 1 Processor -- start 2024-03-06T07:25:07.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.713+0000 7f5a463c7640 1 -- start start 2024-03-06T07:25:07.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.713+0000 7f5a463c7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5a400cf830 0x7f5a401539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.713+0000 7f5a463c7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a400d29c0 0x7f5a4014d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.713+0000 7f5a463c7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5a4015dfe0 0x7f5a40151be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.713+0000 7f5a463c7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a400d1df0 con 0x7f5a400d29c0 2024-03-06T07:25:07.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.713+0000 7f5a463c7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a400d1c70 con 0x7f5a400cf830 2024-03-06T07:25:07.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.713+0000 7f5a463c7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4015e3c0 con 0x7f5a4015dfe0 2024-03-06T07:25:07.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.714+0000 7f5a3f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a400d29c0 0x7f5a4014d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.714+0000 7f5a3f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a400d29c0 0x7f5a4014d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35398/0 (socket says 172.21.15.83:35398) 2024-03-06T07:25:07.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.714+0000 7f5a3f7fe640 1 -- 172.21.15.83:0/2967560590 learned_addr learned my addr 172.21.15.83:0/2967560590 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:07.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.714+0000 7f5a44964640 1 --2- 172.21.15.83:0/2967560590 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5a4015dfe0 0x7f5a40151be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.714+0000 7f5a3ffff640 1 --2- 172.21.15.83:0/2967560590 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5a400cf830 0x7f5a401539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.714+0000 7f5a3f7fe640 1 -- 172.21.15.83:0/2967560590 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5a4015dfe0 msgr2=0x7f5a40151be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.714+0000 7f5a3f7fe640 1 --2- 172.21.15.83:0/2967560590 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5a4015dfe0 0x7f5a40151be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.714+0000 7f5a3f7fe640 1 -- 172.21.15.83:0/2967560590 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5a400cf830 msgr2=0x7f5a401539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.714+0000 7f5a3f7fe640 1 --2- 172.21.15.83:0/2967560590 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5a400cf830 0x7f5a401539e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.714+0000 7f5a3f7fe640 1 -- 172.21.15.83:0/2967560590 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a4014d860 con 0x7f5a400d29c0 2024-03-06T07:25:07.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.714+0000 7f5a44964640 1 --2- 172.21.15.83:0/2967560590 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5a4015dfe0 0x7f5a40151be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:07.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.714+0000 7f5a3ffff640 1 --2- 172.21.15.83:0/2967560590 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5a400cf830 0x7f5a401539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:07.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.714+0000 7f5a3f7fe640 1 --2- 172.21.15.83:0/2967560590 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a400d29c0 0x7f5a4014d320 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f5a3400c470 tx=0x7f5a34001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:07.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.714+0000 7f5a3cff9640 1 -- 172.21.15.83:0/2967560590 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a340074e0 con 0x7f5a400d29c0 2024-03-06T07:25:07.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.714+0000 7f5a3cff9640 1 -- 172.21.15.83:0/2967560590 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a34007680 con 0x7f5a400d29c0 2024-03-06T07:25:07.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.714+0000 7f5a3cff9640 1 -- 172.21.15.83:0/2967560590 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a34004a20 con 0x7f5a400d29c0 2024-03-06T07:25:07.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.714+0000 7f5a463c7640 1 -- 172.21.15.83:0/2967560590 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a40152940 con 0x7f5a400d29c0 2024-03-06T07:25:07.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.714+0000 7f5a463c7640 1 -- 172.21.15.83:0/2967560590 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a40152f00 con 0x7f5a400d29c0 2024-03-06T07:25:07.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.715+0000 7f5a463c7640 1 -- 172.21.15.83:0/2967560590 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a4004f330 con 0x7f5a400d29c0 2024-03-06T07:25:07.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.716+0000 7f5a3cff9640 1 -- 172.21.15.83:0/2967560590 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5a3401a020 con 0x7f5a400d29c0 2024-03-06T07:25:07.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.716+0000 7f5a3cff9640 1 --2- 172.21.15.83:0/2967560590 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5a2c06b4d0 0x7f5a2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.716+0000 7f5a3cff9640 1 -- 172.21.15.83:0/2967560590 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5a34089700 con 0x7f5a400d29c0 2024-03-06T07:25:07.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.716+0000 7f5a3ffff640 1 --2- 172.21.15.83:0/2967560590 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5a2c06b4d0 0x7f5a2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.717+0000 7f5a3ffff640 1 --2- 172.21.15.83:0/2967560590 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5a2c06b4d0 0x7f5a2c06d990 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f5a30009af0 tx=0x7f5a30008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:07.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.720+0000 7f5a3cff9640 1 -- 172.21.15.83:0/2967560590 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5a34010300 con 0x7f5a400d29c0 2024-03-06T07:25:07.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.928+0000 7f5a463c7640 1 -- 172.21.15.83:0/2967560590 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a400b2440 con 0x7f5a400d29c0 2024-03-06T07:25:07.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.928+0000 7f5a3cff9640 1 -- 172.21.15.83:0/2967560590 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+3699 (secure 0 0 0) 0x7f5a34058860 con 0x7f5a400d29c0 2024-03-06T07:25:07.930 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:25:07.930 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":27,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5665,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/3334609507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3334609507},{"type":"v1","addr":"172.21.15.183:6834","nonce":3334609507}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5668,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/691189112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":691189112},{"type":"v1","addr":"172.21.15.183:6835","nonce":691189112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6081,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/3952571284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":3952571284},{"type":"v1","addr":"172.21.15.83:6836","nonce":3952571284}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6462,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/386737356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":386737356},{"type":"v1","addr":"172.21.15.83:6837","nonce":386737356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[]} 2024-03-06T07:25:07.931 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 27 2024-03-06T07:25:07.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.930+0000 7f5a463c7640 1 -- 172.21.15.83:0/2967560590 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5a2c06b4d0 msgr2=0x7f5a2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.930+0000 7f5a463c7640 1 --2- 172.21.15.83:0/2967560590 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5a2c06b4d0 0x7f5a2c06d990 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f5a30009af0 tx=0x7f5a30008040 comp rx=0 tx=0).stop 2024-03-06T07:25:07.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.930+0000 7f5a463c7640 1 -- 172.21.15.83:0/2967560590 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a400d29c0 msgr2=0x7f5a4014d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.930+0000 7f5a463c7640 1 --2- 172.21.15.83:0/2967560590 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a400d29c0 0x7f5a4014d320 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f5a3400c470 tx=0x7f5a34001e70 comp rx=0 tx=0).stop 2024-03-06T07:25:07.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.930+0000 7f5a463c7640 1 -- 172.21.15.83:0/2967560590 shutdown_connections 2024-03-06T07:25:07.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.930+0000 7f5a463c7640 1 --2- 172.21.15.83:0/2967560590 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5a2c06b4d0 0x7f5a2c06d990 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.930+0000 7f5a463c7640 1 --2- 172.21.15.83:0/2967560590 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5a4015dfe0 0x7f5a40151be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.930+0000 7f5a463c7640 1 --2- 172.21.15.83:0/2967560590 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a400d29c0 0x7f5a4014d320 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.930+0000 7f5a463c7640 1 --2- 172.21.15.83:0/2967560590 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5a400cf830 0x7f5a401539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.930+0000 7f5a463c7640 1 -- 172.21.15.83:0/2967560590 >> 172.21.15.83:0/2967560590 conn(0x7f5a400d3cd0 msgr2=0x7f5a400b2990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:07.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.930+0000 7f5a463c7640 1 -- 172.21.15.83:0/2967560590 shutdown_connections 2024-03-06T07:25:07.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:07.930+0000 7f5a463c7640 1 -- 172.21.15.83:0/2967560590 wait complete. 2024-03-06T07:25:07.943 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:25:07.943 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:25:07.944 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:25:07.944 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:25:07.944 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:25:07.944 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:25:07.944 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:25:07.944 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:25:07.944 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:25:07.945 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:25:07.945 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:25:07.945 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:25:07.945 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:25:07.945 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:25:07.946 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:25:07.946 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:25:07.953 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:25:07.953 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:25:07.954 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:25:07.954 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:25:07.959 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:25:07.960 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:25:07.960 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:25:07.961 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:25:07.996 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:25:07.996 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:07.996 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:25:07.997 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:08.003 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:25:08.003 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:08.003 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:25:08.004 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:08.004 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:25:08.073 INFO:tasks.ceph.mds.c.smithi083.stdout:starting mds.c at 2024-03-06T07:25:08.073 INFO:tasks.ceph.mds.a.smithi083.stdout:starting mds.a at 2024-03-06T07:25:08.086 INFO:tasks.ceph.mds.b.smithi183.stdout:starting mds.b at 2024-03-06T07:25:08.086 INFO:tasks.ceph.mds.d.smithi183.stdout:starting mds.d at 2024-03-06T07:25:08.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.107+0000 7f6806eeb640 1 Processor -- start 2024-03-06T07:25:08.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.107+0000 7f6806eeb640 1 -- start start 2024-03-06T07:25:08.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.108+0000 7f6806eeb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f68000b4460 0x7f68000d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.108+0000 7f6806eeb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6800162970 0x7f68000b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.108+0000 7f6806eeb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68000d22c0 con 0x7f6800161cf0 2024-03-06T07:25:08.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.108+0000 7f6806eeb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6800161b00 con 0x7f6800162970 2024-03-06T07:25:08.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.108+0000 7f6806eeb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68000cecc0 con 0x7f68000b4460 2024-03-06T07:25:08.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.108+0000 7f67fffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6800162970 0x7f68000b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.108+0000 7f67fffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6800162970 0x7f68000b3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35436/0 (socket says 172.21.15.83:35436) 2024-03-06T07:25:08.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.108+0000 7f67fffff640 1 -- 172.21.15.83:0/406267256 learned_addr learned my addr 172.21.15.83:0/406267256 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:08.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.109+0000 7f6804c87640 1 --2- 172.21.15.83:0/406267256 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f68000b4460 0x7f68000d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.109+0000 7f67ff7fe640 1 -- 172.21.15.83:0/406267256 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2036471562 0 0) 0x7f68000d22c0 con 0x7f6800161cf0 2024-03-06T07:25:08.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.109+0000 7f67ff7fe640 1 -- 172.21.15.83:0/406267256 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67f0003520 con 0x7f6800161cf0 2024-03-06T07:25:08.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.109+0000 7f6804c87640 1 -- 172.21.15.83:0/406267256 >> v2:172.21.15.83:3300/0 conn(0x7f6800162970 msgr2=0x7f68000b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:08.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.109+0000 7f6804c87640 1 --2- 172.21.15.83:0/406267256 >> v2:172.21.15.83:3300/0 conn(0x7f6800162970 0x7f68000b3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.109+0000 7f6804c87640 1 -- 172.21.15.83:0/406267256 >> v1:172.21.15.83:6789/0 conn(0x7f6800161cf0 legacy=0x7f68000cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.109+0000 7f6804c87640 1 -- 172.21.15.83:0/406267256 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68000d25a0 con 0x7f68000b4460 2024-03-06T07:25:08.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.109+0000 7f67fffff640 1 --2- 172.21.15.83:0/406267256 >> v2:172.21.15.83:3300/0 conn(0x7f6800162970 0x7f68000b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:08.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.110+0000 7f6804c87640 1 --2- 172.21.15.83:0/406267256 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f68000b4460 0x7f68000d4800 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f67f40097a0 tx=0x7f67f402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=979cfde0a94ded1a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:08.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.112+0000 7f67ff7fe640 1 -- 172.21.15.83:0/406267256 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f4035020 con 0x7f68000b4460 2024-03-06T07:25:08.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.112+0000 7f67ff7fe640 1 -- 172.21.15.83:0/406267256 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67f4031810 con 0x7f68000b4460 2024-03-06T07:25:08.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.112+0000 7f67ff7fe640 1 -- 172.21.15.83:0/406267256 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f4031bc0 con 0x7f68000b4460 2024-03-06T07:25:08.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.112+0000 7f6806eeb640 1 -- 172.21.15.83:0/406267256 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f68000b4460 msgr2=0x7f68000d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.112+0000 7f6806eeb640 1 --2- 172.21.15.83:0/406267256 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f68000b4460 0x7f68000d4800 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f67f40097a0 tx=0x7f67f402b8b0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.112+0000 7f6806eeb640 1 -- 172.21.15.83:0/406267256 shutdown_connections 2024-03-06T07:25:08.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.112+0000 7f6806eeb640 1 --2- 172.21.15.83:0/406267256 >> v2:172.21.15.83:3300/0 conn(0x7f6800162970 0x7f68000b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.112+0000 7f6806eeb640 1 --2- 172.21.15.83:0/406267256 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f68000b4460 0x7f68000d4800 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.112+0000 7f6806eeb640 1 -- 172.21.15.83:0/406267256 >> 172.21.15.83:0/406267256 conn(0x7f68000d3ce0 msgr2=0x7f68000d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:08.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.112+0000 7f6806eeb640 1 -- 172.21.15.83:0/406267256 shutdown_connections 2024-03-06T07:25:08.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.112+0000 7f6806eeb640 1 -- 172.21.15.83:0/406267256 wait complete. 2024-03-06T07:25:08.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.114+0000 7f6806eeb640 1 Processor -- start 2024-03-06T07:25:08.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.114+0000 7f6806eeb640 1 -- start start 2024-03-06T07:25:08.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.115+0000 7f6806eeb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68000b4460 0x7f6800158820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.115+0000 7f6806eeb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6800161cf0 0x7f6800156180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.115+0000 7f6806eeb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6800162970 0x7f6800156be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.115+0000 7f6806eeb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68000ce950 con 0x7f68000b4460 2024-03-06T07:25:08.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.115+0000 7f6806eeb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68000d2b60 con 0x7f6800161cf0 2024-03-06T07:25:08.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.115+0000 7f6806eeb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68000b3ac0 con 0x7f6800162970 2024-03-06T07:25:08.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.115+0000 7f6804c87640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68000b4460 0x7f6800158820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.115+0000 7f6804c87640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68000b4460 0x7f6800158820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35442/0 (socket says 172.21.15.83:35442) 2024-03-06T07:25:08.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.115+0000 7f6804c87640 1 -- 172.21.15.83:0/2815606244 learned_addr learned my addr 172.21.15.83:0/2815606244 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:08.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.115+0000 7f67fffff640 1 --2- 172.21.15.83:0/2815606244 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6800161cf0 0x7f6800156180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.115+0000 7f6805488640 1 --2- 172.21.15.83:0/2815606244 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6800162970 0x7f6800156be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.115+0000 7f6804c87640 1 -- 172.21.15.83:0/2815606244 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6800162970 msgr2=0x7f6800156be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.115+0000 7f6804c87640 1 --2- 172.21.15.83:0/2815606244 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6800162970 0x7f6800156be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.115+0000 7f6804c87640 1 -- 172.21.15.83:0/2815606244 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6800161cf0 msgr2=0x7f6800156180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.115+0000 7f6804c87640 1 --2- 172.21.15.83:0/2815606244 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6800161cf0 0x7f6800156180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.115+0000 7f6804c87640 1 -- 172.21.15.83:0/2815606244 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68000b6c10 con 0x7f68000b4460 2024-03-06T07:25:08.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.115+0000 7f6805488640 1 --2- 172.21.15.83:0/2815606244 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6800162970 0x7f6800156be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:08.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.115+0000 7f6804c87640 1 --2- 172.21.15.83:0/2815606244 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68000b4460 0x7f6800158820 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f67f402b8f0 tx=0x7f67f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:08.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.116+0000 7f67fd7fa640 1 -- 172.21.15.83:0/2815606244 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f4040070 con 0x7f68000b4460 2024-03-06T07:25:08.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.116+0000 7f6806eeb640 1 -- 172.21.15.83:0/2815606244 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6800157780 con 0x7f68000b4460 2024-03-06T07:25:08.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.116+0000 7f67fd7fa640 1 -- 172.21.15.83:0/2815606244 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67f4039cf0 con 0x7f68000b4460 2024-03-06T07:25:08.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.116+0000 7f67fd7fa640 1 -- 172.21.15.83:0/2815606244 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f4035040 con 0x7f68000b4460 2024-03-06T07:25:08.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.116+0000 7f6806eeb640 1 -- 172.21.15.83:0/2815606244 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6800157d40 con 0x7f68000b4460 2024-03-06T07:25:08.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.117+0000 7f6806eeb640 1 -- 172.21.15.83:0/2815606244 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f680004f330 con 0x7f68000b4460 2024-03-06T07:25:08.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.117+0000 7f67fd7fa640 1 -- 172.21.15.83:0/2815606244 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f67f404b060 con 0x7f68000b4460 2024-03-06T07:25:08.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.117+0000 7f67fd7fa640 1 --2- 172.21.15.83:0/2815606244 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f67e406b4d0 0x7f67e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.118+0000 7f67fd7fa640 1 -- 172.21.15.83:0/2815606244 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f67f40ab7d0 con 0x7f68000b4460 2024-03-06T07:25:08.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.118+0000 7f67fffff640 1 --2- 172.21.15.83:0/2815606244 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f67e406b4d0 0x7f67e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.118+0000 7f67fffff640 1 --2- 172.21.15.83:0/2815606244 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f67e406b4d0 0x7f67e406d990 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f68001f74e0 tx=0x7f67ec009240 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:08.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.123+0000 7f67fd7fa640 1 -- 172.21.15.83:0/2815606244 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f67f407a300 con 0x7f68000b4460 2024-03-06T07:25:08.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:08.309+0000 7f6806eeb640 1 -- 172.21.15.83:0/2815606244 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f68000b2e30 con 0x7f68000b4460 2024-03-06T07:25:09.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.117+0000 7f67fd7fa640 1 -- 172.21.15.83:0/2815606244 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v65) v1 ==== 97+0+0 (secure 0 0 0) 0x7f67f407e3a0 con 0x7f68000b4460 2024-03-06T07:25:09.119 INFO:teuthology.orchestra.run.smithi083.stderr: removed all blocklist entries 2024-03-06T07:25:09.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.120+0000 7f6806eeb640 1 -- 172.21.15.83:0/2815606244 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f67e406b4d0 msgr2=0x7f67e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.120+0000 7f6806eeb640 1 --2- 172.21.15.83:0/2815606244 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f67e406b4d0 0x7f67e406d990 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f68001f74e0 tx=0x7f67ec009240 comp rx=0 tx=0).stop 2024-03-06T07:25:09.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.120+0000 7f6806eeb640 1 -- 172.21.15.83:0/2815606244 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68000b4460 msgr2=0x7f6800158820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.120+0000 7f6806eeb640 1 --2- 172.21.15.83:0/2815606244 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68000b4460 0x7f6800158820 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f67f402b8f0 tx=0x7f67f40046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.120+0000 7f6806eeb640 1 -- 172.21.15.83:0/2815606244 shutdown_connections 2024-03-06T07:25:09.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.120+0000 7f6806eeb640 1 --2- 172.21.15.83:0/2815606244 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f67e406b4d0 0x7f67e406d990 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.120+0000 7f6806eeb640 1 --2- 172.21.15.83:0/2815606244 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6800162970 0x7f6800156be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.120+0000 7f6806eeb640 1 --2- 172.21.15.83:0/2815606244 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6800161cf0 0x7f6800156180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.120+0000 7f6806eeb640 1 --2- 172.21.15.83:0/2815606244 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68000b4460 0x7f6800158820 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.120+0000 7f6806eeb640 1 -- 172.21.15.83:0/2815606244 >> 172.21.15.83:0/2815606244 conn(0x7f68000d3ce0 msgr2=0x7f68001611b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:09.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.120+0000 7f6806eeb640 1 -- 172.21.15.83:0/2815606244 shutdown_connections 2024-03-06T07:25:09.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.120+0000 7f6806eeb640 1 -- 172.21.15.83:0/2815606244 wait complete. 2024-03-06T07:25:09.135 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:25:09.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.239+0000 7fe74ef8a640 1 Processor -- start 2024-03-06T07:25:09.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.239+0000 7fe74ef8a640 1 -- start start 2024-03-06T07:25:09.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.239+0000 7fe74ef8a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe7480d2710 0x7fe7480b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.239+0000 7fe74ef8a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7480b3430 con 0x7fe7480d22b0 2024-03-06T07:25:09.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.239+0000 7fe74ef8a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7480b3710 con 0x7fe74815dfe0 2024-03-06T07:25:09.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.239+0000 7fe74ef8a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7480b31e0 con 0x7fe7480d2710 2024-03-06T07:25:09.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.240+0000 7fe74d527640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe7480d2710 0x7fe7480b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.240+0000 7fe747fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fe7480d22b0 0x7fe7480b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48774/0 (socket says 172.21.15.83:48774) 2024-03-06T07:25:09.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.240+0000 7fe747fff640 1 -- 172.21.15.83:0/1154560049 learned_addr learned my addr 172.21.15.83:0/1154560049 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:25:09.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.240+0000 7fe74d527640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe7480d2710 0x7fe7480b4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35452/0 (socket says 172.21.15.83:35452) 2024-03-06T07:25:09.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.240+0000 7fe74d527640 1 -- 172.21.15.83:0/1154560049 >> v1:172.21.15.183:6789/0 conn(0x7fe74815dfe0 legacy=0x7fe7480cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:09.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.240+0000 7fe74d527640 1 -- 172.21.15.83:0/1154560049 >> v1:172.21.15.83:6789/0 conn(0x7fe7480d22b0 legacy=0x7fe7480b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.240+0000 7fe74d527640 1 -- 172.21.15.83:0/1154560049 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7480b39f0 con 0x7fe7480d2710 2024-03-06T07:25:09.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.240+0000 7fe7477fe640 1 -- 172.21.15.83:0/1154560049 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3983039731 0 0) 0x7fe7480b3430 con 0x7fe7480d22b0 2024-03-06T07:25:09.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.241+0000 7fe74d527640 1 --2- 172.21.15.83:0/1154560049 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7480d2710 0x7fe7480b4900 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fe740009dc0 tx=0x7fe74002be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b11ba88a59405a65 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.241+0000 7fe7477fe640 1 -- 172.21.15.83:0/1154560049 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe740039070 con 0x7fe7480d2710 2024-03-06T07:25:09.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.241+0000 7fe7477fe640 1 -- 172.21.15.83:0/1154560049 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe74002cc40 con 0x7fe7480d2710 2024-03-06T07:25:09.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.241+0000 7fe7477fe640 1 -- 172.21.15.83:0/1154560049 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe740034440 con 0x7fe7480d2710 2024-03-06T07:25:09.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.241+0000 7fe74ef8a640 1 -- 172.21.15.83:0/1154560049 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7480d2710 msgr2=0x7fe7480b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.241+0000 7fe74ef8a640 1 --2- 172.21.15.83:0/1154560049 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7480d2710 0x7fe7480b4900 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fe740009dc0 tx=0x7fe74002be90 comp rx=0 tx=0).stop 2024-03-06T07:25:09.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.241+0000 7fe74ef8a640 1 -- 172.21.15.83:0/1154560049 shutdown_connections 2024-03-06T07:25:09.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.241+0000 7fe74ef8a640 1 --2- 172.21.15.83:0/1154560049 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7480d2710 0x7fe7480b4900 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.241+0000 7fe74ef8a640 1 -- 172.21.15.83:0/1154560049 >> 172.21.15.83:0/1154560049 conn(0x7fe7480d3cd0 msgr2=0x7fe7480d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:09.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.241+0000 7fe74ef8a640 1 -- 172.21.15.83:0/1154560049 shutdown_connections 2024-03-06T07:25:09.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.241+0000 7fe74ef8a640 1 -- 172.21.15.83:0/1154560049 wait complete. 2024-03-06T07:25:09.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.242+0000 7fe74ef8a640 1 Processor -- start 2024-03-06T07:25:09.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.242+0000 7fe74ef8a640 1 -- start start 2024-03-06T07:25:09.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.243+0000 7fe74ef8a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe7480d22b0 0x7fe7480b8db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.243+0000 7fe74ef8a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7480d2710 0x7fe7481f6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.243+0000 7fe74ef8a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe74815dfe0 0x7fe748159510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.243+0000 7fe74ef8a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7480cfa00 con 0x7fe7480d2710 2024-03-06T07:25:09.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.243+0000 7fe74ef8a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7480b3d60 con 0x7fe74815dfe0 2024-03-06T07:25:09.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.243+0000 7fe74ef8a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7480d1c70 con 0x7fe7480d22b0 2024-03-06T07:25:09.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.243+0000 7fe747fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7480d2710 0x7fe7481f6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.243+0000 7fe747fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7480d2710 0x7fe7481f6f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35462/0 (socket says 172.21.15.83:35462) 2024-03-06T07:25:09.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.243+0000 7fe747fff640 1 -- 172.21.15.83:0/2758499485 learned_addr learned my addr 172.21.15.83:0/2758499485 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:09.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.244+0000 7fe74d527640 1 --2- 172.21.15.83:0/2758499485 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe74815dfe0 0x7fe748159510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.244+0000 7fe74cd26640 1 --2- 172.21.15.83:0/2758499485 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe7480d22b0 0x7fe7480b8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.244+0000 7fe747fff640 1 -- 172.21.15.83:0/2758499485 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe7480d22b0 msgr2=0x7fe7480b8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.244+0000 7fe747fff640 1 --2- 172.21.15.83:0/2758499485 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe7480d22b0 0x7fe7480b8db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.244+0000 7fe747fff640 1 -- 172.21.15.83:0/2758499485 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe74815dfe0 msgr2=0x7fe748159510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.244+0000 7fe747fff640 1 --2- 172.21.15.83:0/2758499485 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe74815dfe0 0x7fe748159510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.244+0000 7fe747fff640 1 -- 172.21.15.83:0/2758499485 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7480b68d0 con 0x7fe7480d2710 2024-03-06T07:25:09.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.244+0000 7fe74cd26640 1 --2- 172.21.15.83:0/2758499485 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe7480d22b0 0x7fe7480b8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:09.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.244+0000 7fe747fff640 1 --2- 172.21.15.83:0/2758499485 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7480d2710 0x7fe7481f6f70 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fe73400c4d0 tx=0x7fe73400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.244+0000 7fe7457fa640 1 -- 172.21.15.83:0/2758499485 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe734018070 con 0x7fe7480d2710 2024-03-06T07:25:09.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.244+0000 7fe7457fa640 1 -- 172.21.15.83:0/2758499485 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe734007500 con 0x7fe7480d2710 2024-03-06T07:25:09.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.244+0000 7fe74ef8a640 1 -- 172.21.15.83:0/2758499485 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7481f7a90 con 0x7fe7480d2710 2024-03-06T07:25:09.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.244+0000 7fe7457fa640 1 -- 172.21.15.83:0/2758499485 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe734004a20 con 0x7fe7480d2710 2024-03-06T07:25:09.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.244+0000 7fe74ef8a640 1 -- 172.21.15.83:0/2758499485 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7480b6e50 con 0x7fe7480d2710 2024-03-06T07:25:09.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.245+0000 7fe74ef8a640 1 -- 172.21.15.83:0/2758499485 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7480b7050 con 0x7fe7480d2710 2024-03-06T07:25:09.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.245+0000 7fe7457fa640 1 -- 172.21.15.83:0/2758499485 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe73401b070 con 0x7fe7480d2710 2024-03-06T07:25:09.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.246+0000 7fe7457fa640 1 --2- 172.21.15.83:0/2758499485 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe72c06b4d0 0x7fe72c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.246+0000 7fe7457fa640 1 -- 172.21.15.83:0/2758499485 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe734089110 con 0x7fe7480d2710 2024-03-06T07:25:09.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.246+0000 7fe74cd26640 1 --2- 172.21.15.83:0/2758499485 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe72c06b4d0 0x7fe72c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.246+0000 7fe74cd26640 1 --2- 172.21.15.83:0/2758499485 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe72c06b4d0 0x7fe72c06d990 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fe73c006fd0 tx=0x7fe73c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.250+0000 7fe7457fa640 1 -- 172.21.15.83:0/2758499485 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe7480b7050 con 0x7fe7480d2710 2024-03-06T07:25:09.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.486+0000 7fe74ef8a640 1 -- 172.21.15.83:0/2758499485 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fe7480cec80 con 0x7fe7480d2710 2024-03-06T07:25:09.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.486+0000 7fe7457fa640 1 -- 172.21.15.83:0/2758499485 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fe7481f7a90 con 0x7fe7480d2710 2024-03-06T07:25:09.488 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:25:09.488 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:25:09.489 INFO:teuthology.orchestra.run.smithi083.stdout: "auth_dump": [ 2024-03-06T07:25:09.489 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:25:09.489 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mds.a", 2024-03-06T07:25:09.489 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlVHaUGhAAVRpLsHYTUkaYDmceq5Rlgw==", 2024-03-06T07:25:09.489 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:25:09.489 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:25:09.489 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow *", 2024-03-06T07:25:09.489 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow *", 2024-03-06T07:25:09.489 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:25:09.489 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:25:09.489 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:25:09.489 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:25:09.489 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mds.b", 2024-03-06T07:25:09.489 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlFlrrJhAAQbY6xU910eYRg1zwgpj9vg==", 2024-03-06T07:25:09.490 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:25:09.490 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:25:09.490 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow *", 2024-03-06T07:25:09.490 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow *", 2024-03-06T07:25:09.490 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:25:09.490 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:25:09.490 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:25:09.490 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:25:09.490 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mds.c", 2024-03-06T07:25:09.490 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlX0jxIRAAxYNi39qxIFXazlL2qNzLCg==", 2024-03-06T07:25:09.490 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:25:09.490 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:25:09.490 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow *", 2024-03-06T07:25:09.490 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow *", 2024-03-06T07:25:09.491 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:25:09.491 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:25:09.491 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:25:09.491 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:25:09.491 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mds.d", 2024-03-06T07:25:09.491 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehl3utCLhAAzEjASLnYpPYrJsWOt3WwpQ==", 2024-03-06T07:25:09.491 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:25:09.491 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:25:09.491 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow *", 2024-03-06T07:25:09.491 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow *", 2024-03-06T07:25:09.491 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:25:09.491 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:25:09.491 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:25:09.491 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:25:09.492 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.0", 2024-03-06T07:25:09.492 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBgGehlkezBExAAZOLMH2ua3CTWAL7H6LGeZw==", 2024-03-06T07:25:09.492 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:25:09.492 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:09.492 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:25:09.492 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:25:09.492 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:25:09.492 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:25:09.492 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:25:09.492 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.1", 2024-03-06T07:25:09.492 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBjGehlC04qEhAAqirTFR6y3NOGw6N+iz3/pg==", 2024-03-06T07:25:09.492 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:25:09.492 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:09.492 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:25:09.493 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:25:09.493 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:25:09.493 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:25:09.493 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:25:09.493 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.2", 2024-03-06T07:25:09.493 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBmGehlfuFWEBAAj/vjZXrv1wVNSJ1R74UmnQ==", 2024-03-06T07:25:09.493 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:25:09.493 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:09.493 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:25:09.493 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:25:09.493 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:25:09.493 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:25:09.493 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:25:09.493 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.3", 2024-03-06T07:25:09.494 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBpGehl5+ItDhAA0YNc83MhKcgb3JgRuP87aw==", 2024-03-06T07:25:09.494 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:25:09.494 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:09.494 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:25:09.494 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:25:09.494 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:25:09.494 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:25:09.494 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:25:09.494 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.4", 2024-03-06T07:25:09.494 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBvGehlYa/TFBAAvGcizpRQ21pvgpEGgSET0w==", 2024-03-06T07:25:09.494 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:25:09.494 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:09.494 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:25:09.494 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:25:09.494 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:25:09.495 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:25:09.495 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:25:09.495 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.5", 2024-03-06T07:25:09.495 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQByGehlE80TExAAFcWDgnLtW8OKlyu4B0jnmg==", 2024-03-06T07:25:09.495 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:25:09.495 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:09.495 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:25:09.495 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:25:09.495 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:25:09.495 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:25:09.495 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:25:09.495 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.6", 2024-03-06T07:25:09.495 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQB1Gehla/LbERAAYfAjTKL11VMVnGUGV5GVcg==", 2024-03-06T07:25:09.495 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:25:09.495 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:09.496 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:25:09.496 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:25:09.496 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:25:09.496 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:25:09.496 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:25:09.496 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.7", 2024-03-06T07:25:09.496 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQB4GehlpSJkEBAA17tUgMf94jfqvXlD51P8DQ==", 2024-03-06T07:25:09.496 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:25:09.496 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:09.496 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:25:09.496 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:25:09.496 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:25:09.496 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:25:09.497 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:25:09.497 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.0", 2024-03-06T07:25:09.497 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlRwCIMhAA1DdbdQZPplu7LQOTK3BmSg==", 2024-03-06T07:25:09.497 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:25:09.497 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:25:09.497 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow r", 2024-03-06T07:25:09.497 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:25:09.497 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:25:09.497 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:25:09.497 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:25:09.497 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.1", 2024-03-06T07:25:09.497 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBdGehl77spABAAsiEf+tXos3MwFrXToEQ8dQ==", 2024-03-06T07:25:09.497 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:25:09.497 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:25:09.497 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow r", 2024-03-06T07:25:09.498 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow rw", 2024-03-06T07:25:09.498 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow rwx" 2024-03-06T07:25:09.498 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:25:09.498 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:25:09.498 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:25:09.498 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.2", 2024-03-06T07:25:09.498 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehltY5SOBAAI8rcIEVqMZaWQGgNaYKJTA==", 2024-03-06T07:25:09.498 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:25:09.498 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:25:09.498 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow r", 2024-03-06T07:25:09.498 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow rw", 2024-03-06T07:25:09.498 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow rwx" 2024-03-06T07:25:09.498 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:25:09.498 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:25:09.498 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:25:09.499 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.3", 2024-03-06T07:25:09.499 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBdGehl9ZX1BRAAA7fKnRyaQ+zqHoSGs4UvDw==", 2024-03-06T07:25:09.499 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:25:09.499 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:25:09.499 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow r", 2024-03-06T07:25:09.499 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow rw", 2024-03-06T07:25:09.499 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow rwx" 2024-03-06T07:25:09.499 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:25:09.499 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:25:09.499 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:25:09.499 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.admin", 2024-03-06T07:25:09.499 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBbGehl4gqRORAAobqSWoeyHYA/lXkp6/EcIQ==", 2024-03-06T07:25:09.499 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:25:09.499 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow *", 2024-03-06T07:25:09.499 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow *", 2024-03-06T07:25:09.500 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow *", 2024-03-06T07:25:09.500 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:25:09.500 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:25:09.500 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:25:09.500 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:25:09.500 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mgr.x", 2024-03-06T07:25:09.500 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlfikrFxAAoZkTUtpHlBcUAMpwWABRxg==", 2024-03-06T07:25:09.500 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:25:09.500 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow *", 2024-03-06T07:25:09.500 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile mgr", 2024-03-06T07:25:09.500 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:25:09.500 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:25:09.500 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:25:09.500 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:25:09.501 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mgr.y", 2024-03-06T07:25:09.501 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlvdcdExAAySXlEfJqLnpMySiiYGFkMQ==", 2024-03-06T07:25:09.501 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:25:09.501 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow *", 2024-03-06T07:25:09.501 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile mgr", 2024-03-06T07:25:09.501 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:25:09.501 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:25:09.501 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:25:09.501 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-03-06T07:25:09.501 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:25:09.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.488+0000 7fe74ef8a640 1 -- 172.21.15.83:0/2758499485 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe72c06b4d0 msgr2=0x7fe72c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.488+0000 7fe74ef8a640 1 --2- 172.21.15.83:0/2758499485 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe72c06b4d0 0x7fe72c06d990 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fe73c006fd0 tx=0x7fe73c008040 comp rx=0 tx=0).stop 2024-03-06T07:25:09.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.488+0000 7fe74ef8a640 1 -- 172.21.15.83:0/2758499485 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7480d2710 msgr2=0x7fe7481f6f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.488+0000 7fe74ef8a640 1 --2- 172.21.15.83:0/2758499485 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7480d2710 0x7fe7481f6f70 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fe73400c4d0 tx=0x7fe73400c990 comp rx=0 tx=0).stop 2024-03-06T07:25:09.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.488+0000 7fe74ef8a640 1 -- 172.21.15.83:0/2758499485 shutdown_connections 2024-03-06T07:25:09.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.488+0000 7fe74ef8a640 1 --2- 172.21.15.83:0/2758499485 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe72c06b4d0 0x7fe72c06d990 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.488+0000 7fe74ef8a640 1 --2- 172.21.15.83:0/2758499485 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe74815dfe0 0x7fe748159510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.488+0000 7fe74ef8a640 1 --2- 172.21.15.83:0/2758499485 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7480d2710 0x7fe7481f6f70 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.488+0000 7fe74ef8a640 1 --2- 172.21.15.83:0/2758499485 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe7480d22b0 0x7fe7480b8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.489+0000 7fe74ef8a640 1 -- 172.21.15.83:0/2758499485 >> 172.21.15.83:0/2758499485 conn(0x7fe7480d3cd0 msgr2=0x7fe7480cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:09.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.489+0000 7fe74ef8a640 1 -- 172.21.15.83:0/2758499485 shutdown_connections 2024-03-06T07:25:09.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.489+0000 7fe74ef8a640 1 -- 172.21.15.83:0/2758499485 wait complete. 2024-03-06T07:25:09.504 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:09.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.602+0000 7f9fc5ef4640 1 Processor -- start 2024-03-06T07:25:09.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.602+0000 7f9fc5ef4640 1 -- start start 2024-03-06T07:25:09.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.603+0000 7f9fc5ef4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9fc00d22b0 0x7f9fc00d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.603+0000 7f9fc5ef4640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f9fc00b41f0 0x7f9fc00b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.603+0000 7f9fc5ef4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc00b3430 con 0x7f9fc015dfe0 2024-03-06T07:25:09.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.603+0000 7f9fc5ef4640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fc00d2c50 con 0x7f9fc00d22b0 2024-03-06T07:25:09.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.603+0000 7f9fc5ef4640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fc00b31e0 con 0x7f9fc00b41f0 2024-03-06T07:25:09.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.603+0000 7f9fb6ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9fc00d22b0 0x7f9fc00d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.603+0000 7f9fb6ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9fc00d22b0 0x7f9fc00d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35474/0 (socket says 172.21.15.83:35474) 2024-03-06T07:25:09.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.603+0000 7f9fb6ffd640 1 -- 172.21.15.83:0/2527748979 learned_addr learned my addr 172.21.15.83:0/2527748979 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:09.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.604+0000 7f9fbffff640 1 --2- 172.21.15.83:0/2527748979 >> v2:172.21.15.183:3300/0 conn(0x7f9fc00b41f0 0x7f9fc00b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.604+0000 7f9fb6ffd640 1 -- 172.21.15.83:0/2527748979 >> v2:172.21.15.183:3300/0 conn(0x7f9fc00b41f0 msgr2=0x7f9fc00b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:09.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.604+0000 7f9fb6ffd640 1 --2- 172.21.15.83:0/2527748979 >> v2:172.21.15.183:3300/0 conn(0x7f9fc00b41f0 0x7f9fc00b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.604+0000 7f9fb6ffd640 1 -- 172.21.15.83:0/2527748979 >> v1:172.21.15.183:6789/0 conn(0x7f9fc015dfe0 legacy=0x7f9fc00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:09.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.604+0000 7f9fb6ffd640 1 -- 172.21.15.83:0/2527748979 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fc00b3710 con 0x7f9fc00d22b0 2024-03-06T07:25:09.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.604+0000 7f9fb6ffd640 1 --2- 172.21.15.83:0/2527748979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9fc00d22b0 0x7f9fc00d2710 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f9fa8009870 tx=0x7f9fa802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da8f01275af0fc98 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.604+0000 7f9fbeffd640 1 -- 172.21.15.83:0/2527748979 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa8035020 con 0x7f9fc00d22b0 2024-03-06T07:25:09.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.605+0000 7f9fbeffd640 1 -- 172.21.15.83:0/2527748979 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fa8031870 con 0x7f9fc00d22b0 2024-03-06T07:25:09.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.605+0000 7f9fbeffd640 1 -- 172.21.15.83:0/2527748979 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa8031c40 con 0x7f9fc00d22b0 2024-03-06T07:25:09.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.605+0000 7f9fc5ef4640 1 -- 172.21.15.83:0/2527748979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9fc00d22b0 msgr2=0x7f9fc00d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.605+0000 7f9fc5ef4640 1 --2- 172.21.15.83:0/2527748979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9fc00d22b0 0x7f9fc00d2710 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f9fa8009870 tx=0x7f9fa802b910 comp rx=0 tx=0).stop 2024-03-06T07:25:09.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.605+0000 7f9fc5ef4640 1 -- 172.21.15.83:0/2527748979 shutdown_connections 2024-03-06T07:25:09.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.605+0000 7f9fc5ef4640 1 --2- 172.21.15.83:0/2527748979 >> v2:172.21.15.183:3300/0 conn(0x7f9fc00b41f0 0x7f9fc00b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.605+0000 7f9fc5ef4640 1 --2- 172.21.15.83:0/2527748979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9fc00d22b0 0x7f9fc00d2710 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.605+0000 7f9fc5ef4640 1 -- 172.21.15.83:0/2527748979 >> 172.21.15.83:0/2527748979 conn(0x7f9fc00d3cd0 msgr2=0x7f9fc00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:09.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.605+0000 7f9fc5ef4640 1 -- 172.21.15.83:0/2527748979 shutdown_connections 2024-03-06T07:25:09.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.605+0000 7f9fc5ef4640 1 -- 172.21.15.83:0/2527748979 wait complete. 2024-03-06T07:25:09.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.607+0000 7f9fc5ef4640 1 Processor -- start 2024-03-06T07:25:09.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.607+0000 7f9fc5ef4640 1 -- start start 2024-03-06T07:25:09.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.608+0000 7f9fc5ef4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9fc00b41f0 0x7f9fc0156f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.608+0000 7f9fc5ef4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9fc00d22b0 0x7f9fc0156870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.608+0000 7f9fc5ef4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9fc015dfe0 0x7f9fc0160500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.608+0000 7f9fc5ef4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc00cfa00 con 0x7f9fc015dfe0 2024-03-06T07:25:09.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.608+0000 7f9fc5ef4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc00b3d60 con 0x7f9fc00b41f0 2024-03-06T07:25:09.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.608+0000 7f9fc5ef4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc00d1c70 con 0x7f9fc00d22b0 2024-03-06T07:25:09.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.608+0000 7f9fbffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9fc015dfe0 0x7f9fc0160500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.608+0000 7f9fbffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9fc015dfe0 0x7f9fc0160500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35486/0 (socket says 172.21.15.83:35486) 2024-03-06T07:25:09.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.608+0000 7f9fbffff640 1 -- 172.21.15.83:0/941983649 learned_addr learned my addr 172.21.15.83:0/941983649 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:09.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.608+0000 7f9fb6ffd640 1 --2- 172.21.15.83:0/941983649 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9fc00d22b0 0x7f9fc0156870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.608+0000 7f9fbf7fe640 1 --2- 172.21.15.83:0/941983649 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9fc00b41f0 0x7f9fc0156f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.608+0000 7f9fbffff640 1 -- 172.21.15.83:0/941983649 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9fc00d22b0 msgr2=0x7f9fc0156870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.608+0000 7f9fbffff640 1 --2- 172.21.15.83:0/941983649 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9fc00d22b0 0x7f9fc0156870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.608+0000 7f9fbffff640 1 -- 172.21.15.83:0/941983649 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9fc00b41f0 msgr2=0x7f9fc0156f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.608+0000 7f9fbffff640 1 --2- 172.21.15.83:0/941983649 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9fc00b41f0 0x7f9fc0156f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.608+0000 7f9fbffff640 1 -- 172.21.15.83:0/941983649 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fc00cdb60 con 0x7f9fc015dfe0 2024-03-06T07:25:09.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.608+0000 7f9fbffff640 1 --2- 172.21.15.83:0/941983649 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9fc015dfe0 0x7f9fc0160500 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f9fb800cd70 tx=0x7f9fb800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.608+0000 7f9fbcff9640 1 -- 172.21.15.83:0/941983649 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb8003b90 con 0x7f9fc015dfe0 2024-03-06T07:25:09.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.608+0000 7f9fbcff9640 1 -- 172.21.15.83:0/941983649 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fb8003d30 con 0x7f9fc015dfe0 2024-03-06T07:25:09.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.608+0000 7f9fc5ef4640 1 -- 172.21.15.83:0/941983649 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fc00cb8e0 con 0x7f9fc015dfe0 2024-03-06T07:25:09.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.608+0000 7f9fbcff9640 1 -- 172.21.15.83:0/941983649 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb8004240 con 0x7f9fc015dfe0 2024-03-06T07:25:09.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.608+0000 7f9fc5ef4640 1 -- 172.21.15.83:0/941983649 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fc00cbea0 con 0x7f9fc015dfe0 2024-03-06T07:25:09.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.610+0000 7f9fc5ef4640 1 -- 172.21.15.83:0/941983649 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fc004f330 con 0x7f9fc015dfe0 2024-03-06T07:25:09.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.610+0000 7f9fbcff9640 1 -- 172.21.15.83:0/941983649 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9fb80043e0 con 0x7f9fc015dfe0 2024-03-06T07:25:09.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.610+0000 7f9fbcff9640 1 --2- 172.21.15.83:0/941983649 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9fa006b4d0 0x7f9fa006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.610+0000 7f9fbcff9640 1 -- 172.21.15.83:0/941983649 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9fb808a9a0 con 0x7f9fc015dfe0 2024-03-06T07:25:09.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.610+0000 7f9fbf7fe640 1 --2- 172.21.15.83:0/941983649 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9fa006b4d0 0x7f9fa006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.611+0000 7f9fbf7fe640 1 --2- 172.21.15.83:0/941983649 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9fa006b4d0 0x7f9fa006d990 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f9fb0006fd0 tx=0x7f9fb0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.614+0000 7f9fbcff9640 1 -- 172.21.15.83:0/941983649 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9fb801d340 con 0x7f9fc015dfe0 2024-03-06T07:25:09.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.788+0000 7f9fc5ef4640 1 -- 172.21.15.83:0/941983649 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9fc006a1b0 con 0x7f9fc015dfe0 2024-03-06T07:25:09.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.789+0000 7f9fbcff9640 1 -- 172.21.15.83:0/941983649 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) v1 ==== 74+0+12903 (secure 0 0 0) 0x7f9fb8059c00 con 0x7f9fc015dfe0 2024-03-06T07:25:09.791 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:25:09.791 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":65,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:25:09.114472+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:09.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.791+0000 7f9fc5ef4640 1 -- 172.21.15.83:0/941983649 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9fa006b4d0 msgr2=0x7f9fa006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.791+0000 7f9fc5ef4640 1 --2- 172.21.15.83:0/941983649 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9fa006b4d0 0x7f9fa006d990 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f9fb0006fd0 tx=0x7f9fb0008040 comp rx=0 tx=0).stop 2024-03-06T07:25:09.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.791+0000 7f9fc5ef4640 1 -- 172.21.15.83:0/941983649 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9fc015dfe0 msgr2=0x7f9fc0160500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.791+0000 7f9fc5ef4640 1 --2- 172.21.15.83:0/941983649 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9fc015dfe0 0x7f9fc0160500 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f9fb800cd70 tx=0x7f9fb800a660 comp rx=0 tx=0).stop 2024-03-06T07:25:09.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.791+0000 7f9fc5ef4640 1 -- 172.21.15.83:0/941983649 shutdown_connections 2024-03-06T07:25:09.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.791+0000 7f9fc5ef4640 1 --2- 172.21.15.83:0/941983649 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9fa006b4d0 0x7f9fa006d990 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.791+0000 7f9fc5ef4640 1 --2- 172.21.15.83:0/941983649 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9fc015dfe0 0x7f9fc0160500 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.791+0000 7f9fc5ef4640 1 --2- 172.21.15.83:0/941983649 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9fc00d22b0 0x7f9fc0156870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.791+0000 7f9fc5ef4640 1 --2- 172.21.15.83:0/941983649 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9fc00b41f0 0x7f9fc0156f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.791+0000 7f9fc5ef4640 1 -- 172.21.15.83:0/941983649 >> 172.21.15.83:0/941983649 conn(0x7f9fc00d3cd0 msgr2=0x7f9fc00cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:09.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.791+0000 7f9fc5ef4640 1 -- 172.21.15.83:0/941983649 shutdown_connections 2024-03-06T07:25:09.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.791+0000 7f9fc5ef4640 1 -- 172.21.15.83:0/941983649 wait complete. 2024-03-06T07:25:09.804 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:25:09.804 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:25:09.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.904+0000 7f6e6f14b640 1 Processor -- start 2024-03-06T07:25:09.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.904+0000 7f6e6f14b640 1 -- start start 2024-03-06T07:25:09.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.905+0000 7f6e6f14b640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f6e6806efd0 0x7f6e6806f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.905+0000 7f6e6f14b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6e680cc5c0 0x7f6e680cca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.905+0000 7f6e6f14b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e6806b6c0 con 0x7f6e68145590 2024-03-06T07:25:09.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.905+0000 7f6e6f14b640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e6806f970 con 0x7f6e680cc5c0 2024-03-06T07:25:09.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.905+0000 7f6e6f14b640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e68145970 con 0x7f6e6806efd0 2024-03-06T07:25:09.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.905+0000 7f6e6cee7640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f6e68145590 0x7f6e6806afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48788/0 (socket says 172.21.15.83:48788) 2024-03-06T07:25:09.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.905+0000 7f6e6cee7640 1 -- 172.21.15.83:0/2801683122 learned_addr learned my addr 172.21.15.83:0/2801683122 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:25:09.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.905+0000 7f6e6d6e8640 1 --2- 172.21.15.83:0/2801683122 >> v2:172.21.15.83:3300/0 conn(0x7f6e680cc5c0 0x7f6e680cca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.906+0000 7f6e67fff640 1 --2- 172.21.15.83:0/2801683122 >> v2:172.21.15.183:3300/0 conn(0x7f6e6806efd0 0x7f6e6806f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.906+0000 7f6e6d6e8640 1 -- 172.21.15.83:0/2801683122 >> v2:172.21.15.183:3300/0 conn(0x7f6e6806efd0 msgr2=0x7f6e6806f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:09.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.906+0000 7f6e6d6e8640 1 --2- 172.21.15.83:0/2801683122 >> v2:172.21.15.183:3300/0 conn(0x7f6e6806efd0 0x7f6e6806f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.906+0000 7f6e6d6e8640 1 -- 172.21.15.83:0/2801683122 >> v1:172.21.15.83:6789/0 conn(0x7f6e68145590 legacy=0x7f6e6806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.906+0000 7f6e6d6e8640 1 -- 172.21.15.83:0/2801683122 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e6806b970 con 0x7f6e680cc5c0 2024-03-06T07:25:09.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.906+0000 7f6e67fff640 1 --2- 172.21.15.83:0/2801683122 >> v2:172.21.15.183:3300/0 conn(0x7f6e6806efd0 0x7f6e6806f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:09.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.906+0000 7f6e6d6e8640 1 --2- 172.21.15.83:0/2801683122 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6e680cc5c0 0x7f6e680cca50 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f6e680d08d0 tx=0x7f6e6002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8383dd59fe655c76 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.906+0000 7f6e677fe640 1 -- 172.21.15.83:0/2801683122 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e60039070 con 0x7f6e680cc5c0 2024-03-06T07:25:09.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.906+0000 7f6e677fe640 1 -- 172.21.15.83:0/2801683122 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e6002cc80 con 0x7f6e680cc5c0 2024-03-06T07:25:09.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.906+0000 7f6e677fe640 1 -- 172.21.15.83:0/2801683122 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e60034660 con 0x7f6e680cc5c0 2024-03-06T07:25:09.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.907+0000 7f6e6f14b640 1 -- 172.21.15.83:0/2801683122 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6e680cc5c0 msgr2=0x7f6e680cca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.907+0000 7f6e6f14b640 1 --2- 172.21.15.83:0/2801683122 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6e680cc5c0 0x7f6e680cca50 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f6e680d08d0 tx=0x7f6e6002bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.907+0000 7f6e6f14b640 1 -- 172.21.15.83:0/2801683122 shutdown_connections 2024-03-06T07:25:09.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.907+0000 7f6e6f14b640 1 --2- 172.21.15.83:0/2801683122 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6e680cc5c0 0x7f6e680cca50 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.907+0000 7f6e6f14b640 1 --2- 172.21.15.83:0/2801683122 >> v2:172.21.15.183:3300/0 conn(0x7f6e6806efd0 0x7f6e6806f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.907+0000 7f6e6f14b640 1 -- 172.21.15.83:0/2801683122 >> 172.21.15.83:0/2801683122 conn(0x7f6e680b2660 msgr2=0x7f6e680b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:09.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.907+0000 7f6e6f14b640 1 -- 172.21.15.83:0/2801683122 shutdown_connections 2024-03-06T07:25:09.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.907+0000 7f6e6f14b640 1 -- 172.21.15.83:0/2801683122 wait complete. 2024-03-06T07:25:09.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.908+0000 7f6e6f14b640 1 Processor -- start 2024-03-06T07:25:09.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.908+0000 7f6e6f14b640 1 -- start start 2024-03-06T07:25:09.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.909+0000 7f6e6f14b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6e6806efd0 0x7f6e681f1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.909+0000 7f6e6f14b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6e680cc5c0 0x7f6e6815a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.909+0000 7f6e6f14b640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6e68145590 0x7f6e6815e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.909+0000 7f6e6f14b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6815fac0 con 0x7f6e6806efd0 2024-03-06T07:25:09.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.909+0000 7f6e6f14b640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e68160110 con 0x7f6e68145590 2024-03-06T07:25:09.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.909+0000 7f6e6f14b640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6815fda0 con 0x7f6e680cc5c0 2024-03-06T07:25:09.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.909+0000 7f6e6cee7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6e6806efd0 0x7f6e681f1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.909+0000 7f6e6cee7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6e6806efd0 0x7f6e681f1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35500/0 (socket says 172.21.15.83:35500) 2024-03-06T07:25:09.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.909+0000 7f6e6cee7640 1 -- 172.21.15.83:0/2754944081 learned_addr learned my addr 172.21.15.83:0/2754944081 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:09.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.909+0000 7f6e67fff640 1 --2- 172.21.15.83:0/2754944081 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6e680cc5c0 0x7f6e6815a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.909+0000 7f6e6d6e8640 1 --2- 172.21.15.83:0/2754944081 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6e68145590 0x7f6e6815e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.909+0000 7f6e6cee7640 1 -- 172.21.15.83:0/2754944081 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6e680cc5c0 msgr2=0x7f6e6815a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.909+0000 7f6e6cee7640 1 --2- 172.21.15.83:0/2754944081 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6e680cc5c0 0x7f6e6815a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.909+0000 7f6e6cee7640 1 -- 172.21.15.83:0/2754944081 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6e68145590 msgr2=0x7f6e6815e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.909+0000 7f6e6cee7640 1 --2- 172.21.15.83:0/2754944081 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6e68145590 0x7f6e6815e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.909+0000 7f6e6cee7640 1 -- 172.21.15.83:0/2754944081 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e681e5560 con 0x7f6e6806efd0 2024-03-06T07:25:09.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.909+0000 7f6e67fff640 1 --2- 172.21.15.83:0/2754944081 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6e680cc5c0 0x7f6e6815a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:09.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.909+0000 7f6e6d6e8640 1 --2- 172.21.15.83:0/2754944081 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6e68145590 0x7f6e6815e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:09.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.910+0000 7f6e6cee7640 1 --2- 172.21.15.83:0/2754944081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6e6806efd0 0x7f6e681f1940 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f6e5800c3c0 tx=0x7f6e5800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.910+0000 7f6e657fa640 1 -- 172.21.15.83:0/2754944081 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e58017070 con 0x7f6e6806efd0 2024-03-06T07:25:09.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.910+0000 7f6e657fa640 1 -- 172.21.15.83:0/2754944081 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e58005ce0 con 0x7f6e6806efd0 2024-03-06T07:25:09.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.910+0000 7f6e6f14b640 1 -- 172.21.15.83:0/2754944081 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e681e5840 con 0x7f6e6806efd0 2024-03-06T07:25:09.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.910+0000 7f6e657fa640 1 -- 172.21.15.83:0/2754944081 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e580072f0 con 0x7f6e6806efd0 2024-03-06T07:25:09.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.910+0000 7f6e6f14b640 1 -- 172.21.15.83:0/2754944081 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e681e5e00 con 0x7f6e6806efd0 2024-03-06T07:25:09.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.911+0000 7f6e6f14b640 1 -- 172.21.15.83:0/2754944081 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e681e6230 con 0x7f6e6806efd0 2024-03-06T07:25:09.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.911+0000 7f6e657fa640 1 -- 172.21.15.83:0/2754944081 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6e5801a070 con 0x7f6e6806efd0 2024-03-06T07:25:09.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.912+0000 7f6e657fa640 1 --2- 172.21.15.83:0/2754944081 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6e5406b4d0 0x7f6e5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.912+0000 7f6e657fa640 1 -- 172.21.15.83:0/2754944081 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6e58089290 con 0x7f6e6806efd0 2024-03-06T07:25:09.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.912+0000 7f6e67fff640 1 --2- 172.21.15.83:0/2754944081 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6e5406b4d0 0x7f6e5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.912+0000 7f6e67fff640 1 --2- 172.21.15.83:0/2754944081 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6e5406b4d0 0x7f6e5406d990 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c006fd0 tx=0x7f6e5c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:09.916+0000 7f6e657fa640 1 -- 172.21.15.83:0/2754944081 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6e58058460 con 0x7f6e6806efd0 2024-03-06T07:25:10.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.084+0000 7f6e6f14b640 1 -- 172.21.15.83:0/2754944081 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6e680733b0 con 0x7f6e6806efd0 2024-03-06T07:25:10.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.085+0000 7f6e657fa640 1 -- 172.21.15.83:0/2754944081 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v65) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6e58057d60 con 0x7f6e6806efd0 2024-03-06T07:25:10.087 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:25:10.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.086+0000 7f6e6f14b640 1 -- 172.21.15.83:0/2754944081 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6e5406b4d0 msgr2=0x7f6e5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:10.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.086+0000 7f6e6f14b640 1 --2- 172.21.15.83:0/2754944081 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6e5406b4d0 0x7f6e5406d990 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c006fd0 tx=0x7f6e5c008040 comp rx=0 tx=0).stop 2024-03-06T07:25:10.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.086+0000 7f6e6f14b640 1 -- 172.21.15.83:0/2754944081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6e6806efd0 msgr2=0x7f6e681f1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:10.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.086+0000 7f6e6f14b640 1 --2- 172.21.15.83:0/2754944081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6e6806efd0 0x7f6e681f1940 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f6e5800c3c0 tx=0x7f6e5800c880 comp rx=0 tx=0).stop 2024-03-06T07:25:10.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.087+0000 7f6e6f14b640 1 -- 172.21.15.83:0/2754944081 shutdown_connections 2024-03-06T07:25:10.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.087+0000 7f6e6f14b640 1 --2- 172.21.15.83:0/2754944081 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6e5406b4d0 0x7f6e5406d990 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.087+0000 7f6e6f14b640 1 --2- 172.21.15.83:0/2754944081 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6e68145590 0x7f6e6815e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.087+0000 7f6e6f14b640 1 --2- 172.21.15.83:0/2754944081 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6e680cc5c0 0x7f6e6815a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.087+0000 7f6e6f14b640 1 --2- 172.21.15.83:0/2754944081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6e6806efd0 0x7f6e681f1940 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.087+0000 7f6e6f14b640 1 -- 172.21.15.83:0/2754944081 >> 172.21.15.83:0/2754944081 conn(0x7f6e680b2660 msgr2=0x7f6e6806afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:10.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.087+0000 7f6e6f14b640 1 -- 172.21.15.83:0/2754944081 shutdown_connections 2024-03-06T07:25:10.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.087+0000 7f6e6f14b640 1 -- 172.21.15.83:0/2754944081 wait complete. 2024-03-06T07:25:10.099 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:25:10.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.197+0000 7f67ab3e9640 1 Processor -- start 2024-03-06T07:25:10.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.197+0000 7f67ab3e9640 1 -- start start 2024-03-06T07:25:10.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.198+0000 7f67ab3e9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f67a415dfe0 0x7f67a40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.198+0000 7f67ab3e9640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f67a40cf830 0x7f67a40d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.198+0000 7f67ab3e9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67a40b3430 con 0x7f67a40d27f0 2024-03-06T07:25:10.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.198+0000 7f67ab3e9640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67a40b31e0 con 0x7f67a415dfe0 2024-03-06T07:25:10.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.198+0000 7f67ab3e9640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67a40b3d60 con 0x7f67a40cf830 2024-03-06T07:25:10.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.198+0000 7f67a9185640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f67a415dfe0 0x7f67a40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.198+0000 7f67a9185640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f67a415dfe0 0x7f67a40cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35504/0 (socket says 172.21.15.83:35504) 2024-03-06T07:25:10.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.198+0000 7f67a9185640 1 -- 172.21.15.83:0/2240675259 learned_addr learned my addr 172.21.15.83:0/2240675259 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:10.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.199+0000 7f67a8984640 1 --2- 172.21.15.83:0/2240675259 >> v2:172.21.15.183:3300/0 conn(0x7f67a40cf830 0x7f67a40d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.199+0000 7f67a9185640 1 -- 172.21.15.83:0/2240675259 >> v2:172.21.15.183:3300/0 conn(0x7f67a40cf830 msgr2=0x7f67a40d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:10.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.199+0000 7f67a9185640 1 --2- 172.21.15.83:0/2240675259 >> v2:172.21.15.183:3300/0 conn(0x7f67a40cf830 0x7f67a40d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.199+0000 7f67a9185640 1 -- 172.21.15.83:0/2240675259 >> v1:172.21.15.183:6789/0 conn(0x7f67a40d27f0 legacy=0x7f67a40d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:10.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.199+0000 7f67a9185640 1 -- 172.21.15.83:0/2240675259 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67a40b3710 con 0x7f67a415dfe0 2024-03-06T07:25:10.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.199+0000 7f67a9185640 1 --2- 172.21.15.83:0/2240675259 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67a415dfe0 0x7f67a40cf2f0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f679c009e50 tx=0x7f679c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b130ce0f31900047 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:10.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.200+0000 7f67a3fff640 1 -- 172.21.15.83:0/2240675259 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679c02ca80 con 0x7f67a415dfe0 2024-03-06T07:25:10.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.200+0000 7f67a3fff640 1 -- 172.21.15.83:0/2240675259 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f679c02cc20 con 0x7f67a415dfe0 2024-03-06T07:25:10.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.200+0000 7f67a3fff640 1 -- 172.21.15.83:0/2240675259 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679c0343f0 con 0x7f67a415dfe0 2024-03-06T07:25:10.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.200+0000 7f67ab3e9640 1 -- 172.21.15.83:0/2240675259 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67a415dfe0 msgr2=0x7f67a40cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:10.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.200+0000 7f67ab3e9640 1 --2- 172.21.15.83:0/2240675259 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67a415dfe0 0x7f67a40cf2f0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f679c009e50 tx=0x7f679c02bee0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.200+0000 7f67ab3e9640 1 -- 172.21.15.83:0/2240675259 shutdown_connections 2024-03-06T07:25:10.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.200+0000 7f67ab3e9640 1 --2- 172.21.15.83:0/2240675259 >> v2:172.21.15.183:3300/0 conn(0x7f67a40cf830 0x7f67a40d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.200+0000 7f67ab3e9640 1 --2- 172.21.15.83:0/2240675259 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67a415dfe0 0x7f67a40cf2f0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.200+0000 7f67ab3e9640 1 -- 172.21.15.83:0/2240675259 >> 172.21.15.83:0/2240675259 conn(0x7f67a40d3cd0 msgr2=0x7f67a40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:10.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.200+0000 7f67ab3e9640 1 -- 172.21.15.83:0/2240675259 shutdown_connections 2024-03-06T07:25:10.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.200+0000 7f67ab3e9640 1 -- 172.21.15.83:0/2240675259 wait complete. 2024-03-06T07:25:10.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.201+0000 7f67ab3e9640 1 Processor -- start 2024-03-06T07:25:10.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.201+0000 7f67ab3e9640 1 -- start start 2024-03-06T07:25:10.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.202+0000 7f67ab3e9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67a40cf830 0x7f67a41610a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.202+0000 7f67ab3e9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f67a40d27f0 0x7f67a4149f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.202+0000 7f67ab3e9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f67a415dfe0 0x7f67a414a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.202+0000 7f67ab3e9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a40d1df0 con 0x7f67a40cf830 2024-03-06T07:25:10.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.202+0000 7f67ab3e9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a40d1c70 con 0x7f67a40d27f0 2024-03-06T07:25:10.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.202+0000 7f67ab3e9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67a415e3c0 con 0x7f67a415dfe0 2024-03-06T07:25:10.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.202+0000 7f67a9185640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67a40cf830 0x7f67a41610a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.202+0000 7f67a9185640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67a40cf830 0x7f67a41610a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35516/0 (socket says 172.21.15.83:35516) 2024-03-06T07:25:10.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.202+0000 7f67a9185640 1 -- 172.21.15.83:0/1030395846 learned_addr learned my addr 172.21.15.83:0/1030395846 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:10.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.202+0000 7f67a8984640 1 --2- 172.21.15.83:0/1030395846 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f67a40d27f0 0x7f67a4149f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.202+0000 7f67a9986640 1 --2- 172.21.15.83:0/1030395846 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f67a415dfe0 0x7f67a414a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.202+0000 7f67a9185640 1 -- 172.21.15.83:0/1030395846 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f67a415dfe0 msgr2=0x7f67a414a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:10.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.202+0000 7f67a9185640 1 --2- 172.21.15.83:0/1030395846 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f67a415dfe0 0x7f67a414a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.202+0000 7f67a9185640 1 -- 172.21.15.83:0/1030395846 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f67a40d27f0 msgr2=0x7f67a4149f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:10.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.202+0000 7f67a9185640 1 --2- 172.21.15.83:0/1030395846 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f67a40d27f0 0x7f67a4149f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.202+0000 7f67a9185640 1 -- 172.21.15.83:0/1030395846 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67a40ccbd0 con 0x7f67a40cf830 2024-03-06T07:25:10.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.203+0000 7f67a9185640 1 --2- 172.21.15.83:0/1030395846 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67a40cf830 0x7f67a41610a0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f679c036040 tx=0x7f679c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:10.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.203+0000 7f67a1ffb640 1 -- 172.21.15.83:0/1030395846 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679c043020 con 0x7f67a40cf830 2024-03-06T07:25:10.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.203+0000 7f67a1ffb640 1 -- 172.21.15.83:0/1030395846 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f679c03e450 con 0x7f67a40cf830 2024-03-06T07:25:10.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.203+0000 7f67a1ffb640 1 -- 172.21.15.83:0/1030395846 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679c009510 con 0x7f67a40cf830 2024-03-06T07:25:10.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.203+0000 7f67ab3e9640 1 -- 172.21.15.83:0/1030395846 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67a40cceb0 con 0x7f67a40cf830 2024-03-06T07:25:10.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.203+0000 7f67ab3e9640 1 -- 172.21.15.83:0/1030395846 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67a40cd470 con 0x7f67a40cf830 2024-03-06T07:25:10.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.204+0000 7f67ab3e9640 1 -- 172.21.15.83:0/1030395846 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67a404f330 con 0x7f67a40cf830 2024-03-06T07:25:10.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.204+0000 7f67a1ffb640 1 -- 172.21.15.83:0/1030395846 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f679c046020 con 0x7f67a40cf830 2024-03-06T07:25:10.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.205+0000 7f67a1ffb640 1 --2- 172.21.15.83:0/1030395846 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f679006b4d0 0x7f679006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.205+0000 7f67a1ffb640 1 -- 172.21.15.83:0/1030395846 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f679c0ac670 con 0x7f67a40cf830 2024-03-06T07:25:10.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.205+0000 7f67a8984640 1 --2- 172.21.15.83:0/1030395846 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f679006b4d0 0x7f679006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.205+0000 7f67a8984640 1 --2- 172.21.15.83:0/1030395846 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f679006b4d0 0x7f679006d990 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f6798006fd0 tx=0x7f6798008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:10.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.209+0000 7f67a1ffb640 1 -- 172.21.15.83:0/1030395846 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f679c04d030 con 0x7f67a40cf830 2024-03-06T07:25:10.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.382+0000 7f67ab3e9640 1 -- 172.21.15.83:0/1030395846 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f67a41615e0 con 0x7f67a40cf830 2024-03-06T07:25:10.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.382+0000 7f67a1ffb640 1 -- 172.21.15.83:0/1030395846 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v65) v1 ==== 84+0+65 (secure 0 0 0) 0x7f679c07b8d0 con 0x7f67a40cf830 2024-03-06T07:25:10.384 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:25:10.385 INFO:teuthology.orchestra.run.smithi083.stdout:[ 2024-03-06T07:25:10.385 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:25:10.385 INFO:teuthology.orchestra.run.smithi083.stdout: "poolnum": 1, 2024-03-06T07:25:10.385 INFO:teuthology.orchestra.run.smithi083.stdout: "poolname": ".mgr" 2024-03-06T07:25:10.385 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:25:10.385 INFO:teuthology.orchestra.run.smithi083.stdout:] 2024-03-06T07:25:10.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.384+0000 7f67ab3e9640 1 -- 172.21.15.83:0/1030395846 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f679006b4d0 msgr2=0x7f679006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:10.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.384+0000 7f67ab3e9640 1 --2- 172.21.15.83:0/1030395846 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f679006b4d0 0x7f679006d990 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f6798006fd0 tx=0x7f6798008040 comp rx=0 tx=0).stop 2024-03-06T07:25:10.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.384+0000 7f67ab3e9640 1 -- 172.21.15.83:0/1030395846 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67a40cf830 msgr2=0x7f67a41610a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:10.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.384+0000 7f67ab3e9640 1 --2- 172.21.15.83:0/1030395846 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67a40cf830 0x7f67a41610a0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f679c036040 tx=0x7f679c004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:10.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.385+0000 7f67ab3e9640 1 -- 172.21.15.83:0/1030395846 shutdown_connections 2024-03-06T07:25:10.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.385+0000 7f67ab3e9640 1 --2- 172.21.15.83:0/1030395846 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f679006b4d0 0x7f679006d990 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.385+0000 7f67ab3e9640 1 --2- 172.21.15.83:0/1030395846 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f67a415dfe0 0x7f67a414a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.385+0000 7f67ab3e9640 1 --2- 172.21.15.83:0/1030395846 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f67a40d27f0 0x7f67a4149f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.385+0000 7f67ab3e9640 1 --2- 172.21.15.83:0/1030395846 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67a40cf830 0x7f67a41610a0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.385+0000 7f67ab3e9640 1 -- 172.21.15.83:0/1030395846 >> 172.21.15.83:0/1030395846 conn(0x7f67a40d3cd0 msgr2=0x7f67a40cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:10.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.385+0000 7f67ab3e9640 1 -- 172.21.15.83:0/1030395846 shutdown_connections 2024-03-06T07:25:10.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.385+0000 7f67ab3e9640 1 -- 172.21.15.83:0/1030395846 wait complete. 2024-03-06T07:25:10.397 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:25:10.397 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:25:10.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.499+0000 7fcfebfff640 1 Processor -- start 2024-03-06T07:25:10.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.499+0000 7fcfebfff640 1 -- start start 2024-03-06T07:25:10.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.500+0000 7fcfebfff640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fcfec0ae6b0 0x7fcfec070fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.500+0000 7fcfebfff640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfec0aee50 con 0x7fcfec0cf720 2024-03-06T07:25:10.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.500+0000 7fcfebfff640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfec0af130 con 0x7fcfec15cb70 2024-03-06T07:25:10.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.500+0000 7fcfebfff640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfec15cfd0 con 0x7fcfec0ae6b0 2024-03-06T07:25:10.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.500+0000 7fcfeaffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fcfec0cf720 0x7fcfec0cfb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48794/0 (socket says 172.21.15.83:48794) 2024-03-06T07:25:10.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.500+0000 7fcfeaffd640 1 -- 172.21.15.83:0/2500430122 learned_addr learned my addr 172.21.15.83:0/2500430122 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:25:10.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.500+0000 7fcfe9ffb640 1 -- 172.21.15.83:0/2500430122 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1364552118 0 0) 0x7fcfec0aee50 con 0x7fcfec0cf720 2024-03-06T07:25:10.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.500+0000 7fcfeb7fe640 1 --2- 172.21.15.83:0/2500430122 >> v2:172.21.15.183:3300/0 conn(0x7fcfec0ae6b0 0x7fcfec070fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.500+0000 7fcfe9ffb640 1 -- 172.21.15.83:0/2500430122 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfd4003520 con 0x7fcfec0cf720 2024-03-06T07:25:10.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.500+0000 7fcfe9ffb640 1 -- 172.21.15.83:0/2500430122 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 984325978 0 0) 0x7fcfd4003520 con 0x7fcfec0cf720 2024-03-06T07:25:10.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.500+0000 7fcfe9ffb640 1 -- 172.21.15.83:0/2500430122 >> v2:172.21.15.183:3300/0 conn(0x7fcfec0ae6b0 msgr2=0x7fcfec070fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:10.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.500+0000 7fcfe9ffb640 1 --2- 172.21.15.83:0/2500430122 >> v2:172.21.15.183:3300/0 conn(0x7fcfec0ae6b0 0x7fcfec070fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.500+0000 7fcfe9ffb640 1 -- 172.21.15.83:0/2500430122 >> v1:172.21.15.183:6789/0 conn(0x7fcfec15cb70 legacy=0x7fcfec0b3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:10.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.500+0000 7fcfe9ffb640 1 -- 172.21.15.83:0/2500430122 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfec0b2ff0 con 0x7fcfec0cf720 2024-03-06T07:25:10.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.500+0000 7fcfe9ffb640 1 -- 172.21.15.83:0/2500430122 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7fcfdc0019f0 con 0x7fcfec0cf720 2024-03-06T07:25:10.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.500+0000 7fcfe9ffb640 1 -- 172.21.15.83:0/2500430122 >> v1:172.21.15.83:6789/0 conn(0x7fcfec0cf720 legacy=0x7fcfec0cfb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:10.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.500+0000 7fcfe9ffb640 1 --2- 172.21.15.83:0/2500430122 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcfd40039b0 0x7fcfd4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.500+0000 7fcfe9ffb640 1 --2- 172.21.15.83:0/2500430122 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcfd4004800 0x7fcfd4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.500+0000 7fcfe9ffb640 1 --2- 172.21.15.83:0/2500430122 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcfd40051c0 0x7fcfd400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.500+0000 7fcfe9ffb640 1 -- 172.21.15.83:0/2500430122 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfec15cfd0 con 0x7fcfd4004800 2024-03-06T07:25:10.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.501+0000 7fcfe9ffb640 1 -- 172.21.15.83:0/2500430122 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd400c060 con 0x7fcfd40051c0 2024-03-06T07:25:10.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.501+0000 7fcfe9ffb640 1 -- 172.21.15.83:0/2500430122 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd400c260 con 0x7fcfd40039b0 2024-03-06T07:25:10.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.501+0000 7fcfea7fc640 1 --2- 172.21.15.83:0/2500430122 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcfd4004800 0x7fcfd4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.501+0000 7fcfeaffd640 1 --2- 172.21.15.83:0/2500430122 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcfd40039b0 0x7fcfd4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.501+0000 7fcfeb7fe640 1 --2- 172.21.15.83:0/2500430122 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcfd40051c0 0x7fcfd400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.501+0000 7fcfea7fc640 1 -- 172.21.15.83:0/2500430122 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcfd40039b0 msgr2=0x7fcfd4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:10.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.501+0000 7fcfea7fc640 1 --2- 172.21.15.83:0/2500430122 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcfd40039b0 0x7fcfd4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.501+0000 7fcfea7fc640 1 -- 172.21.15.83:0/2500430122 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcfd40051c0 msgr2=0x7fcfd400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:10.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.501+0000 7fcfea7fc640 1 --2- 172.21.15.83:0/2500430122 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcfd40051c0 0x7fcfd400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.501+0000 7fcfea7fc640 1 -- 172.21.15.83:0/2500430122 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd4003520 con 0x7fcfd4004800 2024-03-06T07:25:10.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.501+0000 7fcfeb7fe640 1 -- 172.21.15.83:0/2500430122 reap_dead start 2024-03-06T07:25:10.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.501+0000 7fcfea7fc640 1 --2- 172.21.15.83:0/2500430122 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcfd4004800 0x7fcfd4004c80 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fcfe0001750 tx=0x7fcfd4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c037d9fe24c953b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:10.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.502+0000 7fcfe9ffb640 1 -- 172.21.15.83:0/2500430122 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfe0001370 con 0x7fcfd4004800 2024-03-06T07:25:10.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.502+0000 7fcfe9ffb640 1 -- 172.21.15.83:0/2500430122 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfe00052d0 con 0x7fcfd4004800 2024-03-06T07:25:10.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.502+0000 7fcfe9ffb640 1 -- 172.21.15.83:0/2500430122 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfe0003830 con 0x7fcfd4004800 2024-03-06T07:25:10.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.502+0000 7fcfebfff640 1 -- 172.21.15.83:0/2500430122 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcfd4004800 msgr2=0x7fcfd4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:10.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.502+0000 7fcfebfff640 1 --2- 172.21.15.83:0/2500430122 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcfd4004800 0x7fcfd4004c80 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fcfe0001750 tx=0x7fcfd4005830 comp rx=0 tx=0).stop 2024-03-06T07:25:10.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.502+0000 7fcfebfff640 1 -- 172.21.15.83:0/2500430122 shutdown_connections 2024-03-06T07:25:10.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.502+0000 7fcfebfff640 1 --2- 172.21.15.83:0/2500430122 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcfd4004800 0x7fcfd4004c80 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.502+0000 7fcfebfff640 1 -- 172.21.15.83:0/2500430122 >> 172.21.15.83:0/2500430122 conn(0x7fcfec0b1500 msgr2=0x7fcfec0b18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:10.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.502+0000 7fcfebfff640 1 -- 172.21.15.83:0/2500430122 shutdown_connections 2024-03-06T07:25:10.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.502+0000 7fcfebfff640 1 -- 172.21.15.83:0/2500430122 wait complete. 2024-03-06T07:25:10.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.503+0000 7fcfebfff640 1 Processor -- start 2024-03-06T07:25:10.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.504+0000 7fcfebfff640 1 -- start start 2024-03-06T07:25:10.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.504+0000 7fcfebfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcfd4004800 0x7fcfec0adf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.504+0000 7fcfebfff640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcfec151260 0x7fcfec15aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.504+0000 7fcfebfff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcfec15b0e0 0x7fcfec1f49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.504+0000 7fcfebfff640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfec0aeb20 con 0x7fcfd4004800 2024-03-06T07:25:10.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.504+0000 7fcfebfff640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfec0b35b0 con 0x7fcfec151260 2024-03-06T07:25:10.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.504+0000 7fcfebfff640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfec159850 con 0x7fcfec15b0e0 2024-03-06T07:25:10.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.504+0000 7fcfeaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcfd4004800 0x7fcfec0adf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.504+0000 7fcfeaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcfd4004800 0x7fcfec0adf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35542/0 (socket says 172.21.15.83:35542) 2024-03-06T07:25:10.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.504+0000 7fcfeaffd640 1 -- 172.21.15.83:0/2966752374 learned_addr learned my addr 172.21.15.83:0/2966752374 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:10.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.504+0000 7fcfea7fc640 1 --2- 172.21.15.83:0/2966752374 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcfec151260 0x7fcfec15aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.504+0000 7fcfeb7fe640 1 --2- 172.21.15.83:0/2966752374 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcfec15b0e0 0x7fcfec1f49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.504+0000 7fcfeaffd640 1 -- 172.21.15.83:0/2966752374 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcfec15b0e0 msgr2=0x7fcfec1f49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:10.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.504+0000 7fcfeaffd640 1 --2- 172.21.15.83:0/2966752374 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcfec15b0e0 0x7fcfec1f49b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.504+0000 7fcfeaffd640 1 -- 172.21.15.83:0/2966752374 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcfec151260 msgr2=0x7fcfec15aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:10.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.504+0000 7fcfeaffd640 1 --2- 172.21.15.83:0/2966752374 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcfec151260 0x7fcfec15aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.504+0000 7fcfeaffd640 1 -- 172.21.15.83:0/2966752374 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfec0ae4d0 con 0x7fcfd4004800 2024-03-06T07:25:10.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.504+0000 7fcfea7fc640 1 --2- 172.21.15.83:0/2966752374 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcfec151260 0x7fcfec15aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:10.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.504+0000 7fcfeb7fe640 1 --2- 172.21.15.83:0/2966752374 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcfec15b0e0 0x7fcfec1f49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:10.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.505+0000 7fcfeaffd640 1 --2- 172.21.15.83:0/2966752374 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcfd4004800 0x7fcfec0adf90 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc007ae0 tx=0x7fcfdc00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:10.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.505+0000 7fcfd37fe640 1 -- 172.21.15.83:0/2966752374 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc013420 con 0x7fcfd4004800 2024-03-06T07:25:10.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.505+0000 7fcfd37fe640 1 -- 172.21.15.83:0/2966752374 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfdc018070 con 0x7fcfd4004800 2024-03-06T07:25:10.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.505+0000 7fcfebfff640 1 -- 172.21.15.83:0/2966752374 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfec151cc0 con 0x7fcfd4004800 2024-03-06T07:25:10.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.505+0000 7fcfd37fe640 1 -- 172.21.15.83:0/2966752374 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc013820 con 0x7fcfd4004800 2024-03-06T07:25:10.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.505+0000 7fcfebfff640 1 -- 172.21.15.83:0/2966752374 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfec152240 con 0x7fcfd4004800 2024-03-06T07:25:10.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.506+0000 7fcfebfff640 1 -- 172.21.15.83:0/2966752374 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfec068f10 con 0x7fcfd4004800 2024-03-06T07:25:10.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.506+0000 7fcfd37fe640 1 -- 172.21.15.83:0/2966752374 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcfdc0139c0 con 0x7fcfd4004800 2024-03-06T07:25:10.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.506+0000 7fcfd37fe640 1 --2- 172.21.15.83:0/2966752374 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcfd806b4d0 0x7fcfd806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.507+0000 7fcfd37fe640 1 -- 172.21.15.83:0/2966752374 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fcfdc08a600 con 0x7fcfd4004800 2024-03-06T07:25:10.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.507+0000 7fcfea7fc640 1 --2- 172.21.15.83:0/2966752374 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcfd806b4d0 0x7fcfd806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.507+0000 7fcfea7fc640 1 --2- 172.21.15.83:0/2966752374 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcfd806b4d0 0x7fcfd806d990 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fcfe0033000 tx=0x7fcfe0009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:10.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.511+0000 7fcfd37fe640 1 -- 172.21.15.83:0/2966752374 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcfec0ae4d0 con 0x7fcfd4004800 2024-03-06T07:25:10.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:10.682+0000 7fcfebfff640 1 -- 172.21.15.83:0/2966752374 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fcfec071120 con 0x7fcfd4004800 2024-03-06T07:25:11.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.123+0000 7fcfd37fe640 1 -- 172.21.15.83:0/2966752374 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v66) v1 ==== 138+0+0 (secure 0 0 0) 0x7fcfdc0597d0 con 0x7fcfd4004800 2024-03-06T07:25:11.125 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'cephfs_metadata' created 2024-03-06T07:25:11.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.125+0000 7fcfebfff640 1 -- 172.21.15.83:0/2966752374 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcfd806b4d0 msgr2=0x7fcfd806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:11.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.125+0000 7fcfebfff640 1 --2- 172.21.15.83:0/2966752374 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcfd806b4d0 0x7fcfd806d990 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fcfe0033000 tx=0x7fcfe0009030 comp rx=0 tx=0).stop 2024-03-06T07:25:11.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.125+0000 7fcfebfff640 1 -- 172.21.15.83:0/2966752374 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcfd4004800 msgr2=0x7fcfec0adf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:11.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.125+0000 7fcfebfff640 1 --2- 172.21.15.83:0/2966752374 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcfd4004800 0x7fcfec0adf90 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc007ae0 tx=0x7fcfdc00e530 comp rx=0 tx=0).stop 2024-03-06T07:25:11.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.125+0000 7fcfebfff640 1 -- 172.21.15.83:0/2966752374 shutdown_connections 2024-03-06T07:25:11.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.125+0000 7fcfebfff640 1 --2- 172.21.15.83:0/2966752374 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcfd806b4d0 0x7fcfd806d990 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.125+0000 7fcfebfff640 1 --2- 172.21.15.83:0/2966752374 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcfec15b0e0 0x7fcfec1f49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.125+0000 7fcfebfff640 1 --2- 172.21.15.83:0/2966752374 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcfec151260 0x7fcfec15aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.125+0000 7fcfebfff640 1 --2- 172.21.15.83:0/2966752374 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcfd4004800 0x7fcfec0adf90 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.125+0000 7fcfebfff640 1 -- 172.21.15.83:0/2966752374 >> 172.21.15.83:0/2966752374 conn(0x7fcfec0b1500 msgr2=0x7fcfec071ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:11.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.125+0000 7fcfebfff640 1 -- 172.21.15.83:0/2966752374 shutdown_connections 2024-03-06T07:25:11.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.125+0000 7fcfebfff640 1 -- 172.21.15.83:0/2966752374 wait complete. 2024-03-06T07:25:11.141 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:25:11.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.247+0000 7ff1a8ba0640 1 Processor -- start 2024-03-06T07:25:11.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.247+0000 7ff1a8ba0640 1 -- start start 2024-03-06T07:25:11.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.248+0000 7ff1a8ba0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff1a415e010 0x7ff1a40d4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:11.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.248+0000 7ff1a8ba0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff1a40cf130 0x7ff1a40d2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:11.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.248+0000 7ff1a8ba0640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ff1a40d2660 0x7ff1a40d0170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:11.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.248+0000 7ff1a8ba0640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1a40d2ad0 con 0x7ff1a40cf130 2024-03-06T07:25:11.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.248+0000 7ff1a8ba0640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1a415e3f0 con 0x7ff1a40d2660 2024-03-06T07:25:11.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.248+0000 7ff1a8ba0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a40cf890 con 0x7ff1a415e010 2024-03-06T07:25:11.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.248+0000 7ff1a2ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff1a40cf130 0x7ff1a40d2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:11.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.248+0000 7ff1a2ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff1a40cf130 0x7ff1a40d2120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35554/0 (socket says 172.21.15.83:35554) 2024-03-06T07:25:11.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.248+0000 7ff1a2ffd640 1 -- 172.21.15.83:0/1259771798 learned_addr learned my addr 172.21.15.83:0/1259771798 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:11.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.248+0000 7ff1a2ffd640 1 -- 172.21.15.83:0/1259771798 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff1a415e010 msgr2=0x7ff1a40d4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:11.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.248+0000 7ff1a2ffd640 1 --2- 172.21.15.83:0/1259771798 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff1a415e010 0x7ff1a40d4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.248+0000 7ff1a2ffd640 1 -- 172.21.15.83:0/1259771798 >> v2:172.21.15.183:3300/0 conn(0x7ff1a40d2660 msgr2=0x7ff1a40d0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:11.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.248+0000 7ff1a2ffd640 1 --2- 172.21.15.83:0/1259771798 >> v2:172.21.15.183:3300/0 conn(0x7ff1a40d2660 0x7ff1a40d0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.248+0000 7ff1a2ffd640 1 -- 172.21.15.83:0/1259771798 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a40b31a0 con 0x7ff1a40cf130 2024-03-06T07:25:11.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.249+0000 7ff1a2ffd640 1 --2- 172.21.15.83:0/1259771798 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1a40cf130 0x7ff1a40d2120 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7ff1a406b400 tx=0x7ff19802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b0777c1a27496f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:11.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.249+0000 7ff1a27fc640 1 -- 172.21.15.83:0/1259771798 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff198035020 con 0x7ff1a40cf130 2024-03-06T07:25:11.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.249+0000 7ff1a27fc640 1 -- 172.21.15.83:0/1259771798 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff198031870 con 0x7ff1a40cf130 2024-03-06T07:25:11.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.249+0000 7ff1a27fc640 1 -- 172.21.15.83:0/1259771798 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff198031c40 con 0x7ff1a40cf130 2024-03-06T07:25:11.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.250+0000 7ff1a8ba0640 1 -- 172.21.15.83:0/1259771798 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1a40cf130 msgr2=0x7ff1a40d2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:11.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.250+0000 7ff1a8ba0640 1 --2- 172.21.15.83:0/1259771798 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1a40cf130 0x7ff1a40d2120 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7ff1a406b400 tx=0x7ff19802b910 comp rx=0 tx=0).stop 2024-03-06T07:25:11.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.250+0000 7ff1a8ba0640 1 -- 172.21.15.83:0/1259771798 shutdown_connections 2024-03-06T07:25:11.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.250+0000 7ff1a8ba0640 1 --2- 172.21.15.83:0/1259771798 >> v2:172.21.15.183:3300/0 conn(0x7ff1a40d2660 0x7ff1a40d0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.250+0000 7ff1a8ba0640 1 --2- 172.21.15.83:0/1259771798 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1a40cf130 0x7ff1a40d2120 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.250+0000 7ff1a8ba0640 1 --2- 172.21.15.83:0/1259771798 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff1a415e010 0x7ff1a40d4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.250+0000 7ff1a8ba0640 1 -- 172.21.15.83:0/1259771798 >> 172.21.15.83:0/1259771798 conn(0x7ff1a409ce80 msgr2=0x7ff1a40d3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:11.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.250+0000 7ff1a8ba0640 1 -- 172.21.15.83:0/1259771798 shutdown_connections 2024-03-06T07:25:11.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.250+0000 7ff1a8ba0640 1 -- 172.21.15.83:0/1259771798 wait complete. 2024-03-06T07:25:11.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.251+0000 7ff1a8ba0640 1 Processor -- start 2024-03-06T07:25:11.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.251+0000 7ff1a8ba0640 1 -- start start 2024-03-06T07:25:11.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.252+0000 7ff1a8ba0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff1a40cf130 0x7ff1a40b70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:11.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.252+0000 7ff1a8ba0640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1a40d2660 0x7ff1a40b8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:11.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.252+0000 7ff1a8ba0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1a415e010 0x7ff1a4159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:11.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.252+0000 7ff1a8ba0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a40d06b0 con 0x7ff1a415e010 2024-03-06T07:25:11.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.252+0000 7ff1a8ba0640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a40cf590 con 0x7ff1a40d2660 2024-03-06T07:25:11.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.252+0000 7ff1a8ba0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a40d0830 con 0x7ff1a40cf130 2024-03-06T07:25:11.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.252+0000 7ff1a3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1a415e010 0x7ff1a4159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:11.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.252+0000 7ff1a3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1a415e010 0x7ff1a4159ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35556/0 (socket says 172.21.15.83:35556) 2024-03-06T07:25:11.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.252+0000 7ff1a3fff640 1 -- 172.21.15.83:0/114225413 learned_addr learned my addr 172.21.15.83:0/114225413 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:11.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.252+0000 7ff1a37fe640 1 --2- 172.21.15.83:0/114225413 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff1a40cf130 0x7ff1a40b70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:11.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.252+0000 7ff1a2ffd640 1 --2- 172.21.15.83:0/114225413 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1a40d2660 0x7ff1a40b8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:11.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.252+0000 7ff1a3fff640 1 -- 172.21.15.83:0/114225413 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff1a40cf130 msgr2=0x7ff1a40b70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:11.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.252+0000 7ff1a3fff640 1 --2- 172.21.15.83:0/114225413 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff1a40cf130 0x7ff1a40b70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.252+0000 7ff1a3fff640 1 -- 172.21.15.83:0/114225413 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1a40d2660 msgr2=0x7ff1a40b8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:11.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.252+0000 7ff1a3fff640 1 --2- 172.21.15.83:0/114225413 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1a40d2660 0x7ff1a40b8fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.252+0000 7ff1a3fff640 1 -- 172.21.15.83:0/114225413 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a4160390 con 0x7ff1a415e010 2024-03-06T07:25:11.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.252+0000 7ff1a3fff640 1 --2- 172.21.15.83:0/114225413 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1a415e010 0x7ff1a4159ed0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7ff19c00b9a0 tx=0x7ff19c00be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:11.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.252+0000 7ff18bfff640 1 -- 172.21.15.83:0/114225413 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff19c019070 con 0x7ff1a415e010 2024-03-06T07:25:11.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.253+0000 7ff18bfff640 1 -- 172.21.15.83:0/114225413 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff19c00cbc0 con 0x7ff1a415e010 2024-03-06T07:25:11.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.253+0000 7ff1a8ba0640 1 -- 172.21.15.83:0/114225413 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1a4160670 con 0x7ff1a415e010 2024-03-06T07:25:11.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.253+0000 7ff18bfff640 1 -- 172.21.15.83:0/114225413 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff19c014440 con 0x7ff1a415e010 2024-03-06T07:25:11.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.253+0000 7ff1a8ba0640 1 -- 172.21.15.83:0/114225413 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1a4160c30 con 0x7ff1a415e010 2024-03-06T07:25:11.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.254+0000 7ff1a8ba0640 1 -- 172.21.15.83:0/114225413 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1a4068f10 con 0x7ff1a415e010 2024-03-06T07:25:11.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.254+0000 7ff18bfff640 1 -- 172.21.15.83:0/114225413 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff19c01c070 con 0x7ff1a415e010 2024-03-06T07:25:11.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.254+0000 7ff18bfff640 1 --2- 172.21.15.83:0/114225413 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff19006b4d0 0x7ff19006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:11.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.254+0000 7ff18bfff640 1 -- 172.21.15.83:0/114225413 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5227+0+0 (secure 0 0 0) 0x7ff19c089db0 con 0x7ff1a415e010 2024-03-06T07:25:11.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.254+0000 7ff1a37fe640 1 --2- 172.21.15.83:0/114225413 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff19006b4d0 0x7ff19006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:11.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.255+0000 7ff1a37fe640 1 --2- 172.21.15.83:0/114225413 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff19006b4d0 0x7ff19006d990 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7ff19400b810 tx=0x7ff194008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:11.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.259+0000 7ff18bfff640 1 -- 172.21.15.83:0/114225413 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff19c058e10 con 0x7ff1a415e010 2024-03-06T07:25:11.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:11.431+0000 7ff1a8ba0640 1 -- 172.21.15.83:0/114225413 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7ff1a40d4a50 con 0x7ff1a415e010 2024-03-06T07:25:12.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.125+0000 7ff18bfff640 1 -- 172.21.15.83:0/114225413 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v67) v1 ==== 170+0+0 (secure 0 0 0) 0x7ff19c058710 con 0x7ff1a415e010 2024-03-06T07:25:12.127 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'cephfs_data' created 2024-03-06T07:25:12.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.128+0000 7ff1a8ba0640 1 -- 172.21.15.83:0/114225413 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff19006b4d0 msgr2=0x7ff19006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:12.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.128+0000 7ff1a8ba0640 1 --2- 172.21.15.83:0/114225413 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff19006b4d0 0x7ff19006d990 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7ff19400b810 tx=0x7ff194008040 comp rx=0 tx=0).stop 2024-03-06T07:25:12.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.128+0000 7ff1a8ba0640 1 -- 172.21.15.83:0/114225413 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1a415e010 msgr2=0x7ff1a4159ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:12.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.128+0000 7ff1a8ba0640 1 --2- 172.21.15.83:0/114225413 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1a415e010 0x7ff1a4159ed0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7ff19c00b9a0 tx=0x7ff19c00be60 comp rx=0 tx=0).stop 2024-03-06T07:25:12.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.128+0000 7ff1a8ba0640 1 -- 172.21.15.83:0/114225413 shutdown_connections 2024-03-06T07:25:12.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.128+0000 7ff1a8ba0640 1 --2- 172.21.15.83:0/114225413 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff19006b4d0 0x7ff19006d990 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:12.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.128+0000 7ff1a8ba0640 1 --2- 172.21.15.83:0/114225413 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1a415e010 0x7ff1a4159ed0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:12.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.128+0000 7ff1a8ba0640 1 --2- 172.21.15.83:0/114225413 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1a40d2660 0x7ff1a40b8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:12.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.128+0000 7ff1a8ba0640 1 --2- 172.21.15.83:0/114225413 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff1a40cf130 0x7ff1a40b70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:12.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.128+0000 7ff1a8ba0640 1 -- 172.21.15.83:0/114225413 >> 172.21.15.83:0/114225413 conn(0x7ff1a409ce80 msgr2=0x7ff1a40b4db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:12.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.128+0000 7ff1a8ba0640 1 -- 172.21.15.83:0/114225413 shutdown_connections 2024-03-06T07:25:12.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.129+0000 7ff1a8ba0640 1 -- 172.21.15.83:0/114225413 wait complete. 2024-03-06T07:25:12.151 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:25:12.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.256+0000 7fa26135e640 1 Processor -- start 2024-03-06T07:25:12.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.256+0000 7fa26135e640 1 -- start start 2024-03-06T07:25:12.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.257+0000 7fa26135e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fa25c158420 0x7fa25c0b35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:12.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.257+0000 7fa26135e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa25c15ba80 0x7fa25c0b3b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:12.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.257+0000 7fa26135e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa25c15e8a0 con 0x7fa25c158800 2024-03-06T07:25:12.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.257+0000 7fa26135e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa25c15e650 con 0x7fa25c158420 2024-03-06T07:25:12.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.257+0000 7fa26135e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c15eb80 con 0x7fa25c15ba80 2024-03-06T07:25:12.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.257+0000 7fa25b7fe640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa25c15ba80 0x7fa25c0b3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:12.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.257+0000 7fa25affd640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fa25c158420 0x7fa25c0b35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:12.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.257+0000 7fa25affd640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fa25c158420 0x7fa25c0b35c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:38238/0 (socket says 172.21.15.83:38238) 2024-03-06T07:25:12.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.257+0000 7fa25b7fe640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa25c15ba80 0x7fa25c0b3b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:57558/0 (socket says 172.21.15.83:57558) 2024-03-06T07:25:12.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.257+0000 7fa25a7fc640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fa25c158800 0x7fa25c15b370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:57900/0 (socket says 172.21.15.83:57900) 2024-03-06T07:25:12.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.257+0000 7fa25affd640 1 -- 172.21.15.83:0/2173582634 learned_addr learned my addr 172.21.15.83:0/2173582634 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:12.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.258+0000 7fa25affd640 1 -- 172.21.15.83:0/2173582634 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa25c15ba80 msgr2=0x7fa25c0b3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:12.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.258+0000 7fa25affd640 1 --2- 172.21.15.83:0/2173582634 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa25c15ba80 0x7fa25c0b3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:12.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.258+0000 7fa25affd640 1 -- 172.21.15.83:0/2173582634 >> v1:172.21.15.183:6789/0 conn(0x7fa25c158800 legacy=0x7fa25c15b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:12.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.258+0000 7fa25affd640 1 -- 172.21.15.83:0/2173582634 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa25c15ee60 con 0x7fa25c158420 2024-03-06T07:25:12.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.258+0000 7fa25b7fe640 1 --2- 172.21.15.83:0/2173582634 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa25c15ba80 0x7fa25c0b3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:12.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.258+0000 7fa25affd640 1 --2- 172.21.15.83:0/2173582634 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa25c158420 0x7fa25c0b35c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa24c009e50 tx=0x7fa24c02bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e1c63f0bb5a9dcc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:12.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.259+0000 7fa259ffb640 1 -- 172.21.15.83:0/2173582634 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24c02ca70 con 0x7fa25c158420 2024-03-06T07:25:12.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.259+0000 7fa259ffb640 1 -- 172.21.15.83:0/2173582634 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa24c02cc10 con 0x7fa25c158420 2024-03-06T07:25:12.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.259+0000 7fa259ffb640 1 -- 172.21.15.83:0/2173582634 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24c0343f0 con 0x7fa25c158420 2024-03-06T07:25:12.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.259+0000 7fa26135e640 1 -- 172.21.15.83:0/2173582634 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa25c158420 msgr2=0x7fa25c0b35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:12.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.259+0000 7fa26135e640 1 --2- 172.21.15.83:0/2173582634 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa25c158420 0x7fa25c0b35c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa24c009e50 tx=0x7fa24c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:12.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.259+0000 7fa26135e640 1 -- 172.21.15.83:0/2173582634 shutdown_connections 2024-03-06T07:25:12.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.259+0000 7fa26135e640 1 --2- 172.21.15.83:0/2173582634 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa25c15ba80 0x7fa25c0b3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:12.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.259+0000 7fa26135e640 1 --2- 172.21.15.83:0/2173582634 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa25c158420 0x7fa25c0b35c0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:12.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.259+0000 7fa26135e640 1 -- 172.21.15.83:0/2173582634 >> 172.21.15.83:0/2173582634 conn(0x7fa25c06afb0 msgr2=0x7fa25c06b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:12.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.259+0000 7fa26135e640 1 -- 172.21.15.83:0/2173582634 shutdown_connections 2024-03-06T07:25:12.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.259+0000 7fa26135e640 1 -- 172.21.15.83:0/2173582634 wait complete. 2024-03-06T07:25:12.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.260+0000 7fa26135e640 1 Processor -- start 2024-03-06T07:25:12.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.261+0000 7fa26135e640 1 -- start start 2024-03-06T07:25:12.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.261+0000 7fa26135e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa25c158420 0x7fa25c0ae6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:12.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.261+0000 7fa26135e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa25c15ba80 0x7fa25c145d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:12.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.261+0000 7fa26135e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa25c1f3640 0x7fa25c1475c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:12.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.261+0000 7fa26135e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c071ba0 con 0x7fa25c158420 2024-03-06T07:25:12.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.261+0000 7fa26135e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c071a20 con 0x7fa25c1f3640 2024-03-06T07:25:12.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.261+0000 7fa26135e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c071d20 con 0x7fa25c15ba80 2024-03-06T07:25:12.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.261+0000 7fa25affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa25c158420 0x7fa25c0ae6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:12.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.261+0000 7fa25affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa25c158420 0x7fa25c0ae6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35568/0 (socket says 172.21.15.83:35568) 2024-03-06T07:25:12.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.261+0000 7fa25affd640 1 -- 172.21.15.83:0/4085854950 learned_addr learned my addr 172.21.15.83:0/4085854950 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:12.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.261+0000 7fa25a7fc640 1 --2- 172.21.15.83:0/4085854950 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa25c15ba80 0x7fa25c145d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:12.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.261+0000 7fa25b7fe640 1 --2- 172.21.15.83:0/4085854950 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa25c1f3640 0x7fa25c1475c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:12.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.262+0000 7fa25affd640 1 -- 172.21.15.83:0/4085854950 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa25c15ba80 msgr2=0x7fa25c145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:12.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.262+0000 7fa25affd640 1 --2- 172.21.15.83:0/4085854950 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa25c15ba80 0x7fa25c145d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:12.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.262+0000 7fa25affd640 1 -- 172.21.15.83:0/4085854950 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa25c1f3640 msgr2=0x7fa25c1475c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:12.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.262+0000 7fa25affd640 1 --2- 172.21.15.83:0/4085854950 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa25c1f3640 0x7fa25c1475c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:12.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.262+0000 7fa25affd640 1 -- 172.21.15.83:0/4085854950 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa25c146280 con 0x7fa25c158420 2024-03-06T07:25:12.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.262+0000 7fa25affd640 1 --2- 172.21.15.83:0/4085854950 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa25c158420 0x7fa25c0ae6a0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fa24c009f70 tx=0x7fa24c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:12.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.262+0000 7fa2437fe640 1 -- 172.21.15.83:0/4085854950 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24c043020 con 0x7fa25c158420 2024-03-06T07:25:12.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.262+0000 7fa2437fe640 1 -- 172.21.15.83:0/4085854950 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa24c03e450 con 0x7fa25c158420 2024-03-06T07:25:12.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.262+0000 7fa26135e640 1 -- 172.21.15.83:0/4085854950 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa25c146520 con 0x7fa25c158420 2024-03-06T07:25:12.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.262+0000 7fa2437fe640 1 -- 172.21.15.83:0/4085854950 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24c009510 con 0x7fa25c158420 2024-03-06T07:25:12.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.262+0000 7fa26135e640 1 -- 172.21.15.83:0/4085854950 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa25c146ae0 con 0x7fa25c158420 2024-03-06T07:25:12.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.263+0000 7fa26135e640 1 -- 172.21.15.83:0/4085854950 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa25c04f330 con 0x7fa25c158420 2024-03-06T07:25:12.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.264+0000 7fa2437fe640 1 -- 172.21.15.83:0/4085854950 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa24c046020 con 0x7fa25c158420 2024-03-06T07:25:12.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.264+0000 7fa2437fe640 1 --2- 172.21.15.83:0/4085854950 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa24806b500 0x7fa24806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:12.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.264+0000 7fa2437fe640 1 -- 172.21.15.83:0/4085854950 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fa24c0ac5f0 con 0x7fa25c158420 2024-03-06T07:25:12.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.264+0000 7fa25a7fc640 1 --2- 172.21.15.83:0/4085854950 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa24806b500 0x7fa24806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:12.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.264+0000 7fa25a7fc640 1 --2- 172.21.15.83:0/4085854950 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa24806b500 0x7fa24806d9c0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fa25c1f71f0 tx=0x7fa250000ed0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:12.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.269+0000 7fa2437fe640 1 -- 172.21.15.83:0/4085854950 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa24c04d070 con 0x7fa25c158420 2024-03-06T07:25:12.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:12.480+0000 7fa26135e640 1 -- 172.21.15.83:0/4085854950 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fa25c0b2a30 con 0x7fa25c158420 2024-03-06T07:25:13.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.149+0000 7fa2437fe640 1 -- 172.21.15.83:0/4085854950 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v29) v1 ==== 174+0+0 (secure 0 0 0) 0x7fa24c07b4d0 con 0x7fa25c158420 2024-03-06T07:25:13.151 INFO:teuthology.orchestra.run.smithi083.stderr:new fs with metadata pool 10 and data pool 11 2024-03-06T07:25:13.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.158+0000 7fa26135e640 1 -- 172.21.15.83:0/4085854950 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa24806b500 msgr2=0x7fa24806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.158+0000 7fa26135e640 1 --2- 172.21.15.83:0/4085854950 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa24806b500 0x7fa24806d9c0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fa25c1f71f0 tx=0x7fa250000ed0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.158+0000 7fa26135e640 1 -- 172.21.15.83:0/4085854950 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa25c158420 msgr2=0x7fa25c0ae6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.158+0000 7fa26135e640 1 --2- 172.21.15.83:0/4085854950 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa25c158420 0x7fa25c0ae6a0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fa24c009f70 tx=0x7fa24c004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:13.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.161+0000 7fa26135e640 1 -- 172.21.15.83:0/4085854950 shutdown_connections 2024-03-06T07:25:13.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.161+0000 7fa26135e640 1 --2- 172.21.15.83:0/4085854950 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa24806b500 0x7fa24806d9c0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.161+0000 7fa26135e640 1 --2- 172.21.15.83:0/4085854950 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa25c1f3640 0x7fa25c1475c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.161+0000 7fa26135e640 1 --2- 172.21.15.83:0/4085854950 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa25c15ba80 0x7fa25c145d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.161+0000 7fa26135e640 1 --2- 172.21.15.83:0/4085854950 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa25c158420 0x7fa25c0ae6a0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.161+0000 7fa26135e640 1 -- 172.21.15.83:0/4085854950 >> 172.21.15.83:0/4085854950 conn(0x7fa25c06afb0 msgr2=0x7fa25c15c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:13.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.162+0000 7fa26135e640 1 -- 172.21.15.83:0/4085854950 shutdown_connections 2024-03-06T07:25:13.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.162+0000 7fa26135e640 1 -- 172.21.15.83:0/4085854950 wait complete. 2024-03-06T07:25:13.182 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:13.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.284+0000 7fe88bb25640 1 Processor -- start 2024-03-06T07:25:13.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.284+0000 7fe88bb25640 1 -- start start 2024-03-06T07:25:13.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.285+0000 7fe88bb25640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fe88415fa10 0x7fe8840ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.285+0000 7fe88bb25640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8840cebe0 con 0x7fe88415bdd0 2024-03-06T07:25:13.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.285+0000 7fe88bb25640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8840ceec0 con 0x7fe8840d16e0 2024-03-06T07:25:13.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.285+0000 7fe88bb25640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8840cf1a0 con 0x7fe88415fa10 2024-03-06T07:25:13.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.285+0000 7fe8898c1640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fe88415bdd0 0x7fe8840c5610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48808/0 (socket says 172.21.15.83:48808) 2024-03-06T07:25:13.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.285+0000 7fe8898c1640 1 -- 172.21.15.83:0/1748412009 learned_addr learned my addr 172.21.15.83:0/1748412009 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:25:13.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.285+0000 7fe8890c0640 1 --2- 172.21.15.83:0/1748412009 >> v2:172.21.15.183:3300/0 conn(0x7fe88415fa10 0x7fe8840ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.285+0000 7fe8888bf640 1 -- 172.21.15.83:0/1748412009 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4134959558 0 0) 0x7fe8840cebe0 con 0x7fe88415bdd0 2024-03-06T07:25:13.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.285+0000 7fe8888bf640 1 -- 172.21.15.83:0/1748412009 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe870003520 con 0x7fe88415bdd0 2024-03-06T07:25:13.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.285+0000 7fe8888bf640 1 -- 172.21.15.83:0/1748412009 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3783572019 0 0) 0x7fe870003520 con 0x7fe88415bdd0 2024-03-06T07:25:13.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.285+0000 7fe8888bf640 1 -- 172.21.15.83:0/1748412009 >> v2:172.21.15.183:3300/0 conn(0x7fe88415fa10 msgr2=0x7fe8840ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:13.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.285+0000 7fe8888bf640 1 --2- 172.21.15.83:0/1748412009 >> v2:172.21.15.183:3300/0 conn(0x7fe88415fa10 0x7fe8840ce270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.285+0000 7fe8888bf640 1 -- 172.21.15.83:0/1748412009 >> v1:172.21.15.183:6789/0 conn(0x7fe8840d16e0 legacy=0x7fe8840d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.285+0000 7fe8888bf640 1 -- 172.21.15.83:0/1748412009 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8840cc8e0 con 0x7fe88415bdd0 2024-03-06T07:25:13.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.285+0000 7fe8888bf640 1 -- 172.21.15.83:0/1748412009 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7fe880001690 con 0x7fe88415bdd0 2024-03-06T07:25:13.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.285+0000 7fe8888bf640 1 -- 172.21.15.83:0/1748412009 >> v1:172.21.15.83:6789/0 conn(0x7fe88415bdd0 legacy=0x7fe8840c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.285+0000 7fe8888bf640 1 --2- 172.21.15.83:0/1748412009 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe8700039b0 0x7fe870003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.286+0000 7fe8888bf640 1 --2- 172.21.15.83:0/1748412009 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8700043b0 0x7fe870004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.286+0000 7fe8888bf640 1 --2- 172.21.15.83:0/1748412009 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe870005120 0x7fe87000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.286+0000 7fe8888bf640 1 -- 172.21.15.83:0/1748412009 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8840cf1a0 con 0x7fe8700043b0 2024-03-06T07:25:13.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.286+0000 7fe8888bf640 1 -- 172.21.15.83:0/1748412009 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe87000c060 con 0x7fe870005120 2024-03-06T07:25:13.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.286+0000 7fe8888bf640 1 -- 172.21.15.83:0/1748412009 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe87000c260 con 0x7fe8700039b0 2024-03-06T07:25:13.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.286+0000 7fe8890c0640 1 --2- 172.21.15.83:0/1748412009 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8700043b0 0x7fe870004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.286+0000 7fe8898c1640 1 --2- 172.21.15.83:0/1748412009 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe8700039b0 0x7fe870003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.286+0000 7fe88a0c2640 1 --2- 172.21.15.83:0/1748412009 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe870005120 0x7fe87000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.286+0000 7fe8890c0640 1 -- 172.21.15.83:0/1748412009 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe8700039b0 msgr2=0x7fe870003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:13.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.286+0000 7fe8890c0640 1 --2- 172.21.15.83:0/1748412009 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe8700039b0 0x7fe870003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.286+0000 7fe8890c0640 1 -- 172.21.15.83:0/1748412009 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe870005120 msgr2=0x7fe87000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:13.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.286+0000 7fe8890c0640 1 --2- 172.21.15.83:0/1748412009 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe870005120 0x7fe87000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.286+0000 7fe88a0c2640 1 -- 172.21.15.83:0/1748412009 reap_dead start 2024-03-06T07:25:13.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.286+0000 7fe8890c0640 1 -- 172.21.15.83:0/1748412009 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe870003520 con 0x7fe8700043b0 2024-03-06T07:25:13.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.287+0000 7fe8890c0640 1 --2- 172.21.15.83:0/1748412009 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8700043b0 0x7fe870004880 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fe874009a90 tx=0x7fe870005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf0c3ae37bae416b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:13.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.287+0000 7fe8888bf640 1 -- 172.21.15.83:0/1748412009 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe874039070 con 0x7fe8700043b0 2024-03-06T07:25:13.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.287+0000 7fe8888bf640 1 -- 172.21.15.83:0/1748412009 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe87402c680 con 0x7fe8700043b0 2024-03-06T07:25:13.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.287+0000 7fe8888bf640 1 -- 172.21.15.83:0/1748412009 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe874009690 con 0x7fe8700043b0 2024-03-06T07:25:13.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.287+0000 7fe88bb25640 1 -- 172.21.15.83:0/1748412009 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8700043b0 msgr2=0x7fe870004880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.287+0000 7fe88bb25640 1 --2- 172.21.15.83:0/1748412009 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8700043b0 0x7fe870004880 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fe874009a90 tx=0x7fe870005830 comp rx=0 tx=0).stop 2024-03-06T07:25:13.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.287+0000 7fe88bb25640 1 -- 172.21.15.83:0/1748412009 shutdown_connections 2024-03-06T07:25:13.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.287+0000 7fe88bb25640 1 --2- 172.21.15.83:0/1748412009 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8700043b0 0x7fe870004880 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.287+0000 7fe88bb25640 1 -- 172.21.15.83:0/1748412009 >> 172.21.15.83:0/1748412009 conn(0x7fe88409a9f0 msgr2=0x7fe88409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:13.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.287+0000 7fe88bb25640 1 -- 172.21.15.83:0/1748412009 shutdown_connections 2024-03-06T07:25:13.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.287+0000 7fe88bb25640 1 -- 172.21.15.83:0/1748412009 wait complete. 2024-03-06T07:25:13.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.288+0000 7fe88bb25640 1 Processor -- start 2024-03-06T07:25:13.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.289+0000 7fe88bb25640 1 -- start start 2024-03-06T07:25:13.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.289+0000 7fe88bb25640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe8700043b0 0x7fe8840b5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.289+0000 7fe88bb25640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe88415cd40 0x7fe8840b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.289+0000 7fe88bb25640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8840b60b0 0x7fe8840b4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.289+0000 7fe88bb25640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8840c5610 con 0x7fe8840b60b0 2024-03-06T07:25:13.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.289+0000 7fe88bb25640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe88415c1b0 con 0x7fe8700043b0 2024-03-06T07:25:13.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.289+0000 7fe88bb25640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8840c5790 con 0x7fe88415cd40 2024-03-06T07:25:13.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.289+0000 7fe88a0c2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8840b60b0 0x7fe8840b4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.289+0000 7fe88a0c2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8840b60b0 0x7fe8840b4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35600/0 (socket says 172.21.15.83:35600) 2024-03-06T07:25:13.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.289+0000 7fe88a0c2640 1 -- 172.21.15.83:0/1742974669 learned_addr learned my addr 172.21.15.83:0/1742974669 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:13.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.289+0000 7fe8898c1640 1 --2- 172.21.15.83:0/1742974669 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe8700043b0 0x7fe8840b5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.289+0000 7fe8890c0640 1 --2- 172.21.15.83:0/1742974669 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe88415cd40 0x7fe8840b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.290+0000 7fe88a0c2640 1 -- 172.21.15.83:0/1742974669 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe88415cd40 msgr2=0x7fe8840b46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.290+0000 7fe88a0c2640 1 --2- 172.21.15.83:0/1742974669 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe88415cd40 0x7fe8840b46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.290+0000 7fe88a0c2640 1 -- 172.21.15.83:0/1742974669 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe8700043b0 msgr2=0x7fe8840b5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.290+0000 7fe88a0c2640 1 --2- 172.21.15.83:0/1742974669 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe8700043b0 0x7fe8840b5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.290+0000 7fe88a0c2640 1 -- 172.21.15.83:0/1742974669 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe88415d1d0 con 0x7fe8840b60b0 2024-03-06T07:25:13.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.290+0000 7fe88a0c2640 1 --2- 172.21.15.83:0/1742974669 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8840b60b0 0x7fe8840b4c10 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fe87c007fd0 tx=0x7fe87c00eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:13.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.290+0000 7fe87a7fc640 1 -- 172.21.15.83:0/1742974669 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87c003970 con 0x7fe8840b60b0 2024-03-06T07:25:13.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.290+0000 7fe87a7fc640 1 -- 172.21.15.83:0/1742974669 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe87c018070 con 0x7fe8840b60b0 2024-03-06T07:25:13.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.290+0000 7fe88bb25640 1 -- 172.21.15.83:0/1742974669 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe88415d4b0 con 0x7fe8840b60b0 2024-03-06T07:25:13.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.290+0000 7fe87a7fc640 1 -- 172.21.15.83:0/1742974669 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87c003d50 con 0x7fe8840b60b0 2024-03-06T07:25:13.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.290+0000 7fe88bb25640 1 -- 172.21.15.83:0/1742974669 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe88415da70 con 0x7fe8840b60b0 2024-03-06T07:25:13.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.291+0000 7fe88bb25640 1 -- 172.21.15.83:0/1742974669 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe88404f330 con 0x7fe8840b60b0 2024-03-06T07:25:13.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.291+0000 7fe87a7fc640 1 -- 172.21.15.83:0/1742974669 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe87c0050d0 con 0x7fe8840b60b0 2024-03-06T07:25:13.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.292+0000 7fe87a7fc640 1 --2- 172.21.15.83:0/1742974669 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe86806b4d0 0x7fe86806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.292+0000 7fe87a7fc640 1 -- 172.21.15.83:0/1742974669 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe87c089e10 con 0x7fe8840b60b0 2024-03-06T07:25:13.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.292+0000 7fe8898c1640 1 --2- 172.21.15.83:0/1742974669 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe86806b4d0 0x7fe86806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.292+0000 7fe8898c1640 1 --2- 172.21.15.83:0/1742974669 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe86806b4d0 0x7fe86806d990 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe880000c00 tx=0x7fe880001090 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:13.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.296+0000 7fe87a7fc640 1 -- 172.21.15.83:0/1742974669 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe87c0586d0 con 0x7fe8840b60b0 2024-03-06T07:25:13.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.470+0000 7fe88bb25640 1 -- 172.21.15.83:0/1742974669 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe88409af00 con 0x7fe8840b60b0 2024-03-06T07:25:13.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.471+0000 7fe87a7fc640 1 -- 172.21.15.83:0/1742974669 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) v1 ==== 74+0+16693 (secure 0 0 0) 0x7fe87c05c770 con 0x7fe8840b60b0 2024-03-06T07:25:13.473 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:25:13.473 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":68,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:25:12.481367+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:10.701416+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:11.450797+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:13.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.473+0000 7fe88bb25640 1 -- 172.21.15.83:0/1742974669 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe86806b4d0 msgr2=0x7fe86806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.473+0000 7fe88bb25640 1 --2- 172.21.15.83:0/1742974669 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe86806b4d0 0x7fe86806d990 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe880000c00 tx=0x7fe880001090 comp rx=0 tx=0).stop 2024-03-06T07:25:13.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.473+0000 7fe88bb25640 1 -- 172.21.15.83:0/1742974669 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8840b60b0 msgr2=0x7fe8840b4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.473+0000 7fe88bb25640 1 --2- 172.21.15.83:0/1742974669 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8840b60b0 0x7fe8840b4c10 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fe87c007fd0 tx=0x7fe87c00eb90 comp rx=0 tx=0).stop 2024-03-06T07:25:13.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.473+0000 7fe88bb25640 1 -- 172.21.15.83:0/1742974669 shutdown_connections 2024-03-06T07:25:13.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.473+0000 7fe88bb25640 1 --2- 172.21.15.83:0/1742974669 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe86806b4d0 0x7fe86806d990 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.473+0000 7fe88bb25640 1 --2- 172.21.15.83:0/1742974669 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8840b60b0 0x7fe8840b4c10 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.473+0000 7fe88bb25640 1 --2- 172.21.15.83:0/1742974669 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe88415cd40 0x7fe8840b46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.473+0000 7fe88bb25640 1 --2- 172.21.15.83:0/1742974669 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe8700043b0 0x7fe8840b5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.473+0000 7fe88bb25640 1 -- 172.21.15.83:0/1742974669 >> 172.21.15.83:0/1742974669 conn(0x7fe88409a9f0 msgr2=0x7fe88406eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:13.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.473+0000 7fe88bb25640 1 -- 172.21.15.83:0/1742974669 shutdown_connections 2024-03-06T07:25:13.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.473+0000 7fe88bb25640 1 -- 172.21.15.83:0/1742974669 wait complete. 2024-03-06T07:25:13.486 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:13.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.584+0000 7f443b8df640 1 Processor -- start 2024-03-06T07:25:13.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.585+0000 7f443b8df640 1 -- start start 2024-03-06T07:25:13.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.585+0000 7f443b8df640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f443415dfe0 0x7f44340cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.585+0000 7f443b8df640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f44340cf830 0x7f44340d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.585+0000 7f443b8df640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44340b3430 con 0x7f44340d27f0 2024-03-06T07:25:13.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.585+0000 7f443b8df640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44340d2c50 con 0x7f44340cf830 2024-03-06T07:25:13.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.585+0000 7f443b8df640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44340b31e0 con 0x7f443415dfe0 2024-03-06T07:25:13.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.586+0000 7f4438e7a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f44340cf830 0x7f44340d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.586+0000 7f4438e7a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f44340cf830 0x7f44340d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35608/0 (socket says 172.21.15.83:35608) 2024-03-06T07:25:13.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.586+0000 7f4438e7a640 1 -- 172.21.15.83:0/913115386 learned_addr learned my addr 172.21.15.83:0/913115386 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:13.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.586+0000 7f4439e7c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f44340d27f0 0x7f44340d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48812/0 (socket says 172.21.15.83:48812) 2024-03-06T07:25:13.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.586+0000 7f443967b640 1 --2- 172.21.15.83:0/913115386 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f443415dfe0 0x7f44340cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.586+0000 7f4438e7a640 1 -- 172.21.15.83:0/913115386 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f443415dfe0 msgr2=0x7f44340cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:13.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.586+0000 7f4438e7a640 1 --2- 172.21.15.83:0/913115386 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f443415dfe0 0x7f44340cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.586+0000 7f4438e7a640 1 -- 172.21.15.83:0/913115386 >> v1:172.21.15.83:6789/0 conn(0x7f44340d27f0 legacy=0x7f44340d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.586+0000 7f4438e7a640 1 -- 172.21.15.83:0/913115386 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44340b3710 con 0x7f44340cf830 2024-03-06T07:25:13.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.586+0000 7f4433fff640 1 -- 172.21.15.83:0/913115386 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1257235741 0 0) 0x7f44340b3430 con 0x7f44340d27f0 2024-03-06T07:25:13.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.586+0000 7f443967b640 1 --2- 172.21.15.83:0/913115386 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f443415dfe0 0x7f44340cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:13.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.587+0000 7f4438e7a640 1 --2- 172.21.15.83:0/913115386 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44340cf830 0x7f44340d22b0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f4428009870 tx=0x7f442802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=638e27670542c723 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:13.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.587+0000 7f4433fff640 1 -- 172.21.15.83:0/913115386 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4428034070 con 0x7f44340cf830 2024-03-06T07:25:13.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.587+0000 7f4433fff640 1 -- 172.21.15.83:0/913115386 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4428030840 con 0x7f44340cf830 2024-03-06T07:25:13.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.587+0000 7f4433fff640 1 -- 172.21.15.83:0/913115386 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4428030c10 con 0x7f44340cf830 2024-03-06T07:25:13.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.587+0000 7f443b8df640 1 -- 172.21.15.83:0/913115386 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44340cf830 msgr2=0x7f44340d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.587+0000 7f443b8df640 1 --2- 172.21.15.83:0/913115386 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44340cf830 0x7f44340d22b0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f4428009870 tx=0x7f442802b980 comp rx=0 tx=0).stop 2024-03-06T07:25:13.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.587+0000 7f443b8df640 1 -- 172.21.15.83:0/913115386 shutdown_connections 2024-03-06T07:25:13.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.587+0000 7f443b8df640 1 --2- 172.21.15.83:0/913115386 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44340cf830 0x7f44340d22b0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.587+0000 7f443b8df640 1 --2- 172.21.15.83:0/913115386 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f443415dfe0 0x7f44340cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.587+0000 7f443b8df640 1 -- 172.21.15.83:0/913115386 >> 172.21.15.83:0/913115386 conn(0x7f44340d3cd0 msgr2=0x7f44340d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:13.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.587+0000 7f443b8df640 1 -- 172.21.15.83:0/913115386 shutdown_connections 2024-03-06T07:25:13.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.587+0000 7f443b8df640 1 -- 172.21.15.83:0/913115386 wait complete. 2024-03-06T07:25:13.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.588+0000 7f443b8df640 1 Processor -- start 2024-03-06T07:25:13.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.589+0000 7f443b8df640 1 -- start start 2024-03-06T07:25:13.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.589+0000 7f443b8df640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44340cf830 0x7f443414ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.589+0000 7f443b8df640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44340d27f0 0x7f44340cb620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.589+0000 7f443b8df640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f443415dfe0 0x7f44340cd810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.589+0000 7f443b8df640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44340d1c70 con 0x7f44340cf830 2024-03-06T07:25:13.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.589+0000 7f443b8df640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44340b3d60 con 0x7f443415dfe0 2024-03-06T07:25:13.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.589+0000 7f443b8df640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44340d1df0 con 0x7f44340d27f0 2024-03-06T07:25:13.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.589+0000 7f443967b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44340cf830 0x7f443414ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.589+0000 7f443967b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44340cf830 0x7f443414ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35620/0 (socket says 172.21.15.83:35620) 2024-03-06T07:25:13.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.589+0000 7f443967b640 1 -- 172.21.15.83:0/1372938341 learned_addr learned my addr 172.21.15.83:0/1372938341 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:13.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.589+0000 7f443967b640 1 -- 172.21.15.83:0/1372938341 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44340d27f0 msgr2=0x7f44340cb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.589+0000 7f443967b640 1 --2- 172.21.15.83:0/1372938341 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44340d27f0 0x7f44340cb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.589+0000 7f443967b640 1 -- 172.21.15.83:0/1372938341 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f443415dfe0 msgr2=0x7f44340cd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.589+0000 7f443967b640 1 --2- 172.21.15.83:0/1372938341 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f443415dfe0 0x7f44340cd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.590+0000 7f443967b640 1 -- 172.21.15.83:0/1372938341 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44340cdd50 con 0x7f44340cf830 2024-03-06T07:25:13.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.590+0000 7f443967b640 1 --2- 172.21.15.83:0/1372938341 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44340cf830 0x7f443414ca80 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f442c00c7f0 tx=0x7f442c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:13.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.590+0000 7f4431ffb640 1 -- 172.21.15.83:0/1372938341 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f442c015070 con 0x7f44340cf830 2024-03-06T07:25:13.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.590+0000 7f4431ffb640 1 -- 172.21.15.83:0/1372938341 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f442c007820 con 0x7f44340cf830 2024-03-06T07:25:13.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.590+0000 7f4431ffb640 1 -- 172.21.15.83:0/1372938341 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f442c007bd0 con 0x7f44340cf830 2024-03-06T07:25:13.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.590+0000 7f443b8df640 1 -- 172.21.15.83:0/1372938341 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44340cbd90 con 0x7f44340cf830 2024-03-06T07:25:13.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.590+0000 7f443b8df640 1 -- 172.21.15.83:0/1372938341 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44340cc350 con 0x7f44340cf830 2024-03-06T07:25:13.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.591+0000 7f443b8df640 1 -- 172.21.15.83:0/1372938341 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f443404f330 con 0x7f44340cf830 2024-03-06T07:25:13.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.591+0000 7f4431ffb640 1 -- 172.21.15.83:0/1372938341 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f442c018070 con 0x7f44340cf830 2024-03-06T07:25:13.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.592+0000 7f4431ffb640 1 --2- 172.21.15.83:0/1372938341 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f442006b4d0 0x7f442006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.592+0000 7f4431ffb640 1 -- 172.21.15.83:0/1372938341 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f442c08a390 con 0x7f44340cf830 2024-03-06T07:25:13.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.592+0000 7f4438e7a640 1 --2- 172.21.15.83:0/1372938341 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f442006b4d0 0x7f442006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.592+0000 7f4438e7a640 1 --2- 172.21.15.83:0/1372938341 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f442006b4d0 0x7f442006d990 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f4428036000 tx=0x7f442803b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:13.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.596+0000 7f4431ffb640 1 -- 172.21.15.83:0/1372938341 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f442c059180 con 0x7f44340cf830 2024-03-06T07:25:13.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.768+0000 7f443b8df640 1 -- 172.21.15.83:0/1372938341 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f44340b3360 con 0x7f44340cf830 2024-03-06T07:25:13.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.769+0000 7f4431ffb640 1 -- 172.21.15.83:0/1372938341 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) v1 ==== 74+0+16693 (secure 0 0 0) 0x7f442c058a80 con 0x7f44340cf830 2024-03-06T07:25:13.771 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:25:13.771 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":68,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:25:12.481367+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:10.701416+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:11.450797+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:13.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.771+0000 7f443b8df640 1 -- 172.21.15.83:0/1372938341 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f442006b4d0 msgr2=0x7f442006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.771+0000 7f443b8df640 1 --2- 172.21.15.83:0/1372938341 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f442006b4d0 0x7f442006d990 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f4428036000 tx=0x7f442803b040 comp rx=0 tx=0).stop 2024-03-06T07:25:13.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.771+0000 7f443b8df640 1 -- 172.21.15.83:0/1372938341 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44340cf830 msgr2=0x7f443414ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.771+0000 7f443b8df640 1 --2- 172.21.15.83:0/1372938341 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44340cf830 0x7f443414ca80 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f442c00c7f0 tx=0x7f442c00ccb0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.771+0000 7f443b8df640 1 -- 172.21.15.83:0/1372938341 shutdown_connections 2024-03-06T07:25:13.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.771+0000 7f443b8df640 1 --2- 172.21.15.83:0/1372938341 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f442006b4d0 0x7f442006d990 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.771+0000 7f443b8df640 1 --2- 172.21.15.83:0/1372938341 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f443415dfe0 0x7f44340cd810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.771+0000 7f443b8df640 1 --2- 172.21.15.83:0/1372938341 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44340d27f0 0x7f44340cb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.771+0000 7f443b8df640 1 --2- 172.21.15.83:0/1372938341 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44340cf830 0x7f443414ca80 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.771+0000 7f443b8df640 1 -- 172.21.15.83:0/1372938341 >> 172.21.15.83:0/1372938341 conn(0x7f44340d3cd0 msgr2=0x7f44340b0e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:13.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.771+0000 7f443b8df640 1 -- 172.21.15.83:0/1372938341 shutdown_connections 2024-03-06T07:25:13.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.771+0000 7f443b8df640 1 -- 172.21.15.83:0/1372938341 wait complete. 2024-03-06T07:25:13.784 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:25:13.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.885+0000 7f7ab4a88640 1 Processor -- start 2024-03-06T07:25:13.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.885+0000 7f7ab4a88640 1 -- start start 2024-03-06T07:25:13.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.886+0000 7f7ab4a88640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7ab00b1e40 0x7f7ab0168b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.886+0000 7f7ab4a88640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ab00d2730 con 0x7f7ab00d4860 2024-03-06T07:25:13.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.886+0000 7f7ab4a88640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ab01618b0 con 0x7f7ab01614d0 2024-03-06T07:25:13.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.886+0000 7f7ab4a88640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ab00d4cf0 con 0x7f7ab00b1e40 2024-03-06T07:25:13.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.886+0000 7f7aaed9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7ab00b1e40 0x7f7ab0168b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.887+0000 7f7aadd9b640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f7ab00d4860 0x7f7ab00d2020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48814/0 (socket says 172.21.15.83:48814) 2024-03-06T07:25:13.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.887+0000 7f7aadd9b640 1 -- 172.21.15.83:0/3182689604 learned_addr learned my addr 172.21.15.83:0/3182689604 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:25:13.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.887+0000 7f7aaed9d640 1 -- 172.21.15.83:0/3182689604 >> v1:172.21.15.183:6789/0 conn(0x7f7ab01614d0 legacy=0x7f7ab00d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.887+0000 7f7aaed9d640 1 -- 172.21.15.83:0/3182689604 >> v1:172.21.15.83:6789/0 conn(0x7f7ab00d4860 legacy=0x7f7ab00d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.887+0000 7f7aaed9d640 1 -- 172.21.15.83:0/3182689604 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ab00cf400 con 0x7f7ab00b1e40 2024-03-06T07:25:13.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.887+0000 7f7aad59a640 1 -- 172.21.15.83:0/3182689604 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4013600312 0 0) 0x7f7ab00d2730 con 0x7f7ab00d4860 2024-03-06T07:25:13.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.888+0000 7f7aaed9d640 1 --2- 172.21.15.83:0/3182689604 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ab00b1e40 0x7f7ab0168b60 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f7ab00cf5d0 tx=0x7f7aa002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52843ec30659489b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:13.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.888+0000 7f7aad59a640 1 -- 172.21.15.83:0/3182689604 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa0035020 con 0x7f7ab00b1e40 2024-03-06T07:25:13.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.888+0000 7f7aad59a640 1 -- 172.21.15.83:0/3182689604 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aa0031830 con 0x7f7ab00b1e40 2024-03-06T07:25:13.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.888+0000 7f7aad59a640 1 -- 172.21.15.83:0/3182689604 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa0031c00 con 0x7f7ab00b1e40 2024-03-06T07:25:13.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.888+0000 7f7ab4a88640 1 -- 172.21.15.83:0/3182689604 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ab00b1e40 msgr2=0x7f7ab0168b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.888+0000 7f7ab4a88640 1 --2- 172.21.15.83:0/3182689604 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ab00b1e40 0x7f7ab0168b60 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f7ab00cf5d0 tx=0x7f7aa002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.888+0000 7f7ab4a88640 1 -- 172.21.15.83:0/3182689604 shutdown_connections 2024-03-06T07:25:13.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.888+0000 7f7ab4a88640 1 --2- 172.21.15.83:0/3182689604 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ab00b1e40 0x7f7ab0168b60 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.888+0000 7f7ab4a88640 1 -- 172.21.15.83:0/3182689604 >> 172.21.15.83:0/3182689604 conn(0x7f7ab00d3d40 msgr2=0x7f7ab00d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:13.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.888+0000 7f7ab4a88640 1 -- 172.21.15.83:0/3182689604 shutdown_connections 2024-03-06T07:25:13.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.888+0000 7f7ab4a88640 1 -- 172.21.15.83:0/3182689604 wait complete. 2024-03-06T07:25:13.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.890+0000 7f7ab4a88640 1 Processor -- start 2024-03-06T07:25:13.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.890+0000 7f7ab4a88640 1 -- start start 2024-03-06T07:25:13.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.890+0000 7f7ab4a88640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7ab00b1e40 0x7f7ab00b6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.890+0000 7f7ab4a88640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7ab00d4860 0x7f7ab00b6e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.890+0000 7f7ab4a88640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ab01614d0 0x7f7ab01f73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.890+0000 7f7ab4a88640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab00cfc60 con 0x7f7ab01614d0 2024-03-06T07:25:13.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.890+0000 7f7ab4a88640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab00cf980 con 0x7f7ab00b1e40 2024-03-06T07:25:13.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.890+0000 7f7ab4a88640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab00b0910 con 0x7f7ab00d4860 2024-03-06T07:25:13.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.890+0000 7f7aaed9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ab01614d0 0x7f7ab01f73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.890+0000 7f7aaed9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ab01614d0 0x7f7ab01f73b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35632/0 (socket says 172.21.15.83:35632) 2024-03-06T07:25:13.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.890+0000 7f7aaed9d640 1 -- 172.21.15.83:0/833149476 learned_addr learned my addr 172.21.15.83:0/833149476 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:13.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.891+0000 7f7aae59c640 1 --2- 172.21.15.83:0/833149476 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7ab00b1e40 0x7f7ab00b6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.891+0000 7f7aadd9b640 1 --2- 172.21.15.83:0/833149476 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7ab00d4860 0x7f7ab00b6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.891+0000 7f7aaed9d640 1 -- 172.21.15.83:0/833149476 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7ab00d4860 msgr2=0x7f7ab00b6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.891+0000 7f7aaed9d640 1 --2- 172.21.15.83:0/833149476 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7ab00d4860 0x7f7ab00b6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.891+0000 7f7aaed9d640 1 -- 172.21.15.83:0/833149476 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7ab00b1e40 msgr2=0x7f7ab00b6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.891+0000 7f7aaed9d640 1 --2- 172.21.15.83:0/833149476 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7ab00b1e40 0x7f7ab00b6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.891+0000 7f7aaed9d640 1 -- 172.21.15.83:0/833149476 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ab014a8b0 con 0x7f7ab01614d0 2024-03-06T07:25:13.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.891+0000 7f7aae59c640 1 --2- 172.21.15.83:0/833149476 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7ab00b1e40 0x7f7ab00b6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:13.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.891+0000 7f7aaed9d640 1 --2- 172.21.15.83:0/833149476 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ab01614d0 0x7f7ab01f73b0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f7aa002b910 tx=0x7f7aa00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:13.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.891+0000 7f7aa6ffd640 1 -- 172.21.15.83:0/833149476 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa0040070 con 0x7f7ab01614d0 2024-03-06T07:25:13.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.891+0000 7f7aa6ffd640 1 -- 172.21.15.83:0/833149476 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aa0039940 con 0x7f7ab01614d0 2024-03-06T07:25:13.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.891+0000 7f7aa6ffd640 1 -- 172.21.15.83:0/833149476 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa0035040 con 0x7f7ab01614d0 2024-03-06T07:25:13.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.891+0000 7f7ab4a88640 1 -- 172.21.15.83:0/833149476 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ab014ab90 con 0x7f7ab01614d0 2024-03-06T07:25:13.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.891+0000 7f7ab4a88640 1 -- 172.21.15.83:0/833149476 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ab00b3750 con 0x7f7ab01614d0 2024-03-06T07:25:13.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.892+0000 7f7ab4a88640 1 -- 172.21.15.83:0/833149476 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ab00b3540 con 0x7f7ab01614d0 2024-03-06T07:25:13.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.893+0000 7f7aa6ffd640 1 -- 172.21.15.83:0/833149476 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7aa0038020 con 0x7f7ab01614d0 2024-03-06T07:25:13.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.893+0000 7f7aa6ffd640 1 --2- 172.21.15.83:0/833149476 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7a9006b4d0 0x7f7a9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.893+0000 7f7aa6ffd640 1 -- 172.21.15.83:0/833149476 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7aa00abb10 con 0x7f7ab01614d0 2024-03-06T07:25:13.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.893+0000 7f7aae59c640 1 --2- 172.21.15.83:0/833149476 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7a9006b4d0 0x7f7a9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.893+0000 7f7aae59c640 1 --2- 172.21.15.83:0/833149476 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7a9006b4d0 0x7f7a9006d990 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f7aa80012e0 tx=0x7f7aa8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:13.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:13.897+0000 7f7aa6ffd640 1 -- 172.21.15.83:0/833149476 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7aa007aa20 con 0x7f7ab01614d0 2024-03-06T07:25:14.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:14.133+0000 7f7ab4a88640 1 -- 172.21.15.83:0/833149476 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f7ab0068ff0 con 0x7f7ab01614d0 2024-03-06T07:25:14.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:14.979+0000 7f7aa6ffd640 1 -- 172.21.15.83:0/833149476 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v31) v1 ==== 118+0+0 (secure 0 0 0) 0x7f7aa007a320 con 0x7f7ab01614d0 2024-03-06T07:25:14.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:14.981+0000 7f7ab4a88640 1 -- 172.21.15.83:0/833149476 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7a9006b4d0 msgr2=0x7f7a9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:14.981+0000 7f7ab4a88640 1 --2- 172.21.15.83:0/833149476 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7a9006b4d0 0x7f7a9006d990 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f7aa80012e0 tx=0x7f7aa8009040 comp rx=0 tx=0).stop 2024-03-06T07:25:14.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:14.981+0000 7f7ab4a88640 1 -- 172.21.15.83:0/833149476 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ab01614d0 msgr2=0x7f7ab01f73b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:14.981+0000 7f7ab4a88640 1 --2- 172.21.15.83:0/833149476 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ab01614d0 0x7f7ab01f73b0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f7aa002b910 tx=0x7f7aa00046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:14.981+0000 7f7ab4a88640 1 -- 172.21.15.83:0/833149476 shutdown_connections 2024-03-06T07:25:14.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:14.981+0000 7f7ab4a88640 1 --2- 172.21.15.83:0/833149476 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7a9006b4d0 0x7f7a9006d990 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:14.981+0000 7f7ab4a88640 1 --2- 172.21.15.83:0/833149476 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ab01614d0 0x7f7ab01f73b0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:14.981+0000 7f7ab4a88640 1 --2- 172.21.15.83:0/833149476 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7ab00d4860 0x7f7ab00b6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:14.981+0000 7f7ab4a88640 1 --2- 172.21.15.83:0/833149476 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7ab00b1e40 0x7f7ab00b6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:14.981+0000 7f7ab4a88640 1 -- 172.21.15.83:0/833149476 >> 172.21.15.83:0/833149476 conn(0x7f7ab00d3d40 msgr2=0x7f7ab00d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:14.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:14.981+0000 7f7ab4a88640 1 -- 172.21.15.83:0/833149476 shutdown_connections 2024-03-06T07:25:14.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:14.981+0000 7f7ab4a88640 1 -- 172.21.15.83:0/833149476 wait complete. 2024-03-06T07:25:14.994 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:25:14.994 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:25:15.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.096+0000 7ff2a5a0e640 1 Processor -- start 2024-03-06T07:25:15.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.096+0000 7ff2a5a0e640 1 -- start start 2024-03-06T07:25:15.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.098+0000 7ff2a5a0e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff2a006fd10 0x7ff2a00cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.098+0000 7ff2a5a0e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2a00cd4b0 con 0x7ff2a00d0af0 2024-03-06T07:25:15.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.098+0000 7ff2a5a0e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2a00cd790 con 0x7ff2a015c050 2024-03-06T07:25:15.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.098+0000 7ff2a5a0e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2a00cda70 con 0x7ff2a006fd10 2024-03-06T07:25:15.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.098+0000 7ff297fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff2a006fd10 0x7ff2a00cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.098+0000 7ff297fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff2a006fd10 0x7ff2a00cc760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35642/0 (socket says 172.21.15.83:35642) 2024-03-06T07:25:15.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.098+0000 7ff297fff640 1 -- 172.21.15.83:0/4019072030 learned_addr learned my addr 172.21.15.83:0/4019072030 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:15.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.098+0000 7ff29f7fe640 1 --1- 172.21.15.83:0/4019072030 >> v1:172.21.15.83:6789/0 conn(0x7ff2a00d0af0 0x7ff2a00ccca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48818/0 (socket says 172.21.15.83:48818) 2024-03-06T07:25:15.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.098+0000 7ff297fff640 1 -- 172.21.15.83:0/4019072030 >> v1:172.21.15.183:6789/0 conn(0x7ff2a015c050 legacy=0x7ff2a006f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:15.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.098+0000 7ff297fff640 1 -- 172.21.15.83:0/4019072030 >> v1:172.21.15.83:6789/0 conn(0x7ff2a00d0af0 legacy=0x7ff2a00ccca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.098+0000 7ff297fff640 1 -- 172.21.15.83:0/4019072030 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a00cdd50 con 0x7ff2a006fd10 2024-03-06T07:25:15.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.099+0000 7ff297fff640 1 --2- 172.21.15.83:0/4019072030 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2a006fd10 0x7ff2a00cc760 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7ff288009870 tx=0x7ff28802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ce743b3b2811a85 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:15.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.099+0000 7ff29e7fc640 1 -- 172.21.15.83:0/4019072030 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff288035020 con 0x7ff2a006fd10 2024-03-06T07:25:15.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.099+0000 7ff29e7fc640 1 -- 172.21.15.83:0/4019072030 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff288031870 con 0x7ff2a006fd10 2024-03-06T07:25:15.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.099+0000 7ff29e7fc640 1 -- 172.21.15.83:0/4019072030 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff288031c40 con 0x7ff2a006fd10 2024-03-06T07:25:15.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.100+0000 7ff2a5a0e640 1 -- 172.21.15.83:0/4019072030 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2a006fd10 msgr2=0x7ff2a00cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.100+0000 7ff2a5a0e640 1 --2- 172.21.15.83:0/4019072030 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2a006fd10 0x7ff2a00cc760 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7ff288009870 tx=0x7ff28802b910 comp rx=0 tx=0).stop 2024-03-06T07:25:15.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.100+0000 7ff2a5a0e640 1 -- 172.21.15.83:0/4019072030 shutdown_connections 2024-03-06T07:25:15.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.100+0000 7ff2a5a0e640 1 --2- 172.21.15.83:0/4019072030 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2a006fd10 0x7ff2a00cc760 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.100+0000 7ff2a5a0e640 1 -- 172.21.15.83:0/4019072030 >> 172.21.15.83:0/4019072030 conn(0x7ff2a00b3800 msgr2=0x7ff2a00b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:15.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.100+0000 7ff2a5a0e640 1 -- 172.21.15.83:0/4019072030 shutdown_connections 2024-03-06T07:25:15.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.100+0000 7ff2a5a0e640 1 -- 172.21.15.83:0/4019072030 wait complete. 2024-03-06T07:25:15.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.101+0000 7ff2a5a0e640 1 Processor -- start 2024-03-06T07:25:15.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.101+0000 7ff2a5a0e640 1 -- start start 2024-03-06T07:25:15.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.102+0000 7ff2a5a0e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff2a006fd10 0x7ff2a009f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.102+0000 7ff2a5a0e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2a00d0af0 0x7ff2a01507c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.102+0000 7ff2a5a0e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff2a015c050 0x7ff2a00b48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.102+0000 7ff2a5a0e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a0166e50 con 0x7ff2a00d0af0 2024-03-06T07:25:15.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.102+0000 7ff2a5a0e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a0166cd0 con 0x7ff2a006fd10 2024-03-06T07:25:15.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.102+0000 7ff2a5a0e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a0166fd0 con 0x7ff2a015c050 2024-03-06T07:25:15.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.102+0000 7ff297fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2a00d0af0 0x7ff2a01507c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.102+0000 7ff297fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2a00d0af0 0x7ff2a01507c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35648/0 (socket says 172.21.15.83:35648) 2024-03-06T07:25:15.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.102+0000 7ff297fff640 1 -- 172.21.15.83:0/2332022038 learned_addr learned my addr 172.21.15.83:0/2332022038 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:15.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.102+0000 7ff29f7fe640 1 --2- 172.21.15.83:0/2332022038 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff2a015c050 0x7ff2a00b48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.102+0000 7ff29effd640 1 --2- 172.21.15.83:0/2332022038 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff2a006fd10 0x7ff2a009f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.102+0000 7ff297fff640 1 -- 172.21.15.83:0/2332022038 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff2a015c050 msgr2=0x7ff2a00b48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.102+0000 7ff297fff640 1 --2- 172.21.15.83:0/2332022038 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff2a015c050 0x7ff2a00b48f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.102+0000 7ff297fff640 1 -- 172.21.15.83:0/2332022038 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff2a006fd10 msgr2=0x7ff2a009f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.102+0000 7ff297fff640 1 --2- 172.21.15.83:0/2332022038 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff2a006fd10 0x7ff2a009f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.102+0000 7ff297fff640 1 -- 172.21.15.83:0/2332022038 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a01586f0 con 0x7ff2a00d0af0 2024-03-06T07:25:15.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.102+0000 7ff297fff640 1 --2- 172.21.15.83:0/2332022038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2a00d0af0 0x7ff2a01507c0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7ff288000c00 tx=0x7ff2880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:15.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.102+0000 7ff29cff9640 1 -- 172.21.15.83:0/2332022038 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff288040070 con 0x7ff2a00d0af0 2024-03-06T07:25:15.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.103+0000 7ff29cff9640 1 -- 172.21.15.83:0/2332022038 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff28803a420 con 0x7ff2a00d0af0 2024-03-06T07:25:15.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.103+0000 7ff29cff9640 1 -- 172.21.15.83:0/2332022038 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff288035040 con 0x7ff2a00d0af0 2024-03-06T07:25:15.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.103+0000 7ff2a5a0e640 1 -- 172.21.15.83:0/2332022038 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2a0158990 con 0x7ff2a00d0af0 2024-03-06T07:25:15.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.103+0000 7ff2a5a0e640 1 -- 172.21.15.83:0/2332022038 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2a0158f50 con 0x7ff2a00d0af0 2024-03-06T07:25:15.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.104+0000 7ff2a5a0e640 1 -- 172.21.15.83:0/2332022038 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2a0068ff0 con 0x7ff2a00d0af0 2024-03-06T07:25:15.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.104+0000 7ff29cff9640 1 -- 172.21.15.83:0/2332022038 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff28804b060 con 0x7ff2a00d0af0 2024-03-06T07:25:15.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.104+0000 7ff29cff9640 1 --2- 172.21.15.83:0/2332022038 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff28006b4d0 0x7ff28006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.105+0000 7ff29cff9640 1 -- 172.21.15.83:0/2332022038 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff2880ac0f0 con 0x7ff2a00d0af0 2024-03-06T07:25:15.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.105+0000 7ff29effd640 1 --2- 172.21.15.83:0/2332022038 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff28006b4d0 0x7ff28006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.105+0000 7ff29effd640 1 --2- 172.21.15.83:0/2332022038 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff28006b4d0 0x7ff28006d990 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7ff290003d90 tx=0x7ff290009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:15.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.113+0000 7ff29cff9640 1 -- 172.21.15.83:0/2332022038 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff28807af70 con 0x7ff2a00d0af0 2024-03-06T07:25:15.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:15.390+0000 7ff2a5a0e640 1 -- 172.21.15.83:0/2332022038 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7ff2a0070360 con 0x7ff2a00d0af0 2024-03-06T07:25:16.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.161+0000 7ff29cff9640 1 -- 172.21.15.83:0/2332022038 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v32) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff28807a870 con 0x7ff2a00d0af0 2024-03-06T07:25:16.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.163+0000 7ff2a5a0e640 1 -- 172.21.15.83:0/2332022038 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff28006b4d0 msgr2=0x7ff28006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.163+0000 7ff2a5a0e640 1 --2- 172.21.15.83:0/2332022038 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff28006b4d0 0x7ff28006d990 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7ff290003d90 tx=0x7ff290009040 comp rx=0 tx=0).stop 2024-03-06T07:25:16.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.163+0000 7ff2a5a0e640 1 -- 172.21.15.83:0/2332022038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2a00d0af0 msgr2=0x7ff2a01507c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.163+0000 7ff2a5a0e640 1 --2- 172.21.15.83:0/2332022038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2a00d0af0 0x7ff2a01507c0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7ff288000c00 tx=0x7ff2880046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.163+0000 7ff2a5a0e640 1 -- 172.21.15.83:0/2332022038 shutdown_connections 2024-03-06T07:25:16.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.163+0000 7ff2a5a0e640 1 --2- 172.21.15.83:0/2332022038 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff28006b4d0 0x7ff28006d990 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.163+0000 7ff2a5a0e640 1 --2- 172.21.15.83:0/2332022038 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff2a015c050 0x7ff2a00b48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.163+0000 7ff2a5a0e640 1 --2- 172.21.15.83:0/2332022038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2a00d0af0 0x7ff2a01507c0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.163+0000 7ff2a5a0e640 1 --2- 172.21.15.83:0/2332022038 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff2a006fd10 0x7ff2a009f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.163+0000 7ff2a5a0e640 1 -- 172.21.15.83:0/2332022038 >> 172.21.15.83:0/2332022038 conn(0x7ff2a00b3800 msgr2=0x7ff2a0157b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:16.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.163+0000 7ff2a5a0e640 1 -- 172.21.15.83:0/2332022038 shutdown_connections 2024-03-06T07:25:16.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.163+0000 7ff2a5a0e640 1 -- 172.21.15.83:0/2332022038 wait complete. 2024-03-06T07:25:16.176 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:25:16.177 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:16.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.276+0000 7f25d4355640 1 Processor -- start 2024-03-06T07:25:16.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.276+0000 7f25d4355640 1 -- start start 2024-03-06T07:25:16.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.277+0000 7f25d4355640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f25cc15dfe0 0x7f25cc0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.277+0000 7f25d4355640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f25cc0cf830 0x7f25cc0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.277+0000 7f25d4355640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25cc0b3430 con 0x7f25cc0d27f0 2024-03-06T07:25:16.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.277+0000 7f25d4355640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25cc0d2c50 con 0x7f25cc0cf830 2024-03-06T07:25:16.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.277+0000 7f25d4355640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25cc0b31e0 con 0x7f25cc15dfe0 2024-03-06T07:25:16.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.277+0000 7f25d18f0640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f25cc0cf830 0x7f25cc0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:16.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.277+0000 7f25d20f1640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f25cc15dfe0 0x7f25cc0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:16.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.277+0000 7f25d18f0640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f25cc0cf830 0x7f25cc0d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:38292/0 (socket says 172.21.15.83:38292) 2024-03-06T07:25:16.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.277+0000 7f25d28f2640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f25cc0d27f0 0x7f25cc0d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:57952/0 (socket says 172.21.15.83:57952) 2024-03-06T07:25:16.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.277+0000 7f25d20f1640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f25cc15dfe0 0x7f25cc0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:57636/0 (socket says 172.21.15.83:57636) 2024-03-06T07:25:16.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.277+0000 7f25d18f0640 1 -- 172.21.15.83:0/2675349500 learned_addr learned my addr 172.21.15.83:0/2675349500 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:16.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.278+0000 7f25d18f0640 1 -- 172.21.15.83:0/2675349500 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f25cc15dfe0 msgr2=0x7f25cc0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:16.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.278+0000 7f25d18f0640 1 --2- 172.21.15.83:0/2675349500 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f25cc15dfe0 0x7f25cc0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.278+0000 7f25d18f0640 1 -- 172.21.15.83:0/2675349500 >> v1:172.21.15.183:6789/0 conn(0x7f25cc0d27f0 legacy=0x7f25cc0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.278+0000 7f25d10ef640 1 -- 172.21.15.83:0/2675349500 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 747007882 0 0) 0x7f25cc0b3430 con 0x7f25cc0d27f0 2024-03-06T07:25:16.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.278+0000 7f25d18f0640 1 -- 172.21.15.83:0/2675349500 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25cc0b3710 con 0x7f25cc0cf830 2024-03-06T07:25:16.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.278+0000 7f25d20f1640 1 --2- 172.21.15.83:0/2675349500 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f25cc15dfe0 0x7f25cc0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:16.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.278+0000 7f25d18f0640 1 --2- 172.21.15.83:0/2675349500 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f25cc0cf830 0x7f25cc0d22b0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f25c8009870 tx=0x7f25c802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=131847b7f1b8386d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:16.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.279+0000 7f25d10ef640 1 -- 172.21.15.83:0/2675349500 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c8034070 con 0x7f25cc0cf830 2024-03-06T07:25:16.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.279+0000 7f25d10ef640 1 -- 172.21.15.83:0/2675349500 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25c8030840 con 0x7f25cc0cf830 2024-03-06T07:25:16.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.279+0000 7f25d10ef640 1 -- 172.21.15.83:0/2675349500 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c8030c10 con 0x7f25cc0cf830 2024-03-06T07:25:16.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.279+0000 7f25d4355640 1 -- 172.21.15.83:0/2675349500 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f25cc0cf830 msgr2=0x7f25cc0d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.279+0000 7f25d4355640 1 --2- 172.21.15.83:0/2675349500 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f25cc0cf830 0x7f25cc0d22b0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f25c8009870 tx=0x7f25c802b980 comp rx=0 tx=0).stop 2024-03-06T07:25:16.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.279+0000 7f25d4355640 1 -- 172.21.15.83:0/2675349500 shutdown_connections 2024-03-06T07:25:16.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.279+0000 7f25d4355640 1 --2- 172.21.15.83:0/2675349500 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f25cc0cf830 0x7f25cc0d22b0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.279+0000 7f25d4355640 1 --2- 172.21.15.83:0/2675349500 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f25cc15dfe0 0x7f25cc0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.279+0000 7f25d4355640 1 -- 172.21.15.83:0/2675349500 >> 172.21.15.83:0/2675349500 conn(0x7f25cc0d3cd0 msgr2=0x7f25cc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:16.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.279+0000 7f25d4355640 1 -- 172.21.15.83:0/2675349500 shutdown_connections 2024-03-06T07:25:16.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.279+0000 7f25d4355640 1 -- 172.21.15.83:0/2675349500 wait complete. 2024-03-06T07:25:16.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.280+0000 7f25d4355640 1 Processor -- start 2024-03-06T07:25:16.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.280+0000 7f25d4355640 1 -- start start 2024-03-06T07:25:16.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.281+0000 7f25d4355640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f25cc0cf830 0x7f25cc161180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.281+0000 7f25d4355640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f25cc0d27f0 0x7f25cc15e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.281+0000 7f25d4355640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f25cc15dfe0 0x7f25cc15ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.281+0000 7f25d4355640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25cc0d1c70 con 0x7f25cc15dfe0 2024-03-06T07:25:16.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.281+0000 7f25d4355640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25cc0b3d60 con 0x7f25cc0cf830 2024-03-06T07:25:16.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.281+0000 7f25d4355640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25cc0d1df0 con 0x7f25cc0d27f0 2024-03-06T07:25:16.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.281+0000 7f25d28f2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f25cc15dfe0 0x7f25cc15ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:16.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.281+0000 7f25d28f2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f25cc15dfe0 0x7f25cc15ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35664/0 (socket says 172.21.15.83:35664) 2024-03-06T07:25:16.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.281+0000 7f25d28f2640 1 -- 172.21.15.83:0/2954438914 learned_addr learned my addr 172.21.15.83:0/2954438914 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:16.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.281+0000 7f25d20f1640 1 --2- 172.21.15.83:0/2954438914 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f25cc0cf830 0x7f25cc161180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:16.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.281+0000 7f25d28f2640 1 -- 172.21.15.83:0/2954438914 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f25cc0d27f0 msgr2=0x7f25cc15e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.281+0000 7f25d28f2640 1 --2- 172.21.15.83:0/2954438914 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f25cc0d27f0 0x7f25cc15e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.281+0000 7f25d28f2640 1 -- 172.21.15.83:0/2954438914 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f25cc0cf830 msgr2=0x7f25cc161180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.281+0000 7f25d28f2640 1 --2- 172.21.15.83:0/2954438914 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f25cc0cf830 0x7f25cc161180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.281+0000 7f25d28f2640 1 -- 172.21.15.83:0/2954438914 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25cc0b6c90 con 0x7f25cc15dfe0 2024-03-06T07:25:16.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.282+0000 7f25d28f2640 1 --2- 172.21.15.83:0/2954438914 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f25cc15dfe0 0x7f25cc15ed10 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f25c400c7a0 tx=0x7f25c400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:16.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.282+0000 7f25beffd640 1 -- 172.21.15.83:0/2954438914 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c4018070 con 0x7f25cc15dfe0 2024-03-06T07:25:16.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.282+0000 7f25beffd640 1 -- 172.21.15.83:0/2954438914 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25c400a640 con 0x7f25cc15dfe0 2024-03-06T07:25:16.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.282+0000 7f25d4355640 1 -- 172.21.15.83:0/2954438914 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25cc0cc360 con 0x7f25cc15dfe0 2024-03-06T07:25:16.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.282+0000 7f25beffd640 1 -- 172.21.15.83:0/2954438914 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c400aab0 con 0x7f25cc15dfe0 2024-03-06T07:25:16.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.282+0000 7f25d4355640 1 -- 172.21.15.83:0/2954438914 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25cc0cc920 con 0x7f25cc15dfe0 2024-03-06T07:25:16.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.283+0000 7f25d4355640 1 -- 172.21.15.83:0/2954438914 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25cc04f330 con 0x7f25cc15dfe0 2024-03-06T07:25:16.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.284+0000 7f25beffd640 1 -- 172.21.15.83:0/2954438914 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f25c400ac50 con 0x7f25cc15dfe0 2024-03-06T07:25:16.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.284+0000 7f25beffd640 1 --2- 172.21.15.83:0/2954438914 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f25b006b4d0 0x7f25b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.284+0000 7f25beffd640 1 -- 172.21.15.83:0/2954438914 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f25c4089930 con 0x7f25cc15dfe0 2024-03-06T07:25:16.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.284+0000 7f25d20f1640 1 --2- 172.21.15.83:0/2954438914 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f25b006b4d0 0x7f25b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:16.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.285+0000 7f25d20f1640 1 --2- 172.21.15.83:0/2954438914 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f25b006b4d0 0x7f25b006d990 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f25cc1603b0 tx=0x7f25c0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:16.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.289+0000 7f25beffd640 1 -- 172.21.15.83:0/2954438914 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f25c4058730 con 0x7f25cc15dfe0 2024-03-06T07:25:16.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.504+0000 7f25d4355640 1 -- 172.21.15.83:0/2954438914 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25cc15d490 con 0x7f25cc15dfe0 2024-03-06T07:25:16.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.505+0000 7f25beffd640 1 -- 172.21.15.83:0/2954438914 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f25c4058030 con 0x7f25cc15dfe0 2024-03-06T07:25:16.507 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:25:16.507 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":32,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6010,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/2142942385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2142942385},{"type":"v1","addr":"172.21.15.183:6834","nonce":2142942385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":6013,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3742438064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":3742438064},{"type":"v1","addr":"172.21.15.183:6835","nonce":3742438064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":6540,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/1549894851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1549894851},{"type":"v1","addr":"172.21.15.83:6836","nonce":1549894851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:12.481670+0000","modified":"2024-03-06T07:25:15.392072+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6543},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6543":{"gid":6543,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/602775648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":602775648},{"type":"v1","addr":"172.21.15.83:6837","nonce":602775648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6543,"qdb_cluster":[6543]},"id":5}]} 2024-03-06T07:25:16.507 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 32 2024-03-06T07:25:16.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.507+0000 7f25d4355640 1 -- 172.21.15.83:0/2954438914 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f25b006b4d0 msgr2=0x7f25b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.507+0000 7f25d4355640 1 --2- 172.21.15.83:0/2954438914 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f25b006b4d0 0x7f25b006d990 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f25cc1603b0 tx=0x7f25c0008040 comp rx=0 tx=0).stop 2024-03-06T07:25:16.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.507+0000 7f25d4355640 1 -- 172.21.15.83:0/2954438914 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f25cc15dfe0 msgr2=0x7f25cc15ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.507+0000 7f25d4355640 1 --2- 172.21.15.83:0/2954438914 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f25cc15dfe0 0x7f25cc15ed10 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f25c400c7a0 tx=0x7f25c400cc60 comp rx=0 tx=0).stop 2024-03-06T07:25:16.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.507+0000 7f25d4355640 1 -- 172.21.15.83:0/2954438914 shutdown_connections 2024-03-06T07:25:16.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.507+0000 7f25d4355640 1 --2- 172.21.15.83:0/2954438914 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f25b006b4d0 0x7f25b006d990 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.507+0000 7f25d4355640 1 --2- 172.21.15.83:0/2954438914 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f25cc15dfe0 0x7f25cc15ed10 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.507+0000 7f25d4355640 1 --2- 172.21.15.83:0/2954438914 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f25cc0d27f0 0x7f25cc15e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.507+0000 7f25d4355640 1 --2- 172.21.15.83:0/2954438914 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f25cc0cf830 0x7f25cc161180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.507+0000 7f25d4355640 1 -- 172.21.15.83:0/2954438914 >> 172.21.15.83:0/2954438914 conn(0x7f25cc0d3cd0 msgr2=0x7f25cc06bb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:16.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.507+0000 7f25d4355640 1 -- 172.21.15.83:0/2954438914 shutdown_connections 2024-03-06T07:25:16.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.507+0000 7f25d4355640 1 -- 172.21.15.83:0/2954438914 wait complete. 2024-03-06T07:25:16.523 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:16.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.621+0000 7f91e7842640 1 Processor -- start 2024-03-06T07:25:16.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.622+0000 7f91e7842640 1 -- start start 2024-03-06T07:25:16.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.622+0000 7f91e7842640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f91e015dfe0 0x7f91e00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.622+0000 7f91e7842640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f91e00cf830 0x7f91e00d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.622+0000 7f91e7842640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f91e00d27f0 0x7f91e00d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.622+0000 7f91e7842640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91e00d2c50 con 0x7f91e00cf830 2024-03-06T07:25:16.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.622+0000 7f91e7842640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91e00b31e0 con 0x7f91e00d27f0 2024-03-06T07:25:16.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.622+0000 7f91e7842640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91e00b3d60 con 0x7f91e015dfe0 2024-03-06T07:25:16.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.622+0000 7f91e4ddd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f91e00cf830 0x7f91e00d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:16.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.622+0000 7f91e4ddd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f91e00cf830 0x7f91e00d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41340/0 (socket says 172.21.15.83:41340) 2024-03-06T07:25:16.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.622+0000 7f91e4ddd640 1 -- 172.21.15.83:0/2688384268 learned_addr learned my addr 172.21.15.83:0/2688384268 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:16.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.623+0000 7f91e55de640 1 --2- 172.21.15.83:0/2688384268 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f91e015dfe0 0x7f91e00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:16.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.623+0000 7f91e5ddf640 1 --2- 172.21.15.83:0/2688384268 >> v2:172.21.15.183:3300/0 conn(0x7f91e00d27f0 0x7f91e00d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:16.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.623+0000 7f91e4ddd640 1 -- 172.21.15.83:0/2688384268 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f91e015dfe0 msgr2=0x7f91e00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:16.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.623+0000 7f91e4ddd640 1 --2- 172.21.15.83:0/2688384268 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f91e015dfe0 0x7f91e00cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.623+0000 7f91e4ddd640 1 -- 172.21.15.83:0/2688384268 >> v2:172.21.15.183:3300/0 conn(0x7f91e00d27f0 msgr2=0x7f91e00d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:16.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.623+0000 7f91e4ddd640 1 --2- 172.21.15.83:0/2688384268 >> v2:172.21.15.183:3300/0 conn(0x7f91e00d27f0 0x7f91e00d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.623+0000 7f91e4ddd640 1 -- 172.21.15.83:0/2688384268 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e00b3430 con 0x7f91e00cf830 2024-03-06T07:25:16.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.624+0000 7f91e4ddd640 1 --2- 172.21.15.83:0/2688384268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f91e00cf830 0x7f91e00d22b0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f91e0067ad0 tx=0x7f91dc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93a05eb1bcd2d86d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:16.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.624+0000 7f91d7fff640 1 -- 172.21.15.83:0/2688384268 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91dc035020 con 0x7f91e00cf830 2024-03-06T07:25:16.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.624+0000 7f91d7fff640 1 -- 172.21.15.83:0/2688384268 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91dc031870 con 0x7f91e00cf830 2024-03-06T07:25:16.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.624+0000 7f91d7fff640 1 -- 172.21.15.83:0/2688384268 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91dc031c40 con 0x7f91e00cf830 2024-03-06T07:25:16.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.624+0000 7f91e7842640 1 -- 172.21.15.83:0/2688384268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f91e00cf830 msgr2=0x7f91e00d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.624+0000 7f91e7842640 1 --2- 172.21.15.83:0/2688384268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f91e00cf830 0x7f91e00d22b0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f91e0067ad0 tx=0x7f91dc02b910 comp rx=0 tx=0).stop 2024-03-06T07:25:16.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.624+0000 7f91e7842640 1 -- 172.21.15.83:0/2688384268 shutdown_connections 2024-03-06T07:25:16.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.624+0000 7f91e7842640 1 --2- 172.21.15.83:0/2688384268 >> v2:172.21.15.183:3300/0 conn(0x7f91e00d27f0 0x7f91e00d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.624+0000 7f91e7842640 1 --2- 172.21.15.83:0/2688384268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f91e00cf830 0x7f91e00d22b0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.624+0000 7f91e7842640 1 --2- 172.21.15.83:0/2688384268 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f91e015dfe0 0x7f91e00cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.624+0000 7f91e7842640 1 -- 172.21.15.83:0/2688384268 >> 172.21.15.83:0/2688384268 conn(0x7f91e00d3cd0 msgr2=0x7f91e00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:16.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.625+0000 7f91e7842640 1 -- 172.21.15.83:0/2688384268 shutdown_connections 2024-03-06T07:25:16.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.625+0000 7f91e7842640 1 -- 172.21.15.83:0/2688384268 wait complete. 2024-03-06T07:25:16.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.626+0000 7f91e7842640 1 Processor -- start 2024-03-06T07:25:16.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.626+0000 7f91e7842640 1 -- start start 2024-03-06T07:25:16.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.627+0000 7f91e7842640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f91e00cf830 0x7f91e0158560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.627+0000 7f91e7842640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f91e00d27f0 0x7f91e015ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.627+0000 7f91e7842640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f91e015dfe0 0x7f91e00b7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.627+0000 7f91e7842640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e00d1c70 con 0x7f91e00d27f0 2024-03-06T07:25:16.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.627+0000 7f91e7842640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e00b39f0 con 0x7f91e00cf830 2024-03-06T07:25:16.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.627+0000 7f91e7842640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91e00d1df0 con 0x7f91e015dfe0 2024-03-06T07:25:16.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.627+0000 7f91e4ddd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f91e00d27f0 0x7f91e015ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:16.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.627+0000 7f91e4ddd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f91e00d27f0 0x7f91e015ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41352/0 (socket says 172.21.15.83:41352) 2024-03-06T07:25:16.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.627+0000 7f91e4ddd640 1 -- 172.21.15.83:0/1062978454 learned_addr learned my addr 172.21.15.83:0/1062978454 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:16.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.628+0000 7f91e55de640 1 --2- 172.21.15.83:0/1062978454 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f91e00cf830 0x7f91e0158560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:16.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.628+0000 7f91e5ddf640 1 --2- 172.21.15.83:0/1062978454 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f91e015dfe0 0x7f91e00b7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:16.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.628+0000 7f91e4ddd640 1 -- 172.21.15.83:0/1062978454 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f91e015dfe0 msgr2=0x7f91e00b7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.628+0000 7f91e4ddd640 1 --2- 172.21.15.83:0/1062978454 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f91e015dfe0 0x7f91e00b7e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.628+0000 7f91e4ddd640 1 -- 172.21.15.83:0/1062978454 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f91e00cf830 msgr2=0x7f91e0158560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.628+0000 7f91e4ddd640 1 --2- 172.21.15.83:0/1062978454 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f91e00cf830 0x7f91e0158560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.628+0000 7f91e4ddd640 1 -- 172.21.15.83:0/1062978454 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e0158aa0 con 0x7f91e00d27f0 2024-03-06T07:25:16.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.628+0000 7f91e55de640 1 --2- 172.21.15.83:0/1062978454 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f91e00cf830 0x7f91e0158560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:16.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.628+0000 7f91e4ddd640 1 --2- 172.21.15.83:0/1062978454 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f91e00d27f0 0x7f91e015ee10 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f91dc02b950 tx=0x7f91dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:16.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.628+0000 7f91d5ffb640 1 -- 172.21.15.83:0/1062978454 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91dc040070 con 0x7f91e00d27f0 2024-03-06T07:25:16.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.628+0000 7f91d5ffb640 1 -- 172.21.15.83:0/1062978454 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91dc030d00 con 0x7f91e00d27f0 2024-03-06T07:25:16.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.628+0000 7f91d5ffb640 1 -- 172.21.15.83:0/1062978454 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91dc035040 con 0x7f91e00d27f0 2024-03-06T07:25:16.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.628+0000 7f91e7842640 1 -- 172.21.15.83:0/1062978454 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91e015f350 con 0x7f91e00d27f0 2024-03-06T07:25:16.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.628+0000 7f91e7842640 1 -- 172.21.15.83:0/1062978454 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91e00b7040 con 0x7f91e00d27f0 2024-03-06T07:25:16.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.629+0000 7f91e7842640 1 -- 172.21.15.83:0/1062978454 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91e00b5200 con 0x7f91e00d27f0 2024-03-06T07:25:16.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.630+0000 7f91d5ffb640 1 -- 172.21.15.83:0/1062978454 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f91dc04b060 con 0x7f91e00d27f0 2024-03-06T07:25:16.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.630+0000 7f91d5ffb640 1 --2- 172.21.15.83:0/1062978454 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f91c406b4d0 0x7f91c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.630+0000 7f91d5ffb640 1 -- 172.21.15.83:0/1062978454 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f91dc0ac000 con 0x7f91e00d27f0 2024-03-06T07:25:16.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.630+0000 7f91e55de640 1 --2- 172.21.15.83:0/1062978454 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f91c406b4d0 0x7f91c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:16.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.631+0000 7f91e55de640 1 --2- 172.21.15.83:0/1062978454 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f91c406b4d0 0x7f91c406d990 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f91d0006fd0 tx=0x7f91d0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:16.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.635+0000 7f91d5ffb640 1 -- 172.21.15.83:0/1062978454 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f91e00b5200 con 0x7f91e00d27f0 2024-03-06T07:25:16.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.945+0000 7f91e7842640 1 -- 172.21.15.83:0/1062978454 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f91e00b0af0 con 0x7f91e00d27f0 2024-03-06T07:25:16.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.946+0000 7f91d5ffb640 1 -- 172.21.15.83:0/1062978454 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+16680 (secure 0 0 0) 0x7f91dc07ae80 con 0x7f91e00d27f0 2024-03-06T07:25:16.949 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:25:16.949 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":69,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:25:14.149032+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:10.701416+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:11.450797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:16.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.949+0000 7f91e7842640 1 -- 172.21.15.83:0/1062978454 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f91c406b4d0 msgr2=0x7f91c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.949+0000 7f91e7842640 1 --2- 172.21.15.83:0/1062978454 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f91c406b4d0 0x7f91c406d990 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f91d0006fd0 tx=0x7f91d0008040 comp rx=0 tx=0).stop 2024-03-06T07:25:16.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.949+0000 7f91e7842640 1 -- 172.21.15.83:0/1062978454 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f91e00d27f0 msgr2=0x7f91e015ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.949+0000 7f91e7842640 1 --2- 172.21.15.83:0/1062978454 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f91e00d27f0 0x7f91e015ee10 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f91dc02b950 tx=0x7f91dc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.949+0000 7f91e7842640 1 -- 172.21.15.83:0/1062978454 shutdown_connections 2024-03-06T07:25:16.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.949+0000 7f91e7842640 1 --2- 172.21.15.83:0/1062978454 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f91c406b4d0 0x7f91c406d990 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.949+0000 7f91e7842640 1 --2- 172.21.15.83:0/1062978454 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f91e015dfe0 0x7f91e00b7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.949+0000 7f91e7842640 1 --2- 172.21.15.83:0/1062978454 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f91e00d27f0 0x7f91e015ee10 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.949+0000 7f91e7842640 1 --2- 172.21.15.83:0/1062978454 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f91e00cf830 0x7f91e0158560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.949+0000 7f91e7842640 1 -- 172.21.15.83:0/1062978454 >> 172.21.15.83:0/1062978454 conn(0x7f91e00d3cd0 msgr2=0x7f91e006cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:16.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.949+0000 7f91e7842640 1 -- 172.21.15.83:0/1062978454 shutdown_connections 2024-03-06T07:25:16.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:16.949+0000 7f91e7842640 1 -- 172.21.15.83:0/1062978454 wait complete. 2024-03-06T07:25:16.966 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:25:16.966 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:17.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.065+0000 7f7de50ed640 1 Processor -- start 2024-03-06T07:25:17.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.065+0000 7f7de50ed640 1 -- start start 2024-03-06T07:25:17.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.066+0000 7f7de50ed640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f7de00d22b0 0x7f7de00d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.066+0000 7f7de50ed640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7de00b3430 con 0x7f7de015dfe0 2024-03-06T07:25:17.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.066+0000 7f7de50ed640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7de00b3710 con 0x7f7de00b41f0 2024-03-06T07:25:17.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.066+0000 7f7de50ed640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7de00d2c50 con 0x7f7de00d22b0 2024-03-06T07:25:17.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.066+0000 7f7dded9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f7de015dfe0 0x7f7de00cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:40600/0 (socket says 172.21.15.83:40600) 2024-03-06T07:25:17.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.066+0000 7f7dded9d640 1 -- 172.21.15.83:0/373796159 learned_addr learned my addr 172.21.15.83:0/373796159 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:25:17.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.067+0000 7f7dd659c640 1 --2- 172.21.15.83:0/373796159 >> v2:172.21.15.183:3300/0 conn(0x7f7de00d22b0 0x7f7de00d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.067+0000 7f7dde59c640 1 -- 172.21.15.83:0/373796159 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3606281851 0 0) 0x7f7de00b3430 con 0x7f7de015dfe0 2024-03-06T07:25:17.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.067+0000 7f7dde59c640 1 -- 172.21.15.83:0/373796159 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dcc003520 con 0x7f7de015dfe0 2024-03-06T07:25:17.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.067+0000 7f7dde59c640 1 -- 172.21.15.83:0/373796159 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1870940577 0 0) 0x7f7dcc003520 con 0x7f7de015dfe0 2024-03-06T07:25:17.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.067+0000 7f7dde59c640 1 -- 172.21.15.83:0/373796159 >> v2:172.21.15.183:3300/0 conn(0x7f7de00d22b0 msgr2=0x7f7de00d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:17.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.067+0000 7f7dde59c640 1 --2- 172.21.15.83:0/373796159 >> v2:172.21.15.183:3300/0 conn(0x7f7de00d22b0 0x7f7de00d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.067+0000 7f7dde59c640 1 -- 172.21.15.83:0/373796159 >> v1:172.21.15.183:6789/0 conn(0x7f7de00b41f0 legacy=0x7f7de00b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:17.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.067+0000 7f7dde59c640 1 -- 172.21.15.83:0/373796159 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7de00b39f0 con 0x7f7de015dfe0 2024-03-06T07:25:17.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.067+0000 7f7dde59c640 1 -- 172.21.15.83:0/373796159 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f7dd0001670 con 0x7f7de015dfe0 2024-03-06T07:25:17.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.067+0000 7f7dde59c640 1 -- 172.21.15.83:0/373796159 >> v1:172.21.15.83:6789/0 conn(0x7f7de015dfe0 legacy=0x7f7de00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:17.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.067+0000 7f7dde59c640 1 --2- 172.21.15.83:0/373796159 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7dcc0039b0 0x7f7dcc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.067+0000 7f7dde59c640 1 --2- 172.21.15.83:0/373796159 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7dcc004800 0x7f7dcc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.067+0000 7f7dde59c640 1 --2- 172.21.15.83:0/373796159 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7dcc0051c0 0x7f7dcc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.067+0000 7f7dde59c640 1 -- 172.21.15.83:0/373796159 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dcc00c030 con 0x7f7dcc0051c0 2024-03-06T07:25:17.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.067+0000 7f7dde59c640 1 -- 172.21.15.83:0/373796159 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dcc00c1e0 con 0x7f7dcc0039b0 2024-03-06T07:25:17.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.067+0000 7f7dde59c640 1 -- 172.21.15.83:0/373796159 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dcc00c3e0 con 0x7f7dcc004800 2024-03-06T07:25:17.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.067+0000 7f7ddf59e640 1 --2- 172.21.15.83:0/373796159 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7dcc0051c0 0x7f7dcc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.067+0000 7f7dded9d640 1 --2- 172.21.15.83:0/373796159 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7dcc0039b0 0x7f7dcc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.067+0000 7f7dd659c640 1 --2- 172.21.15.83:0/373796159 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7dcc004800 0x7f7dcc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.067+0000 7f7ddf59e640 1 -- 172.21.15.83:0/373796159 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7dcc004800 msgr2=0x7f7dcc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:17.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.067+0000 7f7ddf59e640 1 --2- 172.21.15.83:0/373796159 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7dcc004800 0x7f7dcc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.067+0000 7f7ddf59e640 1 -- 172.21.15.83:0/373796159 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7dcc0039b0 msgr2=0x7f7dcc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:17.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.067+0000 7f7ddf59e640 1 --2- 172.21.15.83:0/373796159 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7dcc0039b0 0x7f7dcc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.067+0000 7f7ddf59e640 1 -- 172.21.15.83:0/373796159 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dcc003520 con 0x7f7dcc0051c0 2024-03-06T07:25:17.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.068+0000 7f7ddf59e640 1 -- 172.21.15.83:0/373796159 reap_dead start 2024-03-06T07:25:17.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.068+0000 7f7ddf59e640 1 --2- 172.21.15.83:0/373796159 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7dcc0051c0 0x7f7dcc00b9c0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f7dd8001e50 tx=0x7f7dcc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1ce72270a19b7d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:17.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.068+0000 7f7dde59c640 1 -- 172.21.15.83:0/373796159 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd8001a70 con 0x7f7dcc0051c0 2024-03-06T07:25:17.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.068+0000 7f7dde59c640 1 -- 172.21.15.83:0/373796159 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dd8003bb0 con 0x7f7dcc0051c0 2024-03-06T07:25:17.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.068+0000 7f7dde59c640 1 -- 172.21.15.83:0/373796159 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd80050d0 con 0x7f7dcc0051c0 2024-03-06T07:25:17.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.069+0000 7f7de50ed640 1 -- 172.21.15.83:0/373796159 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7dcc0051c0 msgr2=0x7f7dcc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:17.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.069+0000 7f7de50ed640 1 --2- 172.21.15.83:0/373796159 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7dcc0051c0 0x7f7dcc00b9c0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f7dd8001e50 tx=0x7f7dcc005830 comp rx=0 tx=0).stop 2024-03-06T07:25:17.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.069+0000 7f7de50ed640 1 -- 172.21.15.83:0/373796159 shutdown_connections 2024-03-06T07:25:17.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.069+0000 7f7de50ed640 1 --2- 172.21.15.83:0/373796159 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7dcc0051c0 0x7f7dcc00b9c0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.069+0000 7f7de50ed640 1 -- 172.21.15.83:0/373796159 >> 172.21.15.83:0/373796159 conn(0x7f7de00d3cd0 msgr2=0x7f7de00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:17.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.069+0000 7f7de50ed640 1 -- 172.21.15.83:0/373796159 shutdown_connections 2024-03-06T07:25:17.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.069+0000 7f7de50ed640 1 -- 172.21.15.83:0/373796159 wait complete. 2024-03-06T07:25:17.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.070+0000 7f7de50ed640 1 Processor -- start 2024-03-06T07:25:17.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.070+0000 7f7de50ed640 1 -- start start 2024-03-06T07:25:17.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.071+0000 7f7de50ed640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7dcc0051c0 0x7f7de01f3410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.071+0000 7f7de50ed640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7de00cc8f0 0x7f7de0156560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.071+0000 7f7de50ed640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7de015b650 0x7f7de015bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.071+0000 7f7de50ed640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de00b3d60 con 0x7f7de015b650 2024-03-06T07:25:17.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.071+0000 7f7de50ed640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de00b31e0 con 0x7f7dcc0051c0 2024-03-06T07:25:17.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.071+0000 7f7de50ed640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7de015e3c0 con 0x7f7de00cc8f0 2024-03-06T07:25:17.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.071+0000 7f7ddf59e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7de015b650 0x7f7de015bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.071+0000 7f7ddf59e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7de015b650 0x7f7de015bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41384/0 (socket says 172.21.15.83:41384) 2024-03-06T07:25:17.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.071+0000 7f7ddf59e640 1 -- 172.21.15.83:0/1304340674 learned_addr learned my addr 172.21.15.83:0/1304340674 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:17.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.071+0000 7f7dd659c640 1 --2- 172.21.15.83:0/1304340674 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7de00cc8f0 0x7f7de0156560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.071+0000 7f7dded9d640 1 --2- 172.21.15.83:0/1304340674 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7dcc0051c0 0x7f7de01f3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.071+0000 7f7ddf59e640 1 -- 172.21.15.83:0/1304340674 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7de00cc8f0 msgr2=0x7f7de0156560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:17.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.071+0000 7f7ddf59e640 1 --2- 172.21.15.83:0/1304340674 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7de00cc8f0 0x7f7de0156560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.071+0000 7f7ddf59e640 1 -- 172.21.15.83:0/1304340674 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7dcc0051c0 msgr2=0x7f7de01f3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:17.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.071+0000 7f7ddf59e640 1 --2- 172.21.15.83:0/1304340674 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7dcc0051c0 0x7f7de01f3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.071+0000 7f7ddf59e640 1 -- 172.21.15.83:0/1304340674 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7de00cce90 con 0x7f7de015b650 2024-03-06T07:25:17.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.071+0000 7f7dded9d640 1 --2- 172.21.15.83:0/1304340674 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7dcc0051c0 0x7f7de01f3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:17.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.071+0000 7f7dd659c640 1 --2- 172.21.15.83:0/1304340674 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7de00cc8f0 0x7f7de0156560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:17.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.072+0000 7f7ddf59e640 1 --2- 172.21.15.83:0/1304340674 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7de015b650 0x7f7de015bad0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f7dd802c060 tx=0x7f7dd80014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:17.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.072+0000 7f7dd7fff640 1 -- 172.21.15.83:0/1304340674 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd8006d60 con 0x7f7de015b650 2024-03-06T07:25:17.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.072+0000 7f7dd7fff640 1 -- 172.21.15.83:0/1304340674 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dd8007030 con 0x7f7de015b650 2024-03-06T07:25:17.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.072+0000 7f7de50ed640 1 -- 172.21.15.83:0/1304340674 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7de015eb60 con 0x7f7de015b650 2024-03-06T07:25:17.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.072+0000 7f7dd7fff640 1 -- 172.21.15.83:0/1304340674 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd80073e0 con 0x7f7de015b650 2024-03-06T07:25:17.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.072+0000 7f7de50ed640 1 -- 172.21.15.83:0/1304340674 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7de01f3950 con 0x7f7de015b650 2024-03-06T07:25:17.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.073+0000 7f7de50ed640 1 -- 172.21.15.83:0/1304340674 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7de004f330 con 0x7f7de015b650 2024-03-06T07:25:17.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.073+0000 7f7dd7fff640 1 -- 172.21.15.83:0/1304340674 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7dd8007580 con 0x7f7de015b650 2024-03-06T07:25:17.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.074+0000 7f7dd7fff640 1 --2- 172.21.15.83:0/1304340674 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7dc006b500 0x7f7dc006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.074+0000 7f7dd7fff640 1 -- 172.21.15.83:0/1304340674 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7dd80ac2f0 con 0x7f7de015b650 2024-03-06T07:25:17.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.074+0000 7f7dded9d640 1 --2- 172.21.15.83:0/1304340674 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7dc006b500 0x7f7dc006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.074+0000 7f7dded9d640 1 --2- 172.21.15.83:0/1304340674 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7dc006b500 0x7f7dc006d9c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f7dd0007cb0 tx=0x7f7dd0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:17.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.082+0000 7f7dd7fff640 1 -- 172.21.15.83:0/1304340674 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7dd8033030 con 0x7f7de015b650 2024-03-06T07:25:17.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.313+0000 7f7de50ed640 1 -- 172.21.15.83:0/1304340674 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7de01e8940 con 0x7f7dc006b500 2024-03-06T07:25:17.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.324+0000 7f7dd7fff640 1 -- 172.21.15.83:0/1304340674 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252067 (secure 0 0 0) 0x7f7de01e8940 con 0x7f7dc006b500 2024-03-06T07:25:17.329 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:25:17.330 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-06T07:25:17.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.328+0000 7f7de50ed640 1 -- 172.21.15.83:0/1304340674 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7dc006b500 msgr2=0x7f7dc006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:17.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.328+0000 7f7de50ed640 1 --2- 172.21.15.83:0/1304340674 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7dc006b500 0x7f7dc006d9c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f7dd0007cb0 tx=0x7f7dd0009040 comp rx=0 tx=0).stop 2024-03-06T07:25:17.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.328+0000 7f7de50ed640 1 -- 172.21.15.83:0/1304340674 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7de015b650 msgr2=0x7f7de015bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:17.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.328+0000 7f7de50ed640 1 --2- 172.21.15.83:0/1304340674 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7de015b650 0x7f7de015bad0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f7dd802c060 tx=0x7f7dd80014a0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.328+0000 7f7de50ed640 1 -- 172.21.15.83:0/1304340674 shutdown_connections 2024-03-06T07:25:17.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.328+0000 7f7de50ed640 1 --2- 172.21.15.83:0/1304340674 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7dc006b500 0x7f7dc006d9c0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.328+0000 7f7de50ed640 1 --2- 172.21.15.83:0/1304340674 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7de015b650 0x7f7de015bad0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.328+0000 7f7de50ed640 1 --2- 172.21.15.83:0/1304340674 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7de00cc8f0 0x7f7de0156560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.328+0000 7f7de50ed640 1 --2- 172.21.15.83:0/1304340674 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7dcc0051c0 0x7f7de01f3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.328+0000 7f7de50ed640 1 -- 172.21.15.83:0/1304340674 >> 172.21.15.83:0/1304340674 conn(0x7f7de00d3cd0 msgr2=0x7f7de00b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:17.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.328+0000 7f7de50ed640 1 -- 172.21.15.83:0/1304340674 shutdown_connections 2024-03-06T07:25:17.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.328+0000 7f7de50ed640 1 -- 172.21.15.83:0/1304340674 wait complete. 2024-03-06T07:25:17.341 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":168,"stamp":"2024-03-06T07:25:16.356344+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":114,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":257228,"kb_used_data":7760,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754717492,"statfs":{"total":773094113280,"available":772830711808,"internally_reserved":0,"allocated":7946240,"data_stored":3994068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12757,"internal_metadata":255315499},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.231864"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228800+0000","last_change":"2024-03-06T07:25:13.165487+0000","last_active":"2024-03-06T07:25:15.228800+0000","last_peered":"2024-03-06T07:25:15.228800+0000","last_clean":"2024-03-06T07:25:15.228800+0000","last_became_active":"2024-03-06T07:25:13.165172+0000","last_became_peered":"2024-03-06T07:25:13.165172+0000","last_unstale":"2024-03-06T07:25:15.228800+0000","last_undegraded":"2024-03-06T07:25:15.228800+0000","last_fullsized":"2024-03-06T07:25:15.228800+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:48:56.369026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156914+0000","last_change":"2024-03-06T07:25:13.156380+0000","last_active":"2024-03-06T07:25:14.156914+0000","last_peered":"2024-03-06T07:25:14.156914+0000","last_clean":"2024-03-06T07:25:14.156914+0000","last_became_active":"2024-03-06T07:25:13.156056+0000","last_became_peered":"2024-03-06T07:25:13.156056+0000","last_unstale":"2024-03-06T07:25:14.156914+0000","last_undegraded":"2024-03-06T07:25:14.156914+0000","last_fullsized":"2024-03-06T07:25:14.156914+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:09.679768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228224+0000","last_change":"2024-03-06T07:25:13.163796+0000","last_active":"2024-03-06T07:25:15.228224+0000","last_peered":"2024-03-06T07:25:15.228224+0000","last_clean":"2024-03-06T07:25:15.228224+0000","last_became_active":"2024-03-06T07:25:13.163511+0000","last_became_peered":"2024-03-06T07:25:13.163511+0000","last_unstale":"2024-03-06T07:25:15.228224+0000","last_undegraded":"2024-03-06T07:25:15.228224+0000","last_fullsized":"2024-03-06T07:25:15.228224+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:24.796057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228090+0000","last_change":"2024-03-06T07:25:13.158139+0000","last_active":"2024-03-06T07:25:15.228090+0000","last_peered":"2024-03-06T07:25:15.228090+0000","last_clean":"2024-03-06T07:25:15.228090+0000","last_became_active":"2024-03-06T07:25:13.157866+0000","last_became_peered":"2024-03-06T07:25:13.157866+0000","last_unstale":"2024-03-06T07:25:15.228090+0000","last_undegraded":"2024-03-06T07:25:15.228090+0000","last_fullsized":"2024-03-06T07:25:15.228090+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:49:21.998709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.154891+0000","last_change":"2024-03-06T07:25:13.163633+0000","last_active":"2024-03-06T07:25:14.154891+0000","last_peered":"2024-03-06T07:25:14.154891+0000","last_clean":"2024-03-06T07:25:14.154891+0000","last_became_active":"2024-03-06T07:25:13.163362+0000","last_became_peered":"2024-03-06T07:25:13.163362+0000","last_unstale":"2024-03-06T07:25:14.154891+0000","last_undegraded":"2024-03-06T07:25:14.154891+0000","last_fullsized":"2024-03-06T07:25:14.154891+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:55.118273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156325+0000","last_change":"2024-03-06T07:25:13.154620+0000","last_active":"2024-03-06T07:25:14.156325+0000","last_peered":"2024-03-06T07:25:14.156325+0000","last_clean":"2024-03-06T07:25:14.156325+0000","last_became_active":"2024-03-06T07:25:13.154363+0000","last_became_peered":"2024-03-06T07:25:13.154363+0000","last_unstale":"2024-03-06T07:25:14.156325+0000","last_undegraded":"2024-03-06T07:25:14.156325+0000","last_fullsized":"2024-03-06T07:25:14.156325+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:53:28.757512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.158955+0000","last_change":"2024-03-06T07:25:13.164757+0000","last_active":"2024-03-06T07:25:14.158955+0000","last_peered":"2024-03-06T07:25:14.158955+0000","last_clean":"2024-03-06T07:25:14.158955+0000","last_became_active":"2024-03-06T07:25:13.164418+0000","last_became_peered":"2024-03-06T07:25:13.164418+0000","last_unstale":"2024-03-06T07:25:14.158955+0000","last_undegraded":"2024-03-06T07:25:14.158955+0000","last_fullsized":"2024-03-06T07:25:14.158955+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:36:46.984162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.155994+0000","last_change":"2024-03-06T07:25:13.165057+0000","last_active":"2024-03-06T07:25:14.155994+0000","last_peered":"2024-03-06T07:25:14.155994+0000","last_clean":"2024-03-06T07:25:14.155994+0000","last_became_active":"2024-03-06T07:25:13.164778+0000","last_became_peered":"2024-03-06T07:25:13.164778+0000","last_unstale":"2024-03-06T07:25:14.155994+0000","last_undegraded":"2024-03-06T07:25:14.155994+0000","last_fullsized":"2024-03-06T07:25:14.155994+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:33:40.938375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.155033+0000","last_change":"2024-03-06T07:25:13.155808+0000","last_active":"2024-03-06T07:25:14.155033+0000","last_peered":"2024-03-06T07:25:14.155033+0000","last_clean":"2024-03-06T07:25:14.155033+0000","last_became_active":"2024-03-06T07:25:13.155222+0000","last_became_peered":"2024-03-06T07:25:13.155222+0000","last_unstale":"2024-03-06T07:25:14.155033+0000","last_undegraded":"2024-03-06T07:25:14.155033+0000","last_fullsized":"2024-03-06T07:25:14.155033+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:05:02.223283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156306+0000","last_change":"2024-03-06T07:25:13.157719+0000","last_active":"2024-03-06T07:25:14.156306+0000","last_peered":"2024-03-06T07:25:14.156306+0000","last_clean":"2024-03-06T07:25:14.156306+0000","last_became_active":"2024-03-06T07:25:13.156832+0000","last_became_peered":"2024-03-06T07:25:13.156832+0000","last_unstale":"2024-03-06T07:25:14.156306+0000","last_undegraded":"2024-03-06T07:25:14.156306+0000","last_fullsized":"2024-03-06T07:25:14.156306+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:50.449834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228893+0000","last_change":"2024-03-06T07:25:13.163124+0000","last_active":"2024-03-06T07:25:15.228893+0000","last_peered":"2024-03-06T07:25:15.228893+0000","last_clean":"2024-03-06T07:25:15.228893+0000","last_became_active":"2024-03-06T07:25:13.162725+0000","last_became_peered":"2024-03-06T07:25:13.162725+0000","last_unstale":"2024-03-06T07:25:15.228893+0000","last_undegraded":"2024-03-06T07:25:15.228893+0000","last_fullsized":"2024-03-06T07:25:15.228893+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:24.796057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228220+0000","last_change":"2024-03-06T07:25:13.157110+0000","last_active":"2024-03-06T07:25:15.228220+0000","last_peered":"2024-03-06T07:25:15.228220+0000","last_clean":"2024-03-06T07:25:15.228220+0000","last_became_active":"2024-03-06T07:25:13.156814+0000","last_became_peered":"2024-03-06T07:25:13.156814+0000","last_unstale":"2024-03-06T07:25:15.228220+0000","last_undegraded":"2024-03-06T07:25:15.228220+0000","last_fullsized":"2024-03-06T07:25:15.228220+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:07.232004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.161003+0000","last_change":"2024-03-06T07:25:13.153995+0000","last_active":"2024-03-06T07:25:14.161003+0000","last_peered":"2024-03-06T07:25:14.161003+0000","last_clean":"2024-03-06T07:25:14.161003+0000","last_became_active":"2024-03-06T07:25:13.153593+0000","last_became_peered":"2024-03-06T07:25:13.153593+0000","last_unstale":"2024-03-06T07:25:14.161003+0000","last_undegraded":"2024-03-06T07:25:14.161003+0000","last_fullsized":"2024-03-06T07:25:14.161003+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:02:06.354776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.155174+0000","last_change":"2024-03-06T07:25:13.157216+0000","last_active":"2024-03-06T07:25:14.155174+0000","last_peered":"2024-03-06T07:25:14.155174+0000","last_clean":"2024-03-06T07:25:14.155174+0000","last_became_active":"2024-03-06T07:25:13.156941+0000","last_became_peered":"2024-03-06T07:25:13.156941+0000","last_unstale":"2024-03-06T07:25:14.155174+0000","last_undegraded":"2024-03-06T07:25:14.155174+0000","last_fullsized":"2024-03-06T07:25:14.155174+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:20:21.040939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.160873+0000","last_change":"2024-03-06T07:25:13.158351+0000","last_active":"2024-03-06T07:25:14.160873+0000","last_peered":"2024-03-06T07:25:14.160873+0000","last_clean":"2024-03-06T07:25:14.160873+0000","last_became_active":"2024-03-06T07:25:13.158092+0000","last_became_peered":"2024-03-06T07:25:13.158092+0000","last_unstale":"2024-03-06T07:25:14.160873+0000","last_undegraded":"2024-03-06T07:25:14.160873+0000","last_fullsized":"2024-03-06T07:25:14.160873+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:50.449834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.155410+0000","last_change":"2024-03-06T07:25:13.152334+0000","last_active":"2024-03-06T07:25:14.155410+0000","last_peered":"2024-03-06T07:25:14.155410+0000","last_clean":"2024-03-06T07:25:14.155410+0000","last_became_active":"2024-03-06T07:25:13.152021+0000","last_became_peered":"2024-03-06T07:25:13.152021+0000","last_unstale":"2024-03-06T07:25:14.155410+0000","last_undegraded":"2024-03-06T07:25:14.155410+0000","last_fullsized":"2024-03-06T07:25:14.155410+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:01:49.348156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.155049+0000","last_change":"2024-03-06T07:25:13.155930+0000","last_active":"2024-03-06T07:25:14.155049+0000","last_peered":"2024-03-06T07:25:14.155049+0000","last_clean":"2024-03-06T07:25:14.155049+0000","last_became_active":"2024-03-06T07:25:13.155643+0000","last_became_peered":"2024-03-06T07:25:13.155643+0000","last_unstale":"2024-03-06T07:25:14.155049+0000","last_undegraded":"2024-03-06T07:25:14.155049+0000","last_fullsized":"2024-03-06T07:25:14.155049+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:59:52.753418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228174+0000","last_change":"2024-03-06T07:25:13.153982+0000","last_active":"2024-03-06T07:25:15.228174+0000","last_peered":"2024-03-06T07:25:15.228174+0000","last_clean":"2024-03-06T07:25:15.228174+0000","last_became_active":"2024-03-06T07:25:13.153358+0000","last_became_peered":"2024-03-06T07:25:13.153358+0000","last_unstale":"2024-03-06T07:25:15.228174+0000","last_undegraded":"2024-03-06T07:25:15.228174+0000","last_fullsized":"2024-03-06T07:25:15.228174+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:59:52.753418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156552+0000","last_change":"2024-03-06T07:25:13.157124+0000","last_active":"2024-03-06T07:25:14.156552+0000","last_peered":"2024-03-06T07:25:14.156552+0000","last_clean":"2024-03-06T07:25:14.156552+0000","last_became_active":"2024-03-06T07:25:13.156610+0000","last_became_peered":"2024-03-06T07:25:13.156610+0000","last_unstale":"2024-03-06T07:25:14.156552+0000","last_undegraded":"2024-03-06T07:25:14.156552+0000","last_fullsized":"2024-03-06T07:25:14.156552+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:37.851591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.161066+0000","last_change":"2024-03-06T07:25:13.164176+0000","last_active":"2024-03-06T07:25:14.161066+0000","last_peered":"2024-03-06T07:25:14.161066+0000","last_clean":"2024-03-06T07:25:14.161066+0000","last_became_active":"2024-03-06T07:25:13.163099+0000","last_became_peered":"2024-03-06T07:25:13.163099+0000","last_unstale":"2024-03-06T07:25:14.161066+0000","last_undegraded":"2024-03-06T07:25:14.161066+0000","last_fullsized":"2024-03-06T07:25:14.161066+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:40.549085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156102+0000","last_change":"2024-03-06T07:25:13.155225+0000","last_active":"2024-03-06T07:25:14.156102+0000","last_peered":"2024-03-06T07:25:14.156102+0000","last_clean":"2024-03-06T07:25:14.156102+0000","last_became_active":"2024-03-06T07:25:13.154937+0000","last_became_peered":"2024-03-06T07:25:13.154937+0000","last_unstale":"2024-03-06T07:25:14.156102+0000","last_undegraded":"2024-03-06T07:25:14.156102+0000","last_fullsized":"2024-03-06T07:25:14.156102+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:49.755068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.155597+0000","last_change":"2024-03-06T07:25:13.157186+0000","last_active":"2024-03-06T07:25:14.155597+0000","last_peered":"2024-03-06T07:25:14.155597+0000","last_clean":"2024-03-06T07:25:14.155597+0000","last_became_active":"2024-03-06T07:25:13.156842+0000","last_became_peered":"2024-03-06T07:25:13.156842+0000","last_unstale":"2024-03-06T07:25:14.155597+0000","last_undegraded":"2024-03-06T07:25:14.155597+0000","last_fullsized":"2024-03-06T07:25:14.155597+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:15:35.393776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228449+0000","last_change":"2024-03-06T07:25:13.169632+0000","last_active":"2024-03-06T07:25:15.228449+0000","last_peered":"2024-03-06T07:25:15.228449+0000","last_clean":"2024-03-06T07:25:15.228449+0000","last_became_active":"2024-03-06T07:25:13.167476+0000","last_became_peered":"2024-03-06T07:25:13.167476+0000","last_unstale":"2024-03-06T07:25:15.228449+0000","last_undegraded":"2024-03-06T07:25:15.228449+0000","last_fullsized":"2024-03-06T07:25:15.228449+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:13.108157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228467+0000","last_change":"2024-03-06T07:25:13.164595+0000","last_active":"2024-03-06T07:25:15.228467+0000","last_peered":"2024-03-06T07:25:15.228467+0000","last_clean":"2024-03-06T07:25:15.228467+0000","last_became_active":"2024-03-06T07:25:13.164313+0000","last_became_peered":"2024-03-06T07:25:13.164313+0000","last_unstale":"2024-03-06T07:25:15.228467+0000","last_undegraded":"2024-03-06T07:25:15.228467+0000","last_fullsized":"2024-03-06T07:25:15.228467+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:24.956262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.155832+0000","last_change":"2024-03-06T07:25:13.158874+0000","last_active":"2024-03-06T07:25:14.155832+0000","last_peered":"2024-03-06T07:25:14.155832+0000","last_clean":"2024-03-06T07:25:14.155832+0000","last_became_active":"2024-03-06T07:25:13.158322+0000","last_became_peered":"2024-03-06T07:25:13.158322+0000","last_unstale":"2024-03-06T07:25:14.155832+0000","last_undegraded":"2024-03-06T07:25:14.155832+0000","last_fullsized":"2024-03-06T07:25:14.155832+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:27.680894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.159090+0000","last_change":"2024-03-06T07:25:13.160666+0000","last_active":"2024-03-06T07:25:14.159090+0000","last_peered":"2024-03-06T07:25:14.159090+0000","last_clean":"2024-03-06T07:25:14.159090+0000","last_became_active":"2024-03-06T07:25:13.160380+0000","last_became_peered":"2024-03-06T07:25:13.160380+0000","last_unstale":"2024-03-06T07:25:14.159090+0000","last_undegraded":"2024-03-06T07:25:14.159090+0000","last_fullsized":"2024-03-06T07:25:14.159090+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:01.195841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228349+0000","last_change":"2024-03-06T07:25:13.160230+0000","last_active":"2024-03-06T07:25:15.228349+0000","last_peered":"2024-03-06T07:25:15.228349+0000","last_clean":"2024-03-06T07:25:15.228349+0000","last_became_active":"2024-03-06T07:25:13.159059+0000","last_became_peered":"2024-03-06T07:25:13.159059+0000","last_unstale":"2024-03-06T07:25:15.228349+0000","last_undegraded":"2024-03-06T07:25:15.228349+0000","last_fullsized":"2024-03-06T07:25:15.228349+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:46:28.717814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156175+0000","last_change":"2024-03-06T07:25:13.157587+0000","last_active":"2024-03-06T07:25:14.156175+0000","last_peered":"2024-03-06T07:25:14.156175+0000","last_clean":"2024-03-06T07:25:14.156175+0000","last_became_active":"2024-03-06T07:25:13.157280+0000","last_became_peered":"2024-03-06T07:25:13.157280+0000","last_unstale":"2024-03-06T07:25:14.156175+0000","last_undegraded":"2024-03-06T07:25:14.156175+0000","last_fullsized":"2024-03-06T07:25:14.156175+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:27:22.280558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228758+0000","last_change":"2024-03-06T07:25:13.160500+0000","last_active":"2024-03-06T07:25:15.228758+0000","last_peered":"2024-03-06T07:25:15.228758+0000","last_clean":"2024-03-06T07:25:15.228758+0000","last_became_active":"2024-03-06T07:25:13.160121+0000","last_became_peered":"2024-03-06T07:25:13.160121+0000","last_unstale":"2024-03-06T07:25:15.228758+0000","last_undegraded":"2024-03-06T07:25:15.228758+0000","last_fullsized":"2024-03-06T07:25:15.228758+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:05.941938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.159034+0000","last_change":"2024-03-06T07:25:13.158496+0000","last_active":"2024-03-06T07:25:14.159034+0000","last_peered":"2024-03-06T07:25:14.159034+0000","last_clean":"2024-03-06T07:25:14.159034+0000","last_became_active":"2024-03-06T07:25:13.158237+0000","last_became_peered":"2024-03-06T07:25:13.158237+0000","last_unstale":"2024-03-06T07:25:14.159034+0000","last_undegraded":"2024-03-06T07:25:14.159034+0000","last_fullsized":"2024-03-06T07:25:14.159034+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:08:35.657078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.159186+0000","last_change":"2024-03-06T07:25:13.162268+0000","last_active":"2024-03-06T07:25:14.159186+0000","last_peered":"2024-03-06T07:25:14.159186+0000","last_clean":"2024-03-06T07:25:14.159186+0000","last_became_active":"2024-03-06T07:25:13.162021+0000","last_became_peered":"2024-03-06T07:25:13.162021+0000","last_unstale":"2024-03-06T07:25:14.159186+0000","last_undegraded":"2024-03-06T07:25:14.159186+0000","last_fullsized":"2024-03-06T07:25:14.159186+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:33:11.084604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156035+0000","last_change":"2024-03-06T07:25:13.156062+0000","last_active":"2024-03-06T07:25:14.156035+0000","last_peered":"2024-03-06T07:25:14.156035+0000","last_clean":"2024-03-06T07:25:14.156035+0000","last_became_active":"2024-03-06T07:25:13.155807+0000","last_became_peered":"2024-03-06T07:25:13.155807+0000","last_unstale":"2024-03-06T07:25:14.156035+0000","last_undegraded":"2024-03-06T07:25:14.156035+0000","last_fullsized":"2024-03-06T07:25:14.156035+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:28:36.984619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228779+0000","last_change":"2024-03-06T07:25:13.165720+0000","last_active":"2024-03-06T07:25:15.228779+0000","last_peered":"2024-03-06T07:25:15.228779+0000","last_clean":"2024-03-06T07:25:15.228779+0000","last_became_active":"2024-03-06T07:25:13.165367+0000","last_became_peered":"2024-03-06T07:25:13.165367+0000","last_unstale":"2024-03-06T07:25:15.228779+0000","last_undegraded":"2024-03-06T07:25:15.228779+0000","last_fullsized":"2024-03-06T07:25:15.228779+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:37.968763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"68'4","reported_seq":18,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228919+0000","last_change":"2024-03-06T07:25:12.150041+0000","last_active":"2024-03-06T07:25:15.228919+0000","last_peered":"2024-03-06T07:25:15.228919+0000","last_clean":"2024-03-06T07:25:15.228919+0000","last_became_active":"2024-03-06T07:25:12.149784+0000","last_became_peered":"2024-03-06T07:25:12.149784+0000","last_unstale":"2024-03-06T07:25:15.228919+0000","last_undegraded":"2024-03-06T07:25:15.228919+0000","last_fullsized":"2024-03-06T07:25:15.228919+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:49.445567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.155153+0000","last_change":"2024-03-06T07:25:13.155523+0000","last_active":"2024-03-06T07:25:14.155153+0000","last_peered":"2024-03-06T07:25:14.155153+0000","last_clean":"2024-03-06T07:25:14.155153+0000","last_became_active":"2024-03-06T07:25:13.154872+0000","last_became_peered":"2024-03-06T07:25:13.154872+0000","last_unstale":"2024-03-06T07:25:14.155153+0000","last_undegraded":"2024-03-06T07:25:14.155153+0000","last_fullsized":"2024-03-06T07:25:14.155153+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:07.232004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.161070+0000","last_change":"2024-03-06T07:25:12.140136+0000","last_active":"2024-03-06T07:25:14.161070+0000","last_peered":"2024-03-06T07:25:14.161070+0000","last_clean":"2024-03-06T07:25:14.161070+0000","last_became_active":"2024-03-06T07:25:12.139794+0000","last_became_peered":"2024-03-06T07:25:12.139794+0000","last_unstale":"2024-03-06T07:25:14.161070+0000","last_undegraded":"2024-03-06T07:25:14.161070+0000","last_fullsized":"2024-03-06T07:25:14.161070+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:39.546776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156755+0000","last_change":"2024-03-06T07:25:13.165146+0000","last_active":"2024-03-06T07:25:14.156755+0000","last_peered":"2024-03-06T07:25:14.156755+0000","last_clean":"2024-03-06T07:25:14.156755+0000","last_became_active":"2024-03-06T07:25:13.164825+0000","last_became_peered":"2024-03-06T07:25:13.164825+0000","last_unstale":"2024-03-06T07:25:14.156755+0000","last_undegraded":"2024-03-06T07:25:14.156755+0000","last_fullsized":"2024-03-06T07:25:14.156755+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:49:30.437740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156134+0000","last_change":"2024-03-06T07:25:12.141294+0000","last_active":"2024-03-06T07:25:14.156134+0000","last_peered":"2024-03-06T07:25:14.156134+0000","last_clean":"2024-03-06T07:25:14.156134+0000","last_became_active":"2024-03-06T07:25:12.141044+0000","last_became_peered":"2024-03-06T07:25:12.141044+0000","last_unstale":"2024-03-06T07:25:14.156134+0000","last_undegraded":"2024-03-06T07:25:14.156134+0000","last_fullsized":"2024-03-06T07:25:14.156134+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:53.918943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228860+0000","last_change":"2024-03-06T07:25:13.167242+0000","last_active":"2024-03-06T07:25:15.228860+0000","last_peered":"2024-03-06T07:25:15.228860+0000","last_clean":"2024-03-06T07:25:15.228860+0000","last_became_active":"2024-03-06T07:25:13.166935+0000","last_became_peered":"2024-03-06T07:25:13.166935+0000","last_unstale":"2024-03-06T07:25:15.228860+0000","last_undegraded":"2024-03-06T07:25:15.228860+0000","last_fullsized":"2024-03-06T07:25:15.228860+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:03.696343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156910+0000","last_change":"2024-03-06T07:25:12.144105+0000","last_active":"2024-03-06T07:25:14.156910+0000","last_peered":"2024-03-06T07:25:14.156910+0000","last_clean":"2024-03-06T07:25:14.156910+0000","last_became_active":"2024-03-06T07:25:12.143797+0000","last_became_peered":"2024-03-06T07:25:12.143797+0000","last_unstale":"2024-03-06T07:25:14.156910+0000","last_undegraded":"2024-03-06T07:25:14.156910+0000","last_fullsized":"2024-03-06T07:25:14.156910+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:54.114006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.159082+0000","last_change":"2024-03-06T07:25:13.161312+0000","last_active":"2024-03-06T07:25:14.159082+0000","last_peered":"2024-03-06T07:25:14.159082+0000","last_clean":"2024-03-06T07:25:14.159082+0000","last_became_active":"2024-03-06T07:25:13.161037+0000","last_became_peered":"2024-03-06T07:25:13.161037+0000","last_unstale":"2024-03-06T07:25:14.159082+0000","last_undegraded":"2024-03-06T07:25:14.159082+0000","last_fullsized":"2024-03-06T07:25:14.159082+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:44:51.540913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228853+0000","last_change":"2024-03-06T07:25:12.142783+0000","last_active":"2024-03-06T07:25:15.228853+0000","last_peered":"2024-03-06T07:25:15.228853+0000","last_clean":"2024-03-06T07:25:15.228853+0000","last_became_active":"2024-03-06T07:25:12.142498+0000","last_became_peered":"2024-03-06T07:25:12.142498+0000","last_unstale":"2024-03-06T07:25:15.228853+0000","last_undegraded":"2024-03-06T07:25:15.228853+0000","last_fullsized":"2024-03-06T07:25:15.228853+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:23.951995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.155941+0000","last_change":"2024-03-06T07:25:13.158674+0000","last_active":"2024-03-06T07:25:14.155941+0000","last_peered":"2024-03-06T07:25:14.155941+0000","last_clean":"2024-03-06T07:25:14.155941+0000","last_became_active":"2024-03-06T07:25:13.158369+0000","last_became_peered":"2024-03-06T07:25:13.158369+0000","last_unstale":"2024-03-06T07:25:14.155941+0000","last_undegraded":"2024-03-06T07:25:14.155941+0000","last_fullsized":"2024-03-06T07:25:14.155941+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:27:59.088271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156292+0000","last_change":"2024-03-06T07:25:12.148361+0000","last_active":"2024-03-06T07:25:14.156292+0000","last_peered":"2024-03-06T07:25:14.156292+0000","last_clean":"2024-03-06T07:25:14.156292+0000","last_became_active":"2024-03-06T07:25:12.148055+0000","last_became_peered":"2024-03-06T07:25:12.148055+0000","last_unstale":"2024-03-06T07:25:14.156292+0000","last_undegraded":"2024-03-06T07:25:14.156292+0000","last_fullsized":"2024-03-06T07:25:14.156292+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:52:02.481328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.154647+0000","last_change":"2024-03-06T07:25:13.157940+0000","last_active":"2024-03-06T07:25:14.154647+0000","last_peered":"2024-03-06T07:25:14.154647+0000","last_clean":"2024-03-06T07:25:14.154647+0000","last_became_active":"2024-03-06T07:25:13.157682+0000","last_became_peered":"2024-03-06T07:25:13.157682+0000","last_unstale":"2024-03-06T07:25:14.154647+0000","last_undegraded":"2024-03-06T07:25:14.154647+0000","last_fullsized":"2024-03-06T07:25:14.154647+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:06.723675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.227986+0000","last_change":"2024-03-06T07:25:12.144427+0000","last_active":"2024-03-06T07:25:15.227986+0000","last_peered":"2024-03-06T07:25:15.227986+0000","last_clean":"2024-03-06T07:25:15.227986+0000","last_became_active":"2024-03-06T07:25:12.144166+0000","last_became_peered":"2024-03-06T07:25:12.144166+0000","last_unstale":"2024-03-06T07:25:15.227986+0000","last_undegraded":"2024-03-06T07:25:15.227986+0000","last_fullsized":"2024-03-06T07:25:15.227986+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:41:07.504129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.160961+0000","last_change":"2024-03-06T07:25:13.160676+0000","last_active":"2024-03-06T07:25:14.160961+0000","last_peered":"2024-03-06T07:25:14.160961+0000","last_clean":"2024-03-06T07:25:14.160961+0000","last_became_active":"2024-03-06T07:25:13.160341+0000","last_became_peered":"2024-03-06T07:25:13.160341+0000","last_unstale":"2024-03-06T07:25:14.160961+0000","last_undegraded":"2024-03-06T07:25:14.160961+0000","last_fullsized":"2024-03-06T07:25:14.160961+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:32.749309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.154646+0000","last_change":"2024-03-06T07:25:12.145436+0000","last_active":"2024-03-06T07:25:14.154646+0000","last_peered":"2024-03-06T07:25:14.154646+0000","last_clean":"2024-03-06T07:25:14.154646+0000","last_became_active":"2024-03-06T07:25:12.145165+0000","last_became_peered":"2024-03-06T07:25:12.145165+0000","last_unstale":"2024-03-06T07:25:14.154646+0000","last_undegraded":"2024-03-06T07:25:14.154646+0000","last_fullsized":"2024-03-06T07:25:14.154646+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:51.126409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.159103+0000","last_change":"2024-03-06T07:25:13.159948+0000","last_active":"2024-03-06T07:25:14.159103+0000","last_peered":"2024-03-06T07:25:14.159103+0000","last_clean":"2024-03-06T07:25:14.159103+0000","last_became_active":"2024-03-06T07:25:13.159672+0000","last_became_peered":"2024-03-06T07:25:13.159672+0000","last_unstale":"2024-03-06T07:25:14.159103+0000","last_undegraded":"2024-03-06T07:25:14.159103+0000","last_fullsized":"2024-03-06T07:25:14.159103+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:44:28.285425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.155696+0000","last_change":"2024-03-06T07:25:12.140394+0000","last_active":"2024-03-06T07:25:14.155696+0000","last_peered":"2024-03-06T07:25:14.155696+0000","last_clean":"2024-03-06T07:25:14.155696+0000","last_became_active":"2024-03-06T07:25:12.140118+0000","last_became_peered":"2024-03-06T07:25:12.140118+0000","last_unstale":"2024-03-06T07:25:14.155696+0000","last_undegraded":"2024-03-06T07:25:14.155696+0000","last_fullsized":"2024-03-06T07:25:14.155696+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:13.758634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228811+0000","last_change":"2024-03-06T07:25:13.169371+0000","last_active":"2024-03-06T07:25:15.228811+0000","last_peered":"2024-03-06T07:25:15.228811+0000","last_clean":"2024-03-06T07:25:15.228811+0000","last_became_active":"2024-03-06T07:25:13.169085+0000","last_became_peered":"2024-03-06T07:25:13.169085+0000","last_unstale":"2024-03-06T07:25:15.228811+0000","last_undegraded":"2024-03-06T07:25:15.228811+0000","last_fullsized":"2024-03-06T07:25:15.228811+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:10.404106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156948+0000","last_change":"2024-03-06T07:25:12.145549+0000","last_active":"2024-03-06T07:25:14.156948+0000","last_peered":"2024-03-06T07:25:14.156948+0000","last_clean":"2024-03-06T07:25:14.156948+0000","last_became_active":"2024-03-06T07:25:12.144847+0000","last_became_peered":"2024-03-06T07:25:12.144847+0000","last_unstale":"2024-03-06T07:25:14.156948+0000","last_undegraded":"2024-03-06T07:25:14.156948+0000","last_fullsized":"2024-03-06T07:25:14.156948+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:05.719408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156729+0000","last_change":"2024-03-06T07:25:13.157940+0000","last_active":"2024-03-06T07:25:14.156729+0000","last_peered":"2024-03-06T07:25:14.156729+0000","last_clean":"2024-03-06T07:25:14.156729+0000","last_became_active":"2024-03-06T07:25:13.157695+0000","last_became_peered":"2024-03-06T07:25:13.157695+0000","last_unstale":"2024-03-06T07:25:14.156729+0000","last_undegraded":"2024-03-06T07:25:14.156729+0000","last_fullsized":"2024-03-06T07:25:14.156729+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:40.551042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"68'3","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228450+0000","last_change":"2024-03-06T07:25:12.147266+0000","last_active":"2024-03-06T07:25:15.228450+0000","last_peered":"2024-03-06T07:25:15.228450+0000","last_clean":"2024-03-06T07:25:15.228450+0000","last_became_active":"2024-03-06T07:25:12.146961+0000","last_became_peered":"2024-03-06T07:25:12.146961+0000","last_unstale":"2024-03-06T07:25:15.228450+0000","last_undegraded":"2024-03-06T07:25:15.228450+0000","last_fullsized":"2024-03-06T07:25:15.228450+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:33:10.080337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.154820+0000","last_change":"2024-03-06T07:25:13.160320+0000","last_active":"2024-03-06T07:25:14.154820+0000","last_peered":"2024-03-06T07:25:14.154820+0000","last_clean":"2024-03-06T07:25:14.154820+0000","last_became_active":"2024-03-06T07:25:13.159684+0000","last_became_peered":"2024-03-06T07:25:13.159684+0000","last_unstale":"2024-03-06T07:25:14.154820+0000","last_undegraded":"2024-03-06T07:25:14.154820+0000","last_fullsized":"2024-03-06T07:25:14.154820+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:52:03.485595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"68'6","reported_seq":20,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228308+0000","last_change":"2024-03-06T07:25:12.145325+0000","last_active":"2024-03-06T07:25:15.228308+0000","last_peered":"2024-03-06T07:25:15.228308+0000","last_clean":"2024-03-06T07:25:15.228308+0000","last_became_active":"2024-03-06T07:25:12.145043+0000","last_became_peered":"2024-03-06T07:25:12.145043+0000","last_unstale":"2024-03-06T07:25:15.228308+0000","last_undegraded":"2024-03-06T07:25:15.228308+0000","last_fullsized":"2024-03-06T07:25:15.228308+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:01.865233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228511+0000","last_change":"2024-03-06T07:25:13.161899+0000","last_active":"2024-03-06T07:25:15.228511+0000","last_peered":"2024-03-06T07:25:15.228511+0000","last_clean":"2024-03-06T07:25:15.228511+0000","last_became_active":"2024-03-06T07:25:13.161618+0000","last_became_peered":"2024-03-06T07:25:13.161618+0000","last_unstale":"2024-03-06T07:25:15.228511+0000","last_undegraded":"2024-03-06T07:25:15.228511+0000","last_fullsized":"2024-03-06T07:25:15.228511+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:13.723493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228203+0000","last_change":"2024-03-06T07:25:12.144337+0000","last_active":"2024-03-06T07:25:15.228203+0000","last_peered":"2024-03-06T07:25:15.228203+0000","last_clean":"2024-03-06T07:25:15.228203+0000","last_became_active":"2024-03-06T07:25:12.144066+0000","last_became_peered":"2024-03-06T07:25:12.144066+0000","last_unstale":"2024-03-06T07:25:15.228203+0000","last_undegraded":"2024-03-06T07:25:15.228203+0000","last_fullsized":"2024-03-06T07:25:15.228203+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:30.030141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.159130+0000","last_change":"2024-03-06T07:25:13.157351+0000","last_active":"2024-03-06T07:25:14.159130+0000","last_peered":"2024-03-06T07:25:14.159130+0000","last_clean":"2024-03-06T07:25:14.159130+0000","last_became_active":"2024-03-06T07:25:13.157071+0000","last_became_peered":"2024-03-06T07:25:13.157071+0000","last_unstale":"2024-03-06T07:25:14.159130+0000","last_undegraded":"2024-03-06T07:25:14.159130+0000","last_fullsized":"2024-03-06T07:25:14.159130+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:59.805941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156408+0000","last_change":"2024-03-06T07:25:12.147133+0000","last_active":"2024-03-06T07:25:14.156408+0000","last_peered":"2024-03-06T07:25:14.156408+0000","last_clean":"2024-03-06T07:25:14.156408+0000","last_became_active":"2024-03-06T07:25:12.146865+0000","last_became_peered":"2024-03-06T07:25:12.146865+0000","last_unstale":"2024-03-06T07:25:14.156408+0000","last_undegraded":"2024-03-06T07:25:14.156408+0000","last_fullsized":"2024-03-06T07:25:14.156408+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:59:55.193963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.160853+0000","last_change":"2024-03-06T07:25:13.159853+0000","last_active":"2024-03-06T07:25:14.160853+0000","last_peered":"2024-03-06T07:25:14.160853+0000","last_clean":"2024-03-06T07:25:14.160853+0000","last_became_active":"2024-03-06T07:25:13.159589+0000","last_became_peered":"2024-03-06T07:25:13.159589+0000","last_unstale":"2024-03-06T07:25:14.160853+0000","last_undegraded":"2024-03-06T07:25:14.160853+0000","last_fullsized":"2024-03-06T07:25:14.160853+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:27:22.280558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.159073+0000","last_change":"2024-03-06T07:25:12.145548+0000","last_active":"2024-03-06T07:25:14.159073+0000","last_peered":"2024-03-06T07:25:14.159073+0000","last_clean":"2024-03-06T07:25:14.159073+0000","last_became_active":"2024-03-06T07:25:12.145214+0000","last_became_peered":"2024-03-06T07:25:12.145214+0000","last_unstale":"2024-03-06T07:25:14.159073+0000","last_undegraded":"2024-03-06T07:25:14.159073+0000","last_fullsized":"2024-03-06T07:25:14.159073+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:35.931140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.154972+0000","last_change":"2024-03-06T07:25:13.156771+0000","last_active":"2024-03-06T07:25:14.154972+0000","last_peered":"2024-03-06T07:25:14.154972+0000","last_clean":"2024-03-06T07:25:14.154972+0000","last_became_active":"2024-03-06T07:25:13.156472+0000","last_became_peered":"2024-03-06T07:25:13.156472+0000","last_unstale":"2024-03-06T07:25:14.154972+0000","last_undegraded":"2024-03-06T07:25:14.154972+0000","last_fullsized":"2024-03-06T07:25:14.154972+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:59:56.198230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"68'8","reported_seq":22,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.155528+0000","last_change":"2024-03-06T07:25:12.143217+0000","last_active":"2024-03-06T07:25:14.155528+0000","last_peered":"2024-03-06T07:25:14.155528+0000","last_clean":"2024-03-06T07:25:14.155528+0000","last_became_active":"2024-03-06T07:25:12.142853+0000","last_became_peered":"2024-03-06T07:25:12.142853+0000","last_unstale":"2024-03-06T07:25:14.155528+0000","last_undegraded":"2024-03-06T07:25:14.155528+0000","last_fullsized":"2024-03-06T07:25:14.155528+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:28:18.153551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156336+0000","last_change":"2024-03-06T07:25:13.155028+0000","last_active":"2024-03-06T07:25:14.156336+0000","last_peered":"2024-03-06T07:25:14.156336+0000","last_clean":"2024-03-06T07:25:14.156336+0000","last_became_active":"2024-03-06T07:25:13.154744+0000","last_became_peered":"2024-03-06T07:25:13.154744+0000","last_unstale":"2024-03-06T07:25:14.156336+0000","last_undegraded":"2024-03-06T07:25:14.156336+0000","last_fullsized":"2024-03-06T07:25:14.156336+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:32.749309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156986+0000","last_change":"2024-03-06T07:25:12.146430+0000","last_active":"2024-03-06T07:25:14.156986+0000","last_peered":"2024-03-06T07:25:14.156986+0000","last_clean":"2024-03-06T07:25:14.156986+0000","last_became_active":"2024-03-06T07:25:12.146111+0000","last_became_peered":"2024-03-06T07:25:12.146111+0000","last_unstale":"2024-03-06T07:25:14.156986+0000","last_undegraded":"2024-03-06T07:25:14.156986+0000","last_fullsized":"2024-03-06T07:25:14.156986+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:51:20.859880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228986+0000","last_change":"2024-03-06T07:25:13.167953+0000","last_active":"2024-03-06T07:25:15.228986+0000","last_peered":"2024-03-06T07:25:15.228986+0000","last_clean":"2024-03-06T07:25:15.228986+0000","last_became_active":"2024-03-06T07:25:13.167703+0000","last_became_peered":"2024-03-06T07:25:13.167703+0000","last_unstale":"2024-03-06T07:25:15.228986+0000","last_undegraded":"2024-03-06T07:25:15.228986+0000","last_fullsized":"2024-03-06T07:25:15.228986+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:19:01.252079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156426+0000","last_change":"2024-03-06T07:25:12.148744+0000","last_active":"2024-03-06T07:25:14.156426+0000","last_peered":"2024-03-06T07:25:14.156426+0000","last_clean":"2024-03-06T07:25:14.156426+0000","last_became_active":"2024-03-06T07:25:12.148418+0000","last_became_peered":"2024-03-06T07:25:12.148418+0000","last_unstale":"2024-03-06T07:25:14.156426+0000","last_undegraded":"2024-03-06T07:25:14.156426+0000","last_fullsized":"2024-03-06T07:25:14.156426+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:33:39.934108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228297+0000","last_change":"2024-03-06T07:25:13.156902+0000","last_active":"2024-03-06T07:25:15.228297+0000","last_peered":"2024-03-06T07:25:15.228297+0000","last_clean":"2024-03-06T07:25:15.228297+0000","last_became_active":"2024-03-06T07:25:13.156626+0000","last_became_peered":"2024-03-06T07:25:13.156626+0000","last_unstale":"2024-03-06T07:25:15.228297+0000","last_undegraded":"2024-03-06T07:25:15.228297+0000","last_fullsized":"2024-03-06T07:25:15.228297+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:20:21.040939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156828+0000","last_change":"2024-03-06T07:25:12.148664+0000","last_active":"2024-03-06T07:25:14.156828+0000","last_peered":"2024-03-06T07:25:14.156828+0000","last_clean":"2024-03-06T07:25:14.156828+0000","last_became_active":"2024-03-06T07:25:12.147501+0000","last_became_peered":"2024-03-06T07:25:12.147501+0000","last_unstale":"2024-03-06T07:25:14.156828+0000","last_undegraded":"2024-03-06T07:25:14.156828+0000","last_fullsized":"2024-03-06T07:25:14.156828+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:59:51.749151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.155041+0000","last_change":"2024-03-06T07:25:13.160545+0000","last_active":"2024-03-06T07:25:14.155041+0000","last_peered":"2024-03-06T07:25:14.155041+0000","last_clean":"2024-03-06T07:25:14.155041+0000","last_became_active":"2024-03-06T07:25:13.160053+0000","last_became_peered":"2024-03-06T07:25:13.160053+0000","last_unstale":"2024-03-06T07:25:14.155041+0000","last_undegraded":"2024-03-06T07:25:14.155041+0000","last_fullsized":"2024-03-06T07:25:14.155041+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:49:21.998709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228755+0000","last_change":"2024-03-06T07:25:12.151623+0000","last_active":"2024-03-06T07:25:15.228755+0000","last_peered":"2024-03-06T07:25:15.228755+0000","last_clean":"2024-03-06T07:25:15.228755+0000","last_became_active":"2024-03-06T07:25:12.151361+0000","last_became_peered":"2024-03-06T07:25:12.151361+0000","last_unstale":"2024-03-06T07:25:15.228755+0000","last_undegraded":"2024-03-06T07:25:15.228755+0000","last_fullsized":"2024-03-06T07:25:15.228755+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:27:21.276291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228797+0000","last_change":"2024-03-06T07:25:13.162254+0000","last_active":"2024-03-06T07:25:15.228797+0000","last_peered":"2024-03-06T07:25:15.228797+0000","last_clean":"2024-03-06T07:25:15.228797+0000","last_became_active":"2024-03-06T07:25:13.161990+0000","last_became_peered":"2024-03-06T07:25:13.161990+0000","last_unstale":"2024-03-06T07:25:15.228797+0000","last_undegraded":"2024-03-06T07:25:15.228797+0000","last_fullsized":"2024-03-06T07:25:15.228797+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:09:24.521844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156442+0000","last_change":"2024-03-06T07:25:12.148747+0000","last_active":"2024-03-06T07:25:14.156442+0000","last_peered":"2024-03-06T07:25:14.156442+0000","last_clean":"2024-03-06T07:25:14.156442+0000","last_became_active":"2024-03-06T07:25:12.147762+0000","last_became_peered":"2024-03-06T07:25:12.147762+0000","last_unstale":"2024-03-06T07:25:14.156442+0000","last_undegraded":"2024-03-06T07:25:14.156442+0000","last_fullsized":"2024-03-06T07:25:14.156442+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:05:01.219016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228646+0000","last_change":"2024-03-06T07:25:13.159680+0000","last_active":"2024-03-06T07:25:15.228646+0000","last_peered":"2024-03-06T07:25:15.228646+0000","last_clean":"2024-03-06T07:25:15.228646+0000","last_became_active":"2024-03-06T07:25:13.159424+0000","last_became_peered":"2024-03-06T07:25:13.159424+0000","last_unstale":"2024-03-06T07:25:15.228646+0000","last_undegraded":"2024-03-06T07:25:15.228646+0000","last_fullsized":"2024-03-06T07:25:15.228646+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:19:01.252079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.159208+0000","last_change":"2024-03-06T07:25:12.149069+0000","last_active":"2024-03-06T07:25:14.159208+0000","last_peered":"2024-03-06T07:25:14.159208+0000","last_clean":"2024-03-06T07:25:14.159208+0000","last_became_active":"2024-03-06T07:25:12.148582+0000","last_became_peered":"2024-03-06T07:25:12.148582+0000","last_unstale":"2024-03-06T07:25:14.159208+0000","last_undegraded":"2024-03-06T07:25:14.159208+0000","last_fullsized":"2024-03-06T07:25:14.159208+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:01:48.343889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228701+0000","last_change":"2024-03-06T07:25:13.156792+0000","last_active":"2024-03-06T07:25:15.228701+0000","last_peered":"2024-03-06T07:25:15.228701+0000","last_clean":"2024-03-06T07:25:15.228701+0000","last_became_active":"2024-03-06T07:25:13.156508+0000","last_became_peered":"2024-03-06T07:25:13.156508+0000","last_unstale":"2024-03-06T07:25:15.228701+0000","last_undegraded":"2024-03-06T07:25:15.228701+0000","last_fullsized":"2024-03-06T07:25:15.228701+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:49.607452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.160796+0000","last_change":"2024-03-06T07:25:12.140076+0000","last_active":"2024-03-06T07:25:14.160796+0000","last_peered":"2024-03-06T07:25:14.160796+0000","last_clean":"2024-03-06T07:25:14.160796+0000","last_became_active":"2024-03-06T07:25:12.139766+0000","last_became_peered":"2024-03-06T07:25:12.139766+0000","last_unstale":"2024-03-06T07:25:14.160796+0000","last_undegraded":"2024-03-06T07:25:14.160796+0000","last_fullsized":"2024-03-06T07:25:14.160796+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:49:29.433473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.161117+0000","last_change":"2024-03-06T07:25:13.159035+0000","last_active":"2024-03-06T07:25:14.161117+0000","last_peered":"2024-03-06T07:25:14.161117+0000","last_clean":"2024-03-06T07:25:14.161117+0000","last_became_active":"2024-03-06T07:25:13.158765+0000","last_became_peered":"2024-03-06T07:25:13.158765+0000","last_unstale":"2024-03-06T07:25:14.161117+0000","last_undegraded":"2024-03-06T07:25:14.161117+0000","last_fullsized":"2024-03-06T07:25:14.161117+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:49:42.624006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156808+0000","last_change":"2024-03-06T07:25:12.145745+0000","last_active":"2024-03-06T07:25:14.156808+0000","last_peered":"2024-03-06T07:25:14.156808+0000","last_clean":"2024-03-06T07:25:14.156808+0000","last_became_active":"2024-03-06T07:25:12.145474+0000","last_became_peered":"2024-03-06T07:25:12.145474+0000","last_unstale":"2024-03-06T07:25:14.156808+0000","last_undegraded":"2024-03-06T07:25:14.156808+0000","last_fullsized":"2024-03-06T07:25:14.156808+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:16:25.481036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.159159+0000","last_change":"2024-03-06T07:25:13.164462+0000","last_active":"2024-03-06T07:25:14.159159+0000","last_peered":"2024-03-06T07:25:14.159159+0000","last_clean":"2024-03-06T07:25:14.159159+0000","last_became_active":"2024-03-06T07:25:13.164167+0000","last_became_peered":"2024-03-06T07:25:13.164167+0000","last_unstale":"2024-03-06T07:25:14.159159+0000","last_undegraded":"2024-03-06T07:25:14.159159+0000","last_fullsized":"2024-03-06T07:25:14.159159+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:45:06.074331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"69'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.160466+0000","last_change":"2024-03-06T07:25:12.145578+0000","last_active":"2024-03-06T07:25:14.160466+0000","last_peered":"2024-03-06T07:25:14.160466+0000","last_clean":"2024-03-06T07:25:14.160466+0000","last_became_active":"2024-03-06T07:25:12.145278+0000","last_became_peered":"2024-03-06T07:25:12.145278+0000","last_unstale":"2024-03-06T07:25:14.160466+0000","last_undegraded":"2024-03-06T07:25:14.160466+0000","last_fullsized":"2024-03-06T07:25:14.160466+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:15:34.389509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228736+0000","last_change":"2024-03-06T07:25:13.167557+0000","last_active":"2024-03-06T07:25:15.228736+0000","last_peered":"2024-03-06T07:25:15.228736+0000","last_clean":"2024-03-06T07:25:15.228736+0000","last_became_active":"2024-03-06T07:25:13.167287+0000","last_became_peered":"2024-03-06T07:25:13.167287+0000","last_unstale":"2024-03-06T07:25:15.228736+0000","last_undegraded":"2024-03-06T07:25:15.228736+0000","last_fullsized":"2024-03-06T07:25:15.228736+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:53:53.770605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.155033+0000","last_change":"2024-03-06T07:25:12.146374+0000","last_active":"2024-03-06T07:25:14.155033+0000","last_peered":"2024-03-06T07:25:14.155033+0000","last_clean":"2024-03-06T07:25:14.155033+0000","last_became_active":"2024-03-06T07:25:12.146106+0000","last_became_peered":"2024-03-06T07:25:12.146106+0000","last_unstale":"2024-03-06T07:25:14.155033+0000","last_undegraded":"2024-03-06T07:25:14.155033+0000","last_fullsized":"2024-03-06T07:25:14.155033+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:44:58.938903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228979+0000","last_change":"2024-03-06T07:25:13.165780+0000","last_active":"2024-03-06T07:25:15.228979+0000","last_peered":"2024-03-06T07:25:15.228979+0000","last_clean":"2024-03-06T07:25:15.228979+0000","last_became_active":"2024-03-06T07:25:13.165477+0000","last_became_peered":"2024-03-06T07:25:13.165477+0000","last_unstale":"2024-03-06T07:25:15.228979+0000","last_undegraded":"2024-03-06T07:25:15.228979+0000","last_fullsized":"2024-03-06T07:25:15.228979+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:46:28.717814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.161029+0000","last_change":"2024-03-06T07:25:12.140996+0000","last_active":"2024-03-06T07:25:14.161029+0000","last_peered":"2024-03-06T07:25:14.161029+0000","last_clean":"2024-03-06T07:25:14.161029+0000","last_became_active":"2024-03-06T07:25:12.139506+0000","last_became_peered":"2024-03-06T07:25:12.139506+0000","last_unstale":"2024-03-06T07:25:14.161029+0000","last_undegraded":"2024-03-06T07:25:14.161029+0000","last_fullsized":"2024-03-06T07:25:14.161029+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:08.675501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.155202+0000","last_change":"2024-03-06T07:25:13.157415+0000","last_active":"2024-03-06T07:25:14.155202+0000","last_peered":"2024-03-06T07:25:14.155202+0000","last_clean":"2024-03-06T07:25:14.155202+0000","last_became_active":"2024-03-06T07:25:13.156547+0000","last_became_peered":"2024-03-06T07:25:13.156547+0000","last_unstale":"2024-03-06T07:25:14.155202+0000","last_undegraded":"2024-03-06T07:25:14.155202+0000","last_fullsized":"2024-03-06T07:25:14.155202+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:26.335671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.154992+0000","last_change":"2024-03-06T07:25:12.149100+0000","last_active":"2024-03-06T07:25:14.154992+0000","last_peered":"2024-03-06T07:25:14.154992+0000","last_clean":"2024-03-06T07:25:14.154992+0000","last_became_active":"2024-03-06T07:25:12.148792+0000","last_became_peered":"2024-03-06T07:25:12.148792+0000","last_unstale":"2024-03-06T07:25:14.154992+0000","last_undegraded":"2024-03-06T07:25:14.154992+0000","last_fullsized":"2024-03-06T07:25:14.154992+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:28.946566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156932+0000","last_change":"2024-03-06T07:25:13.162334+0000","last_active":"2024-03-06T07:25:14.156932+0000","last_peered":"2024-03-06T07:25:14.156932+0000","last_clean":"2024-03-06T07:25:14.156932+0000","last_became_active":"2024-03-06T07:25:13.162080+0000","last_became_peered":"2024-03-06T07:25:13.162080+0000","last_unstale":"2024-03-06T07:25:14.156932+0000","last_undegraded":"2024-03-06T07:25:14.156932+0000","last_fullsized":"2024-03-06T07:25:14.156932+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:36.935406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228017+0000","last_change":"2024-03-06T07:25:12.143767+0000","last_active":"2024-03-06T07:25:15.228017+0000","last_peered":"2024-03-06T07:25:15.228017+0000","last_clean":"2024-03-06T07:25:15.228017+0000","last_became_active":"2024-03-06T07:25:12.143484+0000","last_became_peered":"2024-03-06T07:25:12.143484+0000","last_unstale":"2024-03-06T07:25:15.228017+0000","last_undegraded":"2024-03-06T07:25:15.228017+0000","last_fullsized":"2024-03-06T07:25:15.228017+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:57:36.833773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228041+0000","last_change":"2024-03-06T07:25:13.152398+0000","last_active":"2024-03-06T07:25:15.228041+0000","last_peered":"2024-03-06T07:25:15.228041+0000","last_clean":"2024-03-06T07:25:15.228041+0000","last_became_active":"2024-03-06T07:25:13.152124+0000","last_became_peered":"2024-03-06T07:25:13.152124+0000","last_unstale":"2024-03-06T07:25:15.228041+0000","last_undegraded":"2024-03-06T07:25:15.228041+0000","last_fullsized":"2024-03-06T07:25:15.228041+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:05:02.223283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.155374+0000","last_change":"2024-03-06T07:25:12.146477+0000","last_active":"2024-03-06T07:25:14.155374+0000","last_peered":"2024-03-06T07:25:14.155374+0000","last_clean":"2024-03-06T07:25:14.155374+0000","last_became_active":"2024-03-06T07:25:12.146197+0000","last_became_peered":"2024-03-06T07:25:12.146197+0000","last_unstale":"2024-03-06T07:25:14.155374+0000","last_undegraded":"2024-03-06T07:25:14.155374+0000","last_fullsized":"2024-03-06T07:25:14.155374+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:36.270067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156095+0000","last_change":"2024-03-06T07:25:13.155252+0000","last_active":"2024-03-06T07:25:14.156095+0000","last_peered":"2024-03-06T07:25:14.156095+0000","last_clean":"2024-03-06T07:25:14.156095+0000","last_became_active":"2024-03-06T07:25:13.154953+0000","last_became_peered":"2024-03-06T07:25:13.154953+0000","last_unstale":"2024-03-06T07:25:14.156095+0000","last_undegraded":"2024-03-06T07:25:14.156095+0000","last_fullsized":"2024-03-06T07:25:14.156095+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:03:54.769565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156102+0000","last_change":"2024-03-06T07:25:12.144358+0000","last_active":"2024-03-06T07:25:14.156102+0000","last_peered":"2024-03-06T07:25:14.156102+0000","last_clean":"2024-03-06T07:25:14.156102+0000","last_became_active":"2024-03-06T07:25:12.143221+0000","last_became_peered":"2024-03-06T07:25:12.143221+0000","last_unstale":"2024-03-06T07:25:14.156102+0000","last_undegraded":"2024-03-06T07:25:14.156102+0000","last_fullsized":"2024-03-06T07:25:14.156102+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:06.227738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228835+0000","last_change":"2024-03-06T07:25:13.160729+0000","last_active":"2024-03-06T07:25:15.228835+0000","last_peered":"2024-03-06T07:25:15.228835+0000","last_clean":"2024-03-06T07:25:15.228835+0000","last_became_active":"2024-03-06T07:25:13.159186+0000","last_became_peered":"2024-03-06T07:25:13.159186+0000","last_unstale":"2024-03-06T07:25:15.228835+0000","last_undegraded":"2024-03-06T07:25:15.228835+0000","last_fullsized":"2024-03-06T07:25:15.228835+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:05.164335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156749+0000","last_change":"2024-03-06T07:25:12.142392+0000","last_active":"2024-03-06T07:25:14.156749+0000","last_peered":"2024-03-06T07:25:14.156749+0000","last_clean":"2024-03-06T07:25:14.156749+0000","last_became_active":"2024-03-06T07:25:12.142096+0000","last_became_peered":"2024-03-06T07:25:12.142096+0000","last_unstale":"2024-03-06T07:25:14.156749+0000","last_undegraded":"2024-03-06T07:25:14.156749+0000","last_fullsized":"2024-03-06T07:25:14.156749+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:49:20.994442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":256,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228430+0000","last_change":"2024-03-06T07:21:47.553693+0000","last_active":"2024-03-06T07:25:15.228430+0000","last_peered":"2024-03-06T07:25:15.228430+0000","last_clean":"2024-03-06T07:25:15.228430+0000","last_became_active":"2024-03-06T07:21:47.552664+0000","last_became_peered":"2024-03-06T07:21:47.552664+0000","last_unstale":"2024-03-06T07:25:15.228430+0000","last_undegraded":"2024-03-06T07:25:15.228430+0000","last_fullsized":"2024-03-06T07:25:15.228430+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_clean_scrub_stamp":"2024-03-06T07:21:46.362028+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:02.916426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574895,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32580,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339260,"statfs":{"total":96636764160,"available":96603402240,"internally_reserved":0,"allocated":1429504,"data_stored":937323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574893,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32016,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339824,"statfs":{"total":96636764160,"available":96603979776,"internally_reserved":0,"allocated":847872,"data_stored":352731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574894,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32004,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339836,"statfs":{"total":96636764160,"available":96603992064,"internally_reserved":0,"allocated":843776,"data_stored":352729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574895,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32004,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339836,"statfs":{"total":96636764160,"available":96603992064,"internally_reserved":0,"allocated":843776,"data_stored":352709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574894,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32008,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339832,"statfs":{"total":96636764160,"available":96603987968,"internally_reserved":0,"allocated":847872,"data_stored":352751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574894,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32012,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339828,"statfs":{"total":96636764160,"available":96603983872,"internally_reserved":0,"allocated":851968,"data_stored":354251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574895,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32016,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339824,"statfs":{"total":96636764160,"available":96603979776,"internally_reserved":0,"allocated":847872,"data_stored":348567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574894,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32588,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339252,"statfs":{"total":96636764160,"available":96603394048,"internally_reserved":0,"allocated":1433600,"data_stored":943007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:17.346 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:17.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.479+0000 7ff4dade2640 1 Processor -- start 2024-03-06T07:25:17.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.479+0000 7ff4dade2640 1 -- start start 2024-03-06T07:25:17.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.480+0000 7ff4dade2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff4d40d4760 0x7ff4d40b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.480+0000 7ff4dade2640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ff4d40d0a80 0x7ff4d40cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.480+0000 7ff4dade2640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d40d25d0 con 0x7ff4d40b4170 2024-03-06T07:25:17.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.480+0000 7ff4dade2640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4d40d4bf0 con 0x7ff4d40d0a80 2024-03-06T07:25:17.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.480+0000 7ff4dade2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d40ce9e0 con 0x7ff4d40d4760 2024-03-06T07:25:17.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.480+0000 7ff4d8b7e640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff4d40b4170 0x7ff4d40d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:40604/0 (socket says 172.21.15.83:40604) 2024-03-06T07:25:17.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.480+0000 7ff4d8b7e640 1 -- 172.21.15.83:0/180445527 learned_addr learned my addr 172.21.15.83:0/180445527 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:25:17.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.481+0000 7ff4d3fff640 1 --2- 172.21.15.83:0/180445527 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff4d40d4760 0x7ff4d40b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.481+0000 7ff4d937f640 1 --2- 172.21.15.83:0/180445527 >> v2:172.21.15.183:3300/0 conn(0x7ff4d40d0a80 0x7ff4d40cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.481+0000 7ff4d37fe640 1 -- 172.21.15.83:0/180445527 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2041076090 0 0) 0x7ff4d40d25d0 con 0x7ff4d40b4170 2024-03-06T07:25:17.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.481+0000 7ff4d37fe640 1 -- 172.21.15.83:0/180445527 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4c4003520 con 0x7ff4d40b4170 2024-03-06T07:25:17.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.481+0000 7ff4d37fe640 1 -- 172.21.15.83:0/180445527 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2604098703 0 0) 0x7ff4c4003520 con 0x7ff4d40b4170 2024-03-06T07:25:17.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.481+0000 7ff4d37fe640 1 -- 172.21.15.83:0/180445527 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff4d40d4760 msgr2=0x7ff4d40b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:17.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.481+0000 7ff4d37fe640 1 --2- 172.21.15.83:0/180445527 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff4d40d4760 0x7ff4d40b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.481+0000 7ff4d37fe640 1 -- 172.21.15.83:0/180445527 >> v2:172.21.15.183:3300/0 conn(0x7ff4d40d0a80 msgr2=0x7ff4d40cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:17.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.481+0000 7ff4d37fe640 1 --2- 172.21.15.83:0/180445527 >> v2:172.21.15.183:3300/0 conn(0x7ff4d40d0a80 0x7ff4d40cee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.481+0000 7ff4d37fe640 1 -- 172.21.15.83:0/180445527 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d40d2870 con 0x7ff4d40b4170 2024-03-06T07:25:17.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.481+0000 7ff4d37fe640 1 -- 172.21.15.83:0/180445527 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7ff4c8001670 con 0x7ff4d40b4170 2024-03-06T07:25:17.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.481+0000 7ff4d3fff640 1 --2- 172.21.15.83:0/180445527 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff4d40d4760 0x7ff4d40b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:17.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.481+0000 7ff4d937f640 1 --2- 172.21.15.83:0/180445527 >> v2:172.21.15.183:3300/0 conn(0x7ff4d40d0a80 0x7ff4d40cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:17.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.481+0000 7ff4d37fe640 1 -- 172.21.15.83:0/180445527 >> v1:172.21.15.83:6789/0 conn(0x7ff4d40b4170 legacy=0x7ff4d40d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:17.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.481+0000 7ff4d37fe640 1 --2- 172.21.15.83:0/180445527 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff4c40039b0 0x7ff4c4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.481+0000 7ff4d37fe640 1 --2- 172.21.15.83:0/180445527 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff4c4004800 0x7ff4c4004c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.481+0000 7ff4d37fe640 1 --2- 172.21.15.83:0/180445527 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff4c4005140 0x7ff4c400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.481+0000 7ff4d37fe640 1 -- 172.21.15.83:0/180445527 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d40d4bf0 con 0x7ff4c4005140 2024-03-06T07:25:17.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.481+0000 7ff4d37fe640 1 -- 172.21.15.83:0/180445527 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d40ce9e0 con 0x7ff4c40039b0 2024-03-06T07:25:17.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.481+0000 7ff4d37fe640 1 -- 172.21.15.83:0/180445527 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c4004630 con 0x7ff4c4004800 2024-03-06T07:25:17.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.481+0000 7ff4d937f640 1 --2- 172.21.15.83:0/180445527 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff4c4005140 0x7ff4c400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.481+0000 7ff4d8b7e640 1 --2- 172.21.15.83:0/180445527 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff4c40039b0 0x7ff4c4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.481+0000 7ff4d3fff640 1 --2- 172.21.15.83:0/180445527 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff4c4004800 0x7ff4c4004c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.481+0000 7ff4d937f640 1 -- 172.21.15.83:0/180445527 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff4c4004800 msgr2=0x7ff4c4004c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:17.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.481+0000 7ff4d937f640 1 --2- 172.21.15.83:0/180445527 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff4c4004800 0x7ff4c4004c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.481+0000 7ff4d937f640 1 -- 172.21.15.83:0/180445527 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff4c40039b0 msgr2=0x7ff4c4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:17.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.481+0000 7ff4d937f640 1 --2- 172.21.15.83:0/180445527 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff4c40039b0 0x7ff4c4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.482+0000 7ff4d937f640 1 -- 172.21.15.83:0/180445527 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c4003520 con 0x7ff4c4005140 2024-03-06T07:25:17.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.482+0000 7ff4d937f640 1 --2- 172.21.15.83:0/180445527 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff4c4005140 0x7ff4c400b9c0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc00e320 tx=0x7ff4c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b2d3177fb611237 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:17.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.482+0000 7ff4d37fe640 1 -- 172.21.15.83:0/180445527 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4cc009de0 con 0x7ff4c4005140 2024-03-06T07:25:17.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.482+0000 7ff4d37fe640 1 -- 172.21.15.83:0/180445527 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4cc02cde0 con 0x7ff4c4005140 2024-03-06T07:25:17.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.482+0000 7ff4d37fe640 1 -- 172.21.15.83:0/180445527 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4cc003b90 con 0x7ff4c4005140 2024-03-06T07:25:17.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.483+0000 7ff4dade2640 1 -- 172.21.15.83:0/180445527 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff4c4005140 msgr2=0x7ff4c400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:17.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.483+0000 7ff4dade2640 1 --2- 172.21.15.83:0/180445527 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff4c4005140 0x7ff4c400b9c0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc00e320 tx=0x7ff4c4005830 comp rx=0 tx=0).stop 2024-03-06T07:25:17.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.483+0000 7ff4d937f640 1 -- 172.21.15.83:0/180445527 reap_dead start 2024-03-06T07:25:17.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.483+0000 7ff4dade2640 1 -- 172.21.15.83:0/180445527 shutdown_connections 2024-03-06T07:25:17.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.483+0000 7ff4dade2640 1 -- 172.21.15.83:0/180445527 >> 172.21.15.83:0/180445527 conn(0x7ff4d40d3c40 msgr2=0x7ff4d40d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:17.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.483+0000 7ff4dade2640 1 -- 172.21.15.83:0/180445527 shutdown_connections 2024-03-06T07:25:17.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.483+0000 7ff4dade2640 1 -- 172.21.15.83:0/180445527 wait complete. 2024-03-06T07:25:17.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.485+0000 7ff4dade2640 1 Processor -- start 2024-03-06T07:25:17.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.485+0000 7ff4dade2640 1 -- start start 2024-03-06T07:25:17.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.486+0000 7ff4dade2640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff4c4005140 0x7ff4d415f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.486+0000 7ff4dade2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff4d40a13d0 0x7ff4d40cba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.486+0000 7ff4dade2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff4d40cbf80 0x7ff4d40b2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.486+0000 7ff4dade2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d40b4dd0 con 0x7ff4d40a13d0 2024-03-06T07:25:17.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.486+0000 7ff4dade2640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d40b4550 con 0x7ff4c4005140 2024-03-06T07:25:17.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.486+0000 7ff4dade2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d40b4f50 con 0x7ff4d40cbf80 2024-03-06T07:25:17.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.486+0000 7ff4d3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff4d40a13d0 0x7ff4d40cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.486+0000 7ff4d3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff4d40a13d0 0x7ff4d40cba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41400/0 (socket says 172.21.15.83:41400) 2024-03-06T07:25:17.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.487+0000 7ff4d3fff640 1 -- 172.21.15.83:0/2123889377 learned_addr learned my addr 172.21.15.83:0/2123889377 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:17.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.487+0000 7ff4d8b7e640 1 --2- 172.21.15.83:0/2123889377 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff4c4005140 0x7ff4d415f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.487+0000 7ff4d937f640 1 --2- 172.21.15.83:0/2123889377 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff4d40cbf80 0x7ff4d40b2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.487+0000 7ff4d3fff640 1 -- 172.21.15.83:0/2123889377 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff4d40cbf80 msgr2=0x7ff4d40b2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:17.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.487+0000 7ff4d3fff640 1 --2- 172.21.15.83:0/2123889377 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff4d40cbf80 0x7ff4d40b2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.487+0000 7ff4d3fff640 1 -- 172.21.15.83:0/2123889377 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff4c4005140 msgr2=0x7ff4d415f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:17.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.487+0000 7ff4d3fff640 1 --2- 172.21.15.83:0/2123889377 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff4c4005140 0x7ff4d415f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.487+0000 7ff4d3fff640 1 -- 172.21.15.83:0/2123889377 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d41f36b0 con 0x7ff4d40a13d0 2024-03-06T07:25:17.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.487+0000 7ff4d3fff640 1 --2- 172.21.15.83:0/2123889377 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff4d40a13d0 0x7ff4d40cba40 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7ff4c00119d0 tx=0x7ff4c0011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:17.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.487+0000 7ff4d17fa640 1 -- 172.21.15.83:0/2123889377 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c000faf0 con 0x7ff4d40a13d0 2024-03-06T07:25:17.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.487+0000 7ff4d17fa640 1 -- 172.21.15.83:0/2123889377 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4c000fc90 con 0x7ff4d40a13d0 2024-03-06T07:25:17.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.487+0000 7ff4d17fa640 1 -- 172.21.15.83:0/2123889377 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c0004030 con 0x7ff4d40a13d0 2024-03-06T07:25:17.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.487+0000 7ff4dade2640 1 -- 172.21.15.83:0/2123889377 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4d415f5a0 con 0x7ff4d40a13d0 2024-03-06T07:25:17.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.487+0000 7ff4dade2640 1 -- 172.21.15.83:0/2123889377 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4d41f3920 con 0x7ff4d40a13d0 2024-03-06T07:25:17.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.488+0000 7ff4dade2640 1 -- 172.21.15.83:0/2123889377 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4d404f330 con 0x7ff4d40a13d0 2024-03-06T07:25:17.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.489+0000 7ff4d17fa640 1 -- 172.21.15.83:0/2123889377 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff4c0012040 con 0x7ff4d40a13d0 2024-03-06T07:25:17.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.489+0000 7ff4d17fa640 1 --2- 172.21.15.83:0/2123889377 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff4b806b4d0 0x7ff4b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.489+0000 7ff4d17fa640 1 -- 172.21.15.83:0/2123889377 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff4c008a4a0 con 0x7ff4d40a13d0 2024-03-06T07:25:17.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.489+0000 7ff4d8b7e640 1 --2- 172.21.15.83:0/2123889377 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff4b806b4d0 0x7ff4b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.490+0000 7ff4d8b7e640 1 --2- 172.21.15.83:0/2123889377 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff4b806b4d0 0x7ff4b806d990 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8007d90 tx=0x7ff4c80078f0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:17.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.493+0000 7ff4d17fa640 1 -- 172.21.15.83:0/2123889377 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff4c0019070 con 0x7ff4d40a13d0 2024-03-06T07:25:17.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.664+0000 7ff4dade2640 1 -- 172.21.15.83:0/2123889377 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff4d40733b0 con 0x7ff4b806b4d0 2024-03-06T07:25:17.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.670+0000 7ff4d17fa640 1 -- 172.21.15.83:0/2123889377 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252067 (secure 0 0 0) 0x7ff4d40733b0 con 0x7ff4b806b4d0 2024-03-06T07:25:17.674 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:25:17.675 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-06T07:25:17.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.672+0000 7ff4dade2640 1 -- 172.21.15.83:0/2123889377 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff4b806b4d0 msgr2=0x7ff4b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:17.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.672+0000 7ff4dade2640 1 --2- 172.21.15.83:0/2123889377 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff4b806b4d0 0x7ff4b806d990 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8007d90 tx=0x7ff4c80078f0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.672+0000 7ff4dade2640 1 -- 172.21.15.83:0/2123889377 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff4d40a13d0 msgr2=0x7ff4d40cba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:17.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.672+0000 7ff4dade2640 1 --2- 172.21.15.83:0/2123889377 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff4d40a13d0 0x7ff4d40cba40 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7ff4c00119d0 tx=0x7ff4c0011e90 comp rx=0 tx=0).stop 2024-03-06T07:25:17.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.673+0000 7ff4dade2640 1 -- 172.21.15.83:0/2123889377 shutdown_connections 2024-03-06T07:25:17.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.673+0000 7ff4dade2640 1 --2- 172.21.15.83:0/2123889377 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff4b806b4d0 0x7ff4b806d990 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.673+0000 7ff4dade2640 1 --2- 172.21.15.83:0/2123889377 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff4d40cbf80 0x7ff4d40b2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.673+0000 7ff4dade2640 1 --2- 172.21.15.83:0/2123889377 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff4d40a13d0 0x7ff4d40cba40 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.673+0000 7ff4dade2640 1 --2- 172.21.15.83:0/2123889377 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff4c4005140 0x7ff4d415f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.673+0000 7ff4dade2640 1 -- 172.21.15.83:0/2123889377 >> 172.21.15.83:0/2123889377 conn(0x7ff4d40d3c40 msgr2=0x7ff4d40b7010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:17.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.673+0000 7ff4dade2640 1 -- 172.21.15.83:0/2123889377 shutdown_connections 2024-03-06T07:25:17.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.673+0000 7ff4dade2640 1 -- 172.21.15.83:0/2123889377 wait complete. 2024-03-06T07:25:17.685 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":168,"stamp":"2024-03-06T07:25:16.356344+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":114,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":257228,"kb_used_data":7760,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754717492,"statfs":{"total":773094113280,"available":772830711808,"internally_reserved":0,"allocated":7946240,"data_stored":3994068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12757,"internal_metadata":255315499},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.231864"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228800+0000","last_change":"2024-03-06T07:25:13.165487+0000","last_active":"2024-03-06T07:25:15.228800+0000","last_peered":"2024-03-06T07:25:15.228800+0000","last_clean":"2024-03-06T07:25:15.228800+0000","last_became_active":"2024-03-06T07:25:13.165172+0000","last_became_peered":"2024-03-06T07:25:13.165172+0000","last_unstale":"2024-03-06T07:25:15.228800+0000","last_undegraded":"2024-03-06T07:25:15.228800+0000","last_fullsized":"2024-03-06T07:25:15.228800+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:48:56.369026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156914+0000","last_change":"2024-03-06T07:25:13.156380+0000","last_active":"2024-03-06T07:25:14.156914+0000","last_peered":"2024-03-06T07:25:14.156914+0000","last_clean":"2024-03-06T07:25:14.156914+0000","last_became_active":"2024-03-06T07:25:13.156056+0000","last_became_peered":"2024-03-06T07:25:13.156056+0000","last_unstale":"2024-03-06T07:25:14.156914+0000","last_undegraded":"2024-03-06T07:25:14.156914+0000","last_fullsized":"2024-03-06T07:25:14.156914+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:09.679768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228224+0000","last_change":"2024-03-06T07:25:13.163796+0000","last_active":"2024-03-06T07:25:15.228224+0000","last_peered":"2024-03-06T07:25:15.228224+0000","last_clean":"2024-03-06T07:25:15.228224+0000","last_became_active":"2024-03-06T07:25:13.163511+0000","last_became_peered":"2024-03-06T07:25:13.163511+0000","last_unstale":"2024-03-06T07:25:15.228224+0000","last_undegraded":"2024-03-06T07:25:15.228224+0000","last_fullsized":"2024-03-06T07:25:15.228224+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:24.796057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228090+0000","last_change":"2024-03-06T07:25:13.158139+0000","last_active":"2024-03-06T07:25:15.228090+0000","last_peered":"2024-03-06T07:25:15.228090+0000","last_clean":"2024-03-06T07:25:15.228090+0000","last_became_active":"2024-03-06T07:25:13.157866+0000","last_became_peered":"2024-03-06T07:25:13.157866+0000","last_unstale":"2024-03-06T07:25:15.228090+0000","last_undegraded":"2024-03-06T07:25:15.228090+0000","last_fullsized":"2024-03-06T07:25:15.228090+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:49:21.998709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.154891+0000","last_change":"2024-03-06T07:25:13.163633+0000","last_active":"2024-03-06T07:25:14.154891+0000","last_peered":"2024-03-06T07:25:14.154891+0000","last_clean":"2024-03-06T07:25:14.154891+0000","last_became_active":"2024-03-06T07:25:13.163362+0000","last_became_peered":"2024-03-06T07:25:13.163362+0000","last_unstale":"2024-03-06T07:25:14.154891+0000","last_undegraded":"2024-03-06T07:25:14.154891+0000","last_fullsized":"2024-03-06T07:25:14.154891+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:55.118273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156325+0000","last_change":"2024-03-06T07:25:13.154620+0000","last_active":"2024-03-06T07:25:14.156325+0000","last_peered":"2024-03-06T07:25:14.156325+0000","last_clean":"2024-03-06T07:25:14.156325+0000","last_became_active":"2024-03-06T07:25:13.154363+0000","last_became_peered":"2024-03-06T07:25:13.154363+0000","last_unstale":"2024-03-06T07:25:14.156325+0000","last_undegraded":"2024-03-06T07:25:14.156325+0000","last_fullsized":"2024-03-06T07:25:14.156325+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:53:28.757512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.158955+0000","last_change":"2024-03-06T07:25:13.164757+0000","last_active":"2024-03-06T07:25:14.158955+0000","last_peered":"2024-03-06T07:25:14.158955+0000","last_clean":"2024-03-06T07:25:14.158955+0000","last_became_active":"2024-03-06T07:25:13.164418+0000","last_became_peered":"2024-03-06T07:25:13.164418+0000","last_unstale":"2024-03-06T07:25:14.158955+0000","last_undegraded":"2024-03-06T07:25:14.158955+0000","last_fullsized":"2024-03-06T07:25:14.158955+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:36:46.984162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.155994+0000","last_change":"2024-03-06T07:25:13.165057+0000","last_active":"2024-03-06T07:25:14.155994+0000","last_peered":"2024-03-06T07:25:14.155994+0000","last_clean":"2024-03-06T07:25:14.155994+0000","last_became_active":"2024-03-06T07:25:13.164778+0000","last_became_peered":"2024-03-06T07:25:13.164778+0000","last_unstale":"2024-03-06T07:25:14.155994+0000","last_undegraded":"2024-03-06T07:25:14.155994+0000","last_fullsized":"2024-03-06T07:25:14.155994+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:33:40.938375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.155033+0000","last_change":"2024-03-06T07:25:13.155808+0000","last_active":"2024-03-06T07:25:14.155033+0000","last_peered":"2024-03-06T07:25:14.155033+0000","last_clean":"2024-03-06T07:25:14.155033+0000","last_became_active":"2024-03-06T07:25:13.155222+0000","last_became_peered":"2024-03-06T07:25:13.155222+0000","last_unstale":"2024-03-06T07:25:14.155033+0000","last_undegraded":"2024-03-06T07:25:14.155033+0000","last_fullsized":"2024-03-06T07:25:14.155033+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:05:02.223283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156306+0000","last_change":"2024-03-06T07:25:13.157719+0000","last_active":"2024-03-06T07:25:14.156306+0000","last_peered":"2024-03-06T07:25:14.156306+0000","last_clean":"2024-03-06T07:25:14.156306+0000","last_became_active":"2024-03-06T07:25:13.156832+0000","last_became_peered":"2024-03-06T07:25:13.156832+0000","last_unstale":"2024-03-06T07:25:14.156306+0000","last_undegraded":"2024-03-06T07:25:14.156306+0000","last_fullsized":"2024-03-06T07:25:14.156306+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:50.449834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228893+0000","last_change":"2024-03-06T07:25:13.163124+0000","last_active":"2024-03-06T07:25:15.228893+0000","last_peered":"2024-03-06T07:25:15.228893+0000","last_clean":"2024-03-06T07:25:15.228893+0000","last_became_active":"2024-03-06T07:25:13.162725+0000","last_became_peered":"2024-03-06T07:25:13.162725+0000","last_unstale":"2024-03-06T07:25:15.228893+0000","last_undegraded":"2024-03-06T07:25:15.228893+0000","last_fullsized":"2024-03-06T07:25:15.228893+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:24.796057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228220+0000","last_change":"2024-03-06T07:25:13.157110+0000","last_active":"2024-03-06T07:25:15.228220+0000","last_peered":"2024-03-06T07:25:15.228220+0000","last_clean":"2024-03-06T07:25:15.228220+0000","last_became_active":"2024-03-06T07:25:13.156814+0000","last_became_peered":"2024-03-06T07:25:13.156814+0000","last_unstale":"2024-03-06T07:25:15.228220+0000","last_undegraded":"2024-03-06T07:25:15.228220+0000","last_fullsized":"2024-03-06T07:25:15.228220+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:07.232004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.161003+0000","last_change":"2024-03-06T07:25:13.153995+0000","last_active":"2024-03-06T07:25:14.161003+0000","last_peered":"2024-03-06T07:25:14.161003+0000","last_clean":"2024-03-06T07:25:14.161003+0000","last_became_active":"2024-03-06T07:25:13.153593+0000","last_became_peered":"2024-03-06T07:25:13.153593+0000","last_unstale":"2024-03-06T07:25:14.161003+0000","last_undegraded":"2024-03-06T07:25:14.161003+0000","last_fullsized":"2024-03-06T07:25:14.161003+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:02:06.354776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.155174+0000","last_change":"2024-03-06T07:25:13.157216+0000","last_active":"2024-03-06T07:25:14.155174+0000","last_peered":"2024-03-06T07:25:14.155174+0000","last_clean":"2024-03-06T07:25:14.155174+0000","last_became_active":"2024-03-06T07:25:13.156941+0000","last_became_peered":"2024-03-06T07:25:13.156941+0000","last_unstale":"2024-03-06T07:25:14.155174+0000","last_undegraded":"2024-03-06T07:25:14.155174+0000","last_fullsized":"2024-03-06T07:25:14.155174+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:20:21.040939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.160873+0000","last_change":"2024-03-06T07:25:13.158351+0000","last_active":"2024-03-06T07:25:14.160873+0000","last_peered":"2024-03-06T07:25:14.160873+0000","last_clean":"2024-03-06T07:25:14.160873+0000","last_became_active":"2024-03-06T07:25:13.158092+0000","last_became_peered":"2024-03-06T07:25:13.158092+0000","last_unstale":"2024-03-06T07:25:14.160873+0000","last_undegraded":"2024-03-06T07:25:14.160873+0000","last_fullsized":"2024-03-06T07:25:14.160873+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:50.449834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.155410+0000","last_change":"2024-03-06T07:25:13.152334+0000","last_active":"2024-03-06T07:25:14.155410+0000","last_peered":"2024-03-06T07:25:14.155410+0000","last_clean":"2024-03-06T07:25:14.155410+0000","last_became_active":"2024-03-06T07:25:13.152021+0000","last_became_peered":"2024-03-06T07:25:13.152021+0000","last_unstale":"2024-03-06T07:25:14.155410+0000","last_undegraded":"2024-03-06T07:25:14.155410+0000","last_fullsized":"2024-03-06T07:25:14.155410+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:01:49.348156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.155049+0000","last_change":"2024-03-06T07:25:13.155930+0000","last_active":"2024-03-06T07:25:14.155049+0000","last_peered":"2024-03-06T07:25:14.155049+0000","last_clean":"2024-03-06T07:25:14.155049+0000","last_became_active":"2024-03-06T07:25:13.155643+0000","last_became_peered":"2024-03-06T07:25:13.155643+0000","last_unstale":"2024-03-06T07:25:14.155049+0000","last_undegraded":"2024-03-06T07:25:14.155049+0000","last_fullsized":"2024-03-06T07:25:14.155049+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:59:52.753418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228174+0000","last_change":"2024-03-06T07:25:13.153982+0000","last_active":"2024-03-06T07:25:15.228174+0000","last_peered":"2024-03-06T07:25:15.228174+0000","last_clean":"2024-03-06T07:25:15.228174+0000","last_became_active":"2024-03-06T07:25:13.153358+0000","last_became_peered":"2024-03-06T07:25:13.153358+0000","last_unstale":"2024-03-06T07:25:15.228174+0000","last_undegraded":"2024-03-06T07:25:15.228174+0000","last_fullsized":"2024-03-06T07:25:15.228174+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:59:52.753418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156552+0000","last_change":"2024-03-06T07:25:13.157124+0000","last_active":"2024-03-06T07:25:14.156552+0000","last_peered":"2024-03-06T07:25:14.156552+0000","last_clean":"2024-03-06T07:25:14.156552+0000","last_became_active":"2024-03-06T07:25:13.156610+0000","last_became_peered":"2024-03-06T07:25:13.156610+0000","last_unstale":"2024-03-06T07:25:14.156552+0000","last_undegraded":"2024-03-06T07:25:14.156552+0000","last_fullsized":"2024-03-06T07:25:14.156552+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:37.851591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.161066+0000","last_change":"2024-03-06T07:25:13.164176+0000","last_active":"2024-03-06T07:25:14.161066+0000","last_peered":"2024-03-06T07:25:14.161066+0000","last_clean":"2024-03-06T07:25:14.161066+0000","last_became_active":"2024-03-06T07:25:13.163099+0000","last_became_peered":"2024-03-06T07:25:13.163099+0000","last_unstale":"2024-03-06T07:25:14.161066+0000","last_undegraded":"2024-03-06T07:25:14.161066+0000","last_fullsized":"2024-03-06T07:25:14.161066+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:40.549085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156102+0000","last_change":"2024-03-06T07:25:13.155225+0000","last_active":"2024-03-06T07:25:14.156102+0000","last_peered":"2024-03-06T07:25:14.156102+0000","last_clean":"2024-03-06T07:25:14.156102+0000","last_became_active":"2024-03-06T07:25:13.154937+0000","last_became_peered":"2024-03-06T07:25:13.154937+0000","last_unstale":"2024-03-06T07:25:14.156102+0000","last_undegraded":"2024-03-06T07:25:14.156102+0000","last_fullsized":"2024-03-06T07:25:14.156102+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:49.755068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.155597+0000","last_change":"2024-03-06T07:25:13.157186+0000","last_active":"2024-03-06T07:25:14.155597+0000","last_peered":"2024-03-06T07:25:14.155597+0000","last_clean":"2024-03-06T07:25:14.155597+0000","last_became_active":"2024-03-06T07:25:13.156842+0000","last_became_peered":"2024-03-06T07:25:13.156842+0000","last_unstale":"2024-03-06T07:25:14.155597+0000","last_undegraded":"2024-03-06T07:25:14.155597+0000","last_fullsized":"2024-03-06T07:25:14.155597+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:15:35.393776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228449+0000","last_change":"2024-03-06T07:25:13.169632+0000","last_active":"2024-03-06T07:25:15.228449+0000","last_peered":"2024-03-06T07:25:15.228449+0000","last_clean":"2024-03-06T07:25:15.228449+0000","last_became_active":"2024-03-06T07:25:13.167476+0000","last_became_peered":"2024-03-06T07:25:13.167476+0000","last_unstale":"2024-03-06T07:25:15.228449+0000","last_undegraded":"2024-03-06T07:25:15.228449+0000","last_fullsized":"2024-03-06T07:25:15.228449+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:13.108157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228467+0000","last_change":"2024-03-06T07:25:13.164595+0000","last_active":"2024-03-06T07:25:15.228467+0000","last_peered":"2024-03-06T07:25:15.228467+0000","last_clean":"2024-03-06T07:25:15.228467+0000","last_became_active":"2024-03-06T07:25:13.164313+0000","last_became_peered":"2024-03-06T07:25:13.164313+0000","last_unstale":"2024-03-06T07:25:15.228467+0000","last_undegraded":"2024-03-06T07:25:15.228467+0000","last_fullsized":"2024-03-06T07:25:15.228467+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:24.956262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.155832+0000","last_change":"2024-03-06T07:25:13.158874+0000","last_active":"2024-03-06T07:25:14.155832+0000","last_peered":"2024-03-06T07:25:14.155832+0000","last_clean":"2024-03-06T07:25:14.155832+0000","last_became_active":"2024-03-06T07:25:13.158322+0000","last_became_peered":"2024-03-06T07:25:13.158322+0000","last_unstale":"2024-03-06T07:25:14.155832+0000","last_undegraded":"2024-03-06T07:25:14.155832+0000","last_fullsized":"2024-03-06T07:25:14.155832+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:27.680894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.159090+0000","last_change":"2024-03-06T07:25:13.160666+0000","last_active":"2024-03-06T07:25:14.159090+0000","last_peered":"2024-03-06T07:25:14.159090+0000","last_clean":"2024-03-06T07:25:14.159090+0000","last_became_active":"2024-03-06T07:25:13.160380+0000","last_became_peered":"2024-03-06T07:25:13.160380+0000","last_unstale":"2024-03-06T07:25:14.159090+0000","last_undegraded":"2024-03-06T07:25:14.159090+0000","last_fullsized":"2024-03-06T07:25:14.159090+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:01.195841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228349+0000","last_change":"2024-03-06T07:25:13.160230+0000","last_active":"2024-03-06T07:25:15.228349+0000","last_peered":"2024-03-06T07:25:15.228349+0000","last_clean":"2024-03-06T07:25:15.228349+0000","last_became_active":"2024-03-06T07:25:13.159059+0000","last_became_peered":"2024-03-06T07:25:13.159059+0000","last_unstale":"2024-03-06T07:25:15.228349+0000","last_undegraded":"2024-03-06T07:25:15.228349+0000","last_fullsized":"2024-03-06T07:25:15.228349+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:46:28.717814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156175+0000","last_change":"2024-03-06T07:25:13.157587+0000","last_active":"2024-03-06T07:25:14.156175+0000","last_peered":"2024-03-06T07:25:14.156175+0000","last_clean":"2024-03-06T07:25:14.156175+0000","last_became_active":"2024-03-06T07:25:13.157280+0000","last_became_peered":"2024-03-06T07:25:13.157280+0000","last_unstale":"2024-03-06T07:25:14.156175+0000","last_undegraded":"2024-03-06T07:25:14.156175+0000","last_fullsized":"2024-03-06T07:25:14.156175+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:27:22.280558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228758+0000","last_change":"2024-03-06T07:25:13.160500+0000","last_active":"2024-03-06T07:25:15.228758+0000","last_peered":"2024-03-06T07:25:15.228758+0000","last_clean":"2024-03-06T07:25:15.228758+0000","last_became_active":"2024-03-06T07:25:13.160121+0000","last_became_peered":"2024-03-06T07:25:13.160121+0000","last_unstale":"2024-03-06T07:25:15.228758+0000","last_undegraded":"2024-03-06T07:25:15.228758+0000","last_fullsized":"2024-03-06T07:25:15.228758+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:05.941938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.159034+0000","last_change":"2024-03-06T07:25:13.158496+0000","last_active":"2024-03-06T07:25:14.159034+0000","last_peered":"2024-03-06T07:25:14.159034+0000","last_clean":"2024-03-06T07:25:14.159034+0000","last_became_active":"2024-03-06T07:25:13.158237+0000","last_became_peered":"2024-03-06T07:25:13.158237+0000","last_unstale":"2024-03-06T07:25:14.159034+0000","last_undegraded":"2024-03-06T07:25:14.159034+0000","last_fullsized":"2024-03-06T07:25:14.159034+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:08:35.657078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.159186+0000","last_change":"2024-03-06T07:25:13.162268+0000","last_active":"2024-03-06T07:25:14.159186+0000","last_peered":"2024-03-06T07:25:14.159186+0000","last_clean":"2024-03-06T07:25:14.159186+0000","last_became_active":"2024-03-06T07:25:13.162021+0000","last_became_peered":"2024-03-06T07:25:13.162021+0000","last_unstale":"2024-03-06T07:25:14.159186+0000","last_undegraded":"2024-03-06T07:25:14.159186+0000","last_fullsized":"2024-03-06T07:25:14.159186+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:33:11.084604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156035+0000","last_change":"2024-03-06T07:25:13.156062+0000","last_active":"2024-03-06T07:25:14.156035+0000","last_peered":"2024-03-06T07:25:14.156035+0000","last_clean":"2024-03-06T07:25:14.156035+0000","last_became_active":"2024-03-06T07:25:13.155807+0000","last_became_peered":"2024-03-06T07:25:13.155807+0000","last_unstale":"2024-03-06T07:25:14.156035+0000","last_undegraded":"2024-03-06T07:25:14.156035+0000","last_fullsized":"2024-03-06T07:25:14.156035+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:28:36.984619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228779+0000","last_change":"2024-03-06T07:25:13.165720+0000","last_active":"2024-03-06T07:25:15.228779+0000","last_peered":"2024-03-06T07:25:15.228779+0000","last_clean":"2024-03-06T07:25:15.228779+0000","last_became_active":"2024-03-06T07:25:13.165367+0000","last_became_peered":"2024-03-06T07:25:13.165367+0000","last_unstale":"2024-03-06T07:25:15.228779+0000","last_undegraded":"2024-03-06T07:25:15.228779+0000","last_fullsized":"2024-03-06T07:25:15.228779+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:37.968763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"68'4","reported_seq":18,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228919+0000","last_change":"2024-03-06T07:25:12.150041+0000","last_active":"2024-03-06T07:25:15.228919+0000","last_peered":"2024-03-06T07:25:15.228919+0000","last_clean":"2024-03-06T07:25:15.228919+0000","last_became_active":"2024-03-06T07:25:12.149784+0000","last_became_peered":"2024-03-06T07:25:12.149784+0000","last_unstale":"2024-03-06T07:25:15.228919+0000","last_undegraded":"2024-03-06T07:25:15.228919+0000","last_fullsized":"2024-03-06T07:25:15.228919+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:49.445567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.155153+0000","last_change":"2024-03-06T07:25:13.155523+0000","last_active":"2024-03-06T07:25:14.155153+0000","last_peered":"2024-03-06T07:25:14.155153+0000","last_clean":"2024-03-06T07:25:14.155153+0000","last_became_active":"2024-03-06T07:25:13.154872+0000","last_became_peered":"2024-03-06T07:25:13.154872+0000","last_unstale":"2024-03-06T07:25:14.155153+0000","last_undegraded":"2024-03-06T07:25:14.155153+0000","last_fullsized":"2024-03-06T07:25:14.155153+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:07.232004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.161070+0000","last_change":"2024-03-06T07:25:12.140136+0000","last_active":"2024-03-06T07:25:14.161070+0000","last_peered":"2024-03-06T07:25:14.161070+0000","last_clean":"2024-03-06T07:25:14.161070+0000","last_became_active":"2024-03-06T07:25:12.139794+0000","last_became_peered":"2024-03-06T07:25:12.139794+0000","last_unstale":"2024-03-06T07:25:14.161070+0000","last_undegraded":"2024-03-06T07:25:14.161070+0000","last_fullsized":"2024-03-06T07:25:14.161070+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:39.546776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156755+0000","last_change":"2024-03-06T07:25:13.165146+0000","last_active":"2024-03-06T07:25:14.156755+0000","last_peered":"2024-03-06T07:25:14.156755+0000","last_clean":"2024-03-06T07:25:14.156755+0000","last_became_active":"2024-03-06T07:25:13.164825+0000","last_became_peered":"2024-03-06T07:25:13.164825+0000","last_unstale":"2024-03-06T07:25:14.156755+0000","last_undegraded":"2024-03-06T07:25:14.156755+0000","last_fullsized":"2024-03-06T07:25:14.156755+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:49:30.437740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156134+0000","last_change":"2024-03-06T07:25:12.141294+0000","last_active":"2024-03-06T07:25:14.156134+0000","last_peered":"2024-03-06T07:25:14.156134+0000","last_clean":"2024-03-06T07:25:14.156134+0000","last_became_active":"2024-03-06T07:25:12.141044+0000","last_became_peered":"2024-03-06T07:25:12.141044+0000","last_unstale":"2024-03-06T07:25:14.156134+0000","last_undegraded":"2024-03-06T07:25:14.156134+0000","last_fullsized":"2024-03-06T07:25:14.156134+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:53.918943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228860+0000","last_change":"2024-03-06T07:25:13.167242+0000","last_active":"2024-03-06T07:25:15.228860+0000","last_peered":"2024-03-06T07:25:15.228860+0000","last_clean":"2024-03-06T07:25:15.228860+0000","last_became_active":"2024-03-06T07:25:13.166935+0000","last_became_peered":"2024-03-06T07:25:13.166935+0000","last_unstale":"2024-03-06T07:25:15.228860+0000","last_undegraded":"2024-03-06T07:25:15.228860+0000","last_fullsized":"2024-03-06T07:25:15.228860+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:03.696343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156910+0000","last_change":"2024-03-06T07:25:12.144105+0000","last_active":"2024-03-06T07:25:14.156910+0000","last_peered":"2024-03-06T07:25:14.156910+0000","last_clean":"2024-03-06T07:25:14.156910+0000","last_became_active":"2024-03-06T07:25:12.143797+0000","last_became_peered":"2024-03-06T07:25:12.143797+0000","last_unstale":"2024-03-06T07:25:14.156910+0000","last_undegraded":"2024-03-06T07:25:14.156910+0000","last_fullsized":"2024-03-06T07:25:14.156910+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:54.114006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.159082+0000","last_change":"2024-03-06T07:25:13.161312+0000","last_active":"2024-03-06T07:25:14.159082+0000","last_peered":"2024-03-06T07:25:14.159082+0000","last_clean":"2024-03-06T07:25:14.159082+0000","last_became_active":"2024-03-06T07:25:13.161037+0000","last_became_peered":"2024-03-06T07:25:13.161037+0000","last_unstale":"2024-03-06T07:25:14.159082+0000","last_undegraded":"2024-03-06T07:25:14.159082+0000","last_fullsized":"2024-03-06T07:25:14.159082+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:44:51.540913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228853+0000","last_change":"2024-03-06T07:25:12.142783+0000","last_active":"2024-03-06T07:25:15.228853+0000","last_peered":"2024-03-06T07:25:15.228853+0000","last_clean":"2024-03-06T07:25:15.228853+0000","last_became_active":"2024-03-06T07:25:12.142498+0000","last_became_peered":"2024-03-06T07:25:12.142498+0000","last_unstale":"2024-03-06T07:25:15.228853+0000","last_undegraded":"2024-03-06T07:25:15.228853+0000","last_fullsized":"2024-03-06T07:25:15.228853+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:23.951995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.155941+0000","last_change":"2024-03-06T07:25:13.158674+0000","last_active":"2024-03-06T07:25:14.155941+0000","last_peered":"2024-03-06T07:25:14.155941+0000","last_clean":"2024-03-06T07:25:14.155941+0000","last_became_active":"2024-03-06T07:25:13.158369+0000","last_became_peered":"2024-03-06T07:25:13.158369+0000","last_unstale":"2024-03-06T07:25:14.155941+0000","last_undegraded":"2024-03-06T07:25:14.155941+0000","last_fullsized":"2024-03-06T07:25:14.155941+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:27:59.088271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156292+0000","last_change":"2024-03-06T07:25:12.148361+0000","last_active":"2024-03-06T07:25:14.156292+0000","last_peered":"2024-03-06T07:25:14.156292+0000","last_clean":"2024-03-06T07:25:14.156292+0000","last_became_active":"2024-03-06T07:25:12.148055+0000","last_became_peered":"2024-03-06T07:25:12.148055+0000","last_unstale":"2024-03-06T07:25:14.156292+0000","last_undegraded":"2024-03-06T07:25:14.156292+0000","last_fullsized":"2024-03-06T07:25:14.156292+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:52:02.481328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.154647+0000","last_change":"2024-03-06T07:25:13.157940+0000","last_active":"2024-03-06T07:25:14.154647+0000","last_peered":"2024-03-06T07:25:14.154647+0000","last_clean":"2024-03-06T07:25:14.154647+0000","last_became_active":"2024-03-06T07:25:13.157682+0000","last_became_peered":"2024-03-06T07:25:13.157682+0000","last_unstale":"2024-03-06T07:25:14.154647+0000","last_undegraded":"2024-03-06T07:25:14.154647+0000","last_fullsized":"2024-03-06T07:25:14.154647+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:06.723675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.227986+0000","last_change":"2024-03-06T07:25:12.144427+0000","last_active":"2024-03-06T07:25:15.227986+0000","last_peered":"2024-03-06T07:25:15.227986+0000","last_clean":"2024-03-06T07:25:15.227986+0000","last_became_active":"2024-03-06T07:25:12.144166+0000","last_became_peered":"2024-03-06T07:25:12.144166+0000","last_unstale":"2024-03-06T07:25:15.227986+0000","last_undegraded":"2024-03-06T07:25:15.227986+0000","last_fullsized":"2024-03-06T07:25:15.227986+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:41:07.504129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.160961+0000","last_change":"2024-03-06T07:25:13.160676+0000","last_active":"2024-03-06T07:25:14.160961+0000","last_peered":"2024-03-06T07:25:14.160961+0000","last_clean":"2024-03-06T07:25:14.160961+0000","last_became_active":"2024-03-06T07:25:13.160341+0000","last_became_peered":"2024-03-06T07:25:13.160341+0000","last_unstale":"2024-03-06T07:25:14.160961+0000","last_undegraded":"2024-03-06T07:25:14.160961+0000","last_fullsized":"2024-03-06T07:25:14.160961+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:32.749309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.154646+0000","last_change":"2024-03-06T07:25:12.145436+0000","last_active":"2024-03-06T07:25:14.154646+0000","last_peered":"2024-03-06T07:25:14.154646+0000","last_clean":"2024-03-06T07:25:14.154646+0000","last_became_active":"2024-03-06T07:25:12.145165+0000","last_became_peered":"2024-03-06T07:25:12.145165+0000","last_unstale":"2024-03-06T07:25:14.154646+0000","last_undegraded":"2024-03-06T07:25:14.154646+0000","last_fullsized":"2024-03-06T07:25:14.154646+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:51.126409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.159103+0000","last_change":"2024-03-06T07:25:13.159948+0000","last_active":"2024-03-06T07:25:14.159103+0000","last_peered":"2024-03-06T07:25:14.159103+0000","last_clean":"2024-03-06T07:25:14.159103+0000","last_became_active":"2024-03-06T07:25:13.159672+0000","last_became_peered":"2024-03-06T07:25:13.159672+0000","last_unstale":"2024-03-06T07:25:14.159103+0000","last_undegraded":"2024-03-06T07:25:14.159103+0000","last_fullsized":"2024-03-06T07:25:14.159103+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:44:28.285425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.155696+0000","last_change":"2024-03-06T07:25:12.140394+0000","last_active":"2024-03-06T07:25:14.155696+0000","last_peered":"2024-03-06T07:25:14.155696+0000","last_clean":"2024-03-06T07:25:14.155696+0000","last_became_active":"2024-03-06T07:25:12.140118+0000","last_became_peered":"2024-03-06T07:25:12.140118+0000","last_unstale":"2024-03-06T07:25:14.155696+0000","last_undegraded":"2024-03-06T07:25:14.155696+0000","last_fullsized":"2024-03-06T07:25:14.155696+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:13.758634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228811+0000","last_change":"2024-03-06T07:25:13.169371+0000","last_active":"2024-03-06T07:25:15.228811+0000","last_peered":"2024-03-06T07:25:15.228811+0000","last_clean":"2024-03-06T07:25:15.228811+0000","last_became_active":"2024-03-06T07:25:13.169085+0000","last_became_peered":"2024-03-06T07:25:13.169085+0000","last_unstale":"2024-03-06T07:25:15.228811+0000","last_undegraded":"2024-03-06T07:25:15.228811+0000","last_fullsized":"2024-03-06T07:25:15.228811+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:10.404106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156948+0000","last_change":"2024-03-06T07:25:12.145549+0000","last_active":"2024-03-06T07:25:14.156948+0000","last_peered":"2024-03-06T07:25:14.156948+0000","last_clean":"2024-03-06T07:25:14.156948+0000","last_became_active":"2024-03-06T07:25:12.144847+0000","last_became_peered":"2024-03-06T07:25:12.144847+0000","last_unstale":"2024-03-06T07:25:14.156948+0000","last_undegraded":"2024-03-06T07:25:14.156948+0000","last_fullsized":"2024-03-06T07:25:14.156948+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:05.719408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156729+0000","last_change":"2024-03-06T07:25:13.157940+0000","last_active":"2024-03-06T07:25:14.156729+0000","last_peered":"2024-03-06T07:25:14.156729+0000","last_clean":"2024-03-06T07:25:14.156729+0000","last_became_active":"2024-03-06T07:25:13.157695+0000","last_became_peered":"2024-03-06T07:25:13.157695+0000","last_unstale":"2024-03-06T07:25:14.156729+0000","last_undegraded":"2024-03-06T07:25:14.156729+0000","last_fullsized":"2024-03-06T07:25:14.156729+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:40.551042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"68'3","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228450+0000","last_change":"2024-03-06T07:25:12.147266+0000","last_active":"2024-03-06T07:25:15.228450+0000","last_peered":"2024-03-06T07:25:15.228450+0000","last_clean":"2024-03-06T07:25:15.228450+0000","last_became_active":"2024-03-06T07:25:12.146961+0000","last_became_peered":"2024-03-06T07:25:12.146961+0000","last_unstale":"2024-03-06T07:25:15.228450+0000","last_undegraded":"2024-03-06T07:25:15.228450+0000","last_fullsized":"2024-03-06T07:25:15.228450+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:33:10.080337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.154820+0000","last_change":"2024-03-06T07:25:13.160320+0000","last_active":"2024-03-06T07:25:14.154820+0000","last_peered":"2024-03-06T07:25:14.154820+0000","last_clean":"2024-03-06T07:25:14.154820+0000","last_became_active":"2024-03-06T07:25:13.159684+0000","last_became_peered":"2024-03-06T07:25:13.159684+0000","last_unstale":"2024-03-06T07:25:14.154820+0000","last_undegraded":"2024-03-06T07:25:14.154820+0000","last_fullsized":"2024-03-06T07:25:14.154820+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:52:03.485595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"68'6","reported_seq":20,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228308+0000","last_change":"2024-03-06T07:25:12.145325+0000","last_active":"2024-03-06T07:25:15.228308+0000","last_peered":"2024-03-06T07:25:15.228308+0000","last_clean":"2024-03-06T07:25:15.228308+0000","last_became_active":"2024-03-06T07:25:12.145043+0000","last_became_peered":"2024-03-06T07:25:12.145043+0000","last_unstale":"2024-03-06T07:25:15.228308+0000","last_undegraded":"2024-03-06T07:25:15.228308+0000","last_fullsized":"2024-03-06T07:25:15.228308+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:01.865233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228511+0000","last_change":"2024-03-06T07:25:13.161899+0000","last_active":"2024-03-06T07:25:15.228511+0000","last_peered":"2024-03-06T07:25:15.228511+0000","last_clean":"2024-03-06T07:25:15.228511+0000","last_became_active":"2024-03-06T07:25:13.161618+0000","last_became_peered":"2024-03-06T07:25:13.161618+0000","last_unstale":"2024-03-06T07:25:15.228511+0000","last_undegraded":"2024-03-06T07:25:15.228511+0000","last_fullsized":"2024-03-06T07:25:15.228511+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:13.723493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228203+0000","last_change":"2024-03-06T07:25:12.144337+0000","last_active":"2024-03-06T07:25:15.228203+0000","last_peered":"2024-03-06T07:25:15.228203+0000","last_clean":"2024-03-06T07:25:15.228203+0000","last_became_active":"2024-03-06T07:25:12.144066+0000","last_became_peered":"2024-03-06T07:25:12.144066+0000","last_unstale":"2024-03-06T07:25:15.228203+0000","last_undegraded":"2024-03-06T07:25:15.228203+0000","last_fullsized":"2024-03-06T07:25:15.228203+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:30.030141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.159130+0000","last_change":"2024-03-06T07:25:13.157351+0000","last_active":"2024-03-06T07:25:14.159130+0000","last_peered":"2024-03-06T07:25:14.159130+0000","last_clean":"2024-03-06T07:25:14.159130+0000","last_became_active":"2024-03-06T07:25:13.157071+0000","last_became_peered":"2024-03-06T07:25:13.157071+0000","last_unstale":"2024-03-06T07:25:14.159130+0000","last_undegraded":"2024-03-06T07:25:14.159130+0000","last_fullsized":"2024-03-06T07:25:14.159130+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:59.805941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156408+0000","last_change":"2024-03-06T07:25:12.147133+0000","last_active":"2024-03-06T07:25:14.156408+0000","last_peered":"2024-03-06T07:25:14.156408+0000","last_clean":"2024-03-06T07:25:14.156408+0000","last_became_active":"2024-03-06T07:25:12.146865+0000","last_became_peered":"2024-03-06T07:25:12.146865+0000","last_unstale":"2024-03-06T07:25:14.156408+0000","last_undegraded":"2024-03-06T07:25:14.156408+0000","last_fullsized":"2024-03-06T07:25:14.156408+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:59:55.193963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.160853+0000","last_change":"2024-03-06T07:25:13.159853+0000","last_active":"2024-03-06T07:25:14.160853+0000","last_peered":"2024-03-06T07:25:14.160853+0000","last_clean":"2024-03-06T07:25:14.160853+0000","last_became_active":"2024-03-06T07:25:13.159589+0000","last_became_peered":"2024-03-06T07:25:13.159589+0000","last_unstale":"2024-03-06T07:25:14.160853+0000","last_undegraded":"2024-03-06T07:25:14.160853+0000","last_fullsized":"2024-03-06T07:25:14.160853+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:27:22.280558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.159073+0000","last_change":"2024-03-06T07:25:12.145548+0000","last_active":"2024-03-06T07:25:14.159073+0000","last_peered":"2024-03-06T07:25:14.159073+0000","last_clean":"2024-03-06T07:25:14.159073+0000","last_became_active":"2024-03-06T07:25:12.145214+0000","last_became_peered":"2024-03-06T07:25:12.145214+0000","last_unstale":"2024-03-06T07:25:14.159073+0000","last_undegraded":"2024-03-06T07:25:14.159073+0000","last_fullsized":"2024-03-06T07:25:14.159073+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:35.931140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.154972+0000","last_change":"2024-03-06T07:25:13.156771+0000","last_active":"2024-03-06T07:25:14.154972+0000","last_peered":"2024-03-06T07:25:14.154972+0000","last_clean":"2024-03-06T07:25:14.154972+0000","last_became_active":"2024-03-06T07:25:13.156472+0000","last_became_peered":"2024-03-06T07:25:13.156472+0000","last_unstale":"2024-03-06T07:25:14.154972+0000","last_undegraded":"2024-03-06T07:25:14.154972+0000","last_fullsized":"2024-03-06T07:25:14.154972+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:59:56.198230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"68'8","reported_seq":22,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.155528+0000","last_change":"2024-03-06T07:25:12.143217+0000","last_active":"2024-03-06T07:25:14.155528+0000","last_peered":"2024-03-06T07:25:14.155528+0000","last_clean":"2024-03-06T07:25:14.155528+0000","last_became_active":"2024-03-06T07:25:12.142853+0000","last_became_peered":"2024-03-06T07:25:12.142853+0000","last_unstale":"2024-03-06T07:25:14.155528+0000","last_undegraded":"2024-03-06T07:25:14.155528+0000","last_fullsized":"2024-03-06T07:25:14.155528+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:28:18.153551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156336+0000","last_change":"2024-03-06T07:25:13.155028+0000","last_active":"2024-03-06T07:25:14.156336+0000","last_peered":"2024-03-06T07:25:14.156336+0000","last_clean":"2024-03-06T07:25:14.156336+0000","last_became_active":"2024-03-06T07:25:13.154744+0000","last_became_peered":"2024-03-06T07:25:13.154744+0000","last_unstale":"2024-03-06T07:25:14.156336+0000","last_undegraded":"2024-03-06T07:25:14.156336+0000","last_fullsized":"2024-03-06T07:25:14.156336+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:32.749309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156986+0000","last_change":"2024-03-06T07:25:12.146430+0000","last_active":"2024-03-06T07:25:14.156986+0000","last_peered":"2024-03-06T07:25:14.156986+0000","last_clean":"2024-03-06T07:25:14.156986+0000","last_became_active":"2024-03-06T07:25:12.146111+0000","last_became_peered":"2024-03-06T07:25:12.146111+0000","last_unstale":"2024-03-06T07:25:14.156986+0000","last_undegraded":"2024-03-06T07:25:14.156986+0000","last_fullsized":"2024-03-06T07:25:14.156986+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:51:20.859880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228986+0000","last_change":"2024-03-06T07:25:13.167953+0000","last_active":"2024-03-06T07:25:15.228986+0000","last_peered":"2024-03-06T07:25:15.228986+0000","last_clean":"2024-03-06T07:25:15.228986+0000","last_became_active":"2024-03-06T07:25:13.167703+0000","last_became_peered":"2024-03-06T07:25:13.167703+0000","last_unstale":"2024-03-06T07:25:15.228986+0000","last_undegraded":"2024-03-06T07:25:15.228986+0000","last_fullsized":"2024-03-06T07:25:15.228986+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:19:01.252079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156426+0000","last_change":"2024-03-06T07:25:12.148744+0000","last_active":"2024-03-06T07:25:14.156426+0000","last_peered":"2024-03-06T07:25:14.156426+0000","last_clean":"2024-03-06T07:25:14.156426+0000","last_became_active":"2024-03-06T07:25:12.148418+0000","last_became_peered":"2024-03-06T07:25:12.148418+0000","last_unstale":"2024-03-06T07:25:14.156426+0000","last_undegraded":"2024-03-06T07:25:14.156426+0000","last_fullsized":"2024-03-06T07:25:14.156426+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:33:39.934108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228297+0000","last_change":"2024-03-06T07:25:13.156902+0000","last_active":"2024-03-06T07:25:15.228297+0000","last_peered":"2024-03-06T07:25:15.228297+0000","last_clean":"2024-03-06T07:25:15.228297+0000","last_became_active":"2024-03-06T07:25:13.156626+0000","last_became_peered":"2024-03-06T07:25:13.156626+0000","last_unstale":"2024-03-06T07:25:15.228297+0000","last_undegraded":"2024-03-06T07:25:15.228297+0000","last_fullsized":"2024-03-06T07:25:15.228297+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:20:21.040939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156828+0000","last_change":"2024-03-06T07:25:12.148664+0000","last_active":"2024-03-06T07:25:14.156828+0000","last_peered":"2024-03-06T07:25:14.156828+0000","last_clean":"2024-03-06T07:25:14.156828+0000","last_became_active":"2024-03-06T07:25:12.147501+0000","last_became_peered":"2024-03-06T07:25:12.147501+0000","last_unstale":"2024-03-06T07:25:14.156828+0000","last_undegraded":"2024-03-06T07:25:14.156828+0000","last_fullsized":"2024-03-06T07:25:14.156828+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:59:51.749151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.155041+0000","last_change":"2024-03-06T07:25:13.160545+0000","last_active":"2024-03-06T07:25:14.155041+0000","last_peered":"2024-03-06T07:25:14.155041+0000","last_clean":"2024-03-06T07:25:14.155041+0000","last_became_active":"2024-03-06T07:25:13.160053+0000","last_became_peered":"2024-03-06T07:25:13.160053+0000","last_unstale":"2024-03-06T07:25:14.155041+0000","last_undegraded":"2024-03-06T07:25:14.155041+0000","last_fullsized":"2024-03-06T07:25:14.155041+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:49:21.998709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228755+0000","last_change":"2024-03-06T07:25:12.151623+0000","last_active":"2024-03-06T07:25:15.228755+0000","last_peered":"2024-03-06T07:25:15.228755+0000","last_clean":"2024-03-06T07:25:15.228755+0000","last_became_active":"2024-03-06T07:25:12.151361+0000","last_became_peered":"2024-03-06T07:25:12.151361+0000","last_unstale":"2024-03-06T07:25:15.228755+0000","last_undegraded":"2024-03-06T07:25:15.228755+0000","last_fullsized":"2024-03-06T07:25:15.228755+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:27:21.276291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228797+0000","last_change":"2024-03-06T07:25:13.162254+0000","last_active":"2024-03-06T07:25:15.228797+0000","last_peered":"2024-03-06T07:25:15.228797+0000","last_clean":"2024-03-06T07:25:15.228797+0000","last_became_active":"2024-03-06T07:25:13.161990+0000","last_became_peered":"2024-03-06T07:25:13.161990+0000","last_unstale":"2024-03-06T07:25:15.228797+0000","last_undegraded":"2024-03-06T07:25:15.228797+0000","last_fullsized":"2024-03-06T07:25:15.228797+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:09:24.521844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156442+0000","last_change":"2024-03-06T07:25:12.148747+0000","last_active":"2024-03-06T07:25:14.156442+0000","last_peered":"2024-03-06T07:25:14.156442+0000","last_clean":"2024-03-06T07:25:14.156442+0000","last_became_active":"2024-03-06T07:25:12.147762+0000","last_became_peered":"2024-03-06T07:25:12.147762+0000","last_unstale":"2024-03-06T07:25:14.156442+0000","last_undegraded":"2024-03-06T07:25:14.156442+0000","last_fullsized":"2024-03-06T07:25:14.156442+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:05:01.219016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228646+0000","last_change":"2024-03-06T07:25:13.159680+0000","last_active":"2024-03-06T07:25:15.228646+0000","last_peered":"2024-03-06T07:25:15.228646+0000","last_clean":"2024-03-06T07:25:15.228646+0000","last_became_active":"2024-03-06T07:25:13.159424+0000","last_became_peered":"2024-03-06T07:25:13.159424+0000","last_unstale":"2024-03-06T07:25:15.228646+0000","last_undegraded":"2024-03-06T07:25:15.228646+0000","last_fullsized":"2024-03-06T07:25:15.228646+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:19:01.252079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.159208+0000","last_change":"2024-03-06T07:25:12.149069+0000","last_active":"2024-03-06T07:25:14.159208+0000","last_peered":"2024-03-06T07:25:14.159208+0000","last_clean":"2024-03-06T07:25:14.159208+0000","last_became_active":"2024-03-06T07:25:12.148582+0000","last_became_peered":"2024-03-06T07:25:12.148582+0000","last_unstale":"2024-03-06T07:25:14.159208+0000","last_undegraded":"2024-03-06T07:25:14.159208+0000","last_fullsized":"2024-03-06T07:25:14.159208+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:01:48.343889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228701+0000","last_change":"2024-03-06T07:25:13.156792+0000","last_active":"2024-03-06T07:25:15.228701+0000","last_peered":"2024-03-06T07:25:15.228701+0000","last_clean":"2024-03-06T07:25:15.228701+0000","last_became_active":"2024-03-06T07:25:13.156508+0000","last_became_peered":"2024-03-06T07:25:13.156508+0000","last_unstale":"2024-03-06T07:25:15.228701+0000","last_undegraded":"2024-03-06T07:25:15.228701+0000","last_fullsized":"2024-03-06T07:25:15.228701+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:49.607452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.160796+0000","last_change":"2024-03-06T07:25:12.140076+0000","last_active":"2024-03-06T07:25:14.160796+0000","last_peered":"2024-03-06T07:25:14.160796+0000","last_clean":"2024-03-06T07:25:14.160796+0000","last_became_active":"2024-03-06T07:25:12.139766+0000","last_became_peered":"2024-03-06T07:25:12.139766+0000","last_unstale":"2024-03-06T07:25:14.160796+0000","last_undegraded":"2024-03-06T07:25:14.160796+0000","last_fullsized":"2024-03-06T07:25:14.160796+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:49:29.433473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.161117+0000","last_change":"2024-03-06T07:25:13.159035+0000","last_active":"2024-03-06T07:25:14.161117+0000","last_peered":"2024-03-06T07:25:14.161117+0000","last_clean":"2024-03-06T07:25:14.161117+0000","last_became_active":"2024-03-06T07:25:13.158765+0000","last_became_peered":"2024-03-06T07:25:13.158765+0000","last_unstale":"2024-03-06T07:25:14.161117+0000","last_undegraded":"2024-03-06T07:25:14.161117+0000","last_fullsized":"2024-03-06T07:25:14.161117+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:49:42.624006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156808+0000","last_change":"2024-03-06T07:25:12.145745+0000","last_active":"2024-03-06T07:25:14.156808+0000","last_peered":"2024-03-06T07:25:14.156808+0000","last_clean":"2024-03-06T07:25:14.156808+0000","last_became_active":"2024-03-06T07:25:12.145474+0000","last_became_peered":"2024-03-06T07:25:12.145474+0000","last_unstale":"2024-03-06T07:25:14.156808+0000","last_undegraded":"2024-03-06T07:25:14.156808+0000","last_fullsized":"2024-03-06T07:25:14.156808+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:16:25.481036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.159159+0000","last_change":"2024-03-06T07:25:13.164462+0000","last_active":"2024-03-06T07:25:14.159159+0000","last_peered":"2024-03-06T07:25:14.159159+0000","last_clean":"2024-03-06T07:25:14.159159+0000","last_became_active":"2024-03-06T07:25:13.164167+0000","last_became_peered":"2024-03-06T07:25:13.164167+0000","last_unstale":"2024-03-06T07:25:14.159159+0000","last_undegraded":"2024-03-06T07:25:14.159159+0000","last_fullsized":"2024-03-06T07:25:14.159159+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:45:06.074331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"69'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.160466+0000","last_change":"2024-03-06T07:25:12.145578+0000","last_active":"2024-03-06T07:25:14.160466+0000","last_peered":"2024-03-06T07:25:14.160466+0000","last_clean":"2024-03-06T07:25:14.160466+0000","last_became_active":"2024-03-06T07:25:12.145278+0000","last_became_peered":"2024-03-06T07:25:12.145278+0000","last_unstale":"2024-03-06T07:25:14.160466+0000","last_undegraded":"2024-03-06T07:25:14.160466+0000","last_fullsized":"2024-03-06T07:25:14.160466+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:15:34.389509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228736+0000","last_change":"2024-03-06T07:25:13.167557+0000","last_active":"2024-03-06T07:25:15.228736+0000","last_peered":"2024-03-06T07:25:15.228736+0000","last_clean":"2024-03-06T07:25:15.228736+0000","last_became_active":"2024-03-06T07:25:13.167287+0000","last_became_peered":"2024-03-06T07:25:13.167287+0000","last_unstale":"2024-03-06T07:25:15.228736+0000","last_undegraded":"2024-03-06T07:25:15.228736+0000","last_fullsized":"2024-03-06T07:25:15.228736+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:53:53.770605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.155033+0000","last_change":"2024-03-06T07:25:12.146374+0000","last_active":"2024-03-06T07:25:14.155033+0000","last_peered":"2024-03-06T07:25:14.155033+0000","last_clean":"2024-03-06T07:25:14.155033+0000","last_became_active":"2024-03-06T07:25:12.146106+0000","last_became_peered":"2024-03-06T07:25:12.146106+0000","last_unstale":"2024-03-06T07:25:14.155033+0000","last_undegraded":"2024-03-06T07:25:14.155033+0000","last_fullsized":"2024-03-06T07:25:14.155033+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:44:58.938903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228979+0000","last_change":"2024-03-06T07:25:13.165780+0000","last_active":"2024-03-06T07:25:15.228979+0000","last_peered":"2024-03-06T07:25:15.228979+0000","last_clean":"2024-03-06T07:25:15.228979+0000","last_became_active":"2024-03-06T07:25:13.165477+0000","last_became_peered":"2024-03-06T07:25:13.165477+0000","last_unstale":"2024-03-06T07:25:15.228979+0000","last_undegraded":"2024-03-06T07:25:15.228979+0000","last_fullsized":"2024-03-06T07:25:15.228979+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:46:28.717814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.161029+0000","last_change":"2024-03-06T07:25:12.140996+0000","last_active":"2024-03-06T07:25:14.161029+0000","last_peered":"2024-03-06T07:25:14.161029+0000","last_clean":"2024-03-06T07:25:14.161029+0000","last_became_active":"2024-03-06T07:25:12.139506+0000","last_became_peered":"2024-03-06T07:25:12.139506+0000","last_unstale":"2024-03-06T07:25:14.161029+0000","last_undegraded":"2024-03-06T07:25:14.161029+0000","last_fullsized":"2024-03-06T07:25:14.161029+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:08.675501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.155202+0000","last_change":"2024-03-06T07:25:13.157415+0000","last_active":"2024-03-06T07:25:14.155202+0000","last_peered":"2024-03-06T07:25:14.155202+0000","last_clean":"2024-03-06T07:25:14.155202+0000","last_became_active":"2024-03-06T07:25:13.156547+0000","last_became_peered":"2024-03-06T07:25:13.156547+0000","last_unstale":"2024-03-06T07:25:14.155202+0000","last_undegraded":"2024-03-06T07:25:14.155202+0000","last_fullsized":"2024-03-06T07:25:14.155202+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:26.335671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.154992+0000","last_change":"2024-03-06T07:25:12.149100+0000","last_active":"2024-03-06T07:25:14.154992+0000","last_peered":"2024-03-06T07:25:14.154992+0000","last_clean":"2024-03-06T07:25:14.154992+0000","last_became_active":"2024-03-06T07:25:12.148792+0000","last_became_peered":"2024-03-06T07:25:12.148792+0000","last_unstale":"2024-03-06T07:25:14.154992+0000","last_undegraded":"2024-03-06T07:25:14.154992+0000","last_fullsized":"2024-03-06T07:25:14.154992+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:28.946566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156932+0000","last_change":"2024-03-06T07:25:13.162334+0000","last_active":"2024-03-06T07:25:14.156932+0000","last_peered":"2024-03-06T07:25:14.156932+0000","last_clean":"2024-03-06T07:25:14.156932+0000","last_became_active":"2024-03-06T07:25:13.162080+0000","last_became_peered":"2024-03-06T07:25:13.162080+0000","last_unstale":"2024-03-06T07:25:14.156932+0000","last_undegraded":"2024-03-06T07:25:14.156932+0000","last_fullsized":"2024-03-06T07:25:14.156932+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:36.935406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228017+0000","last_change":"2024-03-06T07:25:12.143767+0000","last_active":"2024-03-06T07:25:15.228017+0000","last_peered":"2024-03-06T07:25:15.228017+0000","last_clean":"2024-03-06T07:25:15.228017+0000","last_became_active":"2024-03-06T07:25:12.143484+0000","last_became_peered":"2024-03-06T07:25:12.143484+0000","last_unstale":"2024-03-06T07:25:15.228017+0000","last_undegraded":"2024-03-06T07:25:15.228017+0000","last_fullsized":"2024-03-06T07:25:15.228017+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:57:36.833773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228041+0000","last_change":"2024-03-06T07:25:13.152398+0000","last_active":"2024-03-06T07:25:15.228041+0000","last_peered":"2024-03-06T07:25:15.228041+0000","last_clean":"2024-03-06T07:25:15.228041+0000","last_became_active":"2024-03-06T07:25:13.152124+0000","last_became_peered":"2024-03-06T07:25:13.152124+0000","last_unstale":"2024-03-06T07:25:15.228041+0000","last_undegraded":"2024-03-06T07:25:15.228041+0000","last_fullsized":"2024-03-06T07:25:15.228041+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:05:02.223283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.155374+0000","last_change":"2024-03-06T07:25:12.146477+0000","last_active":"2024-03-06T07:25:14.155374+0000","last_peered":"2024-03-06T07:25:14.155374+0000","last_clean":"2024-03-06T07:25:14.155374+0000","last_became_active":"2024-03-06T07:25:12.146197+0000","last_became_peered":"2024-03-06T07:25:12.146197+0000","last_unstale":"2024-03-06T07:25:14.155374+0000","last_undegraded":"2024-03-06T07:25:14.155374+0000","last_fullsized":"2024-03-06T07:25:14.155374+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:36.270067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156095+0000","last_change":"2024-03-06T07:25:13.155252+0000","last_active":"2024-03-06T07:25:14.156095+0000","last_peered":"2024-03-06T07:25:14.156095+0000","last_clean":"2024-03-06T07:25:14.156095+0000","last_became_active":"2024-03-06T07:25:13.154953+0000","last_became_peered":"2024-03-06T07:25:13.154953+0000","last_unstale":"2024-03-06T07:25:14.156095+0000","last_undegraded":"2024-03-06T07:25:14.156095+0000","last_fullsized":"2024-03-06T07:25:14.156095+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:03:54.769565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156102+0000","last_change":"2024-03-06T07:25:12.144358+0000","last_active":"2024-03-06T07:25:14.156102+0000","last_peered":"2024-03-06T07:25:14.156102+0000","last_clean":"2024-03-06T07:25:14.156102+0000","last_became_active":"2024-03-06T07:25:12.143221+0000","last_became_peered":"2024-03-06T07:25:12.143221+0000","last_unstale":"2024-03-06T07:25:14.156102+0000","last_undegraded":"2024-03-06T07:25:14.156102+0000","last_fullsized":"2024-03-06T07:25:14.156102+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:06.227738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228835+0000","last_change":"2024-03-06T07:25:13.160729+0000","last_active":"2024-03-06T07:25:15.228835+0000","last_peered":"2024-03-06T07:25:15.228835+0000","last_clean":"2024-03-06T07:25:15.228835+0000","last_became_active":"2024-03-06T07:25:13.159186+0000","last_became_peered":"2024-03-06T07:25:13.159186+0000","last_unstale":"2024-03-06T07:25:15.228835+0000","last_undegraded":"2024-03-06T07:25:15.228835+0000","last_fullsized":"2024-03-06T07:25:15.228835+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:12.124012+0000","last_clean_scrub_stamp":"2024-03-06T07:25:12.124012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:05.164335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:14.156749+0000","last_change":"2024-03-06T07:25:12.142392+0000","last_active":"2024-03-06T07:25:14.156749+0000","last_peered":"2024-03-06T07:25:14.156749+0000","last_clean":"2024-03-06T07:25:14.156749+0000","last_became_active":"2024-03-06T07:25:12.142096+0000","last_became_peered":"2024-03-06T07:25:12.142096+0000","last_unstale":"2024-03-06T07:25:14.156749+0000","last_undegraded":"2024-03-06T07:25:14.156749+0000","last_fullsized":"2024-03-06T07:25:14.156749+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:11.119745+0000","last_clean_scrub_stamp":"2024-03-06T07:25:11.119745+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:49:20.994442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":256,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:15.228430+0000","last_change":"2024-03-06T07:21:47.553693+0000","last_active":"2024-03-06T07:25:15.228430+0000","last_peered":"2024-03-06T07:25:15.228430+0000","last_clean":"2024-03-06T07:25:15.228430+0000","last_became_active":"2024-03-06T07:21:47.552664+0000","last_became_peered":"2024-03-06T07:21:47.552664+0000","last_unstale":"2024-03-06T07:25:15.228430+0000","last_undegraded":"2024-03-06T07:25:15.228430+0000","last_fullsized":"2024-03-06T07:25:15.228430+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_clean_scrub_stamp":"2024-03-06T07:21:46.362028+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:02.916426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574895,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32580,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339260,"statfs":{"total":96636764160,"available":96603402240,"internally_reserved":0,"allocated":1429504,"data_stored":937323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574893,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32016,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339824,"statfs":{"total":96636764160,"available":96603979776,"internally_reserved":0,"allocated":847872,"data_stored":352731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574894,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32004,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339836,"statfs":{"total":96636764160,"available":96603992064,"internally_reserved":0,"allocated":843776,"data_stored":352729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574895,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32004,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339836,"statfs":{"total":96636764160,"available":96603992064,"internally_reserved":0,"allocated":843776,"data_stored":352709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574894,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32008,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339832,"statfs":{"total":96636764160,"available":96603987968,"internally_reserved":0,"allocated":847872,"data_stored":352751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574894,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32012,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339828,"statfs":{"total":96636764160,"available":96603983872,"internally_reserved":0,"allocated":851968,"data_stored":354251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574895,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32016,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339824,"statfs":{"total":96636764160,"available":96603979776,"internally_reserved":0,"allocated":847872,"data_stored":348567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574894,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32588,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339252,"statfs":{"total":96636764160,"available":96603394048,"internally_reserved":0,"allocated":1433600,"data_stored":943007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:17.689 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:25:17.689 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:25:17.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.832+0000 7f86013ae640 1 Processor -- start 2024-03-06T07:25:17.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.832+0000 7f86013ae640 1 -- start start 2024-03-06T07:25:17.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.832+0000 7f86013ae640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f85fc15bb40 0x7f85fc0b4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.832+0000 7f86013ae640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f85fc15f7d0 0x7f85fc06d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.832+0000 7f86013ae640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85fc15ff90 con 0x7f85fc06d5c0 2024-03-06T07:25:17.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.832+0000 7f86013ae640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85fc06c910 con 0x7f85fc15bb40 2024-03-06T07:25:17.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.832+0000 7f86013ae640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85fc160270 con 0x7f85fc15f7d0 2024-03-06T07:25:17.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.833+0000 7f85faffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f85fc15bb40 0x7f85fc0b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.833+0000 7f85faffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f85fc15bb40 0x7f85fc0b4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41412/0 (socket says 172.21.15.83:41412) 2024-03-06T07:25:17.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.833+0000 7f85fb7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f85fc06d5c0 0x7f85fc0cfa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:40616/0 (socket says 172.21.15.83:40616) 2024-03-06T07:25:17.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.833+0000 7f85faffd640 1 -- 172.21.15.83:0/3154297263 learned_addr learned my addr 172.21.15.83:0/3154297263 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:17.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.833+0000 7f85fa7fc640 1 --2- 172.21.15.83:0/3154297263 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f85fc15f7d0 0x7f85fc06d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.833+0000 7f85faffd640 1 -- 172.21.15.83:0/3154297263 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f85fc15f7d0 msgr2=0x7f85fc06d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:17.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.833+0000 7f85faffd640 1 --2- 172.21.15.83:0/3154297263 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f85fc15f7d0 0x7f85fc06d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.833+0000 7f85faffd640 1 -- 172.21.15.83:0/3154297263 >> v1:172.21.15.83:6789/0 conn(0x7f85fc06d5c0 legacy=0x7f85fc0cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:17.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.833+0000 7f85faffd640 1 -- 172.21.15.83:0/3154297263 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85fc160550 con 0x7f85fc15bb40 2024-03-06T07:25:17.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.834+0000 7f85faffd640 1 --2- 172.21.15.83:0/3154297263 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f85fc15bb40 0x7f85fc0b4970 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f85ec009870 tx=0x7f85ec02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53d63d845d69f265 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:17.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.834+0000 7f85f9ffb640 1 -- 172.21.15.83:0/3154297263 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85ec034070 con 0x7f85fc15bb40 2024-03-06T07:25:17.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.834+0000 7f85f9ffb640 1 -- 172.21.15.83:0/3154297263 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85ec030840 con 0x7f85fc15bb40 2024-03-06T07:25:17.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.834+0000 7f85f9ffb640 1 -- 172.21.15.83:0/3154297263 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85ec030c10 con 0x7f85fc15bb40 2024-03-06T07:25:17.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.834+0000 7f86013ae640 1 -- 172.21.15.83:0/3154297263 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f85fc15bb40 msgr2=0x7f85fc0b4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:17.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.834+0000 7f86013ae640 1 --2- 172.21.15.83:0/3154297263 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f85fc15bb40 0x7f85fc0b4970 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f85ec009870 tx=0x7f85ec02b980 comp rx=0 tx=0).stop 2024-03-06T07:25:17.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.834+0000 7f86013ae640 1 -- 172.21.15.83:0/3154297263 shutdown_connections 2024-03-06T07:25:17.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.834+0000 7f86013ae640 1 --2- 172.21.15.83:0/3154297263 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f85fc15f7d0 0x7f85fc06d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.834+0000 7f86013ae640 1 --2- 172.21.15.83:0/3154297263 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f85fc15bb40 0x7f85fc0b4970 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.834+0000 7f86013ae640 1 -- 172.21.15.83:0/3154297263 >> 172.21.15.83:0/3154297263 conn(0x7f85fc06b190 msgr2=0x7f85fc06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:17.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.834+0000 7f86013ae640 1 -- 172.21.15.83:0/3154297263 shutdown_connections 2024-03-06T07:25:17.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.834+0000 7f86013ae640 1 -- 172.21.15.83:0/3154297263 wait complete. 2024-03-06T07:25:17.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.836+0000 7f86013ae640 1 Processor -- start 2024-03-06T07:25:17.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.836+0000 7f86013ae640 1 -- start start 2024-03-06T07:25:17.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.836+0000 7f86013ae640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f85fc06d5c0 0x7f85fc1e70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.836+0000 7f86013ae640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f85fc15bb40 0x7f85fc1503b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.836+0000 7f86013ae640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f85fc15f7d0 0x7f85fc1e55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.836+0000 7f86013ae640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85fc15bf20 con 0x7f85fc15bb40 2024-03-06T07:25:17.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.836+0000 7f86013ae640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85fc15fc30 con 0x7f85fc15f7d0 2024-03-06T07:25:17.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.836+0000 7f86013ae640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85fc15c0a0 con 0x7f85fc06d5c0 2024-03-06T07:25:17.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.836+0000 7f85fa7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f85fc15bb40 0x7f85fc1503b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.836+0000 7f85fa7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f85fc15bb40 0x7f85fc1503b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41420/0 (socket says 172.21.15.83:41420) 2024-03-06T07:25:17.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.836+0000 7f85fa7fc640 1 -- 172.21.15.83:0/2950527922 learned_addr learned my addr 172.21.15.83:0/2950527922 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:17.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.837+0000 7f85faffd640 1 --2- 172.21.15.83:0/2950527922 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f85fc06d5c0 0x7f85fc1e70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.837+0000 7f85fb7fe640 1 --2- 172.21.15.83:0/2950527922 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f85fc15f7d0 0x7f85fc1e55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.837+0000 7f85fa7fc640 1 -- 172.21.15.83:0/2950527922 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f85fc06d5c0 msgr2=0x7f85fc1e70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:17.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.837+0000 7f85fa7fc640 1 --2- 172.21.15.83:0/2950527922 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f85fc06d5c0 0x7f85fc1e70c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.837+0000 7f85fa7fc640 1 -- 172.21.15.83:0/2950527922 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f85fc15f7d0 msgr2=0x7f85fc1e55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:17.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.837+0000 7f85fa7fc640 1 --2- 172.21.15.83:0/2950527922 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f85fc15f7d0 0x7f85fc1e55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.837+0000 7f85fa7fc640 1 -- 172.21.15.83:0/2950527922 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85fc1e6020 con 0x7f85fc15bb40 2024-03-06T07:25:17.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.837+0000 7f85fb7fe640 1 --2- 172.21.15.83:0/2950527922 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f85fc15f7d0 0x7f85fc1e55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:17.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.837+0000 7f85fa7fc640 1 --2- 172.21.15.83:0/2950527922 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f85fc15bb40 0x7f85fc1503b0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f85f000e870 tx=0x7f85f000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:17.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.837+0000 7f85e37fe640 1 -- 172.21.15.83:0/2950527922 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f0017070 con 0x7f85fc15bb40 2024-03-06T07:25:17.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.837+0000 7f85e37fe640 1 -- 172.21.15.83:0/2950527922 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85f000c940 con 0x7f85fc15bb40 2024-03-06T07:25:17.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.837+0000 7f86013ae640 1 -- 172.21.15.83:0/2950527922 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85fc1e6300 con 0x7f85fc15bb40 2024-03-06T07:25:17.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.837+0000 7f85e37fe640 1 -- 172.21.15.83:0/2950527922 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f000cd30 con 0x7f85fc15bb40 2024-03-06T07:25:17.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.837+0000 7f86013ae640 1 -- 172.21.15.83:0/2950527922 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85fc1e68c0 con 0x7f85fc15bb40 2024-03-06T07:25:17.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.838+0000 7f86013ae640 1 -- 172.21.15.83:0/2950527922 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85fc0732b0 con 0x7f85fc15bb40 2024-03-06T07:25:17.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.839+0000 7f85e37fe640 1 -- 172.21.15.83:0/2950527922 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f85f001a070 con 0x7f85fc15bb40 2024-03-06T07:25:17.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.839+0000 7f85e37fe640 1 --2- 172.21.15.83:0/2950527922 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f85e806b4d0 0x7f85e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.839+0000 7f85faffd640 1 --2- 172.21.15.83:0/2950527922 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f85e806b4d0 0x7f85e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.839+0000 7f85e37fe640 1 -- 172.21.15.83:0/2950527922 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f85f008a3d0 con 0x7f85fc15bb40 2024-03-06T07:25:17.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.840+0000 7f85faffd640 1 --2- 172.21.15.83:0/2950527922 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f85e806b4d0 0x7f85e806d990 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f85ec036000 tx=0x7f85ec008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:17.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:17.843+0000 7f85e37fe640 1 -- 172.21.15.83:0/2950527922 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f85f0058c00 con 0x7f85fc15bb40 2024-03-06T07:25:18.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.087+0000 7f86013ae640 1 -- 172.21.15.83:0/2950527922 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f85fc068ff0 con 0x7f85fc15bb40 2024-03-06T07:25:18.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.089+0000 7f85e37fe640 1 -- 172.21.15.83:0/2950527922 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) v1 ==== 190+0+147 (secure 0 0 0) 0x7f85f005cca0 con 0x7f85fc15bb40 2024-03-06T07:25:18.091 INFO:teuthology.orchestra.run.smithi083.stdout:[client.0] 2024-03-06T07:25:18.091 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBcGehlRwCIMhAA1DdbdQZPplu7LQOTK3BmSg== 2024-03-06T07:25:18.091 INFO:teuthology.orchestra.run.smithi083.stdout: caps mds = "allow" 2024-03-06T07:25:18.091 INFO:teuthology.orchestra.run.smithi083.stdout: caps mon = "allow r" 2024-03-06T07:25:18.091 INFO:teuthology.orchestra.run.smithi083.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:25:18.092 INFO:teuthology.orchestra.run.smithi083.stderr:updated caps for client.0 2024-03-06T07:25:18.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.091+0000 7f86013ae640 1 -- 172.21.15.83:0/2950527922 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f85e806b4d0 msgr2=0x7f85e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:18.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.091+0000 7f86013ae640 1 --2- 172.21.15.83:0/2950527922 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f85e806b4d0 0x7f85e806d990 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f85ec036000 tx=0x7f85ec008030 comp rx=0 tx=0).stop 2024-03-06T07:25:18.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.091+0000 7f86013ae640 1 -- 172.21.15.83:0/2950527922 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f85fc15bb40 msgr2=0x7f85fc1503b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:18.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.091+0000 7f86013ae640 1 --2- 172.21.15.83:0/2950527922 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f85fc15bb40 0x7f85fc1503b0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f85f000e870 tx=0x7f85f000ed30 comp rx=0 tx=0).stop 2024-03-06T07:25:18.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.091+0000 7f86013ae640 1 -- 172.21.15.83:0/2950527922 shutdown_connections 2024-03-06T07:25:18.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.091+0000 7f86013ae640 1 --2- 172.21.15.83:0/2950527922 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f85e806b4d0 0x7f85e806d990 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.091+0000 7f86013ae640 1 --2- 172.21.15.83:0/2950527922 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f85fc15f7d0 0x7f85fc1e55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.091+0000 7f86013ae640 1 --2- 172.21.15.83:0/2950527922 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f85fc15bb40 0x7f85fc1503b0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.091+0000 7f86013ae640 1 --2- 172.21.15.83:0/2950527922 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f85fc06d5c0 0x7f85fc1e70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.091+0000 7f86013ae640 1 -- 172.21.15.83:0/2950527922 >> 172.21.15.83:0/2950527922 conn(0x7f85fc06b190 msgr2=0x7f85fc156dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:18.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.091+0000 7f86013ae640 1 -- 172.21.15.83:0/2950527922 shutdown_connections 2024-03-06T07:25:18.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.091+0000 7f86013ae640 1 -- 172.21.15.83:0/2950527922 wait complete. 2024-03-06T07:25:18.104 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:18.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.205+0000 7fccd47ac640 1 Processor -- start 2024-03-06T07:25:18.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.205+0000 7fccd47ac640 1 -- start start 2024-03-06T07:25:18.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.206+0000 7fccd47ac640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcccc0d22b0 0x7fcccc0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.206+0000 7fccd47ac640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcccc0b41f0 0x7fcccc0b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.206+0000 7fccd47ac640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcccc0b3430 con 0x7fcccc15dfe0 2024-03-06T07:25:18.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.206+0000 7fccd47ac640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcccc0d2c50 con 0x7fcccc0d22b0 2024-03-06T07:25:18.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.206+0000 7fccd47ac640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc0b31e0 con 0x7fcccc0b41f0 2024-03-06T07:25:18.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.206+0000 7fccd1d47640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcccc0d22b0 0x7fcccc0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.206+0000 7fccd1d47640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcccc0d22b0 0x7fcccc0d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41430/0 (socket says 172.21.15.83:41430) 2024-03-06T07:25:18.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.206+0000 7fccd1d47640 1 -- 172.21.15.83:0/881207455 learned_addr learned my addr 172.21.15.83:0/881207455 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:18.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.206+0000 7fccd2d49640 1 --2- 172.21.15.83:0/881207455 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcccc0b41f0 0x7fcccc0b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.206+0000 7fccd1d47640 1 -- 172.21.15.83:0/881207455 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcccc0b41f0 msgr2=0x7fcccc0b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:18.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.206+0000 7fccd1d47640 1 --2- 172.21.15.83:0/881207455 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcccc0b41f0 0x7fcccc0b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.207+0000 7fccd1d47640 1 -- 172.21.15.83:0/881207455 >> v1:172.21.15.183:6789/0 conn(0x7fcccc15dfe0 legacy=0x7fcccc0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:18.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.207+0000 7fccd1d47640 1 -- 172.21.15.83:0/881207455 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcccc0b3710 con 0x7fcccc0d22b0 2024-03-06T07:25:18.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.208+0000 7fccd1d47640 1 --2- 172.21.15.83:0/881207455 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcccc0d22b0 0x7fcccc0d2710 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fccc8009870 tx=0x7fccc802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bab80c4e279bea2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:18.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.208+0000 7fccd1546640 1 -- 172.21.15.83:0/881207455 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc8034070 con 0x7fcccc0d22b0 2024-03-06T07:25:18.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.208+0000 7fccd1546640 1 -- 172.21.15.83:0/881207455 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc8030840 con 0x7fcccc0d22b0 2024-03-06T07:25:18.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.208+0000 7fccd1546640 1 -- 172.21.15.83:0/881207455 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc8030c10 con 0x7fcccc0d22b0 2024-03-06T07:25:18.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.208+0000 7fccd47ac640 1 -- 172.21.15.83:0/881207455 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcccc0d22b0 msgr2=0x7fcccc0d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:18.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.208+0000 7fccd47ac640 1 --2- 172.21.15.83:0/881207455 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcccc0d22b0 0x7fcccc0d2710 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fccc8009870 tx=0x7fccc802b980 comp rx=0 tx=0).stop 2024-03-06T07:25:18.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.208+0000 7fccd47ac640 1 -- 172.21.15.83:0/881207455 shutdown_connections 2024-03-06T07:25:18.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.208+0000 7fccd47ac640 1 --2- 172.21.15.83:0/881207455 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcccc0b41f0 0x7fcccc0b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.208+0000 7fccd47ac640 1 --2- 172.21.15.83:0/881207455 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcccc0d22b0 0x7fcccc0d2710 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.208+0000 7fccd47ac640 1 -- 172.21.15.83:0/881207455 >> 172.21.15.83:0/881207455 conn(0x7fcccc0d3cd0 msgr2=0x7fcccc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:18.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.209+0000 7fccd47ac640 1 -- 172.21.15.83:0/881207455 shutdown_connections 2024-03-06T07:25:18.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.209+0000 7fccd47ac640 1 -- 172.21.15.83:0/881207455 wait complete. 2024-03-06T07:25:18.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.210+0000 7fccd47ac640 1 Processor -- start 2024-03-06T07:25:18.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.210+0000 7fccd47ac640 1 -- start start 2024-03-06T07:25:18.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.210+0000 7fccd47ac640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcccc0b41f0 0x7fcccc0b6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.210+0000 7fccd47ac640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcccc0d22b0 0x7fcccc1f3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.210+0000 7fccd47ac640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcccc15dfe0 0x7fcccc1f3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.210+0000 7fccd47ac640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc0cfa00 con 0x7fcccc0d22b0 2024-03-06T07:25:18.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.210+0000 7fccd47ac640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc0b3d60 con 0x7fcccc15dfe0 2024-03-06T07:25:18.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.210+0000 7fccd47ac640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc0d1c70 con 0x7fcccc0b41f0 2024-03-06T07:25:18.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.210+0000 7fccd1d47640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcccc0d22b0 0x7fcccc1f3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.210+0000 7fccd1d47640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcccc0d22b0 0x7fcccc1f3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41446/0 (socket says 172.21.15.83:41446) 2024-03-06T07:25:18.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.210+0000 7fccd1d47640 1 -- 172.21.15.83:0/2557126905 learned_addr learned my addr 172.21.15.83:0/2557126905 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:18.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.211+0000 7fccd2548640 1 --2- 172.21.15.83:0/2557126905 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcccc0b41f0 0x7fcccc0b6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.211+0000 7fccd2d49640 1 --2- 172.21.15.83:0/2557126905 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcccc15dfe0 0x7fcccc1f3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.211+0000 7fccd1d47640 1 -- 172.21.15.83:0/2557126905 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcccc0b41f0 msgr2=0x7fcccc0b6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:18.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.211+0000 7fccd1d47640 1 --2- 172.21.15.83:0/2557126905 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcccc0b41f0 0x7fcccc0b6910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.211+0000 7fccd1d47640 1 -- 172.21.15.83:0/2557126905 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcccc15dfe0 msgr2=0x7fcccc1f3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:18.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.211+0000 7fccd1d47640 1 --2- 172.21.15.83:0/2557126905 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcccc15dfe0 0x7fcccc1f3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.211+0000 7fccd1d47640 1 -- 172.21.15.83:0/2557126905 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcccc0b9490 con 0x7fcccc0d22b0 2024-03-06T07:25:18.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.211+0000 7fccd1d47640 1 --2- 172.21.15.83:0/2557126905 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcccc0d22b0 0x7fcccc1f3900 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fccc802b9c0 tx=0x7fccc8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:18.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.211+0000 7fccc2ffd640 1 -- 172.21.15.83:0/2557126905 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc8048070 con 0x7fcccc0d22b0 2024-03-06T07:25:18.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.211+0000 7fccc2ffd640 1 -- 172.21.15.83:0/2557126905 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc8030640 con 0x7fcccc0d22b0 2024-03-06T07:25:18.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.211+0000 7fccd47ac640 1 -- 172.21.15.83:0/2557126905 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcccc0b9770 con 0x7fcccc0d22b0 2024-03-06T07:25:18.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.211+0000 7fccc2ffd640 1 -- 172.21.15.83:0/2557126905 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc8034040 con 0x7fcccc0d22b0 2024-03-06T07:25:18.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.211+0000 7fccd47ac640 1 -- 172.21.15.83:0/2557126905 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcccc0b1130 con 0x7fcccc0d22b0 2024-03-06T07:25:18.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.212+0000 7fccd47ac640 1 -- 172.21.15.83:0/2557126905 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcccc04f330 con 0x7fcccc0d22b0 2024-03-06T07:25:18.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.213+0000 7fccc2ffd640 1 -- 172.21.15.83:0/2557126905 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fccc804b070 con 0x7fcccc0d22b0 2024-03-06T07:25:18.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.213+0000 7fccc2ffd640 1 --2- 172.21.15.83:0/2557126905 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fccb006b4d0 0x7fccb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.213+0000 7fccc2ffd640 1 -- 172.21.15.83:0/2557126905 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fccc80ac070 con 0x7fcccc0d22b0 2024-03-06T07:25:18.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.213+0000 7fccd2548640 1 --2- 172.21.15.83:0/2557126905 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fccb006b4d0 0x7fccb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.213+0000 7fccd2548640 1 --2- 172.21.15.83:0/2557126905 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fccb006b4d0 0x7fccb006d990 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fccbc007fd0 tx=0x7fccbc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:18.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.217+0000 7fccc2ffd640 1 -- 172.21.15.83:0/2557126905 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fccc807af80 con 0x7fcccc0d22b0 2024-03-06T07:25:18.410 INFO:tasks.ceph.mds.a.smithi083.stderr:2024-03-06T07:25:18.408+0000 7fbe89efd640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:25:18.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.434+0000 7fccd47ac640 1 -- 172.21.15.83:0/2557126905 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcccc0b2f40 con 0x7fcccc0d22b0 2024-03-06T07:25:18.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.435+0000 7fccc2ffd640 1 -- 172.21.15.83:0/2557126905 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4982 (secure 0 0 0) 0x7fccc8030810 con 0x7fcccc0d22b0 2024-03-06T07:25:18.436 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:25:18.437 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":32,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6010,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/2142942385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2142942385},{"type":"v1","addr":"172.21.15.183:6834","nonce":2142942385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":6013,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3742438064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":3742438064},{"type":"v1","addr":"172.21.15.183:6835","nonce":3742438064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":6540,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/1549894851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1549894851},{"type":"v1","addr":"172.21.15.83:6836","nonce":1549894851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:12.481670+0000","modified":"2024-03-06T07:25:15.392072+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6543},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6543":{"gid":6543,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/602775648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":602775648},{"type":"v1","addr":"172.21.15.83:6837","nonce":602775648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6543,"qdb_cluster":[6543]},"id":5}]} 2024-03-06T07:25:18.437 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 32 2024-03-06T07:25:18.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.437+0000 7fccd47ac640 1 -- 172.21.15.83:0/2557126905 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fccb006b4d0 msgr2=0x7fccb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:18.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.437+0000 7fccd47ac640 1 --2- 172.21.15.83:0/2557126905 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fccb006b4d0 0x7fccb006d990 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fccbc007fd0 tx=0x7fccbc009040 comp rx=0 tx=0).stop 2024-03-06T07:25:18.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.437+0000 7fccd47ac640 1 -- 172.21.15.83:0/2557126905 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcccc0d22b0 msgr2=0x7fcccc1f3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:18.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.437+0000 7fccd47ac640 1 --2- 172.21.15.83:0/2557126905 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcccc0d22b0 0x7fcccc1f3900 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fccc802b9c0 tx=0x7fccc8004730 comp rx=0 tx=0).stop 2024-03-06T07:25:18.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.437+0000 7fccd47ac640 1 -- 172.21.15.83:0/2557126905 shutdown_connections 2024-03-06T07:25:18.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.437+0000 7fccd47ac640 1 --2- 172.21.15.83:0/2557126905 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fccb006b4d0 0x7fccb006d990 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.437+0000 7fccd47ac640 1 --2- 172.21.15.83:0/2557126905 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcccc15dfe0 0x7fcccc1f3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.437+0000 7fccd47ac640 1 --2- 172.21.15.83:0/2557126905 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcccc0d22b0 0x7fcccc1f3900 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.437+0000 7fccd47ac640 1 --2- 172.21.15.83:0/2557126905 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcccc0b41f0 0x7fcccc0b6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.437+0000 7fccd47ac640 1 -- 172.21.15.83:0/2557126905 >> 172.21.15.83:0/2557126905 conn(0x7fcccc0d3cd0 msgr2=0x7fcccc06cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:18.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.437+0000 7fccd47ac640 1 -- 172.21.15.83:0/2557126905 shutdown_connections 2024-03-06T07:25:18.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.437+0000 7fccd47ac640 1 -- 172.21.15.83:0/2557126905 wait complete. 2024-03-06T07:25:18.851 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 32, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:25:12.481670+0000', 'modified': '2024-03-06T07:25:15.392072+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6543}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6543': {'gid': 6543, 'name': 'a', 'rank': 0, 'incarnation': 29, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.83:6837/602775648', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 602775648}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 602775648}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6543, 'qdb_cluster': [6543]} 2024-03-06T07:25:18.852 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:25:18.852 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:25:18.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.953+0000 7f00a1e66640 1 Processor -- start 2024-03-06T07:25:18.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.953+0000 7f00a1e66640 1 -- start start 2024-03-06T07:25:18.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.953+0000 7f00a1e66640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f009c0b4fe0 0x7f009c0b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.953+0000 7f00a1e66640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f009c15ba50 0x7f009c15bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.953+0000 7f00a1e66640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f009c15d510 con 0x7f009c0b4c00 2024-03-06T07:25:18.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.953+0000 7f00a1e66640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f009c15e0b0 con 0x7f009c15ba50 2024-03-06T07:25:18.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.953+0000 7f00a1e66640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f009c15d780 con 0x7f009c0b4fe0 2024-03-06T07:25:18.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.953+0000 7f009bfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f009c15ba50 0x7f009c15bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.954+0000 7f009bfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f009c15ba50 0x7f009c15bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41450/0 (socket says 172.21.15.83:41450) 2024-03-06T07:25:18.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.954+0000 7f009bfff640 1 -- 172.21.15.83:0/2286586473 learned_addr learned my addr 172.21.15.83:0/2286586473 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:18.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.954+0000 7f0093fff640 1 --2- 172.21.15.83:0/2286586473 >> v2:172.21.15.183:3300/0 conn(0x7f009c0b4fe0 0x7f009c0b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.954+0000 7f009bfff640 1 -- 172.21.15.83:0/2286586473 >> v2:172.21.15.183:3300/0 conn(0x7f009c0b4fe0 msgr2=0x7f009c0b2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:18.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.954+0000 7f009bfff640 1 --2- 172.21.15.83:0/2286586473 >> v2:172.21.15.183:3300/0 conn(0x7f009c0b4fe0 0x7f009c0b2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.954+0000 7f009bfff640 1 -- 172.21.15.83:0/2286586473 >> v1:172.21.15.183:6789/0 conn(0x7f009c0b4c00 legacy=0x7f009c0b28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:18.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.954+0000 7f009bfff640 1 -- 172.21.15.83:0/2286586473 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f009c15da60 con 0x7f009c15ba50 2024-03-06T07:25:18.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.955+0000 7f009bfff640 1 --2- 172.21.15.83:0/2286586473 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f009c15ba50 0x7f009c15bec0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f009c0730b0 tx=0x7f009402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86bf284a44e1d926 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:18.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.955+0000 7f009affd640 1 -- 172.21.15.83:0/2286586473 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009402cb20 con 0x7f009c15ba50 2024-03-06T07:25:18.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.955+0000 7f009affd640 1 -- 172.21.15.83:0/2286586473 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f009402ccc0 con 0x7f009c15ba50 2024-03-06T07:25:18.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.955+0000 7f009affd640 1 -- 172.21.15.83:0/2286586473 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0094034610 con 0x7f009c15ba50 2024-03-06T07:25:18.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.956+0000 7f00a1e66640 1 -- 172.21.15.83:0/2286586473 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f009c15ba50 msgr2=0x7f009c15bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:18.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.956+0000 7f00a1e66640 1 --2- 172.21.15.83:0/2286586473 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f009c15ba50 0x7f009c15bec0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f009c0730b0 tx=0x7f009402bf80 comp rx=0 tx=0).stop 2024-03-06T07:25:18.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.956+0000 7f00a1e66640 1 -- 172.21.15.83:0/2286586473 shutdown_connections 2024-03-06T07:25:18.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.956+0000 7f00a1e66640 1 --2- 172.21.15.83:0/2286586473 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f009c15ba50 0x7f009c15bec0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.956+0000 7f00a1e66640 1 --2- 172.21.15.83:0/2286586473 >> v2:172.21.15.183:3300/0 conn(0x7f009c0b4fe0 0x7f009c0b2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.956+0000 7f00a1e66640 1 -- 172.21.15.83:0/2286586473 >> 172.21.15.83:0/2286586473 conn(0x7f009c09a9f0 msgr2=0x7f009c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:18.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.956+0000 7f00a1e66640 1 -- 172.21.15.83:0/2286586473 shutdown_connections 2024-03-06T07:25:18.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.956+0000 7f00a1e66640 1 -- 172.21.15.83:0/2286586473 wait complete. 2024-03-06T07:25:18.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.957+0000 7f00a1e66640 1 Processor -- start 2024-03-06T07:25:18.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.957+0000 7f00a1e66640 1 -- start start 2024-03-06T07:25:18.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.958+0000 7f00a1e66640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f009c0b4c00 0x7f009c0b00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.958+0000 7f00a1e66640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f009c0b4fe0 0x7f009c13f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.958+0000 7f00a1e66640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f009c15ba50 0x7f009c1f5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.958+0000 7f00a1e66640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f009c15f6d0 con 0x7f009c0b4c00 2024-03-06T07:25:18.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.958+0000 7f00a1e66640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f009c15f550 con 0x7f009c0b4fe0 2024-03-06T07:25:18.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.958+0000 7f00a1e66640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f009c15c400 con 0x7f009c15ba50 2024-03-06T07:25:18.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.958+0000 7f009b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f009c0b4c00 0x7f009c0b00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.958+0000 7f009b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f009c0b4c00 0x7f009c0b00b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41454/0 (socket says 172.21.15.83:41454) 2024-03-06T07:25:18.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.958+0000 7f009b7fe640 1 -- 172.21.15.83:0/2987479779 learned_addr learned my addr 172.21.15.83:0/2987479779 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:18.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.958+0000 7f009bfff640 1 --2- 172.21.15.83:0/2987479779 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f009c15ba50 0x7f009c1f5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.958+0000 7f009b7fe640 1 -- 172.21.15.83:0/2987479779 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f009c15ba50 msgr2=0x7f009c1f5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:18.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.958+0000 7f009b7fe640 1 --2- 172.21.15.83:0/2987479779 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f009c15ba50 0x7f009c1f5bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.958+0000 7f009b7fe640 1 -- 172.21.15.83:0/2987479779 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f009c0b4fe0 msgr2=0x7f009c13f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:18.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.958+0000 7f0093fff640 1 --2- 172.21.15.83:0/2987479779 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f009c0b4fe0 0x7f009c13f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.958+0000 7f009b7fe640 1 --2- 172.21.15.83:0/2987479779 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f009c0b4fe0 0x7f009c13f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.958+0000 7f009b7fe640 1 -- 172.21.15.83:0/2987479779 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f009c152d30 con 0x7f009c0b4c00 2024-03-06T07:25:18.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.958+0000 7f009b7fe640 1 --2- 172.21.15.83:0/2987479779 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f009c0b4c00 0x7f009c0b00b0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f008c00c430 tx=0x7f008c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:18.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.958+0000 7f0098ff9640 1 -- 172.21.15.83:0/2987479779 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008c011790 con 0x7f009c0b4c00 2024-03-06T07:25:18.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.958+0000 7f0098ff9640 1 -- 172.21.15.83:0/2987479779 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f008c017070 con 0x7f009c0b4c00 2024-03-06T07:25:18.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.958+0000 7f0098ff9640 1 -- 172.21.15.83:0/2987479779 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008c011b70 con 0x7f009c0b4c00 2024-03-06T07:25:18.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.958+0000 7f00a1e66640 1 -- 172.21.15.83:0/2987479779 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f009c153010 con 0x7f009c0b4c00 2024-03-06T07:25:18.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.958+0000 7f00a1e66640 1 -- 172.21.15.83:0/2987479779 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f009c1532f0 con 0x7f009c0b4c00 2024-03-06T07:25:18.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.960+0000 7f0098ff9640 1 -- 172.21.15.83:0/2987479779 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f008c011d10 con 0x7f009c0b4c00 2024-03-06T07:25:18.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.960+0000 7f0098ff9640 1 --2- 172.21.15.83:0/2987479779 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f007c06b4d0 0x7f007c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.960+0000 7f0098ff9640 1 -- 172.21.15.83:0/2987479779 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f008c08a030 con 0x7f009c0b4c00 2024-03-06T07:25:18.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.960+0000 7f0093fff640 1 --2- 172.21.15.83:0/2987479779 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f007c06b4d0 0x7f007c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.961+0000 7f0093fff640 1 --2- 172.21.15.83:0/2987479779 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f007c06b4d0 0x7f007c06d990 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f009c0b2380 tx=0x7f0084008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:18.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.961+0000 7f00a4a48740 1 Processor -- start 2024-03-06T07:25:18.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.961+0000 7f00a4a48740 1 -- start start 2024-03-06T07:25:18.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.962+0000 7f00a4a48740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56452ced8cc0 0x56452ced90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.962+0000 7f00a4a48740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56452cef9090 0x56452cef95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.962+0000 7f00a4a48740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56452cef87b0 0x56452ceab680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.962+0000 7f00a4a48740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x56452ce156b0 con 0x56452cef87b0 2024-03-06T07:25:18.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.962+0000 7f00a4a48740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x56452cdbea60 con 0x56452ced8cc0 2024-03-06T07:25:18.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.962+0000 7f00a4a48740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x56452ce11750 con 0x56452cef9090 2024-03-06T07:25:18.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.962+0000 7f009bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56452cef87b0 0x56452ceab680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.962+0000 7f009bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56452cef87b0 0x56452ceab680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41456/0 (socket says 172.21.15.83:41456) 2024-03-06T07:25:18.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.962+0000 7f009bfff640 1 -- 172.21.15.83:0/2141882054 learned_addr learned my addr 172.21.15.83:0/2141882054 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:18.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.962+0000 7f0093fff640 1 --2- 172.21.15.83:0/2141882054 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56452cef9090 0x56452cef95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.962+0000 7f009b7fe640 1 --2- 172.21.15.83:0/2141882054 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56452ced8cc0 0x56452ced90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.962+0000 7f009bfff640 1 -- 172.21.15.83:0/2141882054 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56452cef9090 msgr2=0x56452cef95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:18.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.962+0000 7f009bfff640 1 --2- 172.21.15.83:0/2141882054 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56452cef9090 0x56452cef95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.962+0000 7f009bfff640 1 -- 172.21.15.83:0/2141882054 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56452ced8cc0 msgr2=0x56452ced90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:18.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.962+0000 7f009bfff640 1 --2- 172.21.15.83:0/2141882054 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56452ced8cc0 0x56452ced90a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.962+0000 7f009bfff640 1 -- 172.21.15.83:0/2141882054 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56452cddcf30 con 0x56452cef87b0 2024-03-06T07:25:18.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.963+0000 7f009b7fe640 1 --2- 172.21.15.83:0/2141882054 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56452ced8cc0 0x56452ced90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:18.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.963+0000 7f009bfff640 1 --2- 172.21.15.83:0/2141882054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56452cef87b0 0x56452ceab680 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f0094009f10 tx=0x7f009402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6fb839c59108dbf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:18.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.963+0000 7f00917fa640 1 -- 172.21.15.83:0/2141882054 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0094045070 con 0x56452cef87b0 2024-03-06T07:25:18.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.963+0000 7f00917fa640 1 -- 172.21.15.83:0/2141882054 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f009402cb20 con 0x56452cef87b0 2024-03-06T07:25:18.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.963+0000 7f00917fa640 1 -- 172.21.15.83:0/2141882054 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009402ccd0 con 0x56452cef87b0 2024-03-06T07:25:18.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.963+0000 7f00a4a48740 1 -- 172.21.15.83:0/2141882054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56452cef87b0 msgr2=0x56452ceab680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:18.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.963+0000 7f00a4a48740 1 --2- 172.21.15.83:0/2141882054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56452cef87b0 0x56452ceab680 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f0094009f10 tx=0x7f009402e000 comp rx=0 tx=0).stop 2024-03-06T07:25:18.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.964+0000 7f00a4a48740 1 -- 172.21.15.83:0/2141882054 shutdown_connections 2024-03-06T07:25:18.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.964+0000 7f00a4a48740 1 --2- 172.21.15.83:0/2141882054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56452cef87b0 0x56452ceab680 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.964+0000 7f00a4a48740 1 --2- 172.21.15.83:0/2141882054 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56452cef9090 0x56452cef95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.964+0000 7f00a4a48740 1 --2- 172.21.15.83:0/2141882054 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56452ced8cc0 0x56452ced90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.964+0000 7f00a4a48740 1 -- 172.21.15.83:0/2141882054 >> 172.21.15.83:0/2141882054 conn(0x56452cead8f0 msgr2=0x56452ceaf1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:18.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.964+0000 7f00a4a48740 1 -- 172.21.15.83:0/2141882054 shutdown_connections 2024-03-06T07:25:18.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.964+0000 7f00a4a48740 1 -- 172.21.15.83:0/2141882054 wait complete. 2024-03-06T07:25:18.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.964+0000 7f00a4a48740 1 -- start start 2024-03-06T07:25:18.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.965+0000 7f00a4a48740 1 Processor -- start 2024-03-06T07:25:18.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.966+0000 7f00a4a48740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56452cef87b0 0x56452cecacb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.966+0000 7f00a4a48740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56452cef9090 0x56452cf0c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.966+0000 7f00a4a48740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56452cecb240 0x56452ced0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.966+0000 7f00a4a48740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x56452cedc010 con 0x56452cecb240 2024-03-06T07:25:18.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.966+0000 7f00a4a48740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x56452cdd1ea0 con 0x56452cef9090 2024-03-06T07:25:18.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.966+0000 7f00a4a48740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x56452cef7a30 con 0x56452cef87b0 2024-03-06T07:25:18.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.966+0000 7f00a4a48740 10 client.0 _ll_register_callbacks cb 0x56452cdd1870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:25:18.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.966+0000 7f009bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56452cecb240 0x56452ced0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.966+0000 7f009bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56452cecb240 0x56452ced0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41468/0 (socket says 172.21.15.83:41468) 2024-03-06T07:25:18.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.966+0000 7f009bfff640 1 -- 172.21.15.83:0/2587226059 learned_addr learned my addr 172.21.15.83:0/2587226059 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:18.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.966+0000 7f009b7fe640 1 --2- 172.21.15.83:0/2587226059 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56452cef87b0 0x56452cecacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.966+0000 7f0093fff640 1 --2- 172.21.15.83:0/2587226059 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56452cef9090 0x56452cf0c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.966+0000 7f009bfff640 1 -- 172.21.15.83:0/2587226059 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56452cef87b0 msgr2=0x56452cecacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:18.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.966+0000 7f009bfff640 1 --2- 172.21.15.83:0/2587226059 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56452cef87b0 0x56452cecacb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.966+0000 7f009bfff640 1 -- 172.21.15.83:0/2587226059 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56452cef9090 msgr2=0x56452cf0c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:18.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.966+0000 7f009bfff640 1 --2- 172.21.15.83:0/2587226059 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56452cef9090 0x56452cf0c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.966+0000 7f009bfff640 1 -- 172.21.15.83:0/2587226059 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56452cdd2270 con 0x56452cecb240 2024-03-06T07:25:18.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.966+0000 7f009bfff640 1 --2- 172.21.15.83:0/2587226059 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56452cecb240 0x56452ced0dd0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f00940072a0 tx=0x7f0094004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1171e5355336f583 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:18.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.966+0000 7f007a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-06T07:25:18.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.967+0000 7f007a7fc640 1 -- 172.21.15.83:0/2587226059 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0094045040 con 0x56452cecb240 2024-03-06T07:25:18.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.967+0000 7f007a7fc640 1 -- 172.21.15.83:0/2587226059 <== mon.0 v2:172.21.15.83:3300/0 2 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0094004240 con 0x56452cecb240 2024-03-06T07:25:18.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.967+0000 7f00a4a48740 1 -- 172.21.15.83:0/2587226059 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f009402cb20 con 0x56452cecb240 2024-03-06T07:25:18.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.967+0000 7f007a7fc640 1 -- 172.21.15.83:0/2587226059 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0094055050 con 0x56452cecb240 2024-03-06T07:25:18.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.967+0000 7f00a4a48740 10 client.6690 fetch_fsmap learned FSMap version 32 2024-03-06T07:25:18.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.967+0000 7f00a4a48740 1 -- 172.21.15.83:0/2587226059 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=2+}) v3 -- 0x56452cb2a2d0 con 0x56452cecb240 2024-03-06T07:25:18.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.967+0000 7f007a7fc640 1 -- 172.21.15.83:0/2587226059 <== mon.0 v2:172.21.15.83:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f009402cb20 con 0x56452cecb240 2024-03-06T07:25:18.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.967+0000 7f007a7fc640 1 -- 172.21.15.83:0/2587226059 <== mon.0 v2:172.21.15.83:3300/0 5 ==== fsmap(e 32) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f0094058020 con 0x56452cecb240 2024-03-06T07:25:18.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.967+0000 7f00a4a48740 10 client.6690 fetch_fsmap finished waiting for FSMap version 32 2024-03-06T07:25:18.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.967+0000 7f00a4a48740 10 client.6690 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:25:18.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.967+0000 7f00a4a48740 20 client.6690 populate_metadata read hostname 'smithi083' 2024-03-06T07:25:18.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.967+0000 7f00a4a48740 1 --2- 172.21.15.83:0/2587226059 >> [v2:172.21.15.83:6835/602775648,v1:172.21.15.83:6837/602775648] conn(0x56452ced1e50 0x56452ced2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.967+0000 7f00a4a48740 4 client.6690 mds_command: new command op to 6543 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:25:18.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.967+0000 7f00a4a48740 1 -- 172.21.15.83:0/2587226059 --> [v2:172.21.15.83:6835/602775648,v1:172.21.15.83:6837/602775648] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56452ce22bf0 con 0x56452ced1e50 2024-03-06T07:25:18.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.967+0000 7f009b7fe640 1 --2- 172.21.15.83:0/2587226059 >> [v2:172.21.15.83:6835/602775648,v1:172.21.15.83:6837/602775648] conn(0x56452ced1e50 0x56452ced2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.967+0000 7f007a7fc640 10 client.6690 ms_handle_connect on v2:172.21.15.83:6835/602775648 2024-03-06T07:25:18.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.967+0000 7f009b7fe640 1 --2- 172.21.15.83:0/2587226059 >> [v2:172.21.15.83:6835/602775648,v1:172.21.15.83:6837/602775648] conn(0x56452ced1e50 0x56452ced2230 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2debcbb26f65eb80 server_cookie=3b731c02169c2081 in_seq=0 out_seq=0 2024-03-06T07:25:18.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.968+0000 7f007a7fc640 1 -- 172.21.15.83:0/2587226059 <== mds.0 v2:172.21.15.83:6835/602775648 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56452ce22bf0 con 0x56452ced1e50 2024-03-06T07:25:18.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.968+0000 7f007a7fc640 10 client.6690 handle_command_reply: tid=0 2024-03-06T07:25:18.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.968+0000 7f00a4a48740 1 client.6690 shutdown 2024-03-06T07:25:18.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.968+0000 7f00a4a48740 1 -- 172.21.15.83:0/2587226059 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56452cecb240 msgr2=0x56452ced0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:18.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.968+0000 7f00a4a48740 1 --2- 172.21.15.83:0/2587226059 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56452cecb240 0x56452ced0dd0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f00940072a0 tx=0x7f0094004ab0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.968+0000 7f00a4a48740 1 -- 172.21.15.83:0/2587226059 shutdown_connections 2024-03-06T07:25:18.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.968+0000 7f00a4a48740 1 --2- 172.21.15.83:0/2587226059 >> [v2:172.21.15.83:6835/602775648,v1:172.21.15.83:6837/602775648] conn(0x56452ced1e50 0x56452ced2230 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.968+0000 7f007a7fc640 0 client.6690 ms_handle_reset on v2:172.21.15.83:6835/602775648 2024-03-06T07:25:18.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.968+0000 7f00a4a48740 1 --2- 172.21.15.83:0/2587226059 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56452cecb240 0x56452ced0dd0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.968+0000 7f00a4a48740 1 --2- 172.21.15.83:0/2587226059 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56452cef9090 0x56452cf0c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.968+0000 7f00a4a48740 1 --2- 172.21.15.83:0/2587226059 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56452cef87b0 0x56452cecacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.968+0000 7f00a4a48740 1 -- 172.21.15.83:0/2587226059 >> 172.21.15.83:0/2587226059 conn(0x56452ced8cc0 msgr2=0x56452ceaf8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:18.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.968+0000 7f00a4a48740 1 -- 172.21.15.83:0/2587226059 shutdown_connections 2024-03-06T07:25:18.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.968+0000 7f00a4a48740 1 -- 172.21.15.83:0/2587226059 wait complete. 2024-03-06T07:25:18.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.968+0000 7f00a4a48740 20 client.6690 trim_cache size 0 max 16384 2024-03-06T07:25:18.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.985+0000 7f00a4a48740 1 Processor -- start 2024-03-06T07:25:18.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.985+0000 7f00a4a48740 1 -- start start 2024-03-06T07:25:18.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.986+0000 7f00a4a48740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56452cecb240 0x56452ceab320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.986+0000 7f00a4a48740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56452ced1e50 0x56452ceb15f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.986+0000 7f00a4a48740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56452cef87b0 0x56452ceaa510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.986+0000 7f00a4a48740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f009c142450 con 0x56452ced1e50 2024-03-06T07:25:18.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.986+0000 7f00a4a48740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f009c141850 con 0x56452cecb240 2024-03-06T07:25:18.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.986+0000 7f00a4a48740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f009c141e50 con 0x56452cef87b0 2024-03-06T07:25:18.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.986+0000 7f0093fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56452ced1e50 0x56452ceb15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.986+0000 7f0093fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56452ced1e50 0x56452ceb15f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41482/0 (socket says 172.21.15.83:41482) 2024-03-06T07:25:18.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.986+0000 7f0093fff640 1 -- 172.21.15.83:0/743089094 learned_addr learned my addr 172.21.15.83:0/743089094 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:18.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.986+0000 7f0093fff640 1 -- 172.21.15.83:0/743089094 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56452cef87b0 msgr2=0x56452ceaa510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:18.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.986+0000 7f009bfff640 1 --2- 172.21.15.83:0/743089094 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56452cef87b0 0x56452ceaa510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.986+0000 7f009b7fe640 1 --2- 172.21.15.83:0/743089094 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56452cecb240 0x56452ceab320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.986+0000 7f0093fff640 1 --2- 172.21.15.83:0/743089094 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56452cef87b0 0x56452ceaa510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.986+0000 7f0093fff640 1 -- 172.21.15.83:0/743089094 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56452cecb240 msgr2=0x56452ceab320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:18.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.986+0000 7f0093fff640 1 --2- 172.21.15.83:0/743089094 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56452cecb240 0x56452ceab320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.986+0000 7f0093fff640 1 -- 172.21.15.83:0/743089094 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56452cbf46b0 con 0x56452ced1e50 2024-03-06T07:25:18.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.986+0000 7f0093fff640 1 --2- 172.21.15.83:0/743089094 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56452ced1e50 0x56452ceb15f0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f0084009f70 tx=0x7f0084014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bd92f803018e80d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:18.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.987+0000 7f0090ff9640 1 -- 172.21.15.83:0/743089094 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008400f040 con 0x56452ced1e50 2024-03-06T07:25:18.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.987+0000 7f0090ff9640 1 -- 172.21.15.83:0/743089094 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0084027070 con 0x56452ced1e50 2024-03-06T07:25:18.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.987+0000 7f0090ff9640 1 -- 172.21.15.83:0/743089094 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0084015680 con 0x56452ced1e50 2024-03-06T07:25:18.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.987+0000 7f00a4a48740 1 -- 172.21.15.83:0/743089094 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56452ced1e50 msgr2=0x56452ceb15f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:18.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.987+0000 7f00a4a48740 1 --2- 172.21.15.83:0/743089094 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56452ced1e50 0x56452ceb15f0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f0084009f70 tx=0x7f0084014000 comp rx=0 tx=0).stop 2024-03-06T07:25:18.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.987+0000 7f00a4a48740 1 -- 172.21.15.83:0/743089094 shutdown_connections 2024-03-06T07:25:18.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.987+0000 7f00a4a48740 1 --2- 172.21.15.83:0/743089094 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56452cef87b0 0x56452ceaa510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.987+0000 7f00a4a48740 1 --2- 172.21.15.83:0/743089094 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56452ced1e50 0x56452ceb15f0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.987+0000 7f00a4a48740 1 --2- 172.21.15.83:0/743089094 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56452cecb240 0x56452ceab320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.987+0000 7f00a4a48740 1 -- 172.21.15.83:0/743089094 >> 172.21.15.83:0/743089094 conn(0x56452ced8cc0 msgr2=0x56452cec8930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:18.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.987+0000 7f00a4a48740 1 -- 172.21.15.83:0/743089094 shutdown_connections 2024-03-06T07:25:18.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.987+0000 7f00a4a48740 1 -- 172.21.15.83:0/743089094 wait complete. 2024-03-06T07:25:18.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.988+0000 7f00a4a48740 1 -- start start 2024-03-06T07:25:18.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.988+0000 7f00a4a48740 1 Processor -- start 2024-03-06T07:25:18.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.989+0000 7f00a4a48740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56452cecb240 0x56452cf6a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.989+0000 7f00a4a48740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56452ced1e50 0x56452ceaf180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.989+0000 7f00a4a48740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56452cef87b0 0x56452cece370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.989+0000 7f00a4a48740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x56452cf0cc00 con 0x56452cecb240 2024-03-06T07:25:18.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.989+0000 7f00a4a48740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x56452ced05d0 con 0x56452cef87b0 2024-03-06T07:25:18.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.989+0000 7f00a4a48740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x56452ced1550 con 0x56452ced1e50 2024-03-06T07:25:18.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.989+0000 7f00a4a48740 10 client.0 _ll_register_callbacks cb 0x7f0094003ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:25:18.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.989+0000 7f009b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56452cecb240 0x56452cf6a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.989+0000 7f009b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56452cecb240 0x56452cf6a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41484/0 (socket says 172.21.15.83:41484) 2024-03-06T07:25:18.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.989+0000 7f009b7fe640 1 -- 172.21.15.83:0/3298277219 learned_addr learned my addr 172.21.15.83:0/3298277219 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:18.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.989+0000 7f0093fff640 1 --2- 172.21.15.83:0/3298277219 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56452ced1e50 0x56452ceaf180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.989+0000 7f009bfff640 1 --2- 172.21.15.83:0/3298277219 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56452cef87b0 0x56452cece370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.989+0000 7f009b7fe640 1 -- 172.21.15.83:0/3298277219 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56452ced1e50 msgr2=0x56452ceaf180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:18.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.989+0000 7f009b7fe640 1 --2- 172.21.15.83:0/3298277219 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56452ced1e50 0x56452ceaf180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.989+0000 7f009b7fe640 1 -- 172.21.15.83:0/3298277219 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56452cef87b0 msgr2=0x56452cece370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:18.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.989+0000 7f009b7fe640 1 --2- 172.21.15.83:0/3298277219 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56452cef87b0 0x56452cece370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.989+0000 7f009b7fe640 1 -- 172.21.15.83:0/3298277219 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56452ceaf6c0 con 0x56452cecb240 2024-03-06T07:25:18.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.989+0000 7f009b7fe640 1 --2- 172.21.15.83:0/3298277219 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56452cecb240 0x56452cf6a6e0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f008c058540 tx=0x7f008c01e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9df109517f73d39c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:18.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.989+0000 7f007a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-06T07:25:18.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.989+0000 7f007a7fc640 1 -- 172.21.15.83:0/3298277219 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008c054eb0 con 0x56452cecb240 2024-03-06T07:25:18.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.989+0000 7f007a7fc640 1 -- 172.21.15.83:0/3298277219 <== mon.0 v2:172.21.15.83:3300/0 2 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f008c070190 con 0x56452cecb240 2024-03-06T07:25:18.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.989+0000 7f00a4a48740 1 -- 172.21.15.83:0/3298277219 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0084027070 con 0x56452cecb240 2024-03-06T07:25:18.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.989+0000 7f007a7fc640 1 -- 172.21.15.83:0/3298277219 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f008c055050 con 0x56452cecb240 2024-03-06T07:25:18.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.989+0000 7f00a4a48740 10 client.6696 fetch_fsmap learned FSMap version 32 2024-03-06T07:25:18.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.989+0000 7f00a4a48740 1 -- 172.21.15.83:0/3298277219 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=2+}) v3 -- 0x56452cf1e750 con 0x56452cecb240 2024-03-06T07:25:18.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.990+0000 7f007a7fc640 1 -- 172.21.15.83:0/3298277219 <== mon.0 v2:172.21.15.83:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0084027070 con 0x56452cecb240 2024-03-06T07:25:18.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.990+0000 7f007a7fc640 1 -- 172.21.15.83:0/3298277219 <== mon.0 v2:172.21.15.83:3300/0 5 ==== fsmap(e 32) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f008c060ac0 con 0x56452cecb240 2024-03-06T07:25:18.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.990+0000 7f00a4a48740 10 client.6696 fetch_fsmap finished waiting for FSMap version 32 2024-03-06T07:25:18.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.990+0000 7f00a4a48740 10 client.6696 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:25:18.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.990+0000 7f00a4a48740 20 client.6696 populate_metadata read hostname 'smithi083' 2024-03-06T07:25:18.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.990+0000 7f00a4a48740 1 --2- 172.21.15.83:0/3298277219 >> [v2:172.21.15.83:6835/602775648,v1:172.21.15.83:6837/602775648] conn(0x56452cef9090 0x56452cec96d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.990+0000 7f00a4a48740 4 client.6696 mds_command: new command op to 6543 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:25:18.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.990+0000 7f00a4a48740 1 -- 172.21.15.83:0/3298277219 --> [v2:172.21.15.83:6835/602775648,v1:172.21.15.83:6837/602775648] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56452ce1cf40 con 0x56452cef9090 2024-03-06T07:25:18.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.990+0000 7f0093fff640 1 --2- 172.21.15.83:0/3298277219 >> [v2:172.21.15.83:6835/602775648,v1:172.21.15.83:6837/602775648] conn(0x56452cef9090 0x56452cec96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.990+0000 7f007a7fc640 10 client.6696 ms_handle_connect on v2:172.21.15.83:6835/602775648 2024-03-06T07:25:18.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.990+0000 7f0093fff640 1 --2- 172.21.15.83:0/3298277219 >> [v2:172.21.15.83:6835/602775648,v1:172.21.15.83:6837/602775648] conn(0x56452cef9090 0x56452cec96d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=709939e6f4f1030 server_cookie=a3d2786f8e7a93f6 in_seq=0 out_seq=0 2024-03-06T07:25:18.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.990+0000 7f007a7fc640 1 -- 172.21.15.83:0/3298277219 <== mds.0 v2:172.21.15.83:6835/602775648 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x56452ce1cf40 con 0x56452cef9090 2024-03-06T07:25:18.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.990+0000 7f007a7fc640 10 client.6696 handle_command_reply: tid=0 2024-03-06T07:25:18.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.990+0000 7f00a4a48740 1 client.6696 shutdown 2024-03-06T07:25:18.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.990+0000 7f00a4a48740 1 -- 172.21.15.83:0/3298277219 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56452cecb240 msgr2=0x56452cf6a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:18.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.990+0000 7f00a4a48740 1 --2- 172.21.15.83:0/3298277219 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56452cecb240 0x56452cf6a6e0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f008c058540 tx=0x7f008c01e040 comp rx=0 tx=0).stop 2024-03-06T07:25:18.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.991+0000 7f00a4a48740 1 -- 172.21.15.83:0/3298277219 shutdown_connections 2024-03-06T07:25:18.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.991+0000 7f00a4a48740 1 --2- 172.21.15.83:0/3298277219 >> [v2:172.21.15.83:6835/602775648,v1:172.21.15.83:6837/602775648] conn(0x56452cef9090 0x56452cec96d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.991+0000 7f00a4a48740 1 --2- 172.21.15.83:0/3298277219 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56452cef87b0 0x56452cece370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.991+0000 7f00a4a48740 1 --2- 172.21.15.83:0/3298277219 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56452ced1e50 0x56452ceaf180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.991+0000 7f00a4a48740 1 --2- 172.21.15.83:0/3298277219 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56452cecb240 0x56452cf6a6e0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.991+0000 7f007a7fc640 0 client.6696 ms_handle_reset on v2:172.21.15.83:6835/602775648 2024-03-06T07:25:19.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.991+0000 7f00a4a48740 1 -- 172.21.15.83:0/3298277219 >> 172.21.15.83:0/3298277219 conn(0x56452ced8cc0 msgr2=0x56452cea6c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:19.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.991+0000 7f00a4a48740 1 -- 172.21.15.83:0/3298277219 shutdown_connections 2024-03-06T07:25:19.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.991+0000 7f00a4a48740 1 -- 172.21.15.83:0/3298277219 wait complete. 2024-03-06T07:25:19.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.991+0000 7f00a4a48740 20 client.6696 trim_cache size 0 max 16384 2024-03-06T07:25:19.000 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:25:19.000 INFO:teuthology.orchestra.run.smithi083.stdout: "cluster_fsid": "039877dd-d2ce-45d2-8e77-ac5eb7e1d55a", 2024-03-06T07:25:19.000 INFO:teuthology.orchestra.run.smithi083.stdout: "whoami": 0, 2024-03-06T07:25:19.000 INFO:teuthology.orchestra.run.smithi083.stdout: "id": 6543, 2024-03-06T07:25:19.001 INFO:teuthology.orchestra.run.smithi083.stdout: "want_state": "up:active", 2024-03-06T07:25:19.001 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "up:active", 2024-03-06T07:25:19.001 INFO:teuthology.orchestra.run.smithi083.stdout: "fs_name": "cephfs", 2024-03-06T07:25:19.001 INFO:teuthology.orchestra.run.smithi083.stdout: "rank_uptime": 5.8395537089999996, 2024-03-06T07:25:19.001 INFO:teuthology.orchestra.run.smithi083.stdout: "mdsmap_epoch": 32, 2024-03-06T07:25:19.001 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_epoch": 69, 2024-03-06T07:25:19.001 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_epoch_barrier": 69, 2024-03-06T07:25:19.001 INFO:teuthology.orchestra.run.smithi083.stdout: "uptime": 10.91783405 2024-03-06T07:25:19.001 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:25:19.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.991+0000 7f00a1e66640 1 -- 172.21.15.83:0/2987479779 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f007c06b4d0 msgr2=0x7f007c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:19.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.991+0000 7f00a1e66640 1 --2- 172.21.15.83:0/2987479779 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f007c06b4d0 0x7f007c06d990 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f009c0b2380 tx=0x7f0084008040 comp rx=0 tx=0).stop 2024-03-06T07:25:19.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.991+0000 7f00a1e66640 1 -- 172.21.15.83:0/2987479779 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f009c0b4c00 msgr2=0x7f009c0b00b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:19.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.991+0000 7f00a1e66640 1 --2- 172.21.15.83:0/2987479779 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f009c0b4c00 0x7f009c0b00b0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f008c00c430 tx=0x7f008c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.991+0000 7f00a1e66640 1 -- 172.21.15.83:0/2987479779 shutdown_connections 2024-03-06T07:25:19.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.991+0000 7f00a1e66640 1 --2- 172.21.15.83:0/2987479779 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f007c06b4d0 0x7f007c06d990 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.991+0000 7f00a1e66640 1 --2- 172.21.15.83:0/2987479779 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f009c15ba50 0x7f009c1f5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.991+0000 7f00a1e66640 1 --2- 172.21.15.83:0/2987479779 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f009c0b4fe0 0x7f009c13f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.991+0000 7f00a1e66640 1 --2- 172.21.15.83:0/2987479779 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f009c0b4c00 0x7f009c0b00b0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.991+0000 7f00a1e66640 1 -- 172.21.15.83:0/2987479779 >> 172.21.15.83:0/2987479779 conn(0x7f009c09a9f0 msgr2=0x7f009c158c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:19.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.991+0000 7f00a1e66640 1 -- 172.21.15.83:0/2987479779 shutdown_connections 2024-03-06T07:25:19.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:18.991+0000 7f00a1e66640 1 -- 172.21.15.83:0/2987479779 wait complete. 2024-03-06T07:25:19.004 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:25:19.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.125+0000 7f486de81640 1 Processor -- start 2024-03-06T07:25:19.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.125+0000 7f486de81640 1 -- start start 2024-03-06T07:25:19.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.125+0000 7f486de81640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f48681424f0 0x7f48680cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:19.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.125+0000 7f486de81640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f48680cd7a0 0x7f486815cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:19.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.125+0000 7f486de81640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f486806d330 0x7f486806d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:19.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.125+0000 7f486de81640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48680cd220 con 0x7f48680cd7a0 2024-03-06T07:25:19.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.125+0000 7f486de81640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48680cc980 con 0x7f48681424f0 2024-03-06T07:25:19.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.125+0000 7f486de81640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48680ccc60 con 0x7f486806d330 2024-03-06T07:25:19.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.126+0000 7f4866ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f48680cd7a0 0x7f486815cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:19.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.126+0000 7f4866ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f48680cd7a0 0x7f486815cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41486/0 (socket says 172.21.15.83:41486) 2024-03-06T07:25:19.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.126+0000 7f4866ffd640 1 -- 172.21.15.83:0/2984180446 learned_addr learned my addr 172.21.15.83:0/2984180446 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:19.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.126+0000 7f4867fff640 1 --2- 172.21.15.83:0/2984180446 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f486806d330 0x7f486806d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:19.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.126+0000 7f48677fe640 1 --2- 172.21.15.83:0/2984180446 >> v2:172.21.15.183:3300/0 conn(0x7f48681424f0 0x7f48680cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:19.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.126+0000 7f4866ffd640 1 -- 172.21.15.83:0/2984180446 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f486806d330 msgr2=0x7f486806d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:19.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.126+0000 7f4866ffd640 1 --2- 172.21.15.83:0/2984180446 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f486806d330 0x7f486806d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.126+0000 7f4866ffd640 1 -- 172.21.15.83:0/2984180446 >> v2:172.21.15.183:3300/0 conn(0x7f48681424f0 msgr2=0x7f48680cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:19.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.126+0000 7f4866ffd640 1 --2- 172.21.15.83:0/2984180446 >> v2:172.21.15.183:3300/0 conn(0x7f48681424f0 0x7f48680cc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.126+0000 7f4866ffd640 1 -- 172.21.15.83:0/2984180446 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48680ccf40 con 0x7f48680cd7a0 2024-03-06T07:25:19.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.127+0000 7f4866ffd640 1 --2- 172.21.15.83:0/2984180446 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f48680cd7a0 0x7f486815cef0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f486806c6d0 tx=0x7f485c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b93d6377f5cad42 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:19.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.127+0000 7f48667fc640 1 -- 172.21.15.83:0/2984180446 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f485c035020 con 0x7f48680cd7a0 2024-03-06T07:25:19.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.127+0000 7f48667fc640 1 -- 172.21.15.83:0/2984180446 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f485c031870 con 0x7f48680cd7a0 2024-03-06T07:25:19.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.127+0000 7f48667fc640 1 -- 172.21.15.83:0/2984180446 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f485c031c40 con 0x7f48680cd7a0 2024-03-06T07:25:19.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.127+0000 7f486de81640 1 -- 172.21.15.83:0/2984180446 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f48680cd7a0 msgr2=0x7f486815cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:19.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.127+0000 7f486de81640 1 --2- 172.21.15.83:0/2984180446 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f48680cd7a0 0x7f486815cef0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f486806c6d0 tx=0x7f485c02b910 comp rx=0 tx=0).stop 2024-03-06T07:25:19.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.127+0000 7f486de81640 1 -- 172.21.15.83:0/2984180446 shutdown_connections 2024-03-06T07:25:19.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.127+0000 7f486de81640 1 --2- 172.21.15.83:0/2984180446 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f486806d330 0x7f486806d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.127+0000 7f486de81640 1 --2- 172.21.15.83:0/2984180446 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f48680cd7a0 0x7f486815cef0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.127+0000 7f486de81640 1 --2- 172.21.15.83:0/2984180446 >> v2:172.21.15.183:3300/0 conn(0x7f48681424f0 0x7f48680cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.127+0000 7f486de81640 1 -- 172.21.15.83:0/2984180446 >> 172.21.15.83:0/2984180446 conn(0x7f48680cf8a0 msgr2=0x7f48680cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:19.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.127+0000 7f486de81640 1 -- 172.21.15.83:0/2984180446 shutdown_connections 2024-03-06T07:25:19.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.127+0000 7f486de81640 1 -- 172.21.15.83:0/2984180446 wait complete. 2024-03-06T07:25:19.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.128+0000 7f486de81640 1 Processor -- start 2024-03-06T07:25:19.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.129+0000 7f486de81640 1 -- start start 2024-03-06T07:25:19.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.129+0000 7f486de81640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f486806d330 0x7f48680b3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:19.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.129+0000 7f486de81640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f48680cd7a0 0x7f486813f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:19.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.129+0000 7f486de81640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f48681424f0 0x7f486815aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:19.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.129+0000 7f486de81640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4868071ce0 con 0x7f486806d330 2024-03-06T07:25:19.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.129+0000 7f486de81640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4868071b60 con 0x7f48681424f0 2024-03-06T07:25:19.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.129+0000 7f486de81640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4868071e60 con 0x7f48680cd7a0 2024-03-06T07:25:19.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.129+0000 7f48677fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f486806d330 0x7f48680b3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:19.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.129+0000 7f48677fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f486806d330 0x7f48680b3a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41498/0 (socket says 172.21.15.83:41498) 2024-03-06T07:25:19.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.130+0000 7f48677fe640 1 -- 172.21.15.83:0/3885814637 learned_addr learned my addr 172.21.15.83:0/3885814637 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:19.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.130+0000 7f4866ffd640 1 --2- 172.21.15.83:0/3885814637 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f48680cd7a0 0x7f486813f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:19.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.130+0000 7f4867fff640 1 --2- 172.21.15.83:0/3885814637 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f48681424f0 0x7f486815aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:19.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.130+0000 7f48677fe640 1 -- 172.21.15.83:0/3885814637 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f48680cd7a0 msgr2=0x7f486813f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:19.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.130+0000 7f48677fe640 1 --2- 172.21.15.83:0/3885814637 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f48680cd7a0 0x7f486813f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.130+0000 7f48677fe640 1 -- 172.21.15.83:0/3885814637 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f48681424f0 msgr2=0x7f486815aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:19.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.130+0000 7f48677fe640 1 --2- 172.21.15.83:0/3885814637 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f48681424f0 0x7f486815aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.130+0000 7f48677fe640 1 -- 172.21.15.83:0/3885814637 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48680b2c60 con 0x7f486806d330 2024-03-06T07:25:19.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.130+0000 7f48677fe640 1 --2- 172.21.15.83:0/3885814637 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f486806d330 0x7f48680b3a20 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f485800c820 tx=0x7f485800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:19.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.130+0000 7f484ffff640 1 -- 172.21.15.83:0/3885814637 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4858015070 con 0x7f486806d330 2024-03-06T07:25:19.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.130+0000 7f484ffff640 1 -- 172.21.15.83:0/3885814637 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4858004020 con 0x7f486806d330 2024-03-06T07:25:19.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.130+0000 7f486de81640 1 -- 172.21.15.83:0/3885814637 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48680b2f40 con 0x7f486806d330 2024-03-06T07:25:19.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.130+0000 7f484ffff640 1 -- 172.21.15.83:0/3885814637 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48580043f0 con 0x7f486806d330 2024-03-06T07:25:19.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.130+0000 7f486de81640 1 -- 172.21.15.83:0/3885814637 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48680b3500 con 0x7f486806d330 2024-03-06T07:25:19.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.131+0000 7f486de81640 1 -- 172.21.15.83:0/3885814637 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4868072940 con 0x7f486806d330 2024-03-06T07:25:19.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.132+0000 7f484ffff640 1 -- 172.21.15.83:0/3885814637 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4858007830 con 0x7f486806d330 2024-03-06T07:25:19.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.132+0000 7f484ffff640 1 --2- 172.21.15.83:0/3885814637 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f485406b4d0 0x7f485406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:19.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.132+0000 7f484ffff640 1 -- 172.21.15.83:0/3885814637 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f485808a020 con 0x7f486806d330 2024-03-06T07:25:19.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.132+0000 7f4866ffd640 1 --2- 172.21.15.83:0/3885814637 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f485406b4d0 0x7f485406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:19.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.133+0000 7f4866ffd640 1 --2- 172.21.15.83:0/3885814637 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f485406b4d0 0x7f485406d990 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f485c036000 tx=0x7f485c03c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:19.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.136+0000 7f484ffff640 1 -- 172.21.15.83:0/3885814637 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4868072940 con 0x7f486806d330 2024-03-06T07:25:19.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.422+0000 7f486de81640 1 -- 172.21.15.83:0/3885814637 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f486804f330 con 0x7f486806d330 2024-03-06T07:25:19.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.422+0000 7f484ffff640 1 -- 172.21.15.83:0/3885814637 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v32) v1 ==== 53+0+73 (secure 0 0 0) 0x7f4858058ea0 con 0x7f486806d330 2024-03-06T07:25:19.424 INFO:teuthology.orchestra.run.smithi083.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:25:19.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.424+0000 7f486de81640 1 -- 172.21.15.83:0/3885814637 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f485406b4d0 msgr2=0x7f485406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:19.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.424+0000 7f486de81640 1 --2- 172.21.15.83:0/3885814637 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f485406b4d0 0x7f485406d990 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f485c036000 tx=0x7f485c03c040 comp rx=0 tx=0).stop 2024-03-06T07:25:19.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.424+0000 7f486de81640 1 -- 172.21.15.83:0/3885814637 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f486806d330 msgr2=0x7f48680b3a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:19.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.424+0000 7f486de81640 1 --2- 172.21.15.83:0/3885814637 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f486806d330 0x7f48680b3a20 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f485800c820 tx=0x7f485800cce0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.424+0000 7f486de81640 1 -- 172.21.15.83:0/3885814637 shutdown_connections 2024-03-06T07:25:19.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.424+0000 7f486de81640 1 --2- 172.21.15.83:0/3885814637 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f485406b4d0 0x7f485406d990 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.424+0000 7f486de81640 1 --2- 172.21.15.83:0/3885814637 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f48681424f0 0x7f486815aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.424+0000 7f486de81640 1 --2- 172.21.15.83:0/3885814637 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f48680cd7a0 0x7f486813f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.424+0000 7f486de81640 1 --2- 172.21.15.83:0/3885814637 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f486806d330 0x7f48680b3a20 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.424+0000 7f486de81640 1 -- 172.21.15.83:0/3885814637 >> 172.21.15.83:0/3885814637 conn(0x7f48680cf8a0 msgr2=0x7f4868072230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:19.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.424+0000 7f486de81640 1 -- 172.21.15.83:0/3885814637 shutdown_connections 2024-03-06T07:25:19.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:19.424+0000 7f486de81640 1 -- 172.21.15.83:0/3885814637 wait complete. 2024-03-06T07:25:19.440 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:25:19.440 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:25:19.441 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi083.front.sepia.ceph.com 2024-03-06T07:25:19.441 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:25:19.441 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:19.441 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:25:19.441 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:25:19.441 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:25:19.441 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:25:19.441 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:19.441 DEBUG:teuthology.orchestra.run.smithi083:> ip addr 2024-03-06T07:25:19.456 INFO:teuthology.orchestra.run.smithi083.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:25:19.457 INFO:teuthology.orchestra.run.smithi083.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:25:19.457 INFO:teuthology.orchestra.run.smithi083.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:25:19.457 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:19.457 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 ::1/128 scope host 2024-03-06T07:25:19.457 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:19.457 INFO:teuthology.orchestra.run.smithi083.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:25:19.457 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:d8:da:1e brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:19.457 INFO:teuthology.orchestra.run.smithi083.stdout: altname enp7s0f0 2024-03-06T07:25:19.457 INFO:teuthology.orchestra.run.smithi083.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:25:19.457 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:d8:da:1f brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:19.457 INFO:teuthology.orchestra.run.smithi083.stdout: altname enp7s0f1 2024-03-06T07:25:19.457 INFO:teuthology.orchestra.run.smithi083.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:25:19.458 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:8f:ce:90 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:19.458 INFO:teuthology.orchestra.run.smithi083.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:25:19.458 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:8f:ce:91 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:19.458 INFO:teuthology.orchestra.run.smithi083.stdout: inet 172.21.15.83/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:25:19.458 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft 42536sec preferred_lft 42536sec 2024-03-06T07:25:19.458 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::ec4:7aff:fe8f:ce91/64 scope link 2024-03-06T07:25:19.458 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:19.458 INFO:teuthology.orchestra.run.smithi083.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:25:19.458 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 1e:29:46:c9:e7:c8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:19.458 INFO:teuthology.orchestra.run.smithi083.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:25:19.458 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:19.458 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::60c5:d7ff:fe70:2b1c/64 scope link 2024-03-06T07:25:19.458 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:19.458 INFO:teuthology.orchestra.run.smithi083.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:25:19.459 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 1e:29:46:c9:e7:c8 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:19.459 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::1c29:46ff:fec9:e7c8/64 scope link 2024-03-06T07:25:19.459 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:19.459 INFO:teuthology.orchestra.run.smithi083.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:25:19.459 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 3e:00:2f:23:d5:45 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:19.459 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::3c00:2fff:fe23:d545/64 scope link 2024-03-06T07:25:19.459 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:19.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:19.459 DEBUG:teuthology.orchestra.run.smithi083:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:25:19.522 INFO:teuthology.orchestra.run.smithi083.stdout:1 2024-03-06T07:25:19.523 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:19.523 DEBUG:teuthology.orchestra.run.smithi083:> route 2024-03-06T07:25:19.578 INFO:teuthology.orchestra.run.smithi083.stdout:Kernel IP routing table 2024-03-06T07:25:19.578 INFO:teuthology.orchestra.run.smithi083.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:25:19.578 INFO:teuthology.orchestra.run.smithi083.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:25:19.578 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:25:19.578 INFO:teuthology.orchestra.run.smithi083.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:25:19.579 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:19.579 DEBUG:teuthology.orchestra.run.smithi083:> (cd / && exec bash -c ' 2024-03-06T07:25:19.579 DEBUG:teuthology.orchestra.run.smithi083:> set -e 2024-03-06T07:25:19.579 DEBUG:teuthology.orchestra.run.smithi083:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:25:19.579 DEBUG:teuthology.orchestra.run.smithi083:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:25:19.579 DEBUG:teuthology.orchestra.run.smithi083:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:25:19.579 DEBUG:teuthology.orchestra.run.smithi083:> ') 2024-03-06T07:25:19.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:19.675 DEBUG:teuthology.orchestra.run.smithi083:> ip netns list 2024-03-06T07:25:19.689 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:25:19.689 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:25:19.689 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:25:19.690 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:19.690 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip link set brx.0 up 2024-03-06T07:25:19.756 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:25:19.756 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:25:19.756 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:19.811 INFO:teuthology.orchestra.run.smithi083.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:25:19.811 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:25:19.811 DEBUG:teuthology.orchestra.run.smithi083:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:19.865 DEBUG:teuthology.orchestra.run.smithi083:> sudo modprobe fuse 2024-03-06T07:25:19.933 DEBUG:teuthology.orchestra.run.smithi083:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:25:19.988 INFO:teuthology.orchestra.run.smithi083.stdout:/proc 2024-03-06T07:25:19.989 INFO:teuthology.orchestra.run.smithi083.stdout:/sys 2024-03-06T07:25:19.989 INFO:teuthology.orchestra.run.smithi083.stdout:/dev 2024-03-06T07:25:19.989 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/security 2024-03-06T07:25:19.989 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/shm 2024-03-06T07:25:19.989 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/pts 2024-03-06T07:25:19.989 INFO:teuthology.orchestra.run.smithi083.stdout:/run 2024-03-06T07:25:19.990 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/cgroup 2024-03-06T07:25:19.990 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/pstore 2024-03-06T07:25:19.990 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/bpf 2024-03-06T07:25:19.990 INFO:teuthology.orchestra.run.smithi083.stdout:/ 2024-03-06T07:25:19.990 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/selinux 2024-03-06T07:25:19.990 INFO:teuthology.orchestra.run.smithi083.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:25:19.990 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/mqueue 2024-03-06T07:25:19.990 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/hugepages 2024-03-06T07:25:19.990 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/debug 2024-03-06T07:25:19.990 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/tracing 2024-03-06T07:25:19.990 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:25:19.990 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/fuse/connections 2024-03-06T07:25:19.990 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/config 2024-03-06T07:25:19.990 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:25:19.990 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:25:19.991 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:25:19.991 INFO:teuthology.orchestra.run.smithi083.stdout:/run/user/1000 2024-03-06T07:25:19.991 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph 2024-03-06T07:25:19.991 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:25:19.991 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:25:19.991 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:25:19.991 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:25:19.991 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns 2024-03-06T07:25:19.991 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:19.991 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:19.991 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:19.991 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:19.992 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:19.993 DEBUG:teuthology.orchestra.run.smithi083:> ls /sys/fs/fuse/connections 2024-03-06T07:25:20.047 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:25:20.048 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:25:20.090 DEBUG:teuthology.orchestra.run.smithi083:> sudo modprobe fuse 2024-03-06T07:25:20.115 DEBUG:teuthology.orchestra.run.smithi083:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:25:20.172 INFO:teuthology.orchestra.run.smithi083.stdout:/proc 2024-03-06T07:25:20.172 INFO:teuthology.orchestra.run.smithi083.stdout:/sys 2024-03-06T07:25:20.172 INFO:teuthology.orchestra.run.smithi083.stdout:/dev 2024-03-06T07:25:20.172 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/security 2024-03-06T07:25:20.172 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/shm 2024-03-06T07:25:20.172 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/pts 2024-03-06T07:25:20.172 INFO:teuthology.orchestra.run.smithi083.stdout:/run 2024-03-06T07:25:20.172 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/cgroup 2024-03-06T07:25:20.172 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/pstore 2024-03-06T07:25:20.173 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/bpf 2024-03-06T07:25:20.173 INFO:teuthology.orchestra.run.smithi083.stdout:/ 2024-03-06T07:25:20.173 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/selinux 2024-03-06T07:25:20.173 INFO:teuthology.orchestra.run.smithi083.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:25:20.173 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/mqueue 2024-03-06T07:25:20.173 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/hugepages 2024-03-06T07:25:20.173 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/debug 2024-03-06T07:25:20.173 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/tracing 2024-03-06T07:25:20.173 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:25:20.173 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/fuse/connections 2024-03-06T07:25:20.173 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/config 2024-03-06T07:25:20.174 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:25:20.174 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:25:20.174 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:25:20.174 INFO:teuthology.orchestra.run.smithi083.stdout:/run/user/1000 2024-03-06T07:25:20.174 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph 2024-03-06T07:25:20.174 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:25:20.174 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:25:20.174 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:25:20.174 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:25:20.174 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns 2024-03-06T07:25:20.174 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:20.174 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:20.174 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:20.175 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:20.175 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:20.175 DEBUG:teuthology.orchestra.run.smithi083:> ls /sys/fs/fuse/connections 2024-03-06T07:25:20.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:2024-03-06T07:25:20.175+0000 7fbeefa0b480 -1 init, newargv = 0x5612016be8c0 newargc=15 2024-03-06T07:25:20.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:ceph-fuse[63034]: starting ceph client 2024-03-06T07:25:20.181 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:ceph-fuse[63034]: starting fuse 2024-03-06T07:25:20.193 INFO:teuthology.orchestra.run.smithi083.stdout:41 2024-03-06T07:25:20.193 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-06T07:25:20.193 DEBUG:teuthology.orchestra.run.smithi083:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:25:20.193 DEBUG:teuthology.orchestra.run.smithi083:> import glob 2024-03-06T07:25:20.193 DEBUG:teuthology.orchestra.run.smithi083:> import re 2024-03-06T07:25:20.193 DEBUG:teuthology.orchestra.run.smithi083:> import os 2024-03-06T07:25:20.193 DEBUG:teuthology.orchestra.run.smithi083:> import subprocess 2024-03-06T07:25:20.193 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:25:20.193 DEBUG:teuthology.orchestra.run.smithi083:> def _find_admin_socket(client_name): 2024-03-06T07:25:20.194 DEBUG:teuthology.orchestra.run.smithi083:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:25:20.194 DEBUG:teuthology.orchestra.run.smithi083:> files = glob.glob(asok_path) 2024-03-06T07:25:20.194 DEBUG:teuthology.orchestra.run.smithi083:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:25:20.194 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:25:20.194 DEBUG:teuthology.orchestra.run.smithi083:> # Given a non-glob path, it better be there 2024-03-06T07:25:20.194 DEBUG:teuthology.orchestra.run.smithi083:> if "*" not in asok_path: 2024-03-06T07:25:20.194 DEBUG:teuthology.orchestra.run.smithi083:> assert(len(files) == 1) 2024-03-06T07:25:20.194 DEBUG:teuthology.orchestra.run.smithi083:> return files[0] 2024-03-06T07:25:20.194 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:25:20.194 DEBUG:teuthology.orchestra.run.smithi083:> for f in files: 2024-03-06T07:25:20.194 DEBUG:teuthology.orchestra.run.smithi083:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:25:20.194 DEBUG:teuthology.orchestra.run.smithi083:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:25:20.194 DEBUG:teuthology.orchestra.run.smithi083:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:25:20.194 DEBUG:teuthology.orchestra.run.smithi083:> contents = proc_f.read() 2024-03-06T07:25:20.194 DEBUG:teuthology.orchestra.run.smithi083:> if mountpoint in contents: 2024-03-06T07:25:20.194 DEBUG:teuthology.orchestra.run.smithi083:> return f 2024-03-06T07:25:20.195 DEBUG:teuthology.orchestra.run.smithi083:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:25:20.195 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:25:20.195 DEBUG:teuthology.orchestra.run.smithi083:> print(_find_admin_socket("client.0")) 2024-03-06T07:25:20.195 DEBUG:teuthology.orchestra.run.smithi083:> ' 2024-03-06T07:25:20.313 INFO:teuthology.orchestra.run.smithi083.stdout:/var/run/ceph/ceph-client.0.63034.asok 2024-03-06T07:25:20.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:25:20.324 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.63034.asok 2024-03-06T07:25:20.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:20.324 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.63034.asok status 2024-03-06T07:25:20.460 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:25:20.461 INFO:teuthology.orchestra.run.smithi083.stdout: "metadata": { 2024-03-06T07:25:20.461 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:25:20.461 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:25:20.461 INFO:teuthology.orchestra.run.smithi083.stdout: "entity_id": "0", 2024-03-06T07:25:20.461 INFO:teuthology.orchestra.run.smithi083.stdout: "hostname": "smithi083", 2024-03-06T07:25:20.461 INFO:teuthology.orchestra.run.smithi083.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:25:20.461 INFO:teuthology.orchestra.run.smithi083.stdout: "pid": "63034", 2024-03-06T07:25:20.461 INFO:teuthology.orchestra.run.smithi083.stdout: "root": "/" 2024-03-06T07:25:20.461 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:25:20.461 INFO:teuthology.orchestra.run.smithi083.stdout: "dentry_count": 0, 2024-03-06T07:25:20.461 INFO:teuthology.orchestra.run.smithi083.stdout: "dentry_pinned_count": 0, 2024-03-06T07:25:20.461 INFO:teuthology.orchestra.run.smithi083.stdout: "id": 6708, 2024-03-06T07:25:20.461 INFO:teuthology.orchestra.run.smithi083.stdout: "inst": { 2024-03-06T07:25:20.462 INFO:teuthology.orchestra.run.smithi083.stdout: "name": { 2024-03-06T07:25:20.462 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "client", 2024-03-06T07:25:20.462 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 6708 2024-03-06T07:25:20.462 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:25:20.462 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": { 2024-03-06T07:25:20.462 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-03-06T07:25:20.462 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:25:20.462 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 1500083332 2024-03-06T07:25:20.462 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:25:20.462 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:25:20.462 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": { 2024-03-06T07:25:20.462 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-03-06T07:25:20.462 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:25:20.462 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 1500083332 2024-03-06T07:25:20.463 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:25:20.463 INFO:teuthology.orchestra.run.smithi083.stdout: "inst_str": "client.6708 192.168.0.1:0/1500083332", 2024-03-06T07:25:20.463 INFO:teuthology.orchestra.run.smithi083.stdout: "addr_str": "192.168.0.1:0/1500083332", 2024-03-06T07:25:20.463 INFO:teuthology.orchestra.run.smithi083.stdout: "inode_count": 1, 2024-03-06T07:25:20.463 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_epoch": 32, 2024-03-06T07:25:20.463 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_epoch": 69, 2024-03-06T07:25:20.463 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:25:20.463 INFO:teuthology.orchestra.run.smithi083.stdout: "blocklisted": false, 2024-03-06T07:25:20.463 INFO:teuthology.orchestra.run.smithi083.stdout: "fs_name": "cephfs" 2024-03-06T07:25:20.463 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:25:20.469 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:20.469 DEBUG:teuthology.orchestra.run.smithi083:> stat --file-system '--printf=%T 2024-03-06T07:25:20.469 DEBUG:teuthology.orchestra.run.smithi083:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:20.484 INFO:teuthology.orchestra.run.smithi083.stdout:fuseblk 2024-03-06T07:25:20.484 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:20.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:20.484 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:20.555 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_fragment_size_max 10 2024-03-06T07:25:20.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.697+0000 7f9c17fe2640 1 Processor -- start 2024-03-06T07:25:20.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.698+0000 7f9c17fe2640 1 -- start start 2024-03-06T07:25:20.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.698+0000 7f9c17fe2640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f9c101614d0 0x7f9c100d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.698+0000 7f9c17fe2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9c100d1910 0x7f9c100d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.698+0000 7f9c17fe2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c100d22e0 0x7f9c100d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.698+0000 7f9c17fe2640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c100d2750 con 0x7f9c100d1910 2024-03-06T07:25:20.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.698+0000 7f9c17fe2640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c101618b0 con 0x7f9c101614d0 2024-03-06T07:25:20.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.698+0000 7f9c17fe2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c100cf400 con 0x7f9c100d22e0 2024-03-06T07:25:20.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.699+0000 7f9c1557d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9c100d1910 0x7f9c100d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.699+0000 7f9c1557d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9c100d1910 0x7f9c100d1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41502/0 (socket says 172.21.15.83:41502) 2024-03-06T07:25:20.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.699+0000 7f9c1557d640 1 -- 172.21.15.83:0/4177784471 learned_addr learned my addr 172.21.15.83:0/4177784471 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:20.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.699+0000 7f9c1657f640 1 --2- 172.21.15.83:0/4177784471 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c100d22e0 0x7f9c100d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.699+0000 7f9c15d7e640 1 --2- 172.21.15.83:0/4177784471 >> v2:172.21.15.183:3300/0 conn(0x7f9c101614d0 0x7f9c100d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.699+0000 7f9c1557d640 1 -- 172.21.15.83:0/4177784471 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c100d22e0 msgr2=0x7f9c100d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:20.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.699+0000 7f9c1557d640 1 --2- 172.21.15.83:0/4177784471 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c100d22e0 0x7f9c100d06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.699+0000 7f9c1557d640 1 -- 172.21.15.83:0/4177784471 >> v2:172.21.15.183:3300/0 conn(0x7f9c101614d0 msgr2=0x7f9c100d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:20.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.699+0000 7f9c1557d640 1 --2- 172.21.15.83:0/4177784471 >> v2:172.21.15.183:3300/0 conn(0x7f9c101614d0 0x7f9c100d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.699+0000 7f9c1557d640 1 -- 172.21.15.83:0/4177784471 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c100cf6a0 con 0x7f9c100d1910 2024-03-06T07:25:20.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.700+0000 7f9c1557d640 1 --2- 172.21.15.83:0/4177784471 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c100d1910 0x7f9c100d1da0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f9c1008c470 tx=0x7f9c0002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0a9d1bad76cc7fe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:20.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.700+0000 7f9c14d7c640 1 -- 172.21.15.83:0/4177784471 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c00035020 con 0x7f9c100d1910 2024-03-06T07:25:20.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.700+0000 7f9c14d7c640 1 -- 172.21.15.83:0/4177784471 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c00031870 con 0x7f9c100d1910 2024-03-06T07:25:20.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.700+0000 7f9c14d7c640 1 -- 172.21.15.83:0/4177784471 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c00031c40 con 0x7f9c100d1910 2024-03-06T07:25:20.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.700+0000 7f9c17fe2640 1 -- 172.21.15.83:0/4177784471 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c100d1910 msgr2=0x7f9c100d1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.700+0000 7f9c17fe2640 1 --2- 172.21.15.83:0/4177784471 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c100d1910 0x7f9c100d1da0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f9c1008c470 tx=0x7f9c0002b910 comp rx=0 tx=0).stop 2024-03-06T07:25:20.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.701+0000 7f9c17fe2640 1 -- 172.21.15.83:0/4177784471 shutdown_connections 2024-03-06T07:25:20.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.701+0000 7f9c17fe2640 1 --2- 172.21.15.83:0/4177784471 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c100d22e0 0x7f9c100d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.701+0000 7f9c17fe2640 1 --2- 172.21.15.83:0/4177784471 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c100d1910 0x7f9c100d1da0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.701+0000 7f9c17fe2640 1 --2- 172.21.15.83:0/4177784471 >> v2:172.21.15.183:3300/0 conn(0x7f9c101614d0 0x7f9c100d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.701+0000 7f9c17fe2640 1 -- 172.21.15.83:0/4177784471 >> 172.21.15.83:0/4177784471 conn(0x7f9c100d3d40 msgr2=0x7f9c100d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:20.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.701+0000 7f9c17fe2640 1 -- 172.21.15.83:0/4177784471 shutdown_connections 2024-03-06T07:25:20.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.701+0000 7f9c17fe2640 1 -- 172.21.15.83:0/4177784471 wait complete. 2024-03-06T07:25:20.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.702+0000 7f9c17fe2640 1 Processor -- start 2024-03-06T07:25:20.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.702+0000 7f9c17fe2640 1 -- start start 2024-03-06T07:25:20.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.704+0000 7f9c17fe2640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c100d1910 0x7f9c10150b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.704+0000 7f9c17fe2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c100d22e0 0x7f9c101f0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.704+0000 7f9c17fe2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c101614d0 0x7f9c10160070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.704+0000 7f9c17fe2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c100b0910 con 0x7f9c101614d0 2024-03-06T07:25:20.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.704+0000 7f9c17fe2640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c100cfc60 con 0x7f9c100d1910 2024-03-06T07:25:20.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.704+0000 7f9c17fe2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c100d0be0 con 0x7f9c100d22e0 2024-03-06T07:25:20.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.704+0000 7f9c1657f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c101614d0 0x7f9c10160070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.704+0000 7f9c1657f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c101614d0 0x7f9c10160070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41506/0 (socket says 172.21.15.83:41506) 2024-03-06T07:25:20.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.704+0000 7f9c1657f640 1 -- 172.21.15.83:0/3985714398 learned_addr learned my addr 172.21.15.83:0/3985714398 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:20.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.704+0000 7f9c1557d640 1 --2- 172.21.15.83:0/3985714398 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c100d22e0 0x7f9c101f0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.704+0000 7f9c15d7e640 1 --2- 172.21.15.83:0/3985714398 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c100d1910 0x7f9c10150b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.704+0000 7f9c1657f640 1 -- 172.21.15.83:0/3985714398 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c100d22e0 msgr2=0x7f9c101f0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.704+0000 7f9c1657f640 1 --2- 172.21.15.83:0/3985714398 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c100d22e0 0x7f9c101f0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.704+0000 7f9c1657f640 1 -- 172.21.15.83:0/3985714398 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c100d1910 msgr2=0x7f9c10150b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.704+0000 7f9c1657f640 1 --2- 172.21.15.83:0/3985714398 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c100d1910 0x7f9c10150b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.704+0000 7f9c1657f640 1 -- 172.21.15.83:0/3985714398 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c101510a0 con 0x7f9c101614d0 2024-03-06T07:25:20.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.704+0000 7f9c15d7e640 1 --2- 172.21.15.83:0/3985714398 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c100d1910 0x7f9c10150b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:20.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.704+0000 7f9c1657f640 1 --2- 172.21.15.83:0/3985714398 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c101614d0 0x7f9c10160070 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f9c08007e40 tx=0x7f9c0800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:20.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.704+0000 7f9c067fc640 1 -- 172.21.15.83:0/3985714398 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c08014620 con 0x7f9c101614d0 2024-03-06T07:25:20.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.704+0000 7f9c067fc640 1 -- 172.21.15.83:0/3985714398 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c08019070 con 0x7f9c101614d0 2024-03-06T07:25:20.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.704+0000 7f9c17fe2640 1 -- 172.21.15.83:0/3985714398 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c101f3df0 con 0x7f9c101614d0 2024-03-06T07:25:20.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.704+0000 7f9c067fc640 1 -- 172.21.15.83:0/3985714398 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c08014a20 con 0x7f9c101614d0 2024-03-06T07:25:20.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.704+0000 7f9c17fe2640 1 -- 172.21.15.83:0/3985714398 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c101f43b0 con 0x7f9c101614d0 2024-03-06T07:25:20.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.705+0000 7f9c17fe2640 1 -- 172.21.15.83:0/3985714398 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c10068ff0 con 0x7f9c101614d0 2024-03-06T07:25:20.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.706+0000 7f9c067fc640 1 -- 172.21.15.83:0/3985714398 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9c08014bc0 con 0x7f9c101614d0 2024-03-06T07:25:20.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.706+0000 7f9c067fc640 1 --2- 172.21.15.83:0/3985714398 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9bf406b4d0 0x7f9bf406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.706+0000 7f9c15d7e640 1 --2- 172.21.15.83:0/3985714398 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9bf406b4d0 0x7f9bf406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.707+0000 7f9c067fc640 1 -- 172.21.15.83:0/3985714398 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9c0808aa70 con 0x7f9c101614d0 2024-03-06T07:25:20.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.707+0000 7f9c15d7e640 1 --2- 172.21.15.83:0/3985714398 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9bf406b4d0 0x7f9bf406d990 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c00b970 tx=0x7f9c0c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:20.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.711+0000 7f9c067fc640 1 -- 172.21.15.83:0/3985714398 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9c101510a0 con 0x7f9c101614d0 2024-03-06T07:25:20.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.888+0000 7f9c17fe2640 1 -- 172.21.15.83:0/3985714398 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) v1 -- 0x7f9c101f1280 con 0x7f9c101614d0 2024-03-06T07:25:20.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.891+0000 7f9c067fc640 1 -- 172.21.15.83:0/3985714398 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v23)=0 v23) v1 ==== 124+0+0 (secure 0 0 0) 0x7f9c080598f0 con 0x7f9c101614d0 2024-03-06T07:25:20.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.894+0000 7f9c17fe2640 1 -- 172.21.15.83:0/3985714398 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9bf406b4d0 msgr2=0x7f9bf406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.894+0000 7f9c17fe2640 1 --2- 172.21.15.83:0/3985714398 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9bf406b4d0 0x7f9bf406d990 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c00b970 tx=0x7f9c0c008040 comp rx=0 tx=0).stop 2024-03-06T07:25:20.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.894+0000 7f9c17fe2640 1 -- 172.21.15.83:0/3985714398 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c101614d0 msgr2=0x7f9c10160070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.894+0000 7f9c17fe2640 1 --2- 172.21.15.83:0/3985714398 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c101614d0 0x7f9c10160070 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f9c08007e40 tx=0x7f9c0800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.894+0000 7f9c17fe2640 1 -- 172.21.15.83:0/3985714398 shutdown_connections 2024-03-06T07:25:20.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.894+0000 7f9c17fe2640 1 --2- 172.21.15.83:0/3985714398 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9bf406b4d0 0x7f9bf406d990 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.894+0000 7f9c17fe2640 1 --2- 172.21.15.83:0/3985714398 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c101614d0 0x7f9c10160070 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.894+0000 7f9c17fe2640 1 --2- 172.21.15.83:0/3985714398 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c100d22e0 0x7f9c101f0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.894+0000 7f9c17fe2640 1 --2- 172.21.15.83:0/3985714398 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c100d1910 0x7f9c10150b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.894+0000 7f9c17fe2640 1 -- 172.21.15.83:0/3985714398 >> 172.21.15.83:0/3985714398 conn(0x7f9c100d3d40 msgr2=0x7f9c100d4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:20.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.894+0000 7f9c17fe2640 1 -- 172.21.15.83:0/3985714398 shutdown_connections 2024-03-06T07:25:20.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:20.894+0000 7f9c17fe2640 1 -- 172.21.15.83:0/3985714398 wait complete. 2024-03-06T07:25:28.867 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:25:28.869 DEBUG:teuthology.orchestra.run.smithi183:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:25:30.907 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:31.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.006+0000 7ff90f2af640 1 Processor -- start 2024-03-06T07:25:31.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.006+0000 7ff90f2af640 1 -- start start 2024-03-06T07:25:31.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.007+0000 7ff90f2af640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff90815dfe0 0x7ff9080cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.007+0000 7ff90f2af640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ff9080cf830 0x7ff9080d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.007+0000 7ff90f2af640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9080d27f0 0x7ff9080d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.007+0000 7ff90f2af640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9080b31e0 con 0x7ff90815dfe0 2024-03-06T07:25:31.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.007+0000 7ff90f2af640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9080b3d60 con 0x7ff9080cf830 2024-03-06T07:25:31.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.007+0000 7ff90f2af640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9080b3430 con 0x7ff9080d27f0 2024-03-06T07:25:31.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.007+0000 7ff90d04b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff90815dfe0 0x7ff9080cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.007+0000 7ff90d04b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff90815dfe0 0x7ff9080cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:39898/0 (socket says 172.21.15.83:39898) 2024-03-06T07:25:31.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.007+0000 7ff90d04b640 1 -- 172.21.15.83:0/4033035539 learned_addr learned my addr 172.21.15.83:0/4033035539 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:31.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.008+0000 7ff90d84c640 1 --2- 172.21.15.83:0/4033035539 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9080d27f0 0x7ff9080d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.008+0000 7ff90c84a640 1 --2- 172.21.15.83:0/4033035539 >> v2:172.21.15.183:3300/0 conn(0x7ff9080cf830 0x7ff9080d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.008+0000 7ff90d04b640 1 -- 172.21.15.83:0/4033035539 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9080d27f0 msgr2=0x7ff9080d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:31.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.008+0000 7ff90d04b640 1 --2- 172.21.15.83:0/4033035539 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9080d27f0 0x7ff9080d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.008+0000 7ff90d04b640 1 -- 172.21.15.83:0/4033035539 >> v2:172.21.15.183:3300/0 conn(0x7ff9080cf830 msgr2=0x7ff9080d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:31.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.008+0000 7ff90d04b640 1 --2- 172.21.15.83:0/4033035539 >> v2:172.21.15.183:3300/0 conn(0x7ff9080cf830 0x7ff9080d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.008+0000 7ff90d04b640 1 -- 172.21.15.83:0/4033035539 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9080b3710 con 0x7ff90815dfe0 2024-03-06T07:25:31.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.008+0000 7ff90d84c640 1 --2- 172.21.15.83:0/4033035539 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9080d27f0 0x7ff9080d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:31.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.008+0000 7ff90c84a640 1 --2- 172.21.15.83:0/4033035539 >> v2:172.21.15.183:3300/0 conn(0x7ff9080cf830 0x7ff9080d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:31.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.008+0000 7ff90d04b640 1 --2- 172.21.15.83:0/4033035539 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff90815dfe0 0x7ff9080cf2f0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7ff9080d16b0 tx=0x7ff8f802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cad8eae1a2fef9b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.008+0000 7ff907fff640 1 -- 172.21.15.83:0/4033035539 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f8035020 con 0x7ff90815dfe0 2024-03-06T07:25:31.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.008+0000 7ff907fff640 1 -- 172.21.15.83:0/4033035539 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8f8031870 con 0x7ff90815dfe0 2024-03-06T07:25:31.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.009+0000 7ff907fff640 1 -- 172.21.15.83:0/4033035539 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f8031c40 con 0x7ff90815dfe0 2024-03-06T07:25:31.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.009+0000 7ff90f2af640 1 -- 172.21.15.83:0/4033035539 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff90815dfe0 msgr2=0x7ff9080cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.009+0000 7ff90f2af640 1 --2- 172.21.15.83:0/4033035539 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff90815dfe0 0x7ff9080cf2f0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7ff9080d16b0 tx=0x7ff8f802b910 comp rx=0 tx=0).stop 2024-03-06T07:25:31.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.009+0000 7ff90f2af640 1 -- 172.21.15.83:0/4033035539 shutdown_connections 2024-03-06T07:25:31.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.009+0000 7ff90f2af640 1 --2- 172.21.15.83:0/4033035539 >> v2:172.21.15.183:3300/0 conn(0x7ff9080cf830 0x7ff9080d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.009+0000 7ff90f2af640 1 --2- 172.21.15.83:0/4033035539 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9080d27f0 0x7ff9080d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.009+0000 7ff90f2af640 1 --2- 172.21.15.83:0/4033035539 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff90815dfe0 0x7ff9080cf2f0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.009+0000 7ff90f2af640 1 -- 172.21.15.83:0/4033035539 >> 172.21.15.83:0/4033035539 conn(0x7ff9080d3cd0 msgr2=0x7ff9080d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:31.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.009+0000 7ff90f2af640 1 -- 172.21.15.83:0/4033035539 shutdown_connections 2024-03-06T07:25:31.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.009+0000 7ff90f2af640 1 -- 172.21.15.83:0/4033035539 wait complete. 2024-03-06T07:25:31.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.011+0000 7ff90f2af640 1 Processor -- start 2024-03-06T07:25:31.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.011+0000 7ff90f2af640 1 -- start start 2024-03-06T07:25:31.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.012+0000 7ff90f2af640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9080cf830 0x7ff9080a1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.012+0000 7ff90f2af640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9080d27f0 0x7ff9081f3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.012+0000 7ff90f2af640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff90815dfe0 0x7ff908160750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.012+0000 7ff90f2af640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9080d1df0 con 0x7ff9080d27f0 2024-03-06T07:25:31.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.012+0000 7ff90f2af640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9080d1c70 con 0x7ff90815dfe0 2024-03-06T07:25:31.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.012+0000 7ff90f2af640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff90815e3c0 con 0x7ff9080cf830 2024-03-06T07:25:31.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.012+0000 7ff90c84a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9080d27f0 0x7ff9081f3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.012+0000 7ff90c84a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9080d27f0 0x7ff9081f3040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:39904/0 (socket says 172.21.15.83:39904) 2024-03-06T07:25:31.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.012+0000 7ff90c84a640 1 -- 172.21.15.83:0/1724108598 learned_addr learned my addr 172.21.15.83:0/1724108598 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:31.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.012+0000 7ff90d04b640 1 --2- 172.21.15.83:0/1724108598 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9080cf830 0x7ff9080a1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.012+0000 7ff90d84c640 1 --2- 172.21.15.83:0/1724108598 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff90815dfe0 0x7ff908160750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.013+0000 7ff90c84a640 1 -- 172.21.15.83:0/1724108598 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9080cf830 msgr2=0x7ff9080a1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.013+0000 7ff90c84a640 1 --2- 172.21.15.83:0/1724108598 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9080cf830 0x7ff9080a1690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.013+0000 7ff90c84a640 1 -- 172.21.15.83:0/1724108598 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff90815dfe0 msgr2=0x7ff908160750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.013+0000 7ff90c84a640 1 --2- 172.21.15.83:0/1724108598 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff90815dfe0 0x7ff908160750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.013+0000 7ff90c84a640 1 -- 172.21.15.83:0/1724108598 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9081f3580 con 0x7ff9080d27f0 2024-03-06T07:25:31.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.013+0000 7ff90d04b640 1 --2- 172.21.15.83:0/1724108598 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9080cf830 0x7ff9080a1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:31.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.013+0000 7ff90d84c640 1 --2- 172.21.15.83:0/1724108598 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff90815dfe0 0x7ff908160750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:31.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.013+0000 7ff90c84a640 1 --2- 172.21.15.83:0/1724108598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9080d27f0 0x7ff9081f3040 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc00c7c0 tx=0x7ff8fc00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.013+0000 7ff905ffb640 1 -- 172.21.15.83:0/1724108598 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8fc015070 con 0x7ff9080d27f0 2024-03-06T07:25:31.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.013+0000 7ff905ffb640 1 -- 172.21.15.83:0/1724108598 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8fc007820 con 0x7ff9080d27f0 2024-03-06T07:25:31.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.013+0000 7ff90f2af640 1 -- 172.21.15.83:0/1724108598 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9080cced0 con 0x7ff9080d27f0 2024-03-06T07:25:31.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.013+0000 7ff905ffb640 1 -- 172.21.15.83:0/1724108598 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8fc007bf0 con 0x7ff9080d27f0 2024-03-06T07:25:31.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.013+0000 7ff90f2af640 1 -- 172.21.15.83:0/1724108598 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9080cd490 con 0x7ff9080d27f0 2024-03-06T07:25:31.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.014+0000 7ff90f2af640 1 -- 172.21.15.83:0/1724108598 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff90804f330 con 0x7ff9080d27f0 2024-03-06T07:25:31.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.014+0000 7ff905ffb640 1 -- 172.21.15.83:0/1724108598 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff8fc018070 con 0x7ff9080d27f0 2024-03-06T07:25:31.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.015+0000 7ff905ffb640 1 --2- 172.21.15.83:0/1724108598 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff8f406b4d0 0x7ff8f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.015+0000 7ff905ffb640 1 -- 172.21.15.83:0/1724108598 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff8fc08a0d0 con 0x7ff9080d27f0 2024-03-06T07:25:31.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.015+0000 7ff90d04b640 1 --2- 172.21.15.83:0/1724108598 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff8f406b4d0 0x7ff8f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.015+0000 7ff90d04b640 1 --2- 172.21.15.83:0/1724108598 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff8f406b4d0 0x7ff8f406d990 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7ff8f8036000 tx=0x7ff8f803c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.019+0000 7ff905ffb640 1 -- 172.21.15.83:0/1724108598 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff8fc058f50 con 0x7ff9080d27f0 2024-03-06T07:25:31.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.234+0000 7ff90f2af640 1 -- 172.21.15.83:0/1724108598 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9080b23d0 con 0x7ff9080d27f0 2024-03-06T07:25:31.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.234+0000 7ff905ffb640 1 -- 172.21.15.83:0/1724108598 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4982 (secure 0 0 0) 0x7ff8fc058850 con 0x7ff9080d27f0 2024-03-06T07:25:31.236 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:25:31.236 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":32,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6010,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/2142942385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2142942385},{"type":"v1","addr":"172.21.15.183:6834","nonce":2142942385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":6013,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3742438064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":3742438064},{"type":"v1","addr":"172.21.15.183:6835","nonce":3742438064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":6540,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/1549894851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1549894851},{"type":"v1","addr":"172.21.15.83:6836","nonce":1549894851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:12.481670+0000","modified":"2024-03-06T07:25:15.392072+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6543},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6543":{"gid":6543,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/602775648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":602775648},{"type":"v1","addr":"172.21.15.83:6837","nonce":602775648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6543,"qdb_cluster":[6543]},"id":5}]} 2024-03-06T07:25:31.236 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 32 2024-03-06T07:25:31.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.236+0000 7ff90f2af640 1 -- 172.21.15.83:0/1724108598 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff8f406b4d0 msgr2=0x7ff8f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.236+0000 7ff90f2af640 1 --2- 172.21.15.83:0/1724108598 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff8f406b4d0 0x7ff8f406d990 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7ff8f8036000 tx=0x7ff8f803c040 comp rx=0 tx=0).stop 2024-03-06T07:25:31.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.236+0000 7ff90f2af640 1 -- 172.21.15.83:0/1724108598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9080d27f0 msgr2=0x7ff9081f3040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.236+0000 7ff90f2af640 1 --2- 172.21.15.83:0/1724108598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9080d27f0 0x7ff9081f3040 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc00c7c0 tx=0x7ff8fc00cc80 comp rx=0 tx=0).stop 2024-03-06T07:25:31.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.236+0000 7ff90f2af640 1 -- 172.21.15.83:0/1724108598 shutdown_connections 2024-03-06T07:25:31.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.236+0000 7ff90f2af640 1 --2- 172.21.15.83:0/1724108598 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff8f406b4d0 0x7ff8f406d990 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.236+0000 7ff90f2af640 1 --2- 172.21.15.83:0/1724108598 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff90815dfe0 0x7ff908160750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.236+0000 7ff90f2af640 1 --2- 172.21.15.83:0/1724108598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9080d27f0 0x7ff9081f3040 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.236+0000 7ff90f2af640 1 --2- 172.21.15.83:0/1724108598 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9080cf830 0x7ff9080a1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.236+0000 7ff90f2af640 1 -- 172.21.15.83:0/1724108598 >> 172.21.15.83:0/1724108598 conn(0x7ff9080d3cd0 msgr2=0x7ff9080b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:31.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.236+0000 7ff90f2af640 1 -- 172.21.15.83:0/1724108598 shutdown_connections 2024-03-06T07:25:31.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.236+0000 7ff90f2af640 1 -- 172.21.15.83:0/1724108598 wait complete. 2024-03-06T07:25:31.249 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:25:31.366 INFO:teuthology.orchestra.run.smithi083.stdout:{"mds_cache":{"strays_created":0}} 2024-03-06T07:25:31.366 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 0 } } 2024-03-06T07:25:31.367 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:25:31.367 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:31.382 DEBUG:teuthology.orchestra.run.smithi083:> stdin-killer -- python3 -c ' 2024-03-06T07:25:31.382 DEBUG:teuthology.orchestra.run.smithi083:> import os 2024-03-06T07:25:31.382 DEBUG:teuthology.orchestra.run.smithi083:> import uuid 2024-03-06T07:25:31.382 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:25:31.382 DEBUG:teuthology.orchestra.run.smithi083:> n = 100 2024-03-06T07:25:31.382 DEBUG:teuthology.orchestra.run.smithi083:> create_hard_links = False 2024-03-06T07:25:31.382 DEBUG:teuthology.orchestra.run.smithi083:> if 0 > 0: 2024-03-06T07:25:31.383 DEBUG:teuthology.orchestra.run.smithi083:> create_hard_links = True 2024-03-06T07:25:31.383 DEBUG:teuthology.orchestra.run.smithi083:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:25:31.383 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:25:31.383 DEBUG:teuthology.orchestra.run.smithi083:> dpath = os.path.dirname(path) 2024-03-06T07:25:31.383 DEBUG:teuthology.orchestra.run.smithi083:> fnameprefix = os.path.basename(path) 2024-03-06T07:25:31.383 DEBUG:teuthology.orchestra.run.smithi083:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:25:31.383 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:25:31.383 DEBUG:teuthology.orchestra.run.smithi083:> try: 2024-03-06T07:25:31.383 DEBUG:teuthology.orchestra.run.smithi083:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:25:31.383 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:25:31.383 DEBUG:teuthology.orchestra.run.smithi083:> for i in range(n): 2024-03-06T07:25:31.383 DEBUG:teuthology.orchestra.run.smithi083:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:25:31.383 DEBUG:teuthology.orchestra.run.smithi083:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:25:31.383 DEBUG:teuthology.orchestra.run.smithi083:> f.write(f"{i}") 2024-03-06T07:25:31.383 DEBUG:teuthology.orchestra.run.smithi083:> if False: 2024-03-06T07:25:31.383 DEBUG:teuthology.orchestra.run.smithi083:> f.flush() 2024-03-06T07:25:31.383 DEBUG:teuthology.orchestra.run.smithi083:> os.fsync(f.fileno()) 2024-03-06T07:25:31.383 DEBUG:teuthology.orchestra.run.smithi083:> if True: 2024-03-06T07:25:31.384 DEBUG:teuthology.orchestra.run.smithi083:> os.unlink(fpath) 2024-03-06T07:25:31.384 DEBUG:teuthology.orchestra.run.smithi083:> if False: 2024-03-06T07:25:31.384 DEBUG:teuthology.orchestra.run.smithi083:> os.fsync(dirfd) 2024-03-06T07:25:31.384 DEBUG:teuthology.orchestra.run.smithi083:> if create_hard_links: 2024-03-06T07:25:31.384 DEBUG:teuthology.orchestra.run.smithi083:> for j in range(0): 2024-03-06T07:25:31.384 DEBUG:teuthology.orchestra.run.smithi083:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:25:31.384 DEBUG:teuthology.orchestra.run.smithi083:> if True: 2024-03-06T07:25:31.384 DEBUG:teuthology.orchestra.run.smithi083:> os.fsync(dirfd) 2024-03-06T07:25:31.384 DEBUG:teuthology.orchestra.run.smithi083:> finally: 2024-03-06T07:25:31.384 DEBUG:teuthology.orchestra.run.smithi083:> os.close(dirfd) 2024-03-06T07:25:31.384 DEBUG:teuthology.orchestra.run.smithi083:> ' 2024-03-06T07:25:31.532 INFO:teuthology.orchestra.run.smithi083.stderr:Traceback (most recent call last): 2024-03-06T07:25:31.532 INFO:teuthology.orchestra.run.smithi083.stderr: File "", line 26, in 2024-03-06T07:25:31.532 INFO:teuthology.orchestra.run.smithi083.stderr:OSError: [Errno 28] No space left on device: '/home/ubuntu/cephtest/mnt.0/subdir/file_10' 2024-03-06T07:25:31.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31 stdin-killer INFO: command exited with status 1: exiting normally with same code! 2024-03-06T07:25:31.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:25:31.544 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:31.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.682+0000 7f32a4ea4640 1 Processor -- start 2024-03-06T07:25:31.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.682+0000 7f32a4ea4640 1 -- start start 2024-03-06T07:25:31.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.683+0000 7f32a4ea4640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f32a00d2710 0x7f32a00b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.683+0000 7f32a4ea4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a00b3430 con 0x7f32a00d22b0 2024-03-06T07:25:31.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.683+0000 7f32a4ea4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a00b3710 con 0x7f32a015dfe0 2024-03-06T07:25:31.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.683+0000 7f32a4ea4640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32a00b31e0 con 0x7f32a00d2710 2024-03-06T07:25:31.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.683+0000 7f3297fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f32a00d22b0 0x7f32a00b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39116/0 (socket says 172.21.15.83:39116) 2024-03-06T07:25:31.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.683+0000 7f3297fff640 1 -- 172.21.15.83:0/3223503244 learned_addr learned my addr 172.21.15.83:0/3223503244 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:25:31.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.683+0000 7f329ed9d640 1 --2- 172.21.15.83:0/3223503244 >> v2:172.21.15.183:3300/0 conn(0x7f32a00d2710 0x7f32a00b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.683+0000 7f329dd9b640 1 -- 172.21.15.83:0/3223503244 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4110041439 0 0) 0x7f32a00b3430 con 0x7f32a00d22b0 2024-03-06T07:25:31.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.683+0000 7f329dd9b640 1 -- 172.21.15.83:0/3223503244 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f328c003520 con 0x7f32a00d22b0 2024-03-06T07:25:31.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.683+0000 7f329dd9b640 1 -- 172.21.15.83:0/3223503244 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3316379533 0 0) 0x7f328c003520 con 0x7f32a00d22b0 2024-03-06T07:25:31.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.683+0000 7f329dd9b640 1 -- 172.21.15.83:0/3223503244 >> v2:172.21.15.183:3300/0 conn(0x7f32a00d2710 msgr2=0x7f32a00b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:31.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.683+0000 7f329dd9b640 1 --2- 172.21.15.83:0/3223503244 >> v2:172.21.15.183:3300/0 conn(0x7f32a00d2710 0x7f32a00b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.683+0000 7f329dd9b640 1 -- 172.21.15.83:0/3223503244 >> v1:172.21.15.183:6789/0 conn(0x7f32a015dfe0 legacy=0x7f32a00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.683+0000 7f329dd9b640 1 -- 172.21.15.83:0/3223503244 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a00b39f0 con 0x7f32a00d22b0 2024-03-06T07:25:31.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.683+0000 7f329dd9b640 1 -- 172.21.15.83:0/3223503244 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f3288001670 con 0x7f32a00d22b0 2024-03-06T07:25:31.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.683+0000 7f329dd9b640 1 -- 172.21.15.83:0/3223503244 >> v1:172.21.15.83:6789/0 conn(0x7f32a00d22b0 legacy=0x7f32a00b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.683+0000 7f329dd9b640 1 --2- 172.21.15.83:0/3223503244 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f328c0039e0 0x7f328c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.684+0000 7f329dd9b640 1 --2- 172.21.15.83:0/3223503244 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f328c004800 0x7f328c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.684+0000 7f329dd9b640 1 --2- 172.21.15.83:0/3223503244 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f328c0051c0 0x7f328c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.684+0000 7f329dd9b640 1 -- 172.21.15.83:0/3223503244 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a00b31e0 con 0x7f328c004800 2024-03-06T07:25:31.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.684+0000 7f329dd9b640 1 -- 172.21.15.83:0/3223503244 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f328c00c090 con 0x7f328c0039e0 2024-03-06T07:25:31.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.684+0000 7f329dd9b640 1 -- 172.21.15.83:0/3223503244 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f328c00c290 con 0x7f328c0051c0 2024-03-06T07:25:31.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.684+0000 7f3297fff640 1 --2- 172.21.15.83:0/3223503244 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f328c004800 0x7f328c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.684+0000 7f329e59c640 1 --2- 172.21.15.83:0/3223503244 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f328c0039e0 0x7f328c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.684+0000 7f329ed9d640 1 --2- 172.21.15.83:0/3223503244 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f328c0051c0 0x7f328c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.684+0000 7f3297fff640 1 -- 172.21.15.83:0/3223503244 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f328c0051c0 msgr2=0x7f328c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:31.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.684+0000 7f3297fff640 1 --2- 172.21.15.83:0/3223503244 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f328c0051c0 0x7f328c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.684+0000 7f3297fff640 1 -- 172.21.15.83:0/3223503244 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f328c0039e0 msgr2=0x7f328c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:31.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.684+0000 7f3297fff640 1 --2- 172.21.15.83:0/3223503244 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f328c0039e0 0x7f328c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.684+0000 7f3297fff640 1 -- 172.21.15.83:0/3223503244 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f328c003520 con 0x7f328c004800 2024-03-06T07:25:31.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.684+0000 7f329ed9d640 1 -- 172.21.15.83:0/3223503244 reap_dead start 2024-03-06T07:25:31.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.685+0000 7f3297fff640 1 --2- 172.21.15.83:0/3223503244 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f328c004800 0x7f328c004c80 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f328800c240 tx=0x7f328c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d20a8c441cdc63e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.685+0000 7f329dd9b640 1 -- 172.21.15.83:0/3223503244 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3288007d10 con 0x7f328c004800 2024-03-06T07:25:31.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.685+0000 7f329dd9b640 1 -- 172.21.15.83:0/3223503244 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3288030070 con 0x7f328c004800 2024-03-06T07:25:31.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.685+0000 7f329dd9b640 1 -- 172.21.15.83:0/3223503244 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3288005d90 con 0x7f328c004800 2024-03-06T07:25:31.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.685+0000 7f32a4ea4640 1 -- 172.21.15.83:0/3223503244 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f328c004800 msgr2=0x7f328c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.685+0000 7f32a4ea4640 1 --2- 172.21.15.83:0/3223503244 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f328c004800 0x7f328c004c80 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f328800c240 tx=0x7f328c005830 comp rx=0 tx=0).stop 2024-03-06T07:25:31.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.685+0000 7f32a4ea4640 1 -- 172.21.15.83:0/3223503244 shutdown_connections 2024-03-06T07:25:31.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.685+0000 7f32a4ea4640 1 --2- 172.21.15.83:0/3223503244 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f328c004800 0x7f328c004c80 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.685+0000 7f32a4ea4640 1 -- 172.21.15.83:0/3223503244 >> 172.21.15.83:0/3223503244 conn(0x7f32a00d3cd0 msgr2=0x7f32a00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:31.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.685+0000 7f32a4ea4640 1 -- 172.21.15.83:0/3223503244 shutdown_connections 2024-03-06T07:25:31.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.686+0000 7f32a4ea4640 1 -- 172.21.15.83:0/3223503244 wait complete. 2024-03-06T07:25:31.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.687+0000 7f32a4ea4640 1 Processor -- start 2024-03-06T07:25:31.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.687+0000 7f32a4ea4640 1 -- start start 2024-03-06T07:25:31.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.687+0000 7f32a4ea4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f328c004800 0x7f32a00b97d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.687+0000 7f32a4ea4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f32a015bc90 0x7f32a015b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.688+0000 7f32a4ea4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f32a006cc10 0x7f32a006d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.688+0000 7f32a4ea4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a00d1c70 con 0x7f328c004800 2024-03-06T07:25:31.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.688+0000 7f32a4ea4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a00b3d60 con 0x7f32a006cc10 2024-03-06T07:25:31.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.688+0000 7f32a4ea4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32a00d1df0 con 0x7f32a015bc90 2024-03-06T07:25:31.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.688+0000 7f329e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f328c004800 0x7f32a00b97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.688+0000 7f329e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f328c004800 0x7f32a00b97d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:39914/0 (socket says 172.21.15.83:39914) 2024-03-06T07:25:31.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.688+0000 7f329e59c640 1 -- 172.21.15.83:0/4177446207 learned_addr learned my addr 172.21.15.83:0/4177446207 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:31.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.688+0000 7f3297fff640 1 --2- 172.21.15.83:0/4177446207 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f32a015bc90 0x7f32a015b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.688+0000 7f329ed9d640 1 --2- 172.21.15.83:0/4177446207 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f32a006cc10 0x7f32a006d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.688+0000 7f329e59c640 1 -- 172.21.15.83:0/4177446207 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f32a015bc90 msgr2=0x7f32a015b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.688+0000 7f329e59c640 1 --2- 172.21.15.83:0/4177446207 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f32a015bc90 0x7f32a015b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.688+0000 7f329e59c640 1 -- 172.21.15.83:0/4177446207 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f32a006cc10 msgr2=0x7f32a006d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.688+0000 7f329e59c640 1 --2- 172.21.15.83:0/4177446207 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f32a006cc10 0x7f32a006d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.688+0000 7f329e59c640 1 -- 172.21.15.83:0/4177446207 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a015c500 con 0x7f328c004800 2024-03-06T07:25:31.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.688+0000 7f3297fff640 1 --2- 172.21.15.83:0/4177446207 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f32a015bc90 0x7f32a015b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:31.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.688+0000 7f329ed9d640 1 --2- 172.21.15.83:0/4177446207 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f32a006cc10 0x7f32a006d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:31.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.688+0000 7f329e59c640 1 --2- 172.21.15.83:0/4177446207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f328c004800 0x7f32a00b97d0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f329000f620 tx=0x7f329000fae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.688+0000 7f3296ffd640 1 -- 172.21.15.83:0/4177446207 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32900052e0 con 0x7f328c004800 2024-03-06T07:25:31.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.688+0000 7f3296ffd640 1 -- 172.21.15.83:0/4177446207 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3290018070 con 0x7f328c004800 2024-03-06T07:25:31.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.688+0000 7f32a4ea4640 1 -- 172.21.15.83:0/4177446207 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32a00b68d0 con 0x7f328c004800 2024-03-06T07:25:31.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.688+0000 7f3296ffd640 1 -- 172.21.15.83:0/4177446207 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3290003810 con 0x7f328c004800 2024-03-06T07:25:31.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.688+0000 7f32a4ea4640 1 -- 172.21.15.83:0/4177446207 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32a00b73f0 con 0x7f328c004800 2024-03-06T07:25:31.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.689+0000 7f32a4ea4640 1 -- 172.21.15.83:0/4177446207 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32a006d5d0 con 0x7f328c004800 2024-03-06T07:25:31.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.690+0000 7f3296ffd640 1 -- 172.21.15.83:0/4177446207 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f32900039b0 con 0x7f328c004800 2024-03-06T07:25:31.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.690+0000 7f3296ffd640 1 --2- 172.21.15.83:0/4177446207 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f328006b500 0x7f328006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.690+0000 7f3297fff640 1 --2- 172.21.15.83:0/4177446207 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f328006b500 0x7f328006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.690+0000 7f3296ffd640 1 -- 172.21.15.83:0/4177446207 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3290089f30 con 0x7f328c004800 2024-03-06T07:25:31.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.691+0000 7f3297fff640 1 --2- 172.21.15.83:0/4177446207 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f328006b500 0x7f328006d9c0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f3288034000 tx=0x7f3288009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.695+0000 7f3296ffd640 1 -- 172.21.15.83:0/4177446207 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3290058db0 con 0x7f328c004800 2024-03-06T07:25:31.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.910+0000 7f32a4ea4640 1 -- 172.21.15.83:0/4177446207 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32a004f330 con 0x7f328c004800 2024-03-06T07:25:31.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.910+0000 7f3296ffd640 1 -- 172.21.15.83:0/4177446207 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f32900586b0 con 0x7f328c004800 2024-03-06T07:25:31.912 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:25:31.913 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":32,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6010,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/2142942385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2142942385},{"type":"v1","addr":"172.21.15.183:6834","nonce":2142942385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":6013,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3742438064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":3742438064},{"type":"v1","addr":"172.21.15.183:6835","nonce":3742438064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":6540,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/1549894851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1549894851},{"type":"v1","addr":"172.21.15.83:6836","nonce":1549894851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:12.481670+0000","modified":"2024-03-06T07:25:15.392072+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6543},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6543":{"gid":6543,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/602775648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":602775648},{"type":"v1","addr":"172.21.15.83:6837","nonce":602775648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6543,"qdb_cluster":[6543]},"id":5}]} 2024-03-06T07:25:31.913 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 32 2024-03-06T07:25:31.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.912+0000 7f32a4ea4640 1 -- 172.21.15.83:0/4177446207 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f328006b500 msgr2=0x7f328006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.912+0000 7f32a4ea4640 1 --2- 172.21.15.83:0/4177446207 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f328006b500 0x7f328006d9c0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f3288034000 tx=0x7f3288009030 comp rx=0 tx=0).stop 2024-03-06T07:25:31.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.912+0000 7f32a4ea4640 1 -- 172.21.15.83:0/4177446207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f328c004800 msgr2=0x7f32a00b97d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.912+0000 7f32a4ea4640 1 --2- 172.21.15.83:0/4177446207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f328c004800 0x7f32a00b97d0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f329000f620 tx=0x7f329000fae0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.912+0000 7f32a4ea4640 1 -- 172.21.15.83:0/4177446207 shutdown_connections 2024-03-06T07:25:31.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.912+0000 7f32a4ea4640 1 --2- 172.21.15.83:0/4177446207 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f328006b500 0x7f328006d9c0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.912+0000 7f32a4ea4640 1 --2- 172.21.15.83:0/4177446207 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f32a006cc10 0x7f32a006d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.912+0000 7f32a4ea4640 1 --2- 172.21.15.83:0/4177446207 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f32a015bc90 0x7f32a015b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.912+0000 7f32a4ea4640 1 --2- 172.21.15.83:0/4177446207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f328c004800 0x7f32a00b97d0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.912+0000 7f32a4ea4640 1 -- 172.21.15.83:0/4177446207 >> 172.21.15.83:0/4177446207 conn(0x7f32a00d3cd0 msgr2=0x7f32a00cedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:31.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.912+0000 7f32a4ea4640 1 -- 172.21.15.83:0/4177446207 shutdown_connections 2024-03-06T07:25:31.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:31.912+0000 7f32a4ea4640 1 -- 172.21.15.83:0/4177446207 wait complete. 2024-03-06T07:25:31.925 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:25:32.039 INFO:teuthology.orchestra.run.smithi083.stdout:{"mds_cache":{"strays_created":10}} 2024-03-06T07:25:32.039 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 10 } } 2024-03-06T07:25:32.040 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:32.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.140+0000 7f756762a640 1 Processor -- start 2024-03-06T07:25:32.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.140+0000 7f756762a640 1 -- start start 2024-03-06T07:25:32.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.140+0000 7f756762a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f75600d22b0 0x7f75600d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.141+0000 7f756762a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75600b3430 con 0x7f756015dfe0 2024-03-06T07:25:32.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.141+0000 7f756762a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75600b3710 con 0x7f75600b41f0 2024-03-06T07:25:32.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.141+0000 7f756762a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75600d2c50 con 0x7f75600d22b0 2024-03-06T07:25:32.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.141+0000 7f75653c6640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f756015dfe0 0x7f75600cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39126/0 (socket says 172.21.15.83:39126) 2024-03-06T07:25:32.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.141+0000 7f75653c6640 1 -- 172.21.15.83:0/1217114260 learned_addr learned my addr 172.21.15.83:0/1217114260 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:25:32.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.141+0000 7f7564bc5640 1 --2- 172.21.15.83:0/1217114260 >> v2:172.21.15.183:3300/0 conn(0x7f75600d22b0 0x7f75600d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.141+0000 7f7557fff640 1 -- 172.21.15.83:0/1217114260 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2479237619 0 0) 0x7f75600b3430 con 0x7f756015dfe0 2024-03-06T07:25:32.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.141+0000 7f7557fff640 1 -- 172.21.15.83:0/1217114260 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f754c003520 con 0x7f756015dfe0 2024-03-06T07:25:32.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.141+0000 7f7557fff640 1 -- 172.21.15.83:0/1217114260 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3377896935 0 0) 0x7f754c003520 con 0x7f756015dfe0 2024-03-06T07:25:32.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.141+0000 7f7557fff640 1 -- 172.21.15.83:0/1217114260 >> v2:172.21.15.183:3300/0 conn(0x7f75600d22b0 msgr2=0x7f75600d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:32.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.141+0000 7f7557fff640 1 --2- 172.21.15.83:0/1217114260 >> v2:172.21.15.183:3300/0 conn(0x7f75600d22b0 0x7f75600d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.141+0000 7f7557fff640 1 -- 172.21.15.83:0/1217114260 >> v1:172.21.15.183:6789/0 conn(0x7f75600b41f0 legacy=0x7f75600b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.141+0000 7f7557fff640 1 -- 172.21.15.83:0/1217114260 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75600b39f0 con 0x7f756015dfe0 2024-03-06T07:25:32.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.141+0000 7f7557fff640 1 -- 172.21.15.83:0/1217114260 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f755c001670 con 0x7f756015dfe0 2024-03-06T07:25:32.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.141+0000 7f7557fff640 1 -- 172.21.15.83:0/1217114260 >> v1:172.21.15.83:6789/0 conn(0x7f756015dfe0 legacy=0x7f75600cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.141+0000 7f7557fff640 1 --2- 172.21.15.83:0/1217114260 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f754c0039b0 0x7f754c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.141+0000 7f7557fff640 1 --2- 172.21.15.83:0/1217114260 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f754c0043b0 0x7f754c004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.141+0000 7f7557fff640 1 --2- 172.21.15.83:0/1217114260 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f754c005080 0x7f754c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.141+0000 7f7557fff640 1 -- 172.21.15.83:0/1217114260 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754c00bfb0 con 0x7f754c0043b0 2024-03-06T07:25:32.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.141+0000 7f7557fff640 1 -- 172.21.15.83:0/1217114260 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754c00c160 con 0x7f754c005080 2024-03-06T07:25:32.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.141+0000 7f7557fff640 1 -- 172.21.15.83:0/1217114260 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f754c00c360 con 0x7f754c0039b0 2024-03-06T07:25:32.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.141+0000 7f7564bc5640 1 --2- 172.21.15.83:0/1217114260 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f754c0043b0 0x7f754c004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.141+0000 7f75653c6640 1 --2- 172.21.15.83:0/1217114260 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f754c0039b0 0x7f754c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.142+0000 7f7565bc7640 1 --2- 172.21.15.83:0/1217114260 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f754c005080 0x7f754c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.142+0000 7f7564bc5640 1 -- 172.21.15.83:0/1217114260 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f754c0039b0 msgr2=0x7f754c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:32.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.142+0000 7f7564bc5640 1 --2- 172.21.15.83:0/1217114260 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f754c0039b0 0x7f754c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.142+0000 7f7564bc5640 1 -- 172.21.15.83:0/1217114260 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f754c005080 msgr2=0x7f754c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:32.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.142+0000 7f7564bc5640 1 --2- 172.21.15.83:0/1217114260 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f754c005080 0x7f754c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.142+0000 7f7564bc5640 1 -- 172.21.15.83:0/1217114260 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f754c003520 con 0x7f754c0043b0 2024-03-06T07:25:32.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.142+0000 7f7565bc7640 1 -- 172.21.15.83:0/1217114260 reap_dead start 2024-03-06T07:25:32.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.142+0000 7f7564bc5640 1 --2- 172.21.15.83:0/1217114260 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f754c0043b0 0x7f754c004880 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f7550009ac0 tx=0x7f754c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db71970a8f237a1e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.143+0000 7f7557fff640 1 -- 172.21.15.83:0/1217114260 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7550039070 con 0x7f754c0043b0 2024-03-06T07:25:32.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.143+0000 7f7557fff640 1 -- 172.21.15.83:0/1217114260 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f755002c6b0 con 0x7f754c0043b0 2024-03-06T07:25:32.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.143+0000 7f7557fff640 1 -- 172.21.15.83:0/1217114260 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7550009690 con 0x7f754c0043b0 2024-03-06T07:25:32.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.143+0000 7f756762a640 1 -- 172.21.15.83:0/1217114260 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f754c0043b0 msgr2=0x7f754c004880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.143+0000 7f756762a640 1 --2- 172.21.15.83:0/1217114260 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f754c0043b0 0x7f754c004880 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f7550009ac0 tx=0x7f754c005830 comp rx=0 tx=0).stop 2024-03-06T07:25:32.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.143+0000 7f756762a640 1 -- 172.21.15.83:0/1217114260 shutdown_connections 2024-03-06T07:25:32.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.143+0000 7f756762a640 1 --2- 172.21.15.83:0/1217114260 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f754c0043b0 0x7f754c004880 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.143+0000 7f756762a640 1 -- 172.21.15.83:0/1217114260 >> 172.21.15.83:0/1217114260 conn(0x7f75600d3cd0 msgr2=0x7f75600d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:32.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.143+0000 7f756762a640 1 -- 172.21.15.83:0/1217114260 shutdown_connections 2024-03-06T07:25:32.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.143+0000 7f756762a640 1 -- 172.21.15.83:0/1217114260 wait complete. 2024-03-06T07:25:32.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.144+0000 7f756762a640 1 Processor -- start 2024-03-06T07:25:32.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.144+0000 7f756762a640 1 -- start start 2024-03-06T07:25:32.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.145+0000 7f756762a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f754c0043b0 0x7f75601f3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.145+0000 7f756762a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f75600cada0 0x7f75600cb220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.145+0000 7f756762a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f75601f6fe0 0x7f75601f7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.145+0000 7f756762a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75600b3d60 con 0x7f75601f6fe0 2024-03-06T07:25:32.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.145+0000 7f756762a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75600b31e0 con 0x7f75600cada0 2024-03-06T07:25:32.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.145+0000 7f756762a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f756015e3c0 con 0x7f754c0043b0 2024-03-06T07:25:32.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.145+0000 7f7565bc7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f75601f6fe0 0x7f75601f7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.145+0000 7f7565bc7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f75601f6fe0 0x7f75601f7470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:39920/0 (socket says 172.21.15.83:39920) 2024-03-06T07:25:32.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.145+0000 7f7565bc7640 1 -- 172.21.15.83:0/41696024 learned_addr learned my addr 172.21.15.83:0/41696024 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:32.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.145+0000 7f7565bc7640 1 -- 172.21.15.83:0/41696024 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f754c0043b0 msgr2=0x7f75601f3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.145+0000 7f7565bc7640 1 --2- 172.21.15.83:0/41696024 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f754c0043b0 0x7f75601f3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.145+0000 7f7565bc7640 1 -- 172.21.15.83:0/41696024 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f75600cada0 msgr2=0x7f75600cb220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.145+0000 7f7565bc7640 1 --2- 172.21.15.83:0/41696024 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f75600cada0 0x7f75600cb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.145+0000 7f7565bc7640 1 -- 172.21.15.83:0/41696024 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7560143990 con 0x7f75601f6fe0 2024-03-06T07:25:32.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.145+0000 7f7565bc7640 1 --2- 172.21.15.83:0/41696024 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f75601f6fe0 0x7f75601f7470 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f7558007fd0 tx=0x7f755800eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.146+0000 7f7555ffb640 1 -- 172.21.15.83:0/41696024 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7558003970 con 0x7f75601f6fe0 2024-03-06T07:25:32.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.146+0000 7f7555ffb640 1 -- 172.21.15.83:0/41696024 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7558018070 con 0x7f75601f6fe0 2024-03-06T07:25:32.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.146+0000 7f756762a640 1 -- 172.21.15.83:0/41696024 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7560143c70 con 0x7f75601f6fe0 2024-03-06T07:25:32.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.146+0000 7f7555ffb640 1 -- 172.21.15.83:0/41696024 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7558003d50 con 0x7f75601f6fe0 2024-03-06T07:25:32.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.146+0000 7f756762a640 1 -- 172.21.15.83:0/41696024 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7560144230 con 0x7f75601f6fe0 2024-03-06T07:25:32.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.147+0000 7f756762a640 1 -- 172.21.15.83:0/41696024 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f756004f330 con 0x7f75601f6fe0 2024-03-06T07:25:32.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.147+0000 7f7555ffb640 1 -- 172.21.15.83:0/41696024 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f75580050d0 con 0x7f75601f6fe0 2024-03-06T07:25:32.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.147+0000 7f7555ffb640 1 --2- 172.21.15.83:0/41696024 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f754406b500 0x7f754406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.148+0000 7f75653c6640 1 --2- 172.21.15.83:0/41696024 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f754406b500 0x7f754406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.148+0000 7f7555ffb640 1 -- 172.21.15.83:0/41696024 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7558089ff0 con 0x7f75601f6fe0 2024-03-06T07:25:32.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.148+0000 7f75653c6640 1 --2- 172.21.15.83:0/41696024 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f754406b500 0x7f754406d9c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f755c001ba0 tx=0x7f755c001b10 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.156+0000 7f7555ffb640 1 -- 172.21.15.83:0/41696024 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7558058e70 con 0x7f75601f6fe0 2024-03-06T07:25:32.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.408+0000 7f756762a640 1 -- 172.21.15.83:0/41696024 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75600d07a0 con 0x7f75601f6fe0 2024-03-06T07:25:32.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.408+0000 7f7555ffb640 1 -- 172.21.15.83:0/41696024 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f7558058770 con 0x7f75601f6fe0 2024-03-06T07:25:32.410 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:25:32.410 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":32,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6010,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/2142942385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2142942385},{"type":"v1","addr":"172.21.15.183:6834","nonce":2142942385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":6013,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3742438064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":3742438064},{"type":"v1","addr":"172.21.15.183:6835","nonce":3742438064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":6540,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/1549894851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1549894851},{"type":"v1","addr":"172.21.15.83:6836","nonce":1549894851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:12.481670+0000","modified":"2024-03-06T07:25:15.392072+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6543},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6543":{"gid":6543,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/602775648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":602775648},{"type":"v1","addr":"172.21.15.83:6837","nonce":602775648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6543,"qdb_cluster":[6543]},"id":5}]} 2024-03-06T07:25:32.411 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 32 2024-03-06T07:25:32.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.411+0000 7f756762a640 1 -- 172.21.15.83:0/41696024 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f754406b500 msgr2=0x7f754406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.411+0000 7f756762a640 1 --2- 172.21.15.83:0/41696024 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f754406b500 0x7f754406d9c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f755c001ba0 tx=0x7f755c001b10 comp rx=0 tx=0).stop 2024-03-06T07:25:32.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.411+0000 7f756762a640 1 -- 172.21.15.83:0/41696024 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f75601f6fe0 msgr2=0x7f75601f7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.411+0000 7f756762a640 1 --2- 172.21.15.83:0/41696024 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f75601f6fe0 0x7f75601f7470 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f7558007fd0 tx=0x7f755800eb90 comp rx=0 tx=0).stop 2024-03-06T07:25:32.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.411+0000 7f756762a640 1 -- 172.21.15.83:0/41696024 shutdown_connections 2024-03-06T07:25:32.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.411+0000 7f756762a640 1 --2- 172.21.15.83:0/41696024 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f754406b500 0x7f754406d9c0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.411+0000 7f756762a640 1 --2- 172.21.15.83:0/41696024 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f75601f6fe0 0x7f75601f7470 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.411+0000 7f756762a640 1 --2- 172.21.15.83:0/41696024 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f75600cada0 0x7f75600cb220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.411+0000 7f756762a640 1 --2- 172.21.15.83:0/41696024 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f754c0043b0 0x7f75601f3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.411+0000 7f756762a640 1 -- 172.21.15.83:0/41696024 >> 172.21.15.83:0/41696024 conn(0x7f75600d3cd0 msgr2=0x7f75600ca4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:32.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.411+0000 7f756762a640 1 -- 172.21.15.83:0/41696024 shutdown_connections 2024-03-06T07:25:32.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:32.411+0000 7f756762a640 1 -- 172.21.15.83:0/41696024 wait complete. 2024-03-06T07:25:32.424 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:25:32.539 INFO:teuthology.orchestra.run.smithi083.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-06T07:25:32.540 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-06T07:25:32.540 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 10, waiting (timeout=60)... 2024-03-06T07:25:37.541 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:37.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.644+0000 7f76efc5c640 1 Processor -- start 2024-03-06T07:25:37.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.644+0000 7f76efc5c640 1 -- start start 2024-03-06T07:25:37.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.645+0000 7f76efc5c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f76e815dfe0 0x7f76e80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.645+0000 7f76efc5c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76e80b3430 con 0x7f76e80cf830 2024-03-06T07:25:37.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.645+0000 7f76efc5c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76e80b3710 con 0x7f76e80d29c0 2024-03-06T07:25:37.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.645+0000 7f76efc5c640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76e80b31e0 con 0x7f76e815dfe0 2024-03-06T07:25:37.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.645+0000 7f76ed9f8640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f76e815dfe0 0x7f76e80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.645+0000 7f76ed1f7640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f76e80cf830 0x7f76e80d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52484/0 (socket says 172.21.15.83:52484) 2024-03-06T07:25:37.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.645+0000 7f76ed1f7640 1 -- 172.21.15.83:0/1547272947 learned_addr learned my addr 172.21.15.83:0/1547272947 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:25:37.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.646+0000 7f76ed9f8640 1 -- 172.21.15.83:0/1547272947 >> v1:172.21.15.183:6789/0 conn(0x7f76e80d29c0 legacy=0x7f76e80b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:37.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.646+0000 7f76ed9f8640 1 -- 172.21.15.83:0/1547272947 >> v1:172.21.15.83:6789/0 conn(0x7f76e80cf830 legacy=0x7f76e80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.646+0000 7f76ec9f6640 1 -- 172.21.15.83:0/1547272947 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3463672020 0 0) 0x7f76e80b3430 con 0x7f76e80cf830 2024-03-06T07:25:37.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.646+0000 7f76ed9f8640 1 -- 172.21.15.83:0/1547272947 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e80b39f0 con 0x7f76e815dfe0 2024-03-06T07:25:37.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.646+0000 7f76ed9f8640 1 --2- 172.21.15.83:0/1547272947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f76e815dfe0 0x7f76e80cf2f0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f76e80b3360 tx=0x7f76e402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa269e90925199ff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:37.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.647+0000 7f76ec9f6640 1 -- 172.21.15.83:0/1547272947 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e4035020 con 0x7f76e815dfe0 2024-03-06T07:25:37.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.647+0000 7f76ec9f6640 1 -- 172.21.15.83:0/1547272947 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76e4031830 con 0x7f76e815dfe0 2024-03-06T07:25:37.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.647+0000 7f76ec9f6640 1 -- 172.21.15.83:0/1547272947 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e4031c00 con 0x7f76e815dfe0 2024-03-06T07:25:37.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.647+0000 7f76efc5c640 1 -- 172.21.15.83:0/1547272947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f76e815dfe0 msgr2=0x7f76e80cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.647+0000 7f76efc5c640 1 --2- 172.21.15.83:0/1547272947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f76e815dfe0 0x7f76e80cf2f0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f76e80b3360 tx=0x7f76e402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.647+0000 7f76efc5c640 1 -- 172.21.15.83:0/1547272947 shutdown_connections 2024-03-06T07:25:37.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.647+0000 7f76efc5c640 1 --2- 172.21.15.83:0/1547272947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f76e815dfe0 0x7f76e80cf2f0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.647+0000 7f76efc5c640 1 -- 172.21.15.83:0/1547272947 >> 172.21.15.83:0/1547272947 conn(0x7f76e80d3cd0 msgr2=0x7f76e80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:37.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.647+0000 7f76efc5c640 1 -- 172.21.15.83:0/1547272947 shutdown_connections 2024-03-06T07:25:37.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.647+0000 7f76efc5c640 1 -- 172.21.15.83:0/1547272947 wait complete. 2024-03-06T07:25:37.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.648+0000 7f76efc5c640 1 Processor -- start 2024-03-06T07:25:37.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.649+0000 7f76efc5c640 1 -- start start 2024-03-06T07:25:37.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.649+0000 7f76efc5c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f76e80cf830 0x7f76e8150230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.649+0000 7f76efc5c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f76e80d29c0 0x7f76e809fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.649+0000 7f76efc5c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f76e815dfe0 0x7f76e80a0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.649+0000 7f76efc5c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76e815e3c0 con 0x7f76e815dfe0 2024-03-06T07:25:37.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.649+0000 7f76efc5c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76e80b3d60 con 0x7f76e80d29c0 2024-03-06T07:25:37.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.649+0000 7f76efc5c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76e815e540 con 0x7f76e80cf830 2024-03-06T07:25:37.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.649+0000 7f76ee1f9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f76e815dfe0 0x7f76e80a0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.649+0000 7f76ee1f9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f76e815dfe0 0x7f76e80a0010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52544/0 (socket says 172.21.15.83:52544) 2024-03-06T07:25:37.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.649+0000 7f76ee1f9640 1 -- 172.21.15.83:0/642467625 learned_addr learned my addr 172.21.15.83:0/642467625 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:37.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.650+0000 7f76ed1f7640 1 --2- 172.21.15.83:0/642467625 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f76e80d29c0 0x7f76e809fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.650+0000 7f76ed9f8640 1 --2- 172.21.15.83:0/642467625 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f76e80cf830 0x7f76e8150230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.650+0000 7f76ee1f9640 1 -- 172.21.15.83:0/642467625 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f76e80cf830 msgr2=0x7f76e8150230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.650+0000 7f76ee1f9640 1 --2- 172.21.15.83:0/642467625 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f76e80cf830 0x7f76e8150230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.650+0000 7f76ee1f9640 1 -- 172.21.15.83:0/642467625 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f76e80d29c0 msgr2=0x7f76e809fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.650+0000 7f76ee1f9640 1 --2- 172.21.15.83:0/642467625 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f76e80d29c0 0x7f76e809fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.650+0000 7f76ee1f9640 1 -- 172.21.15.83:0/642467625 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e80a0850 con 0x7f76e815dfe0 2024-03-06T07:25:37.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.650+0000 7f76ed9f8640 1 --2- 172.21.15.83:0/642467625 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f76e80cf830 0x7f76e8150230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:37.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.650+0000 7f76ee1f9640 1 --2- 172.21.15.83:0/642467625 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f76e815dfe0 0x7f76e80a0010 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f76e000c700 tx=0x7f76e000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:37.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.650+0000 7f76de7fc640 1 -- 172.21.15.83:0/642467625 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e0016020 con 0x7f76e815dfe0 2024-03-06T07:25:37.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.650+0000 7f76de7fc640 1 -- 172.21.15.83:0/642467625 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76e000aa50 con 0x7f76e815dfe0 2024-03-06T07:25:37.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.650+0000 7f76de7fc640 1 -- 172.21.15.83:0/642467625 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e000ae40 con 0x7f76e815dfe0 2024-03-06T07:25:37.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.650+0000 7f76efc5c640 1 -- 172.21.15.83:0/642467625 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76e80a0b30 con 0x7f76e815dfe0 2024-03-06T07:25:37.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.650+0000 7f76efc5c640 1 -- 172.21.15.83:0/642467625 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76e80a10f0 con 0x7f76e815dfe0 2024-03-06T07:25:37.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.651+0000 7f76efc5c640 1 -- 172.21.15.83:0/642467625 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76e804f330 con 0x7f76e815dfe0 2024-03-06T07:25:37.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.652+0000 7f76de7fc640 1 -- 172.21.15.83:0/642467625 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f76e0019020 con 0x7f76e815dfe0 2024-03-06T07:25:37.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.652+0000 7f76de7fc640 1 --2- 172.21.15.83:0/642467625 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f76cc06b4d0 0x7f76cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.652+0000 7f76de7fc640 1 -- 172.21.15.83:0/642467625 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f76e008a3f0 con 0x7f76e815dfe0 2024-03-06T07:25:37.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.652+0000 7f76ed9f8640 1 --2- 172.21.15.83:0/642467625 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f76cc06b4d0 0x7f76cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.652+0000 7f76ed9f8640 1 --2- 172.21.15.83:0/642467625 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f76cc06b4d0 0x7f76cc06d990 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f76e4036000 tx=0x7f76e4008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:37.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.660+0000 7f76de7fc640 1 -- 172.21.15.83:0/642467625 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f76e0020020 con 0x7f76e815dfe0 2024-03-06T07:25:37.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.874+0000 7f76efc5c640 1 -- 172.21.15.83:0/642467625 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76e80b4e50 con 0x7f76e815dfe0 2024-03-06T07:25:37.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.875+0000 7f76de7fc640 1 -- 172.21.15.83:0/642467625 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f76e0059300 con 0x7f76e815dfe0 2024-03-06T07:25:37.877 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:25:37.877 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":32,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6010,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/2142942385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2142942385},{"type":"v1","addr":"172.21.15.183:6834","nonce":2142942385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":6013,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3742438064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":3742438064},{"type":"v1","addr":"172.21.15.183:6835","nonce":3742438064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":6540,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/1549894851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1549894851},{"type":"v1","addr":"172.21.15.83:6836","nonce":1549894851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:12.481670+0000","modified":"2024-03-06T07:25:15.392072+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6543},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6543":{"gid":6543,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/602775648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":602775648},{"type":"v1","addr":"172.21.15.83:6837","nonce":602775648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6543,"qdb_cluster":[6543]},"id":5}]} 2024-03-06T07:25:37.878 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 32 2024-03-06T07:25:37.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.876+0000 7f76efc5c640 1 -- 172.21.15.83:0/642467625 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f76cc06b4d0 msgr2=0x7f76cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.876+0000 7f76efc5c640 1 --2- 172.21.15.83:0/642467625 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f76cc06b4d0 0x7f76cc06d990 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f76e4036000 tx=0x7f76e4008030 comp rx=0 tx=0).stop 2024-03-06T07:25:37.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.876+0000 7f76efc5c640 1 -- 172.21.15.83:0/642467625 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f76e815dfe0 msgr2=0x7f76e80a0010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.876+0000 7f76efc5c640 1 --2- 172.21.15.83:0/642467625 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f76e815dfe0 0x7f76e80a0010 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f76e000c700 tx=0x7f76e000cbc0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.876+0000 7f76efc5c640 1 -- 172.21.15.83:0/642467625 shutdown_connections 2024-03-06T07:25:37.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.876+0000 7f76efc5c640 1 --2- 172.21.15.83:0/642467625 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f76cc06b4d0 0x7f76cc06d990 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.876+0000 7f76efc5c640 1 --2- 172.21.15.83:0/642467625 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f76e815dfe0 0x7f76e80a0010 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.876+0000 7f76efc5c640 1 --2- 172.21.15.83:0/642467625 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f76e80d29c0 0x7f76e809fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.876+0000 7f76efc5c640 1 --2- 172.21.15.83:0/642467625 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f76e80cf830 0x7f76e8150230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.876+0000 7f76efc5c640 1 -- 172.21.15.83:0/642467625 >> 172.21.15.83:0/642467625 conn(0x7f76e80d3cd0 msgr2=0x7f76e806cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:37.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.877+0000 7f76efc5c640 1 -- 172.21.15.83:0/642467625 shutdown_connections 2024-03-06T07:25:37.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:37.877+0000 7f76efc5c640 1 -- 172.21.15.83:0/642467625 wait complete. 2024-03-06T07:25:37.889 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:25:38.005 INFO:teuthology.orchestra.run.smithi083.stdout:{"mds_cache":{"strays_enqueued":10}} 2024-03-06T07:25:38.006 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 10 } } 2024-03-06T07:25:38.006 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:38.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.106+0000 7f33ceb7a640 1 Processor -- start 2024-03-06T07:25:38.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.106+0000 7f33ceb7a640 1 -- start start 2024-03-06T07:25:38.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.107+0000 7f33ceb7a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f33c80d22b0 0x7f33c80d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.107+0000 7f33ceb7a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33c80b3430 con 0x7f33c80b41f0 2024-03-06T07:25:38.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.107+0000 7f33ceb7a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33c80b3710 con 0x7f33c815dfe0 2024-03-06T07:25:38.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.107+0000 7f33ceb7a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33c80d2c50 con 0x7f33c80d22b0 2024-03-06T07:25:38.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.107+0000 7f33cd117640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f33c80b41f0 0x7f33c80b4680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52492/0 (socket says 172.21.15.83:52492) 2024-03-06T07:25:38.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.107+0000 7f33cd117640 1 -- 172.21.15.83:0/2924016270 learned_addr learned my addr 172.21.15.83:0/2924016270 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:25:38.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.107+0000 7f33c7fff640 1 --2- 172.21.15.83:0/2924016270 >> v2:172.21.15.183:3300/0 conn(0x7f33c80d22b0 0x7f33c80d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.107+0000 7f33c77fe640 1 -- 172.21.15.83:0/2924016270 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 633412127 0 0) 0x7f33c80b3430 con 0x7f33c80b41f0 2024-03-06T07:25:38.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.107+0000 7f33c77fe640 1 -- 172.21.15.83:0/2924016270 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33b8003520 con 0x7f33c80b41f0 2024-03-06T07:25:38.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.107+0000 7f33c77fe640 1 -- 172.21.15.83:0/2924016270 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1938074360 0 0) 0x7f33b8003520 con 0x7f33c80b41f0 2024-03-06T07:25:38.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.107+0000 7f33c77fe640 1 -- 172.21.15.83:0/2924016270 >> v2:172.21.15.183:3300/0 conn(0x7f33c80d22b0 msgr2=0x7f33c80d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:38.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.107+0000 7f33c77fe640 1 --2- 172.21.15.83:0/2924016270 >> v2:172.21.15.183:3300/0 conn(0x7f33c80d22b0 0x7f33c80d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.107+0000 7f33c77fe640 1 -- 172.21.15.83:0/2924016270 >> v1:172.21.15.183:6789/0 conn(0x7f33c815dfe0 legacy=0x7f33c80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.107+0000 7f33c77fe640 1 -- 172.21.15.83:0/2924016270 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c80b39f0 con 0x7f33c80b41f0 2024-03-06T07:25:38.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.107+0000 7f33c77fe640 1 -- 172.21.15.83:0/2924016270 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f33c00019f0 con 0x7f33c80b41f0 2024-03-06T07:25:38.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.108+0000 7f33c77fe640 1 -- 172.21.15.83:0/2924016270 >> v1:172.21.15.83:6789/0 conn(0x7f33c80b41f0 legacy=0x7f33c80b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.108+0000 7f33c77fe640 1 --2- 172.21.15.83:0/2924016270 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f33b80039b0 0x7f33b8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.108+0000 7f33c77fe640 1 --2- 172.21.15.83:0/2924016270 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33b8004800 0x7f33b8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.108+0000 7f33c77fe640 1 --2- 172.21.15.83:0/2924016270 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f33b80051c0 0x7f33b800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.108+0000 7f33c77fe640 1 -- 172.21.15.83:0/2924016270 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b800c030 con 0x7f33b8004800 2024-03-06T07:25:38.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.108+0000 7f33c77fe640 1 -- 172.21.15.83:0/2924016270 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b800c1e0 con 0x7f33b80039b0 2024-03-06T07:25:38.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.108+0000 7f33c77fe640 1 -- 172.21.15.83:0/2924016270 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33b800c3e0 con 0x7f33b80051c0 2024-03-06T07:25:38.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.108+0000 7f33c7fff640 1 --2- 172.21.15.83:0/2924016270 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33b8004800 0x7f33b8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.108+0000 7f33cc916640 1 --2- 172.21.15.83:0/2924016270 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f33b80039b0 0x7f33b8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.108+0000 7f33cd117640 1 --2- 172.21.15.83:0/2924016270 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f33b80051c0 0x7f33b800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.108+0000 7f33c7fff640 1 -- 172.21.15.83:0/2924016270 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f33b80051c0 msgr2=0x7f33b800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:38.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.108+0000 7f33c7fff640 1 --2- 172.21.15.83:0/2924016270 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f33b80051c0 0x7f33b800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.108+0000 7f33c7fff640 1 -- 172.21.15.83:0/2924016270 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f33b80039b0 msgr2=0x7f33b8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:38.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.108+0000 7f33c7fff640 1 --2- 172.21.15.83:0/2924016270 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f33b80039b0 0x7f33b8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.108+0000 7f33c7fff640 1 -- 172.21.15.83:0/2924016270 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33b8003520 con 0x7f33b8004800 2024-03-06T07:25:38.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.108+0000 7f33cd117640 1 -- 172.21.15.83:0/2924016270 reap_dead start 2024-03-06T07:25:38.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.109+0000 7f33c7fff640 1 --2- 172.21.15.83:0/2924016270 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33b8004800 0x7f33b8004c80 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f33b4009ac0 tx=0x7f33b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8a15e7651ff70dd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.109+0000 7f33c77fe640 1 -- 172.21.15.83:0/2924016270 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b4039070 con 0x7f33b8004800 2024-03-06T07:25:38.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.109+0000 7f33c77fe640 1 -- 172.21.15.83:0/2924016270 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33b402c6b0 con 0x7f33b8004800 2024-03-06T07:25:38.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.109+0000 7f33c77fe640 1 -- 172.21.15.83:0/2924016270 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b4009690 con 0x7f33b8004800 2024-03-06T07:25:38.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.109+0000 7f33ceb7a640 1 -- 172.21.15.83:0/2924016270 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33b8004800 msgr2=0x7f33b8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.109+0000 7f33ceb7a640 1 --2- 172.21.15.83:0/2924016270 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33b8004800 0x7f33b8004c80 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f33b4009ac0 tx=0x7f33b8005830 comp rx=0 tx=0).stop 2024-03-06T07:25:38.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.109+0000 7f33ceb7a640 1 -- 172.21.15.83:0/2924016270 shutdown_connections 2024-03-06T07:25:38.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.109+0000 7f33ceb7a640 1 --2- 172.21.15.83:0/2924016270 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33b8004800 0x7f33b8004c80 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.109+0000 7f33ceb7a640 1 -- 172.21.15.83:0/2924016270 >> 172.21.15.83:0/2924016270 conn(0x7f33c80d3cd0 msgr2=0x7f33c80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:38.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.109+0000 7f33ceb7a640 1 -- 172.21.15.83:0/2924016270 shutdown_connections 2024-03-06T07:25:38.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.109+0000 7f33ceb7a640 1 -- 172.21.15.83:0/2924016270 wait complete. 2024-03-06T07:25:38.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.111+0000 7f33ceb7a640 1 Processor -- start 2024-03-06T07:25:38.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.111+0000 7f33ceb7a640 1 -- start start 2024-03-06T07:25:38.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.111+0000 7f33ceb7a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f33b8004800 0x7f33c815bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.111+0000 7f33ceb7a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33c81f8040 0x7f33c81601c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.111+0000 7f33ceb7a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f33c8160700 0x7f33c80b85b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.111+0000 7f33ceb7a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c80b3d60 con 0x7f33c81f8040 2024-03-06T07:25:38.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.111+0000 7f33ceb7a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c80b31e0 con 0x7f33b8004800 2024-03-06T07:25:38.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.111+0000 7f33ceb7a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33c815e3c0 con 0x7f33c8160700 2024-03-06T07:25:38.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.111+0000 7f33c7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33c81f8040 0x7f33c81601c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.112+0000 7f33c7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33c81f8040 0x7f33c81601c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52552/0 (socket says 172.21.15.83:52552) 2024-03-06T07:25:38.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.112+0000 7f33c7fff640 1 -- 172.21.15.83:0/115958964 learned_addr learned my addr 172.21.15.83:0/115958964 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:38.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.112+0000 7f33cc916640 1 --2- 172.21.15.83:0/115958964 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f33b8004800 0x7f33c815bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.112+0000 7f33cd117640 1 --2- 172.21.15.83:0/115958964 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f33c8160700 0x7f33c80b85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.112+0000 7f33c7fff640 1 -- 172.21.15.83:0/115958964 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f33c8160700 msgr2=0x7f33c80b85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.112+0000 7f33c7fff640 1 --2- 172.21.15.83:0/115958964 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f33c8160700 0x7f33c80b85b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.112+0000 7f33c7fff640 1 -- 172.21.15.83:0/115958964 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f33b8004800 msgr2=0x7f33c815bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.112+0000 7f33c7fff640 1 --2- 172.21.15.83:0/115958964 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f33b8004800 0x7f33c815bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.112+0000 7f33c7fff640 1 -- 172.21.15.83:0/115958964 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c81f3ec0 con 0x7f33c81f8040 2024-03-06T07:25:38.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.112+0000 7f33cc916640 1 --2- 172.21.15.83:0/115958964 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f33b8004800 0x7f33c815bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:38.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.112+0000 7f33cd117640 1 --2- 172.21.15.83:0/115958964 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f33c8160700 0x7f33c80b85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:38.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.112+0000 7f33c7fff640 1 --2- 172.21.15.83:0/115958964 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33c81f8040 0x7f33c81601c0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f33b4000c00 tx=0x7f33b4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.112+0000 7f33c57fa640 1 -- 172.21.15.83:0/115958964 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b4042020 con 0x7f33c81f8040 2024-03-06T07:25:38.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.112+0000 7f33c57fa640 1 -- 172.21.15.83:0/115958964 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33b403a040 con 0x7f33c81f8040 2024-03-06T07:25:38.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.112+0000 7f33ceb7a640 1 -- 172.21.15.83:0/115958964 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33c81f41a0 con 0x7f33c81f8040 2024-03-06T07:25:38.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.112+0000 7f33c57fa640 1 -- 172.21.15.83:0/115958964 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b4039070 con 0x7f33c81f8040 2024-03-06T07:25:38.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.112+0000 7f33ceb7a640 1 -- 172.21.15.83:0/115958964 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33c81f4760 con 0x7f33c81f8040 2024-03-06T07:25:38.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.113+0000 7f33ceb7a640 1 -- 172.21.15.83:0/115958964 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33c804f330 con 0x7f33c81f8040 2024-03-06T07:25:38.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.114+0000 7f33c57fa640 1 -- 172.21.15.83:0/115958964 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f33b4045020 con 0x7f33c81f8040 2024-03-06T07:25:38.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.114+0000 7f33c57fa640 1 --2- 172.21.15.83:0/115958964 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f33ac06b4d0 0x7f33ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.114+0000 7f33cc916640 1 --2- 172.21.15.83:0/115958964 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f33ac06b4d0 0x7f33ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.114+0000 7f33c57fa640 1 -- 172.21.15.83:0/115958964 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f33b40ab630 con 0x7f33c81f8040 2024-03-06T07:25:38.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.115+0000 7f33cc916640 1 --2- 172.21.15.83:0/115958964 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f33ac06b4d0 0x7f33ac06d990 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f33bc005600 tx=0x7f33bc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.118+0000 7f33c57fa640 1 -- 172.21.15.83:0/115958964 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f33b404c070 con 0x7f33c81f8040 2024-03-06T07:25:38.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.333+0000 7f33ceb7a640 1 -- 172.21.15.83:0/115958964 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f33c815cb30 con 0x7f33c81f8040 2024-03-06T07:25:38.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.333+0000 7f33c57fa640 1 -- 172.21.15.83:0/115958964 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f33b407ee20 con 0x7f33c81f8040 2024-03-06T07:25:38.335 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:25:38.335 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":32,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6010,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/2142942385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2142942385},{"type":"v1","addr":"172.21.15.183:6834","nonce":2142942385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":6013,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3742438064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":3742438064},{"type":"v1","addr":"172.21.15.183:6835","nonce":3742438064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":6540,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/1549894851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1549894851},{"type":"v1","addr":"172.21.15.83:6836","nonce":1549894851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:12.481670+0000","modified":"2024-03-06T07:25:15.392072+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6543},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6543":{"gid":6543,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/602775648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":602775648},{"type":"v1","addr":"172.21.15.83:6837","nonce":602775648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6543,"qdb_cluster":[6543]},"id":5}]} 2024-03-06T07:25:38.336 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 32 2024-03-06T07:25:38.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.335+0000 7f33ceb7a640 1 -- 172.21.15.83:0/115958964 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f33ac06b4d0 msgr2=0x7f33ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.335+0000 7f33ceb7a640 1 --2- 172.21.15.83:0/115958964 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f33ac06b4d0 0x7f33ac06d990 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f33bc005600 tx=0x7f33bc009040 comp rx=0 tx=0).stop 2024-03-06T07:25:38.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.335+0000 7f33ceb7a640 1 -- 172.21.15.83:0/115958964 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33c81f8040 msgr2=0x7f33c81601c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.335+0000 7f33ceb7a640 1 --2- 172.21.15.83:0/115958964 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33c81f8040 0x7f33c81601c0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f33b4000c00 tx=0x7f33b4004730 comp rx=0 tx=0).stop 2024-03-06T07:25:38.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.335+0000 7f33ceb7a640 1 -- 172.21.15.83:0/115958964 shutdown_connections 2024-03-06T07:25:38.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.335+0000 7f33ceb7a640 1 --2- 172.21.15.83:0/115958964 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f33ac06b4d0 0x7f33ac06d990 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.335+0000 7f33ceb7a640 1 --2- 172.21.15.83:0/115958964 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f33c8160700 0x7f33c80b85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.335+0000 7f33ceb7a640 1 --2- 172.21.15.83:0/115958964 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33c81f8040 0x7f33c81601c0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.335+0000 7f33ceb7a640 1 --2- 172.21.15.83:0/115958964 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f33b8004800 0x7f33c815bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.335+0000 7f33ceb7a640 1 -- 172.21.15.83:0/115958964 >> 172.21.15.83:0/115958964 conn(0x7f33c80d3cd0 msgr2=0x7f33c80ceed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:38.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.335+0000 7f33ceb7a640 1 -- 172.21.15.83:0/115958964 shutdown_connections 2024-03-06T07:25:38.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.335+0000 7f33ceb7a640 1 -- 172.21.15.83:0/115958964 wait complete. 2024-03-06T07:25:38.351 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump purge_queue pq_executed 2024-03-06T07:25:38.467 INFO:teuthology.orchestra.run.smithi083.stdout:{"purge_queue":{"pq_executed":10}} 2024-03-06T07:25:38.467 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 10 } } 2024-03-06T07:25:38.468 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:25:38.468 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:38.483 DEBUG:teuthology.orchestra.run.smithi083:> stdin-killer -- python3 -c ' 2024-03-06T07:25:38.483 DEBUG:teuthology.orchestra.run.smithi083:> import os 2024-03-06T07:25:38.483 DEBUG:teuthology.orchestra.run.smithi083:> import uuid 2024-03-06T07:25:38.484 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:25:38.484 DEBUG:teuthology.orchestra.run.smithi083:> n = 10 2024-03-06T07:25:38.484 DEBUG:teuthology.orchestra.run.smithi083:> create_hard_links = False 2024-03-06T07:25:38.484 DEBUG:teuthology.orchestra.run.smithi083:> if 0 > 0: 2024-03-06T07:25:38.484 DEBUG:teuthology.orchestra.run.smithi083:> create_hard_links = True 2024-03-06T07:25:38.484 DEBUG:teuthology.orchestra.run.smithi083:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:25:38.484 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:25:38.484 DEBUG:teuthology.orchestra.run.smithi083:> dpath = os.path.dirname(path) 2024-03-06T07:25:38.484 DEBUG:teuthology.orchestra.run.smithi083:> fnameprefix = os.path.basename(path) 2024-03-06T07:25:38.484 DEBUG:teuthology.orchestra.run.smithi083:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:25:38.484 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:25:38.484 DEBUG:teuthology.orchestra.run.smithi083:> try: 2024-03-06T07:25:38.484 DEBUG:teuthology.orchestra.run.smithi083:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:25:38.484 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:25:38.485 DEBUG:teuthology.orchestra.run.smithi083:> for i in range(n): 2024-03-06T07:25:38.485 DEBUG:teuthology.orchestra.run.smithi083:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:25:38.485 DEBUG:teuthology.orchestra.run.smithi083:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:25:38.485 DEBUG:teuthology.orchestra.run.smithi083:> f.write(f"{i}") 2024-03-06T07:25:38.485 DEBUG:teuthology.orchestra.run.smithi083:> if False: 2024-03-06T07:25:38.485 DEBUG:teuthology.orchestra.run.smithi083:> f.flush() 2024-03-06T07:25:38.485 DEBUG:teuthology.orchestra.run.smithi083:> os.fsync(f.fileno()) 2024-03-06T07:25:38.485 DEBUG:teuthology.orchestra.run.smithi083:> if True: 2024-03-06T07:25:38.485 DEBUG:teuthology.orchestra.run.smithi083:> os.unlink(fpath) 2024-03-06T07:25:38.485 DEBUG:teuthology.orchestra.run.smithi083:> if True: 2024-03-06T07:25:38.485 DEBUG:teuthology.orchestra.run.smithi083:> os.fsync(dirfd) 2024-03-06T07:25:38.485 DEBUG:teuthology.orchestra.run.smithi083:> if create_hard_links: 2024-03-06T07:25:38.485 DEBUG:teuthology.orchestra.run.smithi083:> for j in range(0): 2024-03-06T07:25:38.485 DEBUG:teuthology.orchestra.run.smithi083:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:25:38.485 DEBUG:teuthology.orchestra.run.smithi083:> if False: 2024-03-06T07:25:38.485 DEBUG:teuthology.orchestra.run.smithi083:> os.fsync(dirfd) 2024-03-06T07:25:38.486 DEBUG:teuthology.orchestra.run.smithi083:> finally: 2024-03-06T07:25:38.486 DEBUG:teuthology.orchestra.run.smithi083:> os.close(dirfd) 2024-03-06T07:25:38.486 DEBUG:teuthology.orchestra.run.smithi083:> ' 2024-03-06T07:25:38.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:25:38.659 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:25:38.660 DEBUG:teuthology.orchestra.run.smithi083:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:38.747 DEBUG:teuthology.orchestra.run.smithi183:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:38.797 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:25:38.797 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:25:38.797 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:38.814 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:38.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.953+0000 7f983f5b5640 1 Processor -- start 2024-03-06T07:25:38.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.953+0000 7f983f5b5640 1 -- start start 2024-03-06T07:25:38.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.954+0000 7f983f5b5640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f9838071ce0 0x7f9838072140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.954+0000 7f983f5b5640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f983806d400 0x7f983806d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.954+0000 7f983f5b5640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98380adf70 con 0x7f98380cf790 2024-03-06T07:25:38.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.954+0000 7f983f5b5640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9838072680 con 0x7f983806d400 2024-03-06T07:25:38.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.954+0000 7f983f5b5640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f983806cf70 con 0x7f9838071ce0 2024-03-06T07:25:38.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.954+0000 7f983db52640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f983806d400 0x7f983806d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.954+0000 7f983d351640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f98380cf790 0x7f98380ad860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52498/0 (socket says 172.21.15.83:52498) 2024-03-06T07:25:38.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.954+0000 7f983d351640 1 -- 172.21.15.83:0/1171491568 learned_addr learned my addr 172.21.15.83:0/1171491568 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:25:38.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.955+0000 7f983cb50640 1 --2- 172.21.15.83:0/1171491568 >> v2:172.21.15.183:3300/0 conn(0x7f9838071ce0 0x7f9838072140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.955+0000 7f983db52640 1 -- 172.21.15.83:0/1171491568 >> v2:172.21.15.183:3300/0 conn(0x7f9838071ce0 msgr2=0x7f9838072140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:38.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.955+0000 7f983db52640 1 --2- 172.21.15.83:0/1171491568 >> v2:172.21.15.183:3300/0 conn(0x7f9838071ce0 0x7f9838072140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.955+0000 7f983db52640 1 -- 172.21.15.83:0/1171491568 >> v1:172.21.15.83:6789/0 conn(0x7f98380cf790 legacy=0x7f98380ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.955+0000 7f983db52640 1 -- 172.21.15.83:0/1171491568 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98380ae240 con 0x7f983806d400 2024-03-06T07:25:38.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.956+0000 7f983db52640 1 --2- 172.21.15.83:0/1171491568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f983806d400 0x7f983806d890 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f98380737d0 tx=0x7f983002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bce3e591b85ce126 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.956+0000 7f982ffff640 1 -- 172.21.15.83:0/1171491568 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9830039070 con 0x7f983806d400 2024-03-06T07:25:38.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.956+0000 7f982ffff640 1 -- 172.21.15.83:0/1171491568 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f983002cc80 con 0x7f983806d400 2024-03-06T07:25:38.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.956+0000 7f982ffff640 1 -- 172.21.15.83:0/1171491568 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9830034660 con 0x7f983806d400 2024-03-06T07:25:38.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.956+0000 7f983f5b5640 1 -- 172.21.15.83:0/1171491568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f983806d400 msgr2=0x7f983806d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.956+0000 7f983f5b5640 1 --2- 172.21.15.83:0/1171491568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f983806d400 0x7f983806d890 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f98380737d0 tx=0x7f983002bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.956+0000 7f983f5b5640 1 -- 172.21.15.83:0/1171491568 shutdown_connections 2024-03-06T07:25:38.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.956+0000 7f983f5b5640 1 --2- 172.21.15.83:0/1171491568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f983806d400 0x7f983806d890 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.956+0000 7f983f5b5640 1 --2- 172.21.15.83:0/1171491568 >> v2:172.21.15.183:3300/0 conn(0x7f9838071ce0 0x7f9838072140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.956+0000 7f983f5b5640 1 -- 172.21.15.83:0/1171491568 >> 172.21.15.83:0/1171491568 conn(0x7f98380b1a90 msgr2=0x7f98380b1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:38.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.956+0000 7f983f5b5640 1 -- 172.21.15.83:0/1171491568 shutdown_connections 2024-03-06T07:25:38.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.956+0000 7f983f5b5640 1 -- 172.21.15.83:0/1171491568 wait complete. 2024-03-06T07:25:38.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.958+0000 7f983f5b5640 1 Processor -- start 2024-03-06T07:25:38.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.958+0000 7f983f5b5640 1 -- start start 2024-03-06T07:25:38.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.959+0000 7f983f5b5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f983806d400 0x7f983813f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.959+0000 7f983f5b5640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9838071ce0 0x7f9838156ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.959+0000 7f983f5b5640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98380cf790 0x7f9838156ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.959+0000 7f983f5b5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98380aeae0 con 0x7f983806d400 2024-03-06T07:25:38.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.959+0000 7f983f5b5640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98380ae800 con 0x7f9838071ce0 2024-03-06T07:25:38.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.959+0000 7f983f5b5640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f983815e580 con 0x7f98380cf790 2024-03-06T07:25:38.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.959+0000 7f983d351640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f983806d400 0x7f983813f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.959+0000 7f983d351640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f983806d400 0x7f983813f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52570/0 (socket says 172.21.15.83:52570) 2024-03-06T07:25:38.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.959+0000 7f983d351640 1 -- 172.21.15.83:0/233925761 learned_addr learned my addr 172.21.15.83:0/233925761 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:38.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.959+0000 7f983d351640 1 -- 172.21.15.83:0/233925761 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98380cf790 msgr2=0x7f9838156ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.959+0000 7f983d351640 1 --2- 172.21.15.83:0/233925761 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98380cf790 0x7f9838156ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.959+0000 7f983d351640 1 -- 172.21.15.83:0/233925761 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9838071ce0 msgr2=0x7f9838156ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.959+0000 7f983d351640 1 --2- 172.21.15.83:0/233925761 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9838071ce0 0x7f9838156ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.959+0000 7f983d351640 1 -- 172.21.15.83:0/233925761 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98381f49c0 con 0x7f983806d400 2024-03-06T07:25:38.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.960+0000 7f983d351640 1 --2- 172.21.15.83:0/233925761 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f983806d400 0x7f983813f730 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f982800c3c0 tx=0x7f982800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.960+0000 7f982dffb640 1 -- 172.21.15.83:0/233925761 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9828017070 con 0x7f983806d400 2024-03-06T07:25:38.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.960+0000 7f982dffb640 1 -- 172.21.15.83:0/233925761 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9828005ce0 con 0x7f983806d400 2024-03-06T07:25:38.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.960+0000 7f983f5b5640 1 -- 172.21.15.83:0/233925761 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9838152680 con 0x7f983806d400 2024-03-06T07:25:38.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.960+0000 7f982dffb640 1 -- 172.21.15.83:0/233925761 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98280072f0 con 0x7f983806d400 2024-03-06T07:25:38.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.960+0000 7f983f5b5640 1 -- 172.21.15.83:0/233925761 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9838152c40 con 0x7f983806d400 2024-03-06T07:25:38.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.961+0000 7f982dffb640 1 -- 172.21.15.83:0/233925761 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f982801a070 con 0x7f983806d400 2024-03-06T07:25:38.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.961+0000 7f983f5b5640 1 -- 172.21.15.83:0/233925761 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f983804f330 con 0x7f983806d400 2024-03-06T07:25:38.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.962+0000 7f982dffb640 1 --2- 172.21.15.83:0/233925761 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f981c06b500 0x7f981c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.962+0000 7f983cb50640 1 --2- 172.21.15.83:0/233925761 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f981c06b500 0x7f981c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.962+0000 7f982dffb640 1 -- 172.21.15.83:0/233925761 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f98280894c0 con 0x7f983806d400 2024-03-06T07:25:38.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.962+0000 7f983cb50640 1 --2- 172.21.15.83:0/233925761 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f981c06b500 0x7f981c06d9c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f9838162720 tx=0x7f9834008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:38.966+0000 7f982dffb640 1 -- 172.21.15.83:0/233925761 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f98280583d0 con 0x7f983806d400 2024-03-06T07:25:39.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.214+0000 7f983f5b5640 1 -- 172.21.15.83:0/233925761 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9838072dd0 con 0x7f983806d400 2024-03-06T07:25:39.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.215+0000 7f982dffb640 1 -- 172.21.15.83:0/233925761 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+16680 (secure 0 0 0) 0x7f982805e0d0 con 0x7f983806d400 2024-03-06T07:25:39.217 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:25:39.217 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":69,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:25:14.149032+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:10.701416+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:11.450797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:39.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.217+0000 7f983f5b5640 1 -- 172.21.15.83:0/233925761 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f981c06b500 msgr2=0x7f981c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.217+0000 7f983f5b5640 1 --2- 172.21.15.83:0/233925761 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f981c06b500 0x7f981c06d9c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f9838162720 tx=0x7f9834008040 comp rx=0 tx=0).stop 2024-03-06T07:25:39.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.217+0000 7f983f5b5640 1 -- 172.21.15.83:0/233925761 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f983806d400 msgr2=0x7f983813f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.217+0000 7f983f5b5640 1 --2- 172.21.15.83:0/233925761 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f983806d400 0x7f983813f730 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f982800c3c0 tx=0x7f982800c880 comp rx=0 tx=0).stop 2024-03-06T07:25:39.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.217+0000 7f983f5b5640 1 -- 172.21.15.83:0/233925761 shutdown_connections 2024-03-06T07:25:39.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.217+0000 7f983f5b5640 1 --2- 172.21.15.83:0/233925761 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f981c06b500 0x7f981c06d9c0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.217+0000 7f983f5b5640 1 --2- 172.21.15.83:0/233925761 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98380cf790 0x7f9838156ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.217+0000 7f983f5b5640 1 --2- 172.21.15.83:0/233925761 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9838071ce0 0x7f9838156ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.217+0000 7f983f5b5640 1 --2- 172.21.15.83:0/233925761 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f983806d400 0x7f983813f730 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.217+0000 7f983f5b5640 1 -- 172.21.15.83:0/233925761 >> 172.21.15.83:0/233925761 conn(0x7f98380b1a90 msgr2=0x7f9838161940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:39.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.217+0000 7f983f5b5640 1 -- 172.21.15.83:0/233925761 shutdown_connections 2024-03-06T07:25:39.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.218+0000 7f983f5b5640 1 -- 172.21.15.83:0/233925761 wait complete. 2024-03-06T07:25:39.230 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:25:10.701416+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:25:11.450797+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:25:39.230 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:25:39.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.332+0000 7f11c549d640 1 Processor -- start 2024-03-06T07:25:39.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.333+0000 7f11c549d640 1 -- start start 2024-03-06T07:25:39.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.333+0000 7f11c549d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11c01614d0 0x7f11c00d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.333+0000 7f11c549d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11c01618b0 con 0x7f11c00d1910 2024-03-06T07:25:39.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.333+0000 7f11c549d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11c00cf400 con 0x7f11c00d2480 2024-03-06T07:25:39.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.333+0000 7f11c549d640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11c00cf6a0 con 0x7f11c01614d0 2024-03-06T07:25:39.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.333+0000 7f11be7fc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f11c00d1910 0x7f11c00d1d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52508/0 (socket says 172.21.15.83:52508) 2024-03-06T07:25:39.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.333+0000 7f11be7fc640 1 -- 172.21.15.83:0/2377362577 learned_addr learned my addr 172.21.15.83:0/2377362577 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:25:39.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.334+0000 7f11beffd640 1 --2- 172.21.15.83:0/2377362577 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11c01614d0 0x7f11c00d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.334+0000 7f11bdffb640 1 -- 172.21.15.83:0/2377362577 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 320304653 0 0) 0x7f11c01618b0 con 0x7f11c00d1910 2024-03-06T07:25:39.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.334+0000 7f11bdffb640 1 -- 172.21.15.83:0/2377362577 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11a8003520 con 0x7f11c00d1910 2024-03-06T07:25:39.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.334+0000 7f11beffd640 1 -- 172.21.15.83:0/2377362577 >> v1:172.21.15.183:6789/0 conn(0x7f11c00d2480 legacy=0x7f11c00b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.334+0000 7f11bdffb640 1 -- 172.21.15.83:0/2377362577 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2978050600 0 0) 0x7f11a8003520 con 0x7f11c00d1910 2024-03-06T07:25:39.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.334+0000 7f11beffd640 1 -- 172.21.15.83:0/2377362577 >> v1:172.21.15.83:6789/0 conn(0x7f11c00d1910 legacy=0x7f11c00d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.334+0000 7f11beffd640 1 -- 172.21.15.83:0/2377362577 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c00cf980 con 0x7f11c01614d0 2024-03-06T07:25:39.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.335+0000 7f11beffd640 1 --2- 172.21.15.83:0/2377362577 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11c01614d0 0x7f11c00d4860 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f11b00097d0 tx=0x7f11b002b660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2369e0176a83b85 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.335+0000 7f11bdffb640 1 -- 172.21.15.83:0/2377362577 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b0037070 con 0x7f11c01614d0 2024-03-06T07:25:39.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.335+0000 7f11bdffb640 1 -- 172.21.15.83:0/2377362577 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11b0030510 con 0x7f11c01614d0 2024-03-06T07:25:39.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.335+0000 7f11bdffb640 1 -- 172.21.15.83:0/2377362577 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b002be60 con 0x7f11c01614d0 2024-03-06T07:25:39.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.336+0000 7f11c549d640 1 -- 172.21.15.83:0/2377362577 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11c01614d0 msgr2=0x7f11c00d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.336+0000 7f11c549d640 1 --2- 172.21.15.83:0/2377362577 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11c01614d0 0x7f11c00d4860 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f11b00097d0 tx=0x7f11b002b660 comp rx=0 tx=0).stop 2024-03-06T07:25:39.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.336+0000 7f11c549d640 1 -- 172.21.15.83:0/2377362577 shutdown_connections 2024-03-06T07:25:39.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.336+0000 7f11c549d640 1 --2- 172.21.15.83:0/2377362577 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11c01614d0 0x7f11c00d4860 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.336+0000 7f11c549d640 1 -- 172.21.15.83:0/2377362577 >> 172.21.15.83:0/2377362577 conn(0x7f11c00d3d40 msgr2=0x7f11c00d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:39.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.336+0000 7f11c549d640 1 -- 172.21.15.83:0/2377362577 shutdown_connections 2024-03-06T07:25:39.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.336+0000 7f11c549d640 1 -- 172.21.15.83:0/2377362577 wait complete. 2024-03-06T07:25:39.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.338+0000 7f11c549d640 1 Processor -- start 2024-03-06T07:25:39.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.338+0000 7f11c549d640 1 -- start start 2024-03-06T07:25:39.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.339+0000 7f11c549d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11c00d1910 0x7f11c00cd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.339+0000 7f11c549d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11c00d2480 0x7f11c0152020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.339+0000 7f11c549d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11c01614d0 0x7f11c00b3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.339+0000 7f11c549d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c00b1540 con 0x7f11c00d1910 2024-03-06T07:25:39.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.339+0000 7f11c549d640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c00b0910 con 0x7f11c01614d0 2024-03-06T07:25:39.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.339+0000 7f11c549d640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11c00b16c0 con 0x7f11c00d2480 2024-03-06T07:25:39.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.339+0000 7f11beffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11c00d1910 0x7f11c00cd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.339+0000 7f11beffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11c00d1910 0x7f11c00cd270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52584/0 (socket says 172.21.15.83:52584) 2024-03-06T07:25:39.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.339+0000 7f11beffd640 1 -- 172.21.15.83:0/1602011569 learned_addr learned my addr 172.21.15.83:0/1602011569 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:39.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.339+0000 7f11be7fc640 1 --2- 172.21.15.83:0/1602011569 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11c00d2480 0x7f11c0152020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.339+0000 7f11bf7fe640 1 --2- 172.21.15.83:0/1602011569 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11c01614d0 0x7f11c00b3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.339+0000 7f11beffd640 1 -- 172.21.15.83:0/1602011569 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11c00d2480 msgr2=0x7f11c0152020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.339+0000 7f11beffd640 1 --2- 172.21.15.83:0/1602011569 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11c00d2480 0x7f11c0152020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.339+0000 7f11beffd640 1 -- 172.21.15.83:0/1602011569 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11c01614d0 msgr2=0x7f11c00b3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.339+0000 7f11beffd640 1 --2- 172.21.15.83:0/1602011569 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11c01614d0 0x7f11c00b3cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.339+0000 7f11beffd640 1 -- 172.21.15.83:0/1602011569 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c00cbb00 con 0x7f11c00d1910 2024-03-06T07:25:39.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.340+0000 7f11beffd640 1 --2- 172.21.15.83:0/1602011569 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11c00d1910 0x7f11c00cd270 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f11b0035040 tx=0x7f11b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.340+0000 7f11a77fe640 1 -- 172.21.15.83:0/1602011569 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b0042070 con 0x7f11c00d1910 2024-03-06T07:25:39.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.340+0000 7f11a77fe640 1 -- 172.21.15.83:0/1602011569 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11b003d850 con 0x7f11c00d1910 2024-03-06T07:25:39.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.340+0000 7f11c549d640 1 -- 172.21.15.83:0/1602011569 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11c00cc790 con 0x7f11c00d1910 2024-03-06T07:25:39.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.340+0000 7f11a77fe640 1 -- 172.21.15.83:0/1602011569 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b0037040 con 0x7f11c00d1910 2024-03-06T07:25:39.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.340+0000 7f11c549d640 1 -- 172.21.15.83:0/1602011569 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11c00ccd50 con 0x7f11c00d1910 2024-03-06T07:25:39.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.341+0000 7f11c549d640 1 -- 172.21.15.83:0/1602011569 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11c0068ff0 con 0x7f11c00d1910 2024-03-06T07:25:39.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.341+0000 7f11a77fe640 1 -- 172.21.15.83:0/1602011569 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f11b0045070 con 0x7f11c00d1910 2024-03-06T07:25:39.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.342+0000 7f11a77fe640 1 --2- 172.21.15.83:0/1602011569 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f11ac06b4d0 0x7f11ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.342+0000 7f11a77fe640 1 -- 172.21.15.83:0/1602011569 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f11b00ab9a0 con 0x7f11c00d1910 2024-03-06T07:25:39.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.342+0000 7f11be7fc640 1 --2- 172.21.15.83:0/1602011569 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f11ac06b4d0 0x7f11ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.342+0000 7f11be7fc640 1 --2- 172.21.15.83:0/1602011569 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f11ac06b4d0 0x7f11ac06d990 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f11b4001eb0 tx=0x7f11b4001ac0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.346+0000 7f11a77fe640 1 -- 172.21.15.83:0/1602011569 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f11b007a260 con 0x7f11c00d1910 2024-03-06T07:25:39.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.521+0000 7f11c549d640 1 -- 172.21.15.83:0/1602011569 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f11c006c4e0 con 0x7f11c00d1910 2024-03-06T07:25:39.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.522+0000 7f11a77fe640 1 -- 172.21.15.83:0/1602011569 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v69) v1 ==== 93+0+10 (secure 0 0 0) 0x7f11b007e300 con 0x7f11c00d1910 2024-03-06T07:25:39.524 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:25:39.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.524+0000 7f11c549d640 1 -- 172.21.15.83:0/1602011569 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f11ac06b4d0 msgr2=0x7f11ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.524+0000 7f11c549d640 1 --2- 172.21.15.83:0/1602011569 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f11ac06b4d0 0x7f11ac06d990 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f11b4001eb0 tx=0x7f11b4001ac0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.524+0000 7f11c549d640 1 -- 172.21.15.83:0/1602011569 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11c00d1910 msgr2=0x7f11c00cd270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.524+0000 7f11c549d640 1 --2- 172.21.15.83:0/1602011569 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11c00d1910 0x7f11c00cd270 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f11b0035040 tx=0x7f11b00046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.524+0000 7f11c549d640 1 -- 172.21.15.83:0/1602011569 shutdown_connections 2024-03-06T07:25:39.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.524+0000 7f11c549d640 1 --2- 172.21.15.83:0/1602011569 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f11ac06b4d0 0x7f11ac06d990 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.524+0000 7f11c549d640 1 --2- 172.21.15.83:0/1602011569 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11c01614d0 0x7f11c00b3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.524+0000 7f11c549d640 1 --2- 172.21.15.83:0/1602011569 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11c00d2480 0x7f11c0152020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.524+0000 7f11c549d640 1 --2- 172.21.15.83:0/1602011569 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11c00d1910 0x7f11c00cd270 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.524+0000 7f11c549d640 1 -- 172.21.15.83:0/1602011569 >> 172.21.15.83:0/1602011569 conn(0x7f11c00d3d40 msgr2=0x7f11c00b5830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:39.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.524+0000 7f11c549d640 1 -- 172.21.15.83:0/1602011569 shutdown_connections 2024-03-06T07:25:39.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.524+0000 7f11c549d640 1 -- 172.21.15.83:0/1602011569 wait complete. 2024-03-06T07:25:39.537 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:25:39.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.639+0000 7fe170a2a640 1 Processor -- start 2024-03-06T07:25:39.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.639+0000 7fe170a2a640 1 -- start start 2024-03-06T07:25:39.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.640+0000 7fe170a2a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe16c1614d0 0x7fe16c0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.640+0000 7fe170a2a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe16c0d1910 0x7fe16c0d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.640+0000 7fe170a2a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe16c0d2740 con 0x7fe16c0d22b0 2024-03-06T07:25:39.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.640+0000 7fe170a2a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe16c1618b0 con 0x7fe16c0d1910 2024-03-06T07:25:39.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.640+0000 7fe170a2a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe16c0cf400 con 0x7fe16c1614d0 2024-03-06T07:25:39.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.640+0000 7fe161d9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe16c0d1910 0x7fe16c0d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.640+0000 7fe16ad9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fe16c0d22b0 0x7fe16c0d06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52510/0 (socket says 172.21.15.83:52510) 2024-03-06T07:25:39.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.640+0000 7fe16ad9d640 1 -- 172.21.15.83:0/2768563681 learned_addr learned my addr 172.21.15.83:0/2768563681 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:25:39.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.640+0000 7fe16a59c640 1 --2- 172.21.15.83:0/2768563681 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe16c1614d0 0x7fe16c0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.640+0000 7fe161d9b640 1 -- 172.21.15.83:0/2768563681 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe16c1614d0 msgr2=0x7fe16c0d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:39.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.640+0000 7fe161d9b640 1 --2- 172.21.15.83:0/2768563681 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe16c1614d0 0x7fe16c0d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.640+0000 7fe161d9b640 1 -- 172.21.15.83:0/2768563681 >> v1:172.21.15.83:6789/0 conn(0x7fe16c0d22b0 legacy=0x7fe16c0d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.640+0000 7fe161d9b640 1 -- 172.21.15.83:0/2768563681 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe16c0cf6a0 con 0x7fe16c0d1910 2024-03-06T07:25:39.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.640+0000 7fe169d9b640 1 -- 172.21.15.83:0/2768563681 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3143855435 0 0) 0x7fe164001490 con 0x7fe16c0d22b0 2024-03-06T07:25:39.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.641+0000 7fe161d9b640 1 --2- 172.21.15.83:0/2768563681 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe16c0d1910 0x7fe16c0d1d70 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fe1540097a0 tx=0x7fe15402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2e3d144e9f3e3b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.641+0000 7fe169d9b640 1 -- 172.21.15.83:0/2768563681 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe154035020 con 0x7fe16c0d1910 2024-03-06T07:25:39.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.641+0000 7fe169d9b640 1 -- 172.21.15.83:0/2768563681 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe154031830 con 0x7fe16c0d1910 2024-03-06T07:25:39.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.641+0000 7fe169d9b640 1 -- 172.21.15.83:0/2768563681 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe154031c00 con 0x7fe16c0d1910 2024-03-06T07:25:39.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.641+0000 7fe170a2a640 1 -- 172.21.15.83:0/2768563681 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe16c0d1910 msgr2=0x7fe16c0d1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.641+0000 7fe170a2a640 1 --2- 172.21.15.83:0/2768563681 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe16c0d1910 0x7fe16c0d1d70 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fe1540097a0 tx=0x7fe15402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.642+0000 7fe170a2a640 1 -- 172.21.15.83:0/2768563681 shutdown_connections 2024-03-06T07:25:39.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.642+0000 7fe170a2a640 1 --2- 172.21.15.83:0/2768563681 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe16c0d1910 0x7fe16c0d1d70 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.642+0000 7fe170a2a640 1 --2- 172.21.15.83:0/2768563681 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe16c1614d0 0x7fe16c0d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.642+0000 7fe170a2a640 1 -- 172.21.15.83:0/2768563681 >> 172.21.15.83:0/2768563681 conn(0x7fe16c0d3d40 msgr2=0x7fe16c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:39.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.642+0000 7fe170a2a640 1 -- 172.21.15.83:0/2768563681 shutdown_connections 2024-03-06T07:25:39.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.642+0000 7fe170a2a640 1 -- 172.21.15.83:0/2768563681 wait complete. 2024-03-06T07:25:39.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.643+0000 7fe170a2a640 1 Processor -- start 2024-03-06T07:25:39.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.643+0000 7fe170a2a640 1 -- start start 2024-03-06T07:25:39.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.644+0000 7fe170a2a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe16c0d1910 0x7fe16c0b7c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.644+0000 7fe170a2a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe16c0d22b0 0x7fe16c0cc5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.644+0000 7fe170a2a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe16c1614d0 0x7fe16c0cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.644+0000 7fe170a2a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe16c0b0910 con 0x7fe16c0d1910 2024-03-06T07:25:39.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.644+0000 7fe170a2a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe16c0cfc60 con 0x7fe16c0d22b0 2024-03-06T07:25:39.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.644+0000 7fe170a2a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe16c0b1e40 con 0x7fe16c1614d0 2024-03-06T07:25:39.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.644+0000 7fe16a59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe16c0d1910 0x7fe16c0b7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.644+0000 7fe16a59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe16c0d1910 0x7fe16c0b7c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52594/0 (socket says 172.21.15.83:52594) 2024-03-06T07:25:39.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.644+0000 7fe16a59c640 1 -- 172.21.15.83:0/1429881080 learned_addr learned my addr 172.21.15.83:0/1429881080 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:39.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.644+0000 7fe16ad9d640 1 --2- 172.21.15.83:0/1429881080 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe16c1614d0 0x7fe16c0cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.644+0000 7fe161d9b640 1 --2- 172.21.15.83:0/1429881080 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe16c0d22b0 0x7fe16c0cc5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.644+0000 7fe16a59c640 1 -- 172.21.15.83:0/1429881080 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe16c1614d0 msgr2=0x7fe16c0cae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.644+0000 7fe16a59c640 1 --2- 172.21.15.83:0/1429881080 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe16c1614d0 0x7fe16c0cae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.644+0000 7fe16a59c640 1 -- 172.21.15.83:0/1429881080 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe16c0d22b0 msgr2=0x7fe16c0cc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.644+0000 7fe16a59c640 1 --2- 172.21.15.83:0/1429881080 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe16c0d22b0 0x7fe16c0cc5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.644+0000 7fe16a59c640 1 -- 172.21.15.83:0/1429881080 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe16c15ca80 con 0x7fe16c0d1910 2024-03-06T07:25:39.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.644+0000 7fe16ad9d640 1 --2- 172.21.15.83:0/1429881080 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe16c1614d0 0x7fe16c0cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:39.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.644+0000 7fe16a59c640 1 --2- 172.21.15.83:0/1429881080 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe16c0d1910 0x7fe16c0b7c10 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fe15c00d5f0 tx=0x7fe15c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.645+0000 7fe1637fe640 1 -- 172.21.15.83:0/1429881080 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15c0125f0 con 0x7fe16c0d1910 2024-03-06T07:25:39.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.645+0000 7fe1637fe640 1 -- 172.21.15.83:0/1429881080 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe15c019070 con 0x7fe16c0d1910 2024-03-06T07:25:39.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.645+0000 7fe170a2a640 1 -- 172.21.15.83:0/1429881080 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe16c0cb510 con 0x7fe16c0d1910 2024-03-06T07:25:39.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.645+0000 7fe1637fe640 1 -- 172.21.15.83:0/1429881080 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15c0129d0 con 0x7fe16c0d1910 2024-03-06T07:25:39.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.645+0000 7fe170a2a640 1 -- 172.21.15.83:0/1429881080 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe16c0cbad0 con 0x7fe16c0d1910 2024-03-06T07:25:39.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.645+0000 7fe170a2a640 1 -- 172.21.15.83:0/1429881080 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe16c0ce170 con 0x7fe16c0d1910 2024-03-06T07:25:39.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.646+0000 7fe1637fe640 1 -- 172.21.15.83:0/1429881080 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe15c012b70 con 0x7fe16c0d1910 2024-03-06T07:25:39.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.646+0000 7fe1637fe640 1 --2- 172.21.15.83:0/1429881080 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe14c06b4d0 0x7fe14c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.646+0000 7fe1637fe640 1 -- 172.21.15.83:0/1429881080 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe15c08a610 con 0x7fe16c0d1910 2024-03-06T07:25:39.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.646+0000 7fe161d9b640 1 --2- 172.21.15.83:0/1429881080 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe14c06b4d0 0x7fe14c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.647+0000 7fe161d9b640 1 --2- 172.21.15.83:0/1429881080 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe14c06b4d0 0x7fe14c06d990 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fe154036000 tx=0x7fe154008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.651+0000 7fe1637fe640 1 -- 172.21.15.83:0/1429881080 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe15c059490 con 0x7fe16c0d1910 2024-03-06T07:25:39.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.824+0000 7fe170a2a640 1 -- 172.21.15.83:0/1429881080 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe16c15f890 con 0x7fe16c0d1910 2024-03-06T07:25:39.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.824+0000 7fe1637fe640 1 -- 172.21.15.83:0/1429881080 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v69) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe15c058d90 con 0x7fe16c0d1910 2024-03-06T07:25:39.826 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-03-06T07:25:39.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.826+0000 7fe170a2a640 1 -- 172.21.15.83:0/1429881080 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe14c06b4d0 msgr2=0x7fe14c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.826+0000 7fe170a2a640 1 --2- 172.21.15.83:0/1429881080 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe14c06b4d0 0x7fe14c06d990 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fe154036000 tx=0x7fe154008030 comp rx=0 tx=0).stop 2024-03-06T07:25:39.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.826+0000 7fe170a2a640 1 -- 172.21.15.83:0/1429881080 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe16c0d1910 msgr2=0x7fe16c0b7c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.826+0000 7fe170a2a640 1 --2- 172.21.15.83:0/1429881080 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe16c0d1910 0x7fe16c0b7c10 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fe15c00d5f0 tx=0x7fe15c00dab0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.826+0000 7fe170a2a640 1 -- 172.21.15.83:0/1429881080 shutdown_connections 2024-03-06T07:25:39.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.826+0000 7fe170a2a640 1 --2- 172.21.15.83:0/1429881080 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe14c06b4d0 0x7fe14c06d990 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.826+0000 7fe170a2a640 1 --2- 172.21.15.83:0/1429881080 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe16c1614d0 0x7fe16c0cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.826+0000 7fe170a2a640 1 --2- 172.21.15.83:0/1429881080 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe16c0d22b0 0x7fe16c0cc5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.826+0000 7fe170a2a640 1 --2- 172.21.15.83:0/1429881080 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe16c0d1910 0x7fe16c0b7c10 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.826+0000 7fe170a2a640 1 -- 172.21.15.83:0/1429881080 >> 172.21.15.83:0/1429881080 conn(0x7fe16c0d3d40 msgr2=0x7fe16c0cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:39.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.826+0000 7fe170a2a640 1 -- 172.21.15.83:0/1429881080 shutdown_connections 2024-03-06T07:25:39.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.826+0000 7fe170a2a640 1 -- 172.21.15.83:0/1429881080 wait complete. 2024-03-06T07:25:39.839 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:25:39.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.944+0000 7f795ac1c640 1 Processor -- start 2024-03-06T07:25:39.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.944+0000 7f795ac1c640 1 -- start start 2024-03-06T07:25:39.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.945+0000 7f795ac1c640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f79541586e0 0x7f79540aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.945+0000 7f795ac1c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f79540aef80 0x7f79540cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.945+0000 7f795ac1c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79540d03f0 con 0x7f79540cffd0 2024-03-06T07:25:39.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.945+0000 7f795ac1c640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7954158ac0 con 0x7f79541586e0 2024-03-06T07:25:39.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.945+0000 7f795ac1c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f795415b2a0 con 0x7f79540aef80 2024-03-06T07:25:39.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.945+0000 7f79591b9640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f79540cffd0 0x7f795406da20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52512/0 (socket says 172.21.15.83:52512) 2024-03-06T07:25:39.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.945+0000 7f79591b9640 1 -- 172.21.15.83:0/539202339 learned_addr learned my addr 172.21.15.83:0/539202339 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:25:39.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.945+0000 7f7953fff640 1 --2- 172.21.15.83:0/539202339 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f79540aef80 0x7f79540cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.945+0000 7f79589b8640 1 --2- 172.21.15.83:0/539202339 >> v2:172.21.15.183:3300/0 conn(0x7f79541586e0 0x7f79540aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.945+0000 7f79537fe640 1 -- 172.21.15.83:0/539202339 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3699298429 0 0) 0x7f79540d03f0 con 0x7f79540cffd0 2024-03-06T07:25:39.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.945+0000 7f79537fe640 1 -- 172.21.15.83:0/539202339 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f793c003520 con 0x7f79540cffd0 2024-03-06T07:25:39.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.946+0000 7f79537fe640 1 -- 172.21.15.83:0/539202339 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2148798479 0 0) 0x7f793c003520 con 0x7f79540cffd0 2024-03-06T07:25:39.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.946+0000 7f7953fff640 1 -- 172.21.15.83:0/539202339 >> v2:172.21.15.183:3300/0 conn(0x7f79541586e0 msgr2=0x7f79540aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:39.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.946+0000 7f7953fff640 1 --2- 172.21.15.83:0/539202339 >> v2:172.21.15.183:3300/0 conn(0x7f79541586e0 0x7f79540aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.946+0000 7f7953fff640 1 -- 172.21.15.83:0/539202339 >> v1:172.21.15.83:6789/0 conn(0x7f79540cffd0 legacy=0x7f795406da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.946+0000 7f7953fff640 1 -- 172.21.15.83:0/539202339 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f795415b540 con 0x7f79540aef80 2024-03-06T07:25:39.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.946+0000 7f79589b8640 1 --2- 172.21.15.83:0/539202339 >> v2:172.21.15.183:3300/0 conn(0x7f79541586e0 0x7f79540aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:39.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.946+0000 7f7953fff640 1 --2- 172.21.15.83:0/539202339 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f79540aef80 0x7f79540cfa90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f79480097a0 tx=0x7f794802b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18c05217919a1c55 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.947+0000 7f79537fe640 1 -- 172.21.15.83:0/539202339 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7948037070 con 0x7f79540aef80 2024-03-06T07:25:39.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.947+0000 7f79537fe640 1 -- 172.21.15.83:0/539202339 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7948030510 con 0x7f79540aef80 2024-03-06T07:25:39.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.947+0000 7f79537fe640 1 -- 172.21.15.83:0/539202339 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7944001670 con 0x7f79540aef80 2024-03-06T07:25:39.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.947+0000 7f795ac1c640 1 -- 172.21.15.83:0/539202339 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f79540aef80 msgr2=0x7f79540cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.947+0000 7f795ac1c640 1 --2- 172.21.15.83:0/539202339 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f79540aef80 0x7f79540cfa90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f79480097a0 tx=0x7f794802b790 comp rx=0 tx=0).stop 2024-03-06T07:25:39.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.947+0000 7f795ac1c640 1 -- 172.21.15.83:0/539202339 shutdown_connections 2024-03-06T07:25:39.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.947+0000 7f795ac1c640 1 --2- 172.21.15.83:0/539202339 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f79540aef80 0x7f79540cfa90 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.947+0000 7f795ac1c640 1 --2- 172.21.15.83:0/539202339 >> v2:172.21.15.183:3300/0 conn(0x7f79541586e0 0x7f79540aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.947+0000 7f795ac1c640 1 -- 172.21.15.83:0/539202339 >> 172.21.15.83:0/539202339 conn(0x7f795406b190 msgr2=0x7f795406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:39.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.947+0000 7f795ac1c640 1 -- 172.21.15.83:0/539202339 shutdown_connections 2024-03-06T07:25:39.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.947+0000 7f795ac1c640 1 -- 172.21.15.83:0/539202339 wait complete. 2024-03-06T07:25:39.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.948+0000 7f795ac1c640 1 Processor -- start 2024-03-06T07:25:39.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.948+0000 7f795ac1c640 1 -- start start 2024-03-06T07:25:39.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.949+0000 7f795ac1c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f79540aef80 0x7f795414b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.949+0000 7f795ac1c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f79540cffd0 0x7f795406e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.949+0000 7f795ac1c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79541586e0 0x7f79540ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.949+0000 7f795ac1c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f795415bde0 con 0x7f79541586e0 2024-03-06T07:25:39.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.949+0000 7f795ac1c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f795415bb00 con 0x7f79540cffd0 2024-03-06T07:25:39.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.949+0000 7f795ac1c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7954071350 con 0x7f79540aef80 2024-03-06T07:25:39.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.949+0000 7f79591b9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79541586e0 0x7f79540ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.949+0000 7f79591b9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79541586e0 0x7f79540ad860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52606/0 (socket says 172.21.15.83:52606) 2024-03-06T07:25:39.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.949+0000 7f79591b9640 1 -- 172.21.15.83:0/618395046 learned_addr learned my addr 172.21.15.83:0/618395046 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:39.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.949+0000 7f79589b8640 1 --2- 172.21.15.83:0/618395046 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f79540aef80 0x7f795414b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.949+0000 7f7953fff640 1 --2- 172.21.15.83:0/618395046 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f79540cffd0 0x7f795406e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.950+0000 7f79591b9640 1 -- 172.21.15.83:0/618395046 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f79540aef80 msgr2=0x7f795414b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.950+0000 7f79591b9640 1 --2- 172.21.15.83:0/618395046 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f79540aef80 0x7f795414b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.950+0000 7f79591b9640 1 -- 172.21.15.83:0/618395046 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f79540cffd0 msgr2=0x7f795406e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.950+0000 7f79591b9640 1 --2- 172.21.15.83:0/618395046 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f79540cffd0 0x7f795406e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.950+0000 7f79591b9640 1 -- 172.21.15.83:0/618395046 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f795406c9b0 con 0x7f79541586e0 2024-03-06T07:25:39.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.950+0000 7f79591b9640 1 --2- 172.21.15.83:0/618395046 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79541586e0 0x7f79540ad860 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f794400c750 tx=0x7f794400cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.950+0000 7f79517fa640 1 -- 172.21.15.83:0/618395046 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7944005ad0 con 0x7f79541586e0 2024-03-06T07:25:39.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.950+0000 7f79517fa640 1 -- 172.21.15.83:0/618395046 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7944005c90 con 0x7f79541586e0 2024-03-06T07:25:39.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.950+0000 7f795ac1c640 1 -- 172.21.15.83:0/618395046 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f795406cc90 con 0x7f79541586e0 2024-03-06T07:25:39.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.950+0000 7f79517fa640 1 -- 172.21.15.83:0/618395046 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79440072b0 con 0x7f79541586e0 2024-03-06T07:25:39.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.950+0000 7f795ac1c640 1 -- 172.21.15.83:0/618395046 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f795406d250 con 0x7f79541586e0 2024-03-06T07:25:39.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.951+0000 7f795ac1c640 1 -- 172.21.15.83:0/618395046 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79540732b0 con 0x7f79541586e0 2024-03-06T07:25:39.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.952+0000 7f79517fa640 1 -- 172.21.15.83:0/618395046 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7944007450 con 0x7f79541586e0 2024-03-06T07:25:39.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.952+0000 7f79517fa640 1 --2- 172.21.15.83:0/618395046 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f794006b4d0 0x7f794006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.952+0000 7f79517fa640 1 -- 172.21.15.83:0/618395046 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7944089f40 con 0x7f79541586e0 2024-03-06T07:25:39.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.952+0000 7f79589b8640 1 --2- 172.21.15.83:0/618395046 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f794006b4d0 0x7f794006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.952+0000 7f79589b8640 1 --2- 172.21.15.83:0/618395046 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f794006b4d0 0x7f794006d990 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f79541f6da0 tx=0x7f794c00b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:39.961+0000 7f79517fa640 1 -- 172.21.15.83:0/618395046 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7944058730 con 0x7f79541586e0 2024-03-06T07:25:40.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.137+0000 7f795ac1c640 1 -- 172.21.15.83:0/618395046 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f79540cfb00 con 0x7f79541586e0 2024-03-06T07:25:40.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.138+0000 7f79517fa640 1 -- 172.21.15.83:0/618395046 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v70) v1 ==== 100+0+11 (secure 0 0 0) 0x7f794405c7d0 con 0x7f79541586e0 2024-03-06T07:25:40.139 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 64 2024-03-06T07:25:40.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.139+0000 7f795ac1c640 1 -- 172.21.15.83:0/618395046 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f794006b4d0 msgr2=0x7f794006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.139+0000 7f795ac1c640 1 --2- 172.21.15.83:0/618395046 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f794006b4d0 0x7f794006d990 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f79541f6da0 tx=0x7f794c00b040 comp rx=0 tx=0).stop 2024-03-06T07:25:40.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.139+0000 7f795ac1c640 1 -- 172.21.15.83:0/618395046 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79541586e0 msgr2=0x7f79540ad860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.139+0000 7f795ac1c640 1 --2- 172.21.15.83:0/618395046 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79541586e0 0x7f79540ad860 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f794400c750 tx=0x7f794400cc10 comp rx=0 tx=0).stop 2024-03-06T07:25:40.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.140+0000 7f795ac1c640 1 -- 172.21.15.83:0/618395046 shutdown_connections 2024-03-06T07:25:40.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.140+0000 7f795ac1c640 1 --2- 172.21.15.83:0/618395046 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f794006b4d0 0x7f794006d990 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.140+0000 7f795ac1c640 1 --2- 172.21.15.83:0/618395046 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79541586e0 0x7f79540ad860 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.140+0000 7f795ac1c640 1 --2- 172.21.15.83:0/618395046 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f79540cffd0 0x7f795406e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.140+0000 7f795ac1c640 1 --2- 172.21.15.83:0/618395046 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f79540aef80 0x7f795414b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.140+0000 7f795ac1c640 1 -- 172.21.15.83:0/618395046 >> 172.21.15.83:0/618395046 conn(0x7f795406b190 msgr2=0x7f7954155970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:40.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.140+0000 7f795ac1c640 1 -- 172.21.15.83:0/618395046 shutdown_connections 2024-03-06T07:25:40.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.140+0000 7f795ac1c640 1 -- 172.21.15.83:0/618395046 wait complete. 2024-03-06T07:25:40.152 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:25:40.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.251+0000 7fd05820e640 1 Processor -- start 2024-03-06T07:25:40.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.251+0000 7fd05820e640 1 -- start start 2024-03-06T07:25:40.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.252+0000 7fd05820e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0500b4460 0x7fd0500d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.252+0000 7fd05820e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0500d22c0 con 0x7fd050161cf0 2024-03-06T07:25:40.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.252+0000 7fd05820e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0500d25a0 con 0x7fd050162970 2024-03-06T07:25:40.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.252+0000 7fd05820e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd050161b00 con 0x7fd0500b4460 2024-03-06T07:25:40.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.252+0000 7fd0567ab640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fd050161cf0 0x7fd0500cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52522/0 (socket says 172.21.15.83:52522) 2024-03-06T07:25:40.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.252+0000 7fd0567ab640 1 -- 172.21.15.83:0/2867715478 learned_addr learned my addr 172.21.15.83:0/2867715478 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:25:40.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.252+0000 7fd055faa640 1 --2- 172.21.15.83:0/2867715478 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0500b4460 0x7fd0500d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.252+0000 7fd054fa8640 1 -- 172.21.15.83:0/2867715478 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1237248791 0 0) 0x7fd0500d22c0 con 0x7fd050161cf0 2024-03-06T07:25:40.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.252+0000 7fd054fa8640 1 -- 172.21.15.83:0/2867715478 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd03c003520 con 0x7fd050161cf0 2024-03-06T07:25:40.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.252+0000 7fd055faa640 1 -- 172.21.15.83:0/2867715478 >> v1:172.21.15.183:6789/0 conn(0x7fd050162970 legacy=0x7fd0500b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.252+0000 7fd055faa640 1 -- 172.21.15.83:0/2867715478 >> v1:172.21.15.83:6789/0 conn(0x7fd050161cf0 legacy=0x7fd0500cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.252+0000 7fd055faa640 1 -- 172.21.15.83:0/2867715478 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0500d2880 con 0x7fd0500b4460 2024-03-06T07:25:40.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.253+0000 7fd055faa640 1 --2- 172.21.15.83:0/2867715478 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0500b4460 0x7fd0500d4800 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd04c0097d0 tx=0x7fd04c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4857c68c53fa5945 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.253+0000 7fd054fa8640 1 -- 172.21.15.83:0/2867715478 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd04c035020 con 0x7fd0500b4460 2024-03-06T07:25:40.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.253+0000 7fd054fa8640 1 -- 172.21.15.83:0/2867715478 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd04c031890 con 0x7fd0500b4460 2024-03-06T07:25:40.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.253+0000 7fd054fa8640 1 -- 172.21.15.83:0/2867715478 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd04c031c60 con 0x7fd0500b4460 2024-03-06T07:25:40.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.254+0000 7fd05820e640 1 -- 172.21.15.83:0/2867715478 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0500b4460 msgr2=0x7fd0500d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.254+0000 7fd05820e640 1 --2- 172.21.15.83:0/2867715478 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0500b4460 0x7fd0500d4800 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd04c0097d0 tx=0x7fd04c02b930 comp rx=0 tx=0).stop 2024-03-06T07:25:40.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.254+0000 7fd05820e640 1 -- 172.21.15.83:0/2867715478 shutdown_connections 2024-03-06T07:25:40.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.254+0000 7fd05820e640 1 --2- 172.21.15.83:0/2867715478 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0500b4460 0x7fd0500d4800 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.254+0000 7fd05820e640 1 -- 172.21.15.83:0/2867715478 >> 172.21.15.83:0/2867715478 conn(0x7fd0500d3ce0 msgr2=0x7fd0500d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:40.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.254+0000 7fd05820e640 1 -- 172.21.15.83:0/2867715478 shutdown_connections 2024-03-06T07:25:40.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.254+0000 7fd05820e640 1 -- 172.21.15.83:0/2867715478 wait complete. 2024-03-06T07:25:40.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.255+0000 7fd05820e640 1 Processor -- start 2024-03-06T07:25:40.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.255+0000 7fd05820e640 1 -- start start 2024-03-06T07:25:40.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.256+0000 7fd05820e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0500b4460 0x7fd050155440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.256+0000 7fd05820e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd050161cf0 0x7fd0500c83e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.256+0000 7fd05820e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd050162970 0x7fd0500b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.256+0000 7fd05820e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0500ce950 con 0x7fd0500b4460 2024-03-06T07:25:40.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.256+0000 7fd05820e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0500cecc0 con 0x7fd050161cf0 2024-03-06T07:25:40.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.256+0000 7fd05820e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0500b4840 con 0x7fd050162970 2024-03-06T07:25:40.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.256+0000 7fd055faa640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0500b4460 0x7fd050155440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.256+0000 7fd055faa640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0500b4460 0x7fd050155440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52614/0 (socket says 172.21.15.83:52614) 2024-03-06T07:25:40.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.256+0000 7fd055faa640 1 -- 172.21.15.83:0/3019824109 learned_addr learned my addr 172.21.15.83:0/3019824109 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:40.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.256+0000 7fd0567ab640 1 --2- 172.21.15.83:0/3019824109 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd050162970 0x7fd0500b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.256+0000 7fd0557a9640 1 --2- 172.21.15.83:0/3019824109 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd050161cf0 0x7fd0500c83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.256+0000 7fd055faa640 1 -- 172.21.15.83:0/3019824109 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd050162970 msgr2=0x7fd0500b1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.256+0000 7fd055faa640 1 --2- 172.21.15.83:0/3019824109 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd050162970 0x7fd0500b1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.256+0000 7fd055faa640 1 -- 172.21.15.83:0/3019824109 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd050161cf0 msgr2=0x7fd0500c83e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.256+0000 7fd055faa640 1 --2- 172.21.15.83:0/3019824109 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd050161cf0 0x7fd0500c83e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.256+0000 7fd055faa640 1 -- 172.21.15.83:0/3019824109 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0500cd250 con 0x7fd0500b4460 2024-03-06T07:25:40.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.256+0000 7fd0557a9640 1 --2- 172.21.15.83:0/3019824109 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd050161cf0 0x7fd0500c83e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:40.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.256+0000 7fd055faa640 1 --2- 172.21.15.83:0/3019824109 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0500b4460 0x7fd050155440 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fd04c034040 tx=0x7fd04c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.256+0000 7fd0467fc640 1 -- 172.21.15.83:0/3019824109 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd04c040070 con 0x7fd0500b4460 2024-03-06T07:25:40.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.256+0000 7fd0467fc640 1 -- 172.21.15.83:0/3019824109 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd04c030d20 con 0x7fd0500b4460 2024-03-06T07:25:40.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.256+0000 7fd05820e640 1 -- 172.21.15.83:0/3019824109 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0500c7940 con 0x7fd0500b4460 2024-03-06T07:25:40.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.257+0000 7fd0467fc640 1 -- 172.21.15.83:0/3019824109 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd04c035040 con 0x7fd0500b4460 2024-03-06T07:25:40.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.257+0000 7fd05820e640 1 -- 172.21.15.83:0/3019824109 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0500c7be0 con 0x7fd0500b4460 2024-03-06T07:25:40.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.258+0000 7fd05820e640 1 -- 172.21.15.83:0/3019824109 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd05004f330 con 0x7fd0500b4460 2024-03-06T07:25:40.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.258+0000 7fd0467fc640 1 -- 172.21.15.83:0/3019824109 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd04c04b060 con 0x7fd0500b4460 2024-03-06T07:25:40.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.258+0000 7fd0467fc640 1 --2- 172.21.15.83:0/3019824109 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd03406b4d0 0x7fd03406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.259+0000 7fd0467fc640 1 -- 172.21.15.83:0/3019824109 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd04c0ac0a0 con 0x7fd0500b4460 2024-03-06T07:25:40.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.259+0000 7fd0557a9640 1 --2- 172.21.15.83:0/3019824109 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd03406b4d0 0x7fd03406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.259+0000 7fd0557a9640 1 --2- 172.21.15.83:0/3019824109 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd03406b4d0 0x7fd03406d990 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fd0500cd860 tx=0x7fd040009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.263+0000 7fd0467fc640 1 -- 172.21.15.83:0/3019824109 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd04c02d040 con 0x7fd0500b4460 2024-03-06T07:25:40.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.531+0000 7fd05820e640 1 -- 172.21.15.83:0/3019824109 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd05006be40 con 0x7fd0500b4460 2024-03-06T07:25:40.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.531+0000 7fd0467fc640 1 -- 172.21.15.83:0/3019824109 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v70) v1 ==== 80+0+0 (secure 0 0 0) 0x7fd04c07ae60 con 0x7fd0500b4460 2024-03-06T07:25:40.533 INFO:teuthology.orchestra.run.smithi083.stderr:listed 0 entries 2024-03-06T07:25:40.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.533+0000 7fd05820e640 1 -- 172.21.15.83:0/3019824109 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd03406b4d0 msgr2=0x7fd03406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.533+0000 7fd05820e640 1 --2- 172.21.15.83:0/3019824109 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd03406b4d0 0x7fd03406d990 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fd0500cd860 tx=0x7fd040009040 comp rx=0 tx=0).stop 2024-03-06T07:25:40.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.533+0000 7fd05820e640 1 -- 172.21.15.83:0/3019824109 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0500b4460 msgr2=0x7fd050155440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.533+0000 7fd05820e640 1 --2- 172.21.15.83:0/3019824109 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0500b4460 0x7fd050155440 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fd04c034040 tx=0x7fd04c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.533+0000 7fd05820e640 1 -- 172.21.15.83:0/3019824109 shutdown_connections 2024-03-06T07:25:40.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.533+0000 7fd05820e640 1 --2- 172.21.15.83:0/3019824109 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd03406b4d0 0x7fd03406d990 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.533+0000 7fd05820e640 1 --2- 172.21.15.83:0/3019824109 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd050162970 0x7fd0500b1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.533+0000 7fd05820e640 1 --2- 172.21.15.83:0/3019824109 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd050161cf0 0x7fd0500c83e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.533+0000 7fd05820e640 1 --2- 172.21.15.83:0/3019824109 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0500b4460 0x7fd050155440 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.533+0000 7fd05820e640 1 -- 172.21.15.83:0/3019824109 >> 172.21.15.83:0/3019824109 conn(0x7fd0500d3ce0 msgr2=0x7fd0500d0700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:40.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.533+0000 7fd05820e640 1 -- 172.21.15.83:0/3019824109 shutdown_connections 2024-03-06T07:25:40.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:40.533+0000 7fd05820e640 1 -- 172.21.15.83:0/3019824109 wait complete. 2024-03-06T07:25:40.545 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi083.front.sepia.ceph.com... 2024-03-06T07:25:40.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:40.546 DEBUG:teuthology.orchestra.run.smithi083:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:40.571 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:ceph-fuse[63034]: fuse finished with error 0 and tester_r 0 2024-03-06T07:25:40.572 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi083.front.sepia.ceph.com 2024-03-06T07:25:40.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:40.573 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:25:40.738 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:25:40.738 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:40.756 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=287 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:25:40.756 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi183.front.sepia.ceph.com 2024-03-06T07:25:40.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:40.757 DEBUG:teuthology.orchestra.run.smithi183:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:25:40.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:25:40.813 INFO:teuthology.orchestra.run.smithi183.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:25:40.813 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:25:40.813 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:40.829 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1257 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:25:40.829 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi083.front.sepia.ceph.com 2024-03-06T07:25:40.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:40.829 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:25:40.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:25:40.884 INFO:teuthology.orchestra.run.smithi083.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:25:40.884 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:25:40.884 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:40.901 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=287 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:25:40.901 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi183.front.sepia.ceph.com 2024-03-06T07:25:40.902 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:40.902 DEBUG:teuthology.orchestra.run.smithi183:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:25:40.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:25:40.957 INFO:teuthology.orchestra.run.smithi183.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:25:40.958 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:41.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.056+0000 7f0f979ac640 1 Processor -- start 2024-03-06T07:25:41.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.057+0000 7f0f979ac640 1 -- start start 2024-03-06T07:25:41.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.057+0000 7f0f979ac640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0f9015dfe0 0x7f0f900cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.057+0000 7f0f979ac640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0f900d29c0 0x7f0f900b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.057+0000 7f0f979ac640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f900b3430 con 0x7f0f900cf830 2024-03-06T07:25:41.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.057+0000 7f0f979ac640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f900b31e0 con 0x7f0f900d29c0 2024-03-06T07:25:41.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.057+0000 7f0f979ac640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f900b3d60 con 0x7f0f9015dfe0 2024-03-06T07:25:41.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.058+0000 7f0f95f49640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0f900d29c0 0x7f0f900b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.058+0000 7f0f94f47640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f0f900cf830 0x7f0f900d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52524/0 (socket says 172.21.15.83:52524) 2024-03-06T07:25:41.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.058+0000 7f0f94f47640 1 -- 172.21.15.83:0/1609511409 learned_addr learned my addr 172.21.15.83:0/1609511409 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:25:41.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.058+0000 7f0f95748640 1 --2- 172.21.15.83:0/1609511409 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0f9015dfe0 0x7f0f900cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.058+0000 7f0f95f49640 1 -- 172.21.15.83:0/1609511409 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0f9015dfe0 msgr2=0x7f0f900cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:41.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.058+0000 7f0f95f49640 1 --2- 172.21.15.83:0/1609511409 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0f9015dfe0 0x7f0f900cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.058+0000 7f0f95f49640 1 -- 172.21.15.83:0/1609511409 >> v1:172.21.15.83:6789/0 conn(0x7f0f900cf830 legacy=0x7f0f900d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.058+0000 7f0f95f49640 1 -- 172.21.15.83:0/1609511409 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f900b3710 con 0x7f0f900d29c0 2024-03-06T07:25:41.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.058+0000 7f0f87fff640 1 -- 172.21.15.83:0/1609511409 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1923610778 0 0) 0x7f0f900b3430 con 0x7f0f900cf830 2024-03-06T07:25:41.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.059+0000 7f0f95f49640 1 --2- 172.21.15.83:0/1609511409 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0f900d29c0 0x7f0f900b41f0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f0f88009d90 tx=0x7f0f8802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c91da77e4f4eda5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.059+0000 7f0f87fff640 1 -- 172.21.15.83:0/1609511409 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f88039070 con 0x7f0f900d29c0 2024-03-06T07:25:41.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.059+0000 7f0f87fff640 1 -- 172.21.15.83:0/1609511409 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f8802cc50 con 0x7f0f900d29c0 2024-03-06T07:25:41.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.059+0000 7f0f87fff640 1 -- 172.21.15.83:0/1609511409 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f88034440 con 0x7f0f900d29c0 2024-03-06T07:25:41.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.059+0000 7f0f979ac640 1 -- 172.21.15.83:0/1609511409 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0f900d29c0 msgr2=0x7f0f900b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.059+0000 7f0f979ac640 1 --2- 172.21.15.83:0/1609511409 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0f900d29c0 0x7f0f900b41f0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f0f88009d90 tx=0x7f0f8802bea0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.059+0000 7f0f979ac640 1 -- 172.21.15.83:0/1609511409 shutdown_connections 2024-03-06T07:25:41.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.059+0000 7f0f979ac640 1 --2- 172.21.15.83:0/1609511409 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0f900d29c0 0x7f0f900b41f0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.059+0000 7f0f979ac640 1 --2- 172.21.15.83:0/1609511409 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0f9015dfe0 0x7f0f900cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.059+0000 7f0f979ac640 1 -- 172.21.15.83:0/1609511409 >> 172.21.15.83:0/1609511409 conn(0x7f0f900d3cd0 msgr2=0x7f0f900d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:41.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.059+0000 7f0f979ac640 1 -- 172.21.15.83:0/1609511409 shutdown_connections 2024-03-06T07:25:41.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.059+0000 7f0f979ac640 1 -- 172.21.15.83:0/1609511409 wait complete. 2024-03-06T07:25:41.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.061+0000 7f0f979ac640 1 Processor -- start 2024-03-06T07:25:41.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.061+0000 7f0f979ac640 1 -- start start 2024-03-06T07:25:41.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.062+0000 7f0f979ac640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0f900cf830 0x7f0f900ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.062+0000 7f0f979ac640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f900d29c0 0x7f0f9015f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.062+0000 7f0f979ac640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0f9015dfe0 0x7f0f90150c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.062+0000 7f0f979ac640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f900d1df0 con 0x7f0f900cf830 2024-03-06T07:25:41.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.062+0000 7f0f979ac640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f900d1c70 con 0x7f0f900d29c0 2024-03-06T07:25:41.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.062+0000 7f0f979ac640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9015e3c0 con 0x7f0f9015dfe0 2024-03-06T07:25:41.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.063+0000 7f0f95748640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0f900cf830 0x7f0f900ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.063+0000 7f0f95748640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0f900cf830 0x7f0f900ccb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52636/0 (socket says 172.21.15.83:52636) 2024-03-06T07:25:41.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.063+0000 7f0f95748640 1 -- 172.21.15.83:0/2567429216 learned_addr learned my addr 172.21.15.83:0/2567429216 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:41.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.063+0000 7f0f95f49640 1 --2- 172.21.15.83:0/2567429216 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0f9015dfe0 0x7f0f90150c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.063+0000 7f0f94f47640 1 --2- 172.21.15.83:0/2567429216 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f900d29c0 0x7f0f9015f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.063+0000 7f0f95748640 1 -- 172.21.15.83:0/2567429216 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0f9015dfe0 msgr2=0x7f0f90150c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.063+0000 7f0f95748640 1 --2- 172.21.15.83:0/2567429216 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0f9015dfe0 0x7f0f90150c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.063+0000 7f0f95748640 1 -- 172.21.15.83:0/2567429216 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f900d29c0 msgr2=0x7f0f9015f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.063+0000 7f0f95748640 1 --2- 172.21.15.83:0/2567429216 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f900d29c0 0x7f0f9015f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.063+0000 7f0f95748640 1 -- 172.21.15.83:0/2567429216 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f9015e880 con 0x7f0f900cf830 2024-03-06T07:25:41.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.063+0000 7f0f94f47640 1 --2- 172.21.15.83:0/2567429216 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f900d29c0 0x7f0f9015f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:41.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.063+0000 7f0f95f49640 1 --2- 172.21.15.83:0/2567429216 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0f9015dfe0 0x7f0f90150c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:41.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.063+0000 7f0f95748640 1 --2- 172.21.15.83:0/2567429216 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0f900cf830 0x7f0f900ccb50 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c00d5f0 tx=0x7f0f8c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.063+0000 7f0f85ffb640 1 -- 172.21.15.83:0/2567429216 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f8c0125f0 con 0x7f0f900cf830 2024-03-06T07:25:41.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.063+0000 7f0f85ffb640 1 -- 172.21.15.83:0/2567429216 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f8c019070 con 0x7f0f900cf830 2024-03-06T07:25:41.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.063+0000 7f0f85ffb640 1 -- 172.21.15.83:0/2567429216 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f8c0129d0 con 0x7f0f900cf830 2024-03-06T07:25:41.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.063+0000 7f0f979ac640 1 -- 172.21.15.83:0/2567429216 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f9015eb60 con 0x7f0f900cf830 2024-03-06T07:25:41.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.063+0000 7f0f979ac640 1 -- 172.21.15.83:0/2567429216 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f9015f120 con 0x7f0f900cf830 2024-03-06T07:25:41.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.064+0000 7f0f979ac640 1 -- 172.21.15.83:0/2567429216 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f9004f330 con 0x7f0f900cf830 2024-03-06T07:25:41.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.065+0000 7f0f85ffb640 1 -- 172.21.15.83:0/2567429216 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0f8c012b70 con 0x7f0f900cf830 2024-03-06T07:25:41.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.065+0000 7f0f85ffb640 1 --2- 172.21.15.83:0/2567429216 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0f7406b4d0 0x7f0f7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.065+0000 7f0f85ffb640 1 -- 172.21.15.83:0/2567429216 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0f8c08a5a0 con 0x7f0f900cf830 2024-03-06T07:25:41.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.066+0000 7f0f94f47640 1 --2- 172.21.15.83:0/2567429216 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0f7406b4d0 0x7f0f7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.066+0000 7f0f94f47640 1 --2- 172.21.15.83:0/2567429216 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0f7406b4d0 0x7f0f7406d990 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f0f9006a1b0 tx=0x7f0f80009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.070+0000 7f0f85ffb640 1 -- 172.21.15.83:0/2567429216 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0f8c059470 con 0x7f0f900cf830 2024-03-06T07:25:41.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.313+0000 7f0f979ac640 1 -- 172.21.15.83:0/2567429216 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f9015cfc0 con 0x7f0f900cf830 2024-03-06T07:25:41.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.313+0000 7f0f85ffb640 1 -- 172.21.15.83:0/2567429216 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f0f8c058d70 con 0x7f0f900cf830 2024-03-06T07:25:41.315 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:25:41.315 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":32,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6010,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/2142942385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2142942385},{"type":"v1","addr":"172.21.15.183:6834","nonce":2142942385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":6013,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3742438064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":3742438064},{"type":"v1","addr":"172.21.15.183:6835","nonce":3742438064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":6540,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/1549894851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1549894851},{"type":"v1","addr":"172.21.15.83:6836","nonce":1549894851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:12.481670+0000","modified":"2024-03-06T07:25:15.392072+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6543},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6543":{"gid":6543,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/602775648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":602775648},{"type":"v1","addr":"172.21.15.83:6837","nonce":602775648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6543,"qdb_cluster":[6543]},"id":5}]} 2024-03-06T07:25:41.316 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 32 2024-03-06T07:25:41.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.315+0000 7f0f979ac640 1 -- 172.21.15.83:0/2567429216 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0f7406b4d0 msgr2=0x7f0f7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.315+0000 7f0f979ac640 1 --2- 172.21.15.83:0/2567429216 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0f7406b4d0 0x7f0f7406d990 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f0f9006a1b0 tx=0x7f0f80009040 comp rx=0 tx=0).stop 2024-03-06T07:25:41.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.315+0000 7f0f979ac640 1 -- 172.21.15.83:0/2567429216 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0f900cf830 msgr2=0x7f0f900ccb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.315+0000 7f0f979ac640 1 --2- 172.21.15.83:0/2567429216 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0f900cf830 0x7f0f900ccb50 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c00d5f0 tx=0x7f0f8c00dab0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.315+0000 7f0f979ac640 1 -- 172.21.15.83:0/2567429216 shutdown_connections 2024-03-06T07:25:41.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.315+0000 7f0f979ac640 1 --2- 172.21.15.83:0/2567429216 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0f7406b4d0 0x7f0f7406d990 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.315+0000 7f0f979ac640 1 --2- 172.21.15.83:0/2567429216 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0f9015dfe0 0x7f0f90150c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.315+0000 7f0f979ac640 1 --2- 172.21.15.83:0/2567429216 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f900d29c0 0x7f0f9015f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.315+0000 7f0f979ac640 1 --2- 172.21.15.83:0/2567429216 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0f900cf830 0x7f0f900ccb50 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.315+0000 7f0f979ac640 1 -- 172.21.15.83:0/2567429216 >> 172.21.15.83:0/2567429216 conn(0x7f0f900d3cd0 msgr2=0x7f0f900d22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:41.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.315+0000 7f0f979ac640 1 -- 172.21.15.83:0/2567429216 shutdown_connections 2024-03-06T07:25:41.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.315+0000 7f0f979ac640 1 -- 172.21.15.83:0/2567429216 wait complete. 2024-03-06T07:25:41.331 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:41.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.430+0000 7f5b6844e640 1 Processor -- start 2024-03-06T07:25:41.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.430+0000 7f5b6844e640 1 -- start start 2024-03-06T07:25:41.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.431+0000 7f5b6844e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5b600d2710 0x7f5b600b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.431+0000 7f5b6844e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b600b3430 con 0x7f5b600d22b0 2024-03-06T07:25:41.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.431+0000 7f5b6844e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b600b3710 con 0x7f5b6015dfe0 2024-03-06T07:25:41.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.431+0000 7f5b6844e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b600b31e0 con 0x7f5b600d2710 2024-03-06T07:25:41.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.431+0000 7f5b669eb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5b600d2710 0x7f5b600b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.431+0000 7f5b659e9640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f5b600d22b0 0x7f5b600b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52536/0 (socket says 172.21.15.83:52536) 2024-03-06T07:25:41.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.431+0000 7f5b669eb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5b600d2710 0x7f5b600b4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52650/0 (socket says 172.21.15.83:52650) 2024-03-06T07:25:41.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.431+0000 7f5b659e9640 1 -- 172.21.15.83:0/2294878894 learned_addr learned my addr 172.21.15.83:0/2294878894 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:25:41.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.432+0000 7f5b669eb640 1 -- 172.21.15.83:0/2294878894 >> v1:172.21.15.183:6789/0 conn(0x7f5b6015dfe0 legacy=0x7f5b600cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:41.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.432+0000 7f5b669eb640 1 -- 172.21.15.83:0/2294878894 >> v1:172.21.15.83:6789/0 conn(0x7f5b600d22b0 legacy=0x7f5b600b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.432+0000 7f5b651e8640 1 -- 172.21.15.83:0/2294878894 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1242700583 0 0) 0x7f5b600b3430 con 0x7f5b600d22b0 2024-03-06T07:25:41.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.432+0000 7f5b669eb640 1 -- 172.21.15.83:0/2294878894 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b600b39f0 con 0x7f5b600d2710 2024-03-06T07:25:41.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.433+0000 7f5b669eb640 1 --2- 172.21.15.83:0/2294878894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5b600d2710 0x7f5b600b4900 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f5b58009dc0 tx=0x7f5b5802be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b28cfff3c3e490e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.433+0000 7f5b651e8640 1 -- 172.21.15.83:0/2294878894 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b58039070 con 0x7f5b600d2710 2024-03-06T07:25:41.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.433+0000 7f5b651e8640 1 -- 172.21.15.83:0/2294878894 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b5802cc40 con 0x7f5b600d2710 2024-03-06T07:25:41.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.433+0000 7f5b651e8640 1 -- 172.21.15.83:0/2294878894 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b58034440 con 0x7f5b600d2710 2024-03-06T07:25:41.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.433+0000 7f5b6844e640 1 -- 172.21.15.83:0/2294878894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5b600d2710 msgr2=0x7f5b600b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.433+0000 7f5b6844e640 1 --2- 172.21.15.83:0/2294878894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5b600d2710 0x7f5b600b4900 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f5b58009dc0 tx=0x7f5b5802be90 comp rx=0 tx=0).stop 2024-03-06T07:25:41.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.434+0000 7f5b6844e640 1 -- 172.21.15.83:0/2294878894 shutdown_connections 2024-03-06T07:25:41.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.434+0000 7f5b6844e640 1 --2- 172.21.15.83:0/2294878894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5b600d2710 0x7f5b600b4900 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.434+0000 7f5b6844e640 1 -- 172.21.15.83:0/2294878894 >> 172.21.15.83:0/2294878894 conn(0x7f5b600d3cd0 msgr2=0x7f5b600d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:41.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.434+0000 7f5b6844e640 1 -- 172.21.15.83:0/2294878894 shutdown_connections 2024-03-06T07:25:41.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.434+0000 7f5b6844e640 1 -- 172.21.15.83:0/2294878894 wait complete. 2024-03-06T07:25:41.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.435+0000 7f5b6844e640 1 Processor -- start 2024-03-06T07:25:41.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.435+0000 7f5b6844e640 1 -- start start 2024-03-06T07:25:41.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.436+0000 7f5b6844e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5b600d22b0 0x7f5b600b8db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.436+0000 7f5b6844e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b600d2710 0x7f5b601f6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.436+0000 7f5b6844e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5b6015dfe0 0x7f5b60159510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.436+0000 7f5b6844e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b600cfa00 con 0x7f5b600d22b0 2024-03-06T07:25:41.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.436+0000 7f5b6844e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b600b3d60 con 0x7f5b600d2710 2024-03-06T07:25:41.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.436+0000 7f5b6844e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b600d1c70 con 0x7f5b6015dfe0 2024-03-06T07:25:41.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.436+0000 7f5b661ea640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5b600d22b0 0x7f5b600b8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.436+0000 7f5b661ea640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5b600d22b0 0x7f5b600b8db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52662/0 (socket says 172.21.15.83:52662) 2024-03-06T07:25:41.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.436+0000 7f5b661ea640 1 -- 172.21.15.83:0/1044406873 learned_addr learned my addr 172.21.15.83:0/1044406873 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:41.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.436+0000 7f5b669eb640 1 --2- 172.21.15.83:0/1044406873 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5b6015dfe0 0x7f5b60159510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.436+0000 7f5b659e9640 1 --2- 172.21.15.83:0/1044406873 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b600d2710 0x7f5b601f6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.436+0000 7f5b661ea640 1 -- 172.21.15.83:0/1044406873 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5b6015dfe0 msgr2=0x7f5b60159510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.436+0000 7f5b661ea640 1 --2- 172.21.15.83:0/1044406873 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5b6015dfe0 0x7f5b60159510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.436+0000 7f5b661ea640 1 -- 172.21.15.83:0/1044406873 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b600d2710 msgr2=0x7f5b601f6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.436+0000 7f5b661ea640 1 --2- 172.21.15.83:0/1044406873 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b600d2710 0x7f5b601f6f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.436+0000 7f5b661ea640 1 -- 172.21.15.83:0/1044406873 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b600b68d0 con 0x7f5b600d22b0 2024-03-06T07:25:41.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.436+0000 7f5b661ea640 1 --2- 172.21.15.83:0/1044406873 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5b600d22b0 0x7f5b600b8db0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c00b3f0 tx=0x7f5b5c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.436+0000 7f5b56ffd640 1 -- 172.21.15.83:0/1044406873 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5c015020 con 0x7f5b600d22b0 2024-03-06T07:25:41.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.436+0000 7f5b56ffd640 1 -- 172.21.15.83:0/1044406873 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b5c011830 con 0x7f5b600d22b0 2024-03-06T07:25:41.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.436+0000 7f5b6844e640 1 -- 172.21.15.83:0/1044406873 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b601f7a90 con 0x7f5b600d22b0 2024-03-06T07:25:41.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.436+0000 7f5b56ffd640 1 -- 172.21.15.83:0/1044406873 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5c011c20 con 0x7f5b600d22b0 2024-03-06T07:25:41.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.436+0000 7f5b6844e640 1 -- 172.21.15.83:0/1044406873 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b600b6e50 con 0x7f5b600d22b0 2024-03-06T07:25:41.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.437+0000 7f5b6844e640 1 -- 172.21.15.83:0/1044406873 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b600b7050 con 0x7f5b600d22b0 2024-03-06T07:25:41.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.438+0000 7f5b56ffd640 1 -- 172.21.15.83:0/1044406873 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5b5c018020 con 0x7f5b600d22b0 2024-03-06T07:25:41.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.438+0000 7f5b56ffd640 1 --2- 172.21.15.83:0/1044406873 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5b4406b4d0 0x7f5b4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.438+0000 7f5b659e9640 1 --2- 172.21.15.83:0/1044406873 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5b4406b4d0 0x7f5b4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.438+0000 7f5b56ffd640 1 -- 172.21.15.83:0/1044406873 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5b5c089e40 con 0x7f5b600d22b0 2024-03-06T07:25:41.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.439+0000 7f5b659e9640 1 --2- 172.21.15.83:0/1044406873 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5b4406b4d0 0x7f5b4406d990 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f5b6015c9e0 tx=0x7f5b50007b50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.443+0000 7f5b56ffd640 1 -- 172.21.15.83:0/1044406873 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5b600b7050 con 0x7f5b600d22b0 2024-03-06T07:25:41.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.661+0000 7f5b6844e640 1 -- 172.21.15.83:0/1044406873 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b600cec80 con 0x7f5b600d22b0 2024-03-06T07:25:41.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.662+0000 7f5b56ffd640 1 -- 172.21.15.83:0/1044406873 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+16778 (secure 0 0 0) 0x7f5b5c01f030 con 0x7f5b600d22b0 2024-03-06T07:25:41.665 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:25:41.665 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":71,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:25:40.924823+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:10.701416+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:11.450797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.1e","mappings":[{"from":6,"to":0}]},{"pgid":"11.3e","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:41.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.665+0000 7f5b6844e640 1 -- 172.21.15.83:0/1044406873 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5b4406b4d0 msgr2=0x7f5b4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.665+0000 7f5b6844e640 1 --2- 172.21.15.83:0/1044406873 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5b4406b4d0 0x7f5b4406d990 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f5b6015c9e0 tx=0x7f5b50007b50 comp rx=0 tx=0).stop 2024-03-06T07:25:41.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.665+0000 7f5b6844e640 1 -- 172.21.15.83:0/1044406873 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5b600d22b0 msgr2=0x7f5b600b8db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.665+0000 7f5b6844e640 1 --2- 172.21.15.83:0/1044406873 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5b600d22b0 0x7f5b600b8db0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c00b3f0 tx=0x7f5b5c00b8b0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.665+0000 7f5b6844e640 1 -- 172.21.15.83:0/1044406873 shutdown_connections 2024-03-06T07:25:41.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.665+0000 7f5b6844e640 1 --2- 172.21.15.83:0/1044406873 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5b4406b4d0 0x7f5b4406d990 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.665+0000 7f5b6844e640 1 --2- 172.21.15.83:0/1044406873 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5b6015dfe0 0x7f5b60159510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.665+0000 7f5b6844e640 1 --2- 172.21.15.83:0/1044406873 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b600d2710 0x7f5b601f6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.665+0000 7f5b6844e640 1 --2- 172.21.15.83:0/1044406873 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5b600d22b0 0x7f5b600b8db0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.665+0000 7f5b6844e640 1 -- 172.21.15.83:0/1044406873 >> 172.21.15.83:0/1044406873 conn(0x7f5b600d3cd0 msgr2=0x7f5b600cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:41.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.665+0000 7f5b6844e640 1 -- 172.21.15.83:0/1044406873 shutdown_connections 2024-03-06T07:25:41.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.665+0000 7f5b6844e640 1 -- 172.21.15.83:0/1044406873 wait complete. 2024-03-06T07:25:41.679 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:25:10.701416+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:25:11.450797+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:25:41.679 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:25:41.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.782+0000 7fb0756b5640 1 Processor -- start 2024-03-06T07:25:41.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.782+0000 7fb0756b5640 1 -- start start 2024-03-06T07:25:41.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.783+0000 7fb0756b5640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb0701586e0 0x7fb0700aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.783+0000 7fb0756b5640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fb0700aef80 0x7fb0700cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.783+0000 7fb0756b5640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb070158ac0 con 0x7fb0700cffd0 2024-03-06T07:25:41.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.783+0000 7fb0756b5640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb07015b2a0 con 0x7fb0701586e0 2024-03-06T07:25:41.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.783+0000 7fb0756b5640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb07015b540 con 0x7fb0700aef80 2024-03-06T07:25:41.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.783+0000 7fb06effd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb0701586e0 0x7fb0700aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.783+0000 7fb06effd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb0701586e0 0x7fb0700aea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52674/0 (socket says 172.21.15.83:52674) 2024-03-06T07:25:41.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.783+0000 7fb06effd640 1 -- 172.21.15.83:0/3338597546 learned_addr learned my addr 172.21.15.83:0/3338597546 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:41.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.783+0000 7fb06f7fe640 1 --1- 172.21.15.83:0/3338597546 >> v1:172.21.15.83:6789/0 conn(0x7fb0700cffd0 0x7fb07006da20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52544/0 (socket says 172.21.15.83:52544) 2024-03-06T07:25:41.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.783+0000 7fb067fff640 1 --2- 172.21.15.83:0/3338597546 >> v2:172.21.15.183:3300/0 conn(0x7fb0700aef80 0x7fb0700cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.783+0000 7fb06effd640 1 -- 172.21.15.83:0/3338597546 >> v2:172.21.15.183:3300/0 conn(0x7fb0700aef80 msgr2=0x7fb0700cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:41.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.783+0000 7fb06effd640 1 --2- 172.21.15.83:0/3338597546 >> v2:172.21.15.183:3300/0 conn(0x7fb0700aef80 0x7fb0700cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.783+0000 7fb06effd640 1 -- 172.21.15.83:0/3338597546 >> v1:172.21.15.83:6789/0 conn(0x7fb0700cffd0 legacy=0x7fb07006da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.783+0000 7fb06effd640 1 -- 172.21.15.83:0/3338597546 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb07015b820 con 0x7fb0701586e0 2024-03-06T07:25:41.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.783+0000 7fb06e7fc640 1 -- 172.21.15.83:0/3338597546 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4032875201 0 0) 0x7fb070158ac0 con 0x7fb0700cffd0 2024-03-06T07:25:41.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.784+0000 7fb06effd640 1 --2- 172.21.15.83:0/3338597546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb0701586e0 0x7fb0700aea40 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fb070072e30 tx=0x7fb06002e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2fc6e6f89801ed server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.784+0000 7fb06e7fc640 1 -- 172.21.15.83:0/3338597546 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb06003b070 con 0x7fb0701586e0 2024-03-06T07:25:41.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.784+0000 7fb06e7fc640 1 -- 172.21.15.83:0/3338597546 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb060036440 con 0x7fb0701586e0 2024-03-06T07:25:41.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.784+0000 7fb06e7fc640 1 -- 172.21.15.83:0/3338597546 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb060036810 con 0x7fb0701586e0 2024-03-06T07:25:41.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.785+0000 7fb0756b5640 1 -- 172.21.15.83:0/3338597546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb0701586e0 msgr2=0x7fb0700aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.785+0000 7fb0756b5640 1 --2- 172.21.15.83:0/3338597546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb0701586e0 0x7fb0700aea40 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fb070072e30 tx=0x7fb06002e170 comp rx=0 tx=0).stop 2024-03-06T07:25:41.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.785+0000 7fb0756b5640 1 -- 172.21.15.83:0/3338597546 shutdown_connections 2024-03-06T07:25:41.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.785+0000 7fb0756b5640 1 --2- 172.21.15.83:0/3338597546 >> v2:172.21.15.183:3300/0 conn(0x7fb0700aef80 0x7fb0700cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.785+0000 7fb0756b5640 1 --2- 172.21.15.83:0/3338597546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb0701586e0 0x7fb0700aea40 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.785+0000 7fb0756b5640 1 -- 172.21.15.83:0/3338597546 >> 172.21.15.83:0/3338597546 conn(0x7fb07006b190 msgr2=0x7fb07006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:41.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.785+0000 7fb0756b5640 1 -- 172.21.15.83:0/3338597546 shutdown_connections 2024-03-06T07:25:41.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.785+0000 7fb0756b5640 1 -- 172.21.15.83:0/3338597546 wait complete. 2024-03-06T07:25:41.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.787+0000 7fb0756b5640 1 Processor -- start 2024-03-06T07:25:41.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.787+0000 7fb0756b5640 1 -- start start 2024-03-06T07:25:41.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.788+0000 7fb0756b5640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb0700aef80 0x7fb0700cd5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.788+0000 7fb0756b5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb0700cffd0 0x7fb070152580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.788+0000 7fb0756b5640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0701586e0 0x7fb070152ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.788+0000 7fb0756b5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb070071350 con 0x7fb0700cffd0 2024-03-06T07:25:41.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.788+0000 7fb0756b5640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07015bde0 con 0x7fb0701586e0 2024-03-06T07:25:41.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.788+0000 7fb0756b5640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0700714d0 con 0x7fb0700aef80 2024-03-06T07:25:41.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.788+0000 7fb067fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb0700cffd0 0x7fb070152580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.788+0000 7fb067fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb0700cffd0 0x7fb070152580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52676/0 (socket says 172.21.15.83:52676) 2024-03-06T07:25:41.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.788+0000 7fb067fff640 1 -- 172.21.15.83:0/3712364469 learned_addr learned my addr 172.21.15.83:0/3712364469 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:41.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.788+0000 7fb06effd640 1 --2- 172.21.15.83:0/3712364469 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb0700aef80 0x7fb0700cd5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.788+0000 7fb06f7fe640 1 --2- 172.21.15.83:0/3712364469 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0701586e0 0x7fb070152ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.788+0000 7fb067fff640 1 -- 172.21.15.83:0/3712364469 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb0700aef80 msgr2=0x7fb0700cd5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.788+0000 7fb067fff640 1 --2- 172.21.15.83:0/3712364469 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb0700aef80 0x7fb0700cd5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.788+0000 7fb067fff640 1 -- 172.21.15.83:0/3712364469 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0701586e0 msgr2=0x7fb070152ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.788+0000 7fb067fff640 1 --2- 172.21.15.83:0/3712364469 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0701586e0 0x7fb070152ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.788+0000 7fb067fff640 1 -- 172.21.15.83:0/3712364469 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0701f2e30 con 0x7fb0700cffd0 2024-03-06T07:25:41.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.788+0000 7fb06effd640 1 --2- 172.21.15.83:0/3712364469 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb0700aef80 0x7fb0700cd5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:41.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.788+0000 7fb06f7fe640 1 --2- 172.21.15.83:0/3712364469 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0701586e0 0x7fb070152ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:41.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.789+0000 7fb067fff640 1 --2- 172.21.15.83:0/3712364469 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb0700cffd0 0x7fb070152580 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fb05800d7d0 tx=0x7fb05800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.789+0000 7fb0677fe640 1 -- 172.21.15.83:0/3712364469 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05800b6c0 con 0x7fb0700cffd0 2024-03-06T07:25:41.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.789+0000 7fb0677fe640 1 -- 172.21.15.83:0/3712364469 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb058019070 con 0x7fb0700cffd0 2024-03-06T07:25:41.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.789+0000 7fb0756b5640 1 -- 172.21.15.83:0/3712364469 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0701f3110 con 0x7fb0700cffd0 2024-03-06T07:25:41.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.789+0000 7fb0677fe640 1 -- 172.21.15.83:0/3712364469 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05800ba80 con 0x7fb0700cffd0 2024-03-06T07:25:41.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.789+0000 7fb0756b5640 1 -- 172.21.15.83:0/3712364469 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0701f36d0 con 0x7fb0700cffd0 2024-03-06T07:25:41.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.790+0000 7fb0756b5640 1 -- 172.21.15.83:0/3712364469 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0700b4540 con 0x7fb0700cffd0 2024-03-06T07:25:41.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.790+0000 7fb0677fe640 1 -- 172.21.15.83:0/3712364469 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb05800bc20 con 0x7fb0700cffd0 2024-03-06T07:25:41.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.791+0000 7fb0677fe640 1 --2- 172.21.15.83:0/3712364469 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb05006b4d0 0x7fb05006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.791+0000 7fb06effd640 1 --2- 172.21.15.83:0/3712364469 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb05006b4d0 0x7fb05006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.791+0000 7fb0677fe640 1 -- 172.21.15.83:0/3712364469 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb05808a6a0 con 0x7fb0700cffd0 2024-03-06T07:25:41.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.791+0000 7fb06effd640 1 --2- 172.21.15.83:0/3712364469 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb05006b4d0 0x7fb05006d990 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fb060038000 tx=0x7fb060032000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.795+0000 7fb0677fe640 1 -- 172.21.15.83:0/3712364469 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb0700b4540 con 0x7fb0700cffd0 2024-03-06T07:25:41.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.972+0000 7fb0756b5640 1 -- 172.21.15.83:0/3712364469 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb0700732b0 con 0x7fb0700cffd0 2024-03-06T07:25:41.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.972+0000 7fb0677fe640 1 -- 172.21.15.83:0/3712364469 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v71) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb05801c360 con 0x7fb0700cffd0 2024-03-06T07:25:41.974 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:25:41.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.974+0000 7fb0756b5640 1 -- 172.21.15.83:0/3712364469 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb05006b4d0 msgr2=0x7fb05006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.974+0000 7fb0756b5640 1 --2- 172.21.15.83:0/3712364469 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb05006b4d0 0x7fb05006d990 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fb060038000 tx=0x7fb060032000 comp rx=0 tx=0).stop 2024-03-06T07:25:41.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.974+0000 7fb0756b5640 1 -- 172.21.15.83:0/3712364469 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb0700cffd0 msgr2=0x7fb070152580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.974+0000 7fb0756b5640 1 --2- 172.21.15.83:0/3712364469 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb0700cffd0 0x7fb070152580 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fb05800d7d0 tx=0x7fb05800dc90 comp rx=0 tx=0).stop 2024-03-06T07:25:41.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.974+0000 7fb0756b5640 1 -- 172.21.15.83:0/3712364469 shutdown_connections 2024-03-06T07:25:41.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.974+0000 7fb0756b5640 1 --2- 172.21.15.83:0/3712364469 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb05006b4d0 0x7fb05006d990 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.974+0000 7fb0756b5640 1 --2- 172.21.15.83:0/3712364469 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0701586e0 0x7fb070152ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.974+0000 7fb0756b5640 1 --2- 172.21.15.83:0/3712364469 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb0700cffd0 0x7fb070152580 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.974+0000 7fb0756b5640 1 --2- 172.21.15.83:0/3712364469 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb0700aef80 0x7fb0700cd5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.974+0000 7fb0756b5640 1 -- 172.21.15.83:0/3712364469 >> 172.21.15.83:0/3712364469 conn(0x7fb07006b190 msgr2=0x7fb070153b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:41.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.974+0000 7fb0756b5640 1 -- 172.21.15.83:0/3712364469 shutdown_connections 2024-03-06T07:25:41.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:41.974+0000 7fb0756b5640 1 -- 172.21.15.83:0/3712364469 wait complete. 2024-03-06T07:25:41.987 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:25:42.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.089+0000 7f78a4301640 1 Processor -- start 2024-03-06T07:25:42.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.089+0000 7f78a4301640 1 -- start start 2024-03-06T07:25:42.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.090+0000 7f78a4301640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f789c0d4860 0x7f789c0d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.090+0000 7f78a4301640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f789c0b0e30 0x7f789c0b1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.090+0000 7f78a4301640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f789c1618b0 con 0x7f789c1614d0 2024-03-06T07:25:42.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.090+0000 7f78a4301640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f789c0d4cf0 con 0x7f789c0b0e30 2024-03-06T07:25:42.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.090+0000 7f78a4301640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f789c0cf400 con 0x7f789c0d4860 2024-03-06T07:25:42.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.090+0000 7f78a289e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f789c0b0e30 0x7f789c0b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.090+0000 7f78a209d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f789c1614d0 0x7f789c0d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52548/0 (socket says 172.21.15.83:52548) 2024-03-06T07:25:42.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.090+0000 7f78a209d640 1 -- 172.21.15.83:0/467365254 learned_addr learned my addr 172.21.15.83:0/467365254 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:25:42.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.090+0000 7f78a189c640 1 --2- 172.21.15.83:0/467365254 >> v2:172.21.15.183:3300/0 conn(0x7f789c0d4860 0x7f789c0d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.090+0000 7f78a289e640 1 -- 172.21.15.83:0/467365254 >> v2:172.21.15.183:3300/0 conn(0x7f789c0d4860 msgr2=0x7f789c0d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:42.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.090+0000 7f78a289e640 1 --2- 172.21.15.83:0/467365254 >> v2:172.21.15.183:3300/0 conn(0x7f789c0d4860 0x7f789c0d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.090+0000 7f78a289e640 1 -- 172.21.15.83:0/467365254 >> v1:172.21.15.83:6789/0 conn(0x7f789c1614d0 legacy=0x7f789c0d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.090+0000 7f78a289e640 1 -- 172.21.15.83:0/467365254 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f789c0cf6a0 con 0x7f789c0b0e30 2024-03-06T07:25:42.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.090+0000 7f78a109b640 1 -- 172.21.15.83:0/467365254 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3804125170 0 0) 0x7f789c1618b0 con 0x7f789c1614d0 2024-03-06T07:25:42.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.091+0000 7f78a289e640 1 --2- 172.21.15.83:0/467365254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f789c0b0e30 0x7f789c0b1260 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f789c0ce550 tx=0x7f789402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9974033db93412df server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:42.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.091+0000 7f78a109b640 1 -- 172.21.15.83:0/467365254 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7894039070 con 0x7f789c0b0e30 2024-03-06T07:25:42.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.091+0000 7f78a109b640 1 -- 172.21.15.83:0/467365254 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f789402cc80 con 0x7f789c0b0e30 2024-03-06T07:25:42.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.091+0000 7f78a109b640 1 -- 172.21.15.83:0/467365254 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7894034660 con 0x7f789c0b0e30 2024-03-06T07:25:42.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.092+0000 7f78a4301640 1 -- 172.21.15.83:0/467365254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f789c0b0e30 msgr2=0x7f789c0b1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.092+0000 7f78a4301640 1 --2- 172.21.15.83:0/467365254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f789c0b0e30 0x7f789c0b1260 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f789c0ce550 tx=0x7f789402bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.092+0000 7f78a4301640 1 -- 172.21.15.83:0/467365254 shutdown_connections 2024-03-06T07:25:42.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.092+0000 7f78a4301640 1 --2- 172.21.15.83:0/467365254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f789c0b0e30 0x7f789c0b1260 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.092+0000 7f78a4301640 1 --2- 172.21.15.83:0/467365254 >> v2:172.21.15.183:3300/0 conn(0x7f789c0d4860 0x7f789c0d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.092+0000 7f78a4301640 1 -- 172.21.15.83:0/467365254 >> 172.21.15.83:0/467365254 conn(0x7f789c0d3d40 msgr2=0x7f789c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:42.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.092+0000 7f78a4301640 1 -- 172.21.15.83:0/467365254 shutdown_connections 2024-03-06T07:25:42.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.092+0000 7f78a4301640 1 -- 172.21.15.83:0/467365254 wait complete. 2024-03-06T07:25:42.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.094+0000 7f78a4301640 1 Processor -- start 2024-03-06T07:25:42.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.094+0000 7f78a4301640 1 -- start start 2024-03-06T07:25:42.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.095+0000 7f78a4301640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f789c0b0e30 0x7f789c1f7c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.095+0000 7f78a4301640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f789c0d4860 0x7f789c15a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.095+0000 7f78a4301640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f789c1614d0 0x7f789c0cc250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.095+0000 7f78a4301640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789c0b0910 con 0x7f789c0b0e30 2024-03-06T07:25:42.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.095+0000 7f78a4301640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789c0cfc60 con 0x7f789c0d4860 2024-03-06T07:25:42.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.095+0000 7f78a4301640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f789c0d2560 con 0x7f789c1614d0 2024-03-06T07:25:42.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.095+0000 7f78a209d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f789c0b0e30 0x7f789c1f7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.095+0000 7f78a209d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f789c0b0e30 0x7f789c1f7c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52692/0 (socket says 172.21.15.83:52692) 2024-03-06T07:25:42.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.095+0000 7f78a209d640 1 -- 172.21.15.83:0/366107947 learned_addr learned my addr 172.21.15.83:0/366107947 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:42.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.095+0000 7f78a209d640 1 -- 172.21.15.83:0/366107947 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f789c1614d0 msgr2=0x7f789c0cc250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.095+0000 7f78a209d640 1 --2- 172.21.15.83:0/366107947 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f789c1614d0 0x7f789c0cc250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.095+0000 7f78a209d640 1 -- 172.21.15.83:0/366107947 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f789c0d4860 msgr2=0x7f789c15a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.095+0000 7f78a209d640 1 --2- 172.21.15.83:0/366107947 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f789c0d4860 0x7f789c15a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.095+0000 7f78a209d640 1 -- 172.21.15.83:0/366107947 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f789c1f81a0 con 0x7f789c0b0e30 2024-03-06T07:25:42.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.095+0000 7f78a209d640 1 --2- 172.21.15.83:0/366107947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f789c0b0e30 0x7f789c1f7c60 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f788c00c400 tx=0x7f788c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:42.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.096+0000 7f789affd640 1 -- 172.21.15.83:0/366107947 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788c017070 con 0x7f789c0b0e30 2024-03-06T07:25:42.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.096+0000 7f789affd640 1 -- 172.21.15.83:0/366107947 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f788c005ce0 con 0x7f789c0b0e30 2024-03-06T07:25:42.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.096+0000 7f78a4301640 1 -- 172.21.15.83:0/366107947 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f789c0cb1b0 con 0x7f789c0b0e30 2024-03-06T07:25:42.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.096+0000 7f789affd640 1 -- 172.21.15.83:0/366107947 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788c0072f0 con 0x7f789c0b0e30 2024-03-06T07:25:42.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.096+0000 7f78a4301640 1 -- 172.21.15.83:0/366107947 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f789c0cb770 con 0x7f789c0b0e30 2024-03-06T07:25:42.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.097+0000 7f78a4301640 1 -- 172.21.15.83:0/366107947 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f789c06c070 con 0x7f789c0b0e30 2024-03-06T07:25:42.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.097+0000 7f789affd640 1 -- 172.21.15.83:0/366107947 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f788c01a070 con 0x7f789c0b0e30 2024-03-06T07:25:42.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.098+0000 7f789affd640 1 --2- 172.21.15.83:0/366107947 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f788806b500 0x7f788806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.098+0000 7f789affd640 1 -- 172.21.15.83:0/366107947 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f788c089630 con 0x7f789c0b0e30 2024-03-06T07:25:42.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.098+0000 7f78a189c640 1 --2- 172.21.15.83:0/366107947 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f788806b500 0x7f788806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.098+0000 7f78a189c640 1 --2- 172.21.15.83:0/366107947 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f788806b500 0x7f788806d9c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f7890006fd0 tx=0x7f7890008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:42.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.102+0000 7f789affd640 1 -- 172.21.15.83:0/366107947 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f789c06c070 con 0x7f789c0b0e30 2024-03-06T07:25:42.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.404+0000 7f78a4301640 1 -- 172.21.15.83:0/366107947 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f789c069e80 con 0x7f789c0b0e30 2024-03-06T07:25:42.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.404+0000 7f789affd640 1 -- 172.21.15.83:0/366107947 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v71) v1 ==== 104+0+11 (secure 0 0 0) 0x7f789c069e80 con 0x7f789c0b0e30 2024-03-06T07:25:42.406 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-03-06T07:25:42.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.406+0000 7f78a4301640 1 -- 172.21.15.83:0/366107947 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f788806b500 msgr2=0x7f788806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.406+0000 7f78a4301640 1 --2- 172.21.15.83:0/366107947 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f788806b500 0x7f788806d9c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f7890006fd0 tx=0x7f7890008040 comp rx=0 tx=0).stop 2024-03-06T07:25:42.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.406+0000 7f78a4301640 1 -- 172.21.15.83:0/366107947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f789c0b0e30 msgr2=0x7f789c1f7c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.406+0000 7f78a4301640 1 --2- 172.21.15.83:0/366107947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f789c0b0e30 0x7f789c1f7c60 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f788c00c400 tx=0x7f788c00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.406+0000 7f78a4301640 1 -- 172.21.15.83:0/366107947 shutdown_connections 2024-03-06T07:25:42.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.406+0000 7f78a4301640 1 --2- 172.21.15.83:0/366107947 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f788806b500 0x7f788806d9c0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.406+0000 7f78a4301640 1 --2- 172.21.15.83:0/366107947 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f789c1614d0 0x7f789c0cc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.406+0000 7f78a4301640 1 --2- 172.21.15.83:0/366107947 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f789c0d4860 0x7f789c15a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.406+0000 7f78a4301640 1 --2- 172.21.15.83:0/366107947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f789c0b0e30 0x7f789c1f7c60 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.406+0000 7f78a4301640 1 -- 172.21.15.83:0/366107947 >> 172.21.15.83:0/366107947 conn(0x7f789c0d3d40 msgr2=0x7f789c06b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:42.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.406+0000 7f78a4301640 1 -- 172.21.15.83:0/366107947 shutdown_connections 2024-03-06T07:25:42.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.406+0000 7f78a4301640 1 -- 172.21.15.83:0/366107947 wait complete. 2024-03-06T07:25:42.419 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:25:42.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.521+0000 7f5d52301640 1 Processor -- start 2024-03-06T07:25:42.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.521+0000 7f5d52301640 1 -- start start 2024-03-06T07:25:42.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.522+0000 7f5d52301640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5d4c1614d0 0x7f5d4c0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.522+0000 7f5d52301640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f5d4c0d1910 0x7f5d4c0d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.522+0000 7f5d52301640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d4c1618b0 con 0x7f5d4c0d22e0 2024-03-06T07:25:42.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.522+0000 7f5d52301640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d4c0cf400 con 0x7f5d4c1614d0 2024-03-06T07:25:42.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.522+0000 7f5d52301640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d4c0d2710 con 0x7f5d4c0d1910 2024-03-06T07:25:42.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.523+0000 7f5d4bfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5d4c1614d0 0x7f5d4c0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.523+0000 7f5d4bfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5d4c1614d0 0x7f5d4c0d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52698/0 (socket says 172.21.15.83:52698) 2024-03-06T07:25:42.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.523+0000 7f5d4bfff640 1 -- 172.21.15.83:0/3485659397 learned_addr learned my addr 172.21.15.83:0/3485659397 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:42.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.523+0000 7f5d4b7fe640 1 --2- 172.21.15.83:0/3485659397 >> v2:172.21.15.183:3300/0 conn(0x7f5d4c0d1910 0x7f5d4c0d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.523+0000 7f5d4bfff640 1 -- 172.21.15.83:0/3485659397 >> v2:172.21.15.183:3300/0 conn(0x7f5d4c0d1910 msgr2=0x7f5d4c0d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:42.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.523+0000 7f5d4bfff640 1 --2- 172.21.15.83:0/3485659397 >> v2:172.21.15.183:3300/0 conn(0x7f5d4c0d1910 0x7f5d4c0d1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.523+0000 7f5d4bfff640 1 -- 172.21.15.83:0/3485659397 >> v1:172.21.15.183:6789/0 conn(0x7f5d4c0d22e0 legacy=0x7f5d4c0d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:42.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.523+0000 7f5d4bfff640 1 -- 172.21.15.83:0/3485659397 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d4c0cf6a0 con 0x7f5d4c1614d0 2024-03-06T07:25:42.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.524+0000 7f5d4bfff640 1 --2- 172.21.15.83:0/3485659397 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d4c1614d0 0x7f5d4c0d4860 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c08c470 tx=0x7f5d4002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc68120d3755f572 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:42.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.524+0000 7f5d4affd640 1 -- 172.21.15.83:0/3485659397 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d40035020 con 0x7f5d4c1614d0 2024-03-06T07:25:42.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.524+0000 7f5d4affd640 1 -- 172.21.15.83:0/3485659397 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d40031640 con 0x7f5d4c1614d0 2024-03-06T07:25:42.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.524+0000 7f5d4affd640 1 -- 172.21.15.83:0/3485659397 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d4002d040 con 0x7f5d4c1614d0 2024-03-06T07:25:42.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.524+0000 7f5d52301640 1 -- 172.21.15.83:0/3485659397 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d4c1614d0 msgr2=0x7f5d4c0d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.524+0000 7f5d52301640 1 --2- 172.21.15.83:0/3485659397 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d4c1614d0 0x7f5d4c0d4860 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c08c470 tx=0x7f5d4002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.524+0000 7f5d52301640 1 -- 172.21.15.83:0/3485659397 shutdown_connections 2024-03-06T07:25:42.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.524+0000 7f5d52301640 1 --2- 172.21.15.83:0/3485659397 >> v2:172.21.15.183:3300/0 conn(0x7f5d4c0d1910 0x7f5d4c0d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.524+0000 7f5d52301640 1 --2- 172.21.15.83:0/3485659397 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d4c1614d0 0x7f5d4c0d4860 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.524+0000 7f5d52301640 1 -- 172.21.15.83:0/3485659397 >> 172.21.15.83:0/3485659397 conn(0x7f5d4c0d3d40 msgr2=0x7f5d4c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:42.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.524+0000 7f5d52301640 1 -- 172.21.15.83:0/3485659397 shutdown_connections 2024-03-06T07:25:42.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.524+0000 7f5d52301640 1 -- 172.21.15.83:0/3485659397 wait complete. 2024-03-06T07:25:42.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.526+0000 7f5d52301640 1 Processor -- start 2024-03-06T07:25:42.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.526+0000 7f5d52301640 1 -- start start 2024-03-06T07:25:42.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.527+0000 7f5d52301640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5d4c0d1910 0x7f5d4c15f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.527+0000 7f5d52301640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d4c0d22e0 0x7f5d4c1f6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.527+0000 7f5d52301640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5d4c1614d0 0x7f5d4c1f7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.527+0000 7f5d52301640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4c0b0910 con 0x7f5d4c0d22e0 2024-03-06T07:25:42.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.527+0000 7f5d52301640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4c0cfc60 con 0x7f5d4c0d1910 2024-03-06T07:25:42.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.527+0000 7f5d52301640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4c0b1e40 con 0x7f5d4c1614d0 2024-03-06T07:25:42.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.527+0000 7f5d4b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d4c0d22e0 0x7f5d4c1f6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.527+0000 7f5d4b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d4c0d22e0 0x7f5d4c1f6c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52700/0 (socket says 172.21.15.83:52700) 2024-03-06T07:25:42.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.527+0000 7f5d4b7fe640 1 -- 172.21.15.83:0/3527553462 learned_addr learned my addr 172.21.15.83:0/3527553462 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:42.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.527+0000 7f5d4bfff640 1 --2- 172.21.15.83:0/3527553462 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5d4c0d1910 0x7f5d4c15f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.527+0000 7f5d5089e640 1 --2- 172.21.15.83:0/3527553462 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5d4c1614d0 0x7f5d4c1f7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.528+0000 7f5d4b7fe640 1 -- 172.21.15.83:0/3527553462 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5d4c1614d0 msgr2=0x7f5d4c1f7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.528+0000 7f5d4b7fe640 1 --2- 172.21.15.83:0/3527553462 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5d4c1614d0 0x7f5d4c1f7180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.528+0000 7f5d4b7fe640 1 -- 172.21.15.83:0/3527553462 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5d4c0d1910 msgr2=0x7f5d4c15f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.528+0000 7f5d4b7fe640 1 --2- 172.21.15.83:0/3527553462 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5d4c0d1910 0x7f5d4c15f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.528+0000 7f5d4b7fe640 1 -- 172.21.15.83:0/3527553462 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d4c0cc130 con 0x7f5d4c0d22e0 2024-03-06T07:25:42.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.528+0000 7f5d4bfff640 1 --2- 172.21.15.83:0/3527553462 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5d4c0d1910 0x7f5d4c15f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:42.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.528+0000 7f5d5089e640 1 --2- 172.21.15.83:0/3527553462 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5d4c1614d0 0x7f5d4c1f7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:42.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.528+0000 7f5d4b7fe640 1 --2- 172.21.15.83:0/3527553462 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d4c0d22e0 0x7f5d4c1f6c40 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c00d800 tx=0x7f5d3c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:42.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.528+0000 7f5d48ff9640 1 -- 172.21.15.83:0/3527553462 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d3c00b6c0 con 0x7f5d4c0d22e0 2024-03-06T07:25:42.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.528+0000 7f5d48ff9640 1 -- 172.21.15.83:0/3527553462 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d3c019070 con 0x7f5d4c0d22e0 2024-03-06T07:25:42.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.528+0000 7f5d52301640 1 -- 172.21.15.83:0/3527553462 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d4c0c8680 con 0x7f5d4c0d22e0 2024-03-06T07:25:42.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.528+0000 7f5d48ff9640 1 -- 172.21.15.83:0/3527553462 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d3c00ba80 con 0x7f5d4c0d22e0 2024-03-06T07:25:42.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.528+0000 7f5d52301640 1 -- 172.21.15.83:0/3527553462 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d4c0cc3d0 con 0x7f5d4c0d22e0 2024-03-06T07:25:42.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.529+0000 7f5d52301640 1 -- 172.21.15.83:0/3527553462 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d4c0d05b0 con 0x7f5d4c0d22e0 2024-03-06T07:25:42.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.529+0000 7f5d48ff9640 1 -- 172.21.15.83:0/3527553462 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5d3c00bc20 con 0x7f5d4c0d22e0 2024-03-06T07:25:42.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.530+0000 7f5d48ff9640 1 --2- 172.21.15.83:0/3527553462 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5d3806b4d0 0x7f5d3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.530+0000 7f5d48ff9640 1 -- 172.21.15.83:0/3527553462 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5d3c08a690 con 0x7f5d4c0d22e0 2024-03-06T07:25:42.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.530+0000 7f5d4bfff640 1 --2- 172.21.15.83:0/3527553462 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5d3806b4d0 0x7f5d3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.530+0000 7f5d4bfff640 1 --2- 172.21.15.83:0/3527553462 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5d3806b4d0 0x7f5d3806d990 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f5d40036000 tx=0x7f5d4003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:42.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.534+0000 7f5d48ff9640 1 -- 172.21.15.83:0/3527553462 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5d4c0d05b0 con 0x7f5d4c0d22e0 2024-03-06T07:25:42.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.759+0000 7f5d52301640 1 -- 172.21.15.83:0/3527553462 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5d4c0cc8b0 con 0x7f5d4c0d22e0 2024-03-06T07:25:42.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.760+0000 7f5d48ff9640 1 -- 172.21.15.83:0/3527553462 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v71) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5d4c0cc8b0 con 0x7f5d4c0d22e0 2024-03-06T07:25:42.762 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 64 2024-03-06T07:25:42.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.762+0000 7f5d52301640 1 -- 172.21.15.83:0/3527553462 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5d3806b4d0 msgr2=0x7f5d3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.762+0000 7f5d52301640 1 --2- 172.21.15.83:0/3527553462 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5d3806b4d0 0x7f5d3806d990 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f5d40036000 tx=0x7f5d4003c040 comp rx=0 tx=0).stop 2024-03-06T07:25:42.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.762+0000 7f5d52301640 1 -- 172.21.15.83:0/3527553462 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d4c0d22e0 msgr2=0x7f5d4c1f6c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.762+0000 7f5d52301640 1 --2- 172.21.15.83:0/3527553462 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d4c0d22e0 0x7f5d4c1f6c40 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c00d800 tx=0x7f5d3c00dcc0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.762+0000 7f5d52301640 1 -- 172.21.15.83:0/3527553462 shutdown_connections 2024-03-06T07:25:42.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.762+0000 7f5d52301640 1 --2- 172.21.15.83:0/3527553462 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5d3806b4d0 0x7f5d3806d990 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.762+0000 7f5d52301640 1 --2- 172.21.15.83:0/3527553462 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5d4c1614d0 0x7f5d4c1f7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.762+0000 7f5d52301640 1 --2- 172.21.15.83:0/3527553462 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d4c0d22e0 0x7f5d4c1f6c40 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.762+0000 7f5d52301640 1 --2- 172.21.15.83:0/3527553462 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5d4c0d1910 0x7f5d4c15f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.762+0000 7f5d52301640 1 -- 172.21.15.83:0/3527553462 >> 172.21.15.83:0/3527553462 conn(0x7f5d4c0d3d40 msgr2=0x7f5d4c0cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:42.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.763+0000 7f5d52301640 1 -- 172.21.15.83:0/3527553462 shutdown_connections 2024-03-06T07:25:42.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.763+0000 7f5d52301640 1 -- 172.21.15.83:0/3527553462 wait complete. 2024-03-06T07:25:42.775 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:42.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.875+0000 7fda0e2bb640 1 Processor -- start 2024-03-06T07:25:42.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.875+0000 7fda0e2bb640 1 -- start start 2024-03-06T07:25:42.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.876+0000 7fda0e2bb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fda0815dfe0 0x7fda080cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.876+0000 7fda0e2bb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fda080cf830 0x7fda080d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.876+0000 7fda0e2bb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda080b3430 con 0x7fda080d27f0 2024-03-06T07:25:42.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.876+0000 7fda0e2bb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda080d2c50 con 0x7fda0815dfe0 2024-03-06T07:25:42.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.876+0000 7fda0e2bb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda080b31e0 con 0x7fda080cf830 2024-03-06T07:25:42.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.876+0000 7fda07fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fda0815dfe0 0x7fda080cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.876+0000 7fda0c858640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fda080d27f0 0x7fda080d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52562/0 (socket says 172.21.15.83:52562) 2024-03-06T07:25:42.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.876+0000 7fda07fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fda0815dfe0 0x7fda080cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52714/0 (socket says 172.21.15.83:52714) 2024-03-06T07:25:42.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.876+0000 7fda0c858640 1 -- 172.21.15.83:0/481165207 learned_addr learned my addr 172.21.15.83:0/481165207 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:25:42.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.876+0000 7fda077fe640 1 --2- 172.21.15.83:0/481165207 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fda080cf830 0x7fda080d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.876+0000 7fda07fff640 1 -- 172.21.15.83:0/481165207 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fda080cf830 msgr2=0x7fda080d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:42.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.876+0000 7fda07fff640 1 --2- 172.21.15.83:0/481165207 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fda080cf830 0x7fda080d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.876+0000 7fda07fff640 1 -- 172.21.15.83:0/481165207 >> v1:172.21.15.83:6789/0 conn(0x7fda080d27f0 legacy=0x7fda080d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.876+0000 7fda07fff640 1 -- 172.21.15.83:0/481165207 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda080b3710 con 0x7fda0815dfe0 2024-03-06T07:25:42.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.876+0000 7fda06ffd640 1 -- 172.21.15.83:0/481165207 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1205673993 0 0) 0x7fda080b3430 con 0x7fda080d27f0 2024-03-06T07:25:42.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.877+0000 7fda07fff640 1 --2- 172.21.15.83:0/481165207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fda0815dfe0 0x7fda080cf2f0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fd9f00097a0 tx=0x7fd9f002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b45aee6daa4dba57 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:42.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.877+0000 7fda06ffd640 1 -- 172.21.15.83:0/481165207 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f0035020 con 0x7fda0815dfe0 2024-03-06T07:25:42.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.877+0000 7fda06ffd640 1 -- 172.21.15.83:0/481165207 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9f0031830 con 0x7fda0815dfe0 2024-03-06T07:25:42.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.877+0000 7fda06ffd640 1 -- 172.21.15.83:0/481165207 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f0031c00 con 0x7fda0815dfe0 2024-03-06T07:25:42.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.877+0000 7fda0e2bb640 1 -- 172.21.15.83:0/481165207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fda0815dfe0 msgr2=0x7fda080cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.877+0000 7fda0e2bb640 1 --2- 172.21.15.83:0/481165207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fda0815dfe0 0x7fda080cf2f0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fd9f00097a0 tx=0x7fd9f002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.878+0000 7fda0e2bb640 1 -- 172.21.15.83:0/481165207 shutdown_connections 2024-03-06T07:25:42.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.878+0000 7fda0e2bb640 1 --2- 172.21.15.83:0/481165207 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fda080cf830 0x7fda080d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.878+0000 7fda0e2bb640 1 --2- 172.21.15.83:0/481165207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fda0815dfe0 0x7fda080cf2f0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.878+0000 7fda0e2bb640 1 -- 172.21.15.83:0/481165207 >> 172.21.15.83:0/481165207 conn(0x7fda080d3cd0 msgr2=0x7fda080d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:42.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.878+0000 7fda0e2bb640 1 -- 172.21.15.83:0/481165207 shutdown_connections 2024-03-06T07:25:42.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.878+0000 7fda0e2bb640 1 -- 172.21.15.83:0/481165207 wait complete. 2024-03-06T07:25:42.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.879+0000 7fda0e2bb640 1 Processor -- start 2024-03-06T07:25:42.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.879+0000 7fda0e2bb640 1 -- start start 2024-03-06T07:25:42.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.880+0000 7fda0e2bb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fda080cf830 0x7fda080cce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.880+0000 7fda0e2bb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fda080d27f0 0x7fda0815e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.880+0000 7fda0e2bb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fda0815dfe0 0x7fda0815ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.880+0000 7fda0e2bb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda080d1c70 con 0x7fda080d27f0 2024-03-06T07:25:42.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.880+0000 7fda0e2bb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda080b3d60 con 0x7fda080cf830 2024-03-06T07:25:42.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.880+0000 7fda0e2bb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda080d1df0 con 0x7fda0815dfe0 2024-03-06T07:25:42.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.880+0000 7fda077fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fda080d27f0 0x7fda0815e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.880+0000 7fda077fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fda080d27f0 0x7fda0815e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52718/0 (socket says 172.21.15.83:52718) 2024-03-06T07:25:42.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.880+0000 7fda077fe640 1 -- 172.21.15.83:0/947911465 learned_addr learned my addr 172.21.15.83:0/947911465 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:42.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.880+0000 7fda077fe640 1 -- 172.21.15.83:0/947911465 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fda0815dfe0 msgr2=0x7fda0815ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.880+0000 7fda077fe640 1 --2- 172.21.15.83:0/947911465 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fda0815dfe0 0x7fda0815ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.880+0000 7fda077fe640 1 -- 172.21.15.83:0/947911465 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fda080cf830 msgr2=0x7fda080cce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.880+0000 7fda077fe640 1 --2- 172.21.15.83:0/947911465 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fda080cf830 0x7fda080cce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.880+0000 7fda077fe640 1 -- 172.21.15.83:0/947911465 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda080b6a40 con 0x7fda080d27f0 2024-03-06T07:25:42.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.880+0000 7fda077fe640 1 --2- 172.21.15.83:0/947911465 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fda080d27f0 0x7fda0815e4e0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fd9f800d7d0 tx=0x7fd9f800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:42.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.880+0000 7fda04ff9640 1 -- 172.21.15.83:0/947911465 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f800b6c0 con 0x7fda080d27f0 2024-03-06T07:25:42.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.880+0000 7fda04ff9640 1 -- 172.21.15.83:0/947911465 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9f8019070 con 0x7fda080d27f0 2024-03-06T07:25:42.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.880+0000 7fda0e2bb640 1 -- 172.21.15.83:0/947911465 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda0815f050 con 0x7fda080d27f0 2024-03-06T07:25:42.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.880+0000 7fda04ff9640 1 -- 172.21.15.83:0/947911465 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f800bac0 con 0x7fda080d27f0 2024-03-06T07:25:42.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.880+0000 7fda0e2bb640 1 -- 172.21.15.83:0/947911465 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda080b6ce0 con 0x7fda080d27f0 2024-03-06T07:25:42.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.881+0000 7fda0e2bb640 1 -- 172.21.15.83:0/947911465 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda0804f330 con 0x7fda080d27f0 2024-03-06T07:25:42.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.882+0000 7fda04ff9640 1 -- 172.21.15.83:0/947911465 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd9f800bc60 con 0x7fda080d27f0 2024-03-06T07:25:42.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.882+0000 7fda04ff9640 1 --2- 172.21.15.83:0/947911465 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd9e806b500 0x7fd9e806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.882+0000 7fda04ff9640 1 -- 172.21.15.83:0/947911465 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd9f808a610 con 0x7fda080d27f0 2024-03-06T07:25:42.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.882+0000 7fda07fff640 1 --2- 172.21.15.83:0/947911465 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd9e806b500 0x7fd9e806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.883+0000 7fda07fff640 1 --2- 172.21.15.83:0/947911465 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd9e806b500 0x7fd9e806d9c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fd9f0036000 tx=0x7fd9f0030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:42.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:42.891+0000 7fda04ff9640 1 -- 172.21.15.83:0/947911465 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd9f801c360 con 0x7fda080d27f0 2024-03-06T07:25:43.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.105+0000 7fda0e2bb640 1 -- 172.21.15.83:0/947911465 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda0815f570 con 0x7fda080d27f0 2024-03-06T07:25:43.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.106+0000 7fda04ff9640 1 -- 172.21.15.83:0/947911465 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4982 (secure 0 0 0) 0x7fd9f80594e0 con 0x7fda080d27f0 2024-03-06T07:25:43.108 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:25:43.108 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":32,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6010,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/2142942385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2142942385},{"type":"v1","addr":"172.21.15.183:6834","nonce":2142942385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":6013,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3742438064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":3742438064},{"type":"v1","addr":"172.21.15.183:6835","nonce":3742438064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":6540,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/1549894851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1549894851},{"type":"v1","addr":"172.21.15.83:6836","nonce":1549894851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:12.481670+0000","modified":"2024-03-06T07:25:15.392072+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6543},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6543":{"gid":6543,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/602775648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":602775648},{"type":"v1","addr":"172.21.15.83:6837","nonce":602775648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6543,"qdb_cluster":[6543]},"id":5}]} 2024-03-06T07:25:43.108 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 32 2024-03-06T07:25:43.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.108+0000 7fda0e2bb640 1 -- 172.21.15.83:0/947911465 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd9e806b500 msgr2=0x7fd9e806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.108+0000 7fda0e2bb640 1 --2- 172.21.15.83:0/947911465 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd9e806b500 0x7fd9e806d9c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fd9f0036000 tx=0x7fd9f0030000 comp rx=0 tx=0).stop 2024-03-06T07:25:43.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.108+0000 7fda0e2bb640 1 -- 172.21.15.83:0/947911465 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fda080d27f0 msgr2=0x7fda0815e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.108+0000 7fda0e2bb640 1 --2- 172.21.15.83:0/947911465 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fda080d27f0 0x7fda0815e4e0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fd9f800d7d0 tx=0x7fd9f800dc90 comp rx=0 tx=0).stop 2024-03-06T07:25:43.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.108+0000 7fda0e2bb640 1 -- 172.21.15.83:0/947911465 shutdown_connections 2024-03-06T07:25:43.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.108+0000 7fda0e2bb640 1 --2- 172.21.15.83:0/947911465 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd9e806b500 0x7fd9e806d9c0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.108+0000 7fda0e2bb640 1 --2- 172.21.15.83:0/947911465 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fda0815dfe0 0x7fda0815ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.108+0000 7fda0e2bb640 1 --2- 172.21.15.83:0/947911465 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fda080d27f0 0x7fda0815e4e0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.108+0000 7fda0e2bb640 1 --2- 172.21.15.83:0/947911465 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fda080cf830 0x7fda080cce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.108+0000 7fda0e2bb640 1 -- 172.21.15.83:0/947911465 >> 172.21.15.83:0/947911465 conn(0x7fda080d3cd0 msgr2=0x7fda0806cab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:43.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.108+0000 7fda0e2bb640 1 -- 172.21.15.83:0/947911465 shutdown_connections 2024-03-06T07:25:43.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.108+0000 7fda0e2bb640 1 -- 172.21.15.83:0/947911465 wait complete. 2024-03-06T07:25:43.120 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:43.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.219+0000 7f0c2c450640 1 Processor -- start 2024-03-06T07:25:43.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.219+0000 7f0c2c450640 1 -- start start 2024-03-06T07:25:43.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.220+0000 7f0c2c450640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f0c2415dfe0 0x7f0c240cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.220+0000 7f0c2c450640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0c240cf830 0x7f0c240d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.220+0000 7f0c2c450640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0c240d27f0 0x7f0c240d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.220+0000 7f0c2c450640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c240b31e0 con 0x7f0c240cf830 2024-03-06T07:25:43.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.220+0000 7f0c2c450640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c240b3d60 con 0x7f0c2415dfe0 2024-03-06T07:25:43.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.220+0000 7f0c2c450640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c240b3430 con 0x7f0c240d27f0 2024-03-06T07:25:43.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.221+0000 7f0c299eb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0c240cf830 0x7f0c240d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.221+0000 7f0c299eb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0c240cf830 0x7f0c240d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52732/0 (socket says 172.21.15.83:52732) 2024-03-06T07:25:43.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.221+0000 7f0c299eb640 1 -- 172.21.15.83:0/3347935447 learned_addr learned my addr 172.21.15.83:0/3347935447 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:43.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.221+0000 7f0c2a1ec640 1 --2- 172.21.15.83:0/3347935447 >> v2:172.21.15.183:3300/0 conn(0x7f0c2415dfe0 0x7f0c240cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.221+0000 7f0c2a9ed640 1 --2- 172.21.15.83:0/3347935447 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0c240d27f0 0x7f0c240d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.221+0000 7f0c299eb640 1 -- 172.21.15.83:0/3347935447 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0c240d27f0 msgr2=0x7f0c240d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:43.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.221+0000 7f0c299eb640 1 --2- 172.21.15.83:0/3347935447 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0c240d27f0 0x7f0c240d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.221+0000 7f0c299eb640 1 -- 172.21.15.83:0/3347935447 >> v2:172.21.15.183:3300/0 conn(0x7f0c2415dfe0 msgr2=0x7f0c240cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:43.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.221+0000 7f0c299eb640 1 --2- 172.21.15.83:0/3347935447 >> v2:172.21.15.183:3300/0 conn(0x7f0c2415dfe0 0x7f0c240cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.221+0000 7f0c299eb640 1 -- 172.21.15.83:0/3347935447 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c240b3710 con 0x7f0c240cf830 2024-03-06T07:25:43.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.222+0000 7f0c299eb640 1 --2- 172.21.15.83:0/3347935447 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c240cf830 0x7f0c240d22b0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f0c24067990 tx=0x7f0c1402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2490db64b224815b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:43.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.222+0000 7f0c291ea640 1 -- 172.21.15.83:0/3347935447 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c14035020 con 0x7f0c240cf830 2024-03-06T07:25:43.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.222+0000 7f0c291ea640 1 -- 172.21.15.83:0/3347935447 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c14031870 con 0x7f0c240cf830 2024-03-06T07:25:43.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.222+0000 7f0c291ea640 1 -- 172.21.15.83:0/3347935447 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c14031c40 con 0x7f0c240cf830 2024-03-06T07:25:43.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.222+0000 7f0c2c450640 1 -- 172.21.15.83:0/3347935447 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c240cf830 msgr2=0x7f0c240d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.222+0000 7f0c2c450640 1 --2- 172.21.15.83:0/3347935447 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c240cf830 0x7f0c240d22b0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f0c24067990 tx=0x7f0c1402b910 comp rx=0 tx=0).stop 2024-03-06T07:25:43.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.223+0000 7f0c2c450640 1 -- 172.21.15.83:0/3347935447 shutdown_connections 2024-03-06T07:25:43.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.223+0000 7f0c2c450640 1 --2- 172.21.15.83:0/3347935447 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0c240d27f0 0x7f0c240d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.223+0000 7f0c2c450640 1 --2- 172.21.15.83:0/3347935447 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c240cf830 0x7f0c240d22b0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.223+0000 7f0c2c450640 1 --2- 172.21.15.83:0/3347935447 >> v2:172.21.15.183:3300/0 conn(0x7f0c2415dfe0 0x7f0c240cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.223+0000 7f0c2c450640 1 -- 172.21.15.83:0/3347935447 >> 172.21.15.83:0/3347935447 conn(0x7f0c240d3cd0 msgr2=0x7f0c240d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:43.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.223+0000 7f0c2c450640 1 -- 172.21.15.83:0/3347935447 shutdown_connections 2024-03-06T07:25:43.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.223+0000 7f0c2c450640 1 -- 172.21.15.83:0/3347935447 wait complete. 2024-03-06T07:25:43.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.224+0000 7f0c2c450640 1 Processor -- start 2024-03-06T07:25:43.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.224+0000 7f0c2c450640 1 -- start start 2024-03-06T07:25:43.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.224+0000 7f0c2c450640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c240cf830 0x7f0c2415c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.224+0000 7f0c2c450640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0c240d27f0 0x7f0c2415f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.224+0000 7f0c2c450640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c2415dfe0 0x7f0c241531d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.224+0000 7f0c2c450640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c240d1df0 con 0x7f0c240cf830 2024-03-06T07:25:43.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.224+0000 7f0c2c450640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c240d1c70 con 0x7f0c2415dfe0 2024-03-06T07:25:43.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.224+0000 7f0c2c450640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2415e3c0 con 0x7f0c240d27f0 2024-03-06T07:25:43.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.224+0000 7f0c2a1ec640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c240cf830 0x7f0c2415c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.224+0000 7f0c2a1ec640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c240cf830 0x7f0c2415c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52740/0 (socket says 172.21.15.83:52740) 2024-03-06T07:25:43.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.225+0000 7f0c2a1ec640 1 -- 172.21.15.83:0/1684760529 learned_addr learned my addr 172.21.15.83:0/1684760529 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:43.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.225+0000 7f0c299eb640 1 --2- 172.21.15.83:0/1684760529 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0c240d27f0 0x7f0c2415f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.225+0000 7f0c2a9ed640 1 --2- 172.21.15.83:0/1684760529 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c2415dfe0 0x7f0c241531d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.225+0000 7f0c2a1ec640 1 -- 172.21.15.83:0/1684760529 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0c240d27f0 msgr2=0x7f0c2415f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.225+0000 7f0c2a1ec640 1 --2- 172.21.15.83:0/1684760529 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0c240d27f0 0x7f0c2415f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.225+0000 7f0c2a1ec640 1 -- 172.21.15.83:0/1684760529 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c2415dfe0 msgr2=0x7f0c241531d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.225+0000 7f0c2a1ec640 1 --2- 172.21.15.83:0/1684760529 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c2415dfe0 0x7f0c241531d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.225+0000 7f0c2a1ec640 1 -- 172.21.15.83:0/1684760529 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c2415fe90 con 0x7f0c240cf830 2024-03-06T07:25:43.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.225+0000 7f0c2a1ec640 1 --2- 172.21.15.83:0/1684760529 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c240cf830 0x7f0c2415c0d0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f0c2000d620 tx=0x7f0c2000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:43.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.225+0000 7f0c1affd640 1 -- 172.21.15.83:0/1684760529 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2000b620 con 0x7f0c240cf830 2024-03-06T07:25:43.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.225+0000 7f0c1affd640 1 -- 172.21.15.83:0/1684760529 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c20019070 con 0x7f0c240cf830 2024-03-06T07:25:43.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.225+0000 7f0c2c450640 1 -- 172.21.15.83:0/1684760529 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c24160170 con 0x7f0c240cf830 2024-03-06T07:25:43.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.225+0000 7f0c1affd640 1 -- 172.21.15.83:0/1684760529 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2000ba00 con 0x7f0c240cf830 2024-03-06T07:25:43.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.225+0000 7f0c2c450640 1 -- 172.21.15.83:0/1684760529 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c24160730 con 0x7f0c240cf830 2024-03-06T07:25:43.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.226+0000 7f0c2c450640 1 -- 172.21.15.83:0/1684760529 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c2404f330 con 0x7f0c240cf830 2024-03-06T07:25:43.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.227+0000 7f0c1affd640 1 -- 172.21.15.83:0/1684760529 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0c2000bba0 con 0x7f0c240cf830 2024-03-06T07:25:43.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.227+0000 7f0c1affd640 1 --2- 172.21.15.83:0/1684760529 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0c0806b4d0 0x7f0c0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.227+0000 7f0c1affd640 1 -- 172.21.15.83:0/1684760529 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0c2008a660 con 0x7f0c240cf830 2024-03-06T07:25:43.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.227+0000 7f0c299eb640 1 --2- 172.21.15.83:0/1684760529 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0c0806b4d0 0x7f0c0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.228+0000 7f0c299eb640 1 --2- 172.21.15.83:0/1684760529 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0c0806b4d0 0x7f0c0806d990 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f0c14036000 tx=0x7f0c1403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:43.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.232+0000 7f0c1affd640 1 -- 172.21.15.83:0/1684760529 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0c200594a0 con 0x7f0c240cf830 2024-03-06T07:25:43.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.404+0000 7f0c2c450640 1 -- 172.21.15.83:0/1684760529 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0c2415d450 con 0x7f0c240cf830 2024-03-06T07:25:43.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.405+0000 7f0c1affd640 1 -- 172.21.15.83:0/1684760529 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+16778 (secure 0 0 0) 0x7f0c20058da0 con 0x7f0c240cf830 2024-03-06T07:25:43.407 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:25:43.407 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":71,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:25:40.924823+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:10.701416+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:11.450797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.1e","mappings":[{"from":6,"to":0}]},{"pgid":"11.3e","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:43.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.407+0000 7f0c2c450640 1 -- 172.21.15.83:0/1684760529 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0c0806b4d0 msgr2=0x7f0c0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.407+0000 7f0c2c450640 1 --2- 172.21.15.83:0/1684760529 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0c0806b4d0 0x7f0c0806d990 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f0c14036000 tx=0x7f0c1403c040 comp rx=0 tx=0).stop 2024-03-06T07:25:43.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.407+0000 7f0c2c450640 1 -- 172.21.15.83:0/1684760529 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c240cf830 msgr2=0x7f0c2415c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.407+0000 7f0c2c450640 1 --2- 172.21.15.83:0/1684760529 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c240cf830 0x7f0c2415c0d0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f0c2000d620 tx=0x7f0c2000dae0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.407+0000 7f0c2c450640 1 -- 172.21.15.83:0/1684760529 shutdown_connections 2024-03-06T07:25:43.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.407+0000 7f0c2c450640 1 --2- 172.21.15.83:0/1684760529 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0c0806b4d0 0x7f0c0806d990 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.407+0000 7f0c2c450640 1 --2- 172.21.15.83:0/1684760529 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c2415dfe0 0x7f0c241531d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.407+0000 7f0c2c450640 1 --2- 172.21.15.83:0/1684760529 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0c240d27f0 0x7f0c2415f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.407+0000 7f0c2c450640 1 --2- 172.21.15.83:0/1684760529 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c240cf830 0x7f0c2415c0d0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.407+0000 7f0c2c450640 1 -- 172.21.15.83:0/1684760529 >> 172.21.15.83:0/1684760529 conn(0x7f0c240d3cd0 msgr2=0x7f0c2406bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:43.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.407+0000 7f0c2c450640 1 -- 172.21.15.83:0/1684760529 shutdown_connections 2024-03-06T07:25:43.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.407+0000 7f0c2c450640 1 -- 172.21.15.83:0/1684760529 wait complete. 2024-03-06T07:25:43.420 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:25:43.421 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:43.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.519+0000 7fc8539b6640 1 Processor -- start 2024-03-06T07:25:43.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.519+0000 7fc8539b6640 1 -- start start 2024-03-06T07:25:43.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.520+0000 7fc8539b6640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fc84c15bdd0 0x7fc84c0c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.520+0000 7fc8539b6640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc84c15fa10 0x7fc84c0ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.520+0000 7fc8539b6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc84c0cebe0 con 0x7fc84c0d16e0 2024-03-06T07:25:43.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.520+0000 7fc8539b6640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc84c0ceec0 con 0x7fc84c15fa10 2024-03-06T07:25:43.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.520+0000 7fc8539b6640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc84c0cf1a0 con 0x7fc84c15bdd0 2024-03-06T07:25:43.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.520+0000 7fc850f51640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc84c15fa10 0x7fc84c0ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.520+0000 7fc851f53640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fc84c0d16e0 0x7fc84c0d1af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52568/0 (socket says 172.21.15.83:52568) 2024-03-06T07:25:43.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.520+0000 7fc850f51640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc84c15fa10 0x7fc84c0ce270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52754/0 (socket says 172.21.15.83:52754) 2024-03-06T07:25:43.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.520+0000 7fc851f53640 1 -- 172.21.15.83:0/3538201035 learned_addr learned my addr 172.21.15.83:0/3538201035 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:25:43.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.521+0000 7fc851752640 1 --2- 172.21.15.83:0/3538201035 >> v2:172.21.15.183:3300/0 conn(0x7fc84c15bdd0 0x7fc84c0c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.521+0000 7fc850f51640 1 -- 172.21.15.83:0/3538201035 >> v2:172.21.15.183:3300/0 conn(0x7fc84c15bdd0 msgr2=0x7fc84c0c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:43.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.521+0000 7fc850f51640 1 --2- 172.21.15.83:0/3538201035 >> v2:172.21.15.183:3300/0 conn(0x7fc84c15bdd0 0x7fc84c0c5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.521+0000 7fc850f51640 1 -- 172.21.15.83:0/3538201035 >> v1:172.21.15.83:6789/0 conn(0x7fc84c0d16e0 legacy=0x7fc84c0d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.521+0000 7fc850f51640 1 -- 172.21.15.83:0/3538201035 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc84c0cc8e0 con 0x7fc84c15fa10 2024-03-06T07:25:43.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.521+0000 7fc851752640 1 --2- 172.21.15.83:0/3538201035 >> v2:172.21.15.183:3300/0 conn(0x7fc84c15bdd0 0x7fc84c0c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:43.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.521+0000 7fc850f51640 1 --2- 172.21.15.83:0/3538201035 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc84c15fa10 0x7fc84c0ce270 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fc84c06e3c0 tx=0x7fc83c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb68a261575afad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:43.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.522+0000 7fc843fff640 1 -- 172.21.15.83:0/3538201035 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc83c0315a0 con 0x7fc84c15fa10 2024-03-06T07:25:43.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.522+0000 7fc843fff640 1 -- 172.21.15.83:0/3538201035 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc83c037070 con 0x7fc84c15fa10 2024-03-06T07:25:43.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.522+0000 7fc843fff640 1 -- 172.21.15.83:0/3538201035 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc83c031960 con 0x7fc84c15fa10 2024-03-06T07:25:43.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.522+0000 7fc8539b6640 1 -- 172.21.15.83:0/3538201035 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc84c15fa10 msgr2=0x7fc84c0ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.522+0000 7fc8539b6640 1 --2- 172.21.15.83:0/3538201035 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc84c15fa10 0x7fc84c0ce270 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fc84c06e3c0 tx=0x7fc83c02b840 comp rx=0 tx=0).stop 2024-03-06T07:25:43.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.522+0000 7fc8539b6640 1 -- 172.21.15.83:0/3538201035 shutdown_connections 2024-03-06T07:25:43.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.522+0000 7fc8539b6640 1 --2- 172.21.15.83:0/3538201035 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc84c15fa10 0x7fc84c0ce270 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.522+0000 7fc8539b6640 1 --2- 172.21.15.83:0/3538201035 >> v2:172.21.15.183:3300/0 conn(0x7fc84c15bdd0 0x7fc84c0c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.522+0000 7fc8539b6640 1 -- 172.21.15.83:0/3538201035 >> 172.21.15.83:0/3538201035 conn(0x7fc84c09a9f0 msgr2=0x7fc84c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:43.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.523+0000 7fc8539b6640 1 -- 172.21.15.83:0/3538201035 shutdown_connections 2024-03-06T07:25:43.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.523+0000 7fc8539b6640 1 -- 172.21.15.83:0/3538201035 wait complete. 2024-03-06T07:25:43.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.524+0000 7fc8539b6640 1 Processor -- start 2024-03-06T07:25:43.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.524+0000 7fc8539b6640 1 -- start start 2024-03-06T07:25:43.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.524+0000 7fc8539b6640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc84c0d16e0 0x7fc84c0cfcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.524+0000 7fc8539b6640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc84c15bdd0 0x7fc84c0cb440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.524+0000 7fc8539b6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc84c15fa10 0x7fc84c14eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.524+0000 7fc8539b6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc84c15c1b0 con 0x7fc84c15fa10 2024-03-06T07:25:43.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.524+0000 7fc8539b6640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc84c0cc5d0 con 0x7fc84c15bdd0 2024-03-06T07:25:43.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.524+0000 7fc8539b6640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc84c0ce7b0 con 0x7fc84c0d16e0 2024-03-06T07:25:43.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.524+0000 7fc851f53640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc84c15fa10 0x7fc84c14eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.525+0000 7fc851f53640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc84c15fa10 0x7fc84c14eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52770/0 (socket says 172.21.15.83:52770) 2024-03-06T07:25:43.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.525+0000 7fc851f53640 1 -- 172.21.15.83:0/41991358 learned_addr learned my addr 172.21.15.83:0/41991358 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:43.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.525+0000 7fc851752640 1 --2- 172.21.15.83:0/41991358 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc84c0d16e0 0x7fc84c0cfcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.525+0000 7fc850f51640 1 --2- 172.21.15.83:0/41991358 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc84c15bdd0 0x7fc84c0cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.525+0000 7fc851f53640 1 -- 172.21.15.83:0/41991358 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc84c0d16e0 msgr2=0x7fc84c0cfcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.525+0000 7fc851f53640 1 --2- 172.21.15.83:0/41991358 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc84c0d16e0 0x7fc84c0cfcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.525+0000 7fc851f53640 1 -- 172.21.15.83:0/41991358 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc84c15bdd0 msgr2=0x7fc84c0cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.525+0000 7fc851f53640 1 --2- 172.21.15.83:0/41991358 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc84c15bdd0 0x7fc84c0cb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.525+0000 7fc851f53640 1 -- 172.21.15.83:0/41991358 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc84c1f3cf0 con 0x7fc84c15fa10 2024-03-06T07:25:43.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.525+0000 7fc850f51640 1 --2- 172.21.15.83:0/41991358 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc84c15bdd0 0x7fc84c0cb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:43.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.525+0000 7fc851f53640 1 --2- 172.21.15.83:0/41991358 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc84c15fa10 0x7fc84c14eb00 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fc844002230 tx=0x7fc84400cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:43.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.525+0000 7fc841ffb640 1 -- 172.21.15.83:0/41991358 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc844017070 con 0x7fc84c15fa10 2024-03-06T07:25:43.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.525+0000 7fc841ffb640 1 -- 172.21.15.83:0/41991358 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc84400abd0 con 0x7fc84c15fa10 2024-03-06T07:25:43.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.525+0000 7fc8539b6640 1 -- 172.21.15.83:0/41991358 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc84c1f3fa0 con 0x7fc84c15fa10 2024-03-06T07:25:43.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.525+0000 7fc841ffb640 1 -- 172.21.15.83:0/41991358 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc844012400 con 0x7fc84c15fa10 2024-03-06T07:25:43.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.525+0000 7fc8539b6640 1 -- 172.21.15.83:0/41991358 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc84c1f4560 con 0x7fc84c15fa10 2024-03-06T07:25:43.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.526+0000 7fc8539b6640 1 -- 172.21.15.83:0/41991358 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc84c0c9d10 con 0x7fc84c15fa10 2024-03-06T07:25:43.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.527+0000 7fc841ffb640 1 -- 172.21.15.83:0/41991358 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc84401a070 con 0x7fc84c15fa10 2024-03-06T07:25:43.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.527+0000 7fc841ffb640 1 --2- 172.21.15.83:0/41991358 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc83006b4d0 0x7fc83006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.527+0000 7fc851752640 1 --2- 172.21.15.83:0/41991358 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc83006b4d0 0x7fc83006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.527+0000 7fc841ffb640 1 -- 172.21.15.83:0/41991358 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc844089dd0 con 0x7fc84c15fa10 2024-03-06T07:25:43.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.528+0000 7fc851752640 1 --2- 172.21.15.83:0/41991358 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc83006b4d0 0x7fc83006d990 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fc84800ab60 tx=0x7fc848008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:43.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.532+0000 7fc841ffb640 1 -- 172.21.15.83:0/41991358 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc844058650 con 0x7fc84c15fa10 2024-03-06T07:25:43.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.745+0000 7fc8539b6640 1 -- 172.21.15.83:0/41991358 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc84c0b7a40 con 0x7fc84c15fa10 2024-03-06T07:25:43.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.746+0000 7fc841ffb640 1 -- 172.21.15.83:0/41991358 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4982 (secure 0 0 0) 0x7fc84408b090 con 0x7fc84c15fa10 2024-03-06T07:25:43.748 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:25:43.748 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":32,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6010,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/2142942385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2142942385},{"type":"v1","addr":"172.21.15.183:6834","nonce":2142942385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":6013,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3742438064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":3742438064},{"type":"v1","addr":"172.21.15.183:6835","nonce":3742438064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":6540,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/1549894851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1549894851},{"type":"v1","addr":"172.21.15.83:6836","nonce":1549894851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:12.481670+0000","modified":"2024-03-06T07:25:15.392072+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6543},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6543":{"gid":6543,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/602775648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":602775648},{"type":"v1","addr":"172.21.15.83:6837","nonce":602775648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6543,"qdb_cluster":[6543]},"id":5}]} 2024-03-06T07:25:43.748 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 32 2024-03-06T07:25:43.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.748+0000 7fc8539b6640 1 -- 172.21.15.83:0/41991358 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc83006b4d0 msgr2=0x7fc83006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.748+0000 7fc8539b6640 1 --2- 172.21.15.83:0/41991358 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc83006b4d0 0x7fc83006d990 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fc84800ab60 tx=0x7fc848008040 comp rx=0 tx=0).stop 2024-03-06T07:25:43.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.748+0000 7fc8539b6640 1 -- 172.21.15.83:0/41991358 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc84c15fa10 msgr2=0x7fc84c14eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.748+0000 7fc8539b6640 1 --2- 172.21.15.83:0/41991358 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc84c15fa10 0x7fc84c14eb00 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fc844002230 tx=0x7fc84400cf00 comp rx=0 tx=0).stop 2024-03-06T07:25:43.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.748+0000 7fc8539b6640 1 -- 172.21.15.83:0/41991358 shutdown_connections 2024-03-06T07:25:43.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.748+0000 7fc8539b6640 1 --2- 172.21.15.83:0/41991358 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc83006b4d0 0x7fc83006d990 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.748+0000 7fc8539b6640 1 --2- 172.21.15.83:0/41991358 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc84c15fa10 0x7fc84c14eb00 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.748+0000 7fc8539b6640 1 --2- 172.21.15.83:0/41991358 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc84c15bdd0 0x7fc84c0cb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.748+0000 7fc8539b6640 1 --2- 172.21.15.83:0/41991358 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc84c0d16e0 0x7fc84c0cfcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.748+0000 7fc8539b6640 1 -- 172.21.15.83:0/41991358 >> 172.21.15.83:0/41991358 conn(0x7fc84c09a9f0 msgr2=0x7fc84c070270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:43.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.748+0000 7fc8539b6640 1 -- 172.21.15.83:0/41991358 shutdown_connections 2024-03-06T07:25:43.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.748+0000 7fc8539b6640 1 -- 172.21.15.83:0/41991358 wait complete. 2024-03-06T07:25:43.760 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:43.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.861+0000 7ff12cf81640 1 Processor -- start 2024-03-06T07:25:43.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.861+0000 7ff12cf81640 1 -- start start 2024-03-06T07:25:43.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.862+0000 7ff12cf81640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ff12815dfe0 0x7ff1280cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.862+0000 7ff12cf81640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff1280d29c0 0x7ff1280b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.862+0000 7ff12cf81640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1280b3430 con 0x7ff1280cf830 2024-03-06T07:25:43.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.862+0000 7ff12cf81640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1280b31e0 con 0x7ff12815dfe0 2024-03-06T07:25:43.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.862+0000 7ff12cf81640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1280b3d60 con 0x7ff1280d29c0 2024-03-06T07:25:43.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.862+0000 7ff11dd9b640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff1280cf830 0x7ff1280d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52570/0 (socket says 172.21.15.83:52570) 2024-03-06T07:25:43.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.862+0000 7ff11dd9b640 1 -- 172.21.15.83:0/1149470467 learned_addr learned my addr 172.21.15.83:0/1149470467 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:25:43.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.862+0000 7ff126d9d640 1 --2- 172.21.15.83:0/1149470467 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff1280d29c0 0x7ff1280b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.862+0000 7ff125d9b640 1 -- 172.21.15.83:0/1149470467 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4104092375 0 0) 0x7ff1280b3430 con 0x7ff1280cf830 2024-03-06T07:25:43.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.862+0000 7ff125d9b640 1 -- 172.21.15.83:0/1149470467 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff114003520 con 0x7ff1280cf830 2024-03-06T07:25:43.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.862+0000 7ff12659c640 1 --2- 172.21.15.83:0/1149470467 >> v2:172.21.15.183:3300/0 conn(0x7ff12815dfe0 0x7ff1280cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.863+0000 7ff125d9b640 1 -- 172.21.15.83:0/1149470467 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1054168627 0 0) 0x7ff114003520 con 0x7ff1280cf830 2024-03-06T07:25:43.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.863+0000 7ff125d9b640 1 -- 172.21.15.83:0/1149470467 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff1280d29c0 msgr2=0x7ff1280b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:43.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.863+0000 7ff125d9b640 1 --2- 172.21.15.83:0/1149470467 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff1280d29c0 0x7ff1280b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.863+0000 7ff125d9b640 1 -- 172.21.15.83:0/1149470467 >> v2:172.21.15.183:3300/0 conn(0x7ff12815dfe0 msgr2=0x7ff1280cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:43.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.863+0000 7ff125d9b640 1 --2- 172.21.15.83:0/1149470467 >> v2:172.21.15.183:3300/0 conn(0x7ff12815dfe0 0x7ff1280cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.863+0000 7ff125d9b640 1 -- 172.21.15.83:0/1149470467 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1280b3710 con 0x7ff1280cf830 2024-03-06T07:25:43.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.863+0000 7ff125d9b640 1 -- 172.21.15.83:0/1149470467 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7ff110001670 con 0x7ff1280cf830 2024-03-06T07:25:43.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.863+0000 7ff125d9b640 1 -- 172.21.15.83:0/1149470467 >> v1:172.21.15.83:6789/0 conn(0x7ff1280cf830 legacy=0x7ff1280d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.863+0000 7ff125d9b640 1 --2- 172.21.15.83:0/1149470467 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff1280d29c0 0x7ff1140039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.863+0000 7ff125d9b640 1 --2- 172.21.15.83:0/1149470467 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff114003f30 0x7ff114004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.863+0000 7ff125d9b640 1 --2- 172.21.15.83:0/1149470467 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff114004d40 0x7ff1140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.863+0000 7ff125d9b640 1 -- 172.21.15.83:0/1149470467 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1280b31e0 con 0x7ff114003f30 2024-03-06T07:25:43.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.863+0000 7ff125d9b640 1 -- 172.21.15.83:0/1149470467 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1280b3d60 con 0x7ff114004d40 2024-03-06T07:25:43.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.863+0000 7ff125d9b640 1 -- 172.21.15.83:0/1149470467 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff11400b9c0 con 0x7ff1280d29c0 2024-03-06T07:25:43.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.863+0000 7ff11dd9b640 1 --2- 172.21.15.83:0/1149470467 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff114003f30 0x7ff114004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.863+0000 7ff12659c640 1 --2- 172.21.15.83:0/1149470467 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff1280d29c0 0x7ff1140039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.863+0000 7ff126d9d640 1 --2- 172.21.15.83:0/1149470467 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff114004d40 0x7ff1140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.863+0000 7ff11dd9b640 1 -- 172.21.15.83:0/1149470467 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff1280d29c0 msgr2=0x7ff1140039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:43.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.863+0000 7ff11dd9b640 1 --2- 172.21.15.83:0/1149470467 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff1280d29c0 0x7ff1140039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.863+0000 7ff11dd9b640 1 -- 172.21.15.83:0/1149470467 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff114004d40 msgr2=0x7ff1140051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:43.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.863+0000 7ff11dd9b640 1 --2- 172.21.15.83:0/1149470467 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff114004d40 0x7ff1140051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.863+0000 7ff11dd9b640 1 -- 172.21.15.83:0/1149470467 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff114003520 con 0x7ff114003f30 2024-03-06T07:25:43.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.863+0000 7ff126d9d640 1 --2- 172.21.15.83:0/1149470467 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff114004d40 0x7ff1140051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:43.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.864+0000 7ff11dd9b640 1 --2- 172.21.15.83:0/1149470467 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff114003f30 0x7ff114004800 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7ff1280cfc70 tx=0x7ff114005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42432b9a86bbf4ba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:43.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.864+0000 7ff125d9b640 1 -- 172.21.15.83:0/1149470467 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff110007d50 con 0x7ff114003f30 2024-03-06T07:25:43.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.864+0000 7ff125d9b640 1 -- 172.21.15.83:0/1149470467 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff110030070 con 0x7ff114003f30 2024-03-06T07:25:43.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.864+0000 7ff125d9b640 1 -- 172.21.15.83:0/1149470467 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff110005d10 con 0x7ff114003f30 2024-03-06T07:25:43.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.865+0000 7ff12cf81640 1 -- 172.21.15.83:0/1149470467 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff114003f30 msgr2=0x7ff114004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.865+0000 7ff12cf81640 1 --2- 172.21.15.83:0/1149470467 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff114003f30 0x7ff114004800 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7ff1280cfc70 tx=0x7ff114005830 comp rx=0 tx=0).stop 2024-03-06T07:25:43.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.865+0000 7ff126d9d640 1 -- 172.21.15.83:0/1149470467 reap_dead start 2024-03-06T07:25:43.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.865+0000 7ff12cf81640 1 -- 172.21.15.83:0/1149470467 shutdown_connections 2024-03-06T07:25:43.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.865+0000 7ff12cf81640 1 -- 172.21.15.83:0/1149470467 >> 172.21.15.83:0/1149470467 conn(0x7ff1280d3cd0 msgr2=0x7ff1280d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:43.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.865+0000 7ff12cf81640 1 -- 172.21.15.83:0/1149470467 shutdown_connections 2024-03-06T07:25:43.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.865+0000 7ff12cf81640 1 -- 172.21.15.83:0/1149470467 wait complete. 2024-03-06T07:25:43.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.866+0000 7ff12cf81640 1 Processor -- start 2024-03-06T07:25:43.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.866+0000 7ff12cf81640 1 -- start start 2024-03-06T07:25:43.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.867+0000 7ff12cf81640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff114003f30 0x7ff1280cd030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.867+0000 7ff12cf81640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff12814eb10 0x7ff12814d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.867+0000 7ff12cf81640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff12814d560 0x7ff12814f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.867+0000 7ff12cf81640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1280d1df0 con 0x7ff12814d560 2024-03-06T07:25:43.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.867+0000 7ff12cf81640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1280d1c70 con 0x7ff12814eb10 2024-03-06T07:25:43.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.867+0000 7ff12cf81640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff12815e3c0 con 0x7ff114003f30 2024-03-06T07:25:43.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.867+0000 7ff126d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff12814d560 0x7ff12814f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.867+0000 7ff126d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff12814d560 0x7ff12814f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52790/0 (socket says 172.21.15.83:52790) 2024-03-06T07:25:43.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.867+0000 7ff126d9d640 1 -- 172.21.15.83:0/605532527 learned_addr learned my addr 172.21.15.83:0/605532527 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:43.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.867+0000 7ff12659c640 1 --2- 172.21.15.83:0/605532527 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff114003f30 0x7ff1280cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.867+0000 7ff11dd9b640 1 --2- 172.21.15.83:0/605532527 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff12814eb10 0x7ff12814d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.867+0000 7ff126d9d640 1 -- 172.21.15.83:0/605532527 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff114003f30 msgr2=0x7ff1280cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.867+0000 7ff126d9d640 1 --2- 172.21.15.83:0/605532527 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff114003f30 0x7ff1280cd030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.867+0000 7ff126d9d640 1 -- 172.21.15.83:0/605532527 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff12814eb10 msgr2=0x7ff12814d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.867+0000 7ff126d9d640 1 --2- 172.21.15.83:0/605532527 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff12814eb10 0x7ff12814d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.867+0000 7ff126d9d640 1 -- 172.21.15.83:0/605532527 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff12814da70 con 0x7ff12814d560 2024-03-06T07:25:43.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.867+0000 7ff11dd9b640 1 --2- 172.21.15.83:0/605532527 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff12814eb10 0x7ff12814d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:43.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.868+0000 7ff126d9d640 1 --2- 172.21.15.83:0/605532527 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff12814d560 0x7ff12814f040 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7ff12808c940 tx=0x7ff12000f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:43.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.868+0000 7ff11f7fe640 1 -- 172.21.15.83:0/605532527 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff12001c070 con 0x7ff12814d560 2024-03-06T07:25:43.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.868+0000 7ff11f7fe640 1 -- 172.21.15.83:0/605532527 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff120017400 con 0x7ff12814d560 2024-03-06T07:25:43.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.868+0000 7ff12cf81640 1 -- 172.21.15.83:0/605532527 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff12814dd50 con 0x7ff12814d560 2024-03-06T07:25:43.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.868+0000 7ff11f7fe640 1 -- 172.21.15.83:0/605532527 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1200177f0 con 0x7ff12814d560 2024-03-06T07:25:43.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.868+0000 7ff12cf81640 1 -- 172.21.15.83:0/605532527 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff12814e310 con 0x7ff12814d560 2024-03-06T07:25:43.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.869+0000 7ff12cf81640 1 -- 172.21.15.83:0/605532527 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff12804f330 con 0x7ff12814d560 2024-03-06T07:25:43.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.869+0000 7ff11f7fe640 1 -- 172.21.15.83:0/605532527 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff12001f070 con 0x7ff12814d560 2024-03-06T07:25:43.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.869+0000 7ff11f7fe640 1 --2- 172.21.15.83:0/605532527 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff10806b4d0 0x7ff10806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.869+0000 7ff11f7fe640 1 -- 172.21.15.83:0/605532527 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff12008a0d0 con 0x7ff12814d560 2024-03-06T07:25:43.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.869+0000 7ff12659c640 1 --2- 172.21.15.83:0/605532527 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff10806b4d0 0x7ff10806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.870+0000 7ff12659c640 1 --2- 172.21.15.83:0/605532527 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff10806b4d0 0x7ff10806d990 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7ff118009bc0 tx=0x7ff118009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:43.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:43.874+0000 7ff11f7fe640 1 -- 172.21.15.83:0/605532527 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff120012070 con 0x7ff12814d560 2024-03-06T07:25:44.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.088+0000 7ff12cf81640 1 -- 172.21.15.83:0/605532527 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff1280b4ba0 con 0x7ff12814d560 2024-03-06T07:25:44.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.089+0000 7ff11f7fe640 1 -- 172.21.15.83:0/605532527 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4982 (secure 0 0 0) 0x7ff120058fa0 con 0x7ff12814d560 2024-03-06T07:25:44.091 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:25:44.091 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":32,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6010,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/2142942385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2142942385},{"type":"v1","addr":"172.21.15.183:6834","nonce":2142942385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":6013,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3742438064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":3742438064},{"type":"v1","addr":"172.21.15.183:6835","nonce":3742438064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":6540,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/1549894851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1549894851},{"type":"v1","addr":"172.21.15.83:6836","nonce":1549894851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:12.481670+0000","modified":"2024-03-06T07:25:15.392072+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6543},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6543":{"gid":6543,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/602775648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":602775648},{"type":"v1","addr":"172.21.15.83:6837","nonce":602775648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6543,"qdb_cluster":[6543]},"id":5}]} 2024-03-06T07:25:44.091 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 32 2024-03-06T07:25:44.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.090+0000 7ff12cf81640 1 -- 172.21.15.83:0/605532527 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff10806b4d0 msgr2=0x7ff10806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.090+0000 7ff12cf81640 1 --2- 172.21.15.83:0/605532527 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff10806b4d0 0x7ff10806d990 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7ff118009bc0 tx=0x7ff118009190 comp rx=0 tx=0).stop 2024-03-06T07:25:44.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.091+0000 7ff12cf81640 1 -- 172.21.15.83:0/605532527 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff12814d560 msgr2=0x7ff12814f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.091+0000 7ff12cf81640 1 --2- 172.21.15.83:0/605532527 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff12814d560 0x7ff12814f040 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7ff12808c940 tx=0x7ff12000f4a0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.091+0000 7ff12cf81640 1 -- 172.21.15.83:0/605532527 shutdown_connections 2024-03-06T07:25:44.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.091+0000 7ff12cf81640 1 --2- 172.21.15.83:0/605532527 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff10806b4d0 0x7ff10806d990 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.091+0000 7ff12cf81640 1 --2- 172.21.15.83:0/605532527 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff12814d560 0x7ff12814f040 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.091+0000 7ff12cf81640 1 --2- 172.21.15.83:0/605532527 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff12814eb10 0x7ff12814d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.091+0000 7ff12cf81640 1 --2- 172.21.15.83:0/605532527 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff114003f30 0x7ff1280cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.091+0000 7ff12cf81640 1 -- 172.21.15.83:0/605532527 >> 172.21.15.83:0/605532527 conn(0x7ff1280d3cd0 msgr2=0x7ff1280b40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:44.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.091+0000 7ff12cf81640 1 -- 172.21.15.83:0/605532527 shutdown_connections 2024-03-06T07:25:44.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.091+0000 7ff12cf81640 1 -- 172.21.15.83:0/605532527 wait complete. 2024-03-06T07:25:44.103 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:44.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.204+0000 7fe6068a4640 1 Processor -- start 2024-03-06T07:25:44.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.204+0000 7fe6068a4640 1 -- start start 2024-03-06T07:25:44.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.205+0000 7fe6068a4640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fe6000d02c0 0x7fe6000cf850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.205+0000 7fe6068a4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe6000d06a0 0x7fe6000cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.205+0000 7fe6068a4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe60015c620 con 0x7fe6000ce130 2024-03-06T07:25:44.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.205+0000 7fe6068a4640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6000b50b0 con 0x7fe6000d06a0 2024-03-06T07:25:44.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.205+0000 7fe6068a4640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6000cd800 con 0x7fe6000d02c0 2024-03-06T07:25:44.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.205+0000 7fe5ff7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe6000d06a0 0x7fe6000cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.205+0000 7fe5ff7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe6000d06a0 0x7fe6000cd0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52806/0 (socket says 172.21.15.83:52806) 2024-03-06T07:25:44.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.205+0000 7fe5ff7fe640 1 -- 172.21.15.83:0/2176149917 learned_addr learned my addr 172.21.15.83:0/2176149917 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:44.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.205+0000 7fe5fffff640 1 --2- 172.21.15.83:0/2176149917 >> v2:172.21.15.183:3300/0 conn(0x7fe6000d02c0 0x7fe6000cf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.205+0000 7fe5ff7fe640 1 -- 172.21.15.83:0/2176149917 >> v2:172.21.15.183:3300/0 conn(0x7fe6000d02c0 msgr2=0x7fe6000cf850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:44.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.205+0000 7fe5ff7fe640 1 --2- 172.21.15.83:0/2176149917 >> v2:172.21.15.183:3300/0 conn(0x7fe6000d02c0 0x7fe6000cf850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.205+0000 7fe5ff7fe640 1 -- 172.21.15.83:0/2176149917 >> v1:172.21.15.183:6789/0 conn(0x7fe6000ce130 legacy=0x7fe6000ce540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:44.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.206+0000 7fe5ff7fe640 1 -- 172.21.15.83:0/2176149917 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6000b4a60 con 0x7fe6000d06a0 2024-03-06T07:25:44.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.206+0000 7fe5ff7fe640 1 --2- 172.21.15.83:0/2176149917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6000d06a0 0x7fe6000cd0d0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fe600071cf0 tx=0x7fe5f402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7c43105c3cfab8c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:44.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.206+0000 7fe5feffd640 1 -- 172.21.15.83:0/2176149917 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f4035020 con 0x7fe6000d06a0 2024-03-06T07:25:44.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.206+0000 7fe5feffd640 1 -- 172.21.15.83:0/2176149917 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f4031870 con 0x7fe6000d06a0 2024-03-06T07:25:44.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.206+0000 7fe5feffd640 1 -- 172.21.15.83:0/2176149917 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f4031c40 con 0x7fe6000d06a0 2024-03-06T07:25:44.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.207+0000 7fe6068a4640 1 -- 172.21.15.83:0/2176149917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6000d06a0 msgr2=0x7fe6000cd0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.207+0000 7fe6068a4640 1 --2- 172.21.15.83:0/2176149917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6000d06a0 0x7fe6000cd0d0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fe600071cf0 tx=0x7fe5f402b910 comp rx=0 tx=0).stop 2024-03-06T07:25:44.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.207+0000 7fe6068a4640 1 -- 172.21.15.83:0/2176149917 shutdown_connections 2024-03-06T07:25:44.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.207+0000 7fe6068a4640 1 --2- 172.21.15.83:0/2176149917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6000d06a0 0x7fe6000cd0d0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.207+0000 7fe6068a4640 1 --2- 172.21.15.83:0/2176149917 >> v2:172.21.15.183:3300/0 conn(0x7fe6000d02c0 0x7fe6000cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.207+0000 7fe6068a4640 1 -- 172.21.15.83:0/2176149917 >> 172.21.15.83:0/2176149917 conn(0x7fe6000b35a0 msgr2=0x7fe6000b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:44.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.207+0000 7fe6068a4640 1 -- 172.21.15.83:0/2176149917 shutdown_connections 2024-03-06T07:25:44.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.207+0000 7fe6068a4640 1 -- 172.21.15.83:0/2176149917 wait complete. 2024-03-06T07:25:44.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.208+0000 7fe6068a4640 1 Processor -- start 2024-03-06T07:25:44.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.208+0000 7fe6068a4640 1 -- start start 2024-03-06T07:25:44.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.209+0000 7fe6068a4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6000ce130 0x7fe600147cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.209+0000 7fe6068a4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6000d02c0 0x7fe6000c55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.209+0000 7fe6068a4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe6000d06a0 0x7fe6000c7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.209+0000 7fe6068a4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6000b66d0 con 0x7fe6000d02c0 2024-03-06T07:25:44.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.209+0000 7fe6068a4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6000b6550 con 0x7fe6000ce130 2024-03-06T07:25:44.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.209+0000 7fe6068a4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6000b6850 con 0x7fe6000d06a0 2024-03-06T07:25:44.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.209+0000 7fe5ff7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6000d02c0 0x7fe6000c55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.209+0000 7fe5ff7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6000d02c0 0x7fe6000c55a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52818/0 (socket says 172.21.15.83:52818) 2024-03-06T07:25:44.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.209+0000 7fe5ff7fe640 1 -- 172.21.15.83:0/4107414348 learned_addr learned my addr 172.21.15.83:0/4107414348 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:44.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.209+0000 7fe5fffff640 1 --2- 172.21.15.83:0/4107414348 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6000ce130 0x7fe600147cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.209+0000 7fe604e41640 1 --2- 172.21.15.83:0/4107414348 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe6000d06a0 0x7fe6000c7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.209+0000 7fe5ff7fe640 1 -- 172.21.15.83:0/4107414348 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe6000d06a0 msgr2=0x7fe6000c7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.209+0000 7fe5ff7fe640 1 --2- 172.21.15.83:0/4107414348 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe6000d06a0 0x7fe6000c7090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.209+0000 7fe5ff7fe640 1 -- 172.21.15.83:0/4107414348 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6000ce130 msgr2=0x7fe600147cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.209+0000 7fe5ff7fe640 1 --2- 172.21.15.83:0/4107414348 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6000ce130 0x7fe600147cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.209+0000 7fe5ff7fe640 1 -- 172.21.15.83:0/4107414348 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe60006d640 con 0x7fe6000d02c0 2024-03-06T07:25:44.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.209+0000 7fe5fffff640 1 --2- 172.21.15.83:0/4107414348 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6000ce130 0x7fe600147cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:44.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.210+0000 7fe5ff7fe640 1 --2- 172.21.15.83:0/4107414348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6000d02c0 0x7fe6000c55a0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fe5f402b950 tx=0x7fe5f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:44.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.210+0000 7fe5fcff9640 1 -- 172.21.15.83:0/4107414348 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f4040070 con 0x7fe6000d02c0 2024-03-06T07:25:44.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.210+0000 7fe5fcff9640 1 -- 172.21.15.83:0/4107414348 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f4030d00 con 0x7fe6000d02c0 2024-03-06T07:25:44.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.210+0000 7fe6068a4640 1 -- 172.21.15.83:0/4107414348 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe60006d920 con 0x7fe6000d02c0 2024-03-06T07:25:44.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.210+0000 7fe5fcff9640 1 -- 172.21.15.83:0/4107414348 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f4035040 con 0x7fe6000d02c0 2024-03-06T07:25:44.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.210+0000 7fe6068a4640 1 -- 172.21.15.83:0/4107414348 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6001e5dd0 con 0x7fe6000d02c0 2024-03-06T07:25:44.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.211+0000 7fe6068a4640 1 -- 172.21.15.83:0/4107414348 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe60004f330 con 0x7fe6000d02c0 2024-03-06T07:25:44.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.211+0000 7fe5fcff9640 1 -- 172.21.15.83:0/4107414348 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe5f404b060 con 0x7fe6000d02c0 2024-03-06T07:25:44.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.212+0000 7fe5fcff9640 1 --2- 172.21.15.83:0/4107414348 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe5ec06b4d0 0x7fe5ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.212+0000 7fe5fcff9640 1 -- 172.21.15.83:0/4107414348 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe5f40abf80 con 0x7fe6000d02c0 2024-03-06T07:25:44.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.212+0000 7fe5fffff640 1 --2- 172.21.15.83:0/4107414348 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe5ec06b4d0 0x7fe5ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.212+0000 7fe5fffff640 1 --2- 172.21.15.83:0/4107414348 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe5ec06b4d0 0x7fe5ec06d990 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fe5f0006fd0 tx=0x7fe5f0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:44.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.216+0000 7fe5fcff9640 1 -- 172.21.15.83:0/4107414348 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe5f407adc0 con 0x7fe6000d02c0 2024-03-06T07:25:44.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.394+0000 7fe6068a4640 1 -- 172.21.15.83:0/4107414348 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6000cf3d0 con 0x7fe6000d02c0 2024-03-06T07:25:44.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.395+0000 7fe5fcff9640 1 -- 172.21.15.83:0/4107414348 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+16778 (secure 0 0 0) 0x7fe5f407a6c0 con 0x7fe6000d02c0 2024-03-06T07:25:44.397 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:25:44.398 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":71,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:25:40.924823+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:10.701416+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:11.450797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.1e","mappings":[{"from":6,"to":0}]},{"pgid":"11.3e","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:44.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.397+0000 7fe6068a4640 1 -- 172.21.15.83:0/4107414348 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe5ec06b4d0 msgr2=0x7fe5ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.397+0000 7fe6068a4640 1 --2- 172.21.15.83:0/4107414348 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe5ec06b4d0 0x7fe5ec06d990 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fe5f0006fd0 tx=0x7fe5f0008040 comp rx=0 tx=0).stop 2024-03-06T07:25:44.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.397+0000 7fe6068a4640 1 -- 172.21.15.83:0/4107414348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6000d02c0 msgr2=0x7fe6000c55a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.397+0000 7fe6068a4640 1 --2- 172.21.15.83:0/4107414348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6000d02c0 0x7fe6000c55a0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fe5f402b950 tx=0x7fe5f40046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.397+0000 7fe6068a4640 1 -- 172.21.15.83:0/4107414348 shutdown_connections 2024-03-06T07:25:44.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.397+0000 7fe6068a4640 1 --2- 172.21.15.83:0/4107414348 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe5ec06b4d0 0x7fe5ec06d990 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.397+0000 7fe6068a4640 1 --2- 172.21.15.83:0/4107414348 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe6000d06a0 0x7fe6000c7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.397+0000 7fe6068a4640 1 --2- 172.21.15.83:0/4107414348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6000d02c0 0x7fe6000c55a0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.397+0000 7fe6068a4640 1 --2- 172.21.15.83:0/4107414348 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6000ce130 0x7fe600147cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.397+0000 7fe6068a4640 1 -- 172.21.15.83:0/4107414348 >> 172.21.15.83:0/4107414348 conn(0x7fe6000b35a0 msgr2=0x7fe60006fb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:44.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.398+0000 7fe6068a4640 1 -- 172.21.15.83:0/4107414348 shutdown_connections 2024-03-06T07:25:44.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.398+0000 7fe6068a4640 1 -- 172.21.15.83:0/4107414348 wait complete. 2024-03-06T07:25:44.410 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:25:44.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.531+0000 7f5ee6a4e640 1 Processor -- start 2024-03-06T07:25:44.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.531+0000 7f5ee6a4e640 1 -- start start 2024-03-06T07:25:44.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.531+0000 7f5ee6a4e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5ee01614a0 0x7f5ee006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.531+0000 7f5ee6a4e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5ee00b1a20 0x7f5ee00b5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.531+0000 7f5ee6a4e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ee00af4b0 con 0x7f5ee00b0270 2024-03-06T07:25:44.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.531+0000 7f5ee6a4e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ee00af790 con 0x7f5ee01614a0 2024-03-06T07:25:44.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.531+0000 7f5ee6a4e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee00afa70 con 0x7f5ee00b1a20 2024-03-06T07:25:44.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.532+0000 7f5edffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5ee01614a0 0x7f5ee006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.532+0000 7f5edffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5ee01614a0 0x7f5ee006d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52824/0 (socket says 172.21.15.83:52824) 2024-03-06T07:25:44.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.532+0000 7f5edffff640 1 -- 172.21.15.83:0/2843404936 learned_addr learned my addr 172.21.15.83:0/2843404936 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:44.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.532+0000 7f5edf7fe640 1 --1- 172.21.15.83:0/2843404936 >> v1:172.21.15.83:6789/0 conn(0x7f5ee00b0270 0x7f5ee00b1310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52580/0 (socket says 172.21.15.83:52580) 2024-03-06T07:25:44.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.532+0000 7f5ee4feb640 1 --2- 172.21.15.83:0/2843404936 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5ee00b1a20 0x7f5ee00b5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.532+0000 7f5edffff640 1 -- 172.21.15.83:0/2843404936 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5ee00b1a20 msgr2=0x7f5ee00b5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:44.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.532+0000 7f5edffff640 1 --2- 172.21.15.83:0/2843404936 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5ee00b1a20 0x7f5ee00b5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.532+0000 7f5edffff640 1 -- 172.21.15.83:0/2843404936 >> v1:172.21.15.83:6789/0 conn(0x7f5ee00b0270 legacy=0x7f5ee00b1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.532+0000 7f5edffff640 1 -- 172.21.15.83:0/2843404936 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee00afd50 con 0x7f5ee01614a0 2024-03-06T07:25:44.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.532+0000 7f5edeffd640 1 -- 172.21.15.83:0/2843404936 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2612177037 0 0) 0x7f5ee00af4b0 con 0x7f5ee00b0270 2024-03-06T07:25:44.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.533+0000 7f5edffff640 1 --2- 172.21.15.83:0/2843404936 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ee01614a0 0x7f5ee006d3c0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f5ed0009870 tx=0x7f5ed002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86553c5b72889a49 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:44.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.533+0000 7f5edeffd640 1 -- 172.21.15.83:0/2843404936 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ed0035020 con 0x7f5ee01614a0 2024-03-06T07:25:44.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.533+0000 7f5edeffd640 1 -- 172.21.15.83:0/2843404936 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ed0031640 con 0x7f5ee01614a0 2024-03-06T07:25:44.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.533+0000 7f5edeffd640 1 -- 172.21.15.83:0/2843404936 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ed002d040 con 0x7f5ee01614a0 2024-03-06T07:25:44.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.533+0000 7f5ee6a4e640 1 -- 172.21.15.83:0/2843404936 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ee01614a0 msgr2=0x7f5ee006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.533+0000 7f5ee6a4e640 1 --2- 172.21.15.83:0/2843404936 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ee01614a0 0x7f5ee006d3c0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f5ed0009870 tx=0x7f5ed002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.533+0000 7f5ee6a4e640 1 -- 172.21.15.83:0/2843404936 shutdown_connections 2024-03-06T07:25:44.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.533+0000 7f5ee6a4e640 1 --2- 172.21.15.83:0/2843404936 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5ee00b1a20 0x7f5ee00b5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.533+0000 7f5ee6a4e640 1 --2- 172.21.15.83:0/2843404936 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ee01614a0 0x7f5ee006d3c0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.533+0000 7f5ee6a4e640 1 -- 172.21.15.83:0/2843404936 >> 172.21.15.83:0/2843404936 conn(0x7f5ee009a9f0 msgr2=0x7f5ee009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:44.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.533+0000 7f5ee6a4e640 1 -- 172.21.15.83:0/2843404936 shutdown_connections 2024-03-06T07:25:44.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.533+0000 7f5ee6a4e640 1 -- 172.21.15.83:0/2843404936 wait complete. 2024-03-06T07:25:44.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.535+0000 7f5ee6a4e640 1 Processor -- start 2024-03-06T07:25:44.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.535+0000 7f5ee6a4e640 1 -- start start 2024-03-06T07:25:44.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.535+0000 7f5ee6a4e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ee00b0270 0x7f5ee01f3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.535+0000 7f5ee6a4e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5ee00b1a20 0x7f5ee014fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.535+0000 7f5ee6a4e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ee01614a0 0x7f5ee0152800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.535+0000 7f5ee6a4e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee00b1e30 con 0x7f5ee01614a0 2024-03-06T07:25:44.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.535+0000 7f5ee6a4e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee00b06e0 con 0x7f5ee00b0270 2024-03-06T07:25:44.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.535+0000 7f5ee6a4e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee00b5a50 con 0x7f5ee00b1a20 2024-03-06T07:25:44.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.535+0000 7f5ee4feb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ee01614a0 0x7f5ee0152800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.535+0000 7f5ee4feb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ee01614a0 0x7f5ee0152800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52840/0 (socket says 172.21.15.83:52840) 2024-03-06T07:25:44.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.535+0000 7f5ee4feb640 1 -- 172.21.15.83:0/1663884396 learned_addr learned my addr 172.21.15.83:0/1663884396 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:44.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.536+0000 7f5edf7fe640 1 --2- 172.21.15.83:0/1663884396 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5ee00b1a20 0x7f5ee014fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.536+0000 7f5edffff640 1 --2- 172.21.15.83:0/1663884396 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ee00b0270 0x7f5ee01f3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.536+0000 7f5ee4feb640 1 -- 172.21.15.83:0/1663884396 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5ee00b1a20 msgr2=0x7f5ee014fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.536+0000 7f5ee4feb640 1 --2- 172.21.15.83:0/1663884396 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5ee00b1a20 0x7f5ee014fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.536+0000 7f5ee4feb640 1 -- 172.21.15.83:0/1663884396 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ee00b0270 msgr2=0x7f5ee01f3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.536+0000 7f5ee4feb640 1 --2- 172.21.15.83:0/1663884396 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ee00b0270 0x7f5ee01f3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.536+0000 7f5ee4feb640 1 -- 172.21.15.83:0/1663884396 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee009ee10 con 0x7f5ee01614a0 2024-03-06T07:25:44.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.536+0000 7f5ee4feb640 1 --2- 172.21.15.83:0/1663884396 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ee01614a0 0x7f5ee0152800 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f5ed8007e40 tx=0x7f5ed800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:44.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.536+0000 7f5edcff9640 1 -- 172.21.15.83:0/1663884396 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ed8014620 con 0x7f5ee01614a0 2024-03-06T07:25:44.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.536+0000 7f5edcff9640 1 -- 172.21.15.83:0/1663884396 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ed8019070 con 0x7f5ee01614a0 2024-03-06T07:25:44.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.536+0000 7f5ee6a4e640 1 -- 172.21.15.83:0/1663884396 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ee009f0b0 con 0x7f5ee01614a0 2024-03-06T07:25:44.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.536+0000 7f5edcff9640 1 -- 172.21.15.83:0/1663884396 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ed8014a20 con 0x7f5ee01614a0 2024-03-06T07:25:44.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.536+0000 7f5ee6a4e640 1 -- 172.21.15.83:0/1663884396 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ee009f670 con 0x7f5ee01614a0 2024-03-06T07:25:44.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.537+0000 7f5ee6a4e640 1 -- 172.21.15.83:0/1663884396 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ee0160440 con 0x7f5ee01614a0 2024-03-06T07:25:44.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.537+0000 7f5edcff9640 1 -- 172.21.15.83:0/1663884396 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5ed8014bc0 con 0x7f5ee01614a0 2024-03-06T07:25:44.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.538+0000 7f5edcff9640 1 --2- 172.21.15.83:0/1663884396 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5ecc06b4d0 0x7f5ecc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.538+0000 7f5edcff9640 1 -- 172.21.15.83:0/1663884396 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5ed808aaf0 con 0x7f5ee01614a0 2024-03-06T07:25:44.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.538+0000 7f5edffff640 1 --2- 172.21.15.83:0/1663884396 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5ecc06b4d0 0x7f5ecc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.538+0000 7f5edffff640 1 --2- 172.21.15.83:0/1663884396 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5ecc06b4d0 0x7f5ecc06d990 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f5ed0036000 tx=0x7f5ed0008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:44.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.546+0000 7f5edcff9640 1 -- 172.21.15.83:0/1663884396 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5ee0160440 con 0x7f5ee01614a0 2024-03-06T07:25:44.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.760+0000 7f5ee6a4e640 1 -- 172.21.15.83:0/1663884396 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f5ee0161000 con 0x7f5ee01614a0 2024-03-06T07:25:44.763 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-03-06T07:25:44.761+0000 7fb0dae11640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:25:44.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.762+0000 7f5edcff9640 1 -- 172.21.15.83:0/1663884396 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v33) v1 ==== 161+0+0 (secure 0 0 0) 0x7f5ee0161000 con 0x7f5ee01614a0 2024-03-06T07:25:44.764 INFO:teuthology.orchestra.run.smithi083.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:25:44.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.765+0000 7f5ee6a4e640 1 -- 172.21.15.83:0/1663884396 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5ecc06b4d0 msgr2=0x7f5ecc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.765+0000 7f5ee6a4e640 1 --2- 172.21.15.83:0/1663884396 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5ecc06b4d0 0x7f5ecc06d990 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f5ed0036000 tx=0x7f5ed0008030 comp rx=0 tx=0).stop 2024-03-06T07:25:44.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.765+0000 7f5ee6a4e640 1 -- 172.21.15.83:0/1663884396 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ee01614a0 msgr2=0x7f5ee0152800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.765+0000 7f5ee6a4e640 1 --2- 172.21.15.83:0/1663884396 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ee01614a0 0x7f5ee0152800 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f5ed8007e40 tx=0x7f5ed800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.765+0000 7f5ee6a4e640 1 -- 172.21.15.83:0/1663884396 shutdown_connections 2024-03-06T07:25:44.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.765+0000 7f5ee6a4e640 1 --2- 172.21.15.83:0/1663884396 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5ecc06b4d0 0x7f5ecc06d990 secure :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f5ed0036000 tx=0x7f5ed0008030 comp rx=0 tx=0).stop 2024-03-06T07:25:44.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.765+0000 7f5ee6a4e640 1 --2- 172.21.15.83:0/1663884396 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ee01614a0 0x7f5ee0152800 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.765+0000 7f5ee6a4e640 1 --2- 172.21.15.83:0/1663884396 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5ee00b1a20 0x7f5ee014fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.765+0000 7f5ee6a4e640 1 --2- 172.21.15.83:0/1663884396 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ee00b0270 0x7f5ee01f3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.765+0000 7f5ee6a4e640 1 -- 172.21.15.83:0/1663884396 >> 172.21.15.83:0/1663884396 conn(0x7f5ee009a9f0 msgr2=0x7f5ee006fdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:44.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.765+0000 7f5ee6a4e640 1 -- 172.21.15.83:0/1663884396 shutdown_connections 2024-03-06T07:25:44.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.765+0000 7f5ee6a4e640 1 -- 172.21.15.83:0/1663884396 wait complete. 2024-03-06T07:25:44.779 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:25:44.803 INFO:tasks.ceph.mds.a.smithi083.stdout:starting mds.a at 2024-03-06T07:25:44.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.881+0000 7f98a6871640 1 Processor -- start 2024-03-06T07:25:44.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.881+0000 7f98a6871640 1 -- start start 2024-03-06T07:25:44.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.881+0000 7f98a6871640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98a01614d0 0x7f98a00d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.881+0000 7f98a6871640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f98a00d1910 0x7f98a00d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.881+0000 7f98a6871640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98a00d2740 con 0x7f98a00d22b0 2024-03-06T07:25:44.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.881+0000 7f98a6871640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98a01618b0 con 0x7f98a00d1910 2024-03-06T07:25:44.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.881+0000 7f98a6871640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98a00cf400 con 0x7f98a01614d0 2024-03-06T07:25:44.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.882+0000 7f989ffff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98a01614d0 0x7f98a00d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.882+0000 7f989f7fe640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f98a00d1910 0x7f98a00d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.882+0000 7f989ffff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98a01614d0 0x7f98a00d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:56468/0 (socket says 172.21.15.83:56468) 2024-03-06T07:25:44.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.882+0000 7f989ffff640 1 -- 172.21.15.83:0/1974012300 learned_addr learned my addr 172.21.15.83:0/1974012300 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:44.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.882+0000 7f989ffff640 1 -- 172.21.15.83:0/1974012300 >> v2:172.21.15.183:3300/0 conn(0x7f98a00d1910 msgr2=0x7f98a00d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:44.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.882+0000 7f989ffff640 1 --2- 172.21.15.83:0/1974012300 >> v2:172.21.15.183:3300/0 conn(0x7f98a00d1910 0x7f98a00d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.882+0000 7f989ffff640 1 -- 172.21.15.83:0/1974012300 >> v1:172.21.15.183:6789/0 conn(0x7f98a00d22b0 legacy=0x7f98a00d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.882+0000 7f989ffff640 1 -- 172.21.15.83:0/1974012300 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a00cf6a0 con 0x7f98a01614d0 2024-03-06T07:25:44.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.883+0000 7f989ffff640 1 --2- 172.21.15.83:0/1974012300 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98a01614d0 0x7f98a00d4860 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9894009870 tx=0x7f989402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42745e30c1d75648 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:44.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.883+0000 7f989effd640 1 -- 172.21.15.83:0/1974012300 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9894034070 con 0x7f98a01614d0 2024-03-06T07:25:44.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.883+0000 7f989effd640 1 -- 172.21.15.83:0/1974012300 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9894030840 con 0x7f98a01614d0 2024-03-06T07:25:44.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.883+0000 7f989effd640 1 -- 172.21.15.83:0/1974012300 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9894030c10 con 0x7f98a01614d0 2024-03-06T07:25:44.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.884+0000 7f98a6871640 1 -- 172.21.15.83:0/1974012300 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98a01614d0 msgr2=0x7f98a00d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.884+0000 7f98a6871640 1 --2- 172.21.15.83:0/1974012300 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98a01614d0 0x7f98a00d4860 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9894009870 tx=0x7f989402b980 comp rx=0 tx=0).stop 2024-03-06T07:25:44.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.884+0000 7f98a6871640 1 -- 172.21.15.83:0/1974012300 shutdown_connections 2024-03-06T07:25:44.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.884+0000 7f98a6871640 1 --2- 172.21.15.83:0/1974012300 >> v2:172.21.15.183:3300/0 conn(0x7f98a00d1910 0x7f98a00d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.884+0000 7f98a6871640 1 --2- 172.21.15.83:0/1974012300 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98a01614d0 0x7f98a00d4860 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.884+0000 7f98a6871640 1 -- 172.21.15.83:0/1974012300 >> 172.21.15.83:0/1974012300 conn(0x7f98a00d3d40 msgr2=0x7f98a00d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:44.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.884+0000 7f98a6871640 1 -- 172.21.15.83:0/1974012300 shutdown_connections 2024-03-06T07:25:44.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.884+0000 7f98a6871640 1 -- 172.21.15.83:0/1974012300 wait complete. 2024-03-06T07:25:44.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.885+0000 7f98a6871640 1 Processor -- start 2024-03-06T07:25:44.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.885+0000 7f98a6871640 1 -- start start 2024-03-06T07:25:44.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.886+0000 7f98a6871640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98a00d1910 0x7f98a00cd690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.886+0000 7f98a6871640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98a00d22b0 0x7f98a00cb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.886+0000 7f98a6871640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98a01614d0 0x7f98a01f3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.886+0000 7f98a6871640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a00b0910 con 0x7f98a01614d0 2024-03-06T07:25:44.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.886+0000 7f98a6871640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a00cfc60 con 0x7f98a00d1910 2024-03-06T07:25:44.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.886+0000 7f98a6871640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98a00b1e40 con 0x7f98a00d22b0 2024-03-06T07:25:44.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.886+0000 7f98a4e0e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98a01614d0 0x7f98a01f3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.886+0000 7f98a4e0e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98a01614d0 0x7f98a01f3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52876/0 (socket says 172.21.15.83:52876) 2024-03-06T07:25:44.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.886+0000 7f98a4e0e640 1 -- 172.21.15.83:0/3107941311 learned_addr learned my addr 172.21.15.83:0/3107941311 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:44.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.886+0000 7f989f7fe640 1 --2- 172.21.15.83:0/3107941311 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98a00d22b0 0x7f98a00cb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.886+0000 7f989ffff640 1 --2- 172.21.15.83:0/3107941311 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98a00d1910 0x7f98a00cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.886+0000 7f98a4e0e640 1 -- 172.21.15.83:0/3107941311 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98a00d22b0 msgr2=0x7f98a00cb8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.886+0000 7f98a4e0e640 1 --2- 172.21.15.83:0/3107941311 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98a00d22b0 0x7f98a00cb8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.886+0000 7f98a4e0e640 1 -- 172.21.15.83:0/3107941311 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98a00d1910 msgr2=0x7f98a00cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.886+0000 7f98a4e0e640 1 --2- 172.21.15.83:0/3107941311 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98a00d1910 0x7f98a00cd690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.886+0000 7f98a4e0e640 1 -- 172.21.15.83:0/3107941311 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a00cae10 con 0x7f98a01614d0 2024-03-06T07:25:44.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.886+0000 7f98a4e0e640 1 --2- 172.21.15.83:0/3107941311 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98a01614d0 0x7f98a01f3d40 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f989800c7a0 tx=0x7f989800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:44.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.886+0000 7f989cff9640 1 -- 172.21.15.83:0/3107941311 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9898003b70 con 0x7f98a01614d0 2024-03-06T07:25:44.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.886+0000 7f989cff9640 1 -- 172.21.15.83:0/3107941311 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9898003d30 con 0x7f98a01614d0 2024-03-06T07:25:44.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.887+0000 7f989cff9640 1 -- 172.21.15.83:0/3107941311 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989800aa90 con 0x7f98a01614d0 2024-03-06T07:25:44.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.887+0000 7f98a6871640 1 -- 172.21.15.83:0/3107941311 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98a01f4850 con 0x7f98a01614d0 2024-03-06T07:25:44.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.887+0000 7f98a6871640 1 -- 172.21.15.83:0/3107941311 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98a00cb0b0 con 0x7f98a01614d0 2024-03-06T07:25:44.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.887+0000 7f98a6871640 1 -- 172.21.15.83:0/3107941311 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98a00cdc20 con 0x7f98a01614d0 2024-03-06T07:25:44.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.888+0000 7f989cff9640 1 -- 172.21.15.83:0/3107941311 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f989800ac30 con 0x7f98a01614d0 2024-03-06T07:25:44.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.888+0000 7f989cff9640 1 --2- 172.21.15.83:0/3107941311 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f988c06b4d0 0x7f988c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.888+0000 7f989cff9640 1 -- 172.21.15.83:0/3107941311 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f989808a070 con 0x7f98a01614d0 2024-03-06T07:25:44.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.888+0000 7f989ffff640 1 --2- 172.21.15.83:0/3107941311 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f988c06b4d0 0x7f988c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.889+0000 7f989ffff640 1 --2- 172.21.15.83:0/3107941311 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f988c06b4d0 0x7f988c06d990 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f9894036000 tx=0x7f989403b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:44.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:44.893+0000 7f989cff9640 1 -- 172.21.15.83:0/3107941311 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f98a00d2740 con 0x7f98a01614d0 2024-03-06T07:25:45.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.102+0000 7f98a6871640 1 -- 172.21.15.83:0/3107941311 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f98a0068ff0 con 0x7f98a01614d0 2024-03-06T07:25:45.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.764+0000 7f989cff9640 1 -- 172.21.15.83:0/3107941311 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v34) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9898058820 con 0x7f98a01614d0 2024-03-06T07:25:45.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.767+0000 7f98a6871640 1 -- 172.21.15.83:0/3107941311 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f988c06b4d0 msgr2=0x7f988c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.767+0000 7f98a6871640 1 --2- 172.21.15.83:0/3107941311 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f988c06b4d0 0x7f988c06d990 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f9894036000 tx=0x7f989403b040 comp rx=0 tx=0).stop 2024-03-06T07:25:45.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.767+0000 7f98a6871640 1 -- 172.21.15.83:0/3107941311 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98a01614d0 msgr2=0x7f98a01f3d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.767+0000 7f98a6871640 1 --2- 172.21.15.83:0/3107941311 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98a01614d0 0x7f98a01f3d40 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f989800c7a0 tx=0x7f989800cc60 comp rx=0 tx=0).stop 2024-03-06T07:25:45.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.767+0000 7f98a6871640 1 -- 172.21.15.83:0/3107941311 shutdown_connections 2024-03-06T07:25:45.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.767+0000 7f98a6871640 1 --2- 172.21.15.83:0/3107941311 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f988c06b4d0 0x7f988c06d990 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.767+0000 7f98a6871640 1 --2- 172.21.15.83:0/3107941311 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98a01614d0 0x7f98a01f3d40 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.767+0000 7f98a6871640 1 --2- 172.21.15.83:0/3107941311 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98a00d22b0 0x7f98a00cb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.767+0000 7f98a6871640 1 --2- 172.21.15.83:0/3107941311 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98a00d1910 0x7f98a00cd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.767+0000 7f98a6871640 1 -- 172.21.15.83:0/3107941311 >> 172.21.15.83:0/3107941311 conn(0x7f98a00d3d40 msgr2=0x7f98a00d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:45.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.767+0000 7f98a6871640 1 -- 172.21.15.83:0/3107941311 shutdown_connections 2024-03-06T07:25:45.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.767+0000 7f98a6871640 1 -- 172.21.15.83:0/3107941311 wait complete. 2024-03-06T07:25:45.781 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:25:45.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.925+0000 7f5442beb640 1 Processor -- start 2024-03-06T07:25:45.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.925+0000 7f5442beb640 1 -- start start 2024-03-06T07:25:45.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.926+0000 7f5442beb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f543c0708b0 0x7f543c070d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.926+0000 7f5442beb640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f543c0ce0a0 0x7f543c0ce550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.926+0000 7f5442beb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f543c15afe0 con 0x7f543c15ac00 2024-03-06T07:25:45.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.926+0000 7f5442beb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f543c071250 con 0x7f543c0708b0 2024-03-06T07:25:45.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.926+0000 7f5442beb640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f543c0cdc10 con 0x7f543c0ce0a0 2024-03-06T07:25:45.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.926+0000 7f543bfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f543c0708b0 0x7f543c070d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.926+0000 7f543bfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f543c0708b0 0x7f543c070d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52878/0 (socket says 172.21.15.83:52878) 2024-03-06T07:25:45.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.926+0000 7f5440987640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f543c15ac00 0x7f543c06c910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52584/0 (socket says 172.21.15.83:52584) 2024-03-06T07:25:45.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.926+0000 7f543bfff640 1 -- 172.21.15.83:0/1480297551 learned_addr learned my addr 172.21.15.83:0/1480297551 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:45.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.926+0000 7f5441188640 1 --2- 172.21.15.83:0/1480297551 >> v2:172.21.15.183:3300/0 conn(0x7f543c0ce0a0 0x7f543c0ce550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.926+0000 7f543bfff640 1 -- 172.21.15.83:0/1480297551 >> v2:172.21.15.183:3300/0 conn(0x7f543c0ce0a0 msgr2=0x7f543c0ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:45.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.926+0000 7f543bfff640 1 --2- 172.21.15.83:0/1480297551 >> v2:172.21.15.183:3300/0 conn(0x7f543c0ce0a0 0x7f543c0ce550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.926+0000 7f543bfff640 1 -- 172.21.15.83:0/1480297551 >> v1:172.21.15.83:6789/0 conn(0x7f543c15ac00 legacy=0x7f543c06c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.926+0000 7f543bfff640 1 -- 172.21.15.83:0/1480297551 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f543c0cd000 con 0x7f543c0708b0 2024-03-06T07:25:45.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.926+0000 7f543b7fe640 1 -- 172.21.15.83:0/1480297551 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1234714760 0 0) 0x7f543c15afe0 con 0x7f543c15ac00 2024-03-06T07:25:45.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.927+0000 7f543bfff640 1 --2- 172.21.15.83:0/1480297551 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f543c0708b0 0x7f543c070d10 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f543c0cc0a0 tx=0x7f542c02e0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e2512c7ef5488db server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:45.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.927+0000 7f543b7fe640 1 -- 172.21.15.83:0/1480297551 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f542c02ea90 con 0x7f543c0708b0 2024-03-06T07:25:45.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.927+0000 7f543b7fe640 1 -- 172.21.15.83:0/1480297551 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f542c02ec30 con 0x7f543c0708b0 2024-03-06T07:25:45.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.927+0000 7f543b7fe640 1 -- 172.21.15.83:0/1480297551 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f542c009600 con 0x7f543c0708b0 2024-03-06T07:25:45.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.928+0000 7f5442beb640 1 -- 172.21.15.83:0/1480297551 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f543c0708b0 msgr2=0x7f543c070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.928+0000 7f5442beb640 1 --2- 172.21.15.83:0/1480297551 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f543c0708b0 0x7f543c070d10 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f543c0cc0a0 tx=0x7f542c02e0f0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.928+0000 7f5442beb640 1 -- 172.21.15.83:0/1480297551 shutdown_connections 2024-03-06T07:25:45.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.928+0000 7f5442beb640 1 --2- 172.21.15.83:0/1480297551 >> v2:172.21.15.183:3300/0 conn(0x7f543c0ce0a0 0x7f543c0ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.928+0000 7f5442beb640 1 --2- 172.21.15.83:0/1480297551 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f543c0708b0 0x7f543c070d10 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.928+0000 7f5442beb640 1 -- 172.21.15.83:0/1480297551 >> 172.21.15.83:0/1480297551 conn(0x7f543c0b1650 msgr2=0x7f543c0b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:45.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.928+0000 7f5442beb640 1 -- 172.21.15.83:0/1480297551 shutdown_connections 2024-03-06T07:25:45.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.928+0000 7f5442beb640 1 -- 172.21.15.83:0/1480297551 wait complete. 2024-03-06T07:25:45.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.929+0000 7f5442beb640 1 Processor -- start 2024-03-06T07:25:45.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.929+0000 7f5442beb640 1 -- start start 2024-03-06T07:25:45.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.930+0000 7f5442beb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f543c0708b0 0x7f543c1512f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.930+0000 7f5442beb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f543c0ce0a0 0x7f543c151a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.930+0000 7f5442beb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f543c15ac00 0x7f543c1f2890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.930+0000 7f5442beb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f543c0cd8a0 con 0x7f543c0708b0 2024-03-06T07:25:45.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.930+0000 7f5442beb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f543c0cd5c0 con 0x7f543c15ac00 2024-03-06T07:25:45.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.930+0000 7f5442beb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f543c0cf5d0 con 0x7f543c0ce0a0 2024-03-06T07:25:45.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.930+0000 7f5440987640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f543c0708b0 0x7f543c1512f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.930+0000 7f5440987640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f543c0708b0 0x7f543c1512f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52892/0 (socket says 172.21.15.83:52892) 2024-03-06T07:25:45.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.930+0000 7f5440987640 1 -- 172.21.15.83:0/1457360529 learned_addr learned my addr 172.21.15.83:0/1457360529 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:45.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.930+0000 7f543bfff640 1 --2- 172.21.15.83:0/1457360529 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f543c0ce0a0 0x7f543c151a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.930+0000 7f5441188640 1 --2- 172.21.15.83:0/1457360529 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f543c15ac00 0x7f543c1f2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.930+0000 7f5440987640 1 -- 172.21.15.83:0/1457360529 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f543c0ce0a0 msgr2=0x7f543c151a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.930+0000 7f5440987640 1 --2- 172.21.15.83:0/1457360529 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f543c0ce0a0 0x7f543c151a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.930+0000 7f5440987640 1 -- 172.21.15.83:0/1457360529 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f543c15ac00 msgr2=0x7f543c1f2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.930+0000 7f5440987640 1 --2- 172.21.15.83:0/1457360529 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f543c15ac00 0x7f543c1f2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.930+0000 7f5440987640 1 -- 172.21.15.83:0/1457360529 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f543c1f37a0 con 0x7f543c0708b0 2024-03-06T07:25:45.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.930+0000 7f543bfff640 1 --2- 172.21.15.83:0/1457360529 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f543c0ce0a0 0x7f543c151a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:45.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.930+0000 7f5441188640 1 --2- 172.21.15.83:0/1457360529 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f543c15ac00 0x7f543c1f2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:45.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.930+0000 7f5440987640 1 --2- 172.21.15.83:0/1457360529 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f543c0708b0 0x7f543c1512f0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f543400c460 tx=0x7f543400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:45.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.930+0000 7f54397fa640 1 -- 172.21.15.83:0/1457360529 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5434018070 con 0x7f543c0708b0 2024-03-06T07:25:45.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.930+0000 7f54397fa640 1 -- 172.21.15.83:0/1457360529 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5434007500 con 0x7f543c0708b0 2024-03-06T07:25:45.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.930+0000 7f5442beb640 1 -- 172.21.15.83:0/1457360529 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f543c151830 con 0x7f543c0708b0 2024-03-06T07:25:45.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.930+0000 7f54397fa640 1 -- 172.21.15.83:0/1457360529 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5434004a20 con 0x7f543c0708b0 2024-03-06T07:25:45.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.930+0000 7f5442beb640 1 -- 172.21.15.83:0/1457360529 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f543c1f34c0 con 0x7f543c0708b0 2024-03-06T07:25:45.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.931+0000 7f5442beb640 1 -- 172.21.15.83:0/1457360529 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f543c04f330 con 0x7f543c0708b0 2024-03-06T07:25:45.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.932+0000 7f54397fa640 1 -- 172.21.15.83:0/1457360529 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f543401b070 con 0x7f543c0708b0 2024-03-06T07:25:45.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.932+0000 7f54397fa640 1 --2- 172.21.15.83:0/1457360529 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f541c06b4d0 0x7f541c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.932+0000 7f54397fa640 1 -- 172.21.15.83:0/1457360529 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f5434089460 con 0x7f543c0708b0 2024-03-06T07:25:45.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.932+0000 7f543bfff640 1 --2- 172.21.15.83:0/1457360529 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f541c06b4d0 0x7f541c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.933+0000 7f543bfff640 1 --2- 172.21.15.83:0/1457360529 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f541c06b4d0 0x7f541c06d990 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f542c037000 tx=0x7f542c032000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:45.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:45.937+0000 7f54397fa640 1 -- 172.21.15.83:0/1457360529 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f54340582e0 con 0x7f543c0708b0 2024-03-06T07:25:46.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:46.108+0000 7f5442beb640 1 -- 172.21.15.83:0/1457360529 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f543c0b1ad0 con 0x7f543c0708b0 2024-03-06T07:25:46.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:46.948+0000 7f54397fa640 1 -- 172.21.15.83:0/1457360529 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v73) v1 ==== 181+0+0 (secure 0 0 0) 0x7f5434057be0 con 0x7f543c0708b0 2024-03-06T07:25:46.950 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:25:46.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:46.951+0000 7f5442beb640 1 -- 172.21.15.83:0/1457360529 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f541c06b4d0 msgr2=0x7f541c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:46.951+0000 7f5442beb640 1 --2- 172.21.15.83:0/1457360529 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f541c06b4d0 0x7f541c06d990 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f542c037000 tx=0x7f542c032000 comp rx=0 tx=0).stop 2024-03-06T07:25:46.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:46.951+0000 7f5442beb640 1 -- 172.21.15.83:0/1457360529 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f543c0708b0 msgr2=0x7f543c1512f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:46.951+0000 7f5442beb640 1 --2- 172.21.15.83:0/1457360529 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f543c0708b0 0x7f543c1512f0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f543400c460 tx=0x7f543400c920 comp rx=0 tx=0).stop 2024-03-06T07:25:46.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:46.951+0000 7f5442beb640 1 -- 172.21.15.83:0/1457360529 shutdown_connections 2024-03-06T07:25:46.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:46.951+0000 7f5442beb640 1 --2- 172.21.15.83:0/1457360529 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f541c06b4d0 0x7f541c06d990 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:46.951+0000 7f5442beb640 1 --2- 172.21.15.83:0/1457360529 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f543c15ac00 0x7f543c1f2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:46.951+0000 7f5442beb640 1 --2- 172.21.15.83:0/1457360529 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f543c0ce0a0 0x7f543c151a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:46.951+0000 7f5442beb640 1 --2- 172.21.15.83:0/1457360529 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f543c0708b0 0x7f543c1512f0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:46.951+0000 7f5442beb640 1 -- 172.21.15.83:0/1457360529 >> 172.21.15.83:0/1457360529 conn(0x7f543c0b1650 msgr2=0x7f543c0ce550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:46.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:46.951+0000 7f5442beb640 1 -- 172.21.15.83:0/1457360529 shutdown_connections 2024-03-06T07:25:46.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:46.951+0000 7f5442beb640 1 -- 172.21.15.83:0/1457360529 wait complete. 2024-03-06T07:25:46.968 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:25:47.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.076+0000 7f694ad1e640 1 Processor -- start 2024-03-06T07:25:47.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.076+0000 7f694ad1e640 1 -- start start 2024-03-06T07:25:47.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.077+0000 7f694ad1e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6944161440 0x7f69440b4be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.077+0000 7f694ad1e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69440cf520 con 0x7f69440d0d20 2024-03-06T07:25:47.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.077+0000 7f694ad1e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69440cf800 con 0x7f6944161820 2024-03-06T07:25:47.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.077+0000 7f694ad1e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69440cf2d0 con 0x7f6944161440 2024-03-06T07:25:47.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.077+0000 7f69492bb640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f69440d0d20 0x7f69440b6f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39438/0 (socket says 172.21.15.83:39438) 2024-03-06T07:25:47.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.077+0000 7f69492bb640 1 -- 172.21.15.83:0/3507363227 learned_addr learned my addr 172.21.15.83:0/3507363227 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:25:47.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.077+0000 7f6948aba640 1 --2- 172.21.15.83:0/3507363227 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6944161440 0x7f69440b4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.077+0000 7f69437fe640 1 -- 172.21.15.83:0/3507363227 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3895320504 0 0) 0x7f69440cf520 con 0x7f69440d0d20 2024-03-06T07:25:47.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.077+0000 7f69437fe640 1 -- 172.21.15.83:0/3507363227 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6934003520 con 0x7f69440d0d20 2024-03-06T07:25:47.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.077+0000 7f69437fe640 1 -- 172.21.15.83:0/3507363227 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3548547980 0 0) 0x7f69440cf800 con 0x7f6944161820 2024-03-06T07:25:47.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.077+0000 7f69437fe640 1 -- 172.21.15.83:0/3507363227 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69440cf520 con 0x7f6944161820 2024-03-06T07:25:47.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.077+0000 7f69437fe640 1 -- 172.21.15.83:0/3507363227 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3594133520 0 0) 0x7f6934003520 con 0x7f69440d0d20 2024-03-06T07:25:47.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.077+0000 7f6948aba640 1 -- 172.21.15.83:0/3507363227 >> v1:172.21.15.183:6789/0 conn(0x7f6944161820 legacy=0x7f69440d0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.077+0000 7f6948aba640 1 -- 172.21.15.83:0/3507363227 >> v1:172.21.15.83:6789/0 conn(0x7f69440d0d20 legacy=0x7f69440b6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.077+0000 7f6948aba640 1 -- 172.21.15.83:0/3507363227 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69440cfae0 con 0x7f6944161440 2024-03-06T07:25:47.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.078+0000 7f6948aba640 1 --2- 172.21.15.83:0/3507363227 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6944161440 0x7f69440b4be0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f69380097d0 tx=0x7f693802b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2e33bc69d493346 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:47.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.078+0000 7f69437fe640 1 -- 172.21.15.83:0/3507363227 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6938037070 con 0x7f6944161440 2024-03-06T07:25:47.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.078+0000 7f69437fe640 1 -- 172.21.15.83:0/3507363227 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6938030510 con 0x7f6944161440 2024-03-06T07:25:47.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.078+0000 7f69437fe640 1 -- 172.21.15.83:0/3507363227 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f693c0019f0 con 0x7f6944161440 2024-03-06T07:25:47.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.079+0000 7f694ad1e640 1 -- 172.21.15.83:0/3507363227 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6944161440 msgr2=0x7f69440b4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.079+0000 7f694ad1e640 1 --2- 172.21.15.83:0/3507363227 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6944161440 0x7f69440b4be0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f69380097d0 tx=0x7f693802b770 comp rx=0 tx=0).stop 2024-03-06T07:25:47.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.079+0000 7f694ad1e640 1 -- 172.21.15.83:0/3507363227 shutdown_connections 2024-03-06T07:25:47.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.079+0000 7f694ad1e640 1 --2- 172.21.15.83:0/3507363227 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6944161440 0x7f69440b4be0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.079+0000 7f694ad1e640 1 -- 172.21.15.83:0/3507363227 >> 172.21.15.83:0/3507363227 conn(0x7f69440d3cb0 msgr2=0x7f69440d40c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:47.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.079+0000 7f694ad1e640 1 -- 172.21.15.83:0/3507363227 shutdown_connections 2024-03-06T07:25:47.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.079+0000 7f694ad1e640 1 -- 172.21.15.83:0/3507363227 wait complete. 2024-03-06T07:25:47.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.080+0000 7f694ad1e640 1 Processor -- start 2024-03-06T07:25:47.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.080+0000 7f694ad1e640 1 -- start start 2024-03-06T07:25:47.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.081+0000 7f694ad1e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f69440d0d20 0x7f694415efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.081+0000 7f694ad1e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6944161440 0x7f69441f3b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.081+0000 7f694ad1e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6944161820 0x7f69440cd190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.081+0000 7f694ad1e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69440d1300 con 0x7f69440d0d20 2024-03-06T07:25:47.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.081+0000 7f694ad1e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69440b6b00 con 0x7f6944161820 2024-03-06T07:25:47.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.081+0000 7f694ad1e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69440d1480 con 0x7f6944161440 2024-03-06T07:25:47.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.081+0000 7f6948aba640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f69440d0d20 0x7f694415efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.081+0000 7f6948aba640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f69440d0d20 0x7f694415efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37488/0 (socket says 172.21.15.83:37488) 2024-03-06T07:25:47.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.081+0000 7f6948aba640 1 -- 172.21.15.83:0/364911543 learned_addr learned my addr 172.21.15.83:0/364911543 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:47.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.081+0000 7f6943fff640 1 --2- 172.21.15.83:0/364911543 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6944161440 0x7f69441f3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.081+0000 7f69492bb640 1 --2- 172.21.15.83:0/364911543 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6944161820 0x7f69440cd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.081+0000 7f6948aba640 1 -- 172.21.15.83:0/364911543 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6944161440 msgr2=0x7f69441f3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.081+0000 7f6948aba640 1 --2- 172.21.15.83:0/364911543 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6944161440 0x7f69441f3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.081+0000 7f6948aba640 1 -- 172.21.15.83:0/364911543 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6944161820 msgr2=0x7f69440cd190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.081+0000 7f6948aba640 1 --2- 172.21.15.83:0/364911543 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6944161820 0x7f69440cd190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.081+0000 7f6948aba640 1 -- 172.21.15.83:0/364911543 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69440cad80 con 0x7f69440d0d20 2024-03-06T07:25:47.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.081+0000 7f6943fff640 1 --2- 172.21.15.83:0/364911543 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6944161440 0x7f69441f3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:47.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.082+0000 7f6948aba640 1 --2- 172.21.15.83:0/364911543 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f69440d0d20 0x7f694415efd0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f6938000c00 tx=0x7f69380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:47.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.082+0000 7f69417fa640 1 -- 172.21.15.83:0/364911543 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6938042070 con 0x7f69440d0d20 2024-03-06T07:25:47.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.082+0000 7f69417fa640 1 -- 172.21.15.83:0/364911543 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f693803e870 con 0x7f69440d0d20 2024-03-06T07:25:47.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.082+0000 7f694ad1e640 1 -- 172.21.15.83:0/364911543 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69441f4550 con 0x7f69440d0d20 2024-03-06T07:25:47.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.082+0000 7f69417fa640 1 -- 172.21.15.83:0/364911543 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6938037040 con 0x7f69440d0d20 2024-03-06T07:25:47.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.082+0000 7f694ad1e640 1 -- 172.21.15.83:0/364911543 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69440b0a80 con 0x7f69440d0d20 2024-03-06T07:25:47.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.083+0000 7f694ad1e640 1 -- 172.21.15.83:0/364911543 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f694404f330 con 0x7f69440d0d20 2024-03-06T07:25:47.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.083+0000 7f69417fa640 1 -- 172.21.15.83:0/364911543 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6938045070 con 0x7f69440d0d20 2024-03-06T07:25:47.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.084+0000 7f69417fa640 1 --2- 172.21.15.83:0/364911543 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f692806b4d0 0x7f692806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.084+0000 7f69417fa640 1 -- 172.21.15.83:0/364911543 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5415+0+0 (secure 0 0 0) 0x7f69380abe00 con 0x7f69440d0d20 2024-03-06T07:25:47.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.084+0000 7f6943fff640 1 --2- 172.21.15.83:0/364911543 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f692806b4d0 0x7f692806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.084+0000 7f6943fff640 1 --2- 172.21.15.83:0/364911543 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f692806b4d0 0x7f692806d990 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f69300011e0 tx=0x7f6930009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:47.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.088+0000 7f69417fa640 1 -- 172.21.15.83:0/364911543 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f69440cad80 con 0x7f69440d0d20 2024-03-06T07:25:47.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.259+0000 7f694ad1e640 1 -- 172.21.15.83:0/364911543 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f69440d1fa0 con 0x7f69440d0d20 2024-03-06T07:25:47.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.953+0000 7f69417fa640 1 -- 172.21.15.83:0/364911543 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v74) v1 ==== 169+0+0 (secure 0 0 0) 0x7f693807af50 con 0x7f69440d0d20 2024-03-06T07:25:47.955 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'cephfs_data' removed 2024-03-06T07:25:47.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.956+0000 7f694ad1e640 1 -- 172.21.15.83:0/364911543 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f692806b4d0 msgr2=0x7f692806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.956+0000 7f694ad1e640 1 --2- 172.21.15.83:0/364911543 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f692806b4d0 0x7f692806d990 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f69300011e0 tx=0x7f6930009040 comp rx=0 tx=0).stop 2024-03-06T07:25:47.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.956+0000 7f694ad1e640 1 -- 172.21.15.83:0/364911543 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f69440d0d20 msgr2=0x7f694415efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.956+0000 7f694ad1e640 1 --2- 172.21.15.83:0/364911543 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f69440d0d20 0x7f694415efd0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f6938000c00 tx=0x7f69380046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.956+0000 7f694ad1e640 1 -- 172.21.15.83:0/364911543 shutdown_connections 2024-03-06T07:25:47.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.956+0000 7f694ad1e640 1 --2- 172.21.15.83:0/364911543 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f692806b4d0 0x7f692806d990 secure :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f69300011e0 tx=0x7f6930009040 comp rx=0 tx=0).stop 2024-03-06T07:25:47.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.956+0000 7f694ad1e640 1 --2- 172.21.15.83:0/364911543 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6944161820 0x7f69440cd190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.956+0000 7f694ad1e640 1 --2- 172.21.15.83:0/364911543 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6944161440 0x7f69441f3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.956+0000 7f694ad1e640 1 --2- 172.21.15.83:0/364911543 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f69440d0d20 0x7f694415efd0 secure :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f6938000c00 tx=0x7f69380046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.956+0000 7f694ad1e640 1 -- 172.21.15.83:0/364911543 >> 172.21.15.83:0/364911543 conn(0x7f69440d3cb0 msgr2=0x7f69440b74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:47.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.957+0000 7f694ad1e640 1 -- 172.21.15.83:0/364911543 shutdown_connections 2024-03-06T07:25:47.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:47.957+0000 7f694ad1e640 1 -- 172.21.15.83:0/364911543 wait complete. 2024-03-06T07:25:47.973 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_fragment_size_max 2024-03-06T07:25:48.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.078+0000 7f1a4f888640 1 Processor -- start 2024-03-06T07:25:48.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.078+0000 7f1a4f888640 1 -- start start 2024-03-06T07:25:48.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.079+0000 7f1a4f888640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1a480cfa90 0x7f1a480cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.079+0000 7f1a4f888640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1a480b33a0 0x7f1a480b37c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.079+0000 7f1a4f888640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a48158ac0 con 0x7f1a481586e0 2024-03-06T07:25:48.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.079+0000 7f1a4f888640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a4815b2a0 con 0x7f1a480cfa90 2024-03-06T07:25:48.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.079+0000 7f1a4f888640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4815b540 con 0x7f1a480b33a0 2024-03-06T07:25:48.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.079+0000 7f1a4ce23640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1a480cfa90 0x7f1a480cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.079+0000 7f1a4ce23640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1a480cfa90 0x7f1a480cff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37500/0 (socket says 172.21.15.83:37500) 2024-03-06T07:25:48.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.079+0000 7f1a4ce23640 1 -- 172.21.15.83:0/2963351558 learned_addr learned my addr 172.21.15.83:0/2963351558 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:48.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.079+0000 7f1a4de25640 1 --2- 172.21.15.83:0/2963351558 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1a480b33a0 0x7f1a480b37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.079+0000 7f1a4ce23640 1 -- 172.21.15.83:0/2963351558 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1a480b33a0 msgr2=0x7f1a480b37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:48.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.079+0000 7f1a4ce23640 1 --2- 172.21.15.83:0/2963351558 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1a480b33a0 0x7f1a480b37c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.079+0000 7f1a4ce23640 1 -- 172.21.15.83:0/2963351558 >> v1:172.21.15.183:6789/0 conn(0x7f1a481586e0 legacy=0x7f1a480aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:48.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.079+0000 7f1a4ce23640 1 -- 172.21.15.83:0/2963351558 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a4815b820 con 0x7f1a480cfa90 2024-03-06T07:25:48.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.080+0000 7f1a4ce23640 1 --2- 172.21.15.83:0/2963351558 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a480cfa90 0x7f1a480cff40 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f1a38009870 tx=0x7f1a3802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e80f50656a94ee03 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:48.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.080+0000 7f1a3ffff640 1 -- 172.21.15.83:0/2963351558 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a38034070 con 0x7f1a480cfa90 2024-03-06T07:25:48.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.080+0000 7f1a3ffff640 1 -- 172.21.15.83:0/2963351558 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a38030840 con 0x7f1a480cfa90 2024-03-06T07:25:48.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.080+0000 7f1a3ffff640 1 -- 172.21.15.83:0/2963351558 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a38030c10 con 0x7f1a480cfa90 2024-03-06T07:25:48.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.081+0000 7f1a4f888640 1 -- 172.21.15.83:0/2963351558 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a480cfa90 msgr2=0x7f1a480cff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.081+0000 7f1a4f888640 1 --2- 172.21.15.83:0/2963351558 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a480cfa90 0x7f1a480cff40 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f1a38009870 tx=0x7f1a3802b980 comp rx=0 tx=0).stop 2024-03-06T07:25:48.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.081+0000 7f1a4f888640 1 -- 172.21.15.83:0/2963351558 shutdown_connections 2024-03-06T07:25:48.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.081+0000 7f1a4f888640 1 --2- 172.21.15.83:0/2963351558 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1a480b33a0 0x7f1a480b37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.081+0000 7f1a4f888640 1 --2- 172.21.15.83:0/2963351558 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a480cfa90 0x7f1a480cff40 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.081+0000 7f1a4f888640 1 -- 172.21.15.83:0/2963351558 >> 172.21.15.83:0/2963351558 conn(0x7f1a4806b190 msgr2=0x7f1a4806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:48.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.081+0000 7f1a4f888640 1 -- 172.21.15.83:0/2963351558 shutdown_connections 2024-03-06T07:25:48.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.081+0000 7f1a4f888640 1 -- 172.21.15.83:0/2963351558 wait complete. 2024-03-06T07:25:48.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.082+0000 7f1a4f888640 1 Processor -- start 2024-03-06T07:25:48.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.082+0000 7f1a4f888640 1 -- start start 2024-03-06T07:25:48.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.082+0000 7f1a4f888640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a480b33a0 0x7f1a48148750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.083+0000 7f1a4f888640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a480cfa90 0x7f1a48146c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.083+0000 7f1a4f888640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1a481586e0 0x7f1a48147190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.083+0000 7f1a4f888640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a480af150 con 0x7f1a480b33a0 2024-03-06T07:25:48.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.083+0000 7f1a4f888640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4815bde0 con 0x7f1a480cfa90 2024-03-06T07:25:48.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.083+0000 7f1a4f888640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a48071350 con 0x7f1a481586e0 2024-03-06T07:25:48.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.083+0000 7f1a4d624640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a480b33a0 0x7f1a48148750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.083+0000 7f1a4d624640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a480b33a0 0x7f1a48148750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37516/0 (socket says 172.21.15.83:37516) 2024-03-06T07:25:48.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.083+0000 7f1a4d624640 1 -- 172.21.15.83:0/2536756256 learned_addr learned my addr 172.21.15.83:0/2536756256 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:48.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.083+0000 7f1a4ce23640 1 --2- 172.21.15.83:0/2536756256 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a480cfa90 0x7f1a48146c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.083+0000 7f1a4de25640 1 --2- 172.21.15.83:0/2536756256 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1a481586e0 0x7f1a48147190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.083+0000 7f1a4d624640 1 -- 172.21.15.83:0/2536756256 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1a481586e0 msgr2=0x7f1a48147190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.083+0000 7f1a4d624640 1 --2- 172.21.15.83:0/2536756256 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1a481586e0 0x7f1a48147190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.083+0000 7f1a4d624640 1 -- 172.21.15.83:0/2536756256 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a480cfa90 msgr2=0x7f1a48146c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.083+0000 7f1a4d624640 1 --2- 172.21.15.83:0/2536756256 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a480cfa90 0x7f1a48146c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.083+0000 7f1a4d624640 1 -- 172.21.15.83:0/2536756256 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a480cecc0 con 0x7f1a480b33a0 2024-03-06T07:25:48.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.083+0000 7f1a4ce23640 1 --2- 172.21.15.83:0/2536756256 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a480cfa90 0x7f1a48146c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:48.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.083+0000 7f1a4d624640 1 --2- 172.21.15.83:0/2536756256 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a480b33a0 0x7f1a48148750 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f1a4400c3c0 tx=0x7f1a4400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:48.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.083+0000 7f1a3dffb640 1 -- 172.21.15.83:0/2536756256 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a440052d0 con 0x7f1a480b33a0 2024-03-06T07:25:48.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.083+0000 7f1a3dffb640 1 -- 172.21.15.83:0/2536756256 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a44005490 con 0x7f1a480b33a0 2024-03-06T07:25:48.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.083+0000 7f1a4f888640 1 -- 172.21.15.83:0/2536756256 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a48147810 con 0x7f1a480b33a0 2024-03-06T07:25:48.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.083+0000 7f1a3dffb640 1 -- 172.21.15.83:0/2536756256 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a44007260 con 0x7f1a480b33a0 2024-03-06T07:25:48.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.083+0000 7f1a4f888640 1 -- 172.21.15.83:0/2536756256 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a48147dd0 con 0x7f1a480b33a0 2024-03-06T07:25:48.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.085+0000 7f1a4f888640 1 -- 172.21.15.83:0/2536756256 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a480732b0 con 0x7f1a480b33a0 2024-03-06T07:25:48.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.085+0000 7f1a3dffb640 1 -- 172.21.15.83:0/2536756256 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1a44007400 con 0x7f1a480b33a0 2024-03-06T07:25:48.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.085+0000 7f1a3dffb640 1 --2- 172.21.15.83:0/2536756256 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1a2c06b4d0 0x7f1a2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.085+0000 7f1a3dffb640 1 -- 172.21.15.83:0/2536756256 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f1a440893e0 con 0x7f1a480b33a0 2024-03-06T07:25:48.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.085+0000 7f1a4ce23640 1 --2- 172.21.15.83:0/2536756256 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1a2c06b4d0 0x7f1a2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.085+0000 7f1a4ce23640 1 --2- 172.21.15.83:0/2536756256 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1a2c06b4d0 0x7f1a2c06d990 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f1a38036000 tx=0x7f1a3803b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:48.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.089+0000 7f1a3dffb640 1 -- 172.21.15.83:0/2536756256 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1a480cecc0 con 0x7f1a480b33a0 2024-03-06T07:25:48.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.270+0000 7f1a4f888640 1 -- 172.21.15.83:0/2536756256 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"} v 0) v1 -- 0x7f1a4815a270 con 0x7f1a480b33a0 2024-03-06T07:25:48.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.273+0000 7f1a3dffb640 1 -- 172.21.15.83:0/2536756256 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"}]=0 v24) v1 ==== 108+0+0 (secure 0 0 0) 0x7f1a44058560 con 0x7f1a480b33a0 2024-03-06T07:25:48.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.275+0000 7f1a4f888640 1 -- 172.21.15.83:0/2536756256 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1a2c06b4d0 msgr2=0x7f1a2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.275+0000 7f1a4f888640 1 --2- 172.21.15.83:0/2536756256 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1a2c06b4d0 0x7f1a2c06d990 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f1a38036000 tx=0x7f1a3803b040 comp rx=0 tx=0).stop 2024-03-06T07:25:48.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.275+0000 7f1a4f888640 1 -- 172.21.15.83:0/2536756256 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a480b33a0 msgr2=0x7f1a48148750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.275+0000 7f1a4f888640 1 --2- 172.21.15.83:0/2536756256 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a480b33a0 0x7f1a48148750 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f1a4400c3c0 tx=0x7f1a4400c880 comp rx=0 tx=0).stop 2024-03-06T07:25:48.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.275+0000 7f1a4f888640 1 -- 172.21.15.83:0/2536756256 shutdown_connections 2024-03-06T07:25:48.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.275+0000 7f1a4f888640 1 --2- 172.21.15.83:0/2536756256 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1a2c06b4d0 0x7f1a2c06d990 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.275+0000 7f1a4f888640 1 --2- 172.21.15.83:0/2536756256 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1a481586e0 0x7f1a48147190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.275+0000 7f1a4f888640 1 --2- 172.21.15.83:0/2536756256 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a480cfa90 0x7f1a48146c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.275+0000 7f1a4f888640 1 --2- 172.21.15.83:0/2536756256 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a480b33a0 0x7f1a48148750 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.275+0000 7f1a4f888640 1 -- 172.21.15.83:0/2536756256 >> 172.21.15.83:0/2536756256 conn(0x7f1a4806b190 msgr2=0x7f1a48159330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:48.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.275+0000 7f1a4f888640 1 -- 172.21.15.83:0/2536756256 shutdown_connections 2024-03-06T07:25:48.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.275+0000 7f1a4f888640 1 -- 172.21.15.83:0/2536756256 wait complete. 2024-03-06T07:25:48.288 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:25:48.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.388+0000 7f2423f01640 1 Processor -- start 2024-03-06T07:25:48.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.388+0000 7f2423f01640 1 -- start start 2024-03-06T07:25:48.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.389+0000 7f2423f01640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f241c15bb40 0x7f241c0b4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.389+0000 7f2423f01640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f241c15ff90 con 0x7f241c0cfa90 2024-03-06T07:25:48.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.389+0000 7f2423f01640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f241c160270 con 0x7f241c15f7d0 2024-03-06T07:25:48.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.389+0000 7f2423f01640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f241c06c910 con 0x7f241c15bb40 2024-03-06T07:25:48.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.389+0000 7f242249e640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f241c0cfa90 0x7f241c0cdcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39440/0 (socket says 172.21.15.83:39440) 2024-03-06T07:25:48.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.389+0000 7f242249e640 1 -- 172.21.15.83:0/3636988081 learned_addr learned my addr 172.21.15.83:0/3636988081 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:25:48.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.389+0000 7f2420c9b640 1 -- 172.21.15.83:0/3636988081 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1396206119 0 0) 0x7f241c15ff90 con 0x7f241c0cfa90 2024-03-06T07:25:48.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.389+0000 7f2420c9b640 1 -- 172.21.15.83:0/3636988081 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2408003520 con 0x7f241c0cfa90 2024-03-06T07:25:48.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.389+0000 7f2421c9d640 1 --2- 172.21.15.83:0/3636988081 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f241c15bb40 0x7f241c0b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.389+0000 7f2420c9b640 1 -- 172.21.15.83:0/3636988081 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4285321304 0 0) 0x7f2408003520 con 0x7f241c0cfa90 2024-03-06T07:25:48.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.389+0000 7f2420c9b640 1 -- 172.21.15.83:0/3636988081 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f241c15bb40 msgr2=0x7f241c0b4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:48.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.389+0000 7f2420c9b640 1 --2- 172.21.15.83:0/3636988081 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f241c15bb40 0x7f241c0b4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.389+0000 7f2420c9b640 1 -- 172.21.15.83:0/3636988081 >> v1:172.21.15.183:6789/0 conn(0x7f241c15f7d0 legacy=0x7f241c06d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:48.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.389+0000 7f2420c9b640 1 -- 172.21.15.83:0/3636988081 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f241c160550 con 0x7f241c0cfa90 2024-03-06T07:25:48.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.389+0000 7f2420c9b640 1 -- 172.21.15.83:0/3636988081 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f2418001670 con 0x7f241c0cfa90 2024-03-06T07:25:48.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.389+0000 7f2420c9b640 1 -- 172.21.15.83:0/3636988081 >> v1:172.21.15.83:6789/0 conn(0x7f241c0cfa90 legacy=0x7f241c0cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.390+0000 7f2420c9b640 1 --2- 172.21.15.83:0/3636988081 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f241c15bb40 0x7f24080039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.390+0000 7f2420c9b640 1 --2- 172.21.15.83:0/3636988081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2408004010 0x7f2408004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.390+0000 7f2420c9b640 1 --2- 172.21.15.83:0/3636988081 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2408004d40 0x7f24080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.390+0000 7f2420c9b640 1 -- 172.21.15.83:0/3636988081 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241c06c910 con 0x7f2408004010 2024-03-06T07:25:48.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.390+0000 7f2420c9b640 1 -- 172.21.15.83:0/3636988081 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240800bb80 con 0x7f2408004d40 2024-03-06T07:25:48.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.390+0000 7f2420c9b640 1 -- 172.21.15.83:0/3636988081 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f240800bd80 con 0x7f241c15bb40 2024-03-06T07:25:48.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.390+0000 7f242149c640 1 --2- 172.21.15.83:0/3636988081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2408004010 0x7f2408004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.390+0000 7f2421c9d640 1 --2- 172.21.15.83:0/3636988081 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f241c15bb40 0x7f24080039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.390+0000 7f242249e640 1 --2- 172.21.15.83:0/3636988081 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2408004d40 0x7f24080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.390+0000 7f242149c640 1 -- 172.21.15.83:0/3636988081 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f241c15bb40 msgr2=0x7f24080039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:48.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.390+0000 7f242149c640 1 --2- 172.21.15.83:0/3636988081 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f241c15bb40 0x7f24080039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.390+0000 7f242149c640 1 -- 172.21.15.83:0/3636988081 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2408004d40 msgr2=0x7f24080051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:48.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.390+0000 7f242149c640 1 --2- 172.21.15.83:0/3636988081 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2408004d40 0x7f24080051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.390+0000 7f242149c640 1 -- 172.21.15.83:0/3636988081 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2408003520 con 0x7f2408004010 2024-03-06T07:25:48.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.391+0000 7f242149c640 1 --2- 172.21.15.83:0/3636988081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2408004010 0x7f2408004800 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f240c001750 tx=0x7f2408005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf03c7654623f8ad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:48.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.391+0000 7f2420c9b640 1 -- 172.21.15.83:0/3636988081 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240c001370 con 0x7f2408004010 2024-03-06T07:25:48.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.391+0000 7f2420c9b640 1 -- 172.21.15.83:0/3636988081 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f240c0052d0 con 0x7f2408004010 2024-03-06T07:25:48.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.391+0000 7f2420c9b640 1 -- 172.21.15.83:0/3636988081 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240c003830 con 0x7f2408004010 2024-03-06T07:25:48.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.391+0000 7f2423f01640 1 -- 172.21.15.83:0/3636988081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2408004010 msgr2=0x7f2408004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.391+0000 7f2423f01640 1 --2- 172.21.15.83:0/3636988081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2408004010 0x7f2408004800 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f240c001750 tx=0x7f2408005830 comp rx=0 tx=0).stop 2024-03-06T07:25:48.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.391+0000 7f242249e640 1 -- 172.21.15.83:0/3636988081 reap_dead start 2024-03-06T07:25:48.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.391+0000 7f2423f01640 1 -- 172.21.15.83:0/3636988081 shutdown_connections 2024-03-06T07:25:48.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.391+0000 7f2423f01640 1 -- 172.21.15.83:0/3636988081 >> 172.21.15.83:0/3636988081 conn(0x7f241c06b190 msgr2=0x7f241c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:48.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.391+0000 7f2423f01640 1 -- 172.21.15.83:0/3636988081 shutdown_connections 2024-03-06T07:25:48.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.391+0000 7f2423f01640 1 -- 172.21.15.83:0/3636988081 wait complete. 2024-03-06T07:25:48.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.393+0000 7f2423f01640 1 Processor -- start 2024-03-06T07:25:48.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.393+0000 7f2423f01640 1 -- start start 2024-03-06T07:25:48.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.393+0000 7f2423f01640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2408004010 0x7f241c0adda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.393+0000 7f2423f01640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f241c143270 0x7f241c1436f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.393+0000 7f2423f01640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f241c141250 0x7f241c1416d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.393+0000 7f2423f01640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241c06d790 con 0x7f241c143270 2024-03-06T07:25:48.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.393+0000 7f2423f01640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241c15fc30 con 0x7f241c141250 2024-03-06T07:25:48.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.393+0000 7f2423f01640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f241c15bf20 con 0x7f2408004010 2024-03-06T07:25:48.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.393+0000 7f242149c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f241c143270 0x7f241c1436f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.393+0000 7f242149c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f241c143270 0x7f241c1436f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37534/0 (socket says 172.21.15.83:37534) 2024-03-06T07:25:48.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.393+0000 7f242149c640 1 -- 172.21.15.83:0/3642640660 learned_addr learned my addr 172.21.15.83:0/3642640660 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:48.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.393+0000 7f2421c9d640 1 --2- 172.21.15.83:0/3642640660 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2408004010 0x7f241c0adda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.393+0000 7f242249e640 1 --2- 172.21.15.83:0/3642640660 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f241c141250 0x7f241c1416d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.394+0000 7f242149c640 1 -- 172.21.15.83:0/3642640660 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2408004010 msgr2=0x7f241c0adda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.394+0000 7f242149c640 1 --2- 172.21.15.83:0/3642640660 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2408004010 0x7f241c0adda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.394+0000 7f242149c640 1 -- 172.21.15.83:0/3642640660 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f241c141250 msgr2=0x7f241c1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.394+0000 7f242149c640 1 --2- 172.21.15.83:0/3642640660 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f241c141250 0x7f241c1416d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.394+0000 7f242149c640 1 -- 172.21.15.83:0/3642640660 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f241c0ae2e0 con 0x7f241c143270 2024-03-06T07:25:48.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.394+0000 7f242149c640 1 --2- 172.21.15.83:0/3642640660 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f241c143270 0x7f241c1436f0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f240c008fd0 tx=0x7f240c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:48.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.394+0000 7f24127fc640 1 -- 172.21.15.83:0/3642640660 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240c041070 con 0x7f241c143270 2024-03-06T07:25:48.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.394+0000 7f24127fc640 1 -- 172.21.15.83:0/3642640660 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f240c005480 con 0x7f241c143270 2024-03-06T07:25:48.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.394+0000 7f2423f01640 1 -- 172.21.15.83:0/3642640660 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f241c141fc0 con 0x7f241c143270 2024-03-06T07:25:48.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.394+0000 7f24127fc640 1 -- 172.21.15.83:0/3642640660 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240c001370 con 0x7f241c143270 2024-03-06T07:25:48.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.394+0000 7f2423f01640 1 -- 172.21.15.83:0/3642640660 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f241c142580 con 0x7f241c143270 2024-03-06T07:25:48.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.395+0000 7f2423f01640 1 -- 172.21.15.83:0/3642640660 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f241c143c30 con 0x7f241c143270 2024-03-06T07:25:48.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.395+0000 7f24127fc640 1 -- 172.21.15.83:0/3642640660 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f240c044070 con 0x7f241c143270 2024-03-06T07:25:48.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.396+0000 7f24127fc640 1 --2- 172.21.15.83:0/3642640660 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f240006b500 0x7f240006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.396+0000 7f2421c9d640 1 --2- 172.21.15.83:0/3642640660 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f240006b500 0x7f240006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.396+0000 7f24127fc640 1 -- 172.21.15.83:0/3642640660 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f240c0abc40 con 0x7f241c143270 2024-03-06T07:25:48.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.396+0000 7f2421c9d640 1 --2- 172.21.15.83:0/3642640660 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f240006b500 0x7f240006d9c0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f2414003b30 tx=0x7f2414009760 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:48.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.400+0000 7f24127fc640 1 -- 172.21.15.83:0/3642640660 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f241c143c30 con 0x7f241c143270 2024-03-06T07:25:48.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.572+0000 7f2423f01640 1 -- 172.21.15.83:0/3642640660 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f241c0732b0 con 0x7f241c143270 2024-03-06T07:25:48.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.577+0000 7f24127fc640 1 -- 172.21.15.83:0/3642640660 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v25) v1 ==== 104+0+0 (secure 0 0 0) 0x7f240c07a890 con 0x7f241c143270 2024-03-06T07:25:48.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.579+0000 7f2423f01640 1 -- 172.21.15.83:0/3642640660 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f240006b500 msgr2=0x7f240006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.579+0000 7f2423f01640 1 --2- 172.21.15.83:0/3642640660 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f240006b500 0x7f240006d9c0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f2414003b30 tx=0x7f2414009760 comp rx=0 tx=0).stop 2024-03-06T07:25:48.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.579+0000 7f2423f01640 1 -- 172.21.15.83:0/3642640660 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f241c143270 msgr2=0x7f241c1436f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.579+0000 7f2423f01640 1 --2- 172.21.15.83:0/3642640660 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f241c143270 0x7f241c1436f0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f240c008fd0 tx=0x7f240c001120 comp rx=0 tx=0).stop 2024-03-06T07:25:48.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.579+0000 7f2423f01640 1 -- 172.21.15.83:0/3642640660 shutdown_connections 2024-03-06T07:25:48.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.579+0000 7f2423f01640 1 --2- 172.21.15.83:0/3642640660 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f240006b500 0x7f240006d9c0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.579+0000 7f2423f01640 1 --2- 172.21.15.83:0/3642640660 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f241c141250 0x7f241c1416d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.579+0000 7f2423f01640 1 --2- 172.21.15.83:0/3642640660 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f241c143270 0x7f241c1436f0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.579+0000 7f2423f01640 1 --2- 172.21.15.83:0/3642640660 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2408004010 0x7f241c0adda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.579+0000 7f2423f01640 1 -- 172.21.15.83:0/3642640660 >> 172.21.15.83:0/3642640660 conn(0x7f241c06b190 msgr2=0x7f241c15ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:48.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.579+0000 7f2423f01640 1 -- 172.21.15.83:0/3642640660 shutdown_connections 2024-03-06T07:25:48.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.579+0000 7f2423f01640 1 -- 172.21.15.83:0/3642640660 wait complete. 2024-03-06T07:25:48.592 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays' 2024-03-06T07:25:48.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.690+0000 7f33f84b5640 1 Processor -- start 2024-03-06T07:25:48.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.690+0000 7f33f84b5640 1 -- start start 2024-03-06T07:25:48.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.691+0000 7f33f84b5640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f33f015bb20 0x7f33f00d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.691+0000 7f33f84b5640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33f0161980 con 0x7f33f00cfff0 2024-03-06T07:25:48.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.691+0000 7f33f84b5640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33f0161c60 con 0x7f33f00cd8c0 2024-03-06T07:25:48.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.691+0000 7f33f84b5640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33f0161f40 con 0x7f33f015bb20 2024-03-06T07:25:48.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.691+0000 7f33f5a50640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f33f00cfff0 0x7f33f00ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39450/0 (socket says 172.21.15.83:39450) 2024-03-06T07:25:48.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.691+0000 7f33f5a50640 1 -- 172.21.15.83:0/2787157420 learned_addr learned my addr 172.21.15.83:0/2787157420 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:25:48.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.691+0000 7f33f524f640 1 -- 172.21.15.83:0/2787157420 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 938620056 0 0) 0x7f33f0161980 con 0x7f33f00cfff0 2024-03-06T07:25:48.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.691+0000 7f33f524f640 1 -- 172.21.15.83:0/2787157420 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33dc003520 con 0x7f33f00cfff0 2024-03-06T07:25:48.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.691+0000 7f33f6251640 1 --2- 172.21.15.83:0/2787157420 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f33f015bb20 0x7f33f00d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.692+0000 7f33f524f640 1 -- 172.21.15.83:0/2787157420 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4077705673 0 0) 0x7f33dc003520 con 0x7f33f00cfff0 2024-03-06T07:25:48.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.692+0000 7f33f524f640 1 -- 172.21.15.83:0/2787157420 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f33f015bb20 msgr2=0x7f33f00d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:48.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.692+0000 7f33f524f640 1 --2- 172.21.15.83:0/2787157420 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f33f015bb20 0x7f33f00d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.692+0000 7f33f524f640 1 -- 172.21.15.83:0/2787157420 >> v1:172.21.15.183:6789/0 conn(0x7f33f00cd8c0 legacy=0x7f33f00cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.692+0000 7f33f524f640 1 -- 172.21.15.83:0/2787157420 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f0162220 con 0x7f33f00cfff0 2024-03-06T07:25:48.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.692+0000 7f33f524f640 1 -- 172.21.15.83:0/2787157420 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f33ec001670 con 0x7f33f00cfff0 2024-03-06T07:25:48.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.692+0000 7f33f524f640 1 -- 172.21.15.83:0/2787157420 >> v1:172.21.15.83:6789/0 conn(0x7f33f00cfff0 legacy=0x7f33f00ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.692+0000 7f33f524f640 1 --2- 172.21.15.83:0/2787157420 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f33f015bb20 0x7f33dc0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.692+0000 7f33f524f640 1 --2- 172.21.15.83:0/2787157420 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33dc004010 0x7f33dc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.692+0000 7f33f524f640 1 --2- 172.21.15.83:0/2787157420 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f33dc004d40 0x7f33dc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.692+0000 7f33f524f640 1 -- 172.21.15.83:0/2787157420 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f0161f40 con 0x7f33dc004010 2024-03-06T07:25:48.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.692+0000 7f33f524f640 1 -- 172.21.15.83:0/2787157420 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33dc00bc00 con 0x7f33dc004d40 2024-03-06T07:25:48.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.692+0000 7f33f524f640 1 -- 172.21.15.83:0/2787157420 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33dc00be00 con 0x7f33f015bb20 2024-03-06T07:25:48.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.692+0000 7f33f5a50640 1 --2- 172.21.15.83:0/2787157420 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33dc004010 0x7f33dc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.692+0000 7f33f6251640 1 --2- 172.21.15.83:0/2787157420 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f33f015bb20 0x7f33dc0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.692+0000 7f33f6a52640 1 --2- 172.21.15.83:0/2787157420 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f33dc004d40 0x7f33dc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.692+0000 7f33f5a50640 1 -- 172.21.15.83:0/2787157420 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f33f015bb20 msgr2=0x7f33dc0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:48.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.692+0000 7f33f5a50640 1 --2- 172.21.15.83:0/2787157420 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f33f015bb20 0x7f33dc0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.692+0000 7f33f5a50640 1 -- 172.21.15.83:0/2787157420 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f33dc004d40 msgr2=0x7f33dc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:48.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.692+0000 7f33f5a50640 1 --2- 172.21.15.83:0/2787157420 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f33dc004d40 0x7f33dc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.692+0000 7f33f5a50640 1 -- 172.21.15.83:0/2787157420 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33dc003520 con 0x7f33dc004010 2024-03-06T07:25:48.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.692+0000 7f33f6a52640 1 --2- 172.21.15.83:0/2787157420 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f33dc004d40 0x7f33dc0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:48.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.693+0000 7f33f5a50640 1 --2- 172.21.15.83:0/2787157420 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33dc004010 0x7f33dc004800 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f33ec00c210 tx=0x7f33dc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfecfb1fdf055d05 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:48.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.693+0000 7f33f524f640 1 -- 172.21.15.83:0/2787157420 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ec007d30 con 0x7f33dc004010 2024-03-06T07:25:48.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.693+0000 7f33f524f640 1 -- 172.21.15.83:0/2787157420 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33ec030070 con 0x7f33dc004010 2024-03-06T07:25:48.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.693+0000 7f33f524f640 1 -- 172.21.15.83:0/2787157420 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ec02cda0 con 0x7f33dc004010 2024-03-06T07:25:48.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.693+0000 7f33f84b5640 1 -- 172.21.15.83:0/2787157420 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33dc004010 msgr2=0x7f33dc004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.693+0000 7f33f84b5640 1 --2- 172.21.15.83:0/2787157420 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33dc004010 0x7f33dc004800 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f33ec00c210 tx=0x7f33dc005830 comp rx=0 tx=0).stop 2024-03-06T07:25:48.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.693+0000 7f33f6a52640 1 -- 172.21.15.83:0/2787157420 reap_dead start 2024-03-06T07:25:48.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.694+0000 7f33f84b5640 1 -- 172.21.15.83:0/2787157420 shutdown_connections 2024-03-06T07:25:48.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.694+0000 7f33f84b5640 1 -- 172.21.15.83:0/2787157420 >> 172.21.15.83:0/2787157420 conn(0x7f33f009a9f0 msgr2=0x7f33f009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:48.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.694+0000 7f33f84b5640 1 -- 172.21.15.83:0/2787157420 shutdown_connections 2024-03-06T07:25:48.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.694+0000 7f33f84b5640 1 -- 172.21.15.83:0/2787157420 wait complete. 2024-03-06T07:25:48.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.695+0000 7f33f84b5640 1 Processor -- start 2024-03-06T07:25:48.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.695+0000 7f33f84b5640 1 -- start start 2024-03-06T07:25:48.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.695+0000 7f33f84b5640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f33dc004010 0x7f33f015f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.695+0000 7f33f84b5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33f00c6980 0x7f33f006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.695+0000 7f33f84b5640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f33f006d990 0x7f33f015e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.695+0000 7f33f84b5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f00d1980 con 0x7f33f00c6980 2024-03-06T07:25:48.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.695+0000 7f33f84b5640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f00cf2d0 con 0x7f33f006d990 2024-03-06T07:25:48.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.695+0000 7f33f84b5640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33f00d1b00 con 0x7f33dc004010 2024-03-06T07:25:48.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.696+0000 7f33f5a50640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33f00c6980 0x7f33f006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.696+0000 7f33f5a50640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33f00c6980 0x7f33f006d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37542/0 (socket says 172.21.15.83:37542) 2024-03-06T07:25:48.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.696+0000 7f33f5a50640 1 -- 172.21.15.83:0/2749023205 learned_addr learned my addr 172.21.15.83:0/2749023205 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:48.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.696+0000 7f33f6251640 1 --2- 172.21.15.83:0/2749023205 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f33dc004010 0x7f33f015f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.696+0000 7f33f6a52640 1 --2- 172.21.15.83:0/2749023205 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f33f006d990 0x7f33f015e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.696+0000 7f33f5a50640 1 -- 172.21.15.83:0/2749023205 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f33dc004010 msgr2=0x7f33f015f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.696+0000 7f33f5a50640 1 --2- 172.21.15.83:0/2749023205 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f33dc004010 0x7f33f015f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.696+0000 7f33f5a50640 1 -- 172.21.15.83:0/2749023205 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f33f006d990 msgr2=0x7f33f015e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.696+0000 7f33f5a50640 1 --2- 172.21.15.83:0/2749023205 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f33f006d990 0x7f33f015e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.696+0000 7f33f5a50640 1 -- 172.21.15.83:0/2749023205 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f0160530 con 0x7f33f00c6980 2024-03-06T07:25:48.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.696+0000 7f33f6a52640 1 --2- 172.21.15.83:0/2749023205 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f33f006d990 0x7f33f015e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:48.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.696+0000 7f33f5a50640 1 --2- 172.21.15.83:0/2749023205 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33f00c6980 0x7f33f006d450 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f33ec02c570 tx=0x7f33ec001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:48.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.696+0000 7f33e2ffd640 1 -- 172.21.15.83:0/2749023205 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ec045070 con 0x7f33f00c6980 2024-03-06T07:25:48.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.696+0000 7f33e2ffd640 1 -- 172.21.15.83:0/2749023205 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33ec000ed0 con 0x7f33f00c6980 2024-03-06T07:25:48.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.696+0000 7f33f84b5640 1 -- 172.21.15.83:0/2749023205 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33f0160810 con 0x7f33f00c6980 2024-03-06T07:25:48.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.696+0000 7f33e2ffd640 1 -- 172.21.15.83:0/2749023205 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ec004e30 con 0x7f33f00c6980 2024-03-06T07:25:48.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.696+0000 7f33f84b5640 1 -- 172.21.15.83:0/2749023205 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33f0160dd0 con 0x7f33f00c6980 2024-03-06T07:25:48.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.697+0000 7f33f84b5640 1 -- 172.21.15.83:0/2749023205 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33f004f330 con 0x7f33f00c6980 2024-03-06T07:25:48.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.698+0000 7f33e2ffd640 1 -- 172.21.15.83:0/2749023205 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f33ec007070 con 0x7f33f00c6980 2024-03-06T07:25:48.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.698+0000 7f33e2ffd640 1 --2- 172.21.15.83:0/2749023205 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f33d406b4d0 0x7f33d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.698+0000 7f33f6251640 1 --2- 172.21.15.83:0/2749023205 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f33d406b4d0 0x7f33d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.698+0000 7f33e2ffd640 1 -- 172.21.15.83:0/2749023205 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f33ec0aafe0 con 0x7f33f00c6980 2024-03-06T07:25:48.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.699+0000 7f33f6251640 1 --2- 172.21.15.83:0/2749023205 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f33d406b4d0 0x7f33d406d990 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f33e4006fd0 tx=0x7f33e4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:48.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.702+0000 7f33e2ffd640 1 -- 172.21.15.83:0/2749023205 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f33ec07a240 con 0x7f33f00c6980 2024-03-06T07:25:48.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.938+0000 7f33f84b5640 1 -- 172.21.15.83:0/2749023205 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]} v 0) v1 -- 0x7f33f00b1820 con 0x7f33f00c6980 2024-03-06T07:25:48.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.957+0000 7f33e2ffd640 1 -- 172.21.15.83:0/2749023205 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]}]=0 v221) v1 ==== 140+0+0 (secure 0 0 0) 0x7f33ec079b40 con 0x7f33f00c6980 2024-03-06T07:25:48.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.959+0000 7f33f84b5640 1 -- 172.21.15.83:0/2749023205 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f33d406b4d0 msgr2=0x7f33d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.959+0000 7f33f84b5640 1 --2- 172.21.15.83:0/2749023205 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f33d406b4d0 0x7f33d406d990 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f33e4006fd0 tx=0x7f33e4008040 comp rx=0 tx=0).stop 2024-03-06T07:25:48.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.959+0000 7f33f84b5640 1 -- 172.21.15.83:0/2749023205 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33f00c6980 msgr2=0x7f33f006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.959+0000 7f33f84b5640 1 --2- 172.21.15.83:0/2749023205 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33f00c6980 0x7f33f006d450 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f33ec02c570 tx=0x7f33ec001a40 comp rx=0 tx=0).stop 2024-03-06T07:25:48.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.959+0000 7f33f84b5640 1 -- 172.21.15.83:0/2749023205 shutdown_connections 2024-03-06T07:25:48.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.959+0000 7f33f84b5640 1 --2- 172.21.15.83:0/2749023205 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f33d406b4d0 0x7f33d406d990 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.959+0000 7f33f84b5640 1 --2- 172.21.15.83:0/2749023205 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f33f006d990 0x7f33f015e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.959+0000 7f33f84b5640 1 --2- 172.21.15.83:0/2749023205 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33f00c6980 0x7f33f006d450 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.959+0000 7f33f84b5640 1 --2- 172.21.15.83:0/2749023205 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f33dc004010 0x7f33f015f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.959+0000 7f33f84b5640 1 -- 172.21.15.83:0/2749023205 >> 172.21.15.83:0/2749023205 conn(0x7f33f009a9f0 msgr2=0x7f33f006f710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:48.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.959+0000 7f33f84b5640 1 -- 172.21.15.83:0/2749023205 shutdown_connections 2024-03-06T07:25:48.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:48.959+0000 7f33f84b5640 1 -- 172.21.15.83:0/2749023205 wait complete. 2024-03-06T07:25:48.971 INFO:tasks.cephfs_test_runner:That unlinking fails when the stray directory fragment becomes too ... ok 2024-03-06T07:25:48.972 INFO:tasks.cephfs_test_runner:Starting test: test_fancy_layout (tasks.cephfs.test_strays.TestStrays) purge stray file with fancy layout 2024-03-06T07:25:48.972 INFO:tasks.cephfs_test_runner:test_fancy_layout (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:25:48.972 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:49.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.070+0000 7f530d53a640 1 Processor -- start 2024-03-06T07:25:49.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.070+0000 7f530d53a640 1 -- start start 2024-03-06T07:25:49.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.071+0000 7f530d53a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f53080d4760 0x7f53080b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.071+0000 7f530d53a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53080d25d0 con 0x7f53080d0a80 2024-03-06T07:25:49.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.071+0000 7f530d53a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53080d2870 con 0x7f53080b4170 2024-03-06T07:25:49.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.071+0000 7f530d53a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53080d4bf0 con 0x7f53080d4760 2024-03-06T07:25:49.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.071+0000 7f53067fc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f53080d4760 0x7f53080b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.071+0000 7f53067fc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f53080d4760 0x7f53080b6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37554/0 (socket says 172.21.15.83:37554) 2024-03-06T07:25:49.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.071+0000 7f53067fc640 1 -- 172.21.15.83:0/1636480201 learned_addr learned my addr 172.21.15.83:0/1636480201 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:49.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.071+0000 7f53077fe640 1 --1- 172.21.15.83:0/1636480201 >> v1:172.21.15.83:6789/0 conn(0x7f53080d0a80 0x7f53080cee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39454/0 (socket says 172.21.15.83:39454) 2024-03-06T07:25:49.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.072+0000 7f53067fc640 1 -- 172.21.15.83:0/1636480201 >> v1:172.21.15.183:6789/0 conn(0x7f53080b4170 legacy=0x7f53080d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:49.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.072+0000 7f53067fc640 1 -- 172.21.15.83:0/1636480201 >> v1:172.21.15.83:6789/0 conn(0x7f53080d0a80 legacy=0x7f53080cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.072+0000 7f53067fc640 1 -- 172.21.15.83:0/1636480201 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53080d2b50 con 0x7f53080d4760 2024-03-06T07:25:49.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.072+0000 7f5305ffb640 1 -- 172.21.15.83:0/1636480201 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 321620444 0 0) 0x7f53080d25d0 con 0x7f53080d0a80 2024-03-06T07:25:49.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.072+0000 7f53067fc640 1 --2- 172.21.15.83:0/1636480201 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53080d4760 0x7f53080b6a90 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f52fc009870 tx=0x7f52fc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=979fc3cf1563ede8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:49.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.072+0000 7f5305ffb640 1 -- 172.21.15.83:0/1636480201 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52fc034070 con 0x7f53080d4760 2024-03-06T07:25:49.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.072+0000 7f5305ffb640 1 -- 172.21.15.83:0/1636480201 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52fc030860 con 0x7f53080d4760 2024-03-06T07:25:49.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.072+0000 7f5305ffb640 1 -- 172.21.15.83:0/1636480201 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52fc030c30 con 0x7f53080d4760 2024-03-06T07:25:49.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.073+0000 7f530d53a640 1 -- 172.21.15.83:0/1636480201 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53080d4760 msgr2=0x7f53080b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.073+0000 7f530d53a640 1 --2- 172.21.15.83:0/1636480201 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53080d4760 0x7f53080b6a90 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f52fc009870 tx=0x7f52fc02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.073+0000 7f530d53a640 1 -- 172.21.15.83:0/1636480201 shutdown_connections 2024-03-06T07:25:49.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.073+0000 7f530d53a640 1 --2- 172.21.15.83:0/1636480201 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53080d4760 0x7f53080b6a90 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.073+0000 7f530d53a640 1 -- 172.21.15.83:0/1636480201 >> 172.21.15.83:0/1636480201 conn(0x7f53080d3c40 msgr2=0x7f53080d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:49.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.073+0000 7f530d53a640 1 -- 172.21.15.83:0/1636480201 shutdown_connections 2024-03-06T07:25:49.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.073+0000 7f530d53a640 1 -- 172.21.15.83:0/1636480201 wait complete. 2024-03-06T07:25:49.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.074+0000 7f530d53a640 1 Processor -- start 2024-03-06T07:25:49.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.074+0000 7f530d53a640 1 -- start start 2024-03-06T07:25:49.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.075+0000 7f530d53a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f53080b4170 0x7f53081f4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.075+0000 7f530d53a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53080d0a80 0x7f53080c7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.075+0000 7f530d53a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53080ccca0 0x7f53080cd150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.075+0000 7f530d53a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53080b4dd0 con 0x7f53080d0a80 2024-03-06T07:25:49.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.075+0000 7f530d53a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53080ce9e0 con 0x7f53080ccca0 2024-03-06T07:25:49.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.075+0000 7f530d53a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53080b4f50 con 0x7f53080b4170 2024-03-06T07:25:49.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.075+0000 7f53067fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53080d0a80 0x7f53080c7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.075+0000 7f53067fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53080d0a80 0x7f53080c7910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37568/0 (socket says 172.21.15.83:37568) 2024-03-06T07:25:49.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.075+0000 7f53067fc640 1 -- 172.21.15.83:0/3685903188 learned_addr learned my addr 172.21.15.83:0/3685903188 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:49.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.075+0000 7f5306ffd640 1 --2- 172.21.15.83:0/3685903188 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f53080b4170 0x7f53081f4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.075+0000 7f53077fe640 1 --2- 172.21.15.83:0/3685903188 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53080ccca0 0x7f53080cd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.075+0000 7f53067fc640 1 -- 172.21.15.83:0/3685903188 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f53080b4170 msgr2=0x7f53081f4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.075+0000 7f53067fc640 1 --2- 172.21.15.83:0/3685903188 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f53080b4170 0x7f53081f4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.075+0000 7f53067fc640 1 -- 172.21.15.83:0/3685903188 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53080ccca0 msgr2=0x7f53080cd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.075+0000 7f53067fc640 1 --2- 172.21.15.83:0/3685903188 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53080ccca0 0x7f53080cd150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.075+0000 7f53067fc640 1 -- 172.21.15.83:0/3685903188 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53080cbf00 con 0x7f53080d0a80 2024-03-06T07:25:49.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.076+0000 7f53067fc640 1 --2- 172.21.15.83:0/3685903188 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53080d0a80 0x7f53080c7910 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f52fc000c00 tx=0x7f52fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:49.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.076+0000 7f52ef7fe640 1 -- 172.21.15.83:0/3685903188 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52fc034040 con 0x7f53080d0a80 2024-03-06T07:25:49.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.076+0000 7f52ef7fe640 1 -- 172.21.15.83:0/3685903188 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52fc039d90 con 0x7f53080d0a80 2024-03-06T07:25:49.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.076+0000 7f52ef7fe640 1 -- 172.21.15.83:0/3685903188 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52fc043610 con 0x7f53080d0a80 2024-03-06T07:25:49.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.076+0000 7f530d53a640 1 -- 172.21.15.83:0/3685903188 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53080cc1e0 con 0x7f53080d0a80 2024-03-06T07:25:49.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.076+0000 7f530d53a640 1 -- 172.21.15.83:0/3685903188 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53080cc7a0 con 0x7f53080d0a80 2024-03-06T07:25:49.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.077+0000 7f530d53a640 1 -- 172.21.15.83:0/3685903188 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f530804f330 con 0x7f53080d0a80 2024-03-06T07:25:49.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.077+0000 7f52ef7fe640 1 -- 172.21.15.83:0/3685903188 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f52fc04a060 con 0x7f53080d0a80 2024-03-06T07:25:49.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.077+0000 7f52ef7fe640 1 --2- 172.21.15.83:0/3685903188 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f52f406b4d0 0x7f52f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.078+0000 7f52ef7fe640 1 -- 172.21.15.83:0/3685903188 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f52fc0ac150 con 0x7f53080d0a80 2024-03-06T07:25:49.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.078+0000 7f5306ffd640 1 --2- 172.21.15.83:0/3685903188 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f52f406b4d0 0x7f52f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.078+0000 7f5306ffd640 1 --2- 172.21.15.83:0/3685903188 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f52f406b4d0 0x7f52f406d990 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f52f8005520 tx=0x7f52f8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:49.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.082+0000 7f52ef7fe640 1 -- 172.21.15.83:0/3685903188 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f52fc03e070 con 0x7f53080d0a80 2024-03-06T07:25:49.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.253+0000 7f530d53a640 1 -- 172.21.15.83:0/3685903188 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53080733b0 con 0x7f53080d0a80 2024-03-06T07:25:49.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.254+0000 7f52ef7fe640 1 -- 172.21.15.83:0/3685903188 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) v1 ==== 74+0+13031 (secure 0 0 0) 0x7f52fc037360 con 0x7f53080d0a80 2024-03-06T07:25:49.256 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:25:49.256 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":74,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:25:47.949811+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.83:6837/602775648":"2024-03-07T07:25:44.762189+0000","172.21.15.83:6835/602775648":"2024-03-07T07:25:44.762189+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:49.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.256+0000 7f530d53a640 1 -- 172.21.15.83:0/3685903188 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f52f406b4d0 msgr2=0x7f52f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.256+0000 7f530d53a640 1 --2- 172.21.15.83:0/3685903188 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f52f406b4d0 0x7f52f406d990 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f52f8005520 tx=0x7f52f8009040 comp rx=0 tx=0).stop 2024-03-06T07:25:49.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.256+0000 7f530d53a640 1 -- 172.21.15.83:0/3685903188 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53080d0a80 msgr2=0x7f53080c7910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.256+0000 7f530d53a640 1 --2- 172.21.15.83:0/3685903188 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53080d0a80 0x7f53080c7910 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f52fc000c00 tx=0x7f52fc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.256+0000 7f530d53a640 1 -- 172.21.15.83:0/3685903188 shutdown_connections 2024-03-06T07:25:49.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.256+0000 7f530d53a640 1 --2- 172.21.15.83:0/3685903188 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f52f406b4d0 0x7f52f406d990 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.256+0000 7f530d53a640 1 --2- 172.21.15.83:0/3685903188 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53080ccca0 0x7f53080cd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.256+0000 7f530d53a640 1 --2- 172.21.15.83:0/3685903188 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53080d0a80 0x7f53080c7910 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.256+0000 7f530d53a640 1 --2- 172.21.15.83:0/3685903188 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f53080b4170 0x7f53081f4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.256+0000 7f530d53a640 1 -- 172.21.15.83:0/3685903188 >> 172.21.15.83:0/3685903188 conn(0x7f53080d3c40 msgr2=0x7f53080d0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:49.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.256+0000 7f530d53a640 1 -- 172.21.15.83:0/3685903188 shutdown_connections 2024-03-06T07:25:49.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.256+0000 7f530d53a640 1 -- 172.21.15.83:0/3685903188 wait complete. 2024-03-06T07:25:49.269 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:25:49.270 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:25:49.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.371+0000 7f67d7d5c640 1 Processor -- start 2024-03-06T07:25:49.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.371+0000 7f67d7d5c640 1 -- start start 2024-03-06T07:25:49.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.372+0000 7f67d7d5c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f67d0145590 0x7f67d006afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.372+0000 7f67d7d5c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f67d006b4f0 0x7f67d006efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.372+0000 7f67d7d5c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67d006b970 con 0x7f67d006f510 2024-03-06T07:25:49.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.372+0000 7f67d7d5c640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67d0145970 con 0x7f67d006b4f0 2024-03-06T07:25:49.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.372+0000 7f67d7d5c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67d0160110 con 0x7f67d0145590 2024-03-06T07:25:49.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.372+0000 7f67d52f7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f67d006b4f0 0x7f67d006efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.372+0000 7f67d52f7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f67d006b4f0 0x7f67d006efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37576/0 (socket says 172.21.15.83:37576) 2024-03-06T07:25:49.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.372+0000 7f67d52f7640 1 -- 172.21.15.83:0/2194447870 learned_addr learned my addr 172.21.15.83:0/2194447870 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:49.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.372+0000 7f67d52f7640 1 -- 172.21.15.83:0/2194447870 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f67d0145590 msgr2=0x7f67d006afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:49.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.372+0000 7f67d5af8640 1 --2- 172.21.15.83:0/2194447870 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f67d0145590 0x7f67d006afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.372+0000 7f67d52f7640 1 --2- 172.21.15.83:0/2194447870 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f67d0145590 0x7f67d006afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.372+0000 7f67d52f7640 1 -- 172.21.15.83:0/2194447870 >> v1:172.21.15.183:6789/0 conn(0x7f67d006f510 legacy=0x7f67d006daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:49.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.372+0000 7f67d52f7640 1 -- 172.21.15.83:0/2194447870 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d015f7e0 con 0x7f67d006b4f0 2024-03-06T07:25:49.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.373+0000 7f67d52f7640 1 --2- 172.21.15.83:0/2194447870 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67d006b4f0 0x7f67d006efd0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f67c0009870 tx=0x7f67c002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca6c9608142e606 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:49.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.373+0000 7f67d4af6640 1 -- 172.21.15.83:0/2194447870 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c0035020 con 0x7f67d006b4f0 2024-03-06T07:25:49.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.373+0000 7f67d4af6640 1 -- 172.21.15.83:0/2194447870 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67c0031640 con 0x7f67d006b4f0 2024-03-06T07:25:49.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.373+0000 7f67d4af6640 1 -- 172.21.15.83:0/2194447870 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c002d040 con 0x7f67d006b4f0 2024-03-06T07:25:49.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.373+0000 7f67d7d5c640 1 -- 172.21.15.83:0/2194447870 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67d006b4f0 msgr2=0x7f67d006efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.373+0000 7f67d7d5c640 1 --2- 172.21.15.83:0/2194447870 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67d006b4f0 0x7f67d006efd0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f67c0009870 tx=0x7f67c002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.374+0000 7f67d7d5c640 1 -- 172.21.15.83:0/2194447870 shutdown_connections 2024-03-06T07:25:49.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.374+0000 7f67d7d5c640 1 --2- 172.21.15.83:0/2194447870 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67d006b4f0 0x7f67d006efd0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.374+0000 7f67d7d5c640 1 --2- 172.21.15.83:0/2194447870 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f67d0145590 0x7f67d006afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.374+0000 7f67d7d5c640 1 -- 172.21.15.83:0/2194447870 >> 172.21.15.83:0/2194447870 conn(0x7f67d00b2660 msgr2=0x7f67d00b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:49.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.374+0000 7f67d7d5c640 1 -- 172.21.15.83:0/2194447870 shutdown_connections 2024-03-06T07:25:49.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.374+0000 7f67d7d5c640 1 -- 172.21.15.83:0/2194447870 wait complete. 2024-03-06T07:25:49.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.375+0000 7f67d7d5c640 1 Processor -- start 2024-03-06T07:25:49.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.375+0000 7f67d7d5c640 1 -- start start 2024-03-06T07:25:49.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.375+0000 7f67d7d5c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f67d006b4f0 0x7f67d015b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.375+0000 7f67d7d5c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f67d006f510 0x7f67d00b36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.375+0000 7f67d7d5c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67d0145590 0x7f67d0130570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.375+0000 7f67d7d5c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d00cd5d0 con 0x7f67d0145590 2024-03-06T07:25:49.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.375+0000 7f67d7d5c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d015fda0 con 0x7f67d006b4f0 2024-03-06T07:25:49.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.375+0000 7f67d7d5c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67d01605a0 con 0x7f67d006f510 2024-03-06T07:25:49.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.376+0000 7f67d62f9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67d0145590 0x7f67d0130570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.376+0000 7f67d62f9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67d0145590 0x7f67d0130570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37592/0 (socket says 172.21.15.83:37592) 2024-03-06T07:25:49.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.376+0000 7f67d62f9640 1 -- 172.21.15.83:0/2733918232 learned_addr learned my addr 172.21.15.83:0/2733918232 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:49.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.376+0000 7f67d5af8640 1 --2- 172.21.15.83:0/2733918232 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f67d006b4f0 0x7f67d015b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.376+0000 7f67d62f9640 1 -- 172.21.15.83:0/2733918232 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f67d006f510 msgr2=0x7f67d00b36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.376+0000 7f67d52f7640 1 --2- 172.21.15.83:0/2733918232 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f67d006f510 0x7f67d00b36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.376+0000 7f67d62f9640 1 --2- 172.21.15.83:0/2733918232 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f67d006f510 0x7f67d00b36f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.376+0000 7f67d62f9640 1 -- 172.21.15.83:0/2733918232 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f67d006b4f0 msgr2=0x7f67d015b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.376+0000 7f67d62f9640 1 --2- 172.21.15.83:0/2733918232 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f67d006b4f0 0x7f67d015b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.376+0000 7f67d62f9640 1 -- 172.21.15.83:0/2733918232 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d00b3d80 con 0x7f67d0145590 2024-03-06T07:25:49.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.376+0000 7f67d62f9640 1 --2- 172.21.15.83:0/2733918232 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67d0145590 0x7f67d0130570 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f67c800c7b0 tx=0x7f67c800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:49.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.376+0000 7f67c67fc640 1 -- 172.21.15.83:0/2733918232 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c8004b60 con 0x7f67d0145590 2024-03-06T07:25:49.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.376+0000 7f67c67fc640 1 -- 172.21.15.83:0/2733918232 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67c8004d00 con 0x7f67d0145590 2024-03-06T07:25:49.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.376+0000 7f67d7d5c640 1 -- 172.21.15.83:0/2733918232 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67d00b4060 con 0x7f67d0145590 2024-03-06T07:25:49.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.376+0000 7f67c67fc640 1 -- 172.21.15.83:0/2733918232 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c800aa90 con 0x7f67d0145590 2024-03-06T07:25:49.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.376+0000 7f67d7d5c640 1 -- 172.21.15.83:0/2733918232 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67d00b4620 con 0x7f67d0145590 2024-03-06T07:25:49.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.377+0000 7f67d7d5c640 1 -- 172.21.15.83:0/2733918232 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67d00733b0 con 0x7f67d0145590 2024-03-06T07:25:49.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.378+0000 7f67c67fc640 1 -- 172.21.15.83:0/2733918232 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f67c800ac30 con 0x7f67d0145590 2024-03-06T07:25:49.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.378+0000 7f67c67fc640 1 --2- 172.21.15.83:0/2733918232 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f67b406b4d0 0x7f67b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.378+0000 7f67c67fc640 1 -- 172.21.15.83:0/2733918232 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f67c808a140 con 0x7f67d0145590 2024-03-06T07:25:49.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.378+0000 7f67d5af8640 1 --2- 172.21.15.83:0/2733918232 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f67b406b4d0 0x7f67b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.378+0000 7f67d5af8640 1 --2- 172.21.15.83:0/2733918232 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f67b406b4d0 0x7f67b406d990 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f67cc006fd0 tx=0x7f67cc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:49.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.382+0000 7f67c67fc640 1 -- 172.21.15.83:0/2733918232 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f67c8020070 con 0x7f67d0145590 2024-03-06T07:25:49.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.552+0000 7f67d7d5c640 1 -- 172.21.15.83:0/2733918232 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f67d00d1130 con 0x7f67d0145590 2024-03-06T07:25:49.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.553+0000 7f67c67fc640 1 -- 172.21.15.83:0/2733918232 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v74) v1 ==== 93+0+10 (secure 0 0 0) 0x7f67c8059350 con 0x7f67d0145590 2024-03-06T07:25:49.554 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:25:49.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.554+0000 7f67d7d5c640 1 -- 172.21.15.83:0/2733918232 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f67b406b4d0 msgr2=0x7f67b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.554+0000 7f67d7d5c640 1 --2- 172.21.15.83:0/2733918232 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f67b406b4d0 0x7f67b406d990 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f67cc006fd0 tx=0x7f67cc008040 comp rx=0 tx=0).stop 2024-03-06T07:25:49.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.554+0000 7f67d7d5c640 1 -- 172.21.15.83:0/2733918232 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67d0145590 msgr2=0x7f67d0130570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.554+0000 7f67d7d5c640 1 --2- 172.21.15.83:0/2733918232 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67d0145590 0x7f67d0130570 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f67c800c7b0 tx=0x7f67c800cc70 comp rx=0 tx=0).stop 2024-03-06T07:25:49.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.554+0000 7f67d7d5c640 1 -- 172.21.15.83:0/2733918232 shutdown_connections 2024-03-06T07:25:49.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.554+0000 7f67d7d5c640 1 --2- 172.21.15.83:0/2733918232 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f67b406b4d0 0x7f67b406d990 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.554+0000 7f67d7d5c640 1 --2- 172.21.15.83:0/2733918232 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67d0145590 0x7f67d0130570 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.554+0000 7f67d7d5c640 1 --2- 172.21.15.83:0/2733918232 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f67d006f510 0x7f67d00b36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.554+0000 7f67d7d5c640 1 --2- 172.21.15.83:0/2733918232 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f67d006b4f0 0x7f67d015b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.555+0000 7f67d7d5c640 1 -- 172.21.15.83:0/2733918232 >> 172.21.15.83:0/2733918232 conn(0x7f67d00b2660 msgr2=0x7f67d00b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:49.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.555+0000 7f67d7d5c640 1 -- 172.21.15.83:0/2733918232 shutdown_connections 2024-03-06T07:25:49.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.555+0000 7f67d7d5c640 1 -- 172.21.15.83:0/2733918232 wait complete. 2024-03-06T07:25:49.567 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_fancy_layout' 2024-03-06T07:25:49.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.665+0000 7f03b40eb640 1 Processor -- start 2024-03-06T07:25:49.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.666+0000 7f03b40eb640 1 -- start start 2024-03-06T07:25:49.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.666+0000 7f03b40eb640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f03ac15b7a0 0x7f03ac0b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.666+0000 7f03b40eb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f03ac070930 0x7f03ac0b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.666+0000 7f03b40eb640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03ac070dc0 con 0x7f03ac144640 2024-03-06T07:25:49.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.666+0000 7f03b40eb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03ac0b4fe0 con 0x7f03ac070930 2024-03-06T07:25:49.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.666+0000 7f03b40eb640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03ac0ce5d0 con 0x7f03ac15b7a0 2024-03-06T07:25:49.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.667+0000 7f03b2688640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f03ac070930 0x7f03ac0b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.667+0000 7f03b2688640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f03ac070930 0x7f03ac0b4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37608/0 (socket says 172.21.15.83:37608) 2024-03-06T07:25:49.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.667+0000 7f03b2688640 1 -- 172.21.15.83:0/119338518 learned_addr learned my addr 172.21.15.83:0/119338518 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:49.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.667+0000 7f03b1686640 1 --2- 172.21.15.83:0/119338518 >> v2:172.21.15.183:3300/0 conn(0x7f03ac15b7a0 0x7f03ac0b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.667+0000 7f03b2688640 1 -- 172.21.15.83:0/119338518 >> v2:172.21.15.183:3300/0 conn(0x7f03ac15b7a0 msgr2=0x7f03ac0b34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:49.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.667+0000 7f03b2688640 1 --2- 172.21.15.83:0/119338518 >> v2:172.21.15.183:3300/0 conn(0x7f03ac15b7a0 0x7f03ac0b34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.667+0000 7f03b2688640 1 -- 172.21.15.83:0/119338518 >> v1:172.21.15.183:6789/0 conn(0x7f03ac144640 legacy=0x7f03ac15b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:49.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.667+0000 7f03b2688640 1 -- 172.21.15.83:0/119338518 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03ac0b3bb0 con 0x7f03ac070930 2024-03-06T07:25:49.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.667+0000 7f03b1686640 1 --2- 172.21.15.83:0/119338518 >> v2:172.21.15.183:3300/0 conn(0x7f03ac15b7a0 0x7f03ac0b34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:49.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.668+0000 7f03b2688640 1 --2- 172.21.15.83:0/119338518 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f03ac070930 0x7f03ac0b4aa0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f03ac0ccdf0 tx=0x7f03a402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=548b24f5a423acc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:49.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.668+0000 7f03b0e85640 1 -- 172.21.15.83:0/119338518 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a402c9f0 con 0x7f03ac070930 2024-03-06T07:25:49.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.668+0000 7f03b0e85640 1 -- 172.21.15.83:0/119338518 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03a402cb90 con 0x7f03ac070930 2024-03-06T07:25:49.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.668+0000 7f03b0e85640 1 -- 172.21.15.83:0/119338518 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a40343f0 con 0x7f03ac070930 2024-03-06T07:25:49.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.668+0000 7f03b40eb640 1 -- 172.21.15.83:0/119338518 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f03ac070930 msgr2=0x7f03ac0b4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.668+0000 7f03b40eb640 1 --2- 172.21.15.83:0/119338518 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f03ac070930 0x7f03ac0b4aa0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f03ac0ccdf0 tx=0x7f03a402be50 comp rx=0 tx=0).stop 2024-03-06T07:25:49.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.668+0000 7f03b40eb640 1 -- 172.21.15.83:0/119338518 shutdown_connections 2024-03-06T07:25:49.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.668+0000 7f03b40eb640 1 --2- 172.21.15.83:0/119338518 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f03ac070930 0x7f03ac0b4aa0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.668+0000 7f03b40eb640 1 --2- 172.21.15.83:0/119338518 >> v2:172.21.15.183:3300/0 conn(0x7f03ac15b7a0 0x7f03ac0b34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.668+0000 7f03b40eb640 1 -- 172.21.15.83:0/119338518 >> 172.21.15.83:0/119338518 conn(0x7f03ac06b3d0 msgr2=0x7f03ac06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:49.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.668+0000 7f03b40eb640 1 -- 172.21.15.83:0/119338518 shutdown_connections 2024-03-06T07:25:49.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.668+0000 7f03b40eb640 1 -- 172.21.15.83:0/119338518 wait complete. 2024-03-06T07:25:49.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.669+0000 7f03b40eb640 1 Processor -- start 2024-03-06T07:25:49.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.670+0000 7f03b40eb640 1 -- start start 2024-03-06T07:25:49.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.670+0000 7f03b40eb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f03ac070930 0x7f03ac1360c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.670+0000 7f03b40eb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f03ac144640 0x7f03ac0b2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.670+0000 7f03b40eb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f03ac15b7a0 0x7f03ac0b3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.670+0000 7f03b40eb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03ac0cdee0 con 0x7f03ac15b7a0 2024-03-06T07:25:49.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.670+0000 7f03b40eb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03ac0ce260 con 0x7f03ac070930 2024-03-06T07:25:49.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.670+0000 7f03b40eb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03ac0cea60 con 0x7f03ac144640 2024-03-06T07:25:49.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.670+0000 7f03b2688640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f03ac15b7a0 0x7f03ac0b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.670+0000 7f03b2688640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f03ac15b7a0 0x7f03ac0b3120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37618/0 (socket says 172.21.15.83:37618) 2024-03-06T07:25:49.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.670+0000 7f03b2688640 1 -- 172.21.15.83:0/3887022380 learned_addr learned my addr 172.21.15.83:0/3887022380 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:49.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.671+0000 7f03b2688640 1 -- 172.21.15.83:0/3887022380 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f03ac144640 msgr2=0x7f03ac0b2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.671+0000 7f03b2688640 1 --2- 172.21.15.83:0/3887022380 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f03ac144640 0x7f03ac0b2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.671+0000 7f03b2688640 1 -- 172.21.15.83:0/3887022380 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f03ac070930 msgr2=0x7f03ac1360c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.671+0000 7f03b2688640 1 --2- 172.21.15.83:0/3887022380 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f03ac070930 0x7f03ac1360c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.671+0000 7f03b2688640 1 -- 172.21.15.83:0/3887022380 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03ac13adb0 con 0x7f03ac15b7a0 2024-03-06T07:25:49.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.671+0000 7f03b2688640 1 --2- 172.21.15.83:0/3887022380 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f03ac15b7a0 0x7f03ac0b3120 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f03a4009f40 tx=0x7f03a4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:49.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.671+0000 7f03a27fc640 1 -- 172.21.15.83:0/3887022380 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a4043020 con 0x7f03ac15b7a0 2024-03-06T07:25:49.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.671+0000 7f03a27fc640 1 -- 172.21.15.83:0/3887022380 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03a403e450 con 0x7f03ac15b7a0 2024-03-06T07:25:49.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.671+0000 7f03b40eb640 1 -- 172.21.15.83:0/3887022380 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03ac13b090 con 0x7f03ac15b7a0 2024-03-06T07:25:49.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.671+0000 7f03a27fc640 1 -- 172.21.15.83:0/3887022380 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a4009510 con 0x7f03ac15b7a0 2024-03-06T07:25:49.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.671+0000 7f03b40eb640 1 -- 172.21.15.83:0/3887022380 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03ac13a9d0 con 0x7f03ac15b7a0 2024-03-06T07:25:49.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.672+0000 7f03b40eb640 1 -- 172.21.15.83:0/3887022380 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03ac04f330 con 0x7f03ac15b7a0 2024-03-06T07:25:49.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.672+0000 7f03a27fc640 1 -- 172.21.15.83:0/3887022380 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f03a4046020 con 0x7f03ac15b7a0 2024-03-06T07:25:49.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.673+0000 7f03a27fc640 1 --2- 172.21.15.83:0/3887022380 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f039006b4d0 0x7f039006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.673+0000 7f03a27fc640 1 -- 172.21.15.83:0/3887022380 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f03a40ac730 con 0x7f03ac15b7a0 2024-03-06T07:25:49.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.673+0000 7f03b1e87640 1 --2- 172.21.15.83:0/3887022380 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f039006b4d0 0x7f039006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.673+0000 7f03b1e87640 1 --2- 172.21.15.83:0/3887022380 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f039006b4d0 0x7f039006d990 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f03a8003cc0 tx=0x7f03a80079b0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:49.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.677+0000 7f03a27fc640 1 -- 172.21.15.83:0/3887022380 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f03a404d030 con 0x7f03ac15b7a0 2024-03-06T07:25:49.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.914+0000 7f03b40eb640 1 -- 172.21.15.83:0/3887022380 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]} v 0) v1 -- 0x7f03ac072650 con 0x7f03ac15b7a0 2024-03-06T07:25:49.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.958+0000 7f03a27fc640 1 -- 172.21.15.83:0/3887022380 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]}]=0 v222) v1 ==== 135+0+0 (secure 0 0 0) 0x7f03a407b920 con 0x7f03ac15b7a0 2024-03-06T07:25:49.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.960+0000 7f03b40eb640 1 -- 172.21.15.83:0/3887022380 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f039006b4d0 msgr2=0x7f039006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.960+0000 7f03b40eb640 1 --2- 172.21.15.83:0/3887022380 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f039006b4d0 0x7f039006d990 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f03a8003cc0 tx=0x7f03a80079b0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.960+0000 7f03b40eb640 1 -- 172.21.15.83:0/3887022380 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f03ac15b7a0 msgr2=0x7f03ac0b3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.960+0000 7f03b40eb640 1 --2- 172.21.15.83:0/3887022380 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f03ac15b7a0 0x7f03ac0b3120 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f03a4009f40 tx=0x7f03a4004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:49.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.960+0000 7f03b40eb640 1 -- 172.21.15.83:0/3887022380 shutdown_connections 2024-03-06T07:25:49.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.960+0000 7f03b40eb640 1 --2- 172.21.15.83:0/3887022380 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f039006b4d0 0x7f039006d990 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.960+0000 7f03b40eb640 1 --2- 172.21.15.83:0/3887022380 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f03ac15b7a0 0x7f03ac0b3120 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.960+0000 7f03b40eb640 1 --2- 172.21.15.83:0/3887022380 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f03ac144640 0x7f03ac0b2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.960+0000 7f03b40eb640 1 --2- 172.21.15.83:0/3887022380 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f03ac070930 0x7f03ac1360c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.960+0000 7f03b40eb640 1 -- 172.21.15.83:0/3887022380 >> 172.21.15.83:0/3887022380 conn(0x7f03ac06b3d0 msgr2=0x7f03ac070f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:49.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.960+0000 7f03b40eb640 1 -- 172.21.15.83:0/3887022380 shutdown_connections 2024-03-06T07:25:49.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:49.960+0000 7f03b40eb640 1 -- 172.21.15.83:0/3887022380 wait complete. 2024-03-06T07:25:49.973 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:25:50.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.074+0000 7fdeb16d4640 1 Processor -- start 2024-03-06T07:25:50.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.074+0000 7fdeb16d4640 1 -- start start 2024-03-06T07:25:50.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.075+0000 7fdeb16d4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdeac06fd10 0x7fdeac0cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.075+0000 7fdeb16d4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fdeac0d0af0 0x7fdeac0ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.075+0000 7fdeb16d4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeac0cd1e0 con 0x7fdeac15c050 2024-03-06T07:25:50.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.075+0000 7fdeb16d4640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdeac0cd4b0 con 0x7fdeac0d0af0 2024-03-06T07:25:50.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.075+0000 7fdeb16d4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeac0cd790 con 0x7fdeac06fd10 2024-03-06T07:25:50.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.075+0000 7fdeab7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fdeac0d0af0 0x7fdeac0ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.075+0000 7fdeab7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fdeac0d0af0 0x7fdeac0ccca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37632/0 (socket says 172.21.15.83:37632) 2024-03-06T07:25:50.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.075+0000 7fdeaaffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fdeac15c050 0x7fdeac06f600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39466/0 (socket says 172.21.15.83:39466) 2024-03-06T07:25:50.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.075+0000 7fdeab7fe640 1 -- 172.21.15.83:0/3637429960 learned_addr learned my addr 172.21.15.83:0/3637429960 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:50.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.076+0000 7fdeab7fe640 1 -- 172.21.15.83:0/3637429960 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdeac06fd10 msgr2=0x7fdeac0cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:50.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.076+0000 7fdeaa7fc640 1 --2- 172.21.15.83:0/3637429960 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdeac06fd10 0x7fdeac0cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.076+0000 7fdea9ffb640 1 -- 172.21.15.83:0/3637429960 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2681589158 0 0) 0x7fdeac0cd1e0 con 0x7fdeac15c050 2024-03-06T07:25:50.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.076+0000 7fdeab7fe640 1 --2- 172.21.15.83:0/3637429960 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdeac06fd10 0x7fdeac0cc760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.076+0000 7fdeab7fe640 1 -- 172.21.15.83:0/3637429960 >> v1:172.21.15.83:6789/0 conn(0x7fdeac15c050 legacy=0x7fdeac06f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.076+0000 7fdeab7fe640 1 -- 172.21.15.83:0/3637429960 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeac0cda70 con 0x7fdeac0d0af0 2024-03-06T07:25:50.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.076+0000 7fdeab7fe640 1 --2- 172.21.15.83:0/3637429960 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdeac0d0af0 0x7fdeac0ccca0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fdea4009870 tx=0x7fdea402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7513853f54d5c331 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:50.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.077+0000 7fdea9ffb640 1 -- 172.21.15.83:0/3637429960 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea4035020 con 0x7fdeac0d0af0 2024-03-06T07:25:50.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.077+0000 7fdea9ffb640 1 -- 172.21.15.83:0/3637429960 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea4031640 con 0x7fdeac0d0af0 2024-03-06T07:25:50.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.077+0000 7fdea9ffb640 1 -- 172.21.15.83:0/3637429960 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea402d040 con 0x7fdeac0d0af0 2024-03-06T07:25:50.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.077+0000 7fdeb16d4640 1 -- 172.21.15.83:0/3637429960 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdeac0d0af0 msgr2=0x7fdeac0ccca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.077+0000 7fdeb16d4640 1 --2- 172.21.15.83:0/3637429960 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdeac0d0af0 0x7fdeac0ccca0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fdea4009870 tx=0x7fdea402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.077+0000 7fdeb16d4640 1 -- 172.21.15.83:0/3637429960 shutdown_connections 2024-03-06T07:25:50.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.077+0000 7fdeb16d4640 1 --2- 172.21.15.83:0/3637429960 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdeac0d0af0 0x7fdeac0ccca0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.077+0000 7fdeb16d4640 1 --2- 172.21.15.83:0/3637429960 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdeac06fd10 0x7fdeac0cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.077+0000 7fdeb16d4640 1 -- 172.21.15.83:0/3637429960 >> 172.21.15.83:0/3637429960 conn(0x7fdeac0b3800 msgr2=0x7fdeac0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:50.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.077+0000 7fdeb16d4640 1 -- 172.21.15.83:0/3637429960 shutdown_connections 2024-03-06T07:25:50.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.077+0000 7fdeb16d4640 1 -- 172.21.15.83:0/3637429960 wait complete. 2024-03-06T07:25:50.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.078+0000 7fdeb16d4640 1 Processor -- start 2024-03-06T07:25:50.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.078+0000 7fdeb16d4640 1 -- start start 2024-03-06T07:25:50.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.079+0000 7fdeb16d4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdeac06fd10 0x7fdeac158820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.079+0000 7fdeb16d4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdeac0d0af0 0x7fdeac0b6d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.079+0000 7fdeb16d4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdeac15c050 0x7fdeac1f51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.079+0000 7fdeb16d4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeac165c30 con 0x7fdeac15c050 2024-03-06T07:25:50.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.079+0000 7fdeb16d4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeac165ab0 con 0x7fdeac0d0af0 2024-03-06T07:25:50.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.079+0000 7fdeb16d4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeac165db0 con 0x7fdeac06fd10 2024-03-06T07:25:50.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.079+0000 7fdeab7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdeac15c050 0x7fdeac1f51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.079+0000 7fdeab7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdeac15c050 0x7fdeac1f51b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37642/0 (socket says 172.21.15.83:37642) 2024-03-06T07:25:50.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.079+0000 7fdeab7fe640 1 -- 172.21.15.83:0/3336302672 learned_addr learned my addr 172.21.15.83:0/3336302672 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:50.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.079+0000 7fdeaaffd640 1 --2- 172.21.15.83:0/3336302672 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdeac06fd10 0x7fdeac158820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.079+0000 7fdeab7fe640 1 -- 172.21.15.83:0/3336302672 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdeac06fd10 msgr2=0x7fdeac158820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.079+0000 7fdeaa7fc640 1 --2- 172.21.15.83:0/3336302672 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdeac0d0af0 0x7fdeac0b6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.079+0000 7fdeab7fe640 1 --2- 172.21.15.83:0/3336302672 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdeac06fd10 0x7fdeac158820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.079+0000 7fdeab7fe640 1 -- 172.21.15.83:0/3336302672 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdeac0d0af0 msgr2=0x7fdeac0b6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.079+0000 7fdeab7fe640 1 --2- 172.21.15.83:0/3336302672 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdeac0d0af0 0x7fdeac0b6d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.080+0000 7fdeab7fe640 1 -- 172.21.15.83:0/3336302672 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeac15bc60 con 0x7fdeac15c050 2024-03-06T07:25:50.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.080+0000 7fdeaa7fc640 1 --2- 172.21.15.83:0/3336302672 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdeac0d0af0 0x7fdeac0b6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:50.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.080+0000 7fdeab7fe640 1 --2- 172.21.15.83:0/3336302672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdeac15c050 0x7fdeac1f51b0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fdea402b930 tx=0x7fdea40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:50.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.080+0000 7fde937fe640 1 -- 172.21.15.83:0/3336302672 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea403ac60 con 0x7fdeac15c050 2024-03-06T07:25:50.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.080+0000 7fde937fe640 1 -- 172.21.15.83:0/3336302672 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea403ae20 con 0x7fdeac15c050 2024-03-06T07:25:50.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.080+0000 7fdeb16d4640 1 -- 172.21.15.83:0/3336302672 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdeac09bd90 con 0x7fdeac15c050 2024-03-06T07:25:50.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.080+0000 7fde937fe640 1 -- 172.21.15.83:0/3336302672 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea4044630 con 0x7fdeac15c050 2024-03-06T07:25:50.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.080+0000 7fdeb16d4640 1 -- 172.21.15.83:0/3336302672 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeac09c350 con 0x7fdeac15c050 2024-03-06T07:25:50.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.081+0000 7fdeb16d4640 1 -- 172.21.15.83:0/3336302672 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeac09fe60 con 0x7fdeac15c050 2024-03-06T07:25:50.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.081+0000 7fde937fe640 1 -- 172.21.15.83:0/3336302672 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdea404b060 con 0x7fdeac15c050 2024-03-06T07:25:50.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.082+0000 7fde937fe640 1 --2- 172.21.15.83:0/3336302672 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fde9806b4d0 0x7fde9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.082+0000 7fdeaaffd640 1 --2- 172.21.15.83:0/3336302672 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fde9806b4d0 0x7fde9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.082+0000 7fde937fe640 1 -- 172.21.15.83:0/3336302672 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fdea40abbe0 con 0x7fdeac15c050 2024-03-06T07:25:50.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.082+0000 7fdeaaffd640 1 --2- 172.21.15.83:0/3336302672 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fde9806b4d0 0x7fde9806d990 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fde9c0012e0 tx=0x7fde9c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:50.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.086+0000 7fde937fe640 1 -- 172.21.15.83:0/3336302672 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdea403f070 con 0x7fdeac15c050 2024-03-06T07:25:50.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.257+0000 7fdeb16d4640 1 -- 172.21.15.83:0/3336302672 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fdeac0d2a80 con 0x7fdeac15c050 2024-03-06T07:25:50.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.260+0000 7fde937fe640 1 -- 172.21.15.83:0/3336302672 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v26)=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7fdea407ad60 con 0x7fdeac15c050 2024-03-06T07:25:50.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.262+0000 7fdeb16d4640 1 -- 172.21.15.83:0/3336302672 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fde9806b4d0 msgr2=0x7fde9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.262+0000 7fdeb16d4640 1 --2- 172.21.15.83:0/3336302672 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fde9806b4d0 0x7fde9806d990 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fde9c0012e0 tx=0x7fde9c009040 comp rx=0 tx=0).stop 2024-03-06T07:25:50.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.262+0000 7fdeb16d4640 1 -- 172.21.15.83:0/3336302672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdeac15c050 msgr2=0x7fdeac1f51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.262+0000 7fdeb16d4640 1 --2- 172.21.15.83:0/3336302672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdeac15c050 0x7fdeac1f51b0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fdea402b930 tx=0x7fdea40046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.263+0000 7fdeb16d4640 1 -- 172.21.15.83:0/3336302672 shutdown_connections 2024-03-06T07:25:50.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.263+0000 7fdeb16d4640 1 --2- 172.21.15.83:0/3336302672 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fde9806b4d0 0x7fde9806d990 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.263+0000 7fdeb16d4640 1 --2- 172.21.15.83:0/3336302672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdeac15c050 0x7fdeac1f51b0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.263+0000 7fdeb16d4640 1 --2- 172.21.15.83:0/3336302672 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdeac0d0af0 0x7fdeac0b6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.263+0000 7fdeb16d4640 1 --2- 172.21.15.83:0/3336302672 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdeac06fd10 0x7fdeac158820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.263+0000 7fdeb16d4640 1 -- 172.21.15.83:0/3336302672 >> 172.21.15.83:0/3336302672 conn(0x7fdeac0b3800 msgr2=0x7fdeac0d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:50.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.263+0000 7fdeb16d4640 1 -- 172.21.15.83:0/3336302672 shutdown_connections 2024-03-06T07:25:50.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.263+0000 7fdeb16d4640 1 -- 172.21.15.83:0/3336302672 wait complete. 2024-03-06T07:25:50.275 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:25:50.276 DEBUG:teuthology.orchestra.run.smithi083:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:50.315 DEBUG:teuthology.orchestra.run.smithi183:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:50.369 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:25:50.369 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:25:50.369 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:50.385 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1257 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:25:50.385 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:25:50.385 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:50.402 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=287 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:25:50.402 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:25:50.402 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:50.440 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1257 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:25:50.441 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:25:50.441 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:50.458 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=287 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:25:50.458 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:25:50.459 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:50.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.578+0000 7f11b6b0c640 1 Processor -- start 2024-03-06T07:25:50.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.578+0000 7f11b6b0c640 1 -- start start 2024-03-06T07:25:50.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.579+0000 7f11b6b0c640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f11b01424f0 0x7f11b00cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.579+0000 7f11b6b0c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f11b00cd7a0 0x7f11b015cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.579+0000 7f11b6b0c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11b00cc980 con 0x7f11b006d330 2024-03-06T07:25:50.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.579+0000 7f11b6b0c640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11b00cd220 con 0x7f11b00cd7a0 2024-03-06T07:25:50.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.579+0000 7f11b6b0c640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11b00ccc60 con 0x7f11b01424f0 2024-03-06T07:25:50.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.579+0000 7f11affff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f11b00cd7a0 0x7f11b015cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.579+0000 7f11affff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f11b00cd7a0 0x7f11b015cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37652/0 (socket says 172.21.15.83:37652) 2024-03-06T07:25:50.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.579+0000 7f11affff640 1 -- 172.21.15.83:0/3445864924 learned_addr learned my addr 172.21.15.83:0/3445864924 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:50.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.579+0000 7f11b48a8640 1 --2- 172.21.15.83:0/3445864924 >> v2:172.21.15.183:3300/0 conn(0x7f11b01424f0 0x7f11b00cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.579+0000 7f11affff640 1 -- 172.21.15.83:0/3445864924 >> v2:172.21.15.183:3300/0 conn(0x7f11b01424f0 msgr2=0x7f11b00cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:50.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.579+0000 7f11affff640 1 --2- 172.21.15.83:0/3445864924 >> v2:172.21.15.183:3300/0 conn(0x7f11b01424f0 0x7f11b00cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.579+0000 7f11affff640 1 -- 172.21.15.83:0/3445864924 >> v1:172.21.15.183:6789/0 conn(0x7f11b006d330 legacy=0x7f11b0071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:50.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.579+0000 7f11affff640 1 -- 172.21.15.83:0/3445864924 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b00ccf40 con 0x7f11b00cd7a0 2024-03-06T07:25:50.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.580+0000 7f11affff640 1 --2- 172.21.15.83:0/3445864924 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11b00cd7a0 0x7f11b015cef0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f11b006c6d0 tx=0x7f119c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a38ad2ddd00b0178 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:50.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.580+0000 7f11af7fe640 1 -- 172.21.15.83:0/3445864924 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f119c035020 con 0x7f11b00cd7a0 2024-03-06T07:25:50.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.580+0000 7f11af7fe640 1 -- 172.21.15.83:0/3445864924 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f119c031870 con 0x7f11b00cd7a0 2024-03-06T07:25:50.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.580+0000 7f11af7fe640 1 -- 172.21.15.83:0/3445864924 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f119c031c40 con 0x7f11b00cd7a0 2024-03-06T07:25:50.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.581+0000 7f11b6b0c640 1 -- 172.21.15.83:0/3445864924 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11b00cd7a0 msgr2=0x7f11b015cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.581+0000 7f11b6b0c640 1 --2- 172.21.15.83:0/3445864924 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11b00cd7a0 0x7f11b015cef0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f11b006c6d0 tx=0x7f119c02b910 comp rx=0 tx=0).stop 2024-03-06T07:25:50.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.581+0000 7f11b6b0c640 1 -- 172.21.15.83:0/3445864924 shutdown_connections 2024-03-06T07:25:50.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.581+0000 7f11b6b0c640 1 --2- 172.21.15.83:0/3445864924 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11b00cd7a0 0x7f11b015cef0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.581+0000 7f11b6b0c640 1 --2- 172.21.15.83:0/3445864924 >> v2:172.21.15.183:3300/0 conn(0x7f11b01424f0 0x7f11b00cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.581+0000 7f11b6b0c640 1 -- 172.21.15.83:0/3445864924 >> 172.21.15.83:0/3445864924 conn(0x7f11b00cf8a0 msgr2=0x7f11b00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:50.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.581+0000 7f11b6b0c640 1 -- 172.21.15.83:0/3445864924 shutdown_connections 2024-03-06T07:25:50.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.581+0000 7f11b6b0c640 1 -- 172.21.15.83:0/3445864924 wait complete. 2024-03-06T07:25:50.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.582+0000 7f11b6b0c640 1 Processor -- start 2024-03-06T07:25:50.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.582+0000 7f11b6b0c640 1 -- start start 2024-03-06T07:25:50.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.583+0000 7f11b6b0c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11b006d330 0x7f11b0140da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.583+0000 7f11b6b0c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11b00cd7a0 0x7f11b00af5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.583+0000 7f11b6b0c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11b01424f0 0x7f11b00afb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.583+0000 7f11b6b0c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b006d920 con 0x7f11b00cd7a0 2024-03-06T07:25:50.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.583+0000 7f11b6b0c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b006d7a0 con 0x7f11b006d330 2024-03-06T07:25:50.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.583+0000 7f11b6b0c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11b006daa0 con 0x7f11b01424f0 2024-03-06T07:25:50.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.583+0000 7f11affff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11b00cd7a0 0x7f11b00af5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.583+0000 7f11affff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11b00cd7a0 0x7f11b00af5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37666/0 (socket says 172.21.15.83:37666) 2024-03-06T07:25:50.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.583+0000 7f11affff640 1 -- 172.21.15.83:0/251604335 learned_addr learned my addr 172.21.15.83:0/251604335 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:50.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.583+0000 7f11affff640 1 -- 172.21.15.83:0/251604335 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11b01424f0 msgr2=0x7f11b00afb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.583+0000 7f11affff640 1 --2- 172.21.15.83:0/251604335 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11b01424f0 0x7f11b00afb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.583+0000 7f11affff640 1 -- 172.21.15.83:0/251604335 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11b006d330 msgr2=0x7f11b0140da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.583+0000 7f11affff640 1 --2- 172.21.15.83:0/251604335 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11b006d330 0x7f11b0140da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.583+0000 7f11affff640 1 -- 172.21.15.83:0/251604335 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b00b0360 con 0x7f11b00cd7a0 2024-03-06T07:25:50.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.583+0000 7f11affff640 1 --2- 172.21.15.83:0/251604335 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11b00cd7a0 0x7f11b00af5c0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f119c02b950 tx=0x7f119c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:50.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.583+0000 7f11ad7fa640 1 -- 172.21.15.83:0/251604335 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f119c040070 con 0x7f11b00cd7a0 2024-03-06T07:25:50.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.583+0000 7f11ad7fa640 1 -- 172.21.15.83:0/251604335 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f119c030d00 con 0x7f11b00cd7a0 2024-03-06T07:25:50.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.583+0000 7f11ad7fa640 1 -- 172.21.15.83:0/251604335 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f119c035040 con 0x7f11b00cd7a0 2024-03-06T07:25:50.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.583+0000 7f11b6b0c640 1 -- 172.21.15.83:0/251604335 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11b00b0670 con 0x7f11b00cd7a0 2024-03-06T07:25:50.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.583+0000 7f11b6b0c640 1 -- 172.21.15.83:0/251604335 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11b00b0eb0 con 0x7f11b00cd7a0 2024-03-06T07:25:50.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.584+0000 7f11b6b0c640 1 -- 172.21.15.83:0/251604335 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11b004f330 con 0x7f11b00cd7a0 2024-03-06T07:25:50.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.585+0000 7f11ad7fa640 1 -- 172.21.15.83:0/251604335 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f119c04b060 con 0x7f11b00cd7a0 2024-03-06T07:25:50.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.585+0000 7f11ad7fa640 1 --2- 172.21.15.83:0/251604335 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f119406b4d0 0x7f119406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.585+0000 7f11ad7fa640 1 -- 172.21.15.83:0/251604335 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f119c0abd00 con 0x7f11b00cd7a0 2024-03-06T07:25:50.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.585+0000 7f11b48a8640 1 --2- 172.21.15.83:0/251604335 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f119406b4d0 0x7f119406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.586+0000 7f11b48a8640 1 --2- 172.21.15.83:0/251604335 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f119406b4d0 0x7f119406d990 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f11a4006fd0 tx=0x7f11a4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:50.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.590+0000 7f11ad7fa640 1 -- 172.21.15.83:0/251604335 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f11b00b0670 con 0x7f11b00cd7a0 2024-03-06T07:25:50.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.794+0000 7f11b6b0c640 1 -- 172.21.15.83:0/251604335 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11b00d0620 con 0x7f11b00cd7a0 2024-03-06T07:25:50.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.794+0000 7f11ad7fa640 1 -- 172.21.15.83:0/251604335 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f119c07ae80 con 0x7f11b00cd7a0 2024-03-06T07:25:50.796 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:25:50.797 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":34,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6010,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/2142942385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2142942385},{"type":"v1","addr":"172.21.15.183:6834","nonce":2142942385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":6013,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3742438064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":3742438064},{"type":"v1","addr":"172.21.15.183:6835","nonce":3742438064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":6540,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/1549894851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1549894851},{"type":"v1","addr":"172.21.15.83:6836","nonce":1549894851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":6876,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/1539710402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1539710402},{"type":"v1","addr":"172.21.15.83:6837","nonce":1539710402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34}],"filesystems":[]} 2024-03-06T07:25:50.797 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 34 2024-03-06T07:25:50.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.796+0000 7f11b6b0c640 1 -- 172.21.15.83:0/251604335 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f119406b4d0 msgr2=0x7f119406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.796+0000 7f11b6b0c640 1 --2- 172.21.15.83:0/251604335 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f119406b4d0 0x7f119406d990 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f11a4006fd0 tx=0x7f11a4008040 comp rx=0 tx=0).stop 2024-03-06T07:25:50.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.796+0000 7f11b6b0c640 1 -- 172.21.15.83:0/251604335 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11b00cd7a0 msgr2=0x7f11b00af5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.796+0000 7f11b6b0c640 1 --2- 172.21.15.83:0/251604335 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11b00cd7a0 0x7f11b00af5c0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f119c02b950 tx=0x7f119c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.796+0000 7f11b6b0c640 1 -- 172.21.15.83:0/251604335 shutdown_connections 2024-03-06T07:25:50.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.796+0000 7f11b6b0c640 1 --2- 172.21.15.83:0/251604335 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f119406b4d0 0x7f119406d990 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.796+0000 7f11b6b0c640 1 --2- 172.21.15.83:0/251604335 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11b01424f0 0x7f11b00afb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.796+0000 7f11b6b0c640 1 --2- 172.21.15.83:0/251604335 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11b00cd7a0 0x7f11b00af5c0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.796+0000 7f11b6b0c640 1 --2- 172.21.15.83:0/251604335 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11b006d330 0x7f11b0140da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.796+0000 7f11b6b0c640 1 -- 172.21.15.83:0/251604335 >> 172.21.15.83:0/251604335 conn(0x7f11b00cf8a0 msgr2=0x7f11b015f9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:50.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.796+0000 7f11b6b0c640 1 -- 172.21.15.83:0/251604335 shutdown_connections 2024-03-06T07:25:50.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.796+0000 7f11b6b0c640 1 -- 172.21.15.83:0/251604335 wait complete. 2024-03-06T07:25:50.810 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:25:50.810 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:25:50.810 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:25:50.810 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:25:50.810 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:25:50.810 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:25:50.811 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:25:50.811 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:25:50.811 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:25:50.811 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:25:50.811 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:25:50.811 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:25:50.811 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:25:50.811 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:25:50.812 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:25:50.812 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:25:50.820 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:25:50.820 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:25:50.821 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:25:50.821 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:25:50.826 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:25:50.827 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:25:50.827 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:25:50.827 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:25:50.863 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:25:50.863 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:50.864 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:25:50.864 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:50.869 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:25:50.869 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:50.869 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:25:50.870 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:50.870 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:25:50.939 INFO:tasks.ceph.mds.c.smithi083.stdout:starting mds.c at 2024-03-06T07:25:50.939 INFO:tasks.ceph.mds.a.smithi083.stdout:starting mds.a at 2024-03-06T07:25:50.953 INFO:tasks.ceph.mds.b.smithi183.stdout:starting mds.b at 2024-03-06T07:25:50.953 INFO:tasks.ceph.mds.d.smithi183.stdout:starting mds.d at 2024-03-06T07:25:50.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.974+0000 7f27451ac640 1 Processor -- start 2024-03-06T07:25:50.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.975+0000 7f27451ac640 1 -- start start 2024-03-06T07:25:50.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.975+0000 7f27451ac640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2740162400 0x7f27400d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.975+0000 7f27451ac640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27400d22c0 con 0x7f2740162970 2024-03-06T07:25:50.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.975+0000 7f27451ac640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27400d25a0 con 0x7f27400b4460 2024-03-06T07:25:50.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.975+0000 7f27451ac640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2740161b00 con 0x7f2740162400 2024-03-06T07:25:50.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.975+0000 7f273f59e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2740162400 0x7f27400d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.975+0000 7f273e59c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f2740162970 0x7f2740161cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39488/0 (socket says 172.21.15.83:39488) 2024-03-06T07:25:50.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.975+0000 7f273f59e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2740162400 0x7f27400d4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37700/0 (socket says 172.21.15.83:37700) 2024-03-06T07:25:50.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.975+0000 7f273e59c640 1 -- 172.21.15.83:0/330230533 learned_addr learned my addr 172.21.15.83:0/330230533 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:25:50.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.976+0000 7f273f59e640 1 -- 172.21.15.83:0/330230533 >> v1:172.21.15.183:6789/0 conn(0x7f27400b4460 legacy=0x7f27400b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.976+0000 7f273f59e640 1 -- 172.21.15.83:0/330230533 >> v1:172.21.15.83:6789/0 conn(0x7f2740162970 legacy=0x7f2740161cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.976+0000 7f273dd9b640 1 -- 172.21.15.83:0/330230533 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 421344778 0 0) 0x7f27400d25a0 con 0x7f27400b4460 2024-03-06T07:25:50.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.976+0000 7f273f59e640 1 -- 172.21.15.83:0/330230533 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27400d2880 con 0x7f2740162400 2024-03-06T07:25:50.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.976+0000 7f273f59e640 1 --2- 172.21.15.83:0/330230533 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2740162400 0x7f27400d4800 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f2738009d90 tx=0x7f273802bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3e7caae1dbdbb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:50.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.977+0000 7f273dd9b640 1 -- 172.21.15.83:0/330230533 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2738039070 con 0x7f2740162400 2024-03-06T07:25:50.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.977+0000 7f273dd9b640 1 -- 172.21.15.83:0/330230533 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f273802cab0 con 0x7f2740162400 2024-03-06T07:25:50.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.977+0000 7f273dd9b640 1 -- 172.21.15.83:0/330230533 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2738034440 con 0x7f2740162400 2024-03-06T07:25:50.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.977+0000 7f27451ac640 1 -- 172.21.15.83:0/330230533 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2740162400 msgr2=0x7f27400d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.977+0000 7f27451ac640 1 --2- 172.21.15.83:0/330230533 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2740162400 0x7f27400d4800 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f2738009d90 tx=0x7f273802bd80 comp rx=0 tx=0).stop 2024-03-06T07:25:50.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.977+0000 7f27451ac640 1 -- 172.21.15.83:0/330230533 shutdown_connections 2024-03-06T07:25:50.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.977+0000 7f27451ac640 1 --2- 172.21.15.83:0/330230533 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2740162400 0x7f27400d4800 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.977+0000 7f27451ac640 1 -- 172.21.15.83:0/330230533 >> 172.21.15.83:0/330230533 conn(0x7f27400d3ce0 msgr2=0x7f27400d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:50.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.977+0000 7f27451ac640 1 -- 172.21.15.83:0/330230533 shutdown_connections 2024-03-06T07:25:50.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.977+0000 7f27451ac640 1 -- 172.21.15.83:0/330230533 wait complete. 2024-03-06T07:25:50.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.979+0000 7f27451ac640 1 Processor -- start 2024-03-06T07:25:50.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.979+0000 7f27451ac640 1 -- start start 2024-03-06T07:25:50.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.979+0000 7f27451ac640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27400b4460 0x7f27400a1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.979+0000 7f27451ac640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2740162400 0x7f27400c7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.979+0000 7f27451ac640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2740162970 0x7f27400c7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.979+0000 7f27451ac640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27400ce950 con 0x7f27400b4460 2024-03-06T07:25:50.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.979+0000 7f27451ac640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27400cecc0 con 0x7f2740162970 2024-03-06T07:25:50.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.979+0000 7f27451ac640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27400b4840 con 0x7f2740162400 2024-03-06T07:25:50.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.979+0000 7f273ed9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27400b4460 0x7f27400a1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.980+0000 7f273ed9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27400b4460 0x7f27400a1080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37704/0 (socket says 172.21.15.83:37704) 2024-03-06T07:25:50.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.980+0000 7f273ed9d640 1 -- 172.21.15.83:0/3971951612 learned_addr learned my addr 172.21.15.83:0/3971951612 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:50.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.980+0000 7f273e59c640 1 --2- 172.21.15.83:0/3971951612 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2740162400 0x7f27400c7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.980+0000 7f273f59e640 1 --2- 172.21.15.83:0/3971951612 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2740162970 0x7f27400c7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.980+0000 7f273ed9d640 1 -- 172.21.15.83:0/3971951612 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2740162400 msgr2=0x7f27400c7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.980+0000 7f273ed9d640 1 --2- 172.21.15.83:0/3971951612 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2740162400 0x7f27400c7810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.980+0000 7f273ed9d640 1 -- 172.21.15.83:0/3971951612 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2740162970 msgr2=0x7f27400c7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.980+0000 7f273ed9d640 1 --2- 172.21.15.83:0/3971951612 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2740162970 0x7f27400c7d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.980+0000 7f273ed9d640 1 -- 172.21.15.83:0/3971951612 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27400a15c0 con 0x7f27400b4460 2024-03-06T07:25:50.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.980+0000 7f273ed9d640 1 --2- 172.21.15.83:0/3971951612 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27400b4460 0x7f27400a1080 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f273000c550 tx=0x7f273000ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:50.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.980+0000 7f27277fe640 1 -- 172.21.15.83:0/3971951612 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2730017070 con 0x7f27400b4460 2024-03-06T07:25:50.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.980+0000 7f27277fe640 1 -- 172.21.15.83:0/3971951612 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27300035f0 con 0x7f27400b4460 2024-03-06T07:25:50.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.980+0000 7f27451ac640 1 -- 172.21.15.83:0/3971951612 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27400cd660 con 0x7f27400b4460 2024-03-06T07:25:50.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.981+0000 7f27277fe640 1 -- 172.21.15.83:0/3971951612 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27300039c0 con 0x7f27400b4460 2024-03-06T07:25:50.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.981+0000 7f27451ac640 1 -- 172.21.15.83:0/3971951612 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27400b7790 con 0x7f27400b4460 2024-03-06T07:25:50.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.982+0000 7f27451ac640 1 -- 172.21.15.83:0/3971951612 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f274004f330 con 0x7f27400b4460 2024-03-06T07:25:50.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.982+0000 7f27277fe640 1 -- 172.21.15.83:0/3971951612 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f273001a070 con 0x7f27400b4460 2024-03-06T07:25:50.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.982+0000 7f27277fe640 1 --2- 172.21.15.83:0/3971951612 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f272c06b4d0 0x7f272c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.982+0000 7f27277fe640 1 -- 172.21.15.83:0/3971951612 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f27300891b0 con 0x7f27400b4460 2024-03-06T07:25:50.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.982+0000 7f273e59c640 1 --2- 172.21.15.83:0/3971951612 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f272c06b4d0 0x7f272c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.983+0000 7f273e59c640 1 --2- 172.21.15.83:0/3971951612 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f272c06b4d0 0x7f272c06d990 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f27400b7440 tx=0x7f2734007e20 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:50.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:50.986+0000 7f27277fe640 1 -- 172.21.15.83:0/3971951612 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2730057d70 con 0x7f27400b4460 2024-03-06T07:25:51.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.172+0000 7f27451ac640 1 -- 172.21.15.83:0/3971951612 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f274006b180 con 0x7f27400b4460 2024-03-06T07:25:51.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.269+0000 7f27277fe640 1 -- 172.21.15.83:0/3971951612 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v78) v1 ==== 97+0+0 (secure 0 0 0) 0x7f273005e0c0 con 0x7f27400b4460 2024-03-06T07:25:51.271 INFO:teuthology.orchestra.run.smithi083.stderr: removed all blocklist entries 2024-03-06T07:25:51.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.271+0000 7f27451ac640 1 -- 172.21.15.83:0/3971951612 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f272c06b4d0 msgr2=0x7f272c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.271+0000 7f27451ac640 1 --2- 172.21.15.83:0/3971951612 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f272c06b4d0 0x7f272c06d990 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f27400b7440 tx=0x7f2734007e20 comp rx=0 tx=0).stop 2024-03-06T07:25:51.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.271+0000 7f27451ac640 1 -- 172.21.15.83:0/3971951612 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27400b4460 msgr2=0x7f27400a1080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.271+0000 7f27451ac640 1 --2- 172.21.15.83:0/3971951612 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27400b4460 0x7f27400a1080 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f273000c550 tx=0x7f273000ca10 comp rx=0 tx=0).stop 2024-03-06T07:25:51.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.271+0000 7f27451ac640 1 -- 172.21.15.83:0/3971951612 shutdown_connections 2024-03-06T07:25:51.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.271+0000 7f27451ac640 1 --2- 172.21.15.83:0/3971951612 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f272c06b4d0 0x7f272c06d990 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.271+0000 7f27451ac640 1 --2- 172.21.15.83:0/3971951612 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2740162970 0x7f27400c7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.271+0000 7f27451ac640 1 --2- 172.21.15.83:0/3971951612 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2740162400 0x7f27400c7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.271+0000 7f27451ac640 1 --2- 172.21.15.83:0/3971951612 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27400b4460 0x7f27400a1080 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.271+0000 7f27451ac640 1 -- 172.21.15.83:0/3971951612 >> 172.21.15.83:0/3971951612 conn(0x7f27400d3ce0 msgr2=0x7f27400b5670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:51.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.271+0000 7f27451ac640 1 -- 172.21.15.83:0/3971951612 shutdown_connections 2024-03-06T07:25:51.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.271+0000 7f27451ac640 1 -- 172.21.15.83:0/3971951612 wait complete. 2024-03-06T07:25:51.285 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:25:51.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.382+0000 7f232da88640 1 Processor -- start 2024-03-06T07:25:51.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.382+0000 7f232da88640 1 -- start start 2024-03-06T07:25:51.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.383+0000 7f232da88640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f23280b3060 0x7f232815e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.383+0000 7f232da88640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f232815f580 con 0x7f232813dc10 2024-03-06T07:25:51.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.383+0000 7f232da88640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f232815b450 con 0x7f232815bf30 2024-03-06T07:25:51.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.383+0000 7f232da88640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23280cf3f0 con 0x7f23280b3060 2024-03-06T07:25:51.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.383+0000 7f23277fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f232813dc10 0x7f23280b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39494/0 (socket says 172.21.15.83:39494) 2024-03-06T07:25:51.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.383+0000 7f23277fe640 1 -- 172.21.15.83:0/2591740413 learned_addr learned my addr 172.21.15.83:0/2591740413 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:25:51.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.383+0000 7f2326ffd640 1 --2- 172.21.15.83:0/2591740413 >> v2:172.21.15.183:3300/0 conn(0x7f23280b3060 0x7f232815e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.383+0000 7f23267fc640 1 -- 172.21.15.83:0/2591740413 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 271427176 0 0) 0x7f232815f580 con 0x7f232813dc10 2024-03-06T07:25:51.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.383+0000 7f23267fc640 1 -- 172.21.15.83:0/2591740413 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2310003520 con 0x7f232813dc10 2024-03-06T07:25:51.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.383+0000 7f23267fc640 1 -- 172.21.15.83:0/2591740413 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4256204788 0 0) 0x7f2310003520 con 0x7f232813dc10 2024-03-06T07:25:51.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.383+0000 7f23267fc640 1 -- 172.21.15.83:0/2591740413 >> v2:172.21.15.183:3300/0 conn(0x7f23280b3060 msgr2=0x7f232815e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:51.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.383+0000 7f23267fc640 1 --2- 172.21.15.83:0/2591740413 >> v2:172.21.15.183:3300/0 conn(0x7f23280b3060 0x7f232815e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.383+0000 7f23267fc640 1 -- 172.21.15.83:0/2591740413 >> v1:172.21.15.183:6789/0 conn(0x7f232815bf30 legacy=0x7f232815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.383+0000 7f23267fc640 1 -- 172.21.15.83:0/2591740413 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f232815b730 con 0x7f232813dc10 2024-03-06T07:25:51.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.383+0000 7f23267fc640 1 -- 172.21.15.83:0/2591740413 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f2318001670 con 0x7f232813dc10 2024-03-06T07:25:51.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.383+0000 7f23267fc640 1 -- 172.21.15.83:0/2591740413 >> v1:172.21.15.83:6789/0 conn(0x7f232813dc10 legacy=0x7f23280b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.383+0000 7f23267fc640 1 --2- 172.21.15.83:0/2591740413 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f23100039b0 0x7f2310003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.383+0000 7f23267fc640 1 --2- 172.21.15.83:0/2591740413 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2310004800 0x7f2310004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.383+0000 7f23267fc640 1 --2- 172.21.15.83:0/2591740413 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23100051c0 0x7f231000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.383+0000 7f23267fc640 1 -- 172.21.15.83:0/2591740413 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23280cf3f0 con 0x7f23100051c0 2024-03-06T07:25:51.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.383+0000 7f23267fc640 1 -- 172.21.15.83:0/2591740413 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f231000c160 con 0x7f23100039b0 2024-03-06T07:25:51.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.383+0000 7f23267fc640 1 -- 172.21.15.83:0/2591740413 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f231000c360 con 0x7f2310004800 2024-03-06T07:25:51.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.384+0000 7f2327fff640 1 --2- 172.21.15.83:0/2591740413 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23100051c0 0x7f231000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.384+0000 7f2326ffd640 1 --2- 172.21.15.83:0/2591740413 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2310004800 0x7f2310004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.384+0000 7f23277fe640 1 --2- 172.21.15.83:0/2591740413 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f23100039b0 0x7f2310003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.384+0000 7f2327fff640 1 -- 172.21.15.83:0/2591740413 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2310004800 msgr2=0x7f2310004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:51.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.384+0000 7f2327fff640 1 --2- 172.21.15.83:0/2591740413 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2310004800 0x7f2310004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.384+0000 7f2327fff640 1 -- 172.21.15.83:0/2591740413 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f23100039b0 msgr2=0x7f2310003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:51.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.384+0000 7f2327fff640 1 --2- 172.21.15.83:0/2591740413 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f23100039b0 0x7f2310003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.384+0000 7f2327fff640 1 -- 172.21.15.83:0/2591740413 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2310003520 con 0x7f23100051c0 2024-03-06T07:25:51.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.384+0000 7f2327fff640 1 -- 172.21.15.83:0/2591740413 reap_dead start 2024-03-06T07:25:51.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.384+0000 7f2327fff640 1 --2- 172.21.15.83:0/2591740413 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23100051c0 0x7f231000b9c0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f2320001e50 tx=0x7f2310005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a9e3d944b423e63 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:51.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.385+0000 7f23267fc640 1 -- 172.21.15.83:0/2591740413 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2320001a70 con 0x7f23100051c0 2024-03-06T07:25:51.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.385+0000 7f23267fc640 1 -- 172.21.15.83:0/2591740413 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2320003b90 con 0x7f23100051c0 2024-03-06T07:25:51.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.385+0000 7f23267fc640 1 -- 172.21.15.83:0/2591740413 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23200050d0 con 0x7f23100051c0 2024-03-06T07:25:51.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.385+0000 7f232da88640 1 -- 172.21.15.83:0/2591740413 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23100051c0 msgr2=0x7f231000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.385+0000 7f232da88640 1 --2- 172.21.15.83:0/2591740413 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23100051c0 0x7f231000b9c0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f2320001e50 tx=0x7f2310005830 comp rx=0 tx=0).stop 2024-03-06T07:25:51.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.385+0000 7f232da88640 1 -- 172.21.15.83:0/2591740413 shutdown_connections 2024-03-06T07:25:51.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.385+0000 7f232da88640 1 --2- 172.21.15.83:0/2591740413 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23100051c0 0x7f231000b9c0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.385+0000 7f232da88640 1 -- 172.21.15.83:0/2591740413 >> 172.21.15.83:0/2591740413 conn(0x7f23280cf8a0 msgr2=0x7f23280cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:51.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.385+0000 7f232da88640 1 -- 172.21.15.83:0/2591740413 shutdown_connections 2024-03-06T07:25:51.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.385+0000 7f232da88640 1 -- 172.21.15.83:0/2591740413 wait complete. 2024-03-06T07:25:51.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.386+0000 7f232da88640 1 Processor -- start 2024-03-06T07:25:51.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.387+0000 7f232da88640 1 -- start start 2024-03-06T07:25:51.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.387+0000 7f232da88640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f23100051c0 0x7f232813cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.387+0000 7f232da88640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f232815cd30 0x7f232813e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.387+0000 7f232da88640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f232813e880 0x7f23280cdc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.387+0000 7f232da88640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23280b2950 con 0x7f232813e880 2024-03-06T07:25:51.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.387+0000 7f232da88640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f232815b210 con 0x7f23100051c0 2024-03-06T07:25:51.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.387+0000 7f232da88640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23280b2ad0 con 0x7f232815cd30 2024-03-06T07:25:51.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.387+0000 7f2327fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f232813e880 0x7f23280cdc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.387+0000 7f2327fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f232813e880 0x7f23280cdc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37726/0 (socket says 172.21.15.83:37726) 2024-03-06T07:25:51.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.387+0000 7f2327fff640 1 -- 172.21.15.83:0/1023648878 learned_addr learned my addr 172.21.15.83:0/1023648878 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:51.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.388+0000 7f23277fe640 1 --2- 172.21.15.83:0/1023648878 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f23100051c0 0x7f232813cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.388+0000 7f2326ffd640 1 --2- 172.21.15.83:0/1023648878 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f232815cd30 0x7f232813e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.388+0000 7f2327fff640 1 -- 172.21.15.83:0/1023648878 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f232815cd30 msgr2=0x7f232813e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.388+0000 7f2327fff640 1 --2- 172.21.15.83:0/1023648878 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f232815cd30 0x7f232813e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.388+0000 7f2327fff640 1 -- 172.21.15.83:0/1023648878 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f23100051c0 msgr2=0x7f232813cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.388+0000 7f2327fff640 1 --2- 172.21.15.83:0/1023648878 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f23100051c0 0x7f232813cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.388+0000 7f2327fff640 1 -- 172.21.15.83:0/1023648878 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2328141780 con 0x7f232813e880 2024-03-06T07:25:51.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.388+0000 7f23277fe640 1 --2- 172.21.15.83:0/1023648878 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f23100051c0 0x7f232813cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:51.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.388+0000 7f2327fff640 1 --2- 172.21.15.83:0/1023648878 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f232813e880 0x7f23280cdc30 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f2320001f70 tx=0x7f23200014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:51.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.388+0000 7f230ffff640 1 -- 172.21.15.83:0/1023648878 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2320007db0 con 0x7f232813e880 2024-03-06T07:25:51.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.388+0000 7f230ffff640 1 -- 172.21.15.83:0/1023648878 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2320006d50 con 0x7f232813e880 2024-03-06T07:25:51.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.388+0000 7f232da88640 1 -- 172.21.15.83:0/1023648878 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2328141a60 con 0x7f232813e880 2024-03-06T07:25:51.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.388+0000 7f230ffff640 1 -- 172.21.15.83:0/1023648878 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2320039680 con 0x7f232813e880 2024-03-06T07:25:51.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.388+0000 7f232da88640 1 -- 172.21.15.83:0/1023648878 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2328142020 con 0x7f232813e880 2024-03-06T07:25:51.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.389+0000 7f232da88640 1 -- 172.21.15.83:0/1023648878 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f232804f330 con 0x7f232813e880 2024-03-06T07:25:51.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.389+0000 7f230ffff640 1 -- 172.21.15.83:0/1023648878 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2320039820 con 0x7f232813e880 2024-03-06T07:25:51.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.390+0000 7f230ffff640 1 --2- 172.21.15.83:0/1023648878 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f231406b4d0 0x7f231406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.390+0000 7f230ffff640 1 -- 172.21.15.83:0/1023648878 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f23200abe30 con 0x7f232813e880 2024-03-06T07:25:51.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.390+0000 7f23277fe640 1 --2- 172.21.15.83:0/1023648878 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f231406b4d0 0x7f231406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.390+0000 7f23277fe640 1 --2- 172.21.15.83:0/1023648878 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f231406b4d0 0x7f231406d990 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f23180015d0 tx=0x7f2318009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:51.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.394+0000 7f230ffff640 1 -- 172.21.15.83:0/1023648878 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2320033030 con 0x7f232813e880 2024-03-06T07:25:51.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.636+0000 7f232da88640 1 -- 172.21.15.83:0/1023648878 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f232806eb00 con 0x7f232813e880 2024-03-06T07:25:51.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.636+0000 7f230ffff640 1 -- 172.21.15.83:0/1023648878 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f232007b000 con 0x7f232813e880 2024-03-06T07:25:51.638 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:25:51.638 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:25:51.639 INFO:teuthology.orchestra.run.smithi083.stdout: "auth_dump": [ 2024-03-06T07:25:51.639 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:25:51.639 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mds.a", 2024-03-06T07:25:51.639 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlVHaUGhAAVRpLsHYTUkaYDmceq5Rlgw==", 2024-03-06T07:25:51.639 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:25:51.639 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:25:51.639 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow *", 2024-03-06T07:25:51.639 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow *", 2024-03-06T07:25:51.639 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:25:51.639 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:25:51.639 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:25:51.639 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:25:51.640 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mds.b", 2024-03-06T07:25:51.640 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlFlrrJhAAQbY6xU910eYRg1zwgpj9vg==", 2024-03-06T07:25:51.640 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:25:51.640 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:25:51.640 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow *", 2024-03-06T07:25:51.640 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow *", 2024-03-06T07:25:51.640 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:25:51.640 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:25:51.640 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:25:51.640 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:25:51.640 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mds.c", 2024-03-06T07:25:51.640 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlX0jxIRAAxYNi39qxIFXazlL2qNzLCg==", 2024-03-06T07:25:51.640 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:25:51.641 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:25:51.641 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow *", 2024-03-06T07:25:51.641 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow *", 2024-03-06T07:25:51.641 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:25:51.641 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:25:51.641 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:25:51.641 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:25:51.641 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mds.d", 2024-03-06T07:25:51.641 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehl3utCLhAAzEjASLnYpPYrJsWOt3WwpQ==", 2024-03-06T07:25:51.641 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:25:51.641 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:25:51.641 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow *", 2024-03-06T07:25:51.641 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow *", 2024-03-06T07:25:51.641 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:25:51.642 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:25:51.642 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:25:51.642 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:25:51.642 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.0", 2024-03-06T07:25:51.642 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBgGehlkezBExAAZOLMH2ua3CTWAL7H6LGeZw==", 2024-03-06T07:25:51.642 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:25:51.642 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:51.642 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:25:51.642 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:25:51.642 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:25:51.642 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:25:51.642 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:25:51.642 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.1", 2024-03-06T07:25:51.642 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBjGehlC04qEhAAqirTFR6y3NOGw6N+iz3/pg==", 2024-03-06T07:25:51.643 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:25:51.643 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:51.643 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:25:51.643 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:25:51.643 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:25:51.643 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:25:51.643 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:25:51.643 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.2", 2024-03-06T07:25:51.643 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBmGehlfuFWEBAAj/vjZXrv1wVNSJ1R74UmnQ==", 2024-03-06T07:25:51.643 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:25:51.643 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:51.643 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:25:51.643 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:25:51.643 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:25:51.643 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:25:51.644 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:25:51.644 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.3", 2024-03-06T07:25:51.644 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBpGehl5+ItDhAA0YNc83MhKcgb3JgRuP87aw==", 2024-03-06T07:25:51.644 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:25:51.644 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:51.644 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:25:51.644 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:25:51.644 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:25:51.644 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:25:51.644 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:25:51.644 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.4", 2024-03-06T07:25:51.644 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBvGehlYa/TFBAAvGcizpRQ21pvgpEGgSET0w==", 2024-03-06T07:25:51.644 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:25:51.644 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:51.645 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:25:51.645 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:25:51.645 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:25:51.645 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:25:51.645 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:25:51.645 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.5", 2024-03-06T07:25:51.645 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQByGehlE80TExAAFcWDgnLtW8OKlyu4B0jnmg==", 2024-03-06T07:25:51.645 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:25:51.645 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:51.645 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:25:51.645 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:25:51.645 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:25:51.646 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:25:51.646 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:25:51.646 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.6", 2024-03-06T07:25:51.646 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQB1Gehla/LbERAAYfAjTKL11VMVnGUGV5GVcg==", 2024-03-06T07:25:51.646 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:25:51.646 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:51.646 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:25:51.646 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:25:51.646 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:25:51.646 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:25:51.647 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:25:51.647 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.7", 2024-03-06T07:25:51.647 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQB4GehlpSJkEBAA17tUgMf94jfqvXlD51P8DQ==", 2024-03-06T07:25:51.647 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:25:51.647 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:51.647 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:25:51.647 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:25:51.647 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:25:51.647 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:25:51.647 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:25:51.647 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.0", 2024-03-06T07:25:51.647 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlRwCIMhAA1DdbdQZPplu7LQOTK3BmSg==", 2024-03-06T07:25:51.647 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:25:51.647 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:25:51.647 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow r", 2024-03-06T07:25:51.648 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:25:51.648 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:25:51.648 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:25:51.648 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:25:51.648 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.1", 2024-03-06T07:25:51.648 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBdGehl77spABAAsiEf+tXos3MwFrXToEQ8dQ==", 2024-03-06T07:25:51.648 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:25:51.648 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:25:51.648 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow r", 2024-03-06T07:25:51.648 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow rw", 2024-03-06T07:25:51.648 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow rwx" 2024-03-06T07:25:51.648 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:25:51.649 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:25:51.649 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:25:51.649 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.2", 2024-03-06T07:25:51.649 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehltY5SOBAAI8rcIEVqMZaWQGgNaYKJTA==", 2024-03-06T07:25:51.649 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:25:51.649 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:25:51.649 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow r", 2024-03-06T07:25:51.649 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow rw", 2024-03-06T07:25:51.649 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow rwx" 2024-03-06T07:25:51.649 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:25:51.649 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:25:51.650 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:25:51.650 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.3", 2024-03-06T07:25:51.650 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBdGehl9ZX1BRAAA7fKnRyaQ+zqHoSGs4UvDw==", 2024-03-06T07:25:51.650 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:25:51.650 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:25:51.650 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow r", 2024-03-06T07:25:51.650 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow rw", 2024-03-06T07:25:51.650 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow rwx" 2024-03-06T07:25:51.650 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:25:51.650 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:25:51.650 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:25:51.650 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.admin", 2024-03-06T07:25:51.651 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBbGehl4gqRORAAobqSWoeyHYA/lXkp6/EcIQ==", 2024-03-06T07:25:51.651 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:25:51.651 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow *", 2024-03-06T07:25:51.651 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow *", 2024-03-06T07:25:51.651 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow *", 2024-03-06T07:25:51.651 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:25:51.651 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:25:51.651 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:25:51.651 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:25:51.651 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mgr.x", 2024-03-06T07:25:51.651 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlfikrFxAAoZkTUtpHlBcUAMpwWABRxg==", 2024-03-06T07:25:51.651 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:25:51.651 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow *", 2024-03-06T07:25:51.651 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile mgr", 2024-03-06T07:25:51.651 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:25:51.652 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:25:51.652 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:25:51.652 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:25:51.652 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mgr.y", 2024-03-06T07:25:51.652 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlvdcdExAAySXlEfJqLnpMySiiYGFkMQ==", 2024-03-06T07:25:51.652 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:25:51.652 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow *", 2024-03-06T07:25:51.652 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile mgr", 2024-03-06T07:25:51.652 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:25:51.652 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:25:51.652 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:25:51.652 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-03-06T07:25:51.652 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:25:51.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.638+0000 7f232da88640 1 -- 172.21.15.83:0/1023648878 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f231406b4d0 msgr2=0x7f231406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.638+0000 7f232da88640 1 --2- 172.21.15.83:0/1023648878 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f231406b4d0 0x7f231406d990 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f23180015d0 tx=0x7f2318009040 comp rx=0 tx=0).stop 2024-03-06T07:25:51.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.638+0000 7f232da88640 1 -- 172.21.15.83:0/1023648878 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f232813e880 msgr2=0x7f23280cdc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.638+0000 7f232da88640 1 --2- 172.21.15.83:0/1023648878 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f232813e880 0x7f23280cdc30 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f2320001f70 tx=0x7f23200014a0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.638+0000 7f232da88640 1 -- 172.21.15.83:0/1023648878 shutdown_connections 2024-03-06T07:25:51.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.638+0000 7f232da88640 1 --2- 172.21.15.83:0/1023648878 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f231406b4d0 0x7f231406d990 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.638+0000 7f232da88640 1 --2- 172.21.15.83:0/1023648878 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f232813e880 0x7f23280cdc30 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.638+0000 7f232da88640 1 --2- 172.21.15.83:0/1023648878 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f232815cd30 0x7f232813e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.638+0000 7f232da88640 1 --2- 172.21.15.83:0/1023648878 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f23100051c0 0x7f232813cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.638+0000 7f232da88640 1 -- 172.21.15.83:0/1023648878 >> 172.21.15.83:0/1023648878 conn(0x7f23280cf8a0 msgr2=0x7f2328159880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:51.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.638+0000 7f232da88640 1 -- 172.21.15.83:0/1023648878 shutdown_connections 2024-03-06T07:25:51.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.638+0000 7f232da88640 1 -- 172.21.15.83:0/1023648878 wait complete. 2024-03-06T07:25:51.656 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:51.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.754+0000 7fa051edf640 1 Processor -- start 2024-03-06T07:25:51.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.754+0000 7fa051edf640 1 -- start start 2024-03-06T07:25:51.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.755+0000 7fa051edf640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa04c15bf30 0x7fa04c15ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.755+0000 7fa051edf640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa04c15f580 con 0x7fa04c0b3060 2024-03-06T07:25:51.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.755+0000 7fa051edf640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa04c15b450 con 0x7fa04c13dc10 2024-03-06T07:25:51.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.755+0000 7fa051edf640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa04c0cf3f0 con 0x7fa04c15bf30 2024-03-06T07:25:51.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.755+0000 7fa04affd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fa04c0b3060 0x7fa04c15e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39500/0 (socket says 172.21.15.83:39500) 2024-03-06T07:25:51.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.755+0000 7fa04bfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa04c15bf30 0x7fa04c15ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.755+0000 7fa04affd640 1 -- 172.21.15.83:0/1101651995 learned_addr learned my addr 172.21.15.83:0/1101651995 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:25:51.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.755+0000 7fa04bfff640 1 -- 172.21.15.83:0/1101651995 >> v1:172.21.15.183:6789/0 conn(0x7fa04c13dc10 legacy=0x7fa04c0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.755+0000 7fa04bfff640 1 -- 172.21.15.83:0/1101651995 >> v1:172.21.15.83:6789/0 conn(0x7fa04c0b3060 legacy=0x7fa04c15e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.755+0000 7fa04bfff640 1 -- 172.21.15.83:0/1101651995 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa04c15b730 con 0x7fa04c15bf30 2024-03-06T07:25:51.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.756+0000 7fa04bfff640 1 --2- 172.21.15.83:0/1101651995 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa04c15bf30 0x7fa04c15ef60 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fa044009d90 tx=0x7fa04402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6e5f594cb27a8fe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:51.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.756+0000 7fa04a7fc640 1 -- 172.21.15.83:0/1101651995 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa044039070 con 0x7fa04c15bf30 2024-03-06T07:25:51.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.756+0000 7fa04a7fc640 1 -- 172.21.15.83:0/1101651995 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa04402cc60 con 0x7fa04c15bf30 2024-03-06T07:25:51.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.756+0000 7fa04a7fc640 1 -- 172.21.15.83:0/1101651995 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa044034660 con 0x7fa04c15bf30 2024-03-06T07:25:51.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.757+0000 7fa051edf640 1 -- 172.21.15.83:0/1101651995 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa04c15bf30 msgr2=0x7fa04c15ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.757+0000 7fa051edf640 1 --2- 172.21.15.83:0/1101651995 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa04c15bf30 0x7fa04c15ef60 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fa044009d90 tx=0x7fa04402beb0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.757+0000 7fa051edf640 1 -- 172.21.15.83:0/1101651995 shutdown_connections 2024-03-06T07:25:51.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.757+0000 7fa051edf640 1 --2- 172.21.15.83:0/1101651995 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa04c15bf30 0x7fa04c15ef60 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.757+0000 7fa051edf640 1 -- 172.21.15.83:0/1101651995 >> 172.21.15.83:0/1101651995 conn(0x7fa04c0cf8a0 msgr2=0x7fa04c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:51.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.757+0000 7fa051edf640 1 -- 172.21.15.83:0/1101651995 shutdown_connections 2024-03-06T07:25:51.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.757+0000 7fa051edf640 1 -- 172.21.15.83:0/1101651995 wait complete. 2024-03-06T07:25:51.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.758+0000 7fa051edf640 1 Processor -- start 2024-03-06T07:25:51.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.758+0000 7fa051edf640 1 -- start start 2024-03-06T07:25:51.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.759+0000 7fa051edf640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa04c0b3060 0x7fa04c0ce080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.759+0000 7fa051edf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa04c13dc10 0x7fa04c15d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.759+0000 7fa051edf640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa04c15bf30 0x7fa04c15d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.759+0000 7fa051edf640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa04c0718e0 con 0x7fa04c13dc10 2024-03-06T07:25:51.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.759+0000 7fa051edf640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa04c15b210 con 0x7fa04c0b3060 2024-03-06T07:25:51.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.759+0000 7fa051edf640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa04c071a60 con 0x7fa04c15bf30 2024-03-06T07:25:51.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.759+0000 7fa04affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa04c13dc10 0x7fa04c15d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.759+0000 7fa04affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa04c13dc10 0x7fa04c15d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37734/0 (socket says 172.21.15.83:37734) 2024-03-06T07:25:51.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.759+0000 7fa04affd640 1 -- 172.21.15.83:0/4239661655 learned_addr learned my addr 172.21.15.83:0/4239661655 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:51.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.759+0000 7fa04bfff640 1 --2- 172.21.15.83:0/4239661655 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa04c15bf30 0x7fa04c15d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.759+0000 7fa04b7fe640 1 --2- 172.21.15.83:0/4239661655 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa04c0b3060 0x7fa04c0ce080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.759+0000 7fa04affd640 1 -- 172.21.15.83:0/4239661655 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa04c15bf30 msgr2=0x7fa04c15d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.759+0000 7fa04affd640 1 --2- 172.21.15.83:0/4239661655 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa04c15bf30 0x7fa04c15d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.759+0000 7fa04affd640 1 -- 172.21.15.83:0/4239661655 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa04c0b3060 msgr2=0x7fa04c0ce080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.759+0000 7fa04affd640 1 --2- 172.21.15.83:0/4239661655 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa04c0b3060 0x7fa04c0ce080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.759+0000 7fa04affd640 1 -- 172.21.15.83:0/4239661655 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa04c150cf0 con 0x7fa04c13dc10 2024-03-06T07:25:51.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.759+0000 7fa04bfff640 1 --2- 172.21.15.83:0/4239661655 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa04c15bf30 0x7fa04c15d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:51.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.759+0000 7fa04b7fe640 1 --2- 172.21.15.83:0/4239661655 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa04c0b3060 0x7fa04c0ce080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:51.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.759+0000 7fa04affd640 1 --2- 172.21.15.83:0/4239661655 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa04c13dc10 0x7fa04c15d390 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fa04000c4a0 tx=0x7fa04000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:51.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.759+0000 7fa033fff640 1 -- 172.21.15.83:0/4239661655 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa040017070 con 0x7fa04c13dc10 2024-03-06T07:25:51.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.759+0000 7fa033fff640 1 -- 172.21.15.83:0/4239661655 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa040005ce0 con 0x7fa04c13dc10 2024-03-06T07:25:51.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.760+0000 7fa051edf640 1 -- 172.21.15.83:0/4239661655 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa04c15a7e0 con 0x7fa04c13dc10 2024-03-06T07:25:51.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.760+0000 7fa033fff640 1 -- 172.21.15.83:0/4239661655 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa040007270 con 0x7fa04c13dc10 2024-03-06T07:25:51.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.760+0000 7fa051edf640 1 -- 172.21.15.83:0/4239661655 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa04c13e870 con 0x7fa04c13dc10 2024-03-06T07:25:51.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.760+0000 7fa051edf640 1 -- 172.21.15.83:0/4239661655 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa04c15ac90 con 0x7fa04c13dc10 2024-03-06T07:25:51.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.761+0000 7fa033fff640 1 -- 172.21.15.83:0/4239661655 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa04001a070 con 0x7fa04c13dc10 2024-03-06T07:25:51.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.761+0000 7fa033fff640 1 --2- 172.21.15.83:0/4239661655 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa03806b500 0x7fa03806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.761+0000 7fa033fff640 1 -- 172.21.15.83:0/4239661655 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa040089080 con 0x7fa04c13dc10 2024-03-06T07:25:51.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.761+0000 7fa04b7fe640 1 --2- 172.21.15.83:0/4239661655 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa03806b500 0x7fa03806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.762+0000 7fa04b7fe640 1 --2- 172.21.15.83:0/4239661655 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa03806b500 0x7fa03806d9c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fa04c1520c0 tx=0x7fa03c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:51.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.766+0000 7fa033fff640 1 -- 172.21.15.83:0/4239661655 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa040057c90 con 0x7fa04c13dc10 2024-03-06T07:25:51.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.937+0000 7fa051edf640 1 -- 172.21.15.83:0/4239661655 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa04c04f330 con 0x7fa04c13dc10 2024-03-06T07:25:51.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.938+0000 7fa033fff640 1 -- 172.21.15.83:0/4239661655 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+12904 (secure 0 0 0) 0x7fa04005bd30 con 0x7fa04c13dc10 2024-03-06T07:25:51.940 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:25:51.940 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":78,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:25:51.268672+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:51.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.940+0000 7fa051edf640 1 -- 172.21.15.83:0/4239661655 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa03806b500 msgr2=0x7fa03806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.940+0000 7fa051edf640 1 --2- 172.21.15.83:0/4239661655 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa03806b500 0x7fa03806d9c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fa04c1520c0 tx=0x7fa03c009040 comp rx=0 tx=0).stop 2024-03-06T07:25:51.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.940+0000 7fa051edf640 1 -- 172.21.15.83:0/4239661655 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa04c13dc10 msgr2=0x7fa04c15d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.940+0000 7fa051edf640 1 --2- 172.21.15.83:0/4239661655 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa04c13dc10 0x7fa04c15d390 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fa04000c4a0 tx=0x7fa04000c960 comp rx=0 tx=0).stop 2024-03-06T07:25:51.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.940+0000 7fa051edf640 1 -- 172.21.15.83:0/4239661655 shutdown_connections 2024-03-06T07:25:51.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.940+0000 7fa051edf640 1 --2- 172.21.15.83:0/4239661655 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa03806b500 0x7fa03806d9c0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.940+0000 7fa051edf640 1 --2- 172.21.15.83:0/4239661655 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa04c15bf30 0x7fa04c15d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.940+0000 7fa051edf640 1 --2- 172.21.15.83:0/4239661655 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa04c13dc10 0x7fa04c15d390 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.940+0000 7fa051edf640 1 --2- 172.21.15.83:0/4239661655 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa04c0b3060 0x7fa04c0ce080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.940+0000 7fa051edf640 1 -- 172.21.15.83:0/4239661655 >> 172.21.15.83:0/4239661655 conn(0x7fa04c0cf8a0 msgr2=0x7fa04c06afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:51.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.940+0000 7fa051edf640 1 -- 172.21.15.83:0/4239661655 shutdown_connections 2024-03-06T07:25:51.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:51.940+0000 7fa051edf640 1 -- 172.21.15.83:0/4239661655 wait complete. 2024-03-06T07:25:51.953 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:25:51.953 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:25:52.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.054+0000 7fd9f797a640 1 Processor -- start 2024-03-06T07:25:52.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.054+0000 7fd9f797a640 1 -- start start 2024-03-06T07:25:52.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.055+0000 7fd9f797a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd9f015fc30 0x7fd9f00b47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.055+0000 7fd9f797a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9f00b4d10 0x7fd9f00cf200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.055+0000 7fd9f797a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9f01600a0 con 0x7fd9f00d06e0 2024-03-06T07:25:52.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.055+0000 7fd9f797a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9f00ced70 con 0x7fd9f015fc30 2024-03-06T07:25:52.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.055+0000 7fd9f797a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f00d47f0 con 0x7fd9f00b4d10 2024-03-06T07:25:52.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.055+0000 7fd9f4f15640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd9f015fc30 0x7fd9f00b47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.055+0000 7fd9f5716640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fd9f00d06e0 0x7fd9f00d0ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39510/0 (socket says 172.21.15.83:39510) 2024-03-06T07:25:52.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.055+0000 7fd9f4f15640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd9f015fc30 0x7fd9f00b47d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37740/0 (socket says 172.21.15.83:37740) 2024-03-06T07:25:52.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.055+0000 7fd9f5716640 1 -- 172.21.15.83:0/311311336 learned_addr learned my addr 172.21.15.83:0/311311336 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:25:52.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.056+0000 7fd9f5f17640 1 --2- 172.21.15.83:0/311311336 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9f00b4d10 0x7fd9f00cf200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.056+0000 7fd9f4f15640 1 -- 172.21.15.83:0/311311336 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9f00b4d10 msgr2=0x7fd9f00cf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:52.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.056+0000 7fd9f4f15640 1 --2- 172.21.15.83:0/311311336 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9f00b4d10 0x7fd9f00cf200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.056+0000 7fd9f4f15640 1 -- 172.21.15.83:0/311311336 >> v1:172.21.15.83:6789/0 conn(0x7fd9f00d06e0 legacy=0x7fd9f00d0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.056+0000 7fd9e7fff640 1 -- 172.21.15.83:0/311311336 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3782311393 0 0) 0x7fd9f01600a0 con 0x7fd9f00d06e0 2024-03-06T07:25:52.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.056+0000 7fd9f4f15640 1 -- 172.21.15.83:0/311311336 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9f00d1ab0 con 0x7fd9f015fc30 2024-03-06T07:25:52.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.056+0000 7fd9f4f15640 1 --2- 172.21.15.83:0/311311336 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9f015fc30 0x7fd9f00b47d0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fd9e00097a0 tx=0x7fd9e002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daa892d177df15d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.057+0000 7fd9e7fff640 1 -- 172.21.15.83:0/311311336 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e0035020 con 0x7fd9f015fc30 2024-03-06T07:25:52.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.057+0000 7fd9e7fff640 1 -- 172.21.15.83:0/311311336 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9e0031830 con 0x7fd9f015fc30 2024-03-06T07:25:52.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.057+0000 7fd9e7fff640 1 -- 172.21.15.83:0/311311336 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e0031c00 con 0x7fd9f015fc30 2024-03-06T07:25:52.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.057+0000 7fd9f797a640 1 -- 172.21.15.83:0/311311336 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9f015fc30 msgr2=0x7fd9f00b47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.057+0000 7fd9f797a640 1 --2- 172.21.15.83:0/311311336 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9f015fc30 0x7fd9f00b47d0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fd9e00097a0 tx=0x7fd9e002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.057+0000 7fd9f797a640 1 -- 172.21.15.83:0/311311336 shutdown_connections 2024-03-06T07:25:52.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.057+0000 7fd9f797a640 1 --2- 172.21.15.83:0/311311336 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9f00b4d10 0x7fd9f00cf200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.057+0000 7fd9f797a640 1 --2- 172.21.15.83:0/311311336 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9f015fc30 0x7fd9f00b47d0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.057+0000 7fd9f797a640 1 -- 172.21.15.83:0/311311336 >> 172.21.15.83:0/311311336 conn(0x7fd9f00d3cd0 msgr2=0x7fd9f00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:52.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.057+0000 7fd9f797a640 1 -- 172.21.15.83:0/311311336 shutdown_connections 2024-03-06T07:25:52.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.057+0000 7fd9f797a640 1 -- 172.21.15.83:0/311311336 wait complete. 2024-03-06T07:25:52.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.058+0000 7fd9f797a640 1 Processor -- start 2024-03-06T07:25:52.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.058+0000 7fd9f797a640 1 -- start start 2024-03-06T07:25:52.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.059+0000 7fd9f797a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9f00b4d10 0x7fd9f015b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.059+0000 7fd9f797a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9f00d06e0 0x7fd9f01e9c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.059+0000 7fd9f797a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9f015fc30 0x7fd9f01ea1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.059+0000 7fd9f797a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f00ce990 con 0x7fd9f00b4d10 2024-03-06T07:25:52.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.059+0000 7fd9f797a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f00d4ab0 con 0x7fd9f00d06e0 2024-03-06T07:25:52.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.059+0000 7fd9f797a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f00b44d0 con 0x7fd9f015fc30 2024-03-06T07:25:52.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.059+0000 7fd9f5716640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9f00b4d10 0x7fd9f015b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.059+0000 7fd9f5716640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9f00b4d10 0x7fd9f015b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37756/0 (socket says 172.21.15.83:37756) 2024-03-06T07:25:52.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.059+0000 7fd9f5716640 1 -- 172.21.15.83:0/1257654311 learned_addr learned my addr 172.21.15.83:0/1257654311 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:52.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.059+0000 7fd9f5f17640 1 --2- 172.21.15.83:0/1257654311 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9f015fc30 0x7fd9f01ea1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.060+0000 7fd9f5716640 1 -- 172.21.15.83:0/1257654311 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9f015fc30 msgr2=0x7fd9f01ea1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.060+0000 7fd9f5716640 1 --2- 172.21.15.83:0/1257654311 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9f015fc30 0x7fd9f01ea1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.060+0000 7fd9f5716640 1 -- 172.21.15.83:0/1257654311 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9f00d06e0 msgr2=0x7fd9f01e9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.060+0000 7fd9f5716640 1 --2- 172.21.15.83:0/1257654311 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9f00d06e0 0x7fd9f01e9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.060+0000 7fd9f5716640 1 -- 172.21.15.83:0/1257654311 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9f00cd260 con 0x7fd9f00b4d10 2024-03-06T07:25:52.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.060+0000 7fd9f5716640 1 --2- 172.21.15.83:0/1257654311 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9f00b4d10 0x7fd9f015b370 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec00c7f0 tx=0x7fd9ec00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.060+0000 7fd9e5ffb640 1 -- 172.21.15.83:0/1257654311 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9ec018070 con 0x7fd9f00b4d10 2024-03-06T07:25:52.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.060+0000 7fd9e5ffb640 1 -- 172.21.15.83:0/1257654311 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9ec00a640 con 0x7fd9f00b4d10 2024-03-06T07:25:52.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.060+0000 7fd9f797a640 1 -- 172.21.15.83:0/1257654311 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9f00cd500 con 0x7fd9f00b4d10 2024-03-06T07:25:52.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.060+0000 7fd9e5ffb640 1 -- 172.21.15.83:0/1257654311 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9ec00aa30 con 0x7fd9f00b4d10 2024-03-06T07:25:52.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.060+0000 7fd9f797a640 1 -- 172.21.15.83:0/1257654311 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9f00cdac0 con 0x7fd9f00b4d10 2024-03-06T07:25:52.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.061+0000 7fd9f797a640 1 -- 172.21.15.83:0/1257654311 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9f00a1c80 con 0x7fd9f00b4d10 2024-03-06T07:25:52.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.061+0000 7fd9e5ffb640 1 -- 172.21.15.83:0/1257654311 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd9ec01b070 con 0x7fd9f00b4d10 2024-03-06T07:25:52.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.062+0000 7fd9e5ffb640 1 --2- 172.21.15.83:0/1257654311 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd9d406b4d0 0x7fd9d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.062+0000 7fd9e5ffb640 1 -- 172.21.15.83:0/1257654311 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd9ec089410 con 0x7fd9f00b4d10 2024-03-06T07:25:52.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.062+0000 7fd9f4f15640 1 --2- 172.21.15.83:0/1257654311 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd9d406b4d0 0x7fd9d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.062+0000 7fd9f4f15640 1 --2- 172.21.15.83:0/1257654311 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd9d406b4d0 0x7fd9d406d990 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fd9e0036000 tx=0x7fd9e0030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.066+0000 7fd9e5ffb640 1 -- 172.21.15.83:0/1257654311 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd9ec058670 con 0x7fd9f00b4d10 2024-03-06T07:25:52.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.238+0000 7fd9f797a640 1 -- 172.21.15.83:0/1257654311 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd9f00b5270 con 0x7fd9f00b4d10 2024-03-06T07:25:52.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.239+0000 7fd9e5ffb640 1 -- 172.21.15.83:0/1257654311 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd9ec057f70 con 0x7fd9f00b4d10 2024-03-06T07:25:52.241 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:25:52.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.240+0000 7fd9f797a640 1 -- 172.21.15.83:0/1257654311 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd9d406b4d0 msgr2=0x7fd9d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.240+0000 7fd9f797a640 1 --2- 172.21.15.83:0/1257654311 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd9d406b4d0 0x7fd9d406d990 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fd9e0036000 tx=0x7fd9e0030000 comp rx=0 tx=0).stop 2024-03-06T07:25:52.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.240+0000 7fd9f797a640 1 -- 172.21.15.83:0/1257654311 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9f00b4d10 msgr2=0x7fd9f015b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.240+0000 7fd9f797a640 1 --2- 172.21.15.83:0/1257654311 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9f00b4d10 0x7fd9f015b370 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec00c7f0 tx=0x7fd9ec00ccb0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.241+0000 7fd9f797a640 1 -- 172.21.15.83:0/1257654311 shutdown_connections 2024-03-06T07:25:52.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.241+0000 7fd9f797a640 1 --2- 172.21.15.83:0/1257654311 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd9d406b4d0 0x7fd9d406d990 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.241+0000 7fd9f797a640 1 --2- 172.21.15.83:0/1257654311 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9f015fc30 0x7fd9f01ea1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.241+0000 7fd9f797a640 1 --2- 172.21.15.83:0/1257654311 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9f00d06e0 0x7fd9f01e9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.241+0000 7fd9f797a640 1 --2- 172.21.15.83:0/1257654311 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9f00b4d10 0x7fd9f015b370 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.241+0000 7fd9f797a640 1 -- 172.21.15.83:0/1257654311 >> 172.21.15.83:0/1257654311 conn(0x7fd9f00d3cd0 msgr2=0x7fd9f00b5660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:52.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.241+0000 7fd9f797a640 1 -- 172.21.15.83:0/1257654311 shutdown_connections 2024-03-06T07:25:52.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.241+0000 7fd9f797a640 1 -- 172.21.15.83:0/1257654311 wait complete. 2024-03-06T07:25:52.253 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:25:52.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.351+0000 7f36a33e4640 1 Processor -- start 2024-03-06T07:25:52.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.351+0000 7f36a33e4640 1 -- start start 2024-03-06T07:25:52.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.352+0000 7f36a33e4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f369c0b3060 0x7f369c15e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.352+0000 7f36a33e4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f369c15f580 con 0x7f369c13dc10 2024-03-06T07:25:52.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.352+0000 7f36a33e4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f369c15b450 con 0x7f369c15bf30 2024-03-06T07:25:52.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.352+0000 7f36a33e4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f369c0cf3f0 con 0x7f369c0b3060 2024-03-06T07:25:52.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.352+0000 7f36a1180640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f369c13dc10 0x7f369c0b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39514/0 (socket says 172.21.15.83:39514) 2024-03-06T07:25:52.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.352+0000 7f36a1180640 1 -- 172.21.15.83:0/401386104 learned_addr learned my addr 172.21.15.83:0/401386104 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:25:52.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.352+0000 7f36a097f640 1 --2- 172.21.15.83:0/401386104 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f369c0b3060 0x7f369c15e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.352+0000 7f3693fff640 1 -- 172.21.15.83:0/401386104 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2098735170 0 0) 0x7f369c15f580 con 0x7f369c13dc10 2024-03-06T07:25:52.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.352+0000 7f3693fff640 1 -- 172.21.15.83:0/401386104 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3688003520 con 0x7f369c13dc10 2024-03-06T07:25:52.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.353+0000 7f3693fff640 1 -- 172.21.15.83:0/401386104 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4229294061 0 0) 0x7f369c15b450 con 0x7f369c15bf30 2024-03-06T07:25:52.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.353+0000 7f36a097f640 1 -- 172.21.15.83:0/401386104 >> v1:172.21.15.183:6789/0 conn(0x7f369c15bf30 legacy=0x7f369c15ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.353+0000 7f36a097f640 1 -- 172.21.15.83:0/401386104 >> v1:172.21.15.83:6789/0 conn(0x7f369c13dc10 legacy=0x7f369c0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.353+0000 7f36a097f640 1 -- 172.21.15.83:0/401386104 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f369c15b730 con 0x7f369c0b3060 2024-03-06T07:25:52.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.353+0000 7f36a097f640 1 --2- 172.21.15.83:0/401386104 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f369c0b3060 0x7f369c15e850 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f368c0097d0 tx=0x7f368c02b690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43a77c1242ea6c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.354+0000 7f3693fff640 1 -- 172.21.15.83:0/401386104 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f368c037070 con 0x7f369c0b3060 2024-03-06T07:25:52.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.354+0000 7f3693fff640 1 -- 172.21.15.83:0/401386104 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f368c030510 con 0x7f369c0b3060 2024-03-06T07:25:52.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.354+0000 7f3693fff640 1 -- 172.21.15.83:0/401386104 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36980019f0 con 0x7f369c0b3060 2024-03-06T07:25:52.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.354+0000 7f36a33e4640 1 -- 172.21.15.83:0/401386104 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f369c0b3060 msgr2=0x7f369c15e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.354+0000 7f36a33e4640 1 --2- 172.21.15.83:0/401386104 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f369c0b3060 0x7f369c15e850 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f368c0097d0 tx=0x7f368c02b690 comp rx=0 tx=0).stop 2024-03-06T07:25:52.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.354+0000 7f36a33e4640 1 -- 172.21.15.83:0/401386104 shutdown_connections 2024-03-06T07:25:52.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.354+0000 7f36a33e4640 1 --2- 172.21.15.83:0/401386104 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f369c0b3060 0x7f369c15e850 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.354+0000 7f36a33e4640 1 -- 172.21.15.83:0/401386104 >> 172.21.15.83:0/401386104 conn(0x7f369c0cf8a0 msgr2=0x7f369c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:52.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.354+0000 7f36a33e4640 1 -- 172.21.15.83:0/401386104 shutdown_connections 2024-03-06T07:25:52.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.354+0000 7f36a33e4640 1 -- 172.21.15.83:0/401386104 wait complete. 2024-03-06T07:25:52.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.355+0000 7f36a33e4640 1 Processor -- start 2024-03-06T07:25:52.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.355+0000 7f36a33e4640 1 -- start start 2024-03-06T07:25:52.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.356+0000 7f36a33e4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f369c0b3060 0x7f369c13c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.356+0000 7f36a33e4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f369c13dc10 0x7f369c1ef680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.356+0000 7f36a33e4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f369c15bf30 0x7f369c142370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.356+0000 7f36a33e4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f369c0718e0 con 0x7f369c15bf30 2024-03-06T07:25:52.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.356+0000 7f36a33e4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f369c15b210 con 0x7f369c0b3060 2024-03-06T07:25:52.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.356+0000 7f36a33e4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f369c071a60 con 0x7f369c13dc10 2024-03-06T07:25:52.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.356+0000 7f36a1981640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f369c15bf30 0x7f369c142370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.356+0000 7f36a1981640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f369c15bf30 0x7f369c142370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37762/0 (socket says 172.21.15.83:37762) 2024-03-06T07:25:52.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.356+0000 7f36a1981640 1 -- 172.21.15.83:0/1941104420 learned_addr learned my addr 172.21.15.83:0/1941104420 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:52.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.356+0000 7f36a097f640 1 --2- 172.21.15.83:0/1941104420 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f369c13dc10 0x7f369c1ef680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.356+0000 7f36a1180640 1 --2- 172.21.15.83:0/1941104420 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f369c0b3060 0x7f369c13c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.356+0000 7f36a1981640 1 -- 172.21.15.83:0/1941104420 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f369c13dc10 msgr2=0x7f369c1ef680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.356+0000 7f36a1981640 1 --2- 172.21.15.83:0/1941104420 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f369c13dc10 0x7f369c1ef680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.356+0000 7f36a1981640 1 -- 172.21.15.83:0/1941104420 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f369c0b3060 msgr2=0x7f369c13c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.356+0000 7f36a1981640 1 --2- 172.21.15.83:0/1941104420 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f369c0b3060 0x7f369c13c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.356+0000 7f36a1981640 1 -- 172.21.15.83:0/1941104420 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f369c13d650 con 0x7f369c15bf30 2024-03-06T07:25:52.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.356+0000 7f36a1180640 1 --2- 172.21.15.83:0/1941104420 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f369c0b3060 0x7f369c13c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:52.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.357+0000 7f36a1981640 1 --2- 172.21.15.83:0/1941104420 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f369c15bf30 0x7f369c142370 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f369400c3e0 tx=0x7f369400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.357+0000 7f3691ffb640 1 -- 172.21.15.83:0/1941104420 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3694017070 con 0x7f369c15bf30 2024-03-06T07:25:52.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.357+0000 7f3691ffb640 1 -- 172.21.15.83:0/1941104420 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36940035f0 con 0x7f369c15bf30 2024-03-06T07:25:52.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.357+0000 7f36a33e4640 1 -- 172.21.15.83:0/1941104420 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f369c13cec0 con 0x7f369c15bf30 2024-03-06T07:25:52.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.357+0000 7f3691ffb640 1 -- 172.21.15.83:0/1941104420 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36940039e0 con 0x7f369c15bf30 2024-03-06T07:25:52.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.357+0000 7f36a33e4640 1 -- 172.21.15.83:0/1941104420 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f369c1432b0 con 0x7f369c15bf30 2024-03-06T07:25:52.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.358+0000 7f36a33e4640 1 -- 172.21.15.83:0/1941104420 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f369c04f330 con 0x7f369c15bf30 2024-03-06T07:25:52.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.358+0000 7f3691ffb640 1 -- 172.21.15.83:0/1941104420 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f369401a070 con 0x7f369c15bf30 2024-03-06T07:25:52.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.358+0000 7f3691ffb640 1 --2- 172.21.15.83:0/1941104420 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f368006b4d0 0x7f368006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.359+0000 7f3691ffb640 1 -- 172.21.15.83:0/1941104420 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3694089240 con 0x7f369c15bf30 2024-03-06T07:25:52.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.359+0000 7f36a1180640 1 --2- 172.21.15.83:0/1941104420 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f368006b4d0 0x7f368006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.359+0000 7f36a1180640 1 --2- 172.21.15.83:0/1941104420 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f368006b4d0 0x7f368006d990 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f3698001a80 tx=0x7f3698007cb0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.363+0000 7f3691ffb640 1 -- 172.21.15.83:0/1941104420 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3694057d40 con 0x7f369c15bf30 2024-03-06T07:25:52.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.538+0000 7f36a33e4640 1 -- 172.21.15.83:0/1941104420 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f369c06eb00 con 0x7f369c15bf30 2024-03-06T07:25:52.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.538+0000 7f3691ffb640 1 -- 172.21.15.83:0/1941104420 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v78) v1 ==== 84+0+65 (secure 0 0 0) 0x7f369405bde0 con 0x7f369c15bf30 2024-03-06T07:25:52.541 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:25:52.541 INFO:teuthology.orchestra.run.smithi083.stdout:[ 2024-03-06T07:25:52.541 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:25:52.541 INFO:teuthology.orchestra.run.smithi083.stdout: "poolnum": 1, 2024-03-06T07:25:52.541 INFO:teuthology.orchestra.run.smithi083.stdout: "poolname": ".mgr" 2024-03-06T07:25:52.541 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:25:52.541 INFO:teuthology.orchestra.run.smithi083.stdout:] 2024-03-06T07:25:52.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.540+0000 7f36a33e4640 1 -- 172.21.15.83:0/1941104420 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f368006b4d0 msgr2=0x7f368006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.540+0000 7f36a33e4640 1 --2- 172.21.15.83:0/1941104420 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f368006b4d0 0x7f368006d990 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f3698001a80 tx=0x7f3698007cb0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.540+0000 7f36a33e4640 1 -- 172.21.15.83:0/1941104420 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f369c15bf30 msgr2=0x7f369c142370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.540+0000 7f36a33e4640 1 --2- 172.21.15.83:0/1941104420 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f369c15bf30 0x7f369c142370 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f369400c3e0 tx=0x7f369400c8a0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.540+0000 7f36a33e4640 1 -- 172.21.15.83:0/1941104420 shutdown_connections 2024-03-06T07:25:52.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.540+0000 7f36a33e4640 1 --2- 172.21.15.83:0/1941104420 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f368006b4d0 0x7f368006d990 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.540+0000 7f36a33e4640 1 --2- 172.21.15.83:0/1941104420 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f369c15bf30 0x7f369c142370 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.540+0000 7f36a33e4640 1 --2- 172.21.15.83:0/1941104420 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f369c13dc10 0x7f369c1ef680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.540+0000 7f36a33e4640 1 --2- 172.21.15.83:0/1941104420 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f369c0b3060 0x7f369c13c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.540+0000 7f36a33e4640 1 -- 172.21.15.83:0/1941104420 >> 172.21.15.83:0/1941104420 conn(0x7f369c0cf8a0 msgr2=0x7f369c0cdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:52.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.541+0000 7f36a33e4640 1 -- 172.21.15.83:0/1941104420 shutdown_connections 2024-03-06T07:25:52.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.541+0000 7f36a33e4640 1 -- 172.21.15.83:0/1941104420 wait complete. 2024-03-06T07:25:52.553 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:25:52.553 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:25:52.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.655+0000 7f54ee9f6640 1 Processor -- start 2024-03-06T07:25:52.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.655+0000 7f54ee9f6640 1 -- start start 2024-03-06T07:25:52.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.656+0000 7f54ee9f6640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f54e815df60 0x7f54e80b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.656+0000 7f54ee9f6640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f54e806cde0 0x7f54e80ae340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.656+0000 7f54ee9f6640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f54e80ae880 0x7f54e80aed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.656+0000 7f54ee9f6640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54e815e340 con 0x7f54e80ae880 2024-03-06T07:25:52.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.656+0000 7f54ee9f6640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54e80b3570 con 0x7f54e815df60 2024-03-06T07:25:52.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.656+0000 7f54ee9f6640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54e80b3850 con 0x7f54e806cde0 2024-03-06T07:25:52.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.656+0000 7f54ecf93640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f54e80ae880 0x7f54e80aed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.656+0000 7f54ecf93640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f54e80ae880 0x7f54e80aed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37764/0 (socket says 172.21.15.83:37764) 2024-03-06T07:25:52.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.656+0000 7f54ecf93640 1 -- 172.21.15.83:0/3193739601 learned_addr learned my addr 172.21.15.83:0/3193739601 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:52.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.656+0000 7f54ecf93640 1 -- 172.21.15.83:0/3193739601 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f54e806cde0 msgr2=0x7f54e80ae340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:52.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.656+0000 7f54e77fe640 1 --2- 172.21.15.83:0/3193739601 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f54e806cde0 0x7f54e80ae340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.656+0000 7f54e7fff640 1 --2- 172.21.15.83:0/3193739601 >> v2:172.21.15.183:3300/0 conn(0x7f54e815df60 0x7f54e80b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.656+0000 7f54ecf93640 1 --2- 172.21.15.83:0/3193739601 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f54e806cde0 0x7f54e80ae340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.656+0000 7f54ecf93640 1 -- 172.21.15.83:0/3193739601 >> v2:172.21.15.183:3300/0 conn(0x7f54e815df60 msgr2=0x7f54e80b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:52.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.656+0000 7f54ecf93640 1 --2- 172.21.15.83:0/3193739601 >> v2:172.21.15.183:3300/0 conn(0x7f54e815df60 0x7f54e80b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.656+0000 7f54ecf93640 1 -- 172.21.15.83:0/3193739601 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e80ad860 con 0x7f54e80ae880 2024-03-06T07:25:52.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.656+0000 7f54e77fe640 1 --2- 172.21.15.83:0/3193739601 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f54e806cde0 0x7f54e80ae340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:52.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.656+0000 7f54e7fff640 1 --2- 172.21.15.83:0/3193739601 >> v2:172.21.15.183:3300/0 conn(0x7f54e815df60 0x7f54e80b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:52.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.657+0000 7f54ecf93640 1 --2- 172.21.15.83:0/3193739601 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54e80ae880 0x7f54e80aed30 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f54e806bf40 tx=0x7f54e002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74631f385da5400 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.657+0000 7f54e6ffd640 1 -- 172.21.15.83:0/3193739601 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e002ca70 con 0x7f54e80ae880 2024-03-06T07:25:52.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.657+0000 7f54e6ffd640 1 -- 172.21.15.83:0/3193739601 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54e002cc10 con 0x7f54e80ae880 2024-03-06T07:25:52.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.657+0000 7f54e6ffd640 1 -- 172.21.15.83:0/3193739601 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e00343f0 con 0x7f54e80ae880 2024-03-06T07:25:52.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.657+0000 7f54ee9f6640 1 -- 172.21.15.83:0/3193739601 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54e80ae880 msgr2=0x7f54e80aed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.657+0000 7f54ee9f6640 1 --2- 172.21.15.83:0/3193739601 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54e80ae880 0x7f54e80aed30 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f54e806bf40 tx=0x7f54e002bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.658+0000 7f54ee9f6640 1 -- 172.21.15.83:0/3193739601 shutdown_connections 2024-03-06T07:25:52.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.658+0000 7f54ee9f6640 1 --2- 172.21.15.83:0/3193739601 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54e80ae880 0x7f54e80aed30 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.658+0000 7f54ee9f6640 1 --2- 172.21.15.83:0/3193739601 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f54e806cde0 0x7f54e80ae340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.658+0000 7f54ee9f6640 1 --2- 172.21.15.83:0/3193739601 >> v2:172.21.15.183:3300/0 conn(0x7f54e815df60 0x7f54e80b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.658+0000 7f54ee9f6640 1 -- 172.21.15.83:0/3193739601 >> 172.21.15.83:0/3193739601 conn(0x7f54e80b1650 msgr2=0x7f54e80b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:52.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.658+0000 7f54ee9f6640 1 -- 172.21.15.83:0/3193739601 shutdown_connections 2024-03-06T07:25:52.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.658+0000 7f54ee9f6640 1 -- 172.21.15.83:0/3193739601 wait complete. 2024-03-06T07:25:52.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.659+0000 7f54ee9f6640 1 Processor -- start 2024-03-06T07:25:52.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.659+0000 7f54ee9f6640 1 -- start start 2024-03-06T07:25:52.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.660+0000 7f54ee9f6640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f54e806cde0 0x7f54e814d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.660+0000 7f54ee9f6640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f54e80ae880 0x7f54e81f3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.660+0000 7f54ee9f6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54e815df60 0x7f54e815f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.660+0000 7f54ee9f6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e8161a20 con 0x7f54e815df60 2024-03-06T07:25:52.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.660+0000 7f54ee9f6640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e80ade20 con 0x7f54e80ae880 2024-03-06T07:25:52.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.660+0000 7f54ee9f6640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54e8161ba0 con 0x7f54e806cde0 2024-03-06T07:25:52.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.660+0000 7f54ecf93640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54e815df60 0x7f54e815f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.660+0000 7f54ecf93640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54e815df60 0x7f54e815f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37770/0 (socket says 172.21.15.83:37770) 2024-03-06T07:25:52.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.660+0000 7f54ecf93640 1 -- 172.21.15.83:0/1903877476 learned_addr learned my addr 172.21.15.83:0/1903877476 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:52.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.660+0000 7f54e7fff640 1 --2- 172.21.15.83:0/1903877476 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f54e806cde0 0x7f54e814d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.660+0000 7f54e77fe640 1 --2- 172.21.15.83:0/1903877476 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f54e80ae880 0x7f54e81f3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.660+0000 7f54ecf93640 1 -- 172.21.15.83:0/1903877476 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f54e806cde0 msgr2=0x7f54e814d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.660+0000 7f54ecf93640 1 --2- 172.21.15.83:0/1903877476 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f54e806cde0 0x7f54e814d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.660+0000 7f54ecf93640 1 -- 172.21.15.83:0/1903877476 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f54e80ae880 msgr2=0x7f54e81f3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.660+0000 7f54ecf93640 1 --2- 172.21.15.83:0/1903877476 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f54e80ae880 0x7f54e81f3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.660+0000 7f54ecf93640 1 -- 172.21.15.83:0/1903877476 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e8152db0 con 0x7f54e815df60 2024-03-06T07:25:52.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.660+0000 7f54ecf93640 1 --2- 172.21.15.83:0/1903877476 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54e815df60 0x7f54e815f090 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f54e0036040 tx=0x7f54e0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.660+0000 7f54e4ff9640 1 -- 172.21.15.83:0/1903877476 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e0043020 con 0x7f54e815df60 2024-03-06T07:25:52.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.660+0000 7f54e4ff9640 1 -- 172.21.15.83:0/1903877476 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54e003e450 con 0x7f54e815df60 2024-03-06T07:25:52.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.660+0000 7f54e4ff9640 1 -- 172.21.15.83:0/1903877476 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e003e820 con 0x7f54e815df60 2024-03-06T07:25:52.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.660+0000 7f54ee9f6640 1 -- 172.21.15.83:0/1903877476 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54e8153090 con 0x7f54e815df60 2024-03-06T07:25:52.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.660+0000 7f54ee9f6640 1 -- 172.21.15.83:0/1903877476 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54e8153650 con 0x7f54e815df60 2024-03-06T07:25:52.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.661+0000 7f54ee9f6640 1 -- 172.21.15.83:0/1903877476 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54e804f330 con 0x7f54e815df60 2024-03-06T07:25:52.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.662+0000 7f54e4ff9640 1 -- 172.21.15.83:0/1903877476 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f54e0046020 con 0x7f54e815df60 2024-03-06T07:25:52.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.662+0000 7f54e4ff9640 1 --2- 172.21.15.83:0/1903877476 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f54d406b4d0 0x7f54d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.662+0000 7f54e7fff640 1 --2- 172.21.15.83:0/1903877476 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f54d406b4d0 0x7f54d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.662+0000 7f54e4ff9640 1 -- 172.21.15.83:0/1903877476 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f54e00ac860 con 0x7f54e815df60 2024-03-06T07:25:52.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.663+0000 7f54e7fff640 1 --2- 172.21.15.83:0/1903877476 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f54d406b4d0 0x7f54d406d990 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f54d800a950 tx=0x7f54d8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.666+0000 7f54e4ff9640 1 -- 172.21.15.83:0/1903877476 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f54e004d030 con 0x7f54e815df60 2024-03-06T07:25:52.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:52.837+0000 7f54ee9f6640 1 -- 172.21.15.83:0/1903877476 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f54e80d0410 con 0x7f54e815df60 2024-03-06T07:25:53.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.272+0000 7f54e4ff9640 1 -- 172.21.15.83:0/1903877476 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v79) v1 ==== 138+0+0 (secure 0 0 0) 0x7f54e007b9a0 con 0x7f54e815df60 2024-03-06T07:25:53.274 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'cephfs_metadata' created 2024-03-06T07:25:53.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.275+0000 7f54ee9f6640 1 -- 172.21.15.83:0/1903877476 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f54d406b4d0 msgr2=0x7f54d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.275+0000 7f54ee9f6640 1 --2- 172.21.15.83:0/1903877476 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f54d406b4d0 0x7f54d406d990 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f54d800a950 tx=0x7f54d8008040 comp rx=0 tx=0).stop 2024-03-06T07:25:53.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.275+0000 7f54ee9f6640 1 -- 172.21.15.83:0/1903877476 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54e815df60 msgr2=0x7f54e815f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.275+0000 7f54ee9f6640 1 --2- 172.21.15.83:0/1903877476 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54e815df60 0x7f54e815f090 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f54e0036040 tx=0x7f54e0004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:53.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.275+0000 7f54ee9f6640 1 -- 172.21.15.83:0/1903877476 shutdown_connections 2024-03-06T07:25:53.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.275+0000 7f54ee9f6640 1 --2- 172.21.15.83:0/1903877476 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f54d406b4d0 0x7f54d406d990 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.275+0000 7f54ee9f6640 1 --2- 172.21.15.83:0/1903877476 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54e815df60 0x7f54e815f090 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.275+0000 7f54ee9f6640 1 --2- 172.21.15.83:0/1903877476 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f54e80ae880 0x7f54e81f3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.275+0000 7f54ee9f6640 1 --2- 172.21.15.83:0/1903877476 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f54e806cde0 0x7f54e814d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.275+0000 7f54ee9f6640 1 -- 172.21.15.83:0/1903877476 >> 172.21.15.83:0/1903877476 conn(0x7f54e80b1650 msgr2=0x7f54e806dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:53.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.275+0000 7f54ee9f6640 1 -- 172.21.15.83:0/1903877476 shutdown_connections 2024-03-06T07:25:53.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.276+0000 7f54ee9f6640 1 -- 172.21.15.83:0/1903877476 wait complete. 2024-03-06T07:25:53.291 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:25:53.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.396+0000 7fb954d3e640 1 Processor -- start 2024-03-06T07:25:53.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.397+0000 7fb954d3e640 1 -- start start 2024-03-06T07:25:53.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.397+0000 7fb954d3e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb95015bb40 0x7fb9500b4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.397+0000 7fb954d3e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb95015ff90 con 0x7fb95015f7d0 2024-03-06T07:25:53.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.397+0000 7fb954d3e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb950160270 con 0x7fb9500cfa90 2024-03-06T07:25:53.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.397+0000 7fb954d3e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb95006c910 con 0x7fb95015bb40 2024-03-06T07:25:53.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.397+0000 7fb94e59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb95015bb40 0x7fb9500b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.398+0000 7fb947fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fb95015f7d0 0x7fb95006d080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39524/0 (socket says 172.21.15.83:39524) 2024-03-06T07:25:53.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.398+0000 7fb947fff640 1 -- 172.21.15.83:0/2396841342 learned_addr learned my addr 172.21.15.83:0/2396841342 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:25:53.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.398+0000 7fb94e59c640 1 -- 172.21.15.83:0/2396841342 >> v1:172.21.15.183:6789/0 conn(0x7fb9500cfa90 legacy=0x7fb9500cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:53.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.398+0000 7fb94e59c640 1 -- 172.21.15.83:0/2396841342 >> v1:172.21.15.83:6789/0 conn(0x7fb95015f7d0 legacy=0x7fb95006d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.398+0000 7fb94e59c640 1 -- 172.21.15.83:0/2396841342 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb950160550 con 0x7fb95015bb40 2024-03-06T07:25:53.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.398+0000 7fb94dd9b640 1 -- 172.21.15.83:0/2396841342 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 724582925 0 0) 0x7fb95015ff90 con 0x7fb95015f7d0 2024-03-06T07:25:53.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.398+0000 7fb94e59c640 1 --2- 172.21.15.83:0/2396841342 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb95015bb40 0x7fb9500b4970 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fb95006ca90 tx=0x7fb94002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54f9ea8373407246 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:53.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.399+0000 7fb94dd9b640 1 -- 172.21.15.83:0/2396841342 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb940035020 con 0x7fb95015bb40 2024-03-06T07:25:53.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.399+0000 7fb94dd9b640 1 -- 172.21.15.83:0/2396841342 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb940031830 con 0x7fb95015bb40 2024-03-06T07:25:53.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.399+0000 7fb94dd9b640 1 -- 172.21.15.83:0/2396841342 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb940031c00 con 0x7fb95015bb40 2024-03-06T07:25:53.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.399+0000 7fb954d3e640 1 -- 172.21.15.83:0/2396841342 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb95015bb40 msgr2=0x7fb9500b4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.399+0000 7fb954d3e640 1 --2- 172.21.15.83:0/2396841342 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb95015bb40 0x7fb9500b4970 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fb95006ca90 tx=0x7fb94002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.399+0000 7fb954d3e640 1 -- 172.21.15.83:0/2396841342 shutdown_connections 2024-03-06T07:25:53.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.399+0000 7fb954d3e640 1 --2- 172.21.15.83:0/2396841342 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb95015bb40 0x7fb9500b4970 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.399+0000 7fb954d3e640 1 -- 172.21.15.83:0/2396841342 >> 172.21.15.83:0/2396841342 conn(0x7fb95006b190 msgr2=0x7fb95006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:53.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.399+0000 7fb954d3e640 1 -- 172.21.15.83:0/2396841342 shutdown_connections 2024-03-06T07:25:53.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.399+0000 7fb954d3e640 1 -- 172.21.15.83:0/2396841342 wait complete. 2024-03-06T07:25:53.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.401+0000 7fb954d3e640 1 Processor -- start 2024-03-06T07:25:53.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.401+0000 7fb954d3e640 1 -- start start 2024-03-06T07:25:53.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.401+0000 7fb954d3e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9500cfa90 0x7fb950159070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.401+0000 7fb954d3e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb95015bb40 0x7fb9501e55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.401+0000 7fb954d3e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb95015f7d0 0x7fb9500b2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.401+0000 7fb954d3e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb95006d790 con 0x7fb9500cfa90 2024-03-06T07:25:53.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.401+0000 7fb954d3e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb95015fc30 con 0x7fb95015bb40 2024-03-06T07:25:53.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.401+0000 7fb954d3e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb95015bf20 con 0x7fb95015f7d0 2024-03-06T07:25:53.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.401+0000 7fb94e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9500cfa90 0x7fb950159070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.401+0000 7fb94e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9500cfa90 0x7fb950159070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37782/0 (socket says 172.21.15.83:37782) 2024-03-06T07:25:53.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.401+0000 7fb94e59c640 1 -- 172.21.15.83:0/2492428110 learned_addr learned my addr 172.21.15.83:0/2492428110 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:53.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.402+0000 7fb94ed9d640 1 --2- 172.21.15.83:0/2492428110 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb95015f7d0 0x7fb9500b2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.402+0000 7fb947fff640 1 --2- 172.21.15.83:0/2492428110 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb95015bb40 0x7fb9501e55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.402+0000 7fb94e59c640 1 -- 172.21.15.83:0/2492428110 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb95015f7d0 msgr2=0x7fb9500b2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.402+0000 7fb94e59c640 1 --2- 172.21.15.83:0/2492428110 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb95015f7d0 0x7fb9500b2870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.402+0000 7fb94e59c640 1 -- 172.21.15.83:0/2492428110 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb95015bb40 msgr2=0x7fb9501e55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.402+0000 7fb94e59c640 1 --2- 172.21.15.83:0/2492428110 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb95015bb40 0x7fb9501e55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.402+0000 7fb94e59c640 1 -- 172.21.15.83:0/2492428110 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb950159650 con 0x7fb9500cfa90 2024-03-06T07:25:53.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.402+0000 7fb94e59c640 1 --2- 172.21.15.83:0/2492428110 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9500cfa90 0x7fb950159070 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fb940009890 tx=0x7fb9400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:53.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.402+0000 7fb946ffd640 1 -- 172.21.15.83:0/2492428110 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb94003ba80 con 0x7fb9500cfa90 2024-03-06T07:25:53.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.402+0000 7fb946ffd640 1 -- 172.21.15.83:0/2492428110 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb94003bc40 con 0x7fb9500cfa90 2024-03-06T07:25:53.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.402+0000 7fb954d3e640 1 -- 172.21.15.83:0/2492428110 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb950159930 con 0x7fb9500cfa90 2024-03-06T07:25:53.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.402+0000 7fb946ffd640 1 -- 172.21.15.83:0/2492428110 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9400453f0 con 0x7fb9500cfa90 2024-03-06T07:25:53.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.402+0000 7fb954d3e640 1 -- 172.21.15.83:0/2492428110 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb950159ef0 con 0x7fb9500cfa90 2024-03-06T07:25:53.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.403+0000 7fb954d3e640 1 -- 172.21.15.83:0/2492428110 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9500732b0 con 0x7fb9500cfa90 2024-03-06T07:25:53.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.404+0000 7fb946ffd640 1 -- 172.21.15.83:0/2492428110 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb940038020 con 0x7fb9500cfa90 2024-03-06T07:25:53.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.404+0000 7fb946ffd640 1 --2- 172.21.15.83:0/2492428110 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb93006b4d0 0x7fb93006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.404+0000 7fb947fff640 1 --2- 172.21.15.83:0/2492428110 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb93006b4d0 0x7fb93006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.404+0000 7fb946ffd640 1 -- 172.21.15.83:0/2492428110 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fb9400abe60 con 0x7fb9500cfa90 2024-03-06T07:25:53.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.404+0000 7fb947fff640 1 --2- 172.21.15.83:0/2492428110 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb93006b4d0 0x7fb93006d990 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fb950159000 tx=0x7fb938007d00 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:53.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.409+0000 7fb946ffd640 1 -- 172.21.15.83:0/2492428110 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb940040070 con 0x7fb9500cfa90 2024-03-06T07:25:53.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:53.579+0000 7fb954d3e640 1 -- 172.21.15.83:0/2492428110 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fb9500b32f0 con 0x7fb9500cfa90 2024-03-06T07:25:54.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.277+0000 7fb946ffd640 1 -- 172.21.15.83:0/2492428110 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v80) v1 ==== 170+0+0 (secure 0 0 0) 0x7fb94007aec0 con 0x7fb9500cfa90 2024-03-06T07:25:54.279 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'cephfs_data' created 2024-03-06T07:25:54.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.281+0000 7fb954d3e640 1 -- 172.21.15.83:0/2492428110 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb93006b4d0 msgr2=0x7fb93006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.281+0000 7fb954d3e640 1 --2- 172.21.15.83:0/2492428110 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb93006b4d0 0x7fb93006d990 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fb950159000 tx=0x7fb938007d00 comp rx=0 tx=0).stop 2024-03-06T07:25:54.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.281+0000 7fb954d3e640 1 -- 172.21.15.83:0/2492428110 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9500cfa90 msgr2=0x7fb950159070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.281+0000 7fb954d3e640 1 --2- 172.21.15.83:0/2492428110 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9500cfa90 0x7fb950159070 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fb940009890 tx=0x7fb9400046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.281+0000 7fb954d3e640 1 -- 172.21.15.83:0/2492428110 shutdown_connections 2024-03-06T07:25:54.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.281+0000 7fb954d3e640 1 --2- 172.21.15.83:0/2492428110 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb93006b4d0 0x7fb93006d990 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.281+0000 7fb954d3e640 1 --2- 172.21.15.83:0/2492428110 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb95015f7d0 0x7fb9500b2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.281+0000 7fb954d3e640 1 --2- 172.21.15.83:0/2492428110 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb95015bb40 0x7fb9501e55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.281+0000 7fb954d3e640 1 --2- 172.21.15.83:0/2492428110 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9500cfa90 0x7fb950159070 secure :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fb940009890 tx=0x7fb9400046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.281+0000 7fb954d3e640 1 -- 172.21.15.83:0/2492428110 >> 172.21.15.83:0/2492428110 conn(0x7fb95006b190 msgr2=0x7fb9500b1160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:54.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.292+0000 7fb954d3e640 1 -- 172.21.15.83:0/2492428110 shutdown_connections 2024-03-06T07:25:54.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.293+0000 7fb954d3e640 1 -- 172.21.15.83:0/2492428110 wait complete. 2024-03-06T07:25:54.308 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:25:54.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.411+0000 7f5772d3e640 1 Processor -- start 2024-03-06T07:25:54.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.411+0000 7f5772d3e640 1 -- start start 2024-03-06T07:25:54.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.412+0000 7f5772d3e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f576c15bc50 0x7f576c0cf430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.412+0000 7f5772d3e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f576c0cc280 0x7f576c0cf970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.412+0000 7f5772d3e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f576c06d5b0 con 0x7f576c0aee20 2024-03-06T07:25:54.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.412+0000 7f5772d3e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f576c0b4850 con 0x7f576c15bc50 2024-03-06T07:25:54.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.412+0000 7f5772d3e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f576c0cc7b0 con 0x7f576c0cc280 2024-03-06T07:25:54.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.412+0000 7f57712db640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f576c0aee20 0x7f576c15f8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39530/0 (socket says 172.21.15.83:39530) 2024-03-06T07:25:54.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.412+0000 7f57712db640 1 -- 172.21.15.83:0/2704758420 learned_addr learned my addr 172.21.15.83:0/2704758420 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:25:54.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.412+0000 7f576b7fe640 1 -- 172.21.15.83:0/2704758420 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2180314206 0 0) 0x7f576c06d5b0 con 0x7f576c0aee20 2024-03-06T07:25:54.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.412+0000 7f576bfff640 1 --2- 172.21.15.83:0/2704758420 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f576c0cc280 0x7f576c0cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.412+0000 7f5770ada640 1 --2- 172.21.15.83:0/2704758420 >> v2:172.21.15.183:3300/0 conn(0x7f576c15bc50 0x7f576c0cf430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.412+0000 7f576b7fe640 1 -- 172.21.15.83:0/2704758420 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5754003520 con 0x7f576c0aee20 2024-03-06T07:25:54.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.413+0000 7f576b7fe640 1 -- 172.21.15.83:0/2704758420 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3406013462 0 0) 0x7f5754003520 con 0x7f576c0aee20 2024-03-06T07:25:54.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.413+0000 7f576b7fe640 1 -- 172.21.15.83:0/2704758420 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f576c0cc280 msgr2=0x7f576c0cf970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:54.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.413+0000 7f576b7fe640 1 --2- 172.21.15.83:0/2704758420 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f576c0cc280 0x7f576c0cf970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.413+0000 7f576b7fe640 1 -- 172.21.15.83:0/2704758420 >> v2:172.21.15.183:3300/0 conn(0x7f576c15bc50 msgr2=0x7f576c0cf430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:54.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.413+0000 7f576b7fe640 1 --2- 172.21.15.83:0/2704758420 >> v2:172.21.15.183:3300/0 conn(0x7f576c15bc50 0x7f576c0cf430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.413+0000 7f576b7fe640 1 -- 172.21.15.83:0/2704758420 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f576c0b4ce0 con 0x7f576c0aee20 2024-03-06T07:25:54.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.413+0000 7f576b7fe640 1 -- 172.21.15.83:0/2704758420 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f57640019f0 con 0x7f576c0aee20 2024-03-06T07:25:54.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.413+0000 7f576b7fe640 1 -- 172.21.15.83:0/2704758420 >> v1:172.21.15.83:6789/0 conn(0x7f576c0aee20 legacy=0x7f576c15f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.413+0000 7f576b7fe640 1 --2- 172.21.15.83:0/2704758420 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f576c0cc280 0x7f57540039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.413+0000 7f576b7fe640 1 --2- 172.21.15.83:0/2704758420 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5754003f30 0x7f5754004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.413+0000 7f576b7fe640 1 --2- 172.21.15.83:0/2704758420 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5754004d40 0x7f57540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.413+0000 7f576b7fe640 1 -- 172.21.15.83:0/2704758420 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f576c0b4850 con 0x7f5754004d40 2024-03-06T07:25:54.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.413+0000 7f576b7fe640 1 -- 172.21.15.83:0/2704758420 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f576c0cc7b0 con 0x7f5754003f30 2024-03-06T07:25:54.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.413+0000 7f576b7fe640 1 -- 172.21.15.83:0/2704758420 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f575400ba10 con 0x7f576c0cc280 2024-03-06T07:25:54.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.413+0000 7f57712db640 1 --2- 172.21.15.83:0/2704758420 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5754004d40 0x7f57540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.413+0000 7f5770ada640 1 --2- 172.21.15.83:0/2704758420 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f576c0cc280 0x7f57540039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.413+0000 7f576bfff640 1 --2- 172.21.15.83:0/2704758420 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5754003f30 0x7f5754004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.413+0000 7f57712db640 1 -- 172.21.15.83:0/2704758420 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f576c0cc280 msgr2=0x7f57540039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:54.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.413+0000 7f57712db640 1 --2- 172.21.15.83:0/2704758420 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f576c0cc280 0x7f57540039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.413+0000 7f57712db640 1 -- 172.21.15.83:0/2704758420 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5754003f30 msgr2=0x7f5754004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:54.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.413+0000 7f57712db640 1 --2- 172.21.15.83:0/2704758420 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5754003f30 0x7f5754004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.413+0000 7f57712db640 1 -- 172.21.15.83:0/2704758420 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5754003520 con 0x7f5754004d40 2024-03-06T07:25:54.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.414+0000 7f57712db640 1 --2- 172.21.15.83:0/2704758420 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5754004d40 0x7f57540051c0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f576c072cf0 tx=0x7f5754005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b649a0685e68930 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:54.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.414+0000 7f576b7fe640 1 -- 172.21.15.83:0/2704758420 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576400a4e0 con 0x7f5754004d40 2024-03-06T07:25:54.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.414+0000 7f576b7fe640 1 -- 172.21.15.83:0/2704758420 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5764034650 con 0x7f5754004d40 2024-03-06T07:25:54.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.414+0000 7f576b7fe640 1 -- 172.21.15.83:0/2704758420 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5764034a40 con 0x7f5754004d40 2024-03-06T07:25:54.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.414+0000 7f5772d3e640 1 -- 172.21.15.83:0/2704758420 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5754004d40 msgr2=0x7f57540051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.414+0000 7f5772d3e640 1 --2- 172.21.15.83:0/2704758420 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5754004d40 0x7f57540051c0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f576c072cf0 tx=0x7f5754005830 comp rx=0 tx=0).stop 2024-03-06T07:25:54.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.414+0000 7f57712db640 1 -- 172.21.15.83:0/2704758420 reap_dead start 2024-03-06T07:25:54.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.415+0000 7f5772d3e640 1 -- 172.21.15.83:0/2704758420 shutdown_connections 2024-03-06T07:25:54.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.415+0000 7f5772d3e640 1 -- 172.21.15.83:0/2704758420 >> 172.21.15.83:0/2704758420 conn(0x7f576c06afb0 msgr2=0x7f576c06b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:54.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.415+0000 7f5772d3e640 1 -- 172.21.15.83:0/2704758420 shutdown_connections 2024-03-06T07:25:54.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.415+0000 7f5772d3e640 1 -- 172.21.15.83:0/2704758420 wait complete. 2024-03-06T07:25:54.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.416+0000 7f5772d3e640 1 Processor -- start 2024-03-06T07:25:54.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.416+0000 7f5772d3e640 1 -- start start 2024-03-06T07:25:54.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.416+0000 7f5772d3e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f576c1492c0 0x7f576c0ab2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.417+0000 7f5772d3e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f576c153570 0x7f576c0afde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.417+0000 7f5772d3e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f576c1f1b70 0x7f576c0b0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.417+0000 7f5772d3e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f576c15c030 con 0x7f576c1492c0 2024-03-06T07:25:54.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.417+0000 7f5772d3e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f576c0ae900 con 0x7f576c1f1b70 2024-03-06T07:25:54.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.417+0000 7f5772d3e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f576c15c1b0 con 0x7f576c153570 2024-03-06T07:25:54.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.417+0000 7f5770ada640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f576c1492c0 0x7f576c0ab2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.417+0000 7f5770ada640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f576c1492c0 0x7f576c0ab2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37802/0 (socket says 172.21.15.83:37802) 2024-03-06T07:25:54.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.417+0000 7f5770ada640 1 -- 172.21.15.83:0/2441716918 learned_addr learned my addr 172.21.15.83:0/2441716918 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:54.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.417+0000 7f576bfff640 1 --2- 172.21.15.83:0/2441716918 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f576c153570 0x7f576c0afde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.417+0000 7f57712db640 1 --2- 172.21.15.83:0/2441716918 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f576c1f1b70 0x7f576c0b0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.417+0000 7f5770ada640 1 -- 172.21.15.83:0/2441716918 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f576c153570 msgr2=0x7f576c0afde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.417+0000 7f5770ada640 1 --2- 172.21.15.83:0/2441716918 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f576c153570 0x7f576c0afde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.417+0000 7f5770ada640 1 -- 172.21.15.83:0/2441716918 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f576c1f1b70 msgr2=0x7f576c0b0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.417+0000 7f5770ada640 1 --2- 172.21.15.83:0/2441716918 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f576c1f1b70 0x7f576c0b0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.417+0000 7f5770ada640 1 -- 172.21.15.83:0/2441716918 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f576c0abda0 con 0x7f576c1492c0 2024-03-06T07:25:54.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.417+0000 7f5770ada640 1 --2- 172.21.15.83:0/2441716918 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f576c1492c0 0x7f576c0ab2c0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f575c011a70 tx=0x7f575c011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:54.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.417+0000 7f57697fa640 1 -- 172.21.15.83:0/2441716918 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575c00fb50 con 0x7f576c1492c0 2024-03-06T07:25:54.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.417+0000 7f57697fa640 1 -- 172.21.15.83:0/2441716918 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f575c00fcf0 con 0x7f576c1492c0 2024-03-06T07:25:54.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.417+0000 7f5772d3e640 1 -- 172.21.15.83:0/2441716918 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f576c0ac080 con 0x7f576c1492c0 2024-03-06T07:25:54.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.417+0000 7f57697fa640 1 -- 172.21.15.83:0/2441716918 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575c00fea0 con 0x7f576c1492c0 2024-03-06T07:25:54.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.417+0000 7f5772d3e640 1 -- 172.21.15.83:0/2441716918 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f576c0ac640 con 0x7f576c1492c0 2024-03-06T07:25:54.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.418+0000 7f5772d3e640 1 -- 172.21.15.83:0/2441716918 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f576c04f330 con 0x7f576c1492c0 2024-03-06T07:25:54.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.419+0000 7f57697fa640 1 -- 172.21.15.83:0/2441716918 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f575c012040 con 0x7f576c1492c0 2024-03-06T07:25:54.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.419+0000 7f57697fa640 1 --2- 172.21.15.83:0/2441716918 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f575806b500 0x7f575806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.419+0000 7f57697fa640 1 -- 172.21.15.83:0/2441716918 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f575c08a910 con 0x7f576c1492c0 2024-03-06T07:25:54.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.419+0000 7f576bfff640 1 --2- 172.21.15.83:0/2441716918 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f575806b500 0x7f575806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.419+0000 7f576bfff640 1 --2- 172.21.15.83:0/2441716918 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f575806b500 0x7f575806d9c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f576c15fea0 tx=0x7f5760008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:54.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.423+0000 7f57697fa640 1 -- 172.21.15.83:0/2441716918 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f575c019030 con 0x7f576c1492c0 2024-03-06T07:25:54.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:54.630+0000 7f5772d3e640 1 -- 172.21.15.83:0/2441716918 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f576c06e5b0 con 0x7f576c1492c0 2024-03-06T07:25:55.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.279+0000 7f57697fa640 1 -- 172.21.15.83:0/2441716918 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v37) v1 ==== 174+0+0 (secure 0 0 0) 0x7f575c0597f0 con 0x7f576c1492c0 2024-03-06T07:25:55.281 INFO:teuthology.orchestra.run.smithi083.stderr:new fs with metadata pool 12 and data pool 13 2024-03-06T07:25:55.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.284+0000 7f5772d3e640 1 -- 172.21.15.83:0/2441716918 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f575806b500 msgr2=0x7f575806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.284+0000 7f5772d3e640 1 --2- 172.21.15.83:0/2441716918 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f575806b500 0x7f575806d9c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f576c15fea0 tx=0x7f5760008040 comp rx=0 tx=0).stop 2024-03-06T07:25:55.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.284+0000 7f5772d3e640 1 -- 172.21.15.83:0/2441716918 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f576c1492c0 msgr2=0x7f576c0ab2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.284+0000 7f5772d3e640 1 --2- 172.21.15.83:0/2441716918 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f576c1492c0 0x7f576c0ab2c0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f575c011a70 tx=0x7f575c011f30 comp rx=0 tx=0).stop 2024-03-06T07:25:55.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.284+0000 7f5772d3e640 1 -- 172.21.15.83:0/2441716918 shutdown_connections 2024-03-06T07:25:55.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.284+0000 7f5772d3e640 1 --2- 172.21.15.83:0/2441716918 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f575806b500 0x7f575806d9c0 secure :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f576c15fea0 tx=0x7f5760008040 comp rx=0 tx=0).stop 2024-03-06T07:25:55.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.284+0000 7f5772d3e640 1 --2- 172.21.15.83:0/2441716918 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f576c1f1b70 0x7f576c0b0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.284+0000 7f5772d3e640 1 --2- 172.21.15.83:0/2441716918 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f576c153570 0x7f576c0afde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.284+0000 7f5772d3e640 1 --2- 172.21.15.83:0/2441716918 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f576c1492c0 0x7f576c0ab2c0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.284+0000 7f5772d3e640 1 -- 172.21.15.83:0/2441716918 >> 172.21.15.83:0/2441716918 conn(0x7f576c06afb0 msgr2=0x7f576c0af300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:55.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.287+0000 7f5772d3e640 1 -- 172.21.15.83:0/2441716918 shutdown_connections 2024-03-06T07:25:55.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.290+0000 7f5772d3e640 1 -- 172.21.15.83:0/2441716918 wait complete. 2024-03-06T07:25:55.319 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:55.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.422+0000 7f8fd8b93640 1 Processor -- start 2024-03-06T07:25:55.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.422+0000 7f8fd8b93640 1 -- start start 2024-03-06T07:25:55.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.422+0000 7f8fd8b93640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f8fd415dfe0 0x7f8fd40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.422+0000 7f8fd8b93640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f8fd40cf830 0x7f8fd40d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.422+0000 7f8fd8b93640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8fd40d27f0 0x7f8fd40d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.422+0000 7f8fd8b93640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fd40b31e0 con 0x7f8fd40cf830 2024-03-06T07:25:55.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.422+0000 7f8fd8b93640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fd40b3d60 con 0x7f8fd415dfe0 2024-03-06T07:25:55.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.422+0000 7f8fd8b93640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd40b3430 con 0x7f8fd40d27f0 2024-03-06T07:25:55.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.423+0000 7f8fcbfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f8fd40cf830 0x7f8fd40d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.423+0000 7f8fcbfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f8fd40cf830 0x7f8fd40d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37806/0 (socket says 172.21.15.83:37806) 2024-03-06T07:25:55.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.423+0000 7f8fcbfff640 1 -- 172.21.15.83:0/1303063305 learned_addr learned my addr 172.21.15.83:0/1303063305 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:55.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.423+0000 7f8fd2d9d640 1 --2- 172.21.15.83:0/1303063305 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8fd40d27f0 0x7f8fd40d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.423+0000 7f8fd259c640 1 --2- 172.21.15.83:0/1303063305 >> v2:172.21.15.183:3300/0 conn(0x7f8fd415dfe0 0x7f8fd40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.423+0000 7f8fcbfff640 1 -- 172.21.15.83:0/1303063305 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8fd40d27f0 msgr2=0x7f8fd40d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:55.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.423+0000 7f8fcbfff640 1 --2- 172.21.15.83:0/1303063305 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8fd40d27f0 0x7f8fd40d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.423+0000 7f8fcbfff640 1 -- 172.21.15.83:0/1303063305 >> v2:172.21.15.183:3300/0 conn(0x7f8fd415dfe0 msgr2=0x7f8fd40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:55.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.423+0000 7f8fcbfff640 1 --2- 172.21.15.83:0/1303063305 >> v2:172.21.15.183:3300/0 conn(0x7f8fd415dfe0 0x7f8fd40cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.423+0000 7f8fcbfff640 1 -- 172.21.15.83:0/1303063305 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd40b3710 con 0x7f8fd40cf830 2024-03-06T07:25:55.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.423+0000 7f8fd259c640 1 --2- 172.21.15.83:0/1303063305 >> v2:172.21.15.183:3300/0 conn(0x7f8fd415dfe0 0x7f8fd40cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:55.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.424+0000 7f8fcbfff640 1 --2- 172.21.15.83:0/1303063305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8fd40cf830 0x7f8fd40d22b0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4067990 tx=0x7f8fbc02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3f3d1ea08e8d58a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.424+0000 7f8fd1d9b640 1 -- 172.21.15.83:0/1303063305 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc035020 con 0x7f8fd40cf830 2024-03-06T07:25:55.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.424+0000 7f8fd1d9b640 1 -- 172.21.15.83:0/1303063305 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fbc02d040 con 0x7f8fd40cf830 2024-03-06T07:25:55.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.424+0000 7f8fd1d9b640 1 -- 172.21.15.83:0/1303063305 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc031a80 con 0x7f8fd40cf830 2024-03-06T07:25:55.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.424+0000 7f8fd8b93640 1 -- 172.21.15.83:0/1303063305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8fd40cf830 msgr2=0x7f8fd40d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.424+0000 7f8fd8b93640 1 --2- 172.21.15.83:0/1303063305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8fd40cf830 0x7f8fd40d22b0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4067990 tx=0x7f8fbc02b900 comp rx=0 tx=0).stop 2024-03-06T07:25:55.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.424+0000 7f8fd8b93640 1 -- 172.21.15.83:0/1303063305 shutdown_connections 2024-03-06T07:25:55.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.424+0000 7f8fd8b93640 1 --2- 172.21.15.83:0/1303063305 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8fd40d27f0 0x7f8fd40d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.424+0000 7f8fd8b93640 1 --2- 172.21.15.83:0/1303063305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8fd40cf830 0x7f8fd40d22b0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.424+0000 7f8fd8b93640 1 --2- 172.21.15.83:0/1303063305 >> v2:172.21.15.183:3300/0 conn(0x7f8fd415dfe0 0x7f8fd40cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.424+0000 7f8fd8b93640 1 -- 172.21.15.83:0/1303063305 >> 172.21.15.83:0/1303063305 conn(0x7f8fd40d3cd0 msgr2=0x7f8fd40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:55.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.424+0000 7f8fd8b93640 1 -- 172.21.15.83:0/1303063305 shutdown_connections 2024-03-06T07:25:55.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.424+0000 7f8fd8b93640 1 -- 172.21.15.83:0/1303063305 wait complete. 2024-03-06T07:25:55.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.426+0000 7f8fd8b93640 1 Processor -- start 2024-03-06T07:25:55.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.426+0000 7f8fd8b93640 1 -- start start 2024-03-06T07:25:55.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.426+0000 7f8fd8b93640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8fd40cf830 0x7f8fd414add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.426+0000 7f8fd8b93640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8fd40d27f0 0x7f8fd414cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.426+0000 7f8fd8b93640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8fd415dfe0 0x7f8fd41f7a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.426+0000 7f8fd8b93640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd40d1df0 con 0x7f8fd40cf830 2024-03-06T07:25:55.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.426+0000 7f8fd8b93640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd40d1c70 con 0x7f8fd415dfe0 2024-03-06T07:25:55.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.426+0000 7f8fd8b93640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd415e3c0 con 0x7f8fd40d27f0 2024-03-06T07:25:55.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.426+0000 7f8fd259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8fd40cf830 0x7f8fd414add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.426+0000 7f8fd259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8fd40cf830 0x7f8fd414add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37822/0 (socket says 172.21.15.83:37822) 2024-03-06T07:25:55.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.426+0000 7f8fd259c640 1 -- 172.21.15.83:0/3992164866 learned_addr learned my addr 172.21.15.83:0/3992164866 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:55.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.427+0000 7f8fcbfff640 1 --2- 172.21.15.83:0/3992164866 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8fd40d27f0 0x7f8fd414cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.427+0000 7f8fd2d9d640 1 --2- 172.21.15.83:0/3992164866 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8fd415dfe0 0x7f8fd41f7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.427+0000 7f8fd259c640 1 -- 172.21.15.83:0/3992164866 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8fd40d27f0 msgr2=0x7f8fd414cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.427+0000 7f8fd259c640 1 --2- 172.21.15.83:0/3992164866 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8fd40d27f0 0x7f8fd414cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.427+0000 7f8fd259c640 1 -- 172.21.15.83:0/3992164866 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8fd415dfe0 msgr2=0x7f8fd41f7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.427+0000 7f8fd259c640 1 --2- 172.21.15.83:0/3992164866 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8fd415dfe0 0x7f8fd41f7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.427+0000 7f8fd259c640 1 -- 172.21.15.83:0/3992164866 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd414d480 con 0x7f8fd40cf830 2024-03-06T07:25:55.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.427+0000 7f8fd259c640 1 --2- 172.21.15.83:0/3992164866 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8fd40cf830 0x7f8fd414add0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f8fc400c7f0 tx=0x7f8fc400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.427+0000 7f8fcaffd640 1 -- 172.21.15.83:0/3992164866 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4015070 con 0x7f8fd40cf830 2024-03-06T07:25:55.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.427+0000 7f8fcaffd640 1 -- 172.21.15.83:0/3992164866 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc4007820 con 0x7f8fd40cf830 2024-03-06T07:25:55.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.427+0000 7f8fd8b93640 1 -- 172.21.15.83:0/3992164866 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fd414d760 con 0x7f8fd40cf830 2024-03-06T07:25:55.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.427+0000 7f8fcaffd640 1 -- 172.21.15.83:0/3992164866 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4007bd0 con 0x7f8fd40cf830 2024-03-06T07:25:55.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.427+0000 7f8fd8b93640 1 -- 172.21.15.83:0/3992164866 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fd414dd20 con 0x7f8fd40cf830 2024-03-06T07:25:55.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.428+0000 7f8fd8b93640 1 -- 172.21.15.83:0/3992164866 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fd404f330 con 0x7f8fd40cf830 2024-03-06T07:25:55.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.428+0000 7f8fcaffd640 1 -- 172.21.15.83:0/3992164866 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8fc4018070 con 0x7f8fd40cf830 2024-03-06T07:25:55.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.429+0000 7f8fcaffd640 1 --2- 172.21.15.83:0/3992164866 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8fb406b4d0 0x7f8fb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.429+0000 7f8fcbfff640 1 --2- 172.21.15.83:0/3992164866 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8fb406b4d0 0x7f8fb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.429+0000 7f8fcaffd640 1 -- 172.21.15.83:0/3992164866 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8fc408a2c0 con 0x7f8fd40cf830 2024-03-06T07:25:55.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.429+0000 7f8fcbfff640 1 --2- 172.21.15.83:0/3992164866 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8fb406b4d0 0x7f8fb406d990 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc036000 tx=0x7f8fbc03c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.433+0000 7f8fcaffd640 1 -- 172.21.15.83:0/3992164866 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8fc4058af0 con 0x7f8fd40cf830 2024-03-06T07:25:55.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.605+0000 7f8fd8b93640 1 -- 172.21.15.83:0/3992164866 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8fd40ceb20 con 0x7f8fd40cf830 2024-03-06T07:25:55.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.606+0000 7f8fcaffd640 1 -- 172.21.15.83:0/3992164866 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16685 (secure 0 0 0) 0x7f8fc405cb90 con 0x7f8fd40cf830 2024-03-06T07:25:55.608 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:25:55.608 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":81,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:25:54.631655+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:52.856525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:53.598434+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:55.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.608+0000 7f8fd8b93640 1 -- 172.21.15.83:0/3992164866 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8fb406b4d0 msgr2=0x7f8fb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.608+0000 7f8fd8b93640 1 --2- 172.21.15.83:0/3992164866 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8fb406b4d0 0x7f8fb406d990 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc036000 tx=0x7f8fbc03c040 comp rx=0 tx=0).stop 2024-03-06T07:25:55.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.608+0000 7f8fd8b93640 1 -- 172.21.15.83:0/3992164866 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8fd40cf830 msgr2=0x7f8fd414add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.608+0000 7f8fd8b93640 1 --2- 172.21.15.83:0/3992164866 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8fd40cf830 0x7f8fd414add0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f8fc400c7f0 tx=0x7f8fc400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.608+0000 7f8fd8b93640 1 -- 172.21.15.83:0/3992164866 shutdown_connections 2024-03-06T07:25:55.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.608+0000 7f8fd8b93640 1 --2- 172.21.15.83:0/3992164866 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8fb406b4d0 0x7f8fb406d990 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.608+0000 7f8fd8b93640 1 --2- 172.21.15.83:0/3992164866 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8fd415dfe0 0x7f8fd41f7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.608+0000 7f8fd8b93640 1 --2- 172.21.15.83:0/3992164866 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8fd40d27f0 0x7f8fd414cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.608+0000 7f8fd8b93640 1 --2- 172.21.15.83:0/3992164866 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8fd40cf830 0x7f8fd414add0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.608+0000 7f8fd8b93640 1 -- 172.21.15.83:0/3992164866 >> 172.21.15.83:0/3992164866 conn(0x7f8fd40d3cd0 msgr2=0x7f8fd406bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:55.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.608+0000 7f8fd8b93640 1 -- 172.21.15.83:0/3992164866 shutdown_connections 2024-03-06T07:25:55.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.608+0000 7f8fd8b93640 1 -- 172.21.15.83:0/3992164866 wait complete. 2024-03-06T07:25:55.621 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:55.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.719+0000 7fa2a4a9a640 1 Processor -- start 2024-03-06T07:25:55.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.719+0000 7fa2a4a9a640 1 -- start start 2024-03-06T07:25:55.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.720+0000 7fa2a4a9a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa2a015dfe0 0x7fa2a00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.720+0000 7fa2a4a9a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fa2a00d29c0 0x7fa2a00b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.720+0000 7fa2a4a9a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2a00b3430 con 0x7fa2a00cf830 2024-03-06T07:25:55.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.720+0000 7fa2a4a9a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2a00b31e0 con 0x7fa2a00d29c0 2024-03-06T07:25:55.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.720+0000 7fa2a4a9a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a00b3d60 con 0x7fa2a015dfe0 2024-03-06T07:25:55.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.720+0000 7fa29dd9b640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fa2a00cf830 0x7fa2a00d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39542/0 (socket says 172.21.15.83:39542) 2024-03-06T07:25:55.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.720+0000 7fa29dd9b640 1 -- 172.21.15.83:0/3191206615 learned_addr learned my addr 172.21.15.83:0/3191206615 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:25:55.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.721+0000 7fa29d59a640 1 -- 172.21.15.83:0/3191206615 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3514347371 0 0) 0x7fa2a00b3430 con 0x7fa2a00cf830 2024-03-06T07:25:55.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.721+0000 7fa29e59c640 1 --2- 172.21.15.83:0/3191206615 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa2a015dfe0 0x7fa2a00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.721+0000 7fa29d59a640 1 -- 172.21.15.83:0/3191206615 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa288003520 con 0x7fa2a00cf830 2024-03-06T07:25:55.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.721+0000 7fa29ed9d640 1 --2- 172.21.15.83:0/3191206615 >> v2:172.21.15.183:3300/0 conn(0x7fa2a00d29c0 0x7fa2a00b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.721+0000 7fa29d59a640 1 -- 172.21.15.83:0/3191206615 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3484519919 0 0) 0x7fa288003520 con 0x7fa2a00cf830 2024-03-06T07:25:55.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.721+0000 7fa29d59a640 1 -- 172.21.15.83:0/3191206615 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa2a015dfe0 msgr2=0x7fa2a00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:55.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.721+0000 7fa29d59a640 1 --2- 172.21.15.83:0/3191206615 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa2a015dfe0 0x7fa2a00cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.721+0000 7fa29d59a640 1 -- 172.21.15.83:0/3191206615 >> v2:172.21.15.183:3300/0 conn(0x7fa2a00d29c0 msgr2=0x7fa2a00b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:55.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.721+0000 7fa29d59a640 1 --2- 172.21.15.83:0/3191206615 >> v2:172.21.15.183:3300/0 conn(0x7fa2a00d29c0 0x7fa2a00b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.721+0000 7fa29d59a640 1 -- 172.21.15.83:0/3191206615 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2a00b3710 con 0x7fa2a00cf830 2024-03-06T07:25:55.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.721+0000 7fa29d59a640 1 -- 172.21.15.83:0/3191206615 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7fa294001670 con 0x7fa2a00cf830 2024-03-06T07:25:55.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.721+0000 7fa29d59a640 1 -- 172.21.15.83:0/3191206615 >> v1:172.21.15.83:6789/0 conn(0x7fa2a00cf830 legacy=0x7fa2a00d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.721+0000 7fa29d59a640 1 --2- 172.21.15.83:0/3191206615 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa2880039b0 0x7fa288003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.721+0000 7fa29d59a640 1 --2- 172.21.15.83:0/3191206615 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa2a015dfe0 0x7fa288004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.721+0000 7fa29d59a640 1 --2- 172.21.15.83:0/3191206615 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa288004d40 0x7fa2880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.721+0000 7fa29d59a640 1 -- 172.21.15.83:0/3191206615 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a00b31e0 con 0x7fa288004d40 2024-03-06T07:25:55.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.721+0000 7fa29d59a640 1 -- 172.21.15.83:0/3191206615 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a00b3d60 con 0x7fa2880039b0 2024-03-06T07:25:55.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.721+0000 7fa29d59a640 1 -- 172.21.15.83:0/3191206615 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa28800b9c0 con 0x7fa2a015dfe0 2024-03-06T07:25:55.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.721+0000 7fa29ed9d640 1 --2- 172.21.15.83:0/3191206615 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa288004d40 0x7fa2880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.721+0000 7fa29e59c640 1 --2- 172.21.15.83:0/3191206615 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa2880039b0 0x7fa288003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.721+0000 7fa29dd9b640 1 --2- 172.21.15.83:0/3191206615 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa2a015dfe0 0x7fa288004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.721+0000 7fa29ed9d640 1 -- 172.21.15.83:0/3191206615 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa2a015dfe0 msgr2=0x7fa288004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:55.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.721+0000 7fa29ed9d640 1 --2- 172.21.15.83:0/3191206615 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa2a015dfe0 0x7fa288004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.721+0000 7fa29ed9d640 1 -- 172.21.15.83:0/3191206615 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa2880039b0 msgr2=0x7fa288003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:55.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.721+0000 7fa29ed9d640 1 --2- 172.21.15.83:0/3191206615 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa2880039b0 0x7fa288003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.721+0000 7fa29ed9d640 1 -- 172.21.15.83:0/3191206615 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa288003520 con 0x7fa288004d40 2024-03-06T07:25:55.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.722+0000 7fa29ed9d640 1 --2- 172.21.15.83:0/3191206615 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa288004d40 0x7fa2880051c0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fa290009fc0 tx=0x7fa288005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13d064e3ddde2be2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.722+0000 7fa29d59a640 1 -- 172.21.15.83:0/3191206615 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa290039070 con 0x7fa288004d40 2024-03-06T07:25:55.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.722+0000 7fa29d59a640 1 -- 172.21.15.83:0/3191206615 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa29002cc60 con 0x7fa288004d40 2024-03-06T07:25:55.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.722+0000 7fa29d59a640 1 -- 172.21.15.83:0/3191206615 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa290009c10 con 0x7fa288004d40 2024-03-06T07:25:55.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.723+0000 7fa2a4a9a640 1 -- 172.21.15.83:0/3191206615 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa288004d40 msgr2=0x7fa2880051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.723+0000 7fa2a4a9a640 1 --2- 172.21.15.83:0/3191206615 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa288004d40 0x7fa2880051c0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fa290009fc0 tx=0x7fa288005830 comp rx=0 tx=0).stop 2024-03-06T07:25:55.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.723+0000 7fa29ed9d640 1 -- 172.21.15.83:0/3191206615 reap_dead start 2024-03-06T07:25:55.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.723+0000 7fa2a4a9a640 1 -- 172.21.15.83:0/3191206615 shutdown_connections 2024-03-06T07:25:55.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.723+0000 7fa2a4a9a640 1 -- 172.21.15.83:0/3191206615 >> 172.21.15.83:0/3191206615 conn(0x7fa2a00d3cd0 msgr2=0x7fa2a00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:55.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.723+0000 7fa2a4a9a640 1 -- 172.21.15.83:0/3191206615 shutdown_connections 2024-03-06T07:25:55.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.723+0000 7fa2a4a9a640 1 -- 172.21.15.83:0/3191206615 wait complete. 2024-03-06T07:25:55.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.724+0000 7fa2a4a9a640 1 Processor -- start 2024-03-06T07:25:55.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.724+0000 7fa2a4a9a640 1 -- start start 2024-03-06T07:25:55.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.725+0000 7fa2a4a9a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa288004d40 0x7fa2a01596d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.725+0000 7fa2a4a9a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa2a01571f0 0x7fa2a00d40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.725+0000 7fa2a4a9a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa2a00d4620 0x7fa2a01579e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.725+0000 7fa2a4a9a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a00d1df0 con 0x7fa288004d40 2024-03-06T07:25:55.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.725+0000 7fa2a4a9a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a00d1c70 con 0x7fa2a00d4620 2024-03-06T07:25:55.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.725+0000 7fa2a4a9a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a015e3c0 con 0x7fa2a01571f0 2024-03-06T07:25:55.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.725+0000 7fa29e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa288004d40 0x7fa2a01596d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.725+0000 7fa29e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa288004d40 0x7fa2a01596d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37828/0 (socket says 172.21.15.83:37828) 2024-03-06T07:25:55.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.725+0000 7fa29e59c640 1 -- 172.21.15.83:0/3059198457 learned_addr learned my addr 172.21.15.83:0/3059198457 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:55.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.725+0000 7fa29ed9d640 1 --2- 172.21.15.83:0/3059198457 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa2a00d4620 0x7fa2a01579e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.725+0000 7fa29dd9b640 1 --2- 172.21.15.83:0/3059198457 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa2a01571f0 0x7fa2a00d40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.725+0000 7fa29e59c640 1 -- 172.21.15.83:0/3059198457 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa2a01571f0 msgr2=0x7fa2a00d40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.725+0000 7fa29e59c640 1 --2- 172.21.15.83:0/3059198457 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa2a01571f0 0x7fa2a00d40e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.725+0000 7fa29e59c640 1 -- 172.21.15.83:0/3059198457 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa2a00d4620 msgr2=0x7fa2a01579e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.725+0000 7fa29e59c640 1 --2- 172.21.15.83:0/3059198457 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa2a00d4620 0x7fa2a01579e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.725+0000 7fa29e59c640 1 -- 172.21.15.83:0/3059198457 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2a0158630 con 0x7fa288004d40 2024-03-06T07:25:55.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.725+0000 7fa29ed9d640 1 --2- 172.21.15.83:0/3059198457 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa2a00d4620 0x7fa2a01579e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:55.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.725+0000 7fa29dd9b640 1 --2- 172.21.15.83:0/3059198457 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa2a01571f0 0x7fa2a00d40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:55.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.725+0000 7fa29e59c640 1 --2- 172.21.15.83:0/3059198457 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa288004d40 0x7fa2a01596d0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fa298011910 tx=0x7fa298011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.726+0000 7fa28effd640 1 -- 172.21.15.83:0/3059198457 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa29800faf0 con 0x7fa288004d40 2024-03-06T07:25:55.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.726+0000 7fa28effd640 1 -- 172.21.15.83:0/3059198457 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa29800fc90 con 0x7fa288004d40 2024-03-06T07:25:55.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.726+0000 7fa2a4a9a640 1 -- 172.21.15.83:0/3059198457 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2a0158910 con 0x7fa288004d40 2024-03-06T07:25:55.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.726+0000 7fa28effd640 1 -- 172.21.15.83:0/3059198457 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa298014440 con 0x7fa288004d40 2024-03-06T07:25:55.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.726+0000 7fa2a4a9a640 1 -- 172.21.15.83:0/3059198457 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2a0158ed0 con 0x7fa288004d40 2024-03-06T07:25:55.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.727+0000 7fa2a4a9a640 1 -- 172.21.15.83:0/3059198457 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2a00d4a30 con 0x7fa288004d40 2024-03-06T07:25:55.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.727+0000 7fa28effd640 1 -- 172.21.15.83:0/3059198457 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa298012040 con 0x7fa288004d40 2024-03-06T07:25:55.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.728+0000 7fa28effd640 1 --2- 172.21.15.83:0/3059198457 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa28006b4d0 0x7fa28006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.728+0000 7fa28effd640 1 -- 172.21.15.83:0/3059198457 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa29808a860 con 0x7fa288004d40 2024-03-06T07:25:55.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.728+0000 7fa29dd9b640 1 --2- 172.21.15.83:0/3059198457 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa28006b4d0 0x7fa28006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.728+0000 7fa29dd9b640 1 --2- 172.21.15.83:0/3059198457 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa28006b4d0 0x7fa28006d990 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fa294001eb0 tx=0x7fa294001cb0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.732+0000 7fa28effd640 1 -- 172.21.15.83:0/3059198457 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa2a00d4a30 con 0x7fa288004d40 2024-03-06T07:25:55.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.905+0000 7fa2a4a9a640 1 -- 172.21.15.83:0/3059198457 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa2a00b1e10 con 0x7fa288004d40 2024-03-06T07:25:55.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.907+0000 7fa28effd640 1 -- 172.21.15.83:0/3059198457 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16685 (secure 0 0 0) 0x7fa29808a070 con 0x7fa288004d40 2024-03-06T07:25:55.909 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:25:55.909 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":81,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:25:54.631655+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:52.856525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:53.598434+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:55.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.908+0000 7fa2a4a9a640 1 -- 172.21.15.83:0/3059198457 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa28006b4d0 msgr2=0x7fa28006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.908+0000 7fa2a4a9a640 1 --2- 172.21.15.83:0/3059198457 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa28006b4d0 0x7fa28006d990 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fa294001eb0 tx=0x7fa294001cb0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.908+0000 7fa2a4a9a640 1 -- 172.21.15.83:0/3059198457 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa288004d40 msgr2=0x7fa2a01596d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.908+0000 7fa2a4a9a640 1 --2- 172.21.15.83:0/3059198457 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa288004d40 0x7fa2a01596d0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fa298011910 tx=0x7fa298011dd0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.908+0000 7fa2a4a9a640 1 -- 172.21.15.83:0/3059198457 shutdown_connections 2024-03-06T07:25:55.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.908+0000 7fa2a4a9a640 1 --2- 172.21.15.83:0/3059198457 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa28006b4d0 0x7fa28006d990 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.908+0000 7fa2a4a9a640 1 --2- 172.21.15.83:0/3059198457 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa2a00d4620 0x7fa2a01579e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.908+0000 7fa2a4a9a640 1 --2- 172.21.15.83:0/3059198457 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa2a01571f0 0x7fa2a00d40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.908+0000 7fa2a4a9a640 1 --2- 172.21.15.83:0/3059198457 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa288004d40 0x7fa2a01596d0 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.909+0000 7fa2a4a9a640 1 -- 172.21.15.83:0/3059198457 >> 172.21.15.83:0/3059198457 conn(0x7fa2a00d3cd0 msgr2=0x7fa2a00b29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:55.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.909+0000 7fa2a4a9a640 1 -- 172.21.15.83:0/3059198457 shutdown_connections 2024-03-06T07:25:55.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:55.909+0000 7fa2a4a9a640 1 -- 172.21.15.83:0/3059198457 wait complete. 2024-03-06T07:25:55.922 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:25:56.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.023+0000 7f94140b5640 1 Processor -- start 2024-03-06T07:25:56.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.023+0000 7f94140b5640 1 -- start start 2024-03-06T07:25:56.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.024+0000 7f94140b5640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f940c0d06e0 0x7f940c15fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.024+0000 7f94140b5640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f940c0d0ac0 0x7f940c0b47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.024+0000 7f94140b5640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f940c0d47f0 con 0x7f940c0b4d10 2024-03-06T07:25:56.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.024+0000 7f94140b5640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f940c0ced70 con 0x7f940c0d0ac0 2024-03-06T07:25:56.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.024+0000 7f94140b5640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f940c0d1ab0 con 0x7f940c0d06e0 2024-03-06T07:25:56.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.024+0000 7f9411650640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f940c0d0ac0 0x7f940c0b47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.024+0000 7f9411650640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f940c0d0ac0 0x7f940c0b47d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37830/0 (socket says 172.21.15.83:37830) 2024-03-06T07:25:56.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.024+0000 7f9411650640 1 -- 172.21.15.83:0/4104617273 learned_addr learned my addr 172.21.15.83:0/4104617273 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:56.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.025+0000 7f9411e51640 1 --2- 172.21.15.83:0/4104617273 >> v2:172.21.15.183:3300/0 conn(0x7f940c0d06e0 0x7f940c15fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.025+0000 7f9411650640 1 -- 172.21.15.83:0/4104617273 >> v2:172.21.15.183:3300/0 conn(0x7f940c0d06e0 msgr2=0x7f940c15fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:56.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.025+0000 7f9411650640 1 --2- 172.21.15.83:0/4104617273 >> v2:172.21.15.183:3300/0 conn(0x7f940c0d06e0 0x7f940c15fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.025+0000 7f9411650640 1 -- 172.21.15.83:0/4104617273 >> v1:172.21.15.183:6789/0 conn(0x7f940c0b4d10 legacy=0x7f940c0cf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:56.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.025+0000 7f9411650640 1 -- 172.21.15.83:0/4104617273 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f940c0d1d90 con 0x7f940c0d0ac0 2024-03-06T07:25:56.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.025+0000 7f9411e51640 1 --2- 172.21.15.83:0/4104617273 >> v2:172.21.15.183:3300/0 conn(0x7f940c0d06e0 0x7f940c15fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:56.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.025+0000 7f9411650640 1 --2- 172.21.15.83:0/4104617273 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f940c0d0ac0 0x7f940c0b47d0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f940c0b6850 tx=0x7f93fc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b7b23ee74fdd978 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:56.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.026+0000 7f9410e4f640 1 -- 172.21.15.83:0/4104617273 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93fc035020 con 0x7f940c0d0ac0 2024-03-06T07:25:56.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.026+0000 7f9410e4f640 1 -- 172.21.15.83:0/4104617273 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93fc031870 con 0x7f940c0d0ac0 2024-03-06T07:25:56.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.026+0000 7f9410e4f640 1 -- 172.21.15.83:0/4104617273 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93fc031c40 con 0x7f940c0d0ac0 2024-03-06T07:25:56.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.026+0000 7f94140b5640 1 -- 172.21.15.83:0/4104617273 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f940c0d0ac0 msgr2=0x7f940c0b47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.026+0000 7f94140b5640 1 --2- 172.21.15.83:0/4104617273 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f940c0d0ac0 0x7f940c0b47d0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f940c0b6850 tx=0x7f93fc02b910 comp rx=0 tx=0).stop 2024-03-06T07:25:56.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.026+0000 7f94140b5640 1 -- 172.21.15.83:0/4104617273 shutdown_connections 2024-03-06T07:25:56.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.026+0000 7f94140b5640 1 --2- 172.21.15.83:0/4104617273 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f940c0d0ac0 0x7f940c0b47d0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.026+0000 7f94140b5640 1 --2- 172.21.15.83:0/4104617273 >> v2:172.21.15.183:3300/0 conn(0x7f940c0d06e0 0x7f940c15fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.026+0000 7f94140b5640 1 -- 172.21.15.83:0/4104617273 >> 172.21.15.83:0/4104617273 conn(0x7f940c0d3cd0 msgr2=0x7f940c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:56.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.026+0000 7f94140b5640 1 -- 172.21.15.83:0/4104617273 shutdown_connections 2024-03-06T07:25:56.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.026+0000 7f94140b5640 1 -- 172.21.15.83:0/4104617273 wait complete. 2024-03-06T07:25:56.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.027+0000 7f94140b5640 1 Processor -- start 2024-03-06T07:25:56.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.027+0000 7f94140b5640 1 -- start start 2024-03-06T07:25:56.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.028+0000 7f94140b5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f940c0b4d10 0x7f940c0c8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.028+0000 7f94140b5640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f940c0d06e0 0x7f940c0b1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.028+0000 7f94140b5640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f940c0d0ac0 0x7f940c0b1e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.028+0000 7f94140b5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f940c0b44d0 con 0x7f940c0b4d10 2024-03-06T07:25:56.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.028+0000 7f94140b5640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f940c0ce990 con 0x7f940c0d0ac0 2024-03-06T07:25:56.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.028+0000 7f94140b5640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f940c0d0f30 con 0x7f940c0d06e0 2024-03-06T07:25:56.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.028+0000 7f9411e51640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f940c0b4d10 0x7f940c0c8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.028+0000 7f9411e51640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f940c0b4d10 0x7f940c0c8af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37844/0 (socket says 172.21.15.83:37844) 2024-03-06T07:25:56.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.028+0000 7f9411e51640 1 -- 172.21.15.83:0/1687165583 learned_addr learned my addr 172.21.15.83:0/1687165583 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:56.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.028+0000 7f9411e51640 1 -- 172.21.15.83:0/1687165583 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f940c0d06e0 msgr2=0x7f940c0b1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.028+0000 7f9411e51640 1 --2- 172.21.15.83:0/1687165583 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f940c0d06e0 0x7f940c0b1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.028+0000 7f9411e51640 1 -- 172.21.15.83:0/1687165583 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f940c0d0ac0 msgr2=0x7f940c0b1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.028+0000 7f9411e51640 1 --2- 172.21.15.83:0/1687165583 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f940c0d0ac0 0x7f940c0b1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.028+0000 7f9411e51640 1 -- 172.21.15.83:0/1687165583 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f940c0ca3c0 con 0x7f940c0b4d10 2024-03-06T07:25:56.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.029+0000 7f9411e51640 1 --2- 172.21.15.83:0/1687165583 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f940c0b4d10 0x7f940c0c8af0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f940800ce30 tx=0x7f940800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:56.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.029+0000 7f94027fc640 1 -- 172.21.15.83:0/1687165583 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940800f890 con 0x7f940c0b4d10 2024-03-06T07:25:56.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.029+0000 7f94027fc640 1 -- 172.21.15.83:0/1687165583 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f940800fa30 con 0x7f940c0b4d10 2024-03-06T07:25:56.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.029+0000 7f94027fc640 1 -- 172.21.15.83:0/1687165583 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940800fe00 con 0x7f940c0b4d10 2024-03-06T07:25:56.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.029+0000 7f94140b5640 1 -- 172.21.15.83:0/1687165583 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f940c0c9320 con 0x7f940c0b4d10 2024-03-06T07:25:56.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.029+0000 7f94140b5640 1 -- 172.21.15.83:0/1687165583 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f940c0c98e0 con 0x7f940c0b4d10 2024-03-06T07:25:56.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.030+0000 7f94140b5640 1 -- 172.21.15.83:0/1687165583 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f940c0a1c80 con 0x7f940c0b4d10 2024-03-06T07:25:56.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.031+0000 7f94027fc640 1 -- 172.21.15.83:0/1687165583 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9408018400 con 0x7f940c0b4d10 2024-03-06T07:25:56.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.031+0000 7f94027fc640 1 --2- 172.21.15.83:0/1687165583 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f93f006b4d0 0x7f93f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.031+0000 7f94027fc640 1 -- 172.21.15.83:0/1687165583 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f940808b060 con 0x7f940c0b4d10 2024-03-06T07:25:56.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.031+0000 7f9411650640 1 --2- 172.21.15.83:0/1687165583 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f93f006b4d0 0x7f93f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.031+0000 7f9411650640 1 --2- 172.21.15.83:0/1687165583 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f93f006b4d0 0x7f93f006d990 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f93fc036000 tx=0x7f93fc030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:56.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.035+0000 7f94027fc640 1 -- 172.21.15.83:0/1687165583 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9408014030 con 0x7f940c0b4d10 2024-03-06T07:25:56.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.243+0000 7f94140b5640 1 -- 172.21.15.83:0/1687165583 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f940c0d2b00 con 0x7f940c0b4d10 2024-03-06T07:25:56.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.292+0000 7f94027fc640 1 -- 172.21.15.83:0/1687165583 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v38) v1 ==== 118+0+0 (secure 0 0 0) 0x7f940800ae20 con 0x7f940c0b4d10 2024-03-06T07:25:56.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.294+0000 7f94140b5640 1 -- 172.21.15.83:0/1687165583 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f93f006b4d0 msgr2=0x7f93f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.294+0000 7f94140b5640 1 --2- 172.21.15.83:0/1687165583 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f93f006b4d0 0x7f93f006d990 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f93fc036000 tx=0x7f93fc030000 comp rx=0 tx=0).stop 2024-03-06T07:25:56.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.294+0000 7f94140b5640 1 -- 172.21.15.83:0/1687165583 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f940c0b4d10 msgr2=0x7f940c0c8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.294+0000 7f94140b5640 1 --2- 172.21.15.83:0/1687165583 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f940c0b4d10 0x7f940c0c8af0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f940800ce30 tx=0x7f940800a650 comp rx=0 tx=0).stop 2024-03-06T07:25:56.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.294+0000 7f94140b5640 1 -- 172.21.15.83:0/1687165583 shutdown_connections 2024-03-06T07:25:56.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.294+0000 7f94140b5640 1 --2- 172.21.15.83:0/1687165583 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f93f006b4d0 0x7f93f006d990 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.294+0000 7f94140b5640 1 --2- 172.21.15.83:0/1687165583 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f940c0d0ac0 0x7f940c0b1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.294+0000 7f94140b5640 1 --2- 172.21.15.83:0/1687165583 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f940c0d06e0 0x7f940c0b1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.294+0000 7f94140b5640 1 --2- 172.21.15.83:0/1687165583 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f940c0b4d10 0x7f940c0c8af0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.294+0000 7f94140b5640 1 -- 172.21.15.83:0/1687165583 >> 172.21.15.83:0/1687165583 conn(0x7f940c0d3cd0 msgr2=0x7f940c15e970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:56.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.294+0000 7f94140b5640 1 -- 172.21.15.83:0/1687165583 shutdown_connections 2024-03-06T07:25:56.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.294+0000 7f94140b5640 1 -- 172.21.15.83:0/1687165583 wait complete. 2024-03-06T07:25:56.307 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:25:56.307 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:25:56.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.409+0000 7f0a445d0640 1 Processor -- start 2024-03-06T07:25:56.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.409+0000 7f0a445d0640 1 -- start start 2024-03-06T07:25:56.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.410+0000 7f0a445d0640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f0a3c0d4860 0x7f0a3c0d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.410+0000 7f0a445d0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0a3c0b0e30 0x7f0a3c0b1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.410+0000 7f0a445d0640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a3c1618b0 con 0x7f0a3c1614d0 2024-03-06T07:25:56.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.410+0000 7f0a445d0640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a3c0d4cf0 con 0x7f0a3c0b0e30 2024-03-06T07:25:56.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.410+0000 7f0a445d0640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a3c0cf400 con 0x7f0a3c0d4860 2024-03-06T07:25:56.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.410+0000 7f0a42b6d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0a3c0b0e30 0x7f0a3c0b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.410+0000 7f0a4236c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f0a3c1614d0 0x7f0a3c0d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39558/0 (socket says 172.21.15.83:39558) 2024-03-06T07:25:56.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.410+0000 7f0a42b6d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0a3c0b0e30 0x7f0a3c0b1260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37856/0 (socket says 172.21.15.83:37856) 2024-03-06T07:25:56.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.410+0000 7f0a4236c640 1 -- 172.21.15.83:0/2915818918 learned_addr learned my addr 172.21.15.83:0/2915818918 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:25:56.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.410+0000 7f0a41b6b640 1 --2- 172.21.15.83:0/2915818918 >> v2:172.21.15.183:3300/0 conn(0x7f0a3c0d4860 0x7f0a3c0d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.410+0000 7f0a42b6d640 1 -- 172.21.15.83:0/2915818918 >> v2:172.21.15.183:3300/0 conn(0x7f0a3c0d4860 msgr2=0x7f0a3c0d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:56.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.410+0000 7f0a42b6d640 1 --2- 172.21.15.83:0/2915818918 >> v2:172.21.15.183:3300/0 conn(0x7f0a3c0d4860 0x7f0a3c0d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.410+0000 7f0a42b6d640 1 -- 172.21.15.83:0/2915818918 >> v1:172.21.15.83:6789/0 conn(0x7f0a3c1614d0 legacy=0x7f0a3c0d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.410+0000 7f0a4136a640 1 -- 172.21.15.83:0/2915818918 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3343542112 0 0) 0x7f0a3c1618b0 con 0x7f0a3c1614d0 2024-03-06T07:25:56.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.410+0000 7f0a42b6d640 1 -- 172.21.15.83:0/2915818918 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a3c0cf6a0 con 0x7f0a3c0b0e30 2024-03-06T07:25:56.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.410+0000 7f0a41b6b640 1 --2- 172.21.15.83:0/2915818918 >> v2:172.21.15.183:3300/0 conn(0x7f0a3c0d4860 0x7f0a3c0d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:56.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.411+0000 7f0a42b6d640 1 --2- 172.21.15.83:0/2915818918 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a3c0b0e30 0x7f0a3c0b1260 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c0ce550 tx=0x7f0a3402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd2cdd0f4419f648 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:56.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.411+0000 7f0a4136a640 1 -- 172.21.15.83:0/2915818918 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a34039070 con 0x7f0a3c0b0e30 2024-03-06T07:25:56.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.411+0000 7f0a4136a640 1 -- 172.21.15.83:0/2915818918 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a3402cc80 con 0x7f0a3c0b0e30 2024-03-06T07:25:56.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.411+0000 7f0a4136a640 1 -- 172.21.15.83:0/2915818918 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a34034660 con 0x7f0a3c0b0e30 2024-03-06T07:25:56.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.412+0000 7f0a445d0640 1 -- 172.21.15.83:0/2915818918 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a3c0b0e30 msgr2=0x7f0a3c0b1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.412+0000 7f0a445d0640 1 --2- 172.21.15.83:0/2915818918 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a3c0b0e30 0x7f0a3c0b1260 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c0ce550 tx=0x7f0a3402bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.412+0000 7f0a445d0640 1 -- 172.21.15.83:0/2915818918 shutdown_connections 2024-03-06T07:25:56.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.412+0000 7f0a445d0640 1 --2- 172.21.15.83:0/2915818918 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a3c0b0e30 0x7f0a3c0b1260 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.412+0000 7f0a445d0640 1 --2- 172.21.15.83:0/2915818918 >> v2:172.21.15.183:3300/0 conn(0x7f0a3c0d4860 0x7f0a3c0d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.412+0000 7f0a445d0640 1 -- 172.21.15.83:0/2915818918 >> 172.21.15.83:0/2915818918 conn(0x7f0a3c0d3d40 msgr2=0x7f0a3c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:56.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.412+0000 7f0a445d0640 1 -- 172.21.15.83:0/2915818918 shutdown_connections 2024-03-06T07:25:56.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.412+0000 7f0a445d0640 1 -- 172.21.15.83:0/2915818918 wait complete. 2024-03-06T07:25:56.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.413+0000 7f0a445d0640 1 Processor -- start 2024-03-06T07:25:56.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.413+0000 7f0a445d0640 1 -- start start 2024-03-06T07:25:56.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.414+0000 7f0a445d0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0a3c0b0e30 0x7f0a3c0ccf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.414+0000 7f0a445d0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a3c0d4860 0x7f0a3c14cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.414+0000 7f0a445d0640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a3c1614d0 0x7f0a3c15da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.414+0000 7f0a445d0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a3c0b0910 con 0x7f0a3c0d4860 2024-03-06T07:25:56.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.414+0000 7f0a445d0640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a3c0cfc60 con 0x7f0a3c1614d0 2024-03-06T07:25:56.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.414+0000 7f0a445d0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a3c0d2560 con 0x7f0a3c0b0e30 2024-03-06T07:25:56.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.414+0000 7f0a41b6b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a3c0d4860 0x7f0a3c14cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.414+0000 7f0a41b6b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a3c0d4860 0x7f0a3c14cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37868/0 (socket says 172.21.15.83:37868) 2024-03-06T07:25:56.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.414+0000 7f0a41b6b640 1 -- 172.21.15.83:0/1425249121 learned_addr learned my addr 172.21.15.83:0/1425249121 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:56.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.414+0000 7f0a4236c640 1 --2- 172.21.15.83:0/1425249121 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0a3c0b0e30 0x7f0a3c0ccf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.414+0000 7f0a42b6d640 1 --2- 172.21.15.83:0/1425249121 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a3c1614d0 0x7f0a3c15da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.414+0000 7f0a41b6b640 1 -- 172.21.15.83:0/1425249121 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0a3c0b0e30 msgr2=0x7f0a3c0ccf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.414+0000 7f0a41b6b640 1 --2- 172.21.15.83:0/1425249121 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0a3c0b0e30 0x7f0a3c0ccf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.414+0000 7f0a41b6b640 1 -- 172.21.15.83:0/1425249121 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a3c1614d0 msgr2=0x7f0a3c15da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.414+0000 7f0a41b6b640 1 --2- 172.21.15.83:0/1425249121 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a3c1614d0 0x7f0a3c15da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.414+0000 7f0a41b6b640 1 -- 172.21.15.83:0/1425249121 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a3c0cd6e0 con 0x7f0a3c0d4860 2024-03-06T07:25:56.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.414+0000 7f0a41b6b640 1 --2- 172.21.15.83:0/1425249121 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a3c0d4860 0x7f0a3c14cc90 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f0a3000c7f0 tx=0x7f0a3000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:56.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.414+0000 7f0a3affd640 1 -- 172.21.15.83:0/1425249121 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a30015070 con 0x7f0a3c0d4860 2024-03-06T07:25:56.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.414+0000 7f0a3affd640 1 -- 172.21.15.83:0/1425249121 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a30007a30 con 0x7f0a3c0d4860 2024-03-06T07:25:56.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.414+0000 7f0a445d0640 1 -- 172.21.15.83:0/1425249121 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a3c0cd9c0 con 0x7f0a3c0d4860 2024-03-06T07:25:56.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.414+0000 7f0a3affd640 1 -- 172.21.15.83:0/1425249121 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a30007e00 con 0x7f0a3c0d4860 2024-03-06T07:25:56.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.415+0000 7f0a445d0640 1 -- 172.21.15.83:0/1425249121 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a3c14d1d0 con 0x7f0a3c0d4860 2024-03-06T07:25:56.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.415+0000 7f0a445d0640 1 -- 172.21.15.83:0/1425249121 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a3c0d0f70 con 0x7f0a3c0d4860 2024-03-06T07:25:56.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.416+0000 7f0a3affd640 1 -- 172.21.15.83:0/1425249121 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0a30018070 con 0x7f0a3c0d4860 2024-03-06T07:25:56.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.416+0000 7f0a3affd640 1 --2- 172.21.15.83:0/1425249121 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0a2806b500 0x7f0a2806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.416+0000 7f0a3affd640 1 -- 172.21.15.83:0/1425249121 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0a3008a120 con 0x7f0a3c0d4860 2024-03-06T07:25:56.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.416+0000 7f0a4236c640 1 --2- 172.21.15.83:0/1425249121 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0a2806b500 0x7f0a2806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.417+0000 7f0a4236c640 1 --2- 172.21.15.83:0/1425249121 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0a2806b500 0x7f0a2806d9c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c000f60 tx=0x7f0a2c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:56.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.421+0000 7f0a3affd640 1 -- 172.21.15.83:0/1425249121 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0a30059030 con 0x7f0a3c0d4860 2024-03-06T07:25:56.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:56.635+0000 7f0a445d0640 1 -- 172.21.15.83:0/1425249121 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f0a3c068ff0 con 0x7f0a3c0d4860 2024-03-06T07:25:57.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.300+0000 7f0a3affd640 1 -- 172.21.15.83:0/1425249121 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v39) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0a30058930 con 0x7f0a3c0d4860 2024-03-06T07:25:57.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.303+0000 7f0a445d0640 1 -- 172.21.15.83:0/1425249121 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0a2806b500 msgr2=0x7f0a2806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.303+0000 7f0a445d0640 1 --2- 172.21.15.83:0/1425249121 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0a2806b500 0x7f0a2806d9c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c000f60 tx=0x7f0a2c009040 comp rx=0 tx=0).stop 2024-03-06T07:25:57.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.303+0000 7f0a445d0640 1 -- 172.21.15.83:0/1425249121 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a3c0d4860 msgr2=0x7f0a3c14cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.303+0000 7f0a445d0640 1 --2- 172.21.15.83:0/1425249121 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a3c0d4860 0x7f0a3c14cc90 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f0a3000c7f0 tx=0x7f0a3000cbf0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.303+0000 7f0a445d0640 1 -- 172.21.15.83:0/1425249121 shutdown_connections 2024-03-06T07:25:57.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.303+0000 7f0a445d0640 1 --2- 172.21.15.83:0/1425249121 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0a2806b500 0x7f0a2806d9c0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.303+0000 7f0a445d0640 1 --2- 172.21.15.83:0/1425249121 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a3c1614d0 0x7f0a3c15da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.303+0000 7f0a445d0640 1 --2- 172.21.15.83:0/1425249121 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a3c0d4860 0x7f0a3c14cc90 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.303+0000 7f0a445d0640 1 --2- 172.21.15.83:0/1425249121 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0a3c0b0e30 0x7f0a3c0ccf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.303+0000 7f0a445d0640 1 -- 172.21.15.83:0/1425249121 >> 172.21.15.83:0/1425249121 conn(0x7f0a3c0d3d40 msgr2=0x7f0a3c0d1910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:57.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.303+0000 7f0a445d0640 1 -- 172.21.15.83:0/1425249121 shutdown_connections 2024-03-06T07:25:57.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.303+0000 7f0a445d0640 1 -- 172.21.15.83:0/1425249121 wait complete. 2024-03-06T07:25:57.315 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:25:57.315 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:57.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.414+0000 7fcda2de9640 1 Processor -- start 2024-03-06T07:25:57.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.414+0000 7fcda2de9640 1 -- start start 2024-03-06T07:25:57.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.415+0000 7fcda2de9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcd9c1515e0 0x7fcd9c0cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.415+0000 7fcda2de9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcd9c0cd7a0 0x7fcd9c0cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.415+0000 7fcda2de9640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd9c071f40 con 0x7fcd9c15b430 2024-03-06T07:25:57.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.415+0000 7fcda2de9640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd9c15b8e0 con 0x7fcd9c1515e0 2024-03-06T07:25:57.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.415+0000 7fcda2de9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c070a70 con 0x7fcd9c0cd7a0 2024-03-06T07:25:57.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.415+0000 7fcda0b85640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcd9c1515e0 0x7fcd9c0cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.415+0000 7fcd9bfff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fcd9c15b430 0x7fcd9c071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:33074/0 (socket says 172.21.15.83:33074) 2024-03-06T07:25:57.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.415+0000 7fcd9bfff640 1 -- 172.21.15.83:0/3925035453 learned_addr learned my addr 172.21.15.83:0/3925035453 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:25:57.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.415+0000 7fcda1386640 1 --2- 172.21.15.83:0/3925035453 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcd9c0cd7a0 0x7fcd9c0cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.416+0000 7fcd9b7fe640 1 -- 172.21.15.83:0/3925035453 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3956469030 0 0) 0x7fcd9c071f40 con 0x7fcd9c15b430 2024-03-06T07:25:57.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.416+0000 7fcd9b7fe640 1 -- 172.21.15.83:0/3925035453 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd88003520 con 0x7fcd9c15b430 2024-03-06T07:25:57.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.416+0000 7fcda0b85640 1 -- 172.21.15.83:0/3925035453 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcd9c0cd7a0 msgr2=0x7fcd9c0cdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:57.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.416+0000 7fcda0b85640 1 --2- 172.21.15.83:0/3925035453 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcd9c0cd7a0 0x7fcd9c0cdbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.416+0000 7fcda0b85640 1 -- 172.21.15.83:0/3925035453 >> v1:172.21.15.83:6789/0 conn(0x7fcd9c15b430 legacy=0x7fcd9c071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.416+0000 7fcda0b85640 1 -- 172.21.15.83:0/3925035453 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd9c070d50 con 0x7fcd9c1515e0 2024-03-06T07:25:57.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.416+0000 7fcda1386640 1 --2- 172.21.15.83:0/3925035453 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcd9c0cd7a0 0x7fcd9c0cdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:57.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.416+0000 7fcda0b85640 1 --2- 172.21.15.83:0/3925035453 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd9c1515e0 0x7fcd9c0cce40 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c0097a0 tx=0x7fcd8c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fbd7cc423a10dc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:57.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.416+0000 7fcd9b7fe640 1 -- 172.21.15.83:0/3925035453 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8c035020 con 0x7fcd9c1515e0 2024-03-06T07:25:57.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.417+0000 7fcd9b7fe640 1 -- 172.21.15.83:0/3925035453 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd8c031830 con 0x7fcd9c1515e0 2024-03-06T07:25:57.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.417+0000 7fcd9b7fe640 1 -- 172.21.15.83:0/3925035453 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8c031be0 con 0x7fcd9c1515e0 2024-03-06T07:25:57.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.417+0000 7fcda2de9640 1 -- 172.21.15.83:0/3925035453 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd9c1515e0 msgr2=0x7fcd9c0cce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.417+0000 7fcda2de9640 1 --2- 172.21.15.83:0/3925035453 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd9c1515e0 0x7fcd9c0cce40 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c0097a0 tx=0x7fcd8c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.417+0000 7fcda2de9640 1 -- 172.21.15.83:0/3925035453 shutdown_connections 2024-03-06T07:25:57.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.417+0000 7fcda2de9640 1 --2- 172.21.15.83:0/3925035453 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcd9c0cd7a0 0x7fcd9c0cdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.417+0000 7fcda2de9640 1 --2- 172.21.15.83:0/3925035453 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd9c1515e0 0x7fcd9c0cce40 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.417+0000 7fcda2de9640 1 -- 172.21.15.83:0/3925035453 >> 172.21.15.83:0/3925035453 conn(0x7fcd9c0cf8a0 msgr2=0x7fcd9c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:57.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.417+0000 7fcda2de9640 1 -- 172.21.15.83:0/3925035453 shutdown_connections 2024-03-06T07:25:57.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.417+0000 7fcda2de9640 1 -- 172.21.15.83:0/3925035453 wait complete. 2024-03-06T07:25:57.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.418+0000 7fcda2de9640 1 Processor -- start 2024-03-06T07:25:57.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.418+0000 7fcda2de9640 1 -- start start 2024-03-06T07:25:57.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.419+0000 7fcda2de9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcd9c0cd7a0 0x7fcd9c06d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.419+0000 7fcda2de9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd9c1515e0 0x7fcd9c140320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.419+0000 7fcda2de9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcd9c15b430 0x7fcd9c1f0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.419+0000 7fcda2de9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c0ce0f0 con 0x7fcd9c1515e0 2024-03-06T07:25:57.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.419+0000 7fcda2de9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c071310 con 0x7fcd9c15b430 2024-03-06T07:25:57.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.419+0000 7fcda2de9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c0cc3a0 con 0x7fcd9c0cd7a0 2024-03-06T07:25:57.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.419+0000 7fcd9bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd9c1515e0 0x7fcd9c140320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.419+0000 7fcd9bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd9c1515e0 0x7fcd9c140320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55524/0 (socket says 172.21.15.83:55524) 2024-03-06T07:25:57.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.419+0000 7fcd9bfff640 1 -- 172.21.15.83:0/1721757785 learned_addr learned my addr 172.21.15.83:0/1721757785 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:57.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.419+0000 7fcda1386640 1 --2- 172.21.15.83:0/1721757785 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcd9c15b430 0x7fcd9c1f0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.419+0000 7fcda0b85640 1 --2- 172.21.15.83:0/1721757785 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcd9c0cd7a0 0x7fcd9c06d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.420+0000 7fcd9bfff640 1 -- 172.21.15.83:0/1721757785 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcd9c0cd7a0 msgr2=0x7fcd9c06d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.420+0000 7fcd9bfff640 1 --2- 172.21.15.83:0/1721757785 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcd9c0cd7a0 0x7fcd9c06d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.420+0000 7fcd9bfff640 1 -- 172.21.15.83:0/1721757785 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcd9c15b430 msgr2=0x7fcd9c1f0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.420+0000 7fcd9bfff640 1 --2- 172.21.15.83:0/1721757785 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcd9c15b430 0x7fcd9c1f0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.420+0000 7fcd9bfff640 1 -- 172.21.15.83:0/1721757785 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd9c15ee40 con 0x7fcd9c1515e0 2024-03-06T07:25:57.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.420+0000 7fcda0b85640 1 --2- 172.21.15.83:0/1721757785 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcd9c0cd7a0 0x7fcd9c06d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:57.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.420+0000 7fcda1386640 1 --2- 172.21.15.83:0/1721757785 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcd9c15b430 0x7fcd9c1f0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:57.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.420+0000 7fcd9bfff640 1 --2- 172.21.15.83:0/1721757785 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd9c1515e0 0x7fcd9c140320 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fcd84001200 tx=0x7fcd8400eb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:57.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.420+0000 7fcd997fa640 1 -- 172.21.15.83:0/1721757785 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd84015070 con 0x7fcd9c1515e0 2024-03-06T07:25:57.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.420+0000 7fcd997fa640 1 -- 172.21.15.83:0/1721757785 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd84005ce0 con 0x7fcd9c1515e0 2024-03-06T07:25:57.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.420+0000 7fcda2de9640 1 -- 172.21.15.83:0/1721757785 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd9c1455b0 con 0x7fcd9c1515e0 2024-03-06T07:25:57.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.420+0000 7fcd997fa640 1 -- 172.21.15.83:0/1721757785 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd840072b0 con 0x7fcd9c1515e0 2024-03-06T07:25:57.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.420+0000 7fcda2de9640 1 -- 172.21.15.83:0/1721757785 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd9c145df0 con 0x7fcd9c1515e0 2024-03-06T07:25:57.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.421+0000 7fcda2de9640 1 -- 172.21.15.83:0/1721757785 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd9c068840 con 0x7fcd9c1515e0 2024-03-06T07:25:57.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.421+0000 7fcd997fa640 1 -- 172.21.15.83:0/1721757785 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcd84018070 con 0x7fcd9c1515e0 2024-03-06T07:25:57.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.422+0000 7fcd997fa640 1 --2- 172.21.15.83:0/1721757785 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcd7c06b4d0 0x7fcd7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.422+0000 7fcda0b85640 1 --2- 172.21.15.83:0/1721757785 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcd7c06b4d0 0x7fcd7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.422+0000 7fcd997fa640 1 -- 172.21.15.83:0/1721757785 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcd8408c300 con 0x7fcd9c1515e0 2024-03-06T07:25:57.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.422+0000 7fcda0b85640 1 --2- 172.21.15.83:0/1721757785 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcd7c06b4d0 0x7fcd7c06d990 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c036000 tx=0x7fcd8c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:57.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.426+0000 7fcd997fa640 1 -- 172.21.15.83:0/1721757785 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcd9c1455b0 con 0x7fcd9c1515e0 2024-03-06T07:25:57.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.640+0000 7fcda2de9640 1 -- 172.21.15.83:0/1721757785 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd9c04f330 con 0x7fcd9c1515e0 2024-03-06T07:25:57.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.641+0000 7fcd997fa640 1 -- 172.21.15.83:0/1721757785 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+4982 (secure 0 0 0) 0x7fcd8405b210 con 0x7fcd9c1515e0 2024-03-06T07:25:57.643 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:25:57.643 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":39,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/2525949760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":2525949760},{"type":"v1","addr":"172.21.15.183:6834","nonce":2525949760}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35},{"gid":6262,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2104447869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2104447869},{"type":"v1","addr":"172.21.15.183:6835","nonce":2104447869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6954,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/779621621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":779621621},{"type":"v1","addr":"172.21.15.83:6836","nonce":779621621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:54.631963+0000","modified":"2024-03-06T07:25:56.637047+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6957":{"gid":6957,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/3056214366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":3056214366},{"type":"v1","addr":"172.21.15.83:6837","nonce":3056214366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6957,"qdb_cluster":[6957]},"id":6}]} 2024-03-06T07:25:57.643 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 39 2024-03-06T07:25:57.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.643+0000 7fcda2de9640 1 -- 172.21.15.83:0/1721757785 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcd7c06b4d0 msgr2=0x7fcd7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.643+0000 7fcda2de9640 1 --2- 172.21.15.83:0/1721757785 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcd7c06b4d0 0x7fcd7c06d990 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c036000 tx=0x7fcd8c008030 comp rx=0 tx=0).stop 2024-03-06T07:25:57.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.643+0000 7fcda2de9640 1 -- 172.21.15.83:0/1721757785 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd9c1515e0 msgr2=0x7fcd9c140320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.643+0000 7fcda2de9640 1 --2- 172.21.15.83:0/1721757785 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd9c1515e0 0x7fcd9c140320 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fcd84001200 tx=0x7fcd8400eb00 comp rx=0 tx=0).stop 2024-03-06T07:25:57.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.643+0000 7fcda2de9640 1 -- 172.21.15.83:0/1721757785 shutdown_connections 2024-03-06T07:25:57.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.643+0000 7fcda2de9640 1 --2- 172.21.15.83:0/1721757785 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcd7c06b4d0 0x7fcd7c06d990 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.643+0000 7fcda2de9640 1 --2- 172.21.15.83:0/1721757785 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcd9c15b430 0x7fcd9c1f0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.643+0000 7fcda2de9640 1 --2- 172.21.15.83:0/1721757785 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd9c1515e0 0x7fcd9c140320 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.643+0000 7fcda2de9640 1 --2- 172.21.15.83:0/1721757785 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcd9c0cd7a0 0x7fcd9c06d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.643+0000 7fcda2de9640 1 -- 172.21.15.83:0/1721757785 >> 172.21.15.83:0/1721757785 conn(0x7fcd9c0cf8a0 msgr2=0x7fcd9c0b3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:57.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.643+0000 7fcda2de9640 1 -- 172.21.15.83:0/1721757785 shutdown_connections 2024-03-06T07:25:57.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.643+0000 7fcda2de9640 1 -- 172.21.15.83:0/1721757785 wait complete. 2024-03-06T07:25:57.656 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:57.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.755+0000 7f607cb5e640 1 Processor -- start 2024-03-06T07:25:57.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.755+0000 7f607cb5e640 1 -- start start 2024-03-06T07:25:57.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.756+0000 7f607cb5e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f60780d22b0 0x7f60780d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.756+0000 7f607cb5e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60780b3430 con 0x7f607815dfe0 2024-03-06T07:25:57.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.756+0000 7f607cb5e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60780b3710 con 0x7f60780b41f0 2024-03-06T07:25:57.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.756+0000 7f607cb5e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60780d2c50 con 0x7f60780d22b0 2024-03-06T07:25:57.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.756+0000 7f606ffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f60780d22b0 0x7f60780d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.756+0000 7f607659c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f607815dfe0 0x7f60780cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:33076/0 (socket says 172.21.15.83:33076) 2024-03-06T07:25:57.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.756+0000 7f606ffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f60780d22b0 0x7f60780d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55532/0 (socket says 172.21.15.83:55532) 2024-03-06T07:25:57.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.756+0000 7f607659c640 1 -- 172.21.15.83:0/2952951468 learned_addr learned my addr 172.21.15.83:0/2952951468 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:25:57.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.756+0000 7f606ffff640 1 -- 172.21.15.83:0/2952951468 >> v1:172.21.15.183:6789/0 conn(0x7f60780b41f0 legacy=0x7f60780b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:57.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.756+0000 7f606ffff640 1 -- 172.21.15.83:0/2952951468 >> v1:172.21.15.83:6789/0 conn(0x7f607815dfe0 legacy=0x7f60780cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.756+0000 7f606ffff640 1 -- 172.21.15.83:0/2952951468 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60780b39f0 con 0x7f60780d22b0 2024-03-06T07:25:57.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.756+0000 7f6075d9b640 1 -- 172.21.15.83:0/2952951468 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1343878655 0 0) 0x7f60780b3430 con 0x7f607815dfe0 2024-03-06T07:25:57.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.757+0000 7f606ffff640 1 --2- 172.21.15.83:0/2952951468 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60780d22b0 0x7f60780d2710 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f60600097d0 tx=0x7f606002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba5a36b28813f370 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:57.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.757+0000 7f6075d9b640 1 -- 172.21.15.83:0/2952951468 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6060035020 con 0x7f60780d22b0 2024-03-06T07:25:57.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.757+0000 7f6075d9b640 1 -- 172.21.15.83:0/2952951468 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6060031830 con 0x7f60780d22b0 2024-03-06T07:25:57.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.757+0000 7f6075d9b640 1 -- 172.21.15.83:0/2952951468 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6060031c00 con 0x7f60780d22b0 2024-03-06T07:25:57.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.758+0000 7f607cb5e640 1 -- 172.21.15.83:0/2952951468 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60780d22b0 msgr2=0x7f60780d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.758+0000 7f607cb5e640 1 --2- 172.21.15.83:0/2952951468 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60780d22b0 0x7f60780d2710 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f60600097d0 tx=0x7f606002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.758+0000 7f607cb5e640 1 -- 172.21.15.83:0/2952951468 shutdown_connections 2024-03-06T07:25:57.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.758+0000 7f607cb5e640 1 --2- 172.21.15.83:0/2952951468 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60780d22b0 0x7f60780d2710 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.758+0000 7f607cb5e640 1 -- 172.21.15.83:0/2952951468 >> 172.21.15.83:0/2952951468 conn(0x7f60780d3cd0 msgr2=0x7f60780d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:57.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.758+0000 7f607cb5e640 1 -- 172.21.15.83:0/2952951468 shutdown_connections 2024-03-06T07:25:57.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.758+0000 7f607cb5e640 1 -- 172.21.15.83:0/2952951468 wait complete. 2024-03-06T07:25:57.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.759+0000 7f607cb5e640 1 Processor -- start 2024-03-06T07:25:57.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.759+0000 7f607cb5e640 1 -- start start 2024-03-06T07:25:57.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.760+0000 7f607cb5e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f60780b41f0 0x7f6078156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.760+0000 7f607cb5e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f60780d22b0 0x7f60780b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.760+0000 7f607cb5e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f607815dfe0 0x7f60780b84a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.760+0000 7f607cb5e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60780b3d60 con 0x7f607815dfe0 2024-03-06T07:25:57.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.760+0000 7f607cb5e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60780b31e0 con 0x7f60780b41f0 2024-03-06T07:25:57.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.760+0000 7f607cb5e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60780d1c70 con 0x7f60780d22b0 2024-03-06T07:25:57.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.760+0000 7f6076d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f607815dfe0 0x7f60780b84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.760+0000 7f6076d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f607815dfe0 0x7f60780b84a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55546/0 (socket says 172.21.15.83:55546) 2024-03-06T07:25:57.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.760+0000 7f6076d9d640 1 -- 172.21.15.83:0/2298573121 learned_addr learned my addr 172.21.15.83:0/2298573121 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:57.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.760+0000 7f6076d9d640 1 -- 172.21.15.83:0/2298573121 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f60780d22b0 msgr2=0x7f60780b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.760+0000 7f6076d9d640 1 --2- 172.21.15.83:0/2298573121 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f60780d22b0 0x7f60780b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.760+0000 7f6076d9d640 1 -- 172.21.15.83:0/2298573121 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f60780b41f0 msgr2=0x7f6078156dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.760+0000 7f607659c640 1 --2- 172.21.15.83:0/2298573121 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f60780b41f0 0x7f6078156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.760+0000 7f6076d9d640 1 --2- 172.21.15.83:0/2298573121 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f60780b41f0 0x7f6078156dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.760+0000 7f6076d9d640 1 -- 172.21.15.83:0/2298573121 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6078157800 con 0x7f607815dfe0 2024-03-06T07:25:57.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.760+0000 7f6076d9d640 1 --2- 172.21.15.83:0/2298573121 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f607815dfe0 0x7f60780b84a0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f607000c780 tx=0x7f607000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:57.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.761+0000 7f606effd640 1 -- 172.21.15.83:0/2298573121 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6070017070 con 0x7f607815dfe0 2024-03-06T07:25:57.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.761+0000 7f606effd640 1 -- 172.21.15.83:0/2298573121 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6070003da0 con 0x7f607815dfe0 2024-03-06T07:25:57.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.761+0000 7f607cb5e640 1 -- 172.21.15.83:0/2298573121 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60780b93b0 con 0x7f607815dfe0 2024-03-06T07:25:57.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.761+0000 7f606effd640 1 -- 172.21.15.83:0/2298573121 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60700052b0 con 0x7f607815dfe0 2024-03-06T07:25:57.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.761+0000 7f607cb5e640 1 -- 172.21.15.83:0/2298573121 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60780b9970 con 0x7f607815dfe0 2024-03-06T07:25:57.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.762+0000 7f607cb5e640 1 -- 172.21.15.83:0/2298573121 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f607804f330 con 0x7f607815dfe0 2024-03-06T07:25:57.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.763+0000 7f606effd640 1 -- 172.21.15.83:0/2298573121 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f607001a070 con 0x7f607815dfe0 2024-03-06T07:25:57.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.763+0000 7f606effd640 1 --2- 172.21.15.83:0/2298573121 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f605806b4d0 0x7f605806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.763+0000 7f606effd640 1 -- 172.21.15.83:0/2298573121 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6070089910 con 0x7f607815dfe0 2024-03-06T07:25:57.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.763+0000 7f607659c640 1 --2- 172.21.15.83:0/2298573121 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f605806b4d0 0x7f605806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.764+0000 7f607659c640 1 --2- 172.21.15.83:0/2298573121 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f605806b4d0 0x7f605806d990 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f6068000f60 tx=0x7f6068009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:57.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.767+0000 7f606effd640 1 -- 172.21.15.83:0/2298573121 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6078157800 con 0x7f607815dfe0 2024-03-06T07:25:57.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.945+0000 7f607cb5e640 1 -- 172.21.15.83:0/2298573121 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f607815d3a0 con 0x7f607815dfe0 2024-03-06T07:25:57.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.946+0000 7f606effd640 1 -- 172.21.15.83:0/2298573121 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16672 (secure 0 0 0) 0x7f6070058700 con 0x7f607815dfe0 2024-03-06T07:25:57.948 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:25:57.949 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:25:56.279928+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:52.856525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:53.598434+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:57.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.948+0000 7f607cb5e640 1 -- 172.21.15.83:0/2298573121 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f605806b4d0 msgr2=0x7f605806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.948+0000 7f607cb5e640 1 --2- 172.21.15.83:0/2298573121 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f605806b4d0 0x7f605806d990 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f6068000f60 tx=0x7f6068009040 comp rx=0 tx=0).stop 2024-03-06T07:25:57.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.948+0000 7f607cb5e640 1 -- 172.21.15.83:0/2298573121 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f607815dfe0 msgr2=0x7f60780b84a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.948+0000 7f607cb5e640 1 --2- 172.21.15.83:0/2298573121 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f607815dfe0 0x7f60780b84a0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f607000c780 tx=0x7f607000cc40 comp rx=0 tx=0).stop 2024-03-06T07:25:57.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.948+0000 7f607cb5e640 1 -- 172.21.15.83:0/2298573121 shutdown_connections 2024-03-06T07:25:57.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.948+0000 7f607cb5e640 1 --2- 172.21.15.83:0/2298573121 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f605806b4d0 0x7f605806d990 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.948+0000 7f607cb5e640 1 --2- 172.21.15.83:0/2298573121 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f607815dfe0 0x7f60780b84a0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.948+0000 7f607cb5e640 1 --2- 172.21.15.83:0/2298573121 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f60780d22b0 0x7f60780b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.948+0000 7f607cb5e640 1 --2- 172.21.15.83:0/2298573121 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f60780b41f0 0x7f6078156dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.948+0000 7f607cb5e640 1 -- 172.21.15.83:0/2298573121 >> 172.21.15.83:0/2298573121 conn(0x7f60780d3cd0 msgr2=0x7f607806ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:57.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.948+0000 7f607cb5e640 1 -- 172.21.15.83:0/2298573121 shutdown_connections 2024-03-06T07:25:57.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:57.948+0000 7f607cb5e640 1 -- 172.21.15.83:0/2298573121 wait complete. 2024-03-06T07:25:57.964 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:25:57.965 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:58.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.063+0000 7f30435bc640 1 Processor -- start 2024-03-06T07:25:58.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.063+0000 7f30435bc640 1 -- start start 2024-03-06T07:25:58.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.064+0000 7f30435bc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f303c15dfe0 0x7f303c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.064+0000 7f30435bc640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f303c0d29c0 0x7f303c0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.064+0000 7f30435bc640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f303c0b3430 con 0x7f303c0cf830 2024-03-06T07:25:58.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.064+0000 7f30435bc640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f303c0b31e0 con 0x7f303c15dfe0 2024-03-06T07:25:58.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.064+0000 7f30435bc640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f303c0b3d60 con 0x7f303c0d29c0 2024-03-06T07:25:58.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.065+0000 7f3041358640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f303c15dfe0 0x7f303c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.065+0000 7f3041358640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f303c15dfe0 0x7f303c0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55556/0 (socket says 172.21.15.83:55556) 2024-03-06T07:25:58.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.065+0000 7f3041358640 1 -- 172.21.15.83:0/553882800 learned_addr learned my addr 172.21.15.83:0/553882800 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:58.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.065+0000 7f3041358640 1 -- 172.21.15.83:0/553882800 >> v2:172.21.15.183:3300/0 conn(0x7f303c0d29c0 msgr2=0x7f303c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:58.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.065+0000 7f3041b59640 1 --2- 172.21.15.83:0/553882800 >> v2:172.21.15.183:3300/0 conn(0x7f303c0d29c0 0x7f303c0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.065+0000 7f3041358640 1 --2- 172.21.15.83:0/553882800 >> v2:172.21.15.183:3300/0 conn(0x7f303c0d29c0 0x7f303c0b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.065+0000 7f3041358640 1 -- 172.21.15.83:0/553882800 >> v1:172.21.15.183:6789/0 conn(0x7f303c0cf830 legacy=0x7f303c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:58.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.065+0000 7f3041358640 1 -- 172.21.15.83:0/553882800 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f303c0b3710 con 0x7f303c15dfe0 2024-03-06T07:25:58.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.065+0000 7f3041b59640 1 --2- 172.21.15.83:0/553882800 >> v2:172.21.15.183:3300/0 conn(0x7f303c0d29c0 0x7f303c0b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:58.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.066+0000 7f3041358640 1 --2- 172.21.15.83:0/553882800 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f303c15dfe0 0x7f303c0cf2f0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f303c0b3cd0 tx=0x7f303802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fd028bfd70a94a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:58.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.066+0000 7f3033fff640 1 -- 172.21.15.83:0/553882800 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3038035020 con 0x7f303c15dfe0 2024-03-06T07:25:58.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.066+0000 7f3033fff640 1 -- 172.21.15.83:0/553882800 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3038031870 con 0x7f303c15dfe0 2024-03-06T07:25:58.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.066+0000 7f3033fff640 1 -- 172.21.15.83:0/553882800 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3038031c40 con 0x7f303c15dfe0 2024-03-06T07:25:58.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.066+0000 7f30435bc640 1 -- 172.21.15.83:0/553882800 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f303c15dfe0 msgr2=0x7f303c0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.066+0000 7f30435bc640 1 --2- 172.21.15.83:0/553882800 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f303c15dfe0 0x7f303c0cf2f0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f303c0b3cd0 tx=0x7f303802b910 comp rx=0 tx=0).stop 2024-03-06T07:25:58.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.066+0000 7f30435bc640 1 -- 172.21.15.83:0/553882800 shutdown_connections 2024-03-06T07:25:58.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.066+0000 7f30435bc640 1 --2- 172.21.15.83:0/553882800 >> v2:172.21.15.183:3300/0 conn(0x7f303c0d29c0 0x7f303c0b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.066+0000 7f30435bc640 1 --2- 172.21.15.83:0/553882800 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f303c15dfe0 0x7f303c0cf2f0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.066+0000 7f30435bc640 1 -- 172.21.15.83:0/553882800 >> 172.21.15.83:0/553882800 conn(0x7f303c0d3cd0 msgr2=0x7f303c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:58.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.066+0000 7f30435bc640 1 -- 172.21.15.83:0/553882800 shutdown_connections 2024-03-06T07:25:58.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.066+0000 7f30435bc640 1 -- 172.21.15.83:0/553882800 wait complete. 2024-03-06T07:25:58.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.068+0000 7f30435bc640 1 Processor -- start 2024-03-06T07:25:58.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.068+0000 7f30435bc640 1 -- start start 2024-03-06T07:25:58.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.069+0000 7f30435bc640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f303c0cf830 0x7f303c1531d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.069+0000 7f30435bc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f303c0d29c0 0x7f303c0ce7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.069+0000 7f30435bc640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f303c15dfe0 0x7f303c0cece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.069+0000 7f30435bc640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f303c0d1df0 con 0x7f303c0d29c0 2024-03-06T07:25:58.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.069+0000 7f30435bc640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f303c0d1c70 con 0x7f303c0cf830 2024-03-06T07:25:58.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.069+0000 7f30435bc640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f303c15e3c0 con 0x7f303c15dfe0 2024-03-06T07:25:58.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.070+0000 7f3040b57640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f303c0d29c0 0x7f303c0ce7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.070+0000 7f3040b57640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f303c0d29c0 0x7f303c0ce7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55568/0 (socket says 172.21.15.83:55568) 2024-03-06T07:25:58.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.070+0000 7f3040b57640 1 -- 172.21.15.83:0/2139396080 learned_addr learned my addr 172.21.15.83:0/2139396080 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:58.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.070+0000 7f3041358640 1 --2- 172.21.15.83:0/2139396080 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f303c0cf830 0x7f303c1531d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.070+0000 7f3041b59640 1 --2- 172.21.15.83:0/2139396080 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f303c15dfe0 0x7f303c0cece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.070+0000 7f3040b57640 1 -- 172.21.15.83:0/2139396080 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f303c15dfe0 msgr2=0x7f303c0cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.070+0000 7f3040b57640 1 --2- 172.21.15.83:0/2139396080 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f303c15dfe0 0x7f303c0cece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.070+0000 7f3040b57640 1 -- 172.21.15.83:0/2139396080 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f303c0cf830 msgr2=0x7f303c1531d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.070+0000 7f3040b57640 1 --2- 172.21.15.83:0/2139396080 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f303c0cf830 0x7f303c1531d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.070+0000 7f3040b57640 1 -- 172.21.15.83:0/2139396080 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f303c15c980 con 0x7f303c0d29c0 2024-03-06T07:25:58.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.070+0000 7f3041358640 1 --2- 172.21.15.83:0/2139396080 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f303c0cf830 0x7f303c1531d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:58.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.070+0000 7f3040b57640 1 --2- 172.21.15.83:0/2139396080 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f303c0d29c0 0x7f303c0ce7a0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f302c00c400 tx=0x7f302c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:58.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.070+0000 7f3031ffb640 1 -- 172.21.15.83:0/2139396080 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f302c011790 con 0x7f303c0d29c0 2024-03-06T07:25:58.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.070+0000 7f3031ffb640 1 -- 172.21.15.83:0/2139396080 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f302c017070 con 0x7f303c0d29c0 2024-03-06T07:25:58.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.070+0000 7f30435bc640 1 -- 172.21.15.83:0/2139396080 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f303c0b7db0 con 0x7f303c0d29c0 2024-03-06T07:25:58.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.070+0000 7f3031ffb640 1 -- 172.21.15.83:0/2139396080 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f302c011bb0 con 0x7f303c0d29c0 2024-03-06T07:25:58.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.070+0000 7f30435bc640 1 -- 172.21.15.83:0/2139396080 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f303c0b8370 con 0x7f303c0d29c0 2024-03-06T07:25:58.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.071+0000 7f30435bc640 1 -- 172.21.15.83:0/2139396080 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f303c04f330 con 0x7f303c0d29c0 2024-03-06T07:25:58.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.072+0000 7f3031ffb640 1 -- 172.21.15.83:0/2139396080 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f302c01c460 con 0x7f303c0d29c0 2024-03-06T07:25:58.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.072+0000 7f3031ffb640 1 --2- 172.21.15.83:0/2139396080 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f302006b4d0 0x7f302006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.072+0000 7f3041358640 1 --2- 172.21.15.83:0/2139396080 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f302006b4d0 0x7f302006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.073+0000 7f3031ffb640 1 -- 172.21.15.83:0/2139396080 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f302c08a050 con 0x7f303c0d29c0 2024-03-06T07:25:58.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.073+0000 7f3041358640 1 --2- 172.21.15.83:0/2139396080 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f302006b4d0 0x7f302006d990 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f3038036000 tx=0x7f303803c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:58.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.077+0000 7f3031ffb640 1 -- 172.21.15.83:0/2139396080 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f302c058ed0 con 0x7f303c0d29c0 2024-03-06T07:25:58.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.245+0000 7f30435bc640 1 -- 172.21.15.83:0/2139396080 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f303c0b0f10 con 0x7f302006b4d0 2024-03-06T07:25:58.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.251+0000 7f3031ffb640 1 -- 172.21.15.83:0/2139396080 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252004 (secure 0 0 0) 0x7f303c0b0f10 con 0x7f302006b4d0 2024-03-06T07:25:58.255 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:25:58.256 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-06T07:25:58.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.253+0000 7f30435bc640 1 -- 172.21.15.83:0/2139396080 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f302006b4d0 msgr2=0x7f302006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.253+0000 7f30435bc640 1 --2- 172.21.15.83:0/2139396080 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f302006b4d0 0x7f302006d990 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f3038036000 tx=0x7f303803c040 comp rx=0 tx=0).stop 2024-03-06T07:25:58.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.253+0000 7f30435bc640 1 -- 172.21.15.83:0/2139396080 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f303c0d29c0 msgr2=0x7f303c0ce7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.253+0000 7f30435bc640 1 --2- 172.21.15.83:0/2139396080 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f303c0d29c0 0x7f303c0ce7a0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f302c00c400 tx=0x7f302c00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.254+0000 7f30435bc640 1 -- 172.21.15.83:0/2139396080 shutdown_connections 2024-03-06T07:25:58.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.254+0000 7f30435bc640 1 --2- 172.21.15.83:0/2139396080 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f302006b4d0 0x7f302006d990 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.254+0000 7f30435bc640 1 --2- 172.21.15.83:0/2139396080 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f303c15dfe0 0x7f303c0cece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.254+0000 7f30435bc640 1 --2- 172.21.15.83:0/2139396080 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f303c0d29c0 0x7f303c0ce7a0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.254+0000 7f30435bc640 1 --2- 172.21.15.83:0/2139396080 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f303c0cf830 0x7f303c1531d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.254+0000 7f30435bc640 1 -- 172.21.15.83:0/2139396080 >> 172.21.15.83:0/2139396080 conn(0x7f303c0d3cd0 msgr2=0x7f303c06cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:58.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.254+0000 7f30435bc640 1 -- 172.21.15.83:0/2139396080 shutdown_connections 2024-03-06T07:25:58.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.254+0000 7f30435bc640 1 -- 172.21.15.83:0/2139396080 wait complete. 2024-03-06T07:25:58.266 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":201,"stamp":"2024-03-06T07:25:56.363342+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":67,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258328,"kb_used_data":8864,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716392,"statfs":{"total":773094113280,"available":772829585408,"internally_reserved":0,"allocated":9076736,"data_stored":4508457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12757,"internal_metadata":255315499},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000657"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.296965+0000","last_change":"2024-03-06T07:25:55.292649+0000","last_active":"2024-03-06T07:25:55.296965+0000","last_peered":"2024-03-06T07:25:55.296965+0000","last_clean":"2024-03-06T07:25:55.296965+0000","last_became_active":"2024-03-06T07:25:55.292348+0000","last_became_peered":"2024-03-06T07:25:55.292348+0000","last_unstale":"2024-03-06T07:25:55.296965+0000","last_undegraded":"2024-03-06T07:25:55.296965+0000","last_fullsized":"2024-03-06T07:25:55.296965+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:31.756662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.311939+0000","last_change":"2024-03-06T07:25:55.291769+0000","last_active":"2024-03-06T07:25:55.311939+0000","last_peered":"2024-03-06T07:25:55.311939+0000","last_clean":"2024-03-06T07:25:55.311939+0000","last_became_active":"2024-03-06T07:25:55.291490+0000","last_became_peered":"2024-03-06T07:25:55.291490+0000","last_unstale":"2024-03-06T07:25:55.311939+0000","last_undegraded":"2024-03-06T07:25:55.311939+0000","last_fullsized":"2024-03-06T07:25:55.311939+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:07:31.904278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.300901+0000","last_change":"2024-03-06T07:25:55.296596+0000","last_active":"2024-03-06T07:25:55.300901+0000","last_peered":"2024-03-06T07:25:55.300901+0000","last_clean":"2024-03-06T07:25:55.300901+0000","last_became_active":"2024-03-06T07:25:55.296244+0000","last_became_peered":"2024-03-06T07:25:55.296244+0000","last_unstale":"2024-03-06T07:25:55.300901+0000","last_undegraded":"2024-03-06T07:25:55.300901+0000","last_fullsized":"2024-03-06T07:25:55.300901+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:21.500763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.304202+0000","last_change":"2024-03-06T07:25:55.300898+0000","last_active":"2024-03-06T07:25:55.304202+0000","last_peered":"2024-03-06T07:25:55.304202+0000","last_clean":"2024-03-06T07:25:55.304202+0000","last_became_active":"2024-03-06T07:25:55.300611+0000","last_became_peered":"2024-03-06T07:25:55.300611+0000","last_unstale":"2024-03-06T07:25:55.304202+0000","last_undegraded":"2024-03-06T07:25:55.304202+0000","last_fullsized":"2024-03-06T07:25:55.304202+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:19.084617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.300392+0000","last_change":"2024-03-06T07:25:55.296794+0000","last_active":"2024-03-06T07:25:55.300392+0000","last_peered":"2024-03-06T07:25:55.300392+0000","last_clean":"2024-03-06T07:25:55.300392+0000","last_became_active":"2024-03-06T07:25:55.296514+0000","last_became_peered":"2024-03-06T07:25:55.296514+0000","last_unstale":"2024-03-06T07:25:55.300392+0000","last_undegraded":"2024-03-06T07:25:55.300392+0000","last_fullsized":"2024-03-06T07:25:55.300392+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:47.313545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.305992+0000","last_change":"2024-03-06T07:25:55.305416+0000","last_active":"2024-03-06T07:25:55.305992+0000","last_peered":"2024-03-06T07:25:55.305992+0000","last_clean":"2024-03-06T07:25:55.305992+0000","last_became_active":"2024-03-06T07:25:55.305092+0000","last_became_peered":"2024-03-06T07:25:55.305092+0000","last_unstale":"2024-03-06T07:25:55.305992+0000","last_undegraded":"2024-03-06T07:25:55.305992+0000","last_fullsized":"2024-03-06T07:25:55.305992+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:05:41.806303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.304299+0000","last_change":"2024-03-06T07:25:55.300485+0000","last_active":"2024-03-06T07:25:55.304299+0000","last_peered":"2024-03-06T07:25:55.304299+0000","last_clean":"2024-03-06T07:25:55.304299+0000","last_became_active":"2024-03-06T07:25:55.300186+0000","last_became_peered":"2024-03-06T07:25:55.300186+0000","last_unstale":"2024-03-06T07:25:55.304299+0000","last_undegraded":"2024-03-06T07:25:55.304299+0000","last_fullsized":"2024-03-06T07:25:55.304299+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:12.586950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.294414+0000","last_change":"2024-03-06T07:25:55.293481+0000","last_active":"2024-03-06T07:25:55.294414+0000","last_peered":"2024-03-06T07:25:55.294414+0000","last_clean":"2024-03-06T07:25:55.294414+0000","last_became_active":"2024-03-06T07:25:55.293205+0000","last_became_peered":"2024-03-06T07:25:55.293205+0000","last_unstale":"2024-03-06T07:25:55.294414+0000","last_undegraded":"2024-03-06T07:25:55.294414+0000","last_fullsized":"2024-03-06T07:25:55.294414+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:13.381230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.294431+0000","last_change":"2024-03-06T07:25:55.291760+0000","last_active":"2024-03-06T07:25:55.294431+0000","last_peered":"2024-03-06T07:25:55.294431+0000","last_clean":"2024-03-06T07:25:55.294431+0000","last_became_active":"2024-03-06T07:25:55.291366+0000","last_became_peered":"2024-03-06T07:25:55.291366+0000","last_unstale":"2024-03-06T07:25:55.294431+0000","last_undegraded":"2024-03-06T07:25:55.294431+0000","last_fullsized":"2024-03-06T07:25:55.294431+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:57.470091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.297045+0000","last_change":"2024-03-06T07:25:55.291368+0000","last_active":"2024-03-06T07:25:55.297045+0000","last_peered":"2024-03-06T07:25:55.297045+0000","last_clean":"2024-03-06T07:25:55.297045+0000","last_became_active":"2024-03-06T07:25:55.290947+0000","last_became_peered":"2024-03-06T07:25:55.290947+0000","last_unstale":"2024-03-06T07:25:55.297045+0000","last_undegraded":"2024-03-06T07:25:55.297045+0000","last_fullsized":"2024-03-06T07:25:55.297045+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:47:10.867024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.312025+0000","last_change":"2024-03-06T07:25:55.293123+0000","last_active":"2024-03-06T07:25:55.312025+0000","last_peered":"2024-03-06T07:25:55.312025+0000","last_clean":"2024-03-06T07:25:55.312025+0000","last_became_active":"2024-03-06T07:25:55.292780+0000","last_became_peered":"2024-03-06T07:25:55.292780+0000","last_unstale":"2024-03-06T07:25:55.312025+0000","last_undegraded":"2024-03-06T07:25:55.312025+0000","last_fullsized":"2024-03-06T07:25:55.312025+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:28:04.429768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.297072+0000","last_change":"2024-03-06T07:25:55.292088+0000","last_active":"2024-03-06T07:25:55.297072+0000","last_peered":"2024-03-06T07:25:55.297072+0000","last_clean":"2024-03-06T07:25:55.297072+0000","last_became_active":"2024-03-06T07:25:55.291583+0000","last_became_peered":"2024-03-06T07:25:55.291583+0000","last_unstale":"2024-03-06T07:25:55.297072+0000","last_undegraded":"2024-03-06T07:25:55.297072+0000","last_fullsized":"2024-03-06T07:25:55.297072+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:19:43.401290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.300699+0000","last_change":"2024-03-06T07:25:55.293447+0000","last_active":"2024-03-06T07:25:55.300699+0000","last_peered":"2024-03-06T07:25:55.300699+0000","last_clean":"2024-03-06T07:25:55.300699+0000","last_became_active":"2024-03-06T07:25:55.293167+0000","last_became_peered":"2024-03-06T07:25:55.293167+0000","last_unstale":"2024-03-06T07:25:55.300699+0000","last_undegraded":"2024-03-06T07:25:55.300699+0000","last_fullsized":"2024-03-06T07:25:55.300699+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:36:14.844243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.320157+0000","last_change":"2024-03-06T07:25:55.317724+0000","last_active":"2024-03-06T07:25:55.320157+0000","last_peered":"2024-03-06T07:25:55.320157+0000","last_clean":"2024-03-06T07:25:55.320157+0000","last_became_active":"2024-03-06T07:25:55.317392+0000","last_became_peered":"2024-03-06T07:25:55.317392+0000","last_unstale":"2024-03-06T07:25:55.320157+0000","last_undegraded":"2024-03-06T07:25:55.320157+0000","last_fullsized":"2024-03-06T07:25:55.320157+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:28:04.429768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.298763+0000","last_change":"2024-03-06T07:25:55.293332+0000","last_active":"2024-03-06T07:25:55.298763+0000","last_peered":"2024-03-06T07:25:55.298763+0000","last_clean":"2024-03-06T07:25:55.298763+0000","last_became_active":"2024-03-06T07:25:55.293082+0000","last_became_peered":"2024-03-06T07:25:55.293082+0000","last_unstale":"2024-03-06T07:25:55.298763+0000","last_undegraded":"2024-03-06T07:25:55.298763+0000","last_fullsized":"2024-03-06T07:25:55.298763+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:01:22.060797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.302751+0000","last_change":"2024-03-06T07:25:55.297008+0000","last_active":"2024-03-06T07:25:55.302751+0000","last_peered":"2024-03-06T07:25:55.302751+0000","last_clean":"2024-03-06T07:25:55.302751+0000","last_became_active":"2024-03-06T07:25:55.296714+0000","last_became_peered":"2024-03-06T07:25:55.296714+0000","last_unstale":"2024-03-06T07:25:55.302751+0000","last_undegraded":"2024-03-06T07:25:55.302751+0000","last_fullsized":"2024-03-06T07:25:55.302751+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:26:35.055387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.307634+0000","last_change":"2024-03-06T07:25:55.304655+0000","last_active":"2024-03-06T07:25:55.307634+0000","last_peered":"2024-03-06T07:25:55.307634+0000","last_clean":"2024-03-06T07:25:55.307634+0000","last_became_active":"2024-03-06T07:25:55.304358+0000","last_became_peered":"2024-03-06T07:25:55.304358+0000","last_unstale":"2024-03-06T07:25:55.307634+0000","last_undegraded":"2024-03-06T07:25:55.307634+0000","last_fullsized":"2024-03-06T07:25:55.307634+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:30:51.700835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.305719+0000","last_change":"2024-03-06T07:25:55.304037+0000","last_active":"2024-03-06T07:25:55.305719+0000","last_peered":"2024-03-06T07:25:55.305719+0000","last_clean":"2024-03-06T07:25:55.305719+0000","last_became_active":"2024-03-06T07:25:55.303698+0000","last_became_peered":"2024-03-06T07:25:55.303698+0000","last_unstale":"2024-03-06T07:25:55.305719+0000","last_undegraded":"2024-03-06T07:25:55.305719+0000","last_fullsized":"2024-03-06T07:25:55.305719+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:07:31.904278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.303768+0000","last_change":"2024-03-06T07:25:55.300658+0000","last_active":"2024-03-06T07:25:55.303768+0000","last_peered":"2024-03-06T07:25:55.303768+0000","last_clean":"2024-03-06T07:25:55.303768+0000","last_became_active":"2024-03-06T07:25:55.300409+0000","last_became_peered":"2024-03-06T07:25:55.300409+0000","last_unstale":"2024-03-06T07:25:55.303768+0000","last_undegraded":"2024-03-06T07:25:55.303768+0000","last_fullsized":"2024-03-06T07:25:55.303768+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:05:41.806303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.304650+0000","last_change":"2024-03-06T07:25:55.302928+0000","last_active":"2024-03-06T07:25:55.304650+0000","last_peered":"2024-03-06T07:25:55.304650+0000","last_clean":"2024-03-06T07:25:55.304650+0000","last_became_active":"2024-03-06T07:25:55.302641+0000","last_became_peered":"2024-03-06T07:25:55.302641+0000","last_unstale":"2024-03-06T07:25:55.304650+0000","last_undegraded":"2024-03-06T07:25:55.304650+0000","last_fullsized":"2024-03-06T07:25:55.304650+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:28:41.237481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.304225+0000","last_change":"2024-03-06T07:25:55.300880+0000","last_active":"2024-03-06T07:25:55.304225+0000","last_peered":"2024-03-06T07:25:55.304225+0000","last_clean":"2024-03-06T07:25:55.304225+0000","last_became_active":"2024-03-06T07:25:55.299911+0000","last_became_peered":"2024-03-06T07:25:55.299911+0000","last_unstale":"2024-03-06T07:25:55.304225+0000","last_undegraded":"2024-03-06T07:25:55.304225+0000","last_fullsized":"2024-03-06T07:25:55.304225+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:51.828978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.312967+0000","last_change":"2024-03-06T07:25:55.295597+0000","last_active":"2024-03-06T07:25:55.312967+0000","last_peered":"2024-03-06T07:25:55.312967+0000","last_clean":"2024-03-06T07:25:55.312967+0000","last_became_active":"2024-03-06T07:25:55.295303+0000","last_became_peered":"2024-03-06T07:25:55.295303+0000","last_unstale":"2024-03-06T07:25:55.312967+0000","last_undegraded":"2024-03-06T07:25:55.312967+0000","last_fullsized":"2024-03-06T07:25:55.312967+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:26:14.898519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.297876+0000","last_change":"2024-03-06T07:25:55.296213+0000","last_active":"2024-03-06T07:25:55.297876+0000","last_peered":"2024-03-06T07:25:55.297876+0000","last_clean":"2024-03-06T07:25:55.297876+0000","last_became_active":"2024-03-06T07:25:55.295884+0000","last_became_peered":"2024-03-06T07:25:55.295884+0000","last_unstale":"2024-03-06T07:25:55.297876+0000","last_undegraded":"2024-03-06T07:25:55.297876+0000","last_fullsized":"2024-03-06T07:25:55.297876+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:51:44.302090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.304243+0000","last_change":"2024-03-06T07:25:55.301130+0000","last_active":"2024-03-06T07:25:55.304243+0000","last_peered":"2024-03-06T07:25:55.304243+0000","last_clean":"2024-03-06T07:25:55.304243+0000","last_became_active":"2024-03-06T07:25:55.300197+0000","last_became_peered":"2024-03-06T07:25:55.300197+0000","last_unstale":"2024-03-06T07:25:55.304243+0000","last_undegraded":"2024-03-06T07:25:55.304243+0000","last_fullsized":"2024-03-06T07:25:55.304243+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:31.497366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.296098+0000","last_change":"2024-03-06T07:25:55.287535+0000","last_active":"2024-03-06T07:25:55.296098+0000","last_peered":"2024-03-06T07:25:55.296098+0000","last_clean":"2024-03-06T07:25:55.296098+0000","last_became_active":"2024-03-06T07:25:55.287237+0000","last_became_peered":"2024-03-06T07:25:55.287237+0000","last_unstale":"2024-03-06T07:25:55.296098+0000","last_undegraded":"2024-03-06T07:25:55.296098+0000","last_fullsized":"2024-03-06T07:25:55.296098+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:17.542986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.301024+0000","last_change":"2024-03-06T07:25:55.297752+0000","last_active":"2024-03-06T07:25:55.301024+0000","last_peered":"2024-03-06T07:25:55.301024+0000","last_clean":"2024-03-06T07:25:55.301024+0000","last_became_active":"2024-03-06T07:25:55.297468+0000","last_became_peered":"2024-03-06T07:25:55.297468+0000","last_unstale":"2024-03-06T07:25:55.301024+0000","last_undegraded":"2024-03-06T07:25:55.301024+0000","last_fullsized":"2024-03-06T07:25:55.301024+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:07:09.583134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.293770+0000","last_change":"2024-03-06T07:25:55.292818+0000","last_active":"2024-03-06T07:25:55.293770+0000","last_peered":"2024-03-06T07:25:55.293770+0000","last_clean":"2024-03-06T07:25:55.293770+0000","last_became_active":"2024-03-06T07:25:55.292411+0000","last_became_peered":"2024-03-06T07:25:55.292411+0000","last_unstale":"2024-03-06T07:25:55.293770+0000","last_undegraded":"2024-03-06T07:25:55.293770+0000","last_fullsized":"2024-03-06T07:25:55.293770+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:31.756662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.297090+0000","last_change":"2024-03-06T07:25:55.292902+0000","last_active":"2024-03-06T07:25:55.297090+0000","last_peered":"2024-03-06T07:25:55.297090+0000","last_clean":"2024-03-06T07:25:55.297090+0000","last_became_active":"2024-03-06T07:25:55.292170+0000","last_became_peered":"2024-03-06T07:25:55.292170+0000","last_unstale":"2024-03-06T07:25:55.297090+0000","last_undegraded":"2024-03-06T07:25:55.297090+0000","last_fullsized":"2024-03-06T07:25:55.297090+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:48.091148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.296136+0000","last_change":"2024-03-06T07:25:55.290918+0000","last_active":"2024-03-06T07:25:55.296136+0000","last_peered":"2024-03-06T07:25:55.296136+0000","last_clean":"2024-03-06T07:25:55.296136+0000","last_became_active":"2024-03-06T07:25:55.290668+0000","last_became_peered":"2024-03-06T07:25:55.290668+0000","last_unstale":"2024-03-06T07:25:55.296136+0000","last_undegraded":"2024-03-06T07:25:55.296136+0000","last_fullsized":"2024-03-06T07:25:55.296136+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:28:41.237481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.317074+0000","last_change":"2024-03-06T07:25:55.314084+0000","last_active":"2024-03-06T07:25:55.317074+0000","last_peered":"2024-03-06T07:25:55.317074+0000","last_clean":"2024-03-06T07:25:55.317074+0000","last_became_active":"2024-03-06T07:25:55.313802+0000","last_became_peered":"2024-03-06T07:25:55.313802+0000","last_unstale":"2024-03-06T07:25:55.317074+0000","last_undegraded":"2024-03-06T07:25:55.317074+0000","last_fullsized":"2024-03-06T07:25:55.317074+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:50:24.773216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.302769+0000","last_change":"2024-03-06T07:25:55.299543+0000","last_active":"2024-03-06T07:25:55.302769+0000","last_peered":"2024-03-06T07:25:55.302769+0000","last_clean":"2024-03-06T07:25:55.302769+0000","last_became_active":"2024-03-06T07:25:55.299235+0000","last_became_peered":"2024-03-06T07:25:55.299235+0000","last_unstale":"2024-03-06T07:25:55.302769+0000","last_undegraded":"2024-03-06T07:25:55.302769+0000","last_fullsized":"2024-03-06T07:25:55.302769+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:57.470091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.294659+0000","last_change":"2024-03-06T07:25:55.286654+0000","last_active":"2024-03-06T07:25:55.294659+0000","last_peered":"2024-03-06T07:25:55.294659+0000","last_clean":"2024-03-06T07:25:55.294659+0000","last_became_active":"2024-03-06T07:25:55.286367+0000","last_became_peered":"2024-03-06T07:25:55.286367+0000","last_unstale":"2024-03-06T07:25:55.294659+0000","last_undegraded":"2024-03-06T07:25:55.294659+0000","last_fullsized":"2024-03-06T07:25:55.294659+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:04:06.945268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.300307+0000","last_change":"2024-03-06T07:25:55.296703+0000","last_active":"2024-03-06T07:25:55.300307+0000","last_peered":"2024-03-06T07:25:55.300307+0000","last_clean":"2024-03-06T07:25:55.300307+0000","last_became_active":"2024-03-06T07:25:55.296417+0000","last_became_peered":"2024-03-06T07:25:55.296417+0000","last_unstale":"2024-03-06T07:25:55.300307+0000","last_undegraded":"2024-03-06T07:25:55.300307+0000","last_fullsized":"2024-03-06T07:25:55.300307+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:10:06.671054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"81'4","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.315310+0000","last_change":"2024-03-06T07:25:54.291182+0000","last_active":"2024-03-06T07:25:55.315310+0000","last_peered":"2024-03-06T07:25:55.315310+0000","last_clean":"2024-03-06T07:25:55.315310+0000","last_became_active":"2024-03-06T07:25:54.290745+0000","last_became_peered":"2024-03-06T07:25:54.290745+0000","last_unstale":"2024-03-06T07:25:55.315310+0000","last_undegraded":"2024-03-06T07:25:55.315310+0000","last_fullsized":"2024-03-06T07:25:55.315310+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:37.513716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.297991+0000","last_change":"2024-03-06T07:25:55.295155+0000","last_active":"2024-03-06T07:25:55.297991+0000","last_peered":"2024-03-06T07:25:55.297991+0000","last_clean":"2024-03-06T07:25:55.297991+0000","last_became_active":"2024-03-06T07:25:55.294875+0000","last_became_peered":"2024-03-06T07:25:55.294875+0000","last_unstale":"2024-03-06T07:25:55.297991+0000","last_undegraded":"2024-03-06T07:25:55.297991+0000","last_fullsized":"2024-03-06T07:25:55.297991+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:26:35.055387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"81'1","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.308432+0000","last_change":"2024-03-06T07:25:54.291074+0000","last_active":"2024-03-06T07:25:55.308432+0000","last_peered":"2024-03-06T07:25:55.308432+0000","last_clean":"2024-03-06T07:25:55.308432+0000","last_became_active":"2024-03-06T07:25:54.290582+0000","last_became_peered":"2024-03-06T07:25:54.290582+0000","last_unstale":"2024-03-06T07:25:55.308432+0000","last_undegraded":"2024-03-06T07:25:55.308432+0000","last_fullsized":"2024-03-06T07:25:55.308432+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:19:42.396769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.297843+0000","last_change":"2024-03-06T07:25:55.294906+0000","last_active":"2024-03-06T07:25:55.297843+0000","last_peered":"2024-03-06T07:25:55.297843+0000","last_clean":"2024-03-06T07:25:55.297843+0000","last_became_active":"2024-03-06T07:25:55.294456+0000","last_became_peered":"2024-03-06T07:25:55.294456+0000","last_unstale":"2024-03-06T07:25:55.297843+0000","last_undegraded":"2024-03-06T07:25:55.297843+0000","last_fullsized":"2024-03-06T07:25:55.297843+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:47.313545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"81'2","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.311462+0000","last_change":"2024-03-06T07:25:54.292060+0000","last_active":"2024-03-06T07:25:55.311462+0000","last_peered":"2024-03-06T07:25:55.311462+0000","last_clean":"2024-03-06T07:25:55.311462+0000","last_became_active":"2024-03-06T07:25:54.291766+0000","last_became_peered":"2024-03-06T07:25:54.291766+0000","last_unstale":"2024-03-06T07:25:55.311462+0000","last_undegraded":"2024-03-06T07:25:55.311462+0000","last_fullsized":"2024-03-06T07:25:55.311462+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:51:43.297569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.307936+0000","last_change":"2024-03-06T07:25:55.307027+0000","last_active":"2024-03-06T07:25:55.307936+0000","last_peered":"2024-03-06T07:25:55.307936+0000","last_clean":"2024-03-06T07:25:55.307936+0000","last_became_active":"2024-03-06T07:25:55.306704+0000","last_became_peered":"2024-03-06T07:25:55.306704+0000","last_unstale":"2024-03-06T07:25:55.307936+0000","last_undegraded":"2024-03-06T07:25:55.307936+0000","last_fullsized":"2024-03-06T07:25:55.307936+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:30:10.567259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"81'2","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.315218+0000","last_change":"2024-03-06T07:25:54.292709+0000","last_active":"2024-03-06T07:25:55.315218+0000","last_peered":"2024-03-06T07:25:55.315218+0000","last_clean":"2024-03-06T07:25:55.315218+0000","last_became_active":"2024-03-06T07:25:54.292462+0000","last_became_peered":"2024-03-06T07:25:54.292462+0000","last_unstale":"2024-03-06T07:25:55.315218+0000","last_undegraded":"2024-03-06T07:25:55.315218+0000","last_fullsized":"2024-03-06T07:25:55.315218+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:12.376709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.317055+0000","last_change":"2024-03-06T07:25:55.313962+0000","last_active":"2024-03-06T07:25:55.317055+0000","last_peered":"2024-03-06T07:25:55.317055+0000","last_clean":"2024-03-06T07:25:55.317055+0000","last_became_active":"2024-03-06T07:25:55.313591+0000","last_became_peered":"2024-03-06T07:25:55.313591+0000","last_unstale":"2024-03-06T07:25:55.317055+0000","last_undegraded":"2024-03-06T07:25:55.317055+0000","last_fullsized":"2024-03-06T07:25:55.317055+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:35:22.698296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"81'1","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.317207+0000","last_change":"2024-03-06T07:25:54.293197+0000","last_active":"2024-03-06T07:25:55.317207+0000","last_peered":"2024-03-06T07:25:55.317207+0000","last_clean":"2024-03-06T07:25:55.317207+0000","last_became_active":"2024-03-06T07:25:54.292915+0000","last_became_peered":"2024-03-06T07:25:54.292915+0000","last_unstale":"2024-03-06T07:25:55.317207+0000","last_undegraded":"2024-03-06T07:25:55.317207+0000","last_fullsized":"2024-03-06T07:25:55.317207+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:44.841033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.307704+0000","last_change":"2024-03-06T07:25:55.304875+0000","last_active":"2024-03-06T07:25:55.307704+0000","last_peered":"2024-03-06T07:25:55.307704+0000","last_clean":"2024-03-06T07:25:55.307704+0000","last_became_active":"2024-03-06T07:25:55.304504+0000","last_became_peered":"2024-03-06T07:25:55.304504+0000","last_unstale":"2024-03-06T07:25:55.307704+0000","last_undegraded":"2024-03-06T07:25:55.307704+0000","last_fullsized":"2024-03-06T07:25:55.307704+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:05:41.806303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"81'1","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.307737+0000","last_change":"2024-03-06T07:25:54.290635+0000","last_active":"2024-03-06T07:25:55.307737+0000","last_peered":"2024-03-06T07:25:55.307737+0000","last_clean":"2024-03-06T07:25:55.307737+0000","last_became_active":"2024-03-06T07:25:54.290357+0000","last_became_peered":"2024-03-06T07:25:54.290357+0000","last_unstale":"2024-03-06T07:25:55.307737+0000","last_undegraded":"2024-03-06T07:25:55.307737+0000","last_fullsized":"2024-03-06T07:25:55.307737+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:04:05.940747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.302682+0000","last_change":"2024-03-06T07:25:55.298515+0000","last_active":"2024-03-06T07:25:55.302682+0000","last_peered":"2024-03-06T07:25:55.302682+0000","last_clean":"2024-03-06T07:25:55.302682+0000","last_became_active":"2024-03-06T07:25:55.298237+0000","last_became_peered":"2024-03-06T07:25:55.298237+0000","last_unstale":"2024-03-06T07:25:55.302682+0000","last_undegraded":"2024-03-06T07:25:55.302682+0000","last_fullsized":"2024-03-06T07:25:55.302682+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:20.461335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.291830+0000","last_change":"2024-03-06T07:25:54.294694+0000","last_active":"2024-03-06T07:25:55.291830+0000","last_peered":"2024-03-06T07:25:55.291830+0000","last_clean":"2024-03-06T07:25:55.291830+0000","last_became_active":"2024-03-06T07:25:54.294417+0000","last_became_peered":"2024-03-06T07:25:54.294417+0000","last_unstale":"2024-03-06T07:25:55.291830+0000","last_undegraded":"2024-03-06T07:25:55.291830+0000","last_fullsized":"2024-03-06T07:25:55.291830+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:26:34.050866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.298737+0000","last_change":"2024-03-06T07:25:55.292659+0000","last_active":"2024-03-06T07:25:55.298737+0000","last_peered":"2024-03-06T07:25:55.298737+0000","last_clean":"2024-03-06T07:25:55.298737+0000","last_became_active":"2024-03-06T07:25:55.292364+0000","last_became_peered":"2024-03-06T07:25:55.292364+0000","last_unstale":"2024-03-06T07:25:55.298737+0000","last_undegraded":"2024-03-06T07:25:55.298737+0000","last_fullsized":"2024-03-06T07:25:55.298737+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:30:10.567259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.282771+0000","last_change":"2024-03-06T07:25:54.290900+0000","last_active":"2024-03-06T07:25:55.282771+0000","last_peered":"2024-03-06T07:25:55.282771+0000","last_clean":"2024-03-06T07:25:55.282771+0000","last_became_active":"2024-03-06T07:25:54.290597+0000","last_became_peered":"2024-03-06T07:25:54.290597+0000","last_unstale":"2024-03-06T07:25:55.282771+0000","last_undegraded":"2024-03-06T07:25:55.282771+0000","last_fullsized":"2024-03-06T07:25:55.282771+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:11.582429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.299203+0000","last_change":"2024-03-06T07:25:55.297343+0000","last_active":"2024-03-06T07:25:55.299203+0000","last_peered":"2024-03-06T07:25:55.299203+0000","last_clean":"2024-03-06T07:25:55.299203+0000","last_became_active":"2024-03-06T07:25:55.296919+0000","last_became_peered":"2024-03-06T07:25:55.296919+0000","last_unstale":"2024-03-06T07:25:55.299203+0000","last_undegraded":"2024-03-06T07:25:55.299203+0000","last_fullsized":"2024-03-06T07:25:55.299203+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:55:57.004976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.283857+0000","last_change":"2024-03-06T07:25:54.293844+0000","last_active":"2024-03-06T07:25:55.283857+0000","last_peered":"2024-03-06T07:25:55.283857+0000","last_clean":"2024-03-06T07:25:55.283857+0000","last_became_active":"2024-03-06T07:25:54.293582+0000","last_became_peered":"2024-03-06T07:25:54.293582+0000","last_unstale":"2024-03-06T07:25:55.283857+0000","last_undegraded":"2024-03-06T07:25:55.283857+0000","last_fullsized":"2024-03-06T07:25:55.283857+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:52:19.113453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.305684+0000","last_change":"2024-03-06T07:25:55.304754+0000","last_active":"2024-03-06T07:25:55.305684+0000","last_peered":"2024-03-06T07:25:55.305684+0000","last_clean":"2024-03-06T07:25:55.305684+0000","last_became_active":"2024-03-06T07:25:55.304416+0000","last_became_peered":"2024-03-06T07:25:55.304416+0000","last_unstale":"2024-03-06T07:25:55.305684+0000","last_undegraded":"2024-03-06T07:25:55.305684+0000","last_fullsized":"2024-03-06T07:25:55.305684+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:30:51.700835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.283013+0000","last_change":"2024-03-06T07:25:54.296247+0000","last_active":"2024-03-06T07:25:55.283013+0000","last_peered":"2024-03-06T07:25:55.283013+0000","last_clean":"2024-03-06T07:25:55.283013+0000","last_became_active":"2024-03-06T07:25:54.295983+0000","last_became_peered":"2024-03-06T07:25:54.295983+0000","last_unstale":"2024-03-06T07:25:55.283013+0000","last_undegraded":"2024-03-06T07:25:55.283013+0000","last_fullsized":"2024-03-06T07:25:55.283013+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:54.868183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.303448+0000","last_change":"2024-03-06T07:25:55.299284+0000","last_active":"2024-03-06T07:25:55.303448+0000","last_peered":"2024-03-06T07:25:55.303448+0000","last_clean":"2024-03-06T07:25:55.303448+0000","last_became_active":"2024-03-06T07:25:55.298960+0000","last_became_peered":"2024-03-06T07:25:55.298960+0000","last_unstale":"2024-03-06T07:25:55.303448+0000","last_undegraded":"2024-03-06T07:25:55.303448+0000","last_fullsized":"2024-03-06T07:25:55.303448+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:30:51.700835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"81'3","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.317228+0000","last_change":"2024-03-06T07:25:54.292027+0000","last_active":"2024-03-06T07:25:55.317228+0000","last_peered":"2024-03-06T07:25:55.317228+0000","last_clean":"2024-03-06T07:25:55.317228+0000","last_became_active":"2024-03-06T07:25:54.291681+0000","last_became_peered":"2024-03-06T07:25:54.291681+0000","last_unstale":"2024-03-06T07:25:55.317228+0000","last_undegraded":"2024-03-06T07:25:55.317228+0000","last_fullsized":"2024-03-06T07:25:55.317228+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:54.252846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.302831+0000","last_change":"2024-03-06T07:25:55.299922+0000","last_active":"2024-03-06T07:25:55.302831+0000","last_peered":"2024-03-06T07:25:55.302831+0000","last_clean":"2024-03-06T07:25:55.302831+0000","last_became_active":"2024-03-06T07:25:55.299617+0000","last_became_peered":"2024-03-06T07:25:55.299617+0000","last_unstale":"2024-03-06T07:25:55.302831+0000","last_undegraded":"2024-03-06T07:25:55.302831+0000","last_fullsized":"2024-03-06T07:25:55.302831+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:13.381230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"81'6","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.319810+0000","last_change":"2024-03-06T07:25:54.291313+0000","last_active":"2024-03-06T07:25:55.319810+0000","last_peered":"2024-03-06T07:25:55.319810+0000","last_clean":"2024-03-06T07:25:55.319810+0000","last_became_active":"2024-03-06T07:25:54.291033+0000","last_became_peered":"2024-03-06T07:25:54.291033+0000","last_unstale":"2024-03-06T07:25:55.319810+0000","last_undegraded":"2024-03-06T07:25:55.319810+0000","last_fullsized":"2024-03-06T07:25:55.319810+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:46.309024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.308971+0000","last_change":"2024-03-06T07:25:55.304419+0000","last_active":"2024-03-06T07:25:55.308971+0000","last_peered":"2024-03-06T07:25:55.308971+0000","last_clean":"2024-03-06T07:25:55.308971+0000","last_became_active":"2024-03-06T07:25:55.304096+0000","last_became_peered":"2024-03-06T07:25:55.304096+0000","last_unstale":"2024-03-06T07:25:55.308971+0000","last_undegraded":"2024-03-06T07:25:55.308971+0000","last_fullsized":"2024-03-06T07:25:55.308971+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:26.561006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"81'2","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.319953+0000","last_change":"2024-03-06T07:25:54.290233+0000","last_active":"2024-03-06T07:25:55.319953+0000","last_peered":"2024-03-06T07:25:55.319953+0000","last_clean":"2024-03-06T07:25:55.319953+0000","last_became_active":"2024-03-06T07:25:54.289942+0000","last_became_peered":"2024-03-06T07:25:54.289942+0000","last_unstale":"2024-03-06T07:25:55.319953+0000","last_undegraded":"2024-03-06T07:25:55.319953+0000","last_fullsized":"2024-03-06T07:25:55.319953+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:50.824458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.302823+0000","last_change":"2024-03-06T07:25:55.298840+0000","last_active":"2024-03-06T07:25:55.302823+0000","last_peered":"2024-03-06T07:25:55.302823+0000","last_clean":"2024-03-06T07:25:55.302823+0000","last_became_active":"2024-03-06T07:25:55.298556+0000","last_became_peered":"2024-03-06T07:25:55.298556+0000","last_unstale":"2024-03-06T07:25:55.302823+0000","last_undegraded":"2024-03-06T07:25:55.302823+0000","last_fullsized":"2024-03-06T07:25:55.302823+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:27:15.358402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.288550+0000","last_change":"2024-03-06T07:25:54.294006+0000","last_active":"2024-03-06T07:25:55.288550+0000","last_peered":"2024-03-06T07:25:55.288550+0000","last_clean":"2024-03-06T07:25:55.288550+0000","last_became_active":"2024-03-06T07:25:54.293606+0000","last_became_peered":"2024-03-06T07:25:54.293606+0000","last_unstale":"2024-03-06T07:25:55.288550+0000","last_undegraded":"2024-03-06T07:25:55.288550+0000","last_fullsized":"2024-03-06T07:25:55.288550+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:19.456815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.316886+0000","last_change":"2024-03-06T07:25:55.312501+0000","last_active":"2024-03-06T07:25:55.316886+0000","last_peered":"2024-03-06T07:25:55.316886+0000","last_clean":"2024-03-06T07:25:55.316886+0000","last_became_active":"2024-03-06T07:25:55.312229+0000","last_became_peered":"2024-03-06T07:25:55.312229+0000","last_unstale":"2024-03-06T07:25:55.316886+0000","last_undegraded":"2024-03-06T07:25:55.316886+0000","last_fullsized":"2024-03-06T07:25:55.316886+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:02:48.503986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.282982+0000","last_change":"2024-03-06T07:25:54.295701+0000","last_active":"2024-03-06T07:25:55.282982+0000","last_peered":"2024-03-06T07:25:55.282982+0000","last_clean":"2024-03-06T07:25:55.282982+0000","last_became_active":"2024-03-06T07:25:54.295359+0000","last_became_peered":"2024-03-06T07:25:54.295359+0000","last_unstale":"2024-03-06T07:25:55.282982+0000","last_undegraded":"2024-03-06T07:25:55.282982+0000","last_fullsized":"2024-03-06T07:25:55.282982+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:10:05.666533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.296244+0000","last_change":"2024-03-06T07:25:55.289553+0000","last_active":"2024-03-06T07:25:55.296244+0000","last_peered":"2024-03-06T07:25:55.296244+0000","last_clean":"2024-03-06T07:25:55.296244+0000","last_became_active":"2024-03-06T07:25:55.289273+0000","last_became_peered":"2024-03-06T07:25:55.289273+0000","last_unstale":"2024-03-06T07:25:55.296244+0000","last_undegraded":"2024-03-06T07:25:55.296244+0000","last_fullsized":"2024-03-06T07:25:55.296244+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:09.830105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"81'8","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.315188+0000","last_change":"2024-03-06T07:25:54.297143+0000","last_active":"2024-03-06T07:25:55.315188+0000","last_peered":"2024-03-06T07:25:55.315188+0000","last_clean":"2024-03-06T07:25:55.315188+0000","last_became_active":"2024-03-06T07:25:54.296886+0000","last_became_peered":"2024-03-06T07:25:54.296886+0000","last_unstale":"2024-03-06T07:25:55.315188+0000","last_undegraded":"2024-03-06T07:25:55.315188+0000","last_fullsized":"2024-03-06T07:25:55.315188+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:10:05.666533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.294795+0000","last_change":"2024-03-06T07:25:55.291671+0000","last_active":"2024-03-06T07:25:55.294795+0000","last_peered":"2024-03-06T07:25:55.294795+0000","last_clean":"2024-03-06T07:25:55.294795+0000","last_became_active":"2024-03-06T07:25:55.291395+0000","last_became_peered":"2024-03-06T07:25:55.291395+0000","last_unstale":"2024-03-06T07:25:55.294795+0000","last_undegraded":"2024-03-06T07:25:55.294795+0000","last_fullsized":"2024-03-06T07:25:55.294795+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:05.339370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"81'1","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.309053+0000","last_change":"2024-03-06T07:25:54.292182+0000","last_active":"2024-03-06T07:25:55.309053+0000","last_peered":"2024-03-06T07:25:55.309053+0000","last_clean":"2024-03-06T07:25:55.309053+0000","last_became_active":"2024-03-06T07:25:54.291904+0000","last_became_peered":"2024-03-06T07:25:54.291904+0000","last_unstale":"2024-03-06T07:25:55.309053+0000","last_undegraded":"2024-03-06T07:25:55.309053+0000","last_fullsized":"2024-03-06T07:25:55.309053+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:51.548796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.296989+0000","last_change":"2024-03-06T07:25:55.293197+0000","last_active":"2024-03-06T07:25:55.296989+0000","last_peered":"2024-03-06T07:25:55.296989+0000","last_clean":"2024-03-06T07:25:55.296989+0000","last_became_active":"2024-03-06T07:25:55.292949+0000","last_became_peered":"2024-03-06T07:25:55.292949+0000","last_unstale":"2024-03-06T07:25:55.296989+0000","last_undegraded":"2024-03-06T07:25:55.296989+0000","last_fullsized":"2024-03-06T07:25:55.296989+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:10:06.671054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"81'2","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.311538+0000","last_change":"2024-03-06T07:25:54.296445+0000","last_active":"2024-03-06T07:25:55.311538+0000","last_peered":"2024-03-06T07:25:55.311538+0000","last_clean":"2024-03-06T07:25:55.311538+0000","last_became_active":"2024-03-06T07:25:54.296169+0000","last_became_peered":"2024-03-06T07:25:54.296169+0000","last_unstale":"2024-03-06T07:25:55.311538+0000","last_undegraded":"2024-03-06T07:25:55.311538+0000","last_fullsized":"2024-03-06T07:25:55.311538+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:46.309024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.302811+0000","last_change":"2024-03-06T07:25:55.297736+0000","last_active":"2024-03-06T07:25:55.302811+0000","last_peered":"2024-03-06T07:25:55.302811+0000","last_clean":"2024-03-06T07:25:55.302811+0000","last_became_active":"2024-03-06T07:25:55.297466+0000","last_became_peered":"2024-03-06T07:25:55.297466+0000","last_unstale":"2024-03-06T07:25:55.302811+0000","last_undegraded":"2024-03-06T07:25:55.302811+0000","last_fullsized":"2024-03-06T07:25:55.302811+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:51:44.302090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.294184+0000","last_change":"2024-03-06T07:25:54.291395+0000","last_active":"2024-03-06T07:25:55.294184+0000","last_peered":"2024-03-06T07:25:55.294184+0000","last_clean":"2024-03-06T07:25:55.294184+0000","last_became_active":"2024-03-06T07:25:54.291139+0000","last_became_peered":"2024-03-06T07:25:54.291139+0000","last_unstale":"2024-03-06T07:25:55.294184+0000","last_undegraded":"2024-03-06T07:25:55.294184+0000","last_fullsized":"2024-03-06T07:25:55.294184+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:52:03.922846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.320172+0000","last_change":"2024-03-06T07:25:55.317720+0000","last_active":"2024-03-06T07:25:55.320172+0000","last_peered":"2024-03-06T07:25:55.320172+0000","last_clean":"2024-03-06T07:25:55.320172+0000","last_became_active":"2024-03-06T07:25:55.317371+0000","last_became_peered":"2024-03-06T07:25:55.317371+0000","last_unstale":"2024-03-06T07:25:55.320172+0000","last_undegraded":"2024-03-06T07:25:55.320172+0000","last_fullsized":"2024-03-06T07:25:55.320172+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:55:32.599045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"81'2","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.309368+0000","last_change":"2024-03-06T07:25:54.288785+0000","last_active":"2024-03-06T07:25:55.309368+0000","last_peered":"2024-03-06T07:25:55.309368+0000","last_clean":"2024-03-06T07:25:55.309368+0000","last_became_active":"2024-03-06T07:25:54.288529+0000","last_became_peered":"2024-03-06T07:25:54.288529+0000","last_unstale":"2024-03-06T07:25:55.309368+0000","last_undegraded":"2024-03-06T07:25:55.309368+0000","last_fullsized":"2024-03-06T07:25:55.309368+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:27:14.353882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.293885+0000","last_change":"2024-03-06T07:25:55.290064+0000","last_active":"2024-03-06T07:25:55.293885+0000","last_peered":"2024-03-06T07:25:55.293885+0000","last_clean":"2024-03-06T07:25:55.293885+0000","last_became_active":"2024-03-06T07:25:55.289798+0000","last_became_peered":"2024-03-06T07:25:55.289798+0000","last_unstale":"2024-03-06T07:25:55.293885+0000","last_undegraded":"2024-03-06T07:25:55.293885+0000","last_fullsized":"2024-03-06T07:25:55.293885+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:26:35.055387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.302641+0000","last_change":"2024-03-06T07:25:54.293118+0000","last_active":"2024-03-06T07:25:55.302641+0000","last_peered":"2024-03-06T07:25:55.302641+0000","last_clean":"2024-03-06T07:25:55.302641+0000","last_became_active":"2024-03-06T07:25:54.292868+0000","last_became_peered":"2024-03-06T07:25:54.292868+0000","last_unstale":"2024-03-06T07:25:55.302641+0000","last_undegraded":"2024-03-06T07:25:55.302641+0000","last_fullsized":"2024-03-06T07:25:55.302641+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:08.825584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.304263+0000","last_change":"2024-03-06T07:25:55.301253+0000","last_active":"2024-03-06T07:25:55.304263+0000","last_peered":"2024-03-06T07:25:55.304263+0000","last_clean":"2024-03-06T07:25:55.304263+0000","last_became_active":"2024-03-06T07:25:55.300954+0000","last_became_peered":"2024-03-06T07:25:55.300954+0000","last_unstale":"2024-03-06T07:25:55.304263+0000","last_undegraded":"2024-03-06T07:25:55.304263+0000","last_fullsized":"2024-03-06T07:25:55.304263+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:17.542986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.297539+0000","last_change":"2024-03-06T07:25:54.294699+0000","last_active":"2024-03-06T07:25:55.297539+0000","last_peered":"2024-03-06T07:25:55.297539+0000","last_clean":"2024-03-06T07:25:55.297539+0000","last_became_active":"2024-03-06T07:25:54.294443+0000","last_became_peered":"2024-03-06T07:25:54.294443+0000","last_unstale":"2024-03-06T07:25:55.297539+0000","last_undegraded":"2024-03-06T07:25:55.297539+0000","last_fullsized":"2024-03-06T07:25:55.297539+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:21.695732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.297203+0000","last_change":"2024-03-06T07:25:55.296010+0000","last_active":"2024-03-06T07:25:55.297203+0000","last_peered":"2024-03-06T07:25:55.297203+0000","last_clean":"2024-03-06T07:25:55.297203+0000","last_became_active":"2024-03-06T07:25:55.295733+0000","last_became_peered":"2024-03-06T07:25:55.295733+0000","last_unstale":"2024-03-06T07:25:55.297203+0000","last_undegraded":"2024-03-06T07:25:55.297203+0000","last_fullsized":"2024-03-06T07:25:55.297203+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:52:04.927367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.286259+0000","last_change":"2024-03-06T07:25:54.291208+0000","last_active":"2024-03-06T07:25:55.286259+0000","last_peered":"2024-03-06T07:25:55.286259+0000","last_clean":"2024-03-06T07:25:55.286259+0000","last_became_active":"2024-03-06T07:25:54.290582+0000","last_became_peered":"2024-03-06T07:25:54.290582+0000","last_unstale":"2024-03-06T07:25:55.286259+0000","last_undegraded":"2024-03-06T07:25:55.286259+0000","last_fullsized":"2024-03-06T07:25:55.286259+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:07:08.578613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.294547+0000","last_change":"2024-03-06T07:25:55.293174+0000","last_active":"2024-03-06T07:25:55.294547+0000","last_peered":"2024-03-06T07:25:55.294547+0000","last_clean":"2024-03-06T07:25:55.294547+0000","last_became_active":"2024-03-06T07:25:55.292343+0000","last_became_peered":"2024-03-06T07:25:55.292343+0000","last_unstale":"2024-03-06T07:25:55.294547+0000","last_undegraded":"2024-03-06T07:25:55.294547+0000","last_fullsized":"2024-03-06T07:25:55.294547+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:48.091148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"81'2","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.317089+0000","last_change":"2024-03-06T07:25:54.296487+0000","last_active":"2024-03-06T07:25:55.317089+0000","last_peered":"2024-03-06T07:25:55.317089+0000","last_clean":"2024-03-06T07:25:55.317089+0000","last_became_active":"2024-03-06T07:25:54.296235+0000","last_became_peered":"2024-03-06T07:25:54.296235+0000","last_unstale":"2024-03-06T07:25:55.317089+0000","last_undegraded":"2024-03-06T07:25:55.317089+0000","last_fullsized":"2024-03-06T07:25:55.317089+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:30.492845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.300876+0000","last_change":"2024-03-06T07:25:55.296551+0000","last_active":"2024-03-06T07:25:55.300876+0000","last_peered":"2024-03-06T07:25:55.300876+0000","last_clean":"2024-03-06T07:25:55.300876+0000","last_became_active":"2024-03-06T07:25:55.296204+0000","last_became_peered":"2024-03-06T07:25:55.296204+0000","last_unstale":"2024-03-06T07:25:55.300876+0000","last_undegraded":"2024-03-06T07:25:55.300876+0000","last_fullsized":"2024-03-06T07:25:55.300876+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:15.632815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.284017+0000","last_change":"2024-03-06T07:25:54.292685+0000","last_active":"2024-03-06T07:25:55.284017+0000","last_peered":"2024-03-06T07:25:55.284017+0000","last_clean":"2024-03-06T07:25:55.284017+0000","last_became_active":"2024-03-06T07:25:54.291463+0000","last_became_peered":"2024-03-06T07:25:54.291463+0000","last_unstale":"2024-03-06T07:25:55.284017+0000","last_undegraded":"2024-03-06T07:25:55.284017+0000","last_fullsized":"2024-03-06T07:25:55.284017+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:18.080096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.305393+0000","last_change":"2024-03-06T07:25:55.300773+0000","last_active":"2024-03-06T07:25:55.305393+0000","last_peered":"2024-03-06T07:25:55.305393+0000","last_clean":"2024-03-06T07:25:55.305393+0000","last_became_active":"2024-03-06T07:25:55.300434+0000","last_became_peered":"2024-03-06T07:25:55.300434+0000","last_unstale":"2024-03-06T07:25:55.305393+0000","last_undegraded":"2024-03-06T07:25:55.305393+0000","last_fullsized":"2024-03-06T07:25:55.305393+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:57.470091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.282666+0000","last_change":"2024-03-06T07:25:54.291158+0000","last_active":"2024-03-06T07:25:55.282666+0000","last_peered":"2024-03-06T07:25:55.282666+0000","last_clean":"2024-03-06T07:25:55.282666+0000","last_became_active":"2024-03-06T07:25:54.290849+0000","last_became_peered":"2024-03-06T07:25:54.290849+0000","last_unstale":"2024-03-06T07:25:55.282666+0000","last_undegraded":"2024-03-06T07:25:55.282666+0000","last_fullsized":"2024-03-06T07:25:55.282666+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:46:06.100952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.302848+0000","last_change":"2024-03-06T07:25:55.299979+0000","last_active":"2024-03-06T07:25:55.302848+0000","last_peered":"2024-03-06T07:25:55.302848+0000","last_clean":"2024-03-06T07:25:55.302848+0000","last_became_active":"2024-03-06T07:25:55.299260+0000","last_became_peered":"2024-03-06T07:25:55.299260+0000","last_unstale":"2024-03-06T07:25:55.302848+0000","last_undegraded":"2024-03-06T07:25:55.302848+0000","last_fullsized":"2024-03-06T07:25:55.302848+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:52:04.927367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.285252+0000","last_change":"2024-03-06T07:25:54.292616+0000","last_active":"2024-03-06T07:25:55.285252+0000","last_peered":"2024-03-06T07:25:55.285252+0000","last_clean":"2024-03-06T07:25:55.285252+0000","last_became_active":"2024-03-06T07:25:54.292359+0000","last_became_peered":"2024-03-06T07:25:54.292359+0000","last_unstale":"2024-03-06T07:25:55.285252+0000","last_undegraded":"2024-03-06T07:25:55.285252+0000","last_fullsized":"2024-03-06T07:25:55.285252+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:44.841033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.293853+0000","last_change":"2024-03-06T07:25:55.290783+0000","last_active":"2024-03-06T07:25:55.293853+0000","last_peered":"2024-03-06T07:25:55.293853+0000","last_clean":"2024-03-06T07:25:55.293853+0000","last_became_active":"2024-03-06T07:25:55.290485+0000","last_became_peered":"2024-03-06T07:25:55.290485+0000","last_unstale":"2024-03-06T07:25:55.293853+0000","last_undegraded":"2024-03-06T07:25:55.293853+0000","last_fullsized":"2024-03-06T07:25:55.293853+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:51:44.302090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.293745+0000","last_change":"2024-03-06T07:25:54.296533+0000","last_active":"2024-03-06T07:25:55.293745+0000","last_peered":"2024-03-06T07:25:55.293745+0000","last_clean":"2024-03-06T07:25:55.293745+0000","last_became_active":"2024-03-06T07:25:54.296291+0000","last_became_peered":"2024-03-06T07:25:54.296291+0000","last_unstale":"2024-03-06T07:25:55.293745+0000","last_undegraded":"2024-03-06T07:25:55.293745+0000","last_fullsized":"2024-03-06T07:25:55.293745+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:56.465571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.309087+0000","last_change":"2024-03-06T07:25:55.303601+0000","last_active":"2024-03-06T07:25:55.309087+0000","last_peered":"2024-03-06T07:25:55.309087+0000","last_clean":"2024-03-06T07:25:55.309087+0000","last_became_active":"2024-03-06T07:25:55.303315+0000","last_became_peered":"2024-03-06T07:25:55.303315+0000","last_unstale":"2024-03-06T07:25:55.309087+0000","last_undegraded":"2024-03-06T07:25:55.309087+0000","last_fullsized":"2024-03-06T07:25:55.309087+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:35.427361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.288404+0000","last_change":"2024-03-06T07:25:54.290731+0000","last_active":"2024-03-06T07:25:55.288404+0000","last_peered":"2024-03-06T07:25:55.288404+0000","last_clean":"2024-03-06T07:25:55.288404+0000","last_became_active":"2024-03-06T07:25:54.290414+0000","last_became_peered":"2024-03-06T07:25:54.290414+0000","last_unstale":"2024-03-06T07:25:55.288404+0000","last_undegraded":"2024-03-06T07:25:55.288404+0000","last_fullsized":"2024-03-06T07:25:55.288404+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:47:09.862504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.300857+0000","last_change":"2024-03-06T07:25:55.297028+0000","last_active":"2024-03-06T07:25:55.300857+0000","last_peered":"2024-03-06T07:25:55.300857+0000","last_clean":"2024-03-06T07:25:55.300857+0000","last_became_active":"2024-03-06T07:25:55.296767+0000","last_became_peered":"2024-03-06T07:25:55.296767+0000","last_unstale":"2024-03-06T07:25:55.300857+0000","last_undegraded":"2024-03-06T07:25:55.300857+0000","last_fullsized":"2024-03-06T07:25:55.300857+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:07:02.526172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.283519+0000","last_change":"2024-03-06T07:25:54.293230+0000","last_active":"2024-03-06T07:25:55.283519+0000","last_peered":"2024-03-06T07:25:55.283519+0000","last_clean":"2024-03-06T07:25:55.283519+0000","last_became_active":"2024-03-06T07:25:54.292980+0000","last_became_peered":"2024-03-06T07:25:54.292980+0000","last_unstale":"2024-03-06T07:25:55.283519+0000","last_undegraded":"2024-03-06T07:25:55.283519+0000","last_fullsized":"2024-03-06T07:25:55.283519+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:54:34.915295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.306581+0000","last_change":"2024-03-06T07:25:55.303769+0000","last_active":"2024-03-06T07:25:55.306581+0000","last_peered":"2024-03-06T07:25:55.306581+0000","last_clean":"2024-03-06T07:25:55.306581+0000","last_became_active":"2024-03-06T07:25:55.303526+0000","last_became_peered":"2024-03-06T07:25:55.303526+0000","last_unstale":"2024-03-06T07:25:55.306581+0000","last_undegraded":"2024-03-06T07:25:55.306581+0000","last_fullsized":"2024-03-06T07:25:55.306581+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:13.381230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.288075+0000","last_change":"2024-03-06T07:25:54.294078+0000","last_active":"2024-03-06T07:25:55.288075+0000","last_peered":"2024-03-06T07:25:55.288075+0000","last_clean":"2024-03-06T07:25:55.288075+0000","last_became_active":"2024-03-06T07:25:54.293826+0000","last_became_peered":"2024-03-06T07:25:54.293826+0000","last_unstale":"2024-03-06T07:25:55.288075+0000","last_undegraded":"2024-03-06T07:25:55.288075+0000","last_fullsized":"2024-03-06T07:25:55.288075+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:51.548796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.312612+0000","last_change":"2024-03-06T07:25:55.294179+0000","last_active":"2024-03-06T07:25:55.312612+0000","last_peered":"2024-03-06T07:25:55.312612+0000","last_clean":"2024-03-06T07:25:55.312612+0000","last_became_active":"2024-03-06T07:25:55.293871+0000","last_became_peered":"2024-03-06T07:25:55.293871+0000","last_unstale":"2024-03-06T07:25:55.312612+0000","last_undegraded":"2024-03-06T07:25:55.312612+0000","last_fullsized":"2024-03-06T07:25:55.312612+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:55:32.599045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"81'1","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.317266+0000","last_change":"2024-03-06T07:25:54.295348+0000","last_active":"2024-03-06T07:25:55.317266+0000","last_peered":"2024-03-06T07:25:55.317266+0000","last_clean":"2024-03-06T07:25:55.317266+0000","last_became_active":"2024-03-06T07:25:54.295035+0000","last_became_peered":"2024-03-06T07:25:54.295035+0000","last_unstale":"2024-03-06T07:25:55.317266+0000","last_undegraded":"2024-03-06T07:25:55.317266+0000","last_fullsized":"2024-03-06T07:25:55.317266+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:25.556485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":281,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.288705+0000","last_change":"2024-03-06T07:21:47.553693+0000","last_active":"2024-03-06T07:25:55.288705+0000","last_peered":"2024-03-06T07:25:55.288705+0000","last_clean":"2024-03-06T07:25:55.288705+0000","last_became_active":"2024-03-06T07:21:47.552664+0000","last_became_peered":"2024-03-06T07:21:47.552664+0000","last_unstale":"2024-03-06T07:25:55.288705+0000","last_undegraded":"2024-03-06T07:25:55.288705+0000","last_fullsized":"2024-03-06T07:25:55.288705+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_clean_scrub_stamp":"2024-03-06T07:21:46.362028+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:02.916426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574903,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":1005462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574901,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32140,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339700,"statfs":{"total":96636764160,"available":96603852800,"internally_reserved":0,"allocated":983040,"data_stored":415094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574902,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32144,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339696,"statfs":{"total":96636764160,"available":96603848704,"internally_reserved":0,"allocated":983040,"data_stored":415094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574903,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32144,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339696,"statfs":{"total":96636764160,"available":96603848704,"internally_reserved":0,"allocated":983040,"data_stored":415094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574902,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32144,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339696,"statfs":{"total":96636764160,"available":96603848704,"internally_reserved":0,"allocated":983040,"data_stored":415094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574902,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32144,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339696,"statfs":{"total":96636764160,"available":96603848704,"internally_reserved":0,"allocated":983040,"data_stored":415094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574903,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32172,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339668,"statfs":{"total":96636764160,"available":96603820032,"internally_reserved":0,"allocated":1007616,"data_stored":422063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574902,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":1005462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:58.271 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:58.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.405+0000 7faa35676640 1 Processor -- start 2024-03-06T07:25:58.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.406+0000 7faa35676640 1 -- start start 2024-03-06T07:25:58.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.406+0000 7faa35676640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faa3015dfe0 0x7faa300cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.406+0000 7faa35676640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7faa300cf830 0x7faa300d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.406+0000 7faa35676640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7faa300d27f0 0x7faa300d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.406+0000 7faa35676640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa300d2c50 con 0x7faa300cf830 2024-03-06T07:25:58.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.406+0000 7faa35676640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa300b31e0 con 0x7faa300d27f0 2024-03-06T07:25:58.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.406+0000 7faa35676640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa300b3d60 con 0x7faa3015dfe0 2024-03-06T07:25:58.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.406+0000 7faa2e7fc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7faa300cf830 0x7faa300d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.406+0000 7faa2e7fc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7faa300cf830 0x7faa300d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55584/0 (socket says 172.21.15.83:55584) 2024-03-06T07:25:58.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.406+0000 7faa2e7fc640 1 -- 172.21.15.83:0/1758302790 learned_addr learned my addr 172.21.15.83:0/1758302790 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:58.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.407+0000 7faa2effd640 1 --2- 172.21.15.83:0/1758302790 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faa3015dfe0 0x7faa300cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.407+0000 7faa2f7fe640 1 --2- 172.21.15.83:0/1758302790 >> v2:172.21.15.183:3300/0 conn(0x7faa300d27f0 0x7faa300d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.407+0000 7faa2e7fc640 1 -- 172.21.15.83:0/1758302790 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faa3015dfe0 msgr2=0x7faa300cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:58.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.407+0000 7faa2e7fc640 1 --2- 172.21.15.83:0/1758302790 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faa3015dfe0 0x7faa300cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.407+0000 7faa2e7fc640 1 -- 172.21.15.83:0/1758302790 >> v2:172.21.15.183:3300/0 conn(0x7faa300d27f0 msgr2=0x7faa300d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:58.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.407+0000 7faa2e7fc640 1 --2- 172.21.15.83:0/1758302790 >> v2:172.21.15.183:3300/0 conn(0x7faa300d27f0 0x7faa300d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.407+0000 7faa2e7fc640 1 -- 172.21.15.83:0/1758302790 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa300b3430 con 0x7faa300cf830 2024-03-06T07:25:58.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.407+0000 7faa2f7fe640 1 --2- 172.21.15.83:0/1758302790 >> v2:172.21.15.183:3300/0 conn(0x7faa300d27f0 0x7faa300d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:58.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.407+0000 7faa2e7fc640 1 --2- 172.21.15.83:0/1758302790 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faa300cf830 0x7faa300d22b0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7faa30067ad0 tx=0x7faa1802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2443291d597461e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:58.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.408+0000 7faa2dffb640 1 -- 172.21.15.83:0/1758302790 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa18035020 con 0x7faa300cf830 2024-03-06T07:25:58.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.408+0000 7faa2dffb640 1 -- 172.21.15.83:0/1758302790 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa18031870 con 0x7faa300cf830 2024-03-06T07:25:58.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.408+0000 7faa2dffb640 1 -- 172.21.15.83:0/1758302790 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa18031c40 con 0x7faa300cf830 2024-03-06T07:25:58.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.408+0000 7faa35676640 1 -- 172.21.15.83:0/1758302790 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faa300cf830 msgr2=0x7faa300d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.408+0000 7faa35676640 1 --2- 172.21.15.83:0/1758302790 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faa300cf830 0x7faa300d22b0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7faa30067ad0 tx=0x7faa1802b910 comp rx=0 tx=0).stop 2024-03-06T07:25:58.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.408+0000 7faa35676640 1 -- 172.21.15.83:0/1758302790 shutdown_connections 2024-03-06T07:25:58.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.408+0000 7faa35676640 1 --2- 172.21.15.83:0/1758302790 >> v2:172.21.15.183:3300/0 conn(0x7faa300d27f0 0x7faa300d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.408+0000 7faa35676640 1 --2- 172.21.15.83:0/1758302790 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faa300cf830 0x7faa300d22b0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.408+0000 7faa35676640 1 --2- 172.21.15.83:0/1758302790 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faa3015dfe0 0x7faa300cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.408+0000 7faa35676640 1 -- 172.21.15.83:0/1758302790 >> 172.21.15.83:0/1758302790 conn(0x7faa300d3cd0 msgr2=0x7faa300d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:58.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.408+0000 7faa35676640 1 -- 172.21.15.83:0/1758302790 shutdown_connections 2024-03-06T07:25:58.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.408+0000 7faa35676640 1 -- 172.21.15.83:0/1758302790 wait complete. 2024-03-06T07:25:58.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.409+0000 7faa35676640 1 Processor -- start 2024-03-06T07:25:58.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.410+0000 7faa35676640 1 -- start start 2024-03-06T07:25:58.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.410+0000 7faa35676640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faa300cf830 0x7faa300b6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.410+0000 7faa35676640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faa300d27f0 0x7faa301f3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.410+0000 7faa35676640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faa3015dfe0 0x7faa30149d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.410+0000 7faa35676640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa300d1c70 con 0x7faa300d27f0 2024-03-06T07:25:58.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.410+0000 7faa35676640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa300b39f0 con 0x7faa300cf830 2024-03-06T07:25:58.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.410+0000 7faa35676640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa300d1df0 con 0x7faa3015dfe0 2024-03-06T07:25:58.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.410+0000 7faa2e7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faa300d27f0 0x7faa301f3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.410+0000 7faa2e7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faa300d27f0 0x7faa301f3020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55592/0 (socket says 172.21.15.83:55592) 2024-03-06T07:25:58.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.410+0000 7faa2e7fc640 1 -- 172.21.15.83:0/1236738705 learned_addr learned my addr 172.21.15.83:0/1236738705 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:58.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.410+0000 7faa2effd640 1 --2- 172.21.15.83:0/1236738705 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faa300cf830 0x7faa300b6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.410+0000 7faa2f7fe640 1 --2- 172.21.15.83:0/1236738705 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faa3015dfe0 0x7faa30149d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.411+0000 7faa2e7fc640 1 -- 172.21.15.83:0/1236738705 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faa3015dfe0 msgr2=0x7faa30149d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.411+0000 7faa2e7fc640 1 --2- 172.21.15.83:0/1236738705 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faa3015dfe0 0x7faa30149d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.411+0000 7faa2e7fc640 1 -- 172.21.15.83:0/1236738705 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faa300cf830 msgr2=0x7faa300b6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.411+0000 7faa2e7fc640 1 --2- 172.21.15.83:0/1236738705 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faa300cf830 0x7faa300b6940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.411+0000 7faa2e7fc640 1 -- 172.21.15.83:0/1236738705 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa300b8ca0 con 0x7faa300d27f0 2024-03-06T07:25:58.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.411+0000 7faa2e7fc640 1 --2- 172.21.15.83:0/1236738705 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faa300d27f0 0x7faa301f3020 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7faa1802b950 tx=0x7faa180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:58.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.411+0000 7faa277fe640 1 -- 172.21.15.83:0/1236738705 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa18040070 con 0x7faa300d27f0 2024-03-06T07:25:58.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.411+0000 7faa277fe640 1 -- 172.21.15.83:0/1236738705 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa18030d00 con 0x7faa300d27f0 2024-03-06T07:25:58.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.411+0000 7faa277fe640 1 -- 172.21.15.83:0/1236738705 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa18035040 con 0x7faa300d27f0 2024-03-06T07:25:58.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.411+0000 7faa35676640 1 -- 172.21.15.83:0/1236738705 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa300b8f20 con 0x7faa300d27f0 2024-03-06T07:25:58.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.411+0000 7faa35676640 1 -- 172.21.15.83:0/1236738705 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa300b94e0 con 0x7faa300d27f0 2024-03-06T07:25:58.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.412+0000 7faa35676640 1 -- 172.21.15.83:0/1236738705 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa3004f330 con 0x7faa300d27f0 2024-03-06T07:25:58.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.412+0000 7faa277fe640 1 -- 172.21.15.83:0/1236738705 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faa18038020 con 0x7faa300d27f0 2024-03-06T07:25:58.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.413+0000 7faa277fe640 1 --2- 172.21.15.83:0/1236738705 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7faa1006b4d0 0x7faa1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.413+0000 7faa2effd640 1 --2- 172.21.15.83:0/1236738705 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7faa1006b4d0 0x7faa1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.413+0000 7faa277fe640 1 -- 172.21.15.83:0/1236738705 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faa180ac080 con 0x7faa300d27f0 2024-03-06T07:25:58.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.413+0000 7faa2effd640 1 --2- 172.21.15.83:0/1236738705 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7faa1006b4d0 0x7faa1006d990 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7faa28006fd0 tx=0x7faa28008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:58.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.417+0000 7faa277fe640 1 -- 172.21.15.83:0/1236738705 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faa300b8ca0 con 0x7faa300d27f0 2024-03-06T07:25:58.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.590+0000 7faa35676640 1 -- 172.21.15.83:0/1236738705 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7faa300747a0 con 0x7faa1006b4d0 2024-03-06T07:25:58.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.596+0000 7faa277fe640 1 -- 172.21.15.83:0/1236738705 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252004 (secure 0 0 0) 0x7faa300747a0 con 0x7faa1006b4d0 2024-03-06T07:25:58.601 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:25:58.602 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-06T07:25:58.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.600+0000 7faa35676640 1 -- 172.21.15.83:0/1236738705 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7faa1006b4d0 msgr2=0x7faa1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.600+0000 7faa35676640 1 --2- 172.21.15.83:0/1236738705 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7faa1006b4d0 0x7faa1006d990 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7faa28006fd0 tx=0x7faa28008040 comp rx=0 tx=0).stop 2024-03-06T07:25:58.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.600+0000 7faa35676640 1 -- 172.21.15.83:0/1236738705 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faa300d27f0 msgr2=0x7faa301f3020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.600+0000 7faa35676640 1 --2- 172.21.15.83:0/1236738705 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faa300d27f0 0x7faa301f3020 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7faa1802b950 tx=0x7faa180046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.600+0000 7faa35676640 1 -- 172.21.15.83:0/1236738705 shutdown_connections 2024-03-06T07:25:58.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.600+0000 7faa35676640 1 --2- 172.21.15.83:0/1236738705 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7faa1006b4d0 0x7faa1006d990 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.600+0000 7faa35676640 1 --2- 172.21.15.83:0/1236738705 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faa3015dfe0 0x7faa30149d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.600+0000 7faa35676640 1 --2- 172.21.15.83:0/1236738705 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faa300d27f0 0x7faa301f3020 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.600+0000 7faa35676640 1 --2- 172.21.15.83:0/1236738705 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faa300cf830 0x7faa300b6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.600+0000 7faa35676640 1 -- 172.21.15.83:0/1236738705 >> 172.21.15.83:0/1236738705 conn(0x7faa300d3cd0 msgr2=0x7faa300d0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:58.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.600+0000 7faa35676640 1 -- 172.21.15.83:0/1236738705 shutdown_connections 2024-03-06T07:25:58.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.600+0000 7faa35676640 1 -- 172.21.15.83:0/1236738705 wait complete. 2024-03-06T07:25:58.613 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":202,"stamp":"2024-03-06T07:25:58.363629+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":67,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258328,"kb_used_data":8864,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716392,"statfs":{"total":773094113280,"available":772829585408,"internally_reserved":0,"allocated":9076736,"data_stored":4508457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12757,"internal_metadata":255315499},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.089992"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.296965+0000","last_change":"2024-03-06T07:25:55.292649+0000","last_active":"2024-03-06T07:25:55.296965+0000","last_peered":"2024-03-06T07:25:55.296965+0000","last_clean":"2024-03-06T07:25:55.296965+0000","last_became_active":"2024-03-06T07:25:55.292348+0000","last_became_peered":"2024-03-06T07:25:55.292348+0000","last_unstale":"2024-03-06T07:25:55.296965+0000","last_undegraded":"2024-03-06T07:25:55.296965+0000","last_fullsized":"2024-03-06T07:25:55.296965+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:31.756662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.311939+0000","last_change":"2024-03-06T07:25:55.291769+0000","last_active":"2024-03-06T07:25:55.311939+0000","last_peered":"2024-03-06T07:25:55.311939+0000","last_clean":"2024-03-06T07:25:55.311939+0000","last_became_active":"2024-03-06T07:25:55.291490+0000","last_became_peered":"2024-03-06T07:25:55.291490+0000","last_unstale":"2024-03-06T07:25:55.311939+0000","last_undegraded":"2024-03-06T07:25:55.311939+0000","last_fullsized":"2024-03-06T07:25:55.311939+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:07:31.904278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.300901+0000","last_change":"2024-03-06T07:25:55.296596+0000","last_active":"2024-03-06T07:25:55.300901+0000","last_peered":"2024-03-06T07:25:55.300901+0000","last_clean":"2024-03-06T07:25:55.300901+0000","last_became_active":"2024-03-06T07:25:55.296244+0000","last_became_peered":"2024-03-06T07:25:55.296244+0000","last_unstale":"2024-03-06T07:25:55.300901+0000","last_undegraded":"2024-03-06T07:25:55.300901+0000","last_fullsized":"2024-03-06T07:25:55.300901+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:21.500763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.304202+0000","last_change":"2024-03-06T07:25:55.300898+0000","last_active":"2024-03-06T07:25:55.304202+0000","last_peered":"2024-03-06T07:25:55.304202+0000","last_clean":"2024-03-06T07:25:55.304202+0000","last_became_active":"2024-03-06T07:25:55.300611+0000","last_became_peered":"2024-03-06T07:25:55.300611+0000","last_unstale":"2024-03-06T07:25:55.304202+0000","last_undegraded":"2024-03-06T07:25:55.304202+0000","last_fullsized":"2024-03-06T07:25:55.304202+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:19.084617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.300392+0000","last_change":"2024-03-06T07:25:55.296794+0000","last_active":"2024-03-06T07:25:55.300392+0000","last_peered":"2024-03-06T07:25:55.300392+0000","last_clean":"2024-03-06T07:25:55.300392+0000","last_became_active":"2024-03-06T07:25:55.296514+0000","last_became_peered":"2024-03-06T07:25:55.296514+0000","last_unstale":"2024-03-06T07:25:55.300392+0000","last_undegraded":"2024-03-06T07:25:55.300392+0000","last_fullsized":"2024-03-06T07:25:55.300392+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:47.313545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.305992+0000","last_change":"2024-03-06T07:25:55.305416+0000","last_active":"2024-03-06T07:25:55.305992+0000","last_peered":"2024-03-06T07:25:55.305992+0000","last_clean":"2024-03-06T07:25:55.305992+0000","last_became_active":"2024-03-06T07:25:55.305092+0000","last_became_peered":"2024-03-06T07:25:55.305092+0000","last_unstale":"2024-03-06T07:25:55.305992+0000","last_undegraded":"2024-03-06T07:25:55.305992+0000","last_fullsized":"2024-03-06T07:25:55.305992+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:05:41.806303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.304299+0000","last_change":"2024-03-06T07:25:55.300485+0000","last_active":"2024-03-06T07:25:55.304299+0000","last_peered":"2024-03-06T07:25:55.304299+0000","last_clean":"2024-03-06T07:25:55.304299+0000","last_became_active":"2024-03-06T07:25:55.300186+0000","last_became_peered":"2024-03-06T07:25:55.300186+0000","last_unstale":"2024-03-06T07:25:55.304299+0000","last_undegraded":"2024-03-06T07:25:55.304299+0000","last_fullsized":"2024-03-06T07:25:55.304299+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:12.586950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.294414+0000","last_change":"2024-03-06T07:25:55.293481+0000","last_active":"2024-03-06T07:25:55.294414+0000","last_peered":"2024-03-06T07:25:55.294414+0000","last_clean":"2024-03-06T07:25:55.294414+0000","last_became_active":"2024-03-06T07:25:55.293205+0000","last_became_peered":"2024-03-06T07:25:55.293205+0000","last_unstale":"2024-03-06T07:25:55.294414+0000","last_undegraded":"2024-03-06T07:25:55.294414+0000","last_fullsized":"2024-03-06T07:25:55.294414+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:13.381230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.294431+0000","last_change":"2024-03-06T07:25:55.291760+0000","last_active":"2024-03-06T07:25:55.294431+0000","last_peered":"2024-03-06T07:25:55.294431+0000","last_clean":"2024-03-06T07:25:55.294431+0000","last_became_active":"2024-03-06T07:25:55.291366+0000","last_became_peered":"2024-03-06T07:25:55.291366+0000","last_unstale":"2024-03-06T07:25:55.294431+0000","last_undegraded":"2024-03-06T07:25:55.294431+0000","last_fullsized":"2024-03-06T07:25:55.294431+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:57.470091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.297045+0000","last_change":"2024-03-06T07:25:55.291368+0000","last_active":"2024-03-06T07:25:55.297045+0000","last_peered":"2024-03-06T07:25:55.297045+0000","last_clean":"2024-03-06T07:25:55.297045+0000","last_became_active":"2024-03-06T07:25:55.290947+0000","last_became_peered":"2024-03-06T07:25:55.290947+0000","last_unstale":"2024-03-06T07:25:55.297045+0000","last_undegraded":"2024-03-06T07:25:55.297045+0000","last_fullsized":"2024-03-06T07:25:55.297045+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:47:10.867024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.312025+0000","last_change":"2024-03-06T07:25:55.293123+0000","last_active":"2024-03-06T07:25:55.312025+0000","last_peered":"2024-03-06T07:25:55.312025+0000","last_clean":"2024-03-06T07:25:55.312025+0000","last_became_active":"2024-03-06T07:25:55.292780+0000","last_became_peered":"2024-03-06T07:25:55.292780+0000","last_unstale":"2024-03-06T07:25:55.312025+0000","last_undegraded":"2024-03-06T07:25:55.312025+0000","last_fullsized":"2024-03-06T07:25:55.312025+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:28:04.429768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.297072+0000","last_change":"2024-03-06T07:25:55.292088+0000","last_active":"2024-03-06T07:25:55.297072+0000","last_peered":"2024-03-06T07:25:55.297072+0000","last_clean":"2024-03-06T07:25:55.297072+0000","last_became_active":"2024-03-06T07:25:55.291583+0000","last_became_peered":"2024-03-06T07:25:55.291583+0000","last_unstale":"2024-03-06T07:25:55.297072+0000","last_undegraded":"2024-03-06T07:25:55.297072+0000","last_fullsized":"2024-03-06T07:25:55.297072+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:19:43.401290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.300699+0000","last_change":"2024-03-06T07:25:55.293447+0000","last_active":"2024-03-06T07:25:55.300699+0000","last_peered":"2024-03-06T07:25:55.300699+0000","last_clean":"2024-03-06T07:25:55.300699+0000","last_became_active":"2024-03-06T07:25:55.293167+0000","last_became_peered":"2024-03-06T07:25:55.293167+0000","last_unstale":"2024-03-06T07:25:55.300699+0000","last_undegraded":"2024-03-06T07:25:55.300699+0000","last_fullsized":"2024-03-06T07:25:55.300699+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:36:14.844243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.320157+0000","last_change":"2024-03-06T07:25:55.317724+0000","last_active":"2024-03-06T07:25:55.320157+0000","last_peered":"2024-03-06T07:25:55.320157+0000","last_clean":"2024-03-06T07:25:55.320157+0000","last_became_active":"2024-03-06T07:25:55.317392+0000","last_became_peered":"2024-03-06T07:25:55.317392+0000","last_unstale":"2024-03-06T07:25:55.320157+0000","last_undegraded":"2024-03-06T07:25:55.320157+0000","last_fullsized":"2024-03-06T07:25:55.320157+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:28:04.429768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.298763+0000","last_change":"2024-03-06T07:25:55.293332+0000","last_active":"2024-03-06T07:25:55.298763+0000","last_peered":"2024-03-06T07:25:55.298763+0000","last_clean":"2024-03-06T07:25:55.298763+0000","last_became_active":"2024-03-06T07:25:55.293082+0000","last_became_peered":"2024-03-06T07:25:55.293082+0000","last_unstale":"2024-03-06T07:25:55.298763+0000","last_undegraded":"2024-03-06T07:25:55.298763+0000","last_fullsized":"2024-03-06T07:25:55.298763+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:01:22.060797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.302751+0000","last_change":"2024-03-06T07:25:55.297008+0000","last_active":"2024-03-06T07:25:55.302751+0000","last_peered":"2024-03-06T07:25:55.302751+0000","last_clean":"2024-03-06T07:25:55.302751+0000","last_became_active":"2024-03-06T07:25:55.296714+0000","last_became_peered":"2024-03-06T07:25:55.296714+0000","last_unstale":"2024-03-06T07:25:55.302751+0000","last_undegraded":"2024-03-06T07:25:55.302751+0000","last_fullsized":"2024-03-06T07:25:55.302751+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:26:35.055387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.307634+0000","last_change":"2024-03-06T07:25:55.304655+0000","last_active":"2024-03-06T07:25:55.307634+0000","last_peered":"2024-03-06T07:25:55.307634+0000","last_clean":"2024-03-06T07:25:55.307634+0000","last_became_active":"2024-03-06T07:25:55.304358+0000","last_became_peered":"2024-03-06T07:25:55.304358+0000","last_unstale":"2024-03-06T07:25:55.307634+0000","last_undegraded":"2024-03-06T07:25:55.307634+0000","last_fullsized":"2024-03-06T07:25:55.307634+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:30:51.700835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.305719+0000","last_change":"2024-03-06T07:25:55.304037+0000","last_active":"2024-03-06T07:25:55.305719+0000","last_peered":"2024-03-06T07:25:55.305719+0000","last_clean":"2024-03-06T07:25:55.305719+0000","last_became_active":"2024-03-06T07:25:55.303698+0000","last_became_peered":"2024-03-06T07:25:55.303698+0000","last_unstale":"2024-03-06T07:25:55.305719+0000","last_undegraded":"2024-03-06T07:25:55.305719+0000","last_fullsized":"2024-03-06T07:25:55.305719+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:07:31.904278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.303768+0000","last_change":"2024-03-06T07:25:55.300658+0000","last_active":"2024-03-06T07:25:55.303768+0000","last_peered":"2024-03-06T07:25:55.303768+0000","last_clean":"2024-03-06T07:25:55.303768+0000","last_became_active":"2024-03-06T07:25:55.300409+0000","last_became_peered":"2024-03-06T07:25:55.300409+0000","last_unstale":"2024-03-06T07:25:55.303768+0000","last_undegraded":"2024-03-06T07:25:55.303768+0000","last_fullsized":"2024-03-06T07:25:55.303768+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:05:41.806303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.304650+0000","last_change":"2024-03-06T07:25:55.302928+0000","last_active":"2024-03-06T07:25:55.304650+0000","last_peered":"2024-03-06T07:25:55.304650+0000","last_clean":"2024-03-06T07:25:55.304650+0000","last_became_active":"2024-03-06T07:25:55.302641+0000","last_became_peered":"2024-03-06T07:25:55.302641+0000","last_unstale":"2024-03-06T07:25:55.304650+0000","last_undegraded":"2024-03-06T07:25:55.304650+0000","last_fullsized":"2024-03-06T07:25:55.304650+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:28:41.237481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.304225+0000","last_change":"2024-03-06T07:25:55.300880+0000","last_active":"2024-03-06T07:25:55.304225+0000","last_peered":"2024-03-06T07:25:55.304225+0000","last_clean":"2024-03-06T07:25:55.304225+0000","last_became_active":"2024-03-06T07:25:55.299911+0000","last_became_peered":"2024-03-06T07:25:55.299911+0000","last_unstale":"2024-03-06T07:25:55.304225+0000","last_undegraded":"2024-03-06T07:25:55.304225+0000","last_fullsized":"2024-03-06T07:25:55.304225+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:51.828978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.312967+0000","last_change":"2024-03-06T07:25:55.295597+0000","last_active":"2024-03-06T07:25:55.312967+0000","last_peered":"2024-03-06T07:25:55.312967+0000","last_clean":"2024-03-06T07:25:55.312967+0000","last_became_active":"2024-03-06T07:25:55.295303+0000","last_became_peered":"2024-03-06T07:25:55.295303+0000","last_unstale":"2024-03-06T07:25:55.312967+0000","last_undegraded":"2024-03-06T07:25:55.312967+0000","last_fullsized":"2024-03-06T07:25:55.312967+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:26:14.898519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.297876+0000","last_change":"2024-03-06T07:25:55.296213+0000","last_active":"2024-03-06T07:25:55.297876+0000","last_peered":"2024-03-06T07:25:55.297876+0000","last_clean":"2024-03-06T07:25:55.297876+0000","last_became_active":"2024-03-06T07:25:55.295884+0000","last_became_peered":"2024-03-06T07:25:55.295884+0000","last_unstale":"2024-03-06T07:25:55.297876+0000","last_undegraded":"2024-03-06T07:25:55.297876+0000","last_fullsized":"2024-03-06T07:25:55.297876+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:51:44.302090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.304243+0000","last_change":"2024-03-06T07:25:55.301130+0000","last_active":"2024-03-06T07:25:55.304243+0000","last_peered":"2024-03-06T07:25:55.304243+0000","last_clean":"2024-03-06T07:25:55.304243+0000","last_became_active":"2024-03-06T07:25:55.300197+0000","last_became_peered":"2024-03-06T07:25:55.300197+0000","last_unstale":"2024-03-06T07:25:55.304243+0000","last_undegraded":"2024-03-06T07:25:55.304243+0000","last_fullsized":"2024-03-06T07:25:55.304243+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:31.497366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.296098+0000","last_change":"2024-03-06T07:25:55.287535+0000","last_active":"2024-03-06T07:25:55.296098+0000","last_peered":"2024-03-06T07:25:55.296098+0000","last_clean":"2024-03-06T07:25:55.296098+0000","last_became_active":"2024-03-06T07:25:55.287237+0000","last_became_peered":"2024-03-06T07:25:55.287237+0000","last_unstale":"2024-03-06T07:25:55.296098+0000","last_undegraded":"2024-03-06T07:25:55.296098+0000","last_fullsized":"2024-03-06T07:25:55.296098+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:17.542986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.301024+0000","last_change":"2024-03-06T07:25:55.297752+0000","last_active":"2024-03-06T07:25:55.301024+0000","last_peered":"2024-03-06T07:25:55.301024+0000","last_clean":"2024-03-06T07:25:55.301024+0000","last_became_active":"2024-03-06T07:25:55.297468+0000","last_became_peered":"2024-03-06T07:25:55.297468+0000","last_unstale":"2024-03-06T07:25:55.301024+0000","last_undegraded":"2024-03-06T07:25:55.301024+0000","last_fullsized":"2024-03-06T07:25:55.301024+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:07:09.583134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.293770+0000","last_change":"2024-03-06T07:25:55.292818+0000","last_active":"2024-03-06T07:25:55.293770+0000","last_peered":"2024-03-06T07:25:55.293770+0000","last_clean":"2024-03-06T07:25:55.293770+0000","last_became_active":"2024-03-06T07:25:55.292411+0000","last_became_peered":"2024-03-06T07:25:55.292411+0000","last_unstale":"2024-03-06T07:25:55.293770+0000","last_undegraded":"2024-03-06T07:25:55.293770+0000","last_fullsized":"2024-03-06T07:25:55.293770+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:31.756662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.297090+0000","last_change":"2024-03-06T07:25:55.292902+0000","last_active":"2024-03-06T07:25:55.297090+0000","last_peered":"2024-03-06T07:25:55.297090+0000","last_clean":"2024-03-06T07:25:55.297090+0000","last_became_active":"2024-03-06T07:25:55.292170+0000","last_became_peered":"2024-03-06T07:25:55.292170+0000","last_unstale":"2024-03-06T07:25:55.297090+0000","last_undegraded":"2024-03-06T07:25:55.297090+0000","last_fullsized":"2024-03-06T07:25:55.297090+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:48.091148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.296136+0000","last_change":"2024-03-06T07:25:55.290918+0000","last_active":"2024-03-06T07:25:55.296136+0000","last_peered":"2024-03-06T07:25:55.296136+0000","last_clean":"2024-03-06T07:25:55.296136+0000","last_became_active":"2024-03-06T07:25:55.290668+0000","last_became_peered":"2024-03-06T07:25:55.290668+0000","last_unstale":"2024-03-06T07:25:55.296136+0000","last_undegraded":"2024-03-06T07:25:55.296136+0000","last_fullsized":"2024-03-06T07:25:55.296136+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:28:41.237481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.317074+0000","last_change":"2024-03-06T07:25:55.314084+0000","last_active":"2024-03-06T07:25:55.317074+0000","last_peered":"2024-03-06T07:25:55.317074+0000","last_clean":"2024-03-06T07:25:55.317074+0000","last_became_active":"2024-03-06T07:25:55.313802+0000","last_became_peered":"2024-03-06T07:25:55.313802+0000","last_unstale":"2024-03-06T07:25:55.317074+0000","last_undegraded":"2024-03-06T07:25:55.317074+0000","last_fullsized":"2024-03-06T07:25:55.317074+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:50:24.773216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.302769+0000","last_change":"2024-03-06T07:25:55.299543+0000","last_active":"2024-03-06T07:25:55.302769+0000","last_peered":"2024-03-06T07:25:55.302769+0000","last_clean":"2024-03-06T07:25:55.302769+0000","last_became_active":"2024-03-06T07:25:55.299235+0000","last_became_peered":"2024-03-06T07:25:55.299235+0000","last_unstale":"2024-03-06T07:25:55.302769+0000","last_undegraded":"2024-03-06T07:25:55.302769+0000","last_fullsized":"2024-03-06T07:25:55.302769+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:57.470091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.294659+0000","last_change":"2024-03-06T07:25:55.286654+0000","last_active":"2024-03-06T07:25:55.294659+0000","last_peered":"2024-03-06T07:25:55.294659+0000","last_clean":"2024-03-06T07:25:55.294659+0000","last_became_active":"2024-03-06T07:25:55.286367+0000","last_became_peered":"2024-03-06T07:25:55.286367+0000","last_unstale":"2024-03-06T07:25:55.294659+0000","last_undegraded":"2024-03-06T07:25:55.294659+0000","last_fullsized":"2024-03-06T07:25:55.294659+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:04:06.945268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.300307+0000","last_change":"2024-03-06T07:25:55.296703+0000","last_active":"2024-03-06T07:25:55.300307+0000","last_peered":"2024-03-06T07:25:55.300307+0000","last_clean":"2024-03-06T07:25:55.300307+0000","last_became_active":"2024-03-06T07:25:55.296417+0000","last_became_peered":"2024-03-06T07:25:55.296417+0000","last_unstale":"2024-03-06T07:25:55.300307+0000","last_undegraded":"2024-03-06T07:25:55.300307+0000","last_fullsized":"2024-03-06T07:25:55.300307+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:10:06.671054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"81'4","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.315310+0000","last_change":"2024-03-06T07:25:54.291182+0000","last_active":"2024-03-06T07:25:55.315310+0000","last_peered":"2024-03-06T07:25:55.315310+0000","last_clean":"2024-03-06T07:25:55.315310+0000","last_became_active":"2024-03-06T07:25:54.290745+0000","last_became_peered":"2024-03-06T07:25:54.290745+0000","last_unstale":"2024-03-06T07:25:55.315310+0000","last_undegraded":"2024-03-06T07:25:55.315310+0000","last_fullsized":"2024-03-06T07:25:55.315310+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:37.513716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.297991+0000","last_change":"2024-03-06T07:25:55.295155+0000","last_active":"2024-03-06T07:25:55.297991+0000","last_peered":"2024-03-06T07:25:55.297991+0000","last_clean":"2024-03-06T07:25:55.297991+0000","last_became_active":"2024-03-06T07:25:55.294875+0000","last_became_peered":"2024-03-06T07:25:55.294875+0000","last_unstale":"2024-03-06T07:25:55.297991+0000","last_undegraded":"2024-03-06T07:25:55.297991+0000","last_fullsized":"2024-03-06T07:25:55.297991+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:26:35.055387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"81'1","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.308432+0000","last_change":"2024-03-06T07:25:54.291074+0000","last_active":"2024-03-06T07:25:55.308432+0000","last_peered":"2024-03-06T07:25:55.308432+0000","last_clean":"2024-03-06T07:25:55.308432+0000","last_became_active":"2024-03-06T07:25:54.290582+0000","last_became_peered":"2024-03-06T07:25:54.290582+0000","last_unstale":"2024-03-06T07:25:55.308432+0000","last_undegraded":"2024-03-06T07:25:55.308432+0000","last_fullsized":"2024-03-06T07:25:55.308432+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:19:42.396769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.297843+0000","last_change":"2024-03-06T07:25:55.294906+0000","last_active":"2024-03-06T07:25:55.297843+0000","last_peered":"2024-03-06T07:25:55.297843+0000","last_clean":"2024-03-06T07:25:55.297843+0000","last_became_active":"2024-03-06T07:25:55.294456+0000","last_became_peered":"2024-03-06T07:25:55.294456+0000","last_unstale":"2024-03-06T07:25:55.297843+0000","last_undegraded":"2024-03-06T07:25:55.297843+0000","last_fullsized":"2024-03-06T07:25:55.297843+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:47.313545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"81'2","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.311462+0000","last_change":"2024-03-06T07:25:54.292060+0000","last_active":"2024-03-06T07:25:55.311462+0000","last_peered":"2024-03-06T07:25:55.311462+0000","last_clean":"2024-03-06T07:25:55.311462+0000","last_became_active":"2024-03-06T07:25:54.291766+0000","last_became_peered":"2024-03-06T07:25:54.291766+0000","last_unstale":"2024-03-06T07:25:55.311462+0000","last_undegraded":"2024-03-06T07:25:55.311462+0000","last_fullsized":"2024-03-06T07:25:55.311462+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:51:43.297569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.307936+0000","last_change":"2024-03-06T07:25:55.307027+0000","last_active":"2024-03-06T07:25:55.307936+0000","last_peered":"2024-03-06T07:25:55.307936+0000","last_clean":"2024-03-06T07:25:55.307936+0000","last_became_active":"2024-03-06T07:25:55.306704+0000","last_became_peered":"2024-03-06T07:25:55.306704+0000","last_unstale":"2024-03-06T07:25:55.307936+0000","last_undegraded":"2024-03-06T07:25:55.307936+0000","last_fullsized":"2024-03-06T07:25:55.307936+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:30:10.567259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"81'2","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.315218+0000","last_change":"2024-03-06T07:25:54.292709+0000","last_active":"2024-03-06T07:25:55.315218+0000","last_peered":"2024-03-06T07:25:55.315218+0000","last_clean":"2024-03-06T07:25:55.315218+0000","last_became_active":"2024-03-06T07:25:54.292462+0000","last_became_peered":"2024-03-06T07:25:54.292462+0000","last_unstale":"2024-03-06T07:25:55.315218+0000","last_undegraded":"2024-03-06T07:25:55.315218+0000","last_fullsized":"2024-03-06T07:25:55.315218+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:12.376709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.317055+0000","last_change":"2024-03-06T07:25:55.313962+0000","last_active":"2024-03-06T07:25:55.317055+0000","last_peered":"2024-03-06T07:25:55.317055+0000","last_clean":"2024-03-06T07:25:55.317055+0000","last_became_active":"2024-03-06T07:25:55.313591+0000","last_became_peered":"2024-03-06T07:25:55.313591+0000","last_unstale":"2024-03-06T07:25:55.317055+0000","last_undegraded":"2024-03-06T07:25:55.317055+0000","last_fullsized":"2024-03-06T07:25:55.317055+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:35:22.698296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"81'1","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.317207+0000","last_change":"2024-03-06T07:25:54.293197+0000","last_active":"2024-03-06T07:25:55.317207+0000","last_peered":"2024-03-06T07:25:55.317207+0000","last_clean":"2024-03-06T07:25:55.317207+0000","last_became_active":"2024-03-06T07:25:54.292915+0000","last_became_peered":"2024-03-06T07:25:54.292915+0000","last_unstale":"2024-03-06T07:25:55.317207+0000","last_undegraded":"2024-03-06T07:25:55.317207+0000","last_fullsized":"2024-03-06T07:25:55.317207+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:44.841033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.307704+0000","last_change":"2024-03-06T07:25:55.304875+0000","last_active":"2024-03-06T07:25:55.307704+0000","last_peered":"2024-03-06T07:25:55.307704+0000","last_clean":"2024-03-06T07:25:55.307704+0000","last_became_active":"2024-03-06T07:25:55.304504+0000","last_became_peered":"2024-03-06T07:25:55.304504+0000","last_unstale":"2024-03-06T07:25:55.307704+0000","last_undegraded":"2024-03-06T07:25:55.307704+0000","last_fullsized":"2024-03-06T07:25:55.307704+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:05:41.806303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"81'1","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.307737+0000","last_change":"2024-03-06T07:25:54.290635+0000","last_active":"2024-03-06T07:25:55.307737+0000","last_peered":"2024-03-06T07:25:55.307737+0000","last_clean":"2024-03-06T07:25:55.307737+0000","last_became_active":"2024-03-06T07:25:54.290357+0000","last_became_peered":"2024-03-06T07:25:54.290357+0000","last_unstale":"2024-03-06T07:25:55.307737+0000","last_undegraded":"2024-03-06T07:25:55.307737+0000","last_fullsized":"2024-03-06T07:25:55.307737+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:04:05.940747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.302682+0000","last_change":"2024-03-06T07:25:55.298515+0000","last_active":"2024-03-06T07:25:55.302682+0000","last_peered":"2024-03-06T07:25:55.302682+0000","last_clean":"2024-03-06T07:25:55.302682+0000","last_became_active":"2024-03-06T07:25:55.298237+0000","last_became_peered":"2024-03-06T07:25:55.298237+0000","last_unstale":"2024-03-06T07:25:55.302682+0000","last_undegraded":"2024-03-06T07:25:55.302682+0000","last_fullsized":"2024-03-06T07:25:55.302682+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:20.461335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.291830+0000","last_change":"2024-03-06T07:25:54.294694+0000","last_active":"2024-03-06T07:25:55.291830+0000","last_peered":"2024-03-06T07:25:55.291830+0000","last_clean":"2024-03-06T07:25:55.291830+0000","last_became_active":"2024-03-06T07:25:54.294417+0000","last_became_peered":"2024-03-06T07:25:54.294417+0000","last_unstale":"2024-03-06T07:25:55.291830+0000","last_undegraded":"2024-03-06T07:25:55.291830+0000","last_fullsized":"2024-03-06T07:25:55.291830+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:26:34.050866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.298737+0000","last_change":"2024-03-06T07:25:55.292659+0000","last_active":"2024-03-06T07:25:55.298737+0000","last_peered":"2024-03-06T07:25:55.298737+0000","last_clean":"2024-03-06T07:25:55.298737+0000","last_became_active":"2024-03-06T07:25:55.292364+0000","last_became_peered":"2024-03-06T07:25:55.292364+0000","last_unstale":"2024-03-06T07:25:55.298737+0000","last_undegraded":"2024-03-06T07:25:55.298737+0000","last_fullsized":"2024-03-06T07:25:55.298737+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:30:10.567259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.282771+0000","last_change":"2024-03-06T07:25:54.290900+0000","last_active":"2024-03-06T07:25:55.282771+0000","last_peered":"2024-03-06T07:25:55.282771+0000","last_clean":"2024-03-06T07:25:55.282771+0000","last_became_active":"2024-03-06T07:25:54.290597+0000","last_became_peered":"2024-03-06T07:25:54.290597+0000","last_unstale":"2024-03-06T07:25:55.282771+0000","last_undegraded":"2024-03-06T07:25:55.282771+0000","last_fullsized":"2024-03-06T07:25:55.282771+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:11.582429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.299203+0000","last_change":"2024-03-06T07:25:55.297343+0000","last_active":"2024-03-06T07:25:55.299203+0000","last_peered":"2024-03-06T07:25:55.299203+0000","last_clean":"2024-03-06T07:25:55.299203+0000","last_became_active":"2024-03-06T07:25:55.296919+0000","last_became_peered":"2024-03-06T07:25:55.296919+0000","last_unstale":"2024-03-06T07:25:55.299203+0000","last_undegraded":"2024-03-06T07:25:55.299203+0000","last_fullsized":"2024-03-06T07:25:55.299203+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:55:57.004976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.283857+0000","last_change":"2024-03-06T07:25:54.293844+0000","last_active":"2024-03-06T07:25:55.283857+0000","last_peered":"2024-03-06T07:25:55.283857+0000","last_clean":"2024-03-06T07:25:55.283857+0000","last_became_active":"2024-03-06T07:25:54.293582+0000","last_became_peered":"2024-03-06T07:25:54.293582+0000","last_unstale":"2024-03-06T07:25:55.283857+0000","last_undegraded":"2024-03-06T07:25:55.283857+0000","last_fullsized":"2024-03-06T07:25:55.283857+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:52:19.113453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.305684+0000","last_change":"2024-03-06T07:25:55.304754+0000","last_active":"2024-03-06T07:25:55.305684+0000","last_peered":"2024-03-06T07:25:55.305684+0000","last_clean":"2024-03-06T07:25:55.305684+0000","last_became_active":"2024-03-06T07:25:55.304416+0000","last_became_peered":"2024-03-06T07:25:55.304416+0000","last_unstale":"2024-03-06T07:25:55.305684+0000","last_undegraded":"2024-03-06T07:25:55.305684+0000","last_fullsized":"2024-03-06T07:25:55.305684+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:30:51.700835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.283013+0000","last_change":"2024-03-06T07:25:54.296247+0000","last_active":"2024-03-06T07:25:55.283013+0000","last_peered":"2024-03-06T07:25:55.283013+0000","last_clean":"2024-03-06T07:25:55.283013+0000","last_became_active":"2024-03-06T07:25:54.295983+0000","last_became_peered":"2024-03-06T07:25:54.295983+0000","last_unstale":"2024-03-06T07:25:55.283013+0000","last_undegraded":"2024-03-06T07:25:55.283013+0000","last_fullsized":"2024-03-06T07:25:55.283013+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:54.868183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.303448+0000","last_change":"2024-03-06T07:25:55.299284+0000","last_active":"2024-03-06T07:25:55.303448+0000","last_peered":"2024-03-06T07:25:55.303448+0000","last_clean":"2024-03-06T07:25:55.303448+0000","last_became_active":"2024-03-06T07:25:55.298960+0000","last_became_peered":"2024-03-06T07:25:55.298960+0000","last_unstale":"2024-03-06T07:25:55.303448+0000","last_undegraded":"2024-03-06T07:25:55.303448+0000","last_fullsized":"2024-03-06T07:25:55.303448+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:30:51.700835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"81'3","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.317228+0000","last_change":"2024-03-06T07:25:54.292027+0000","last_active":"2024-03-06T07:25:55.317228+0000","last_peered":"2024-03-06T07:25:55.317228+0000","last_clean":"2024-03-06T07:25:55.317228+0000","last_became_active":"2024-03-06T07:25:54.291681+0000","last_became_peered":"2024-03-06T07:25:54.291681+0000","last_unstale":"2024-03-06T07:25:55.317228+0000","last_undegraded":"2024-03-06T07:25:55.317228+0000","last_fullsized":"2024-03-06T07:25:55.317228+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:54.252846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.302831+0000","last_change":"2024-03-06T07:25:55.299922+0000","last_active":"2024-03-06T07:25:55.302831+0000","last_peered":"2024-03-06T07:25:55.302831+0000","last_clean":"2024-03-06T07:25:55.302831+0000","last_became_active":"2024-03-06T07:25:55.299617+0000","last_became_peered":"2024-03-06T07:25:55.299617+0000","last_unstale":"2024-03-06T07:25:55.302831+0000","last_undegraded":"2024-03-06T07:25:55.302831+0000","last_fullsized":"2024-03-06T07:25:55.302831+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:13.381230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"81'6","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.319810+0000","last_change":"2024-03-06T07:25:54.291313+0000","last_active":"2024-03-06T07:25:55.319810+0000","last_peered":"2024-03-06T07:25:55.319810+0000","last_clean":"2024-03-06T07:25:55.319810+0000","last_became_active":"2024-03-06T07:25:54.291033+0000","last_became_peered":"2024-03-06T07:25:54.291033+0000","last_unstale":"2024-03-06T07:25:55.319810+0000","last_undegraded":"2024-03-06T07:25:55.319810+0000","last_fullsized":"2024-03-06T07:25:55.319810+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:46.309024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.308971+0000","last_change":"2024-03-06T07:25:55.304419+0000","last_active":"2024-03-06T07:25:55.308971+0000","last_peered":"2024-03-06T07:25:55.308971+0000","last_clean":"2024-03-06T07:25:55.308971+0000","last_became_active":"2024-03-06T07:25:55.304096+0000","last_became_peered":"2024-03-06T07:25:55.304096+0000","last_unstale":"2024-03-06T07:25:55.308971+0000","last_undegraded":"2024-03-06T07:25:55.308971+0000","last_fullsized":"2024-03-06T07:25:55.308971+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:26.561006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"81'2","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.319953+0000","last_change":"2024-03-06T07:25:54.290233+0000","last_active":"2024-03-06T07:25:55.319953+0000","last_peered":"2024-03-06T07:25:55.319953+0000","last_clean":"2024-03-06T07:25:55.319953+0000","last_became_active":"2024-03-06T07:25:54.289942+0000","last_became_peered":"2024-03-06T07:25:54.289942+0000","last_unstale":"2024-03-06T07:25:55.319953+0000","last_undegraded":"2024-03-06T07:25:55.319953+0000","last_fullsized":"2024-03-06T07:25:55.319953+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:50.824458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.302823+0000","last_change":"2024-03-06T07:25:55.298840+0000","last_active":"2024-03-06T07:25:55.302823+0000","last_peered":"2024-03-06T07:25:55.302823+0000","last_clean":"2024-03-06T07:25:55.302823+0000","last_became_active":"2024-03-06T07:25:55.298556+0000","last_became_peered":"2024-03-06T07:25:55.298556+0000","last_unstale":"2024-03-06T07:25:55.302823+0000","last_undegraded":"2024-03-06T07:25:55.302823+0000","last_fullsized":"2024-03-06T07:25:55.302823+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:27:15.358402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.288550+0000","last_change":"2024-03-06T07:25:54.294006+0000","last_active":"2024-03-06T07:25:55.288550+0000","last_peered":"2024-03-06T07:25:55.288550+0000","last_clean":"2024-03-06T07:25:55.288550+0000","last_became_active":"2024-03-06T07:25:54.293606+0000","last_became_peered":"2024-03-06T07:25:54.293606+0000","last_unstale":"2024-03-06T07:25:55.288550+0000","last_undegraded":"2024-03-06T07:25:55.288550+0000","last_fullsized":"2024-03-06T07:25:55.288550+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:19.456815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.316886+0000","last_change":"2024-03-06T07:25:55.312501+0000","last_active":"2024-03-06T07:25:55.316886+0000","last_peered":"2024-03-06T07:25:55.316886+0000","last_clean":"2024-03-06T07:25:55.316886+0000","last_became_active":"2024-03-06T07:25:55.312229+0000","last_became_peered":"2024-03-06T07:25:55.312229+0000","last_unstale":"2024-03-06T07:25:55.316886+0000","last_undegraded":"2024-03-06T07:25:55.316886+0000","last_fullsized":"2024-03-06T07:25:55.316886+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:02:48.503986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.282982+0000","last_change":"2024-03-06T07:25:54.295701+0000","last_active":"2024-03-06T07:25:55.282982+0000","last_peered":"2024-03-06T07:25:55.282982+0000","last_clean":"2024-03-06T07:25:55.282982+0000","last_became_active":"2024-03-06T07:25:54.295359+0000","last_became_peered":"2024-03-06T07:25:54.295359+0000","last_unstale":"2024-03-06T07:25:55.282982+0000","last_undegraded":"2024-03-06T07:25:55.282982+0000","last_fullsized":"2024-03-06T07:25:55.282982+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:10:05.666533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.296244+0000","last_change":"2024-03-06T07:25:55.289553+0000","last_active":"2024-03-06T07:25:55.296244+0000","last_peered":"2024-03-06T07:25:55.296244+0000","last_clean":"2024-03-06T07:25:55.296244+0000","last_became_active":"2024-03-06T07:25:55.289273+0000","last_became_peered":"2024-03-06T07:25:55.289273+0000","last_unstale":"2024-03-06T07:25:55.296244+0000","last_undegraded":"2024-03-06T07:25:55.296244+0000","last_fullsized":"2024-03-06T07:25:55.296244+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:09.830105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"81'8","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.315188+0000","last_change":"2024-03-06T07:25:54.297143+0000","last_active":"2024-03-06T07:25:55.315188+0000","last_peered":"2024-03-06T07:25:55.315188+0000","last_clean":"2024-03-06T07:25:55.315188+0000","last_became_active":"2024-03-06T07:25:54.296886+0000","last_became_peered":"2024-03-06T07:25:54.296886+0000","last_unstale":"2024-03-06T07:25:55.315188+0000","last_undegraded":"2024-03-06T07:25:55.315188+0000","last_fullsized":"2024-03-06T07:25:55.315188+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:10:05.666533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.294795+0000","last_change":"2024-03-06T07:25:55.291671+0000","last_active":"2024-03-06T07:25:55.294795+0000","last_peered":"2024-03-06T07:25:55.294795+0000","last_clean":"2024-03-06T07:25:55.294795+0000","last_became_active":"2024-03-06T07:25:55.291395+0000","last_became_peered":"2024-03-06T07:25:55.291395+0000","last_unstale":"2024-03-06T07:25:55.294795+0000","last_undegraded":"2024-03-06T07:25:55.294795+0000","last_fullsized":"2024-03-06T07:25:55.294795+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:05.339370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"81'1","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.309053+0000","last_change":"2024-03-06T07:25:54.292182+0000","last_active":"2024-03-06T07:25:55.309053+0000","last_peered":"2024-03-06T07:25:55.309053+0000","last_clean":"2024-03-06T07:25:55.309053+0000","last_became_active":"2024-03-06T07:25:54.291904+0000","last_became_peered":"2024-03-06T07:25:54.291904+0000","last_unstale":"2024-03-06T07:25:55.309053+0000","last_undegraded":"2024-03-06T07:25:55.309053+0000","last_fullsized":"2024-03-06T07:25:55.309053+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:51.548796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.296989+0000","last_change":"2024-03-06T07:25:55.293197+0000","last_active":"2024-03-06T07:25:55.296989+0000","last_peered":"2024-03-06T07:25:55.296989+0000","last_clean":"2024-03-06T07:25:55.296989+0000","last_became_active":"2024-03-06T07:25:55.292949+0000","last_became_peered":"2024-03-06T07:25:55.292949+0000","last_unstale":"2024-03-06T07:25:55.296989+0000","last_undegraded":"2024-03-06T07:25:55.296989+0000","last_fullsized":"2024-03-06T07:25:55.296989+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:10:06.671054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"81'2","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.311538+0000","last_change":"2024-03-06T07:25:54.296445+0000","last_active":"2024-03-06T07:25:55.311538+0000","last_peered":"2024-03-06T07:25:55.311538+0000","last_clean":"2024-03-06T07:25:55.311538+0000","last_became_active":"2024-03-06T07:25:54.296169+0000","last_became_peered":"2024-03-06T07:25:54.296169+0000","last_unstale":"2024-03-06T07:25:55.311538+0000","last_undegraded":"2024-03-06T07:25:55.311538+0000","last_fullsized":"2024-03-06T07:25:55.311538+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:46.309024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.302811+0000","last_change":"2024-03-06T07:25:55.297736+0000","last_active":"2024-03-06T07:25:55.302811+0000","last_peered":"2024-03-06T07:25:55.302811+0000","last_clean":"2024-03-06T07:25:55.302811+0000","last_became_active":"2024-03-06T07:25:55.297466+0000","last_became_peered":"2024-03-06T07:25:55.297466+0000","last_unstale":"2024-03-06T07:25:55.302811+0000","last_undegraded":"2024-03-06T07:25:55.302811+0000","last_fullsized":"2024-03-06T07:25:55.302811+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:51:44.302090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.294184+0000","last_change":"2024-03-06T07:25:54.291395+0000","last_active":"2024-03-06T07:25:55.294184+0000","last_peered":"2024-03-06T07:25:55.294184+0000","last_clean":"2024-03-06T07:25:55.294184+0000","last_became_active":"2024-03-06T07:25:54.291139+0000","last_became_peered":"2024-03-06T07:25:54.291139+0000","last_unstale":"2024-03-06T07:25:55.294184+0000","last_undegraded":"2024-03-06T07:25:55.294184+0000","last_fullsized":"2024-03-06T07:25:55.294184+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:52:03.922846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.320172+0000","last_change":"2024-03-06T07:25:55.317720+0000","last_active":"2024-03-06T07:25:55.320172+0000","last_peered":"2024-03-06T07:25:55.320172+0000","last_clean":"2024-03-06T07:25:55.320172+0000","last_became_active":"2024-03-06T07:25:55.317371+0000","last_became_peered":"2024-03-06T07:25:55.317371+0000","last_unstale":"2024-03-06T07:25:55.320172+0000","last_undegraded":"2024-03-06T07:25:55.320172+0000","last_fullsized":"2024-03-06T07:25:55.320172+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:55:32.599045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"81'2","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.309368+0000","last_change":"2024-03-06T07:25:54.288785+0000","last_active":"2024-03-06T07:25:55.309368+0000","last_peered":"2024-03-06T07:25:55.309368+0000","last_clean":"2024-03-06T07:25:55.309368+0000","last_became_active":"2024-03-06T07:25:54.288529+0000","last_became_peered":"2024-03-06T07:25:54.288529+0000","last_unstale":"2024-03-06T07:25:55.309368+0000","last_undegraded":"2024-03-06T07:25:55.309368+0000","last_fullsized":"2024-03-06T07:25:55.309368+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:27:14.353882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.293885+0000","last_change":"2024-03-06T07:25:55.290064+0000","last_active":"2024-03-06T07:25:55.293885+0000","last_peered":"2024-03-06T07:25:55.293885+0000","last_clean":"2024-03-06T07:25:55.293885+0000","last_became_active":"2024-03-06T07:25:55.289798+0000","last_became_peered":"2024-03-06T07:25:55.289798+0000","last_unstale":"2024-03-06T07:25:55.293885+0000","last_undegraded":"2024-03-06T07:25:55.293885+0000","last_fullsized":"2024-03-06T07:25:55.293885+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:26:35.055387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.302641+0000","last_change":"2024-03-06T07:25:54.293118+0000","last_active":"2024-03-06T07:25:55.302641+0000","last_peered":"2024-03-06T07:25:55.302641+0000","last_clean":"2024-03-06T07:25:55.302641+0000","last_became_active":"2024-03-06T07:25:54.292868+0000","last_became_peered":"2024-03-06T07:25:54.292868+0000","last_unstale":"2024-03-06T07:25:55.302641+0000","last_undegraded":"2024-03-06T07:25:55.302641+0000","last_fullsized":"2024-03-06T07:25:55.302641+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:08.825584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.304263+0000","last_change":"2024-03-06T07:25:55.301253+0000","last_active":"2024-03-06T07:25:55.304263+0000","last_peered":"2024-03-06T07:25:55.304263+0000","last_clean":"2024-03-06T07:25:55.304263+0000","last_became_active":"2024-03-06T07:25:55.300954+0000","last_became_peered":"2024-03-06T07:25:55.300954+0000","last_unstale":"2024-03-06T07:25:55.304263+0000","last_undegraded":"2024-03-06T07:25:55.304263+0000","last_fullsized":"2024-03-06T07:25:55.304263+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:17.542986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.297539+0000","last_change":"2024-03-06T07:25:54.294699+0000","last_active":"2024-03-06T07:25:55.297539+0000","last_peered":"2024-03-06T07:25:55.297539+0000","last_clean":"2024-03-06T07:25:55.297539+0000","last_became_active":"2024-03-06T07:25:54.294443+0000","last_became_peered":"2024-03-06T07:25:54.294443+0000","last_unstale":"2024-03-06T07:25:55.297539+0000","last_undegraded":"2024-03-06T07:25:55.297539+0000","last_fullsized":"2024-03-06T07:25:55.297539+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:21.695732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.297203+0000","last_change":"2024-03-06T07:25:55.296010+0000","last_active":"2024-03-06T07:25:55.297203+0000","last_peered":"2024-03-06T07:25:55.297203+0000","last_clean":"2024-03-06T07:25:55.297203+0000","last_became_active":"2024-03-06T07:25:55.295733+0000","last_became_peered":"2024-03-06T07:25:55.295733+0000","last_unstale":"2024-03-06T07:25:55.297203+0000","last_undegraded":"2024-03-06T07:25:55.297203+0000","last_fullsized":"2024-03-06T07:25:55.297203+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:52:04.927367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.286259+0000","last_change":"2024-03-06T07:25:54.291208+0000","last_active":"2024-03-06T07:25:55.286259+0000","last_peered":"2024-03-06T07:25:55.286259+0000","last_clean":"2024-03-06T07:25:55.286259+0000","last_became_active":"2024-03-06T07:25:54.290582+0000","last_became_peered":"2024-03-06T07:25:54.290582+0000","last_unstale":"2024-03-06T07:25:55.286259+0000","last_undegraded":"2024-03-06T07:25:55.286259+0000","last_fullsized":"2024-03-06T07:25:55.286259+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:07:08.578613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.294547+0000","last_change":"2024-03-06T07:25:55.293174+0000","last_active":"2024-03-06T07:25:55.294547+0000","last_peered":"2024-03-06T07:25:55.294547+0000","last_clean":"2024-03-06T07:25:55.294547+0000","last_became_active":"2024-03-06T07:25:55.292343+0000","last_became_peered":"2024-03-06T07:25:55.292343+0000","last_unstale":"2024-03-06T07:25:55.294547+0000","last_undegraded":"2024-03-06T07:25:55.294547+0000","last_fullsized":"2024-03-06T07:25:55.294547+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:48.091148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"81'2","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.317089+0000","last_change":"2024-03-06T07:25:54.296487+0000","last_active":"2024-03-06T07:25:55.317089+0000","last_peered":"2024-03-06T07:25:55.317089+0000","last_clean":"2024-03-06T07:25:55.317089+0000","last_became_active":"2024-03-06T07:25:54.296235+0000","last_became_peered":"2024-03-06T07:25:54.296235+0000","last_unstale":"2024-03-06T07:25:55.317089+0000","last_undegraded":"2024-03-06T07:25:55.317089+0000","last_fullsized":"2024-03-06T07:25:55.317089+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:30.492845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.300876+0000","last_change":"2024-03-06T07:25:55.296551+0000","last_active":"2024-03-06T07:25:55.300876+0000","last_peered":"2024-03-06T07:25:55.300876+0000","last_clean":"2024-03-06T07:25:55.300876+0000","last_became_active":"2024-03-06T07:25:55.296204+0000","last_became_peered":"2024-03-06T07:25:55.296204+0000","last_unstale":"2024-03-06T07:25:55.300876+0000","last_undegraded":"2024-03-06T07:25:55.300876+0000","last_fullsized":"2024-03-06T07:25:55.300876+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:15.632815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.284017+0000","last_change":"2024-03-06T07:25:54.292685+0000","last_active":"2024-03-06T07:25:55.284017+0000","last_peered":"2024-03-06T07:25:55.284017+0000","last_clean":"2024-03-06T07:25:55.284017+0000","last_became_active":"2024-03-06T07:25:54.291463+0000","last_became_peered":"2024-03-06T07:25:54.291463+0000","last_unstale":"2024-03-06T07:25:55.284017+0000","last_undegraded":"2024-03-06T07:25:55.284017+0000","last_fullsized":"2024-03-06T07:25:55.284017+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:18.080096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.305393+0000","last_change":"2024-03-06T07:25:55.300773+0000","last_active":"2024-03-06T07:25:55.305393+0000","last_peered":"2024-03-06T07:25:55.305393+0000","last_clean":"2024-03-06T07:25:55.305393+0000","last_became_active":"2024-03-06T07:25:55.300434+0000","last_became_peered":"2024-03-06T07:25:55.300434+0000","last_unstale":"2024-03-06T07:25:55.305393+0000","last_undegraded":"2024-03-06T07:25:55.305393+0000","last_fullsized":"2024-03-06T07:25:55.305393+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:57.470091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.282666+0000","last_change":"2024-03-06T07:25:54.291158+0000","last_active":"2024-03-06T07:25:55.282666+0000","last_peered":"2024-03-06T07:25:55.282666+0000","last_clean":"2024-03-06T07:25:55.282666+0000","last_became_active":"2024-03-06T07:25:54.290849+0000","last_became_peered":"2024-03-06T07:25:54.290849+0000","last_unstale":"2024-03-06T07:25:55.282666+0000","last_undegraded":"2024-03-06T07:25:55.282666+0000","last_fullsized":"2024-03-06T07:25:55.282666+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:46:06.100952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.302848+0000","last_change":"2024-03-06T07:25:55.299979+0000","last_active":"2024-03-06T07:25:55.302848+0000","last_peered":"2024-03-06T07:25:55.302848+0000","last_clean":"2024-03-06T07:25:55.302848+0000","last_became_active":"2024-03-06T07:25:55.299260+0000","last_became_peered":"2024-03-06T07:25:55.299260+0000","last_unstale":"2024-03-06T07:25:55.302848+0000","last_undegraded":"2024-03-06T07:25:55.302848+0000","last_fullsized":"2024-03-06T07:25:55.302848+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:52:04.927367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.285252+0000","last_change":"2024-03-06T07:25:54.292616+0000","last_active":"2024-03-06T07:25:55.285252+0000","last_peered":"2024-03-06T07:25:55.285252+0000","last_clean":"2024-03-06T07:25:55.285252+0000","last_became_active":"2024-03-06T07:25:54.292359+0000","last_became_peered":"2024-03-06T07:25:54.292359+0000","last_unstale":"2024-03-06T07:25:55.285252+0000","last_undegraded":"2024-03-06T07:25:55.285252+0000","last_fullsized":"2024-03-06T07:25:55.285252+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:44.841033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.293853+0000","last_change":"2024-03-06T07:25:55.290783+0000","last_active":"2024-03-06T07:25:55.293853+0000","last_peered":"2024-03-06T07:25:55.293853+0000","last_clean":"2024-03-06T07:25:55.293853+0000","last_became_active":"2024-03-06T07:25:55.290485+0000","last_became_peered":"2024-03-06T07:25:55.290485+0000","last_unstale":"2024-03-06T07:25:55.293853+0000","last_undegraded":"2024-03-06T07:25:55.293853+0000","last_fullsized":"2024-03-06T07:25:55.293853+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:51:44.302090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.293745+0000","last_change":"2024-03-06T07:25:54.296533+0000","last_active":"2024-03-06T07:25:55.293745+0000","last_peered":"2024-03-06T07:25:55.293745+0000","last_clean":"2024-03-06T07:25:55.293745+0000","last_became_active":"2024-03-06T07:25:54.296291+0000","last_became_peered":"2024-03-06T07:25:54.296291+0000","last_unstale":"2024-03-06T07:25:55.293745+0000","last_undegraded":"2024-03-06T07:25:55.293745+0000","last_fullsized":"2024-03-06T07:25:55.293745+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:56.465571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.309087+0000","last_change":"2024-03-06T07:25:55.303601+0000","last_active":"2024-03-06T07:25:55.309087+0000","last_peered":"2024-03-06T07:25:55.309087+0000","last_clean":"2024-03-06T07:25:55.309087+0000","last_became_active":"2024-03-06T07:25:55.303315+0000","last_became_peered":"2024-03-06T07:25:55.303315+0000","last_unstale":"2024-03-06T07:25:55.309087+0000","last_undegraded":"2024-03-06T07:25:55.309087+0000","last_fullsized":"2024-03-06T07:25:55.309087+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:35.427361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.288404+0000","last_change":"2024-03-06T07:25:54.290731+0000","last_active":"2024-03-06T07:25:55.288404+0000","last_peered":"2024-03-06T07:25:55.288404+0000","last_clean":"2024-03-06T07:25:55.288404+0000","last_became_active":"2024-03-06T07:25:54.290414+0000","last_became_peered":"2024-03-06T07:25:54.290414+0000","last_unstale":"2024-03-06T07:25:55.288404+0000","last_undegraded":"2024-03-06T07:25:55.288404+0000","last_fullsized":"2024-03-06T07:25:55.288404+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:47:09.862504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.300857+0000","last_change":"2024-03-06T07:25:55.297028+0000","last_active":"2024-03-06T07:25:55.300857+0000","last_peered":"2024-03-06T07:25:55.300857+0000","last_clean":"2024-03-06T07:25:55.300857+0000","last_became_active":"2024-03-06T07:25:55.296767+0000","last_became_peered":"2024-03-06T07:25:55.296767+0000","last_unstale":"2024-03-06T07:25:55.300857+0000","last_undegraded":"2024-03-06T07:25:55.300857+0000","last_fullsized":"2024-03-06T07:25:55.300857+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:07:02.526172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.283519+0000","last_change":"2024-03-06T07:25:54.293230+0000","last_active":"2024-03-06T07:25:55.283519+0000","last_peered":"2024-03-06T07:25:55.283519+0000","last_clean":"2024-03-06T07:25:55.283519+0000","last_became_active":"2024-03-06T07:25:54.292980+0000","last_became_peered":"2024-03-06T07:25:54.292980+0000","last_unstale":"2024-03-06T07:25:55.283519+0000","last_undegraded":"2024-03-06T07:25:55.283519+0000","last_fullsized":"2024-03-06T07:25:55.283519+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:54:34.915295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.306581+0000","last_change":"2024-03-06T07:25:55.303769+0000","last_active":"2024-03-06T07:25:55.306581+0000","last_peered":"2024-03-06T07:25:55.306581+0000","last_clean":"2024-03-06T07:25:55.306581+0000","last_became_active":"2024-03-06T07:25:55.303526+0000","last_became_peered":"2024-03-06T07:25:55.303526+0000","last_unstale":"2024-03-06T07:25:55.306581+0000","last_undegraded":"2024-03-06T07:25:55.306581+0000","last_fullsized":"2024-03-06T07:25:55.306581+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:13.381230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.288075+0000","last_change":"2024-03-06T07:25:54.294078+0000","last_active":"2024-03-06T07:25:55.288075+0000","last_peered":"2024-03-06T07:25:55.288075+0000","last_clean":"2024-03-06T07:25:55.288075+0000","last_became_active":"2024-03-06T07:25:54.293826+0000","last_became_peered":"2024-03-06T07:25:54.293826+0000","last_unstale":"2024-03-06T07:25:55.288075+0000","last_undegraded":"2024-03-06T07:25:55.288075+0000","last_fullsized":"2024-03-06T07:25:55.288075+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:51.548796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.312612+0000","last_change":"2024-03-06T07:25:55.294179+0000","last_active":"2024-03-06T07:25:55.312612+0000","last_peered":"2024-03-06T07:25:55.312612+0000","last_clean":"2024-03-06T07:25:55.312612+0000","last_became_active":"2024-03-06T07:25:55.293871+0000","last_became_peered":"2024-03-06T07:25:55.293871+0000","last_unstale":"2024-03-06T07:25:55.312612+0000","last_undegraded":"2024-03-06T07:25:55.312612+0000","last_fullsized":"2024-03-06T07:25:55.312612+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:54.273222+0000","last_clean_scrub_stamp":"2024-03-06T07:25:54.273222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:55:32.599045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"81'1","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.317266+0000","last_change":"2024-03-06T07:25:54.295348+0000","last_active":"2024-03-06T07:25:55.317266+0000","last_peered":"2024-03-06T07:25:55.317266+0000","last_clean":"2024-03-06T07:25:55.317266+0000","last_became_active":"2024-03-06T07:25:54.295035+0000","last_became_peered":"2024-03-06T07:25:54.295035+0000","last_unstale":"2024-03-06T07:25:55.317266+0000","last_undegraded":"2024-03-06T07:25:55.317266+0000","last_fullsized":"2024-03-06T07:25:55.317266+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:53.268701+0000","last_clean_scrub_stamp":"2024-03-06T07:25:53.268701+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:25.556485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":281,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:55.288705+0000","last_change":"2024-03-06T07:21:47.553693+0000","last_active":"2024-03-06T07:25:55.288705+0000","last_peered":"2024-03-06T07:25:55.288705+0000","last_clean":"2024-03-06T07:25:55.288705+0000","last_became_active":"2024-03-06T07:21:47.552664+0000","last_became_peered":"2024-03-06T07:21:47.552664+0000","last_unstale":"2024-03-06T07:25:55.288705+0000","last_undegraded":"2024-03-06T07:25:55.288705+0000","last_fullsized":"2024-03-06T07:25:55.288705+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_clean_scrub_stamp":"2024-03-06T07:21:46.362028+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:02.916426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574903,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":1005462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574901,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32140,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339700,"statfs":{"total":96636764160,"available":96603852800,"internally_reserved":0,"allocated":983040,"data_stored":415094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574902,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32144,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339696,"statfs":{"total":96636764160,"available":96603848704,"internally_reserved":0,"allocated":983040,"data_stored":415094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574903,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32144,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339696,"statfs":{"total":96636764160,"available":96603848704,"internally_reserved":0,"allocated":983040,"data_stored":415094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574902,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32144,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339696,"statfs":{"total":96636764160,"available":96603848704,"internally_reserved":0,"allocated":983040,"data_stored":415094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574902,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32144,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339696,"statfs":{"total":96636764160,"available":96603848704,"internally_reserved":0,"allocated":983040,"data_stored":415094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574903,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32172,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339668,"statfs":{"total":96636764160,"available":96603820032,"internally_reserved":0,"allocated":1007616,"data_stored":422063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574902,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":1005462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:58.616 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:25:58.617 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:25:58.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.762+0000 7f3a739bc640 1 Processor -- start 2024-03-06T07:25:58.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.763+0000 7f3a739bc640 1 -- start start 2024-03-06T07:25:58.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.763+0000 7f3a739bc640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f3a6c1614d0 0x7f3a6c0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.763+0000 7f3a739bc640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a6c0d24b0 0x7f3a6c0b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.763+0000 7f3a739bc640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a6c1618b0 con 0x7f3a6c0d1910 2024-03-06T07:25:58.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.763+0000 7f3a739bc640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a6c0cf400 con 0x7f3a6c1614d0 2024-03-06T07:25:58.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.763+0000 7f3a739bc640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6c0cf6a0 con 0x7f3a6c0d24b0 2024-03-06T07:25:58.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.764+0000 7f3a71758640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f3a6c1614d0 0x7f3a6c0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.764+0000 7f3a71f59640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a6c0d24b0 0x7f3a6c0b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.764+0000 7f3a71758640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f3a6c1614d0 0x7f3a6c0d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:60682/0 (socket says 172.21.15.83:60682) 2024-03-06T07:25:58.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.764+0000 7f3a70f57640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f3a6c0d1910 0x7f3a6c0d1da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:44588/0 (socket says 172.21.15.83:44588) 2024-03-06T07:25:58.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.764+0000 7f3a71758640 1 -- 172.21.15.83:0/3465920714 learned_addr learned my addr 172.21.15.83:0/3465920714 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:58.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.764+0000 7f3a63fff640 1 -- 172.21.15.83:0/3465920714 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 193447325 0 0) 0x7f3a6c1618b0 con 0x7f3a6c0d1910 2024-03-06T07:25:58.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.764+0000 7f3a63fff640 1 -- 172.21.15.83:0/3465920714 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a58003520 con 0x7f3a6c0d1910 2024-03-06T07:25:58.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.764+0000 7f3a71758640 1 -- 172.21.15.83:0/3465920714 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a6c0d24b0 msgr2=0x7f3a6c0b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:58.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.764+0000 7f3a71758640 1 --2- 172.21.15.83:0/3465920714 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a6c0d24b0 0x7f3a6c0b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.764+0000 7f3a71758640 1 -- 172.21.15.83:0/3465920714 >> v1:172.21.15.183:6789/0 conn(0x7f3a6c0d1910 legacy=0x7f3a6c0d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.764+0000 7f3a71758640 1 -- 172.21.15.83:0/3465920714 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a6c0cf980 con 0x7f3a6c1614d0 2024-03-06T07:25:58.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.764+0000 7f3a71f59640 1 --2- 172.21.15.83:0/3465920714 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a6c0d24b0 0x7f3a6c0b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:58.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.765+0000 7f3a71758640 1 --2- 172.21.15.83:0/3465920714 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a6c1614d0 0x7f3a6c0d4860 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f3a68009870 tx=0x7f3a6802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2deabdc064f50d8b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:58.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.765+0000 7f3a63fff640 1 -- 172.21.15.83:0/3465920714 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a68034070 con 0x7f3a6c1614d0 2024-03-06T07:25:58.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.765+0000 7f3a63fff640 1 -- 172.21.15.83:0/3465920714 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a68030840 con 0x7f3a6c1614d0 2024-03-06T07:25:58.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.765+0000 7f3a63fff640 1 -- 172.21.15.83:0/3465920714 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a68030c10 con 0x7f3a6c1614d0 2024-03-06T07:25:58.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.766+0000 7f3a739bc640 1 -- 172.21.15.83:0/3465920714 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a6c1614d0 msgr2=0x7f3a6c0d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.766+0000 7f3a739bc640 1 --2- 172.21.15.83:0/3465920714 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a6c1614d0 0x7f3a6c0d4860 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f3a68009870 tx=0x7f3a6802b980 comp rx=0 tx=0).stop 2024-03-06T07:25:58.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.766+0000 7f3a739bc640 1 -- 172.21.15.83:0/3465920714 shutdown_connections 2024-03-06T07:25:58.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.766+0000 7f3a739bc640 1 --2- 172.21.15.83:0/3465920714 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a6c0d24b0 0x7f3a6c0b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.766+0000 7f3a739bc640 1 --2- 172.21.15.83:0/3465920714 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a6c1614d0 0x7f3a6c0d4860 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.766+0000 7f3a739bc640 1 -- 172.21.15.83:0/3465920714 >> 172.21.15.83:0/3465920714 conn(0x7f3a6c0d3d40 msgr2=0x7f3a6c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:58.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.766+0000 7f3a739bc640 1 -- 172.21.15.83:0/3465920714 shutdown_connections 2024-03-06T07:25:58.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.766+0000 7f3a739bc640 1 -- 172.21.15.83:0/3465920714 wait complete. 2024-03-06T07:25:58.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.768+0000 7f3a739bc640 1 Processor -- start 2024-03-06T07:25:58.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.768+0000 7f3a739bc640 1 -- start start 2024-03-06T07:25:58.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.769+0000 7f3a739bc640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a6c0d1910 0x7f3a6c1575a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.769+0000 7f3a739bc640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a6c0d24b0 0x7f3a6c0ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.769+0000 7f3a739bc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a6c1614d0 0x7f3a6c0c9b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.769+0000 7f3a739bc640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6c0b1370 con 0x7f3a6c1614d0 2024-03-06T07:25:58.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.769+0000 7f3a739bc640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6c0b0910 con 0x7f3a6c0d1910 2024-03-06T07:25:58.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.769+0000 7f3a739bc640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6c0b14f0 con 0x7f3a6c0d24b0 2024-03-06T07:25:58.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.769+0000 7f3a71f59640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a6c1614d0 0x7f3a6c0c9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.769+0000 7f3a71f59640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a6c1614d0 0x7f3a6c0c9b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55596/0 (socket says 172.21.15.83:55596) 2024-03-06T07:25:58.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.769+0000 7f3a71f59640 1 -- 172.21.15.83:0/1106324212 learned_addr learned my addr 172.21.15.83:0/1106324212 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:58.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.769+0000 7f3a70f57640 1 --2- 172.21.15.83:0/1106324212 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a6c0d24b0 0x7f3a6c0ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.769+0000 7f3a71758640 1 --2- 172.21.15.83:0/1106324212 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a6c0d1910 0x7f3a6c1575a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.769+0000 7f3a71f59640 1 -- 172.21.15.83:0/1106324212 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a6c0d24b0 msgr2=0x7f3a6c0ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.769+0000 7f3a71f59640 1 --2- 172.21.15.83:0/1106324212 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a6c0d24b0 0x7f3a6c0ccfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.769+0000 7f3a71f59640 1 -- 172.21.15.83:0/1106324212 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a6c0d1910 msgr2=0x7f3a6c1575a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.769+0000 7f3a71f59640 1 --2- 172.21.15.83:0/1106324212 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a6c0d1910 0x7f3a6c1575a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.769+0000 7f3a71f59640 1 -- 172.21.15.83:0/1106324212 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a6c0cbf00 con 0x7f3a6c1614d0 2024-03-06T07:25:58.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.770+0000 7f3a71f59640 1 --2- 172.21.15.83:0/1106324212 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a6c1614d0 0x7f3a6c0c9b60 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f3a6400efc0 tx=0x7f3a64009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:58.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.770+0000 7f3a61ffb640 1 -- 172.21.15.83:0/1106324212 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a64018030 con 0x7f3a6c1614d0 2024-03-06T07:25:58.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.770+0000 7f3a61ffb640 1 -- 172.21.15.83:0/1106324212 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a6400cde0 con 0x7f3a6c1614d0 2024-03-06T07:25:58.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.770+0000 7f3a739bc640 1 -- 172.21.15.83:0/1106324212 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a6c0cc1e0 con 0x7f3a6c1614d0 2024-03-06T07:25:58.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.770+0000 7f3a61ffb640 1 -- 172.21.15.83:0/1106324212 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a64003bb0 con 0x7f3a6c1614d0 2024-03-06T07:25:58.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.770+0000 7f3a739bc640 1 -- 172.21.15.83:0/1106324212 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a6c0cca80 con 0x7f3a6c1614d0 2024-03-06T07:25:58.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.771+0000 7f3a739bc640 1 -- 172.21.15.83:0/1106324212 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a6c0ccc80 con 0x7f3a6c1614d0 2024-03-06T07:25:58.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.772+0000 7f3a61ffb640 1 -- 172.21.15.83:0/1106324212 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3a64005d10 con 0x7f3a6c1614d0 2024-03-06T07:25:58.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.772+0000 7f3a61ffb640 1 --2- 172.21.15.83:0/1106324212 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3a5006b500 0x7f3a5006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.772+0000 7f3a61ffb640 1 -- 172.21.15.83:0/1106324212 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3a6408aae0 con 0x7f3a6c1614d0 2024-03-06T07:25:58.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.772+0000 7f3a71758640 1 --2- 172.21.15.83:0/1106324212 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3a5006b500 0x7f3a5006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.773+0000 7f3a71758640 1 --2- 172.21.15.83:0/1106324212 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3a5006b500 0x7f3a5006d9c0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f3a68036000 tx=0x7f3a6803b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:58.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:58.777+0000 7f3a61ffb640 1 -- 172.21.15.83:0/1106324212 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3a6c0ccc80 con 0x7f3a6c1614d0 2024-03-06T07:25:59.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.022+0000 7f3a739bc640 1 -- 172.21.15.83:0/1106324212 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f3a6c068ff0 con 0x7f3a6c1614d0 2024-03-06T07:25:59.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.024+0000 7f3a61ffb640 1 -- 172.21.15.83:0/1106324212 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+147 (secure 0 0 0) 0x7f3a64022070 con 0x7f3a6c1614d0 2024-03-06T07:25:59.026 INFO:teuthology.orchestra.run.smithi083.stdout:[client.0] 2024-03-06T07:25:59.026 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBcGehlRwCIMhAA1DdbdQZPplu7LQOTK3BmSg== 2024-03-06T07:25:59.026 INFO:teuthology.orchestra.run.smithi083.stdout: caps mds = "allow" 2024-03-06T07:25:59.026 INFO:teuthology.orchestra.run.smithi083.stdout: caps mon = "allow r" 2024-03-06T07:25:59.026 INFO:teuthology.orchestra.run.smithi083.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:25:59.026 INFO:teuthology.orchestra.run.smithi083.stderr:updated caps for client.0 2024-03-06T07:25:59.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.026+0000 7f3a739bc640 1 -- 172.21.15.83:0/1106324212 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3a5006b500 msgr2=0x7f3a5006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.026+0000 7f3a739bc640 1 --2- 172.21.15.83:0/1106324212 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3a5006b500 0x7f3a5006d9c0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f3a68036000 tx=0x7f3a6803b040 comp rx=0 tx=0).stop 2024-03-06T07:25:59.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.026+0000 7f3a739bc640 1 -- 172.21.15.83:0/1106324212 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a6c1614d0 msgr2=0x7f3a6c0c9b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.026+0000 7f3a739bc640 1 --2- 172.21.15.83:0/1106324212 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a6c1614d0 0x7f3a6c0c9b60 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f3a6400efc0 tx=0x7f3a64009f90 comp rx=0 tx=0).stop 2024-03-06T07:25:59.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.026+0000 7f3a739bc640 1 -- 172.21.15.83:0/1106324212 shutdown_connections 2024-03-06T07:25:59.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.026+0000 7f3a739bc640 1 --2- 172.21.15.83:0/1106324212 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3a5006b500 0x7f3a5006d9c0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.026+0000 7f3a739bc640 1 --2- 172.21.15.83:0/1106324212 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a6c1614d0 0x7f3a6c0c9b60 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.026+0000 7f3a739bc640 1 --2- 172.21.15.83:0/1106324212 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a6c0d24b0 0x7f3a6c0ccfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.026+0000 7f3a739bc640 1 --2- 172.21.15.83:0/1106324212 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a6c0d1910 0x7f3a6c1575a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.026+0000 7f3a739bc640 1 -- 172.21.15.83:0/1106324212 >> 172.21.15.83:0/1106324212 conn(0x7f3a6c0d3d40 msgr2=0x7f3a6c0cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:59.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.027+0000 7f3a739bc640 1 -- 172.21.15.83:0/1106324212 shutdown_connections 2024-03-06T07:25:59.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.027+0000 7f3a739bc640 1 -- 172.21.15.83:0/1106324212 wait complete. 2024-03-06T07:25:59.038 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:25:59.039 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:59.040 DEBUG:teuthology.orchestra.run.smithi183:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:25:59.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.218+0000 7ffb4d9bc640 1 Processor -- start 2024-03-06T07:25:59.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.218+0000 7ffb4d9bc640 1 -- start start 2024-03-06T07:25:59.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.219+0000 7ffb4d9bc640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ffb4815dfe0 0x7ffb480cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.219+0000 7ffb4d9bc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ffb480cf830 0x7ffb480d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.219+0000 7ffb4d9bc640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffb480d27f0 0x7ffb480d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.219+0000 7ffb4d9bc640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb480b31e0 con 0x7ffb480cf830 2024-03-06T07:25:59.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.219+0000 7ffb4d9bc640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb480b3d60 con 0x7ffb4815dfe0 2024-03-06T07:25:59.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.219+0000 7ffb4d9bc640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb480b3430 con 0x7ffb480d27f0 2024-03-06T07:25:59.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.219+0000 7ffb467fc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ffb480cf830 0x7ffb480d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.219+0000 7ffb467fc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ffb480cf830 0x7ffb480d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55606/0 (socket says 172.21.15.83:55606) 2024-03-06T07:25:59.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.219+0000 7ffb467fc640 1 -- 172.21.15.83:0/3497117108 learned_addr learned my addr 172.21.15.83:0/3497117108 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:59.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.219+0000 7ffb477fe640 1 --2- 172.21.15.83:0/3497117108 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffb480d27f0 0x7ffb480d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.219+0000 7ffb46ffd640 1 --2- 172.21.15.83:0/3497117108 >> v2:172.21.15.183:3300/0 conn(0x7ffb4815dfe0 0x7ffb480cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.219+0000 7ffb467fc640 1 -- 172.21.15.83:0/3497117108 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffb480d27f0 msgr2=0x7ffb480d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:59.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.219+0000 7ffb467fc640 1 --2- 172.21.15.83:0/3497117108 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffb480d27f0 0x7ffb480d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.219+0000 7ffb467fc640 1 -- 172.21.15.83:0/3497117108 >> v2:172.21.15.183:3300/0 conn(0x7ffb4815dfe0 msgr2=0x7ffb480cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:59.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.219+0000 7ffb467fc640 1 --2- 172.21.15.83:0/3497117108 >> v2:172.21.15.183:3300/0 conn(0x7ffb4815dfe0 0x7ffb480cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.219+0000 7ffb467fc640 1 -- 172.21.15.83:0/3497117108 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb480b3710 con 0x7ffb480cf830 2024-03-06T07:25:59.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.219+0000 7ffb46ffd640 1 --2- 172.21.15.83:0/3497117108 >> v2:172.21.15.183:3300/0 conn(0x7ffb4815dfe0 0x7ffb480cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:59.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.220+0000 7ffb467fc640 1 --2- 172.21.15.83:0/3497117108 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb480cf830 0x7ffb480d22b0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7ffb48067990 tx=0x7ffb3c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=801e25e8761efc05 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:59.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.220+0000 7ffb45ffb640 1 -- 172.21.15.83:0/3497117108 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3c035020 con 0x7ffb480cf830 2024-03-06T07:25:59.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.220+0000 7ffb45ffb640 1 -- 172.21.15.83:0/3497117108 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb3c031870 con 0x7ffb480cf830 2024-03-06T07:25:59.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.220+0000 7ffb45ffb640 1 -- 172.21.15.83:0/3497117108 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3c031c40 con 0x7ffb480cf830 2024-03-06T07:25:59.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.221+0000 7ffb4d9bc640 1 -- 172.21.15.83:0/3497117108 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb480cf830 msgr2=0x7ffb480d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.221+0000 7ffb4d9bc640 1 --2- 172.21.15.83:0/3497117108 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb480cf830 0x7ffb480d22b0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7ffb48067990 tx=0x7ffb3c02b910 comp rx=0 tx=0).stop 2024-03-06T07:25:59.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.221+0000 7ffb4d9bc640 1 -- 172.21.15.83:0/3497117108 shutdown_connections 2024-03-06T07:25:59.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.221+0000 7ffb4d9bc640 1 --2- 172.21.15.83:0/3497117108 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffb480d27f0 0x7ffb480d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.221+0000 7ffb4d9bc640 1 --2- 172.21.15.83:0/3497117108 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb480cf830 0x7ffb480d22b0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.221+0000 7ffb4d9bc640 1 --2- 172.21.15.83:0/3497117108 >> v2:172.21.15.183:3300/0 conn(0x7ffb4815dfe0 0x7ffb480cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.221+0000 7ffb4d9bc640 1 -- 172.21.15.83:0/3497117108 >> 172.21.15.83:0/3497117108 conn(0x7ffb480d3cd0 msgr2=0x7ffb480d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:59.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.221+0000 7ffb4d9bc640 1 -- 172.21.15.83:0/3497117108 shutdown_connections 2024-03-06T07:25:59.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.221+0000 7ffb4d9bc640 1 -- 172.21.15.83:0/3497117108 wait complete. 2024-03-06T07:25:59.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.223+0000 7ffb4d9bc640 1 Processor -- start 2024-03-06T07:25:59.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.223+0000 7ffb4d9bc640 1 -- start start 2024-03-06T07:25:59.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.224+0000 7ffb4d9bc640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffb480cf830 0x7ffb480b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.224+0000 7ffb4d9bc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb480d27f0 0x7ffb4815be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.224+0000 7ffb4d9bc640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffb4815dfe0 0x7ffb480b80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.224+0000 7ffb4d9bc640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb480d1df0 con 0x7ffb480d27f0 2024-03-06T07:25:59.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.224+0000 7ffb4d9bc640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb480d1c70 con 0x7ffb4815dfe0 2024-03-06T07:25:59.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.224+0000 7ffb4d9bc640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4815e3c0 con 0x7ffb480cf830 2024-03-06T07:25:59.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.224+0000 7ffb467fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb480d27f0 0x7ffb4815be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.224+0000 7ffb467fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb480d27f0 0x7ffb4815be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55614/0 (socket says 172.21.15.83:55614) 2024-03-06T07:25:59.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.224+0000 7ffb467fc640 1 -- 172.21.15.83:0/1883959155 learned_addr learned my addr 172.21.15.83:0/1883959155 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:25:59.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.224+0000 7ffb46ffd640 1 --2- 172.21.15.83:0/1883959155 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffb480cf830 0x7ffb480b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.224+0000 7ffb477fe640 1 --2- 172.21.15.83:0/1883959155 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffb4815dfe0 0x7ffb480b80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.225+0000 7ffb467fc640 1 -- 172.21.15.83:0/1883959155 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffb480cf830 msgr2=0x7ffb480b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.225+0000 7ffb467fc640 1 --2- 172.21.15.83:0/1883959155 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffb480cf830 0x7ffb480b68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.225+0000 7ffb467fc640 1 -- 172.21.15.83:0/1883959155 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffb4815dfe0 msgr2=0x7ffb480b80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.225+0000 7ffb467fc640 1 --2- 172.21.15.83:0/1883959155 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffb4815dfe0 0x7ffb480b80f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.225+0000 7ffb467fc640 1 -- 172.21.15.83:0/1883959155 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb480a1b70 con 0x7ffb480d27f0 2024-03-06T07:25:59.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.225+0000 7ffb467fc640 1 --2- 172.21.15.83:0/1883959155 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb480d27f0 0x7ffb4815be30 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c02b950 tx=0x7ffb3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:59.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.225+0000 7ffb2ffff640 1 -- 172.21.15.83:0/1883959155 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3c040070 con 0x7ffb480d27f0 2024-03-06T07:25:59.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.225+0000 7ffb2ffff640 1 -- 172.21.15.83:0/1883959155 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb3c030d00 con 0x7ffb480d27f0 2024-03-06T07:25:59.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.225+0000 7ffb4d9bc640 1 -- 172.21.15.83:0/1883959155 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb480b6fd0 con 0x7ffb480d27f0 2024-03-06T07:25:59.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.225+0000 7ffb2ffff640 1 -- 172.21.15.83:0/1883959155 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3c035040 con 0x7ffb480d27f0 2024-03-06T07:25:59.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.225+0000 7ffb4d9bc640 1 -- 172.21.15.83:0/1883959155 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb480b7590 con 0x7ffb480d27f0 2024-03-06T07:25:59.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.226+0000 7ffb4d9bc640 1 -- 172.21.15.83:0/1883959155 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb4804f330 con 0x7ffb480d27f0 2024-03-06T07:25:59.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.226+0000 7ffb2ffff640 1 -- 172.21.15.83:0/1883959155 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffb3c04b060 con 0x7ffb480d27f0 2024-03-06T07:25:59.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.227+0000 7ffb2ffff640 1 --2- 172.21.15.83:0/1883959155 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffb3406b4d0 0x7ffb3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.227+0000 7ffb2ffff640 1 -- 172.21.15.83:0/1883959155 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffb3c0abdf0 con 0x7ffb480d27f0 2024-03-06T07:25:59.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.227+0000 7ffb46ffd640 1 --2- 172.21.15.83:0/1883959155 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffb3406b4d0 0x7ffb3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.227+0000 7ffb46ffd640 1 --2- 172.21.15.83:0/1883959155 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffb3406b4d0 0x7ffb3406d990 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7ffb38006fd0 tx=0x7ffb38008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:59.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.231+0000 7ffb2ffff640 1 -- 172.21.15.83:0/1883959155 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffb3c07a620 con 0x7ffb480d27f0 2024-03-06T07:25:59.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.481+0000 7ffb4d9bc640 1 -- 172.21.15.83:0/1883959155 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb480747a0 con 0x7ffb480d27f0 2024-03-06T07:25:59.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.481+0000 7ffb2ffff640 1 -- 172.21.15.83:0/1883959155 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+4982 (secure 0 0 0) 0x7ffb3c07e6c0 con 0x7ffb480d27f0 2024-03-06T07:25:59.483 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:25:59.483 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":39,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/2525949760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":2525949760},{"type":"v1","addr":"172.21.15.183:6834","nonce":2525949760}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35},{"gid":6262,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2104447869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2104447869},{"type":"v1","addr":"172.21.15.183:6835","nonce":2104447869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6954,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/779621621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":779621621},{"type":"v1","addr":"172.21.15.83:6836","nonce":779621621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:54.631963+0000","modified":"2024-03-06T07:25:56.637047+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6957":{"gid":6957,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/3056214366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":3056214366},{"type":"v1","addr":"172.21.15.83:6837","nonce":3056214366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6957,"qdb_cluster":[6957]},"id":6}]} 2024-03-06T07:25:59.483 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 39 2024-03-06T07:25:59.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.483+0000 7ffb4d9bc640 1 -- 172.21.15.83:0/1883959155 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffb3406b4d0 msgr2=0x7ffb3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.483+0000 7ffb4d9bc640 1 --2- 172.21.15.83:0/1883959155 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffb3406b4d0 0x7ffb3406d990 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7ffb38006fd0 tx=0x7ffb38008040 comp rx=0 tx=0).stop 2024-03-06T07:25:59.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.483+0000 7ffb4d9bc640 1 -- 172.21.15.83:0/1883959155 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb480d27f0 msgr2=0x7ffb4815be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.483+0000 7ffb4d9bc640 1 --2- 172.21.15.83:0/1883959155 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb480d27f0 0x7ffb4815be30 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c02b950 tx=0x7ffb3c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.483+0000 7ffb4d9bc640 1 -- 172.21.15.83:0/1883959155 shutdown_connections 2024-03-06T07:25:59.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.483+0000 7ffb4d9bc640 1 --2- 172.21.15.83:0/1883959155 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffb3406b4d0 0x7ffb3406d990 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.483+0000 7ffb4d9bc640 1 --2- 172.21.15.83:0/1883959155 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffb4815dfe0 0x7ffb480b80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.483+0000 7ffb4d9bc640 1 --2- 172.21.15.83:0/1883959155 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb480d27f0 0x7ffb4815be30 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.483+0000 7ffb4d9bc640 1 --2- 172.21.15.83:0/1883959155 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffb480cf830 0x7ffb480b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.483+0000 7ffb4d9bc640 1 -- 172.21.15.83:0/1883959155 >> 172.21.15.83:0/1883959155 conn(0x7ffb480d3cd0 msgr2=0x7ffb4806bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:59.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.483+0000 7ffb4d9bc640 1 -- 172.21.15.83:0/1883959155 shutdown_connections 2024-03-06T07:25:59.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:25:59.483+0000 7ffb4d9bc640 1 -- 172.21.15.83:0/1883959155 wait complete. 2024-03-06T07:25:59.902 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 39, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:25:54.631963+0000', 'modified': '2024-03-06T07:25:56.637047+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6957}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6957': {'gid': 6957, 'name': 'c', 'rank': 0, 'incarnation': 37, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.83:6837/3056214366', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 3056214366}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 3056214366}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6957, 'qdb_cluster': [6957]} 2024-03-06T07:25:59.902 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:25:59.902 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:26:00.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.002+0000 7f4e38d0c640 1 Processor -- start 2024-03-06T07:26:00.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.002+0000 7f4e38d0c640 1 -- start start 2024-03-06T07:26:00.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.003+0000 7f4e38d0c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4e34158880 0x7f4e3406c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.003+0000 7f4e38d0c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e3406d030 con 0x7f4e3406cb90 2024-03-06T07:26:00.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.003+0000 7f4e38d0c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e3406d310 con 0x7f4e3415b380 2024-03-06T07:26:00.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.003+0000 7f4e38d0c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3406d5f0 con 0x7f4e34158880 2024-03-06T07:26:00.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.003+0000 7f4e32d9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f4e3406cb90 0x7f4e340cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:33080/0 (socket says 172.21.15.83:33080) 2024-03-06T07:26:00.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.003+0000 7f4e32d9d640 1 -- 172.21.15.83:0/3665068335 learned_addr learned my addr 172.21.15.83:0/3665068335 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:00.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.003+0000 7f4e3259c640 1 --2- 172.21.15.83:0/3665068335 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4e34158880 0x7f4e3406c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.004+0000 7f4e3159a640 1 -- 172.21.15.83:0/3665068335 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 897239749 0 0) 0x7f4e3406d030 con 0x7f4e3406cb90 2024-03-06T07:26:00.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.004+0000 7f4e3159a640 1 -- 172.21.15.83:0/3665068335 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e1c003520 con 0x7f4e3406cb90 2024-03-06T07:26:00.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.004+0000 7f4e3159a640 1 -- 172.21.15.83:0/3665068335 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2188145257 0 0) 0x7f4e3406d310 con 0x7f4e3415b380 2024-03-06T07:26:00.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.004+0000 7f4e3159a640 1 -- 172.21.15.83:0/3665068335 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e3406d030 con 0x7f4e3415b380 2024-03-06T07:26:00.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.004+0000 7f4e3259c640 1 -- 172.21.15.83:0/3665068335 >> v1:172.21.15.183:6789/0 conn(0x7f4e3415b380 legacy=0x7f4e340708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.004+0000 7f4e3159a640 1 -- 172.21.15.83:0/3665068335 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2768039687 0 0) 0x7f4e1c003520 con 0x7f4e3406cb90 2024-03-06T07:26:00.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.004+0000 7f4e3259c640 1 -- 172.21.15.83:0/3665068335 >> v1:172.21.15.83:6789/0 conn(0x7f4e3406cb90 legacy=0x7f4e340cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.004+0000 7f4e3259c640 1 -- 172.21.15.83:0/3665068335 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e3406d8d0 con 0x7f4e34158880 2024-03-06T07:26:00.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.005+0000 7f4e3259c640 1 --2- 172.21.15.83:0/3665068335 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4e34158880 0x7f4e3406c650 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4e240097d0 tx=0x7f4e2402b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd25eb20b44afbd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.005+0000 7f4e3159a640 1 -- 172.21.15.83:0/3665068335 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e24037070 con 0x7f4e34158880 2024-03-06T07:26:00.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.005+0000 7f4e3159a640 1 -- 172.21.15.83:0/3665068335 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e24030510 con 0x7f4e34158880 2024-03-06T07:26:00.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.005+0000 7f4e3159a640 1 -- 172.21.15.83:0/3665068335 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c0019f0 con 0x7f4e34158880 2024-03-06T07:26:00.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.005+0000 7f4e38d0c640 1 -- 172.21.15.83:0/3665068335 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4e34158880 msgr2=0x7f4e3406c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.005+0000 7f4e38d0c640 1 --2- 172.21.15.83:0/3665068335 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4e34158880 0x7f4e3406c650 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4e240097d0 tx=0x7f4e2402b800 comp rx=0 tx=0).stop 2024-03-06T07:26:00.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.005+0000 7f4e38d0c640 1 -- 172.21.15.83:0/3665068335 shutdown_connections 2024-03-06T07:26:00.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.005+0000 7f4e38d0c640 1 --2- 172.21.15.83:0/3665068335 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4e34158880 0x7f4e3406c650 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.006+0000 7f4e38d0c640 1 -- 172.21.15.83:0/3665068335 >> 172.21.15.83:0/3665068335 conn(0x7f4e340b1c80 msgr2=0x7f4e340cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:00.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.006+0000 7f4e38d0c640 1 -- 172.21.15.83:0/3665068335 shutdown_connections 2024-03-06T07:26:00.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.006+0000 7f4e38d0c640 1 -- 172.21.15.83:0/3665068335 wait complete. 2024-03-06T07:26:00.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.007+0000 7f4e38d0c640 1 Processor -- start 2024-03-06T07:26:00.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.007+0000 7f4e38d0c640 1 -- start start 2024-03-06T07:26:00.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.007+0000 7f4e38d0c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4e3406cb90 0x7f4e340cc640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.007+0000 7f4e38d0c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4e34158880 0x7f4e34141000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.007+0000 7f4e38d0c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e3415b380 0x7f4e34141540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.007+0000 7f4e38d0c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e34071160 con 0x7f4e3415b380 2024-03-06T07:26:00.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.008+0000 7f4e38d0c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e34070fe0 con 0x7f4e34158880 2024-03-06T07:26:00.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.008+0000 7f4e38d0c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e340cded0 con 0x7f4e3406cb90 2024-03-06T07:26:00.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.008+0000 7f4e32d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e3415b380 0x7f4e34141540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.008+0000 7f4e32d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e3415b380 0x7f4e34141540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55616/0 (socket says 172.21.15.83:55616) 2024-03-06T07:26:00.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.008+0000 7f4e32d9d640 1 -- 172.21.15.83:0/1863313603 learned_addr learned my addr 172.21.15.83:0/1863313603 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:00.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.008+0000 7f4e3259c640 1 --2- 172.21.15.83:0/1863313603 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4e3406cb90 0x7f4e340cc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.008+0000 7f4e31d9b640 1 --2- 172.21.15.83:0/1863313603 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4e34158880 0x7f4e34141000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.008+0000 7f4e32d9d640 1 -- 172.21.15.83:0/1863313603 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4e3406cb90 msgr2=0x7f4e340cc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.008+0000 7f4e32d9d640 1 --2- 172.21.15.83:0/1863313603 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4e3406cb90 0x7f4e340cc640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.008+0000 7f4e32d9d640 1 -- 172.21.15.83:0/1863313603 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4e34158880 msgr2=0x7f4e34141000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.008+0000 7f4e32d9d640 1 --2- 172.21.15.83:0/1863313603 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4e34158880 0x7f4e34141000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.008+0000 7f4e32d9d640 1 -- 172.21.15.83:0/1863313603 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e341f2d60 con 0x7f4e3415b380 2024-03-06T07:26:00.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.008+0000 7f4e31d9b640 1 --2- 172.21.15.83:0/1863313603 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4e34158880 0x7f4e34141000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:00.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.008+0000 7f4e32d9d640 1 --2- 172.21.15.83:0/1863313603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e3415b380 0x7f4e34141540 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c001460 tx=0x7f4e2c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.008+0000 7f4e22ffd640 1 -- 172.21.15.83:0/1863313603 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c012400 con 0x7f4e3415b380 2024-03-06T07:26:00.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.008+0000 7f4e22ffd640 1 -- 172.21.15.83:0/1863313603 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e2c0125a0 con 0x7f4e3415b380 2024-03-06T07:26:00.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.008+0000 7f4e38d0c640 1 -- 172.21.15.83:0/1863313603 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e341f57e0 con 0x7f4e3415b380 2024-03-06T07:26:00.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.008+0000 7f4e22ffd640 1 -- 172.21.15.83:0/1863313603 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c012990 con 0x7f4e3415b380 2024-03-06T07:26:00.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.008+0000 7f4e38d0c640 1 -- 172.21.15.83:0/1863313603 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e341f62c0 con 0x7f4e3415b380 2024-03-06T07:26:00.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.010+0000 7f4e22ffd640 1 -- 172.21.15.83:0/1863313603 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4e2c012b30 con 0x7f4e3415b380 2024-03-06T07:26:00.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.011+0000 7f4e22ffd640 1 --2- 172.21.15.83:0/1863313603 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4e1406b4d0 0x7f4e1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.011+0000 7f4e3259c640 1 --2- 172.21.15.83:0/1863313603 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4e1406b4d0 0x7f4e1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.011+0000 7f4e22ffd640 1 -- 172.21.15.83:0/1863313603 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4e2c08a9f0 con 0x7f4e3415b380 2024-03-06T07:26:00.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.011+0000 7f4e3b9d2740 1 Processor -- start 2024-03-06T07:26:00.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.011+0000 7f4e3259c640 1 --2- 172.21.15.83:0/1863313603 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4e1406b4d0 0x7f4e1406d990 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f4e24034000 tx=0x7f4e24008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.011+0000 7f4e3b9d2740 1 -- start start 2024-03-06T07:26:00.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.012+0000 7f4e3b9d2740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55f14e1d4cc0 0x55f14e1d50a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.012+0000 7f4e3b9d2740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f14e1f5090 0x55f14e1f55e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.012+0000 7f4e3b9d2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f14e1f47b0 0x55f14e1a7680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.012+0000 7f4e3b9d2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f14e1116b0 con 0x55f14e1f47b0 2024-03-06T07:26:00.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.012+0000 7f4e3b9d2740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f14e0baa60 con 0x55f14e1f5090 2024-03-06T07:26:00.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.012+0000 7f4e3b9d2740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f14e10d750 con 0x55f14e1d4cc0 2024-03-06T07:26:00.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.012+0000 7f4e32d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f14e1f47b0 0x55f14e1a7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.012+0000 7f4e32d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f14e1f47b0 0x55f14e1a7680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55624/0 (socket says 172.21.15.83:55624) 2024-03-06T07:26:00.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.012+0000 7f4e32d9d640 1 -- 172.21.15.83:0/3567853626 learned_addr learned my addr 172.21.15.83:0/3567853626 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:00.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.012+0000 7f4e32d9d640 1 -- 172.21.15.83:0/3567853626 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55f14e1d4cc0 msgr2=0x55f14e1d50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.012+0000 7f4e3259c640 1 --2- 172.21.15.83:0/3567853626 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55f14e1d4cc0 0x55f14e1d50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.013+0000 7f4e31d9b640 1 --2- 172.21.15.83:0/3567853626 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f14e1f5090 0x55f14e1f55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.013+0000 7f4e32d9d640 1 --2- 172.21.15.83:0/3567853626 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55f14e1d4cc0 0x55f14e1d50a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.013+0000 7f4e32d9d640 1 -- 172.21.15.83:0/3567853626 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f14e1f5090 msgr2=0x55f14e1f55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.013+0000 7f4e32d9d640 1 --2- 172.21.15.83:0/3567853626 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f14e1f5090 0x55f14e1f55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.013+0000 7f4e32d9d640 1 -- 172.21.15.83:0/3567853626 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f14e0d8f30 con 0x55f14e1f47b0 2024-03-06T07:26:00.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.013+0000 7f4e3259c640 1 --2- 172.21.15.83:0/3567853626 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55f14e1d4cc0 0x55f14e1d50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:00.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.013+0000 7f4e31d9b640 1 --2- 172.21.15.83:0/3567853626 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f14e1f5090 0x55f14e1f55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:00.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.013+0000 7f4e32d9d640 1 --2- 172.21.15.83:0/3567853626 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f14e1f47b0 0x55f14e1a7680 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c0590f0 tx=0x7f4e2c058b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b389409321dbff3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.013+0000 7f4e13fff640 1 -- 172.21.15.83:0/3567853626 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c087070 con 0x55f14e1f47b0 2024-03-06T07:26:00.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.013+0000 7f4e13fff640 1 -- 172.21.15.83:0/3567853626 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e2c059160 con 0x55f14e1f47b0 2024-03-06T07:26:00.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.013+0000 7f4e13fff640 1 -- 172.21.15.83:0/3567853626 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c05d2b0 con 0x55f14e1f47b0 2024-03-06T07:26:00.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.013+0000 7f4e3b9d2740 1 -- 172.21.15.83:0/3567853626 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f14e1f47b0 msgr2=0x55f14e1a7680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.013+0000 7f4e3b9d2740 1 --2- 172.21.15.83:0/3567853626 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f14e1f47b0 0x55f14e1a7680 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c0590f0 tx=0x7f4e2c058b80 comp rx=0 tx=0).stop 2024-03-06T07:26:00.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.014+0000 7f4e3b9d2740 1 -- 172.21.15.83:0/3567853626 shutdown_connections 2024-03-06T07:26:00.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.014+0000 7f4e3b9d2740 1 --2- 172.21.15.83:0/3567853626 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f14e1f47b0 0x55f14e1a7680 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.014+0000 7f4e3b9d2740 1 --2- 172.21.15.83:0/3567853626 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f14e1f5090 0x55f14e1f55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.014+0000 7f4e3b9d2740 1 --2- 172.21.15.83:0/3567853626 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55f14e1d4cc0 0x55f14e1d50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.014+0000 7f4e3b9d2740 1 -- 172.21.15.83:0/3567853626 >> 172.21.15.83:0/3567853626 conn(0x55f14e1a98f0 msgr2=0x55f14e1ab1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:00.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.014+0000 7f4e3b9d2740 1 -- 172.21.15.83:0/3567853626 shutdown_connections 2024-03-06T07:26:00.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.014+0000 7f4e3b9d2740 1 -- 172.21.15.83:0/3567853626 wait complete. 2024-03-06T07:26:00.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.014+0000 7f4e3b9d2740 1 -- start start 2024-03-06T07:26:00.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.014+0000 7f4e3b9d2740 1 Processor -- start 2024-03-06T07:26:00.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.015+0000 7f4e3b9d2740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f14e1f47b0 0x55f14e1f3030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.015+0000 7f4e3b9d2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f14e1f5090 0x55f14e1d33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.015+0000 7f4e3b9d2740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55f14e1d38f0 0x55f14e1d51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.015+0000 7f4e3b9d2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f14e1d8010 con 0x55f14e1f5090 2024-03-06T07:26:00.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.015+0000 7f4e3b9d2740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f14e0cdea0 con 0x55f14e1f47b0 2024-03-06T07:26:00.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.015+0000 7f4e3b9d2740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f14e1f3a30 con 0x55f14e1d38f0 2024-03-06T07:26:00.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.015+0000 7f4e3b9d2740 10 client.0 _ll_register_callbacks cb 0x55f14e0cd870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:26:00.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.015+0000 7f4e31d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f14e1f5090 0x55f14e1d33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.015+0000 7f4e31d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f14e1f5090 0x55f14e1d33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55632/0 (socket says 172.21.15.83:55632) 2024-03-06T07:26:00.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.015+0000 7f4e31d9b640 1 -- 172.21.15.83:0/2377773212 learned_addr learned my addr 172.21.15.83:0/2377773212 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:00.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.015+0000 7f4e3259c640 1 --2- 172.21.15.83:0/2377773212 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f14e1f47b0 0x55f14e1f3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.015+0000 7f4e32d9d640 1 --2- 172.21.15.83:0/2377773212 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55f14e1d38f0 0x55f14e1d51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.016+0000 7f4e31d9b640 1 -- 172.21.15.83:0/2377773212 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55f14e1d38f0 msgr2=0x55f14e1d51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.016+0000 7f4e31d9b640 1 --2- 172.21.15.83:0/2377773212 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55f14e1d38f0 0x55f14e1d51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.016+0000 7f4e31d9b640 1 -- 172.21.15.83:0/2377773212 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f14e1f47b0 msgr2=0x55f14e1f3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.016+0000 7f4e31d9b640 1 --2- 172.21.15.83:0/2377773212 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f14e1f47b0 0x55f14e1f3030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.016+0000 7f4e31d9b640 1 -- 172.21.15.83:0/2377773212 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f14e0ce270 con 0x55f14e1f5090 2024-03-06T07:26:00.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.016+0000 7f4e3259c640 1 --2- 172.21.15.83:0/2377773212 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f14e1f47b0 0x55f14e1f3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:00.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.016+0000 7f4e31d9b640 1 --2- 172.21.15.83:0/2377773212 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f14e1f5090 0x55f14e1d33b0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f4e280011e0 tx=0x7f4e2800f660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf842ad61e234935 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.016+0000 7f4e117fa640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-06T07:26:00.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.016+0000 7f4e117fa640 1 -- 172.21.15.83:0/2377773212 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2800fe90 con 0x55f14e1f5090 2024-03-06T07:26:00.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.016+0000 7f4e117fa640 1 -- 172.21.15.83:0/2377773212 <== mon.0 v2:172.21.15.83:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4e280052e0 con 0x55f14e1f5090 2024-03-06T07:26:00.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.016+0000 7f4e3b9d2740 1 -- 172.21.15.83:0/2377773212 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4e2c059160 con 0x55f14e1f5090 2024-03-06T07:26:00.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.016+0000 7f4e117fa640 1 -- 172.21.15.83:0/2377773212 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4e28020050 con 0x55f14e1f5090 2024-03-06T07:26:00.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.016+0000 7f4e3b9d2740 10 client.7104 fetch_fsmap learned FSMap version 39 2024-03-06T07:26:00.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.016+0000 7f4e3b9d2740 1 -- 172.21.15.83:0/2377773212 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=2+}) v3 -- 0x55f14de262d0 con 0x55f14e1f5090 2024-03-06T07:26:00.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.016+0000 7f4e117fa640 1 -- 172.21.15.83:0/2377773212 <== mon.0 v2:172.21.15.83:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e2c059160 con 0x55f14e1f5090 2024-03-06T07:26:00.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.016+0000 7f4e117fa640 1 -- 172.21.15.83:0/2377773212 <== mon.0 v2:172.21.15.83:3300/0 5 ==== fsmap(e 39) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f4e28023020 con 0x55f14e1f5090 2024-03-06T07:26:00.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.016+0000 7f4e3b9d2740 10 client.7104 fetch_fsmap finished waiting for FSMap version 39 2024-03-06T07:26:00.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.016+0000 7f4e3b9d2740 10 client.7104 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:26:00.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.016+0000 7f4e3b9d2740 20 client.7104 populate_metadata read hostname 'smithi083' 2024-03-06T07:26:00.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.016+0000 7f4e3b9d2740 1 --2- 172.21.15.83:0/2377773212 >> [v2:172.21.15.83:6835/3056214366,v1:172.21.15.83:6837/3056214366] conn(0x55f14e1b6340 0x55f14e1acab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.016+0000 7f4e3b9d2740 4 client.7104 mds_command: new command op to 6957 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:26:00.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.016+0000 7f4e3b9d2740 1 -- 172.21.15.83:0/2377773212 --> [v2:172.21.15.83:6835/3056214366,v1:172.21.15.83:6837/3056214366] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f14e11ebf0 con 0x55f14e1b6340 2024-03-06T07:26:00.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.017+0000 7f4e3259c640 1 --2- 172.21.15.83:0/2377773212 >> [v2:172.21.15.83:6835/3056214366,v1:172.21.15.83:6837/3056214366] conn(0x55f14e1b6340 0x55f14e1acab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.017+0000 7f4e117fa640 10 client.7104 ms_handle_connect on v2:172.21.15.83:6835/3056214366 2024-03-06T07:26:00.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.017+0000 7f4e3259c640 1 --2- 172.21.15.83:0/2377773212 >> [v2:172.21.15.83:6835/3056214366,v1:172.21.15.83:6837/3056214366] conn(0x55f14e1b6340 0x55f14e1acab0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9db3b08889498fe9 server_cookie=5fd075595e502140 in_seq=0 out_seq=0 2024-03-06T07:26:00.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.017+0000 7f4e117fa640 1 -- 172.21.15.83:0/2377773212 <== mds.0 v2:172.21.15.83:6835/3056214366 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f14e11ebf0 con 0x55f14e1b6340 2024-03-06T07:26:00.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.017+0000 7f4e117fa640 10 client.7104 handle_command_reply: tid=0 2024-03-06T07:26:00.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.017+0000 7f4e3b9d2740 1 client.7104 shutdown 2024-03-06T07:26:00.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.018+0000 7f4e3b9d2740 1 -- 172.21.15.83:0/2377773212 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f14e1f5090 msgr2=0x55f14e1d33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.018+0000 7f4e3b9d2740 1 --2- 172.21.15.83:0/2377773212 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f14e1f5090 0x55f14e1d33b0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f4e280011e0 tx=0x7f4e2800f660 comp rx=0 tx=0).stop 2024-03-06T07:26:00.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.018+0000 7f4e3b9d2740 1 -- 172.21.15.83:0/2377773212 shutdown_connections 2024-03-06T07:26:00.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.018+0000 7f4e3b9d2740 1 --2- 172.21.15.83:0/2377773212 >> [v2:172.21.15.83:6835/3056214366,v1:172.21.15.83:6837/3056214366] conn(0x55f14e1b6340 0x55f14e1acab0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.018+0000 7f4e3b9d2740 1 --2- 172.21.15.83:0/2377773212 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55f14e1d38f0 0x55f14e1d51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.018+0000 7f4e3b9d2740 1 --2- 172.21.15.83:0/2377773212 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f14e1f5090 0x55f14e1d33b0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.018+0000 7f4e117fa640 0 client.7104 ms_handle_reset on v2:172.21.15.83:6835/3056214366 2024-03-06T07:26:00.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.018+0000 7f4e3b9d2740 1 --2- 172.21.15.83:0/2377773212 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f14e1f47b0 0x55f14e1f3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.018+0000 7f4e3b9d2740 1 -- 172.21.15.83:0/2377773212 >> 172.21.15.83:0/2377773212 conn(0x55f14e1d4cc0 msgr2=0x55f14e1a6510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:00.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.018+0000 7f4e3b9d2740 1 -- 172.21.15.83:0/2377773212 shutdown_connections 2024-03-06T07:26:00.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.018+0000 7f4e3b9d2740 1 -- 172.21.15.83:0/2377773212 wait complete. 2024-03-06T07:26:00.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.018+0000 7f4e3b9d2740 20 client.7104 trim_cache size 0 max 16384 2024-03-06T07:26:00.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.034+0000 7f4e3b9d2740 1 Processor -- start 2024-03-06T07:26:00.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.035+0000 7f4e3b9d2740 1 -- start start 2024-03-06T07:26:00.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.035+0000 7f4e3b9d2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f14e1d38f0 0x55f14e1a78a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.035+0000 7f4e3b9d2740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f14e1f47b0 0x55f14e1fe0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.035+0000 7f4e3b9d2740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55f14e1f5090 0x55f14e1c9250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.035+0000 7f4e3b9d2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e341e3ea0 con 0x55f14e1d38f0 2024-03-06T07:26:00.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.035+0000 7f4e3b9d2740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3415a6f0 con 0x55f14e1f47b0 2024-03-06T07:26:00.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.035+0000 7f4e3b9d2740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e341e38a0 con 0x55f14e1f5090 2024-03-06T07:26:00.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.035+0000 7f4e3259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f14e1d38f0 0x55f14e1a78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.035+0000 7f4e3259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f14e1d38f0 0x55f14e1a78a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55638/0 (socket says 172.21.15.83:55638) 2024-03-06T07:26:00.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.035+0000 7f4e3259c640 1 -- 172.21.15.83:0/3111206457 learned_addr learned my addr 172.21.15.83:0/3111206457 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:00.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.036+0000 7f4e3259c640 1 -- 172.21.15.83:0/3111206457 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55f14e1f5090 msgr2=0x55f14e1c9250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.036+0000 7f4e3259c640 1 --2- 172.21.15.83:0/3111206457 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55f14e1f5090 0x55f14e1c9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.036+0000 7f4e3259c640 1 -- 172.21.15.83:0/3111206457 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f14e1f47b0 msgr2=0x55f14e1fe0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.036+0000 7f4e3259c640 1 --2- 172.21.15.83:0/3111206457 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f14e1f47b0 0x55f14e1fe0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.036+0000 7f4e3259c640 1 -- 172.21.15.83:0/3111206457 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f14def06b0 con 0x55f14e1d38f0 2024-03-06T07:26:00.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.036+0000 7f4e3259c640 1 --2- 172.21.15.83:0/3111206457 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f14e1d38f0 0x55f14e1a78a0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f4e2403f040 tx=0x7f4e2402f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=153b864f18ef9744 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.036+0000 7f4e137fe640 1 -- 172.21.15.83:0/3111206457 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e24037040 con 0x55f14e1d38f0 2024-03-06T07:26:00.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.036+0000 7f4e137fe640 1 -- 172.21.15.83:0/3111206457 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e24055070 con 0x55f14e1d38f0 2024-03-06T07:26:00.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.036+0000 7f4e137fe640 1 -- 172.21.15.83:0/3111206457 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2404f2d0 con 0x55f14e1d38f0 2024-03-06T07:26:00.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.036+0000 7f4e3b9d2740 1 -- 172.21.15.83:0/3111206457 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f14e1d38f0 msgr2=0x55f14e1a78a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.036+0000 7f4e3b9d2740 1 --2- 172.21.15.83:0/3111206457 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f14e1d38f0 0x55f14e1a78a0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f4e2403f040 tx=0x7f4e2402f000 comp rx=0 tx=0).stop 2024-03-06T07:26:00.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.037+0000 7f4e3b9d2740 1 -- 172.21.15.83:0/3111206457 shutdown_connections 2024-03-06T07:26:00.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.037+0000 7f4e3b9d2740 1 --2- 172.21.15.83:0/3111206457 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55f14e1f5090 0x55f14e1c9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.037+0000 7f4e3b9d2740 1 --2- 172.21.15.83:0/3111206457 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f14e1f47b0 0x55f14e1fe0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.037+0000 7f4e3b9d2740 1 --2- 172.21.15.83:0/3111206457 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f14e1d38f0 0x55f14e1a78a0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.037+0000 7f4e3b9d2740 1 -- 172.21.15.83:0/3111206457 >> 172.21.15.83:0/3111206457 conn(0x55f14e1d4cc0 msgr2=0x55f14e1ce200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:00.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.037+0000 7f4e3b9d2740 1 -- 172.21.15.83:0/3111206457 shutdown_connections 2024-03-06T07:26:00.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.037+0000 7f4e3b9d2740 1 -- 172.21.15.83:0/3111206457 wait complete. 2024-03-06T07:26:00.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.037+0000 7f4e3b9d2740 1 -- start start 2024-03-06T07:26:00.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.037+0000 7f4e3b9d2740 1 Processor -- start 2024-03-06T07:26:00.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.039+0000 7f4e3b9d2740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55f14e1d38f0 0x55f14e1f5a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.039+0000 7f4e3b9d2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f14e1f47b0 0x55f14e1a77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.039+0000 7f4e3b9d2740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f14e1f5090 0x55f14e1aca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.039+0000 7f4e3b9d2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f14e1b61c0 con 0x55f14e1f47b0 2024-03-06T07:26:00.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.039+0000 7f4e3b9d2740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f14e1cac30 con 0x55f14e1f5090 2024-03-06T07:26:00.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.039+0000 7f4e3b9d2740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f14e1b6760 con 0x55f14e1d38f0 2024-03-06T07:26:00.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.039+0000 7f4e3b9d2740 10 client.0 _ll_register_callbacks cb 0x7f4e28001ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:26:00.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.039+0000 7f4e31d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f14e1f47b0 0x55f14e1a77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.039+0000 7f4e31d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f14e1f47b0 0x55f14e1a77e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55640/0 (socket says 172.21.15.83:55640) 2024-03-06T07:26:00.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.039+0000 7f4e31d9b640 1 -- 172.21.15.83:0/2865455086 learned_addr learned my addr 172.21.15.83:0/2865455086 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:00.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.039+0000 7f4e3259c640 1 --2- 172.21.15.83:0/2865455086 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55f14e1d38f0 0x55f14e1f5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.039+0000 7f4e32d9d640 1 --2- 172.21.15.83:0/2865455086 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f14e1f5090 0x55f14e1aca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.039+0000 7f4e31d9b640 1 -- 172.21.15.83:0/2865455086 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55f14e1d38f0 msgr2=0x55f14e1f5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.039+0000 7f4e31d9b640 1 --2- 172.21.15.83:0/2865455086 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55f14e1d38f0 0x55f14e1f5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.039+0000 7f4e31d9b640 1 -- 172.21.15.83:0/2865455086 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f14e1f5090 msgr2=0x55f14e1aca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.039+0000 7f4e31d9b640 1 --2- 172.21.15.83:0/2865455086 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f14e1f5090 0x55f14e1aca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.039+0000 7f4e31d9b640 1 -- 172.21.15.83:0/2865455086 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f14e1fb8a0 con 0x55f14e1f47b0 2024-03-06T07:26:00.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.039+0000 7f4e31d9b640 1 --2- 172.21.15.83:0/2865455086 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f14e1f47b0 0x55f14e1a77e0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f4e28003780 tx=0x7f4e28000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce51ff55195fb1b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.039+0000 7f4e117fa640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-06T07:26:00.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.039+0000 7f4e117fa640 1 -- 172.21.15.83:0/2865455086 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e28003c10 con 0x55f14e1f47b0 2024-03-06T07:26:00.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.039+0000 7f4e117fa640 1 -- 172.21.15.83:0/2865455086 <== mon.0 v2:172.21.15.83:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4e28003db0 con 0x55f14e1f47b0 2024-03-06T07:26:00.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.039+0000 7f4e3b9d2740 1 -- 172.21.15.83:0/2865455086 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4e24055070 con 0x55f14e1f47b0 2024-03-06T07:26:00.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.040+0000 7f4e117fa640 1 -- 172.21.15.83:0/2865455086 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4e280133f0 con 0x55f14e1f47b0 2024-03-06T07:26:00.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.040+0000 7f4e3b9d2740 10 client.7110 fetch_fsmap learned FSMap version 39 2024-03-06T07:26:00.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.040+0000 7f4e3b9d2740 1 -- 172.21.15.83:0/2865455086 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=2+}) v3 -- 0x55f14e1f6130 con 0x55f14e1f47b0 2024-03-06T07:26:00.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.040+0000 7f4e117fa640 1 -- 172.21.15.83:0/2865455086 <== mon.0 v2:172.21.15.83:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e24055070 con 0x55f14e1f47b0 2024-03-06T07:26:00.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.040+0000 7f4e117fa640 1 -- 172.21.15.83:0/2865455086 <== mon.0 v2:172.21.15.83:3300/0 5 ==== fsmap(e 39) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f4e28034020 con 0x55f14e1f47b0 2024-03-06T07:26:00.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.040+0000 7f4e3b9d2740 10 client.7110 fetch_fsmap finished waiting for FSMap version 39 2024-03-06T07:26:00.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.040+0000 7f4e3b9d2740 10 client.7110 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:26:00.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.040+0000 7f4e3b9d2740 20 client.7110 populate_metadata read hostname 'smithi083' 2024-03-06T07:26:00.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.040+0000 7f4e3b9d2740 1 --2- 172.21.15.83:0/2865455086 >> [v2:172.21.15.83:6835/3056214366,v1:172.21.15.83:6837/3056214366] conn(0x55f14e1ad0b0 0x55f14e1da550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.040+0000 7f4e3b9d2740 4 client.7110 mds_command: new command op to 6957 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:26:00.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.040+0000 7f4e3b9d2740 1 -- 172.21.15.83:0/2865455086 --> [v2:172.21.15.83:6835/3056214366,v1:172.21.15.83:6837/3056214366] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f14e1aa050 con 0x55f14e1ad0b0 2024-03-06T07:26:00.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.040+0000 7f4e3259c640 1 --2- 172.21.15.83:0/2865455086 >> [v2:172.21.15.83:6835/3056214366,v1:172.21.15.83:6837/3056214366] conn(0x55f14e1ad0b0 0x55f14e1da550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.040+0000 7f4e117fa640 10 client.7110 ms_handle_connect on v2:172.21.15.83:6835/3056214366 2024-03-06T07:26:00.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.040+0000 7f4e3259c640 1 --2- 172.21.15.83:0/2865455086 >> [v2:172.21.15.83:6835/3056214366,v1:172.21.15.83:6837/3056214366] conn(0x55f14e1ad0b0 0x55f14e1da550 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33c46771f7b5f55a server_cookie=435b49fcaf40e42d in_seq=0 out_seq=0 2024-03-06T07:26:00.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.040+0000 7f4e117fa640 1 -- 172.21.15.83:0/2865455086 <== mds.0 v2:172.21.15.83:6835/3056214366 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55f14e1aa050 con 0x55f14e1ad0b0 2024-03-06T07:26:00.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.040+0000 7f4e117fa640 10 client.7110 handle_command_reply: tid=0 2024-03-06T07:26:00.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.040+0000 7f4e3b9d2740 1 client.7110 shutdown 2024-03-06T07:26:00.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.041+0000 7f4e3b9d2740 1 -- 172.21.15.83:0/2865455086 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f14e1f47b0 msgr2=0x55f14e1a77e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.041+0000 7f4e3b9d2740 1 --2- 172.21.15.83:0/2865455086 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f14e1f47b0 0x55f14e1a77e0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f4e28003780 tx=0x7f4e28000ed0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.041+0000 7f4e3b9d2740 1 -- 172.21.15.83:0/2865455086 shutdown_connections 2024-03-06T07:26:00.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.041+0000 7f4e3b9d2740 1 --2- 172.21.15.83:0/2865455086 >> [v2:172.21.15.83:6835/3056214366,v1:172.21.15.83:6837/3056214366] conn(0x55f14e1ad0b0 0x55f14e1da550 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.041+0000 7f4e3b9d2740 1 --2- 172.21.15.83:0/2865455086 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f14e1f5090 0x55f14e1aca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.041+0000 7f4e3b9d2740 1 --2- 172.21.15.83:0/2865455086 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f14e1f47b0 0x55f14e1a77e0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.041+0000 7f4e117fa640 0 client.7110 ms_handle_reset on v2:172.21.15.83:6835/3056214366 2024-03-06T07:26:00.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.041+0000 7f4e3b9d2740 1 --2- 172.21.15.83:0/2865455086 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55f14e1d38f0 0x55f14e1f5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.041+0000 7f4e3b9d2740 1 -- 172.21.15.83:0/2865455086 >> 172.21.15.83:0/2865455086 conn(0x55f14e1d4cc0 msgr2=0x55f14e1f3a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:00.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.041+0000 7f4e3b9d2740 1 -- 172.21.15.83:0/2865455086 shutdown_connections 2024-03-06T07:26:00.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.041+0000 7f4e3b9d2740 1 -- 172.21.15.83:0/2865455086 wait complete. 2024-03-06T07:26:00.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.041+0000 7f4e3b9d2740 20 client.7110 trim_cache size 0 max 16384 2024-03-06T07:26:00.048 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:26:00.048 INFO:teuthology.orchestra.run.smithi083.stdout: "cluster_fsid": "039877dd-d2ce-45d2-8e77-ac5eb7e1d55a", 2024-03-06T07:26:00.048 INFO:teuthology.orchestra.run.smithi083.stdout: "whoami": 0, 2024-03-06T07:26:00.048 INFO:teuthology.orchestra.run.smithi083.stdout: "id": 6957, 2024-03-06T07:26:00.048 INFO:teuthology.orchestra.run.smithi083.stdout: "want_state": "up:active", 2024-03-06T07:26:00.049 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "up:active", 2024-03-06T07:26:00.049 INFO:teuthology.orchestra.run.smithi083.stdout: "fs_name": "cephfs", 2024-03-06T07:26:00.049 INFO:teuthology.orchestra.run.smithi083.stdout: "rank_uptime": 4.7613076530000003, 2024-03-06T07:26:00.049 INFO:teuthology.orchestra.run.smithi083.stdout: "mdsmap_epoch": 39, 2024-03-06T07:26:00.049 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_epoch": 82, 2024-03-06T07:26:00.049 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_epoch_barrier": 82, 2024-03-06T07:26:00.049 INFO:teuthology.orchestra.run.smithi083.stdout: "uptime": 9.1020162150000008 2024-03-06T07:26:00.049 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:26:00.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.041+0000 7f4e38d0c640 1 -- 172.21.15.83:0/1863313603 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4e1406b4d0 msgr2=0x7f4e1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.041+0000 7f4e38d0c640 1 --2- 172.21.15.83:0/1863313603 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4e1406b4d0 0x7f4e1406d990 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f4e24034000 tx=0x7f4e24008030 comp rx=0 tx=0).stop 2024-03-06T07:26:00.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.041+0000 7f4e38d0c640 1 -- 172.21.15.83:0/1863313603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e3415b380 msgr2=0x7f4e34141540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.041+0000 7f4e38d0c640 1 --2- 172.21.15.83:0/1863313603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e3415b380 0x7f4e34141540 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c001460 tx=0x7f4e2c00a430 comp rx=0 tx=0).stop 2024-03-06T07:26:00.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.041+0000 7f4e38d0c640 1 -- 172.21.15.83:0/1863313603 shutdown_connections 2024-03-06T07:26:00.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.041+0000 7f4e38d0c640 1 --2- 172.21.15.83:0/1863313603 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4e1406b4d0 0x7f4e1406d990 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.041+0000 7f4e38d0c640 1 --2- 172.21.15.83:0/1863313603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e3415b380 0x7f4e34141540 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.041+0000 7f4e38d0c640 1 --2- 172.21.15.83:0/1863313603 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4e34158880 0x7f4e34141000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.041+0000 7f4e38d0c640 1 --2- 172.21.15.83:0/1863313603 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4e3406cb90 0x7f4e340cc640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.041+0000 7f4e38d0c640 1 -- 172.21.15.83:0/1863313603 >> 172.21.15.83:0/1863313603 conn(0x7f4e340b1c80 msgr2=0x7f4e340b3a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:00.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.041+0000 7f4e38d0c640 1 -- 172.21.15.83:0/1863313603 shutdown_connections 2024-03-06T07:26:00.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.041+0000 7f4e38d0c640 1 -- 172.21.15.83:0/1863313603 wait complete. 2024-03-06T07:26:00.054 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:26:00.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.153+0000 7faf2610c640 1 Processor -- start 2024-03-06T07:26:00.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.153+0000 7faf2610c640 1 -- start start 2024-03-06T07:26:00.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.154+0000 7faf2610c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7faf2015bdd0 0x7faf200c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.154+0000 7faf2610c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf200d16e0 0x7faf200d1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.154+0000 7faf2610c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf200cebe0 con 0x7faf2015fa10 2024-03-06T07:26:00.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.154+0000 7faf2610c640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf200ce980 con 0x7faf2015bdd0 2024-03-06T07:26:00.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.154+0000 7faf2610c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf200ceec0 con 0x7faf200d16e0 2024-03-06T07:26:00.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.154+0000 7faf1f7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7faf2015bdd0 0x7faf200c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.154+0000 7faf1f7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7faf2015bdd0 0x7faf200c5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55642/0 (socket says 172.21.15.83:55642) 2024-03-06T07:26:00.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.154+0000 7faf1f7fe640 1 -- 172.21.15.83:0/1469517609 learned_addr learned my addr 172.21.15.83:0/1469517609 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:00.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.154+0000 7faf1ffff640 1 --2- 172.21.15.83:0/1469517609 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf200d16e0 0x7faf200d1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.154+0000 7faf1f7fe640 1 -- 172.21.15.83:0/1469517609 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf200d16e0 msgr2=0x7faf200d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.154+0000 7faf1f7fe640 1 --2- 172.21.15.83:0/1469517609 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf200d16e0 0x7faf200d1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.154+0000 7faf1f7fe640 1 -- 172.21.15.83:0/1469517609 >> v1:172.21.15.183:6789/0 conn(0x7faf2015fa10 legacy=0x7faf200ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.154+0000 7faf1f7fe640 1 -- 172.21.15.83:0/1469517609 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf200cf1a0 con 0x7faf2015bdd0 2024-03-06T07:26:00.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.154+0000 7faf1ffff640 1 --2- 172.21.15.83:0/1469517609 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf200d16e0 0x7faf200d1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:00.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.155+0000 7faf1f7fe640 1 --2- 172.21.15.83:0/1469517609 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf2015bdd0 0x7faf200c5610 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7faf10009870 tx=0x7faf1002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48e6cf022ea72df4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.155+0000 7faf1e7fc640 1 -- 172.21.15.83:0/1469517609 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf100317c0 con 0x7faf2015bdd0 2024-03-06T07:26:00.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.155+0000 7faf1e7fc640 1 -- 172.21.15.83:0/1469517609 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf10037070 con 0x7faf2015bdd0 2024-03-06T07:26:00.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.155+0000 7faf1e7fc640 1 -- 172.21.15.83:0/1469517609 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf10031b80 con 0x7faf2015bdd0 2024-03-06T07:26:00.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.156+0000 7faf2610c640 1 -- 172.21.15.83:0/1469517609 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf2015bdd0 msgr2=0x7faf200c5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.156+0000 7faf2610c640 1 --2- 172.21.15.83:0/1469517609 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf2015bdd0 0x7faf200c5610 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7faf10009870 tx=0x7faf1002b860 comp rx=0 tx=0).stop 2024-03-06T07:26:00.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.156+0000 7faf2610c640 1 -- 172.21.15.83:0/1469517609 shutdown_connections 2024-03-06T07:26:00.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.156+0000 7faf2610c640 1 --2- 172.21.15.83:0/1469517609 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf200d16e0 0x7faf200d1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.156+0000 7faf2610c640 1 --2- 172.21.15.83:0/1469517609 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf2015bdd0 0x7faf200c5610 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.156+0000 7faf2610c640 1 -- 172.21.15.83:0/1469517609 >> 172.21.15.83:0/1469517609 conn(0x7faf2009a9f0 msgr2=0x7faf2009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:00.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.156+0000 7faf2610c640 1 -- 172.21.15.83:0/1469517609 shutdown_connections 2024-03-06T07:26:00.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.156+0000 7faf2610c640 1 -- 172.21.15.83:0/1469517609 wait complete. 2024-03-06T07:26:00.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.157+0000 7faf2610c640 1 Processor -- start 2024-03-06T07:26:00.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.157+0000 7faf2610c640 1 -- start start 2024-03-06T07:26:00.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.158+0000 7faf2610c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf200d16e0 0x7faf20152b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.158+0000 7faf2610c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf2015bdd0 0x7faf200b72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.158+0000 7faf2610c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf2015fa10 0x7faf200c9c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.158+0000 7faf2610c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf200d2050 con 0x7faf2015fa10 2024-03-06T07:26:00.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.158+0000 7faf2610c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf200c5b50 con 0x7faf200d16e0 2024-03-06T07:26:00.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.158+0000 7faf2610c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf200cc5d0 con 0x7faf2015bdd0 2024-03-06T07:26:00.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.158+0000 7faf1ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf2015fa10 0x7faf200c9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.158+0000 7faf1ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf2015fa10 0x7faf200c9c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55646/0 (socket says 172.21.15.83:55646) 2024-03-06T07:26:00.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.158+0000 7faf1ffff640 1 -- 172.21.15.83:0/2331394219 learned_addr learned my addr 172.21.15.83:0/2331394219 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:00.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.158+0000 7faf1f7fe640 1 --2- 172.21.15.83:0/2331394219 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf200d16e0 0x7faf20152b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.158+0000 7faf1ffff640 1 -- 172.21.15.83:0/2331394219 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf2015bdd0 msgr2=0x7faf200b72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.158+0000 7faf1effd640 1 --2- 172.21.15.83:0/2331394219 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf2015bdd0 0x7faf200b72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.158+0000 7faf1ffff640 1 --2- 172.21.15.83:0/2331394219 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf2015bdd0 0x7faf200b72b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.158+0000 7faf1ffff640 1 -- 172.21.15.83:0/2331394219 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf200d16e0 msgr2=0x7faf20152b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.158+0000 7faf1ffff640 1 --2- 172.21.15.83:0/2331394219 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf200d16e0 0x7faf20152b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.158+0000 7faf1ffff640 1 -- 172.21.15.83:0/2331394219 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf200b6210 con 0x7faf2015fa10 2024-03-06T07:26:00.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.158+0000 7faf1effd640 1 --2- 172.21.15.83:0/2331394219 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf2015bdd0 0x7faf200b72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:00.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.158+0000 7faf1ffff640 1 --2- 172.21.15.83:0/2331394219 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf2015fa10 0x7faf200c9c90 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7faf1800cd50 tx=0x7faf1800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.158+0000 7faf07fff640 1 -- 172.21.15.83:0/2331394219 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf1800f680 con 0x7faf2015fa10 2024-03-06T07:26:00.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.158+0000 7faf07fff640 1 -- 172.21.15.83:0/2331394219 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf1800f840 con 0x7faf2015fa10 2024-03-06T07:26:00.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.158+0000 7faf2610c640 1 -- 172.21.15.83:0/2331394219 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf200b64f0 con 0x7faf2015fa10 2024-03-06T07:26:00.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.159+0000 7faf07fff640 1 -- 172.21.15.83:0/2331394219 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf1800fc30 con 0x7faf2015fa10 2024-03-06T07:26:00.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.159+0000 7faf2610c640 1 -- 172.21.15.83:0/2331394219 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf200b6ab0 con 0x7faf2015fa10 2024-03-06T07:26:00.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.159+0000 7faf2610c640 1 -- 172.21.15.83:0/2331394219 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf2004f330 con 0x7faf2015fa10 2024-03-06T07:26:00.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.160+0000 7faf07fff640 1 -- 172.21.15.83:0/2331394219 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faf180213f0 con 0x7faf2015fa10 2024-03-06T07:26:00.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.160+0000 7faf07fff640 1 --2- 172.21.15.83:0/2331394219 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7faf0c06b4d0 0x7faf0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.160+0000 7faf07fff640 1 -- 172.21.15.83:0/2331394219 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faf1808b0c0 con 0x7faf2015fa10 2024-03-06T07:26:00.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.160+0000 7faf1f7fe640 1 --2- 172.21.15.83:0/2331394219 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7faf0c06b4d0 0x7faf0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.161+0000 7faf1f7fe640 1 --2- 172.21.15.83:0/2331394219 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7faf0c06b4d0 0x7faf0c06d990 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7faf10036000 tx=0x7faf10008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.168+0000 7faf07fff640 1 -- 172.21.15.83:0/2331394219 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faf18014030 con 0x7faf2015fa10 2024-03-06T07:26:00.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.400+0000 7faf2610c640 1 -- 172.21.15.83:0/2331394219 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7faf2006e860 con 0x7faf2015fa10 2024-03-06T07:26:00.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.401+0000 7faf07fff640 1 -- 172.21.15.83:0/2331394219 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v39) v1 ==== 53+0+73 (secure 0 0 0) 0x7faf18059eb0 con 0x7faf2015fa10 2024-03-06T07:26:00.403 INFO:teuthology.orchestra.run.smithi083.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:26:00.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.403+0000 7faf2610c640 1 -- 172.21.15.83:0/2331394219 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7faf0c06b4d0 msgr2=0x7faf0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.403+0000 7faf2610c640 1 --2- 172.21.15.83:0/2331394219 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7faf0c06b4d0 0x7faf0c06d990 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7faf10036000 tx=0x7faf10008030 comp rx=0 tx=0).stop 2024-03-06T07:26:00.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.403+0000 7faf2610c640 1 -- 172.21.15.83:0/2331394219 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf2015fa10 msgr2=0x7faf200c9c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.403+0000 7faf2610c640 1 --2- 172.21.15.83:0/2331394219 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf2015fa10 0x7faf200c9c90 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7faf1800cd50 tx=0x7faf1800a650 comp rx=0 tx=0).stop 2024-03-06T07:26:00.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.403+0000 7faf2610c640 1 -- 172.21.15.83:0/2331394219 shutdown_connections 2024-03-06T07:26:00.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.403+0000 7faf2610c640 1 --2- 172.21.15.83:0/2331394219 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7faf0c06b4d0 0x7faf0c06d990 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.403+0000 7faf2610c640 1 --2- 172.21.15.83:0/2331394219 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf2015fa10 0x7faf200c9c90 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.403+0000 7faf2610c640 1 --2- 172.21.15.83:0/2331394219 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf2015bdd0 0x7faf200b72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.403+0000 7faf2610c640 1 --2- 172.21.15.83:0/2331394219 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf200d16e0 0x7faf20152b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.403+0000 7faf2610c640 1 -- 172.21.15.83:0/2331394219 >> 172.21.15.83:0/2331394219 conn(0x7faf2009a9f0 msgr2=0x7faf200b3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:00.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.403+0000 7faf2610c640 1 -- 172.21.15.83:0/2331394219 shutdown_connections 2024-03-06T07:26:00.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:00.403+0000 7faf2610c640 1 -- 172.21.15.83:0/2331394219 wait complete. 2024-03-06T07:26:00.415 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:26:00.416 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:26:00.416 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi083.front.sepia.ceph.com 2024-03-06T07:26:00.416 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:26:00.416 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:00.416 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:26:00.416 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:26:00.416 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:26:00.416 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:26:00.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:00.416 DEBUG:teuthology.orchestra.run.smithi083:> ip addr 2024-03-06T07:26:00.431 INFO:teuthology.orchestra.run.smithi083.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:26:00.431 INFO:teuthology.orchestra.run.smithi083.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:26:00.431 INFO:teuthology.orchestra.run.smithi083.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:26:00.432 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:00.432 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 ::1/128 scope host 2024-03-06T07:26:00.432 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:00.432 INFO:teuthology.orchestra.run.smithi083.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:26:00.432 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:d8:da:1e brd ff:ff:ff:ff:ff:ff 2024-03-06T07:26:00.432 INFO:teuthology.orchestra.run.smithi083.stdout: altname enp7s0f0 2024-03-06T07:26:00.432 INFO:teuthology.orchestra.run.smithi083.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:26:00.432 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:d8:da:1f brd ff:ff:ff:ff:ff:ff 2024-03-06T07:26:00.432 INFO:teuthology.orchestra.run.smithi083.stdout: altname enp7s0f1 2024-03-06T07:26:00.432 INFO:teuthology.orchestra.run.smithi083.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:26:00.432 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:8f:ce:90 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:26:00.432 INFO:teuthology.orchestra.run.smithi083.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:26:00.432 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:8f:ce:91 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:26:00.433 INFO:teuthology.orchestra.run.smithi083.stdout: inet 172.21.15.83/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:26:00.433 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft 42495sec preferred_lft 42495sec 2024-03-06T07:26:00.433 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::ec4:7aff:fe8f:ce91/64 scope link 2024-03-06T07:26:00.433 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:00.433 INFO:teuthology.orchestra.run.smithi083.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:26:00.433 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 1e:29:46:c9:e7:c8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:26:00.433 INFO:teuthology.orchestra.run.smithi083.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:26:00.433 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:00.433 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::60c5:d7ff:fe70:2b1c/64 scope link 2024-03-06T07:26:00.433 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:00.433 INFO:teuthology.orchestra.run.smithi083.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:26:00.433 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 1e:29:46:c9:e7:c8 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:26:00.433 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::1c29:46ff:fec9:e7c8/64 scope link 2024-03-06T07:26:00.433 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:00.433 INFO:teuthology.orchestra.run.smithi083.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:26:00.434 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 3e:00:2f:23:d5:45 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:26:00.434 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::3c00:2fff:fe23:d545/64 scope link 2024-03-06T07:26:00.434 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:00.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:00.434 DEBUG:teuthology.orchestra.run.smithi083:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:26:00.496 INFO:teuthology.orchestra.run.smithi083.stdout:1 2024-03-06T07:26:00.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:00.498 DEBUG:teuthology.orchestra.run.smithi083:> route 2024-03-06T07:26:00.554 INFO:teuthology.orchestra.run.smithi083.stdout:Kernel IP routing table 2024-03-06T07:26:00.555 INFO:teuthology.orchestra.run.smithi083.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:26:00.555 INFO:teuthology.orchestra.run.smithi083.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:26:00.555 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:26:00.555 INFO:teuthology.orchestra.run.smithi083.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:26:00.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:00.555 DEBUG:teuthology.orchestra.run.smithi083:> (cd / && exec bash -c ' 2024-03-06T07:26:00.555 DEBUG:teuthology.orchestra.run.smithi083:> set -e 2024-03-06T07:26:00.555 DEBUG:teuthology.orchestra.run.smithi083:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:26:00.555 DEBUG:teuthology.orchestra.run.smithi083:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:26:00.555 DEBUG:teuthology.orchestra.run.smithi083:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:26:00.556 DEBUG:teuthology.orchestra.run.smithi083:> ') 2024-03-06T07:26:00.649 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:00.649 DEBUG:teuthology.orchestra.run.smithi083:> ip netns list 2024-03-06T07:26:00.664 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:26:00.664 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:26:00.664 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:26:00.664 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:00.664 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip link set brx.0 up 2024-03-06T07:26:00.735 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:26:00.735 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:26:00.735 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:00.790 INFO:teuthology.orchestra.run.smithi083.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:26:00.791 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:26:00.791 DEBUG:teuthology.orchestra.run.smithi083:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:00.844 DEBUG:teuthology.orchestra.run.smithi083:> sudo modprobe fuse 2024-03-06T07:26:00.911 DEBUG:teuthology.orchestra.run.smithi083:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:26:00.966 INFO:teuthology.orchestra.run.smithi083.stdout:/proc 2024-03-06T07:26:00.966 INFO:teuthology.orchestra.run.smithi083.stdout:/sys 2024-03-06T07:26:00.966 INFO:teuthology.orchestra.run.smithi083.stdout:/dev 2024-03-06T07:26:00.967 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/security 2024-03-06T07:26:00.967 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/shm 2024-03-06T07:26:00.967 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/pts 2024-03-06T07:26:00.967 INFO:teuthology.orchestra.run.smithi083.stdout:/run 2024-03-06T07:26:00.967 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/cgroup 2024-03-06T07:26:00.967 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/pstore 2024-03-06T07:26:00.967 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/bpf 2024-03-06T07:26:00.967 INFO:teuthology.orchestra.run.smithi083.stdout:/ 2024-03-06T07:26:00.967 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/selinux 2024-03-06T07:26:00.967 INFO:teuthology.orchestra.run.smithi083.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:26:00.967 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/mqueue 2024-03-06T07:26:00.967 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/hugepages 2024-03-06T07:26:00.967 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/debug 2024-03-06T07:26:00.968 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/tracing 2024-03-06T07:26:00.968 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:26:00.968 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/fuse/connections 2024-03-06T07:26:00.968 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/config 2024-03-06T07:26:00.968 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:26:00.968 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:26:00.968 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:26:00.968 INFO:teuthology.orchestra.run.smithi083.stdout:/run/user/1000 2024-03-06T07:26:00.968 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph 2024-03-06T07:26:00.968 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:26:00.968 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:26:00.968 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:26:00.969 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:26:00.969 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns 2024-03-06T07:26:00.969 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:26:00.969 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:26:00.969 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:26:00.969 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:26:00.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:00.969 DEBUG:teuthology.orchestra.run.smithi083:> ls /sys/fs/fuse/connections 2024-03-06T07:26:01.020 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:26:01.020 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:26:01.063 DEBUG:teuthology.orchestra.run.smithi083:> sudo modprobe fuse 2024-03-06T07:26:01.091 DEBUG:teuthology.orchestra.run.smithi083:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:26:01.145 INFO:teuthology.orchestra.run.smithi083.stdout:/proc 2024-03-06T07:26:01.146 INFO:teuthology.orchestra.run.smithi083.stdout:/sys 2024-03-06T07:26:01.146 INFO:teuthology.orchestra.run.smithi083.stdout:/dev 2024-03-06T07:26:01.146 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/security 2024-03-06T07:26:01.146 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/shm 2024-03-06T07:26:01.146 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/pts 2024-03-06T07:26:01.146 INFO:teuthology.orchestra.run.smithi083.stdout:/run 2024-03-06T07:26:01.146 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/cgroup 2024-03-06T07:26:01.146 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/pstore 2024-03-06T07:26:01.146 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/bpf 2024-03-06T07:26:01.146 INFO:teuthology.orchestra.run.smithi083.stdout:/ 2024-03-06T07:26:01.146 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/selinux 2024-03-06T07:26:01.146 INFO:teuthology.orchestra.run.smithi083.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:26:01.146 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/mqueue 2024-03-06T07:26:01.147 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/hugepages 2024-03-06T07:26:01.147 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/debug 2024-03-06T07:26:01.147 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/tracing 2024-03-06T07:26:01.147 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:26:01.147 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/fuse/connections 2024-03-06T07:26:01.147 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/config 2024-03-06T07:26:01.147 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:26:01.147 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:26:01.147 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:26:01.147 INFO:teuthology.orchestra.run.smithi083.stdout:/run/user/1000 2024-03-06T07:26:01.147 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph 2024-03-06T07:26:01.147 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:26:01.147 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:26:01.148 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:26:01.148 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:26:01.148 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns 2024-03-06T07:26:01.148 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:26:01.148 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:26:01.148 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:26:01.148 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:26:01.149 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:01.149 DEBUG:teuthology.orchestra.run.smithi083:> ls /sys/fs/fuse/connections 2024-03-06T07:26:01.152 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:2024-03-06T07:26:01.150+0000 7fc862c3f480 -1 init, newargv = 0x563423a19090 newargc=15 2024-03-06T07:26:01.152 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:ceph-fuse[66134]: starting ceph client 2024-03-06T07:26:01.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:ceph-fuse[66134]: starting fuse 2024-03-06T07:26:01.166 INFO:teuthology.orchestra.run.smithi083.stdout:41 2024-03-06T07:26:01.166 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-06T07:26:01.167 DEBUG:teuthology.orchestra.run.smithi083:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:26:01.167 DEBUG:teuthology.orchestra.run.smithi083:> import glob 2024-03-06T07:26:01.167 DEBUG:teuthology.orchestra.run.smithi083:> import re 2024-03-06T07:26:01.167 DEBUG:teuthology.orchestra.run.smithi083:> import os 2024-03-06T07:26:01.167 DEBUG:teuthology.orchestra.run.smithi083:> import subprocess 2024-03-06T07:26:01.167 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:26:01.167 DEBUG:teuthology.orchestra.run.smithi083:> def _find_admin_socket(client_name): 2024-03-06T07:26:01.167 DEBUG:teuthology.orchestra.run.smithi083:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:26:01.167 DEBUG:teuthology.orchestra.run.smithi083:> files = glob.glob(asok_path) 2024-03-06T07:26:01.167 DEBUG:teuthology.orchestra.run.smithi083:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:26:01.167 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:26:01.167 DEBUG:teuthology.orchestra.run.smithi083:> # Given a non-glob path, it better be there 2024-03-06T07:26:01.167 DEBUG:teuthology.orchestra.run.smithi083:> if "*" not in asok_path: 2024-03-06T07:26:01.167 DEBUG:teuthology.orchestra.run.smithi083:> assert(len(files) == 1) 2024-03-06T07:26:01.168 DEBUG:teuthology.orchestra.run.smithi083:> return files[0] 2024-03-06T07:26:01.168 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:26:01.168 DEBUG:teuthology.orchestra.run.smithi083:> for f in files: 2024-03-06T07:26:01.168 DEBUG:teuthology.orchestra.run.smithi083:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:26:01.168 DEBUG:teuthology.orchestra.run.smithi083:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:26:01.168 DEBUG:teuthology.orchestra.run.smithi083:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:26:01.168 DEBUG:teuthology.orchestra.run.smithi083:> contents = proc_f.read() 2024-03-06T07:26:01.168 DEBUG:teuthology.orchestra.run.smithi083:> if mountpoint in contents: 2024-03-06T07:26:01.168 DEBUG:teuthology.orchestra.run.smithi083:> return f 2024-03-06T07:26:01.168 DEBUG:teuthology.orchestra.run.smithi083:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:26:01.168 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:26:01.168 DEBUG:teuthology.orchestra.run.smithi083:> print(_find_admin_socket("client.0")) 2024-03-06T07:26:01.168 DEBUG:teuthology.orchestra.run.smithi083:> ' 2024-03-06T07:26:01.276 INFO:tasks.ceph.mds.c.smithi083.stderr:2024-03-06T07:26:01.275+0000 7f18123cd640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:26:01.287 INFO:teuthology.orchestra.run.smithi083.stdout:/var/run/ceph/ceph-client.0.66134.asok 2024-03-06T07:26:01.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:26:01.298 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.66134.asok 2024-03-06T07:26:01.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:01.298 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.66134.asok status 2024-03-06T07:26:01.434 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:26:01.434 INFO:teuthology.orchestra.run.smithi083.stdout: "metadata": { 2024-03-06T07:26:01.434 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:26:01.434 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:26:01.434 INFO:teuthology.orchestra.run.smithi083.stdout: "entity_id": "0", 2024-03-06T07:26:01.435 INFO:teuthology.orchestra.run.smithi083.stdout: "hostname": "smithi083", 2024-03-06T07:26:01.435 INFO:teuthology.orchestra.run.smithi083.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:26:01.435 INFO:teuthology.orchestra.run.smithi083.stdout: "pid": "66134", 2024-03-06T07:26:01.435 INFO:teuthology.orchestra.run.smithi083.stdout: "root": "/" 2024-03-06T07:26:01.435 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:26:01.435 INFO:teuthology.orchestra.run.smithi083.stdout: "dentry_count": 0, 2024-03-06T07:26:01.435 INFO:teuthology.orchestra.run.smithi083.stdout: "dentry_pinned_count": 0, 2024-03-06T07:26:01.435 INFO:teuthology.orchestra.run.smithi083.stdout: "id": 7125, 2024-03-06T07:26:01.435 INFO:teuthology.orchestra.run.smithi083.stdout: "inst": { 2024-03-06T07:26:01.435 INFO:teuthology.orchestra.run.smithi083.stdout: "name": { 2024-03-06T07:26:01.435 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "client", 2024-03-06T07:26:01.435 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 7125 2024-03-06T07:26:01.435 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:26:01.435 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": { 2024-03-06T07:26:01.436 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-03-06T07:26:01.436 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:26:01.436 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 2082981069 2024-03-06T07:26:01.436 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:26:01.436 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:26:01.436 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": { 2024-03-06T07:26:01.436 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-03-06T07:26:01.436 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:26:01.436 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 2082981069 2024-03-06T07:26:01.436 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:26:01.436 INFO:teuthology.orchestra.run.smithi083.stdout: "inst_str": "client.7125 192.168.0.1:0/2082981069", 2024-03-06T07:26:01.436 INFO:teuthology.orchestra.run.smithi083.stdout: "addr_str": "192.168.0.1:0/2082981069", 2024-03-06T07:26:01.436 INFO:teuthology.orchestra.run.smithi083.stdout: "inode_count": 1, 2024-03-06T07:26:01.436 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_epoch": 39, 2024-03-06T07:26:01.437 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_epoch": 82, 2024-03-06T07:26:01.437 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:26:01.437 INFO:teuthology.orchestra.run.smithi083.stdout: "blocklisted": false, 2024-03-06T07:26:01.437 INFO:teuthology.orchestra.run.smithi083.stdout: "fs_name": "cephfs" 2024-03-06T07:26:01.437 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:26:01.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:01.443 DEBUG:teuthology.orchestra.run.smithi083:> stat --file-system '--printf=%T 2024-03-06T07:26:01.443 DEBUG:teuthology.orchestra.run.smithi083:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:01.458 INFO:teuthology.orchestra.run.smithi083.stdout:fuseblk 2024-03-06T07:26:01.458 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:01.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:01.458 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:01.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:01.526 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch fancy_layout_file) 2024-03-06T07:26:01.587 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:01.587 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.file.layout -v 'stripe_unit=1048576 stripe_count=4 object_size=8388608' fancy_layout_file) 2024-03-06T07:26:01.654 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:26:01.654 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:01.709 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:01.709 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=fancy_layout_file bs=1M conv=fdatasync count=35 seek=0) 2024-03-06T07:26:01.928 INFO:teuthology.orchestra.run.smithi083.stderr:35+0 records in 2024-03-06T07:26:01.928 INFO:teuthology.orchestra.run.smithi083.stderr:35+0 records out 2024-03-06T07:26:01.928 INFO:teuthology.orchestra.run.smithi083.stderr:36700160 bytes (37 MB, 35 MiB) copied, 0.159598 s, 230 MB/s 2024-03-06T07:26:01.928 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:01.928 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f fancy_layout_file) 2024-03-06T07:26:01.946 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:02.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.087+0000 7fce41313640 1 Processor -- start 2024-03-06T07:26:02.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.087+0000 7fce41313640 1 -- start start 2024-03-06T07:26:02.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.088+0000 7fce41313640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fce3c06b220 0x7fce3c15e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.088+0000 7fce41313640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fce3c0ce920 0x7fce3c071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.088+0000 7fce41313640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce3c15ea30 con 0x7fce3c071ec0 2024-03-06T07:26:02.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.088+0000 7fce41313640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce3c161b70 con 0x7fce3c06b220 2024-03-06T07:26:02.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.088+0000 7fce41313640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce3c15ed10 con 0x7fce3c0ce920 2024-03-06T07:26:02.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.088+0000 7fce3affd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fce3c06b220 0x7fce3c15e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.088+0000 7fce3b7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fce3c071ec0 0x7fce3c162000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:33090/0 (socket says 172.21.15.83:33090) 2024-03-06T07:26:02.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.088+0000 7fce3b7fe640 1 -- 172.21.15.83:0/1285492501 learned_addr learned my addr 172.21.15.83:0/1285492501 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:02.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.089+0000 7fce3a7fc640 1 --2- 172.21.15.83:0/1285492501 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fce3c0ce920 0x7fce3c071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.089+0000 7fce3affd640 1 -- 172.21.15.83:0/1285492501 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fce3c0ce920 msgr2=0x7fce3c071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:02.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.089+0000 7fce3affd640 1 --2- 172.21.15.83:0/1285492501 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fce3c0ce920 0x7fce3c071980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.089+0000 7fce3affd640 1 -- 172.21.15.83:0/1285492501 >> v1:172.21.15.83:6789/0 conn(0x7fce3c071ec0 legacy=0x7fce3c162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.089+0000 7fce3affd640 1 -- 172.21.15.83:0/1285492501 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce3c15eff0 con 0x7fce3c06b220 2024-03-06T07:26:02.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.089+0000 7fce3affd640 1 --2- 172.21.15.83:0/1285492501 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce3c06b220 0x7fce3c15e450 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fce300097a0 tx=0x7fce3002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c09e8df8d0a0a4b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:02.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.090+0000 7fce39ffb640 1 -- 172.21.15.83:0/1285492501 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce30035020 con 0x7fce3c06b220 2024-03-06T07:26:02.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.090+0000 7fce39ffb640 1 -- 172.21.15.83:0/1285492501 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce30031830 con 0x7fce3c06b220 2024-03-06T07:26:02.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.090+0000 7fce39ffb640 1 -- 172.21.15.83:0/1285492501 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce30031c00 con 0x7fce3c06b220 2024-03-06T07:26:02.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.090+0000 7fce41313640 1 -- 172.21.15.83:0/1285492501 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce3c06b220 msgr2=0x7fce3c15e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.090+0000 7fce41313640 1 --2- 172.21.15.83:0/1285492501 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce3c06b220 0x7fce3c15e450 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fce300097a0 tx=0x7fce3002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.090+0000 7fce41313640 1 -- 172.21.15.83:0/1285492501 shutdown_connections 2024-03-06T07:26:02.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.090+0000 7fce41313640 1 --2- 172.21.15.83:0/1285492501 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fce3c0ce920 0x7fce3c071980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.090+0000 7fce41313640 1 --2- 172.21.15.83:0/1285492501 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce3c06b220 0x7fce3c15e450 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.090+0000 7fce41313640 1 -- 172.21.15.83:0/1285492501 >> 172.21.15.83:0/1285492501 conn(0x7fce3c06c0c0 msgr2=0x7fce3c06c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:02.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.090+0000 7fce41313640 1 -- 172.21.15.83:0/1285492501 shutdown_connections 2024-03-06T07:26:02.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.090+0000 7fce41313640 1 -- 172.21.15.83:0/1285492501 wait complete. 2024-03-06T07:26:02.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.091+0000 7fce41313640 1 Processor -- start 2024-03-06T07:26:02.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.092+0000 7fce41313640 1 -- start start 2024-03-06T07:26:02.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.092+0000 7fce41313640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fce3c06b220 0x7fce3c1f4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.092+0000 7fce41313640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce3c071ec0 0x7fce3c1e8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.092+0000 7fce41313640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce3c0ce920 0x7fce3c15c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.092+0000 7fce41313640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce3c0ce380 con 0x7fce3c071ec0 2024-03-06T07:26:02.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.092+0000 7fce41313640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce3c0ce200 con 0x7fce3c0ce920 2024-03-06T07:26:02.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.092+0000 7fce41313640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce3c162710 con 0x7fce3c06b220 2024-03-06T07:26:02.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.092+0000 7fce3a7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce3c071ec0 0x7fce3c1e8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.092+0000 7fce3a7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce3c071ec0 0x7fce3c1e8a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55676/0 (socket says 172.21.15.83:55676) 2024-03-06T07:26:02.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.092+0000 7fce3a7fc640 1 -- 172.21.15.83:0/1903085910 learned_addr learned my addr 172.21.15.83:0/1903085910 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:02.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.092+0000 7fce3affd640 1 --2- 172.21.15.83:0/1903085910 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fce3c06b220 0x7fce3c1f4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.092+0000 7fce3b7fe640 1 --2- 172.21.15.83:0/1903085910 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce3c0ce920 0x7fce3c15c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.093+0000 7fce3a7fc640 1 -- 172.21.15.83:0/1903085910 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fce3c06b220 msgr2=0x7fce3c1f4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.093+0000 7fce3a7fc640 1 --2- 172.21.15.83:0/1903085910 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fce3c06b220 0x7fce3c1f4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.093+0000 7fce3a7fc640 1 -- 172.21.15.83:0/1903085910 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce3c0ce920 msgr2=0x7fce3c15c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.093+0000 7fce3a7fc640 1 --2- 172.21.15.83:0/1903085910 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce3c0ce920 0x7fce3c15c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.093+0000 7fce3a7fc640 1 -- 172.21.15.83:0/1903085910 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce3c1e8f80 con 0x7fce3c071ec0 2024-03-06T07:26:02.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.093+0000 7fce3a7fc640 1 --2- 172.21.15.83:0/1903085910 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce3c071ec0 0x7fce3c1e8a40 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fce2c00d800 tx=0x7fce2c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:02.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.093+0000 7fce237fe640 1 -- 172.21.15.83:0/1903085910 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce2c00b6c0 con 0x7fce3c071ec0 2024-03-06T07:26:02.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.093+0000 7fce237fe640 1 -- 172.21.15.83:0/1903085910 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce2c019070 con 0x7fce3c071ec0 2024-03-06T07:26:02.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.093+0000 7fce41313640 1 -- 172.21.15.83:0/1903085910 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce3c1f4660 con 0x7fce3c071ec0 2024-03-06T07:26:02.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.093+0000 7fce237fe640 1 -- 172.21.15.83:0/1903085910 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce2c00ba80 con 0x7fce3c071ec0 2024-03-06T07:26:02.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.093+0000 7fce41313640 1 -- 172.21.15.83:0/1903085910 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce3c1f4c20 con 0x7fce3c071ec0 2024-03-06T07:26:02.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.094+0000 7fce41313640 1 -- 172.21.15.83:0/1903085910 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce3c1f78a0 con 0x7fce3c071ec0 2024-03-06T07:26:02.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.094+0000 7fce237fe640 1 -- 172.21.15.83:0/1903085910 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fce2c00bc20 con 0x7fce3c071ec0 2024-03-06T07:26:02.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.095+0000 7fce237fe640 1 --2- 172.21.15.83:0/1903085910 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fce2806b4d0 0x7fce2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.095+0000 7fce237fe640 1 -- 172.21.15.83:0/1903085910 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fce2c08a5e0 con 0x7fce3c071ec0 2024-03-06T07:26:02.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.095+0000 7fce3affd640 1 --2- 172.21.15.83:0/1903085910 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fce2806b4d0 0x7fce2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.095+0000 7fce3affd640 1 --2- 172.21.15.83:0/1903085910 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fce2806b4d0 0x7fce2806d990 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fce30036000 tx=0x7fce3003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:02.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.099+0000 7fce237fe640 1 -- 172.21.15.83:0/1903085910 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fce3c1f78a0 con 0x7fce3c071ec0 2024-03-06T07:26:02.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.315+0000 7fce41313640 1 -- 172.21.15.83:0/1903085910 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce3c04f330 con 0x7fce3c071ec0 2024-03-06T07:26:02.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.315+0000 7fce237fe640 1 -- 172.21.15.83:0/1903085910 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+4982 (secure 0 0 0) 0x7fce2c058d90 con 0x7fce3c071ec0 2024-03-06T07:26:02.317 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:02.317 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":39,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/2525949760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":2525949760},{"type":"v1","addr":"172.21.15.183:6834","nonce":2525949760}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35},{"gid":6262,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2104447869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2104447869},{"type":"v1","addr":"172.21.15.183:6835","nonce":2104447869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6954,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/779621621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":779621621},{"type":"v1","addr":"172.21.15.83:6836","nonce":779621621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:54.631963+0000","modified":"2024-03-06T07:25:56.637047+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6957":{"gid":6957,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/3056214366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":3056214366},{"type":"v1","addr":"172.21.15.83:6837","nonce":3056214366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6957,"qdb_cluster":[6957]},"id":6}]} 2024-03-06T07:26:02.318 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 39 2024-03-06T07:26:02.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.317+0000 7fce41313640 1 -- 172.21.15.83:0/1903085910 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fce2806b4d0 msgr2=0x7fce2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.317+0000 7fce41313640 1 --2- 172.21.15.83:0/1903085910 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fce2806b4d0 0x7fce2806d990 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fce30036000 tx=0x7fce3003c040 comp rx=0 tx=0).stop 2024-03-06T07:26:02.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.317+0000 7fce41313640 1 -- 172.21.15.83:0/1903085910 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce3c071ec0 msgr2=0x7fce3c1e8a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.317+0000 7fce41313640 1 --2- 172.21.15.83:0/1903085910 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce3c071ec0 0x7fce3c1e8a40 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fce2c00d800 tx=0x7fce2c00dcc0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.317+0000 7fce41313640 1 -- 172.21.15.83:0/1903085910 shutdown_connections 2024-03-06T07:26:02.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.317+0000 7fce41313640 1 --2- 172.21.15.83:0/1903085910 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fce2806b4d0 0x7fce2806d990 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.317+0000 7fce41313640 1 --2- 172.21.15.83:0/1903085910 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce3c0ce920 0x7fce3c15c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.317+0000 7fce41313640 1 --2- 172.21.15.83:0/1903085910 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce3c071ec0 0x7fce3c1e8a40 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.317+0000 7fce41313640 1 --2- 172.21.15.83:0/1903085910 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fce3c06b220 0x7fce3c1f4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.317+0000 7fce41313640 1 -- 172.21.15.83:0/1903085910 >> 172.21.15.83:0/1903085910 conn(0x7fce3c06c0c0 msgr2=0x7fce3c0d06a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:02.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.317+0000 7fce41313640 1 -- 172.21.15.83:0/1903085910 shutdown_connections 2024-03-06T07:26:02.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.317+0000 7fce41313640 1 -- 172.21.15.83:0/1903085910 wait complete. 2024-03-06T07:26:02.330 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:26:02.463 INFO:teuthology.orchestra.run.smithi083.stdout:{"message":"","return_code":0} 2024-03-06T07:26:02.463 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:26:02.463 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2024-03-06T07:26:02.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.562+0000 7f7d525f7640 1 Processor -- start 2024-03-06T07:26:02.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.562+0000 7f7d525f7640 1 -- start start 2024-03-06T07:26:02.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.563+0000 7f7d525f7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7d4c162970 0x7f7d4c0d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.563+0000 7f7d525f7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d4c161cf0 0x7f7d4c162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.563+0000 7f7d525f7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d4c0d22c0 con 0x7f7d4c0b4460 2024-03-06T07:26:02.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.563+0000 7f7d525f7640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d4c161b00 con 0x7f7d4c162970 2024-03-06T07:26:02.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.563+0000 7f7d525f7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4c0cecc0 con 0x7f7d4c161cf0 2024-03-06T07:26:02.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.563+0000 7f7d4b7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7d4c162970 0x7f7d4c0d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.563+0000 7f7d4bfff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f7d4c0b4460 0x7f7d4c0b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:33094/0 (socket says 172.21.15.83:33094) 2024-03-06T07:26:02.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.563+0000 7f7d4bfff640 1 -- 172.21.15.83:0/3359738632 learned_addr learned my addr 172.21.15.83:0/3359738632 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:02.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.563+0000 7f7d50b94640 1 --2- 172.21.15.83:0/3359738632 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d4c161cf0 0x7f7d4c162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.563+0000 7f7d4b7fe640 1 -- 172.21.15.83:0/3359738632 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d4c161cf0 msgr2=0x7f7d4c162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:02.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.563+0000 7f7d4b7fe640 1 --2- 172.21.15.83:0/3359738632 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d4c161cf0 0x7f7d4c162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.563+0000 7f7d4b7fe640 1 -- 172.21.15.83:0/3359738632 >> v1:172.21.15.83:6789/0 conn(0x7f7d4c0b4460 legacy=0x7f7d4c0b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.563+0000 7f7d4b7fe640 1 -- 172.21.15.83:0/3359738632 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d4c0d25a0 con 0x7f7d4c162970 2024-03-06T07:26:02.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.563+0000 7f7d4affd640 1 -- 172.21.15.83:0/3359738632 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 42264644 0 0) 0x7f7d4c0d22c0 con 0x7f7d4c0b4460 2024-03-06T07:26:02.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.564+0000 7f7d4b7fe640 1 --2- 172.21.15.83:0/3359738632 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d4c162970 0x7f7d4c0d4800 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f7d400097a0 tx=0x7f7d4002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0c9ecd3ce9161c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:02.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.564+0000 7f7d4affd640 1 -- 172.21.15.83:0/3359738632 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d40035020 con 0x7f7d4c162970 2024-03-06T07:26:02.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.564+0000 7f7d4affd640 1 -- 172.21.15.83:0/3359738632 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d40031830 con 0x7f7d4c162970 2024-03-06T07:26:02.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.564+0000 7f7d4affd640 1 -- 172.21.15.83:0/3359738632 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d40031c00 con 0x7f7d4c162970 2024-03-06T07:26:02.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.565+0000 7f7d525f7640 1 -- 172.21.15.83:0/3359738632 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d4c162970 msgr2=0x7f7d4c0d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.565+0000 7f7d525f7640 1 --2- 172.21.15.83:0/3359738632 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d4c162970 0x7f7d4c0d4800 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f7d400097a0 tx=0x7f7d4002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.565+0000 7f7d525f7640 1 -- 172.21.15.83:0/3359738632 shutdown_connections 2024-03-06T07:26:02.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.565+0000 7f7d525f7640 1 --2- 172.21.15.83:0/3359738632 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d4c161cf0 0x7f7d4c162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.565+0000 7f7d525f7640 1 --2- 172.21.15.83:0/3359738632 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d4c162970 0x7f7d4c0d4800 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.565+0000 7f7d525f7640 1 -- 172.21.15.83:0/3359738632 >> 172.21.15.83:0/3359738632 conn(0x7f7d4c0d3ce0 msgr2=0x7f7d4c0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:02.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.565+0000 7f7d525f7640 1 -- 172.21.15.83:0/3359738632 shutdown_connections 2024-03-06T07:26:02.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.565+0000 7f7d525f7640 1 -- 172.21.15.83:0/3359738632 wait complete. 2024-03-06T07:26:02.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.566+0000 7f7d525f7640 1 Processor -- start 2024-03-06T07:26:02.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.567+0000 7f7d525f7640 1 -- start start 2024-03-06T07:26:02.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.568+0000 7f7d525f7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d4c0b4460 0x7f7d4c155ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.568+0000 7f7d525f7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d4c161cf0 0x7f7d4c0c7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.568+0000 7f7d525f7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d4c162970 0x7f7d4c0c8240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.568+0000 7f7d525f7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4c0ce950 con 0x7f7d4c162970 2024-03-06T07:26:02.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.568+0000 7f7d525f7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4c0d2b60 con 0x7f7d4c161cf0 2024-03-06T07:26:02.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.568+0000 7f7d525f7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4c0b4840 con 0x7f7d4c0b4460 2024-03-06T07:26:02.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.568+0000 7f7d50b94640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d4c162970 0x7f7d4c0c8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.568+0000 7f7d50b94640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d4c162970 0x7f7d4c0c8240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55698/0 (socket says 172.21.15.83:55698) 2024-03-06T07:26:02.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.568+0000 7f7d50b94640 1 -- 172.21.15.83:0/3079008417 learned_addr learned my addr 172.21.15.83:0/3079008417 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:02.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.568+0000 7f7d4bfff640 1 --2- 172.21.15.83:0/3079008417 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d4c0b4460 0x7f7d4c155ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.568+0000 7f7d4b7fe640 1 --2- 172.21.15.83:0/3079008417 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d4c161cf0 0x7f7d4c0c7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.568+0000 7f7d50b94640 1 -- 172.21.15.83:0/3079008417 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d4c0b4460 msgr2=0x7f7d4c155ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.568+0000 7f7d50b94640 1 --2- 172.21.15.83:0/3079008417 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d4c0b4460 0x7f7d4c155ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.568+0000 7f7d50b94640 1 -- 172.21.15.83:0/3079008417 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d4c161cf0 msgr2=0x7f7d4c0c7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.568+0000 7f7d50b94640 1 --2- 172.21.15.83:0/3079008417 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d4c161cf0 0x7f7d4c0c7810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.568+0000 7f7d50b94640 1 -- 172.21.15.83:0/3079008417 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d4c0c7d50 con 0x7f7d4c162970 2024-03-06T07:26:02.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.568+0000 7f7d50b94640 1 --2- 172.21.15.83:0/3079008417 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d4c162970 0x7f7d4c0c8240 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f7d44007e90 tx=0x7f7d4400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:02.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.569+0000 7f7d48ff9640 1 -- 172.21.15.83:0/3079008417 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d44014420 con 0x7f7d4c162970 2024-03-06T07:26:02.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.569+0000 7f7d48ff9640 1 -- 172.21.15.83:0/3079008417 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d44019070 con 0x7f7d4c162970 2024-03-06T07:26:02.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.569+0000 7f7d525f7640 1 -- 172.21.15.83:0/3079008417 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d4c0b2270 con 0x7f7d4c162970 2024-03-06T07:26:02.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.569+0000 7f7d48ff9640 1 -- 172.21.15.83:0/3079008417 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d44014800 con 0x7f7d4c162970 2024-03-06T07:26:02.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.569+0000 7f7d525f7640 1 -- 172.21.15.83:0/3079008417 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d4c0b2830 con 0x7f7d4c162970 2024-03-06T07:26:02.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.570+0000 7f7d525f7640 1 -- 172.21.15.83:0/3079008417 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d4c04f330 con 0x7f7d4c162970 2024-03-06T07:26:02.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.570+0000 7f7d48ff9640 1 -- 172.21.15.83:0/3079008417 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7d440149a0 con 0x7f7d4c162970 2024-03-06T07:26:02.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.571+0000 7f7d48ff9640 1 --2- 172.21.15.83:0/3079008417 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7d3806b500 0x7f7d3806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.571+0000 7f7d48ff9640 1 -- 172.21.15.83:0/3079008417 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7d4408aa30 con 0x7f7d4c162970 2024-03-06T07:26:02.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.571+0000 7f7d4bfff640 1 --2- 172.21.15.83:0/3079008417 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7d3806b500 0x7f7d3806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.571+0000 7f7d4bfff640 1 --2- 172.21.15.83:0/3079008417 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7d3806b500 0x7f7d3806d9c0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c000f60 tx=0x7f7d3c007c00 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:02.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.575+0000 7f7d48ff9640 1 -- 172.21.15.83:0/3079008417 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7d440598b0 con 0x7f7d4c162970 2024-03-06T07:26:02.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.813+0000 7f7d525f7640 1 -- 172.21.15.83:0/3079008417 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7f7d4c06b180 con 0x7f7d4c162970 2024-03-06T07:26:02.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.813+0000 7f7d48ff9640 1 -- 172.21.15.83:0/3079008417 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1653 (secure 0 0 0) 0x7f7d440591b0 con 0x7f7d4c162970 2024-03-06T07:26:02.815 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:02.815 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:26:02.815 INFO:teuthology.orchestra.run.smithi083.stdout: "stats": { 2024-03-06T07:26:02.815 INFO:teuthology.orchestra.run.smithi083.stdout: "total_bytes": 773094113280, 2024-03-06T07:26:02.816 INFO:teuthology.orchestra.run.smithi083.stdout: "total_avail_bytes": 772829487104, 2024-03-06T07:26:02.816 INFO:teuthology.orchestra.run.smithi083.stdout: "total_used_bytes": 264626176, 2024-03-06T07:26:02.816 INFO:teuthology.orchestra.run.smithi083.stdout: "total_used_raw_bytes": 264626176, 2024-03-06T07:26:02.816 INFO:teuthology.orchestra.run.smithi083.stdout: "total_used_raw_ratio": 0.00034229489392600954, 2024-03-06T07:26:02.816 INFO:teuthology.orchestra.run.smithi083.stdout: "num_osds": 8, 2024-03-06T07:26:02.816 INFO:teuthology.orchestra.run.smithi083.stdout: "num_per_pool_osds": 8, 2024-03-06T07:26:02.816 INFO:teuthology.orchestra.run.smithi083.stdout: "num_per_pool_omap_osds": 8 2024-03-06T07:26:02.816 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:26:02.816 INFO:teuthology.orchestra.run.smithi083.stdout: "stats_by_class": { 2024-03-06T07:26:02.816 INFO:teuthology.orchestra.run.smithi083.stdout: "ssd": { 2024-03-06T07:26:02.816 INFO:teuthology.orchestra.run.smithi083.stdout: "total_bytes": 773094113280, 2024-03-06T07:26:02.816 INFO:teuthology.orchestra.run.smithi083.stdout: "total_avail_bytes": 772829487104, 2024-03-06T07:26:02.816 INFO:teuthology.orchestra.run.smithi083.stdout: "total_used_bytes": 264626176, 2024-03-06T07:26:02.817 INFO:teuthology.orchestra.run.smithi083.stdout: "total_used_raw_bytes": 264626176, 2024-03-06T07:26:02.817 INFO:teuthology.orchestra.run.smithi083.stdout: "total_used_raw_ratio": 0.00034229489392600954 2024-03-06T07:26:02.817 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:26:02.817 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:26:02.817 INFO:teuthology.orchestra.run.smithi083.stdout: "pools": [ 2024-03-06T07:26:02.817 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:26:02.817 INFO:teuthology.orchestra.run.smithi083.stdout: "name": ".mgr", 2024-03-06T07:26:02.817 INFO:teuthology.orchestra.run.smithi083.stdout: "id": 1, 2024-03-06T07:26:02.817 INFO:teuthology.orchestra.run.smithi083.stdout: "stats": { 2024-03-06T07:26:02.817 INFO:teuthology.orchestra.run.smithi083.stdout: "stored": 590368, 2024-03-06T07:26:02.817 INFO:teuthology.orchestra.run.smithi083.stdout: "objects": 2, 2024-03-06T07:26:02.817 INFO:teuthology.orchestra.run.smithi083.stdout: "kb_used": 1160, 2024-03-06T07:26:02.817 INFO:teuthology.orchestra.run.smithi083.stdout: "bytes_used": 1187840, 2024-03-06T07:26:02.817 INFO:teuthology.orchestra.run.smithi083.stdout: "percent_used": 1.6179310478037223e-06, 2024-03-06T07:26:02.818 INFO:teuthology.orchestra.run.smithi083.stdout: "max_avail": 367085518848 2024-03-06T07:26:02.818 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:26:02.818 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:26:02.818 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:26:02.818 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "cephfs_metadata", 2024-03-06T07:26:02.818 INFO:teuthology.orchestra.run.smithi083.stdout: "id": 12, 2024-03-06T07:26:02.818 INFO:teuthology.orchestra.run.smithi083.stdout: "stats": { 2024-03-06T07:26:02.818 INFO:teuthology.orchestra.run.smithi083.stdout: "stored": 2410, 2024-03-06T07:26:02.818 INFO:teuthology.orchestra.run.smithi083.stdout: "objects": 22, 2024-03-06T07:26:02.818 INFO:teuthology.orchestra.run.smithi083.stdout: "kb_used": 64, 2024-03-06T07:26:02.818 INFO:teuthology.orchestra.run.smithi083.stdout: "bytes_used": 65536, 2024-03-06T07:26:02.818 INFO:teuthology.orchestra.run.smithi083.stdout: "percent_used": 8.9265292047002731e-08, 2024-03-06T07:26:02.818 INFO:teuthology.orchestra.run.smithi083.stdout: "max_avail": 367085518848 2024-03-06T07:26:02.818 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:26:02.818 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:26:02.819 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:26:02.819 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "cephfs_data", 2024-03-06T07:26:02.819 INFO:teuthology.orchestra.run.smithi083.stdout: "id": 13, 2024-03-06T07:26:02.819 INFO:teuthology.orchestra.run.smithi083.stdout: "stats": { 2024-03-06T07:26:02.819 INFO:teuthology.orchestra.run.smithi083.stdout: "stored": 0, 2024-03-06T07:26:02.819 INFO:teuthology.orchestra.run.smithi083.stdout: "objects": 0, 2024-03-06T07:26:02.819 INFO:teuthology.orchestra.run.smithi083.stdout: "kb_used": 0, 2024-03-06T07:26:02.819 INFO:teuthology.orchestra.run.smithi083.stdout: "bytes_used": 0, 2024-03-06T07:26:02.819 INFO:teuthology.orchestra.run.smithi083.stdout: "percent_used": 0, 2024-03-06T07:26:02.819 INFO:teuthology.orchestra.run.smithi083.stdout: "max_avail": 367085518848 2024-03-06T07:26:02.819 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:26:02.819 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:26:02.819 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-03-06T07:26:02.819 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:26:02.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.816+0000 7f7d525f7640 1 -- 172.21.15.83:0/3079008417 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7d3806b500 msgr2=0x7f7d3806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.816+0000 7f7d525f7640 1 --2- 172.21.15.83:0/3079008417 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7d3806b500 0x7f7d3806d9c0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c000f60 tx=0x7f7d3c007c00 comp rx=0 tx=0).stop 2024-03-06T07:26:02.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.816+0000 7f7d525f7640 1 -- 172.21.15.83:0/3079008417 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d4c162970 msgr2=0x7f7d4c0c8240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.816+0000 7f7d525f7640 1 --2- 172.21.15.83:0/3079008417 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d4c162970 0x7f7d4c0c8240 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f7d44007e90 tx=0x7f7d4400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.816+0000 7f7d525f7640 1 -- 172.21.15.83:0/3079008417 shutdown_connections 2024-03-06T07:26:02.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.816+0000 7f7d525f7640 1 --2- 172.21.15.83:0/3079008417 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7d3806b500 0x7f7d3806d9c0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.816+0000 7f7d525f7640 1 --2- 172.21.15.83:0/3079008417 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d4c162970 0x7f7d4c0c8240 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.816+0000 7f7d525f7640 1 --2- 172.21.15.83:0/3079008417 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d4c161cf0 0x7f7d4c0c7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.816+0000 7f7d525f7640 1 --2- 172.21.15.83:0/3079008417 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d4c0b4460 0x7f7d4c155ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.816+0000 7f7d525f7640 1 -- 172.21.15.83:0/3079008417 >> 172.21.15.83:0/3079008417 conn(0x7f7d4c0d3ce0 msgr2=0x7f7d4c0d0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:02.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.816+0000 7f7d525f7640 1 -- 172.21.15.83:0/3079008417 shutdown_connections 2024-03-06T07:26:02.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:02.816+0000 7f7d525f7640 1 -- 172.21.15.83:0/3079008417 wait complete. 2024-03-06T07:26:02.831 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-03-06T07:26:02.831 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:26:02.831 DEBUG:teuthology.orchestra.run.smithi083:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:02.874 DEBUG:teuthology.orchestra.run.smithi183:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:02.919 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:26:02.919 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:26:02.919 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:02.934 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:03.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.072+0000 7fa186cd4640 1 Processor -- start 2024-03-06T07:26:03.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.072+0000 7fa186cd4640 1 -- start start 2024-03-06T07:26:03.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.073+0000 7fa186cd4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa1800d22b0 0x7fa1800d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.073+0000 7fa186cd4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa1800b41f0 0x7fa1800b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.073+0000 7fa186cd4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1800b3430 con 0x7fa18015dfe0 2024-03-06T07:26:03.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.073+0000 7fa186cd4640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1800d2c50 con 0x7fa1800b41f0 2024-03-06T07:26:03.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.073+0000 7fa186cd4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1800b31e0 con 0x7fa1800d22b0 2024-03-06T07:26:03.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.073+0000 7fa185271640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa1800b41f0 0x7fa1800b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.073+0000 7fa185271640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa1800b41f0 0x7fa1800b4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55710/0 (socket says 172.21.15.83:55710) 2024-03-06T07:26:03.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.073+0000 7fa184a70640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fa18015dfe0 0x7fa1800cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:33108/0 (socket says 172.21.15.83:33108) 2024-03-06T07:26:03.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.073+0000 7fa185271640 1 -- 172.21.15.83:0/3276303762 learned_addr learned my addr 172.21.15.83:0/3276303762 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:03.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.074+0000 7fa17ffff640 1 --2- 172.21.15.83:0/3276303762 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa1800d22b0 0x7fa1800d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.074+0000 7fa185271640 1 -- 172.21.15.83:0/3276303762 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa1800d22b0 msgr2=0x7fa1800d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:03.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.074+0000 7fa185271640 1 --2- 172.21.15.83:0/3276303762 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa1800d22b0 0x7fa1800d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.074+0000 7fa185271640 1 -- 172.21.15.83:0/3276303762 >> v1:172.21.15.83:6789/0 conn(0x7fa18015dfe0 legacy=0x7fa1800cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.074+0000 7fa185271640 1 -- 172.21.15.83:0/3276303762 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1800b3710 con 0x7fa1800b41f0 2024-03-06T07:26:03.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.074+0000 7fa17f7fe640 1 -- 172.21.15.83:0/3276303762 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2505611299 0 0) 0x7fa1800b3430 con 0x7fa18015dfe0 2024-03-06T07:26:03.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.074+0000 7fa185271640 1 --2- 172.21.15.83:0/3276303762 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1800b41f0 0x7fa1800b4650 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fa178009e50 tx=0x7fa17802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0f616f974e7c7dd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.075+0000 7fa17f7fe640 1 -- 172.21.15.83:0/3276303762 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa17802cb00 con 0x7fa1800b41f0 2024-03-06T07:26:03.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.075+0000 7fa17f7fe640 1 -- 172.21.15.83:0/3276303762 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa17802cca0 con 0x7fa1800b41f0 2024-03-06T07:26:03.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.075+0000 7fa17f7fe640 1 -- 172.21.15.83:0/3276303762 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa178034610 con 0x7fa1800b41f0 2024-03-06T07:26:03.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.075+0000 7fa186cd4640 1 -- 172.21.15.83:0/3276303762 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1800b41f0 msgr2=0x7fa1800b4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.075+0000 7fa186cd4640 1 --2- 172.21.15.83:0/3276303762 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1800b41f0 0x7fa1800b4650 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fa178009e50 tx=0x7fa17802bf60 comp rx=0 tx=0).stop 2024-03-06T07:26:03.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.075+0000 7fa186cd4640 1 -- 172.21.15.83:0/3276303762 shutdown_connections 2024-03-06T07:26:03.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.075+0000 7fa186cd4640 1 --2- 172.21.15.83:0/3276303762 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1800b41f0 0x7fa1800b4650 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.075+0000 7fa186cd4640 1 --2- 172.21.15.83:0/3276303762 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa1800d22b0 0x7fa1800d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.075+0000 7fa186cd4640 1 -- 172.21.15.83:0/3276303762 >> 172.21.15.83:0/3276303762 conn(0x7fa1800d3cd0 msgr2=0x7fa1800d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:03.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.075+0000 7fa186cd4640 1 -- 172.21.15.83:0/3276303762 shutdown_connections 2024-03-06T07:26:03.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.075+0000 7fa186cd4640 1 -- 172.21.15.83:0/3276303762 wait complete. 2024-03-06T07:26:03.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.077+0000 7fa186cd4640 1 Processor -- start 2024-03-06T07:26:03.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.077+0000 7fa186cd4640 1 -- start start 2024-03-06T07:26:03.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.078+0000 7fa186cd4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1800b41f0 0x7fa1800cbe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.078+0000 7fa186cd4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa1800d22b0 0x7fa180147e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.078+0000 7fa186cd4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa18015dfe0 0x7fa1800cc900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.078+0000 7fa186cd4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1800cfa00 con 0x7fa1800b41f0 2024-03-06T07:26:03.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.078+0000 7fa186cd4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1800b3d60 con 0x7fa1800d22b0 2024-03-06T07:26:03.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.078+0000 7fa186cd4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1800d1c70 con 0x7fa18015dfe0 2024-03-06T07:26:03.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.078+0000 7fa184a70640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1800b41f0 0x7fa1800cbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.078+0000 7fa184a70640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1800b41f0 0x7fa1800cbe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55722/0 (socket says 172.21.15.83:55722) 2024-03-06T07:26:03.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.078+0000 7fa184a70640 1 -- 172.21.15.83:0/1347643905 learned_addr learned my addr 172.21.15.83:0/1347643905 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:03.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.078+0000 7fa17ffff640 1 --2- 172.21.15.83:0/1347643905 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa1800d22b0 0x7fa180147e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.078+0000 7fa185271640 1 --2- 172.21.15.83:0/1347643905 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa18015dfe0 0x7fa1800cc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.078+0000 7fa184a70640 1 -- 172.21.15.83:0/1347643905 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa18015dfe0 msgr2=0x7fa1800cc900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.078+0000 7fa184a70640 1 --2- 172.21.15.83:0/1347643905 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa18015dfe0 0x7fa1800cc900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.078+0000 7fa184a70640 1 -- 172.21.15.83:0/1347643905 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa1800d22b0 msgr2=0x7fa180147e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.078+0000 7fa184a70640 1 --2- 172.21.15.83:0/1347643905 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa1800d22b0 0x7fa180147e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.078+0000 7fa184a70640 1 -- 172.21.15.83:0/1347643905 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1800b6940 con 0x7fa1800b41f0 2024-03-06T07:26:03.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.078+0000 7fa17ffff640 1 --2- 172.21.15.83:0/1347643905 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa1800d22b0 0x7fa180147e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:03.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.078+0000 7fa184a70640 1 --2- 172.21.15.83:0/1347643905 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1800b41f0 0x7fa1800cbe40 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fa17400c500 tx=0x7fa17400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.078+0000 7fa17d7fa640 1 -- 172.21.15.83:0/1347643905 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa174018070 con 0x7fa1800b41f0 2024-03-06T07:26:03.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.078+0000 7fa17d7fa640 1 -- 172.21.15.83:0/1347643905 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa174007580 con 0x7fa1800b41f0 2024-03-06T07:26:03.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.078+0000 7fa186cd4640 1 -- 172.21.15.83:0/1347643905 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa18014dc20 con 0x7fa1800b41f0 2024-03-06T07:26:03.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.078+0000 7fa17d7fa640 1 -- 172.21.15.83:0/1347643905 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa174004a20 con 0x7fa1800b41f0 2024-03-06T07:26:03.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.078+0000 7fa186cd4640 1 -- 172.21.15.83:0/1347643905 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1800b6e90 con 0x7fa1800b41f0 2024-03-06T07:26:03.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.080+0000 7fa186cd4640 1 -- 172.21.15.83:0/1347643905 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa18004f330 con 0x7fa1800b41f0 2024-03-06T07:26:03.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.080+0000 7fa17d7fa640 1 -- 172.21.15.83:0/1347643905 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa17401b070 con 0x7fa1800b41f0 2024-03-06T07:26:03.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.080+0000 7fa17d7fa640 1 --2- 172.21.15.83:0/1347643905 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa16406b4d0 0x7fa16406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.081+0000 7fa17d7fa640 1 -- 172.21.15.83:0/1347643905 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa1740893f0 con 0x7fa1800b41f0 2024-03-06T07:26:03.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.081+0000 7fa17ffff640 1 --2- 172.21.15.83:0/1347643905 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa16406b4d0 0x7fa16406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.081+0000 7fa17ffff640 1 --2- 172.21.15.83:0/1347643905 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa16406b4d0 0x7fa16406d990 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fa1800ce0d0 tx=0x7fa16c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.089+0000 7fa17d7fa640 1 -- 172.21.15.83:0/1347643905 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa174057cb0 con 0x7fa1800b41f0 2024-03-06T07:26:03.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.294+0000 7fa186cd4640 1 -- 172.21.15.83:0/1347643905 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1800d2890 con 0x7fa1800b41f0 2024-03-06T07:26:03.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.295+0000 7fa17d7fa640 1 -- 172.21.15.83:0/1347643905 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16672 (secure 0 0 0) 0x7fa17405bd50 con 0x7fa1800b41f0 2024-03-06T07:26:03.297 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:03.297 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:25:56.279928+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:52.856525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:53.598434+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:03.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.297+0000 7fa186cd4640 1 -- 172.21.15.83:0/1347643905 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa16406b4d0 msgr2=0x7fa16406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.297+0000 7fa186cd4640 1 --2- 172.21.15.83:0/1347643905 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa16406b4d0 0x7fa16406d990 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fa1800ce0d0 tx=0x7fa16c008040 comp rx=0 tx=0).stop 2024-03-06T07:26:03.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.297+0000 7fa186cd4640 1 -- 172.21.15.83:0/1347643905 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1800b41f0 msgr2=0x7fa1800cbe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.297+0000 7fa186cd4640 1 --2- 172.21.15.83:0/1347643905 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1800b41f0 0x7fa1800cbe40 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fa17400c500 tx=0x7fa17400c9c0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.298+0000 7fa186cd4640 1 -- 172.21.15.83:0/1347643905 shutdown_connections 2024-03-06T07:26:03.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.298+0000 7fa186cd4640 1 --2- 172.21.15.83:0/1347643905 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa16406b4d0 0x7fa16406d990 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.298+0000 7fa186cd4640 1 --2- 172.21.15.83:0/1347643905 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa18015dfe0 0x7fa1800cc900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.298+0000 7fa186cd4640 1 --2- 172.21.15.83:0/1347643905 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa1800d22b0 0x7fa180147e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.298+0000 7fa186cd4640 1 --2- 172.21.15.83:0/1347643905 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1800b41f0 0x7fa1800cbe40 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.298+0000 7fa186cd4640 1 -- 172.21.15.83:0/1347643905 >> 172.21.15.83:0/1347643905 conn(0x7fa1800d3cd0 msgr2=0x7fa1800cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:03.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.298+0000 7fa186cd4640 1 -- 172.21.15.83:0/1347643905 shutdown_connections 2024-03-06T07:26:03.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.298+0000 7fa186cd4640 1 -- 172.21.15.83:0/1347643905 wait complete. 2024-03-06T07:26:03.310 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:25:52.856525+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:25:53.598434+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:26:03.311 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:26:03.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.414+0000 7fedf906f640 1 Processor -- start 2024-03-06T07:26:03.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.414+0000 7fedf906f640 1 -- start start 2024-03-06T07:26:03.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.415+0000 7fedf906f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fedf40d4860 0x7fedf40d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.415+0000 7fedf906f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedf41618b0 con 0x7fedf41614d0 2024-03-06T07:26:03.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.415+0000 7fedf906f640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedf40cf400 con 0x7fedf40b0e30 2024-03-06T07:26:03.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.415+0000 7fedf906f640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedf40d4cf0 con 0x7fedf40d4860 2024-03-06T07:26:03.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.415+0000 7fedf259c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fedf40d4860 0x7fedf40d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.415+0000 7fedf2d9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fedf41614d0 0x7fedf40d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:33116/0 (socket says 172.21.15.83:33116) 2024-03-06T07:26:03.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.415+0000 7fedf2d9d640 1 -- 172.21.15.83:0/275543591 learned_addr learned my addr 172.21.15.83:0/275543591 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:03.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.415+0000 7fedf259c640 1 -- 172.21.15.83:0/275543591 >> v1:172.21.15.183:6789/0 conn(0x7fedf40b0e30 legacy=0x7fedf40b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:03.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.415+0000 7fedf259c640 1 -- 172.21.15.83:0/275543591 >> v1:172.21.15.83:6789/0 conn(0x7fedf41614d0 legacy=0x7fedf40d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.415+0000 7fedf259c640 1 -- 172.21.15.83:0/275543591 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf40cf6a0 con 0x7fedf40d4860 2024-03-06T07:26:03.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.415+0000 7fedf1d9b640 1 -- 172.21.15.83:0/275543591 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2541084058 0 0) 0x7fedf41618b0 con 0x7fedf41614d0 2024-03-06T07:26:03.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.416+0000 7fedf259c640 1 --2- 172.21.15.83:0/275543591 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedf40d4860 0x7fedf40d2020 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fede80097a0 tx=0x7fede802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23236777462e74bd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.416+0000 7fedf1d9b640 1 -- 172.21.15.83:0/275543591 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede8035020 con 0x7fedf40d4860 2024-03-06T07:26:03.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.416+0000 7fedf1d9b640 1 -- 172.21.15.83:0/275543591 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fede8031830 con 0x7fedf40d4860 2024-03-06T07:26:03.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.416+0000 7fedf1d9b640 1 -- 172.21.15.83:0/275543591 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede8031c00 con 0x7fedf40d4860 2024-03-06T07:26:03.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.416+0000 7fedf906f640 1 -- 172.21.15.83:0/275543591 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedf40d4860 msgr2=0x7fedf40d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.416+0000 7fedf906f640 1 --2- 172.21.15.83:0/275543591 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedf40d4860 0x7fedf40d2020 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fede80097a0 tx=0x7fede802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.416+0000 7fedf906f640 1 -- 172.21.15.83:0/275543591 shutdown_connections 2024-03-06T07:26:03.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.416+0000 7fedf906f640 1 --2- 172.21.15.83:0/275543591 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedf40d4860 0x7fedf40d2020 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.416+0000 7fedf906f640 1 -- 172.21.15.83:0/275543591 >> 172.21.15.83:0/275543591 conn(0x7fedf40d3d40 msgr2=0x7fedf40d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:03.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.416+0000 7fedf906f640 1 -- 172.21.15.83:0/275543591 shutdown_connections 2024-03-06T07:26:03.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.416+0000 7fedf906f640 1 -- 172.21.15.83:0/275543591 wait complete. 2024-03-06T07:26:03.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.418+0000 7fedf906f640 1 Processor -- start 2024-03-06T07:26:03.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.418+0000 7fedf906f640 1 -- start start 2024-03-06T07:26:03.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.419+0000 7fedf906f640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fedf40b0e30 0x7fedf4157410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.419+0000 7fedf906f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedf40d4860 0x7fedf41f3b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.419+0000 7fedf906f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fedf41614d0 0x7fedf40cd180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.419+0000 7fedf906f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf40b0910 con 0x7fedf40d4860 2024-03-06T07:26:03.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.419+0000 7fedf906f640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf40cfc60 con 0x7fedf40b0e30 2024-03-06T07:26:03.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.419+0000 7fedf906f640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedf40d2560 con 0x7fedf41614d0 2024-03-06T07:26:03.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.419+0000 7fedf259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedf40d4860 0x7fedf41f3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.419+0000 7fedf259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedf40d4860 0x7fedf41f3b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55746/0 (socket says 172.21.15.83:55746) 2024-03-06T07:26:03.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.419+0000 7fedf259c640 1 -- 172.21.15.83:0/654890890 learned_addr learned my addr 172.21.15.83:0/654890890 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:03.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.419+0000 7fedf2d9d640 1 --2- 172.21.15.83:0/654890890 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fedf40b0e30 0x7fedf4157410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.419+0000 7fedf359e640 1 --2- 172.21.15.83:0/654890890 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fedf41614d0 0x7fedf40cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.419+0000 7fedf259c640 1 -- 172.21.15.83:0/654890890 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fedf41614d0 msgr2=0x7fedf40cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.419+0000 7fedf259c640 1 --2- 172.21.15.83:0/654890890 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fedf41614d0 0x7fedf40cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.419+0000 7fedf259c640 1 -- 172.21.15.83:0/654890890 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fedf40b0e30 msgr2=0x7fedf4157410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.419+0000 7fedf259c640 1 --2- 172.21.15.83:0/654890890 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fedf40b0e30 0x7fedf4157410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.419+0000 7fedf259c640 1 -- 172.21.15.83:0/654890890 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf41f4580 con 0x7fedf40d4860 2024-03-06T07:26:03.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.419+0000 7fedf259c640 1 --2- 172.21.15.83:0/654890890 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedf40d4860 0x7fedf41f3b20 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fede80098c0 tx=0x7fede80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.419+0000 7feddb7fe640 1 -- 172.21.15.83:0/654890890 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede8004230 con 0x7fedf40d4860 2024-03-06T07:26:03.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.419+0000 7feddb7fe640 1 -- 172.21.15.83:0/654890890 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fede80043d0 con 0x7fedf40d4860 2024-03-06T07:26:03.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.419+0000 7feddb7fe640 1 -- 172.21.15.83:0/654890890 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede803cad0 con 0x7fedf40d4860 2024-03-06T07:26:03.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.419+0000 7fedf906f640 1 -- 172.21.15.83:0/654890890 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedf41f4860 con 0x7fedf40d4860 2024-03-06T07:26:03.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.419+0000 7fedf906f640 1 -- 172.21.15.83:0/654890890 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedf41f4d10 con 0x7fedf40d4860 2024-03-06T07:26:03.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.420+0000 7fedf906f640 1 -- 172.21.15.83:0/654890890 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedf4068ff0 con 0x7fedf40d4860 2024-03-06T07:26:03.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.421+0000 7feddb7fe640 1 -- 172.21.15.83:0/654890890 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fede8038020 con 0x7fedf40d4860 2024-03-06T07:26:03.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.421+0000 7feddb7fe640 1 --2- 172.21.15.83:0/654890890 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fede006b500 0x7fede006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.421+0000 7feddb7fe640 1 -- 172.21.15.83:0/654890890 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fede80ac3c0 con 0x7fedf40d4860 2024-03-06T07:26:03.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.421+0000 7fedf2d9d640 1 --2- 172.21.15.83:0/654890890 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fede006b500 0x7fede006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.422+0000 7fedf2d9d640 1 --2- 172.21.15.83:0/654890890 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fede006b500 0x7fede006d9c0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fede4000f60 tx=0x7fede4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.426+0000 7feddb7fe640 1 -- 172.21.15.83:0/654890890 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fede8041070 con 0x7fedf40d4860 2024-03-06T07:26:03.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.610+0000 7fedf906f640 1 -- 172.21.15.83:0/654890890 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fedf40747a0 con 0x7fedf40d4860 2024-03-06T07:26:03.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.611+0000 7feddb7fe640 1 -- 172.21.15.83:0/654890890 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7fede807b3f0 con 0x7fedf40d4860 2024-03-06T07:26:03.613 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:26:03.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.613+0000 7fedf906f640 1 -- 172.21.15.83:0/654890890 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fede006b500 msgr2=0x7fede006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.613+0000 7fedf906f640 1 --2- 172.21.15.83:0/654890890 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fede006b500 0x7fede006d9c0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fede4000f60 tx=0x7fede4009040 comp rx=0 tx=0).stop 2024-03-06T07:26:03.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.613+0000 7fedf906f640 1 -- 172.21.15.83:0/654890890 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedf40d4860 msgr2=0x7fedf41f3b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.613+0000 7fedf906f640 1 --2- 172.21.15.83:0/654890890 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedf40d4860 0x7fedf41f3b20 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fede80098c0 tx=0x7fede80046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.613+0000 7fedf906f640 1 -- 172.21.15.83:0/654890890 shutdown_connections 2024-03-06T07:26:03.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.613+0000 7fedf906f640 1 --2- 172.21.15.83:0/654890890 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fede006b500 0x7fede006d9c0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.613+0000 7fedf906f640 1 --2- 172.21.15.83:0/654890890 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fedf41614d0 0x7fedf40cd180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.613+0000 7fedf906f640 1 --2- 172.21.15.83:0/654890890 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedf40d4860 0x7fedf41f3b20 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.613+0000 7fedf906f640 1 --2- 172.21.15.83:0/654890890 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fedf40b0e30 0x7fedf4157410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.613+0000 7fedf906f640 1 -- 172.21.15.83:0/654890890 >> 172.21.15.83:0/654890890 conn(0x7fedf40d3d40 msgr2=0x7fedf40ce9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:03.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.613+0000 7fedf906f640 1 -- 172.21.15.83:0/654890890 shutdown_connections 2024-03-06T07:26:03.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.613+0000 7fedf906f640 1 -- 172.21.15.83:0/654890890 wait complete. 2024-03-06T07:26:03.625 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:26:03.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.729+0000 7ff2125bf640 1 Processor -- start 2024-03-06T07:26:03.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.729+0000 7ff2125bf640 1 -- start start 2024-03-06T07:26:03.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.730+0000 7ff2125bf640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff20c1614d0 0x7ff20c0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.730+0000 7ff2125bf640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff20c0d1910 0x7ff20c0d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.730+0000 7ff2125bf640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff20c0d2740 con 0x7ff20c0d22b0 2024-03-06T07:26:03.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.730+0000 7ff2125bf640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff20c1618b0 con 0x7ff20c0d1910 2024-03-06T07:26:03.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.730+0000 7ff2125bf640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff20c0cf400 con 0x7ff20c1614d0 2024-03-06T07:26:03.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.731+0000 7ff20b7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff20c0d1910 0x7ff20c0d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.731+0000 7ff20b7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff20c0d1910 0x7ff20c0d1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55760/0 (socket says 172.21.15.83:55760) 2024-03-06T07:26:03.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.731+0000 7ff20b7fe640 1 -- 172.21.15.83:0/2359732297 learned_addr learned my addr 172.21.15.83:0/2359732297 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:03.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.731+0000 7ff20bfff640 1 --2- 172.21.15.83:0/2359732297 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff20c1614d0 0x7ff20c0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.731+0000 7ff20b7fe640 1 -- 172.21.15.83:0/2359732297 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff20c1614d0 msgr2=0x7ff20c0d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:03.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.731+0000 7ff20b7fe640 1 --2- 172.21.15.83:0/2359732297 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff20c1614d0 0x7ff20c0d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.731+0000 7ff20b7fe640 1 -- 172.21.15.83:0/2359732297 >> v1:172.21.15.183:6789/0 conn(0x7ff20c0d22b0 legacy=0x7ff20c0d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.731+0000 7ff20b7fe640 1 -- 172.21.15.83:0/2359732297 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff20c0cf6a0 con 0x7ff20c0d1910 2024-03-06T07:26:03.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.732+0000 7ff20b7fe640 1 --2- 172.21.15.83:0/2359732297 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff20c0d1910 0x7ff20c0d1d70 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc009870 tx=0x7ff1fc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83beeb1a47809632 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.732+0000 7ff20affd640 1 -- 172.21.15.83:0/2359732297 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc034070 con 0x7ff20c0d1910 2024-03-06T07:26:03.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.732+0000 7ff20affd640 1 -- 172.21.15.83:0/2359732297 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1fc030840 con 0x7ff20c0d1910 2024-03-06T07:26:03.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.732+0000 7ff20affd640 1 -- 172.21.15.83:0/2359732297 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc030c10 con 0x7ff20c0d1910 2024-03-06T07:26:03.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.732+0000 7ff2125bf640 1 -- 172.21.15.83:0/2359732297 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff20c0d1910 msgr2=0x7ff20c0d1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.732+0000 7ff2125bf640 1 --2- 172.21.15.83:0/2359732297 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff20c0d1910 0x7ff20c0d1d70 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc009870 tx=0x7ff1fc02b980 comp rx=0 tx=0).stop 2024-03-06T07:26:03.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.732+0000 7ff2125bf640 1 -- 172.21.15.83:0/2359732297 shutdown_connections 2024-03-06T07:26:03.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.732+0000 7ff2125bf640 1 --2- 172.21.15.83:0/2359732297 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff20c0d1910 0x7ff20c0d1d70 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.732+0000 7ff2125bf640 1 --2- 172.21.15.83:0/2359732297 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff20c1614d0 0x7ff20c0d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.732+0000 7ff2125bf640 1 -- 172.21.15.83:0/2359732297 >> 172.21.15.83:0/2359732297 conn(0x7ff20c0d3d40 msgr2=0x7ff20c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:03.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.732+0000 7ff2125bf640 1 -- 172.21.15.83:0/2359732297 shutdown_connections 2024-03-06T07:26:03.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.732+0000 7ff2125bf640 1 -- 172.21.15.83:0/2359732297 wait complete. 2024-03-06T07:26:03.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.733+0000 7ff2125bf640 1 Processor -- start 2024-03-06T07:26:03.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.734+0000 7ff2125bf640 1 -- start start 2024-03-06T07:26:03.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.734+0000 7ff2125bf640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff20c0d1910 0x7ff20c15c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.734+0000 7ff2125bf640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff20c0d22b0 0x7ff20c0cd420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.734+0000 7ff2125bf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff20c1614d0 0x7ff20c1602b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.734+0000 7ff2125bf640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff20c0b0910 con 0x7ff20c1614d0 2024-03-06T07:26:03.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.734+0000 7ff2125bf640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff20c0cfc60 con 0x7ff20c0d22b0 2024-03-06T07:26:03.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.734+0000 7ff2125bf640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff20c0b1e40 con 0x7ff20c0d1910 2024-03-06T07:26:03.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.735+0000 7ff210b5c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff20c1614d0 0x7ff20c1602b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.735+0000 7ff210b5c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff20c1614d0 0x7ff20c1602b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55774/0 (socket says 172.21.15.83:55774) 2024-03-06T07:26:03.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.735+0000 7ff210b5c640 1 -- 172.21.15.83:0/123480742 learned_addr learned my addr 172.21.15.83:0/123480742 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:03.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.735+0000 7ff20bfff640 1 --2- 172.21.15.83:0/123480742 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff20c0d1910 0x7ff20c15c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.735+0000 7ff20b7fe640 1 --2- 172.21.15.83:0/123480742 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff20c0d22b0 0x7ff20c0cd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.735+0000 7ff210b5c640 1 -- 172.21.15.83:0/123480742 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff20c0d1910 msgr2=0x7ff20c15c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.735+0000 7ff210b5c640 1 --2- 172.21.15.83:0/123480742 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff20c0d1910 0x7ff20c15c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.735+0000 7ff210b5c640 1 -- 172.21.15.83:0/123480742 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff20c0d22b0 msgr2=0x7ff20c0cd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.735+0000 7ff210b5c640 1 --2- 172.21.15.83:0/123480742 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff20c0d22b0 0x7ff20c0cd420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.735+0000 7ff210b5c640 1 -- 172.21.15.83:0/123480742 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff20c0b7b30 con 0x7ff20c1614d0 2024-03-06T07:26:03.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.735+0000 7ff210b5c640 1 --2- 172.21.15.83:0/123480742 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff20c1614d0 0x7ff20c1602b0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7ff20400c7d0 tx=0x7ff20400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.735+0000 7ff208ff9640 1 -- 172.21.15.83:0/123480742 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff204003b70 con 0x7ff20c1614d0 2024-03-06T07:26:03.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.735+0000 7ff208ff9640 1 -- 172.21.15.83:0/123480742 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff204003d30 con 0x7ff20c1614d0 2024-03-06T07:26:03.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.735+0000 7ff208ff9640 1 -- 172.21.15.83:0/123480742 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20400aa90 con 0x7ff20c1614d0 2024-03-06T07:26:03.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.735+0000 7ff2125bf640 1 -- 172.21.15.83:0/123480742 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff20c15c880 con 0x7ff20c1614d0 2024-03-06T07:26:03.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.735+0000 7ff2125bf640 1 -- 172.21.15.83:0/123480742 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff20c0b80b0 con 0x7ff20c1614d0 2024-03-06T07:26:03.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.737+0000 7ff2125bf640 1 -- 172.21.15.83:0/123480742 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff20c068ff0 con 0x7ff20c1614d0 2024-03-06T07:26:03.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.737+0000 7ff208ff9640 1 -- 172.21.15.83:0/123480742 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff20400ac30 con 0x7ff20c1614d0 2024-03-06T07:26:03.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.737+0000 7ff208ff9640 1 --2- 172.21.15.83:0/123480742 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff1ec06b4d0 0x7ff1ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.737+0000 7ff20bfff640 1 --2- 172.21.15.83:0/123480742 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff1ec06b4d0 0x7ff1ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.737+0000 7ff208ff9640 1 -- 172.21.15.83:0/123480742 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff204089f40 con 0x7ff20c1614d0 2024-03-06T07:26:03.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.738+0000 7ff20bfff640 1 --2- 172.21.15.83:0/123480742 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff1ec06b4d0 0x7ff1ec06d990 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7ff20c0ca4a0 tx=0x7ff1f4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.742+0000 7ff208ff9640 1 -- 172.21.15.83:0/123480742 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff20c0d2740 con 0x7ff20c1614d0 2024-03-06T07:26:03.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.918+0000 7ff2125bf640 1 -- 172.21.15.83:0/123480742 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff20c0c7410 con 0x7ff20c1614d0 2024-03-06T07:26:03.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.918+0000 7ff208ff9640 1 -- 172.21.15.83:0/123480742 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v82) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff204058800 con 0x7ff20c1614d0 2024-03-06T07:26:03.920 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-03-06T07:26:03.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.920+0000 7ff2125bf640 1 -- 172.21.15.83:0/123480742 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff1ec06b4d0 msgr2=0x7ff1ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.920+0000 7ff2125bf640 1 --2- 172.21.15.83:0/123480742 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff1ec06b4d0 0x7ff1ec06d990 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7ff20c0ca4a0 tx=0x7ff1f4008040 comp rx=0 tx=0).stop 2024-03-06T07:26:03.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.920+0000 7ff2125bf640 1 -- 172.21.15.83:0/123480742 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff20c1614d0 msgr2=0x7ff20c1602b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.920+0000 7ff2125bf640 1 --2- 172.21.15.83:0/123480742 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff20c1614d0 0x7ff20c1602b0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7ff20400c7d0 tx=0x7ff20400cc90 comp rx=0 tx=0).stop 2024-03-06T07:26:03.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.920+0000 7ff2125bf640 1 -- 172.21.15.83:0/123480742 shutdown_connections 2024-03-06T07:26:03.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.920+0000 7ff2125bf640 1 --2- 172.21.15.83:0/123480742 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff1ec06b4d0 0x7ff1ec06d990 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.920+0000 7ff2125bf640 1 --2- 172.21.15.83:0/123480742 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff20c1614d0 0x7ff20c1602b0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.920+0000 7ff2125bf640 1 --2- 172.21.15.83:0/123480742 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff20c0d22b0 0x7ff20c0cd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.920+0000 7ff2125bf640 1 --2- 172.21.15.83:0/123480742 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff20c0d1910 0x7ff20c15c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.920+0000 7ff2125bf640 1 -- 172.21.15.83:0/123480742 >> 172.21.15.83:0/123480742 conn(0x7ff20c0d3d40 msgr2=0x7ff20c15fad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:03.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.920+0000 7ff2125bf640 1 -- 172.21.15.83:0/123480742 shutdown_connections 2024-03-06T07:26:03.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:03.920+0000 7ff2125bf640 1 -- 172.21.15.83:0/123480742 wait complete. 2024-03-06T07:26:03.933 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:26:04.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.034+0000 7f1594888640 1 Processor -- start 2024-03-06T07:26:04.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.034+0000 7f1594888640 1 -- start start 2024-03-06T07:26:04.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.035+0000 7f1594888640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f15901614d0 0x7f15900d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.035+0000 7f1594888640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f15900d1910 0x7f15900d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.035+0000 7f1594888640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f15900d22b0 0x7f15900d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.035+0000 7f1594888640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15900d2740 con 0x7f15901614d0 2024-03-06T07:26:04.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.035+0000 7f1594888640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15901618b0 con 0x7f15900d22b0 2024-03-06T07:26:04.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.035+0000 7f1594888640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15900cf400 con 0x7f15900d1910 2024-03-06T07:26:04.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.035+0000 7f158e59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f15901614d0 0x7f15900d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.035+0000 7f158e59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f15901614d0 0x7f15900d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55780/0 (socket says 172.21.15.83:55780) 2024-03-06T07:26:04.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.035+0000 7f158e59c640 1 -- 172.21.15.83:0/3485914685 learned_addr learned my addr 172.21.15.83:0/3485914685 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:04.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.036+0000 7f158dd9b640 1 --2- 172.21.15.83:0/3485914685 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f15900d1910 0x7f15900d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.036+0000 7f158ed9d640 1 --2- 172.21.15.83:0/3485914685 >> v2:172.21.15.183:3300/0 conn(0x7f15900d22b0 0x7f15900d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.036+0000 7f158e59c640 1 -- 172.21.15.83:0/3485914685 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f15900d1910 msgr2=0x7f15900d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:04.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.036+0000 7f158e59c640 1 --2- 172.21.15.83:0/3485914685 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f15900d1910 0x7f15900d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.036+0000 7f158e59c640 1 -- 172.21.15.83:0/3485914685 >> v2:172.21.15.183:3300/0 conn(0x7f15900d22b0 msgr2=0x7f15900d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:04.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.036+0000 7f158e59c640 1 --2- 172.21.15.83:0/3485914685 >> v2:172.21.15.183:3300/0 conn(0x7f15900d22b0 0x7f15900d06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.036+0000 7f158e59c640 1 -- 172.21.15.83:0/3485914685 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15900cf6a0 con 0x7f15901614d0 2024-03-06T07:26:04.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.036+0000 7f158e59c640 1 --2- 172.21.15.83:0/3485914685 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15901614d0 0x7f15900d4860 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f15900b57d0 tx=0x7f158002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63e90e6d23c1e2f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.037+0000 7f158d59a640 1 -- 172.21.15.83:0/3485914685 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f158002ca70 con 0x7f15901614d0 2024-03-06T07:26:04.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.037+0000 7f158d59a640 1 -- 172.21.15.83:0/3485914685 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f158002cc10 con 0x7f15901614d0 2024-03-06T07:26:04.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.037+0000 7f158d59a640 1 -- 172.21.15.83:0/3485914685 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15800343f0 con 0x7f15901614d0 2024-03-06T07:26:04.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.037+0000 7f1594888640 1 -- 172.21.15.83:0/3485914685 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15901614d0 msgr2=0x7f15900d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.037+0000 7f1594888640 1 --2- 172.21.15.83:0/3485914685 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15901614d0 0x7f15900d4860 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f15900b57d0 tx=0x7f158002bed0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.037+0000 7f1594888640 1 -- 172.21.15.83:0/3485914685 shutdown_connections 2024-03-06T07:26:04.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.037+0000 7f1594888640 1 --2- 172.21.15.83:0/3485914685 >> v2:172.21.15.183:3300/0 conn(0x7f15900d22b0 0x7f15900d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.037+0000 7f1594888640 1 --2- 172.21.15.83:0/3485914685 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f15900d1910 0x7f15900d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.037+0000 7f1594888640 1 --2- 172.21.15.83:0/3485914685 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15901614d0 0x7f15900d4860 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.037+0000 7f1594888640 1 -- 172.21.15.83:0/3485914685 >> 172.21.15.83:0/3485914685 conn(0x7f15900d3d40 msgr2=0x7f15900d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:04.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.037+0000 7f1594888640 1 -- 172.21.15.83:0/3485914685 shutdown_connections 2024-03-06T07:26:04.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.037+0000 7f1594888640 1 -- 172.21.15.83:0/3485914685 wait complete. 2024-03-06T07:26:04.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.039+0000 7f1594888640 1 Processor -- start 2024-03-06T07:26:04.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.039+0000 7f1594888640 1 -- start start 2024-03-06T07:26:04.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.039+0000 7f1594888640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f15900d1910 0x7f159015cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.039+0000 7f1594888640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f15900d22b0 0x7f15900c8840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.039+0000 7f1594888640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15901614d0 0x7f159006b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.039+0000 7f1594888640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15900b0910 con 0x7f15901614d0 2024-03-06T07:26:04.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.039+0000 7f1594888640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15900cfc60 con 0x7f15900d22b0 2024-03-06T07:26:04.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.039+0000 7f1594888640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15900d0be0 con 0x7f15900d1910 2024-03-06T07:26:04.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.039+0000 7f158ed9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15901614d0 0x7f159006b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.039+0000 7f158ed9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15901614d0 0x7f159006b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55784/0 (socket says 172.21.15.83:55784) 2024-03-06T07:26:04.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.039+0000 7f158ed9d640 1 -- 172.21.15.83:0/3610782167 learned_addr learned my addr 172.21.15.83:0/3610782167 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:04.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.040+0000 7f158e59c640 1 --2- 172.21.15.83:0/3610782167 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f15900d1910 0x7f159015cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.040+0000 7f158dd9b640 1 --2- 172.21.15.83:0/3610782167 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f15900d22b0 0x7f15900c8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.040+0000 7f158ed9d640 1 -- 172.21.15.83:0/3610782167 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f15900d1910 msgr2=0x7f159015cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.040+0000 7f158ed9d640 1 --2- 172.21.15.83:0/3610782167 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f15900d1910 0x7f159015cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.040+0000 7f158ed9d640 1 -- 172.21.15.83:0/3610782167 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f15900d22b0 msgr2=0x7f15900c8840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.040+0000 7f158ed9d640 1 --2- 172.21.15.83:0/3610782167 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f15900d22b0 0x7f15900c8840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.040+0000 7f158ed9d640 1 -- 172.21.15.83:0/3610782167 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15900c7a80 con 0x7f15901614d0 2024-03-06T07:26:04.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.040+0000 7f158ed9d640 1 --2- 172.21.15.83:0/3610782167 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15901614d0 0x7f159006b9d0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f158800d800 tx=0x7f158800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.040+0000 7f1586ffd640 1 -- 172.21.15.83:0/3610782167 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f158800b6c0 con 0x7f15901614d0 2024-03-06T07:26:04.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.040+0000 7f1586ffd640 1 -- 172.21.15.83:0/3610782167 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1588019070 con 0x7f15901614d0 2024-03-06T07:26:04.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.040+0000 7f1594888640 1 -- 172.21.15.83:0/3610782167 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15900c7d60 con 0x7f15901614d0 2024-03-06T07:26:04.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.040+0000 7f1586ffd640 1 -- 172.21.15.83:0/3610782167 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f158800bac0 con 0x7f15901614d0 2024-03-06T07:26:04.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.040+0000 7f1594888640 1 -- 172.21.15.83:0/3610782167 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15900c8320 con 0x7f15901614d0 2024-03-06T07:26:04.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.041+0000 7f1594888640 1 -- 172.21.15.83:0/3610782167 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15900c84f0 con 0x7f15901614d0 2024-03-06T07:26:04.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.041+0000 7f1586ffd640 1 -- 172.21.15.83:0/3610782167 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f158800bc60 con 0x7f15901614d0 2024-03-06T07:26:04.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.042+0000 7f1586ffd640 1 --2- 172.21.15.83:0/3610782167 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f157006b4d0 0x7f157006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.042+0000 7f1586ffd640 1 -- 172.21.15.83:0/3610782167 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f158808a4c0 con 0x7f15901614d0 2024-03-06T07:26:04.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.042+0000 7f158e59c640 1 --2- 172.21.15.83:0/3610782167 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f157006b4d0 0x7f157006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.042+0000 7f158e59c640 1 --2- 172.21.15.83:0/3610782167 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f157006b4d0 0x7f157006d990 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f1580035000 tx=0x7f1580030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.046+0000 7f1586ffd640 1 -- 172.21.15.83:0/3610782167 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1588058cf0 con 0x7f15901614d0 2024-03-06T07:26:04.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.223+0000 7f1594888640 1 -- 172.21.15.83:0/3610782167 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f15900cef10 con 0x7f15901614d0 2024-03-06T07:26:04.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.223+0000 7f1586ffd640 1 -- 172.21.15.83:0/3610782167 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v82) v1 ==== 100+0+11 (secure 0 0 0) 0x7f158805cd90 con 0x7f15901614d0 2024-03-06T07:26:04.226 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 64 2024-03-06T07:26:04.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.225+0000 7f1594888640 1 -- 172.21.15.83:0/3610782167 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f157006b4d0 msgr2=0x7f157006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.225+0000 7f1594888640 1 --2- 172.21.15.83:0/3610782167 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f157006b4d0 0x7f157006d990 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f1580035000 tx=0x7f1580030000 comp rx=0 tx=0).stop 2024-03-06T07:26:04.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.225+0000 7f1594888640 1 -- 172.21.15.83:0/3610782167 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15901614d0 msgr2=0x7f159006b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.225+0000 7f1594888640 1 --2- 172.21.15.83:0/3610782167 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15901614d0 0x7f159006b9d0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f158800d800 tx=0x7f158800dcc0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.225+0000 7f1594888640 1 -- 172.21.15.83:0/3610782167 shutdown_connections 2024-03-06T07:26:04.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.225+0000 7f1594888640 1 --2- 172.21.15.83:0/3610782167 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f157006b4d0 0x7f157006d990 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.225+0000 7f1594888640 1 --2- 172.21.15.83:0/3610782167 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f15901614d0 0x7f159006b9d0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.225+0000 7f1594888640 1 --2- 172.21.15.83:0/3610782167 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f15900d22b0 0x7f15900c8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.225+0000 7f1594888640 1 --2- 172.21.15.83:0/3610782167 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f15900d1910 0x7f159015cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.225+0000 7f1594888640 1 -- 172.21.15.83:0/3610782167 >> 172.21.15.83:0/3610782167 conn(0x7f15900d3d40 msgr2=0x7f159006cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:04.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.225+0000 7f1594888640 1 -- 172.21.15.83:0/3610782167 shutdown_connections 2024-03-06T07:26:04.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.225+0000 7f1594888640 1 -- 172.21.15.83:0/3610782167 wait complete. 2024-03-06T07:26:04.239 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:26:04.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.344+0000 7f89fcc9a640 1 Processor -- start 2024-03-06T07:26:04.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.344+0000 7f89fcc9a640 1 -- start start 2024-03-06T07:26:04.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.346+0000 7f89fcc9a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f89f80d02b0 0x7f89f806afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.346+0000 7f89fcc9a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f89f80b50d0 0x7f89f80ce5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.346+0000 7f89fcc9a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89f80b5c30 con 0x7f89f80670a0 2024-03-06T07:26:04.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.346+0000 7f89fcc9a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89f80cd940 con 0x7f89f80d02b0 2024-03-06T07:26:04.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.346+0000 7f89fcc9a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89f80b5f10 con 0x7f89f80b50d0 2024-03-06T07:26:04.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.346+0000 7f89f5d9b640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f89f80670a0 0x7f89f806b4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:33124/0 (socket says 172.21.15.83:33124) 2024-03-06T07:26:04.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.346+0000 7f89f5d9b640 1 -- 172.21.15.83:0/3508048547 learned_addr learned my addr 172.21.15.83:0/3508048547 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:04.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.346+0000 7f89f559a640 1 -- 172.21.15.83:0/3508048547 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4189080865 0 0) 0x7f89f80b5c30 con 0x7f89f80670a0 2024-03-06T07:26:04.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.346+0000 7f89f6d9d640 1 --2- 172.21.15.83:0/3508048547 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f89f80b50d0 0x7f89f80ce5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.346+0000 7f89f659c640 1 --2- 172.21.15.83:0/3508048547 >> v2:172.21.15.183:3300/0 conn(0x7f89f80d02b0 0x7f89f806afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.346+0000 7f89f559a640 1 -- 172.21.15.83:0/3508048547 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89e4003520 con 0x7f89f80670a0 2024-03-06T07:26:04.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.346+0000 7f89f559a640 1 -- 172.21.15.83:0/3508048547 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1824448582 0 0) 0x7f89e4003520 con 0x7f89f80670a0 2024-03-06T07:26:04.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.346+0000 7f89f559a640 1 -- 172.21.15.83:0/3508048547 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f89f80b50d0 msgr2=0x7f89f80ce5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:04.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.346+0000 7f89f559a640 1 --2- 172.21.15.83:0/3508048547 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f89f80b50d0 0x7f89f80ce5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.346+0000 7f89f559a640 1 -- 172.21.15.83:0/3508048547 >> v2:172.21.15.183:3300/0 conn(0x7f89f80d02b0 msgr2=0x7f89f806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:04.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.346+0000 7f89f559a640 1 --2- 172.21.15.83:0/3508048547 >> v2:172.21.15.183:3300/0 conn(0x7f89f80d02b0 0x7f89f806afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.346+0000 7f89f559a640 1 -- 172.21.15.83:0/3508048547 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f80cd5d0 con 0x7f89f80670a0 2024-03-06T07:26:04.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.346+0000 7f89f559a640 1 -- 172.21.15.83:0/3508048547 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f89e0001690 con 0x7f89f80670a0 2024-03-06T07:26:04.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.346+0000 7f89f559a640 1 -- 172.21.15.83:0/3508048547 >> v1:172.21.15.83:6789/0 conn(0x7f89f80670a0 legacy=0x7f89f806b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.346+0000 7f89f559a640 1 --2- 172.21.15.83:0/3508048547 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f89e40039b0 0x7f89e4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.346+0000 7f89f559a640 1 --2- 172.21.15.83:0/3508048547 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89e4004800 0x7f89e4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.346+0000 7f89f559a640 1 --2- 172.21.15.83:0/3508048547 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f89f80b50d0 0x7f89e40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.346+0000 7f89f559a640 1 -- 172.21.15.83:0/3508048547 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f80b5f10 con 0x7f89e4004800 2024-03-06T07:26:04.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.346+0000 7f89f559a640 1 -- 172.21.15.83:0/3508048547 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e400b9c0 con 0x7f89e40039b0 2024-03-06T07:26:04.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.346+0000 7f89f559a640 1 -- 172.21.15.83:0/3508048547 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89e400bbc0 con 0x7f89f80b50d0 2024-03-06T07:26:04.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.347+0000 7f89f5d9b640 1 --2- 172.21.15.83:0/3508048547 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89e4004800 0x7f89e4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.347+0000 7f89f659c640 1 --2- 172.21.15.83:0/3508048547 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f89e40039b0 0x7f89e4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.347+0000 7f89f6d9d640 1 --2- 172.21.15.83:0/3508048547 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f89f80b50d0 0x7f89e40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.347+0000 7f89f5d9b640 1 -- 172.21.15.83:0/3508048547 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f89f80b50d0 msgr2=0x7f89e40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:04.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.347+0000 7f89f5d9b640 1 --2- 172.21.15.83:0/3508048547 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f89f80b50d0 0x7f89e40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.347+0000 7f89f5d9b640 1 -- 172.21.15.83:0/3508048547 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f89e40039b0 msgr2=0x7f89e4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:04.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.347+0000 7f89f5d9b640 1 --2- 172.21.15.83:0/3508048547 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f89e40039b0 0x7f89e4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.347+0000 7f89f5d9b640 1 -- 172.21.15.83:0/3508048547 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e4003520 con 0x7f89e4004800 2024-03-06T07:26:04.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.347+0000 7f89f5d9b640 1 --2- 172.21.15.83:0/3508048547 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89e4004800 0x7f89e4004c80 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f89e0007fd0 tx=0x7f89e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cef0adbafa7c4dd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.348+0000 7f89f559a640 1 -- 172.21.15.83:0/3508048547 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e0007ca0 con 0x7f89e4004800 2024-03-06T07:26:04.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.348+0000 7f89f559a640 1 -- 172.21.15.83:0/3508048547 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89e0030070 con 0x7f89e4004800 2024-03-06T07:26:04.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.348+0000 7f89f559a640 1 -- 172.21.15.83:0/3508048547 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e0005d30 con 0x7f89e4004800 2024-03-06T07:26:04.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.348+0000 7f89fcc9a640 1 -- 172.21.15.83:0/3508048547 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89e4004800 msgr2=0x7f89e4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.348+0000 7f89fcc9a640 1 --2- 172.21.15.83:0/3508048547 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89e4004800 0x7f89e4004c80 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f89e0007fd0 tx=0x7f89e4005830 comp rx=0 tx=0).stop 2024-03-06T07:26:04.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.348+0000 7f89f6d9d640 1 -- 172.21.15.83:0/3508048547 reap_dead start 2024-03-06T07:26:04.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.348+0000 7f89fcc9a640 1 -- 172.21.15.83:0/3508048547 shutdown_connections 2024-03-06T07:26:04.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.348+0000 7f89fcc9a640 1 -- 172.21.15.83:0/3508048547 >> 172.21.15.83:0/3508048547 conn(0x7f89f8072520 msgr2=0x7f89f8072930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:04.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.348+0000 7f89fcc9a640 1 -- 172.21.15.83:0/3508048547 shutdown_connections 2024-03-06T07:26:04.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.348+0000 7f89fcc9a640 1 -- 172.21.15.83:0/3508048547 wait complete. 2024-03-06T07:26:04.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.350+0000 7f89fcc9a640 1 Processor -- start 2024-03-06T07:26:04.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.350+0000 7f89fcc9a640 1 -- start start 2024-03-06T07:26:04.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.350+0000 7f89fcc9a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f89e4004800 0x7f89f815c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.350+0000 7f89fcc9a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f89f8158740 0x7f89f806e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.350+0000 7f89fcc9a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89f80c5240 0x7f89f80c56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.350+0000 7f89fcc9a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f806bd80 con 0x7f89f80c5240 2024-03-06T07:26:04.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.350+0000 7f89fcc9a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f806bc00 con 0x7f89f8158740 2024-03-06T07:26:04.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.350+0000 7f89fcc9a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89f80d0690 con 0x7f89e4004800 2024-03-06T07:26:04.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.350+0000 7f89f6d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89f80c5240 0x7f89f80c56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.350+0000 7f89f6d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89f80c5240 0x7f89f80c56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55802/0 (socket says 172.21.15.83:55802) 2024-03-06T07:26:04.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.350+0000 7f89f6d9d640 1 -- 172.21.15.83:0/2931180320 learned_addr learned my addr 172.21.15.83:0/2931180320 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:04.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.351+0000 7f89f659c640 1 --2- 172.21.15.83:0/2931180320 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f89e4004800 0x7f89f815c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.351+0000 7f89f5d9b640 1 --2- 172.21.15.83:0/2931180320 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f89f8158740 0x7f89f806e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.351+0000 7f89f6d9d640 1 -- 172.21.15.83:0/2931180320 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f89e4004800 msgr2=0x7f89f815c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.351+0000 7f89f6d9d640 1 --2- 172.21.15.83:0/2931180320 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f89e4004800 0x7f89f815c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.351+0000 7f89f6d9d640 1 -- 172.21.15.83:0/2931180320 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f89f8158740 msgr2=0x7f89f806e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.351+0000 7f89f6d9d640 1 --2- 172.21.15.83:0/2931180320 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f89f8158740 0x7f89f806e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.351+0000 7f89f6d9d640 1 -- 172.21.15.83:0/2931180320 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f806f190 con 0x7f89f80c5240 2024-03-06T07:26:04.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.351+0000 7f89f6d9d640 1 --2- 172.21.15.83:0/2931180320 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89f80c5240 0x7f89f80c56c0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f89f80710e0 tx=0x7f89e800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.351+0000 7f89eeffd640 1 -- 172.21.15.83:0/2931180320 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e800fd80 con 0x7f89f80c5240 2024-03-06T07:26:04.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.351+0000 7f89eeffd640 1 -- 172.21.15.83:0/2931180320 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89e801d070 con 0x7f89f80c5240 2024-03-06T07:26:04.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.351+0000 7f89fcc9a640 1 -- 172.21.15.83:0/2931180320 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89f806eeb0 con 0x7f89f80c5240 2024-03-06T07:26:04.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.351+0000 7f89eeffd640 1 -- 172.21.15.83:0/2931180320 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e8015670 con 0x7f89f80c5240 2024-03-06T07:26:04.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.351+0000 7f89fcc9a640 1 -- 172.21.15.83:0/2931180320 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89f80b11e0 con 0x7f89f80c5240 2024-03-06T07:26:04.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.352+0000 7f89fcc9a640 1 -- 172.21.15.83:0/2931180320 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89f806afb0 con 0x7f89f80c5240 2024-03-06T07:26:04.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.353+0000 7f89eeffd640 1 -- 172.21.15.83:0/2931180320 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f89e8015810 con 0x7f89f80c5240 2024-03-06T07:26:04.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.353+0000 7f89eeffd640 1 --2- 172.21.15.83:0/2931180320 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f89d806b500 0x7f89d806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.353+0000 7f89eeffd640 1 -- 172.21.15.83:0/2931180320 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f89e808aa00 con 0x7f89f80c5240 2024-03-06T07:26:04.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.354+0000 7f89f659c640 1 --2- 172.21.15.83:0/2931180320 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f89d806b500 0x7f89d806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.354+0000 7f89f659c640 1 --2- 172.21.15.83:0/2931180320 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f89d806b500 0x7f89d806d9c0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f89f0009b00 tx=0x7f89f0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.358+0000 7f89eeffd640 1 -- 172.21.15.83:0/2931180320 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f89f806afb0 con 0x7f89f80c5240 2024-03-06T07:26:04.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.549+0000 7f89fcc9a640 1 -- 172.21.15.83:0/2931180320 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f89f804f330 con 0x7f89f80c5240 2024-03-06T07:26:04.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.549+0000 7f89eeffd640 1 -- 172.21.15.83:0/2931180320 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v82) v1 ==== 80+0+0 (secure 0 0 0) 0x7f89e8010040 con 0x7f89f80c5240 2024-03-06T07:26:04.552 INFO:teuthology.orchestra.run.smithi083.stderr:listed 0 entries 2024-03-06T07:26:04.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.551+0000 7f89fcc9a640 1 -- 172.21.15.83:0/2931180320 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f89d806b500 msgr2=0x7f89d806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.551+0000 7f89fcc9a640 1 --2- 172.21.15.83:0/2931180320 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f89d806b500 0x7f89d806d9c0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f89f0009b00 tx=0x7f89f0008040 comp rx=0 tx=0).stop 2024-03-06T07:26:04.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.551+0000 7f89fcc9a640 1 -- 172.21.15.83:0/2931180320 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89f80c5240 msgr2=0x7f89f80c56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.551+0000 7f89fcc9a640 1 --2- 172.21.15.83:0/2931180320 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89f80c5240 0x7f89f80c56c0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f89f80710e0 tx=0x7f89e800f580 comp rx=0 tx=0).stop 2024-03-06T07:26:04.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.551+0000 7f89fcc9a640 1 -- 172.21.15.83:0/2931180320 shutdown_connections 2024-03-06T07:26:04.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.551+0000 7f89fcc9a640 1 --2- 172.21.15.83:0/2931180320 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f89d806b500 0x7f89d806d9c0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.551+0000 7f89fcc9a640 1 --2- 172.21.15.83:0/2931180320 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f89f80c5240 0x7f89f80c56c0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.551+0000 7f89fcc9a640 1 --2- 172.21.15.83:0/2931180320 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f89f8158740 0x7f89f806e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.551+0000 7f89fcc9a640 1 --2- 172.21.15.83:0/2931180320 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f89e4004800 0x7f89f815c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.551+0000 7f89fcc9a640 1 -- 172.21.15.83:0/2931180320 >> 172.21.15.83:0/2931180320 conn(0x7f89f8072520 msgr2=0x7f89f80c6470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:04.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.551+0000 7f89fcc9a640 1 -- 172.21.15.83:0/2931180320 shutdown_connections 2024-03-06T07:26:04.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:04.551+0000 7f89fcc9a640 1 -- 172.21.15.83:0/2931180320 wait complete. 2024-03-06T07:26:04.564 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi083.front.sepia.ceph.com... 2024-03-06T07:26:04.564 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:04.565 DEBUG:teuthology.orchestra.run.smithi083:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:04.589 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:ceph-fuse[66134]: fuse finished with error 0 and tester_r 0 2024-03-06T07:26:04.591 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi083.front.sepia.ceph.com 2024-03-06T07:26:04.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:04.592 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:26:04.742 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:26:04.742 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:04.761 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=287 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:26:04.761 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi183.front.sepia.ceph.com 2024-03-06T07:26:04.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:04.761 DEBUG:teuthology.orchestra.run.smithi183:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:26:04.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:26:04.819 INFO:teuthology.orchestra.run.smithi183.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:26:04.819 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:26:04.820 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:04.834 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1257 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:26:04.835 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi083.front.sepia.ceph.com 2024-03-06T07:26:04.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:04.835 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:26:04.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:26:04.889 INFO:teuthology.orchestra.run.smithi083.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:26:04.889 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:26:04.889 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:04.907 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=287 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:26:04.907 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi183.front.sepia.ceph.com 2024-03-06T07:26:04.907 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:04.907 DEBUG:teuthology.orchestra.run.smithi183:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:26:04.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:26:04.964 INFO:teuthology.orchestra.run.smithi183.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:26:04.965 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:05.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.062+0000 7f2735484640 1 Processor -- start 2024-03-06T07:26:05.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.062+0000 7f2735484640 1 -- start start 2024-03-06T07:26:05.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.063+0000 7f2735484640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f273006da40 0x7f27300cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.063+0000 7f2735484640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f27300cd7a0 0x7f273015cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.063+0000 7f2735484640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27300cc980 con 0x7f27301424f0 2024-03-06T07:26:05.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.063+0000 7f2735484640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27300cd220 con 0x7f27300cd7a0 2024-03-06T07:26:05.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.063+0000 7f2735484640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27300ccc60 con 0x7f273006da40 2024-03-06T07:26:05.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.064+0000 7f272e7fc640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f273006da40 0x7f27300cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.064+0000 7f272f7fe640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f27300cd7a0 0x7f273015cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.064+0000 7f272f7fe640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f27300cd7a0 0x7f273015cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:60870/0 (socket says 172.21.15.83:60870) 2024-03-06T07:26:05.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.064+0000 7f272e7fc640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f273006da40 0x7f27300cc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:53062/0 (socket says 172.21.15.83:53062) 2024-03-06T07:26:05.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.064+0000 7f272effd640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f27301424f0 0x7f273006d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:44622/0 (socket says 172.21.15.83:44622) 2024-03-06T07:26:05.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.064+0000 7f272f7fe640 1 -- 172.21.15.83:0/4216071278 learned_addr learned my addr 172.21.15.83:0/4216071278 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:05.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.064+0000 7f272e7fc640 1 -- 172.21.15.83:0/4216071278 >> v2:172.21.15.183:3300/0 conn(0x7f27300cd7a0 msgr2=0x7f273015cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:05.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.064+0000 7f272e7fc640 1 --2- 172.21.15.83:0/4216071278 >> v2:172.21.15.183:3300/0 conn(0x7f27300cd7a0 0x7f273015cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.064+0000 7f272e7fc640 1 -- 172.21.15.83:0/4216071278 >> v1:172.21.15.183:6789/0 conn(0x7f27301424f0 legacy=0x7f273006d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.064+0000 7f272e7fc640 1 -- 172.21.15.83:0/4216071278 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27300ccf40 con 0x7f273006da40 2024-03-06T07:26:05.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.064+0000 7f272f7fe640 1 --2- 172.21.15.83:0/4216071278 >> v2:172.21.15.183:3300/0 conn(0x7f27300cd7a0 0x7f273015cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:05.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.065+0000 7f272e7fc640 1 --2- 172.21.15.83:0/4216071278 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f273006da40 0x7f27300cc2f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f27180097a0 tx=0x7f271802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd45a297f8582afc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:05.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.065+0000 7f272dffb640 1 -- 172.21.15.83:0/4216071278 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2718035020 con 0x7f273006da40 2024-03-06T07:26:05.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.065+0000 7f272dffb640 1 -- 172.21.15.83:0/4216071278 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2718031830 con 0x7f273006da40 2024-03-06T07:26:05.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.065+0000 7f272dffb640 1 -- 172.21.15.83:0/4216071278 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2718031c00 con 0x7f273006da40 2024-03-06T07:26:05.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.066+0000 7f2735484640 1 -- 172.21.15.83:0/4216071278 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f273006da40 msgr2=0x7f27300cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.066+0000 7f2735484640 1 --2- 172.21.15.83:0/4216071278 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f273006da40 0x7f27300cc2f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f27180097a0 tx=0x7f271802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.066+0000 7f2735484640 1 -- 172.21.15.83:0/4216071278 shutdown_connections 2024-03-06T07:26:05.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.066+0000 7f2735484640 1 --2- 172.21.15.83:0/4216071278 >> v2:172.21.15.183:3300/0 conn(0x7f27300cd7a0 0x7f273015cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.066+0000 7f2735484640 1 --2- 172.21.15.83:0/4216071278 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f273006da40 0x7f27300cc2f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.066+0000 7f2735484640 1 -- 172.21.15.83:0/4216071278 >> 172.21.15.83:0/4216071278 conn(0x7f27300cf8a0 msgr2=0x7f27300cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:05.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.066+0000 7f2735484640 1 -- 172.21.15.83:0/4216071278 shutdown_connections 2024-03-06T07:26:05.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.066+0000 7f2735484640 1 -- 172.21.15.83:0/4216071278 wait complete. 2024-03-06T07:26:05.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.068+0000 7f2735484640 1 Processor -- start 2024-03-06T07:26:05.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.068+0000 7f2735484640 1 -- start start 2024-03-06T07:26:05.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.069+0000 7f2735484640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f273006da40 0x7f27301e4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.069+0000 7f2735484640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f27300cd7a0 0x7f273013e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.069+0000 7f2735484640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f27301424f0 0x7f27301f1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.069+0000 7f2735484640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2730071ce0 con 0x7f273006da40 2024-03-06T07:26:05.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.069+0000 7f2735484640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2730071b60 con 0x7f27300cd7a0 2024-03-06T07:26:05.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.069+0000 7f2735484640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2730071e60 con 0x7f27301424f0 2024-03-06T07:26:05.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.069+0000 7f272effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f273006da40 0x7f27301e4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.069+0000 7f272effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f273006da40 0x7f27301e4700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55814/0 (socket says 172.21.15.83:55814) 2024-03-06T07:26:05.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.069+0000 7f272effd640 1 -- 172.21.15.83:0/2932545530 learned_addr learned my addr 172.21.15.83:0/2932545530 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:05.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.069+0000 7f272f7fe640 1 --2- 172.21.15.83:0/2932545530 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f27301424f0 0x7f27301f1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.069+0000 7f272e7fc640 1 --2- 172.21.15.83:0/2932545530 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f27300cd7a0 0x7f273013e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.069+0000 7f272effd640 1 -- 172.21.15.83:0/2932545530 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f27301424f0 msgr2=0x7f27301f1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.069+0000 7f272effd640 1 --2- 172.21.15.83:0/2932545530 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f27301424f0 0x7f27301f1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.069+0000 7f272effd640 1 -- 172.21.15.83:0/2932545530 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f27300cd7a0 msgr2=0x7f273013e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.069+0000 7f272effd640 1 --2- 172.21.15.83:0/2932545530 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f27300cd7a0 0x7f273013e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.069+0000 7f272effd640 1 -- 172.21.15.83:0/2932545530 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f273013cf80 con 0x7f273006da40 2024-03-06T07:26:05.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.069+0000 7f272e7fc640 1 --2- 172.21.15.83:0/2932545530 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f27300cd7a0 0x7f273013e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:05.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.069+0000 7f272effd640 1 --2- 172.21.15.83:0/2932545530 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f273006da40 0x7f27301e4700 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f272800c730 tx=0x7f272800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:05.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.070+0000 7f27277fe640 1 -- 172.21.15.83:0/2932545530 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2728017070 con 0x7f273006da40 2024-03-06T07:26:05.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.070+0000 7f27277fe640 1 -- 172.21.15.83:0/2932545530 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f272800ce40 con 0x7f273006da40 2024-03-06T07:26:05.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.070+0000 7f2735484640 1 -- 172.21.15.83:0/2932545530 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f273013d540 con 0x7f273006da40 2024-03-06T07:26:05.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.070+0000 7f27277fe640 1 -- 172.21.15.83:0/2932545530 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f272800a860 con 0x7f273006da40 2024-03-06T07:26:05.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.070+0000 7f2735484640 1 -- 172.21.15.83:0/2932545530 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f273013db00 con 0x7f273006da40 2024-03-06T07:26:05.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.071+0000 7f2735484640 1 -- 172.21.15.83:0/2932545530 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f273004f330 con 0x7f273006da40 2024-03-06T07:26:05.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.071+0000 7f27277fe640 1 -- 172.21.15.83:0/2932545530 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f272801a070 con 0x7f273006da40 2024-03-06T07:26:05.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.071+0000 7f27277fe640 1 --2- 172.21.15.83:0/2932545530 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f271006b4d0 0x7f271006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.072+0000 7f27277fe640 1 -- 172.21.15.83:0/2932545530 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2728089780 con 0x7f273006da40 2024-03-06T07:26:05.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.072+0000 7f272e7fc640 1 --2- 172.21.15.83:0/2932545530 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f271006b4d0 0x7f271006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.072+0000 7f272e7fc640 1 --2- 172.21.15.83:0/2932545530 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f271006b4d0 0x7f271006d990 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f2718036000 tx=0x7f271803c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:05.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.076+0000 7f27277fe640 1 -- 172.21.15.83:0/2932545530 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2728057fb0 con 0x7f273006da40 2024-03-06T07:26:05.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.290+0000 7f2735484640 1 -- 172.21.15.83:0/2932545530 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27300d0620 con 0x7f273006da40 2024-03-06T07:26:05.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.290+0000 7f27277fe640 1 -- 172.21.15.83:0/2932545530 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f272805c050 con 0x7f273006da40 2024-03-06T07:26:05.293 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:05.294 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":39,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/2525949760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":2525949760},{"type":"v1","addr":"172.21.15.183:6834","nonce":2525949760}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35},{"gid":6262,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2104447869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2104447869},{"type":"v1","addr":"172.21.15.183:6835","nonce":2104447869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6954,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/779621621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":779621621},{"type":"v1","addr":"172.21.15.83:6836","nonce":779621621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:54.631963+0000","modified":"2024-03-06T07:25:56.637047+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6957":{"gid":6957,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/3056214366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":3056214366},{"type":"v1","addr":"172.21.15.83:6837","nonce":3056214366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6957,"qdb_cluster":[6957]},"id":6}]} 2024-03-06T07:26:05.294 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 39 2024-03-06T07:26:05.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.292+0000 7f2735484640 1 -- 172.21.15.83:0/2932545530 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f271006b4d0 msgr2=0x7f271006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.292+0000 7f2735484640 1 --2- 172.21.15.83:0/2932545530 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f271006b4d0 0x7f271006d990 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f2718036000 tx=0x7f271803c040 comp rx=0 tx=0).stop 2024-03-06T07:26:05.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.292+0000 7f2735484640 1 -- 172.21.15.83:0/2932545530 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f273006da40 msgr2=0x7f27301e4700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.292+0000 7f2735484640 1 --2- 172.21.15.83:0/2932545530 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f273006da40 0x7f27301e4700 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f272800c730 tx=0x7f272800cbf0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.292+0000 7f2735484640 1 -- 172.21.15.83:0/2932545530 shutdown_connections 2024-03-06T07:26:05.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.292+0000 7f2735484640 1 --2- 172.21.15.83:0/2932545530 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f271006b4d0 0x7f271006d990 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.292+0000 7f2735484640 1 --2- 172.21.15.83:0/2932545530 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f27301424f0 0x7f27301f1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.292+0000 7f2735484640 1 --2- 172.21.15.83:0/2932545530 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f27300cd7a0 0x7f273013e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.292+0000 7f2735484640 1 --2- 172.21.15.83:0/2932545530 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f273006da40 0x7f27301e4700 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.292+0000 7f2735484640 1 -- 172.21.15.83:0/2932545530 >> 172.21.15.83:0/2932545530 conn(0x7f27300cf8a0 msgr2=0x7f273015d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:05.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.292+0000 7f2735484640 1 -- 172.21.15.83:0/2932545530 shutdown_connections 2024-03-06T07:26:05.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.292+0000 7f2735484640 1 -- 172.21.15.83:0/2932545530 wait complete. 2024-03-06T07:26:05.306 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:05.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.427+0000 7f7667864640 1 Processor -- start 2024-03-06T07:26:05.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.427+0000 7f7667864640 1 -- start start 2024-03-06T07:26:05.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.428+0000 7f7667864640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f76600d22b0 0x7f76600d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.428+0000 7f7667864640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f76600b41f0 0x7f76600b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.428+0000 7f7667864640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76600b3430 con 0x7f766015dfe0 2024-03-06T07:26:05.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.428+0000 7f7667864640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76600d2c50 con 0x7f76600b41f0 2024-03-06T07:26:05.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.428+0000 7f7667864640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76600b31e0 con 0x7f76600d22b0 2024-03-06T07:26:05.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.428+0000 7f7664dff640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f76600d22b0 0x7f76600d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.428+0000 7f7665600640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f766015dfe0 0x7f76600cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:44628/0 (socket says 172.21.15.83:44628) 2024-03-06T07:26:05.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.428+0000 7f7665600640 1 -- 172.21.15.83:0/1760579538 learned_addr learned my addr 172.21.15.83:0/1760579538 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:05.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.428+0000 7f7665e01640 1 --2- 172.21.15.83:0/1760579538 >> v2:172.21.15.83:3300/0 conn(0x7f76600b41f0 0x7f76600b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.428+0000 7f7665e01640 1 -- 172.21.15.83:0/1760579538 >> v2:172.21.15.183:3300/0 conn(0x7f76600d22b0 msgr2=0x7f76600d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:05.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.428+0000 7f7665e01640 1 --2- 172.21.15.83:0/1760579538 >> v2:172.21.15.183:3300/0 conn(0x7f76600d22b0 0x7f76600d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.428+0000 7f7665e01640 1 -- 172.21.15.83:0/1760579538 >> v1:172.21.15.183:6789/0 conn(0x7f766015dfe0 legacy=0x7f76600cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.428+0000 7f7665e01640 1 -- 172.21.15.83:0/1760579538 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76600b3710 con 0x7f76600b41f0 2024-03-06T07:26:05.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.429+0000 7f7665e01640 1 --2- 172.21.15.83:0/1760579538 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f76600b41f0 0x7f76600b4680 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f76580097a0 tx=0x7f765802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29e9813f1aa34780 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:05.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.429+0000 7f7657fff640 1 -- 172.21.15.83:0/1760579538 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7658035020 con 0x7f76600b41f0 2024-03-06T07:26:05.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.429+0000 7f7657fff640 1 -- 172.21.15.83:0/1760579538 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7658031830 con 0x7f76600b41f0 2024-03-06T07:26:05.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.429+0000 7f7657fff640 1 -- 172.21.15.83:0/1760579538 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7658031c00 con 0x7f76600b41f0 2024-03-06T07:26:05.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.430+0000 7f7667864640 1 -- 172.21.15.83:0/1760579538 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f76600b41f0 msgr2=0x7f76600b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.430+0000 7f7667864640 1 --2- 172.21.15.83:0/1760579538 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f76600b41f0 0x7f76600b4680 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f76580097a0 tx=0x7f765802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.430+0000 7f7667864640 1 -- 172.21.15.83:0/1760579538 shutdown_connections 2024-03-06T07:26:05.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.430+0000 7f7667864640 1 --2- 172.21.15.83:0/1760579538 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f76600b41f0 0x7f76600b4680 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.430+0000 7f7667864640 1 --2- 172.21.15.83:0/1760579538 >> v2:172.21.15.183:3300/0 conn(0x7f76600d22b0 0x7f76600d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.430+0000 7f7667864640 1 -- 172.21.15.83:0/1760579538 >> 172.21.15.83:0/1760579538 conn(0x7f76600d3cd0 msgr2=0x7f76600d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:05.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.430+0000 7f7667864640 1 -- 172.21.15.83:0/1760579538 shutdown_connections 2024-03-06T07:26:05.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.430+0000 7f7667864640 1 -- 172.21.15.83:0/1760579538 wait complete. 2024-03-06T07:26:05.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.431+0000 7f7667864640 1 Processor -- start 2024-03-06T07:26:05.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.431+0000 7f7667864640 1 -- start start 2024-03-06T07:26:05.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.432+0000 7f7667864640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f76600b41f0 0x7f766015c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.432+0000 7f7667864640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f76600d22b0 0x7f76600b85f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.432+0000 7f7667864640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f766015dfe0 0x7f76601603d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.432+0000 7f7667864640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76600cfa00 con 0x7f766015dfe0 2024-03-06T07:26:05.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.432+0000 7f7667864640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76600b3d60 con 0x7f76600b41f0 2024-03-06T07:26:05.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.432+0000 7f7667864640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76600d1c70 con 0x7f76600d22b0 2024-03-06T07:26:05.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.432+0000 7f7665e01640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f766015dfe0 0x7f76601603d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.432+0000 7f7665e01640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f766015dfe0 0x7f76601603d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55836/0 (socket says 172.21.15.83:55836) 2024-03-06T07:26:05.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.432+0000 7f7665e01640 1 -- 172.21.15.83:0/1131382038 learned_addr learned my addr 172.21.15.83:0/1131382038 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:05.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.432+0000 7f7664dff640 1 --2- 172.21.15.83:0/1131382038 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f76600d22b0 0x7f76600b85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.432+0000 7f7665e01640 1 -- 172.21.15.83:0/1131382038 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f76600d22b0 msgr2=0x7f76600b85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.432+0000 7f7665e01640 1 --2- 172.21.15.83:0/1131382038 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f76600d22b0 0x7f76600b85f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.432+0000 7f7665600640 1 --2- 172.21.15.83:0/1131382038 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f76600b41f0 0x7f766015c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.432+0000 7f7665e01640 1 -- 172.21.15.83:0/1131382038 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f76600b41f0 msgr2=0x7f766015c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.432+0000 7f7665e01640 1 --2- 172.21.15.83:0/1131382038 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f76600b41f0 0x7f766015c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.432+0000 7f7665e01640 1 -- 172.21.15.83:0/1131382038 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76600cd170 con 0x7f766015dfe0 2024-03-06T07:26:05.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.432+0000 7f7664dff640 1 --2- 172.21.15.83:0/1131382038 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f76600d22b0 0x7f76600b85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:05.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.432+0000 7f7665600640 1 --2- 172.21.15.83:0/1131382038 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f76600b41f0 0x7f766015c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:05.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.432+0000 7f7665e01640 1 --2- 172.21.15.83:0/1131382038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f766015dfe0 0x7f76601603d0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f76580098c0 tx=0x7f76580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:05.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.432+0000 7f7655ffb640 1 -- 172.21.15.83:0/1131382038 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7658035040 con 0x7f766015dfe0 2024-03-06T07:26:05.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.433+0000 7f7655ffb640 1 -- 172.21.15.83:0/1131382038 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f765803b840 con 0x7f766015dfe0 2024-03-06T07:26:05.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.433+0000 7f7667864640 1 -- 172.21.15.83:0/1131382038 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76600cd450 con 0x7f766015dfe0 2024-03-06T07:26:05.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.433+0000 7f7655ffb640 1 -- 172.21.15.83:0/1131382038 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f765803bc30 con 0x7f766015dfe0 2024-03-06T07:26:05.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.433+0000 7f7667864640 1 -- 172.21.15.83:0/1131382038 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76600cda10 con 0x7f766015dfe0 2024-03-06T07:26:05.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.434+0000 7f7667864640 1 -- 172.21.15.83:0/1131382038 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f766004f330 con 0x7f766015dfe0 2024-03-06T07:26:05.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.434+0000 7f7655ffb640 1 -- 172.21.15.83:0/1131382038 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7658038020 con 0x7f766015dfe0 2024-03-06T07:26:05.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.435+0000 7f7655ffb640 1 --2- 172.21.15.83:0/1131382038 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f764406b4d0 0x7f764406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.435+0000 7f7665600640 1 --2- 172.21.15.83:0/1131382038 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f764406b4d0 0x7f764406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.435+0000 7f7655ffb640 1 -- 172.21.15.83:0/1131382038 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f76580ac240 con 0x7f766015dfe0 2024-03-06T07:26:05.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.435+0000 7f7665600640 1 --2- 172.21.15.83:0/1131382038 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f764406b4d0 0x7f764406d990 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f765c0012e0 tx=0x7f765c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:05.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.440+0000 7f7655ffb640 1 -- 172.21.15.83:0/1131382038 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7658031800 con 0x7f766015dfe0 2024-03-06T07:26:05.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.613+0000 7f7667864640 1 -- 172.21.15.83:0/1131382038 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f766015d2e0 con 0x7f766015dfe0 2024-03-06T07:26:05.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.614+0000 7f7655ffb640 1 -- 172.21.15.83:0/1131382038 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16672 (secure 0 0 0) 0x7f765807b0c0 con 0x7f766015dfe0 2024-03-06T07:26:05.617 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:05.617 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:25:56.279928+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:52.856525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:53.598434+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:05.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.616+0000 7f7667864640 1 -- 172.21.15.83:0/1131382038 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f764406b4d0 msgr2=0x7f764406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.616+0000 7f7667864640 1 --2- 172.21.15.83:0/1131382038 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f764406b4d0 0x7f764406d990 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f765c0012e0 tx=0x7f765c009040 comp rx=0 tx=0).stop 2024-03-06T07:26:05.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.616+0000 7f7667864640 1 -- 172.21.15.83:0/1131382038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f766015dfe0 msgr2=0x7f76601603d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.616+0000 7f7667864640 1 --2- 172.21.15.83:0/1131382038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f766015dfe0 0x7f76601603d0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f76580098c0 tx=0x7f76580046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.616+0000 7f7667864640 1 -- 172.21.15.83:0/1131382038 shutdown_connections 2024-03-06T07:26:05.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.616+0000 7f7667864640 1 --2- 172.21.15.83:0/1131382038 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f764406b4d0 0x7f764406d990 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.616+0000 7f7667864640 1 --2- 172.21.15.83:0/1131382038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f766015dfe0 0x7f76601603d0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.616+0000 7f7667864640 1 --2- 172.21.15.83:0/1131382038 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f76600d22b0 0x7f76600b85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.616+0000 7f7667864640 1 --2- 172.21.15.83:0/1131382038 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f76600b41f0 0x7f766015c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.616+0000 7f7667864640 1 -- 172.21.15.83:0/1131382038 >> 172.21.15.83:0/1131382038 conn(0x7f76600d3cd0 msgr2=0x7f766006ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:05.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.616+0000 7f7667864640 1 -- 172.21.15.83:0/1131382038 shutdown_connections 2024-03-06T07:26:05.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.616+0000 7f7667864640 1 -- 172.21.15.83:0/1131382038 wait complete. 2024-03-06T07:26:05.630 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:25:52.856525+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:25:53.598434+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:26:05.630 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:26:05.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.732+0000 7f52fdeed640 1 Processor -- start 2024-03-06T07:26:05.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.732+0000 7f52fdeed640 1 -- start start 2024-03-06T07:26:05.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.733+0000 7f52fdeed640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f52f8071a70 0x7f52f8071e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.733+0000 7f52fdeed640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52f8072390 con 0x7f52f80b42e0 2024-03-06T07:26:05.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.733+0000 7f52fdeed640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52f815b260 con 0x7f52f815ad90 2024-03-06T07:26:05.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.733+0000 7f52fdeed640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52f815c020 con 0x7f52f8071a70 2024-03-06T07:26:05.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.733+0000 7f52f77fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f52f8071a70 0x7f52f8071e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.733+0000 7f52f77fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f52f8071a70 0x7f52f8071e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55848/0 (socket says 172.21.15.83:55848) 2024-03-06T07:26:05.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.733+0000 7f52f7fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f52f80b42e0 0x7f52f815c210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:33130/0 (socket says 172.21.15.83:33130) 2024-03-06T07:26:05.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.733+0000 7f52f77fe640 1 -- 172.21.15.83:0/52938075 learned_addr learned my addr 172.21.15.83:0/52938075 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:05.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.733+0000 7f52f77fe640 1 -- 172.21.15.83:0/52938075 >> v1:172.21.15.183:6789/0 conn(0x7f52f815ad90 legacy=0x7f52f80b3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:05.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.733+0000 7f52f77fe640 1 -- 172.21.15.83:0/52938075 >> v1:172.21.15.83:6789/0 conn(0x7f52f80b42e0 legacy=0x7f52f815c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.733+0000 7f52f67fc640 1 -- 172.21.15.83:0/52938075 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 338594206 0 0) 0x7f52f8072390 con 0x7f52f80b42e0 2024-03-06T07:26:05.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.733+0000 7f52f77fe640 1 -- 172.21.15.83:0/52938075 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f815b540 con 0x7f52f8071a70 2024-03-06T07:26:05.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.734+0000 7f52f77fe640 1 --2- 172.21.15.83:0/52938075 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f52f8071a70 0x7f52f8071e50 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f52e8009870 tx=0x7f52e802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38610b1274b867fa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:05.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.734+0000 7f52f67fc640 1 -- 172.21.15.83:0/52938075 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e8035020 con 0x7f52f8071a70 2024-03-06T07:26:05.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.734+0000 7f52f67fc640 1 -- 172.21.15.83:0/52938075 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52e8031870 con 0x7f52f8071a70 2024-03-06T07:26:05.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.734+0000 7f52f67fc640 1 -- 172.21.15.83:0/52938075 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e8031c40 con 0x7f52f8071a70 2024-03-06T07:26:05.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.735+0000 7f52fdeed640 1 -- 172.21.15.83:0/52938075 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f52f8071a70 msgr2=0x7f52f8071e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.735+0000 7f52fdeed640 1 --2- 172.21.15.83:0/52938075 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f52f8071a70 0x7f52f8071e50 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f52e8009870 tx=0x7f52e802b910 comp rx=0 tx=0).stop 2024-03-06T07:26:05.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.735+0000 7f52fdeed640 1 -- 172.21.15.83:0/52938075 shutdown_connections 2024-03-06T07:26:05.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.735+0000 7f52fdeed640 1 --2- 172.21.15.83:0/52938075 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f52f8071a70 0x7f52f8071e50 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.735+0000 7f52fdeed640 1 -- 172.21.15.83:0/52938075 >> 172.21.15.83:0/52938075 conn(0x7f52f80b2660 msgr2=0x7f52f80b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:05.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.735+0000 7f52fdeed640 1 -- 172.21.15.83:0/52938075 shutdown_connections 2024-03-06T07:26:05.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.735+0000 7f52fdeed640 1 -- 172.21.15.83:0/52938075 wait complete. 2024-03-06T07:26:05.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.736+0000 7f52fdeed640 1 Processor -- start 2024-03-06T07:26:05.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.736+0000 7f52fdeed640 1 -- start start 2024-03-06T07:26:05.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.737+0000 7f52fdeed640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f52f8071a70 0x7f52f81490e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.737+0000 7f52fdeed640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f52f80b42e0 0x7f52f80c9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.737+0000 7f52fdeed640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f52f815ad90 0x7f52f80b3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.737+0000 7f52fdeed640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f815bb00 con 0x7f52f8071a70 2024-03-06T07:26:05.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.737+0000 7f52fdeed640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f80b4720 con 0x7f52f80b42e0 2024-03-06T07:26:05.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.737+0000 7f52fdeed640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52f815c920 con 0x7f52f815ad90 2024-03-06T07:26:05.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.737+0000 7f52f77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f52f8071a70 0x7f52f81490e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.737+0000 7f52f77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f52f8071a70 0x7f52f81490e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55852/0 (socket says 172.21.15.83:55852) 2024-03-06T07:26:05.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.737+0000 7f52f77fe640 1 -- 172.21.15.83:0/3530454199 learned_addr learned my addr 172.21.15.83:0/3530454199 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:05.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.737+0000 7f52f7fff640 1 --2- 172.21.15.83:0/3530454199 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f52f815ad90 0x7f52f80b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.737+0000 7f52f77fe640 1 -- 172.21.15.83:0/3530454199 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f52f815ad90 msgr2=0x7f52f80b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.737+0000 7f52f6ffd640 1 --2- 172.21.15.83:0/3530454199 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f52f80b42e0 0x7f52f80c9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.737+0000 7f52f77fe640 1 --2- 172.21.15.83:0/3530454199 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f52f815ad90 0x7f52f80b3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.737+0000 7f52f77fe640 1 -- 172.21.15.83:0/3530454199 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f52f80b42e0 msgr2=0x7f52f80c9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.737+0000 7f52f77fe640 1 --2- 172.21.15.83:0/3530454199 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f52f80b42e0 0x7f52f80c9ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.737+0000 7f52f77fe640 1 -- 172.21.15.83:0/3530454199 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f80ca5e0 con 0x7f52f8071a70 2024-03-06T07:26:05.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.737+0000 7f52f77fe640 1 --2- 172.21.15.83:0/3530454199 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f52f8071a70 0x7f52f81490e0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f52e8009870 tx=0x7f52e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:05.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.738+0000 7f52dffff640 1 -- 172.21.15.83:0/3530454199 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e803bb60 con 0x7f52f8071a70 2024-03-06T07:26:05.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.738+0000 7f52dffff640 1 -- 172.21.15.83:0/3530454199 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52e803bd20 con 0x7f52f8071a70 2024-03-06T07:26:05.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.738+0000 7f52fdeed640 1 -- 172.21.15.83:0/3530454199 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52f80ca8c0 con 0x7f52f8071a70 2024-03-06T07:26:05.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.738+0000 7f52dffff640 1 -- 172.21.15.83:0/3530454199 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e8004270 con 0x7f52f8071a70 2024-03-06T07:26:05.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.738+0000 7f52fdeed640 1 -- 172.21.15.83:0/3530454199 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52f81f7850 con 0x7f52f8071a70 2024-03-06T07:26:05.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.739+0000 7f52fdeed640 1 -- 172.21.15.83:0/3530454199 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52f8159f30 con 0x7f52f8071a70 2024-03-06T07:26:05.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.739+0000 7f52dffff640 1 -- 172.21.15.83:0/3530454199 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f52e804c060 con 0x7f52f8071a70 2024-03-06T07:26:05.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.740+0000 7f52dffff640 1 --2- 172.21.15.83:0/3530454199 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f52e406b4d0 0x7f52e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.740+0000 7f52dffff640 1 -- 172.21.15.83:0/3530454199 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f52e80ac1a0 con 0x7f52f8071a70 2024-03-06T07:26:05.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.740+0000 7f52f6ffd640 1 --2- 172.21.15.83:0/3530454199 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f52e406b4d0 0x7f52e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.740+0000 7f52f6ffd640 1 --2- 172.21.15.83:0/3530454199 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f52e406b4d0 0x7f52e406d990 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f52f81f7330 tx=0x7f52ec008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:05.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.744+0000 7f52dffff640 1 -- 172.21.15.83:0/3530454199 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f52f8159f30 con 0x7f52f8071a70 2024-03-06T07:26:05.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.918+0000 7f52fdeed640 1 -- 172.21.15.83:0/3530454199 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f52f80d07d0 con 0x7f52f8071a70 2024-03-06T07:26:05.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.919+0000 7f52dffff640 1 -- 172.21.15.83:0/3530454199 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7f52e8040070 con 0x7f52f8071a70 2024-03-06T07:26:05.922 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:26:05.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.921+0000 7f52fdeed640 1 -- 172.21.15.83:0/3530454199 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f52e406b4d0 msgr2=0x7f52e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.921+0000 7f52fdeed640 1 --2- 172.21.15.83:0/3530454199 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f52e406b4d0 0x7f52e406d990 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f52f81f7330 tx=0x7f52ec008040 comp rx=0 tx=0).stop 2024-03-06T07:26:05.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.921+0000 7f52fdeed640 1 -- 172.21.15.83:0/3530454199 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f52f8071a70 msgr2=0x7f52f81490e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.921+0000 7f52fdeed640 1 --2- 172.21.15.83:0/3530454199 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f52f8071a70 0x7f52f81490e0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f52e8009870 tx=0x7f52e80046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.921+0000 7f52fdeed640 1 -- 172.21.15.83:0/3530454199 shutdown_connections 2024-03-06T07:26:05.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.921+0000 7f52fdeed640 1 --2- 172.21.15.83:0/3530454199 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f52e406b4d0 0x7f52e406d990 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.921+0000 7f52fdeed640 1 --2- 172.21.15.83:0/3530454199 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f52f815ad90 0x7f52f80b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.921+0000 7f52fdeed640 1 --2- 172.21.15.83:0/3530454199 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f52f80b42e0 0x7f52f80c9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.921+0000 7f52fdeed640 1 --2- 172.21.15.83:0/3530454199 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f52f8071a70 0x7f52f81490e0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.921+0000 7f52fdeed640 1 -- 172.21.15.83:0/3530454199 >> 172.21.15.83:0/3530454199 conn(0x7f52f80b2660 msgr2=0x7f52f80b1050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:05.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.921+0000 7f52fdeed640 1 -- 172.21.15.83:0/3530454199 shutdown_connections 2024-03-06T07:26:05.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:05.921+0000 7f52fdeed640 1 -- 172.21.15.83:0/3530454199 wait complete. 2024-03-06T07:26:05.935 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:26:06.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.035+0000 7f08723f7640 1 Processor -- start 2024-03-06T07:26:06.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.035+0000 7f08723f7640 1 -- start start 2024-03-06T07:26:06.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.036+0000 7f08723f7640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f086c0d4860 0x7f086c0d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.036+0000 7f08723f7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f086c0b0e30 0x7f086c0b1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.036+0000 7f08723f7640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f086c1618b0 con 0x7f086c1614d0 2024-03-06T07:26:06.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.036+0000 7f08723f7640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f086c0d4cf0 con 0x7f086c0b0e30 2024-03-06T07:26:06.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.036+0000 7f08723f7640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f086c0cf400 con 0x7f086c0d4860 2024-03-06T07:26:06.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.036+0000 7f0870994640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f086c0b0e30 0x7f086c0b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.036+0000 7f0870994640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f086c0b0e30 0x7f086c0b1260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55866/0 (socket says 172.21.15.83:55866) 2024-03-06T07:26:06.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.036+0000 7f0870994640 1 -- 172.21.15.83:0/2205944486 learned_addr learned my addr 172.21.15.83:0/2205944486 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:06.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.037+0000 7f086b7fe640 1 --2- 172.21.15.83:0/2205944486 >> v2:172.21.15.183:3300/0 conn(0x7f086c0d4860 0x7f086c0d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.037+0000 7f0870994640 1 -- 172.21.15.83:0/2205944486 >> v2:172.21.15.183:3300/0 conn(0x7f086c0d4860 msgr2=0x7f086c0d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:06.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.037+0000 7f0870994640 1 --2- 172.21.15.83:0/2205944486 >> v2:172.21.15.183:3300/0 conn(0x7f086c0d4860 0x7f086c0d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.037+0000 7f0870994640 1 -- 172.21.15.83:0/2205944486 >> v1:172.21.15.183:6789/0 conn(0x7f086c1614d0 legacy=0x7f086c0d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:06.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.037+0000 7f0870994640 1 -- 172.21.15.83:0/2205944486 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f086c0cf6a0 con 0x7f086c0b0e30 2024-03-06T07:26:06.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.037+0000 7f0870994640 1 --2- 172.21.15.83:0/2205944486 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f086c0b0e30 0x7f086c0b1260 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f086c0b2a70 tx=0x7f086402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b45445e04020da8f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.038+0000 7f086affd640 1 -- 172.21.15.83:0/2205944486 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086402ca50 con 0x7f086c0b0e30 2024-03-06T07:26:06.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.038+0000 7f086affd640 1 -- 172.21.15.83:0/2205944486 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f086402cbf0 con 0x7f086c0b0e30 2024-03-06T07:26:06.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.038+0000 7f086affd640 1 -- 172.21.15.83:0/2205944486 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08640343f0 con 0x7f086c0b0e30 2024-03-06T07:26:06.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.038+0000 7f08723f7640 1 -- 172.21.15.83:0/2205944486 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f086c0b0e30 msgr2=0x7f086c0b1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.038+0000 7f08723f7640 1 --2- 172.21.15.83:0/2205944486 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f086c0b0e30 0x7f086c0b1260 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f086c0b2a70 tx=0x7f086402beb0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.038+0000 7f08723f7640 1 -- 172.21.15.83:0/2205944486 shutdown_connections 2024-03-06T07:26:06.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.038+0000 7f08723f7640 1 --2- 172.21.15.83:0/2205944486 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f086c0b0e30 0x7f086c0b1260 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.038+0000 7f08723f7640 1 --2- 172.21.15.83:0/2205944486 >> v2:172.21.15.183:3300/0 conn(0x7f086c0d4860 0x7f086c0d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.038+0000 7f08723f7640 1 -- 172.21.15.83:0/2205944486 >> 172.21.15.83:0/2205944486 conn(0x7f086c0d3d40 msgr2=0x7f086c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:06.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.038+0000 7f08723f7640 1 -- 172.21.15.83:0/2205944486 shutdown_connections 2024-03-06T07:26:06.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.038+0000 7f08723f7640 1 -- 172.21.15.83:0/2205944486 wait complete. 2024-03-06T07:26:06.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.039+0000 7f08723f7640 1 Processor -- start 2024-03-06T07:26:06.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.040+0000 7f08723f7640 1 -- start start 2024-03-06T07:26:06.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.040+0000 7f08723f7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f086c0b0e30 0x7f086c0cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.040+0000 7f08723f7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f086c0d4860 0x7f086c0cd880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.040+0000 7f08723f7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f086c1614d0 0x7f086c156300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.040+0000 7f08723f7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f086c0b0910 con 0x7f086c1614d0 2024-03-06T07:26:06.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.040+0000 7f08723f7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f086c0cfc60 con 0x7f086c0d4860 2024-03-06T07:26:06.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.040+0000 7f08723f7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f086c0d2560 con 0x7f086c0b0e30 2024-03-06T07:26:06.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.040+0000 7f0870994640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f086c1614d0 0x7f086c156300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.040+0000 7f0870994640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f086c1614d0 0x7f086c156300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55870/0 (socket says 172.21.15.83:55870) 2024-03-06T07:26:06.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.040+0000 7f0870994640 1 -- 172.21.15.83:0/1095028660 learned_addr learned my addr 172.21.15.83:0/1095028660 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:06.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.041+0000 7f086b7fe640 1 --2- 172.21.15.83:0/1095028660 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f086c0d4860 0x7f086c0cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.040+0000 7f086bfff640 1 --2- 172.21.15.83:0/1095028660 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f086c0b0e30 0x7f086c0cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.041+0000 7f0870994640 1 -- 172.21.15.83:0/1095028660 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f086c0b0e30 msgr2=0x7f086c0cae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.041+0000 7f0870994640 1 --2- 172.21.15.83:0/1095028660 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f086c0b0e30 0x7f086c0cae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.041+0000 7f0870994640 1 -- 172.21.15.83:0/1095028660 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f086c0d4860 msgr2=0x7f086c0cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.041+0000 7f0870994640 1 --2- 172.21.15.83:0/1095028660 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f086c0d4860 0x7f086c0cd880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.041+0000 7f0870994640 1 -- 172.21.15.83:0/1095028660 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f086c156840 con 0x7f086c1614d0 2024-03-06T07:26:06.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.041+0000 7f0870994640 1 --2- 172.21.15.83:0/1095028660 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f086c1614d0 0x7f086c156300 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f0864036040 tx=0x7f0864004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.041+0000 7f0868ff9640 1 -- 172.21.15.83:0/1095028660 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0864043020 con 0x7f086c1614d0 2024-03-06T07:26:06.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.041+0000 7f0868ff9640 1 -- 172.21.15.83:0/1095028660 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f086403e450 con 0x7f086c1614d0 2024-03-06T07:26:06.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.041+0000 7f0868ff9640 1 -- 172.21.15.83:0/1095028660 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0864009510 con 0x7f086c1614d0 2024-03-06T07:26:06.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.041+0000 7f08723f7640 1 -- 172.21.15.83:0/1095028660 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f086c0cb4a0 con 0x7f086c1614d0 2024-03-06T07:26:06.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.041+0000 7f08723f7640 1 -- 172.21.15.83:0/1095028660 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f086c0cba60 con 0x7f086c1614d0 2024-03-06T07:26:06.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.042+0000 7f08723f7640 1 -- 172.21.15.83:0/1095028660 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f086c068ff0 con 0x7f086c1614d0 2024-03-06T07:26:06.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.043+0000 7f0868ff9640 1 -- 172.21.15.83:0/1095028660 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0864046020 con 0x7f086c1614d0 2024-03-06T07:26:06.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.043+0000 7f0868ff9640 1 --2- 172.21.15.83:0/1095028660 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f084c06b4d0 0x7f084c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.043+0000 7f086bfff640 1 --2- 172.21.15.83:0/1095028660 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f084c06b4d0 0x7f084c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.043+0000 7f0868ff9640 1 -- 172.21.15.83:0/1095028660 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f08640acaa0 con 0x7f086c1614d0 2024-03-06T07:26:06.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.044+0000 7f086bfff640 1 --2- 172.21.15.83:0/1095028660 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f084c06b4d0 0x7f084c06d990 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f085c006fd0 tx=0x7f085c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.048+0000 7f0868ff9640 1 -- 172.21.15.83:0/1095028660 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f086404d030 con 0x7f086c1614d0 2024-03-06T07:26:06.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.225+0000 7f08723f7640 1 -- 172.21.15.83:0/1095028660 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f086c0b5550 con 0x7f086c1614d0 2024-03-06T07:26:06.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.225+0000 7f0868ff9640 1 -- 172.21.15.83:0/1095028660 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v82) v1 ==== 104+0+11 (secure 0 0 0) 0x7f086407b920 con 0x7f086c1614d0 2024-03-06T07:26:06.228 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-03-06T07:26:06.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.227+0000 7f08723f7640 1 -- 172.21.15.83:0/1095028660 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f084c06b4d0 msgr2=0x7f084c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.227+0000 7f08723f7640 1 --2- 172.21.15.83:0/1095028660 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f084c06b4d0 0x7f084c06d990 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f085c006fd0 tx=0x7f085c008040 comp rx=0 tx=0).stop 2024-03-06T07:26:06.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.227+0000 7f08723f7640 1 -- 172.21.15.83:0/1095028660 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f086c1614d0 msgr2=0x7f086c156300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.227+0000 7f08723f7640 1 --2- 172.21.15.83:0/1095028660 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f086c1614d0 0x7f086c156300 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f0864036040 tx=0x7f0864004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:06.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.227+0000 7f08723f7640 1 -- 172.21.15.83:0/1095028660 shutdown_connections 2024-03-06T07:26:06.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.227+0000 7f08723f7640 1 --2- 172.21.15.83:0/1095028660 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f084c06b4d0 0x7f084c06d990 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.227+0000 7f08723f7640 1 --2- 172.21.15.83:0/1095028660 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f086c1614d0 0x7f086c156300 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.227+0000 7f08723f7640 1 --2- 172.21.15.83:0/1095028660 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f086c0d4860 0x7f086c0cd880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.227+0000 7f08723f7640 1 --2- 172.21.15.83:0/1095028660 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f086c0b0e30 0x7f086c0cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.227+0000 7f08723f7640 1 -- 172.21.15.83:0/1095028660 >> 172.21.15.83:0/1095028660 conn(0x7f086c0d3d40 msgr2=0x7f086c0d0c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:06.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.227+0000 7f08723f7640 1 -- 172.21.15.83:0/1095028660 shutdown_connections 2024-03-06T07:26:06.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.227+0000 7f08723f7640 1 -- 172.21.15.83:0/1095028660 wait complete. 2024-03-06T07:26:06.241 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:26:06.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.342+0000 7f09219e2640 1 Processor -- start 2024-03-06T07:26:06.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.342+0000 7f09219e2640 1 -- start start 2024-03-06T07:26:06.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.345+0000 7f09219e2640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f091c15c050 0x7f091c06f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.345+0000 7f09219e2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f091c06fb40 0x7f091c0cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.345+0000 7f09219e2640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f091c0cd4b0 con 0x7f091c0ccca0 2024-03-06T07:26:06.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.345+0000 7f09219e2640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f091c06ffb0 con 0x7f091c06fb40 2024-03-06T07:26:06.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.345+0000 7f09219e2640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f091c0cd790 con 0x7f091c15c050 2024-03-06T07:26:06.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.345+0000 7f09127fc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f091c06fb40 0x7f091c0cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.345+0000 7f091b7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f091c0ccca0 0x7f091c0cfae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:33138/0 (socket says 172.21.15.83:33138) 2024-03-06T07:26:06.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.345+0000 7f091b7fe640 1 -- 172.21.15.83:0/3597872113 learned_addr learned my addr 172.21.15.83:0/3597872113 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:06.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.345+0000 7f091affd640 1 --2- 172.21.15.83:0/3597872113 >> v2:172.21.15.183:3300/0 conn(0x7f091c15c050 0x7f091c06f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.345+0000 7f09127fc640 1 -- 172.21.15.83:0/3597872113 >> v2:172.21.15.183:3300/0 conn(0x7f091c15c050 msgr2=0x7f091c06f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:06.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.345+0000 7f09127fc640 1 --2- 172.21.15.83:0/3597872113 >> v2:172.21.15.183:3300/0 conn(0x7f091c15c050 0x7f091c06f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.345+0000 7f09127fc640 1 -- 172.21.15.83:0/3597872113 >> v1:172.21.15.83:6789/0 conn(0x7f091c0ccca0 legacy=0x7f091c0cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.345+0000 7f091a7fc640 1 -- 172.21.15.83:0/3597872113 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1244677701 0 0) 0x7f091c0cd4b0 con 0x7f091c0ccca0 2024-03-06T07:26:06.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.345+0000 7f09127fc640 1 -- 172.21.15.83:0/3597872113 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f091c0cda70 con 0x7f091c06fb40 2024-03-06T07:26:06.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.345+0000 7f091affd640 1 --2- 172.21.15.83:0/3597872113 >> v2:172.21.15.183:3300/0 conn(0x7f091c15c050 0x7f091c06f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:06.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.345+0000 7f09127fc640 1 --2- 172.21.15.83:0/3597872113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f091c06fb40 0x7f091c0cc760 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f091c070ef0 tx=0x7f090402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7fb66edd45b5b4d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.345+0000 7f091a7fc640 1 -- 172.21.15.83:0/3597872113 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09040315a0 con 0x7f091c06fb40 2024-03-06T07:26:06.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.345+0000 7f091a7fc640 1 -- 172.21.15.83:0/3597872113 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0904037070 con 0x7f091c06fb40 2024-03-06T07:26:06.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.345+0000 7f091a7fc640 1 -- 172.21.15.83:0/3597872113 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0904031960 con 0x7f091c06fb40 2024-03-06T07:26:06.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.346+0000 7f09219e2640 1 -- 172.21.15.83:0/3597872113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f091c06fb40 msgr2=0x7f091c0cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.346+0000 7f09219e2640 1 --2- 172.21.15.83:0/3597872113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f091c06fb40 0x7f091c0cc760 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f091c070ef0 tx=0x7f090402b840 comp rx=0 tx=0).stop 2024-03-06T07:26:06.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.346+0000 7f09219e2640 1 -- 172.21.15.83:0/3597872113 shutdown_connections 2024-03-06T07:26:06.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.346+0000 7f09219e2640 1 --2- 172.21.15.83:0/3597872113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f091c06fb40 0x7f091c0cc760 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.346+0000 7f09219e2640 1 --2- 172.21.15.83:0/3597872113 >> v2:172.21.15.183:3300/0 conn(0x7f091c15c050 0x7f091c06f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.346+0000 7f09219e2640 1 -- 172.21.15.83:0/3597872113 >> 172.21.15.83:0/3597872113 conn(0x7f091c0b3800 msgr2=0x7f091c0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:06.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.346+0000 7f09219e2640 1 -- 172.21.15.83:0/3597872113 shutdown_connections 2024-03-06T07:26:06.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.346+0000 7f09219e2640 1 -- 172.21.15.83:0/3597872113 wait complete. 2024-03-06T07:26:06.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.347+0000 7f09219e2640 1 Processor -- start 2024-03-06T07:26:06.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.347+0000 7f09219e2640 1 -- start start 2024-03-06T07:26:06.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.348+0000 7f09219e2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f091c06fb40 0x7f091c09f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.348+0000 7f09219e2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f091c0ccca0 0x7f091c1582f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.348+0000 7f09219e2640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f091c15c050 0x7f091c0cb030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.348+0000 7f09219e2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091c0d01f0 con 0x7f091c0ccca0 2024-03-06T07:26:06.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.348+0000 7f09219e2640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091c0cd130 con 0x7f091c15c050 2024-03-06T07:26:06.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.348+0000 7f09219e2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f091c0d0370 con 0x7f091c06fb40 2024-03-06T07:26:06.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.348+0000 7f09127fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f091c0ccca0 0x7f091c1582f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.348+0000 7f09127fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f091c0ccca0 0x7f091c1582f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55886/0 (socket says 172.21.15.83:55886) 2024-03-06T07:26:06.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.348+0000 7f09127fc640 1 -- 172.21.15.83:0/2888052662 learned_addr learned my addr 172.21.15.83:0/2888052662 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:06.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.348+0000 7f091affd640 1 --2- 172.21.15.83:0/2888052662 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f091c06fb40 0x7f091c09f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.348+0000 7f091b7fe640 1 --2- 172.21.15.83:0/2888052662 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f091c15c050 0x7f091c0cb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.348+0000 7f09127fc640 1 -- 172.21.15.83:0/2888052662 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f091c06fb40 msgr2=0x7f091c09f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.348+0000 7f09127fc640 1 --2- 172.21.15.83:0/2888052662 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f091c06fb40 0x7f091c09f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.348+0000 7f09127fc640 1 -- 172.21.15.83:0/2888052662 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f091c15c050 msgr2=0x7f091c0cb030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.348+0000 7f09127fc640 1 --2- 172.21.15.83:0/2888052662 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f091c15c050 0x7f091c0cb030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.348+0000 7f09127fc640 1 -- 172.21.15.83:0/2888052662 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f091c09d250 con 0x7f091c0ccca0 2024-03-06T07:26:06.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.348+0000 7f091affd640 1 --2- 172.21.15.83:0/2888052662 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f091c06fb40 0x7f091c09f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:06.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.348+0000 7f091b7fe640 1 --2- 172.21.15.83:0/2888052662 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f091c15c050 0x7f091c0cb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:06.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.348+0000 7f09127fc640 1 --2- 172.21.15.83:0/2888052662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f091c0ccca0 0x7f091c1582f0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f0904009890 tx=0x7f09040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.348+0000 7f0918ff9640 1 -- 172.21.15.83:0/2888052662 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0904032840 con 0x7f091c0ccca0 2024-03-06T07:26:06.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.349+0000 7f0918ff9640 1 -- 172.21.15.83:0/2888052662 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09040329e0 con 0x7f091c0ccca0 2024-03-06T07:26:06.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.349+0000 7f0918ff9640 1 -- 172.21.15.83:0/2888052662 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0904032dd0 con 0x7f091c0ccca0 2024-03-06T07:26:06.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.349+0000 7f09219e2640 1 -- 172.21.15.83:0/2888052662 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f091c09d530 con 0x7f091c0ccca0 2024-03-06T07:26:06.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.349+0000 7f09219e2640 1 -- 172.21.15.83:0/2888052662 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f091c09daf0 con 0x7f091c0ccca0 2024-03-06T07:26:06.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.350+0000 7f09219e2640 1 -- 172.21.15.83:0/2888052662 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f091c0b59f0 con 0x7f091c0ccca0 2024-03-06T07:26:06.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.350+0000 7f0918ff9640 1 -- 172.21.15.83:0/2888052662 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f090403b3f0 con 0x7f091c0ccca0 2024-03-06T07:26:06.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.351+0000 7f0918ff9640 1 --2- 172.21.15.83:0/2888052662 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f08fc06b4d0 0x7f08fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.351+0000 7f091affd640 1 --2- 172.21.15.83:0/2888052662 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f08fc06b4d0 0x7f08fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.351+0000 7f0918ff9640 1 -- 172.21.15.83:0/2888052662 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f090402f030 con 0x7f091c0ccca0 2024-03-06T07:26:06.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.351+0000 7f091affd640 1 --2- 172.21.15.83:0/2888052662 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f08fc06b4d0 0x7f08fc06d990 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f090c00aa30 tx=0x7f090c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.355+0000 7f0918ff9640 1 -- 172.21.15.83:0/2888052662 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0904040070 con 0x7f091c0ccca0 2024-03-06T07:26:06.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.528+0000 7f09219e2640 1 -- 172.21.15.83:0/2888052662 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f091c068ff0 con 0x7f091c0ccca0 2024-03-06T07:26:06.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.529+0000 7f0918ff9640 1 -- 172.21.15.83:0/2888052662 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v82) v1 ==== 100+0+11 (secure 0 0 0) 0x7f090407b090 con 0x7f091c0ccca0 2024-03-06T07:26:06.531 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 64 2024-03-06T07:26:06.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.531+0000 7f09219e2640 1 -- 172.21.15.83:0/2888052662 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f08fc06b4d0 msgr2=0x7f08fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.531+0000 7f09219e2640 1 --2- 172.21.15.83:0/2888052662 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f08fc06b4d0 0x7f08fc06d990 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f090c00aa30 tx=0x7f090c008040 comp rx=0 tx=0).stop 2024-03-06T07:26:06.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.531+0000 7f09219e2640 1 -- 172.21.15.83:0/2888052662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f091c0ccca0 msgr2=0x7f091c1582f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.531+0000 7f09219e2640 1 --2- 172.21.15.83:0/2888052662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f091c0ccca0 0x7f091c1582f0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f0904009890 tx=0x7f09040046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.531+0000 7f09219e2640 1 -- 172.21.15.83:0/2888052662 shutdown_connections 2024-03-06T07:26:06.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.531+0000 7f09219e2640 1 --2- 172.21.15.83:0/2888052662 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f08fc06b4d0 0x7f08fc06d990 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.531+0000 7f09219e2640 1 --2- 172.21.15.83:0/2888052662 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f091c15c050 0x7f091c0cb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.531+0000 7f09219e2640 1 --2- 172.21.15.83:0/2888052662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f091c0ccca0 0x7f091c1582f0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.531+0000 7f09219e2640 1 --2- 172.21.15.83:0/2888052662 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f091c06fb40 0x7f091c09f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.531+0000 7f09219e2640 1 -- 172.21.15.83:0/2888052662 >> 172.21.15.83:0/2888052662 conn(0x7f091c0b3800 msgr2=0x7f091c070680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:06.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.531+0000 7f09219e2640 1 -- 172.21.15.83:0/2888052662 shutdown_connections 2024-03-06T07:26:06.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.531+0000 7f09219e2640 1 -- 172.21.15.83:0/2888052662 wait complete. 2024-03-06T07:26:06.544 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:06.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.679+0000 7f280fc88640 1 Processor -- start 2024-03-06T07:26:06.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.679+0000 7f280fc88640 1 -- start start 2024-03-06T07:26:06.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.680+0000 7f280fc88640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f28080d22b0 0x7f28080d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.680+0000 7f280fc88640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f28080b41f0 0x7f28080b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.680+0000 7f280fc88640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28080b3430 con 0x7f280815dfe0 2024-03-06T07:26:06.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.680+0000 7f280fc88640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28080d2c50 con 0x7f28080b41f0 2024-03-06T07:26:06.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.680+0000 7f280fc88640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28080b31e0 con 0x7f28080d22b0 2024-03-06T07:26:06.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.680+0000 7f280da24640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f280815dfe0 0x7f28080cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60324/0 (socket says 172.21.15.83:60324) 2024-03-06T07:26:06.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.680+0000 7f280da24640 1 -- 172.21.15.83:0/1585718385 learned_addr learned my addr 172.21.15.83:0/1585718385 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:06.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.681+0000 7f280d223640 1 --2- 172.21.15.83:0/1585718385 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f28080d22b0 0x7f28080d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.681+0000 7f280e225640 1 --2- 172.21.15.83:0/1585718385 >> v2:172.21.15.183:3300/0 conn(0x7f28080b41f0 0x7f28080b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.681+0000 7f280ca22640 1 -- 172.21.15.83:0/1585718385 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2688156415 0 0) 0x7f28080b3430 con 0x7f280815dfe0 2024-03-06T07:26:06.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.681+0000 7f280ca22640 1 -- 172.21.15.83:0/1585718385 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27f4003520 con 0x7f280815dfe0 2024-03-06T07:26:06.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.681+0000 7f280ca22640 1 -- 172.21.15.83:0/1585718385 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2971241514 0 0) 0x7f27f4003520 con 0x7f280815dfe0 2024-03-06T07:26:06.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.681+0000 7f280ca22640 1 -- 172.21.15.83:0/1585718385 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f28080d22b0 msgr2=0x7f28080d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:06.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.681+0000 7f280ca22640 1 --2- 172.21.15.83:0/1585718385 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f28080d22b0 0x7f28080d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.681+0000 7f280ca22640 1 -- 172.21.15.83:0/1585718385 >> v2:172.21.15.183:3300/0 conn(0x7f28080b41f0 msgr2=0x7f28080b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:06.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.681+0000 7f280ca22640 1 --2- 172.21.15.83:0/1585718385 >> v2:172.21.15.183:3300/0 conn(0x7f28080b41f0 0x7f28080b4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.681+0000 7f280ca22640 1 -- 172.21.15.83:0/1585718385 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28080b3710 con 0x7f280815dfe0 2024-03-06T07:26:06.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.681+0000 7f280ca22640 1 -- 172.21.15.83:0/1585718385 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f2800001670 con 0x7f280815dfe0 2024-03-06T07:26:06.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.681+0000 7f280ca22640 1 -- 172.21.15.83:0/1585718385 >> v1:172.21.15.83:6789/0 conn(0x7f280815dfe0 legacy=0x7f28080cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.681+0000 7f280ca22640 1 --2- 172.21.15.83:0/1585718385 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f28080d22b0 0x7f27f40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.681+0000 7f280ca22640 1 --2- 172.21.15.83:0/1585718385 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27f4003f30 0x7f27f4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.681+0000 7f280ca22640 1 --2- 172.21.15.83:0/1585718385 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f27f4004d40 0x7f27f40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.681+0000 7f280ca22640 1 -- 172.21.15.83:0/1585718385 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28080b31e0 con 0x7f27f4003f30 2024-03-06T07:26:06.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.681+0000 7f280ca22640 1 -- 172.21.15.83:0/1585718385 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f400b9c0 con 0x7f27f4004d40 2024-03-06T07:26:06.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.681+0000 7f280ca22640 1 -- 172.21.15.83:0/1585718385 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27f400bb40 con 0x7f28080d22b0 2024-03-06T07:26:06.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.681+0000 7f280d223640 1 --2- 172.21.15.83:0/1585718385 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27f4003f30 0x7f27f4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.681+0000 7f280da24640 1 --2- 172.21.15.83:0/1585718385 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f28080d22b0 0x7f27f40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.681+0000 7f280e225640 1 --2- 172.21.15.83:0/1585718385 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f27f4004d40 0x7f27f40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.681+0000 7f280d223640 1 -- 172.21.15.83:0/1585718385 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f28080d22b0 msgr2=0x7f27f40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:06.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.681+0000 7f280d223640 1 --2- 172.21.15.83:0/1585718385 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f28080d22b0 0x7f27f40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.681+0000 7f280d223640 1 -- 172.21.15.83:0/1585718385 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f27f4004d40 msgr2=0x7f27f40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:06.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.681+0000 7f280d223640 1 --2- 172.21.15.83:0/1585718385 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f27f4004d40 0x7f27f40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.681+0000 7f280d223640 1 -- 172.21.15.83:0/1585718385 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f4003520 con 0x7f27f4003f30 2024-03-06T07:26:06.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.681+0000 7f280da24640 1 --2- 172.21.15.83:0/1585718385 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f28080d22b0 0x7f27f40039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:06.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.682+0000 7f280d223640 1 --2- 172.21.15.83:0/1585718385 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27f4003f30 0x7f27f4004800 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f27f8009ac0 tx=0x7f27f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26d7d968aea5fcd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.682+0000 7f280ca22640 1 -- 172.21.15.83:0/1585718385 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f8039070 con 0x7f27f4003f30 2024-03-06T07:26:06.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.682+0000 7f280ca22640 1 -- 172.21.15.83:0/1585718385 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27f802c6b0 con 0x7f27f4003f30 2024-03-06T07:26:06.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.682+0000 7f280ca22640 1 -- 172.21.15.83:0/1585718385 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f8009690 con 0x7f27f4003f30 2024-03-06T07:26:06.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.683+0000 7f280fc88640 1 -- 172.21.15.83:0/1585718385 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27f4003f30 msgr2=0x7f27f4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.683+0000 7f280fc88640 1 --2- 172.21.15.83:0/1585718385 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27f4003f30 0x7f27f4004800 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f27f8009ac0 tx=0x7f27f4005830 comp rx=0 tx=0).stop 2024-03-06T07:26:06.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.683+0000 7f280e225640 1 -- 172.21.15.83:0/1585718385 reap_dead start 2024-03-06T07:26:06.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.683+0000 7f280fc88640 1 -- 172.21.15.83:0/1585718385 shutdown_connections 2024-03-06T07:26:06.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.683+0000 7f280fc88640 1 -- 172.21.15.83:0/1585718385 >> 172.21.15.83:0/1585718385 conn(0x7f28080d3cd0 msgr2=0x7f28080d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:06.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.683+0000 7f280fc88640 1 -- 172.21.15.83:0/1585718385 shutdown_connections 2024-03-06T07:26:06.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.683+0000 7f280fc88640 1 -- 172.21.15.83:0/1585718385 wait complete. 2024-03-06T07:26:06.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.684+0000 7f280fc88640 1 Processor -- start 2024-03-06T07:26:06.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.684+0000 7f280fc88640 1 -- start start 2024-03-06T07:26:06.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.685+0000 7f280fc88640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f27f4003f30 0x7f280815d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.685+0000 7f280fc88640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f280809f0b0 0x7f28080cd550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.685+0000 7f280fc88640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f280809f530 0x7f28080cda90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.685+0000 7f280fc88640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28080cfa00 con 0x7f280809f530 2024-03-06T07:26:06.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.685+0000 7f280fc88640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28080b3d60 con 0x7f27f4003f30 2024-03-06T07:26:06.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.685+0000 7f280fc88640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28080d1c70 con 0x7f280809f0b0 2024-03-06T07:26:06.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.685+0000 7f280e225640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f280809f530 0x7f28080cda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.685+0000 7f280e225640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f280809f530 0x7f28080cda90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52748/0 (socket says 172.21.15.83:52748) 2024-03-06T07:26:06.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.685+0000 7f280e225640 1 -- 172.21.15.83:0/3548629808 learned_addr learned my addr 172.21.15.83:0/3548629808 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:06.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.685+0000 7f280d223640 1 --2- 172.21.15.83:0/3548629808 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f280809f0b0 0x7f28080cd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.685+0000 7f280da24640 1 --2- 172.21.15.83:0/3548629808 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f27f4003f30 0x7f280815d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.685+0000 7f280e225640 1 -- 172.21.15.83:0/3548629808 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f280809f0b0 msgr2=0x7f28080cd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.685+0000 7f280e225640 1 --2- 172.21.15.83:0/3548629808 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f280809f0b0 0x7f28080cd550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.685+0000 7f280e225640 1 -- 172.21.15.83:0/3548629808 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f27f4003f30 msgr2=0x7f280815d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.685+0000 7f280e225640 1 --2- 172.21.15.83:0/3548629808 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f27f4003f30 0x7f280815d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.685+0000 7f280e225640 1 -- 172.21.15.83:0/3548629808 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28080caec0 con 0x7f280809f530 2024-03-06T07:26:06.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.685+0000 7f280da24640 1 --2- 172.21.15.83:0/3548629808 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f27f4003f30 0x7f280815d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:06.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.685+0000 7f280e225640 1 --2- 172.21.15.83:0/3548629808 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f280809f530 0x7f28080cda90 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f280808c940 tx=0x7f28040115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.686+0000 7f27fe7fc640 1 -- 172.21.15.83:0/3548629808 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2804011e10 con 0x7f280809f530 2024-03-06T07:26:06.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.686+0000 7f27fe7fc640 1 -- 172.21.15.83:0/3548629808 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f280401b070 con 0x7f280809f530 2024-03-06T07:26:06.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.686+0000 7f27fe7fc640 1 -- 172.21.15.83:0/3548629808 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2804003990 con 0x7f280809f530 2024-03-06T07:26:06.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.686+0000 7f280fc88640 1 -- 172.21.15.83:0/3548629808 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28080cb160 con 0x7f280809f530 2024-03-06T07:26:06.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.686+0000 7f280fc88640 1 -- 172.21.15.83:0/3548629808 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28080cb720 con 0x7f280809f530 2024-03-06T07:26:06.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.687+0000 7f280fc88640 1 -- 172.21.15.83:0/3548629808 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f280804f330 con 0x7f280809f530 2024-03-06T07:26:06.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.687+0000 7f27fe7fc640 1 -- 172.21.15.83:0/3548629808 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2804005d10 con 0x7f280809f530 2024-03-06T07:26:06.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.687+0000 7f27fe7fc640 1 --2- 172.21.15.83:0/3548629808 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f27ec06b500 0x7f27ec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.688+0000 7f27fe7fc640 1 -- 172.21.15.83:0/3548629808 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f280408a760 con 0x7f280809f530 2024-03-06T07:26:06.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.688+0000 7f280da24640 1 --2- 172.21.15.83:0/3548629808 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f27ec06b500 0x7f27ec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.688+0000 7f280da24640 1 --2- 172.21.15.83:0/3548629808 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f27ec06b500 0x7f27ec06d9c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f2800001050 tx=0x7f2800009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.692+0000 7f27fe7fc640 1 -- 172.21.15.83:0/3548629808 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2804025020 con 0x7f280809f530 2024-03-06T07:26:06.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.908+0000 7f280fc88640 1 -- 172.21.15.83:0/3548629808 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f280806bfe0 con 0x7f280809f530 2024-03-06T07:26:06.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.908+0000 7f27fe7fc640 1 -- 172.21.15.83:0/3548629808 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f28040595e0 con 0x7f280809f530 2024-03-06T07:26:06.910 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:06.910 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":39,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/2525949760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":2525949760},{"type":"v1","addr":"172.21.15.183:6834","nonce":2525949760}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35},{"gid":6262,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2104447869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2104447869},{"type":"v1","addr":"172.21.15.183:6835","nonce":2104447869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6954,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/779621621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":779621621},{"type":"v1","addr":"172.21.15.83:6836","nonce":779621621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:54.631963+0000","modified":"2024-03-06T07:25:56.637047+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6957":{"gid":6957,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/3056214366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":3056214366},{"type":"v1","addr":"172.21.15.83:6837","nonce":3056214366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6957,"qdb_cluster":[6957]},"id":6}]} 2024-03-06T07:26:06.911 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 39 2024-03-06T07:26:06.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.910+0000 7f280fc88640 1 -- 172.21.15.83:0/3548629808 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f27ec06b500 msgr2=0x7f27ec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.910+0000 7f280fc88640 1 --2- 172.21.15.83:0/3548629808 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f27ec06b500 0x7f27ec06d9c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f2800001050 tx=0x7f2800009040 comp rx=0 tx=0).stop 2024-03-06T07:26:06.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.910+0000 7f280fc88640 1 -- 172.21.15.83:0/3548629808 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f280809f530 msgr2=0x7f28080cda90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.910+0000 7f280fc88640 1 --2- 172.21.15.83:0/3548629808 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f280809f530 0x7f28080cda90 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f280808c940 tx=0x7f28040115a0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.910+0000 7f280fc88640 1 -- 172.21.15.83:0/3548629808 shutdown_connections 2024-03-06T07:26:06.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.910+0000 7f280fc88640 1 --2- 172.21.15.83:0/3548629808 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f27ec06b500 0x7f27ec06d9c0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.910+0000 7f280fc88640 1 --2- 172.21.15.83:0/3548629808 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f280809f530 0x7f28080cda90 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.910+0000 7f280fc88640 1 --2- 172.21.15.83:0/3548629808 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f280809f0b0 0x7f28080cd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.910+0000 7f280fc88640 1 --2- 172.21.15.83:0/3548629808 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f27f4003f30 0x7f280815d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.910+0000 7f280fc88640 1 -- 172.21.15.83:0/3548629808 >> 172.21.15.83:0/3548629808 conn(0x7f28080d3cd0 msgr2=0x7f28080b0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:06.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.910+0000 7f280fc88640 1 -- 172.21.15.83:0/3548629808 shutdown_connections 2024-03-06T07:26:06.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:06.910+0000 7f280fc88640 1 -- 172.21.15.83:0/3548629808 wait complete. 2024-03-06T07:26:06.923 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:07.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.022+0000 7f63616bb640 1 Processor -- start 2024-03-06T07:26:07.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.022+0000 7f63616bb640 1 -- start start 2024-03-06T07:26:07.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.023+0000 7f63616bb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f635c0d0a80 0x7f635c0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.023+0000 7f63616bb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f635c0d25d0 con 0x7f635c0b4170 2024-03-06T07:26:07.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.023+0000 7f63616bb640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f635c0d2870 con 0x7f635c0d4760 2024-03-06T07:26:07.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.023+0000 7f63616bb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f635c0d4bf0 con 0x7f635c0d0a80 2024-03-06T07:26:07.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.023+0000 7f635affd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f635c0b4170 0x7f635c0d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60326/0 (socket says 172.21.15.83:60326) 2024-03-06T07:26:07.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.023+0000 7f635affd640 1 -- 172.21.15.83:0/2670579839 learned_addr learned my addr 172.21.15.83:0/2670579839 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:07.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.023+0000 7f635b7fe640 1 --2- 172.21.15.83:0/2670579839 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f635c0d0a80 0x7f635c0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.024+0000 7f6359ffb640 1 -- 172.21.15.83:0/2670579839 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3497859523 0 0) 0x7f635c0d25d0 con 0x7f635c0b4170 2024-03-06T07:26:07.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.024+0000 7f6359ffb640 1 -- 172.21.15.83:0/2670579839 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6344003520 con 0x7f635c0b4170 2024-03-06T07:26:07.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.024+0000 7f635b7fe640 1 -- 172.21.15.83:0/2670579839 >> v1:172.21.15.183:6789/0 conn(0x7f635c0d4760 legacy=0x7f635c0cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.024+0000 7f635b7fe640 1 -- 172.21.15.83:0/2670579839 >> v1:172.21.15.83:6789/0 conn(0x7f635c0b4170 legacy=0x7f635c0d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.024+0000 7f635b7fe640 1 -- 172.21.15.83:0/2670579839 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f635c0d2b50 con 0x7f635c0d0a80 2024-03-06T07:26:07.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.025+0000 7f635b7fe640 1 --2- 172.21.15.83:0/2670579839 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f635c0d0a80 0x7f635c0b6a90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f634c009dc0 tx=0x7f634c02bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bef45ef53f3de487 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:07.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.025+0000 7f6359ffb640 1 -- 172.21.15.83:0/2670579839 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f634c039070 con 0x7f635c0d0a80 2024-03-06T07:26:07.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.025+0000 7f6359ffb640 1 -- 172.21.15.83:0/2670579839 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f634c02ccd0 con 0x7f635c0d0a80 2024-03-06T07:26:07.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.025+0000 7f6359ffb640 1 -- 172.21.15.83:0/2670579839 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f634c034640 con 0x7f635c0d0a80 2024-03-06T07:26:07.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.025+0000 7f63616bb640 1 -- 172.21.15.83:0/2670579839 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f635c0d0a80 msgr2=0x7f635c0b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.025+0000 7f63616bb640 1 --2- 172.21.15.83:0/2670579839 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f635c0d0a80 0x7f635c0b6a90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f634c009dc0 tx=0x7f634c02bf20 comp rx=0 tx=0).stop 2024-03-06T07:26:07.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.025+0000 7f63616bb640 1 -- 172.21.15.83:0/2670579839 shutdown_connections 2024-03-06T07:26:07.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.025+0000 7f63616bb640 1 --2- 172.21.15.83:0/2670579839 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f635c0d0a80 0x7f635c0b6a90 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.025+0000 7f63616bb640 1 -- 172.21.15.83:0/2670579839 >> 172.21.15.83:0/2670579839 conn(0x7f635c0d3c40 msgr2=0x7f635c0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:07.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.025+0000 7f63616bb640 1 -- 172.21.15.83:0/2670579839 shutdown_connections 2024-03-06T07:26:07.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.025+0000 7f63616bb640 1 -- 172.21.15.83:0/2670579839 wait complete. 2024-03-06T07:26:07.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.027+0000 7f63616bb640 1 Processor -- start 2024-03-06T07:26:07.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.027+0000 7f63616bb640 1 -- start start 2024-03-06T07:26:07.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.027+0000 7f63616bb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f635c0b4170 0x7f635c0b9250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.027+0000 7f63616bb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f635c0d0a80 0x7f635c156f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.027+0000 7f63616bb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f635c157470 0x7f635c0c9700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.027+0000 7f63616bb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635c0b4dd0 con 0x7f635c0d0a80 2024-03-06T07:26:07.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.027+0000 7f63616bb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635c0ce9e0 con 0x7f635c0b4170 2024-03-06T07:26:07.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.027+0000 7f63616bb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f635c0b4f50 con 0x7f635c157470 2024-03-06T07:26:07.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.028+0000 7f635a7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f635c0d0a80 0x7f635c156f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.028+0000 7f635a7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f635c0d0a80 0x7f635c156f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52756/0 (socket says 172.21.15.83:52756) 2024-03-06T07:26:07.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.028+0000 7f635a7fc640 1 -- 172.21.15.83:0/2494351721 learned_addr learned my addr 172.21.15.83:0/2494351721 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:07.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.028+0000 7f635b7fe640 1 --2- 172.21.15.83:0/2494351721 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f635c157470 0x7f635c0c9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.028+0000 7f635affd640 1 --2- 172.21.15.83:0/2494351721 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f635c0b4170 0x7f635c0b9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.028+0000 7f635a7fc640 1 -- 172.21.15.83:0/2494351721 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f635c157470 msgr2=0x7f635c0c9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.028+0000 7f635a7fc640 1 --2- 172.21.15.83:0/2494351721 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f635c157470 0x7f635c0c9700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.028+0000 7f635a7fc640 1 -- 172.21.15.83:0/2494351721 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f635c0b4170 msgr2=0x7f635c0b9250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.028+0000 7f635a7fc640 1 --2- 172.21.15.83:0/2494351721 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f635c0b4170 0x7f635c0b9250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.028+0000 7f635a7fc640 1 -- 172.21.15.83:0/2494351721 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f635c0c8940 con 0x7f635c0d0a80 2024-03-06T07:26:07.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.028+0000 7f635a7fc640 1 --2- 172.21.15.83:0/2494351721 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f635c0d0a80 0x7f635c156f30 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f635000c3c0 tx=0x7f635000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:07.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.028+0000 7f63437fe640 1 -- 172.21.15.83:0/2494351721 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6350017070 con 0x7f635c0d0a80 2024-03-06T07:26:07.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.028+0000 7f63437fe640 1 -- 172.21.15.83:0/2494351721 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63500052f0 con 0x7f635c0d0a80 2024-03-06T07:26:07.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.028+0000 7f63437fe640 1 -- 172.21.15.83:0/2494351721 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6350003830 con 0x7f635c0d0a80 2024-03-06T07:26:07.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.028+0000 7f63616bb640 1 -- 172.21.15.83:0/2494351721 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f635c0c8c20 con 0x7f635c0d0a80 2024-03-06T07:26:07.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.028+0000 7f63616bb640 1 -- 172.21.15.83:0/2494351721 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f635c0c91e0 con 0x7f635c0d0a80 2024-03-06T07:26:07.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.029+0000 7f63616bb640 1 -- 172.21.15.83:0/2494351721 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f635c0c93e0 con 0x7f635c0d0a80 2024-03-06T07:26:07.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.030+0000 7f63437fe640 1 -- 172.21.15.83:0/2494351721 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f635001a070 con 0x7f635c0d0a80 2024-03-06T07:26:07.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.030+0000 7f63437fe640 1 --2- 172.21.15.83:0/2494351721 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f634806b4d0 0x7f634806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.030+0000 7f63437fe640 1 -- 172.21.15.83:0/2494351721 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6350089410 con 0x7f635c0d0a80 2024-03-06T07:26:07.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.030+0000 7f635affd640 1 --2- 172.21.15.83:0/2494351721 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f634806b4d0 0x7f634806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.031+0000 7f635affd640 1 --2- 172.21.15.83:0/2494351721 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f634806b4d0 0x7f634806d990 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f63540011e0 tx=0x7f6354007d30 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:07.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.035+0000 7f63437fe640 1 -- 172.21.15.83:0/2494351721 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f635c0c93e0 con 0x7f635c0d0a80 2024-03-06T07:26:07.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.206+0000 7f63616bb640 1 -- 172.21.15.83:0/2494351721 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f635c04f330 con 0x7f635c0d0a80 2024-03-06T07:26:07.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.208+0000 7f63437fe640 1 -- 172.21.15.83:0/2494351721 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16672 (secure 0 0 0) 0x7f6350057cd0 con 0x7f635c0d0a80 2024-03-06T07:26:07.210 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:07.210 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:25:56.279928+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:52.856525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:53.598434+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:07.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.210+0000 7f63616bb640 1 -- 172.21.15.83:0/2494351721 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f634806b4d0 msgr2=0x7f634806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.210+0000 7f63616bb640 1 --2- 172.21.15.83:0/2494351721 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f634806b4d0 0x7f634806d990 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f63540011e0 tx=0x7f6354007d30 comp rx=0 tx=0).stop 2024-03-06T07:26:07.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.210+0000 7f63616bb640 1 -- 172.21.15.83:0/2494351721 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f635c0d0a80 msgr2=0x7f635c156f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.210+0000 7f63616bb640 1 --2- 172.21.15.83:0/2494351721 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f635c0d0a80 0x7f635c156f30 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f635000c3c0 tx=0x7f635000c880 comp rx=0 tx=0).stop 2024-03-06T07:26:07.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.210+0000 7f63616bb640 1 -- 172.21.15.83:0/2494351721 shutdown_connections 2024-03-06T07:26:07.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.210+0000 7f63616bb640 1 --2- 172.21.15.83:0/2494351721 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f634806b4d0 0x7f634806d990 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.210+0000 7f63616bb640 1 --2- 172.21.15.83:0/2494351721 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f635c157470 0x7f635c0c9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.210+0000 7f63616bb640 1 --2- 172.21.15.83:0/2494351721 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f635c0d0a80 0x7f635c156f30 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.210+0000 7f63616bb640 1 --2- 172.21.15.83:0/2494351721 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f635c0b4170 0x7f635c0b9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.210+0000 7f63616bb640 1 -- 172.21.15.83:0/2494351721 >> 172.21.15.83:0/2494351721 conn(0x7f635c0d3c40 msgr2=0x7f635c0ced70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:07.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.210+0000 7f63616bb640 1 -- 172.21.15.83:0/2494351721 shutdown_connections 2024-03-06T07:26:07.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.210+0000 7f63616bb640 1 -- 172.21.15.83:0/2494351721 wait complete. 2024-03-06T07:26:07.223 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:26:07.223 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:07.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.322+0000 7f341fa08640 1 Processor -- start 2024-03-06T07:26:07.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.323+0000 7f341fa08640 1 -- start start 2024-03-06T07:26:07.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.323+0000 7f341fa08640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f341815dfe0 0x7f34180cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.323+0000 7f341fa08640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34180b3430 con 0x7f34180cf830 2024-03-06T07:26:07.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.323+0000 7f341fa08640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34180b3710 con 0x7f34180d29c0 2024-03-06T07:26:07.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.323+0000 7f341fa08640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34180b31e0 con 0x7f341815dfe0 2024-03-06T07:26:07.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.323+0000 7f341cfa3640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f34180cf830 0x7f34180d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60342/0 (socket says 172.21.15.83:60342) 2024-03-06T07:26:07.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.323+0000 7f341cfa3640 1 -- 172.21.15.83:0/995777898 learned_addr learned my addr 172.21.15.83:0/995777898 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:07.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.324+0000 7f341d7a4640 1 --2- 172.21.15.83:0/995777898 >> v2:172.21.15.183:3300/0 conn(0x7f341815dfe0 0x7f34180cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.324+0000 7f340ffff640 1 -- 172.21.15.83:0/995777898 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 136105507 0 0) 0x7f34180b3430 con 0x7f34180cf830 2024-03-06T07:26:07.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.324+0000 7f340ffff640 1 -- 172.21.15.83:0/995777898 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3404003520 con 0x7f34180cf830 2024-03-06T07:26:07.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.324+0000 7f340ffff640 1 -- 172.21.15.83:0/995777898 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2748483121 0 0) 0x7f34180b3710 con 0x7f34180d29c0 2024-03-06T07:26:07.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.324+0000 7f340ffff640 1 -- 172.21.15.83:0/995777898 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34180b3430 con 0x7f34180d29c0 2024-03-06T07:26:07.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.324+0000 7f341d7a4640 1 -- 172.21.15.83:0/995777898 >> v1:172.21.15.183:6789/0 conn(0x7f34180d29c0 legacy=0x7f34180b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.324+0000 7f341d7a4640 1 -- 172.21.15.83:0/995777898 >> v1:172.21.15.83:6789/0 conn(0x7f34180cf830 legacy=0x7f34180d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.324+0000 7f341d7a4640 1 -- 172.21.15.83:0/995777898 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34180b39f0 con 0x7f341815dfe0 2024-03-06T07:26:07.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.325+0000 7f341d7a4640 1 --2- 172.21.15.83:0/995777898 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f341815dfe0 0x7f34180cf2f0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f34180b3360 tx=0x7f341402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97bb72c4c7ec6321 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:07.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.325+0000 7f340ffff640 1 -- 172.21.15.83:0/995777898 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3414034070 con 0x7f341815dfe0 2024-03-06T07:26:07.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.325+0000 7f340ffff640 1 -- 172.21.15.83:0/995777898 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3414030610 con 0x7f341815dfe0 2024-03-06T07:26:07.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.325+0000 7f340ffff640 1 -- 172.21.15.83:0/995777898 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34140309e0 con 0x7f341815dfe0 2024-03-06T07:26:07.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.326+0000 7f341fa08640 1 -- 172.21.15.83:0/995777898 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f341815dfe0 msgr2=0x7f34180cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.326+0000 7f341fa08640 1 --2- 172.21.15.83:0/995777898 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f341815dfe0 0x7f34180cf2f0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f34180b3360 tx=0x7f341402b960 comp rx=0 tx=0).stop 2024-03-06T07:26:07.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.326+0000 7f341fa08640 1 -- 172.21.15.83:0/995777898 shutdown_connections 2024-03-06T07:26:07.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.326+0000 7f341fa08640 1 --2- 172.21.15.83:0/995777898 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f341815dfe0 0x7f34180cf2f0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.326+0000 7f341fa08640 1 -- 172.21.15.83:0/995777898 >> 172.21.15.83:0/995777898 conn(0x7f34180d3cd0 msgr2=0x7f34180d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:07.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.326+0000 7f341fa08640 1 -- 172.21.15.83:0/995777898 shutdown_connections 2024-03-06T07:26:07.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.326+0000 7f341fa08640 1 -- 172.21.15.83:0/995777898 wait complete. 2024-03-06T07:26:07.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.327+0000 7f341fa08640 1 Processor -- start 2024-03-06T07:26:07.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.327+0000 7f341fa08640 1 -- start start 2024-03-06T07:26:07.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.328+0000 7f341fa08640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f34180cf830 0x7f34180b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.328+0000 7f341fa08640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34180d29c0 0x7f34180b8140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.328+0000 7f341fa08640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f341815dfe0 0x7f34180b1420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.328+0000 7f341fa08640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f341815e3c0 con 0x7f34180d29c0 2024-03-06T07:26:07.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.328+0000 7f341fa08640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34180b3d60 con 0x7f341815dfe0 2024-03-06T07:26:07.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.328+0000 7f341fa08640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f341815e540 con 0x7f34180cf830 2024-03-06T07:26:07.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.328+0000 7f341cfa3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34180d29c0 0x7f34180b8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.328+0000 7f341cfa3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34180d29c0 0x7f34180b8140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52760/0 (socket says 172.21.15.83:52760) 2024-03-06T07:26:07.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.328+0000 7f341cfa3640 1 -- 172.21.15.83:0/2566790818 learned_addr learned my addr 172.21.15.83:0/2566790818 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:07.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.328+0000 7f341d7a4640 1 --2- 172.21.15.83:0/2566790818 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f34180cf830 0x7f34180b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.328+0000 7f341dfa5640 1 --2- 172.21.15.83:0/2566790818 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f341815dfe0 0x7f34180b1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.328+0000 7f341cfa3640 1 -- 172.21.15.83:0/2566790818 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f34180cf830 msgr2=0x7f34180b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.328+0000 7f341cfa3640 1 --2- 172.21.15.83:0/2566790818 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f34180cf830 0x7f34180b68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.328+0000 7f341cfa3640 1 -- 172.21.15.83:0/2566790818 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f341815dfe0 msgr2=0x7f34180b1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.328+0000 7f341cfa3640 1 --2- 172.21.15.83:0/2566790818 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f341815dfe0 0x7f34180b1420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.328+0000 7f341cfa3640 1 -- 172.21.15.83:0/2566790818 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34180b6e10 con 0x7f34180d29c0 2024-03-06T07:26:07.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.328+0000 7f341dfa5640 1 --2- 172.21.15.83:0/2566790818 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f341815dfe0 0x7f34180b1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:07.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.329+0000 7f341cfa3640 1 --2- 172.21.15.83:0/2566790818 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34180d29c0 0x7f34180b8140 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f3408001eb0 tx=0x7f340800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:07.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.329+0000 7f340dffb640 1 -- 172.21.15.83:0/2566790818 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3408017070 con 0x7f34180d29c0 2024-03-06T07:26:07.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.329+0000 7f340dffb640 1 -- 172.21.15.83:0/2566790818 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3408005ce0 con 0x7f34180d29c0 2024-03-06T07:26:07.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.329+0000 7f341fa08640 1 -- 172.21.15.83:0/2566790818 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34180b8680 con 0x7f34180d29c0 2024-03-06T07:26:07.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.329+0000 7f340dffb640 1 -- 172.21.15.83:0/2566790818 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3408007270 con 0x7f34180d29c0 2024-03-06T07:26:07.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.329+0000 7f341fa08640 1 -- 172.21.15.83:0/2566790818 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34180b9100 con 0x7f34180d29c0 2024-03-06T07:26:07.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.330+0000 7f341fa08640 1 -- 172.21.15.83:0/2566790818 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f341804f330 con 0x7f34180d29c0 2024-03-06T07:26:07.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.330+0000 7f340dffb640 1 -- 172.21.15.83:0/2566790818 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f340801a070 con 0x7f34180d29c0 2024-03-06T07:26:07.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.331+0000 7f340dffb640 1 --2- 172.21.15.83:0/2566790818 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f33fc06b500 0x7f33fc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.331+0000 7f340dffb640 1 -- 172.21.15.83:0/2566790818 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3408089420 con 0x7f34180d29c0 2024-03-06T07:26:07.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.331+0000 7f341d7a4640 1 --2- 172.21.15.83:0/2566790818 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f33fc06b500 0x7f33fc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.331+0000 7f341d7a4640 1 --2- 172.21.15.83:0/2566790818 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f33fc06b500 0x7f33fc06d9c0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f3414036000 tx=0x7f341403b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:07.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.336+0000 7f340dffb640 1 -- 172.21.15.83:0/2566790818 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3408057c50 con 0x7f34180d29c0 2024-03-06T07:26:07.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.547+0000 7f341fa08640 1 -- 172.21.15.83:0/2566790818 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f341806bd20 con 0x7f34180d29c0 2024-03-06T07:26:07.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.548+0000 7f340dffb640 1 -- 172.21.15.83:0/2566790818 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f340805bcf0 con 0x7f34180d29c0 2024-03-06T07:26:07.552 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:07.552 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":39,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/2525949760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":2525949760},{"type":"v1","addr":"172.21.15.183:6834","nonce":2525949760}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35},{"gid":6262,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2104447869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2104447869},{"type":"v1","addr":"172.21.15.183:6835","nonce":2104447869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6954,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/779621621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":779621621},{"type":"v1","addr":"172.21.15.83:6836","nonce":779621621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:54.631963+0000","modified":"2024-03-06T07:25:56.637047+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6957":{"gid":6957,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/3056214366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":3056214366},{"type":"v1","addr":"172.21.15.83:6837","nonce":3056214366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6957,"qdb_cluster":[6957]},"id":6}]} 2024-03-06T07:26:07.553 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 39 2024-03-06T07:26:07.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.550+0000 7f341fa08640 1 -- 172.21.15.83:0/2566790818 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f33fc06b500 msgr2=0x7f33fc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.550+0000 7f341fa08640 1 --2- 172.21.15.83:0/2566790818 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f33fc06b500 0x7f33fc06d9c0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f3414036000 tx=0x7f341403b040 comp rx=0 tx=0).stop 2024-03-06T07:26:07.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.550+0000 7f341fa08640 1 -- 172.21.15.83:0/2566790818 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34180d29c0 msgr2=0x7f34180b8140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.550+0000 7f341fa08640 1 --2- 172.21.15.83:0/2566790818 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34180d29c0 0x7f34180b8140 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f3408001eb0 tx=0x7f340800c960 comp rx=0 tx=0).stop 2024-03-06T07:26:07.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.550+0000 7f341fa08640 1 -- 172.21.15.83:0/2566790818 shutdown_connections 2024-03-06T07:26:07.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.550+0000 7f341fa08640 1 --2- 172.21.15.83:0/2566790818 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f33fc06b500 0x7f33fc06d9c0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.550+0000 7f341fa08640 1 --2- 172.21.15.83:0/2566790818 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f341815dfe0 0x7f34180b1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.550+0000 7f341fa08640 1 --2- 172.21.15.83:0/2566790818 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34180d29c0 0x7f34180b8140 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.550+0000 7f341fa08640 1 --2- 172.21.15.83:0/2566790818 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f34180cf830 0x7f34180b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.550+0000 7f341fa08640 1 -- 172.21.15.83:0/2566790818 >> 172.21.15.83:0/2566790818 conn(0x7f34180d3cd0 msgr2=0x7f34180b5f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:07.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.550+0000 7f341fa08640 1 -- 172.21.15.83:0/2566790818 shutdown_connections 2024-03-06T07:26:07.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.550+0000 7f341fa08640 1 -- 172.21.15.83:0/2566790818 wait complete. 2024-03-06T07:26:07.563 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:07.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.663+0000 7fb3e9172640 1 Processor -- start 2024-03-06T07:26:07.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.663+0000 7fb3e9172640 1 -- start start 2024-03-06T07:26:07.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.664+0000 7fb3e9172640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb3e415bb20 0x7fb3e40d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.664+0000 7fb3e9172640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fb3e40cd8c0 0x7fb3e40d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.664+0000 7fb3e9172640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3e4161980 con 0x7fb3e40cfff0 2024-03-06T07:26:07.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.664+0000 7fb3e9172640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3e4161c60 con 0x7fb3e40cd8c0 2024-03-06T07:26:07.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.664+0000 7fb3e9172640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e4161f40 con 0x7fb3e415bb20 2024-03-06T07:26:07.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.665+0000 7fb3e2d9d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb3e415bb20 0x7fb3e40d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.665+0000 7fb3e359e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fb3e40cd8c0 0x7fb3e40d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.665+0000 7fb3e2d9d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb3e415bb20 0x7fb3e40d1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:58834/0 (socket says 172.21.15.83:58834) 2024-03-06T07:26:07.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.665+0000 7fb3e359e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fb3e40cd8c0 0x7fb3e40d0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:53666/0 (socket says 172.21.15.83:53666) 2024-03-06T07:26:07.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.665+0000 7fb3e2d9d640 1 -- 172.21.15.83:0/2826975630 learned_addr learned my addr 172.21.15.83:0/2826975630 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:07.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.665+0000 7fb3e259c640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fb3e40cfff0 0x7fb3e40ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:37750/0 (socket says 172.21.15.83:37750) 2024-03-06T07:26:07.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.665+0000 7fb3e359e640 1 -- 172.21.15.83:0/2826975630 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb3e415bb20 msgr2=0x7fb3e40d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:07.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.665+0000 7fb3e359e640 1 --2- 172.21.15.83:0/2826975630 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb3e415bb20 0x7fb3e40d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.665+0000 7fb3e359e640 1 -- 172.21.15.83:0/2826975630 >> v1:172.21.15.183:6789/0 conn(0x7fb3e40cfff0 legacy=0x7fb3e40ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.665+0000 7fb3e359e640 1 -- 172.21.15.83:0/2826975630 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e4162220 con 0x7fb3e40cd8c0 2024-03-06T07:26:07.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.665+0000 7fb3e1d9b640 1 -- 172.21.15.83:0/2826975630 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2177565740 0 0) 0x7fb3e4161980 con 0x7fb3e40cfff0 2024-03-06T07:26:07.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.665+0000 7fb3e2d9d640 1 --2- 172.21.15.83:0/2826975630 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb3e415bb20 0x7fb3e40d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:07.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.666+0000 7fb3e359e640 1 --2- 172.21.15.83:0/2826975630 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb3e40cd8c0 0x7fb3e40d0470 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc009d90 tx=0x7fb3dc02be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2140c51f41987d5e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:07.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.666+0000 7fb3e1d9b640 1 -- 172.21.15.83:0/2826975630 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3dc039070 con 0x7fb3e40cd8c0 2024-03-06T07:26:07.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.666+0000 7fb3e1d9b640 1 -- 172.21.15.83:0/2826975630 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3dc02cbc0 con 0x7fb3e40cd8c0 2024-03-06T07:26:07.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.666+0000 7fb3e1d9b640 1 -- 172.21.15.83:0/2826975630 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3dc034440 con 0x7fb3e40cd8c0 2024-03-06T07:26:07.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.666+0000 7fb3e9172640 1 -- 172.21.15.83:0/2826975630 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb3e40cd8c0 msgr2=0x7fb3e40d0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.666+0000 7fb3e9172640 1 --2- 172.21.15.83:0/2826975630 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb3e40cd8c0 0x7fb3e40d0470 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc009d90 tx=0x7fb3dc02be10 comp rx=0 tx=0).stop 2024-03-06T07:26:07.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.667+0000 7fb3e9172640 1 -- 172.21.15.83:0/2826975630 shutdown_connections 2024-03-06T07:26:07.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.667+0000 7fb3e9172640 1 --2- 172.21.15.83:0/2826975630 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb3e40cd8c0 0x7fb3e40d0470 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.667+0000 7fb3e9172640 1 --2- 172.21.15.83:0/2826975630 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb3e415bb20 0x7fb3e40d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.667+0000 7fb3e9172640 1 -- 172.21.15.83:0/2826975630 >> 172.21.15.83:0/2826975630 conn(0x7fb3e409a9f0 msgr2=0x7fb3e409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:07.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.667+0000 7fb3e9172640 1 -- 172.21.15.83:0/2826975630 shutdown_connections 2024-03-06T07:26:07.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.667+0000 7fb3e9172640 1 -- 172.21.15.83:0/2826975630 wait complete. 2024-03-06T07:26:07.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.668+0000 7fb3e9172640 1 Processor -- start 2024-03-06T07:26:07.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.668+0000 7fb3e9172640 1 -- start start 2024-03-06T07:26:07.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.669+0000 7fb3e9172640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb3e40cd8c0 0x7fb3e414aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.669+0000 7fb3e9172640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb3e40cfff0 0x7fb3e40c8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.669+0000 7fb3e9172640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3e415bb20 0x7fb3e415a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.669+0000 7fb3e9172640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e40d24b0 con 0x7fb3e415bb20 2024-03-06T07:26:07.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.669+0000 7fb3e9172640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e40d2330 con 0x7fb3e40cfff0 2024-03-06T07:26:07.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.669+0000 7fb3e9172640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e40d1980 con 0x7fb3e40cd8c0 2024-03-06T07:26:07.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.669+0000 7fb3e359e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3e415bb20 0x7fb3e415a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.669+0000 7fb3e359e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3e415bb20 0x7fb3e415a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52772/0 (socket says 172.21.15.83:52772) 2024-03-06T07:26:07.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.669+0000 7fb3e359e640 1 -- 172.21.15.83:0/2789483077 learned_addr learned my addr 172.21.15.83:0/2789483077 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:07.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.669+0000 7fb3e259c640 1 --2- 172.21.15.83:0/2789483077 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb3e40cfff0 0x7fb3e40c8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.669+0000 7fb3e2d9d640 1 --2- 172.21.15.83:0/2789483077 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb3e40cd8c0 0x7fb3e414aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.669+0000 7fb3e359e640 1 -- 172.21.15.83:0/2789483077 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb3e40cd8c0 msgr2=0x7fb3e414aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.669+0000 7fb3e359e640 1 --2- 172.21.15.83:0/2789483077 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb3e40cd8c0 0x7fb3e414aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.669+0000 7fb3e359e640 1 -- 172.21.15.83:0/2789483077 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb3e40cfff0 msgr2=0x7fb3e40c8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.669+0000 7fb3e359e640 1 --2- 172.21.15.83:0/2789483077 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb3e40cfff0 0x7fb3e40c8cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.669+0000 7fb3e359e640 1 -- 172.21.15.83:0/2789483077 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e415b320 con 0x7fb3e415bb20 2024-03-06T07:26:07.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.670+0000 7fb3e359e640 1 --2- 172.21.15.83:0/2789483077 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3e415bb20 0x7fb3e415a7b0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc009d90 tx=0x7fb3dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:07.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.670+0000 7fb3cb7fe640 1 -- 172.21.15.83:0/2789483077 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3dc044020 con 0x7fb3e415bb20 2024-03-06T07:26:07.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.670+0000 7fb3cb7fe640 1 -- 172.21.15.83:0/2789483077 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3dc03a040 con 0x7fb3e415bb20 2024-03-06T07:26:07.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.670+0000 7fb3e9172640 1 -- 172.21.15.83:0/2789483077 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3e40c5670 con 0x7fb3e415bb20 2024-03-06T07:26:07.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.670+0000 7fb3cb7fe640 1 -- 172.21.15.83:0/2789483077 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3dc039050 con 0x7fb3e415bb20 2024-03-06T07:26:07.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.670+0000 7fb3e9172640 1 -- 172.21.15.83:0/2789483077 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3e40c5c30 con 0x7fb3e415bb20 2024-03-06T07:26:07.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.671+0000 7fb3e9172640 1 -- 172.21.15.83:0/2789483077 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3e404f330 con 0x7fb3e415bb20 2024-03-06T07:26:07.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.671+0000 7fb3cb7fe640 1 -- 172.21.15.83:0/2789483077 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb3dc003bb0 con 0x7fb3e415bb20 2024-03-06T07:26:07.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.671+0000 7fb3cb7fe640 1 --2- 172.21.15.83:0/2789483077 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb3d006b500 0x7fb3d006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.672+0000 7fb3cb7fe640 1 -- 172.21.15.83:0/2789483077 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb3dc0ac6d0 con 0x7fb3e415bb20 2024-03-06T07:26:07.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.672+0000 7fb3e2d9d640 1 --2- 172.21.15.83:0/2789483077 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb3d006b500 0x7fb3d006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.672+0000 7fb3e2d9d640 1 --2- 172.21.15.83:0/2789483077 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb3d006b500 0x7fb3d006d9c0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fb3d8006ca0 tx=0x7fb3d8009240 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:07.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.680+0000 7fb3cb7fe640 1 -- 172.21.15.83:0/2789483077 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb3dc04e060 con 0x7fb3e415bb20 2024-03-06T07:26:07.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.894+0000 7fb3e9172640 1 -- 172.21.15.83:0/2789483077 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3e40b11c0 con 0x7fb3e415bb20 2024-03-06T07:26:07.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.895+0000 7fb3cb7fe640 1 -- 172.21.15.83:0/2789483077 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+4982 (secure 0 0 0) 0x7fb3dc07af10 con 0x7fb3e415bb20 2024-03-06T07:26:07.896 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:07.897 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":39,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/2525949760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":2525949760},{"type":"v1","addr":"172.21.15.183:6834","nonce":2525949760}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35},{"gid":6262,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2104447869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2104447869},{"type":"v1","addr":"172.21.15.183:6835","nonce":2104447869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6954,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/779621621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":779621621},{"type":"v1","addr":"172.21.15.83:6836","nonce":779621621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:54.631963+0000","modified":"2024-03-06T07:25:56.637047+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6957":{"gid":6957,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/3056214366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":3056214366},{"type":"v1","addr":"172.21.15.83:6837","nonce":3056214366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6957,"qdb_cluster":[6957]},"id":6}]} 2024-03-06T07:26:07.897 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 39 2024-03-06T07:26:07.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.897+0000 7fb3e9172640 1 -- 172.21.15.83:0/2789483077 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb3d006b500 msgr2=0x7fb3d006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.897+0000 7fb3e9172640 1 --2- 172.21.15.83:0/2789483077 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb3d006b500 0x7fb3d006d9c0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fb3d8006ca0 tx=0x7fb3d8009240 comp rx=0 tx=0).stop 2024-03-06T07:26:07.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.897+0000 7fb3e9172640 1 -- 172.21.15.83:0/2789483077 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3e415bb20 msgr2=0x7fb3e415a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.897+0000 7fb3e9172640 1 --2- 172.21.15.83:0/2789483077 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3e415bb20 0x7fb3e415a7b0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc009d90 tx=0x7fb3dc004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:07.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.897+0000 7fb3e9172640 1 -- 172.21.15.83:0/2789483077 shutdown_connections 2024-03-06T07:26:07.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.897+0000 7fb3e9172640 1 --2- 172.21.15.83:0/2789483077 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb3d006b500 0x7fb3d006d9c0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.897+0000 7fb3e9172640 1 --2- 172.21.15.83:0/2789483077 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3e415bb20 0x7fb3e415a7b0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.897+0000 7fb3e9172640 1 --2- 172.21.15.83:0/2789483077 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb3e40cfff0 0x7fb3e40c8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.897+0000 7fb3e9172640 1 --2- 172.21.15.83:0/2789483077 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb3e40cd8c0 0x7fb3e414aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.897+0000 7fb3e9172640 1 -- 172.21.15.83:0/2789483077 >> 172.21.15.83:0/2789483077 conn(0x7fb3e409a9f0 msgr2=0x7fb3e406eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:07.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.897+0000 7fb3e9172640 1 -- 172.21.15.83:0/2789483077 shutdown_connections 2024-03-06T07:26:07.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:07.897+0000 7fb3e9172640 1 -- 172.21.15.83:0/2789483077 wait complete. 2024-03-06T07:26:07.910 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:08.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.009+0000 7fa4330b8640 1 Processor -- start 2024-03-06T07:26:08.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.009+0000 7fa4330b8640 1 -- start start 2024-03-06T07:26:08.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.010+0000 7fa4330b8640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa42c1424f0 0x7fa42c0cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.010+0000 7fa4330b8640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fa42c0cd7a0 0x7fa42c15cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.010+0000 7fa4330b8640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa42c0cc980 con 0x7fa42c06d330 2024-03-06T07:26:08.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.010+0000 7fa4330b8640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa42c0cd220 con 0x7fa42c1424f0 2024-03-06T07:26:08.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.010+0000 7fa4330b8640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa42c0ccc60 con 0x7fa42c0cd7a0 2024-03-06T07:26:08.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.010+0000 7fa430e54640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa42c1424f0 0x7fa42c0cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.010+0000 7fa430e54640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa42c1424f0 0x7fa42c0cc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52782/0 (socket says 172.21.15.83:52782) 2024-03-06T07:26:08.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.010+0000 7fa430e54640 1 -- 172.21.15.83:0/3037933400 learned_addr learned my addr 172.21.15.83:0/3037933400 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:08.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.010+0000 7fa42bfff640 1 --2- 172.21.15.83:0/3037933400 >> v2:172.21.15.183:3300/0 conn(0x7fa42c0cd7a0 0x7fa42c15cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.010+0000 7fa430e54640 1 -- 172.21.15.83:0/3037933400 >> v2:172.21.15.183:3300/0 conn(0x7fa42c0cd7a0 msgr2=0x7fa42c15cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:08.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.010+0000 7fa430e54640 1 --2- 172.21.15.83:0/3037933400 >> v2:172.21.15.183:3300/0 conn(0x7fa42c0cd7a0 0x7fa42c15cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.010+0000 7fa430e54640 1 -- 172.21.15.83:0/3037933400 >> v1:172.21.15.183:6789/0 conn(0x7fa42c06d330 legacy=0x7fa42c071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.010+0000 7fa430e54640 1 -- 172.21.15.83:0/3037933400 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa42c0ccf40 con 0x7fa42c1424f0 2024-03-06T07:26:08.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.011+0000 7fa430e54640 1 --2- 172.21.15.83:0/3037933400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa42c1424f0 0x7fa42c0cc2f0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fa42c0b2a00 tx=0x7fa42002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d93b8ea6c53a2097 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.011+0000 7fa42b7fe640 1 -- 172.21.15.83:0/3037933400 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa420034070 con 0x7fa42c1424f0 2024-03-06T07:26:08.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.011+0000 7fa42b7fe640 1 -- 172.21.15.83:0/3037933400 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa420030860 con 0x7fa42c1424f0 2024-03-06T07:26:08.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.011+0000 7fa42b7fe640 1 -- 172.21.15.83:0/3037933400 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa420030c30 con 0x7fa42c1424f0 2024-03-06T07:26:08.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.012+0000 7fa4330b8640 1 -- 172.21.15.83:0/3037933400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa42c1424f0 msgr2=0x7fa42c0cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.012+0000 7fa4330b8640 1 --2- 172.21.15.83:0/3037933400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa42c1424f0 0x7fa42c0cc2f0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fa42c0b2a00 tx=0x7fa42002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.012+0000 7fa4330b8640 1 -- 172.21.15.83:0/3037933400 shutdown_connections 2024-03-06T07:26:08.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.012+0000 7fa4330b8640 1 --2- 172.21.15.83:0/3037933400 >> v2:172.21.15.183:3300/0 conn(0x7fa42c0cd7a0 0x7fa42c15cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.012+0000 7fa4330b8640 1 --2- 172.21.15.83:0/3037933400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa42c1424f0 0x7fa42c0cc2f0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.012+0000 7fa4330b8640 1 -- 172.21.15.83:0/3037933400 >> 172.21.15.83:0/3037933400 conn(0x7fa42c0cf8a0 msgr2=0x7fa42c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:08.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.012+0000 7fa4330b8640 1 -- 172.21.15.83:0/3037933400 shutdown_connections 2024-03-06T07:26:08.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.012+0000 7fa4330b8640 1 -- 172.21.15.83:0/3037933400 wait complete. 2024-03-06T07:26:08.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.013+0000 7fa4330b8640 1 Processor -- start 2024-03-06T07:26:08.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.013+0000 7fa4330b8640 1 -- start start 2024-03-06T07:26:08.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.015+0000 7fa4330b8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa42c06d330 0x7fa42c154a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.015+0000 7fa4330b8640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa42c0cd7a0 0x7fa42c137220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.015+0000 7fa4330b8640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa42c1424f0 0x7fa42c1550e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.015+0000 7fa4330b8640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa42c06d920 con 0x7fa42c06d330 2024-03-06T07:26:08.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.015+0000 7fa4330b8640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa42c06d7a0 con 0x7fa42c0cd7a0 2024-03-06T07:26:08.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.015+0000 7fa4330b8640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa42c06daa0 con 0x7fa42c1424f0 2024-03-06T07:26:08.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.015+0000 7fa430e54640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa42c06d330 0x7fa42c154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.015+0000 7fa430e54640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa42c06d330 0x7fa42c154a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52798/0 (socket says 172.21.15.83:52798) 2024-03-06T07:26:08.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.015+0000 7fa430e54640 1 -- 172.21.15.83:0/4227643440 learned_addr learned my addr 172.21.15.83:0/4227643440 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:08.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.015+0000 7fa42bfff640 1 --2- 172.21.15.83:0/4227643440 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa42c0cd7a0 0x7fa42c137220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.015+0000 7fa431655640 1 --2- 172.21.15.83:0/4227643440 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa42c1424f0 0x7fa42c1550e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.015+0000 7fa430e54640 1 -- 172.21.15.83:0/4227643440 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa42c1424f0 msgr2=0x7fa42c1550e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.015+0000 7fa430e54640 1 --2- 172.21.15.83:0/4227643440 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa42c1424f0 0x7fa42c1550e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.015+0000 7fa430e54640 1 -- 172.21.15.83:0/4227643440 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa42c0cd7a0 msgr2=0x7fa42c137220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.015+0000 7fa430e54640 1 --2- 172.21.15.83:0/4227643440 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa42c0cd7a0 0x7fa42c137220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.015+0000 7fa430e54640 1 -- 172.21.15.83:0/4227643440 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa42c137760 con 0x7fa42c06d330 2024-03-06T07:26:08.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.015+0000 7fa431655640 1 --2- 172.21.15.83:0/4227643440 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa42c1424f0 0x7fa42c1550e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:08.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.015+0000 7fa430e54640 1 --2- 172.21.15.83:0/4227643440 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa42c06d330 0x7fa42c154a60 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fa42002b9e0 tx=0x7fa4200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.015+0000 7fa4297fa640 1 -- 172.21.15.83:0/4227643440 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa420034040 con 0x7fa42c06d330 2024-03-06T07:26:08.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.015+0000 7fa4297fa640 1 -- 172.21.15.83:0/4227643440 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa420039d90 con 0x7fa42c06d330 2024-03-06T07:26:08.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.015+0000 7fa4297fa640 1 -- 172.21.15.83:0/4227643440 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa420043630 con 0x7fa42c06d330 2024-03-06T07:26:08.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.015+0000 7fa4330b8640 1 -- 172.21.15.83:0/4227643440 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa42c1f58e0 con 0x7fa42c06d330 2024-03-06T07:26:08.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.016+0000 7fa4330b8640 1 -- 172.21.15.83:0/4227643440 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa42c1f5ea0 con 0x7fa42c06d330 2024-03-06T07:26:08.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.016+0000 7fa4330b8640 1 -- 172.21.15.83:0/4227643440 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa42c04f330 con 0x7fa42c06d330 2024-03-06T07:26:08.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.017+0000 7fa4297fa640 1 -- 172.21.15.83:0/4227643440 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa42004a060 con 0x7fa42c06d330 2024-03-06T07:26:08.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.017+0000 7fa4297fa640 1 --2- 172.21.15.83:0/4227643440 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa41006b4d0 0x7fa41006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.017+0000 7fa4297fa640 1 -- 172.21.15.83:0/4227643440 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa4200ac290 con 0x7fa42c06d330 2024-03-06T07:26:08.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.017+0000 7fa42bfff640 1 --2- 172.21.15.83:0/4227643440 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa41006b4d0 0x7fa41006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.018+0000 7fa42bfff640 1 --2- 172.21.15.83:0/4227643440 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa41006b4d0 0x7fa41006d990 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fa42c1f54d0 tx=0x7fa418008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.021+0000 7fa4297fa640 1 -- 172.21.15.83:0/4227643440 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa42c137760 con 0x7fa42c06d330 2024-03-06T07:26:08.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.306+0000 7fa4330b8640 1 -- 172.21.15.83:0/4227643440 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa42c141ca0 con 0x7fa42c06d330 2024-03-06T07:26:08.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.307+0000 7fa4297fa640 1 -- 172.21.15.83:0/4227643440 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16672 (secure 0 0 0) 0x7fa42003e070 con 0x7fa42c06d330 2024-03-06T07:26:08.310 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:08.310 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:25:56.279928+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:52.856525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:53.598434+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:08.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.310+0000 7fa4330b8640 1 -- 172.21.15.83:0/4227643440 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa41006b4d0 msgr2=0x7fa41006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.310+0000 7fa4330b8640 1 --2- 172.21.15.83:0/4227643440 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa41006b4d0 0x7fa41006d990 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fa42c1f54d0 tx=0x7fa418008040 comp rx=0 tx=0).stop 2024-03-06T07:26:08.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.310+0000 7fa4330b8640 1 -- 172.21.15.83:0/4227643440 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa42c06d330 msgr2=0x7fa42c154a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.310+0000 7fa4330b8640 1 --2- 172.21.15.83:0/4227643440 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa42c06d330 0x7fa42c154a60 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fa42002b9e0 tx=0x7fa4200046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.310+0000 7fa4330b8640 1 -- 172.21.15.83:0/4227643440 shutdown_connections 2024-03-06T07:26:08.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.310+0000 7fa4330b8640 1 --2- 172.21.15.83:0/4227643440 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa41006b4d0 0x7fa41006d990 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.310+0000 7fa4330b8640 1 --2- 172.21.15.83:0/4227643440 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa42c1424f0 0x7fa42c1550e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.310+0000 7fa4330b8640 1 --2- 172.21.15.83:0/4227643440 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa42c0cd7a0 0x7fa42c137220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.310+0000 7fa4330b8640 1 --2- 172.21.15.83:0/4227643440 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa42c06d330 0x7fa42c154a60 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.310+0000 7fa4330b8640 1 -- 172.21.15.83:0/4227643440 >> 172.21.15.83:0/4227643440 conn(0x7fa42c0cf8a0 msgr2=0x7fa42c0ce710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:08.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.310+0000 7fa4330b8640 1 -- 172.21.15.83:0/4227643440 shutdown_connections 2024-03-06T07:26:08.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.310+0000 7fa4330b8640 1 -- 172.21.15.83:0/4227643440 wait complete. 2024-03-06T07:26:08.323 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:26:08.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.423+0000 7f4f41a47640 1 Processor -- start 2024-03-06T07:26:08.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.423+0000 7f4f41a47640 1 -- start start 2024-03-06T07:26:08.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.424+0000 7f4f41a47640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4f3c0b4460 0x7f4f3c0d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.424+0000 7f4f41a47640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4f3c162970 0x7f4f3c0b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.424+0000 7f4f41a47640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f3c0d22c0 con 0x7f4f3c161cf0 2024-03-06T07:26:08.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.424+0000 7f4f41a47640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f3c161b00 con 0x7f4f3c0b4460 2024-03-06T07:26:08.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.424+0000 7f4f41a47640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3c0cecc0 con 0x7f4f3c162970 2024-03-06T07:26:08.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.424+0000 7f4f3affd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4f3c0b4460 0x7f4f3c0d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.424+0000 7f4f3b7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f4f3c161cf0 0x7f4f3c0cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60350/0 (socket says 172.21.15.83:60350) 2024-03-06T07:26:08.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.424+0000 7f4f3b7fe640 1 -- 172.21.15.83:0/3790108513 learned_addr learned my addr 172.21.15.83:0/3790108513 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:08.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.424+0000 7f4f3affd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4f3c0b4460 0x7f4f3c0d4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52810/0 (socket says 172.21.15.83:52810) 2024-03-06T07:26:08.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.425+0000 7f4f33fff640 1 --2- 172.21.15.83:0/3790108513 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4f3c162970 0x7f4f3c0b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.425+0000 7f4f3affd640 1 -- 172.21.15.83:0/3790108513 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4f3c162970 msgr2=0x7f4f3c0b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:08.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.425+0000 7f4f3affd640 1 --2- 172.21.15.83:0/3790108513 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4f3c162970 0x7f4f3c0b3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.425+0000 7f4f3affd640 1 -- 172.21.15.83:0/3790108513 >> v1:172.21.15.83:6789/0 conn(0x7f4f3c161cf0 legacy=0x7f4f3c0cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.425+0000 7f4f3affd640 1 -- 172.21.15.83:0/3790108513 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f3c0d25a0 con 0x7f4f3c0b4460 2024-03-06T07:26:08.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.425+0000 7f4f3a7fc640 1 -- 172.21.15.83:0/3790108513 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2784777987 0 0) 0x7f4f3c0d22c0 con 0x7f4f3c161cf0 2024-03-06T07:26:08.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.425+0000 7f4f3affd640 1 --2- 172.21.15.83:0/3790108513 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f3c0b4460 0x7f4f3c0d4800 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c0097a0 tx=0x7f4f2c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b461a6939c7a675a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.426+0000 7f4f3a7fc640 1 -- 172.21.15.83:0/3790108513 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c035020 con 0x7f4f3c0b4460 2024-03-06T07:26:08.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.426+0000 7f4f3a7fc640 1 -- 172.21.15.83:0/3790108513 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f2c031830 con 0x7f4f3c0b4460 2024-03-06T07:26:08.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.426+0000 7f4f3a7fc640 1 -- 172.21.15.83:0/3790108513 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c031c00 con 0x7f4f3c0b4460 2024-03-06T07:26:08.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.426+0000 7f4f41a47640 1 -- 172.21.15.83:0/3790108513 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f3c0b4460 msgr2=0x7f4f3c0d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.426+0000 7f4f41a47640 1 --2- 172.21.15.83:0/3790108513 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f3c0b4460 0x7f4f3c0d4800 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c0097a0 tx=0x7f4f2c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.426+0000 7f4f41a47640 1 -- 172.21.15.83:0/3790108513 shutdown_connections 2024-03-06T07:26:08.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.426+0000 7f4f41a47640 1 --2- 172.21.15.83:0/3790108513 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4f3c162970 0x7f4f3c0b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.426+0000 7f4f41a47640 1 --2- 172.21.15.83:0/3790108513 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f3c0b4460 0x7f4f3c0d4800 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.426+0000 7f4f41a47640 1 -- 172.21.15.83:0/3790108513 >> 172.21.15.83:0/3790108513 conn(0x7f4f3c0d3ce0 msgr2=0x7f4f3c0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:08.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.426+0000 7f4f41a47640 1 -- 172.21.15.83:0/3790108513 shutdown_connections 2024-03-06T07:26:08.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.426+0000 7f4f41a47640 1 -- 172.21.15.83:0/3790108513 wait complete. 2024-03-06T07:26:08.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.427+0000 7f4f41a47640 1 Processor -- start 2024-03-06T07:26:08.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.427+0000 7f4f41a47640 1 -- start start 2024-03-06T07:26:08.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.428+0000 7f4f41a47640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f3c0b4460 0x7f4f3c0b28c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.428+0000 7f4f41a47640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4f3c161cf0 0x7f4f3c1f4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.428+0000 7f4f41a47640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4f3c162970 0x7f4f3c1560a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.428+0000 7f4f41a47640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3c0ce950 con 0x7f4f3c0b4460 2024-03-06T07:26:08.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.428+0000 7f4f41a47640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3c0d2b60 con 0x7f4f3c161cf0 2024-03-06T07:26:08.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.428+0000 7f4f41a47640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3c0b3ac0 con 0x7f4f3c162970 2024-03-06T07:26:08.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.428+0000 7f4f3affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f3c0b4460 0x7f4f3c0b28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.428+0000 7f4f3affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f3c0b4460 0x7f4f3c0b28c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52826/0 (socket says 172.21.15.83:52826) 2024-03-06T07:26:08.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.428+0000 7f4f3affd640 1 -- 172.21.15.83:0/3653335173 learned_addr learned my addr 172.21.15.83:0/3653335173 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:08.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.428+0000 7f4f3affd640 1 -- 172.21.15.83:0/3653335173 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4f3c162970 msgr2=0x7f4f3c1560a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.428+0000 7f4f3affd640 1 --2- 172.21.15.83:0/3653335173 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4f3c162970 0x7f4f3c1560a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.428+0000 7f4f3affd640 1 -- 172.21.15.83:0/3653335173 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4f3c161cf0 msgr2=0x7f4f3c1f4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.428+0000 7f4f3affd640 1 --2- 172.21.15.83:0/3653335173 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4f3c161cf0 0x7f4f3c1f4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.428+0000 7f4f3affd640 1 -- 172.21.15.83:0/3653335173 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f3c0b8140 con 0x7f4f3c0b4460 2024-03-06T07:26:08.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.429+0000 7f4f3affd640 1 --2- 172.21.15.83:0/3653335173 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f3c0b4460 0x7f4f3c0b28c0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c0098c0 tx=0x7f4f2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.429+0000 7f4f38ff9640 1 -- 172.21.15.83:0/3653335173 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c040070 con 0x7f4f3c0b4460 2024-03-06T07:26:08.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.429+0000 7f4f38ff9640 1 -- 172.21.15.83:0/3653335173 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f2c03e070 con 0x7f4f3c0b4460 2024-03-06T07:26:08.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.429+0000 7f4f41a47640 1 -- 172.21.15.83:0/3653335173 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f3c1f4ef0 con 0x7f4f3c0b4460 2024-03-06T07:26:08.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.429+0000 7f4f38ff9640 1 -- 172.21.15.83:0/3653335173 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c035040 con 0x7f4f3c0b4460 2024-03-06T07:26:08.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.429+0000 7f4f41a47640 1 -- 172.21.15.83:0/3653335173 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f3c1f54b0 con 0x7f4f3c0b4460 2024-03-06T07:26:08.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.430+0000 7f4f41a47640 1 -- 172.21.15.83:0/3653335173 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f3c04f330 con 0x7f4f3c0b4460 2024-03-06T07:26:08.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.430+0000 7f4f38ff9640 1 -- 172.21.15.83:0/3653335173 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4f2c038020 con 0x7f4f3c0b4460 2024-03-06T07:26:08.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.431+0000 7f4f38ff9640 1 --2- 172.21.15.83:0/3653335173 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4f1c06b500 0x7f4f1c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.431+0000 7f4f38ff9640 1 -- 172.21.15.83:0/3653335173 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4f2c0abb00 con 0x7f4f3c0b4460 2024-03-06T07:26:08.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.431+0000 7f4f33fff640 1 --2- 172.21.15.83:0/3653335173 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4f1c06b500 0x7f4f1c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.431+0000 7f4f33fff640 1 --2- 172.21.15.83:0/3653335173 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4f1c06b500 0x7f4f1c06d9c0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f4f24006a90 tx=0x7f4f24006a20 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.435+0000 7f4f38ff9640 1 -- 172.21.15.83:0/3653335173 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4f2c07a980 con 0x7f4f3c0b4460 2024-03-06T07:26:08.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.645+0000 7f4f41a47640 1 -- 172.21.15.83:0/3653335173 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f4f3c06b180 con 0x7f4f3c0b4460 2024-03-06T07:26:08.647 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-03-06T07:26:08.646+0000 7fb0dae11640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:26:08.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.647+0000 7f4f38ff9640 1 -- 172.21.15.83:0/3653335173 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v40) v1 ==== 161+0+0 (secure 0 0 0) 0x7f4f2c031800 con 0x7f4f3c0b4460 2024-03-06T07:26:08.649 INFO:teuthology.orchestra.run.smithi083.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:26:08.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.650+0000 7f4f41a47640 1 -- 172.21.15.83:0/3653335173 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4f1c06b500 msgr2=0x7f4f1c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.650+0000 7f4f41a47640 1 --2- 172.21.15.83:0/3653335173 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4f1c06b500 0x7f4f1c06d9c0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f4f24006a90 tx=0x7f4f24006a20 comp rx=0 tx=0).stop 2024-03-06T07:26:08.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.650+0000 7f4f41a47640 1 -- 172.21.15.83:0/3653335173 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f3c0b4460 msgr2=0x7f4f3c0b28c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.650+0000 7f4f41a47640 1 --2- 172.21.15.83:0/3653335173 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f3c0b4460 0x7f4f3c0b28c0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c0098c0 tx=0x7f4f2c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.650+0000 7f4f41a47640 1 -- 172.21.15.83:0/3653335173 shutdown_connections 2024-03-06T07:26:08.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.650+0000 7f4f41a47640 1 --2- 172.21.15.83:0/3653335173 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4f1c06b500 0x7f4f1c06d9c0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.650+0000 7f4f41a47640 1 --2- 172.21.15.83:0/3653335173 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4f3c162970 0x7f4f3c1560a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.650+0000 7f4f41a47640 1 --2- 172.21.15.83:0/3653335173 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4f3c161cf0 0x7f4f3c1f4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.650+0000 7f4f41a47640 1 --2- 172.21.15.83:0/3653335173 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f3c0b4460 0x7f4f3c0b28c0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.650+0000 7f4f41a47640 1 -- 172.21.15.83:0/3653335173 >> 172.21.15.83:0/3653335173 conn(0x7f4f3c0d3ce0 msgr2=0x7f4f3c0d08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:08.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.650+0000 7f4f41a47640 1 -- 172.21.15.83:0/3653335173 shutdown_connections 2024-03-06T07:26:08.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.650+0000 7f4f41a47640 1 -- 172.21.15.83:0/3653335173 wait complete. 2024-03-06T07:26:08.654 INFO:tasks.ceph.mds.c.smithi083.stderr: -2501> 2024-03-06T07:26:01.275+0000 7f18123cd640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:26:08.664 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:26:08.683 INFO:tasks.ceph.mds.c.smithi083.stdout:starting mds.c at 2024-03-06T07:26:08.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.765+0000 7fd85a6e0640 1 Processor -- start 2024-03-06T07:26:08.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.765+0000 7fd85a6e0640 1 -- start start 2024-03-06T07:26:08.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.766+0000 7fd85a6e0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd85415b940 0x7fd8540cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.766+0000 7fd85a6e0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd8540cffd0 0x7fd85415e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.766+0000 7fd85a6e0640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fd85415ea20 0x7fd854161910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.766+0000 7fd85a6e0640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8540ad900 con 0x7fd85415b940 2024-03-06T07:26:08.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.766+0000 7fd85a6e0640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd85415bd20 con 0x7fd85415ea20 2024-03-06T07:26:08.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.766+0000 7fd85a6e0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8540adb70 con 0x7fd8540cffd0 2024-03-06T07:26:08.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.766+0000 7fd853fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd85415b940 0x7fd8540cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.766+0000 7fd853fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd85415b940 0x7fd8540cfa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52848/0 (socket says 172.21.15.83:52848) 2024-03-06T07:26:08.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.766+0000 7fd853fff640 1 -- 172.21.15.83:0/277238722 learned_addr learned my addr 172.21.15.83:0/277238722 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:08.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.766+0000 7fd8537fe640 1 --2- 172.21.15.83:0/277238722 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd8540cffd0 0x7fd85415e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.766+0000 7fd858c7d640 1 --2- 172.21.15.83:0/277238722 >> v2:172.21.15.183:3300/0 conn(0x7fd85415ea20 0x7fd854161910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.766+0000 7fd853fff640 1 -- 172.21.15.83:0/277238722 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd8540cffd0 msgr2=0x7fd85415e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:08.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.766+0000 7fd853fff640 1 --2- 172.21.15.83:0/277238722 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd8540cffd0 0x7fd85415e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.766+0000 7fd853fff640 1 -- 172.21.15.83:0/277238722 >> v2:172.21.15.183:3300/0 conn(0x7fd85415ea20 msgr2=0x7fd854161910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:08.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.766+0000 7fd853fff640 1 --2- 172.21.15.83:0/277238722 >> v2:172.21.15.183:3300/0 conn(0x7fd85415ea20 0x7fd854161910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.766+0000 7fd853fff640 1 -- 172.21.15.83:0/277238722 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8540ade50 con 0x7fd85415b940 2024-03-06T07:26:08.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.767+0000 7fd853fff640 1 --2- 172.21.15.83:0/277238722 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd85415b940 0x7fd8540cfa90 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fd8540b4f60 tx=0x7fd84802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd8b000b6b37bc08 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.767+0000 7fd852ffd640 1 -- 172.21.15.83:0/277238722 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd848035020 con 0x7fd85415b940 2024-03-06T07:26:08.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.767+0000 7fd852ffd640 1 -- 172.21.15.83:0/277238722 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd848031870 con 0x7fd85415b940 2024-03-06T07:26:08.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.767+0000 7fd852ffd640 1 -- 172.21.15.83:0/277238722 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd848031c40 con 0x7fd85415b940 2024-03-06T07:26:08.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.767+0000 7fd85a6e0640 1 -- 172.21.15.83:0/277238722 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd85415b940 msgr2=0x7fd8540cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.767+0000 7fd85a6e0640 1 --2- 172.21.15.83:0/277238722 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd85415b940 0x7fd8540cfa90 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fd8540b4f60 tx=0x7fd84802b910 comp rx=0 tx=0).stop 2024-03-06T07:26:08.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.767+0000 7fd85a6e0640 1 -- 172.21.15.83:0/277238722 shutdown_connections 2024-03-06T07:26:08.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.767+0000 7fd85a6e0640 1 --2- 172.21.15.83:0/277238722 >> v2:172.21.15.183:3300/0 conn(0x7fd85415ea20 0x7fd854161910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.767+0000 7fd85a6e0640 1 --2- 172.21.15.83:0/277238722 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd8540cffd0 0x7fd85415e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.767+0000 7fd85a6e0640 1 --2- 172.21.15.83:0/277238722 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd85415b940 0x7fd8540cfa90 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.768+0000 7fd85a6e0640 1 -- 172.21.15.83:0/277238722 >> 172.21.15.83:0/277238722 conn(0x7fd85406b190 msgr2=0x7fd85406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:08.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.768+0000 7fd85a6e0640 1 -- 172.21.15.83:0/277238722 shutdown_connections 2024-03-06T07:26:08.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.768+0000 7fd85a6e0640 1 -- 172.21.15.83:0/277238722 wait complete. 2024-03-06T07:26:08.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.769+0000 7fd85a6e0640 1 Processor -- start 2024-03-06T07:26:08.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.769+0000 7fd85a6e0640 1 -- start start 2024-03-06T07:26:08.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.770+0000 7fd85a6e0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd8540cffd0 0x7fd8541e76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.770+0000 7fd85a6e0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd85415b940 0x7fd854145300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.770+0000 7fd85a6e0640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd85415ea20 0x7fd854160b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.770+0000 7fd85a6e0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd85415eeb0 con 0x7fd8540cffd0 2024-03-06T07:26:08.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.770+0000 7fd85a6e0640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd85415f210 con 0x7fd85415ea20 2024-03-06T07:26:08.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.770+0000 7fd85a6e0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8540ae650 con 0x7fd85415b940 2024-03-06T07:26:08.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.770+0000 7fd853fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd8540cffd0 0x7fd8541e76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.770+0000 7fd853fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd8540cffd0 0x7fd8541e76f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52856/0 (socket says 172.21.15.83:52856) 2024-03-06T07:26:08.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.770+0000 7fd853fff640 1 -- 172.21.15.83:0/2852110215 learned_addr learned my addr 172.21.15.83:0/2852110215 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:08.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.770+0000 7fd8537fe640 1 --2- 172.21.15.83:0/2852110215 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd85415b940 0x7fd854145300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.770+0000 7fd858c7d640 1 --2- 172.21.15.83:0/2852110215 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd85415ea20 0x7fd854160b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.770+0000 7fd853fff640 1 -- 172.21.15.83:0/2852110215 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd85415b940 msgr2=0x7fd854145300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.770+0000 7fd853fff640 1 --2- 172.21.15.83:0/2852110215 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd85415b940 0x7fd854145300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.770+0000 7fd853fff640 1 -- 172.21.15.83:0/2852110215 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd85415ea20 msgr2=0x7fd854160b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.770+0000 7fd853fff640 1 --2- 172.21.15.83:0/2852110215 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd85415ea20 0x7fd854160b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.770+0000 7fd853fff640 1 -- 172.21.15.83:0/2852110215 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8541603a0 con 0x7fd8540cffd0 2024-03-06T07:26:08.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.770+0000 7fd853fff640 1 --2- 172.21.15.83:0/2852110215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd8540cffd0 0x7fd8541e76f0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fd84802b950 tx=0x7fd8480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.770+0000 7fd850ff9640 1 -- 172.21.15.83:0/2852110215 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd848040070 con 0x7fd8540cffd0 2024-03-06T07:26:08.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.770+0000 7fd850ff9640 1 -- 172.21.15.83:0/2852110215 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd848030d00 con 0x7fd8540cffd0 2024-03-06T07:26:08.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.770+0000 7fd85a6e0640 1 -- 172.21.15.83:0/2852110215 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd854145840 con 0x7fd8540cffd0 2024-03-06T07:26:08.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.770+0000 7fd850ff9640 1 -- 172.21.15.83:0/2852110215 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd848035040 con 0x7fd8540cffd0 2024-03-06T07:26:08.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.770+0000 7fd85a6e0640 1 -- 172.21.15.83:0/2852110215 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8541e8250 con 0x7fd8540cffd0 2024-03-06T07:26:08.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.771+0000 7fd85a6e0640 1 -- 172.21.15.83:0/2852110215 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8540732b0 con 0x7fd8540cffd0 2024-03-06T07:26:08.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.772+0000 7fd850ff9640 1 -- 172.21.15.83:0/2852110215 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd84804b060 con 0x7fd8540cffd0 2024-03-06T07:26:08.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.772+0000 7fd850ff9640 1 --2- 172.21.15.83:0/2852110215 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd84006b4d0 0x7fd84006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.772+0000 7fd850ff9640 1 -- 172.21.15.83:0/2852110215 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fd8480abef0 con 0x7fd8540cffd0 2024-03-06T07:26:08.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.772+0000 7fd8537fe640 1 --2- 172.21.15.83:0/2852110215 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd84006b4d0 0x7fd84006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.772+0000 7fd8537fe640 1 --2- 172.21.15.83:0/2852110215 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd84006b4d0 0x7fd84006d990 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fd844006fd0 tx=0x7fd844008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.776+0000 7fd850ff9640 1 -- 172.21.15.83:0/2852110215 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd854145840 con 0x7fd8540cffd0 2024-03-06T07:26:08.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:08.983+0000 7fd85a6e0640 1 -- 172.21.15.83:0/2852110215 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd854068ff0 con 0x7fd8540cffd0 2024-03-06T07:26:09.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.649+0000 7fd850ff9640 1 -- 172.21.15.83:0/2852110215 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v41) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd8541603a0 con 0x7fd8540cffd0 2024-03-06T07:26:09.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.652+0000 7fd85a6e0640 1 -- 172.21.15.83:0/2852110215 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd84006b4d0 msgr2=0x7fd84006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.652+0000 7fd85a6e0640 1 --2- 172.21.15.83:0/2852110215 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd84006b4d0 0x7fd84006d990 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fd844006fd0 tx=0x7fd844008040 comp rx=0 tx=0).stop 2024-03-06T07:26:09.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.652+0000 7fd85a6e0640 1 -- 172.21.15.83:0/2852110215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd8540cffd0 msgr2=0x7fd8541e76f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.652+0000 7fd85a6e0640 1 --2- 172.21.15.83:0/2852110215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd8540cffd0 0x7fd8541e76f0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fd84802b950 tx=0x7fd8480046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.652+0000 7fd85a6e0640 1 -- 172.21.15.83:0/2852110215 shutdown_connections 2024-03-06T07:26:09.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.652+0000 7fd85a6e0640 1 --2- 172.21.15.83:0/2852110215 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd84006b4d0 0x7fd84006d990 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.652+0000 7fd85a6e0640 1 --2- 172.21.15.83:0/2852110215 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd85415ea20 0x7fd854160b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.652+0000 7fd85a6e0640 1 --2- 172.21.15.83:0/2852110215 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd85415b940 0x7fd854145300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.652+0000 7fd85a6e0640 1 --2- 172.21.15.83:0/2852110215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd8540cffd0 0x7fd8541e76f0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.652+0000 7fd85a6e0640 1 -- 172.21.15.83:0/2852110215 >> 172.21.15.83:0/2852110215 conn(0x7fd85406b190 msgr2=0x7fd854070b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:09.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.652+0000 7fd85a6e0640 1 -- 172.21.15.83:0/2852110215 shutdown_connections 2024-03-06T07:26:09.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.652+0000 7fd85a6e0640 1 -- 172.21.15.83:0/2852110215 wait complete. 2024-03-06T07:26:09.667 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:26:09.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.816+0000 7f6fcb579640 1 Processor -- start 2024-03-06T07:26:09.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.816+0000 7f6fcb579640 1 -- start start 2024-03-06T07:26:09.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.817+0000 7f6fcb579640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f6fc406be80 0x7f6fc40b11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.817+0000 7f6fcb579640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6fc415f9d0 0x7f6fc415fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.817+0000 7f6fcb579640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fc4068b00 con 0x7f6fc40b5ca0 2024-03-06T07:26:09.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.817+0000 7f6fcb579640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fc415edf0 con 0x7f6fc415f9d0 2024-03-06T07:26:09.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.817+0000 7f6fcb579640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fc41618a0 con 0x7f6fc406be80 2024-03-06T07:26:09.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.817+0000 7f6fc9b16640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6fc415f9d0 0x7f6fc415fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.817+0000 7f6fc9b16640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6fc415f9d0 0x7f6fc415fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52864/0 (socket says 172.21.15.83:52864) 2024-03-06T07:26:09.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.817+0000 7f6fc9b16640 1 -- 172.21.15.83:0/2866847371 learned_addr learned my addr 172.21.15.83:0/2866847371 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:09.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.817+0000 7f6fc9315640 1 --2- 172.21.15.83:0/2866847371 >> v2:172.21.15.183:3300/0 conn(0x7f6fc406be80 0x7f6fc40b11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.817+0000 7f6fc9b16640 1 -- 172.21.15.83:0/2866847371 >> v2:172.21.15.183:3300/0 conn(0x7f6fc406be80 msgr2=0x7f6fc40b11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:09.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.817+0000 7f6fc9b16640 1 --2- 172.21.15.83:0/2866847371 >> v2:172.21.15.183:3300/0 conn(0x7f6fc406be80 0x7f6fc40b11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.817+0000 7f6fc9b16640 1 -- 172.21.15.83:0/2866847371 >> v1:172.21.15.183:6789/0 conn(0x7f6fc40b5ca0 legacy=0x7f6fc406c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.817+0000 7f6fc9b16640 1 -- 172.21.15.83:0/2866847371 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc415e4e0 con 0x7f6fc415f9d0 2024-03-06T07:26:09.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.818+0000 7f6fc9b16640 1 --2- 172.21.15.83:0/2866847371 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6fc415f9d0 0x7f6fc415fe00 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f6fc415ef70 tx=0x7f6fbc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dd7a5741a00e1da server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:09.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.818+0000 7f6fbbfff640 1 -- 172.21.15.83:0/2866847371 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc02ca70 con 0x7f6fc415f9d0 2024-03-06T07:26:09.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.818+0000 7f6fbbfff640 1 -- 172.21.15.83:0/2866847371 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fbc02cc10 con 0x7f6fc415f9d0 2024-03-06T07:26:09.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.818+0000 7f6fbbfff640 1 -- 172.21.15.83:0/2866847371 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc0343f0 con 0x7f6fc415f9d0 2024-03-06T07:26:09.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.819+0000 7f6fcb579640 1 -- 172.21.15.83:0/2866847371 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6fc415f9d0 msgr2=0x7f6fc415fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.819+0000 7f6fcb579640 1 --2- 172.21.15.83:0/2866847371 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6fc415f9d0 0x7f6fc415fe00 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f6fc415ef70 tx=0x7f6fbc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.819+0000 7f6fcb579640 1 -- 172.21.15.83:0/2866847371 shutdown_connections 2024-03-06T07:26:09.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.819+0000 7f6fcb579640 1 --2- 172.21.15.83:0/2866847371 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6fc415f9d0 0x7f6fc415fe00 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.819+0000 7f6fcb579640 1 --2- 172.21.15.83:0/2866847371 >> v2:172.21.15.183:3300/0 conn(0x7f6fc406be80 0x7f6fc40b11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.819+0000 7f6fcb579640 1 -- 172.21.15.83:0/2866847371 >> 172.21.15.83:0/2866847371 conn(0x7f6fc40b4ed0 msgr2=0x7f6fc40b5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:09.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.819+0000 7f6fcb579640 1 -- 172.21.15.83:0/2866847371 shutdown_connections 2024-03-06T07:26:09.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.819+0000 7f6fcb579640 1 -- 172.21.15.83:0/2866847371 wait complete. 2024-03-06T07:26:09.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.820+0000 7f6fcb579640 1 Processor -- start 2024-03-06T07:26:09.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.820+0000 7f6fcb579640 1 -- start start 2024-03-06T07:26:09.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.821+0000 7f6fcb579640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6fc406be80 0x7f6fc41506a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.821+0000 7f6fcb579640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6fc40b5ca0 0x7f6fc40d0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.821+0000 7f6fcb579640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6fc415f9d0 0x7f6fc40cdf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.821+0000 7f6fcb579640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc415ea80 con 0x7f6fc415f9d0 2024-03-06T07:26:09.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.821+0000 7f6fcb579640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc415e7a0 con 0x7f6fc406be80 2024-03-06T07:26:09.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.821+0000 7f6fcb579640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc415f280 con 0x7f6fc40b5ca0 2024-03-06T07:26:09.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.821+0000 7f6fc9b16640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6fc415f9d0 0x7f6fc40cdf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.821+0000 7f6fc9b16640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6fc415f9d0 0x7f6fc40cdf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52866/0 (socket says 172.21.15.83:52866) 2024-03-06T07:26:09.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.821+0000 7f6fc9b16640 1 -- 172.21.15.83:0/3062349633 learned_addr learned my addr 172.21.15.83:0/3062349633 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:09.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.821+0000 7f6fc9315640 1 --2- 172.21.15.83:0/3062349633 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6fc406be80 0x7f6fc41506a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.821+0000 7f6fc8b14640 1 --2- 172.21.15.83:0/3062349633 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6fc40b5ca0 0x7f6fc40d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.821+0000 7f6fc9b16640 1 -- 172.21.15.83:0/3062349633 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6fc40b5ca0 msgr2=0x7f6fc40d0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.821+0000 7f6fc9b16640 1 --2- 172.21.15.83:0/3062349633 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6fc40b5ca0 0x7f6fc40d0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.821+0000 7f6fc9b16640 1 -- 172.21.15.83:0/3062349633 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6fc406be80 msgr2=0x7f6fc41506a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.821+0000 7f6fc9b16640 1 --2- 172.21.15.83:0/3062349633 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6fc406be80 0x7f6fc41506a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.821+0000 7f6fc9b16640 1 -- 172.21.15.83:0/3062349633 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc40cce60 con 0x7f6fc415f9d0 2024-03-06T07:26:09.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.821+0000 7f6fc8b14640 1 --2- 172.21.15.83:0/3062349633 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6fc40b5ca0 0x7f6fc40d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:09.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.821+0000 7f6fc9315640 1 --2- 172.21.15.83:0/3062349633 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6fc406be80 0x7f6fc41506a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:09.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.821+0000 7f6fc9b16640 1 --2- 172.21.15.83:0/3062349633 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6fc415f9d0 0x7f6fc40cdf00 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc009f10 tx=0x7f6fbc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:09.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.821+0000 7f6fb9ffb640 1 -- 172.21.15.83:0/3062349633 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc043020 con 0x7f6fc415f9d0 2024-03-06T07:26:09.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.821+0000 7f6fb9ffb640 1 -- 172.21.15.83:0/3062349633 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fbc03e450 con 0x7f6fc415f9d0 2024-03-06T07:26:09.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.821+0000 7f6fb9ffb640 1 -- 172.21.15.83:0/3062349633 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc009510 con 0x7f6fc415f9d0 2024-03-06T07:26:09.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.821+0000 7f6fcb579640 1 -- 172.21.15.83:0/3062349633 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fc40cd140 con 0x7f6fc415f9d0 2024-03-06T07:26:09.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.822+0000 7f6fcb579640 1 -- 172.21.15.83:0/3062349633 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fc40cd700 con 0x7f6fc415f9d0 2024-03-06T07:26:09.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.823+0000 7f6fcb579640 1 -- 172.21.15.83:0/3062349633 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fc404f330 con 0x7f6fc415f9d0 2024-03-06T07:26:09.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.823+0000 7f6fb9ffb640 1 -- 172.21.15.83:0/3062349633 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6fbc046020 con 0x7f6fc415f9d0 2024-03-06T07:26:09.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.823+0000 7f6fb9ffb640 1 --2- 172.21.15.83:0/3062349633 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6fa806b4d0 0x7f6fa806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.824+0000 7f6fc9315640 1 --2- 172.21.15.83:0/3062349633 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6fa806b4d0 0x7f6fa806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.824+0000 7f6fb9ffb640 1 -- 172.21.15.83:0/3062349633 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f6fbc0aca70 con 0x7f6fc415f9d0 2024-03-06T07:26:09.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.824+0000 7f6fc9315640 1 --2- 172.21.15.83:0/3062349633 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6fa806b4d0 0x7f6fa806d990 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f6fc0006fd0 tx=0x7f6fc0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:09.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.828+0000 7f6fb9ffb640 1 -- 172.21.15.83:0/3062349633 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6fbc04d030 con 0x7f6fc415f9d0 2024-03-06T07:26:09.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:09.997+0000 7f6fcb579640 1 -- 172.21.15.83:0/3062349633 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6fc4071d90 con 0x7f6fc415f9d0 2024-03-06T07:26:10.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.652+0000 7f6fb9ffb640 1 -- 172.21.15.83:0/3062349633 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v84) v1 ==== 181+0+0 (secure 0 0 0) 0x7f6fbc07b890 con 0x7f6fc415f9d0 2024-03-06T07:26:10.655 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:26:10.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.655+0000 7f6fcb579640 1 -- 172.21.15.83:0/3062349633 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6fa806b4d0 msgr2=0x7f6fa806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.655+0000 7f6fcb579640 1 --2- 172.21.15.83:0/3062349633 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6fa806b4d0 0x7f6fa806d990 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f6fc0006fd0 tx=0x7f6fc0008040 comp rx=0 tx=0).stop 2024-03-06T07:26:10.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.655+0000 7f6fcb579640 1 -- 172.21.15.83:0/3062349633 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6fc415f9d0 msgr2=0x7f6fc40cdf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.655+0000 7f6fcb579640 1 --2- 172.21.15.83:0/3062349633 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6fc415f9d0 0x7f6fc40cdf00 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc009f10 tx=0x7f6fbc004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:10.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.656+0000 7f6fcb579640 1 -- 172.21.15.83:0/3062349633 shutdown_connections 2024-03-06T07:26:10.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.656+0000 7f6fcb579640 1 --2- 172.21.15.83:0/3062349633 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6fa806b4d0 0x7f6fa806d990 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.656+0000 7f6fcb579640 1 --2- 172.21.15.83:0/3062349633 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6fc415f9d0 0x7f6fc40cdf00 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.656+0000 7f6fcb579640 1 --2- 172.21.15.83:0/3062349633 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6fc40b5ca0 0x7f6fc40d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.656+0000 7f6fcb579640 1 --2- 172.21.15.83:0/3062349633 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6fc406be80 0x7f6fc41506a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.656+0000 7f6fcb579640 1 -- 172.21.15.83:0/3062349633 >> 172.21.15.83:0/3062349633 conn(0x7f6fc40b4ed0 msgr2=0x7f6fc40b5570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:10.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.656+0000 7f6fcb579640 1 -- 172.21.15.83:0/3062349633 shutdown_connections 2024-03-06T07:26:10.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.656+0000 7f6fcb579640 1 -- 172.21.15.83:0/3062349633 wait complete. 2024-03-06T07:26:10.672 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:26:10.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.779+0000 7f97f5a81640 1 Processor -- start 2024-03-06T07:26:10.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.779+0000 7f97f5a81640 1 -- start start 2024-03-06T07:26:10.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.779+0000 7f97f5a81640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97f00d29f0 0x7f97f0168ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.779+0000 7f97f5a81640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97f015e380 con 0x7f97f015dfa0 2024-03-06T07:26:10.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.779+0000 7f97f5a81640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97f00cf470 con 0x7f97f00d4630 2024-03-06T07:26:10.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.779+0000 7f97f5a81640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97f00d2100 con 0x7f97f00d29f0 2024-03-06T07:26:10.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.780+0000 7f97ef7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f97f015dfa0 0x7f97f00d18f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60374/0 (socket says 172.21.15.83:60374) 2024-03-06T07:26:10.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.780+0000 7f97ef7fe640 1 -- 172.21.15.83:0/3259384426 learned_addr learned my addr 172.21.15.83:0/3259384426 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:10.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.780+0000 7f97effff640 1 --2- 172.21.15.83:0/3259384426 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97f00d29f0 0x7f97f0168ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.780+0000 7f97ee7fc640 1 -- 172.21.15.83:0/3259384426 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1636360004 0 0) 0x7f97f015e380 con 0x7f97f015dfa0 2024-03-06T07:26:10.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.780+0000 7f97ee7fc640 1 -- 172.21.15.83:0/3259384426 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97d8003520 con 0x7f97f015dfa0 2024-03-06T07:26:10.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.780+0000 7f97ee7fc640 1 -- 172.21.15.83:0/3259384426 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 225079978 0 0) 0x7f97d8003520 con 0x7f97f015dfa0 2024-03-06T07:26:10.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.780+0000 7f97ee7fc640 1 -- 172.21.15.83:0/3259384426 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97f00d29f0 msgr2=0x7f97f0168ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:10.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.780+0000 7f97ee7fc640 1 --2- 172.21.15.83:0/3259384426 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97f00d29f0 0x7f97f0168ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.780+0000 7f97ee7fc640 1 -- 172.21.15.83:0/3259384426 >> v1:172.21.15.183:6789/0 conn(0x7f97f00d4630 legacy=0x7f97f00d22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.780+0000 7f97ee7fc640 1 -- 172.21.15.83:0/3259384426 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f00cf750 con 0x7f97f015dfa0 2024-03-06T07:26:10.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.780+0000 7f97ee7fc640 1 -- 172.21.15.83:0/3259384426 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f97e4001670 con 0x7f97f015dfa0 2024-03-06T07:26:10.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.780+0000 7f97ee7fc640 1 -- 172.21.15.83:0/3259384426 >> v1:172.21.15.83:6789/0 conn(0x7f97f015dfa0 legacy=0x7f97f00d18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.780+0000 7f97ee7fc640 1 --2- 172.21.15.83:0/3259384426 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97d80039e0 0x7f97d8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.780+0000 7f97ee7fc640 1 --2- 172.21.15.83:0/3259384426 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97f00d29f0 0x7f97d8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.780+0000 7f97ee7fc640 1 --2- 172.21.15.83:0/3259384426 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97d8004d40 0x7f97d80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.780+0000 7f97ee7fc640 1 -- 172.21.15.83:0/3259384426 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f00d2100 con 0x7f97d8004d40 2024-03-06T07:26:10.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.780+0000 7f97ee7fc640 1 -- 172.21.15.83:0/3259384426 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d800baa0 con 0x7f97d80039e0 2024-03-06T07:26:10.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.780+0000 7f97ee7fc640 1 -- 172.21.15.83:0/3259384426 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97d800bca0 con 0x7f97f00d29f0 2024-03-06T07:26:10.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.780+0000 7f97effff640 1 --2- 172.21.15.83:0/3259384426 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97d8004d40 0x7f97d80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.780+0000 7f97eeffd640 1 --2- 172.21.15.83:0/3259384426 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97f00d29f0 0x7f97d8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.780+0000 7f97ef7fe640 1 --2- 172.21.15.83:0/3259384426 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97d80039e0 0x7f97d8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.781+0000 7f97effff640 1 -- 172.21.15.83:0/3259384426 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97f00d29f0 msgr2=0x7f97d8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:10.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.781+0000 7f97effff640 1 --2- 172.21.15.83:0/3259384426 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97f00d29f0 0x7f97d8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.781+0000 7f97effff640 1 -- 172.21.15.83:0/3259384426 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97d80039e0 msgr2=0x7f97d8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:10.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.781+0000 7f97effff640 1 --2- 172.21.15.83:0/3259384426 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97d80039e0 0x7f97d8003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.781+0000 7f97effff640 1 -- 172.21.15.83:0/3259384426 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97d8003520 con 0x7f97d8004d40 2024-03-06T07:26:10.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.781+0000 7f97effff640 1 --2- 172.21.15.83:0/3259384426 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97d8004d40 0x7f97d80051c0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f97f008d530 tx=0x7f97d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=916e8fe01d979778 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.782+0000 7f97ee7fc640 1 -- 172.21.15.83:0/3259384426 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e8039070 con 0x7f97d8004d40 2024-03-06T07:26:10.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.782+0000 7f97ee7fc640 1 -- 172.21.15.83:0/3259384426 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97e802cb90 con 0x7f97d8004d40 2024-03-06T07:26:10.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.782+0000 7f97ee7fc640 1 -- 172.21.15.83:0/3259384426 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e8009c10 con 0x7f97d8004d40 2024-03-06T07:26:10.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.782+0000 7f97f5a81640 1 -- 172.21.15.83:0/3259384426 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97d8004d40 msgr2=0x7f97d80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.782+0000 7f97f5a81640 1 --2- 172.21.15.83:0/3259384426 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97d8004d40 0x7f97d80051c0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f97f008d530 tx=0x7f97d8005830 comp rx=0 tx=0).stop 2024-03-06T07:26:10.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.782+0000 7f97effff640 1 -- 172.21.15.83:0/3259384426 reap_dead start 2024-03-06T07:26:10.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.782+0000 7f97f5a81640 1 -- 172.21.15.83:0/3259384426 shutdown_connections 2024-03-06T07:26:10.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.782+0000 7f97f5a81640 1 -- 172.21.15.83:0/3259384426 >> 172.21.15.83:0/3259384426 conn(0x7f97f009d000 msgr2=0x7f97f00d3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:10.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.782+0000 7f97f5a81640 1 -- 172.21.15.83:0/3259384426 shutdown_connections 2024-03-06T07:26:10.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.782+0000 7f97f5a81640 1 -- 172.21.15.83:0/3259384426 wait complete. 2024-03-06T07:26:10.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.783+0000 7f97f5a81640 1 Processor -- start 2024-03-06T07:26:10.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.784+0000 7f97f5a81640 1 -- start start 2024-03-06T07:26:10.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.784+0000 7f97f5a81640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97f00b3fa0 0x7f97f01508a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.784+0000 7f97f5a81640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97f00b4380 0x7f97f0150de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.784+0000 7f97f5a81640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97f00c9880 0x7f97f00c9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.784+0000 7f97f5a81640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f00cc320 con 0x7f97f00b4380 2024-03-06T07:26:10.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.784+0000 7f97f5a81640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f00cc1a0 con 0x7f97f00b3fa0 2024-03-06T07:26:10.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.784+0000 7f97f5a81640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97f00cc4a0 con 0x7f97f00c9880 2024-03-06T07:26:10.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.784+0000 7f97eeffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97f00b4380 0x7f97f0150de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.784+0000 7f97eeffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97f00b4380 0x7f97f0150de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52888/0 (socket says 172.21.15.83:52888) 2024-03-06T07:26:10.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.784+0000 7f97eeffd640 1 -- 172.21.15.83:0/3981419215 learned_addr learned my addr 172.21.15.83:0/3981419215 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:10.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.785+0000 7f97effff640 1 --2- 172.21.15.83:0/3981419215 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97f00c9880 0x7f97f00c9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.785+0000 7f97ef7fe640 1 --2- 172.21.15.83:0/3981419215 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97f00b3fa0 0x7f97f01508a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.785+0000 7f97eeffd640 1 -- 172.21.15.83:0/3981419215 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97f00c9880 msgr2=0x7f97f00c9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.785+0000 7f97eeffd640 1 --2- 172.21.15.83:0/3981419215 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97f00c9880 0x7f97f00c9d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.785+0000 7f97eeffd640 1 -- 172.21.15.83:0/3981419215 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97f00b3fa0 msgr2=0x7f97f01508a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.785+0000 7f97eeffd640 1 --2- 172.21.15.83:0/3981419215 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97f00b3fa0 0x7f97f01508a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.785+0000 7f97eeffd640 1 -- 172.21.15.83:0/3981419215 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f00ca620 con 0x7f97f00b4380 2024-03-06T07:26:10.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.785+0000 7f97eeffd640 1 --2- 172.21.15.83:0/3981419215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97f00b4380 0x7f97f0150de0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f97e000f750 tx=0x7f97e000fc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.785+0000 7f97d7fff640 1 -- 172.21.15.83:0/3981419215 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e000fe60 con 0x7f97f00b4380 2024-03-06T07:26:10.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.785+0000 7f97d7fff640 1 -- 172.21.15.83:0/3981419215 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97e0018070 con 0x7f97f00b4380 2024-03-06T07:26:10.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.785+0000 7f97f5a81640 1 -- 172.21.15.83:0/3981419215 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97f0151560 con 0x7f97f00b4380 2024-03-06T07:26:10.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.785+0000 7f97d7fff640 1 -- 172.21.15.83:0/3981419215 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e000d730 con 0x7f97f00b4380 2024-03-06T07:26:10.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.785+0000 7f97f5a81640 1 -- 172.21.15.83:0/3981419215 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97f0151b20 con 0x7f97f00b4380 2024-03-06T07:26:10.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.786+0000 7f97f5a81640 1 -- 172.21.15.83:0/3981419215 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97f004f330 con 0x7f97f00b4380 2024-03-06T07:26:10.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.786+0000 7f97d7fff640 1 -- 172.21.15.83:0/3981419215 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f97e000d8d0 con 0x7f97f00b4380 2024-03-06T07:26:10.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.787+0000 7f97d7fff640 1 --2- 172.21.15.83:0/3981419215 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f97dc06b4d0 0x7f97dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.787+0000 7f97ef7fe640 1 --2- 172.21.15.83:0/3981419215 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f97dc06b4d0 0x7f97dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.787+0000 7f97d7fff640 1 -- 172.21.15.83:0/3981419215 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5357+0+0 (secure 0 0 0) 0x7f97e008a0a0 con 0x7f97f00b4380 2024-03-06T07:26:10.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.787+0000 7f97ef7fe640 1 --2- 172.21.15.83:0/3981419215 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f97dc06b4d0 0x7f97dc06d990 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f97e4004840 tx=0x7f97e4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.791+0000 7f97d7fff640 1 -- 172.21.15.83:0/3981419215 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f97e001a310 con 0x7f97f00b4380 2024-03-06T07:26:10.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:10.969+0000 7f97f5a81640 1 -- 172.21.15.83:0/3981419215 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f97f00b4b40 con 0x7f97f00b4380 2024-03-06T07:26:11.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.654+0000 7f97d7fff640 1 -- 172.21.15.83:0/3981419215 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v85) v1 ==== 169+0+0 (secure 0 0 0) 0x7f97e0059080 con 0x7f97f00b4380 2024-03-06T07:26:11.656 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'cephfs_data' removed 2024-03-06T07:26:11.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.657+0000 7f97f5a81640 1 -- 172.21.15.83:0/3981419215 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f97dc06b4d0 msgr2=0x7f97dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.657+0000 7f97f5a81640 1 --2- 172.21.15.83:0/3981419215 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f97dc06b4d0 0x7f97dc06d990 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f97e4004840 tx=0x7f97e4009040 comp rx=0 tx=0).stop 2024-03-06T07:26:11.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.657+0000 7f97f5a81640 1 -- 172.21.15.83:0/3981419215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97f00b4380 msgr2=0x7f97f0150de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.657+0000 7f97f5a81640 1 --2- 172.21.15.83:0/3981419215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97f00b4380 0x7f97f0150de0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f97e000f750 tx=0x7f97e000fc10 comp rx=0 tx=0).stop 2024-03-06T07:26:11.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.657+0000 7f97f5a81640 1 -- 172.21.15.83:0/3981419215 shutdown_connections 2024-03-06T07:26:11.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.657+0000 7f97f5a81640 1 --2- 172.21.15.83:0/3981419215 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f97dc06b4d0 0x7f97dc06d990 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.657+0000 7f97f5a81640 1 --2- 172.21.15.83:0/3981419215 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97f00c9880 0x7f97f00c9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.657+0000 7f97f5a81640 1 --2- 172.21.15.83:0/3981419215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97f00b4380 0x7f97f0150de0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.657+0000 7f97f5a81640 1 --2- 172.21.15.83:0/3981419215 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97f00b3fa0 0x7f97f01508a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.657+0000 7f97f5a81640 1 -- 172.21.15.83:0/3981419215 >> 172.21.15.83:0/3981419215 conn(0x7f97f009d000 msgr2=0x7f97f00b8920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:11.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.658+0000 7f97f5a81640 1 -- 172.21.15.83:0/3981419215 shutdown_connections 2024-03-06T07:26:11.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.658+0000 7f97f5a81640 1 -- 172.21.15.83:0/3981419215 wait complete. 2024-03-06T07:26:11.673 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:26:11.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.775+0000 7f48a17c7640 1 Processor -- start 2024-03-06T07:26:11.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.775+0000 7f48a17c7640 1 -- start start 2024-03-06T07:26:11.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.775+0000 7f48a17c7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f489c1614d0 0x7f489c0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.775+0000 7f48a17c7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f489c0d2480 0x7f489c0b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.775+0000 7f48a17c7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f489c1618b0 con 0x7f489c0d1910 2024-03-06T07:26:11.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.776+0000 7f48a17c7640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f489c0cf400 con 0x7f489c0d2480 2024-03-06T07:26:11.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.776+0000 7f48a17c7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f489c0cf6a0 con 0x7f489c1614d0 2024-03-06T07:26:11.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.776+0000 7f489b7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f489c0d2480 0x7f489c0b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.776+0000 7f489b7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f489c0d2480 0x7f489c0b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52900/0 (socket says 172.21.15.83:52900) 2024-03-06T07:26:11.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.776+0000 7f4893fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f489c0d1910 0x7f489c0d1d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60376/0 (socket says 172.21.15.83:60376) 2024-03-06T07:26:11.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.776+0000 7f489b7fe640 1 -- 172.21.15.83:0/2882705702 learned_addr learned my addr 172.21.15.83:0/2882705702 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:11.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.776+0000 7f489affd640 1 --2- 172.21.15.83:0/2882705702 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f489c1614d0 0x7f489c0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.776+0000 7f489b7fe640 1 -- 172.21.15.83:0/2882705702 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f489c1614d0 msgr2=0x7f489c0d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:11.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.776+0000 7f489b7fe640 1 --2- 172.21.15.83:0/2882705702 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f489c1614d0 0x7f489c0d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.776+0000 7f489b7fe640 1 -- 172.21.15.83:0/2882705702 >> v1:172.21.15.83:6789/0 conn(0x7f489c0d1910 legacy=0x7f489c0d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.776+0000 7f489b7fe640 1 -- 172.21.15.83:0/2882705702 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f489c0cf980 con 0x7f489c0d2480 2024-03-06T07:26:11.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.776+0000 7f489affd640 1 --2- 172.21.15.83:0/2882705702 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f489c1614d0 0x7f489c0d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:11.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.777+0000 7f489b7fe640 1 --2- 172.21.15.83:0/2882705702 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f489c0d2480 0x7f489c0b0e30 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f4894009e50 tx=0x7f489402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d1104105c006035 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:11.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.777+0000 7f489a7fc640 1 -- 172.21.15.83:0/2882705702 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489402cb00 con 0x7f489c0d2480 2024-03-06T07:26:11.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.777+0000 7f489a7fc640 1 -- 172.21.15.83:0/2882705702 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f489402cca0 con 0x7f489c0d2480 2024-03-06T07:26:11.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.777+0000 7f489a7fc640 1 -- 172.21.15.83:0/2882705702 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4894034610 con 0x7f489c0d2480 2024-03-06T07:26:11.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.777+0000 7f48a17c7640 1 -- 172.21.15.83:0/2882705702 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f489c0d2480 msgr2=0x7f489c0b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.777+0000 7f48a17c7640 1 --2- 172.21.15.83:0/2882705702 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f489c0d2480 0x7f489c0b0e30 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f4894009e50 tx=0x7f489402bf60 comp rx=0 tx=0).stop 2024-03-06T07:26:11.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.777+0000 7f48a17c7640 1 -- 172.21.15.83:0/2882705702 shutdown_connections 2024-03-06T07:26:11.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.777+0000 7f48a17c7640 1 --2- 172.21.15.83:0/2882705702 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f489c0d2480 0x7f489c0b0e30 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.777+0000 7f48a17c7640 1 --2- 172.21.15.83:0/2882705702 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f489c1614d0 0x7f489c0d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.777+0000 7f48a17c7640 1 -- 172.21.15.83:0/2882705702 >> 172.21.15.83:0/2882705702 conn(0x7f489c0d3d40 msgr2=0x7f489c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:11.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.777+0000 7f48a17c7640 1 -- 172.21.15.83:0/2882705702 shutdown_connections 2024-03-06T07:26:11.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.777+0000 7f48a17c7640 1 -- 172.21.15.83:0/2882705702 wait complete. 2024-03-06T07:26:11.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.779+0000 7f48a17c7640 1 Processor -- start 2024-03-06T07:26:11.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.779+0000 7f48a17c7640 1 -- start start 2024-03-06T07:26:11.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.779+0000 7f48a17c7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f489c0d1910 0x7f489c15bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.779+0000 7f48a17c7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f489c0d2480 0x7f489c0cd680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.779+0000 7f48a17c7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f489c1614d0 0x7f489c0c89b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.779+0000 7f48a17c7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f489c0b1370 con 0x7f489c0d1910 2024-03-06T07:26:11.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.779+0000 7f48a17c7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f489c0b0910 con 0x7f489c0d2480 2024-03-06T07:26:11.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.779+0000 7f48a17c7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f489c0b14f0 con 0x7f489c1614d0 2024-03-06T07:26:11.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.779+0000 7f489affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f489c0d1910 0x7f489c15bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.779+0000 7f489affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f489c0d1910 0x7f489c15bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52906/0 (socket says 172.21.15.83:52906) 2024-03-06T07:26:11.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.779+0000 7f489affd640 1 -- 172.21.15.83:0/3095621079 learned_addr learned my addr 172.21.15.83:0/3095621079 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:11.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.780+0000 7f489b7fe640 1 --2- 172.21.15.83:0/3095621079 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f489c1614d0 0x7f489c0c89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.780+0000 7f4893fff640 1 --2- 172.21.15.83:0/3095621079 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f489c0d2480 0x7f489c0cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.780+0000 7f489affd640 1 -- 172.21.15.83:0/3095621079 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f489c1614d0 msgr2=0x7f489c0c89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.780+0000 7f489affd640 1 --2- 172.21.15.83:0/3095621079 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f489c1614d0 0x7f489c0c89b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.780+0000 7f489affd640 1 -- 172.21.15.83:0/3095621079 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f489c0d2480 msgr2=0x7f489c0cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.780+0000 7f489affd640 1 --2- 172.21.15.83:0/3095621079 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f489c0d2480 0x7f489c0cd680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.780+0000 7f489affd640 1 -- 172.21.15.83:0/3095621079 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f489c0c7910 con 0x7f489c0d1910 2024-03-06T07:26:11.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.780+0000 7f489b7fe640 1 --2- 172.21.15.83:0/3095621079 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f489c1614d0 0x7f489c0c89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:11.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.780+0000 7f489affd640 1 --2- 172.21.15.83:0/3095621079 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f489c0d1910 0x7f489c15bbe0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f488c00c7f0 tx=0x7f488c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:11.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.780+0000 7f48937fe640 1 -- 172.21.15.83:0/3095621079 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f488c015070 con 0x7f489c0d1910 2024-03-06T07:26:11.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.780+0000 7f48937fe640 1 -- 172.21.15.83:0/3095621079 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f488c007820 con 0x7f489c0d1910 2024-03-06T07:26:11.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.780+0000 7f48a17c7640 1 -- 172.21.15.83:0/3095621079 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f489c0c7bf0 con 0x7f489c0d1910 2024-03-06T07:26:11.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.780+0000 7f48937fe640 1 -- 172.21.15.83:0/3095621079 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f488c007bd0 con 0x7f489c0d1910 2024-03-06T07:26:11.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.780+0000 7f48a17c7640 1 -- 172.21.15.83:0/3095621079 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f489c0c81b0 con 0x7f489c0d1910 2024-03-06T07:26:11.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.781+0000 7f48a17c7640 1 -- 172.21.15.83:0/3095621079 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f489c15f870 con 0x7f489c0d1910 2024-03-06T07:26:11.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.781+0000 7f48937fe640 1 -- 172.21.15.83:0/3095621079 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f488c018070 con 0x7f489c0d1910 2024-03-06T07:26:11.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.782+0000 7f48937fe640 1 --2- 172.21.15.83:0/3095621079 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f487c06b500 0x7f487c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.782+0000 7f48937fe640 1 -- 172.21.15.83:0/3095621079 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f488c089fb0 con 0x7f489c0d1910 2024-03-06T07:26:11.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.782+0000 7f4893fff640 1 --2- 172.21.15.83:0/3095621079 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f487c06b500 0x7f487c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.782+0000 7f4893fff640 1 --2- 172.21.15.83:0/3095621079 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f487c06b500 0x7f487c06d9c0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f489c0d0920 tx=0x7f4884007cc0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:11.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.786+0000 7f48937fe640 1 -- 172.21.15.83:0/3095621079 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f488c059130 con 0x7f489c0d1910 2024-03-06T07:26:11.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.958+0000 7f48a17c7640 1 -- 172.21.15.83:0/3095621079 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f489c0ce170 con 0x7f489c0d1910 2024-03-06T07:26:11.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.961+0000 7f48937fe640 1 -- 172.21.15.83:0/3095621079 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v27) v1 ==== 104+0+0 (secure 0 0 0) 0x7f488c058a30 con 0x7f489c0d1910 2024-03-06T07:26:11.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.963+0000 7f48a17c7640 1 -- 172.21.15.83:0/3095621079 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f487c06b500 msgr2=0x7f487c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.963+0000 7f48a17c7640 1 --2- 172.21.15.83:0/3095621079 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f487c06b500 0x7f487c06d9c0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f489c0d0920 tx=0x7f4884007cc0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.963+0000 7f48a17c7640 1 -- 172.21.15.83:0/3095621079 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f489c0d1910 msgr2=0x7f489c15bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.963+0000 7f48a17c7640 1 --2- 172.21.15.83:0/3095621079 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f489c0d1910 0x7f489c15bbe0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f488c00c7f0 tx=0x7f488c00ccb0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.963+0000 7f48a17c7640 1 -- 172.21.15.83:0/3095621079 shutdown_connections 2024-03-06T07:26:11.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.963+0000 7f48a17c7640 1 --2- 172.21.15.83:0/3095621079 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f487c06b500 0x7f487c06d9c0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.963+0000 7f48a17c7640 1 --2- 172.21.15.83:0/3095621079 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f489c1614d0 0x7f489c0c89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.963+0000 7f48a17c7640 1 --2- 172.21.15.83:0/3095621079 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f489c0d2480 0x7f489c0cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.963+0000 7f48a17c7640 1 --2- 172.21.15.83:0/3095621079 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f489c0d1910 0x7f489c15bbe0 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.963+0000 7f48a17c7640 1 -- 172.21.15.83:0/3095621079 >> 172.21.15.83:0/3095621079 conn(0x7f489c0d3d40 msgr2=0x7f489c0d0e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:11.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.963+0000 7f48a17c7640 1 -- 172.21.15.83:0/3095621079 shutdown_connections 2024-03-06T07:26:11.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:11.963+0000 7f48a17c7640 1 -- 172.21.15.83:0/3095621079 wait complete. 2024-03-06T07:26:11.976 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_fancy_layout' 2024-03-06T07:26:12.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.074+0000 7fdb880d4640 1 Processor -- start 2024-03-06T07:26:12.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.074+0000 7fdb880d4640 1 -- start start 2024-03-06T07:26:12.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.075+0000 7fdb880d4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fdb800b3060 0x7fdb8015e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.075+0000 7fdb880d4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdb8015bf30 0x7fdb8015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.075+0000 7fdb880d4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb8015f2d0 con 0x7fdb8013dc10 2024-03-06T07:26:12.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.075+0000 7fdb880d4640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb800cf3f0 con 0x7fdb800b3060 2024-03-06T07:26:12.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.075+0000 7fdb880d4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8015b210 con 0x7fdb8015bf30 2024-03-06T07:26:12.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.075+0000 7fdb8566f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fdb800b3060 0x7fdb8015e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.075+0000 7fdb85e70640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fdb8013dc10 0x7fdb800b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60380/0 (socket says 172.21.15.83:60380) 2024-03-06T07:26:12.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.075+0000 7fdb85e70640 1 -- 172.21.15.83:0/3834404521 learned_addr learned my addr 172.21.15.83:0/3834404521 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:12.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.075+0000 7fdb8566f640 1 -- 172.21.15.83:0/3834404521 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdb8015bf30 msgr2=0x7fdb8015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:12.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.075+0000 7fdb8566f640 1 --2- 172.21.15.83:0/3834404521 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdb8015bf30 0x7fdb8015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.075+0000 7fdb8566f640 1 -- 172.21.15.83:0/3834404521 >> v1:172.21.15.83:6789/0 conn(0x7fdb8013dc10 legacy=0x7fdb800b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.075+0000 7fdb8566f640 1 -- 172.21.15.83:0/3834404521 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb8015f580 con 0x7fdb800b3060 2024-03-06T07:26:12.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.075+0000 7fdb84e6e640 1 -- 172.21.15.83:0/3834404521 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3252282108 0 0) 0x7fdb8015f2d0 con 0x7fdb8013dc10 2024-03-06T07:26:12.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.076+0000 7fdb8566f640 1 --2- 172.21.15.83:0/3834404521 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb800b3060 0x7fdb8015e850 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fdb7000ba40 tx=0x7fdb7002dff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c155465f85ac8ce8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:12.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.076+0000 7fdb84e6e640 1 -- 172.21.15.83:0/3834404521 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb7003b070 con 0x7fdb800b3060 2024-03-06T07:26:12.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.076+0000 7fdb84e6e640 1 -- 172.21.15.83:0/3834404521 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb7002eda0 con 0x7fdb800b3060 2024-03-06T07:26:12.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.076+0000 7fdb84e6e640 1 -- 172.21.15.83:0/3834404521 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb70036660 con 0x7fdb800b3060 2024-03-06T07:26:12.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.076+0000 7fdb880d4640 1 -- 172.21.15.83:0/3834404521 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb800b3060 msgr2=0x7fdb8015e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.076+0000 7fdb880d4640 1 --2- 172.21.15.83:0/3834404521 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb800b3060 0x7fdb8015e850 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fdb7000ba40 tx=0x7fdb7002dff0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.076+0000 7fdb880d4640 1 -- 172.21.15.83:0/3834404521 shutdown_connections 2024-03-06T07:26:12.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.076+0000 7fdb880d4640 1 --2- 172.21.15.83:0/3834404521 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdb8015bf30 0x7fdb8015ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.076+0000 7fdb880d4640 1 --2- 172.21.15.83:0/3834404521 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb800b3060 0x7fdb8015e850 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.076+0000 7fdb880d4640 1 -- 172.21.15.83:0/3834404521 >> 172.21.15.83:0/3834404521 conn(0x7fdb800cf8a0 msgr2=0x7fdb800cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:12.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.077+0000 7fdb880d4640 1 -- 172.21.15.83:0/3834404521 shutdown_connections 2024-03-06T07:26:12.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.077+0000 7fdb880d4640 1 -- 172.21.15.83:0/3834404521 wait complete. 2024-03-06T07:26:12.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.078+0000 7fdb880d4640 1 Processor -- start 2024-03-06T07:26:12.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.078+0000 7fdb880d4640 1 -- start start 2024-03-06T07:26:12.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.078+0000 7fdb880d4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb800b3060 0x7fdb80139e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.078+0000 7fdb880d4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdb8013dc10 0x7fdb80152f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.078+0000 7fdb880d4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdb8015bf30 0x7fdb80140770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.078+0000 7fdb880d4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8013dff0 con 0x7fdb800b3060 2024-03-06T07:26:12.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.078+0000 7fdb880d4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8015b730 con 0x7fdb8013dc10 2024-03-06T07:26:12.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.078+0000 7fdb880d4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb800cd7a0 con 0x7fdb8015bf30 2024-03-06T07:26:12.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.078+0000 7fdb85e70640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb800b3060 0x7fdb80139e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.078+0000 7fdb85e70640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb800b3060 0x7fdb80139e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52930/0 (socket says 172.21.15.83:52930) 2024-03-06T07:26:12.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.078+0000 7fdb85e70640 1 -- 172.21.15.83:0/95630326 learned_addr learned my addr 172.21.15.83:0/95630326 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:12.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.079+0000 7fdb8566f640 1 --2- 172.21.15.83:0/95630326 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdb8013dc10 0x7fdb80152f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.079+0000 7fdb86671640 1 --2- 172.21.15.83:0/95630326 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdb8015bf30 0x7fdb80140770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.079+0000 7fdb85e70640 1 -- 172.21.15.83:0/95630326 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdb8015bf30 msgr2=0x7fdb80140770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.079+0000 7fdb85e70640 1 --2- 172.21.15.83:0/95630326 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdb8015bf30 0x7fdb80140770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.079+0000 7fdb85e70640 1 -- 172.21.15.83:0/95630326 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdb8013dc10 msgr2=0x7fdb80152f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.079+0000 7fdb85e70640 1 --2- 172.21.15.83:0/95630326 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdb8013dc10 0x7fdb80152f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.079+0000 7fdb85e70640 1 -- 172.21.15.83:0/95630326 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb80153ca0 con 0x7fdb800b3060 2024-03-06T07:26:12.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.079+0000 7fdb86671640 1 --2- 172.21.15.83:0/95630326 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdb8015bf30 0x7fdb80140770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:12.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.079+0000 7fdb8566f640 1 --2- 172.21.15.83:0/95630326 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdb8013dc10 0x7fdb80152f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:12.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.079+0000 7fdb85e70640 1 --2- 172.21.15.83:0/95630326 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb800b3060 0x7fdb80139e00 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c00c4d0 tx=0x7fdb7c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:12.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.079+0000 7fdb767fc640 1 -- 172.21.15.83:0/95630326 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c018070 con 0x7fdb800b3060 2024-03-06T07:26:12.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.079+0000 7fdb767fc640 1 -- 172.21.15.83:0/95630326 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb7c007500 con 0x7fdb800b3060 2024-03-06T07:26:12.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.079+0000 7fdb880d4640 1 -- 172.21.15.83:0/95630326 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb801539c0 con 0x7fdb800b3060 2024-03-06T07:26:12.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.079+0000 7fdb767fc640 1 -- 172.21.15.83:0/95630326 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c004a20 con 0x7fdb800b3060 2024-03-06T07:26:12.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.079+0000 7fdb880d4640 1 -- 172.21.15.83:0/95630326 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb80071220 con 0x7fdb800b3060 2024-03-06T07:26:12.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.080+0000 7fdb880d4640 1 -- 172.21.15.83:0/95630326 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb8004f330 con 0x7fdb800b3060 2024-03-06T07:26:12.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.081+0000 7fdb767fc640 1 -- 172.21.15.83:0/95630326 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdb7c01b070 con 0x7fdb800b3060 2024-03-06T07:26:12.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.081+0000 7fdb767fc640 1 --2- 172.21.15.83:0/95630326 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdb6406b4d0 0x7fdb6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.081+0000 7fdb767fc640 1 -- 172.21.15.83:0/95630326 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fdb7c089190 con 0x7fdb800b3060 2024-03-06T07:26:12.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.081+0000 7fdb8566f640 1 --2- 172.21.15.83:0/95630326 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdb6406b4d0 0x7fdb6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.081+0000 7fdb8566f640 1 --2- 172.21.15.83:0/95630326 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdb6406b4d0 0x7fdb6406d990 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fdb70038000 tx=0x7fdb70032000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:12.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.085+0000 7fdb767fc640 1 -- 172.21.15.83:0/95630326 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdb80153ca0 con 0x7fdb800b3060 2024-03-06T07:26:12.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.323+0000 7fdb880d4640 1 -- 172.21.15.83:0/95630326 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]} v 0) v1 -- 0x7fdb800b04c0 con 0x7fdb800b3060 2024-03-06T07:26:12.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.660+0000 7fdb767fc640 1 -- 172.21.15.83:0/95630326 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]}]=0 v244) v1 ==== 132+0+0 (secure 0 0 0) 0x7fdb7c058310 con 0x7fdb800b3060 2024-03-06T07:26:12.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.662+0000 7fdb880d4640 1 -- 172.21.15.83:0/95630326 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdb6406b4d0 msgr2=0x7fdb6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.662+0000 7fdb880d4640 1 --2- 172.21.15.83:0/95630326 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdb6406b4d0 0x7fdb6406d990 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fdb70038000 tx=0x7fdb70032000 comp rx=0 tx=0).stop 2024-03-06T07:26:12.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.662+0000 7fdb880d4640 1 -- 172.21.15.83:0/95630326 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb800b3060 msgr2=0x7fdb80139e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.662+0000 7fdb880d4640 1 --2- 172.21.15.83:0/95630326 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb800b3060 0x7fdb80139e00 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c00c4d0 tx=0x7fdb7c00c990 comp rx=0 tx=0).stop 2024-03-06T07:26:12.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.662+0000 7fdb880d4640 1 -- 172.21.15.83:0/95630326 shutdown_connections 2024-03-06T07:26:12.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.662+0000 7fdb880d4640 1 --2- 172.21.15.83:0/95630326 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdb6406b4d0 0x7fdb6406d990 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.662+0000 7fdb880d4640 1 --2- 172.21.15.83:0/95630326 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdb8015bf30 0x7fdb80140770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.662+0000 7fdb880d4640 1 --2- 172.21.15.83:0/95630326 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdb8013dc10 0x7fdb80152f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.662+0000 7fdb880d4640 1 --2- 172.21.15.83:0/95630326 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb800b3060 0x7fdb80139e00 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.662+0000 7fdb880d4640 1 -- 172.21.15.83:0/95630326 >> 172.21.15.83:0/95630326 conn(0x7fdb800cf8a0 msgr2=0x7fdb800b0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:12.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.662+0000 7fdb880d4640 1 -- 172.21.15.83:0/95630326 shutdown_connections 2024-03-06T07:26:12.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.663+0000 7fdb880d4640 1 -- 172.21.15.83:0/95630326 wait complete. 2024-03-06T07:26:12.675 INFO:tasks.cephfs_test_runner:purge stray file with fancy layout ... ok 2024-03-06T07:26:12.676 INFO:tasks.cephfs_test_runner:Starting test: test_files_throttle (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:26:12.676 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:12.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.774+0000 7f8e5e7f7640 1 Processor -- start 2024-03-06T07:26:12.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.774+0000 7f8e5e7f7640 1 -- start start 2024-03-06T07:26:12.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.775+0000 7f8e5e7f7640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f8e580cf790 0x7f8e580ad860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.775+0000 7f8e5e7f7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e580ae240 con 0x7f8e580723f0 2024-03-06T07:26:12.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.775+0000 7f8e5e7f7640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e580ae520 con 0x7f8e580adda0 2024-03-06T07:26:12.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.775+0000 7f8e5e7f7640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e5806cf70 con 0x7f8e580cf790 2024-03-06T07:26:12.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.775+0000 7f8e5cd94640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f8e580723f0 0x7f8e5806d400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60386/0 (socket says 172.21.15.83:60386) 2024-03-06T07:26:12.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.775+0000 7f8e5cd94640 1 -- 172.21.15.83:0/908715527 learned_addr learned my addr 172.21.15.83:0/908715527 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:12.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.775+0000 7f8e56ffd640 1 -- 172.21.15.83:0/908715527 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3466102748 0 0) 0x7f8e580ae240 con 0x7f8e580723f0 2024-03-06T07:26:12.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.775+0000 7f8e56ffd640 1 -- 172.21.15.83:0/908715527 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e40003520 con 0x7f8e580723f0 2024-03-06T07:26:12.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.775+0000 7f8e57fff640 1 --2- 172.21.15.83:0/908715527 >> v2:172.21.15.183:3300/0 conn(0x7f8e580cf790 0x7f8e580ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.776+0000 7f8e56ffd640 1 -- 172.21.15.83:0/908715527 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1389194556 0 0) 0x7f8e40003520 con 0x7f8e580723f0 2024-03-06T07:26:12.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.776+0000 7f8e56ffd640 1 -- 172.21.15.83:0/908715527 >> v2:172.21.15.183:3300/0 conn(0x7f8e580cf790 msgr2=0x7f8e580ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:12.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.776+0000 7f8e56ffd640 1 --2- 172.21.15.83:0/908715527 >> v2:172.21.15.183:3300/0 conn(0x7f8e580cf790 0x7f8e580ad860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.776+0000 7f8e56ffd640 1 -- 172.21.15.83:0/908715527 >> v1:172.21.15.183:6789/0 conn(0x7f8e580adda0 legacy=0x7f8e58071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:12.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.776+0000 7f8e56ffd640 1 -- 172.21.15.83:0/908715527 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e580ae800 con 0x7f8e580723f0 2024-03-06T07:26:12.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.776+0000 7f8e56ffd640 1 -- 172.21.15.83:0/908715527 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f8e50001a10 con 0x7f8e580723f0 2024-03-06T07:26:12.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.776+0000 7f8e56ffd640 1 -- 172.21.15.83:0/908715527 >> v1:172.21.15.83:6789/0 conn(0x7f8e580723f0 legacy=0x7f8e5806d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.776+0000 7f8e56ffd640 1 --2- 172.21.15.83:0/908715527 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8e400039b0 0x7f8e40003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.776+0000 7f8e56ffd640 1 --2- 172.21.15.83:0/908715527 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8e40004800 0x7f8e40004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.776+0000 7f8e56ffd640 1 --2- 172.21.15.83:0/908715527 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e400051c0 0x7f8e4000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.776+0000 7f8e56ffd640 1 -- 172.21.15.83:0/908715527 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5806cf70 con 0x7f8e400051c0 2024-03-06T07:26:12.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.776+0000 7f8e56ffd640 1 -- 172.21.15.83:0/908715527 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4000c110 con 0x7f8e40004800 2024-03-06T07:26:12.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.776+0000 7f8e56ffd640 1 -- 172.21.15.83:0/908715527 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4000c310 con 0x7f8e400039b0 2024-03-06T07:26:12.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.776+0000 7f8e5cd94640 1 --2- 172.21.15.83:0/908715527 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e400051c0 0x7f8e4000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.776+0000 7f8e57fff640 1 --2- 172.21.15.83:0/908715527 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8e400039b0 0x7f8e40003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.776+0000 7f8e577fe640 1 --2- 172.21.15.83:0/908715527 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8e40004800 0x7f8e40004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.776+0000 7f8e5cd94640 1 -- 172.21.15.83:0/908715527 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8e400039b0 msgr2=0x7f8e40003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:12.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.776+0000 7f8e5cd94640 1 --2- 172.21.15.83:0/908715527 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8e400039b0 0x7f8e40003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.776+0000 7f8e5cd94640 1 -- 172.21.15.83:0/908715527 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8e40004800 msgr2=0x7f8e40004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:12.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.776+0000 7f8e5cd94640 1 --2- 172.21.15.83:0/908715527 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8e40004800 0x7f8e40004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.776+0000 7f8e5cd94640 1 -- 172.21.15.83:0/908715527 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e40003520 con 0x7f8e400051c0 2024-03-06T07:26:12.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.777+0000 7f8e5cd94640 1 -- 172.21.15.83:0/908715527 reap_dead start 2024-03-06T07:26:12.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.777+0000 7f8e5cd94640 1 --2- 172.21.15.83:0/908715527 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e400051c0 0x7f8e4000b9c0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f8e58074940 tx=0x7f8e40005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11aae7bb0ffc00d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:12.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.777+0000 7f8e56ffd640 1 -- 172.21.15.83:0/908715527 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e5000a480 con 0x7f8e400051c0 2024-03-06T07:26:12.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.777+0000 7f8e56ffd640 1 -- 172.21.15.83:0/908715527 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e50034650 con 0x7f8e400051c0 2024-03-06T07:26:12.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.777+0000 7f8e56ffd640 1 -- 172.21.15.83:0/908715527 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e50034a40 con 0x7f8e400051c0 2024-03-06T07:26:12.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.777+0000 7f8e5e7f7640 1 -- 172.21.15.83:0/908715527 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e400051c0 msgr2=0x7f8e4000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.777+0000 7f8e5e7f7640 1 --2- 172.21.15.83:0/908715527 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e400051c0 0x7f8e4000b9c0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f8e58074940 tx=0x7f8e40005830 comp rx=0 tx=0).stop 2024-03-06T07:26:12.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.777+0000 7f8e5e7f7640 1 -- 172.21.15.83:0/908715527 shutdown_connections 2024-03-06T07:26:12.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.777+0000 7f8e5e7f7640 1 --2- 172.21.15.83:0/908715527 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e400051c0 0x7f8e4000b9c0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.777+0000 7f8e5e7f7640 1 -- 172.21.15.83:0/908715527 >> 172.21.15.83:0/908715527 conn(0x7f8e580b1a90 msgr2=0x7f8e580b1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:12.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.778+0000 7f8e5e7f7640 1 -- 172.21.15.83:0/908715527 shutdown_connections 2024-03-06T07:26:12.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.778+0000 7f8e5e7f7640 1 -- 172.21.15.83:0/908715527 wait complete. 2024-03-06T07:26:12.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.779+0000 7f8e5e7f7640 1 Processor -- start 2024-03-06T07:26:12.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.779+0000 7f8e5e7f7640 1 -- start start 2024-03-06T07:26:12.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.780+0000 7f8e5e7f7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8e400051c0 0x7f8e581426d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.780+0000 7f8e5e7f7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8e58143260 0x7f8e58155750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.780+0000 7f8e5e7f7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e58155c90 0x7f8e580b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.780+0000 7f8e5e7f7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5815e700 con 0x7f8e58155c90 2024-03-06T07:26:12.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.780+0000 7f8e5e7f7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5815e580 con 0x7f8e400051c0 2024-03-06T07:26:12.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.780+0000 7f8e5e7f7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e580ad860 con 0x7f8e58143260 2024-03-06T07:26:12.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.780+0000 7f8e5cd94640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e58155c90 0x7f8e580b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.780+0000 7f8e5cd94640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e58155c90 0x7f8e580b36d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52952/0 (socket says 172.21.15.83:52952) 2024-03-06T07:26:12.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.780+0000 7f8e5cd94640 1 -- 172.21.15.83:0/1732716724 learned_addr learned my addr 172.21.15.83:0/1732716724 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:12.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.780+0000 7f8e577fe640 1 --2- 172.21.15.83:0/1732716724 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8e58143260 0x7f8e58155750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.780+0000 7f8e57fff640 1 --2- 172.21.15.83:0/1732716724 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8e400051c0 0x7f8e581426d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.780+0000 7f8e5cd94640 1 -- 172.21.15.83:0/1732716724 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8e58143260 msgr2=0x7f8e58155750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.780+0000 7f8e5cd94640 1 --2- 172.21.15.83:0/1732716724 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8e58143260 0x7f8e58155750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.780+0000 7f8e5cd94640 1 -- 172.21.15.83:0/1732716724 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8e400051c0 msgr2=0x7f8e581426d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.780+0000 7f8e5cd94640 1 --2- 172.21.15.83:0/1732716724 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8e400051c0 0x7f8e581426d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.780+0000 7f8e5cd94640 1 -- 172.21.15.83:0/1732716724 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e580b43c0 con 0x7f8e58155c90 2024-03-06T07:26:12.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.780+0000 7f8e577fe640 1 --2- 172.21.15.83:0/1732716724 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8e58143260 0x7f8e58155750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:12.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.780+0000 7f8e5cd94640 1 --2- 172.21.15.83:0/1732716724 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e58155c90 0x7f8e580b36d0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f8e58072da0 tx=0x7f8e5806cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:12.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.780+0000 7f8e54ff9640 1 -- 172.21.15.83:0/1732716724 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e50043070 con 0x7f8e58155c90 2024-03-06T07:26:12.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.780+0000 7f8e54ff9640 1 -- 172.21.15.83:0/1732716724 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e500019d0 con 0x7f8e58155c90 2024-03-06T07:26:12.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.780+0000 7f8e54ff9640 1 -- 172.21.15.83:0/1732716724 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e50039040 con 0x7f8e58155c90 2024-03-06T07:26:12.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.780+0000 7f8e5e7f7640 1 -- 172.21.15.83:0/1732716724 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e581512b0 con 0x7f8e58155c90 2024-03-06T07:26:12.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.780+0000 7f8e5e7f7640 1 -- 172.21.15.83:0/1732716724 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e58151870 con 0x7f8e58155c90 2024-03-06T07:26:12.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.781+0000 7f8e5e7f7640 1 -- 172.21.15.83:0/1732716724 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e5804f330 con 0x7f8e58155c90 2024-03-06T07:26:12.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.782+0000 7f8e54ff9640 1 -- 172.21.15.83:0/1732716724 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8e50046070 con 0x7f8e58155c90 2024-03-06T07:26:12.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.782+0000 7f8e54ff9640 1 --2- 172.21.15.83:0/1732716724 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8e4406b500 0x7f8e4406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.782+0000 7f8e57fff640 1 --2- 172.21.15.83:0/1732716724 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8e4406b500 0x7f8e4406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.782+0000 7f8e54ff9640 1 -- 172.21.15.83:0/1732716724 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8e500ab290 con 0x7f8e58155c90 2024-03-06T07:26:12.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.783+0000 7f8e57fff640 1 --2- 172.21.15.83:0/1732716724 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8e4406b500 0x7f8e4406d9c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f8e48006fd0 tx=0x7f8e48008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:12.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.787+0000 7f8e54ff9640 1 -- 172.21.15.83:0/1732716724 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8e50079dd0 con 0x7f8e58155c90 2024-03-06T07:26:12.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.960+0000 7f8e5e7f7640 1 -- 172.21.15.83:0/1732716724 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e58151a40 con 0x7f8e58155c90 2024-03-06T07:26:12.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.960+0000 7f8e54ff9640 1 -- 172.21.15.83:0/1732716724 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+13033 (secure 0 0 0) 0x7f8e5007de70 con 0x7f8e58155c90 2024-03-06T07:26:12.962 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:12.962 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":85,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:26:11.653726+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.83:6837/3056214366":"2024-03-07T07:26:08.646986+0000","172.21.15.83:6835/3056214366":"2024-03-07T07:26:08.646986+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:12.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.963+0000 7f8e5e7f7640 1 -- 172.21.15.83:0/1732716724 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8e4406b500 msgr2=0x7f8e4406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.963+0000 7f8e5e7f7640 1 --2- 172.21.15.83:0/1732716724 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8e4406b500 0x7f8e4406d9c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f8e48006fd0 tx=0x7f8e48008040 comp rx=0 tx=0).stop 2024-03-06T07:26:12.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.963+0000 7f8e5e7f7640 1 -- 172.21.15.83:0/1732716724 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e58155c90 msgr2=0x7f8e580b36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.963+0000 7f8e5e7f7640 1 --2- 172.21.15.83:0/1732716724 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e58155c90 0x7f8e580b36d0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f8e58072da0 tx=0x7f8e5806cee0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.963+0000 7f8e5e7f7640 1 -- 172.21.15.83:0/1732716724 shutdown_connections 2024-03-06T07:26:12.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.963+0000 7f8e5e7f7640 1 --2- 172.21.15.83:0/1732716724 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8e4406b500 0x7f8e4406d9c0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.963+0000 7f8e5e7f7640 1 --2- 172.21.15.83:0/1732716724 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e58155c90 0x7f8e580b36d0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.963+0000 7f8e5e7f7640 1 --2- 172.21.15.83:0/1732716724 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8e58143260 0x7f8e58155750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.963+0000 7f8e5e7f7640 1 --2- 172.21.15.83:0/1732716724 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8e400051c0 0x7f8e581426d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.963+0000 7f8e5e7f7640 1 -- 172.21.15.83:0/1732716724 >> 172.21.15.83:0/1732716724 conn(0x7f8e580b1a90 msgr2=0x7f8e5815f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:12.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.963+0000 7f8e5e7f7640 1 -- 172.21.15.83:0/1732716724 shutdown_connections 2024-03-06T07:26:12.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:12.963+0000 7f8e5e7f7640 1 -- 172.21.15.83:0/1732716724 wait complete. 2024-03-06T07:26:12.976 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:26:12.976 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:26:13.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.077+0000 7fbfbbc0e640 1 Processor -- start 2024-03-06T07:26:13.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.077+0000 7fbfbbc0e640 1 -- start start 2024-03-06T07:26:13.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.078+0000 7fbfbbc0e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbfb41586e0 0x7fbfb40aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.078+0000 7fbfbbc0e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fbfb40aef80 0x7fbfb40cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.078+0000 7fbfbbc0e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfb4158ac0 con 0x7fbfb40cffd0 2024-03-06T07:26:13.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.078+0000 7fbfbbc0e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfb415b2a0 con 0x7fbfb41586e0 2024-03-06T07:26:13.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.078+0000 7fbfbbc0e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfb415b540 con 0x7fbfb40aef80 2024-03-06T07:26:13.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.078+0000 7fbfb99aa640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbfb41586e0 0x7fbfb40aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.078+0000 7fbfba1ab640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fbfb40cffd0 0x7fbfb406da20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60388/0 (socket says 172.21.15.83:60388) 2024-03-06T07:26:13.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.078+0000 7fbfba1ab640 1 -- 172.21.15.83:0/2816145256 learned_addr learned my addr 172.21.15.83:0/2816145256 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:13.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.078+0000 7fbfb99aa640 1 -- 172.21.15.83:0/2816145256 >> v2:172.21.15.183:3300/0 conn(0x7fbfb40aef80 msgr2=0x7fbfb40cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:13.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.078+0000 7fbfb91a9640 1 --2- 172.21.15.83:0/2816145256 >> v2:172.21.15.183:3300/0 conn(0x7fbfb40aef80 0x7fbfb40cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.078+0000 7fbfb89a8640 1 -- 172.21.15.83:0/2816145256 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 214705382 0 0) 0x7fbfb4158ac0 con 0x7fbfb40cffd0 2024-03-06T07:26:13.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.078+0000 7fbfb99aa640 1 --2- 172.21.15.83:0/2816145256 >> v2:172.21.15.183:3300/0 conn(0x7fbfb40aef80 0x7fbfb40cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.078+0000 7fbfb99aa640 1 -- 172.21.15.83:0/2816145256 >> v1:172.21.15.83:6789/0 conn(0x7fbfb40cffd0 legacy=0x7fbfb406da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.078+0000 7fbfb99aa640 1 -- 172.21.15.83:0/2816145256 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb415b820 con 0x7fbfb41586e0 2024-03-06T07:26:13.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.078+0000 7fbfb91a9640 1 --2- 172.21.15.83:0/2816145256 >> v2:172.21.15.183:3300/0 conn(0x7fbfb40aef80 0x7fbfb40cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:13.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.079+0000 7fbfb99aa640 1 --2- 172.21.15.83:0/2816145256 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbfb41586e0 0x7fbfb40aea40 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fbfb4072e30 tx=0x7fbfb002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0620e0c298af595 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.079+0000 7fbfb89a8640 1 -- 172.21.15.83:0/2816145256 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfb0035020 con 0x7fbfb41586e0 2024-03-06T07:26:13.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.079+0000 7fbfb89a8640 1 -- 172.21.15.83:0/2816145256 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfb0031830 con 0x7fbfb41586e0 2024-03-06T07:26:13.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.079+0000 7fbfb89a8640 1 -- 172.21.15.83:0/2816145256 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfb0031c00 con 0x7fbfb41586e0 2024-03-06T07:26:13.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.080+0000 7fbfbbc0e640 1 -- 172.21.15.83:0/2816145256 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbfb41586e0 msgr2=0x7fbfb40aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.080+0000 7fbfbbc0e640 1 --2- 172.21.15.83:0/2816145256 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbfb41586e0 0x7fbfb40aea40 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fbfb4072e30 tx=0x7fbfb002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.080+0000 7fbfbbc0e640 1 -- 172.21.15.83:0/2816145256 shutdown_connections 2024-03-06T07:26:13.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.080+0000 7fbfbbc0e640 1 --2- 172.21.15.83:0/2816145256 >> v2:172.21.15.183:3300/0 conn(0x7fbfb40aef80 0x7fbfb40cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.080+0000 7fbfbbc0e640 1 --2- 172.21.15.83:0/2816145256 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbfb41586e0 0x7fbfb40aea40 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.080+0000 7fbfbbc0e640 1 -- 172.21.15.83:0/2816145256 >> 172.21.15.83:0/2816145256 conn(0x7fbfb406b190 msgr2=0x7fbfb406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:13.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.080+0000 7fbfbbc0e640 1 -- 172.21.15.83:0/2816145256 shutdown_connections 2024-03-06T07:26:13.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.080+0000 7fbfbbc0e640 1 -- 172.21.15.83:0/2816145256 wait complete. 2024-03-06T07:26:13.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.081+0000 7fbfbbc0e640 1 Processor -- start 2024-03-06T07:26:13.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.081+0000 7fbfbbc0e640 1 -- start start 2024-03-06T07:26:13.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.081+0000 7fbfbbc0e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbfb40aef80 0x7fbfb40a9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.081+0000 7fbfbbc0e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbfb40cffd0 0x7fbfb4141250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.082+0000 7fbfbbc0e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbfb41586e0 0x7fbfb40aaab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.082+0000 7fbfbbc0e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb4071350 con 0x7fbfb40cffd0 2024-03-06T07:26:13.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.082+0000 7fbfbbc0e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb415bde0 con 0x7fbfb40aef80 2024-03-06T07:26:13.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.082+0000 7fbfbbc0e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb40714d0 con 0x7fbfb41586e0 2024-03-06T07:26:13.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.082+0000 7fbfb91a9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbfb40cffd0 0x7fbfb4141250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.082+0000 7fbfb91a9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbfb40cffd0 0x7fbfb4141250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52968/0 (socket says 172.21.15.83:52968) 2024-03-06T07:26:13.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.082+0000 7fbfb91a9640 1 -- 172.21.15.83:0/2520747443 learned_addr learned my addr 172.21.15.83:0/2520747443 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:13.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.082+0000 7fbfb91a9640 1 -- 172.21.15.83:0/2520747443 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbfb41586e0 msgr2=0x7fbfb40aaab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.082+0000 7fbfba1ab640 1 --2- 172.21.15.83:0/2520747443 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbfb41586e0 0x7fbfb40aaab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.082+0000 7fbfb99aa640 1 --2- 172.21.15.83:0/2520747443 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbfb40aef80 0x7fbfb40a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.082+0000 7fbfb91a9640 1 --2- 172.21.15.83:0/2520747443 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbfb41586e0 0x7fbfb40aaab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.082+0000 7fbfb91a9640 1 -- 172.21.15.83:0/2520747443 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbfb40aef80 msgr2=0x7fbfb40a9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.082+0000 7fbfb91a9640 1 --2- 172.21.15.83:0/2520747443 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbfb40aef80 0x7fbfb40a9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.082+0000 7fbfb91a9640 1 -- 172.21.15.83:0/2520747443 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb40a9b50 con 0x7fbfb40cffd0 2024-03-06T07:26:13.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.082+0000 7fbfba1ab640 1 --2- 172.21.15.83:0/2520747443 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbfb41586e0 0x7fbfb40aaab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:13.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.082+0000 7fbfb99aa640 1 --2- 172.21.15.83:0/2520747443 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbfb40aef80 0x7fbfb40a9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:13.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.082+0000 7fbfb91a9640 1 --2- 172.21.15.83:0/2520747443 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbfb40cffd0 0x7fbfb4141250 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fbfa400c7f0 tx=0x7fbfa400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.082+0000 7fbfaa7fc640 1 -- 172.21.15.83:0/2520747443 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa4015070 con 0x7fbfb40cffd0 2024-03-06T07:26:13.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.082+0000 7fbfaa7fc640 1 -- 172.21.15.83:0/2520747443 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfa4007a30 con 0x7fbfb40cffd0 2024-03-06T07:26:13.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.082+0000 7fbfbbc0e640 1 -- 172.21.15.83:0/2520747443 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfb40a9e30 con 0x7fbfb40cffd0 2024-03-06T07:26:13.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.082+0000 7fbfaa7fc640 1 -- 172.21.15.83:0/2520747443 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa4007e00 con 0x7fbfb40cffd0 2024-03-06T07:26:13.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.082+0000 7fbfbbc0e640 1 -- 172.21.15.83:0/2520747443 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfb40aa3f0 con 0x7fbfb40cffd0 2024-03-06T07:26:13.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.083+0000 7fbfbbc0e640 1 -- 172.21.15.83:0/2520747443 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfb415c250 con 0x7fbfb40cffd0 2024-03-06T07:26:13.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.084+0000 7fbfaa7fc640 1 -- 172.21.15.83:0/2520747443 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbfa4018070 con 0x7fbfb40cffd0 2024-03-06T07:26:13.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.084+0000 7fbfaa7fc640 1 --2- 172.21.15.83:0/2520747443 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbf9806b500 0x7fbf9806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.084+0000 7fbfb99aa640 1 --2- 172.21.15.83:0/2520747443 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbf9806b500 0x7fbf9806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.084+0000 7fbfaa7fc640 1 -- 172.21.15.83:0/2520747443 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fbfa4089e20 con 0x7fbfb40cffd0 2024-03-06T07:26:13.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.085+0000 7fbfb99aa640 1 --2- 172.21.15.83:0/2520747443 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbf9806b500 0x7fbf9806d9c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0036000 tx=0x7fbfb0008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.089+0000 7fbfaa7fc640 1 -- 172.21.15.83:0/2520747443 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbfb415c250 con 0x7fbfb40cffd0 2024-03-06T07:26:13.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.263+0000 7fbfbbc0e640 1 -- 172.21.15.83:0/2520747443 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbfb40b4820 con 0x7fbfb40cffd0 2024-03-06T07:26:13.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.263+0000 7fbfaa7fc640 1 -- 172.21.15.83:0/2520747443 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbfb40b4820 con 0x7fbfb40cffd0 2024-03-06T07:26:13.265 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:26:13.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.265+0000 7fbfbbc0e640 1 -- 172.21.15.83:0/2520747443 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbf9806b500 msgr2=0x7fbf9806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.265+0000 7fbfbbc0e640 1 --2- 172.21.15.83:0/2520747443 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbf9806b500 0x7fbf9806d9c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0036000 tx=0x7fbfb0008030 comp rx=0 tx=0).stop 2024-03-06T07:26:13.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.265+0000 7fbfbbc0e640 1 -- 172.21.15.83:0/2520747443 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbfb40cffd0 msgr2=0x7fbfb4141250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.265+0000 7fbfbbc0e640 1 --2- 172.21.15.83:0/2520747443 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbfb40cffd0 0x7fbfb4141250 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fbfa400c7f0 tx=0x7fbfa400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.265+0000 7fbfbbc0e640 1 -- 172.21.15.83:0/2520747443 shutdown_connections 2024-03-06T07:26:13.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.265+0000 7fbfbbc0e640 1 --2- 172.21.15.83:0/2520747443 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbf9806b500 0x7fbf9806d9c0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.265+0000 7fbfbbc0e640 1 --2- 172.21.15.83:0/2520747443 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbfb41586e0 0x7fbfb40aaab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.265+0000 7fbfbbc0e640 1 --2- 172.21.15.83:0/2520747443 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbfb40cffd0 0x7fbfb4141250 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.265+0000 7fbfbbc0e640 1 --2- 172.21.15.83:0/2520747443 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbfb40aef80 0x7fbfb40a9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.265+0000 7fbfbbc0e640 1 -- 172.21.15.83:0/2520747443 >> 172.21.15.83:0/2520747443 conn(0x7fbfb406b190 msgr2=0x7fbfb41592c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:13.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.265+0000 7fbfbbc0e640 1 -- 172.21.15.83:0/2520747443 shutdown_connections 2024-03-06T07:26:13.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.265+0000 7fbfbbc0e640 1 -- 172.21.15.83:0/2520747443 wait complete. 2024-03-06T07:26:13.278 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_files_throttle' 2024-03-06T07:26:13.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.376+0000 7ff75bc33640 1 Processor -- start 2024-03-06T07:26:13.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.376+0000 7ff75bc33640 1 -- start start 2024-03-06T07:26:13.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.377+0000 7ff75bc33640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ff754066d70 0x7ff75415e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.377+0000 7ff75bc33640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff75406d2a0 0x7ff7540d0600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.377+0000 7ff75bc33640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff754067200 con 0x7ff75406bf10 2024-03-06T07:26:13.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.377+0000 7ff75bc33640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff75415f560 con 0x7ff754066d70 2024-03-06T07:26:13.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.377+0000 7ff75bc33640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff75415ecc0 con 0x7ff75406d2a0 2024-03-06T07:26:13.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.377+0000 7ff7599cf640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff75406bf10 0x7ff75406c2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60398/0 (socket says 172.21.15.83:60398) 2024-03-06T07:26:13.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.377+0000 7ff7599cf640 1 -- 172.21.15.83:0/345567245 learned_addr learned my addr 172.21.15.83:0/345567245 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:13.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.377+0000 7ff7589cd640 1 -- 172.21.15.83:0/345567245 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3592863905 0 0) 0x7ff754067200 con 0x7ff75406bf10 2024-03-06T07:26:13.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.378+0000 7ff7589cd640 1 -- 172.21.15.83:0/345567245 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff740003520 con 0x7ff75406bf10 2024-03-06T07:26:13.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.378+0000 7ff75a1d0640 1 --2- 172.21.15.83:0/345567245 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff75406d2a0 0x7ff7540d0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.378+0000 7ff7591ce640 1 --2- 172.21.15.83:0/345567245 >> v2:172.21.15.183:3300/0 conn(0x7ff754066d70 0x7ff75415e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.378+0000 7ff7589cd640 1 -- 172.21.15.83:0/345567245 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 340912516 0 0) 0x7ff740003520 con 0x7ff75406bf10 2024-03-06T07:26:13.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.378+0000 7ff7589cd640 1 -- 172.21.15.83:0/345567245 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff75406d2a0 msgr2=0x7ff7540d0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:13.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.378+0000 7ff7589cd640 1 --2- 172.21.15.83:0/345567245 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff75406d2a0 0x7ff7540d0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.378+0000 7ff7589cd640 1 -- 172.21.15.83:0/345567245 >> v2:172.21.15.183:3300/0 conn(0x7ff754066d70 msgr2=0x7ff75415e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:13.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.378+0000 7ff7589cd640 1 --2- 172.21.15.83:0/345567245 >> v2:172.21.15.183:3300/0 conn(0x7ff754066d70 0x7ff75415e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.378+0000 7ff7589cd640 1 -- 172.21.15.83:0/345567245 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff75415efa0 con 0x7ff75406bf10 2024-03-06T07:26:13.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.378+0000 7ff7589cd640 1 -- 172.21.15.83:0/345567245 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7ff750001670 con 0x7ff75406bf10 2024-03-06T07:26:13.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.378+0000 7ff7589cd640 1 -- 172.21.15.83:0/345567245 >> v1:172.21.15.83:6789/0 conn(0x7ff75406bf10 legacy=0x7ff75406c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.378+0000 7ff7589cd640 1 --2- 172.21.15.83:0/345567245 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7400039b0 0x7ff740003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.378+0000 7ff7589cd640 1 --2- 172.21.15.83:0/345567245 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff75406d2a0 0x7ff740004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.378+0000 7ff7589cd640 1 --2- 172.21.15.83:0/345567245 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff740004d40 0x7ff7400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.378+0000 7ff7589cd640 1 -- 172.21.15.83:0/345567245 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff75415f560 con 0x7ff740004d40 2024-03-06T07:26:13.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.378+0000 7ff7589cd640 1 -- 172.21.15.83:0/345567245 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff75415ecc0 con 0x7ff7400039b0 2024-03-06T07:26:13.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.378+0000 7ff7589cd640 1 -- 172.21.15.83:0/345567245 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff74000ba40 con 0x7ff75406d2a0 2024-03-06T07:26:13.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.378+0000 7ff75a1d0640 1 --2- 172.21.15.83:0/345567245 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff740004d40 0x7ff7400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.378+0000 7ff7599cf640 1 --2- 172.21.15.83:0/345567245 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7400039b0 0x7ff740003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.378+0000 7ff7591ce640 1 --2- 172.21.15.83:0/345567245 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff75406d2a0 0x7ff740004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.378+0000 7ff75a1d0640 1 -- 172.21.15.83:0/345567245 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff75406d2a0 msgr2=0x7ff740004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:13.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.378+0000 7ff75a1d0640 1 --2- 172.21.15.83:0/345567245 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff75406d2a0 0x7ff740004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.378+0000 7ff75a1d0640 1 -- 172.21.15.83:0/345567245 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7400039b0 msgr2=0x7ff740003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:13.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.378+0000 7ff75a1d0640 1 --2- 172.21.15.83:0/345567245 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7400039b0 0x7ff740003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.378+0000 7ff75a1d0640 1 -- 172.21.15.83:0/345567245 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff740003520 con 0x7ff740004d40 2024-03-06T07:26:13.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.379+0000 7ff75a1d0640 1 --2- 172.21.15.83:0/345567245 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff740004d40 0x7ff7400051c0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7ff74c009fc0 tx=0x7ff740005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5c8b302bfa2161e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.379+0000 7ff7589cd640 1 -- 172.21.15.83:0/345567245 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff74c039070 con 0x7ff740004d40 2024-03-06T07:26:13.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.379+0000 7ff7589cd640 1 -- 172.21.15.83:0/345567245 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff74c02cc20 con 0x7ff740004d40 2024-03-06T07:26:13.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.379+0000 7ff7589cd640 1 -- 172.21.15.83:0/345567245 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff74c009c10 con 0x7ff740004d40 2024-03-06T07:26:13.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.380+0000 7ff75bc33640 1 -- 172.21.15.83:0/345567245 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff740004d40 msgr2=0x7ff7400051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.380+0000 7ff75bc33640 1 --2- 172.21.15.83:0/345567245 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff740004d40 0x7ff7400051c0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7ff74c009fc0 tx=0x7ff740005830 comp rx=0 tx=0).stop 2024-03-06T07:26:13.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.380+0000 7ff75a1d0640 1 -- 172.21.15.83:0/345567245 reap_dead start 2024-03-06T07:26:13.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.380+0000 7ff75bc33640 1 -- 172.21.15.83:0/345567245 shutdown_connections 2024-03-06T07:26:13.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.380+0000 7ff75bc33640 1 -- 172.21.15.83:0/345567245 >> 172.21.15.83:0/345567245 conn(0x7ff7540725f0 msgr2=0x7ff754072a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:13.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.380+0000 7ff75bc33640 1 -- 172.21.15.83:0/345567245 shutdown_connections 2024-03-06T07:26:13.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.380+0000 7ff75bc33640 1 -- 172.21.15.83:0/345567245 wait complete. 2024-03-06T07:26:13.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.381+0000 7ff75bc33640 1 Processor -- start 2024-03-06T07:26:13.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.381+0000 7ff75bc33640 1 -- start start 2024-03-06T07:26:13.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.382+0000 7ff75bc33640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff740004d40 0x7ff75414c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.382+0000 7ff75bc33640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7540af270 0x7ff7540ad730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.382+0000 7ff75bc33640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7540adc70 0x7ff7540b0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.382+0000 7ff75bc33640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff75415be70 con 0x7ff7540adc70 2024-03-06T07:26:13.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.382+0000 7ff75bc33640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7540d00e0 con 0x7ff7540af270 2024-03-06T07:26:13.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.382+0000 7ff75bc33640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff75415bff0 con 0x7ff740004d40 2024-03-06T07:26:13.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.382+0000 7ff75a1d0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7540adc70 0x7ff7540b0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.382+0000 7ff75a1d0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7540adc70 0x7ff7540b0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52998/0 (socket says 172.21.15.83:52998) 2024-03-06T07:26:13.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.382+0000 7ff75a1d0640 1 -- 172.21.15.83:0/2644538115 learned_addr learned my addr 172.21.15.83:0/2644538115 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:13.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.382+0000 7ff7599cf640 1 --2- 172.21.15.83:0/2644538115 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff740004d40 0x7ff75414c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.382+0000 7ff7591ce640 1 --2- 172.21.15.83:0/2644538115 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7540af270 0x7ff7540ad730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.382+0000 7ff75a1d0640 1 -- 172.21.15.83:0/2644538115 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff740004d40 msgr2=0x7ff75414c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.382+0000 7ff75a1d0640 1 --2- 172.21.15.83:0/2644538115 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff740004d40 0x7ff75414c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.382+0000 7ff75a1d0640 1 -- 172.21.15.83:0/2644538115 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7540af270 msgr2=0x7ff7540ad730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.382+0000 7ff75a1d0640 1 --2- 172.21.15.83:0/2644538115 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7540af270 0x7ff7540ad730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.382+0000 7ff75a1d0640 1 -- 172.21.15.83:0/2644538115 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7540ae1d0 con 0x7ff7540adc70 2024-03-06T07:26:13.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.382+0000 7ff7591ce640 1 --2- 172.21.15.83:0/2644538115 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7540af270 0x7ff7540ad730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:13.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.382+0000 7ff75a1d0640 1 --2- 172.21.15.83:0/2644538115 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7540adc70 0x7ff7540b0690 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7ff74c002230 tx=0x7ff74c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.382+0000 7ff74a7fc640 1 -- 172.21.15.83:0/2644538115 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff74c039030 con 0x7ff7540adc70 2024-03-06T07:26:13.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.382+0000 7ff74a7fc640 1 -- 172.21.15.83:0/2644538115 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff74c043070 con 0x7ff7540adc70 2024-03-06T07:26:13.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.382+0000 7ff75bc33640 1 -- 172.21.15.83:0/2644538115 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7540ae4b0 con 0x7ff7540adc70 2024-03-06T07:26:13.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.382+0000 7ff74a7fc640 1 -- 172.21.15.83:0/2644538115 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff74c03d7f0 con 0x7ff7540adc70 2024-03-06T07:26:13.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.382+0000 7ff75bc33640 1 -- 172.21.15.83:0/2644538115 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7540aea70 con 0x7ff7540adc70 2024-03-06T07:26:13.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.383+0000 7ff75bc33640 1 -- 172.21.15.83:0/2644538115 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7540682b0 con 0x7ff7540adc70 2024-03-06T07:26:13.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.384+0000 7ff74a7fc640 1 -- 172.21.15.83:0/2644538115 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff74c005d10 con 0x7ff7540adc70 2024-03-06T07:26:13.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.384+0000 7ff74a7fc640 1 --2- 172.21.15.83:0/2644538115 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff73806b500 0x7ff73806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.384+0000 7ff74a7fc640 1 -- 172.21.15.83:0/2644538115 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff74c0ac1a0 con 0x7ff7540adc70 2024-03-06T07:26:13.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.384+0000 7ff7599cf640 1 --2- 172.21.15.83:0/2644538115 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff73806b500 0x7ff73806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.385+0000 7ff7599cf640 1 --2- 172.21.15.83:0/2644538115 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff73806b500 0x7ff73806d9c0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7ff750000c00 tx=0x7ff750009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.389+0000 7ff74a7fc640 1 -- 172.21.15.83:0/2644538115 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff7540682b0 con 0x7ff7540adc70 2024-03-06T07:26:13.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.624+0000 7ff75bc33640 1 -- 172.21.15.83:0/2644538115 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]} v 0) v1 -- 0x7ff75404f330 con 0x7ff7540adc70 2024-03-06T07:26:13.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.661+0000 7ff74a7fc640 1 -- 172.21.15.83:0/2644538115 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]}]=0 v245) v1 ==== 137+0+0 (secure 0 0 0) 0x7ff74c07b320 con 0x7ff7540adc70 2024-03-06T07:26:13.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.664+0000 7ff75bc33640 1 -- 172.21.15.83:0/2644538115 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff73806b500 msgr2=0x7ff73806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.664+0000 7ff75bc33640 1 --2- 172.21.15.83:0/2644538115 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff73806b500 0x7ff73806d9c0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7ff750000c00 tx=0x7ff750009040 comp rx=0 tx=0).stop 2024-03-06T07:26:13.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.664+0000 7ff75bc33640 1 -- 172.21.15.83:0/2644538115 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7540adc70 msgr2=0x7ff7540b0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.664+0000 7ff75bc33640 1 --2- 172.21.15.83:0/2644538115 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7540adc70 0x7ff7540b0690 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7ff74c002230 tx=0x7ff74c009750 comp rx=0 tx=0).stop 2024-03-06T07:26:13.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.664+0000 7ff75bc33640 1 -- 172.21.15.83:0/2644538115 shutdown_connections 2024-03-06T07:26:13.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.664+0000 7ff75bc33640 1 --2- 172.21.15.83:0/2644538115 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff73806b500 0x7ff73806d9c0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.664+0000 7ff75bc33640 1 --2- 172.21.15.83:0/2644538115 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7540adc70 0x7ff7540b0690 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.664+0000 7ff75bc33640 1 --2- 172.21.15.83:0/2644538115 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7540af270 0x7ff7540ad730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.664+0000 7ff75bc33640 1 --2- 172.21.15.83:0/2644538115 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff740004d40 0x7ff75414c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.664+0000 7ff75bc33640 1 -- 172.21.15.83:0/2644538115 >> 172.21.15.83:0/2644538115 conn(0x7ff7540725f0 msgr2=0x7ff75415fcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:13.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.664+0000 7ff75bc33640 1 -- 172.21.15.83:0/2644538115 shutdown_connections 2024-03-06T07:26:13.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.664+0000 7ff75bc33640 1 -- 172.21.15.83:0/2644538115 wait complete. 2024-03-06T07:26:13.677 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:26:13.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.778+0000 7fe3aae0e640 1 Processor -- start 2024-03-06T07:26:13.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.778+0000 7fe3aae0e640 1 -- start start 2024-03-06T07:26:13.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.779+0000 7fe3aae0e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fe3a415f4f0 0x7fe3a4066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.779+0000 7fe3aae0e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3a406b050 con 0x7fe3a40b0db0 2024-03-06T07:26:13.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.779+0000 7fe3aae0e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3a406b2c0 con 0x7fe3a40ce750 2024-03-06T07:26:13.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.779+0000 7fe3aae0e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3a40b0890 con 0x7fe3a415f4f0 2024-03-06T07:26:13.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.779+0000 7fe3a3fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fe3a40b0db0 0x7fe3a40ce040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60402/0 (socket says 172.21.15.83:60402) 2024-03-06T07:26:13.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.779+0000 7fe3a3fff640 1 -- 172.21.15.83:0/3548342935 learned_addr learned my addr 172.21.15.83:0/3548342935 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:13.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.779+0000 7fe3a37fe640 1 -- 172.21.15.83:0/3548342935 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 162907261 0 0) 0x7fe3a406b050 con 0x7fe3a40b0db0 2024-03-06T07:26:13.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.779+0000 7fe3a37fe640 1 -- 172.21.15.83:0/3548342935 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe38c003520 con 0x7fe3a40b0db0 2024-03-06T07:26:13.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.779+0000 7fe3a8baa640 1 --2- 172.21.15.83:0/3548342935 >> v2:172.21.15.183:3300/0 conn(0x7fe3a415f4f0 0x7fe3a4066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.780+0000 7fe3a37fe640 1 -- 172.21.15.83:0/3548342935 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1954305232 0 0) 0x7fe38c003520 con 0x7fe3a40b0db0 2024-03-06T07:26:13.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.780+0000 7fe3a37fe640 1 -- 172.21.15.83:0/3548342935 >> v2:172.21.15.183:3300/0 conn(0x7fe3a415f4f0 msgr2=0x7fe3a4066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:13.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.780+0000 7fe3a37fe640 1 --2- 172.21.15.83:0/3548342935 >> v2:172.21.15.183:3300/0 conn(0x7fe3a415f4f0 0x7fe3a4066d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.780+0000 7fe3a37fe640 1 -- 172.21.15.83:0/3548342935 >> v1:172.21.15.183:6789/0 conn(0x7fe3a40ce750 legacy=0x7fe3a40b4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:13.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.780+0000 7fe3a37fe640 1 -- 172.21.15.83:0/3548342935 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a40672e0 con 0x7fe3a40b0db0 2024-03-06T07:26:13.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.780+0000 7fe3a37fe640 1 -- 172.21.15.83:0/3548342935 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7fe398001670 con 0x7fe3a40b0db0 2024-03-06T07:26:13.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.780+0000 7fe3a37fe640 1 -- 172.21.15.83:0/3548342935 >> v1:172.21.15.83:6789/0 conn(0x7fe3a40b0db0 legacy=0x7fe3a40ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.780+0000 7fe3a37fe640 1 --2- 172.21.15.83:0/3548342935 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe38c0039b0 0x7fe38c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.780+0000 7fe3a37fe640 1 --2- 172.21.15.83:0/3548342935 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe38c004800 0x7fe38c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.780+0000 7fe3a37fe640 1 --2- 172.21.15.83:0/3548342935 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe38c0051c0 0x7fe38c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.780+0000 7fe3a37fe640 1 -- 172.21.15.83:0/3548342935 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a40b0890 con 0x7fe38c0039b0 2024-03-06T07:26:13.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.780+0000 7fe3a37fe640 1 -- 172.21.15.83:0/3548342935 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe38c00c060 con 0x7fe38c004800 2024-03-06T07:26:13.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.780+0000 7fe3a37fe640 1 -- 172.21.15.83:0/3548342935 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe38c00c260 con 0x7fe38c0051c0 2024-03-06T07:26:13.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.780+0000 7fe3a8baa640 1 --2- 172.21.15.83:0/3548342935 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe38c0039b0 0x7fe38c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.780+0000 7fe3a3fff640 1 --2- 172.21.15.83:0/3548342935 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe38c004800 0x7fe38c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.780+0000 7fe3a93ab640 1 --2- 172.21.15.83:0/3548342935 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe38c0051c0 0x7fe38c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.780+0000 7fe3a8baa640 1 -- 172.21.15.83:0/3548342935 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe38c0051c0 msgr2=0x7fe38c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:13.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.780+0000 7fe3a8baa640 1 --2- 172.21.15.83:0/3548342935 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe38c0051c0 0x7fe38c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.780+0000 7fe3a8baa640 1 -- 172.21.15.83:0/3548342935 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe38c004800 msgr2=0x7fe38c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:13.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.780+0000 7fe3a8baa640 1 --2- 172.21.15.83:0/3548342935 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe38c004800 0x7fe38c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.780+0000 7fe3a8baa640 1 -- 172.21.15.83:0/3548342935 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe38c003520 con 0x7fe38c0039b0 2024-03-06T07:26:13.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.780+0000 7fe3a93ab640 1 -- 172.21.15.83:0/3548342935 reap_dead start 2024-03-06T07:26:13.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.781+0000 7fe3a8baa640 1 --2- 172.21.15.83:0/3548342935 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe38c0039b0 0x7fe38c003dd0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fe394009ac0 tx=0x7fe38c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34f537aeca615f35 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.781+0000 7fe3a37fe640 1 -- 172.21.15.83:0/3548342935 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe394039070 con 0x7fe38c0039b0 2024-03-06T07:26:13.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.781+0000 7fe3a37fe640 1 -- 172.21.15.83:0/3548342935 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe39402c6d0 con 0x7fe38c0039b0 2024-03-06T07:26:13.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.781+0000 7fe3a37fe640 1 -- 172.21.15.83:0/3548342935 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe394009690 con 0x7fe38c0039b0 2024-03-06T07:26:13.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.781+0000 7fe3aae0e640 1 -- 172.21.15.83:0/3548342935 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe38c0039b0 msgr2=0x7fe38c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.781+0000 7fe3aae0e640 1 --2- 172.21.15.83:0/3548342935 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe38c0039b0 0x7fe38c003dd0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fe394009ac0 tx=0x7fe38c005830 comp rx=0 tx=0).stop 2024-03-06T07:26:13.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.782+0000 7fe3aae0e640 1 -- 172.21.15.83:0/3548342935 shutdown_connections 2024-03-06T07:26:13.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.782+0000 7fe3aae0e640 1 --2- 172.21.15.83:0/3548342935 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe38c0039b0 0x7fe38c003dd0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.782+0000 7fe3aae0e640 1 -- 172.21.15.83:0/3548342935 >> 172.21.15.83:0/3548342935 conn(0x7fe3a40b1d70 msgr2=0x7fe3a40b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:13.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.782+0000 7fe3aae0e640 1 -- 172.21.15.83:0/3548342935 shutdown_connections 2024-03-06T07:26:13.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.782+0000 7fe3aae0e640 1 -- 172.21.15.83:0/3548342935 wait complete. 2024-03-06T07:26:13.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.783+0000 7fe3aae0e640 1 Processor -- start 2024-03-06T07:26:13.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.783+0000 7fe3aae0e640 1 -- start start 2024-03-06T07:26:13.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.784+0000 7fe3aae0e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe38c0039b0 0x7fe3a4152790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.784+0000 7fe3aae0e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3a40c5560 0x7fe3a40c8430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.784+0000 7fe3aae0e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe3a40c65f0 0x7fe3a406d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.784+0000 7fe3aae0e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a415f8d0 con 0x7fe3a40c5560 2024-03-06T07:26:13.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.784+0000 7fe3aae0e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a406b5a0 con 0x7fe38c0039b0 2024-03-06T07:26:13.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.784+0000 7fe3aae0e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a40c8970 con 0x7fe3a40c65f0 2024-03-06T07:26:13.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.784+0000 7fe3a3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3a40c5560 0x7fe3a40c8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.784+0000 7fe3a3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3a40c5560 0x7fe3a40c8430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53018/0 (socket says 172.21.15.83:53018) 2024-03-06T07:26:13.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.784+0000 7fe3a3fff640 1 -- 172.21.15.83:0/173238226 learned_addr learned my addr 172.21.15.83:0/173238226 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:13.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.784+0000 7fe3a8baa640 1 --2- 172.21.15.83:0/173238226 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe38c0039b0 0x7fe3a4152790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.784+0000 7fe3a93ab640 1 --2- 172.21.15.83:0/173238226 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe3a40c65f0 0x7fe3a406d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.784+0000 7fe3a3fff640 1 -- 172.21.15.83:0/173238226 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe3a40c65f0 msgr2=0x7fe3a406d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.784+0000 7fe3a3fff640 1 --2- 172.21.15.83:0/173238226 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe3a40c65f0 0x7fe3a406d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.784+0000 7fe3a3fff640 1 -- 172.21.15.83:0/173238226 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe38c0039b0 msgr2=0x7fe3a4152790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.784+0000 7fe3a3fff640 1 --2- 172.21.15.83:0/173238226 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe38c0039b0 0x7fe3a4152790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.784+0000 7fe3a3fff640 1 -- 172.21.15.83:0/173238226 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a40c7670 con 0x7fe3a40c5560 2024-03-06T07:26:13.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.784+0000 7fe3a93ab640 1 --2- 172.21.15.83:0/173238226 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe3a40c65f0 0x7fe3a406d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:13.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.784+0000 7fe3a3fff640 1 --2- 172.21.15.83:0/173238226 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3a40c5560 0x7fe3a40c8430 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fe3980072f0 tx=0x7fe3980077b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.784+0000 7fe3a17fa640 1 -- 172.21.15.83:0/173238226 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39800f9c0 con 0x7fe3a40c5560 2024-03-06T07:26:13.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.784+0000 7fe3a17fa640 1 -- 172.21.15.83:0/173238226 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe398018070 con 0x7fe3a40c5560 2024-03-06T07:26:13.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.784+0000 7fe3aae0e640 1 -- 172.21.15.83:0/173238226 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3a40c7950 con 0x7fe3a40c5560 2024-03-06T07:26:13.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.784+0000 7fe3a17fa640 1 -- 172.21.15.83:0/173238226 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39800fde0 con 0x7fe3a40c5560 2024-03-06T07:26:13.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.784+0000 7fe3aae0e640 1 -- 172.21.15.83:0/173238226 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3a40c6a80 con 0x7fe3a40c5560 2024-03-06T07:26:13.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.785+0000 7fe3aae0e640 1 -- 172.21.15.83:0/173238226 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3a40c6ce0 con 0x7fe3a40c5560 2024-03-06T07:26:13.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.786+0000 7fe3a17fa640 1 -- 172.21.15.83:0/173238226 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe39801d460 con 0x7fe3a40c5560 2024-03-06T07:26:13.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.786+0000 7fe3a17fa640 1 --2- 172.21.15.83:0/173238226 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe39006b4d0 0x7fe39006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.786+0000 7fe3a17fa640 1 -- 172.21.15.83:0/173238226 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe398089fb0 con 0x7fe3a40c5560 2024-03-06T07:26:13.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.786+0000 7fe3a8baa640 1 --2- 172.21.15.83:0/173238226 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe39006b4d0 0x7fe39006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.787+0000 7fe3a8baa640 1 --2- 172.21.15.83:0/173238226 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe39006b4d0 0x7fe39006d990 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fe394036000 tx=0x7fe394030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.791+0000 7fe3a17fa640 1 -- 172.21.15.83:0/173238226 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe3a40c6ce0 con 0x7fe3a40c5560 2024-03-06T07:26:13.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.962+0000 7fe3aae0e640 1 -- 172.21.15.83:0/173238226 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fe3a409e3d0 con 0x7fe3a40c5560 2024-03-06T07:26:13.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.965+0000 7fe3a17fa640 1 -- 172.21.15.83:0/173238226 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe398059130 con 0x7fe3a40c5560 2024-03-06T07:26:13.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.967+0000 7fe3aae0e640 1 -- 172.21.15.83:0/173238226 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe39006b4d0 msgr2=0x7fe39006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.967+0000 7fe3aae0e640 1 --2- 172.21.15.83:0/173238226 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe39006b4d0 0x7fe39006d990 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fe394036000 tx=0x7fe394030000 comp rx=0 tx=0).stop 2024-03-06T07:26:13.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.967+0000 7fe3aae0e640 1 -- 172.21.15.83:0/173238226 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3a40c5560 msgr2=0x7fe3a40c8430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.967+0000 7fe3aae0e640 1 --2- 172.21.15.83:0/173238226 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3a40c5560 0x7fe3a40c8430 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fe3980072f0 tx=0x7fe3980077b0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.967+0000 7fe3aae0e640 1 -- 172.21.15.83:0/173238226 shutdown_connections 2024-03-06T07:26:13.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.967+0000 7fe3aae0e640 1 --2- 172.21.15.83:0/173238226 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe39006b4d0 0x7fe39006d990 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.967+0000 7fe3aae0e640 1 --2- 172.21.15.83:0/173238226 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe3a40c65f0 0x7fe3a406d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.967+0000 7fe3aae0e640 1 --2- 172.21.15.83:0/173238226 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3a40c5560 0x7fe3a40c8430 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.967+0000 7fe3aae0e640 1 --2- 172.21.15.83:0/173238226 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe38c0039b0 0x7fe3a4152790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.967+0000 7fe3aae0e640 1 -- 172.21.15.83:0/173238226 >> 172.21.15.83:0/173238226 conn(0x7fe3a40b1d70 msgr2=0x7fe3a4072cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:13.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.967+0000 7fe3aae0e640 1 -- 172.21.15.83:0/173238226 shutdown_connections 2024-03-06T07:26:13.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:13.967+0000 7fe3aae0e640 1 -- 172.21.15.83:0/173238226 wait complete. 2024-03-06T07:26:13.980 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:26:13.980 DEBUG:teuthology.orchestra.run.smithi083:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:14.017 DEBUG:teuthology.orchestra.run.smithi183:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:14.064 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:26:14.064 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:26:14.064 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:14.079 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1257 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:26:14.080 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:26:14.080 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:14.097 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=287 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:26:14.097 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:26:14.097 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:14.134 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1257 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:26:14.134 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:26:14.134 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:14.154 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=287 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:26:14.154 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:26:14.154 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:14.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.272+0000 7f132b97a640 1 Processor -- start 2024-03-06T07:26:14.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.272+0000 7f132b97a640 1 -- start start 2024-03-06T07:26:14.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.273+0000 7f132b97a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f13240d22b0 0x7f13240d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.273+0000 7f132b97a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13240b3430 con 0x7f13240b41f0 2024-03-06T07:26:14.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.273+0000 7f132b97a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13240b3710 con 0x7f132415dfe0 2024-03-06T07:26:14.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.273+0000 7f132b97a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13240d2c50 con 0x7f13240d22b0 2024-03-06T07:26:14.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.273+0000 7f1328f15640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f13240d22b0 0x7f13240d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.273+0000 7f1329f17640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f13240b41f0 0x7f13240b4680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60404/0 (socket says 172.21.15.83:60404) 2024-03-06T07:26:14.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.273+0000 7f1329f17640 1 -- 172.21.15.83:0/1480064181 learned_addr learned my addr 172.21.15.83:0/1480064181 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:14.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.274+0000 7f1328f15640 1 -- 172.21.15.83:0/1480064181 >> v1:172.21.15.183:6789/0 conn(0x7f132415dfe0 legacy=0x7f13240cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:14.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.274+0000 7f1328f15640 1 -- 172.21.15.83:0/1480064181 >> v1:172.21.15.83:6789/0 conn(0x7f13240b41f0 legacy=0x7f13240b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.274+0000 7f131bfff640 1 -- 172.21.15.83:0/1480064181 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4242499069 0 0) 0x7f13240b3430 con 0x7f13240b41f0 2024-03-06T07:26:14.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.274+0000 7f1328f15640 1 -- 172.21.15.83:0/1480064181 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13240b39f0 con 0x7f13240d22b0 2024-03-06T07:26:14.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.274+0000 7f1328f15640 1 --2- 172.21.15.83:0/1480064181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13240d22b0 0x7f13240d2710 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f13140097d0 tx=0x7f131402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=466735d62a6360ac server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:14.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.274+0000 7f131bfff640 1 -- 172.21.15.83:0/1480064181 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1314035020 con 0x7f13240d22b0 2024-03-06T07:26:14.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.275+0000 7f131bfff640 1 -- 172.21.15.83:0/1480064181 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1314031830 con 0x7f13240d22b0 2024-03-06T07:26:14.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.275+0000 7f131bfff640 1 -- 172.21.15.83:0/1480064181 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1314031c00 con 0x7f13240d22b0 2024-03-06T07:26:14.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.275+0000 7f132b97a640 1 -- 172.21.15.83:0/1480064181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13240d22b0 msgr2=0x7f13240d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.275+0000 7f132b97a640 1 --2- 172.21.15.83:0/1480064181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13240d22b0 0x7f13240d2710 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f13140097d0 tx=0x7f131402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.275+0000 7f132b97a640 1 -- 172.21.15.83:0/1480064181 shutdown_connections 2024-03-06T07:26:14.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.275+0000 7f132b97a640 1 --2- 172.21.15.83:0/1480064181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13240d22b0 0x7f13240d2710 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.275+0000 7f132b97a640 1 -- 172.21.15.83:0/1480064181 >> 172.21.15.83:0/1480064181 conn(0x7f13240d3cd0 msgr2=0x7f13240d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:14.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.275+0000 7f132b97a640 1 -- 172.21.15.83:0/1480064181 shutdown_connections 2024-03-06T07:26:14.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.275+0000 7f132b97a640 1 -- 172.21.15.83:0/1480064181 wait complete. 2024-03-06T07:26:14.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.276+0000 7f132b97a640 1 Processor -- start 2024-03-06T07:26:14.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.276+0000 7f132b97a640 1 -- start start 2024-03-06T07:26:14.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.277+0000 7f132b97a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f13240b41f0 0x7f13241610c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.277+0000 7f132b97a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f13240d22b0 0x7f13240b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.277+0000 7f132b97a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f132415dfe0 0x7f1324149e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.277+0000 7f132b97a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13240b3d60 con 0x7f132415dfe0 2024-03-06T07:26:14.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.277+0000 7f132b97a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13240b31e0 con 0x7f13240b41f0 2024-03-06T07:26:14.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.277+0000 7f132b97a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13240d1c70 con 0x7f13240d22b0 2024-03-06T07:26:14.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.277+0000 7f1329f17640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f132415dfe0 0x7f1324149e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.277+0000 7f1329f17640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f132415dfe0 0x7f1324149e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53036/0 (socket says 172.21.15.83:53036) 2024-03-06T07:26:14.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.277+0000 7f1329f17640 1 -- 172.21.15.83:0/2334351253 learned_addr learned my addr 172.21.15.83:0/2334351253 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:14.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.277+0000 7f1328f15640 1 --2- 172.21.15.83:0/2334351253 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f13240d22b0 0x7f13240b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.277+0000 7f1329716640 1 --2- 172.21.15.83:0/2334351253 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f13240b41f0 0x7f13241610c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.277+0000 7f1329f17640 1 -- 172.21.15.83:0/2334351253 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f13240d22b0 msgr2=0x7f13240b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.277+0000 7f1329f17640 1 --2- 172.21.15.83:0/2334351253 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f13240d22b0 0x7f13240b68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.277+0000 7f1329f17640 1 -- 172.21.15.83:0/2334351253 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f13240b41f0 msgr2=0x7f13241610c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.277+0000 7f1329f17640 1 --2- 172.21.15.83:0/2334351253 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f13240b41f0 0x7f13241610c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.277+0000 7f1329f17640 1 -- 172.21.15.83:0/2334351253 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13240b9070 con 0x7f132415dfe0 2024-03-06T07:26:14.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.277+0000 7f1329716640 1 --2- 172.21.15.83:0/2334351253 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f13240b41f0 0x7f13241610c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:14.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.277+0000 7f1328f15640 1 --2- 172.21.15.83:0/2334351253 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f13240d22b0 0x7f13240b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:14.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.278+0000 7f1329f17640 1 --2- 172.21.15.83:0/2334351253 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f132415dfe0 0x7f1324149e50 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f131c00c450 tx=0x7f131c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:14.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.278+0000 7f1319ffb640 1 -- 172.21.15.83:0/2334351253 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f131c018070 con 0x7f132415dfe0 2024-03-06T07:26:14.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.278+0000 7f1319ffb640 1 -- 172.21.15.83:0/2334351253 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f131c007500 con 0x7f132415dfe0 2024-03-06T07:26:14.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.278+0000 7f1319ffb640 1 -- 172.21.15.83:0/2334351253 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f131c004a20 con 0x7f132415dfe0 2024-03-06T07:26:14.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.278+0000 7f132b97a640 1 -- 172.21.15.83:0/2334351253 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13240b9350 con 0x7f132415dfe0 2024-03-06T07:26:14.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.278+0000 7f132b97a640 1 -- 172.21.15.83:0/2334351253 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13240b9910 con 0x7f132415dfe0 2024-03-06T07:26:14.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.279+0000 7f132b97a640 1 -- 172.21.15.83:0/2334351253 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f132404f330 con 0x7f132415dfe0 2024-03-06T07:26:14.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.279+0000 7f1319ffb640 1 -- 172.21.15.83:0/2334351253 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f131c004bc0 con 0x7f132415dfe0 2024-03-06T07:26:14.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.279+0000 7f1319ffb640 1 --2- 172.21.15.83:0/2334351253 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f130806b4d0 0x7f130806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.280+0000 7f1319ffb640 1 -- 172.21.15.83:0/2334351253 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f131c089120 con 0x7f132415dfe0 2024-03-06T07:26:14.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.280+0000 7f1329716640 1 --2- 172.21.15.83:0/2334351253 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f130806b4d0 0x7f130806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.280+0000 7f1329716640 1 --2- 172.21.15.83:0/2334351253 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f130806b4d0 0x7f130806d990 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f1320007e90 tx=0x7f13200073a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:14.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.284+0000 7f1319ffb640 1 -- 172.21.15.83:0/2334351253 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f131c0582b0 con 0x7f132415dfe0 2024-03-06T07:26:14.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.493+0000 7f132b97a640 1 -- 172.21.15.83:0/2334351253 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13240c9fc0 con 0x7f132415dfe0 2024-03-06T07:26:14.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.493+0000 7f1319ffb640 1 -- 172.21.15.83:0/2334351253 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+3702 (secure 0 0 0) 0x7f131c057bb0 con 0x7f132415dfe0 2024-03-06T07:26:14.495 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:14.495 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":41,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/2525949760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":2525949760},{"type":"v1","addr":"172.21.15.183:6834","nonce":2525949760}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35},{"gid":6262,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2104447869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2104447869},{"type":"v1","addr":"172.21.15.183:6835","nonce":2104447869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6954,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/779621621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":779621621},{"type":"v1","addr":"172.21.15.83:6836","nonce":779621621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35},{"gid":7254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/2242048991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":2242048991},{"type":"v1","addr":"172.21.15.83:6837","nonce":2242048991}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41}],"filesystems":[]} 2024-03-06T07:26:14.495 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 41 2024-03-06T07:26:14.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.495+0000 7f132b97a640 1 -- 172.21.15.83:0/2334351253 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f130806b4d0 msgr2=0x7f130806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.495+0000 7f132b97a640 1 --2- 172.21.15.83:0/2334351253 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f130806b4d0 0x7f130806d990 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f1320007e90 tx=0x7f13200073a0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.495+0000 7f132b97a640 1 -- 172.21.15.83:0/2334351253 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f132415dfe0 msgr2=0x7f1324149e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.495+0000 7f132b97a640 1 --2- 172.21.15.83:0/2334351253 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f132415dfe0 0x7f1324149e50 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f131c00c450 tx=0x7f131c00c910 comp rx=0 tx=0).stop 2024-03-06T07:26:14.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.495+0000 7f132b97a640 1 -- 172.21.15.83:0/2334351253 shutdown_connections 2024-03-06T07:26:14.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.495+0000 7f132b97a640 1 --2- 172.21.15.83:0/2334351253 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f130806b4d0 0x7f130806d990 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.495+0000 7f132b97a640 1 --2- 172.21.15.83:0/2334351253 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f132415dfe0 0x7f1324149e50 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.495+0000 7f132b97a640 1 --2- 172.21.15.83:0/2334351253 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f13240d22b0 0x7f13240b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.495+0000 7f132b97a640 1 --2- 172.21.15.83:0/2334351253 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f13240b41f0 0x7f13241610c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.495+0000 7f132b97a640 1 -- 172.21.15.83:0/2334351253 >> 172.21.15.83:0/2334351253 conn(0x7f13240d3cd0 msgr2=0x7f13240b5750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:14.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.495+0000 7f132b97a640 1 -- 172.21.15.83:0/2334351253 shutdown_connections 2024-03-06T07:26:14.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.495+0000 7f132b97a640 1 -- 172.21.15.83:0/2334351253 wait complete. 2024-03-06T07:26:14.508 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:26:14.508 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:26:14.508 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:26:14.508 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:14.509 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:26:14.509 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:26:14.509 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:26:14.509 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:14.509 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:26:14.509 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:26:14.509 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:26:14.510 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:14.510 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:26:14.510 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:26:14.510 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:26:14.510 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:14.518 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:26:14.518 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:26:14.519 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:26:14.519 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:26:14.525 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:26:14.525 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:26:14.526 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:26:14.526 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:26:14.561 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:26:14.561 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:14.561 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:26:14.561 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:14.568 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:26:14.568 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:14.568 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:26:14.569 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:14.569 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:26:14.636 INFO:tasks.ceph.mds.c.smithi083.stdout:starting mds.c at 2024-03-06T07:26:14.637 INFO:tasks.ceph.mds.a.smithi083.stdout:starting mds.a at 2024-03-06T07:26:14.653 INFO:tasks.ceph.mds.b.smithi183.stdout:starting mds.b at 2024-03-06T07:26:14.653 INFO:tasks.ceph.mds.d.smithi183.stdout:starting mds.d at 2024-03-06T07:26:14.654 INFO:tasks.ceph.mds.b.smithi183.stderr:2024-03-06T07:26:14.652+0000 7f6ca322f640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2024-03-06T07:26:14.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.672+0000 7f515e02a640 1 Processor -- start 2024-03-06T07:26:14.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.672+0000 7f515e02a640 1 -- start start 2024-03-06T07:26:14.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.673+0000 7f515e02a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f515815e000 0x7f51580d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.673+0000 7f515e02a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f51580d4810 0x7f51580d0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.673+0000 7f515e02a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51580d25e0 con 0x7f51580c80e0 2024-03-06T07:26:14.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.673+0000 7f515e02a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51580c7c50 con 0x7f515815e000 2024-03-06T07:26:14.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.673+0000 7f515e02a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51580d2880 con 0x7f51580d4810 2024-03-06T07:26:14.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.673+0000 7f51577fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f515815e000 0x7f51580d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.673+0000 7f5157fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f51580c80e0 0x7f51580c8570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60426/0 (socket says 172.21.15.83:60426) 2024-03-06T07:26:14.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.673+0000 7f5157fff640 1 -- 172.21.15.83:0/2935542945 learned_addr learned my addr 172.21.15.83:0/2935542945 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:14.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.673+0000 7f514ffff640 1 --2- 172.21.15.83:0/2935542945 >> v2:172.21.15.183:3300/0 conn(0x7f51580d4810 0x7f51580d0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.673+0000 7f51577fe640 1 -- 172.21.15.83:0/2935542945 >> v2:172.21.15.183:3300/0 conn(0x7f51580d4810 msgr2=0x7f51580d0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:14.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.674+0000 7f51577fe640 1 --2- 172.21.15.83:0/2935542945 >> v2:172.21.15.183:3300/0 conn(0x7f51580d4810 0x7f51580d0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.674+0000 7f51577fe640 1 -- 172.21.15.83:0/2935542945 >> v1:172.21.15.83:6789/0 conn(0x7f51580c80e0 legacy=0x7f51580c8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.674+0000 7f51577fe640 1 -- 172.21.15.83:0/2935542945 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51580d2b40 con 0x7f515815e000 2024-03-06T07:26:14.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.674+0000 7f51577fe640 1 --2- 172.21.15.83:0/2935542945 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f515815e000 0x7f51580d0680 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f515806b7f0 tx=0x7f514802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46adb2d4d0b9be05 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:14.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.678+0000 7f5156ffd640 1 -- 172.21.15.83:0/2935542945 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5148034070 con 0x7f515815e000 2024-03-06T07:26:14.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.678+0000 7f5156ffd640 1 -- 172.21.15.83:0/2935542945 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5148030610 con 0x7f515815e000 2024-03-06T07:26:14.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.678+0000 7f5156ffd640 1 -- 172.21.15.83:0/2935542945 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51480309e0 con 0x7f515815e000 2024-03-06T07:26:14.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.679+0000 7f515e02a640 1 -- 172.21.15.83:0/2935542945 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f515815e000 msgr2=0x7f51580d0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.679+0000 7f515e02a640 1 --2- 172.21.15.83:0/2935542945 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f515815e000 0x7f51580d0680 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f515806b7f0 tx=0x7f514802b960 comp rx=0 tx=0).stop 2024-03-06T07:26:14.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.679+0000 7f515e02a640 1 -- 172.21.15.83:0/2935542945 shutdown_connections 2024-03-06T07:26:14.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.679+0000 7f515e02a640 1 --2- 172.21.15.83:0/2935542945 >> v2:172.21.15.183:3300/0 conn(0x7f51580d4810 0x7f51580d0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.679+0000 7f515e02a640 1 --2- 172.21.15.83:0/2935542945 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f515815e000 0x7f51580d0680 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.679+0000 7f515e02a640 1 -- 172.21.15.83:0/2935542945 >> 172.21.15.83:0/2935542945 conn(0x7f51580d3cf0 msgr2=0x7f51580d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:14.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.679+0000 7f515e02a640 1 -- 172.21.15.83:0/2935542945 shutdown_connections 2024-03-06T07:26:14.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.679+0000 7f515e02a640 1 -- 172.21.15.83:0/2935542945 wait complete. 2024-03-06T07:26:14.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.680+0000 7f515e02a640 1 Processor -- start 2024-03-06T07:26:14.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.680+0000 7f515e02a640 1 -- start start 2024-03-06T07:26:14.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.681+0000 7f515e02a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f51580c80e0 0x7f51580b0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.681+0000 7f515e02a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f51580d4810 0x7f51580cc820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.681+0000 7f515e02a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f515815e000 0x7f51580b8920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.681+0000 7f515e02a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515815e3e0 con 0x7f515815e000 2024-03-06T07:26:14.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.681+0000 7f515e02a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51580d22d0 con 0x7f51580c80e0 2024-03-06T07:26:14.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.681+0000 7f515e02a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51580cf7d0 con 0x7f51580d4810 2024-03-06T07:26:14.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.681+0000 7f5157fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f515815e000 0x7f51580b8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.681+0000 7f5157fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f515815e000 0x7f51580b8920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53084/0 (socket says 172.21.15.83:53084) 2024-03-06T07:26:14.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.681+0000 7f5157fff640 1 -- 172.21.15.83:0/3236828526 learned_addr learned my addr 172.21.15.83:0/3236828526 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:14.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.681+0000 7f514ffff640 1 --2- 172.21.15.83:0/3236828526 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f51580d4810 0x7f51580cc820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.681+0000 7f51577fe640 1 --2- 172.21.15.83:0/3236828526 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f51580c80e0 0x7f51580b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.681+0000 7f5157fff640 1 -- 172.21.15.83:0/3236828526 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f51580d4810 msgr2=0x7f51580cc820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.681+0000 7f5157fff640 1 --2- 172.21.15.83:0/3236828526 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f51580d4810 0x7f51580cc820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.681+0000 7f5157fff640 1 -- 172.21.15.83:0/3236828526 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f51580c80e0 msgr2=0x7f51580b0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.681+0000 7f5157fff640 1 --2- 172.21.15.83:0/3236828526 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f51580c80e0 0x7f51580b0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.681+0000 7f5157fff640 1 -- 172.21.15.83:0/3236828526 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5158150770 con 0x7f515815e000 2024-03-06T07:26:14.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.682+0000 7f5157fff640 1 --2- 172.21.15.83:0/3236828526 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f515815e000 0x7f51580b8920 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f515000caa0 tx=0x7f515000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:14.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.686+0000 7f5154ff9640 1 -- 172.21.15.83:0/3236828526 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5150017070 con 0x7f515815e000 2024-03-06T07:26:14.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.686+0000 7f515e02a640 1 -- 172.21.15.83:0/3236828526 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5158150a50 con 0x7f515815e000 2024-03-06T07:26:14.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.686+0000 7f515e02a640 1 -- 172.21.15.83:0/3236828526 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5158151010 con 0x7f515815e000 2024-03-06T07:26:14.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.686+0000 7f5154ff9640 1 -- 172.21.15.83:0/3236828526 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f515000aba0 con 0x7f515815e000 2024-03-06T07:26:14.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.686+0000 7f5154ff9640 1 -- 172.21.15.83:0/3236828526 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5150012400 con 0x7f515815e000 2024-03-06T07:26:14.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.687+0000 7f5154ff9640 1 -- 172.21.15.83:0/3236828526 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f515001a070 con 0x7f515815e000 2024-03-06T07:26:14.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.687+0000 7f515e02a640 1 -- 172.21.15.83:0/3236828526 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f513c005290 con 0x7f515815e000 2024-03-06T07:26:14.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.688+0000 7f5154ff9640 1 --2- 172.21.15.83:0/3236828526 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f513806b500 0x7f513806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.688+0000 7f51577fe640 1 --2- 172.21.15.83:0/3236828526 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f513806b500 0x7f513806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.688+0000 7f5154ff9640 1 -- 172.21.15.83:0/3236828526 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5150089a20 con 0x7f515815e000 2024-03-06T07:26:14.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.688+0000 7f51577fe640 1 --2- 172.21.15.83:0/3236828526 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f513806b500 0x7f513806d9c0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f5148036000 tx=0x7f514803c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:14.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.693+0000 7f5154ff9640 1 -- 172.21.15.83:0/3236828526 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f51500585e0 con 0x7f515815e000 2024-03-06T07:26:14.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:14.878+0000 7f515e02a640 1 -- 172.21.15.83:0/3236828526 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f513c005850 con 0x7f515815e000 2024-03-06T07:26:15.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.688+0000 7f5154ff9640 1 -- 172.21.15.83:0/3236828526 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v90) v1 ==== 97+0+0 (secure 0 0 0) 0x7f515005c680 con 0x7f515815e000 2024-03-06T07:26:15.690 INFO:teuthology.orchestra.run.smithi083.stderr: removed all blocklist entries 2024-03-06T07:26:15.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.691+0000 7f515e02a640 1 -- 172.21.15.83:0/3236828526 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f513806b500 msgr2=0x7f513806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.691+0000 7f515e02a640 1 --2- 172.21.15.83:0/3236828526 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f513806b500 0x7f513806d9c0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f5148036000 tx=0x7f514803c040 comp rx=0 tx=0).stop 2024-03-06T07:26:15.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.691+0000 7f515e02a640 1 -- 172.21.15.83:0/3236828526 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f515815e000 msgr2=0x7f51580b8920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.691+0000 7f515e02a640 1 --2- 172.21.15.83:0/3236828526 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f515815e000 0x7f51580b8920 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f515000caa0 tx=0x7f515000cf60 comp rx=0 tx=0).stop 2024-03-06T07:26:15.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.691+0000 7f515e02a640 1 -- 172.21.15.83:0/3236828526 shutdown_connections 2024-03-06T07:26:15.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.691+0000 7f515e02a640 1 --2- 172.21.15.83:0/3236828526 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f513806b500 0x7f513806d9c0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.691+0000 7f515e02a640 1 --2- 172.21.15.83:0/3236828526 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f515815e000 0x7f51580b8920 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.691+0000 7f515e02a640 1 --2- 172.21.15.83:0/3236828526 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f51580d4810 0x7f51580cc820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.691+0000 7f515e02a640 1 --2- 172.21.15.83:0/3236828526 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f51580c80e0 0x7f51580b0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.691+0000 7f515e02a640 1 -- 172.21.15.83:0/3236828526 >> 172.21.15.83:0/3236828526 conn(0x7f51580d3cf0 msgr2=0x7f51580c8930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:15.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.691+0000 7f515e02a640 1 -- 172.21.15.83:0/3236828526 shutdown_connections 2024-03-06T07:26:15.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.691+0000 7f515e02a640 1 -- 172.21.15.83:0/3236828526 wait complete. 2024-03-06T07:26:15.705 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:26:15.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.808+0000 7f66540a4640 1 Processor -- start 2024-03-06T07:26:15.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.809+0000 7f66540a4640 1 -- start start 2024-03-06T07:26:15.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.809+0000 7f66540a4640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f664c06b6c0 0x7f664c0b38f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.809+0000 7f66540a4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f664c0b0160 con 0x7f664c0ae4c0 2024-03-06T07:26:15.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.809+0000 7f66540a4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f664c0ad9e0 con 0x7f664c15e110 2024-03-06T07:26:15.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.809+0000 7f66540a4640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f664c0ae030 con 0x7f664c06b6c0 2024-03-06T07:26:15.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.810+0000 7f665163f640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f664c0ae4c0 0x7f664c06afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60438/0 (socket says 172.21.15.83:60438) 2024-03-06T07:26:15.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.810+0000 7f665163f640 1 -- 172.21.15.83:0/3364253829 learned_addr learned my addr 172.21.15.83:0/3364253829 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:15.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.810+0000 7f6652641640 1 --2- 172.21.15.83:0/3364253829 >> v2:172.21.15.183:3300/0 conn(0x7f664c06b6c0 0x7f664c0b38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.810+0000 7f6650e3e640 1 -- 172.21.15.83:0/3364253829 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 581056743 0 0) 0x7f664c0b0160 con 0x7f664c0ae4c0 2024-03-06T07:26:15.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.810+0000 7f6650e3e640 1 -- 172.21.15.83:0/3364253829 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6638003520 con 0x7f664c0ae4c0 2024-03-06T07:26:15.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.810+0000 7f6650e3e640 1 -- 172.21.15.83:0/3364253829 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1076832993 0 0) 0x7f6638003520 con 0x7f664c0ae4c0 2024-03-06T07:26:15.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.810+0000 7f6650e3e640 1 -- 172.21.15.83:0/3364253829 >> v2:172.21.15.183:3300/0 conn(0x7f664c06b6c0 msgr2=0x7f664c0b38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:15.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.810+0000 7f6650e3e640 1 --2- 172.21.15.83:0/3364253829 >> v2:172.21.15.183:3300/0 conn(0x7f664c06b6c0 0x7f664c0b38f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.810+0000 7f6650e3e640 1 -- 172.21.15.83:0/3364253829 >> v1:172.21.15.183:6789/0 conn(0x7f664c15e110 legacy=0x7f664c0b5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.810+0000 7f6650e3e640 1 -- 172.21.15.83:0/3364253829 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f664c0adcc0 con 0x7f664c0ae4c0 2024-03-06T07:26:15.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.810+0000 7f6650e3e640 1 -- 172.21.15.83:0/3364253829 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f6648001670 con 0x7f664c0ae4c0 2024-03-06T07:26:15.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.810+0000 7f6650e3e640 1 -- 172.21.15.83:0/3364253829 >> v1:172.21.15.83:6789/0 conn(0x7f664c0ae4c0 legacy=0x7f664c06afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.810+0000 7f6650e3e640 1 --2- 172.21.15.83:0/3364253829 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66380039b0 0x7f6638003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.810+0000 7f6650e3e640 1 --2- 172.21.15.83:0/3364253829 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6638004800 0x7f6638004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.810+0000 7f6650e3e640 1 --2- 172.21.15.83:0/3364253829 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f66380051c0 0x7f663800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.810+0000 7f6650e3e640 1 -- 172.21.15.83:0/3364253829 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f664c0ae030 con 0x7f66380039b0 2024-03-06T07:26:15.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.810+0000 7f6650e3e640 1 -- 172.21.15.83:0/3364253829 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f663800c070 con 0x7f6638004800 2024-03-06T07:26:15.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.810+0000 7f6650e3e640 1 -- 172.21.15.83:0/3364253829 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f663800c270 con 0x7f66380051c0 2024-03-06T07:26:15.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.810+0000 7f6651e40640 1 --2- 172.21.15.83:0/3364253829 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66380039b0 0x7f6638003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.810+0000 7f665163f640 1 --2- 172.21.15.83:0/3364253829 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6638004800 0x7f6638004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.810+0000 7f6652641640 1 --2- 172.21.15.83:0/3364253829 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f66380051c0 0x7f663800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.810+0000 7f6651e40640 1 -- 172.21.15.83:0/3364253829 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f66380051c0 msgr2=0x7f663800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:15.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.810+0000 7f6651e40640 1 --2- 172.21.15.83:0/3364253829 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f66380051c0 0x7f663800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.811+0000 7f6651e40640 1 -- 172.21.15.83:0/3364253829 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6638004800 msgr2=0x7f6638004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:15.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.811+0000 7f6651e40640 1 --2- 172.21.15.83:0/3364253829 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6638004800 0x7f6638004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.811+0000 7f6651e40640 1 -- 172.21.15.83:0/3364253829 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6638003520 con 0x7f66380039b0 2024-03-06T07:26:15.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.811+0000 7f6652641640 1 -- 172.21.15.83:0/3364253829 reap_dead start 2024-03-06T07:26:15.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.811+0000 7f665163f640 1 --2- 172.21.15.83:0/3364253829 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6638004800 0x7f6638004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:15.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.811+0000 7f6651e40640 1 --2- 172.21.15.83:0/3364253829 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66380039b0 0x7f6638003dd0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f663c001750 tx=0x7f6638005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9d895590d0a828c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:15.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.811+0000 7f6650e3e640 1 -- 172.21.15.83:0/3364253829 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663c001370 con 0x7f66380039b0 2024-03-06T07:26:15.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.811+0000 7f6650e3e640 1 -- 172.21.15.83:0/3364253829 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f663c0052d0 con 0x7f66380039b0 2024-03-06T07:26:15.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.811+0000 7f6650e3e640 1 -- 172.21.15.83:0/3364253829 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663c003830 con 0x7f66380039b0 2024-03-06T07:26:15.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.812+0000 7f66540a4640 1 -- 172.21.15.83:0/3364253829 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66380039b0 msgr2=0x7f6638003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.812+0000 7f66540a4640 1 --2- 172.21.15.83:0/3364253829 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66380039b0 0x7f6638003dd0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f663c001750 tx=0x7f6638005830 comp rx=0 tx=0).stop 2024-03-06T07:26:15.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.812+0000 7f66540a4640 1 -- 172.21.15.83:0/3364253829 shutdown_connections 2024-03-06T07:26:15.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.812+0000 7f66540a4640 1 --2- 172.21.15.83:0/3364253829 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66380039b0 0x7f6638003dd0 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.812+0000 7f66540a4640 1 -- 172.21.15.83:0/3364253829 >> 172.21.15.83:0/3364253829 conn(0x7f664c0d0960 msgr2=0x7f664c0d0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:15.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.812+0000 7f66540a4640 1 -- 172.21.15.83:0/3364253829 shutdown_connections 2024-03-06T07:26:15.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.812+0000 7f66540a4640 1 -- 172.21.15.83:0/3364253829 wait complete. 2024-03-06T07:26:15.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.813+0000 7f66540a4640 1 Processor -- start 2024-03-06T07:26:15.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.813+0000 7f66540a4640 1 -- start start 2024-03-06T07:26:15.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.814+0000 7f66540a4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66380039b0 0x7f664c1f67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.814+0000 7f66540a4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f664c1f7b50 0x7f664c160490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.814+0000 7f66540a4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f664c0b4bc0 0x7f664c0b5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.814+0000 7f66540a4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f664c0d17e0 con 0x7f66380039b0 2024-03-06T07:26:15.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.814+0000 7f66540a4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f664c0d1660 con 0x7f664c0b4bc0 2024-03-06T07:26:15.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.814+0000 7f66540a4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f664c0afcf0 con 0x7f664c1f7b50 2024-03-06T07:26:15.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.814+0000 7f6651e40640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66380039b0 0x7f664c1f67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.814+0000 7f6651e40640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66380039b0 0x7f664c1f67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53108/0 (socket says 172.21.15.83:53108) 2024-03-06T07:26:15.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.814+0000 7f6651e40640 1 -- 172.21.15.83:0/301943187 learned_addr learned my addr 172.21.15.83:0/301943187 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:15.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.814+0000 7f6652641640 1 --2- 172.21.15.83:0/301943187 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f664c0b4bc0 0x7f664c0b5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.814+0000 7f665163f640 1 --2- 172.21.15.83:0/301943187 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f664c1f7b50 0x7f664c160490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.814+0000 7f6651e40640 1 -- 172.21.15.83:0/301943187 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f664c1f7b50 msgr2=0x7f664c160490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.814+0000 7f6651e40640 1 --2- 172.21.15.83:0/301943187 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f664c1f7b50 0x7f664c160490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.814+0000 7f6651e40640 1 -- 172.21.15.83:0/301943187 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f664c0b4bc0 msgr2=0x7f664c0b5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.814+0000 7f6651e40640 1 --2- 172.21.15.83:0/301943187 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f664c0b4bc0 0x7f664c0b5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.814+0000 7f6651e40640 1 -- 172.21.15.83:0/301943187 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f664c1f3b90 con 0x7f66380039b0 2024-03-06T07:26:15.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.814+0000 7f6651e40640 1 --2- 172.21.15.83:0/301943187 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66380039b0 0x7f664c1f67b0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f663c005040 tx=0x7f663c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:15.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.815+0000 7f66427fc640 1 -- 172.21.15.83:0/301943187 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663c041070 con 0x7f66380039b0 2024-03-06T07:26:15.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.815+0000 7f66427fc640 1 -- 172.21.15.83:0/301943187 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f663c005480 con 0x7f66380039b0 2024-03-06T07:26:15.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.815+0000 7f66540a4640 1 -- 172.21.15.83:0/301943187 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f664c1f3e70 con 0x7f66380039b0 2024-03-06T07:26:15.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.815+0000 7f66427fc640 1 -- 172.21.15.83:0/301943187 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663c001370 con 0x7f66380039b0 2024-03-06T07:26:15.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.815+0000 7f66540a4640 1 -- 172.21.15.83:0/301943187 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f664c1609d0 con 0x7f66380039b0 2024-03-06T07:26:15.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.816+0000 7f66540a4640 1 -- 172.21.15.83:0/301943187 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f664c04f330 con 0x7f66380039b0 2024-03-06T07:26:15.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.816+0000 7f66427fc640 1 -- 172.21.15.83:0/301943187 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f663c044070 con 0x7f66380039b0 2024-03-06T07:26:15.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.816+0000 7f66427fc640 1 --2- 172.21.15.83:0/301943187 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f663006b500 0x7f663006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.817+0000 7f665163f640 1 --2- 172.21.15.83:0/301943187 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f663006b500 0x7f663006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.817+0000 7f66427fc640 1 -- 172.21.15.83:0/301943187 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f663c0ab8a0 con 0x7f66380039b0 2024-03-06T07:26:15.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.817+0000 7f665163f640 1 --2- 172.21.15.83:0/301943187 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f663006b500 0x7f663006d9c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f664c06ddd0 tx=0x7f6648009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:15.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:15.821+0000 7f66427fc640 1 -- 172.21.15.83:0/301943187 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f663c07aa70 con 0x7f66380039b0 2024-03-06T07:26:16.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.057+0000 7f66540a4640 1 -- 172.21.15.83:0/301943187 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f664c0d06b0 con 0x7f66380039b0 2024-03-06T07:26:16.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.058+0000 7f66427fc640 1 -- 172.21.15.83:0/301943187 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f663c07a370 con 0x7f66380039b0 2024-03-06T07:26:16.060 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:16.060 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:26:16.060 INFO:teuthology.orchestra.run.smithi083.stdout: "auth_dump": [ 2024-03-06T07:26:16.060 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:26:16.060 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mds.a", 2024-03-06T07:26:16.060 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlVHaUGhAAVRpLsHYTUkaYDmceq5Rlgw==", 2024-03-06T07:26:16.060 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:26:16.060 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:26:16.060 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow *", 2024-03-06T07:26:16.060 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow *", 2024-03-06T07:26:16.060 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:26:16.060 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:26:16.061 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:26:16.061 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:26:16.061 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mds.b", 2024-03-06T07:26:16.061 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlFlrrJhAAQbY6xU910eYRg1zwgpj9vg==", 2024-03-06T07:26:16.061 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:26:16.061 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:26:16.061 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow *", 2024-03-06T07:26:16.061 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow *", 2024-03-06T07:26:16.061 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:26:16.061 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:26:16.061 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:26:16.061 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:26:16.061 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mds.c", 2024-03-06T07:26:16.061 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlX0jxIRAAxYNi39qxIFXazlL2qNzLCg==", 2024-03-06T07:26:16.062 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:26:16.062 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:26:16.062 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow *", 2024-03-06T07:26:16.062 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow *", 2024-03-06T07:26:16.062 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:26:16.062 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:26:16.062 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:26:16.062 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:26:16.062 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mds.d", 2024-03-06T07:26:16.062 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehl3utCLhAAzEjASLnYpPYrJsWOt3WwpQ==", 2024-03-06T07:26:16.062 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:26:16.062 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:26:16.062 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow *", 2024-03-06T07:26:16.062 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow *", 2024-03-06T07:26:16.062 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:26:16.062 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:26:16.063 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:26:16.063 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:26:16.063 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.0", 2024-03-06T07:26:16.063 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBgGehlkezBExAAZOLMH2ua3CTWAL7H6LGeZw==", 2024-03-06T07:26:16.063 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:26:16.063 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:16.063 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:26:16.063 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:26:16.063 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:26:16.063 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:26:16.063 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:26:16.063 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.1", 2024-03-06T07:26:16.063 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBjGehlC04qEhAAqirTFR6y3NOGw6N+iz3/pg==", 2024-03-06T07:26:16.063 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:26:16.063 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:16.064 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:26:16.064 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:26:16.064 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:26:16.064 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:26:16.064 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:26:16.064 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.2", 2024-03-06T07:26:16.064 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBmGehlfuFWEBAAj/vjZXrv1wVNSJ1R74UmnQ==", 2024-03-06T07:26:16.064 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:26:16.064 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:16.064 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:26:16.064 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:26:16.065 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:26:16.065 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:26:16.065 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:26:16.065 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.3", 2024-03-06T07:26:16.065 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBpGehl5+ItDhAA0YNc83MhKcgb3JgRuP87aw==", 2024-03-06T07:26:16.065 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:26:16.065 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:16.065 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:26:16.065 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:26:16.065 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:26:16.065 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:26:16.065 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:26:16.065 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.4", 2024-03-06T07:26:16.065 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBvGehlYa/TFBAAvGcizpRQ21pvgpEGgSET0w==", 2024-03-06T07:26:16.065 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:26:16.065 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:16.066 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:26:16.066 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:26:16.066 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:26:16.066 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:26:16.066 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:26:16.066 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.5", 2024-03-06T07:26:16.066 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQByGehlE80TExAAFcWDgnLtW8OKlyu4B0jnmg==", 2024-03-06T07:26:16.066 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:26:16.066 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:16.066 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:26:16.066 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:26:16.066 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:26:16.066 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:26:16.066 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:26:16.067 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.6", 2024-03-06T07:26:16.067 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQB1Gehla/LbERAAYfAjTKL11VMVnGUGV5GVcg==", 2024-03-06T07:26:16.067 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:26:16.067 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:16.067 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:26:16.067 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:26:16.067 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:26:16.067 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:26:16.067 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:26:16.067 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.7", 2024-03-06T07:26:16.067 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQB4GehlpSJkEBAA17tUgMf94jfqvXlD51P8DQ==", 2024-03-06T07:26:16.067 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:26:16.067 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:16.067 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:26:16.067 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:26:16.068 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:26:16.068 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:26:16.068 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:26:16.068 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.0", 2024-03-06T07:26:16.068 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlRwCIMhAA1DdbdQZPplu7LQOTK3BmSg==", 2024-03-06T07:26:16.068 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:26:16.068 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:26:16.068 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow r", 2024-03-06T07:26:16.068 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:26:16.068 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:26:16.068 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:26:16.068 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:26:16.068 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.1", 2024-03-06T07:26:16.069 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBdGehl77spABAAsiEf+tXos3MwFrXToEQ8dQ==", 2024-03-06T07:26:16.069 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:26:16.069 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:26:16.069 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow r", 2024-03-06T07:26:16.069 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow rw", 2024-03-06T07:26:16.069 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow rwx" 2024-03-06T07:26:16.069 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:26:16.069 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:26:16.069 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:26:16.069 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.2", 2024-03-06T07:26:16.069 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehltY5SOBAAI8rcIEVqMZaWQGgNaYKJTA==", 2024-03-06T07:26:16.069 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:26:16.069 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:26:16.069 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow r", 2024-03-06T07:26:16.069 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow rw", 2024-03-06T07:26:16.069 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow rwx" 2024-03-06T07:26:16.070 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:26:16.070 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:26:16.070 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:26:16.070 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.3", 2024-03-06T07:26:16.070 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBdGehl9ZX1BRAAA7fKnRyaQ+zqHoSGs4UvDw==", 2024-03-06T07:26:16.070 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:26:16.070 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:26:16.070 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow r", 2024-03-06T07:26:16.070 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow rw", 2024-03-06T07:26:16.070 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow rwx" 2024-03-06T07:26:16.070 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:26:16.070 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:26:16.070 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:26:16.070 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.admin", 2024-03-06T07:26:16.071 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBbGehl4gqRORAAobqSWoeyHYA/lXkp6/EcIQ==", 2024-03-06T07:26:16.071 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:26:16.071 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow *", 2024-03-06T07:26:16.071 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow *", 2024-03-06T07:26:16.071 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow *", 2024-03-06T07:26:16.071 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:26:16.071 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:26:16.071 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:26:16.071 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:26:16.071 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mgr.x", 2024-03-06T07:26:16.071 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlfikrFxAAoZkTUtpHlBcUAMpwWABRxg==", 2024-03-06T07:26:16.071 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:26:16.071 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow *", 2024-03-06T07:26:16.071 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile mgr", 2024-03-06T07:26:16.072 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:26:16.072 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:26:16.072 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:26:16.072 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:26:16.072 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mgr.y", 2024-03-06T07:26:16.072 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlvdcdExAAySXlEfJqLnpMySiiYGFkMQ==", 2024-03-06T07:26:16.072 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:26:16.072 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow *", 2024-03-06T07:26:16.072 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile mgr", 2024-03-06T07:26:16.072 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:26:16.072 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:26:16.072 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:26:16.072 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-03-06T07:26:16.072 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:26:16.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.059+0000 7f66540a4640 1 -- 172.21.15.83:0/301943187 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f663006b500 msgr2=0x7f663006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.059+0000 7f66540a4640 1 --2- 172.21.15.83:0/301943187 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f663006b500 0x7f663006d9c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f664c06ddd0 tx=0x7f6648009040 comp rx=0 tx=0).stop 2024-03-06T07:26:16.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.059+0000 7f66540a4640 1 -- 172.21.15.83:0/301943187 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66380039b0 msgr2=0x7f664c1f67b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.059+0000 7f66540a4640 1 --2- 172.21.15.83:0/301943187 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66380039b0 0x7f664c1f67b0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f663c005040 tx=0x7f663c001120 comp rx=0 tx=0).stop 2024-03-06T07:26:16.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.060+0000 7f66540a4640 1 -- 172.21.15.83:0/301943187 shutdown_connections 2024-03-06T07:26:16.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.060+0000 7f66540a4640 1 --2- 172.21.15.83:0/301943187 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f663006b500 0x7f663006d9c0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.060+0000 7f66540a4640 1 --2- 172.21.15.83:0/301943187 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f664c0b4bc0 0x7f664c0b5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.060+0000 7f66540a4640 1 --2- 172.21.15.83:0/301943187 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f664c1f7b50 0x7f664c160490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.060+0000 7f66540a4640 1 --2- 172.21.15.83:0/301943187 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66380039b0 0x7f664c1f67b0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.060+0000 7f66540a4640 1 -- 172.21.15.83:0/301943187 >> 172.21.15.83:0/301943187 conn(0x7f664c0d0960 msgr2=0x7f664c15c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:16.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.060+0000 7f66540a4640 1 -- 172.21.15.83:0/301943187 shutdown_connections 2024-03-06T07:26:16.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.060+0000 7f66540a4640 1 -- 172.21.15.83:0/301943187 wait complete. 2024-03-06T07:26:16.076 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:16.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.175+0000 7f41f8439640 1 Processor -- start 2024-03-06T07:26:16.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.175+0000 7f41f8439640 1 -- start start 2024-03-06T07:26:16.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.176+0000 7f41f8439640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f41f006b600 0x7f41f015e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.176+0000 7f41f8439640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41f015ea30 con 0x7f41f00ce920 2024-03-06T07:26:16.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.176+0000 7f41f8439640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41f015ed10 con 0x7f41f006b220 2024-03-06T07:26:16.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.176+0000 7f41f8439640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41f0161b70 con 0x7f41f006b600 2024-03-06T07:26:16.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.176+0000 7f41f69d6640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f41f006b600 0x7f41f015e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.176+0000 7f41f59d4640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f41f00ce920 0x7f41f0162000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60454/0 (socket says 172.21.15.83:60454) 2024-03-06T07:26:16.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.176+0000 7f41f59d4640 1 -- 172.21.15.83:0/587660647 learned_addr learned my addr 172.21.15.83:0/587660647 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:16.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.176+0000 7f41f69d6640 1 -- 172.21.15.83:0/587660647 >> v1:172.21.15.183:6789/0 conn(0x7f41f006b220 legacy=0x7f41f0071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:16.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.176+0000 7f41f69d6640 1 -- 172.21.15.83:0/587660647 >> v1:172.21.15.83:6789/0 conn(0x7f41f00ce920 legacy=0x7f41f0162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.176+0000 7f41f51d3640 1 -- 172.21.15.83:0/587660647 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3134587420 0 0) 0x7f41f015ea30 con 0x7f41f00ce920 2024-03-06T07:26:16.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.176+0000 7f41f69d6640 1 -- 172.21.15.83:0/587660647 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41f015eff0 con 0x7f41f006b600 2024-03-06T07:26:16.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.177+0000 7f41f69d6640 1 --2- 172.21.15.83:0/587660647 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41f006b600 0x7f41f015e450 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f41f0161cf0 tx=0x7f41e802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a27da0bac1929946 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:16.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.177+0000 7f41f51d3640 1 -- 172.21.15.83:0/587660647 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e8039070 con 0x7f41f006b600 2024-03-06T07:26:16.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.177+0000 7f41f51d3640 1 -- 172.21.15.83:0/587660647 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41e802cc50 con 0x7f41f006b600 2024-03-06T07:26:16.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.177+0000 7f41f51d3640 1 -- 172.21.15.83:0/587660647 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e8034440 con 0x7f41f006b600 2024-03-06T07:26:16.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.177+0000 7f41f8439640 1 -- 172.21.15.83:0/587660647 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41f006b600 msgr2=0x7f41f015e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.177+0000 7f41f8439640 1 --2- 172.21.15.83:0/587660647 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41f006b600 0x7f41f015e450 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f41f0161cf0 tx=0x7f41e802bea0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.178+0000 7f41f8439640 1 -- 172.21.15.83:0/587660647 shutdown_connections 2024-03-06T07:26:16.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.178+0000 7f41f8439640 1 --2- 172.21.15.83:0/587660647 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41f006b600 0x7f41f015e450 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.178+0000 7f41f8439640 1 -- 172.21.15.83:0/587660647 >> 172.21.15.83:0/587660647 conn(0x7f41f006c0c0 msgr2=0x7f41f006c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:16.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.178+0000 7f41f8439640 1 -- 172.21.15.83:0/587660647 shutdown_connections 2024-03-06T07:26:16.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.178+0000 7f41f8439640 1 -- 172.21.15.83:0/587660647 wait complete. 2024-03-06T07:26:16.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.179+0000 7f41f8439640 1 Processor -- start 2024-03-06T07:26:16.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.179+0000 7f41f8439640 1 -- start start 2024-03-06T07:26:16.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.179+0000 7f41f8439640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f41f006b220 0x7f41f0147540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.179+0000 7f41f8439640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41f006b600 0x7f41f0147cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.179+0000 7f41f8439640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f41f00ce920 0x7f41f00b14b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.179+0000 7f41f8439640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f0072090 con 0x7f41f006b600 2024-03-06T07:26:16.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.179+0000 7f41f8439640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f00ce200 con 0x7f41f00ce920 2024-03-06T07:26:16.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.179+0000 7f41f8439640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41f00b01b0 con 0x7f41f006b220 2024-03-06T07:26:16.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.179+0000 7f41f59d4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41f006b600 0x7f41f0147cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.180+0000 7f41f59d4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41f006b600 0x7f41f0147cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53122/0 (socket says 172.21.15.83:53122) 2024-03-06T07:26:16.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.180+0000 7f41f59d4640 1 -- 172.21.15.83:0/4164619716 learned_addr learned my addr 172.21.15.83:0/4164619716 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:16.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.180+0000 7f41f69d6640 1 --2- 172.21.15.83:0/4164619716 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f41f00ce920 0x7f41f00b14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.180+0000 7f41f61d5640 1 --2- 172.21.15.83:0/4164619716 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f41f006b220 0x7f41f0147540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.180+0000 7f41f59d4640 1 -- 172.21.15.83:0/4164619716 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f41f006b220 msgr2=0x7f41f0147540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.180+0000 7f41f59d4640 1 --2- 172.21.15.83:0/4164619716 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f41f006b220 0x7f41f0147540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.180+0000 7f41f59d4640 1 -- 172.21.15.83:0/4164619716 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f41f00ce920 msgr2=0x7f41f00b14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.180+0000 7f41f59d4640 1 --2- 172.21.15.83:0/4164619716 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f41f00ce920 0x7f41f00b14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.180+0000 7f41f59d4640 1 -- 172.21.15.83:0/4164619716 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41f00b0430 con 0x7f41f006b600 2024-03-06T07:26:16.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.180+0000 7f41f69d6640 1 --2- 172.21.15.83:0/4164619716 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f41f00ce920 0x7f41f00b14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:16.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.180+0000 7f41f59d4640 1 --2- 172.21.15.83:0/4164619716 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41f006b600 0x7f41f0147cd0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f41e000c4d0 tx=0x7f41e000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:16.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.180+0000 7f41e6ffd640 1 -- 172.21.15.83:0/4164619716 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e0018070 con 0x7f41f006b600 2024-03-06T07:26:16.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.180+0000 7f41e6ffd640 1 -- 172.21.15.83:0/4164619716 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41e0007500 con 0x7f41f006b600 2024-03-06T07:26:16.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.180+0000 7f41f8439640 1 -- 172.21.15.83:0/4164619716 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41f00b06f0 con 0x7f41f006b600 2024-03-06T07:26:16.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.180+0000 7f41e6ffd640 1 -- 172.21.15.83:0/4164619716 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e0004a20 con 0x7f41f006b600 2024-03-06T07:26:16.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.180+0000 7f41f8439640 1 -- 172.21.15.83:0/4164619716 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41f00b0cb0 con 0x7f41f006b600 2024-03-06T07:26:16.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.181+0000 7f41f8439640 1 -- 172.21.15.83:0/4164619716 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41f004f330 con 0x7f41f006b600 2024-03-06T07:26:16.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.181+0000 7f41e6ffd640 1 -- 172.21.15.83:0/4164619716 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f41e001b070 con 0x7f41f006b600 2024-03-06T07:26:16.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.182+0000 7f41e6ffd640 1 --2- 172.21.15.83:0/4164619716 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f41d406b4d0 0x7f41d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.182+0000 7f41e6ffd640 1 -- 172.21.15.83:0/4164619716 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f41e00891a0 con 0x7f41f006b600 2024-03-06T07:26:16.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.182+0000 7f41f61d5640 1 --2- 172.21.15.83:0/4164619716 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f41d406b4d0 0x7f41d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.182+0000 7f41f61d5640 1 --2- 172.21.15.83:0/4164619716 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f41d406b4d0 0x7f41d406d990 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f41ec006fd0 tx=0x7f41ec008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:16.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.186+0000 7f41e6ffd640 1 -- 172.21.15.83:0/4164619716 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f41e0058360 con 0x7f41f006b600 2024-03-06T07:26:16.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.358+0000 7f41f8439640 1 -- 172.21.15.83:0/4164619716 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41f00cdf00 con 0x7f41f006b600 2024-03-06T07:26:16.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.359+0000 7f41e6ffd640 1 -- 172.21.15.83:0/4164619716 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+12904 (secure 0 0 0) 0x7f41e0057c60 con 0x7f41f006b600 2024-03-06T07:26:16.360 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:16.361 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":90,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:26:15.687774+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:16.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.360+0000 7f41f8439640 1 -- 172.21.15.83:0/4164619716 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f41d406b4d0 msgr2=0x7f41d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.360+0000 7f41f8439640 1 --2- 172.21.15.83:0/4164619716 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f41d406b4d0 0x7f41d406d990 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f41ec006fd0 tx=0x7f41ec008040 comp rx=0 tx=0).stop 2024-03-06T07:26:16.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.360+0000 7f41f8439640 1 -- 172.21.15.83:0/4164619716 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41f006b600 msgr2=0x7f41f0147cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.360+0000 7f41f8439640 1 --2- 172.21.15.83:0/4164619716 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41f006b600 0x7f41f0147cd0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f41e000c4d0 tx=0x7f41e000c990 comp rx=0 tx=0).stop 2024-03-06T07:26:16.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.360+0000 7f41f8439640 1 -- 172.21.15.83:0/4164619716 shutdown_connections 2024-03-06T07:26:16.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.360+0000 7f41f8439640 1 --2- 172.21.15.83:0/4164619716 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f41d406b4d0 0x7f41d406d990 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.360+0000 7f41f8439640 1 --2- 172.21.15.83:0/4164619716 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f41f00ce920 0x7f41f00b14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.360+0000 7f41f8439640 1 --2- 172.21.15.83:0/4164619716 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41f006b600 0x7f41f0147cd0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.360+0000 7f41f8439640 1 --2- 172.21.15.83:0/4164619716 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f41f006b220 0x7f41f0147540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.360+0000 7f41f8439640 1 -- 172.21.15.83:0/4164619716 >> 172.21.15.83:0/4164619716 conn(0x7f41f006c0c0 msgr2=0x7f41f006e210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:16.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.361+0000 7f41f8439640 1 -- 172.21.15.83:0/4164619716 shutdown_connections 2024-03-06T07:26:16.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.361+0000 7f41f8439640 1 -- 172.21.15.83:0/4164619716 wait complete. 2024-03-06T07:26:16.373 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:26:16.374 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:26:16.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.474+0000 7ff73ed65640 1 Processor -- start 2024-03-06T07:26:16.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.474+0000 7ff73ed65640 1 -- start start 2024-03-06T07:26:16.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.475+0000 7ff73ed65640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ff7380d4860 0x7ff7380d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.475+0000 7ff73ed65640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff7380b0e30 0x7ff7380b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.475+0000 7ff73ed65640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7381618b0 con 0x7ff7381614d0 2024-03-06T07:26:16.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.475+0000 7ff73ed65640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7380d4cf0 con 0x7ff7380d4860 2024-03-06T07:26:16.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.475+0000 7ff73ed65640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7380cf400 con 0x7ff7380b0e30 2024-03-06T07:26:16.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.475+0000 7ff73cb01640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff7381614d0 0x7ff7380d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60466/0 (socket says 172.21.15.83:60466) 2024-03-06T07:26:16.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.475+0000 7ff73cb01640 1 -- 172.21.15.83:0/767867370 learned_addr learned my addr 172.21.15.83:0/767867370 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:16.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.476+0000 7ff7377fe640 1 -- 172.21.15.83:0/767867370 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3040091759 0 0) 0x7ff7381618b0 con 0x7ff7381614d0 2024-03-06T07:26:16.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.476+0000 7ff7377fe640 1 -- 172.21.15.83:0/767867370 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff720003520 con 0x7ff7381614d0 2024-03-06T07:26:16.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.476+0000 7ff737fff640 1 --2- 172.21.15.83:0/767867370 >> v2:172.21.15.183:3300/0 conn(0x7ff7380d4860 0x7ff7380d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.476+0000 7ff73d302640 1 --2- 172.21.15.83:0/767867370 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff7380b0e30 0x7ff7380b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.476+0000 7ff7377fe640 1 -- 172.21.15.83:0/767867370 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1278351144 0 0) 0x7ff720003520 con 0x7ff7381614d0 2024-03-06T07:26:16.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.476+0000 7ff7377fe640 1 -- 172.21.15.83:0/767867370 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff7380b0e30 msgr2=0x7ff7380b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:16.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.476+0000 7ff7377fe640 1 --2- 172.21.15.83:0/767867370 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff7380b0e30 0x7ff7380b12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.476+0000 7ff7377fe640 1 -- 172.21.15.83:0/767867370 >> v2:172.21.15.183:3300/0 conn(0x7ff7380d4860 msgr2=0x7ff7380d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:16.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.476+0000 7ff7377fe640 1 --2- 172.21.15.83:0/767867370 >> v2:172.21.15.183:3300/0 conn(0x7ff7380d4860 0x7ff7380d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.476+0000 7ff7377fe640 1 -- 172.21.15.83:0/767867370 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7380cf6a0 con 0x7ff7381614d0 2024-03-06T07:26:16.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.476+0000 7ff7377fe640 1 -- 172.21.15.83:0/767867370 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7ff728001670 con 0x7ff7381614d0 2024-03-06T07:26:16.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.476+0000 7ff7377fe640 1 -- 172.21.15.83:0/767867370 >> v1:172.21.15.83:6789/0 conn(0x7ff7381614d0 legacy=0x7ff7380d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.476+0000 7ff7377fe640 1 --2- 172.21.15.83:0/767867370 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7200039b0 0x7ff720003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.476+0000 7ff7377fe640 1 --2- 172.21.15.83:0/767867370 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff720004800 0x7ff720004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.476+0000 7ff7377fe640 1 --2- 172.21.15.83:0/767867370 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff7380b0e30 0x7ff7200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.476+0000 7ff7377fe640 1 -- 172.21.15.83:0/767867370 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7380d4cf0 con 0x7ff7200039b0 2024-03-06T07:26:16.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.476+0000 7ff7377fe640 1 -- 172.21.15.83:0/767867370 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7380cf400 con 0x7ff720004800 2024-03-06T07:26:16.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.476+0000 7ff7377fe640 1 -- 172.21.15.83:0/767867370 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff72000b9c0 con 0x7ff7380b0e30 2024-03-06T07:26:16.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.476+0000 7ff73cb01640 1 --2- 172.21.15.83:0/767867370 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7200039b0 0x7ff720003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.476+0000 7ff737fff640 1 --2- 172.21.15.83:0/767867370 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff720004800 0x7ff720004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.476+0000 7ff73d302640 1 --2- 172.21.15.83:0/767867370 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff7380b0e30 0x7ff7200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.476+0000 7ff73cb01640 1 -- 172.21.15.83:0/767867370 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff7380b0e30 msgr2=0x7ff7200051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:16.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.476+0000 7ff73cb01640 1 --2- 172.21.15.83:0/767867370 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff7380b0e30 0x7ff7200051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.476+0000 7ff73cb01640 1 -- 172.21.15.83:0/767867370 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff720004800 msgr2=0x7ff720004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:16.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.476+0000 7ff73cb01640 1 --2- 172.21.15.83:0/767867370 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff720004800 0x7ff720004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.476+0000 7ff73cb01640 1 -- 172.21.15.83:0/767867370 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff720003520 con 0x7ff7200039b0 2024-03-06T07:26:16.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.477+0000 7ff73cb01640 1 --2- 172.21.15.83:0/767867370 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7200039b0 0x7ff720003dd0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7ff728001050 tx=0x7ff720005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7de97bc1dfd3d57a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:16.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.477+0000 7ff7377fe640 1 -- 172.21.15.83:0/767867370 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff728007d70 con 0x7ff7200039b0 2024-03-06T07:26:16.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.477+0000 7ff7377fe640 1 -- 172.21.15.83:0/767867370 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff728030070 con 0x7ff7200039b0 2024-03-06T07:26:16.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.477+0000 7ff7377fe640 1 -- 172.21.15.83:0/767867370 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff72802cd80 con 0x7ff7200039b0 2024-03-06T07:26:16.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.478+0000 7ff73ed65640 1 -- 172.21.15.83:0/767867370 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7200039b0 msgr2=0x7ff720003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.478+0000 7ff73ed65640 1 --2- 172.21.15.83:0/767867370 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7200039b0 0x7ff720003dd0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7ff728001050 tx=0x7ff720005830 comp rx=0 tx=0).stop 2024-03-06T07:26:16.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.478+0000 7ff73d302640 1 -- 172.21.15.83:0/767867370 reap_dead start 2024-03-06T07:26:16.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.478+0000 7ff73ed65640 1 -- 172.21.15.83:0/767867370 shutdown_connections 2024-03-06T07:26:16.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.478+0000 7ff73ed65640 1 -- 172.21.15.83:0/767867370 >> 172.21.15.83:0/767867370 conn(0x7ff7380d3d40 msgr2=0x7ff7380d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:16.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.478+0000 7ff73ed65640 1 -- 172.21.15.83:0/767867370 shutdown_connections 2024-03-06T07:26:16.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.478+0000 7ff73ed65640 1 -- 172.21.15.83:0/767867370 wait complete. 2024-03-06T07:26:16.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.479+0000 7ff73ed65640 1 Processor -- start 2024-03-06T07:26:16.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.479+0000 7ff73ed65640 1 -- start start 2024-03-06T07:26:16.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.480+0000 7ff73ed65640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff7200039b0 0x7ff7380caf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.480+0000 7ff73ed65640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff738152250 0x7ff7380cd9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.480+0000 7ff73ed65640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7380b3540 0x7ff7380b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.480+0000 7ff73ed65640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7380b0910 con 0x7ff7380b3540 2024-03-06T07:26:16.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.480+0000 7ff73ed65640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7380cfc60 con 0x7ff738152250 2024-03-06T07:26:16.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.480+0000 7ff73ed65640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7380d2560 con 0x7ff7200039b0 2024-03-06T07:26:16.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.480+0000 7ff73d302640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7380b3540 0x7ff7380b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.480+0000 7ff73d302640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7380b3540 0x7ff7380b39c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53140/0 (socket says 172.21.15.83:53140) 2024-03-06T07:26:16.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.480+0000 7ff73d302640 1 -- 172.21.15.83:0/4200154981 learned_addr learned my addr 172.21.15.83:0/4200154981 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:16.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.480+0000 7ff737fff640 1 --2- 172.21.15.83:0/4200154981 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff738152250 0x7ff7380cd9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.480+0000 7ff73cb01640 1 --2- 172.21.15.83:0/4200154981 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff7200039b0 0x7ff7380caf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.480+0000 7ff73d302640 1 -- 172.21.15.83:0/4200154981 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff7200039b0 msgr2=0x7ff7380caf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.480+0000 7ff73d302640 1 --2- 172.21.15.83:0/4200154981 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff7200039b0 0x7ff7380caf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.480+0000 7ff73d302640 1 -- 172.21.15.83:0/4200154981 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff738152250 msgr2=0x7ff7380cd9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.480+0000 7ff73d302640 1 --2- 172.21.15.83:0/4200154981 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff738152250 0x7ff7380cd9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.480+0000 7ff73d302640 1 -- 172.21.15.83:0/4200154981 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7381526d0 con 0x7ff7380b3540 2024-03-06T07:26:16.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.480+0000 7ff73d302640 1 --2- 172.21.15.83:0/4200154981 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7380b3540 0x7ff7380b39c0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7ff7380cadc0 tx=0x7ff7300115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:16.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.480+0000 7ff7357fa640 1 -- 172.21.15.83:0/4200154981 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff730011db0 con 0x7ff7380b3540 2024-03-06T07:26:16.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.481+0000 7ff7357fa640 1 -- 172.21.15.83:0/4200154981 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff73001b070 con 0x7ff7380b3540 2024-03-06T07:26:16.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.481+0000 7ff73ed65640 1 -- 172.21.15.83:0/4200154981 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7380cdf10 con 0x7ff7380b3540 2024-03-06T07:26:16.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.481+0000 7ff7357fa640 1 -- 172.21.15.83:0/4200154981 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff730003990 con 0x7ff7380b3540 2024-03-06T07:26:16.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.481+0000 7ff73ed65640 1 -- 172.21.15.83:0/4200154981 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7380b6d40 con 0x7ff7380b3540 2024-03-06T07:26:16.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.482+0000 7ff73ed65640 1 -- 172.21.15.83:0/4200154981 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff738068ff0 con 0x7ff7380b3540 2024-03-06T07:26:16.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.482+0000 7ff7357fa640 1 -- 172.21.15.83:0/4200154981 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff730005d10 con 0x7ff7380b3540 2024-03-06T07:26:16.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.482+0000 7ff7357fa640 1 --2- 172.21.15.83:0/4200154981 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff72406b4d0 0x7ff72406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.482+0000 7ff7357fa640 1 -- 172.21.15.83:0/4200154981 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff73008a3d0 con 0x7ff7380b3540 2024-03-06T07:26:16.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.482+0000 7ff73cb01640 1 --2- 172.21.15.83:0/4200154981 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff72406b4d0 0x7ff72406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.483+0000 7ff73cb01640 1 --2- 172.21.15.83:0/4200154981 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff72406b4d0 0x7ff72406d990 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7ff728034000 tx=0x7ff728007000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:16.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.487+0000 7ff7357fa640 1 -- 172.21.15.83:0/4200154981 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff730025020 con 0x7ff7380b3540 2024-03-06T07:26:16.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.656+0000 7ff73ed65640 1 -- 172.21.15.83:0/4200154981 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff7380c71c0 con 0x7ff7380b3540 2024-03-06T07:26:16.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.656+0000 7ff7357fa640 1 -- 172.21.15.83:0/4200154981 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff73001d310 con 0x7ff7380b3540 2024-03-06T07:26:16.658 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:26:16.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.658+0000 7ff73ed65640 1 -- 172.21.15.83:0/4200154981 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff72406b4d0 msgr2=0x7ff72406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.658+0000 7ff73ed65640 1 --2- 172.21.15.83:0/4200154981 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff72406b4d0 0x7ff72406d990 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7ff728034000 tx=0x7ff728007000 comp rx=0 tx=0).stop 2024-03-06T07:26:16.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.658+0000 7ff73ed65640 1 -- 172.21.15.83:0/4200154981 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7380b3540 msgr2=0x7ff7380b39c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.658+0000 7ff73ed65640 1 --2- 172.21.15.83:0/4200154981 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7380b3540 0x7ff7380b39c0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7ff7380cadc0 tx=0x7ff7300115a0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.658+0000 7ff73ed65640 1 -- 172.21.15.83:0/4200154981 shutdown_connections 2024-03-06T07:26:16.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.658+0000 7ff73ed65640 1 --2- 172.21.15.83:0/4200154981 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff72406b4d0 0x7ff72406d990 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.658+0000 7ff73ed65640 1 --2- 172.21.15.83:0/4200154981 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7380b3540 0x7ff7380b39c0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.658+0000 7ff73ed65640 1 --2- 172.21.15.83:0/4200154981 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff738152250 0x7ff7380cd9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.658+0000 7ff73ed65640 1 --2- 172.21.15.83:0/4200154981 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff7200039b0 0x7ff7380caf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.658+0000 7ff73ed65640 1 -- 172.21.15.83:0/4200154981 >> 172.21.15.83:0/4200154981 conn(0x7ff7380d3d40 msgr2=0x7ff7380cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:16.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.658+0000 7ff73ed65640 1 -- 172.21.15.83:0/4200154981 shutdown_connections 2024-03-06T07:26:16.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.658+0000 7ff73ed65640 1 -- 172.21.15.83:0/4200154981 wait complete. 2024-03-06T07:26:16.671 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:26:16.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.769+0000 7f94642af640 1 Processor -- start 2024-03-06T07:26:16.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.769+0000 7f94642af640 1 -- start start 2024-03-06T07:26:16.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.770+0000 7f94642af640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f945c15bdd0 0x7f945c0c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.770+0000 7f94642af640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f945c0cebe0 con 0x7f945c0d16e0 2024-03-06T07:26:16.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.770+0000 7f94642af640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f945c0ceec0 con 0x7f945c15fa10 2024-03-06T07:26:16.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.770+0000 7f94642af640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f945c0ce980 con 0x7f945c15bdd0 2024-03-06T07:26:16.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.770+0000 7f946284c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f945c0d16e0 0x7f945c0d1b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:35376/0 (socket says 172.21.15.83:35376) 2024-03-06T07:26:16.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.770+0000 7f946284c640 1 -- 172.21.15.83:0/3841825282 learned_addr learned my addr 172.21.15.83:0/3841825282 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:16.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.771+0000 7f946204b640 1 --2- 172.21.15.83:0/3841825282 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f945c15bdd0 0x7f945c0c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.771+0000 7f9461049640 1 -- 172.21.15.83:0/3841825282 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2840641275 0 0) 0x7f945c0cebe0 con 0x7f945c0d16e0 2024-03-06T07:26:16.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.771+0000 7f9461049640 1 -- 172.21.15.83:0/3841825282 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9448003520 con 0x7f945c0d16e0 2024-03-06T07:26:16.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.771+0000 7f9461049640 1 -- 172.21.15.83:0/3841825282 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2029206136 0 0) 0x7f945c0ceec0 con 0x7f945c15fa10 2024-03-06T07:26:16.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.771+0000 7f9461049640 1 -- 172.21.15.83:0/3841825282 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f945c0cebe0 con 0x7f945c15fa10 2024-03-06T07:26:16.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.771+0000 7f9461049640 1 -- 172.21.15.83:0/3841825282 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 23783060 0 0) 0x7f9448003520 con 0x7f945c0d16e0 2024-03-06T07:26:16.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.771+0000 7f946204b640 1 -- 172.21.15.83:0/3841825282 >> v1:172.21.15.183:6789/0 conn(0x7f945c15fa10 legacy=0x7f945c0ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.771+0000 7f946204b640 1 -- 172.21.15.83:0/3841825282 >> v1:172.21.15.83:6789/0 conn(0x7f945c0d16e0 legacy=0x7f945c0d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.771+0000 7f946204b640 1 -- 172.21.15.83:0/3841825282 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f945c0cf1a0 con 0x7f945c15bdd0 2024-03-06T07:26:16.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.772+0000 7f946204b640 1 --2- 172.21.15.83:0/3841825282 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f945c15bdd0 0x7f945c0c5610 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9458009dc0 tx=0x7f945802bd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eccb429375d67e2a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:16.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.772+0000 7f9461049640 1 -- 172.21.15.83:0/3841825282 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9458039070 con 0x7f945c15bdd0 2024-03-06T07:26:16.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.772+0000 7f9461049640 1 -- 172.21.15.83:0/3841825282 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f945802c880 con 0x7f945c15bdd0 2024-03-06T07:26:16.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.772+0000 7f9461049640 1 -- 172.21.15.83:0/3841825282 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9454001670 con 0x7f945c15bdd0 2024-03-06T07:26:16.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.773+0000 7f94642af640 1 -- 172.21.15.83:0/3841825282 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f945c15bdd0 msgr2=0x7f945c0c5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.773+0000 7f94642af640 1 --2- 172.21.15.83:0/3841825282 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f945c15bdd0 0x7f945c0c5610 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9458009dc0 tx=0x7f945802bd50 comp rx=0 tx=0).stop 2024-03-06T07:26:16.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.773+0000 7f94642af640 1 -- 172.21.15.83:0/3841825282 shutdown_connections 2024-03-06T07:26:16.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.773+0000 7f94642af640 1 --2- 172.21.15.83:0/3841825282 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f945c15bdd0 0x7f945c0c5610 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.773+0000 7f94642af640 1 -- 172.21.15.83:0/3841825282 >> 172.21.15.83:0/3841825282 conn(0x7f945c09a9f0 msgr2=0x7f945c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:16.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.773+0000 7f94642af640 1 -- 172.21.15.83:0/3841825282 shutdown_connections 2024-03-06T07:26:16.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.773+0000 7f94642af640 1 -- 172.21.15.83:0/3841825282 wait complete. 2024-03-06T07:26:16.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.774+0000 7f94642af640 1 Processor -- start 2024-03-06T07:26:16.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.774+0000 7f94642af640 1 -- start start 2024-03-06T07:26:16.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.774+0000 7f94642af640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f945c0d16e0 0x7f945c15a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.774+0000 7f94642af640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f945c15bdd0 0x7f945c15d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.774+0000 7f94642af640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f945c15fa10 0x7f945c0b51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.774+0000 7f94642af640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945c0cc5d0 con 0x7f945c0d16e0 2024-03-06T07:26:16.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.774+0000 7f94642af640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945c0c5b50 con 0x7f945c15fa10 2024-03-06T07:26:16.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.774+0000 7f94642af640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f945c15c1b0 con 0x7f945c15bdd0 2024-03-06T07:26:16.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.775+0000 7f946204b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f945c0d16e0 0x7f945c15a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.775+0000 7f946204b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f945c0d16e0 0x7f945c15a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50320/0 (socket says 172.21.15.83:50320) 2024-03-06T07:26:16.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.775+0000 7f946204b640 1 -- 172.21.15.83:0/4252659761 learned_addr learned my addr 172.21.15.83:0/4252659761 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:16.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.775+0000 7f946284c640 1 --2- 172.21.15.83:0/4252659761 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f945c15fa10 0x7f945c0b51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.775+0000 7f946184a640 1 --2- 172.21.15.83:0/4252659761 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f945c15bdd0 0x7f945c15d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.775+0000 7f946204b640 1 -- 172.21.15.83:0/4252659761 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f945c15bdd0 msgr2=0x7f945c15d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.775+0000 7f946204b640 1 --2- 172.21.15.83:0/4252659761 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f945c15bdd0 0x7f945c15d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.775+0000 7f946204b640 1 -- 172.21.15.83:0/4252659761 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f945c15fa10 msgr2=0x7f945c0b51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.775+0000 7f946204b640 1 --2- 172.21.15.83:0/4252659761 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f945c15fa10 0x7f945c0b51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.775+0000 7f946204b640 1 -- 172.21.15.83:0/4252659761 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f945c15ae90 con 0x7f945c0d16e0 2024-03-06T07:26:16.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.775+0000 7f946184a640 1 --2- 172.21.15.83:0/4252659761 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f945c15bdd0 0x7f945c15d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:16.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.775+0000 7f946284c640 1 --2- 172.21.15.83:0/4252659761 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f945c15fa10 0x7f945c0b51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:16.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.775+0000 7f946204b640 1 --2- 172.21.15.83:0/4252659761 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f945c0d16e0 0x7f945c15a950 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f9458000f80 tx=0x7f9458004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:16.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.775+0000 7f9452ffd640 1 -- 172.21.15.83:0/4252659761 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9458039050 con 0x7f945c0d16e0 2024-03-06T07:26:16.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.775+0000 7f9452ffd640 1 -- 172.21.15.83:0/4252659761 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9458045070 con 0x7f945c0d16e0 2024-03-06T07:26:16.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.775+0000 7f94642af640 1 -- 172.21.15.83:0/4252659761 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f945c15e1a0 con 0x7f945c0d16e0 2024-03-06T07:26:16.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.775+0000 7f9452ffd640 1 -- 172.21.15.83:0/4252659761 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9458009450 con 0x7f945c0d16e0 2024-03-06T07:26:16.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.775+0000 7f94642af640 1 -- 172.21.15.83:0/4252659761 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f945c15e760 con 0x7f945c0d16e0 2024-03-06T07:26:16.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.776+0000 7f94642af640 1 -- 172.21.15.83:0/4252659761 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f945c04f330 con 0x7f945c0d16e0 2024-03-06T07:26:16.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.777+0000 7f9452ffd640 1 -- 172.21.15.83:0/4252659761 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9458040690 con 0x7f945c0d16e0 2024-03-06T07:26:16.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.777+0000 7f9452ffd640 1 --2- 172.21.15.83:0/4252659761 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f944006b4d0 0x7f944006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.777+0000 7f9452ffd640 1 -- 172.21.15.83:0/4252659761 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f94580ac720 con 0x7f945c0d16e0 2024-03-06T07:26:16.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.777+0000 7f946184a640 1 --2- 172.21.15.83:0/4252659761 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f944006b4d0 0x7f944006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.777+0000 7f946184a640 1 --2- 172.21.15.83:0/4252659761 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f944006b4d0 0x7f944006d990 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f945c0c53a0 tx=0x7f944c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:16.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.781+0000 7f9452ffd640 1 -- 172.21.15.83:0/4252659761 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f945807b900 con 0x7f945c0d16e0 2024-03-06T07:26:16.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.955+0000 7f94642af640 1 -- 172.21.15.83:0/4252659761 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f945c0b7a40 con 0x7f945c0d16e0 2024-03-06T07:26:16.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.955+0000 7f9452ffd640 1 -- 172.21.15.83:0/4252659761 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v90) v1 ==== 84+0+65 (secure 0 0 0) 0x7f945807b200 con 0x7f945c0d16e0 2024-03-06T07:26:16.957 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:16.957 INFO:teuthology.orchestra.run.smithi083.stdout:[ 2024-03-06T07:26:16.957 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:26:16.957 INFO:teuthology.orchestra.run.smithi083.stdout: "poolnum": 1, 2024-03-06T07:26:16.957 INFO:teuthology.orchestra.run.smithi083.stdout: "poolname": ".mgr" 2024-03-06T07:26:16.957 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:26:16.958 INFO:teuthology.orchestra.run.smithi083.stdout:] 2024-03-06T07:26:16.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.957+0000 7f94642af640 1 -- 172.21.15.83:0/4252659761 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f944006b4d0 msgr2=0x7f944006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.957+0000 7f94642af640 1 --2- 172.21.15.83:0/4252659761 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f944006b4d0 0x7f944006d990 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f945c0c53a0 tx=0x7f944c009040 comp rx=0 tx=0).stop 2024-03-06T07:26:16.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.957+0000 7f94642af640 1 -- 172.21.15.83:0/4252659761 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f945c0d16e0 msgr2=0x7f945c15a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.957+0000 7f94642af640 1 --2- 172.21.15.83:0/4252659761 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f945c0d16e0 0x7f945c15a950 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f9458000f80 tx=0x7f9458004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:16.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.957+0000 7f94642af640 1 -- 172.21.15.83:0/4252659761 shutdown_connections 2024-03-06T07:26:16.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.957+0000 7f94642af640 1 --2- 172.21.15.83:0/4252659761 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f944006b4d0 0x7f944006d990 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.957+0000 7f94642af640 1 --2- 172.21.15.83:0/4252659761 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f945c15fa10 0x7f945c0b51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.957+0000 7f94642af640 1 --2- 172.21.15.83:0/4252659761 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f945c15bdd0 0x7f945c15d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.957+0000 7f94642af640 1 --2- 172.21.15.83:0/4252659761 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f945c0d16e0 0x7f945c15a950 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.957+0000 7f94642af640 1 -- 172.21.15.83:0/4252659761 >> 172.21.15.83:0/4252659761 conn(0x7f945c09a9f0 msgr2=0x7f945c0cd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:16.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.957+0000 7f94642af640 1 -- 172.21.15.83:0/4252659761 shutdown_connections 2024-03-06T07:26:16.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:16.957+0000 7f94642af640 1 -- 172.21.15.83:0/4252659761 wait complete. 2024-03-06T07:26:16.970 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:26:16.970 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:26:17.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.072+0000 7fe7b56aa640 1 Processor -- start 2024-03-06T07:26:17.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.072+0000 7fe7b56aa640 1 -- start start 2024-03-06T07:26:17.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.073+0000 7fe7b56aa640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fe7b0161440 0x7fe7b00b4be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.073+0000 7fe7b56aa640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7b00cf520 con 0x7fe7b00d0d20 2024-03-06T07:26:17.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.073+0000 7fe7b56aa640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7b00cf800 con 0x7fe7b0161820 2024-03-06T07:26:17.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.073+0000 7fe7b56aa640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7b00cf2d0 con 0x7fe7b0161440 2024-03-06T07:26:17.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.073+0000 7fe7af7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fe7b00d0d20 0x7fe7b00b6f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:35390/0 (socket says 172.21.15.83:35390) 2024-03-06T07:26:17.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.073+0000 7fe7af7fe640 1 -- 172.21.15.83:0/3393846111 learned_addr learned my addr 172.21.15.83:0/3393846111 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:17.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.073+0000 7fe7aeffd640 1 --2- 172.21.15.83:0/3393846111 >> v2:172.21.15.183:3300/0 conn(0x7fe7b0161440 0x7fe7b00b4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.074+0000 7fe7adffb640 1 -- 172.21.15.83:0/3393846111 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3453441297 0 0) 0x7fe7b00cf520 con 0x7fe7b00d0d20 2024-03-06T07:26:17.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.074+0000 7fe7adffb640 1 -- 172.21.15.83:0/3393846111 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe798003520 con 0x7fe7b00d0d20 2024-03-06T07:26:17.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.074+0000 7fe7adffb640 1 -- 172.21.15.83:0/3393846111 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 853099180 0 0) 0x7fe7b00cf800 con 0x7fe7b0161820 2024-03-06T07:26:17.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.074+0000 7fe7adffb640 1 -- 172.21.15.83:0/3393846111 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7b00cf520 con 0x7fe7b0161820 2024-03-06T07:26:17.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.074+0000 7fe7adffb640 1 -- 172.21.15.83:0/3393846111 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 233308052 0 0) 0x7fe798003520 con 0x7fe7b00d0d20 2024-03-06T07:26:17.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.074+0000 7fe7aeffd640 1 -- 172.21.15.83:0/3393846111 >> v1:172.21.15.183:6789/0 conn(0x7fe7b0161820 legacy=0x7fe7b00d0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.074+0000 7fe7aeffd640 1 -- 172.21.15.83:0/3393846111 >> v1:172.21.15.83:6789/0 conn(0x7fe7b00d0d20 legacy=0x7fe7b00b6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.074+0000 7fe7aeffd640 1 -- 172.21.15.83:0/3393846111 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b00cfae0 con 0x7fe7b0161440 2024-03-06T07:26:17.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.075+0000 7fe7aeffd640 1 --2- 172.21.15.83:0/3393846111 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe7b0161440 0x7fe7b00b4be0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fe7a00097a0 tx=0x7fe7a002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a398d4407ecfe35 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:17.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.075+0000 7fe7adffb640 1 -- 172.21.15.83:0/3393846111 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a0037070 con 0x7fe7b0161440 2024-03-06T07:26:17.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.075+0000 7fe7adffb640 1 -- 172.21.15.83:0/3393846111 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7a0030510 con 0x7fe7b0161440 2024-03-06T07:26:17.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.075+0000 7fe7adffb640 1 -- 172.21.15.83:0/3393846111 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a8001a10 con 0x7fe7b0161440 2024-03-06T07:26:17.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.075+0000 7fe7b56aa640 1 -- 172.21.15.83:0/3393846111 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe7b0161440 msgr2=0x7fe7b00b4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.075+0000 7fe7b56aa640 1 --2- 172.21.15.83:0/3393846111 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe7b0161440 0x7fe7b00b4be0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fe7a00097a0 tx=0x7fe7a002b790 comp rx=0 tx=0).stop 2024-03-06T07:26:17.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.075+0000 7fe7b56aa640 1 -- 172.21.15.83:0/3393846111 shutdown_connections 2024-03-06T07:26:17.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.075+0000 7fe7b56aa640 1 --2- 172.21.15.83:0/3393846111 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe7b0161440 0x7fe7b00b4be0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.075+0000 7fe7b56aa640 1 -- 172.21.15.83:0/3393846111 >> 172.21.15.83:0/3393846111 conn(0x7fe7b00d3cb0 msgr2=0x7fe7b00d40c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:17.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.075+0000 7fe7b56aa640 1 -- 172.21.15.83:0/3393846111 shutdown_connections 2024-03-06T07:26:17.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.075+0000 7fe7b56aa640 1 -- 172.21.15.83:0/3393846111 wait complete. 2024-03-06T07:26:17.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.077+0000 7fe7b56aa640 1 Processor -- start 2024-03-06T07:26:17.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.077+0000 7fe7b56aa640 1 -- start start 2024-03-06T07:26:17.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.077+0000 7fe7b56aa640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe7b00d0d20 0x7fe7b009da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.077+0000 7fe7b56aa640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7b0161440 0x7fe7b00c9890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.077+0000 7fe7b56aa640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe7b0161820 0x7fe7b0150d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.077+0000 7fe7b56aa640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b00d1300 con 0x7fe7b0161440 2024-03-06T07:26:17.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.077+0000 7fe7b56aa640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b00b6b00 con 0x7fe7b00d0d20 2024-03-06T07:26:17.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.077+0000 7fe7b56aa640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b00d1480 con 0x7fe7b0161820 2024-03-06T07:26:17.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.077+0000 7fe7ae7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7b0161440 0x7fe7b00c9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.077+0000 7fe7ae7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7b0161440 0x7fe7b00c9890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50326/0 (socket says 172.21.15.83:50326) 2024-03-06T07:26:17.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.077+0000 7fe7ae7fc640 1 -- 172.21.15.83:0/3969485375 learned_addr learned my addr 172.21.15.83:0/3969485375 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:17.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.077+0000 7fe7aeffd640 1 --2- 172.21.15.83:0/3969485375 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe7b00d0d20 0x7fe7b009da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.078+0000 7fe7af7fe640 1 --2- 172.21.15.83:0/3969485375 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe7b0161820 0x7fe7b0150d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.078+0000 7fe7ae7fc640 1 -- 172.21.15.83:0/3969485375 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe7b0161820 msgr2=0x7fe7b0150d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.078+0000 7fe7ae7fc640 1 --2- 172.21.15.83:0/3969485375 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe7b0161820 0x7fe7b0150d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.078+0000 7fe7ae7fc640 1 -- 172.21.15.83:0/3969485375 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe7b00d0d20 msgr2=0x7fe7b009da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.078+0000 7fe7ae7fc640 1 --2- 172.21.15.83:0/3969485375 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe7b00d0d20 0x7fe7b009da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.078+0000 7fe7ae7fc640 1 -- 172.21.15.83:0/3969485375 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b015db10 con 0x7fe7b0161440 2024-03-06T07:26:17.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.078+0000 7fe7ae7fc640 1 --2- 172.21.15.83:0/3969485375 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7b0161440 0x7fe7b00c9890 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fe7a400c450 tx=0x7fe7a400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:17.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.078+0000 7fe7977fe640 1 -- 172.21.15.83:0/3969485375 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a4017070 con 0x7fe7b0161440 2024-03-06T07:26:17.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.078+0000 7fe7977fe640 1 -- 172.21.15.83:0/3969485375 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7a40052d0 con 0x7fe7b0161440 2024-03-06T07:26:17.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.078+0000 7fe7b56aa640 1 -- 172.21.15.83:0/3969485375 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7b00a09d0 con 0x7fe7b0161440 2024-03-06T07:26:17.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.078+0000 7fe7977fe640 1 -- 172.21.15.83:0/3969485375 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a4003830 con 0x7fe7b0161440 2024-03-06T07:26:17.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.078+0000 7fe7b56aa640 1 -- 172.21.15.83:0/3969485375 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7b00a0f90 con 0x7fe7b0161440 2024-03-06T07:26:17.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.079+0000 7fe7b56aa640 1 -- 172.21.15.83:0/3969485375 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7b004f330 con 0x7fe7b0161440 2024-03-06T07:26:17.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.080+0000 7fe7977fe640 1 -- 172.21.15.83:0/3969485375 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe7a401a070 con 0x7fe7b0161440 2024-03-06T07:26:17.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.080+0000 7fe7977fe640 1 --2- 172.21.15.83:0/3969485375 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe79c06b4d0 0x7fe79c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.080+0000 7fe7977fe640 1 -- 172.21.15.83:0/3969485375 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe7a4089240 con 0x7fe7b0161440 2024-03-06T07:26:17.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.080+0000 7fe7aeffd640 1 --2- 172.21.15.83:0/3969485375 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe79c06b4d0 0x7fe79c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.080+0000 7fe7aeffd640 1 --2- 172.21.15.83:0/3969485375 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe79c06b4d0 0x7fe79c06d990 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fe7a0034000 tx=0x7fe7a002f000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:17.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.084+0000 7fe7977fe640 1 -- 172.21.15.83:0/3969485375 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe7a4057e80 con 0x7fe7b0161440 2024-03-06T07:26:17.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.256+0000 7fe7b56aa640 1 -- 172.21.15.83:0/3969485375 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fe7b0160ac0 con 0x7fe7b0161440 2024-03-06T07:26:17.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.702+0000 7fe7977fe640 1 -- 172.21.15.83:0/3969485375 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v91) v1 ==== 138+0+0 (secure 0 0 0) 0x7fe7a405bf20 con 0x7fe7b0161440 2024-03-06T07:26:17.704 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'cephfs_metadata' created 2024-03-06T07:26:17.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.704+0000 7fe7b56aa640 1 -- 172.21.15.83:0/3969485375 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe79c06b4d0 msgr2=0x7fe79c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.704+0000 7fe7b56aa640 1 --2- 172.21.15.83:0/3969485375 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe79c06b4d0 0x7fe79c06d990 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fe7a0034000 tx=0x7fe7a002f000 comp rx=0 tx=0).stop 2024-03-06T07:26:17.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.704+0000 7fe7b56aa640 1 -- 172.21.15.83:0/3969485375 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7b0161440 msgr2=0x7fe7b00c9890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.704+0000 7fe7b56aa640 1 --2- 172.21.15.83:0/3969485375 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7b0161440 0x7fe7b00c9890 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fe7a400c450 tx=0x7fe7a400c910 comp rx=0 tx=0).stop 2024-03-06T07:26:17.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.704+0000 7fe7b56aa640 1 -- 172.21.15.83:0/3969485375 shutdown_connections 2024-03-06T07:26:17.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.704+0000 7fe7b56aa640 1 --2- 172.21.15.83:0/3969485375 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe79c06b4d0 0x7fe79c06d990 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.704+0000 7fe7b56aa640 1 --2- 172.21.15.83:0/3969485375 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe7b0161820 0x7fe7b0150d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.704+0000 7fe7b56aa640 1 --2- 172.21.15.83:0/3969485375 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7b0161440 0x7fe7b00c9890 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.704+0000 7fe7b56aa640 1 --2- 172.21.15.83:0/3969485375 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe7b00d0d20 0x7fe7b009da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.704+0000 7fe7b56aa640 1 -- 172.21.15.83:0/3969485375 >> 172.21.15.83:0/3969485375 conn(0x7fe7b00d3cb0 msgr2=0x7fe7b015fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:17.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.704+0000 7fe7b56aa640 1 -- 172.21.15.83:0/3969485375 shutdown_connections 2024-03-06T07:26:17.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.704+0000 7fe7b56aa640 1 -- 172.21.15.83:0/3969485375 wait complete. 2024-03-06T07:26:17.721 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:26:17.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.826+0000 7f83b845c640 1 Processor -- start 2024-03-06T07:26:17.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.827+0000 7f83b845c640 1 -- start start 2024-03-06T07:26:17.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.827+0000 7f83b845c640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f83b01586e0 0x7f83b00aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.827+0000 7f83b845c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f83b00aef80 0x7f83b00cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.828+0000 7f83b845c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b0158ac0 con 0x7f83b00cffd0 2024-03-06T07:26:17.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.828+0000 7f83b845c640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83b015b2a0 con 0x7f83b00aef80 2024-03-06T07:26:17.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.828+0000 7f83b845c640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83b015b540 con 0x7f83b01586e0 2024-03-06T07:26:17.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.828+0000 7f83b59f7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f83b00aef80 0x7f83b00cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.828+0000 7f83b59f7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f83b00aef80 0x7f83b00cfa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50334/0 (socket says 172.21.15.83:50334) 2024-03-06T07:26:17.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.828+0000 7f83b59f7640 1 -- 172.21.15.83:0/3306097549 learned_addr learned my addr 172.21.15.83:0/3306097549 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:17.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.828+0000 7f83b59f7640 1 -- 172.21.15.83:0/3306097549 >> v2:172.21.15.183:3300/0 conn(0x7f83b01586e0 msgr2=0x7f83b00aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:17.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.828+0000 7f83b59f7640 1 --2- 172.21.15.83:0/3306097549 >> v2:172.21.15.183:3300/0 conn(0x7f83b01586e0 0x7f83b00aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.828+0000 7f83b59f7640 1 -- 172.21.15.83:0/3306097549 >> v1:172.21.15.183:6789/0 conn(0x7f83b00cffd0 legacy=0x7f83b006da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:17.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.828+0000 7f83b59f7640 1 -- 172.21.15.83:0/3306097549 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b015b820 con 0x7f83b00aef80 2024-03-06T07:26:17.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.829+0000 7f83b59f7640 1 --2- 172.21.15.83:0/3306097549 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83b00aef80 0x7f83b00cfa90 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f83b0070d10 tx=0x7f83a002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=484e483f759ab19f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:17.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.829+0000 7f83b51f6640 1 -- 172.21.15.83:0/3306097549 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a0035020 con 0x7f83b00aef80 2024-03-06T07:26:17.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.829+0000 7f83b51f6640 1 -- 172.21.15.83:0/3306097549 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a0031870 con 0x7f83b00aef80 2024-03-06T07:26:17.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.829+0000 7f83b51f6640 1 -- 172.21.15.83:0/3306097549 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a0031c40 con 0x7f83b00aef80 2024-03-06T07:26:17.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.829+0000 7f83b845c640 1 -- 172.21.15.83:0/3306097549 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83b00aef80 msgr2=0x7f83b00cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.829+0000 7f83b845c640 1 --2- 172.21.15.83:0/3306097549 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83b00aef80 0x7f83b00cfa90 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f83b0070d10 tx=0x7f83a002b910 comp rx=0 tx=0).stop 2024-03-06T07:26:17.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.829+0000 7f83b845c640 1 -- 172.21.15.83:0/3306097549 shutdown_connections 2024-03-06T07:26:17.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.829+0000 7f83b845c640 1 --2- 172.21.15.83:0/3306097549 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83b00aef80 0x7f83b00cfa90 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.829+0000 7f83b845c640 1 --2- 172.21.15.83:0/3306097549 >> v2:172.21.15.183:3300/0 conn(0x7f83b01586e0 0x7f83b00aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.829+0000 7f83b845c640 1 -- 172.21.15.83:0/3306097549 >> 172.21.15.83:0/3306097549 conn(0x7f83b006b190 msgr2=0x7f83b006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:17.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.830+0000 7f83b845c640 1 -- 172.21.15.83:0/3306097549 shutdown_connections 2024-03-06T07:26:17.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.830+0000 7f83b845c640 1 -- 172.21.15.83:0/3306097549 wait complete. 2024-03-06T07:26:17.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.831+0000 7f83b845c640 1 Processor -- start 2024-03-06T07:26:17.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.831+0000 7f83b845c640 1 -- start start 2024-03-06T07:26:17.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.831+0000 7f83b845c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f83b00aef80 0x7f83b013fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.831+0000 7f83b845c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83b00cffd0 0x7f83b0149d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.831+0000 7f83b845c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f83b01586e0 0x7f83b00ce550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.831+0000 7f83b845c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b0071350 con 0x7f83b00cffd0 2024-03-06T07:26:17.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.831+0000 7f83b845c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b015bde0 con 0x7f83b00aef80 2024-03-06T07:26:17.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.831+0000 7f83b845c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b00714d0 con 0x7f83b01586e0 2024-03-06T07:26:17.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.831+0000 7f83b59f7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83b00cffd0 0x7f83b0149d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.831+0000 7f83b59f7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83b00cffd0 0x7f83b0149d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50346/0 (socket says 172.21.15.83:50346) 2024-03-06T07:26:17.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.831+0000 7f83b59f7640 1 -- 172.21.15.83:0/925817566 learned_addr learned my addr 172.21.15.83:0/925817566 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:17.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.832+0000 7f83b69f9640 1 --2- 172.21.15.83:0/925817566 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f83b01586e0 0x7f83b00ce550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.832+0000 7f83b61f8640 1 --2- 172.21.15.83:0/925817566 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f83b00aef80 0x7f83b013fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.832+0000 7f83b59f7640 1 -- 172.21.15.83:0/925817566 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f83b01586e0 msgr2=0x7f83b00ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.832+0000 7f83b59f7640 1 --2- 172.21.15.83:0/925817566 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f83b01586e0 0x7f83b00ce550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.832+0000 7f83b59f7640 1 -- 172.21.15.83:0/925817566 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f83b00aef80 msgr2=0x7f83b013fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.832+0000 7f83b59f7640 1 --2- 172.21.15.83:0/925817566 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f83b00aef80 0x7f83b013fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.832+0000 7f83b59f7640 1 -- 172.21.15.83:0/925817566 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b01416b0 con 0x7f83b00cffd0 2024-03-06T07:26:17.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.832+0000 7f83b61f8640 1 --2- 172.21.15.83:0/925817566 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f83b00aef80 0x7f83b013fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:17.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.832+0000 7f83b69f9640 1 --2- 172.21.15.83:0/925817566 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f83b01586e0 0x7f83b00ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:17.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.832+0000 7f83b59f7640 1 --2- 172.21.15.83:0/925817566 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83b00cffd0 0x7f83b0149d80 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f83a0000c00 tx=0x7f83a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:17.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.832+0000 7f83a6ffd640 1 -- 172.21.15.83:0/925817566 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a003ad60 con 0x7f83b00cffd0 2024-03-06T07:26:17.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.832+0000 7f83a6ffd640 1 -- 172.21.15.83:0/925817566 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a00443f0 con 0x7f83b00cffd0 2024-03-06T07:26:17.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.832+0000 7f83b845c640 1 -- 172.21.15.83:0/925817566 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83b00cda70 con 0x7f83b00cffd0 2024-03-06T07:26:17.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.832+0000 7f83a6ffd640 1 -- 172.21.15.83:0/925817566 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a00447e0 con 0x7f83b00cffd0 2024-03-06T07:26:17.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.832+0000 7f83b845c640 1 -- 172.21.15.83:0/925817566 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83b01403a0 con 0x7f83b00cffd0 2024-03-06T07:26:17.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.833+0000 7f83b845c640 1 -- 172.21.15.83:0/925817566 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83b00732b0 con 0x7f83b00cffd0 2024-03-06T07:26:17.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.834+0000 7f83a6ffd640 1 -- 172.21.15.83:0/925817566 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f83a004b060 con 0x7f83b00cffd0 2024-03-06T07:26:17.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.834+0000 7f83a6ffd640 1 --2- 172.21.15.83:0/925817566 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f839406b4d0 0x7f839406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.834+0000 7f83a6ffd640 1 -- 172.21.15.83:0/925817566 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f83a00ac1f0 con 0x7f83b00cffd0 2024-03-06T07:26:17.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.834+0000 7f83b61f8640 1 --2- 172.21.15.83:0/925817566 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f839406b4d0 0x7f839406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.834+0000 7f83b61f8640 1 --2- 172.21.15.83:0/925817566 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f839406b4d0 0x7f839406d990 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f83b014c130 tx=0x7f83ac008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:17.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:17.839+0000 7f83a6ffd640 1 -- 172.21.15.83:0/925817566 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f83a003f070 con 0x7f83b00cffd0 2024-03-06T07:26:18.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.010+0000 7f83b845c640 1 -- 172.21.15.83:0/925817566 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f83b0068ff0 con 0x7f83b00cffd0 2024-03-06T07:26:18.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.709+0000 7f83a6ffd640 1 -- 172.21.15.83:0/925817566 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v92) v1 ==== 170+0+0 (secure 0 0 0) 0x7f83a002f320 con 0x7f83b00cffd0 2024-03-06T07:26:18.711 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'cephfs_data' created 2024-03-06T07:26:18.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.712+0000 7f83b845c640 1 -- 172.21.15.83:0/925817566 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f839406b4d0 msgr2=0x7f839406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.712+0000 7f83b845c640 1 --2- 172.21.15.83:0/925817566 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f839406b4d0 0x7f839406d990 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f83b014c130 tx=0x7f83ac008040 comp rx=0 tx=0).stop 2024-03-06T07:26:18.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.712+0000 7f83b845c640 1 -- 172.21.15.83:0/925817566 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83b00cffd0 msgr2=0x7f83b0149d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.712+0000 7f83b845c640 1 --2- 172.21.15.83:0/925817566 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83b00cffd0 0x7f83b0149d80 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f83a0000c00 tx=0x7f83a00046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.712+0000 7f83b845c640 1 -- 172.21.15.83:0/925817566 shutdown_connections 2024-03-06T07:26:18.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.712+0000 7f83b845c640 1 --2- 172.21.15.83:0/925817566 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f839406b4d0 0x7f839406d990 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.712+0000 7f83b845c640 1 --2- 172.21.15.83:0/925817566 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f83b01586e0 0x7f83b00ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.712+0000 7f83b845c640 1 --2- 172.21.15.83:0/925817566 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83b00cffd0 0x7f83b0149d80 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.712+0000 7f83b845c640 1 --2- 172.21.15.83:0/925817566 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f83b00aef80 0x7f83b013fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.712+0000 7f83b845c640 1 -- 172.21.15.83:0/925817566 >> 172.21.15.83:0/925817566 conn(0x7f83b006b190 msgr2=0x7f83b0156ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:18.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.712+0000 7f83b845c640 1 -- 172.21.15.83:0/925817566 shutdown_connections 2024-03-06T07:26:18.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.712+0000 7f83b845c640 1 -- 172.21.15.83:0/925817566 wait complete. 2024-03-06T07:26:18.740 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:26:18.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.842+0000 7f860f688640 1 Processor -- start 2024-03-06T07:26:18.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.842+0000 7f860f688640 1 -- start start 2024-03-06T07:26:18.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.843+0000 7f860f688640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f860806b6c0 0x7f86081601b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.843+0000 7f860f688640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f860806fd80 0x7f860806da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.843+0000 7f860f688640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8608145850 con 0x7f8608145470 2024-03-06T07:26:18.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.843+0000 7f860f688640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86080cfac0 con 0x7f860806b6c0 2024-03-06T07:26:18.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.843+0000 7f860f688640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86080ceeb0 con 0x7f860806fd80 2024-03-06T07:26:18.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.843+0000 7f860cc23640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f860806b6c0 0x7f86081601b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.843+0000 7f860cc23640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f860806b6c0 0x7f86081601b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50354/0 (socket says 172.21.15.83:50354) 2024-03-06T07:26:18.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.843+0000 7f860cc23640 1 -- 172.21.15.83:0/1814658990 learned_addr learned my addr 172.21.15.83:0/1814658990 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:18.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.843+0000 7f860dc25640 1 --2- 172.21.15.83:0/1814658990 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f860806fd80 0x7f860806da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.843+0000 7f860cc23640 1 -- 172.21.15.83:0/1814658990 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f860806fd80 msgr2=0x7f860806da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:18.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.843+0000 7f860cc23640 1 --2- 172.21.15.83:0/1814658990 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f860806fd80 0x7f860806da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.843+0000 7f860cc23640 1 -- 172.21.15.83:0/1814658990 >> v1:172.21.15.183:6789/0 conn(0x7f8608145470 legacy=0x7f860806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:18.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.843+0000 7f860cc23640 1 -- 172.21.15.83:0/1814658990 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86080cf190 con 0x7f860806b6c0 2024-03-06T07:26:18.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.843+0000 7f860dc25640 1 --2- 172.21.15.83:0/1814658990 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f860806fd80 0x7f860806da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:18.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.844+0000 7f860cc23640 1 --2- 172.21.15.83:0/1814658990 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f860806b6c0 0x7f86081601b0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f85f8009870 tx=0x7f85f802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=550e1188c90b8a72 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:18.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.844+0000 7f85fffff640 1 -- 172.21.15.83:0/1814658990 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f80317c0 con 0x7f860806b6c0 2024-03-06T07:26:18.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.844+0000 7f85fffff640 1 -- 172.21.15.83:0/1814658990 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85f8037070 con 0x7f860806b6c0 2024-03-06T07:26:18.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.844+0000 7f85fffff640 1 -- 172.21.15.83:0/1814658990 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f8031b80 con 0x7f860806b6c0 2024-03-06T07:26:18.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.845+0000 7f860f688640 1 -- 172.21.15.83:0/1814658990 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f860806b6c0 msgr2=0x7f86081601b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.845+0000 7f860f688640 1 --2- 172.21.15.83:0/1814658990 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f860806b6c0 0x7f86081601b0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f85f8009870 tx=0x7f85f802b860 comp rx=0 tx=0).stop 2024-03-06T07:26:18.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.845+0000 7f860f688640 1 -- 172.21.15.83:0/1814658990 shutdown_connections 2024-03-06T07:26:18.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.845+0000 7f860f688640 1 --2- 172.21.15.83:0/1814658990 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f860806fd80 0x7f860806da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.845+0000 7f860f688640 1 --2- 172.21.15.83:0/1814658990 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f860806b6c0 0x7f86081601b0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.845+0000 7f860f688640 1 -- 172.21.15.83:0/1814658990 >> 172.21.15.83:0/1814658990 conn(0x7f860806c970 msgr2=0x7f86080b2580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:18.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.845+0000 7f860f688640 1 -- 172.21.15.83:0/1814658990 shutdown_connections 2024-03-06T07:26:18.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.845+0000 7f860f688640 1 -- 172.21.15.83:0/1814658990 wait complete. 2024-03-06T07:26:18.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.846+0000 7f860f688640 1 Processor -- start 2024-03-06T07:26:18.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.846+0000 7f860f688640 1 -- start start 2024-03-06T07:26:18.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.847+0000 7f860f688640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f860806b6c0 0x7f860815b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.847+0000 7f860f688640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f860806fd80 0x7f8608140ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.847+0000 7f860f688640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8608145470 0x7f860815e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.847+0000 7f860f688640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86080cff50 con 0x7f860806b6c0 2024-03-06T07:26:18.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.847+0000 7f860f688640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86080cf750 con 0x7f8608145470 2024-03-06T07:26:18.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.847+0000 7f860f688640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f860815d1b0 con 0x7f860806fd80 2024-03-06T07:26:18.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.847+0000 7f860d424640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f860806b6c0 0x7f860815b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.847+0000 7f860d424640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f860806b6c0 0x7f860815b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50368/0 (socket says 172.21.15.83:50368) 2024-03-06T07:26:18.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.847+0000 7f860d424640 1 -- 172.21.15.83:0/3320541466 learned_addr learned my addr 172.21.15.83:0/3320541466 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:18.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.847+0000 7f860d424640 1 -- 172.21.15.83:0/3320541466 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f860806fd80 msgr2=0x7f8608140ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.847+0000 7f860d424640 1 --2- 172.21.15.83:0/3320541466 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f860806fd80 0x7f8608140ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.847+0000 7f860d424640 1 -- 172.21.15.83:0/3320541466 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8608145470 msgr2=0x7f860815e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.847+0000 7f860d424640 1 --2- 172.21.15.83:0/3320541466 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8608145470 0x7f860815e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.847+0000 7f860d424640 1 -- 172.21.15.83:0/3320541466 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f860815d490 con 0x7f860806b6c0 2024-03-06T07:26:18.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.847+0000 7f860d424640 1 --2- 172.21.15.83:0/3320541466 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f860806b6c0 0x7f860815b2c0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f860400b3f0 tx=0x7f860400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:18.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.847+0000 7f85fdffb640 1 -- 172.21.15.83:0/3320541466 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8604015020 con 0x7f860806b6c0 2024-03-06T07:26:18.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.847+0000 7f85fdffb640 1 -- 172.21.15.83:0/3320541466 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8604011830 con 0x7f860806b6c0 2024-03-06T07:26:18.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.847+0000 7f860f688640 1 -- 172.21.15.83:0/3320541466 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f860815d770 con 0x7f860806b6c0 2024-03-06T07:26:18.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.847+0000 7f85fdffb640 1 -- 172.21.15.83:0/3320541466 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8604011c20 con 0x7f860806b6c0 2024-03-06T07:26:18.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.848+0000 7f860f688640 1 -- 172.21.15.83:0/3320541466 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f860815dd30 con 0x7f860806b6c0 2024-03-06T07:26:18.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.849+0000 7f860f688640 1 -- 172.21.15.83:0/3320541466 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f860804f330 con 0x7f860806b6c0 2024-03-06T07:26:18.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.849+0000 7f85fdffb640 1 -- 172.21.15.83:0/3320541466 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8604018020 con 0x7f860806b6c0 2024-03-06T07:26:18.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.849+0000 7f85fdffb640 1 --2- 172.21.15.83:0/3320541466 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f85ec06b4d0 0x7f85ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.850+0000 7f85fdffb640 1 -- 172.21.15.83:0/3320541466 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f860408a0a0 con 0x7f860806b6c0 2024-03-06T07:26:18.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.850+0000 7f860cc23640 1 --2- 172.21.15.83:0/3320541466 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f85ec06b4d0 0x7f85ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.850+0000 7f860cc23640 1 --2- 172.21.15.83:0/3320541466 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f85ec06b4d0 0x7f85ec06d990 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f85f8036000 tx=0x7f85f8030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:18.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:18.854+0000 7f85fdffb640 1 -- 172.21.15.83:0/3320541466 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f860401f020 con 0x7f860806b6c0 2024-03-06T07:26:19.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.062+0000 7f860f688640 1 -- 172.21.15.83:0/3320541466 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f86080b32d0 con 0x7f860806b6c0 2024-03-06T07:26:19.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.710+0000 7f85fdffb640 1 -- 172.21.15.83:0/3320541466 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 14 and data pool 15 v43) v1 ==== 174+0+0 (secure 0 0 0) 0x7f86040172c0 con 0x7f860806b6c0 2024-03-06T07:26:19.712 INFO:teuthology.orchestra.run.smithi083.stderr:new fs with metadata pool 14 and data pool 15 2024-03-06T07:26:19.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.716+0000 7f860f688640 1 -- 172.21.15.83:0/3320541466 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f85ec06b4d0 msgr2=0x7f85ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.716+0000 7f860f688640 1 --2- 172.21.15.83:0/3320541466 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f85ec06b4d0 0x7f85ec06d990 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f85f8036000 tx=0x7f85f8030000 comp rx=0 tx=0).stop 2024-03-06T07:26:19.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.716+0000 7f860f688640 1 -- 172.21.15.83:0/3320541466 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f860806b6c0 msgr2=0x7f860815b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.716+0000 7f860f688640 1 --2- 172.21.15.83:0/3320541466 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f860806b6c0 0x7f860815b2c0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f860400b3f0 tx=0x7f860400b8b0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.717+0000 7f860f688640 1 -- 172.21.15.83:0/3320541466 shutdown_connections 2024-03-06T07:26:19.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.717+0000 7f860f688640 1 --2- 172.21.15.83:0/3320541466 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f85ec06b4d0 0x7f85ec06d990 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.717+0000 7f860f688640 1 --2- 172.21.15.83:0/3320541466 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8608145470 0x7f860815e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.717+0000 7f860f688640 1 --2- 172.21.15.83:0/3320541466 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f860806fd80 0x7f8608140ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.717+0000 7f860f688640 1 --2- 172.21.15.83:0/3320541466 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f860806b6c0 0x7f860815b2c0 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.717+0000 7f860f688640 1 -- 172.21.15.83:0/3320541466 >> 172.21.15.83:0/3320541466 conn(0x7f860806c970 msgr2=0x7f86080b1ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:19.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.720+0000 7f860f688640 1 -- 172.21.15.83:0/3320541466 shutdown_connections 2024-03-06T07:26:19.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.721+0000 7f860f688640 1 -- 172.21.15.83:0/3320541466 wait complete. 2024-03-06T07:26:19.746 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:19.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.846+0000 7ffa8dcc3640 1 Processor -- start 2024-03-06T07:26:19.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.846+0000 7ffa8dcc3640 1 -- start start 2024-03-06T07:26:19.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.846+0000 7ffa8dcc3640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffa881424f0 0x7ffa880cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.846+0000 7ffa8dcc3640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ffa8806da40 0x7ffa8815cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.846+0000 7ffa8dcc3640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa880cc980 con 0x7ffa880cd7a0 2024-03-06T07:26:19.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.846+0000 7ffa8dcc3640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa880cd220 con 0x7ffa8806da40 2024-03-06T07:26:19.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.846+0000 7ffa8dcc3640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa880ccc60 con 0x7ffa881424f0 2024-03-06T07:26:19.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.847+0000 7ffa87fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ffa8806da40 0x7ffa8815cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.847+0000 7ffa87fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ffa8806da40 0x7ffa8815cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50374/0 (socket says 172.21.15.83:50374) 2024-03-06T07:26:19.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.847+0000 7ffa87fff640 1 -- 172.21.15.83:0/3064650158 learned_addr learned my addr 172.21.15.83:0/3064650158 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:19.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.847+0000 7ffa86ffd640 1 --1- 172.21.15.83:0/3064650158 >> v1:172.21.15.83:6789/0 conn(0x7ffa880cd7a0 0x7ffa8806d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:35396/0 (socket says 172.21.15.83:35396) 2024-03-06T07:26:19.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.847+0000 7ffa87fff640 1 -- 172.21.15.83:0/3064650158 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffa881424f0 msgr2=0x7ffa880cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:19.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.847+0000 7ffa87fff640 1 --2- 172.21.15.83:0/3064650158 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffa881424f0 0x7ffa880cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.847+0000 7ffa87fff640 1 -- 172.21.15.83:0/3064650158 >> v1:172.21.15.83:6789/0 conn(0x7ffa880cd7a0 legacy=0x7ffa8806d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.847+0000 7ffa867fc640 1 -- 172.21.15.83:0/3064650158 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4006574849 0 0) 0x7ffa880cc980 con 0x7ffa880cd7a0 2024-03-06T07:26:19.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.847+0000 7ffa87fff640 1 -- 172.21.15.83:0/3064650158 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa880ccf40 con 0x7ffa8806da40 2024-03-06T07:26:19.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.848+0000 7ffa87fff640 1 --2- 172.21.15.83:0/3064650158 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa8806da40 0x7ffa8815cef0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7ffa80009e50 tx=0x7ffa8002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fc3f5fcf586faf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:19.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.848+0000 7ffa867fc640 1 -- 172.21.15.83:0/3064650158 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa8002cb00 con 0x7ffa8806da40 2024-03-06T07:26:19.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.848+0000 7ffa867fc640 1 -- 172.21.15.83:0/3064650158 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa8002cca0 con 0x7ffa8806da40 2024-03-06T07:26:19.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.848+0000 7ffa867fc640 1 -- 172.21.15.83:0/3064650158 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa80034610 con 0x7ffa8806da40 2024-03-06T07:26:19.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.848+0000 7ffa8dcc3640 1 -- 172.21.15.83:0/3064650158 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa8806da40 msgr2=0x7ffa8815cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.848+0000 7ffa8dcc3640 1 --2- 172.21.15.83:0/3064650158 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa8806da40 0x7ffa8815cef0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7ffa80009e50 tx=0x7ffa8002bf60 comp rx=0 tx=0).stop 2024-03-06T07:26:19.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.848+0000 7ffa8dcc3640 1 -- 172.21.15.83:0/3064650158 shutdown_connections 2024-03-06T07:26:19.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.848+0000 7ffa8dcc3640 1 --2- 172.21.15.83:0/3064650158 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa8806da40 0x7ffa8815cef0 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.848+0000 7ffa8dcc3640 1 --2- 172.21.15.83:0/3064650158 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffa881424f0 0x7ffa880cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.848+0000 7ffa8dcc3640 1 -- 172.21.15.83:0/3064650158 >> 172.21.15.83:0/3064650158 conn(0x7ffa880cf8a0 msgr2=0x7ffa880cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:19.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.848+0000 7ffa8dcc3640 1 -- 172.21.15.83:0/3064650158 shutdown_connections 2024-03-06T07:26:19.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.848+0000 7ffa8dcc3640 1 -- 172.21.15.83:0/3064650158 wait complete. 2024-03-06T07:26:19.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.849+0000 7ffa8dcc3640 1 Processor -- start 2024-03-06T07:26:19.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.850+0000 7ffa8dcc3640 1 -- start start 2024-03-06T07:26:19.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.850+0000 7ffa8dcc3640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffa8806da40 0x7ffa8813e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.850+0000 7ffa8dcc3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa880cd7a0 0x7ffa88145310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.850+0000 7ffa8dcc3640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffa881424f0 0x7ffa880aafa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.850+0000 7ffa8dcc3640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa88071ce0 con 0x7ffa880cd7a0 2024-03-06T07:26:19.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.850+0000 7ffa8dcc3640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa88071b60 con 0x7ffa881424f0 2024-03-06T07:26:19.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.850+0000 7ffa8dcc3640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa88071e60 con 0x7ffa8806da40 2024-03-06T07:26:19.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.850+0000 7ffa86ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa880cd7a0 0x7ffa88145310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.850+0000 7ffa86ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa880cd7a0 0x7ffa88145310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50376/0 (socket says 172.21.15.83:50376) 2024-03-06T07:26:19.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.850+0000 7ffa86ffd640 1 -- 172.21.15.83:0/1586970975 learned_addr learned my addr 172.21.15.83:0/1586970975 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:19.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.851+0000 7ffa87fff640 1 --2- 172.21.15.83:0/1586970975 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffa881424f0 0x7ffa880aafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.851+0000 7ffa877fe640 1 --2- 172.21.15.83:0/1586970975 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffa8806da40 0x7ffa8813e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.851+0000 7ffa86ffd640 1 -- 172.21.15.83:0/1586970975 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffa8806da40 msgr2=0x7ffa8813e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.851+0000 7ffa86ffd640 1 --2- 172.21.15.83:0/1586970975 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffa8806da40 0x7ffa8813e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.851+0000 7ffa86ffd640 1 -- 172.21.15.83:0/1586970975 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffa881424f0 msgr2=0x7ffa880aafa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.851+0000 7ffa86ffd640 1 --2- 172.21.15.83:0/1586970975 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffa881424f0 0x7ffa880aafa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.851+0000 7ffa86ffd640 1 -- 172.21.15.83:0/1586970975 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa880a94f0 con 0x7ffa880cd7a0 2024-03-06T07:26:19.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.851+0000 7ffa87fff640 1 --2- 172.21.15.83:0/1586970975 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffa881424f0 0x7ffa880aafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:19.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.851+0000 7ffa86ffd640 1 --2- 172.21.15.83:0/1586970975 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa880cd7a0 0x7ffa88145310 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c00c4a0 tx=0x7ffa7c00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:19.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.851+0000 7ffa6ffff640 1 -- 172.21.15.83:0/1586970975 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa7c018070 con 0x7ffa880cd7a0 2024-03-06T07:26:19.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.851+0000 7ffa6ffff640 1 -- 172.21.15.83:0/1586970975 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa7c0074e0 con 0x7ffa880cd7a0 2024-03-06T07:26:19.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.851+0000 7ffa8dcc3640 1 -- 172.21.15.83:0/1586970975 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa880a97d0 con 0x7ffa880cd7a0 2024-03-06T07:26:19.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.851+0000 7ffa6ffff640 1 -- 172.21.15.83:0/1586970975 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa7c004a20 con 0x7ffa880cd7a0 2024-03-06T07:26:19.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.851+0000 7ffa8dcc3640 1 -- 172.21.15.83:0/1586970975 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa880a9d90 con 0x7ffa880cd7a0 2024-03-06T07:26:19.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.852+0000 7ffa8dcc3640 1 -- 172.21.15.83:0/1586970975 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa8804f330 con 0x7ffa880cd7a0 2024-03-06T07:26:19.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.853+0000 7ffa6ffff640 1 -- 172.21.15.83:0/1586970975 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffa7c01b070 con 0x7ffa880cd7a0 2024-03-06T07:26:19.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.853+0000 7ffa6ffff640 1 --2- 172.21.15.83:0/1586970975 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffa7406b4d0 0x7ffa7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.853+0000 7ffa6ffff640 1 -- 172.21.15.83:0/1586970975 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffa7c0894e0 con 0x7ffa880cd7a0 2024-03-06T07:26:19.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.853+0000 7ffa877fe640 1 --2- 172.21.15.83:0/1586970975 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffa7406b4d0 0x7ffa7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.854+0000 7ffa877fe640 1 --2- 172.21.15.83:0/1586970975 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffa7406b4d0 0x7ffa7406d990 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7ffa7800a770 tx=0x7ffa78008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:19.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:19.857+0000 7ffa6ffff640 1 -- 172.21.15.83:0/1586970975 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffa7c0583f0 con 0x7ffa880cd7a0 2024-03-06T07:26:20.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.029+0000 7ffa8dcc3640 1 -- 172.21.15.83:0/1586970975 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa8806a120 con 0x7ffa880cd7a0 2024-03-06T07:26:20.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.030+0000 7ffa6ffff640 1 -- 172.21.15.83:0/1586970975 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+16633 (secure 0 0 0) 0x7ffa7c057cf0 con 0x7ffa880cd7a0 2024-03-06T07:26:20.032 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:20.032 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":93,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:26:19.064032+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:17.275618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:18.028872+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:20.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.032+0000 7ffa8dcc3640 1 -- 172.21.15.83:0/1586970975 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffa7406b4d0 msgr2=0x7ffa7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.032+0000 7ffa8dcc3640 1 --2- 172.21.15.83:0/1586970975 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffa7406b4d0 0x7ffa7406d990 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7ffa7800a770 tx=0x7ffa78008040 comp rx=0 tx=0).stop 2024-03-06T07:26:20.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.032+0000 7ffa8dcc3640 1 -- 172.21.15.83:0/1586970975 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa880cd7a0 msgr2=0x7ffa88145310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.032+0000 7ffa8dcc3640 1 --2- 172.21.15.83:0/1586970975 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa880cd7a0 0x7ffa88145310 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c00c4a0 tx=0x7ffa7c00c960 comp rx=0 tx=0).stop 2024-03-06T07:26:20.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.032+0000 7ffa8dcc3640 1 -- 172.21.15.83:0/1586970975 shutdown_connections 2024-03-06T07:26:20.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.032+0000 7ffa8dcc3640 1 --2- 172.21.15.83:0/1586970975 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffa7406b4d0 0x7ffa7406d990 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.032+0000 7ffa8dcc3640 1 --2- 172.21.15.83:0/1586970975 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffa881424f0 0x7ffa880aafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.032+0000 7ffa8dcc3640 1 --2- 172.21.15.83:0/1586970975 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa880cd7a0 0x7ffa88145310 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.032+0000 7ffa8dcc3640 1 --2- 172.21.15.83:0/1586970975 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffa8806da40 0x7ffa8813e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.032+0000 7ffa8dcc3640 1 -- 172.21.15.83:0/1586970975 >> 172.21.15.83:0/1586970975 conn(0x7ffa880cf8a0 msgr2=0x7ffa8815cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:20.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.032+0000 7ffa8dcc3640 1 -- 172.21.15.83:0/1586970975 shutdown_connections 2024-03-06T07:26:20.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.032+0000 7ffa8dcc3640 1 -- 172.21.15.83:0/1586970975 wait complete. 2024-03-06T07:26:20.045 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:20.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.144+0000 7f05a13ac640 1 Processor -- start 2024-03-06T07:26:20.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.144+0000 7f05a13ac640 1 -- start start 2024-03-06T07:26:20.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.144+0000 7f05a13ac640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f059c0d4920 0x7f059c0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.144+0000 7f05a13ac640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f059c1625a0 0x7f059c15dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.144+0000 7f05a13ac640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f059c0ceac0 con 0x7f059c1621c0 2024-03-06T07:26:20.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.144+0000 7f05a13ac640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f059c0cf690 con 0x7f059c0d4920 2024-03-06T07:26:20.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.144+0000 7f05a13ac640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f059c0d2dc0 con 0x7f059c1625a0 2024-03-06T07:26:20.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.145+0000 7f0593fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f059c0d4920 0x7f059c0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.145+0000 7f0593fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f059c0d4920 0x7f059c0cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50384/0 (socket says 172.21.15.83:50384) 2024-03-06T07:26:20.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.145+0000 7f0593fff640 1 -- 172.21.15.83:0/3514502510 learned_addr learned my addr 172.21.15.83:0/3514502510 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:20.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.145+0000 7f0593fff640 1 -- 172.21.15.83:0/3514502510 >> v2:172.21.15.183:3300/0 conn(0x7f059c1625a0 msgr2=0x7f059c15dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:20.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.145+0000 7f0593fff640 1 --2- 172.21.15.83:0/3514502510 >> v2:172.21.15.183:3300/0 conn(0x7f059c1625a0 0x7f059c15dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.145+0000 7f0593fff640 1 -- 172.21.15.83:0/3514502510 >> v1:172.21.15.183:6789/0 conn(0x7f059c1621c0 legacy=0x7f059c0d26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:20.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.145+0000 7f0593fff640 1 -- 172.21.15.83:0/3514502510 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f059c0ced60 con 0x7f059c0d4920 2024-03-06T07:26:20.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.146+0000 7f0593fff640 1 --2- 172.21.15.83:0/3514502510 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f059c0d4920 0x7f059c0cfb20 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f059c0d3be0 tx=0x7f058402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a1f398207748206 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.146+0000 7f059a7fc640 1 -- 172.21.15.83:0/3514502510 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058402e070 con 0x7f059c0d4920 2024-03-06T07:26:20.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.146+0000 7f059a7fc640 1 -- 172.21.15.83:0/3514502510 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0584037070 con 0x7f059c0d4920 2024-03-06T07:26:20.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.146+0000 7f059a7fc640 1 -- 172.21.15.83:0/3514502510 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0584031a00 con 0x7f059c0d4920 2024-03-06T07:26:20.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.146+0000 7f05a13ac640 1 -- 172.21.15.83:0/3514502510 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f059c0d4920 msgr2=0x7f059c0cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.146+0000 7f05a13ac640 1 --2- 172.21.15.83:0/3514502510 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f059c0d4920 0x7f059c0cfb20 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f059c0d3be0 tx=0x7f058402b880 comp rx=0 tx=0).stop 2024-03-06T07:26:20.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.146+0000 7f05a13ac640 1 -- 172.21.15.83:0/3514502510 shutdown_connections 2024-03-06T07:26:20.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.146+0000 7f05a13ac640 1 --2- 172.21.15.83:0/3514502510 >> v2:172.21.15.183:3300/0 conn(0x7f059c1625a0 0x7f059c15dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.146+0000 7f05a13ac640 1 --2- 172.21.15.83:0/3514502510 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f059c0d4920 0x7f059c0cfb20 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.146+0000 7f05a13ac640 1 -- 172.21.15.83:0/3514502510 >> 172.21.15.83:0/3514502510 conn(0x7f059c0d3d10 msgr2=0x7f059c0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:20.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.146+0000 7f05a13ac640 1 -- 172.21.15.83:0/3514502510 shutdown_connections 2024-03-06T07:26:20.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.146+0000 7f05a13ac640 1 -- 172.21.15.83:0/3514502510 wait complete. 2024-03-06T07:26:20.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.148+0000 7f05a13ac640 1 Processor -- start 2024-03-06T07:26:20.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.148+0000 7f05a13ac640 1 -- start start 2024-03-06T07:26:20.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.148+0000 7f05a13ac640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f059c0d4920 0x7f059c152060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.148+0000 7f05a13ac640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f059c1621c0 0x7f059c1e7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.148+0000 7f05a13ac640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f059c1625a0 0x7f059c1531d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.148+0000 7f05a13ac640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f059c15e280 con 0x7f059c1621c0 2024-03-06T07:26:20.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.148+0000 7f05a13ac640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f059c0cf320 con 0x7f059c1625a0 2024-03-06T07:26:20.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.148+0000 7f05a13ac640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f059c15e400 con 0x7f059c0d4920 2024-03-06T07:26:20.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.148+0000 7f0593fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f059c1621c0 0x7f059c1e7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.148+0000 7f0593fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f059c1621c0 0x7f059c1e7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50400/0 (socket says 172.21.15.83:50400) 2024-03-06T07:26:20.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.148+0000 7f0593fff640 1 -- 172.21.15.83:0/2388312018 learned_addr learned my addr 172.21.15.83:0/2388312018 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:20.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.148+0000 7f059b7fe640 1 --2- 172.21.15.83:0/2388312018 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f059c1625a0 0x7f059c1531d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.148+0000 7f059affd640 1 --2- 172.21.15.83:0/2388312018 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f059c0d4920 0x7f059c152060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.149+0000 7f0593fff640 1 -- 172.21.15.83:0/2388312018 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f059c0d4920 msgr2=0x7f059c152060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.149+0000 7f0593fff640 1 --2- 172.21.15.83:0/2388312018 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f059c0d4920 0x7f059c152060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.149+0000 7f0593fff640 1 -- 172.21.15.83:0/2388312018 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f059c1625a0 msgr2=0x7f059c1531d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.149+0000 7f0593fff640 1 --2- 172.21.15.83:0/2388312018 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f059c1625a0 0x7f059c1531d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.149+0000 7f0593fff640 1 -- 172.21.15.83:0/2388312018 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f059c1529f0 con 0x7f059c1621c0 2024-03-06T07:26:20.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.149+0000 7f059affd640 1 --2- 172.21.15.83:0/2388312018 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f059c0d4920 0x7f059c152060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:20.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.149+0000 7f0593fff640 1 --2- 172.21.15.83:0/2388312018 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f059c1621c0 0x7f059c1e7790 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f058402b8c0 tx=0x7f05840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.149+0000 7f05937fe640 1 -- 172.21.15.83:0/2388312018 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0584032880 con 0x7f059c1621c0 2024-03-06T07:26:20.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.149+0000 7f05937fe640 1 -- 172.21.15.83:0/2388312018 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0584032a20 con 0x7f059c1621c0 2024-03-06T07:26:20.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.149+0000 7f05a13ac640 1 -- 172.21.15.83:0/2388312018 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f059c1e8210 con 0x7f059c1621c0 2024-03-06T07:26:20.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.149+0000 7f05937fe640 1 -- 172.21.15.83:0/2388312018 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0584032e10 con 0x7f059c1621c0 2024-03-06T07:26:20.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.149+0000 7f05a13ac640 1 -- 172.21.15.83:0/2388312018 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f059c152cb0 con 0x7f059c1621c0 2024-03-06T07:26:20.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.150+0000 7f05a13ac640 1 -- 172.21.15.83:0/2388312018 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f059c1e8730 con 0x7f059c1621c0 2024-03-06T07:26:20.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.150+0000 7f05937fe640 1 -- 172.21.15.83:0/2388312018 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f058404c060 con 0x7f059c1621c0 2024-03-06T07:26:20.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.151+0000 7f05937fe640 1 --2- 172.21.15.83:0/2388312018 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f057c06b4d0 0x7f057c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.151+0000 7f05937fe640 1 -- 172.21.15.83:0/2388312018 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f05840abcb0 con 0x7f059c1621c0 2024-03-06T07:26:20.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.151+0000 7f059affd640 1 --2- 172.21.15.83:0/2388312018 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f057c06b4d0 0x7f057c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.151+0000 7f059affd640 1 --2- 172.21.15.83:0/2388312018 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f057c06b4d0 0x7f057c06d990 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f058c001750 tx=0x7f058c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.155+0000 7f05937fe640 1 -- 172.21.15.83:0/2388312018 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0584040070 con 0x7f059c1621c0 2024-03-06T07:26:20.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.333+0000 7f05a13ac640 1 -- 172.21.15.83:0/2388312018 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f059c04f330 con 0x7f059c1621c0 2024-03-06T07:26:20.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.334+0000 7f05937fe640 1 -- 172.21.15.83:0/2388312018 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+16633 (secure 0 0 0) 0x7f058407ab30 con 0x7f059c1621c0 2024-03-06T07:26:20.336 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:20.336 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":93,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:26:19.064032+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:17.275618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:18.028872+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:20.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.336+0000 7f05a13ac640 1 -- 172.21.15.83:0/2388312018 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f057c06b4d0 msgr2=0x7f057c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.336+0000 7f05a13ac640 1 --2- 172.21.15.83:0/2388312018 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f057c06b4d0 0x7f057c06d990 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f058c001750 tx=0x7f058c009040 comp rx=0 tx=0).stop 2024-03-06T07:26:20.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.336+0000 7f05a13ac640 1 -- 172.21.15.83:0/2388312018 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f059c1621c0 msgr2=0x7f059c1e7790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.336+0000 7f05a13ac640 1 --2- 172.21.15.83:0/2388312018 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f059c1621c0 0x7f059c1e7790 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f058402b8c0 tx=0x7f05840046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.336+0000 7f05a13ac640 1 -- 172.21.15.83:0/2388312018 shutdown_connections 2024-03-06T07:26:20.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.336+0000 7f05a13ac640 1 --2- 172.21.15.83:0/2388312018 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f057c06b4d0 0x7f057c06d990 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.336+0000 7f05a13ac640 1 --2- 172.21.15.83:0/2388312018 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f059c1625a0 0x7f059c1531d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.336+0000 7f05a13ac640 1 --2- 172.21.15.83:0/2388312018 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f059c1621c0 0x7f059c1e7790 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.336+0000 7f05a13ac640 1 --2- 172.21.15.83:0/2388312018 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f059c0d4920 0x7f059c152060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.336+0000 7f05a13ac640 1 -- 172.21.15.83:0/2388312018 >> 172.21.15.83:0/2388312018 conn(0x7f059c0d3d10 msgr2=0x7f059c15e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:20.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.336+0000 7f05a13ac640 1 -- 172.21.15.83:0/2388312018 shutdown_connections 2024-03-06T07:26:20.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.336+0000 7f05a13ac640 1 -- 172.21.15.83:0/2388312018 wait complete. 2024-03-06T07:26:20.349 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:26:20.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.450+0000 7f1b16716640 1 Processor -- start 2024-03-06T07:26:20.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.450+0000 7f1b16716640 1 -- start start 2024-03-06T07:26:20.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.451+0000 7f1b16716640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1b1015f4f0 0x7f1b10066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.451+0000 7f1b16716640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f1b100b0db0 0x7f1b100ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.451+0000 7f1b16716640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b1006b050 con 0x7f1b100ce580 2024-03-06T07:26:20.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.451+0000 7f1b16716640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b1006b2c0 con 0x7f1b1015f4f0 2024-03-06T07:26:20.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.451+0000 7f1b16716640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b100b0890 con 0x7f1b100b0db0 2024-03-06T07:26:20.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.451+0000 7f1b0ffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1b1015f4f0 0x7f1b10066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.451+0000 7f1b14cb3640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f1b100ce580 0x7f1b10072550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:35398/0 (socket says 172.21.15.83:35398) 2024-03-06T07:26:20.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.451+0000 7f1b14cb3640 1 -- 172.21.15.83:0/3086977406 learned_addr learned my addr 172.21.15.83:0/3086977406 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:20.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.451+0000 7f1b0f7fe640 1 --2- 172.21.15.83:0/3086977406 >> v2:172.21.15.183:3300/0 conn(0x7f1b100b0db0 0x7f1b100ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.451+0000 7f1b0ffff640 1 -- 172.21.15.83:0/3086977406 >> v2:172.21.15.183:3300/0 conn(0x7f1b100b0db0 msgr2=0x7f1b100ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:20.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.451+0000 7f1b0ffff640 1 --2- 172.21.15.83:0/3086977406 >> v2:172.21.15.183:3300/0 conn(0x7f1b100b0db0 0x7f1b100ce040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.451+0000 7f1b0ffff640 1 -- 172.21.15.83:0/3086977406 >> v1:172.21.15.83:6789/0 conn(0x7f1b100ce580 legacy=0x7f1b10072550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.451+0000 7f1b0ffff640 1 -- 172.21.15.83:0/3086977406 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b100672e0 con 0x7f1b1015f4f0 2024-03-06T07:26:20.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.451+0000 7f1b0effd640 1 -- 172.21.15.83:0/3086977406 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 797381914 0 0) 0x7f1b1006b050 con 0x7f1b100ce580 2024-03-06T07:26:20.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.452+0000 7f1b0ffff640 1 --2- 172.21.15.83:0/3086977406 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b1015f4f0 0x7f1b10066d00 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f1b100d11d0 tx=0x7f1b0002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26f02775a59d33b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.452+0000 7f1b0effd640 1 -- 172.21.15.83:0/3086977406 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b00035020 con 0x7f1b1015f4f0 2024-03-06T07:26:20.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.452+0000 7f1b0effd640 1 -- 172.21.15.83:0/3086977406 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b00031830 con 0x7f1b1015f4f0 2024-03-06T07:26:20.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.452+0000 7f1b0effd640 1 -- 172.21.15.83:0/3086977406 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b00031c00 con 0x7f1b1015f4f0 2024-03-06T07:26:20.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.452+0000 7f1b16716640 1 -- 172.21.15.83:0/3086977406 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b1015f4f0 msgr2=0x7f1b10066d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.452+0000 7f1b16716640 1 --2- 172.21.15.83:0/3086977406 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b1015f4f0 0x7f1b10066d00 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f1b100d11d0 tx=0x7f1b0002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.453+0000 7f1b16716640 1 -- 172.21.15.83:0/3086977406 shutdown_connections 2024-03-06T07:26:20.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.453+0000 7f1b16716640 1 --2- 172.21.15.83:0/3086977406 >> v2:172.21.15.183:3300/0 conn(0x7f1b100b0db0 0x7f1b100ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.453+0000 7f1b16716640 1 --2- 172.21.15.83:0/3086977406 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b1015f4f0 0x7f1b10066d00 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.453+0000 7f1b16716640 1 -- 172.21.15.83:0/3086977406 >> 172.21.15.83:0/3086977406 conn(0x7f1b100b1d70 msgr2=0x7f1b100b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:20.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.453+0000 7f1b16716640 1 -- 172.21.15.83:0/3086977406 shutdown_connections 2024-03-06T07:26:20.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.453+0000 7f1b16716640 1 -- 172.21.15.83:0/3086977406 wait complete. 2024-03-06T07:26:20.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.454+0000 7f1b16716640 1 Processor -- start 2024-03-06T07:26:20.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.454+0000 7f1b16716640 1 -- start start 2024-03-06T07:26:20.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.455+0000 7f1b16716640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1b100b0db0 0x7f1b100ae6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.455+0000 7f1b16716640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b100ce580 0x7f1b100cbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.455+0000 7f1b16716640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1b1015f4f0 0x7f1b100cb090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.455+0000 7f1b16716640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b1015f8d0 con 0x7f1b100ce580 2024-03-06T07:26:20.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.455+0000 7f1b16716640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b1006b5a0 con 0x7f1b100b0db0 2024-03-06T07:26:20.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.455+0000 7f1b16716640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b100ae210 con 0x7f1b1015f4f0 2024-03-06T07:26:20.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.455+0000 7f1b0f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b100ce580 0x7f1b100cbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.455+0000 7f1b0f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b100ce580 0x7f1b100cbcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50416/0 (socket says 172.21.15.83:50416) 2024-03-06T07:26:20.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.455+0000 7f1b0f7fe640 1 -- 172.21.15.83:0/2758396857 learned_addr learned my addr 172.21.15.83:0/2758396857 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:20.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.455+0000 7f1b0ffff640 1 --2- 172.21.15.83:0/2758396857 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1b100b0db0 0x7f1b100ae6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.455+0000 7f1b0f7fe640 1 -- 172.21.15.83:0/2758396857 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1b1015f4f0 msgr2=0x7f1b100cb090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.455+0000 7f1b0f7fe640 1 --2- 172.21.15.83:0/2758396857 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1b1015f4f0 0x7f1b100cb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.455+0000 7f1b0f7fe640 1 -- 172.21.15.83:0/2758396857 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1b100b0db0 msgr2=0x7f1b100ae6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.455+0000 7f1b0f7fe640 1 --2- 172.21.15.83:0/2758396857 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1b100b0db0 0x7f1b100ae6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.455+0000 7f1b0f7fe640 1 -- 172.21.15.83:0/2758396857 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b100ad600 con 0x7f1b100ce580 2024-03-06T07:26:20.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.455+0000 7f1b0ffff640 1 --2- 172.21.15.83:0/2758396857 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1b100b0db0 0x7f1b100ae6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:20.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.455+0000 7f1b0f7fe640 1 --2- 172.21.15.83:0/2758396857 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b100ce580 0x7f1b100cbcc0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f1b0400d800 tx=0x7f1b0400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.455+0000 7f1b0cff9640 1 -- 172.21.15.83:0/2758396857 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b0400b6c0 con 0x7f1b100ce580 2024-03-06T07:26:20.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.455+0000 7f1b0cff9640 1 -- 172.21.15.83:0/2758396857 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b04019070 con 0x7f1b100ce580 2024-03-06T07:26:20.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.455+0000 7f1b16716640 1 -- 172.21.15.83:0/2758396857 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b100ad8e0 con 0x7f1b100ce580 2024-03-06T07:26:20.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.455+0000 7f1b0cff9640 1 -- 172.21.15.83:0/2758396857 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b0400ba80 con 0x7f1b100ce580 2024-03-06T07:26:20.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.455+0000 7f1b16716640 1 -- 172.21.15.83:0/2758396857 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b100adea0 con 0x7f1b100ce580 2024-03-06T07:26:20.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.456+0000 7f1b16716640 1 -- 172.21.15.83:0/2758396857 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b1009e3d0 con 0x7f1b100ce580 2024-03-06T07:26:20.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.457+0000 7f1b0cff9640 1 -- 172.21.15.83:0/2758396857 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1b0400bc20 con 0x7f1b100ce580 2024-03-06T07:26:20.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.457+0000 7f1b0cff9640 1 --2- 172.21.15.83:0/2758396857 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1afc06b500 0x7f1afc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.457+0000 7f1b0cff9640 1 -- 172.21.15.83:0/2758396857 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1b0408a6f0 con 0x7f1b100ce580 2024-03-06T07:26:20.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.457+0000 7f1b0ffff640 1 --2- 172.21.15.83:0/2758396857 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1afc06b500 0x7f1afc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.457+0000 7f1b0ffff640 1 --2- 172.21.15.83:0/2758396857 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1afc06b500 0x7f1afc06d9c0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f1b00036000 tx=0x7f1b00008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.461+0000 7f1b0cff9640 1 -- 172.21.15.83:0/2758396857 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1b0401c360 con 0x7f1b100ce580 2024-03-06T07:26:20.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.669+0000 7f1b16716640 1 -- 172.21.15.83:0/2758396857 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f1b10153d10 con 0x7f1b100ce580 2024-03-06T07:26:20.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.724+0000 7f1b0cff9640 1 -- 172.21.15.83:0/2758396857 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v44) v1 ==== 118+0+0 (secure 0 0 0) 0x7f1b040594e0 con 0x7f1b100ce580 2024-03-06T07:26:20.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.727+0000 7f1b16716640 1 -- 172.21.15.83:0/2758396857 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1afc06b500 msgr2=0x7f1afc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.727+0000 7f1b16716640 1 --2- 172.21.15.83:0/2758396857 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1afc06b500 0x7f1afc06d9c0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f1b00036000 tx=0x7f1b00008030 comp rx=0 tx=0).stop 2024-03-06T07:26:20.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.727+0000 7f1b16716640 1 -- 172.21.15.83:0/2758396857 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b100ce580 msgr2=0x7f1b100cbcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.727+0000 7f1b16716640 1 --2- 172.21.15.83:0/2758396857 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b100ce580 0x7f1b100cbcc0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f1b0400d800 tx=0x7f1b0400dcc0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.727+0000 7f1b16716640 1 -- 172.21.15.83:0/2758396857 shutdown_connections 2024-03-06T07:26:20.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.727+0000 7f1b16716640 1 --2- 172.21.15.83:0/2758396857 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1afc06b500 0x7f1afc06d9c0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.727+0000 7f1b16716640 1 --2- 172.21.15.83:0/2758396857 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1b1015f4f0 0x7f1b100cb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.727+0000 7f1b16716640 1 --2- 172.21.15.83:0/2758396857 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b100ce580 0x7f1b100cbcc0 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.727+0000 7f1b16716640 1 --2- 172.21.15.83:0/2758396857 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1b100b0db0 0x7f1b100ae6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.727+0000 7f1b16716640 1 -- 172.21.15.83:0/2758396857 >> 172.21.15.83:0/2758396857 conn(0x7f1b100b1d70 msgr2=0x7f1b100afbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:20.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.727+0000 7f1b16716640 1 -- 172.21.15.83:0/2758396857 shutdown_connections 2024-03-06T07:26:20.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.727+0000 7f1b16716640 1 -- 172.21.15.83:0/2758396857 wait complete. 2024-03-06T07:26:20.741 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:26:20.742 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:26:20.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.843+0000 7ff9b945c640 1 Processor -- start 2024-03-06T07:26:20.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.843+0000 7ff9b945c640 1 -- start start 2024-03-06T07:26:20.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.844+0000 7ff9b945c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff9b41614d0 0x7ff9b40d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.844+0000 7ff9b945c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9b40d24b0 0x7ff9b40b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.844+0000 7ff9b945c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9b41618b0 con 0x7ff9b40d1910 2024-03-06T07:26:20.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.844+0000 7ff9b945c640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9b40cf400 con 0x7ff9b41614d0 2024-03-06T07:26:20.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.844+0000 7ff9b945c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b40cf6a0 con 0x7ff9b40d24b0 2024-03-06T07:26:20.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.844+0000 7ff9b2ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff9b41614d0 0x7ff9b40d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.844+0000 7ff9aa7fc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff9b40d1910 0x7ff9b40d1da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:35400/0 (socket says 172.21.15.83:35400) 2024-03-06T07:26:20.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.844+0000 7ff9b2ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff9b41614d0 0x7ff9b40d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50430/0 (socket says 172.21.15.83:50430) 2024-03-06T07:26:20.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.844+0000 7ff9aa7fc640 1 -- 172.21.15.83:0/515693793 learned_addr learned my addr 172.21.15.83:0/515693793 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:20.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.844+0000 7ff9b37fe640 1 --2- 172.21.15.83:0/515693793 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9b40d24b0 0x7ff9b40b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.844+0000 7ff9b2ffd640 1 -- 172.21.15.83:0/515693793 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9b40d24b0 msgr2=0x7ff9b40b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:20.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.844+0000 7ff9b2ffd640 1 --2- 172.21.15.83:0/515693793 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9b40d24b0 0x7ff9b40b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.844+0000 7ff9b2ffd640 1 -- 172.21.15.83:0/515693793 >> v1:172.21.15.83:6789/0 conn(0x7ff9b40d1910 legacy=0x7ff9b40d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.844+0000 7ff9b2ffd640 1 -- 172.21.15.83:0/515693793 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b40cf980 con 0x7ff9b41614d0 2024-03-06T07:26:20.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.844+0000 7ff9b27fc640 1 -- 172.21.15.83:0/515693793 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1312117908 0 0) 0x7ff9b41618b0 con 0x7ff9b40d1910 2024-03-06T07:26:20.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.844+0000 7ff9b37fe640 1 --2- 172.21.15.83:0/515693793 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9b40d24b0 0x7ff9b40b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:20.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.845+0000 7ff9b2ffd640 1 --2- 172.21.15.83:0/515693793 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9b41614d0 0x7ff9b40d4860 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7ff9a40097a0 tx=0x7ff9a402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=570be111bd9ca517 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.845+0000 7ff9b27fc640 1 -- 172.21.15.83:0/515693793 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a4035020 con 0x7ff9b41614d0 2024-03-06T07:26:20.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.845+0000 7ff9b27fc640 1 -- 172.21.15.83:0/515693793 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9a4031830 con 0x7ff9b41614d0 2024-03-06T07:26:20.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.845+0000 7ff9b27fc640 1 -- 172.21.15.83:0/515693793 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a4031c00 con 0x7ff9b41614d0 2024-03-06T07:26:20.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.846+0000 7ff9b945c640 1 -- 172.21.15.83:0/515693793 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9b41614d0 msgr2=0x7ff9b40d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.846+0000 7ff9b945c640 1 --2- 172.21.15.83:0/515693793 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9b41614d0 0x7ff9b40d4860 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7ff9a40097a0 tx=0x7ff9a402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.846+0000 7ff9b945c640 1 -- 172.21.15.83:0/515693793 shutdown_connections 2024-03-06T07:26:20.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.846+0000 7ff9b945c640 1 --2- 172.21.15.83:0/515693793 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9b40d24b0 0x7ff9b40b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.846+0000 7ff9b945c640 1 --2- 172.21.15.83:0/515693793 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9b41614d0 0x7ff9b40d4860 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.846+0000 7ff9b945c640 1 -- 172.21.15.83:0/515693793 >> 172.21.15.83:0/515693793 conn(0x7ff9b40d3d40 msgr2=0x7ff9b40d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:20.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.846+0000 7ff9b945c640 1 -- 172.21.15.83:0/515693793 shutdown_connections 2024-03-06T07:26:20.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.846+0000 7ff9b945c640 1 -- 172.21.15.83:0/515693793 wait complete. 2024-03-06T07:26:20.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.847+0000 7ff9b945c640 1 Processor -- start 2024-03-06T07:26:20.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.847+0000 7ff9b945c640 1 -- start start 2024-03-06T07:26:20.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.848+0000 7ff9b945c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9b40d1910 0x7ff9b40cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.848+0000 7ff9b945c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff9b40d24b0 0x7ff9b4158440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.848+0000 7ff9b945c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9b41614d0 0x7ff9b40b4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.848+0000 7ff9b945c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b40b1370 con 0x7ff9b41614d0 2024-03-06T07:26:20.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.848+0000 7ff9b945c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b40b0910 con 0x7ff9b40d24b0 2024-03-06T07:26:20.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.848+0000 7ff9b945c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b40b14f0 con 0x7ff9b40d1910 2024-03-06T07:26:20.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.848+0000 7ff9b37fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9b41614d0 0x7ff9b40b4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.848+0000 7ff9b37fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9b41614d0 0x7ff9b40b4120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50432/0 (socket says 172.21.15.83:50432) 2024-03-06T07:26:20.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.848+0000 7ff9b37fe640 1 -- 172.21.15.83:0/1746539593 learned_addr learned my addr 172.21.15.83:0/1746539593 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:20.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.848+0000 7ff9b2ffd640 1 --2- 172.21.15.83:0/1746539593 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9b40d1910 0x7ff9b40cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.848+0000 7ff9aa7fc640 1 --2- 172.21.15.83:0/1746539593 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff9b40d24b0 0x7ff9b4158440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.848+0000 7ff9b37fe640 1 -- 172.21.15.83:0/1746539593 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9b40d1910 msgr2=0x7ff9b40cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.848+0000 7ff9b37fe640 1 --2- 172.21.15.83:0/1746539593 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9b40d1910 0x7ff9b40cd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.848+0000 7ff9b37fe640 1 -- 172.21.15.83:0/1746539593 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff9b40d24b0 msgr2=0x7ff9b4158440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.848+0000 7ff9b37fe640 1 --2- 172.21.15.83:0/1746539593 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff9b40d24b0 0x7ff9b4158440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.849+0000 7ff9b37fe640 1 -- 172.21.15.83:0/1746539593 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b41f7e30 con 0x7ff9b41614d0 2024-03-06T07:26:20.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.849+0000 7ff9b37fe640 1 --2- 172.21.15.83:0/1746539593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9b41614d0 0x7ff9b40b4120 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac00cef0 tx=0x7ff9ac00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.849+0000 7ff9abfff640 1 -- 172.21.15.83:0/1746539593 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9ac01a070 con 0x7ff9b41614d0 2024-03-06T07:26:20.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.849+0000 7ff9abfff640 1 -- 172.21.15.83:0/1746539593 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9ac00d860 con 0x7ff9b41614d0 2024-03-06T07:26:20.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.849+0000 7ff9b945c640 1 -- 172.21.15.83:0/1746539593 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9b40ccc00 con 0x7ff9b41614d0 2024-03-06T07:26:20.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.849+0000 7ff9abfff640 1 -- 172.21.15.83:0/1746539593 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9ac00dc50 con 0x7ff9b41614d0 2024-03-06T07:26:20.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.849+0000 7ff9b945c640 1 -- 172.21.15.83:0/1746539593 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9b40cdc20 con 0x7ff9b41614d0 2024-03-06T07:26:20.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.850+0000 7ff9b945c640 1 -- 172.21.15.83:0/1746539593 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9b4068ff0 con 0x7ff9b41614d0 2024-03-06T07:26:20.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.850+0000 7ff9abfff640 1 -- 172.21.15.83:0/1746539593 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff9ac01d070 con 0x7ff9b41614d0 2024-03-06T07:26:20.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.851+0000 7ff9abfff640 1 --2- 172.21.15.83:0/1746539593 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff99406b4d0 0x7ff99406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.851+0000 7ff9abfff640 1 -- 172.21.15.83:0/1746539593 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff9ac08ab10 con 0x7ff9b41614d0 2024-03-06T07:26:20.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.851+0000 7ff9b2ffd640 1 --2- 172.21.15.83:0/1746539593 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff99406b4d0 0x7ff99406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.851+0000 7ff9b2ffd640 1 --2- 172.21.15.83:0/1746539593 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff99406b4d0 0x7ff99406d990 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4036000 tx=0x7ff9a403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:20.855+0000 7ff9abfff640 1 -- 172.21.15.83:0/1746539593 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff9ac0599c0 con 0x7ff9b41614d0 2024-03-06T07:26:21.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.070+0000 7ff9b945c640 1 -- 172.21.15.83:0/1746539593 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7ff9b4160c00 con 0x7ff9b41614d0 2024-03-06T07:26:21.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.730+0000 7ff9abfff640 1 -- 172.21.15.83:0/1746539593 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v45) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff9ac0592c0 con 0x7ff9b41614d0 2024-03-06T07:26:21.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.733+0000 7ff9b945c640 1 -- 172.21.15.83:0/1746539593 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff99406b4d0 msgr2=0x7ff99406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.733+0000 7ff9b945c640 1 --2- 172.21.15.83:0/1746539593 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff99406b4d0 0x7ff99406d990 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4036000 tx=0x7ff9a403c040 comp rx=0 tx=0).stop 2024-03-06T07:26:21.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.733+0000 7ff9b945c640 1 -- 172.21.15.83:0/1746539593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9b41614d0 msgr2=0x7ff9b40b4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.733+0000 7ff9b945c640 1 --2- 172.21.15.83:0/1746539593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9b41614d0 0x7ff9b40b4120 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac00cef0 tx=0x7ff9ac00a710 comp rx=0 tx=0).stop 2024-03-06T07:26:21.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.733+0000 7ff9b945c640 1 -- 172.21.15.83:0/1746539593 shutdown_connections 2024-03-06T07:26:21.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.733+0000 7ff9b945c640 1 --2- 172.21.15.83:0/1746539593 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff99406b4d0 0x7ff99406d990 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.733+0000 7ff9b945c640 1 --2- 172.21.15.83:0/1746539593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9b41614d0 0x7ff9b40b4120 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.733+0000 7ff9b945c640 1 --2- 172.21.15.83:0/1746539593 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff9b40d24b0 0x7ff9b4158440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.733+0000 7ff9b945c640 1 --2- 172.21.15.83:0/1746539593 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9b40d1910 0x7ff9b40cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.733+0000 7ff9b945c640 1 -- 172.21.15.83:0/1746539593 >> 172.21.15.83:0/1746539593 conn(0x7ff9b40d3d40 msgr2=0x7ff9b40cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:21.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.733+0000 7ff9b945c640 1 -- 172.21.15.83:0/1746539593 shutdown_connections 2024-03-06T07:26:21.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.733+0000 7ff9b945c640 1 -- 172.21.15.83:0/1746539593 wait complete. 2024-03-06T07:26:21.749 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:26:21.749 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:21.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.848+0000 7f5009d2c640 1 Processor -- start 2024-03-06T07:26:21.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.848+0000 7f5009d2c640 1 -- start start 2024-03-06T07:26:21.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.849+0000 7f5009d2c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f500406c680 0x7f500406c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.849+0000 7f5009d2c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f50040b0aa0 0x7f500406ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.849+0000 7f5009d2c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f500406cfa0 con 0x7f500415e180 2024-03-06T07:26:21.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.849+0000 7f5009d2c640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50040d06c0 con 0x7f500406c680 2024-03-06T07:26:21.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.849+0000 7f5009d2c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f500415f380 con 0x7f50040b0aa0 2024-03-06T07:26:21.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.849+0000 7f50037fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f500406c680 0x7f500406c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.849+0000 7f4ffaffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f500415e180 0x7f50040d0b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:35402/0 (socket says 172.21.15.83:35402) 2024-03-06T07:26:21.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.849+0000 7f4ffaffd640 1 -- 172.21.15.83:0/4098245756 learned_addr learned my addr 172.21.15.83:0/4098245756 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:21.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.849+0000 7f5003fff640 1 --2- 172.21.15.83:0/4098245756 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f50040b0aa0 0x7f500406ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.849+0000 7f50037fe640 1 -- 172.21.15.83:0/4098245756 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f50040b0aa0 msgr2=0x7f500406ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:21.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.849+0000 7f50037fe640 1 --2- 172.21.15.83:0/4098245756 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f50040b0aa0 0x7f500406ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.849+0000 7f50037fe640 1 -- 172.21.15.83:0/4098245756 >> v1:172.21.15.83:6789/0 conn(0x7f500415e180 legacy=0x7f50040d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.849+0000 7f50037fe640 1 -- 172.21.15.83:0/4098245756 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50040d0070 con 0x7f500406c680 2024-03-06T07:26:21.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.850+0000 7f50037fe640 1 --2- 172.21.15.83:0/4098245756 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f500406c680 0x7f500406c0c0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f4ff40097a0 tx=0x7f4ff402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb50f80294550dcb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.850+0000 7f5002ffd640 1 -- 172.21.15.83:0/4098245756 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff4035020 con 0x7f500406c680 2024-03-06T07:26:21.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.851+0000 7f5002ffd640 1 -- 172.21.15.83:0/4098245756 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ff4031830 con 0x7f500406c680 2024-03-06T07:26:21.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.851+0000 7f5002ffd640 1 -- 172.21.15.83:0/4098245756 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff4031c00 con 0x7f500406c680 2024-03-06T07:26:21.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.851+0000 7f5009d2c640 1 -- 172.21.15.83:0/4098245756 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f500406c680 msgr2=0x7f500406c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.851+0000 7f5009d2c640 1 --2- 172.21.15.83:0/4098245756 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f500406c680 0x7f500406c0c0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f4ff40097a0 tx=0x7f4ff402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.851+0000 7f5009d2c640 1 -- 172.21.15.83:0/4098245756 shutdown_connections 2024-03-06T07:26:21.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.851+0000 7f5009d2c640 1 --2- 172.21.15.83:0/4098245756 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f50040b0aa0 0x7f500406ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.851+0000 7f5009d2c640 1 --2- 172.21.15.83:0/4098245756 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f500406c680 0x7f500406c0c0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.851+0000 7f5009d2c640 1 -- 172.21.15.83:0/4098245756 >> 172.21.15.83:0/4098245756 conn(0x7f50040d1b20 msgr2=0x7f50040d1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:21.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.851+0000 7f5009d2c640 1 -- 172.21.15.83:0/4098245756 shutdown_connections 2024-03-06T07:26:21.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.851+0000 7f5009d2c640 1 -- 172.21.15.83:0/4098245756 wait complete. 2024-03-06T07:26:21.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.852+0000 7f5009d2c640 1 Processor -- start 2024-03-06T07:26:21.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.852+0000 7f5009d2c640 1 -- start start 2024-03-06T07:26:21.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.853+0000 7f5009d2c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f500406c680 0x7f50040b4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.853+0000 7f5009d2c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50040b0aa0 0x7f50041f3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.853+0000 7f5009d2c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f500415e180 0x7f500415ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.853+0000 7f5009d2c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50040cfe30 con 0x7f50040b0aa0 2024-03-06T07:26:21.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.853+0000 7f5009d2c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50040cfcb0 con 0x7f500406c680 2024-03-06T07:26:21.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.853+0000 7f5009d2c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f500415efa0 con 0x7f500415e180 2024-03-06T07:26:21.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.853+0000 7f4ffaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50040b0aa0 0x7f50041f3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.853+0000 7f4ffaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50040b0aa0 0x7f50041f3c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50442/0 (socket says 172.21.15.83:50442) 2024-03-06T07:26:21.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.853+0000 7f4ffaffd640 1 -- 172.21.15.83:0/1256597876 learned_addr learned my addr 172.21.15.83:0/1256597876 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:21.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.853+0000 7f5003fff640 1 --2- 172.21.15.83:0/1256597876 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f500415e180 0x7f500415ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.853+0000 7f50037fe640 1 --2- 172.21.15.83:0/1256597876 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f500406c680 0x7f50040b4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.854+0000 7f4ffaffd640 1 -- 172.21.15.83:0/1256597876 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f500415e180 msgr2=0x7f500415ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.854+0000 7f4ffaffd640 1 --2- 172.21.15.83:0/1256597876 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f500415e180 0x7f500415ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.854+0000 7f4ffaffd640 1 -- 172.21.15.83:0/1256597876 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f500406c680 msgr2=0x7f50040b4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.854+0000 7f4ffaffd640 1 --2- 172.21.15.83:0/1256597876 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f500406c680 0x7f50040b4930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.854+0000 7f4ffaffd640 1 -- 172.21.15.83:0/1256597876 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50040b4f10 con 0x7f50040b0aa0 2024-03-06T07:26:21.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.854+0000 7f50037fe640 1 --2- 172.21.15.83:0/1256597876 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f500406c680 0x7f50040b4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:21.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.854+0000 7f4ffaffd640 1 --2- 172.21.15.83:0/1256597876 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50040b0aa0 0x7f50041f3c20 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f4fec00c400 tx=0x7f4fec00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.854+0000 7f5000ff9640 1 -- 172.21.15.83:0/1256597876 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fec017070 con 0x7f50040b0aa0 2024-03-06T07:26:21.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.854+0000 7f5000ff9640 1 -- 172.21.15.83:0/1256597876 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fec005ce0 con 0x7f50040b0aa0 2024-03-06T07:26:21.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.854+0000 7f5009d2c640 1 -- 172.21.15.83:0/1256597876 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50040b51f0 con 0x7f50040b0aa0 2024-03-06T07:26:21.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.854+0000 7f5000ff9640 1 -- 172.21.15.83:0/1256597876 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fec0072f0 con 0x7f50040b0aa0 2024-03-06T07:26:21.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.854+0000 7f5009d2c640 1 -- 172.21.15.83:0/1256597876 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50040b57b0 con 0x7f50040b0aa0 2024-03-06T07:26:21.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.855+0000 7f5009d2c640 1 -- 172.21.15.83:0/1256597876 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f500404f330 con 0x7f50040b0aa0 2024-03-06T07:26:21.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.856+0000 7f5000ff9640 1 -- 172.21.15.83:0/1256597876 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4fec01a070 con 0x7f50040b0aa0 2024-03-06T07:26:21.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.856+0000 7f5000ff9640 1 --2- 172.21.15.83:0/1256597876 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4fe406b4d0 0x7f4fe406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.856+0000 7f5000ff9640 1 -- 172.21.15.83:0/1256597876 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4fec089500 con 0x7f50040b0aa0 2024-03-06T07:26:21.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.856+0000 7f50037fe640 1 --2- 172.21.15.83:0/1256597876 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4fe406b4d0 0x7f4fe406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.857+0000 7f50037fe640 1 --2- 172.21.15.83:0/1256597876 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4fe406b4d0 0x7f4fe406d990 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f4ff4036000 tx=0x7f4ff403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:21.860+0000 7f5000ff9640 1 -- 172.21.15.83:0/1256597876 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4fec057d30 con 0x7f50040b0aa0 2024-03-06T07:26:22.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.075+0000 7f5009d2c640 1 -- 172.21.15.83:0/1256597876 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f500406e850 con 0x7f50040b0aa0 2024-03-06T07:26:22.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.076+0000 7f5000ff9640 1 -- 172.21.15.83:0/1256597876 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f4fec05bdd0 con 0x7f50040b0aa0 2024-03-06T07:26:22.078 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:22.078 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":45,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5924,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/2933830298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2933830298},{"type":"v1","addr":"172.21.15.183:6834","nonce":2933830298}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":6490,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1025428736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":1025428736},{"type":"v1","addr":"172.21.15.183:6835","nonce":1025428736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":7335,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/3634489597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3634489597},{"type":"v1","addr":"172.21.15.83:6836","nonce":3634489597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.064335+0000","modified":"2024-03-06T07:26:21.071371+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7338},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7338":{"gid":7338,"name":"a","rank":0,"incarnation":43,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/2924411033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":2924411033},{"type":"v1","addr":"172.21.15.83:6837","nonce":2924411033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7338,"qdb_cluster":[7338]},"id":7}]} 2024-03-06T07:26:22.078 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 45 2024-03-06T07:26:22.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.078+0000 7f5009d2c640 1 -- 172.21.15.83:0/1256597876 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4fe406b4d0 msgr2=0x7f4fe406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.078+0000 7f5009d2c640 1 --2- 172.21.15.83:0/1256597876 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4fe406b4d0 0x7f4fe406d990 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f4ff4036000 tx=0x7f4ff403c040 comp rx=0 tx=0).stop 2024-03-06T07:26:22.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.078+0000 7f5009d2c640 1 -- 172.21.15.83:0/1256597876 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50040b0aa0 msgr2=0x7f50041f3c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.078+0000 7f5009d2c640 1 --2- 172.21.15.83:0/1256597876 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50040b0aa0 0x7f50041f3c20 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f4fec00c400 tx=0x7f4fec00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.078+0000 7f5009d2c640 1 -- 172.21.15.83:0/1256597876 shutdown_connections 2024-03-06T07:26:22.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.078+0000 7f5009d2c640 1 --2- 172.21.15.83:0/1256597876 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4fe406b4d0 0x7f4fe406d990 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.078+0000 7f5009d2c640 1 --2- 172.21.15.83:0/1256597876 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f500415e180 0x7f500415ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.078+0000 7f5009d2c640 1 --2- 172.21.15.83:0/1256597876 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50040b0aa0 0x7f50041f3c20 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.078+0000 7f5009d2c640 1 --2- 172.21.15.83:0/1256597876 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f500406c680 0x7f50040b4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.078+0000 7f5009d2c640 1 -- 172.21.15.83:0/1256597876 >> 172.21.15.83:0/1256597876 conn(0x7f50040d1b20 msgr2=0x7f5004154020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:22.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.078+0000 7f5009d2c640 1 -- 172.21.15.83:0/1256597876 shutdown_connections 2024-03-06T07:26:22.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.078+0000 7f5009d2c640 1 -- 172.21.15.83:0/1256597876 wait complete. 2024-03-06T07:26:22.091 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:22.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.190+0000 7fcd39081640 1 Processor -- start 2024-03-06T07:26:22.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.190+0000 7fcd39081640 1 -- start start 2024-03-06T07:26:22.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.190+0000 7fcd39081640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcd3415bdd0 0x7fcd340c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.190+0000 7fcd39081640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcd3415fa10 0x7fcd340ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.190+0000 7fcd39081640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fcd340d16e0 0x7fcd340d1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.190+0000 7fcd39081640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd340cebe0 con 0x7fcd3415fa10 2024-03-06T07:26:22.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.190+0000 7fcd39081640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd340ceec0 con 0x7fcd340d16e0 2024-03-06T07:26:22.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.190+0000 7fcd39081640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd340cf1a0 con 0x7fcd3415bdd0 2024-03-06T07:26:22.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.191+0000 7fcd3259c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcd3415fa10 0x7fcd340ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.191+0000 7fcd3259c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcd3415fa10 0x7fcd340ce270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50452/0 (socket says 172.21.15.83:50452) 2024-03-06T07:26:22.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.191+0000 7fcd3259c640 1 -- 172.21.15.83:0/3069379118 learned_addr learned my addr 172.21.15.83:0/3069379118 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:22.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.191+0000 7fcd3359e640 1 --2- 172.21.15.83:0/3069379118 >> v2:172.21.15.183:3300/0 conn(0x7fcd340d16e0 0x7fcd340d1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.191+0000 7fcd3259c640 1 -- 172.21.15.83:0/3069379118 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcd3415bdd0 msgr2=0x7fcd340c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:22.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.191+0000 7fcd32d9d640 1 --2- 172.21.15.83:0/3069379118 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcd3415bdd0 0x7fcd340c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.191+0000 7fcd3259c640 1 --2- 172.21.15.83:0/3069379118 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcd3415bdd0 0x7fcd340c5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.191+0000 7fcd3259c640 1 -- 172.21.15.83:0/3069379118 >> v2:172.21.15.183:3300/0 conn(0x7fcd340d16e0 msgr2=0x7fcd340d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:22.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.191+0000 7fcd3259c640 1 --2- 172.21.15.83:0/3069379118 >> v2:172.21.15.183:3300/0 conn(0x7fcd340d16e0 0x7fcd340d1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.191+0000 7fcd3259c640 1 -- 172.21.15.83:0/3069379118 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd340cc8e0 con 0x7fcd3415fa10 2024-03-06T07:26:22.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.191+0000 7fcd3359e640 1 --2- 172.21.15.83:0/3069379118 >> v2:172.21.15.183:3300/0 conn(0x7fcd340d16e0 0x7fcd340d1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:22.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.192+0000 7fcd3259c640 1 --2- 172.21.15.83:0/3069379118 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd3415fa10 0x7fcd340ce270 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fcd340729e0 tx=0x7fcd2802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a66b5b5d8acbf96c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.192+0000 7fcd31d9b640 1 -- 172.21.15.83:0/3069379118 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2802e070 con 0x7fcd3415fa10 2024-03-06T07:26:22.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.192+0000 7fcd31d9b640 1 -- 172.21.15.83:0/3069379118 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd28037070 con 0x7fcd3415fa10 2024-03-06T07:26:22.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.192+0000 7fcd31d9b640 1 -- 172.21.15.83:0/3069379118 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd28031a00 con 0x7fcd3415fa10 2024-03-06T07:26:22.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.192+0000 7fcd39081640 1 -- 172.21.15.83:0/3069379118 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd3415fa10 msgr2=0x7fcd340ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.192+0000 7fcd39081640 1 --2- 172.21.15.83:0/3069379118 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd3415fa10 0x7fcd340ce270 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fcd340729e0 tx=0x7fcd2802b880 comp rx=0 tx=0).stop 2024-03-06T07:26:22.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.192+0000 7fcd39081640 1 -- 172.21.15.83:0/3069379118 shutdown_connections 2024-03-06T07:26:22.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.192+0000 7fcd39081640 1 --2- 172.21.15.83:0/3069379118 >> v2:172.21.15.183:3300/0 conn(0x7fcd340d16e0 0x7fcd340d1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.192+0000 7fcd39081640 1 --2- 172.21.15.83:0/3069379118 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd3415fa10 0x7fcd340ce270 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.192+0000 7fcd39081640 1 --2- 172.21.15.83:0/3069379118 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcd3415bdd0 0x7fcd340c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.192+0000 7fcd39081640 1 -- 172.21.15.83:0/3069379118 >> 172.21.15.83:0/3069379118 conn(0x7fcd3409a9f0 msgr2=0x7fcd3409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:22.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.192+0000 7fcd39081640 1 -- 172.21.15.83:0/3069379118 shutdown_connections 2024-03-06T07:26:22.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.192+0000 7fcd39081640 1 -- 172.21.15.83:0/3069379118 wait complete. 2024-03-06T07:26:22.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.194+0000 7fcd39081640 1 Processor -- start 2024-03-06T07:26:22.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.194+0000 7fcd39081640 1 -- start start 2024-03-06T07:26:22.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.195+0000 7fcd39081640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd340d16e0 0x7fcd340cb760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.195+0000 7fcd39081640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcd3415bdd0 0x7fcd3415aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.195+0000 7fcd39081640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcd3415fa10 0x7fcd341563c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.195+0000 7fcd39081640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd340cc5d0 con 0x7fcd340d16e0 2024-03-06T07:26:22.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.195+0000 7fcd39081640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd340d2070 con 0x7fcd3415bdd0 2024-03-06T07:26:22.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.195+0000 7fcd39081640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3415c1b0 con 0x7fcd3415fa10 2024-03-06T07:26:22.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.195+0000 7fcd32d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd340d16e0 0x7fcd340cb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.195+0000 7fcd32d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd340d16e0 0x7fcd340cb760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50456/0 (socket says 172.21.15.83:50456) 2024-03-06T07:26:22.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.195+0000 7fcd32d9d640 1 -- 172.21.15.83:0/1149097598 learned_addr learned my addr 172.21.15.83:0/1149097598 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:22.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.195+0000 7fcd3259c640 1 --2- 172.21.15.83:0/1149097598 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcd3415bdd0 0x7fcd3415aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.195+0000 7fcd3359e640 1 --2- 172.21.15.83:0/1149097598 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcd3415fa10 0x7fcd341563c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.195+0000 7fcd32d9d640 1 -- 172.21.15.83:0/1149097598 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcd3415fa10 msgr2=0x7fcd341563c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.195+0000 7fcd32d9d640 1 --2- 172.21.15.83:0/1149097598 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcd3415fa10 0x7fcd341563c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.195+0000 7fcd32d9d640 1 -- 172.21.15.83:0/1149097598 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcd3415bdd0 msgr2=0x7fcd3415aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.195+0000 7fcd32d9d640 1 --2- 172.21.15.83:0/1149097598 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcd3415bdd0 0x7fcd3415aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.195+0000 7fcd32d9d640 1 -- 172.21.15.83:0/1149097598 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd3415b1e0 con 0x7fcd340d16e0 2024-03-06T07:26:22.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.195+0000 7fcd3359e640 1 --2- 172.21.15.83:0/1149097598 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcd3415fa10 0x7fcd341563c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:22.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.195+0000 7fcd3259c640 1 --2- 172.21.15.83:0/1149097598 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcd3415bdd0 0x7fcd3415aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:22.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.195+0000 7fcd32d9d640 1 --2- 172.21.15.83:0/1149097598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd340d16e0 0x7fcd340cb760 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fcd2400c820 tx=0x7fcd2400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.195+0000 7fcd1b7fe640 1 -- 172.21.15.83:0/1149097598 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd24015070 con 0x7fcd340d16e0 2024-03-06T07:26:22.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.195+0000 7fcd1b7fe640 1 -- 172.21.15.83:0/1149097598 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd24004020 con 0x7fcd340d16e0 2024-03-06T07:26:22.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.195+0000 7fcd39081640 1 -- 172.21.15.83:0/1149097598 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd340b5970 con 0x7fcd340d16e0 2024-03-06T07:26:22.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.195+0000 7fcd1b7fe640 1 -- 172.21.15.83:0/1149097598 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd240043f0 con 0x7fcd340d16e0 2024-03-06T07:26:22.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.195+0000 7fcd39081640 1 -- 172.21.15.83:0/1149097598 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd340b5f30 con 0x7fcd340d16e0 2024-03-06T07:26:22.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.197+0000 7fcd39081640 1 -- 172.21.15.83:0/1149097598 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd3404f330 con 0x7fcd340d16e0 2024-03-06T07:26:22.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.197+0000 7fcd1b7fe640 1 -- 172.21.15.83:0/1149097598 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcd24007830 con 0x7fcd340d16e0 2024-03-06T07:26:22.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.197+0000 7fcd1b7fe640 1 --2- 172.21.15.83:0/1149097598 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcd2006b4d0 0x7fcd2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.197+0000 7fcd1b7fe640 1 -- 172.21.15.83:0/1149097598 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcd24089f90 con 0x7fcd340d16e0 2024-03-06T07:26:22.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.197+0000 7fcd3259c640 1 --2- 172.21.15.83:0/1149097598 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcd2006b4d0 0x7fcd2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.198+0000 7fcd3259c640 1 --2- 172.21.15.83:0/1149097598 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcd2006b4d0 0x7fcd2006d990 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fcd28036000 tx=0x7fcd2803d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.202+0000 7fcd1b7fe640 1 -- 172.21.15.83:0/1149097598 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcd24058e20 con 0x7fcd340d16e0 2024-03-06T07:26:22.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.403+0000 7fcd39081640 1 -- 172.21.15.83:0/1149097598 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcd340b7a40 con 0x7fcd340d16e0 2024-03-06T07:26:22.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.404+0000 7fcd1b7fe640 1 -- 172.21.15.83:0/1149097598 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+16620 (secure 0 0 0) 0x7fcd24058720 con 0x7fcd340d16e0 2024-03-06T07:26:22.406 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:22.406 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":94,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:26:20.721381+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:17.275618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:18.028872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:22.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.407+0000 7fcd39081640 1 -- 172.21.15.83:0/1149097598 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcd2006b4d0 msgr2=0x7fcd2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.407+0000 7fcd39081640 1 --2- 172.21.15.83:0/1149097598 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcd2006b4d0 0x7fcd2006d990 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fcd28036000 tx=0x7fcd2803d040 comp rx=0 tx=0).stop 2024-03-06T07:26:22.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.407+0000 7fcd39081640 1 -- 172.21.15.83:0/1149097598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd340d16e0 msgr2=0x7fcd340cb760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.407+0000 7fcd39081640 1 --2- 172.21.15.83:0/1149097598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd340d16e0 0x7fcd340cb760 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fcd2400c820 tx=0x7fcd2400cce0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.407+0000 7fcd39081640 1 -- 172.21.15.83:0/1149097598 shutdown_connections 2024-03-06T07:26:22.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.407+0000 7fcd39081640 1 --2- 172.21.15.83:0/1149097598 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcd2006b4d0 0x7fcd2006d990 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.407+0000 7fcd39081640 1 --2- 172.21.15.83:0/1149097598 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcd3415fa10 0x7fcd341563c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.407+0000 7fcd39081640 1 --2- 172.21.15.83:0/1149097598 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcd3415bdd0 0x7fcd3415aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.407+0000 7fcd39081640 1 --2- 172.21.15.83:0/1149097598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcd340d16e0 0x7fcd340cb760 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.407+0000 7fcd39081640 1 -- 172.21.15.83:0/1149097598 >> 172.21.15.83:0/1149097598 conn(0x7fcd3409a9f0 msgr2=0x7fcd3415e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:22.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.407+0000 7fcd39081640 1 -- 172.21.15.83:0/1149097598 shutdown_connections 2024-03-06T07:26:22.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.407+0000 7fcd39081640 1 -- 172.21.15.83:0/1149097598 wait complete. 2024-03-06T07:26:22.422 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:26:22.422 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:26:22.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.521+0000 7ff9ddc2a640 1 Processor -- start 2024-03-06T07:26:22.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.521+0000 7ff9ddc2a640 1 -- start start 2024-03-06T07:26:22.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.522+0000 7ff9ddc2a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff9d815e0f0 0x7ff9d80cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.522+0000 7ff9ddc2a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9d815e4d0 con 0x7ff9d80d0660 2024-03-06T07:26:22.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.522+0000 7ff9ddc2a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9d80cdac0 con 0x7ff9d80cf850 2024-03-06T07:26:22.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.522+0000 7ff9ddc2a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9d80b1410 con 0x7ff9d815e0f0 2024-03-06T07:26:22.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.522+0000 7ff9d77fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff9d815e0f0 0x7ff9d80cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.522+0000 7ff9d7fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff9d80d0660 0x7ff9d80b25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:35410/0 (socket says 172.21.15.83:35410) 2024-03-06T07:26:22.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.522+0000 7ff9d7fff640 1 -- 172.21.15.83:0/627196602 learned_addr learned my addr 172.21.15.83:0/627196602 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:22.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.523+0000 7ff9d77fe640 1 -- 172.21.15.83:0/627196602 >> v1:172.21.15.183:6789/0 conn(0x7ff9d80cf850 legacy=0x7ff9d80cff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:22.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.523+0000 7ff9d77fe640 1 -- 172.21.15.83:0/627196602 >> v1:172.21.15.83:6789/0 conn(0x7ff9d80d0660 legacy=0x7ff9d80b25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.523+0000 7ff9d6ffd640 1 -- 172.21.15.83:0/627196602 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2084026078 0 0) 0x7ff9d815e4d0 con 0x7ff9d80d0660 2024-03-06T07:26:22.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.523+0000 7ff9d77fe640 1 -- 172.21.15.83:0/627196602 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d806c260 con 0x7ff9d815e0f0 2024-03-06T07:26:22.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.523+0000 7ff9d77fe640 1 --2- 172.21.15.83:0/627196602 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9d815e0f0 0x7ff9d80cd3c0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7ff9c80097a0 tx=0x7ff9c802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6d67849799020aa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.523+0000 7ff9d6ffd640 1 -- 172.21.15.83:0/627196602 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c8035020 con 0x7ff9d815e0f0 2024-03-06T07:26:22.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.524+0000 7ff9d6ffd640 1 -- 172.21.15.83:0/627196602 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9c8031830 con 0x7ff9d815e0f0 2024-03-06T07:26:22.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.524+0000 7ff9d6ffd640 1 -- 172.21.15.83:0/627196602 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c8031c00 con 0x7ff9d815e0f0 2024-03-06T07:26:22.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.524+0000 7ff9ddc2a640 1 -- 172.21.15.83:0/627196602 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9d815e0f0 msgr2=0x7ff9d80cd3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.524+0000 7ff9ddc2a640 1 --2- 172.21.15.83:0/627196602 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9d815e0f0 0x7ff9d80cd3c0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7ff9c80097a0 tx=0x7ff9c802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.524+0000 7ff9ddc2a640 1 -- 172.21.15.83:0/627196602 shutdown_connections 2024-03-06T07:26:22.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.524+0000 7ff9ddc2a640 1 --2- 172.21.15.83:0/627196602 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9d815e0f0 0x7ff9d80cd3c0 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.524+0000 7ff9ddc2a640 1 -- 172.21.15.83:0/627196602 >> 172.21.15.83:0/627196602 conn(0x7ff9d80b35a0 msgr2=0x7ff9d80b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:22.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.524+0000 7ff9ddc2a640 1 -- 172.21.15.83:0/627196602 shutdown_connections 2024-03-06T07:26:22.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.524+0000 7ff9ddc2a640 1 -- 172.21.15.83:0/627196602 wait complete. 2024-03-06T07:26:22.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.526+0000 7ff9ddc2a640 1 Processor -- start 2024-03-06T07:26:22.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.526+0000 7ff9ddc2a640 1 -- start start 2024-03-06T07:26:22.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.527+0000 7ff9ddc2a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9d80cf850 0x7ff9d80b74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.527+0000 7ff9ddc2a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9d80d0660 0x7ff9d80cb3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.527+0000 7ff9ddc2a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff9d815e0f0 0x7ff9d815e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.527+0000 7ff9ddc2a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d80d0cc0 con 0x7ff9d80cf850 2024-03-06T07:26:22.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.527+0000 7ff9ddc2a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d806c8b0 con 0x7ff9d815e0f0 2024-03-06T07:26:22.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.527+0000 7ff9ddc2a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d806dd50 con 0x7ff9d80d0660 2024-03-06T07:26:22.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.527+0000 7ff9d77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9d80cf850 0x7ff9d80b74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.527+0000 7ff9d77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9d80cf850 0x7ff9d80b74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50478/0 (socket says 172.21.15.83:50478) 2024-03-06T07:26:22.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.527+0000 7ff9d77fe640 1 -- 172.21.15.83:0/2196079870 learned_addr learned my addr 172.21.15.83:0/2196079870 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:22.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.528+0000 7ff9d7fff640 1 --2- 172.21.15.83:0/2196079870 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff9d815e0f0 0x7ff9d815e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.528+0000 7ff9ceffd640 1 --2- 172.21.15.83:0/2196079870 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9d80d0660 0x7ff9d80cb3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.528+0000 7ff9d77fe640 1 -- 172.21.15.83:0/2196079870 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9d80d0660 msgr2=0x7ff9d80cb3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.528+0000 7ff9d77fe640 1 --2- 172.21.15.83:0/2196079870 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9d80d0660 0x7ff9d80cb3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.528+0000 7ff9d77fe640 1 -- 172.21.15.83:0/2196079870 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff9d815e0f0 msgr2=0x7ff9d815e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.528+0000 7ff9d77fe640 1 --2- 172.21.15.83:0/2196079870 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff9d815e0f0 0x7ff9d815e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.528+0000 7ff9d77fe640 1 -- 172.21.15.83:0/2196079870 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d81f7350 con 0x7ff9d80cf850 2024-03-06T07:26:22.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.528+0000 7ff9d7fff640 1 --2- 172.21.15.83:0/2196079870 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff9d815e0f0 0x7ff9d815e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:22.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.528+0000 7ff9ceffd640 1 --2- 172.21.15.83:0/2196079870 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9d80d0660 0x7ff9d80cb3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:22.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.528+0000 7ff9d77fe640 1 --2- 172.21.15.83:0/2196079870 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9d80cf850 0x7ff9d80b74a0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7ff9c80097a0 tx=0x7ff9c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.528+0000 7ff9d4ff9640 1 -- 172.21.15.83:0/2196079870 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c803aa70 con 0x7ff9d80cf850 2024-03-06T07:26:22.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.528+0000 7ff9d4ff9640 1 -- 172.21.15.83:0/2196079870 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9c803ac10 con 0x7ff9d80cf850 2024-03-06T07:26:22.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.528+0000 7ff9d4ff9640 1 -- 172.21.15.83:0/2196079870 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c80443f0 con 0x7ff9d80cf850 2024-03-06T07:26:22.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.528+0000 7ff9ddc2a640 1 -- 172.21.15.83:0/2196079870 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9d81f7630 con 0x7ff9d80cf850 2024-03-06T07:26:22.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.528+0000 7ff9ddc2a640 1 -- 172.21.15.83:0/2196079870 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9d81f7bf0 con 0x7ff9d80cf850 2024-03-06T07:26:22.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.529+0000 7ff9ddc2a640 1 -- 172.21.15.83:0/2196079870 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9d8068700 con 0x7ff9d80cf850 2024-03-06T07:26:22.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.530+0000 7ff9d4ff9640 1 -- 172.21.15.83:0/2196079870 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff9c804b060 con 0x7ff9d80cf850 2024-03-06T07:26:22.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.530+0000 7ff9d4ff9640 1 --2- 172.21.15.83:0/2196079870 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff9b806b4d0 0x7ff9b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.530+0000 7ff9ceffd640 1 --2- 172.21.15.83:0/2196079870 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff9b806b4d0 0x7ff9b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.530+0000 7ff9d4ff9640 1 -- 172.21.15.83:0/2196079870 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff9c80abf60 con 0x7ff9d80cf850 2024-03-06T07:26:22.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.531+0000 7ff9ceffd640 1 --2- 172.21.15.83:0/2196079870 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff9b806b4d0 0x7ff9b806d990 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7ff9d815f960 tx=0x7ff9c0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.534+0000 7ff9d4ff9640 1 -- 172.21.15.83:0/2196079870 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff9c803f070 con 0x7ff9d80cf850 2024-03-06T07:26:22.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.708+0000 7ff9ddc2a640 1 -- 172.21.15.83:0/2196079870 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff9d804f330 con 0x7ff9b806b4d0 2024-03-06T07:26:22.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.714+0000 7ff9d4ff9640 1 -- 172.21.15.83:0/2196079870 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252015 (secure 0 0 0) 0x7ff9c0055880 con 0x7ff9b806b4d0 2024-03-06T07:26:22.719 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:22.720 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-06T07:26:22.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.717+0000 7ff9ddc2a640 1 -- 172.21.15.83:0/2196079870 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff9b806b4d0 msgr2=0x7ff9b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.717+0000 7ff9ddc2a640 1 --2- 172.21.15.83:0/2196079870 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff9b806b4d0 0x7ff9b806d990 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7ff9d815f960 tx=0x7ff9c0009040 comp rx=0 tx=0).stop 2024-03-06T07:26:22.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.717+0000 7ff9ddc2a640 1 -- 172.21.15.83:0/2196079870 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9d80cf850 msgr2=0x7ff9d80b74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.717+0000 7ff9ddc2a640 1 --2- 172.21.15.83:0/2196079870 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9d80cf850 0x7ff9d80b74a0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7ff9c80097a0 tx=0x7ff9c80046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.717+0000 7ff9ddc2a640 1 -- 172.21.15.83:0/2196079870 shutdown_connections 2024-03-06T07:26:22.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.717+0000 7ff9ddc2a640 1 --2- 172.21.15.83:0/2196079870 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff9b806b4d0 0x7ff9b806d990 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.717+0000 7ff9ddc2a640 1 --2- 172.21.15.83:0/2196079870 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff9d815e0f0 0x7ff9d815e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.717+0000 7ff9ddc2a640 1 --2- 172.21.15.83:0/2196079870 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9d80d0660 0x7ff9d80cb3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.717+0000 7ff9ddc2a640 1 --2- 172.21.15.83:0/2196079870 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9d80cf850 0x7ff9d80b74a0 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.717+0000 7ff9ddc2a640 1 -- 172.21.15.83:0/2196079870 >> 172.21.15.83:0/2196079870 conn(0x7ff9d80b35a0 msgr2=0x7ff9d806ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:22.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.717+0000 7ff9ddc2a640 1 -- 172.21.15.83:0/2196079870 shutdown_connections 2024-03-06T07:26:22.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.717+0000 7ff9ddc2a640 1 -- 172.21.15.83:0/2196079870 wait complete. 2024-03-06T07:26:22.730 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":226,"stamp":"2024-03-06T07:26:22.367900+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":59,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":259656,"kb_used_data":10164,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754715064,"statfs":{"total":773094113280,"available":772828225536,"internally_reserved":0,"allocated":10407936,"data_stored":5092690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12757,"internal_metadata":255315499},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.661962"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.742127+0000","last_change":"2024-03-06T07:26:19.735768+0000","last_active":"2024-03-06T07:26:19.742127+0000","last_peered":"2024-03-06T07:26:19.742127+0000","last_clean":"2024-03-06T07:26:19.742127+0000","last_became_active":"2024-03-06T07:26:19.735493+0000","last_became_peered":"2024-03-06T07:26:19.735493+0000","last_unstale":"2024-03-06T07:26:19.742127+0000","last_undegraded":"2024-03-06T07:26:19.742127+0000","last_fullsized":"2024-03-06T07:26:19.742127+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:52:29.357695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.732073+0000","last_change":"2024-03-06T07:26:19.727695+0000","last_active":"2024-03-06T07:26:19.732073+0000","last_peered":"2024-03-06T07:26:19.732073+0000","last_clean":"2024-03-06T07:26:19.732073+0000","last_became_active":"2024-03-06T07:26:19.727435+0000","last_became_peered":"2024-03-06T07:26:19.727435+0000","last_unstale":"2024-03-06T07:26:19.732073+0000","last_undegraded":"2024-03-06T07:26:19.732073+0000","last_fullsized":"2024-03-06T07:26:19.732073+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:55.844050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.742193+0000","last_change":"2024-03-06T07:26:19.735708+0000","last_active":"2024-03-06T07:26:19.742193+0000","last_peered":"2024-03-06T07:26:19.742193+0000","last_clean":"2024-03-06T07:26:19.742193+0000","last_became_active":"2024-03-06T07:26:19.735457+0000","last_became_peered":"2024-03-06T07:26:19.735457+0000","last_unstale":"2024-03-06T07:26:19.742193+0000","last_undegraded":"2024-03-06T07:26:19.742193+0000","last_fullsized":"2024-03-06T07:26:19.742193+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:37.811558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.726497+0000","last_change":"2024-03-06T07:26:19.723522+0000","last_active":"2024-03-06T07:26:19.726497+0000","last_peered":"2024-03-06T07:26:19.726497+0000","last_clean":"2024-03-06T07:26:19.726497+0000","last_became_active":"2024-03-06T07:26:19.723265+0000","last_became_peered":"2024-03-06T07:26:19.723265+0000","last_unstale":"2024-03-06T07:26:19.726497+0000","last_undegraded":"2024-03-06T07:26:19.726497+0000","last_fullsized":"2024-03-06T07:26:19.726497+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:17.128778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.739496+0000","last_change":"2024-03-06T07:26:19.737877+0000","last_active":"2024-03-06T07:26:19.739496+0000","last_peered":"2024-03-06T07:26:19.739496+0000","last_clean":"2024-03-06T07:26:19.739496+0000","last_became_active":"2024-03-06T07:26:19.737483+0000","last_became_peered":"2024-03-06T07:26:19.737483+0000","last_unstale":"2024-03-06T07:26:19.739496+0000","last_undegraded":"2024-03-06T07:26:19.739496+0000","last_fullsized":"2024-03-06T07:26:19.739496+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:46:28.616508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.726743+0000","last_change":"2024-03-06T07:26:19.725055+0000","last_active":"2024-03-06T07:26:19.726743+0000","last_peered":"2024-03-06T07:26:19.726743+0000","last_clean":"2024-03-06T07:26:19.726743+0000","last_became_active":"2024-03-06T07:26:19.724782+0000","last_became_peered":"2024-03-06T07:26:19.724782+0000","last_unstale":"2024-03-06T07:26:19.726743+0000","last_undegraded":"2024-03-06T07:26:19.726743+0000","last_fullsized":"2024-03-06T07:26:19.726743+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:27:46.844174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.725722+0000","last_change":"2024-03-06T07:26:19.724068+0000","last_active":"2024-03-06T07:26:19.725722+0000","last_peered":"2024-03-06T07:26:19.725722+0000","last_clean":"2024-03-06T07:26:19.725722+0000","last_became_active":"2024-03-06T07:26:19.723769+0000","last_became_peered":"2024-03-06T07:26:19.723769+0000","last_unstale":"2024-03-06T07:26:19.725722+0000","last_undegraded":"2024-03-06T07:26:19.725722+0000","last_fullsized":"2024-03-06T07:26:19.725722+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:53:35.777661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.731836+0000","last_change":"2024-03-06T07:26:19.729272+0000","last_active":"2024-03-06T07:26:19.731836+0000","last_peered":"2024-03-06T07:26:19.731836+0000","last_clean":"2024-03-06T07:26:19.731836+0000","last_became_active":"2024-03-06T07:26:19.728926+0000","last_became_peered":"2024-03-06T07:26:19.728926+0000","last_unstale":"2024-03-06T07:26:19.731836+0000","last_undegraded":"2024-03-06T07:26:19.731836+0000","last_fullsized":"2024-03-06T07:26:19.731836+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:23.513679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.728910+0000","last_change":"2024-03-06T07:26:19.726431+0000","last_active":"2024-03-06T07:26:19.728910+0000","last_peered":"2024-03-06T07:26:19.728910+0000","last_clean":"2024-03-06T07:26:19.728910+0000","last_became_active":"2024-03-06T07:26:19.726113+0000","last_became_peered":"2024-03-06T07:26:19.726113+0000","last_unstale":"2024-03-06T07:26:19.728910+0000","last_undegraded":"2024-03-06T07:26:19.728910+0000","last_fullsized":"2024-03-06T07:26:19.728910+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:30.017299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.736857+0000","last_change":"2024-03-06T07:26:19.733949+0000","last_active":"2024-03-06T07:26:19.736857+0000","last_peered":"2024-03-06T07:26:19.736857+0000","last_clean":"2024-03-06T07:26:19.736857+0000","last_became_active":"2024-03-06T07:26:19.733655+0000","last_became_peered":"2024-03-06T07:26:19.733655+0000","last_unstale":"2024-03-06T07:26:19.736857+0000","last_undegraded":"2024-03-06T07:26:19.736857+0000","last_fullsized":"2024-03-06T07:26:19.736857+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:27:46.844174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.728330+0000","last_change":"2024-03-06T07:26:19.726190+0000","last_active":"2024-03-06T07:26:19.728330+0000","last_peered":"2024-03-06T07:26:19.728330+0000","last_clean":"2024-03-06T07:26:19.728330+0000","last_became_active":"2024-03-06T07:26:19.725936+0000","last_became_peered":"2024-03-06T07:26:19.725936+0000","last_unstale":"2024-03-06T07:26:19.728330+0000","last_undegraded":"2024-03-06T07:26:19.728330+0000","last_fullsized":"2024-03-06T07:26:19.728330+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:46:28.616508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.739285+0000","last_change":"2024-03-06T07:26:19.734412+0000","last_active":"2024-03-06T07:26:19.739285+0000","last_peered":"2024-03-06T07:26:19.739285+0000","last_clean":"2024-03-06T07:26:19.739285+0000","last_became_active":"2024-03-06T07:26:19.734139+0000","last_became_peered":"2024-03-06T07:26:19.734139+0000","last_unstale":"2024-03-06T07:26:19.739285+0000","last_undegraded":"2024-03-06T07:26:19.739285+0000","last_fullsized":"2024-03-06T07:26:19.739285+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:53:35.777661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.736598+0000","last_change":"2024-03-06T07:26:19.734391+0000","last_active":"2024-03-06T07:26:19.736598+0000","last_peered":"2024-03-06T07:26:19.736598+0000","last_clean":"2024-03-06T07:26:19.736598+0000","last_became_active":"2024-03-06T07:26:19.734138+0000","last_became_peered":"2024-03-06T07:26:19.734138+0000","last_unstale":"2024-03-06T07:26:19.736598+0000","last_undegraded":"2024-03-06T07:26:19.736598+0000","last_fullsized":"2024-03-06T07:26:19.736598+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:54.623542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.736745+0000","last_change":"2024-03-06T07:26:19.731014+0000","last_active":"2024-03-06T07:26:19.736745+0000","last_peered":"2024-03-06T07:26:19.736745+0000","last_clean":"2024-03-06T07:26:19.736745+0000","last_became_active":"2024-03-06T07:26:19.730286+0000","last_became_peered":"2024-03-06T07:26:19.730286+0000","last_unstale":"2024-03-06T07:26:19.736745+0000","last_undegraded":"2024-03-06T07:26:19.736745+0000","last_fullsized":"2024-03-06T07:26:19.736745+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:52:29.357695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.741787+0000","last_change":"2024-03-06T07:26:19.740852+0000","last_active":"2024-03-06T07:26:19.741787+0000","last_peered":"2024-03-06T07:26:19.741787+0000","last_clean":"2024-03-06T07:26:19.741787+0000","last_became_active":"2024-03-06T07:26:19.740588+0000","last_became_peered":"2024-03-06T07:26:19.740588+0000","last_unstale":"2024-03-06T07:26:19.741787+0000","last_undegraded":"2024-03-06T07:26:19.741787+0000","last_fullsized":"2024-03-06T07:26:19.741787+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:27:46.844174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.736126+0000","last_change":"2024-03-06T07:26:19.732371+0000","last_active":"2024-03-06T07:26:19.736126+0000","last_peered":"2024-03-06T07:26:19.736126+0000","last_clean":"2024-03-06T07:26:19.736126+0000","last_became_active":"2024-03-06T07:26:19.732095+0000","last_became_peered":"2024-03-06T07:26:19.732095+0000","last_unstale":"2024-03-06T07:26:19.736126+0000","last_undegraded":"2024-03-06T07:26:19.736126+0000","last_fullsized":"2024-03-06T07:26:19.736126+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:55.844050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.731677+0000","last_change":"2024-03-06T07:26:19.724714+0000","last_active":"2024-03-06T07:26:19.731677+0000","last_peered":"2024-03-06T07:26:19.731677+0000","last_clean":"2024-03-06T07:26:19.731677+0000","last_became_active":"2024-03-06T07:26:19.724362+0000","last_became_peered":"2024-03-06T07:26:19.724362+0000","last_unstale":"2024-03-06T07:26:19.731677+0000","last_undegraded":"2024-03-06T07:26:19.731677+0000","last_fullsized":"2024-03-06T07:26:19.731677+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:27:46.844174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.736550+0000","last_change":"2024-03-06T07:26:19.735101+0000","last_active":"2024-03-06T07:26:19.736550+0000","last_peered":"2024-03-06T07:26:19.736550+0000","last_clean":"2024-03-06T07:26:19.736550+0000","last_became_active":"2024-03-06T07:26:19.734816+0000","last_became_peered":"2024-03-06T07:26:19.734816+0000","last_unstale":"2024-03-06T07:26:19.736550+0000","last_undegraded":"2024-03-06T07:26:19.736550+0000","last_fullsized":"2024-03-06T07:26:19.736550+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:11:14.556391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.738754+0000","last_change":"2024-03-06T07:26:19.734210+0000","last_active":"2024-03-06T07:26:19.738754+0000","last_peered":"2024-03-06T07:26:19.738754+0000","last_clean":"2024-03-06T07:26:19.738754+0000","last_became_active":"2024-03-06T07:26:19.733952+0000","last_became_peered":"2024-03-06T07:26:19.733952+0000","last_unstale":"2024-03-06T07:26:19.738754+0000","last_undegraded":"2024-03-06T07:26:19.738754+0000","last_fullsized":"2024-03-06T07:26:19.738754+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:06.236632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.732778+0000","last_change":"2024-03-06T07:26:19.728609+0000","last_active":"2024-03-06T07:26:19.732778+0000","last_peered":"2024-03-06T07:26:19.732778+0000","last_clean":"2024-03-06T07:26:19.732778+0000","last_became_active":"2024-03-06T07:26:19.727143+0000","last_became_peered":"2024-03-06T07:26:19.727143+0000","last_unstale":"2024-03-06T07:26:19.732778+0000","last_undegraded":"2024-03-06T07:26:19.732778+0000","last_fullsized":"2024-03-06T07:26:19.732778+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:17.550360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.736051+0000","last_change":"2024-03-06T07:26:19.727073+0000","last_active":"2024-03-06T07:26:19.736051+0000","last_peered":"2024-03-06T07:26:19.736051+0000","last_clean":"2024-03-06T07:26:19.736051+0000","last_became_active":"2024-03-06T07:26:19.726789+0000","last_became_peered":"2024-03-06T07:26:19.726789+0000","last_unstale":"2024-03-06T07:26:19.736051+0000","last_undegraded":"2024-03-06T07:26:19.736051+0000","last_fullsized":"2024-03-06T07:26:19.736051+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:52:11.743874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.735278+0000","last_change":"2024-03-06T07:26:19.724480+0000","last_active":"2024-03-06T07:26:19.735278+0000","last_peered":"2024-03-06T07:26:19.735278+0000","last_clean":"2024-03-06T07:26:19.735278+0000","last_became_active":"2024-03-06T07:26:19.724164+0000","last_became_peered":"2024-03-06T07:26:19.724164+0000","last_unstale":"2024-03-06T07:26:19.735278+0000","last_undegraded":"2024-03-06T07:26:19.735278+0000","last_fullsized":"2024-03-06T07:26:19.735278+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:56.348682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.731413+0000","last_change":"2024-03-06T07:26:19.725097+0000","last_active":"2024-03-06T07:26:19.731413+0000","last_peered":"2024-03-06T07:26:19.731413+0000","last_clean":"2024-03-06T07:26:19.731413+0000","last_became_active":"2024-03-06T07:26:19.724372+0000","last_became_peered":"2024-03-06T07:26:19.724372+0000","last_unstale":"2024-03-06T07:26:19.731413+0000","last_undegraded":"2024-03-06T07:26:19.731413+0000","last_fullsized":"2024-03-06T07:26:19.731413+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:46:28.616508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.740057+0000","last_change":"2024-03-06T07:26:19.739513+0000","last_active":"2024-03-06T07:26:19.740057+0000","last_peered":"2024-03-06T07:26:19.740057+0000","last_clean":"2024-03-06T07:26:19.740057+0000","last_became_active":"2024-03-06T07:26:19.739139+0000","last_became_peered":"2024-03-06T07:26:19.739139+0000","last_unstale":"2024-03-06T07:26:19.740057+0000","last_undegraded":"2024-03-06T07:26:19.740057+0000","last_fullsized":"2024-03-06T07:26:19.740057+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:19:22.032163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.732967+0000","last_change":"2024-03-06T07:26:19.730127+0000","last_active":"2024-03-06T07:26:19.732967+0000","last_peered":"2024-03-06T07:26:19.732967+0000","last_clean":"2024-03-06T07:26:19.732967+0000","last_became_active":"2024-03-06T07:26:19.729672+0000","last_became_peered":"2024-03-06T07:26:19.729672+0000","last_unstale":"2024-03-06T07:26:19.732967+0000","last_undegraded":"2024-03-06T07:26:19.732967+0000","last_fullsized":"2024-03-06T07:26:19.732967+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:17.550360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.740295+0000","last_change":"2024-03-06T07:26:19.738982+0000","last_active":"2024-03-06T07:26:19.740295+0000","last_peered":"2024-03-06T07:26:19.740295+0000","last_clean":"2024-03-06T07:26:19.740295+0000","last_became_active":"2024-03-06T07:26:19.738687+0000","last_became_peered":"2024-03-06T07:26:19.738687+0000","last_unstale":"2024-03-06T07:26:19.740295+0000","last_undegraded":"2024-03-06T07:26:19.740295+0000","last_fullsized":"2024-03-06T07:26:19.740295+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:46:25.686772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.738793+0000","last_change":"2024-03-06T07:26:19.736690+0000","last_active":"2024-03-06T07:26:19.738793+0000","last_peered":"2024-03-06T07:26:19.738793+0000","last_clean":"2024-03-06T07:26:19.738793+0000","last_became_active":"2024-03-06T07:26:19.736240+0000","last_became_peered":"2024-03-06T07:26:19.736240+0000","last_unstale":"2024-03-06T07:26:19.738793+0000","last_undegraded":"2024-03-06T07:26:19.738793+0000","last_fullsized":"2024-03-06T07:26:19.738793+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:23.513679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.727156+0000","last_change":"2024-03-06T07:26:19.724494+0000","last_active":"2024-03-06T07:26:19.727156+0000","last_peered":"2024-03-06T07:26:19.727156+0000","last_clean":"2024-03-06T07:26:19.727156+0000","last_became_active":"2024-03-06T07:26:19.724204+0000","last_became_peered":"2024-03-06T07:26:19.724204+0000","last_unstale":"2024-03-06T07:26:19.727156+0000","last_undegraded":"2024-03-06T07:26:19.727156+0000","last_fullsized":"2024-03-06T07:26:19.727156+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:30.017299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.738409+0000","last_change":"2024-03-06T07:26:19.733074+0000","last_active":"2024-03-06T07:26:19.738409+0000","last_peered":"2024-03-06T07:26:19.738409+0000","last_clean":"2024-03-06T07:26:19.738409+0000","last_became_active":"2024-03-06T07:26:19.732813+0000","last_became_peered":"2024-03-06T07:26:19.732813+0000","last_unstale":"2024-03-06T07:26:19.738409+0000","last_undegraded":"2024-03-06T07:26:19.738409+0000","last_fullsized":"2024-03-06T07:26:19.738409+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:19:22.032163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.738549+0000","last_change":"2024-03-06T07:26:19.736540+0000","last_active":"2024-03-06T07:26:19.738549+0000","last_peered":"2024-03-06T07:26:19.738549+0000","last_clean":"2024-03-06T07:26:19.738549+0000","last_became_active":"2024-03-06T07:26:19.736246+0000","last_became_peered":"2024-03-06T07:26:19.736246+0000","last_unstale":"2024-03-06T07:26:19.738549+0000","last_undegraded":"2024-03-06T07:26:19.738549+0000","last_fullsized":"2024-03-06T07:26:19.738549+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:59.857690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.737575+0000","last_change":"2024-03-06T07:26:19.736012+0000","last_active":"2024-03-06T07:26:19.737575+0000","last_peered":"2024-03-06T07:26:19.737575+0000","last_clean":"2024-03-06T07:26:19.737575+0000","last_became_active":"2024-03-06T07:26:19.735750+0000","last_became_peered":"2024-03-06T07:26:19.735750+0000","last_unstale":"2024-03-06T07:26:19.737575+0000","last_undegraded":"2024-03-06T07:26:19.737575+0000","last_fullsized":"2024-03-06T07:26:19.737575+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:17.550360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.742770+0000","last_change":"2024-03-06T07:26:19.738603+0000","last_active":"2024-03-06T07:26:19.742770+0000","last_peered":"2024-03-06T07:26:19.742770+0000","last_clean":"2024-03-06T07:26:19.742770+0000","last_became_active":"2024-03-06T07:26:19.738352+0000","last_became_peered":"2024-03-06T07:26:19.738352+0000","last_unstale":"2024-03-06T07:26:19.742770+0000","last_undegraded":"2024-03-06T07:26:19.742770+0000","last_fullsized":"2024-03-06T07:26:19.742770+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:16.131164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.736014+0000","last_change":"2024-03-06T07:26:19.727501+0000","last_active":"2024-03-06T07:26:19.736014+0000","last_peered":"2024-03-06T07:26:19.736014+0000","last_clean":"2024-03-06T07:26:19.736014+0000","last_became_active":"2024-03-06T07:26:19.727061+0000","last_became_peered":"2024-03-06T07:26:19.727061+0000","last_unstale":"2024-03-06T07:26:19.736014+0000","last_undegraded":"2024-03-06T07:26:19.736014+0000","last_fullsized":"2024-03-06T07:26:19.736014+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:52:08.732418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"93'4","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.737746+0000","last_change":"2024-03-06T07:26:18.726716+0000","last_active":"2024-03-06T07:26:19.737746+0000","last_peered":"2024-03-06T07:26:19.737746+0000","last_clean":"2024-03-06T07:26:19.737746+0000","last_became_active":"2024-03-06T07:26:18.726460+0000","last_became_peered":"2024-03-06T07:26:18.726460+0000","last_unstale":"2024-03-06T07:26:19.737746+0000","last_undegraded":"2024-03-06T07:26:19.737746+0000","last_fullsized":"2024-03-06T07:26:19.737746+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:13:09.899872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.739673+0000","last_change":"2024-03-06T07:26:19.737967+0000","last_active":"2024-03-06T07:26:19.739673+0000","last_peered":"2024-03-06T07:26:19.739673+0000","last_clean":"2024-03-06T07:26:19.739673+0000","last_became_active":"2024-03-06T07:26:19.737673+0000","last_became_peered":"2024-03-06T07:26:19.737673+0000","last_unstale":"2024-03-06T07:26:19.739673+0000","last_undegraded":"2024-03-06T07:26:19.739673+0000","last_fullsized":"2024-03-06T07:26:19.739673+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:53:35.777661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"93'1","reported_seq":13,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.742244+0000","last_change":"2024-03-06T07:26:18.726268+0000","last_active":"2024-03-06T07:26:19.742244+0000","last_peered":"2024-03-06T07:26:19.742244+0000","last_clean":"2024-03-06T07:26:19.742244+0000","last_became_active":"2024-03-06T07:26:18.725937+0000","last_became_peered":"2024-03-06T07:26:18.725937+0000","last_unstale":"2024-03-06T07:26:19.742244+0000","last_undegraded":"2024-03-06T07:26:19.742244+0000","last_fullsized":"2024-03-06T07:26:19.742244+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:11:13.546951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.742843+0000","last_change":"2024-03-06T07:26:19.740104+0000","last_active":"2024-03-06T07:26:19.742843+0000","last_peered":"2024-03-06T07:26:19.742843+0000","last_clean":"2024-03-06T07:26:19.742843+0000","last_became_active":"2024-03-06T07:26:19.739816+0000","last_became_peered":"2024-03-06T07:26:19.739816+0000","last_unstale":"2024-03-06T07:26:19.742843+0000","last_undegraded":"2024-03-06T07:26:19.742843+0000","last_fullsized":"2024-03-06T07:26:19.742843+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:06.236632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"93'2","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.737061+0000","last_change":"2024-03-06T07:26:18.727278+0000","last_active":"2024-03-06T07:26:19.737061+0000","last_peered":"2024-03-06T07:26:19.737061+0000","last_clean":"2024-03-06T07:26:19.737061+0000","last_became_active":"2024-03-06T07:26:18.727010+0000","last_became_peered":"2024-03-06T07:26:18.727010+0000","last_unstale":"2024-03-06T07:26:19.737061+0000","last_undegraded":"2024-03-06T07:26:19.737061+0000","last_fullsized":"2024-03-06T07:26:19.737061+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:16.540921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.728229+0000","last_change":"2024-03-06T07:26:19.725287+0000","last_active":"2024-03-06T07:26:19.728229+0000","last_peered":"2024-03-06T07:26:19.728229+0000","last_clean":"2024-03-06T07:26:19.728229+0000","last_became_active":"2024-03-06T07:26:19.725009+0000","last_became_peered":"2024-03-06T07:26:19.725009+0000","last_unstale":"2024-03-06T07:26:19.728229+0000","last_undegraded":"2024-03-06T07:26:19.728229+0000","last_fullsized":"2024-03-06T07:26:19.728229+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:17.128778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"93'2","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.736627+0000","last_change":"2024-03-06T07:26:18.729060+0000","last_active":"2024-03-06T07:26:19.736627+0000","last_peered":"2024-03-06T07:26:19.736627+0000","last_clean":"2024-03-06T07:26:19.736627+0000","last_became_active":"2024-03-06T07:26:18.728795+0000","last_became_peered":"2024-03-06T07:26:18.728795+0000","last_unstale":"2024-03-06T07:26:19.736627+0000","last_undegraded":"2024-03-06T07:26:19.736627+0000","last_fullsized":"2024-03-06T07:26:19.736627+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:38:12.922468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.731219+0000","last_change":"2024-03-06T07:26:19.725732+0000","last_active":"2024-03-06T07:26:19.731219+0000","last_peered":"2024-03-06T07:26:19.731219+0000","last_clean":"2024-03-06T07:26:19.731219+0000","last_became_active":"2024-03-06T07:26:19.725443+0000","last_became_peered":"2024-03-06T07:26:19.725443+0000","last_unstale":"2024-03-06T07:26:19.731219+0000","last_undegraded":"2024-03-06T07:26:19.731219+0000","last_fullsized":"2024-03-06T07:26:19.731219+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:41.542738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"93'1","reported_seq":13,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.741861+0000","last_change":"2024-03-06T07:26:18.728494+0000","last_active":"2024-03-06T07:26:19.741861+0000","last_peered":"2024-03-06T07:26:19.741861+0000","last_clean":"2024-03-06T07:26:19.741861+0000","last_became_active":"2024-03-06T07:26:18.728198+0000","last_became_peered":"2024-03-06T07:26:18.728198+0000","last_unstale":"2024-03-06T07:26:19.741861+0000","last_undegraded":"2024-03-06T07:26:19.741861+0000","last_fullsized":"2024-03-06T07:26:19.741861+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:10:30.091943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.731110+0000","last_change":"2024-03-06T07:26:19.728888+0000","last_active":"2024-03-06T07:26:19.731110+0000","last_peered":"2024-03-06T07:26:19.731110+0000","last_clean":"2024-03-06T07:26:19.731110+0000","last_became_active":"2024-03-06T07:26:19.728656+0000","last_became_peered":"2024-03-06T07:26:19.728656+0000","last_unstale":"2024-03-06T07:26:19.731110+0000","last_undegraded":"2024-03-06T07:26:19.731110+0000","last_fullsized":"2024-03-06T07:26:19.731110+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:08.203747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"93'1","reported_seq":13,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.740089+0000","last_change":"2024-03-06T07:26:18.724482+0000","last_active":"2024-03-06T07:26:19.740089+0000","last_peered":"2024-03-06T07:26:19.740089+0000","last_clean":"2024-03-06T07:26:19.740089+0000","last_became_active":"2024-03-06T07:26:18.724150+0000","last_became_peered":"2024-03-06T07:26:18.724150+0000","last_unstale":"2024-03-06T07:26:19.740089+0000","last_undegraded":"2024-03-06T07:26:19.740089+0000","last_fullsized":"2024-03-06T07:26:19.740089+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:36:42.959958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.741858+0000","last_change":"2024-03-06T07:26:19.740964+0000","last_active":"2024-03-06T07:26:19.741858+0000","last_peered":"2024-03-06T07:26:19.741858+0000","last_clean":"2024-03-06T07:26:19.741858+0000","last_became_active":"2024-03-06T07:26:19.740694+0000","last_became_peered":"2024-03-06T07:26:19.740694+0000","last_unstale":"2024-03-06T07:26:19.741858+0000","last_undegraded":"2024-03-06T07:26:19.741858+0000","last_fullsized":"2024-03-06T07:26:19.741858+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:46:28.616508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.715608+0000","last_change":"2024-03-06T07:26:18.732422+0000","last_active":"2024-03-06T07:26:19.715608+0000","last_peered":"2024-03-06T07:26:19.715608+0000","last_clean":"2024-03-06T07:26:19.715608+0000","last_became_active":"2024-03-06T07:26:18.732149+0000","last_became_peered":"2024-03-06T07:26:18.732149+0000","last_unstale":"2024-03-06T07:26:19.715608+0000","last_undegraded":"2024-03-06T07:26:19.715608+0000","last_fullsized":"2024-03-06T07:26:19.715608+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:29.117687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.728054+0000","last_change":"2024-03-06T07:26:19.723401+0000","last_active":"2024-03-06T07:26:19.728054+0000","last_peered":"2024-03-06T07:26:19.728054+0000","last_clean":"2024-03-06T07:26:19.728054+0000","last_became_active":"2024-03-06T07:26:19.722863+0000","last_became_peered":"2024-03-06T07:26:19.722863+0000","last_unstale":"2024-03-06T07:26:19.728054+0000","last_undegraded":"2024-03-06T07:26:19.728054+0000","last_fullsized":"2024-03-06T07:26:19.728054+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:36:43.969398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.726106+0000","last_change":"2024-03-06T07:26:18.725961+0000","last_active":"2024-03-06T07:26:19.726106+0000","last_peered":"2024-03-06T07:26:19.726106+0000","last_clean":"2024-03-06T07:26:19.726106+0000","last_became_active":"2024-03-06T07:26:18.725715+0000","last_became_peered":"2024-03-06T07:26:18.725715+0000","last_unstale":"2024-03-06T07:26:19.726106+0000","last_undegraded":"2024-03-06T07:26:19.726106+0000","last_fullsized":"2024-03-06T07:26:19.726106+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:26:58.476276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.732090+0000","last_change":"2024-03-06T07:26:19.728430+0000","last_active":"2024-03-06T07:26:19.732090+0000","last_peered":"2024-03-06T07:26:19.732090+0000","last_clean":"2024-03-06T07:26:19.732090+0000","last_became_active":"2024-03-06T07:26:19.728192+0000","last_became_peered":"2024-03-06T07:26:19.728192+0000","last_unstale":"2024-03-06T07:26:19.732090+0000","last_undegraded":"2024-03-06T07:26:19.732090+0000","last_fullsized":"2024-03-06T07:26:19.732090+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:04.601612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.724910+0000","last_change":"2024-03-06T07:26:18.724249+0000","last_active":"2024-03-06T07:26:19.724910+0000","last_peered":"2024-03-06T07:26:19.724910+0000","last_clean":"2024-03-06T07:26:19.724910+0000","last_became_active":"2024-03-06T07:26:18.723977+0000","last_became_peered":"2024-03-06T07:26:18.723977+0000","last_unstale":"2024-03-06T07:26:19.724910+0000","last_undegraded":"2024-03-06T07:26:19.724910+0000","last_fullsized":"2024-03-06T07:26:19.724910+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:52:10.734434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.728929+0000","last_change":"2024-03-06T07:26:19.727115+0000","last_active":"2024-03-06T07:26:19.728929+0000","last_peered":"2024-03-06T07:26:19.728929+0000","last_clean":"2024-03-06T07:26:19.728929+0000","last_became_active":"2024-03-06T07:26:19.726801+0000","last_became_peered":"2024-03-06T07:26:19.726801+0000","last_unstale":"2024-03-06T07:26:19.728929+0000","last_undegraded":"2024-03-06T07:26:19.728929+0000","last_fullsized":"2024-03-06T07:26:19.728929+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:54.623542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.722761+0000","last_change":"2024-03-06T07:26:18.723149+0000","last_active":"2024-03-06T07:26:19.722761+0000","last_peered":"2024-03-06T07:26:19.722761+0000","last_clean":"2024-03-06T07:26:19.722761+0000","last_became_active":"2024-03-06T07:26:18.722892+0000","last_became_peered":"2024-03-06T07:26:18.722892+0000","last_unstale":"2024-03-06T07:26:19.722761+0000","last_undegraded":"2024-03-06T07:26:19.722761+0000","last_fullsized":"2024-03-06T07:26:19.722761+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:36:42.959958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.727137+0000","last_change":"2024-03-06T07:26:19.723523+0000","last_active":"2024-03-06T07:26:19.727137+0000","last_peered":"2024-03-06T07:26:19.727137+0000","last_clean":"2024-03-06T07:26:19.727137+0000","last_became_active":"2024-03-06T07:26:19.723232+0000","last_became_peered":"2024-03-06T07:26:19.723232+0000","last_unstale":"2024-03-06T07:26:19.727137+0000","last_undegraded":"2024-03-06T07:26:19.727137+0000","last_fullsized":"2024-03-06T07:26:19.727137+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:36:43.969398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"93'3","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.739949+0000","last_change":"2024-03-06T07:26:18.728852+0000","last_active":"2024-03-06T07:26:19.739949+0000","last_peered":"2024-03-06T07:26:19.739949+0000","last_clean":"2024-03-06T07:26:19.739949+0000","last_became_active":"2024-03-06T07:26:18.728539+0000","last_became_peered":"2024-03-06T07:26:18.728539+0000","last_unstale":"2024-03-06T07:26:19.739949+0000","last_undegraded":"2024-03-06T07:26:19.739949+0000","last_fullsized":"2024-03-06T07:26:19.739949+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:16.540921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.736134+0000","last_change":"2024-03-06T07:26:19.732572+0000","last_active":"2024-03-06T07:26:19.736134+0000","last_peered":"2024-03-06T07:26:19.736134+0000","last_clean":"2024-03-06T07:26:19.736134+0000","last_became_active":"2024-03-06T07:26:19.732306+0000","last_became_peered":"2024-03-06T07:26:19.732306+0000","last_unstale":"2024-03-06T07:26:19.736134+0000","last_undegraded":"2024-03-06T07:26:19.736134+0000","last_fullsized":"2024-03-06T07:26:19.736134+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:04.601612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"93'6","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.744342+0000","last_change":"2024-03-06T07:26:18.726061+0000","last_active":"2024-03-06T07:26:19.744342+0000","last_peered":"2024-03-06T07:26:19.744342+0000","last_clean":"2024-03-06T07:26:19.744342+0000","last_became_active":"2024-03-06T07:26:18.725791+0000","last_became_peered":"2024-03-06T07:26:18.725791+0000","last_unstale":"2024-03-06T07:26:19.744342+0000","last_undegraded":"2024-03-06T07:26:19.744342+0000","last_fullsized":"2024-03-06T07:26:19.744342+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:38:12.922468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.727660+0000","last_change":"2024-03-06T07:26:19.726823+0000","last_active":"2024-03-06T07:26:19.727660+0000","last_peered":"2024-03-06T07:26:19.727660+0000","last_clean":"2024-03-06T07:26:19.727660+0000","last_became_active":"2024-03-06T07:26:19.726453+0000","last_became_peered":"2024-03-06T07:26:19.726453+0000","last_unstale":"2024-03-06T07:26:19.727660+0000","last_undegraded":"2024-03-06T07:26:19.727660+0000","last_fullsized":"2024-03-06T07:26:19.727660+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:30.017299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"93'2","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.739773+0000","last_change":"2024-03-06T07:26:18.726135+0000","last_active":"2024-03-06T07:26:19.739773+0000","last_peered":"2024-03-06T07:26:19.739773+0000","last_clean":"2024-03-06T07:26:19.739773+0000","last_became_active":"2024-03-06T07:26:18.725858+0000","last_became_peered":"2024-03-06T07:26:18.725858+0000","last_unstale":"2024-03-06T07:26:19.739773+0000","last_undegraded":"2024-03-06T07:26:19.739773+0000","last_fullsized":"2024-03-06T07:26:19.739773+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:47:34.287913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.742918+0000","last_change":"2024-03-06T07:26:19.741214+0000","last_active":"2024-03-06T07:26:19.742918+0000","last_peered":"2024-03-06T07:26:19.742918+0000","last_clean":"2024-03-06T07:26:19.742918+0000","last_became_active":"2024-03-06T07:26:19.740963+0000","last_became_peered":"2024-03-06T07:26:19.740963+0000","last_unstale":"2024-03-06T07:26:19.742918+0000","last_undegraded":"2024-03-06T07:26:19.742918+0000","last_fullsized":"2024-03-06T07:26:19.742918+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:50.991334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.719461+0000","last_change":"2024-03-06T07:26:18.725543+0000","last_active":"2024-03-06T07:26:19.719461+0000","last_peered":"2024-03-06T07:26:19.719461+0000","last_clean":"2024-03-06T07:26:19.719461+0000","last_became_active":"2024-03-06T07:26:18.725302+0000","last_became_peered":"2024-03-06T07:26:18.725302+0000","last_unstale":"2024-03-06T07:26:19.719461+0000","last_undegraded":"2024-03-06T07:26:19.719461+0000","last_fullsized":"2024-03-06T07:26:19.719461+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:06.822178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.731318+0000","last_change":"2024-03-06T07:26:19.728351+0000","last_active":"2024-03-06T07:26:19.731318+0000","last_peered":"2024-03-06T07:26:19.731318+0000","last_clean":"2024-03-06T07:26:19.731318+0000","last_became_active":"2024-03-06T07:26:19.726177+0000","last_became_peered":"2024-03-06T07:26:19.726177+0000","last_unstale":"2024-03-06T07:26:19.731318+0000","last_undegraded":"2024-03-06T07:26:19.731318+0000","last_fullsized":"2024-03-06T07:26:19.731318+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:11:14.556391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.716552+0000","last_change":"2024-03-06T07:26:18.727884+0000","last_active":"2024-03-06T07:26:19.716552+0000","last_peered":"2024-03-06T07:26:19.716552+0000","last_clean":"2024-03-06T07:26:19.716552+0000","last_became_active":"2024-03-06T07:26:18.727582+0000","last_became_peered":"2024-03-06T07:26:18.727582+0000","last_unstale":"2024-03-06T07:26:19.716552+0000","last_undegraded":"2024-03-06T07:26:19.716552+0000","last_fullsized":"2024-03-06T07:26:19.716552+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:29.117687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.736386+0000","last_change":"2024-03-06T07:26:19.730980+0000","last_active":"2024-03-06T07:26:19.736386+0000","last_peered":"2024-03-06T07:26:19.736386+0000","last_clean":"2024-03-06T07:26:19.736386+0000","last_became_active":"2024-03-06T07:26:19.730687+0000","last_became_peered":"2024-03-06T07:26:19.730687+0000","last_unstale":"2024-03-06T07:26:19.736386+0000","last_undegraded":"2024-03-06T07:26:19.736386+0000","last_fullsized":"2024-03-06T07:26:19.736386+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:37.811558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"93'8","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.742730+0000","last_change":"2024-03-06T07:26:18.724919+0000","last_active":"2024-03-06T07:26:19.742730+0000","last_peered":"2024-03-06T07:26:19.742730+0000","last_clean":"2024-03-06T07:26:19.742730+0000","last_became_active":"2024-03-06T07:26:18.724649+0000","last_became_peered":"2024-03-06T07:26:18.724649+0000","last_unstale":"2024-03-06T07:26:19.742730+0000","last_undegraded":"2024-03-06T07:26:19.742730+0000","last_fullsized":"2024-03-06T07:26:19.742730+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:52:07.722978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.728628+0000","last_change":"2024-03-06T07:26:19.725595+0000","last_active":"2024-03-06T07:26:19.728628+0000","last_peered":"2024-03-06T07:26:19.728628+0000","last_clean":"2024-03-06T07:26:19.728628+0000","last_became_active":"2024-03-06T07:26:19.723788+0000","last_became_peered":"2024-03-06T07:26:19.723788+0000","last_unstale":"2024-03-06T07:26:19.728628+0000","last_undegraded":"2024-03-06T07:26:19.728628+0000","last_fullsized":"2024-03-06T07:26:19.728628+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:11:14.556391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"93'2","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.740647+0000","last_change":"2024-03-06T07:26:18.727027+0000","last_active":"2024-03-06T07:26:19.740647+0000","last_peered":"2024-03-06T07:26:19.740647+0000","last_clean":"2024-03-06T07:26:19.740647+0000","last_became_active":"2024-03-06T07:26:18.726712+0000","last_became_peered":"2024-03-06T07:26:18.726712+0000","last_unstale":"2024-03-06T07:26:19.740647+0000","last_undegraded":"2024-03-06T07:26:19.740647+0000","last_fullsized":"2024-03-06T07:26:19.740647+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:52.931110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.731258+0000","last_change":"2024-03-06T07:26:19.727130+0000","last_active":"2024-03-06T07:26:19.731258+0000","last_peered":"2024-03-06T07:26:19.731258+0000","last_clean":"2024-03-06T07:26:19.731258+0000","last_became_active":"2024-03-06T07:26:19.726842+0000","last_became_peered":"2024-03-06T07:26:19.726842+0000","last_unstale":"2024-03-06T07:26:19.731258+0000","last_undegraded":"2024-03-06T07:26:19.731258+0000","last_fullsized":"2024-03-06T07:26:19.731258+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:12:20.891606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"93'1","reported_seq":13,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.736922+0000","last_change":"2024-03-06T07:26:18.725982+0000","last_active":"2024-03-06T07:26:19.736922+0000","last_peered":"2024-03-06T07:26:19.736922+0000","last_clean":"2024-03-06T07:26:19.736922+0000","last_became_active":"2024-03-06T07:26:18.724772+0000","last_became_peered":"2024-03-06T07:26:18.724772+0000","last_unstale":"2024-03-06T07:26:19.736922+0000","last_undegraded":"2024-03-06T07:26:19.736922+0000","last_fullsized":"2024-03-06T07:26:19.736922+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:13:09.899872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.728622+0000","last_change":"2024-03-06T07:26:19.726585+0000","last_active":"2024-03-06T07:26:19.728622+0000","last_peered":"2024-03-06T07:26:19.728622+0000","last_clean":"2024-03-06T07:26:19.728622+0000","last_became_active":"2024-03-06T07:26:19.726302+0000","last_became_peered":"2024-03-06T07:26:19.726302+0000","last_unstale":"2024-03-06T07:26:19.728622+0000","last_undegraded":"2024-03-06T07:26:19.728622+0000","last_fullsized":"2024-03-06T07:26:19.728622+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:56.348682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.714921+0000","last_change":"2024-03-06T07:26:18.729197+0000","last_active":"2024-03-06T07:26:19.714921+0000","last_peered":"2024-03-06T07:26:19.714921+0000","last_clean":"2024-03-06T07:26:19.714921+0000","last_became_active":"2024-03-06T07:26:18.728584+0000","last_became_peered":"2024-03-06T07:26:18.728584+0000","last_unstale":"2024-03-06T07:26:19.714921+0000","last_undegraded":"2024-03-06T07:26:19.714921+0000","last_fullsized":"2024-03-06T07:26:19.714921+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:29.007859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.736710+0000","last_change":"2024-03-06T07:26:19.725650+0000","last_active":"2024-03-06T07:26:19.736710+0000","last_peered":"2024-03-06T07:26:19.736710+0000","last_clean":"2024-03-06T07:26:19.736710+0000","last_became_active":"2024-03-06T07:26:19.725402+0000","last_became_peered":"2024-03-06T07:26:19.725402+0000","last_unstale":"2024-03-06T07:26:19.736710+0000","last_undegraded":"2024-03-06T07:26:19.736710+0000","last_fullsized":"2024-03-06T07:26:19.736710+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:10:31.101383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"93'2","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.737395+0000","last_change":"2024-03-06T07:26:18.728232+0000","last_active":"2024-03-06T07:26:19.737395+0000","last_peered":"2024-03-06T07:26:19.737395+0000","last_clean":"2024-03-06T07:26:19.737395+0000","last_became_active":"2024-03-06T07:26:18.728019+0000","last_became_peered":"2024-03-06T07:26:18.728019+0000","last_unstale":"2024-03-06T07:26:19.737395+0000","last_undegraded":"2024-03-06T07:26:19.737395+0000","last_fullsized":"2024-03-06T07:26:19.737395+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:53.614103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.738587+0000","last_change":"2024-03-06T07:26:19.732170+0000","last_active":"2024-03-06T07:26:19.738587+0000","last_peered":"2024-03-06T07:26:19.738587+0000","last_clean":"2024-03-06T07:26:19.738587+0000","last_became_active":"2024-03-06T07:26:19.731374+0000","last_became_peered":"2024-03-06T07:26:19.731374+0000","last_unstale":"2024-03-06T07:26:19.738587+0000","last_undegraded":"2024-03-06T07:26:19.738587+0000","last_fullsized":"2024-03-06T07:26:19.738587+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:16.131164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.720659+0000","last_change":"2024-03-06T07:26:18.727063+0000","last_active":"2024-03-06T07:26:19.720659+0000","last_peered":"2024-03-06T07:26:19.720659+0000","last_clean":"2024-03-06T07:26:19.720659+0000","last_became_active":"2024-03-06T07:26:18.726585+0000","last_became_peered":"2024-03-06T07:26:18.726585+0000","last_unstale":"2024-03-06T07:26:19.720659+0000","last_undegraded":"2024-03-06T07:26:19.720659+0000","last_fullsized":"2024-03-06T07:26:19.720659+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:51.405230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.731778+0000","last_change":"2024-03-06T07:26:19.723739+0000","last_active":"2024-03-06T07:26:19.731778+0000","last_peered":"2024-03-06T07:26:19.731778+0000","last_clean":"2024-03-06T07:26:19.731778+0000","last_became_active":"2024-03-06T07:26:19.723457+0000","last_became_peered":"2024-03-06T07:26:19.723457+0000","last_unstale":"2024-03-06T07:26:19.731778+0000","last_undegraded":"2024-03-06T07:26:19.731778+0000","last_fullsized":"2024-03-06T07:26:19.731778+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:56.186991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.720717+0000","last_change":"2024-03-06T07:26:18.730150+0000","last_active":"2024-03-06T07:26:19.720717+0000","last_peered":"2024-03-06T07:26:19.720717+0000","last_clean":"2024-03-06T07:26:19.720717+0000","last_became_active":"2024-03-06T07:26:18.729809+0000","last_became_peered":"2024-03-06T07:26:18.729809+0000","last_unstale":"2024-03-06T07:26:19.720717+0000","last_undegraded":"2024-03-06T07:26:19.720717+0000","last_fullsized":"2024-03-06T07:26:19.720717+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:42.398745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.736363+0000","last_change":"2024-03-06T07:26:19.732240+0000","last_active":"2024-03-06T07:26:19.736363+0000","last_peered":"2024-03-06T07:26:19.736363+0000","last_clean":"2024-03-06T07:26:19.736363+0000","last_became_active":"2024-03-06T07:26:19.731971+0000","last_became_peered":"2024-03-06T07:26:19.731971+0000","last_unstale":"2024-03-06T07:26:19.736363+0000","last_undegraded":"2024-03-06T07:26:19.736363+0000","last_fullsized":"2024-03-06T07:26:19.736363+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:53:35.777661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.714072+0000","last_change":"2024-03-06T07:26:18.722623+0000","last_active":"2024-03-06T07:26:19.714072+0000","last_peered":"2024-03-06T07:26:19.714072+0000","last_clean":"2024-03-06T07:26:19.714072+0000","last_became_active":"2024-03-06T07:26:18.722379+0000","last_became_peered":"2024-03-06T07:26:18.722379+0000","last_unstale":"2024-03-06T07:26:19.714072+0000","last_undegraded":"2024-03-06T07:26:19.714072+0000","last_fullsized":"2024-03-06T07:26:19.714072+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:52.931110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.732051+0000","last_change":"2024-03-06T07:26:19.727054+0000","last_active":"2024-03-06T07:26:19.732051+0000","last_peered":"2024-03-06T07:26:19.732051+0000","last_clean":"2024-03-06T07:26:19.732051+0000","last_became_active":"2024-03-06T07:26:19.726755+0000","last_became_peered":"2024-03-06T07:26:19.726755+0000","last_unstale":"2024-03-06T07:26:19.732051+0000","last_undegraded":"2024-03-06T07:26:19.732051+0000","last_fullsized":"2024-03-06T07:26:19.732051+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:08.203747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"93'2","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.739854+0000","last_change":"2024-03-06T07:26:18.728905+0000","last_active":"2024-03-06T07:26:19.739854+0000","last_peered":"2024-03-06T07:26:19.739854+0000","last_clean":"2024-03-06T07:26:19.739854+0000","last_became_active":"2024-03-06T07:26:18.728587+0000","last_became_peered":"2024-03-06T07:26:18.728587+0000","last_unstale":"2024-03-06T07:26:19.739854+0000","last_undegraded":"2024-03-06T07:26:19.739854+0000","last_fullsized":"2024-03-06T07:26:19.739854+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:14:35.299761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.742874+0000","last_change":"2024-03-06T07:26:19.740512+0000","last_active":"2024-03-06T07:26:19.742874+0000","last_peered":"2024-03-06T07:26:19.742874+0000","last_clean":"2024-03-06T07:26:19.742874+0000","last_became_active":"2024-03-06T07:26:19.740284+0000","last_became_peered":"2024-03-06T07:26:19.740284+0000","last_unstale":"2024-03-06T07:26:19.742874+0000","last_undegraded":"2024-03-06T07:26:19.742874+0000","last_fullsized":"2024-03-06T07:26:19.742874+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:59.857690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.715749+0000","last_change":"2024-03-06T07:26:18.727585+0000","last_active":"2024-03-06T07:26:19.715749+0000","last_peered":"2024-03-06T07:26:19.715749+0000","last_clean":"2024-03-06T07:26:19.715749+0000","last_became_active":"2024-03-06T07:26:18.727315+0000","last_became_peered":"2024-03-06T07:26:18.727315+0000","last_unstale":"2024-03-06T07:26:19.715749+0000","last_undegraded":"2024-03-06T07:26:19.715749+0000","last_fullsized":"2024-03-06T07:26:19.715749+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:53:34.768221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.741979+0000","last_change":"2024-03-06T07:26:19.733995+0000","last_active":"2024-03-06T07:26:19.741979+0000","last_peered":"2024-03-06T07:26:19.741979+0000","last_clean":"2024-03-06T07:26:19.741979+0000","last_became_active":"2024-03-06T07:26:19.733749+0000","last_became_peered":"2024-03-06T07:26:19.733749+0000","last_unstale":"2024-03-06T07:26:19.741979+0000","last_undegraded":"2024-03-06T07:26:19.741979+0000","last_fullsized":"2024-03-06T07:26:19.741979+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:54:21.900420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.714613+0000","last_change":"2024-03-06T07:26:18.729239+0000","last_active":"2024-03-06T07:26:19.714613+0000","last_peered":"2024-03-06T07:26:19.714613+0000","last_clean":"2024-03-06T07:26:19.714613+0000","last_became_active":"2024-03-06T07:26:18.728737+0000","last_became_peered":"2024-03-06T07:26:18.728737+0000","last_unstale":"2024-03-06T07:26:19.714613+0000","last_undegraded":"2024-03-06T07:26:19.714613+0000","last_fullsized":"2024-03-06T07:26:19.714613+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:16.119338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.727103+0000","last_change":"2024-03-06T07:26:19.724422+0000","last_active":"2024-03-06T07:26:19.727103+0000","last_peered":"2024-03-06T07:26:19.727103+0000","last_clean":"2024-03-06T07:26:19.727103+0000","last_became_active":"2024-03-06T07:26:19.724072+0000","last_became_peered":"2024-03-06T07:26:19.724072+0000","last_unstale":"2024-03-06T07:26:19.727103+0000","last_undegraded":"2024-03-06T07:26:19.727103+0000","last_fullsized":"2024-03-06T07:26:19.727103+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:17.128778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.715963+0000","last_change":"2024-03-06T07:26:18.728248+0000","last_active":"2024-03-06T07:26:19.715963+0000","last_peered":"2024-03-06T07:26:19.715963+0000","last_clean":"2024-03-06T07:26:19.715963+0000","last_became_active":"2024-03-06T07:26:18.727982+0000","last_became_peered":"2024-03-06T07:26:18.727982+0000","last_unstale":"2024-03-06T07:26:19.715963+0000","last_undegraded":"2024-03-06T07:26:19.715963+0000","last_fullsized":"2024-03-06T07:26:19.715963+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:38:12.922468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.736729+0000","last_change":"2024-03-06T07:26:19.730542+0000","last_active":"2024-03-06T07:26:19.736729+0000","last_peered":"2024-03-06T07:26:19.736729+0000","last_clean":"2024-03-06T07:26:19.736729+0000","last_became_active":"2024-03-06T07:26:19.729923+0000","last_became_peered":"2024-03-06T07:26:19.729923+0000","last_unstale":"2024-03-06T07:26:19.736729+0000","last_undegraded":"2024-03-06T07:26:19.736729+0000","last_fullsized":"2024-03-06T07:26:19.736729+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:54:21.900420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.715587+0000","last_change":"2024-03-06T07:26:18.728166+0000","last_active":"2024-03-06T07:26:19.715587+0000","last_peered":"2024-03-06T07:26:19.715587+0000","last_clean":"2024-03-06T07:26:19.715587+0000","last_became_active":"2024-03-06T07:26:18.727886+0000","last_became_peered":"2024-03-06T07:26:18.727886+0000","last_unstale":"2024-03-06T07:26:19.715587+0000","last_undegraded":"2024-03-06T07:26:19.715587+0000","last_fullsized":"2024-03-06T07:26:19.715587+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:11:13.546951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.723489+0000","last_change":"2024-03-06T07:26:19.720690+0000","last_active":"2024-03-06T07:26:19.723489+0000","last_peered":"2024-03-06T07:26:19.723489+0000","last_clean":"2024-03-06T07:26:19.723489+0000","last_became_active":"2024-03-06T07:26:19.720264+0000","last_became_peered":"2024-03-06T07:26:19.720264+0000","last_unstale":"2024-03-06T07:26:19.723489+0000","last_undegraded":"2024-03-06T07:26:19.723489+0000","last_fullsized":"2024-03-06T07:26:19.723489+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:13:10.909312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.715801+0000","last_change":"2024-03-06T07:26:18.727030+0000","last_active":"2024-03-06T07:26:19.715801+0000","last_peered":"2024-03-06T07:26:19.715801+0000","last_clean":"2024-03-06T07:26:19.715801+0000","last_became_active":"2024-03-06T07:26:18.725274+0000","last_became_peered":"2024-03-06T07:26:18.725274+0000","last_unstale":"2024-03-06T07:26:19.715801+0000","last_undegraded":"2024-03-06T07:26:19.715801+0000","last_fullsized":"2024-03-06T07:26:19.715801+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:52.931110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.736042+0000","last_change":"2024-03-06T07:26:19.727321+0000","last_active":"2024-03-06T07:26:19.736042+0000","last_peered":"2024-03-06T07:26:19.736042+0000","last_clean":"2024-03-06T07:26:19.736042+0000","last_became_active":"2024-03-06T07:26:19.726619+0000","last_became_peered":"2024-03-06T07:26:19.726619+0000","last_unstale":"2024-03-06T07:26:19.736042+0000","last_undegraded":"2024-03-06T07:26:19.736042+0000","last_fullsized":"2024-03-06T07:26:19.736042+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:26:59.485716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.720753+0000","last_change":"2024-03-06T07:26:18.723615+0000","last_active":"2024-03-06T07:26:19.720753+0000","last_peered":"2024-03-06T07:26:19.720753+0000","last_clean":"2024-03-06T07:26:19.720753+0000","last_became_active":"2024-03-06T07:26:18.723349+0000","last_became_peered":"2024-03-06T07:26:18.723349+0000","last_unstale":"2024-03-06T07:26:19.720753+0000","last_undegraded":"2024-03-06T07:26:19.720753+0000","last_fullsized":"2024-03-06T07:26:19.720753+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:13:09.899872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.727449+0000","last_change":"2024-03-06T07:26:19.725056+0000","last_active":"2024-03-06T07:26:19.727449+0000","last_peered":"2024-03-06T07:26:19.727449+0000","last_clean":"2024-03-06T07:26:19.727449+0000","last_became_active":"2024-03-06T07:26:19.724803+0000","last_became_peered":"2024-03-06T07:26:19.724803+0000","last_unstale":"2024-03-06T07:26:19.727449+0000","last_undegraded":"2024-03-06T07:26:19.727449+0000","last_fullsized":"2024-03-06T07:26:19.727449+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:54.623542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.718132+0000","last_change":"2024-03-06T07:26:18.728464+0000","last_active":"2024-03-06T07:26:19.718132+0000","last_peered":"2024-03-06T07:26:19.718132+0000","last_clean":"2024-03-06T07:26:19.718132+0000","last_became_active":"2024-03-06T07:26:18.728202+0000","last_became_peered":"2024-03-06T07:26:18.728202+0000","last_unstale":"2024-03-06T07:26:19.718132+0000","last_undegraded":"2024-03-06T07:26:19.718132+0000","last_fullsized":"2024-03-06T07:26:19.718132+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:04:30.366156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.731749+0000","last_change":"2024-03-06T07:26:19.725026+0000","last_active":"2024-03-06T07:26:19.731749+0000","last_peered":"2024-03-06T07:26:19.731749+0000","last_clean":"2024-03-06T07:26:19.731749+0000","last_became_active":"2024-03-06T07:26:19.724759+0000","last_became_peered":"2024-03-06T07:26:19.724759+0000","last_unstale":"2024-03-06T07:26:19.731749+0000","last_undegraded":"2024-03-06T07:26:19.731749+0000","last_fullsized":"2024-03-06T07:26:19.731749+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:24:12.521477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"93'1","reported_seq":13,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.738800+0000","last_change":"2024-03-06T07:26:18.728873+0000","last_active":"2024-03-06T07:26:19.738800+0000","last_peered":"2024-03-06T07:26:19.738800+0000","last_clean":"2024-03-06T07:26:19.738800+0000","last_became_active":"2024-03-06T07:26:18.728290+0000","last_became_peered":"2024-03-06T07:26:18.728290+0000","last_unstale":"2024-03-06T07:26:19.738800+0000","last_undegraded":"2024-03-06T07:26:19.738800+0000","last_fullsized":"2024-03-06T07:26:19.738800+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:53.614103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":303,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.720104+0000","last_change":"2024-03-06T07:21:47.553693+0000","last_active":"2024-03-06T07:26:19.720104+0000","last_peered":"2024-03-06T07:26:19.720104+0000","last_clean":"2024-03-06T07:26:19.720104+0000","last_became_active":"2024-03-06T07:21:47.552664+0000","last_became_peered":"2024-03-06T07:21:47.552664+0000","last_unstale":"2024-03-06T07:26:19.720104+0000","last_undegraded":"2024-03-06T07:26:19.720104+0000","last_fullsized":"2024-03-06T07:26:19.720104+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_clean_scrub_stamp":"2024-03-06T07:21:46.362028+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:02.916426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574908,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32900,"kb_used_data":1712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338940,"statfs":{"total":96636764160,"available":96603074560,"internally_reserved":0,"allocated":1753088,"data_stored":1080864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574906,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32296,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339544,"statfs":{"total":96636764160,"available":96603693056,"internally_reserved":0,"allocated":1138688,"data_stored":483609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574907,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32316,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339524,"statfs":{"total":96636764160,"available":96603672576,"internally_reserved":0,"allocated":1155072,"data_stored":490454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574908,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32296,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339544,"statfs":{"total":96636764160,"available":96603693056,"internally_reserved":0,"allocated":1138688,"data_stored":483609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32328,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339512,"statfs":{"total":96636764160,"available":96603660288,"internally_reserved":0,"allocated":1167360,"data_stored":492018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574907,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32316,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339524,"statfs":{"total":96636764160,"available":96603672576,"internally_reserved":0,"allocated":1155072,"data_stored":490418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574908,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32312,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339528,"statfs":{"total":96636764160,"available":96603676672,"internally_reserved":0,"allocated":1150976,"data_stored":490384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32892,"kb_used_data":1708,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338948,"statfs":{"total":96636764160,"available":96603082752,"internally_reserved":0,"allocated":1748992,"data_stored":1081334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:26:22.734 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:26:22.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.873+0000 7f44f3239640 1 Processor -- start 2024-03-06T07:26:22.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.873+0000 7f44f3239640 1 -- start start 2024-03-06T07:26:22.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.874+0000 7f44f3239640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44ec0d2710 0x7f44ec0b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.874+0000 7f44f3239640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44ec0b3430 con 0x7f44ec15dfe0 2024-03-06T07:26:22.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.874+0000 7f44f3239640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44ec0b3710 con 0x7f44ec0d22b0 2024-03-06T07:26:22.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.874+0000 7f44f3239640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44ec0b31e0 con 0x7f44ec0d2710 2024-03-06T07:26:22.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.874+0000 7f44f0fd5640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f44ec15dfe0 0x7f44ec0cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:35426/0 (socket says 172.21.15.83:35426) 2024-03-06T07:26:22.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.874+0000 7f44f0fd5640 1 -- 172.21.15.83:0/3970812837 learned_addr learned my addr 172.21.15.83:0/3970812837 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:22.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.875+0000 7f44f17d6640 1 --2- 172.21.15.83:0/3970812837 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44ec0d2710 0x7f44ec0b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.875+0000 7f44eb7fe640 1 -- 172.21.15.83:0/3970812837 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 911548573 0 0) 0x7f44ec0b3430 con 0x7f44ec15dfe0 2024-03-06T07:26:22.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.875+0000 7f44eb7fe640 1 -- 172.21.15.83:0/3970812837 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44d4003520 con 0x7f44ec15dfe0 2024-03-06T07:26:22.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.875+0000 7f44eb7fe640 1 -- 172.21.15.83:0/3970812837 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4068429032 0 0) 0x7f44ec0b3710 con 0x7f44ec0d22b0 2024-03-06T07:26:22.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.875+0000 7f44eb7fe640 1 -- 172.21.15.83:0/3970812837 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44ec0b3430 con 0x7f44ec0d22b0 2024-03-06T07:26:22.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.875+0000 7f44eb7fe640 1 -- 172.21.15.83:0/3970812837 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 797682759 0 0) 0x7f44d4003520 con 0x7f44ec15dfe0 2024-03-06T07:26:22.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.875+0000 7f44eb7fe640 1 -- 172.21.15.83:0/3970812837 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44ec0d2710 msgr2=0x7f44ec0b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:22.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.875+0000 7f44eb7fe640 1 --2- 172.21.15.83:0/3970812837 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44ec0d2710 0x7f44ec0b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.875+0000 7f44eb7fe640 1 -- 172.21.15.83:0/3970812837 >> v1:172.21.15.183:6789/0 conn(0x7f44ec0d22b0 legacy=0x7f44ec0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.875+0000 7f44eb7fe640 1 -- 172.21.15.83:0/3970812837 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44ec0b39f0 con 0x7f44ec15dfe0 2024-03-06T07:26:22.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.875+0000 7f44eb7fe640 1 -- 172.21.15.83:0/3970812837 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f44dc001670 con 0x7f44ec15dfe0 2024-03-06T07:26:22.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.875+0000 7f44f17d6640 1 --2- 172.21.15.83:0/3970812837 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44ec0d2710 0x7f44ec0b4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:22.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.875+0000 7f44eb7fe640 1 -- 172.21.15.83:0/3970812837 >> v1:172.21.15.83:6789/0 conn(0x7f44ec15dfe0 legacy=0x7f44ec0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.875+0000 7f44eb7fe640 1 --2- 172.21.15.83:0/3970812837 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44ec0d2710 0x7f44d4003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.875+0000 7f44eb7fe640 1 --2- 172.21.15.83:0/3970812837 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f44d4004100 0x7f44d4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.875+0000 7f44eb7fe640 1 --2- 172.21.15.83:0/3970812837 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44d4004d40 0x7f44d40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.875+0000 7f44eb7fe640 1 -- 172.21.15.83:0/3970812837 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44ec0b31e0 con 0x7f44d4004d40 2024-03-06T07:26:22.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.875+0000 7f44eb7fe640 1 -- 172.21.15.83:0/3970812837 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d400bad0 con 0x7f44d4004100 2024-03-06T07:26:22.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.875+0000 7f44eb7fe640 1 -- 172.21.15.83:0/3970812837 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44d400bcd0 con 0x7f44ec0d2710 2024-03-06T07:26:22.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.875+0000 7f44f17d6640 1 --2- 172.21.15.83:0/3970812837 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44d4004d40 0x7f44d40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.875+0000 7f44f0fd5640 1 --2- 172.21.15.83:0/3970812837 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44ec0d2710 0x7f44d4003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.875+0000 7f44ebfff640 1 --2- 172.21.15.83:0/3970812837 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f44d4004100 0x7f44d4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.875+0000 7f44f17d6640 1 -- 172.21.15.83:0/3970812837 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44ec0d2710 msgr2=0x7f44d4003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:22.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.875+0000 7f44f17d6640 1 --2- 172.21.15.83:0/3970812837 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44ec0d2710 0x7f44d4003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.875+0000 7f44f17d6640 1 -- 172.21.15.83:0/3970812837 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f44d4004100 msgr2=0x7f44d4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:22.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.875+0000 7f44f17d6640 1 --2- 172.21.15.83:0/3970812837 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f44d4004100 0x7f44d4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.875+0000 7f44f17d6640 1 -- 172.21.15.83:0/3970812837 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d4003520 con 0x7f44d4004d40 2024-03-06T07:26:22.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.876+0000 7f44f17d6640 1 --2- 172.21.15.83:0/3970812837 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44d4004d40 0x7f44d40051c0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f44e400e320 tx=0x7f44d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d5afa2f4a54690f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.876+0000 7f44eb7fe640 1 -- 172.21.15.83:0/3970812837 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e4039070 con 0x7f44d4004d40 2024-03-06T07:26:22.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.876+0000 7f44eb7fe640 1 -- 172.21.15.83:0/3970812837 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44e402cd60 con 0x7f44d4004d40 2024-03-06T07:26:22.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.876+0000 7f44eb7fe640 1 -- 172.21.15.83:0/3970812837 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e4009de0 con 0x7f44d4004d40 2024-03-06T07:26:22.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.877+0000 7f44f3239640 1 -- 172.21.15.83:0/3970812837 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44d4004d40 msgr2=0x7f44d40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.877+0000 7f44f3239640 1 --2- 172.21.15.83:0/3970812837 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44d4004d40 0x7f44d40051c0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f44e400e320 tx=0x7f44d4005830 comp rx=0 tx=0).stop 2024-03-06T07:26:22.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.877+0000 7f44f17d6640 1 -- 172.21.15.83:0/3970812837 reap_dead start 2024-03-06T07:26:22.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.877+0000 7f44f3239640 1 -- 172.21.15.83:0/3970812837 shutdown_connections 2024-03-06T07:26:22.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.877+0000 7f44f3239640 1 -- 172.21.15.83:0/3970812837 >> 172.21.15.83:0/3970812837 conn(0x7f44ec0d3cd0 msgr2=0x7f44ec0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:22.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.877+0000 7f44f3239640 1 -- 172.21.15.83:0/3970812837 shutdown_connections 2024-03-06T07:26:22.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.877+0000 7f44f3239640 1 -- 172.21.15.83:0/3970812837 wait complete. 2024-03-06T07:26:22.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.879+0000 7f44f3239640 1 Processor -- start 2024-03-06T07:26:22.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.879+0000 7f44f3239640 1 -- start start 2024-03-06T07:26:22.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.880+0000 7f44f3239640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f44d4004d40 0x7f44ec158220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.880+0000 7f44f3239640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44ec158a60 0x7f44ec158ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.880+0000 7f44f3239640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44ec0d40e0 0x7f44ec0d4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.880+0000 7f44f3239640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44ec0cfa00 con 0x7f44ec0d40e0 2024-03-06T07:26:22.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.880+0000 7f44f3239640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44ec0b3d60 con 0x7f44d4004d40 2024-03-06T07:26:22.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.880+0000 7f44f3239640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44ec0d1c70 con 0x7f44ec158a60 2024-03-06T07:26:22.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.880+0000 7f44f17d6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44ec0d40e0 0x7f44ec0d4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.880+0000 7f44f17d6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44ec0d40e0 0x7f44ec0d4560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50494/0 (socket says 172.21.15.83:50494) 2024-03-06T07:26:22.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.880+0000 7f44f17d6640 1 -- 172.21.15.83:0/3792242265 learned_addr learned my addr 172.21.15.83:0/3792242265 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:22.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.880+0000 7f44f0fd5640 1 --2- 172.21.15.83:0/3792242265 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f44d4004d40 0x7f44ec158220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.880+0000 7f44ebfff640 1 --2- 172.21.15.83:0/3792242265 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44ec158a60 0x7f44ec158ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.880+0000 7f44f17d6640 1 -- 172.21.15.83:0/3792242265 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44ec158a60 msgr2=0x7f44ec158ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.880+0000 7f44f17d6640 1 --2- 172.21.15.83:0/3792242265 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44ec158a60 0x7f44ec158ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.880+0000 7f44f17d6640 1 -- 172.21.15.83:0/3792242265 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f44d4004d40 msgr2=0x7f44ec158220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.880+0000 7f44f17d6640 1 --2- 172.21.15.83:0/3792242265 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f44d4004d40 0x7f44ec158220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.880+0000 7f44f17d6640 1 -- 172.21.15.83:0/3792242265 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44ec0ccc60 con 0x7f44ec0d40e0 2024-03-06T07:26:22.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.880+0000 7f44f0fd5640 1 --2- 172.21.15.83:0/3792242265 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f44d4004d40 0x7f44ec158220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:22.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.880+0000 7f44f17d6640 1 --2- 172.21.15.83:0/3792242265 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44ec0d40e0 0x7f44ec0d4560 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f44e4037040 tx=0x7f44e4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.880+0000 7f44e97fa640 1 -- 172.21.15.83:0/3792242265 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e4039070 con 0x7f44ec0d40e0 2024-03-06T07:26:22.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.880+0000 7f44e97fa640 1 -- 172.21.15.83:0/3792242265 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44e4044070 con 0x7f44ec0d40e0 2024-03-06T07:26:22.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.880+0000 7f44f3239640 1 -- 172.21.15.83:0/3792242265 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44ec0ccf40 con 0x7f44ec0d40e0 2024-03-06T07:26:22.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.880+0000 7f44e97fa640 1 -- 172.21.15.83:0/3792242265 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e4009de0 con 0x7f44ec0d40e0 2024-03-06T07:26:22.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.881+0000 7f44f3239640 1 -- 172.21.15.83:0/3792242265 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44ec0cd500 con 0x7f44ec0d40e0 2024-03-06T07:26:22.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.881+0000 7f44f3239640 1 -- 172.21.15.83:0/3792242265 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44ec04f330 con 0x7f44ec0d40e0 2024-03-06T07:26:22.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.882+0000 7f44e97fa640 1 -- 172.21.15.83:0/3792242265 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f44e4005d10 con 0x7f44ec0d40e0 2024-03-06T07:26:22.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.882+0000 7f44e97fa640 1 --2- 172.21.15.83:0/3792242265 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f44d806b4d0 0x7f44d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.882+0000 7f44e97fa640 1 -- 172.21.15.83:0/3792242265 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f44e40ac1c0 con 0x7f44ec0d40e0 2024-03-06T07:26:22.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.882+0000 7f44f0fd5640 1 --2- 172.21.15.83:0/3792242265 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f44d806b4d0 0x7f44d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.883+0000 7f44f0fd5640 1 --2- 172.21.15.83:0/3792242265 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f44d806b4d0 0x7f44d806d990 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f44ec0726b0 tx=0x7f44dc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:22.887+0000 7f44e97fa640 1 -- 172.21.15.83:0/3792242265 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f44e404e020 con 0x7f44ec0d40e0 2024-03-06T07:26:23.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.060+0000 7f44f3239640 1 -- 172.21.15.83:0/3792242265 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f44ec0b4320 con 0x7f44d806b4d0 2024-03-06T07:26:23.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.066+0000 7f44e97fa640 1 -- 172.21.15.83:0/3792242265 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252015 (secure 0 0 0) 0x7f44ec0b4320 con 0x7f44d806b4d0 2024-03-06T07:26:23.070 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:23.071 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-06T07:26:23.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.069+0000 7f44f3239640 1 -- 172.21.15.83:0/3792242265 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f44d806b4d0 msgr2=0x7f44d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.069+0000 7f44f3239640 1 --2- 172.21.15.83:0/3792242265 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f44d806b4d0 0x7f44d806d990 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f44ec0726b0 tx=0x7f44dc009040 comp rx=0 tx=0).stop 2024-03-06T07:26:23.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.069+0000 7f44f3239640 1 -- 172.21.15.83:0/3792242265 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44ec0d40e0 msgr2=0x7f44ec0d4560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.069+0000 7f44f3239640 1 --2- 172.21.15.83:0/3792242265 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44ec0d40e0 0x7f44ec0d4560 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f44e4037040 tx=0x7f44e4009750 comp rx=0 tx=0).stop 2024-03-06T07:26:23.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.069+0000 7f44f3239640 1 -- 172.21.15.83:0/3792242265 shutdown_connections 2024-03-06T07:26:23.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.069+0000 7f44f3239640 1 --2- 172.21.15.83:0/3792242265 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f44d806b4d0 0x7f44d806d990 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.069+0000 7f44f3239640 1 --2- 172.21.15.83:0/3792242265 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44ec0d40e0 0x7f44ec0d4560 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.069+0000 7f44f3239640 1 --2- 172.21.15.83:0/3792242265 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44ec158a60 0x7f44ec158ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.069+0000 7f44f3239640 1 --2- 172.21.15.83:0/3792242265 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f44d4004d40 0x7f44ec158220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.069+0000 7f44f3239640 1 -- 172.21.15.83:0/3792242265 >> 172.21.15.83:0/3792242265 conn(0x7f44ec0d3cd0 msgr2=0x7f44ec0b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:23.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.069+0000 7f44f3239640 1 -- 172.21.15.83:0/3792242265 shutdown_connections 2024-03-06T07:26:23.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.069+0000 7f44f3239640 1 -- 172.21.15.83:0/3792242265 wait complete. 2024-03-06T07:26:23.082 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":226,"stamp":"2024-03-06T07:26:22.367900+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":59,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":259656,"kb_used_data":10164,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754715064,"statfs":{"total":773094113280,"available":772828225536,"internally_reserved":0,"allocated":10407936,"data_stored":5092690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12757,"internal_metadata":255315499},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.661962"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.742127+0000","last_change":"2024-03-06T07:26:19.735768+0000","last_active":"2024-03-06T07:26:19.742127+0000","last_peered":"2024-03-06T07:26:19.742127+0000","last_clean":"2024-03-06T07:26:19.742127+0000","last_became_active":"2024-03-06T07:26:19.735493+0000","last_became_peered":"2024-03-06T07:26:19.735493+0000","last_unstale":"2024-03-06T07:26:19.742127+0000","last_undegraded":"2024-03-06T07:26:19.742127+0000","last_fullsized":"2024-03-06T07:26:19.742127+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:52:29.357695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.732073+0000","last_change":"2024-03-06T07:26:19.727695+0000","last_active":"2024-03-06T07:26:19.732073+0000","last_peered":"2024-03-06T07:26:19.732073+0000","last_clean":"2024-03-06T07:26:19.732073+0000","last_became_active":"2024-03-06T07:26:19.727435+0000","last_became_peered":"2024-03-06T07:26:19.727435+0000","last_unstale":"2024-03-06T07:26:19.732073+0000","last_undegraded":"2024-03-06T07:26:19.732073+0000","last_fullsized":"2024-03-06T07:26:19.732073+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:55.844050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.742193+0000","last_change":"2024-03-06T07:26:19.735708+0000","last_active":"2024-03-06T07:26:19.742193+0000","last_peered":"2024-03-06T07:26:19.742193+0000","last_clean":"2024-03-06T07:26:19.742193+0000","last_became_active":"2024-03-06T07:26:19.735457+0000","last_became_peered":"2024-03-06T07:26:19.735457+0000","last_unstale":"2024-03-06T07:26:19.742193+0000","last_undegraded":"2024-03-06T07:26:19.742193+0000","last_fullsized":"2024-03-06T07:26:19.742193+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:37.811558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.726497+0000","last_change":"2024-03-06T07:26:19.723522+0000","last_active":"2024-03-06T07:26:19.726497+0000","last_peered":"2024-03-06T07:26:19.726497+0000","last_clean":"2024-03-06T07:26:19.726497+0000","last_became_active":"2024-03-06T07:26:19.723265+0000","last_became_peered":"2024-03-06T07:26:19.723265+0000","last_unstale":"2024-03-06T07:26:19.726497+0000","last_undegraded":"2024-03-06T07:26:19.726497+0000","last_fullsized":"2024-03-06T07:26:19.726497+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:17.128778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.739496+0000","last_change":"2024-03-06T07:26:19.737877+0000","last_active":"2024-03-06T07:26:19.739496+0000","last_peered":"2024-03-06T07:26:19.739496+0000","last_clean":"2024-03-06T07:26:19.739496+0000","last_became_active":"2024-03-06T07:26:19.737483+0000","last_became_peered":"2024-03-06T07:26:19.737483+0000","last_unstale":"2024-03-06T07:26:19.739496+0000","last_undegraded":"2024-03-06T07:26:19.739496+0000","last_fullsized":"2024-03-06T07:26:19.739496+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:46:28.616508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.726743+0000","last_change":"2024-03-06T07:26:19.725055+0000","last_active":"2024-03-06T07:26:19.726743+0000","last_peered":"2024-03-06T07:26:19.726743+0000","last_clean":"2024-03-06T07:26:19.726743+0000","last_became_active":"2024-03-06T07:26:19.724782+0000","last_became_peered":"2024-03-06T07:26:19.724782+0000","last_unstale":"2024-03-06T07:26:19.726743+0000","last_undegraded":"2024-03-06T07:26:19.726743+0000","last_fullsized":"2024-03-06T07:26:19.726743+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:27:46.844174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.725722+0000","last_change":"2024-03-06T07:26:19.724068+0000","last_active":"2024-03-06T07:26:19.725722+0000","last_peered":"2024-03-06T07:26:19.725722+0000","last_clean":"2024-03-06T07:26:19.725722+0000","last_became_active":"2024-03-06T07:26:19.723769+0000","last_became_peered":"2024-03-06T07:26:19.723769+0000","last_unstale":"2024-03-06T07:26:19.725722+0000","last_undegraded":"2024-03-06T07:26:19.725722+0000","last_fullsized":"2024-03-06T07:26:19.725722+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:53:35.777661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.731836+0000","last_change":"2024-03-06T07:26:19.729272+0000","last_active":"2024-03-06T07:26:19.731836+0000","last_peered":"2024-03-06T07:26:19.731836+0000","last_clean":"2024-03-06T07:26:19.731836+0000","last_became_active":"2024-03-06T07:26:19.728926+0000","last_became_peered":"2024-03-06T07:26:19.728926+0000","last_unstale":"2024-03-06T07:26:19.731836+0000","last_undegraded":"2024-03-06T07:26:19.731836+0000","last_fullsized":"2024-03-06T07:26:19.731836+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:23.513679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.728910+0000","last_change":"2024-03-06T07:26:19.726431+0000","last_active":"2024-03-06T07:26:19.728910+0000","last_peered":"2024-03-06T07:26:19.728910+0000","last_clean":"2024-03-06T07:26:19.728910+0000","last_became_active":"2024-03-06T07:26:19.726113+0000","last_became_peered":"2024-03-06T07:26:19.726113+0000","last_unstale":"2024-03-06T07:26:19.728910+0000","last_undegraded":"2024-03-06T07:26:19.728910+0000","last_fullsized":"2024-03-06T07:26:19.728910+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:30.017299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.736857+0000","last_change":"2024-03-06T07:26:19.733949+0000","last_active":"2024-03-06T07:26:19.736857+0000","last_peered":"2024-03-06T07:26:19.736857+0000","last_clean":"2024-03-06T07:26:19.736857+0000","last_became_active":"2024-03-06T07:26:19.733655+0000","last_became_peered":"2024-03-06T07:26:19.733655+0000","last_unstale":"2024-03-06T07:26:19.736857+0000","last_undegraded":"2024-03-06T07:26:19.736857+0000","last_fullsized":"2024-03-06T07:26:19.736857+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:27:46.844174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.728330+0000","last_change":"2024-03-06T07:26:19.726190+0000","last_active":"2024-03-06T07:26:19.728330+0000","last_peered":"2024-03-06T07:26:19.728330+0000","last_clean":"2024-03-06T07:26:19.728330+0000","last_became_active":"2024-03-06T07:26:19.725936+0000","last_became_peered":"2024-03-06T07:26:19.725936+0000","last_unstale":"2024-03-06T07:26:19.728330+0000","last_undegraded":"2024-03-06T07:26:19.728330+0000","last_fullsized":"2024-03-06T07:26:19.728330+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:46:28.616508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.739285+0000","last_change":"2024-03-06T07:26:19.734412+0000","last_active":"2024-03-06T07:26:19.739285+0000","last_peered":"2024-03-06T07:26:19.739285+0000","last_clean":"2024-03-06T07:26:19.739285+0000","last_became_active":"2024-03-06T07:26:19.734139+0000","last_became_peered":"2024-03-06T07:26:19.734139+0000","last_unstale":"2024-03-06T07:26:19.739285+0000","last_undegraded":"2024-03-06T07:26:19.739285+0000","last_fullsized":"2024-03-06T07:26:19.739285+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:53:35.777661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.736598+0000","last_change":"2024-03-06T07:26:19.734391+0000","last_active":"2024-03-06T07:26:19.736598+0000","last_peered":"2024-03-06T07:26:19.736598+0000","last_clean":"2024-03-06T07:26:19.736598+0000","last_became_active":"2024-03-06T07:26:19.734138+0000","last_became_peered":"2024-03-06T07:26:19.734138+0000","last_unstale":"2024-03-06T07:26:19.736598+0000","last_undegraded":"2024-03-06T07:26:19.736598+0000","last_fullsized":"2024-03-06T07:26:19.736598+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:54.623542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.736745+0000","last_change":"2024-03-06T07:26:19.731014+0000","last_active":"2024-03-06T07:26:19.736745+0000","last_peered":"2024-03-06T07:26:19.736745+0000","last_clean":"2024-03-06T07:26:19.736745+0000","last_became_active":"2024-03-06T07:26:19.730286+0000","last_became_peered":"2024-03-06T07:26:19.730286+0000","last_unstale":"2024-03-06T07:26:19.736745+0000","last_undegraded":"2024-03-06T07:26:19.736745+0000","last_fullsized":"2024-03-06T07:26:19.736745+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:52:29.357695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.741787+0000","last_change":"2024-03-06T07:26:19.740852+0000","last_active":"2024-03-06T07:26:19.741787+0000","last_peered":"2024-03-06T07:26:19.741787+0000","last_clean":"2024-03-06T07:26:19.741787+0000","last_became_active":"2024-03-06T07:26:19.740588+0000","last_became_peered":"2024-03-06T07:26:19.740588+0000","last_unstale":"2024-03-06T07:26:19.741787+0000","last_undegraded":"2024-03-06T07:26:19.741787+0000","last_fullsized":"2024-03-06T07:26:19.741787+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:27:46.844174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.736126+0000","last_change":"2024-03-06T07:26:19.732371+0000","last_active":"2024-03-06T07:26:19.736126+0000","last_peered":"2024-03-06T07:26:19.736126+0000","last_clean":"2024-03-06T07:26:19.736126+0000","last_became_active":"2024-03-06T07:26:19.732095+0000","last_became_peered":"2024-03-06T07:26:19.732095+0000","last_unstale":"2024-03-06T07:26:19.736126+0000","last_undegraded":"2024-03-06T07:26:19.736126+0000","last_fullsized":"2024-03-06T07:26:19.736126+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:55.844050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.731677+0000","last_change":"2024-03-06T07:26:19.724714+0000","last_active":"2024-03-06T07:26:19.731677+0000","last_peered":"2024-03-06T07:26:19.731677+0000","last_clean":"2024-03-06T07:26:19.731677+0000","last_became_active":"2024-03-06T07:26:19.724362+0000","last_became_peered":"2024-03-06T07:26:19.724362+0000","last_unstale":"2024-03-06T07:26:19.731677+0000","last_undegraded":"2024-03-06T07:26:19.731677+0000","last_fullsized":"2024-03-06T07:26:19.731677+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:27:46.844174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.736550+0000","last_change":"2024-03-06T07:26:19.735101+0000","last_active":"2024-03-06T07:26:19.736550+0000","last_peered":"2024-03-06T07:26:19.736550+0000","last_clean":"2024-03-06T07:26:19.736550+0000","last_became_active":"2024-03-06T07:26:19.734816+0000","last_became_peered":"2024-03-06T07:26:19.734816+0000","last_unstale":"2024-03-06T07:26:19.736550+0000","last_undegraded":"2024-03-06T07:26:19.736550+0000","last_fullsized":"2024-03-06T07:26:19.736550+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:11:14.556391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.738754+0000","last_change":"2024-03-06T07:26:19.734210+0000","last_active":"2024-03-06T07:26:19.738754+0000","last_peered":"2024-03-06T07:26:19.738754+0000","last_clean":"2024-03-06T07:26:19.738754+0000","last_became_active":"2024-03-06T07:26:19.733952+0000","last_became_peered":"2024-03-06T07:26:19.733952+0000","last_unstale":"2024-03-06T07:26:19.738754+0000","last_undegraded":"2024-03-06T07:26:19.738754+0000","last_fullsized":"2024-03-06T07:26:19.738754+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:06.236632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.732778+0000","last_change":"2024-03-06T07:26:19.728609+0000","last_active":"2024-03-06T07:26:19.732778+0000","last_peered":"2024-03-06T07:26:19.732778+0000","last_clean":"2024-03-06T07:26:19.732778+0000","last_became_active":"2024-03-06T07:26:19.727143+0000","last_became_peered":"2024-03-06T07:26:19.727143+0000","last_unstale":"2024-03-06T07:26:19.732778+0000","last_undegraded":"2024-03-06T07:26:19.732778+0000","last_fullsized":"2024-03-06T07:26:19.732778+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:17.550360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.736051+0000","last_change":"2024-03-06T07:26:19.727073+0000","last_active":"2024-03-06T07:26:19.736051+0000","last_peered":"2024-03-06T07:26:19.736051+0000","last_clean":"2024-03-06T07:26:19.736051+0000","last_became_active":"2024-03-06T07:26:19.726789+0000","last_became_peered":"2024-03-06T07:26:19.726789+0000","last_unstale":"2024-03-06T07:26:19.736051+0000","last_undegraded":"2024-03-06T07:26:19.736051+0000","last_fullsized":"2024-03-06T07:26:19.736051+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:52:11.743874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.735278+0000","last_change":"2024-03-06T07:26:19.724480+0000","last_active":"2024-03-06T07:26:19.735278+0000","last_peered":"2024-03-06T07:26:19.735278+0000","last_clean":"2024-03-06T07:26:19.735278+0000","last_became_active":"2024-03-06T07:26:19.724164+0000","last_became_peered":"2024-03-06T07:26:19.724164+0000","last_unstale":"2024-03-06T07:26:19.735278+0000","last_undegraded":"2024-03-06T07:26:19.735278+0000","last_fullsized":"2024-03-06T07:26:19.735278+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:56.348682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.731413+0000","last_change":"2024-03-06T07:26:19.725097+0000","last_active":"2024-03-06T07:26:19.731413+0000","last_peered":"2024-03-06T07:26:19.731413+0000","last_clean":"2024-03-06T07:26:19.731413+0000","last_became_active":"2024-03-06T07:26:19.724372+0000","last_became_peered":"2024-03-06T07:26:19.724372+0000","last_unstale":"2024-03-06T07:26:19.731413+0000","last_undegraded":"2024-03-06T07:26:19.731413+0000","last_fullsized":"2024-03-06T07:26:19.731413+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:46:28.616508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.740057+0000","last_change":"2024-03-06T07:26:19.739513+0000","last_active":"2024-03-06T07:26:19.740057+0000","last_peered":"2024-03-06T07:26:19.740057+0000","last_clean":"2024-03-06T07:26:19.740057+0000","last_became_active":"2024-03-06T07:26:19.739139+0000","last_became_peered":"2024-03-06T07:26:19.739139+0000","last_unstale":"2024-03-06T07:26:19.740057+0000","last_undegraded":"2024-03-06T07:26:19.740057+0000","last_fullsized":"2024-03-06T07:26:19.740057+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:19:22.032163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.732967+0000","last_change":"2024-03-06T07:26:19.730127+0000","last_active":"2024-03-06T07:26:19.732967+0000","last_peered":"2024-03-06T07:26:19.732967+0000","last_clean":"2024-03-06T07:26:19.732967+0000","last_became_active":"2024-03-06T07:26:19.729672+0000","last_became_peered":"2024-03-06T07:26:19.729672+0000","last_unstale":"2024-03-06T07:26:19.732967+0000","last_undegraded":"2024-03-06T07:26:19.732967+0000","last_fullsized":"2024-03-06T07:26:19.732967+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:17.550360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.740295+0000","last_change":"2024-03-06T07:26:19.738982+0000","last_active":"2024-03-06T07:26:19.740295+0000","last_peered":"2024-03-06T07:26:19.740295+0000","last_clean":"2024-03-06T07:26:19.740295+0000","last_became_active":"2024-03-06T07:26:19.738687+0000","last_became_peered":"2024-03-06T07:26:19.738687+0000","last_unstale":"2024-03-06T07:26:19.740295+0000","last_undegraded":"2024-03-06T07:26:19.740295+0000","last_fullsized":"2024-03-06T07:26:19.740295+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:46:25.686772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.738793+0000","last_change":"2024-03-06T07:26:19.736690+0000","last_active":"2024-03-06T07:26:19.738793+0000","last_peered":"2024-03-06T07:26:19.738793+0000","last_clean":"2024-03-06T07:26:19.738793+0000","last_became_active":"2024-03-06T07:26:19.736240+0000","last_became_peered":"2024-03-06T07:26:19.736240+0000","last_unstale":"2024-03-06T07:26:19.738793+0000","last_undegraded":"2024-03-06T07:26:19.738793+0000","last_fullsized":"2024-03-06T07:26:19.738793+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:23.513679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.727156+0000","last_change":"2024-03-06T07:26:19.724494+0000","last_active":"2024-03-06T07:26:19.727156+0000","last_peered":"2024-03-06T07:26:19.727156+0000","last_clean":"2024-03-06T07:26:19.727156+0000","last_became_active":"2024-03-06T07:26:19.724204+0000","last_became_peered":"2024-03-06T07:26:19.724204+0000","last_unstale":"2024-03-06T07:26:19.727156+0000","last_undegraded":"2024-03-06T07:26:19.727156+0000","last_fullsized":"2024-03-06T07:26:19.727156+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:30.017299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.738409+0000","last_change":"2024-03-06T07:26:19.733074+0000","last_active":"2024-03-06T07:26:19.738409+0000","last_peered":"2024-03-06T07:26:19.738409+0000","last_clean":"2024-03-06T07:26:19.738409+0000","last_became_active":"2024-03-06T07:26:19.732813+0000","last_became_peered":"2024-03-06T07:26:19.732813+0000","last_unstale":"2024-03-06T07:26:19.738409+0000","last_undegraded":"2024-03-06T07:26:19.738409+0000","last_fullsized":"2024-03-06T07:26:19.738409+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:19:22.032163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.738549+0000","last_change":"2024-03-06T07:26:19.736540+0000","last_active":"2024-03-06T07:26:19.738549+0000","last_peered":"2024-03-06T07:26:19.738549+0000","last_clean":"2024-03-06T07:26:19.738549+0000","last_became_active":"2024-03-06T07:26:19.736246+0000","last_became_peered":"2024-03-06T07:26:19.736246+0000","last_unstale":"2024-03-06T07:26:19.738549+0000","last_undegraded":"2024-03-06T07:26:19.738549+0000","last_fullsized":"2024-03-06T07:26:19.738549+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:59.857690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.737575+0000","last_change":"2024-03-06T07:26:19.736012+0000","last_active":"2024-03-06T07:26:19.737575+0000","last_peered":"2024-03-06T07:26:19.737575+0000","last_clean":"2024-03-06T07:26:19.737575+0000","last_became_active":"2024-03-06T07:26:19.735750+0000","last_became_peered":"2024-03-06T07:26:19.735750+0000","last_unstale":"2024-03-06T07:26:19.737575+0000","last_undegraded":"2024-03-06T07:26:19.737575+0000","last_fullsized":"2024-03-06T07:26:19.737575+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:17.550360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.742770+0000","last_change":"2024-03-06T07:26:19.738603+0000","last_active":"2024-03-06T07:26:19.742770+0000","last_peered":"2024-03-06T07:26:19.742770+0000","last_clean":"2024-03-06T07:26:19.742770+0000","last_became_active":"2024-03-06T07:26:19.738352+0000","last_became_peered":"2024-03-06T07:26:19.738352+0000","last_unstale":"2024-03-06T07:26:19.742770+0000","last_undegraded":"2024-03-06T07:26:19.742770+0000","last_fullsized":"2024-03-06T07:26:19.742770+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:16.131164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.736014+0000","last_change":"2024-03-06T07:26:19.727501+0000","last_active":"2024-03-06T07:26:19.736014+0000","last_peered":"2024-03-06T07:26:19.736014+0000","last_clean":"2024-03-06T07:26:19.736014+0000","last_became_active":"2024-03-06T07:26:19.727061+0000","last_became_peered":"2024-03-06T07:26:19.727061+0000","last_unstale":"2024-03-06T07:26:19.736014+0000","last_undegraded":"2024-03-06T07:26:19.736014+0000","last_fullsized":"2024-03-06T07:26:19.736014+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:52:08.732418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"93'4","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.737746+0000","last_change":"2024-03-06T07:26:18.726716+0000","last_active":"2024-03-06T07:26:19.737746+0000","last_peered":"2024-03-06T07:26:19.737746+0000","last_clean":"2024-03-06T07:26:19.737746+0000","last_became_active":"2024-03-06T07:26:18.726460+0000","last_became_peered":"2024-03-06T07:26:18.726460+0000","last_unstale":"2024-03-06T07:26:19.737746+0000","last_undegraded":"2024-03-06T07:26:19.737746+0000","last_fullsized":"2024-03-06T07:26:19.737746+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:13:09.899872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.739673+0000","last_change":"2024-03-06T07:26:19.737967+0000","last_active":"2024-03-06T07:26:19.739673+0000","last_peered":"2024-03-06T07:26:19.739673+0000","last_clean":"2024-03-06T07:26:19.739673+0000","last_became_active":"2024-03-06T07:26:19.737673+0000","last_became_peered":"2024-03-06T07:26:19.737673+0000","last_unstale":"2024-03-06T07:26:19.739673+0000","last_undegraded":"2024-03-06T07:26:19.739673+0000","last_fullsized":"2024-03-06T07:26:19.739673+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:53:35.777661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"93'1","reported_seq":13,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.742244+0000","last_change":"2024-03-06T07:26:18.726268+0000","last_active":"2024-03-06T07:26:19.742244+0000","last_peered":"2024-03-06T07:26:19.742244+0000","last_clean":"2024-03-06T07:26:19.742244+0000","last_became_active":"2024-03-06T07:26:18.725937+0000","last_became_peered":"2024-03-06T07:26:18.725937+0000","last_unstale":"2024-03-06T07:26:19.742244+0000","last_undegraded":"2024-03-06T07:26:19.742244+0000","last_fullsized":"2024-03-06T07:26:19.742244+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:11:13.546951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.742843+0000","last_change":"2024-03-06T07:26:19.740104+0000","last_active":"2024-03-06T07:26:19.742843+0000","last_peered":"2024-03-06T07:26:19.742843+0000","last_clean":"2024-03-06T07:26:19.742843+0000","last_became_active":"2024-03-06T07:26:19.739816+0000","last_became_peered":"2024-03-06T07:26:19.739816+0000","last_unstale":"2024-03-06T07:26:19.742843+0000","last_undegraded":"2024-03-06T07:26:19.742843+0000","last_fullsized":"2024-03-06T07:26:19.742843+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:06.236632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"93'2","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.737061+0000","last_change":"2024-03-06T07:26:18.727278+0000","last_active":"2024-03-06T07:26:19.737061+0000","last_peered":"2024-03-06T07:26:19.737061+0000","last_clean":"2024-03-06T07:26:19.737061+0000","last_became_active":"2024-03-06T07:26:18.727010+0000","last_became_peered":"2024-03-06T07:26:18.727010+0000","last_unstale":"2024-03-06T07:26:19.737061+0000","last_undegraded":"2024-03-06T07:26:19.737061+0000","last_fullsized":"2024-03-06T07:26:19.737061+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:16.540921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.728229+0000","last_change":"2024-03-06T07:26:19.725287+0000","last_active":"2024-03-06T07:26:19.728229+0000","last_peered":"2024-03-06T07:26:19.728229+0000","last_clean":"2024-03-06T07:26:19.728229+0000","last_became_active":"2024-03-06T07:26:19.725009+0000","last_became_peered":"2024-03-06T07:26:19.725009+0000","last_unstale":"2024-03-06T07:26:19.728229+0000","last_undegraded":"2024-03-06T07:26:19.728229+0000","last_fullsized":"2024-03-06T07:26:19.728229+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:17.128778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"93'2","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.736627+0000","last_change":"2024-03-06T07:26:18.729060+0000","last_active":"2024-03-06T07:26:19.736627+0000","last_peered":"2024-03-06T07:26:19.736627+0000","last_clean":"2024-03-06T07:26:19.736627+0000","last_became_active":"2024-03-06T07:26:18.728795+0000","last_became_peered":"2024-03-06T07:26:18.728795+0000","last_unstale":"2024-03-06T07:26:19.736627+0000","last_undegraded":"2024-03-06T07:26:19.736627+0000","last_fullsized":"2024-03-06T07:26:19.736627+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:38:12.922468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.731219+0000","last_change":"2024-03-06T07:26:19.725732+0000","last_active":"2024-03-06T07:26:19.731219+0000","last_peered":"2024-03-06T07:26:19.731219+0000","last_clean":"2024-03-06T07:26:19.731219+0000","last_became_active":"2024-03-06T07:26:19.725443+0000","last_became_peered":"2024-03-06T07:26:19.725443+0000","last_unstale":"2024-03-06T07:26:19.731219+0000","last_undegraded":"2024-03-06T07:26:19.731219+0000","last_fullsized":"2024-03-06T07:26:19.731219+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:41.542738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"93'1","reported_seq":13,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.741861+0000","last_change":"2024-03-06T07:26:18.728494+0000","last_active":"2024-03-06T07:26:19.741861+0000","last_peered":"2024-03-06T07:26:19.741861+0000","last_clean":"2024-03-06T07:26:19.741861+0000","last_became_active":"2024-03-06T07:26:18.728198+0000","last_became_peered":"2024-03-06T07:26:18.728198+0000","last_unstale":"2024-03-06T07:26:19.741861+0000","last_undegraded":"2024-03-06T07:26:19.741861+0000","last_fullsized":"2024-03-06T07:26:19.741861+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:10:30.091943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.731110+0000","last_change":"2024-03-06T07:26:19.728888+0000","last_active":"2024-03-06T07:26:19.731110+0000","last_peered":"2024-03-06T07:26:19.731110+0000","last_clean":"2024-03-06T07:26:19.731110+0000","last_became_active":"2024-03-06T07:26:19.728656+0000","last_became_peered":"2024-03-06T07:26:19.728656+0000","last_unstale":"2024-03-06T07:26:19.731110+0000","last_undegraded":"2024-03-06T07:26:19.731110+0000","last_fullsized":"2024-03-06T07:26:19.731110+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:08.203747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"93'1","reported_seq":13,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.740089+0000","last_change":"2024-03-06T07:26:18.724482+0000","last_active":"2024-03-06T07:26:19.740089+0000","last_peered":"2024-03-06T07:26:19.740089+0000","last_clean":"2024-03-06T07:26:19.740089+0000","last_became_active":"2024-03-06T07:26:18.724150+0000","last_became_peered":"2024-03-06T07:26:18.724150+0000","last_unstale":"2024-03-06T07:26:19.740089+0000","last_undegraded":"2024-03-06T07:26:19.740089+0000","last_fullsized":"2024-03-06T07:26:19.740089+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:36:42.959958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.741858+0000","last_change":"2024-03-06T07:26:19.740964+0000","last_active":"2024-03-06T07:26:19.741858+0000","last_peered":"2024-03-06T07:26:19.741858+0000","last_clean":"2024-03-06T07:26:19.741858+0000","last_became_active":"2024-03-06T07:26:19.740694+0000","last_became_peered":"2024-03-06T07:26:19.740694+0000","last_unstale":"2024-03-06T07:26:19.741858+0000","last_undegraded":"2024-03-06T07:26:19.741858+0000","last_fullsized":"2024-03-06T07:26:19.741858+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:46:28.616508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.715608+0000","last_change":"2024-03-06T07:26:18.732422+0000","last_active":"2024-03-06T07:26:19.715608+0000","last_peered":"2024-03-06T07:26:19.715608+0000","last_clean":"2024-03-06T07:26:19.715608+0000","last_became_active":"2024-03-06T07:26:18.732149+0000","last_became_peered":"2024-03-06T07:26:18.732149+0000","last_unstale":"2024-03-06T07:26:19.715608+0000","last_undegraded":"2024-03-06T07:26:19.715608+0000","last_fullsized":"2024-03-06T07:26:19.715608+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:29.117687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.728054+0000","last_change":"2024-03-06T07:26:19.723401+0000","last_active":"2024-03-06T07:26:19.728054+0000","last_peered":"2024-03-06T07:26:19.728054+0000","last_clean":"2024-03-06T07:26:19.728054+0000","last_became_active":"2024-03-06T07:26:19.722863+0000","last_became_peered":"2024-03-06T07:26:19.722863+0000","last_unstale":"2024-03-06T07:26:19.728054+0000","last_undegraded":"2024-03-06T07:26:19.728054+0000","last_fullsized":"2024-03-06T07:26:19.728054+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:36:43.969398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.726106+0000","last_change":"2024-03-06T07:26:18.725961+0000","last_active":"2024-03-06T07:26:19.726106+0000","last_peered":"2024-03-06T07:26:19.726106+0000","last_clean":"2024-03-06T07:26:19.726106+0000","last_became_active":"2024-03-06T07:26:18.725715+0000","last_became_peered":"2024-03-06T07:26:18.725715+0000","last_unstale":"2024-03-06T07:26:19.726106+0000","last_undegraded":"2024-03-06T07:26:19.726106+0000","last_fullsized":"2024-03-06T07:26:19.726106+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:26:58.476276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.732090+0000","last_change":"2024-03-06T07:26:19.728430+0000","last_active":"2024-03-06T07:26:19.732090+0000","last_peered":"2024-03-06T07:26:19.732090+0000","last_clean":"2024-03-06T07:26:19.732090+0000","last_became_active":"2024-03-06T07:26:19.728192+0000","last_became_peered":"2024-03-06T07:26:19.728192+0000","last_unstale":"2024-03-06T07:26:19.732090+0000","last_undegraded":"2024-03-06T07:26:19.732090+0000","last_fullsized":"2024-03-06T07:26:19.732090+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:04.601612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.724910+0000","last_change":"2024-03-06T07:26:18.724249+0000","last_active":"2024-03-06T07:26:19.724910+0000","last_peered":"2024-03-06T07:26:19.724910+0000","last_clean":"2024-03-06T07:26:19.724910+0000","last_became_active":"2024-03-06T07:26:18.723977+0000","last_became_peered":"2024-03-06T07:26:18.723977+0000","last_unstale":"2024-03-06T07:26:19.724910+0000","last_undegraded":"2024-03-06T07:26:19.724910+0000","last_fullsized":"2024-03-06T07:26:19.724910+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:52:10.734434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.728929+0000","last_change":"2024-03-06T07:26:19.727115+0000","last_active":"2024-03-06T07:26:19.728929+0000","last_peered":"2024-03-06T07:26:19.728929+0000","last_clean":"2024-03-06T07:26:19.728929+0000","last_became_active":"2024-03-06T07:26:19.726801+0000","last_became_peered":"2024-03-06T07:26:19.726801+0000","last_unstale":"2024-03-06T07:26:19.728929+0000","last_undegraded":"2024-03-06T07:26:19.728929+0000","last_fullsized":"2024-03-06T07:26:19.728929+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:54.623542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.722761+0000","last_change":"2024-03-06T07:26:18.723149+0000","last_active":"2024-03-06T07:26:19.722761+0000","last_peered":"2024-03-06T07:26:19.722761+0000","last_clean":"2024-03-06T07:26:19.722761+0000","last_became_active":"2024-03-06T07:26:18.722892+0000","last_became_peered":"2024-03-06T07:26:18.722892+0000","last_unstale":"2024-03-06T07:26:19.722761+0000","last_undegraded":"2024-03-06T07:26:19.722761+0000","last_fullsized":"2024-03-06T07:26:19.722761+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:36:42.959958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.727137+0000","last_change":"2024-03-06T07:26:19.723523+0000","last_active":"2024-03-06T07:26:19.727137+0000","last_peered":"2024-03-06T07:26:19.727137+0000","last_clean":"2024-03-06T07:26:19.727137+0000","last_became_active":"2024-03-06T07:26:19.723232+0000","last_became_peered":"2024-03-06T07:26:19.723232+0000","last_unstale":"2024-03-06T07:26:19.727137+0000","last_undegraded":"2024-03-06T07:26:19.727137+0000","last_fullsized":"2024-03-06T07:26:19.727137+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:36:43.969398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"93'3","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.739949+0000","last_change":"2024-03-06T07:26:18.728852+0000","last_active":"2024-03-06T07:26:19.739949+0000","last_peered":"2024-03-06T07:26:19.739949+0000","last_clean":"2024-03-06T07:26:19.739949+0000","last_became_active":"2024-03-06T07:26:18.728539+0000","last_became_peered":"2024-03-06T07:26:18.728539+0000","last_unstale":"2024-03-06T07:26:19.739949+0000","last_undegraded":"2024-03-06T07:26:19.739949+0000","last_fullsized":"2024-03-06T07:26:19.739949+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:16.540921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.736134+0000","last_change":"2024-03-06T07:26:19.732572+0000","last_active":"2024-03-06T07:26:19.736134+0000","last_peered":"2024-03-06T07:26:19.736134+0000","last_clean":"2024-03-06T07:26:19.736134+0000","last_became_active":"2024-03-06T07:26:19.732306+0000","last_became_peered":"2024-03-06T07:26:19.732306+0000","last_unstale":"2024-03-06T07:26:19.736134+0000","last_undegraded":"2024-03-06T07:26:19.736134+0000","last_fullsized":"2024-03-06T07:26:19.736134+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:04.601612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"93'6","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.744342+0000","last_change":"2024-03-06T07:26:18.726061+0000","last_active":"2024-03-06T07:26:19.744342+0000","last_peered":"2024-03-06T07:26:19.744342+0000","last_clean":"2024-03-06T07:26:19.744342+0000","last_became_active":"2024-03-06T07:26:18.725791+0000","last_became_peered":"2024-03-06T07:26:18.725791+0000","last_unstale":"2024-03-06T07:26:19.744342+0000","last_undegraded":"2024-03-06T07:26:19.744342+0000","last_fullsized":"2024-03-06T07:26:19.744342+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:38:12.922468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.727660+0000","last_change":"2024-03-06T07:26:19.726823+0000","last_active":"2024-03-06T07:26:19.727660+0000","last_peered":"2024-03-06T07:26:19.727660+0000","last_clean":"2024-03-06T07:26:19.727660+0000","last_became_active":"2024-03-06T07:26:19.726453+0000","last_became_peered":"2024-03-06T07:26:19.726453+0000","last_unstale":"2024-03-06T07:26:19.727660+0000","last_undegraded":"2024-03-06T07:26:19.727660+0000","last_fullsized":"2024-03-06T07:26:19.727660+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:30.017299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"93'2","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.739773+0000","last_change":"2024-03-06T07:26:18.726135+0000","last_active":"2024-03-06T07:26:19.739773+0000","last_peered":"2024-03-06T07:26:19.739773+0000","last_clean":"2024-03-06T07:26:19.739773+0000","last_became_active":"2024-03-06T07:26:18.725858+0000","last_became_peered":"2024-03-06T07:26:18.725858+0000","last_unstale":"2024-03-06T07:26:19.739773+0000","last_undegraded":"2024-03-06T07:26:19.739773+0000","last_fullsized":"2024-03-06T07:26:19.739773+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:47:34.287913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.742918+0000","last_change":"2024-03-06T07:26:19.741214+0000","last_active":"2024-03-06T07:26:19.742918+0000","last_peered":"2024-03-06T07:26:19.742918+0000","last_clean":"2024-03-06T07:26:19.742918+0000","last_became_active":"2024-03-06T07:26:19.740963+0000","last_became_peered":"2024-03-06T07:26:19.740963+0000","last_unstale":"2024-03-06T07:26:19.742918+0000","last_undegraded":"2024-03-06T07:26:19.742918+0000","last_fullsized":"2024-03-06T07:26:19.742918+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:50.991334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.719461+0000","last_change":"2024-03-06T07:26:18.725543+0000","last_active":"2024-03-06T07:26:19.719461+0000","last_peered":"2024-03-06T07:26:19.719461+0000","last_clean":"2024-03-06T07:26:19.719461+0000","last_became_active":"2024-03-06T07:26:18.725302+0000","last_became_peered":"2024-03-06T07:26:18.725302+0000","last_unstale":"2024-03-06T07:26:19.719461+0000","last_undegraded":"2024-03-06T07:26:19.719461+0000","last_fullsized":"2024-03-06T07:26:19.719461+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:06.822178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.731318+0000","last_change":"2024-03-06T07:26:19.728351+0000","last_active":"2024-03-06T07:26:19.731318+0000","last_peered":"2024-03-06T07:26:19.731318+0000","last_clean":"2024-03-06T07:26:19.731318+0000","last_became_active":"2024-03-06T07:26:19.726177+0000","last_became_peered":"2024-03-06T07:26:19.726177+0000","last_unstale":"2024-03-06T07:26:19.731318+0000","last_undegraded":"2024-03-06T07:26:19.731318+0000","last_fullsized":"2024-03-06T07:26:19.731318+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:11:14.556391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.716552+0000","last_change":"2024-03-06T07:26:18.727884+0000","last_active":"2024-03-06T07:26:19.716552+0000","last_peered":"2024-03-06T07:26:19.716552+0000","last_clean":"2024-03-06T07:26:19.716552+0000","last_became_active":"2024-03-06T07:26:18.727582+0000","last_became_peered":"2024-03-06T07:26:18.727582+0000","last_unstale":"2024-03-06T07:26:19.716552+0000","last_undegraded":"2024-03-06T07:26:19.716552+0000","last_fullsized":"2024-03-06T07:26:19.716552+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:29.117687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.736386+0000","last_change":"2024-03-06T07:26:19.730980+0000","last_active":"2024-03-06T07:26:19.736386+0000","last_peered":"2024-03-06T07:26:19.736386+0000","last_clean":"2024-03-06T07:26:19.736386+0000","last_became_active":"2024-03-06T07:26:19.730687+0000","last_became_peered":"2024-03-06T07:26:19.730687+0000","last_unstale":"2024-03-06T07:26:19.736386+0000","last_undegraded":"2024-03-06T07:26:19.736386+0000","last_fullsized":"2024-03-06T07:26:19.736386+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:37.811558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"93'8","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.742730+0000","last_change":"2024-03-06T07:26:18.724919+0000","last_active":"2024-03-06T07:26:19.742730+0000","last_peered":"2024-03-06T07:26:19.742730+0000","last_clean":"2024-03-06T07:26:19.742730+0000","last_became_active":"2024-03-06T07:26:18.724649+0000","last_became_peered":"2024-03-06T07:26:18.724649+0000","last_unstale":"2024-03-06T07:26:19.742730+0000","last_undegraded":"2024-03-06T07:26:19.742730+0000","last_fullsized":"2024-03-06T07:26:19.742730+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:52:07.722978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.728628+0000","last_change":"2024-03-06T07:26:19.725595+0000","last_active":"2024-03-06T07:26:19.728628+0000","last_peered":"2024-03-06T07:26:19.728628+0000","last_clean":"2024-03-06T07:26:19.728628+0000","last_became_active":"2024-03-06T07:26:19.723788+0000","last_became_peered":"2024-03-06T07:26:19.723788+0000","last_unstale":"2024-03-06T07:26:19.728628+0000","last_undegraded":"2024-03-06T07:26:19.728628+0000","last_fullsized":"2024-03-06T07:26:19.728628+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:11:14.556391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"93'2","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.740647+0000","last_change":"2024-03-06T07:26:18.727027+0000","last_active":"2024-03-06T07:26:19.740647+0000","last_peered":"2024-03-06T07:26:19.740647+0000","last_clean":"2024-03-06T07:26:19.740647+0000","last_became_active":"2024-03-06T07:26:18.726712+0000","last_became_peered":"2024-03-06T07:26:18.726712+0000","last_unstale":"2024-03-06T07:26:19.740647+0000","last_undegraded":"2024-03-06T07:26:19.740647+0000","last_fullsized":"2024-03-06T07:26:19.740647+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:52.931110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.731258+0000","last_change":"2024-03-06T07:26:19.727130+0000","last_active":"2024-03-06T07:26:19.731258+0000","last_peered":"2024-03-06T07:26:19.731258+0000","last_clean":"2024-03-06T07:26:19.731258+0000","last_became_active":"2024-03-06T07:26:19.726842+0000","last_became_peered":"2024-03-06T07:26:19.726842+0000","last_unstale":"2024-03-06T07:26:19.731258+0000","last_undegraded":"2024-03-06T07:26:19.731258+0000","last_fullsized":"2024-03-06T07:26:19.731258+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:12:20.891606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"93'1","reported_seq":13,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.736922+0000","last_change":"2024-03-06T07:26:18.725982+0000","last_active":"2024-03-06T07:26:19.736922+0000","last_peered":"2024-03-06T07:26:19.736922+0000","last_clean":"2024-03-06T07:26:19.736922+0000","last_became_active":"2024-03-06T07:26:18.724772+0000","last_became_peered":"2024-03-06T07:26:18.724772+0000","last_unstale":"2024-03-06T07:26:19.736922+0000","last_undegraded":"2024-03-06T07:26:19.736922+0000","last_fullsized":"2024-03-06T07:26:19.736922+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:13:09.899872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.728622+0000","last_change":"2024-03-06T07:26:19.726585+0000","last_active":"2024-03-06T07:26:19.728622+0000","last_peered":"2024-03-06T07:26:19.728622+0000","last_clean":"2024-03-06T07:26:19.728622+0000","last_became_active":"2024-03-06T07:26:19.726302+0000","last_became_peered":"2024-03-06T07:26:19.726302+0000","last_unstale":"2024-03-06T07:26:19.728622+0000","last_undegraded":"2024-03-06T07:26:19.728622+0000","last_fullsized":"2024-03-06T07:26:19.728622+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:56.348682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.714921+0000","last_change":"2024-03-06T07:26:18.729197+0000","last_active":"2024-03-06T07:26:19.714921+0000","last_peered":"2024-03-06T07:26:19.714921+0000","last_clean":"2024-03-06T07:26:19.714921+0000","last_became_active":"2024-03-06T07:26:18.728584+0000","last_became_peered":"2024-03-06T07:26:18.728584+0000","last_unstale":"2024-03-06T07:26:19.714921+0000","last_undegraded":"2024-03-06T07:26:19.714921+0000","last_fullsized":"2024-03-06T07:26:19.714921+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:29.007859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.736710+0000","last_change":"2024-03-06T07:26:19.725650+0000","last_active":"2024-03-06T07:26:19.736710+0000","last_peered":"2024-03-06T07:26:19.736710+0000","last_clean":"2024-03-06T07:26:19.736710+0000","last_became_active":"2024-03-06T07:26:19.725402+0000","last_became_peered":"2024-03-06T07:26:19.725402+0000","last_unstale":"2024-03-06T07:26:19.736710+0000","last_undegraded":"2024-03-06T07:26:19.736710+0000","last_fullsized":"2024-03-06T07:26:19.736710+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:10:31.101383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"93'2","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.737395+0000","last_change":"2024-03-06T07:26:18.728232+0000","last_active":"2024-03-06T07:26:19.737395+0000","last_peered":"2024-03-06T07:26:19.737395+0000","last_clean":"2024-03-06T07:26:19.737395+0000","last_became_active":"2024-03-06T07:26:18.728019+0000","last_became_peered":"2024-03-06T07:26:18.728019+0000","last_unstale":"2024-03-06T07:26:19.737395+0000","last_undegraded":"2024-03-06T07:26:19.737395+0000","last_fullsized":"2024-03-06T07:26:19.737395+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:53.614103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.738587+0000","last_change":"2024-03-06T07:26:19.732170+0000","last_active":"2024-03-06T07:26:19.738587+0000","last_peered":"2024-03-06T07:26:19.738587+0000","last_clean":"2024-03-06T07:26:19.738587+0000","last_became_active":"2024-03-06T07:26:19.731374+0000","last_became_peered":"2024-03-06T07:26:19.731374+0000","last_unstale":"2024-03-06T07:26:19.738587+0000","last_undegraded":"2024-03-06T07:26:19.738587+0000","last_fullsized":"2024-03-06T07:26:19.738587+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:16.131164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.720659+0000","last_change":"2024-03-06T07:26:18.727063+0000","last_active":"2024-03-06T07:26:19.720659+0000","last_peered":"2024-03-06T07:26:19.720659+0000","last_clean":"2024-03-06T07:26:19.720659+0000","last_became_active":"2024-03-06T07:26:18.726585+0000","last_became_peered":"2024-03-06T07:26:18.726585+0000","last_unstale":"2024-03-06T07:26:19.720659+0000","last_undegraded":"2024-03-06T07:26:19.720659+0000","last_fullsized":"2024-03-06T07:26:19.720659+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:51.405230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.731778+0000","last_change":"2024-03-06T07:26:19.723739+0000","last_active":"2024-03-06T07:26:19.731778+0000","last_peered":"2024-03-06T07:26:19.731778+0000","last_clean":"2024-03-06T07:26:19.731778+0000","last_became_active":"2024-03-06T07:26:19.723457+0000","last_became_peered":"2024-03-06T07:26:19.723457+0000","last_unstale":"2024-03-06T07:26:19.731778+0000","last_undegraded":"2024-03-06T07:26:19.731778+0000","last_fullsized":"2024-03-06T07:26:19.731778+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:56.186991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.720717+0000","last_change":"2024-03-06T07:26:18.730150+0000","last_active":"2024-03-06T07:26:19.720717+0000","last_peered":"2024-03-06T07:26:19.720717+0000","last_clean":"2024-03-06T07:26:19.720717+0000","last_became_active":"2024-03-06T07:26:18.729809+0000","last_became_peered":"2024-03-06T07:26:18.729809+0000","last_unstale":"2024-03-06T07:26:19.720717+0000","last_undegraded":"2024-03-06T07:26:19.720717+0000","last_fullsized":"2024-03-06T07:26:19.720717+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:42.398745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.736363+0000","last_change":"2024-03-06T07:26:19.732240+0000","last_active":"2024-03-06T07:26:19.736363+0000","last_peered":"2024-03-06T07:26:19.736363+0000","last_clean":"2024-03-06T07:26:19.736363+0000","last_became_active":"2024-03-06T07:26:19.731971+0000","last_became_peered":"2024-03-06T07:26:19.731971+0000","last_unstale":"2024-03-06T07:26:19.736363+0000","last_undegraded":"2024-03-06T07:26:19.736363+0000","last_fullsized":"2024-03-06T07:26:19.736363+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:53:35.777661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.714072+0000","last_change":"2024-03-06T07:26:18.722623+0000","last_active":"2024-03-06T07:26:19.714072+0000","last_peered":"2024-03-06T07:26:19.714072+0000","last_clean":"2024-03-06T07:26:19.714072+0000","last_became_active":"2024-03-06T07:26:18.722379+0000","last_became_peered":"2024-03-06T07:26:18.722379+0000","last_unstale":"2024-03-06T07:26:19.714072+0000","last_undegraded":"2024-03-06T07:26:19.714072+0000","last_fullsized":"2024-03-06T07:26:19.714072+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:52.931110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.732051+0000","last_change":"2024-03-06T07:26:19.727054+0000","last_active":"2024-03-06T07:26:19.732051+0000","last_peered":"2024-03-06T07:26:19.732051+0000","last_clean":"2024-03-06T07:26:19.732051+0000","last_became_active":"2024-03-06T07:26:19.726755+0000","last_became_peered":"2024-03-06T07:26:19.726755+0000","last_unstale":"2024-03-06T07:26:19.732051+0000","last_undegraded":"2024-03-06T07:26:19.732051+0000","last_fullsized":"2024-03-06T07:26:19.732051+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:08.203747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"93'2","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.739854+0000","last_change":"2024-03-06T07:26:18.728905+0000","last_active":"2024-03-06T07:26:19.739854+0000","last_peered":"2024-03-06T07:26:19.739854+0000","last_clean":"2024-03-06T07:26:19.739854+0000","last_became_active":"2024-03-06T07:26:18.728587+0000","last_became_peered":"2024-03-06T07:26:18.728587+0000","last_unstale":"2024-03-06T07:26:19.739854+0000","last_undegraded":"2024-03-06T07:26:19.739854+0000","last_fullsized":"2024-03-06T07:26:19.739854+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:14:35.299761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.742874+0000","last_change":"2024-03-06T07:26:19.740512+0000","last_active":"2024-03-06T07:26:19.742874+0000","last_peered":"2024-03-06T07:26:19.742874+0000","last_clean":"2024-03-06T07:26:19.742874+0000","last_became_active":"2024-03-06T07:26:19.740284+0000","last_became_peered":"2024-03-06T07:26:19.740284+0000","last_unstale":"2024-03-06T07:26:19.742874+0000","last_undegraded":"2024-03-06T07:26:19.742874+0000","last_fullsized":"2024-03-06T07:26:19.742874+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:59.857690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.715749+0000","last_change":"2024-03-06T07:26:18.727585+0000","last_active":"2024-03-06T07:26:19.715749+0000","last_peered":"2024-03-06T07:26:19.715749+0000","last_clean":"2024-03-06T07:26:19.715749+0000","last_became_active":"2024-03-06T07:26:18.727315+0000","last_became_peered":"2024-03-06T07:26:18.727315+0000","last_unstale":"2024-03-06T07:26:19.715749+0000","last_undegraded":"2024-03-06T07:26:19.715749+0000","last_fullsized":"2024-03-06T07:26:19.715749+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:53:34.768221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.741979+0000","last_change":"2024-03-06T07:26:19.733995+0000","last_active":"2024-03-06T07:26:19.741979+0000","last_peered":"2024-03-06T07:26:19.741979+0000","last_clean":"2024-03-06T07:26:19.741979+0000","last_became_active":"2024-03-06T07:26:19.733749+0000","last_became_peered":"2024-03-06T07:26:19.733749+0000","last_unstale":"2024-03-06T07:26:19.741979+0000","last_undegraded":"2024-03-06T07:26:19.741979+0000","last_fullsized":"2024-03-06T07:26:19.741979+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:54:21.900420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.714613+0000","last_change":"2024-03-06T07:26:18.729239+0000","last_active":"2024-03-06T07:26:19.714613+0000","last_peered":"2024-03-06T07:26:19.714613+0000","last_clean":"2024-03-06T07:26:19.714613+0000","last_became_active":"2024-03-06T07:26:18.728737+0000","last_became_peered":"2024-03-06T07:26:18.728737+0000","last_unstale":"2024-03-06T07:26:19.714613+0000","last_undegraded":"2024-03-06T07:26:19.714613+0000","last_fullsized":"2024-03-06T07:26:19.714613+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:16.119338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.727103+0000","last_change":"2024-03-06T07:26:19.724422+0000","last_active":"2024-03-06T07:26:19.727103+0000","last_peered":"2024-03-06T07:26:19.727103+0000","last_clean":"2024-03-06T07:26:19.727103+0000","last_became_active":"2024-03-06T07:26:19.724072+0000","last_became_peered":"2024-03-06T07:26:19.724072+0000","last_unstale":"2024-03-06T07:26:19.727103+0000","last_undegraded":"2024-03-06T07:26:19.727103+0000","last_fullsized":"2024-03-06T07:26:19.727103+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:17.128778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.715963+0000","last_change":"2024-03-06T07:26:18.728248+0000","last_active":"2024-03-06T07:26:19.715963+0000","last_peered":"2024-03-06T07:26:19.715963+0000","last_clean":"2024-03-06T07:26:19.715963+0000","last_became_active":"2024-03-06T07:26:18.727982+0000","last_became_peered":"2024-03-06T07:26:18.727982+0000","last_unstale":"2024-03-06T07:26:19.715963+0000","last_undegraded":"2024-03-06T07:26:19.715963+0000","last_fullsized":"2024-03-06T07:26:19.715963+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:38:12.922468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.736729+0000","last_change":"2024-03-06T07:26:19.730542+0000","last_active":"2024-03-06T07:26:19.736729+0000","last_peered":"2024-03-06T07:26:19.736729+0000","last_clean":"2024-03-06T07:26:19.736729+0000","last_became_active":"2024-03-06T07:26:19.729923+0000","last_became_peered":"2024-03-06T07:26:19.729923+0000","last_unstale":"2024-03-06T07:26:19.736729+0000","last_undegraded":"2024-03-06T07:26:19.736729+0000","last_fullsized":"2024-03-06T07:26:19.736729+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:54:21.900420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.715587+0000","last_change":"2024-03-06T07:26:18.728166+0000","last_active":"2024-03-06T07:26:19.715587+0000","last_peered":"2024-03-06T07:26:19.715587+0000","last_clean":"2024-03-06T07:26:19.715587+0000","last_became_active":"2024-03-06T07:26:18.727886+0000","last_became_peered":"2024-03-06T07:26:18.727886+0000","last_unstale":"2024-03-06T07:26:19.715587+0000","last_undegraded":"2024-03-06T07:26:19.715587+0000","last_fullsized":"2024-03-06T07:26:19.715587+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:11:13.546951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.723489+0000","last_change":"2024-03-06T07:26:19.720690+0000","last_active":"2024-03-06T07:26:19.723489+0000","last_peered":"2024-03-06T07:26:19.723489+0000","last_clean":"2024-03-06T07:26:19.723489+0000","last_became_active":"2024-03-06T07:26:19.720264+0000","last_became_peered":"2024-03-06T07:26:19.720264+0000","last_unstale":"2024-03-06T07:26:19.723489+0000","last_undegraded":"2024-03-06T07:26:19.723489+0000","last_fullsized":"2024-03-06T07:26:19.723489+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:13:10.909312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.715801+0000","last_change":"2024-03-06T07:26:18.727030+0000","last_active":"2024-03-06T07:26:19.715801+0000","last_peered":"2024-03-06T07:26:19.715801+0000","last_clean":"2024-03-06T07:26:19.715801+0000","last_became_active":"2024-03-06T07:26:18.725274+0000","last_became_peered":"2024-03-06T07:26:18.725274+0000","last_unstale":"2024-03-06T07:26:19.715801+0000","last_undegraded":"2024-03-06T07:26:19.715801+0000","last_fullsized":"2024-03-06T07:26:19.715801+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:52.931110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.736042+0000","last_change":"2024-03-06T07:26:19.727321+0000","last_active":"2024-03-06T07:26:19.736042+0000","last_peered":"2024-03-06T07:26:19.736042+0000","last_clean":"2024-03-06T07:26:19.736042+0000","last_became_active":"2024-03-06T07:26:19.726619+0000","last_became_peered":"2024-03-06T07:26:19.726619+0000","last_unstale":"2024-03-06T07:26:19.736042+0000","last_undegraded":"2024-03-06T07:26:19.736042+0000","last_fullsized":"2024-03-06T07:26:19.736042+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:26:59.485716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.720753+0000","last_change":"2024-03-06T07:26:18.723615+0000","last_active":"2024-03-06T07:26:19.720753+0000","last_peered":"2024-03-06T07:26:19.720753+0000","last_clean":"2024-03-06T07:26:19.720753+0000","last_became_active":"2024-03-06T07:26:18.723349+0000","last_became_peered":"2024-03-06T07:26:18.723349+0000","last_unstale":"2024-03-06T07:26:19.720753+0000","last_undegraded":"2024-03-06T07:26:19.720753+0000","last_fullsized":"2024-03-06T07:26:19.720753+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:13:09.899872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.727449+0000","last_change":"2024-03-06T07:26:19.725056+0000","last_active":"2024-03-06T07:26:19.727449+0000","last_peered":"2024-03-06T07:26:19.727449+0000","last_clean":"2024-03-06T07:26:19.727449+0000","last_became_active":"2024-03-06T07:26:19.724803+0000","last_became_peered":"2024-03-06T07:26:19.724803+0000","last_unstale":"2024-03-06T07:26:19.727449+0000","last_undegraded":"2024-03-06T07:26:19.727449+0000","last_fullsized":"2024-03-06T07:26:19.727449+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:54.623542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.718132+0000","last_change":"2024-03-06T07:26:18.728464+0000","last_active":"2024-03-06T07:26:19.718132+0000","last_peered":"2024-03-06T07:26:19.718132+0000","last_clean":"2024-03-06T07:26:19.718132+0000","last_became_active":"2024-03-06T07:26:18.728202+0000","last_became_peered":"2024-03-06T07:26:18.728202+0000","last_unstale":"2024-03-06T07:26:19.718132+0000","last_undegraded":"2024-03-06T07:26:19.718132+0000","last_fullsized":"2024-03-06T07:26:19.718132+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:04:30.366156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.731749+0000","last_change":"2024-03-06T07:26:19.725026+0000","last_active":"2024-03-06T07:26:19.731749+0000","last_peered":"2024-03-06T07:26:19.731749+0000","last_clean":"2024-03-06T07:26:19.731749+0000","last_became_active":"2024-03-06T07:26:19.724759+0000","last_became_peered":"2024-03-06T07:26:19.724759+0000","last_unstale":"2024-03-06T07:26:19.731749+0000","last_undegraded":"2024-03-06T07:26:19.731749+0000","last_fullsized":"2024-03-06T07:26:19.731749+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.703551+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.703551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:24:12.521477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"93'1","reported_seq":13,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.738800+0000","last_change":"2024-03-06T07:26:18.728873+0000","last_active":"2024-03-06T07:26:19.738800+0000","last_peered":"2024-03-06T07:26:19.738800+0000","last_clean":"2024-03-06T07:26:19.738800+0000","last_became_active":"2024-03-06T07:26:18.728290+0000","last_became_peered":"2024-03-06T07:26:18.728290+0000","last_unstale":"2024-03-06T07:26:19.738800+0000","last_undegraded":"2024-03-06T07:26:19.738800+0000","last_fullsized":"2024-03-06T07:26:19.738800+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:17.694111+0000","last_clean_scrub_stamp":"2024-03-06T07:26:17.694111+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:53.614103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":303,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.720104+0000","last_change":"2024-03-06T07:21:47.553693+0000","last_active":"2024-03-06T07:26:19.720104+0000","last_peered":"2024-03-06T07:26:19.720104+0000","last_clean":"2024-03-06T07:26:19.720104+0000","last_became_active":"2024-03-06T07:21:47.552664+0000","last_became_peered":"2024-03-06T07:21:47.552664+0000","last_unstale":"2024-03-06T07:26:19.720104+0000","last_undegraded":"2024-03-06T07:26:19.720104+0000","last_fullsized":"2024-03-06T07:26:19.720104+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_clean_scrub_stamp":"2024-03-06T07:21:46.362028+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:02.916426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574908,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32900,"kb_used_data":1712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338940,"statfs":{"total":96636764160,"available":96603074560,"internally_reserved":0,"allocated":1753088,"data_stored":1080864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574906,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32296,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339544,"statfs":{"total":96636764160,"available":96603693056,"internally_reserved":0,"allocated":1138688,"data_stored":483609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574907,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32316,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339524,"statfs":{"total":96636764160,"available":96603672576,"internally_reserved":0,"allocated":1155072,"data_stored":490454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574908,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32296,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339544,"statfs":{"total":96636764160,"available":96603693056,"internally_reserved":0,"allocated":1138688,"data_stored":483609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32328,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339512,"statfs":{"total":96636764160,"available":96603660288,"internally_reserved":0,"allocated":1167360,"data_stored":492018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574907,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32316,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339524,"statfs":{"total":96636764160,"available":96603672576,"internally_reserved":0,"allocated":1155072,"data_stored":490418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574908,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32312,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339528,"statfs":{"total":96636764160,"available":96603676672,"internally_reserved":0,"allocated":1150976,"data_stored":490384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32892,"kb_used_data":1708,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338948,"statfs":{"total":96636764160,"available":96603082752,"internally_reserved":0,"allocated":1748992,"data_stored":1081334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:26:23.119 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:26:23.119 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:26:23.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.231+0000 7fb149d20640 1 Processor -- start 2024-03-06T07:26:23.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.231+0000 7fb149d20640 1 -- start start 2024-03-06T07:26:23.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.232+0000 7fb149d20640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb1440cf9a0 0x7fb1440b5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.232+0000 7fb149d20640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1440b55e0 0x7fb1440ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.232+0000 7fb149d20640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb14406b710 con 0x7fb144066d90 2024-03-06T07:26:23.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.232+0000 7fb149d20640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb14406b150 con 0x7fb1440cf9a0 2024-03-06T07:26:23.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.232+0000 7fb149d20640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb14406b430 con 0x7fb1440b55e0 2024-03-06T07:26:23.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.232+0000 7fb1437fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb1440cf9a0 0x7fb1440b5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.232+0000 7fb142ffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fb144066d90 0x7fb1440b4ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:35428/0 (socket says 172.21.15.83:35428) 2024-03-06T07:26:23.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.232+0000 7fb142ffd640 1 -- 172.21.15.83:0/3543307516 learned_addr learned my addr 172.21.15.83:0/3543307516 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:23.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.232+0000 7fb143fff640 1 --2- 172.21.15.83:0/3543307516 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1440b55e0 0x7fb1440ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.232+0000 7fb1437fe640 1 -- 172.21.15.83:0/3543307516 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1440b55e0 msgr2=0x7fb1440ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:23.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.232+0000 7fb1437fe640 1 --2- 172.21.15.83:0/3543307516 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1440b55e0 0x7fb1440ce040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.232+0000 7fb1437fe640 1 -- 172.21.15.83:0/3543307516 >> v1:172.21.15.83:6789/0 conn(0x7fb144066d90 legacy=0x7fb1440b4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.232+0000 7fb1437fe640 1 -- 172.21.15.83:0/3543307516 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1440b0970 con 0x7fb1440cf9a0 2024-03-06T07:26:23.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.233+0000 7fb1437fe640 1 --2- 172.21.15.83:0/3543307516 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1440cf9a0 0x7fb1440b5ab0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fb1380097a0 tx=0x7fb13802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f383d0e5d803e068 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:23.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.233+0000 7fb1427fc640 1 -- 172.21.15.83:0/3543307516 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb138035020 con 0x7fb1440cf9a0 2024-03-06T07:26:23.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.233+0000 7fb1427fc640 1 -- 172.21.15.83:0/3543307516 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb138031810 con 0x7fb1440cf9a0 2024-03-06T07:26:23.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.233+0000 7fb1427fc640 1 -- 172.21.15.83:0/3543307516 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb138031be0 con 0x7fb1440cf9a0 2024-03-06T07:26:23.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.233+0000 7fb149d20640 1 -- 172.21.15.83:0/3543307516 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1440cf9a0 msgr2=0x7fb1440b5ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.233+0000 7fb149d20640 1 --2- 172.21.15.83:0/3543307516 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1440cf9a0 0x7fb1440b5ab0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fb1380097a0 tx=0x7fb13802b8b0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.233+0000 7fb149d20640 1 -- 172.21.15.83:0/3543307516 shutdown_connections 2024-03-06T07:26:23.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.233+0000 7fb149d20640 1 --2- 172.21.15.83:0/3543307516 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1440b55e0 0x7fb1440ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.233+0000 7fb149d20640 1 --2- 172.21.15.83:0/3543307516 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1440cf9a0 0x7fb1440b5ab0 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.233+0000 7fb149d20640 1 -- 172.21.15.83:0/3543307516 >> 172.21.15.83:0/3543307516 conn(0x7fb1440b1d70 msgr2=0x7fb1440b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:23.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.233+0000 7fb149d20640 1 -- 172.21.15.83:0/3543307516 shutdown_connections 2024-03-06T07:26:23.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.233+0000 7fb149d20640 1 -- 172.21.15.83:0/3543307516 wait complete. 2024-03-06T07:26:23.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.235+0000 7fb149d20640 1 Processor -- start 2024-03-06T07:26:23.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.235+0000 7fb149d20640 1 -- start start 2024-03-06T07:26:23.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.235+0000 7fb149d20640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb144066d90 0x7fb144160100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.235+0000 7fb149d20640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1440b55e0 0x7fb1440caac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.235+0000 7fb149d20640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1440cf9a0 0x7fb1440cb000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.235+0000 7fb149d20640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1440b6170 con 0x7fb1440cf9a0 2024-03-06T07:26:23.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.235+0000 7fb149d20640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1440b5ff0 con 0x7fb144066d90 2024-03-06T07:26:23.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.235+0000 7fb149d20640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1440cfd80 con 0x7fb1440b55e0 2024-03-06T07:26:23.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.236+0000 7fb143fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1440cf9a0 0x7fb1440cb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.236+0000 7fb143fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1440cf9a0 0x7fb1440cb000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50514/0 (socket says 172.21.15.83:50514) 2024-03-06T07:26:23.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.236+0000 7fb143fff640 1 -- 172.21.15.83:0/2664686972 learned_addr learned my addr 172.21.15.83:0/2664686972 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:23.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.236+0000 7fb1437fe640 1 --2- 172.21.15.83:0/2664686972 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb144066d90 0x7fb144160100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.236+0000 7fb142ffd640 1 --2- 172.21.15.83:0/2664686972 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1440b55e0 0x7fb1440caac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.236+0000 7fb143fff640 1 -- 172.21.15.83:0/2664686972 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1440b55e0 msgr2=0x7fb1440caac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.236+0000 7fb143fff640 1 --2- 172.21.15.83:0/2664686972 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1440b55e0 0x7fb1440caac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.236+0000 7fb143fff640 1 -- 172.21.15.83:0/2664686972 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb144066d90 msgr2=0x7fb144160100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.236+0000 7fb143fff640 1 --2- 172.21.15.83:0/2664686972 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb144066d90 0x7fb144160100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.236+0000 7fb143fff640 1 -- 172.21.15.83:0/2664686972 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb144160cf0 con 0x7fb1440cf9a0 2024-03-06T07:26:23.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.236+0000 7fb143fff640 1 --2- 172.21.15.83:0/2664686972 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1440cf9a0 0x7fb1440cb000 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fb13c007e60 tx=0x7fb13c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:23.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.236+0000 7fb12bfff640 1 -- 172.21.15.83:0/2664686972 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13c014620 con 0x7fb1440cf9a0 2024-03-06T07:26:23.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.236+0000 7fb12bfff640 1 -- 172.21.15.83:0/2664686972 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb13c019070 con 0x7fb1440cf9a0 2024-03-06T07:26:23.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.236+0000 7fb12bfff640 1 -- 172.21.15.83:0/2664686972 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13c014a20 con 0x7fb1440cf9a0 2024-03-06T07:26:23.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.236+0000 7fb149d20640 1 -- 172.21.15.83:0/2664686972 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1440cb630 con 0x7fb1440cf9a0 2024-03-06T07:26:23.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.236+0000 7fb149d20640 1 -- 172.21.15.83:0/2664686972 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1440cbbf0 con 0x7fb1440cf9a0 2024-03-06T07:26:23.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.237+0000 7fb149d20640 1 -- 172.21.15.83:0/2664686972 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb14409e3d0 con 0x7fb1440cf9a0 2024-03-06T07:26:23.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.238+0000 7fb12bfff640 1 -- 172.21.15.83:0/2664686972 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb13c014bc0 con 0x7fb1440cf9a0 2024-03-06T07:26:23.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.238+0000 7fb12bfff640 1 --2- 172.21.15.83:0/2664686972 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb13006b4d0 0x7fb13006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.238+0000 7fb12bfff640 1 -- 172.21.15.83:0/2664686972 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb13c08a9a0 con 0x7fb1440cf9a0 2024-03-06T07:26:23.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.238+0000 7fb1437fe640 1 --2- 172.21.15.83:0/2664686972 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb13006b4d0 0x7fb13006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.239+0000 7fb1437fe640 1 --2- 172.21.15.83:0/2664686972 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb13006b4d0 0x7fb13006d990 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fb138036000 tx=0x7fb13803c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:23.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.242+0000 7fb12bfff640 1 -- 172.21.15.83:0/2664686972 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb13c059820 con 0x7fb1440cf9a0 2024-03-06T07:26:23.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.518+0000 7fb149d20640 1 -- 172.21.15.83:0/2664686972 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fb1440ce300 con 0x7fb1440cf9a0 2024-03-06T07:26:23.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.520+0000 7fb12bfff640 1 -- 172.21.15.83:0/2664686972 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) v1 ==== 190+0+147 (secure 0 0 0) 0x7fb13c059120 con 0x7fb1440cf9a0 2024-03-06T07:26:23.522 INFO:teuthology.orchestra.run.smithi083.stdout:[client.0] 2024-03-06T07:26:23.523 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBcGehlRwCIMhAA1DdbdQZPplu7LQOTK3BmSg== 2024-03-06T07:26:23.523 INFO:teuthology.orchestra.run.smithi083.stdout: caps mds = "allow" 2024-03-06T07:26:23.523 INFO:teuthology.orchestra.run.smithi083.stdout: caps mon = "allow r" 2024-03-06T07:26:23.523 INFO:teuthology.orchestra.run.smithi083.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:26:23.523 INFO:teuthology.orchestra.run.smithi083.stderr:updated caps for client.0 2024-03-06T07:26:23.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.522+0000 7fb149d20640 1 -- 172.21.15.83:0/2664686972 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb13006b4d0 msgr2=0x7fb13006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.522+0000 7fb149d20640 1 --2- 172.21.15.83:0/2664686972 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb13006b4d0 0x7fb13006d990 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fb138036000 tx=0x7fb13803c040 comp rx=0 tx=0).stop 2024-03-06T07:26:23.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.522+0000 7fb149d20640 1 -- 172.21.15.83:0/2664686972 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1440cf9a0 msgr2=0x7fb1440cb000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.522+0000 7fb149d20640 1 --2- 172.21.15.83:0/2664686972 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1440cf9a0 0x7fb1440cb000 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fb13c007e60 tx=0x7fb13c00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.522+0000 7fb149d20640 1 -- 172.21.15.83:0/2664686972 shutdown_connections 2024-03-06T07:26:23.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.522+0000 7fb149d20640 1 --2- 172.21.15.83:0/2664686972 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb13006b4d0 0x7fb13006d990 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.522+0000 7fb149d20640 1 --2- 172.21.15.83:0/2664686972 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1440cf9a0 0x7fb1440cb000 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.522+0000 7fb149d20640 1 --2- 172.21.15.83:0/2664686972 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1440b55e0 0x7fb1440caac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.522+0000 7fb149d20640 1 --2- 172.21.15.83:0/2664686972 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb144066d90 0x7fb144160100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.523+0000 7fb149d20640 1 -- 172.21.15.83:0/2664686972 >> 172.21.15.83:0/2664686972 conn(0x7fb1440b1d70 msgr2=0x7fb1440af060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:23.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.523+0000 7fb149d20640 1 -- 172.21.15.83:0/2664686972 shutdown_connections 2024-03-06T07:26:23.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.523+0000 7fb149d20640 1 -- 172.21.15.83:0/2664686972 wait complete. 2024-03-06T07:26:23.536 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:23.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.634+0000 7fe0aade9640 1 Processor -- start 2024-03-06T07:26:23.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.634+0000 7fe0aade9640 1 -- start start 2024-03-06T07:26:23.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.635+0000 7fe0aade9640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fe0a415dfe0 0x7fe0a40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.635+0000 7fe0aade9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe0a40cf830 0x7fe0a40d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.635+0000 7fe0aade9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a40b3430 con 0x7fe0a40d27f0 2024-03-06T07:26:23.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.635+0000 7fe0aade9640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0a40b31e0 con 0x7fe0a40cf830 2024-03-06T07:26:23.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.635+0000 7fe0aade9640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0a40b3d60 con 0x7fe0a415dfe0 2024-03-06T07:26:23.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.635+0000 7fe0a3fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe0a40cf830 0x7fe0a40d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.635+0000 7fe0a3fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe0a40cf830 0x7fe0a40d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50524/0 (socket says 172.21.15.83:50524) 2024-03-06T07:26:23.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.635+0000 7fe0a3fff640 1 -- 172.21.15.83:0/1318627340 learned_addr learned my addr 172.21.15.83:0/1318627340 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:23.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.636+0000 7fe0a3fff640 1 -- 172.21.15.83:0/1318627340 >> v2:172.21.15.183:3300/0 conn(0x7fe0a415dfe0 msgr2=0x7fe0a40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:23.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.636+0000 7fe0a8b85640 1 --2- 172.21.15.83:0/1318627340 >> v2:172.21.15.183:3300/0 conn(0x7fe0a415dfe0 0x7fe0a40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.636+0000 7fe0a3fff640 1 --2- 172.21.15.83:0/1318627340 >> v2:172.21.15.183:3300/0 conn(0x7fe0a415dfe0 0x7fe0a40cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.636+0000 7fe0a3fff640 1 -- 172.21.15.83:0/1318627340 >> v1:172.21.15.183:6789/0 conn(0x7fe0a40d27f0 legacy=0x7fe0a40d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:23.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.636+0000 7fe0a3fff640 1 -- 172.21.15.83:0/1318627340 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a40b3710 con 0x7fe0a40cf830 2024-03-06T07:26:23.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.636+0000 7fe0a8b85640 1 --2- 172.21.15.83:0/1318627340 >> v2:172.21.15.183:3300/0 conn(0x7fe0a415dfe0 0x7fe0a40cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:23.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.637+0000 7fe0a3fff640 1 --2- 172.21.15.83:0/1318627340 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0a40cf830 0x7fe0a40d22b0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fe090009870 tx=0x7fe09002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ec8384ddab1985e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:23.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.637+0000 7fe0a37fe640 1 -- 172.21.15.83:0/1318627340 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe090035020 con 0x7fe0a40cf830 2024-03-06T07:26:23.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.637+0000 7fe0a37fe640 1 -- 172.21.15.83:0/1318627340 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe090031870 con 0x7fe0a40cf830 2024-03-06T07:26:23.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.637+0000 7fe0a37fe640 1 -- 172.21.15.83:0/1318627340 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe090031c40 con 0x7fe0a40cf830 2024-03-06T07:26:23.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.638+0000 7fe0aade9640 1 -- 172.21.15.83:0/1318627340 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0a40cf830 msgr2=0x7fe0a40d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.638+0000 7fe0aade9640 1 --2- 172.21.15.83:0/1318627340 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0a40cf830 0x7fe0a40d22b0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fe090009870 tx=0x7fe09002b910 comp rx=0 tx=0).stop 2024-03-06T07:26:23.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.638+0000 7fe0aade9640 1 -- 172.21.15.83:0/1318627340 shutdown_connections 2024-03-06T07:26:23.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.638+0000 7fe0aade9640 1 --2- 172.21.15.83:0/1318627340 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0a40cf830 0x7fe0a40d22b0 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.638+0000 7fe0aade9640 1 --2- 172.21.15.83:0/1318627340 >> v2:172.21.15.183:3300/0 conn(0x7fe0a415dfe0 0x7fe0a40cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.638+0000 7fe0aade9640 1 -- 172.21.15.83:0/1318627340 >> 172.21.15.83:0/1318627340 conn(0x7fe0a40d3cd0 msgr2=0x7fe0a40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:23.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.638+0000 7fe0aade9640 1 -- 172.21.15.83:0/1318627340 shutdown_connections 2024-03-06T07:26:23.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.638+0000 7fe0aade9640 1 -- 172.21.15.83:0/1318627340 wait complete. 2024-03-06T07:26:23.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.639+0000 7fe0aade9640 1 Processor -- start 2024-03-06T07:26:23.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.639+0000 7fe0aade9640 1 -- start start 2024-03-06T07:26:23.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.641+0000 7fe0aade9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0a40cf830 0x7fe0a415bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.641+0000 7fe0aade9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe0a40d27f0 0x7fe0a4160230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.641+0000 7fe0aade9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe0a415dfe0 0x7fe0a40ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.641+0000 7fe0aade9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a40d1df0 con 0x7fe0a40cf830 2024-03-06T07:26:23.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.641+0000 7fe0aade9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a40d1c70 con 0x7fe0a40d27f0 2024-03-06T07:26:23.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.641+0000 7fe0aade9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a415e3c0 con 0x7fe0a415dfe0 2024-03-06T07:26:23.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.641+0000 7fe0a8b85640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0a40cf830 0x7fe0a415bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.641+0000 7fe0a8b85640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0a40cf830 0x7fe0a415bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50530/0 (socket says 172.21.15.83:50530) 2024-03-06T07:26:23.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.641+0000 7fe0a8b85640 1 -- 172.21.15.83:0/1255567079 learned_addr learned my addr 172.21.15.83:0/1255567079 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:23.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.641+0000 7fe0a8b85640 1 -- 172.21.15.83:0/1255567079 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe0a415dfe0 msgr2=0x7fe0a40ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.641+0000 7fe0a8b85640 1 --2- 172.21.15.83:0/1255567079 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe0a415dfe0 0x7fe0a40ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.641+0000 7fe0a8b85640 1 -- 172.21.15.83:0/1255567079 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe0a40d27f0 msgr2=0x7fe0a4160230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.641+0000 7fe0a8b85640 1 --2- 172.21.15.83:0/1255567079 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe0a40d27f0 0x7fe0a4160230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.641+0000 7fe0a8b85640 1 -- 172.21.15.83:0/1255567079 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a41f3e90 con 0x7fe0a40cf830 2024-03-06T07:26:23.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.641+0000 7fe0a8b85640 1 --2- 172.21.15.83:0/1255567079 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0a40cf830 0x7fe0a415bd80 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fe09800c7f0 tx=0x7fe09800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:23.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.641+0000 7fe0a17fa640 1 -- 172.21.15.83:0/1255567079 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe098015070 con 0x7fe0a40cf830 2024-03-06T07:26:23.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.642+0000 7fe0a17fa640 1 -- 172.21.15.83:0/1255567079 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0980079d0 con 0x7fe0a40cf830 2024-03-06T07:26:23.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.642+0000 7fe0aade9640 1 -- 172.21.15.83:0/1255567079 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0a41f4130 con 0x7fe0a40cf830 2024-03-06T07:26:23.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.642+0000 7fe0a17fa640 1 -- 172.21.15.83:0/1255567079 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe098007da0 con 0x7fe0a40cf830 2024-03-06T07:26:23.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.642+0000 7fe0aade9640 1 -- 172.21.15.83:0/1255567079 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0a41f46f0 con 0x7fe0a40cf830 2024-03-06T07:26:23.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.642+0000 7fe0aade9640 1 -- 172.21.15.83:0/1255567079 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0a404f330 con 0x7fe0a40cf830 2024-03-06T07:26:23.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.643+0000 7fe0a17fa640 1 -- 172.21.15.83:0/1255567079 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe098018070 con 0x7fe0a40cf830 2024-03-06T07:26:23.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.643+0000 7fe0a17fa640 1 --2- 172.21.15.83:0/1255567079 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe08806b4d0 0x7fe08806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.644+0000 7fe0a3fff640 1 --2- 172.21.15.83:0/1255567079 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe08806b4d0 0x7fe08806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.644+0000 7fe0a17fa640 1 -- 172.21.15.83:0/1255567079 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe09808a0b0 con 0x7fe0a40cf830 2024-03-06T07:26:23.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.644+0000 7fe0a3fff640 1 --2- 172.21.15.83:0/1255567079 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe08806b4d0 0x7fe08806d990 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fe090036000 tx=0x7fe090030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:23.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.648+0000 7fe0a17fa640 1 -- 172.21.15.83:0/1255567079 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe098058fc0 con 0x7fe0a40cf830 2024-03-06T07:26:23.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.865+0000 7fe0aade9640 1 -- 172.21.15.83:0/1255567079 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0a40b09c0 con 0x7fe0a40cf830 2024-03-06T07:26:23.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.865+0000 7fe0a17fa640 1 -- 172.21.15.83:0/1255567079 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4985 (secure 0 0 0) 0x7fe0980588c0 con 0x7fe0a40cf830 2024-03-06T07:26:23.867 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:23.868 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":45,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5924,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/2933830298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2933830298},{"type":"v1","addr":"172.21.15.183:6834","nonce":2933830298}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":6490,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1025428736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":1025428736},{"type":"v1","addr":"172.21.15.183:6835","nonce":1025428736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":7335,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/3634489597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3634489597},{"type":"v1","addr":"172.21.15.83:6836","nonce":3634489597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.064335+0000","modified":"2024-03-06T07:26:21.071371+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7338},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7338":{"gid":7338,"name":"a","rank":0,"incarnation":43,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6837/2924411033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":2924411033},{"type":"v1","addr":"172.21.15.83:6837","nonce":2924411033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7338,"qdb_cluster":[7338]},"id":7}]} 2024-03-06T07:26:23.868 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 45 2024-03-06T07:26:23.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.867+0000 7fe0aade9640 1 -- 172.21.15.83:0/1255567079 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe08806b4d0 msgr2=0x7fe08806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.867+0000 7fe0aade9640 1 --2- 172.21.15.83:0/1255567079 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe08806b4d0 0x7fe08806d990 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fe090036000 tx=0x7fe090030000 comp rx=0 tx=0).stop 2024-03-06T07:26:23.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.867+0000 7fe0aade9640 1 -- 172.21.15.83:0/1255567079 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0a40cf830 msgr2=0x7fe0a415bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.867+0000 7fe0aade9640 1 --2- 172.21.15.83:0/1255567079 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0a40cf830 0x7fe0a415bd80 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fe09800c7f0 tx=0x7fe09800cbf0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.867+0000 7fe0aade9640 1 -- 172.21.15.83:0/1255567079 shutdown_connections 2024-03-06T07:26:23.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.867+0000 7fe0aade9640 1 --2- 172.21.15.83:0/1255567079 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe08806b4d0 0x7fe08806d990 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.867+0000 7fe0aade9640 1 --2- 172.21.15.83:0/1255567079 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe0a415dfe0 0x7fe0a40ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.867+0000 7fe0aade9640 1 --2- 172.21.15.83:0/1255567079 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe0a40d27f0 0x7fe0a4160230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.867+0000 7fe0aade9640 1 --2- 172.21.15.83:0/1255567079 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0a40cf830 0x7fe0a415bd80 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.867+0000 7fe0aade9640 1 -- 172.21.15.83:0/1255567079 >> 172.21.15.83:0/1255567079 conn(0x7fe0a40d3cd0 msgr2=0x7fe0a40d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:23.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.867+0000 7fe0aade9640 1 -- 172.21.15.83:0/1255567079 shutdown_connections 2024-03-06T07:26:23.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:23.867+0000 7fe0aade9640 1 -- 172.21.15.83:0/1255567079 wait complete. 2024-03-06T07:26:24.282 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 45, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:26:19.064335+0000', 'modified': '2024-03-06T07:26:21.071371+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7338}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7338': {'gid': 7338, 'name': 'a', 'rank': 0, 'incarnation': 43, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.83:6837/2924411033', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 2924411033}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 2924411033}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7338, 'qdb_cluster': [7338]} 2024-03-06T07:26:24.282 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:26:24.283 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:26:24.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.384+0000 7f5c4fb5c640 1 Processor -- start 2024-03-06T07:26:24.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.384+0000 7f5c4fb5c640 1 -- start start 2024-03-06T07:26:24.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.385+0000 7f5c4fb5c640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f5c480b3240 0x7f5c480b23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.385+0000 7f5c4fb5c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c480b2910 0x7f5c480d1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.385+0000 7f5c4fb5c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c4815e3f0 con 0x7f5c4815e010 2024-03-06T07:26:24.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.385+0000 7f5c4fb5c640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c480b1f40 con 0x7f5c480b3240 2024-03-06T07:26:24.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.385+0000 7f5c4fb5c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c480b1c10 con 0x7f5c480b2910 2024-03-06T07:26:24.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.385+0000 7f5c4d8f8640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f5c4815e010 0x7f5c480cedd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:35440/0 (socket says 172.21.15.83:35440) 2024-03-06T07:26:24.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.385+0000 7f5c4d8f8640 1 -- 172.21.15.83:0/3972839395 learned_addr learned my addr 172.21.15.83:0/3972839395 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:24.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.385+0000 7f5c4e0f9640 1 --2- 172.21.15.83:0/3972839395 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c480b2910 0x7f5c480d1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.385+0000 7f5c4d0f7640 1 --2- 172.21.15.83:0/3972839395 >> v2:172.21.15.183:3300/0 conn(0x7f5c480b3240 0x7f5c480b23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.385+0000 7f5c4c8f6640 1 -- 172.21.15.83:0/3972839395 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2775293311 0 0) 0x7f5c4815e3f0 con 0x7f5c4815e010 2024-03-06T07:26:24.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.385+0000 7f5c4c8f6640 1 -- 172.21.15.83:0/3972839395 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c34003520 con 0x7f5c4815e010 2024-03-06T07:26:24.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.385+0000 7f5c4c8f6640 1 -- 172.21.15.83:0/3972839395 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 148540671 0 0) 0x7f5c34003520 con 0x7f5c4815e010 2024-03-06T07:26:24.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.385+0000 7f5c4c8f6640 1 -- 172.21.15.83:0/3972839395 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c480b2910 msgr2=0x7f5c480d1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:24.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.385+0000 7f5c4c8f6640 1 --2- 172.21.15.83:0/3972839395 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c480b2910 0x7f5c480d1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.385+0000 7f5c4c8f6640 1 -- 172.21.15.83:0/3972839395 >> v2:172.21.15.183:3300/0 conn(0x7f5c480b3240 msgr2=0x7f5c480b23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:24.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.385+0000 7f5c4c8f6640 1 --2- 172.21.15.83:0/3972839395 >> v2:172.21.15.183:3300/0 conn(0x7f5c480b3240 0x7f5c480b23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.385+0000 7f5c4c8f6640 1 -- 172.21.15.83:0/3972839395 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c4806d1b0 con 0x7f5c4815e010 2024-03-06T07:26:24.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.385+0000 7f5c4c8f6640 1 -- 172.21.15.83:0/3972839395 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f5c44001670 con 0x7f5c4815e010 2024-03-06T07:26:24.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.385+0000 7f5c4c8f6640 1 -- 172.21.15.83:0/3972839395 >> v1:172.21.15.83:6789/0 conn(0x7f5c4815e010 legacy=0x7f5c480cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.385+0000 7f5c4c8f6640 1 --2- 172.21.15.83:0/3972839395 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c340039b0 0x7f5c34003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.385+0000 7f5c4c8f6640 1 --2- 172.21.15.83:0/3972839395 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c34004800 0x7f5c34004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.385+0000 7f5c4c8f6640 1 --2- 172.21.15.83:0/3972839395 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5c34005180 0x7f5c3400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.385+0000 7f5c4c8f6640 1 -- 172.21.15.83:0/3972839395 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c480b1f40 con 0x7f5c340039b0 2024-03-06T07:26:24.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.385+0000 7f5c4c8f6640 1 -- 172.21.15.83:0/3972839395 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c480b1c10 con 0x7f5c34005180 2024-03-06T07:26:24.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.385+0000 7f5c4c8f6640 1 -- 172.21.15.83:0/3972839395 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3400bf00 con 0x7f5c34004800 2024-03-06T07:26:24.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.385+0000 7f5c4d8f8640 1 --2- 172.21.15.83:0/3972839395 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c340039b0 0x7f5c34003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.386+0000 7f5c4d0f7640 1 --2- 172.21.15.83:0/3972839395 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c34004800 0x7f5c34004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.386+0000 7f5c4e0f9640 1 --2- 172.21.15.83:0/3972839395 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5c34005180 0x7f5c3400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.386+0000 7f5c4d8f8640 1 -- 172.21.15.83:0/3972839395 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c34004800 msgr2=0x7f5c34004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:24.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.386+0000 7f5c4d8f8640 1 --2- 172.21.15.83:0/3972839395 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c34004800 0x7f5c34004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.386+0000 7f5c4d8f8640 1 -- 172.21.15.83:0/3972839395 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5c34005180 msgr2=0x7f5c3400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:24.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.386+0000 7f5c4d8f8640 1 --2- 172.21.15.83:0/3972839395 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5c34005180 0x7f5c3400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.386+0000 7f5c4d8f8640 1 -- 172.21.15.83:0/3972839395 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c34003520 con 0x7f5c340039b0 2024-03-06T07:26:24.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.386+0000 7f5c4d8f8640 1 --2- 172.21.15.83:0/3972839395 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c340039b0 0x7f5c34003dd0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f5c44001050 tx=0x7f5c34005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dff17bc84e28d7a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:24.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.387+0000 7f5c4c8f6640 1 -- 172.21.15.83:0/3972839395 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c44007d70 con 0x7f5c340039b0 2024-03-06T07:26:24.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.387+0000 7f5c4c8f6640 1 -- 172.21.15.83:0/3972839395 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c44030070 con 0x7f5c340039b0 2024-03-06T07:26:24.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.387+0000 7f5c4c8f6640 1 -- 172.21.15.83:0/3972839395 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4402cd80 con 0x7f5c340039b0 2024-03-06T07:26:24.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.387+0000 7f5c4fb5c640 1 -- 172.21.15.83:0/3972839395 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c340039b0 msgr2=0x7f5c34003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.387+0000 7f5c4fb5c640 1 --2- 172.21.15.83:0/3972839395 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c340039b0 0x7f5c34003dd0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f5c44001050 tx=0x7f5c34005830 comp rx=0 tx=0).stop 2024-03-06T07:26:24.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.387+0000 7f5c4e0f9640 1 -- 172.21.15.83:0/3972839395 reap_dead start 2024-03-06T07:26:24.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.387+0000 7f5c4fb5c640 1 -- 172.21.15.83:0/3972839395 shutdown_connections 2024-03-06T07:26:24.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.387+0000 7f5c4fb5c640 1 -- 172.21.15.83:0/3972839395 >> 172.21.15.83:0/3972839395 conn(0x7f5c4809a9f0 msgr2=0x7f5c4809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:24.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.387+0000 7f5c4fb5c640 1 -- 172.21.15.83:0/3972839395 shutdown_connections 2024-03-06T07:26:24.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.387+0000 7f5c4fb5c640 1 -- 172.21.15.83:0/3972839395 wait complete. 2024-03-06T07:26:24.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.389+0000 7f5c4fb5c640 1 Processor -- start 2024-03-06T07:26:24.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.389+0000 7f5c4fb5c640 1 -- start start 2024-03-06T07:26:24.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.390+0000 7f5c4fb5c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5c340039b0 0x7f5c481e7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.390+0000 7f5c4fb5c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c480b6b10 0x7f5c481e9d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.390+0000 7f5c4fb5c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c480b6f90 0x7f5c481ea290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.390+0000 7f5c4fb5c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c480d1ad0 con 0x7f5c480b6b10 2024-03-06T07:26:24.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.390+0000 7f5c4fb5c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4806ced0 con 0x7f5c340039b0 2024-03-06T07:26:24.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.390+0000 7f5c4fb5c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c480d1c50 con 0x7f5c480b6f90 2024-03-06T07:26:24.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.390+0000 7f5c4d0f7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c480b6b10 0x7f5c481e9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.390+0000 7f5c4d0f7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c480b6b10 0x7f5c481e9d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50554/0 (socket says 172.21.15.83:50554) 2024-03-06T07:26:24.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.390+0000 7f5c4d0f7640 1 -- 172.21.15.83:0/44736411 learned_addr learned my addr 172.21.15.83:0/44736411 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:24.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.391+0000 7f5c4d8f8640 1 --2- 172.21.15.83:0/44736411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5c340039b0 0x7f5c481e7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.391+0000 7f5c4e0f9640 1 --2- 172.21.15.83:0/44736411 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c480b6f90 0x7f5c481ea290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.391+0000 7f5c4d0f7640 1 -- 172.21.15.83:0/44736411 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c480b6f90 msgr2=0x7f5c481ea290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.391+0000 7f5c4d0f7640 1 --2- 172.21.15.83:0/44736411 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c480b6f90 0x7f5c481ea290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.391+0000 7f5c4d0f7640 1 -- 172.21.15.83:0/44736411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5c340039b0 msgr2=0x7f5c481e7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.391+0000 7f5c4d0f7640 1 --2- 172.21.15.83:0/44736411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5c340039b0 0x7f5c481e7170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.391+0000 7f5c4d0f7640 1 -- 172.21.15.83:0/44736411 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c481e76b0 con 0x7f5c480b6b10 2024-03-06T07:26:24.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.391+0000 7f5c4e0f9640 1 --2- 172.21.15.83:0/44736411 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c480b6f90 0x7f5c481ea290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:24.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.391+0000 7f5c4d8f8640 1 --2- 172.21.15.83:0/44736411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5c340039b0 0x7f5c481e7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:24.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.391+0000 7f5c4d0f7640 1 --2- 172.21.15.83:0/44736411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c480b6b10 0x7f5c481e9d50 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f5c38011960 tx=0x7f5c38011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:24.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.391+0000 7f5c3e7fc640 1 -- 172.21.15.83:0/44736411 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3800faf0 con 0x7f5c480b6b10 2024-03-06T07:26:24.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.391+0000 7f5c3e7fc640 1 -- 172.21.15.83:0/44736411 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c3800fc90 con 0x7f5c480b6b10 2024-03-06T07:26:24.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.391+0000 7f5c3e7fc640 1 -- 172.21.15.83:0/44736411 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c38014440 con 0x7f5c480b6b10 2024-03-06T07:26:24.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.391+0000 7f5c4fb5c640 1 -- 172.21.15.83:0/44736411 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c480b4990 con 0x7f5c480b6b10 2024-03-06T07:26:24.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.391+0000 7f5c4fb5c640 1 -- 172.21.15.83:0/44736411 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c480b4f50 con 0x7f5c480b6b10 2024-03-06T07:26:24.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.393+0000 7f5c3e7fc640 1 -- 172.21.15.83:0/44736411 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5c38012040 con 0x7f5c480b6b10 2024-03-06T07:26:24.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.393+0000 7f5c3e7fc640 1 --2- 172.21.15.83:0/44736411 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5c2c06b4d0 0x7f5c2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.393+0000 7f5c4d8f8640 1 --2- 172.21.15.83:0/44736411 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5c2c06b4d0 0x7f5c2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.393+0000 7f5c3e7fc640 1 -- 172.21.15.83:0/44736411 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5c3808a780 con 0x7f5c480b6b10 2024-03-06T07:26:24.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.394+0000 7f5c4d8f8640 1 --2- 172.21.15.83:0/44736411 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5c2c06b4d0 0x7f5c2c06d990 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f5c44034000 tx=0x7f5c44007000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:24.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.394+0000 7f5c52815740 1 Processor -- start 2024-03-06T07:26:24.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.395+0000 7f5c52815740 1 -- start start 2024-03-06T07:26:24.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.395+0000 7f5c52815740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cab31f9870 0x55cab31f9c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.395+0000 7f5c52815740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55cab31fd8c0 0x55cab31ff360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.395+0000 7f5c52815740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cab3229600 0x55cab3229a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.395+0000 7f5c52815740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cab31636b0 con 0x55cab3229600 2024-03-06T07:26:24.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.395+0000 7f5c52815740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cab310ca60 con 0x55cab31f9870 2024-03-06T07:26:24.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.395+0000 7f5c52815740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cab315f750 con 0x55cab31fd8c0 2024-03-06T07:26:24.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.395+0000 7f5c4e0f9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cab3229600 0x55cab3229a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.395+0000 7f5c4e0f9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cab3229600 0x55cab3229a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50566/0 (socket says 172.21.15.83:50566) 2024-03-06T07:26:24.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.395+0000 7f5c4e0f9640 1 -- 172.21.15.83:0/2297725196 learned_addr learned my addr 172.21.15.83:0/2297725196 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:24.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.395+0000 7f5c4d8f8640 1 --2- 172.21.15.83:0/2297725196 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cab31f9870 0x55cab31f9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.395+0000 7f5c4d0f7640 1 --2- 172.21.15.83:0/2297725196 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55cab31fd8c0 0x55cab31ff360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.396+0000 7f5c4e0f9640 1 -- 172.21.15.83:0/2297725196 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55cab31fd8c0 msgr2=0x55cab31ff360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:24.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.396+0000 7f5c4e0f9640 1 --2- 172.21.15.83:0/2297725196 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55cab31fd8c0 0x55cab31ff360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.396+0000 7f5c4e0f9640 1 -- 172.21.15.83:0/2297725196 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cab31f9870 msgr2=0x55cab31f9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:24.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.396+0000 7f5c4e0f9640 1 --2- 172.21.15.83:0/2297725196 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cab31f9870 0x55cab31f9c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.396+0000 7f5c4e0f9640 1 -- 172.21.15.83:0/2297725196 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cab312af30 con 0x55cab3229600 2024-03-06T07:26:24.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.396+0000 7f5c4d0f7640 1 --2- 172.21.15.83:0/2297725196 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55cab31fd8c0 0x55cab31ff360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:24.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.396+0000 7f5c4d8f8640 1 --2- 172.21.15.83:0/2297725196 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cab31f9870 0x55cab31f9c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:24.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.396+0000 7f5c4e0f9640 1 --2- 172.21.15.83:0/2297725196 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cab3229600 0x55cab3229a80 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f5c4806e8e0 tx=0x7f5c40009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec715742586e0564 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:24.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.396+0000 7f5c2b7fe640 1 -- 172.21.15.83:0/2297725196 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c40003da0 con 0x55cab3229600 2024-03-06T07:26:24.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.396+0000 7f5c2b7fe640 1 -- 172.21.15.83:0/2297725196 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c4001d070 con 0x55cab3229600 2024-03-06T07:26:24.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.396+0000 7f5c2b7fe640 1 -- 172.21.15.83:0/2297725196 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c40004230 con 0x55cab3229600 2024-03-06T07:26:24.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.396+0000 7f5c52815740 1 -- 172.21.15.83:0/2297725196 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cab3229600 msgr2=0x55cab3229a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.396+0000 7f5c52815740 1 --2- 172.21.15.83:0/2297725196 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cab3229600 0x55cab3229a80 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f5c4806e8e0 tx=0x7f5c40009750 comp rx=0 tx=0).stop 2024-03-06T07:26:24.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.396+0000 7f5c52815740 1 -- 172.21.15.83:0/2297725196 shutdown_connections 2024-03-06T07:26:24.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.396+0000 7f5c52815740 1 --2- 172.21.15.83:0/2297725196 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cab3229600 0x55cab3229a80 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.397+0000 7f5c52815740 1 --2- 172.21.15.83:0/2297725196 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55cab31fd8c0 0x55cab31ff360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.397+0000 7f5c52815740 1 --2- 172.21.15.83:0/2297725196 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cab31f9870 0x55cab31f9c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.397+0000 7f5c52815740 1 -- 172.21.15.83:0/2297725196 >> 172.21.15.83:0/2297725196 conn(0x55cab31fb8f0 msgr2=0x55cab31fd1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:24.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.397+0000 7f5c52815740 1 -- 172.21.15.83:0/2297725196 shutdown_connections 2024-03-06T07:26:24.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.397+0000 7f5c52815740 1 -- 172.21.15.83:0/2297725196 wait complete. 2024-03-06T07:26:24.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.397+0000 7f5c52815740 1 -- start start 2024-03-06T07:26:24.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.397+0000 7f5c52815740 1 Processor -- start 2024-03-06T07:26:24.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.399+0000 7f5c52815740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cab3229600 0x55cab323f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.399+0000 7f5c52815740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55cab3224360 0x55cab32237b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.399+0000 7f5c52815740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cab3223cf0 0x55cab3242fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.399+0000 7f5c52815740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cab31fdd50 con 0x55cab3223cf0 2024-03-06T07:26:24.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.399+0000 7f5c52815740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cab311fea0 con 0x55cab3229600 2024-03-06T07:26:24.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.399+0000 7f5c52815740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cab31fe030 con 0x55cab3224360 2024-03-06T07:26:24.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.399+0000 7f5c52815740 10 client.0 _ll_register_callbacks cb 0x55cab311f870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:26:24.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.399+0000 7f5c4e0f9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cab3223cf0 0x55cab3242fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.399+0000 7f5c4e0f9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cab3223cf0 0x55cab3242fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50576/0 (socket says 172.21.15.83:50576) 2024-03-06T07:26:24.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.399+0000 7f5c4e0f9640 1 -- 172.21.15.83:0/3136939663 learned_addr learned my addr 172.21.15.83:0/3136939663 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:24.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.399+0000 7f5c4e0f9640 1 -- 172.21.15.83:0/3136939663 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55cab3224360 msgr2=0x55cab32237b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:24.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.399+0000 7f5c4e0f9640 1 --2- 172.21.15.83:0/3136939663 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55cab3224360 0x55cab32237b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.399+0000 7f5c4e0f9640 1 -- 172.21.15.83:0/3136939663 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cab3229600 msgr2=0x55cab323f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:24.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.399+0000 7f5c4e0f9640 1 --2- 172.21.15.83:0/3136939663 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cab3229600 0x55cab323f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.399+0000 7f5c4e0f9640 1 -- 172.21.15.83:0/3136939663 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cab3120270 con 0x55cab3223cf0 2024-03-06T07:26:24.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.399+0000 7f5c4e0f9640 1 --2- 172.21.15.83:0/3136939663 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cab3223cf0 0x55cab3242fc0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x55cab322dcf0 tx=0x7f5c40004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc54f4c3a9d27f27 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:24.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.399+0000 7f5c28ff9640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-06T07:26:24.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.399+0000 7f5c28ff9640 1 -- 172.21.15.83:0/3136939663 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c400045e0 con 0x55cab3223cf0 2024-03-06T07:26:24.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.399+0000 7f5c28ff9640 1 -- 172.21.15.83:0/3136939663 <== mon.0 v2:172.21.15.83:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5c40004780 con 0x55cab3223cf0 2024-03-06T07:26:24.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.399+0000 7f5c52815740 1 -- 172.21.15.83:0/3136939663 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5c4001d070 con 0x55cab3223cf0 2024-03-06T07:26:24.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.399+0000 7f5c28ff9640 1 -- 172.21.15.83:0/3136939663 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5c40016430 con 0x55cab3223cf0 2024-03-06T07:26:24.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.399+0000 7f5c52815740 10 client.7494 fetch_fsmap learned FSMap version 45 2024-03-06T07:26:24.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.400+0000 7f5c52815740 1 -- 172.21.15.83:0/3136939663 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=2+}) v3 -- 0x55cab2e782d0 con 0x55cab3223cf0 2024-03-06T07:26:24.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.400+0000 7f5c28ff9640 1 -- 172.21.15.83:0/3136939663 <== mon.0 v2:172.21.15.83:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c4001d070 con 0x55cab3223cf0 2024-03-06T07:26:24.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.400+0000 7f5c28ff9640 1 -- 172.21.15.83:0/3136939663 <== mon.0 v2:172.21.15.83:3300/0 5 ==== fsmap(e 45) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f5c400165d0 con 0x55cab3223cf0 2024-03-06T07:26:24.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.400+0000 7f5c52815740 10 client.7494 fetch_fsmap finished waiting for FSMap version 45 2024-03-06T07:26:24.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.400+0000 7f5c52815740 10 client.7494 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:26:24.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.400+0000 7f5c52815740 20 client.7494 populate_metadata read hostname 'smithi083' 2024-03-06T07:26:24.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.400+0000 7f5c52815740 1 --2- 172.21.15.83:0/3136939663 >> [v2:172.21.15.83:6835/2924411033,v1:172.21.15.83:6837/2924411033] conn(0x55cab3241150 0x55cab3241530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.400+0000 7f5c52815740 4 client.7494 mds_command: new command op to 7338 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:26:24.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.400+0000 7f5c52815740 1 -- 172.21.15.83:0/3136939663 --> [v2:172.21.15.83:6835/2924411033,v1:172.21.15.83:6837/2924411033] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cab3170bf0 con 0x55cab3241150 2024-03-06T07:26:24.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.400+0000 7f5c4d8f8640 1 --2- 172.21.15.83:0/3136939663 >> [v2:172.21.15.83:6835/2924411033,v1:172.21.15.83:6837/2924411033] conn(0x55cab3241150 0x55cab3241530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.400+0000 7f5c28ff9640 10 client.7494 ms_handle_connect on v2:172.21.15.83:6835/2924411033 2024-03-06T07:26:24.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.400+0000 7f5c4d8f8640 1 --2- 172.21.15.83:0/3136939663 >> [v2:172.21.15.83:6835/2924411033,v1:172.21.15.83:6837/2924411033] conn(0x55cab3241150 0x55cab3241530 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57c58a1e74c8e8f2 server_cookie=dc8e61df1d411bd5 in_seq=0 out_seq=0 2024-03-06T07:26:24.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.401+0000 7f5c28ff9640 1 -- 172.21.15.83:0/3136939663 <== mds.0 v2:172.21.15.83:6835/2924411033 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55cab3170bf0 con 0x55cab3241150 2024-03-06T07:26:24.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.401+0000 7f5c28ff9640 10 client.7494 handle_command_reply: tid=0 2024-03-06T07:26:24.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.401+0000 7f5c52815740 1 client.7494 shutdown 2024-03-06T07:26:24.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.401+0000 7f5c52815740 1 -- 172.21.15.83:0/3136939663 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cab3223cf0 msgr2=0x55cab3242fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.401+0000 7f5c52815740 1 --2- 172.21.15.83:0/3136939663 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cab3223cf0 0x55cab3242fc0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x55cab322dcf0 tx=0x7f5c40004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:24.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.401+0000 7f5c52815740 1 -- 172.21.15.83:0/3136939663 shutdown_connections 2024-03-06T07:26:24.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.401+0000 7f5c52815740 1 --2- 172.21.15.83:0/3136939663 >> [v2:172.21.15.83:6835/2924411033,v1:172.21.15.83:6837/2924411033] conn(0x55cab3241150 0x55cab3241530 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.401+0000 7f5c52815740 1 --2- 172.21.15.83:0/3136939663 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cab3223cf0 0x55cab3242fc0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.401+0000 7f5c52815740 1 --2- 172.21.15.83:0/3136939663 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55cab3224360 0x55cab32237b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.401+0000 7f5c28ff9640 0 client.7494 ms_handle_reset on v2:172.21.15.83:6835/2924411033 2024-03-06T07:26:24.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.401+0000 7f5c52815740 1 --2- 172.21.15.83:0/3136939663 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cab3229600 0x55cab323f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.401+0000 7f5c52815740 1 -- 172.21.15.83:0/3136939663 >> 172.21.15.83:0/3136939663 conn(0x55cab31f9870 msgr2=0x55cab31f8510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:24.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.401+0000 7f5c52815740 1 -- 172.21.15.83:0/3136939663 shutdown_connections 2024-03-06T07:26:24.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.401+0000 7f5c52815740 1 -- 172.21.15.83:0/3136939663 wait complete. 2024-03-06T07:26:24.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.401+0000 7f5c52815740 20 client.7494 trim_cache size 0 max 16384 2024-03-06T07:26:24.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.433+0000 7f5c52815740 1 Processor -- start 2024-03-06T07:26:24.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.433+0000 7f5c52815740 1 -- start start 2024-03-06T07:26:24.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.434+0000 7f5c52815740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55cab3223cf0 0x55cab32231c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.434+0000 7f5c52815740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cab3224360 0x55cab3223700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.434+0000 7f5c52815740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cab3229600 0x55cab322acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.434+0000 7f5c52815740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c48200410 con 0x55cab3229600 2024-03-06T07:26:24.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.434+0000 7f5c52815740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c481ff610 con 0x55cab3224360 2024-03-06T07:26:24.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.434+0000 7f5c52815740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c481e6110 con 0x55cab3223cf0 2024-03-06T07:26:24.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.434+0000 7f5c4e0f9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cab3229600 0x55cab322acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.434+0000 7f5c4e0f9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cab3229600 0x55cab322acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50580/0 (socket says 172.21.15.83:50580) 2024-03-06T07:26:24.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.434+0000 7f5c4e0f9640 1 -- 172.21.15.83:0/256917721 learned_addr learned my addr 172.21.15.83:0/256917721 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:24.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.435+0000 7f5c4d8f8640 1 --2- 172.21.15.83:0/256917721 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55cab3223cf0 0x55cab32231c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.435+0000 7f5c4d0f7640 1 --2- 172.21.15.83:0/256917721 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cab3224360 0x55cab3223700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.435+0000 7f5c4e0f9640 1 -- 172.21.15.83:0/256917721 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55cab3223cf0 msgr2=0x55cab32231c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:24.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.435+0000 7f5c4e0f9640 1 --2- 172.21.15.83:0/256917721 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55cab3223cf0 0x55cab32231c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.435+0000 7f5c4e0f9640 1 -- 172.21.15.83:0/256917721 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cab3224360 msgr2=0x55cab3223700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:24.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.435+0000 7f5c4e0f9640 1 --2- 172.21.15.83:0/256917721 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cab3224360 0x55cab3223700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.435+0000 7f5c4e0f9640 1 -- 172.21.15.83:0/256917721 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cab2f426b0 con 0x55cab3229600 2024-03-06T07:26:24.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.435+0000 7f5c4e0f9640 1 --2- 172.21.15.83:0/256917721 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cab3229600 0x55cab322acb0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f5c40007ec0 tx=0x7f5c40024070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c464614422e3516 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:24.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.435+0000 7f5c2affd640 1 -- 172.21.15.83:0/256917721 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c40016cd0 con 0x55cab3229600 2024-03-06T07:26:24.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.435+0000 7f5c2affd640 1 -- 172.21.15.83:0/256917721 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c4001d070 con 0x55cab3229600 2024-03-06T07:26:24.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.435+0000 7f5c2affd640 1 -- 172.21.15.83:0/256917721 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c40004480 con 0x55cab3229600 2024-03-06T07:26:24.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.436+0000 7f5c52815740 1 -- 172.21.15.83:0/256917721 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cab3229600 msgr2=0x55cab322acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.436+0000 7f5c52815740 1 --2- 172.21.15.83:0/256917721 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cab3229600 0x55cab322acb0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f5c40007ec0 tx=0x7f5c40024070 comp rx=0 tx=0).stop 2024-03-06T07:26:24.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.436+0000 7f5c52815740 1 -- 172.21.15.83:0/256917721 shutdown_connections 2024-03-06T07:26:24.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.436+0000 7f5c52815740 1 --2- 172.21.15.83:0/256917721 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cab3229600 0x55cab322acb0 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.436+0000 7f5c52815740 1 --2- 172.21.15.83:0/256917721 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cab3224360 0x55cab3223700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.436+0000 7f5c52815740 1 --2- 172.21.15.83:0/256917721 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55cab3223cf0 0x55cab32231c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.436+0000 7f5c52815740 1 -- 172.21.15.83:0/256917721 >> 172.21.15.83:0/256917721 conn(0x55cab31f9870 msgr2=0x55cab323ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:24.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.436+0000 7f5c52815740 1 -- 172.21.15.83:0/256917721 shutdown_connections 2024-03-06T07:26:24.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.436+0000 7f5c52815740 1 -- 172.21.15.83:0/256917721 wait complete. 2024-03-06T07:26:24.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.436+0000 7f5c52815740 1 -- start start 2024-03-06T07:26:24.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.436+0000 7f5c52815740 1 Processor -- start 2024-03-06T07:26:24.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.437+0000 7f5c52815740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55cab3223cf0 0x55cab322aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.437+0000 7f5c52815740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cab3224360 0x55cab322afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.437+0000 7f5c52815740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cab3229600 0x55cab32b4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.437+0000 7f5c52815740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cab323f570 con 0x55cab3229600 2024-03-06T07:26:24.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.437+0000 7f5c52815740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c481e6710 con 0x55cab3224360 2024-03-06T07:26:24.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.437+0000 7f5c52815740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cab32427c0 con 0x55cab3223cf0 2024-03-06T07:26:24.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.437+0000 7f5c52815740 10 client.0 _ll_register_callbacks cb 0x7f5c40007330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:26:24.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.437+0000 7f5c4e0f9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cab3229600 0x55cab32b4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.437+0000 7f5c4e0f9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cab3229600 0x55cab32b4760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50588/0 (socket says 172.21.15.83:50588) 2024-03-06T07:26:24.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.437+0000 7f5c4e0f9640 1 -- 172.21.15.83:0/3860671871 learned_addr learned my addr 172.21.15.83:0/3860671871 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:24.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.437+0000 7f5c4d8f8640 1 --2- 172.21.15.83:0/3860671871 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55cab3223cf0 0x55cab322aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.437+0000 7f5c4d0f7640 1 --2- 172.21.15.83:0/3860671871 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cab3224360 0x55cab322afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.438+0000 7f5c4e0f9640 1 -- 172.21.15.83:0/3860671871 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55cab3223cf0 msgr2=0x55cab322aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:24.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.438+0000 7f5c4e0f9640 1 --2- 172.21.15.83:0/3860671871 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55cab3223cf0 0x55cab322aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.438+0000 7f5c4e0f9640 1 -- 172.21.15.83:0/3860671871 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cab3224360 msgr2=0x55cab322afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:24.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.438+0000 7f5c4e0f9640 1 --2- 172.21.15.83:0/3860671871 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cab3224360 0x55cab322afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.438+0000 7f5c4e0f9640 1 -- 172.21.15.83:0/3860671871 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cab32443f0 con 0x55cab3229600 2024-03-06T07:26:24.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.438+0000 7f5c4e0f9640 1 --2- 172.21.15.83:0/3860671871 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cab3229600 0x55cab32b4760 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f5c40039040 tx=0x7f5c4000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60d21641fe1fe0a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:24.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.438+0000 7f5c28ff9640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-06T07:26:24.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.438+0000 7f5c28ff9640 1 -- 172.21.15.83:0/3860671871 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4003b040 con 0x55cab3229600 2024-03-06T07:26:24.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.438+0000 7f5c28ff9640 1 -- 172.21.15.83:0/3860671871 <== mon.0 v2:172.21.15.83:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5c40004440 con 0x55cab3229600 2024-03-06T07:26:24.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.438+0000 7f5c52815740 1 -- 172.21.15.83:0/3860671871 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5c4001d070 con 0x55cab3229600 2024-03-06T07:26:24.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.438+0000 7f5c28ff9640 1 -- 172.21.15.83:0/3860671871 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5c4004c050 con 0x55cab3229600 2024-03-06T07:26:24.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.438+0000 7f5c52815740 10 client.7500 fetch_fsmap learned FSMap version 45 2024-03-06T07:26:24.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.438+0000 7f5c52815740 1 -- 172.21.15.83:0/3860671871 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=2+}) v3 -- 0x55cab32b5840 con 0x55cab3229600 2024-03-06T07:26:24.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.438+0000 7f5c28ff9640 1 -- 172.21.15.83:0/3860671871 <== mon.0 v2:172.21.15.83:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c4001d070 con 0x55cab3229600 2024-03-06T07:26:24.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.438+0000 7f5c28ff9640 1 -- 172.21.15.83:0/3860671871 <== mon.0 v2:172.21.15.83:3300/0 5 ==== fsmap(e 45) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f5c4004f020 con 0x55cab3229600 2024-03-06T07:26:24.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.439+0000 7f5c52815740 10 client.7500 fetch_fsmap finished waiting for FSMap version 45 2024-03-06T07:26:24.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.439+0000 7f5c52815740 10 client.7500 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:26:24.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.439+0000 7f5c52815740 20 client.7500 populate_metadata read hostname 'smithi083' 2024-03-06T07:26:24.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.439+0000 7f5c52815740 1 --2- 172.21.15.83:0/3860671871 >> [v2:172.21.15.83:6835/2924411033,v1:172.21.15.83:6837/2924411033] conn(0x55cab32b53c0 0x55cab322c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.439+0000 7f5c52815740 4 client.7500 mds_command: new command op to 7338 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:26:24.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.439+0000 7f5c52815740 1 -- 172.21.15.83:0/3860671871 --> [v2:172.21.15.83:6835/2924411033,v1:172.21.15.83:6837/2924411033] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55cab31faed0 con 0x55cab32b53c0 2024-03-06T07:26:24.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.439+0000 7f5c4d8f8640 1 --2- 172.21.15.83:0/3860671871 >> [v2:172.21.15.83:6835/2924411033,v1:172.21.15.83:6837/2924411033] conn(0x55cab32b53c0 0x55cab322c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.439+0000 7f5c28ff9640 10 client.7500 ms_handle_connect on v2:172.21.15.83:6835/2924411033 2024-03-06T07:26:24.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.439+0000 7f5c4d8f8640 1 --2- 172.21.15.83:0/3860671871 >> [v2:172.21.15.83:6835/2924411033,v1:172.21.15.83:6837/2924411033] conn(0x55cab32b53c0 0x55cab322c380 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27ee7fe216624996 server_cookie=af75477d3b7e624c in_seq=0 out_seq=0 2024-03-06T07:26:24.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.439+0000 7f5c28ff9640 1 -- 172.21.15.83:0/3860671871 <== mds.0 v2:172.21.15.83:6835/2924411033 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55cab31faed0 con 0x55cab32b53c0 2024-03-06T07:26:24.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.439+0000 7f5c28ff9640 10 client.7500 handle_command_reply: tid=0 2024-03-06T07:26:24.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.439+0000 7f5c52815740 1 client.7500 shutdown 2024-03-06T07:26:24.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.439+0000 7f5c52815740 1 -- 172.21.15.83:0/3860671871 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cab3229600 msgr2=0x55cab32b4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.439+0000 7f5c52815740 1 --2- 172.21.15.83:0/3860671871 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cab3229600 0x55cab32b4760 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f5c40039040 tx=0x7f5c4000b040 comp rx=0 tx=0).stop 2024-03-06T07:26:24.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.439+0000 7f5c52815740 1 -- 172.21.15.83:0/3860671871 shutdown_connections 2024-03-06T07:26:24.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.439+0000 7f5c52815740 1 --2- 172.21.15.83:0/3860671871 >> [v2:172.21.15.83:6835/2924411033,v1:172.21.15.83:6837/2924411033] conn(0x55cab32b53c0 0x55cab322c380 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.439+0000 7f5c52815740 1 --2- 172.21.15.83:0/3860671871 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cab3229600 0x55cab32b4760 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.439+0000 7f5c52815740 1 --2- 172.21.15.83:0/3860671871 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cab3224360 0x55cab322afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.439+0000 7f5c52815740 1 --2- 172.21.15.83:0/3860671871 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55cab3223cf0 0x55cab322aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.439+0000 7f5c52815740 1 -- 172.21.15.83:0/3860671871 >> 172.21.15.83:0/3860671871 conn(0x55cab31f9870 msgr2=0x55cab3226d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:24.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.439+0000 7f5c28ff9640 0 client.7500 ms_handle_reset on v2:172.21.15.83:6835/2924411033 2024-03-06T07:26:24.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.439+0000 7f5c52815740 1 -- 172.21.15.83:0/3860671871 shutdown_connections 2024-03-06T07:26:24.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.439+0000 7f5c52815740 1 -- 172.21.15.83:0/3860671871 wait complete. 2024-03-06T07:26:24.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.440+0000 7f5c52815740 20 client.7500 trim_cache size 0 max 16384 2024-03-06T07:26:24.447 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:26:24.447 INFO:teuthology.orchestra.run.smithi083.stdout: "cluster_fsid": "039877dd-d2ce-45d2-8e77-ac5eb7e1d55a", 2024-03-06T07:26:24.447 INFO:teuthology.orchestra.run.smithi083.stdout: "whoami": 0, 2024-03-06T07:26:24.447 INFO:teuthology.orchestra.run.smithi083.stdout: "id": 7338, 2024-03-06T07:26:24.447 INFO:teuthology.orchestra.run.smithi083.stdout: "want_state": "up:active", 2024-03-06T07:26:24.447 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "up:active", 2024-03-06T07:26:24.447 INFO:teuthology.orchestra.run.smithi083.stdout: "fs_name": "cephfs", 2024-03-06T07:26:24.447 INFO:teuthology.orchestra.run.smithi083.stdout: "rank_uptime": 4.7281584929999996, 2024-03-06T07:26:24.447 INFO:teuthology.orchestra.run.smithi083.stdout: "mdsmap_epoch": 45, 2024-03-06T07:26:24.447 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_epoch": 94, 2024-03-06T07:26:24.447 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_epoch_barrier": 94, 2024-03-06T07:26:24.447 INFO:teuthology.orchestra.run.smithi083.stdout: "uptime": 9.8023932620000007 2024-03-06T07:26:24.448 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:26:24.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.440+0000 7f5c4fb5c640 1 -- 172.21.15.83:0/44736411 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5c2c06b4d0 msgr2=0x7f5c2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.440+0000 7f5c4fb5c640 1 --2- 172.21.15.83:0/44736411 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5c2c06b4d0 0x7f5c2c06d990 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f5c44034000 tx=0x7f5c44007000 comp rx=0 tx=0).stop 2024-03-06T07:26:24.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.440+0000 7f5c4fb5c640 1 -- 172.21.15.83:0/44736411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c480b6b10 msgr2=0x7f5c481e9d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.440+0000 7f5c4fb5c640 1 --2- 172.21.15.83:0/44736411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c480b6b10 0x7f5c481e9d50 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f5c38011960 tx=0x7f5c38011e20 comp rx=0 tx=0).stop 2024-03-06T07:26:24.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.440+0000 7f5c4fb5c640 1 -- 172.21.15.83:0/44736411 shutdown_connections 2024-03-06T07:26:24.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.440+0000 7f5c4fb5c640 1 --2- 172.21.15.83:0/44736411 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5c2c06b4d0 0x7f5c2c06d990 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.440+0000 7f5c4fb5c640 1 --2- 172.21.15.83:0/44736411 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c480b6f90 0x7f5c481ea290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.440+0000 7f5c4fb5c640 1 --2- 172.21.15.83:0/44736411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c480b6b10 0x7f5c481e9d50 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.440+0000 7f5c4fb5c640 1 --2- 172.21.15.83:0/44736411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5c340039b0 0x7f5c481e7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.440+0000 7f5c4fb5c640 1 -- 172.21.15.83:0/44736411 >> 172.21.15.83:0/44736411 conn(0x7f5c4809a9f0 msgr2=0x7f5c480d10b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:24.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.440+0000 7f5c4fb5c640 1 -- 172.21.15.83:0/44736411 shutdown_connections 2024-03-06T07:26:24.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.440+0000 7f5c4fb5c640 1 -- 172.21.15.83:0/44736411 wait complete. 2024-03-06T07:26:24.452 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:26:24.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.551+0000 7f4193fb6640 1 Processor -- start 2024-03-06T07:26:24.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.551+0000 7f4193fb6640 1 -- start start 2024-03-06T07:26:24.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.552+0000 7f4193fb6640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f418c15bb20 0x7f418c0d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.552+0000 7f4193fb6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f418c161980 con 0x7f418c0cfff0 2024-03-06T07:26:24.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.552+0000 7f4193fb6640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f418c161c60 con 0x7f418c0cd8c0 2024-03-06T07:26:24.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.552+0000 7f4193fb6640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f418c161f40 con 0x7f418c15bb20 2024-03-06T07:26:24.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.552+0000 7f4191d52640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f418c15bb20 0x7f418c0d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.552+0000 7f4191551640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f418c0cfff0 0x7f418c0ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:35442/0 (socket says 172.21.15.83:35442) 2024-03-06T07:26:24.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.552+0000 7f4191551640 1 -- 172.21.15.83:0/3098521919 learned_addr learned my addr 172.21.15.83:0/3098521919 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:24.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.553+0000 7f4191d52640 1 -- 172.21.15.83:0/3098521919 >> v1:172.21.15.183:6789/0 conn(0x7f418c0cd8c0 legacy=0x7f418c0cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:24.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.553+0000 7f4191d52640 1 -- 172.21.15.83:0/3098521919 >> v1:172.21.15.83:6789/0 conn(0x7f418c0cfff0 legacy=0x7f418c0ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.553+0000 7f4191d52640 1 -- 172.21.15.83:0/3098521919 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f418c162220 con 0x7f418c15bb20 2024-03-06T07:26:24.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.553+0000 7f4191d52640 1 --2- 172.21.15.83:0/3098521919 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f418c15bb20 0x7f418c0d1df0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f417c0097a0 tx=0x7f417c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b68bb1e84091d45 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:24.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.553+0000 7f4190d50640 1 -- 172.21.15.83:0/3098521919 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c035020 con 0x7f418c15bb20 2024-03-06T07:26:24.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.553+0000 7f4190d50640 1 -- 172.21.15.83:0/3098521919 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f417c031830 con 0x7f418c15bb20 2024-03-06T07:26:24.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.553+0000 7f4190d50640 1 -- 172.21.15.83:0/3098521919 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c031c00 con 0x7f418c15bb20 2024-03-06T07:26:24.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.554+0000 7f4193fb6640 1 -- 172.21.15.83:0/3098521919 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f418c15bb20 msgr2=0x7f418c0d1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.554+0000 7f4193fb6640 1 --2- 172.21.15.83:0/3098521919 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f418c15bb20 0x7f418c0d1df0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f417c0097a0 tx=0x7f417c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.554+0000 7f4193fb6640 1 -- 172.21.15.83:0/3098521919 shutdown_connections 2024-03-06T07:26:24.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.554+0000 7f4193fb6640 1 --2- 172.21.15.83:0/3098521919 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f418c15bb20 0x7f418c0d1df0 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.554+0000 7f4193fb6640 1 -- 172.21.15.83:0/3098521919 >> 172.21.15.83:0/3098521919 conn(0x7f418c09a9f0 msgr2=0x7f418c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:24.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.554+0000 7f4193fb6640 1 -- 172.21.15.83:0/3098521919 shutdown_connections 2024-03-06T07:26:24.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.554+0000 7f4193fb6640 1 -- 172.21.15.83:0/3098521919 wait complete. 2024-03-06T07:26:24.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.555+0000 7f4193fb6640 1 Processor -- start 2024-03-06T07:26:24.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.555+0000 7f4193fb6640 1 -- start start 2024-03-06T07:26:24.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.556+0000 7f4193fb6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f418c0cd8c0 0x7f418c0c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.556+0000 7f4193fb6640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f418c0cfff0 0x7f418c149f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.556+0000 7f4193fb6640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f418c15bb20 0x7f418c14a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.556+0000 7f4193fb6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f418c0d2330 con 0x7f418c0cd8c0 2024-03-06T07:26:24.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.556+0000 7f4193fb6640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f418c0cf2d0 con 0x7f418c0cfff0 2024-03-06T07:26:24.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.556+0000 7f4193fb6640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f418c0d24b0 con 0x7f418c15bb20 2024-03-06T07:26:24.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.556+0000 7f4191d52640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f418c0cd8c0 0x7f418c0c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.556+0000 7f4191d52640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f418c0cd8c0 0x7f418c0c8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50600/0 (socket says 172.21.15.83:50600) 2024-03-06T07:26:24.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.556+0000 7f4191d52640 1 -- 172.21.15.83:0/833098767 learned_addr learned my addr 172.21.15.83:0/833098767 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:24.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.557+0000 7f4191551640 1 --2- 172.21.15.83:0/833098767 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f418c0cfff0 0x7f418c149f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.557+0000 7f4191d52640 1 -- 172.21.15.83:0/833098767 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f418c15bb20 msgr2=0x7f418c14a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.557+0000 7f4191d52640 1 --2- 172.21.15.83:0/833098767 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f418c15bb20 0x7f418c14a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.557+0000 7f4191d52640 1 -- 172.21.15.83:0/833098767 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f418c0cfff0 msgr2=0x7f418c149f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.557+0000 7f4191d52640 1 --2- 172.21.15.83:0/833098767 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f418c0cfff0 0x7f418c149f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.557+0000 7f4191d52640 1 -- 172.21.15.83:0/833098767 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f418c0c7260 con 0x7f418c0cd8c0 2024-03-06T07:26:24.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.557+0000 7f4191d52640 1 --2- 172.21.15.83:0/833098767 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f418c0cd8c0 0x7f418c0c8b90 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f417c0098c0 tx=0x7f417c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:24.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.557+0000 7f41827fc640 1 -- 172.21.15.83:0/833098767 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c041070 con 0x7f418c0cd8c0 2024-03-06T07:26:24.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.557+0000 7f41827fc640 1 -- 172.21.15.83:0/833098767 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f417c03aa30 con 0x7f418c0cd8c0 2024-03-06T07:26:24.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.557+0000 7f41827fc640 1 -- 172.21.15.83:0/833098767 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c035040 con 0x7f418c0cd8c0 2024-03-06T07:26:24.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.557+0000 7f4193fb6640 1 -- 172.21.15.83:0/833098767 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f418c0c7540 con 0x7f418c0cd8c0 2024-03-06T07:26:24.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.557+0000 7f4193fb6640 1 -- 172.21.15.83:0/833098767 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f418c0c7b00 con 0x7f418c0cd8c0 2024-03-06T07:26:24.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.558+0000 7f4193fb6640 1 -- 172.21.15.83:0/833098767 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f418c04f330 con 0x7f418c0cd8c0 2024-03-06T07:26:24.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.558+0000 7f41827fc640 1 -- 172.21.15.83:0/833098767 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f417c04c060 con 0x7f418c0cd8c0 2024-03-06T07:26:24.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.559+0000 7f41827fc640 1 --2- 172.21.15.83:0/833098767 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f417006b500 0x7f417006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.559+0000 7f4191551640 1 --2- 172.21.15.83:0/833098767 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f417006b500 0x7f417006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.559+0000 7f41827fc640 1 -- 172.21.15.83:0/833098767 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f417c0abc20 con 0x7f418c0cd8c0 2024-03-06T07:26:24.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.559+0000 7f4191551640 1 --2- 172.21.15.83:0/833098767 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f417006b500 0x7f417006d9c0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f418c1e7a90 tx=0x7f4188009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:24.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.563+0000 7f41827fc640 1 -- 172.21.15.83:0/833098767 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f417c07ab30 con 0x7f418c0cd8c0 2024-03-06T07:26:24.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.793+0000 7f4193fb6640 1 -- 172.21.15.83:0/833098767 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f418c0b2cf0 con 0x7f418c0cd8c0 2024-03-06T07:26:24.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.793+0000 7f41827fc640 1 -- 172.21.15.83:0/833098767 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v45) v1 ==== 53+0+73 (secure 0 0 0) 0x7f417c07a430 con 0x7f418c0cd8c0 2024-03-06T07:26:24.795 INFO:teuthology.orchestra.run.smithi083.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:26:24.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.795+0000 7f4193fb6640 1 -- 172.21.15.83:0/833098767 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f417006b500 msgr2=0x7f417006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.795+0000 7f4193fb6640 1 --2- 172.21.15.83:0/833098767 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f417006b500 0x7f417006d9c0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f418c1e7a90 tx=0x7f4188009040 comp rx=0 tx=0).stop 2024-03-06T07:26:24.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.795+0000 7f4193fb6640 1 -- 172.21.15.83:0/833098767 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f418c0cd8c0 msgr2=0x7f418c0c8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.795+0000 7f4193fb6640 1 --2- 172.21.15.83:0/833098767 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f418c0cd8c0 0x7f418c0c8b90 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f417c0098c0 tx=0x7f417c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.795+0000 7f4193fb6640 1 -- 172.21.15.83:0/833098767 shutdown_connections 2024-03-06T07:26:24.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.795+0000 7f4193fb6640 1 --2- 172.21.15.83:0/833098767 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f417006b500 0x7f417006d9c0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.795+0000 7f4193fb6640 1 --2- 172.21.15.83:0/833098767 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f418c15bb20 0x7f418c14a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.795+0000 7f4193fb6640 1 --2- 172.21.15.83:0/833098767 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f418c0cfff0 0x7f418c149f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.795+0000 7f4193fb6640 1 --2- 172.21.15.83:0/833098767 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f418c0cd8c0 0x7f418c0c8b90 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.795+0000 7f4193fb6640 1 -- 172.21.15.83:0/833098767 >> 172.21.15.83:0/833098767 conn(0x7f418c09a9f0 msgr2=0x7f418c0b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:24.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.795+0000 7f4193fb6640 1 -- 172.21.15.83:0/833098767 shutdown_connections 2024-03-06T07:26:24.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:24.795+0000 7f4193fb6640 1 -- 172.21.15.83:0/833098767 wait complete. 2024-03-06T07:26:24.808 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:26:24.808 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:26:24.808 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi083.front.sepia.ceph.com 2024-03-06T07:26:24.808 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:26:24.808 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:24.808 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:26:24.808 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:26:24.808 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:26:24.808 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:26:24.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:24.809 DEBUG:teuthology.orchestra.run.smithi083:> ip addr 2024-03-06T07:26:24.824 INFO:teuthology.orchestra.run.smithi083.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:26:24.824 INFO:teuthology.orchestra.run.smithi083.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:26:24.824 INFO:teuthology.orchestra.run.smithi083.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:26:24.824 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:24.824 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 ::1/128 scope host 2024-03-06T07:26:24.824 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:24.824 INFO:teuthology.orchestra.run.smithi083.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:26:24.824 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:d8:da:1e brd ff:ff:ff:ff:ff:ff 2024-03-06T07:26:24.824 INFO:teuthology.orchestra.run.smithi083.stdout: altname enp7s0f0 2024-03-06T07:26:24.824 INFO:teuthology.orchestra.run.smithi083.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:26:24.824 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:d8:da:1f brd ff:ff:ff:ff:ff:ff 2024-03-06T07:26:24.825 INFO:teuthology.orchestra.run.smithi083.stdout: altname enp7s0f1 2024-03-06T07:26:24.825 INFO:teuthology.orchestra.run.smithi083.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:26:24.825 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:8f:ce:90 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:26:24.825 INFO:teuthology.orchestra.run.smithi083.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:26:24.825 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:8f:ce:91 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:26:24.825 INFO:teuthology.orchestra.run.smithi083.stdout: inet 172.21.15.83/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:26:24.825 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft 42471sec preferred_lft 42471sec 2024-03-06T07:26:24.825 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::ec4:7aff:fe8f:ce91/64 scope link 2024-03-06T07:26:24.825 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:24.825 INFO:teuthology.orchestra.run.smithi083.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:26:24.825 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 1e:29:46:c9:e7:c8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:26:24.825 INFO:teuthology.orchestra.run.smithi083.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:26:24.825 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:24.825 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::60c5:d7ff:fe70:2b1c/64 scope link 2024-03-06T07:26:24.826 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:24.826 INFO:teuthology.orchestra.run.smithi083.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:26:24.826 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 1e:29:46:c9:e7:c8 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:26:24.826 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::1c29:46ff:fec9:e7c8/64 scope link 2024-03-06T07:26:24.826 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:24.826 INFO:teuthology.orchestra.run.smithi083.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:26:24.826 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 3e:00:2f:23:d5:45 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:26:24.826 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::3c00:2fff:fe23:d545/64 scope link 2024-03-06T07:26:24.826 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:24.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:24.827 DEBUG:teuthology.orchestra.run.smithi083:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:26:24.889 INFO:teuthology.orchestra.run.smithi083.stdout:1 2024-03-06T07:26:24.891 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:24.891 DEBUG:teuthology.orchestra.run.smithi083:> route 2024-03-06T07:26:24.947 INFO:teuthology.orchestra.run.smithi083.stdout:Kernel IP routing table 2024-03-06T07:26:24.947 INFO:teuthology.orchestra.run.smithi083.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:26:24.947 INFO:teuthology.orchestra.run.smithi083.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:26:24.947 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:26:24.947 INFO:teuthology.orchestra.run.smithi083.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:26:24.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:24.948 DEBUG:teuthology.orchestra.run.smithi083:> (cd / && exec bash -c ' 2024-03-06T07:26:24.948 DEBUG:teuthology.orchestra.run.smithi083:> set -e 2024-03-06T07:26:24.948 DEBUG:teuthology.orchestra.run.smithi083:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:26:24.948 DEBUG:teuthology.orchestra.run.smithi083:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:26:24.948 DEBUG:teuthology.orchestra.run.smithi083:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:26:24.948 DEBUG:teuthology.orchestra.run.smithi083:> ') 2024-03-06T07:26:24.974 INFO:tasks.ceph.mds.a.smithi083.stderr:2024-03-06T07:26:24.973+0000 7f67fe08a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:26:25.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:25.029 DEBUG:teuthology.orchestra.run.smithi083:> ip netns list 2024-03-06T07:26:25.044 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:26:25.044 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:26:25.044 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:26:25.044 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:25.044 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip link set brx.0 up 2024-03-06T07:26:25.111 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:26:25.111 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:26:25.111 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:25.166 INFO:teuthology.orchestra.run.smithi083.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:26:25.166 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:26:25.166 DEBUG:teuthology.orchestra.run.smithi083:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:25.221 DEBUG:teuthology.orchestra.run.smithi083:> sudo modprobe fuse 2024-03-06T07:26:25.288 DEBUG:teuthology.orchestra.run.smithi083:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:26:25.344 INFO:teuthology.orchestra.run.smithi083.stdout:/proc 2024-03-06T07:26:25.344 INFO:teuthology.orchestra.run.smithi083.stdout:/sys 2024-03-06T07:26:25.345 INFO:teuthology.orchestra.run.smithi083.stdout:/dev 2024-03-06T07:26:25.345 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/security 2024-03-06T07:26:25.345 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/shm 2024-03-06T07:26:25.345 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/pts 2024-03-06T07:26:25.345 INFO:teuthology.orchestra.run.smithi083.stdout:/run 2024-03-06T07:26:25.345 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/cgroup 2024-03-06T07:26:25.345 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/pstore 2024-03-06T07:26:25.345 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/bpf 2024-03-06T07:26:25.345 INFO:teuthology.orchestra.run.smithi083.stdout:/ 2024-03-06T07:26:25.345 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/selinux 2024-03-06T07:26:25.345 INFO:teuthology.orchestra.run.smithi083.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:26:25.345 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/mqueue 2024-03-06T07:26:25.345 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/hugepages 2024-03-06T07:26:25.345 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/debug 2024-03-06T07:26:25.345 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/tracing 2024-03-06T07:26:25.346 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:26:25.346 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/fuse/connections 2024-03-06T07:26:25.346 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/config 2024-03-06T07:26:25.346 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:26:25.346 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:26:25.346 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:26:25.346 INFO:teuthology.orchestra.run.smithi083.stdout:/run/user/1000 2024-03-06T07:26:25.346 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph 2024-03-06T07:26:25.346 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:26:25.346 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:26:25.346 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:26:25.346 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:26:25.346 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns 2024-03-06T07:26:25.346 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:26:25.346 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:26:25.347 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:26:25.347 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:26:25.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:25.347 DEBUG:teuthology.orchestra.run.smithi083:> ls /sys/fs/fuse/connections 2024-03-06T07:26:25.400 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:26:25.400 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:26:25.442 DEBUG:teuthology.orchestra.run.smithi083:> sudo modprobe fuse 2024-03-06T07:26:25.469 DEBUG:teuthology.orchestra.run.smithi083:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:26:25.526 INFO:teuthology.orchestra.run.smithi083.stdout:/proc 2024-03-06T07:26:25.526 INFO:teuthology.orchestra.run.smithi083.stdout:/sys 2024-03-06T07:26:25.526 INFO:teuthology.orchestra.run.smithi083.stdout:/dev 2024-03-06T07:26:25.526 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/security 2024-03-06T07:26:25.526 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/shm 2024-03-06T07:26:25.526 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/pts 2024-03-06T07:26:25.526 INFO:teuthology.orchestra.run.smithi083.stdout:/run 2024-03-06T07:26:25.526 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/cgroup 2024-03-06T07:26:25.526 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/pstore 2024-03-06T07:26:25.526 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/bpf 2024-03-06T07:26:25.526 INFO:teuthology.orchestra.run.smithi083.stdout:/ 2024-03-06T07:26:25.526 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/selinux 2024-03-06T07:26:25.527 INFO:teuthology.orchestra.run.smithi083.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:26:25.527 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/mqueue 2024-03-06T07:26:25.527 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/hugepages 2024-03-06T07:26:25.527 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/debug 2024-03-06T07:26:25.527 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/tracing 2024-03-06T07:26:25.527 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:26:25.527 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/fuse/connections 2024-03-06T07:26:25.527 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/config 2024-03-06T07:26:25.527 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:26:25.527 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:26:25.527 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:26:25.527 INFO:teuthology.orchestra.run.smithi083.stdout:/run/user/1000 2024-03-06T07:26:25.527 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph 2024-03-06T07:26:25.527 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:26:25.528 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:26:25.528 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:26:25.528 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:26:25.528 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns 2024-03-06T07:26:25.528 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:26:25.528 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:26:25.528 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:26:25.528 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:26:25.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:25.528 DEBUG:teuthology.orchestra.run.smithi083:> ls /sys/fs/fuse/connections 2024-03-06T07:26:25.535 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:2024-03-06T07:26:25.533+0000 7f1c2ead8480 -1 init, newargv = 0x55992a0b6fd0 newargc=15 2024-03-06T07:26:25.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:ceph-fuse[68926]: starting ceph client 2024-03-06T07:26:25.540 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:ceph-fuse[68926]: starting fuse 2024-03-06T07:26:25.550 INFO:teuthology.orchestra.run.smithi083.stdout:41 2024-03-06T07:26:25.550 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-06T07:26:25.550 DEBUG:teuthology.orchestra.run.smithi083:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:26:25.551 DEBUG:teuthology.orchestra.run.smithi083:> import glob 2024-03-06T07:26:25.551 DEBUG:teuthology.orchestra.run.smithi083:> import re 2024-03-06T07:26:25.551 DEBUG:teuthology.orchestra.run.smithi083:> import os 2024-03-06T07:26:25.551 DEBUG:teuthology.orchestra.run.smithi083:> import subprocess 2024-03-06T07:26:25.551 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:26:25.551 DEBUG:teuthology.orchestra.run.smithi083:> def _find_admin_socket(client_name): 2024-03-06T07:26:25.551 DEBUG:teuthology.orchestra.run.smithi083:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:26:25.551 DEBUG:teuthology.orchestra.run.smithi083:> files = glob.glob(asok_path) 2024-03-06T07:26:25.551 DEBUG:teuthology.orchestra.run.smithi083:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:26:25.551 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:26:25.551 DEBUG:teuthology.orchestra.run.smithi083:> # Given a non-glob path, it better be there 2024-03-06T07:26:25.551 DEBUG:teuthology.orchestra.run.smithi083:> if "*" not in asok_path: 2024-03-06T07:26:25.551 DEBUG:teuthology.orchestra.run.smithi083:> assert(len(files) == 1) 2024-03-06T07:26:25.551 DEBUG:teuthology.orchestra.run.smithi083:> return files[0] 2024-03-06T07:26:25.551 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:26:25.551 DEBUG:teuthology.orchestra.run.smithi083:> for f in files: 2024-03-06T07:26:25.552 DEBUG:teuthology.orchestra.run.smithi083:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:26:25.552 DEBUG:teuthology.orchestra.run.smithi083:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:26:25.552 DEBUG:teuthology.orchestra.run.smithi083:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:26:25.552 DEBUG:teuthology.orchestra.run.smithi083:> contents = proc_f.read() 2024-03-06T07:26:25.552 DEBUG:teuthology.orchestra.run.smithi083:> if mountpoint in contents: 2024-03-06T07:26:25.552 DEBUG:teuthology.orchestra.run.smithi083:> return f 2024-03-06T07:26:25.552 DEBUG:teuthology.orchestra.run.smithi083:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:26:25.552 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:26:25.552 DEBUG:teuthology.orchestra.run.smithi083:> print(_find_admin_socket("client.0")) 2024-03-06T07:26:25.552 DEBUG:teuthology.orchestra.run.smithi083:> ' 2024-03-06T07:26:25.670 INFO:teuthology.orchestra.run.smithi083.stdout:/var/run/ceph/ceph-client.0.68926.asok 2024-03-06T07:26:25.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:26:25.680 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.68926.asok 2024-03-06T07:26:25.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:25.680 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.68926.asok status 2024-03-06T07:26:25.820 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:26:25.820 INFO:teuthology.orchestra.run.smithi083.stdout: "metadata": { 2024-03-06T07:26:25.820 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:26:25.820 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:26:25.820 INFO:teuthology.orchestra.run.smithi083.stdout: "entity_id": "0", 2024-03-06T07:26:25.820 INFO:teuthology.orchestra.run.smithi083.stdout: "hostname": "smithi083", 2024-03-06T07:26:25.820 INFO:teuthology.orchestra.run.smithi083.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:26:25.820 INFO:teuthology.orchestra.run.smithi083.stdout: "pid": "68926", 2024-03-06T07:26:25.821 INFO:teuthology.orchestra.run.smithi083.stdout: "root": "/" 2024-03-06T07:26:25.821 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:26:25.821 INFO:teuthology.orchestra.run.smithi083.stdout: "dentry_count": 0, 2024-03-06T07:26:25.821 INFO:teuthology.orchestra.run.smithi083.stdout: "dentry_pinned_count": 0, 2024-03-06T07:26:25.821 INFO:teuthology.orchestra.run.smithi083.stdout: "id": 7515, 2024-03-06T07:26:25.821 INFO:teuthology.orchestra.run.smithi083.stdout: "inst": { 2024-03-06T07:26:25.821 INFO:teuthology.orchestra.run.smithi083.stdout: "name": { 2024-03-06T07:26:25.821 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "client", 2024-03-06T07:26:25.821 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 7515 2024-03-06T07:26:25.821 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:26:25.821 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": { 2024-03-06T07:26:25.821 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-03-06T07:26:25.821 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:26:25.821 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 1063027794 2024-03-06T07:26:25.822 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:26:25.822 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:26:25.822 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": { 2024-03-06T07:26:25.822 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-03-06T07:26:25.822 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:26:25.822 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 1063027794 2024-03-06T07:26:25.822 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:26:25.822 INFO:teuthology.orchestra.run.smithi083.stdout: "inst_str": "client.7515 192.168.0.1:0/1063027794", 2024-03-06T07:26:25.822 INFO:teuthology.orchestra.run.smithi083.stdout: "addr_str": "192.168.0.1:0/1063027794", 2024-03-06T07:26:25.822 INFO:teuthology.orchestra.run.smithi083.stdout: "inode_count": 1, 2024-03-06T07:26:25.822 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_epoch": 45, 2024-03-06T07:26:25.822 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_epoch": 94, 2024-03-06T07:26:25.822 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:26:25.822 INFO:teuthology.orchestra.run.smithi083.stdout: "blocklisted": false, 2024-03-06T07:26:25.822 INFO:teuthology.orchestra.run.smithi083.stdout: "fs_name": "cephfs" 2024-03-06T07:26:25.823 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:26:25.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:25.829 DEBUG:teuthology.orchestra.run.smithi083:> stat --file-system '--printf=%T 2024-03-06T07:26:25.829 DEBUG:teuthology.orchestra.run.smithi083:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:25.844 INFO:teuthology.orchestra.run.smithi083.stdout:fuseblk 2024-03-06T07:26:25.844 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:25.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:25.845 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:25.913 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:25.955 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:26.003 INFO:teuthology.orchestra.run.smithi083.stdout:[global] 2024-03-06T07:26:26.003 INFO:teuthology.orchestra.run.smithi083.stdout: chdir = "" 2024-03-06T07:26:26.003 INFO:teuthology.orchestra.run.smithi083.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:26.004 INFO:teuthology.orchestra.run.smithi083.stdout: auth supported = cephx 2024-03-06T07:26:26.004 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.004 INFO:teuthology.orchestra.run.smithi083.stdout: filestore xattr use omap = true 2024-03-06T07:26:26.004 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.004 INFO:teuthology.orchestra.run.smithi083.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:26.004 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.004 INFO:teuthology.orchestra.run.smithi083.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:26.004 INFO:teuthology.orchestra.run.smithi083.stdout: auth debug = true 2024-03-06T07:26:26.004 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.004 INFO:teuthology.orchestra.run.smithi083.stdout: ms die on old message = true 2024-03-06T07:26:26.004 INFO:teuthology.orchestra.run.smithi083.stdout: ms die on bug = true 2024-03-06T07:26:26.004 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.005 INFO:teuthology.orchestra.run.smithi083.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:26.005 INFO:teuthology.orchestra.run.smithi083.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:26.005 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.005 INFO:teuthology.orchestra.run.smithi083.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:26.005 INFO:teuthology.orchestra.run.smithi083.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:26.005 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.005 INFO:teuthology.orchestra.run.smithi083.stdout: osd pool default size = 2 2024-03-06T07:26:26.005 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.005 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:26.005 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd allow pg remap = true 2024-03-06T07:26:26.005 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:26.005 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:26.005 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:26.005 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:26.005 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on too few osds = false 2024-03-06T07:26:26.006 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:26.006 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:26.006 INFO:teuthology.orchestra.run.smithi083.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:26.006 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.006 INFO:teuthology.orchestra.run.smithi083.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:26.006 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.006 INFO:teuthology.orchestra.run.smithi083.stdout: osd default data pool replay window = 5 2024-03-06T07:26:26.006 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.006 INFO:teuthology.orchestra.run.smithi083.stdout: mon allow pool delete = true 2024-03-06T07:26:26.006 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.006 INFO:teuthology.orchestra.run.smithi083.stdout: mon cluster log file level = debug 2024-03-06T07:26:26.006 INFO:teuthology.orchestra.run.smithi083.stdout: debug asserts on shutdown = true 2024-03-06T07:26:26.006 INFO:teuthology.orchestra.run.smithi083.stdout: mon health detail to clog = false 2024-03-06T07:26:26.006 INFO:teuthology.orchestra.run.smithi083.stdout: mon host = "172.21.15.83,172.21.15.183,[v2:172.21.15.183:3301,v1:172.21.15.183:6790]" 2024-03-06T07:26:26.006 INFO:teuthology.orchestra.run.smithi083.stdout: fsid = 039877dd-d2ce-45d2-8e77-ac5eb7e1d55a 2024-03-06T07:26:26.007 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.007 INFO:teuthology.orchestra.run.smithi083.stdout:[osd] 2024-03-06T07:26:26.007 INFO:teuthology.orchestra.run.smithi083.stdout: osd journal size = 100 2024-03-06T07:26:26.007 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.007 INFO:teuthology.orchestra.run.smithi083.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:26.007 INFO:teuthology.orchestra.run.smithi083.stdout: osd scrub max interval = 600 2024-03-06T07:26:26.007 INFO:teuthology.orchestra.run.smithi083.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:26.007 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.007 INFO:teuthology.orchestra.run.smithi083.stdout: osd recover clone overlap = true 2024-03-06T07:26:26.007 INFO:teuthology.orchestra.run.smithi083.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:26.007 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.007 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug shutdown = true 2024-03-06T07:26:26.007 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug op order = true 2024-03-06T07:26:26.007 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:26.007 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug trim objects = true 2024-03-06T07:26:26.007 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.008 INFO:teuthology.orchestra.run.smithi083.stdout: osd open classes on start = true 2024-03-06T07:26:26.008 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug pg log writeout = true 2024-03-06T07:26:26.008 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.008 INFO:teuthology.orchestra.run.smithi083.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:26.008 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.008 INFO:teuthology.orchestra.run.smithi083.stdout: osd map max advance = 10 2024-03-06T07:26:26.008 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.008 INFO:teuthology.orchestra.run.smithi083.stdout: journal zero on create = true 2024-03-06T07:26:26.008 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.008 INFO:teuthology.orchestra.run.smithi083.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:26.008 INFO:teuthology.orchestra.run.smithi083.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:26.008 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.008 INFO:teuthology.orchestra.run.smithi083.stdout: bdev debug aio = true 2024-03-06T07:26:26.008 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug misdirected ops = true 2024-03-06T07:26:26.008 INFO:teuthology.orchestra.run.smithi083.stdout: bdev async discard = True 2024-03-06T07:26:26.008 INFO:teuthology.orchestra.run.smithi083.stdout: bdev enable discard = True 2024-03-06T07:26:26.009 INFO:teuthology.orchestra.run.smithi083.stdout: bluestore allocator = bitmap 2024-03-06T07:26:26.009 INFO:teuthology.orchestra.run.smithi083.stdout: bluestore block size = 96636764160 2024-03-06T07:26:26.009 INFO:teuthology.orchestra.run.smithi083.stdout: bluestore fsck on mount = True 2024-03-06T07:26:26.009 INFO:teuthology.orchestra.run.smithi083.stdout: debug bluefs = 1/20 2024-03-06T07:26:26.009 INFO:teuthology.orchestra.run.smithi083.stdout: debug bluestore = 1/20 2024-03-06T07:26:26.009 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-06T07:26:26.009 INFO:teuthology.orchestra.run.smithi083.stdout: debug osd = 20 2024-03-06T07:26:26.009 INFO:teuthology.orchestra.run.smithi083.stdout: debug rocksdb = 4/10 2024-03-06T07:26:26.009 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:26.009 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:26.009 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:26.009 INFO:teuthology.orchestra.run.smithi083.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:26.009 INFO:teuthology.orchestra.run.smithi083.stdout: osd objectstore = bluestore 2024-03-06T07:26:26.009 INFO:teuthology.orchestra.run.smithi083.stdout: osd op complaint time = 180 2024-03-06T07:26:26.009 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.009 INFO:teuthology.orchestra.run.smithi083.stdout:[mgr] 2024-03-06T07:26:26.010 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-06T07:26:26.010 INFO:teuthology.orchestra.run.smithi083.stdout: debug mgr = 20 2024-03-06T07:26:26.010 INFO:teuthology.orchestra.run.smithi083.stdout: debug mon = 20 2024-03-06T07:26:26.010 INFO:teuthology.orchestra.run.smithi083.stdout: debug auth = 20 2024-03-06T07:26:26.010 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:26.010 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:26.010 INFO:teuthology.orchestra.run.smithi083.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:26.010 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.010 INFO:teuthology.orchestra.run.smithi083.stdout:[mon] 2024-03-06T07:26:26.010 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-06T07:26:26.010 INFO:teuthology.orchestra.run.smithi083.stdout: debug mon = 20 2024-03-06T07:26:26.010 INFO:teuthology.orchestra.run.smithi083.stdout: debug paxos = 20 2024-03-06T07:26:26.010 INFO:teuthology.orchestra.run.smithi083.stdout: debug auth = 20 2024-03-06T07:26:26.011 INFO:teuthology.orchestra.run.smithi083.stdout: mon data avail warn = 5 2024-03-06T07:26:26.011 INFO:teuthology.orchestra.run.smithi083.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:26.011 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:26.011 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:26.011 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd prime pg temp = true 2024-03-06T07:26:26.011 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:26.011 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.011 INFO:teuthology.orchestra.run.smithi083.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:26.011 INFO:teuthology.orchestra.run.smithi083.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:26.011 INFO:teuthology.orchestra.run.smithi083.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:26.011 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.011 INFO:teuthology.orchestra.run.smithi083.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:26.011 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:26.011 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:26.012 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.012 INFO:teuthology.orchestra.run.smithi083.stdout: # 1m isn't quite enough 2024-03-06T07:26:26.012 INFO:teuthology.orchestra.run.smithi083.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:26.012 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.012 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:26.012 INFO:teuthology.orchestra.run.smithi083.stdout: mon op complaint time = 120 2024-03-06T07:26:26.012 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.012 INFO:teuthology.orchestra.run.smithi083.stdout:[client] 2024-03-06T07:26:26.012 INFO:teuthology.orchestra.run.smithi083.stdout: rgw cache enabled = true 2024-03-06T07:26:26.012 INFO:teuthology.orchestra.run.smithi083.stdout: rgw enable ops log = true 2024-03-06T07:26:26.012 INFO:teuthology.orchestra.run.smithi083.stdout: rgw enable usage log = true 2024-03-06T07:26:26.012 INFO:teuthology.orchestra.run.smithi083.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:26.012 INFO:teuthology.orchestra.run.smithi083.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:26.012 INFO:teuthology.orchestra.run.smithi083.stdout: client mount timeout = 600 2024-03-06T07:26:26.013 INFO:teuthology.orchestra.run.smithi083.stdout: debug client = 20 2024-03-06T07:26:26.013 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-06T07:26:26.013 INFO:teuthology.orchestra.run.smithi083.stdout: pid file = "" 2024-03-06T07:26:26.013 INFO:teuthology.orchestra.run.smithi083.stdout: rados mon op timeout = 900 2024-03-06T07:26:26.013 INFO:teuthology.orchestra.run.smithi083.stdout: rados osd op timeout = 900 2024-03-06T07:26:26.013 INFO:teuthology.orchestra.run.smithi083.stdout:[mon.a] 2024-03-06T07:26:26.013 INFO:teuthology.orchestra.run.smithi083.stdout:[mon.b] 2024-03-06T07:26:26.013 INFO:teuthology.orchestra.run.smithi083.stdout:[mon.c] 2024-03-06T07:26:26.013 INFO:teuthology.orchestra.run.smithi083.stdout:[mds.a] 2024-03-06T07:26:26.013 INFO:teuthology.orchestra.run.smithi083.stdout:[mds.c] 2024-03-06T07:26:26.013 INFO:teuthology.orchestra.run.smithi083.stdout:[mds.b] 2024-03-06T07:26:26.013 INFO:teuthology.orchestra.run.smithi083.stdout:[mds.d] 2024-03-06T07:26:26.013 INFO:teuthology.orchestra.run.smithi083.stdout:[mds] 2024-03-06T07:26:26.013 INFO:teuthology.orchestra.run.smithi083.stdout: debug mds = 20 2024-03-06T07:26:26.014 INFO:teuthology.orchestra.run.smithi083.stdout: debug mds balancer = 20 2024-03-06T07:26:26.014 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-06T07:26:26.014 INFO:teuthology.orchestra.run.smithi083.stdout: mds debug frag = True 2024-03-06T07:26:26.014 INFO:teuthology.orchestra.run.smithi083.stdout: mds debug scatterstat = True 2024-03-06T07:26:26.014 INFO:teuthology.orchestra.run.smithi083.stdout: mds op complaint time = 180 2024-03-06T07:26:26.014 INFO:teuthology.orchestra.run.smithi083.stdout: mds verify scatter = True 2024-03-06T07:26:26.014 INFO:teuthology.orchestra.run.smithi083.stdout: osd op complaint time = 180 2024-03-06T07:26:26.014 INFO:teuthology.orchestra.run.smithi083.stdout: rados mon op timeout = 900 2024-03-06T07:26:26.014 INFO:teuthology.orchestra.run.smithi083.stdout: rados osd op timeout = 900 2024-03-06T07:26:26.014 INFO:teuthology.orchestra.run.smithi083.stdout: mds_max_purge_files = 6 2024-03-06T07:26:26.015 INFO:teuthology.orchestra.run.smithi183.stdout:[global] 2024-03-06T07:26:26.015 INFO:teuthology.orchestra.run.smithi183.stdout: chdir = "" 2024-03-06T07:26:26.015 INFO:teuthology.orchestra.run.smithi183.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:26.015 INFO:teuthology.orchestra.run.smithi183.stdout: auth supported = cephx 2024-03-06T07:26:26.015 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.015 INFO:teuthology.orchestra.run.smithi183.stdout: filestore xattr use omap = true 2024-03-06T07:26:26.015 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.015 INFO:teuthology.orchestra.run.smithi183.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:26.015 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.015 INFO:teuthology.orchestra.run.smithi183.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:26.015 INFO:teuthology.orchestra.run.smithi183.stdout: auth debug = true 2024-03-06T07:26:26.015 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.015 INFO:teuthology.orchestra.run.smithi183.stdout: ms die on old message = true 2024-03-06T07:26:26.015 INFO:teuthology.orchestra.run.smithi183.stdout: ms die on bug = true 2024-03-06T07:26:26.016 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.016 INFO:teuthology.orchestra.run.smithi183.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:26.016 INFO:teuthology.orchestra.run.smithi183.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:26.016 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.016 INFO:teuthology.orchestra.run.smithi183.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:26.016 INFO:teuthology.orchestra.run.smithi183.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:26.016 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.016 INFO:teuthology.orchestra.run.smithi183.stdout: osd pool default size = 2 2024-03-06T07:26:26.016 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.016 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:26.016 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd allow pg remap = true 2024-03-06T07:26:26.016 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:26.016 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:26.016 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:26.017 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:26.017 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on too few osds = false 2024-03-06T07:26:26.017 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:26.017 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:26.017 INFO:teuthology.orchestra.run.smithi183.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:26.017 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.017 INFO:teuthology.orchestra.run.smithi183.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:26.017 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.017 INFO:teuthology.orchestra.run.smithi183.stdout: osd default data pool replay window = 5 2024-03-06T07:26:26.017 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.017 INFO:teuthology.orchestra.run.smithi183.stdout: mon allow pool delete = true 2024-03-06T07:26:26.017 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.017 INFO:teuthology.orchestra.run.smithi183.stdout: mon cluster log file level = debug 2024-03-06T07:26:26.017 INFO:teuthology.orchestra.run.smithi183.stdout: debug asserts on shutdown = true 2024-03-06T07:26:26.017 INFO:teuthology.orchestra.run.smithi183.stdout: mon health detail to clog = false 2024-03-06T07:26:26.018 INFO:teuthology.orchestra.run.smithi183.stdout: mon host = "172.21.15.83,172.21.15.183,[v2:172.21.15.183:3301,v1:172.21.15.183:6790]" 2024-03-06T07:26:26.018 INFO:teuthology.orchestra.run.smithi183.stdout: fsid = 039877dd-d2ce-45d2-8e77-ac5eb7e1d55a 2024-03-06T07:26:26.018 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.018 INFO:teuthology.orchestra.run.smithi183.stdout:[osd] 2024-03-06T07:26:26.018 INFO:teuthology.orchestra.run.smithi183.stdout: osd journal size = 100 2024-03-06T07:26:26.018 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.018 INFO:teuthology.orchestra.run.smithi183.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:26.018 INFO:teuthology.orchestra.run.smithi183.stdout: osd scrub max interval = 600 2024-03-06T07:26:26.018 INFO:teuthology.orchestra.run.smithi183.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:26.018 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.018 INFO:teuthology.orchestra.run.smithi183.stdout: osd recover clone overlap = true 2024-03-06T07:26:26.018 INFO:teuthology.orchestra.run.smithi183.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:26.018 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.018 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug shutdown = true 2024-03-06T07:26:26.018 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug op order = true 2024-03-06T07:26:26.019 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:26.019 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug trim objects = true 2024-03-06T07:26:26.019 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.019 INFO:teuthology.orchestra.run.smithi183.stdout: osd open classes on start = true 2024-03-06T07:26:26.019 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug pg log writeout = true 2024-03-06T07:26:26.019 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.019 INFO:teuthology.orchestra.run.smithi183.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:26.019 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.019 INFO:teuthology.orchestra.run.smithi183.stdout: osd map max advance = 10 2024-03-06T07:26:26.019 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.019 INFO:teuthology.orchestra.run.smithi183.stdout: journal zero on create = true 2024-03-06T07:26:26.019 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.019 INFO:teuthology.orchestra.run.smithi183.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:26.019 INFO:teuthology.orchestra.run.smithi183.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:26.019 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.019 INFO:teuthology.orchestra.run.smithi183.stdout: bdev debug aio = true 2024-03-06T07:26:26.020 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug misdirected ops = true 2024-03-06T07:26:26.020 INFO:teuthology.orchestra.run.smithi183.stdout: bdev async discard = True 2024-03-06T07:26:26.020 INFO:teuthology.orchestra.run.smithi183.stdout: bdev enable discard = True 2024-03-06T07:26:26.020 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore allocator = bitmap 2024-03-06T07:26:26.020 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore block size = 96636764160 2024-03-06T07:26:26.020 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore fsck on mount = True 2024-03-06T07:26:26.020 INFO:teuthology.orchestra.run.smithi183.stdout: debug bluefs = 1/20 2024-03-06T07:26:26.020 INFO:teuthology.orchestra.run.smithi183.stdout: debug bluestore = 1/20 2024-03-06T07:26:26.020 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-06T07:26:26.020 INFO:teuthology.orchestra.run.smithi183.stdout: debug osd = 20 2024-03-06T07:26:26.020 INFO:teuthology.orchestra.run.smithi183.stdout: debug rocksdb = 4/10 2024-03-06T07:26:26.020 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:26.020 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:26.020 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:26.021 INFO:teuthology.orchestra.run.smithi183.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:26.021 INFO:teuthology.orchestra.run.smithi183.stdout: osd objectstore = bluestore 2024-03-06T07:26:26.021 INFO:teuthology.orchestra.run.smithi183.stdout: osd op complaint time = 180 2024-03-06T07:26:26.021 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.021 INFO:teuthology.orchestra.run.smithi183.stdout:[mgr] 2024-03-06T07:26:26.021 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-06T07:26:26.021 INFO:teuthology.orchestra.run.smithi183.stdout: debug mgr = 20 2024-03-06T07:26:26.021 INFO:teuthology.orchestra.run.smithi183.stdout: debug mon = 20 2024-03-06T07:26:26.021 INFO:teuthology.orchestra.run.smithi183.stdout: debug auth = 20 2024-03-06T07:26:26.021 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:26.021 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:26.021 INFO:teuthology.orchestra.run.smithi183.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:26.021 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.021 INFO:teuthology.orchestra.run.smithi183.stdout:[mon] 2024-03-06T07:26:26.021 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-06T07:26:26.022 INFO:teuthology.orchestra.run.smithi183.stdout: debug mon = 20 2024-03-06T07:26:26.022 INFO:teuthology.orchestra.run.smithi183.stdout: debug paxos = 20 2024-03-06T07:26:26.022 INFO:teuthology.orchestra.run.smithi183.stdout: debug auth = 20 2024-03-06T07:26:26.022 INFO:teuthology.orchestra.run.smithi183.stdout: mon data avail warn = 5 2024-03-06T07:26:26.022 INFO:teuthology.orchestra.run.smithi183.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:26.022 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:26.022 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:26.022 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd prime pg temp = true 2024-03-06T07:26:26.022 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:26.022 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.022 INFO:teuthology.orchestra.run.smithi183.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:26.022 INFO:teuthology.orchestra.run.smithi183.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:26.022 INFO:teuthology.orchestra.run.smithi183.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:26.022 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.022 INFO:teuthology.orchestra.run.smithi183.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:26.023 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:26.023 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:26.023 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.023 INFO:teuthology.orchestra.run.smithi183.stdout: # 1m isn't quite enough 2024-03-06T07:26:26.023 INFO:teuthology.orchestra.run.smithi183.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:26.023 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.023 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:26.023 INFO:teuthology.orchestra.run.smithi183.stdout: mon op complaint time = 120 2024-03-06T07:26:26.023 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.023 INFO:teuthology.orchestra.run.smithi183.stdout:[client] 2024-03-06T07:26:26.023 INFO:teuthology.orchestra.run.smithi183.stdout: rgw cache enabled = true 2024-03-06T07:26:26.023 INFO:teuthology.orchestra.run.smithi183.stdout: rgw enable ops log = true 2024-03-06T07:26:26.023 INFO:teuthology.orchestra.run.smithi183.stdout: rgw enable usage log = true 2024-03-06T07:26:26.023 INFO:teuthology.orchestra.run.smithi183.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:26.024 INFO:teuthology.orchestra.run.smithi183.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:26.024 INFO:teuthology.orchestra.run.smithi183.stdout: client mount timeout = 600 2024-03-06T07:26:26.024 INFO:teuthology.orchestra.run.smithi183.stdout: debug client = 20 2024-03-06T07:26:26.024 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-06T07:26:26.024 INFO:teuthology.orchestra.run.smithi183.stdout: pid file = "" 2024-03-06T07:26:26.024 INFO:teuthology.orchestra.run.smithi183.stdout: rados mon op timeout = 900 2024-03-06T07:26:26.024 INFO:teuthology.orchestra.run.smithi183.stdout: rados osd op timeout = 900 2024-03-06T07:26:26.024 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.a] 2024-03-06T07:26:26.024 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.b] 2024-03-06T07:26:26.024 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.c] 2024-03-06T07:26:26.024 INFO:teuthology.orchestra.run.smithi183.stdout:[mds.a] 2024-03-06T07:26:26.024 INFO:teuthology.orchestra.run.smithi183.stdout:[mds.c] 2024-03-06T07:26:26.024 INFO:teuthology.orchestra.run.smithi183.stdout:[mds.b] 2024-03-06T07:26:26.024 INFO:teuthology.orchestra.run.smithi183.stdout:[mds.d] 2024-03-06T07:26:26.024 INFO:teuthology.orchestra.run.smithi183.stdout:[mds] 2024-03-06T07:26:26.025 INFO:teuthology.orchestra.run.smithi183.stdout: debug mds = 20 2024-03-06T07:26:26.025 INFO:teuthology.orchestra.run.smithi183.stdout: debug mds balancer = 20 2024-03-06T07:26:26.025 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-06T07:26:26.025 INFO:teuthology.orchestra.run.smithi183.stdout: mds debug frag = True 2024-03-06T07:26:26.025 INFO:teuthology.orchestra.run.smithi183.stdout: mds debug scatterstat = True 2024-03-06T07:26:26.025 INFO:teuthology.orchestra.run.smithi183.stdout: mds op complaint time = 180 2024-03-06T07:26:26.025 INFO:teuthology.orchestra.run.smithi183.stdout: mds verify scatter = True 2024-03-06T07:26:26.025 INFO:teuthology.orchestra.run.smithi183.stdout: osd op complaint time = 180 2024-03-06T07:26:26.025 INFO:teuthology.orchestra.run.smithi183.stdout: rados mon op timeout = 900 2024-03-06T07:26:26.025 INFO:teuthology.orchestra.run.smithi183.stdout: rados osd op timeout = 900 2024-03-06T07:26:26.025 INFO:teuthology.orchestra.run.smithi183.stdout: mds_max_purge_files = 6 2024-03-06T07:26:26.029 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:26.069 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:26.118 INFO:teuthology.orchestra.run.smithi083.stdout:[global] 2024-03-06T07:26:26.118 INFO:teuthology.orchestra.run.smithi083.stdout: chdir = "" 2024-03-06T07:26:26.118 INFO:teuthology.orchestra.run.smithi083.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:26.118 INFO:teuthology.orchestra.run.smithi083.stdout: auth supported = cephx 2024-03-06T07:26:26.118 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.118 INFO:teuthology.orchestra.run.smithi083.stdout: filestore xattr use omap = true 2024-03-06T07:26:26.118 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.118 INFO:teuthology.orchestra.run.smithi083.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:26.119 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.119 INFO:teuthology.orchestra.run.smithi083.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:26.119 INFO:teuthology.orchestra.run.smithi083.stdout: auth debug = true 2024-03-06T07:26:26.119 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.119 INFO:teuthology.orchestra.run.smithi083.stdout: ms die on old message = true 2024-03-06T07:26:26.119 INFO:teuthology.orchestra.run.smithi083.stdout: ms die on bug = true 2024-03-06T07:26:26.119 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.119 INFO:teuthology.orchestra.run.smithi083.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:26.119 INFO:teuthology.orchestra.run.smithi083.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:26.119 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.119 INFO:teuthology.orchestra.run.smithi083.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:26.119 INFO:teuthology.orchestra.run.smithi083.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:26.119 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.119 INFO:teuthology.orchestra.run.smithi083.stdout: osd pool default size = 2 2024-03-06T07:26:26.119 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.120 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:26.120 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd allow pg remap = true 2024-03-06T07:26:26.120 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:26.120 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:26.120 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:26.120 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:26.120 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on too few osds = false 2024-03-06T07:26:26.120 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:26.120 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:26.120 INFO:teuthology.orchestra.run.smithi083.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:26.120 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.120 INFO:teuthology.orchestra.run.smithi083.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:26.120 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.120 INFO:teuthology.orchestra.run.smithi083.stdout: osd default data pool replay window = 5 2024-03-06T07:26:26.121 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.121 INFO:teuthology.orchestra.run.smithi083.stdout: mon allow pool delete = true 2024-03-06T07:26:26.121 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.121 INFO:teuthology.orchestra.run.smithi083.stdout: mon cluster log file level = debug 2024-03-06T07:26:26.121 INFO:teuthology.orchestra.run.smithi083.stdout: debug asserts on shutdown = true 2024-03-06T07:26:26.121 INFO:teuthology.orchestra.run.smithi083.stdout: mon health detail to clog = false 2024-03-06T07:26:26.121 INFO:teuthology.orchestra.run.smithi083.stdout: mon host = "172.21.15.83,172.21.15.183,[v2:172.21.15.183:3301,v1:172.21.15.183:6790]" 2024-03-06T07:26:26.121 INFO:teuthology.orchestra.run.smithi083.stdout: fsid = 039877dd-d2ce-45d2-8e77-ac5eb7e1d55a 2024-03-06T07:26:26.121 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.121 INFO:teuthology.orchestra.run.smithi083.stdout:[osd] 2024-03-06T07:26:26.121 INFO:teuthology.orchestra.run.smithi083.stdout: osd journal size = 100 2024-03-06T07:26:26.121 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.121 INFO:teuthology.orchestra.run.smithi083.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:26.121 INFO:teuthology.orchestra.run.smithi083.stdout: osd scrub max interval = 600 2024-03-06T07:26:26.121 INFO:teuthology.orchestra.run.smithi083.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:26.122 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.122 INFO:teuthology.orchestra.run.smithi083.stdout: osd recover clone overlap = true 2024-03-06T07:26:26.122 INFO:teuthology.orchestra.run.smithi083.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:26.122 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.122 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug shutdown = true 2024-03-06T07:26:26.122 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug op order = true 2024-03-06T07:26:26.122 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:26.122 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug trim objects = true 2024-03-06T07:26:26.122 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.122 INFO:teuthology.orchestra.run.smithi083.stdout: osd open classes on start = true 2024-03-06T07:26:26.122 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug pg log writeout = true 2024-03-06T07:26:26.122 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.122 INFO:teuthology.orchestra.run.smithi083.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:26.122 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.122 INFO:teuthology.orchestra.run.smithi083.stdout: osd map max advance = 10 2024-03-06T07:26:26.123 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.123 INFO:teuthology.orchestra.run.smithi083.stdout: journal zero on create = true 2024-03-06T07:26:26.123 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.123 INFO:teuthology.orchestra.run.smithi083.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:26.123 INFO:teuthology.orchestra.run.smithi083.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:26.123 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.123 INFO:teuthology.orchestra.run.smithi083.stdout: bdev debug aio = true 2024-03-06T07:26:26.123 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug misdirected ops = true 2024-03-06T07:26:26.123 INFO:teuthology.orchestra.run.smithi083.stdout: bdev async discard = True 2024-03-06T07:26:26.123 INFO:teuthology.orchestra.run.smithi083.stdout: bdev enable discard = True 2024-03-06T07:26:26.123 INFO:teuthology.orchestra.run.smithi083.stdout: bluestore allocator = bitmap 2024-03-06T07:26:26.123 INFO:teuthology.orchestra.run.smithi083.stdout: bluestore block size = 96636764160 2024-03-06T07:26:26.123 INFO:teuthology.orchestra.run.smithi083.stdout: bluestore fsck on mount = True 2024-03-06T07:26:26.123 INFO:teuthology.orchestra.run.smithi083.stdout: debug bluefs = 1/20 2024-03-06T07:26:26.123 INFO:teuthology.orchestra.run.smithi083.stdout: debug bluestore = 1/20 2024-03-06T07:26:26.123 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-06T07:26:26.124 INFO:teuthology.orchestra.run.smithi083.stdout: debug osd = 20 2024-03-06T07:26:26.124 INFO:teuthology.orchestra.run.smithi083.stdout: debug rocksdb = 4/10 2024-03-06T07:26:26.124 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:26.124 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:26.124 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:26.124 INFO:teuthology.orchestra.run.smithi083.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:26.124 INFO:teuthology.orchestra.run.smithi083.stdout: osd objectstore = bluestore 2024-03-06T07:26:26.124 INFO:teuthology.orchestra.run.smithi083.stdout: osd op complaint time = 180 2024-03-06T07:26:26.124 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.124 INFO:teuthology.orchestra.run.smithi083.stdout:[mgr] 2024-03-06T07:26:26.124 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-06T07:26:26.124 INFO:teuthology.orchestra.run.smithi083.stdout: debug mgr = 20 2024-03-06T07:26:26.124 INFO:teuthology.orchestra.run.smithi083.stdout: debug mon = 20 2024-03-06T07:26:26.124 INFO:teuthology.orchestra.run.smithi083.stdout: debug auth = 20 2024-03-06T07:26:26.124 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:26.125 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:26.125 INFO:teuthology.orchestra.run.smithi083.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:26.125 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.125 INFO:teuthology.orchestra.run.smithi083.stdout:[mon] 2024-03-06T07:26:26.125 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-06T07:26:26.125 INFO:teuthology.orchestra.run.smithi083.stdout: debug mon = 20 2024-03-06T07:26:26.125 INFO:teuthology.orchestra.run.smithi083.stdout: debug paxos = 20 2024-03-06T07:26:26.125 INFO:teuthology.orchestra.run.smithi083.stdout: debug auth = 20 2024-03-06T07:26:26.125 INFO:teuthology.orchestra.run.smithi083.stdout: mon data avail warn = 5 2024-03-06T07:26:26.125 INFO:teuthology.orchestra.run.smithi083.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:26.125 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:26.125 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:26.125 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd prime pg temp = true 2024-03-06T07:26:26.125 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:26.126 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.126 INFO:teuthology.orchestra.run.smithi083.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:26.126 INFO:teuthology.orchestra.run.smithi083.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:26.126 INFO:teuthology.orchestra.run.smithi083.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:26.126 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.126 INFO:teuthology.orchestra.run.smithi083.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:26.126 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:26.126 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:26.126 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.126 INFO:teuthology.orchestra.run.smithi083.stdout: # 1m isn't quite enough 2024-03-06T07:26:26.126 INFO:teuthology.orchestra.run.smithi083.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:26.126 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.126 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:26.126 INFO:teuthology.orchestra.run.smithi083.stdout: mon op complaint time = 120 2024-03-06T07:26:26.127 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.127 INFO:teuthology.orchestra.run.smithi083.stdout:[client] 2024-03-06T07:26:26.127 INFO:teuthology.orchestra.run.smithi083.stdout: rgw cache enabled = true 2024-03-06T07:26:26.127 INFO:teuthology.orchestra.run.smithi083.stdout: rgw enable ops log = true 2024-03-06T07:26:26.127 INFO:teuthology.orchestra.run.smithi083.stdout: rgw enable usage log = true 2024-03-06T07:26:26.127 INFO:teuthology.orchestra.run.smithi083.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:26.127 INFO:teuthology.orchestra.run.smithi083.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:26.127 INFO:teuthology.orchestra.run.smithi083.stdout: client mount timeout = 600 2024-03-06T07:26:26.127 INFO:teuthology.orchestra.run.smithi083.stdout: debug client = 20 2024-03-06T07:26:26.127 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-06T07:26:26.127 INFO:teuthology.orchestra.run.smithi083.stdout: pid file = "" 2024-03-06T07:26:26.127 INFO:teuthology.orchestra.run.smithi083.stdout: rados mon op timeout = 900 2024-03-06T07:26:26.127 INFO:teuthology.orchestra.run.smithi083.stdout: rados osd op timeout = 900 2024-03-06T07:26:26.128 INFO:teuthology.orchestra.run.smithi083.stdout:[mon.a] 2024-03-06T07:26:26.128 INFO:teuthology.orchestra.run.smithi083.stdout:[mon.b] 2024-03-06T07:26:26.128 INFO:teuthology.orchestra.run.smithi083.stdout:[mon.c] 2024-03-06T07:26:26.128 INFO:teuthology.orchestra.run.smithi083.stdout:[mds.a] 2024-03-06T07:26:26.128 INFO:teuthology.orchestra.run.smithi083.stdout:[mds.c] 2024-03-06T07:26:26.128 INFO:teuthology.orchestra.run.smithi083.stdout:[mds.b] 2024-03-06T07:26:26.128 INFO:teuthology.orchestra.run.smithi083.stdout:[mds.d] 2024-03-06T07:26:26.128 INFO:teuthology.orchestra.run.smithi083.stdout:[mds] 2024-03-06T07:26:26.128 INFO:teuthology.orchestra.run.smithi083.stdout: debug mds = 20 2024-03-06T07:26:26.128 INFO:teuthology.orchestra.run.smithi083.stdout: debug mds balancer = 20 2024-03-06T07:26:26.128 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-06T07:26:26.128 INFO:teuthology.orchestra.run.smithi083.stdout: mds debug frag = True 2024-03-06T07:26:26.128 INFO:teuthology.orchestra.run.smithi083.stdout: mds debug scatterstat = True 2024-03-06T07:26:26.128 INFO:teuthology.orchestra.run.smithi083.stdout: mds op complaint time = 180 2024-03-06T07:26:26.129 INFO:teuthology.orchestra.run.smithi083.stdout: mds verify scatter = True 2024-03-06T07:26:26.129 INFO:teuthology.orchestra.run.smithi083.stdout: osd op complaint time = 180 2024-03-06T07:26:26.129 INFO:teuthology.orchestra.run.smithi083.stdout: rados mon op timeout = 900 2024-03-06T07:26:26.129 INFO:teuthology.orchestra.run.smithi083.stdout: rados osd op timeout = 900 2024-03-06T07:26:26.129 INFO:teuthology.orchestra.run.smithi083.stdout: mds_max_purge_files = 6 2024-03-06T07:26:26.129 INFO:teuthology.orchestra.run.smithi083.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:26:26.129 INFO:teuthology.orchestra.run.smithi183.stdout:[global] 2024-03-06T07:26:26.129 INFO:teuthology.orchestra.run.smithi183.stdout: chdir = "" 2024-03-06T07:26:26.130 INFO:teuthology.orchestra.run.smithi183.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:26.130 INFO:teuthology.orchestra.run.smithi183.stdout: auth supported = cephx 2024-03-06T07:26:26.130 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.130 INFO:teuthology.orchestra.run.smithi183.stdout: filestore xattr use omap = true 2024-03-06T07:26:26.130 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.130 INFO:teuthology.orchestra.run.smithi183.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:26.130 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.130 INFO:teuthology.orchestra.run.smithi183.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:26.130 INFO:teuthology.orchestra.run.smithi183.stdout: auth debug = true 2024-03-06T07:26:26.130 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.130 INFO:teuthology.orchestra.run.smithi183.stdout: ms die on old message = true 2024-03-06T07:26:26.130 INFO:teuthology.orchestra.run.smithi183.stdout: ms die on bug = true 2024-03-06T07:26:26.130 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.131 INFO:teuthology.orchestra.run.smithi183.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:26.131 INFO:teuthology.orchestra.run.smithi183.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:26.131 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.131 INFO:teuthology.orchestra.run.smithi183.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:26.131 INFO:teuthology.orchestra.run.smithi183.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:26.131 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.131 INFO:teuthology.orchestra.run.smithi183.stdout: osd pool default size = 2 2024-03-06T07:26:26.131 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.131 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:26.131 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd allow pg remap = true 2024-03-06T07:26:26.131 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:26.131 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:26.131 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:26.131 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:26.132 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on too few osds = false 2024-03-06T07:26:26.132 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:26.132 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:26.132 INFO:teuthology.orchestra.run.smithi183.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:26.132 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.132 INFO:teuthology.orchestra.run.smithi183.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:26.132 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.132 INFO:teuthology.orchestra.run.smithi183.stdout: osd default data pool replay window = 5 2024-03-06T07:26:26.132 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.132 INFO:teuthology.orchestra.run.smithi183.stdout: mon allow pool delete = true 2024-03-06T07:26:26.132 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.132 INFO:teuthology.orchestra.run.smithi183.stdout: mon cluster log file level = debug 2024-03-06T07:26:26.132 INFO:teuthology.orchestra.run.smithi183.stdout: debug asserts on shutdown = true 2024-03-06T07:26:26.132 INFO:teuthology.orchestra.run.smithi183.stdout: mon health detail to clog = false 2024-03-06T07:26:26.133 INFO:teuthology.orchestra.run.smithi183.stdout: mon host = "172.21.15.83,172.21.15.183,[v2:172.21.15.183:3301,v1:172.21.15.183:6790]" 2024-03-06T07:26:26.133 INFO:teuthology.orchestra.run.smithi183.stdout: fsid = 039877dd-d2ce-45d2-8e77-ac5eb7e1d55a 2024-03-06T07:26:26.133 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.133 INFO:teuthology.orchestra.run.smithi183.stdout:[osd] 2024-03-06T07:26:26.133 INFO:teuthology.orchestra.run.smithi183.stdout: osd journal size = 100 2024-03-06T07:26:26.133 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.133 INFO:teuthology.orchestra.run.smithi183.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:26.133 INFO:teuthology.orchestra.run.smithi183.stdout: osd scrub max interval = 600 2024-03-06T07:26:26.133 INFO:teuthology.orchestra.run.smithi183.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:26.133 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.133 INFO:teuthology.orchestra.run.smithi183.stdout: osd recover clone overlap = true 2024-03-06T07:26:26.133 INFO:teuthology.orchestra.run.smithi183.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:26.133 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.134 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug shutdown = true 2024-03-06T07:26:26.134 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug op order = true 2024-03-06T07:26:26.134 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:26.134 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug trim objects = true 2024-03-06T07:26:26.134 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.134 INFO:teuthology.orchestra.run.smithi183.stdout: osd open classes on start = true 2024-03-06T07:26:26.134 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug pg log writeout = true 2024-03-06T07:26:26.134 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.134 INFO:teuthology.orchestra.run.smithi183.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:26.134 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.134 INFO:teuthology.orchestra.run.smithi183.stdout: osd map max advance = 10 2024-03-06T07:26:26.134 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.134 INFO:teuthology.orchestra.run.smithi183.stdout: journal zero on create = true 2024-03-06T07:26:26.135 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.135 INFO:teuthology.orchestra.run.smithi183.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:26.135 INFO:teuthology.orchestra.run.smithi183.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:26.135 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.135 INFO:teuthology.orchestra.run.smithi183.stdout: bdev debug aio = true 2024-03-06T07:26:26.135 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug misdirected ops = true 2024-03-06T07:26:26.135 INFO:teuthology.orchestra.run.smithi183.stdout: bdev async discard = True 2024-03-06T07:26:26.135 INFO:teuthology.orchestra.run.smithi183.stdout: bdev enable discard = True 2024-03-06T07:26:26.135 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore allocator = bitmap 2024-03-06T07:26:26.135 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore block size = 96636764160 2024-03-06T07:26:26.135 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore fsck on mount = True 2024-03-06T07:26:26.135 INFO:teuthology.orchestra.run.smithi183.stdout: debug bluefs = 1/20 2024-03-06T07:26:26.135 INFO:teuthology.orchestra.run.smithi183.stdout: debug bluestore = 1/20 2024-03-06T07:26:26.135 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-06T07:26:26.135 INFO:teuthology.orchestra.run.smithi183.stdout: debug osd = 20 2024-03-06T07:26:26.136 INFO:teuthology.orchestra.run.smithi183.stdout: debug rocksdb = 4/10 2024-03-06T07:26:26.136 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:26.136 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:26.136 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:26.136 INFO:teuthology.orchestra.run.smithi183.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:26.136 INFO:teuthology.orchestra.run.smithi183.stdout: osd objectstore = bluestore 2024-03-06T07:26:26.136 INFO:teuthology.orchestra.run.smithi183.stdout: osd op complaint time = 180 2024-03-06T07:26:26.136 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.136 INFO:teuthology.orchestra.run.smithi183.stdout:[mgr] 2024-03-06T07:26:26.136 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-06T07:26:26.136 INFO:teuthology.orchestra.run.smithi183.stdout: debug mgr = 20 2024-03-06T07:26:26.136 INFO:teuthology.orchestra.run.smithi183.stdout: debug mon = 20 2024-03-06T07:26:26.136 INFO:teuthology.orchestra.run.smithi183.stdout: debug auth = 20 2024-03-06T07:26:26.137 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:26.137 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:26.137 INFO:teuthology.orchestra.run.smithi183.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:26.137 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.137 INFO:teuthology.orchestra.run.smithi183.stdout:[mon] 2024-03-06T07:26:26.137 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-06T07:26:26.137 INFO:teuthology.orchestra.run.smithi183.stdout: debug mon = 20 2024-03-06T07:26:26.137 INFO:teuthology.orchestra.run.smithi183.stdout: debug paxos = 20 2024-03-06T07:26:26.137 INFO:teuthology.orchestra.run.smithi183.stdout: debug auth = 20 2024-03-06T07:26:26.137 INFO:teuthology.orchestra.run.smithi183.stdout: mon data avail warn = 5 2024-03-06T07:26:26.137 INFO:teuthology.orchestra.run.smithi183.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:26.137 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:26.137 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:26.137 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd prime pg temp = true 2024-03-06T07:26:26.137 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:26.137 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.138 INFO:teuthology.orchestra.run.smithi183.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:26.138 INFO:teuthology.orchestra.run.smithi183.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:26.138 INFO:teuthology.orchestra.run.smithi183.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:26.138 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.138 INFO:teuthology.orchestra.run.smithi183.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:26.138 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:26.138 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:26.138 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.138 INFO:teuthology.orchestra.run.smithi183.stdout: # 1m isn't quite enough 2024-03-06T07:26:26.138 INFO:teuthology.orchestra.run.smithi183.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:26.138 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.138 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:26.138 INFO:teuthology.orchestra.run.smithi183.stdout: mon op complaint time = 120 2024-03-06T07:26:26.138 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.139 INFO:teuthology.orchestra.run.smithi183.stdout:[client] 2024-03-06T07:26:26.139 INFO:teuthology.orchestra.run.smithi183.stdout: rgw cache enabled = true 2024-03-06T07:26:26.139 INFO:teuthology.orchestra.run.smithi183.stdout: rgw enable ops log = true 2024-03-06T07:26:26.139 INFO:teuthology.orchestra.run.smithi183.stdout: rgw enable usage log = true 2024-03-06T07:26:26.139 INFO:teuthology.orchestra.run.smithi183.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:26.139 INFO:teuthology.orchestra.run.smithi183.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:26.139 INFO:teuthology.orchestra.run.smithi183.stdout: client mount timeout = 600 2024-03-06T07:26:26.139 INFO:teuthology.orchestra.run.smithi183.stdout: debug client = 20 2024-03-06T07:26:26.139 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-06T07:26:26.139 INFO:teuthology.orchestra.run.smithi183.stdout: pid file = "" 2024-03-06T07:26:26.139 INFO:teuthology.orchestra.run.smithi183.stdout: rados mon op timeout = 900 2024-03-06T07:26:26.139 INFO:teuthology.orchestra.run.smithi183.stdout: rados osd op timeout = 900 2024-03-06T07:26:26.139 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.a] 2024-03-06T07:26:26.139 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.b] 2024-03-06T07:26:26.140 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.c] 2024-03-06T07:26:26.140 INFO:teuthology.orchestra.run.smithi183.stdout:[mds.a] 2024-03-06T07:26:26.140 INFO:teuthology.orchestra.run.smithi183.stdout:[mds.c] 2024-03-06T07:26:26.140 INFO:teuthology.orchestra.run.smithi183.stdout:[mds.b] 2024-03-06T07:26:26.140 INFO:teuthology.orchestra.run.smithi183.stdout:[mds.d] 2024-03-06T07:26:26.140 INFO:teuthology.orchestra.run.smithi183.stdout:[mds] 2024-03-06T07:26:26.140 INFO:teuthology.orchestra.run.smithi183.stdout: debug mds = 20 2024-03-06T07:26:26.140 INFO:teuthology.orchestra.run.smithi183.stdout: debug mds balancer = 20 2024-03-06T07:26:26.140 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-06T07:26:26.140 INFO:teuthology.orchestra.run.smithi183.stdout: mds debug frag = True 2024-03-06T07:26:26.140 INFO:teuthology.orchestra.run.smithi183.stdout: mds debug scatterstat = True 2024-03-06T07:26:26.140 INFO:teuthology.orchestra.run.smithi183.stdout: mds op complaint time = 180 2024-03-06T07:26:26.140 INFO:teuthology.orchestra.run.smithi183.stdout: mds verify scatter = True 2024-03-06T07:26:26.140 INFO:teuthology.orchestra.run.smithi183.stdout: osd op complaint time = 180 2024-03-06T07:26:26.141 INFO:teuthology.orchestra.run.smithi183.stdout: rados mon op timeout = 900 2024-03-06T07:26:26.141 INFO:teuthology.orchestra.run.smithi183.stdout: rados osd op timeout = 900 2024-03-06T07:26:26.141 INFO:teuthology.orchestra.run.smithi183.stdout: mds_max_purge_files = 6 2024-03-06T07:26:26.141 INFO:teuthology.orchestra.run.smithi183.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:26:26.142 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:26:26.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.285+0000 7f84d518a640 1 Processor -- start 2024-03-06T07:26:26.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.285+0000 7f84d518a640 1 -- start start 2024-03-06T07:26:26.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.286+0000 7f84d518a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f84d015c070 0x7f84d0072100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.286+0000 7f84d518a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f84d006e4c0 0x7f84d0072640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.286+0000 7f84d518a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84d006ea80 con 0x7f84d015bc90 2024-03-06T07:26:26.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.286+0000 7f84d518a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84d006d920 con 0x7f84d006e4c0 2024-03-06T07:26:26.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.286+0000 7f84d518a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84d0072b80 con 0x7f84d015c070 2024-03-06T07:26:26.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.286+0000 7f84ced9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f84d015bc90 0x7f84d006dd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:35446/0 (socket says 172.21.15.83:35446) 2024-03-06T07:26:26.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.286+0000 7f84cf59e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f84d006e4c0 0x7f84d0072640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.286+0000 7f84ced9d640 1 -- 172.21.15.83:0/360856102 learned_addr learned my addr 172.21.15.83:0/360856102 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:26.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.287+0000 7f84ce59c640 1 --2- 172.21.15.83:0/360856102 >> v2:172.21.15.183:3300/0 conn(0x7f84d015c070 0x7f84d0072100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.287+0000 7f84cdd9b640 1 -- 172.21.15.83:0/360856102 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1334502966 0 0) 0x7f84d006ea80 con 0x7f84d015bc90 2024-03-06T07:26:26.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.287+0000 7f84cdd9b640 1 -- 172.21.15.83:0/360856102 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84b8003520 con 0x7f84d015bc90 2024-03-06T07:26:26.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.287+0000 7f84cf59e640 1 -- 172.21.15.83:0/360856102 >> v2:172.21.15.183:3300/0 conn(0x7f84d015c070 msgr2=0x7f84d0072100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:26.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.287+0000 7f84cf59e640 1 --2- 172.21.15.83:0/360856102 >> v2:172.21.15.183:3300/0 conn(0x7f84d015c070 0x7f84d0072100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.287+0000 7f84cf59e640 1 -- 172.21.15.83:0/360856102 >> v1:172.21.15.83:6789/0 conn(0x7f84d015bc90 legacy=0x7f84d006dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.287+0000 7f84cf59e640 1 -- 172.21.15.83:0/360856102 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d015fa40 con 0x7f84d006e4c0 2024-03-06T07:26:26.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.287+0000 7f84cf59e640 1 --2- 172.21.15.83:0/360856102 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84d006e4c0 0x7f84d0072640 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f84d006edc0 tx=0x7f84c002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d86d31987017eff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.288+0000 7f84cdd9b640 1 -- 172.21.15.83:0/360856102 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c00315a0 con 0x7f84d006e4c0 2024-03-06T07:26:26.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.288+0000 7f84cdd9b640 1 -- 172.21.15.83:0/360856102 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84c0037070 con 0x7f84d006e4c0 2024-03-06T07:26:26.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.288+0000 7f84cdd9b640 1 -- 172.21.15.83:0/360856102 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c0031960 con 0x7f84d006e4c0 2024-03-06T07:26:26.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.288+0000 7f84d518a640 1 -- 172.21.15.83:0/360856102 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84d006e4c0 msgr2=0x7f84d0072640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.288+0000 7f84d518a640 1 --2- 172.21.15.83:0/360856102 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84d006e4c0 0x7f84d0072640 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f84d006edc0 tx=0x7f84c002b840 comp rx=0 tx=0).stop 2024-03-06T07:26:26.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.288+0000 7f84d518a640 1 -- 172.21.15.83:0/360856102 shutdown_connections 2024-03-06T07:26:26.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.288+0000 7f84d518a640 1 --2- 172.21.15.83:0/360856102 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84d006e4c0 0x7f84d0072640 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.288+0000 7f84d518a640 1 --2- 172.21.15.83:0/360856102 >> v2:172.21.15.183:3300/0 conn(0x7f84d015c070 0x7f84d0072100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.288+0000 7f84d518a640 1 -- 172.21.15.83:0/360856102 >> 172.21.15.83:0/360856102 conn(0x7f84d00b1500 msgr2=0x7f84d00b18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:26.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.288+0000 7f84d518a640 1 -- 172.21.15.83:0/360856102 shutdown_connections 2024-03-06T07:26:26.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.288+0000 7f84d518a640 1 -- 172.21.15.83:0/360856102 wait complete. 2024-03-06T07:26:26.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.290+0000 7f84d518a640 1 Processor -- start 2024-03-06T07:26:26.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.290+0000 7f84d518a640 1 -- start start 2024-03-06T07:26:26.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.291+0000 7f84d518a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84d006e4c0 0x7f84d014de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.291+0000 7f84d518a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84d015bc90 0x7f84d015e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.291+0000 7f84d518a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f84d015c070 0x7f84d015ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.291+0000 7f84d518a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d01602e0 con 0x7f84d006e4c0 2024-03-06T07:26:26.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.291+0000 7f84d518a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d0160000 con 0x7f84d015bc90 2024-03-06T07:26:26.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.291+0000 7f84d518a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84d0160800 con 0x7f84d015c070 2024-03-06T07:26:26.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.291+0000 7f84ced9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84d006e4c0 0x7f84d014de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.291+0000 7f84ced9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84d006e4c0 0x7f84d014de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50620/0 (socket says 172.21.15.83:50620) 2024-03-06T07:26:26.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.291+0000 7f84ced9d640 1 -- 172.21.15.83:0/992456245 learned_addr learned my addr 172.21.15.83:0/992456245 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:26.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.291+0000 7f84ce59c640 1 --2- 172.21.15.83:0/992456245 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84d015bc90 0x7f84d015e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.291+0000 7f84cf59e640 1 --2- 172.21.15.83:0/992456245 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f84d015c070 0x7f84d015ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.291+0000 7f84ced9d640 1 -- 172.21.15.83:0/992456245 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f84d015c070 msgr2=0x7f84d015ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.291+0000 7f84ced9d640 1 --2- 172.21.15.83:0/992456245 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f84d015c070 0x7f84d015ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.291+0000 7f84ced9d640 1 -- 172.21.15.83:0/992456245 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84d015bc90 msgr2=0x7f84d015e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.291+0000 7f84ced9d640 1 --2- 172.21.15.83:0/992456245 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84d015bc90 0x7f84d015e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.291+0000 7f84ced9d640 1 -- 172.21.15.83:0/992456245 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d014e390 con 0x7f84d006e4c0 2024-03-06T07:26:26.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.291+0000 7f84ce59c640 1 --2- 172.21.15.83:0/992456245 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84d015bc90 0x7f84d015e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:26.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.291+0000 7f84ced9d640 1 --2- 172.21.15.83:0/992456245 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84d006e4c0 0x7f84d014de50 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f84c800caa0 tx=0x7f84c800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.291+0000 7f84b77fe640 1 -- 172.21.15.83:0/992456245 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c8017070 con 0x7f84d006e4c0 2024-03-06T07:26:26.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.292+0000 7f84b77fe640 1 -- 172.21.15.83:0/992456245 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84c800ac00 con 0x7f84d006e4c0 2024-03-06T07:26:26.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.292+0000 7f84d518a640 1 -- 172.21.15.83:0/992456245 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84d00c8a80 con 0x7f84d006e4c0 2024-03-06T07:26:26.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.292+0000 7f84b77fe640 1 -- 172.21.15.83:0/992456245 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c8012400 con 0x7f84d006e4c0 2024-03-06T07:26:26.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.292+0000 7f84d518a640 1 -- 172.21.15.83:0/992456245 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84d00c9040 con 0x7f84d006e4c0 2024-03-06T07:26:26.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.293+0000 7f84d518a640 1 -- 172.21.15.83:0/992456245 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84d00b1320 con 0x7f84d006e4c0 2024-03-06T07:26:26.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.293+0000 7f84b77fe640 1 -- 172.21.15.83:0/992456245 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f84c801a070 con 0x7f84d006e4c0 2024-03-06T07:26:26.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.294+0000 7f84b77fe640 1 --2- 172.21.15.83:0/992456245 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f84bc06b4d0 0x7f84bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.294+0000 7f84ce59c640 1 --2- 172.21.15.83:0/992456245 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f84bc06b4d0 0x7f84bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.294+0000 7f84b77fe640 1 -- 172.21.15.83:0/992456245 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f84c808a000 con 0x7f84d006e4c0 2024-03-06T07:26:26.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.294+0000 7f84ce59c640 1 --2- 172.21.15.83:0/992456245 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f84bc06b4d0 0x7f84bc06d990 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f84d00c9e50 tx=0x7f84c4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.298+0000 7f84b77fe640 1 -- 172.21.15.83:0/992456245 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f84d00b1320 con 0x7f84d006e4c0 2024-03-06T07:26:26.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.500+0000 7f84d518a640 1 -- 172.21.15.83:0/992456245 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f84d0068fb0 con 0x7f84d006e4c0 2024-03-06T07:26:26.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.500+0000 7f84b77fe640 1 -- 172.21.15.83:0/992456245 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) v1 ==== 100+0+11 (secure 0 0 0) 0x7f84c8058e80 con 0x7f84d006e4c0 2024-03-06T07:26:26.502 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 64 2024-03-06T07:26:26.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.503+0000 7f84d518a640 1 -- 172.21.15.83:0/992456245 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f84bc06b4d0 msgr2=0x7f84bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.503+0000 7f84d518a640 1 --2- 172.21.15.83:0/992456245 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f84bc06b4d0 0x7f84bc06d990 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f84d00c9e50 tx=0x7f84c4008040 comp rx=0 tx=0).stop 2024-03-06T07:26:26.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.503+0000 7f84d518a640 1 -- 172.21.15.83:0/992456245 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84d006e4c0 msgr2=0x7f84d014de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.503+0000 7f84d518a640 1 --2- 172.21.15.83:0/992456245 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84d006e4c0 0x7f84d014de50 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f84c800caa0 tx=0x7f84c800cf60 comp rx=0 tx=0).stop 2024-03-06T07:26:26.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.503+0000 7f84d518a640 1 -- 172.21.15.83:0/992456245 shutdown_connections 2024-03-06T07:26:26.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.503+0000 7f84d518a640 1 --2- 172.21.15.83:0/992456245 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f84bc06b4d0 0x7f84bc06d990 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.503+0000 7f84d518a640 1 --2- 172.21.15.83:0/992456245 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f84d015c070 0x7f84d015ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.503+0000 7f84d518a640 1 --2- 172.21.15.83:0/992456245 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84d015bc90 0x7f84d015e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.503+0000 7f84d518a640 1 --2- 172.21.15.83:0/992456245 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84d006e4c0 0x7f84d014de50 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.503+0000 7f84d518a640 1 -- 172.21.15.83:0/992456245 >> 172.21.15.83:0/992456245 conn(0x7f84d00b1500 msgr2=0x7f84d00720c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:26.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.503+0000 7f84d518a640 1 -- 172.21.15.83:0/992456245 shutdown_connections 2024-03-06T07:26:26.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.503+0000 7f84d518a640 1 -- 172.21.15.83:0/992456245 wait complete. 2024-03-06T07:26:26.516 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:26.518 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:26.566 INFO:teuthology.orchestra.run.smithi083.stdout:[global] 2024-03-06T07:26:26.566 INFO:teuthology.orchestra.run.smithi083.stdout: chdir = "" 2024-03-06T07:26:26.566 INFO:teuthology.orchestra.run.smithi083.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:26.567 INFO:teuthology.orchestra.run.smithi083.stdout: auth supported = cephx 2024-03-06T07:26:26.567 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.567 INFO:teuthology.orchestra.run.smithi083.stdout: filestore xattr use omap = true 2024-03-06T07:26:26.567 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.567 INFO:teuthology.orchestra.run.smithi083.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:26.567 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.567 INFO:teuthology.orchestra.run.smithi083.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:26.567 INFO:teuthology.orchestra.run.smithi083.stdout: auth debug = true 2024-03-06T07:26:26.567 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.567 INFO:teuthology.orchestra.run.smithi083.stdout: ms die on old message = true 2024-03-06T07:26:26.567 INFO:teuthology.orchestra.run.smithi083.stdout: ms die on bug = true 2024-03-06T07:26:26.567 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.567 INFO:teuthology.orchestra.run.smithi083.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:26.567 INFO:teuthology.orchestra.run.smithi083.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:26.568 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.568 INFO:teuthology.orchestra.run.smithi083.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:26.568 INFO:teuthology.orchestra.run.smithi083.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:26.568 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.568 INFO:teuthology.orchestra.run.smithi083.stdout: osd pool default size = 2 2024-03-06T07:26:26.568 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.568 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:26.568 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd allow pg remap = true 2024-03-06T07:26:26.568 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:26.568 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:26.568 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:26.568 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:26.568 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on too few osds = false 2024-03-06T07:26:26.568 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:26.569 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:26.569 INFO:teuthology.orchestra.run.smithi083.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:26.569 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.569 INFO:teuthology.orchestra.run.smithi083.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:26.569 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.569 INFO:teuthology.orchestra.run.smithi083.stdout: osd default data pool replay window = 5 2024-03-06T07:26:26.569 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.569 INFO:teuthology.orchestra.run.smithi083.stdout: mon allow pool delete = true 2024-03-06T07:26:26.569 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.569 INFO:teuthology.orchestra.run.smithi083.stdout: mon cluster log file level = debug 2024-03-06T07:26:26.569 INFO:teuthology.orchestra.run.smithi083.stdout: debug asserts on shutdown = true 2024-03-06T07:26:26.569 INFO:teuthology.orchestra.run.smithi083.stdout: mon health detail to clog = false 2024-03-06T07:26:26.569 INFO:teuthology.orchestra.run.smithi083.stdout: mon host = "172.21.15.83,172.21.15.183,[v2:172.21.15.183:3301,v1:172.21.15.183:6790]" 2024-03-06T07:26:26.569 INFO:teuthology.orchestra.run.smithi083.stdout: fsid = 039877dd-d2ce-45d2-8e77-ac5eb7e1d55a 2024-03-06T07:26:26.569 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.570 INFO:teuthology.orchestra.run.smithi083.stdout:[osd] 2024-03-06T07:26:26.570 INFO:teuthology.orchestra.run.smithi083.stdout: osd journal size = 100 2024-03-06T07:26:26.570 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.570 INFO:teuthology.orchestra.run.smithi083.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:26.570 INFO:teuthology.orchestra.run.smithi083.stdout: osd scrub max interval = 600 2024-03-06T07:26:26.570 INFO:teuthology.orchestra.run.smithi083.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:26.570 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.570 INFO:teuthology.orchestra.run.smithi083.stdout: osd recover clone overlap = true 2024-03-06T07:26:26.570 INFO:teuthology.orchestra.run.smithi083.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:26.570 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.570 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug shutdown = true 2024-03-06T07:26:26.570 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug op order = true 2024-03-06T07:26:26.570 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:26.570 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug trim objects = true 2024-03-06T07:26:26.571 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.571 INFO:teuthology.orchestra.run.smithi083.stdout: osd open classes on start = true 2024-03-06T07:26:26.571 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug pg log writeout = true 2024-03-06T07:26:26.571 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.571 INFO:teuthology.orchestra.run.smithi083.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:26.571 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.571 INFO:teuthology.orchestra.run.smithi083.stdout: osd map max advance = 10 2024-03-06T07:26:26.571 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.571 INFO:teuthology.orchestra.run.smithi083.stdout: journal zero on create = true 2024-03-06T07:26:26.571 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.571 INFO:teuthology.orchestra.run.smithi083.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:26.571 INFO:teuthology.orchestra.run.smithi083.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:26.571 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.571 INFO:teuthology.orchestra.run.smithi083.stdout: bdev debug aio = true 2024-03-06T07:26:26.572 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug misdirected ops = true 2024-03-06T07:26:26.572 INFO:teuthology.orchestra.run.smithi083.stdout: bdev async discard = True 2024-03-06T07:26:26.572 INFO:teuthology.orchestra.run.smithi083.stdout: bdev enable discard = True 2024-03-06T07:26:26.572 INFO:teuthology.orchestra.run.smithi083.stdout: bluestore allocator = bitmap 2024-03-06T07:26:26.572 INFO:teuthology.orchestra.run.smithi083.stdout: bluestore block size = 96636764160 2024-03-06T07:26:26.572 INFO:teuthology.orchestra.run.smithi083.stdout: bluestore fsck on mount = True 2024-03-06T07:26:26.572 INFO:teuthology.orchestra.run.smithi083.stdout: debug bluefs = 1/20 2024-03-06T07:26:26.572 INFO:teuthology.orchestra.run.smithi083.stdout: debug bluestore = 1/20 2024-03-06T07:26:26.572 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-06T07:26:26.572 INFO:teuthology.orchestra.run.smithi083.stdout: debug osd = 20 2024-03-06T07:26:26.572 INFO:teuthology.orchestra.run.smithi083.stdout: debug rocksdb = 4/10 2024-03-06T07:26:26.572 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:26.572 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:26.572 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:26.572 INFO:teuthology.orchestra.run.smithi083.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:26.573 INFO:teuthology.orchestra.run.smithi083.stdout: osd objectstore = bluestore 2024-03-06T07:26:26.573 INFO:teuthology.orchestra.run.smithi083.stdout: osd op complaint time = 180 2024-03-06T07:26:26.573 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.573 INFO:teuthology.orchestra.run.smithi083.stdout:[mgr] 2024-03-06T07:26:26.573 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-06T07:26:26.573 INFO:teuthology.orchestra.run.smithi083.stdout: debug mgr = 20 2024-03-06T07:26:26.573 INFO:teuthology.orchestra.run.smithi083.stdout: debug mon = 20 2024-03-06T07:26:26.573 INFO:teuthology.orchestra.run.smithi083.stdout: debug auth = 20 2024-03-06T07:26:26.573 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:26.573 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:26.573 INFO:teuthology.orchestra.run.smithi083.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:26.573 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.573 INFO:teuthology.orchestra.run.smithi083.stdout:[mon] 2024-03-06T07:26:26.573 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-06T07:26:26.573 INFO:teuthology.orchestra.run.smithi083.stdout: debug mon = 20 2024-03-06T07:26:26.573 INFO:teuthology.orchestra.run.smithi083.stdout: debug paxos = 20 2024-03-06T07:26:26.574 INFO:teuthology.orchestra.run.smithi083.stdout: debug auth = 20 2024-03-06T07:26:26.574 INFO:teuthology.orchestra.run.smithi083.stdout: mon data avail warn = 5 2024-03-06T07:26:26.574 INFO:teuthology.orchestra.run.smithi083.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:26.574 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:26.574 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:26.574 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd prime pg temp = true 2024-03-06T07:26:26.574 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:26.574 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.574 INFO:teuthology.orchestra.run.smithi083.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:26.574 INFO:teuthology.orchestra.run.smithi083.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:26.574 INFO:teuthology.orchestra.run.smithi083.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:26.574 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.574 INFO:teuthology.orchestra.run.smithi083.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:26.574 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:26.574 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:26.574 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.574 INFO:teuthology.orchestra.run.smithi083.stdout: # 1m isn't quite enough 2024-03-06T07:26:26.575 INFO:teuthology.orchestra.run.smithi083.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:26.575 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.575 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:26.575 INFO:teuthology.orchestra.run.smithi083.stdout: mon op complaint time = 120 2024-03-06T07:26:26.575 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:26.575 INFO:teuthology.orchestra.run.smithi083.stdout:[client] 2024-03-06T07:26:26.575 INFO:teuthology.orchestra.run.smithi083.stdout: rgw cache enabled = true 2024-03-06T07:26:26.575 INFO:teuthology.orchestra.run.smithi083.stdout: rgw enable ops log = true 2024-03-06T07:26:26.575 INFO:teuthology.orchestra.run.smithi083.stdout: rgw enable usage log = true 2024-03-06T07:26:26.575 INFO:teuthology.orchestra.run.smithi083.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:26.575 INFO:teuthology.orchestra.run.smithi083.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:26.575 INFO:teuthology.orchestra.run.smithi083.stdout: client mount timeout = 600 2024-03-06T07:26:26.575 INFO:teuthology.orchestra.run.smithi083.stdout: debug client = 20 2024-03-06T07:26:26.575 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-06T07:26:26.575 INFO:teuthology.orchestra.run.smithi083.stdout: pid file = "" 2024-03-06T07:26:26.575 INFO:teuthology.orchestra.run.smithi083.stdout: rados mon op timeout = 900 2024-03-06T07:26:26.576 INFO:teuthology.orchestra.run.smithi083.stdout: rados osd op timeout = 900 2024-03-06T07:26:26.576 INFO:teuthology.orchestra.run.smithi083.stdout:[mon.a] 2024-03-06T07:26:26.576 INFO:teuthology.orchestra.run.smithi083.stdout:[mon.b] 2024-03-06T07:26:26.576 INFO:teuthology.orchestra.run.smithi083.stdout:[mon.c] 2024-03-06T07:26:26.576 INFO:teuthology.orchestra.run.smithi083.stdout:[mds.a] 2024-03-06T07:26:26.576 INFO:teuthology.orchestra.run.smithi083.stdout:[mds.c] 2024-03-06T07:26:26.576 INFO:teuthology.orchestra.run.smithi083.stdout:[mds.b] 2024-03-06T07:26:26.576 INFO:teuthology.orchestra.run.smithi083.stdout:[mds.d] 2024-03-06T07:26:26.576 INFO:teuthology.orchestra.run.smithi083.stdout:[mds] 2024-03-06T07:26:26.576 INFO:teuthology.orchestra.run.smithi083.stdout: debug mds = 20 2024-03-06T07:26:26.576 INFO:teuthology.orchestra.run.smithi083.stdout: debug mds balancer = 20 2024-03-06T07:26:26.576 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-06T07:26:26.576 INFO:teuthology.orchestra.run.smithi083.stdout: mds debug frag = True 2024-03-06T07:26:26.576 INFO:teuthology.orchestra.run.smithi083.stdout: mds debug scatterstat = True 2024-03-06T07:26:26.577 INFO:teuthology.orchestra.run.smithi083.stdout: mds op complaint time = 180 2024-03-06T07:26:26.577 INFO:teuthology.orchestra.run.smithi083.stdout: mds verify scatter = True 2024-03-06T07:26:26.577 INFO:teuthology.orchestra.run.smithi083.stdout: osd op complaint time = 180 2024-03-06T07:26:26.577 INFO:teuthology.orchestra.run.smithi083.stdout: rados mon op timeout = 900 2024-03-06T07:26:26.577 INFO:teuthology.orchestra.run.smithi083.stdout: rados osd op timeout = 900 2024-03-06T07:26:26.577 INFO:teuthology.orchestra.run.smithi083.stdout: mds_max_purge_files = 6 2024-03-06T07:26:26.577 INFO:teuthology.orchestra.run.smithi083.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:26:26.577 INFO:teuthology.orchestra.run.smithi083.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:26:26.577 INFO:teuthology.orchestra.run.smithi183.stdout:[global] 2024-03-06T07:26:26.578 INFO:teuthology.orchestra.run.smithi183.stdout: chdir = "" 2024-03-06T07:26:26.578 INFO:teuthology.orchestra.run.smithi183.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:26.578 INFO:teuthology.orchestra.run.smithi183.stdout: auth supported = cephx 2024-03-06T07:26:26.578 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.578 INFO:teuthology.orchestra.run.smithi183.stdout: filestore xattr use omap = true 2024-03-06T07:26:26.578 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.578 INFO:teuthology.orchestra.run.smithi183.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:26.578 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.578 INFO:teuthology.orchestra.run.smithi183.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:26.578 INFO:teuthology.orchestra.run.smithi183.stdout: auth debug = true 2024-03-06T07:26:26.578 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.578 INFO:teuthology.orchestra.run.smithi183.stdout: ms die on old message = true 2024-03-06T07:26:26.578 INFO:teuthology.orchestra.run.smithi183.stdout: ms die on bug = true 2024-03-06T07:26:26.579 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.579 INFO:teuthology.orchestra.run.smithi183.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:26.579 INFO:teuthology.orchestra.run.smithi183.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:26.579 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.579 INFO:teuthology.orchestra.run.smithi183.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:26.579 INFO:teuthology.orchestra.run.smithi183.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:26.579 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.579 INFO:teuthology.orchestra.run.smithi183.stdout: osd pool default size = 2 2024-03-06T07:26:26.579 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.579 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:26.579 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd allow pg remap = true 2024-03-06T07:26:26.579 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:26.579 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:26.580 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:26.580 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:26.580 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on too few osds = false 2024-03-06T07:26:26.580 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:26.580 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:26.580 INFO:teuthology.orchestra.run.smithi183.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:26.580 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.580 INFO:teuthology.orchestra.run.smithi183.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:26.580 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.580 INFO:teuthology.orchestra.run.smithi183.stdout: osd default data pool replay window = 5 2024-03-06T07:26:26.580 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.580 INFO:teuthology.orchestra.run.smithi183.stdout: mon allow pool delete = true 2024-03-06T07:26:26.580 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.580 INFO:teuthology.orchestra.run.smithi183.stdout: mon cluster log file level = debug 2024-03-06T07:26:26.581 INFO:teuthology.orchestra.run.smithi183.stdout: debug asserts on shutdown = true 2024-03-06T07:26:26.581 INFO:teuthology.orchestra.run.smithi183.stdout: mon health detail to clog = false 2024-03-06T07:26:26.581 INFO:teuthology.orchestra.run.smithi183.stdout: mon host = "172.21.15.83,172.21.15.183,[v2:172.21.15.183:3301,v1:172.21.15.183:6790]" 2024-03-06T07:26:26.581 INFO:teuthology.orchestra.run.smithi183.stdout: fsid = 039877dd-d2ce-45d2-8e77-ac5eb7e1d55a 2024-03-06T07:26:26.581 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.581 INFO:teuthology.orchestra.run.smithi183.stdout:[osd] 2024-03-06T07:26:26.581 INFO:teuthology.orchestra.run.smithi183.stdout: osd journal size = 100 2024-03-06T07:26:26.581 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.581 INFO:teuthology.orchestra.run.smithi183.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:26.581 INFO:teuthology.orchestra.run.smithi183.stdout: osd scrub max interval = 600 2024-03-06T07:26:26.581 INFO:teuthology.orchestra.run.smithi183.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:26.581 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.581 INFO:teuthology.orchestra.run.smithi183.stdout: osd recover clone overlap = true 2024-03-06T07:26:26.581 INFO:teuthology.orchestra.run.smithi183.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:26.581 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.582 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug shutdown = true 2024-03-06T07:26:26.582 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug op order = true 2024-03-06T07:26:26.582 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:26.582 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug trim objects = true 2024-03-06T07:26:26.582 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.582 INFO:teuthology.orchestra.run.smithi183.stdout: osd open classes on start = true 2024-03-06T07:26:26.582 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug pg log writeout = true 2024-03-06T07:26:26.582 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.582 INFO:teuthology.orchestra.run.smithi183.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:26.582 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.582 INFO:teuthology.orchestra.run.smithi183.stdout: osd map max advance = 10 2024-03-06T07:26:26.582 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.582 INFO:teuthology.orchestra.run.smithi183.stdout: journal zero on create = true 2024-03-06T07:26:26.582 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.582 INFO:teuthology.orchestra.run.smithi183.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:26.582 INFO:teuthology.orchestra.run.smithi183.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:26.583 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.583 INFO:teuthology.orchestra.run.smithi183.stdout: bdev debug aio = true 2024-03-06T07:26:26.583 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug misdirected ops = true 2024-03-06T07:26:26.583 INFO:teuthology.orchestra.run.smithi183.stdout: bdev async discard = True 2024-03-06T07:26:26.583 INFO:teuthology.orchestra.run.smithi183.stdout: bdev enable discard = True 2024-03-06T07:26:26.583 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore allocator = bitmap 2024-03-06T07:26:26.583 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore block size = 96636764160 2024-03-06T07:26:26.583 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore fsck on mount = True 2024-03-06T07:26:26.583 INFO:teuthology.orchestra.run.smithi183.stdout: debug bluefs = 1/20 2024-03-06T07:26:26.583 INFO:teuthology.orchestra.run.smithi183.stdout: debug bluestore = 1/20 2024-03-06T07:26:26.583 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-06T07:26:26.583 INFO:teuthology.orchestra.run.smithi183.stdout: debug osd = 20 2024-03-06T07:26:26.583 INFO:teuthology.orchestra.run.smithi183.stdout: debug rocksdb = 4/10 2024-03-06T07:26:26.583 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:26.584 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:26.584 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:26.584 INFO:teuthology.orchestra.run.smithi183.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:26.584 INFO:teuthology.orchestra.run.smithi183.stdout: osd objectstore = bluestore 2024-03-06T07:26:26.584 INFO:teuthology.orchestra.run.smithi183.stdout: osd op complaint time = 180 2024-03-06T07:26:26.584 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.584 INFO:teuthology.orchestra.run.smithi183.stdout:[mgr] 2024-03-06T07:26:26.584 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-06T07:26:26.584 INFO:teuthology.orchestra.run.smithi183.stdout: debug mgr = 20 2024-03-06T07:26:26.584 INFO:teuthology.orchestra.run.smithi183.stdout: debug mon = 20 2024-03-06T07:26:26.584 INFO:teuthology.orchestra.run.smithi183.stdout: debug auth = 20 2024-03-06T07:26:26.584 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:26.584 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:26.584 INFO:teuthology.orchestra.run.smithi183.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:26.584 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.585 INFO:teuthology.orchestra.run.smithi183.stdout:[mon] 2024-03-06T07:26:26.585 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-06T07:26:26.585 INFO:teuthology.orchestra.run.smithi183.stdout: debug mon = 20 2024-03-06T07:26:26.585 INFO:teuthology.orchestra.run.smithi183.stdout: debug paxos = 20 2024-03-06T07:26:26.585 INFO:teuthology.orchestra.run.smithi183.stdout: debug auth = 20 2024-03-06T07:26:26.585 INFO:teuthology.orchestra.run.smithi183.stdout: mon data avail warn = 5 2024-03-06T07:26:26.585 INFO:teuthology.orchestra.run.smithi183.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:26.585 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:26.585 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:26.585 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd prime pg temp = true 2024-03-06T07:26:26.585 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:26.585 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.585 INFO:teuthology.orchestra.run.smithi183.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:26.585 INFO:teuthology.orchestra.run.smithi183.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:26.585 INFO:teuthology.orchestra.run.smithi183.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:26.585 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.586 INFO:teuthology.orchestra.run.smithi183.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:26.586 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:26.586 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:26.586 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.586 INFO:teuthology.orchestra.run.smithi183.stdout: # 1m isn't quite enough 2024-03-06T07:26:26.586 INFO:teuthology.orchestra.run.smithi183.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:26.586 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.586 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:26.586 INFO:teuthology.orchestra.run.smithi183.stdout: mon op complaint time = 120 2024-03-06T07:26:26.586 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:26:26.586 INFO:teuthology.orchestra.run.smithi183.stdout:[client] 2024-03-06T07:26:26.586 INFO:teuthology.orchestra.run.smithi183.stdout: rgw cache enabled = true 2024-03-06T07:26:26.586 INFO:teuthology.orchestra.run.smithi183.stdout: rgw enable ops log = true 2024-03-06T07:26:26.586 INFO:teuthology.orchestra.run.smithi183.stdout: rgw enable usage log = true 2024-03-06T07:26:26.586 INFO:teuthology.orchestra.run.smithi183.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:26.586 INFO:teuthology.orchestra.run.smithi183.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:26.587 INFO:teuthology.orchestra.run.smithi183.stdout: client mount timeout = 600 2024-03-06T07:26:26.587 INFO:teuthology.orchestra.run.smithi183.stdout: debug client = 20 2024-03-06T07:26:26.587 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-06T07:26:26.587 INFO:teuthology.orchestra.run.smithi183.stdout: pid file = "" 2024-03-06T07:26:26.587 INFO:teuthology.orchestra.run.smithi183.stdout: rados mon op timeout = 900 2024-03-06T07:26:26.587 INFO:teuthology.orchestra.run.smithi183.stdout: rados osd op timeout = 900 2024-03-06T07:26:26.587 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.a] 2024-03-06T07:26:26.587 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.b] 2024-03-06T07:26:26.587 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.c] 2024-03-06T07:26:26.587 INFO:teuthology.orchestra.run.smithi183.stdout:[mds.a] 2024-03-06T07:26:26.587 INFO:teuthology.orchestra.run.smithi183.stdout:[mds.c] 2024-03-06T07:26:26.587 INFO:teuthology.orchestra.run.smithi183.stdout:[mds.b] 2024-03-06T07:26:26.587 INFO:teuthology.orchestra.run.smithi183.stdout:[mds.d] 2024-03-06T07:26:26.587 INFO:teuthology.orchestra.run.smithi183.stdout:[mds] 2024-03-06T07:26:26.587 INFO:teuthology.orchestra.run.smithi183.stdout: debug mds = 20 2024-03-06T07:26:26.587 INFO:teuthology.orchestra.run.smithi183.stdout: debug mds balancer = 20 2024-03-06T07:26:26.587 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-06T07:26:26.588 INFO:teuthology.orchestra.run.smithi183.stdout: mds debug frag = True 2024-03-06T07:26:26.588 INFO:teuthology.orchestra.run.smithi183.stdout: mds debug scatterstat = True 2024-03-06T07:26:26.588 INFO:teuthology.orchestra.run.smithi183.stdout: mds op complaint time = 180 2024-03-06T07:26:26.588 INFO:teuthology.orchestra.run.smithi183.stdout: mds verify scatter = True 2024-03-06T07:26:26.588 INFO:teuthology.orchestra.run.smithi183.stdout: osd op complaint time = 180 2024-03-06T07:26:26.588 INFO:teuthology.orchestra.run.smithi183.stdout: rados mon op timeout = 900 2024-03-06T07:26:26.588 INFO:teuthology.orchestra.run.smithi183.stdout: rados osd op timeout = 900 2024-03-06T07:26:26.588 INFO:teuthology.orchestra.run.smithi183.stdout: mds_max_purge_files = 6 2024-03-06T07:26:26.588 INFO:teuthology.orchestra.run.smithi183.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:26:26.588 INFO:teuthology.orchestra.run.smithi183.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:26:26.590 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:26:26.590 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:26.590 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:26:26.590 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:26.590 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:26:26.591 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:26.591 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:26:26.591 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:26.639 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:26:26.640 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-03-06T07:26:26.640 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:26:26.641 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-03-06T07:26:26.642 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:26:26.642 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail c 2024-03-06T07:26:26.643 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:26:26.643 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2024-03-06T07:26:26.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.787+0000 7f5c94ce0640 1 Processor -- start 2024-03-06T07:26:26.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.787+0000 7f446759e640 1 Processor -- start 2024-03-06T07:26:26.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.787+0000 7f5c94ce0640 1 -- start start 2024-03-06T07:26:26.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.787+0000 7f446759e640 1 -- start start 2024-03-06T07:26:26.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.787+0000 7feff999a640 1 Processor -- start 2024-03-06T07:26:26.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.787+0000 7feff999a640 1 -- start start 2024-03-06T07:26:26.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.788+0000 7f5c94ce0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c900d0950 0x7f5c900d0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.788+0000 7f446759e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4468072810 0x7f446806c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.788+0000 7f446759e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4468072ca0 con 0x7f446815eb00 2024-03-06T07:26:26.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.788+0000 7f446759e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44680cd750 con 0x7f446806c090 2024-03-06T07:26:26.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.788+0000 7f446759e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f446806dc10 con 0x7f4468072810 2024-03-06T07:26:26.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.788+0000 7f5c94ce0640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c900d1ea0 con 0x7f5c900b9060 2024-03-06T07:26:26.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.788+0000 7f5c94ce0640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c900d2180 con 0x7f5c9015e150 2024-03-06T07:26:26.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.788+0000 7f5c94ce0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c900d12f0 con 0x7f5c900d0950 2024-03-06T07:26:26.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.788+0000 7f4465d9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4468072810 0x7f446806c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.788+0000 7f5c8ed9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f5c900b9060 0x7f5c900b94f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:36832/0 (socket says 172.21.15.83:36832) 2024-03-06T07:26:26.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.788+0000 7f5c8ed9d640 1 -- 172.21.15.83:0/3268068628 learned_addr learned my addr 172.21.15.83:0/3268068628 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:26.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.788+0000 7f4466d9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f446815eb00 0x7f446815ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:36846/0 (socket says 172.21.15.83:36846) 2024-03-06T07:26:26.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.788+0000 7f4466d9d640 1 -- 172.21.15.83:0/2152383227 learned_addr learned my addr 172.21.15.83:0/2152383227 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:26.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.788+0000 7feff999a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7feff415e150 0x7feff40b34e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.789+0000 7feff999a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feff40ad8b0 0x7feff40b3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.788+0000 7f5c8dd9b640 1 --2- 172.21.15.83:0/3268068628 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c900d0950 0x7f5c900d0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.789+0000 7feff999a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feff406e240 con 0x7feff406ef30 2024-03-06T07:26:26.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.789+0000 7feff999a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feff406b670 con 0x7feff415e150 2024-03-06T07:26:26.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.789+0000 7feff999a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feff406e520 con 0x7feff40ad8b0 2024-03-06T07:26:26.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.789+0000 7feff2ffd640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7feff415e150 0x7feff40b34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.789+0000 7feff2ffd640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7feff415e150 0x7feff40b34e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:40516/0 (socket says 172.21.15.83:40516) 2024-03-06T07:26:26.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.789+0000 7f4465d9b640 1 -- 172.21.15.83:0/2152383227 >> v1:172.21.15.183:6789/0 conn(0x7f446806c090 legacy=0x7f446815bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.789+0000 7f4465d9b640 1 -- 172.21.15.83:0/2152383227 >> v1:172.21.15.83:6789/0 conn(0x7f446815eb00 legacy=0x7f446815ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.789+0000 7f4465d9b640 1 -- 172.21.15.83:0/2152383227 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44680ce670 con 0x7f4468072810 2024-03-06T07:26:26.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.789+0000 7feff2ffd640 1 -- 172.21.15.83:0/276486399 learned_addr learned my addr 172.21.15.83:0/276486399 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:26.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.789+0000 7feff27fc640 1 --2- 172.21.15.83:0/276486399 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feff40ad8b0 0x7feff40b3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.789+0000 7f5c8dd9b640 1 -- 172.21.15.83:0/3268068628 >> v1:172.21.15.183:6789/0 conn(0x7f5c9015e150 legacy=0x7f5c900cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.789+0000 7f5c8dd9b640 1 -- 172.21.15.83:0/3268068628 >> v1:172.21.15.83:6789/0 conn(0x7f5c900b9060 legacy=0x7f5c900b94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.789+0000 7f5c8dd9b640 1 -- 172.21.15.83:0/3268068628 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c900d2460 con 0x7f5c900d0950 2024-03-06T07:26:26.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.789+0000 7feff2ffd640 1 -- 172.21.15.83:0/276486399 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feff40ad8b0 msgr2=0x7feff40b3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:26.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.789+0000 7feff2ffd640 1 --2- 172.21.15.83:0/276486399 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feff40ad8b0 0x7feff40b3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.789+0000 7feff2ffd640 1 -- 172.21.15.83:0/276486399 >> v1:172.21.15.183:6789/0 conn(0x7feff406ef30 legacy=0x7feff4161be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.789+0000 7feff2ffd640 1 -- 172.21.15.83:0/276486399 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff406b020 con 0x7feff415e150 2024-03-06T07:26:26.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.790+0000 7feff2ffd640 1 --2- 172.21.15.83:0/276486399 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feff415e150 0x7feff40b34e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fefe4009870 tx=0x7fefe402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31e0b54d67bbc75d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.790+0000 7f4465d9b640 1 --2- 172.21.15.83:0/2152383227 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4468072810 0x7f446806c470 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f445c0097a0 tx=0x7f445c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ef2068d8b525982 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.790+0000 7feff1ffb640 1 -- 172.21.15.83:0/276486399 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe4034070 con 0x7feff415e150 2024-03-06T07:26:26.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.790+0000 7feff1ffb640 1 -- 172.21.15.83:0/276486399 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefe4030840 con 0x7feff415e150 2024-03-06T07:26:26.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.790+0000 7feff1ffb640 1 -- 172.21.15.83:0/276486399 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe4030c10 con 0x7feff415e150 2024-03-06T07:26:26.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.790+0000 7f5c8dd9b640 1 --2- 172.21.15.83:0/3268068628 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c900d0950 0x7f5c900d0db0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5c84009750 tx=0x7f5c8402b680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac601b9701e8fca7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.790+0000 7f446559a640 1 -- 172.21.15.83:0/2152383227 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f445c035020 con 0x7f4468072810 2024-03-06T07:26:26.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.790+0000 7f5c8d59a640 1 -- 172.21.15.83:0/3268068628 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c84037070 con 0x7f5c900d0950 2024-03-06T07:26:26.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.790+0000 7f5c8d59a640 1 -- 172.21.15.83:0/3268068628 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c84030710 con 0x7f5c900d0950 2024-03-06T07:26:26.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.790+0000 7f5c8d59a640 1 -- 172.21.15.83:0/3268068628 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c84030ae0 con 0x7f5c900d0950 2024-03-06T07:26:26.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.791+0000 7feff999a640 1 -- 172.21.15.83:0/276486399 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feff415e150 msgr2=0x7feff40b34e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.791+0000 7feff999a640 1 --2- 172.21.15.83:0/276486399 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feff415e150 0x7feff40b34e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fefe4009870 tx=0x7fefe402b980 comp rx=0 tx=0).stop 2024-03-06T07:26:26.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.790+0000 7f446559a640 1 -- 172.21.15.83:0/2152383227 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f445c031830 con 0x7f4468072810 2024-03-06T07:26:26.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.790+0000 7f446559a640 1 -- 172.21.15.83:0/2152383227 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f445c031c00 con 0x7f4468072810 2024-03-06T07:26:26.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.791+0000 7f446759e640 1 -- 172.21.15.83:0/2152383227 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4468072810 msgr2=0x7f446806c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.791+0000 7f446759e640 1 --2- 172.21.15.83:0/2152383227 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4468072810 0x7f446806c470 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f445c0097a0 tx=0x7f445c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.791+0000 7feff999a640 1 -- 172.21.15.83:0/276486399 shutdown_connections 2024-03-06T07:26:26.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.791+0000 7feff999a640 1 --2- 172.21.15.83:0/276486399 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feff40ad8b0 0x7feff40b3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.791+0000 7feff999a640 1 --2- 172.21.15.83:0/276486399 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feff415e150 0x7feff40b34e0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.791+0000 7feff999a640 1 -- 172.21.15.83:0/276486399 >> 172.21.15.83:0/276486399 conn(0x7feff40d0470 msgr2=0x7feff40d0880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:26.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.791+0000 7f5c94ce0640 1 -- 172.21.15.83:0/3268068628 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c900d0950 msgr2=0x7f5c900d0db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.791+0000 7f5c94ce0640 1 --2- 172.21.15.83:0/3268068628 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c900d0950 0x7f5c900d0db0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5c84009750 tx=0x7f5c8402b680 comp rx=0 tx=0).stop 2024-03-06T07:26:26.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.791+0000 7feff999a640 1 -- 172.21.15.83:0/276486399 shutdown_connections 2024-03-06T07:26:26.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.791+0000 7f446759e640 1 -- 172.21.15.83:0/2152383227 shutdown_connections 2024-03-06T07:26:26.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.791+0000 7f446759e640 1 --2- 172.21.15.83:0/2152383227 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4468072810 0x7f446806c470 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.791+0000 7feff999a640 1 -- 172.21.15.83:0/276486399 wait complete. 2024-03-06T07:26:26.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.791+0000 7f5c94ce0640 1 -- 172.21.15.83:0/3268068628 shutdown_connections 2024-03-06T07:26:26.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.791+0000 7f446759e640 1 -- 172.21.15.83:0/2152383227 >> 172.21.15.83:0/2152383227 conn(0x7f44680b2930 msgr2=0x7f44680b2d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:26.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.791+0000 7f5c94ce0640 1 --2- 172.21.15.83:0/3268068628 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c900d0950 0x7f5c900d0db0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.791+0000 7f5c94ce0640 1 -- 172.21.15.83:0/3268068628 >> 172.21.15.83:0/3268068628 conn(0x7f5c9009d230 msgr2=0x7f5c900d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:26.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.791+0000 7f446759e640 1 -- 172.21.15.83:0/2152383227 shutdown_connections 2024-03-06T07:26:26.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.791+0000 7f446759e640 1 -- 172.21.15.83:0/2152383227 wait complete. 2024-03-06T07:26:26.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.791+0000 7f5c94ce0640 1 -- 172.21.15.83:0/3268068628 shutdown_connections 2024-03-06T07:26:26.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.791+0000 7f5c94ce0640 1 -- 172.21.15.83:0/3268068628 wait complete. 2024-03-06T07:26:26.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.792+0000 7feff999a640 1 Processor -- start 2024-03-06T07:26:26.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.793+0000 7f446759e640 1 Processor -- start 2024-03-06T07:26:26.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.793+0000 7f5c94ce0640 1 Processor -- start 2024-03-06T07:26:26.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.793+0000 7feff999a640 1 -- start start 2024-03-06T07:26:26.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.793+0000 7f5c94ce0640 1 -- start start 2024-03-06T07:26:26.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.793+0000 7f446759e640 1 -- start start 2024-03-06T07:26:26.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.793+0000 7feff999a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feff406ef30 0x7feff4157fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.793+0000 7f446759e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f446806c090 0x7f44681511c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.793+0000 7f446759e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4468072810 0x7f44680af980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.793+0000 7feff999a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feff40ad8b0 0x7feff415bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.793+0000 7f5c94ce0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c900b9060 0x7f5c901f5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.793+0000 7f5c94ce0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c900d0950 0x7f5c900b4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.793+0000 7f5c94ce0640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5c9015e150 0x7f5c90151340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.793+0000 7f5c94ce0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c900d2f40 con 0x7f5c900d0950 2024-03-06T07:26:26.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.793+0000 7f5c94ce0640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c900d2ab0 con 0x7f5c9015e150 2024-03-06T07:26:26.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.793+0000 7f5c94ce0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c900b7dc0 con 0x7f5c900b9060 2024-03-06T07:26:26.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.793+0000 7feff999a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feff415e150 0x7feff40ae610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.794+0000 7feff999a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff4066f10 con 0x7feff415e150 2024-03-06T07:26:26.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.793+0000 7f446759e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f446815eb00 0x7f44680b1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.793+0000 7f446759e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f446806e140 con 0x7f4468072810 2024-03-06T07:26:26.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.793+0000 7f446759e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f446806de60 con 0x7f446806c090 2024-03-06T07:26:26.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.793+0000 7f446759e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44680d05c0 con 0x7f446815eb00 2024-03-06T07:26:26.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.794+0000 7f4465d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4468072810 0x7f44680af980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.794+0000 7f4465d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4468072810 0x7f44680af980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47890/0 (socket says 172.21.15.83:47890) 2024-03-06T07:26:26.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.794+0000 7f4465d9b640 1 -- 172.21.15.83:0/886613971 learned_addr learned my addr 172.21.15.83:0/886613971 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:26.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.794+0000 7f4466d9d640 1 --2- 172.21.15.83:0/886613971 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f446815eb00 0x7f44680b1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.794+0000 7feff999a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff406df10 con 0x7feff40ad8b0 2024-03-06T07:26:26.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.794+0000 7feff999a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feff4067090 con 0x7feff406ef30 2024-03-06T07:26:26.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.794+0000 7f446659c640 1 --2- 172.21.15.83:0/886613971 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f446806c090 0x7f44681511c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.794+0000 7feff37fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feff415e150 0x7feff40ae610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.794+0000 7feff37fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feff415e150 0x7feff40ae610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47904/0 (socket says 172.21.15.83:47904) 2024-03-06T07:26:26.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.794+0000 7feff37fe640 1 -- 172.21.15.83:0/631912530 learned_addr learned my addr 172.21.15.83:0/631912530 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:26.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.794+0000 7f5c8dd9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c900d0950 0x7f5c900b4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.794+0000 7f5c8dd9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c900d0950 0x7f5c900b4020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47906/0 (socket says 172.21.15.83:47906) 2024-03-06T07:26:26.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.794+0000 7f5c8dd9b640 1 -- 172.21.15.83:0/604173816 learned_addr learned my addr 172.21.15.83:0/604173816 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:26.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.794+0000 7f5c8ed9d640 1 --2- 172.21.15.83:0/604173816 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5c9015e150 0x7f5c90151340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.794+0000 7f5c8e59c640 1 --2- 172.21.15.83:0/604173816 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c900b9060 0x7f5c901f5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.794+0000 7feff27fc640 1 --2- 172.21.15.83:0/631912530 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feff40ad8b0 0x7feff415bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.794+0000 7feff2ffd640 1 --2- 172.21.15.83:0/631912530 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feff406ef30 0x7feff4157fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.794+0000 7f4465d9b640 1 -- 172.21.15.83:0/886613971 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f446815eb00 msgr2=0x7f44680b1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.794+0000 7f4465d9b640 1 --2- 172.21.15.83:0/886613971 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f446815eb00 0x7f44680b1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.794+0000 7f4465d9b640 1 -- 172.21.15.83:0/886613971 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f446806c090 msgr2=0x7f44681511c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.794+0000 7f4465d9b640 1 --2- 172.21.15.83:0/886613971 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f446806c090 0x7f44681511c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.794+0000 7f4465d9b640 1 -- 172.21.15.83:0/886613971 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4468168ff0 con 0x7f4468072810 2024-03-06T07:26:26.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.794+0000 7f4466d9d640 1 --2- 172.21.15.83:0/886613971 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f446815eb00 0x7f44680b1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:26.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.794+0000 7feff2ffd640 1 -- 172.21.15.83:0/631912530 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feff40ad8b0 msgr2=0x7feff415bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.794+0000 7feff2ffd640 1 --2- 172.21.15.83:0/631912530 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feff40ad8b0 0x7feff415bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.794+0000 7feff2ffd640 1 -- 172.21.15.83:0/631912530 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feff415e150 msgr2=0x7feff40ae610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.794+0000 7feff2ffd640 1 --2- 172.21.15.83:0/631912530 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feff415e150 0x7feff40ae610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.794+0000 7f5c8dd9b640 1 -- 172.21.15.83:0/604173816 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c900b9060 msgr2=0x7f5c901f5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.794+0000 7feff2ffd640 1 -- 172.21.15.83:0/631912530 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff41584e0 con 0x7feff406ef30 2024-03-06T07:26:26.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.794+0000 7f5c8dd9b640 1 --2- 172.21.15.83:0/604173816 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c900b9060 0x7f5c901f5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.794+0000 7f5c8dd9b640 1 -- 172.21.15.83:0/604173816 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5c9015e150 msgr2=0x7f5c90151340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.794+0000 7f5c8dd9b640 1 --2- 172.21.15.83:0/604173816 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5c9015e150 0x7f5c90151340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.794+0000 7f5c8dd9b640 1 -- 172.21.15.83:0/604173816 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c900b4560 con 0x7f5c900d0950 2024-03-06T07:26:26.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.794+0000 7f4465d9b640 1 --2- 172.21.15.83:0/886613971 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4468072810 0x7f44680af980 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f445c02bee0 tx=0x7f445c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.794+0000 7feff2ffd640 1 --2- 172.21.15.83:0/631912530 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feff406ef30 0x7feff4157fa0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fefe4033000 tx=0x7fefe40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.795+0000 7f5c8dd9b640 1 --2- 172.21.15.83:0/604173816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c900d0950 0x7f5c900b4020 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f5c84009870 tx=0x7f5c840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.795+0000 7f4456ffd640 1 -- 172.21.15.83:0/886613971 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f445c041070 con 0x7f4468072810 2024-03-06T07:26:26.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.795+0000 7f4456ffd640 1 -- 172.21.15.83:0/886613971 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f445c03a950 con 0x7f4468072810 2024-03-06T07:26:26.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.795+0000 7f446759e640 1 -- 172.21.15.83:0/886613971 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44680adeb0 con 0x7f4468072810 2024-03-06T07:26:26.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.795+0000 7fefdbfff640 1 -- 172.21.15.83:0/631912530 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe4034040 con 0x7feff406ef30 2024-03-06T07:26:26.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.795+0000 7f4456ffd640 1 -- 172.21.15.83:0/886613971 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f445c035040 con 0x7f4468072810 2024-03-06T07:26:26.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.795+0000 7fefdbfff640 1 -- 172.21.15.83:0/631912530 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefe4039d90 con 0x7feff406ef30 2024-03-06T07:26:26.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.795+0000 7fefdbfff640 1 -- 172.21.15.83:0/631912530 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe4043610 con 0x7feff406ef30 2024-03-06T07:26:26.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.795+0000 7f5c7effd640 1 -- 172.21.15.83:0/604173816 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c84041020 con 0x7f5c900d0950 2024-03-06T07:26:26.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.795+0000 7feff999a640 1 -- 172.21.15.83:0/631912530 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feff4159f90 con 0x7feff406ef30 2024-03-06T07:26:26.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.795+0000 7f5c7effd640 1 -- 172.21.15.83:0/604173816 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c84032a40 con 0x7f5c900d0950 2024-03-06T07:26:26.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.795+0000 7f5c7effd640 1 -- 172.21.15.83:0/604173816 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c84037040 con 0x7f5c900d0950 2024-03-06T07:26:26.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.795+0000 7feff999a640 1 -- 172.21.15.83:0/631912530 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feff415a550 con 0x7feff406ef30 2024-03-06T07:26:26.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.795+0000 7f5c94ce0640 1 -- 172.21.15.83:0/604173816 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c900b47f0 con 0x7f5c900d0950 2024-03-06T07:26:26.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.795+0000 7f5c94ce0640 1 -- 172.21.15.83:0/604173816 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c900b1010 con 0x7f5c900d0950 2024-03-06T07:26:26.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.795+0000 7f446759e640 1 -- 172.21.15.83:0/886613971 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44680ae470 con 0x7f4468072810 2024-03-06T07:26:26.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.796+0000 7f4454ff9640 1 -- 172.21.15.83:0/886613971 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4440005290 con 0x7f4468072810 2024-03-06T07:26:26.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.796+0000 7feff999a640 1 -- 172.21.15.83:0/631912530 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feff4068050 con 0x7feff406ef30 2024-03-06T07:26:26.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.797+0000 7fefdbfff640 1 -- 172.21.15.83:0/631912530 <== mon.2 v2:172.21.15.183:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fefe404a060 con 0x7feff406ef30 2024-03-06T07:26:26.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.797+0000 7fefdbfff640 1 --2- 172.21.15.83:0/631912530 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fefe006b4d0 0x7fefe006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.797+0000 7f4456ffd640 1 -- 172.21.15.83:0/886613971 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f445c038020 con 0x7f4468072810 2024-03-06T07:26:26.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.797+0000 7fefdbfff640 1 -- 172.21.15.83:0/631912530 <== mon.2 v2:172.21.15.183:3301/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fefe40abf30 con 0x7feff406ef30 2024-03-06T07:26:26.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.797+0000 7feff27fc640 1 --2- 172.21.15.83:0/631912530 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fefe006b4d0 0x7fefe006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.797+0000 7f4456ffd640 1 --2- 172.21.15.83:0/886613971 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f444806b4d0 0x7f444806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.797+0000 7f5c94ce0640 1 -- 172.21.15.83:0/604173816 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c900d3230 con 0x7f5c900d0950 2024-03-06T07:26:26.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.798+0000 7f4456ffd640 1 -- 172.21.15.83:0/886613971 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f445c0abba0 con 0x7f4468072810 2024-03-06T07:26:26.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.798+0000 7f5c7effd640 1 -- 172.21.15.83:0/604173816 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5c84044020 con 0x7f5c900d0950 2024-03-06T07:26:26.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.798+0000 7feff27fc640 1 --2- 172.21.15.83:0/631912530 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fefe006b4d0 0x7fefe006d990 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7feff415f5b0 tx=0x7fefe8009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.798+0000 7f446659c640 1 --2- 172.21.15.83:0/886613971 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f444806b4d0 0x7f444806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.798+0000 7f5c7effd640 1 --2- 172.21.15.83:0/604173816 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5c7006b5a0 0x7f5c7006da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.798+0000 7f446659c640 1 --2- 172.21.15.83:0/886613971 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f444806b4d0 0x7f444806d990 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f4458001780 tx=0x7f4458009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.798+0000 7f5c8e59c640 1 --2- 172.21.15.83:0/604173816 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5c7006b5a0 0x7f5c7006da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.798+0000 7f5c7effd640 1 -- 172.21.15.83:0/604173816 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5c840abc40 con 0x7f5c900d0950 2024-03-06T07:26:26.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.799+0000 7f5c8e59c640 1 --2- 172.21.15.83:0/604173816 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5c7006b5a0 0x7f5c7006da60 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f5c900cd4f0 tx=0x7f5c80009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.802+0000 7fefdbfff640 1 -- 172.21.15.83:0/631912530 <== mon.2 v2:172.21.15.183:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fefe403e070 con 0x7feff406ef30 2024-03-06T07:26:26.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.802+0000 7f4456ffd640 1 -- 172.21.15.83:0/886613971 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f445c07a3e0 con 0x7f4468072810 2024-03-06T07:26:26.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.807+0000 7f5c7effd640 1 -- 172.21.15.83:0/604173816 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5c900d3230 con 0x7f5c900d0950 2024-03-06T07:26:26.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.849+0000 7f81c6b3e640 1 Processor -- start 2024-03-06T07:26:26.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.849+0000 7f81c6b3e640 1 -- start start 2024-03-06T07:26:26.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.849+0000 7f81c6b3e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f81c015e150 0x7f81c00cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.850+0000 7f81c6b3e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f81c00d1060 0x7f81c00b9060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.850+0000 7f81c6b3e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81c00d1ea0 con 0x7f81c00cfba0 2024-03-06T07:26:26.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.850+0000 7f81c6b3e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81c00d2ab0 con 0x7f81c00d1060 2024-03-06T07:26:26.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.850+0000 7f81bffff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f81c00cfba0 0x7f81c00d0950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:36848/0 (socket says 172.21.15.83:36848) 2024-03-06T07:26:26.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.850+0000 7f81bffff640 1 -- 172.21.15.83:0/351996389 learned_addr learned my addr 172.21.15.83:0/351996389 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:26.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.850+0000 7f81c6b3e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81c00d2180 con 0x7f81c015e150 2024-03-06T07:26:26.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.850+0000 7f81c48da640 1 --2- 172.21.15.83:0/351996389 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f81c015e150 0x7f81c00cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.850+0000 7f81c50db640 1 --2- 172.21.15.83:0/351996389 >> v2:172.21.15.183:3300/0 conn(0x7f81c00d1060 0x7f81c00b9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.850+0000 7f81bf7fe640 1 -- 172.21.15.83:0/351996389 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 805886924 0 0) 0x7f81c00d1ea0 con 0x7f81c00cfba0 2024-03-06T07:26:26.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.850+0000 7f81bf7fe640 1 -- 172.21.15.83:0/351996389 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81b0003520 con 0x7f81c00cfba0 2024-03-06T07:26:26.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.850+0000 7f81c50db640 1 -- 172.21.15.83:0/351996389 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f81c015e150 msgr2=0x7f81c00cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:26.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.850+0000 7f81c50db640 1 --2- 172.21.15.83:0/351996389 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f81c015e150 0x7f81c00cf660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.850+0000 7f81c50db640 1 -- 172.21.15.83:0/351996389 >> v1:172.21.15.83:6789/0 conn(0x7f81c00cfba0 legacy=0x7f81c00d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.850+0000 7f81c50db640 1 -- 172.21.15.83:0/351996389 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81c00d2460 con 0x7f81c00d1060 2024-03-06T07:26:26.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.851+0000 7f81c50db640 1 --2- 172.21.15.83:0/351996389 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f81c00d1060 0x7f81c00b9060 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f81b8009d90 tx=0x7f81b802be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=415bdf8a67f38587 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.851+0000 7f81bf7fe640 1 -- 172.21.15.83:0/351996389 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b8039070 con 0x7f81c00d1060 2024-03-06T07:26:26.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.851+0000 7f81bf7fe640 1 -- 172.21.15.83:0/351996389 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81b802cbd0 con 0x7f81c00d1060 2024-03-06T07:26:26.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.851+0000 7f81bf7fe640 1 -- 172.21.15.83:0/351996389 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b8034440 con 0x7f81c00d1060 2024-03-06T07:26:26.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.852+0000 7f81c6b3e640 1 -- 172.21.15.83:0/351996389 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f81c00d1060 msgr2=0x7f81c00b9060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.852+0000 7f81c6b3e640 1 --2- 172.21.15.83:0/351996389 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f81c00d1060 0x7f81c00b9060 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f81b8009d90 tx=0x7f81b802be20 comp rx=0 tx=0).stop 2024-03-06T07:26:26.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.852+0000 7f81c6b3e640 1 -- 172.21.15.83:0/351996389 shutdown_connections 2024-03-06T07:26:26.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.852+0000 7f81c6b3e640 1 --2- 172.21.15.83:0/351996389 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f81c00d1060 0x7f81c00b9060 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.852+0000 7f81c6b3e640 1 --2- 172.21.15.83:0/351996389 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f81c015e150 0x7f81c00cf660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.852+0000 7f81c6b3e640 1 -- 172.21.15.83:0/351996389 >> 172.21.15.83:0/351996389 conn(0x7f81c009d230 msgr2=0x7f81c00d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:26.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.854+0000 7f81c6b3e640 1 -- 172.21.15.83:0/351996389 shutdown_connections 2024-03-06T07:26:26.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.854+0000 7f81c6b3e640 1 -- 172.21.15.83:0/351996389 wait complete. 2024-03-06T07:26:26.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.855+0000 7f81c6b3e640 1 Processor -- start 2024-03-06T07:26:26.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.855+0000 7f81c6b3e640 1 -- start start 2024-03-06T07:26:26.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.855+0000 7f81c6b3e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f81c00cfba0 0x7f81c0153940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.856+0000 7f81c6b3e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f81c00d1060 0x7f81c0154ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.856+0000 7f81c6b3e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f81c015e150 0x7f81c00c7610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.856+0000 7f81c6b3e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c00d1b30 con 0x7f81c015e150 2024-03-06T07:26:26.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.856+0000 7f81c6b3e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c00d2f40 con 0x7f81c00d1060 2024-03-06T07:26:26.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.856+0000 7f81c6b3e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81c015e530 con 0x7f81c00cfba0 2024-03-06T07:26:26.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.856+0000 7f81c50db640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f81c015e150 0x7f81c00c7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.856+0000 7f81c50db640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f81c015e150 0x7f81c00c7610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47916/0 (socket says 172.21.15.83:47916) 2024-03-06T07:26:26.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.856+0000 7f81c50db640 1 -- 172.21.15.83:0/308822418 learned_addr learned my addr 172.21.15.83:0/308822418 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:26.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.856+0000 7f81c48da640 1 --2- 172.21.15.83:0/308822418 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f81c00cfba0 0x7f81c0153940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.856+0000 7f81bffff640 1 --2- 172.21.15.83:0/308822418 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f81c00d1060 0x7f81c0154ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.856+0000 7f81c50db640 1 -- 172.21.15.83:0/308822418 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f81c00cfba0 msgr2=0x7f81c0153940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.856+0000 7f81c50db640 1 --2- 172.21.15.83:0/308822418 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f81c00cfba0 0x7f81c0153940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.856+0000 7f81c50db640 1 -- 172.21.15.83:0/308822418 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f81c00d1060 msgr2=0x7f81c0154ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.856+0000 7f81c50db640 1 --2- 172.21.15.83:0/308822418 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f81c00d1060 0x7f81c0154ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.856+0000 7f81c50db640 1 -- 172.21.15.83:0/308822418 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81c00b9940 con 0x7f81c015e150 2024-03-06T07:26:26.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.856+0000 7f81bffff640 1 --2- 172.21.15.83:0/308822418 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f81c00d1060 0x7f81c0154ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:26.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.857+0000 7f81c50db640 1 --2- 172.21.15.83:0/308822418 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f81c015e150 0x7f81c00c7610 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f81b8009eb0 tx=0x7f81b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.857+0000 7f81bd7fa640 1 -- 172.21.15.83:0/308822418 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b8039050 con 0x7f81c015e150 2024-03-06T07:26:26.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.857+0000 7f81bd7fa640 1 -- 172.21.15.83:0/308822418 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81b8045070 con 0x7f81c015e150 2024-03-06T07:26:26.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.857+0000 7f81bd7fa640 1 -- 172.21.15.83:0/308822418 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b8009450 con 0x7f81c015e150 2024-03-06T07:26:26.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.857+0000 7f81c6b3e640 1 -- 172.21.15.83:0/308822418 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81c0154200 con 0x7f81c015e150 2024-03-06T07:26:26.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.857+0000 7f81c6b3e640 1 -- 172.21.15.83:0/308822418 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81c01547c0 con 0x7f81c015e150 2024-03-06T07:26:26.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.859+0000 7f81c6b3e640 1 -- 172.21.15.83:0/308822418 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81c0069ee0 con 0x7f81c015e150 2024-03-06T07:26:26.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.859+0000 7f81bd7fa640 1 -- 172.21.15.83:0/308822418 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f81b803e850 con 0x7f81c015e150 2024-03-06T07:26:26.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.860+0000 7f81bd7fa640 1 --2- 172.21.15.83:0/308822418 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f81a406b4d0 0x7f81a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.860+0000 7f81bd7fa640 1 -- 172.21.15.83:0/308822418 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f81b80acf10 con 0x7f81c015e150 2024-03-06T07:26:26.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.860+0000 7f81c48da640 1 --2- 172.21.15.83:0/308822418 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f81a406b4d0 0x7f81a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.860+0000 7f81c48da640 1 --2- 172.21.15.83:0/308822418 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f81a406b4d0 0x7f81a406d990 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f81b4009aa0 tx=0x7f81b4009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:26.868+0000 7f81bd7fa640 1 -- 172.21.15.83:0/308822418 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f81b807bf40 con 0x7f81c015e150 2024-03-06T07:26:27.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.057+0000 7f5c94ce0640 1 -- 172.21.15.83:0/604173816 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7f5c90069ee0 con 0x7f5c900d0950 2024-03-06T07:26:27.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.059+0000 7feff999a640 1 -- 172.21.15.83:0/631912530 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "c"} v 0) v1 -- 0x7feff40d09c0 con 0x7feff406ef30 2024-03-06T07:26:27.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.081+0000 7f4454ff9640 1 -- 172.21.15.83:0/886613971 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7f4440005850 con 0x7f4468072810 2024-03-06T07:26:27.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.116+0000 7f81c6b3e640 1 -- 172.21.15.83:0/308822418 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) v1 -- 0x7f81c00d4670 con 0x7f81c015e150 2024-03-06T07:26:27.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.765+0000 7f5c7effd640 1 -- 172.21.15.83:0/604173816 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 6490 v46) v1 ==== 95+0+0 (secure 0 0 0) 0x7f5c8404b030 con 0x7f5c900d0950 2024-03-06T07:26:27.767 INFO:teuthology.orchestra.run.smithi083.stderr:failed mds gid 6490 2024-03-06T07:26:27.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.768+0000 7f5c94ce0640 1 -- 172.21.15.83:0/604173816 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5c7006b5a0 msgr2=0x7f5c7006da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.768+0000 7f5c94ce0640 1 --2- 172.21.15.83:0/604173816 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5c7006b5a0 0x7f5c7006da60 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f5c900cd4f0 tx=0x7f5c80009040 comp rx=0 tx=0).stop 2024-03-06T07:26:27.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.768+0000 7f5c94ce0640 1 -- 172.21.15.83:0/604173816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c900d0950 msgr2=0x7f5c900b4020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.768+0000 7f5c94ce0640 1 --2- 172.21.15.83:0/604173816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c900d0950 0x7f5c900b4020 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f5c84009870 tx=0x7f5c840046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.768+0000 7f5c94ce0640 1 -- 172.21.15.83:0/604173816 shutdown_connections 2024-03-06T07:26:27.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.768+0000 7f5c94ce0640 1 --2- 172.21.15.83:0/604173816 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5c7006b5a0 0x7f5c7006da60 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.768+0000 7f5c94ce0640 1 --2- 172.21.15.83:0/604173816 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5c9015e150 0x7f5c90151340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.768+0000 7f5c94ce0640 1 --2- 172.21.15.83:0/604173816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c900d0950 0x7f5c900b4020 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.768+0000 7f5c94ce0640 1 --2- 172.21.15.83:0/604173816 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c900b9060 0x7f5c901f5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.768+0000 7f5c94ce0640 1 -- 172.21.15.83:0/604173816 >> 172.21.15.83:0/604173816 conn(0x7f5c9009d230 msgr2=0x7f5c900b6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:27.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.769+0000 7f5c94ce0640 1 -- 172.21.15.83:0/604173816 shutdown_connections 2024-03-06T07:26:27.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.769+0000 7f5c94ce0640 1 -- 172.21.15.83:0/604173816 wait complete. 2024-03-06T07:26:27.783 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:26:27.783 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:26:27.785 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:26:27.786 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:27.786 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-03-06T07:26:27.784+0000 7fb0d860c640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:26:27.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.787+0000 7f4456ffd640 1 -- 172.21.15.83:0/886613971 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 5924 v47) v1 ==== 95+0+0 (secure 0 0 0) 0x7f445c07e480 con 0x7f4468072810 2024-03-06T07:26:27.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.787+0000 7f81bd7fa640 1 -- 172.21.15.83:0/308822418 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 7338 v47) v1 ==== 95+0+0 (secure 0 0 0) 0x7f81b807b840 con 0x7f81c015e150 2024-03-06T07:26:27.789 INFO:teuthology.orchestra.run.smithi083.stderr:failed mds gid 7338 2024-03-06T07:26:27.789 INFO:teuthology.orchestra.run.smithi083.stderr:failed mds gid 5924 2024-03-06T07:26:27.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.789+0000 7f81c6b3e640 1 -- 172.21.15.83:0/308822418 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f81a406b4d0 msgr2=0x7f81a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.789+0000 7f81c6b3e640 1 --2- 172.21.15.83:0/308822418 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f81a406b4d0 0x7f81a406d990 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f81b4009aa0 tx=0x7f81b4009190 comp rx=0 tx=0).stop 2024-03-06T07:26:27.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.789+0000 7f81c6b3e640 1 -- 172.21.15.83:0/308822418 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f81c015e150 msgr2=0x7f81c00c7610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.789+0000 7f81c6b3e640 1 --2- 172.21.15.83:0/308822418 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f81c015e150 0x7f81c00c7610 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f81b8009eb0 tx=0x7f81b8004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:27.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.789+0000 7f4454ff9640 1 -- 172.21.15.83:0/886613971 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f444806b4d0 msgr2=0x7f444806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.789+0000 7f4454ff9640 1 --2- 172.21.15.83:0/886613971 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f444806b4d0 0x7f444806d990 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f4458001780 tx=0x7f4458009040 comp rx=0 tx=0).stop 2024-03-06T07:26:27.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.790+0000 7f4454ff9640 1 -- 172.21.15.83:0/886613971 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4468072810 msgr2=0x7f44680af980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.790+0000 7f4454ff9640 1 --2- 172.21.15.83:0/886613971 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4468072810 0x7f44680af980 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f445c02bee0 tx=0x7f445c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.790+0000 7f81c6b3e640 1 -- 172.21.15.83:0/308822418 shutdown_connections 2024-03-06T07:26:27.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.790+0000 7f81c6b3e640 1 --2- 172.21.15.83:0/308822418 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f81a406b4d0 0x7f81a406d990 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.790+0000 7f81c6b3e640 1 --2- 172.21.15.83:0/308822418 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f81c015e150 0x7f81c00c7610 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.790+0000 7f81c6b3e640 1 --2- 172.21.15.83:0/308822418 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f81c00d1060 0x7f81c0154ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.790+0000 7f81c6b3e640 1 --2- 172.21.15.83:0/308822418 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f81c00cfba0 0x7f81c0153940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.790+0000 7f81c6b3e640 1 -- 172.21.15.83:0/308822418 >> 172.21.15.83:0/308822418 conn(0x7f81c009d230 msgr2=0x7f81c006d190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:27.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.790+0000 7f4454ff9640 1 -- 172.21.15.83:0/886613971 shutdown_connections 2024-03-06T07:26:27.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.790+0000 7f4454ff9640 1 --2- 172.21.15.83:0/886613971 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f444806b4d0 0x7f444806d990 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.790+0000 7f4454ff9640 1 --2- 172.21.15.83:0/886613971 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f446815eb00 0x7f44680b1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.790+0000 7f4454ff9640 1 --2- 172.21.15.83:0/886613971 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4468072810 0x7f44680af980 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.790+0000 7f4454ff9640 1 --2- 172.21.15.83:0/886613971 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f446806c090 0x7f44681511c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.790+0000 7f4454ff9640 1 -- 172.21.15.83:0/886613971 >> 172.21.15.83:0/886613971 conn(0x7f44680b2930 msgr2=0x7f44680b08a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:27.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.790+0000 7f4454ff9640 1 -- 172.21.15.83:0/886613971 shutdown_connections 2024-03-06T07:26:27.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.790+0000 7f4454ff9640 1 -- 172.21.15.83:0/886613971 wait complete. 2024-03-06T07:26:27.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.790+0000 7f81c6b3e640 1 -- 172.21.15.83:0/308822418 shutdown_connections 2024-03-06T07:26:27.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.790+0000 7f81c6b3e640 1 -- 172.21.15.83:0/308822418 wait complete. 2024-03-06T07:26:27.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.790+0000 7fefdbfff640 1 -- 172.21.15.83:0/631912530 <== mon.2 v2:172.21.15.183:3301/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "c"}]=0 failed mds gid 7335 v47) v1 ==== 95+0+0 (secure 0 0 0) 0x7fefe407a7f0 con 0x7feff406ef30 2024-03-06T07:26:27.795 INFO:teuthology.orchestra.run.smithi083.stderr:failed mds gid 7335 2024-03-06T07:26:27.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.794+0000 7feff999a640 1 -- 172.21.15.83:0/631912530 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fefe006b4d0 msgr2=0x7fefe006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.794+0000 7feff999a640 1 --2- 172.21.15.83:0/631912530 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fefe006b4d0 0x7fefe006d990 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7feff415f5b0 tx=0x7fefe8009190 comp rx=0 tx=0).stop 2024-03-06T07:26:27.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.794+0000 7feff999a640 1 -- 172.21.15.83:0/631912530 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feff406ef30 msgr2=0x7feff4157fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.794+0000 7feff999a640 1 --2- 172.21.15.83:0/631912530 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feff406ef30 0x7feff4157fa0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fefe4033000 tx=0x7fefe40046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.794+0000 7feff999a640 1 -- 172.21.15.83:0/631912530 shutdown_connections 2024-03-06T07:26:27.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.794+0000 7feff999a640 1 --2- 172.21.15.83:0/631912530 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fefe006b4d0 0x7fefe006d990 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.794+0000 7feff999a640 1 --2- 172.21.15.83:0/631912530 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feff415e150 0x7feff40ae610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.794+0000 7feff999a640 1 --2- 172.21.15.83:0/631912530 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feff40ad8b0 0x7feff415bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.794+0000 7feff999a640 1 --2- 172.21.15.83:0/631912530 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feff406ef30 0x7feff4157fa0 secure :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fefe4033000 tx=0x7fefe40046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.794+0000 7feff999a640 1 -- 172.21.15.83:0/631912530 >> 172.21.15.83:0/631912530 conn(0x7feff40d0470 msgr2=0x7feff4158910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:27.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.795+0000 7feff999a640 1 -- 172.21.15.83:0/631912530 shutdown_connections 2024-03-06T07:26:27.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.795+0000 7feff999a640 1 -- 172.21.15.83:0/631912530 wait complete. 2024-03-06T07:26:27.806 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:26:27.806 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:26:27.808 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:26:27.808 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:27.809 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:26:27.809 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:26:27.812 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:26:27.812 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:26:27.854 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:26:27.854 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:27.856 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:26:27.856 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:27.856 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:27.865 INFO:tasks.ceph.mds.b.smithi183.stdout:starting mds.b at 2024-03-06T07:26:27.885 INFO:tasks.ceph.mds.d.smithi183.stdout:starting mds.d at 2024-03-06T07:26:27.931 INFO:tasks.ceph.mds.c.smithi083.stdout:starting mds.c at 2024-03-06T07:26:27.931 INFO:tasks.ceph.mds.a.smithi083.stdout:starting mds.a at 2024-03-06T07:26:27.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.958+0000 7f9d094f4640 1 Processor -- start 2024-03-06T07:26:27.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.958+0000 7f9d094f4640 1 -- start start 2024-03-06T07:26:27.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.959+0000 7f9d094f4640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f9d0415e150 0x7f9d040cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.959+0000 7f9d094f4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d040d1ea0 con 0x7f9d040cfba0 2024-03-06T07:26:27.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.959+0000 7f9d094f4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d040d2180 con 0x7f9d040d1060 2024-03-06T07:26:27.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.959+0000 7f9d094f4640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d040d2ab0 con 0x7f9d0415e150 2024-03-06T07:26:27.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.959+0000 7f9d027fc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f9d040cfba0 0x7f9d040d0950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:36870/0 (socket says 172.21.15.83:36870) 2024-03-06T07:26:27.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.959+0000 7f9d027fc640 1 -- 172.21.15.83:0/3480676125 learned_addr learned my addr 172.21.15.83:0/3480676125 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:27.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.959+0000 7f9d02ffd640 1 --2- 172.21.15.83:0/3480676125 >> v2:172.21.15.183:3300/0 conn(0x7f9d0415e150 0x7f9d040cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.959+0000 7f9d01ffb640 1 -- 172.21.15.83:0/3480676125 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1492407171 0 0) 0x7f9d040d1ea0 con 0x7f9d040cfba0 2024-03-06T07:26:27.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.959+0000 7f9d01ffb640 1 -- 172.21.15.83:0/3480676125 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cec003520 con 0x7f9d040cfba0 2024-03-06T07:26:27.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.959+0000 7f9d02ffd640 1 -- 172.21.15.83:0/3480676125 >> v1:172.21.15.183:6789/0 conn(0x7f9d040d1060 legacy=0x7f9d040b9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.959+0000 7f9d02ffd640 1 -- 172.21.15.83:0/3480676125 >> v1:172.21.15.83:6789/0 conn(0x7f9d040cfba0 legacy=0x7f9d040d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.959+0000 7f9d01ffb640 1 -- 172.21.15.83:0/3480676125 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2830483669 0 0) 0x7f9cec003520 con 0x7f9d040cfba0 2024-03-06T07:26:27.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.959+0000 7f9d02ffd640 1 -- 172.21.15.83:0/3480676125 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d040d2460 con 0x7f9d0415e150 2024-03-06T07:26:27.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.960+0000 7f9d02ffd640 1 --2- 172.21.15.83:0/3480676125 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9d0415e150 0x7f9d040cf660 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f9cf40097a0 tx=0x7f9cf402b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1587b4d12ac1d0cc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:27.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.960+0000 7f9d01ffb640 1 -- 172.21.15.83:0/3480676125 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf4037070 con 0x7f9d0415e150 2024-03-06T07:26:27.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.960+0000 7f9d01ffb640 1 -- 172.21.15.83:0/3480676125 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf4030510 con 0x7f9d0415e150 2024-03-06T07:26:27.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.960+0000 7f9d01ffb640 1 -- 172.21.15.83:0/3480676125 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf8001670 con 0x7f9d0415e150 2024-03-06T07:26:27.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.961+0000 7f9d094f4640 1 -- 172.21.15.83:0/3480676125 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9d0415e150 msgr2=0x7f9d040cf660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.961+0000 7f9d094f4640 1 --2- 172.21.15.83:0/3480676125 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9d0415e150 0x7f9d040cf660 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f9cf40097a0 tx=0x7f9cf402b790 comp rx=0 tx=0).stop 2024-03-06T07:26:27.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.961+0000 7f9d094f4640 1 -- 172.21.15.83:0/3480676125 shutdown_connections 2024-03-06T07:26:27.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.961+0000 7f9d094f4640 1 --2- 172.21.15.83:0/3480676125 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9d0415e150 0x7f9d040cf660 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.961+0000 7f9d094f4640 1 -- 172.21.15.83:0/3480676125 >> 172.21.15.83:0/3480676125 conn(0x7f9d0409d230 msgr2=0x7f9d040d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:27.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.961+0000 7f9d094f4640 1 -- 172.21.15.83:0/3480676125 shutdown_connections 2024-03-06T07:26:27.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.961+0000 7f9d094f4640 1 -- 172.21.15.83:0/3480676125 wait complete. 2024-03-06T07:26:27.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.962+0000 7f9d094f4640 1 Processor -- start 2024-03-06T07:26:27.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.962+0000 7f9d094f4640 1 -- start start 2024-03-06T07:26:27.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.963+0000 7f9d094f4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9d040cfba0 0x7f9d0415b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.963+0000 7f9d094f4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d040d1060 0x7f9d040b92c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.963+0000 7f9d094f4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9d0415e150 0x7f9d040b9800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.963+0000 7f9d094f4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d040b7dc0 con 0x7f9d040d1060 2024-03-06T07:26:27.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.963+0000 7f9d094f4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d040d2f40 con 0x7f9d0415e150 2024-03-06T07:26:27.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.963+0000 7f9d094f4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d040d1b30 con 0x7f9d040cfba0 2024-03-06T07:26:27.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.963+0000 7f9d027fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d040d1060 0x7f9d040b92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.963+0000 7f9d027fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d040d1060 0x7f9d040b92c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47940/0 (socket says 172.21.15.83:47940) 2024-03-06T07:26:27.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.963+0000 7f9d027fc640 1 -- 172.21.15.83:0/1187783764 learned_addr learned my addr 172.21.15.83:0/1187783764 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:27.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.963+0000 7f9d02ffd640 1 --2- 172.21.15.83:0/1187783764 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9d040cfba0 0x7f9d0415b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.963+0000 7f9d037fe640 1 --2- 172.21.15.83:0/1187783764 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9d0415e150 0x7f9d040b9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.963+0000 7f9d027fc640 1 -- 172.21.15.83:0/1187783764 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9d040cfba0 msgr2=0x7f9d0415b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.963+0000 7f9d027fc640 1 --2- 172.21.15.83:0/1187783764 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9d040cfba0 0x7f9d0415b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.963+0000 7f9d027fc640 1 -- 172.21.15.83:0/1187783764 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9d0415e150 msgr2=0x7f9d040b9800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.963+0000 7f9d027fc640 1 --2- 172.21.15.83:0/1187783764 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9d0415e150 0x7f9d040b9800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.963+0000 7f9d027fc640 1 -- 172.21.15.83:0/1187783764 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d0415bab0 con 0x7f9d040d1060 2024-03-06T07:26:27.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.963+0000 7f9d027fc640 1 --2- 172.21.15.83:0/1187783764 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d040d1060 0x7f9d040b92c0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f9cf800c720 tx=0x7f9cf800cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:27.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.963+0000 7f9ceb7fe640 1 -- 172.21.15.83:0/1187783764 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf8005ad0 con 0x7f9d040d1060 2024-03-06T07:26:27.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.964+0000 7f9ceb7fe640 1 -- 172.21.15.83:0/1187783764 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf8005c70 con 0x7f9d040d1060 2024-03-06T07:26:27.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.964+0000 7f9ceb7fe640 1 -- 172.21.15.83:0/1187783764 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf8005e60 con 0x7f9d040d1060 2024-03-06T07:26:27.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.964+0000 7f9d094f4640 1 -- 172.21.15.83:0/1187783764 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d0415bd90 con 0x7f9d040d1060 2024-03-06T07:26:27.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.964+0000 7f9d094f4640 1 -- 172.21.15.83:0/1187783764 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d040cd280 con 0x7f9d040d1060 2024-03-06T07:26:27.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.965+0000 7f9d094f4640 1 -- 172.21.15.83:0/1187783764 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d0406d830 con 0x7f9d040d1060 2024-03-06T07:26:27.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.965+0000 7f9ceb7fe640 1 -- 172.21.15.83:0/1187783764 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9cf8007270 con 0x7f9d040d1060 2024-03-06T07:26:27.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.965+0000 7f9ceb7fe640 1 --2- 172.21.15.83:0/1187783764 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9cf006b4d0 0x7f9cf006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.966+0000 7f9ceb7fe640 1 -- 172.21.15.83:0/1187783764 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f9cf8089ff0 con 0x7f9d040d1060 2024-03-06T07:26:27.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.966+0000 7f9d02ffd640 1 --2- 172.21.15.83:0/1187783764 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9cf006b4d0 0x7f9cf006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.966+0000 7f9d02ffd640 1 --2- 172.21.15.83:0/1187783764 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9cf006b4d0 0x7f9cf006d990 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f9cf4034000 tx=0x7f9cf402f000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:27.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:27.970+0000 7f9ceb7fe640 1 -- 172.21.15.83:0/1187783764 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9d0406d830 con 0x7f9d040d1060 2024-03-06T07:26:28.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:28.176+0000 7f9d094f4640 1 -- 172.21.15.83:0/1187783764 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d04069ee0 con 0x7f9d040d1060 2024-03-06T07:26:28.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:28.176+0000 7f9ceb7fe640 1 -- 172.21.15.83:0/1187783764 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+1801 (secure 0 0 0) 0x7f9cf8007560 con 0x7f9d040d1060 2024-03-06T07:26:28.178 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:28.178 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":47,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.064335+0000","modified":"2024-03-06T07:26:27.785245+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-03-06T07:26:28.179 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 47 2024-03-06T07:26:28.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:28.178+0000 7f9d094f4640 1 -- 172.21.15.83:0/1187783764 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9cf006b4d0 msgr2=0x7f9cf006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:28.178+0000 7f9d094f4640 1 --2- 172.21.15.83:0/1187783764 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9cf006b4d0 0x7f9cf006d990 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f9cf4034000 tx=0x7f9cf402f000 comp rx=0 tx=0).stop 2024-03-06T07:26:28.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:28.178+0000 7f9d094f4640 1 -- 172.21.15.83:0/1187783764 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d040d1060 msgr2=0x7f9d040b92c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:28.178+0000 7f9d094f4640 1 --2- 172.21.15.83:0/1187783764 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d040d1060 0x7f9d040b92c0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f9cf800c720 tx=0x7f9cf800cbe0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:28.178+0000 7f9d094f4640 1 -- 172.21.15.83:0/1187783764 shutdown_connections 2024-03-06T07:26:28.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:28.178+0000 7f9d094f4640 1 --2- 172.21.15.83:0/1187783764 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9cf006b4d0 0x7f9cf006d990 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:28.178+0000 7f9d094f4640 1 --2- 172.21.15.83:0/1187783764 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9d0415e150 0x7f9d040b9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:28.178+0000 7f9d094f4640 1 --2- 172.21.15.83:0/1187783764 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d040d1060 0x7f9d040b92c0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:28.178+0000 7f9d094f4640 1 --2- 172.21.15.83:0/1187783764 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9d040cfba0 0x7f9d0415b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:28.178+0000 7f9d094f4640 1 -- 172.21.15.83:0/1187783764 >> 172.21.15.83:0/1187783764 conn(0x7f9d0409d230 msgr2=0x7f9d04160540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:28.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:28.178+0000 7f9d094f4640 1 -- 172.21.15.83:0/1187783764 shutdown_connections 2024-03-06T07:26:28.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:28.178+0000 7f9d094f4640 1 -- 172.21.15.83:0/1187783764 wait complete. 2024-03-06T07:26:28.596 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 47, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:26:19.064335+0000', 'modified': '2024-03-06T07:26:27.785245+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 98, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:26:28.596 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-03-06T07:26:29.185 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:29.187 DEBUG:teuthology.orchestra.run.smithi183:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:29.597 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:29.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.706+0000 7f278c881640 1 Processor -- start 2024-03-06T07:26:29.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.706+0000 7f278c881640 1 -- start start 2024-03-06T07:26:29.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.707+0000 7f278c881640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f27880ce980 0x7f27880cee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.707+0000 7f278c881640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27880b5300 con 0x7f2788162940 2024-03-06T07:26:29.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.707+0000 7f278c881640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27880ccbd0 con 0x7f27880ce5a0 2024-03-06T07:26:29.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.707+0000 7f278c881640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27880b3b00 con 0x7f27880ce980 2024-03-06T07:26:29.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.708+0000 7f2786d9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f2788162940 0x7f2788164d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:36874/0 (socket says 172.21.15.83:36874) 2024-03-06T07:26:29.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.708+0000 7f2786d9d640 1 -- 172.21.15.83:0/1704208024 learned_addr learned my addr 172.21.15.83:0/1704208024 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:29.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.708+0000 7f2785d9b640 1 --2- 172.21.15.83:0/1704208024 >> v2:172.21.15.183:3300/0 conn(0x7f27880ce980 0x7f27880cee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.708+0000 7f278559a640 1 -- 172.21.15.83:0/1704208024 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 680714370 0 0) 0x7f27880b5300 con 0x7f2788162940 2024-03-06T07:26:29.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.708+0000 7f278559a640 1 -- 172.21.15.83:0/1704208024 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2774003520 con 0x7f2788162940 2024-03-06T07:26:29.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.708+0000 7f278559a640 1 -- 172.21.15.83:0/1704208024 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3855745320 0 0) 0x7f27880ccbd0 con 0x7f27880ce5a0 2024-03-06T07:26:29.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.708+0000 7f278559a640 1 -- 172.21.15.83:0/1704208024 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27880b5300 con 0x7f27880ce5a0 2024-03-06T07:26:29.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.708+0000 7f278559a640 1 -- 172.21.15.83:0/1704208024 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1448813769 0 0) 0x7f2774003520 con 0x7f2788162940 2024-03-06T07:26:29.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.708+0000 7f2785d9b640 1 -- 172.21.15.83:0/1704208024 >> v1:172.21.15.183:6789/0 conn(0x7f27880ce5a0 legacy=0x7f27880b4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.708+0000 7f2785d9b640 1 -- 172.21.15.83:0/1704208024 >> v1:172.21.15.83:6789/0 conn(0x7f2788162940 legacy=0x7f2788164d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.708+0000 7f2785d9b640 1 -- 172.21.15.83:0/1704208024 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27880b3de0 con 0x7f27880ce980 2024-03-06T07:26:29.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.709+0000 7f2785d9b640 1 --2- 172.21.15.83:0/1704208024 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f27880ce980 0x7f27880cee10 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f27780097a0 tx=0x7f277802b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=840253d559948d85 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:29.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.709+0000 7f278559a640 1 -- 172.21.15.83:0/1704208024 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2778038070 con 0x7f27880ce980 2024-03-06T07:26:29.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.709+0000 7f278559a640 1 -- 172.21.15.83:0/1704208024 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2778031500 con 0x7f27880ce980 2024-03-06T07:26:29.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.709+0000 7f278559a640 1 -- 172.21.15.83:0/1704208024 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27800019f0 con 0x7f27880ce980 2024-03-06T07:26:29.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.710+0000 7f278c881640 1 -- 172.21.15.83:0/1704208024 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f27880ce980 msgr2=0x7f27880cee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.710+0000 7f278c881640 1 --2- 172.21.15.83:0/1704208024 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f27880ce980 0x7f27880cee10 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f27780097a0 tx=0x7f277802b820 comp rx=0 tx=0).stop 2024-03-06T07:26:29.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.710+0000 7f278c881640 1 -- 172.21.15.83:0/1704208024 shutdown_connections 2024-03-06T07:26:29.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.710+0000 7f278c881640 1 --2- 172.21.15.83:0/1704208024 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f27880ce980 0x7f27880cee10 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.710+0000 7f278c881640 1 -- 172.21.15.83:0/1704208024 >> 172.21.15.83:0/1704208024 conn(0x7f278806b160 msgr2=0x7f278806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:29.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.710+0000 7f278c881640 1 -- 172.21.15.83:0/1704208024 shutdown_connections 2024-03-06T07:26:29.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.710+0000 7f278c881640 1 -- 172.21.15.83:0/1704208024 wait complete. 2024-03-06T07:26:29.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.711+0000 7f278c881640 1 Processor -- start 2024-03-06T07:26:29.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.711+0000 7f278c881640 1 -- start start 2024-03-06T07:26:29.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.712+0000 7f278c881640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27880ce5a0 0x7f2788150620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.712+0000 7f278c881640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f27880ce980 0x7f27880ade90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.712+0000 7f278c881640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2788162940 0x7f27880ae3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.712+0000 7f278c881640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27880cc830 con 0x7f27880ce5a0 2024-03-06T07:26:29.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.712+0000 7f278c881640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27880cf350 con 0x7f2788162940 2024-03-06T07:26:29.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.712+0000 7f278c881640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27880cc9b0 con 0x7f27880ce980 2024-03-06T07:26:29.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.712+0000 7f278659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27880ce5a0 0x7f2788150620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.712+0000 7f278659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27880ce5a0 0x7f2788150620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47956/0 (socket says 172.21.15.83:47956) 2024-03-06T07:26:29.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.712+0000 7f278659c640 1 -- 172.21.15.83:0/1134406072 learned_addr learned my addr 172.21.15.83:0/1134406072 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:29.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.712+0000 7f2786d9d640 1 --2- 172.21.15.83:0/1134406072 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2788162940 0x7f27880ae3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.712+0000 7f278659c640 1 -- 172.21.15.83:0/1134406072 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f27880ce980 msgr2=0x7f27880ade90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.712+0000 7f2785d9b640 1 --2- 172.21.15.83:0/1134406072 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f27880ce980 0x7f27880ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.712+0000 7f278659c640 1 --2- 172.21.15.83:0/1134406072 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f27880ce980 0x7f27880ade90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.712+0000 7f278659c640 1 -- 172.21.15.83:0/1134406072 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2788162940 msgr2=0x7f27880ae3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.712+0000 7f278659c640 1 --2- 172.21.15.83:0/1134406072 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2788162940 0x7f27880ae3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.712+0000 7f278659c640 1 -- 172.21.15.83:0/1134406072 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2788150b60 con 0x7f27880ce5a0 2024-03-06T07:26:29.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.712+0000 7f2785d9b640 1 --2- 172.21.15.83:0/1134406072 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f27880ce980 0x7f27880ade90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:29.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.712+0000 7f2786d9d640 1 --2- 172.21.15.83:0/1134406072 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2788162940 0x7f27880ae3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:29.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.713+0000 7f278659c640 1 --2- 172.21.15.83:0/1134406072 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27880ce5a0 0x7f2788150620 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f277000c530 tx=0x7f277000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:29.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.713+0000 7f277effd640 1 -- 172.21.15.83:0/1134406072 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27700052d0 con 0x7f27880ce5a0 2024-03-06T07:26:29.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.713+0000 7f277effd640 1 -- 172.21.15.83:0/1134406072 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2770005490 con 0x7f27880ce5a0 2024-03-06T07:26:29.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.713+0000 7f278c881640 1 -- 172.21.15.83:0/1134406072 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f278815b500 con 0x7f27880ce5a0 2024-03-06T07:26:29.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.713+0000 7f277effd640 1 -- 172.21.15.83:0/1134406072 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2770012870 con 0x7f27880ce5a0 2024-03-06T07:26:29.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.713+0000 7f278c881640 1 -- 172.21.15.83:0/1134406072 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f278815bac0 con 0x7f27880ce5a0 2024-03-06T07:26:29.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.714+0000 7f278c881640 1 -- 172.21.15.83:0/1134406072 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27880ccbd0 con 0x7f27880ce5a0 2024-03-06T07:26:29.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.714+0000 7f277effd640 1 -- 172.21.15.83:0/1134406072 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2770012a10 con 0x7f27880ce5a0 2024-03-06T07:26:29.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.715+0000 7f277effd640 1 --2- 172.21.15.83:0/1134406072 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f276806b4d0 0x7f276806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.715+0000 7f2785d9b640 1 --2- 172.21.15.83:0/1134406072 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f276806b4d0 0x7f276806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.715+0000 7f277effd640 1 -- 172.21.15.83:0/1134406072 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f2770089ec0 con 0x7f27880ce5a0 2024-03-06T07:26:29.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.715+0000 7f2785d9b640 1 --2- 172.21.15.83:0/1134406072 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f276806b4d0 0x7f276806d990 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f2778035000 tx=0x7f2778008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:29.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.719+0000 7f277effd640 1 -- 172.21.15.83:0/1134406072 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f27880ccbd0 con 0x7f27880ce5a0 2024-03-06T07:26:29.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.929+0000 7f278c881640 1 -- 172.21.15.83:0/1134406072 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2788069ee0 con 0x7f27880ce5a0 2024-03-06T07:26:29.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.929+0000 7f277effd640 1 -- 172.21.15.83:0/1134406072 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f2770058cf0 con 0x7f27880ce5a0 2024-03-06T07:26:29.931 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:29.932 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":50,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6023,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/601236077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":601236077},{"type":"v1","addr":"172.21.15.183:6833","nonce":601236077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6029,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1435055961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1435055961},{"type":"v1","addr":"172.21.15.183:6835","nonce":1435055961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":7572,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/2726045966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2726045966},{"type":"v1","addr":"172.21.15.83:6836","nonce":2726045966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.064335+0000","modified":"2024-03-06T07:26:28.812110+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7575":{"gid":7575,"name":"a","rank":0,"incarnation":49,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.83:6837/3254575964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":3254575964},{"type":"v1","addr":"172.21.15.83:6837","nonce":3254575964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-03-06T07:26:29.932 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 50 2024-03-06T07:26:29.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.932+0000 7f278c881640 1 -- 172.21.15.83:0/1134406072 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f276806b4d0 msgr2=0x7f276806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.932+0000 7f278c881640 1 --2- 172.21.15.83:0/1134406072 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f276806b4d0 0x7f276806d990 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f2778035000 tx=0x7f2778008030 comp rx=0 tx=0).stop 2024-03-06T07:26:29.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.932+0000 7f278c881640 1 -- 172.21.15.83:0/1134406072 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27880ce5a0 msgr2=0x7f2788150620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.932+0000 7f278c881640 1 --2- 172.21.15.83:0/1134406072 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27880ce5a0 0x7f2788150620 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f277000c530 tx=0x7f277000c9f0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.932+0000 7f278c881640 1 -- 172.21.15.83:0/1134406072 shutdown_connections 2024-03-06T07:26:29.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.932+0000 7f278c881640 1 --2- 172.21.15.83:0/1134406072 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f276806b4d0 0x7f276806d990 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.932+0000 7f278c881640 1 --2- 172.21.15.83:0/1134406072 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2788162940 0x7f27880ae3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.932+0000 7f278c881640 1 --2- 172.21.15.83:0/1134406072 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f27880ce980 0x7f27880ade90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.932+0000 7f278c881640 1 --2- 172.21.15.83:0/1134406072 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27880ce5a0 0x7f2788150620 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.932+0000 7f278c881640 1 -- 172.21.15.83:0/1134406072 >> 172.21.15.83:0/1134406072 conn(0x7f278806b160 msgr2=0x7f27880b2c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:29.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.932+0000 7f278c881640 1 -- 172.21.15.83:0/1134406072 shutdown_connections 2024-03-06T07:26:29.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:29.932+0000 7f278c881640 1 -- 172.21.15.83:0/1134406072 wait complete. 2024-03-06T07:26:30.347 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 50, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:26:19.064335+0000', 'modified': '2024-03-06T07:26:28.812110+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 98, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7575}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7575': {'gid': 7575, 'name': 'a', 'rank': 0, 'incarnation': 49, 'state': 'up:reconnect', 'state_seq': 2, 'addr': '172.21.15.83:6837/3254575964', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 3254575964}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 3254575964}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:26:30.347 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7575:up:reconnect 2024-03-06T07:26:31.349 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:31.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.458+0000 7fbd5defb640 1 Processor -- start 2024-03-06T07:26:31.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.458+0000 7fbd5defb640 1 -- start start 2024-03-06T07:26:31.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.459+0000 7fbd5defb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbd580ce5a0 0x7fbd580b4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:31.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.459+0000 7fbd5defb640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fbd58162940 0x7fbd58164d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:31.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.459+0000 7fbd5defb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd580ccbd0 con 0x7fbd580ce980 2024-03-06T07:26:31.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.459+0000 7fbd5defb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd580b3b00 con 0x7fbd580ce5a0 2024-03-06T07:26:31.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.459+0000 7fbd5defb640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd580b3de0 con 0x7fbd58162940 2024-03-06T07:26:31.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.459+0000 7fbd577fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbd580ce5a0 0x7fbd580b4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:31.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.459+0000 7fbd56ffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fbd580ce980 0x7fbd580cee10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:36886/0 (socket says 172.21.15.83:36886) 2024-03-06T07:26:31.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.459+0000 7fbd56ffd640 1 -- 172.21.15.83:0/459795642 learned_addr learned my addr 172.21.15.83:0/459795642 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:31.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.460+0000 7fbd57fff640 1 --2- 172.21.15.83:0/459795642 >> v2:172.21.15.183:3300/0 conn(0x7fbd58162940 0x7fbd58164d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:31.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.460+0000 7fbd577fe640 1 -- 172.21.15.83:0/459795642 >> v2:172.21.15.183:3300/0 conn(0x7fbd58162940 msgr2=0x7fbd58164d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:31.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.460+0000 7fbd577fe640 1 --2- 172.21.15.83:0/459795642 >> v2:172.21.15.183:3300/0 conn(0x7fbd58162940 0x7fbd58164d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:31.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.460+0000 7fbd577fe640 1 -- 172.21.15.83:0/459795642 >> v1:172.21.15.83:6789/0 conn(0x7fbd580ce980 legacy=0x7fbd580cee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:31.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.460+0000 7fbd567fc640 1 -- 172.21.15.83:0/459795642 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 367003542 0 0) 0x7fbd580ccbd0 con 0x7fbd580ce980 2024-03-06T07:26:31.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.460+0000 7fbd577fe640 1 -- 172.21.15.83:0/459795642 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd580b40c0 con 0x7fbd580ce5a0 2024-03-06T07:26:31.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.460+0000 7fbd577fe640 1 --2- 172.21.15.83:0/459795642 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd580ce5a0 0x7fbd580b4bf0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7fbd5815b330 tx=0x7fbd4802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3f775162674e026 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:31.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.461+0000 7fbd567fc640 1 -- 172.21.15.83:0/459795642 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd480315a0 con 0x7fbd580ce5a0 2024-03-06T07:26:31.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.461+0000 7fbd567fc640 1 -- 172.21.15.83:0/459795642 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd48037070 con 0x7fbd580ce5a0 2024-03-06T07:26:31.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.461+0000 7fbd567fc640 1 -- 172.21.15.83:0/459795642 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd48031960 con 0x7fbd580ce5a0 2024-03-06T07:26:31.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.461+0000 7fbd5defb640 1 -- 172.21.15.83:0/459795642 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd580ce5a0 msgr2=0x7fbd580b4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:31.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.461+0000 7fbd5defb640 1 --2- 172.21.15.83:0/459795642 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd580ce5a0 0x7fbd580b4bf0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7fbd5815b330 tx=0x7fbd4802b840 comp rx=0 tx=0).stop 2024-03-06T07:26:31.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.461+0000 7fbd5defb640 1 -- 172.21.15.83:0/459795642 shutdown_connections 2024-03-06T07:26:31.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.461+0000 7fbd5defb640 1 --2- 172.21.15.83:0/459795642 >> v2:172.21.15.183:3300/0 conn(0x7fbd58162940 0x7fbd58164d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:31.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.461+0000 7fbd5defb640 1 --2- 172.21.15.83:0/459795642 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd580ce5a0 0x7fbd580b4bf0 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:31.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.461+0000 7fbd5defb640 1 -- 172.21.15.83:0/459795642 >> 172.21.15.83:0/459795642 conn(0x7fbd5806b160 msgr2=0x7fbd5806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:31.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.461+0000 7fbd5defb640 1 -- 172.21.15.83:0/459795642 shutdown_connections 2024-03-06T07:26:31.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.461+0000 7fbd5defb640 1 -- 172.21.15.83:0/459795642 wait complete. 2024-03-06T07:26:31.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.462+0000 7fbd5defb640 1 Processor -- start 2024-03-06T07:26:31.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.463+0000 7fbd5defb640 1 -- start start 2024-03-06T07:26:31.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.463+0000 7fbd5defb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbd580ce5a0 0x7fbd5813b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:31.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.463+0000 7fbd5defb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd580ce980 0x7fbd58140610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:31.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.463+0000 7fbd5defb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbd58162940 0x7fbd5806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:31.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.463+0000 7fbd5defb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd580cd030 con 0x7fbd580ce980 2024-03-06T07:26:31.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.463+0000 7fbd5defb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd580cceb0 con 0x7fbd580ce5a0 2024-03-06T07:26:31.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.463+0000 7fbd5defb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd58071d80 con 0x7fbd58162940 2024-03-06T07:26:31.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.463+0000 7fbd56ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd580ce980 0x7fbd58140610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:31.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.463+0000 7fbd56ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd580ce980 0x7fbd58140610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47976/0 (socket says 172.21.15.83:47976) 2024-03-06T07:26:31.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.463+0000 7fbd56ffd640 1 -- 172.21.15.83:0/1468127672 learned_addr learned my addr 172.21.15.83:0/1468127672 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:31.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.464+0000 7fbd577fe640 1 --2- 172.21.15.83:0/1468127672 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbd580ce5a0 0x7fbd5813b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:31.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.464+0000 7fbd57fff640 1 --2- 172.21.15.83:0/1468127672 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbd58162940 0x7fbd5806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:31.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.464+0000 7fbd56ffd640 1 -- 172.21.15.83:0/1468127672 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbd58162940 msgr2=0x7fbd5806d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:31.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.464+0000 7fbd56ffd640 1 --2- 172.21.15.83:0/1468127672 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbd58162940 0x7fbd5806d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:31.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.464+0000 7fbd56ffd640 1 -- 172.21.15.83:0/1468127672 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbd580ce5a0 msgr2=0x7fbd5813b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:31.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.464+0000 7fbd56ffd640 1 --2- 172.21.15.83:0/1468127672 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbd580ce5a0 0x7fbd5813b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:31.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.464+0000 7fbd56ffd640 1 -- 172.21.15.83:0/1468127672 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd58143960 con 0x7fbd580ce980 2024-03-06T07:26:31.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.464+0000 7fbd577fe640 1 --2- 172.21.15.83:0/1468127672 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbd580ce5a0 0x7fbd5813b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:31.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.464+0000 7fbd57fff640 1 --2- 172.21.15.83:0/1468127672 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbd58162940 0x7fbd5806d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:31.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.464+0000 7fbd56ffd640 1 --2- 172.21.15.83:0/1468127672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd580ce980 0x7fbd58140610 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c00c4a0 tx=0x7fbd4c00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:31.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.464+0000 7fbd3ffff640 1 -- 172.21.15.83:0/1468127672 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c018070 con 0x7fbd580ce980 2024-03-06T07:26:31.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.464+0000 7fbd3ffff640 1 -- 172.21.15.83:0/1468127672 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd4c0074e0 con 0x7fbd580ce980 2024-03-06T07:26:31.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.464+0000 7fbd5defb640 1 -- 172.21.15.83:0/1468127672 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd581430c0 con 0x7fbd580ce980 2024-03-06T07:26:31.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.464+0000 7fbd3ffff640 1 -- 172.21.15.83:0/1468127672 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c004a20 con 0x7fbd580ce980 2024-03-06T07:26:31.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.464+0000 7fbd5defb640 1 -- 172.21.15.83:0/1468127672 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd581550e0 con 0x7fbd580ce980 2024-03-06T07:26:31.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.465+0000 7fbd5defb640 1 -- 172.21.15.83:0/1468127672 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd581434b0 con 0x7fbd580ce980 2024-03-06T07:26:31.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.465+0000 7fbd3ffff640 1 -- 172.21.15.83:0/1468127672 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbd4c01b070 con 0x7fbd580ce980 2024-03-06T07:26:31.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.466+0000 7fbd3ffff640 1 --2- 172.21.15.83:0/1468127672 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbd4406b4d0 0x7fbd4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:31.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.466+0000 7fbd577fe640 1 --2- 172.21.15.83:0/1468127672 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbd4406b4d0 0x7fbd4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:31.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.466+0000 7fbd3ffff640 1 -- 172.21.15.83:0/1468127672 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fbd4c0894b0 con 0x7fbd580ce980 2024-03-06T07:26:31.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.466+0000 7fbd577fe640 1 --2- 172.21.15.83:0/1468127672 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbd4406b4d0 0x7fbd4406d990 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fbd48036000 tx=0x7fbd4803d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:31.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.470+0000 7fbd3ffff640 1 -- 172.21.15.83:0/1468127672 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbd4c058360 con 0x7fbd580ce980 2024-03-06T07:26:31.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.681+0000 7fbd5defb640 1 -- 172.21.15.83:0/1468127672 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd58069ee0 con 0x7fbd580ce980 2024-03-06T07:26:31.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.681+0000 7fbd3ffff640 1 -- 172.21.15.83:0/1468127672 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4976 (secure 0 0 0) 0x7fbd4c057c60 con 0x7fbd580ce980 2024-03-06T07:26:31.683 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:31.684 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":51,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6023,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/601236077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":601236077},{"type":"v1","addr":"172.21.15.183:6833","nonce":601236077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6029,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1435055961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1435055961},{"type":"v1","addr":"172.21.15.183:6835","nonce":1435055961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":7572,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/2726045966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2726045966},{"type":"v1","addr":"172.21.15.83:6836","nonce":2726045966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.064335+0000","modified":"2024-03-06T07:26:29.806098+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7575":{"gid":7575,"name":"a","rank":0,"incarnation":49,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.83:6837/3254575964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":3254575964},{"type":"v1","addr":"172.21.15.83:6837","nonce":3254575964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-03-06T07:26:31.684 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 51 2024-03-06T07:26:31.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.683+0000 7fbd5defb640 1 -- 172.21.15.83:0/1468127672 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbd4406b4d0 msgr2=0x7fbd4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:31.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.683+0000 7fbd5defb640 1 --2- 172.21.15.83:0/1468127672 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbd4406b4d0 0x7fbd4406d990 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fbd48036000 tx=0x7fbd4803d040 comp rx=0 tx=0).stop 2024-03-06T07:26:31.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.683+0000 7fbd5defb640 1 -- 172.21.15.83:0/1468127672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd580ce980 msgr2=0x7fbd58140610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:31.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.683+0000 7fbd5defb640 1 --2- 172.21.15.83:0/1468127672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd580ce980 0x7fbd58140610 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c00c4a0 tx=0x7fbd4c00c960 comp rx=0 tx=0).stop 2024-03-06T07:26:31.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.683+0000 7fbd5defb640 1 -- 172.21.15.83:0/1468127672 shutdown_connections 2024-03-06T07:26:31.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.683+0000 7fbd5defb640 1 --2- 172.21.15.83:0/1468127672 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbd4406b4d0 0x7fbd4406d990 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:31.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.683+0000 7fbd5defb640 1 --2- 172.21.15.83:0/1468127672 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbd58162940 0x7fbd5806d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:31.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.683+0000 7fbd5defb640 1 --2- 172.21.15.83:0/1468127672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd580ce980 0x7fbd58140610 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:31.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.684+0000 7fbd5defb640 1 --2- 172.21.15.83:0/1468127672 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbd580ce5a0 0x7fbd5813b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:31.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.684+0000 7fbd5defb640 1 -- 172.21.15.83:0/1468127672 >> 172.21.15.83:0/1468127672 conn(0x7fbd5806b160 msgr2=0x7fbd5815d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:31.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.684+0000 7fbd5defb640 1 -- 172.21.15.83:0/1468127672 shutdown_connections 2024-03-06T07:26:31.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:31.684+0000 7fbd5defb640 1 -- 172.21.15.83:0/1468127672 wait complete. 2024-03-06T07:26:32.099 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 51, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:26:19.064335+0000', 'modified': '2024-03-06T07:26:29.806098+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 98, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7575}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7575': {'gid': 7575, 'name': 'a', 'rank': 0, 'incarnation': 49, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.83:6837/3254575964', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 3254575964}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 3254575964}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:26:32.099 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7575:up:rejoin 2024-03-06T07:26:33.100 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:33.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.199+0000 7fa8dd58a640 1 Processor -- start 2024-03-06T07:26:33.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.199+0000 7fa8dd58a640 1 -- start start 2024-03-06T07:26:33.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.200+0000 7fa8dd58a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fa8d8161760 0x7fa8d80b4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:33.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.200+0000 7fa8dd58a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8d80b5240 0x7fa8d80d1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:33.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.200+0000 7fa8dd58a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8d8161b40 con 0x7fa8d80d2070 2024-03-06T07:26:33.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.200+0000 7fa8dd58a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8d80d2520 con 0x7fa8d8161760 2024-03-06T07:26:33.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.200+0000 7fa8dd58a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d80c8ac0 con 0x7fa8d80b5240 2024-03-06T07:26:33.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.200+0000 7fa8d67fc640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8d80b5240 0x7fa8d80d1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:33.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.200+0000 7fa8d6ffd640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fa8d8161760 0x7fa8d80b4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:33.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.200+0000 7fa8d67fc640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8d80b5240 0x7fa8d80d1b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:55222/0 (socket says 172.21.15.83:55222) 2024-03-06T07:26:33.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.200+0000 7fa8d67fc640 1 -- 172.21.15.83:0/1300154106 learned_addr learned my addr 172.21.15.83:0/1300154106 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:33.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.200+0000 7fa8d77fe640 1 --1- 172.21.15.83:0/1300154106 >> v1:172.21.15.183:6789/0 conn(0x7fa8d80d2070 0x7fa8d80d0790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:34028/0 (socket says 172.21.15.83:34028) 2024-03-06T07:26:33.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.201+0000 7fa8d6ffd640 1 -- 172.21.15.83:0/1300154106 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8d80b5240 msgr2=0x7fa8d80d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:33.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.201+0000 7fa8d6ffd640 1 --2- 172.21.15.83:0/1300154106 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8d80b5240 0x7fa8d80d1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.201+0000 7fa8d6ffd640 1 -- 172.21.15.83:0/1300154106 >> v1:172.21.15.183:6789/0 conn(0x7fa8d80d2070 legacy=0x7fa8d80d0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:33.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.201+0000 7fa8d6ffd640 1 -- 172.21.15.83:0/1300154106 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d80d27e0 con 0x7fa8d8161760 2024-03-06T07:26:33.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.201+0000 7fa8d67fc640 1 --2- 172.21.15.83:0/1300154106 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8d80b5240 0x7fa8d80d1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:33.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.201+0000 7fa8d6ffd640 1 --2- 172.21.15.83:0/1300154106 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8d8161760 0x7fa8d80b4cd0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0009d90 tx=0x7fa8d002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0780d01d15f3f51 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:33.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.202+0000 7fa8d5ffb640 1 -- 172.21.15.83:0/1300154106 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d0039070 con 0x7fa8d8161760 2024-03-06T07:26:33.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.202+0000 7fa8d5ffb640 1 -- 172.21.15.83:0/1300154106 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8d002cc50 con 0x7fa8d8161760 2024-03-06T07:26:33.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.202+0000 7fa8d5ffb640 1 -- 172.21.15.83:0/1300154106 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d0034440 con 0x7fa8d8161760 2024-03-06T07:26:33.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.202+0000 7fa8dd58a640 1 -- 172.21.15.83:0/1300154106 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8d8161760 msgr2=0x7fa8d80b4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:33.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.202+0000 7fa8dd58a640 1 --2- 172.21.15.83:0/1300154106 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8d8161760 0x7fa8d80b4cd0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0009d90 tx=0x7fa8d002bea0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.202+0000 7fa8dd58a640 1 -- 172.21.15.83:0/1300154106 shutdown_connections 2024-03-06T07:26:33.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.202+0000 7fa8dd58a640 1 --2- 172.21.15.83:0/1300154106 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8d80b5240 0x7fa8d80d1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.202+0000 7fa8dd58a640 1 --2- 172.21.15.83:0/1300154106 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8d8161760 0x7fa8d80b4cd0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.202+0000 7fa8dd58a640 1 -- 172.21.15.83:0/1300154106 >> 172.21.15.83:0/1300154106 conn(0x7fa8d80d3f60 msgr2=0x7fa8d80d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:33.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.202+0000 7fa8dd58a640 1 -- 172.21.15.83:0/1300154106 shutdown_connections 2024-03-06T07:26:33.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.202+0000 7fa8dd58a640 1 -- 172.21.15.83:0/1300154106 wait complete. 2024-03-06T07:26:33.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.204+0000 7fa8dd58a640 1 Processor -- start 2024-03-06T07:26:33.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.204+0000 7fa8dd58a640 1 -- start start 2024-03-06T07:26:33.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.206+0000 7fa8dd58a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8d80b5240 0x7fa8d81f7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:33.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.206+0000 7fa8dd58a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8d80d2070 0x7fa8d8159570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:33.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.206+0000 7fa8dd58a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8d8161760 0x7fa8d815d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:33.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.206+0000 7fa8dd58a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d80cfd80 con 0x7fa8d80d2070 2024-03-06T07:26:33.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.206+0000 7fa8dd58a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d80cf860 con 0x7fa8d8161760 2024-03-06T07:26:33.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.206+0000 7fa8dd58a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d80d2ac0 con 0x7fa8d80b5240 2024-03-06T07:26:33.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.206+0000 7fa8d67fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8d80d2070 0x7fa8d8159570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:33.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.206+0000 7fa8d67fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8d80d2070 0x7fa8d8159570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47982/0 (socket says 172.21.15.83:47982) 2024-03-06T07:26:33.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.206+0000 7fa8d67fc640 1 -- 172.21.15.83:0/1264547632 learned_addr learned my addr 172.21.15.83:0/1264547632 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:33.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.206+0000 7fa8d6ffd640 1 --2- 172.21.15.83:0/1264547632 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8d80b5240 0x7fa8d81f7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:33.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.206+0000 7fa8d77fe640 1 --2- 172.21.15.83:0/1264547632 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8d8161760 0x7fa8d815d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:33.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.206+0000 7fa8d67fc640 1 -- 172.21.15.83:0/1264547632 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8d80b5240 msgr2=0x7fa8d81f7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:33.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.206+0000 7fa8d67fc640 1 --2- 172.21.15.83:0/1264547632 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8d80b5240 0x7fa8d81f7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.206+0000 7fa8d67fc640 1 -- 172.21.15.83:0/1264547632 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8d8161760 msgr2=0x7fa8d815d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:33.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.206+0000 7fa8d67fc640 1 --2- 172.21.15.83:0/1264547632 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8d8161760 0x7fa8d815d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.206+0000 7fa8d67fc640 1 -- 172.21.15.83:0/1264547632 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d80a0330 con 0x7fa8d80d2070 2024-03-06T07:26:33.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.206+0000 7fa8d67fc640 1 --2- 172.21.15.83:0/1264547632 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8d80d2070 0x7fa8d8159570 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7fa8c000e9a0 tx=0x7fa8c000ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:33.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.207+0000 7fa8cf7fe640 1 -- 172.21.15.83:0/1264547632 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c0017070 con 0x7fa8d80d2070 2024-03-06T07:26:33.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.207+0000 7fa8cf7fe640 1 -- 172.21.15.83:0/1264547632 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c000ca00 con 0x7fa8d80d2070 2024-03-06T07:26:33.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.207+0000 7fa8dd58a640 1 -- 172.21.15.83:0/1264547632 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8d809e960 con 0x7fa8d80d2070 2024-03-06T07:26:33.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.207+0000 7fa8cf7fe640 1 -- 172.21.15.83:0/1264547632 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c000cdb0 con 0x7fa8d80d2070 2024-03-06T07:26:33.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.207+0000 7fa8dd58a640 1 -- 172.21.15.83:0/1264547632 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8d815a430 con 0x7fa8d80d2070 2024-03-06T07:26:33.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.208+0000 7fa8dd58a640 1 -- 172.21.15.83:0/1264547632 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8d8069ee0 con 0x7fa8d80d2070 2024-03-06T07:26:33.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.208+0000 7fa8cf7fe640 1 -- 172.21.15.83:0/1264547632 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa8c001a070 con 0x7fa8d80d2070 2024-03-06T07:26:33.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.209+0000 7fa8cf7fe640 1 --2- 172.21.15.83:0/1264547632 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa8b806b500 0x7fa8b806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:33.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.209+0000 7fa8cf7fe640 1 -- 172.21.15.83:0/1264547632 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fa8c008a090 con 0x7fa8d80d2070 2024-03-06T07:26:33.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.209+0000 7fa8d6ffd640 1 --2- 172.21.15.83:0/1264547632 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa8b806b500 0x7fa8b806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:33.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.209+0000 7fa8d6ffd640 1 --2- 172.21.15.83:0/1264547632 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa8b806b500 0x7fa8b806d9c0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0036000 tx=0x7fa8d0030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:33.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.213+0000 7fa8cf7fe640 1 -- 172.21.15.83:0/1264547632 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa8c0058860 con 0x7fa8d80d2070 2024-03-06T07:26:33.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.427+0000 7fa8dd58a640 1 -- 172.21.15.83:0/1264547632 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8d815ff20 con 0x7fa8d80d2070 2024-03-06T07:26:33.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.427+0000 7fa8cf7fe640 1 -- 172.21.15.83:0/1264547632 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+4983 (secure 0 0 0) 0x7fa8c005c900 con 0x7fa8d80d2070 2024-03-06T07:26:33.429 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:33.429 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":53,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6023,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/601236077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":601236077},{"type":"v1","addr":"172.21.15.183:6833","nonce":601236077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6029,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1435055961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1435055961},{"type":"v1","addr":"172.21.15.183:6835","nonce":1435055961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":7572,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/2726045966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2726045966},{"type":"v1","addr":"172.21.15.83:6836","nonce":2726045966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.064335+0000","modified":"2024-03-06T07:26:32.814889+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7575":{"gid":7575,"name":"a","rank":0,"incarnation":49,"state":"up:active","state_seq":5,"addr":"172.21.15.83:6837/3254575964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":3254575964},{"type":"v1","addr":"172.21.15.83:6837","nonce":3254575964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7575,"qdb_cluster":[7575]},"id":7}]} 2024-03-06T07:26:33.430 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 53 2024-03-06T07:26:33.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.430+0000 7fa8dd58a640 1 -- 172.21.15.83:0/1264547632 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa8b806b500 msgr2=0x7fa8b806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:33.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.430+0000 7fa8dd58a640 1 --2- 172.21.15.83:0/1264547632 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa8b806b500 0x7fa8b806d9c0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0036000 tx=0x7fa8d0030000 comp rx=0 tx=0).stop 2024-03-06T07:26:33.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.430+0000 7fa8dd58a640 1 -- 172.21.15.83:0/1264547632 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8d80d2070 msgr2=0x7fa8d8159570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:33.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.430+0000 7fa8dd58a640 1 --2- 172.21.15.83:0/1264547632 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8d80d2070 0x7fa8d8159570 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7fa8c000e9a0 tx=0x7fa8c000ee60 comp rx=0 tx=0).stop 2024-03-06T07:26:33.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.430+0000 7fa8dd58a640 1 -- 172.21.15.83:0/1264547632 shutdown_connections 2024-03-06T07:26:33.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.430+0000 7fa8dd58a640 1 --2- 172.21.15.83:0/1264547632 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa8b806b500 0x7fa8b806d9c0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.430+0000 7fa8dd58a640 1 --2- 172.21.15.83:0/1264547632 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8d8161760 0x7fa8d815d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.430+0000 7fa8dd58a640 1 --2- 172.21.15.83:0/1264547632 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8d80d2070 0x7fa8d8159570 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.430+0000 7fa8dd58a640 1 --2- 172.21.15.83:0/1264547632 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8d80b5240 0x7fa8d81f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.430+0000 7fa8dd58a640 1 -- 172.21.15.83:0/1264547632 >> 172.21.15.83:0/1264547632 conn(0x7fa8d80d3f60 msgr2=0x7fa8d8160970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:33.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.430+0000 7fa8dd58a640 1 -- 172.21.15.83:0/1264547632 shutdown_connections 2024-03-06T07:26:33.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.430+0000 7fa8dd58a640 1 -- 172.21.15.83:0/1264547632 wait complete. 2024-03-06T07:26:33.805 INFO:tasks.ceph.mds.a.smithi083.stderr:2024-03-06T07:26:33.803+0000 7f5ada039640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:26:33.845 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 53, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:26:19.064335+0000', 'modified': '2024-03-06T07:26:32.814889+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 98, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7575}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7575': {'gid': 7575, 'name': 'a', 'rank': 0, 'incarnation': 49, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.83:6837/3254575964', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 3254575964}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 3254575964}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7575, 'qdb_cluster': [7575]} 2024-03-06T07:26:33.845 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:26:33.846 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:26:33.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.946+0000 7f8bc9313640 1 Processor -- start 2024-03-06T07:26:33.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.946+0000 7f8bc9313640 1 -- start start 2024-03-06T07:26:33.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.947+0000 7f8bc9313640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8bc4155560 0x7f8bc415b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:33.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.947+0000 7f8bc9313640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f8bc415e8f0 0x7f8bc4071c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:33.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.947+0000 7f8bc9313640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bc415f0d0 con 0x7f8bc40721d0 2024-03-06T07:26:33.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.947+0000 7f8bc9313640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bc415f3b0 con 0x7f8bc415e8f0 2024-03-06T07:26:33.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.947+0000 7f8bc9313640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc415f690 con 0x7f8bc4155560 2024-03-06T07:26:33.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.947+0000 7f8bba7fc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f8bc415e8f0 0x7f8bc4071c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:33.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.947+0000 7f8bba7fc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f8bc415e8f0 0x7f8bc4071c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47984/0 (socket says 172.21.15.83:47984) 2024-03-06T07:26:33.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.947+0000 7f8bba7fc640 1 -- 172.21.15.83:0/3012863911 learned_addr learned my addr 172.21.15.83:0/3012863911 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:33.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.948+0000 7f8bc2ffd640 1 --2- 172.21.15.83:0/3012863911 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8bc4155560 0x7f8bc415b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:33.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.948+0000 7f8bba7fc640 1 -- 172.21.15.83:0/3012863911 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8bc4155560 msgr2=0x7f8bc415b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:33.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.948+0000 7f8bba7fc640 1 --2- 172.21.15.83:0/3012863911 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8bc4155560 0x7f8bc415b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.948+0000 7f8bba7fc640 1 -- 172.21.15.83:0/3012863911 >> v1:172.21.15.183:6789/0 conn(0x7f8bc40721d0 legacy=0x7f8bc406da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:33.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.948+0000 7f8bba7fc640 1 -- 172.21.15.83:0/3012863911 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc40b3da0 con 0x7f8bc415e8f0 2024-03-06T07:26:33.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.948+0000 7f8bba7fc640 1 --2- 172.21.15.83:0/3012863911 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8bc415e8f0 0x7f8bc4071c90 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f8bac009870 tx=0x7f8bac02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2fa791aa862fa0f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:33.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.949+0000 7f8bc27fc640 1 -- 172.21.15.83:0/3012863911 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bac034070 con 0x7f8bc415e8f0 2024-03-06T07:26:33.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.949+0000 7f8bc27fc640 1 -- 172.21.15.83:0/3012863911 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bac030840 con 0x7f8bc415e8f0 2024-03-06T07:26:33.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.949+0000 7f8bc27fc640 1 -- 172.21.15.83:0/3012863911 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bac030c10 con 0x7f8bc415e8f0 2024-03-06T07:26:33.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.949+0000 7f8bc9313640 1 -- 172.21.15.83:0/3012863911 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8bc415e8f0 msgr2=0x7f8bc4071c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:33.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.949+0000 7f8bc9313640 1 --2- 172.21.15.83:0/3012863911 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8bc415e8f0 0x7f8bc4071c90 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f8bac009870 tx=0x7f8bac02b980 comp rx=0 tx=0).stop 2024-03-06T07:26:33.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.949+0000 7f8bc9313640 1 -- 172.21.15.83:0/3012863911 shutdown_connections 2024-03-06T07:26:33.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.949+0000 7f8bc9313640 1 --2- 172.21.15.83:0/3012863911 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8bc415e8f0 0x7f8bc4071c90 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.949+0000 7f8bc9313640 1 --2- 172.21.15.83:0/3012863911 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8bc4155560 0x7f8bc415b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.949+0000 7f8bc9313640 1 -- 172.21.15.83:0/3012863911 >> 172.21.15.83:0/3012863911 conn(0x7f8bc40b2890 msgr2=0x7f8bc40b2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:33.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.949+0000 7f8bc9313640 1 -- 172.21.15.83:0/3012863911 shutdown_connections 2024-03-06T07:26:33.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.949+0000 7f8bc9313640 1 -- 172.21.15.83:0/3012863911 wait complete. 2024-03-06T07:26:33.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.950+0000 7f8bc9313640 1 Processor -- start 2024-03-06T07:26:33.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.951+0000 7f8bc9313640 1 -- start start 2024-03-06T07:26:33.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.951+0000 7f8bc9313640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8bc40721d0 0x7f8bc415cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:33.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.951+0000 7f8bc9313640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8bc4155560 0x7f8bc413eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:33.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.951+0000 7f8bc9313640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8bc415e8f0 0x7f8bc40b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:33.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.951+0000 7f8bc9313640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc40b5450 con 0x7f8bc4155560 2024-03-06T07:26:33.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.951+0000 7f8bc9313640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc40b52d0 con 0x7f8bc40721d0 2024-03-06T07:26:33.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.951+0000 7f8bc9313640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc40b55d0 con 0x7f8bc415e8f0 2024-03-06T07:26:33.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.951+0000 7f8bba7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8bc4155560 0x7f8bc413eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:33.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.951+0000 7f8bba7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8bc4155560 0x7f8bc413eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47996/0 (socket says 172.21.15.83:47996) 2024-03-06T07:26:33.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.951+0000 7f8bba7fc640 1 -- 172.21.15.83:0/1433039432 learned_addr learned my addr 172.21.15.83:0/1433039432 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:33.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.951+0000 7f8bc37fe640 1 --2- 172.21.15.83:0/1433039432 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8bc415e8f0 0x7f8bc40b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:33.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.951+0000 7f8bc2ffd640 1 --2- 172.21.15.83:0/1433039432 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8bc40721d0 0x7f8bc415cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:33.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.952+0000 7f8bba7fc640 1 -- 172.21.15.83:0/1433039432 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8bc415e8f0 msgr2=0x7f8bc40b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:33.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.952+0000 7f8bba7fc640 1 --2- 172.21.15.83:0/1433039432 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8bc415e8f0 0x7f8bc40b41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.952+0000 7f8bba7fc640 1 -- 172.21.15.83:0/1433039432 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8bc40721d0 msgr2=0x7f8bc415cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:33.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.952+0000 7f8bba7fc640 1 --2- 172.21.15.83:0/1433039432 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8bc40721d0 0x7f8bc415cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.952+0000 7f8bba7fc640 1 -- 172.21.15.83:0/1433039432 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc40ce2f0 con 0x7f8bc4155560 2024-03-06T07:26:33.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.952+0000 7f8bba7fc640 1 --2- 172.21.15.83:0/1433039432 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8bc4155560 0x7f8bc413eae0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f8bac000c00 tx=0x7f8bac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:33.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.952+0000 7f8bbbfff640 1 -- 172.21.15.83:0/1433039432 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bac034040 con 0x7f8bc4155560 2024-03-06T07:26:33.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.952+0000 7f8bbbfff640 1 -- 172.21.15.83:0/1433039432 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bac039d90 con 0x7f8bc4155560 2024-03-06T07:26:33.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.952+0000 7f8bc9313640 1 -- 172.21.15.83:0/1433039432 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bc40ce5d0 con 0x7f8bc4155560 2024-03-06T07:26:33.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.952+0000 7f8bbbfff640 1 -- 172.21.15.83:0/1433039432 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bac043610 con 0x7f8bc4155560 2024-03-06T07:26:33.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.952+0000 7f8bc9313640 1 -- 172.21.15.83:0/1433039432 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bc40ceb90 con 0x7f8bc4155560 2024-03-06T07:26:33.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.953+0000 7f8bbbfff640 1 -- 172.21.15.83:0/1433039432 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8bac04a060 con 0x7f8bc4155560 2024-03-06T07:26:33.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.954+0000 7f8bbbfff640 1 --2- 172.21.15.83:0/1433039432 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8ba406b500 0x7f8ba406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:33.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.954+0000 7f8bbbfff640 1 -- 172.21.15.83:0/1433039432 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f8bac0ac150 con 0x7f8bc4155560 2024-03-06T07:26:33.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.954+0000 7f8bc2ffd640 1 --2- 172.21.15.83:0/1433039432 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8ba406b500 0x7f8ba406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:33.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.954+0000 7f8bc2ffd640 1 --2- 172.21.15.83:0/1433039432 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8ba406b500 0x7f8ba406d9c0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f8bb4006fd0 tx=0x7f8bb4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:33.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.955+0000 7f8bcbfda740 1 Processor -- start 2024-03-06T07:26:33.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.955+0000 7f8bcbfda740 1 -- start start 2024-03-06T07:26:33.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.956+0000 7f8bcbfda740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cbda722870 0x55cbda722c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:33.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.956+0000 7f8bcbfda740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55cbda7268c0 0x55cbda728360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:33.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.956+0000 7f8bcbfda740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbda752600 0x55cbda752a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:33.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.956+0000 7f8bcbfda740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cbda68c6b0 con 0x55cbda752600 2024-03-06T07:26:33.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.956+0000 7f8bcbfda740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cbda635a60 con 0x55cbda722870 2024-03-06T07:26:33.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.956+0000 7f8bcbfda740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cbda688750 con 0x55cbda7268c0 2024-03-06T07:26:33.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.956+0000 7f8bc37fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbda752600 0x55cbda752a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:33.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.956+0000 7f8bc37fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbda752600 0x55cbda752a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47998/0 (socket says 172.21.15.83:47998) 2024-03-06T07:26:33.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.956+0000 7f8bc37fe640 1 -- 172.21.15.83:0/3536054270 learned_addr learned my addr 172.21.15.83:0/3536054270 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:33.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.956+0000 7f8bc37fe640 1 -- 172.21.15.83:0/3536054270 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55cbda7268c0 msgr2=0x55cbda728360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:33.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.956+0000 7f8bc2ffd640 1 --2- 172.21.15.83:0/3536054270 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cbda722870 0x55cbda722c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:33.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.956+0000 7f8bc37fe640 1 --2- 172.21.15.83:0/3536054270 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55cbda7268c0 0x55cbda728360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.956+0000 7f8bc37fe640 1 -- 172.21.15.83:0/3536054270 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cbda722870 msgr2=0x55cbda722c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:33.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.956+0000 7f8bc37fe640 1 --2- 172.21.15.83:0/3536054270 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cbda722870 0x55cbda722c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.956+0000 7f8bc37fe640 1 -- 172.21.15.83:0/3536054270 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cbda653f30 con 0x55cbda752600 2024-03-06T07:26:33.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.956+0000 7f8bc37fe640 1 --2- 172.21.15.83:0/3536054270 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbda752600 0x55cbda752a80 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc001f20 tx=0x7f8bbc001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1320990535e7838 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:33.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.956+0000 7f8bb8ff9640 1 -- 172.21.15.83:0/3536054270 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bbc00fe50 con 0x55cbda752600 2024-03-06T07:26:33.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.956+0000 7f8bb8ff9640 1 -- 172.21.15.83:0/3536054270 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bbc006b50 con 0x55cbda752600 2024-03-06T07:26:33.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.956+0000 7f8bb8ff9640 1 -- 172.21.15.83:0/3536054270 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bbc006d00 con 0x55cbda752600 2024-03-06T07:26:33.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.957+0000 7f8bcbfda740 1 -- 172.21.15.83:0/3536054270 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbda752600 msgr2=0x55cbda752a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:33.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.957+0000 7f8bcbfda740 1 --2- 172.21.15.83:0/3536054270 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbda752600 0x55cbda752a80 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc001f20 tx=0x7f8bbc001670 comp rx=0 tx=0).stop 2024-03-06T07:26:33.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.957+0000 7f8bcbfda740 1 -- 172.21.15.83:0/3536054270 shutdown_connections 2024-03-06T07:26:33.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.957+0000 7f8bcbfda740 1 --2- 172.21.15.83:0/3536054270 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbda752600 0x55cbda752a80 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.957+0000 7f8bcbfda740 1 --2- 172.21.15.83:0/3536054270 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55cbda7268c0 0x55cbda728360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.957+0000 7f8bcbfda740 1 --2- 172.21.15.83:0/3536054270 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cbda722870 0x55cbda722c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.957+0000 7f8bcbfda740 1 -- 172.21.15.83:0/3536054270 >> 172.21.15.83:0/3536054270 conn(0x55cbda7248f0 msgr2=0x55cbda7261b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:33.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.957+0000 7f8bcbfda740 1 -- 172.21.15.83:0/3536054270 shutdown_connections 2024-03-06T07:26:33.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.957+0000 7f8bcbfda740 1 -- 172.21.15.83:0/3536054270 wait complete. 2024-03-06T07:26:33.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.958+0000 7f8bcbfda740 1 -- start start 2024-03-06T07:26:33.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.958+0000 7f8bcbfda740 1 Processor -- start 2024-03-06T07:26:33.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.958+0000 7f8bcbfda740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbda752600 0x55cbda74d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:33.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.958+0000 7f8bcbfda740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cbda749070 0x55cbda74b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:33.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.958+0000 7f8bcbfda740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55cbda754130 0x55cbda74b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:33.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.958+0000 7f8bcbfda740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cbda726d50 con 0x55cbda752600 2024-03-06T07:26:33.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.958+0000 7f8bcbfda740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cbda648ea0 con 0x55cbda749070 2024-03-06T07:26:33.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.958+0000 7f8bcbfda740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cbda727030 con 0x55cbda754130 2024-03-06T07:26:33.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.959+0000 7f8bcbfda740 10 client.0 _ll_register_callbacks cb 0x55cbda648870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:26:33.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.959+0000 7f8bc2ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbda752600 0x55cbda74d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:33.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.959+0000 7f8bc2ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbda752600 0x55cbda74d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48008/0 (socket says 172.21.15.83:48008) 2024-03-06T07:26:33.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.959+0000 7f8bc2ffd640 1 -- 172.21.15.83:0/3636196112 learned_addr learned my addr 172.21.15.83:0/3636196112 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:33.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.959+0000 7f8bc37fe640 1 --2- 172.21.15.83:0/3636196112 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55cbda754130 0x55cbda74b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:33.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.959+0000 7f8bc2ffd640 1 -- 172.21.15.83:0/3636196112 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55cbda754130 msgr2=0x55cbda74b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:33.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.959+0000 7f8bba7fc640 1 --2- 172.21.15.83:0/3636196112 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cbda749070 0x55cbda74b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:33.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.959+0000 7f8bc2ffd640 1 --2- 172.21.15.83:0/3636196112 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55cbda754130 0x55cbda74b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.959+0000 7f8bc2ffd640 1 -- 172.21.15.83:0/3636196112 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cbda749070 msgr2=0x55cbda74b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:33.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.959+0000 7f8bc2ffd640 1 --2- 172.21.15.83:0/3636196112 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cbda749070 0x55cbda74b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.959+0000 7f8bc2ffd640 1 -- 172.21.15.83:0/3636196112 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cbda649270 con 0x55cbda752600 2024-03-06T07:26:33.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.959+0000 7f8bc2ffd640 1 --2- 172.21.15.83:0/3636196112 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbda752600 0x55cbda74d2a0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f8bb4016800 tx=0x7f8bb4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8c31ed1412ee7c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:33.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.959+0000 7f8ba1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-06T07:26:33.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.959+0000 7f8ba1ffb640 1 -- 172.21.15.83:0/3636196112 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb40149b0 con 0x55cbda752600 2024-03-06T07:26:33.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.959+0000 7f8ba1ffb640 1 -- 172.21.15.83:0/3636196112 <== mon.0 v2:172.21.15.83:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f8bb4014b50 con 0x55cbda752600 2024-03-06T07:26:33.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.959+0000 7f8bcbfda740 1 -- 172.21.15.83:0/3636196112 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8bbc006b50 con 0x55cbda752600 2024-03-06T07:26:33.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.959+0000 7f8ba1ffb640 1 -- 172.21.15.83:0/3636196112 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8bb402f050 con 0x55cbda752600 2024-03-06T07:26:33.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.959+0000 7f8bcbfda740 10 client.7611 fetch_fsmap learned FSMap version 53 2024-03-06T07:26:33.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.959+0000 7f8bcbfda740 1 -- 172.21.15.83:0/3636196112 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=2+}) v3 -- 0x55cbda3a12d0 con 0x55cbda752600 2024-03-06T07:26:33.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.960+0000 7f8ba1ffb640 1 -- 172.21.15.83:0/3636196112 <== mon.0 v2:172.21.15.83:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bbc006b50 con 0x55cbda752600 2024-03-06T07:26:33.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.960+0000 7f8ba1ffb640 1 -- 172.21.15.83:0/3636196112 <== mon.0 v2:172.21.15.83:3300/0 5 ==== fsmap(e 53) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f8bb4032020 con 0x55cbda752600 2024-03-06T07:26:33.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.960+0000 7f8bcbfda740 10 client.7611 fetch_fsmap finished waiting for FSMap version 53 2024-03-06T07:26:33.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.960+0000 7f8bcbfda740 10 client.7611 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:26:33.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.960+0000 7f8bcbfda740 20 client.7611 populate_metadata read hostname 'smithi083' 2024-03-06T07:26:33.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.960+0000 7f8bcbfda740 1 --2- 172.21.15.83:0/3636196112 >> [v2:172.21.15.83:6835/3254575964,v1:172.21.15.83:6837/3254575964] conn(0x55cbda7474b0 0x55cbda78c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:33.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.960+0000 7f8bcbfda740 4 client.7611 mds_command: new command op to 7575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:26:33.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.960+0000 7f8bcbfda740 1 -- 172.21.15.83:0/3636196112 --> [v2:172.21.15.83:6835/3254575964,v1:172.21.15.83:6837/3254575964] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cbda699bf0 con 0x55cbda7474b0 2024-03-06T07:26:33.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.960+0000 7f8bba7fc640 1 --2- 172.21.15.83:0/3636196112 >> [v2:172.21.15.83:6835/3254575964,v1:172.21.15.83:6837/3254575964] conn(0x55cbda7474b0 0x55cbda78c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:33.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.960+0000 7f8ba1ffb640 10 client.7611 ms_handle_connect on v2:172.21.15.83:6835/3254575964 2024-03-06T07:26:33.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.960+0000 7f8bba7fc640 1 --2- 172.21.15.83:0/3636196112 >> [v2:172.21.15.83:6835/3254575964,v1:172.21.15.83:6837/3254575964] conn(0x55cbda7474b0 0x55cbda78c880 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5c28a6e5715fb32 server_cookie=f16c2f7125905370 in_seq=0 out_seq=0 2024-03-06T07:26:33.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.961+0000 7f8ba1ffb640 1 -- 172.21.15.83:0/3636196112 <== mds.0 v2:172.21.15.83:6835/3254575964 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55cbda699bf0 con 0x55cbda7474b0 2024-03-06T07:26:33.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.961+0000 7f8ba1ffb640 10 client.7611 handle_command_reply: tid=0 2024-03-06T07:26:33.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.961+0000 7f8bcbfda740 1 client.7611 shutdown 2024-03-06T07:26:33.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.961+0000 7f8bcbfda740 1 -- 172.21.15.83:0/3636196112 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbda752600 msgr2=0x55cbda74d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:33.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.961+0000 7f8bcbfda740 1 --2- 172.21.15.83:0/3636196112 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbda752600 0x55cbda74d2a0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f8bb4016800 tx=0x7f8bb4004730 comp rx=0 tx=0).stop 2024-03-06T07:26:33.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.961+0000 7f8bcbfda740 1 -- 172.21.15.83:0/3636196112 shutdown_connections 2024-03-06T07:26:33.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.961+0000 7f8bcbfda740 1 --2- 172.21.15.83:0/3636196112 >> [v2:172.21.15.83:6835/3254575964,v1:172.21.15.83:6837/3254575964] conn(0x55cbda7474b0 0x55cbda78c880 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.961+0000 7f8bcbfda740 1 --2- 172.21.15.83:0/3636196112 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55cbda754130 0x55cbda74b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.961+0000 7f8bcbfda740 1 --2- 172.21.15.83:0/3636196112 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cbda749070 0x55cbda74b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.961+0000 7f8bcbfda740 1 --2- 172.21.15.83:0/3636196112 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbda752600 0x55cbda74d2a0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.961+0000 7f8ba1ffb640 0 client.7611 ms_handle_reset on v2:172.21.15.83:6835/3254575964 2024-03-06T07:26:33.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.961+0000 7f8bcbfda740 1 -- 172.21.15.83:0/3636196112 >> 172.21.15.83:0/3636196112 conn(0x55cbda722870 msgr2=0x55cbda7505c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:33.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.961+0000 7f8bcbfda740 1 -- 172.21.15.83:0/3636196112 shutdown_connections 2024-03-06T07:26:33.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.961+0000 7f8bcbfda740 1 -- 172.21.15.83:0/3636196112 wait complete. 2024-03-06T07:26:33.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.961+0000 7f8bcbfda740 20 client.7611 trim_cache size 0 max 16384 2024-03-06T07:26:33.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.978+0000 7f8bcbfda740 1 Processor -- start 2024-03-06T07:26:33.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.978+0000 7f8bcbfda740 1 -- start start 2024-03-06T07:26:33.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.978+0000 7f8bcbfda740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55cbda7474b0 0x55cbda7506e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:33.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.978+0000 7f8bcbfda740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbda749070 0x55cbda755de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:33.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.978+0000 7f8bcbfda740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cbda752600 0x55cbda7908a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:33.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.978+0000 7f8bcbfda740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc41ff560 con 0x55cbda749070 2024-03-06T07:26:33.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.978+0000 7f8bcbfda740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc40cfae0 con 0x55cbda752600 2024-03-06T07:26:33.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.978+0000 7f8bcbfda740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc40d00e0 con 0x55cbda7474b0 2024-03-06T07:26:33.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.978+0000 7f8bba7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbda749070 0x55cbda755de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:33.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.978+0000 7f8bba7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbda749070 0x55cbda755de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48014/0 (socket says 172.21.15.83:48014) 2024-03-06T07:26:33.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.978+0000 7f8bba7fc640 1 -- 172.21.15.83:0/2764625172 learned_addr learned my addr 172.21.15.83:0/2764625172 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:33.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.979+0000 7f8bc2ffd640 1 --2- 172.21.15.83:0/2764625172 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55cbda7474b0 0x55cbda7506e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:33.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.979+0000 7f8bba7fc640 1 -- 172.21.15.83:0/2764625172 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55cbda7474b0 msgr2=0x55cbda7506e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:33.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.979+0000 7f8bba7fc640 1 --2- 172.21.15.83:0/2764625172 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55cbda7474b0 0x55cbda7506e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.979+0000 7f8bc37fe640 1 --2- 172.21.15.83:0/2764625172 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cbda752600 0x55cbda7908a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:33.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.979+0000 7f8bba7fc640 1 -- 172.21.15.83:0/2764625172 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cbda752600 msgr2=0x55cbda7908a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:33.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.979+0000 7f8bba7fc640 1 --2- 172.21.15.83:0/2764625172 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cbda752600 0x55cbda7908a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.979+0000 7f8bba7fc640 1 -- 172.21.15.83:0/2764625172 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cbda46b6b0 con 0x55cbda749070 2024-03-06T07:26:33.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.979+0000 7f8bc2ffd640 1 --2- 172.21.15.83:0/2764625172 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55cbda7474b0 0x55cbda7506e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:33.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.979+0000 7f8bc37fe640 1 --2- 172.21.15.83:0/2764625172 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cbda752600 0x55cbda7908a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:33.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.979+0000 7f8bba7fc640 1 --2- 172.21.15.83:0/2764625172 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbda749070 0x55cbda755de0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f8bac076d80 tx=0x7f8bac0ac040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fff837de825b1c29 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:33.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.979+0000 7f8ba3fff640 1 -- 172.21.15.83:0/2764625172 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bac082f80 con 0x55cbda749070 2024-03-06T07:26:33.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.979+0000 7f8ba3fff640 1 -- 172.21.15.83:0/2764625172 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bac0a8070 con 0x55cbda749070 2024-03-06T07:26:33.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.979+0000 7f8ba3fff640 1 -- 172.21.15.83:0/2764625172 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bac078c20 con 0x55cbda749070 2024-03-06T07:26:33.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.979+0000 7f8bcbfda740 1 -- 172.21.15.83:0/2764625172 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbda749070 msgr2=0x55cbda755de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:33.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.979+0000 7f8bcbfda740 1 --2- 172.21.15.83:0/2764625172 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbda749070 0x55cbda755de0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f8bac076d80 tx=0x7f8bac0ac040 comp rx=0 tx=0).stop 2024-03-06T07:26:33.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.979+0000 7f8bcbfda740 1 -- 172.21.15.83:0/2764625172 shutdown_connections 2024-03-06T07:26:33.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.979+0000 7f8bcbfda740 1 --2- 172.21.15.83:0/2764625172 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cbda752600 0x55cbda7908a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.980+0000 7f8bcbfda740 1 --2- 172.21.15.83:0/2764625172 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbda749070 0x55cbda755de0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.980+0000 7f8bcbfda740 1 --2- 172.21.15.83:0/2764625172 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55cbda7474b0 0x55cbda7506e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.980+0000 7f8bcbfda740 1 -- 172.21.15.83:0/2764625172 >> 172.21.15.83:0/2764625172 conn(0x55cbda722870 msgr2=0x55cbda743f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:33.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.980+0000 7f8bcbfda740 1 -- 172.21.15.83:0/2764625172 shutdown_connections 2024-03-06T07:26:33.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.980+0000 7f8bcbfda740 1 -- 172.21.15.83:0/2764625172 wait complete. 2024-03-06T07:26:33.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.980+0000 7f8bcbfda740 1 -- start start 2024-03-06T07:26:33.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.980+0000 7f8bcbfda740 1 Processor -- start 2024-03-06T07:26:33.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.981+0000 7f8bcbfda740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55cbda7474b0 0x55cbda787c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:33.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.981+0000 7f8bcbfda740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cbda749070 0x55cbda74a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:33.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.981+0000 7f8bcbfda740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbda752600 0x55cbda755ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:33.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.981+0000 7f8bcbfda740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cbda754650 con 0x55cbda752600 2024-03-06T07:26:33.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.981+0000 7f8bcbfda740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cbda74a5a0 con 0x55cbda749070 2024-03-06T07:26:33.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.981+0000 7f8bcbfda740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cbda7450a0 con 0x55cbda7474b0 2024-03-06T07:26:33.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.981+0000 7f8bcbfda740 10 client.0 _ll_register_callbacks cb 0x7f8bb4007790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:26:33.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.981+0000 7f8bc37fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbda752600 0x55cbda755ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:33.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.981+0000 7f8bc37fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbda752600 0x55cbda755ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48020/0 (socket says 172.21.15.83:48020) 2024-03-06T07:26:33.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.981+0000 7f8bc37fe640 1 -- 172.21.15.83:0/1129362971 learned_addr learned my addr 172.21.15.83:0/1129362971 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:33.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.981+0000 7f8bc2ffd640 1 --2- 172.21.15.83:0/1129362971 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55cbda7474b0 0x55cbda787c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:33.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.981+0000 7f8bba7fc640 1 --2- 172.21.15.83:0/1129362971 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cbda749070 0x55cbda74a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:33.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.981+0000 7f8bc37fe640 1 -- 172.21.15.83:0/1129362971 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55cbda7474b0 msgr2=0x55cbda787c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:33.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.981+0000 7f8bc37fe640 1 --2- 172.21.15.83:0/1129362971 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55cbda7474b0 0x55cbda787c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.981+0000 7f8bc37fe640 1 -- 172.21.15.83:0/1129362971 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cbda749070 msgr2=0x55cbda74a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:33.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.981+0000 7f8bc37fe640 1 --2- 172.21.15.83:0/1129362971 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cbda749070 0x55cbda74a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.981+0000 7f8bc37fe640 1 -- 172.21.15.83:0/1129362971 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cbda747bc0 con 0x55cbda752600 2024-03-06T07:26:33.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.982+0000 7f8bc37fe640 1 --2- 172.21.15.83:0/1129362971 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbda752600 0x55cbda755ce0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc007000 tx=0x7f8bbc001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a987604e476c7ba1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:33.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.982+0000 7f8ba1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-06T07:26:33.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.982+0000 7f8ba1ffb640 1 -- 172.21.15.83:0/1129362971 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bbc023070 con 0x55cbda752600 2024-03-06T07:26:33.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.982+0000 7f8ba1ffb640 1 -- 172.21.15.83:0/1129362971 <== mon.0 v2:172.21.15.83:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f8bbc003b80 con 0x55cbda752600 2024-03-06T07:26:33.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.982+0000 7f8bcbfda740 1 -- 172.21.15.83:0/1129362971 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8bac0a8070 con 0x55cbda752600 2024-03-06T07:26:33.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.982+0000 7f8ba1ffb640 1 -- 172.21.15.83:0/1129362971 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8bbc02e050 con 0x55cbda752600 2024-03-06T07:26:33.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.982+0000 7f8bcbfda740 10 client.7617 fetch_fsmap learned FSMap version 53 2024-03-06T07:26:33.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.982+0000 7f8bcbfda740 1 -- 172.21.15.83:0/1129362971 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=2+}) v3 -- 0x55cbda7ddf80 con 0x55cbda752600 2024-03-06T07:26:33.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.982+0000 7f8ba1ffb640 1 -- 172.21.15.83:0/1129362971 <== mon.0 v2:172.21.15.83:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bac0a8070 con 0x55cbda752600 2024-03-06T07:26:33.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.982+0000 7f8ba1ffb640 1 -- 172.21.15.83:0/1129362971 <== mon.0 v2:172.21.15.83:3300/0 5 ==== fsmap(e 53) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f8bbc031020 con 0x55cbda752600 2024-03-06T07:26:33.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.982+0000 7f8bcbfda740 10 client.7617 fetch_fsmap finished waiting for FSMap version 53 2024-03-06T07:26:33.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.982+0000 7f8bcbfda740 10 client.7617 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:26:33.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.982+0000 7f8bcbfda740 20 client.7617 populate_metadata read hostname 'smithi083' 2024-03-06T07:26:33.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.982+0000 7f8bcbfda740 1 --2- 172.21.15.83:0/1129362971 >> [v2:172.21.15.83:6835/3254575964,v1:172.21.15.83:6837/3254575964] conn(0x55cbda754130 0x55cbda7512b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:33.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.982+0000 7f8bcbfda740 4 client.7617 mds_command: new command op to 7575 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:26:33.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.982+0000 7f8bcbfda740 1 -- 172.21.15.83:0/1129362971 --> [v2:172.21.15.83:6835/3254575964,v1:172.21.15.83:6837/3254575964] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55cbda725050 con 0x55cbda754130 2024-03-06T07:26:33.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.982+0000 7f8bc2ffd640 1 --2- 172.21.15.83:0/1129362971 >> [v2:172.21.15.83:6835/3254575964,v1:172.21.15.83:6837/3254575964] conn(0x55cbda754130 0x55cbda7512b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:33.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.983+0000 7f8ba1ffb640 10 client.7617 ms_handle_connect on v2:172.21.15.83:6835/3254575964 2024-03-06T07:26:33.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.983+0000 7f8bc2ffd640 1 --2- 172.21.15.83:0/1129362971 >> [v2:172.21.15.83:6835/3254575964,v1:172.21.15.83:6837/3254575964] conn(0x55cbda754130 0x55cbda7512b0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c277979969452dd server_cookie=3f2b8e6b0c78f1ac in_seq=0 out_seq=0 2024-03-06T07:26:33.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.983+0000 7f8ba1ffb640 1 -- 172.21.15.83:0/1129362971 <== mds.0 v2:172.21.15.83:6835/3254575964 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55cbda725050 con 0x55cbda754130 2024-03-06T07:26:33.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.983+0000 7f8ba1ffb640 10 client.7617 handle_command_reply: tid=0 2024-03-06T07:26:33.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.983+0000 7f8bcbfda740 1 client.7617 shutdown 2024-03-06T07:26:33.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.983+0000 7f8bcbfda740 1 -- 172.21.15.83:0/1129362971 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbda752600 msgr2=0x55cbda755ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:33.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.983+0000 7f8bcbfda740 1 --2- 172.21.15.83:0/1129362971 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbda752600 0x55cbda755ce0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc007000 tx=0x7f8bbc001250 comp rx=0 tx=0).stop 2024-03-06T07:26:33.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.983+0000 7f8bcbfda740 1 -- 172.21.15.83:0/1129362971 shutdown_connections 2024-03-06T07:26:33.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.983+0000 7f8bcbfda740 1 --2- 172.21.15.83:0/1129362971 >> [v2:172.21.15.83:6835/3254575964,v1:172.21.15.83:6837/3254575964] conn(0x55cbda754130 0x55cbda7512b0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.983+0000 7f8bcbfda740 1 --2- 172.21.15.83:0/1129362971 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbda752600 0x55cbda755ce0 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.983+0000 7f8bcbfda740 1 --2- 172.21.15.83:0/1129362971 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cbda749070 0x55cbda74a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.983+0000 7f8ba1ffb640 0 client.7617 ms_handle_reset on v2:172.21.15.83:6835/3254575964 2024-03-06T07:26:33.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.983+0000 7f8bcbfda740 1 --2- 172.21.15.83:0/1129362971 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55cbda7474b0 0x55cbda787c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.983+0000 7f8bcbfda740 1 -- 172.21.15.83:0/1129362971 >> 172.21.15.83:0/1129362971 conn(0x55cbda722870 msgr2=0x55cbda7506e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:33.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.983+0000 7f8bcbfda740 1 -- 172.21.15.83:0/1129362971 shutdown_connections 2024-03-06T07:26:33.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.983+0000 7f8bcbfda740 1 -- 172.21.15.83:0/1129362971 wait complete. 2024-03-06T07:26:33.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.983+0000 7f8bcbfda740 20 client.7617 trim_cache size 0 max 16384 2024-03-06T07:26:33.992 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:26:33.992 INFO:teuthology.orchestra.run.smithi083.stdout: "cluster_fsid": "039877dd-d2ce-45d2-8e77-ac5eb7e1d55a", 2024-03-06T07:26:33.992 INFO:teuthology.orchestra.run.smithi083.stdout: "whoami": 0, 2024-03-06T07:26:33.992 INFO:teuthology.orchestra.run.smithi083.stdout: "id": 7575, 2024-03-06T07:26:33.993 INFO:teuthology.orchestra.run.smithi083.stdout: "want_state": "up:active", 2024-03-06T07:26:33.993 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "up:active", 2024-03-06T07:26:33.993 INFO:teuthology.orchestra.run.smithi083.stdout: "fs_name": "cephfs", 2024-03-06T07:26:33.993 INFO:teuthology.orchestra.run.smithi083.stdout: "rank_uptime": 5.1811710050000004, 2024-03-06T07:26:33.993 INFO:teuthology.orchestra.run.smithi083.stdout: "mdsmap_epoch": 53, 2024-03-06T07:26:33.993 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_epoch": 98, 2024-03-06T07:26:33.993 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_epoch_barrier": 98, 2024-03-06T07:26:33.993 INFO:teuthology.orchestra.run.smithi083.stdout: "uptime": 6.0519709449999999 2024-03-06T07:26:33.993 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:26:33.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.983+0000 7f8bc9313640 1 -- 172.21.15.83:0/1433039432 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8ba406b500 msgr2=0x7f8ba406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:33.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.983+0000 7f8bc9313640 1 --2- 172.21.15.83:0/1433039432 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8ba406b500 0x7f8ba406d9c0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f8bb4006fd0 tx=0x7f8bb4008040 comp rx=0 tx=0).stop 2024-03-06T07:26:33.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.983+0000 7f8bc9313640 1 -- 172.21.15.83:0/1433039432 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8bc4155560 msgr2=0x7f8bc413eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:33.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.983+0000 7f8bc9313640 1 --2- 172.21.15.83:0/1433039432 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8bc4155560 0x7f8bc413eae0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f8bac000c00 tx=0x7f8bac0046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.984+0000 7f8bc9313640 1 -- 172.21.15.83:0/1433039432 shutdown_connections 2024-03-06T07:26:33.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.984+0000 7f8bc9313640 1 --2- 172.21.15.83:0/1433039432 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8ba406b500 0x7f8ba406d9c0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.984+0000 7f8bc9313640 1 --2- 172.21.15.83:0/1433039432 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8bc415e8f0 0x7f8bc40b41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.984+0000 7f8bc9313640 1 --2- 172.21.15.83:0/1433039432 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8bc4155560 0x7f8bc413eae0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.984+0000 7f8bc9313640 1 --2- 172.21.15.83:0/1433039432 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8bc40721d0 0x7f8bc415cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.984+0000 7f8bc9313640 1 -- 172.21.15.83:0/1433039432 >> 172.21.15.83:0/1433039432 conn(0x7f8bc40b2890 msgr2=0x7f8bc40d0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:33.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.984+0000 7f8bc9313640 1 -- 172.21.15.83:0/1433039432 shutdown_connections 2024-03-06T07:26:33.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:33.984+0000 7f8bc9313640 1 -- 172.21.15.83:0/1433039432 wait complete. 2024-03-06T07:26:33.996 DEBUG:teuthology.orchestra.run.smithi083:> stdin-killer -- python3 -c ' 2024-03-06T07:26:33.996 DEBUG:teuthology.orchestra.run.smithi083:> import os 2024-03-06T07:26:33.997 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:26:33.997 DEBUG:teuthology.orchestra.run.smithi083:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:26:33.997 DEBUG:teuthology.orchestra.run.smithi083:> subdir = "delete_me" 2024-03-06T07:26:33.997 DEBUG:teuthology.orchestra.run.smithi083:> size_unit = 1024 2024-03-06T07:26:33.997 DEBUG:teuthology.orchestra.run.smithi083:> file_multiplier = 200 2024-03-06T07:26:33.997 DEBUG:teuthology.orchestra.run.smithi083:> os.mkdir(os.path.join(mountpoint, subdir)) 2024-03-06T07:26:33.997 DEBUG:teuthology.orchestra.run.smithi083:> for i in range(0, file_multiplier): 2024-03-06T07:26:33.997 DEBUG:teuthology.orchestra.run.smithi083:> for size in range(0, 16*size_unit, size_unit): 2024-03-06T07:26:33.997 DEBUG:teuthology.orchestra.run.smithi083:> filename = "{0}_{1}.bin".format(i, size // size_unit) 2024-03-06T07:26:33.997 DEBUG:teuthology.orchestra.run.smithi083:> with open(os.path.join(mountpoint, subdir, filename), '"'"'w'"'"') as f: 2024-03-06T07:26:33.997 DEBUG:teuthology.orchestra.run.smithi083:> f.write(size * '"'"'x'"'"') 2024-03-06T07:26:33.997 DEBUG:teuthology.orchestra.run.smithi083:> ' 2024-03-06T07:26:40.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:26:40.479 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json config get mds_max_purge_ops 2024-03-06T07:26:40.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:40.479 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -rf delete_me) 2024-03-06T07:26:40.667 INFO:teuthology.orchestra.run.smithi083.stdout:{"mds_max_purge_ops":"100000000"} 2024-03-06T07:26:40.667 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_max_purge_ops": "100000000" } 2024-03-06T07:26:40.667 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json config get mds_max_purge_files 2024-03-06T07:26:40.785 INFO:teuthology.orchestra.run.smithi083.stdout:{"mds_max_purge_files":"6"} 2024-03-06T07:26:40.785 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_max_purge_files": "6" } 2024-03-06T07:26:40.785 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:40.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.906+0000 7f144db08640 1 Processor -- start 2024-03-06T07:26:40.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.906+0000 7f144db08640 1 -- start start 2024-03-06T07:26:40.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.907+0000 7f144db08640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f14480ce5a0 0x7f14480b4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:40.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.907+0000 7f144db08640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14480ccbd0 con 0x7f14480ce980 2024-03-06T07:26:40.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.907+0000 7f144db08640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14480b3b00 con 0x7f1448162940 2024-03-06T07:26:40.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.907+0000 7f144db08640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14480b3de0 con 0x7f14480ce5a0 2024-03-06T07:26:40.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.907+0000 7f1446ffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f14480ce980 0x7f14480cede0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:50932/0 (socket says 172.21.15.83:50932) 2024-03-06T07:26:40.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.907+0000 7f1446ffd640 1 -- 172.21.15.83:0/1945497465 learned_addr learned my addr 172.21.15.83:0/1945497465 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:40.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.907+0000 7f14477fe640 1 --2- 172.21.15.83:0/1945497465 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f14480ce5a0 0x7f14480b4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:40.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.907+0000 7f14467fc640 1 -- 172.21.15.83:0/1945497465 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 565850506 0 0) 0x7f14480ccbd0 con 0x7f14480ce980 2024-03-06T07:26:40.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.907+0000 7f14467fc640 1 -- 172.21.15.83:0/1945497465 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1430003520 con 0x7f14480ce980 2024-03-06T07:26:40.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.907+0000 7f14477fe640 1 -- 172.21.15.83:0/1945497465 >> v1:172.21.15.183:6789/0 conn(0x7f1448162940 legacy=0x7f1448164d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:40.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.907+0000 7f14477fe640 1 -- 172.21.15.83:0/1945497465 >> v1:172.21.15.83:6789/0 conn(0x7f14480ce980 legacy=0x7f14480cede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:40.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.907+0000 7f14477fe640 1 -- 172.21.15.83:0/1945497465 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14480b40c0 con 0x7f14480ce5a0 2024-03-06T07:26:40.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.908+0000 7f14477fe640 1 --2- 172.21.15.83:0/1945497465 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f14480ce5a0 0x7f14480b4bf0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f14380097d0 tx=0x7f143802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=437cc6e85bdb82e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:40.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.908+0000 7f14467fc640 1 -- 172.21.15.83:0/1945497465 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1438035020 con 0x7f14480ce5a0 2024-03-06T07:26:40.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.908+0000 7f14467fc640 1 -- 172.21.15.83:0/1945497465 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1438031890 con 0x7f14480ce5a0 2024-03-06T07:26:40.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.908+0000 7f14467fc640 1 -- 172.21.15.83:0/1945497465 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1438031c60 con 0x7f14480ce5a0 2024-03-06T07:26:40.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.909+0000 7f144db08640 1 -- 172.21.15.83:0/1945497465 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f14480ce5a0 msgr2=0x7f14480b4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:40.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.909+0000 7f144db08640 1 --2- 172.21.15.83:0/1945497465 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f14480ce5a0 0x7f14480b4bf0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f14380097d0 tx=0x7f143802b930 comp rx=0 tx=0).stop 2024-03-06T07:26:40.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.909+0000 7f144db08640 1 -- 172.21.15.83:0/1945497465 shutdown_connections 2024-03-06T07:26:40.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.909+0000 7f144db08640 1 --2- 172.21.15.83:0/1945497465 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f14480ce5a0 0x7f14480b4bf0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:40.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.909+0000 7f144db08640 1 -- 172.21.15.83:0/1945497465 >> 172.21.15.83:0/1945497465 conn(0x7f144806b160 msgr2=0x7f144806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:40.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.909+0000 7f144db08640 1 -- 172.21.15.83:0/1945497465 shutdown_connections 2024-03-06T07:26:40.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.909+0000 7f144db08640 1 -- 172.21.15.83:0/1945497465 wait complete. 2024-03-06T07:26:40.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.910+0000 7f144db08640 1 Processor -- start 2024-03-06T07:26:40.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.910+0000 7f144db08640 1 -- start start 2024-03-06T07:26:40.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.911+0000 7f144db08640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14480ce5a0 0x7f144813b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:40.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.911+0000 7f144db08640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f14480ce980 0x7f144813f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:40.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.911+0000 7f144db08640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1448162940 0x7f144813fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:40.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.911+0000 7f144db08640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14480cd030 con 0x7f14480ce5a0 2024-03-06T07:26:40.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.911+0000 7f144db08640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14480cceb0 con 0x7f1448162940 2024-03-06T07:26:40.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.911+0000 7f144db08640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1448071d80 con 0x7f14480ce980 2024-03-06T07:26:40.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.911+0000 7f14477fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14480ce5a0 0x7f144813b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:40.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.911+0000 7f14477fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14480ce5a0 0x7f144813b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33312/0 (socket says 172.21.15.83:33312) 2024-03-06T07:26:40.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.911+0000 7f14477fe640 1 -- 172.21.15.83:0/2871858759 learned_addr learned my addr 172.21.15.83:0/2871858759 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:40.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.911+0000 7f1446ffd640 1 --2- 172.21.15.83:0/2871858759 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f14480ce980 0x7f144813f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:40.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.911+0000 7f1447fff640 1 --2- 172.21.15.83:0/2871858759 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1448162940 0x7f144813fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:40.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.911+0000 7f14477fe640 1 -- 172.21.15.83:0/2871858759 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f14480ce980 msgr2=0x7f144813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:40.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.911+0000 7f14477fe640 1 --2- 172.21.15.83:0/2871858759 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f14480ce980 0x7f144813f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:40.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.911+0000 7f14477fe640 1 -- 172.21.15.83:0/2871858759 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1448162940 msgr2=0x7f144813fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:40.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.911+0000 7f14477fe640 1 --2- 172.21.15.83:0/2871858759 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1448162940 0x7f144813fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:40.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.912+0000 7f14477fe640 1 -- 172.21.15.83:0/2871858759 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f144815f040 con 0x7f14480ce5a0 2024-03-06T07:26:40.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.912+0000 7f14477fe640 1 --2- 172.21.15.83:0/2871858759 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14480ce5a0 0x7f144813b310 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f143802bf40 tx=0x7f14380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:40.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.912+0000 7f142ffff640 1 -- 172.21.15.83:0/2871858759 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1438035040 con 0x7f14480ce5a0 2024-03-06T07:26:40.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.912+0000 7f144db08640 1 -- 172.21.15.83:0/2871858759 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f144815f320 con 0x7f14480ce5a0 2024-03-06T07:26:40.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.912+0000 7f142ffff640 1 -- 172.21.15.83:0/2871858759 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1438030b10 con 0x7f14480ce5a0 2024-03-06T07:26:40.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.912+0000 7f142ffff640 1 -- 172.21.15.83:0/2871858759 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1438044430 con 0x7f14480ce5a0 2024-03-06T07:26:40.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.912+0000 7f144db08640 1 -- 172.21.15.83:0/2871858759 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f144815f8e0 con 0x7f14480ce5a0 2024-03-06T07:26:40.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.914+0000 7f144db08640 1 -- 172.21.15.83:0/2871858759 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1448069ee0 con 0x7f14480ce5a0 2024-03-06T07:26:40.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.914+0000 7f142ffff640 1 -- 172.21.15.83:0/2871858759 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1438038020 con 0x7f14480ce5a0 2024-03-06T07:26:40.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.915+0000 7f142ffff640 1 --2- 172.21.15.83:0/2871858759 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f143406b500 0x7f143406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:40.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.915+0000 7f1446ffd640 1 --2- 172.21.15.83:0/2871858759 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f143406b500 0x7f143406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:40.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.915+0000 7f142ffff640 1 -- 172.21.15.83:0/2871858759 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f14380ac5c0 con 0x7f14480ce5a0 2024-03-06T07:26:40.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.915+0000 7f1446ffd640 1 --2- 172.21.15.83:0/2871858759 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f143406b500 0x7f143406d9c0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f14481f4aa0 tx=0x7f143c007d30 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:40.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:40.919+0000 7f142ffff640 1 -- 172.21.15.83:0/2871858759 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f143802d040 con 0x7f14480ce5a0 2024-03-06T07:26:41.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:41.194+0000 7f144db08640 1 -- 172.21.15.83:0/2871858759 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f144815ffa0 con 0x7f14480ce5a0 2024-03-06T07:26:41.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:41.194+0000 7f142ffff640 1 -- 172.21.15.83:0/2871858759 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f143807ada0 con 0x7f14480ce5a0 2024-03-06T07:26:41.196 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:41.196 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":53,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6023,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/601236077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":601236077},{"type":"v1","addr":"172.21.15.183:6833","nonce":601236077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6029,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1435055961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1435055961},{"type":"v1","addr":"172.21.15.183:6835","nonce":1435055961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":7572,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/2726045966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2726045966},{"type":"v1","addr":"172.21.15.83:6836","nonce":2726045966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.064335+0000","modified":"2024-03-06T07:26:32.814889+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7575":{"gid":7575,"name":"a","rank":0,"incarnation":49,"state":"up:active","state_seq":5,"addr":"172.21.15.83:6837/3254575964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":3254575964},{"type":"v1","addr":"172.21.15.83:6837","nonce":3254575964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7575,"qdb_cluster":[7575]},"id":7}]} 2024-03-06T07:26:41.197 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 53 2024-03-06T07:26:41.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:41.196+0000 7f144db08640 1 -- 172.21.15.83:0/2871858759 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f143406b500 msgr2=0x7f143406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:41.196+0000 7f144db08640 1 --2- 172.21.15.83:0/2871858759 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f143406b500 0x7f143406d9c0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f14481f4aa0 tx=0x7f143c007d30 comp rx=0 tx=0).stop 2024-03-06T07:26:41.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:41.196+0000 7f144db08640 1 -- 172.21.15.83:0/2871858759 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14480ce5a0 msgr2=0x7f144813b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:41.196+0000 7f144db08640 1 --2- 172.21.15.83:0/2871858759 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14480ce5a0 0x7f144813b310 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f143802bf40 tx=0x7f14380046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:41.196+0000 7f144db08640 1 -- 172.21.15.83:0/2871858759 shutdown_connections 2024-03-06T07:26:41.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:41.197+0000 7f144db08640 1 --2- 172.21.15.83:0/2871858759 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f143406b500 0x7f143406d9c0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:41.197+0000 7f144db08640 1 --2- 172.21.15.83:0/2871858759 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1448162940 0x7f144813fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:41.197+0000 7f144db08640 1 --2- 172.21.15.83:0/2871858759 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f14480ce980 0x7f144813f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:41.197+0000 7f144db08640 1 --2- 172.21.15.83:0/2871858759 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14480ce5a0 0x7f144813b310 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:41.197+0000 7f144db08640 1 -- 172.21.15.83:0/2871858759 >> 172.21.15.83:0/2871858759 conn(0x7f144806b160 msgr2=0x7f144815d400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:41.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:41.197+0000 7f144db08640 1 -- 172.21.15.83:0/2871858759 shutdown_connections 2024-03-06T07:26:41.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:41.197+0000 7f144db08640 1 -- 172.21.15.83:0/2871858759 wait complete. 2024-03-06T07:26:41.211 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:26:41.362 INFO:teuthology.orchestra.run.smithi083.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":22,"msgr_send_messages":22,"msgr_recv_bytes":14845,"msgr_send_bytes":4222868,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.015522638,"msgr_running_send_time":0.008383978,"msgr_running_recv_time":0.003943106,"msgr_running_fast_dispatch_time":0.001961906,"msgr_send_messages_queue_lat":{"avgcount":21,"sum":0.007183135,"avgtime":0.000342054},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000209,"avgtime":0.000000104},"msgr_recv_encrypted_bytes":1568,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":4438,"msgr_send_messages":10424,"msgr_recv_bytes":1638154,"msgr_send_bytes":10202126,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.445846913,"msgr_running_send_time":0.240808394,"msgr_running_recv_time":0.243833078,"msgr_running_fast_dispatch_time":0.004635290,"msgr_send_messages_queue_lat":{"avgcount":10424,"sum":0.103302878,"avgtime":0.000009910},"msgr_handle_ack_lat":{"avgcount":14717,"sum":0.002188310,"avgtime":0.000000148},"msgr_recv_encrypted_bytes":107504,"msgr_send_encrypted_bytes":6672},"AsyncMessenger::Worker-2":{"msgr_recv_messages":10,"msgr_send_messages":15,"msgr_recv_bytes":4045,"msgr_send_bytes":95408,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.003204357,"msgr_running_send_time":0.000411942,"msgr_running_recv_time":0.002533518,"msgr_running_fast_dispatch_time":0.000144192,"msgr_send_messages_queue_lat":{"avgcount":14,"sum":0.003145270,"avgtime":0.000224662},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":92224},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":703,"complete_latency":{"avgcount":76,"sum":0.794377392,"avgtime":0.010452334}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4,"sum":0.000094113,"avgtime":0.000023528}},"mds":{"request":3362,"reply":3359,"reply_latency":{"avgcount":3359,"sum":3.647648075,"avgtime":0.001085932},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":3211,"inodes_top":0,"inodes_bottom":3211,"inodes_pin_tail":0,"inodes_pinned":3211,"inodes_expired":0,"inodes_with_caps":3202,"caps":3202,"subtrees":2,"traverse":3368,"traverse_hit":3367,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":3012,"root_rbytes":2380800,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":144,"ceph_cap_op_grant":3255,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":1004,"handle_client_caps_dirty":906,"handle_client_cap_release":0,"process_request_cap_release":3501},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":150,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":150,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":4339,"evex":0,"evtrm":0,"ev":4342,"evexg":0,"evexd":0,"segadd":6,"segex":0,"segtrm":0,"seg":8,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":12879432,"rdpos":4195244,"jlat":{"avgcount":12,"sum":0.053763131,"avgtime":0.004480260},"replayed":3},"mds_mem":{"ino":3214,"ino+":3214,"ino-":0,"dir":13,"dir+":13,"dir-":0,"dn":3360,"dn+":3360,"dn-":0,"cap":3202,"cap+":3202,"cap-":0,"rss":86080,"heap":248164},"mds_server":{"dispatch_client_request":3365,"dispatch_server_request":0,"handle_client_request":3362,"handle_client_session":0,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.114640424,"avgtime":0.000973325},"req_getattr_latency":{"avgcount":6,"sum":0.197880257,"avgtime":0.032980042},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002834805,"avgtime":0.002834805},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.120240153,"avgtime":0.040080051},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":149,"sum":0.212052436,"avgtime":0.001423170},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":3204,"average_load":3204,"avg_session_uptime":764,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":499972,"buffer_anon_items":174,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":6544720,"mds_co_items":32743,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":65,"op_send_bytes":8742706,"op_resend":0,"op_reply":65,"op_latency":{"avgcount":65,"sum":0.145996591,"avgtime":0.002246101},"op_inflight":0,"oplen_avg":{"avgcount":65,"sum":120},"op":65,"op_r":36,"op_w":29,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":8,"osdop_write":12,"osdop_writefull":1,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":7,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":15,"omap_rd":48,"omap_del":2},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed_ops":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":557,"max":104857600,"get_started":0,"get":4465,"get_sum":1252410,"get_or_fail_fail":0,"get_or_fail_success":4465,"take":0,"take_sum":0,"put":4463,"put_sum":1251853,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":65,"take_sum":8743823,"put":65,"put_sum":8743823,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":65,"take_sum":65,"put":65,"put_sum":65,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":0,"put_sum":0,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55b1d3c8aa00":{"val":296520,"max":3758096384,"get_started":0,"get":4339,"get_sum":8683747,"get_or_fail_fail":0,"get_or_fail_success":4339,"take":0,"take_sum":0,"put":12,"put_sum":8387227,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:41.364 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.04e-07, "sum": 2.09e-07 }, "msgr_recv_bytes": 14845, "msgr_recv_encrypted_bytes": 1568, "msgr_recv_messages": 22, "msgr_running_fast_dispatch_time": 0.001961906, "msgr_running_recv_time": 0.003943106, "msgr_running_send_time": 0.008383978, "msgr_running_total_time": 0.015522638, "msgr_send_bytes": 4222868, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 22, "msgr_send_messages_queue_lat": { "avgcount": 21, "avgtime": 0.000342054, "sum": 0.007183135 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 14717, "avgtime": 1.48e-07, "sum": 0.00218831 }, "msgr_recv_bytes": 1638154, "msgr_recv_encrypted_bytes": 107504, "msgr_recv_messages": 4438, "msgr_running_fast_dispatch_time": 0.00463529, "msgr_running_recv_time": 0.243833078, "msgr_running_send_time": 0.240808394, "msgr_running_total_time": 0.445846913, "msgr_send_bytes": 10202126, "msgr_send_encrypted_bytes": 6672, "msgr_send_messages": 10424, "msgr_send_messages_queue_lat": { "avgcount": 10424, "avgtime": 9.91e-06, "sum": 0.103302878 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 4045, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 10, "msgr_running_fast_dispatch_time": 0.000144192, "msgr_running_recv_time": 0.002533518, "msgr_running_send_time": 0.000411942, "msgr_running_total_time": 0.003204357, "msgr_send_bytes": 95408, "msgr_send_encrypted_bytes": 92224, "msgr_send_messages": 15, "msgr_send_messages_queue_lat": { "avgcount": 14, "avgtime": 0.000224662, "sum": 0.00314527 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 76, "avgtime": 0.010452334, "sum": 0.794377392 }, "queue_len": 703 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4, "avgtime": 2.3528e-05, "sum": 9.4113e-05 }, "queue_len": 0 }, "mds": { "caps": 3202, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 3255, "ceph_cap_op_revoke": 144, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 0, "handle_client_caps": 1004, "handle_client_caps_dirty": 906, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 3211, "inodes_bottom": 3211, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 3211, "inodes_top": 0, "inodes_with_caps": 3202, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3501, "q": 0, "reply": 3359, "reply_latency": { "avgcount": 3359, "avgtime": 0.001085932, "sum": 3.647648075 }, "request": 3362, "root_rbytes": 2380800, "root_rfiles": 3012, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 3368, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3367, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 150, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 150, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 4342, "evadd": 4339, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 12, "avgtime": 0.00448026, "sum": 0.053763131 }, "rdpos": 4195244, "replayed": 3, "seg": 8, "segadd": 6, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 12879432 }, "mds_mem": { "cap": 3202, "cap+": 3202, "cap-": 0, "dir": 13, "dir+": 13, "dir-": 0, "dn": 3360, "dn+": 3360, "dn-": 0, "heap": 248164, "ino": 3214, "ino+": 3214, "ino-": 0, "rss": 86080 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3365, "dispatch_server_request": 0, "handle_client_request": 3362, "handle_client_session": 0, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000973325, "sum": 3.114640424 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.032980042, "sum": 0.197880257 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002834805, "sum": 0.002834805 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.040080051, "sum": 0.120240153 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 149, "avgtime": 0.00142317, "sum": 0.212052436 } }, "mds_sessions": { "average_load": 3204, "avg_session_uptime": 764, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 3204 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 499972, "buffer_anon_items": 174, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 6544720, "mds_co_items": 32743, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 2, "omap_rd": 48, "omap_wr": 15, "op": 65, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 65, "avgtime": 0.002246101, "sum": 0.145996591 }, "op_pg": 0, "op_r": 36, "op_reply": 65, "op_resend": 0, "op_rmw": 0, "op_send": 65, "op_send_bytes": 8742706, "op_w": 29, "oplen_avg": { "avgcount": 65, "sum": 120 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 7, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 8, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 12, "osdop_writefull": 1, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 0, "pq_executed_ops": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 4465, "get_or_fail_fail": 0, "get_or_fail_success": 4465, "get_started": 0, "get_sum": 1252410, "max": 104857600, "put": 4463, "put_sum": 1251853, "take": 0, "take_sum": 0, "val": 557, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 65, "put_sum": 8743823, "take": 65, "take_sum": 8743823, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 65, "put_sum": 65, "take": 65, "take_sum": 65, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 3758096384, "put": 0, "put_sum": 0, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55b1d3c8aa00": { "get": 4339, "get_or_fail_fail": 0, "get_or_fail_success": 4339, "get_started": 0, "get_sum": 8683747, "max": 3758096384, "put": 12, "put_sum": 8387227, "take": 0, "take_sum": 0, "val": 296520, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:41.364 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/150, 0/150 2024-03-06T07:26:42.365 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:42.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.488+0000 7f5f0eb79640 1 Processor -- start 2024-03-06T07:26:42.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.488+0000 7f5f0eb79640 1 -- start start 2024-03-06T07:26:42.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.489+0000 7f5f0eb79640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5f0815f5e0 0x7f5f080b2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:42.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.489+0000 7f5f0eb79640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f080b16b0 0x7f5f080ce550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:42.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.489+0000 7f5f0eb79640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f5f080d2870 0x7f5f080cea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:42.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.489+0000 7f5f0eb79640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f080cf920 con 0x7f5f0815f5e0 2024-03-06T07:26:42.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.489+0000 7f5f0eb79640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f080cd4d0 con 0x7f5f080d2870 2024-03-06T07:26:42.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.489+0000 7f5f0eb79640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f080cd7b0 con 0x7f5f080b16b0 2024-03-06T07:26:42.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.489+0000 7f5f0c915640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5f0815f5e0 0x7f5f080b2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:42.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.489+0000 7f5f0c915640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5f0815f5e0 0x7f5f080b2bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33318/0 (socket says 172.21.15.83:33318) 2024-03-06T07:26:42.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.489+0000 7f5f0c915640 1 -- 172.21.15.83:0/3439489895 learned_addr learned my addr 172.21.15.83:0/3439489895 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:42.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.489+0000 7f5f0d116640 1 --2- 172.21.15.83:0/3439489895 >> v2:172.21.15.183:3300/0 conn(0x7f5f080d2870 0x7f5f080cea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:42.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.489+0000 7f5f07fff640 1 --2- 172.21.15.83:0/3439489895 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f080b16b0 0x7f5f080ce550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:42.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.489+0000 7f5f0c915640 1 -- 172.21.15.83:0/3439489895 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f080b16b0 msgr2=0x7f5f080ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:42.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.489+0000 7f5f0c915640 1 --2- 172.21.15.83:0/3439489895 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f080b16b0 0x7f5f080ce550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.489+0000 7f5f0c915640 1 -- 172.21.15.83:0/3439489895 >> v2:172.21.15.183:3300/0 conn(0x7f5f080d2870 msgr2=0x7f5f080cea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:42.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.489+0000 7f5f0c915640 1 --2- 172.21.15.83:0/3439489895 >> v2:172.21.15.183:3300/0 conn(0x7f5f080d2870 0x7f5f080cea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.489+0000 7f5f0c915640 1 -- 172.21.15.83:0/3439489895 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f080cda90 con 0x7f5f0815f5e0 2024-03-06T07:26:42.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.489+0000 7f5f0d116640 1 --2- 172.21.15.83:0/3439489895 >> v2:172.21.15.183:3300/0 conn(0x7f5f080d2870 0x7f5f080cea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:42.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.489+0000 7f5f07fff640 1 --2- 172.21.15.83:0/3439489895 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f080b16b0 0x7f5f080ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:42.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.490+0000 7f5f0c915640 1 --2- 172.21.15.83:0/3439489895 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f0815f5e0 0x7f5f080b2bf0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f5f080d1930 tx=0x7f5ef802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48a47244a6036669 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:42.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.490+0000 7f5f077fe640 1 -- 172.21.15.83:0/3439489895 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef802e070 con 0x7f5f0815f5e0 2024-03-06T07:26:42.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.490+0000 7f5f077fe640 1 -- 172.21.15.83:0/3439489895 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ef8037070 con 0x7f5f0815f5e0 2024-03-06T07:26:42.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.490+0000 7f5f077fe640 1 -- 172.21.15.83:0/3439489895 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef8031a00 con 0x7f5f0815f5e0 2024-03-06T07:26:42.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.491+0000 7f5f0eb79640 1 -- 172.21.15.83:0/3439489895 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f0815f5e0 msgr2=0x7f5f080b2bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:42.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.491+0000 7f5f0eb79640 1 --2- 172.21.15.83:0/3439489895 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f0815f5e0 0x7f5f080b2bf0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f5f080d1930 tx=0x7f5ef802b880 comp rx=0 tx=0).stop 2024-03-06T07:26:42.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.491+0000 7f5f0eb79640 1 -- 172.21.15.83:0/3439489895 shutdown_connections 2024-03-06T07:26:42.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.491+0000 7f5f0eb79640 1 --2- 172.21.15.83:0/3439489895 >> v2:172.21.15.183:3300/0 conn(0x7f5f080d2870 0x7f5f080cea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.491+0000 7f5f0eb79640 1 --2- 172.21.15.83:0/3439489895 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f080b16b0 0x7f5f080ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.491+0000 7f5f0eb79640 1 --2- 172.21.15.83:0/3439489895 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f0815f5e0 0x7f5f080b2bf0 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.491+0000 7f5f0eb79640 1 -- 172.21.15.83:0/3439489895 >> 172.21.15.83:0/3439489895 conn(0x7f5f080d1d50 msgr2=0x7f5f080d2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:42.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.491+0000 7f5f0eb79640 1 -- 172.21.15.83:0/3439489895 shutdown_connections 2024-03-06T07:26:42.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.491+0000 7f5f0eb79640 1 -- 172.21.15.83:0/3439489895 wait complete. 2024-03-06T07:26:42.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.493+0000 7f5f0eb79640 1 Processor -- start 2024-03-06T07:26:42.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.493+0000 7f5f0eb79640 1 -- start start 2024-03-06T07:26:42.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.494+0000 7f5f0eb79640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f080b16b0 0x7f5f0815d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:42.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.494+0000 7f5f0eb79640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f080d2870 0x7f5f08070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:42.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.494+0000 7f5f0eb79640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f0815f5e0 0x7f5f0815ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:42.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.494+0000 7f5f0eb79640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f080cf270 con 0x7f5f080b16b0 2024-03-06T07:26:42.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.494+0000 7f5f0eb79640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f080cf0f0 con 0x7f5f080d2870 2024-03-06T07:26:42.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.494+0000 7f5f0eb79640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0815fcd0 con 0x7f5f0815f5e0 2024-03-06T07:26:42.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.494+0000 7f5f0c915640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f080b16b0 0x7f5f0815d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:42.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.494+0000 7f5f0c915640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f080b16b0 0x7f5f0815d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33322/0 (socket says 172.21.15.83:33322) 2024-03-06T07:26:42.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.494+0000 7f5f0c915640 1 -- 172.21.15.83:0/2214528181 learned_addr learned my addr 172.21.15.83:0/2214528181 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:42.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.494+0000 7f5f0d116640 1 --2- 172.21.15.83:0/2214528181 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f0815f5e0 0x7f5f0815ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:42.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.494+0000 7f5f07fff640 1 --2- 172.21.15.83:0/2214528181 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f080d2870 0x7f5f08070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:42.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.494+0000 7f5f0c915640 1 -- 172.21.15.83:0/2214528181 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f0815f5e0 msgr2=0x7f5f0815ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:42.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.494+0000 7f5f0c915640 1 --2- 172.21.15.83:0/2214528181 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f0815f5e0 0x7f5f0815ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.494+0000 7f5f0c915640 1 -- 172.21.15.83:0/2214528181 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f080d2870 msgr2=0x7f5f08070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:42.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.494+0000 7f5f0c915640 1 --2- 172.21.15.83:0/2214528181 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f080d2870 0x7f5f08070150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.494+0000 7f5f0c915640 1 -- 172.21.15.83:0/2214528181 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f0815c750 con 0x7f5f080b16b0 2024-03-06T07:26:42.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.494+0000 7f5f0d116640 1 --2- 172.21.15.83:0/2214528181 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f0815f5e0 0x7f5f0815ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:42.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.494+0000 7f5f07fff640 1 --2- 172.21.15.83:0/2214528181 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f080d2870 0x7f5f08070150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:42.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.495+0000 7f5f0c915640 1 --2- 172.21.15.83:0/2214528181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f080b16b0 0x7f5f0815d7c0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f5ef8000c00 tx=0x7f5ef80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:42.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.495+0000 7f5f057fa640 1 -- 172.21.15.83:0/2214528181 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef8032670 con 0x7f5f080b16b0 2024-03-06T07:26:42.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.495+0000 7f5f057fa640 1 -- 172.21.15.83:0/2214528181 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ef8032850 con 0x7f5f080b16b0 2024-03-06T07:26:42.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.495+0000 7f5f0eb79640 1 -- 172.21.15.83:0/2214528181 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f0815ca60 con 0x7f5f080b16b0 2024-03-06T07:26:42.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.495+0000 7f5f057fa640 1 -- 172.21.15.83:0/2214528181 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef8032c40 con 0x7f5f080b16b0 2024-03-06T07:26:42.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.495+0000 7f5f0eb79640 1 -- 172.21.15.83:0/2214528181 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f0815d2a0 con 0x7f5f080b16b0 2024-03-06T07:26:42.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.496+0000 7f5f0eb79640 1 -- 172.21.15.83:0/2214528181 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f08068050 con 0x7f5f080b16b0 2024-03-06T07:26:42.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.497+0000 7f5f057fa640 1 -- 172.21.15.83:0/2214528181 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5ef804c060 con 0x7f5f080b16b0 2024-03-06T07:26:42.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.497+0000 7f5f057fa640 1 --2- 172.21.15.83:0/2214528181 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5ef406b4d0 0x7f5ef406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:42.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.497+0000 7f5f057fa640 1 -- 172.21.15.83:0/2214528181 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f5ef80abd30 con 0x7f5f080b16b0 2024-03-06T07:26:42.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.497+0000 7f5f07fff640 1 --2- 172.21.15.83:0/2214528181 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5ef406b4d0 0x7f5ef406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:42.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.498+0000 7f5f07fff640 1 --2- 172.21.15.83:0/2214528181 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5ef406b4d0 0x7f5ef406d990 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f5f0815b310 tx=0x7f5efc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:42.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.503+0000 7f5f057fa640 1 -- 172.21.15.83:0/2214528181 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5f0815ca60 con 0x7f5f080b16b0 2024-03-06T07:26:42.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.801+0000 7f5f0eb79640 1 -- 172.21.15.83:0/2214528181 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f080c8600 con 0x7f5f080b16b0 2024-03-06T07:26:42.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.802+0000 7f5f057fa640 1 -- 172.21.15.83:0/2214528181 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f5ef8040070 con 0x7f5f080b16b0 2024-03-06T07:26:42.804 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:42.804 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":53,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6023,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/601236077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":601236077},{"type":"v1","addr":"172.21.15.183:6833","nonce":601236077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6029,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1435055961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1435055961},{"type":"v1","addr":"172.21.15.183:6835","nonce":1435055961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":7572,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/2726045966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2726045966},{"type":"v1","addr":"172.21.15.83:6836","nonce":2726045966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.064335+0000","modified":"2024-03-06T07:26:32.814889+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7575":{"gid":7575,"name":"a","rank":0,"incarnation":49,"state":"up:active","state_seq":5,"addr":"172.21.15.83:6837/3254575964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":3254575964},{"type":"v1","addr":"172.21.15.83:6837","nonce":3254575964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7575,"qdb_cluster":[7575]},"id":7}]} 2024-03-06T07:26:42.804 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 53 2024-03-06T07:26:42.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.804+0000 7f5f0eb79640 1 -- 172.21.15.83:0/2214528181 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5ef406b4d0 msgr2=0x7f5ef406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:42.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.804+0000 7f5f0eb79640 1 --2- 172.21.15.83:0/2214528181 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5ef406b4d0 0x7f5ef406d990 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f5f0815b310 tx=0x7f5efc008040 comp rx=0 tx=0).stop 2024-03-06T07:26:42.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.804+0000 7f5f0eb79640 1 -- 172.21.15.83:0/2214528181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f080b16b0 msgr2=0x7f5f0815d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:42.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.804+0000 7f5f0eb79640 1 --2- 172.21.15.83:0/2214528181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f080b16b0 0x7f5f0815d7c0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f5ef8000c00 tx=0x7f5ef80046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.804+0000 7f5f0eb79640 1 -- 172.21.15.83:0/2214528181 shutdown_connections 2024-03-06T07:26:42.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.804+0000 7f5f0eb79640 1 --2- 172.21.15.83:0/2214528181 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5ef406b4d0 0x7f5ef406d990 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.804+0000 7f5f0eb79640 1 --2- 172.21.15.83:0/2214528181 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f0815f5e0 0x7f5f0815ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.804+0000 7f5f0eb79640 1 --2- 172.21.15.83:0/2214528181 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f080d2870 0x7f5f08070150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.804+0000 7f5f0eb79640 1 --2- 172.21.15.83:0/2214528181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f080b16b0 0x7f5f0815d7c0 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.804+0000 7f5f0eb79640 1 -- 172.21.15.83:0/2214528181 >> 172.21.15.83:0/2214528181 conn(0x7f5f080d1d50 msgr2=0x7f5f0806f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:42.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.804+0000 7f5f0eb79640 1 -- 172.21.15.83:0/2214528181 shutdown_connections 2024-03-06T07:26:42.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:42.804+0000 7f5f0eb79640 1 -- 172.21.15.83:0/2214528181 wait complete. 2024-03-06T07:26:42.818 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:26:42.949 INFO:teuthology.orchestra.run.smithi083.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":23,"msgr_send_messages":23,"msgr_recv_bytes":15091,"msgr_send_bytes":4223186,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.015629925,"msgr_running_send_time":0.008429047,"msgr_running_recv_time":0.003966640,"msgr_running_fast_dispatch_time":0.001994395,"msgr_send_messages_queue_lat":{"avgcount":22,"sum":0.007205052,"avgtime":0.000327502},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000209,"avgtime":0.000000104},"msgr_recv_encrypted_bytes":1568,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":6304,"msgr_send_messages":12549,"msgr_recv_bytes":2310762,"msgr_send_bytes":15552801,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.561816108,"msgr_running_send_time":0.305412857,"msgr_running_recv_time":0.307850521,"msgr_running_fast_dispatch_time":0.006569323,"msgr_send_messages_queue_lat":{"avgcount":12549,"sum":0.128796076,"avgtime":0.000010263},"msgr_handle_ack_lat":{"avgcount":18528,"sum":0.002763530,"avgtime":0.000000149},"msgr_recv_encrypted_bytes":107504,"msgr_send_encrypted_bytes":6672},"AsyncMessenger::Worker-2":{"msgr_recv_messages":10,"msgr_send_messages":15,"msgr_recv_bytes":4045,"msgr_send_bytes":95408,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.003204357,"msgr_running_send_time":0.000411942,"msgr_running_recv_time":0.002533518,"msgr_running_fast_dispatch_time":0.000144192,"msgr_send_messages_queue_lat":{"avgcount":14,"sum":0.003145270,"avgtime":0.000224662},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":92224},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":1779,"complete_latency":{"avgcount":80,"sum":1.465141739,"avgtime":0.018314271}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4,"sum":0.000094113,"avgtime":0.000023528}},"mds":{"request":3891,"reply":3889,"reply_latency":{"avgcount":3889,"sum":4.648115258,"avgtime":0.001195195},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":3890,"inodes_top":0,"inodes_bottom":3890,"inodes_pin_tail":0,"inodes_pinned":3890,"inodes_expired":0,"inodes_with_caps":3202,"caps":3202,"subtrees":2,"traverse":3897,"traverse_hit":3896,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":3160,"root_rbytes":6518784,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":559,"ceph_cap_op_grant":3777,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":2300,"handle_client_caps_dirty":2040,"handle_client_cap_release":0,"process_request_cap_release":4559},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":679,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":679,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":6167,"evex":0,"evtrm":0,"ev":6170,"evexg":0,"evexd":0,"segadd":8,"segex":0,"segtrm":0,"seg":10,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":17093003,"rdpos":4195244,"jlat":{"avgcount":13,"sum":0.077944408,"avgtime":0.005995723},"replayed":3},"mds_mem":{"ino":3214,"ino+":3214,"ino-":0,"dir":13,"dir+":13,"dir-":0,"dn":3890,"dn+":3890,"dn-":0,"cap":3202,"cap+":3202,"cap-":0,"rss":107328,"heap":248164},"mds_server":{"dispatch_client_request":3894,"dispatch_server_request":0,"handle_client_request":3891,"handle_client_session":0,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.114640424,"avgtime":0.000973325},"req_getattr_latency":{"avgcount":6,"sum":0.197880257,"avgtime":0.032980042},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002834805,"avgtime":0.002834805},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.120240153,"avgtime":0.040080051},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":679,"sum":1.212519619,"avgtime":0.001785743},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":3683,"average_load":3683,"avg_session_uptime":769,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":2348911,"buffer_anon_items":1195,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":15263669,"mds_co_items":113352,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":1,"op_laggy":0,"op_send":70,"op_send_bytes":13270044,"op_resend":0,"op_reply":69,"op_latency":{"avgcount":69,"sum":0.174995913,"avgtime":0.002536172},"op_inflight":0,"oplen_avg":{"avgcount":70,"sum":129},"op":70,"op_r":36,"op_w":34,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":8,"osdop_write":14,"osdop_writefull":1,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":8,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":19,"omap_rd":48,"omap_del":4},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed_ops":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":9814,"max":104857600,"get_started":0,"get":6333,"get_sum":1757390,"get_or_fail_fail":0,"get_or_fail_success":6333,"take":0,"take_sum":0,"put":6295,"put_sum":1747576,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":70,"take_sum":13271161,"put":70,"put_sum":13271161,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":70,"take_sum":70,"put":70,"put_sum":70,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":0,"put_sum":0,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55b1d3c8aa00":{"val":3842,"max":3758096384,"get_started":0,"get":6167,"get_sum":12897318,"get_or_fail_fail":0,"get_or_fail_success":6167,"take":0,"take_sum":0,"put":14,"put_sum":12893476,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:42.950 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.04e-07, "sum": 2.09e-07 }, "msgr_recv_bytes": 15091, "msgr_recv_encrypted_bytes": 1568, "msgr_recv_messages": 23, "msgr_running_fast_dispatch_time": 0.001994395, "msgr_running_recv_time": 0.00396664, "msgr_running_send_time": 0.008429047, "msgr_running_total_time": 0.015629925, "msgr_send_bytes": 4223186, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 23, "msgr_send_messages_queue_lat": { "avgcount": 22, "avgtime": 0.000327502, "sum": 0.007205052 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 18528, "avgtime": 1.49e-07, "sum": 0.00276353 }, "msgr_recv_bytes": 2310762, "msgr_recv_encrypted_bytes": 107504, "msgr_recv_messages": 6304, "msgr_running_fast_dispatch_time": 0.006569323, "msgr_running_recv_time": 0.307850521, "msgr_running_send_time": 0.305412857, "msgr_running_total_time": 0.561816108, "msgr_send_bytes": 15552801, "msgr_send_encrypted_bytes": 6672, "msgr_send_messages": 12549, "msgr_send_messages_queue_lat": { "avgcount": 12549, "avgtime": 1.0263e-05, "sum": 0.128796076 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 4045, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 10, "msgr_running_fast_dispatch_time": 0.000144192, "msgr_running_recv_time": 0.002533518, "msgr_running_send_time": 0.000411942, "msgr_running_total_time": 0.003204357, "msgr_send_bytes": 95408, "msgr_send_encrypted_bytes": 92224, "msgr_send_messages": 15, "msgr_send_messages_queue_lat": { "avgcount": 14, "avgtime": 0.000224662, "sum": 0.00314527 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 80, "avgtime": 0.018314271, "sum": 1.465141739 }, "queue_len": 1779 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4, "avgtime": 2.3528e-05, "sum": 9.4113e-05 }, "queue_len": 0 }, "mds": { "caps": 3202, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 3777, "ceph_cap_op_revoke": 559, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 0, "handle_client_caps": 2300, "handle_client_caps_dirty": 2040, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 3890, "inodes_bottom": 3890, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 3890, "inodes_top": 0, "inodes_with_caps": 3202, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 4559, "q": 0, "reply": 3889, "reply_latency": { "avgcount": 3889, "avgtime": 0.001195195, "sum": 4.648115258 }, "request": 3891, "root_rbytes": 6518784, "root_rfiles": 3160, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 3897, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3896, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 679, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 679, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 6170, "evadd": 6167, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 13, "avgtime": 0.005995723, "sum": 0.077944408 }, "rdpos": 4195244, "replayed": 3, "seg": 10, "segadd": 8, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 17093003 }, "mds_mem": { "cap": 3202, "cap+": 3202, "cap-": 0, "dir": 13, "dir+": 13, "dir-": 0, "dn": 3890, "dn+": 3890, "dn-": 0, "heap": 248164, "ino": 3214, "ino+": 3214, "ino-": 0, "rss": 107328 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3894, "dispatch_server_request": 0, "handle_client_request": 3891, "handle_client_session": 0, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000973325, "sum": 3.114640424 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.032980042, "sum": 0.197880257 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002834805, "sum": 0.002834805 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.040080051, "sum": 0.120240153 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 679, "avgtime": 0.001785743, "sum": 1.212519619 } }, "mds_sessions": { "average_load": 3683, "avg_session_uptime": 769, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 3683 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 2348911, "buffer_anon_items": 1195, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 15263669, "mds_co_items": 113352, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 4, "omap_rd": 48, "omap_wr": 19, "op": 70, "op_Infinitylight": 0, "op_active": 1, "op_laggy": 0, "op_latency": { "avgcount": 69, "avgtime": 0.002536172, "sum": 0.174995913 }, "op_pg": 0, "op_r": 36, "op_reply": 69, "op_resend": 0, "op_rmw": 0, "op_send": 70, "op_send_bytes": 13270044, "op_w": 34, "oplen_avg": { "avgcount": 70, "sum": 129 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 8, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 8, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 14, "osdop_writefull": 1, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 0, "pq_executed_ops": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 6333, "get_or_fail_fail": 0, "get_or_fail_success": 6333, "get_started": 0, "get_sum": 1757390, "max": 104857600, "put": 6295, "put_sum": 1747576, "take": 0, "take_sum": 0, "val": 9814, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 70, "put_sum": 13271161, "take": 70, "take_sum": 13271161, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 70, "put_sum": 70, "take": 70, "take_sum": 70, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 3758096384, "put": 0, "put_sum": 0, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55b1d3c8aa00": { "get": 6167, "get_or_fail_fail": 0, "get_or_fail_success": 6167, "get_started": 0, "get_sum": 12897318, "max": 3758096384, "put": 14, "put_sum": 12893476, "take": 0, "take_sum": 0, "val": 3842, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:42.951 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/679, 0/679 2024-03-06T07:26:43.952 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:44.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.085+0000 7fbc30d02640 1 Processor -- start 2024-03-06T07:26:44.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.085+0000 7fbc30d02640 1 -- start start 2024-03-06T07:26:44.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.086+0000 7fbc30d02640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbc2c15e150 0x7fbc2c0cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.086+0000 7fbc30d02640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fbc2c0d1060 0x7fbc2c0b9060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.086+0000 7fbc30d02640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc2c0d1ea0 con 0x7fbc2c0cfba0 2024-03-06T07:26:44.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.086+0000 7fbc30d02640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc2c0d2ab0 con 0x7fbc2c0d1060 2024-03-06T07:26:44.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.086+0000 7fbc30d02640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2c0d2180 con 0x7fbc2c15e150 2024-03-06T07:26:44.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.086+0000 7fbc2a59c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbc2c15e150 0x7fbc2c0cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.086+0000 7fbc2ad9d640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fbc2c0d1060 0x7fbc2c0b9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.086+0000 7fbc29d9b640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fbc2c0cfba0 0x7fbc2c0d0950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:36946/0 (socket says 172.21.15.83:36946) 2024-03-06T07:26:44.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.086+0000 7fbc29d9b640 1 -- 172.21.15.83:0/1745244751 learned_addr learned my addr 172.21.15.83:0/1745244751 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:44.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.086+0000 7fbc2a59c640 1 -- 172.21.15.83:0/1745244751 >> v2:172.21.15.183:3300/0 conn(0x7fbc2c0d1060 msgr2=0x7fbc2c0b9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:44.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.086+0000 7fbc2a59c640 1 --2- 172.21.15.83:0/1745244751 >> v2:172.21.15.183:3300/0 conn(0x7fbc2c0d1060 0x7fbc2c0b9060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.086+0000 7fbc2a59c640 1 -- 172.21.15.83:0/1745244751 >> v1:172.21.15.183:6789/0 conn(0x7fbc2c0cfba0 legacy=0x7fbc2c0d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.086+0000 7fbc2a59c640 1 -- 172.21.15.83:0/1745244751 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc2c0d2460 con 0x7fbc2c15e150 2024-03-06T07:26:44.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.087+0000 7fbc2ad9d640 1 --2- 172.21.15.83:0/1745244751 >> v2:172.21.15.183:3300/0 conn(0x7fbc2c0d1060 0x7fbc2c0b9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:44.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.087+0000 7fbc2a59c640 1 --2- 172.21.15.83:0/1745244751 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbc2c15e150 0x7fbc2c0cf660 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c0097a0 tx=0x7fbc1c02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd7187737b24a51c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:44.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.087+0000 7fbc2959a640 1 -- 172.21.15.83:0/1745244751 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c0315a0 con 0x7fbc2c15e150 2024-03-06T07:26:44.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.088+0000 7fbc2959a640 1 -- 172.21.15.83:0/1745244751 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc1c037070 con 0x7fbc2c15e150 2024-03-06T07:26:44.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.088+0000 7fbc2959a640 1 -- 172.21.15.83:0/1745244751 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c031960 con 0x7fbc2c15e150 2024-03-06T07:26:44.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.088+0000 7fbc30d02640 1 -- 172.21.15.83:0/1745244751 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbc2c15e150 msgr2=0x7fbc2c0cf660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.088+0000 7fbc30d02640 1 --2- 172.21.15.83:0/1745244751 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbc2c15e150 0x7fbc2c0cf660 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c0097a0 tx=0x7fbc1c02b840 comp rx=0 tx=0).stop 2024-03-06T07:26:44.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.088+0000 7fbc30d02640 1 -- 172.21.15.83:0/1745244751 shutdown_connections 2024-03-06T07:26:44.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.088+0000 7fbc30d02640 1 --2- 172.21.15.83:0/1745244751 >> v2:172.21.15.183:3300/0 conn(0x7fbc2c0d1060 0x7fbc2c0b9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.088+0000 7fbc30d02640 1 --2- 172.21.15.83:0/1745244751 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbc2c15e150 0x7fbc2c0cf660 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.088+0000 7fbc30d02640 1 -- 172.21.15.83:0/1745244751 >> 172.21.15.83:0/1745244751 conn(0x7fbc2c09d230 msgr2=0x7fbc2c0d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:44.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.088+0000 7fbc30d02640 1 -- 172.21.15.83:0/1745244751 shutdown_connections 2024-03-06T07:26:44.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.088+0000 7fbc30d02640 1 -- 172.21.15.83:0/1745244751 wait complete. 2024-03-06T07:26:44.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.090+0000 7fbc30d02640 1 Processor -- start 2024-03-06T07:26:44.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.090+0000 7fbc30d02640 1 -- start start 2024-03-06T07:26:44.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.091+0000 7fbc30d02640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbc2c0cfba0 0x7fbc2c148cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.091+0000 7fbc30d02640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbc2c0d1060 0x7fbc2c160c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.091+0000 7fbc30d02640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc2c15e150 0x7fbc2c1611a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.091+0000 7fbc30d02640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2c0d1b30 con 0x7fbc2c15e150 2024-03-06T07:26:44.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.091+0000 7fbc30d02640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2c0d2f40 con 0x7fbc2c0cfba0 2024-03-06T07:26:44.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.091+0000 7fbc30d02640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2c15e530 con 0x7fbc2c0d1060 2024-03-06T07:26:44.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.091+0000 7fbc2ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc2c15e150 0x7fbc2c1611a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.091+0000 7fbc2ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc2c15e150 0x7fbc2c1611a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33328/0 (socket says 172.21.15.83:33328) 2024-03-06T07:26:44.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.091+0000 7fbc2ad9d640 1 -- 172.21.15.83:0/299827012 learned_addr learned my addr 172.21.15.83:0/299827012 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:44.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.091+0000 7fbc2a59c640 1 --2- 172.21.15.83:0/299827012 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbc2c0cfba0 0x7fbc2c148cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.091+0000 7fbc29d9b640 1 --2- 172.21.15.83:0/299827012 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbc2c0d1060 0x7fbc2c160c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.091+0000 7fbc2ad9d640 1 -- 172.21.15.83:0/299827012 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbc2c0d1060 msgr2=0x7fbc2c160c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.092+0000 7fbc2ad9d640 1 --2- 172.21.15.83:0/299827012 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbc2c0d1060 0x7fbc2c160c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.092+0000 7fbc2ad9d640 1 -- 172.21.15.83:0/299827012 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbc2c0cfba0 msgr2=0x7fbc2c148cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.092+0000 7fbc2ad9d640 1 --2- 172.21.15.83:0/299827012 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbc2c0cfba0 0x7fbc2c148cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.092+0000 7fbc2ad9d640 1 -- 172.21.15.83:0/299827012 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc2c14a080 con 0x7fbc2c15e150 2024-03-06T07:26:44.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.092+0000 7fbc2ad9d640 1 --2- 172.21.15.83:0/299827012 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc2c15e150 0x7fbc2c1611a0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fbc2400efc0 tx=0x7fbc24009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:44.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.092+0000 7fbc1affd640 1 -- 172.21.15.83:0/299827012 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc24018030 con 0x7fbc2c15e150 2024-03-06T07:26:44.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.092+0000 7fbc1affd640 1 -- 172.21.15.83:0/299827012 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc2400cde0 con 0x7fbc2c15e150 2024-03-06T07:26:44.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.092+0000 7fbc30d02640 1 -- 172.21.15.83:0/299827012 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc2c14a360 con 0x7fbc2c15e150 2024-03-06T07:26:44.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.092+0000 7fbc1affd640 1 -- 172.21.15.83:0/299827012 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc24003bb0 con 0x7fbc2c15e150 2024-03-06T07:26:44.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.092+0000 7fbc30d02640 1 -- 172.21.15.83:0/299827012 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc2c14a920 con 0x7fbc2c15e150 2024-03-06T07:26:44.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.094+0000 7fbc30d02640 1 -- 172.21.15.83:0/299827012 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc2c069ee0 con 0x7fbc2c15e150 2024-03-06T07:26:44.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.094+0000 7fbc1affd640 1 -- 172.21.15.83:0/299827012 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbc2401b070 con 0x7fbc2c15e150 2024-03-06T07:26:44.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.094+0000 7fbc1affd640 1 --2- 172.21.15.83:0/299827012 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbc0c06b4d0 0x7fbc0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.095+0000 7fbc1affd640 1 -- 172.21.15.83:0/299827012 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fbc2408ab90 con 0x7fbc2c15e150 2024-03-06T07:26:44.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.095+0000 7fbc2a59c640 1 --2- 172.21.15.83:0/299827012 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbc0c06b4d0 0x7fbc0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.096+0000 7fbc2a59c640 1 --2- 172.21.15.83:0/299827012 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbc0c06b4d0 0x7fbc0c06d990 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c036000 tx=0x7fbc1c03d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:44.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.099+0000 7fbc1affd640 1 -- 172.21.15.83:0/299827012 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbc24010040 con 0x7fbc2c15e150 2024-03-06T07:26:44.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.427+0000 7fbc30d02640 1 -- 172.21.15.83:0/299827012 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc2c0b9e20 con 0x7fbc2c15e150 2024-03-06T07:26:44.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.427+0000 7fbc1affd640 1 -- 172.21.15.83:0/299827012 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+4983 (secure 0 0 0) 0x7fbc24059370 con 0x7fbc2c15e150 2024-03-06T07:26:44.429 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:44.429 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":53,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6023,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/601236077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":601236077},{"type":"v1","addr":"172.21.15.183:6833","nonce":601236077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6029,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1435055961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1435055961},{"type":"v1","addr":"172.21.15.183:6835","nonce":1435055961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":7572,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/2726045966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2726045966},{"type":"v1","addr":"172.21.15.83:6836","nonce":2726045966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.064335+0000","modified":"2024-03-06T07:26:32.814889+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7575":{"gid":7575,"name":"a","rank":0,"incarnation":49,"state":"up:active","state_seq":5,"addr":"172.21.15.83:6837/3254575964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":3254575964},{"type":"v1","addr":"172.21.15.83:6837","nonce":3254575964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7575,"qdb_cluster":[7575]},"id":7}]} 2024-03-06T07:26:44.430 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 53 2024-03-06T07:26:44.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.430+0000 7fbc30d02640 1 -- 172.21.15.83:0/299827012 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbc0c06b4d0 msgr2=0x7fbc0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.430+0000 7fbc30d02640 1 --2- 172.21.15.83:0/299827012 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbc0c06b4d0 0x7fbc0c06d990 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c036000 tx=0x7fbc1c03d040 comp rx=0 tx=0).stop 2024-03-06T07:26:44.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.430+0000 7fbc30d02640 1 -- 172.21.15.83:0/299827012 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc2c15e150 msgr2=0x7fbc2c1611a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.430+0000 7fbc30d02640 1 --2- 172.21.15.83:0/299827012 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc2c15e150 0x7fbc2c1611a0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fbc2400efc0 tx=0x7fbc24009f90 comp rx=0 tx=0).stop 2024-03-06T07:26:44.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.430+0000 7fbc30d02640 1 -- 172.21.15.83:0/299827012 shutdown_connections 2024-03-06T07:26:44.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.430+0000 7fbc30d02640 1 --2- 172.21.15.83:0/299827012 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbc0c06b4d0 0x7fbc0c06d990 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.430+0000 7fbc30d02640 1 --2- 172.21.15.83:0/299827012 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc2c15e150 0x7fbc2c1611a0 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.430+0000 7fbc30d02640 1 --2- 172.21.15.83:0/299827012 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbc2c0d1060 0x7fbc2c160c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.430+0000 7fbc30d02640 1 --2- 172.21.15.83:0/299827012 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbc2c0cfba0 0x7fbc2c148cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.430+0000 7fbc30d02640 1 -- 172.21.15.83:0/299827012 >> 172.21.15.83:0/299827012 conn(0x7fbc2c09d230 msgr2=0x7fbc2c0cd040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:44.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.430+0000 7fbc30d02640 1 -- 172.21.15.83:0/299827012 shutdown_connections 2024-03-06T07:26:44.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:44.430+0000 7fbc30d02640 1 -- 172.21.15.83:0/299827012 wait complete. 2024-03-06T07:26:44.447 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:26:44.600 INFO:teuthology.orchestra.run.smithi083.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":72,"msgr_send_messages":72,"msgr_recv_bytes":27529,"msgr_send_bytes":4239152,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.019680092,"msgr_running_send_time":0.010610487,"msgr_running_recv_time":0.004727973,"msgr_running_fast_dispatch_time":0.002850851,"msgr_send_messages_queue_lat":{"avgcount":71,"sum":0.009233090,"avgtime":0.000130043},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000209,"avgtime":0.000000104},"msgr_recv_encrypted_bytes":1568,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":7548,"msgr_send_messages":15912,"msgr_recv_bytes":2767600,"msgr_send_bytes":16736821,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.687736084,"msgr_running_send_time":0.377157625,"msgr_running_recv_time":0.367680969,"msgr_running_fast_dispatch_time":0.007288639,"msgr_send_messages_queue_lat":{"avgcount":15912,"sum":0.171805183,"avgtime":0.000010797},"msgr_handle_ack_lat":{"avgcount":22519,"sum":0.003743016,"avgtime":0.000000166},"msgr_recv_encrypted_bytes":107776,"msgr_send_encrypted_bytes":7296},"AsyncMessenger::Worker-2":{"msgr_recv_messages":41,"msgr_send_messages":47,"msgr_recv_bytes":20836,"msgr_send_bytes":116403,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.005883850,"msgr_running_send_time":0.001521508,"msgr_running_recv_time":0.003267510,"msgr_running_fast_dispatch_time":0.000796301,"msgr_send_messages_queue_lat":{"avgcount":46,"sum":0.004060318,"avgtime":0.000088267},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":94016},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":163,"complete_latency":{"avgcount":82,"sum":3.222770014,"avgtime":0.039302073}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":148,"sum":0.099175820,"avgtime":0.000670106}},"mds":{"request":4486,"reply":4484,"reply_latency":{"avgcount":4484,"sum":6.196424752,"avgtime":0.001381896},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":4382,"inodes_top":0,"inodes_bottom":4382,"inodes_pin_tail":0,"inodes_pinned":4382,"inodes_expired":0,"inodes_with_caps":3113,"caps":3113,"subtrees":2,"traverse":4492,"traverse_hit":4491,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":2831,"root_rbytes":9556992,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":835,"ceph_cap_op_grant":4713,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":2958,"handle_client_caps_dirty":2570,"handle_client_cap_release":1,"process_request_cap_release":5750},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1185,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":1274,"strays_enqueued":89,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":7510,"evex":0,"evtrm":0,"ev":7513,"evexg":0,"evexd":0,"segadd":9,"segex":0,"segtrm":0,"seg":11,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":20326433,"rdpos":4195244,"jlat":{"avgcount":14,"sum":1.706413880,"avgtime":0.121886705},"replayed":3},"mds_mem":{"ino":3214,"ino+":3214,"ino-":0,"dir":13,"dir+":13,"dir-":0,"dn":4485,"dn+":4485,"dn-":0,"cap":3113,"cap+":3202,"cap-":89,"rss":108480,"heap":248164},"mds_server":{"dispatch_client_request":4490,"dispatch_server_request":0,"handle_client_request":4487,"handle_client_session":0,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.114640424,"avgtime":0.000973325},"req_getattr_latency":{"avgcount":6,"sum":0.197880257,"avgtime":0.032980042},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002834805,"avgtime":0.002834805},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.120240153,"avgtime":0.040080051},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1274,"sum":2.760829113,"avgtime":0.002167055},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":4191,"average_load":4191,"avg_session_uptime":769,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":2348911,"buffer_anon_items":1195,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":15263669,"mds_co_items":113352,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":173,"op_send_bytes":13279213,"op_resend":0,"op_reply":173,"op_latency":{"avgcount":173,"sum":0.353991734,"avgtime":0.002046194},"op_inflight":0,"oplen_avg":{"avgcount":173,"sum":232},"op":173,"op_r":39,"op_w":134,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":11,"osdop_write":17,"osdop_writefull":3,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":103,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":19,"omap_rd":48,"omap_del":4},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":89,"pq_executed":89,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":296,"max":104857600,"get_started":0,"get":7656,"get_sum":2124089,"get_or_fail_fail":0,"get_or_fail_success":7656,"take":0,"take_sum":0,"put":7655,"put_sum":2123793,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":173,"take_sum":13289319,"put":173,"put_sum":13289319,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":173,"take_sum":173,"put":173,"put_sum":173,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":89,"get_sum":8989,"get_or_fail_fail":0,"get_or_fail_success":89,"take":0,"take_sum":0,"put":3,"put_sum":8989,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55b1d3c8aa00":{"val":3237272,"max":3758096384,"get_started":0,"get":7510,"get_sum":16130748,"get_or_fail_fail":0,"get_or_fail_success":7510,"take":0,"take_sum":0,"put":14,"put_sum":12893476,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:44.601 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.04e-07, "sum": 2.09e-07 }, "msgr_recv_bytes": 27529, "msgr_recv_encrypted_bytes": 1568, "msgr_recv_messages": 72, "msgr_running_fast_dispatch_time": 0.002850851, "msgr_running_recv_time": 0.004727973, "msgr_running_send_time": 0.010610487, "msgr_running_total_time": 0.019680092, "msgr_send_bytes": 4239152, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 72, "msgr_send_messages_queue_lat": { "avgcount": 71, "avgtime": 0.000130043, "sum": 0.00923309 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 22519, "avgtime": 1.66e-07, "sum": 0.003743016 }, "msgr_recv_bytes": 2767600, "msgr_recv_encrypted_bytes": 107776, "msgr_recv_messages": 7548, "msgr_running_fast_dispatch_time": 0.007288639, "msgr_running_recv_time": 0.367680969, "msgr_running_send_time": 0.377157625, "msgr_running_total_time": 0.687736084, "msgr_send_bytes": 16736821, "msgr_send_encrypted_bytes": 7296, "msgr_send_messages": 15912, "msgr_send_messages_queue_lat": { "avgcount": 15912, "avgtime": 1.0797e-05, "sum": 0.171805183 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 20836, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 41, "msgr_running_fast_dispatch_time": 0.000796301, "msgr_running_recv_time": 0.00326751, "msgr_running_send_time": 0.001521508, "msgr_running_total_time": 0.00588385, "msgr_send_bytes": 116403, "msgr_send_encrypted_bytes": 94016, "msgr_send_messages": 47, "msgr_send_messages_queue_lat": { "avgcount": 46, "avgtime": 8.8267e-05, "sum": 0.004060318 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 82, "avgtime": 0.039302073, "sum": 3.222770014 }, "queue_len": 163 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 148, "avgtime": 0.000670106, "sum": 0.09917582 }, "queue_len": 0 }, "mds": { "caps": 3113, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4713, "ceph_cap_op_revoke": 835, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 1, "handle_client_caps": 2958, "handle_client_caps_dirty": 2570, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4382, "inodes_bottom": 4382, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4382, "inodes_top": 0, "inodes_with_caps": 3113, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 5750, "q": 0, "reply": 4484, "reply_latency": { "avgcount": 4484, "avgtime": 0.001381896, "sum": 6.196424752 }, "request": 4486, "root_rbytes": 9556992, "root_rfiles": 2831, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 4492, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 4491, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1185, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1274, "strays_enqueued": 89, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 7513, "evadd": 7510, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 14, "avgtime": 0.121886705, "sum": 1.70641388 }, "rdpos": 4195244, "replayed": 3, "seg": 11, "segadd": 9, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 20326433 }, "mds_mem": { "cap": 3113, "cap+": 3202, "cap-": 89, "dir": 13, "dir+": 13, "dir-": 0, "dn": 4485, "dn+": 4485, "dn-": 0, "heap": 248164, "ino": 3214, "ino+": 3214, "ino-": 0, "rss": 108480 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 4490, "dispatch_server_request": 0, "handle_client_request": 4487, "handle_client_session": 0, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000973325, "sum": 3.114640424 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.032980042, "sum": 0.197880257 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002834805, "sum": 0.002834805 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.040080051, "sum": 0.120240153 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1274, "avgtime": 0.002167055, "sum": 2.760829113 } }, "mds_sessions": { "average_load": 4191, "avg_session_uptime": 769, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 4191 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 2348911, "buffer_anon_items": 1195, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 15263669, "mds_co_items": 113352, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 4, "omap_rd": 48, "omap_wr": 19, "op": 173, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 173, "avgtime": 0.002046194, "sum": 0.353991734 }, "op_pg": 0, "op_r": 39, "op_reply": 173, "op_resend": 0, "op_rmw": 0, "op_send": 173, "op_send_bytes": 13279213, "op_w": 134, "oplen_avg": { "avgcount": 173, "sum": 232 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 103, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 11, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 17, "osdop_writefull": 3, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 89, "pq_executed_ops": 89, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 7656, "get_or_fail_fail": 0, "get_or_fail_success": 7656, "get_started": 0, "get_sum": 2124089, "max": 104857600, "put": 7655, "put_sum": 2123793, "take": 0, "take_sum": 0, "val": 296, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 173, "put_sum": 13289319, "take": 173, "take_sum": 13289319, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 173, "put_sum": 173, "take": 173, "take_sum": 173, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 89, "get_or_fail_fail": 0, "get_or_fail_success": 89, "get_started": 0, "get_sum": 8989, "max": 3758096384, "put": 3, "put_sum": 8989, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55b1d3c8aa00": { "get": 7510, "get_or_fail_fail": 0, "get_or_fail_success": 7510, "get_started": 0, "get_sum": 16130748, "max": 3758096384, "put": 14, "put_sum": 12893476, "take": 0, "take_sum": 0, "val": 3237272, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:44.601 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/1185, 89/1274 2024-03-06T07:26:45.602 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:45.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.722+0000 7fb7dde47640 1 Processor -- start 2024-03-06T07:26:45.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.723+0000 7fb7dde47640 1 -- start start 2024-03-06T07:26:45.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.724+0000 7fb7dde47640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fb7d80d2880 0x7fb7d80d0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.724+0000 7fb7dde47640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb7d806ee60 0x7fb7d806f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.724+0000 7fb7dde47640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7d80cdc90 con 0x7fb7d815c070 2024-03-06T07:26:45.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.724+0000 7fb7dde47640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7d80cc870 con 0x7fb7d806ee60 2024-03-06T07:26:45.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.724+0000 7fb7dde47640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7d80cf660 con 0x7fb7d80d2880 2024-03-06T07:26:45.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.724+0000 7fb7d7fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb7d806ee60 0x7fb7d806f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.724+0000 7fb7d77fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fb7d815c070 0x7fb7d80cd580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:50934/0 (socket says 172.21.15.83:50934) 2024-03-06T07:26:45.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.724+0000 7fb7d77fe640 1 -- 172.21.15.83:0/3425635737 learned_addr learned my addr 172.21.15.83:0/3425635737 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:45.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.724+0000 7fb7d6ffd640 1 --2- 172.21.15.83:0/3425635737 >> v2:172.21.15.183:3300/0 conn(0x7fb7d80d2880 0x7fb7d80d0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.724+0000 7fb7d7fff640 1 -- 172.21.15.83:0/3425635737 >> v2:172.21.15.183:3300/0 conn(0x7fb7d80d2880 msgr2=0x7fb7d80d0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:45.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.724+0000 7fb7d7fff640 1 --2- 172.21.15.83:0/3425635737 >> v2:172.21.15.183:3300/0 conn(0x7fb7d80d2880 0x7fb7d80d0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.724+0000 7fb7d7fff640 1 -- 172.21.15.83:0/3425635737 >> v1:172.21.15.83:6789/0 conn(0x7fb7d815c070 legacy=0x7fb7d80cd580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.724+0000 7fb7d7fff640 1 -- 172.21.15.83:0/3425635737 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d80d09d0 con 0x7fb7d806ee60 2024-03-06T07:26:45.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.725+0000 7fb7d7fff640 1 --2- 172.21.15.83:0/3425635737 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7d806ee60 0x7fb7d806f2d0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fb7d8071a70 tx=0x7fb7d002bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a633b9d1830ca2bb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:45.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.725+0000 7fb7d67fc640 1 -- 172.21.15.83:0/3425635737 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d0039070 con 0x7fb7d806ee60 2024-03-06T07:26:45.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.726+0000 7fb7d67fc640 1 -- 172.21.15.83:0/3425635737 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7d002cc70 con 0x7fb7d806ee60 2024-03-06T07:26:45.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.726+0000 7fb7d67fc640 1 -- 172.21.15.83:0/3425635737 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d0034660 con 0x7fb7d806ee60 2024-03-06T07:26:45.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.726+0000 7fb7dde47640 1 -- 172.21.15.83:0/3425635737 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7d806ee60 msgr2=0x7fb7d806f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.726+0000 7fb7dde47640 1 --2- 172.21.15.83:0/3425635737 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7d806ee60 0x7fb7d806f2d0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fb7d8071a70 tx=0x7fb7d002bec0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.726+0000 7fb7dde47640 1 -- 172.21.15.83:0/3425635737 shutdown_connections 2024-03-06T07:26:45.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.726+0000 7fb7dde47640 1 --2- 172.21.15.83:0/3425635737 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7d806ee60 0x7fb7d806f2d0 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.726+0000 7fb7dde47640 1 --2- 172.21.15.83:0/3425635737 >> v2:172.21.15.183:3300/0 conn(0x7fb7d80d2880 0x7fb7d80d0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.726+0000 7fb7dde47640 1 -- 172.21.15.83:0/3425635737 >> 172.21.15.83:0/3425635737 conn(0x7fb7d80d1d60 msgr2=0x7fb7d80d2170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:45.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.726+0000 7fb7dde47640 1 -- 172.21.15.83:0/3425635737 shutdown_connections 2024-03-06T07:26:45.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.726+0000 7fb7dde47640 1 -- 172.21.15.83:0/3425635737 wait complete. 2024-03-06T07:26:45.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.728+0000 7fb7dde47640 1 Processor -- start 2024-03-06T07:26:45.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.728+0000 7fb7dde47640 1 -- start start 2024-03-06T07:26:45.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.729+0000 7fb7dde47640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7d806ee60 0x7fb7d8140ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.729+0000 7fb7dde47640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb7d80d2880 0x7fb7d8150c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.729+0000 7fb7dde47640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb7d815c070 0x7fb7d8151190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.729+0000 7fb7dde47640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d80cd060 con 0x7fb7d806ee60 2024-03-06T07:26:45.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.729+0000 7fb7dde47640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d80ccd80 con 0x7fb7d815c070 2024-03-06T07:26:45.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.729+0000 7fb7d77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7d806ee60 0x7fb7d8140ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.729+0000 7fb7dde47640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d80d0cb0 con 0x7fb7d80d2880 2024-03-06T07:26:45.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.729+0000 7fb7d77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7d806ee60 0x7fb7d8140ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33346/0 (socket says 172.21.15.83:33346) 2024-03-06T07:26:45.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.729+0000 7fb7d77fe640 1 -- 172.21.15.83:0/1686390688 learned_addr learned my addr 172.21.15.83:0/1686390688 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:45.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.729+0000 7fb7d6ffd640 1 --2- 172.21.15.83:0/1686390688 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb7d80d2880 0x7fb7d8150c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.729+0000 7fb7d7fff640 1 --2- 172.21.15.83:0/1686390688 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb7d815c070 0x7fb7d8151190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.729+0000 7fb7d77fe640 1 -- 172.21.15.83:0/1686390688 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb7d80d2880 msgr2=0x7fb7d8150c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.729+0000 7fb7d77fe640 1 --2- 172.21.15.83:0/1686390688 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb7d80d2880 0x7fb7d8150c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.729+0000 7fb7d77fe640 1 -- 172.21.15.83:0/1686390688 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb7d815c070 msgr2=0x7fb7d8151190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.729+0000 7fb7d77fe640 1 --2- 172.21.15.83:0/1686390688 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb7d815c070 0x7fb7d8151190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.729+0000 7fb7d77fe640 1 -- 172.21.15.83:0/1686390688 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d8150970 con 0x7fb7d806ee60 2024-03-06T07:26:45.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.730+0000 7fb7d77fe640 1 --2- 172.21.15.83:0/1686390688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7d806ee60 0x7fb7d8140ba0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fb7c800c400 tx=0x7fb7c800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:45.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.730+0000 7fb7bffff640 1 -- 172.21.15.83:0/1686390688 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c8017070 con 0x7fb7d806ee60 2024-03-06T07:26:45.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.730+0000 7fb7dde47640 1 -- 172.21.15.83:0/1686390688 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7d81517c0 con 0x7fb7d806ee60 2024-03-06T07:26:45.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.730+0000 7fb7bffff640 1 -- 172.21.15.83:0/1686390688 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7c8005ce0 con 0x7fb7d806ee60 2024-03-06T07:26:45.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.730+0000 7fb7bffff640 1 -- 172.21.15.83:0/1686390688 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c8017070 con 0x7fb7d806ee60 2024-03-06T07:26:45.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.730+0000 7fb7dde47640 1 -- 172.21.15.83:0/1686390688 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7d8151d80 con 0x7fb7d806ee60 2024-03-06T07:26:45.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.731+0000 7fb7dde47640 1 -- 172.21.15.83:0/1686390688 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7d8069ee0 con 0x7fb7d806ee60 2024-03-06T07:26:45.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.732+0000 7fb7bffff640 1 -- 172.21.15.83:0/1686390688 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb7c801a070 con 0x7fb7d806ee60 2024-03-06T07:26:45.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.732+0000 7fb7bffff640 1 --2- 172.21.15.83:0/1686390688 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb7c406b6d0 0x7fb7c406db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.732+0000 7fb7bffff640 1 -- 172.21.15.83:0/1686390688 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb7c8089120 con 0x7fb7d806ee60 2024-03-06T07:26:45.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.732+0000 7fb7d6ffd640 1 --2- 172.21.15.83:0/1686390688 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb7c406b6d0 0x7fb7c406db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.733+0000 7fb7d6ffd640 1 --2- 172.21.15.83:0/1686390688 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb7c406b6d0 0x7fb7c406db90 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fb7d813df40 tx=0x7fb7cc009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:45.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:45.740+0000 7fb7bffff640 1 -- 172.21.15.83:0/1686390688 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb7c8057900 con 0x7fb7d806ee60 2024-03-06T07:26:46.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:46.031+0000 7fb7dde47640 1 -- 172.21.15.83:0/1686390688 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7d80b79a0 con 0x7fb7d806ee60 2024-03-06T07:26:46.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:46.032+0000 7fb7bffff640 1 -- 172.21.15.83:0/1686390688 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+4983 (secure 0 0 0) 0x7fb7c805b9a0 con 0x7fb7d806ee60 2024-03-06T07:26:46.034 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:46.034 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":53,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6023,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/601236077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":601236077},{"type":"v1","addr":"172.21.15.183:6833","nonce":601236077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6029,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1435055961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1435055961},{"type":"v1","addr":"172.21.15.183:6835","nonce":1435055961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":7572,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/2726045966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2726045966},{"type":"v1","addr":"172.21.15.83:6836","nonce":2726045966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.064335+0000","modified":"2024-03-06T07:26:32.814889+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7575":{"gid":7575,"name":"a","rank":0,"incarnation":49,"state":"up:active","state_seq":5,"addr":"172.21.15.83:6837/3254575964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":3254575964},{"type":"v1","addr":"172.21.15.83:6837","nonce":3254575964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7575,"qdb_cluster":[7575]},"id":7}]} 2024-03-06T07:26:46.034 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 53 2024-03-06T07:26:46.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:46.034+0000 7fb7dde47640 1 -- 172.21.15.83:0/1686390688 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb7c406b6d0 msgr2=0x7fb7c406db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:46.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:46.034+0000 7fb7dde47640 1 --2- 172.21.15.83:0/1686390688 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb7c406b6d0 0x7fb7c406db90 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fb7d813df40 tx=0x7fb7cc009190 comp rx=0 tx=0).stop 2024-03-06T07:26:46.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:46.034+0000 7fb7dde47640 1 -- 172.21.15.83:0/1686390688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7d806ee60 msgr2=0x7fb7d8140ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:46.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:46.034+0000 7fb7dde47640 1 --2- 172.21.15.83:0/1686390688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7d806ee60 0x7fb7d8140ba0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fb7c800c400 tx=0x7fb7c800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:46.034+0000 7fb7dde47640 1 -- 172.21.15.83:0/1686390688 shutdown_connections 2024-03-06T07:26:46.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:46.034+0000 7fb7dde47640 1 --2- 172.21.15.83:0/1686390688 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb7c406b6d0 0x7fb7c406db90 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:46.034+0000 7fb7dde47640 1 --2- 172.21.15.83:0/1686390688 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb7d815c070 0x7fb7d8151190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:46.034+0000 7fb7dde47640 1 --2- 172.21.15.83:0/1686390688 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb7d80d2880 0x7fb7d8150c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:46.034+0000 7fb7dde47640 1 --2- 172.21.15.83:0/1686390688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7d806ee60 0x7fb7d8140ba0 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:46.034+0000 7fb7dde47640 1 -- 172.21.15.83:0/1686390688 >> 172.21.15.83:0/1686390688 conn(0x7fb7d80d1d60 msgr2=0x7fb7d80b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:46.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:46.035+0000 7fb7dde47640 1 -- 172.21.15.83:0/1686390688 shutdown_connections 2024-03-06T07:26:46.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:46.035+0000 7fb7dde47640 1 -- 172.21.15.83:0/1686390688 wait complete. 2024-03-06T07:26:46.051 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:26:46.226 INFO:teuthology.orchestra.run.smithi083.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":278,"msgr_send_messages":282,"msgr_recv_bytes":79845,"msgr_send_bytes":4307604,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.048000677,"msgr_running_send_time":0.026251573,"msgr_running_recv_time":0.009254864,"msgr_running_fast_dispatch_time":0.009887199,"msgr_send_messages_queue_lat":{"avgcount":281,"sum":0.027702922,"avgtime":0.000098586},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000209,"avgtime":0.000000104},"msgr_recv_encrypted_bytes":1568,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":8759,"msgr_send_messages":19064,"msgr_recv_bytes":3215498,"msgr_send_bytes":24100176,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.832214840,"msgr_running_send_time":0.465699924,"msgr_running_recv_time":0.424685262,"msgr_running_fast_dispatch_time":0.013126132,"msgr_send_messages_queue_lat":{"avgcount":19063,"sum":0.260274336,"avgtime":0.000013653},"msgr_handle_ack_lat":{"avgcount":25932,"sum":0.004487269,"avgtime":0.000000173},"msgr_recv_encrypted_bytes":107776,"msgr_send_encrypted_bytes":7296},"AsyncMessenger::Worker-2":{"msgr_recv_messages":191,"msgr_send_messages":199,"msgr_recv_bytes":118422,"msgr_send_bytes":225531,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.023259807,"msgr_running_send_time":0.011318532,"msgr_running_recv_time":0.006099665,"msgr_running_fast_dispatch_time":0.004374728,"msgr_send_messages_queue_lat":{"avgcount":198,"sum":0.013748114,"avgtime":0.000069434},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":94016},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":1438,"complete_latency":{"avgcount":88,"sum":4.766872963,"avgtime":0.054169010}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":886,"sum":0.663698932,"avgtime":0.000749095}},"mds":{"request":5066,"reply":5064,"reply_latency":{"avgcount":5064,"sum":7.432131967,"avgtime":0.001467640},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":4957,"inodes_top":0,"inodes_bottom":4957,"inodes_pin_tail":0,"inodes_pinned":4957,"inodes_expired":0,"inodes_with_caps":2523,"caps":2523,"subtrees":2,"traverse":5072,"traverse_hit":5071,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":1870,"root_rbytes":11083776,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":938,"ceph_cap_op_grant":5672,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3470,"handle_client_caps_dirty":3000,"handle_client_cap_release":3,"process_request_cap_release":6909},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1175,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":1854,"strays_enqueued":679,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":9196,"evex":0,"evtrm":0,"ev":9199,"evexg":0,"evexd":0,"segadd":11,"segex":0,"segtrm":0,"seg":13,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":23524616,"rdpos":4195244,"jlat":{"avgcount":17,"sum":2.969231146,"avgtime":0.174660655},"replayed":3},"mds_mem":{"ino":3125,"ino+":3214,"ino-":89,"dir":13,"dir+":13,"dir-":0,"dn":4976,"dn+":5065,"dn-":89,"cap":2523,"cap+":3202,"cap-":679,"rss":108480,"heap":248164},"mds_server":{"dispatch_client_request":5069,"dispatch_server_request":0,"handle_client_request":5066,"handle_client_session":0,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.114640424,"avgtime":0.000973325},"req_getattr_latency":{"avgcount":6,"sum":0.197880257,"avgtime":0.032980042},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002834805,"avgtime":0.002834805},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.120240153,"avgtime":0.040080051},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1854,"sum":3.996536328,"avgtime":0.002155629},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":4688,"average_load":4688,"avg_session_uptime":769,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":2348911,"buffer_anon_items":1195,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":15263669,"mds_co_items":113352,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":6,"op_laggy":0,"op_send":651,"op_send_bytes":19628164,"op_resend":0,"op_reply":645,"op_latency":{"avgcount":645,"sum":1.239971010,"avgtime":0.001922435},"op_inflight":5,"oplen_avg":{"avgcount":651,"sum":712},"op":651,"op_r":45,"op_w":606,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":17,"osdop_write":26,"osdop_writefull":5,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":563,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":21,"omap_rd":48,"omap_del":5},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":6,"pq_executing_ops_high_water":6,"pq_executing":6,"pq_executing_high_water":6,"pq_executed_ops":542,"pq_executed":542,"pq_item_in_journal":140},"throttle-msgr_dispatch_throttler-mds":{"val":297,"max":104857600,"get_started":0,"get":9225,"get_sum":2581320,"get_or_fail_fail":0,"get_or_fail_success":9225,"take":0,"take_sum":0,"put":9224,"put_sum":2581023,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":651,"take_sum":19697860,"put":646,"put_sum":19697860,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":6,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":652,"take_sum":652,"put":646,"put_sum":646,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":679,"get_sum":68579,"get_or_fail_fail":0,"get_or_fail_success":679,"take":0,"take_sum":0,"put":9,"put_sum":68579,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55b1d3c8aa00":{"val":160310,"max":3758096384,"get_started":0,"get":9196,"get_sum":19328931,"get_or_fail_fail":0,"get_or_fail_success":9196,"take":0,"take_sum":0,"put":17,"put_sum":19168621,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:46.228 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.04e-07, "sum": 2.09e-07 }, "msgr_recv_bytes": 79845, "msgr_recv_encrypted_bytes": 1568, "msgr_recv_messages": 278, "msgr_running_fast_dispatch_time": 0.009887199, "msgr_running_recv_time": 0.009254864, "msgr_running_send_time": 0.026251573, "msgr_running_total_time": 0.048000677, "msgr_send_bytes": 4307604, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 282, "msgr_send_messages_queue_lat": { "avgcount": 281, "avgtime": 9.8586e-05, "sum": 0.027702922 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 25932, "avgtime": 1.73e-07, "sum": 0.004487269 }, "msgr_recv_bytes": 3215498, "msgr_recv_encrypted_bytes": 107776, "msgr_recv_messages": 8759, "msgr_running_fast_dispatch_time": 0.013126132, "msgr_running_recv_time": 0.424685262, "msgr_running_send_time": 0.465699924, "msgr_running_total_time": 0.83221484, "msgr_send_bytes": 24100176, "msgr_send_encrypted_bytes": 7296, "msgr_send_messages": 19064, "msgr_send_messages_queue_lat": { "avgcount": 19063, "avgtime": 1.3653e-05, "sum": 0.260274336 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 118422, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 191, "msgr_running_fast_dispatch_time": 0.004374728, "msgr_running_recv_time": 0.006099665, "msgr_running_send_time": 0.011318532, "msgr_running_total_time": 0.023259807, "msgr_send_bytes": 225531, "msgr_send_encrypted_bytes": 94016, "msgr_send_messages": 199, "msgr_send_messages_queue_lat": { "avgcount": 198, "avgtime": 6.9434e-05, "sum": 0.013748114 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 88, "avgtime": 0.05416901, "sum": 4.766872963 }, "queue_len": 1438 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 886, "avgtime": 0.000749095, "sum": 0.663698932 }, "queue_len": 0 }, "mds": { "caps": 2523, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 5672, "ceph_cap_op_revoke": 938, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 3, "handle_client_caps": 3470, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4957, "inodes_bottom": 4957, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4957, "inodes_top": 0, "inodes_with_caps": 2523, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 6909, "q": 0, "reply": 5064, "reply_latency": { "avgcount": 5064, "avgtime": 0.00146764, "sum": 7.432131967 }, "request": 5066, "root_rbytes": 11083776, "root_rfiles": 1870, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 5072, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 5071, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1175, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1854, "strays_enqueued": 679, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 9199, "evadd": 9196, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 17, "avgtime": 0.174660655, "sum": 2.969231146 }, "rdpos": 4195244, "replayed": 3, "seg": 13, "segadd": 11, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 23524616 }, "mds_mem": { "cap": 2523, "cap+": 3202, "cap-": 679, "dir": 13, "dir+": 13, "dir-": 0, "dn": 4976, "dn+": 5065, "dn-": 89, "heap": 248164, "ino": 3125, "ino+": 3214, "ino-": 89, "rss": 108480 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 5069, "dispatch_server_request": 0, "handle_client_request": 5066, "handle_client_session": 0, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000973325, "sum": 3.114640424 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.032980042, "sum": 0.197880257 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002834805, "sum": 0.002834805 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.040080051, "sum": 0.120240153 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1854, "avgtime": 0.002155629, "sum": 3.996536328 } }, "mds_sessions": { "average_load": 4688, "avg_session_uptime": 769, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 4688 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 2348911, "buffer_anon_items": 1195, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 15263669, "mds_co_items": 113352, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 5, "omap_rd": 48, "omap_wr": 21, "op": 651, "op_Infinitylight": 5, "op_active": 6, "op_laggy": 0, "op_latency": { "avgcount": 645, "avgtime": 0.001922435, "sum": 1.23997101 }, "op_pg": 0, "op_r": 45, "op_reply": 645, "op_resend": 0, "op_rmw": 0, "op_send": 651, "op_send_bytes": 19628164, "op_w": 606, "oplen_avg": { "avgcount": 651, "sum": 712 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 563, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 17, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 26, "osdop_writefull": 5, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 542, "pq_executed_ops": 542, "pq_executing": 6, "pq_executing_high_water": 6, "pq_executing_ops": 6, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 140 }, "throttle-msgr_dispatch_throttler-mds": { "get": 9225, "get_or_fail_fail": 0, "get_or_fail_success": 9225, "get_started": 0, "get_sum": 2581320, "max": 104857600, "put": 9224, "put_sum": 2581023, "take": 0, "take_sum": 0, "val": 297, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 646, "put_sum": 19697860, "take": 651, "take_sum": 19697860, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 646, "put_sum": 646, "take": 652, "take_sum": 652, "val": 6, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 679, "get_or_fail_fail": 0, "get_or_fail_success": 679, "get_started": 0, "get_sum": 68579, "max": 3758096384, "put": 9, "put_sum": 68579, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55b1d3c8aa00": { "get": 9196, "get_or_fail_fail": 0, "get_or_fail_success": 9196, "get_started": 0, "get_sum": 19328931, "max": 3758096384, "put": 17, "put_sum": 19168621, "take": 0, "take_sum": 0, "val": 160310, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:46.228 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 6/1175, 542/1854 2024-03-06T07:26:47.229 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:47.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.385+0000 7f5972c15640 1 Processor -- start 2024-03-06T07:26:47.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.385+0000 7f5972c15640 1 -- start start 2024-03-06T07:26:47.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.386+0000 7f5972c15640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f596c162550 0x7f596c0ccc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:47.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.386+0000 7f5972c15640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f596c162930 0x7f596c0b5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:47.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.386+0000 7f5972c15640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f596c0cd2f0 con 0x7f596c06af80 2024-03-06T07:26:47.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.386+0000 7f5972c15640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f596c06efa0 con 0x7f596c162930 2024-03-06T07:26:47.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.386+0000 7f5972c15640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f596c0cd5d0 con 0x7f596c162550 2024-03-06T07:26:47.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.386+0000 7f596bfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f596c162930 0x7f596c0b5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:47.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.386+0000 7f596bfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f596c162930 0x7f596c0b5c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53294/0 (socket says 172.21.15.83:53294) 2024-03-06T07:26:47.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.386+0000 7f596bfff640 1 -- 172.21.15.83:0/2584845207 learned_addr learned my addr 172.21.15.83:0/2584845207 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:47.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.386+0000 7f59709b1640 1 --2- 172.21.15.83:0/2584845207 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f596c162550 0x7f596c0ccc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:47.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.386+0000 7f596bfff640 1 -- 172.21.15.83:0/2584845207 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f596c162550 msgr2=0x7f596c0ccc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:47.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.386+0000 7f596bfff640 1 --2- 172.21.15.83:0/2584845207 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f596c162550 0x7f596c0ccc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.386+0000 7f596bfff640 1 -- 172.21.15.83:0/2584845207 >> v1:172.21.15.183:6789/0 conn(0x7f596c06af80 legacy=0x7f596c0cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:47.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.386+0000 7f596bfff640 1 -- 172.21.15.83:0/2584845207 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f596c0cd8b0 con 0x7f596c162930 2024-03-06T07:26:47.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.386+0000 7f59709b1640 1 --2- 172.21.15.83:0/2584845207 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f596c162550 0x7f596c0ccc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:47.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.387+0000 7f596bfff640 1 --2- 172.21.15.83:0/2584845207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f596c162930 0x7f596c0b5c20 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f5958009870 tx=0x7f595802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10951c319887cd78 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:47.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.387+0000 7f596b7fe640 1 -- 172.21.15.83:0/2584845207 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5958034070 con 0x7f596c162930 2024-03-06T07:26:47.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.387+0000 7f596b7fe640 1 -- 172.21.15.83:0/2584845207 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5958030840 con 0x7f596c162930 2024-03-06T07:26:47.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.387+0000 7f596b7fe640 1 -- 172.21.15.83:0/2584845207 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5958030c10 con 0x7f596c162930 2024-03-06T07:26:47.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.388+0000 7f5972c15640 1 -- 172.21.15.83:0/2584845207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f596c162930 msgr2=0x7f596c0b5c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:47.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.388+0000 7f5972c15640 1 --2- 172.21.15.83:0/2584845207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f596c162930 0x7f596c0b5c20 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f5958009870 tx=0x7f595802b980 comp rx=0 tx=0).stop 2024-03-06T07:26:47.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.388+0000 7f5972c15640 1 -- 172.21.15.83:0/2584845207 shutdown_connections 2024-03-06T07:26:47.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.388+0000 7f5972c15640 1 --2- 172.21.15.83:0/2584845207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f596c162930 0x7f596c0b5c20 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.388+0000 7f5972c15640 1 --2- 172.21.15.83:0/2584845207 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f596c162550 0x7f596c0ccc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.388+0000 7f5972c15640 1 -- 172.21.15.83:0/2584845207 >> 172.21.15.83:0/2584845207 conn(0x7f596c072800 msgr2=0x7f596c072c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:47.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.388+0000 7f5972c15640 1 -- 172.21.15.83:0/2584845207 shutdown_connections 2024-03-06T07:26:47.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.388+0000 7f5972c15640 1 -- 172.21.15.83:0/2584845207 wait complete. 2024-03-06T07:26:47.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.390+0000 7f5972c15640 1 Processor -- start 2024-03-06T07:26:47.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.390+0000 7f5972c15640 1 -- start start 2024-03-06T07:26:47.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.391+0000 7f5972c15640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f596c06af80 0x7f596c1602d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:47.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.391+0000 7f59709b1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f596c06af80 0x7f596c1602d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:47.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.391+0000 7f59709b1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f596c06af80 0x7f596c1602d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53306/0 (socket says 172.21.15.83:53306) 2024-03-06T07:26:47.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.391+0000 7f59709b1640 1 -- 172.21.15.83:0/2465018768 learned_addr learned my addr 172.21.15.83:0/2465018768 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:47.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.391+0000 7f5972c15640 1 --2- 172.21.15.83:0/2465018768 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f596c162550 0x7f596c160810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:47.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.391+0000 7f5972c15640 1 --2- 172.21.15.83:0/2465018768 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f596c162930 0x7f596c0b07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:47.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.391+0000 7f5972c15640 1 -- 172.21.15.83:0/2465018768 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596c06d970 con 0x7f596c06af80 2024-03-06T07:26:47.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.391+0000 7f5972c15640 1 -- 172.21.15.83:0/2465018768 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596c0caa30 con 0x7f596c162930 2024-03-06T07:26:47.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.391+0000 7f5972c15640 1 -- 172.21.15.83:0/2465018768 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f596c06daf0 con 0x7f596c162550 2024-03-06T07:26:47.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.391+0000 7f596bfff640 1 --2- 172.21.15.83:0/2465018768 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f596c162550 0x7f596c160810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:47.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.391+0000 7f59711b2640 1 --2- 172.21.15.83:0/2465018768 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f596c162930 0x7f596c0b07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:47.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.392+0000 7f59709b1640 1 -- 172.21.15.83:0/2465018768 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f596c162550 msgr2=0x7f596c160810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:47.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.392+0000 7f59709b1640 1 --2- 172.21.15.83:0/2465018768 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f596c162550 0x7f596c160810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.392+0000 7f59709b1640 1 -- 172.21.15.83:0/2465018768 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f596c162930 msgr2=0x7f596c0b07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:47.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.392+0000 7f59709b1640 1 --2- 172.21.15.83:0/2465018768 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f596c162930 0x7f596c0b07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.392+0000 7f59709b1640 1 -- 172.21.15.83:0/2465018768 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f596c1f8980 con 0x7f596c06af80 2024-03-06T07:26:47.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.392+0000 7f59711b2640 1 --2- 172.21.15.83:0/2465018768 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f596c162930 0x7f596c0b07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:47.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.392+0000 7f59709b1640 1 --2- 172.21.15.83:0/2465018768 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f596c06af80 0x7f596c1602d0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f596000e7e0 tx=0x7f596000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:47.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.392+0000 7f59697fa640 1 -- 172.21.15.83:0/2465018768 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596000f040 con 0x7f596c06af80 2024-03-06T07:26:47.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.392+0000 7f59697fa640 1 -- 172.21.15.83:0/2465018768 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5960004030 con 0x7f596c06af80 2024-03-06T07:26:47.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.392+0000 7f5972c15640 1 -- 172.21.15.83:0/2465018768 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f596c163020 con 0x7f596c06af80 2024-03-06T07:26:47.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.392+0000 7f59697fa640 1 -- 172.21.15.83:0/2465018768 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5960004400 con 0x7f596c06af80 2024-03-06T07:26:47.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.392+0000 7f5972c15640 1 -- 172.21.15.83:0/2465018768 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f596c163b00 con 0x7f596c06af80 2024-03-06T07:26:47.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.393+0000 7f5972c15640 1 -- 172.21.15.83:0/2465018768 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f596c068050 con 0x7f596c06af80 2024-03-06T07:26:47.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.395+0000 7f59697fa640 1 -- 172.21.15.83:0/2465018768 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f596000c6d0 con 0x7f596c06af80 2024-03-06T07:26:47.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.395+0000 7f59697fa640 1 --2- 172.21.15.83:0/2465018768 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f595006b520 0x7f595006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:47.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.395+0000 7f59697fa640 1 -- 172.21.15.83:0/2465018768 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f5960089e00 con 0x7f596c06af80 2024-03-06T07:26:47.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.395+0000 7f596bfff640 1 --2- 172.21.15.83:0/2465018768 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f595006b520 0x7f595006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:47.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.396+0000 7f596bfff640 1 --2- 172.21.15.83:0/2465018768 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f595006b520 0x7f595006d9e0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f5958036000 tx=0x7f5958008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:47.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.403+0000 7f59697fa640 1 -- 172.21.15.83:0/2465018768 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5960017020 con 0x7f596c06af80 2024-03-06T07:26:47.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.736+0000 7f5972c15640 1 -- 172.21.15.83:0/2465018768 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f596c1636e0 con 0x7f596c06af80 2024-03-06T07:26:47.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.736+0000 7f59697fa640 1 -- 172.21.15.83:0/2465018768 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f5960058740 con 0x7f596c06af80 2024-03-06T07:26:47.739 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:47.739 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":53,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6023,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/601236077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":601236077},{"type":"v1","addr":"172.21.15.183:6833","nonce":601236077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6029,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1435055961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1435055961},{"type":"v1","addr":"172.21.15.183:6835","nonce":1435055961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":7572,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/2726045966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2726045966},{"type":"v1","addr":"172.21.15.83:6836","nonce":2726045966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.064335+0000","modified":"2024-03-06T07:26:32.814889+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7575":{"gid":7575,"name":"a","rank":0,"incarnation":49,"state":"up:active","state_seq":5,"addr":"172.21.15.83:6837/3254575964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":3254575964},{"type":"v1","addr":"172.21.15.83:6837","nonce":3254575964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7575,"qdb_cluster":[7575]},"id":7}]} 2024-03-06T07:26:47.740 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 53 2024-03-06T07:26:47.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.738+0000 7f5972c15640 1 -- 172.21.15.83:0/2465018768 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f595006b520 msgr2=0x7f595006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:47.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.738+0000 7f5972c15640 1 --2- 172.21.15.83:0/2465018768 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f595006b520 0x7f595006d9e0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f5958036000 tx=0x7f5958008030 comp rx=0 tx=0).stop 2024-03-06T07:26:47.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.738+0000 7f5972c15640 1 -- 172.21.15.83:0/2465018768 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f596c06af80 msgr2=0x7f596c1602d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:47.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.738+0000 7f5972c15640 1 --2- 172.21.15.83:0/2465018768 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f596c06af80 0x7f596c1602d0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f596000e7e0 tx=0x7f596000eca0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.739+0000 7f5972c15640 1 -- 172.21.15.83:0/2465018768 shutdown_connections 2024-03-06T07:26:47.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.739+0000 7f5972c15640 1 --2- 172.21.15.83:0/2465018768 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f595006b520 0x7f595006d9e0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.739+0000 7f5972c15640 1 --2- 172.21.15.83:0/2465018768 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f596c162930 0x7f596c0b07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.739+0000 7f5972c15640 1 --2- 172.21.15.83:0/2465018768 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f596c162550 0x7f596c160810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.739+0000 7f5972c15640 1 --2- 172.21.15.83:0/2465018768 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f596c06af80 0x7f596c1602d0 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.739+0000 7f5972c15640 1 -- 172.21.15.83:0/2465018768 >> 172.21.15.83:0/2465018768 conn(0x7f596c072800 msgr2=0x7f596c0cc2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:47.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.739+0000 7f5972c15640 1 -- 172.21.15.83:0/2465018768 shutdown_connections 2024-03-06T07:26:47.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:47.739+0000 7f5972c15640 1 -- 172.21.15.83:0/2465018768 wait complete. 2024-03-06T07:26:47.754 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:26:47.919 INFO:teuthology.orchestra.run.smithi083.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":534,"msgr_send_messages":534,"msgr_recv_bytes":144861,"msgr_send_bytes":4389748,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.085782529,"msgr_running_send_time":0.050242017,"msgr_running_recv_time":0.015216965,"msgr_running_fast_dispatch_time":0.016050401,"msgr_send_messages_queue_lat":{"avgcount":533,"sum":0.046965997,"avgtime":0.000088116},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000209,"avgtime":0.000000104},"msgr_recv_encrypted_bytes":1568,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":9796,"msgr_send_messages":22323,"msgr_recv_bytes":3615794,"msgr_send_bytes":27022863,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.977210789,"msgr_running_send_time":0.553594283,"msgr_running_recv_time":0.481183142,"msgr_running_fast_dispatch_time":0.019666563,"msgr_send_messages_queue_lat":{"avgcount":22323,"sum":0.364759420,"avgtime":0.000016340},"msgr_handle_ack_lat":{"avgcount":28904,"sum":0.005039829,"avgtime":0.000000174},"msgr_recv_encrypted_bytes":108048,"msgr_send_encrypted_bytes":7920},"AsyncMessenger::Worker-2":{"msgr_recv_messages":369,"msgr_send_messages":375,"msgr_recv_bytes":211202,"msgr_send_bytes":373176,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.044810417,"msgr_running_send_time":0.022884714,"msgr_running_recv_time":0.010259837,"msgr_running_fast_dispatch_time":0.008889283,"msgr_send_messages_queue_lat":{"avgcount":374,"sum":0.029009056,"avgtime":0.000077564},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":94016},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":889,"complete_latency":{"avgcount":90,"sum":5.903619193,"avgtime":0.065595768}},"finisher-PurgeQueue":{"queue_len":32,"complete_latency":{"avgcount":1766,"sum":1.079997057,"avgtime":0.000611549}},"mds":{"request":5935,"reply":5932,"reply_latency":{"avgcount":5932,"sum":8.622140240,"avgtime":0.001453496},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":5218,"inodes_top":0,"inodes_bottom":5218,"inodes_pin_tail":0,"inodes_pinned":5218,"inodes_expired":0,"inodes_with_caps":1630,"caps":1630,"subtrees":2,"traverse":5941,"traverse_hit":5940,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":1370,"root_rbytes":10646528,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":938,"ceph_cap_op_grant":6651,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3470,"handle_client_caps_dirty":3000,"handle_client_cap_release":5,"process_request_cap_release":8647},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1576,"num_strays_delayed":0,"num_strays_enqueuing":425,"strays_created":2723,"strays_enqueued":1147,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":10534,"evex":0,"evtrm":0,"ev":10537,"evexg":0,"evexd":0,"segadd":13,"segex":0,"segtrm":0,"seg":15,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":26570755,"rdpos":4195244,"jlat":{"avgcount":18,"sum":3.177238838,"avgtime":0.176513268},"replayed":3},"mds_mem":{"ino":2500,"ino+":3214,"ino-":714,"dir":13,"dir+":13,"dir-":0,"dn":5219,"dn+":5933,"dn-":714,"cap":1630,"cap+":3202,"cap-":1572,"rss":97284,"heap":248164},"mds_server":{"dispatch_client_request":5938,"dispatch_server_request":0,"handle_client_request":5935,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.114640424,"avgtime":0.000973325},"req_getattr_latency":{"avgcount":6,"sum":0.197880257,"avgtime":0.032980042},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002834805,"avgtime":0.002834805},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.120240153,"avgtime":0.040080051},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":2722,"sum":5.186544601,"avgtime":0.001905416},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5457,"average_load":5457,"avg_session_uptime":769,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":2348911,"buffer_anon_items":1195,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":15263669,"mds_co_items":113352,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":1241,"op_send_bytes":21525736,"op_resend":0,"op_reply":1241,"op_latency":{"avgcount":1241,"sum":2.302946150,"avgtime":0.001855718},"op_inflight":0,"oplen_avg":{"avgcount":1241,"sum":1306},"op":1241,"op_r":50,"op_w":1191,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":22,"osdop_write":33,"osdop_writefull":7,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":1137,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":25,"omap_rd":48,"omap_del":7},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":1121,"pq_executed":1121,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":10694,"get_sum":3006509,"get_or_fail_fail":0,"get_or_fail_success":10694,"take":0,"take_sum":0,"put":10694,"put_sum":3006509,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1241,"take_sum":21643104,"put":1241,"put_sum":21643104,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1241,"take_sum":1241,"put":1241,"put_sum":1241,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1572,"get_sum":158772,"get_or_fail_fail":0,"get_or_fail_success":1572,"take":0,"take_sum":0,"put":15,"put_sum":158772,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55b1d3c8aa00":{"val":1408117,"max":3758096384,"get_started":0,"get":10535,"get_sum":22378256,"get_or_fail_fail":0,"get_or_fail_success":10535,"take":0,"take_sum":0,"put":18,"put_sum":20970139,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:47.921 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.04e-07, "sum": 2.09e-07 }, "msgr_recv_bytes": 144861, "msgr_recv_encrypted_bytes": 1568, "msgr_recv_messages": 534, "msgr_running_fast_dispatch_time": 0.016050401, "msgr_running_recv_time": 0.015216965, "msgr_running_send_time": 0.050242017, "msgr_running_total_time": 0.085782529, "msgr_send_bytes": 4389748, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 534, "msgr_send_messages_queue_lat": { "avgcount": 533, "avgtime": 8.8116e-05, "sum": 0.046965997 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 28904, "avgtime": 1.74e-07, "sum": 0.005039829 }, "msgr_recv_bytes": 3615794, "msgr_recv_encrypted_bytes": 108048, "msgr_recv_messages": 9796, "msgr_running_fast_dispatch_time": 0.019666563, "msgr_running_recv_time": 0.481183142, "msgr_running_send_time": 0.553594283, "msgr_running_total_time": 0.977210789, "msgr_send_bytes": 27022863, "msgr_send_encrypted_bytes": 7920, "msgr_send_messages": 22323, "msgr_send_messages_queue_lat": { "avgcount": 22323, "avgtime": 1.634e-05, "sum": 0.36475942 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 211202, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 369, "msgr_running_fast_dispatch_time": 0.008889283, "msgr_running_recv_time": 0.010259837, "msgr_running_send_time": 0.022884714, "msgr_running_total_time": 0.044810417, "msgr_send_bytes": 373176, "msgr_send_encrypted_bytes": 94016, "msgr_send_messages": 375, "msgr_send_messages_queue_lat": { "avgcount": 374, "avgtime": 7.7564e-05, "sum": 0.029009056 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 90, "avgtime": 0.065595768, "sum": 5.903619193 }, "queue_len": 889 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 1766, "avgtime": 0.000611549, "sum": 1.079997057 }, "queue_len": 32 }, "mds": { "caps": 1630, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 6651, "ceph_cap_op_revoke": 938, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 5, "handle_client_caps": 3470, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 5218, "inodes_bottom": 5218, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 5218, "inodes_top": 0, "inodes_with_caps": 1630, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 8647, "q": 0, "reply": 5932, "reply_latency": { "avgcount": 5932, "avgtime": 0.001453496, "sum": 8.62214024 }, "request": 5935, "root_rbytes": 10646528, "root_rfiles": 1370, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 5941, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 5940, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1576, "num_strays_delayed": 0, "num_strays_enqueuing": 425, "recovery_completed": 0, "recovery_started": 0, "strays_created": 2723, "strays_enqueued": 1147, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 10537, "evadd": 10534, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 18, "avgtime": 0.176513268, "sum": 3.177238838 }, "rdpos": 4195244, "replayed": 3, "seg": 15, "segadd": 13, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 26570755 }, "mds_mem": { "cap": 1630, "cap+": 3202, "cap-": 1572, "dir": 13, "dir+": 13, "dir-": 0, "dn": 5219, "dn+": 5933, "dn-": 714, "heap": 248164, "ino": 2500, "ino+": 3214, "ino-": 714, "rss": 97284 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 5938, "dispatch_server_request": 0, "handle_client_request": 5935, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000973325, "sum": 3.114640424 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.032980042, "sum": 0.197880257 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002834805, "sum": 0.002834805 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.040080051, "sum": 0.120240153 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 2722, "avgtime": 0.001905416, "sum": 5.186544601 } }, "mds_sessions": { "average_load": 5457, "avg_session_uptime": 769, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5457 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 2348911, "buffer_anon_items": 1195, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 15263669, "mds_co_items": 113352, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 7, "omap_rd": 48, "omap_wr": 25, "op": 1241, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 1241, "avgtime": 0.001855718, "sum": 2.30294615 }, "op_pg": 0, "op_r": 50, "op_reply": 1241, "op_resend": 0, "op_rmw": 0, "op_send": 1241, "op_send_bytes": 21525736, "op_w": 1191, "oplen_avg": { "avgcount": 1241, "sum": 1306 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 1137, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 22, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 33, "osdop_writefull": 7, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 1121, "pq_executed_ops": 1121, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 10694, "get_or_fail_fail": 0, "get_or_fail_success": 10694, "get_started": 0, "get_sum": 3006509, "max": 104857600, "put": 10694, "put_sum": 3006509, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 1241, "put_sum": 21643104, "take": 1241, "take_sum": 21643104, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 1241, "put_sum": 1241, "take": 1241, "take_sum": 1241, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1572, "get_or_fail_fail": 0, "get_or_fail_success": 1572, "get_started": 0, "get_sum": 158772, "max": 3758096384, "put": 15, "put_sum": 158772, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55b1d3c8aa00": { "get": 10535, "get_or_fail_fail": 0, "get_or_fail_success": 10535, "get_started": 0, "get_sum": 22378256, "max": 3758096384, "put": 18, "put_sum": 20970139, "take": 0, "take_sum": 0, "val": 1408117, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:47.921 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/1576, 1121/2723 2024-03-06T07:26:48.921 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:49.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.037+0000 7f5642c2a640 1 Processor -- start 2024-03-06T07:26:49.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.037+0000 7f5642c2a640 1 -- start start 2024-03-06T07:26:49.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.038+0000 7f5642c2a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f563c0d0950 0x7f563c0d0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.038+0000 7f5642c2a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f563c0b9060 0x7f563c0b94d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.038+0000 7f5642c2a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f563c0d1ea0 con 0x7f563c15e150 2024-03-06T07:26:49.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.038+0000 7f5642c2a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f563c0d12f0 con 0x7f563c0d0950 2024-03-06T07:26:49.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.038+0000 7f5642c2a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f563c0d2ab0 con 0x7f563c0b9060 2024-03-06T07:26:49.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.038+0000 7f563bfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f563c0d0950 0x7f563c0d0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.038+0000 7f563bfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f563c0d0950 0x7f563c0d0db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53308/0 (socket says 172.21.15.83:53308) 2024-03-06T07:26:49.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.038+0000 7f563bfff640 1 -- 172.21.15.83:0/2842059428 learned_addr learned my addr 172.21.15.83:0/2842059428 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:49.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.039+0000 7f56411c7640 1 --2- 172.21.15.83:0/2842059428 >> v2:172.21.15.183:3300/0 conn(0x7f563c0b9060 0x7f563c0b94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.039+0000 7f563bfff640 1 -- 172.21.15.83:0/2842059428 >> v2:172.21.15.183:3300/0 conn(0x7f563c0b9060 msgr2=0x7f563c0b94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:49.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.039+0000 7f563bfff640 1 --2- 172.21.15.83:0/2842059428 >> v2:172.21.15.183:3300/0 conn(0x7f563c0b9060 0x7f563c0b94d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.039+0000 7f563bfff640 1 -- 172.21.15.83:0/2842059428 >> v1:172.21.15.183:6789/0 conn(0x7f563c15e150 legacy=0x7f563c0cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:49.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.039+0000 7f563bfff640 1 -- 172.21.15.83:0/2842059428 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f563c0d2180 con 0x7f563c0d0950 2024-03-06T07:26:49.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.039+0000 7f56411c7640 1 --2- 172.21.15.83:0/2842059428 >> v2:172.21.15.183:3300/0 conn(0x7f563c0b9060 0x7f563c0b94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:49.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.040+0000 7f563bfff640 1 --2- 172.21.15.83:0/2842059428 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f563c0d0950 0x7f563c0d0db0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f563c08c8d0 tx=0x7f562802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44f28a328ad5176b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:49.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.040+0000 7f563b7fe640 1 -- 172.21.15.83:0/2842059428 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f562802e070 con 0x7f563c0d0950 2024-03-06T07:26:49.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.040+0000 7f563b7fe640 1 -- 172.21.15.83:0/2842059428 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5628037070 con 0x7f563c0d0950 2024-03-06T07:26:49.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.040+0000 7f563b7fe640 1 -- 172.21.15.83:0/2842059428 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5628031a00 con 0x7f563c0d0950 2024-03-06T07:26:49.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.041+0000 7f5642c2a640 1 -- 172.21.15.83:0/2842059428 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f563c0d0950 msgr2=0x7f563c0d0db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.041+0000 7f5642c2a640 1 --2- 172.21.15.83:0/2842059428 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f563c0d0950 0x7f563c0d0db0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f563c08c8d0 tx=0x7f562802b880 comp rx=0 tx=0).stop 2024-03-06T07:26:49.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.041+0000 7f5642c2a640 1 -- 172.21.15.83:0/2842059428 shutdown_connections 2024-03-06T07:26:49.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.041+0000 7f5642c2a640 1 --2- 172.21.15.83:0/2842059428 >> v2:172.21.15.183:3300/0 conn(0x7f563c0b9060 0x7f563c0b94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.041+0000 7f5642c2a640 1 --2- 172.21.15.83:0/2842059428 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f563c0d0950 0x7f563c0d0db0 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.041+0000 7f5642c2a640 1 -- 172.21.15.83:0/2842059428 >> 172.21.15.83:0/2842059428 conn(0x7f563c09d230 msgr2=0x7f563c0d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:49.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.041+0000 7f5642c2a640 1 -- 172.21.15.83:0/2842059428 shutdown_connections 2024-03-06T07:26:49.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.041+0000 7f5642c2a640 1 -- 172.21.15.83:0/2842059428 wait complete. 2024-03-06T07:26:49.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.043+0000 7f5642c2a640 1 Processor -- start 2024-03-06T07:26:49.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.043+0000 7f5642c2a640 1 -- start start 2024-03-06T07:26:49.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.044+0000 7f5642c2a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f563c0b9060 0x7f563c151460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.044+0000 7f5642c2a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f563c0d0950 0x7f563c0ccd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.044+0000 7f5642c2a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f563c15e150 0x7f563c0b6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.044+0000 7f5642c2a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563c0d2f40 con 0x7f563c15e150 2024-03-06T07:26:49.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.044+0000 7f5642c2a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563c0d2740 con 0x7f563c0b9060 2024-03-06T07:26:49.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.044+0000 7f5642c2a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f563c0d1b30 con 0x7f563c0d0950 2024-03-06T07:26:49.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.044+0000 7f56411c7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f563c15e150 0x7f563c0b6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.044+0000 7f56411c7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f563c15e150 0x7f563c0b6da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53310/0 (socket says 172.21.15.83:53310) 2024-03-06T07:26:49.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.044+0000 7f56411c7640 1 -- 172.21.15.83:0/1030176459 learned_addr learned my addr 172.21.15.83:0/1030176459 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:49.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.044+0000 7f563bfff640 1 --2- 172.21.15.83:0/1030176459 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f563c0d0950 0x7f563c0ccd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.044+0000 7f56409c6640 1 --2- 172.21.15.83:0/1030176459 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f563c0b9060 0x7f563c151460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.044+0000 7f56411c7640 1 -- 172.21.15.83:0/1030176459 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f563c0d0950 msgr2=0x7f563c0ccd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.044+0000 7f56411c7640 1 --2- 172.21.15.83:0/1030176459 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f563c0d0950 0x7f563c0ccd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.044+0000 7f56411c7640 1 -- 172.21.15.83:0/1030176459 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f563c0b9060 msgr2=0x7f563c151460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.044+0000 7f56411c7640 1 --2- 172.21.15.83:0/1030176459 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f563c0b9060 0x7f563c151460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.044+0000 7f56411c7640 1 -- 172.21.15.83:0/1030176459 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f563c0c8620 con 0x7f563c15e150 2024-03-06T07:26:49.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.046+0000 7f56411c7640 1 --2- 172.21.15.83:0/1030176459 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f563c15e150 0x7f563c0b6da0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f563400cef0 tx=0x7f563400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:49.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.046+0000 7f56397fa640 1 -- 172.21.15.83:0/1030176459 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563401a070 con 0x7f563c15e150 2024-03-06T07:26:49.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.046+0000 7f56397fa640 1 -- 172.21.15.83:0/1030176459 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f563400d860 con 0x7f563c15e150 2024-03-06T07:26:49.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.046+0000 7f56397fa640 1 -- 172.21.15.83:0/1030176459 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563400dc50 con 0x7f563c15e150 2024-03-06T07:26:49.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.046+0000 7f5642c2a640 1 -- 172.21.15.83:0/1030176459 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f563c0c8900 con 0x7f563c15e150 2024-03-06T07:26:49.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.046+0000 7f5642c2a640 1 -- 172.21.15.83:0/1030176459 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f563c0c8ec0 con 0x7f563c15e150 2024-03-06T07:26:49.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.046+0000 7f5642c2a640 1 -- 172.21.15.83:0/1030176459 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f563c069ee0 con 0x7f563c15e150 2024-03-06T07:26:49.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.046+0000 7f56397fa640 1 -- 172.21.15.83:0/1030176459 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f563401d070 con 0x7f563c15e150 2024-03-06T07:26:49.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.047+0000 7f56397fa640 1 --2- 172.21.15.83:0/1030176459 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f562006b4d0 0x7f562006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.047+0000 7f56409c6640 1 --2- 172.21.15.83:0/1030176459 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f562006b4d0 0x7f562006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.047+0000 7f56397fa640 1 -- 172.21.15.83:0/1030176459 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f563408aaf0 con 0x7f563c15e150 2024-03-06T07:26:49.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.047+0000 7f56409c6640 1 --2- 172.21.15.83:0/1030176459 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f562006b4d0 0x7f562006d990 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f5630006fd0 tx=0x7f5630008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:49.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.052+0000 7f56397fa640 1 -- 172.21.15.83:0/1030176459 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f56340599d0 con 0x7f563c15e150 2024-03-06T07:26:49.156 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:49.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.307+0000 7f8c83237640 1 Processor -- start 2024-03-06T07:26:49.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.307+0000 7f8c83237640 1 -- start start 2024-03-06T07:26:49.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.309+0000 7f8c83237640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f8c7c15e150 0x7f8c7c0cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.309+0000 7f8c83237640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c7c0d1ea0 con 0x7f8c7c0cfba0 2024-03-06T07:26:49.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.309+0000 7f8c83237640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c7c0d2180 con 0x7f8c7c0d1060 2024-03-06T07:26:49.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.309+0000 7f8c83237640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c7c0d2ab0 con 0x7f8c7c15e150 2024-03-06T07:26:49.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.309+0000 7f8c80fd3640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f8c7c15e150 0x7f8c7c0cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.309+0000 7f8c7bfff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f8c7c0cfba0 0x7f8c7c0d0950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:33402/0 (socket says 172.21.15.83:33402) 2024-03-06T07:26:49.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.309+0000 7f8c7bfff640 1 -- 172.21.15.83:0/1756347085 learned_addr learned my addr 172.21.15.83:0/1756347085 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:49.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.309+0000 7f8c80fd3640 1 -- 172.21.15.83:0/1756347085 >> v1:172.21.15.183:6789/0 conn(0x7f8c7c0d1060 legacy=0x7f8c7c0b9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:49.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.309+0000 7f8c80fd3640 1 -- 172.21.15.83:0/1756347085 >> v1:172.21.15.83:6789/0 conn(0x7f8c7c0cfba0 legacy=0x7f8c7c0d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.309+0000 7f8c80fd3640 1 -- 172.21.15.83:0/1756347085 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c7c0d2460 con 0x7f8c7c15e150 2024-03-06T07:26:49.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.310+0000 7f8c80fd3640 1 --2- 172.21.15.83:0/1756347085 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8c7c15e150 0x7f8c7c0cf660 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f8c700097a0 tx=0x7f8c7002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c879c11bc1064293 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:49.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.310+0000 7f8c7b7fe640 1 -- 172.21.15.83:0/1756347085 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c700315a0 con 0x7f8c7c15e150 2024-03-06T07:26:49.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.310+0000 7f8c7b7fe640 1 -- 172.21.15.83:0/1756347085 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c70037070 con 0x7f8c7c15e150 2024-03-06T07:26:49.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.310+0000 7f8c7b7fe640 1 -- 172.21.15.83:0/1756347085 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c70031960 con 0x7f8c7c15e150 2024-03-06T07:26:49.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.310+0000 7f8c83237640 1 -- 172.21.15.83:0/1756347085 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8c7c15e150 msgr2=0x7f8c7c0cf660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.310+0000 7f8c83237640 1 --2- 172.21.15.83:0/1756347085 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8c7c15e150 0x7f8c7c0cf660 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f8c700097a0 tx=0x7f8c7002b840 comp rx=0 tx=0).stop 2024-03-06T07:26:49.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.311+0000 7f8c83237640 1 -- 172.21.15.83:0/1756347085 shutdown_connections 2024-03-06T07:26:49.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.311+0000 7f8c83237640 1 --2- 172.21.15.83:0/1756347085 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8c7c15e150 0x7f8c7c0cf660 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.311+0000 7f8c83237640 1 -- 172.21.15.83:0/1756347085 >> 172.21.15.83:0/1756347085 conn(0x7f8c7c09d230 msgr2=0x7f8c7c0d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:49.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.311+0000 7f8c83237640 1 -- 172.21.15.83:0/1756347085 shutdown_connections 2024-03-06T07:26:49.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.311+0000 7f8c83237640 1 -- 172.21.15.83:0/1756347085 wait complete. 2024-03-06T07:26:49.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.312+0000 7f8c83237640 1 Processor -- start 2024-03-06T07:26:49.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.312+0000 7f8c83237640 1 -- start start 2024-03-06T07:26:49.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.313+0000 7f8c83237640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8c7c0cfba0 0x7f8c7c15a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.313+0000 7f8c83237640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c7c0d1060 0x7f8c7c0c8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.313+0000 7f8c83237640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8c7c15e150 0x7f8c7c0cd200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.313+0000 7f8c83237640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c0b7dc0 con 0x7f8c7c0cfba0 2024-03-06T07:26:49.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.313+0000 7f8c83237640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c0d2f40 con 0x7f8c7c0d1060 2024-03-06T07:26:49.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.313+0000 7f8c83237640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c0d1b30 con 0x7f8c7c15e150 2024-03-06T07:26:49.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.313+0000 7f8c80fd3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8c7c0cfba0 0x7f8c7c15a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.313+0000 7f8c80fd3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8c7c0cfba0 0x7f8c7c15a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53328/0 (socket says 172.21.15.83:53328) 2024-03-06T07:26:49.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.313+0000 7f8c80fd3640 1 -- 172.21.15.83:0/1580706593 learned_addr learned my addr 172.21.15.83:0/1580706593 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:49.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.313+0000 7f8c7bfff640 1 --2- 172.21.15.83:0/1580706593 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c7c0d1060 0x7f8c7c0c8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.313+0000 7f8c817d4640 1 --2- 172.21.15.83:0/1580706593 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8c7c15e150 0x7f8c7c0cd200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.313+0000 7f8c80fd3640 1 -- 172.21.15.83:0/1580706593 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8c7c15e150 msgr2=0x7f8c7c0cd200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.313+0000 7f8c80fd3640 1 --2- 172.21.15.83:0/1580706593 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8c7c15e150 0x7f8c7c0cd200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.313+0000 7f8c80fd3640 1 -- 172.21.15.83:0/1580706593 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c7c0d1060 msgr2=0x7f8c7c0c8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.313+0000 7f8c80fd3640 1 --2- 172.21.15.83:0/1580706593 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c7c0d1060 0x7f8c7c0c8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.313+0000 7f8c80fd3640 1 -- 172.21.15.83:0/1580706593 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c7c0c9920 con 0x7f8c7c0cfba0 2024-03-06T07:26:49.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.313+0000 7f8c817d4640 1 --2- 172.21.15.83:0/1580706593 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8c7c15e150 0x7f8c7c0cd200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:49.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.313+0000 7f8c80fd3640 1 --2- 172.21.15.83:0/1580706593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8c7c0cfba0 0x7f8c7c15a2a0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f8c70000c00 tx=0x7f8c700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:49.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.313+0000 7f8c797fa640 1 -- 172.21.15.83:0/1580706593 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c70032840 con 0x7f8c7c0cfba0 2024-03-06T07:26:49.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.313+0000 7f8c797fa640 1 -- 172.21.15.83:0/1580706593 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c700329e0 con 0x7f8c7c0cfba0 2024-03-06T07:26:49.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.313+0000 7f8c83237640 1 -- 172.21.15.83:0/1580706593 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c7c0c9c00 con 0x7f8c7c0cfba0 2024-03-06T07:26:49.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.313+0000 7f8c797fa640 1 -- 172.21.15.83:0/1580706593 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c70032dd0 con 0x7f8c7c0cfba0 2024-03-06T07:26:49.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.314+0000 7f8c83237640 1 -- 172.21.15.83:0/1580706593 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c7c0ca4a0 con 0x7f8c7c0cfba0 2024-03-06T07:26:49.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.315+0000 7f8c83237640 1 -- 172.21.15.83:0/1580706593 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c7c069ee0 con 0x7f8c7c0cfba0 2024-03-06T07:26:49.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.315+0000 7f8c797fa640 1 -- 172.21.15.83:0/1580706593 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8c7003b3f0 con 0x7f8c7c0cfba0 2024-03-06T07:26:49.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.315+0000 7f8c797fa640 1 --2- 172.21.15.83:0/1580706593 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8c6006b4d0 0x7f8c6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.315+0000 7f8c797fa640 1 -- 172.21.15.83:0/1580706593 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f8c7002f030 con 0x7f8c7c0cfba0 2024-03-06T07:26:49.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.315+0000 7f8c7bfff640 1 --2- 172.21.15.83:0/1580706593 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8c6006b4d0 0x7f8c6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.316+0000 7f8c7bfff640 1 --2- 172.21.15.83:0/1580706593 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8c6006b4d0 0x7f8c6006d990 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c0a1e40 tx=0x7f8c68007e80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:49.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.324+0000 7f8c797fa640 1 -- 172.21.15.83:0/1580706593 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8c70040070 con 0x7f8c7c0cfba0 2024-03-06T07:26:49.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.380+0000 7f5642c2a640 1 -- 172.21.15.83:0/1030176459 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f563c0d4670 con 0x7f563c15e150 2024-03-06T07:26:49.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.381+0000 7f56397fa640 1 -- 172.21.15.83:0/1030176459 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f56340592d0 con 0x7f563c15e150 2024-03-06T07:26:49.383 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:49.383 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":53,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6023,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/601236077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":601236077},{"type":"v1","addr":"172.21.15.183:6833","nonce":601236077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6029,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1435055961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1435055961},{"type":"v1","addr":"172.21.15.183:6835","nonce":1435055961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":7572,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/2726045966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2726045966},{"type":"v1","addr":"172.21.15.83:6836","nonce":2726045966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.064335+0000","modified":"2024-03-06T07:26:32.814889+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7575":{"gid":7575,"name":"a","rank":0,"incarnation":49,"state":"up:active","state_seq":5,"addr":"172.21.15.83:6837/3254575964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":3254575964},{"type":"v1","addr":"172.21.15.83:6837","nonce":3254575964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7575,"qdb_cluster":[7575]},"id":7}]} 2024-03-06T07:26:49.383 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 53 2024-03-06T07:26:49.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.384+0000 7f5642c2a640 1 -- 172.21.15.83:0/1030176459 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f562006b4d0 msgr2=0x7f562006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.384+0000 7f5642c2a640 1 --2- 172.21.15.83:0/1030176459 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f562006b4d0 0x7f562006d990 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f5630006fd0 tx=0x7f5630008040 comp rx=0 tx=0).stop 2024-03-06T07:26:49.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.384+0000 7f5642c2a640 1 -- 172.21.15.83:0/1030176459 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f563c15e150 msgr2=0x7f563c0b6da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.384+0000 7f5642c2a640 1 --2- 172.21.15.83:0/1030176459 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f563c15e150 0x7f563c0b6da0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f563400cef0 tx=0x7f563400a710 comp rx=0 tx=0).stop 2024-03-06T07:26:49.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.384+0000 7f5642c2a640 1 -- 172.21.15.83:0/1030176459 shutdown_connections 2024-03-06T07:26:49.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.384+0000 7f5642c2a640 1 --2- 172.21.15.83:0/1030176459 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f562006b4d0 0x7f562006d990 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.384+0000 7f5642c2a640 1 --2- 172.21.15.83:0/1030176459 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f563c15e150 0x7f563c0b6da0 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.384+0000 7f5642c2a640 1 --2- 172.21.15.83:0/1030176459 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f563c0d0950 0x7f563c0ccd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.384+0000 7f5642c2a640 1 --2- 172.21.15.83:0/1030176459 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f563c0b9060 0x7f563c151460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.384+0000 7f5642c2a640 1 -- 172.21.15.83:0/1030176459 >> 172.21.15.83:0/1030176459 conn(0x7f563c09d230 msgr2=0x7f563c06b9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:49.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.385+0000 7f5642c2a640 1 -- 172.21.15.83:0/1030176459 shutdown_connections 2024-03-06T07:26:49.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.385+0000 7f5642c2a640 1 -- 172.21.15.83:0/1030176459 wait complete. 2024-03-06T07:26:49.403 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:26:49.523 INFO:teuthology.orchestra.run.smithi083.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":982,"msgr_send_messages":982,"msgr_recv_bytes":258653,"msgr_send_bytes":4535796,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.134083570,"msgr_running_send_time":0.075158901,"msgr_running_recv_time":0.023735230,"msgr_running_fast_dispatch_time":0.027133826,"msgr_send_messages_queue_lat":{"avgcount":981,"sum":0.069801511,"avgtime":0.000071153},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000209,"avgtime":0.000000104},"msgr_recv_encrypted_bytes":1568,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":10547,"msgr_send_messages":24999,"msgr_recv_bytes":3884488,"msgr_send_bytes":27833441,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":1.072152909,"msgr_running_send_time":0.609113613,"msgr_running_recv_time":0.514858106,"msgr_running_fast_dispatch_time":0.026112498,"msgr_send_messages_queue_lat":{"avgcount":24999,"sum":0.414783138,"avgtime":0.000016591},"msgr_handle_ack_lat":{"avgcount":30793,"sum":0.005372236,"avgtime":0.000000174},"msgr_recv_encrypted_bytes":108048,"msgr_send_encrypted_bytes":7920},"AsyncMessenger::Worker-2":{"msgr_recv_messages":706,"msgr_send_messages":713,"msgr_recv_bytes":398815,"msgr_send_bytes":2400660,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.082709426,"msgr_running_send_time":0.040596534,"msgr_running_recv_time":0.017832125,"msgr_running_fast_dispatch_time":0.019278650,"msgr_send_messages_queue_lat":{"avgcount":712,"sum":0.048667337,"avgtime":0.000068353},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":95808},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":96,"sum":7.699752282,"avgtime":0.080205752}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":3425,"sum":1.671902658,"avgtime":0.000488146}},"mds":{"request":6413,"reply":6411,"reply_latency":{"avgcount":6411,"sum":9.604517880,"avgtime":0.001498131},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":4840,"inodes_top":0,"inodes_bottom":4840,"inodes_pin_tail":0,"inodes_pinned":4840,"inodes_expired":0,"inodes_with_caps":1040,"caps":1040,"subtrees":2,"traverse":6419,"traverse_hit":6418,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":306,"root_rbytes":2430976,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":940,"ceph_cap_op_grant":7615,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3472,"handle_client_caps_dirty":3000,"handle_client_cap_release":6,"process_request_cap_release":9603},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1039,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":2162,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":12031,"evex":0,"evtrm":0,"ev":12034,"evexg":0,"evexd":0,"segadd":15,"segex":0,"segtrm":0,"seg":17,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":28711929,"rdpos":4195244,"jlat":{"avgcount":21,"sum":3.459018456,"avgtime":0.164715164},"replayed":3},"mds_mem":{"ino":1642,"ino+":3214,"ino-":1572,"dir":13,"dir+":13,"dir-":0,"dn":4840,"dn+":6412,"dn-":1572,"cap":1040,"cap+":3202,"cap-":2162,"rss":98052,"heap":248164},"mds_server":{"dispatch_client_request":6419,"dispatch_server_request":0,"handle_client_request":6413,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.114640424,"avgtime":0.000973325},"req_getattr_latency":{"avgcount":6,"sum":0.197880257,"avgtime":0.032980042},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002834805,"avgtime":0.002834805},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.120240153,"avgtime":0.040080051},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.330014075,"avgtime":0.330014075},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":5.838908166,"avgtime":0.001824658},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5853,"average_load":5853,"avg_session_uptime":774,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":10708958,"buffer_anon_items":3161,"buffer_meta_bytes":176,"buffer_meta_items":2,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":11436119,"mds_co_items":137238,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":2295,"op_send_bytes":24775409,"op_resend":0,"op_reply":2295,"op_latency":{"avgcount":2295,"sum":4.477895256,"avgtime":0.001951152},"op_inflight":0,"oplen_avg":{"avgcount":2295,"sum":2362},"op":2295,"op_r":54,"op_w":2241,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":26,"osdop_write":39,"osdop_writefull":9,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":2178,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":27,"omap_rd":48,"omap_del":8},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":2162,"pq_executed":2162,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":12230,"get_sum":3438368,"get_or_fail_fail":0,"get_or_fail_success":12230,"take":0,"take_sum":0,"put":12230,"put_sum":3438368,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":2295,"take_sum":24994888,"put":2295,"put_sum":24994888,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":2295,"take_sum":2295,"put":2295,"put_sum":2295,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":2162,"get_sum":218362,"get_or_fail_fail":0,"get_or_fail_success":2162,"take":0,"take_sum":0,"put":18,"put_sum":218362,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55b1d3c8aa00":{"val":356371,"max":3758096384,"get_started":0,"get":12031,"get_sum":24516244,"get_or_fail_fail":0,"get_or_fail_success":12031,"take":0,"take_sum":0,"put":21,"put_sum":24159873,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:49.524 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.04e-07, "sum": 2.09e-07 }, "msgr_recv_bytes": 258653, "msgr_recv_encrypted_bytes": 1568, "msgr_recv_messages": 982, "msgr_running_fast_dispatch_time": 0.027133826, "msgr_running_recv_time": 0.02373523, "msgr_running_send_time": 0.075158901, "msgr_running_total_time": 0.13408357, "msgr_send_bytes": 4535796, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 982, "msgr_send_messages_queue_lat": { "avgcount": 981, "avgtime": 7.1153e-05, "sum": 0.069801511 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 30793, "avgtime": 1.74e-07, "sum": 0.005372236 }, "msgr_recv_bytes": 3884488, "msgr_recv_encrypted_bytes": 108048, "msgr_recv_messages": 10547, "msgr_running_fast_dispatch_time": 0.026112498, "msgr_running_recv_time": 0.514858106, "msgr_running_send_time": 0.609113613, "msgr_running_total_time": 1.072152909, "msgr_send_bytes": 27833441, "msgr_send_encrypted_bytes": 7920, "msgr_send_messages": 24999, "msgr_send_messages_queue_lat": { "avgcount": 24999, "avgtime": 1.6591e-05, "sum": 0.414783138 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 398815, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 706, "msgr_running_fast_dispatch_time": 0.01927865, "msgr_running_recv_time": 0.017832125, "msgr_running_send_time": 0.040596534, "msgr_running_total_time": 0.082709426, "msgr_send_bytes": 2400660, "msgr_send_encrypted_bytes": 95808, "msgr_send_messages": 713, "msgr_send_messages_queue_lat": { "avgcount": 712, "avgtime": 6.8353e-05, "sum": 0.048667337 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 96, "avgtime": 0.080205752, "sum": 7.699752282 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 3425, "avgtime": 0.000488146, "sum": 1.671902658 }, "queue_len": 0 }, "mds": { "caps": 1040, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7615, "ceph_cap_op_revoke": 940, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 6, "handle_client_caps": 3472, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4840, "inodes_bottom": 4840, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4840, "inodes_top": 0, "inodes_with_caps": 1040, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 9603, "q": 0, "reply": 6411, "reply_latency": { "avgcount": 6411, "avgtime": 0.001498131, "sum": 9.60451788 }, "request": 6413, "root_rbytes": 2430976, "root_rfiles": 306, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6419, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6418, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1039, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 2162, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 12034, "evadd": 12031, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 21, "avgtime": 0.164715164, "sum": 3.459018456 }, "rdpos": 4195244, "replayed": 3, "seg": 17, "segadd": 15, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 28711929 }, "mds_mem": { "cap": 1040, "cap+": 3202, "cap-": 2162, "dir": 13, "dir+": 13, "dir-": 0, "dn": 4840, "dn+": 6412, "dn-": 1572, "heap": 248164, "ino": 1642, "ino+": 3214, "ino-": 1572, "rss": 98052 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6419, "dispatch_server_request": 0, "handle_client_request": 6413, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000973325, "sum": 3.114640424 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.032980042, "sum": 0.197880257 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002834805, "sum": 0.002834805 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.040080051, "sum": 0.120240153 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.330014075, "sum": 0.330014075 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.001824658, "sum": 5.838908166 } }, "mds_sessions": { "average_load": 5853, "avg_session_uptime": 774, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5853 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 10708958, "buffer_anon_items": 3161, "buffer_meta_bytes": 176, "buffer_meta_items": 2, "mds_co_bytes": 11436119, "mds_co_items": 137238, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 8, "omap_rd": 48, "omap_wr": 27, "op": 2295, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 2295, "avgtime": 0.001951152, "sum": 4.477895256 }, "op_pg": 0, "op_r": 54, "op_reply": 2295, "op_resend": 0, "op_rmw": 0, "op_send": 2295, "op_send_bytes": 24775409, "op_w": 2241, "oplen_avg": { "avgcount": 2295, "sum": 2362 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 2178, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 26, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 39, "osdop_writefull": 9, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 2162, "pq_executed_ops": 2162, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 12230, "get_or_fail_fail": 0, "get_or_fail_success": 12230, "get_started": 0, "get_sum": 3438368, "max": 104857600, "put": 12230, "put_sum": 3438368, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 2295, "put_sum": 24994888, "take": 2295, "take_sum": 24994888, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 2295, "put_sum": 2295, "take": 2295, "take_sum": 2295, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 2162, "get_or_fail_fail": 0, "get_or_fail_success": 2162, "get_started": 0, "get_sum": 218362, "max": 3758096384, "put": 18, "put_sum": 218362, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55b1d3c8aa00": { "get": 12031, "get_or_fail_fail": 0, "get_or_fail_success": 12031, "get_started": 0, "get_sum": 24516244, "max": 3758096384, "put": 21, "put_sum": 24159873, "take": 0, "take_sum": 0, "val": 356371, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:49.525 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/1039, 2162/3201 2024-03-06T07:26:49.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.604+0000 7f8c83237640 1 -- 172.21.15.83:0/1580706593 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c7c0cf5d0 con 0x7f8c7c0cfba0 2024-03-06T07:26:49.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.604+0000 7f8c797fa640 1 -- 172.21.15.83:0/1580706593 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f8c7007af80 con 0x7f8c7c0cfba0 2024-03-06T07:26:49.606 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:49.606 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":53,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6023,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/601236077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":601236077},{"type":"v1","addr":"172.21.15.183:6833","nonce":601236077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6029,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1435055961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1435055961},{"type":"v1","addr":"172.21.15.183:6835","nonce":1435055961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":7572,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/2726045966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2726045966},{"type":"v1","addr":"172.21.15.83:6836","nonce":2726045966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.064335+0000","modified":"2024-03-06T07:26:32.814889+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7575":{"gid":7575,"name":"a","rank":0,"incarnation":49,"state":"up:active","state_seq":5,"addr":"172.21.15.83:6837/3254575964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":3254575964},{"type":"v1","addr":"172.21.15.83:6837","nonce":3254575964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7575,"qdb_cluster":[7575]},"id":7}]} 2024-03-06T07:26:49.606 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 53 2024-03-06T07:26:49.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.606+0000 7f8c83237640 1 -- 172.21.15.83:0/1580706593 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8c6006b4d0 msgr2=0x7f8c6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.606+0000 7f8c83237640 1 --2- 172.21.15.83:0/1580706593 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8c6006b4d0 0x7f8c6006d990 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c0a1e40 tx=0x7f8c68007e80 comp rx=0 tx=0).stop 2024-03-06T07:26:49.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.606+0000 7f8c83237640 1 -- 172.21.15.83:0/1580706593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8c7c0cfba0 msgr2=0x7f8c7c15a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.606+0000 7f8c83237640 1 --2- 172.21.15.83:0/1580706593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8c7c0cfba0 0x7f8c7c15a2a0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f8c70000c00 tx=0x7f8c700046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.606+0000 7f8c83237640 1 -- 172.21.15.83:0/1580706593 shutdown_connections 2024-03-06T07:26:49.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.606+0000 7f8c83237640 1 --2- 172.21.15.83:0/1580706593 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8c6006b4d0 0x7f8c6006d990 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.606+0000 7f8c83237640 1 --2- 172.21.15.83:0/1580706593 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8c7c15e150 0x7f8c7c0cd200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.606+0000 7f8c83237640 1 --2- 172.21.15.83:0/1580706593 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c7c0d1060 0x7f8c7c0c8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.606+0000 7f8c83237640 1 --2- 172.21.15.83:0/1580706593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8c7c0cfba0 0x7f8c7c15a2a0 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.606+0000 7f8c83237640 1 -- 172.21.15.83:0/1580706593 >> 172.21.15.83:0/1580706593 conn(0x7f8c7c09d230 msgr2=0x7f8c7c0b9080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:49.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.606+0000 7f8c83237640 1 -- 172.21.15.83:0/1580706593 shutdown_connections 2024-03-06T07:26:49.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:49.606+0000 7f8c83237640 1 -- 172.21.15.83:0/1580706593 wait complete. 2024-03-06T07:26:49.619 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:26:49.942 INFO:teuthology.orchestra.run.smithi083.stdout:{"message":"","return_code":0} 2024-03-06T07:26:49.943 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:26:50.526 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:50.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.627+0000 7f79e1fd7640 1 Processor -- start 2024-03-06T07:26:50.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.627+0000 7f79e1fd7640 1 -- start start 2024-03-06T07:26:50.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.628+0000 7f79e1fd7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f79dc0ce980 0x7f79dc0cede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.628+0000 7f79e1fd7640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f79dc162940 0x7f79dc164d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.628+0000 7f79e1fd7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79dc0b5300 con 0x7f79dc0ce5a0 2024-03-06T07:26:50.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.628+0000 7f79e1fd7640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79dc0ccbd0 con 0x7f79dc162940 2024-03-06T07:26:50.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.628+0000 7f79e1fd7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79dc0b3b00 con 0x7f79dc0ce980 2024-03-06T07:26:50.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.628+0000 7f79db7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f79dc0ce5a0 0x7f79dc0b4bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:33416/0 (socket says 172.21.15.83:33416) 2024-03-06T07:26:50.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.628+0000 7f79db7fe640 1 -- 172.21.15.83:0/16165978 learned_addr learned my addr 172.21.15.83:0/16165978 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:50.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.628+0000 7f79dbfff640 1 --2- 172.21.15.83:0/16165978 >> v2:172.21.15.183:3300/0 conn(0x7f79dc162940 0x7f79dc164d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.628+0000 7f79daffd640 1 --2- 172.21.15.83:0/16165978 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f79dc0ce980 0x7f79dc0cede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.628+0000 7f79da7fc640 1 -- 172.21.15.83:0/16165978 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3666525527 0 0) 0x7f79dc0b5300 con 0x7f79dc0ce5a0 2024-03-06T07:26:50.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.628+0000 7f79da7fc640 1 -- 172.21.15.83:0/16165978 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79c4003520 con 0x7f79dc0ce5a0 2024-03-06T07:26:50.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.629+0000 7f79da7fc640 1 -- 172.21.15.83:0/16165978 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1580847574 0 0) 0x7f79c4003520 con 0x7f79dc0ce5a0 2024-03-06T07:26:50.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.629+0000 7f79da7fc640 1 -- 172.21.15.83:0/16165978 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f79dc0ce980 msgr2=0x7f79dc0cede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:50.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.629+0000 7f79da7fc640 1 --2- 172.21.15.83:0/16165978 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f79dc0ce980 0x7f79dc0cede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.629+0000 7f79da7fc640 1 -- 172.21.15.83:0/16165978 >> v2:172.21.15.183:3300/0 conn(0x7f79dc162940 msgr2=0x7f79dc164d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:50.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.629+0000 7f79da7fc640 1 --2- 172.21.15.83:0/16165978 >> v2:172.21.15.183:3300/0 conn(0x7f79dc162940 0x7f79dc164d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.629+0000 7f79da7fc640 1 -- 172.21.15.83:0/16165978 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79dc0b3de0 con 0x7f79dc0ce5a0 2024-03-06T07:26:50.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.629+0000 7f79da7fc640 1 -- 172.21.15.83:0/16165978 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f79cc001670 con 0x7f79dc0ce5a0 2024-03-06T07:26:50.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.629+0000 7f79da7fc640 1 -- 172.21.15.83:0/16165978 >> v1:172.21.15.83:6789/0 conn(0x7f79dc0ce5a0 legacy=0x7f79dc0b4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.629+0000 7f79da7fc640 1 --2- 172.21.15.83:0/16165978 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f79dc0ce980 0x7f79c40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.629+0000 7f79da7fc640 1 --2- 172.21.15.83:0/16165978 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f79c4003f30 0x7f79c4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.629+0000 7f79da7fc640 1 --2- 172.21.15.83:0/16165978 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79c4004d40 0x7f79c40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.629+0000 7f79da7fc640 1 -- 172.21.15.83:0/16165978 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79dc0ccbd0 con 0x7f79c4004d40 2024-03-06T07:26:50.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.629+0000 7f79da7fc640 1 -- 172.21.15.83:0/16165978 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79dc0b3b00 con 0x7f79c4003f30 2024-03-06T07:26:50.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.629+0000 7f79da7fc640 1 -- 172.21.15.83:0/16165978 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79c400b9c0 con 0x7f79dc0ce980 2024-03-06T07:26:50.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.629+0000 7f79dbfff640 1 --2- 172.21.15.83:0/16165978 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79c4004d40 0x7f79c40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.629+0000 7f79db7fe640 1 --2- 172.21.15.83:0/16165978 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f79dc0ce980 0x7f79c40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.629+0000 7f79daffd640 1 --2- 172.21.15.83:0/16165978 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f79c4003f30 0x7f79c4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.629+0000 7f79db7fe640 1 -- 172.21.15.83:0/16165978 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f79c4003f30 msgr2=0x7f79c4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:50.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.629+0000 7f79db7fe640 1 --2- 172.21.15.83:0/16165978 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f79c4003f30 0x7f79c4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.629+0000 7f79db7fe640 1 -- 172.21.15.83:0/16165978 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79c4004d40 msgr2=0x7f79c40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:50.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.629+0000 7f79db7fe640 1 --2- 172.21.15.83:0/16165978 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79c4004d40 0x7f79c40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.629+0000 7f79db7fe640 1 -- 172.21.15.83:0/16165978 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79c4003520 con 0x7f79dc0ce980 2024-03-06T07:26:50.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.629+0000 7f79dbfff640 1 --2- 172.21.15.83:0/16165978 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79c4004d40 0x7f79c40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:50.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.630+0000 7f79db7fe640 1 --2- 172.21.15.83:0/16165978 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f79dc0ce980 0x7f79c40039b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f79cc00c290 tx=0x7f79c4005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4679788b07cd4811 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:50.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.630+0000 7f79da7fc640 1 -- 172.21.15.83:0/16165978 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79cc007d50 con 0x7f79dc0ce980 2024-03-06T07:26:50.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.630+0000 7f79da7fc640 1 -- 172.21.15.83:0/16165978 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79cc030070 con 0x7f79dc0ce980 2024-03-06T07:26:50.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.630+0000 7f79da7fc640 1 -- 172.21.15.83:0/16165978 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79cc02cde0 con 0x7f79dc0ce980 2024-03-06T07:26:50.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.631+0000 7f79e1fd7640 1 -- 172.21.15.83:0/16165978 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f79dc0ce980 msgr2=0x7f79c40039b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.631+0000 7f79e1fd7640 1 --2- 172.21.15.83:0/16165978 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f79dc0ce980 0x7f79c40039b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f79cc00c290 tx=0x7f79c4005830 comp rx=0 tx=0).stop 2024-03-06T07:26:50.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.631+0000 7f79dbfff640 1 -- 172.21.15.83:0/16165978 reap_dead start 2024-03-06T07:26:50.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.631+0000 7f79e1fd7640 1 -- 172.21.15.83:0/16165978 shutdown_connections 2024-03-06T07:26:50.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.631+0000 7f79e1fd7640 1 -- 172.21.15.83:0/16165978 >> 172.21.15.83:0/16165978 conn(0x7f79dc06b160 msgr2=0x7f79dc06b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:50.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.631+0000 7f79e1fd7640 1 -- 172.21.15.83:0/16165978 shutdown_connections 2024-03-06T07:26:50.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.631+0000 7f79e1fd7640 1 -- 172.21.15.83:0/16165978 wait complete. 2024-03-06T07:26:50.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.632+0000 7f79e1fd7640 1 Processor -- start 2024-03-06T07:26:50.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.632+0000 7f79e1fd7640 1 -- start start 2024-03-06T07:26:50.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.633+0000 7f79e1fd7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f79dc0ce980 0x7f79dc1f7740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.633+0000 7f79e1fd7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79dc15a800 0x7f79dc06dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.633+0000 7f79e1fd7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f79dc13b440 0x7f79dc13b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.633+0000 7f79e1fd7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79dc0cc9b0 con 0x7f79dc15a800 2024-03-06T07:26:50.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.633+0000 7f79e1fd7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79dc0cc830 con 0x7f79dc0ce980 2024-03-06T07:26:50.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.633+0000 7f79e1fd7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79dc0cceb0 con 0x7f79dc13b440 2024-03-06T07:26:50.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.633+0000 7f79daffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79dc15a800 0x7f79dc06dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.633+0000 7f79daffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79dc15a800 0x7f79dc06dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53346/0 (socket says 172.21.15.83:53346) 2024-03-06T07:26:50.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.633+0000 7f79daffd640 1 -- 172.21.15.83:0/1946390429 learned_addr learned my addr 172.21.15.83:0/1946390429 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:50.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.633+0000 7f79dbfff640 1 --2- 172.21.15.83:0/1946390429 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f79dc13b440 0x7f79dc13b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.633+0000 7f79db7fe640 1 --2- 172.21.15.83:0/1946390429 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f79dc0ce980 0x7f79dc1f7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.633+0000 7f79daffd640 1 -- 172.21.15.83:0/1946390429 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f79dc13b440 msgr2=0x7f79dc13b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.633+0000 7f79daffd640 1 --2- 172.21.15.83:0/1946390429 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f79dc13b440 0x7f79dc13b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.633+0000 7f79daffd640 1 -- 172.21.15.83:0/1946390429 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f79dc0ce980 msgr2=0x7f79dc1f7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.633+0000 7f79daffd640 1 --2- 172.21.15.83:0/1946390429 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f79dc0ce980 0x7f79dc1f7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.633+0000 7f79daffd640 1 -- 172.21.15.83:0/1946390429 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79dc13be00 con 0x7f79dc15a800 2024-03-06T07:26:50.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.633+0000 7f79daffd640 1 --2- 172.21.15.83:0/1946390429 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79dc15a800 0x7f79dc06dc50 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f79d0011990 tx=0x7f79d0011e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:50.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.633+0000 7f79c3fff640 1 -- 172.21.15.83:0/1946390429 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d000fa10 con 0x7f79dc15a800 2024-03-06T07:26:50.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.634+0000 7f79c3fff640 1 -- 172.21.15.83:0/1946390429 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79d000fbb0 con 0x7f79dc15a800 2024-03-06T07:26:50.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.634+0000 7f79c3fff640 1 -- 172.21.15.83:0/1946390429 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d0014440 con 0x7f79dc15a800 2024-03-06T07:26:50.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.634+0000 7f79e1fd7640 1 -- 172.21.15.83:0/1946390429 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79dc0c91d0 con 0x7f79dc15a800 2024-03-06T07:26:50.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.634+0000 7f79e1fd7640 1 -- 172.21.15.83:0/1946390429 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79dc1f8270 con 0x7f79dc15a800 2024-03-06T07:26:50.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.635+0000 7f79e1fd7640 1 -- 172.21.15.83:0/1946390429 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79dc069ee0 con 0x7f79dc15a800 2024-03-06T07:26:50.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.635+0000 7f79c3fff640 1 -- 172.21.15.83:0/1946390429 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f79d0012040 con 0x7f79dc15a800 2024-03-06T07:26:50.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.635+0000 7f79c3fff640 1 --2- 172.21.15.83:0/1946390429 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f79c806b500 0x7f79c806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.635+0000 7f79c3fff640 1 -- 172.21.15.83:0/1946390429 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f79d008a650 con 0x7f79dc15a800 2024-03-06T07:26:50.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.635+0000 7f79db7fe640 1 --2- 172.21.15.83:0/1946390429 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f79c806b500 0x7f79c806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.636+0000 7f79db7fe640 1 --2- 172.21.15.83:0/1946390429 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f79c806b500 0x7f79c806d9c0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f79cc034000 tx=0x7f79cc007000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:50.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.640+0000 7f79c3fff640 1 -- 172.21.15.83:0/1946390429 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f79d0019030 con 0x7f79dc15a800 2024-03-06T07:26:50.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.873+0000 7f79e1fd7640 1 -- 172.21.15.83:0/1946390429 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79dc072720 con 0x7f79dc15a800 2024-03-06T07:26:50.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.873+0000 7f79c3fff640 1 -- 172.21.15.83:0/1946390429 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f79d0058e20 con 0x7f79dc15a800 2024-03-06T07:26:50.875 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:50.875 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":53,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6023,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/601236077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":601236077},{"type":"v1","addr":"172.21.15.183:6833","nonce":601236077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6029,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1435055961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1435055961},{"type":"v1","addr":"172.21.15.183:6835","nonce":1435055961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":7572,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/2726045966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2726045966},{"type":"v1","addr":"172.21.15.83:6836","nonce":2726045966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.064335+0000","modified":"2024-03-06T07:26:32.814889+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7575":{"gid":7575,"name":"a","rank":0,"incarnation":49,"state":"up:active","state_seq":5,"addr":"172.21.15.83:6837/3254575964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":3254575964},{"type":"v1","addr":"172.21.15.83:6837","nonce":3254575964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7575,"qdb_cluster":[7575]},"id":7}]} 2024-03-06T07:26:50.876 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 53 2024-03-06T07:26:50.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.875+0000 7f79e1fd7640 1 -- 172.21.15.83:0/1946390429 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f79c806b500 msgr2=0x7f79c806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.875+0000 7f79e1fd7640 1 --2- 172.21.15.83:0/1946390429 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f79c806b500 0x7f79c806d9c0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f79cc034000 tx=0x7f79cc007000 comp rx=0 tx=0).stop 2024-03-06T07:26:50.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.875+0000 7f79e1fd7640 1 -- 172.21.15.83:0/1946390429 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79dc15a800 msgr2=0x7f79dc06dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.875+0000 7f79e1fd7640 1 --2- 172.21.15.83:0/1946390429 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79dc15a800 0x7f79dc06dc50 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f79d0011990 tx=0x7f79d0011e50 comp rx=0 tx=0).stop 2024-03-06T07:26:50.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.875+0000 7f79e1fd7640 1 -- 172.21.15.83:0/1946390429 shutdown_connections 2024-03-06T07:26:50.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.875+0000 7f79e1fd7640 1 --2- 172.21.15.83:0/1946390429 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f79c806b500 0x7f79c806d9c0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.875+0000 7f79e1fd7640 1 --2- 172.21.15.83:0/1946390429 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f79dc13b440 0x7f79dc13b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.875+0000 7f79e1fd7640 1 --2- 172.21.15.83:0/1946390429 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79dc15a800 0x7f79dc06dc50 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.875+0000 7f79e1fd7640 1 --2- 172.21.15.83:0/1946390429 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f79dc0ce980 0x7f79dc1f7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.875+0000 7f79e1fd7640 1 -- 172.21.15.83:0/1946390429 >> 172.21.15.83:0/1946390429 conn(0x7f79dc06b160 msgr2=0x7f79dc15cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:50.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.876+0000 7f79e1fd7640 1 -- 172.21.15.83:0/1946390429 shutdown_connections 2024-03-06T07:26:50.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:50.876+0000 7f79e1fd7640 1 -- 172.21.15.83:0/1946390429 wait complete. 2024-03-06T07:26:50.888 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:26:51.040 INFO:teuthology.orchestra.run.smithi083.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1595,"msgr_send_messages":1598,"msgr_recv_bytes":451052,"msgr_send_bytes":5192177,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.167038564,"msgr_running_send_time":0.089576112,"msgr_running_recv_time":0.028722803,"msgr_running_fast_dispatch_time":0.037621322,"msgr_send_messages_queue_lat":{"avgcount":1596,"sum":0.418763176,"avgtime":0.000262382},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000209,"avgtime":0.000000104},"msgr_recv_encrypted_bytes":1568,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":10977,"msgr_send_messages":25426,"msgr_recv_bytes":4044625,"msgr_send_bytes":28053062,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":1.110677286,"msgr_running_send_time":0.627142636,"msgr_running_recv_time":0.522424641,"msgr_running_fast_dispatch_time":0.038106320,"msgr_send_messages_queue_lat":{"avgcount":25426,"sum":1.114987499,"avgtime":0.000043852},"msgr_handle_ack_lat":{"avgcount":30798,"sum":0.005372939,"avgtime":0.000000174},"msgr_recv_encrypted_bytes":108048,"msgr_send_encrypted_bytes":7920},"AsyncMessenger::Worker-2":{"msgr_recv_messages":1129,"msgr_send_messages":1139,"msgr_recv_bytes":637302,"msgr_send_bytes":3282131,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.112621879,"msgr_running_send_time":0.049668241,"msgr_running_recv_time":0.021982569,"msgr_running_fast_dispatch_time":0.035141347,"msgr_send_messages_queue_lat":{"avgcount":1137,"sum":0.592781380,"avgtime":0.000521355},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":95808},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":119,"sum":7.878914406,"avgtime":0.066209364}},"finisher-PurgeQueue":{"queue_len":1,"complete_latency":{"avgcount":4072,"sum":1.755830341,"avgtime":0.000431196}},"mds":{"request":6413,"reply":6411,"reply_latency":{"avgcount":6411,"sum":9.604517880,"avgtime":0.001498131},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":4356,"inodes_top":0,"inodes_bottom":4356,"inodes_pin_tail":0,"inodes_pinned":4356,"inodes_expired":0,"inodes_with_caps":1040,"caps":1040,"subtrees":2,"traverse":6419,"traverse_hit":6418,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":940,"ceph_cap_op_grant":7615,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3472,"handle_client_caps_dirty":3000,"handle_client_cap_release":8,"process_request_cap_release":9603},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3201,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":13074,"evex":12034,"evtrm":12034,"ev":1043,"evexg":0,"evexd":12033,"segadd":17,"segex":17,"segtrm":17,"seg":2,"segmjr":1,"segexg":0,"segexd":16,"expos":28711929,"wrpos":29340566,"rdpos":4195244,"jlat":{"avgcount":24,"sum":3.524300847,"avgtime":0.146845868},"replayed":3},"mds_mem":{"ino":1052,"ino+":3214,"ino-":2162,"dir":13,"dir+":13,"dir-":0,"dn":1050,"dn+":6412,"dn-":5362,"cap":1,"cap+":3202,"cap-":3201,"rss":105604,"heap":248164},"mds_server":{"dispatch_client_request":6419,"dispatch_server_request":0,"handle_client_request":6413,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.114640424,"avgtime":0.000973325},"req_getattr_latency":{"avgcount":6,"sum":0.197880257,"avgtime":0.032980042},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002834805,"avgtime":0.002834805},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.120240153,"avgtime":0.040080051},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.330014075,"avgtime":0.330014075},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":5.838908166,"avgtime":0.001824658},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5853,"average_load":5853,"avg_session_uptime":774,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":10708958,"buffer_anon_items":3161,"buffer_meta_bytes":176,"buffer_meta_items":2,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":11436119,"mds_co_items":137238,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":6,"op_laggy":0,"op_send":3763,"op_send_bytes":25974573,"op_resend":0,"op_reply":3757,"op_latency":{"avgcount":3757,"sum":66.486444752,"avgtime":0.017696684},"op_inflight":4,"oplen_avg":{"avgcount":3763,"sum":5953},"op":3763,"op_r":59,"op_w":3704,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":1051,"osdop_read":31,"osdop_write":47,"osdop_writefull":15,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":2560,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":2102,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":50,"omap_rd":48,"omap_del":12},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":6,"pq_executing_ops_high_water":6,"pq_executing":6,"pq_executing_high_water":6,"pq_executed_ops":2533,"pq_executed":2533,"pq_item_in_journal":670},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":13697,"get_sum":3897615,"get_or_fail_fail":0,"get_or_fail_success":13697,"take":0,"take_sum":0,"put":13697,"put_sum":3897615,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3763,"take_sum":26298991,"put":3758,"put_sum":26298991,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":5,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3764,"take_sum":3764,"put":3758,"put_sum":3758,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3201,"get_sum":323301,"get_or_fail_fail":0,"get_or_fail_success":3201,"take":0,"take_sum":0,"put":23,"put_sum":323301,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55b1d3c8aa00":{"val":624956,"max":3758096384,"get_started":0,"get":13074,"get_sum":25144881,"get_or_fail_fail":0,"get_or_fail_success":13074,"take":0,"take_sum":0,"put":24,"put_sum":24519925,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:51.041 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.04e-07, "sum": 2.09e-07 }, "msgr_recv_bytes": 451052, "msgr_recv_encrypted_bytes": 1568, "msgr_recv_messages": 1595, "msgr_running_fast_dispatch_time": 0.037621322, "msgr_running_recv_time": 0.028722803, "msgr_running_send_time": 0.089576112, "msgr_running_total_time": 0.167038564, "msgr_send_bytes": 5192177, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 1598, "msgr_send_messages_queue_lat": { "avgcount": 1596, "avgtime": 0.000262382, "sum": 0.418763176 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 30798, "avgtime": 1.74e-07, "sum": 0.005372939 }, "msgr_recv_bytes": 4044625, "msgr_recv_encrypted_bytes": 108048, "msgr_recv_messages": 10977, "msgr_running_fast_dispatch_time": 0.03810632, "msgr_running_recv_time": 0.522424641, "msgr_running_send_time": 0.627142636, "msgr_running_total_time": 1.110677286, "msgr_send_bytes": 28053062, "msgr_send_encrypted_bytes": 7920, "msgr_send_messages": 25426, "msgr_send_messages_queue_lat": { "avgcount": 25426, "avgtime": 4.3852e-05, "sum": 1.114987499 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 637302, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 1129, "msgr_running_fast_dispatch_time": 0.035141347, "msgr_running_recv_time": 0.021982569, "msgr_running_send_time": 0.049668241, "msgr_running_total_time": 0.112621879, "msgr_send_bytes": 3282131, "msgr_send_encrypted_bytes": 95808, "msgr_send_messages": 1139, "msgr_send_messages_queue_lat": { "avgcount": 1137, "avgtime": 0.000521355, "sum": 0.59278138 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 119, "avgtime": 0.066209364, "sum": 7.878914406 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4072, "avgtime": 0.000431196, "sum": 1.755830341 }, "queue_len": 1 }, "mds": { "caps": 1040, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7615, "ceph_cap_op_revoke": 940, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 8, "handle_client_caps": 3472, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4356, "inodes_bottom": 4356, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4356, "inodes_top": 0, "inodes_with_caps": 1040, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 9603, "q": 0, "reply": 6411, "reply_latency": { "avgcount": 6411, "avgtime": 0.001498131, "sum": 9.60451788 }, "request": 6413, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6419, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6418, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3201, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 1043, "evadd": 13074, "evex": 12034, "evexd": 12033, "evexg": 0, "evlrg": 0, "evtrm": 12034, "expos": 28711929, "jlat": { "avgcount": 24, "avgtime": 0.146845868, "sum": 3.524300847 }, "rdpos": 4195244, "replayed": 3, "seg": 2, "segadd": 17, "segex": 17, "segexd": 16, "segexg": 0, "segmjr": 1, "segtrm": 17, "wrpos": 29340566 }, "mds_mem": { "cap": 1, "cap+": 3202, "cap-": 3201, "dir": 13, "dir+": 13, "dir-": 0, "dn": 1050, "dn+": 6412, "dn-": 5362, "heap": 248164, "ino": 1052, "ino+": 3214, "ino-": 2162, "rss": 105604 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6419, "dispatch_server_request": 0, "handle_client_request": 6413, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000973325, "sum": 3.114640424 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.032980042, "sum": 0.197880257 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002834805, "sum": 0.002834805 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.040080051, "sum": 0.120240153 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.330014075, "sum": 0.330014075 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.001824658, "sum": 5.838908166 } }, "mds_sessions": { "average_load": 5853, "avg_session_uptime": 774, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5853 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 10708958, "buffer_anon_items": 3161, "buffer_meta_bytes": 176, "buffer_meta_items": 2, "mds_co_bytes": 11436119, "mds_co_items": 137238, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 12, "omap_rd": 48, "omap_wr": 50, "op": 3763, "op_Infinitylight": 4, "op_active": 6, "op_laggy": 0, "op_latency": { "avgcount": 3757, "avgtime": 0.017696684, "sum": 66.486444752 }, "op_pg": 0, "op_r": 59, "op_reply": 3757, "op_resend": 0, "op_rmw": 0, "op_send": 3763, "op_send_bytes": 25974573, "op_w": 3704, "oplen_avg": { "avgcount": 3763, "sum": 5953 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 1051, "osdop_delete": 2560, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 31, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 2102, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 47, "osdop_writefull": 15, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 2533, "pq_executed_ops": 2533, "pq_executing": 6, "pq_executing_high_water": 6, "pq_executing_ops": 6, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 670 }, "throttle-msgr_dispatch_throttler-mds": { "get": 13697, "get_or_fail_fail": 0, "get_or_fail_success": 13697, "get_started": 0, "get_sum": 3897615, "max": 104857600, "put": 13697, "put_sum": 3897615, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 3758, "put_sum": 26298991, "take": 3763, "take_sum": 26298991, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3758, "put_sum": 3758, "take": 3764, "take_sum": 3764, "val": 5, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3201, "get_or_fail_fail": 0, "get_or_fail_success": 3201, "get_started": 0, "get_sum": 323301, "max": 3758096384, "put": 23, "put_sum": 323301, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55b1d3c8aa00": { "get": 13074, "get_or_fail_fail": 0, "get_or_fail_success": 13074, "get_started": 0, "get_sum": 25144881, "max": 3758096384, "put": 24, "put_sum": 24519925, "take": 0, "take_sum": 0, "val": 624956, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:51.042 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 6/0, 2533/3201 2024-03-06T07:26:52.042 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:52.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.140+0000 7faf58193640 1 Processor -- start 2024-03-06T07:26:52.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.141+0000 7faf58193640 1 -- start start 2024-03-06T07:26:52.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.141+0000 7faf58193640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf5015e150 0x7faf500cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.141+0000 7faf58193640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7faf500d1060 0x7faf500b9060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.141+0000 7faf58193640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf500d1ea0 con 0x7faf500cfba0 2024-03-06T07:26:52.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.141+0000 7faf58193640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf500d2ab0 con 0x7faf500d1060 2024-03-06T07:26:52.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.141+0000 7faf58193640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf500d2180 con 0x7faf5015e150 2024-03-06T07:26:52.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.142+0000 7faf56730640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7faf500d1060 0x7faf500b9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.142+0000 7faf5572e640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7faf500cfba0 0x7faf500d0950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:33428/0 (socket says 172.21.15.83:33428) 2024-03-06T07:26:52.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.142+0000 7faf5572e640 1 -- 172.21.15.83:0/4206895367 learned_addr learned my addr 172.21.15.83:0/4206895367 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:52.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.142+0000 7faf55f2f640 1 --2- 172.21.15.83:0/4206895367 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf5015e150 0x7faf500cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.142+0000 7faf56730640 1 -- 172.21.15.83:0/4206895367 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf5015e150 msgr2=0x7faf500cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:52.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.142+0000 7faf56730640 1 --2- 172.21.15.83:0/4206895367 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf5015e150 0x7faf500cf660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.142+0000 7faf56730640 1 -- 172.21.15.83:0/4206895367 >> v1:172.21.15.83:6789/0 conn(0x7faf500cfba0 legacy=0x7faf500d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.142+0000 7faf54f2d640 1 -- 172.21.15.83:0/4206895367 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3759709255 0 0) 0x7faf500d1ea0 con 0x7faf500cfba0 2024-03-06T07:26:52.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.142+0000 7faf56730640 1 -- 172.21.15.83:0/4206895367 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf500d2460 con 0x7faf500d1060 2024-03-06T07:26:52.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.143+0000 7faf56730640 1 --2- 172.21.15.83:0/4206895367 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf500d1060 0x7faf500b9060 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7faf4c0097a0 tx=0x7faf4c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d5d657d27efa316 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:52.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.143+0000 7faf54f2d640 1 -- 172.21.15.83:0/4206895367 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf4c0315a0 con 0x7faf500d1060 2024-03-06T07:26:52.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.143+0000 7faf54f2d640 1 -- 172.21.15.83:0/4206895367 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf4c037070 con 0x7faf500d1060 2024-03-06T07:26:52.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.143+0000 7faf54f2d640 1 -- 172.21.15.83:0/4206895367 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf4c031960 con 0x7faf500d1060 2024-03-06T07:26:52.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.143+0000 7faf58193640 1 -- 172.21.15.83:0/4206895367 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf500d1060 msgr2=0x7faf500b9060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.143+0000 7faf58193640 1 --2- 172.21.15.83:0/4206895367 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf500d1060 0x7faf500b9060 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7faf4c0097a0 tx=0x7faf4c02b840 comp rx=0 tx=0).stop 2024-03-06T07:26:52.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.144+0000 7faf58193640 1 -- 172.21.15.83:0/4206895367 shutdown_connections 2024-03-06T07:26:52.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.144+0000 7faf58193640 1 --2- 172.21.15.83:0/4206895367 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf500d1060 0x7faf500b9060 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.144+0000 7faf58193640 1 --2- 172.21.15.83:0/4206895367 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf5015e150 0x7faf500cf660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.144+0000 7faf58193640 1 -- 172.21.15.83:0/4206895367 >> 172.21.15.83:0/4206895367 conn(0x7faf5009d230 msgr2=0x7faf500d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:52.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.144+0000 7faf58193640 1 -- 172.21.15.83:0/4206895367 shutdown_connections 2024-03-06T07:26:52.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.144+0000 7faf58193640 1 -- 172.21.15.83:0/4206895367 wait complete. 2024-03-06T07:26:52.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.145+0000 7faf58193640 1 Processor -- start 2024-03-06T07:26:52.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.145+0000 7faf58193640 1 -- start start 2024-03-06T07:26:52.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.145+0000 7faf58193640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf500cfba0 0x7faf501f1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.145+0000 7faf58193640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf500d1060 0x7faf500c9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.145+0000 7faf58193640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf5015e150 0x7faf500b8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.145+0000 7faf58193640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf500d1b30 con 0x7faf500cfba0 2024-03-06T07:26:52.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.145+0000 7faf58193640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf500d2f40 con 0x7faf5015e150 2024-03-06T07:26:52.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.145+0000 7faf58193640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf5015e530 con 0x7faf500d1060 2024-03-06T07:26:52.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.145+0000 7faf55f2f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf500cfba0 0x7faf501f1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.146+0000 7faf55f2f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf500cfba0 0x7faf501f1340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53360/0 (socket says 172.21.15.83:53360) 2024-03-06T07:26:52.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.146+0000 7faf55f2f640 1 -- 172.21.15.83:0/285921129 learned_addr learned my addr 172.21.15.83:0/285921129 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:52.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.146+0000 7faf5572e640 1 --2- 172.21.15.83:0/285921129 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf500d1060 0x7faf500c9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.146+0000 7faf56730640 1 --2- 172.21.15.83:0/285921129 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf5015e150 0x7faf500b8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.146+0000 7faf55f2f640 1 -- 172.21.15.83:0/285921129 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf500d1060 msgr2=0x7faf500c9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.146+0000 7faf55f2f640 1 --2- 172.21.15.83:0/285921129 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf500d1060 0x7faf500c9080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.146+0000 7faf55f2f640 1 -- 172.21.15.83:0/285921129 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf5015e150 msgr2=0x7faf500b8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.146+0000 7faf55f2f640 1 --2- 172.21.15.83:0/285921129 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf5015e150 0x7faf500b8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.146+0000 7faf55f2f640 1 -- 172.21.15.83:0/285921129 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf501f1880 con 0x7faf500cfba0 2024-03-06T07:26:52.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.146+0000 7faf56730640 1 --2- 172.21.15.83:0/285921129 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf5015e150 0x7faf500b8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:52.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.146+0000 7faf5572e640 1 --2- 172.21.15.83:0/285921129 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf500d1060 0x7faf500c9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:52.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.146+0000 7faf55f2f640 1 --2- 172.21.15.83:0/285921129 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf500cfba0 0x7faf501f1340 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7faf4800ccf0 tx=0x7faf4800a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:52.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.146+0000 7faf427fc640 1 -- 172.21.15.83:0/285921129 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf48003990 con 0x7faf500cfba0 2024-03-06T07:26:52.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.146+0000 7faf427fc640 1 -- 172.21.15.83:0/285921129 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf48003b30 con 0x7faf500cfba0 2024-03-06T07:26:52.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.146+0000 7faf58193640 1 -- 172.21.15.83:0/285921129 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf500c95c0 con 0x7faf500cfba0 2024-03-06T07:26:52.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.146+0000 7faf427fc640 1 -- 172.21.15.83:0/285921129 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf48004020 con 0x7faf500cfba0 2024-03-06T07:26:52.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.146+0000 7faf58193640 1 -- 172.21.15.83:0/285921129 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf501f4d90 con 0x7faf500cfba0 2024-03-06T07:26:52.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.147+0000 7faf58193640 1 -- 172.21.15.83:0/285921129 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf50069ee0 con 0x7faf500cfba0 2024-03-06T07:26:52.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.148+0000 7faf427fc640 1 -- 172.21.15.83:0/285921129 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faf480041c0 con 0x7faf500cfba0 2024-03-06T07:26:52.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.148+0000 7faf427fc640 1 --2- 172.21.15.83:0/285921129 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7faf3406b4d0 0x7faf3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.148+0000 7faf427fc640 1 -- 172.21.15.83:0/285921129 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5795+0+0 (secure 0 0 0) 0x7faf4808a780 con 0x7faf500cfba0 2024-03-06T07:26:52.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.149+0000 7faf5572e640 1 --2- 172.21.15.83:0/285921129 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7faf3406b4d0 0x7faf3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.149+0000 7faf5572e640 1 --2- 172.21.15.83:0/285921129 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7faf3406b4d0 0x7faf3406d990 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7faf501574e0 tx=0x7faf44007c10 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:52.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.153+0000 7faf427fc640 1 -- 172.21.15.83:0/285921129 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faf48014030 con 0x7faf500cfba0 2024-03-06T07:26:52.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.395+0000 7faf58193640 1 -- 172.21.15.83:0/285921129 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf500d4670 con 0x7faf500cfba0 2024-03-06T07:26:52.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.395+0000 7faf427fc640 1 -- 172.21.15.83:0/285921129 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+4983 (secure 0 0 0) 0x7faf480595a0 con 0x7faf500cfba0 2024-03-06T07:26:52.397 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:52.397 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":53,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6023,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/601236077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":601236077},{"type":"v1","addr":"172.21.15.183:6833","nonce":601236077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6029,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1435055961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1435055961},{"type":"v1","addr":"172.21.15.183:6835","nonce":1435055961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":7572,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/2726045966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2726045966},{"type":"v1","addr":"172.21.15.83:6836","nonce":2726045966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.064335+0000","modified":"2024-03-06T07:26:32.814889+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7575":{"gid":7575,"name":"a","rank":0,"incarnation":49,"state":"up:active","state_seq":5,"addr":"172.21.15.83:6837/3254575964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":3254575964},{"type":"v1","addr":"172.21.15.83:6837","nonce":3254575964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7575,"qdb_cluster":[7575]},"id":7}]} 2024-03-06T07:26:52.397 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 53 2024-03-06T07:26:52.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.397+0000 7faf58193640 1 -- 172.21.15.83:0/285921129 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7faf3406b4d0 msgr2=0x7faf3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.397+0000 7faf58193640 1 --2- 172.21.15.83:0/285921129 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7faf3406b4d0 0x7faf3406d990 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7faf501574e0 tx=0x7faf44007c10 comp rx=0 tx=0).stop 2024-03-06T07:26:52.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.397+0000 7faf58193640 1 -- 172.21.15.83:0/285921129 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf500cfba0 msgr2=0x7faf501f1340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.397+0000 7faf58193640 1 --2- 172.21.15.83:0/285921129 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf500cfba0 0x7faf501f1340 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7faf4800ccf0 tx=0x7faf4800a5c0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.397+0000 7faf58193640 1 -- 172.21.15.83:0/285921129 shutdown_connections 2024-03-06T07:26:52.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.397+0000 7faf58193640 1 --2- 172.21.15.83:0/285921129 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7faf3406b4d0 0x7faf3406d990 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.397+0000 7faf58193640 1 --2- 172.21.15.83:0/285921129 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf5015e150 0x7faf500b8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.397+0000 7faf58193640 1 --2- 172.21.15.83:0/285921129 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf500d1060 0x7faf500c9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.397+0000 7faf58193640 1 --2- 172.21.15.83:0/285921129 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf500cfba0 0x7faf501f1340 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.397+0000 7faf58193640 1 -- 172.21.15.83:0/285921129 >> 172.21.15.83:0/285921129 conn(0x7faf5009d230 msgr2=0x7faf5015fb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:52.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.397+0000 7faf58193640 1 -- 172.21.15.83:0/285921129 shutdown_connections 2024-03-06T07:26:52.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.397+0000 7faf58193640 1 -- 172.21.15.83:0/285921129 wait complete. 2024-03-06T07:26:52.410 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:26:52.527 INFO:teuthology.orchestra.run.smithi083.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1852,"msgr_send_messages":1852,"msgr_recv_bytes":516330,"msgr_send_bytes":5275307,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.186047210,"msgr_running_send_time":0.096803110,"msgr_running_recv_time":0.033128621,"msgr_running_fast_dispatch_time":0.043227075,"msgr_send_messages_queue_lat":{"avgcount":1851,"sum":0.427594116,"avgtime":0.000231007},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000209,"avgtime":0.000000104},"msgr_recv_encrypted_bytes":1568,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":11178,"msgr_send_messages":25626,"msgr_recv_bytes":4095857,"msgr_send_bytes":28118598,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":1.123843278,"msgr_running_send_time":0.631387581,"msgr_running_recv_time":0.525744783,"msgr_running_fast_dispatch_time":0.042653640,"msgr_send_messages_queue_lat":{"avgcount":25626,"sum":1.121386100,"avgtime":0.000043759},"msgr_handle_ack_lat":{"avgcount":30799,"sum":0.005373068,"avgtime":0.000000174},"msgr_recv_encrypted_bytes":108320,"msgr_send_encrypted_bytes":8544},"AsyncMessenger::Worker-2":{"msgr_recv_messages":1343,"msgr_send_messages":1350,"msgr_recv_bytes":691650,"msgr_send_bytes":3351325,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.126991325,"msgr_running_send_time":0.055037145,"msgr_running_recv_time":0.025375514,"msgr_running_fast_dispatch_time":0.039243855,"msgr_send_messages_queue_lat":{"avgcount":1349,"sum":0.599749422,"avgtime":0.000444588},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":95808},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":120,"sum":7.878923993,"avgtime":0.065657699}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5213,"sum":1.790232553,"avgtime":0.000343416}},"mds":{"request":6413,"reply":6411,"reply_latency":{"avgcount":6411,"sum":9.604517880,"avgtime":0.001498131},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":4356,"inodes_top":0,"inodes_bottom":4356,"inodes_pin_tail":0,"inodes_pinned":4356,"inodes_expired":0,"inodes_with_caps":1040,"caps":1040,"subtrees":2,"traverse":6419,"traverse_hit":6418,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":940,"ceph_cap_op_grant":7615,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3472,"handle_client_caps_dirty":3000,"handle_client_cap_release":8,"process_request_cap_release":9603},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3201,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":13074,"evex":12034,"evtrm":12034,"ev":1043,"evexg":0,"evexd":12033,"segadd":17,"segex":17,"segtrm":17,"seg":2,"segmjr":1,"segexg":0,"segexd":16,"expos":28711929,"wrpos":29340566,"rdpos":4195244,"jlat":{"avgcount":24,"sum":3.524300847,"avgtime":0.146845868},"replayed":3},"mds_mem":{"ino":1052,"ino+":3214,"ino-":2162,"dir":12,"dir+":13,"dir-":1,"dn":1050,"dn+":6412,"dn-":5362,"cap":1,"cap+":3202,"cap-":3201,"rss":105604,"heap":248164},"mds_server":{"dispatch_client_request":6419,"dispatch_server_request":0,"handle_client_request":6413,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.114640424,"avgtime":0.000973325},"req_getattr_latency":{"avgcount":6,"sum":0.197880257,"avgtime":0.032980042},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002834805,"avgtime":0.002834805},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.120240153,"avgtime":0.040080051},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.330014075,"avgtime":0.330014075},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":5.838908166,"avgtime":0.001824658},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5853,"average_load":5853,"avg_session_uptime":774,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":10708958,"buffer_anon_items":3161,"buffer_meta_bytes":176,"buffer_meta_items":2,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":11436119,"mds_co_items":137238,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":4427,"op_send_bytes":25974705,"op_resend":0,"op_reply":4427,"op_latency":{"avgcount":4427,"sum":67.441422413,"avgtime":0.015234113},"op_inflight":0,"oplen_avg":{"avgcount":4427,"sum":6617},"op":4427,"op_r":59,"op_w":4368,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":1051,"osdop_read":31,"osdop_write":47,"osdop_writefull":16,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3222,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":2102,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":51,"omap_rd":48,"omap_del":12},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3201,"pq_executed":3201,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":14368,"get_sum":4007777,"get_or_fail_fail":0,"get_or_fail_success":14368,"take":0,"take_sum":0,"put":14368,"put_sum":4007777,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":4427,"take_sum":26299123,"put":4427,"put_sum":26299123,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":4427,"take_sum":4427,"put":4427,"put_sum":4427,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3201,"get_sum":323301,"get_or_fail_fail":0,"get_or_fail_success":3201,"take":0,"take_sum":0,"put":23,"put_sum":323301,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55b1d3c8aa00":{"val":624956,"max":3758096384,"get_started":0,"get":13074,"get_sum":25144881,"get_or_fail_fail":0,"get_or_fail_success":13074,"take":0,"take_sum":0,"put":24,"put_sum":24519925,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:52.528 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.04e-07, "sum": 2.09e-07 }, "msgr_recv_bytes": 516330, "msgr_recv_encrypted_bytes": 1568, "msgr_recv_messages": 1852, "msgr_running_fast_dispatch_time": 0.043227075, "msgr_running_recv_time": 0.033128621, "msgr_running_send_time": 0.09680311, "msgr_running_total_time": 0.18604721, "msgr_send_bytes": 5275307, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 1852, "msgr_send_messages_queue_lat": { "avgcount": 1851, "avgtime": 0.000231007, "sum": 0.427594116 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 30799, "avgtime": 1.74e-07, "sum": 0.005373068 }, "msgr_recv_bytes": 4095857, "msgr_recv_encrypted_bytes": 108320, "msgr_recv_messages": 11178, "msgr_running_fast_dispatch_time": 0.04265364, "msgr_running_recv_time": 0.525744783, "msgr_running_send_time": 0.631387581, "msgr_running_total_time": 1.123843278, "msgr_send_bytes": 28118598, "msgr_send_encrypted_bytes": 8544, "msgr_send_messages": 25626, "msgr_send_messages_queue_lat": { "avgcount": 25626, "avgtime": 4.3759e-05, "sum": 1.1213861 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 691650, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 1343, "msgr_running_fast_dispatch_time": 0.039243855, "msgr_running_recv_time": 0.025375514, "msgr_running_send_time": 0.055037145, "msgr_running_total_time": 0.126991325, "msgr_send_bytes": 3351325, "msgr_send_encrypted_bytes": 95808, "msgr_send_messages": 1350, "msgr_send_messages_queue_lat": { "avgcount": 1349, "avgtime": 0.000444588, "sum": 0.599749422 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 120, "avgtime": 0.065657699, "sum": 7.878923993 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5213, "avgtime": 0.000343416, "sum": 1.790232553 }, "queue_len": 0 }, "mds": { "caps": 1040, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7615, "ceph_cap_op_revoke": 940, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 8, "handle_client_caps": 3472, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4356, "inodes_bottom": 4356, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4356, "inodes_top": 0, "inodes_with_caps": 1040, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 9603, "q": 0, "reply": 6411, "reply_latency": { "avgcount": 6411, "avgtime": 0.001498131, "sum": 9.60451788 }, "request": 6413, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6419, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6418, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3201, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 1043, "evadd": 13074, "evex": 12034, "evexd": 12033, "evexg": 0, "evlrg": 0, "evtrm": 12034, "expos": 28711929, "jlat": { "avgcount": 24, "avgtime": 0.146845868, "sum": 3.524300847 }, "rdpos": 4195244, "replayed": 3, "seg": 2, "segadd": 17, "segex": 17, "segexd": 16, "segexg": 0, "segmjr": 1, "segtrm": 17, "wrpos": 29340566 }, "mds_mem": { "cap": 1, "cap+": 3202, "cap-": 3201, "dir": 12, "dir+": 13, "dir-": 1, "dn": 1050, "dn+": 6412, "dn-": 5362, "heap": 248164, "ino": 1052, "ino+": 3214, "ino-": 2162, "rss": 105604 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6419, "dispatch_server_request": 0, "handle_client_request": 6413, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000973325, "sum": 3.114640424 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.032980042, "sum": 0.197880257 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002834805, "sum": 0.002834805 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.040080051, "sum": 0.120240153 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.330014075, "sum": 0.330014075 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.001824658, "sum": 5.838908166 } }, "mds_sessions": { "average_load": 5853, "avg_session_uptime": 774, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5853 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 10708958, "buffer_anon_items": 3161, "buffer_meta_bytes": 176, "buffer_meta_items": 2, "mds_co_bytes": 11436119, "mds_co_items": 137238, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 12, "omap_rd": 48, "omap_wr": 51, "op": 4427, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 4427, "avgtime": 0.015234113, "sum": 67.441422413 }, "op_pg": 0, "op_r": 59, "op_reply": 4427, "op_resend": 0, "op_rmw": 0, "op_send": 4427, "op_send_bytes": 25974705, "op_w": 4368, "oplen_avg": { "avgcount": 4427, "sum": 6617 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 1051, "osdop_delete": 3222, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 31, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 2102, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 47, "osdop_writefull": 16, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 3201, "pq_executed_ops": 3201, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 14368, "get_or_fail_fail": 0, "get_or_fail_success": 14368, "get_started": 0, "get_sum": 4007777, "max": 104857600, "put": 14368, "put_sum": 4007777, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 4427, "put_sum": 26299123, "take": 4427, "take_sum": 26299123, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 4427, "put_sum": 4427, "take": 4427, "take_sum": 4427, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3201, "get_or_fail_fail": 0, "get_or_fail_success": 3201, "get_started": 0, "get_sum": 323301, "max": 3758096384, "put": 23, "put_sum": 323301, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55b1d3c8aa00": { "get": 13074, "get_or_fail_fail": 0, "get_or_fail_success": 13074, "get_started": 0, "get_sum": 25144881, "max": 3758096384, "put": 24, "put_sum": 24519925, "take": 0, "take_sum": 0, "val": 624956, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:52.528 INFO:tasks.cephfs.test_strays:Complete purge in 7 seconds 2024-03-06T07:26:52.528 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:52.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.627+0000 7fd118288640 1 Processor -- start 2024-03-06T07:26:52.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.627+0000 7fd118288640 1 -- start start 2024-03-06T07:26:52.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.628+0000 7fd118288640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd11015e150 0x7fd1100cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.628+0000 7fd118288640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd1100cfba0 0x7fd1100d0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.628+0000 7fd118288640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1100d1ea0 con 0x7fd1100d0e90 2024-03-06T07:26:52.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.628+0000 7fd118288640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1100d2ab0 con 0x7fd1100cfba0 2024-03-06T07:26:52.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.628+0000 7fd118288640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1100d2180 con 0x7fd11015e150 2024-03-06T07:26:52.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.629+0000 7fd115823640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd1100cfba0 0x7fd1100d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.629+0000 7fd115823640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd1100cfba0 0x7fd1100d0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53374/0 (socket says 172.21.15.83:53374) 2024-03-06T07:26:52.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.629+0000 7fd115823640 1 -- 172.21.15.83:0/3887652804 learned_addr learned my addr 172.21.15.83:0/3887652804 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:52.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.629+0000 7fd116024640 1 --2- 172.21.15.83:0/3887652804 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd11015e150 0x7fd1100cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.629+0000 7fd115823640 1 -- 172.21.15.83:0/3887652804 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd11015e150 msgr2=0x7fd1100cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:52.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.629+0000 7fd115823640 1 --2- 172.21.15.83:0/3887652804 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd11015e150 0x7fd1100cf660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.629+0000 7fd115823640 1 -- 172.21.15.83:0/3887652804 >> v1:172.21.15.183:6789/0 conn(0x7fd1100d0e90 legacy=0x7fd1100b9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.629+0000 7fd115823640 1 -- 172.21.15.83:0/3887652804 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1100d2460 con 0x7fd1100cfba0 2024-03-06T07:26:52.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.630+0000 7fd115823640 1 --2- 172.21.15.83:0/3887652804 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1100cfba0 0x7fd1100d0950 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fd100009870 tx=0x7fd10002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=985cc12581a86d2b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:52.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.630+0000 7fd115022640 1 -- 172.21.15.83:0/3887652804 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd100035020 con 0x7fd1100cfba0 2024-03-06T07:26:52.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.630+0000 7fd115022640 1 -- 172.21.15.83:0/3887652804 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd100031640 con 0x7fd1100cfba0 2024-03-06T07:26:52.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.630+0000 7fd115022640 1 -- 172.21.15.83:0/3887652804 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd10002d040 con 0x7fd1100cfba0 2024-03-06T07:26:52.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.630+0000 7fd118288640 1 -- 172.21.15.83:0/3887652804 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1100cfba0 msgr2=0x7fd1100d0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.630+0000 7fd118288640 1 --2- 172.21.15.83:0/3887652804 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1100cfba0 0x7fd1100d0950 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fd100009870 tx=0x7fd10002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.630+0000 7fd118288640 1 -- 172.21.15.83:0/3887652804 shutdown_connections 2024-03-06T07:26:52.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.630+0000 7fd118288640 1 --2- 172.21.15.83:0/3887652804 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1100cfba0 0x7fd1100d0950 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.630+0000 7fd118288640 1 --2- 172.21.15.83:0/3887652804 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd11015e150 0x7fd1100cf660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.630+0000 7fd118288640 1 -- 172.21.15.83:0/3887652804 >> 172.21.15.83:0/3887652804 conn(0x7fd11009d230 msgr2=0x7fd1100d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:52.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.630+0000 7fd118288640 1 -- 172.21.15.83:0/3887652804 shutdown_connections 2024-03-06T07:26:52.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.630+0000 7fd118288640 1 -- 172.21.15.83:0/3887652804 wait complete. 2024-03-06T07:26:52.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.631+0000 7fd118288640 1 Processor -- start 2024-03-06T07:26:52.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.632+0000 7fd118288640 1 -- start start 2024-03-06T07:26:52.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.632+0000 7fd118288640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd1100cfba0 0x7fd1100c77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.632+0000 7fd118288640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1100d0e90 0x7fd11014f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.632+0000 7fd118288640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd11015e150 0x7fd1100b8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.632+0000 7fd118288640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1100d1b30 con 0x7fd1100d0e90 2024-03-06T07:26:52.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.632+0000 7fd118288640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1100d2f40 con 0x7fd11015e150 2024-03-06T07:26:52.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.632+0000 7fd118288640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd11015e530 con 0x7fd1100cfba0 2024-03-06T07:26:52.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.632+0000 7fd115823640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1100d0e90 0x7fd11014f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.632+0000 7fd115823640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1100d0e90 0x7fd11014f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53386/0 (socket says 172.21.15.83:53386) 2024-03-06T07:26:52.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.632+0000 7fd115823640 1 -- 172.21.15.83:0/3277968596 learned_addr learned my addr 172.21.15.83:0/3277968596 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:52.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.632+0000 7fd116024640 1 --2- 172.21.15.83:0/3277968596 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd1100cfba0 0x7fd1100c77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.632+0000 7fd116825640 1 --2- 172.21.15.83:0/3277968596 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd11015e150 0x7fd1100b8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.633+0000 7fd115823640 1 -- 172.21.15.83:0/3277968596 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd1100cfba0 msgr2=0x7fd1100c77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.633+0000 7fd115823640 1 --2- 172.21.15.83:0/3277968596 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd1100cfba0 0x7fd1100c77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.633+0000 7fd115823640 1 -- 172.21.15.83:0/3277968596 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd11015e150 msgr2=0x7fd1100b8ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.633+0000 7fd115823640 1 --2- 172.21.15.83:0/3277968596 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd11015e150 0x7fd1100b8ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.633+0000 7fd115823640 1 -- 172.21.15.83:0/3277968596 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1100ca620 con 0x7fd1100d0e90 2024-03-06T07:26:52.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.633+0000 7fd115823640 1 --2- 172.21.15.83:0/3277968596 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1100d0e90 0x7fd11014f280 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7fd100000c00 tx=0x7fd1000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:52.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.633+0000 7fd106ffd640 1 -- 172.21.15.83:0/3277968596 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd100040070 con 0x7fd1100d0e90 2024-03-06T07:26:52.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.633+0000 7fd106ffd640 1 -- 172.21.15.83:0/3277968596 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1000399f0 con 0x7fd1100d0e90 2024-03-06T07:26:52.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.633+0000 7fd118288640 1 -- 172.21.15.83:0/3277968596 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd11014f7c0 con 0x7fd1100d0e90 2024-03-06T07:26:52.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.633+0000 7fd106ffd640 1 -- 172.21.15.83:0/3277968596 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd100035040 con 0x7fd1100d0e90 2024-03-06T07:26:52.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.633+0000 7fd118288640 1 -- 172.21.15.83:0/3277968596 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1100c81c0 con 0x7fd1100d0e90 2024-03-06T07:26:52.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.634+0000 7fd118288640 1 -- 172.21.15.83:0/3277968596 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd110069ee0 con 0x7fd1100d0e90 2024-03-06T07:26:52.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.634+0000 7fd106ffd640 1 -- 172.21.15.83:0/3277968596 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd10004b060 con 0x7fd1100d0e90 2024-03-06T07:26:52.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.635+0000 7fd106ffd640 1 --2- 172.21.15.83:0/3277968596 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd0f406b4d0 0x7fd0f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.635+0000 7fd106ffd640 1 -- 172.21.15.83:0/3277968596 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fd1000aba10 con 0x7fd1100d0e90 2024-03-06T07:26:52.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.635+0000 7fd116024640 1 --2- 172.21.15.83:0/3277968596 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd0f406b4d0 0x7fd0f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.635+0000 7fd116024640 1 --2- 172.21.15.83:0/3277968596 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd0f406b4d0 0x7fd0f406d990 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fd10c009aa0 tx=0x7fd10c009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:52.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.639+0000 7fd106ffd640 1 -- 172.21.15.83:0/3277968596 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd10007a1e0 con 0x7fd1100d0e90 2024-03-06T07:26:52.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.851+0000 7fd118288640 1 -- 172.21.15.83:0/3277968596 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1100d4670 con 0x7fd1100d0e90 2024-03-06T07:26:52.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.852+0000 7fd106ffd640 1 -- 172.21.15.83:0/3277968596 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+4983 (secure 0 0 0) 0x7fd10007e280 con 0x7fd1100d0e90 2024-03-06T07:26:52.854 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:52.854 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":53,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6023,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/601236077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":601236077},{"type":"v1","addr":"172.21.15.183:6833","nonce":601236077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6029,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1435055961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1435055961},{"type":"v1","addr":"172.21.15.183:6835","nonce":1435055961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":7572,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/2726045966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2726045966},{"type":"v1","addr":"172.21.15.83:6836","nonce":2726045966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.064335+0000","modified":"2024-03-06T07:26:32.814889+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7575":{"gid":7575,"name":"a","rank":0,"incarnation":49,"state":"up:active","state_seq":5,"addr":"172.21.15.83:6837/3254575964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":3254575964},{"type":"v1","addr":"172.21.15.83:6837","nonce":3254575964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7575,"qdb_cluster":[7575]},"id":7}]} 2024-03-06T07:26:52.855 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 53 2024-03-06T07:26:52.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.854+0000 7fd118288640 1 -- 172.21.15.83:0/3277968596 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd0f406b4d0 msgr2=0x7fd0f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.854+0000 7fd118288640 1 --2- 172.21.15.83:0/3277968596 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd0f406b4d0 0x7fd0f406d990 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fd10c009aa0 tx=0x7fd10c009190 comp rx=0 tx=0).stop 2024-03-06T07:26:52.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.854+0000 7fd118288640 1 -- 172.21.15.83:0/3277968596 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1100d0e90 msgr2=0x7fd11014f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.854+0000 7fd118288640 1 --2- 172.21.15.83:0/3277968596 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1100d0e90 0x7fd11014f280 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7fd100000c00 tx=0x7fd1000046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.854+0000 7fd118288640 1 -- 172.21.15.83:0/3277968596 shutdown_connections 2024-03-06T07:26:52.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.854+0000 7fd118288640 1 --2- 172.21.15.83:0/3277968596 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd0f406b4d0 0x7fd0f406d990 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.854+0000 7fd118288640 1 --2- 172.21.15.83:0/3277968596 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd11015e150 0x7fd1100b8ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.854+0000 7fd118288640 1 --2- 172.21.15.83:0/3277968596 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1100d0e90 0x7fd11014f280 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.854+0000 7fd118288640 1 --2- 172.21.15.83:0/3277968596 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd1100cfba0 0x7fd1100c77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.854+0000 7fd118288640 1 -- 172.21.15.83:0/3277968596 >> 172.21.15.83:0/3277968596 conn(0x7fd11009d230 msgr2=0x7fd11006d300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:52.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.854+0000 7fd118288640 1 -- 172.21.15.83:0/3277968596 shutdown_connections 2024-03-06T07:26:52.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:52.854+0000 7fd118288640 1 -- 172.21.15.83:0/3277968596 wait complete. 2024-03-06T07:26:52.868 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:26:53.014 INFO:teuthology.orchestra.run.smithi083.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1852,"msgr_send_messages":1852,"msgr_recv_bytes":516330,"msgr_send_bytes":5275307,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.186047210,"msgr_running_send_time":0.096803110,"msgr_running_recv_time":0.033128621,"msgr_running_fast_dispatch_time":0.043227075,"msgr_send_messages_queue_lat":{"avgcount":1851,"sum":0.427594116,"avgtime":0.000231007},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000209,"avgtime":0.000000104},"msgr_recv_encrypted_bytes":1568,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":11179,"msgr_send_messages":25626,"msgr_recv_bytes":4096275,"msgr_send_bytes":28118598,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":1.123922923,"msgr_running_send_time":0.631415299,"msgr_running_recv_time":0.525765987,"msgr_running_fast_dispatch_time":0.042682120,"msgr_send_messages_queue_lat":{"avgcount":25626,"sum":1.121386100,"avgtime":0.000043759},"msgr_handle_ack_lat":{"avgcount":30800,"sum":0.005373201,"avgtime":0.000000174},"msgr_recv_encrypted_bytes":108320,"msgr_send_encrypted_bytes":8544},"AsyncMessenger::Worker-2":{"msgr_recv_messages":1344,"msgr_send_messages":1351,"msgr_recv_bytes":691896,"msgr_send_bytes":3976599,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.129652885,"msgr_running_send_time":0.056712553,"msgr_running_recv_time":0.025404664,"msgr_running_fast_dispatch_time":0.040192891,"msgr_send_messages_queue_lat":{"avgcount":1350,"sum":0.600014066,"avgtime":0.000444454},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":95808},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":122,"sum":7.918109393,"avgtime":0.064902536}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5213,"sum":1.790232553,"avgtime":0.000343416}},"mds":{"request":6413,"reply":6411,"reply_latency":{"avgcount":6411,"sum":9.604517880,"avgtime":0.001498131},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":1050,"inodes_top":0,"inodes_bottom":1050,"inodes_pin_tail":0,"inodes_pinned":1050,"inodes_expired":0,"inodes_with_caps":1,"caps":1,"subtrees":2,"traverse":6419,"traverse_hit":6418,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":940,"ceph_cap_op_grant":7615,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3472,"handle_client_caps_dirty":3000,"handle_client_cap_release":8,"process_request_cap_release":9603},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3201,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":13074,"evex":12034,"evtrm":12034,"ev":1043,"evexg":0,"evexd":12033,"segadd":17,"segex":17,"segtrm":17,"seg":2,"segmjr":1,"segexg":0,"segexd":16,"expos":28711929,"wrpos":29340566,"rdpos":4195244,"jlat":{"avgcount":25,"sum":3.531266590,"avgtime":0.141250663},"replayed":3},"mds_mem":{"ino":1052,"ino+":3214,"ino-":2162,"dir":12,"dir+":13,"dir-":1,"dn":1050,"dn+":6412,"dn-":5362,"cap":1,"cap+":3202,"cap-":3201,"rss":105604,"heap":248164},"mds_server":{"dispatch_client_request":6419,"dispatch_server_request":0,"handle_client_request":6413,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.114640424,"avgtime":0.000973325},"req_getattr_latency":{"avgcount":6,"sum":0.197880257,"avgtime":0.032980042},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002834805,"avgtime":0.002834805},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.120240153,"avgtime":0.040080051},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.330014075,"avgtime":0.330014075},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":5.838908166,"avgtime":0.001824658},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5853,"average_load":5853,"avg_session_uptime":779,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":8611868,"buffer_anon_items":2176,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":3343738,"mds_co_items":55331,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":4428,"op_send_bytes":26599661,"op_resend":0,"op_reply":4428,"op_latency":{"avgcount":4428,"sum":67.447422273,"avgtime":0.015232028},"op_inflight":0,"oplen_avg":{"avgcount":4428,"sum":6618},"op":4428,"op_r":59,"op_w":4369,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":1051,"osdop_read":31,"osdop_write":48,"osdop_writefull":16,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3222,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":2102,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":51,"omap_rd":48,"omap_del":12},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3201,"pq_executed":3201,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":14370,"get_sum":4008261,"get_or_fail_fail":0,"get_or_fail_success":14370,"take":0,"take_sum":0,"put":14370,"put_sum":4008261,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":4428,"take_sum":26924079,"put":4428,"put_sum":26924079,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":4428,"take_sum":4428,"put":4428,"put_sum":4428,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3201,"get_sum":323301,"get_or_fail_fail":0,"get_or_fail_success":3201,"take":0,"take_sum":0,"put":23,"put_sum":323301,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55b1d3c8aa00":{"val":0,"max":3758096384,"get_started":0,"get":13074,"get_sum":25144881,"get_or_fail_fail":0,"get_or_fail_success":13074,"take":0,"take_sum":0,"put":25,"put_sum":25144881,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:53.016 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.04e-07, "sum": 2.09e-07 }, "msgr_recv_bytes": 516330, "msgr_recv_encrypted_bytes": 1568, "msgr_recv_messages": 1852, "msgr_running_fast_dispatch_time": 0.043227075, "msgr_running_recv_time": 0.033128621, "msgr_running_send_time": 0.09680311, "msgr_running_total_time": 0.18604721, "msgr_send_bytes": 5275307, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 1852, "msgr_send_messages_queue_lat": { "avgcount": 1851, "avgtime": 0.000231007, "sum": 0.427594116 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 30800, "avgtime": 1.74e-07, "sum": 0.005373201 }, "msgr_recv_bytes": 4096275, "msgr_recv_encrypted_bytes": 108320, "msgr_recv_messages": 11179, "msgr_running_fast_dispatch_time": 0.04268212, "msgr_running_recv_time": 0.525765987, "msgr_running_send_time": 0.631415299, "msgr_running_total_time": 1.123922923, "msgr_send_bytes": 28118598, "msgr_send_encrypted_bytes": 8544, "msgr_send_messages": 25626, "msgr_send_messages_queue_lat": { "avgcount": 25626, "avgtime": 4.3759e-05, "sum": 1.1213861 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 691896, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 1344, "msgr_running_fast_dispatch_time": 0.040192891, "msgr_running_recv_time": 0.025404664, "msgr_running_send_time": 0.056712553, "msgr_running_total_time": 0.129652885, "msgr_send_bytes": 3976599, "msgr_send_encrypted_bytes": 95808, "msgr_send_messages": 1351, "msgr_send_messages_queue_lat": { "avgcount": 1350, "avgtime": 0.000444454, "sum": 0.600014066 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 122, "avgtime": 0.064902536, "sum": 7.918109393 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5213, "avgtime": 0.000343416, "sum": 1.790232553 }, "queue_len": 0 }, "mds": { "caps": 1, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7615, "ceph_cap_op_revoke": 940, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 8, "handle_client_caps": 3472, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 1050, "inodes_bottom": 1050, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 1050, "inodes_top": 0, "inodes_with_caps": 1, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 9603, "q": 0, "reply": 6411, "reply_latency": { "avgcount": 6411, "avgtime": 0.001498131, "sum": 9.60451788 }, "request": 6413, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6419, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6418, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3201, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 1043, "evadd": 13074, "evex": 12034, "evexd": 12033, "evexg": 0, "evlrg": 0, "evtrm": 12034, "expos": 28711929, "jlat": { "avgcount": 25, "avgtime": 0.141250663, "sum": 3.53126659 }, "rdpos": 4195244, "replayed": 3, "seg": 2, "segadd": 17, "segex": 17, "segexd": 16, "segexg": 0, "segmjr": 1, "segtrm": 17, "wrpos": 29340566 }, "mds_mem": { "cap": 1, "cap+": 3202, "cap-": 3201, "dir": 12, "dir+": 13, "dir-": 1, "dn": 1050, "dn+": 6412, "dn-": 5362, "heap": 248164, "ino": 1052, "ino+": 3214, "ino-": 2162, "rss": 105604 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6419, "dispatch_server_request": 0, "handle_client_request": 6413, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000973325, "sum": 3.114640424 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.032980042, "sum": 0.197880257 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002834805, "sum": 0.002834805 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.040080051, "sum": 0.120240153 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.330014075, "sum": 0.330014075 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.001824658, "sum": 5.838908166 } }, "mds_sessions": { "average_load": 5853, "avg_session_uptime": 779, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5853 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 8611868, "buffer_anon_items": 2176, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 3343738, "mds_co_items": 55331, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 12, "omap_rd": 48, "omap_wr": 51, "op": 4428, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 4428, "avgtime": 0.015232028, "sum": 67.447422273 }, "op_pg": 0, "op_r": 59, "op_reply": 4428, "op_resend": 0, "op_rmw": 0, "op_send": 4428, "op_send_bytes": 26599661, "op_w": 4369, "oplen_avg": { "avgcount": 4428, "sum": 6618 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 1051, "osdop_delete": 3222, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 31, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 2102, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 48, "osdop_writefull": 16, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 3201, "pq_executed_ops": 3201, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 14370, "get_or_fail_fail": 0, "get_or_fail_success": 14370, "get_started": 0, "get_sum": 4008261, "max": 104857600, "put": 14370, "put_sum": 4008261, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 4428, "put_sum": 26924079, "take": 4428, "take_sum": 26924079, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 4428, "put_sum": 4428, "take": 4428, "take_sum": 4428, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3201, "get_or_fail_fail": 0, "get_or_fail_success": 3201, "get_started": 0, "get_sum": 323301, "max": 3758096384, "put": 23, "put_sum": 323301, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55b1d3c8aa00": { "get": 13074, "get_or_fail_fail": 0, "get_or_fail_success": 13074, "get_started": 0, "get_sum": 25144881, "max": 3758096384, "put": 25, "put_sum": 25144881, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:53.016 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:26:53.017 DEBUG:teuthology.orchestra.run.smithi083:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:53.060 DEBUG:teuthology.orchestra.run.smithi183:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:53.104 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:26:53.104 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:26:53.104 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:53.119 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:53.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.257+0000 7f64930ec640 1 Processor -- start 2024-03-06T07:26:53.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.257+0000 7f64930ec640 1 -- start start 2024-03-06T07:26:53.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.258+0000 7f64930ec640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f648c15e150 0x7f648c0cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.258+0000 7f64930ec640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f648c0cfba0 0x7f648c0d0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.258+0000 7f64930ec640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f648c0d0e90 0x7f648c0b9060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.258+0000 7f64930ec640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f648c0d2ab0 con 0x7f648c0d0e90 2024-03-06T07:26:53.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.258+0000 7f64930ec640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f648c0d1ea0 con 0x7f648c0cfba0 2024-03-06T07:26:53.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.258+0000 7f64930ec640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f648c0d2180 con 0x7f648c15e150 2024-03-06T07:26:53.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.258+0000 7f6491689640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f648c0d0e90 0x7f648c0b9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.258+0000 7f6491689640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f648c0d0e90 0x7f648c0b9060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53394/0 (socket says 172.21.15.83:53394) 2024-03-06T07:26:53.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.258+0000 7f6491689640 1 -- 172.21.15.83:0/4282737649 learned_addr learned my addr 172.21.15.83:0/4282737649 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:53.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.258+0000 7f6490e88640 1 --2- 172.21.15.83:0/4282737649 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f648c15e150 0x7f648c0cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.258+0000 7f6491689640 1 -- 172.21.15.83:0/4282737649 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f648c15e150 msgr2=0x7f648c0cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:53.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.258+0000 7f6491689640 1 --2- 172.21.15.83:0/4282737649 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f648c15e150 0x7f648c0cf660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.258+0000 7f6491689640 1 -- 172.21.15.83:0/4282737649 >> v2:172.21.15.183:3300/0 conn(0x7f648c0cfba0 msgr2=0x7f648c0d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:53.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.258+0000 7f6491689640 1 --2- 172.21.15.83:0/4282737649 >> v2:172.21.15.183:3300/0 conn(0x7f648c0cfba0 0x7f648c0d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.258+0000 7f6491689640 1 -- 172.21.15.83:0/4282737649 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f648c0d2460 con 0x7f648c0d0e90 2024-03-06T07:26:53.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.259+0000 7f6491689640 1 --2- 172.21.15.83:0/4282737649 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f648c0d0e90 0x7f648c0b9060 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f648c08c920 tx=0x7f648402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed904e7103de457b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:53.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.259+0000 7f648b7fe640 1 -- 172.21.15.83:0/4282737649 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648402c9e0 con 0x7f648c0d0e90 2024-03-06T07:26:53.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.259+0000 7f648b7fe640 1 -- 172.21.15.83:0/4282737649 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f648402cb80 con 0x7f648c0d0e90 2024-03-06T07:26:53.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.259+0000 7f648b7fe640 1 -- 172.21.15.83:0/4282737649 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64840343f0 con 0x7f648c0d0e90 2024-03-06T07:26:53.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.260+0000 7f64930ec640 1 -- 172.21.15.83:0/4282737649 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f648c0d0e90 msgr2=0x7f648c0b9060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.260+0000 7f64930ec640 1 --2- 172.21.15.83:0/4282737649 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f648c0d0e90 0x7f648c0b9060 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f648c08c920 tx=0x7f648402be40 comp rx=0 tx=0).stop 2024-03-06T07:26:53.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.260+0000 7f64930ec640 1 -- 172.21.15.83:0/4282737649 shutdown_connections 2024-03-06T07:26:53.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.260+0000 7f64930ec640 1 --2- 172.21.15.83:0/4282737649 >> v2:172.21.15.183:3300/0 conn(0x7f648c0cfba0 0x7f648c0d0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.260+0000 7f64930ec640 1 --2- 172.21.15.83:0/4282737649 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f648c0d0e90 0x7f648c0b9060 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.260+0000 7f64930ec640 1 --2- 172.21.15.83:0/4282737649 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f648c15e150 0x7f648c0cf660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.260+0000 7f64930ec640 1 -- 172.21.15.83:0/4282737649 >> 172.21.15.83:0/4282737649 conn(0x7f648c09d230 msgr2=0x7f648c0d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:53.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.260+0000 7f64930ec640 1 -- 172.21.15.83:0/4282737649 shutdown_connections 2024-03-06T07:26:53.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.260+0000 7f64930ec640 1 -- 172.21.15.83:0/4282737649 wait complete. 2024-03-06T07:26:53.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.261+0000 7f64930ec640 1 Processor -- start 2024-03-06T07:26:53.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.261+0000 7f64930ec640 1 -- start start 2024-03-06T07:26:53.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.262+0000 7f64930ec640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f648c0cfba0 0x7f648c0c93b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.262+0000 7f64930ec640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f648c0d0e90 0x7f648c06cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.262+0000 7f64930ec640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f648c15e150 0x7f648c15a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.262+0000 7f64930ec640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f648c0d1b30 con 0x7f648c15e150 2024-03-06T07:26:53.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.262+0000 7f64930ec640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f648c0d2f40 con 0x7f648c0d0e90 2024-03-06T07:26:53.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.262+0000 7f64930ec640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f648c15e530 con 0x7f648c0cfba0 2024-03-06T07:26:53.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.262+0000 7f6491689640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f648c15e150 0x7f648c15a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.262+0000 7f6491689640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f648c15e150 0x7f648c15a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53402/0 (socket says 172.21.15.83:53402) 2024-03-06T07:26:53.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.262+0000 7f6491689640 1 -- 172.21.15.83:0/3946614485 learned_addr learned my addr 172.21.15.83:0/3946614485 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:53.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.262+0000 7f6490e88640 1 --2- 172.21.15.83:0/3946614485 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f648c0cfba0 0x7f648c0c93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.262+0000 7f648bfff640 1 --2- 172.21.15.83:0/3946614485 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f648c0d0e90 0x7f648c06cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.262+0000 7f6491689640 1 -- 172.21.15.83:0/3946614485 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f648c0cfba0 msgr2=0x7f648c0c93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.262+0000 7f6491689640 1 --2- 172.21.15.83:0/3946614485 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f648c0cfba0 0x7f648c0c93b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.262+0000 7f6491689640 1 -- 172.21.15.83:0/3946614485 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f648c0d0e90 msgr2=0x7f648c06cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.262+0000 7f6491689640 1 --2- 172.21.15.83:0/3946614485 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f648c0d0e90 0x7f648c06cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.262+0000 7f6491689640 1 -- 172.21.15.83:0/3946614485 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f648c0c9b20 con 0x7f648c15e150 2024-03-06T07:26:53.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.262+0000 7f648bfff640 1 --2- 172.21.15.83:0/3946614485 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f648c0d0e90 0x7f648c06cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:53.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.262+0000 7f6490e88640 1 --2- 172.21.15.83:0/3946614485 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f648c0cfba0 0x7f648c0c93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:53.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.262+0000 7f6491689640 1 --2- 172.21.15.83:0/3946614485 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f648c15e150 0x7f648c15a0c0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f6484009f40 tx=0x7f6484004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:53.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.262+0000 7f64897fa640 1 -- 172.21.15.83:0/3946614485 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6484043020 con 0x7f648c15e150 2024-03-06T07:26:53.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.263+0000 7f64897fa640 1 -- 172.21.15.83:0/3946614485 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f648403e450 con 0x7f648c15e150 2024-03-06T07:26:53.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.263+0000 7f64930ec640 1 -- 172.21.15.83:0/3946614485 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f648c0c9e00 con 0x7f648c15e150 2024-03-06T07:26:53.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.263+0000 7f64897fa640 1 -- 172.21.15.83:0/3946614485 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6484009510 con 0x7f648c15e150 2024-03-06T07:26:53.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.263+0000 7f64930ec640 1 -- 172.21.15.83:0/3946614485 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f648c0ca3c0 con 0x7f648c15e150 2024-03-06T07:26:53.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.264+0000 7f64930ec640 1 -- 172.21.15.83:0/3946614485 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f648c069ee0 con 0x7f648c15e150 2024-03-06T07:26:53.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.264+0000 7f64897fa640 1 -- 172.21.15.83:0/3946614485 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6484046020 con 0x7f648c15e150 2024-03-06T07:26:53.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.264+0000 7f64897fa640 1 --2- 172.21.15.83:0/3946614485 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f647006b4d0 0x7f647006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.264+0000 7f64897fa640 1 -- 172.21.15.83:0/3946614485 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f64840acb70 con 0x7f648c15e150 2024-03-06T07:26:53.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.264+0000 7f6490e88640 1 --2- 172.21.15.83:0/3946614485 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f647006b4d0 0x7f647006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.265+0000 7f6490e88640 1 --2- 172.21.15.83:0/3946614485 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f647006b4d0 0x7f647006d990 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f648000a9d0 tx=0x7f6480008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:53.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.268+0000 7f64897fa640 1 -- 172.21.15.83:0/3946614485 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f648404d030 con 0x7f648c15e150 2024-03-06T07:26:53.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.446+0000 7f64930ec640 1 -- 172.21.15.83:0/3946614485 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f648c0d4670 con 0x7f648c15e150 2024-03-06T07:26:53.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.447+0000 7f64897fa640 1 -- 172.21.15.83:0/3946614485 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+16749 (secure 0 0 0) 0x7f648407ba20 con 0x7f648c15e150 2024-03-06T07:26:53.449 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:53.449 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":98,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:26:27.785066+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:17.275618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:18.028872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.83:6837/2924411033":"2024-03-07T07:26:27.785046+0000","172.21.15.83:6835/2924411033":"2024-03-07T07:26:27.785046+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:53.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.449+0000 7f64930ec640 1 -- 172.21.15.83:0/3946614485 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f647006b4d0 msgr2=0x7f647006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.449+0000 7f64930ec640 1 --2- 172.21.15.83:0/3946614485 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f647006b4d0 0x7f647006d990 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f648000a9d0 tx=0x7f6480008040 comp rx=0 tx=0).stop 2024-03-06T07:26:53.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.449+0000 7f64930ec640 1 -- 172.21.15.83:0/3946614485 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f648c15e150 msgr2=0x7f648c15a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.449+0000 7f64930ec640 1 --2- 172.21.15.83:0/3946614485 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f648c15e150 0x7f648c15a0c0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f6484009f40 tx=0x7f6484004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:53.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.449+0000 7f64930ec640 1 -- 172.21.15.83:0/3946614485 shutdown_connections 2024-03-06T07:26:53.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.449+0000 7f64930ec640 1 --2- 172.21.15.83:0/3946614485 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f647006b4d0 0x7f647006d990 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.449+0000 7f64930ec640 1 --2- 172.21.15.83:0/3946614485 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f648c15e150 0x7f648c15a0c0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.449+0000 7f64930ec640 1 --2- 172.21.15.83:0/3946614485 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f648c0d0e90 0x7f648c06cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.449+0000 7f64930ec640 1 --2- 172.21.15.83:0/3946614485 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f648c0cfba0 0x7f648c0c93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.449+0000 7f64930ec640 1 -- 172.21.15.83:0/3946614485 >> 172.21.15.83:0/3946614485 conn(0x7f648c09d230 msgr2=0x7f648c15f970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:53.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.449+0000 7f64930ec640 1 -- 172.21.15.83:0/3946614485 shutdown_connections 2024-03-06T07:26:53.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.449+0000 7f64930ec640 1 -- 172.21.15.83:0/3946614485 wait complete. 2024-03-06T07:26:53.462 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:26:17.275618+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '93', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:26:18.028872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '94', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:26:53.462 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:26:53.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.567+0000 7fd47a73e640 1 Processor -- start 2024-03-06T07:26:53.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.567+0000 7fd47a73e640 1 -- start start 2024-03-06T07:26:53.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.567+0000 7fd47a73e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd4740d0950 0x7fd4740d0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.567+0000 7fd47a73e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fd4740b9060 0x7fd4740b94f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.567+0000 7fd47a73e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4740d1ea0 con 0x7fd47415e150 2024-03-06T07:26:53.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.567+0000 7fd47a73e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4740d12f0 con 0x7fd4740b9060 2024-03-06T07:26:53.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.567+0000 7fd47a73e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4740d2ab0 con 0x7fd4740d0950 2024-03-06T07:26:53.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.568+0000 7fd4737fe640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd4740d0950 0x7fd4740d0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.568+0000 7fd478cdb640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fd4740b9060 0x7fd4740b94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.568+0000 7fd4737fe640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd4740d0950 0x7fd4740d0db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:46462/0 (socket says 172.21.15.83:46462) 2024-03-06T07:26:53.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.568+0000 7fd478cdb640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fd4740b9060 0x7fd4740b94f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:58124/0 (socket says 172.21.15.83:58124) 2024-03-06T07:26:53.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.568+0000 7fd4737fe640 1 -- 172.21.15.83:0/4189420249 learned_addr learned my addr 172.21.15.83:0/4189420249 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:53.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.568+0000 7fd473fff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fd47415e150 0x7fd4740cf660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:50662/0 (socket says 172.21.15.83:50662) 2024-03-06T07:26:53.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.568+0000 7fd478cdb640 1 -- 172.21.15.83:0/4189420249 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd4740d0950 msgr2=0x7fd4740d0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:53.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.568+0000 7fd478cdb640 1 --2- 172.21.15.83:0/4189420249 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd4740d0950 0x7fd4740d0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.568+0000 7fd478cdb640 1 -- 172.21.15.83:0/4189420249 >> v1:172.21.15.183:6789/0 conn(0x7fd47415e150 legacy=0x7fd4740cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.568+0000 7fd478cdb640 1 -- 172.21.15.83:0/4189420249 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4740d2180 con 0x7fd4740b9060 2024-03-06T07:26:53.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.568+0000 7fd4737fe640 1 --2- 172.21.15.83:0/4189420249 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd4740d0950 0x7fd4740d0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:53.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.569+0000 7fd478cdb640 1 --2- 172.21.15.83:0/4189420249 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd4740b9060 0x7fd4740b94f0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fd46c009d90 tx=0x7fd46c02be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7e31a40de17a39a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:53.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.569+0000 7fd472ffd640 1 -- 172.21.15.83:0/4189420249 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd46c039070 con 0x7fd4740b9060 2024-03-06T07:26:53.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.569+0000 7fd472ffd640 1 -- 172.21.15.83:0/4189420249 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd46c02cbc0 con 0x7fd4740b9060 2024-03-06T07:26:53.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.569+0000 7fd472ffd640 1 -- 172.21.15.83:0/4189420249 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd46c034440 con 0x7fd4740b9060 2024-03-06T07:26:53.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.570+0000 7fd47a73e640 1 -- 172.21.15.83:0/4189420249 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd4740b9060 msgr2=0x7fd4740b94f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.570+0000 7fd47a73e640 1 --2- 172.21.15.83:0/4189420249 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd4740b9060 0x7fd4740b94f0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fd46c009d90 tx=0x7fd46c02be10 comp rx=0 tx=0).stop 2024-03-06T07:26:53.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.570+0000 7fd47a73e640 1 -- 172.21.15.83:0/4189420249 shutdown_connections 2024-03-06T07:26:53.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.570+0000 7fd47a73e640 1 --2- 172.21.15.83:0/4189420249 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd4740b9060 0x7fd4740b94f0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.570+0000 7fd47a73e640 1 --2- 172.21.15.83:0/4189420249 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd4740d0950 0x7fd4740d0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.570+0000 7fd47a73e640 1 -- 172.21.15.83:0/4189420249 >> 172.21.15.83:0/4189420249 conn(0x7fd47409d230 msgr2=0x7fd4740d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:53.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.570+0000 7fd47a73e640 1 -- 172.21.15.83:0/4189420249 shutdown_connections 2024-03-06T07:26:53.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.570+0000 7fd47a73e640 1 -- 172.21.15.83:0/4189420249 wait complete. 2024-03-06T07:26:53.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.571+0000 7fd47a73e640 1 Processor -- start 2024-03-06T07:26:53.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.571+0000 7fd47a73e640 1 -- start start 2024-03-06T07:26:53.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.572+0000 7fd47a73e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd4740b9060 0x7fd4740ccd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.572+0000 7fd47a73e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd4740d0950 0x7fd4741514d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.572+0000 7fd47a73e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd47415e150 0x7fd47415f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.572+0000 7fd47a73e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4740d2f40 con 0x7fd4740b9060 2024-03-06T07:26:53.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.572+0000 7fd47a73e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4740d2740 con 0x7fd4740d0950 2024-03-06T07:26:53.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.572+0000 7fd47a73e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4740d1b30 con 0x7fd47415e150 2024-03-06T07:26:53.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.572+0000 7fd473fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd4740b9060 0x7fd4740ccd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.572+0000 7fd473fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd4740b9060 0x7fd4740ccd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53408/0 (socket says 172.21.15.83:53408) 2024-03-06T07:26:53.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.572+0000 7fd473fff640 1 -- 172.21.15.83:0/4222717736 learned_addr learned my addr 172.21.15.83:0/4222717736 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:53.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.572+0000 7fd478cdb640 1 --2- 172.21.15.83:0/4222717736 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd47415e150 0x7fd47415f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.572+0000 7fd4737fe640 1 --2- 172.21.15.83:0/4222717736 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd4740d0950 0x7fd4741514d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.572+0000 7fd473fff640 1 -- 172.21.15.83:0/4222717736 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd47415e150 msgr2=0x7fd47415f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.572+0000 7fd473fff640 1 --2- 172.21.15.83:0/4222717736 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd47415e150 0x7fd47415f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.572+0000 7fd473fff640 1 -- 172.21.15.83:0/4222717736 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd4740d0950 msgr2=0x7fd4741514d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.572+0000 7fd473fff640 1 --2- 172.21.15.83:0/4222717736 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd4740d0950 0x7fd4741514d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.572+0000 7fd473fff640 1 -- 172.21.15.83:0/4222717736 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4740cbcd0 con 0x7fd4740b9060 2024-03-06T07:26:53.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.573+0000 7fd473fff640 1 --2- 172.21.15.83:0/4222717736 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd4740b9060 0x7fd4740ccd70 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fd46400c3c0 tx=0x7fd46400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:53.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.573+0000 7fd470ff9640 1 -- 172.21.15.83:0/4222717736 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4640074e0 con 0x7fd4740b9060 2024-03-06T07:26:53.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.573+0000 7fd470ff9640 1 -- 172.21.15.83:0/4222717736 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd464007680 con 0x7fd4740b9060 2024-03-06T07:26:53.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.573+0000 7fd470ff9640 1 -- 172.21.15.83:0/4222717736 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd464004a00 con 0x7fd4740b9060 2024-03-06T07:26:53.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.573+0000 7fd47a73e640 1 -- 172.21.15.83:0/4222717736 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4740cbfb0 con 0x7fd4740b9060 2024-03-06T07:26:53.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.573+0000 7fd47a73e640 1 -- 172.21.15.83:0/4222717736 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4740cc570 con 0x7fd4740b9060 2024-03-06T07:26:53.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.574+0000 7fd47a73e640 1 -- 172.21.15.83:0/4222717736 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd474069ee0 con 0x7fd4740b9060 2024-03-06T07:26:53.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.574+0000 7fd470ff9640 1 -- 172.21.15.83:0/4222717736 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd46401a020 con 0x7fd4740b9060 2024-03-06T07:26:53.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.575+0000 7fd470ff9640 1 --2- 172.21.15.83:0/4222717736 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd46006b4d0 0x7fd46006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.575+0000 7fd4737fe640 1 --2- 172.21.15.83:0/4222717736 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd46006b4d0 0x7fd46006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.575+0000 7fd470ff9640 1 -- 172.21.15.83:0/4222717736 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fd4640897f0 con 0x7fd4740b9060 2024-03-06T07:26:53.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.575+0000 7fd4737fe640 1 --2- 172.21.15.83:0/4222717736 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd46006b4d0 0x7fd46006d990 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fd4740b6d70 tx=0x7fd468009240 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:53.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.579+0000 7fd470ff9640 1 -- 172.21.15.83:0/4222717736 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd464058050 con 0x7fd4740b9060 2024-03-06T07:26:53.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.752+0000 7fd47a73e640 1 -- 172.21.15.83:0/4222717736 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd4740d4670 con 0x7fd4740b9060 2024-03-06T07:26:53.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.752+0000 7fd470ff9640 1 -- 172.21.15.83:0/4222717736 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v98) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd46405c0f0 con 0x7fd4740b9060 2024-03-06T07:26:53.754 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:26:53.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.754+0000 7fd47a73e640 1 -- 172.21.15.83:0/4222717736 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd46006b4d0 msgr2=0x7fd46006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.755+0000 7fd47a73e640 1 --2- 172.21.15.83:0/4222717736 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd46006b4d0 0x7fd46006d990 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fd4740b6d70 tx=0x7fd468009240 comp rx=0 tx=0).stop 2024-03-06T07:26:53.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.755+0000 7fd47a73e640 1 -- 172.21.15.83:0/4222717736 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd4740b9060 msgr2=0x7fd4740ccd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.755+0000 7fd47a73e640 1 --2- 172.21.15.83:0/4222717736 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd4740b9060 0x7fd4740ccd70 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fd46400c3c0 tx=0x7fd46400c880 comp rx=0 tx=0).stop 2024-03-06T07:26:53.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.755+0000 7fd47a73e640 1 -- 172.21.15.83:0/4222717736 shutdown_connections 2024-03-06T07:26:53.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.755+0000 7fd47a73e640 1 --2- 172.21.15.83:0/4222717736 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd46006b4d0 0x7fd46006d990 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.755+0000 7fd47a73e640 1 --2- 172.21.15.83:0/4222717736 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd47415e150 0x7fd47415f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.755+0000 7fd47a73e640 1 --2- 172.21.15.83:0/4222717736 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd4740d0950 0x7fd4741514d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.755+0000 7fd47a73e640 1 --2- 172.21.15.83:0/4222717736 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd4740b9060 0x7fd4740ccd70 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.755+0000 7fd47a73e640 1 -- 172.21.15.83:0/4222717736 >> 172.21.15.83:0/4222717736 conn(0x7fd47409d230 msgr2=0x7fd4740ceaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:53.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.755+0000 7fd47a73e640 1 -- 172.21.15.83:0/4222717736 shutdown_connections 2024-03-06T07:26:53.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.755+0000 7fd47a73e640 1 -- 172.21.15.83:0/4222717736 wait complete. 2024-03-06T07:26:53.767 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:26:53.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.869+0000 7fb9965e2640 1 Processor -- start 2024-03-06T07:26:53.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.869+0000 7fb9965e2640 1 -- start start 2024-03-06T07:26:53.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.870+0000 7fb9965e2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb990161670 0x7fb990161e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.870+0000 7fb9965e2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb9900b2da0 0x7fb9901623a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.870+0000 7fb9965e2640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb990162c70 con 0x7fb9900cc680 2024-03-06T07:26:53.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.870+0000 7fb9965e2640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9900b3590 con 0x7fb990161670 2024-03-06T07:26:53.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.870+0000 7fb9965e2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb99006cd80 con 0x7fb9900b2da0 2024-03-06T07:26:53.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.870+0000 7fb98ffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb990161670 0x7fb990161e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.870+0000 7fb994b7f640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fb9900cc680 0x7fb9900ccb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:33432/0 (socket says 172.21.15.83:33432) 2024-03-06T07:26:53.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.870+0000 7fb994b7f640 1 -- 172.21.15.83:0/2543529750 learned_addr learned my addr 172.21.15.83:0/2543529750 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:53.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.870+0000 7fb98ffff640 1 -- 172.21.15.83:0/2543529750 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb9900b2da0 msgr2=0x7fb9901623a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:53.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.870+0000 7fb98ffff640 1 --2- 172.21.15.83:0/2543529750 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb9900b2da0 0x7fb9901623a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.870+0000 7fb98ffff640 1 -- 172.21.15.83:0/2543529750 >> v1:172.21.15.83:6789/0 conn(0x7fb9900cc680 legacy=0x7fb9900ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.870+0000 7fb98ffff640 1 -- 172.21.15.83:0/2543529750 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb99006d060 con 0x7fb990161670 2024-03-06T07:26:53.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.870+0000 7fb98effd640 1 -- 172.21.15.83:0/2543529750 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2964259721 0 0) 0x7fb990162c70 con 0x7fb9900cc680 2024-03-06T07:26:53.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.871+0000 7fb98ffff640 1 --2- 172.21.15.83:0/2543529750 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb990161670 0x7fb990161e60 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fb9800097a0 tx=0x7fb98002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aa0220f23ba3122 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:53.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.871+0000 7fb98effd640 1 -- 172.21.15.83:0/2543529750 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb980035020 con 0x7fb990161670 2024-03-06T07:26:53.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.871+0000 7fb98effd640 1 -- 172.21.15.83:0/2543529750 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb980031830 con 0x7fb990161670 2024-03-06T07:26:53.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.871+0000 7fb98effd640 1 -- 172.21.15.83:0/2543529750 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb980031c00 con 0x7fb990161670 2024-03-06T07:26:53.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.872+0000 7fb9965e2640 1 -- 172.21.15.83:0/2543529750 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb990161670 msgr2=0x7fb990161e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.872+0000 7fb9965e2640 1 --2- 172.21.15.83:0/2543529750 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb990161670 0x7fb990161e60 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fb9800097a0 tx=0x7fb98002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.872+0000 7fb9965e2640 1 -- 172.21.15.83:0/2543529750 shutdown_connections 2024-03-06T07:26:53.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.872+0000 7fb9965e2640 1 --2- 172.21.15.83:0/2543529750 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb9900b2da0 0x7fb9901623a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.872+0000 7fb9965e2640 1 --2- 172.21.15.83:0/2543529750 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb990161670 0x7fb990161e60 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.872+0000 7fb9965e2640 1 -- 172.21.15.83:0/2543529750 >> 172.21.15.83:0/2543529750 conn(0x7fb9900d1d50 msgr2=0x7fb9900d2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:53.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.872+0000 7fb9965e2640 1 -- 172.21.15.83:0/2543529750 shutdown_connections 2024-03-06T07:26:53.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.872+0000 7fb9965e2640 1 -- 172.21.15.83:0/2543529750 wait complete. 2024-03-06T07:26:53.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.873+0000 7fb9965e2640 1 Processor -- start 2024-03-06T07:26:53.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.873+0000 7fb9965e2640 1 -- start start 2024-03-06T07:26:53.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.874+0000 7fb9965e2640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9900b2da0 0x7fb9901f4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.874+0000 7fb9965e2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9900cc680 0x7fb9900b4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.874+0000 7fb9965e2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb990161670 0x7fb9901452b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.874+0000 7fb9965e2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb99006d340 con 0x7fb9900cc680 2024-03-06T07:26:53.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.874+0000 7fb9965e2640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9901628e0 con 0x7fb9900b2da0 2024-03-06T07:26:53.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.874+0000 7fb9965e2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb99006d4c0 con 0x7fb990161670 2024-03-06T07:26:53.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.874+0000 7fb98f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9900cc680 0x7fb9900b4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.874+0000 7fb98f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9900cc680 0x7fb9900b4ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53434/0 (socket says 172.21.15.83:53434) 2024-03-06T07:26:53.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.874+0000 7fb98f7fe640 1 -- 172.21.15.83:0/1514725646 learned_addr learned my addr 172.21.15.83:0/1514725646 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:53.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.874+0000 7fb994b7f640 1 --2- 172.21.15.83:0/1514725646 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb990161670 0x7fb9901452b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.874+0000 7fb98ffff640 1 --2- 172.21.15.83:0/1514725646 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9900b2da0 0x7fb9901f4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.874+0000 7fb98f7fe640 1 -- 172.21.15.83:0/1514725646 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb990161670 msgr2=0x7fb9901452b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.874+0000 7fb98f7fe640 1 --2- 172.21.15.83:0/1514725646 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb990161670 0x7fb9901452b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.874+0000 7fb98f7fe640 1 -- 172.21.15.83:0/1514725646 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9900b2da0 msgr2=0x7fb9901f4120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.874+0000 7fb98f7fe640 1 --2- 172.21.15.83:0/1514725646 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9900b2da0 0x7fb9901f4120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.874+0000 7fb98f7fe640 1 -- 172.21.15.83:0/1514725646 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9900b50e0 con 0x7fb9900cc680 2024-03-06T07:26:53.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.874+0000 7fb98f7fe640 1 --2- 172.21.15.83:0/1514725646 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9900cc680 0x7fb9900b4ba0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fb98400b4a0 tx=0x7fb98400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:53.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.875+0000 7fb98cff9640 1 -- 172.21.15.83:0/1514725646 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb984014070 con 0x7fb9900cc680 2024-03-06T07:26:53.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.875+0000 7fb98cff9640 1 -- 172.21.15.83:0/1514725646 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb984010610 con 0x7fb9900cc680 2024-03-06T07:26:53.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.875+0000 7fb9965e2640 1 -- 172.21.15.83:0/1514725646 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9901457f0 con 0x7fb9900cc680 2024-03-06T07:26:53.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.875+0000 7fb98cff9640 1 -- 172.21.15.83:0/1514725646 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb984010a00 con 0x7fb9900cc680 2024-03-06T07:26:53.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.875+0000 7fb9965e2640 1 -- 172.21.15.83:0/1514725646 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9900b77b0 con 0x7fb9900cc680 2024-03-06T07:26:53.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.875+0000 7fb9965e2640 1 -- 172.21.15.83:0/1514725646 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb990068050 con 0x7fb9900cc680 2024-03-06T07:26:53.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.876+0000 7fb98cff9640 1 -- 172.21.15.83:0/1514725646 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb984017070 con 0x7fb9900cc680 2024-03-06T07:26:53.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.876+0000 7fb98cff9640 1 --2- 172.21.15.83:0/1514725646 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb97c06b4d0 0x7fb97c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.876+0000 7fb98cff9640 1 -- 172.21.15.83:0/1514725646 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb984089c80 con 0x7fb9900cc680 2024-03-06T07:26:53.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.876+0000 7fb98ffff640 1 --2- 172.21.15.83:0/1514725646 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb97c06b4d0 0x7fb97c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.877+0000 7fb98ffff640 1 --2- 172.21.15.83:0/1514725646 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb97c06b4d0 0x7fb97c06d990 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fb980036000 tx=0x7fb980008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:53.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:53.881+0000 7fb98cff9640 1 -- 172.21.15.83:0/1514725646 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb9901457f0 con 0x7fb9900cc680 2024-03-06T07:26:54.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.058+0000 7fb9965e2640 1 -- 172.21.15.83:0/1514725646 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb9900c8b90 con 0x7fb9900cc680 2024-03-06T07:26:54.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.058+0000 7fb98cff9640 1 -- 172.21.15.83:0/1514725646 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v98) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb984058aa0 con 0x7fb9900cc680 2024-03-06T07:26:54.060 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-03-06T07:26:54.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.060+0000 7fb9965e2640 1 -- 172.21.15.83:0/1514725646 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb97c06b4d0 msgr2=0x7fb97c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.060+0000 7fb9965e2640 1 --2- 172.21.15.83:0/1514725646 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb97c06b4d0 0x7fb97c06d990 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fb980036000 tx=0x7fb980008030 comp rx=0 tx=0).stop 2024-03-06T07:26:54.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.060+0000 7fb9965e2640 1 -- 172.21.15.83:0/1514725646 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9900cc680 msgr2=0x7fb9900b4ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.060+0000 7fb9965e2640 1 --2- 172.21.15.83:0/1514725646 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9900cc680 0x7fb9900b4ba0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fb98400b4a0 tx=0x7fb98400b960 comp rx=0 tx=0).stop 2024-03-06T07:26:54.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.060+0000 7fb9965e2640 1 -- 172.21.15.83:0/1514725646 shutdown_connections 2024-03-06T07:26:54.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.060+0000 7fb9965e2640 1 --2- 172.21.15.83:0/1514725646 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb97c06b4d0 0x7fb97c06d990 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.060+0000 7fb9965e2640 1 --2- 172.21.15.83:0/1514725646 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb990161670 0x7fb9901452b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.060+0000 7fb9965e2640 1 --2- 172.21.15.83:0/1514725646 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9900cc680 0x7fb9900b4ba0 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.060+0000 7fb9965e2640 1 --2- 172.21.15.83:0/1514725646 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9900b2da0 0x7fb9901f4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.060+0000 7fb9965e2640 1 -- 172.21.15.83:0/1514725646 >> 172.21.15.83:0/1514725646 conn(0x7fb9900d1d50 msgr2=0x7fb990162110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:54.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.060+0000 7fb9965e2640 1 -- 172.21.15.83:0/1514725646 shutdown_connections 2024-03-06T07:26:54.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.060+0000 7fb9965e2640 1 -- 172.21.15.83:0/1514725646 wait complete. 2024-03-06T07:26:54.072 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:26:54.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.175+0000 7f46359c7640 1 Processor -- start 2024-03-06T07:26:54.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.175+0000 7f46359c7640 1 -- start start 2024-03-06T07:26:54.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.176+0000 7f46359c7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46300b2da0 0x7f4630162570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.176+0000 7f46359c7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4630162c70 con 0x7f4630161670 2024-03-06T07:26:54.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.176+0000 7f46359c7640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f463006cd80 con 0x7f46300cd690 2024-03-06T07:26:54.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.176+0000 7f46359c7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46300b3590 con 0x7f46300b2da0 2024-03-06T07:26:54.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.176+0000 7f462effd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f4630161670 0x7f4630161e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:33436/0 (socket says 172.21.15.83:33436) 2024-03-06T07:26:54.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.176+0000 7f462effd640 1 -- 172.21.15.83:0/1392078210 learned_addr learned my addr 172.21.15.83:0/1392078210 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:54.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.176+0000 7f462f7fe640 1 --2- 172.21.15.83:0/1392078210 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46300b2da0 0x7f4630162570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.176+0000 7f462e7fc640 1 -- 172.21.15.83:0/1392078210 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 359538753 0 0) 0x7f4630162c70 con 0x7f4630161670 2024-03-06T07:26:54.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.176+0000 7f462e7fc640 1 -- 172.21.15.83:0/1392078210 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f461c003520 con 0x7f4630161670 2024-03-06T07:26:54.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.177+0000 7f462e7fc640 1 -- 172.21.15.83:0/1392078210 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 611075623 0 0) 0x7f461c003520 con 0x7f4630161670 2024-03-06T07:26:54.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.177+0000 7f462f7fe640 1 -- 172.21.15.83:0/1392078210 >> v1:172.21.15.183:6789/0 conn(0x7f46300cd690 legacy=0x7f46300cff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.177+0000 7f462f7fe640 1 -- 172.21.15.83:0/1392078210 >> v1:172.21.15.83:6789/0 conn(0x7f4630161670 legacy=0x7f4630161e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.177+0000 7f462f7fe640 1 -- 172.21.15.83:0/1392078210 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f463006d060 con 0x7f46300b2da0 2024-03-06T07:26:54.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.177+0000 7f462f7fe640 1 --2- 172.21.15.83:0/1392078210 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46300b2da0 0x7f4630162570 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4628009df0 tx=0x7f462802bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9920fd1b2510a511 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:54.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.178+0000 7f462e7fc640 1 -- 172.21.15.83:0/1392078210 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4628039070 con 0x7f46300b2da0 2024-03-06T07:26:54.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.178+0000 7f462e7fc640 1 -- 172.21.15.83:0/1392078210 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f462802c7b0 con 0x7f46300b2da0 2024-03-06T07:26:54.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.178+0000 7f462e7fc640 1 -- 172.21.15.83:0/1392078210 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4620001670 con 0x7f46300b2da0 2024-03-06T07:26:54.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.178+0000 7f46359c7640 1 -- 172.21.15.83:0/1392078210 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46300b2da0 msgr2=0x7f4630162570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.178+0000 7f46359c7640 1 --2- 172.21.15.83:0/1392078210 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46300b2da0 0x7f4630162570 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4628009df0 tx=0x7f462802bc80 comp rx=0 tx=0).stop 2024-03-06T07:26:54.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.178+0000 7f46359c7640 1 -- 172.21.15.83:0/1392078210 shutdown_connections 2024-03-06T07:26:54.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.178+0000 7f46359c7640 1 --2- 172.21.15.83:0/1392078210 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46300b2da0 0x7f4630162570 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.178+0000 7f46359c7640 1 -- 172.21.15.83:0/1392078210 >> 172.21.15.83:0/1392078210 conn(0x7f46300d1d50 msgr2=0x7f46300d2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:54.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.178+0000 7f46359c7640 1 -- 172.21.15.83:0/1392078210 shutdown_connections 2024-03-06T07:26:54.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.178+0000 7f46359c7640 1 -- 172.21.15.83:0/1392078210 wait complete. 2024-03-06T07:26:54.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.179+0000 7f46359c7640 1 Processor -- start 2024-03-06T07:26:54.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.179+0000 7f46359c7640 1 -- start start 2024-03-06T07:26:54.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.180+0000 7f46359c7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46300b2da0 0x7f4630148250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.180+0000 7f46359c7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46300cd690 0x7f46300b5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.180+0000 7f46359c7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4630161670 0x7f463015d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.180+0000 7f46359c7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463006d4c0 con 0x7f46300cd690 2024-03-06T07:26:54.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.180+0000 7f46359c7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463006d340 con 0x7f46300b2da0 2024-03-06T07:26:54.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.180+0000 7f46359c7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4630161a50 con 0x7f4630161670 2024-03-06T07:26:54.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.180+0000 7f4627fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46300cd690 0x7f46300b5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.180+0000 7f4627fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46300cd690 0x7f46300b5b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53450/0 (socket says 172.21.15.83:53450) 2024-03-06T07:26:54.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.180+0000 7f4627fff640 1 -- 172.21.15.83:0/1882321992 learned_addr learned my addr 172.21.15.83:0/1882321992 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:54.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.180+0000 7f462f7fe640 1 --2- 172.21.15.83:0/1882321992 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4630161670 0x7f463015d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.180+0000 7f462effd640 1 --2- 172.21.15.83:0/1882321992 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46300b2da0 0x7f4630148250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.180+0000 7f4627fff640 1 -- 172.21.15.83:0/1882321992 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4630161670 msgr2=0x7f463015d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.180+0000 7f4627fff640 1 --2- 172.21.15.83:0/1882321992 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4630161670 0x7f463015d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.180+0000 7f4627fff640 1 -- 172.21.15.83:0/1882321992 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46300b2da0 msgr2=0x7f4630148250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.181+0000 7f4627fff640 1 --2- 172.21.15.83:0/1882321992 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46300b2da0 0x7f4630148250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.181+0000 7f4627fff640 1 -- 172.21.15.83:0/1882321992 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46300b6420 con 0x7f46300cd690 2024-03-06T07:26:54.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.181+0000 7f462effd640 1 --2- 172.21.15.83:0/1882321992 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46300b2da0 0x7f4630148250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:54.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.181+0000 7f4627fff640 1 --2- 172.21.15.83:0/1882321992 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46300cd690 0x7f46300b5b40 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f461800c7b0 tx=0x7f461800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:54.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.181+0000 7f462cff9640 1 -- 172.21.15.83:0/1882321992 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4618017070 con 0x7f46300cd690 2024-03-06T07:26:54.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.181+0000 7f462cff9640 1 -- 172.21.15.83:0/1882321992 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4618005270 con 0x7f46300cd690 2024-03-06T07:26:54.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.181+0000 7f462cff9640 1 -- 172.21.15.83:0/1882321992 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f461800a640 con 0x7f46300cd690 2024-03-06T07:26:54.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.181+0000 7f46359c7640 1 -- 172.21.15.83:0/1882321992 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46300b6700 con 0x7f46300cd690 2024-03-06T07:26:54.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.181+0000 7f46359c7640 1 -- 172.21.15.83:0/1882321992 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46300b6cc0 con 0x7f46300cd690 2024-03-06T07:26:54.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.182+0000 7f46359c7640 1 -- 172.21.15.83:0/1882321992 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4630068050 con 0x7f46300cd690 2024-03-06T07:26:54.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.182+0000 7f462cff9640 1 -- 172.21.15.83:0/1882321992 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f461801a070 con 0x7f46300cd690 2024-03-06T07:26:54.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.183+0000 7f462cff9640 1 --2- 172.21.15.83:0/1882321992 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f461006b4d0 0x7f461006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.183+0000 7f462effd640 1 --2- 172.21.15.83:0/1882321992 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f461006b4d0 0x7f461006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.183+0000 7f462cff9640 1 -- 172.21.15.83:0/1882321992 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f4618089800 con 0x7f46300cd690 2024-03-06T07:26:54.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.183+0000 7f462effd640 1 --2- 172.21.15.83:0/1882321992 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f461006b4d0 0x7f461006d990 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f4620005900 tx=0x7f4620001bc0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:54.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.187+0000 7f462cff9640 1 -- 172.21.15.83:0/1882321992 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4618057f50 con 0x7f46300cd690 2024-03-06T07:26:54.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.380+0000 7f46359c7640 1 -- 172.21.15.83:0/1882321992 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f46300b7cd0 con 0x7f46300cd690 2024-03-06T07:26:54.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.380+0000 7f462cff9640 1 -- 172.21.15.83:0/1882321992 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v98) v1 ==== 100+0+11 (secure 0 0 0) 0x7f461805bff0 con 0x7f46300cd690 2024-03-06T07:26:54.383 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 64 2024-03-06T07:26:54.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.382+0000 7f46359c7640 1 -- 172.21.15.83:0/1882321992 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f461006b4d0 msgr2=0x7f461006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.382+0000 7f46359c7640 1 --2- 172.21.15.83:0/1882321992 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f461006b4d0 0x7f461006d990 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f4620005900 tx=0x7f4620001bc0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.382+0000 7f46359c7640 1 -- 172.21.15.83:0/1882321992 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46300cd690 msgr2=0x7f46300b5b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.382+0000 7f46359c7640 1 --2- 172.21.15.83:0/1882321992 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46300cd690 0x7f46300b5b40 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f461800c7b0 tx=0x7f461800cc70 comp rx=0 tx=0).stop 2024-03-06T07:26:54.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.382+0000 7f46359c7640 1 -- 172.21.15.83:0/1882321992 shutdown_connections 2024-03-06T07:26:54.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.382+0000 7f46359c7640 1 --2- 172.21.15.83:0/1882321992 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f461006b4d0 0x7f461006d990 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.382+0000 7f46359c7640 1 --2- 172.21.15.83:0/1882321992 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4630161670 0x7f463015d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.382+0000 7f46359c7640 1 --2- 172.21.15.83:0/1882321992 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46300cd690 0x7f46300b5b40 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.382+0000 7f46359c7640 1 --2- 172.21.15.83:0/1882321992 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46300b2da0 0x7f4630148250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.382+0000 7f46359c7640 1 -- 172.21.15.83:0/1882321992 >> 172.21.15.83:0/1882321992 conn(0x7f46300d1d50 msgr2=0x7f46300b3720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:54.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.382+0000 7f46359c7640 1 -- 172.21.15.83:0/1882321992 shutdown_connections 2024-03-06T07:26:54.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.382+0000 7f46359c7640 1 -- 172.21.15.83:0/1882321992 wait complete. 2024-03-06T07:26:54.398 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:26:54.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.499+0000 7f8750088640 1 Processor -- start 2024-03-06T07:26:54.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.500+0000 7f8750088640 1 -- start start 2024-03-06T07:26:54.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.500+0000 7f8750088640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f8748161a80 0x7f87480d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.500+0000 7f8750088640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f87480b5010 0x7f87480b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.500+0000 7f8750088640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8748161e60 con 0x7f87480d25a0 2024-03-06T07:26:54.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.500+0000 7f8750088640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87480b3e30 con 0x7f8748161a80 2024-03-06T07:26:54.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.500+0000 7f8750088640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87480d1470 con 0x7f87480b5010 2024-03-06T07:26:54.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.500+0000 7f874e625640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f87480d25a0 0x7f87480b42c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:33442/0 (socket says 172.21.15.83:33442) 2024-03-06T07:26:54.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.501+0000 7f874e625640 1 -- 172.21.15.83:0/1912591702 learned_addr learned my addr 172.21.15.83:0/1912591702 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:54.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.501+0000 7f874d623640 1 --2- 172.21.15.83:0/1912591702 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f87480b5010 0x7f87480b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.501+0000 7f874de24640 1 --2- 172.21.15.83:0/1912591702 >> v2:172.21.15.183:3300/0 conn(0x7f8748161a80 0x7f87480d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.501+0000 7f874ce22640 1 -- 172.21.15.83:0/1912591702 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3418167585 0 0) 0x7f8740001490 con 0x7f87480d25a0 2024-03-06T07:26:54.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.501+0000 7f874ce22640 1 -- 172.21.15.83:0/1912591702 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8734003520 con 0x7f87480d25a0 2024-03-06T07:26:54.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.501+0000 7f874ce22640 1 -- 172.21.15.83:0/1912591702 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1444736565 0 0) 0x7f8734003520 con 0x7f87480d25a0 2024-03-06T07:26:54.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.501+0000 7f874d623640 1 -- 172.21.15.83:0/1912591702 >> v2:172.21.15.183:3300/0 conn(0x7f8748161a80 msgr2=0x7f87480d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:54.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.501+0000 7f874d623640 1 --2- 172.21.15.83:0/1912591702 >> v2:172.21.15.183:3300/0 conn(0x7f8748161a80 0x7f87480d1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.501+0000 7f874d623640 1 -- 172.21.15.83:0/1912591702 >> v1:172.21.15.83:6789/0 conn(0x7f87480d25a0 legacy=0x7f87480b42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.501+0000 7f874d623640 1 -- 172.21.15.83:0/1912591702 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87480d1750 con 0x7f87480b5010 2024-03-06T07:26:54.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.502+0000 7f874d623640 1 --2- 172.21.15.83:0/1912591702 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f87480b5010 0x7f87480b0b40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f87380097a0 tx=0x7f873802b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1072519140c95b15 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:54.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.502+0000 7f874ce22640 1 -- 172.21.15.83:0/1912591702 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8738037070 con 0x7f87480b5010 2024-03-06T07:26:54.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.502+0000 7f874ce22640 1 -- 172.21.15.83:0/1912591702 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8738030510 con 0x7f87480b5010 2024-03-06T07:26:54.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.502+0000 7f874ce22640 1 -- 172.21.15.83:0/1912591702 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8740001bc0 con 0x7f87480b5010 2024-03-06T07:26:54.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.503+0000 7f8750088640 1 -- 172.21.15.83:0/1912591702 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f87480b5010 msgr2=0x7f87480b0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.503+0000 7f8750088640 1 --2- 172.21.15.83:0/1912591702 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f87480b5010 0x7f87480b0b40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f87380097a0 tx=0x7f873802b710 comp rx=0 tx=0).stop 2024-03-06T07:26:54.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.503+0000 7f8750088640 1 -- 172.21.15.83:0/1912591702 shutdown_connections 2024-03-06T07:26:54.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.503+0000 7f8750088640 1 --2- 172.21.15.83:0/1912591702 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f87480b5010 0x7f87480b0b40 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.503+0000 7f8750088640 1 --2- 172.21.15.83:0/1912591702 >> v2:172.21.15.183:3300/0 conn(0x7f8748161a80 0x7f87480d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.503+0000 7f8750088640 1 -- 172.21.15.83:0/1912591702 >> 172.21.15.83:0/1912591702 conn(0x7f87480d3fc0 msgr2=0x7f87480d43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:54.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.503+0000 7f8750088640 1 -- 172.21.15.83:0/1912591702 shutdown_connections 2024-03-06T07:26:54.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.503+0000 7f8750088640 1 -- 172.21.15.83:0/1912591702 wait complete. 2024-03-06T07:26:54.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.504+0000 7f8750088640 1 Processor -- start 2024-03-06T07:26:54.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.504+0000 7f8750088640 1 -- start start 2024-03-06T07:26:54.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.505+0000 7f8750088640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f87480b5010 0x7f8748159620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.505+0000 7f8750088640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87480d25a0 0x7f87480cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.505+0000 7f8750088640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8748161a80 0x7f87480cdbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.505+0000 7f8750088640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87480d1210 con 0x7f87480d25a0 2024-03-06T07:26:54.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.505+0000 7f8750088640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87480d1090 con 0x7f87480b5010 2024-03-06T07:26:54.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.505+0000 7f8750088640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87480b1080 con 0x7f8748161a80 2024-03-06T07:26:54.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.505+0000 7f874d623640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87480d25a0 0x7f87480cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.506+0000 7f874d623640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87480d25a0 0x7f87480cd6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53458/0 (socket says 172.21.15.83:53458) 2024-03-06T07:26:54.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.506+0000 7f874d623640 1 -- 172.21.15.83:0/2805567838 learned_addr learned my addr 172.21.15.83:0/2805567838 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:54.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.506+0000 7f874de24640 1 --2- 172.21.15.83:0/2805567838 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f87480b5010 0x7f8748159620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.506+0000 7f874e625640 1 --2- 172.21.15.83:0/2805567838 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8748161a80 0x7f87480cdbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.506+0000 7f874d623640 1 -- 172.21.15.83:0/2805567838 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8748161a80 msgr2=0x7f87480cdbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.506+0000 7f874d623640 1 --2- 172.21.15.83:0/2805567838 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8748161a80 0x7f87480cdbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.506+0000 7f874d623640 1 -- 172.21.15.83:0/2805567838 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f87480b5010 msgr2=0x7f8748159620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.506+0000 7f874d623640 1 --2- 172.21.15.83:0/2805567838 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f87480b5010 0x7f8748159620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.506+0000 7f874d623640 1 -- 172.21.15.83:0/2805567838 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f874815a6d0 con 0x7f87480d25a0 2024-03-06T07:26:54.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.506+0000 7f874e625640 1 --2- 172.21.15.83:0/2805567838 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8748161a80 0x7f87480cdbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:54.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.506+0000 7f874de24640 1 --2- 172.21.15.83:0/2805567838 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f87480b5010 0x7f8748159620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:54.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.506+0000 7f874d623640 1 --2- 172.21.15.83:0/2805567838 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87480d25a0 0x7f87480cd6a0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f873802b750 tx=0x7f87380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:54.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.506+0000 7f873e7fc640 1 -- 172.21.15.83:0/2805567838 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8738042020 con 0x7f87480d25a0 2024-03-06T07:26:54.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.506+0000 7f873e7fc640 1 -- 172.21.15.83:0/2805567838 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87380044c0 con 0x7f87480d25a0 2024-03-06T07:26:54.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.506+0000 7f8750088640 1 -- 172.21.15.83:0/2805567838 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87480cc0a0 con 0x7f87480d25a0 2024-03-06T07:26:54.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.506+0000 7f873e7fc640 1 -- 172.21.15.83:0/2805567838 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8738037040 con 0x7f87480d25a0 2024-03-06T07:26:54.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.506+0000 7f8750088640 1 -- 172.21.15.83:0/2805567838 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87480cc660 con 0x7f87480d25a0 2024-03-06T07:26:54.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.507+0000 7f8750088640 1 -- 172.21.15.83:0/2805567838 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f874815a8a0 con 0x7f87480d25a0 2024-03-06T07:26:54.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.508+0000 7f873e7fc640 1 -- 172.21.15.83:0/2805567838 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8738045020 con 0x7f87480d25a0 2024-03-06T07:26:54.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.508+0000 7f873e7fc640 1 --2- 172.21.15.83:0/2805567838 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f872c06b4d0 0x7f872c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.508+0000 7f873e7fc640 1 -- 172.21.15.83:0/2805567838 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f87380ac010 con 0x7f87480d25a0 2024-03-06T07:26:54.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.508+0000 7f874de24640 1 --2- 172.21.15.83:0/2805567838 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f872c06b4d0 0x7f872c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.509+0000 7f874de24640 1 --2- 172.21.15.83:0/2805567838 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f872c06b4d0 0x7f872c06d990 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f8744009a20 tx=0x7f8744008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:54.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.513+0000 7f873e7fc640 1 -- 172.21.15.83:0/2805567838 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f873804c030 con 0x7f87480d25a0 2024-03-06T07:26:54.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.703+0000 7f8750088640 1 -- 172.21.15.83:0/2805567838 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f8748068050 con 0x7f87480d25a0 2024-03-06T07:26:54.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.703+0000 7f873e7fc640 1 -- 172.21.15.83:0/2805567838 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v98) v1 ==== 80+0+122 (secure 0 0 0) 0x7f873807ada0 con 0x7f87480d25a0 2024-03-06T07:26:54.705 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6837/2924411033 2024-03-07T07:26:27.785046+0000 2024-03-06T07:26:54.706 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6835/2924411033 2024-03-07T07:26:27.785046+0000 2024-03-06T07:26:54.706 INFO:teuthology.orchestra.run.smithi083.stderr:listed 2 entries 2024-03-06T07:26:54.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.705+0000 7f8750088640 1 -- 172.21.15.83:0/2805567838 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f872c06b4d0 msgr2=0x7f872c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.705+0000 7f8750088640 1 --2- 172.21.15.83:0/2805567838 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f872c06b4d0 0x7f872c06d990 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f8744009a20 tx=0x7f8744008040 comp rx=0 tx=0).stop 2024-03-06T07:26:54.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.705+0000 7f8750088640 1 -- 172.21.15.83:0/2805567838 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87480d25a0 msgr2=0x7f87480cd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.705+0000 7f8750088640 1 --2- 172.21.15.83:0/2805567838 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87480d25a0 0x7f87480cd6a0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f873802b750 tx=0x7f87380046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.705+0000 7f8750088640 1 -- 172.21.15.83:0/2805567838 shutdown_connections 2024-03-06T07:26:54.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.705+0000 7f8750088640 1 --2- 172.21.15.83:0/2805567838 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f872c06b4d0 0x7f872c06d990 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.705+0000 7f8750088640 1 --2- 172.21.15.83:0/2805567838 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8748161a80 0x7f87480cdbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.705+0000 7f8750088640 1 --2- 172.21.15.83:0/2805567838 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87480d25a0 0x7f87480cd6a0 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.705+0000 7f8750088640 1 --2- 172.21.15.83:0/2805567838 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f87480b5010 0x7f8748159620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.705+0000 7f8750088640 1 -- 172.21.15.83:0/2805567838 >> 172.21.15.83:0/2805567838 conn(0x7f87480d3fc0 msgr2=0x7f87480d29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:54.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.706+0000 7f8750088640 1 -- 172.21.15.83:0/2805567838 shutdown_connections 2024-03-06T07:26:54.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:54.706+0000 7f8750088640 1 -- 172.21.15.83:0/2805567838 wait complete. 2024-03-06T07:26:54.718 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi083.front.sepia.ceph.com... 2024-03-06T07:26:54.718 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:54.718 DEBUG:teuthology.orchestra.run.smithi083:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:54.743 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:ceph-fuse[68926]: fuse finished with error 0 and tester_r 0 2024-03-06T07:26:54.745 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi083.front.sepia.ceph.com 2024-03-06T07:26:54.745 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:54.745 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:26:54.914 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:26:54.915 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:54.931 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=287 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:26:54.931 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi183.front.sepia.ceph.com 2024-03-06T07:26:54.931 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:54.931 DEBUG:teuthology.orchestra.run.smithi183:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:26:54.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:26:54.987 INFO:teuthology.orchestra.run.smithi183.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:26:54.987 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:26:54.987 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:55.003 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1257 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:26:55.004 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi083.front.sepia.ceph.com 2024-03-06T07:26:55.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:55.004 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:26:55.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:26:55.058 INFO:teuthology.orchestra.run.smithi083.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:26:55.058 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:26:55.058 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:55.074 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=287 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:26:55.075 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi183.front.sepia.ceph.com 2024-03-06T07:26:55.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:55.075 DEBUG:teuthology.orchestra.run.smithi183:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:26:55.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:26:55.130 INFO:teuthology.orchestra.run.smithi183.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:26:55.130 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:55.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.228+0000 7f29dae65640 1 Processor -- start 2024-03-06T07:26:55.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.228+0000 7f29dae65640 1 -- start start 2024-03-06T07:26:55.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.229+0000 7f29dae65640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f29d4161670 0x7f29d4161e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.229+0000 7f29dae65640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f29d40b2da0 0x7f29d41623a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.229+0000 7f29dae65640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29d4162c70 con 0x7f29d40cc680 2024-03-06T07:26:55.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.229+0000 7f29dae65640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29d40b3590 con 0x7f29d40b2da0 2024-03-06T07:26:55.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.229+0000 7f29dae65640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29d406cd80 con 0x7f29d4161670 2024-03-06T07:26:55.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.229+0000 7f29cbfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f29d40b2da0 0x7f29d41623a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.229+0000 7f29cbfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f29d40b2da0 0x7f29d41623a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53464/0 (socket says 172.21.15.83:53464) 2024-03-06T07:26:55.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.229+0000 7f29cbfff640 1 -- 172.21.15.83:0/166825965 learned_addr learned my addr 172.21.15.83:0/166825965 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:55.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.229+0000 7f29d8c01640 1 --2- 172.21.15.83:0/166825965 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f29d4161670 0x7f29d4161e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.230+0000 7f29cbfff640 1 -- 172.21.15.83:0/166825965 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f29d4161670 msgr2=0x7f29d4161e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:55.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.230+0000 7f29cbfff640 1 --2- 172.21.15.83:0/166825965 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f29d4161670 0x7f29d4161e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.230+0000 7f29cbfff640 1 -- 172.21.15.83:0/166825965 >> v1:172.21.15.83:6789/0 conn(0x7f29d40cc680 legacy=0x7f29d40ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.230+0000 7f29cb7fe640 1 -- 172.21.15.83:0/166825965 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2080469687 0 0) 0x7f29d4162c70 con 0x7f29d40cc680 2024-03-06T07:26:55.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.230+0000 7f29cbfff640 1 -- 172.21.15.83:0/166825965 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d406d060 con 0x7f29d40b2da0 2024-03-06T07:26:55.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.230+0000 7f29cbfff640 1 --2- 172.21.15.83:0/166825965 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29d40b2da0 0x7f29d41623a0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f29c4009870 tx=0x7f29c402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e47dfdc7854c2eae server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.230+0000 7f29cb7fe640 1 -- 172.21.15.83:0/166825965 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c4034070 con 0x7f29d40b2da0 2024-03-06T07:26:55.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.230+0000 7f29cb7fe640 1 -- 172.21.15.83:0/166825965 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29c4030840 con 0x7f29d40b2da0 2024-03-06T07:26:55.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.231+0000 7f29cb7fe640 1 -- 172.21.15.83:0/166825965 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c4030c10 con 0x7f29d40b2da0 2024-03-06T07:26:55.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.231+0000 7f29dae65640 1 -- 172.21.15.83:0/166825965 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29d40b2da0 msgr2=0x7f29d41623a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.231+0000 7f29dae65640 1 --2- 172.21.15.83:0/166825965 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29d40b2da0 0x7f29d41623a0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f29c4009870 tx=0x7f29c402b980 comp rx=0 tx=0).stop 2024-03-06T07:26:55.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.231+0000 7f29dae65640 1 -- 172.21.15.83:0/166825965 shutdown_connections 2024-03-06T07:26:55.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.231+0000 7f29dae65640 1 --2- 172.21.15.83:0/166825965 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29d40b2da0 0x7f29d41623a0 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.231+0000 7f29dae65640 1 --2- 172.21.15.83:0/166825965 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f29d4161670 0x7f29d4161e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.231+0000 7f29dae65640 1 -- 172.21.15.83:0/166825965 >> 172.21.15.83:0/166825965 conn(0x7f29d40d1d50 msgr2=0x7f29d40d2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:55.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.231+0000 7f29dae65640 1 -- 172.21.15.83:0/166825965 shutdown_connections 2024-03-06T07:26:55.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.231+0000 7f29dae65640 1 -- 172.21.15.83:0/166825965 wait complete. 2024-03-06T07:26:55.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.233+0000 7f29dae65640 1 Processor -- start 2024-03-06T07:26:55.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.233+0000 7f29dae65640 1 -- start start 2024-03-06T07:26:55.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.235+0000 7f29dae65640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f29d40b2da0 0x7f29d40c7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.235+0000 7f29dae65640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29d40cc680 0x7f29d40b73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.235+0000 7f29dae65640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f29d4161670 0x7f29d4150dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.235+0000 7f29dae65640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29d406d340 con 0x7f29d40cc680 2024-03-06T07:26:55.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.235+0000 7f29dae65640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29d41628e0 con 0x7f29d4161670 2024-03-06T07:26:55.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.235+0000 7f29dae65640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29d406d4c0 con 0x7f29d40b2da0 2024-03-06T07:26:55.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.235+0000 7f29cbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29d40cc680 0x7f29d40b73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.235+0000 7f29cbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29d40cc680 0x7f29d40b73b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53470/0 (socket says 172.21.15.83:53470) 2024-03-06T07:26:55.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.235+0000 7f29cbfff640 1 -- 172.21.15.83:0/978925754 learned_addr learned my addr 172.21.15.83:0/978925754 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:55.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.235+0000 7f29d8c01640 1 --2- 172.21.15.83:0/978925754 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f29d40b2da0 0x7f29d40c7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.235+0000 7f29d9402640 1 --2- 172.21.15.83:0/978925754 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f29d4161670 0x7f29d4150dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.235+0000 7f29cbfff640 1 -- 172.21.15.83:0/978925754 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f29d40b2da0 msgr2=0x7f29d40c7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.235+0000 7f29cbfff640 1 --2- 172.21.15.83:0/978925754 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f29d40b2da0 0x7f29d40c7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.235+0000 7f29cbfff640 1 -- 172.21.15.83:0/978925754 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f29d4161670 msgr2=0x7f29d4150dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.235+0000 7f29cbfff640 1 --2- 172.21.15.83:0/978925754 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f29d4161670 0x7f29d4150dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.235+0000 7f29cbfff640 1 -- 172.21.15.83:0/978925754 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d40c65d0 con 0x7f29d40cc680 2024-03-06T07:26:55.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.235+0000 7f29cbfff640 1 --2- 172.21.15.83:0/978925754 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29d40cc680 0x7f29d40b73b0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f29c40090b0 tx=0x7f29c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.235+0000 7f29c97fa640 1 -- 172.21.15.83:0/978925754 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c4034040 con 0x7f29d40cc680 2024-03-06T07:26:55.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.235+0000 7f29c97fa640 1 -- 172.21.15.83:0/978925754 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29c4030640 con 0x7f29d40cc680 2024-03-06T07:26:55.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.235+0000 7f29c97fa640 1 -- 172.21.15.83:0/978925754 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c4039d20 con 0x7f29d40cc680 2024-03-06T07:26:55.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.235+0000 7f29dae65640 1 -- 172.21.15.83:0/978925754 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29d40c68b0 con 0x7f29d40cc680 2024-03-06T07:26:55.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.236+0000 7f29dae65640 1 -- 172.21.15.83:0/978925754 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29d40b4cd0 con 0x7f29d40cc680 2024-03-06T07:26:55.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.237+0000 7f29dae65640 1 -- 172.21.15.83:0/978925754 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29d4068050 con 0x7f29d40cc680 2024-03-06T07:26:55.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.237+0000 7f29c97fa640 1 -- 172.21.15.83:0/978925754 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f29c4037070 con 0x7f29d40cc680 2024-03-06T07:26:55.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.237+0000 7f29c97fa640 1 --2- 172.21.15.83:0/978925754 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f29c006b4d0 0x7f29c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.237+0000 7f29c97fa640 1 -- 172.21.15.83:0/978925754 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f29c40abfd0 con 0x7f29d40cc680 2024-03-06T07:26:55.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.237+0000 7f29d8c01640 1 --2- 172.21.15.83:0/978925754 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f29c006b4d0 0x7f29c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.238+0000 7f29d8c01640 1 --2- 172.21.15.83:0/978925754 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f29c006b4d0 0x7f29c006d990 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f29d0006fd0 tx=0x7f29d0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.242+0000 7f29c97fa640 1 -- 172.21.15.83:0/978925754 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f29c407af40 con 0x7f29d40cc680 2024-03-06T07:26:55.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.457+0000 7f29dae65640 1 -- 172.21.15.83:0/978925754 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29d406b350 con 0x7f29d40cc680 2024-03-06T07:26:55.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.457+0000 7f29c97fa640 1 -- 172.21.15.83:0/978925754 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f29c4030810 con 0x7f29d40cc680 2024-03-06T07:26:55.459 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:55.459 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":53,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6023,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/601236077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":601236077},{"type":"v1","addr":"172.21.15.183:6833","nonce":601236077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6029,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1435055961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1435055961},{"type":"v1","addr":"172.21.15.183:6835","nonce":1435055961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":7572,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/2726045966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2726045966},{"type":"v1","addr":"172.21.15.83:6836","nonce":2726045966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.064335+0000","modified":"2024-03-06T07:26:32.814889+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7575":{"gid":7575,"name":"a","rank":0,"incarnation":49,"state":"up:active","state_seq":5,"addr":"172.21.15.83:6837/3254575964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":3254575964},{"type":"v1","addr":"172.21.15.83:6837","nonce":3254575964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7575,"qdb_cluster":[7575]},"id":7}]} 2024-03-06T07:26:55.459 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 53 2024-03-06T07:26:55.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.459+0000 7f29dae65640 1 -- 172.21.15.83:0/978925754 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f29c006b4d0 msgr2=0x7f29c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.459+0000 7f29dae65640 1 --2- 172.21.15.83:0/978925754 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f29c006b4d0 0x7f29c006d990 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f29d0006fd0 tx=0x7f29d0008040 comp rx=0 tx=0).stop 2024-03-06T07:26:55.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.459+0000 7f29dae65640 1 -- 172.21.15.83:0/978925754 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29d40cc680 msgr2=0x7f29d40b73b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.459+0000 7f29dae65640 1 --2- 172.21.15.83:0/978925754 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29d40cc680 0x7f29d40b73b0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f29c40090b0 tx=0x7f29c40046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.459+0000 7f29dae65640 1 -- 172.21.15.83:0/978925754 shutdown_connections 2024-03-06T07:26:55.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.459+0000 7f29dae65640 1 --2- 172.21.15.83:0/978925754 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f29c006b4d0 0x7f29c006d990 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.459+0000 7f29dae65640 1 --2- 172.21.15.83:0/978925754 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f29d4161670 0x7f29d4150dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.459+0000 7f29dae65640 1 --2- 172.21.15.83:0/978925754 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29d40cc680 0x7f29d40b73b0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.459+0000 7f29dae65640 1 --2- 172.21.15.83:0/978925754 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f29d40b2da0 0x7f29d40c7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.459+0000 7f29dae65640 1 -- 172.21.15.83:0/978925754 >> 172.21.15.83:0/978925754 conn(0x7f29d40d1d50 msgr2=0x7f29d40702b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:55.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.459+0000 7f29dae65640 1 -- 172.21.15.83:0/978925754 shutdown_connections 2024-03-06T07:26:55.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.459+0000 7f29dae65640 1 -- 172.21.15.83:0/978925754 wait complete. 2024-03-06T07:26:55.472 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:55.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.571+0000 7fcf9d0b5640 1 Processor -- start 2024-03-06T07:26:55.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.571+0000 7fcf9d0b5640 1 -- start start 2024-03-06T07:26:55.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.572+0000 7fcf9d0b5640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcf9815e150 0x7fcf980cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.572+0000 7fcf9d0b5640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf980d1ea0 con 0x7fcf980d1060 2024-03-06T07:26:55.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.572+0000 7fcf9d0b5640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf980d2180 con 0x7fcf980cfba0 2024-03-06T07:26:55.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.572+0000 7fcf9d0b5640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf980d2ab0 con 0x7fcf9815e150 2024-03-06T07:26:55.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.572+0000 7fcf9759e640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fcf980d1060 0x7fcf980b9060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:33460/0 (socket says 172.21.15.83:33460) 2024-03-06T07:26:55.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.572+0000 7fcf9759e640 1 -- 172.21.15.83:0/1285666548 learned_addr learned my addr 172.21.15.83:0/1285666548 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:55.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.572+0000 7fcf95d9b640 1 -- 172.21.15.83:0/1285666548 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 317121090 0 0) 0x7fcf980d1ea0 con 0x7fcf980d1060 2024-03-06T07:26:55.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.572+0000 7fcf95d9b640 1 -- 172.21.15.83:0/1285666548 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf80003520 con 0x7fcf980d1060 2024-03-06T07:26:55.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.572+0000 7fcf96d9d640 1 --2- 172.21.15.83:0/1285666548 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcf9815e150 0x7fcf980cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.572+0000 7fcf95d9b640 1 -- 172.21.15.83:0/1285666548 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1611466207 0 0) 0x7fcf80003520 con 0x7fcf980d1060 2024-03-06T07:26:55.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.573+0000 7fcf95d9b640 1 -- 172.21.15.83:0/1285666548 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcf9815e150 msgr2=0x7fcf980cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:55.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.573+0000 7fcf95d9b640 1 --2- 172.21.15.83:0/1285666548 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcf9815e150 0x7fcf980cf660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.573+0000 7fcf95d9b640 1 -- 172.21.15.83:0/1285666548 >> v1:172.21.15.183:6789/0 conn(0x7fcf980cfba0 legacy=0x7fcf980d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.573+0000 7fcf95d9b640 1 -- 172.21.15.83:0/1285666548 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf980d2460 con 0x7fcf980d1060 2024-03-06T07:26:55.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.573+0000 7fcf95d9b640 1 -- 172.21.15.83:0/1285666548 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7fcf900019f0 con 0x7fcf980d1060 2024-03-06T07:26:55.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.573+0000 7fcf95d9b640 1 -- 172.21.15.83:0/1285666548 >> v1:172.21.15.83:6789/0 conn(0x7fcf980d1060 legacy=0x7fcf980b9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.573+0000 7fcf95d9b640 1 --2- 172.21.15.83:0/1285666548 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcf800039e0 0x7fcf80003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.573+0000 7fcf95d9b640 1 --2- 172.21.15.83:0/1285666548 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcf9815e150 0x7fcf80004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.573+0000 7fcf95d9b640 1 --2- 172.21.15.83:0/1285666548 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf80004d40 0x7fcf800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.573+0000 7fcf95d9b640 1 -- 172.21.15.83:0/1285666548 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf980d2ab0 con 0x7fcf80004d40 2024-03-06T07:26:55.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.573+0000 7fcf95d9b640 1 -- 172.21.15.83:0/1285666548 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8000baf0 con 0x7fcf800039e0 2024-03-06T07:26:55.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.573+0000 7fcf95d9b640 1 -- 172.21.15.83:0/1285666548 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8000bcf0 con 0x7fcf9815e150 2024-03-06T07:26:55.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.573+0000 7fcf9759e640 1 --2- 172.21.15.83:0/1285666548 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf80004d40 0x7fcf800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.573+0000 7fcf96d9d640 1 --2- 172.21.15.83:0/1285666548 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcf800039e0 0x7fcf80003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.573+0000 7fcf9659c640 1 --2- 172.21.15.83:0/1285666548 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcf9815e150 0x7fcf80004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.573+0000 7fcf9759e640 1 -- 172.21.15.83:0/1285666548 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcf9815e150 msgr2=0x7fcf80004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:55.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.573+0000 7fcf9759e640 1 --2- 172.21.15.83:0/1285666548 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcf9815e150 0x7fcf80004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.573+0000 7fcf9759e640 1 -- 172.21.15.83:0/1285666548 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcf800039e0 msgr2=0x7fcf80003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:55.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.573+0000 7fcf9759e640 1 --2- 172.21.15.83:0/1285666548 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcf800039e0 0x7fcf80003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.573+0000 7fcf9759e640 1 -- 172.21.15.83:0/1285666548 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf80003520 con 0x7fcf80004d40 2024-03-06T07:26:55.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.574+0000 7fcf9759e640 1 --2- 172.21.15.83:0/1285666548 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf80004d40 0x7fcf800051c0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7fcf9000a830 tx=0x7fcf80005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaab609507f9eceb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.574+0000 7fcf95d9b640 1 -- 172.21.15.83:0/1285666548 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9000a480 con 0x7fcf80004d40 2024-03-06T07:26:55.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.574+0000 7fcf95d9b640 1 -- 172.21.15.83:0/1285666548 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf90034650 con 0x7fcf80004d40 2024-03-06T07:26:55.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.574+0000 7fcf95d9b640 1 -- 172.21.15.83:0/1285666548 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf90034a40 con 0x7fcf80004d40 2024-03-06T07:26:55.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.575+0000 7fcf9d0b5640 1 -- 172.21.15.83:0/1285666548 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf80004d40 msgr2=0x7fcf800051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.575+0000 7fcf9d0b5640 1 --2- 172.21.15.83:0/1285666548 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf80004d40 0x7fcf800051c0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7fcf9000a830 tx=0x7fcf80005830 comp rx=0 tx=0).stop 2024-03-06T07:26:55.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.575+0000 7fcf9759e640 1 -- 172.21.15.83:0/1285666548 reap_dead start 2024-03-06T07:26:55.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.575+0000 7fcf9d0b5640 1 -- 172.21.15.83:0/1285666548 shutdown_connections 2024-03-06T07:26:55.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.575+0000 7fcf9d0b5640 1 -- 172.21.15.83:0/1285666548 >> 172.21.15.83:0/1285666548 conn(0x7fcf9809d230 msgr2=0x7fcf980d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:55.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.575+0000 7fcf9d0b5640 1 -- 172.21.15.83:0/1285666548 shutdown_connections 2024-03-06T07:26:55.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.575+0000 7fcf9d0b5640 1 -- 172.21.15.83:0/1285666548 wait complete. 2024-03-06T07:26:55.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.576+0000 7fcf9d0b5640 1 Processor -- start 2024-03-06T07:26:55.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.576+0000 7fcf9d0b5640 1 -- start start 2024-03-06T07:26:55.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.577+0000 7fcf9d0b5640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcf9809df10 0x7fcf980cbb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.577+0000 7fcf9d0b5640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcf980cc090 0x7fcf9815c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.577+0000 7fcf9d0b5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf9809e2f0 0x7fcf9815ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.577+0000 7fcf9d0b5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf980b7dc0 con 0x7fcf9809e2f0 2024-03-06T07:26:55.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.577+0000 7fcf9d0b5640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf980d2f40 con 0x7fcf980cc090 2024-03-06T07:26:55.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.577+0000 7fcf9d0b5640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf980d1b30 con 0x7fcf9809df10 2024-03-06T07:26:55.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.577+0000 7fcf9759e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf9809e2f0 0x7fcf9815ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.577+0000 7fcf9759e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf9809e2f0 0x7fcf9815ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53488/0 (socket says 172.21.15.83:53488) 2024-03-06T07:26:55.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.577+0000 7fcf9759e640 1 -- 172.21.15.83:0/3657775016 learned_addr learned my addr 172.21.15.83:0/3657775016 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:55.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.577+0000 7fcf96d9d640 1 --2- 172.21.15.83:0/3657775016 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcf9809df10 0x7fcf980cbb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.577+0000 7fcf9659c640 1 --2- 172.21.15.83:0/3657775016 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcf980cc090 0x7fcf9815c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.577+0000 7fcf9759e640 1 -- 172.21.15.83:0/3657775016 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcf9809df10 msgr2=0x7fcf980cbb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.578+0000 7fcf9759e640 1 --2- 172.21.15.83:0/3657775016 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcf9809df10 0x7fcf980cbb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.578+0000 7fcf9759e640 1 -- 172.21.15.83:0/3657775016 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcf980cc090 msgr2=0x7fcf9815c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.578+0000 7fcf9759e640 1 --2- 172.21.15.83:0/3657775016 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcf980cc090 0x7fcf9815c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.578+0000 7fcf9759e640 1 -- 172.21.15.83:0/3657775016 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf980ca180 con 0x7fcf9809e2f0 2024-03-06T07:26:55.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.578+0000 7fcf96d9d640 1 --2- 172.21.15.83:0/3657775016 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcf9809df10 0x7fcf980cbb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:55.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.578+0000 7fcf9659c640 1 --2- 172.21.15.83:0/3657775016 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcf980cc090 0x7fcf9815c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:55.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.578+0000 7fcf9759e640 1 --2- 172.21.15.83:0/3657775016 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf9809e2f0 0x7fcf9815ca70 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fcf9000a830 tx=0x7fcf980d2a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.578+0000 7fcf7f7fe640 1 -- 172.21.15.83:0/3657775016 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf90045070 con 0x7fcf9809e2f0 2024-03-06T07:26:55.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.578+0000 7fcf7f7fe640 1 -- 172.21.15.83:0/3657775016 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf9003a040 con 0x7fcf9809e2f0 2024-03-06T07:26:55.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.578+0000 7fcf9d0b5640 1 -- 172.21.15.83:0/3657775016 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf980c9600 con 0x7fcf9809e2f0 2024-03-06T07:26:55.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.578+0000 7fcf7f7fe640 1 -- 172.21.15.83:0/3657775016 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf90001ff0 con 0x7fcf9809e2f0 2024-03-06T07:26:55.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.578+0000 7fcf9d0b5640 1 -- 172.21.15.83:0/3657775016 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf980c9bc0 con 0x7fcf9809e2f0 2024-03-06T07:26:55.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.579+0000 7fcf9d0b5640 1 -- 172.21.15.83:0/3657775016 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf98069ee0 con 0x7fcf9809e2f0 2024-03-06T07:26:55.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.579+0000 7fcf7f7fe640 1 -- 172.21.15.83:0/3657775016 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcf90048070 con 0x7fcf9809e2f0 2024-03-06T07:26:55.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.580+0000 7fcf7f7fe640 1 --2- 172.21.15.83:0/3657775016 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcf8406b4d0 0x7fcf8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.580+0000 7fcf96d9d640 1 --2- 172.21.15.83:0/3657775016 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcf8406b4d0 0x7fcf8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.580+0000 7fcf7f7fe640 1 -- 172.21.15.83:0/3657775016 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fcf900abd90 con 0x7fcf9809e2f0 2024-03-06T07:26:55.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.580+0000 7fcf96d9d640 1 --2- 172.21.15.83:0/3657775016 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcf8406b4d0 0x7fcf8406d990 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fcf980ca870 tx=0x7fcf8c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.584+0000 7fcf7f7fe640 1 -- 172.21.15.83:0/3657775016 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcf9007a560 con 0x7fcf9809e2f0 2024-03-06T07:26:55.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.762+0000 7fcf9d0b5640 1 -- 172.21.15.83:0/3657775016 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf980cadf0 con 0x7fcf9809e2f0 2024-03-06T07:26:55.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.763+0000 7fcf7f7fe640 1 -- 172.21.15.83:0/3657775016 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+16749 (secure 0 0 0) 0x7fcf9007e600 con 0x7fcf9809e2f0 2024-03-06T07:26:55.765 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:55.765 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":98,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:26:27.785066+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:17.275618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:18.028872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.83:6837/2924411033":"2024-03-07T07:26:27.785046+0000","172.21.15.83:6835/2924411033":"2024-03-07T07:26:27.785046+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:55.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.765+0000 7fcf9d0b5640 1 -- 172.21.15.83:0/3657775016 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcf8406b4d0 msgr2=0x7fcf8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.765+0000 7fcf9d0b5640 1 --2- 172.21.15.83:0/3657775016 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcf8406b4d0 0x7fcf8406d990 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fcf980ca870 tx=0x7fcf8c008040 comp rx=0 tx=0).stop 2024-03-06T07:26:55.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.765+0000 7fcf9d0b5640 1 -- 172.21.15.83:0/3657775016 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf9809e2f0 msgr2=0x7fcf9815ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.765+0000 7fcf9d0b5640 1 --2- 172.21.15.83:0/3657775016 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf9809e2f0 0x7fcf9815ca70 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fcf9000a830 tx=0x7fcf980d2a20 comp rx=0 tx=0).stop 2024-03-06T07:26:55.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.765+0000 7fcf9d0b5640 1 -- 172.21.15.83:0/3657775016 shutdown_connections 2024-03-06T07:26:55.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.765+0000 7fcf9d0b5640 1 --2- 172.21.15.83:0/3657775016 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcf8406b4d0 0x7fcf8406d990 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.765+0000 7fcf9d0b5640 1 --2- 172.21.15.83:0/3657775016 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf9809e2f0 0x7fcf9815ca70 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.765+0000 7fcf9d0b5640 1 --2- 172.21.15.83:0/3657775016 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcf980cc090 0x7fcf9815c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.765+0000 7fcf9d0b5640 1 --2- 172.21.15.83:0/3657775016 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcf9809df10 0x7fcf980cbb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.765+0000 7fcf9d0b5640 1 -- 172.21.15.83:0/3657775016 >> 172.21.15.83:0/3657775016 conn(0x7fcf9809d230 msgr2=0x7fcf980b5df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:55.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.765+0000 7fcf9d0b5640 1 -- 172.21.15.83:0/3657775016 shutdown_connections 2024-03-06T07:26:55.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.765+0000 7fcf9d0b5640 1 -- 172.21.15.83:0/3657775016 wait complete. 2024-03-06T07:26:55.778 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:26:17.275618+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '93', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:26:18.028872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '94', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:26:55.779 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:26:55.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.885+0000 7fe73c1bc640 1 Processor -- start 2024-03-06T07:26:55.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.885+0000 7fe73c1bc640 1 -- start start 2024-03-06T07:26:55.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.886+0000 7fe73c1bc640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fe734158690 0x7fe7340b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.886+0000 7fe73c1bc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe7340b40d0 0x7fe7340afa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.886+0000 7fe73c1bc640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7340b5e20 con 0x7fe7340affa0 2024-03-06T07:26:55.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.886+0000 7fe73c1bc640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe73415f4d0 con 0x7fe7340b40d0 2024-03-06T07:26:55.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.886+0000 7fe73c1bc640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe73406ddc0 con 0x7fe734158690 2024-03-06T07:26:55.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.886+0000 7fe739757640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe7340b40d0 0x7fe7340afa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.886+0000 7fe739757640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe7340b40d0 0x7fe7340afa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53502/0 (socket says 172.21.15.83:53502) 2024-03-06T07:26:55.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.886+0000 7fe739757640 1 -- 172.21.15.83:0/3226753305 learned_addr learned my addr 172.21.15.83:0/3226753305 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:55.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.886+0000 7fe739f58640 1 --2- 172.21.15.83:0/3226753305 >> v2:172.21.15.183:3300/0 conn(0x7fe734158690 0x7fe7340b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.886+0000 7fe739757640 1 -- 172.21.15.83:0/3226753305 >> v2:172.21.15.183:3300/0 conn(0x7fe734158690 msgr2=0x7fe7340b3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:55.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.886+0000 7fe739757640 1 --2- 172.21.15.83:0/3226753305 >> v2:172.21.15.183:3300/0 conn(0x7fe734158690 0x7fe7340b3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.886+0000 7fe739757640 1 -- 172.21.15.83:0/3226753305 >> v1:172.21.15.183:6789/0 conn(0x7fe7340affa0 legacy=0x7fe734161b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:55.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.886+0000 7fe739757640 1 -- 172.21.15.83:0/3226753305 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe73415e710 con 0x7fe7340b40d0 2024-03-06T07:26:55.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.886+0000 7fe739f58640 1 --2- 172.21.15.83:0/3226753305 >> v2:172.21.15.183:3300/0 conn(0x7fe734158690 0x7fe7340b3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:55.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.887+0000 7fe739757640 1 --2- 172.21.15.83:0/3226753305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7340b40d0 0x7fe7340afa60 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7fe73406be40 tx=0x7fe72802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10265225c71d8334 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.887+0000 7fe738f56640 1 -- 172.21.15.83:0/3226753305 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe728035020 con 0x7fe7340b40d0 2024-03-06T07:26:55.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.887+0000 7fe738f56640 1 -- 172.21.15.83:0/3226753305 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe728031870 con 0x7fe7340b40d0 2024-03-06T07:26:55.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.887+0000 7fe738f56640 1 -- 172.21.15.83:0/3226753305 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe728031c40 con 0x7fe7340b40d0 2024-03-06T07:26:55.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.887+0000 7fe73c1bc640 1 -- 172.21.15.83:0/3226753305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7340b40d0 msgr2=0x7fe7340afa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.887+0000 7fe73c1bc640 1 --2- 172.21.15.83:0/3226753305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7340b40d0 0x7fe7340afa60 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7fe73406be40 tx=0x7fe72802b910 comp rx=0 tx=0).stop 2024-03-06T07:26:55.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.887+0000 7fe73c1bc640 1 -- 172.21.15.83:0/3226753305 shutdown_connections 2024-03-06T07:26:55.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.887+0000 7fe73c1bc640 1 --2- 172.21.15.83:0/3226753305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7340b40d0 0x7fe7340afa60 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.887+0000 7fe73c1bc640 1 --2- 172.21.15.83:0/3226753305 >> v2:172.21.15.183:3300/0 conn(0x7fe734158690 0x7fe7340b3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.887+0000 7fe73c1bc640 1 -- 172.21.15.83:0/3226753305 >> 172.21.15.83:0/3226753305 conn(0x7fe7340b2890 msgr2=0x7fe7340b2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:55.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.888+0000 7fe73c1bc640 1 -- 172.21.15.83:0/3226753305 shutdown_connections 2024-03-06T07:26:55.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.888+0000 7fe73c1bc640 1 -- 172.21.15.83:0/3226753305 wait complete. 2024-03-06T07:26:55.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.889+0000 7fe73c1bc640 1 Processor -- start 2024-03-06T07:26:55.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.889+0000 7fe73c1bc640 1 -- start start 2024-03-06T07:26:55.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.889+0000 7fe73c1bc640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe7340affa0 0x7fe7341e8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.889+0000 7fe73c1bc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7340b40d0 0x7fe73414a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.889+0000 7fe73c1bc640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe734158690 0x7fe7341e8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.890+0000 7fe73c1bc640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe73415f040 con 0x7fe7340b40d0 2024-03-06T07:26:55.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.890+0000 7fe73c1bc640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe73415ecd0 con 0x7fe734158690 2024-03-06T07:26:55.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.890+0000 7fe73c1bc640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7341e9870 con 0x7fe7340affa0 2024-03-06T07:26:55.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.890+0000 7fe739757640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7340b40d0 0x7fe73414a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.890+0000 7fe739757640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7340b40d0 0x7fe73414a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53506/0 (socket says 172.21.15.83:53506) 2024-03-06T07:26:55.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.890+0000 7fe739757640 1 -- 172.21.15.83:0/107164175 learned_addr learned my addr 172.21.15.83:0/107164175 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:55.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.890+0000 7fe739f58640 1 --2- 172.21.15.83:0/107164175 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe7340affa0 0x7fe7341e8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.890+0000 7fe73a759640 1 --2- 172.21.15.83:0/107164175 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe734158690 0x7fe7341e8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.890+0000 7fe739757640 1 -- 172.21.15.83:0/107164175 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe7340affa0 msgr2=0x7fe7341e8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.890+0000 7fe739757640 1 --2- 172.21.15.83:0/107164175 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe7340affa0 0x7fe7341e8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.890+0000 7fe739757640 1 -- 172.21.15.83:0/107164175 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe734158690 msgr2=0x7fe7341e8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.890+0000 7fe739757640 1 --2- 172.21.15.83:0/107164175 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe734158690 0x7fe7341e8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.890+0000 7fe739757640 1 -- 172.21.15.83:0/107164175 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7341e7b20 con 0x7fe7340b40d0 2024-03-06T07:26:55.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.890+0000 7fe73a759640 1 --2- 172.21.15.83:0/107164175 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe734158690 0x7fe7341e8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:55.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.890+0000 7fe739757640 1 --2- 172.21.15.83:0/107164175 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7340b40d0 0x7fe73414a290 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fe728000c00 tx=0x7fe7280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.890+0000 7fe7267fc640 1 -- 172.21.15.83:0/107164175 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe728035040 con 0x7fe7340b40d0 2024-03-06T07:26:55.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.890+0000 7fe7267fc640 1 -- 172.21.15.83:0/107164175 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe728045480 con 0x7fe7340b40d0 2024-03-06T07:26:55.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.890+0000 7fe73c1bc640 1 -- 172.21.15.83:0/107164175 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7341e7e00 con 0x7fe7340b40d0 2024-03-06T07:26:55.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.890+0000 7fe7267fc640 1 -- 172.21.15.83:0/107164175 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe728045870 con 0x7fe7340b40d0 2024-03-06T07:26:55.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.890+0000 7fe73c1bc640 1 -- 172.21.15.83:0/107164175 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7341e94b0 con 0x7fe7340b40d0 2024-03-06T07:26:55.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.891+0000 7fe73c1bc640 1 -- 172.21.15.83:0/107164175 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe734068050 con 0x7fe7340b40d0 2024-03-06T07:26:55.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.892+0000 7fe7267fc640 1 -- 172.21.15.83:0/107164175 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe728038020 con 0x7fe7340b40d0 2024-03-06T07:26:55.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.892+0000 7fe7267fc640 1 --2- 172.21.15.83:0/107164175 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe71806b4d0 0x7fe71806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.893+0000 7fe739f58640 1 --2- 172.21.15.83:0/107164175 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe71806b4d0 0x7fe71806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.893+0000 7fe7267fc640 1 -- 172.21.15.83:0/107164175 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fe7280ac6e0 con 0x7fe7340b40d0 2024-03-06T07:26:55.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.893+0000 7fe739f58640 1 --2- 172.21.15.83:0/107164175 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe71806b4d0 0x7fe71806d990 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fe73000ab60 tx=0x7fe730008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:55.897+0000 7fe7267fc640 1 -- 172.21.15.83:0/107164175 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe72807b500 con 0x7fe7340b40d0 2024-03-06T07:26:56.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.201+0000 7fe73c1bc640 1 -- 172.21.15.83:0/107164175 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe7340b3450 con 0x7fe7340b40d0 2024-03-06T07:26:56.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.201+0000 7fe7267fc640 1 -- 172.21.15.83:0/107164175 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v98) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe72807ae00 con 0x7fe7340b40d0 2024-03-06T07:26:56.203 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:26:56.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.204+0000 7fe73c1bc640 1 -- 172.21.15.83:0/107164175 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe71806b4d0 msgr2=0x7fe71806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.204+0000 7fe73c1bc640 1 --2- 172.21.15.83:0/107164175 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe71806b4d0 0x7fe71806d990 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fe73000ab60 tx=0x7fe730008040 comp rx=0 tx=0).stop 2024-03-06T07:26:56.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.204+0000 7fe73c1bc640 1 -- 172.21.15.83:0/107164175 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7340b40d0 msgr2=0x7fe73414a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.204+0000 7fe73c1bc640 1 --2- 172.21.15.83:0/107164175 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7340b40d0 0x7fe73414a290 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fe728000c00 tx=0x7fe7280046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.204+0000 7fe73c1bc640 1 -- 172.21.15.83:0/107164175 shutdown_connections 2024-03-06T07:26:56.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.204+0000 7fe73c1bc640 1 --2- 172.21.15.83:0/107164175 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe71806b4d0 0x7fe71806d990 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.204+0000 7fe73c1bc640 1 --2- 172.21.15.83:0/107164175 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe734158690 0x7fe7341e8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.204+0000 7fe73c1bc640 1 --2- 172.21.15.83:0/107164175 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7340b40d0 0x7fe73414a290 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.204+0000 7fe73c1bc640 1 --2- 172.21.15.83:0/107164175 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe7340affa0 0x7fe7341e8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.204+0000 7fe73c1bc640 1 -- 172.21.15.83:0/107164175 >> 172.21.15.83:0/107164175 conn(0x7fe7340b2890 msgr2=0x7fe7341618d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:56.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.204+0000 7fe73c1bc640 1 -- 172.21.15.83:0/107164175 shutdown_connections 2024-03-06T07:26:56.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.204+0000 7fe73c1bc640 1 -- 172.21.15.83:0/107164175 wait complete. 2024-03-06T07:26:56.217 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:26:56.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.320+0000 7f100f181640 1 Processor -- start 2024-03-06T07:26:56.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.320+0000 7f100f181640 1 -- start start 2024-03-06T07:26:56.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.321+0000 7f100f181640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f100815e150 0x7f10080cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.321+0000 7f100f181640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f10080cfba0 0x7f10080d0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.321+0000 7f100f181640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10080d1ea0 con 0x7f10080d0e90 2024-03-06T07:26:56.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.321+0000 7f100f181640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10080d2ab0 con 0x7f10080cfba0 2024-03-06T07:26:56.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.321+0000 7f100f181640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10080d2180 con 0x7f100815e150 2024-03-06T07:26:56.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.322+0000 7f100cf1d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f100815e150 0x7f10080cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.322+0000 7f1007fff640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f10080cfba0 0x7f10080d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.322+0000 7f100d71e640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f10080d0e90 0x7f10080b9060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:50710/0 (socket says 172.21.15.83:50710) 2024-03-06T07:26:56.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.322+0000 7f100cf1d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f100815e150 0x7f10080cf660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:46604/0 (socket says 172.21.15.83:46604) 2024-03-06T07:26:56.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.322+0000 7f100d71e640 1 -- 172.21.15.83:0/944659035 learned_addr learned my addr 172.21.15.83:0/944659035 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:56.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.322+0000 7f100cf1d640 1 -- 172.21.15.83:0/944659035 >> v2:172.21.15.183:3300/0 conn(0x7f10080cfba0 msgr2=0x7f10080d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:56.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.322+0000 7f100cf1d640 1 --2- 172.21.15.83:0/944659035 >> v2:172.21.15.183:3300/0 conn(0x7f10080cfba0 0x7f10080d0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.322+0000 7f10077fe640 1 -- 172.21.15.83:0/944659035 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1198702799 0 0) 0x7f10080d1ea0 con 0x7f10080d0e90 2024-03-06T07:26:56.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.322+0000 7f100cf1d640 1 -- 172.21.15.83:0/944659035 >> v1:172.21.15.183:6789/0 conn(0x7f10080d0e90 legacy=0x7f10080b9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.322+0000 7f100cf1d640 1 -- 172.21.15.83:0/944659035 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10080d2460 con 0x7f100815e150 2024-03-06T07:26:56.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.323+0000 7f100cf1d640 1 --2- 172.21.15.83:0/944659035 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f100815e150 0x7f10080cf660 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc0097a0 tx=0x7f0ffc02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7460957290668a37 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.323+0000 7f10077fe640 1 -- 172.21.15.83:0/944659035 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc0315a0 con 0x7f100815e150 2024-03-06T07:26:56.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.323+0000 7f10077fe640 1 -- 172.21.15.83:0/944659035 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ffc037070 con 0x7f100815e150 2024-03-06T07:26:56.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.323+0000 7f10077fe640 1 -- 172.21.15.83:0/944659035 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc031960 con 0x7f100815e150 2024-03-06T07:26:56.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.323+0000 7f100f181640 1 -- 172.21.15.83:0/944659035 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f100815e150 msgr2=0x7f10080cf660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.323+0000 7f100f181640 1 --2- 172.21.15.83:0/944659035 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f100815e150 0x7f10080cf660 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc0097a0 tx=0x7f0ffc02b840 comp rx=0 tx=0).stop 2024-03-06T07:26:56.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.323+0000 7f100f181640 1 -- 172.21.15.83:0/944659035 shutdown_connections 2024-03-06T07:26:56.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.323+0000 7f100f181640 1 --2- 172.21.15.83:0/944659035 >> v2:172.21.15.183:3300/0 conn(0x7f10080cfba0 0x7f10080d0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.323+0000 7f100f181640 1 --2- 172.21.15.83:0/944659035 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f100815e150 0x7f10080cf660 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.323+0000 7f100f181640 1 -- 172.21.15.83:0/944659035 >> 172.21.15.83:0/944659035 conn(0x7f100809d230 msgr2=0x7f10080d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:56.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.324+0000 7f100f181640 1 -- 172.21.15.83:0/944659035 shutdown_connections 2024-03-06T07:26:56.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.324+0000 7f100f181640 1 -- 172.21.15.83:0/944659035 wait complete. 2024-03-06T07:26:56.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.325+0000 7f100f181640 1 Processor -- start 2024-03-06T07:26:56.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.325+0000 7f100f181640 1 -- start start 2024-03-06T07:26:56.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.325+0000 7f100f181640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10080cfba0 0x7f10081514d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.325+0000 7f100f181640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10080d0e90 0x7f10080b6b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.325+0000 7f100f181640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f100815e150 0x7f10080ca050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.325+0000 7f100f181640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10080d1b30 con 0x7f10080d0e90 2024-03-06T07:26:56.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.325+0000 7f100f181640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10080d2f40 con 0x7f10080cfba0 2024-03-06T07:26:56.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.325+0000 7f100f181640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f100815e530 con 0x7f100815e150 2024-03-06T07:26:56.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.326+0000 7f1007fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10080d0e90 0x7f10080b6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.326+0000 7f1007fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10080d0e90 0x7f10080b6b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53514/0 (socket says 172.21.15.83:53514) 2024-03-06T07:26:56.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.326+0000 7f1007fff640 1 -- 172.21.15.83:0/2527142125 learned_addr learned my addr 172.21.15.83:0/2527142125 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:56.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.326+0000 7f100d71e640 1 --2- 172.21.15.83:0/2527142125 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f100815e150 0x7f10080ca050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.326+0000 7f100cf1d640 1 --2- 172.21.15.83:0/2527142125 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10080cfba0 0x7f10081514d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.326+0000 7f1007fff640 1 -- 172.21.15.83:0/2527142125 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f100815e150 msgr2=0x7f10080ca050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.326+0000 7f1007fff640 1 --2- 172.21.15.83:0/2527142125 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f100815e150 0x7f10080ca050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.326+0000 7f1007fff640 1 -- 172.21.15.83:0/2527142125 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10080cfba0 msgr2=0x7f10081514d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.326+0000 7f1007fff640 1 --2- 172.21.15.83:0/2527142125 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10080cfba0 0x7f10081514d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.326+0000 7f1007fff640 1 -- 172.21.15.83:0/2527142125 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10080c8ec0 con 0x7f10080d0e90 2024-03-06T07:26:56.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.326+0000 7f1007fff640 1 --2- 172.21.15.83:0/2527142125 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10080d0e90 0x7f10080b6b20 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f0ff400e9b0 tx=0x7f0ff400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.326+0000 7f10057fa640 1 -- 172.21.15.83:0/2527142125 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff4017070 con 0x7f10080d0e90 2024-03-06T07:26:56.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.326+0000 7f10057fa640 1 -- 172.21.15.83:0/2527142125 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ff400c9a0 con 0x7f10080d0e90 2024-03-06T07:26:56.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.326+0000 7f10057fa640 1 -- 172.21.15.83:0/2527142125 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff400cd90 con 0x7f10080d0e90 2024-03-06T07:26:56.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.326+0000 7f100f181640 1 -- 172.21.15.83:0/2527142125 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10080c91a0 con 0x7f10080d0e90 2024-03-06T07:26:56.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.326+0000 7f100f181640 1 -- 172.21.15.83:0/2527142125 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10080ca590 con 0x7f10080d0e90 2024-03-06T07:26:56.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.327+0000 7f100f181640 1 -- 172.21.15.83:0/2527142125 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1008069ee0 con 0x7f10080d0e90 2024-03-06T07:26:56.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.328+0000 7f10057fa640 1 -- 172.21.15.83:0/2527142125 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0ff401a070 con 0x7f10080d0e90 2024-03-06T07:26:56.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.328+0000 7f10057fa640 1 --2- 172.21.15.83:0/2527142125 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0fec06b4d0 0x7f0fec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.328+0000 7f10057fa640 1 -- 172.21.15.83:0/2527142125 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0ff408a5e0 con 0x7f10080d0e90 2024-03-06T07:26:56.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.328+0000 7f100cf1d640 1 --2- 172.21.15.83:0/2527142125 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0fec06b4d0 0x7f0fec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.329+0000 7f100cf1d640 1 --2- 172.21.15.83:0/2527142125 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0fec06b4d0 0x7f0fec06d990 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc036000 tx=0x7f0ffc008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.334+0000 7f10057fa640 1 -- 172.21.15.83:0/2527142125 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0ff4058d10 con 0x7f10080d0e90 2024-03-06T07:26:56.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.520+0000 7f100f181640 1 -- 172.21.15.83:0/2527142125 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f10080d42f0 con 0x7f10080d0e90 2024-03-06T07:26:56.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.521+0000 7f10057fa640 1 -- 172.21.15.83:0/2527142125 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v98) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0ff405cdb0 con 0x7f10080d0e90 2024-03-06T07:26:56.523 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-03-06T07:26:56.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.523+0000 7f100f181640 1 -- 172.21.15.83:0/2527142125 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0fec06b4d0 msgr2=0x7f0fec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.523+0000 7f100f181640 1 --2- 172.21.15.83:0/2527142125 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0fec06b4d0 0x7f0fec06d990 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc036000 tx=0x7f0ffc008030 comp rx=0 tx=0).stop 2024-03-06T07:26:56.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.523+0000 7f100f181640 1 -- 172.21.15.83:0/2527142125 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10080d0e90 msgr2=0x7f10080b6b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.523+0000 7f100f181640 1 --2- 172.21.15.83:0/2527142125 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10080d0e90 0x7f10080b6b20 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f0ff400e9b0 tx=0x7f0ff400ee70 comp rx=0 tx=0).stop 2024-03-06T07:26:56.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.523+0000 7f100f181640 1 -- 172.21.15.83:0/2527142125 shutdown_connections 2024-03-06T07:26:56.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.523+0000 7f100f181640 1 --2- 172.21.15.83:0/2527142125 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0fec06b4d0 0x7f0fec06d990 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.523+0000 7f100f181640 1 --2- 172.21.15.83:0/2527142125 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f100815e150 0x7f10080ca050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.523+0000 7f100f181640 1 --2- 172.21.15.83:0/2527142125 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10080d0e90 0x7f10080b6b20 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.523+0000 7f100f181640 1 --2- 172.21.15.83:0/2527142125 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10080cfba0 0x7f10081514d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.523+0000 7f100f181640 1 -- 172.21.15.83:0/2527142125 >> 172.21.15.83:0/2527142125 conn(0x7f100809d230 msgr2=0x7f1008161240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:56.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.523+0000 7f100f181640 1 -- 172.21.15.83:0/2527142125 shutdown_connections 2024-03-06T07:26:56.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.523+0000 7f100f181640 1 -- 172.21.15.83:0/2527142125 wait complete. 2024-03-06T07:26:56.536 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:26:56.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.651+0000 7f63c8ec9640 1 Processor -- start 2024-03-06T07:26:56.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.651+0000 7f63c8ec9640 1 -- start start 2024-03-06T07:26:56.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.652+0000 7f63c8ec9640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f63c40cdb30 0x7f63c415c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.652+0000 7f63c8ec9640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c40cdf90 con 0x7f63c4145900 2024-03-06T07:26:56.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.652+0000 7f63c8ec9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c406d3a0 con 0x7f63c415c970 2024-03-06T07:26:56.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.652+0000 7f63c8ec9640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63c406d150 con 0x7f63c40cdb30 2024-03-06T07:26:56.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.652+0000 7f63c259c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f63c4145900 0x7f63c40cf610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:55218/0 (socket says 172.21.15.83:55218) 2024-03-06T07:26:56.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.652+0000 7f63c259c640 1 -- 172.21.15.83:0/3806680236 learned_addr learned my addr 172.21.15.83:0/3806680236 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:56.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.652+0000 7f63c159a640 1 -- 172.21.15.83:0/3806680236 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3164298450 0 0) 0x7f63c40cdf90 con 0x7f63c4145900 2024-03-06T07:26:56.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.652+0000 7f63c159a640 1 -- 172.21.15.83:0/3806680236 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63ac003520 con 0x7f63c4145900 2024-03-06T07:26:56.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.652+0000 7f63c1d9b640 1 --2- 172.21.15.83:0/3806680236 >> v2:172.21.15.183:3300/0 conn(0x7f63c40cdb30 0x7f63c415c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.652+0000 7f63c159a640 1 -- 172.21.15.83:0/3806680236 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3154165345 0 0) 0x7f63ac003520 con 0x7f63c4145900 2024-03-06T07:26:56.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.653+0000 7f63c159a640 1 -- 172.21.15.83:0/3806680236 >> v2:172.21.15.183:3300/0 conn(0x7f63c40cdb30 msgr2=0x7f63c415c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:56.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.653+0000 7f63c159a640 1 --2- 172.21.15.83:0/3806680236 >> v2:172.21.15.183:3300/0 conn(0x7f63c40cdb30 0x7f63c415c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.653+0000 7f63c159a640 1 -- 172.21.15.83:0/3806680236 >> v1:172.21.15.183:6789/0 conn(0x7f63c415c970 legacy=0x7f63c406e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.653+0000 7f63c159a640 1 -- 172.21.15.83:0/3806680236 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c406d680 con 0x7f63c4145900 2024-03-06T07:26:56.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.653+0000 7f63c159a640 1 -- 172.21.15.83:0/3806680236 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f63bc001670 con 0x7f63c4145900 2024-03-06T07:26:56.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.653+0000 7f63c159a640 1 -- 172.21.15.83:0/3806680236 >> v1:172.21.15.83:6789/0 conn(0x7f63c4145900 legacy=0x7f63c40cf610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.653+0000 7f63c159a640 1 --2- 172.21.15.83:0/3806680236 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f63ac0039b0 0x7f63ac003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.653+0000 7f63c159a640 1 --2- 172.21.15.83:0/3806680236 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f63ac004800 0x7f63ac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.653+0000 7f63c159a640 1 --2- 172.21.15.83:0/3806680236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63ac0052e0 0x7f63ac00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.653+0000 7f63c159a640 1 -- 172.21.15.83:0/3806680236 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c406d150 con 0x7f63ac0052e0 2024-03-06T07:26:56.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.653+0000 7f63c159a640 1 -- 172.21.15.83:0/3806680236 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63ac00c1b0 con 0x7f63ac004800 2024-03-06T07:26:56.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.653+0000 7f63c159a640 1 -- 172.21.15.83:0/3806680236 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63ac00c3b0 con 0x7f63ac0039b0 2024-03-06T07:26:56.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.653+0000 7f63c2d9d640 1 --2- 172.21.15.83:0/3806680236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63ac0052e0 0x7f63ac00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.653+0000 7f63c259c640 1 --2- 172.21.15.83:0/3806680236 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f63ac0039b0 0x7f63ac003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.653+0000 7f63c1d9b640 1 --2- 172.21.15.83:0/3806680236 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f63ac004800 0x7f63ac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.653+0000 7f63c2d9d640 1 -- 172.21.15.83:0/3806680236 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f63ac0039b0 msgr2=0x7f63ac003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:56.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.653+0000 7f63c2d9d640 1 --2- 172.21.15.83:0/3806680236 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f63ac0039b0 0x7f63ac003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.653+0000 7f63c2d9d640 1 -- 172.21.15.83:0/3806680236 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f63ac004800 msgr2=0x7f63ac004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:56.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.653+0000 7f63c2d9d640 1 --2- 172.21.15.83:0/3806680236 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f63ac004800 0x7f63ac004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.653+0000 7f63c2d9d640 1 -- 172.21.15.83:0/3806680236 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63ac003520 con 0x7f63ac0052e0 2024-03-06T07:26:56.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.653+0000 7f63c259c640 1 --2- 172.21.15.83:0/3806680236 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f63ac0039b0 0x7f63ac003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:56.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.654+0000 7f63c2d9d640 1 -- 172.21.15.83:0/3806680236 reap_dead start 2024-03-06T07:26:56.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.654+0000 7f63c2d9d640 1 --2- 172.21.15.83:0/3806680236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63ac0052e0 0x7f63ac00b9c0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f63c415cde0 tx=0x7f63ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89f1f28a2f0f12ee server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.654+0000 7f63c159a640 1 -- 172.21.15.83:0/3806680236 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b4001370 con 0x7f63ac0052e0 2024-03-06T07:26:56.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.654+0000 7f63c159a640 1 -- 172.21.15.83:0/3806680236 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b40052d0 con 0x7f63ac0052e0 2024-03-06T07:26:56.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.654+0000 7f63c159a640 1 -- 172.21.15.83:0/3806680236 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b4003810 con 0x7f63ac0052e0 2024-03-06T07:26:56.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.654+0000 7f63c8ec9640 1 -- 172.21.15.83:0/3806680236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63ac0052e0 msgr2=0x7f63ac00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.654+0000 7f63c8ec9640 1 --2- 172.21.15.83:0/3806680236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63ac0052e0 0x7f63ac00b9c0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f63c415cde0 tx=0x7f63ac005830 comp rx=0 tx=0).stop 2024-03-06T07:26:56.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.654+0000 7f63c8ec9640 1 -- 172.21.15.83:0/3806680236 shutdown_connections 2024-03-06T07:26:56.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.654+0000 7f63c8ec9640 1 --2- 172.21.15.83:0/3806680236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63ac0052e0 0x7f63ac00b9c0 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.654+0000 7f63c8ec9640 1 -- 172.21.15.83:0/3806680236 >> 172.21.15.83:0/3806680236 conn(0x7f63c406b160 msgr2=0x7f63c406b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:56.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.655+0000 7f63c8ec9640 1 -- 172.21.15.83:0/3806680236 shutdown_connections 2024-03-06T07:26:56.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.655+0000 7f63c8ec9640 1 -- 172.21.15.83:0/3806680236 wait complete. 2024-03-06T07:26:56.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.656+0000 7f63c8ec9640 1 Processor -- start 2024-03-06T07:26:56.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.656+0000 7f63c8ec9640 1 -- start start 2024-03-06T07:26:56.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.657+0000 7f63c8ec9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f63ac0052e0 0x7f63c413b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.657+0000 7f63c8ec9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63c413cb50 0x7f63c4143f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.657+0000 7f63c8ec9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f63c4144480 0x7f63c4139050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.657+0000 7f63c8ec9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c415c430 con 0x7f63c413cb50 2024-03-06T07:26:56.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.657+0000 7f63c8ec9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c406dc40 con 0x7f63c4144480 2024-03-06T07:26:56.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.657+0000 7f63c8ec9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c415c5b0 con 0x7f63ac0052e0 2024-03-06T07:26:56.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.657+0000 7f63c1d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63c413cb50 0x7f63c4143f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.657+0000 7f63c1d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63c413cb50 0x7f63c4143f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51344/0 (socket says 172.21.15.83:51344) 2024-03-06T07:26:56.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.657+0000 7f63c1d9b640 1 -- 172.21.15.83:0/2825774301 learned_addr learned my addr 172.21.15.83:0/2825774301 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:56.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.657+0000 7f63c259c640 1 --2- 172.21.15.83:0/2825774301 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f63ac0052e0 0x7f63c413b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.657+0000 7f63c2d9d640 1 --2- 172.21.15.83:0/2825774301 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f63c4144480 0x7f63c4139050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.657+0000 7f63c1d9b640 1 -- 172.21.15.83:0/2825774301 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f63ac0052e0 msgr2=0x7f63c413b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.657+0000 7f63c1d9b640 1 --2- 172.21.15.83:0/2825774301 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f63ac0052e0 0x7f63c413b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.657+0000 7f63c1d9b640 1 -- 172.21.15.83:0/2825774301 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f63c4144480 msgr2=0x7f63c4139050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.657+0000 7f63c1d9b640 1 --2- 172.21.15.83:0/2825774301 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f63c4144480 0x7f63c4139050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.657+0000 7f63c1d9b640 1 -- 172.21.15.83:0/2825774301 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c413bab0 con 0x7f63c413cb50 2024-03-06T07:26:56.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.657+0000 7f63c2d9d640 1 --2- 172.21.15.83:0/2825774301 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f63c4144480 0x7f63c4139050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:56.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.657+0000 7f63c1d9b640 1 --2- 172.21.15.83:0/2825774301 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63c413cb50 0x7f63c4143f40 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f63b8009fb0 tx=0x7f63b80115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.657+0000 7f63b2ffd640 1 -- 172.21.15.83:0/2825774301 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b801a020 con 0x7f63c413cb50 2024-03-06T07:26:56.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.657+0000 7f63b2ffd640 1 -- 172.21.15.83:0/2825774301 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b8016470 con 0x7f63c413cb50 2024-03-06T07:26:56.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.657+0000 7f63b2ffd640 1 -- 172.21.15.83:0/2825774301 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b8016860 con 0x7f63c413cb50 2024-03-06T07:26:56.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.657+0000 7f63c8ec9640 1 -- 172.21.15.83:0/2825774301 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63c413bd90 con 0x7f63c413cb50 2024-03-06T07:26:56.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.657+0000 7f63c8ec9640 1 -- 172.21.15.83:0/2825774301 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63c413c350 con 0x7f63c413cb50 2024-03-06T07:26:56.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.658+0000 7f63c8ec9640 1 -- 172.21.15.83:0/2825774301 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63c4069ee0 con 0x7f63c413cb50 2024-03-06T07:26:56.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.659+0000 7f63b2ffd640 1 -- 172.21.15.83:0/2825774301 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f63b8005d10 con 0x7f63c413cb50 2024-03-06T07:26:56.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.659+0000 7f63b2ffd640 1 --2- 172.21.15.83:0/2825774301 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f63a406b500 0x7f63a406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.659+0000 7f63b2ffd640 1 -- 172.21.15.83:0/2825774301 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f63b808b070 con 0x7f63c413cb50 2024-03-06T07:26:56.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.659+0000 7f63c259c640 1 --2- 172.21.15.83:0/2825774301 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f63a406b500 0x7f63a406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.660+0000 7f63c259c640 1 --2- 172.21.15.83:0/2825774301 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f63a406b500 0x7f63a406d9c0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f63bc001eb0 tx=0x7f63bc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.668+0000 7f63b2ffd640 1 -- 172.21.15.83:0/2825774301 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f63b8024030 con 0x7f63c413cb50 2024-03-06T07:26:56.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.843+0000 7f63c8ec9640 1 -- 172.21.15.83:0/2825774301 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f63c4144860 con 0x7f63c413cb50 2024-03-06T07:26:56.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.843+0000 7f63b2ffd640 1 -- 172.21.15.83:0/2825774301 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v98) v1 ==== 100+0+11 (secure 0 0 0) 0x7f63b8059f20 con 0x7f63c413cb50 2024-03-06T07:26:56.846 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 64 2024-03-06T07:26:56.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.845+0000 7f63c8ec9640 1 -- 172.21.15.83:0/2825774301 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f63a406b500 msgr2=0x7f63a406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.845+0000 7f63c8ec9640 1 --2- 172.21.15.83:0/2825774301 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f63a406b500 0x7f63a406d9c0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f63bc001eb0 tx=0x7f63bc009040 comp rx=0 tx=0).stop 2024-03-06T07:26:56.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.845+0000 7f63c8ec9640 1 -- 172.21.15.83:0/2825774301 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63c413cb50 msgr2=0x7f63c4143f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.845+0000 7f63c8ec9640 1 --2- 172.21.15.83:0/2825774301 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63c413cb50 0x7f63c4143f40 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f63b8009fb0 tx=0x7f63b80115a0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.845+0000 7f63c8ec9640 1 -- 172.21.15.83:0/2825774301 shutdown_connections 2024-03-06T07:26:56.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.845+0000 7f63c8ec9640 1 --2- 172.21.15.83:0/2825774301 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f63a406b500 0x7f63a406d9c0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.845+0000 7f63c8ec9640 1 --2- 172.21.15.83:0/2825774301 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f63c4144480 0x7f63c4139050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.845+0000 7f63c8ec9640 1 --2- 172.21.15.83:0/2825774301 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63c413cb50 0x7f63c4143f40 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.845+0000 7f63c8ec9640 1 --2- 172.21.15.83:0/2825774301 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f63ac0052e0 0x7f63c413b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.845+0000 7f63c8ec9640 1 -- 172.21.15.83:0/2825774301 >> 172.21.15.83:0/2825774301 conn(0x7f63c406b160 msgr2=0x7f63c406e060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:56.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.845+0000 7f63c8ec9640 1 -- 172.21.15.83:0/2825774301 shutdown_connections 2024-03-06T07:26:56.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.845+0000 7f63c8ec9640 1 -- 172.21.15.83:0/2825774301 wait complete. 2024-03-06T07:26:56.858 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:56.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.958+0000 7f5c794c2640 1 Processor -- start 2024-03-06T07:26:56.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.958+0000 7f5c794c2640 1 -- start start 2024-03-06T07:26:56.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.959+0000 7f5c794c2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5c740b5010 0x7f5c740d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.959+0000 7f5c794c2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c740d25a0 0x7f5c740d2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.959+0000 7f5c794c2640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c74161e60 con 0x7f5c74161a80 2024-03-06T07:26:56.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.959+0000 7f5c794c2640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c740b1250 con 0x7f5c740b5010 2024-03-06T07:26:56.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.959+0000 7f5c794c2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c740b3e30 con 0x7f5c740d25a0 2024-03-06T07:26:56.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.959+0000 7f5c727fc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5c740b5010 0x7f5c740d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.959+0000 7f5c72ffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f5c74161a80 0x7f5c740b0b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:55222/0 (socket says 172.21.15.83:55222) 2024-03-06T07:26:56.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.959+0000 7f5c72ffd640 1 -- 172.21.15.83:0/1819180051 learned_addr learned my addr 172.21.15.83:0/1819180051 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:56.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.959+0000 7f5c737fe640 1 --2- 172.21.15.83:0/1819180051 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c740d25a0 0x7f5c740d2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.959+0000 7f5c727fc640 1 -- 172.21.15.83:0/1819180051 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c740d25a0 msgr2=0x7f5c740d2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:56.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.959+0000 7f5c727fc640 1 --2- 172.21.15.83:0/1819180051 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c740d25a0 0x7f5c740d2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.959+0000 7f5c727fc640 1 -- 172.21.15.83:0/1819180051 >> v1:172.21.15.83:6789/0 conn(0x7f5c74161a80 legacy=0x7f5c740b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.959+0000 7f5c71ffb640 1 -- 172.21.15.83:0/1819180051 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2946801626 0 0) 0x7f5c68001110 con 0x7f5c74161a80 2024-03-06T07:26:56.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.959+0000 7f5c727fc640 1 -- 172.21.15.83:0/1819180051 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c740d1470 con 0x7f5c740b5010 2024-03-06T07:26:56.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.960+0000 7f5c727fc640 1 --2- 172.21.15.83:0/1819180051 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c740b5010 0x7f5c740d1da0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f5c640097a0 tx=0x7f5c6402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b830f15249a5c379 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.960+0000 7f5c71ffb640 1 -- 172.21.15.83:0/1819180051 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c640315a0 con 0x7f5c740b5010 2024-03-06T07:26:56.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.960+0000 7f5c71ffb640 1 -- 172.21.15.83:0/1819180051 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c64037070 con 0x7f5c740b5010 2024-03-06T07:26:56.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.960+0000 7f5c71ffb640 1 -- 172.21.15.83:0/1819180051 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c64031960 con 0x7f5c740b5010 2024-03-06T07:26:56.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.961+0000 7f5c794c2640 1 -- 172.21.15.83:0/1819180051 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c740b5010 msgr2=0x7f5c740d1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.961+0000 7f5c794c2640 1 --2- 172.21.15.83:0/1819180051 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c740b5010 0x7f5c740d1da0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f5c640097a0 tx=0x7f5c6402b840 comp rx=0 tx=0).stop 2024-03-06T07:26:56.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.961+0000 7f5c794c2640 1 -- 172.21.15.83:0/1819180051 shutdown_connections 2024-03-06T07:26:56.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.961+0000 7f5c794c2640 1 --2- 172.21.15.83:0/1819180051 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c740d25a0 0x7f5c740d2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.961+0000 7f5c794c2640 1 --2- 172.21.15.83:0/1819180051 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c740b5010 0x7f5c740d1da0 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.961+0000 7f5c794c2640 1 -- 172.21.15.83:0/1819180051 >> 172.21.15.83:0/1819180051 conn(0x7f5c740d3fc0 msgr2=0x7f5c740d43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:56.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.961+0000 7f5c794c2640 1 -- 172.21.15.83:0/1819180051 shutdown_connections 2024-03-06T07:26:56.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.961+0000 7f5c794c2640 1 -- 172.21.15.83:0/1819180051 wait complete. 2024-03-06T07:26:56.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.963+0000 7f5c794c2640 1 Processor -- start 2024-03-06T07:26:56.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.963+0000 7f5c794c2640 1 -- start start 2024-03-06T07:26:56.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.964+0000 7f5c794c2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c740b5010 0x7f5c7414de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.964+0000 7f5c794c2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c740d25a0 0x7f5c7414e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.964+0000 7f5c794c2640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5c74161a80 0x7f5c74150860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.964+0000 7f5c794c2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c740d1090 con 0x7f5c740d25a0 2024-03-06T07:26:56.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.964+0000 7f5c794c2640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c740b3ac0 con 0x7f5c74161a80 2024-03-06T07:26:56.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.964+0000 7f5c794c2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c740d1210 con 0x7f5c740b5010 2024-03-06T07:26:56.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.964+0000 7f5c727fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c740d25a0 0x7f5c7414e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.964+0000 7f5c727fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c740d25a0 0x7f5c7414e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51368/0 (socket says 172.21.15.83:51368) 2024-03-06T07:26:56.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.964+0000 7f5c727fc640 1 -- 172.21.15.83:0/2695034938 learned_addr learned my addr 172.21.15.83:0/2695034938 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:56.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.964+0000 7f5c72ffd640 1 --2- 172.21.15.83:0/2695034938 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c740b5010 0x7f5c7414de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.964+0000 7f5c737fe640 1 --2- 172.21.15.83:0/2695034938 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5c74161a80 0x7f5c74150860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.965+0000 7f5c727fc640 1 -- 172.21.15.83:0/2695034938 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c740b5010 msgr2=0x7f5c7414de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.965+0000 7f5c727fc640 1 --2- 172.21.15.83:0/2695034938 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c740b5010 0x7f5c7414de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.965+0000 7f5c727fc640 1 -- 172.21.15.83:0/2695034938 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5c74161a80 msgr2=0x7f5c74150860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.965+0000 7f5c727fc640 1 --2- 172.21.15.83:0/2695034938 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5c74161a80 0x7f5c74150860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.965+0000 7f5c727fc640 1 -- 172.21.15.83:0/2695034938 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c7414f290 con 0x7f5c740d25a0 2024-03-06T07:26:56.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.965+0000 7f5c72ffd640 1 --2- 172.21.15.83:0/2695034938 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c740b5010 0x7f5c7414de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:56.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.965+0000 7f5c727fc640 1 --2- 172.21.15.83:0/2695034938 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c740d25a0 0x7f5c7414e7d0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f5c64034040 tx=0x7f5c640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.965+0000 7f5c5b7fe640 1 -- 172.21.15.83:0/2695034938 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c64004230 con 0x7f5c740d25a0 2024-03-06T07:26:56.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.965+0000 7f5c5b7fe640 1 -- 172.21.15.83:0/2695034938 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c640043d0 con 0x7f5c740d25a0 2024-03-06T07:26:56.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.965+0000 7f5c794c2640 1 -- 172.21.15.83:0/2695034938 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c7414f570 con 0x7f5c740d25a0 2024-03-06T07:26:56.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.965+0000 7f5c5b7fe640 1 -- 172.21.15.83:0/2695034938 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c64003850 con 0x7f5c740d25a0 2024-03-06T07:26:56.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.965+0000 7f5c794c2640 1 -- 172.21.15.83:0/2695034938 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c7414fe10 con 0x7f5c740d25a0 2024-03-06T07:26:56.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.966+0000 7f5c794c2640 1 -- 172.21.15.83:0/2695034938 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c74068050 con 0x7f5c740d25a0 2024-03-06T07:26:56.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.966+0000 7f5c5b7fe640 1 -- 172.21.15.83:0/2695034938 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5c640039f0 con 0x7f5c740d25a0 2024-03-06T07:26:56.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.967+0000 7f5c5b7fe640 1 --2- 172.21.15.83:0/2695034938 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5c6006b500 0x7f5c6006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.967+0000 7f5c5b7fe640 1 -- 172.21.15.83:0/2695034938 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f5c6402f030 con 0x7f5c740d25a0 2024-03-06T07:26:56.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.967+0000 7f5c72ffd640 1 --2- 172.21.15.83:0/2695034938 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5c6006b500 0x7f5c6006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.967+0000 7f5c72ffd640 1 --2- 172.21.15.83:0/2695034938 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5c6006b500 0x7f5c6006d9c0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f5c68000f60 tx=0x7f5c68007ca0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:56.971+0000 7f5c5b7fe640 1 -- 172.21.15.83:0/2695034938 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5c64041070 con 0x7f5c740d25a0 2024-03-06T07:26:57.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.218+0000 7f5c794c2640 1 -- 172.21.15.83:0/2695034938 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c740d2380 con 0x7f5c740d25a0 2024-03-06T07:26:57.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.218+0000 7f5c5b7fe640 1 -- 172.21.15.83:0/2695034938 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f5c6407b010 con 0x7f5c740d25a0 2024-03-06T07:26:57.220 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:57.220 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":53,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6023,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/601236077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":601236077},{"type":"v1","addr":"172.21.15.183:6833","nonce":601236077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6029,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1435055961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1435055961},{"type":"v1","addr":"172.21.15.183:6835","nonce":1435055961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":7572,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/2726045966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2726045966},{"type":"v1","addr":"172.21.15.83:6836","nonce":2726045966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.064335+0000","modified":"2024-03-06T07:26:32.814889+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7575":{"gid":7575,"name":"a","rank":0,"incarnation":49,"state":"up:active","state_seq":5,"addr":"172.21.15.83:6837/3254575964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":3254575964},{"type":"v1","addr":"172.21.15.83:6837","nonce":3254575964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7575,"qdb_cluster":[7575]},"id":7}]} 2024-03-06T07:26:57.221 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 53 2024-03-06T07:26:57.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.220+0000 7f5c794c2640 1 -- 172.21.15.83:0/2695034938 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5c6006b500 msgr2=0x7f5c6006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.220+0000 7f5c794c2640 1 --2- 172.21.15.83:0/2695034938 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5c6006b500 0x7f5c6006d9c0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f5c68000f60 tx=0x7f5c68007ca0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.220+0000 7f5c794c2640 1 -- 172.21.15.83:0/2695034938 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c740d25a0 msgr2=0x7f5c7414e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.220+0000 7f5c794c2640 1 --2- 172.21.15.83:0/2695034938 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c740d25a0 0x7f5c7414e7d0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f5c64034040 tx=0x7f5c640046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.220+0000 7f5c794c2640 1 -- 172.21.15.83:0/2695034938 shutdown_connections 2024-03-06T07:26:57.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.220+0000 7f5c794c2640 1 --2- 172.21.15.83:0/2695034938 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5c6006b500 0x7f5c6006d9c0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.220+0000 7f5c794c2640 1 --2- 172.21.15.83:0/2695034938 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5c74161a80 0x7f5c74150860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.220+0000 7f5c794c2640 1 --2- 172.21.15.83:0/2695034938 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c740d25a0 0x7f5c7414e7d0 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.220+0000 7f5c794c2640 1 --2- 172.21.15.83:0/2695034938 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c740b5010 0x7f5c7414de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.220+0000 7f5c794c2640 1 -- 172.21.15.83:0/2695034938 >> 172.21.15.83:0/2695034938 conn(0x7f5c740d3fc0 msgr2=0x7f5c740d0350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:57.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.220+0000 7f5c794c2640 1 -- 172.21.15.83:0/2695034938 shutdown_connections 2024-03-06T07:26:57.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.220+0000 7f5c794c2640 1 -- 172.21.15.83:0/2695034938 wait complete. 2024-03-06T07:26:57.237 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:57.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.335+0000 7f92c5525640 1 Processor -- start 2024-03-06T07:26:57.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.336+0000 7f92c5525640 1 -- start start 2024-03-06T07:26:57.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.337+0000 7f92c5525640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f92c006c6a0 0x7f92c00cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.337+0000 7f92c5525640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c00cd2b0 con 0x7f92c015bd30 2024-03-06T07:26:57.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.337+0000 7f92c5525640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c00cd590 con 0x7f92c00721b0 2024-03-06T07:26:57.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.337+0000 7f92c5525640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92c00cefd0 con 0x7f92c006c6a0 2024-03-06T07:26:57.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.337+0000 7f92beffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f92c015bd30 0x7f92c00b3570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:55228/0 (socket says 172.21.15.83:55228) 2024-03-06T07:26:57.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.337+0000 7f92beffd640 1 -- 172.21.15.83:0/3224103731 learned_addr learned my addr 172.21.15.83:0/3224103731 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:57.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.337+0000 7f92be7fc640 1 -- 172.21.15.83:0/3224103731 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2401628342 0 0) 0x7f92c00cd2b0 con 0x7f92c015bd30 2024-03-06T07:26:57.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.337+0000 7f92be7fc640 1 -- 172.21.15.83:0/3224103731 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92ac003520 con 0x7f92c015bd30 2024-03-06T07:26:57.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.337+0000 7f92bf7fe640 1 --2- 172.21.15.83:0/3224103731 >> v2:172.21.15.183:3300/0 conn(0x7f92c006c6a0 0x7f92c00cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.337+0000 7f92be7fc640 1 -- 172.21.15.83:0/3224103731 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3582712062 0 0) 0x7f92ac003520 con 0x7f92c015bd30 2024-03-06T07:26:57.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.337+0000 7f92be7fc640 1 -- 172.21.15.83:0/3224103731 >> v2:172.21.15.183:3300/0 conn(0x7f92c006c6a0 msgr2=0x7f92c00cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:57.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.337+0000 7f92be7fc640 1 --2- 172.21.15.83:0/3224103731 >> v2:172.21.15.183:3300/0 conn(0x7f92c006c6a0 0x7f92c00cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.337+0000 7f92be7fc640 1 -- 172.21.15.83:0/3224103731 >> v1:172.21.15.183:6789/0 conn(0x7f92c00721b0 legacy=0x7f92c006bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.338+0000 7f92be7fc640 1 -- 172.21.15.83:0/3224103731 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c00cf2b0 con 0x7f92c015bd30 2024-03-06T07:26:57.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.338+0000 7f92be7fc640 1 -- 172.21.15.83:0/3224103731 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f92b0001670 con 0x7f92c015bd30 2024-03-06T07:26:57.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.338+0000 7f92be7fc640 1 -- 172.21.15.83:0/3224103731 >> v1:172.21.15.83:6789/0 conn(0x7f92c015bd30 legacy=0x7f92c00b3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.338+0000 7f92be7fc640 1 --2- 172.21.15.83:0/3224103731 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92ac0039b0 0x7f92ac003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.338+0000 7f92be7fc640 1 --2- 172.21.15.83:0/3224103731 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92ac004800 0x7f92ac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.338+0000 7f92be7fc640 1 --2- 172.21.15.83:0/3224103731 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92ac0051c0 0x7f92ac00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.338+0000 7f92be7fc640 1 -- 172.21.15.83:0/3224103731 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c00cefd0 con 0x7f92ac0039b0 2024-03-06T07:26:57.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.338+0000 7f92be7fc640 1 -- 172.21.15.83:0/3224103731 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92ac00c160 con 0x7f92ac0051c0 2024-03-06T07:26:57.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.338+0000 7f92be7fc640 1 -- 172.21.15.83:0/3224103731 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92ac00c360 con 0x7f92ac004800 2024-03-06T07:26:57.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.338+0000 7f92beffd640 1 --2- 172.21.15.83:0/3224103731 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92ac0039b0 0x7f92ac003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.338+0000 7f92b7fff640 1 --2- 172.21.15.83:0/3224103731 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92ac004800 0x7f92ac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.338+0000 7f92bf7fe640 1 --2- 172.21.15.83:0/3224103731 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92ac0051c0 0x7f92ac00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.338+0000 7f92beffd640 1 -- 172.21.15.83:0/3224103731 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92ac004800 msgr2=0x7f92ac004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:57.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.338+0000 7f92beffd640 1 --2- 172.21.15.83:0/3224103731 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92ac004800 0x7f92ac004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.338+0000 7f92beffd640 1 -- 172.21.15.83:0/3224103731 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92ac0051c0 msgr2=0x7f92ac00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:57.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.338+0000 7f92beffd640 1 --2- 172.21.15.83:0/3224103731 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92ac0051c0 0x7f92ac00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.338+0000 7f92beffd640 1 -- 172.21.15.83:0/3224103731 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92ac003520 con 0x7f92ac0039b0 2024-03-06T07:26:57.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.338+0000 7f92bf7fe640 1 --2- 172.21.15.83:0/3224103731 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92ac0051c0 0x7f92ac00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:57.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.338+0000 7f92b7fff640 1 --2- 172.21.15.83:0/3224103731 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92ac004800 0x7f92ac004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:57.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.338+0000 7f92bf7fe640 1 -- 172.21.15.83:0/3224103731 reap_dead start 2024-03-06T07:26:57.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.339+0000 7f92beffd640 1 --2- 172.21.15.83:0/3224103731 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92ac0039b0 0x7f92ac003dd0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f92b000c240 tx=0x7f92ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=585b0c1421815ca8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:57.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.339+0000 7f92be7fc640 1 -- 172.21.15.83:0/3224103731 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b0007d10 con 0x7f92ac0039b0 2024-03-06T07:26:57.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.339+0000 7f92be7fc640 1 -- 172.21.15.83:0/3224103731 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92b0030070 con 0x7f92ac0039b0 2024-03-06T07:26:57.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.339+0000 7f92be7fc640 1 -- 172.21.15.83:0/3224103731 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b0005d90 con 0x7f92ac0039b0 2024-03-06T07:26:57.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.340+0000 7f92c5525640 1 -- 172.21.15.83:0/3224103731 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92ac0039b0 msgr2=0x7f92ac003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.340+0000 7f92c5525640 1 --2- 172.21.15.83:0/3224103731 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92ac0039b0 0x7f92ac003dd0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f92b000c240 tx=0x7f92ac005830 comp rx=0 tx=0).stop 2024-03-06T07:26:57.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.340+0000 7f92c5525640 1 -- 172.21.15.83:0/3224103731 shutdown_connections 2024-03-06T07:26:57.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.340+0000 7f92c5525640 1 --2- 172.21.15.83:0/3224103731 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92ac0039b0 0x7f92ac003dd0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.340+0000 7f92c5525640 1 -- 172.21.15.83:0/3224103731 >> 172.21.15.83:0/3224103731 conn(0x7f92c0071690 msgr2=0x7f92c0071aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:57.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.340+0000 7f92c5525640 1 -- 172.21.15.83:0/3224103731 shutdown_connections 2024-03-06T07:26:57.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.340+0000 7f92c5525640 1 -- 172.21.15.83:0/3224103731 wait complete. 2024-03-06T07:26:57.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.341+0000 7f92c5525640 1 Processor -- start 2024-03-06T07:26:57.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.341+0000 7f92c5525640 1 -- start start 2024-03-06T07:26:57.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.342+0000 7f92c5525640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92ac0039b0 0x7f92c0145680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.342+0000 7f92c5525640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92c0145dc0 0x7f92c00a98c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.342+0000 7f92c5525640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92c015a350 0x7f92c015a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.342+0000 7f92c5525640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c00ccf10 con 0x7f92c015a350 2024-03-06T07:26:57.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.342+0000 7f92c5525640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c00cec80 con 0x7f92ac0039b0 2024-03-06T07:26:57.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.342+0000 7f92c5525640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92c00cd090 con 0x7f92c0145dc0 2024-03-06T07:26:57.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.342+0000 7f92bf7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92c015a350 0x7f92c015a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.342+0000 7f92bf7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92c015a350 0x7f92c015a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51384/0 (socket says 172.21.15.83:51384) 2024-03-06T07:26:57.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.342+0000 7f92bf7fe640 1 -- 172.21.15.83:0/302376860 learned_addr learned my addr 172.21.15.83:0/302376860 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:57.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.342+0000 7f92beffd640 1 --2- 172.21.15.83:0/302376860 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92ac0039b0 0x7f92c0145680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.342+0000 7f92b7fff640 1 --2- 172.21.15.83:0/302376860 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92c0145dc0 0x7f92c00a98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.342+0000 7f92bf7fe640 1 -- 172.21.15.83:0/302376860 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92c0145dc0 msgr2=0x7f92c00a98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.342+0000 7f92bf7fe640 1 --2- 172.21.15.83:0/302376860 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92c0145dc0 0x7f92c00a98c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.342+0000 7f92bf7fe640 1 -- 172.21.15.83:0/302376860 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92ac0039b0 msgr2=0x7f92c0145680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.342+0000 7f92bf7fe640 1 --2- 172.21.15.83:0/302376860 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92ac0039b0 0x7f92c0145680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.342+0000 7f92bf7fe640 1 -- 172.21.15.83:0/302376860 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c01f5650 con 0x7f92c015a350 2024-03-06T07:26:57.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.342+0000 7f92bf7fe640 1 --2- 172.21.15.83:0/302376860 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92c015a350 0x7f92c015a7d0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f92b8009e90 tx=0x7f92b80115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:57.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.342+0000 7f92b77fe640 1 -- 172.21.15.83:0/302376860 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b8011e20 con 0x7f92c015a350 2024-03-06T07:26:57.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.342+0000 7f92b77fe640 1 -- 172.21.15.83:0/302376860 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92b801b070 con 0x7f92c015a350 2024-03-06T07:26:57.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.342+0000 7f92b77fe640 1 -- 172.21.15.83:0/302376860 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b8003990 con 0x7f92c015a350 2024-03-06T07:26:57.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.343+0000 7f92c5525640 1 -- 172.21.15.83:0/302376860 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92c01f5930 con 0x7f92c015a350 2024-03-06T07:26:57.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.343+0000 7f92c5525640 1 -- 172.21.15.83:0/302376860 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92c01f5ef0 con 0x7f92c015a350 2024-03-06T07:26:57.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.343+0000 7f92c5525640 1 -- 172.21.15.83:0/302376860 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92c0068050 con 0x7f92c015a350 2024-03-06T07:26:57.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.344+0000 7f92b77fe640 1 -- 172.21.15.83:0/302376860 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f92b8003b30 con 0x7f92c015a350 2024-03-06T07:26:57.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.344+0000 7f92b77fe640 1 --2- 172.21.15.83:0/302376860 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f92a006b4d0 0x7f92a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.344+0000 7f92b77fe640 1 -- 172.21.15.83:0/302376860 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f92b808aad0 con 0x7f92c015a350 2024-03-06T07:26:57.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.344+0000 7f92beffd640 1 --2- 172.21.15.83:0/302376860 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f92a006b4d0 0x7f92a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.345+0000 7f92beffd640 1 --2- 172.21.15.83:0/302376860 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f92a006b4d0 0x7f92a006d990 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f92b0034000 tx=0x7f92b0039040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:57.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.349+0000 7f92b77fe640 1 -- 172.21.15.83:0/302376860 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f92b8003e20 con 0x7f92c015a350 2024-03-06T07:26:57.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.685+0000 7f92c5525640 1 -- 172.21.15.83:0/302376860 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92c006ffa0 con 0x7f92c015a350 2024-03-06T07:26:57.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.686+0000 7f92b77fe640 1 -- 172.21.15.83:0/302376860 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+16749 (secure 0 0 0) 0x7f92b80598f0 con 0x7f92c015a350 2024-03-06T07:26:57.688 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:57.688 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":98,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:26:27.785066+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:17.275618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:18.028872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.83:6837/2924411033":"2024-03-07T07:26:27.785046+0000","172.21.15.83:6835/2924411033":"2024-03-07T07:26:27.785046+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:57.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.688+0000 7f92c5525640 1 -- 172.21.15.83:0/302376860 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f92a006b4d0 msgr2=0x7f92a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.688+0000 7f92c5525640 1 --2- 172.21.15.83:0/302376860 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f92a006b4d0 0x7f92a006d990 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f92b0034000 tx=0x7f92b0039040 comp rx=0 tx=0).stop 2024-03-06T07:26:57.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.688+0000 7f92c5525640 1 -- 172.21.15.83:0/302376860 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92c015a350 msgr2=0x7f92c015a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.688+0000 7f92c5525640 1 --2- 172.21.15.83:0/302376860 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92c015a350 0x7f92c015a7d0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f92b8009e90 tx=0x7f92b80115a0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.688+0000 7f92c5525640 1 -- 172.21.15.83:0/302376860 shutdown_connections 2024-03-06T07:26:57.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.688+0000 7f92c5525640 1 --2- 172.21.15.83:0/302376860 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f92a006b4d0 0x7f92a006d990 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.688+0000 7f92c5525640 1 --2- 172.21.15.83:0/302376860 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92c015a350 0x7f92c015a7d0 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.688+0000 7f92c5525640 1 --2- 172.21.15.83:0/302376860 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92c0145dc0 0x7f92c00a98c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.688+0000 7f92c5525640 1 --2- 172.21.15.83:0/302376860 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92ac0039b0 0x7f92c0145680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.688+0000 7f92c5525640 1 -- 172.21.15.83:0/302376860 >> 172.21.15.83:0/302376860 conn(0x7f92c0071690 msgr2=0x7f92c00cf6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:57.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.688+0000 7f92c5525640 1 -- 172.21.15.83:0/302376860 shutdown_connections 2024-03-06T07:26:57.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.688+0000 7f92c5525640 1 -- 172.21.15.83:0/302376860 wait complete. 2024-03-06T07:26:57.702 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:26:57.702 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:57.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.801+0000 7f420284e640 1 Processor -- start 2024-03-06T07:26:57.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.801+0000 7f420284e640 1 -- start start 2024-03-06T07:26:57.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.801+0000 7f420284e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f41fc15e150 0x7f41fc0cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.801+0000 7f420284e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f41fc0cfba0 0x7f41fc0d0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.801+0000 7f420284e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41fc0d1ea0 con 0x7f41fc0d0e90 2024-03-06T07:26:57.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.801+0000 7f420284e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41fc0d2ab0 con 0x7f41fc15e150 2024-03-06T07:26:57.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.801+0000 7f420284e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41fc0d2180 con 0x7f41fc0cfba0 2024-03-06T07:26:57.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.802+0000 7f41fbfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f41fc15e150 0x7f41fc0cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.802+0000 7f4200deb640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f41fc0d0e90 0x7f41fc0b9060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:55234/0 (socket says 172.21.15.83:55234) 2024-03-06T07:26:57.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.802+0000 7f4200deb640 1 -- 172.21.15.83:0/4169660363 learned_addr learned my addr 172.21.15.83:0/4169660363 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:57.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.802+0000 7f41fb7fe640 1 --2- 172.21.15.83:0/4169660363 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f41fc0cfba0 0x7f41fc0d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.802+0000 7f41fbfff640 1 -- 172.21.15.83:0/4169660363 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f41fc0cfba0 msgr2=0x7f41fc0d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:57.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.802+0000 7f41fbfff640 1 --2- 172.21.15.83:0/4169660363 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f41fc0cfba0 0x7f41fc0d0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.802+0000 7f41fbfff640 1 -- 172.21.15.83:0/4169660363 >> v1:172.21.15.83:6789/0 conn(0x7f41fc0d0e90 legacy=0x7f41fc0b9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.802+0000 7f41fbfff640 1 -- 172.21.15.83:0/4169660363 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41fc0d2460 con 0x7f41fc15e150 2024-03-06T07:26:57.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.803+0000 7f41fbfff640 1 --2- 172.21.15.83:0/4169660363 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41fc15e150 0x7f41fc0cf660 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f41ec0097a0 tx=0x7f41ec02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d7c705db37b0242 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:57.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.803+0000 7f41faffd640 1 -- 172.21.15.83:0/4169660363 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41ec0315a0 con 0x7f41fc15e150 2024-03-06T07:26:57.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.803+0000 7f41faffd640 1 -- 172.21.15.83:0/4169660363 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41ec037070 con 0x7f41fc15e150 2024-03-06T07:26:57.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.803+0000 7f41faffd640 1 -- 172.21.15.83:0/4169660363 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41ec031960 con 0x7f41fc15e150 2024-03-06T07:26:57.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.803+0000 7f420284e640 1 -- 172.21.15.83:0/4169660363 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41fc15e150 msgr2=0x7f41fc0cf660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.803+0000 7f420284e640 1 --2- 172.21.15.83:0/4169660363 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41fc15e150 0x7f41fc0cf660 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f41ec0097a0 tx=0x7f41ec02b840 comp rx=0 tx=0).stop 2024-03-06T07:26:57.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.803+0000 7f420284e640 1 -- 172.21.15.83:0/4169660363 shutdown_connections 2024-03-06T07:26:57.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.803+0000 7f420284e640 1 --2- 172.21.15.83:0/4169660363 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f41fc0cfba0 0x7f41fc0d0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.803+0000 7f420284e640 1 --2- 172.21.15.83:0/4169660363 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41fc15e150 0x7f41fc0cf660 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.803+0000 7f420284e640 1 -- 172.21.15.83:0/4169660363 >> 172.21.15.83:0/4169660363 conn(0x7f41fc09d230 msgr2=0x7f41fc0d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:57.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.803+0000 7f420284e640 1 -- 172.21.15.83:0/4169660363 shutdown_connections 2024-03-06T07:26:57.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.803+0000 7f420284e640 1 -- 172.21.15.83:0/4169660363 wait complete. 2024-03-06T07:26:57.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.805+0000 7f420284e640 1 Processor -- start 2024-03-06T07:26:57.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.805+0000 7f420284e640 1 -- start start 2024-03-06T07:26:57.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.805+0000 7f420284e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f41fc0cfba0 0x7f41fc15a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.805+0000 7f420284e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41fc0d0e90 0x7f41fc09e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.805+0000 7f420284e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f41fc15e150 0x7f41fc0cd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.805+0000 7f420284e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41fc0d1b30 con 0x7f41fc0d0e90 2024-03-06T07:26:57.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.805+0000 7f420284e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41fc0d2f40 con 0x7f41fc15e150 2024-03-06T07:26:57.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.805+0000 7f420284e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41fc15e530 con 0x7f41fc0cfba0 2024-03-06T07:26:57.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.805+0000 7f41fb7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41fc0d0e90 0x7f41fc09e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.805+0000 7f41fb7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41fc0d0e90 0x7f41fc09e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51410/0 (socket says 172.21.15.83:51410) 2024-03-06T07:26:57.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.805+0000 7f41fb7fe640 1 -- 172.21.15.83:0/3783132257 learned_addr learned my addr 172.21.15.83:0/3783132257 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:57.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.806+0000 7f41fbfff640 1 --2- 172.21.15.83:0/3783132257 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f41fc0cfba0 0x7f41fc15a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.806+0000 7f4200deb640 1 --2- 172.21.15.83:0/3783132257 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f41fc15e150 0x7f41fc0cd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.806+0000 7f41fb7fe640 1 -- 172.21.15.83:0/3783132257 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f41fc0cfba0 msgr2=0x7f41fc15a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.806+0000 7f41fb7fe640 1 --2- 172.21.15.83:0/3783132257 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f41fc0cfba0 0x7f41fc15a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.806+0000 7f41fb7fe640 1 -- 172.21.15.83:0/3783132257 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f41fc15e150 msgr2=0x7f41fc0cd2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.806+0000 7f41fb7fe640 1 --2- 172.21.15.83:0/3783132257 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f41fc15e150 0x7f41fc0cd2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.806+0000 7f41fb7fe640 1 -- 172.21.15.83:0/3783132257 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41fc0a0dc0 con 0x7f41fc0d0e90 2024-03-06T07:26:57.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.806+0000 7f4200deb640 1 --2- 172.21.15.83:0/3783132257 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f41fc15e150 0x7f41fc0cd2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:57.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.806+0000 7f41fb7fe640 1 --2- 172.21.15.83:0/3783132257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41fc0d0e90 0x7f41fc09e640 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f41f000d800 tx=0x7f41f000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:57.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.806+0000 7f41f8ff9640 1 -- 172.21.15.83:0/3783132257 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f000b6c0 con 0x7f41fc0d0e90 2024-03-06T07:26:57.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.806+0000 7f41f8ff9640 1 -- 172.21.15.83:0/3783132257 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41f0019070 con 0x7f41fc0d0e90 2024-03-06T07:26:57.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.806+0000 7f420284e640 1 -- 172.21.15.83:0/3783132257 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41fc0a10a0 con 0x7f41fc0d0e90 2024-03-06T07:26:57.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.806+0000 7f41f8ff9640 1 -- 172.21.15.83:0/3783132257 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f000ba80 con 0x7f41fc0d0e90 2024-03-06T07:26:57.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.806+0000 7f420284e640 1 -- 172.21.15.83:0/3783132257 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41fc0a1660 con 0x7f41fc0d0e90 2024-03-06T07:26:57.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.807+0000 7f420284e640 1 -- 172.21.15.83:0/3783132257 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41fc069ee0 con 0x7f41fc0d0e90 2024-03-06T07:26:57.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.808+0000 7f41f8ff9640 1 -- 172.21.15.83:0/3783132257 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f41f000bc20 con 0x7f41fc0d0e90 2024-03-06T07:26:57.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.808+0000 7f41f8ff9640 1 --2- 172.21.15.83:0/3783132257 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f41e806b4d0 0x7f41e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.808+0000 7f41f8ff9640 1 -- 172.21.15.83:0/3783132257 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f41f008a5e0 con 0x7f41fc0d0e90 2024-03-06T07:26:57.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.808+0000 7f41fbfff640 1 --2- 172.21.15.83:0/3783132257 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f41e806b4d0 0x7f41e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.809+0000 7f41fbfff640 1 --2- 172.21.15.83:0/3783132257 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f41e806b4d0 0x7f41e806d990 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f41ec036000 tx=0x7f41ec03d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:57.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:57.813+0000 7f41f8ff9640 1 -- 172.21.15.83:0/3783132257 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f41f0059490 con 0x7f41fc0d0e90 2024-03-06T07:26:58.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.028+0000 7f420284e640 1 -- 172.21.15.83:0/3783132257 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41fc0d4670 con 0x7f41fc0d0e90 2024-03-06T07:26:58.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.028+0000 7f41f8ff9640 1 -- 172.21.15.83:0/3783132257 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f41f0058d90 con 0x7f41fc0d0e90 2024-03-06T07:26:58.030 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:58.030 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":53,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6023,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/601236077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":601236077},{"type":"v1","addr":"172.21.15.183:6833","nonce":601236077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6029,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1435055961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1435055961},{"type":"v1","addr":"172.21.15.183:6835","nonce":1435055961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":7572,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/2726045966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2726045966},{"type":"v1","addr":"172.21.15.83:6836","nonce":2726045966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.064335+0000","modified":"2024-03-06T07:26:32.814889+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7575":{"gid":7575,"name":"a","rank":0,"incarnation":49,"state":"up:active","state_seq":5,"addr":"172.21.15.83:6837/3254575964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":3254575964},{"type":"v1","addr":"172.21.15.83:6837","nonce":3254575964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7575,"qdb_cluster":[7575]},"id":7}]} 2024-03-06T07:26:58.031 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 53 2024-03-06T07:26:58.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.030+0000 7f420284e640 1 -- 172.21.15.83:0/3783132257 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f41e806b4d0 msgr2=0x7f41e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.030+0000 7f420284e640 1 --2- 172.21.15.83:0/3783132257 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f41e806b4d0 0x7f41e806d990 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f41ec036000 tx=0x7f41ec03d040 comp rx=0 tx=0).stop 2024-03-06T07:26:58.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.030+0000 7f420284e640 1 -- 172.21.15.83:0/3783132257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41fc0d0e90 msgr2=0x7f41fc09e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.030+0000 7f420284e640 1 --2- 172.21.15.83:0/3783132257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41fc0d0e90 0x7f41fc09e640 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f41f000d800 tx=0x7f41f000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.030+0000 7f420284e640 1 -- 172.21.15.83:0/3783132257 shutdown_connections 2024-03-06T07:26:58.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.030+0000 7f420284e640 1 --2- 172.21.15.83:0/3783132257 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f41e806b4d0 0x7f41e806d990 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.030+0000 7f420284e640 1 --2- 172.21.15.83:0/3783132257 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f41fc15e150 0x7f41fc0cd2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.030+0000 7f420284e640 1 --2- 172.21.15.83:0/3783132257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41fc0d0e90 0x7f41fc09e640 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.030+0000 7f420284e640 1 --2- 172.21.15.83:0/3783132257 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f41fc0cfba0 0x7f41fc15a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.030+0000 7f420284e640 1 -- 172.21.15.83:0/3783132257 >> 172.21.15.83:0/3783132257 conn(0x7f41fc09d230 msgr2=0x7f41fc06d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:58.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.030+0000 7f420284e640 1 -- 172.21.15.83:0/3783132257 shutdown_connections 2024-03-06T07:26:58.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.030+0000 7f420284e640 1 -- 172.21.15.83:0/3783132257 wait complete. 2024-03-06T07:26:58.044 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:58.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.143+0000 7f5143c40640 1 Processor -- start 2024-03-06T07:26:58.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.144+0000 7f5143c40640 1 -- start start 2024-03-06T07:26:58.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.145+0000 7f5143c40640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f513c161a80 0x7f513c0d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.145+0000 7f5143c40640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f513c0d25a0 0x7f513c0d29f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.145+0000 7f5143c40640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f513c161e60 con 0x7f513c0b5010 2024-03-06T07:26:58.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.145+0000 7f5143c40640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f513c0b1250 con 0x7f513c0d25a0 2024-03-06T07:26:58.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.145+0000 7f5143c40640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f513c0d2f30 con 0x7f513c161a80 2024-03-06T07:26:58.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.145+0000 7f51411db640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f513c0b5010 0x7f513c0b0b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:55240/0 (socket says 172.21.15.83:55240) 2024-03-06T07:26:58.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.145+0000 7f51411db640 1 -- 172.21.15.83:0/910925038 learned_addr learned my addr 172.21.15.83:0/910925038 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:58.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.145+0000 7f51421dd640 1 --2- 172.21.15.83:0/910925038 >> v2:172.21.15.83:3300/0 conn(0x7f513c0d25a0 0x7f513c0d29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.145+0000 7f51419dc640 1 --2- 172.21.15.83:0/910925038 >> v2:172.21.15.183:3300/0 conn(0x7f513c161a80 0x7f513c0d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.145+0000 7f51421dd640 1 -- 172.21.15.83:0/910925038 >> v2:172.21.15.183:3300/0 conn(0x7f513c161a80 msgr2=0x7f513c0d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:58.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.145+0000 7f51421dd640 1 --2- 172.21.15.83:0/910925038 >> v2:172.21.15.183:3300/0 conn(0x7f513c161a80 0x7f513c0d1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.145+0000 7f51421dd640 1 -- 172.21.15.83:0/910925038 >> v1:172.21.15.83:6789/0 conn(0x7f513c0b5010 legacy=0x7f513c0b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.145+0000 7f51421dd640 1 -- 172.21.15.83:0/910925038 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f513c0d1470 con 0x7f513c0d25a0 2024-03-06T07:26:58.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.145+0000 7f51409da640 1 -- 172.21.15.83:0/910925038 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2656356611 0 0) 0x7f512c001110 con 0x7f513c0b5010 2024-03-06T07:26:58.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.146+0000 7f51421dd640 1 --2- 172.21.15.83:0/910925038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f513c0d25a0 0x7f513c0d29f0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f513c0b6bb0 tx=0x7f513802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcb5b54c08502152 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:58.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.146+0000 7f51409da640 1 -- 172.21.15.83:0/910925038 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5138039070 con 0x7f513c0d25a0 2024-03-06T07:26:58.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.146+0000 7f51409da640 1 -- 172.21.15.83:0/910925038 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f513802cbf0 con 0x7f513c0d25a0 2024-03-06T07:26:58.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.146+0000 7f51409da640 1 -- 172.21.15.83:0/910925038 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5138034440 con 0x7f513c0d25a0 2024-03-06T07:26:58.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.147+0000 7f5143c40640 1 -- 172.21.15.83:0/910925038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f513c0d25a0 msgr2=0x7f513c0d29f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.147+0000 7f5143c40640 1 --2- 172.21.15.83:0/910925038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f513c0d25a0 0x7f513c0d29f0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f513c0b6bb0 tx=0x7f513802be40 comp rx=0 tx=0).stop 2024-03-06T07:26:58.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.147+0000 7f5143c40640 1 -- 172.21.15.83:0/910925038 shutdown_connections 2024-03-06T07:26:58.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.147+0000 7f5143c40640 1 --2- 172.21.15.83:0/910925038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f513c0d25a0 0x7f513c0d29f0 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.147+0000 7f5143c40640 1 --2- 172.21.15.83:0/910925038 >> v2:172.21.15.183:3300/0 conn(0x7f513c161a80 0x7f513c0d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.147+0000 7f5143c40640 1 -- 172.21.15.83:0/910925038 >> 172.21.15.83:0/910925038 conn(0x7f513c0d3fc0 msgr2=0x7f513c0d43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:58.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.147+0000 7f5143c40640 1 -- 172.21.15.83:0/910925038 shutdown_connections 2024-03-06T07:26:58.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.147+0000 7f5143c40640 1 -- 172.21.15.83:0/910925038 wait complete. 2024-03-06T07:26:58.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.148+0000 7f5143c40640 1 Processor -- start 2024-03-06T07:26:58.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.148+0000 7f5143c40640 1 -- start start 2024-03-06T07:26:58.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.149+0000 7f5143c40640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f513c0b5010 0x7f513c15ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.149+0000 7f5143c40640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f513c0d25a0 0x7f513c0b89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.149+0000 7f5143c40640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f513c161a80 0x7f513c0b75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.149+0000 7f5143c40640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f513c0b3ac0 con 0x7f513c0b5010 2024-03-06T07:26:58.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.149+0000 7f5143c40640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f513c0b3e30 con 0x7f513c161a80 2024-03-06T07:26:58.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.149+0000 7f5143c40640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f513c0d1090 con 0x7f513c0d25a0 2024-03-06T07:26:58.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.149+0000 7f51419dc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f513c0b5010 0x7f513c15ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.149+0000 7f51419dc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f513c0b5010 0x7f513c15ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51424/0 (socket says 172.21.15.83:51424) 2024-03-06T07:26:58.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.149+0000 7f51419dc640 1 -- 172.21.15.83:0/2743323744 learned_addr learned my addr 172.21.15.83:0/2743323744 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:58.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.149+0000 7f51419dc640 1 -- 172.21.15.83:0/2743323744 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f513c0d25a0 msgr2=0x7f513c0b89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.149+0000 7f51419dc640 1 --2- 172.21.15.83:0/2743323744 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f513c0d25a0 0x7f513c0b89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.149+0000 7f51419dc640 1 -- 172.21.15.83:0/2743323744 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f513c161a80 msgr2=0x7f513c0b75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.149+0000 7f51419dc640 1 --2- 172.21.15.83:0/2743323744 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f513c161a80 0x7f513c0b75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.149+0000 7f51419dc640 1 -- 172.21.15.83:0/2743323744 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f513c0cb230 con 0x7f513c0b5010 2024-03-06T07:26:58.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.149+0000 7f51419dc640 1 --2- 172.21.15.83:0/2743323744 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f513c0b5010 0x7f513c15ad90 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f513400d5f0 tx=0x7f513400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:58.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.149+0000 7f51327fc640 1 -- 172.21.15.83:0/2743323744 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51340125f0 con 0x7f513c0b5010 2024-03-06T07:26:58.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.149+0000 7f51327fc640 1 -- 172.21.15.83:0/2743323744 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5134019070 con 0x7f513c0b5010 2024-03-06T07:26:58.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.149+0000 7f5143c40640 1 -- 172.21.15.83:0/2743323744 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f513c0cb4d0 con 0x7f513c0b5010 2024-03-06T07:26:58.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.149+0000 7f51327fc640 1 -- 172.21.15.83:0/2743323744 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51340129d0 con 0x7f513c0b5010 2024-03-06T07:26:58.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.150+0000 7f5143c40640 1 -- 172.21.15.83:0/2743323744 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f513c0cba90 con 0x7f513c0b5010 2024-03-06T07:26:58.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.150+0000 7f5143c40640 1 -- 172.21.15.83:0/2743323744 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f513c068050 con 0x7f513c0b5010 2024-03-06T07:26:58.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.151+0000 7f51327fc640 1 -- 172.21.15.83:0/2743323744 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5134012b70 con 0x7f513c0b5010 2024-03-06T07:26:58.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.151+0000 7f51327fc640 1 --2- 172.21.15.83:0/2743323744 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f512006b500 0x7f512006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.152+0000 7f51411db640 1 --2- 172.21.15.83:0/2743323744 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f512006b500 0x7f512006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.152+0000 7f51327fc640 1 -- 172.21.15.83:0/2743323744 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f513408a5a0 con 0x7f513c0b5010 2024-03-06T07:26:58.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.152+0000 7f51411db640 1 --2- 172.21.15.83:0/2743323744 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f512006b500 0x7f512006d9c0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f513c0ce090 tx=0x7f512c001310 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:58.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.156+0000 7f51327fc640 1 -- 172.21.15.83:0/2743323744 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5134059450 con 0x7f513c0b5010 2024-03-06T07:26:58.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.370+0000 7f5143c40640 1 -- 172.21.15.83:0/2743323744 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f513c0d2380 con 0x7f513c0b5010 2024-03-06T07:26:58.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.371+0000 7f51327fc640 1 -- 172.21.15.83:0/2743323744 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f5134058d50 con 0x7f513c0b5010 2024-03-06T07:26:58.373 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:58.373 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":53,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6023,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/601236077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":601236077},{"type":"v1","addr":"172.21.15.183:6833","nonce":601236077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6029,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1435055961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1435055961},{"type":"v1","addr":"172.21.15.183:6835","nonce":1435055961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":7572,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/2726045966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2726045966},{"type":"v1","addr":"172.21.15.83:6836","nonce":2726045966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.064335+0000","modified":"2024-03-06T07:26:32.814889+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7575":{"gid":7575,"name":"a","rank":0,"incarnation":49,"state":"up:active","state_seq":5,"addr":"172.21.15.83:6837/3254575964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":3254575964},{"type":"v1","addr":"172.21.15.83:6837","nonce":3254575964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7575,"qdb_cluster":[7575]},"id":7}]} 2024-03-06T07:26:58.373 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 53 2024-03-06T07:26:58.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.373+0000 7f5143c40640 1 -- 172.21.15.83:0/2743323744 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f512006b500 msgr2=0x7f512006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.373+0000 7f5143c40640 1 --2- 172.21.15.83:0/2743323744 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f512006b500 0x7f512006d9c0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f513c0ce090 tx=0x7f512c001310 comp rx=0 tx=0).stop 2024-03-06T07:26:58.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.373+0000 7f5143c40640 1 -- 172.21.15.83:0/2743323744 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f513c0b5010 msgr2=0x7f513c15ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.373+0000 7f5143c40640 1 --2- 172.21.15.83:0/2743323744 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f513c0b5010 0x7f513c15ad90 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f513400d5f0 tx=0x7f513400dab0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.373+0000 7f5143c40640 1 -- 172.21.15.83:0/2743323744 shutdown_connections 2024-03-06T07:26:58.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.373+0000 7f5143c40640 1 --2- 172.21.15.83:0/2743323744 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f512006b500 0x7f512006d9c0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.373+0000 7f5143c40640 1 --2- 172.21.15.83:0/2743323744 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f513c161a80 0x7f513c0b75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.373+0000 7f5143c40640 1 --2- 172.21.15.83:0/2743323744 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f513c0d25a0 0x7f513c0b89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.373+0000 7f5143c40640 1 --2- 172.21.15.83:0/2743323744 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f513c0b5010 0x7f513c15ad90 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.374+0000 7f5143c40640 1 -- 172.21.15.83:0/2743323744 >> 172.21.15.83:0/2743323744 conn(0x7f513c0d3fc0 msgr2=0x7f513c06c0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:58.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.374+0000 7f5143c40640 1 -- 172.21.15.83:0/2743323744 shutdown_connections 2024-03-06T07:26:58.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.374+0000 7f5143c40640 1 -- 172.21.15.83:0/2743323744 wait complete. 2024-03-06T07:26:58.387 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:58.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.489+0000 7f1fe4081640 1 Processor -- start 2024-03-06T07:26:58.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.489+0000 7f1fe4081640 1 -- start start 2024-03-06T07:26:58.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.490+0000 7f1fe4081640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1fdc0d0950 0x7f1fdc0d0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.490+0000 7f1fe4081640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fdc0d1ea0 con 0x7f1fdc0b9060 2024-03-06T07:26:58.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.490+0000 7f1fe4081640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fdc0d2180 con 0x7f1fdc15e150 2024-03-06T07:26:58.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.490+0000 7f1fe4081640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fdc0d12f0 con 0x7f1fdc0d0950 2024-03-06T07:26:58.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.490+0000 7f1fe161c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1fdc0d0950 0x7f1fdc0d0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.490+0000 7f1fe261e640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f1fdc0b9060 0x7f1fdc0b94d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:55242/0 (socket says 172.21.15.83:55242) 2024-03-06T07:26:58.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.490+0000 7f1fe261e640 1 -- 172.21.15.83:0/2162073307 learned_addr learned my addr 172.21.15.83:0/2162073307 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:58.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.491+0000 7f1fe161c640 1 -- 172.21.15.83:0/2162073307 >> v1:172.21.15.183:6789/0 conn(0x7f1fdc15e150 legacy=0x7f1fdc0cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:58.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.491+0000 7f1fe161c640 1 -- 172.21.15.83:0/2162073307 >> v1:172.21.15.83:6789/0 conn(0x7f1fdc0b9060 legacy=0x7f1fdc0b94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.491+0000 7f1fe161c640 1 -- 172.21.15.83:0/2162073307 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fdc0d2460 con 0x7f1fdc0d0950 2024-03-06T07:26:58.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.491+0000 7f1fe0e1b640 1 -- 172.21.15.83:0/2162073307 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 738625835 0 0) 0x7f1fdc0d1ea0 con 0x7f1fdc0b9060 2024-03-06T07:26:58.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.491+0000 7f1fe161c640 1 --2- 172.21.15.83:0/2162073307 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1fdc0d0950 0x7f1fdc0d0db0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc0097a0 tx=0x7f1fcc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dcd71e91eb150f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:58.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.491+0000 7f1fe0e1b640 1 -- 172.21.15.83:0/2162073307 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fcc0315a0 con 0x7f1fdc0d0950 2024-03-06T07:26:58.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.491+0000 7f1fe0e1b640 1 -- 172.21.15.83:0/2162073307 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fcc037070 con 0x7f1fdc0d0950 2024-03-06T07:26:58.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.491+0000 7f1fe0e1b640 1 -- 172.21.15.83:0/2162073307 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fcc031960 con 0x7f1fdc0d0950 2024-03-06T07:26:58.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.492+0000 7f1fe4081640 1 -- 172.21.15.83:0/2162073307 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1fdc0d0950 msgr2=0x7f1fdc0d0db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.492+0000 7f1fe4081640 1 --2- 172.21.15.83:0/2162073307 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1fdc0d0950 0x7f1fdc0d0db0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc0097a0 tx=0x7f1fcc02b840 comp rx=0 tx=0).stop 2024-03-06T07:26:58.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.492+0000 7f1fe4081640 1 -- 172.21.15.83:0/2162073307 shutdown_connections 2024-03-06T07:26:58.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.492+0000 7f1fe4081640 1 --2- 172.21.15.83:0/2162073307 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1fdc0d0950 0x7f1fdc0d0db0 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.492+0000 7f1fe4081640 1 -- 172.21.15.83:0/2162073307 >> 172.21.15.83:0/2162073307 conn(0x7f1fdc09d230 msgr2=0x7f1fdc0d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:58.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.492+0000 7f1fe4081640 1 -- 172.21.15.83:0/2162073307 shutdown_connections 2024-03-06T07:26:58.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.492+0000 7f1fe4081640 1 -- 172.21.15.83:0/2162073307 wait complete. 2024-03-06T07:26:58.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.493+0000 7f1fe4081640 1 Processor -- start 2024-03-06T07:26:58.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.493+0000 7f1fe4081640 1 -- start start 2024-03-06T07:26:58.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.494+0000 7f1fe4081640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1fdc0b9060 0x7f1fdc1514d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.494+0000 7f1fe4081640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1fdc0d0950 0x7f1fdc1f40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.494+0000 7f1fe4081640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1fdc15e150 0x7f1fdc0cdb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.494+0000 7f1fe4081640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fdc0d2f40 con 0x7f1fdc0b9060 2024-03-06T07:26:58.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.494+0000 7f1fe4081640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fdc0d2ab0 con 0x7f1fdc0d0950 2024-03-06T07:26:58.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.494+0000 7f1fe4081640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fdc0b7dc0 con 0x7f1fdc15e150 2024-03-06T07:26:58.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.494+0000 7f1fe1e1d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1fdc0b9060 0x7f1fdc1514d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.494+0000 7f1fe1e1d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1fdc0b9060 0x7f1fdc1514d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51442/0 (socket says 172.21.15.83:51442) 2024-03-06T07:26:58.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.494+0000 7f1fe1e1d640 1 -- 172.21.15.83:0/846627186 learned_addr learned my addr 172.21.15.83:0/846627186 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:58.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.494+0000 7f1fe161c640 1 --2- 172.21.15.83:0/846627186 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1fdc0d0950 0x7f1fdc1f40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.494+0000 7f1fe1e1d640 1 -- 172.21.15.83:0/846627186 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1fdc15e150 msgr2=0x7f1fdc0cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.494+0000 7f1fe1e1d640 1 --2- 172.21.15.83:0/846627186 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1fdc15e150 0x7f1fdc0cdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.494+0000 7f1fe1e1d640 1 -- 172.21.15.83:0/846627186 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1fdc0d0950 msgr2=0x7f1fdc1f40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.494+0000 7f1fe1e1d640 1 --2- 172.21.15.83:0/846627186 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1fdc0d0950 0x7f1fdc1f40f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.494+0000 7f1fe1e1d640 1 -- 172.21.15.83:0/846627186 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fdc0c8c40 con 0x7f1fdc0b9060 2024-03-06T07:26:58.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.494+0000 7f1fe1e1d640 1 --2- 172.21.15.83:0/846627186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1fdc0b9060 0x7f1fdc1514d0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f1fd800b410 tx=0x7f1fd800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:58.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.495+0000 7f1fd27fc640 1 -- 172.21.15.83:0/846627186 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd8015020 con 0x7f1fdc0b9060 2024-03-06T07:26:58.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.495+0000 7f1fd27fc640 1 -- 172.21.15.83:0/846627186 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd8011830 con 0x7f1fdc0b9060 2024-03-06T07:26:58.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.495+0000 7f1fd27fc640 1 -- 172.21.15.83:0/846627186 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd8011c20 con 0x7f1fdc0b9060 2024-03-06T07:26:58.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.495+0000 7f1fe4081640 1 -- 172.21.15.83:0/846627186 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fdc0c8f20 con 0x7f1fdc0b9060 2024-03-06T07:26:58.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.495+0000 7f1fe4081640 1 -- 172.21.15.83:0/846627186 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fdc0c94e0 con 0x7f1fdc0b9060 2024-03-06T07:26:58.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.496+0000 7f1fe4081640 1 -- 172.21.15.83:0/846627186 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fdc069ee0 con 0x7f1fdc0b9060 2024-03-06T07:26:58.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.496+0000 7f1fd27fc640 1 -- 172.21.15.83:0/846627186 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1fd8018020 con 0x7f1fdc0b9060 2024-03-06T07:26:58.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.496+0000 7f1fd27fc640 1 --2- 172.21.15.83:0/846627186 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1fc006b4d0 0x7f1fc006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.497+0000 7f1fd27fc640 1 -- 172.21.15.83:0/846627186 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f1fd808a170 con 0x7f1fdc0b9060 2024-03-06T07:26:58.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.497+0000 7f1fe161c640 1 --2- 172.21.15.83:0/846627186 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1fc006b4d0 0x7f1fc006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.497+0000 7f1fe161c640 1 --2- 172.21.15.83:0/846627186 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1fc006b4d0 0x7f1fc006d990 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc036000 tx=0x7f1fcc008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:58.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.505+0000 7f1fd27fc640 1 -- 172.21.15.83:0/846627186 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1fd801f020 con 0x7f1fdc0b9060 2024-03-06T07:26:58.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.680+0000 7f1fe4081640 1 -- 172.21.15.83:0/846627186 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1fdc0d4670 con 0x7f1fdc0b9060 2024-03-06T07:26:58.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.681+0000 7f1fd27fc640 1 -- 172.21.15.83:0/846627186 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+16749 (secure 0 0 0) 0x7f1fd80172c0 con 0x7f1fdc0b9060 2024-03-06T07:26:58.683 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:26:58.683 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":98,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:26:27.785066+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:17.275618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:18.028872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.83:6837/2924411033":"2024-03-07T07:26:27.785046+0000","172.21.15.83:6835/2924411033":"2024-03-07T07:26:27.785046+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:58.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.683+0000 7f1fe4081640 1 -- 172.21.15.83:0/846627186 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1fc006b4d0 msgr2=0x7f1fc006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.683+0000 7f1fe4081640 1 --2- 172.21.15.83:0/846627186 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1fc006b4d0 0x7f1fc006d990 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc036000 tx=0x7f1fcc008030 comp rx=0 tx=0).stop 2024-03-06T07:26:58.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.683+0000 7f1fe4081640 1 -- 172.21.15.83:0/846627186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1fdc0b9060 msgr2=0x7f1fdc1514d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.683+0000 7f1fe4081640 1 --2- 172.21.15.83:0/846627186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1fdc0b9060 0x7f1fdc1514d0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f1fd800b410 tx=0x7f1fd800b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.683+0000 7f1fe4081640 1 -- 172.21.15.83:0/846627186 shutdown_connections 2024-03-06T07:26:58.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.683+0000 7f1fe4081640 1 --2- 172.21.15.83:0/846627186 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1fc006b4d0 0x7f1fc006d990 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.683+0000 7f1fe4081640 1 --2- 172.21.15.83:0/846627186 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1fdc15e150 0x7f1fdc0cdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.683+0000 7f1fe4081640 1 --2- 172.21.15.83:0/846627186 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1fdc0d0950 0x7f1fdc1f40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.683+0000 7f1fe4081640 1 --2- 172.21.15.83:0/846627186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1fdc0b9060 0x7f1fdc1514d0 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.683+0000 7f1fe4081640 1 -- 172.21.15.83:0/846627186 >> 172.21.15.83:0/846627186 conn(0x7f1fdc09d230 msgr2=0x7f1fdc06bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:58.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.683+0000 7f1fe4081640 1 -- 172.21.15.83:0/846627186 shutdown_connections 2024-03-06T07:26:58.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.683+0000 7f1fe4081640 1 -- 172.21.15.83:0/846627186 wait complete. 2024-03-06T07:26:58.696 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:26:58.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.833+0000 7f97e649d640 1 Processor -- start 2024-03-06T07:26:58.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.833+0000 7f97e649d640 1 -- start start 2024-03-06T07:26:58.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.834+0000 7f97e649d640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f97e015e150 0x7f97e00cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.834+0000 7f97e649d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97e00d1ea0 con 0x7f97e00d1060 2024-03-06T07:26:58.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.834+0000 7f97e649d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97e00d2180 con 0x7f97e00cfba0 2024-03-06T07:26:58.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.834+0000 7f97e649d640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97e00d2ab0 con 0x7f97e015e150 2024-03-06T07:26:58.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.834+0000 7f97e4a3a640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f97e00d1060 0x7f97e00b9060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:55254/0 (socket says 172.21.15.83:55254) 2024-03-06T07:26:58.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.834+0000 7f97e4a3a640 1 -- 172.21.15.83:0/3681202936 learned_addr learned my addr 172.21.15.83:0/3681202936 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:58.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.834+0000 7f97dffff640 1 --2- 172.21.15.83:0/3681202936 >> v2:172.21.15.183:3300/0 conn(0x7f97e015e150 0x7f97e00cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.834+0000 7f97deffd640 1 -- 172.21.15.83:0/3681202936 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3601504868 0 0) 0x7f97e00d1ea0 con 0x7f97e00d1060 2024-03-06T07:26:58.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.834+0000 7f97deffd640 1 -- 172.21.15.83:0/3681202936 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97c8003520 con 0x7f97e00d1060 2024-03-06T07:26:58.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.834+0000 7f97deffd640 1 -- 172.21.15.83:0/3681202936 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3325134060 0 0) 0x7f97c8003520 con 0x7f97e00d1060 2024-03-06T07:26:58.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.834+0000 7f97deffd640 1 -- 172.21.15.83:0/3681202936 >> v2:172.21.15.183:3300/0 conn(0x7f97e015e150 msgr2=0x7f97e00cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:58.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.834+0000 7f97deffd640 1 --2- 172.21.15.83:0/3681202936 >> v2:172.21.15.183:3300/0 conn(0x7f97e015e150 0x7f97e00cf660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.834+0000 7f97deffd640 1 -- 172.21.15.83:0/3681202936 >> v1:172.21.15.183:6789/0 conn(0x7f97e00cfba0 legacy=0x7f97e00d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.834+0000 7f97deffd640 1 -- 172.21.15.83:0/3681202936 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e00d2460 con 0x7f97e00d1060 2024-03-06T07:26:58.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.834+0000 7f97deffd640 1 -- 172.21.15.83:0/3681202936 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f97d80019f0 con 0x7f97e00d1060 2024-03-06T07:26:58.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.834+0000 7f97deffd640 1 -- 172.21.15.83:0/3681202936 >> v1:172.21.15.83:6789/0 conn(0x7f97e00d1060 legacy=0x7f97e00b9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.835+0000 7f97deffd640 1 --2- 172.21.15.83:0/3681202936 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97c80039b0 0x7f97c8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.835+0000 7f97deffd640 1 --2- 172.21.15.83:0/3681202936 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97c8004800 0x7f97c8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.835+0000 7f97deffd640 1 --2- 172.21.15.83:0/3681202936 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97c80051c0 0x7f97c800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.835+0000 7f97deffd640 1 -- 172.21.15.83:0/3681202936 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e00d2ab0 con 0x7f97c8004800 2024-03-06T07:26:58.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.835+0000 7f97deffd640 1 -- 172.21.15.83:0/3681202936 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c800c160 con 0x7f97c80039b0 2024-03-06T07:26:58.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.835+0000 7f97deffd640 1 -- 172.21.15.83:0/3681202936 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97c800c360 con 0x7f97c80051c0 2024-03-06T07:26:58.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.835+0000 7f97df7fe640 1 --2- 172.21.15.83:0/3681202936 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97c8004800 0x7f97c8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.835+0000 7f97dffff640 1 --2- 172.21.15.83:0/3681202936 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97c80039b0 0x7f97c8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.835+0000 7f97e4a3a640 1 --2- 172.21.15.83:0/3681202936 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97c80051c0 0x7f97c800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.835+0000 7f97df7fe640 1 -- 172.21.15.83:0/3681202936 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97c80051c0 msgr2=0x7f97c800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:58.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.835+0000 7f97df7fe640 1 --2- 172.21.15.83:0/3681202936 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97c80051c0 0x7f97c800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.835+0000 7f97df7fe640 1 -- 172.21.15.83:0/3681202936 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97c80039b0 msgr2=0x7f97c8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:58.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.835+0000 7f97df7fe640 1 --2- 172.21.15.83:0/3681202936 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97c80039b0 0x7f97c8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.835+0000 7f97df7fe640 1 -- 172.21.15.83:0/3681202936 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c8003520 con 0x7f97c8004800 2024-03-06T07:26:58.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.835+0000 7f97e4a3a640 1 -- 172.21.15.83:0/3681202936 reap_dead start 2024-03-06T07:26:58.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.836+0000 7f97df7fe640 1 --2- 172.21.15.83:0/3681202936 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97c8004800 0x7f97c8004c80 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f97d0001750 tx=0x7f97c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb5f33da6111a0a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:58.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.836+0000 7f97deffd640 1 -- 172.21.15.83:0/3681202936 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d0001370 con 0x7f97c8004800 2024-03-06T07:26:58.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.836+0000 7f97deffd640 1 -- 172.21.15.83:0/3681202936 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97d00052d0 con 0x7f97c8004800 2024-03-06T07:26:58.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.836+0000 7f97deffd640 1 -- 172.21.15.83:0/3681202936 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d0003830 con 0x7f97c8004800 2024-03-06T07:26:58.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.836+0000 7f97e649d640 1 -- 172.21.15.83:0/3681202936 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97c8004800 msgr2=0x7f97c8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.836+0000 7f97e649d640 1 --2- 172.21.15.83:0/3681202936 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97c8004800 0x7f97c8004c80 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f97d0001750 tx=0x7f97c8005830 comp rx=0 tx=0).stop 2024-03-06T07:26:58.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.836+0000 7f97e649d640 1 -- 172.21.15.83:0/3681202936 shutdown_connections 2024-03-06T07:26:58.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.836+0000 7f97e649d640 1 --2- 172.21.15.83:0/3681202936 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97c8004800 0x7f97c8004c80 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.836+0000 7f97e649d640 1 -- 172.21.15.83:0/3681202936 >> 172.21.15.83:0/3681202936 conn(0x7f97e009d230 msgr2=0x7f97e00d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:58.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.836+0000 7f97e649d640 1 -- 172.21.15.83:0/3681202936 shutdown_connections 2024-03-06T07:26:58.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.836+0000 7f97e649d640 1 -- 172.21.15.83:0/3681202936 wait complete. 2024-03-06T07:26:58.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.837+0000 7f97e649d640 1 Processor -- start 2024-03-06T07:26:58.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.838+0000 7f97e649d640 1 -- start start 2024-03-06T07:26:58.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.838+0000 7f97e649d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97c8004800 0x7f97e00b8f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.838+0000 7f97e649d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97e01f4ec0 0x7f97e01f7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.838+0000 7f97e649d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97e01f7d70 0x7f97e00ca010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.838+0000 7f97e649d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e00d1b30 con 0x7f97c8004800 2024-03-06T07:26:58.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.838+0000 7f97e649d640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e00d2f40 con 0x7f97e01f4ec0 2024-03-06T07:26:58.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.838+0000 7f97e649d640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97e015e530 con 0x7f97e01f7d70 2024-03-06T07:26:58.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.838+0000 7f97dffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97c8004800 0x7f97e00b8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.838+0000 7f97dffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97c8004800 0x7f97e00b8f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51446/0 (socket says 172.21.15.83:51446) 2024-03-06T07:26:58.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.838+0000 7f97dffff640 1 -- 172.21.15.83:0/3663278043 learned_addr learned my addr 172.21.15.83:0/3663278043 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:58.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.839+0000 7f97e4a3a640 1 --2- 172.21.15.83:0/3663278043 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97e01f7d70 0x7f97e00ca010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.839+0000 7f97df7fe640 1 --2- 172.21.15.83:0/3663278043 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97e01f4ec0 0x7f97e01f7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.839+0000 7f97dffff640 1 -- 172.21.15.83:0/3663278043 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97e01f7d70 msgr2=0x7f97e00ca010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.839+0000 7f97dffff640 1 --2- 172.21.15.83:0/3663278043 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97e01f7d70 0x7f97e00ca010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.839+0000 7f97dffff640 1 -- 172.21.15.83:0/3663278043 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97e01f4ec0 msgr2=0x7f97e01f7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.839+0000 7f97dffff640 1 --2- 172.21.15.83:0/3663278043 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97e01f4ec0 0x7f97e01f7830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.839+0000 7f97dffff640 1 -- 172.21.15.83:0/3663278043 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e01f81f0 con 0x7f97c8004800 2024-03-06T07:26:58.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.839+0000 7f97e4a3a640 1 --2- 172.21.15.83:0/3663278043 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97e01f7d70 0x7f97e00ca010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:58.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.839+0000 7f97dffff640 1 --2- 172.21.15.83:0/3663278043 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97c8004800 0x7f97e00b8f60 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f97d4011aa0 tx=0x7f97d4011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:58.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.839+0000 7f97dcff9640 1 -- 172.21.15.83:0/3663278043 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d400fb80 con 0x7f97c8004800 2024-03-06T07:26:58.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.839+0000 7f97dcff9640 1 -- 172.21.15.83:0/3663278043 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97d400fd20 con 0x7f97c8004800 2024-03-06T07:26:58.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.839+0000 7f97e649d640 1 -- 172.21.15.83:0/3663278043 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97e01f8480 con 0x7f97c8004800 2024-03-06T07:26:58.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.839+0000 7f97dcff9640 1 -- 172.21.15.83:0/3663278043 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d4014660 con 0x7f97c8004800 2024-03-06T07:26:58.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.839+0000 7f97e649d640 1 -- 172.21.15.83:0/3663278043 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97e01f46c0 con 0x7f97c8004800 2024-03-06T07:26:58.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.840+0000 7f97e649d640 1 -- 172.21.15.83:0/3663278043 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97e0069ee0 con 0x7f97c8004800 2024-03-06T07:26:58.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.841+0000 7f97dcff9640 1 -- 172.21.15.83:0/3663278043 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f97d4012040 con 0x7f97c8004800 2024-03-06T07:26:58.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.841+0000 7f97dcff9640 1 --2- 172.21.15.83:0/3663278043 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f97cc06b4d0 0x7f97cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.841+0000 7f97df7fe640 1 --2- 172.21.15.83:0/3663278043 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f97cc06b4d0 0x7f97cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.841+0000 7f97dcff9640 1 -- 172.21.15.83:0/3663278043 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f97d408a970 con 0x7f97c8004800 2024-03-06T07:26:58.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.842+0000 7f97df7fe640 1 --2- 172.21.15.83:0/3663278043 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f97cc06b4d0 0x7f97cc06d990 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f97d0033000 tx=0x7f97d0009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:58.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:58.845+0000 7f97dcff9640 1 -- 172.21.15.83:0/3663278043 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f97d4019030 con 0x7f97c8004800 2024-03-06T07:26:59.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.062+0000 7f97e649d640 1 -- 172.21.15.83:0/3663278043 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f97e00d4670 con 0x7f97c8004800 2024-03-06T07:26:59.064 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-03-06T07:26:59.062+0000 7fb0dae11640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:26:59.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.064+0000 7f97dcff9640 1 -- 172.21.15.83:0/3663278043 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v54) v1 ==== 161+0+0 (secure 0 0 0) 0x7f97d4059820 con 0x7f97c8004800 2024-03-06T07:26:59.065 INFO:teuthology.orchestra.run.smithi083.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:26:59.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.067+0000 7f97e649d640 1 -- 172.21.15.83:0/3663278043 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f97cc06b4d0 msgr2=0x7f97cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.067+0000 7f97e649d640 1 --2- 172.21.15.83:0/3663278043 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f97cc06b4d0 0x7f97cc06d990 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f97d0033000 tx=0x7f97d0009030 comp rx=0 tx=0).stop 2024-03-06T07:26:59.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.067+0000 7f97e649d640 1 -- 172.21.15.83:0/3663278043 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97c8004800 msgr2=0x7f97e00b8f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.067+0000 7f97e649d640 1 --2- 172.21.15.83:0/3663278043 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97c8004800 0x7f97e00b8f60 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f97d4011aa0 tx=0x7f97d4011f60 comp rx=0 tx=0).stop 2024-03-06T07:26:59.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.067+0000 7f97e649d640 1 -- 172.21.15.83:0/3663278043 shutdown_connections 2024-03-06T07:26:59.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.067+0000 7f97e649d640 1 --2- 172.21.15.83:0/3663278043 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f97cc06b4d0 0x7f97cc06d990 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.067+0000 7f97e649d640 1 --2- 172.21.15.83:0/3663278043 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97e01f7d70 0x7f97e00ca010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.067+0000 7f97e649d640 1 --2- 172.21.15.83:0/3663278043 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97e01f4ec0 0x7f97e01f7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.067+0000 7f97e649d640 1 --2- 172.21.15.83:0/3663278043 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97c8004800 0x7f97e00b8f60 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.067+0000 7f97e649d640 1 -- 172.21.15.83:0/3663278043 >> 172.21.15.83:0/3663278043 conn(0x7f97e009d230 msgr2=0x7f97e006ce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:59.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.067+0000 7f97e649d640 1 -- 172.21.15.83:0/3663278043 shutdown_connections 2024-03-06T07:26:59.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.067+0000 7f97e649d640 1 -- 172.21.15.83:0/3663278043 wait complete. 2024-03-06T07:26:59.081 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:26:59.117 INFO:tasks.ceph.mds.a.smithi083.stdout:starting mds.a at 2024-03-06T07:26:59.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.182+0000 7f743ed5e640 1 Processor -- start 2024-03-06T07:26:59.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.182+0000 7f743ed5e640 1 -- start start 2024-03-06T07:26:59.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.183+0000 7f743ed5e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f74380b2da0 0x7f7438162570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.183+0000 7f743ed5e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7438162c70 con 0x7f74380cd690 2024-03-06T07:26:59.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.183+0000 7f743ed5e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f743806cd80 con 0x7f7438161670 2024-03-06T07:26:59.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.183+0000 7f743ed5e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74380b3590 con 0x7f74380b2da0 2024-03-06T07:26:59.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.183+0000 7f743d2fb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f74380b2da0 0x7f7438162570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.183+0000 7f7437fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f74380cd690 0x7f74380cff50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:55260/0 (socket says 172.21.15.83:55260) 2024-03-06T07:26:59.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.183+0000 7f7437fff640 1 -- 172.21.15.83:0/1652909993 learned_addr learned my addr 172.21.15.83:0/1652909993 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:26:59.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.184+0000 7f743d2fb640 1 -- 172.21.15.83:0/1652909993 >> v1:172.21.15.183:6789/0 conn(0x7f7438161670 legacy=0x7f7438161e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:59.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.184+0000 7f743d2fb640 1 -- 172.21.15.83:0/1652909993 >> v1:172.21.15.83:6789/0 conn(0x7f74380cd690 legacy=0x7f74380cff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.184+0000 7f743d2fb640 1 -- 172.21.15.83:0/1652909993 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f743806d060 con 0x7f74380b2da0 2024-03-06T07:26:59.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.184+0000 7f74377fe640 1 -- 172.21.15.83:0/1652909993 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2308134801 0 0) 0x7f7438162c70 con 0x7f74380cd690 2024-03-06T07:26:59.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.184+0000 7f743d2fb640 1 --2- 172.21.15.83:0/1652909993 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74380b2da0 0x7f7438162570 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f7430009dc0 tx=0x7f743002be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca73e9cbd628177a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:59.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.184+0000 7f74377fe640 1 -- 172.21.15.83:0/1652909993 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7430039070 con 0x7f74380b2da0 2024-03-06T07:26:59.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.184+0000 7f74377fe640 1 -- 172.21.15.83:0/1652909993 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f743002cc40 con 0x7f74380b2da0 2024-03-06T07:26:59.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.185+0000 7f74377fe640 1 -- 172.21.15.83:0/1652909993 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7430034440 con 0x7f74380b2da0 2024-03-06T07:26:59.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.185+0000 7f743ed5e640 1 -- 172.21.15.83:0/1652909993 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74380b2da0 msgr2=0x7f7438162570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.185+0000 7f743ed5e640 1 --2- 172.21.15.83:0/1652909993 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74380b2da0 0x7f7438162570 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f7430009dc0 tx=0x7f743002be90 comp rx=0 tx=0).stop 2024-03-06T07:26:59.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.185+0000 7f743ed5e640 1 -- 172.21.15.83:0/1652909993 shutdown_connections 2024-03-06T07:26:59.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.185+0000 7f743ed5e640 1 --2- 172.21.15.83:0/1652909993 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74380b2da0 0x7f7438162570 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.185+0000 7f743ed5e640 1 -- 172.21.15.83:0/1652909993 >> 172.21.15.83:0/1652909993 conn(0x7f74380d1d50 msgr2=0x7f74380d2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:59.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.185+0000 7f743ed5e640 1 -- 172.21.15.83:0/1652909993 shutdown_connections 2024-03-06T07:26:59.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.185+0000 7f743ed5e640 1 -- 172.21.15.83:0/1652909993 wait complete. 2024-03-06T07:26:59.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.186+0000 7f743ed5e640 1 Processor -- start 2024-03-06T07:26:59.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.186+0000 7f743ed5e640 1 -- start start 2024-03-06T07:26:59.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.187+0000 7f743ed5e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74380b2da0 0x7f743815b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.187+0000 7f743ed5e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f74380cd690 0x7f74380b5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.187+0000 7f743ed5e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7438161670 0x7f74380c5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.187+0000 7f743ed5e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f743806d4c0 con 0x7f74380b2da0 2024-03-06T07:26:59.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.187+0000 7f743ed5e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f743806d340 con 0x7f74380cd690 2024-03-06T07:26:59.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.187+0000 7f743ed5e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7438161a50 con 0x7f7438161670 2024-03-06T07:26:59.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.187+0000 7f743cafa640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74380b2da0 0x7f743815b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.187+0000 7f743cafa640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74380b2da0 0x7f743815b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51488/0 (socket says 172.21.15.83:51488) 2024-03-06T07:26:59.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.187+0000 7f743cafa640 1 -- 172.21.15.83:0/1188286784 learned_addr learned my addr 172.21.15.83:0/1188286784 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:26:59.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.187+0000 7f743cafa640 1 -- 172.21.15.83:0/1188286784 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7438161670 msgr2=0x7f74380c5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.187+0000 7f743cafa640 1 --2- 172.21.15.83:0/1188286784 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7438161670 0x7f74380c5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.187+0000 7f743cafa640 1 -- 172.21.15.83:0/1188286784 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f74380cd690 msgr2=0x7f74380b5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.187+0000 7f743cafa640 1 --2- 172.21.15.83:0/1188286784 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f74380cd690 0x7f74380b5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.187+0000 7f743cafa640 1 -- 172.21.15.83:0/1188286784 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f743815b8c0 con 0x7f74380b2da0 2024-03-06T07:26:59.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.188+0000 7f743cafa640 1 --2- 172.21.15.83:0/1188286784 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74380b2da0 0x7f743815b380 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f742800c460 tx=0x7f742800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:59.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.188+0000 7f74357fa640 1 -- 172.21.15.83:0/1188286784 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7428016020 con 0x7f74380b2da0 2024-03-06T07:26:59.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.188+0000 7f74357fa640 1 -- 172.21.15.83:0/1188286784 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7428012870 con 0x7f74380b2da0 2024-03-06T07:26:59.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.188+0000 7f74357fa640 1 -- 172.21.15.83:0/1188286784 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7428012c60 con 0x7f74380b2da0 2024-03-06T07:26:59.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.188+0000 7f743ed5e640 1 -- 172.21.15.83:0/1188286784 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74380c9f10 con 0x7f74380b2da0 2024-03-06T07:26:59.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.188+0000 7f743ed5e640 1 -- 172.21.15.83:0/1188286784 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74380b6780 con 0x7f74380b2da0 2024-03-06T07:26:59.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.189+0000 7f743ed5e640 1 -- 172.21.15.83:0/1188286784 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7438068050 con 0x7f74380b2da0 2024-03-06T07:26:59.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.189+0000 7f74357fa640 1 -- 172.21.15.83:0/1188286784 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7428019020 con 0x7f74380b2da0 2024-03-06T07:26:59.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.189+0000 7f74357fa640 1 --2- 172.21.15.83:0/1188286784 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f741806b4d0 0x7f741806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.190+0000 7f74357fa640 1 -- 172.21.15.83:0/1188286784 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f7428089c50 con 0x7f74380b2da0 2024-03-06T07:26:59.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.190+0000 7f7437fff640 1 --2- 172.21.15.83:0/1188286784 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f741806b4d0 0x7f741806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.190+0000 7f7437fff640 1 --2- 172.21.15.83:0/1188286784 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f741806b4d0 0x7f741806d990 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f7420007fd0 tx=0x7f7420007920 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:59.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.194+0000 7f74357fa640 1 -- 172.21.15.83:0/1188286784 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f742808f020 con 0x7f74380b2da0 2024-03-06T07:26:59.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:26:59.401+0000 7f743ed5e640 1 -- 172.21.15.83:0/1188286784 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f74380b7cd0 con 0x7f74380b2da0 2024-03-06T07:26:59.491 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:59.493 DEBUG:teuthology.orchestra.run.smithi183:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:00.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.065+0000 7f74357fa640 1 -- 172.21.15.83:0/1188286784 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v55) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7428020070 con 0x7f74380b2da0 2024-03-06T07:27:00.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.068+0000 7f743ed5e640 1 -- 172.21.15.83:0/1188286784 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f741806b4d0 msgr2=0x7f741806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.068+0000 7f743ed5e640 1 --2- 172.21.15.83:0/1188286784 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f741806b4d0 0x7f741806d990 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f7420007fd0 tx=0x7f7420007920 comp rx=0 tx=0).stop 2024-03-06T07:27:00.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.068+0000 7f743ed5e640 1 -- 172.21.15.83:0/1188286784 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74380b2da0 msgr2=0x7f743815b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.068+0000 7f743ed5e640 1 --2- 172.21.15.83:0/1188286784 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74380b2da0 0x7f743815b380 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f742800c460 tx=0x7f742800c920 comp rx=0 tx=0).stop 2024-03-06T07:27:00.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.068+0000 7f743ed5e640 1 -- 172.21.15.83:0/1188286784 shutdown_connections 2024-03-06T07:27:00.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.068+0000 7f743ed5e640 1 --2- 172.21.15.83:0/1188286784 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f741806b4d0 0x7f741806d990 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.068+0000 7f743ed5e640 1 --2- 172.21.15.83:0/1188286784 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7438161670 0x7f74380c5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.068+0000 7f743ed5e640 1 --2- 172.21.15.83:0/1188286784 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f74380cd690 0x7f74380b5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.068+0000 7f743ed5e640 1 --2- 172.21.15.83:0/1188286784 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74380b2da0 0x7f743815b380 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.068+0000 7f743ed5e640 1 -- 172.21.15.83:0/1188286784 >> 172.21.15.83:0/1188286784 conn(0x7f74380d1d50 msgr2=0x7f74380b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:00.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.068+0000 7f743ed5e640 1 -- 172.21.15.83:0/1188286784 shutdown_connections 2024-03-06T07:27:00.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.068+0000 7f743ed5e640 1 -- 172.21.15.83:0/1188286784 wait complete. 2024-03-06T07:27:00.083 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:27:00.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.186+0000 7f9c61c81640 1 Processor -- start 2024-03-06T07:27:00.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.186+0000 7f9c61c81640 1 -- start start 2024-03-06T07:27:00.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.187+0000 7f9c61c81640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f9c5c0b3b80 0x7f9c5c071d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.187+0000 7f9c61c81640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c5c06edc0 con 0x7f9c5c16aab0 2024-03-06T07:27:00.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.187+0000 7f9c61c81640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c5c0b30a0 con 0x7f9c5c16ae90 2024-03-06T07:27:00.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.187+0000 7f9c61c81640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c5c08b5d0 con 0x7f9c5c0b3b80 2024-03-06T07:27:00.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.187+0000 7f9c5b7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f9c5c16aab0 0x7f9c5c06e250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:55268/0 (socket says 172.21.15.83:55268) 2024-03-06T07:27:00.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.187+0000 7f9c5b7fe640 1 -- 172.21.15.83:0/4082320641 learned_addr learned my addr 172.21.15.83:0/4082320641 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:27:00.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.188+0000 7f9c5a7fc640 1 -- 172.21.15.83:0/4082320641 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 636924432 0 0) 0x7f9c5c06edc0 con 0x7f9c5c16aab0 2024-03-06T07:27:00.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.188+0000 7f9c5affd640 1 --2- 172.21.15.83:0/4082320641 >> v2:172.21.15.183:3300/0 conn(0x7f9c5c0b3b80 0x7f9c5c071d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.199+0000 7f9c5a7fc640 1 -- 172.21.15.83:0/4082320641 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c44003970 con 0x7f9c5c16aab0 2024-03-06T07:27:00.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.199+0000 7f9c5a7fc640 1 -- 172.21.15.83:0/4082320641 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 229060106 0 0) 0x7f9c5c0b30a0 con 0x7f9c5c16ae90 2024-03-06T07:27:00.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.199+0000 7f9c5a7fc640 1 -- 172.21.15.83:0/4082320641 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c5c06edc0 con 0x7f9c5c16ae90 2024-03-06T07:27:00.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.199+0000 7f9c5a7fc640 1 -- 172.21.15.83:0/4082320641 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1965521821 0 0) 0x7f9c44003970 con 0x7f9c5c16aab0 2024-03-06T07:27:00.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.199+0000 7f9c5a7fc640 1 -- 172.21.15.83:0/4082320641 >> v2:172.21.15.183:3300/0 conn(0x7f9c5c0b3b80 msgr2=0x7f9c5c071d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:00.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.199+0000 7f9c5a7fc640 1 --2- 172.21.15.83:0/4082320641 >> v2:172.21.15.183:3300/0 conn(0x7f9c5c0b3b80 0x7f9c5c071d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.199+0000 7f9c5a7fc640 1 -- 172.21.15.83:0/4082320641 >> v1:172.21.15.183:6789/0 conn(0x7f9c5c16ae90 legacy=0x7f9c5c0cd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.199+0000 7f9c5a7fc640 1 -- 172.21.15.83:0/4082320641 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c5c0b3380 con 0x7f9c5c16aab0 2024-03-06T07:27:00.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.199+0000 7f9c5a7fc640 1 -- 172.21.15.83:0/4082320641 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f9c4c001670 con 0x7f9c5c16aab0 2024-03-06T07:27:00.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.199+0000 7f9c5a7fc640 1 -- 172.21.15.83:0/4082320641 >> v1:172.21.15.83:6789/0 conn(0x7f9c5c16aab0 legacy=0x7f9c5c06e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.199+0000 7f9c5a7fc640 1 --2- 172.21.15.83:0/4082320641 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c44004c50 0x7f9c44005030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.199+0000 7f9c5a7fc640 1 --2- 172.21.15.83:0/4082320641 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c44005630 0x7f9c44003bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.199+0000 7f9c5a7fc640 1 --2- 172.21.15.83:0/4082320641 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c44004200 0x7f9c440046b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.199+0000 7f9c5a7fc640 1 -- 172.21.15.83:0/4082320641 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c44008160 con 0x7f9c44004c50 2024-03-06T07:27:00.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.199+0000 7f9c5a7fc640 1 -- 172.21.15.83:0/4082320641 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c44008310 con 0x7f9c44004200 2024-03-06T07:27:00.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.199+0000 7f9c5a7fc640 1 -- 172.21.15.83:0/4082320641 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c44008510 con 0x7f9c44005630 2024-03-06T07:27:00.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.199+0000 7f9c5b7fe640 1 --2- 172.21.15.83:0/4082320641 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c44004c50 0x7f9c44005030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.200+0000 7f9c5bfff640 1 --2- 172.21.15.83:0/4082320641 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c44004200 0x7f9c440046b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.200+0000 7f9c5affd640 1 --2- 172.21.15.83:0/4082320641 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c44005630 0x7f9c44003bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.200+0000 7f9c5b7fe640 1 -- 172.21.15.83:0/4082320641 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c44005630 msgr2=0x7f9c44003bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:00.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.200+0000 7f9c5b7fe640 1 --2- 172.21.15.83:0/4082320641 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c44005630 0x7f9c44003bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.200+0000 7f9c5b7fe640 1 -- 172.21.15.83:0/4082320641 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c44004200 msgr2=0x7f9c440046b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:00.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.200+0000 7f9c5b7fe640 1 --2- 172.21.15.83:0/4082320641 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c44004200 0x7f9c440046b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.200+0000 7f9c5b7fe640 1 -- 172.21.15.83:0/4082320641 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c44003970 con 0x7f9c44004c50 2024-03-06T07:27:00.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.200+0000 7f9c5bfff640 1 -- 172.21.15.83:0/4082320641 reap_dead start 2024-03-06T07:27:00.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.201+0000 7f9c5b7fe640 1 --2- 172.21.15.83:0/4082320641 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c44004c50 0x7f9c44005030 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c004f90 tx=0x7f9c44005c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c701c67eef19ba55 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:00.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.201+0000 7f9c5a7fc640 1 -- 172.21.15.83:0/4082320641 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c4c004b60 con 0x7f9c44004c50 2024-03-06T07:27:00.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.201+0000 7f9c5a7fc640 1 -- 172.21.15.83:0/4082320641 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c4c005ad0 con 0x7f9c44004c50 2024-03-06T07:27:00.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.201+0000 7f9c5a7fc640 1 -- 172.21.15.83:0/4082320641 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c4c0080d0 con 0x7f9c44004c50 2024-03-06T07:27:00.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.201+0000 7f9c61c81640 1 -- 172.21.15.83:0/4082320641 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c44004c50 msgr2=0x7f9c44005030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.201+0000 7f9c61c81640 1 --2- 172.21.15.83:0/4082320641 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c44004c50 0x7f9c44005030 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c004f90 tx=0x7f9c44005c80 comp rx=0 tx=0).stop 2024-03-06T07:27:00.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.201+0000 7f9c61c81640 1 -- 172.21.15.83:0/4082320641 shutdown_connections 2024-03-06T07:27:00.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.201+0000 7f9c61c81640 1 --2- 172.21.15.83:0/4082320641 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c44004c50 0x7f9c44005030 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.201+0000 7f9c61c81640 1 -- 172.21.15.83:0/4082320641 >> 172.21.15.83:0/4082320641 conn(0x7f9c5c0b1b50 msgr2=0x7f9c5c0b1f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:00.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.201+0000 7f9c61c81640 1 -- 172.21.15.83:0/4082320641 shutdown_connections 2024-03-06T07:27:00.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.201+0000 7f9c61c81640 1 -- 172.21.15.83:0/4082320641 wait complete. 2024-03-06T07:27:00.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.203+0000 7f9c61c81640 1 Processor -- start 2024-03-06T07:27:00.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.203+0000 7f9c61c81640 1 -- start start 2024-03-06T07:27:00.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.204+0000 7f9c61c81640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c44004c50 0x7f9c5c203cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.204+0000 7f9c61c81640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c5c201180 0x7f9c5c163480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.204+0000 7f9c61c81640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c5c16cae0 0x7f9c5c167ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.204+0000 7f9c61c81640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5c06e250 con 0x7f9c5c16cae0 2024-03-06T07:27:00.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.204+0000 7f9c61c81640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5c0b2e50 con 0x7f9c44004c50 2024-03-06T07:27:00.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.204+0000 7f9c61c81640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5c06e3d0 con 0x7f9c5c201180 2024-03-06T07:27:00.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.204+0000 7f9c5bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c5c16cae0 0x7f9c5c167ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.204+0000 7f9c5bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c5c16cae0 0x7f9c5c167ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51516/0 (socket says 172.21.15.83:51516) 2024-03-06T07:27:00.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.204+0000 7f9c5bfff640 1 -- 172.21.15.83:0/2579980403 learned_addr learned my addr 172.21.15.83:0/2579980403 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:00.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.204+0000 7f9c5bfff640 1 -- 172.21.15.83:0/2579980403 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c5c201180 msgr2=0x7f9c5c163480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.204+0000 7f9c5bfff640 1 --2- 172.21.15.83:0/2579980403 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c5c201180 0x7f9c5c163480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.204+0000 7f9c5bfff640 1 -- 172.21.15.83:0/2579980403 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c44004c50 msgr2=0x7f9c5c203cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.204+0000 7f9c5bfff640 1 --2- 172.21.15.83:0/2579980403 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c44004c50 0x7f9c5c203cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.204+0000 7f9c5bfff640 1 -- 172.21.15.83:0/2579980403 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c5c201600 con 0x7f9c5c16cae0 2024-03-06T07:27:00.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.204+0000 7f9c5bfff640 1 --2- 172.21.15.83:0/2579980403 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c5c16cae0 0x7f9c5c167ec0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f9c5400fae0 tx=0x7f9c540020e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:00.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.204+0000 7f9c43fff640 1 -- 172.21.15.83:0/2579980403 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c54019070 con 0x7f9c5c16cae0 2024-03-06T07:27:00.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.204+0000 7f9c43fff640 1 -- 172.21.15.83:0/2579980403 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c54006e20 con 0x7f9c5c16cae0 2024-03-06T07:27:00.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.204+0000 7f9c61c81640 1 -- 172.21.15.83:0/2579980403 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c5c204390 con 0x7f9c5c16cae0 2024-03-06T07:27:00.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.204+0000 7f9c43fff640 1 -- 172.21.15.83:0/2579980403 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c54003bb0 con 0x7f9c5c16cae0 2024-03-06T07:27:00.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.204+0000 7f9c61c81640 1 -- 172.21.15.83:0/2579980403 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c5c204950 con 0x7f9c5c16cae0 2024-03-06T07:27:00.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.205+0000 7f9c61c81640 1 -- 172.21.15.83:0/2579980403 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c5c069ee0 con 0x7f9c5c16cae0 2024-03-06T07:27:00.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.206+0000 7f9c43fff640 1 -- 172.21.15.83:0/2579980403 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9c5400d770 con 0x7f9c5c16cae0 2024-03-06T07:27:00.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.206+0000 7f9c43fff640 1 --2- 172.21.15.83:0/2579980403 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9c4806b500 0x7f9c4806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.206+0000 7f9c43fff640 1 -- 172.21.15.83:0/2579980403 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f9c54089810 con 0x7f9c5c16cae0 2024-03-06T07:27:00.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.206+0000 7f9c5b7fe640 1 --2- 172.21.15.83:0/2579980403 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9c4806b500 0x7f9c4806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.207+0000 7f9c5b7fe640 1 --2- 172.21.15.83:0/2579980403 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9c4806b500 0x7f9c4806d9c0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c004f60 tx=0x7f9c4c007000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:00.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.211+0000 7f9c43fff640 1 -- 172.21.15.83:0/2579980403 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9c540585e0 con 0x7f9c5c16cae0 2024-03-06T07:27:00.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:00.385+0000 7f9c61c81640 1 -- 172.21.15.83:0/2579980403 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9c5c1639c0 con 0x7f9c5c16cae0 2024-03-06T07:27:01.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.070+0000 7f9c43fff640 1 -- 172.21.15.83:0/2579980403 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v100) v1 ==== 181+0+0 (secure 0 0 0) 0x7f9c54057ee0 con 0x7f9c5c16cae0 2024-03-06T07:27:01.072 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:27:01.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.073+0000 7f9c61c81640 1 -- 172.21.15.83:0/2579980403 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9c4806b500 msgr2=0x7f9c4806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.073+0000 7f9c61c81640 1 --2- 172.21.15.83:0/2579980403 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9c4806b500 0x7f9c4806d9c0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c004f60 tx=0x7f9c4c007000 comp rx=0 tx=0).stop 2024-03-06T07:27:01.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.073+0000 7f9c61c81640 1 -- 172.21.15.83:0/2579980403 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c5c16cae0 msgr2=0x7f9c5c167ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.073+0000 7f9c61c81640 1 --2- 172.21.15.83:0/2579980403 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c5c16cae0 0x7f9c5c167ec0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f9c5400fae0 tx=0x7f9c540020e0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.074+0000 7f9c61c81640 1 -- 172.21.15.83:0/2579980403 shutdown_connections 2024-03-06T07:27:01.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.074+0000 7f9c61c81640 1 --2- 172.21.15.83:0/2579980403 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9c4806b500 0x7f9c4806d9c0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.074+0000 7f9c61c81640 1 --2- 172.21.15.83:0/2579980403 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c5c16cae0 0x7f9c5c167ec0 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.074+0000 7f9c61c81640 1 --2- 172.21.15.83:0/2579980403 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c5c201180 0x7f9c5c163480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.074+0000 7f9c61c81640 1 --2- 172.21.15.83:0/2579980403 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c44004c50 0x7f9c5c203cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.074+0000 7f9c61c81640 1 -- 172.21.15.83:0/2579980403 >> 172.21.15.83:0/2579980403 conn(0x7f9c5c0b1b50 msgr2=0x7f9c5c0cda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:01.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.074+0000 7f9c61c81640 1 -- 172.21.15.83:0/2579980403 shutdown_connections 2024-03-06T07:27:01.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.074+0000 7f9c61c81640 1 -- 172.21.15.83:0/2579980403 wait complete. 2024-03-06T07:27:01.090 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:27:01.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.194+0000 7f86cb647640 1 Processor -- start 2024-03-06T07:27:01.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.194+0000 7f86cb647640 1 -- start start 2024-03-06T07:27:01.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.195+0000 7f86cb647640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f86c40ce5a0 0x7f86c40b4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.195+0000 7f86cb647640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f86c4162940 0x7f86c4164d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.195+0000 7f86cb647640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c40ccbd0 con 0x7f86c40ce980 2024-03-06T07:27:01.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.195+0000 7f86cb647640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86c40b3b00 con 0x7f86c4162940 2024-03-06T07:27:01.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.195+0000 7f86cb647640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86c40b3de0 con 0x7f86c40ce5a0 2024-03-06T07:27:01.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.195+0000 7f86c9be4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f86c4162940 0x7f86c4164d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.195+0000 7f86c9be4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f86c4162940 0x7f86c4164d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51518/0 (socket says 172.21.15.83:51518) 2024-03-06T07:27:01.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.195+0000 7f86c9be4640 1 -- 172.21.15.83:0/1144779950 learned_addr learned my addr 172.21.15.83:0/1144779950 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:01.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.195+0000 7f86c93e3640 1 --2- 172.21.15.83:0/1144779950 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f86c40ce5a0 0x7f86c40b4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.195+0000 7f86c9be4640 1 -- 172.21.15.83:0/1144779950 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f86c40ce5a0 msgr2=0x7f86c40b4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:01.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.195+0000 7f86c9be4640 1 --2- 172.21.15.83:0/1144779950 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f86c40ce5a0 0x7f86c40b4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.195+0000 7f86c9be4640 1 -- 172.21.15.83:0/1144779950 >> v1:172.21.15.183:6789/0 conn(0x7f86c40ce980 legacy=0x7f86c40cede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:01.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.195+0000 7f86c9be4640 1 -- 172.21.15.83:0/1144779950 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c40b40c0 con 0x7f86c4162940 2024-03-06T07:27:01.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.195+0000 7f86c93e3640 1 --2- 172.21.15.83:0/1144779950 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f86c40ce5a0 0x7f86c40b4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:01.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.196+0000 7f86c9be4640 1 --2- 172.21.15.83:0/1144779950 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f86c4162940 0x7f86c4164d30 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f86bc009e50 tx=0x7f86bc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b8155acc23ab949 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:01.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.196+0000 7f86bbfff640 1 -- 172.21.15.83:0/1144779950 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86bc02ca80 con 0x7f86c4162940 2024-03-06T07:27:01.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.196+0000 7f86bbfff640 1 -- 172.21.15.83:0/1144779950 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86bc02cc20 con 0x7f86c4162940 2024-03-06T07:27:01.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.196+0000 7f86bbfff640 1 -- 172.21.15.83:0/1144779950 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86bc0343f0 con 0x7f86c4162940 2024-03-06T07:27:01.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.197+0000 7f86cb647640 1 -- 172.21.15.83:0/1144779950 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f86c4162940 msgr2=0x7f86c4164d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.197+0000 7f86cb647640 1 --2- 172.21.15.83:0/1144779950 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f86c4162940 0x7f86c4164d30 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f86bc009e50 tx=0x7f86bc02bee0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.197+0000 7f86cb647640 1 -- 172.21.15.83:0/1144779950 shutdown_connections 2024-03-06T07:27:01.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.197+0000 7f86cb647640 1 --2- 172.21.15.83:0/1144779950 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f86c4162940 0x7f86c4164d30 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.197+0000 7f86cb647640 1 --2- 172.21.15.83:0/1144779950 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f86c40ce5a0 0x7f86c40b4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.197+0000 7f86cb647640 1 -- 172.21.15.83:0/1144779950 >> 172.21.15.83:0/1144779950 conn(0x7f86c406b160 msgr2=0x7f86c406b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:01.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.197+0000 7f86cb647640 1 -- 172.21.15.83:0/1144779950 shutdown_connections 2024-03-06T07:27:01.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.197+0000 7f86cb647640 1 -- 172.21.15.83:0/1144779950 wait complete. 2024-03-06T07:27:01.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.198+0000 7f86cb647640 1 Processor -- start 2024-03-06T07:27:01.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.198+0000 7f86cb647640 1 -- start start 2024-03-06T07:27:01.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.199+0000 7f86cb647640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f86c40ce5a0 0x7f86c413a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.199+0000 7f86cb647640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f86c40ce980 0x7f86c415bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.199+0000 7f86cb647640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f86c4162940 0x7f86c415c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.199+0000 7f86cb647640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c40cd030 con 0x7f86c4162940 2024-03-06T07:27:01.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.199+0000 7f86cb647640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c40cceb0 con 0x7f86c40ce980 2024-03-06T07:27:01.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.199+0000 7f86cb647640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86c4071d80 con 0x7f86c40ce5a0 2024-03-06T07:27:01.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.199+0000 7f86c9be4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f86c4162940 0x7f86c415c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.199+0000 7f86c9be4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f86c4162940 0x7f86c415c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51532/0 (socket says 172.21.15.83:51532) 2024-03-06T07:27:01.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.199+0000 7f86c9be4640 1 -- 172.21.15.83:0/823966965 learned_addr learned my addr 172.21.15.83:0/823966965 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:01.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.199+0000 7f86c93e3640 1 --2- 172.21.15.83:0/823966965 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f86c40ce5a0 0x7f86c413a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.199+0000 7f86c9be4640 1 -- 172.21.15.83:0/823966965 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f86c40ce5a0 msgr2=0x7f86c413a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.199+0000 7f86c9be4640 1 --2- 172.21.15.83:0/823966965 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f86c40ce5a0 0x7f86c413a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.199+0000 7f86c9be4640 1 -- 172.21.15.83:0/823966965 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f86c40ce980 msgr2=0x7f86c415bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.199+0000 7f86c9be4640 1 --2- 172.21.15.83:0/823966965 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f86c40ce980 0x7f86c415bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.199+0000 7f86c9be4640 1 -- 172.21.15.83:0/823966965 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c415c910 con 0x7f86c4162940 2024-03-06T07:27:01.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.199+0000 7f86c9be4640 1 --2- 172.21.15.83:0/823966965 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f86c4162940 0x7f86c415c2e0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f86bc009e50 tx=0x7f86bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:01.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.199+0000 7f86b9ffb640 1 -- 172.21.15.83:0/823966965 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86bc043020 con 0x7f86c4162940 2024-03-06T07:27:01.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.199+0000 7f86b9ffb640 1 -- 172.21.15.83:0/823966965 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86bc03e450 con 0x7f86c4162940 2024-03-06T07:27:01.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.199+0000 7f86cb647640 1 -- 172.21.15.83:0/823966965 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86c415cbf0 con 0x7f86c4162940 2024-03-06T07:27:01.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.199+0000 7f86b9ffb640 1 -- 172.21.15.83:0/823966965 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86bc009510 con 0x7f86c4162940 2024-03-06T07:27:01.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.200+0000 7f86cb647640 1 -- 172.21.15.83:0/823966965 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86c415d1b0 con 0x7f86c4162940 2024-03-06T07:27:01.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.200+0000 7f86cb647640 1 -- 172.21.15.83:0/823966965 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86c4069ee0 con 0x7f86c4162940 2024-03-06T07:27:01.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.201+0000 7f86b9ffb640 1 -- 172.21.15.83:0/823966965 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f86bc046020 con 0x7f86c4162940 2024-03-06T07:27:01.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.201+0000 7f86b9ffb640 1 --2- 172.21.15.83:0/823966965 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f86a806b4d0 0x7f86a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.201+0000 7f86b9ffb640 1 -- 172.21.15.83:0/823966965 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5443+0+0 (secure 0 0 0) 0x7f86bc0ac9d0 con 0x7f86c4162940 2024-03-06T07:27:01.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.201+0000 7f86c93e3640 1 --2- 172.21.15.83:0/823966965 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f86a806b4d0 0x7f86a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.202+0000 7f86c93e3640 1 --2- 172.21.15.83:0/823966965 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f86a806b4d0 0x7f86a806d990 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f86c406ea20 tx=0x7f86c0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:01.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.206+0000 7f86b9ffb640 1 -- 172.21.15.83:0/823966965 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f86bc04d030 con 0x7f86c4162940 2024-03-06T07:27:01.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:01.380+0000 7f86cb647640 1 -- 172.21.15.83:0/823966965 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f86c41600b0 con 0x7f86c4162940 2024-03-06T07:27:02.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.079+0000 7f86b9ffb640 1 -- 172.21.15.83:0/823966965 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v101) v1 ==== 169+0+0 (secure 0 0 0) 0x7f86bc07b950 con 0x7f86c4162940 2024-03-06T07:27:02.081 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'cephfs_data' removed 2024-03-06T07:27:02.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.081+0000 7f86cb647640 1 -- 172.21.15.83:0/823966965 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f86a806b4d0 msgr2=0x7f86a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.081+0000 7f86cb647640 1 --2- 172.21.15.83:0/823966965 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f86a806b4d0 0x7f86a806d990 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f86c406ea20 tx=0x7f86c0008040 comp rx=0 tx=0).stop 2024-03-06T07:27:02.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.081+0000 7f86cb647640 1 -- 172.21.15.83:0/823966965 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f86c4162940 msgr2=0x7f86c415c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.081+0000 7f86cb647640 1 --2- 172.21.15.83:0/823966965 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f86c4162940 0x7f86c415c2e0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f86bc009e50 tx=0x7f86bc004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:02.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.081+0000 7f86cb647640 1 -- 172.21.15.83:0/823966965 shutdown_connections 2024-03-06T07:27:02.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.081+0000 7f86cb647640 1 --2- 172.21.15.83:0/823966965 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f86a806b4d0 0x7f86a806d990 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.081+0000 7f86cb647640 1 --2- 172.21.15.83:0/823966965 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f86c4162940 0x7f86c415c2e0 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.081+0000 7f86cb647640 1 --2- 172.21.15.83:0/823966965 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f86c40ce980 0x7f86c415bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.081+0000 7f86cb647640 1 --2- 172.21.15.83:0/823966965 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f86c40ce5a0 0x7f86c413a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.081+0000 7f86cb647640 1 -- 172.21.15.83:0/823966965 >> 172.21.15.83:0/823966965 conn(0x7f86c406b160 msgr2=0x7f86c406d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:02.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.082+0000 7f86cb647640 1 -- 172.21.15.83:0/823966965 shutdown_connections 2024-03-06T07:27:02.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.082+0000 7f86cb647640 1 -- 172.21.15.83:0/823966965 wait complete. 2024-03-06T07:27:02.099 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:27:02.101 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:27:02.149 INFO:teuthology.orchestra.run.smithi083.stdout:[global] 2024-03-06T07:27:02.149 INFO:teuthology.orchestra.run.smithi083.stdout: chdir = "" 2024-03-06T07:27:02.149 INFO:teuthology.orchestra.run.smithi083.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:27:02.149 INFO:teuthology.orchestra.run.smithi083.stdout: auth supported = cephx 2024-03-06T07:27:02.150 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.150 INFO:teuthology.orchestra.run.smithi083.stdout: filestore xattr use omap = true 2024-03-06T07:27:02.150 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.150 INFO:teuthology.orchestra.run.smithi083.stdout: mon clock drift allowed = 1.000 2024-03-06T07:27:02.150 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.150 INFO:teuthology.orchestra.run.smithi083.stdout: osd crush chooseleaf type = 0 2024-03-06T07:27:02.150 INFO:teuthology.orchestra.run.smithi083.stdout: auth debug = true 2024-03-06T07:27:02.150 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.150 INFO:teuthology.orchestra.run.smithi083.stdout: ms die on old message = true 2024-03-06T07:27:02.150 INFO:teuthology.orchestra.run.smithi083.stdout: ms die on bug = true 2024-03-06T07:27:02.150 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.150 INFO:teuthology.orchestra.run.smithi083.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:27:02.150 INFO:teuthology.orchestra.run.smithi083.stdout: mon pg warn max object skew = 0 2024-03-06T07:27:02.151 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.151 INFO:teuthology.orchestra.run.smithi083.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:27:02.151 INFO:teuthology.orchestra.run.smithi083.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:27:02.151 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.151 INFO:teuthology.orchestra.run.smithi083.stdout: osd pool default size = 2 2024-03-06T07:27:02.151 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.151 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd allow primary affinity = true 2024-03-06T07:27:02.151 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd allow pg remap = true 2024-03-06T07:27:02.151 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:27:02.151 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:27:02.151 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on no sortbitwise = false 2024-03-06T07:27:02.151 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:27:02.151 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on too few osds = false 2024-03-06T07:27:02.151 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:27:02.151 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:27:02.151 INFO:teuthology.orchestra.run.smithi083.stdout: mon_allow_pool_size_one = true 2024-03-06T07:27:02.152 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.152 INFO:teuthology.orchestra.run.smithi083.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:27:02.152 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.152 INFO:teuthology.orchestra.run.smithi083.stdout: osd default data pool replay window = 5 2024-03-06T07:27:02.152 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.152 INFO:teuthology.orchestra.run.smithi083.stdout: mon allow pool delete = true 2024-03-06T07:27:02.152 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.152 INFO:teuthology.orchestra.run.smithi083.stdout: mon cluster log file level = debug 2024-03-06T07:27:02.152 INFO:teuthology.orchestra.run.smithi083.stdout: debug asserts on shutdown = true 2024-03-06T07:27:02.152 INFO:teuthology.orchestra.run.smithi083.stdout: mon health detail to clog = false 2024-03-06T07:27:02.152 INFO:teuthology.orchestra.run.smithi083.stdout: mon host = "172.21.15.83,172.21.15.183,[v2:172.21.15.183:3301,v1:172.21.15.183:6790]" 2024-03-06T07:27:02.152 INFO:teuthology.orchestra.run.smithi083.stdout: fsid = 039877dd-d2ce-45d2-8e77-ac5eb7e1d55a 2024-03-06T07:27:02.152 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.152 INFO:teuthology.orchestra.run.smithi083.stdout:[osd] 2024-03-06T07:27:02.153 INFO:teuthology.orchestra.run.smithi083.stdout: osd journal size = 100 2024-03-06T07:27:02.153 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.153 INFO:teuthology.orchestra.run.smithi083.stdout: osd scrub load threshold = 5.0 2024-03-06T07:27:02.153 INFO:teuthology.orchestra.run.smithi083.stdout: osd scrub max interval = 600 2024-03-06T07:27:02.153 INFO:teuthology.orchestra.run.smithi083.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:27:02.153 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.153 INFO:teuthology.orchestra.run.smithi083.stdout: osd recover clone overlap = true 2024-03-06T07:27:02.153 INFO:teuthology.orchestra.run.smithi083.stdout: osd recovery max chunk = 1048576 2024-03-06T07:27:02.153 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.153 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug shutdown = true 2024-03-06T07:27:02.153 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug op order = true 2024-03-06T07:27:02.153 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug verify stray on activate = true 2024-03-06T07:27:02.153 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug trim objects = true 2024-03-06T07:27:02.153 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.153 INFO:teuthology.orchestra.run.smithi083.stdout: osd open classes on start = true 2024-03-06T07:27:02.154 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug pg log writeout = true 2024-03-06T07:27:02.154 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.154 INFO:teuthology.orchestra.run.smithi083.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:27:02.154 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.154 INFO:teuthology.orchestra.run.smithi083.stdout: osd map max advance = 10 2024-03-06T07:27:02.154 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.154 INFO:teuthology.orchestra.run.smithi083.stdout: journal zero on create = true 2024-03-06T07:27:02.154 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.154 INFO:teuthology.orchestra.run.smithi083.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:27:02.154 INFO:teuthology.orchestra.run.smithi083.stdout: filestore apply finisher threads = 3 2024-03-06T07:27:02.154 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.154 INFO:teuthology.orchestra.run.smithi083.stdout: bdev debug aio = true 2024-03-06T07:27:02.154 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug misdirected ops = true 2024-03-06T07:27:02.154 INFO:teuthology.orchestra.run.smithi083.stdout: bdev async discard = True 2024-03-06T07:27:02.154 INFO:teuthology.orchestra.run.smithi083.stdout: bdev enable discard = True 2024-03-06T07:27:02.155 INFO:teuthology.orchestra.run.smithi083.stdout: bluestore allocator = bitmap 2024-03-06T07:27:02.155 INFO:teuthology.orchestra.run.smithi083.stdout: bluestore block size = 96636764160 2024-03-06T07:27:02.155 INFO:teuthology.orchestra.run.smithi083.stdout: bluestore fsck on mount = True 2024-03-06T07:27:02.155 INFO:teuthology.orchestra.run.smithi083.stdout: debug bluefs = 1/20 2024-03-06T07:27:02.155 INFO:teuthology.orchestra.run.smithi083.stdout: debug bluestore = 1/20 2024-03-06T07:27:02.155 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-06T07:27:02.155 INFO:teuthology.orchestra.run.smithi083.stdout: debug osd = 20 2024-03-06T07:27:02.155 INFO:teuthology.orchestra.run.smithi083.stdout: debug rocksdb = 4/10 2024-03-06T07:27:02.155 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:27:02.155 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd full ratio = 0.9 2024-03-06T07:27:02.155 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:27:02.155 INFO:teuthology.orchestra.run.smithi083.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:27:02.155 INFO:teuthology.orchestra.run.smithi083.stdout: osd objectstore = bluestore 2024-03-06T07:27:02.155 INFO:teuthology.orchestra.run.smithi083.stdout: osd op complaint time = 180 2024-03-06T07:27:02.155 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.156 INFO:teuthology.orchestra.run.smithi083.stdout:[mgr] 2024-03-06T07:27:02.156 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-06T07:27:02.156 INFO:teuthology.orchestra.run.smithi083.stdout: debug mgr = 20 2024-03-06T07:27:02.156 INFO:teuthology.orchestra.run.smithi083.stdout: debug mon = 20 2024-03-06T07:27:02.156 INFO:teuthology.orchestra.run.smithi083.stdout: debug auth = 20 2024-03-06T07:27:02.156 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:02.156 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:02.156 INFO:teuthology.orchestra.run.smithi083.stdout: mgr/telemetry/nag = false 2024-03-06T07:27:02.156 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.156 INFO:teuthology.orchestra.run.smithi083.stdout:[mon] 2024-03-06T07:27:02.156 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-06T07:27:02.156 INFO:teuthology.orchestra.run.smithi083.stdout: debug mon = 20 2024-03-06T07:27:02.156 INFO:teuthology.orchestra.run.smithi083.stdout: debug paxos = 20 2024-03-06T07:27:02.156 INFO:teuthology.orchestra.run.smithi083.stdout: debug auth = 20 2024-03-06T07:27:02.156 INFO:teuthology.orchestra.run.smithi083.stdout: mon data avail warn = 5 2024-03-06T07:27:02.157 INFO:teuthology.orchestra.run.smithi083.stdout: mon mgr mkfs grace = 240 2024-03-06T07:27:02.157 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:02.157 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd reporter subtree level = osd 2024-03-06T07:27:02.157 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd prime pg temp = true 2024-03-06T07:27:02.157 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:02.157 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.157 INFO:teuthology.orchestra.run.smithi083.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:27:02.157 INFO:teuthology.orchestra.run.smithi083.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:27:02.157 INFO:teuthology.orchestra.run.smithi083.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:27:02.157 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.157 INFO:teuthology.orchestra.run.smithi083.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:27:02.157 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:27:02.157 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:27:02.157 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.157 INFO:teuthology.orchestra.run.smithi083.stdout: # 1m isn't quite enough 2024-03-06T07:27:02.157 INFO:teuthology.orchestra.run.smithi083.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:27:02.158 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.158 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:27:02.158 INFO:teuthology.orchestra.run.smithi083.stdout: mon op complaint time = 120 2024-03-06T07:27:02.158 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.158 INFO:teuthology.orchestra.run.smithi083.stdout:[client] 2024-03-06T07:27:02.158 INFO:teuthology.orchestra.run.smithi083.stdout: rgw cache enabled = true 2024-03-06T07:27:02.158 INFO:teuthology.orchestra.run.smithi083.stdout: rgw enable ops log = true 2024-03-06T07:27:02.158 INFO:teuthology.orchestra.run.smithi083.stdout: rgw enable usage log = true 2024-03-06T07:27:02.158 INFO:teuthology.orchestra.run.smithi083.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:27:02.158 INFO:teuthology.orchestra.run.smithi083.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:27:02.158 INFO:teuthology.orchestra.run.smithi083.stdout: client mount timeout = 600 2024-03-06T07:27:02.158 INFO:teuthology.orchestra.run.smithi083.stdout: debug client = 20 2024-03-06T07:27:02.158 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-06T07:27:02.158 INFO:teuthology.orchestra.run.smithi083.stdout: pid file = "" 2024-03-06T07:27:02.158 INFO:teuthology.orchestra.run.smithi083.stdout: rados mon op timeout = 900 2024-03-06T07:27:02.158 INFO:teuthology.orchestra.run.smithi083.stdout: rados osd op timeout = 900 2024-03-06T07:27:02.158 INFO:teuthology.orchestra.run.smithi083.stdout:[mon.a] 2024-03-06T07:27:02.159 INFO:teuthology.orchestra.run.smithi083.stdout:[mon.b] 2024-03-06T07:27:02.159 INFO:teuthology.orchestra.run.smithi083.stdout:[mon.c] 2024-03-06T07:27:02.159 INFO:teuthology.orchestra.run.smithi083.stdout:[mds.a] 2024-03-06T07:27:02.159 INFO:teuthology.orchestra.run.smithi083.stdout:[mds.c] 2024-03-06T07:27:02.159 INFO:teuthology.orchestra.run.smithi083.stdout:[mds.b] 2024-03-06T07:27:02.159 INFO:teuthology.orchestra.run.smithi083.stdout:[mds.d] 2024-03-06T07:27:02.159 INFO:teuthology.orchestra.run.smithi083.stdout:[mds] 2024-03-06T07:27:02.159 INFO:teuthology.orchestra.run.smithi083.stdout: debug mds = 20 2024-03-06T07:27:02.159 INFO:teuthology.orchestra.run.smithi083.stdout: debug mds balancer = 20 2024-03-06T07:27:02.159 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-06T07:27:02.159 INFO:teuthology.orchestra.run.smithi083.stdout: mds debug frag = True 2024-03-06T07:27:02.159 INFO:teuthology.orchestra.run.smithi083.stdout: mds debug scatterstat = True 2024-03-06T07:27:02.159 INFO:teuthology.orchestra.run.smithi083.stdout: mds op complaint time = 180 2024-03-06T07:27:02.159 INFO:teuthology.orchestra.run.smithi083.stdout: mds verify scatter = True 2024-03-06T07:27:02.159 INFO:teuthology.orchestra.run.smithi083.stdout: osd op complaint time = 180 2024-03-06T07:27:02.159 INFO:teuthology.orchestra.run.smithi083.stdout: rados mon op timeout = 900 2024-03-06T07:27:02.159 INFO:teuthology.orchestra.run.smithi083.stdout: rados osd op timeout = 900 2024-03-06T07:27:02.160 INFO:teuthology.orchestra.run.smithi083.stdout: mds_max_purge_files = 6 2024-03-06T07:27:02.160 INFO:teuthology.orchestra.run.smithi083.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:27:02.160 INFO:teuthology.orchestra.run.smithi183.stdout:[global] 2024-03-06T07:27:02.160 INFO:teuthology.orchestra.run.smithi183.stdout: chdir = "" 2024-03-06T07:27:02.160 INFO:teuthology.orchestra.run.smithi183.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:27:02.160 INFO:teuthology.orchestra.run.smithi183.stdout: auth supported = cephx 2024-03-06T07:27:02.160 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.160 INFO:teuthology.orchestra.run.smithi183.stdout: filestore xattr use omap = true 2024-03-06T07:27:02.160 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.161 INFO:teuthology.orchestra.run.smithi183.stdout: mon clock drift allowed = 1.000 2024-03-06T07:27:02.161 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.161 INFO:teuthology.orchestra.run.smithi183.stdout: osd crush chooseleaf type = 0 2024-03-06T07:27:02.161 INFO:teuthology.orchestra.run.smithi183.stdout: auth debug = true 2024-03-06T07:27:02.161 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.161 INFO:teuthology.orchestra.run.smithi183.stdout: ms die on old message = true 2024-03-06T07:27:02.161 INFO:teuthology.orchestra.run.smithi183.stdout: ms die on bug = true 2024-03-06T07:27:02.161 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.161 INFO:teuthology.orchestra.run.smithi183.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:27:02.161 INFO:teuthology.orchestra.run.smithi183.stdout: mon pg warn max object skew = 0 2024-03-06T07:27:02.161 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.161 INFO:teuthology.orchestra.run.smithi183.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:27:02.161 INFO:teuthology.orchestra.run.smithi183.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:27:02.161 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.161 INFO:teuthology.orchestra.run.smithi183.stdout: osd pool default size = 2 2024-03-06T07:27:02.162 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.162 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd allow primary affinity = true 2024-03-06T07:27:02.162 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd allow pg remap = true 2024-03-06T07:27:02.162 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:27:02.162 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:27:02.162 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on no sortbitwise = false 2024-03-06T07:27:02.162 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:27:02.162 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on too few osds = false 2024-03-06T07:27:02.162 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:27:02.162 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:27:02.162 INFO:teuthology.orchestra.run.smithi183.stdout: mon_allow_pool_size_one = true 2024-03-06T07:27:02.162 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.162 INFO:teuthology.orchestra.run.smithi183.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:27:02.162 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.162 INFO:teuthology.orchestra.run.smithi183.stdout: osd default data pool replay window = 5 2024-03-06T07:27:02.163 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.163 INFO:teuthology.orchestra.run.smithi183.stdout: mon allow pool delete = true 2024-03-06T07:27:02.163 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.163 INFO:teuthology.orchestra.run.smithi183.stdout: mon cluster log file level = debug 2024-03-06T07:27:02.163 INFO:teuthology.orchestra.run.smithi183.stdout: debug asserts on shutdown = true 2024-03-06T07:27:02.163 INFO:teuthology.orchestra.run.smithi183.stdout: mon health detail to clog = false 2024-03-06T07:27:02.163 INFO:teuthology.orchestra.run.smithi183.stdout: mon host = "172.21.15.83,172.21.15.183,[v2:172.21.15.183:3301,v1:172.21.15.183:6790]" 2024-03-06T07:27:02.163 INFO:teuthology.orchestra.run.smithi183.stdout: fsid = 039877dd-d2ce-45d2-8e77-ac5eb7e1d55a 2024-03-06T07:27:02.163 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.163 INFO:teuthology.orchestra.run.smithi183.stdout:[osd] 2024-03-06T07:27:02.163 INFO:teuthology.orchestra.run.smithi183.stdout: osd journal size = 100 2024-03-06T07:27:02.163 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.163 INFO:teuthology.orchestra.run.smithi183.stdout: osd scrub load threshold = 5.0 2024-03-06T07:27:02.163 INFO:teuthology.orchestra.run.smithi183.stdout: osd scrub max interval = 600 2024-03-06T07:27:02.163 INFO:teuthology.orchestra.run.smithi183.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:27:02.164 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.164 INFO:teuthology.orchestra.run.smithi183.stdout: osd recover clone overlap = true 2024-03-06T07:27:02.164 INFO:teuthology.orchestra.run.smithi183.stdout: osd recovery max chunk = 1048576 2024-03-06T07:27:02.164 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.164 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug shutdown = true 2024-03-06T07:27:02.164 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug op order = true 2024-03-06T07:27:02.164 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug verify stray on activate = true 2024-03-06T07:27:02.164 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug trim objects = true 2024-03-06T07:27:02.164 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.164 INFO:teuthology.orchestra.run.smithi183.stdout: osd open classes on start = true 2024-03-06T07:27:02.164 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug pg log writeout = true 2024-03-06T07:27:02.164 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.164 INFO:teuthology.orchestra.run.smithi183.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:27:02.164 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.165 INFO:teuthology.orchestra.run.smithi183.stdout: osd map max advance = 10 2024-03-06T07:27:02.165 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.165 INFO:teuthology.orchestra.run.smithi183.stdout: journal zero on create = true 2024-03-06T07:27:02.165 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.165 INFO:teuthology.orchestra.run.smithi183.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:27:02.165 INFO:teuthology.orchestra.run.smithi183.stdout: filestore apply finisher threads = 3 2024-03-06T07:27:02.165 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.165 INFO:teuthology.orchestra.run.smithi183.stdout: bdev debug aio = true 2024-03-06T07:27:02.165 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug misdirected ops = true 2024-03-06T07:27:02.165 INFO:teuthology.orchestra.run.smithi183.stdout: bdev async discard = True 2024-03-06T07:27:02.165 INFO:teuthology.orchestra.run.smithi183.stdout: bdev enable discard = True 2024-03-06T07:27:02.165 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore allocator = bitmap 2024-03-06T07:27:02.165 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore block size = 96636764160 2024-03-06T07:27:02.165 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore fsck on mount = True 2024-03-06T07:27:02.166 INFO:teuthology.orchestra.run.smithi183.stdout: debug bluefs = 1/20 2024-03-06T07:27:02.166 INFO:teuthology.orchestra.run.smithi183.stdout: debug bluestore = 1/20 2024-03-06T07:27:02.166 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-06T07:27:02.166 INFO:teuthology.orchestra.run.smithi183.stdout: debug osd = 20 2024-03-06T07:27:02.166 INFO:teuthology.orchestra.run.smithi183.stdout: debug rocksdb = 4/10 2024-03-06T07:27:02.166 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:27:02.166 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd full ratio = 0.9 2024-03-06T07:27:02.166 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:27:02.166 INFO:teuthology.orchestra.run.smithi183.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:27:02.166 INFO:teuthology.orchestra.run.smithi183.stdout: osd objectstore = bluestore 2024-03-06T07:27:02.166 INFO:teuthology.orchestra.run.smithi183.stdout: osd op complaint time = 180 2024-03-06T07:27:02.166 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.166 INFO:teuthology.orchestra.run.smithi183.stdout:[mgr] 2024-03-06T07:27:02.166 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-06T07:27:02.167 INFO:teuthology.orchestra.run.smithi183.stdout: debug mgr = 20 2024-03-06T07:27:02.167 INFO:teuthology.orchestra.run.smithi183.stdout: debug mon = 20 2024-03-06T07:27:02.167 INFO:teuthology.orchestra.run.smithi183.stdout: debug auth = 20 2024-03-06T07:27:02.167 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:02.167 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:02.167 INFO:teuthology.orchestra.run.smithi183.stdout: mgr/telemetry/nag = false 2024-03-06T07:27:02.167 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.167 INFO:teuthology.orchestra.run.smithi183.stdout:[mon] 2024-03-06T07:27:02.167 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-06T07:27:02.167 INFO:teuthology.orchestra.run.smithi183.stdout: debug mon = 20 2024-03-06T07:27:02.167 INFO:teuthology.orchestra.run.smithi183.stdout: debug paxos = 20 2024-03-06T07:27:02.167 INFO:teuthology.orchestra.run.smithi183.stdout: debug auth = 20 2024-03-06T07:27:02.167 INFO:teuthology.orchestra.run.smithi183.stdout: mon data avail warn = 5 2024-03-06T07:27:02.167 INFO:teuthology.orchestra.run.smithi183.stdout: mon mgr mkfs grace = 240 2024-03-06T07:27:02.167 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:02.168 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd reporter subtree level = osd 2024-03-06T07:27:02.168 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd prime pg temp = true 2024-03-06T07:27:02.168 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:02.168 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.168 INFO:teuthology.orchestra.run.smithi183.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:27:02.168 INFO:teuthology.orchestra.run.smithi183.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:27:02.168 INFO:teuthology.orchestra.run.smithi183.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:27:02.168 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.168 INFO:teuthology.orchestra.run.smithi183.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:27:02.168 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:27:02.168 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:27:02.168 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.168 INFO:teuthology.orchestra.run.smithi183.stdout: # 1m isn't quite enough 2024-03-06T07:27:02.169 INFO:teuthology.orchestra.run.smithi183.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:27:02.169 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.169 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:27:02.169 INFO:teuthology.orchestra.run.smithi183.stdout: mon op complaint time = 120 2024-03-06T07:27:02.169 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.169 INFO:teuthology.orchestra.run.smithi183.stdout:[client] 2024-03-06T07:27:02.169 INFO:teuthology.orchestra.run.smithi183.stdout: rgw cache enabled = true 2024-03-06T07:27:02.169 INFO:teuthology.orchestra.run.smithi183.stdout: rgw enable ops log = true 2024-03-06T07:27:02.169 INFO:teuthology.orchestra.run.smithi183.stdout: rgw enable usage log = true 2024-03-06T07:27:02.169 INFO:teuthology.orchestra.run.smithi183.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:27:02.169 INFO:teuthology.orchestra.run.smithi183.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:27:02.169 INFO:teuthology.orchestra.run.smithi183.stdout: client mount timeout = 600 2024-03-06T07:27:02.169 INFO:teuthology.orchestra.run.smithi183.stdout: debug client = 20 2024-03-06T07:27:02.169 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-06T07:27:02.169 INFO:teuthology.orchestra.run.smithi183.stdout: pid file = "" 2024-03-06T07:27:02.169 INFO:teuthology.orchestra.run.smithi183.stdout: rados mon op timeout = 900 2024-03-06T07:27:02.170 INFO:teuthology.orchestra.run.smithi183.stdout: rados osd op timeout = 900 2024-03-06T07:27:02.170 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.a] 2024-03-06T07:27:02.170 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.b] 2024-03-06T07:27:02.170 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.c] 2024-03-06T07:27:02.170 INFO:teuthology.orchestra.run.smithi183.stdout:[mds.a] 2024-03-06T07:27:02.170 INFO:teuthology.orchestra.run.smithi183.stdout:[mds.c] 2024-03-06T07:27:02.170 INFO:teuthology.orchestra.run.smithi183.stdout:[mds.b] 2024-03-06T07:27:02.170 INFO:teuthology.orchestra.run.smithi183.stdout:[mds.d] 2024-03-06T07:27:02.170 INFO:teuthology.orchestra.run.smithi183.stdout:[mds] 2024-03-06T07:27:02.170 INFO:teuthology.orchestra.run.smithi183.stdout: debug mds = 20 2024-03-06T07:27:02.170 INFO:teuthology.orchestra.run.smithi183.stdout: debug mds balancer = 20 2024-03-06T07:27:02.170 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-06T07:27:02.170 INFO:teuthology.orchestra.run.smithi183.stdout: mds debug frag = True 2024-03-06T07:27:02.170 INFO:teuthology.orchestra.run.smithi183.stdout: mds debug scatterstat = True 2024-03-06T07:27:02.170 INFO:teuthology.orchestra.run.smithi183.stdout: mds op complaint time = 180 2024-03-06T07:27:02.170 INFO:teuthology.orchestra.run.smithi183.stdout: mds verify scatter = True 2024-03-06T07:27:02.170 INFO:teuthology.orchestra.run.smithi183.stdout: osd op complaint time = 180 2024-03-06T07:27:02.171 INFO:teuthology.orchestra.run.smithi183.stdout: rados mon op timeout = 900 2024-03-06T07:27:02.171 INFO:teuthology.orchestra.run.smithi183.stdout: rados osd op timeout = 900 2024-03-06T07:27:02.171 INFO:teuthology.orchestra.run.smithi183.stdout: mds_max_purge_files = 6 2024-03-06T07:27:02.171 INFO:teuthology.orchestra.run.smithi183.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:27:02.173 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:27:02.215 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:27:02.264 INFO:teuthology.orchestra.run.smithi083.stdout:[global] 2024-03-06T07:27:02.264 INFO:teuthology.orchestra.run.smithi083.stdout: chdir = "" 2024-03-06T07:27:02.264 INFO:teuthology.orchestra.run.smithi083.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:27:02.264 INFO:teuthology.orchestra.run.smithi083.stdout: auth supported = cephx 2024-03-06T07:27:02.264 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.264 INFO:teuthology.orchestra.run.smithi083.stdout: filestore xattr use omap = true 2024-03-06T07:27:02.264 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.265 INFO:teuthology.orchestra.run.smithi083.stdout: mon clock drift allowed = 1.000 2024-03-06T07:27:02.265 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.265 INFO:teuthology.orchestra.run.smithi083.stdout: osd crush chooseleaf type = 0 2024-03-06T07:27:02.265 INFO:teuthology.orchestra.run.smithi083.stdout: auth debug = true 2024-03-06T07:27:02.265 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.265 INFO:teuthology.orchestra.run.smithi083.stdout: ms die on old message = true 2024-03-06T07:27:02.265 INFO:teuthology.orchestra.run.smithi083.stdout: ms die on bug = true 2024-03-06T07:27:02.265 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.265 INFO:teuthology.orchestra.run.smithi083.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:27:02.265 INFO:teuthology.orchestra.run.smithi083.stdout: mon pg warn max object skew = 0 2024-03-06T07:27:02.266 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.266 INFO:teuthology.orchestra.run.smithi083.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:27:02.266 INFO:teuthology.orchestra.run.smithi083.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:27:02.266 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.266 INFO:teuthology.orchestra.run.smithi083.stdout: osd pool default size = 2 2024-03-06T07:27:02.266 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.266 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd allow primary affinity = true 2024-03-06T07:27:02.266 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd allow pg remap = true 2024-03-06T07:27:02.266 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:27:02.266 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:27:02.266 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on no sortbitwise = false 2024-03-06T07:27:02.267 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:27:02.267 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on too few osds = false 2024-03-06T07:27:02.267 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:27:02.267 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:27:02.267 INFO:teuthology.orchestra.run.smithi083.stdout: mon_allow_pool_size_one = true 2024-03-06T07:27:02.267 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.267 INFO:teuthology.orchestra.run.smithi083.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:27:02.267 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.267 INFO:teuthology.orchestra.run.smithi083.stdout: osd default data pool replay window = 5 2024-03-06T07:27:02.267 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.267 INFO:teuthology.orchestra.run.smithi083.stdout: mon allow pool delete = true 2024-03-06T07:27:02.268 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.268 INFO:teuthology.orchestra.run.smithi083.stdout: mon cluster log file level = debug 2024-03-06T07:27:02.268 INFO:teuthology.orchestra.run.smithi083.stdout: debug asserts on shutdown = true 2024-03-06T07:27:02.271 INFO:teuthology.orchestra.run.smithi083.stdout: mon health detail to clog = false 2024-03-06T07:27:02.271 INFO:teuthology.orchestra.run.smithi083.stdout: mon host = "172.21.15.83,172.21.15.183,[v2:172.21.15.183:3301,v1:172.21.15.183:6790]" 2024-03-06T07:27:02.271 INFO:teuthology.orchestra.run.smithi083.stdout: fsid = 039877dd-d2ce-45d2-8e77-ac5eb7e1d55a 2024-03-06T07:27:02.271 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.272 INFO:teuthology.orchestra.run.smithi083.stdout:[osd] 2024-03-06T07:27:02.272 INFO:teuthology.orchestra.run.smithi083.stdout: osd journal size = 100 2024-03-06T07:27:02.272 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.272 INFO:teuthology.orchestra.run.smithi083.stdout: osd scrub load threshold = 5.0 2024-03-06T07:27:02.272 INFO:teuthology.orchestra.run.smithi083.stdout: osd scrub max interval = 600 2024-03-06T07:27:02.272 INFO:teuthology.orchestra.run.smithi083.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:27:02.272 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.272 INFO:teuthology.orchestra.run.smithi083.stdout: osd recover clone overlap = true 2024-03-06T07:27:02.272 INFO:teuthology.orchestra.run.smithi083.stdout: osd recovery max chunk = 1048576 2024-03-06T07:27:02.272 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.272 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug shutdown = true 2024-03-06T07:27:02.272 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug op order = true 2024-03-06T07:27:02.272 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug verify stray on activate = true 2024-03-06T07:27:02.272 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug trim objects = true 2024-03-06T07:27:02.273 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.273 INFO:teuthology.orchestra.run.smithi083.stdout: osd open classes on start = true 2024-03-06T07:27:02.273 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug pg log writeout = true 2024-03-06T07:27:02.273 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.273 INFO:teuthology.orchestra.run.smithi083.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:27:02.273 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.273 INFO:teuthology.orchestra.run.smithi083.stdout: osd map max advance = 10 2024-03-06T07:27:02.273 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.273 INFO:teuthology.orchestra.run.smithi083.stdout: journal zero on create = true 2024-03-06T07:27:02.273 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.273 INFO:teuthology.orchestra.run.smithi083.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:27:02.273 INFO:teuthology.orchestra.run.smithi083.stdout: filestore apply finisher threads = 3 2024-03-06T07:27:02.273 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.273 INFO:teuthology.orchestra.run.smithi083.stdout: bdev debug aio = true 2024-03-06T07:27:02.273 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug misdirected ops = true 2024-03-06T07:27:02.274 INFO:teuthology.orchestra.run.smithi083.stdout: bdev async discard = True 2024-03-06T07:27:02.274 INFO:teuthology.orchestra.run.smithi083.stdout: bdev enable discard = True 2024-03-06T07:27:02.274 INFO:teuthology.orchestra.run.smithi083.stdout: bluestore allocator = bitmap 2024-03-06T07:27:02.274 INFO:teuthology.orchestra.run.smithi083.stdout: bluestore block size = 96636764160 2024-03-06T07:27:02.274 INFO:teuthology.orchestra.run.smithi083.stdout: bluestore fsck on mount = True 2024-03-06T07:27:02.274 INFO:teuthology.orchestra.run.smithi083.stdout: debug bluefs = 1/20 2024-03-06T07:27:02.274 INFO:teuthology.orchestra.run.smithi083.stdout: debug bluestore = 1/20 2024-03-06T07:27:02.274 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-06T07:27:02.274 INFO:teuthology.orchestra.run.smithi083.stdout: debug osd = 20 2024-03-06T07:27:02.274 INFO:teuthology.orchestra.run.smithi083.stdout: debug rocksdb = 4/10 2024-03-06T07:27:02.274 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:27:02.275 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd full ratio = 0.9 2024-03-06T07:27:02.275 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:27:02.275 INFO:teuthology.orchestra.run.smithi083.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:27:02.275 INFO:teuthology.orchestra.run.smithi083.stdout: osd objectstore = bluestore 2024-03-06T07:27:02.275 INFO:teuthology.orchestra.run.smithi083.stdout: osd op complaint time = 180 2024-03-06T07:27:02.275 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.275 INFO:teuthology.orchestra.run.smithi083.stdout:[mgr] 2024-03-06T07:27:02.275 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-06T07:27:02.275 INFO:teuthology.orchestra.run.smithi083.stdout: debug mgr = 20 2024-03-06T07:27:02.275 INFO:teuthology.orchestra.run.smithi083.stdout: debug mon = 20 2024-03-06T07:27:02.275 INFO:teuthology.orchestra.run.smithi083.stdout: debug auth = 20 2024-03-06T07:27:02.275 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:02.276 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:02.276 INFO:teuthology.orchestra.run.smithi083.stdout: mgr/telemetry/nag = false 2024-03-06T07:27:02.276 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.276 INFO:teuthology.orchestra.run.smithi083.stdout:[mon] 2024-03-06T07:27:02.276 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-06T07:27:02.276 INFO:teuthology.orchestra.run.smithi083.stdout: debug mon = 20 2024-03-06T07:27:02.276 INFO:teuthology.orchestra.run.smithi083.stdout: debug paxos = 20 2024-03-06T07:27:02.276 INFO:teuthology.orchestra.run.smithi083.stdout: debug auth = 20 2024-03-06T07:27:02.276 INFO:teuthology.orchestra.run.smithi083.stdout: mon data avail warn = 5 2024-03-06T07:27:02.276 INFO:teuthology.orchestra.run.smithi083.stdout: mon mgr mkfs grace = 240 2024-03-06T07:27:02.276 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:02.276 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd reporter subtree level = osd 2024-03-06T07:27:02.276 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd prime pg temp = true 2024-03-06T07:27:02.277 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:02.277 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.277 INFO:teuthology.orchestra.run.smithi083.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:27:02.277 INFO:teuthology.orchestra.run.smithi083.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:27:02.277 INFO:teuthology.orchestra.run.smithi083.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:27:02.277 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.277 INFO:teuthology.orchestra.run.smithi083.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:27:02.277 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:27:02.277 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:27:02.277 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.277 INFO:teuthology.orchestra.run.smithi083.stdout: # 1m isn't quite enough 2024-03-06T07:27:02.277 INFO:teuthology.orchestra.run.smithi083.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:27:02.277 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.277 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:27:02.278 INFO:teuthology.orchestra.run.smithi083.stdout: mon op complaint time = 120 2024-03-06T07:27:02.278 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.278 INFO:teuthology.orchestra.run.smithi083.stdout:[client] 2024-03-06T07:27:02.278 INFO:teuthology.orchestra.run.smithi083.stdout: rgw cache enabled = true 2024-03-06T07:27:02.278 INFO:teuthology.orchestra.run.smithi083.stdout: rgw enable ops log = true 2024-03-06T07:27:02.278 INFO:teuthology.orchestra.run.smithi083.stdout: rgw enable usage log = true 2024-03-06T07:27:02.278 INFO:teuthology.orchestra.run.smithi083.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:27:02.278 INFO:teuthology.orchestra.run.smithi083.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:27:02.278 INFO:teuthology.orchestra.run.smithi083.stdout: client mount timeout = 600 2024-03-06T07:27:02.278 INFO:teuthology.orchestra.run.smithi083.stdout: debug client = 20 2024-03-06T07:27:02.278 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-06T07:27:02.278 INFO:teuthology.orchestra.run.smithi083.stdout: pid file = "" 2024-03-06T07:27:02.278 INFO:teuthology.orchestra.run.smithi083.stdout: rados mon op timeout = 900 2024-03-06T07:27:02.278 INFO:teuthology.orchestra.run.smithi083.stdout: rados osd op timeout = 900 2024-03-06T07:27:02.278 INFO:teuthology.orchestra.run.smithi083.stdout:[mon.a] 2024-03-06T07:27:02.279 INFO:teuthology.orchestra.run.smithi083.stdout:[mon.b] 2024-03-06T07:27:02.279 INFO:teuthology.orchestra.run.smithi083.stdout:[mon.c] 2024-03-06T07:27:02.279 INFO:teuthology.orchestra.run.smithi083.stdout:[mds.a] 2024-03-06T07:27:02.279 INFO:teuthology.orchestra.run.smithi083.stdout:[mds.c] 2024-03-06T07:27:02.279 INFO:teuthology.orchestra.run.smithi083.stdout:[mds.b] 2024-03-06T07:27:02.279 INFO:teuthology.orchestra.run.smithi083.stdout:[mds.d] 2024-03-06T07:27:02.279 INFO:teuthology.orchestra.run.smithi083.stdout:[mds] 2024-03-06T07:27:02.279 INFO:teuthology.orchestra.run.smithi083.stdout: debug mds = 20 2024-03-06T07:27:02.279 INFO:teuthology.orchestra.run.smithi083.stdout: debug mds balancer = 20 2024-03-06T07:27:02.279 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-06T07:27:02.279 INFO:teuthology.orchestra.run.smithi083.stdout: mds debug frag = True 2024-03-06T07:27:02.279 INFO:teuthology.orchestra.run.smithi083.stdout: mds debug scatterstat = True 2024-03-06T07:27:02.280 INFO:teuthology.orchestra.run.smithi083.stdout: mds op complaint time = 180 2024-03-06T07:27:02.280 INFO:teuthology.orchestra.run.smithi083.stdout: mds verify scatter = True 2024-03-06T07:27:02.280 INFO:teuthology.orchestra.run.smithi083.stdout: osd op complaint time = 180 2024-03-06T07:27:02.280 INFO:teuthology.orchestra.run.smithi083.stdout: rados mon op timeout = 900 2024-03-06T07:27:02.280 INFO:teuthology.orchestra.run.smithi083.stdout: rados osd op timeout = 900 2024-03-06T07:27:02.280 INFO:teuthology.orchestra.run.smithi083.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:27:02.281 INFO:teuthology.orchestra.run.smithi183.stdout:[global] 2024-03-06T07:27:02.281 INFO:teuthology.orchestra.run.smithi183.stdout: chdir = "" 2024-03-06T07:27:02.281 INFO:teuthology.orchestra.run.smithi183.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:27:02.281 INFO:teuthology.orchestra.run.smithi183.stdout: auth supported = cephx 2024-03-06T07:27:02.281 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.281 INFO:teuthology.orchestra.run.smithi183.stdout: filestore xattr use omap = true 2024-03-06T07:27:02.281 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.281 INFO:teuthology.orchestra.run.smithi183.stdout: mon clock drift allowed = 1.000 2024-03-06T07:27:02.281 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.281 INFO:teuthology.orchestra.run.smithi183.stdout: osd crush chooseleaf type = 0 2024-03-06T07:27:02.281 INFO:teuthology.orchestra.run.smithi183.stdout: auth debug = true 2024-03-06T07:27:02.281 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.281 INFO:teuthology.orchestra.run.smithi183.stdout: ms die on old message = true 2024-03-06T07:27:02.282 INFO:teuthology.orchestra.run.smithi183.stdout: ms die on bug = true 2024-03-06T07:27:02.282 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.282 INFO:teuthology.orchestra.run.smithi183.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:27:02.282 INFO:teuthology.orchestra.run.smithi183.stdout: mon pg warn max object skew = 0 2024-03-06T07:27:02.282 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.282 INFO:teuthology.orchestra.run.smithi183.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:27:02.282 INFO:teuthology.orchestra.run.smithi183.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:27:02.282 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.282 INFO:teuthology.orchestra.run.smithi183.stdout: osd pool default size = 2 2024-03-06T07:27:02.282 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.282 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd allow primary affinity = true 2024-03-06T07:27:02.282 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd allow pg remap = true 2024-03-06T07:27:02.282 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:27:02.282 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:27:02.282 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on no sortbitwise = false 2024-03-06T07:27:02.283 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:27:02.283 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on too few osds = false 2024-03-06T07:27:02.283 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:27:02.283 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:27:02.283 INFO:teuthology.orchestra.run.smithi183.stdout: mon_allow_pool_size_one = true 2024-03-06T07:27:02.283 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.283 INFO:teuthology.orchestra.run.smithi183.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:27:02.283 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.283 INFO:teuthology.orchestra.run.smithi183.stdout: osd default data pool replay window = 5 2024-03-06T07:27:02.283 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.283 INFO:teuthology.orchestra.run.smithi183.stdout: mon allow pool delete = true 2024-03-06T07:27:02.283 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.283 INFO:teuthology.orchestra.run.smithi183.stdout: mon cluster log file level = debug 2024-03-06T07:27:02.283 INFO:teuthology.orchestra.run.smithi183.stdout: debug asserts on shutdown = true 2024-03-06T07:27:02.283 INFO:teuthology.orchestra.run.smithi183.stdout: mon health detail to clog = false 2024-03-06T07:27:02.284 INFO:teuthology.orchestra.run.smithi183.stdout: mon host = "172.21.15.83,172.21.15.183,[v2:172.21.15.183:3301,v1:172.21.15.183:6790]" 2024-03-06T07:27:02.284 INFO:teuthology.orchestra.run.smithi183.stdout: fsid = 039877dd-d2ce-45d2-8e77-ac5eb7e1d55a 2024-03-06T07:27:02.284 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.284 INFO:teuthology.orchestra.run.smithi183.stdout:[osd] 2024-03-06T07:27:02.284 INFO:teuthology.orchestra.run.smithi183.stdout: osd journal size = 100 2024-03-06T07:27:02.284 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.284 INFO:teuthology.orchestra.run.smithi183.stdout: osd scrub load threshold = 5.0 2024-03-06T07:27:02.284 INFO:teuthology.orchestra.run.smithi183.stdout: osd scrub max interval = 600 2024-03-06T07:27:02.284 INFO:teuthology.orchestra.run.smithi183.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:27:02.284 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.284 INFO:teuthology.orchestra.run.smithi183.stdout: osd recover clone overlap = true 2024-03-06T07:27:02.284 INFO:teuthology.orchestra.run.smithi183.stdout: osd recovery max chunk = 1048576 2024-03-06T07:27:02.284 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.284 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug shutdown = true 2024-03-06T07:27:02.285 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug op order = true 2024-03-06T07:27:02.285 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug verify stray on activate = true 2024-03-06T07:27:02.285 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug trim objects = true 2024-03-06T07:27:02.285 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.285 INFO:teuthology.orchestra.run.smithi183.stdout: osd open classes on start = true 2024-03-06T07:27:02.285 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug pg log writeout = true 2024-03-06T07:27:02.285 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.285 INFO:teuthology.orchestra.run.smithi183.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:27:02.285 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.285 INFO:teuthology.orchestra.run.smithi183.stdout: osd map max advance = 10 2024-03-06T07:27:02.285 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.285 INFO:teuthology.orchestra.run.smithi183.stdout: journal zero on create = true 2024-03-06T07:27:02.285 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.285 INFO:teuthology.orchestra.run.smithi183.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:27:02.286 INFO:teuthology.orchestra.run.smithi183.stdout: filestore apply finisher threads = 3 2024-03-06T07:27:02.286 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.286 INFO:teuthology.orchestra.run.smithi183.stdout: bdev debug aio = true 2024-03-06T07:27:02.286 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug misdirected ops = true 2024-03-06T07:27:02.286 INFO:teuthology.orchestra.run.smithi183.stdout: bdev async discard = True 2024-03-06T07:27:02.286 INFO:teuthology.orchestra.run.smithi183.stdout: bdev enable discard = True 2024-03-06T07:27:02.286 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore allocator = bitmap 2024-03-06T07:27:02.286 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore block size = 96636764160 2024-03-06T07:27:02.286 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore fsck on mount = True 2024-03-06T07:27:02.286 INFO:teuthology.orchestra.run.smithi183.stdout: debug bluefs = 1/20 2024-03-06T07:27:02.286 INFO:teuthology.orchestra.run.smithi183.stdout: debug bluestore = 1/20 2024-03-06T07:27:02.286 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-06T07:27:02.286 INFO:teuthology.orchestra.run.smithi183.stdout: debug osd = 20 2024-03-06T07:27:02.286 INFO:teuthology.orchestra.run.smithi183.stdout: debug rocksdb = 4/10 2024-03-06T07:27:02.287 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:27:02.287 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd full ratio = 0.9 2024-03-06T07:27:02.287 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:27:02.287 INFO:teuthology.orchestra.run.smithi183.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:27:02.287 INFO:teuthology.orchestra.run.smithi183.stdout: osd objectstore = bluestore 2024-03-06T07:27:02.287 INFO:teuthology.orchestra.run.smithi183.stdout: osd op complaint time = 180 2024-03-06T07:27:02.287 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.287 INFO:teuthology.orchestra.run.smithi183.stdout:[mgr] 2024-03-06T07:27:02.287 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-06T07:27:02.287 INFO:teuthology.orchestra.run.smithi183.stdout: debug mgr = 20 2024-03-06T07:27:02.287 INFO:teuthology.orchestra.run.smithi183.stdout: debug mon = 20 2024-03-06T07:27:02.287 INFO:teuthology.orchestra.run.smithi183.stdout: debug auth = 20 2024-03-06T07:27:02.287 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:02.287 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:02.288 INFO:teuthology.orchestra.run.smithi183.stdout: mgr/telemetry/nag = false 2024-03-06T07:27:02.288 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.288 INFO:teuthology.orchestra.run.smithi183.stdout:[mon] 2024-03-06T07:27:02.288 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-06T07:27:02.288 INFO:teuthology.orchestra.run.smithi183.stdout: debug mon = 20 2024-03-06T07:27:02.288 INFO:teuthology.orchestra.run.smithi183.stdout: debug paxos = 20 2024-03-06T07:27:02.288 INFO:teuthology.orchestra.run.smithi183.stdout: debug auth = 20 2024-03-06T07:27:02.288 INFO:teuthology.orchestra.run.smithi183.stdout: mon data avail warn = 5 2024-03-06T07:27:02.288 INFO:teuthology.orchestra.run.smithi183.stdout: mon mgr mkfs grace = 240 2024-03-06T07:27:02.288 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:02.288 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd reporter subtree level = osd 2024-03-06T07:27:02.288 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd prime pg temp = true 2024-03-06T07:27:02.288 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:02.288 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.288 INFO:teuthology.orchestra.run.smithi183.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:27:02.289 INFO:teuthology.orchestra.run.smithi183.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:27:02.289 INFO:teuthology.orchestra.run.smithi183.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:27:02.289 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.289 INFO:teuthology.orchestra.run.smithi183.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:27:02.289 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:27:02.289 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:27:02.289 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.289 INFO:teuthology.orchestra.run.smithi183.stdout: # 1m isn't quite enough 2024-03-06T07:27:02.289 INFO:teuthology.orchestra.run.smithi183.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:27:02.289 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.289 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:27:02.289 INFO:teuthology.orchestra.run.smithi183.stdout: mon op complaint time = 120 2024-03-06T07:27:02.289 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.289 INFO:teuthology.orchestra.run.smithi183.stdout:[client] 2024-03-06T07:27:02.290 INFO:teuthology.orchestra.run.smithi183.stdout: rgw cache enabled = true 2024-03-06T07:27:02.290 INFO:teuthology.orchestra.run.smithi183.stdout: rgw enable ops log = true 2024-03-06T07:27:02.290 INFO:teuthology.orchestra.run.smithi183.stdout: rgw enable usage log = true 2024-03-06T07:27:02.290 INFO:teuthology.orchestra.run.smithi183.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:27:02.290 INFO:teuthology.orchestra.run.smithi183.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:27:02.290 INFO:teuthology.orchestra.run.smithi183.stdout: client mount timeout = 600 2024-03-06T07:27:02.290 INFO:teuthology.orchestra.run.smithi183.stdout: debug client = 20 2024-03-06T07:27:02.290 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-06T07:27:02.290 INFO:teuthology.orchestra.run.smithi183.stdout: pid file = "" 2024-03-06T07:27:02.290 INFO:teuthology.orchestra.run.smithi183.stdout: rados mon op timeout = 900 2024-03-06T07:27:02.290 INFO:teuthology.orchestra.run.smithi183.stdout: rados osd op timeout = 900 2024-03-06T07:27:02.290 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.a] 2024-03-06T07:27:02.290 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.b] 2024-03-06T07:27:02.290 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.c] 2024-03-06T07:27:02.291 INFO:teuthology.orchestra.run.smithi183.stdout:[mds.a] 2024-03-06T07:27:02.291 INFO:teuthology.orchestra.run.smithi183.stdout:[mds.c] 2024-03-06T07:27:02.291 INFO:teuthology.orchestra.run.smithi183.stdout:[mds.b] 2024-03-06T07:27:02.291 INFO:teuthology.orchestra.run.smithi183.stdout:[mds.d] 2024-03-06T07:27:02.291 INFO:teuthology.orchestra.run.smithi183.stdout:[mds] 2024-03-06T07:27:02.291 INFO:teuthology.orchestra.run.smithi183.stdout: debug mds = 20 2024-03-06T07:27:02.291 INFO:teuthology.orchestra.run.smithi183.stdout: debug mds balancer = 20 2024-03-06T07:27:02.291 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-06T07:27:02.291 INFO:teuthology.orchestra.run.smithi183.stdout: mds debug frag = True 2024-03-06T07:27:02.291 INFO:teuthology.orchestra.run.smithi183.stdout: mds debug scatterstat = True 2024-03-06T07:27:02.291 INFO:teuthology.orchestra.run.smithi183.stdout: mds op complaint time = 180 2024-03-06T07:27:02.291 INFO:teuthology.orchestra.run.smithi183.stdout: mds verify scatter = True 2024-03-06T07:27:02.291 INFO:teuthology.orchestra.run.smithi183.stdout: osd op complaint time = 180 2024-03-06T07:27:02.291 INFO:teuthology.orchestra.run.smithi183.stdout: rados mon op timeout = 900 2024-03-06T07:27:02.291 INFO:teuthology.orchestra.run.smithi183.stdout: rados osd op timeout = 900 2024-03-06T07:27:02.291 INFO:teuthology.orchestra.run.smithi183.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:27:02.294 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:27:02.335 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:27:02.383 INFO:teuthology.orchestra.run.smithi083.stdout:[global] 2024-03-06T07:27:02.383 INFO:teuthology.orchestra.run.smithi083.stdout: chdir = "" 2024-03-06T07:27:02.383 INFO:teuthology.orchestra.run.smithi083.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:27:02.383 INFO:teuthology.orchestra.run.smithi083.stdout: auth supported = cephx 2024-03-06T07:27:02.383 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.383 INFO:teuthology.orchestra.run.smithi083.stdout: filestore xattr use omap = true 2024-03-06T07:27:02.384 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.384 INFO:teuthology.orchestra.run.smithi083.stdout: mon clock drift allowed = 1.000 2024-03-06T07:27:02.384 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.384 INFO:teuthology.orchestra.run.smithi083.stdout: osd crush chooseleaf type = 0 2024-03-06T07:27:02.384 INFO:teuthology.orchestra.run.smithi083.stdout: auth debug = true 2024-03-06T07:27:02.384 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.384 INFO:teuthology.orchestra.run.smithi083.stdout: ms die on old message = true 2024-03-06T07:27:02.384 INFO:teuthology.orchestra.run.smithi083.stdout: ms die on bug = true 2024-03-06T07:27:02.384 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.384 INFO:teuthology.orchestra.run.smithi083.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:27:02.384 INFO:teuthology.orchestra.run.smithi083.stdout: mon pg warn max object skew = 0 2024-03-06T07:27:02.384 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.384 INFO:teuthology.orchestra.run.smithi083.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:27:02.384 INFO:teuthology.orchestra.run.smithi083.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:27:02.385 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.385 INFO:teuthology.orchestra.run.smithi083.stdout: osd pool default size = 2 2024-03-06T07:27:02.385 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.385 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd allow primary affinity = true 2024-03-06T07:27:02.385 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd allow pg remap = true 2024-03-06T07:27:02.385 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:27:02.385 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:27:02.385 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on no sortbitwise = false 2024-03-06T07:27:02.385 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:27:02.385 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on too few osds = false 2024-03-06T07:27:02.385 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:27:02.385 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:27:02.386 INFO:teuthology.orchestra.run.smithi083.stdout: mon_allow_pool_size_one = true 2024-03-06T07:27:02.386 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.386 INFO:teuthology.orchestra.run.smithi083.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:27:02.386 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.386 INFO:teuthology.orchestra.run.smithi083.stdout: osd default data pool replay window = 5 2024-03-06T07:27:02.386 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.386 INFO:teuthology.orchestra.run.smithi083.stdout: mon allow pool delete = true 2024-03-06T07:27:02.386 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.386 INFO:teuthology.orchestra.run.smithi083.stdout: mon cluster log file level = debug 2024-03-06T07:27:02.386 INFO:teuthology.orchestra.run.smithi083.stdout: debug asserts on shutdown = true 2024-03-06T07:27:02.386 INFO:teuthology.orchestra.run.smithi083.stdout: mon health detail to clog = false 2024-03-06T07:27:02.386 INFO:teuthology.orchestra.run.smithi083.stdout: mon host = "172.21.15.83,172.21.15.183,[v2:172.21.15.183:3301,v1:172.21.15.183:6790]" 2024-03-06T07:27:02.386 INFO:teuthology.orchestra.run.smithi083.stdout: fsid = 039877dd-d2ce-45d2-8e77-ac5eb7e1d55a 2024-03-06T07:27:02.386 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.387 INFO:teuthology.orchestra.run.smithi083.stdout:[osd] 2024-03-06T07:27:02.387 INFO:teuthology.orchestra.run.smithi083.stdout: osd journal size = 100 2024-03-06T07:27:02.387 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.387 INFO:teuthology.orchestra.run.smithi083.stdout: osd scrub load threshold = 5.0 2024-03-06T07:27:02.387 INFO:teuthology.orchestra.run.smithi083.stdout: osd scrub max interval = 600 2024-03-06T07:27:02.387 INFO:teuthology.orchestra.run.smithi083.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:27:02.387 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.387 INFO:teuthology.orchestra.run.smithi083.stdout: osd recover clone overlap = true 2024-03-06T07:27:02.387 INFO:teuthology.orchestra.run.smithi083.stdout: osd recovery max chunk = 1048576 2024-03-06T07:27:02.387 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.387 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug shutdown = true 2024-03-06T07:27:02.387 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug op order = true 2024-03-06T07:27:02.387 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug verify stray on activate = true 2024-03-06T07:27:02.387 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug trim objects = true 2024-03-06T07:27:02.387 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.388 INFO:teuthology.orchestra.run.smithi083.stdout: osd open classes on start = true 2024-03-06T07:27:02.388 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug pg log writeout = true 2024-03-06T07:27:02.388 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.388 INFO:teuthology.orchestra.run.smithi083.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:27:02.388 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.388 INFO:teuthology.orchestra.run.smithi083.stdout: osd map max advance = 10 2024-03-06T07:27:02.388 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.388 INFO:teuthology.orchestra.run.smithi083.stdout: journal zero on create = true 2024-03-06T07:27:02.388 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.388 INFO:teuthology.orchestra.run.smithi083.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:27:02.388 INFO:teuthology.orchestra.run.smithi083.stdout: filestore apply finisher threads = 3 2024-03-06T07:27:02.388 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.388 INFO:teuthology.orchestra.run.smithi083.stdout: bdev debug aio = true 2024-03-06T07:27:02.388 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug misdirected ops = true 2024-03-06T07:27:02.389 INFO:teuthology.orchestra.run.smithi083.stdout: bdev async discard = True 2024-03-06T07:27:02.389 INFO:teuthology.orchestra.run.smithi083.stdout: bdev enable discard = True 2024-03-06T07:27:02.389 INFO:teuthology.orchestra.run.smithi083.stdout: bluestore allocator = bitmap 2024-03-06T07:27:02.389 INFO:teuthology.orchestra.run.smithi083.stdout: bluestore block size = 96636764160 2024-03-06T07:27:02.389 INFO:teuthology.orchestra.run.smithi083.stdout: bluestore fsck on mount = True 2024-03-06T07:27:02.389 INFO:teuthology.orchestra.run.smithi083.stdout: debug bluefs = 1/20 2024-03-06T07:27:02.389 INFO:teuthology.orchestra.run.smithi083.stdout: debug bluestore = 1/20 2024-03-06T07:27:02.389 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-06T07:27:02.389 INFO:teuthology.orchestra.run.smithi083.stdout: debug osd = 20 2024-03-06T07:27:02.389 INFO:teuthology.orchestra.run.smithi083.stdout: debug rocksdb = 4/10 2024-03-06T07:27:02.389 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:27:02.389 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd full ratio = 0.9 2024-03-06T07:27:02.389 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:27:02.389 INFO:teuthology.orchestra.run.smithi083.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:27:02.389 INFO:teuthology.orchestra.run.smithi083.stdout: osd objectstore = bluestore 2024-03-06T07:27:02.389 INFO:teuthology.orchestra.run.smithi083.stdout: osd op complaint time = 180 2024-03-06T07:27:02.390 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.390 INFO:teuthology.orchestra.run.smithi083.stdout:[mgr] 2024-03-06T07:27:02.390 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-06T07:27:02.390 INFO:teuthology.orchestra.run.smithi083.stdout: debug mgr = 20 2024-03-06T07:27:02.390 INFO:teuthology.orchestra.run.smithi083.stdout: debug mon = 20 2024-03-06T07:27:02.390 INFO:teuthology.orchestra.run.smithi083.stdout: debug auth = 20 2024-03-06T07:27:02.390 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:02.390 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:02.390 INFO:teuthology.orchestra.run.smithi083.stdout: mgr/telemetry/nag = false 2024-03-06T07:27:02.390 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.390 INFO:teuthology.orchestra.run.smithi083.stdout:[mon] 2024-03-06T07:27:02.390 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-06T07:27:02.390 INFO:teuthology.orchestra.run.smithi083.stdout: debug mon = 20 2024-03-06T07:27:02.390 INFO:teuthology.orchestra.run.smithi083.stdout: debug paxos = 20 2024-03-06T07:27:02.390 INFO:teuthology.orchestra.run.smithi083.stdout: debug auth = 20 2024-03-06T07:27:02.390 INFO:teuthology.orchestra.run.smithi083.stdout: mon data avail warn = 5 2024-03-06T07:27:02.391 INFO:teuthology.orchestra.run.smithi083.stdout: mon mgr mkfs grace = 240 2024-03-06T07:27:02.391 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:02.391 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd reporter subtree level = osd 2024-03-06T07:27:02.391 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd prime pg temp = true 2024-03-06T07:27:02.391 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:02.391 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.391 INFO:teuthology.orchestra.run.smithi083.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:27:02.391 INFO:teuthology.orchestra.run.smithi083.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:27:02.391 INFO:teuthology.orchestra.run.smithi083.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:27:02.391 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.391 INFO:teuthology.orchestra.run.smithi083.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:27:02.391 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:27:02.391 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:27:02.391 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.391 INFO:teuthology.orchestra.run.smithi083.stdout: # 1m isn't quite enough 2024-03-06T07:27:02.391 INFO:teuthology.orchestra.run.smithi083.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:27:02.392 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.392 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:27:02.392 INFO:teuthology.orchestra.run.smithi083.stdout: mon op complaint time = 120 2024-03-06T07:27:02.392 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:02.392 INFO:teuthology.orchestra.run.smithi083.stdout:[client] 2024-03-06T07:27:02.392 INFO:teuthology.orchestra.run.smithi083.stdout: rgw cache enabled = true 2024-03-06T07:27:02.392 INFO:teuthology.orchestra.run.smithi083.stdout: rgw enable ops log = true 2024-03-06T07:27:02.392 INFO:teuthology.orchestra.run.smithi083.stdout: rgw enable usage log = true 2024-03-06T07:27:02.392 INFO:teuthology.orchestra.run.smithi083.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:27:02.392 INFO:teuthology.orchestra.run.smithi083.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:27:02.392 INFO:teuthology.orchestra.run.smithi083.stdout: client mount timeout = 600 2024-03-06T07:27:02.392 INFO:teuthology.orchestra.run.smithi083.stdout: debug client = 20 2024-03-06T07:27:02.392 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-06T07:27:02.392 INFO:teuthology.orchestra.run.smithi083.stdout: pid file = "" 2024-03-06T07:27:02.392 INFO:teuthology.orchestra.run.smithi083.stdout: rados mon op timeout = 900 2024-03-06T07:27:02.393 INFO:teuthology.orchestra.run.smithi083.stdout: rados osd op timeout = 900 2024-03-06T07:27:02.393 INFO:teuthology.orchestra.run.smithi083.stdout:[mon.a] 2024-03-06T07:27:02.393 INFO:teuthology.orchestra.run.smithi083.stdout:[mon.b] 2024-03-06T07:27:02.393 INFO:teuthology.orchestra.run.smithi083.stdout:[mon.c] 2024-03-06T07:27:02.393 INFO:teuthology.orchestra.run.smithi083.stdout:[mds.a] 2024-03-06T07:27:02.393 INFO:teuthology.orchestra.run.smithi083.stdout:[mds.c] 2024-03-06T07:27:02.393 INFO:teuthology.orchestra.run.smithi083.stdout:[mds.b] 2024-03-06T07:27:02.393 INFO:teuthology.orchestra.run.smithi083.stdout:[mds.d] 2024-03-06T07:27:02.393 INFO:teuthology.orchestra.run.smithi083.stdout:[mds] 2024-03-06T07:27:02.393 INFO:teuthology.orchestra.run.smithi083.stdout: debug mds = 20 2024-03-06T07:27:02.393 INFO:teuthology.orchestra.run.smithi083.stdout: debug mds balancer = 20 2024-03-06T07:27:02.393 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-06T07:27:02.393 INFO:teuthology.orchestra.run.smithi083.stdout: mds debug frag = True 2024-03-06T07:27:02.393 INFO:teuthology.orchestra.run.smithi083.stdout: mds debug scatterstat = True 2024-03-06T07:27:02.393 INFO:teuthology.orchestra.run.smithi083.stdout: mds op complaint time = 180 2024-03-06T07:27:02.393 INFO:teuthology.orchestra.run.smithi083.stdout: mds verify scatter = True 2024-03-06T07:27:02.394 INFO:teuthology.orchestra.run.smithi083.stdout: osd op complaint time = 180 2024-03-06T07:27:02.394 INFO:teuthology.orchestra.run.smithi083.stdout: rados mon op timeout = 900 2024-03-06T07:27:02.394 INFO:teuthology.orchestra.run.smithi083.stdout: rados osd op timeout = 900 2024-03-06T07:27:02.394 INFO:teuthology.orchestra.run.smithi183.stdout:[global] 2024-03-06T07:27:02.394 INFO:teuthology.orchestra.run.smithi183.stdout: chdir = "" 2024-03-06T07:27:02.394 INFO:teuthology.orchestra.run.smithi183.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:27:02.394 INFO:teuthology.orchestra.run.smithi183.stdout: auth supported = cephx 2024-03-06T07:27:02.394 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.394 INFO:teuthology.orchestra.run.smithi183.stdout: filestore xattr use omap = true 2024-03-06T07:27:02.394 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.395 INFO:teuthology.orchestra.run.smithi183.stdout: mon clock drift allowed = 1.000 2024-03-06T07:27:02.395 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.395 INFO:teuthology.orchestra.run.smithi183.stdout: osd crush chooseleaf type = 0 2024-03-06T07:27:02.395 INFO:teuthology.orchestra.run.smithi183.stdout: auth debug = true 2024-03-06T07:27:02.395 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.395 INFO:teuthology.orchestra.run.smithi183.stdout: ms die on old message = true 2024-03-06T07:27:02.395 INFO:teuthology.orchestra.run.smithi183.stdout: ms die on bug = true 2024-03-06T07:27:02.395 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.395 INFO:teuthology.orchestra.run.smithi183.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:27:02.395 INFO:teuthology.orchestra.run.smithi183.stdout: mon pg warn max object skew = 0 2024-03-06T07:27:02.395 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.395 INFO:teuthology.orchestra.run.smithi183.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:27:02.395 INFO:teuthology.orchestra.run.smithi183.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:27:02.395 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.396 INFO:teuthology.orchestra.run.smithi183.stdout: osd pool default size = 2 2024-03-06T07:27:02.396 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.396 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd allow primary affinity = true 2024-03-06T07:27:02.396 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd allow pg remap = true 2024-03-06T07:27:02.396 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:27:02.396 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:27:02.396 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on no sortbitwise = false 2024-03-06T07:27:02.396 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:27:02.396 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on too few osds = false 2024-03-06T07:27:02.396 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:27:02.396 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:27:02.396 INFO:teuthology.orchestra.run.smithi183.stdout: mon_allow_pool_size_one = true 2024-03-06T07:27:02.396 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.397 INFO:teuthology.orchestra.run.smithi183.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:27:02.397 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.397 INFO:teuthology.orchestra.run.smithi183.stdout: osd default data pool replay window = 5 2024-03-06T07:27:02.397 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.397 INFO:teuthology.orchestra.run.smithi183.stdout: mon allow pool delete = true 2024-03-06T07:27:02.397 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.397 INFO:teuthology.orchestra.run.smithi183.stdout: mon cluster log file level = debug 2024-03-06T07:27:02.397 INFO:teuthology.orchestra.run.smithi183.stdout: debug asserts on shutdown = true 2024-03-06T07:27:02.397 INFO:teuthology.orchestra.run.smithi183.stdout: mon health detail to clog = false 2024-03-06T07:27:02.397 INFO:teuthology.orchestra.run.smithi183.stdout: mon host = "172.21.15.83,172.21.15.183,[v2:172.21.15.183:3301,v1:172.21.15.183:6790]" 2024-03-06T07:27:02.397 INFO:teuthology.orchestra.run.smithi183.stdout: fsid = 039877dd-d2ce-45d2-8e77-ac5eb7e1d55a 2024-03-06T07:27:02.397 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.398 INFO:teuthology.orchestra.run.smithi183.stdout:[osd] 2024-03-06T07:27:02.398 INFO:teuthology.orchestra.run.smithi183.stdout: osd journal size = 100 2024-03-06T07:27:02.398 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.398 INFO:teuthology.orchestra.run.smithi183.stdout: osd scrub load threshold = 5.0 2024-03-06T07:27:02.398 INFO:teuthology.orchestra.run.smithi183.stdout: osd scrub max interval = 600 2024-03-06T07:27:02.398 INFO:teuthology.orchestra.run.smithi183.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:27:02.398 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.398 INFO:teuthology.orchestra.run.smithi183.stdout: osd recover clone overlap = true 2024-03-06T07:27:02.398 INFO:teuthology.orchestra.run.smithi183.stdout: osd recovery max chunk = 1048576 2024-03-06T07:27:02.398 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.398 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug shutdown = true 2024-03-06T07:27:02.398 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug op order = true 2024-03-06T07:27:02.398 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug verify stray on activate = true 2024-03-06T07:27:02.398 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug trim objects = true 2024-03-06T07:27:02.399 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.399 INFO:teuthology.orchestra.run.smithi183.stdout: osd open classes on start = true 2024-03-06T07:27:02.399 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug pg log writeout = true 2024-03-06T07:27:02.399 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.399 INFO:teuthology.orchestra.run.smithi183.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:27:02.399 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.399 INFO:teuthology.orchestra.run.smithi183.stdout: osd map max advance = 10 2024-03-06T07:27:02.399 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.399 INFO:teuthology.orchestra.run.smithi183.stdout: journal zero on create = true 2024-03-06T07:27:02.399 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.399 INFO:teuthology.orchestra.run.smithi183.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:27:02.399 INFO:teuthology.orchestra.run.smithi183.stdout: filestore apply finisher threads = 3 2024-03-06T07:27:02.399 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.399 INFO:teuthology.orchestra.run.smithi183.stdout: bdev debug aio = true 2024-03-06T07:27:02.400 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug misdirected ops = true 2024-03-06T07:27:02.400 INFO:teuthology.orchestra.run.smithi183.stdout: bdev async discard = True 2024-03-06T07:27:02.400 INFO:teuthology.orchestra.run.smithi183.stdout: bdev enable discard = True 2024-03-06T07:27:02.400 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore allocator = bitmap 2024-03-06T07:27:02.400 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore block size = 96636764160 2024-03-06T07:27:02.400 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore fsck on mount = True 2024-03-06T07:27:02.400 INFO:teuthology.orchestra.run.smithi183.stdout: debug bluefs = 1/20 2024-03-06T07:27:02.400 INFO:teuthology.orchestra.run.smithi183.stdout: debug bluestore = 1/20 2024-03-06T07:27:02.400 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-06T07:27:02.400 INFO:teuthology.orchestra.run.smithi183.stdout: debug osd = 20 2024-03-06T07:27:02.400 INFO:teuthology.orchestra.run.smithi183.stdout: debug rocksdb = 4/10 2024-03-06T07:27:02.400 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:27:02.400 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd full ratio = 0.9 2024-03-06T07:27:02.400 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:27:02.400 INFO:teuthology.orchestra.run.smithi183.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:27:02.401 INFO:teuthology.orchestra.run.smithi183.stdout: osd objectstore = bluestore 2024-03-06T07:27:02.401 INFO:teuthology.orchestra.run.smithi183.stdout: osd op complaint time = 180 2024-03-06T07:27:02.401 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.401 INFO:teuthology.orchestra.run.smithi183.stdout:[mgr] 2024-03-06T07:27:02.401 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-06T07:27:02.401 INFO:teuthology.orchestra.run.smithi183.stdout: debug mgr = 20 2024-03-06T07:27:02.401 INFO:teuthology.orchestra.run.smithi183.stdout: debug mon = 20 2024-03-06T07:27:02.401 INFO:teuthology.orchestra.run.smithi183.stdout: debug auth = 20 2024-03-06T07:27:02.401 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:02.401 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:02.401 INFO:teuthology.orchestra.run.smithi183.stdout: mgr/telemetry/nag = false 2024-03-06T07:27:02.401 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.401 INFO:teuthology.orchestra.run.smithi183.stdout:[mon] 2024-03-06T07:27:02.401 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-06T07:27:02.401 INFO:teuthology.orchestra.run.smithi183.stdout: debug mon = 20 2024-03-06T07:27:02.401 INFO:teuthology.orchestra.run.smithi183.stdout: debug paxos = 20 2024-03-06T07:27:02.402 INFO:teuthology.orchestra.run.smithi183.stdout: debug auth = 20 2024-03-06T07:27:02.402 INFO:teuthology.orchestra.run.smithi183.stdout: mon data avail warn = 5 2024-03-06T07:27:02.402 INFO:teuthology.orchestra.run.smithi183.stdout: mon mgr mkfs grace = 240 2024-03-06T07:27:02.402 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:02.402 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd reporter subtree level = osd 2024-03-06T07:27:02.402 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd prime pg temp = true 2024-03-06T07:27:02.402 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:02.402 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.402 INFO:teuthology.orchestra.run.smithi183.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:27:02.402 INFO:teuthology.orchestra.run.smithi183.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:27:02.402 INFO:teuthology.orchestra.run.smithi183.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:27:02.402 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.402 INFO:teuthology.orchestra.run.smithi183.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:27:02.402 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:27:02.402 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:27:02.402 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.403 INFO:teuthology.orchestra.run.smithi183.stdout: # 1m isn't quite enough 2024-03-06T07:27:02.403 INFO:teuthology.orchestra.run.smithi183.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:27:02.403 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.403 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:27:02.403 INFO:teuthology.orchestra.run.smithi183.stdout: mon op complaint time = 120 2024-03-06T07:27:02.403 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-06T07:27:02.403 INFO:teuthology.orchestra.run.smithi183.stdout:[client] 2024-03-06T07:27:02.403 INFO:teuthology.orchestra.run.smithi183.stdout: rgw cache enabled = true 2024-03-06T07:27:02.403 INFO:teuthology.orchestra.run.smithi183.stdout: rgw enable ops log = true 2024-03-06T07:27:02.403 INFO:teuthology.orchestra.run.smithi183.stdout: rgw enable usage log = true 2024-03-06T07:27:02.403 INFO:teuthology.orchestra.run.smithi183.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:27:02.403 INFO:teuthology.orchestra.run.smithi183.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:27:02.403 INFO:teuthology.orchestra.run.smithi183.stdout: client mount timeout = 600 2024-03-06T07:27:02.403 INFO:teuthology.orchestra.run.smithi183.stdout: debug client = 20 2024-03-06T07:27:02.404 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-06T07:27:02.404 INFO:teuthology.orchestra.run.smithi183.stdout: pid file = "" 2024-03-06T07:27:02.404 INFO:teuthology.orchestra.run.smithi183.stdout: rados mon op timeout = 900 2024-03-06T07:27:02.404 INFO:teuthology.orchestra.run.smithi183.stdout: rados osd op timeout = 900 2024-03-06T07:27:02.404 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.a] 2024-03-06T07:27:02.404 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.b] 2024-03-06T07:27:02.404 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.c] 2024-03-06T07:27:02.404 INFO:teuthology.orchestra.run.smithi183.stdout:[mds.a] 2024-03-06T07:27:02.404 INFO:teuthology.orchestra.run.smithi183.stdout:[mds.c] 2024-03-06T07:27:02.404 INFO:teuthology.orchestra.run.smithi183.stdout:[mds.b] 2024-03-06T07:27:02.404 INFO:teuthology.orchestra.run.smithi183.stdout:[mds.d] 2024-03-06T07:27:02.404 INFO:teuthology.orchestra.run.smithi183.stdout:[mds] 2024-03-06T07:27:02.404 INFO:teuthology.orchestra.run.smithi183.stdout: debug mds = 20 2024-03-06T07:27:02.405 INFO:teuthology.orchestra.run.smithi183.stdout: debug mds balancer = 20 2024-03-06T07:27:02.405 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-06T07:27:02.405 INFO:teuthology.orchestra.run.smithi183.stdout: mds debug frag = True 2024-03-06T07:27:02.405 INFO:teuthology.orchestra.run.smithi183.stdout: mds debug scatterstat = True 2024-03-06T07:27:02.405 INFO:teuthology.orchestra.run.smithi183.stdout: mds op complaint time = 180 2024-03-06T07:27:02.405 INFO:teuthology.orchestra.run.smithi183.stdout: mds verify scatter = True 2024-03-06T07:27:02.405 INFO:teuthology.orchestra.run.smithi183.stdout: osd op complaint time = 180 2024-03-06T07:27:02.405 INFO:teuthology.orchestra.run.smithi183.stdout: rados mon op timeout = 900 2024-03-06T07:27:02.405 INFO:teuthology.orchestra.run.smithi183.stdout: rados osd op timeout = 900 2024-03-06T07:27:02.406 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:27:02.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.549+0000 7fb4f3871640 1 Processor -- start 2024-03-06T07:27:02.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.549+0000 7fb4f3871640 1 -- start start 2024-03-06T07:27:02.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.550+0000 7fb4f3871640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb4ec15f4f0 0x7fb4ec066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.550+0000 7fb4f3871640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb4ec0b0db0 0x7fb4ec0ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.550+0000 7fb4f3871640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4ec06b050 con 0x7fb4ec0ce580 2024-03-06T07:27:02.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.550+0000 7fb4f3871640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4ec06b2c0 con 0x7fb4ec0b0db0 2024-03-06T07:27:02.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.550+0000 7fb4f3871640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ec0b0890 con 0x7fb4ec15f4f0 2024-03-06T07:27:02.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.550+0000 7fb4f0e0c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb4ec0b0db0 0x7fb4ec0ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.550+0000 7fb4f1e0e640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fb4ec0ce580 0x7fb4ec072550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:55280/0 (socket says 172.21.15.83:55280) 2024-03-06T07:27:02.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.550+0000 7fb4f1e0e640 1 -- 172.21.15.83:0/2408522499 learned_addr learned my addr 172.21.15.83:0/2408522499 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:27:02.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.550+0000 7fb4f160d640 1 --2- 172.21.15.83:0/2408522499 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb4ec15f4f0 0x7fb4ec066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.550+0000 7fb4f0e0c640 1 -- 172.21.15.83:0/2408522499 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb4ec15f4f0 msgr2=0x7fb4ec066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:02.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.550+0000 7fb4f0e0c640 1 --2- 172.21.15.83:0/2408522499 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb4ec15f4f0 0x7fb4ec066d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.550+0000 7fb4f0e0c640 1 -- 172.21.15.83:0/2408522499 >> v1:172.21.15.83:6789/0 conn(0x7fb4ec0ce580 legacy=0x7fb4ec072550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.550+0000 7fb4f0e0c640 1 -- 172.21.15.83:0/2408522499 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4ec0672e0 con 0x7fb4ec0b0db0 2024-03-06T07:27:02.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.550+0000 7fb4e3fff640 1 -- 172.21.15.83:0/2408522499 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3503408775 0 0) 0x7fb4ec06b050 con 0x7fb4ec0ce580 2024-03-06T07:27:02.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.551+0000 7fb4f160d640 1 --2- 172.21.15.83:0/2408522499 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb4ec15f4f0 0x7fb4ec066d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:02.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.551+0000 7fb4f0e0c640 1 --2- 172.21.15.83:0/2408522499 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4ec0b0db0 0x7fb4ec0ce040 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc0097a0 tx=0x7fb4dc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4ef67b4cf4966b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:02.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.551+0000 7fb4e3fff640 1 -- 172.21.15.83:0/2408522499 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4dc035020 con 0x7fb4ec0b0db0 2024-03-06T07:27:02.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.551+0000 7fb4e3fff640 1 -- 172.21.15.83:0/2408522499 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4dc031830 con 0x7fb4ec0b0db0 2024-03-06T07:27:02.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.551+0000 7fb4e3fff640 1 -- 172.21.15.83:0/2408522499 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4dc031c00 con 0x7fb4ec0b0db0 2024-03-06T07:27:02.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.552+0000 7fb4f3871640 1 -- 172.21.15.83:0/2408522499 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4ec0b0db0 msgr2=0x7fb4ec0ce040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.552+0000 7fb4f3871640 1 --2- 172.21.15.83:0/2408522499 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4ec0b0db0 0x7fb4ec0ce040 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc0097a0 tx=0x7fb4dc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.552+0000 7fb4f3871640 1 -- 172.21.15.83:0/2408522499 shutdown_connections 2024-03-06T07:27:02.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.552+0000 7fb4f3871640 1 --2- 172.21.15.83:0/2408522499 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4ec0b0db0 0x7fb4ec0ce040 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.552+0000 7fb4f3871640 1 --2- 172.21.15.83:0/2408522499 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb4ec15f4f0 0x7fb4ec066d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.552+0000 7fb4f3871640 1 -- 172.21.15.83:0/2408522499 >> 172.21.15.83:0/2408522499 conn(0x7fb4ec0b1d70 msgr2=0x7fb4ec0b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:02.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.552+0000 7fb4f3871640 1 -- 172.21.15.83:0/2408522499 shutdown_connections 2024-03-06T07:27:02.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.552+0000 7fb4f3871640 1 -- 172.21.15.83:0/2408522499 wait complete. 2024-03-06T07:27:02.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.553+0000 7fb4f3871640 1 Processor -- start 2024-03-06T07:27:02.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.553+0000 7fb4f3871640 1 -- start start 2024-03-06T07:27:02.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.554+0000 7fb4f3871640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb4ec0b0db0 0x7fb4ec0caac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.554+0000 7fb4f3871640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4ec0ce580 0x7fb4ec0ae310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.554+0000 7fb4f3871640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb4ec15f4f0 0x7fb4ec0c9900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.554+0000 7fb4f3871640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ec15f8d0 con 0x7fb4ec0ce580 2024-03-06T07:27:02.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.554+0000 7fb4f3871640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ec06b5a0 con 0x7fb4ec0b0db0 2024-03-06T07:27:02.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.554+0000 7fb4f3871640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ec158ee0 con 0x7fb4ec15f4f0 2024-03-06T07:27:02.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.554+0000 7fb4f0e0c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4ec0ce580 0x7fb4ec0ae310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.554+0000 7fb4f0e0c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4ec0ce580 0x7fb4ec0ae310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51554/0 (socket says 172.21.15.83:51554) 2024-03-06T07:27:02.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.554+0000 7fb4f0e0c640 1 -- 172.21.15.83:0/1242411170 learned_addr learned my addr 172.21.15.83:0/1242411170 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:02.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.554+0000 7fb4f1e0e640 1 --2- 172.21.15.83:0/1242411170 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb4ec15f4f0 0x7fb4ec0c9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.554+0000 7fb4f160d640 1 --2- 172.21.15.83:0/1242411170 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb4ec0b0db0 0x7fb4ec0caac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.554+0000 7fb4f0e0c640 1 -- 172.21.15.83:0/1242411170 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb4ec15f4f0 msgr2=0x7fb4ec0c9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.554+0000 7fb4f0e0c640 1 --2- 172.21.15.83:0/1242411170 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb4ec15f4f0 0x7fb4ec0c9900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.554+0000 7fb4f0e0c640 1 -- 172.21.15.83:0/1242411170 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb4ec0b0db0 msgr2=0x7fb4ec0caac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.554+0000 7fb4f0e0c640 1 --2- 172.21.15.83:0/1242411170 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb4ec0b0db0 0x7fb4ec0caac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.554+0000 7fb4f0e0c640 1 -- 172.21.15.83:0/1242411170 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4ec153930 con 0x7fb4ec0ce580 2024-03-06T07:27:02.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.554+0000 7fb4f160d640 1 --2- 172.21.15.83:0/1242411170 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb4ec0b0db0 0x7fb4ec0caac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:02.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.554+0000 7fb4f1e0e640 1 --2- 172.21.15.83:0/1242411170 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb4ec15f4f0 0x7fb4ec0c9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:02.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.554+0000 7fb4f0e0c640 1 --2- 172.21.15.83:0/1242411170 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4ec0ce580 0x7fb4ec0ae310 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc02bee0 tx=0x7fb4dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:02.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.555+0000 7fb4e1ffb640 1 -- 172.21.15.83:0/1242411170 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4dc041070 con 0x7fb4ec0ce580 2024-03-06T07:27:02.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.555+0000 7fb4e1ffb640 1 -- 172.21.15.83:0/1242411170 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4dc03a950 con 0x7fb4ec0ce580 2024-03-06T07:27:02.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.555+0000 7fb4f3871640 1 -- 172.21.15.83:0/1242411170 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4ec158b90 con 0x7fb4ec0ce580 2024-03-06T07:27:02.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.555+0000 7fb4e1ffb640 1 -- 172.21.15.83:0/1242411170 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4dc035040 con 0x7fb4ec0ce580 2024-03-06T07:27:02.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.555+0000 7fb4f3871640 1 -- 172.21.15.83:0/1242411170 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4ec0c7f00 con 0x7fb4ec0ce580 2024-03-06T07:27:02.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.556+0000 7fb4f3871640 1 -- 172.21.15.83:0/1242411170 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4ec09e3d0 con 0x7fb4ec0ce580 2024-03-06T07:27:02.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.556+0000 7fb4e1ffb640 1 -- 172.21.15.83:0/1242411170 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb4dc038020 con 0x7fb4ec0ce580 2024-03-06T07:27:02.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.556+0000 7fb4e1ffb640 1 --2- 172.21.15.83:0/1242411170 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb4d006b500 0x7fb4d006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.556+0000 7fb4e1ffb640 1 -- 172.21.15.83:0/1242411170 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fb4dc0ab980 con 0x7fb4ec0ce580 2024-03-06T07:27:02.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.556+0000 7fb4f160d640 1 --2- 172.21.15.83:0/1242411170 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb4d006b500 0x7fb4d006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.557+0000 7fb4f160d640 1 --2- 172.21.15.83:0/1242411170 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb4d006b500 0x7fb4d006d9c0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fb4e800a990 tx=0x7fb4e8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:02.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.561+0000 7fb4e1ffb640 1 -- 172.21.15.83:0/1242411170 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb4ec153930 con 0x7fb4ec0ce580 2024-03-06T07:27:02.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.733+0000 7fb4f3871640 1 -- 172.21.15.83:0/1242411170 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fb4ec06cd40 con 0x7fb4ec0ce580 2024-03-06T07:27:02.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.735+0000 7fb4e1ffb640 1 -- 172.21.15.83:0/1242411170 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v29) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb4dc07ab30 con 0x7fb4ec0ce580 2024-03-06T07:27:02.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.738+0000 7fb4f3871640 1 -- 172.21.15.83:0/1242411170 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb4d006b500 msgr2=0x7fb4d006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.738+0000 7fb4f3871640 1 --2- 172.21.15.83:0/1242411170 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb4d006b500 0x7fb4d006d9c0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fb4e800a990 tx=0x7fb4e8008040 comp rx=0 tx=0).stop 2024-03-06T07:27:02.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.738+0000 7fb4f3871640 1 -- 172.21.15.83:0/1242411170 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4ec0ce580 msgr2=0x7fb4ec0ae310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.738+0000 7fb4f3871640 1 --2- 172.21.15.83:0/1242411170 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4ec0ce580 0x7fb4ec0ae310 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc02bee0 tx=0x7fb4dc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.738+0000 7fb4f3871640 1 -- 172.21.15.83:0/1242411170 shutdown_connections 2024-03-06T07:27:02.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.738+0000 7fb4f3871640 1 --2- 172.21.15.83:0/1242411170 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb4d006b500 0x7fb4d006d9c0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.738+0000 7fb4f3871640 1 --2- 172.21.15.83:0/1242411170 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb4ec15f4f0 0x7fb4ec0c9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.738+0000 7fb4f3871640 1 --2- 172.21.15.83:0/1242411170 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4ec0ce580 0x7fb4ec0ae310 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.738+0000 7fb4f3871640 1 --2- 172.21.15.83:0/1242411170 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb4ec0b0db0 0x7fb4ec0caac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.738+0000 7fb4f3871640 1 -- 172.21.15.83:0/1242411170 >> 172.21.15.83:0/1242411170 conn(0x7fb4ec0b1d70 msgr2=0x7fb4ec06ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:02.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.738+0000 7fb4f3871640 1 -- 172.21.15.83:0/1242411170 shutdown_connections 2024-03-06T07:27:02.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.738+0000 7fb4f3871640 1 -- 172.21.15.83:0/1242411170 wait complete. 2024-03-06T07:27:02.751 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_files_throttle' 2024-03-06T07:27:02.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.849+0000 7ff894ac2640 1 Processor -- start 2024-03-06T07:27:02.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.849+0000 7ff894ac2640 1 -- start start 2024-03-06T07:27:02.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.849+0000 7ff894ac2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8900b4230 0x7ff8900ccd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.849+0000 7ff894ac2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff8900cd2d0 0x7ff89006d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.849+0000 7ff894ac2640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8900b4830 con 0x7ff890145350 2024-03-06T07:27:02.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.849+0000 7ff894ac2640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8900b4b10 con 0x7ff8900cd2d0 2024-03-06T07:27:02.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.849+0000 7ff894ac2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8900b4df0 con 0x7ff8900b4230 2024-03-06T07:27:02.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.850+0000 7ff88e59c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff890145350 0x7ff8900cf2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:55296/0 (socket says 172.21.15.83:55296) 2024-03-06T07:27:02.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.850+0000 7ff88e59c640 1 -- 172.21.15.83:0/511659277 learned_addr learned my addr 172.21.15.83:0/511659277 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:27:02.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.850+0000 7ff88ed9d640 1 --2- 172.21.15.83:0/511659277 >> v2:172.21.15.83:3300/0 conn(0x7ff8900cd2d0 0x7ff89006d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.850+0000 7ff885d9b640 1 --2- 172.21.15.83:0/511659277 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8900b4230 0x7ff8900ccd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.850+0000 7ff88ed9d640 1 -- 172.21.15.83:0/511659277 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8900b4230 msgr2=0x7ff8900ccd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:02.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.850+0000 7ff88ed9d640 1 --2- 172.21.15.83:0/511659277 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8900b4230 0x7ff8900ccd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.850+0000 7ff88ed9d640 1 -- 172.21.15.83:0/511659277 >> v1:172.21.15.83:6789/0 conn(0x7ff890145350 legacy=0x7ff8900cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.850+0000 7ff88dd9b640 1 -- 172.21.15.83:0/511659277 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2223928991 0 0) 0x7ff8900b4830 con 0x7ff890145350 2024-03-06T07:27:02.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.850+0000 7ff88ed9d640 1 -- 172.21.15.83:0/511659277 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff89006ce00 con 0x7ff8900cd2d0 2024-03-06T07:27:02.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.851+0000 7ff88ed9d640 1 --2- 172.21.15.83:0/511659277 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff8900cd2d0 0x7ff89006d320 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7ff888009d90 tx=0x7ff88802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0d59f6ece595a33 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:02.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.851+0000 7ff88dd9b640 1 -- 172.21.15.83:0/511659277 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff888039070 con 0x7ff8900cd2d0 2024-03-06T07:27:02.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.851+0000 7ff88dd9b640 1 -- 172.21.15.83:0/511659277 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff88802cc50 con 0x7ff8900cd2d0 2024-03-06T07:27:02.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.851+0000 7ff88dd9b640 1 -- 172.21.15.83:0/511659277 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff888034440 con 0x7ff8900cd2d0 2024-03-06T07:27:02.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.851+0000 7ff894ac2640 1 -- 172.21.15.83:0/511659277 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff8900cd2d0 msgr2=0x7ff89006d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.851+0000 7ff894ac2640 1 --2- 172.21.15.83:0/511659277 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff8900cd2d0 0x7ff89006d320 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7ff888009d90 tx=0x7ff88802bea0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.851+0000 7ff894ac2640 1 -- 172.21.15.83:0/511659277 shutdown_connections 2024-03-06T07:27:02.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.851+0000 7ff894ac2640 1 --2- 172.21.15.83:0/511659277 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff8900cd2d0 0x7ff89006d320 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.851+0000 7ff894ac2640 1 --2- 172.21.15.83:0/511659277 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8900b4230 0x7ff8900ccd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.851+0000 7ff894ac2640 1 -- 172.21.15.83:0/511659277 >> 172.21.15.83:0/511659277 conn(0x7ff89006b3d0 msgr2=0x7ff89006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:02.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.851+0000 7ff894ac2640 1 -- 172.21.15.83:0/511659277 shutdown_connections 2024-03-06T07:27:02.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.851+0000 7ff894ac2640 1 -- 172.21.15.83:0/511659277 wait complete. 2024-03-06T07:27:02.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.853+0000 7ff894ac2640 1 Processor -- start 2024-03-06T07:27:02.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.853+0000 7ff894ac2640 1 -- start start 2024-03-06T07:27:02.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.854+0000 7ff894ac2640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff8900b4230 0x7ff8901310e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.854+0000 7ff894ac2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8900cd2d0 0x7ff890141020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.854+0000 7ff894ac2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff890145350 0x7ff890141560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.854+0000 7ff894ac2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff89006d9e0 con 0x7ff890145350 2024-03-06T07:27:02.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.854+0000 7ff894ac2640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff89006d860 con 0x7ff8900b4230 2024-03-06T07:27:02.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.854+0000 7ff894ac2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff89006db60 con 0x7ff8900cd2d0 2024-03-06T07:27:02.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.854+0000 7ff88ed9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff890145350 0x7ff890141560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.854+0000 7ff88ed9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff890145350 0x7ff890141560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51580/0 (socket says 172.21.15.83:51580) 2024-03-06T07:27:02.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.854+0000 7ff88ed9d640 1 -- 172.21.15.83:0/634816650 learned_addr learned my addr 172.21.15.83:0/634816650 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:02.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.854+0000 7ff885d9b640 1 --2- 172.21.15.83:0/634816650 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8900cd2d0 0x7ff890141020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.854+0000 7ff88e59c640 1 --2- 172.21.15.83:0/634816650 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff8900b4230 0x7ff8901310e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.854+0000 7ff88ed9d640 1 -- 172.21.15.83:0/634816650 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8900cd2d0 msgr2=0x7ff890141020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.854+0000 7ff88ed9d640 1 --2- 172.21.15.83:0/634816650 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8900cd2d0 0x7ff890141020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.854+0000 7ff88ed9d640 1 -- 172.21.15.83:0/634816650 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff8900b4230 msgr2=0x7ff8901310e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.854+0000 7ff88ed9d640 1 --2- 172.21.15.83:0/634816650 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff8900b4230 0x7ff8901310e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.854+0000 7ff88ed9d640 1 -- 172.21.15.83:0/634816650 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff890131620 con 0x7ff890145350 2024-03-06T07:27:02.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.854+0000 7ff88ed9d640 1 --2- 172.21.15.83:0/634816650 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff890145350 0x7ff890141560 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7ff888009d90 tx=0x7ff888004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:02.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.854+0000 7ff8877fe640 1 -- 172.21.15.83:0/634816650 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff888044020 con 0x7ff890145350 2024-03-06T07:27:02.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.854+0000 7ff8877fe640 1 -- 172.21.15.83:0/634816650 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff88803a040 con 0x7ff890145350 2024-03-06T07:27:02.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.854+0000 7ff894ac2640 1 -- 172.21.15.83:0/634816650 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8901e7370 con 0x7ff890145350 2024-03-06T07:27:02.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.854+0000 7ff8877fe640 1 -- 172.21.15.83:0/634816650 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff888039050 con 0x7ff890145350 2024-03-06T07:27:02.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.854+0000 7ff894ac2640 1 -- 172.21.15.83:0/634816650 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8901e7930 con 0x7ff890145350 2024-03-06T07:27:02.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.855+0000 7ff894ac2640 1 -- 172.21.15.83:0/634816650 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff89004f330 con 0x7ff890145350 2024-03-06T07:27:02.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.856+0000 7ff8877fe640 1 -- 172.21.15.83:0/634816650 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff888003990 con 0x7ff890145350 2024-03-06T07:27:02.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.856+0000 7ff8877fe640 1 --2- 172.21.15.83:0/634816650 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff87006b4d0 0x7ff87006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.856+0000 7ff88e59c640 1 --2- 172.21.15.83:0/634816650 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff87006b4d0 0x7ff87006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.857+0000 7ff8877fe640 1 -- 172.21.15.83:0/634816650 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5029+0+0 (secure 0 0 0) 0x7ff8880ac220 con 0x7ff890145350 2024-03-06T07:27:02.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.857+0000 7ff88e59c640 1 --2- 172.21.15.83:0/634816650 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff87006b4d0 0x7ff87006d990 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ff880007fd0 tx=0x7ff880007de0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:02.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:02.861+0000 7ff8877fe640 1 -- 172.21.15.83:0/634816650 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff88804e060 con 0x7ff890145350 2024-03-06T07:27:03.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:03.097+0000 7ff894ac2640 1 -- 172.21.15.83:0/634816650 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]} v 0) v1 -- 0x7ff8900cf470 con 0x7ff890145350 2024-03-06T07:27:04.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.085+0000 7ff8877fe640 1 -- 172.21.15.83:0/634816650 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]}]=0 v289) v1 ==== 134+0+0 (secure 0 0 0) 0x7ff88807ace0 con 0x7ff890145350 2024-03-06T07:27:04.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.088+0000 7ff894ac2640 1 -- 172.21.15.83:0/634816650 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff87006b4d0 msgr2=0x7ff87006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.088+0000 7ff894ac2640 1 --2- 172.21.15.83:0/634816650 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff87006b4d0 0x7ff87006d990 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ff880007fd0 tx=0x7ff880007de0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.088+0000 7ff894ac2640 1 -- 172.21.15.83:0/634816650 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff890145350 msgr2=0x7ff890141560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.088+0000 7ff894ac2640 1 --2- 172.21.15.83:0/634816650 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff890145350 0x7ff890141560 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7ff888009d90 tx=0x7ff888004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:04.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.088+0000 7ff894ac2640 1 -- 172.21.15.83:0/634816650 shutdown_connections 2024-03-06T07:27:04.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.088+0000 7ff894ac2640 1 --2- 172.21.15.83:0/634816650 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff87006b4d0 0x7ff87006d990 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.088+0000 7ff894ac2640 1 --2- 172.21.15.83:0/634816650 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff890145350 0x7ff890141560 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.088+0000 7ff894ac2640 1 --2- 172.21.15.83:0/634816650 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8900cd2d0 0x7ff890141020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.088+0000 7ff894ac2640 1 --2- 172.21.15.83:0/634816650 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff8900b4230 0x7ff8901310e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.088+0000 7ff894ac2640 1 -- 172.21.15.83:0/634816650 >> 172.21.15.83:0/634816650 conn(0x7ff89006b3d0 msgr2=0x7ff890142d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:04.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.088+0000 7ff894ac2640 1 -- 172.21.15.83:0/634816650 shutdown_connections 2024-03-06T07:27:04.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.088+0000 7ff894ac2640 1 -- 172.21.15.83:0/634816650 wait complete. 2024-03-06T07:27:04.104 INFO:tasks.cephfs_test_runner:test_files_throttle (tasks.cephfs.test_strays.TestStrays) ... ok 2024-03-06T07:27:04.104 INFO:tasks.cephfs_test_runner:Starting test: test_hardlink_reintegration (tasks.cephfs.test_strays.TestStrays) That removal of primary dentry of hardlinked inode results 2024-03-06T07:27:04.104 INFO:tasks.cephfs_test_runner:test_hardlink_reintegration (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:27:04.105 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:04.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.208+0000 7ff64015c640 1 Processor -- start 2024-03-06T07:27:04.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.209+0000 7ff64015c640 1 -- start start 2024-03-06T07:27:04.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.209+0000 7ff64015c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff6380d22b0 0x7ff6380d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.209+0000 7ff64015c640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ff6380b41f0 0x7ff6380b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.209+0000 7ff64015c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6380b3430 con 0x7ff63815dfe0 2024-03-06T07:27:04.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.209+0000 7ff64015c640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6380d2c50 con 0x7ff6380d22b0 2024-03-06T07:27:04.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.209+0000 7ff64015c640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6380b31e0 con 0x7ff6380b41f0 2024-03-06T07:27:04.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.209+0000 7ff63d6f7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff6380d22b0 0x7ff6380d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.210+0000 7ff63d6f7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff6380d22b0 0x7ff6380d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51586/0 (socket says 172.21.15.83:51586) 2024-03-06T07:27:04.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.210+0000 7ff63d6f7640 1 -- 172.21.15.83:0/1236131133 learned_addr learned my addr 172.21.15.83:0/1236131133 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:04.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.210+0000 7ff63e6f9640 1 --2- 172.21.15.83:0/1236131133 >> v2:172.21.15.183:3300/0 conn(0x7ff6380b41f0 0x7ff6380b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.210+0000 7ff63d6f7640 1 -- 172.21.15.83:0/1236131133 >> v2:172.21.15.183:3300/0 conn(0x7ff6380b41f0 msgr2=0x7ff6380b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:04.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.210+0000 7ff63d6f7640 1 --2- 172.21.15.83:0/1236131133 >> v2:172.21.15.183:3300/0 conn(0x7ff6380b41f0 0x7ff6380b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.210+0000 7ff63d6f7640 1 -- 172.21.15.83:0/1236131133 >> v1:172.21.15.183:6789/0 conn(0x7ff63815dfe0 legacy=0x7ff6380cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:04.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.210+0000 7ff63d6f7640 1 -- 172.21.15.83:0/1236131133 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6380b3710 con 0x7ff6380d22b0 2024-03-06T07:27:04.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.211+0000 7ff63d6f7640 1 --2- 172.21.15.83:0/1236131133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6380d22b0 0x7ff6380d2710 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7ff628009870 tx=0x7ff62802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de0e873c9f6ac2e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:04.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.211+0000 7ff63cef6640 1 -- 172.21.15.83:0/1236131133 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff628035020 con 0x7ff6380d22b0 2024-03-06T07:27:04.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.211+0000 7ff63cef6640 1 -- 172.21.15.83:0/1236131133 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff628031870 con 0x7ff6380d22b0 2024-03-06T07:27:04.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.211+0000 7ff63cef6640 1 -- 172.21.15.83:0/1236131133 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff628031c40 con 0x7ff6380d22b0 2024-03-06T07:27:04.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.211+0000 7ff64015c640 1 -- 172.21.15.83:0/1236131133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6380d22b0 msgr2=0x7ff6380d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.211+0000 7ff64015c640 1 --2- 172.21.15.83:0/1236131133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6380d22b0 0x7ff6380d2710 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7ff628009870 tx=0x7ff62802b910 comp rx=0 tx=0).stop 2024-03-06T07:27:04.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.211+0000 7ff64015c640 1 -- 172.21.15.83:0/1236131133 shutdown_connections 2024-03-06T07:27:04.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.211+0000 7ff64015c640 1 --2- 172.21.15.83:0/1236131133 >> v2:172.21.15.183:3300/0 conn(0x7ff6380b41f0 0x7ff6380b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.211+0000 7ff64015c640 1 --2- 172.21.15.83:0/1236131133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6380d22b0 0x7ff6380d2710 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.211+0000 7ff64015c640 1 -- 172.21.15.83:0/1236131133 >> 172.21.15.83:0/1236131133 conn(0x7ff6380d3cd0 msgr2=0x7ff6380d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:04.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.211+0000 7ff64015c640 1 -- 172.21.15.83:0/1236131133 shutdown_connections 2024-03-06T07:27:04.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.211+0000 7ff64015c640 1 -- 172.21.15.83:0/1236131133 wait complete. 2024-03-06T07:27:04.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.212+0000 7ff64015c640 1 Processor -- start 2024-03-06T07:27:04.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.213+0000 7ff64015c640 1 -- start start 2024-03-06T07:27:04.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.213+0000 7ff64015c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff6380b41f0 0x7ff6380b84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.213+0000 7ff64015c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6380d22b0 0x7ff6380b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.213+0000 7ff64015c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff63815dfe0 0x7ff6380b6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.213+0000 7ff64015c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6380cfa00 con 0x7ff6380d22b0 2024-03-06T07:27:04.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.213+0000 7ff64015c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6380b3d60 con 0x7ff63815dfe0 2024-03-06T07:27:04.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.213+0000 7ff64015c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6380d1c70 con 0x7ff6380b41f0 2024-03-06T07:27:04.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.213+0000 7ff63d6f7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6380d22b0 0x7ff6380b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.213+0000 7ff63d6f7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6380d22b0 0x7ff6380b68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51594/0 (socket says 172.21.15.83:51594) 2024-03-06T07:27:04.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.213+0000 7ff63d6f7640 1 -- 172.21.15.83:0/1629855281 learned_addr learned my addr 172.21.15.83:0/1629855281 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:04.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.214+0000 7ff63e6f9640 1 --2- 172.21.15.83:0/1629855281 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff63815dfe0 0x7ff6380b6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.214+0000 7ff63def8640 1 --2- 172.21.15.83:0/1629855281 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff6380b41f0 0x7ff6380b84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.214+0000 7ff63d6f7640 1 -- 172.21.15.83:0/1629855281 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff6380b41f0 msgr2=0x7ff6380b84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.214+0000 7ff63d6f7640 1 --2- 172.21.15.83:0/1629855281 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff6380b41f0 0x7ff6380b84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.214+0000 7ff63d6f7640 1 -- 172.21.15.83:0/1629855281 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff63815dfe0 msgr2=0x7ff6380b6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.214+0000 7ff63d6f7640 1 --2- 172.21.15.83:0/1629855281 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff63815dfe0 0x7ff6380b6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.214+0000 7ff63d6f7640 1 -- 172.21.15.83:0/1629855281 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6381607d0 con 0x7ff6380d22b0 2024-03-06T07:27:04.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.214+0000 7ff63def8640 1 --2- 172.21.15.83:0/1629855281 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff6380b41f0 0x7ff6380b84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:04.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.214+0000 7ff63e6f9640 1 --2- 172.21.15.83:0/1629855281 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff63815dfe0 0x7ff6380b6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:04.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.214+0000 7ff63d6f7640 1 --2- 172.21.15.83:0/1629855281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6380d22b0 0x7ff6380b68a0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7ff628000c00 tx=0x7ff6280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:04.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.214+0000 7ff62e7fc640 1 -- 172.21.15.83:0/1629855281 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff628040070 con 0x7ff6380d22b0 2024-03-06T07:27:04.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.214+0000 7ff62e7fc640 1 -- 172.21.15.83:0/1629855281 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff62803a420 con 0x7ff6380d22b0 2024-03-06T07:27:04.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.214+0000 7ff64015c640 1 -- 172.21.15.83:0/1629855281 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6380b7410 con 0x7ff6380d22b0 2024-03-06T07:27:04.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.214+0000 7ff62e7fc640 1 -- 172.21.15.83:0/1629855281 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff628035040 con 0x7ff6380d22b0 2024-03-06T07:27:04.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.214+0000 7ff64015c640 1 -- 172.21.15.83:0/1629855281 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6380b79d0 con 0x7ff6380d22b0 2024-03-06T07:27:04.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.215+0000 7ff64015c640 1 -- 172.21.15.83:0/1629855281 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff63804f330 con 0x7ff6380d22b0 2024-03-06T07:27:04.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.216+0000 7ff62e7fc640 1 -- 172.21.15.83:0/1629855281 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff62804b060 con 0x7ff6380d22b0 2024-03-06T07:27:04.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.216+0000 7ff62e7fc640 1 --2- 172.21.15.83:0/1629855281 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff61c06b500 0x7ff61c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.216+0000 7ff63def8640 1 --2- 172.21.15.83:0/1629855281 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff61c06b500 0x7ff61c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.216+0000 7ff62e7fc640 1 -- 172.21.15.83:0/1629855281 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5029+0+0 (secure 0 0 0) 0x7ff6280abe10 con 0x7ff6380d22b0 2024-03-06T07:27:04.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.217+0000 7ff63def8640 1 --2- 172.21.15.83:0/1629855281 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff61c06b500 0x7ff61c06d9c0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ff634007e90 tx=0x7ff6340073a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:04.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.221+0000 7ff62e7fc640 1 -- 172.21.15.83:0/1629855281 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff62807afc0 con 0x7ff6380d22b0 2024-03-06T07:27:04.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.394+0000 7ff64015c640 1 -- 172.21.15.83:0/1629855281 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6380747a0 con 0x7ff6380d22b0 2024-03-06T07:27:04.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.395+0000 7ff62e7fc640 1 -- 172.21.15.83:0/1629855281 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) v1 ==== 74+0+13164 (secure 0 0 0) 0x7ff62807a8c0 con 0x7ff6380d22b0 2024-03-06T07:27:04.397 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:04.397 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":101,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:27:02.076917+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.83:6837/3254575964":"2024-03-07T07:26:59.063588+0000","172.21.15.83:6837/2924411033":"2024-03-07T07:26:27.785046+0000","172.21.15.83:6835/3254575964":"2024-03-07T07:26:59.063588+0000","172.21.15.83:6835/2924411033":"2024-03-07T07:26:27.785046+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:04.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.397+0000 7ff64015c640 1 -- 172.21.15.83:0/1629855281 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff61c06b500 msgr2=0x7ff61c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.397+0000 7ff64015c640 1 --2- 172.21.15.83:0/1629855281 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff61c06b500 0x7ff61c06d9c0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ff634007e90 tx=0x7ff6340073a0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.397+0000 7ff64015c640 1 -- 172.21.15.83:0/1629855281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6380d22b0 msgr2=0x7ff6380b68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.397+0000 7ff64015c640 1 --2- 172.21.15.83:0/1629855281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6380d22b0 0x7ff6380b68a0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7ff628000c00 tx=0x7ff6280046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.397+0000 7ff64015c640 1 -- 172.21.15.83:0/1629855281 shutdown_connections 2024-03-06T07:27:04.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.397+0000 7ff64015c640 1 --2- 172.21.15.83:0/1629855281 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff61c06b500 0x7ff61c06d9c0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.397+0000 7ff64015c640 1 --2- 172.21.15.83:0/1629855281 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff63815dfe0 0x7ff6380b6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.397+0000 7ff64015c640 1 --2- 172.21.15.83:0/1629855281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6380d22b0 0x7ff6380b68a0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.397+0000 7ff64015c640 1 --2- 172.21.15.83:0/1629855281 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff6380b41f0 0x7ff6380b84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.397+0000 7ff64015c640 1 -- 172.21.15.83:0/1629855281 >> 172.21.15.83:0/1629855281 conn(0x7ff6380d3cd0 msgr2=0x7ff6380cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:04.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.397+0000 7ff64015c640 1 -- 172.21.15.83:0/1629855281 shutdown_connections 2024-03-06T07:27:04.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.397+0000 7ff64015c640 1 -- 172.21.15.83:0/1629855281 wait complete. 2024-03-06T07:27:04.410 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:27:04.410 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:27:04.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.511+0000 7fd144864640 1 Processor -- start 2024-03-06T07:27:04.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.511+0000 7fd144864640 1 -- start start 2024-03-06T07:27:04.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.512+0000 7fd144864640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd1400d4860 0x7fd1400d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.512+0000 7fd144864640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd1400b0e30 0x7fd1400b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.512+0000 7fd144864640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1401618b0 con 0x7fd1401614d0 2024-03-06T07:27:04.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.512+0000 7fd144864640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1400d4cc0 con 0x7fd1400b0e30 2024-03-06T07:27:04.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.512+0000 7fd144864640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1400cf400 con 0x7fd1400d4860 2024-03-06T07:27:04.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.512+0000 7fd13ed9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd1400b0e30 0x7fd1400b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.512+0000 7fd13e59c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fd1401614d0 0x7fd1400d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:55300/0 (socket says 172.21.15.83:55300) 2024-03-06T07:27:04.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.512+0000 7fd13e59c640 1 -- 172.21.15.83:0/3018625086 learned_addr learned my addr 172.21.15.83:0/3018625086 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:27:04.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.513+0000 7fd13dd9b640 1 --2- 172.21.15.83:0/3018625086 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd1400d4860 0x7fd1400d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.513+0000 7fd13ed9d640 1 -- 172.21.15.83:0/3018625086 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd1400d4860 msgr2=0x7fd1400d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:04.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.513+0000 7fd13ed9d640 1 --2- 172.21.15.83:0/3018625086 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd1400d4860 0x7fd1400d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.513+0000 7fd13ed9d640 1 -- 172.21.15.83:0/3018625086 >> v1:172.21.15.83:6789/0 conn(0x7fd1401614d0 legacy=0x7fd1400d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.513+0000 7fd13d59a640 1 -- 172.21.15.83:0/3018625086 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3630591920 0 0) 0x7fd1401618b0 con 0x7fd1401614d0 2024-03-06T07:27:04.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.513+0000 7fd13ed9d640 1 -- 172.21.15.83:0/3018625086 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1400cf6a0 con 0x7fd1400b0e30 2024-03-06T07:27:04.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.513+0000 7fd13ed9d640 1 --2- 172.21.15.83:0/3018625086 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1400b0e30 0x7fd1400b12c0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fd1380097a0 tx=0x7fd13802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1e6f70a3d30de46 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:04.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.513+0000 7fd13d59a640 1 -- 172.21.15.83:0/3018625086 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd138035020 con 0x7fd1400b0e30 2024-03-06T07:27:04.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.514+0000 7fd13d59a640 1 -- 172.21.15.83:0/3018625086 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd138031830 con 0x7fd1400b0e30 2024-03-06T07:27:04.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.514+0000 7fd13d59a640 1 -- 172.21.15.83:0/3018625086 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd138031c00 con 0x7fd1400b0e30 2024-03-06T07:27:04.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.514+0000 7fd144864640 1 -- 172.21.15.83:0/3018625086 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1400b0e30 msgr2=0x7fd1400b12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.514+0000 7fd144864640 1 --2- 172.21.15.83:0/3018625086 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1400b0e30 0x7fd1400b12c0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fd1380097a0 tx=0x7fd13802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.514+0000 7fd144864640 1 -- 172.21.15.83:0/3018625086 shutdown_connections 2024-03-06T07:27:04.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.514+0000 7fd144864640 1 --2- 172.21.15.83:0/3018625086 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1400b0e30 0x7fd1400b12c0 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.514+0000 7fd144864640 1 --2- 172.21.15.83:0/3018625086 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd1400d4860 0x7fd1400d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.514+0000 7fd144864640 1 -- 172.21.15.83:0/3018625086 >> 172.21.15.83:0/3018625086 conn(0x7fd1400d3d40 msgr2=0x7fd1400d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:04.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.514+0000 7fd144864640 1 -- 172.21.15.83:0/3018625086 shutdown_connections 2024-03-06T07:27:04.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.514+0000 7fd144864640 1 -- 172.21.15.83:0/3018625086 wait complete. 2024-03-06T07:27:04.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.515+0000 7fd144864640 1 Processor -- start 2024-03-06T07:27:04.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.515+0000 7fd144864640 1 -- start start 2024-03-06T07:27:04.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.516+0000 7fd144864640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd1400b0e30 0x7fd1400b8740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.516+0000 7fd144864640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1400d4860 0x7fd1400c7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.516+0000 7fd144864640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd1401614d0 0x7fd1400b7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.516+0000 7fd144864640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1400b0910 con 0x7fd1400d4860 2024-03-06T07:27:04.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.516+0000 7fd144864640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1400cfc60 con 0x7fd1401614d0 2024-03-06T07:27:04.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.516+0000 7fd144864640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1400d2560 con 0x7fd1400b0e30 2024-03-06T07:27:04.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.516+0000 7fd13dd9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1400d4860 0x7fd1400c7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.516+0000 7fd13dd9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1400d4860 0x7fd1400c7040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51618/0 (socket says 172.21.15.83:51618) 2024-03-06T07:27:04.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.516+0000 7fd13dd9b640 1 -- 172.21.15.83:0/1440142472 learned_addr learned my addr 172.21.15.83:0/1440142472 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:04.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.516+0000 7fd13e59c640 1 --2- 172.21.15.83:0/1440142472 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd1400b0e30 0x7fd1400b8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.516+0000 7fd13dd9b640 1 -- 172.21.15.83:0/1440142472 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd1400b0e30 msgr2=0x7fd1400b8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.517+0000 7fd13dd9b640 1 --2- 172.21.15.83:0/1440142472 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd1400b0e30 0x7fd1400b8740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.517+0000 7fd13dd9b640 1 -- 172.21.15.83:0/1440142472 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd1401614d0 msgr2=0x7fd1400b7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.517+0000 7fd13dd9b640 1 --2- 172.21.15.83:0/1440142472 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd1401614d0 0x7fd1400b7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.517+0000 7fd13dd9b640 1 -- 172.21.15.83:0/1440142472 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1400c7580 con 0x7fd1400d4860 2024-03-06T07:27:04.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.517+0000 7fd13dd9b640 1 --2- 172.21.15.83:0/1440142472 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1400d4860 0x7fd1400c7040 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7fd13400d830 tx=0x7fd13400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:04.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.517+0000 7fd12effd640 1 -- 172.21.15.83:0/1440142472 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13400b6c0 con 0x7fd1400d4860 2024-03-06T07:27:04.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.517+0000 7fd12effd640 1 -- 172.21.15.83:0/1440142472 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd134019070 con 0x7fd1400d4860 2024-03-06T07:27:04.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.517+0000 7fd12effd640 1 -- 172.21.15.83:0/1440142472 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13400ba80 con 0x7fd1400d4860 2024-03-06T07:27:04.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.517+0000 7fd144864640 1 -- 172.21.15.83:0/1440142472 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1400cb8c0 con 0x7fd1400d4860 2024-03-06T07:27:04.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.517+0000 7fd144864640 1 -- 172.21.15.83:0/1440142472 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1400cbe80 con 0x7fd1400d4860 2024-03-06T07:27:04.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.518+0000 7fd144864640 1 -- 172.21.15.83:0/1440142472 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd140068ff0 con 0x7fd1400d4860 2024-03-06T07:27:04.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.518+0000 7fd12effd640 1 -- 172.21.15.83:0/1440142472 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd13401e460 con 0x7fd1400d4860 2024-03-06T07:27:04.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.519+0000 7fd12effd640 1 --2- 172.21.15.83:0/1440142472 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd12006b4d0 0x7fd12006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.519+0000 7fd12effd640 1 -- 172.21.15.83:0/1440142472 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fd13408a580 con 0x7fd1400d4860 2024-03-06T07:27:04.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.519+0000 7fd13e59c640 1 --2- 172.21.15.83:0/1440142472 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd12006b4d0 0x7fd12006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.519+0000 7fd13e59c640 1 --2- 172.21.15.83:0/1440142472 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd12006b4d0 0x7fd12006d990 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fd1300012e0 tx=0x7fd130009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:04.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.523+0000 7fd12effd640 1 -- 172.21.15.83:0/1440142472 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd134059730 con 0x7fd1400d4860 2024-03-06T07:27:04.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.691+0000 7fd144864640 1 -- 172.21.15.83:0/1440142472 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd1400d4530 con 0x7fd1400d4860 2024-03-06T07:27:04.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.692+0000 7fd12effd640 1 -- 172.21.15.83:0/1440142472 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v101) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd134059030 con 0x7fd1400d4860 2024-03-06T07:27:04.694 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:27:04.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.694+0000 7fd144864640 1 -- 172.21.15.83:0/1440142472 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd12006b4d0 msgr2=0x7fd12006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.694+0000 7fd144864640 1 --2- 172.21.15.83:0/1440142472 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd12006b4d0 0x7fd12006d990 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fd1300012e0 tx=0x7fd130009040 comp rx=0 tx=0).stop 2024-03-06T07:27:04.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.694+0000 7fd144864640 1 -- 172.21.15.83:0/1440142472 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1400d4860 msgr2=0x7fd1400c7040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.694+0000 7fd144864640 1 --2- 172.21.15.83:0/1440142472 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1400d4860 0x7fd1400c7040 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7fd13400d830 tx=0x7fd13400dcf0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.694+0000 7fd144864640 1 -- 172.21.15.83:0/1440142472 shutdown_connections 2024-03-06T07:27:04.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.694+0000 7fd144864640 1 --2- 172.21.15.83:0/1440142472 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd12006b4d0 0x7fd12006d990 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.694+0000 7fd144864640 1 --2- 172.21.15.83:0/1440142472 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd1401614d0 0x7fd1400b7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.694+0000 7fd144864640 1 --2- 172.21.15.83:0/1440142472 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1400d4860 0x7fd1400c7040 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.694+0000 7fd144864640 1 --2- 172.21.15.83:0/1440142472 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd1400b0e30 0x7fd1400b8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.694+0000 7fd144864640 1 -- 172.21.15.83:0/1440142472 >> 172.21.15.83:0/1440142472 conn(0x7fd1400d3d40 msgr2=0x7fd1400b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:04.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.694+0000 7fd144864640 1 -- 172.21.15.83:0/1440142472 shutdown_connections 2024-03-06T07:27:04.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.694+0000 7fd144864640 1 -- 172.21.15.83:0/1440142472 wait complete. 2024-03-06T07:27:04.707 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration' 2024-03-06T07:27:04.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.805+0000 7f6966b5e640 1 Processor -- start 2024-03-06T07:27:04.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.805+0000 7f6966b5e640 1 -- start start 2024-03-06T07:27:04.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.806+0000 7f6966b5e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f69600d0320 0x7f69600b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.806+0000 7f6966b5e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f69600d0700 0x7f69600b54b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.806+0000 7f6966b5e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f69600ccae0 0x7f69600ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.806+0000 7f6966b5e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69600b04e0 con 0x7f69600d0320 2024-03-06T07:27:04.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.806+0000 7f6966b5e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f696015c270 con 0x7f69600ccae0 2024-03-06T07:27:04.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.806+0000 7f6966b5e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69600d0e90 con 0x7f69600d0700 2024-03-06T07:27:04.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.806+0000 7f69648fa640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f69600d0320 0x7f69600b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.806+0000 7f69648fa640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f69600d0320 0x7f69600b4f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51628/0 (socket says 172.21.15.83:51628) 2024-03-06T07:27:04.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.806+0000 7f69648fa640 1 -- 172.21.15.83:0/4065159768 learned_addr learned my addr 172.21.15.83:0/4065159768 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:04.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.806+0000 7f695ffff640 1 --2- 172.21.15.83:0/4065159768 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f69600d0700 0x7f69600b54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.806+0000 7f69650fb640 1 --2- 172.21.15.83:0/4065159768 >> v2:172.21.15.183:3300/0 conn(0x7f69600ccae0 0x7f69600ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.806+0000 7f69648fa640 1 -- 172.21.15.83:0/4065159768 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f69600d0700 msgr2=0x7f69600b54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:04.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.806+0000 7f69648fa640 1 --2- 172.21.15.83:0/4065159768 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f69600d0700 0x7f69600b54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.806+0000 7f69648fa640 1 -- 172.21.15.83:0/4065159768 >> v2:172.21.15.183:3300/0 conn(0x7f69600ccae0 msgr2=0x7f69600ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:04.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.806+0000 7f69648fa640 1 --2- 172.21.15.83:0/4065159768 >> v2:172.21.15.183:3300/0 conn(0x7f69600ccae0 0x7f69600ccf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.806+0000 7f69648fa640 1 -- 172.21.15.83:0/4065159768 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69600b5ae0 con 0x7f69600d0320 2024-03-06T07:27:04.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.807+0000 7f69648fa640 1 --2- 172.21.15.83:0/4065159768 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f69600d0320 0x7f69600b4f70 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f6960071470 tx=0x7f695002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53bfc8a959201a00 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:04.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.807+0000 7f695f7fe640 1 -- 172.21.15.83:0/4065159768 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6950034070 con 0x7f69600d0320 2024-03-06T07:27:04.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.807+0000 7f695f7fe640 1 -- 172.21.15.83:0/4065159768 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6950030860 con 0x7f69600d0320 2024-03-06T07:27:04.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.807+0000 7f695f7fe640 1 -- 172.21.15.83:0/4065159768 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6950030c30 con 0x7f69600d0320 2024-03-06T07:27:04.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.808+0000 7f6966b5e640 1 -- 172.21.15.83:0/4065159768 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f69600d0320 msgr2=0x7f69600b4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.808+0000 7f6966b5e640 1 --2- 172.21.15.83:0/4065159768 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f69600d0320 0x7f69600b4f70 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f6960071470 tx=0x7f695002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.808+0000 7f6966b5e640 1 -- 172.21.15.83:0/4065159768 shutdown_connections 2024-03-06T07:27:04.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.808+0000 7f6966b5e640 1 --2- 172.21.15.83:0/4065159768 >> v2:172.21.15.183:3300/0 conn(0x7f69600ccae0 0x7f69600ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.808+0000 7f6966b5e640 1 --2- 172.21.15.83:0/4065159768 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f69600d0700 0x7f69600b54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.808+0000 7f6966b5e640 1 --2- 172.21.15.83:0/4065159768 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f69600d0320 0x7f69600b4f70 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.808+0000 7f6966b5e640 1 -- 172.21.15.83:0/4065159768 >> 172.21.15.83:0/4065159768 conn(0x7f69600724d0 msgr2=0x7f69600728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:04.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.808+0000 7f6966b5e640 1 -- 172.21.15.83:0/4065159768 shutdown_connections 2024-03-06T07:27:04.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.808+0000 7f6966b5e640 1 -- 172.21.15.83:0/4065159768 wait complete. 2024-03-06T07:27:04.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.809+0000 7f6966b5e640 1 Processor -- start 2024-03-06T07:27:04.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.809+0000 7f6966b5e640 1 -- start start 2024-03-06T07:27:04.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.810+0000 7f6966b5e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f69600ccae0 0x7f6960148860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.810+0000 7f6966b5e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f69600d0320 0x7f69600ae3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.810+0000 7f6966b5e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f69600d0700 0x7f6960142d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.810+0000 7f6966b5e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69600b60a0 con 0x7f69600d0700 2024-03-06T07:27:04.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.810+0000 7f6966b5e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69600b0170 con 0x7f69600ccae0 2024-03-06T07:27:04.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.810+0000 7f6966b5e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69600ca790 con 0x7f69600d0320 2024-03-06T07:27:04.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.810+0000 7f69650fb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f69600d0700 0x7f6960142d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.810+0000 7f69650fb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f69600d0700 0x7f6960142d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51634/0 (socket says 172.21.15.83:51634) 2024-03-06T07:27:04.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.810+0000 7f69650fb640 1 -- 172.21.15.83:0/2114604114 learned_addr learned my addr 172.21.15.83:0/2114604114 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:04.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.810+0000 7f695ffff640 1 --2- 172.21.15.83:0/2114604114 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f69600d0320 0x7f69600ae3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.810+0000 7f69650fb640 1 -- 172.21.15.83:0/2114604114 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f69600d0320 msgr2=0x7f69600ae3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.810+0000 7f69648fa640 1 --2- 172.21.15.83:0/2114604114 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f69600ccae0 0x7f6960148860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.810+0000 7f69650fb640 1 --2- 172.21.15.83:0/2114604114 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f69600d0320 0x7f69600ae3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.810+0000 7f69650fb640 1 -- 172.21.15.83:0/2114604114 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f69600ccae0 msgr2=0x7f6960148860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.810+0000 7f69650fb640 1 --2- 172.21.15.83:0/2114604114 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f69600ccae0 0x7f6960148860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.810+0000 7f69650fb640 1 -- 172.21.15.83:0/2114604114 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69600ad340 con 0x7f69600d0700 2024-03-06T07:27:04.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.810+0000 7f69650fb640 1 --2- 172.21.15.83:0/2114604114 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f69600d0700 0x7f6960142d20 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f6958007d50 tx=0x7f695800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:04.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.811+0000 7f695d7fa640 1 -- 172.21.15.83:0/2114604114 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6958014420 con 0x7f69600d0700 2024-03-06T07:27:04.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.811+0000 7f695d7fa640 1 -- 172.21.15.83:0/2114604114 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6958019070 con 0x7f69600d0700 2024-03-06T07:27:04.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.811+0000 7f695d7fa640 1 -- 172.21.15.83:0/2114604114 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6958014820 con 0x7f69600d0700 2024-03-06T07:27:04.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.811+0000 7f6966b5e640 1 -- 172.21.15.83:0/2114604114 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69600ad620 con 0x7f69600d0700 2024-03-06T07:27:04.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.811+0000 7f6966b5e640 1 -- 172.21.15.83:0/2114604114 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69600adbe0 con 0x7f69600d0700 2024-03-06T07:27:04.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.812+0000 7f6966b5e640 1 -- 172.21.15.83:0/2114604114 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f696004f330 con 0x7f69600d0700 2024-03-06T07:27:04.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.812+0000 7f695d7fa640 1 -- 172.21.15.83:0/2114604114 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f69580149c0 con 0x7f69600d0700 2024-03-06T07:27:04.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.812+0000 7f695d7fa640 1 --2- 172.21.15.83:0/2114604114 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f694c06b500 0x7f694c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.812+0000 7f695d7fa640 1 -- 172.21.15.83:0/2114604114 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f695808a690 con 0x7f69600d0700 2024-03-06T07:27:04.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.812+0000 7f69648fa640 1 --2- 172.21.15.83:0/2114604114 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f694c06b500 0x7f694c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.813+0000 7f69648fa640 1 --2- 172.21.15.83:0/2114604114 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f694c06b500 0x7f694c06d9c0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f6950036000 tx=0x7f695003b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:04.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:04.816+0000 7f695d7fa640 1 -- 172.21.15.83:0/2114604114 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f69580597b0 con 0x7f69600d0700 2024-03-06T07:27:05.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.055+0000 7f6966b5e640 1 -- 172.21.15.83:0/2114604114 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]} v 0) v1 -- 0x7f69601571c0 con 0x7f69600d0700 2024-03-06T07:27:05.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.089+0000 7f695d7fa640 1 -- 172.21.15.83:0/2114604114 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]}]=0 v290) v1 ==== 145+0+0 (secure 0 0 0) 0x7f69580590b0 con 0x7f69600d0700 2024-03-06T07:27:05.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.092+0000 7f6966b5e640 1 -- 172.21.15.83:0/2114604114 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f694c06b500 msgr2=0x7f694c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.092+0000 7f6966b5e640 1 --2- 172.21.15.83:0/2114604114 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f694c06b500 0x7f694c06d9c0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f6950036000 tx=0x7f695003b040 comp rx=0 tx=0).stop 2024-03-06T07:27:05.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.092+0000 7f6966b5e640 1 -- 172.21.15.83:0/2114604114 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f69600d0700 msgr2=0x7f6960142d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.092+0000 7f6966b5e640 1 --2- 172.21.15.83:0/2114604114 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f69600d0700 0x7f6960142d20 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f6958007d50 tx=0x7f695800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.092+0000 7f6966b5e640 1 -- 172.21.15.83:0/2114604114 shutdown_connections 2024-03-06T07:27:05.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.092+0000 7f6966b5e640 1 --2- 172.21.15.83:0/2114604114 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f694c06b500 0x7f694c06d9c0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.092+0000 7f6966b5e640 1 --2- 172.21.15.83:0/2114604114 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f69600d0700 0x7f6960142d20 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.092+0000 7f6966b5e640 1 --2- 172.21.15.83:0/2114604114 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f69600d0320 0x7f69600ae3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.092+0000 7f6966b5e640 1 --2- 172.21.15.83:0/2114604114 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f69600ccae0 0x7f6960148860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.092+0000 7f6966b5e640 1 -- 172.21.15.83:0/2114604114 >> 172.21.15.83:0/2114604114 conn(0x7f69600724d0 msgr2=0x7f69600af6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:05.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.092+0000 7f6966b5e640 1 -- 172.21.15.83:0/2114604114 shutdown_connections 2024-03-06T07:27:05.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.092+0000 7f6966b5e640 1 -- 172.21.15.83:0/2114604114 wait complete. 2024-03-06T07:27:05.105 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:27:05.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.206+0000 7f19e1afb640 1 Processor -- start 2024-03-06T07:27:05.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.206+0000 7f19e1afb640 1 -- start start 2024-03-06T07:27:05.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.207+0000 7f19e1afb640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f19dc1614d0 0x7f19dc0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.207+0000 7f19e1afb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19dc1618b0 con 0x7f19dc0d24b0 2024-03-06T07:27:05.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.207+0000 7f19e1afb640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19dc0cf400 con 0x7f19dc0d1910 2024-03-06T07:27:05.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.207+0000 7f19e1afb640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19dc0cf6a0 con 0x7f19dc1614d0 2024-03-06T07:27:05.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.207+0000 7f19dbfff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f19dc0d24b0 0x7f19dc0b0e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:55312/0 (socket says 172.21.15.83:55312) 2024-03-06T07:27:05.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.207+0000 7f19dbfff640 1 -- 172.21.15.83:0/2692334461 learned_addr learned my addr 172.21.15.83:0/2692334461 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:27:05.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.208+0000 7f19db7fe640 1 --2- 172.21.15.83:0/2692334461 >> v2:172.21.15.183:3300/0 conn(0x7f19dc1614d0 0x7f19dc0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.208+0000 7f19da7fc640 1 -- 172.21.15.83:0/2692334461 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4279468219 0 0) 0x7f19dc1618b0 con 0x7f19dc0d24b0 2024-03-06T07:27:05.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.208+0000 7f19da7fc640 1 -- 172.21.15.83:0/2692334461 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19c4003520 con 0x7f19dc0d24b0 2024-03-06T07:27:05.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.208+0000 7f19da7fc640 1 -- 172.21.15.83:0/2692334461 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 431043116 0 0) 0x7f19c4003520 con 0x7f19dc0d24b0 2024-03-06T07:27:05.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.208+0000 7f19da7fc640 1 -- 172.21.15.83:0/2692334461 >> v2:172.21.15.183:3300/0 conn(0x7f19dc1614d0 msgr2=0x7f19dc0d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:05.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.208+0000 7f19da7fc640 1 --2- 172.21.15.83:0/2692334461 >> v2:172.21.15.183:3300/0 conn(0x7f19dc1614d0 0x7f19dc0d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.208+0000 7f19da7fc640 1 -- 172.21.15.83:0/2692334461 >> v1:172.21.15.183:6789/0 conn(0x7f19dc0d1910 legacy=0x7f19dc0d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.208+0000 7f19da7fc640 1 -- 172.21.15.83:0/2692334461 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19dc0cf980 con 0x7f19dc0d24b0 2024-03-06T07:27:05.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.208+0000 7f19da7fc640 1 -- 172.21.15.83:0/2692334461 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f19d40019f0 con 0x7f19dc0d24b0 2024-03-06T07:27:05.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.208+0000 7f19da7fc640 1 -- 172.21.15.83:0/2692334461 >> v1:172.21.15.83:6789/0 conn(0x7f19dc0d24b0 legacy=0x7f19dc0b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.208+0000 7f19da7fc640 1 --2- 172.21.15.83:0/2692334461 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f19c4003a00 0x7f19c4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.208+0000 7f19da7fc640 1 --2- 172.21.15.83:0/2692334461 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f19c4004800 0x7f19c4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.208+0000 7f19da7fc640 1 --2- 172.21.15.83:0/2692334461 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f19c40051c0 0x7f19c400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.208+0000 7f19da7fc640 1 -- 172.21.15.83:0/2692334461 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc0cf6a0 con 0x7f19c4004800 2024-03-06T07:27:05.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.208+0000 7f19da7fc640 1 -- 172.21.15.83:0/2692334461 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c400c0c0 con 0x7f19c4003a00 2024-03-06T07:27:05.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.208+0000 7f19da7fc640 1 -- 172.21.15.83:0/2692334461 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19c400c2c0 con 0x7f19c40051c0 2024-03-06T07:27:05.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.208+0000 7f19daffd640 1 --2- 172.21.15.83:0/2692334461 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f19c4004800 0x7f19c4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.208+0000 7f19db7fe640 1 --2- 172.21.15.83:0/2692334461 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f19c4003a00 0x7f19c4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.208+0000 7f19dbfff640 1 --2- 172.21.15.83:0/2692334461 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f19c40051c0 0x7f19c400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.208+0000 7f19daffd640 1 -- 172.21.15.83:0/2692334461 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f19c40051c0 msgr2=0x7f19c400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:05.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.208+0000 7f19daffd640 1 --2- 172.21.15.83:0/2692334461 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f19c40051c0 0x7f19c400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.208+0000 7f19daffd640 1 -- 172.21.15.83:0/2692334461 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f19c4003a00 msgr2=0x7f19c4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:05.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.208+0000 7f19daffd640 1 --2- 172.21.15.83:0/2692334461 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f19c4003a00 0x7f19c4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.208+0000 7f19dbfff640 1 -- 172.21.15.83:0/2692334461 reap_dead start 2024-03-06T07:27:05.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.208+0000 7f19daffd640 1 -- 172.21.15.83:0/2692334461 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c4003520 con 0x7f19c4004800 2024-03-06T07:27:05.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.209+0000 7f19daffd640 1 --2- 172.21.15.83:0/2692334461 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f19c4004800 0x7f19c4004c80 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f19d0001720 tx=0x7f19c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=307759b29566fa1c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.209+0000 7f19da7fc640 1 -- 172.21.15.83:0/2692334461 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d0001370 con 0x7f19c4004800 2024-03-06T07:27:05.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.209+0000 7f19da7fc640 1 -- 172.21.15.83:0/2692334461 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19d00052d0 con 0x7f19c4004800 2024-03-06T07:27:05.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.209+0000 7f19da7fc640 1 -- 172.21.15.83:0/2692334461 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d0003830 con 0x7f19c4004800 2024-03-06T07:27:05.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.210+0000 7f19e1afb640 1 -- 172.21.15.83:0/2692334461 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f19c4004800 msgr2=0x7f19c4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.210+0000 7f19e1afb640 1 --2- 172.21.15.83:0/2692334461 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f19c4004800 0x7f19c4004c80 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f19d0001720 tx=0x7f19c4005830 comp rx=0 tx=0).stop 2024-03-06T07:27:05.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.210+0000 7f19e1afb640 1 -- 172.21.15.83:0/2692334461 shutdown_connections 2024-03-06T07:27:05.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.210+0000 7f19e1afb640 1 --2- 172.21.15.83:0/2692334461 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f19c4004800 0x7f19c4004c80 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.210+0000 7f19e1afb640 1 -- 172.21.15.83:0/2692334461 >> 172.21.15.83:0/2692334461 conn(0x7f19dc0d3d40 msgr2=0x7f19dc0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:05.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.210+0000 7f19e1afb640 1 -- 172.21.15.83:0/2692334461 shutdown_connections 2024-03-06T07:27:05.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.210+0000 7f19e1afb640 1 -- 172.21.15.83:0/2692334461 wait complete. 2024-03-06T07:27:05.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.211+0000 7f19e1afb640 1 Processor -- start 2024-03-06T07:27:05.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.211+0000 7f19e1afb640 1 -- start start 2024-03-06T07:27:05.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.212+0000 7f19e1afb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f19c4004800 0x7f19dc151540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.212+0000 7f19e1afb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f19dc0cd320 0x7f19dc0cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.212+0000 7f19e1afb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f19dc0cb7d0 0x7f19dc0cbc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.212+0000 7f19e1afb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc0d4860 con 0x7f19c4004800 2024-03-06T07:27:05.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.212+0000 7f19e1afb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc0b0910 con 0x7f19dc0cd320 2024-03-06T07:27:05.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.212+0000 7f19e1afb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc0d49e0 con 0x7f19dc0cb7d0 2024-03-06T07:27:05.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.212+0000 7f19db7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f19c4004800 0x7f19dc151540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.212+0000 7f19db7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f19c4004800 0x7f19dc151540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51648/0 (socket says 172.21.15.83:51648) 2024-03-06T07:27:05.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.212+0000 7f19db7fe640 1 -- 172.21.15.83:0/3374682225 learned_addr learned my addr 172.21.15.83:0/3374682225 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:05.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.212+0000 7f19db7fe640 1 -- 172.21.15.83:0/3374682225 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f19dc0cb7d0 msgr2=0x7f19dc0cbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.212+0000 7f19db7fe640 1 --2- 172.21.15.83:0/3374682225 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f19dc0cb7d0 0x7f19dc0cbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.212+0000 7f19db7fe640 1 -- 172.21.15.83:0/3374682225 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f19dc0cd320 msgr2=0x7f19dc0cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.212+0000 7f19db7fe640 1 --2- 172.21.15.83:0/3374682225 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f19dc0cd320 0x7f19dc0cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.212+0000 7f19db7fe640 1 -- 172.21.15.83:0/3374682225 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19dc0cc190 con 0x7f19c4004800 2024-03-06T07:27:05.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.212+0000 7f19db7fe640 1 --2- 172.21.15.83:0/3374682225 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f19c4004800 0x7f19dc151540 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f19cc011960 tx=0x7f19cc011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.212+0000 7f19c3fff640 1 -- 172.21.15.83:0/3374682225 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19cc00fa70 con 0x7f19c4004800 2024-03-06T07:27:05.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.212+0000 7f19c3fff640 1 -- 172.21.15.83:0/3374682225 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19cc00fc10 con 0x7f19c4004800 2024-03-06T07:27:05.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.213+0000 7f19e1afb640 1 -- 172.21.15.83:0/3374682225 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19dc0cc400 con 0x7f19c4004800 2024-03-06T07:27:05.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.213+0000 7f19c3fff640 1 -- 172.21.15.83:0/3374682225 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19cc014440 con 0x7f19c4004800 2024-03-06T07:27:05.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.213+0000 7f19e1afb640 1 -- 172.21.15.83:0/3374682225 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19dc0c7870 con 0x7f19c4004800 2024-03-06T07:27:05.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.214+0000 7f19e1afb640 1 -- 172.21.15.83:0/3374682225 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19dc0c9900 con 0x7f19c4004800 2024-03-06T07:27:05.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.214+0000 7f19c3fff640 1 -- 172.21.15.83:0/3374682225 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f19cc012040 con 0x7f19c4004800 2024-03-06T07:27:05.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.214+0000 7f19c3fff640 1 --2- 172.21.15.83:0/3374682225 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f19c806b500 0x7f19c806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.215+0000 7f19daffd640 1 --2- 172.21.15.83:0/3374682225 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f19c806b500 0x7f19c806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.215+0000 7f19c3fff640 1 -- 172.21.15.83:0/3374682225 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f19cc08a570 con 0x7f19c4004800 2024-03-06T07:27:05.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.215+0000 7f19daffd640 1 --2- 172.21.15.83:0/3374682225 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f19c806b500 0x7f19c806d9c0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f19d0033000 tx=0x7f19d002f000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.219+0000 7f19c3fff640 1 -- 172.21.15.83:0/3374682225 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f19cc019030 con 0x7f19c4004800 2024-03-06T07:27:05.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.392+0000 7f19e1afb640 1 -- 172.21.15.83:0/3374682225 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f19dc068ff0 con 0x7f19c4004800 2024-03-06T07:27:05.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.395+0000 7f19c3fff640 1 -- 172.21.15.83:0/3374682225 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v30)=0 v30) v1 ==== 122+0+0 (secure 0 0 0) 0x7f19cc059040 con 0x7f19c4004800 2024-03-06T07:27:05.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.397+0000 7f19e1afb640 1 -- 172.21.15.83:0/3374682225 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f19c806b500 msgr2=0x7f19c806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.397+0000 7f19e1afb640 1 --2- 172.21.15.83:0/3374682225 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f19c806b500 0x7f19c806d9c0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f19d0033000 tx=0x7f19d002f000 comp rx=0 tx=0).stop 2024-03-06T07:27:05.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.397+0000 7f19e1afb640 1 -- 172.21.15.83:0/3374682225 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f19c4004800 msgr2=0x7f19dc151540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.397+0000 7f19e1afb640 1 --2- 172.21.15.83:0/3374682225 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f19c4004800 0x7f19dc151540 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f19cc011960 tx=0x7f19cc011e20 comp rx=0 tx=0).stop 2024-03-06T07:27:05.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.397+0000 7f19e1afb640 1 -- 172.21.15.83:0/3374682225 shutdown_connections 2024-03-06T07:27:05.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.397+0000 7f19e1afb640 1 --2- 172.21.15.83:0/3374682225 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f19c806b500 0x7f19c806d9c0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.397+0000 7f19e1afb640 1 --2- 172.21.15.83:0/3374682225 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f19dc0cb7d0 0x7f19dc0cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.397+0000 7f19e1afb640 1 --2- 172.21.15.83:0/3374682225 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f19dc0cd320 0x7f19dc0cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.397+0000 7f19e1afb640 1 --2- 172.21.15.83:0/3374682225 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f19c4004800 0x7f19dc151540 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.397+0000 7f19e1afb640 1 -- 172.21.15.83:0/3374682225 >> 172.21.15.83:0/3374682225 conn(0x7f19dc0d3d40 msgr2=0x7f19dc06c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:05.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.397+0000 7f19e1afb640 1 -- 172.21.15.83:0/3374682225 shutdown_connections 2024-03-06T07:27:05.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.398+0000 7f19e1afb640 1 -- 172.21.15.83:0/3374682225 wait complete. 2024-03-06T07:27:05.412 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:27:05.412 DEBUG:teuthology.orchestra.run.smithi083:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:27:05.451 DEBUG:teuthology.orchestra.run.smithi183:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:27:05.498 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:27:05.498 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:27:05.498 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:05.514 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1257 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:27:05.514 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:27:05.514 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:05.531 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=287 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:27:05.531 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:27:05.531 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:05.568 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1257 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:27:05.569 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:27:05.569 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:05.588 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=287 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:27:05.589 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:27:05.589 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:05.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.706+0000 7f7abeb3e640 1 Processor -- start 2024-03-06T07:27:05.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.706+0000 7f7abeb3e640 1 -- start start 2024-03-06T07:27:05.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.707+0000 7f7abeb3e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7ab815dfe0 0x7f7ab80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.707+0000 7f7abeb3e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7ab80cf830 0x7f7ab80d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.707+0000 7f7abeb3e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ab80b3430 con 0x7f7ab80d27f0 2024-03-06T07:27:05.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.707+0000 7f7abeb3e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ab80d2c50 con 0x7f7ab80cf830 2024-03-06T07:27:05.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.707+0000 7f7abeb3e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab80b31e0 con 0x7f7ab815dfe0 2024-03-06T07:27:05.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.707+0000 7f7ab7fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7ab80cf830 0x7f7ab80d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.707+0000 7f7abd0db640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f7ab80d27f0 0x7f7ab80d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:55318/0 (socket says 172.21.15.83:55318) 2024-03-06T07:27:05.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.707+0000 7f7abd0db640 1 -- 172.21.15.83:0/3203474044 learned_addr learned my addr 172.21.15.83:0/3203474044 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:27:05.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.707+0000 7f7ab7fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7ab80cf830 0x7f7ab80d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51656/0 (socket says 172.21.15.83:51656) 2024-03-06T07:27:05.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.708+0000 7f7abc8da640 1 --2- 172.21.15.83:0/3203474044 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7ab815dfe0 0x7f7ab80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.708+0000 7f7ab7fff640 1 -- 172.21.15.83:0/3203474044 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7ab815dfe0 msgr2=0x7f7ab80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:05.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.708+0000 7f7ab7fff640 1 --2- 172.21.15.83:0/3203474044 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7ab815dfe0 0x7f7ab80cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.708+0000 7f7ab7fff640 1 -- 172.21.15.83:0/3203474044 >> v1:172.21.15.83:6789/0 conn(0x7f7ab80d27f0 legacy=0x7f7ab80d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.708+0000 7f7ab77fe640 1 -- 172.21.15.83:0/3203474044 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3935608630 0 0) 0x7f7ab80b3430 con 0x7f7ab80d27f0 2024-03-06T07:27:05.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.708+0000 7f7ab7fff640 1 -- 172.21.15.83:0/3203474044 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ab80b3710 con 0x7f7ab80cf830 2024-03-06T07:27:05.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.708+0000 7f7abc8da640 1 --2- 172.21.15.83:0/3203474044 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7ab815dfe0 0x7f7ab80cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:05.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.708+0000 7f7ab7fff640 1 --2- 172.21.15.83:0/3203474044 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ab80cf830 0x7f7ab80d22b0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f7aac0097a0 tx=0x7f7aac02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77be4fef7fec6ccd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.709+0000 7f7ab77fe640 1 -- 172.21.15.83:0/3203474044 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aac035020 con 0x7f7ab80cf830 2024-03-06T07:27:05.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.709+0000 7f7ab77fe640 1 -- 172.21.15.83:0/3203474044 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aac031830 con 0x7f7ab80cf830 2024-03-06T07:27:05.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.709+0000 7f7ab77fe640 1 -- 172.21.15.83:0/3203474044 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aac031c00 con 0x7f7ab80cf830 2024-03-06T07:27:05.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.709+0000 7f7abeb3e640 1 -- 172.21.15.83:0/3203474044 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ab80cf830 msgr2=0x7f7ab80d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.709+0000 7f7abeb3e640 1 --2- 172.21.15.83:0/3203474044 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ab80cf830 0x7f7ab80d22b0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f7aac0097a0 tx=0x7f7aac02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.709+0000 7f7abeb3e640 1 -- 172.21.15.83:0/3203474044 shutdown_connections 2024-03-06T07:27:05.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.709+0000 7f7abeb3e640 1 --2- 172.21.15.83:0/3203474044 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ab80cf830 0x7f7ab80d22b0 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.709+0000 7f7abeb3e640 1 --2- 172.21.15.83:0/3203474044 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7ab815dfe0 0x7f7ab80cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.709+0000 7f7abeb3e640 1 -- 172.21.15.83:0/3203474044 >> 172.21.15.83:0/3203474044 conn(0x7f7ab80d3cd0 msgr2=0x7f7ab80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:05.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.709+0000 7f7abeb3e640 1 -- 172.21.15.83:0/3203474044 shutdown_connections 2024-03-06T07:27:05.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.709+0000 7f7abeb3e640 1 -- 172.21.15.83:0/3203474044 wait complete. 2024-03-06T07:27:05.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.710+0000 7f7abeb3e640 1 Processor -- start 2024-03-06T07:27:05.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.710+0000 7f7abeb3e640 1 -- start start 2024-03-06T07:27:05.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.711+0000 7f7abeb3e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7ab80cf830 0x7f7ab80cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.711+0000 7f7abeb3e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7ab80d27f0 0x7f7ab80cd490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.711+0000 7f7abeb3e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ab815dfe0 0x7f7ab8152b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.711+0000 7f7abeb3e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab80d1c70 con 0x7f7ab815dfe0 2024-03-06T07:27:05.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.711+0000 7f7abeb3e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab80b3d60 con 0x7f7ab80d27f0 2024-03-06T07:27:05.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.711+0000 7f7abeb3e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab80d1df0 con 0x7f7ab80cf830 2024-03-06T07:27:05.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.711+0000 7f7abd0db640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ab815dfe0 0x7f7ab8152b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.711+0000 7f7abd0db640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ab815dfe0 0x7f7ab8152b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51666/0 (socket says 172.21.15.83:51666) 2024-03-06T07:27:05.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.711+0000 7f7abd0db640 1 -- 172.21.15.83:0/1821898296 learned_addr learned my addr 172.21.15.83:0/1821898296 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:05.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.711+0000 7f7abc8da640 1 --2- 172.21.15.83:0/1821898296 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7ab80cf830 0x7f7ab80cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.711+0000 7f7ab7fff640 1 --2- 172.21.15.83:0/1821898296 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7ab80d27f0 0x7f7ab80cd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.711+0000 7f7abd0db640 1 -- 172.21.15.83:0/1821898296 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7ab80cf830 msgr2=0x7f7ab80cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.711+0000 7f7abd0db640 1 --2- 172.21.15.83:0/1821898296 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7ab80cf830 0x7f7ab80cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.711+0000 7f7abd0db640 1 -- 172.21.15.83:0/1821898296 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7ab80d27f0 msgr2=0x7f7ab80cd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.711+0000 7f7abd0db640 1 --2- 172.21.15.83:0/1821898296 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7ab80d27f0 0x7f7ab80cd490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.712+0000 7f7abd0db640 1 -- 172.21.15.83:0/1821898296 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ab80cb2e0 con 0x7f7ab815dfe0 2024-03-06T07:27:05.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.712+0000 7f7abc8da640 1 --2- 172.21.15.83:0/1821898296 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7ab80cf830 0x7f7ab80cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:05.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.712+0000 7f7abd0db640 1 --2- 172.21.15.83:0/1821898296 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ab815dfe0 0x7f7ab8152b30 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f7ab000c810 tx=0x7f7ab000ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.712+0000 7f7ab57fa640 1 -- 172.21.15.83:0/1821898296 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab0018070 con 0x7f7ab815dfe0 2024-03-06T07:27:05.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.712+0000 7f7ab57fa640 1 -- 172.21.15.83:0/1821898296 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ab000a640 con 0x7f7ab815dfe0 2024-03-06T07:27:05.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.712+0000 7f7abeb3e640 1 -- 172.21.15.83:0/1821898296 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ab80cb5b0 con 0x7f7ab815dfe0 2024-03-06T07:27:05.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.712+0000 7f7ab57fa640 1 -- 172.21.15.83:0/1821898296 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab000aa30 con 0x7f7ab815dfe0 2024-03-06T07:27:05.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.712+0000 7f7abeb3e640 1 -- 172.21.15.83:0/1821898296 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ab80cbb70 con 0x7f7ab815dfe0 2024-03-06T07:27:05.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.713+0000 7f7abeb3e640 1 -- 172.21.15.83:0/1821898296 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ab804f330 con 0x7f7ab815dfe0 2024-03-06T07:27:05.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.713+0000 7f7ab57fa640 1 -- 172.21.15.83:0/1821898296 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7ab001b070 con 0x7f7ab815dfe0 2024-03-06T07:27:05.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.714+0000 7f7ab57fa640 1 --2- 172.21.15.83:0/1821898296 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7aa406b4d0 0x7f7aa406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.714+0000 7f7ab57fa640 1 -- 172.21.15.83:0/1821898296 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f7ab0089570 con 0x7f7ab815dfe0 2024-03-06T07:27:05.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.714+0000 7f7abc8da640 1 --2- 172.21.15.83:0/1821898296 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7aa406b4d0 0x7f7aa406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.714+0000 7f7abc8da640 1 --2- 172.21.15.83:0/1821898296 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7aa406b4d0 0x7f7aa406d990 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8006fd0 tx=0x7f7aa8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.718+0000 7f7ab57fa640 1 -- 172.21.15.83:0/1821898296 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7ab0058690 con 0x7f7ab815dfe0 2024-03-06T07:27:05.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.926+0000 7f7abeb3e640 1 -- 172.21.15.83:0/1821898296 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ab815d370 con 0x7f7ab815dfe0 2024-03-06T07:27:05.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.926+0000 7f7ab57fa640 1 -- 172.21.15.83:0/1821898296 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+3699 (secure 0 0 0) 0x7f7ab0057f90 con 0x7f7ab815dfe0 2024-03-06T07:27:05.928 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:05.928 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":55,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6023,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/601236077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":601236077},{"type":"v1","addr":"172.21.15.183:6833","nonce":601236077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6029,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1435055961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1435055961},{"type":"v1","addr":"172.21.15.183:6835","nonce":1435055961}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":7572,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/2726045966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":2726045966},{"type":"v1","addr":"172.21.15.83:6836","nonce":2726045966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":7800,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/128741407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":128741407},{"type":"v1","addr":"172.21.15.83:6837","nonce":128741407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[]} 2024-03-06T07:27:05.928 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 55 2024-03-06T07:27:05.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.928+0000 7f7abeb3e640 1 -- 172.21.15.83:0/1821898296 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7aa406b4d0 msgr2=0x7f7aa406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.928+0000 7f7abeb3e640 1 --2- 172.21.15.83:0/1821898296 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7aa406b4d0 0x7f7aa406d990 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8006fd0 tx=0x7f7aa8008040 comp rx=0 tx=0).stop 2024-03-06T07:27:05.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.928+0000 7f7abeb3e640 1 -- 172.21.15.83:0/1821898296 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ab815dfe0 msgr2=0x7f7ab8152b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.928+0000 7f7abeb3e640 1 --2- 172.21.15.83:0/1821898296 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ab815dfe0 0x7f7ab8152b30 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f7ab000c810 tx=0x7f7ab000ccd0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.928+0000 7f7abeb3e640 1 -- 172.21.15.83:0/1821898296 shutdown_connections 2024-03-06T07:27:05.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.928+0000 7f7abeb3e640 1 --2- 172.21.15.83:0/1821898296 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7aa406b4d0 0x7f7aa406d990 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.928+0000 7f7abeb3e640 1 --2- 172.21.15.83:0/1821898296 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ab815dfe0 0x7f7ab8152b30 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.928+0000 7f7abeb3e640 1 --2- 172.21.15.83:0/1821898296 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7ab80d27f0 0x7f7ab80cd490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.928+0000 7f7abeb3e640 1 --2- 172.21.15.83:0/1821898296 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7ab80cf830 0x7f7ab80cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.928+0000 7f7abeb3e640 1 -- 172.21.15.83:0/1821898296 >> 172.21.15.83:0/1821898296 conn(0x7f7ab80d3cd0 msgr2=0x7f7ab80b4db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:05.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.928+0000 7f7abeb3e640 1 -- 172.21.15.83:0/1821898296 shutdown_connections 2024-03-06T07:27:05.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:05.928+0000 7f7abeb3e640 1 -- 172.21.15.83:0/1821898296 wait complete. 2024-03-06T07:27:05.942 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:27:05.942 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:27:05.942 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:27:05.942 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:05.943 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:27:05.943 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:27:05.943 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:27:05.943 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:05.943 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:27:05.943 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:27:05.943 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:27:05.943 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:05.944 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:27:05.944 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:27:05.944 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:27:05.944 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:05.951 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:27:05.951 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:27:05.952 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:27:05.952 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:27:05.955 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:27:05.955 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:27:05.956 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:27:05.956 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:27:05.994 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:27:05.994 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:05.995 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:27:05.995 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:05.998 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:27:05.998 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:05.998 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:27:05.999 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:05.999 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:27:06.070 INFO:tasks.ceph.mds.a.smithi083.stdout:starting mds.a at 2024-03-06T07:27:06.070 INFO:tasks.ceph.mds.c.smithi083.stdout:starting mds.c at 2024-03-06T07:27:06.080 INFO:tasks.ceph.mds.b.smithi183.stdout:starting mds.b at 2024-03-06T07:27:06.080 INFO:tasks.ceph.mds.d.smithi183.stdout:starting mds.d at 2024-03-06T07:27:06.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.105+0000 7fb1f0884640 1 Processor -- start 2024-03-06T07:27:06.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.105+0000 7fb1f0884640 1 -- start start 2024-03-06T07:27:06.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.106+0000 7fb1f0884640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1ec0cc920 0x7fb1ec15eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.106+0000 7fb1f0884640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fb1ec06e000 0x7fb1ec072070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.106+0000 7fb1f0884640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1ec06e770 con 0x7fb1ec0b4a50 2024-03-06T07:27:06.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.106+0000 7fb1f0884640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1ec15e660 con 0x7fb1ec06e000 2024-03-06T07:27:06.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.106+0000 7fb1f0884640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1ec06e490 con 0x7fb1ec0cc920 2024-03-06T07:27:06.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.106+0000 7fb1ead9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fb1ec0b4a50 0x7fb1ec150b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:55338/0 (socket says 172.21.15.83:55338) 2024-03-06T07:27:06.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.106+0000 7fb1ead9d640 1 -- 172.21.15.83:0/1987518636 learned_addr learned my addr 172.21.15.83:0/1987518636 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:27:06.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.106+0000 7fb1e3fff640 1 --2- 172.21.15.83:0/1987518636 >> v2:172.21.15.183:3300/0 conn(0x7fb1ec06e000 0x7fb1ec072070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.106+0000 7fb1ea59c640 1 --2- 172.21.15.83:0/1987518636 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1ec0cc920 0x7fb1ec15eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.106+0000 7fb1e3fff640 1 -- 172.21.15.83:0/1987518636 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1ec0cc920 msgr2=0x7fb1ec15eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:06.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.106+0000 7fb1e3fff640 1 --2- 172.21.15.83:0/1987518636 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1ec0cc920 0x7fb1ec15eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.106+0000 7fb1e3fff640 1 -- 172.21.15.83:0/1987518636 >> v1:172.21.15.83:6789/0 conn(0x7fb1ec0b4a50 legacy=0x7fb1ec150b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.106+0000 7fb1e3fff640 1 -- 172.21.15.83:0/1987518636 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1ec15e010 con 0x7fb1ec06e000 2024-03-06T07:27:06.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.107+0000 7fb1e3fff640 1 --2- 172.21.15.83:0/1987518636 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1ec06e000 0x7fb1ec072070 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec15e7e0 tx=0x7fb1d402b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a81fed80efc663ef server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.112+0000 7fb1e9d9b640 1 -- 172.21.15.83:0/1987518636 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d4035020 con 0x7fb1ec06e000 2024-03-06T07:27:06.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.112+0000 7fb1e9d9b640 1 -- 172.21.15.83:0/1987518636 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1d4031800 con 0x7fb1ec06e000 2024-03-06T07:27:06.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.112+0000 7fb1e9d9b640 1 -- 172.21.15.83:0/1987518636 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d4031bd0 con 0x7fb1ec06e000 2024-03-06T07:27:06.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.113+0000 7fb1f0884640 1 -- 172.21.15.83:0/1987518636 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1ec06e000 msgr2=0x7fb1ec072070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.113+0000 7fb1f0884640 1 --2- 172.21.15.83:0/1987518636 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1ec06e000 0x7fb1ec072070 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec15e7e0 tx=0x7fb1d402b8a0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.113+0000 7fb1f0884640 1 -- 172.21.15.83:0/1987518636 shutdown_connections 2024-03-06T07:27:06.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.113+0000 7fb1f0884640 1 --2- 172.21.15.83:0/1987518636 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1ec06e000 0x7fb1ec072070 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.113+0000 7fb1f0884640 1 --2- 172.21.15.83:0/1987518636 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1ec0cc920 0x7fb1ec15eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.113+0000 7fb1f0884640 1 -- 172.21.15.83:0/1987518636 >> 172.21.15.83:0/1987518636 conn(0x7fb1ec0b1c80 msgr2=0x7fb1ec0cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:06.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.113+0000 7fb1f0884640 1 -- 172.21.15.83:0/1987518636 shutdown_connections 2024-03-06T07:27:06.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.113+0000 7fb1f0884640 1 -- 172.21.15.83:0/1987518636 wait complete. 2024-03-06T07:27:06.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.114+0000 7fb1f0884640 1 Processor -- start 2024-03-06T07:27:06.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.115+0000 7fb1f0884640 1 -- start start 2024-03-06T07:27:06.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.115+0000 7fb1f0884640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1ec0b4a50 0x7fb1ec148e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.115+0000 7fb1f0884640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1ec0cc920 0x7fb1ec1f3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.115+0000 7fb1f0884640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1ec13c080 0x7fb1ec0b0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.115+0000 7fb1f0884640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1ec0cce80 con 0x7fb1ec13c080 2024-03-06T07:27:06.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.115+0000 7fb1f0884640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1ec0ccd00 con 0x7fb1ec0cc920 2024-03-06T07:27:06.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.115+0000 7fb1f0884640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1ec070940 con 0x7fb1ec0b4a50 2024-03-06T07:27:06.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.115+0000 7fb1ead9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1ec13c080 0x7fb1ec0b0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.116+0000 7fb1ead9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1ec13c080 0x7fb1ec0b0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51710/0 (socket says 172.21.15.83:51710) 2024-03-06T07:27:06.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.116+0000 7fb1ead9d640 1 -- 172.21.15.83:0/3843525510 learned_addr learned my addr 172.21.15.83:0/3843525510 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:06.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.116+0000 7fb1ea59c640 1 --2- 172.21.15.83:0/3843525510 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1ec0b4a50 0x7fb1ec148e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.116+0000 7fb1e3fff640 1 --2- 172.21.15.83:0/3843525510 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1ec0cc920 0x7fb1ec1f3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.116+0000 7fb1ead9d640 1 -- 172.21.15.83:0/3843525510 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1ec0b4a50 msgr2=0x7fb1ec148e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.116+0000 7fb1ead9d640 1 --2- 172.21.15.83:0/3843525510 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1ec0b4a50 0x7fb1ec148e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.116+0000 7fb1ead9d640 1 -- 172.21.15.83:0/3843525510 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1ec0cc920 msgr2=0x7fb1ec1f3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.116+0000 7fb1ead9d640 1 --2- 172.21.15.83:0/3843525510 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1ec0cc920 0x7fb1ec1f3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.116+0000 7fb1ead9d640 1 -- 172.21.15.83:0/3843525510 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1ec0af5c0 con 0x7fb1ec13c080 2024-03-06T07:27:06.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.116+0000 7fb1e3fff640 1 --2- 172.21.15.83:0/3843525510 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1ec0cc920 0x7fb1ec1f3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:06.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.116+0000 7fb1ea59c640 1 --2- 172.21.15.83:0/3843525510 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1ec0b4a50 0x7fb1ec148e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:06.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.116+0000 7fb1ead9d640 1 --2- 172.21.15.83:0/3843525510 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1ec13c080 0x7fb1ec0b0620 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fb1e400caa0 tx=0x7fb1e400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.117+0000 7fb1e2ffd640 1 -- 172.21.15.83:0/3843525510 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e4017070 con 0x7fb1ec13c080 2024-03-06T07:27:06.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.117+0000 7fb1e2ffd640 1 -- 172.21.15.83:0/3843525510 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1e400ac00 con 0x7fb1ec13c080 2024-03-06T07:27:06.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.117+0000 7fb1e2ffd640 1 -- 172.21.15.83:0/3843525510 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e4012620 con 0x7fb1ec13c080 2024-03-06T07:27:06.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.117+0000 7fb1f0884640 1 -- 172.21.15.83:0/3843525510 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1ec0af860 con 0x7fb1ec13c080 2024-03-06T07:27:06.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.117+0000 7fb1f0884640 1 -- 172.21.15.83:0/3843525510 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1ec0afe20 con 0x7fb1ec13c080 2024-03-06T07:27:06.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.118+0000 7fb1f0884640 1 -- 172.21.15.83:0/3843525510 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1ec04f330 con 0x7fb1ec13c080 2024-03-06T07:27:06.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.118+0000 7fb1e2ffd640 1 -- 172.21.15.83:0/3843525510 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb1e401a070 con 0x7fb1ec13c080 2024-03-06T07:27:06.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.119+0000 7fb1e2ffd640 1 --2- 172.21.15.83:0/3843525510 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb1cc06b500 0x7fb1cc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.119+0000 7fb1e2ffd640 1 -- 172.21.15.83:0/3843525510 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fb1e4089c60 con 0x7fb1ec13c080 2024-03-06T07:27:06.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.119+0000 7fb1ea59c640 1 --2- 172.21.15.83:0/3843525510 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb1cc06b500 0x7fb1cc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.119+0000 7fb1ea59c640 1 --2- 172.21.15.83:0/3843525510 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb1cc06b500 0x7fb1cc06d9c0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc009af0 tx=0x7fb1dc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.123+0000 7fb1e2ffd640 1 -- 172.21.15.83:0/3843525510 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb1e4058d80 con 0x7fb1ec13c080 2024-03-06T07:27:06.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:06.309+0000 7fb1f0884640 1 -- 172.21.15.83:0/3843525510 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fb1ec0ceae0 con 0x7fb1ec13c080 2024-03-06T07:27:07.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.119+0000 7fb1e2ffd640 1 -- 172.21.15.83:0/3843525510 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v106) v1 ==== 97+0+0 (secure 0 0 0) 0x7fb1e4058680 con 0x7fb1ec13c080 2024-03-06T07:27:07.121 INFO:teuthology.orchestra.run.smithi083.stderr: removed all blocklist entries 2024-03-06T07:27:07.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.122+0000 7fb1f0884640 1 -- 172.21.15.83:0/3843525510 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb1cc06b500 msgr2=0x7fb1cc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.122+0000 7fb1f0884640 1 --2- 172.21.15.83:0/3843525510 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb1cc06b500 0x7fb1cc06d9c0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc009af0 tx=0x7fb1dc008040 comp rx=0 tx=0).stop 2024-03-06T07:27:07.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.122+0000 7fb1f0884640 1 -- 172.21.15.83:0/3843525510 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1ec13c080 msgr2=0x7fb1ec0b0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.122+0000 7fb1f0884640 1 --2- 172.21.15.83:0/3843525510 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1ec13c080 0x7fb1ec0b0620 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fb1e400caa0 tx=0x7fb1e400cf60 comp rx=0 tx=0).stop 2024-03-06T07:27:07.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.122+0000 7fb1f0884640 1 -- 172.21.15.83:0/3843525510 shutdown_connections 2024-03-06T07:27:07.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.122+0000 7fb1f0884640 1 --2- 172.21.15.83:0/3843525510 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb1cc06b500 0x7fb1cc06d9c0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.122+0000 7fb1f0884640 1 --2- 172.21.15.83:0/3843525510 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1ec13c080 0x7fb1ec0b0620 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.122+0000 7fb1f0884640 1 --2- 172.21.15.83:0/3843525510 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1ec0cc920 0x7fb1ec1f3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.122+0000 7fb1f0884640 1 --2- 172.21.15.83:0/3843525510 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1ec0b4a50 0x7fb1ec148e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.122+0000 7fb1f0884640 1 -- 172.21.15.83:0/3843525510 >> 172.21.15.83:0/3843525510 conn(0x7fb1ec0b1c80 msgr2=0x7fb1ec06d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:07.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.122+0000 7fb1f0884640 1 -- 172.21.15.83:0/3843525510 shutdown_connections 2024-03-06T07:27:07.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.122+0000 7fb1f0884640 1 -- 172.21.15.83:0/3843525510 wait complete. 2024-03-06T07:27:07.137 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:27:07.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.242+0000 7f23f9281640 1 Processor -- start 2024-03-06T07:27:07.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.242+0000 7f23f9281640 1 -- start start 2024-03-06T07:27:07.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.243+0000 7f23f9281640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f23f415bdd0 0x7f23f40c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.243+0000 7f23f9281640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f23f40d16e0 0x7f23f40d1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.243+0000 7f23f9281640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23f40cebe0 con 0x7f23f415fa10 2024-03-06T07:27:07.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.243+0000 7f23f9281640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23f40ce980 con 0x7f23f40d16e0 2024-03-06T07:27:07.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.243+0000 7f23f9281640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23f40ceec0 con 0x7f23f415bdd0 2024-03-06T07:27:07.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.243+0000 7f23f27fc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f23f415fa10 0x7f23f40ce270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:37566/0 (socket says 172.21.15.83:37566) 2024-03-06T07:27:07.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.243+0000 7f23f37fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f23f40d16e0 0x7f23f40d1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.243+0000 7f23f27fc640 1 -- 172.21.15.83:0/783741462 learned_addr learned my addr 172.21.15.83:0/783741462 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:27:07.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.243+0000 7f23f2ffd640 1 --2- 172.21.15.83:0/783741462 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f23f415bdd0 0x7f23f40c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.243+0000 7f23f37fe640 1 -- 172.21.15.83:0/783741462 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f23f415bdd0 msgr2=0x7f23f40c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:07.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.243+0000 7f23f37fe640 1 --2- 172.21.15.83:0/783741462 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f23f415bdd0 0x7f23f40c5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.243+0000 7f23f37fe640 1 -- 172.21.15.83:0/783741462 >> v1:172.21.15.83:6789/0 conn(0x7f23f415fa10 legacy=0x7f23f40ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.243+0000 7f23f37fe640 1 -- 172.21.15.83:0/783741462 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f40cf1a0 con 0x7f23f40d16e0 2024-03-06T07:27:07.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.243+0000 7f23f1ffb640 1 -- 172.21.15.83:0/783741462 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3325773462 0 0) 0x7f23f40cebe0 con 0x7f23f415fa10 2024-03-06T07:27:07.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.243+0000 7f23f2ffd640 1 --2- 172.21.15.83:0/783741462 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f23f415bdd0 0x7f23f40c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:07.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.244+0000 7f23f37fe640 1 --2- 172.21.15.83:0/783741462 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23f40d16e0 0x7f23f40d1b30 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f23ec009d90 tx=0x7f23ec02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=654310f89713c172 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.244+0000 7f23f1ffb640 1 -- 172.21.15.83:0/783741462 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ec039070 con 0x7f23f40d16e0 2024-03-06T07:27:07.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.244+0000 7f23f1ffb640 1 -- 172.21.15.83:0/783741462 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23ec02cbc0 con 0x7f23f40d16e0 2024-03-06T07:27:07.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.244+0000 7f23f1ffb640 1 -- 172.21.15.83:0/783741462 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ec034440 con 0x7f23f40d16e0 2024-03-06T07:27:07.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.245+0000 7f23f9281640 1 -- 172.21.15.83:0/783741462 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23f40d16e0 msgr2=0x7f23f40d1b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.245+0000 7f23f9281640 1 --2- 172.21.15.83:0/783741462 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23f40d16e0 0x7f23f40d1b30 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f23ec009d90 tx=0x7f23ec02be10 comp rx=0 tx=0).stop 2024-03-06T07:27:07.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.245+0000 7f23f9281640 1 -- 172.21.15.83:0/783741462 shutdown_connections 2024-03-06T07:27:07.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.245+0000 7f23f9281640 1 --2- 172.21.15.83:0/783741462 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23f40d16e0 0x7f23f40d1b30 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.245+0000 7f23f9281640 1 --2- 172.21.15.83:0/783741462 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f23f415bdd0 0x7f23f40c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.245+0000 7f23f9281640 1 -- 172.21.15.83:0/783741462 >> 172.21.15.83:0/783741462 conn(0x7f23f409a9f0 msgr2=0x7f23f409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:07.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.245+0000 7f23f9281640 1 -- 172.21.15.83:0/783741462 shutdown_connections 2024-03-06T07:27:07.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.245+0000 7f23f9281640 1 -- 172.21.15.83:0/783741462 wait complete. 2024-03-06T07:27:07.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.246+0000 7f23f9281640 1 Processor -- start 2024-03-06T07:27:07.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.246+0000 7f23f9281640 1 -- start start 2024-03-06T07:27:07.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.247+0000 7f23f9281640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f23f40d16e0 0x7f23f4155000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.247+0000 7f23f9281640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23f415bdd0 0x7f23f40b5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.247+0000 7f23f9281640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f23f415fa10 0x7f23f40c96c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.247+0000 7f23f9281640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f40d2070 con 0x7f23f415bdd0 2024-03-06T07:27:07.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.247+0000 7f23f9281640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f40c5b50 con 0x7f23f40d16e0 2024-03-06T07:27:07.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.247+0000 7f23f9281640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23f40cc5d0 con 0x7f23f415fa10 2024-03-06T07:27:07.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.247+0000 7f23f27fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23f415bdd0 0x7f23f40b5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.247+0000 7f23f27fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23f415bdd0 0x7f23f40b5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47444/0 (socket says 172.21.15.83:47444) 2024-03-06T07:27:07.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.247+0000 7f23f27fc640 1 -- 172.21.15.83:0/716490486 learned_addr learned my addr 172.21.15.83:0/716490486 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:07.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.247+0000 7f23f37fe640 1 --2- 172.21.15.83:0/716490486 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f23f415fa10 0x7f23f40c96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.247+0000 7f23f2ffd640 1 --2- 172.21.15.83:0/716490486 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f23f40d16e0 0x7f23f4155000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.247+0000 7f23f27fc640 1 -- 172.21.15.83:0/716490486 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f23f415fa10 msgr2=0x7f23f40c96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.247+0000 7f23f27fc640 1 --2- 172.21.15.83:0/716490486 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f23f415fa10 0x7f23f40c96c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.247+0000 7f23f27fc640 1 -- 172.21.15.83:0/716490486 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f23f40d16e0 msgr2=0x7f23f4155000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.247+0000 7f23f27fc640 1 --2- 172.21.15.83:0/716490486 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f23f40d16e0 0x7f23f4155000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.247+0000 7f23f27fc640 1 -- 172.21.15.83:0/716490486 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f4155540 con 0x7f23f415bdd0 2024-03-06T07:27:07.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.248+0000 7f23f27fc640 1 --2- 172.21.15.83:0/716490486 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23f415bdd0 0x7f23f40b5720 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f23e800c430 tx=0x7f23e800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.248+0000 7f23db7fe640 1 -- 172.21.15.83:0/716490486 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e80074e0 con 0x7f23f415bdd0 2024-03-06T07:27:07.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.248+0000 7f23db7fe640 1 -- 172.21.15.83:0/716490486 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23e80047e0 con 0x7f23f415bdd0 2024-03-06T07:27:07.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.248+0000 7f23f9281640 1 -- 172.21.15.83:0/716490486 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23f40b61e0 con 0x7f23f415bdd0 2024-03-06T07:27:07.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.248+0000 7f23db7fe640 1 -- 172.21.15.83:0/716490486 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e8004bd0 con 0x7f23f415bdd0 2024-03-06T07:27:07.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.248+0000 7f23f9281640 1 -- 172.21.15.83:0/716490486 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23f40b67a0 con 0x7f23f415bdd0 2024-03-06T07:27:07.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.249+0000 7f23f9281640 1 -- 172.21.15.83:0/716490486 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23f404f330 con 0x7f23f415bdd0 2024-03-06T07:27:07.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.249+0000 7f23db7fe640 1 -- 172.21.15.83:0/716490486 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f23e8011070 con 0x7f23f415bdd0 2024-03-06T07:27:07.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.249+0000 7f23db7fe640 1 --2- 172.21.15.83:0/716490486 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f23e006b4d0 0x7f23e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.249+0000 7f23db7fe640 1 -- 172.21.15.83:0/716490486 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f23e8089a30 con 0x7f23f415bdd0 2024-03-06T07:27:07.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.250+0000 7f23f2ffd640 1 --2- 172.21.15.83:0/716490486 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f23e006b4d0 0x7f23e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.250+0000 7f23f2ffd640 1 --2- 172.21.15.83:0/716490486 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f23e006b4d0 0x7f23e006d990 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f23e400a950 tx=0x7f23e4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.254+0000 7f23db7fe640 1 -- 172.21.15.83:0/716490486 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f23f4155540 con 0x7f23f415bdd0 2024-03-06T07:27:07.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.491+0000 7f23f9281640 1 -- 172.21.15.83:0/716490486 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f23f40b7a40 con 0x7f23f415bdd0 2024-03-06T07:27:07.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.492+0000 7f23db7fe640 1 -- 172.21.15.83:0/716490486 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f23e800e280 con 0x7f23f415bdd0 2024-03-06T07:27:07.494 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:07.494 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:27:07.494 INFO:teuthology.orchestra.run.smithi083.stdout: "auth_dump": [ 2024-03-06T07:27:07.494 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:27:07.494 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mds.a", 2024-03-06T07:27:07.494 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlVHaUGhAAVRpLsHYTUkaYDmceq5Rlgw==", 2024-03-06T07:27:07.494 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:27:07.495 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:27:07.495 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow *", 2024-03-06T07:27:07.495 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow *", 2024-03-06T07:27:07.495 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:27:07.495 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:27:07.495 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:27:07.495 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:27:07.495 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mds.b", 2024-03-06T07:27:07.495 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlFlrrJhAAQbY6xU910eYRg1zwgpj9vg==", 2024-03-06T07:27:07.495 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:27:07.495 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:27:07.495 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow *", 2024-03-06T07:27:07.495 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow *", 2024-03-06T07:27:07.495 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:27:07.496 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:27:07.496 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:27:07.496 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:27:07.496 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mds.c", 2024-03-06T07:27:07.496 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlX0jxIRAAxYNi39qxIFXazlL2qNzLCg==", 2024-03-06T07:27:07.496 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:27:07.496 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:27:07.496 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow *", 2024-03-06T07:27:07.496 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow *", 2024-03-06T07:27:07.496 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:27:07.496 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:27:07.496 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:27:07.496 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:27:07.496 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mds.d", 2024-03-06T07:27:07.497 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehl3utCLhAAzEjASLnYpPYrJsWOt3WwpQ==", 2024-03-06T07:27:07.497 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:27:07.497 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:27:07.497 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow *", 2024-03-06T07:27:07.497 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow *", 2024-03-06T07:27:07.497 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:27:07.497 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:27:07.497 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:27:07.497 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:27:07.497 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.0", 2024-03-06T07:27:07.497 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBgGehlkezBExAAZOLMH2ua3CTWAL7H6LGeZw==", 2024-03-06T07:27:07.497 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:27:07.497 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:07.497 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:27:07.497 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:27:07.497 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:27:07.498 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:27:07.498 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:27:07.498 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.1", 2024-03-06T07:27:07.498 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBjGehlC04qEhAAqirTFR6y3NOGw6N+iz3/pg==", 2024-03-06T07:27:07.498 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:27:07.498 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:07.498 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:27:07.498 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:27:07.498 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:27:07.498 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:27:07.498 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:27:07.498 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.2", 2024-03-06T07:27:07.498 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBmGehlfuFWEBAAj/vjZXrv1wVNSJ1R74UmnQ==", 2024-03-06T07:27:07.498 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:27:07.498 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:07.499 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:27:07.499 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:27:07.499 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:27:07.499 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:27:07.499 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:27:07.499 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.3", 2024-03-06T07:27:07.499 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBpGehl5+ItDhAA0YNc83MhKcgb3JgRuP87aw==", 2024-03-06T07:27:07.499 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:27:07.499 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:07.499 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:27:07.499 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:27:07.499 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:27:07.499 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:27:07.499 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:27:07.500 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.4", 2024-03-06T07:27:07.500 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBvGehlYa/TFBAAvGcizpRQ21pvgpEGgSET0w==", 2024-03-06T07:27:07.500 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:27:07.500 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:07.500 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:27:07.500 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:27:07.500 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:27:07.500 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:27:07.500 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:27:07.500 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.5", 2024-03-06T07:27:07.500 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQByGehlE80TExAAFcWDgnLtW8OKlyu4B0jnmg==", 2024-03-06T07:27:07.500 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:27:07.500 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:07.500 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:27:07.501 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:27:07.501 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:27:07.501 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:27:07.501 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:27:07.501 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.6", 2024-03-06T07:27:07.501 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQB1Gehla/LbERAAYfAjTKL11VMVnGUGV5GVcg==", 2024-03-06T07:27:07.501 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:27:07.501 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:07.501 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:27:07.501 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:27:07.501 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:27:07.501 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:27:07.501 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:27:07.501 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.7", 2024-03-06T07:27:07.501 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQB4GehlpSJkEBAA17tUgMf94jfqvXlD51P8DQ==", 2024-03-06T07:27:07.502 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:27:07.502 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:07.502 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:27:07.502 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:27:07.502 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:27:07.502 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:27:07.502 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:27:07.502 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.0", 2024-03-06T07:27:07.502 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlRwCIMhAA1DdbdQZPplu7LQOTK3BmSg==", 2024-03-06T07:27:07.502 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:27:07.502 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:27:07.502 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow r", 2024-03-06T07:27:07.502 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:27:07.502 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:27:07.502 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:27:07.503 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:27:07.503 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.1", 2024-03-06T07:27:07.503 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBdGehl77spABAAsiEf+tXos3MwFrXToEQ8dQ==", 2024-03-06T07:27:07.503 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:27:07.503 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:27:07.503 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow r", 2024-03-06T07:27:07.503 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow rw", 2024-03-06T07:27:07.503 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow rwx" 2024-03-06T07:27:07.503 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:27:07.503 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:27:07.503 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:27:07.503 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.2", 2024-03-06T07:27:07.503 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehltY5SOBAAI8rcIEVqMZaWQGgNaYKJTA==", 2024-03-06T07:27:07.503 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:27:07.503 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:27:07.504 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow r", 2024-03-06T07:27:07.504 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow rw", 2024-03-06T07:27:07.504 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow rwx" 2024-03-06T07:27:07.504 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:27:07.504 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:27:07.504 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:27:07.504 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.3", 2024-03-06T07:27:07.504 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBdGehl9ZX1BRAAA7fKnRyaQ+zqHoSGs4UvDw==", 2024-03-06T07:27:07.504 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:27:07.504 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:27:07.504 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow r", 2024-03-06T07:27:07.504 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow rw", 2024-03-06T07:27:07.504 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow rwx" 2024-03-06T07:27:07.504 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:27:07.505 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:27:07.505 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:27:07.505 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.admin", 2024-03-06T07:27:07.505 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBbGehl4gqRORAAobqSWoeyHYA/lXkp6/EcIQ==", 2024-03-06T07:27:07.505 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:27:07.505 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow *", 2024-03-06T07:27:07.505 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow *", 2024-03-06T07:27:07.505 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow *", 2024-03-06T07:27:07.505 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:27:07.505 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:27:07.505 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:27:07.505 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:27:07.505 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mgr.x", 2024-03-06T07:27:07.505 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlfikrFxAAoZkTUtpHlBcUAMpwWABRxg==", 2024-03-06T07:27:07.505 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:27:07.506 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow *", 2024-03-06T07:27:07.506 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile mgr", 2024-03-06T07:27:07.506 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:27:07.506 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:27:07.506 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:27:07.506 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:27:07.506 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mgr.y", 2024-03-06T07:27:07.506 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlvdcdExAAySXlEfJqLnpMySiiYGFkMQ==", 2024-03-06T07:27:07.506 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:27:07.506 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow *", 2024-03-06T07:27:07.506 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile mgr", 2024-03-06T07:27:07.506 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:27:07.506 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:27:07.506 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:27:07.507 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-03-06T07:27:07.507 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:27:07.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.494+0000 7f23f9281640 1 -- 172.21.15.83:0/716490486 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f23e006b4d0 msgr2=0x7f23e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.494+0000 7f23f9281640 1 --2- 172.21.15.83:0/716490486 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f23e006b4d0 0x7f23e006d990 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f23e400a950 tx=0x7f23e4008040 comp rx=0 tx=0).stop 2024-03-06T07:27:07.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.494+0000 7f23f9281640 1 -- 172.21.15.83:0/716490486 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23f415bdd0 msgr2=0x7f23f40b5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.494+0000 7f23f9281640 1 --2- 172.21.15.83:0/716490486 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23f415bdd0 0x7f23f40b5720 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f23e800c430 tx=0x7f23e800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.494+0000 7f23f9281640 1 -- 172.21.15.83:0/716490486 shutdown_connections 2024-03-06T07:27:07.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.494+0000 7f23f9281640 1 --2- 172.21.15.83:0/716490486 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f23e006b4d0 0x7f23e006d990 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.494+0000 7f23f9281640 1 --2- 172.21.15.83:0/716490486 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f23f415fa10 0x7f23f40c96c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.494+0000 7f23f9281640 1 --2- 172.21.15.83:0/716490486 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23f415bdd0 0x7f23f40b5720 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.494+0000 7f23f9281640 1 --2- 172.21.15.83:0/716490486 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f23f40d16e0 0x7f23f4155000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.494+0000 7f23f9281640 1 -- 172.21.15.83:0/716490486 >> 172.21.15.83:0/716490486 conn(0x7f23f409a9f0 msgr2=0x7f23f409add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:07.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.494+0000 7f23f9281640 1 -- 172.21.15.83:0/716490486 shutdown_connections 2024-03-06T07:27:07.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.494+0000 7f23f9281640 1 -- 172.21.15.83:0/716490486 wait complete. 2024-03-06T07:27:07.509 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:07.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.608+0000 7f10a926d640 1 Processor -- start 2024-03-06T07:27:07.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.608+0000 7f10a926d640 1 -- start start 2024-03-06T07:27:07.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.609+0000 7f10a926d640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f10a413dc10 0x7f10a40b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.609+0000 7f10a926d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f10a415ef60 0x7f10a40b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.609+0000 7f10a926d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10a415f580 con 0x7f10a415bf30 2024-03-06T07:27:07.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.609+0000 7f10a926d640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10a40cf3f0 con 0x7f10a415ef60 2024-03-06T07:27:07.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.609+0000 7f10a926d640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10a415b210 con 0x7f10a413dc10 2024-03-06T07:27:07.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.609+0000 7f10a37fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f10a415ef60 0x7f10a40b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.609+0000 7f10a37fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f10a415ef60 0x7f10a40b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47456/0 (socket says 172.21.15.83:47456) 2024-03-06T07:27:07.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.609+0000 7f10a37fe640 1 -- 172.21.15.83:0/3657328588 learned_addr learned my addr 172.21.15.83:0/3657328588 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:07.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.609+0000 7f10a2ffd640 1 --2- 172.21.15.83:0/3657328588 >> v2:172.21.15.183:3300/0 conn(0x7f10a413dc10 0x7f10a40b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.609+0000 7f10a37fe640 1 -- 172.21.15.83:0/3657328588 >> v2:172.21.15.183:3300/0 conn(0x7f10a413dc10 msgr2=0x7f10a40b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:07.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.609+0000 7f10a37fe640 1 --2- 172.21.15.83:0/3657328588 >> v2:172.21.15.183:3300/0 conn(0x7f10a413dc10 0x7f10a40b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.609+0000 7f10a37fe640 1 -- 172.21.15.83:0/3657328588 >> v1:172.21.15.183:6789/0 conn(0x7f10a415bf30 legacy=0x7f10a415e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.609+0000 7f10a37fe640 1 -- 172.21.15.83:0/3657328588 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10a415b450 con 0x7f10a415ef60 2024-03-06T07:27:07.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.609+0000 7f10a2ffd640 1 --2- 172.21.15.83:0/3657328588 >> v2:172.21.15.183:3300/0 conn(0x7f10a413dc10 0x7f10a40b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:07.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.610+0000 7f10a37fe640 1 --2- 172.21.15.83:0/3657328588 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10a415ef60 0x7f10a40b2e90 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f10a40731a0 tx=0x7f109c02e0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53e3db92593acc7f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.610+0000 7f10a27fc640 1 -- 172.21.15.83:0/3657328588 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f109c02ea90 con 0x7f10a415ef60 2024-03-06T07:27:07.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.610+0000 7f10a27fc640 1 -- 172.21.15.83:0/3657328588 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f109c02ec30 con 0x7f10a415ef60 2024-03-06T07:27:07.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.610+0000 7f10a27fc640 1 -- 172.21.15.83:0/3657328588 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f109c009600 con 0x7f10a415ef60 2024-03-06T07:27:07.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.610+0000 7f10a926d640 1 -- 172.21.15.83:0/3657328588 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10a415ef60 msgr2=0x7f10a40b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.610+0000 7f10a926d640 1 --2- 172.21.15.83:0/3657328588 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10a415ef60 0x7f10a40b2e90 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f10a40731a0 tx=0x7f109c02e0f0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.611+0000 7f10a926d640 1 -- 172.21.15.83:0/3657328588 shutdown_connections 2024-03-06T07:27:07.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.611+0000 7f10a926d640 1 --2- 172.21.15.83:0/3657328588 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10a415ef60 0x7f10a40b2e90 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.611+0000 7f10a926d640 1 --2- 172.21.15.83:0/3657328588 >> v2:172.21.15.183:3300/0 conn(0x7f10a413dc10 0x7f10a40b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.611+0000 7f10a926d640 1 -- 172.21.15.83:0/3657328588 >> 172.21.15.83:0/3657328588 conn(0x7f10a40cf8a0 msgr2=0x7f10a40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:07.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.611+0000 7f10a926d640 1 -- 172.21.15.83:0/3657328588 shutdown_connections 2024-03-06T07:27:07.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.611+0000 7f10a926d640 1 -- 172.21.15.83:0/3657328588 wait complete. 2024-03-06T07:27:07.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.612+0000 7f10a926d640 1 Processor -- start 2024-03-06T07:27:07.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.612+0000 7f10a926d640 1 -- start start 2024-03-06T07:27:07.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.612+0000 7f10a926d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10a413dc10 0x7f10a406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.612+0000 7f10a926d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f10a415bf30 0x7f10a415ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.612+0000 7f10a926d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10a415ef60 0x7f10a415cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.612+0000 7f10a926d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a40cd7a0 con 0x7f10a413dc10 2024-03-06T07:27:07.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.612+0000 7f10a926d640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a413dff0 con 0x7f10a415ef60 2024-03-06T07:27:07.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.612+0000 7f10a926d640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10a40cd920 con 0x7f10a415bf30 2024-03-06T07:27:07.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.613+0000 7f10a2ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10a413dc10 0x7f10a406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.613+0000 7f10a2ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10a413dc10 0x7f10a406d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47464/0 (socket says 172.21.15.83:47464) 2024-03-06T07:27:07.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.613+0000 7f10a2ffd640 1 -- 172.21.15.83:0/1597948555 learned_addr learned my addr 172.21.15.83:0/1597948555 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:07.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.613+0000 7f10a37fe640 1 --2- 172.21.15.83:0/1597948555 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10a415ef60 0x7f10a415cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.613+0000 7f10a2ffd640 1 -- 172.21.15.83:0/1597948555 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f10a415bf30 msgr2=0x7f10a415ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.613+0000 7f10a2ffd640 1 --2- 172.21.15.83:0/1597948555 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f10a415bf30 0x7f10a415ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.613+0000 7f10a2ffd640 1 -- 172.21.15.83:0/1597948555 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10a415ef60 msgr2=0x7f10a415cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.613+0000 7f10a2ffd640 1 --2- 172.21.15.83:0/1597948555 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10a415ef60 0x7f10a415cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.613+0000 7f10a2ffd640 1 -- 172.21.15.83:0/1597948555 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10a406da90 con 0x7f10a413dc10 2024-03-06T07:27:07.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.613+0000 7f10a2ffd640 1 --2- 172.21.15.83:0/1597948555 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10a413dc10 0x7f10a406d390 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f109400e810 tx=0x7f109400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.613+0000 7f109b7fe640 1 -- 172.21.15.83:0/1597948555 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f109400f040 con 0x7f10a413dc10 2024-03-06T07:27:07.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.613+0000 7f109b7fe640 1 -- 172.21.15.83:0/1597948555 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1094004030 con 0x7f10a413dc10 2024-03-06T07:27:07.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.613+0000 7f10a926d640 1 -- 172.21.15.83:0/1597948555 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10a406dd70 con 0x7f10a413dc10 2024-03-06T07:27:07.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.613+0000 7f109b7fe640 1 -- 172.21.15.83:0/1597948555 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1094004400 con 0x7f10a413dc10 2024-03-06T07:27:07.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.613+0000 7f10a926d640 1 -- 172.21.15.83:0/1597948555 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10a406e330 con 0x7f10a413dc10 2024-03-06T07:27:07.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.614+0000 7f10a926d640 1 -- 172.21.15.83:0/1597948555 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10a404f330 con 0x7f10a413dc10 2024-03-06T07:27:07.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.614+0000 7f109b7fe640 1 -- 172.21.15.83:0/1597948555 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f109400c6d0 con 0x7f10a413dc10 2024-03-06T07:27:07.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.615+0000 7f109b7fe640 1 --2- 172.21.15.83:0/1597948555 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f108406b4d0 0x7f108406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.615+0000 7f109b7fe640 1 -- 172.21.15.83:0/1597948555 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1094089c00 con 0x7f10a413dc10 2024-03-06T07:27:07.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.615+0000 7f109bfff640 1 --2- 172.21.15.83:0/1597948555 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f108406b4d0 0x7f108406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.615+0000 7f109bfff640 1 --2- 172.21.15.83:0/1597948555 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f108406b4d0 0x7f108406d990 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f108c000f60 tx=0x7f108c0079d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.619+0000 7f109b7fe640 1 -- 172.21.15.83:0/1597948555 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1094090080 con 0x7f10a413dc10 2024-03-06T07:27:07.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.792+0000 7f10a926d640 1 -- 172.21.15.83:0/1597948555 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f10a40cd1a0 con 0x7f10a413dc10 2024-03-06T07:27:07.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.793+0000 7f109b7fe640 1 -- 172.21.15.83:0/1597948555 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) v1 ==== 74+0+12905 (secure 0 0 0) 0x7f1094017020 con 0x7f10a413dc10 2024-03-06T07:27:07.795 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:07.795 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":106,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:27:07.116497+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:07.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.795+0000 7f10a926d640 1 -- 172.21.15.83:0/1597948555 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f108406b4d0 msgr2=0x7f108406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.795+0000 7f10a926d640 1 --2- 172.21.15.83:0/1597948555 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f108406b4d0 0x7f108406d990 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f108c000f60 tx=0x7f108c0079d0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.795+0000 7f10a926d640 1 -- 172.21.15.83:0/1597948555 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10a413dc10 msgr2=0x7f10a406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.795+0000 7f10a926d640 1 --2- 172.21.15.83:0/1597948555 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10a413dc10 0x7f10a406d390 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f109400e810 tx=0x7f109400ecd0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.795+0000 7f10a926d640 1 -- 172.21.15.83:0/1597948555 shutdown_connections 2024-03-06T07:27:07.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.795+0000 7f10a926d640 1 --2- 172.21.15.83:0/1597948555 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f108406b4d0 0x7f108406d990 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.795+0000 7f10a926d640 1 --2- 172.21.15.83:0/1597948555 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10a415ef60 0x7f10a415cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.795+0000 7f10a926d640 1 --2- 172.21.15.83:0/1597948555 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f10a415bf30 0x7f10a415ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.795+0000 7f10a926d640 1 --2- 172.21.15.83:0/1597948555 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10a413dc10 0x7f10a406d390 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.795+0000 7f10a926d640 1 -- 172.21.15.83:0/1597948555 >> 172.21.15.83:0/1597948555 conn(0x7f10a40cf8a0 msgr2=0x7f10a40707d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:07.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.795+0000 7f10a926d640 1 -- 172.21.15.83:0/1597948555 shutdown_connections 2024-03-06T07:27:07.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.795+0000 7f10a926d640 1 -- 172.21.15.83:0/1597948555 wait complete. 2024-03-06T07:27:07.808 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:27:07.808 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:27:07.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.909+0000 7f1118bdd640 1 Processor -- start 2024-03-06T07:27:07.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.909+0000 7f1118bdd640 1 -- start start 2024-03-06T07:27:07.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.910+0000 7f1118bdd640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f11141614d0 0x7f11140d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.910+0000 7f1118bdd640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11141618b0 con 0x7f11140d24b0 2024-03-06T07:27:07.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.910+0000 7f1118bdd640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11140cf400 con 0x7f11140d1910 2024-03-06T07:27:07.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.910+0000 7f1118bdd640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11140cf6a0 con 0x7f11141614d0 2024-03-06T07:27:07.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.910+0000 7f1112d9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f11140d24b0 0x7f11140b0e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:37576/0 (socket says 172.21.15.83:37576) 2024-03-06T07:27:07.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.910+0000 7f1112d9d640 1 -- 172.21.15.83:0/312367394 learned_addr learned my addr 172.21.15.83:0/312367394 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:27:07.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.911+0000 7f111159a640 1 -- 172.21.15.83:0/312367394 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3837548782 0 0) 0x7f11141618b0 con 0x7f11140d24b0 2024-03-06T07:27:07.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.911+0000 7f111259c640 1 --2- 172.21.15.83:0/312367394 >> v2:172.21.15.183:3300/0 conn(0x7f11141614d0 0x7f11140d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.911+0000 7f111159a640 1 -- 172.21.15.83:0/312367394 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10fc003520 con 0x7f11140d24b0 2024-03-06T07:27:07.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.911+0000 7f111159a640 1 -- 172.21.15.83:0/312367394 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2548604032 0 0) 0x7f10fc003520 con 0x7f11140d24b0 2024-03-06T07:27:07.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.911+0000 7f111159a640 1 -- 172.21.15.83:0/312367394 >> v2:172.21.15.183:3300/0 conn(0x7f11141614d0 msgr2=0x7f11140d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:07.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.911+0000 7f111159a640 1 --2- 172.21.15.83:0/312367394 >> v2:172.21.15.183:3300/0 conn(0x7f11141614d0 0x7f11140d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.911+0000 7f111159a640 1 -- 172.21.15.83:0/312367394 >> v1:172.21.15.183:6789/0 conn(0x7f11140d1910 legacy=0x7f11140d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.911+0000 7f111159a640 1 -- 172.21.15.83:0/312367394 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11140cf980 con 0x7f11140d24b0 2024-03-06T07:27:07.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.911+0000 7f111159a640 1 -- 172.21.15.83:0/312367394 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f110c001670 con 0x7f11140d24b0 2024-03-06T07:27:07.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.911+0000 7f111159a640 1 -- 172.21.15.83:0/312367394 >> v1:172.21.15.83:6789/0 conn(0x7f11140d24b0 legacy=0x7f11140b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.911+0000 7f111159a640 1 --2- 172.21.15.83:0/312367394 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10fc0039b0 0x7f10fc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.911+0000 7f111159a640 1 --2- 172.21.15.83:0/312367394 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10fc004800 0x7f10fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.911+0000 7f111159a640 1 --2- 172.21.15.83:0/312367394 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f10fc0051c0 0x7f10fc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.911+0000 7f111159a640 1 -- 172.21.15.83:0/312367394 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11140cf6a0 con 0x7f10fc0039b0 2024-03-06T07:27:07.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.911+0000 7f111159a640 1 -- 172.21.15.83:0/312367394 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10fc00c060 con 0x7f10fc004800 2024-03-06T07:27:07.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.911+0000 7f111159a640 1 -- 172.21.15.83:0/312367394 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10fc00c260 con 0x7f10fc0051c0 2024-03-06T07:27:07.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.911+0000 7f111259c640 1 --2- 172.21.15.83:0/312367394 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10fc0039b0 0x7f10fc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.911+0000 7f1111d9b640 1 --2- 172.21.15.83:0/312367394 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10fc004800 0x7f10fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.911+0000 7f1112d9d640 1 --2- 172.21.15.83:0/312367394 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f10fc0051c0 0x7f10fc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.911+0000 7f111259c640 1 -- 172.21.15.83:0/312367394 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f10fc0051c0 msgr2=0x7f10fc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:07.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.911+0000 7f111259c640 1 --2- 172.21.15.83:0/312367394 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f10fc0051c0 0x7f10fc00b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.911+0000 7f111259c640 1 -- 172.21.15.83:0/312367394 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10fc004800 msgr2=0x7f10fc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:07.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.911+0000 7f111259c640 1 --2- 172.21.15.83:0/312367394 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10fc004800 0x7f10fc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.911+0000 7f111259c640 1 -- 172.21.15.83:0/312367394 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10fc003520 con 0x7f10fc0039b0 2024-03-06T07:27:07.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.911+0000 7f1111d9b640 1 --2- 172.21.15.83:0/312367394 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10fc004800 0x7f10fc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:07.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.911+0000 7f1112d9d640 1 -- 172.21.15.83:0/312367394 reap_dead start 2024-03-06T07:27:07.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.912+0000 7f111259c640 1 --2- 172.21.15.83:0/312367394 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10fc0039b0 0x7f10fc003dd0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f1104009fc0 tx=0x7f10fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2d6c6bdf9cb6125 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.912+0000 7f111159a640 1 -- 172.21.15.83:0/312367394 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1104039070 con 0x7f10fc0039b0 2024-03-06T07:27:07.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.912+0000 7f111159a640 1 -- 172.21.15.83:0/312367394 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f110402cc20 con 0x7f10fc0039b0 2024-03-06T07:27:07.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.912+0000 7f111159a640 1 -- 172.21.15.83:0/312367394 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1104009c10 con 0x7f10fc0039b0 2024-03-06T07:27:07.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.913+0000 7f1118bdd640 1 -- 172.21.15.83:0/312367394 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10fc0039b0 msgr2=0x7f10fc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.913+0000 7f1118bdd640 1 --2- 172.21.15.83:0/312367394 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10fc0039b0 0x7f10fc003dd0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f1104009fc0 tx=0x7f10fc005830 comp rx=0 tx=0).stop 2024-03-06T07:27:07.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.913+0000 7f1118bdd640 1 -- 172.21.15.83:0/312367394 shutdown_connections 2024-03-06T07:27:07.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.913+0000 7f1118bdd640 1 --2- 172.21.15.83:0/312367394 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10fc0039b0 0x7f10fc003dd0 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.913+0000 7f1118bdd640 1 -- 172.21.15.83:0/312367394 >> 172.21.15.83:0/312367394 conn(0x7f11140d3d40 msgr2=0x7f11140d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:07.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.913+0000 7f1118bdd640 1 -- 172.21.15.83:0/312367394 shutdown_connections 2024-03-06T07:27:07.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.913+0000 7f1118bdd640 1 -- 172.21.15.83:0/312367394 wait complete. 2024-03-06T07:27:07.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.914+0000 7f1118bdd640 1 Processor -- start 2024-03-06T07:27:07.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.914+0000 7f1118bdd640 1 -- start start 2024-03-06T07:27:07.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.915+0000 7f1118bdd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f10fc0039b0 0x7f11140b7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.915+0000 7f1118bdd640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11140b91e0 0x7f111415d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.915+0000 7f1118bdd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11140cce60 0x7f11140cd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.915+0000 7f1118bdd640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11140d4860 con 0x7f11140cce60 2024-03-06T07:27:07.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.915+0000 7f1118bdd640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11140b0910 con 0x7f11140b91e0 2024-03-06T07:27:07.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.915+0000 7f1118bdd640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11140d49e0 con 0x7f10fc0039b0 2024-03-06T07:27:07.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.915+0000 7f1112d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11140cce60 0x7f11140cd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.915+0000 7f1112d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11140cce60 0x7f11140cd2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47486/0 (socket says 172.21.15.83:47486) 2024-03-06T07:27:07.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.915+0000 7f1112d9d640 1 -- 172.21.15.83:0/3931054007 learned_addr learned my addr 172.21.15.83:0/3931054007 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:07.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.915+0000 7f1111d9b640 1 --2- 172.21.15.83:0/3931054007 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11140b91e0 0x7f111415d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.915+0000 7f111259c640 1 --2- 172.21.15.83:0/3931054007 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f10fc0039b0 0x7f11140b7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.915+0000 7f1112d9d640 1 -- 172.21.15.83:0/3931054007 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f10fc0039b0 msgr2=0x7f11140b7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.915+0000 7f1112d9d640 1 --2- 172.21.15.83:0/3931054007 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f10fc0039b0 0x7f11140b7310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.915+0000 7f1112d9d640 1 -- 172.21.15.83:0/3931054007 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11140b91e0 msgr2=0x7f111415d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.915+0000 7f1112d9d640 1 --2- 172.21.15.83:0/3931054007 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11140b91e0 0x7f111415d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.915+0000 7f1112d9d640 1 -- 172.21.15.83:0/3931054007 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11140a1720 con 0x7f11140cce60 2024-03-06T07:27:07.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.915+0000 7f111259c640 1 --2- 172.21.15.83:0/3931054007 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f10fc0039b0 0x7f11140b7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:07.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.915+0000 7f1112d9d640 1 --2- 172.21.15.83:0/3931054007 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11140cce60 0x7f11140cd2f0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f11140b6900 tx=0x7f110c00d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.915+0000 7f1102ffd640 1 -- 172.21.15.83:0/3931054007 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110c0075a0 con 0x7f11140cce60 2024-03-06T07:27:07.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.915+0000 7f1102ffd640 1 -- 172.21.15.83:0/3931054007 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f110c0047e0 con 0x7f11140cce60 2024-03-06T07:27:07.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.915+0000 7f1118bdd640 1 -- 172.21.15.83:0/3931054007 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11140b7a40 con 0x7f11140cce60 2024-03-06T07:27:07.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.915+0000 7f1102ffd640 1 -- 172.21.15.83:0/3931054007 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110c004bd0 con 0x7f11140cce60 2024-03-06T07:27:07.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.915+0000 7f1118bdd640 1 -- 172.21.15.83:0/3931054007 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11140b8000 con 0x7f11140cce60 2024-03-06T07:27:07.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.916+0000 7f1118bdd640 1 -- 172.21.15.83:0/3931054007 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11140b8200 con 0x7f11140cce60 2024-03-06T07:27:07.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.917+0000 7f1102ffd640 1 -- 172.21.15.83:0/3931054007 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f110c014430 con 0x7f11140cce60 2024-03-06T07:27:07.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.917+0000 7f1102ffd640 1 --2- 172.21.15.83:0/3931054007 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f10f406b500 0x7f10f406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.917+0000 7f1102ffd640 1 -- 172.21.15.83:0/3931054007 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f110c089900 con 0x7f11140cce60 2024-03-06T07:27:07.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.917+0000 7f111259c640 1 --2- 172.21.15.83:0/3931054007 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f10f406b500 0x7f10f406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.918+0000 7f111259c640 1 --2- 172.21.15.83:0/3931054007 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f10f406b500 0x7f10f406d9c0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f1104036000 tx=0x7f110402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:07.922+0000 7f1102ffd640 1 -- 172.21.15.83:0/3931054007 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f110c004d70 con 0x7f11140cce60 2024-03-06T07:27:08.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.090+0000 7f1118bdd640 1 -- 172.21.15.83:0/3931054007 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1114068fd0 con 0x7f11140cce60 2024-03-06T07:27:08.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.090+0000 7f1102ffd640 1 -- 172.21.15.83:0/3931054007 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v106) v1 ==== 93+0+10 (secure 0 0 0) 0x7f110c058480 con 0x7f11140cce60 2024-03-06T07:27:08.092 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:27:08.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.092+0000 7f1118bdd640 1 -- 172.21.15.83:0/3931054007 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f10f406b500 msgr2=0x7f10f406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.092+0000 7f1118bdd640 1 --2- 172.21.15.83:0/3931054007 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f10f406b500 0x7f10f406d9c0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f1104036000 tx=0x7f110402e000 comp rx=0 tx=0).stop 2024-03-06T07:27:08.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.092+0000 7f1118bdd640 1 -- 172.21.15.83:0/3931054007 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11140cce60 msgr2=0x7f11140cd2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.092+0000 7f1118bdd640 1 --2- 172.21.15.83:0/3931054007 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11140cce60 0x7f11140cd2f0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f11140b6900 tx=0x7f110c00d9d0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.092+0000 7f1118bdd640 1 -- 172.21.15.83:0/3931054007 shutdown_connections 2024-03-06T07:27:08.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.092+0000 7f1118bdd640 1 --2- 172.21.15.83:0/3931054007 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f10f406b500 0x7f10f406d9c0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.092+0000 7f1118bdd640 1 --2- 172.21.15.83:0/3931054007 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11140cce60 0x7f11140cd2f0 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.092+0000 7f1118bdd640 1 --2- 172.21.15.83:0/3931054007 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11140b91e0 0x7f111415d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.092+0000 7f1118bdd640 1 --2- 172.21.15.83:0/3931054007 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f10fc0039b0 0x7f11140b7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.092+0000 7f1118bdd640 1 -- 172.21.15.83:0/3931054007 >> 172.21.15.83:0/3931054007 conn(0x7f11140d3d40 msgr2=0x7f11140b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:08.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.092+0000 7f1118bdd640 1 -- 172.21.15.83:0/3931054007 shutdown_connections 2024-03-06T07:27:08.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.092+0000 7f1118bdd640 1 -- 172.21.15.83:0/3931054007 wait complete. 2024-03-06T07:27:08.105 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:27:08.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.203+0000 7f7e35efa640 1 Processor -- start 2024-03-06T07:27:08.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.203+0000 7f7e35efa640 1 -- start start 2024-03-06T07:27:08.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.204+0000 7f7e35efa640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7e301515e0 0x7f7e300cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.204+0000 7f7e35efa640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e300cd7a0 0x7f7e300cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.204+0000 7f7e35efa640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e30071f40 con 0x7f7e3015b430 2024-03-06T07:27:08.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.204+0000 7f7e35efa640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e3015b8e0 con 0x7f7e301515e0 2024-03-06T07:27:08.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.204+0000 7f7e35efa640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e30070a70 con 0x7f7e300cd7a0 2024-03-06T07:27:08.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.204+0000 7f7e2f7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7e301515e0 0x7f7e300cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.204+0000 7f7e2effd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f7e3015b430 0x7f7e30071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:37586/0 (socket says 172.21.15.83:37586) 2024-03-06T07:27:08.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.204+0000 7f7e2effd640 1 -- 172.21.15.83:0/3055266311 learned_addr learned my addr 172.21.15.83:0/3055266311 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:27:08.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.204+0000 7f7e2ffff640 1 --2- 172.21.15.83:0/3055266311 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e300cd7a0 0x7f7e300cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.205+0000 7f7e2f7fe640 1 -- 172.21.15.83:0/3055266311 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e300cd7a0 msgr2=0x7f7e300cdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:08.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.205+0000 7f7e2f7fe640 1 --2- 172.21.15.83:0/3055266311 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e300cd7a0 0x7f7e300cdbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.205+0000 7f7e2f7fe640 1 -- 172.21.15.83:0/3055266311 >> v1:172.21.15.83:6789/0 conn(0x7f7e3015b430 legacy=0x7f7e30071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.205+0000 7f7e2e7fc640 1 -- 172.21.15.83:0/3055266311 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2052983613 0 0) 0x7f7e30071f40 con 0x7f7e3015b430 2024-03-06T07:27:08.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.205+0000 7f7e2f7fe640 1 -- 172.21.15.83:0/3055266311 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e30070d50 con 0x7f7e301515e0 2024-03-06T07:27:08.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.205+0000 7f7e2f7fe640 1 --2- 172.21.15.83:0/3055266311 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e301515e0 0x7f7e300cce40 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f7e200097a0 tx=0x7f7e2002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d15c2b46aecf4c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:08.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.205+0000 7f7e2e7fc640 1 -- 172.21.15.83:0/3055266311 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e20035020 con 0x7f7e301515e0 2024-03-06T07:27:08.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.205+0000 7f7e2e7fc640 1 -- 172.21.15.83:0/3055266311 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e20031830 con 0x7f7e301515e0 2024-03-06T07:27:08.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.205+0000 7f7e2e7fc640 1 -- 172.21.15.83:0/3055266311 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e20031c00 con 0x7f7e301515e0 2024-03-06T07:27:08.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.206+0000 7f7e35efa640 1 -- 172.21.15.83:0/3055266311 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e301515e0 msgr2=0x7f7e300cce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.206+0000 7f7e35efa640 1 --2- 172.21.15.83:0/3055266311 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e301515e0 0x7f7e300cce40 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f7e200097a0 tx=0x7f7e2002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.206+0000 7f7e35efa640 1 -- 172.21.15.83:0/3055266311 shutdown_connections 2024-03-06T07:27:08.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.206+0000 7f7e35efa640 1 --2- 172.21.15.83:0/3055266311 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e300cd7a0 0x7f7e300cdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.206+0000 7f7e35efa640 1 --2- 172.21.15.83:0/3055266311 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e301515e0 0x7f7e300cce40 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.206+0000 7f7e35efa640 1 -- 172.21.15.83:0/3055266311 >> 172.21.15.83:0/3055266311 conn(0x7f7e300cf8a0 msgr2=0x7f7e300cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:08.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.206+0000 7f7e35efa640 1 -- 172.21.15.83:0/3055266311 shutdown_connections 2024-03-06T07:27:08.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.206+0000 7f7e35efa640 1 -- 172.21.15.83:0/3055266311 wait complete. 2024-03-06T07:27:08.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.207+0000 7f7e35efa640 1 Processor -- start 2024-03-06T07:27:08.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.207+0000 7f7e35efa640 1 -- start start 2024-03-06T07:27:08.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.208+0000 7f7e35efa640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e300cd7a0 0x7f7e301444f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.208+0000 7f7e35efa640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e301515e0 0x7f7e300c80c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.208+0000 7f7e35efa640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e3015b430 0x7f7e3006afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.208+0000 7f7e35efa640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e300ce0f0 con 0x7f7e3015b430 2024-03-06T07:27:08.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.208+0000 7f7e35efa640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e30071310 con 0x7f7e301515e0 2024-03-06T07:27:08.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.208+0000 7f7e35efa640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e300b3f90 con 0x7f7e300cd7a0 2024-03-06T07:27:08.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.208+0000 7f7e2ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e3015b430 0x7f7e3006afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.208+0000 7f7e2ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e3015b430 0x7f7e3006afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47500/0 (socket says 172.21.15.83:47500) 2024-03-06T07:27:08.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.208+0000 7f7e2ffff640 1 -- 172.21.15.83:0/9459606 learned_addr learned my addr 172.21.15.83:0/9459606 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:08.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.208+0000 7f7e2ffff640 1 -- 172.21.15.83:0/9459606 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e300cd7a0 msgr2=0x7f7e301444f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.208+0000 7f7e2f7fe640 1 --2- 172.21.15.83:0/9459606 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e300cd7a0 0x7f7e301444f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.208+0000 7f7e2effd640 1 --2- 172.21.15.83:0/9459606 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e301515e0 0x7f7e300c80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.208+0000 7f7e2ffff640 1 --2- 172.21.15.83:0/9459606 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e300cd7a0 0x7f7e301444f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.208+0000 7f7e2ffff640 1 -- 172.21.15.83:0/9459606 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e301515e0 msgr2=0x7f7e300c80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.208+0000 7f7e2ffff640 1 --2- 172.21.15.83:0/9459606 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e301515e0 0x7f7e300c80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.208+0000 7f7e2ffff640 1 -- 172.21.15.83:0/9459606 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e300c8aa0 con 0x7f7e3015b430 2024-03-06T07:27:08.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.208+0000 7f7e2effd640 1 --2- 172.21.15.83:0/9459606 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e301515e0 0x7f7e300c80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:08.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.209+0000 7f7e2ffff640 1 --2- 172.21.15.83:0/9459606 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e3015b430 0x7f7e3006afb0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f7e28007e40 tx=0x7f7e2800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:08.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.209+0000 7f7e17fff640 1 -- 172.21.15.83:0/9459606 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e28014620 con 0x7f7e3015b430 2024-03-06T07:27:08.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.209+0000 7f7e17fff640 1 -- 172.21.15.83:0/9459606 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e28019070 con 0x7f7e3015b430 2024-03-06T07:27:08.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.209+0000 7f7e17fff640 1 -- 172.21.15.83:0/9459606 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e28014a20 con 0x7f7e3015b430 2024-03-06T07:27:08.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.209+0000 7f7e35efa640 1 -- 172.21.15.83:0/9459606 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e300c8d80 con 0x7f7e3015b430 2024-03-06T07:27:08.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.209+0000 7f7e35efa640 1 -- 172.21.15.83:0/9459606 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e300c9340 con 0x7f7e3015b430 2024-03-06T07:27:08.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.210+0000 7f7e35efa640 1 -- 172.21.15.83:0/9459606 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e30068840 con 0x7f7e3015b430 2024-03-06T07:27:08.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.210+0000 7f7e17fff640 1 -- 172.21.15.83:0/9459606 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7e28014bc0 con 0x7f7e3015b430 2024-03-06T07:27:08.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.210+0000 7f7e17fff640 1 --2- 172.21.15.83:0/9459606 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7e1c06b4d0 0x7f7e1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.211+0000 7f7e17fff640 1 -- 172.21.15.83:0/9459606 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7e2808a7b0 con 0x7f7e3015b430 2024-03-06T07:27:08.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.211+0000 7f7e2f7fe640 1 --2- 172.21.15.83:0/9459606 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7e1c06b4d0 0x7f7e1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.211+0000 7f7e2f7fe640 1 --2- 172.21.15.83:0/9459606 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7e1c06b4d0 0x7f7e1c06d990 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f7e20036000 tx=0x7f7e20008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:08.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.215+0000 7f7e17fff640 1 -- 172.21.15.83:0/9459606 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7e28059a10 con 0x7f7e3015b430 2024-03-06T07:27:08.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.390+0000 7f7e35efa640 1 -- 172.21.15.83:0/9459606 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f7e3004f330 con 0x7f7e3015b430 2024-03-06T07:27:08.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.391+0000 7f7e17fff640 1 -- 172.21.15.83:0/9459606 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v106) v1 ==== 84+0+65 (secure 0 0 0) 0x7f7e28059310 con 0x7f7e3015b430 2024-03-06T07:27:08.393 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:08.393 INFO:teuthology.orchestra.run.smithi083.stdout:[ 2024-03-06T07:27:08.393 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:27:08.393 INFO:teuthology.orchestra.run.smithi083.stdout: "poolnum": 1, 2024-03-06T07:27:08.393 INFO:teuthology.orchestra.run.smithi083.stdout: "poolname": ".mgr" 2024-03-06T07:27:08.393 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:27:08.393 INFO:teuthology.orchestra.run.smithi083.stdout:] 2024-03-06T07:27:08.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.393+0000 7f7e35efa640 1 -- 172.21.15.83:0/9459606 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7e1c06b4d0 msgr2=0x7f7e1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.393+0000 7f7e35efa640 1 --2- 172.21.15.83:0/9459606 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7e1c06b4d0 0x7f7e1c06d990 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f7e20036000 tx=0x7f7e20008030 comp rx=0 tx=0).stop 2024-03-06T07:27:08.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.393+0000 7f7e35efa640 1 -- 172.21.15.83:0/9459606 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e3015b430 msgr2=0x7f7e3006afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.393+0000 7f7e35efa640 1 --2- 172.21.15.83:0/9459606 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e3015b430 0x7f7e3006afb0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f7e28007e40 tx=0x7f7e2800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.393+0000 7f7e35efa640 1 -- 172.21.15.83:0/9459606 shutdown_connections 2024-03-06T07:27:08.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.393+0000 7f7e35efa640 1 --2- 172.21.15.83:0/9459606 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7e1c06b4d0 0x7f7e1c06d990 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.393+0000 7f7e35efa640 1 --2- 172.21.15.83:0/9459606 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e3015b430 0x7f7e3006afb0 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.393+0000 7f7e35efa640 1 --2- 172.21.15.83:0/9459606 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e301515e0 0x7f7e300c80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.393+0000 7f7e35efa640 1 --2- 172.21.15.83:0/9459606 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e300cd7a0 0x7f7e301444f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.393+0000 7f7e35efa640 1 -- 172.21.15.83:0/9459606 >> 172.21.15.83:0/9459606 conn(0x7f7e300cf8a0 msgr2=0x7f7e3014e540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:08.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.393+0000 7f7e35efa640 1 -- 172.21.15.83:0/9459606 shutdown_connections 2024-03-06T07:27:08.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.393+0000 7f7e35efa640 1 -- 172.21.15.83:0/9459606 wait complete. 2024-03-06T07:27:08.405 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:27:08.406 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:27:08.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.508+0000 7fe9299a0640 1 Processor -- start 2024-03-06T07:27:08.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.508+0000 7fe9299a0640 1 -- start start 2024-03-06T07:27:08.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.509+0000 7fe9299a0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe924144f60 0x7fe92415f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.509+0000 7fe9299a0640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe92415fd00 con 0x7fe92406b6c0 2024-03-06T07:27:08.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.509+0000 7fe9299a0640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe92415ffe0 con 0x7fe9240cb860 2024-03-06T07:27:08.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.509+0000 7fe9299a0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9241602c0 con 0x7fe924144f60 2024-03-06T07:27:08.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.509+0000 7fe923fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fe92406b6c0 0x7fe9240ceb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:37590/0 (socket says 172.21.15.83:37590) 2024-03-06T07:27:08.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.509+0000 7fe923fff640 1 -- 172.21.15.83:0/1051130060 learned_addr learned my addr 172.21.15.83:0/1051130060 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:27:08.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.510+0000 7fe92919f640 1 --2- 172.21.15.83:0/1051130060 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe924144f60 0x7fe92415f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.510+0000 7fe9237fe640 1 -- 172.21.15.83:0/1051130060 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2612625329 0 0) 0x7fe92415fd00 con 0x7fe92406b6c0 2024-03-06T07:27:08.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.510+0000 7fe9237fe640 1 -- 172.21.15.83:0/1051130060 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe90c003520 con 0x7fe92406b6c0 2024-03-06T07:27:08.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.510+0000 7fe9237fe640 1 -- 172.21.15.83:0/1051130060 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2886629117 0 0) 0x7fe92415ffe0 con 0x7fe9240cb860 2024-03-06T07:27:08.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.510+0000 7fe9237fe640 1 -- 172.21.15.83:0/1051130060 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe92415fd00 con 0x7fe9240cb860 2024-03-06T07:27:08.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.510+0000 7fe9237fe640 1 -- 172.21.15.83:0/1051130060 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1777443710 0 0) 0x7fe90c003520 con 0x7fe92406b6c0 2024-03-06T07:27:08.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.510+0000 7fe9237fe640 1 -- 172.21.15.83:0/1051130060 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe924144f60 msgr2=0x7fe92415f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:08.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.510+0000 7fe9237fe640 1 --2- 172.21.15.83:0/1051130060 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe924144f60 0x7fe92415f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.510+0000 7fe9237fe640 1 -- 172.21.15.83:0/1051130060 >> v1:172.21.15.183:6789/0 conn(0x7fe9240cb860 legacy=0x7fe92406afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.510+0000 7fe9237fe640 1 -- 172.21.15.83:0/1051130060 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9241605a0 con 0x7fe92406b6c0 2024-03-06T07:27:08.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.510+0000 7fe9237fe640 1 -- 172.21.15.83:0/1051130060 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7fe9180015f0 con 0x7fe92406b6c0 2024-03-06T07:27:08.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.510+0000 7fe9237fe640 1 -- 172.21.15.83:0/1051130060 >> v1:172.21.15.83:6789/0 conn(0x7fe92406b6c0 legacy=0x7fe9240ceb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.510+0000 7fe9237fe640 1 --2- 172.21.15.83:0/1051130060 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe924144f60 0x7fe90c003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.510+0000 7fe9237fe640 1 --2- 172.21.15.83:0/1051130060 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe90c004120 0x7fe90c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.510+0000 7fe9237fe640 1 --2- 172.21.15.83:0/1051130060 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe90c004d40 0x7fe90c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.510+0000 7fe9237fe640 1 -- 172.21.15.83:0/1051130060 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9241602c0 con 0x7fe90c004d40 2024-03-06T07:27:08.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.510+0000 7fe9237fe640 1 -- 172.21.15.83:0/1051130060 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe90c00bc20 con 0x7fe90c004120 2024-03-06T07:27:08.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.510+0000 7fe9237fe640 1 -- 172.21.15.83:0/1051130060 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe90c00be20 con 0x7fe924144f60 2024-03-06T07:27:08.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.510+0000 7fe92919f640 1 --2- 172.21.15.83:0/1051130060 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe90c004d40 0x7fe90c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.510+0000 7fe92899e640 1 --2- 172.21.15.83:0/1051130060 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe924144f60 0x7fe90c003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.510+0000 7fe923fff640 1 --2- 172.21.15.83:0/1051130060 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe90c004120 0x7fe90c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.510+0000 7fe92919f640 1 -- 172.21.15.83:0/1051130060 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe924144f60 msgr2=0x7fe90c003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:08.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.510+0000 7fe92919f640 1 --2- 172.21.15.83:0/1051130060 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe924144f60 0x7fe90c003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.510+0000 7fe92919f640 1 -- 172.21.15.83:0/1051130060 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe90c004120 msgr2=0x7fe90c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:08.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.510+0000 7fe92919f640 1 --2- 172.21.15.83:0/1051130060 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe90c004120 0x7fe90c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.510+0000 7fe92919f640 1 -- 172.21.15.83:0/1051130060 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe90c003520 con 0x7fe90c004d40 2024-03-06T07:27:08.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.510+0000 7fe923fff640 1 --2- 172.21.15.83:0/1051130060 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe90c004120 0x7fe90c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:08.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.510+0000 7fe92899e640 1 --2- 172.21.15.83:0/1051130060 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe924144f60 0x7fe90c003b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:08.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.511+0000 7fe92919f640 1 --2- 172.21.15.83:0/1051130060 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe90c004d40 0x7fe90c0051c0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7fe91c009fc0 tx=0x7fe90c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af990c7893a205a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:08.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.511+0000 7fe9237fe640 1 -- 172.21.15.83:0/1051130060 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91c039070 con 0x7fe90c004d40 2024-03-06T07:27:08.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.511+0000 7fe9237fe640 1 -- 172.21.15.83:0/1051130060 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe91c02cc20 con 0x7fe90c004d40 2024-03-06T07:27:08.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.511+0000 7fe9237fe640 1 -- 172.21.15.83:0/1051130060 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91c009c10 con 0x7fe90c004d40 2024-03-06T07:27:08.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.512+0000 7fe9299a0640 1 -- 172.21.15.83:0/1051130060 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe90c004d40 msgr2=0x7fe90c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.512+0000 7fe9299a0640 1 --2- 172.21.15.83:0/1051130060 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe90c004d40 0x7fe90c0051c0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7fe91c009fc0 tx=0x7fe90c005830 comp rx=0 tx=0).stop 2024-03-06T07:27:08.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.512+0000 7fe92919f640 1 -- 172.21.15.83:0/1051130060 reap_dead start 2024-03-06T07:27:08.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.512+0000 7fe9299a0640 1 -- 172.21.15.83:0/1051130060 shutdown_connections 2024-03-06T07:27:08.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.512+0000 7fe9299a0640 1 -- 172.21.15.83:0/1051130060 >> 172.21.15.83:0/1051130060 conn(0x7fe9240d09c0 msgr2=0x7fe9240d0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:08.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.512+0000 7fe9299a0640 1 -- 172.21.15.83:0/1051130060 shutdown_connections 2024-03-06T07:27:08.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.512+0000 7fe9299a0640 1 -- 172.21.15.83:0/1051130060 wait complete. 2024-03-06T07:27:08.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.513+0000 7fe9299a0640 1 Processor -- start 2024-03-06T07:27:08.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.513+0000 7fe9299a0640 1 -- start start 2024-03-06T07:27:08.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.514+0000 7fe9299a0640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe924141810 0x7fe9240b5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.514+0000 7fe9299a0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe9240b4730 0x7fe9241f29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.514+0000 7fe9299a0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9240c7dd0 0x7fe9240c8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.514+0000 7fe9299a0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9240cbdc0 con 0x7fe9240c7dd0 2024-03-06T07:27:08.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.514+0000 7fe9299a0640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9240cbc40 con 0x7fe924141810 2024-03-06T07:27:08.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.514+0000 7fe9299a0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe92415f720 con 0x7fe9240b4730 2024-03-06T07:27:08.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.514+0000 7fe92919f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9240c7dd0 0x7fe9240c8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.514+0000 7fe92919f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9240c7dd0 0x7fe9240c8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47512/0 (socket says 172.21.15.83:47512) 2024-03-06T07:27:08.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.514+0000 7fe92919f640 1 -- 172.21.15.83:0/1931831807 learned_addr learned my addr 172.21.15.83:0/1931831807 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:08.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.514+0000 7fe92899e640 1 --2- 172.21.15.83:0/1931831807 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe924141810 0x7fe9240b5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.514+0000 7fe923fff640 1 --2- 172.21.15.83:0/1931831807 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe9240b4730 0x7fe9241f29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.514+0000 7fe92919f640 1 -- 172.21.15.83:0/1931831807 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe9240b4730 msgr2=0x7fe9241f29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.514+0000 7fe92919f640 1 --2- 172.21.15.83:0/1931831807 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe9240b4730 0x7fe9241f29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.514+0000 7fe92919f640 1 -- 172.21.15.83:0/1931831807 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe924141810 msgr2=0x7fe9240b5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.514+0000 7fe92919f640 1 --2- 172.21.15.83:0/1931831807 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe924141810 0x7fe9240b5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.514+0000 7fe92919f640 1 -- 172.21.15.83:0/1931831807 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9240af280 con 0x7fe9240c7dd0 2024-03-06T07:27:08.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.514+0000 7fe92899e640 1 --2- 172.21.15.83:0/1931831807 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe924141810 0x7fe9240b5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:08.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.514+0000 7fe92919f640 1 --2- 172.21.15.83:0/1931831807 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9240c7dd0 0x7fe9240c8260 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7fe91c002230 tx=0x7fe91c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:08.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.514+0000 7fe9217fa640 1 -- 172.21.15.83:0/1931831807 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91c039030 con 0x7fe9240c7dd0 2024-03-06T07:27:08.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.515+0000 7fe9217fa640 1 -- 172.21.15.83:0/1931831807 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe91c043070 con 0x7fe9240c7dd0 2024-03-06T07:27:08.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.515+0000 7fe9299a0640 1 -- 172.21.15.83:0/1931831807 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9240b4bc0 con 0x7fe9240c7dd0 2024-03-06T07:27:08.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.515+0000 7fe9217fa640 1 -- 172.21.15.83:0/1931831807 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91c009c10 con 0x7fe9240c7dd0 2024-03-06T07:27:08.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.515+0000 7fe9299a0640 1 -- 172.21.15.83:0/1931831807 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9240b5180 con 0x7fe9240c7dd0 2024-03-06T07:27:08.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.516+0000 7fe9299a0640 1 -- 172.21.15.83:0/1931831807 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe924068f10 con 0x7fe9240c7dd0 2024-03-06T07:27:08.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.516+0000 7fe9217fa640 1 -- 172.21.15.83:0/1931831807 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe91c005d10 con 0x7fe9240c7dd0 2024-03-06T07:27:08.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.516+0000 7fe9217fa640 1 --2- 172.21.15.83:0/1931831807 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe91006b4d0 0x7fe91006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.517+0000 7fe92899e640 1 --2- 172.21.15.83:0/1931831807 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe91006b4d0 0x7fe91006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.517+0000 7fe9217fa640 1 -- 172.21.15.83:0/1931831807 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe91c0ac000 con 0x7fe9240c7dd0 2024-03-06T07:27:08.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.517+0000 7fe92899e640 1 --2- 172.21.15.83:0/1931831807 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe91006b4d0 0x7fe91006d990 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fe9240b62e0 tx=0x7fe914009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:08.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.521+0000 7fe9217fa640 1 -- 172.21.15.83:0/1931831807 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe91c07b260 con 0x7fe9240c7dd0 2024-03-06T07:27:08.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:08.690+0000 7fe9299a0640 1 -- 172.21.15.83:0/1931831807 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fe924068480 con 0x7fe9240c7dd0 2024-03-06T07:27:09.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.125+0000 7fe9217fa640 1 -- 172.21.15.83:0/1931831807 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v107) v1 ==== 138+0+0 (secure 0 0 0) 0x7fe91c07ab60 con 0x7fe9240c7dd0 2024-03-06T07:27:09.127 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'cephfs_metadata' created 2024-03-06T07:27:09.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.127+0000 7fe9299a0640 1 -- 172.21.15.83:0/1931831807 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe91006b4d0 msgr2=0x7fe91006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.127+0000 7fe9299a0640 1 --2- 172.21.15.83:0/1931831807 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe91006b4d0 0x7fe91006d990 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fe9240b62e0 tx=0x7fe914009040 comp rx=0 tx=0).stop 2024-03-06T07:27:09.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.128+0000 7fe9299a0640 1 -- 172.21.15.83:0/1931831807 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9240c7dd0 msgr2=0x7fe9240c8260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.128+0000 7fe9299a0640 1 --2- 172.21.15.83:0/1931831807 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9240c7dd0 0x7fe9240c8260 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7fe91c002230 tx=0x7fe91c009750 comp rx=0 tx=0).stop 2024-03-06T07:27:09.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.128+0000 7fe9299a0640 1 -- 172.21.15.83:0/1931831807 shutdown_connections 2024-03-06T07:27:09.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.128+0000 7fe9299a0640 1 --2- 172.21.15.83:0/1931831807 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe91006b4d0 0x7fe91006d990 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.128+0000 7fe9299a0640 1 --2- 172.21.15.83:0/1931831807 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9240c7dd0 0x7fe9240c8260 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.128+0000 7fe9299a0640 1 --2- 172.21.15.83:0/1931831807 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe9240b4730 0x7fe9241f29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.128+0000 7fe9299a0640 1 --2- 172.21.15.83:0/1931831807 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe924141810 0x7fe9240b5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.128+0000 7fe9299a0640 1 -- 172.21.15.83:0/1931831807 >> 172.21.15.83:0/1931831807 conn(0x7fe9240d09c0 msgr2=0x7fe9240cd260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:09.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.128+0000 7fe9299a0640 1 -- 172.21.15.83:0/1931831807 shutdown_connections 2024-03-06T07:27:09.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.128+0000 7fe9299a0640 1 -- 172.21.15.83:0/1931831807 wait complete. 2024-03-06T07:27:09.145 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:27:09.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.251+0000 7f16be0c2640 1 Processor -- start 2024-03-06T07:27:09.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.251+0000 7f16be0c2640 1 -- start start 2024-03-06T07:27:09.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.252+0000 7f16be0c2640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f16b80cf9a0 0x7f16b80b5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.252+0000 7f16be0c2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16b80b55e0 0x7f16b80ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.252+0000 7f16be0c2640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b806b710 con 0x7f16b8066d90 2024-03-06T07:27:09.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.252+0000 7f16be0c2640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16b806b150 con 0x7f16b80cf9a0 2024-03-06T07:27:09.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.252+0000 7f16be0c2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16b806b430 con 0x7f16b80b55e0 2024-03-06T07:27:09.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.252+0000 7f16b6ffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f16b8066d90 0x7f16b80b4ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:37596/0 (socket says 172.21.15.83:37596) 2024-03-06T07:27:09.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.252+0000 7f16b6ffd640 1 -- 172.21.15.83:0/2136587941 learned_addr learned my addr 172.21.15.83:0/2136587941 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:27:09.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.252+0000 7f16b67fc640 1 -- 172.21.15.83:0/2136587941 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2075489369 0 0) 0x7f16b806b710 con 0x7f16b8066d90 2024-03-06T07:27:09.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.252+0000 7f16b67fc640 1 -- 172.21.15.83:0/2136587941 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16a0003520 con 0x7f16b8066d90 2024-03-06T07:27:09.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.252+0000 7f16b7fff640 1 --2- 172.21.15.83:0/2136587941 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16b80b55e0 0x7f16b80ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.252+0000 7f16b77fe640 1 --2- 172.21.15.83:0/2136587941 >> v2:172.21.15.183:3300/0 conn(0x7f16b80cf9a0 0x7f16b80b5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.252+0000 7f16b67fc640 1 -- 172.21.15.83:0/2136587941 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2145380857 0 0) 0x7f16a0003520 con 0x7f16b8066d90 2024-03-06T07:27:09.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.253+0000 7f16b67fc640 1 -- 172.21.15.83:0/2136587941 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16b80b55e0 msgr2=0x7f16b80ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:09.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.253+0000 7f16b67fc640 1 --2- 172.21.15.83:0/2136587941 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16b80b55e0 0x7f16b80ce040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.253+0000 7f16b67fc640 1 -- 172.21.15.83:0/2136587941 >> v2:172.21.15.183:3300/0 conn(0x7f16b80cf9a0 msgr2=0x7f16b80b5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:09.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.253+0000 7f16b67fc640 1 --2- 172.21.15.83:0/2136587941 >> v2:172.21.15.183:3300/0 conn(0x7f16b80cf9a0 0x7f16b80b5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.253+0000 7f16b67fc640 1 -- 172.21.15.83:0/2136587941 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b80b0970 con 0x7f16b8066d90 2024-03-06T07:27:09.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.253+0000 7f16b67fc640 1 -- 172.21.15.83:0/2136587941 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f16ac001670 con 0x7f16b8066d90 2024-03-06T07:27:09.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.253+0000 7f16b67fc640 1 -- 172.21.15.83:0/2136587941 >> v1:172.21.15.83:6789/0 conn(0x7f16b8066d90 legacy=0x7f16b80b4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.253+0000 7f16b67fc640 1 --2- 172.21.15.83:0/2136587941 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16a00039b0 0x7f16a0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.253+0000 7f16b67fc640 1 --2- 172.21.15.83:0/2136587941 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16a0004800 0x7f16a0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.253+0000 7f16b67fc640 1 --2- 172.21.15.83:0/2136587941 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16a00051c0 0x7f16a000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.253+0000 7f16b67fc640 1 -- 172.21.15.83:0/2136587941 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b806b150 con 0x7f16a0004800 2024-03-06T07:27:09.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.253+0000 7f16b67fc640 1 -- 172.21.15.83:0/2136587941 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b806b430 con 0x7f16a00051c0 2024-03-06T07:27:09.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.253+0000 7f16b67fc640 1 -- 172.21.15.83:0/2136587941 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16a000c000 con 0x7f16a00039b0 2024-03-06T07:27:09.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.253+0000 7f16b7fff640 1 --2- 172.21.15.83:0/2136587941 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16b80b55e0 0x7f16b80ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:09.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.253+0000 7f16b6ffd640 1 --2- 172.21.15.83:0/2136587941 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16a0004800 0x7f16a0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.253+0000 7f16b77fe640 1 --2- 172.21.15.83:0/2136587941 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16a00039b0 0x7f16a0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.253+0000 7f16b7fff640 1 --2- 172.21.15.83:0/2136587941 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16a00051c0 0x7f16a000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.253+0000 7f16b6ffd640 1 -- 172.21.15.83:0/2136587941 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16a00039b0 msgr2=0x7f16a0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:09.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.253+0000 7f16b6ffd640 1 --2- 172.21.15.83:0/2136587941 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16a00039b0 0x7f16a0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.253+0000 7f16b6ffd640 1 -- 172.21.15.83:0/2136587941 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16a00051c0 msgr2=0x7f16a000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:09.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.253+0000 7f16b6ffd640 1 --2- 172.21.15.83:0/2136587941 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16a00051c0 0x7f16a000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.253+0000 7f16b6ffd640 1 -- 172.21.15.83:0/2136587941 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a0003520 con 0x7f16a0004800 2024-03-06T07:27:09.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.254+0000 7f16b6ffd640 1 --2- 172.21.15.83:0/2136587941 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16a0004800 0x7f16a0004c80 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f16ac00c210 tx=0x7f16a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e097c92097cf7d68 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:09.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.254+0000 7f16b67fc640 1 -- 172.21.15.83:0/2136587941 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ac007d50 con 0x7f16a0004800 2024-03-06T07:27:09.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.254+0000 7f16b67fc640 1 -- 172.21.15.83:0/2136587941 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16ac030070 con 0x7f16a0004800 2024-03-06T07:27:09.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.254+0000 7f16b67fc640 1 -- 172.21.15.83:0/2136587941 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ac02cda0 con 0x7f16a0004800 2024-03-06T07:27:09.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.254+0000 7f16be0c2640 1 -- 172.21.15.83:0/2136587941 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16a0004800 msgr2=0x7f16a0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.254+0000 7f16be0c2640 1 --2- 172.21.15.83:0/2136587941 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16a0004800 0x7f16a0004c80 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f16ac00c210 tx=0x7f16a0005830 comp rx=0 tx=0).stop 2024-03-06T07:27:09.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.254+0000 7f16b7fff640 1 -- 172.21.15.83:0/2136587941 reap_dead start 2024-03-06T07:27:09.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.254+0000 7f16be0c2640 1 -- 172.21.15.83:0/2136587941 shutdown_connections 2024-03-06T07:27:09.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.254+0000 7f16be0c2640 1 -- 172.21.15.83:0/2136587941 >> 172.21.15.83:0/2136587941 conn(0x7f16b80b1d70 msgr2=0x7f16b80b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:09.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.254+0000 7f16be0c2640 1 -- 172.21.15.83:0/2136587941 shutdown_connections 2024-03-06T07:27:09.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.254+0000 7f16be0c2640 1 -- 172.21.15.83:0/2136587941 wait complete. 2024-03-06T07:27:09.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.256+0000 7f16be0c2640 1 Processor -- start 2024-03-06T07:27:09.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.256+0000 7f16be0c2640 1 -- start start 2024-03-06T07:27:09.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.256+0000 7f16be0c2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16a0004800 0x7f16b80c5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.256+0000 7f16be0c2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16b80caac0 0x7f16b8155950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.256+0000 7f16be0c2640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16b81583b0 0x7f16b80ae0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.256+0000 7f16be0c2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b80b6170 con 0x7f16b80caac0 2024-03-06T07:27:09.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.256+0000 7f16be0c2640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b80b5ff0 con 0x7f16b81583b0 2024-03-06T07:27:09.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.256+0000 7f16be0c2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16b80cfd80 con 0x7f16a0004800 2024-03-06T07:27:09.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.257+0000 7f16b6ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16b80caac0 0x7f16b8155950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.257+0000 7f16b6ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16b80caac0 0x7f16b8155950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47544/0 (socket says 172.21.15.83:47544) 2024-03-06T07:27:09.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.257+0000 7f16b6ffd640 1 -- 172.21.15.83:0/3756357634 learned_addr learned my addr 172.21.15.83:0/3756357634 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:09.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.257+0000 7f16b77fe640 1 --2- 172.21.15.83:0/3756357634 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16a0004800 0x7f16b80c5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.257+0000 7f16b7fff640 1 --2- 172.21.15.83:0/3756357634 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16b81583b0 0x7f16b80ae0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.257+0000 7f16b6ffd640 1 -- 172.21.15.83:0/3756357634 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16a0004800 msgr2=0x7f16b80c5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.257+0000 7f16b6ffd640 1 --2- 172.21.15.83:0/3756357634 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16a0004800 0x7f16b80c5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.257+0000 7f16b6ffd640 1 -- 172.21.15.83:0/3756357634 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16b81583b0 msgr2=0x7f16b80ae0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.257+0000 7f16b6ffd640 1 --2- 172.21.15.83:0/3756357634 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16b81583b0 0x7f16b80ae0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.257+0000 7f16b6ffd640 1 -- 172.21.15.83:0/3756357634 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b80c6410 con 0x7f16b80caac0 2024-03-06T07:27:09.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.257+0000 7f16b77fe640 1 --2- 172.21.15.83:0/3756357634 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16a0004800 0x7f16b80c5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:09.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.257+0000 7f16b7fff640 1 --2- 172.21.15.83:0/3756357634 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16b81583b0 0x7f16b80ae0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:09.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.257+0000 7f16b6ffd640 1 --2- 172.21.15.83:0/3756357634 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16b80caac0 0x7f16b8155950 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f16ac00c330 tx=0x7f16ac001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:09.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.257+0000 7f169ffff640 1 -- 172.21.15.83:0/3756357634 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ac001c90 con 0x7f16b80caac0 2024-03-06T07:27:09.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.257+0000 7f169ffff640 1 -- 172.21.15.83:0/3756357634 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16ac004bf0 con 0x7f16b80caac0 2024-03-06T07:27:09.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.257+0000 7f16be0c2640 1 -- 172.21.15.83:0/3756357634 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16b80c52a0 con 0x7f16b80caac0 2024-03-06T07:27:09.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.257+0000 7f169ffff640 1 -- 172.21.15.83:0/3756357634 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ac007070 con 0x7f16b80caac0 2024-03-06T07:27:09.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.257+0000 7f16be0c2640 1 -- 172.21.15.83:0/3756357634 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16b80cb230 con 0x7f16b80caac0 2024-03-06T07:27:09.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.258+0000 7f16be0c2640 1 -- 172.21.15.83:0/3756357634 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16b80af760 con 0x7f16b80caac0 2024-03-06T07:27:09.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.259+0000 7f169ffff640 1 -- 172.21.15.83:0/3756357634 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f16ac047020 con 0x7f16b80caac0 2024-03-06T07:27:09.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.259+0000 7f169ffff640 1 --2- 172.21.15.83:0/3756357634 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f16a406b4d0 0x7f16a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.259+0000 7f169ffff640 1 -- 172.21.15.83:0/3756357634 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f16ac0ab2e0 con 0x7f16b80caac0 2024-03-06T07:27:09.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.259+0000 7f16b77fe640 1 --2- 172.21.15.83:0/3756357634 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f16a406b4d0 0x7f16a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.260+0000 7f16b77fe640 1 --2- 172.21.15.83:0/3756357634 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f16a406b4d0 0x7f16a406d990 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f16a8006fd0 tx=0x7f16a8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:09.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.264+0000 7f169ffff640 1 -- 172.21.15.83:0/3756357634 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f16b80af760 con 0x7f16b80caac0 2024-03-06T07:27:09.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:09.437+0000 7f16be0c2640 1 -- 172.21.15.83:0/3756357634 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f16b8068ff0 con 0x7f16b80caac0 2024-03-06T07:27:10.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.127+0000 7f169ffff640 1 -- 172.21.15.83:0/3756357634 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v108) v1 ==== 170+0+0 (secure 0 0 0) 0x7f16b80c6410 con 0x7f16b80caac0 2024-03-06T07:27:10.129 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'cephfs_data' created 2024-03-06T07:27:10.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.131+0000 7f16be0c2640 1 -- 172.21.15.83:0/3756357634 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f16a406b4d0 msgr2=0x7f16a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.131+0000 7f16be0c2640 1 --2- 172.21.15.83:0/3756357634 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f16a406b4d0 0x7f16a406d990 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f16a8006fd0 tx=0x7f16a8008040 comp rx=0 tx=0).stop 2024-03-06T07:27:10.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.131+0000 7f16be0c2640 1 -- 172.21.15.83:0/3756357634 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16b80caac0 msgr2=0x7f16b8155950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.131+0000 7f16be0c2640 1 --2- 172.21.15.83:0/3756357634 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16b80caac0 0x7f16b8155950 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f16ac00c330 tx=0x7f16ac001a40 comp rx=0 tx=0).stop 2024-03-06T07:27:10.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.131+0000 7f16be0c2640 1 -- 172.21.15.83:0/3756357634 shutdown_connections 2024-03-06T07:27:10.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.131+0000 7f16be0c2640 1 --2- 172.21.15.83:0/3756357634 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f16a406b4d0 0x7f16a406d990 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.131+0000 7f16be0c2640 1 --2- 172.21.15.83:0/3756357634 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16b81583b0 0x7f16b80ae0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.131+0000 7f16be0c2640 1 --2- 172.21.15.83:0/3756357634 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16b80caac0 0x7f16b8155950 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.131+0000 7f16be0c2640 1 --2- 172.21.15.83:0/3756357634 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16a0004800 0x7f16b80c5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.131+0000 7f16be0c2640 1 -- 172.21.15.83:0/3756357634 >> 172.21.15.83:0/3756357634 conn(0x7f16b80b1d70 msgr2=0x7f16b80997e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:10.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.131+0000 7f16be0c2640 1 -- 172.21.15.83:0/3756357634 shutdown_connections 2024-03-06T07:27:10.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.131+0000 7f16be0c2640 1 -- 172.21.15.83:0/3756357634 wait complete. 2024-03-06T07:27:10.158 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:27:10.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.262+0000 7ff6fc502640 1 Processor -- start 2024-03-06T07:27:10.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.262+0000 7ff6fc502640 1 -- start start 2024-03-06T07:27:10.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.263+0000 7ff6fc502640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff6f415e270 0x7ff6f40b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.263+0000 7ff6fc502640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ff6f40b5f80 0x7ff6f40b5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.263+0000 7ff6fc502640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f406b050 con 0x7ff6f4066d00 2024-03-06T07:27:10.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.263+0000 7ff6fc502640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6f415e650 con 0x7ff6f40b5f80 2024-03-06T07:27:10.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.263+0000 7ff6fc502640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f40afdf0 con 0x7ff6f415e270 2024-03-06T07:27:10.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.263+0000 7ff6f9a9d640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ff6f40b5f80 0x7ff6f40b5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.263+0000 7ff6fa29e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff6f415e270 0x7ff6f40b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.263+0000 7ff6faa9f640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7ff6f4066d00 0x7ff6f40cc9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:48168/0 (socket says 172.21.15.83:48168) 2024-03-06T07:27:10.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.263+0000 7ff6fa29e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff6f415e270 0x7ff6f40b4ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:44746/0 (socket says 172.21.15.83:44746) 2024-03-06T07:27:10.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.263+0000 7ff6f9a9d640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ff6f40b5f80 0x7ff6f40b5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:41284/0 (socket says 172.21.15.83:41284) 2024-03-06T07:27:10.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.263+0000 7ff6fa29e640 1 -- 172.21.15.83:0/1636482716 learned_addr learned my addr 172.21.15.83:0/1636482716 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:10.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.264+0000 7ff6f9a9d640 1 -- 172.21.15.83:0/1636482716 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff6f415e270 msgr2=0x7ff6f40b4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:10.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.264+0000 7ff6f9a9d640 1 --2- 172.21.15.83:0/1636482716 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff6f415e270 0x7ff6f40b4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.264+0000 7ff6f9a9d640 1 -- 172.21.15.83:0/1636482716 >> v1:172.21.15.183:6789/0 conn(0x7ff6f4066d00 legacy=0x7ff6f40cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.264+0000 7ff6f9a9d640 1 -- 172.21.15.83:0/1636482716 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f406b330 con 0x7ff6f40b5f80 2024-03-06T07:27:10.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.265+0000 7ff6f9a9d640 1 --2- 172.21.15.83:0/1636482716 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff6f40b5f80 0x7ff6f40b5410 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7ff6e40097a0 tx=0x7ff6e402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b00458c51ad1897 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:10.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.265+0000 7ff6f929c640 1 -- 172.21.15.83:0/1636482716 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e4035020 con 0x7ff6f40b5f80 2024-03-06T07:27:10.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.265+0000 7ff6f929c640 1 -- 172.21.15.83:0/1636482716 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6e4031830 con 0x7ff6f40b5f80 2024-03-06T07:27:10.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.265+0000 7ff6f929c640 1 -- 172.21.15.83:0/1636482716 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e4031c00 con 0x7ff6f40b5f80 2024-03-06T07:27:10.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.265+0000 7ff6fc502640 1 -- 172.21.15.83:0/1636482716 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff6f40b5f80 msgr2=0x7ff6f40b5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.265+0000 7ff6fc502640 1 --2- 172.21.15.83:0/1636482716 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff6f40b5f80 0x7ff6f40b5410 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7ff6e40097a0 tx=0x7ff6e402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.265+0000 7ff6fc502640 1 -- 172.21.15.83:0/1636482716 shutdown_connections 2024-03-06T07:27:10.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.265+0000 7ff6fc502640 1 --2- 172.21.15.83:0/1636482716 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff6f40b5f80 0x7ff6f40b5410 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.265+0000 7ff6fc502640 1 --2- 172.21.15.83:0/1636482716 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff6f415e270 0x7ff6f40b4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.265+0000 7ff6fc502640 1 -- 172.21.15.83:0/1636482716 >> 172.21.15.83:0/1636482716 conn(0x7ff6f40725d0 msgr2=0x7ff6f40729e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:10.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.265+0000 7ff6fc502640 1 -- 172.21.15.83:0/1636482716 shutdown_connections 2024-03-06T07:27:10.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.265+0000 7ff6fc502640 1 -- 172.21.15.83:0/1636482716 wait complete. 2024-03-06T07:27:10.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.266+0000 7ff6fc502640 1 Processor -- start 2024-03-06T07:27:10.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.267+0000 7ff6fc502640 1 -- start start 2024-03-06T07:27:10.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.267+0000 7ff6fc502640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff6f4066d00 0x7ff6f41526b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.267+0000 7ff6fc502640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6f40b5f80 0x7ff6f40c7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.267+0000 7ff6fc502640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff6f415e270 0x7ff6f40c7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.267+0000 7ff6fc502640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f40b0c70 con 0x7ff6f40b5f80 2024-03-06T07:27:10.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.267+0000 7ff6fc502640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f40b0990 con 0x7ff6f415e270 2024-03-06T07:27:10.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.267+0000 7ff6fc502640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f40b1100 con 0x7ff6f4066d00 2024-03-06T07:27:10.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.267+0000 7ff6f9a9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6f40b5f80 0x7ff6f40c7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.267+0000 7ff6faa9f640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff6f415e270 0x7ff6f40c7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.267+0000 7ff6fa29e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff6f4066d00 0x7ff6f41526b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.267+0000 7ff6fa29e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff6f4066d00 0x7ff6f41526b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:44762/0 (socket says 172.21.15.83:44762) 2024-03-06T07:27:10.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.267+0000 7ff6faa9f640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff6f415e270 0x7ff6f40c7750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:41288/0 (socket says 172.21.15.83:41288) 2024-03-06T07:27:10.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.267+0000 7ff6fa29e640 1 -- 172.21.15.83:0/143696448 learned_addr learned my addr 172.21.15.83:0/143696448 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:10.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.268+0000 7ff6f9a9d640 1 -- 172.21.15.83:0/143696448 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff6f4066d00 msgr2=0x7ff6f41526b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.268+0000 7ff6f9a9d640 1 --2- 172.21.15.83:0/143696448 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff6f4066d00 0x7ff6f41526b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.268+0000 7ff6f9a9d640 1 -- 172.21.15.83:0/143696448 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff6f415e270 msgr2=0x7ff6f40c7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.268+0000 7ff6f9a9d640 1 --2- 172.21.15.83:0/143696448 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff6f415e270 0x7ff6f40c7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.268+0000 7ff6f9a9d640 1 -- 172.21.15.83:0/143696448 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f4151610 con 0x7ff6f40b5f80 2024-03-06T07:27:10.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.268+0000 7ff6fa29e640 1 --2- 172.21.15.83:0/143696448 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff6f4066d00 0x7ff6f41526b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:10.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.268+0000 7ff6faa9f640 1 --2- 172.21.15.83:0/143696448 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff6f415e270 0x7ff6f40c7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:10.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.268+0000 7ff6f9a9d640 1 --2- 172.21.15.83:0/143696448 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6f40b5f80 0x7ff6f40c7210 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7ff6e4033000 tx=0x7ff6e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:10.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.268+0000 7ff6eaffd640 1 -- 172.21.15.83:0/143696448 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e4035040 con 0x7ff6f40b5f80 2024-03-06T07:27:10.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.268+0000 7ff6eaffd640 1 -- 172.21.15.83:0/143696448 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6e403b920 con 0x7ff6f40b5f80 2024-03-06T07:27:10.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.268+0000 7ff6eaffd640 1 -- 172.21.15.83:0/143696448 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e403bd10 con 0x7ff6f40b5f80 2024-03-06T07:27:10.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.268+0000 7ff6fc502640 1 -- 172.21.15.83:0/143696448 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6f41518f0 con 0x7ff6f40b5f80 2024-03-06T07:27:10.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.268+0000 7ff6fc502640 1 -- 172.21.15.83:0/143696448 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f4151eb0 con 0x7ff6f40b5f80 2024-03-06T07:27:10.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.269+0000 7ff6fc502640 1 -- 172.21.15.83:0/143696448 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6f40ad1e0 con 0x7ff6f40b5f80 2024-03-06T07:27:10.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.270+0000 7ff6eaffd640 1 -- 172.21.15.83:0/143696448 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff6e4038020 con 0x7ff6f40b5f80 2024-03-06T07:27:10.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.270+0000 7ff6eaffd640 1 --2- 172.21.15.83:0/143696448 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff6d806b4d0 0x7ff6d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.270+0000 7ff6eaffd640 1 -- 172.21.15.83:0/143696448 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5609+0+0 (secure 0 0 0) 0x7ff6e40ac180 con 0x7ff6f40b5f80 2024-03-06T07:27:10.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.270+0000 7ff6fa29e640 1 --2- 172.21.15.83:0/143696448 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff6d806b4d0 0x7ff6d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.270+0000 7ff6fa29e640 1 --2- 172.21.15.83:0/143696448 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff6d806b4d0 0x7ff6d806d990 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7ff6f40d10b0 tx=0x7ff6f0009240 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:10.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.275+0000 7ff6eaffd640 1 -- 172.21.15.83:0/143696448 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff6f40ad1e0 con 0x7ff6f40b5f80 2024-03-06T07:27:10.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:10.489+0000 7ff6fc502640 1 -- 172.21.15.83:0/143696448 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7ff6f40729b0 con 0x7ff6f40b5f80 2024-03-06T07:27:11.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.137+0000 7ff6eaffd640 1 -- 172.21.15.83:0/143696448 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 16 and data pool 17 v57) v1 ==== 174+0+0 (secure 0 0 0) 0x7ff6f40729b0 con 0x7ff6f40b5f80 2024-03-06T07:27:11.139 INFO:teuthology.orchestra.run.smithi083.stderr:new fs with metadata pool 16 and data pool 17 2024-03-06T07:27:11.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.141+0000 7ff6fc502640 1 -- 172.21.15.83:0/143696448 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff6d806b4d0 msgr2=0x7ff6d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.141+0000 7ff6fc502640 1 --2- 172.21.15.83:0/143696448 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff6d806b4d0 0x7ff6d806d990 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7ff6f40d10b0 tx=0x7ff6f0009240 comp rx=0 tx=0).stop 2024-03-06T07:27:11.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.142+0000 7ff6fc502640 1 -- 172.21.15.83:0/143696448 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6f40b5f80 msgr2=0x7ff6f40c7210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.142+0000 7ff6fc502640 1 --2- 172.21.15.83:0/143696448 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6f40b5f80 0x7ff6f40c7210 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7ff6e4033000 tx=0x7ff6e40046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.145+0000 7ff6fc502640 1 -- 172.21.15.83:0/143696448 shutdown_connections 2024-03-06T07:27:11.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.145+0000 7ff6fc502640 1 --2- 172.21.15.83:0/143696448 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff6d806b4d0 0x7ff6d806d990 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.145+0000 7ff6fc502640 1 --2- 172.21.15.83:0/143696448 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff6f415e270 0x7ff6f40c7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.145+0000 7ff6fc502640 1 --2- 172.21.15.83:0/143696448 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6f40b5f80 0x7ff6f40c7210 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.145+0000 7ff6fc502640 1 --2- 172.21.15.83:0/143696448 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff6f4066d00 0x7ff6f41526b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.145+0000 7ff6fc502640 1 -- 172.21.15.83:0/143696448 >> 172.21.15.83:0/143696448 conn(0x7ff6f40725d0 msgr2=0x7ff6f40ad6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:11.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.152+0000 7ff6fc502640 1 -- 172.21.15.83:0/143696448 shutdown_connections 2024-03-06T07:27:11.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.154+0000 7ff6fc502640 1 -- 172.21.15.83:0/143696448 wait complete. 2024-03-06T07:27:11.178 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:11.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.279+0000 7fe85f66b640 1 Processor -- start 2024-03-06T07:27:11.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.279+0000 7fe85f66b640 1 -- start start 2024-03-06T07:27:11.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.280+0000 7fe85f66b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe85815e090 0x7fe8580d47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.280+0000 7fe85f66b640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fe8580d0650 0x7fe8580d0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.280+0000 7fe85f66b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe85815e470 con 0x7fe8580d2920 2024-03-06T07:27:11.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.280+0000 7fe85f66b640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8580d0fe0 con 0x7fe85815e090 2024-03-06T07:27:11.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.280+0000 7fe85f66b640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8580d18f0 con 0x7fe8580d0650 2024-03-06T07:27:11.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.280+0000 7fe85d407640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe85815e090 0x7fe8580d47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.280+0000 7fe85d407640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe85815e090 0x7fe8580d47a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47576/0 (socket says 172.21.15.83:47576) 2024-03-06T07:27:11.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.280+0000 7fe85d407640 1 -- 172.21.15.83:0/3442401990 learned_addr learned my addr 172.21.15.83:0/3442401990 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:11.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.281+0000 7fe85d407640 1 -- 172.21.15.83:0/3442401990 >> v2:172.21.15.183:3300/0 conn(0x7fe8580d0650 msgr2=0x7fe8580d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:11.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.281+0000 7fe85d407640 1 --2- 172.21.15.83:0/3442401990 >> v2:172.21.15.183:3300/0 conn(0x7fe8580d0650 0x7fe8580d0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.281+0000 7fe85d407640 1 -- 172.21.15.83:0/3442401990 >> v1:172.21.15.183:6789/0 conn(0x7fe8580d2920 legacy=0x7fe8580b35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:11.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.281+0000 7fe85d407640 1 -- 172.21.15.83:0/3442401990 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8580b3cf0 con 0x7fe85815e090 2024-03-06T07:27:11.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.281+0000 7fe85d407640 1 --2- 172.21.15.83:0/3442401990 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe85815e090 0x7fe8580d47a0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fe85808c550 tx=0x7fe85402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3ed160cf4d46038 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:11.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.281+0000 7fe84ffff640 1 -- 172.21.15.83:0/3442401990 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe85402ca80 con 0x7fe85815e090 2024-03-06T07:27:11.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.281+0000 7fe84ffff640 1 -- 172.21.15.83:0/3442401990 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe85402cc20 con 0x7fe85815e090 2024-03-06T07:27:11.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.281+0000 7fe84ffff640 1 -- 172.21.15.83:0/3442401990 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8540343f0 con 0x7fe85815e090 2024-03-06T07:27:11.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.282+0000 7fe85f66b640 1 -- 172.21.15.83:0/3442401990 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe85815e090 msgr2=0x7fe8580d47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.282+0000 7fe85f66b640 1 --2- 172.21.15.83:0/3442401990 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe85815e090 0x7fe8580d47a0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fe85808c550 tx=0x7fe85402bee0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.282+0000 7fe85f66b640 1 -- 172.21.15.83:0/3442401990 shutdown_connections 2024-03-06T07:27:11.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.282+0000 7fe85f66b640 1 --2- 172.21.15.83:0/3442401990 >> v2:172.21.15.183:3300/0 conn(0x7fe8580d0650 0x7fe8580d0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.282+0000 7fe85f66b640 1 --2- 172.21.15.83:0/3442401990 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe85815e090 0x7fe8580d47a0 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.282+0000 7fe85f66b640 1 -- 172.21.15.83:0/3442401990 >> 172.21.15.83:0/3442401990 conn(0x7fe8580d3c80 msgr2=0x7fe8580d4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:11.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.282+0000 7fe85f66b640 1 -- 172.21.15.83:0/3442401990 shutdown_connections 2024-03-06T07:27:11.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.282+0000 7fe85f66b640 1 -- 172.21.15.83:0/3442401990 wait complete. 2024-03-06T07:27:11.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.283+0000 7fe85f66b640 1 Processor -- start 2024-03-06T07:27:11.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.283+0000 7fe85f66b640 1 -- start start 2024-03-06T07:27:11.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.284+0000 7fe85f66b640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe8580d0650 0x7fe858156a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.284+0000 7fe85f66b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe8580d2920 0x7fe8581e92f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.284+0000 7fe85f66b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe85815e090 0x7fe8581e9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.284+0000 7fe85f66b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8580cee30 con 0x7fe85815e090 2024-03-06T07:27:11.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.284+0000 7fe85f66b640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8580ceb50 con 0x7fe8580d0650 2024-03-06T07:27:11.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.284+0000 7fe85f66b640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8580b2df0 con 0x7fe8580d2920 2024-03-06T07:27:11.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.284+0000 7fe85dc08640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe85815e090 0x7fe8581e9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.284+0000 7fe85dc08640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe85815e090 0x7fe8581e9830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47586/0 (socket says 172.21.15.83:47586) 2024-03-06T07:27:11.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.284+0000 7fe85dc08640 1 -- 172.21.15.83:0/1003121842 learned_addr learned my addr 172.21.15.83:0/1003121842 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:11.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.284+0000 7fe85d407640 1 --2- 172.21.15.83:0/1003121842 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe8580d0650 0x7fe858156a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.284+0000 7fe85cc06640 1 --2- 172.21.15.83:0/1003121842 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe8580d2920 0x7fe8581e92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.284+0000 7fe85dc08640 1 -- 172.21.15.83:0/1003121842 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe8580d2920 msgr2=0x7fe8581e92f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.284+0000 7fe85dc08640 1 --2- 172.21.15.83:0/1003121842 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe8580d2920 0x7fe8581e92f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.284+0000 7fe85dc08640 1 -- 172.21.15.83:0/1003121842 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe8580d0650 msgr2=0x7fe858156a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.284+0000 7fe85dc08640 1 --2- 172.21.15.83:0/1003121842 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe8580d0650 0x7fe858156a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.284+0000 7fe85dc08640 1 -- 172.21.15.83:0/1003121842 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8581e9e60 con 0x7fe85815e090 2024-03-06T07:27:11.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.284+0000 7fe85d407640 1 --2- 172.21.15.83:0/1003121842 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe8580d0650 0x7fe858156a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:11.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.285+0000 7fe85dc08640 1 --2- 172.21.15.83:0/1003121842 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe85815e090 0x7fe8581e9830 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7fe85000b440 tx=0x7fe85000b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:11.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.285+0000 7fe84dffb640 1 -- 172.21.15.83:0/1003121842 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe850015020 con 0x7fe85815e090 2024-03-06T07:27:11.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.285+0000 7fe84dffb640 1 -- 172.21.15.83:0/1003121842 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe850011830 con 0x7fe85815e090 2024-03-06T07:27:11.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.285+0000 7fe85f66b640 1 -- 172.21.15.83:0/1003121842 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe85815f850 con 0x7fe85815e090 2024-03-06T07:27:11.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.285+0000 7fe84dffb640 1 -- 172.21.15.83:0/1003121842 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe850011c20 con 0x7fe85815e090 2024-03-06T07:27:11.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.285+0000 7fe85f66b640 1 -- 172.21.15.83:0/1003121842 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8581574b0 con 0x7fe85815e090 2024-03-06T07:27:11.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.286+0000 7fe85f66b640 1 -- 172.21.15.83:0/1003121842 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe85804f330 con 0x7fe85815e090 2024-03-06T07:27:11.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.286+0000 7fe84dffb640 1 -- 172.21.15.83:0/1003121842 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe850018020 con 0x7fe85815e090 2024-03-06T07:27:11.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.287+0000 7fe84dffb640 1 --2- 172.21.15.83:0/1003121842 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe83c06b4d0 0x7fe83c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.287+0000 7fe84dffb640 1 -- 172.21.15.83:0/1003121842 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe85008a110 con 0x7fe85815e090 2024-03-06T07:27:11.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.287+0000 7fe85d407640 1 --2- 172.21.15.83:0/1003121842 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe83c06b4d0 0x7fe83c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.287+0000 7fe85d407640 1 --2- 172.21.15.83:0/1003121842 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe83c06b4d0 0x7fe83c06d990 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fe854035000 tx=0x7fe854030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:11.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.291+0000 7fe84dffb640 1 -- 172.21.15.83:0/1003121842 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe85001f020 con 0x7fe85815e090 2024-03-06T07:27:11.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.464+0000 7fe85f66b640 1 -- 172.21.15.83:0/1003121842 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8580d1230 con 0x7fe85815e090 2024-03-06T07:27:11.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.465+0000 7fe84dffb640 1 -- 172.21.15.83:0/1003121842 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+16652 (secure 0 0 0) 0x7fe8500172c0 con 0x7fe85815e090 2024-03-06T07:27:11.467 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:11.467 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":109,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:27:10.490624+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:08.709967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:09.456518+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:11.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.467+0000 7fe85f66b640 1 -- 172.21.15.83:0/1003121842 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe83c06b4d0 msgr2=0x7fe83c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.467+0000 7fe85f66b640 1 --2- 172.21.15.83:0/1003121842 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe83c06b4d0 0x7fe83c06d990 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fe854035000 tx=0x7fe854030000 comp rx=0 tx=0).stop 2024-03-06T07:27:11.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.467+0000 7fe85f66b640 1 -- 172.21.15.83:0/1003121842 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe85815e090 msgr2=0x7fe8581e9830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.467+0000 7fe85f66b640 1 --2- 172.21.15.83:0/1003121842 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe85815e090 0x7fe8581e9830 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7fe85000b440 tx=0x7fe85000b900 comp rx=0 tx=0).stop 2024-03-06T07:27:11.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.467+0000 7fe85f66b640 1 -- 172.21.15.83:0/1003121842 shutdown_connections 2024-03-06T07:27:11.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.467+0000 7fe85f66b640 1 --2- 172.21.15.83:0/1003121842 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe83c06b4d0 0x7fe83c06d990 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.467+0000 7fe85f66b640 1 --2- 172.21.15.83:0/1003121842 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe85815e090 0x7fe8581e9830 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.467+0000 7fe85f66b640 1 --2- 172.21.15.83:0/1003121842 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe8580d2920 0x7fe8581e92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.467+0000 7fe85f66b640 1 --2- 172.21.15.83:0/1003121842 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe8580d0650 0x7fe858156a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.467+0000 7fe85f66b640 1 -- 172.21.15.83:0/1003121842 >> 172.21.15.83:0/1003121842 conn(0x7fe8580d3c80 msgr2=0x7fe8580cf1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:11.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.467+0000 7fe85f66b640 1 -- 172.21.15.83:0/1003121842 shutdown_connections 2024-03-06T07:27:11.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.467+0000 7fe85f66b640 1 -- 172.21.15.83:0/1003121842 wait complete. 2024-03-06T07:27:11.481 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:11.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.579+0000 7fea790c3640 1 Processor -- start 2024-03-06T07:27:11.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.579+0000 7fea790c3640 1 -- start start 2024-03-06T07:27:11.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.580+0000 7fea790c3640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fea740d00c0 0x7fea740d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.580+0000 7fea790c3640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea740d09e0 con 0x7fea7406cf20 2024-03-06T07:27:11.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.580+0000 7fea790c3640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea7415e380 con 0x7fea740cf850 2024-03-06T07:27:11.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.580+0000 7fea790c3640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea7406ca90 con 0x7fea740d00c0 2024-03-06T07:27:11.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.580+0000 7fea7359e640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fea7406cf20 0x7fea7406d3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:37606/0 (socket says 172.21.15.83:37606) 2024-03-06T07:27:11.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.580+0000 7fea7359e640 1 -- 172.21.15.83:0/2393839969 learned_addr learned my addr 172.21.15.83:0/2393839969 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:27:11.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.580+0000 7fea7259c640 1 -- 172.21.15.83:0/2393839969 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2268495760 0 0) 0x7fea740d09e0 con 0x7fea7406cf20 2024-03-06T07:27:11.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.580+0000 7fea7259c640 1 -- 172.21.15.83:0/2393839969 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea60003520 con 0x7fea7406cf20 2024-03-06T07:27:11.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.580+0000 7fea72d9d640 1 --2- 172.21.15.83:0/2393839969 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fea740d00c0 0x7fea740d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.580+0000 7fea7259c640 1 -- 172.21.15.83:0/2393839969 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2274932235 0 0) 0x7fea60003520 con 0x7fea7406cf20 2024-03-06T07:27:11.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.580+0000 7fea7259c640 1 -- 172.21.15.83:0/2393839969 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fea740d00c0 msgr2=0x7fea740d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:11.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.580+0000 7fea7259c640 1 --2- 172.21.15.83:0/2393839969 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fea740d00c0 0x7fea740d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.580+0000 7fea7259c640 1 -- 172.21.15.83:0/2393839969 >> v1:172.21.15.183:6789/0 conn(0x7fea740cf850 legacy=0x7fea740cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.580+0000 7fea7259c640 1 -- 172.21.15.83:0/2393839969 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea740cdc10 con 0x7fea7406cf20 2024-03-06T07:27:11.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.580+0000 7fea7259c640 1 -- 172.21.15.83:0/2393839969 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7fea6c0019f0 con 0x7fea7406cf20 2024-03-06T07:27:11.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.580+0000 7fea72d9d640 1 --2- 172.21.15.83:0/2393839969 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fea740d00c0 0x7fea740d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:11.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.580+0000 7fea7259c640 1 -- 172.21.15.83:0/2393839969 >> v1:172.21.15.83:6789/0 conn(0x7fea7406cf20 legacy=0x7fea7406d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.580+0000 7fea7259c640 1 --2- 172.21.15.83:0/2393839969 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fea600039e0 0x7fea60003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.580+0000 7fea7259c640 1 --2- 172.21.15.83:0/2393839969 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea60004800 0x7fea60004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.581+0000 7fea7259c640 1 --2- 172.21.15.83:0/2393839969 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fea740d00c0 0x7fea600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.581+0000 7fea7259c640 1 -- 172.21.15.83:0/2393839969 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea7406ca90 con 0x7fea60004800 2024-03-06T07:27:11.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.581+0000 7fea7259c640 1 -- 172.21.15.83:0/2393839969 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6000bac0 con 0x7fea600039e0 2024-03-06T07:27:11.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.581+0000 7fea7259c640 1 -- 172.21.15.83:0/2393839969 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea6000bcc0 con 0x7fea740d00c0 2024-03-06T07:27:11.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.581+0000 7fea6bfff640 1 --2- 172.21.15.83:0/2393839969 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea60004800 0x7fea60004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.581+0000 7fea72d9d640 1 --2- 172.21.15.83:0/2393839969 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fea600039e0 0x7fea60003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.581+0000 7fea7359e640 1 --2- 172.21.15.83:0/2393839969 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fea740d00c0 0x7fea600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.581+0000 7fea6bfff640 1 -- 172.21.15.83:0/2393839969 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fea740d00c0 msgr2=0x7fea600051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:11.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.581+0000 7fea6bfff640 1 --2- 172.21.15.83:0/2393839969 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fea740d00c0 0x7fea600051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.581+0000 7fea6bfff640 1 -- 172.21.15.83:0/2393839969 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fea600039e0 msgr2=0x7fea60003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:11.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.581+0000 7fea6bfff640 1 --2- 172.21.15.83:0/2393839969 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fea600039e0 0x7fea60003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.581+0000 7fea6bfff640 1 -- 172.21.15.83:0/2393839969 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea60003520 con 0x7fea60004800 2024-03-06T07:27:11.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.582+0000 7fea6bfff640 1 --2- 172.21.15.83:0/2393839969 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea60004800 0x7fea60004c80 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fea5c001750 tx=0x7fea60005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54119ba1cfbc3fa9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:11.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.582+0000 7fea7259c640 1 -- 172.21.15.83:0/2393839969 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5c001370 con 0x7fea60004800 2024-03-06T07:27:11.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.582+0000 7fea7259c640 1 -- 172.21.15.83:0/2393839969 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea5c0052d0 con 0x7fea60004800 2024-03-06T07:27:11.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.582+0000 7fea7259c640 1 -- 172.21.15.83:0/2393839969 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5c003830 con 0x7fea60004800 2024-03-06T07:27:11.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.582+0000 7fea790c3640 1 -- 172.21.15.83:0/2393839969 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea60004800 msgr2=0x7fea60004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.582+0000 7fea790c3640 1 --2- 172.21.15.83:0/2393839969 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea60004800 0x7fea60004c80 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fea5c001750 tx=0x7fea60005830 comp rx=0 tx=0).stop 2024-03-06T07:27:11.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.582+0000 7fea7359e640 1 -- 172.21.15.83:0/2393839969 reap_dead start 2024-03-06T07:27:11.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.582+0000 7fea790c3640 1 -- 172.21.15.83:0/2393839969 shutdown_connections 2024-03-06T07:27:11.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.582+0000 7fea790c3640 1 -- 172.21.15.83:0/2393839969 >> 172.21.15.83:0/2393839969 conn(0x7fea740b35a0 msgr2=0x7fea740b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:11.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.582+0000 7fea790c3640 1 -- 172.21.15.83:0/2393839969 shutdown_connections 2024-03-06T07:27:11.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.582+0000 7fea790c3640 1 -- 172.21.15.83:0/2393839969 wait complete. 2024-03-06T07:27:11.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.584+0000 7fea790c3640 1 Processor -- start 2024-03-06T07:27:11.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.584+0000 7fea790c3640 1 -- start start 2024-03-06T07:27:11.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.584+0000 7fea790c3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea60004800 0x7fea740c9df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.584+0000 7fea790c3640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fea740b4690 0x7fea740cb570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.584+0000 7fea790c3640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fea7414ee70 0x7fea740c6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.584+0000 7fea790c3640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea7406c720 con 0x7fea60004800 2024-03-06T07:27:11.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.584+0000 7fea790c3640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea7406c440 con 0x7fea740b4690 2024-03-06T07:27:11.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.584+0000 7fea790c3640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea740b1030 con 0x7fea7414ee70 2024-03-06T07:27:11.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.584+0000 7fea72d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea60004800 0x7fea740c9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.584+0000 7fea72d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea60004800 0x7fea740c9df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47602/0 (socket says 172.21.15.83:47602) 2024-03-06T07:27:11.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.584+0000 7fea72d9d640 1 -- 172.21.15.83:0/54606930 learned_addr learned my addr 172.21.15.83:0/54606930 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:11.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.584+0000 7fea6bfff640 1 --2- 172.21.15.83:0/54606930 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fea740b4690 0x7fea740cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.584+0000 7fea7359e640 1 --2- 172.21.15.83:0/54606930 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fea7414ee70 0x7fea740c6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.585+0000 7fea72d9d640 1 -- 172.21.15.83:0/54606930 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fea7414ee70 msgr2=0x7fea740c6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.585+0000 7fea72d9d640 1 --2- 172.21.15.83:0/54606930 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fea7414ee70 0x7fea740c6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.585+0000 7fea72d9d640 1 -- 172.21.15.83:0/54606930 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fea740b4690 msgr2=0x7fea740cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.585+0000 7fea72d9d640 1 --2- 172.21.15.83:0/54606930 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fea740b4690 0x7fea740cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.585+0000 7fea72d9d640 1 -- 172.21.15.83:0/54606930 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea740cbab0 con 0x7fea60004800 2024-03-06T07:27:11.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.585+0000 7fea72d9d640 1 --2- 172.21.15.83:0/54606930 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea60004800 0x7fea740c9df0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fea64010860 tx=0x7fea64010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:11.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.585+0000 7fea6b7fe640 1 -- 172.21.15.83:0/54606930 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea6401a070 con 0x7fea60004800 2024-03-06T07:27:11.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.585+0000 7fea6b7fe640 1 -- 172.21.15.83:0/54606930 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea6400cb40 con 0x7fea60004800 2024-03-06T07:27:11.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.585+0000 7fea790c3640 1 -- 172.21.15.83:0/54606930 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea740ca4d0 con 0x7fea60004800 2024-03-06T07:27:11.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.585+0000 7fea6b7fe640 1 -- 172.21.15.83:0/54606930 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea6400f040 con 0x7fea60004800 2024-03-06T07:27:11.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.585+0000 7fea790c3640 1 -- 172.21.15.83:0/54606930 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea740caa90 con 0x7fea60004800 2024-03-06T07:27:11.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.586+0000 7fea790c3640 1 -- 172.21.15.83:0/54606930 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea7404f330 con 0x7fea60004800 2024-03-06T07:27:11.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.586+0000 7fea6b7fe640 1 -- 172.21.15.83:0/54606930 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fea64015470 con 0x7fea60004800 2024-03-06T07:27:11.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.587+0000 7fea6b7fe640 1 --2- 172.21.15.83:0/54606930 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fea5406b4d0 0x7fea5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.587+0000 7fea6b7fe640 1 -- 172.21.15.83:0/54606930 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fea6408a770 con 0x7fea60004800 2024-03-06T07:27:11.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.587+0000 7fea6bfff640 1 --2- 172.21.15.83:0/54606930 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fea5406b4d0 0x7fea5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.587+0000 7fea6bfff640 1 --2- 172.21.15.83:0/54606930 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fea5406b4d0 0x7fea5406d990 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fea5c033000 tx=0x7fea5c009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:11.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.591+0000 7fea6b7fe640 1 -- 172.21.15.83:0/54606930 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fea740cbab0 con 0x7fea60004800 2024-03-06T07:27:11.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.762+0000 7fea790c3640 1 -- 172.21.15.83:0/54606930 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea740d0540 con 0x7fea60004800 2024-03-06T07:27:11.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.763+0000 7fea6b7fe640 1 -- 172.21.15.83:0/54606930 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+16652 (secure 0 0 0) 0x7fea64059680 con 0x7fea60004800 2024-03-06T07:27:11.766 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:11.766 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":109,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:27:10.490624+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:08.709967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:09.456518+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:11.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.765+0000 7fea790c3640 1 -- 172.21.15.83:0/54606930 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fea5406b4d0 msgr2=0x7fea5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.765+0000 7fea790c3640 1 --2- 172.21.15.83:0/54606930 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fea5406b4d0 0x7fea5406d990 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fea5c033000 tx=0x7fea5c009030 comp rx=0 tx=0).stop 2024-03-06T07:27:11.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.765+0000 7fea790c3640 1 -- 172.21.15.83:0/54606930 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea60004800 msgr2=0x7fea740c9df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.765+0000 7fea790c3640 1 --2- 172.21.15.83:0/54606930 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea60004800 0x7fea740c9df0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fea64010860 tx=0x7fea64010d20 comp rx=0 tx=0).stop 2024-03-06T07:27:11.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.765+0000 7fea790c3640 1 -- 172.21.15.83:0/54606930 shutdown_connections 2024-03-06T07:27:11.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.765+0000 7fea790c3640 1 --2- 172.21.15.83:0/54606930 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fea5406b4d0 0x7fea5406d990 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.765+0000 7fea790c3640 1 --2- 172.21.15.83:0/54606930 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fea7414ee70 0x7fea740c6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.765+0000 7fea790c3640 1 --2- 172.21.15.83:0/54606930 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fea740b4690 0x7fea740cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.765+0000 7fea790c3640 1 --2- 172.21.15.83:0/54606930 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea60004800 0x7fea740c9df0 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.765+0000 7fea790c3640 1 -- 172.21.15.83:0/54606930 >> 172.21.15.83:0/54606930 conn(0x7fea740b35a0 msgr2=0x7fea7406fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:11.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.765+0000 7fea790c3640 1 -- 172.21.15.83:0/54606930 shutdown_connections 2024-03-06T07:27:11.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.765+0000 7fea790c3640 1 -- 172.21.15.83:0/54606930 wait complete. 2024-03-06T07:27:11.778 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:27:11.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.880+0000 7f174e878640 1 Processor -- start 2024-03-06T07:27:11.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.880+0000 7f174e878640 1 -- start start 2024-03-06T07:27:11.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.881+0000 7f174e878640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f174815bbb0 0x7f174806afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.881+0000 7f174e878640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f174806b4f0 0x7f174815e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.881+0000 7f174e878640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f174815ec40 0x7f17480cd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.881+0000 7f174e878640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1748161970 con 0x7f174815bbb0 2024-03-06T07:27:11.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.881+0000 7f174e878640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17481624f0 con 0x7f174806b4f0 2024-03-06T07:27:11.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.881+0000 7f174e878640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17480cea90 con 0x7f174815ec40 2024-03-06T07:27:11.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.881+0000 7f1747fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f174815bbb0 0x7f174806afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.881+0000 7f1747fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f174815bbb0 0x7f174806afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47610/0 (socket says 172.21.15.83:47610) 2024-03-06T07:27:11.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.881+0000 7f1747fff640 1 -- 172.21.15.83:0/1869234079 learned_addr learned my addr 172.21.15.83:0/1869234079 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:11.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.881+0000 7f1747fff640 1 -- 172.21.15.83:0/1869234079 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f174815ec40 msgr2=0x7f17480cd3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:11.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.881+0000 7f1747fff640 1 --2- 172.21.15.83:0/1869234079 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f174815ec40 0x7f17480cd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.881+0000 7f1747fff640 1 -- 172.21.15.83:0/1869234079 >> v2:172.21.15.183:3300/0 conn(0x7f174806b4f0 msgr2=0x7f174815e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:11.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.881+0000 7f17477fe640 1 --2- 172.21.15.83:0/1869234079 >> v2:172.21.15.183:3300/0 conn(0x7f174806b4f0 0x7f174815e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.881+0000 7f1747fff640 1 --2- 172.21.15.83:0/1869234079 >> v2:172.21.15.183:3300/0 conn(0x7f174806b4f0 0x7f174815e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.881+0000 7f1747fff640 1 -- 172.21.15.83:0/1869234079 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1748161bc0 con 0x7f174815bbb0 2024-03-06T07:27:11.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.881+0000 7f17477fe640 1 --2- 172.21.15.83:0/1869234079 >> v2:172.21.15.183:3300/0 conn(0x7f174806b4f0 0x7f174815e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:11.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.882+0000 7f1747fff640 1 --2- 172.21.15.83:0/1869234079 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f174815bbb0 0x7f174806afb0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f174806dcc0 tx=0x7f173802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e50f0719e6fdd6f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:11.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.882+0000 7f1746ffd640 1 -- 172.21.15.83:0/1869234079 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1738035020 con 0x7f174815bbb0 2024-03-06T07:27:11.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.882+0000 7f1746ffd640 1 -- 172.21.15.83:0/1869234079 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1738031870 con 0x7f174815bbb0 2024-03-06T07:27:11.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.882+0000 7f1746ffd640 1 -- 172.21.15.83:0/1869234079 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1738031c40 con 0x7f174815bbb0 2024-03-06T07:27:11.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.883+0000 7f174e878640 1 -- 172.21.15.83:0/1869234079 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f174815bbb0 msgr2=0x7f174806afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.883+0000 7f174e878640 1 --2- 172.21.15.83:0/1869234079 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f174815bbb0 0x7f174806afb0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f174806dcc0 tx=0x7f173802b910 comp rx=0 tx=0).stop 2024-03-06T07:27:11.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.883+0000 7f174e878640 1 -- 172.21.15.83:0/1869234079 shutdown_connections 2024-03-06T07:27:11.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.883+0000 7f174e878640 1 --2- 172.21.15.83:0/1869234079 >> v2:172.21.15.183:3300/0 conn(0x7f174806b4f0 0x7f174815e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.883+0000 7f174e878640 1 --2- 172.21.15.83:0/1869234079 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f174815ec40 0x7f17480cd3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.883+0000 7f174e878640 1 --2- 172.21.15.83:0/1869234079 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f174815bbb0 0x7f174806afb0 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.883+0000 7f174e878640 1 -- 172.21.15.83:0/1869234079 >> 172.21.15.83:0/1869234079 conn(0x7f17480b2660 msgr2=0x7f17480b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:11.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.883+0000 7f174e878640 1 -- 172.21.15.83:0/1869234079 shutdown_connections 2024-03-06T07:27:11.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.883+0000 7f174e878640 1 -- 172.21.15.83:0/1869234079 wait complete. 2024-03-06T07:27:11.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.884+0000 7f174e878640 1 Processor -- start 2024-03-06T07:27:11.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.884+0000 7f174e878640 1 -- start start 2024-03-06T07:27:11.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.885+0000 7f174e878640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f174806b4f0 0x7f174814e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.885+0000 7f174e878640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f174815ec40 0x7f1748153f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.885+0000 7f174e878640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f174814ea80 0x7f17480b05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.885+0000 7f174e878640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f174815f0d0 con 0x7f174806b4f0 2024-03-06T07:27:11.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.885+0000 7f174e878640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1748162180 con 0x7f174815ec40 2024-03-06T07:27:11.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.885+0000 7f174e878640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17480cd8e0 con 0x7f174814ea80 2024-03-06T07:27:11.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.885+0000 7f1747fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f174806b4f0 0x7f174814e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.885+0000 7f1747fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f174806b4f0 0x7f174814e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47616/0 (socket says 172.21.15.83:47616) 2024-03-06T07:27:11.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.885+0000 7f1747fff640 1 -- 172.21.15.83:0/4267263172 learned_addr learned my addr 172.21.15.83:0/4267263172 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:11.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.885+0000 7f17477fe640 1 --2- 172.21.15.83:0/4267263172 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f174815ec40 0x7f1748153f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.885+0000 7f174ce15640 1 --2- 172.21.15.83:0/4267263172 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f174814ea80 0x7f17480b05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.885+0000 7f1747fff640 1 -- 172.21.15.83:0/4267263172 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f174814ea80 msgr2=0x7f17480b05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.885+0000 7f1747fff640 1 --2- 172.21.15.83:0/4267263172 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f174814ea80 0x7f17480b05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.885+0000 7f1747fff640 1 -- 172.21.15.83:0/4267263172 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f174815ec40 msgr2=0x7f1748153f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.885+0000 7f1747fff640 1 --2- 172.21.15.83:0/4267263172 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f174815ec40 0x7f1748153f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.885+0000 7f1747fff640 1 -- 172.21.15.83:0/4267263172 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1748160730 con 0x7f174806b4f0 2024-03-06T07:27:11.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.885+0000 7f1747fff640 1 --2- 172.21.15.83:0/4267263172 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f174806b4f0 0x7f174814e400 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f1738000c00 tx=0x7f17380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:11.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.886+0000 7f1744ff9640 1 -- 172.21.15.83:0/4267263172 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1738035040 con 0x7f174806b4f0 2024-03-06T07:27:11.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.886+0000 7f1744ff9640 1 -- 172.21.15.83:0/4267263172 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1738045480 con 0x7f174806b4f0 2024-03-06T07:27:11.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.886+0000 7f174e878640 1 -- 172.21.15.83:0/4267263172 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1748160a10 con 0x7f174806b4f0 2024-03-06T07:27:11.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.886+0000 7f1744ff9640 1 -- 172.21.15.83:0/4267263172 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1738045870 con 0x7f174806b4f0 2024-03-06T07:27:11.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.886+0000 7f174e878640 1 -- 172.21.15.83:0/4267263172 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1748160fd0 con 0x7f174806b4f0 2024-03-06T07:27:11.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.887+0000 7f174e878640 1 -- 172.21.15.83:0/4267263172 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1748161300 con 0x7f174806b4f0 2024-03-06T07:27:11.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.887+0000 7f1744ff9640 1 -- 172.21.15.83:0/4267263172 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1738038020 con 0x7f174806b4f0 2024-03-06T07:27:11.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.887+0000 7f1744ff9640 1 --2- 172.21.15.83:0/4267263172 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f173406b4d0 0x7f173406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.887+0000 7f1744ff9640 1 -- 172.21.15.83:0/4267263172 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f17380ac610 con 0x7f174806b4f0 2024-03-06T07:27:11.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.887+0000 7f17477fe640 1 --2- 172.21.15.83:0/4267263172 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f173406b4d0 0x7f173406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.888+0000 7f17477fe640 1 --2- 172.21.15.83:0/4267263172 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f173406b4d0 0x7f173406d990 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f1748072f10 tx=0x7f173c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:11.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:11.892+0000 7f1744ff9640 1 -- 172.21.15.83:0/4267263172 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f173807b520 con 0x7f174806b4f0 2024-03-06T07:27:12.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.099+0000 7f174e878640 1 -- 172.21.15.83:0/4267263172 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f17480733b0 con 0x7f174806b4f0 2024-03-06T07:27:12.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.146+0000 7f1744ff9640 1 -- 172.21.15.83:0/4267263172 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v58) v1 ==== 118+0+0 (secure 0 0 0) 0x7f173807ae20 con 0x7f174806b4f0 2024-03-06T07:27:12.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.148+0000 7f174e878640 1 -- 172.21.15.83:0/4267263172 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f173406b4d0 msgr2=0x7f173406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.148+0000 7f174e878640 1 --2- 172.21.15.83:0/4267263172 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f173406b4d0 0x7f173406d990 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f1748072f10 tx=0x7f173c008040 comp rx=0 tx=0).stop 2024-03-06T07:27:12.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.148+0000 7f174e878640 1 -- 172.21.15.83:0/4267263172 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f174806b4f0 msgr2=0x7f174814e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.148+0000 7f174e878640 1 --2- 172.21.15.83:0/4267263172 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f174806b4f0 0x7f174814e400 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f1738000c00 tx=0x7f17380046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.148+0000 7f174e878640 1 -- 172.21.15.83:0/4267263172 shutdown_connections 2024-03-06T07:27:12.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.149+0000 7f174e878640 1 --2- 172.21.15.83:0/4267263172 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f173406b4d0 0x7f173406d990 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.149+0000 7f174e878640 1 --2- 172.21.15.83:0/4267263172 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f174814ea80 0x7f17480b05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.149+0000 7f174e878640 1 --2- 172.21.15.83:0/4267263172 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f174815ec40 0x7f1748153f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.149+0000 7f174e878640 1 --2- 172.21.15.83:0/4267263172 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f174806b4f0 0x7f174814e400 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.149+0000 7f174e878640 1 -- 172.21.15.83:0/4267263172 >> 172.21.15.83:0/4267263172 conn(0x7f17480b2660 msgr2=0x7f1748071990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:12.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.149+0000 7f174e878640 1 -- 172.21.15.83:0/4267263172 shutdown_connections 2024-03-06T07:27:12.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.149+0000 7f174e878640 1 -- 172.21.15.83:0/4267263172 wait complete. 2024-03-06T07:27:12.163 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:27:12.163 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:27:12.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.266+0000 7f9d9633e640 1 Processor -- start 2024-03-06T07:27:12.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.266+0000 7f9d9633e640 1 -- start start 2024-03-06T07:27:12.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.267+0000 7f9d9633e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9d901614d0 0x7f9d900d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.267+0000 7f9d9633e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9d900d2480 0x7f9d900b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.267+0000 7f9d9633e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d901618b0 con 0x7f9d900d1910 2024-03-06T07:27:12.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.267+0000 7f9d9633e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d900cf400 con 0x7f9d900d2480 2024-03-06T07:27:12.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.267+0000 7f9d9633e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d900cf6a0 con 0x7f9d901614d0 2024-03-06T07:27:12.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.267+0000 7f9d948db640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9d900d2480 0x7f9d900b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.267+0000 7f9d8f7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f9d900d1910 0x7f9d900d1d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:37612/0 (socket says 172.21.15.83:37612) 2024-03-06T07:27:12.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.267+0000 7f9d8f7fe640 1 -- 172.21.15.83:0/3756529588 learned_addr learned my addr 172.21.15.83:0/3756529588 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:27:12.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.267+0000 7f9d948db640 1 -- 172.21.15.83:0/3756529588 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9d901614d0 msgr2=0x7f9d900d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:12.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.267+0000 7f9d948db640 1 --2- 172.21.15.83:0/3756529588 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9d901614d0 0x7f9d900d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.267+0000 7f9d948db640 1 -- 172.21.15.83:0/3756529588 >> v1:172.21.15.83:6789/0 conn(0x7f9d900d1910 legacy=0x7f9d900d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.267+0000 7f9d948db640 1 -- 172.21.15.83:0/3756529588 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d900cf980 con 0x7f9d900d2480 2024-03-06T07:27:12.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.267+0000 7f9d8effd640 1 -- 172.21.15.83:0/3756529588 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 87489145 0 0) 0x7f9d901618b0 con 0x7f9d900d1910 2024-03-06T07:27:12.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.268+0000 7f9d948db640 1 --2- 172.21.15.83:0/3756529588 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d900d2480 0x7f9d900b0e30 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f9d88009d90 tx=0x7f9d8802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3a1ee5b49ca7e66 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:12.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.268+0000 7f9d8effd640 1 -- 172.21.15.83:0/3756529588 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d88039070 con 0x7f9d900d2480 2024-03-06T07:27:12.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.268+0000 7f9d8effd640 1 -- 172.21.15.83:0/3756529588 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d8802cc50 con 0x7f9d900d2480 2024-03-06T07:27:12.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.268+0000 7f9d8effd640 1 -- 172.21.15.83:0/3756529588 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d88034440 con 0x7f9d900d2480 2024-03-06T07:27:12.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.269+0000 7f9d9633e640 1 -- 172.21.15.83:0/3756529588 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d900d2480 msgr2=0x7f9d900b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.269+0000 7f9d9633e640 1 --2- 172.21.15.83:0/3756529588 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d900d2480 0x7f9d900b0e30 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f9d88009d90 tx=0x7f9d8802bea0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.269+0000 7f9d9633e640 1 -- 172.21.15.83:0/3756529588 shutdown_connections 2024-03-06T07:27:12.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.269+0000 7f9d9633e640 1 --2- 172.21.15.83:0/3756529588 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d900d2480 0x7f9d900b0e30 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.269+0000 7f9d9633e640 1 --2- 172.21.15.83:0/3756529588 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9d901614d0 0x7f9d900d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.269+0000 7f9d9633e640 1 -- 172.21.15.83:0/3756529588 >> 172.21.15.83:0/3756529588 conn(0x7f9d900d3d40 msgr2=0x7f9d900d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:12.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.269+0000 7f9d9633e640 1 -- 172.21.15.83:0/3756529588 shutdown_connections 2024-03-06T07:27:12.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.269+0000 7f9d9633e640 1 -- 172.21.15.83:0/3756529588 wait complete. 2024-03-06T07:27:12.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.270+0000 7f9d9633e640 1 Processor -- start 2024-03-06T07:27:12.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.270+0000 7f9d9633e640 1 -- start start 2024-03-06T07:27:12.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.271+0000 7f9d9633e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9d900d1910 0x7f9d901594e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.271+0000 7f9d9633e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9d900d2480 0x7f9d900cc090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.271+0000 7f9d9633e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d901614d0 0x7f9d900cc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.271+0000 7f9d9633e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d900b1370 con 0x7f9d901614d0 2024-03-06T07:27:12.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.271+0000 7f9d9633e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d900b0910 con 0x7f9d900d1910 2024-03-06T07:27:12.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.271+0000 7f9d9633e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d900b14f0 con 0x7f9d900d2480 2024-03-06T07:27:12.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.271+0000 7f9d948db640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d901614d0 0x7f9d900cc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.271+0000 7f9d948db640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d901614d0 0x7f9d900cc5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47626/0 (socket says 172.21.15.83:47626) 2024-03-06T07:27:12.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.271+0000 7f9d948db640 1 -- 172.21.15.83:0/2977868695 learned_addr learned my addr 172.21.15.83:0/2977868695 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:12.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.271+0000 7f9d8f7fe640 1 --2- 172.21.15.83:0/2977868695 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9d900d2480 0x7f9d900cc090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.271+0000 7f9d8ffff640 1 --2- 172.21.15.83:0/2977868695 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9d900d1910 0x7f9d901594e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.271+0000 7f9d948db640 1 -- 172.21.15.83:0/2977868695 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9d900d2480 msgr2=0x7f9d900cc090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.271+0000 7f9d948db640 1 --2- 172.21.15.83:0/2977868695 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9d900d2480 0x7f9d900cc090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.271+0000 7f9d948db640 1 -- 172.21.15.83:0/2977868695 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9d900d1910 msgr2=0x7f9d901594e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.271+0000 7f9d948db640 1 --2- 172.21.15.83:0/2977868695 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9d900d1910 0x7f9d901594e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.271+0000 7f9d948db640 1 -- 172.21.15.83:0/2977868695 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d90159ca0 con 0x7f9d901614d0 2024-03-06T07:27:12.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.271+0000 7f9d948db640 1 --2- 172.21.15.83:0/2977868695 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d901614d0 0x7f9d900cc5d0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f9d88009eb0 tx=0x7f9d88004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:12.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.271+0000 7f9d8cff9640 1 -- 172.21.15.83:0/2977868695 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d88044030 con 0x7f9d901614d0 2024-03-06T07:27:12.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.272+0000 7f9d8cff9640 1 -- 172.21.15.83:0/2977868695 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d8802ca50 con 0x7f9d901614d0 2024-03-06T07:27:12.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.272+0000 7f9d9633e640 1 -- 172.21.15.83:0/2977868695 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d90159f80 con 0x7f9d901614d0 2024-03-06T07:27:12.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.272+0000 7f9d8cff9640 1 -- 172.21.15.83:0/2977868695 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d88039040 con 0x7f9d901614d0 2024-03-06T07:27:12.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.272+0000 7f9d9633e640 1 -- 172.21.15.83:0/2977868695 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d9015a540 con 0x7f9d901614d0 2024-03-06T07:27:12.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.273+0000 7f9d9633e640 1 -- 172.21.15.83:0/2977868695 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d90068ff0 con 0x7f9d901614d0 2024-03-06T07:27:12.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.273+0000 7f9d8cff9640 1 -- 172.21.15.83:0/2977868695 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9d88047070 con 0x7f9d901614d0 2024-03-06T07:27:12.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.273+0000 7f9d8cff9640 1 --2- 172.21.15.83:0/2977868695 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9d7c06b4d0 0x7f9d7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.274+0000 7f9d8ffff640 1 --2- 172.21.15.83:0/2977868695 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9d7c06b4d0 0x7f9d7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.274+0000 7f9d8cff9640 1 -- 172.21.15.83:0/2977868695 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9d880ac8c0 con 0x7f9d901614d0 2024-03-06T07:27:12.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.274+0000 7f9d8ffff640 1 --2- 172.21.15.83:0/2977868695 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9d7c06b4d0 0x7f9d7c06d990 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f9d8000b810 tx=0x7f9d80008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:12.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.278+0000 7f9d8cff9640 1 -- 172.21.15.83:0/2977868695 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9d8804e070 con 0x7f9d901614d0 2024-03-06T07:27:12.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:12.511+0000 7f9d9633e640 1 -- 172.21.15.83:0/2977868695 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f9d900b5e00 con 0x7f9d901614d0 2024-03-06T07:27:13.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.154+0000 7f9d8cff9640 1 -- 172.21.15.83:0/2977868695 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v59) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9d88047360 con 0x7f9d901614d0 2024-03-06T07:27:13.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.156+0000 7f9d9633e640 1 -- 172.21.15.83:0/2977868695 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9d7c06b4d0 msgr2=0x7f9d7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.156+0000 7f9d9633e640 1 --2- 172.21.15.83:0/2977868695 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9d7c06b4d0 0x7f9d7c06d990 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f9d8000b810 tx=0x7f9d80008040 comp rx=0 tx=0).stop 2024-03-06T07:27:13.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.156+0000 7f9d9633e640 1 -- 172.21.15.83:0/2977868695 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d901614d0 msgr2=0x7f9d900cc5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.156+0000 7f9d9633e640 1 --2- 172.21.15.83:0/2977868695 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d901614d0 0x7f9d900cc5d0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f9d88009eb0 tx=0x7f9d88004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:13.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.156+0000 7f9d9633e640 1 -- 172.21.15.83:0/2977868695 shutdown_connections 2024-03-06T07:27:13.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.156+0000 7f9d9633e640 1 --2- 172.21.15.83:0/2977868695 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9d7c06b4d0 0x7f9d7c06d990 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.156+0000 7f9d9633e640 1 --2- 172.21.15.83:0/2977868695 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d901614d0 0x7f9d900cc5d0 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.156+0000 7f9d9633e640 1 --2- 172.21.15.83:0/2977868695 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9d900d2480 0x7f9d900cc090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.156+0000 7f9d9633e640 1 --2- 172.21.15.83:0/2977868695 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9d900d1910 0x7f9d901594e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.156+0000 7f9d9633e640 1 -- 172.21.15.83:0/2977868695 >> 172.21.15.83:0/2977868695 conn(0x7f9d900d3d40 msgr2=0x7f9d900b5550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:13.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.156+0000 7f9d9633e640 1 -- 172.21.15.83:0/2977868695 shutdown_connections 2024-03-06T07:27:13.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.156+0000 7f9d9633e640 1 -- 172.21.15.83:0/2977868695 wait complete. 2024-03-06T07:27:13.169 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:27:13.169 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:13.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.300+0000 7f0f711bd640 1 Processor -- start 2024-03-06T07:27:13.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.300+0000 7f0f711bd640 1 -- start start 2024-03-06T07:27:13.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.301+0000 7f0f711bd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0f6c0cf790 0x7f0f6c0ad860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.301+0000 7f0f711bd640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f0f6c0adda0 0x7f0f6c071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.301+0000 7f0f711bd640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f6c0ae240 con 0x7f0f6c072220 2024-03-06T07:27:13.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.301+0000 7f0f711bd640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f6c06cf70 con 0x7f0f6c0adda0 2024-03-06T07:27:13.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.301+0000 7f0f711bd640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f6c0ae520 con 0x7f0f6c0cf790 2024-03-06T07:27:13.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.301+0000 7f0f6b59e640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f0f6c072220 0x7f0f6c1619b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:37624/0 (socket says 172.21.15.83:37624) 2024-03-06T07:27:13.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.301+0000 7f0f6b59e640 1 -- 172.21.15.83:0/3769017905 learned_addr learned my addr 172.21.15.83:0/3769017905 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:27:13.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.301+0000 7f0f6a59c640 1 -- 172.21.15.83:0/3769017905 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 479703367 0 0) 0x7f0f6c0ae240 con 0x7f0f6c072220 2024-03-06T07:27:13.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.301+0000 7f0f6a59c640 1 -- 172.21.15.83:0/3769017905 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f58003520 con 0x7f0f6c072220 2024-03-06T07:27:13.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.301+0000 7f0f63fff640 1 --2- 172.21.15.83:0/3769017905 >> v2:172.21.15.183:3300/0 conn(0x7f0f6c0adda0 0x7f0f6c071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.301+0000 7f0f6ad9d640 1 --2- 172.21.15.83:0/3769017905 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0f6c0cf790 0x7f0f6c0ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.301+0000 7f0f6a59c640 1 -- 172.21.15.83:0/3769017905 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1428265647 0 0) 0x7f0f58003520 con 0x7f0f6c072220 2024-03-06T07:27:13.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.301+0000 7f0f6a59c640 1 -- 172.21.15.83:0/3769017905 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0f6c0cf790 msgr2=0x7f0f6c0ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:13.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.301+0000 7f0f6a59c640 1 --2- 172.21.15.83:0/3769017905 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0f6c0cf790 0x7f0f6c0ad860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.301+0000 7f0f6a59c640 1 -- 172.21.15.83:0/3769017905 >> v2:172.21.15.183:3300/0 conn(0x7f0f6c0adda0 msgr2=0x7f0f6c071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:13.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.301+0000 7f0f6a59c640 1 --2- 172.21.15.83:0/3769017905 >> v2:172.21.15.183:3300/0 conn(0x7f0f6c0adda0 0x7f0f6c071ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.301+0000 7f0f6a59c640 1 -- 172.21.15.83:0/3769017905 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f6c0ae800 con 0x7f0f6c072220 2024-03-06T07:27:13.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.301+0000 7f0f6a59c640 1 -- 172.21.15.83:0/3769017905 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f0f640019f0 con 0x7f0f6c072220 2024-03-06T07:27:13.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.301+0000 7f0f63fff640 1 --2- 172.21.15.83:0/3769017905 >> v2:172.21.15.183:3300/0 conn(0x7f0f6c0adda0 0x7f0f6c071ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:13.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.301+0000 7f0f6a59c640 1 -- 172.21.15.83:0/3769017905 >> v1:172.21.15.83:6789/0 conn(0x7f0f6c072220 legacy=0x7f0f6c1619b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.302+0000 7f0f6a59c640 1 --2- 172.21.15.83:0/3769017905 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f580039b0 0x7f0f58003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.302+0000 7f0f6a59c640 1 --2- 172.21.15.83:0/3769017905 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0f58004800 0x7f0f58004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.302+0000 7f0f6a59c640 1 --2- 172.21.15.83:0/3769017905 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0f6c0cf790 0x7f0f580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.302+0000 7f0f6a59c640 1 -- 172.21.15.83:0/3769017905 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f6c06cf70 con 0x7f0f58004800 2024-03-06T07:27:13.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.302+0000 7f0f6a59c640 1 -- 172.21.15.83:0/3769017905 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f6c0ae520 con 0x7f0f580039b0 2024-03-06T07:27:13.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.302+0000 7f0f6a59c640 1 -- 172.21.15.83:0/3769017905 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5800ba70 con 0x7f0f6c0cf790 2024-03-06T07:27:13.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.302+0000 7f0f63fff640 1 --2- 172.21.15.83:0/3769017905 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0f58004800 0x7f0f58004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.302+0000 7f0f6ad9d640 1 --2- 172.21.15.83:0/3769017905 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f580039b0 0x7f0f58003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.302+0000 7f0f6b59e640 1 --2- 172.21.15.83:0/3769017905 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0f6c0cf790 0x7f0f580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.302+0000 7f0f63fff640 1 -- 172.21.15.83:0/3769017905 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0f6c0cf790 msgr2=0x7f0f580051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:13.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.302+0000 7f0f63fff640 1 --2- 172.21.15.83:0/3769017905 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0f6c0cf790 0x7f0f580051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.302+0000 7f0f63fff640 1 -- 172.21.15.83:0/3769017905 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f580039b0 msgr2=0x7f0f58003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:13.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.302+0000 7f0f63fff640 1 --2- 172.21.15.83:0/3769017905 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f580039b0 0x7f0f58003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.302+0000 7f0f63fff640 1 -- 172.21.15.83:0/3769017905 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f58003520 con 0x7f0f58004800 2024-03-06T07:27:13.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.303+0000 7f0f63fff640 1 --2- 172.21.15.83:0/3769017905 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0f58004800 0x7f0f58004c80 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f0f54006fd0 tx=0x7f0f58005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c845fdadb521b64 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.303+0000 7f0f6a59c640 1 -- 172.21.15.83:0/3769017905 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f5400f040 con 0x7f0f58004800 2024-03-06T07:27:13.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.303+0000 7f0f6a59c640 1 -- 172.21.15.83:0/3769017905 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f54004230 con 0x7f0f58004800 2024-03-06T07:27:13.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.303+0000 7f0f6a59c640 1 -- 172.21.15.83:0/3769017905 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f54003610 con 0x7f0f58004800 2024-03-06T07:27:13.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.303+0000 7f0f711bd640 1 -- 172.21.15.83:0/3769017905 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0f58004800 msgr2=0x7f0f58004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.303+0000 7f0f711bd640 1 --2- 172.21.15.83:0/3769017905 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0f58004800 0x7f0f58004c80 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f0f54006fd0 tx=0x7f0f58005830 comp rx=0 tx=0).stop 2024-03-06T07:27:13.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.303+0000 7f0f6b59e640 1 -- 172.21.15.83:0/3769017905 reap_dead start 2024-03-06T07:27:13.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.303+0000 7f0f711bd640 1 -- 172.21.15.83:0/3769017905 shutdown_connections 2024-03-06T07:27:13.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.303+0000 7f0f711bd640 1 -- 172.21.15.83:0/3769017905 >> 172.21.15.83:0/3769017905 conn(0x7f0f6c0b1a90 msgr2=0x7f0f6c0b1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:13.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.303+0000 7f0f711bd640 1 -- 172.21.15.83:0/3769017905 shutdown_connections 2024-03-06T07:27:13.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.304+0000 7f0f711bd640 1 -- 172.21.15.83:0/3769017905 wait complete. 2024-03-06T07:27:13.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.305+0000 7f0f711bd640 1 Processor -- start 2024-03-06T07:27:13.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.305+0000 7f0f711bd640 1 -- start start 2024-03-06T07:27:13.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.306+0000 7f0f711bd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0f58004800 0x7f0f6c153b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.306+0000 7f0f711bd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0f6c152570 0x7f0f6c151c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.306+0000 7f0f711bd640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f6c156b40 0x7f0f6c156fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.306+0000 7f0f711bd640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f6c15e700 con 0x7f0f6c152570 2024-03-06T07:27:13.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.306+0000 7f0f711bd640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f6c15e580 con 0x7f0f6c156b40 2024-03-06T07:27:13.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.306+0000 7f0f711bd640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f6c06e410 con 0x7f0f58004800 2024-03-06T07:27:13.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.306+0000 7f0f63fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0f6c152570 0x7f0f6c151c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.306+0000 7f0f63fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0f6c152570 0x7f0f6c151c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47652/0 (socket says 172.21.15.83:47652) 2024-03-06T07:27:13.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.306+0000 7f0f63fff640 1 -- 172.21.15.83:0/307975025 learned_addr learned my addr 172.21.15.83:0/307975025 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:13.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.306+0000 7f0f6b59e640 1 --2- 172.21.15.83:0/307975025 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f6c156b40 0x7f0f6c156fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.306+0000 7f0f6ad9d640 1 --2- 172.21.15.83:0/307975025 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0f58004800 0x7f0f6c153b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.306+0000 7f0f63fff640 1 -- 172.21.15.83:0/307975025 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0f58004800 msgr2=0x7f0f6c153b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.306+0000 7f0f63fff640 1 --2- 172.21.15.83:0/307975025 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0f58004800 0x7f0f6c153b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.306+0000 7f0f63fff640 1 -- 172.21.15.83:0/307975025 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f6c156b40 msgr2=0x7f0f6c156fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.306+0000 7f0f63fff640 1 --2- 172.21.15.83:0/307975025 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f6c156b40 0x7f0f6c156fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.306+0000 7f0f63fff640 1 -- 172.21.15.83:0/307975025 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f6c1e7a20 con 0x7f0f6c152570 2024-03-06T07:27:13.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.306+0000 7f0f63fff640 1 --2- 172.21.15.83:0/307975025 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0f6c152570 0x7f0f6c151c90 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f0f54009960 tx=0x7f0f54004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.306+0000 7f0f637fe640 1 -- 172.21.15.83:0/307975025 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f54048070 con 0x7f0f6c152570 2024-03-06T07:27:13.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.306+0000 7f0f637fe640 1 -- 172.21.15.83:0/307975025 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f54030460 con 0x7f0f6c152570 2024-03-06T07:27:13.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.306+0000 7f0f711bd640 1 -- 172.21.15.83:0/307975025 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f6c1e7d00 con 0x7f0f6c152570 2024-03-06T07:27:13.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.306+0000 7f0f637fe640 1 -- 172.21.15.83:0/307975025 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f54030830 con 0x7f0f6c152570 2024-03-06T07:27:13.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.306+0000 7f0f711bd640 1 -- 172.21.15.83:0/307975025 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f6c1e82c0 con 0x7f0f6c152570 2024-03-06T07:27:13.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.308+0000 7f0f711bd640 1 -- 172.21.15.83:0/307975025 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f6c04f330 con 0x7f0f6c152570 2024-03-06T07:27:13.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.308+0000 7f0f637fe640 1 -- 172.21.15.83:0/307975025 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0f5404b070 con 0x7f0f6c152570 2024-03-06T07:27:13.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.308+0000 7f0f637fe640 1 --2- 172.21.15.83:0/307975025 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0f4c06b500 0x7f0f4c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.308+0000 7f0f6ad9d640 1 --2- 172.21.15.83:0/307975025 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0f4c06b500 0x7f0f4c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.308+0000 7f0f637fe640 1 -- 172.21.15.83:0/307975025 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0f540abc20 con 0x7f0f6c152570 2024-03-06T07:27:13.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.309+0000 7f0f6ad9d640 1 --2- 172.21.15.83:0/307975025 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0f4c06b500 0x7f0f4c06d9c0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c006fd0 tx=0x7f0f5c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.313+0000 7f0f637fe640 1 -- 172.21.15.83:0/307975025 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0f54034020 con 0x7f0f6c152570 2024-03-06T07:27:13.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.528+0000 7f0f711bd640 1 -- 172.21.15.83:0/307975025 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f6c072dd0 con 0x7f0f6c152570 2024-03-06T07:27:13.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.529+0000 7f0f637fe640 1 -- 172.21.15.83:0/307975025 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f0f5407ab30 con 0x7f0f6c152570 2024-03-06T07:27:13.531 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:13.531 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":59,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6200,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/2154269119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":2154269119},{"type":"v1","addr":"172.21.15.83:6837","nonce":2154269119}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6206,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3898960075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":3898960075},{"type":"v1","addr":"172.21.15.183:6835","nonce":3898960075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6838,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/1383294888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":1383294888},{"type":"v1","addr":"172.21.15.183:6834","nonce":1383294888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.491022+0000","modified":"2024-03-06T07:27:12.512838+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7887},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7887":{"gid":7887,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6836/855387870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":855387870},{"type":"v1","addr":"172.21.15.83:6836","nonce":855387870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7887,"qdb_cluster":[7887]},"id":8}]} 2024-03-06T07:27:13.531 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 59 2024-03-06T07:27:13.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.531+0000 7f0f711bd640 1 -- 172.21.15.83:0/307975025 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0f4c06b500 msgr2=0x7f0f4c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.531+0000 7f0f711bd640 1 --2- 172.21.15.83:0/307975025 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0f4c06b500 0x7f0f4c06d9c0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c006fd0 tx=0x7f0f5c008040 comp rx=0 tx=0).stop 2024-03-06T07:27:13.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.531+0000 7f0f711bd640 1 -- 172.21.15.83:0/307975025 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0f6c152570 msgr2=0x7f0f6c151c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.531+0000 7f0f711bd640 1 --2- 172.21.15.83:0/307975025 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0f6c152570 0x7f0f6c151c90 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f0f54009960 tx=0x7f0f54004730 comp rx=0 tx=0).stop 2024-03-06T07:27:13.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.531+0000 7f0f711bd640 1 -- 172.21.15.83:0/307975025 shutdown_connections 2024-03-06T07:27:13.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.531+0000 7f0f711bd640 1 --2- 172.21.15.83:0/307975025 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0f4c06b500 0x7f0f4c06d9c0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.531+0000 7f0f711bd640 1 --2- 172.21.15.83:0/307975025 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f6c156b40 0x7f0f6c156fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.531+0000 7f0f711bd640 1 --2- 172.21.15.83:0/307975025 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0f6c152570 0x7f0f6c151c90 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.531+0000 7f0f711bd640 1 --2- 172.21.15.83:0/307975025 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0f58004800 0x7f0f6c153b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.531+0000 7f0f711bd640 1 -- 172.21.15.83:0/307975025 >> 172.21.15.83:0/307975025 conn(0x7f0f6c0b1a90 msgr2=0x7f0f6c070bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:13.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.531+0000 7f0f711bd640 1 -- 172.21.15.83:0/307975025 shutdown_connections 2024-03-06T07:27:13.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.531+0000 7f0f711bd640 1 -- 172.21.15.83:0/307975025 wait complete. 2024-03-06T07:27:13.546 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:13.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.645+0000 7f1e768eb640 1 Processor -- start 2024-03-06T07:27:13.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.645+0000 7f1e768eb640 1 -- start start 2024-03-06T07:27:13.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.646+0000 7f1e768eb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1e7015dfe0 0x7f1e700cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.646+0000 7f1e768eb640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f1e700cf830 0x7f1e700d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.646+0000 7f1e768eb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1e700d27f0 0x7f1e700d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.646+0000 7f1e768eb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e700b31e0 con 0x7f1e7015dfe0 2024-03-06T07:27:13.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.646+0000 7f1e768eb640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e700b3d60 con 0x7f1e700cf830 2024-03-06T07:27:13.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.646+0000 7f1e768eb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e700b3430 con 0x7f1e700d27f0 2024-03-06T07:27:13.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.646+0000 7f1e6ffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1e7015dfe0 0x7f1e700cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.646+0000 7f1e6ffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1e7015dfe0 0x7f1e700cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47654/0 (socket says 172.21.15.83:47654) 2024-03-06T07:27:13.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.646+0000 7f1e6ffff640 1 -- 172.21.15.83:0/2262007149 learned_addr learned my addr 172.21.15.83:0/2262007149 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:13.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.647+0000 7f1e6f7fe640 1 --2- 172.21.15.83:0/2262007149 >> v2:172.21.15.183:3300/0 conn(0x7f1e700cf830 0x7f1e700d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.647+0000 7f1e74e88640 1 --2- 172.21.15.83:0/2262007149 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1e700d27f0 0x7f1e700d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.647+0000 7f1e6ffff640 1 -- 172.21.15.83:0/2262007149 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1e700d27f0 msgr2=0x7f1e700d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:13.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.647+0000 7f1e6ffff640 1 --2- 172.21.15.83:0/2262007149 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1e700d27f0 0x7f1e700d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.647+0000 7f1e6ffff640 1 -- 172.21.15.83:0/2262007149 >> v2:172.21.15.183:3300/0 conn(0x7f1e700cf830 msgr2=0x7f1e700d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:13.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.647+0000 7f1e6ffff640 1 --2- 172.21.15.83:0/2262007149 >> v2:172.21.15.183:3300/0 conn(0x7f1e700cf830 0x7f1e700d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.647+0000 7f1e6ffff640 1 -- 172.21.15.83:0/2262007149 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e700b3710 con 0x7f1e7015dfe0 2024-03-06T07:27:13.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.647+0000 7f1e6f7fe640 1 --2- 172.21.15.83:0/2262007149 >> v2:172.21.15.183:3300/0 conn(0x7f1e700cf830 0x7f1e700d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:13.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.647+0000 7f1e74e88640 1 --2- 172.21.15.83:0/2262007149 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1e700d27f0 0x7f1e700d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:13.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.647+0000 7f1e6ffff640 1 --2- 172.21.15.83:0/2262007149 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e7015dfe0 0x7f1e700cf2f0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f1e700d16b0 tx=0x7f1e6002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2760276895f68965 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.648+0000 7f1e6effd640 1 -- 172.21.15.83:0/2262007149 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e60035020 con 0x7f1e7015dfe0 2024-03-06T07:27:13.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.648+0000 7f1e6effd640 1 -- 172.21.15.83:0/2262007149 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e60031870 con 0x7f1e7015dfe0 2024-03-06T07:27:13.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.648+0000 7f1e6effd640 1 -- 172.21.15.83:0/2262007149 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e60031c40 con 0x7f1e7015dfe0 2024-03-06T07:27:13.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.648+0000 7f1e768eb640 1 -- 172.21.15.83:0/2262007149 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e7015dfe0 msgr2=0x7f1e700cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.648+0000 7f1e768eb640 1 --2- 172.21.15.83:0/2262007149 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e7015dfe0 0x7f1e700cf2f0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f1e700d16b0 tx=0x7f1e6002b910 comp rx=0 tx=0).stop 2024-03-06T07:27:13.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.648+0000 7f1e768eb640 1 -- 172.21.15.83:0/2262007149 shutdown_connections 2024-03-06T07:27:13.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.648+0000 7f1e768eb640 1 --2- 172.21.15.83:0/2262007149 >> v2:172.21.15.183:3300/0 conn(0x7f1e700cf830 0x7f1e700d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.648+0000 7f1e768eb640 1 --2- 172.21.15.83:0/2262007149 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1e700d27f0 0x7f1e700d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.648+0000 7f1e768eb640 1 --2- 172.21.15.83:0/2262007149 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e7015dfe0 0x7f1e700cf2f0 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.648+0000 7f1e768eb640 1 -- 172.21.15.83:0/2262007149 >> 172.21.15.83:0/2262007149 conn(0x7f1e700d3cd0 msgr2=0x7f1e700d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:13.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.648+0000 7f1e768eb640 1 -- 172.21.15.83:0/2262007149 shutdown_connections 2024-03-06T07:27:13.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.648+0000 7f1e768eb640 1 -- 172.21.15.83:0/2262007149 wait complete. 2024-03-06T07:27:13.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.649+0000 7f1e768eb640 1 Processor -- start 2024-03-06T07:27:13.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.650+0000 7f1e768eb640 1 -- start start 2024-03-06T07:27:13.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.650+0000 7f1e768eb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e700cf830 0x7f1e700ca1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.650+0000 7f1e768eb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1e700d27f0 0x7f1e700b9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.650+0000 7f1e768eb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e7015dfe0 0x7f1e7015b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.650+0000 7f1e768eb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e700d1df0 con 0x7f1e700cf830 2024-03-06T07:27:13.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.650+0000 7f1e768eb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e700d1c70 con 0x7f1e7015dfe0 2024-03-06T07:27:13.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.650+0000 7f1e768eb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7015e3c0 con 0x7f1e700d27f0 2024-03-06T07:27:13.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.650+0000 7f1e6ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e700cf830 0x7f1e700ca1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.650+0000 7f1e6ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e700cf830 0x7f1e700ca1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47670/0 (socket says 172.21.15.83:47670) 2024-03-06T07:27:13.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.650+0000 7f1e6ffff640 1 -- 172.21.15.83:0/774806877 learned_addr learned my addr 172.21.15.83:0/774806877 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:13.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.651+0000 7f1e74e88640 1 --2- 172.21.15.83:0/774806877 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e7015dfe0 0x7f1e7015b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.651+0000 7f1e6f7fe640 1 --2- 172.21.15.83:0/774806877 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1e700d27f0 0x7f1e700b9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.651+0000 7f1e6ffff640 1 -- 172.21.15.83:0/774806877 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1e700d27f0 msgr2=0x7f1e700b9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.651+0000 7f1e6ffff640 1 --2- 172.21.15.83:0/774806877 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1e700d27f0 0x7f1e700b9740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.651+0000 7f1e6ffff640 1 -- 172.21.15.83:0/774806877 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e7015dfe0 msgr2=0x7f1e7015b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.651+0000 7f1e6ffff640 1 --2- 172.21.15.83:0/774806877 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e7015dfe0 0x7f1e7015b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.651+0000 7f1e6ffff640 1 -- 172.21.15.83:0/774806877 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e7014e050 con 0x7f1e700cf830 2024-03-06T07:27:13.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.651+0000 7f1e6ffff640 1 --2- 172.21.15.83:0/774806877 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e700cf830 0x7f1e700ca1c0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f1e6002b950 tx=0x7f1e600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.651+0000 7f1e6cff9640 1 -- 172.21.15.83:0/774806877 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e60040070 con 0x7f1e700cf830 2024-03-06T07:27:13.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.651+0000 7f1e6cff9640 1 -- 172.21.15.83:0/774806877 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e60030d00 con 0x7f1e700cf830 2024-03-06T07:27:13.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.651+0000 7f1e768eb640 1 -- 172.21.15.83:0/774806877 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e700b8980 con 0x7f1e700cf830 2024-03-06T07:27:13.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.651+0000 7f1e6cff9640 1 -- 172.21.15.83:0/774806877 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e60035040 con 0x7f1e700cf830 2024-03-06T07:27:13.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.651+0000 7f1e768eb640 1 -- 172.21.15.83:0/774806877 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e700b8f40 con 0x7f1e700cf830 2024-03-06T07:27:13.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.652+0000 7f1e768eb640 1 -- 172.21.15.83:0/774806877 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e7004f330 con 0x7f1e700cf830 2024-03-06T07:27:13.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.652+0000 7f1e6cff9640 1 -- 172.21.15.83:0/774806877 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1e60038020 con 0x7f1e700cf830 2024-03-06T07:27:13.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.653+0000 7f1e6cff9640 1 --2- 172.21.15.83:0/774806877 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1e5006b4d0 0x7f1e5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.653+0000 7f1e6cff9640 1 -- 172.21.15.83:0/774806877 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1e600abf10 con 0x7f1e700cf830 2024-03-06T07:27:13.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.653+0000 7f1e6f7fe640 1 --2- 172.21.15.83:0/774806877 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1e5006b4d0 0x7f1e5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.653+0000 7f1e6f7fe640 1 --2- 172.21.15.83:0/774806877 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1e5006b4d0 0x7f1e5006d990 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f1e58006fd0 tx=0x7f1e58008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.657+0000 7f1e6cff9640 1 -- 172.21.15.83:0/774806877 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1e6007ae20 con 0x7f1e700cf830 2024-03-06T07:27:13.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.837+0000 7f1e768eb640 1 -- 172.21.15.83:0/774806877 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e700ceac0 con 0x7f1e700cf830 2024-03-06T07:27:13.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.838+0000 7f1e6cff9640 1 -- 172.21.15.83:0/774806877 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+16639 (secure 0 0 0) 0x7f1e6007a720 con 0x7f1e700cf830 2024-03-06T07:27:13.840 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:13.840 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":110,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:27:12.146137+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:08.709967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:09.456518+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:13.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.839+0000 7f1e768eb640 1 -- 172.21.15.83:0/774806877 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1e5006b4d0 msgr2=0x7f1e5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.839+0000 7f1e768eb640 1 --2- 172.21.15.83:0/774806877 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1e5006b4d0 0x7f1e5006d990 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f1e58006fd0 tx=0x7f1e58008040 comp rx=0 tx=0).stop 2024-03-06T07:27:13.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.839+0000 7f1e768eb640 1 -- 172.21.15.83:0/774806877 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e700cf830 msgr2=0x7f1e700ca1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.839+0000 7f1e768eb640 1 --2- 172.21.15.83:0/774806877 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e700cf830 0x7f1e700ca1c0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f1e6002b950 tx=0x7f1e600046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.840+0000 7f1e768eb640 1 -- 172.21.15.83:0/774806877 shutdown_connections 2024-03-06T07:27:13.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.840+0000 7f1e768eb640 1 --2- 172.21.15.83:0/774806877 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1e5006b4d0 0x7f1e5006d990 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.840+0000 7f1e768eb640 1 --2- 172.21.15.83:0/774806877 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e7015dfe0 0x7f1e7015b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.840+0000 7f1e768eb640 1 --2- 172.21.15.83:0/774806877 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1e700d27f0 0x7f1e700b9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.840+0000 7f1e768eb640 1 --2- 172.21.15.83:0/774806877 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e700cf830 0x7f1e700ca1c0 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.840+0000 7f1e768eb640 1 -- 172.21.15.83:0/774806877 >> 172.21.15.83:0/774806877 conn(0x7f1e700d3cd0 msgr2=0x7f1e7006bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:13.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.840+0000 7f1e768eb640 1 -- 172.21.15.83:0/774806877 shutdown_connections 2024-03-06T07:27:13.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.840+0000 7f1e768eb640 1 -- 172.21.15.83:0/774806877 wait complete. 2024-03-06T07:27:13.853 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:27:13.853 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:13.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.952+0000 7f99b453a640 1 Processor -- start 2024-03-06T07:27:13.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.952+0000 7f99b453a640 1 -- start start 2024-03-06T07:27:13.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.954+0000 7f99b453a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f99ac15dfe0 0x7f99ac0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.954+0000 7f99b453a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99ac0b3430 con 0x7f99ac0d29c0 2024-03-06T07:27:13.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.954+0000 7f99b453a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99ac0b3710 con 0x7f99ac0cf830 2024-03-06T07:27:13.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.954+0000 7f99b453a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99ac0b31e0 con 0x7f99ac15dfe0 2024-03-06T07:27:13.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.954+0000 7f99b2ad7640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f99ac0d29c0 0x7f99ac0b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:37628/0 (socket says 172.21.15.83:37628) 2024-03-06T07:27:13.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.954+0000 7f99b2ad7640 1 -- 172.21.15.83:0/1148505443 learned_addr learned my addr 172.21.15.83:0/1148505443 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:27:13.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.954+0000 7f99b22d6640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f99ac15dfe0 0x7f99ac0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.954+0000 7f99b22d6640 1 -- 172.21.15.83:0/1148505443 >> v1:172.21.15.183:6789/0 conn(0x7f99ac0cf830 legacy=0x7f99ac0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.954+0000 7f99b22d6640 1 -- 172.21.15.83:0/1148505443 >> v1:172.21.15.83:6789/0 conn(0x7f99ac0d29c0 legacy=0x7f99ac0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.954+0000 7f99b22d6640 1 -- 172.21.15.83:0/1148505443 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99ac0b39f0 con 0x7f99ac15dfe0 2024-03-06T07:27:13.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.954+0000 7f99b12d4640 1 -- 172.21.15.83:0/1148505443 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1205220843 0 0) 0x7f99ac0b3430 con 0x7f99ac0d29c0 2024-03-06T07:27:13.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.955+0000 7f99b22d6640 1 --2- 172.21.15.83:0/1148505443 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99ac15dfe0 0x7f99ac0cf2f0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f99ac0b3360 tx=0x7f99a402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f42eb56c7c6f476 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.955+0000 7f99b12d4640 1 -- 172.21.15.83:0/1148505443 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a4034070 con 0x7f99ac15dfe0 2024-03-06T07:27:13.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.955+0000 7f99b12d4640 1 -- 172.21.15.83:0/1148505443 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99a4030610 con 0x7f99ac15dfe0 2024-03-06T07:27:13.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.955+0000 7f99b12d4640 1 -- 172.21.15.83:0/1148505443 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a40309e0 con 0x7f99ac15dfe0 2024-03-06T07:27:13.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.956+0000 7f99b453a640 1 -- 172.21.15.83:0/1148505443 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99ac15dfe0 msgr2=0x7f99ac0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.956+0000 7f99b453a640 1 --2- 172.21.15.83:0/1148505443 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99ac15dfe0 0x7f99ac0cf2f0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f99ac0b3360 tx=0x7f99a402b960 comp rx=0 tx=0).stop 2024-03-06T07:27:13.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.956+0000 7f99b453a640 1 -- 172.21.15.83:0/1148505443 shutdown_connections 2024-03-06T07:27:13.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.956+0000 7f99b453a640 1 --2- 172.21.15.83:0/1148505443 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99ac15dfe0 0x7f99ac0cf2f0 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.956+0000 7f99b453a640 1 -- 172.21.15.83:0/1148505443 >> 172.21.15.83:0/1148505443 conn(0x7f99ac0d3cd0 msgr2=0x7f99ac0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:13.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.956+0000 7f99b453a640 1 -- 172.21.15.83:0/1148505443 shutdown_connections 2024-03-06T07:27:13.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.956+0000 7f99b453a640 1 -- 172.21.15.83:0/1148505443 wait complete. 2024-03-06T07:27:13.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.958+0000 7f99b453a640 1 Processor -- start 2024-03-06T07:27:13.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.958+0000 7f99b453a640 1 -- start start 2024-03-06T07:27:13.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.959+0000 7f99b453a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99ac0cf830 0x7f99ac0b0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.959+0000 7f99b453a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f99ac0d29c0 0x7f99ac1f3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.959+0000 7f99b453a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f99ac15dfe0 0x7f99ac0cb920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.959+0000 7f99b453a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99ac15e3c0 con 0x7f99ac0cf830 2024-03-06T07:27:13.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.959+0000 7f99b453a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99ac0b3d60 con 0x7f99ac15dfe0 2024-03-06T07:27:13.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.959+0000 7f99b453a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99ac15e540 con 0x7f99ac0d29c0 2024-03-06T07:27:13.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.959+0000 7f99b22d6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99ac0cf830 0x7f99ac0b0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.959+0000 7f99b22d6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99ac0cf830 0x7f99ac0b0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47682/0 (socket says 172.21.15.83:47682) 2024-03-06T07:27:13.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.959+0000 7f99b22d6640 1 -- 172.21.15.83:0/3794888010 learned_addr learned my addr 172.21.15.83:0/3794888010 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:13.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.959+0000 7f99b22d6640 1 -- 172.21.15.83:0/3794888010 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f99ac0d29c0 msgr2=0x7f99ac1f3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.959+0000 7f99b22d6640 1 --2- 172.21.15.83:0/3794888010 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f99ac0d29c0 0x7f99ac1f3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.959+0000 7f99b2ad7640 1 --2- 172.21.15.83:0/3794888010 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f99ac15dfe0 0x7f99ac0cb920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.959+0000 7f99b22d6640 1 -- 172.21.15.83:0/3794888010 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f99ac15dfe0 msgr2=0x7f99ac0cb920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.959+0000 7f99b22d6640 1 --2- 172.21.15.83:0/3794888010 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f99ac15dfe0 0x7f99ac0cb920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.959+0000 7f99b22d6640 1 -- 172.21.15.83:0/3794888010 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99ac0b1880 con 0x7f99ac0cf830 2024-03-06T07:27:13.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.959+0000 7f99b2ad7640 1 --2- 172.21.15.83:0/3794888010 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f99ac15dfe0 0x7f99ac0cb920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:13.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.959+0000 7f99b22d6640 1 --2- 172.21.15.83:0/3794888010 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99ac0cf830 0x7f99ac0b0e50 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f99a4009890 tx=0x7f99a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.960+0000 7f99a2ffd640 1 -- 172.21.15.83:0/3794888010 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a4047070 con 0x7f99ac0cf830 2024-03-06T07:27:13.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.960+0000 7f99a2ffd640 1 -- 172.21.15.83:0/3794888010 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99a4004030 con 0x7f99ac0cf830 2024-03-06T07:27:13.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.960+0000 7f99b453a640 1 -- 172.21.15.83:0/3794888010 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99ac0b15a0 con 0x7f99ac0cf830 2024-03-06T07:27:13.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.960+0000 7f99a2ffd640 1 -- 172.21.15.83:0/3794888010 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a4034040 con 0x7f99ac0cf830 2024-03-06T07:27:13.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.960+0000 7f99b453a640 1 -- 172.21.15.83:0/3794888010 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99ac0cb120 con 0x7f99ac0cf830 2024-03-06T07:27:13.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.961+0000 7f99b453a640 1 -- 172.21.15.83:0/3794888010 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99ac04f330 con 0x7f99ac0cf830 2024-03-06T07:27:13.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.961+0000 7f99a2ffd640 1 -- 172.21.15.83:0/3794888010 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f99a404a070 con 0x7f99ac0cf830 2024-03-06T07:27:13.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.962+0000 7f99a2ffd640 1 --2- 172.21.15.83:0/3794888010 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f999006b500 0x7f999006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.962+0000 7f99a2ffd640 1 -- 172.21.15.83:0/3794888010 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f99a40abcc0 con 0x7f99ac0cf830 2024-03-06T07:27:13.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.962+0000 7f99b1ad5640 1 --2- 172.21.15.83:0/3794888010 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f999006b500 0x7f999006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.962+0000 7f99b1ad5640 1 --2- 172.21.15.83:0/3794888010 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f999006b500 0x7f999006d9c0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f999c000f60 tx=0x7f999c0079b0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:13.966+0000 7f99a2ffd640 1 -- 172.21.15.83:0/3794888010 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f99a407ab40 con 0x7f99ac0cf830 2024-03-06T07:27:14.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.137+0000 7f99b453a640 1 -- 172.21.15.83:0/3794888010 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f99ac1f3790 con 0x7f999006b500 2024-03-06T07:27:14.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.145+0000 7f99a2ffd640 1 -- 172.21.15.83:0/3794888010 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247089 (secure 0 0 0) 0x7f99ac1f3790 con 0x7f999006b500 2024-03-06T07:27:14.150 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:14.152 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-06T07:27:14.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.148+0000 7f99b453a640 1 -- 172.21.15.83:0/3794888010 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f999006b500 msgr2=0x7f999006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.148+0000 7f99b453a640 1 --2- 172.21.15.83:0/3794888010 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f999006b500 0x7f999006d9c0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f999c000f60 tx=0x7f999c0079b0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.149+0000 7f99b453a640 1 -- 172.21.15.83:0/3794888010 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99ac0cf830 msgr2=0x7f99ac0b0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.149+0000 7f99b453a640 1 --2- 172.21.15.83:0/3794888010 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99ac0cf830 0x7f99ac0b0e50 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f99a4009890 tx=0x7f99a40046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.149+0000 7f99b453a640 1 -- 172.21.15.83:0/3794888010 shutdown_connections 2024-03-06T07:27:14.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.149+0000 7f99b453a640 1 --2- 172.21.15.83:0/3794888010 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f999006b500 0x7f999006d9c0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.149+0000 7f99b453a640 1 --2- 172.21.15.83:0/3794888010 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f99ac15dfe0 0x7f99ac0cb920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.149+0000 7f99b453a640 1 --2- 172.21.15.83:0/3794888010 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f99ac0d29c0 0x7f99ac1f3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.149+0000 7f99b453a640 1 --2- 172.21.15.83:0/3794888010 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99ac0cf830 0x7f99ac0b0e50 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.149+0000 7f99b453a640 1 -- 172.21.15.83:0/3794888010 >> 172.21.15.83:0/3794888010 conn(0x7f99ac0d3cd0 msgr2=0x7f99ac0b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:14.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.149+0000 7f99b453a640 1 -- 172.21.15.83:0/3794888010 shutdown_connections 2024-03-06T07:27:14.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.149+0000 7f99b453a640 1 -- 172.21.15.83:0/3794888010 wait complete. 2024-03-06T07:27:14.161 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":267,"stamp":"2024-03-06T07:27:12.377185+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":29,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":323472,"kb_used_data":11608,"kb_used_omap":12,"kb_used_meta":311731,"kb_avail":754651248,"statfs":{"total":773094113280,"available":772762877952,"internally_reserved":0,"allocated":11886592,"data_stored":5768468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12757,"internal_metadata":319213099},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000718"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.143449+0000","last_change":"2024-03-06T07:27:10.143449+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.143449+0000","last_undegraded":"2024-03-06T07:27:10.143449+0000","last_fullsized":"2024-03-06T07:27:10.143449+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.131926+0000","last_change":"2024-03-06T07:27:10.131926+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.131926+0000","last_undegraded":"2024-03-06T07:27:10.131926+0000","last_fullsized":"2024-03-06T07:27:10.131926+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.140106+0000","last_change":"2024-03-06T07:27:10.140106+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.140106+0000","last_undegraded":"2024-03-06T07:27:10.140106+0000","last_fullsized":"2024-03-06T07:27:10.140106+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.141376+0000","last_change":"2024-03-06T07:27:10.141376+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.141376+0000","last_undegraded":"2024-03-06T07:27:10.141376+0000","last_fullsized":"2024-03-06T07:27:10.141376+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.134023+0000","last_change":"2024-03-06T07:27:10.134023+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.134023+0000","last_undegraded":"2024-03-06T07:27:10.134023+0000","last_fullsized":"2024-03-06T07:27:10.134023+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.141567+0000","last_change":"2024-03-06T07:27:10.141567+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.141567+0000","last_undegraded":"2024-03-06T07:27:10.141567+0000","last_fullsized":"2024-03-06T07:27:10.141567+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.135202+0000","last_change":"2024-03-06T07:27:10.135202+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.135202+0000","last_undegraded":"2024-03-06T07:27:10.135202+0000","last_fullsized":"2024-03-06T07:27:10.135202+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.137109+0000","last_change":"2024-03-06T07:27:10.137109+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.137109+0000","last_undegraded":"2024-03-06T07:27:10.137109+0000","last_fullsized":"2024-03-06T07:27:10.137109+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.140884+0000","last_change":"2024-03-06T07:27:10.140884+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.140884+0000","last_undegraded":"2024-03-06T07:27:10.140884+0000","last_fullsized":"2024-03-06T07:27:10.140884+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.140929+0000","last_change":"2024-03-06T07:27:10.140929+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.140929+0000","last_undegraded":"2024-03-06T07:27:10.140929+0000","last_fullsized":"2024-03-06T07:27:10.140929+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.131689+0000","last_change":"2024-03-06T07:27:10.131689+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.131689+0000","last_undegraded":"2024-03-06T07:27:10.131689+0000","last_fullsized":"2024-03-06T07:27:10.131689+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.139813+0000","last_change":"2024-03-06T07:27:10.139813+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.139813+0000","last_undegraded":"2024-03-06T07:27:10.139813+0000","last_fullsized":"2024-03-06T07:27:10.139813+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.136022+0000","last_change":"2024-03-06T07:27:10.136022+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.136022+0000","last_undegraded":"2024-03-06T07:27:10.136022+0000","last_fullsized":"2024-03-06T07:27:10.136022+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.136551+0000","last_change":"2024-03-06T07:27:10.136551+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.136551+0000","last_undegraded":"2024-03-06T07:27:10.136551+0000","last_fullsized":"2024-03-06T07:27:10.136551+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.143562+0000","last_change":"2024-03-06T07:27:10.143562+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.143562+0000","last_undegraded":"2024-03-06T07:27:10.143562+0000","last_fullsized":"2024-03-06T07:27:10.143562+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.141170+0000","last_change":"2024-03-06T07:27:10.141170+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.141170+0000","last_undegraded":"2024-03-06T07:27:10.141170+0000","last_fullsized":"2024-03-06T07:27:10.141170+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.133703+0000","last_change":"2024-03-06T07:27:10.133703+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.133703+0000","last_undegraded":"2024-03-06T07:27:10.133703+0000","last_fullsized":"2024-03-06T07:27:10.133703+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.134084+0000","last_change":"2024-03-06T07:27:10.134084+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.134084+0000","last_undegraded":"2024-03-06T07:27:10.134084+0000","last_fullsized":"2024-03-06T07:27:10.134084+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.139490+0000","last_change":"2024-03-06T07:27:10.139490+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.139490+0000","last_undegraded":"2024-03-06T07:27:10.139490+0000","last_fullsized":"2024-03-06T07:27:10.139490+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.133392+0000","last_change":"2024-03-06T07:27:10.133392+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.133392+0000","last_undegraded":"2024-03-06T07:27:10.133392+0000","last_fullsized":"2024-03-06T07:27:10.133392+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.131525+0000","last_change":"2024-03-06T07:27:10.131525+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.131525+0000","last_undegraded":"2024-03-06T07:27:10.131525+0000","last_fullsized":"2024-03-06T07:27:10.131525+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.138692+0000","last_change":"2024-03-06T07:27:10.138692+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.138692+0000","last_undegraded":"2024-03-06T07:27:10.138692+0000","last_fullsized":"2024-03-06T07:27:10.138692+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.131673+0000","last_change":"2024-03-06T07:27:10.131673+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.131673+0000","last_undegraded":"2024-03-06T07:27:10.131673+0000","last_fullsized":"2024-03-06T07:27:10.131673+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.138495+0000","last_change":"2024-03-06T07:27:10.138495+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.138495+0000","last_undegraded":"2024-03-06T07:27:10.138495+0000","last_fullsized":"2024-03-06T07:27:10.138495+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.132448+0000","last_change":"2024-03-06T07:27:10.132448+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.132448+0000","last_undegraded":"2024-03-06T07:27:10.132448+0000","last_fullsized":"2024-03-06T07:27:10.132448+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.139794+0000","last_change":"2024-03-06T07:27:10.139794+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.139794+0000","last_undegraded":"2024-03-06T07:27:10.139794+0000","last_fullsized":"2024-03-06T07:27:10.139794+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.141443+0000","last_change":"2024-03-06T07:27:10.141443+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.141443+0000","last_undegraded":"2024-03-06T07:27:10.141443+0000","last_fullsized":"2024-03-06T07:27:10.141443+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.139294+0000","last_change":"2024-03-06T07:27:10.139294+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.139294+0000","last_undegraded":"2024-03-06T07:27:10.139294+0000","last_fullsized":"2024-03-06T07:27:10.139294+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.134116+0000","last_change":"2024-03-06T07:27:10.134116+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.134116+0000","last_undegraded":"2024-03-06T07:27:10.134116+0000","last_fullsized":"2024-03-06T07:27:10.134116+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.134159+0000","last_change":"2024-03-06T07:27:10.134159+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.134159+0000","last_undegraded":"2024-03-06T07:27:10.134159+0000","last_fullsized":"2024-03-06T07:27:10.134159+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.141268+0000","last_change":"2024-03-06T07:27:10.141268+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.141268+0000","last_undegraded":"2024-03-06T07:27:10.141268+0000","last_fullsized":"2024-03-06T07:27:10.141268+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.140860+0000","last_change":"2024-03-06T07:27:10.140860+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.140860+0000","last_undegraded":"2024-03-06T07:27:10.140860+0000","last_fullsized":"2024-03-06T07:27:10.140860+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.140859+0000","last_change":"2024-03-06T07:27:10.140859+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.140859+0000","last_undegraded":"2024-03-06T07:27:10.140859+0000","last_fullsized":"2024-03-06T07:27:10.140859+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.144441+0000","last_change":"2024-03-06T07:27:10.143561+0000","last_active":"2024-03-06T07:27:10.144441+0000","last_peered":"2024-03-06T07:27:10.144441+0000","last_clean":"2024-03-06T07:27:10.144441+0000","last_became_active":"2024-03-06T07:27:10.143244+0000","last_became_peered":"2024-03-06T07:27:10.143244+0000","last_unstale":"2024-03-06T07:27:10.144441+0000","last_undegraded":"2024-03-06T07:27:10.144441+0000","last_fullsized":"2024-03-06T07:27:10.144441+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:31:13.932038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.135118+0000","last_change":"2024-03-06T07:27:10.135118+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.135118+0000","last_undegraded":"2024-03-06T07:27:10.135118+0000","last_fullsized":"2024-03-06T07:27:10.135118+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.146696+0000","last_change":"2024-03-06T07:27:10.145366+0000","last_active":"2024-03-06T07:27:10.146696+0000","last_peered":"2024-03-06T07:27:10.146696+0000","last_clean":"2024-03-06T07:27:10.146696+0000","last_became_active":"2024-03-06T07:27:10.145028+0000","last_became_peered":"2024-03-06T07:27:10.145028+0000","last_unstale":"2024-03-06T07:27:10.146696+0000","last_undegraded":"2024-03-06T07:27:10.146696+0000","last_fullsized":"2024-03-06T07:27:10.146696+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:56:36.810364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.141075+0000","last_change":"2024-03-06T07:27:10.141075+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.141075+0000","last_undegraded":"2024-03-06T07:27:10.141075+0000","last_fullsized":"2024-03-06T07:27:10.141075+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.148901+0000","last_change":"2024-03-06T07:27:10.146400+0000","last_active":"2024-03-06T07:27:10.148901+0000","last_peered":"2024-03-06T07:27:10.148901+0000","last_clean":"2024-03-06T07:27:10.148901+0000","last_became_active":"2024-03-06T07:27:10.146106+0000","last_became_peered":"2024-03-06T07:27:10.146106+0000","last_unstale":"2024-03-06T07:27:10.148901+0000","last_undegraded":"2024-03-06T07:27:10.148901+0000","last_fullsized":"2024-03-06T07:27:10.148901+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:24:07.968719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.137861+0000","last_change":"2024-03-06T07:27:10.137861+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.137861+0000","last_undegraded":"2024-03-06T07:27:10.137861+0000","last_fullsized":"2024-03-06T07:27:10.137861+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.140038+0000","last_change":"2024-03-06T07:27:10.137036+0000","last_active":"2024-03-06T07:27:10.140038+0000","last_peered":"2024-03-06T07:27:10.140038+0000","last_clean":"2024-03-06T07:27:10.140038+0000","last_became_active":"2024-03-06T07:27:10.136771+0000","last_became_peered":"2024-03-06T07:27:10.136771+0000","last_unstale":"2024-03-06T07:27:10.140038+0000","last_undegraded":"2024-03-06T07:27:10.140038+0000","last_fullsized":"2024-03-06T07:27:10.140038+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:03:45.041901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.139239+0000","last_change":"2024-03-06T07:27:10.139239+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.139239+0000","last_undegraded":"2024-03-06T07:27:10.139239+0000","last_fullsized":"2024-03-06T07:27:10.139239+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.153078+0000","last_change":"2024-03-06T07:27:10.152619+0000","last_active":"2024-03-06T07:27:10.153078+0000","last_peered":"2024-03-06T07:27:10.153078+0000","last_clean":"2024-03-06T07:27:10.153078+0000","last_became_active":"2024-03-06T07:27:10.152377+0000","last_became_peered":"2024-03-06T07:27:10.152377+0000","last_unstale":"2024-03-06T07:27:10.153078+0000","last_undegraded":"2024-03-06T07:27:10.153078+0000","last_fullsized":"2024-03-06T07:27:10.153078+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:30.226075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.133490+0000","last_change":"2024-03-06T07:27:10.133490+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.133490+0000","last_undegraded":"2024-03-06T07:27:10.133490+0000","last_fullsized":"2024-03-06T07:27:10.133490+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.151980+0000","last_change":"2024-03-06T07:27:10.148748+0000","last_active":"2024-03-06T07:27:10.151980+0000","last_peered":"2024-03-06T07:27:10.151980+0000","last_clean":"2024-03-06T07:27:10.151980+0000","last_became_active":"2024-03-06T07:27:10.148438+0000","last_became_peered":"2024-03-06T07:27:10.148438+0000","last_unstale":"2024-03-06T07:27:10.151980+0000","last_undegraded":"2024-03-06T07:27:10.151980+0000","last_fullsized":"2024-03-06T07:27:10.151980+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:09.417957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.139850+0000","last_change":"2024-03-06T07:27:10.139850+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.139850+0000","last_undegraded":"2024-03-06T07:27:10.139850+0000","last_fullsized":"2024-03-06T07:27:10.139850+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.149200+0000","last_change":"2024-03-06T07:27:10.147680+0000","last_active":"2024-03-06T07:27:10.149200+0000","last_peered":"2024-03-06T07:27:10.149200+0000","last_clean":"2024-03-06T07:27:10.149200+0000","last_became_active":"2024-03-06T07:27:10.147438+0000","last_became_peered":"2024-03-06T07:27:10.147438+0000","last_unstale":"2024-03-06T07:27:10.149200+0000","last_undegraded":"2024-03-06T07:27:10.149200+0000","last_fullsized":"2024-03-06T07:27:10.149200+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:57:44.152110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.135641+0000","last_change":"2024-03-06T07:27:10.135641+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.135641+0000","last_undegraded":"2024-03-06T07:27:10.135641+0000","last_fullsized":"2024-03-06T07:27:10.135641+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.153973+0000","last_change":"2024-03-06T07:27:10.149089+0000","last_active":"2024-03-06T07:27:10.153973+0000","last_peered":"2024-03-06T07:27:10.153973+0000","last_clean":"2024-03-06T07:27:10.153973+0000","last_became_active":"2024-03-06T07:27:10.148126+0000","last_became_peered":"2024-03-06T07:27:10.148126+0000","last_unstale":"2024-03-06T07:27:10.153973+0000","last_undegraded":"2024-03-06T07:27:10.153973+0000","last_fullsized":"2024-03-06T07:27:10.153973+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:23:31.120689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.135927+0000","last_change":"2024-03-06T07:27:10.135927+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.135927+0000","last_undegraded":"2024-03-06T07:27:10.135927+0000","last_fullsized":"2024-03-06T07:27:10.135927+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.149657+0000","last_change":"2024-03-06T07:27:10.146728+0000","last_active":"2024-03-06T07:27:10.149657+0000","last_peered":"2024-03-06T07:27:10.149657+0000","last_clean":"2024-03-06T07:27:10.149657+0000","last_became_active":"2024-03-06T07:27:10.146456+0000","last_became_peered":"2024-03-06T07:27:10.146456+0000","last_unstale":"2024-03-06T07:27:10.149657+0000","last_undegraded":"2024-03-06T07:27:10.149657+0000","last_fullsized":"2024-03-06T07:27:10.149657+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:26.196020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.134728+0000","last_change":"2024-03-06T07:27:10.134728+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.134728+0000","last_undegraded":"2024-03-06T07:27:10.134728+0000","last_fullsized":"2024-03-06T07:27:10.134728+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.151549+0000","last_change":"2024-03-06T07:27:10.148013+0000","last_active":"2024-03-06T07:27:10.151549+0000","last_peered":"2024-03-06T07:27:10.151549+0000","last_clean":"2024-03-06T07:27:10.151549+0000","last_became_active":"2024-03-06T07:27:10.147723+0000","last_became_peered":"2024-03-06T07:27:10.147723+0000","last_unstale":"2024-03-06T07:27:10.151549+0000","last_undegraded":"2024-03-06T07:27:10.151549+0000","last_fullsized":"2024-03-06T07:27:10.151549+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:41.083659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.138899+0000","last_change":"2024-03-06T07:27:10.138899+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.138899+0000","last_undegraded":"2024-03-06T07:27:10.138899+0000","last_fullsized":"2024-03-06T07:27:10.138899+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.151643+0000","last_change":"2024-03-06T07:27:10.145307+0000","last_active":"2024-03-06T07:27:10.151643+0000","last_peered":"2024-03-06T07:27:10.151643+0000","last_clean":"2024-03-06T07:27:10.151643+0000","last_became_active":"2024-03-06T07:27:10.145027+0000","last_became_peered":"2024-03-06T07:27:10.145027+0000","last_unstale":"2024-03-06T07:27:10.151643+0000","last_undegraded":"2024-03-06T07:27:10.151643+0000","last_fullsized":"2024-03-06T07:27:10.151643+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:49:40.057171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.134203+0000","last_change":"2024-03-06T07:27:10.134203+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.134203+0000","last_undegraded":"2024-03-06T07:27:10.134203+0000","last_fullsized":"2024-03-06T07:27:10.134203+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.154023+0000","last_change":"2024-03-06T07:27:10.147579+0000","last_active":"2024-03-06T07:27:10.154023+0000","last_peered":"2024-03-06T07:27:10.154023+0000","last_clean":"2024-03-06T07:27:10.154023+0000","last_became_active":"2024-03-06T07:27:10.147292+0000","last_became_peered":"2024-03-06T07:27:10.147292+0000","last_unstale":"2024-03-06T07:27:10.154023+0000","last_undegraded":"2024-03-06T07:27:10.154023+0000","last_fullsized":"2024-03-06T07:27:10.154023+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:38.297655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.138207+0000","last_change":"2024-03-06T07:27:10.138207+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.138207+0000","last_undegraded":"2024-03-06T07:27:10.138207+0000","last_fullsized":"2024-03-06T07:27:10.138207+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.152149+0000","last_change":"2024-03-06T07:27:10.145448+0000","last_active":"2024-03-06T07:27:10.152149+0000","last_peered":"2024-03-06T07:27:10.152149+0000","last_clean":"2024-03-06T07:27:10.152149+0000","last_became_active":"2024-03-06T07:27:10.145063+0000","last_became_peered":"2024-03-06T07:27:10.145063+0000","last_unstale":"2024-03-06T07:27:10.152149+0000","last_undegraded":"2024-03-06T07:27:10.152149+0000","last_fullsized":"2024-03-06T07:27:10.152149+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:14:01.327671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.137691+0000","last_change":"2024-03-06T07:27:10.137691+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.137691+0000","last_undegraded":"2024-03-06T07:27:10.137691+0000","last_fullsized":"2024-03-06T07:27:10.137691+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.149341+0000","last_change":"2024-03-06T07:27:10.146708+0000","last_active":"2024-03-06T07:27:10.149341+0000","last_peered":"2024-03-06T07:27:10.149341+0000","last_clean":"2024-03-06T07:27:10.149341+0000","last_became_active":"2024-03-06T07:27:10.146387+0000","last_became_peered":"2024-03-06T07:27:10.146387+0000","last_unstale":"2024-03-06T07:27:10.149341+0000","last_undegraded":"2024-03-06T07:27:10.149341+0000","last_fullsized":"2024-03-06T07:27:10.149341+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:12.816920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.139264+0000","last_change":"2024-03-06T07:27:10.139264+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.139264+0000","last_undegraded":"2024-03-06T07:27:10.139264+0000","last_fullsized":"2024-03-06T07:27:10.139264+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.139998+0000","last_change":"2024-03-06T07:27:10.136377+0000","last_active":"2024-03-06T07:27:10.139998+0000","last_peered":"2024-03-06T07:27:10.139998+0000","last_clean":"2024-03-06T07:27:10.139998+0000","last_became_active":"2024-03-06T07:27:10.136062+0000","last_became_peered":"2024-03-06T07:27:10.136062+0000","last_unstale":"2024-03-06T07:27:10.139998+0000","last_undegraded":"2024-03-06T07:27:10.139998+0000","last_fullsized":"2024-03-06T07:27:10.139998+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:44:20.435658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.133922+0000","last_change":"2024-03-06T07:27:10.133922+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.133922+0000","last_undegraded":"2024-03-06T07:27:10.133922+0000","last_fullsized":"2024-03-06T07:27:10.133922+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.153881+0000","last_change":"2024-03-06T07:27:10.147575+0000","last_active":"2024-03-06T07:27:10.153881+0000","last_peered":"2024-03-06T07:27:10.153881+0000","last_clean":"2024-03-06T07:27:10.153881+0000","last_became_active":"2024-03-06T07:27:10.147304+0000","last_became_peered":"2024-03-06T07:27:10.147304+0000","last_unstale":"2024-03-06T07:27:10.153881+0000","last_undegraded":"2024-03-06T07:27:10.153881+0000","last_fullsized":"2024-03-06T07:27:10.153881+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:19:26.310561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.140347+0000","last_change":"2024-03-06T07:27:10.140347+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.140347+0000","last_undegraded":"2024-03-06T07:27:10.140347+0000","last_fullsized":"2024-03-06T07:27:10.140347+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.140158+0000","last_change":"2024-03-06T07:27:10.137380+0000","last_active":"2024-03-06T07:27:10.140158+0000","last_peered":"2024-03-06T07:27:10.140158+0000","last_clean":"2024-03-06T07:27:10.140158+0000","last_became_active":"2024-03-06T07:27:10.137056+0000","last_became_peered":"2024-03-06T07:27:10.137056+0000","last_unstale":"2024-03-06T07:27:10.140158+0000","last_undegraded":"2024-03-06T07:27:10.140158+0000","last_fullsized":"2024-03-06T07:27:10.140158+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:12:04.974749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.135554+0000","last_change":"2024-03-06T07:27:10.135554+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.135554+0000","last_undegraded":"2024-03-06T07:27:10.135554+0000","last_fullsized":"2024-03-06T07:27:10.135554+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.146355+0000","last_change":"2024-03-06T07:27:10.145731+0000","last_active":"2024-03-06T07:27:10.146355+0000","last_peered":"2024-03-06T07:27:10.146355+0000","last_clean":"2024-03-06T07:27:10.146355+0000","last_became_active":"2024-03-06T07:27:10.145424+0000","last_became_peered":"2024-03-06T07:27:10.145424+0000","last_unstale":"2024-03-06T07:27:10.146355+0000","last_undegraded":"2024-03-06T07:27:10.146355+0000","last_fullsized":"2024-03-06T07:27:10.146355+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:09.417957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.137107+0000","last_change":"2024-03-06T07:27:10.137107+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.137107+0000","last_undegraded":"2024-03-06T07:27:10.137107+0000","last_fullsized":"2024-03-06T07:27:10.137107+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.151758+0000","last_change":"2024-03-06T07:27:10.151277+0000","last_active":"2024-03-06T07:27:10.151758+0000","last_peered":"2024-03-06T07:27:10.151758+0000","last_clean":"2024-03-06T07:27:10.151758+0000","last_became_active":"2024-03-06T07:27:10.150933+0000","last_became_peered":"2024-03-06T07:27:10.150933+0000","last_unstale":"2024-03-06T07:27:10.151758+0000","last_undegraded":"2024-03-06T07:27:10.151758+0000","last_fullsized":"2024-03-06T07:27:10.151758+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:16.105130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.135497+0000","last_change":"2024-03-06T07:27:10.135497+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.135497+0000","last_undegraded":"2024-03-06T07:27:10.135497+0000","last_fullsized":"2024-03-06T07:27:10.135497+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.154056+0000","last_change":"2024-03-06T07:27:10.150893+0000","last_active":"2024-03-06T07:27:10.154056+0000","last_peered":"2024-03-06T07:27:10.154056+0000","last_clean":"2024-03-06T07:27:10.154056+0000","last_became_active":"2024-03-06T07:27:10.150648+0000","last_became_peered":"2024-03-06T07:27:10.150648+0000","last_unstale":"2024-03-06T07:27:10.154056+0000","last_undegraded":"2024-03-06T07:27:10.154056+0000","last_fullsized":"2024-03-06T07:27:10.154056+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:34.387756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.139184+0000","last_change":"2024-03-06T07:27:10.139184+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.139184+0000","last_undegraded":"2024-03-06T07:27:10.139184+0000","last_fullsized":"2024-03-06T07:27:10.139184+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.151835+0000","last_change":"2024-03-06T07:27:10.148882+0000","last_active":"2024-03-06T07:27:10.151835+0000","last_peered":"2024-03-06T07:27:10.151835+0000","last_clean":"2024-03-06T07:27:10.151835+0000","last_became_active":"2024-03-06T07:27:10.148626+0000","last_became_peered":"2024-03-06T07:27:10.148626+0000","last_unstale":"2024-03-06T07:27:10.151835+0000","last_undegraded":"2024-03-06T07:27:10.151835+0000","last_fullsized":"2024-03-06T07:27:10.151835+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:19.034867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.136666+0000","last_change":"2024-03-06T07:27:10.136666+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.136666+0000","last_undegraded":"2024-03-06T07:27:10.136666+0000","last_fullsized":"2024-03-06T07:27:10.136666+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.150333+0000","last_change":"2024-03-06T07:27:10.144592+0000","last_active":"2024-03-06T07:27:10.150333+0000","last_peered":"2024-03-06T07:27:10.150333+0000","last_clean":"2024-03-06T07:27:10.150333+0000","last_became_active":"2024-03-06T07:27:10.144328+0000","last_became_peered":"2024-03-06T07:27:10.144328+0000","last_unstale":"2024-03-06T07:27:10.150333+0000","last_undegraded":"2024-03-06T07:27:10.150333+0000","last_fullsized":"2024-03-06T07:27:10.150333+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:39:04.350267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.138259+0000","last_change":"2024-03-06T07:27:10.138259+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.138259+0000","last_undegraded":"2024-03-06T07:27:10.138259+0000","last_fullsized":"2024-03-06T07:27:10.138259+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.154069+0000","last_change":"2024-03-06T07:27:10.151020+0000","last_active":"2024-03-06T07:27:10.154069+0000","last_peered":"2024-03-06T07:27:10.154069+0000","last_clean":"2024-03-06T07:27:10.154069+0000","last_became_active":"2024-03-06T07:27:10.150747+0000","last_became_peered":"2024-03-06T07:27:10.150747+0000","last_unstale":"2024-03-06T07:27:10.154069+0000","last_undegraded":"2024-03-06T07:27:10.154069+0000","last_fullsized":"2024-03-06T07:27:10.154069+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:28:32.938413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.133669+0000","last_change":"2024-03-06T07:27:10.133669+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.133669+0000","last_undegraded":"2024-03-06T07:27:10.133669+0000","last_fullsized":"2024-03-06T07:27:10.133669+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.151911+0000","last_change":"2024-03-06T07:27:10.150171+0000","last_active":"2024-03-06T07:27:10.151911+0000","last_peered":"2024-03-06T07:27:10.151911+0000","last_clean":"2024-03-06T07:27:10.151911+0000","last_became_active":"2024-03-06T07:27:10.149895+0000","last_became_peered":"2024-03-06T07:27:10.149895+0000","last_unstale":"2024-03-06T07:27:10.151911+0000","last_undegraded":"2024-03-06T07:27:10.151911+0000","last_fullsized":"2024-03-06T07:27:10.151911+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:12.816920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.138704+0000","last_change":"2024-03-06T07:27:10.138704+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.138704+0000","last_undegraded":"2024-03-06T07:27:10.138704+0000","last_fullsized":"2024-03-06T07:27:10.138704+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.146334+0000","last_change":"2024-03-06T07:27:10.142952+0000","last_active":"2024-03-06T07:27:10.146334+0000","last_peered":"2024-03-06T07:27:10.146334+0000","last_clean":"2024-03-06T07:27:10.146334+0000","last_became_active":"2024-03-06T07:27:10.142643+0000","last_became_peered":"2024-03-06T07:27:10.142643+0000","last_unstale":"2024-03-06T07:27:10.146334+0000","last_undegraded":"2024-03-06T07:27:10.146334+0000","last_fullsized":"2024-03-06T07:27:10.146334+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:41.083659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.138689+0000","last_change":"2024-03-06T07:27:10.138689+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.138689+0000","last_undegraded":"2024-03-06T07:27:10.138689+0000","last_fullsized":"2024-03-06T07:27:10.138689+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.153932+0000","last_change":"2024-03-06T07:27:10.148289+0000","last_active":"2024-03-06T07:27:10.153932+0000","last_peered":"2024-03-06T07:27:10.153932+0000","last_clean":"2024-03-06T07:27:10.153932+0000","last_became_active":"2024-03-06T07:27:10.148026+0000","last_became_peered":"2024-03-06T07:27:10.148026+0000","last_unstale":"2024-03-06T07:27:10.153932+0000","last_undegraded":"2024-03-06T07:27:10.153932+0000","last_fullsized":"2024-03-06T07:27:10.153932+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:25.955212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.132879+0000","last_change":"2024-03-06T07:27:10.132879+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.132879+0000","last_undegraded":"2024-03-06T07:27:10.132879+0000","last_fullsized":"2024-03-06T07:27:10.132879+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.153944+0000","last_change":"2024-03-06T07:27:10.151397+0000","last_active":"2024-03-06T07:27:10.153944+0000","last_peered":"2024-03-06T07:27:10.153944+0000","last_clean":"2024-03-06T07:27:10.153944+0000","last_became_active":"2024-03-06T07:27:10.150827+0000","last_became_peered":"2024-03-06T07:27:10.150827+0000","last_unstale":"2024-03-06T07:27:10.153944+0000","last_undegraded":"2024-03-06T07:27:10.153944+0000","last_fullsized":"2024-03-06T07:27:10.153944+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:52.397388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.131712+0000","last_change":"2024-03-06T07:27:10.131712+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.131712+0000","last_undegraded":"2024-03-06T07:27:10.131712+0000","last_fullsized":"2024-03-06T07:27:10.131712+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.142168+0000","last_change":"2024-03-06T07:27:10.141228+0000","last_active":"2024-03-06T07:27:10.142168+0000","last_peered":"2024-03-06T07:27:10.142168+0000","last_clean":"2024-03-06T07:27:10.142168+0000","last_became_active":"2024-03-06T07:27:10.140919+0000","last_became_peered":"2024-03-06T07:27:10.140919+0000","last_unstale":"2024-03-06T07:27:10.142168+0000","last_undegraded":"2024-03-06T07:27:10.142168+0000","last_fullsized":"2024-03-06T07:27:10.142168+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:55.019970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.132772+0000","last_change":"2024-03-06T07:27:10.132772+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.132772+0000","last_undegraded":"2024-03-06T07:27:10.132772+0000","last_fullsized":"2024-03-06T07:27:10.132772+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.141015+0000","last_change":"2024-03-06T07:27:10.138499+0000","last_active":"2024-03-06T07:27:10.141015+0000","last_peered":"2024-03-06T07:27:10.141015+0000","last_clean":"2024-03-06T07:27:10.141015+0000","last_became_active":"2024-03-06T07:27:10.138222+0000","last_became_peered":"2024-03-06T07:27:10.138222+0000","last_unstale":"2024-03-06T07:27:10.141015+0000","last_undegraded":"2024-03-06T07:27:10.141015+0000","last_fullsized":"2024-03-06T07:27:10.141015+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:18:48.877918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.136141+0000","last_change":"2024-03-06T07:27:10.136141+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.136141+0000","last_undegraded":"2024-03-06T07:27:10.136141+0000","last_fullsized":"2024-03-06T07:27:10.136141+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.149422+0000","last_change":"2024-03-06T07:27:10.147408+0000","last_active":"2024-03-06T07:27:10.149422+0000","last_peered":"2024-03-06T07:27:10.149422+0000","last_clean":"2024-03-06T07:27:10.149422+0000","last_became_active":"2024-03-06T07:27:10.147159+0000","last_became_peered":"2024-03-06T07:27:10.147159+0000","last_unstale":"2024-03-06T07:27:10.149422+0000","last_undegraded":"2024-03-06T07:27:10.149422+0000","last_fullsized":"2024-03-06T07:27:10.149422+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:28:37.262533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.140866+0000","last_change":"2024-03-06T07:27:10.140866+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.140866+0000","last_undegraded":"2024-03-06T07:27:10.140866+0000","last_fullsized":"2024-03-06T07:27:10.140866+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.153996+0000","last_change":"2024-03-06T07:27:10.149308+0000","last_active":"2024-03-06T07:27:10.153996+0000","last_peered":"2024-03-06T07:27:10.153996+0000","last_clean":"2024-03-06T07:27:10.153996+0000","last_became_active":"2024-03-06T07:27:10.148472+0000","last_became_peered":"2024-03-06T07:27:10.148472+0000","last_unstale":"2024-03-06T07:27:10.153996+0000","last_undegraded":"2024-03-06T07:27:10.153996+0000","last_fullsized":"2024-03-06T07:27:10.153996+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:35:41.626248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.137469+0000","last_change":"2024-03-06T07:27:10.137469+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.137469+0000","last_undegraded":"2024-03-06T07:27:10.137469+0000","last_fullsized":"2024-03-06T07:27:10.137469+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.151732+0000","last_change":"2024-03-06T07:27:10.151075+0000","last_active":"2024-03-06T07:27:10.151732+0000","last_peered":"2024-03-06T07:27:10.151732+0000","last_clean":"2024-03-06T07:27:10.151732+0000","last_became_active":"2024-03-06T07:27:10.150826+0000","last_became_peered":"2024-03-06T07:27:10.150826+0000","last_unstale":"2024-03-06T07:27:10.151732+0000","last_undegraded":"2024-03-06T07:27:10.151732+0000","last_fullsized":"2024-03-06T07:27:10.151732+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:20:12.450522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":327,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.139798+0000","last_change":"2024-03-06T07:21:47.553693+0000","last_active":"2024-03-06T07:27:10.139798+0000","last_peered":"2024-03-06T07:27:10.139798+0000","last_clean":"2024-03-06T07:27:10.139798+0000","last_became_active":"2024-03-06T07:21:47.552664+0000","last_became_peered":"2024-03-06T07:21:47.552664+0000","last_unstale":"2024-03-06T07:27:10.139798+0000","last_undegraded":"2024-03-06T07:27:10.139798+0000","last_fullsized":"2024-03-06T07:27:10.139798+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_clean_scrub_stamp":"2024-03-06T07:21:46.362028+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:02.916426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574918,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37236,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334604,"statfs":{"total":96636764160,"available":96598634496,"internally_reserved":0,"allocated":1937408,"data_stored":1166972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":36174281},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574916,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40800,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331040,"statfs":{"total":96636764160,"available":96594984960,"internally_reserved":0,"allocated":1331200,"data_stored":570329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574917,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40816,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331024,"statfs":{"total":96636764160,"available":96594968576,"internally_reserved":0,"allocated":1343488,"data_stored":576604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574918,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40816,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331024,"statfs":{"total":96636764160,"available":96594968576,"internally_reserved":0,"allocated":1343488,"data_stored":576604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574917,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40800,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331040,"statfs":{"total":96636764160,"available":96594984960,"internally_reserved":0,"allocated":1331200,"data_stored":570329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574917,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40804,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331036,"statfs":{"total":96636764160,"available":96594980864,"internally_reserved":0,"allocated":1331200,"data_stored":570329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574918,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40816,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331024,"statfs":{"total":96636764160,"available":96594968576,"internally_reserved":0,"allocated":1343488,"data_stored":576604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574917,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41384,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330456,"statfs":{"total":96636764160,"available":96594386944,"internally_reserved":0,"allocated":1925120,"data_stored":1160697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:14.166 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:14.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.301+0000 7fc7b4381640 1 Processor -- start 2024-03-06T07:27:14.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.301+0000 7fc7b4381640 1 -- start start 2024-03-06T07:27:14.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.303+0000 7fc7b4381640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc7ac15dfe0 0x7fc7ac0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.303+0000 7fc7b4381640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7ac0b3430 con 0x7fc7ac0d29c0 2024-03-06T07:27:14.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.303+0000 7fc7b4381640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7ac0b3710 con 0x7fc7ac0cf830 2024-03-06T07:27:14.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.303+0000 7fc7b4381640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac0b31e0 con 0x7fc7ac15dfe0 2024-03-06T07:27:14.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.303+0000 7fc7b291e640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fc7ac0d29c0 0x7fc7ac0b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:37632/0 (socket says 172.21.15.83:37632) 2024-03-06T07:27:14.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.303+0000 7fc7b291e640 1 -- 172.21.15.83:0/3467483089 learned_addr learned my addr 172.21.15.83:0/3467483089 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:27:14.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.303+0000 7fc7b211d640 1 --2- 172.21.15.83:0/3467483089 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc7ac15dfe0 0x7fc7ac0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.303+0000 7fc7b111b640 1 -- 172.21.15.83:0/3467483089 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3534470478 0 0) 0x7fc7ac0b3430 con 0x7fc7ac0d29c0 2024-03-06T07:27:14.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.303+0000 7fc7b111b640 1 -- 172.21.15.83:0/3467483089 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc798003520 con 0x7fc7ac0d29c0 2024-03-06T07:27:14.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.303+0000 7fc7b111b640 1 -- 172.21.15.83:0/3467483089 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3058931013 0 0) 0x7fc798003520 con 0x7fc7ac0d29c0 2024-03-06T07:27:14.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.303+0000 7fc7b111b640 1 -- 172.21.15.83:0/3467483089 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc7ac15dfe0 msgr2=0x7fc7ac0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:14.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.303+0000 7fc7b111b640 1 --2- 172.21.15.83:0/3467483089 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc7ac15dfe0 0x7fc7ac0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.303+0000 7fc7b111b640 1 -- 172.21.15.83:0/3467483089 >> v1:172.21.15.183:6789/0 conn(0x7fc7ac0cf830 legacy=0x7fc7ac0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.303+0000 7fc7b111b640 1 -- 172.21.15.83:0/3467483089 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7ac0b39f0 con 0x7fc7ac0d29c0 2024-03-06T07:27:14.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.303+0000 7fc7b111b640 1 -- 172.21.15.83:0/3467483089 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7fc7a40019f0 con 0x7fc7ac0d29c0 2024-03-06T07:27:14.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.303+0000 7fc7b111b640 1 -- 172.21.15.83:0/3467483089 >> v1:172.21.15.83:6789/0 conn(0x7fc7ac0d29c0 legacy=0x7fc7ac0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.303+0000 7fc7b111b640 1 --2- 172.21.15.83:0/3467483089 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7980039e0 0x7fc798003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.303+0000 7fc7b111b640 1 --2- 172.21.15.83:0/3467483089 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc798004390 0x7fc798004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.303+0000 7fc7b111b640 1 --2- 172.21.15.83:0/3467483089 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc7ac15dfe0 0x7fc7980050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.303+0000 7fc7b111b640 1 -- 172.21.15.83:0/3467483089 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac0b31e0 con 0x7fc7980039e0 2024-03-06T07:27:14.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.303+0000 7fc7b111b640 1 -- 172.21.15.83:0/3467483089 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc79800b9c0 con 0x7fc798004390 2024-03-06T07:27:14.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.303+0000 7fc7b111b640 1 -- 172.21.15.83:0/3467483089 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc79800bbc0 con 0x7fc7ac15dfe0 2024-03-06T07:27:14.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.304+0000 7fc7b211d640 1 --2- 172.21.15.83:0/3467483089 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7980039e0 0x7fc798003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.304+0000 7fc7b191c640 1 --2- 172.21.15.83:0/3467483089 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc798004390 0x7fc798004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.304+0000 7fc7b291e640 1 --2- 172.21.15.83:0/3467483089 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc7ac15dfe0 0x7fc7980050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.304+0000 7fc7b211d640 1 -- 172.21.15.83:0/3467483089 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc7ac15dfe0 msgr2=0x7fc7980050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:14.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.304+0000 7fc7b211d640 1 --2- 172.21.15.83:0/3467483089 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc7ac15dfe0 0x7fc7980050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.304+0000 7fc7b211d640 1 -- 172.21.15.83:0/3467483089 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc798004390 msgr2=0x7fc798004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:14.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.304+0000 7fc7b211d640 1 --2- 172.21.15.83:0/3467483089 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc798004390 0x7fc798004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.304+0000 7fc7b211d640 1 -- 172.21.15.83:0/3467483089 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc798003520 con 0x7fc7980039e0 2024-03-06T07:27:14.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.304+0000 7fc7b291e640 1 --2- 172.21.15.83:0/3467483089 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc7ac15dfe0 0x7fc7980050c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:14.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.304+0000 7fc7b191c640 1 --2- 172.21.15.83:0/3467483089 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc798004390 0x7fc798004860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:14.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.305+0000 7fc7b211d640 1 --2- 172.21.15.83:0/3467483089 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7980039e0 0x7fc798003dc0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fc7a8009ac0 tx=0x7fc798005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b668d3dc5ed239 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.305+0000 7fc7b111b640 1 -- 172.21.15.83:0/3467483089 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a8039070 con 0x7fc7980039e0 2024-03-06T07:27:14.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.305+0000 7fc7b111b640 1 -- 172.21.15.83:0/3467483089 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7a802c6b0 con 0x7fc7980039e0 2024-03-06T07:27:14.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.305+0000 7fc7b111b640 1 -- 172.21.15.83:0/3467483089 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a8009690 con 0x7fc7980039e0 2024-03-06T07:27:14.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.305+0000 7fc7b4381640 1 -- 172.21.15.83:0/3467483089 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7980039e0 msgr2=0x7fc798003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.305+0000 7fc7b4381640 1 --2- 172.21.15.83:0/3467483089 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7980039e0 0x7fc798003dc0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fc7a8009ac0 tx=0x7fc798005830 comp rx=0 tx=0).stop 2024-03-06T07:27:14.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.305+0000 7fc7b291e640 1 -- 172.21.15.83:0/3467483089 reap_dead start 2024-03-06T07:27:14.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.305+0000 7fc7b4381640 1 -- 172.21.15.83:0/3467483089 shutdown_connections 2024-03-06T07:27:14.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.305+0000 7fc7b4381640 1 -- 172.21.15.83:0/3467483089 >> 172.21.15.83:0/3467483089 conn(0x7fc7ac0d3cd0 msgr2=0x7fc7ac0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:14.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.305+0000 7fc7b4381640 1 -- 172.21.15.83:0/3467483089 shutdown_connections 2024-03-06T07:27:14.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.305+0000 7fc7b4381640 1 -- 172.21.15.83:0/3467483089 wait complete. 2024-03-06T07:27:14.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.306+0000 7fc7b4381640 1 Processor -- start 2024-03-06T07:27:14.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.307+0000 7fc7b4381640 1 -- start start 2024-03-06T07:27:14.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.307+0000 7fc7b4381640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7980039e0 0x7fc7ac15c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.307+0000 7fc7b4381640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7ac1f7020 0x7fc7ac1f74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.307+0000 7fc7b4381640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc7ac15f3b0 0x7fc7ac15f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.307+0000 7fc7b4381640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac15e3c0 con 0x7fc7ac1f7020 2024-03-06T07:27:14.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.307+0000 7fc7b4381640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac0b3d60 con 0x7fc7980039e0 2024-03-06T07:27:14.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.307+0000 7fc7b4381640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac15e540 con 0x7fc7ac15f3b0 2024-03-06T07:27:14.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.307+0000 7fc7b191c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7ac1f7020 0x7fc7ac1f74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.307+0000 7fc7b191c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7ac1f7020 0x7fc7ac1f74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47688/0 (socket says 172.21.15.83:47688) 2024-03-06T07:27:14.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.307+0000 7fc7b191c640 1 -- 172.21.15.83:0/2943285760 learned_addr learned my addr 172.21.15.83:0/2943285760 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:14.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.307+0000 7fc7b211d640 1 --2- 172.21.15.83:0/2943285760 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7980039e0 0x7fc7ac15c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.307+0000 7fc7b291e640 1 --2- 172.21.15.83:0/2943285760 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc7ac15f3b0 0x7fc7ac15f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.308+0000 7fc7b191c640 1 -- 172.21.15.83:0/2943285760 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc7ac15f3b0 msgr2=0x7fc7ac15f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.308+0000 7fc7b191c640 1 --2- 172.21.15.83:0/2943285760 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc7ac15f3b0 0x7fc7ac15f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.308+0000 7fc7b191c640 1 -- 172.21.15.83:0/2943285760 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7980039e0 msgr2=0x7fc7ac15c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.308+0000 7fc7b191c640 1 --2- 172.21.15.83:0/2943285760 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7980039e0 0x7fc7ac15c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.308+0000 7fc7b191c640 1 -- 172.21.15.83:0/2943285760 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7ac0b68d0 con 0x7fc7ac1f7020 2024-03-06T07:27:14.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.308+0000 7fc7b211d640 1 --2- 172.21.15.83:0/2943285760 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7980039e0 0x7fc7ac15c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:14.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.308+0000 7fc7b191c640 1 --2- 172.21.15.83:0/2943285760 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7ac1f7020 0x7fc7ac1f74a0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7fc79c00e830 tx=0x7fc79c00ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.308+0000 7fc7a2ffd640 1 -- 172.21.15.83:0/2943285760 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79c0050d0 con 0x7fc7ac1f7020 2024-03-06T07:27:14.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.308+0000 7fc7a2ffd640 1 -- 172.21.15.83:0/2943285760 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc79c018070 con 0x7fc7ac1f7020 2024-03-06T07:27:14.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.308+0000 7fc7a2ffd640 1 -- 172.21.15.83:0/2943285760 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79c005470 con 0x7fc7ac1f7020 2024-03-06T07:27:14.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.308+0000 7fc7b4381640 1 -- 172.21.15.83:0/2943285760 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7ac1600e0 con 0x7fc7ac1f7020 2024-03-06T07:27:14.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.308+0000 7fc7b4381640 1 -- 172.21.15.83:0/2943285760 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7ac0b6e50 con 0x7fc7ac1f7020 2024-03-06T07:27:14.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.309+0000 7fc7b4381640 1 -- 172.21.15.83:0/2943285760 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7ac04f330 con 0x7fc7ac1f7020 2024-03-06T07:27:14.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.309+0000 7fc7a2ffd640 1 -- 172.21.15.83:0/2943285760 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc79c00c8d0 con 0x7fc7ac1f7020 2024-03-06T07:27:14.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.310+0000 7fc7a2ffd640 1 --2- 172.21.15.83:0/2943285760 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc79006b500 0x7fc79006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.310+0000 7fc7a2ffd640 1 -- 172.21.15.83:0/2943285760 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc79c08a170 con 0x7fc7ac1f7020 2024-03-06T07:27:14.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.310+0000 7fc7b211d640 1 --2- 172.21.15.83:0/2943285760 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc79006b500 0x7fc79006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.310+0000 7fc7b211d640 1 --2- 172.21.15.83:0/2943285760 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc79006b500 0x7fc79006d9c0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fc7a8009ac0 tx=0x7fc7a8030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.314+0000 7fc7a2ffd640 1 -- 172.21.15.83:0/2943285760 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc79c058ff0 con 0x7fc7ac1f7020 2024-03-06T07:27:14.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.488+0000 7fc7b4381640 1 -- 172.21.15.83:0/2943285760 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc7ac15c7c0 con 0x7fc79006b500 2024-03-06T07:27:14.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.494+0000 7fc7a2ffd640 1 -- 172.21.15.83:0/2943285760 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247089 (secure 0 0 0) 0x7fc7a80040c0 con 0x7fc79006b500 2024-03-06T07:27:14.498 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:14.499 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-06T07:27:14.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.496+0000 7fc7b4381640 1 -- 172.21.15.83:0/2943285760 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc79006b500 msgr2=0x7fc79006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.496+0000 7fc7b4381640 1 --2- 172.21.15.83:0/2943285760 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc79006b500 0x7fc79006d9c0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fc7a8009ac0 tx=0x7fc7a8030000 comp rx=0 tx=0).stop 2024-03-06T07:27:14.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.496+0000 7fc7b4381640 1 -- 172.21.15.83:0/2943285760 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7ac1f7020 msgr2=0x7fc7ac1f74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.496+0000 7fc7b4381640 1 --2- 172.21.15.83:0/2943285760 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7ac1f7020 0x7fc7ac1f74a0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7fc79c00e830 tx=0x7fc79c00ecf0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.496+0000 7fc7b4381640 1 -- 172.21.15.83:0/2943285760 shutdown_connections 2024-03-06T07:27:14.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.496+0000 7fc7b4381640 1 --2- 172.21.15.83:0/2943285760 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc79006b500 0x7fc79006d9c0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.496+0000 7fc7b4381640 1 --2- 172.21.15.83:0/2943285760 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc7ac15f3b0 0x7fc7ac15f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.496+0000 7fc7b4381640 1 --2- 172.21.15.83:0/2943285760 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7ac1f7020 0x7fc7ac1f74a0 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.496+0000 7fc7b4381640 1 --2- 172.21.15.83:0/2943285760 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7980039e0 0x7fc7ac15c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.497+0000 7fc7b4381640 1 -- 172.21.15.83:0/2943285760 >> 172.21.15.83:0/2943285760 conn(0x7fc7ac0d3cd0 msgr2=0x7fc7ac0ced20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:14.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.497+0000 7fc7b4381640 1 -- 172.21.15.83:0/2943285760 shutdown_connections 2024-03-06T07:27:14.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.497+0000 7fc7b4381640 1 -- 172.21.15.83:0/2943285760 wait complete. 2024-03-06T07:27:14.511 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":268,"stamp":"2024-03-06T07:27:14.377541+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":29,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":323472,"kb_used_data":11608,"kb_used_omap":12,"kb_used_meta":311731,"kb_avail":754651248,"statfs":{"total":773094113280,"available":772762877952,"internally_reserved":0,"allocated":11886592,"data_stored":5768468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12757,"internal_metadata":319213099},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.250662"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.143449+0000","last_change":"2024-03-06T07:27:10.143449+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.143449+0000","last_undegraded":"2024-03-06T07:27:10.143449+0000","last_fullsized":"2024-03-06T07:27:10.143449+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.131926+0000","last_change":"2024-03-06T07:27:10.131926+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.131926+0000","last_undegraded":"2024-03-06T07:27:10.131926+0000","last_fullsized":"2024-03-06T07:27:10.131926+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.140106+0000","last_change":"2024-03-06T07:27:10.140106+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.140106+0000","last_undegraded":"2024-03-06T07:27:10.140106+0000","last_fullsized":"2024-03-06T07:27:10.140106+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.141376+0000","last_change":"2024-03-06T07:27:10.141376+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.141376+0000","last_undegraded":"2024-03-06T07:27:10.141376+0000","last_fullsized":"2024-03-06T07:27:10.141376+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.134023+0000","last_change":"2024-03-06T07:27:10.134023+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.134023+0000","last_undegraded":"2024-03-06T07:27:10.134023+0000","last_fullsized":"2024-03-06T07:27:10.134023+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.141567+0000","last_change":"2024-03-06T07:27:10.141567+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.141567+0000","last_undegraded":"2024-03-06T07:27:10.141567+0000","last_fullsized":"2024-03-06T07:27:10.141567+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.135202+0000","last_change":"2024-03-06T07:27:10.135202+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.135202+0000","last_undegraded":"2024-03-06T07:27:10.135202+0000","last_fullsized":"2024-03-06T07:27:10.135202+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.137109+0000","last_change":"2024-03-06T07:27:10.137109+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.137109+0000","last_undegraded":"2024-03-06T07:27:10.137109+0000","last_fullsized":"2024-03-06T07:27:10.137109+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.140884+0000","last_change":"2024-03-06T07:27:10.140884+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.140884+0000","last_undegraded":"2024-03-06T07:27:10.140884+0000","last_fullsized":"2024-03-06T07:27:10.140884+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.140929+0000","last_change":"2024-03-06T07:27:10.140929+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.140929+0000","last_undegraded":"2024-03-06T07:27:10.140929+0000","last_fullsized":"2024-03-06T07:27:10.140929+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.131689+0000","last_change":"2024-03-06T07:27:10.131689+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.131689+0000","last_undegraded":"2024-03-06T07:27:10.131689+0000","last_fullsized":"2024-03-06T07:27:10.131689+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.139813+0000","last_change":"2024-03-06T07:27:10.139813+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.139813+0000","last_undegraded":"2024-03-06T07:27:10.139813+0000","last_fullsized":"2024-03-06T07:27:10.139813+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.136022+0000","last_change":"2024-03-06T07:27:10.136022+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.136022+0000","last_undegraded":"2024-03-06T07:27:10.136022+0000","last_fullsized":"2024-03-06T07:27:10.136022+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.136551+0000","last_change":"2024-03-06T07:27:10.136551+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.136551+0000","last_undegraded":"2024-03-06T07:27:10.136551+0000","last_fullsized":"2024-03-06T07:27:10.136551+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.143562+0000","last_change":"2024-03-06T07:27:10.143562+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.143562+0000","last_undegraded":"2024-03-06T07:27:10.143562+0000","last_fullsized":"2024-03-06T07:27:10.143562+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.141170+0000","last_change":"2024-03-06T07:27:10.141170+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.141170+0000","last_undegraded":"2024-03-06T07:27:10.141170+0000","last_fullsized":"2024-03-06T07:27:10.141170+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.133703+0000","last_change":"2024-03-06T07:27:10.133703+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.133703+0000","last_undegraded":"2024-03-06T07:27:10.133703+0000","last_fullsized":"2024-03-06T07:27:10.133703+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.134084+0000","last_change":"2024-03-06T07:27:10.134084+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.134084+0000","last_undegraded":"2024-03-06T07:27:10.134084+0000","last_fullsized":"2024-03-06T07:27:10.134084+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.139490+0000","last_change":"2024-03-06T07:27:10.139490+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.139490+0000","last_undegraded":"2024-03-06T07:27:10.139490+0000","last_fullsized":"2024-03-06T07:27:10.139490+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.133392+0000","last_change":"2024-03-06T07:27:10.133392+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.133392+0000","last_undegraded":"2024-03-06T07:27:10.133392+0000","last_fullsized":"2024-03-06T07:27:10.133392+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.131525+0000","last_change":"2024-03-06T07:27:10.131525+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.131525+0000","last_undegraded":"2024-03-06T07:27:10.131525+0000","last_fullsized":"2024-03-06T07:27:10.131525+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.138692+0000","last_change":"2024-03-06T07:27:10.138692+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.138692+0000","last_undegraded":"2024-03-06T07:27:10.138692+0000","last_fullsized":"2024-03-06T07:27:10.138692+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.131673+0000","last_change":"2024-03-06T07:27:10.131673+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.131673+0000","last_undegraded":"2024-03-06T07:27:10.131673+0000","last_fullsized":"2024-03-06T07:27:10.131673+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.138495+0000","last_change":"2024-03-06T07:27:10.138495+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.138495+0000","last_undegraded":"2024-03-06T07:27:10.138495+0000","last_fullsized":"2024-03-06T07:27:10.138495+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.132448+0000","last_change":"2024-03-06T07:27:10.132448+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.132448+0000","last_undegraded":"2024-03-06T07:27:10.132448+0000","last_fullsized":"2024-03-06T07:27:10.132448+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.139794+0000","last_change":"2024-03-06T07:27:10.139794+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.139794+0000","last_undegraded":"2024-03-06T07:27:10.139794+0000","last_fullsized":"2024-03-06T07:27:10.139794+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.141443+0000","last_change":"2024-03-06T07:27:10.141443+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.141443+0000","last_undegraded":"2024-03-06T07:27:10.141443+0000","last_fullsized":"2024-03-06T07:27:10.141443+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.139294+0000","last_change":"2024-03-06T07:27:10.139294+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.139294+0000","last_undegraded":"2024-03-06T07:27:10.139294+0000","last_fullsized":"2024-03-06T07:27:10.139294+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.134116+0000","last_change":"2024-03-06T07:27:10.134116+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.134116+0000","last_undegraded":"2024-03-06T07:27:10.134116+0000","last_fullsized":"2024-03-06T07:27:10.134116+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.134159+0000","last_change":"2024-03-06T07:27:10.134159+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.134159+0000","last_undegraded":"2024-03-06T07:27:10.134159+0000","last_fullsized":"2024-03-06T07:27:10.134159+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.141268+0000","last_change":"2024-03-06T07:27:10.141268+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.141268+0000","last_undegraded":"2024-03-06T07:27:10.141268+0000","last_fullsized":"2024-03-06T07:27:10.141268+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.140860+0000","last_change":"2024-03-06T07:27:10.140860+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.140860+0000","last_undegraded":"2024-03-06T07:27:10.140860+0000","last_fullsized":"2024-03-06T07:27:10.140860+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.140859+0000","last_change":"2024-03-06T07:27:10.140859+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.140859+0000","last_undegraded":"2024-03-06T07:27:10.140859+0000","last_fullsized":"2024-03-06T07:27:10.140859+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.144441+0000","last_change":"2024-03-06T07:27:10.143561+0000","last_active":"2024-03-06T07:27:10.144441+0000","last_peered":"2024-03-06T07:27:10.144441+0000","last_clean":"2024-03-06T07:27:10.144441+0000","last_became_active":"2024-03-06T07:27:10.143244+0000","last_became_peered":"2024-03-06T07:27:10.143244+0000","last_unstale":"2024-03-06T07:27:10.144441+0000","last_undegraded":"2024-03-06T07:27:10.144441+0000","last_fullsized":"2024-03-06T07:27:10.144441+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:31:13.932038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.135118+0000","last_change":"2024-03-06T07:27:10.135118+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.135118+0000","last_undegraded":"2024-03-06T07:27:10.135118+0000","last_fullsized":"2024-03-06T07:27:10.135118+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.146696+0000","last_change":"2024-03-06T07:27:10.145366+0000","last_active":"2024-03-06T07:27:10.146696+0000","last_peered":"2024-03-06T07:27:10.146696+0000","last_clean":"2024-03-06T07:27:10.146696+0000","last_became_active":"2024-03-06T07:27:10.145028+0000","last_became_peered":"2024-03-06T07:27:10.145028+0000","last_unstale":"2024-03-06T07:27:10.146696+0000","last_undegraded":"2024-03-06T07:27:10.146696+0000","last_fullsized":"2024-03-06T07:27:10.146696+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:56:36.810364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.141075+0000","last_change":"2024-03-06T07:27:10.141075+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.141075+0000","last_undegraded":"2024-03-06T07:27:10.141075+0000","last_fullsized":"2024-03-06T07:27:10.141075+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.148901+0000","last_change":"2024-03-06T07:27:10.146400+0000","last_active":"2024-03-06T07:27:10.148901+0000","last_peered":"2024-03-06T07:27:10.148901+0000","last_clean":"2024-03-06T07:27:10.148901+0000","last_became_active":"2024-03-06T07:27:10.146106+0000","last_became_peered":"2024-03-06T07:27:10.146106+0000","last_unstale":"2024-03-06T07:27:10.148901+0000","last_undegraded":"2024-03-06T07:27:10.148901+0000","last_fullsized":"2024-03-06T07:27:10.148901+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:24:07.968719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.137861+0000","last_change":"2024-03-06T07:27:10.137861+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.137861+0000","last_undegraded":"2024-03-06T07:27:10.137861+0000","last_fullsized":"2024-03-06T07:27:10.137861+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.140038+0000","last_change":"2024-03-06T07:27:10.137036+0000","last_active":"2024-03-06T07:27:10.140038+0000","last_peered":"2024-03-06T07:27:10.140038+0000","last_clean":"2024-03-06T07:27:10.140038+0000","last_became_active":"2024-03-06T07:27:10.136771+0000","last_became_peered":"2024-03-06T07:27:10.136771+0000","last_unstale":"2024-03-06T07:27:10.140038+0000","last_undegraded":"2024-03-06T07:27:10.140038+0000","last_fullsized":"2024-03-06T07:27:10.140038+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:03:45.041901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.139239+0000","last_change":"2024-03-06T07:27:10.139239+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.139239+0000","last_undegraded":"2024-03-06T07:27:10.139239+0000","last_fullsized":"2024-03-06T07:27:10.139239+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.153078+0000","last_change":"2024-03-06T07:27:10.152619+0000","last_active":"2024-03-06T07:27:10.153078+0000","last_peered":"2024-03-06T07:27:10.153078+0000","last_clean":"2024-03-06T07:27:10.153078+0000","last_became_active":"2024-03-06T07:27:10.152377+0000","last_became_peered":"2024-03-06T07:27:10.152377+0000","last_unstale":"2024-03-06T07:27:10.153078+0000","last_undegraded":"2024-03-06T07:27:10.153078+0000","last_fullsized":"2024-03-06T07:27:10.153078+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:30.226075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.133490+0000","last_change":"2024-03-06T07:27:10.133490+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.133490+0000","last_undegraded":"2024-03-06T07:27:10.133490+0000","last_fullsized":"2024-03-06T07:27:10.133490+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.151980+0000","last_change":"2024-03-06T07:27:10.148748+0000","last_active":"2024-03-06T07:27:10.151980+0000","last_peered":"2024-03-06T07:27:10.151980+0000","last_clean":"2024-03-06T07:27:10.151980+0000","last_became_active":"2024-03-06T07:27:10.148438+0000","last_became_peered":"2024-03-06T07:27:10.148438+0000","last_unstale":"2024-03-06T07:27:10.151980+0000","last_undegraded":"2024-03-06T07:27:10.151980+0000","last_fullsized":"2024-03-06T07:27:10.151980+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:09.417957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.139850+0000","last_change":"2024-03-06T07:27:10.139850+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.139850+0000","last_undegraded":"2024-03-06T07:27:10.139850+0000","last_fullsized":"2024-03-06T07:27:10.139850+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.149200+0000","last_change":"2024-03-06T07:27:10.147680+0000","last_active":"2024-03-06T07:27:10.149200+0000","last_peered":"2024-03-06T07:27:10.149200+0000","last_clean":"2024-03-06T07:27:10.149200+0000","last_became_active":"2024-03-06T07:27:10.147438+0000","last_became_peered":"2024-03-06T07:27:10.147438+0000","last_unstale":"2024-03-06T07:27:10.149200+0000","last_undegraded":"2024-03-06T07:27:10.149200+0000","last_fullsized":"2024-03-06T07:27:10.149200+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:57:44.152110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.135641+0000","last_change":"2024-03-06T07:27:10.135641+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.135641+0000","last_undegraded":"2024-03-06T07:27:10.135641+0000","last_fullsized":"2024-03-06T07:27:10.135641+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.153973+0000","last_change":"2024-03-06T07:27:10.149089+0000","last_active":"2024-03-06T07:27:10.153973+0000","last_peered":"2024-03-06T07:27:10.153973+0000","last_clean":"2024-03-06T07:27:10.153973+0000","last_became_active":"2024-03-06T07:27:10.148126+0000","last_became_peered":"2024-03-06T07:27:10.148126+0000","last_unstale":"2024-03-06T07:27:10.153973+0000","last_undegraded":"2024-03-06T07:27:10.153973+0000","last_fullsized":"2024-03-06T07:27:10.153973+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:23:31.120689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.135927+0000","last_change":"2024-03-06T07:27:10.135927+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.135927+0000","last_undegraded":"2024-03-06T07:27:10.135927+0000","last_fullsized":"2024-03-06T07:27:10.135927+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.149657+0000","last_change":"2024-03-06T07:27:10.146728+0000","last_active":"2024-03-06T07:27:10.149657+0000","last_peered":"2024-03-06T07:27:10.149657+0000","last_clean":"2024-03-06T07:27:10.149657+0000","last_became_active":"2024-03-06T07:27:10.146456+0000","last_became_peered":"2024-03-06T07:27:10.146456+0000","last_unstale":"2024-03-06T07:27:10.149657+0000","last_undegraded":"2024-03-06T07:27:10.149657+0000","last_fullsized":"2024-03-06T07:27:10.149657+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:26.196020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.134728+0000","last_change":"2024-03-06T07:27:10.134728+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.134728+0000","last_undegraded":"2024-03-06T07:27:10.134728+0000","last_fullsized":"2024-03-06T07:27:10.134728+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.151549+0000","last_change":"2024-03-06T07:27:10.148013+0000","last_active":"2024-03-06T07:27:10.151549+0000","last_peered":"2024-03-06T07:27:10.151549+0000","last_clean":"2024-03-06T07:27:10.151549+0000","last_became_active":"2024-03-06T07:27:10.147723+0000","last_became_peered":"2024-03-06T07:27:10.147723+0000","last_unstale":"2024-03-06T07:27:10.151549+0000","last_undegraded":"2024-03-06T07:27:10.151549+0000","last_fullsized":"2024-03-06T07:27:10.151549+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:41.083659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.138899+0000","last_change":"2024-03-06T07:27:10.138899+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.138899+0000","last_undegraded":"2024-03-06T07:27:10.138899+0000","last_fullsized":"2024-03-06T07:27:10.138899+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.151643+0000","last_change":"2024-03-06T07:27:10.145307+0000","last_active":"2024-03-06T07:27:10.151643+0000","last_peered":"2024-03-06T07:27:10.151643+0000","last_clean":"2024-03-06T07:27:10.151643+0000","last_became_active":"2024-03-06T07:27:10.145027+0000","last_became_peered":"2024-03-06T07:27:10.145027+0000","last_unstale":"2024-03-06T07:27:10.151643+0000","last_undegraded":"2024-03-06T07:27:10.151643+0000","last_fullsized":"2024-03-06T07:27:10.151643+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:49:40.057171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.134203+0000","last_change":"2024-03-06T07:27:10.134203+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.134203+0000","last_undegraded":"2024-03-06T07:27:10.134203+0000","last_fullsized":"2024-03-06T07:27:10.134203+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.154023+0000","last_change":"2024-03-06T07:27:10.147579+0000","last_active":"2024-03-06T07:27:10.154023+0000","last_peered":"2024-03-06T07:27:10.154023+0000","last_clean":"2024-03-06T07:27:10.154023+0000","last_became_active":"2024-03-06T07:27:10.147292+0000","last_became_peered":"2024-03-06T07:27:10.147292+0000","last_unstale":"2024-03-06T07:27:10.154023+0000","last_undegraded":"2024-03-06T07:27:10.154023+0000","last_fullsized":"2024-03-06T07:27:10.154023+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:38.297655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.138207+0000","last_change":"2024-03-06T07:27:10.138207+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.138207+0000","last_undegraded":"2024-03-06T07:27:10.138207+0000","last_fullsized":"2024-03-06T07:27:10.138207+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.152149+0000","last_change":"2024-03-06T07:27:10.145448+0000","last_active":"2024-03-06T07:27:10.152149+0000","last_peered":"2024-03-06T07:27:10.152149+0000","last_clean":"2024-03-06T07:27:10.152149+0000","last_became_active":"2024-03-06T07:27:10.145063+0000","last_became_peered":"2024-03-06T07:27:10.145063+0000","last_unstale":"2024-03-06T07:27:10.152149+0000","last_undegraded":"2024-03-06T07:27:10.152149+0000","last_fullsized":"2024-03-06T07:27:10.152149+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:14:01.327671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.137691+0000","last_change":"2024-03-06T07:27:10.137691+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.137691+0000","last_undegraded":"2024-03-06T07:27:10.137691+0000","last_fullsized":"2024-03-06T07:27:10.137691+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.149341+0000","last_change":"2024-03-06T07:27:10.146708+0000","last_active":"2024-03-06T07:27:10.149341+0000","last_peered":"2024-03-06T07:27:10.149341+0000","last_clean":"2024-03-06T07:27:10.149341+0000","last_became_active":"2024-03-06T07:27:10.146387+0000","last_became_peered":"2024-03-06T07:27:10.146387+0000","last_unstale":"2024-03-06T07:27:10.149341+0000","last_undegraded":"2024-03-06T07:27:10.149341+0000","last_fullsized":"2024-03-06T07:27:10.149341+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:12.816920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.139264+0000","last_change":"2024-03-06T07:27:10.139264+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.139264+0000","last_undegraded":"2024-03-06T07:27:10.139264+0000","last_fullsized":"2024-03-06T07:27:10.139264+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.139998+0000","last_change":"2024-03-06T07:27:10.136377+0000","last_active":"2024-03-06T07:27:10.139998+0000","last_peered":"2024-03-06T07:27:10.139998+0000","last_clean":"2024-03-06T07:27:10.139998+0000","last_became_active":"2024-03-06T07:27:10.136062+0000","last_became_peered":"2024-03-06T07:27:10.136062+0000","last_unstale":"2024-03-06T07:27:10.139998+0000","last_undegraded":"2024-03-06T07:27:10.139998+0000","last_fullsized":"2024-03-06T07:27:10.139998+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:44:20.435658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.133922+0000","last_change":"2024-03-06T07:27:10.133922+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.133922+0000","last_undegraded":"2024-03-06T07:27:10.133922+0000","last_fullsized":"2024-03-06T07:27:10.133922+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.153881+0000","last_change":"2024-03-06T07:27:10.147575+0000","last_active":"2024-03-06T07:27:10.153881+0000","last_peered":"2024-03-06T07:27:10.153881+0000","last_clean":"2024-03-06T07:27:10.153881+0000","last_became_active":"2024-03-06T07:27:10.147304+0000","last_became_peered":"2024-03-06T07:27:10.147304+0000","last_unstale":"2024-03-06T07:27:10.153881+0000","last_undegraded":"2024-03-06T07:27:10.153881+0000","last_fullsized":"2024-03-06T07:27:10.153881+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:19:26.310561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.140347+0000","last_change":"2024-03-06T07:27:10.140347+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.140347+0000","last_undegraded":"2024-03-06T07:27:10.140347+0000","last_fullsized":"2024-03-06T07:27:10.140347+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.140158+0000","last_change":"2024-03-06T07:27:10.137380+0000","last_active":"2024-03-06T07:27:10.140158+0000","last_peered":"2024-03-06T07:27:10.140158+0000","last_clean":"2024-03-06T07:27:10.140158+0000","last_became_active":"2024-03-06T07:27:10.137056+0000","last_became_peered":"2024-03-06T07:27:10.137056+0000","last_unstale":"2024-03-06T07:27:10.140158+0000","last_undegraded":"2024-03-06T07:27:10.140158+0000","last_fullsized":"2024-03-06T07:27:10.140158+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:12:04.974749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.135554+0000","last_change":"2024-03-06T07:27:10.135554+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.135554+0000","last_undegraded":"2024-03-06T07:27:10.135554+0000","last_fullsized":"2024-03-06T07:27:10.135554+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.146355+0000","last_change":"2024-03-06T07:27:10.145731+0000","last_active":"2024-03-06T07:27:10.146355+0000","last_peered":"2024-03-06T07:27:10.146355+0000","last_clean":"2024-03-06T07:27:10.146355+0000","last_became_active":"2024-03-06T07:27:10.145424+0000","last_became_peered":"2024-03-06T07:27:10.145424+0000","last_unstale":"2024-03-06T07:27:10.146355+0000","last_undegraded":"2024-03-06T07:27:10.146355+0000","last_fullsized":"2024-03-06T07:27:10.146355+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:09.417957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.137107+0000","last_change":"2024-03-06T07:27:10.137107+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.137107+0000","last_undegraded":"2024-03-06T07:27:10.137107+0000","last_fullsized":"2024-03-06T07:27:10.137107+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.151758+0000","last_change":"2024-03-06T07:27:10.151277+0000","last_active":"2024-03-06T07:27:10.151758+0000","last_peered":"2024-03-06T07:27:10.151758+0000","last_clean":"2024-03-06T07:27:10.151758+0000","last_became_active":"2024-03-06T07:27:10.150933+0000","last_became_peered":"2024-03-06T07:27:10.150933+0000","last_unstale":"2024-03-06T07:27:10.151758+0000","last_undegraded":"2024-03-06T07:27:10.151758+0000","last_fullsized":"2024-03-06T07:27:10.151758+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:16.105130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.135497+0000","last_change":"2024-03-06T07:27:10.135497+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.135497+0000","last_undegraded":"2024-03-06T07:27:10.135497+0000","last_fullsized":"2024-03-06T07:27:10.135497+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.154056+0000","last_change":"2024-03-06T07:27:10.150893+0000","last_active":"2024-03-06T07:27:10.154056+0000","last_peered":"2024-03-06T07:27:10.154056+0000","last_clean":"2024-03-06T07:27:10.154056+0000","last_became_active":"2024-03-06T07:27:10.150648+0000","last_became_peered":"2024-03-06T07:27:10.150648+0000","last_unstale":"2024-03-06T07:27:10.154056+0000","last_undegraded":"2024-03-06T07:27:10.154056+0000","last_fullsized":"2024-03-06T07:27:10.154056+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:34.387756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.139184+0000","last_change":"2024-03-06T07:27:10.139184+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.139184+0000","last_undegraded":"2024-03-06T07:27:10.139184+0000","last_fullsized":"2024-03-06T07:27:10.139184+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.151835+0000","last_change":"2024-03-06T07:27:10.148882+0000","last_active":"2024-03-06T07:27:10.151835+0000","last_peered":"2024-03-06T07:27:10.151835+0000","last_clean":"2024-03-06T07:27:10.151835+0000","last_became_active":"2024-03-06T07:27:10.148626+0000","last_became_peered":"2024-03-06T07:27:10.148626+0000","last_unstale":"2024-03-06T07:27:10.151835+0000","last_undegraded":"2024-03-06T07:27:10.151835+0000","last_fullsized":"2024-03-06T07:27:10.151835+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:19.034867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.136666+0000","last_change":"2024-03-06T07:27:10.136666+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.136666+0000","last_undegraded":"2024-03-06T07:27:10.136666+0000","last_fullsized":"2024-03-06T07:27:10.136666+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.150333+0000","last_change":"2024-03-06T07:27:10.144592+0000","last_active":"2024-03-06T07:27:10.150333+0000","last_peered":"2024-03-06T07:27:10.150333+0000","last_clean":"2024-03-06T07:27:10.150333+0000","last_became_active":"2024-03-06T07:27:10.144328+0000","last_became_peered":"2024-03-06T07:27:10.144328+0000","last_unstale":"2024-03-06T07:27:10.150333+0000","last_undegraded":"2024-03-06T07:27:10.150333+0000","last_fullsized":"2024-03-06T07:27:10.150333+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:39:04.350267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.138259+0000","last_change":"2024-03-06T07:27:10.138259+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.138259+0000","last_undegraded":"2024-03-06T07:27:10.138259+0000","last_fullsized":"2024-03-06T07:27:10.138259+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.154069+0000","last_change":"2024-03-06T07:27:10.151020+0000","last_active":"2024-03-06T07:27:10.154069+0000","last_peered":"2024-03-06T07:27:10.154069+0000","last_clean":"2024-03-06T07:27:10.154069+0000","last_became_active":"2024-03-06T07:27:10.150747+0000","last_became_peered":"2024-03-06T07:27:10.150747+0000","last_unstale":"2024-03-06T07:27:10.154069+0000","last_undegraded":"2024-03-06T07:27:10.154069+0000","last_fullsized":"2024-03-06T07:27:10.154069+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:28:32.938413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.133669+0000","last_change":"2024-03-06T07:27:10.133669+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.133669+0000","last_undegraded":"2024-03-06T07:27:10.133669+0000","last_fullsized":"2024-03-06T07:27:10.133669+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.151911+0000","last_change":"2024-03-06T07:27:10.150171+0000","last_active":"2024-03-06T07:27:10.151911+0000","last_peered":"2024-03-06T07:27:10.151911+0000","last_clean":"2024-03-06T07:27:10.151911+0000","last_became_active":"2024-03-06T07:27:10.149895+0000","last_became_peered":"2024-03-06T07:27:10.149895+0000","last_unstale":"2024-03-06T07:27:10.151911+0000","last_undegraded":"2024-03-06T07:27:10.151911+0000","last_fullsized":"2024-03-06T07:27:10.151911+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:12.816920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.138704+0000","last_change":"2024-03-06T07:27:10.138704+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.138704+0000","last_undegraded":"2024-03-06T07:27:10.138704+0000","last_fullsized":"2024-03-06T07:27:10.138704+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.146334+0000","last_change":"2024-03-06T07:27:10.142952+0000","last_active":"2024-03-06T07:27:10.146334+0000","last_peered":"2024-03-06T07:27:10.146334+0000","last_clean":"2024-03-06T07:27:10.146334+0000","last_became_active":"2024-03-06T07:27:10.142643+0000","last_became_peered":"2024-03-06T07:27:10.142643+0000","last_unstale":"2024-03-06T07:27:10.146334+0000","last_undegraded":"2024-03-06T07:27:10.146334+0000","last_fullsized":"2024-03-06T07:27:10.146334+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:41.083659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.138689+0000","last_change":"2024-03-06T07:27:10.138689+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.138689+0000","last_undegraded":"2024-03-06T07:27:10.138689+0000","last_fullsized":"2024-03-06T07:27:10.138689+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.153932+0000","last_change":"2024-03-06T07:27:10.148289+0000","last_active":"2024-03-06T07:27:10.153932+0000","last_peered":"2024-03-06T07:27:10.153932+0000","last_clean":"2024-03-06T07:27:10.153932+0000","last_became_active":"2024-03-06T07:27:10.148026+0000","last_became_peered":"2024-03-06T07:27:10.148026+0000","last_unstale":"2024-03-06T07:27:10.153932+0000","last_undegraded":"2024-03-06T07:27:10.153932+0000","last_fullsized":"2024-03-06T07:27:10.153932+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:25.955212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.132879+0000","last_change":"2024-03-06T07:27:10.132879+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.132879+0000","last_undegraded":"2024-03-06T07:27:10.132879+0000","last_fullsized":"2024-03-06T07:27:10.132879+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.153944+0000","last_change":"2024-03-06T07:27:10.151397+0000","last_active":"2024-03-06T07:27:10.153944+0000","last_peered":"2024-03-06T07:27:10.153944+0000","last_clean":"2024-03-06T07:27:10.153944+0000","last_became_active":"2024-03-06T07:27:10.150827+0000","last_became_peered":"2024-03-06T07:27:10.150827+0000","last_unstale":"2024-03-06T07:27:10.153944+0000","last_undegraded":"2024-03-06T07:27:10.153944+0000","last_fullsized":"2024-03-06T07:27:10.153944+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:52.397388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.131712+0000","last_change":"2024-03-06T07:27:10.131712+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.131712+0000","last_undegraded":"2024-03-06T07:27:10.131712+0000","last_fullsized":"2024-03-06T07:27:10.131712+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.142168+0000","last_change":"2024-03-06T07:27:10.141228+0000","last_active":"2024-03-06T07:27:10.142168+0000","last_peered":"2024-03-06T07:27:10.142168+0000","last_clean":"2024-03-06T07:27:10.142168+0000","last_became_active":"2024-03-06T07:27:10.140919+0000","last_became_peered":"2024-03-06T07:27:10.140919+0000","last_unstale":"2024-03-06T07:27:10.142168+0000","last_undegraded":"2024-03-06T07:27:10.142168+0000","last_fullsized":"2024-03-06T07:27:10.142168+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:55.019970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.132772+0000","last_change":"2024-03-06T07:27:10.132772+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.132772+0000","last_undegraded":"2024-03-06T07:27:10.132772+0000","last_fullsized":"2024-03-06T07:27:10.132772+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.141015+0000","last_change":"2024-03-06T07:27:10.138499+0000","last_active":"2024-03-06T07:27:10.141015+0000","last_peered":"2024-03-06T07:27:10.141015+0000","last_clean":"2024-03-06T07:27:10.141015+0000","last_became_active":"2024-03-06T07:27:10.138222+0000","last_became_peered":"2024-03-06T07:27:10.138222+0000","last_unstale":"2024-03-06T07:27:10.141015+0000","last_undegraded":"2024-03-06T07:27:10.141015+0000","last_fullsized":"2024-03-06T07:27:10.141015+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:18:48.877918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.136141+0000","last_change":"2024-03-06T07:27:10.136141+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.136141+0000","last_undegraded":"2024-03-06T07:27:10.136141+0000","last_fullsized":"2024-03-06T07:27:10.136141+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.149422+0000","last_change":"2024-03-06T07:27:10.147408+0000","last_active":"2024-03-06T07:27:10.149422+0000","last_peered":"2024-03-06T07:27:10.149422+0000","last_clean":"2024-03-06T07:27:10.149422+0000","last_became_active":"2024-03-06T07:27:10.147159+0000","last_became_peered":"2024-03-06T07:27:10.147159+0000","last_unstale":"2024-03-06T07:27:10.149422+0000","last_undegraded":"2024-03-06T07:27:10.149422+0000","last_fullsized":"2024-03-06T07:27:10.149422+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:28:37.262533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.140866+0000","last_change":"2024-03-06T07:27:10.140866+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.140866+0000","last_undegraded":"2024-03-06T07:27:10.140866+0000","last_fullsized":"2024-03-06T07:27:10.140866+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.153996+0000","last_change":"2024-03-06T07:27:10.149308+0000","last_active":"2024-03-06T07:27:10.153996+0000","last_peered":"2024-03-06T07:27:10.153996+0000","last_clean":"2024-03-06T07:27:10.153996+0000","last_became_active":"2024-03-06T07:27:10.148472+0000","last_became_peered":"2024-03-06T07:27:10.148472+0000","last_unstale":"2024-03-06T07:27:10.153996+0000","last_undegraded":"2024-03-06T07:27:10.153996+0000","last_fullsized":"2024-03-06T07:27:10.153996+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:35:41.626248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.137469+0000","last_change":"2024-03-06T07:27:10.137469+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.137469+0000","last_undegraded":"2024-03-06T07:27:10.137469+0000","last_fullsized":"2024-03-06T07:27:10.137469+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.151732+0000","last_change":"2024-03-06T07:27:10.151075+0000","last_active":"2024-03-06T07:27:10.151732+0000","last_peered":"2024-03-06T07:27:10.151732+0000","last_clean":"2024-03-06T07:27:10.151732+0000","last_became_active":"2024-03-06T07:27:10.150826+0000","last_became_peered":"2024-03-06T07:27:10.150826+0000","last_unstale":"2024-03-06T07:27:10.151732+0000","last_undegraded":"2024-03-06T07:27:10.151732+0000","last_fullsized":"2024-03-06T07:27:10.151732+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:20:12.450522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":327,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.139798+0000","last_change":"2024-03-06T07:21:47.553693+0000","last_active":"2024-03-06T07:27:10.139798+0000","last_peered":"2024-03-06T07:27:10.139798+0000","last_clean":"2024-03-06T07:27:10.139798+0000","last_became_active":"2024-03-06T07:21:47.552664+0000","last_became_peered":"2024-03-06T07:21:47.552664+0000","last_unstale":"2024-03-06T07:27:10.139798+0000","last_undegraded":"2024-03-06T07:27:10.139798+0000","last_fullsized":"2024-03-06T07:27:10.139798+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_clean_scrub_stamp":"2024-03-06T07:21:46.362028+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:02.916426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574918,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37236,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334604,"statfs":{"total":96636764160,"available":96598634496,"internally_reserved":0,"allocated":1937408,"data_stored":1166972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":36174281},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574916,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40800,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331040,"statfs":{"total":96636764160,"available":96594984960,"internally_reserved":0,"allocated":1331200,"data_stored":570329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574917,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40816,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331024,"statfs":{"total":96636764160,"available":96594968576,"internally_reserved":0,"allocated":1343488,"data_stored":576604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574918,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40816,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331024,"statfs":{"total":96636764160,"available":96594968576,"internally_reserved":0,"allocated":1343488,"data_stored":576604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574917,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40800,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331040,"statfs":{"total":96636764160,"available":96594984960,"internally_reserved":0,"allocated":1331200,"data_stored":570329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574917,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40804,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331036,"statfs":{"total":96636764160,"available":96594980864,"internally_reserved":0,"allocated":1331200,"data_stored":570329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574918,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40816,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331024,"statfs":{"total":96636764160,"available":96594968576,"internally_reserved":0,"allocated":1343488,"data_stored":576604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574917,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41384,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330456,"statfs":{"total":96636764160,"available":96594386944,"internally_reserved":0,"allocated":1925120,"data_stored":1160697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:14.515 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:14.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.650+0000 7f0427eed640 1 Processor -- start 2024-03-06T07:27:14.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.650+0000 7f0427eed640 1 -- start start 2024-03-06T07:27:14.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.651+0000 7f0427eed640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f04200d22b0 0x7f04200d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.651+0000 7f0427eed640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f04200b41f0 0x7f04200b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.651+0000 7f0427eed640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04200b3430 con 0x7f042015dfe0 2024-03-06T07:27:14.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.651+0000 7f0427eed640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04200d2c50 con 0x7f04200d22b0 2024-03-06T07:27:14.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.651+0000 7f0427eed640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04200b31e0 con 0x7f04200b41f0 2024-03-06T07:27:14.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.651+0000 7f0425488640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f04200d22b0 0x7f04200d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.651+0000 7f0425488640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f04200d22b0 0x7f04200d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47696/0 (socket says 172.21.15.83:47696) 2024-03-06T07:27:14.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.651+0000 7f0425488640 1 -- 172.21.15.83:0/2120369621 learned_addr learned my addr 172.21.15.83:0/2120369621 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:14.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.651+0000 7f042648a640 1 --2- 172.21.15.83:0/2120369621 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f04200b41f0 0x7f04200b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.651+0000 7f0425488640 1 -- 172.21.15.83:0/2120369621 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f04200b41f0 msgr2=0x7f04200b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:14.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.651+0000 7f0425488640 1 --2- 172.21.15.83:0/2120369621 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f04200b41f0 0x7f04200b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.651+0000 7f0425488640 1 -- 172.21.15.83:0/2120369621 >> v1:172.21.15.183:6789/0 conn(0x7f042015dfe0 legacy=0x7f04200cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.651+0000 7f0425488640 1 -- 172.21.15.83:0/2120369621 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04200b3710 con 0x7f04200d22b0 2024-03-06T07:27:14.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.652+0000 7f0425488640 1 --2- 172.21.15.83:0/2120369621 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f04200d22b0 0x7f04200d2710 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f0410009870 tx=0x7f041002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3505fdfb64b3d6e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.652+0000 7f0424c87640 1 -- 172.21.15.83:0/2120369621 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0410034070 con 0x7f04200d22b0 2024-03-06T07:27:14.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.652+0000 7f0424c87640 1 -- 172.21.15.83:0/2120369621 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0410030840 con 0x7f04200d22b0 2024-03-06T07:27:14.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.652+0000 7f0424c87640 1 -- 172.21.15.83:0/2120369621 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0410030c10 con 0x7f04200d22b0 2024-03-06T07:27:14.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.652+0000 7f0427eed640 1 -- 172.21.15.83:0/2120369621 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f04200d22b0 msgr2=0x7f04200d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.652+0000 7f0427eed640 1 --2- 172.21.15.83:0/2120369621 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f04200d22b0 0x7f04200d2710 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f0410009870 tx=0x7f041002b980 comp rx=0 tx=0).stop 2024-03-06T07:27:14.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.652+0000 7f0427eed640 1 -- 172.21.15.83:0/2120369621 shutdown_connections 2024-03-06T07:27:14.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.652+0000 7f0427eed640 1 --2- 172.21.15.83:0/2120369621 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f04200b41f0 0x7f04200b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.652+0000 7f0427eed640 1 --2- 172.21.15.83:0/2120369621 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f04200d22b0 0x7f04200d2710 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.652+0000 7f0427eed640 1 -- 172.21.15.83:0/2120369621 >> 172.21.15.83:0/2120369621 conn(0x7f04200d3cd0 msgr2=0x7f04200d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:14.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.653+0000 7f0427eed640 1 -- 172.21.15.83:0/2120369621 shutdown_connections 2024-03-06T07:27:14.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.653+0000 7f0427eed640 1 -- 172.21.15.83:0/2120369621 wait complete. 2024-03-06T07:27:14.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.654+0000 7f0427eed640 1 Processor -- start 2024-03-06T07:27:14.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.654+0000 7f0427eed640 1 -- start start 2024-03-06T07:27:14.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.654+0000 7f0427eed640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f04200b41f0 0x7f04201510b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.654+0000 7f0427eed640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f04200d22b0 0x7f04200cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.654+0000 7f0427eed640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f042015dfe0 0x7f04200b6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.654+0000 7f0427eed640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04200cfa00 con 0x7f04200d22b0 2024-03-06T07:27:14.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.654+0000 7f0427eed640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04200b3d60 con 0x7f04200b41f0 2024-03-06T07:27:14.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.654+0000 7f0427eed640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04200d1c70 con 0x7f042015dfe0 2024-03-06T07:27:14.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.654+0000 7f0425488640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f04200d22b0 0x7f04200cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.654+0000 7f0425488640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f04200d22b0 0x7f04200cccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47708/0 (socket says 172.21.15.83:47708) 2024-03-06T07:27:14.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.654+0000 7f0425488640 1 -- 172.21.15.83:0/246173702 learned_addr learned my addr 172.21.15.83:0/246173702 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:14.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.654+0000 7f0425c89640 1 --2- 172.21.15.83:0/246173702 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f04200b41f0 0x7f04201510b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.655+0000 7f042648a640 1 --2- 172.21.15.83:0/246173702 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f042015dfe0 0x7f04200b6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.655+0000 7f0425488640 1 -- 172.21.15.83:0/246173702 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f042015dfe0 msgr2=0x7f04200b6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.655+0000 7f0425488640 1 --2- 172.21.15.83:0/246173702 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f042015dfe0 0x7f04200b6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.655+0000 7f0425488640 1 -- 172.21.15.83:0/246173702 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f04200b41f0 msgr2=0x7f04201510b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.655+0000 7f0425488640 1 --2- 172.21.15.83:0/246173702 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f04200b41f0 0x7f04201510b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.655+0000 7f0425488640 1 -- 172.21.15.83:0/246173702 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04200cd5f0 con 0x7f04200d22b0 2024-03-06T07:27:14.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.655+0000 7f0425488640 1 --2- 172.21.15.83:0/246173702 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f04200d22b0 0x7f04200cccd0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f0410000c00 tx=0x7f0410004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.655+0000 7f04167fc640 1 -- 172.21.15.83:0/246173702 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0410048070 con 0x7f04200d22b0 2024-03-06T07:27:14.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.655+0000 7f04167fc640 1 -- 172.21.15.83:0/246173702 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0410030640 con 0x7f04200d22b0 2024-03-06T07:27:14.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.655+0000 7f0427eed640 1 -- 172.21.15.83:0/246173702 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f042015f1d0 con 0x7f04200d22b0 2024-03-06T07:27:14.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.655+0000 7f04167fc640 1 -- 172.21.15.83:0/246173702 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0410034040 con 0x7f04200d22b0 2024-03-06T07:27:14.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.655+0000 7f0427eed640 1 -- 172.21.15.83:0/246173702 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04201515f0 con 0x7f04200d22b0 2024-03-06T07:27:14.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.656+0000 7f0427eed640 1 -- 172.21.15.83:0/246173702 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04200d3740 con 0x7f04200d22b0 2024-03-06T07:27:14.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.657+0000 7f04167fc640 1 -- 172.21.15.83:0/246173702 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f041004b070 con 0x7f04200d22b0 2024-03-06T07:27:14.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.657+0000 7f04167fc640 1 --2- 172.21.15.83:0/246173702 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f040406b4d0 0x7f040406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.657+0000 7f04167fc640 1 -- 172.21.15.83:0/246173702 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f04100abf90 con 0x7f04200d22b0 2024-03-06T07:27:14.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.657+0000 7f0425c89640 1 --2- 172.21.15.83:0/246173702 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f040406b4d0 0x7f040406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.657+0000 7f0425c89640 1 --2- 172.21.15.83:0/246173702 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f040406b4d0 0x7f040406d990 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f041c0016e0 tx=0x7f041c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.661+0000 7f04167fc640 1 -- 172.21.15.83:0/246173702 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f04200d3740 con 0x7f04200d22b0 2024-03-06T07:27:14.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.836+0000 7f0427eed640 1 -- 172.21.15.83:0/246173702 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f042004f330 con 0x7f040406b4d0 2024-03-06T07:27:14.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.841+0000 7f04167fc640 1 -- 172.21.15.83:0/246173702 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247089 (secure 0 0 0) 0x7f042004f330 con 0x7f040406b4d0 2024-03-06T07:27:14.846 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:14.847 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-06T07:27:14.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.844+0000 7f0427eed640 1 -- 172.21.15.83:0/246173702 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f040406b4d0 msgr2=0x7f040406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.844+0000 7f0427eed640 1 --2- 172.21.15.83:0/246173702 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f040406b4d0 0x7f040406d990 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f041c0016e0 tx=0x7f041c009040 comp rx=0 tx=0).stop 2024-03-06T07:27:14.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.844+0000 7f0427eed640 1 -- 172.21.15.83:0/246173702 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f04200d22b0 msgr2=0x7f04200cccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.844+0000 7f0427eed640 1 --2- 172.21.15.83:0/246173702 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f04200d22b0 0x7f04200cccd0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f0410000c00 tx=0x7f0410004730 comp rx=0 tx=0).stop 2024-03-06T07:27:14.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.844+0000 7f0427eed640 1 -- 172.21.15.83:0/246173702 shutdown_connections 2024-03-06T07:27:14.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.844+0000 7f0427eed640 1 --2- 172.21.15.83:0/246173702 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f040406b4d0 0x7f040406d990 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.844+0000 7f0427eed640 1 --2- 172.21.15.83:0/246173702 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f042015dfe0 0x7f04200b6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.844+0000 7f0427eed640 1 --2- 172.21.15.83:0/246173702 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f04200d22b0 0x7f04200cccd0 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.844+0000 7f0427eed640 1 --2- 172.21.15.83:0/246173702 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f04200b41f0 0x7f04201510b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.844+0000 7f0427eed640 1 -- 172.21.15.83:0/246173702 >> 172.21.15.83:0/246173702 conn(0x7f04200d3cd0 msgr2=0x7f04200cff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:14.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.844+0000 7f0427eed640 1 -- 172.21.15.83:0/246173702 shutdown_connections 2024-03-06T07:27:14.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.844+0000 7f0427eed640 1 -- 172.21.15.83:0/246173702 wait complete. 2024-03-06T07:27:14.857 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":268,"stamp":"2024-03-06T07:27:14.377541+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":29,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":323472,"kb_used_data":11608,"kb_used_omap":12,"kb_used_meta":311731,"kb_avail":754651248,"statfs":{"total":773094113280,"available":772762877952,"internally_reserved":0,"allocated":11886592,"data_stored":5768468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12757,"internal_metadata":319213099},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.250662"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.143449+0000","last_change":"2024-03-06T07:27:10.143449+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.143449+0000","last_undegraded":"2024-03-06T07:27:10.143449+0000","last_fullsized":"2024-03-06T07:27:10.143449+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.131926+0000","last_change":"2024-03-06T07:27:10.131926+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.131926+0000","last_undegraded":"2024-03-06T07:27:10.131926+0000","last_fullsized":"2024-03-06T07:27:10.131926+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.140106+0000","last_change":"2024-03-06T07:27:10.140106+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.140106+0000","last_undegraded":"2024-03-06T07:27:10.140106+0000","last_fullsized":"2024-03-06T07:27:10.140106+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.141376+0000","last_change":"2024-03-06T07:27:10.141376+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.141376+0000","last_undegraded":"2024-03-06T07:27:10.141376+0000","last_fullsized":"2024-03-06T07:27:10.141376+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.134023+0000","last_change":"2024-03-06T07:27:10.134023+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.134023+0000","last_undegraded":"2024-03-06T07:27:10.134023+0000","last_fullsized":"2024-03-06T07:27:10.134023+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.141567+0000","last_change":"2024-03-06T07:27:10.141567+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.141567+0000","last_undegraded":"2024-03-06T07:27:10.141567+0000","last_fullsized":"2024-03-06T07:27:10.141567+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.135202+0000","last_change":"2024-03-06T07:27:10.135202+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.135202+0000","last_undegraded":"2024-03-06T07:27:10.135202+0000","last_fullsized":"2024-03-06T07:27:10.135202+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.137109+0000","last_change":"2024-03-06T07:27:10.137109+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.137109+0000","last_undegraded":"2024-03-06T07:27:10.137109+0000","last_fullsized":"2024-03-06T07:27:10.137109+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.140884+0000","last_change":"2024-03-06T07:27:10.140884+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.140884+0000","last_undegraded":"2024-03-06T07:27:10.140884+0000","last_fullsized":"2024-03-06T07:27:10.140884+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.140929+0000","last_change":"2024-03-06T07:27:10.140929+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.140929+0000","last_undegraded":"2024-03-06T07:27:10.140929+0000","last_fullsized":"2024-03-06T07:27:10.140929+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.131689+0000","last_change":"2024-03-06T07:27:10.131689+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.131689+0000","last_undegraded":"2024-03-06T07:27:10.131689+0000","last_fullsized":"2024-03-06T07:27:10.131689+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.139813+0000","last_change":"2024-03-06T07:27:10.139813+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.139813+0000","last_undegraded":"2024-03-06T07:27:10.139813+0000","last_fullsized":"2024-03-06T07:27:10.139813+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.136022+0000","last_change":"2024-03-06T07:27:10.136022+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.136022+0000","last_undegraded":"2024-03-06T07:27:10.136022+0000","last_fullsized":"2024-03-06T07:27:10.136022+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.136551+0000","last_change":"2024-03-06T07:27:10.136551+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.136551+0000","last_undegraded":"2024-03-06T07:27:10.136551+0000","last_fullsized":"2024-03-06T07:27:10.136551+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.143562+0000","last_change":"2024-03-06T07:27:10.143562+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.143562+0000","last_undegraded":"2024-03-06T07:27:10.143562+0000","last_fullsized":"2024-03-06T07:27:10.143562+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.141170+0000","last_change":"2024-03-06T07:27:10.141170+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.141170+0000","last_undegraded":"2024-03-06T07:27:10.141170+0000","last_fullsized":"2024-03-06T07:27:10.141170+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.133703+0000","last_change":"2024-03-06T07:27:10.133703+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.133703+0000","last_undegraded":"2024-03-06T07:27:10.133703+0000","last_fullsized":"2024-03-06T07:27:10.133703+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.134084+0000","last_change":"2024-03-06T07:27:10.134084+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.134084+0000","last_undegraded":"2024-03-06T07:27:10.134084+0000","last_fullsized":"2024-03-06T07:27:10.134084+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.139490+0000","last_change":"2024-03-06T07:27:10.139490+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.139490+0000","last_undegraded":"2024-03-06T07:27:10.139490+0000","last_fullsized":"2024-03-06T07:27:10.139490+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.133392+0000","last_change":"2024-03-06T07:27:10.133392+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.133392+0000","last_undegraded":"2024-03-06T07:27:10.133392+0000","last_fullsized":"2024-03-06T07:27:10.133392+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.131525+0000","last_change":"2024-03-06T07:27:10.131525+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.131525+0000","last_undegraded":"2024-03-06T07:27:10.131525+0000","last_fullsized":"2024-03-06T07:27:10.131525+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.138692+0000","last_change":"2024-03-06T07:27:10.138692+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.138692+0000","last_undegraded":"2024-03-06T07:27:10.138692+0000","last_fullsized":"2024-03-06T07:27:10.138692+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.131673+0000","last_change":"2024-03-06T07:27:10.131673+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.131673+0000","last_undegraded":"2024-03-06T07:27:10.131673+0000","last_fullsized":"2024-03-06T07:27:10.131673+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.138495+0000","last_change":"2024-03-06T07:27:10.138495+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.138495+0000","last_undegraded":"2024-03-06T07:27:10.138495+0000","last_fullsized":"2024-03-06T07:27:10.138495+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.132448+0000","last_change":"2024-03-06T07:27:10.132448+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.132448+0000","last_undegraded":"2024-03-06T07:27:10.132448+0000","last_fullsized":"2024-03-06T07:27:10.132448+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.139794+0000","last_change":"2024-03-06T07:27:10.139794+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.139794+0000","last_undegraded":"2024-03-06T07:27:10.139794+0000","last_fullsized":"2024-03-06T07:27:10.139794+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.141443+0000","last_change":"2024-03-06T07:27:10.141443+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.141443+0000","last_undegraded":"2024-03-06T07:27:10.141443+0000","last_fullsized":"2024-03-06T07:27:10.141443+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.139294+0000","last_change":"2024-03-06T07:27:10.139294+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.139294+0000","last_undegraded":"2024-03-06T07:27:10.139294+0000","last_fullsized":"2024-03-06T07:27:10.139294+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.134116+0000","last_change":"2024-03-06T07:27:10.134116+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.134116+0000","last_undegraded":"2024-03-06T07:27:10.134116+0000","last_fullsized":"2024-03-06T07:27:10.134116+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.134159+0000","last_change":"2024-03-06T07:27:10.134159+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.134159+0000","last_undegraded":"2024-03-06T07:27:10.134159+0000","last_fullsized":"2024-03-06T07:27:10.134159+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.141268+0000","last_change":"2024-03-06T07:27:10.141268+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.141268+0000","last_undegraded":"2024-03-06T07:27:10.141268+0000","last_fullsized":"2024-03-06T07:27:10.141268+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.140860+0000","last_change":"2024-03-06T07:27:10.140860+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.140860+0000","last_undegraded":"2024-03-06T07:27:10.140860+0000","last_fullsized":"2024-03-06T07:27:10.140860+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.140859+0000","last_change":"2024-03-06T07:27:10.140859+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.140859+0000","last_undegraded":"2024-03-06T07:27:10.140859+0000","last_fullsized":"2024-03-06T07:27:10.140859+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.144441+0000","last_change":"2024-03-06T07:27:10.143561+0000","last_active":"2024-03-06T07:27:10.144441+0000","last_peered":"2024-03-06T07:27:10.144441+0000","last_clean":"2024-03-06T07:27:10.144441+0000","last_became_active":"2024-03-06T07:27:10.143244+0000","last_became_peered":"2024-03-06T07:27:10.143244+0000","last_unstale":"2024-03-06T07:27:10.144441+0000","last_undegraded":"2024-03-06T07:27:10.144441+0000","last_fullsized":"2024-03-06T07:27:10.144441+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:31:13.932038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.135118+0000","last_change":"2024-03-06T07:27:10.135118+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.135118+0000","last_undegraded":"2024-03-06T07:27:10.135118+0000","last_fullsized":"2024-03-06T07:27:10.135118+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.146696+0000","last_change":"2024-03-06T07:27:10.145366+0000","last_active":"2024-03-06T07:27:10.146696+0000","last_peered":"2024-03-06T07:27:10.146696+0000","last_clean":"2024-03-06T07:27:10.146696+0000","last_became_active":"2024-03-06T07:27:10.145028+0000","last_became_peered":"2024-03-06T07:27:10.145028+0000","last_unstale":"2024-03-06T07:27:10.146696+0000","last_undegraded":"2024-03-06T07:27:10.146696+0000","last_fullsized":"2024-03-06T07:27:10.146696+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:56:36.810364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.141075+0000","last_change":"2024-03-06T07:27:10.141075+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.141075+0000","last_undegraded":"2024-03-06T07:27:10.141075+0000","last_fullsized":"2024-03-06T07:27:10.141075+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.148901+0000","last_change":"2024-03-06T07:27:10.146400+0000","last_active":"2024-03-06T07:27:10.148901+0000","last_peered":"2024-03-06T07:27:10.148901+0000","last_clean":"2024-03-06T07:27:10.148901+0000","last_became_active":"2024-03-06T07:27:10.146106+0000","last_became_peered":"2024-03-06T07:27:10.146106+0000","last_unstale":"2024-03-06T07:27:10.148901+0000","last_undegraded":"2024-03-06T07:27:10.148901+0000","last_fullsized":"2024-03-06T07:27:10.148901+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:24:07.968719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.137861+0000","last_change":"2024-03-06T07:27:10.137861+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.137861+0000","last_undegraded":"2024-03-06T07:27:10.137861+0000","last_fullsized":"2024-03-06T07:27:10.137861+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.140038+0000","last_change":"2024-03-06T07:27:10.137036+0000","last_active":"2024-03-06T07:27:10.140038+0000","last_peered":"2024-03-06T07:27:10.140038+0000","last_clean":"2024-03-06T07:27:10.140038+0000","last_became_active":"2024-03-06T07:27:10.136771+0000","last_became_peered":"2024-03-06T07:27:10.136771+0000","last_unstale":"2024-03-06T07:27:10.140038+0000","last_undegraded":"2024-03-06T07:27:10.140038+0000","last_fullsized":"2024-03-06T07:27:10.140038+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:03:45.041901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.139239+0000","last_change":"2024-03-06T07:27:10.139239+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.139239+0000","last_undegraded":"2024-03-06T07:27:10.139239+0000","last_fullsized":"2024-03-06T07:27:10.139239+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.153078+0000","last_change":"2024-03-06T07:27:10.152619+0000","last_active":"2024-03-06T07:27:10.153078+0000","last_peered":"2024-03-06T07:27:10.153078+0000","last_clean":"2024-03-06T07:27:10.153078+0000","last_became_active":"2024-03-06T07:27:10.152377+0000","last_became_peered":"2024-03-06T07:27:10.152377+0000","last_unstale":"2024-03-06T07:27:10.153078+0000","last_undegraded":"2024-03-06T07:27:10.153078+0000","last_fullsized":"2024-03-06T07:27:10.153078+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:30.226075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.133490+0000","last_change":"2024-03-06T07:27:10.133490+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.133490+0000","last_undegraded":"2024-03-06T07:27:10.133490+0000","last_fullsized":"2024-03-06T07:27:10.133490+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.151980+0000","last_change":"2024-03-06T07:27:10.148748+0000","last_active":"2024-03-06T07:27:10.151980+0000","last_peered":"2024-03-06T07:27:10.151980+0000","last_clean":"2024-03-06T07:27:10.151980+0000","last_became_active":"2024-03-06T07:27:10.148438+0000","last_became_peered":"2024-03-06T07:27:10.148438+0000","last_unstale":"2024-03-06T07:27:10.151980+0000","last_undegraded":"2024-03-06T07:27:10.151980+0000","last_fullsized":"2024-03-06T07:27:10.151980+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:09.417957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.139850+0000","last_change":"2024-03-06T07:27:10.139850+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.139850+0000","last_undegraded":"2024-03-06T07:27:10.139850+0000","last_fullsized":"2024-03-06T07:27:10.139850+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.149200+0000","last_change":"2024-03-06T07:27:10.147680+0000","last_active":"2024-03-06T07:27:10.149200+0000","last_peered":"2024-03-06T07:27:10.149200+0000","last_clean":"2024-03-06T07:27:10.149200+0000","last_became_active":"2024-03-06T07:27:10.147438+0000","last_became_peered":"2024-03-06T07:27:10.147438+0000","last_unstale":"2024-03-06T07:27:10.149200+0000","last_undegraded":"2024-03-06T07:27:10.149200+0000","last_fullsized":"2024-03-06T07:27:10.149200+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:57:44.152110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.135641+0000","last_change":"2024-03-06T07:27:10.135641+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.135641+0000","last_undegraded":"2024-03-06T07:27:10.135641+0000","last_fullsized":"2024-03-06T07:27:10.135641+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.153973+0000","last_change":"2024-03-06T07:27:10.149089+0000","last_active":"2024-03-06T07:27:10.153973+0000","last_peered":"2024-03-06T07:27:10.153973+0000","last_clean":"2024-03-06T07:27:10.153973+0000","last_became_active":"2024-03-06T07:27:10.148126+0000","last_became_peered":"2024-03-06T07:27:10.148126+0000","last_unstale":"2024-03-06T07:27:10.153973+0000","last_undegraded":"2024-03-06T07:27:10.153973+0000","last_fullsized":"2024-03-06T07:27:10.153973+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:23:31.120689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.135927+0000","last_change":"2024-03-06T07:27:10.135927+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.135927+0000","last_undegraded":"2024-03-06T07:27:10.135927+0000","last_fullsized":"2024-03-06T07:27:10.135927+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.149657+0000","last_change":"2024-03-06T07:27:10.146728+0000","last_active":"2024-03-06T07:27:10.149657+0000","last_peered":"2024-03-06T07:27:10.149657+0000","last_clean":"2024-03-06T07:27:10.149657+0000","last_became_active":"2024-03-06T07:27:10.146456+0000","last_became_peered":"2024-03-06T07:27:10.146456+0000","last_unstale":"2024-03-06T07:27:10.149657+0000","last_undegraded":"2024-03-06T07:27:10.149657+0000","last_fullsized":"2024-03-06T07:27:10.149657+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:26.196020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.134728+0000","last_change":"2024-03-06T07:27:10.134728+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.134728+0000","last_undegraded":"2024-03-06T07:27:10.134728+0000","last_fullsized":"2024-03-06T07:27:10.134728+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.151549+0000","last_change":"2024-03-06T07:27:10.148013+0000","last_active":"2024-03-06T07:27:10.151549+0000","last_peered":"2024-03-06T07:27:10.151549+0000","last_clean":"2024-03-06T07:27:10.151549+0000","last_became_active":"2024-03-06T07:27:10.147723+0000","last_became_peered":"2024-03-06T07:27:10.147723+0000","last_unstale":"2024-03-06T07:27:10.151549+0000","last_undegraded":"2024-03-06T07:27:10.151549+0000","last_fullsized":"2024-03-06T07:27:10.151549+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:41.083659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.138899+0000","last_change":"2024-03-06T07:27:10.138899+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.138899+0000","last_undegraded":"2024-03-06T07:27:10.138899+0000","last_fullsized":"2024-03-06T07:27:10.138899+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.151643+0000","last_change":"2024-03-06T07:27:10.145307+0000","last_active":"2024-03-06T07:27:10.151643+0000","last_peered":"2024-03-06T07:27:10.151643+0000","last_clean":"2024-03-06T07:27:10.151643+0000","last_became_active":"2024-03-06T07:27:10.145027+0000","last_became_peered":"2024-03-06T07:27:10.145027+0000","last_unstale":"2024-03-06T07:27:10.151643+0000","last_undegraded":"2024-03-06T07:27:10.151643+0000","last_fullsized":"2024-03-06T07:27:10.151643+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:49:40.057171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.134203+0000","last_change":"2024-03-06T07:27:10.134203+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.134203+0000","last_undegraded":"2024-03-06T07:27:10.134203+0000","last_fullsized":"2024-03-06T07:27:10.134203+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.154023+0000","last_change":"2024-03-06T07:27:10.147579+0000","last_active":"2024-03-06T07:27:10.154023+0000","last_peered":"2024-03-06T07:27:10.154023+0000","last_clean":"2024-03-06T07:27:10.154023+0000","last_became_active":"2024-03-06T07:27:10.147292+0000","last_became_peered":"2024-03-06T07:27:10.147292+0000","last_unstale":"2024-03-06T07:27:10.154023+0000","last_undegraded":"2024-03-06T07:27:10.154023+0000","last_fullsized":"2024-03-06T07:27:10.154023+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:38.297655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.138207+0000","last_change":"2024-03-06T07:27:10.138207+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.138207+0000","last_undegraded":"2024-03-06T07:27:10.138207+0000","last_fullsized":"2024-03-06T07:27:10.138207+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.152149+0000","last_change":"2024-03-06T07:27:10.145448+0000","last_active":"2024-03-06T07:27:10.152149+0000","last_peered":"2024-03-06T07:27:10.152149+0000","last_clean":"2024-03-06T07:27:10.152149+0000","last_became_active":"2024-03-06T07:27:10.145063+0000","last_became_peered":"2024-03-06T07:27:10.145063+0000","last_unstale":"2024-03-06T07:27:10.152149+0000","last_undegraded":"2024-03-06T07:27:10.152149+0000","last_fullsized":"2024-03-06T07:27:10.152149+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:14:01.327671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.137691+0000","last_change":"2024-03-06T07:27:10.137691+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.137691+0000","last_undegraded":"2024-03-06T07:27:10.137691+0000","last_fullsized":"2024-03-06T07:27:10.137691+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.149341+0000","last_change":"2024-03-06T07:27:10.146708+0000","last_active":"2024-03-06T07:27:10.149341+0000","last_peered":"2024-03-06T07:27:10.149341+0000","last_clean":"2024-03-06T07:27:10.149341+0000","last_became_active":"2024-03-06T07:27:10.146387+0000","last_became_peered":"2024-03-06T07:27:10.146387+0000","last_unstale":"2024-03-06T07:27:10.149341+0000","last_undegraded":"2024-03-06T07:27:10.149341+0000","last_fullsized":"2024-03-06T07:27:10.149341+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:12.816920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.139264+0000","last_change":"2024-03-06T07:27:10.139264+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.139264+0000","last_undegraded":"2024-03-06T07:27:10.139264+0000","last_fullsized":"2024-03-06T07:27:10.139264+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.139998+0000","last_change":"2024-03-06T07:27:10.136377+0000","last_active":"2024-03-06T07:27:10.139998+0000","last_peered":"2024-03-06T07:27:10.139998+0000","last_clean":"2024-03-06T07:27:10.139998+0000","last_became_active":"2024-03-06T07:27:10.136062+0000","last_became_peered":"2024-03-06T07:27:10.136062+0000","last_unstale":"2024-03-06T07:27:10.139998+0000","last_undegraded":"2024-03-06T07:27:10.139998+0000","last_fullsized":"2024-03-06T07:27:10.139998+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:44:20.435658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.133922+0000","last_change":"2024-03-06T07:27:10.133922+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.133922+0000","last_undegraded":"2024-03-06T07:27:10.133922+0000","last_fullsized":"2024-03-06T07:27:10.133922+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.153881+0000","last_change":"2024-03-06T07:27:10.147575+0000","last_active":"2024-03-06T07:27:10.153881+0000","last_peered":"2024-03-06T07:27:10.153881+0000","last_clean":"2024-03-06T07:27:10.153881+0000","last_became_active":"2024-03-06T07:27:10.147304+0000","last_became_peered":"2024-03-06T07:27:10.147304+0000","last_unstale":"2024-03-06T07:27:10.153881+0000","last_undegraded":"2024-03-06T07:27:10.153881+0000","last_fullsized":"2024-03-06T07:27:10.153881+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:19:26.310561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.140347+0000","last_change":"2024-03-06T07:27:10.140347+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.140347+0000","last_undegraded":"2024-03-06T07:27:10.140347+0000","last_fullsized":"2024-03-06T07:27:10.140347+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.140158+0000","last_change":"2024-03-06T07:27:10.137380+0000","last_active":"2024-03-06T07:27:10.140158+0000","last_peered":"2024-03-06T07:27:10.140158+0000","last_clean":"2024-03-06T07:27:10.140158+0000","last_became_active":"2024-03-06T07:27:10.137056+0000","last_became_peered":"2024-03-06T07:27:10.137056+0000","last_unstale":"2024-03-06T07:27:10.140158+0000","last_undegraded":"2024-03-06T07:27:10.140158+0000","last_fullsized":"2024-03-06T07:27:10.140158+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:12:04.974749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.135554+0000","last_change":"2024-03-06T07:27:10.135554+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.135554+0000","last_undegraded":"2024-03-06T07:27:10.135554+0000","last_fullsized":"2024-03-06T07:27:10.135554+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.146355+0000","last_change":"2024-03-06T07:27:10.145731+0000","last_active":"2024-03-06T07:27:10.146355+0000","last_peered":"2024-03-06T07:27:10.146355+0000","last_clean":"2024-03-06T07:27:10.146355+0000","last_became_active":"2024-03-06T07:27:10.145424+0000","last_became_peered":"2024-03-06T07:27:10.145424+0000","last_unstale":"2024-03-06T07:27:10.146355+0000","last_undegraded":"2024-03-06T07:27:10.146355+0000","last_fullsized":"2024-03-06T07:27:10.146355+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:09.417957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.137107+0000","last_change":"2024-03-06T07:27:10.137107+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.137107+0000","last_undegraded":"2024-03-06T07:27:10.137107+0000","last_fullsized":"2024-03-06T07:27:10.137107+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.151758+0000","last_change":"2024-03-06T07:27:10.151277+0000","last_active":"2024-03-06T07:27:10.151758+0000","last_peered":"2024-03-06T07:27:10.151758+0000","last_clean":"2024-03-06T07:27:10.151758+0000","last_became_active":"2024-03-06T07:27:10.150933+0000","last_became_peered":"2024-03-06T07:27:10.150933+0000","last_unstale":"2024-03-06T07:27:10.151758+0000","last_undegraded":"2024-03-06T07:27:10.151758+0000","last_fullsized":"2024-03-06T07:27:10.151758+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:16.105130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.135497+0000","last_change":"2024-03-06T07:27:10.135497+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.135497+0000","last_undegraded":"2024-03-06T07:27:10.135497+0000","last_fullsized":"2024-03-06T07:27:10.135497+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.154056+0000","last_change":"2024-03-06T07:27:10.150893+0000","last_active":"2024-03-06T07:27:10.154056+0000","last_peered":"2024-03-06T07:27:10.154056+0000","last_clean":"2024-03-06T07:27:10.154056+0000","last_became_active":"2024-03-06T07:27:10.150648+0000","last_became_peered":"2024-03-06T07:27:10.150648+0000","last_unstale":"2024-03-06T07:27:10.154056+0000","last_undegraded":"2024-03-06T07:27:10.154056+0000","last_fullsized":"2024-03-06T07:27:10.154056+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:34.387756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.139184+0000","last_change":"2024-03-06T07:27:10.139184+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.139184+0000","last_undegraded":"2024-03-06T07:27:10.139184+0000","last_fullsized":"2024-03-06T07:27:10.139184+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.151835+0000","last_change":"2024-03-06T07:27:10.148882+0000","last_active":"2024-03-06T07:27:10.151835+0000","last_peered":"2024-03-06T07:27:10.151835+0000","last_clean":"2024-03-06T07:27:10.151835+0000","last_became_active":"2024-03-06T07:27:10.148626+0000","last_became_peered":"2024-03-06T07:27:10.148626+0000","last_unstale":"2024-03-06T07:27:10.151835+0000","last_undegraded":"2024-03-06T07:27:10.151835+0000","last_fullsized":"2024-03-06T07:27:10.151835+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:19.034867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.136666+0000","last_change":"2024-03-06T07:27:10.136666+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.136666+0000","last_undegraded":"2024-03-06T07:27:10.136666+0000","last_fullsized":"2024-03-06T07:27:10.136666+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.150333+0000","last_change":"2024-03-06T07:27:10.144592+0000","last_active":"2024-03-06T07:27:10.150333+0000","last_peered":"2024-03-06T07:27:10.150333+0000","last_clean":"2024-03-06T07:27:10.150333+0000","last_became_active":"2024-03-06T07:27:10.144328+0000","last_became_peered":"2024-03-06T07:27:10.144328+0000","last_unstale":"2024-03-06T07:27:10.150333+0000","last_undegraded":"2024-03-06T07:27:10.150333+0000","last_fullsized":"2024-03-06T07:27:10.150333+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:39:04.350267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.138259+0000","last_change":"2024-03-06T07:27:10.138259+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.138259+0000","last_undegraded":"2024-03-06T07:27:10.138259+0000","last_fullsized":"2024-03-06T07:27:10.138259+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.154069+0000","last_change":"2024-03-06T07:27:10.151020+0000","last_active":"2024-03-06T07:27:10.154069+0000","last_peered":"2024-03-06T07:27:10.154069+0000","last_clean":"2024-03-06T07:27:10.154069+0000","last_became_active":"2024-03-06T07:27:10.150747+0000","last_became_peered":"2024-03-06T07:27:10.150747+0000","last_unstale":"2024-03-06T07:27:10.154069+0000","last_undegraded":"2024-03-06T07:27:10.154069+0000","last_fullsized":"2024-03-06T07:27:10.154069+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:28:32.938413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.133669+0000","last_change":"2024-03-06T07:27:10.133669+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.133669+0000","last_undegraded":"2024-03-06T07:27:10.133669+0000","last_fullsized":"2024-03-06T07:27:10.133669+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.151911+0000","last_change":"2024-03-06T07:27:10.150171+0000","last_active":"2024-03-06T07:27:10.151911+0000","last_peered":"2024-03-06T07:27:10.151911+0000","last_clean":"2024-03-06T07:27:10.151911+0000","last_became_active":"2024-03-06T07:27:10.149895+0000","last_became_peered":"2024-03-06T07:27:10.149895+0000","last_unstale":"2024-03-06T07:27:10.151911+0000","last_undegraded":"2024-03-06T07:27:10.151911+0000","last_fullsized":"2024-03-06T07:27:10.151911+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:12.816920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.138704+0000","last_change":"2024-03-06T07:27:10.138704+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.138704+0000","last_undegraded":"2024-03-06T07:27:10.138704+0000","last_fullsized":"2024-03-06T07:27:10.138704+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.146334+0000","last_change":"2024-03-06T07:27:10.142952+0000","last_active":"2024-03-06T07:27:10.146334+0000","last_peered":"2024-03-06T07:27:10.146334+0000","last_clean":"2024-03-06T07:27:10.146334+0000","last_became_active":"2024-03-06T07:27:10.142643+0000","last_became_peered":"2024-03-06T07:27:10.142643+0000","last_unstale":"2024-03-06T07:27:10.146334+0000","last_undegraded":"2024-03-06T07:27:10.146334+0000","last_fullsized":"2024-03-06T07:27:10.146334+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:41.083659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.138689+0000","last_change":"2024-03-06T07:27:10.138689+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.138689+0000","last_undegraded":"2024-03-06T07:27:10.138689+0000","last_fullsized":"2024-03-06T07:27:10.138689+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.153932+0000","last_change":"2024-03-06T07:27:10.148289+0000","last_active":"2024-03-06T07:27:10.153932+0000","last_peered":"2024-03-06T07:27:10.153932+0000","last_clean":"2024-03-06T07:27:10.153932+0000","last_became_active":"2024-03-06T07:27:10.148026+0000","last_became_peered":"2024-03-06T07:27:10.148026+0000","last_unstale":"2024-03-06T07:27:10.153932+0000","last_undegraded":"2024-03-06T07:27:10.153932+0000","last_fullsized":"2024-03-06T07:27:10.153932+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:25.955212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.132879+0000","last_change":"2024-03-06T07:27:10.132879+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.132879+0000","last_undegraded":"2024-03-06T07:27:10.132879+0000","last_fullsized":"2024-03-06T07:27:10.132879+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.153944+0000","last_change":"2024-03-06T07:27:10.151397+0000","last_active":"2024-03-06T07:27:10.153944+0000","last_peered":"2024-03-06T07:27:10.153944+0000","last_clean":"2024-03-06T07:27:10.153944+0000","last_became_active":"2024-03-06T07:27:10.150827+0000","last_became_peered":"2024-03-06T07:27:10.150827+0000","last_unstale":"2024-03-06T07:27:10.153944+0000","last_undegraded":"2024-03-06T07:27:10.153944+0000","last_fullsized":"2024-03-06T07:27:10.153944+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:52.397388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.131712+0000","last_change":"2024-03-06T07:27:10.131712+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.131712+0000","last_undegraded":"2024-03-06T07:27:10.131712+0000","last_fullsized":"2024-03-06T07:27:10.131712+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.142168+0000","last_change":"2024-03-06T07:27:10.141228+0000","last_active":"2024-03-06T07:27:10.142168+0000","last_peered":"2024-03-06T07:27:10.142168+0000","last_clean":"2024-03-06T07:27:10.142168+0000","last_became_active":"2024-03-06T07:27:10.140919+0000","last_became_peered":"2024-03-06T07:27:10.140919+0000","last_unstale":"2024-03-06T07:27:10.142168+0000","last_undegraded":"2024-03-06T07:27:10.142168+0000","last_fullsized":"2024-03-06T07:27:10.142168+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:55.019970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.132772+0000","last_change":"2024-03-06T07:27:10.132772+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.132772+0000","last_undegraded":"2024-03-06T07:27:10.132772+0000","last_fullsized":"2024-03-06T07:27:10.132772+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.141015+0000","last_change":"2024-03-06T07:27:10.138499+0000","last_active":"2024-03-06T07:27:10.141015+0000","last_peered":"2024-03-06T07:27:10.141015+0000","last_clean":"2024-03-06T07:27:10.141015+0000","last_became_active":"2024-03-06T07:27:10.138222+0000","last_became_peered":"2024-03-06T07:27:10.138222+0000","last_unstale":"2024-03-06T07:27:10.141015+0000","last_undegraded":"2024-03-06T07:27:10.141015+0000","last_fullsized":"2024-03-06T07:27:10.141015+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:18:48.877918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.136141+0000","last_change":"2024-03-06T07:27:10.136141+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.136141+0000","last_undegraded":"2024-03-06T07:27:10.136141+0000","last_fullsized":"2024-03-06T07:27:10.136141+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.149422+0000","last_change":"2024-03-06T07:27:10.147408+0000","last_active":"2024-03-06T07:27:10.149422+0000","last_peered":"2024-03-06T07:27:10.149422+0000","last_clean":"2024-03-06T07:27:10.149422+0000","last_became_active":"2024-03-06T07:27:10.147159+0000","last_became_peered":"2024-03-06T07:27:10.147159+0000","last_unstale":"2024-03-06T07:27:10.149422+0000","last_undegraded":"2024-03-06T07:27:10.149422+0000","last_fullsized":"2024-03-06T07:27:10.149422+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:28:37.262533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.140866+0000","last_change":"2024-03-06T07:27:10.140866+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.140866+0000","last_undegraded":"2024-03-06T07:27:10.140866+0000","last_fullsized":"2024-03-06T07:27:10.140866+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.153996+0000","last_change":"2024-03-06T07:27:10.149308+0000","last_active":"2024-03-06T07:27:10.153996+0000","last_peered":"2024-03-06T07:27:10.153996+0000","last_clean":"2024-03-06T07:27:10.153996+0000","last_became_active":"2024-03-06T07:27:10.148472+0000","last_became_peered":"2024-03-06T07:27:10.148472+0000","last_unstale":"2024-03-06T07:27:10.153996+0000","last_undegraded":"2024-03-06T07:27:10.153996+0000","last_fullsized":"2024-03-06T07:27:10.153996+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:35:41.626248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.137469+0000","last_change":"2024-03-06T07:27:10.137469+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.137469+0000","last_undegraded":"2024-03-06T07:27:10.137469+0000","last_fullsized":"2024-03-06T07:27:10.137469+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.151732+0000","last_change":"2024-03-06T07:27:10.151075+0000","last_active":"2024-03-06T07:27:10.151732+0000","last_peered":"2024-03-06T07:27:10.151732+0000","last_clean":"2024-03-06T07:27:10.151732+0000","last_became_active":"2024-03-06T07:27:10.150826+0000","last_became_peered":"2024-03-06T07:27:10.150826+0000","last_unstale":"2024-03-06T07:27:10.151732+0000","last_undegraded":"2024-03-06T07:27:10.151732+0000","last_fullsized":"2024-03-06T07:27:10.151732+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:20:12.450522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":327,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.139798+0000","last_change":"2024-03-06T07:21:47.553693+0000","last_active":"2024-03-06T07:27:10.139798+0000","last_peered":"2024-03-06T07:27:10.139798+0000","last_clean":"2024-03-06T07:27:10.139798+0000","last_became_active":"2024-03-06T07:21:47.552664+0000","last_became_peered":"2024-03-06T07:21:47.552664+0000","last_unstale":"2024-03-06T07:27:10.139798+0000","last_undegraded":"2024-03-06T07:27:10.139798+0000","last_fullsized":"2024-03-06T07:27:10.139798+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_clean_scrub_stamp":"2024-03-06T07:21:46.362028+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:02.916426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574918,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37236,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334604,"statfs":{"total":96636764160,"available":96598634496,"internally_reserved":0,"allocated":1937408,"data_stored":1166972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":36174281},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574916,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40800,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331040,"statfs":{"total":96636764160,"available":96594984960,"internally_reserved":0,"allocated":1331200,"data_stored":570329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574917,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40816,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331024,"statfs":{"total":96636764160,"available":96594968576,"internally_reserved":0,"allocated":1343488,"data_stored":576604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574918,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40816,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331024,"statfs":{"total":96636764160,"available":96594968576,"internally_reserved":0,"allocated":1343488,"data_stored":576604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574917,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40800,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331040,"statfs":{"total":96636764160,"available":96594984960,"internally_reserved":0,"allocated":1331200,"data_stored":570329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574917,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40804,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331036,"statfs":{"total":96636764160,"available":96594980864,"internally_reserved":0,"allocated":1331200,"data_stored":570329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574918,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40816,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331024,"statfs":{"total":96636764160,"available":96594968576,"internally_reserved":0,"allocated":1343488,"data_stored":576604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574917,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41384,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330456,"statfs":{"total":96636764160,"available":96594386944,"internally_reserved":0,"allocated":1925120,"data_stored":1160697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:14.861 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.3f is not active+clean 2024-03-06T07:27:14.861 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.143449+0000', 'last_change': '2024-03-06T07:27:10.143449+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.143449+0000', 'last_undegraded': '2024-03-06T07:27:10.143449+0000', 'last_fullsized': '2024-03-06T07:27:10.143449+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:14.861 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.3e is not active+clean 2024-03-06T07:27:14.861 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.131926+0000', 'last_change': '2024-03-06T07:27:10.131926+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.131926+0000', 'last_undegraded': '2024-03-06T07:27:10.131926+0000', 'last_fullsized': '2024-03-06T07:27:10.131926+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:14.861 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.3d is not active+clean 2024-03-06T07:27:14.861 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.140106+0000', 'last_change': '2024-03-06T07:27:10.140106+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.140106+0000', 'last_undegraded': '2024-03-06T07:27:10.140106+0000', 'last_fullsized': '2024-03-06T07:27:10.140106+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:14.861 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.3c is not active+clean 2024-03-06T07:27:14.862 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.141376+0000', 'last_change': '2024-03-06T07:27:10.141376+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.141376+0000', 'last_undegraded': '2024-03-06T07:27:10.141376+0000', 'last_fullsized': '2024-03-06T07:27:10.141376+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:14.862 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.3b is not active+clean 2024-03-06T07:27:14.862 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.134023+0000', 'last_change': '2024-03-06T07:27:10.134023+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.134023+0000', 'last_undegraded': '2024-03-06T07:27:10.134023+0000', 'last_fullsized': '2024-03-06T07:27:10.134023+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:27:14.862 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.3a is not active+clean 2024-03-06T07:27:14.862 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.141567+0000', 'last_change': '2024-03-06T07:27:10.141567+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.141567+0000', 'last_undegraded': '2024-03-06T07:27:10.141567+0000', 'last_fullsized': '2024-03-06T07:27:10.141567+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:27:14.862 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.39 is not active+clean 2024-03-06T07:27:14.862 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.135202+0000', 'last_change': '2024-03-06T07:27:10.135202+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.135202+0000', 'last_undegraded': '2024-03-06T07:27:10.135202+0000', 'last_fullsized': '2024-03-06T07:27:10.135202+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:27:14.862 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.38 is not active+clean 2024-03-06T07:27:14.862 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.137109+0000', 'last_change': '2024-03-06T07:27:10.137109+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.137109+0000', 'last_undegraded': '2024-03-06T07:27:10.137109+0000', 'last_fullsized': '2024-03-06T07:27:10.137109+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:14.863 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.37 is not active+clean 2024-03-06T07:27:14.863 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.140884+0000', 'last_change': '2024-03-06T07:27:10.140884+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.140884+0000', 'last_undegraded': '2024-03-06T07:27:10.140884+0000', 'last_fullsized': '2024-03-06T07:27:10.140884+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:14.863 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.36 is not active+clean 2024-03-06T07:27:14.863 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.140929+0000', 'last_change': '2024-03-06T07:27:10.140929+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.140929+0000', 'last_undegraded': '2024-03-06T07:27:10.140929+0000', 'last_fullsized': '2024-03-06T07:27:10.140929+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:14.863 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.35 is not active+clean 2024-03-06T07:27:14.863 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.131689+0000', 'last_change': '2024-03-06T07:27:10.131689+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.131689+0000', 'last_undegraded': '2024-03-06T07:27:10.131689+0000', 'last_fullsized': '2024-03-06T07:27:10.131689+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:14.863 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.34 is not active+clean 2024-03-06T07:27:14.863 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.139813+0000', 'last_change': '2024-03-06T07:27:10.139813+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.139813+0000', 'last_undegraded': '2024-03-06T07:27:10.139813+0000', 'last_fullsized': '2024-03-06T07:27:10.139813+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:14.863 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.33 is not active+clean 2024-03-06T07:27:14.863 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.136022+0000', 'last_change': '2024-03-06T07:27:10.136022+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.136022+0000', 'last_undegraded': '2024-03-06T07:27:10.136022+0000', 'last_fullsized': '2024-03-06T07:27:10.136022+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:14.863 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.32 is not active+clean 2024-03-06T07:27:14.864 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.136551+0000', 'last_change': '2024-03-06T07:27:10.136551+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.136551+0000', 'last_undegraded': '2024-03-06T07:27:10.136551+0000', 'last_fullsized': '2024-03-06T07:27:10.136551+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:14.864 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.31 is not active+clean 2024-03-06T07:27:14.864 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.143562+0000', 'last_change': '2024-03-06T07:27:10.143562+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.143562+0000', 'last_undegraded': '2024-03-06T07:27:10.143562+0000', 'last_fullsized': '2024-03-06T07:27:10.143562+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:27:14.864 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.30 is not active+clean 2024-03-06T07:27:14.864 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.141170+0000', 'last_change': '2024-03-06T07:27:10.141170+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.141170+0000', 'last_undegraded': '2024-03-06T07:27:10.141170+0000', 'last_fullsized': '2024-03-06T07:27:10.141170+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:14.864 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.2f is not active+clean 2024-03-06T07:27:14.864 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.133703+0000', 'last_change': '2024-03-06T07:27:10.133703+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.133703+0000', 'last_undegraded': '2024-03-06T07:27:10.133703+0000', 'last_fullsized': '2024-03-06T07:27:10.133703+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:14.864 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.2e is not active+clean 2024-03-06T07:27:14.864 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.134084+0000', 'last_change': '2024-03-06T07:27:10.134084+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.134084+0000', 'last_undegraded': '2024-03-06T07:27:10.134084+0000', 'last_fullsized': '2024-03-06T07:27:10.134084+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:27:14.864 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.2d is not active+clean 2024-03-06T07:27:14.865 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.139490+0000', 'last_change': '2024-03-06T07:27:10.139490+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.139490+0000', 'last_undegraded': '2024-03-06T07:27:10.139490+0000', 'last_fullsized': '2024-03-06T07:27:10.139490+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:14.865 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.2c is not active+clean 2024-03-06T07:27:14.865 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.133392+0000', 'last_change': '2024-03-06T07:27:10.133392+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.133392+0000', 'last_undegraded': '2024-03-06T07:27:10.133392+0000', 'last_fullsized': '2024-03-06T07:27:10.133392+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:14.865 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.2b is not active+clean 2024-03-06T07:27:14.865 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.131525+0000', 'last_change': '2024-03-06T07:27:10.131525+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.131525+0000', 'last_undegraded': '2024-03-06T07:27:10.131525+0000', 'last_fullsized': '2024-03-06T07:27:10.131525+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:14.865 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.2a is not active+clean 2024-03-06T07:27:14.865 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.138692+0000', 'last_change': '2024-03-06T07:27:10.138692+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.138692+0000', 'last_undegraded': '2024-03-06T07:27:10.138692+0000', 'last_fullsized': '2024-03-06T07:27:10.138692+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:27:14.865 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.29 is not active+clean 2024-03-06T07:27:14.865 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.131673+0000', 'last_change': '2024-03-06T07:27:10.131673+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.131673+0000', 'last_undegraded': '2024-03-06T07:27:10.131673+0000', 'last_fullsized': '2024-03-06T07:27:10.131673+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:14.865 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.28 is not active+clean 2024-03-06T07:27:14.865 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.138495+0000', 'last_change': '2024-03-06T07:27:10.138495+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.138495+0000', 'last_undegraded': '2024-03-06T07:27:10.138495+0000', 'last_fullsized': '2024-03-06T07:27:10.138495+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:27:14.866 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.27 is not active+clean 2024-03-06T07:27:14.866 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.132448+0000', 'last_change': '2024-03-06T07:27:10.132448+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.132448+0000', 'last_undegraded': '2024-03-06T07:27:10.132448+0000', 'last_fullsized': '2024-03-06T07:27:10.132448+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:14.866 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.26 is not active+clean 2024-03-06T07:27:14.866 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.139794+0000', 'last_change': '2024-03-06T07:27:10.139794+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.139794+0000', 'last_undegraded': '2024-03-06T07:27:10.139794+0000', 'last_fullsized': '2024-03-06T07:27:10.139794+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:14.866 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.25 is not active+clean 2024-03-06T07:27:14.866 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.141443+0000', 'last_change': '2024-03-06T07:27:10.141443+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.141443+0000', 'last_undegraded': '2024-03-06T07:27:10.141443+0000', 'last_fullsized': '2024-03-06T07:27:10.141443+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:27:14.866 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.24 is not active+clean 2024-03-06T07:27:14.866 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.139294+0000', 'last_change': '2024-03-06T07:27:10.139294+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.139294+0000', 'last_undegraded': '2024-03-06T07:27:10.139294+0000', 'last_fullsized': '2024-03-06T07:27:10.139294+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:27:14.866 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.23 is not active+clean 2024-03-06T07:27:14.866 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.134116+0000', 'last_change': '2024-03-06T07:27:10.134116+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.134116+0000', 'last_undegraded': '2024-03-06T07:27:10.134116+0000', 'last_fullsized': '2024-03-06T07:27:10.134116+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:14.866 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.22 is not active+clean 2024-03-06T07:27:14.866 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.134159+0000', 'last_change': '2024-03-06T07:27:10.134159+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.134159+0000', 'last_undegraded': '2024-03-06T07:27:10.134159+0000', 'last_fullsized': '2024-03-06T07:27:10.134159+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:27:14.867 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.21 is not active+clean 2024-03-06T07:27:14.867 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.141268+0000', 'last_change': '2024-03-06T07:27:10.141268+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.141268+0000', 'last_undegraded': '2024-03-06T07:27:10.141268+0000', 'last_fullsized': '2024-03-06T07:27:10.141268+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:14.867 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.20 is not active+clean 2024-03-06T07:27:14.867 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.140860+0000', 'last_change': '2024-03-06T07:27:10.140860+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.140860+0000', 'last_undegraded': '2024-03-06T07:27:10.140860+0000', 'last_fullsized': '2024-03-06T07:27:10.140860+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:27:14.867 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1e is not active+clean 2024-03-06T07:27:14.867 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.140859+0000', 'last_change': '2024-03-06T07:27:10.140859+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.140859+0000', 'last_undegraded': '2024-03-06T07:27:10.140859+0000', 'last_fullsized': '2024-03-06T07:27:10.140859+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:27:14.867 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1f is not active+clean 2024-03-06T07:27:14.867 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.135118+0000', 'last_change': '2024-03-06T07:27:10.135118+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.135118+0000', 'last_undegraded': '2024-03-06T07:27:10.135118+0000', 'last_fullsized': '2024-03-06T07:27:10.135118+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:27:14.867 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1c is not active+clean 2024-03-06T07:27:14.867 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.141075+0000', 'last_change': '2024-03-06T07:27:10.141075+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.141075+0000', 'last_undegraded': '2024-03-06T07:27:10.141075+0000', 'last_fullsized': '2024-03-06T07:27:10.141075+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:27:14.868 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1d is not active+clean 2024-03-06T07:27:14.868 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.137861+0000', 'last_change': '2024-03-06T07:27:10.137861+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.137861+0000', 'last_undegraded': '2024-03-06T07:27:10.137861+0000', 'last_fullsized': '2024-03-06T07:27:10.137861+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:14.868 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1a is not active+clean 2024-03-06T07:27:14.868 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.139239+0000', 'last_change': '2024-03-06T07:27:10.139239+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.139239+0000', 'last_undegraded': '2024-03-06T07:27:10.139239+0000', 'last_fullsized': '2024-03-06T07:27:10.139239+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:14.868 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1b is not active+clean 2024-03-06T07:27:14.868 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.133490+0000', 'last_change': '2024-03-06T07:27:10.133490+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.133490+0000', 'last_undegraded': '2024-03-06T07:27:10.133490+0000', 'last_fullsized': '2024-03-06T07:27:10.133490+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:14.868 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.18 is not active+clean 2024-03-06T07:27:14.868 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.139850+0000', 'last_change': '2024-03-06T07:27:10.139850+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.139850+0000', 'last_undegraded': '2024-03-06T07:27:10.139850+0000', 'last_fullsized': '2024-03-06T07:27:10.139850+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:14.868 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.19 is not active+clean 2024-03-06T07:27:14.869 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.135641+0000', 'last_change': '2024-03-06T07:27:10.135641+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.135641+0000', 'last_undegraded': '2024-03-06T07:27:10.135641+0000', 'last_fullsized': '2024-03-06T07:27:10.135641+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:27:14.869 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.16 is not active+clean 2024-03-06T07:27:14.869 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.135927+0000', 'last_change': '2024-03-06T07:27:10.135927+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.135927+0000', 'last_undegraded': '2024-03-06T07:27:10.135927+0000', 'last_fullsized': '2024-03-06T07:27:10.135927+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:27:14.869 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.17 is not active+clean 2024-03-06T07:27:14.869 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.134728+0000', 'last_change': '2024-03-06T07:27:10.134728+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.134728+0000', 'last_undegraded': '2024-03-06T07:27:10.134728+0000', 'last_fullsized': '2024-03-06T07:27:10.134728+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:27:14.869 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.14 is not active+clean 2024-03-06T07:27:14.869 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.138899+0000', 'last_change': '2024-03-06T07:27:10.138899+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.138899+0000', 'last_undegraded': '2024-03-06T07:27:10.138899+0000', 'last_fullsized': '2024-03-06T07:27:10.138899+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:27:14.869 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.15 is not active+clean 2024-03-06T07:27:14.869 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.134203+0000', 'last_change': '2024-03-06T07:27:10.134203+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.134203+0000', 'last_undegraded': '2024-03-06T07:27:10.134203+0000', 'last_fullsized': '2024-03-06T07:27:10.134203+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:27:14.869 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.12 is not active+clean 2024-03-06T07:27:14.869 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.138207+0000', 'last_change': '2024-03-06T07:27:10.138207+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.138207+0000', 'last_undegraded': '2024-03-06T07:27:10.138207+0000', 'last_fullsized': '2024-03-06T07:27:10.138207+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:27:14.870 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.13 is not active+clean 2024-03-06T07:27:14.870 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.137691+0000', 'last_change': '2024-03-06T07:27:10.137691+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.137691+0000', 'last_undegraded': '2024-03-06T07:27:10.137691+0000', 'last_fullsized': '2024-03-06T07:27:10.137691+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:27:14.870 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.11 is not active+clean 2024-03-06T07:27:14.870 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.139264+0000', 'last_change': '2024-03-06T07:27:10.139264+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.139264+0000', 'last_undegraded': '2024-03-06T07:27:10.139264+0000', 'last_fullsized': '2024-03-06T07:27:10.139264+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:27:14.870 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.e is not active+clean 2024-03-06T07:27:14.870 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.133922+0000', 'last_change': '2024-03-06T07:27:10.133922+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.133922+0000', 'last_undegraded': '2024-03-06T07:27:10.133922+0000', 'last_fullsized': '2024-03-06T07:27:10.133922+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:27:14.870 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.f is not active+clean 2024-03-06T07:27:14.870 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.140347+0000', 'last_change': '2024-03-06T07:27:10.140347+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.140347+0000', 'last_undegraded': '2024-03-06T07:27:10.140347+0000', 'last_fullsized': '2024-03-06T07:27:10.140347+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:27:14.870 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.c is not active+clean 2024-03-06T07:27:14.871 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.135554+0000', 'last_change': '2024-03-06T07:27:10.135554+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.135554+0000', 'last_undegraded': '2024-03-06T07:27:10.135554+0000', 'last_fullsized': '2024-03-06T07:27:10.135554+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:14.871 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.d is not active+clean 2024-03-06T07:27:14.871 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.137107+0000', 'last_change': '2024-03-06T07:27:10.137107+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.137107+0000', 'last_undegraded': '2024-03-06T07:27:10.137107+0000', 'last_fullsized': '2024-03-06T07:27:10.137107+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:14.871 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.a is not active+clean 2024-03-06T07:27:14.871 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.135497+0000', 'last_change': '2024-03-06T07:27:10.135497+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.135497+0000', 'last_undegraded': '2024-03-06T07:27:10.135497+0000', 'last_fullsized': '2024-03-06T07:27:10.135497+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:14.871 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.b is not active+clean 2024-03-06T07:27:14.871 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.139184+0000', 'last_change': '2024-03-06T07:27:10.139184+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.139184+0000', 'last_undegraded': '2024-03-06T07:27:10.139184+0000', 'last_fullsized': '2024-03-06T07:27:10.139184+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:14.872 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.8 is not active+clean 2024-03-06T07:27:14.872 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.136666+0000', 'last_change': '2024-03-06T07:27:10.136666+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.136666+0000', 'last_undegraded': '2024-03-06T07:27:10.136666+0000', 'last_fullsized': '2024-03-06T07:27:10.136666+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:14.872 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.9 is not active+clean 2024-03-06T07:27:14.872 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.138259+0000', 'last_change': '2024-03-06T07:27:10.138259+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.138259+0000', 'last_undegraded': '2024-03-06T07:27:10.138259+0000', 'last_fullsized': '2024-03-06T07:27:10.138259+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:27:14.872 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.6 is not active+clean 2024-03-06T07:27:14.872 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.133669+0000', 'last_change': '2024-03-06T07:27:10.133669+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.133669+0000', 'last_undegraded': '2024-03-06T07:27:10.133669+0000', 'last_fullsized': '2024-03-06T07:27:10.133669+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:27:14.872 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.7 is not active+clean 2024-03-06T07:27:14.872 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.138704+0000', 'last_change': '2024-03-06T07:27:10.138704+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.138704+0000', 'last_undegraded': '2024-03-06T07:27:10.138704+0000', 'last_fullsized': '2024-03-06T07:27:10.138704+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:14.873 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.4 is not active+clean 2024-03-06T07:27:14.873 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.138689+0000', 'last_change': '2024-03-06T07:27:10.138689+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.138689+0000', 'last_undegraded': '2024-03-06T07:27:10.138689+0000', 'last_fullsized': '2024-03-06T07:27:10.138689+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:14.873 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.5 is not active+clean 2024-03-06T07:27:14.873 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.132879+0000', 'last_change': '2024-03-06T07:27:10.132879+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.132879+0000', 'last_undegraded': '2024-03-06T07:27:10.132879+0000', 'last_fullsized': '2024-03-06T07:27:10.132879+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:27:14.873 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.2 is not active+clean 2024-03-06T07:27:14.873 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.131712+0000', 'last_change': '2024-03-06T07:27:10.131712+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.131712+0000', 'last_undegraded': '2024-03-06T07:27:10.131712+0000', 'last_fullsized': '2024-03-06T07:27:10.131712+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:14.873 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.3 is not active+clean 2024-03-06T07:27:14.873 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.132772+0000', 'last_change': '2024-03-06T07:27:10.132772+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.132772+0000', 'last_undegraded': '2024-03-06T07:27:10.132772+0000', 'last_fullsized': '2024-03-06T07:27:10.132772+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:14.873 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.0 is not active+clean 2024-03-06T07:27:14.873 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.136141+0000', 'last_change': '2024-03-06T07:27:10.136141+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.136141+0000', 'last_undegraded': '2024-03-06T07:27:10.136141+0000', 'last_fullsized': '2024-03-06T07:27:10.136141+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:14.874 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1 is not active+clean 2024-03-06T07:27:14.874 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.140866+0000', 'last_change': '2024-03-06T07:27:10.140866+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.140866+0000', 'last_undegraded': '2024-03-06T07:27:10.140866+0000', 'last_fullsized': '2024-03-06T07:27:10.140866+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:14.874 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.10 is not active+clean 2024-03-06T07:27:14.874 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 108, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:10.137469+0000', 'last_change': '2024-03-06T07:27:10.137469+0000', 'last_active': '2024-03-06T07:27:10.126521+0000', 'last_peered': '2024-03-06T07:27:10.126521+0000', 'last_clean': '2024-03-06T07:27:10.126521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:10.137469+0000', 'last_undegraded': '2024-03-06T07:27:10.137469+0000', 'last_fullsized': '2024-03-06T07:27:10.137469+0000', 'mapping_epoch': 108, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 108, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:10.126521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:27:14.874 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:27:15.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.998+0000 7f0b323bc640 1 Processor -- start 2024-03-06T07:27:15.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:14.999+0000 7f0b323bc640 1 -- start start 2024-03-06T07:27:15.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.000+0000 7f0b323bc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0b2c0b4460 0x7f0b2c0d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.000+0000 7f0b323bc640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f0b2c162970 0x7f0b2c0b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.000+0000 7f0b323bc640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b2c161cf0 0x7f0b2c162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.000+0000 7f0b323bc640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b2c161b00 con 0x7f0b2c0b4460 2024-03-06T07:27:15.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.000+0000 7f0b323bc640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b2c0cecc0 con 0x7f0b2c162970 2024-03-06T07:27:15.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.000+0000 7f0b323bc640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2c0d22c0 con 0x7f0b2c161cf0 2024-03-06T07:27:15.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.000+0000 7f0b2bfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0b2c0b4460 0x7f0b2c0d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.000+0000 7f0b2bfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0b2c0b4460 0x7f0b2c0d4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47714/0 (socket says 172.21.15.83:47714) 2024-03-06T07:27:15.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.000+0000 7f0b2bfff640 1 -- 172.21.15.83:0/1399232874 learned_addr learned my addr 172.21.15.83:0/1399232874 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:15.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.000+0000 7f0b2b7fe640 1 --2- 172.21.15.83:0/1399232874 >> v2:172.21.15.183:3300/0 conn(0x7f0b2c162970 0x7f0b2c0b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.000+0000 7f0b30959640 1 --2- 172.21.15.83:0/1399232874 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b2c161cf0 0x7f0b2c162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.000+0000 7f0b2bfff640 1 -- 172.21.15.83:0/1399232874 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b2c161cf0 msgr2=0x7f0b2c162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:15.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.000+0000 7f0b2bfff640 1 --2- 172.21.15.83:0/1399232874 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b2c161cf0 0x7f0b2c162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.000+0000 7f0b2bfff640 1 -- 172.21.15.83:0/1399232874 >> v2:172.21.15.183:3300/0 conn(0x7f0b2c162970 msgr2=0x7f0b2c0b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:15.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.000+0000 7f0b2bfff640 1 --2- 172.21.15.83:0/1399232874 >> v2:172.21.15.183:3300/0 conn(0x7f0b2c162970 0x7f0b2c0b3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.000+0000 7f0b2bfff640 1 -- 172.21.15.83:0/1399232874 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b2c0d25a0 con 0x7f0b2c0b4460 2024-03-06T07:27:15.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.001+0000 7f0b2bfff640 1 --2- 172.21.15.83:0/1399232874 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b2c0b4460 0x7f0b2c0d4800 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c0d17e0 tx=0x7f0b1c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e7db84085a19b41 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.001+0000 7f0b2affd640 1 -- 172.21.15.83:0/1399232874 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1c035020 con 0x7f0b2c0b4460 2024-03-06T07:27:15.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.001+0000 7f0b2affd640 1 -- 172.21.15.83:0/1399232874 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b1c031870 con 0x7f0b2c0b4460 2024-03-06T07:27:15.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.001+0000 7f0b2affd640 1 -- 172.21.15.83:0/1399232874 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1c031c40 con 0x7f0b2c0b4460 2024-03-06T07:27:15.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.002+0000 7f0b323bc640 1 -- 172.21.15.83:0/1399232874 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b2c0b4460 msgr2=0x7f0b2c0d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.002+0000 7f0b323bc640 1 --2- 172.21.15.83:0/1399232874 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b2c0b4460 0x7f0b2c0d4800 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c0d17e0 tx=0x7f0b1c02b910 comp rx=0 tx=0).stop 2024-03-06T07:27:15.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.002+0000 7f0b323bc640 1 -- 172.21.15.83:0/1399232874 shutdown_connections 2024-03-06T07:27:15.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.002+0000 7f0b323bc640 1 --2- 172.21.15.83:0/1399232874 >> v2:172.21.15.183:3300/0 conn(0x7f0b2c162970 0x7f0b2c0b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.002+0000 7f0b323bc640 1 --2- 172.21.15.83:0/1399232874 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b2c161cf0 0x7f0b2c162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.002+0000 7f0b323bc640 1 --2- 172.21.15.83:0/1399232874 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b2c0b4460 0x7f0b2c0d4800 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.002+0000 7f0b323bc640 1 -- 172.21.15.83:0/1399232874 >> 172.21.15.83:0/1399232874 conn(0x7f0b2c0d3ce0 msgr2=0x7f0b2c0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:15.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.002+0000 7f0b323bc640 1 -- 172.21.15.83:0/1399232874 shutdown_connections 2024-03-06T07:27:15.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.002+0000 7f0b323bc640 1 -- 172.21.15.83:0/1399232874 wait complete. 2024-03-06T07:27:15.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.004+0000 7f0b323bc640 1 Processor -- start 2024-03-06T07:27:15.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.004+0000 7f0b323bc640 1 -- start start 2024-03-06T07:27:15.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.005+0000 7f0b323bc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b2c0b4460 0x7f0b2c155d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.005+0000 7f0b323bc640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b2c161cf0 0x7f0b2c0c81b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.005+0000 7f0b323bc640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b2c162970 0x7f0b2c0cadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.005+0000 7f0b323bc640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2c0ce950 con 0x7f0b2c0b4460 2024-03-06T07:27:15.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.005+0000 7f0b323bc640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2c0d2b60 con 0x7f0b2c162970 2024-03-06T07:27:15.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.005+0000 7f0b323bc640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2c0b3ac0 con 0x7f0b2c161cf0 2024-03-06T07:27:15.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.005+0000 7f0b2bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b2c0b4460 0x7f0b2c155d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.005+0000 7f0b2bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b2c0b4460 0x7f0b2c155d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47728/0 (socket says 172.21.15.83:47728) 2024-03-06T07:27:15.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.005+0000 7f0b2bfff640 1 -- 172.21.15.83:0/965503261 learned_addr learned my addr 172.21.15.83:0/965503261 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:15.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.005+0000 7f0b30959640 1 --2- 172.21.15.83:0/965503261 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b2c162970 0x7f0b2c0cadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.005+0000 7f0b2b7fe640 1 --2- 172.21.15.83:0/965503261 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b2c161cf0 0x7f0b2c0c81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.005+0000 7f0b2bfff640 1 -- 172.21.15.83:0/965503261 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b2c161cf0 msgr2=0x7f0b2c0c81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.005+0000 7f0b2bfff640 1 --2- 172.21.15.83:0/965503261 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b2c161cf0 0x7f0b2c0c81b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.005+0000 7f0b2bfff640 1 -- 172.21.15.83:0/965503261 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b2c162970 msgr2=0x7f0b2c0cadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.005+0000 7f0b2bfff640 1 --2- 172.21.15.83:0/965503261 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b2c162970 0x7f0b2c0cadb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.005+0000 7f0b2bfff640 1 -- 172.21.15.83:0/965503261 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b2c156290 con 0x7f0b2c0b4460 2024-03-06T07:27:15.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.006+0000 7f0b30959640 1 --2- 172.21.15.83:0/965503261 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b2c162970 0x7f0b2c0cadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:15.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.006+0000 7f0b2bfff640 1 --2- 172.21.15.83:0/965503261 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b2c0b4460 0x7f0b2c155d50 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c02b950 tx=0x7f0b1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.006+0000 7f0b28ff9640 1 -- 172.21.15.83:0/965503261 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1c040070 con 0x7f0b2c0b4460 2024-03-06T07:27:15.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.006+0000 7f0b28ff9640 1 -- 172.21.15.83:0/965503261 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b1c030d00 con 0x7f0b2c0b4460 2024-03-06T07:27:15.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.006+0000 7f0b323bc640 1 -- 172.21.15.83:0/965503261 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b2c0b9ab0 con 0x7f0b2c0b4460 2024-03-06T07:27:15.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.006+0000 7f0b28ff9640 1 -- 172.21.15.83:0/965503261 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1c035040 con 0x7f0b2c0b4460 2024-03-06T07:27:15.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.006+0000 7f0b323bc640 1 -- 172.21.15.83:0/965503261 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b2c0cbc00 con 0x7f0b2c0b4460 2024-03-06T07:27:15.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.007+0000 7f0b323bc640 1 -- 172.21.15.83:0/965503261 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b2c04f330 con 0x7f0b2c0b4460 2024-03-06T07:27:15.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.007+0000 7f0b28ff9640 1 -- 172.21.15.83:0/965503261 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0b1c04b060 con 0x7f0b2c0b4460 2024-03-06T07:27:15.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.008+0000 7f0b28ff9640 1 --2- 172.21.15.83:0/965503261 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0b1806b4d0 0x7f0b1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.008+0000 7f0b2b7fe640 1 --2- 172.21.15.83:0/965503261 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0b1806b4d0 0x7f0b1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.008+0000 7f0b28ff9640 1 -- 172.21.15.83:0/965503261 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0b1c0abe40 con 0x7f0b2c0b4460 2024-03-06T07:27:15.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.008+0000 7f0b2b7fe640 1 --2- 172.21.15.83:0/965503261 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0b1806b4d0 0x7f0b1806d990 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c15a050 tx=0x7f0b20008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.016+0000 7f0b28ff9640 1 -- 172.21.15.83:0/965503261 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0b2c156290 con 0x7f0b2c0b4460 2024-03-06T07:27:15.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.284+0000 7f0b323bc640 1 -- 172.21.15.83:0/965503261 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f0b2c0cbdd0 con 0x7f0b2c0b4460 2024-03-06T07:27:15.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.285+0000 7f0b28ff9640 1 -- 172.21.15.83:0/965503261 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1035 (secure 0 0 0) 0x7f0b1c02f320 con 0x7f0b2c0b4460 2024-03-06T07:27:15.287 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:15.287 INFO:teuthology.orchestra.run.smithi083.stdout:{"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":341,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":110,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709705,"num_in_osds":8,"osd_in_since":1709709696,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":331235328,"bytes_avail":772762877952,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":59,"id":8,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":8,"rank":0,"name":"a","status":"up:active","gid":7887}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-03-06T07:26:28.369187+0000","services":{}},"progress_events":{}} 2024-03-06T07:27:15.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.287+0000 7f0b323bc640 1 -- 172.21.15.83:0/965503261 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0b1806b4d0 msgr2=0x7f0b1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.287+0000 7f0b323bc640 1 --2- 172.21.15.83:0/965503261 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0b1806b4d0 0x7f0b1806d990 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c15a050 tx=0x7f0b20008040 comp rx=0 tx=0).stop 2024-03-06T07:27:15.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.287+0000 7f0b323bc640 1 -- 172.21.15.83:0/965503261 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b2c0b4460 msgr2=0x7f0b2c155d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.287+0000 7f0b323bc640 1 --2- 172.21.15.83:0/965503261 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b2c0b4460 0x7f0b2c155d50 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c02b950 tx=0x7f0b1c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.287+0000 7f0b323bc640 1 -- 172.21.15.83:0/965503261 shutdown_connections 2024-03-06T07:27:15.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.287+0000 7f0b323bc640 1 --2- 172.21.15.83:0/965503261 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0b1806b4d0 0x7f0b1806d990 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.287+0000 7f0b323bc640 1 --2- 172.21.15.83:0/965503261 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b2c162970 0x7f0b2c0cadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.287+0000 7f0b323bc640 1 --2- 172.21.15.83:0/965503261 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0b2c161cf0 0x7f0b2c0c81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.287+0000 7f0b323bc640 1 --2- 172.21.15.83:0/965503261 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0b2c0b4460 0x7f0b2c155d50 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.287+0000 7f0b323bc640 1 -- 172.21.15.83:0/965503261 >> 172.21.15.83:0/965503261 conn(0x7f0b2c0d3ce0 msgr2=0x7f0b2c0d0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:15.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.287+0000 7f0b323bc640 1 -- 172.21.15.83:0/965503261 shutdown_connections 2024-03-06T07:27:15.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.287+0000 7f0b323bc640 1 -- 172.21.15.83:0/965503261 wait complete. 2024-03-06T07:27:15.299 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:27:15.300 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:15.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.399+0000 7f56f5935640 1 Processor -- start 2024-03-06T07:27:15.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.399+0000 7f56f5935640 1 -- start start 2024-03-06T07:27:15.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.400+0000 7f56f5935640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f56f00d22b0 0x7f56f00d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.400+0000 7f56f5935640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f56f00b41f0 0x7f56f00b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.400+0000 7f56f5935640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56f00b3430 con 0x7f56f015dfe0 2024-03-06T07:27:15.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.400+0000 7f56f5935640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56f00d2c50 con 0x7f56f00d22b0 2024-03-06T07:27:15.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.400+0000 7f56f5935640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56f00b31e0 con 0x7f56f00b41f0 2024-03-06T07:27:15.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.400+0000 7f56eeffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f56f015dfe0 0x7f56f00cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:37646/0 (socket says 172.21.15.83:37646) 2024-03-06T07:27:15.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.400+0000 7f56eeffd640 1 -- 172.21.15.83:0/2142421315 learned_addr learned my addr 172.21.15.83:0/2142421315 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:27:15.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.400+0000 7f56e67fc640 1 --2- 172.21.15.83:0/2142421315 >> v2:172.21.15.183:3300/0 conn(0x7f56f00d22b0 0x7f56f00d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.400+0000 7f56ef7fe640 1 --2- 172.21.15.83:0/2142421315 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f56f00b41f0 0x7f56f00b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.400+0000 7f56ee7fc640 1 -- 172.21.15.83:0/2142421315 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3158782739 0 0) 0x7f56f00b3430 con 0x7f56f015dfe0 2024-03-06T07:27:15.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.400+0000 7f56ee7fc640 1 -- 172.21.15.83:0/2142421315 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56d0003520 con 0x7f56f015dfe0 2024-03-06T07:27:15.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.401+0000 7f56ee7fc640 1 -- 172.21.15.83:0/2142421315 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4186730500 0 0) 0x7f56d0003520 con 0x7f56f015dfe0 2024-03-06T07:27:15.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.401+0000 7f56ee7fc640 1 -- 172.21.15.83:0/2142421315 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f56f00b41f0 msgr2=0x7f56f00b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:15.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.401+0000 7f56ee7fc640 1 --2- 172.21.15.83:0/2142421315 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f56f00b41f0 0x7f56f00b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.401+0000 7f56ee7fc640 1 -- 172.21.15.83:0/2142421315 >> v2:172.21.15.183:3300/0 conn(0x7f56f00d22b0 msgr2=0x7f56f00d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:15.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.401+0000 7f56ee7fc640 1 --2- 172.21.15.83:0/2142421315 >> v2:172.21.15.183:3300/0 conn(0x7f56f00d22b0 0x7f56f00d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.401+0000 7f56ee7fc640 1 -- 172.21.15.83:0/2142421315 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f00b3710 con 0x7f56f015dfe0 2024-03-06T07:27:15.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.401+0000 7f56ee7fc640 1 -- 172.21.15.83:0/2142421315 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f56e0001670 con 0x7f56f015dfe0 2024-03-06T07:27:15.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.401+0000 7f56ee7fc640 1 -- 172.21.15.83:0/2142421315 >> v1:172.21.15.83:6789/0 conn(0x7f56f015dfe0 legacy=0x7f56f00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.401+0000 7f56ee7fc640 1 --2- 172.21.15.83:0/2142421315 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f56f00b41f0 0x7f56d00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.401+0000 7f56ee7fc640 1 --2- 172.21.15.83:0/2142421315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56d0003f30 0x7f56d0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.401+0000 7f56ee7fc640 1 --2- 172.21.15.83:0/2142421315 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f56d0004d40 0x7f56d00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.401+0000 7f56ee7fc640 1 -- 172.21.15.83:0/2142421315 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f00b31e0 con 0x7f56d0003f30 2024-03-06T07:27:15.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.401+0000 7f56ee7fc640 1 -- 172.21.15.83:0/2142421315 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d000b9c0 con 0x7f56d0004d40 2024-03-06T07:27:15.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.401+0000 7f56ee7fc640 1 -- 172.21.15.83:0/2142421315 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56d000bb40 con 0x7f56f00b41f0 2024-03-06T07:27:15.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.401+0000 7f56e67fc640 1 --2- 172.21.15.83:0/2142421315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56d0003f30 0x7f56d0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.401+0000 7f56eeffd640 1 --2- 172.21.15.83:0/2142421315 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f56f00b41f0 0x7f56d00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.401+0000 7f56ef7fe640 1 --2- 172.21.15.83:0/2142421315 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f56d0004d40 0x7f56d00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.401+0000 7f56e67fc640 1 -- 172.21.15.83:0/2142421315 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f56f00b41f0 msgr2=0x7f56d00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:15.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.401+0000 7f56e67fc640 1 --2- 172.21.15.83:0/2142421315 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f56f00b41f0 0x7f56d00039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.401+0000 7f56e67fc640 1 -- 172.21.15.83:0/2142421315 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f56d0004d40 msgr2=0x7f56d00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:15.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.401+0000 7f56e67fc640 1 --2- 172.21.15.83:0/2142421315 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f56d0004d40 0x7f56d00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.401+0000 7f56e67fc640 1 -- 172.21.15.83:0/2142421315 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d0003520 con 0x7f56d0003f30 2024-03-06T07:27:15.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.402+0000 7f56e67fc640 1 --2- 172.21.15.83:0/2142421315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56d0003f30 0x7f56d0004800 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f56d8009ac0 tx=0x7f56d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f724bbff3cd76ff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.402+0000 7f56ee7fc640 1 -- 172.21.15.83:0/2142421315 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d8039070 con 0x7f56d0003f30 2024-03-06T07:27:15.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.402+0000 7f56ee7fc640 1 -- 172.21.15.83:0/2142421315 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56d802c6d0 con 0x7f56d0003f30 2024-03-06T07:27:15.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.402+0000 7f56ee7fc640 1 -- 172.21.15.83:0/2142421315 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d8009690 con 0x7f56d0003f30 2024-03-06T07:27:15.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.403+0000 7f56f5935640 1 -- 172.21.15.83:0/2142421315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56d0003f30 msgr2=0x7f56d0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.403+0000 7f56f5935640 1 --2- 172.21.15.83:0/2142421315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56d0003f30 0x7f56d0004800 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f56d8009ac0 tx=0x7f56d0005830 comp rx=0 tx=0).stop 2024-03-06T07:27:15.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.403+0000 7f56ef7fe640 1 -- 172.21.15.83:0/2142421315 reap_dead start 2024-03-06T07:27:15.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.403+0000 7f56f5935640 1 -- 172.21.15.83:0/2142421315 shutdown_connections 2024-03-06T07:27:15.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.403+0000 7f56f5935640 1 -- 172.21.15.83:0/2142421315 >> 172.21.15.83:0/2142421315 conn(0x7f56f00d3cd0 msgr2=0x7f56f00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:15.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.403+0000 7f56f5935640 1 -- 172.21.15.83:0/2142421315 shutdown_connections 2024-03-06T07:27:15.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.403+0000 7f56f5935640 1 -- 172.21.15.83:0/2142421315 wait complete. 2024-03-06T07:27:15.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.404+0000 7f56f5935640 1 Processor -- start 2024-03-06T07:27:15.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.404+0000 7f56f5935640 1 -- start start 2024-03-06T07:27:15.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.405+0000 7f56f5935640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f56d0003f30 0x7f56f00cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.405+0000 7f56f5935640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56f00b96b0 0x7f56f0150e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.405+0000 7f56f5935640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f56f00b6d30 0x7f56f00b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.405+0000 7f56f5935640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f00cfa00 con 0x7f56f00b96b0 2024-03-06T07:27:15.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.405+0000 7f56f5935640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f00b3d60 con 0x7f56d0003f30 2024-03-06T07:27:15.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.405+0000 7f56f5935640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56f00d1c70 con 0x7f56f00b6d30 2024-03-06T07:27:15.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.405+0000 7f56e67fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56f00b96b0 0x7f56f0150e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.405+0000 7f56e67fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56f00b96b0 0x7f56f0150e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47734/0 (socket says 172.21.15.83:47734) 2024-03-06T07:27:15.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.405+0000 7f56e67fc640 1 -- 172.21.15.83:0/1425933322 learned_addr learned my addr 172.21.15.83:0/1425933322 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:15.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.405+0000 7f56eeffd640 1 --2- 172.21.15.83:0/1425933322 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f56d0003f30 0x7f56f00cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.405+0000 7f56ef7fe640 1 --2- 172.21.15.83:0/1425933322 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f56f00b6d30 0x7f56f00b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.405+0000 7f56e67fc640 1 -- 172.21.15.83:0/1425933322 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f56f00b6d30 msgr2=0x7f56f00b71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.405+0000 7f56e67fc640 1 --2- 172.21.15.83:0/1425933322 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f56f00b6d30 0x7f56f00b71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.405+0000 7f56e67fc640 1 -- 172.21.15.83:0/1425933322 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f56d0003f30 msgr2=0x7f56f00cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.405+0000 7f56e67fc640 1 --2- 172.21.15.83:0/1425933322 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f56d0003f30 0x7f56f00cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.405+0000 7f56e67fc640 1 -- 172.21.15.83:0/1425933322 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f00cd270 con 0x7f56f00b96b0 2024-03-06T07:27:15.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.405+0000 7f56e67fc640 1 --2- 172.21.15.83:0/1425933322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56f00b96b0 0x7f56f0150e80 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f56d8009be0 tx=0x7f56d8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.405+0000 7f56ecff9640 1 -- 172.21.15.83:0/1425933322 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d8039050 con 0x7f56f00b96b0 2024-03-06T07:27:15.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.406+0000 7f56ecff9640 1 -- 172.21.15.83:0/1425933322 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56d8045070 con 0x7f56f00b96b0 2024-03-06T07:27:15.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.406+0000 7f56ecff9640 1 -- 172.21.15.83:0/1425933322 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d8003a50 con 0x7f56f00b96b0 2024-03-06T07:27:15.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.406+0000 7f56f5935640 1 -- 172.21.15.83:0/1425933322 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56f00cd4e0 con 0x7f56f00b96b0 2024-03-06T07:27:15.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.406+0000 7f56f5935640 1 -- 172.21.15.83:0/1425933322 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56f00cdaa0 con 0x7f56f00b96b0 2024-03-06T07:27:15.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.406+0000 7f56f5935640 1 -- 172.21.15.83:0/1425933322 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56f004f330 con 0x7f56f00b96b0 2024-03-06T07:27:15.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.407+0000 7f56ecff9640 1 -- 172.21.15.83:0/1425933322 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f56d8003bf0 con 0x7f56f00b96b0 2024-03-06T07:27:15.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.407+0000 7f56ecff9640 1 --2- 172.21.15.83:0/1425933322 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f56d406b4d0 0x7f56d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.407+0000 7f56ecff9640 1 -- 172.21.15.83:0/1425933322 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f56d80abe20 con 0x7f56f00b96b0 2024-03-06T07:27:15.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.407+0000 7f56eeffd640 1 --2- 172.21.15.83:0/1425933322 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f56d406b4d0 0x7f56d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.408+0000 7f56eeffd640 1 --2- 172.21.15.83:0/1425933322 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f56d406b4d0 0x7f56d406d990 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f56e0001050 tx=0x7f56e0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.412+0000 7f56ecff9640 1 -- 172.21.15.83:0/1425933322 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f56d804e020 con 0x7f56f00b96b0 2024-03-06T07:27:15.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.649+0000 7f56f5935640 1 -- 172.21.15.83:0/1425933322 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f56f006bb80 con 0x7f56d406b4d0 2024-03-06T07:27:15.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.655+0000 7f56ecff9640 1 -- 172.21.15.83:0/1425933322 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247089 (secure 0 0 0) 0x7f56f006bb80 con 0x7f56d406b4d0 2024-03-06T07:27:15.660 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:15.661 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-06T07:27:15.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.659+0000 7f56f5935640 1 -- 172.21.15.83:0/1425933322 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f56d406b4d0 msgr2=0x7f56d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.659+0000 7f56f5935640 1 --2- 172.21.15.83:0/1425933322 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f56d406b4d0 0x7f56d406d990 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f56e0001050 tx=0x7f56e0009040 comp rx=0 tx=0).stop 2024-03-06T07:27:15.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.659+0000 7f56f5935640 1 -- 172.21.15.83:0/1425933322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56f00b96b0 msgr2=0x7f56f0150e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.659+0000 7f56f5935640 1 --2- 172.21.15.83:0/1425933322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56f00b96b0 0x7f56f0150e80 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f56d8009be0 tx=0x7f56d8004730 comp rx=0 tx=0).stop 2024-03-06T07:27:15.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.659+0000 7f56f5935640 1 -- 172.21.15.83:0/1425933322 shutdown_connections 2024-03-06T07:27:15.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.659+0000 7f56f5935640 1 --2- 172.21.15.83:0/1425933322 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f56d406b4d0 0x7f56d406d990 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.659+0000 7f56f5935640 1 --2- 172.21.15.83:0/1425933322 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f56f00b6d30 0x7f56f00b71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.659+0000 7f56f5935640 1 --2- 172.21.15.83:0/1425933322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56f00b96b0 0x7f56f0150e80 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.659+0000 7f56f5935640 1 --2- 172.21.15.83:0/1425933322 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f56d0003f30 0x7f56f00cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.659+0000 7f56f5935640 1 -- 172.21.15.83:0/1425933322 >> 172.21.15.83:0/1425933322 conn(0x7f56f00d3cd0 msgr2=0x7f56f00d0a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:15.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.659+0000 7f56f5935640 1 -- 172.21.15.83:0/1425933322 shutdown_connections 2024-03-06T07:27:15.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:15.659+0000 7f56f5935640 1 -- 172.21.15.83:0/1425933322 wait complete. 2024-03-06T07:27:15.671 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":268,"stamp":"2024-03-06T07:27:14.377541+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":29,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":323472,"kb_used_data":11608,"kb_used_omap":12,"kb_used_meta":311731,"kb_avail":754651248,"statfs":{"total":773094113280,"available":772762877952,"internally_reserved":0,"allocated":11886592,"data_stored":5768468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12757,"internal_metadata":319213099},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.250662"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.143449+0000","last_change":"2024-03-06T07:27:10.143449+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.143449+0000","last_undegraded":"2024-03-06T07:27:10.143449+0000","last_fullsized":"2024-03-06T07:27:10.143449+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.131926+0000","last_change":"2024-03-06T07:27:10.131926+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.131926+0000","last_undegraded":"2024-03-06T07:27:10.131926+0000","last_fullsized":"2024-03-06T07:27:10.131926+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.140106+0000","last_change":"2024-03-06T07:27:10.140106+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.140106+0000","last_undegraded":"2024-03-06T07:27:10.140106+0000","last_fullsized":"2024-03-06T07:27:10.140106+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.141376+0000","last_change":"2024-03-06T07:27:10.141376+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.141376+0000","last_undegraded":"2024-03-06T07:27:10.141376+0000","last_fullsized":"2024-03-06T07:27:10.141376+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.134023+0000","last_change":"2024-03-06T07:27:10.134023+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.134023+0000","last_undegraded":"2024-03-06T07:27:10.134023+0000","last_fullsized":"2024-03-06T07:27:10.134023+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.141567+0000","last_change":"2024-03-06T07:27:10.141567+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.141567+0000","last_undegraded":"2024-03-06T07:27:10.141567+0000","last_fullsized":"2024-03-06T07:27:10.141567+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.135202+0000","last_change":"2024-03-06T07:27:10.135202+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.135202+0000","last_undegraded":"2024-03-06T07:27:10.135202+0000","last_fullsized":"2024-03-06T07:27:10.135202+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.137109+0000","last_change":"2024-03-06T07:27:10.137109+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.137109+0000","last_undegraded":"2024-03-06T07:27:10.137109+0000","last_fullsized":"2024-03-06T07:27:10.137109+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.140884+0000","last_change":"2024-03-06T07:27:10.140884+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.140884+0000","last_undegraded":"2024-03-06T07:27:10.140884+0000","last_fullsized":"2024-03-06T07:27:10.140884+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.140929+0000","last_change":"2024-03-06T07:27:10.140929+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.140929+0000","last_undegraded":"2024-03-06T07:27:10.140929+0000","last_fullsized":"2024-03-06T07:27:10.140929+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.131689+0000","last_change":"2024-03-06T07:27:10.131689+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.131689+0000","last_undegraded":"2024-03-06T07:27:10.131689+0000","last_fullsized":"2024-03-06T07:27:10.131689+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.139813+0000","last_change":"2024-03-06T07:27:10.139813+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.139813+0000","last_undegraded":"2024-03-06T07:27:10.139813+0000","last_fullsized":"2024-03-06T07:27:10.139813+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.136022+0000","last_change":"2024-03-06T07:27:10.136022+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.136022+0000","last_undegraded":"2024-03-06T07:27:10.136022+0000","last_fullsized":"2024-03-06T07:27:10.136022+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.136551+0000","last_change":"2024-03-06T07:27:10.136551+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.136551+0000","last_undegraded":"2024-03-06T07:27:10.136551+0000","last_fullsized":"2024-03-06T07:27:10.136551+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.143562+0000","last_change":"2024-03-06T07:27:10.143562+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.143562+0000","last_undegraded":"2024-03-06T07:27:10.143562+0000","last_fullsized":"2024-03-06T07:27:10.143562+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.141170+0000","last_change":"2024-03-06T07:27:10.141170+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.141170+0000","last_undegraded":"2024-03-06T07:27:10.141170+0000","last_fullsized":"2024-03-06T07:27:10.141170+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.133703+0000","last_change":"2024-03-06T07:27:10.133703+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.133703+0000","last_undegraded":"2024-03-06T07:27:10.133703+0000","last_fullsized":"2024-03-06T07:27:10.133703+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.134084+0000","last_change":"2024-03-06T07:27:10.134084+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.134084+0000","last_undegraded":"2024-03-06T07:27:10.134084+0000","last_fullsized":"2024-03-06T07:27:10.134084+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.139490+0000","last_change":"2024-03-06T07:27:10.139490+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.139490+0000","last_undegraded":"2024-03-06T07:27:10.139490+0000","last_fullsized":"2024-03-06T07:27:10.139490+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.133392+0000","last_change":"2024-03-06T07:27:10.133392+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.133392+0000","last_undegraded":"2024-03-06T07:27:10.133392+0000","last_fullsized":"2024-03-06T07:27:10.133392+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.131525+0000","last_change":"2024-03-06T07:27:10.131525+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.131525+0000","last_undegraded":"2024-03-06T07:27:10.131525+0000","last_fullsized":"2024-03-06T07:27:10.131525+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.138692+0000","last_change":"2024-03-06T07:27:10.138692+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.138692+0000","last_undegraded":"2024-03-06T07:27:10.138692+0000","last_fullsized":"2024-03-06T07:27:10.138692+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.131673+0000","last_change":"2024-03-06T07:27:10.131673+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.131673+0000","last_undegraded":"2024-03-06T07:27:10.131673+0000","last_fullsized":"2024-03-06T07:27:10.131673+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.138495+0000","last_change":"2024-03-06T07:27:10.138495+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.138495+0000","last_undegraded":"2024-03-06T07:27:10.138495+0000","last_fullsized":"2024-03-06T07:27:10.138495+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.132448+0000","last_change":"2024-03-06T07:27:10.132448+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.132448+0000","last_undegraded":"2024-03-06T07:27:10.132448+0000","last_fullsized":"2024-03-06T07:27:10.132448+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.139794+0000","last_change":"2024-03-06T07:27:10.139794+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.139794+0000","last_undegraded":"2024-03-06T07:27:10.139794+0000","last_fullsized":"2024-03-06T07:27:10.139794+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.141443+0000","last_change":"2024-03-06T07:27:10.141443+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.141443+0000","last_undegraded":"2024-03-06T07:27:10.141443+0000","last_fullsized":"2024-03-06T07:27:10.141443+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.139294+0000","last_change":"2024-03-06T07:27:10.139294+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.139294+0000","last_undegraded":"2024-03-06T07:27:10.139294+0000","last_fullsized":"2024-03-06T07:27:10.139294+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.134116+0000","last_change":"2024-03-06T07:27:10.134116+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.134116+0000","last_undegraded":"2024-03-06T07:27:10.134116+0000","last_fullsized":"2024-03-06T07:27:10.134116+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.134159+0000","last_change":"2024-03-06T07:27:10.134159+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.134159+0000","last_undegraded":"2024-03-06T07:27:10.134159+0000","last_fullsized":"2024-03-06T07:27:10.134159+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.141268+0000","last_change":"2024-03-06T07:27:10.141268+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.141268+0000","last_undegraded":"2024-03-06T07:27:10.141268+0000","last_fullsized":"2024-03-06T07:27:10.141268+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.140860+0000","last_change":"2024-03-06T07:27:10.140860+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.140860+0000","last_undegraded":"2024-03-06T07:27:10.140860+0000","last_fullsized":"2024-03-06T07:27:10.140860+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.140859+0000","last_change":"2024-03-06T07:27:10.140859+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.140859+0000","last_undegraded":"2024-03-06T07:27:10.140859+0000","last_fullsized":"2024-03-06T07:27:10.140859+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.144441+0000","last_change":"2024-03-06T07:27:10.143561+0000","last_active":"2024-03-06T07:27:10.144441+0000","last_peered":"2024-03-06T07:27:10.144441+0000","last_clean":"2024-03-06T07:27:10.144441+0000","last_became_active":"2024-03-06T07:27:10.143244+0000","last_became_peered":"2024-03-06T07:27:10.143244+0000","last_unstale":"2024-03-06T07:27:10.144441+0000","last_undegraded":"2024-03-06T07:27:10.144441+0000","last_fullsized":"2024-03-06T07:27:10.144441+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:31:13.932038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.135118+0000","last_change":"2024-03-06T07:27:10.135118+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.135118+0000","last_undegraded":"2024-03-06T07:27:10.135118+0000","last_fullsized":"2024-03-06T07:27:10.135118+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.146696+0000","last_change":"2024-03-06T07:27:10.145366+0000","last_active":"2024-03-06T07:27:10.146696+0000","last_peered":"2024-03-06T07:27:10.146696+0000","last_clean":"2024-03-06T07:27:10.146696+0000","last_became_active":"2024-03-06T07:27:10.145028+0000","last_became_peered":"2024-03-06T07:27:10.145028+0000","last_unstale":"2024-03-06T07:27:10.146696+0000","last_undegraded":"2024-03-06T07:27:10.146696+0000","last_fullsized":"2024-03-06T07:27:10.146696+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:56:36.810364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.141075+0000","last_change":"2024-03-06T07:27:10.141075+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.141075+0000","last_undegraded":"2024-03-06T07:27:10.141075+0000","last_fullsized":"2024-03-06T07:27:10.141075+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.148901+0000","last_change":"2024-03-06T07:27:10.146400+0000","last_active":"2024-03-06T07:27:10.148901+0000","last_peered":"2024-03-06T07:27:10.148901+0000","last_clean":"2024-03-06T07:27:10.148901+0000","last_became_active":"2024-03-06T07:27:10.146106+0000","last_became_peered":"2024-03-06T07:27:10.146106+0000","last_unstale":"2024-03-06T07:27:10.148901+0000","last_undegraded":"2024-03-06T07:27:10.148901+0000","last_fullsized":"2024-03-06T07:27:10.148901+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:24:07.968719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.137861+0000","last_change":"2024-03-06T07:27:10.137861+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.137861+0000","last_undegraded":"2024-03-06T07:27:10.137861+0000","last_fullsized":"2024-03-06T07:27:10.137861+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.140038+0000","last_change":"2024-03-06T07:27:10.137036+0000","last_active":"2024-03-06T07:27:10.140038+0000","last_peered":"2024-03-06T07:27:10.140038+0000","last_clean":"2024-03-06T07:27:10.140038+0000","last_became_active":"2024-03-06T07:27:10.136771+0000","last_became_peered":"2024-03-06T07:27:10.136771+0000","last_unstale":"2024-03-06T07:27:10.140038+0000","last_undegraded":"2024-03-06T07:27:10.140038+0000","last_fullsized":"2024-03-06T07:27:10.140038+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:03:45.041901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.139239+0000","last_change":"2024-03-06T07:27:10.139239+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.139239+0000","last_undegraded":"2024-03-06T07:27:10.139239+0000","last_fullsized":"2024-03-06T07:27:10.139239+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.153078+0000","last_change":"2024-03-06T07:27:10.152619+0000","last_active":"2024-03-06T07:27:10.153078+0000","last_peered":"2024-03-06T07:27:10.153078+0000","last_clean":"2024-03-06T07:27:10.153078+0000","last_became_active":"2024-03-06T07:27:10.152377+0000","last_became_peered":"2024-03-06T07:27:10.152377+0000","last_unstale":"2024-03-06T07:27:10.153078+0000","last_undegraded":"2024-03-06T07:27:10.153078+0000","last_fullsized":"2024-03-06T07:27:10.153078+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:30.226075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.133490+0000","last_change":"2024-03-06T07:27:10.133490+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.133490+0000","last_undegraded":"2024-03-06T07:27:10.133490+0000","last_fullsized":"2024-03-06T07:27:10.133490+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.151980+0000","last_change":"2024-03-06T07:27:10.148748+0000","last_active":"2024-03-06T07:27:10.151980+0000","last_peered":"2024-03-06T07:27:10.151980+0000","last_clean":"2024-03-06T07:27:10.151980+0000","last_became_active":"2024-03-06T07:27:10.148438+0000","last_became_peered":"2024-03-06T07:27:10.148438+0000","last_unstale":"2024-03-06T07:27:10.151980+0000","last_undegraded":"2024-03-06T07:27:10.151980+0000","last_fullsized":"2024-03-06T07:27:10.151980+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:09.417957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.139850+0000","last_change":"2024-03-06T07:27:10.139850+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.139850+0000","last_undegraded":"2024-03-06T07:27:10.139850+0000","last_fullsized":"2024-03-06T07:27:10.139850+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.149200+0000","last_change":"2024-03-06T07:27:10.147680+0000","last_active":"2024-03-06T07:27:10.149200+0000","last_peered":"2024-03-06T07:27:10.149200+0000","last_clean":"2024-03-06T07:27:10.149200+0000","last_became_active":"2024-03-06T07:27:10.147438+0000","last_became_peered":"2024-03-06T07:27:10.147438+0000","last_unstale":"2024-03-06T07:27:10.149200+0000","last_undegraded":"2024-03-06T07:27:10.149200+0000","last_fullsized":"2024-03-06T07:27:10.149200+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:57:44.152110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.135641+0000","last_change":"2024-03-06T07:27:10.135641+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.135641+0000","last_undegraded":"2024-03-06T07:27:10.135641+0000","last_fullsized":"2024-03-06T07:27:10.135641+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.153973+0000","last_change":"2024-03-06T07:27:10.149089+0000","last_active":"2024-03-06T07:27:10.153973+0000","last_peered":"2024-03-06T07:27:10.153973+0000","last_clean":"2024-03-06T07:27:10.153973+0000","last_became_active":"2024-03-06T07:27:10.148126+0000","last_became_peered":"2024-03-06T07:27:10.148126+0000","last_unstale":"2024-03-06T07:27:10.153973+0000","last_undegraded":"2024-03-06T07:27:10.153973+0000","last_fullsized":"2024-03-06T07:27:10.153973+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:23:31.120689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.135927+0000","last_change":"2024-03-06T07:27:10.135927+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.135927+0000","last_undegraded":"2024-03-06T07:27:10.135927+0000","last_fullsized":"2024-03-06T07:27:10.135927+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.149657+0000","last_change":"2024-03-06T07:27:10.146728+0000","last_active":"2024-03-06T07:27:10.149657+0000","last_peered":"2024-03-06T07:27:10.149657+0000","last_clean":"2024-03-06T07:27:10.149657+0000","last_became_active":"2024-03-06T07:27:10.146456+0000","last_became_peered":"2024-03-06T07:27:10.146456+0000","last_unstale":"2024-03-06T07:27:10.149657+0000","last_undegraded":"2024-03-06T07:27:10.149657+0000","last_fullsized":"2024-03-06T07:27:10.149657+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:26.196020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.134728+0000","last_change":"2024-03-06T07:27:10.134728+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.134728+0000","last_undegraded":"2024-03-06T07:27:10.134728+0000","last_fullsized":"2024-03-06T07:27:10.134728+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.151549+0000","last_change":"2024-03-06T07:27:10.148013+0000","last_active":"2024-03-06T07:27:10.151549+0000","last_peered":"2024-03-06T07:27:10.151549+0000","last_clean":"2024-03-06T07:27:10.151549+0000","last_became_active":"2024-03-06T07:27:10.147723+0000","last_became_peered":"2024-03-06T07:27:10.147723+0000","last_unstale":"2024-03-06T07:27:10.151549+0000","last_undegraded":"2024-03-06T07:27:10.151549+0000","last_fullsized":"2024-03-06T07:27:10.151549+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:41.083659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.138899+0000","last_change":"2024-03-06T07:27:10.138899+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.138899+0000","last_undegraded":"2024-03-06T07:27:10.138899+0000","last_fullsized":"2024-03-06T07:27:10.138899+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.151643+0000","last_change":"2024-03-06T07:27:10.145307+0000","last_active":"2024-03-06T07:27:10.151643+0000","last_peered":"2024-03-06T07:27:10.151643+0000","last_clean":"2024-03-06T07:27:10.151643+0000","last_became_active":"2024-03-06T07:27:10.145027+0000","last_became_peered":"2024-03-06T07:27:10.145027+0000","last_unstale":"2024-03-06T07:27:10.151643+0000","last_undegraded":"2024-03-06T07:27:10.151643+0000","last_fullsized":"2024-03-06T07:27:10.151643+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:49:40.057171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.134203+0000","last_change":"2024-03-06T07:27:10.134203+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.134203+0000","last_undegraded":"2024-03-06T07:27:10.134203+0000","last_fullsized":"2024-03-06T07:27:10.134203+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.154023+0000","last_change":"2024-03-06T07:27:10.147579+0000","last_active":"2024-03-06T07:27:10.154023+0000","last_peered":"2024-03-06T07:27:10.154023+0000","last_clean":"2024-03-06T07:27:10.154023+0000","last_became_active":"2024-03-06T07:27:10.147292+0000","last_became_peered":"2024-03-06T07:27:10.147292+0000","last_unstale":"2024-03-06T07:27:10.154023+0000","last_undegraded":"2024-03-06T07:27:10.154023+0000","last_fullsized":"2024-03-06T07:27:10.154023+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:38.297655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.138207+0000","last_change":"2024-03-06T07:27:10.138207+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.138207+0000","last_undegraded":"2024-03-06T07:27:10.138207+0000","last_fullsized":"2024-03-06T07:27:10.138207+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.152149+0000","last_change":"2024-03-06T07:27:10.145448+0000","last_active":"2024-03-06T07:27:10.152149+0000","last_peered":"2024-03-06T07:27:10.152149+0000","last_clean":"2024-03-06T07:27:10.152149+0000","last_became_active":"2024-03-06T07:27:10.145063+0000","last_became_peered":"2024-03-06T07:27:10.145063+0000","last_unstale":"2024-03-06T07:27:10.152149+0000","last_undegraded":"2024-03-06T07:27:10.152149+0000","last_fullsized":"2024-03-06T07:27:10.152149+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:14:01.327671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.137691+0000","last_change":"2024-03-06T07:27:10.137691+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.137691+0000","last_undegraded":"2024-03-06T07:27:10.137691+0000","last_fullsized":"2024-03-06T07:27:10.137691+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.149341+0000","last_change":"2024-03-06T07:27:10.146708+0000","last_active":"2024-03-06T07:27:10.149341+0000","last_peered":"2024-03-06T07:27:10.149341+0000","last_clean":"2024-03-06T07:27:10.149341+0000","last_became_active":"2024-03-06T07:27:10.146387+0000","last_became_peered":"2024-03-06T07:27:10.146387+0000","last_unstale":"2024-03-06T07:27:10.149341+0000","last_undegraded":"2024-03-06T07:27:10.149341+0000","last_fullsized":"2024-03-06T07:27:10.149341+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:12.816920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.139264+0000","last_change":"2024-03-06T07:27:10.139264+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.139264+0000","last_undegraded":"2024-03-06T07:27:10.139264+0000","last_fullsized":"2024-03-06T07:27:10.139264+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.139998+0000","last_change":"2024-03-06T07:27:10.136377+0000","last_active":"2024-03-06T07:27:10.139998+0000","last_peered":"2024-03-06T07:27:10.139998+0000","last_clean":"2024-03-06T07:27:10.139998+0000","last_became_active":"2024-03-06T07:27:10.136062+0000","last_became_peered":"2024-03-06T07:27:10.136062+0000","last_unstale":"2024-03-06T07:27:10.139998+0000","last_undegraded":"2024-03-06T07:27:10.139998+0000","last_fullsized":"2024-03-06T07:27:10.139998+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:44:20.435658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.133922+0000","last_change":"2024-03-06T07:27:10.133922+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.133922+0000","last_undegraded":"2024-03-06T07:27:10.133922+0000","last_fullsized":"2024-03-06T07:27:10.133922+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.153881+0000","last_change":"2024-03-06T07:27:10.147575+0000","last_active":"2024-03-06T07:27:10.153881+0000","last_peered":"2024-03-06T07:27:10.153881+0000","last_clean":"2024-03-06T07:27:10.153881+0000","last_became_active":"2024-03-06T07:27:10.147304+0000","last_became_peered":"2024-03-06T07:27:10.147304+0000","last_unstale":"2024-03-06T07:27:10.153881+0000","last_undegraded":"2024-03-06T07:27:10.153881+0000","last_fullsized":"2024-03-06T07:27:10.153881+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:19:26.310561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.140347+0000","last_change":"2024-03-06T07:27:10.140347+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.140347+0000","last_undegraded":"2024-03-06T07:27:10.140347+0000","last_fullsized":"2024-03-06T07:27:10.140347+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.140158+0000","last_change":"2024-03-06T07:27:10.137380+0000","last_active":"2024-03-06T07:27:10.140158+0000","last_peered":"2024-03-06T07:27:10.140158+0000","last_clean":"2024-03-06T07:27:10.140158+0000","last_became_active":"2024-03-06T07:27:10.137056+0000","last_became_peered":"2024-03-06T07:27:10.137056+0000","last_unstale":"2024-03-06T07:27:10.140158+0000","last_undegraded":"2024-03-06T07:27:10.140158+0000","last_fullsized":"2024-03-06T07:27:10.140158+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:12:04.974749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.135554+0000","last_change":"2024-03-06T07:27:10.135554+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.135554+0000","last_undegraded":"2024-03-06T07:27:10.135554+0000","last_fullsized":"2024-03-06T07:27:10.135554+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.146355+0000","last_change":"2024-03-06T07:27:10.145731+0000","last_active":"2024-03-06T07:27:10.146355+0000","last_peered":"2024-03-06T07:27:10.146355+0000","last_clean":"2024-03-06T07:27:10.146355+0000","last_became_active":"2024-03-06T07:27:10.145424+0000","last_became_peered":"2024-03-06T07:27:10.145424+0000","last_unstale":"2024-03-06T07:27:10.146355+0000","last_undegraded":"2024-03-06T07:27:10.146355+0000","last_fullsized":"2024-03-06T07:27:10.146355+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:09.417957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.137107+0000","last_change":"2024-03-06T07:27:10.137107+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.137107+0000","last_undegraded":"2024-03-06T07:27:10.137107+0000","last_fullsized":"2024-03-06T07:27:10.137107+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.151758+0000","last_change":"2024-03-06T07:27:10.151277+0000","last_active":"2024-03-06T07:27:10.151758+0000","last_peered":"2024-03-06T07:27:10.151758+0000","last_clean":"2024-03-06T07:27:10.151758+0000","last_became_active":"2024-03-06T07:27:10.150933+0000","last_became_peered":"2024-03-06T07:27:10.150933+0000","last_unstale":"2024-03-06T07:27:10.151758+0000","last_undegraded":"2024-03-06T07:27:10.151758+0000","last_fullsized":"2024-03-06T07:27:10.151758+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:16.105130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.135497+0000","last_change":"2024-03-06T07:27:10.135497+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.135497+0000","last_undegraded":"2024-03-06T07:27:10.135497+0000","last_fullsized":"2024-03-06T07:27:10.135497+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.154056+0000","last_change":"2024-03-06T07:27:10.150893+0000","last_active":"2024-03-06T07:27:10.154056+0000","last_peered":"2024-03-06T07:27:10.154056+0000","last_clean":"2024-03-06T07:27:10.154056+0000","last_became_active":"2024-03-06T07:27:10.150648+0000","last_became_peered":"2024-03-06T07:27:10.150648+0000","last_unstale":"2024-03-06T07:27:10.154056+0000","last_undegraded":"2024-03-06T07:27:10.154056+0000","last_fullsized":"2024-03-06T07:27:10.154056+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:34.387756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.139184+0000","last_change":"2024-03-06T07:27:10.139184+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.139184+0000","last_undegraded":"2024-03-06T07:27:10.139184+0000","last_fullsized":"2024-03-06T07:27:10.139184+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.151835+0000","last_change":"2024-03-06T07:27:10.148882+0000","last_active":"2024-03-06T07:27:10.151835+0000","last_peered":"2024-03-06T07:27:10.151835+0000","last_clean":"2024-03-06T07:27:10.151835+0000","last_became_active":"2024-03-06T07:27:10.148626+0000","last_became_peered":"2024-03-06T07:27:10.148626+0000","last_unstale":"2024-03-06T07:27:10.151835+0000","last_undegraded":"2024-03-06T07:27:10.151835+0000","last_fullsized":"2024-03-06T07:27:10.151835+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:19.034867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.136666+0000","last_change":"2024-03-06T07:27:10.136666+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.136666+0000","last_undegraded":"2024-03-06T07:27:10.136666+0000","last_fullsized":"2024-03-06T07:27:10.136666+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.150333+0000","last_change":"2024-03-06T07:27:10.144592+0000","last_active":"2024-03-06T07:27:10.150333+0000","last_peered":"2024-03-06T07:27:10.150333+0000","last_clean":"2024-03-06T07:27:10.150333+0000","last_became_active":"2024-03-06T07:27:10.144328+0000","last_became_peered":"2024-03-06T07:27:10.144328+0000","last_unstale":"2024-03-06T07:27:10.150333+0000","last_undegraded":"2024-03-06T07:27:10.150333+0000","last_fullsized":"2024-03-06T07:27:10.150333+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:39:04.350267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.138259+0000","last_change":"2024-03-06T07:27:10.138259+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.138259+0000","last_undegraded":"2024-03-06T07:27:10.138259+0000","last_fullsized":"2024-03-06T07:27:10.138259+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.154069+0000","last_change":"2024-03-06T07:27:10.151020+0000","last_active":"2024-03-06T07:27:10.154069+0000","last_peered":"2024-03-06T07:27:10.154069+0000","last_clean":"2024-03-06T07:27:10.154069+0000","last_became_active":"2024-03-06T07:27:10.150747+0000","last_became_peered":"2024-03-06T07:27:10.150747+0000","last_unstale":"2024-03-06T07:27:10.154069+0000","last_undegraded":"2024-03-06T07:27:10.154069+0000","last_fullsized":"2024-03-06T07:27:10.154069+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:28:32.938413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.133669+0000","last_change":"2024-03-06T07:27:10.133669+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.133669+0000","last_undegraded":"2024-03-06T07:27:10.133669+0000","last_fullsized":"2024-03-06T07:27:10.133669+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.151911+0000","last_change":"2024-03-06T07:27:10.150171+0000","last_active":"2024-03-06T07:27:10.151911+0000","last_peered":"2024-03-06T07:27:10.151911+0000","last_clean":"2024-03-06T07:27:10.151911+0000","last_became_active":"2024-03-06T07:27:10.149895+0000","last_became_peered":"2024-03-06T07:27:10.149895+0000","last_unstale":"2024-03-06T07:27:10.151911+0000","last_undegraded":"2024-03-06T07:27:10.151911+0000","last_fullsized":"2024-03-06T07:27:10.151911+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:12.816920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.138704+0000","last_change":"2024-03-06T07:27:10.138704+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.138704+0000","last_undegraded":"2024-03-06T07:27:10.138704+0000","last_fullsized":"2024-03-06T07:27:10.138704+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.146334+0000","last_change":"2024-03-06T07:27:10.142952+0000","last_active":"2024-03-06T07:27:10.146334+0000","last_peered":"2024-03-06T07:27:10.146334+0000","last_clean":"2024-03-06T07:27:10.146334+0000","last_became_active":"2024-03-06T07:27:10.142643+0000","last_became_peered":"2024-03-06T07:27:10.142643+0000","last_unstale":"2024-03-06T07:27:10.146334+0000","last_undegraded":"2024-03-06T07:27:10.146334+0000","last_fullsized":"2024-03-06T07:27:10.146334+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:41.083659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.138689+0000","last_change":"2024-03-06T07:27:10.138689+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.138689+0000","last_undegraded":"2024-03-06T07:27:10.138689+0000","last_fullsized":"2024-03-06T07:27:10.138689+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.153932+0000","last_change":"2024-03-06T07:27:10.148289+0000","last_active":"2024-03-06T07:27:10.153932+0000","last_peered":"2024-03-06T07:27:10.153932+0000","last_clean":"2024-03-06T07:27:10.153932+0000","last_became_active":"2024-03-06T07:27:10.148026+0000","last_became_peered":"2024-03-06T07:27:10.148026+0000","last_unstale":"2024-03-06T07:27:10.153932+0000","last_undegraded":"2024-03-06T07:27:10.153932+0000","last_fullsized":"2024-03-06T07:27:10.153932+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:25.955212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.132879+0000","last_change":"2024-03-06T07:27:10.132879+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.132879+0000","last_undegraded":"2024-03-06T07:27:10.132879+0000","last_fullsized":"2024-03-06T07:27:10.132879+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.153944+0000","last_change":"2024-03-06T07:27:10.151397+0000","last_active":"2024-03-06T07:27:10.153944+0000","last_peered":"2024-03-06T07:27:10.153944+0000","last_clean":"2024-03-06T07:27:10.153944+0000","last_became_active":"2024-03-06T07:27:10.150827+0000","last_became_peered":"2024-03-06T07:27:10.150827+0000","last_unstale":"2024-03-06T07:27:10.153944+0000","last_undegraded":"2024-03-06T07:27:10.153944+0000","last_fullsized":"2024-03-06T07:27:10.153944+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:52.397388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.131712+0000","last_change":"2024-03-06T07:27:10.131712+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.131712+0000","last_undegraded":"2024-03-06T07:27:10.131712+0000","last_fullsized":"2024-03-06T07:27:10.131712+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.142168+0000","last_change":"2024-03-06T07:27:10.141228+0000","last_active":"2024-03-06T07:27:10.142168+0000","last_peered":"2024-03-06T07:27:10.142168+0000","last_clean":"2024-03-06T07:27:10.142168+0000","last_became_active":"2024-03-06T07:27:10.140919+0000","last_became_peered":"2024-03-06T07:27:10.140919+0000","last_unstale":"2024-03-06T07:27:10.142168+0000","last_undegraded":"2024-03-06T07:27:10.142168+0000","last_fullsized":"2024-03-06T07:27:10.142168+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:55.019970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.132772+0000","last_change":"2024-03-06T07:27:10.132772+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.132772+0000","last_undegraded":"2024-03-06T07:27:10.132772+0000","last_fullsized":"2024-03-06T07:27:10.132772+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.141015+0000","last_change":"2024-03-06T07:27:10.138499+0000","last_active":"2024-03-06T07:27:10.141015+0000","last_peered":"2024-03-06T07:27:10.141015+0000","last_clean":"2024-03-06T07:27:10.141015+0000","last_became_active":"2024-03-06T07:27:10.138222+0000","last_became_peered":"2024-03-06T07:27:10.138222+0000","last_unstale":"2024-03-06T07:27:10.141015+0000","last_undegraded":"2024-03-06T07:27:10.141015+0000","last_fullsized":"2024-03-06T07:27:10.141015+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:18:48.877918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.136141+0000","last_change":"2024-03-06T07:27:10.136141+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.136141+0000","last_undegraded":"2024-03-06T07:27:10.136141+0000","last_fullsized":"2024-03-06T07:27:10.136141+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.149422+0000","last_change":"2024-03-06T07:27:10.147408+0000","last_active":"2024-03-06T07:27:10.149422+0000","last_peered":"2024-03-06T07:27:10.149422+0000","last_clean":"2024-03-06T07:27:10.149422+0000","last_became_active":"2024-03-06T07:27:10.147159+0000","last_became_peered":"2024-03-06T07:27:10.147159+0000","last_unstale":"2024-03-06T07:27:10.149422+0000","last_undegraded":"2024-03-06T07:27:10.149422+0000","last_fullsized":"2024-03-06T07:27:10.149422+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:28:37.262533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.140866+0000","last_change":"2024-03-06T07:27:10.140866+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.140866+0000","last_undegraded":"2024-03-06T07:27:10.140866+0000","last_fullsized":"2024-03-06T07:27:10.140866+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.153996+0000","last_change":"2024-03-06T07:27:10.149308+0000","last_active":"2024-03-06T07:27:10.153996+0000","last_peered":"2024-03-06T07:27:10.153996+0000","last_clean":"2024-03-06T07:27:10.153996+0000","last_became_active":"2024-03-06T07:27:10.148472+0000","last_became_peered":"2024-03-06T07:27:10.148472+0000","last_unstale":"2024-03-06T07:27:10.153996+0000","last_undegraded":"2024-03-06T07:27:10.153996+0000","last_fullsized":"2024-03-06T07:27:10.153996+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:35:41.626248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":1,"reported_epoch":108,"state":"creating+peering","last_fresh":"2024-03-06T07:27:10.137469+0000","last_change":"2024-03-06T07:27:10.137469+0000","last_active":"2024-03-06T07:27:10.126521+0000","last_peered":"2024-03-06T07:27:10.126521+0000","last_clean":"2024-03-06T07:27:10.126521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:10.137469+0000","last_undegraded":"2024-03-06T07:27:10.137469+0000","last_fullsized":"2024-03-06T07:27:10.137469+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":9,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.151732+0000","last_change":"2024-03-06T07:27:10.151075+0000","last_active":"2024-03-06T07:27:10.151732+0000","last_peered":"2024-03-06T07:27:10.151732+0000","last_clean":"2024-03-06T07:27:10.151732+0000","last_became_active":"2024-03-06T07:27:10.150826+0000","last_became_peered":"2024-03-06T07:27:10.150826+0000","last_unstale":"2024-03-06T07:27:10.151732+0000","last_undegraded":"2024-03-06T07:27:10.151732+0000","last_fullsized":"2024-03-06T07:27:10.151732+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:20:12.450522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":327,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:10.139798+0000","last_change":"2024-03-06T07:21:47.553693+0000","last_active":"2024-03-06T07:27:10.139798+0000","last_peered":"2024-03-06T07:27:10.139798+0000","last_clean":"2024-03-06T07:27:10.139798+0000","last_became_active":"2024-03-06T07:21:47.552664+0000","last_became_peered":"2024-03-06T07:21:47.552664+0000","last_unstale":"2024-03-06T07:27:10.139798+0000","last_undegraded":"2024-03-06T07:27:10.139798+0000","last_fullsized":"2024-03-06T07:27:10.139798+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_clean_scrub_stamp":"2024-03-06T07:21:46.362028+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:02.916426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574918,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37236,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334604,"statfs":{"total":96636764160,"available":96598634496,"internally_reserved":0,"allocated":1937408,"data_stored":1166972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":36174281},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574916,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40800,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331040,"statfs":{"total":96636764160,"available":96594984960,"internally_reserved":0,"allocated":1331200,"data_stored":570329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574917,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40816,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331024,"statfs":{"total":96636764160,"available":96594968576,"internally_reserved":0,"allocated":1343488,"data_stored":576604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574918,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40816,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331024,"statfs":{"total":96636764160,"available":96594968576,"internally_reserved":0,"allocated":1343488,"data_stored":576604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574917,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40800,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331040,"statfs":{"total":96636764160,"available":96594984960,"internally_reserved":0,"allocated":1331200,"data_stored":570329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574917,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40804,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331036,"statfs":{"total":96636764160,"available":96594980864,"internally_reserved":0,"allocated":1331200,"data_stored":570329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574918,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40816,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331024,"statfs":{"total":96636764160,"available":96594968576,"internally_reserved":0,"allocated":1343488,"data_stored":576604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574917,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41384,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330456,"statfs":{"total":96636764160,"available":96594386944,"internally_reserved":0,"allocated":1925120,"data_stored":1160697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:16.403 INFO:tasks.ceph.mds.a.smithi083.stderr:2024-03-06T07:27:16.401+0000 7f3719071640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:27:18.676 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:18.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.777+0000 7fe575cc9640 1 Processor -- start 2024-03-06T07:27:18.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.777+0000 7fe575cc9640 1 -- start start 2024-03-06T07:27:18.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.778+0000 7fe575cc9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe5700d4760 0x7fe5700b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.778+0000 7fe575cc9640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5700d25d0 con 0x7fe5700b4170 2024-03-06T07:27:18.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.778+0000 7fe575cc9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5700d2870 con 0x7fe5700d0a80 2024-03-06T07:27:18.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.778+0000 7fe575cc9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5700d4bf0 con 0x7fe5700d4760 2024-03-06T07:27:18.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.779+0000 7fe56f7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fe5700b4170 0x7fe5700d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57074/0 (socket says 172.21.15.83:57074) 2024-03-06T07:27:18.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.779+0000 7fe56f7fe640 1 -- 172.21.15.83:0/3421230528 learned_addr learned my addr 172.21.15.83:0/3421230528 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:27:18.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.779+0000 7fe56e7fc640 1 -- 172.21.15.83:0/3421230528 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3299356078 0 0) 0x7fe5700d25d0 con 0x7fe5700b4170 2024-03-06T07:27:18.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.779+0000 7fe56e7fc640 1 -- 172.21.15.83:0/3421230528 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe55c003520 con 0x7fe5700b4170 2024-03-06T07:27:18.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.779+0000 7fe56effd640 1 --2- 172.21.15.83:0/3421230528 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe5700d4760 0x7fe5700b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.779+0000 7fe56e7fc640 1 -- 172.21.15.83:0/3421230528 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2972086351 0 0) 0x7fe55c003520 con 0x7fe5700b4170 2024-03-06T07:27:18.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.779+0000 7fe56e7fc640 1 -- 172.21.15.83:0/3421230528 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe5700d4760 msgr2=0x7fe5700b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:18.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.779+0000 7fe56e7fc640 1 --2- 172.21.15.83:0/3421230528 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe5700d4760 0x7fe5700b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.779+0000 7fe56e7fc640 1 -- 172.21.15.83:0/3421230528 >> v1:172.21.15.183:6789/0 conn(0x7fe5700d0a80 legacy=0x7fe5700cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.779+0000 7fe56e7fc640 1 -- 172.21.15.83:0/3421230528 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5700d2b50 con 0x7fe5700b4170 2024-03-06T07:27:18.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.779+0000 7fe56e7fc640 1 -- 172.21.15.83:0/3421230528 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7fe560001670 con 0x7fe5700b4170 2024-03-06T07:27:18.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.779+0000 7fe56effd640 1 --2- 172.21.15.83:0/3421230528 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe5700d4760 0x7fe5700b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:18.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.779+0000 7fe56e7fc640 1 -- 172.21.15.83:0/3421230528 >> v1:172.21.15.83:6789/0 conn(0x7fe5700b4170 legacy=0x7fe5700d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.779+0000 7fe56e7fc640 1 --2- 172.21.15.83:0/3421230528 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe55c0039e0 0x7fe55c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.779+0000 7fe56e7fc640 1 --2- 172.21.15.83:0/3421230528 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe55c004800 0x7fe55c004c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.779+0000 7fe56e7fc640 1 --2- 172.21.15.83:0/3421230528 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe55c005140 0x7fe55c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.779+0000 7fe56e7fc640 1 -- 172.21.15.83:0/3421230528 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5700d4bf0 con 0x7fe55c0039e0 2024-03-06T07:27:18.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.779+0000 7fe56e7fc640 1 -- 172.21.15.83:0/3421230528 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe55c00bf00 con 0x7fe55c005140 2024-03-06T07:27:18.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.779+0000 7fe56e7fc640 1 -- 172.21.15.83:0/3421230528 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe55c00c100 con 0x7fe55c004800 2024-03-06T07:27:18.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.779+0000 7fe56f7fe640 1 --2- 172.21.15.83:0/3421230528 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe55c0039e0 0x7fe55c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.779+0000 7fe56effd640 1 --2- 172.21.15.83:0/3421230528 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe55c004800 0x7fe55c004c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.779+0000 7fe56ffff640 1 --2- 172.21.15.83:0/3421230528 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe55c005140 0x7fe55c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.779+0000 7fe56f7fe640 1 -- 172.21.15.83:0/3421230528 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe55c004800 msgr2=0x7fe55c004c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:18.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.780+0000 7fe56f7fe640 1 --2- 172.21.15.83:0/3421230528 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe55c004800 0x7fe55c004c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.780+0000 7fe56f7fe640 1 -- 172.21.15.83:0/3421230528 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe55c005140 msgr2=0x7fe55c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:18.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.780+0000 7fe56f7fe640 1 --2- 172.21.15.83:0/3421230528 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe55c005140 0x7fe55c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.780+0000 7fe56f7fe640 1 -- 172.21.15.83:0/3421230528 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe55c003520 con 0x7fe55c0039e0 2024-03-06T07:27:18.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.780+0000 7fe56f7fe640 1 --2- 172.21.15.83:0/3421230528 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe55c0039e0 0x7fe55c003e00 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fe56000c210 tx=0x7fe55c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e847f77f787789c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:18.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.780+0000 7fe56e7fc640 1 -- 172.21.15.83:0/3421230528 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe560007d30 con 0x7fe55c0039e0 2024-03-06T07:27:18.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.780+0000 7fe56e7fc640 1 -- 172.21.15.83:0/3421230528 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe560030070 con 0x7fe55c0039e0 2024-03-06T07:27:18.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.780+0000 7fe56e7fc640 1 -- 172.21.15.83:0/3421230528 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56002cd60 con 0x7fe55c0039e0 2024-03-06T07:27:18.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.781+0000 7fe575cc9640 1 -- 172.21.15.83:0/3421230528 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe55c0039e0 msgr2=0x7fe55c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.781+0000 7fe575cc9640 1 --2- 172.21.15.83:0/3421230528 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe55c0039e0 0x7fe55c003e00 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fe56000c210 tx=0x7fe55c005830 comp rx=0 tx=0).stop 2024-03-06T07:27:18.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.781+0000 7fe56ffff640 1 -- 172.21.15.83:0/3421230528 reap_dead start 2024-03-06T07:27:18.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.781+0000 7fe575cc9640 1 -- 172.21.15.83:0/3421230528 shutdown_connections 2024-03-06T07:27:18.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.781+0000 7fe575cc9640 1 -- 172.21.15.83:0/3421230528 >> 172.21.15.83:0/3421230528 conn(0x7fe5700d3c40 msgr2=0x7fe5700d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:18.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.781+0000 7fe575cc9640 1 -- 172.21.15.83:0/3421230528 shutdown_connections 2024-03-06T07:27:18.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.781+0000 7fe575cc9640 1 -- 172.21.15.83:0/3421230528 wait complete. 2024-03-06T07:27:18.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.783+0000 7fe575cc9640 1 Processor -- start 2024-03-06T07:27:18.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.783+0000 7fe575cc9640 1 -- start start 2024-03-06T07:27:18.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.784+0000 7fe575cc9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe55c0039e0 0x7fe5701f7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.784+0000 7fe575cc9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe570157e60 0x7fe5701f6870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.784+0000 7fe575cc9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe5701582c0 0x7fe5701f6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.784+0000 7fe575cc9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5700b4dd0 con 0x7fe570157e60 2024-03-06T07:27:18.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.784+0000 7fe575cc9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5700ce9e0 con 0x7fe55c0039e0 2024-03-06T07:27:18.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.784+0000 7fe575cc9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5700b4f50 con 0x7fe5701582c0 2024-03-06T07:27:18.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.784+0000 7fe56effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe570157e60 0x7fe5701f6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.784+0000 7fe56effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe570157e60 0x7fe5701f6870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41432/0 (socket says 172.21.15.83:41432) 2024-03-06T07:27:18.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.784+0000 7fe56effd640 1 -- 172.21.15.83:0/217465165 learned_addr learned my addr 172.21.15.83:0/217465165 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:18.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.784+0000 7fe56f7fe640 1 --2- 172.21.15.83:0/217465165 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe55c0039e0 0x7fe5701f7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.784+0000 7fe56ffff640 1 --2- 172.21.15.83:0/217465165 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe5701582c0 0x7fe5701f6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.784+0000 7fe56effd640 1 -- 172.21.15.83:0/217465165 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe5701582c0 msgr2=0x7fe5701f6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.784+0000 7fe56effd640 1 --2- 172.21.15.83:0/217465165 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe5701582c0 0x7fe5701f6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.784+0000 7fe56effd640 1 -- 172.21.15.83:0/217465165 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe55c0039e0 msgr2=0x7fe5701f7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.784+0000 7fe56effd640 1 --2- 172.21.15.83:0/217465165 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe55c0039e0 0x7fe5701f7f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.784+0000 7fe56effd640 1 -- 172.21.15.83:0/217465165 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe57014aab0 con 0x7fe570157e60 2024-03-06T07:27:18.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.784+0000 7fe56f7fe640 1 --2- 172.21.15.83:0/217465165 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe55c0039e0 0x7fe5701f7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:18.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.785+0000 7fe56effd640 1 --2- 172.21.15.83:0/217465165 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe570157e60 0x7fe5701f6870 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7fe558010860 tx=0x7fe558010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:18.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.785+0000 7fe567fff640 1 -- 172.21.15.83:0/217465165 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55801a070 con 0x7fe570157e60 2024-03-06T07:27:18.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.785+0000 7fe567fff640 1 -- 172.21.15.83:0/217465165 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe55800cb40 con 0x7fe570157e60 2024-03-06T07:27:18.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.785+0000 7fe575cc9640 1 -- 172.21.15.83:0/217465165 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe57014ad90 con 0x7fe570157e60 2024-03-06T07:27:18.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.785+0000 7fe567fff640 1 -- 172.21.15.83:0/217465165 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55800f040 con 0x7fe570157e60 2024-03-06T07:27:18.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.785+0000 7fe575cc9640 1 -- 172.21.15.83:0/217465165 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5700b18e0 con 0x7fe570157e60 2024-03-06T07:27:18.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.786+0000 7fe575cc9640 1 -- 172.21.15.83:0/217465165 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe57004f330 con 0x7fe570157e60 2024-03-06T07:27:18.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.786+0000 7fe567fff640 1 -- 172.21.15.83:0/217465165 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe558015470 con 0x7fe570157e60 2024-03-06T07:27:18.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.787+0000 7fe567fff640 1 --2- 172.21.15.83:0/217465165 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe55006b500 0x7fe55006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.787+0000 7fe567fff640 1 -- 172.21.15.83:0/217465165 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe55808a7d0 con 0x7fe570157e60 2024-03-06T07:27:18.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.787+0000 7fe56f7fe640 1 --2- 172.21.15.83:0/217465165 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe55006b500 0x7fe55006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.787+0000 7fe56f7fe640 1 --2- 172.21.15.83:0/217465165 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe55006b500 0x7fe55006d9c0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fe56000c210 tx=0x7fe560009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:18.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:18.791+0000 7fe567fff640 1 -- 172.21.15.83:0/217465165 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe5580595d0 con 0x7fe570157e60 2024-03-06T07:27:19.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.034+0000 7fe575cc9640 1 -- 172.21.15.83:0/217465165 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe5700d1490 con 0x7fe55006b500 2024-03-06T07:27:19.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.040+0000 7fe567fff640 1 -- 172.21.15.83:0/217465165 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252483 (secure 0 0 0) 0x7fe560082a60 con 0x7fe55006b500 2024-03-06T07:27:19.045 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:19.046 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-06T07:27:19.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.044+0000 7fe575cc9640 1 -- 172.21.15.83:0/217465165 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe55006b500 msgr2=0x7fe55006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.044+0000 7fe575cc9640 1 --2- 172.21.15.83:0/217465165 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe55006b500 0x7fe55006d9c0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fe56000c210 tx=0x7fe560009030 comp rx=0 tx=0).stop 2024-03-06T07:27:19.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.044+0000 7fe575cc9640 1 -- 172.21.15.83:0/217465165 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe570157e60 msgr2=0x7fe5701f6870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.044+0000 7fe575cc9640 1 --2- 172.21.15.83:0/217465165 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe570157e60 0x7fe5701f6870 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7fe558010860 tx=0x7fe558010d20 comp rx=0 tx=0).stop 2024-03-06T07:27:19.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.044+0000 7fe575cc9640 1 -- 172.21.15.83:0/217465165 shutdown_connections 2024-03-06T07:27:19.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.044+0000 7fe575cc9640 1 --2- 172.21.15.83:0/217465165 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe55006b500 0x7fe55006d9c0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.044+0000 7fe575cc9640 1 --2- 172.21.15.83:0/217465165 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe5701582c0 0x7fe5701f6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.044+0000 7fe575cc9640 1 --2- 172.21.15.83:0/217465165 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe570157e60 0x7fe5701f6870 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.044+0000 7fe575cc9640 1 --2- 172.21.15.83:0/217465165 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe55c0039e0 0x7fe5701f7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.044+0000 7fe575cc9640 1 -- 172.21.15.83:0/217465165 >> 172.21.15.83:0/217465165 conn(0x7fe5700d3c40 msgr2=0x7fe5701611b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:19.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.044+0000 7fe575cc9640 1 -- 172.21.15.83:0/217465165 shutdown_connections 2024-03-06T07:27:19.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.044+0000 7fe575cc9640 1 -- 172.21.15.83:0/217465165 wait complete. 2024-03-06T07:27:19.056 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":270,"stamp":"2024-03-06T07:27:18.378345+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":185,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":323756,"kb_used_data":11912,"kb_used_omap":12,"kb_used_meta":311731,"kb_avail":754650964,"statfs":{"total":773094113280,"available":772762587136,"internally_reserved":0,"allocated":12197888,"data_stored":5902988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12757,"internal_metadata":319213099},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001532"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.340989+0000","last_change":"2024-03-06T07:27:11.164458+0000","last_active":"2024-03-06T07:27:12.340989+0000","last_peered":"2024-03-06T07:27:12.340989+0000","last_clean":"2024-03-06T07:27:12.340989+0000","last_became_active":"2024-03-06T07:27:11.161895+0000","last_became_peered":"2024-03-06T07:27:11.161895+0000","last_unstale":"2024-03-06T07:27:12.340989+0000","last_undegraded":"2024-03-06T07:27:12.340989+0000","last_fullsized":"2024-03-06T07:27:12.340989+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:36:53.548112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.154032+0000","last_change":"2024-03-06T07:27:11.158649+0000","last_active":"2024-03-06T07:27:12.154032+0000","last_peered":"2024-03-06T07:27:12.154032+0000","last_clean":"2024-03-06T07:27:12.154032+0000","last_became_active":"2024-03-06T07:27:11.158364+0000","last_became_peered":"2024-03-06T07:27:11.158364+0000","last_unstale":"2024-03-06T07:27:12.154032+0000","last_undegraded":"2024-03-06T07:27:12.154032+0000","last_fullsized":"2024-03-06T07:27:12.154032+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:23:32.125301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.341520+0000","last_change":"2024-03-06T07:27:11.161062+0000","last_active":"2024-03-06T07:27:12.341520+0000","last_peered":"2024-03-06T07:27:12.341520+0000","last_clean":"2024-03-06T07:27:12.341520+0000","last_became_active":"2024-03-06T07:27:11.160743+0000","last_became_peered":"2024-03-06T07:27:11.160743+0000","last_unstale":"2024-03-06T07:27:12.341520+0000","last_undegraded":"2024-03-06T07:27:12.341520+0000","last_fullsized":"2024-03-06T07:27:12.341520+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:02.473244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.340785+0000","last_change":"2024-03-06T07:27:11.170930+0000","last_active":"2024-03-06T07:27:12.340785+0000","last_peered":"2024-03-06T07:27:12.340785+0000","last_clean":"2024-03-06T07:27:12.340785+0000","last_became_active":"2024-03-06T07:27:11.169973+0000","last_became_peered":"2024-03-06T07:27:11.169973+0000","last_unstale":"2024-03-06T07:27:12.340785+0000","last_undegraded":"2024-03-06T07:27:12.340785+0000","last_fullsized":"2024-03-06T07:27:12.340785+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:19:08.987395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.153010+0000","last_change":"2024-03-06T07:27:11.154601+0000","last_active":"2024-03-06T07:27:12.153010+0000","last_peered":"2024-03-06T07:27:12.153010+0000","last_clean":"2024-03-06T07:27:12.153010+0000","last_became_active":"2024-03-06T07:27:11.153863+0000","last_became_peered":"2024-03-06T07:27:11.153863+0000","last_unstale":"2024-03-06T07:27:12.153010+0000","last_undegraded":"2024-03-06T07:27:12.153010+0000","last_fullsized":"2024-03-06T07:27:12.153010+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:19.841476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.341474+0000","last_change":"2024-03-06T07:27:11.154830+0000","last_active":"2024-03-06T07:27:12.341474+0000","last_peered":"2024-03-06T07:27:12.341474+0000","last_clean":"2024-03-06T07:27:12.341474+0000","last_became_active":"2024-03-06T07:27:11.154539+0000","last_became_peered":"2024-03-06T07:27:11.154539+0000","last_unstale":"2024-03-06T07:27:12.341474+0000","last_undegraded":"2024-03-06T07:27:12.341474+0000","last_fullsized":"2024-03-06T07:27:12.341474+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:56.024582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.340682+0000","last_change":"2024-03-06T07:27:11.151737+0000","last_active":"2024-03-06T07:27:12.340682+0000","last_peered":"2024-03-06T07:27:12.340682+0000","last_clean":"2024-03-06T07:27:12.340682+0000","last_became_active":"2024-03-06T07:27:11.150994+0000","last_became_peered":"2024-03-06T07:27:11.150994+0000","last_unstale":"2024-03-06T07:27:12.340682+0000","last_undegraded":"2024-03-06T07:27:12.340682+0000","last_fullsized":"2024-03-06T07:27:12.340682+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:15.396305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.340793+0000","last_change":"2024-03-06T07:27:11.160650+0000","last_active":"2024-03-06T07:27:12.340793+0000","last_peered":"2024-03-06T07:27:12.340793+0000","last_clean":"2024-03-06T07:27:12.340793+0000","last_became_active":"2024-03-06T07:27:11.159326+0000","last_became_peered":"2024-03-06T07:27:11.159326+0000","last_unstale":"2024-03-06T07:27:12.340793+0000","last_undegraded":"2024-03-06T07:27:12.340793+0000","last_fullsized":"2024-03-06T07:27:12.340793+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:56:33.426454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.341038+0000","last_change":"2024-03-06T07:27:11.155203+0000","last_active":"2024-03-06T07:27:12.341038+0000","last_peered":"2024-03-06T07:27:12.341038+0000","last_clean":"2024-03-06T07:27:12.341038+0000","last_became_active":"2024-03-06T07:27:11.154929+0000","last_became_peered":"2024-03-06T07:27:11.154929+0000","last_unstale":"2024-03-06T07:27:12.341038+0000","last_undegraded":"2024-03-06T07:27:12.341038+0000","last_fullsized":"2024-03-06T07:27:12.341038+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:01:49.637222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.340844+0000","last_change":"2024-03-06T07:27:11.168729+0000","last_active":"2024-03-06T07:27:12.340844+0000","last_peered":"2024-03-06T07:27:12.340844+0000","last_clean":"2024-03-06T07:27:12.340844+0000","last_became_active":"2024-03-06T07:27:11.167902+0000","last_became_peered":"2024-03-06T07:27:11.167902+0000","last_unstale":"2024-03-06T07:27:12.340844+0000","last_undegraded":"2024-03-06T07:27:12.340844+0000","last_fullsized":"2024-03-06T07:27:12.340844+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:28:21.043774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.154528+0000","last_change":"2024-03-06T07:27:11.147686+0000","last_active":"2024-03-06T07:27:12.154528+0000","last_peered":"2024-03-06T07:27:12.154528+0000","last_clean":"2024-03-06T07:27:12.154528+0000","last_became_active":"2024-03-06T07:27:11.147406+0000","last_became_peered":"2024-03-06T07:27:11.147406+0000","last_unstale":"2024-03-06T07:27:12.154528+0000","last_undegraded":"2024-03-06T07:27:12.154528+0000","last_fullsized":"2024-03-06T07:27:12.154528+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:18:43.658000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.341126+0000","last_change":"2024-03-06T07:27:11.170266+0000","last_active":"2024-03-06T07:27:12.341126+0000","last_peered":"2024-03-06T07:27:12.341126+0000","last_clean":"2024-03-06T07:27:12.341126+0000","last_became_active":"2024-03-06T07:27:11.169033+0000","last_became_peered":"2024-03-06T07:27:11.169033+0000","last_unstale":"2024-03-06T07:27:12.341126+0000","last_undegraded":"2024-03-06T07:27:12.341126+0000","last_fullsized":"2024-03-06T07:27:12.341126+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:43.937920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.341646+0000","last_change":"2024-03-06T07:27:11.165486+0000","last_active":"2024-03-06T07:27:12.341646+0000","last_peered":"2024-03-06T07:27:12.341646+0000","last_clean":"2024-03-06T07:27:12.341646+0000","last_became_active":"2024-03-06T07:27:11.165141+0000","last_became_peered":"2024-03-06T07:27:11.165141+0000","last_unstale":"2024-03-06T07:27:12.341646+0000","last_undegraded":"2024-03-06T07:27:12.341646+0000","last_fullsized":"2024-03-06T07:27:12.341646+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:20.060365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.340771+0000","last_change":"2024-03-06T07:27:11.157246+0000","last_active":"2024-03-06T07:27:12.340771+0000","last_peered":"2024-03-06T07:27:12.340771+0000","last_clean":"2024-03-06T07:27:12.340771+0000","last_became_active":"2024-03-06T07:27:11.156731+0000","last_became_peered":"2024-03-06T07:27:11.156731+0000","last_unstale":"2024-03-06T07:27:12.340771+0000","last_undegraded":"2024-03-06T07:27:12.340771+0000","last_fullsized":"2024-03-06T07:27:12.340771+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:35:54.384802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.341618+0000","last_change":"2024-03-06T07:27:11.155900+0000","last_active":"2024-03-06T07:27:12.341618+0000","last_peered":"2024-03-06T07:27:12.341618+0000","last_clean":"2024-03-06T07:27:12.341618+0000","last_became_active":"2024-03-06T07:27:11.155578+0000","last_became_peered":"2024-03-06T07:27:11.155578+0000","last_unstale":"2024-03-06T07:27:12.341618+0000","last_undegraded":"2024-03-06T07:27:12.341618+0000","last_fullsized":"2024-03-06T07:27:12.341618+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:31:14.936650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.340962+0000","last_change":"2024-03-06T07:27:11.169776+0000","last_active":"2024-03-06T07:27:12.340962+0000","last_peered":"2024-03-06T07:27:12.340962+0000","last_clean":"2024-03-06T07:27:12.340962+0000","last_became_active":"2024-03-06T07:27:11.167424+0000","last_became_peered":"2024-03-06T07:27:11.167424+0000","last_unstale":"2024-03-06T07:27:12.340962+0000","last_undegraded":"2024-03-06T07:27:12.340962+0000","last_fullsized":"2024-03-06T07:27:12.340962+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:25.820698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.154689+0000","last_change":"2024-03-06T07:27:11.146611+0000","last_active":"2024-03-06T07:27:12.154689+0000","last_peered":"2024-03-06T07:27:12.154689+0000","last_clean":"2024-03-06T07:27:12.154689+0000","last_became_active":"2024-03-06T07:27:11.146349+0000","last_became_peered":"2024-03-06T07:27:11.146349+0000","last_unstale":"2024-03-06T07:27:12.154689+0000","last_undegraded":"2024-03-06T07:27:12.154689+0000","last_fullsized":"2024-03-06T07:27:12.154689+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:35:54.384802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.154110+0000","last_change":"2024-03-06T07:27:11.146457+0000","last_active":"2024-03-06T07:27:12.154110+0000","last_peered":"2024-03-06T07:27:12.154110+0000","last_clean":"2024-03-06T07:27:12.154110+0000","last_became_active":"2024-03-06T07:27:11.146183+0000","last_became_peered":"2024-03-06T07:27:11.146183+0000","last_unstale":"2024-03-06T07:27:12.154110+0000","last_undegraded":"2024-03-06T07:27:12.154110+0000","last_fullsized":"2024-03-06T07:27:12.154110+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:39.302267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.341605+0000","last_change":"2024-03-06T07:27:11.165243+0000","last_active":"2024-03-06T07:27:12.341605+0000","last_peered":"2024-03-06T07:27:12.341605+0000","last_clean":"2024-03-06T07:27:12.341605+0000","last_became_active":"2024-03-06T07:27:11.164857+0000","last_became_peered":"2024-03-06T07:27:11.164857+0000","last_unstale":"2024-03-06T07:27:12.341605+0000","last_undegraded":"2024-03-06T07:27:12.341605+0000","last_fullsized":"2024-03-06T07:27:12.341605+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:13:08.249896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.154028+0000","last_change":"2024-03-06T07:27:11.156282+0000","last_active":"2024-03-06T07:27:12.154028+0000","last_peered":"2024-03-06T07:27:12.154028+0000","last_clean":"2024-03-06T07:27:12.154028+0000","last_became_active":"2024-03-06T07:27:11.156013+0000","last_became_peered":"2024-03-06T07:27:11.156013+0000","last_unstale":"2024-03-06T07:27:12.154028+0000","last_undegraded":"2024-03-06T07:27:12.154028+0000","last_fullsized":"2024-03-06T07:27:12.154028+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:26.959824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.154540+0000","last_change":"2024-03-06T07:27:11.154456+0000","last_active":"2024-03-06T07:27:12.154540+0000","last_peered":"2024-03-06T07:27:12.154540+0000","last_clean":"2024-03-06T07:27:12.154540+0000","last_became_active":"2024-03-06T07:27:11.154182+0000","last_became_peered":"2024-03-06T07:27:11.154182+0000","last_unstale":"2024-03-06T07:27:12.154540+0000","last_undegraded":"2024-03-06T07:27:12.154540+0000","last_fullsized":"2024-03-06T07:27:12.154540+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:56.472401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.341471+0000","last_change":"2024-03-06T07:27:11.151029+0000","last_active":"2024-03-06T07:27:12.341471+0000","last_peered":"2024-03-06T07:27:12.341471+0000","last_clean":"2024-03-06T07:27:12.341471+0000","last_became_active":"2024-03-06T07:27:11.150722+0000","last_became_peered":"2024-03-06T07:27:11.150722+0000","last_unstale":"2024-03-06T07:27:12.341471+0000","last_undegraded":"2024-03-06T07:27:12.341471+0000","last_fullsized":"2024-03-06T07:27:12.341471+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:31:56.895803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.153849+0000","last_change":"2024-03-06T07:27:11.156376+0000","last_active":"2024-03-06T07:27:12.153849+0000","last_peered":"2024-03-06T07:27:12.153849+0000","last_clean":"2024-03-06T07:27:12.153849+0000","last_became_active":"2024-03-06T07:27:11.156084+0000","last_became_peered":"2024-03-06T07:27:11.156084+0000","last_unstale":"2024-03-06T07:27:12.153849+0000","last_undegraded":"2024-03-06T07:27:12.153849+0000","last_fullsized":"2024-03-06T07:27:12.153849+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:39.302267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.340468+0000","last_change":"2024-03-06T07:27:11.163332+0000","last_active":"2024-03-06T07:27:12.340468+0000","last_peered":"2024-03-06T07:27:12.340468+0000","last_clean":"2024-03-06T07:27:12.340468+0000","last_became_active":"2024-03-06T07:27:11.163098+0000","last_became_peered":"2024-03-06T07:27:11.163098+0000","last_unstale":"2024-03-06T07:27:12.340468+0000","last_undegraded":"2024-03-06T07:27:12.340468+0000","last_fullsized":"2024-03-06T07:27:12.340468+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:53.229467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.154694+0000","last_change":"2024-03-06T07:27:11.146926+0000","last_active":"2024-03-06T07:27:12.154694+0000","last_peered":"2024-03-06T07:27:12.154694+0000","last_clean":"2024-03-06T07:27:12.154694+0000","last_became_active":"2024-03-06T07:27:11.146633+0000","last_became_peered":"2024-03-06T07:27:11.146633+0000","last_unstale":"2024-03-06T07:27:12.154694+0000","last_undegraded":"2024-03-06T07:27:12.154694+0000","last_fullsized":"2024-03-06T07:27:12.154694+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:31:56.895803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.340989+0000","last_change":"2024-03-06T07:27:11.168266+0000","last_active":"2024-03-06T07:27:12.340989+0000","last_peered":"2024-03-06T07:27:12.340989+0000","last_clean":"2024-03-06T07:27:12.340989+0000","last_became_active":"2024-03-06T07:27:11.167479+0000","last_became_peered":"2024-03-06T07:27:11.167479+0000","last_unstale":"2024-03-06T07:27:12.340989+0000","last_undegraded":"2024-03-06T07:27:12.340989+0000","last_fullsized":"2024-03-06T07:27:12.340989+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:48:59.627858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.342459+0000","last_change":"2024-03-06T07:27:11.164909+0000","last_active":"2024-03-06T07:27:12.342459+0000","last_peered":"2024-03-06T07:27:12.342459+0000","last_clean":"2024-03-06T07:27:12.342459+0000","last_became_active":"2024-03-06T07:27:11.164631+0000","last_became_peered":"2024-03-06T07:27:11.164631+0000","last_unstale":"2024-03-06T07:27:12.342459+0000","last_undegraded":"2024-03-06T07:27:12.342459+0000","last_fullsized":"2024-03-06T07:27:12.342459+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:55:41.242688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.342009+0000","last_change":"2024-03-06T07:27:11.170885+0000","last_active":"2024-03-06T07:27:12.342009+0000","last_peered":"2024-03-06T07:27:12.342009+0000","last_clean":"2024-03-06T07:27:12.342009+0000","last_became_active":"2024-03-06T07:27:11.170606+0000","last_became_peered":"2024-03-06T07:27:11.170606+0000","last_unstale":"2024-03-06T07:27:12.342009+0000","last_undegraded":"2024-03-06T07:27:12.342009+0000","last_fullsized":"2024-03-06T07:27:12.342009+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:56:33.426454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.154252+0000","last_change":"2024-03-06T07:27:11.154429+0000","last_active":"2024-03-06T07:27:12.154252+0000","last_peered":"2024-03-06T07:27:12.154252+0000","last_clean":"2024-03-06T07:27:12.154252+0000","last_became_active":"2024-03-06T07:27:11.153710+0000","last_became_peered":"2024-03-06T07:27:11.153710+0000","last_unstale":"2024-03-06T07:27:12.154252+0000","last_undegraded":"2024-03-06T07:27:12.154252+0000","last_fullsized":"2024-03-06T07:27:12.154252+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:53.229467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.341031+0000","last_change":"2024-03-06T07:27:11.153576+0000","last_active":"2024-03-06T07:27:12.341031+0000","last_peered":"2024-03-06T07:27:12.341031+0000","last_clean":"2024-03-06T07:27:12.341031+0000","last_became_active":"2024-03-06T07:27:11.153274+0000","last_became_peered":"2024-03-06T07:27:11.153274+0000","last_unstale":"2024-03-06T07:27:12.341031+0000","last_undegraded":"2024-03-06T07:27:12.341031+0000","last_fullsized":"2024-03-06T07:27:12.341031+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:42.088271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.340832+0000","last_change":"2024-03-06T07:27:11.169202+0000","last_active":"2024-03-06T07:27:12.340832+0000","last_peered":"2024-03-06T07:27:12.340832+0000","last_clean":"2024-03-06T07:27:12.340832+0000","last_became_active":"2024-03-06T07:27:11.168904+0000","last_became_peered":"2024-03-06T07:27:11.168904+0000","last_unstale":"2024-03-06T07:27:12.340832+0000","last_undegraded":"2024-03-06T07:27:12.340832+0000","last_fullsized":"2024-03-06T07:27:12.340832+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:04:08.180273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.340468+0000","last_change":"2024-03-06T07:27:11.155597+0000","last_active":"2024-03-06T07:27:12.340468+0000","last_peered":"2024-03-06T07:27:12.340468+0000","last_clean":"2024-03-06T07:27:12.340468+0000","last_became_active":"2024-03-06T07:27:11.155348+0000","last_became_peered":"2024-03-06T07:27:11.155348+0000","last_unstale":"2024-03-06T07:27:12.340468+0000","last_undegraded":"2024-03-06T07:27:12.340468+0000","last_fullsized":"2024-03-06T07:27:12.340468+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:47.267020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.342166+0000","last_change":"2024-03-06T07:27:11.156750+0000","last_active":"2024-03-06T07:27:12.342166+0000","last_peered":"2024-03-06T07:27:12.342166+0000","last_clean":"2024-03-06T07:27:12.342166+0000","last_became_active":"2024-03-06T07:27:11.156486+0000","last_became_peered":"2024-03-06T07:27:11.156486+0000","last_unstale":"2024-03-06T07:27:12.342166+0000","last_undegraded":"2024-03-06T07:27:12.342166+0000","last_fullsized":"2024-03-06T07:27:12.342166+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:20:13.455134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"109'4","reported_seq":18,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.341330+0000","last_change":"2024-03-06T07:27:10.143561+0000","last_active":"2024-03-06T07:27:12.341330+0000","last_peered":"2024-03-06T07:27:12.341330+0000","last_clean":"2024-03-06T07:27:12.341330+0000","last_became_active":"2024-03-06T07:27:10.143244+0000","last_became_peered":"2024-03-06T07:27:10.143244+0000","last_unstale":"2024-03-06T07:27:12.341330+0000","last_undegraded":"2024-03-06T07:27:12.341330+0000","last_fullsized":"2024-03-06T07:27:12.341330+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:19.848712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.154177+0000","last_change":"2024-03-06T07:27:11.156729+0000","last_active":"2024-03-06T07:27:12.154177+0000","last_peered":"2024-03-06T07:27:12.154177+0000","last_clean":"2024-03-06T07:27:12.154177+0000","last_became_active":"2024-03-06T07:27:11.156346+0000","last_became_peered":"2024-03-06T07:27:11.156346+0000","last_unstale":"2024-03-06T07:27:12.154177+0000","last_undegraded":"2024-03-06T07:27:12.154177+0000","last_fullsized":"2024-03-06T07:27:12.154177+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:26.959824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"109'1","reported_seq":15,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.341665+0000","last_change":"2024-03-06T07:27:10.145366+0000","last_active":"2024-03-06T07:27:12.341665+0000","last_peered":"2024-03-06T07:27:12.341665+0000","last_clean":"2024-03-06T07:27:12.341665+0000","last_became_active":"2024-03-06T07:27:10.145028+0000","last_became_peered":"2024-03-06T07:27:10.145028+0000","last_unstale":"2024-03-06T07:27:12.341665+0000","last_undegraded":"2024-03-06T07:27:12.341665+0000","last_fullsized":"2024-03-06T07:27:12.341665+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:49:40.057171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.342352+0000","last_change":"2024-03-06T07:27:11.156224+0000","last_active":"2024-03-06T07:27:12.342352+0000","last_peered":"2024-03-06T07:27:12.342352+0000","last_clean":"2024-03-06T07:27:12.342352+0000","last_became_active":"2024-03-06T07:27:11.155919+0000","last_became_peered":"2024-03-06T07:27:11.155919+0000","last_unstale":"2024-03-06T07:27:12.342352+0000","last_undegraded":"2024-03-06T07:27:12.342352+0000","last_fullsized":"2024-03-06T07:27:12.342352+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:17.109742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"109'2","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.153036+0000","last_change":"2024-03-06T07:27:10.146400+0000","last_active":"2024-03-06T07:27:12.153036+0000","last_peered":"2024-03-06T07:27:12.153036+0000","last_clean":"2024-03-06T07:27:12.153036+0000","last_became_active":"2024-03-06T07:27:10.146106+0000","last_became_peered":"2024-03-06T07:27:10.146106+0000","last_unstale":"2024-03-06T07:27:12.153036+0000","last_undegraded":"2024-03-06T07:27:12.153036+0000","last_fullsized":"2024-03-06T07:27:12.153036+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:46.262408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.341787+0000","last_change":"2024-03-06T07:27:11.165718+0000","last_active":"2024-03-06T07:27:12.341787+0000","last_peered":"2024-03-06T07:27:12.341787+0000","last_clean":"2024-03-06T07:27:12.341787+0000","last_became_active":"2024-03-06T07:27:11.165447+0000","last_became_peered":"2024-03-06T07:27:11.165447+0000","last_unstale":"2024-03-06T07:27:12.341787+0000","last_undegraded":"2024-03-06T07:27:12.341787+0000","last_fullsized":"2024-03-06T07:27:12.341787+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:11:34.434994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"109'2","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.152869+0000","last_change":"2024-03-06T07:27:10.137036+0000","last_active":"2024-03-06T07:27:12.152869+0000","last_peered":"2024-03-06T07:27:12.152869+0000","last_clean":"2024-03-06T07:27:12.152869+0000","last_became_active":"2024-03-06T07:27:10.136771+0000","last_became_peered":"2024-03-06T07:27:10.136771+0000","last_unstale":"2024-03-06T07:27:12.152869+0000","last_undegraded":"2024-03-06T07:27:12.152869+0000","last_fullsized":"2024-03-06T07:27:12.152869+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:55.019970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.340612+0000","last_change":"2024-03-06T07:27:11.154563+0000","last_active":"2024-03-06T07:27:12.340612+0000","last_peered":"2024-03-06T07:27:12.340612+0000","last_clean":"2024-03-06T07:27:12.340612+0000","last_became_active":"2024-03-06T07:27:11.154265+0000","last_became_peered":"2024-03-06T07:27:11.154265+0000","last_unstale":"2024-03-06T07:27:12.340612+0000","last_undegraded":"2024-03-06T07:27:12.340612+0000","last_fullsized":"2024-03-06T07:27:12.340612+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:12:51.868028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"109'1","reported_seq":15,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.341005+0000","last_change":"2024-03-06T07:27:10.152619+0000","last_active":"2024-03-06T07:27:12.341005+0000","last_peered":"2024-03-06T07:27:12.341005+0000","last_clean":"2024-03-06T07:27:12.341005+0000","last_became_active":"2024-03-06T07:27:10.152377+0000","last_became_peered":"2024-03-06T07:27:10.152377+0000","last_unstale":"2024-03-06T07:27:12.341005+0000","last_undegraded":"2024-03-06T07:27:12.341005+0000","last_fullsized":"2024-03-06T07:27:12.341005+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:18.836864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.154076+0000","last_change":"2024-03-06T07:27:11.158457+0000","last_active":"2024-03-06T07:27:12.154076+0000","last_peered":"2024-03-06T07:27:12.154076+0000","last_clean":"2024-03-06T07:27:12.154076+0000","last_became_active":"2024-03-06T07:27:11.158110+0000","last_became_peered":"2024-03-06T07:27:11.158110+0000","last_unstale":"2024-03-06T07:27:12.154076+0000","last_undegraded":"2024-03-06T07:27:12.154076+0000","last_fullsized":"2024-03-06T07:27:12.154076+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:35:42.630859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"109'1","reported_seq":15,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.341700+0000","last_change":"2024-03-06T07:27:10.148748+0000","last_active":"2024-03-06T07:27:12.341700+0000","last_peered":"2024-03-06T07:27:12.341700+0000","last_clean":"2024-03-06T07:27:12.341700+0000","last_became_active":"2024-03-06T07:27:10.148438+0000","last_became_peered":"2024-03-06T07:27:10.148438+0000","last_unstale":"2024-03-06T07:27:12.341700+0000","last_undegraded":"2024-03-06T07:27:12.341700+0000","last_fullsized":"2024-03-06T07:27:12.341700+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:44:45.337949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.341111+0000","last_change":"2024-03-06T07:27:11.169550+0000","last_active":"2024-03-06T07:27:12.341111+0000","last_peered":"2024-03-06T07:27:12.341111+0000","last_clean":"2024-03-06T07:27:12.341111+0000","last_became_active":"2024-03-06T07:27:11.165336+0000","last_became_peered":"2024-03-06T07:27:11.165336+0000","last_unstale":"2024-03-06T07:27:12.341111+0000","last_undegraded":"2024-03-06T07:27:12.341111+0000","last_fullsized":"2024-03-06T07:27:12.341111+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:13.281998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.153709+0000","last_change":"2024-03-06T07:27:10.147680+0000","last_active":"2024-03-06T07:27:12.153709+0000","last_peered":"2024-03-06T07:27:12.153709+0000","last_clean":"2024-03-06T07:27:12.153709+0000","last_became_active":"2024-03-06T07:27:10.147438+0000","last_became_peered":"2024-03-06T07:27:10.147438+0000","last_unstale":"2024-03-06T07:27:12.153709+0000","last_undegraded":"2024-03-06T07:27:12.153709+0000","last_fullsized":"2024-03-06T07:27:12.153709+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:14.391693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.340621+0000","last_change":"2024-03-06T07:27:11.152006+0000","last_active":"2024-03-06T07:27:12.340621+0000","last_peered":"2024-03-06T07:27:12.340621+0000","last_clean":"2024-03-06T07:27:12.340621+0000","last_became_active":"2024-03-06T07:27:11.151397+0000","last_became_peered":"2024-03-06T07:27:11.151397+0000","last_unstale":"2024-03-06T07:27:12.340621+0000","last_undegraded":"2024-03-06T07:27:12.340621+0000","last_fullsized":"2024-03-06T07:27:12.340621+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:57:45.156722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.341317+0000","last_change":"2024-03-06T07:27:10.149089+0000","last_active":"2024-03-06T07:27:12.341317+0000","last_peered":"2024-03-06T07:27:12.341317+0000","last_clean":"2024-03-06T07:27:12.341317+0000","last_became_active":"2024-03-06T07:27:10.148126+0000","last_became_peered":"2024-03-06T07:27:10.148126+0000","last_unstale":"2024-03-06T07:27:12.341317+0000","last_undegraded":"2024-03-06T07:27:12.341317+0000","last_fullsized":"2024-03-06T07:27:12.341317+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:55.467789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.341430+0000","last_change":"2024-03-06T07:27:11.156747+0000","last_active":"2024-03-06T07:27:12.341430+0000","last_peered":"2024-03-06T07:27:12.341430+0000","last_clean":"2024-03-06T07:27:12.341430+0000","last_became_active":"2024-03-06T07:27:11.156431+0000","last_became_peered":"2024-03-06T07:27:11.156431+0000","last_unstale":"2024-03-06T07:27:12.341430+0000","last_undegraded":"2024-03-06T07:27:12.341430+0000","last_fullsized":"2024-03-06T07:27:12.341430+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:13.821532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.153069+0000","last_change":"2024-03-06T07:27:10.146728+0000","last_active":"2024-03-06T07:27:12.153069+0000","last_peered":"2024-03-06T07:27:12.153069+0000","last_clean":"2024-03-06T07:27:12.153069+0000","last_became_active":"2024-03-06T07:27:10.146456+0000","last_became_peered":"2024-03-06T07:27:10.146456+0000","last_unstale":"2024-03-06T07:27:12.153069+0000","last_undegraded":"2024-03-06T07:27:12.153069+0000","last_fullsized":"2024-03-06T07:27:12.153069+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:20:12.450522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.154319+0000","last_change":"2024-03-06T07:27:11.154350+0000","last_active":"2024-03-06T07:27:12.154319+0000","last_peered":"2024-03-06T07:27:12.154319+0000","last_clean":"2024-03-06T07:27:12.154319+0000","last_became_active":"2024-03-06T07:27:11.154017+0000","last_became_peered":"2024-03-06T07:27:11.154017+0000","last_unstale":"2024-03-06T07:27:12.154319+0000","last_undegraded":"2024-03-06T07:27:12.154319+0000","last_fullsized":"2024-03-06T07:27:12.154319+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:23:32.125301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.340620+0000","last_change":"2024-03-06T07:27:10.148013+0000","last_active":"2024-03-06T07:27:12.340620+0000","last_peered":"2024-03-06T07:27:12.340620+0000","last_clean":"2024-03-06T07:27:12.340620+0000","last_became_active":"2024-03-06T07:27:10.147723+0000","last_became_peered":"2024-03-06T07:27:10.147723+0000","last_unstale":"2024-03-06T07:27:12.340620+0000","last_undegraded":"2024-03-06T07:27:12.340620+0000","last_fullsized":"2024-03-06T07:27:12.340620+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:55:08.669740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.342088+0000","last_change":"2024-03-06T07:27:11.151862+0000","last_active":"2024-03-06T07:27:12.342088+0000","last_peered":"2024-03-06T07:27:12.342088+0000","last_clean":"2024-03-06T07:27:12.342088+0000","last_became_active":"2024-03-06T07:27:11.151574+0000","last_became_peered":"2024-03-06T07:27:11.151574+0000","last_unstale":"2024-03-06T07:27:12.342088+0000","last_undegraded":"2024-03-06T07:27:12.342088+0000","last_fullsized":"2024-03-06T07:27:12.342088+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:04.749175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"109'3","reported_seq":17,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.340861+0000","last_change":"2024-03-06T07:27:10.145307+0000","last_active":"2024-03-06T07:27:12.340861+0000","last_peered":"2024-03-06T07:27:12.340861+0000","last_clean":"2024-03-06T07:27:12.340861+0000","last_became_active":"2024-03-06T07:27:10.145027+0000","last_became_peered":"2024-03-06T07:27:10.145027+0000","last_unstale":"2024-03-06T07:27:12.340861+0000","last_undegraded":"2024-03-06T07:27:12.340861+0000","last_fullsized":"2024-03-06T07:27:12.340861+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:56:32.421843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.341185+0000","last_change":"2024-03-06T07:27:11.157009+0000","last_active":"2024-03-06T07:27:12.341185+0000","last_peered":"2024-03-06T07:27:12.341185+0000","last_clean":"2024-03-06T07:27:12.341185+0000","last_became_active":"2024-03-06T07:27:11.156735+0000","last_became_peered":"2024-03-06T07:27:11.156735+0000","last_unstale":"2024-03-06T07:27:12.341185+0000","last_undegraded":"2024-03-06T07:27:12.341185+0000","last_fullsized":"2024-03-06T07:27:12.341185+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:10.422569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"109'6","reported_seq":20,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.341310+0000","last_change":"2024-03-06T07:27:10.147579+0000","last_active":"2024-03-06T07:27:12.341310+0000","last_peered":"2024-03-06T07:27:12.341310+0000","last_clean":"2024-03-06T07:27:12.341310+0000","last_became_active":"2024-03-06T07:27:10.147292+0000","last_became_peered":"2024-03-06T07:27:10.147292+0000","last_unstale":"2024-03-06T07:27:12.341310+0000","last_undegraded":"2024-03-06T07:27:12.341310+0000","last_fullsized":"2024-03-06T07:27:12.341310+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:35:53.380190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.341610+0000","last_change":"2024-03-06T07:27:11.159075+0000","last_active":"2024-03-06T07:27:12.341610+0000","last_peered":"2024-03-06T07:27:12.341610+0000","last_clean":"2024-03-06T07:27:12.341610+0000","last_became_active":"2024-03-06T07:27:11.158802+0000","last_became_peered":"2024-03-06T07:27:11.158802+0000","last_unstale":"2024-03-06T07:27:12.341610+0000","last_undegraded":"2024-03-06T07:27:12.341610+0000","last_fullsized":"2024-03-06T07:27:12.341610+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:35:54.384802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"109'2","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.342080+0000","last_change":"2024-03-06T07:27:10.145448+0000","last_active":"2024-03-06T07:27:12.342080+0000","last_peered":"2024-03-06T07:27:12.342080+0000","last_clean":"2024-03-06T07:27:12.342080+0000","last_became_active":"2024-03-06T07:27:10.145063+0000","last_became_peered":"2024-03-06T07:27:10.145063+0000","last_unstale":"2024-03-06T07:27:12.342080+0000","last_undegraded":"2024-03-06T07:27:12.342080+0000","last_fullsized":"2024-03-06T07:27:12.342080+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:44:20.435658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.340793+0000","last_change":"2024-03-06T07:27:11.161978+0000","last_active":"2024-03-06T07:27:12.340793+0000","last_peered":"2024-03-06T07:27:12.340793+0000","last_clean":"2024-03-06T07:27:12.340793+0000","last_became_active":"2024-03-06T07:27:11.161685+0000","last_became_peered":"2024-03-06T07:27:11.161685+0000","last_unstale":"2024-03-06T07:27:12.340793+0000","last_undegraded":"2024-03-06T07:27:12.340793+0000","last_fullsized":"2024-03-06T07:27:12.340793+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:56.472401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.154508+0000","last_change":"2024-03-06T07:27:10.146708+0000","last_active":"2024-03-06T07:27:12.154508+0000","last_peered":"2024-03-06T07:27:12.154508+0000","last_clean":"2024-03-06T07:27:12.154508+0000","last_became_active":"2024-03-06T07:27:10.146387+0000","last_became_peered":"2024-03-06T07:27:10.146387+0000","last_unstale":"2024-03-06T07:27:12.154508+0000","last_undegraded":"2024-03-06T07:27:12.154508+0000","last_fullsized":"2024-03-06T07:27:12.154508+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:44:45.337949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.340646+0000","last_change":"2024-03-06T07:27:11.153166+0000","last_active":"2024-03-06T07:27:12.340646+0000","last_peered":"2024-03-06T07:27:12.340646+0000","last_clean":"2024-03-06T07:27:12.340646+0000","last_became_active":"2024-03-06T07:27:11.152840+0000","last_became_peered":"2024-03-06T07:27:11.152840+0000","last_unstale":"2024-03-06T07:27:12.340646+0000","last_undegraded":"2024-03-06T07:27:12.340646+0000","last_fullsized":"2024-03-06T07:27:12.340646+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:18:43.658000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"109'8","reported_seq":22,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.152739+0000","last_change":"2024-03-06T07:27:10.136377+0000","last_active":"2024-03-06T07:27:12.152739+0000","last_peered":"2024-03-06T07:27:12.152739+0000","last_clean":"2024-03-06T07:27:12.152739+0000","last_became_active":"2024-03-06T07:27:10.136062+0000","last_became_peered":"2024-03-06T07:27:10.136062+0000","last_unstale":"2024-03-06T07:27:12.152739+0000","last_undegraded":"2024-03-06T07:27:12.152739+0000","last_fullsized":"2024-03-06T07:27:12.152739+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:04:46.767041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.153145+0000","last_change":"2024-03-06T07:27:11.166282+0000","last_active":"2024-03-06T07:27:12.153145+0000","last_peered":"2024-03-06T07:27:12.153145+0000","last_clean":"2024-03-06T07:27:12.153145+0000","last_became_active":"2024-03-06T07:27:11.166042+0000","last_became_peered":"2024-03-06T07:27:11.166042+0000","last_unstale":"2024-03-06T07:27:12.153145+0000","last_undegraded":"2024-03-06T07:27:12.153145+0000","last_fullsized":"2024-03-06T07:27:12.153145+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:13.369455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"109'1","reported_seq":15,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.341158+0000","last_change":"2024-03-06T07:27:10.147575+0000","last_active":"2024-03-06T07:27:12.341158+0000","last_peered":"2024-03-06T07:27:12.341158+0000","last_clean":"2024-03-06T07:27:12.341158+0000","last_became_active":"2024-03-06T07:27:10.147304+0000","last_became_peered":"2024-03-06T07:27:10.147304+0000","last_unstale":"2024-03-06T07:27:12.341158+0000","last_undegraded":"2024-03-06T07:27:12.341158+0000","last_fullsized":"2024-03-06T07:27:12.341158+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:31:55.891191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.342373+0000","last_change":"2024-03-06T07:27:11.161796+0000","last_active":"2024-03-06T07:27:12.342373+0000","last_peered":"2024-03-06T07:27:12.342373+0000","last_clean":"2024-03-06T07:27:12.342373+0000","last_became_active":"2024-03-06T07:27:11.161485+0000","last_became_peered":"2024-03-06T07:27:11.161485+0000","last_unstale":"2024-03-06T07:27:12.342373+0000","last_undegraded":"2024-03-06T07:27:12.342373+0000","last_fullsized":"2024-03-06T07:27:12.342373+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:31.230687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"109'2","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.153143+0000","last_change":"2024-03-06T07:27:10.137380+0000","last_active":"2024-03-06T07:27:12.153143+0000","last_peered":"2024-03-06T07:27:12.153143+0000","last_clean":"2024-03-06T07:27:12.153143+0000","last_became_active":"2024-03-06T07:27:10.137056+0000","last_became_peered":"2024-03-06T07:27:10.137056+0000","last_unstale":"2024-03-06T07:27:12.153143+0000","last_undegraded":"2024-03-06T07:27:12.153143+0000","last_fullsized":"2024-03-06T07:27:12.153143+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:16.105130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.341412+0000","last_change":"2024-03-06T07:27:11.171575+0000","last_active":"2024-03-06T07:27:12.341412+0000","last_peered":"2024-03-06T07:27:12.341412+0000","last_clean":"2024-03-06T07:27:12.341412+0000","last_became_active":"2024-03-06T07:27:11.171354+0000","last_became_peered":"2024-03-06T07:27:11.171354+0000","last_unstale":"2024-03-06T07:27:12.341412+0000","last_undegraded":"2024-03-06T07:27:12.341412+0000","last_fullsized":"2024-03-06T07:27:12.341412+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:28:14.790581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"109'1","reported_seq":15,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.154099+0000","last_change":"2024-03-06T07:27:10.145731+0000","last_active":"2024-03-06T07:27:12.154099+0000","last_peered":"2024-03-06T07:27:12.154099+0000","last_clean":"2024-03-06T07:27:12.154099+0000","last_became_active":"2024-03-06T07:27:10.145424+0000","last_became_peered":"2024-03-06T07:27:10.145424+0000","last_unstale":"2024-03-06T07:27:12.154099+0000","last_undegraded":"2024-03-06T07:27:12.154099+0000","last_fullsized":"2024-03-06T07:27:12.154099+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:55:08.669740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.341801+0000","last_change":"2024-03-06T07:27:11.158948+0000","last_active":"2024-03-06T07:27:12.341801+0000","last_peered":"2024-03-06T07:27:12.341801+0000","last_clean":"2024-03-06T07:27:12.341801+0000","last_became_active":"2024-03-06T07:27:11.158652+0000","last_became_peered":"2024-03-06T07:27:11.158652+0000","last_unstale":"2024-03-06T07:27:12.341801+0000","last_undegraded":"2024-03-06T07:27:12.341801+0000","last_fullsized":"2024-03-06T07:27:12.341801+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:54:08.545924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.341068+0000","last_change":"2024-03-06T07:27:10.151277+0000","last_active":"2024-03-06T07:27:12.341068+0000","last_peered":"2024-03-06T07:27:12.341068+0000","last_clean":"2024-03-06T07:27:12.341068+0000","last_became_active":"2024-03-06T07:27:10.150933+0000","last_became_peered":"2024-03-06T07:27:10.150933+0000","last_unstale":"2024-03-06T07:27:12.341068+0000","last_undegraded":"2024-03-06T07:27:12.341068+0000","last_fullsized":"2024-03-06T07:27:12.341068+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:52.270666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.340939+0000","last_change":"2024-03-06T07:27:11.157018+0000","last_active":"2024-03-06T07:27:12.340939+0000","last_peered":"2024-03-06T07:27:12.340939+0000","last_clean":"2024-03-06T07:27:12.340939+0000","last_became_active":"2024-03-06T07:27:11.156319+0000","last_became_peered":"2024-03-06T07:27:11.156319+0000","last_unstale":"2024-03-06T07:27:12.340939+0000","last_undegraded":"2024-03-06T07:27:12.340939+0000","last_fullsized":"2024-03-06T07:27:12.340939+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:04.749175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"109'2","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.342177+0000","last_change":"2024-03-06T07:27:10.150893+0000","last_active":"2024-03-06T07:27:12.342177+0000","last_peered":"2024-03-06T07:27:12.342177+0000","last_clean":"2024-03-06T07:27:12.342177+0000","last_became_active":"2024-03-06T07:27:10.150648+0000","last_became_peered":"2024-03-06T07:27:10.150648+0000","last_unstale":"2024-03-06T07:27:12.342177+0000","last_undegraded":"2024-03-06T07:27:12.342177+0000","last_fullsized":"2024-03-06T07:27:12.342177+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:03:45.041901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.340494+0000","last_change":"2024-03-06T07:27:11.164955+0000","last_active":"2024-03-06T07:27:12.340494+0000","last_peered":"2024-03-06T07:27:12.340494+0000","last_clean":"2024-03-06T07:27:12.340494+0000","last_became_active":"2024-03-06T07:27:11.164658+0000","last_became_peered":"2024-03-06T07:27:11.164658+0000","last_unstale":"2024-03-06T07:27:12.340494+0000","last_undegraded":"2024-03-06T07:27:12.340494+0000","last_fullsized":"2024-03-06T07:27:12.340494+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:11:50.991642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.152808+0000","last_change":"2024-03-06T07:27:10.148882+0000","last_active":"2024-03-06T07:27:12.152808+0000","last_peered":"2024-03-06T07:27:12.152808+0000","last_clean":"2024-03-06T07:27:12.152808+0000","last_became_active":"2024-03-06T07:27:10.148626+0000","last_became_peered":"2024-03-06T07:27:10.148626+0000","last_unstale":"2024-03-06T07:27:12.152808+0000","last_undegraded":"2024-03-06T07:27:12.152808+0000","last_fullsized":"2024-03-06T07:27:12.152808+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:31:13.932038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.340991+0000","last_change":"2024-03-06T07:27:11.166064+0000","last_active":"2024-03-06T07:27:12.340991+0000","last_peered":"2024-03-06T07:27:12.340991+0000","last_clean":"2024-03-06T07:27:12.340991+0000","last_became_active":"2024-03-06T07:27:11.165681+0000","last_became_peered":"2024-03-06T07:27:11.165681+0000","last_unstale":"2024-03-06T07:27:12.340991+0000","last_undegraded":"2024-03-06T07:27:12.340991+0000","last_fullsized":"2024-03-06T07:27:12.340991+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:41:22.717235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.341689+0000","last_change":"2024-03-06T07:27:10.144592+0000","last_active":"2024-03-06T07:27:12.341689+0000","last_peered":"2024-03-06T07:27:12.341689+0000","last_clean":"2024-03-06T07:27:12.341689+0000","last_became_active":"2024-03-06T07:27:10.144328+0000","last_became_peered":"2024-03-06T07:27:10.144328+0000","last_unstale":"2024-03-06T07:27:12.341689+0000","last_undegraded":"2024-03-06T07:27:12.341689+0000","last_fullsized":"2024-03-06T07:27:12.341689+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:07.547136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.342212+0000","last_change":"2024-03-06T07:27:11.170678+0000","last_active":"2024-03-06T07:27:12.342212+0000","last_peered":"2024-03-06T07:27:12.342212+0000","last_clean":"2024-03-06T07:27:12.342212+0000","last_became_active":"2024-03-06T07:27:11.170320+0000","last_became_peered":"2024-03-06T07:27:11.170320+0000","last_unstale":"2024-03-06T07:27:12.342212+0000","last_undegraded":"2024-03-06T07:27:12.342212+0000","last_fullsized":"2024-03-06T07:27:12.342212+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:41:22.717235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.341271+0000","last_change":"2024-03-06T07:27:10.151020+0000","last_active":"2024-03-06T07:27:12.341271+0000","last_peered":"2024-03-06T07:27:12.341271+0000","last_clean":"2024-03-06T07:27:12.341271+0000","last_became_active":"2024-03-06T07:27:10.150747+0000","last_became_peered":"2024-03-06T07:27:10.150747+0000","last_unstale":"2024-03-06T07:27:12.341271+0000","last_undegraded":"2024-03-06T07:27:12.341271+0000","last_fullsized":"2024-03-06T07:27:12.341271+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:12.742435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.154258+0000","last_change":"2024-03-06T07:27:11.160942+0000","last_active":"2024-03-06T07:27:12.154258+0000","last_peered":"2024-03-06T07:27:12.154258+0000","last_clean":"2024-03-06T07:27:12.154258+0000","last_became_active":"2024-03-06T07:27:11.160586+0000","last_became_peered":"2024-03-06T07:27:11.160586+0000","last_unstale":"2024-03-06T07:27:12.154258+0000","last_undegraded":"2024-03-06T07:27:12.154258+0000","last_fullsized":"2024-03-06T07:27:12.154258+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:35:42.630859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"109'2","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.341104+0000","last_change":"2024-03-06T07:27:10.150171+0000","last_active":"2024-03-06T07:27:12.341104+0000","last_peered":"2024-03-06T07:27:12.341104+0000","last_clean":"2024-03-06T07:27:12.341104+0000","last_became_active":"2024-03-06T07:27:10.149895+0000","last_became_peered":"2024-03-06T07:27:10.149895+0000","last_unstale":"2024-03-06T07:27:12.341104+0000","last_undegraded":"2024-03-06T07:27:12.341104+0000","last_fullsized":"2024-03-06T07:27:12.341104+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:51.244218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.341180+0000","last_change":"2024-03-06T07:27:11.165035+0000","last_active":"2024-03-06T07:27:12.341180+0000","last_peered":"2024-03-06T07:27:12.341180+0000","last_clean":"2024-03-06T07:27:12.341180+0000","last_became_active":"2024-03-06T07:27:11.164760+0000","last_became_peered":"2024-03-06T07:27:11.164760+0000","last_unstale":"2024-03-06T07:27:12.341180+0000","last_undegraded":"2024-03-06T07:27:12.341180+0000","last_fullsized":"2024-03-06T07:27:12.341180+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:13:57.168618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"110'1","reported_seq":15,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.154495+0000","last_change":"2024-03-06T07:27:10.142952+0000","last_active":"2024-03-06T07:27:12.154495+0000","last_peered":"2024-03-06T07:27:12.154495+0000","last_clean":"2024-03-06T07:27:12.154495+0000","last_became_active":"2024-03-06T07:27:10.142643+0000","last_became_peered":"2024-03-06T07:27:10.142643+0000","last_unstale":"2024-03-06T07:27:12.154495+0000","last_undegraded":"2024-03-06T07:27:12.154495+0000","last_fullsized":"2024-03-06T07:27:12.154495+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:51.244218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.341220+0000","last_change":"2024-03-06T07:27:11.169456+0000","last_active":"2024-03-06T07:27:12.341220+0000","last_peered":"2024-03-06T07:27:12.341220+0000","last_clean":"2024-03-06T07:27:12.341220+0000","last_became_active":"2024-03-06T07:27:11.167648+0000","last_became_peered":"2024-03-06T07:27:11.167648+0000","last_unstale":"2024-03-06T07:27:12.341220+0000","last_undegraded":"2024-03-06T07:27:12.341220+0000","last_fullsized":"2024-03-06T07:27:12.341220+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:53:23.168680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.340776+0000","last_change":"2024-03-06T07:27:10.148289+0000","last_active":"2024-03-06T07:27:12.340776+0000","last_peered":"2024-03-06T07:27:12.340776+0000","last_clean":"2024-03-06T07:27:12.340776+0000","last_became_active":"2024-03-06T07:27:10.148026+0000","last_became_peered":"2024-03-06T07:27:10.148026+0000","last_unstale":"2024-03-06T07:27:12.340776+0000","last_undegraded":"2024-03-06T07:27:12.340776+0000","last_fullsized":"2024-03-06T07:27:12.340776+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:41:21.712623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.154049+0000","last_change":"2024-03-06T07:27:11.162233+0000","last_active":"2024-03-06T07:27:12.154049+0000","last_peered":"2024-03-06T07:27:12.154049+0000","last_clean":"2024-03-06T07:27:12.154049+0000","last_became_active":"2024-03-06T07:27:11.161953+0000","last_became_peered":"2024-03-06T07:27:11.161953+0000","last_unstale":"2024-03-06T07:27:12.154049+0000","last_undegraded":"2024-03-06T07:27:12.154049+0000","last_fullsized":"2024-03-06T07:27:12.154049+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:28:33.943025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.342096+0000","last_change":"2024-03-06T07:27:10.151397+0000","last_active":"2024-03-06T07:27:12.342096+0000","last_peered":"2024-03-06T07:27:12.342096+0000","last_clean":"2024-03-06T07:27:12.342096+0000","last_became_active":"2024-03-06T07:27:10.150827+0000","last_became_peered":"2024-03-06T07:27:10.150827+0000","last_unstale":"2024-03-06T07:27:12.342096+0000","last_undegraded":"2024-03-06T07:27:12.342096+0000","last_fullsized":"2024-03-06T07:27:12.342096+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:12:02.412060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.154150+0000","last_change":"2024-03-06T07:27:11.154817+0000","last_active":"2024-03-06T07:27:12.154150+0000","last_peered":"2024-03-06T07:27:12.154150+0000","last_clean":"2024-03-06T07:27:12.154150+0000","last_became_active":"2024-03-06T07:27:11.154544+0000","last_became_peered":"2024-03-06T07:27:11.154544+0000","last_unstale":"2024-03-06T07:27:12.154150+0000","last_undegraded":"2024-03-06T07:27:12.154150+0000","last_fullsized":"2024-03-06T07:27:12.154150+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:28:33.943025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.341235+0000","last_change":"2024-03-06T07:27:10.141228+0000","last_active":"2024-03-06T07:27:12.341235+0000","last_peered":"2024-03-06T07:27:12.341235+0000","last_clean":"2024-03-06T07:27:12.341235+0000","last_became_active":"2024-03-06T07:27:10.140919+0000","last_became_peered":"2024-03-06T07:27:10.140919+0000","last_unstale":"2024-03-06T07:27:12.341235+0000","last_undegraded":"2024-03-06T07:27:12.341235+0000","last_fullsized":"2024-03-06T07:27:12.341235+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:12.364843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.154363+0000","last_change":"2024-03-06T07:27:11.151693+0000","last_active":"2024-03-06T07:27:12.154363+0000","last_peered":"2024-03-06T07:27:12.154363+0000","last_clean":"2024-03-06T07:27:12.154363+0000","last_became_active":"2024-03-06T07:27:11.151395+0000","last_became_peered":"2024-03-06T07:27:11.151395+0000","last_unstale":"2024-03-06T07:27:12.154363+0000","last_undegraded":"2024-03-06T07:27:12.154363+0000","last_fullsized":"2024-03-06T07:27:12.154363+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:13.747047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.154291+0000","last_change":"2024-03-06T07:27:10.138499+0000","last_active":"2024-03-06T07:27:12.154291+0000","last_peered":"2024-03-06T07:27:12.154291+0000","last_clean":"2024-03-06T07:27:12.154291+0000","last_became_active":"2024-03-06T07:27:10.138222+0000","last_became_peered":"2024-03-06T07:27:10.138222+0000","last_unstale":"2024-03-06T07:27:12.154291+0000","last_undegraded":"2024-03-06T07:27:12.154291+0000","last_fullsized":"2024-03-06T07:27:12.154291+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:41.083659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.340992+0000","last_change":"2024-03-06T07:27:11.157143+0000","last_active":"2024-03-06T07:27:12.340992+0000","last_peered":"2024-03-06T07:27:12.340992+0000","last_clean":"2024-03-06T07:27:12.340992+0000","last_became_active":"2024-03-06T07:27:11.156769+0000","last_became_peered":"2024-03-06T07:27:11.156769+0000","last_unstale":"2024-03-06T07:27:12.340992+0000","last_undegraded":"2024-03-06T07:27:12.340992+0000","last_fullsized":"2024-03-06T07:27:12.340992+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:42:33.219676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.152808+0000","last_change":"2024-03-06T07:27:10.147408+0000","last_active":"2024-03-06T07:27:12.152808+0000","last_peered":"2024-03-06T07:27:12.152808+0000","last_clean":"2024-03-06T07:27:12.152808+0000","last_became_active":"2024-03-06T07:27:10.147159+0000","last_became_peered":"2024-03-06T07:27:10.147159+0000","last_unstale":"2024-03-06T07:27:12.152808+0000","last_undegraded":"2024-03-06T07:27:12.152808+0000","last_fullsized":"2024-03-06T07:27:12.152808+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:58.622106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.340987+0000","last_change":"2024-03-06T07:27:11.169241+0000","last_active":"2024-03-06T07:27:12.340987+0000","last_peered":"2024-03-06T07:27:12.340987+0000","last_clean":"2024-03-06T07:27:12.340987+0000","last_became_active":"2024-03-06T07:27:11.168207+0000","last_became_peered":"2024-03-06T07:27:11.168207+0000","last_unstale":"2024-03-06T07:27:12.340987+0000","last_undegraded":"2024-03-06T07:27:12.340987+0000","last_fullsized":"2024-03-06T07:27:12.340987+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:30.179073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.341384+0000","last_change":"2024-03-06T07:27:10.149308+0000","last_active":"2024-03-06T07:27:12.341384+0000","last_peered":"2024-03-06T07:27:12.341384+0000","last_clean":"2024-03-06T07:27:12.341384+0000","last_became_active":"2024-03-06T07:27:10.148472+0000","last_became_peered":"2024-03-06T07:27:10.148472+0000","last_unstale":"2024-03-06T07:27:12.341384+0000","last_undegraded":"2024-03-06T07:27:12.341384+0000","last_fullsized":"2024-03-06T07:27:12.341384+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:18:42.653388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.340490+0000","last_change":"2024-03-06T07:27:11.162715+0000","last_active":"2024-03-06T07:27:12.340490+0000","last_peered":"2024-03-06T07:27:12.340490+0000","last_clean":"2024-03-06T07:27:12.340490+0000","last_became_active":"2024-03-06T07:27:11.162436+0000","last_became_peered":"2024-03-06T07:27:11.162436+0000","last_unstale":"2024-03-06T07:27:12.340490+0000","last_undegraded":"2024-03-06T07:27:12.340490+0000","last_fullsized":"2024-03-06T07:27:12.340490+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:10.126521+0000","last_clean_scrub_stamp":"2024-03-06T07:27:10.126521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:13.747047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.340733+0000","last_change":"2024-03-06T07:27:10.151075+0000","last_active":"2024-03-06T07:27:12.340733+0000","last_peered":"2024-03-06T07:27:12.340733+0000","last_clean":"2024-03-06T07:27:12.340733+0000","last_became_active":"2024-03-06T07:27:10.150826+0000","last_became_peered":"2024-03-06T07:27:10.150826+0000","last_unstale":"2024-03-06T07:27:12.340733+0000","last_undegraded":"2024-03-06T07:27:12.340733+0000","last_fullsized":"2024-03-06T07:27:12.340733+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.121909+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.121909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:12:46.075963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":331,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:12.341316+0000","last_change":"2024-03-06T07:21:47.553693+0000","last_active":"2024-03-06T07:27:12.341316+0000","last_peered":"2024-03-06T07:27:12.341316+0000","last_clean":"2024-03-06T07:27:12.341316+0000","last_became_active":"2024-03-06T07:21:47.552664+0000","last_became_peered":"2024-03-06T07:21:47.552664+0000","last_unstale":"2024-03-06T07:27:12.341316+0000","last_undegraded":"2024-03-06T07:27:12.341316+0000","last_fullsized":"2024-03-06T07:27:12.341316+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_clean_scrub_stamp":"2024-03-06T07:21:46.362028+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:02.916426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574919,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37264,"kb_used_data":1924,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334576,"statfs":{"total":96636764160,"available":96598605824,"internally_reserved":0,"allocated":1970176,"data_stored":1180663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":36174281},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574917,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40840,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331000,"statfs":{"total":96636764160,"available":96594944000,"internally_reserved":0,"allocated":1372160,"data_stored":589749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574918,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40844,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330996,"statfs":{"total":96636764160,"available":96594939904,"internally_reserved":0,"allocated":1376256,"data_stored":590283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574919,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40840,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331000,"statfs":{"total":96636764160,"available":96594944000,"internally_reserved":0,"allocated":1368064,"data_stored":589679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574918,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40852,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330988,"statfs":{"total":96636764160,"available":96594931712,"internally_reserved":0,"allocated":1384448,"data_stored":591361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574918,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40836,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331004,"statfs":{"total":96636764160,"available":96594948096,"internally_reserved":0,"allocated":1368064,"data_stored":589679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574919,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40844,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330996,"statfs":{"total":96636764160,"available":96594939904,"internally_reserved":0,"allocated":1376256,"data_stored":589803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574918,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41436,"kb_used_data":1936,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330404,"statfs":{"total":96636764160,"available":96594333696,"internally_reserved":0,"allocated":1982464,"data_stored":1181771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:19.060 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:27:19.061 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:27:19.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.211+0000 7f076186b640 1 Processor -- start 2024-03-06T07:27:19.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.211+0000 7f076186b640 1 -- start start 2024-03-06T07:27:19.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.212+0000 7f076186b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f075c0cf200 0x7f075c0cf690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.212+0000 7f076186b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f075c1600a0 con 0x7f075c0d06e0 2024-03-06T07:27:19.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.212+0000 7f076186b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f075c0d47f0 con 0x7f075c15fc30 2024-03-06T07:27:19.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.212+0000 7f076186b640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f075c0ced70 con 0x7f075c0cf200 2024-03-06T07:27:19.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.212+0000 7f075affd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f075c0d06e0 0x7f075c0d0ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57086/0 (socket says 172.21.15.83:57086) 2024-03-06T07:27:19.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.212+0000 7f075affd640 1 -- 172.21.15.83:0/4244371614 learned_addr learned my addr 172.21.15.83:0/4244371614 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:27:19.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.212+0000 7f075b7fe640 1 --2- 172.21.15.83:0/4244371614 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f075c0cf200 0x7f075c0cf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.213+0000 7f0759ffb640 1 -- 172.21.15.83:0/4244371614 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4090665735 0 0) 0x7f075c1600a0 con 0x7f075c0d06e0 2024-03-06T07:27:19.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.213+0000 7f0759ffb640 1 -- 172.21.15.83:0/4244371614 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0744003520 con 0x7f075c0d06e0 2024-03-06T07:27:19.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.213+0000 7f0759ffb640 1 -- 172.21.15.83:0/4244371614 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4139856204 0 0) 0x7f075c0d47f0 con 0x7f075c15fc30 2024-03-06T07:27:19.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.213+0000 7f0759ffb640 1 -- 172.21.15.83:0/4244371614 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f075c1600a0 con 0x7f075c15fc30 2024-03-06T07:27:19.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.213+0000 7f0759ffb640 1 -- 172.21.15.83:0/4244371614 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1965536616 0 0) 0x7f0744003520 con 0x7f075c0d06e0 2024-03-06T07:27:19.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.213+0000 7f075b7fe640 1 -- 172.21.15.83:0/4244371614 >> v1:172.21.15.183:6789/0 conn(0x7f075c15fc30 legacy=0x7f075c0b47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.213+0000 7f075b7fe640 1 -- 172.21.15.83:0/4244371614 >> v1:172.21.15.83:6789/0 conn(0x7f075c0d06e0 legacy=0x7f075c0d0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.213+0000 7f075b7fe640 1 -- 172.21.15.83:0/4244371614 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f075c0d1ab0 con 0x7f075c0cf200 2024-03-06T07:27:19.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.214+0000 7f075b7fe640 1 --2- 172.21.15.83:0/4244371614 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f075c0cf200 0x7f075c0cf690 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0754009dc0 tx=0x7f075402bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a55a15c56530d0be server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:19.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.214+0000 7f0759ffb640 1 -- 172.21.15.83:0/4244371614 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0754039070 con 0x7f075c0cf200 2024-03-06T07:27:19.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.214+0000 7f0759ffb640 1 -- 172.21.15.83:0/4244371614 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f075402c910 con 0x7f075c0cf200 2024-03-06T07:27:19.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.214+0000 7f0759ffb640 1 -- 172.21.15.83:0/4244371614 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f074c001670 con 0x7f075c0cf200 2024-03-06T07:27:19.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.215+0000 7f076186b640 1 -- 172.21.15.83:0/4244371614 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f075c0cf200 msgr2=0x7f075c0cf690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.215+0000 7f076186b640 1 --2- 172.21.15.83:0/4244371614 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f075c0cf200 0x7f075c0cf690 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0754009dc0 tx=0x7f075402bde0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.215+0000 7f076186b640 1 -- 172.21.15.83:0/4244371614 shutdown_connections 2024-03-06T07:27:19.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.215+0000 7f076186b640 1 --2- 172.21.15.83:0/4244371614 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f075c0cf200 0x7f075c0cf690 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.215+0000 7f076186b640 1 -- 172.21.15.83:0/4244371614 >> 172.21.15.83:0/4244371614 conn(0x7f075c0d3cd0 msgr2=0x7f075c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:19.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.215+0000 7f076186b640 1 -- 172.21.15.83:0/4244371614 shutdown_connections 2024-03-06T07:27:19.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.215+0000 7f076186b640 1 -- 172.21.15.83:0/4244371614 wait complete. 2024-03-06T07:27:19.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.216+0000 7f076186b640 1 Processor -- start 2024-03-06T07:27:19.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.216+0000 7f076186b640 1 -- start start 2024-03-06T07:27:19.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.217+0000 7f076186b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f075c0cf200 0x7f075c1f4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.217+0000 7f076186b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f075c0d06e0 0x7f075c1f7f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.217+0000 7f076186b640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f075c15fc30 0x7f075c153be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.217+0000 7f076186b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f075c0ce990 con 0x7f075c0cf200 2024-03-06T07:27:19.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.217+0000 7f076186b640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f075c0d4ab0 con 0x7f075c15fc30 2024-03-06T07:27:19.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.217+0000 7f076186b640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f075c0b44d0 con 0x7f075c0d06e0 2024-03-06T07:27:19.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.217+0000 7f075affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f075c0cf200 0x7f075c1f4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.217+0000 7f075affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f075c0cf200 0x7f075c1f4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41442/0 (socket says 172.21.15.83:41442) 2024-03-06T07:27:19.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.217+0000 7f075affd640 1 -- 172.21.15.83:0/3532840891 learned_addr learned my addr 172.21.15.83:0/3532840891 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:19.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.217+0000 7f075affd640 1 -- 172.21.15.83:0/3532840891 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f075c0d06e0 msgr2=0x7f075c1f7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.217+0000 7f075affd640 1 --2- 172.21.15.83:0/3532840891 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f075c0d06e0 0x7f075c1f7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.217+0000 7f075affd640 1 -- 172.21.15.83:0/3532840891 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f075c15fc30 msgr2=0x7f075c153be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.217+0000 7f075affd640 1 --2- 172.21.15.83:0/3532840891 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f075c15fc30 0x7f075c153be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.217+0000 7f075affd640 1 -- 172.21.15.83:0/3532840891 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f075c1f89b0 con 0x7f075c0cf200 2024-03-06T07:27:19.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.217+0000 7f075affd640 1 --2- 172.21.15.83:0/3532840891 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f075c0cf200 0x7f075c1f4290 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f074c00c760 tx=0x7f074c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:19.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.218+0000 7f0743fff640 1 -- 172.21.15.83:0/3532840891 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f074c017070 con 0x7f075c0cf200 2024-03-06T07:27:19.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.218+0000 7f0743fff640 1 -- 172.21.15.83:0/3532840891 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f074c005ad0 con 0x7f075c0cf200 2024-03-06T07:27:19.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.218+0000 7f076186b640 1 -- 172.21.15.83:0/3532840891 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f075c1f8c90 con 0x7f075c0cf200 2024-03-06T07:27:19.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.218+0000 7f0743fff640 1 -- 172.21.15.83:0/3532840891 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f074c007070 con 0x7f075c0cf200 2024-03-06T07:27:19.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.218+0000 7f076186b640 1 -- 172.21.15.83:0/3532840891 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f075c1f47d0 con 0x7f075c0cf200 2024-03-06T07:27:19.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.219+0000 7f076186b640 1 -- 172.21.15.83:0/3532840891 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f075c0a1c80 con 0x7f075c0cf200 2024-03-06T07:27:19.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.219+0000 7f0743fff640 1 -- 172.21.15.83:0/3532840891 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f074c01a070 con 0x7f075c0cf200 2024-03-06T07:27:19.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.219+0000 7f0743fff640 1 --2- 172.21.15.83:0/3532840891 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f074806b4d0 0x7f074806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.220+0000 7f0743fff640 1 -- 172.21.15.83:0/3532840891 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f074c089700 con 0x7f075c0cf200 2024-03-06T07:27:19.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.220+0000 7f075a7fc640 1 --2- 172.21.15.83:0/3532840891 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f074806b4d0 0x7f074806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.220+0000 7f075a7fc640 1 --2- 172.21.15.83:0/3532840891 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f074806b4d0 0x7f074806d990 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f07500010e0 tx=0x7f0750007950 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:19.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.224+0000 7f0743fff640 1 -- 172.21.15.83:0/3532840891 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f074c058610 con 0x7f075c0cf200 2024-03-06T07:27:19.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.470+0000 7f076186b640 1 -- 172.21.15.83:0/3532840891 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f075c0ce080 con 0x7f075c0cf200 2024-03-06T07:27:19.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.472+0000 7f0743fff640 1 -- 172.21.15.83:0/3532840891 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) v1 ==== 190+0+147 (secure 0 0 0) 0x7f074c057f10 con 0x7f075c0cf200 2024-03-06T07:27:19.474 INFO:teuthology.orchestra.run.smithi083.stdout:[client.0] 2024-03-06T07:27:19.474 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBcGehlRwCIMhAA1DdbdQZPplu7LQOTK3BmSg== 2024-03-06T07:27:19.474 INFO:teuthology.orchestra.run.smithi083.stdout: caps mds = "allow" 2024-03-06T07:27:19.474 INFO:teuthology.orchestra.run.smithi083.stdout: caps mon = "allow r" 2024-03-06T07:27:19.474 INFO:teuthology.orchestra.run.smithi083.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:27:19.475 INFO:teuthology.orchestra.run.smithi083.stderr:updated caps for client.0 2024-03-06T07:27:19.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.474+0000 7f076186b640 1 -- 172.21.15.83:0/3532840891 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f074806b4d0 msgr2=0x7f074806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.474+0000 7f076186b640 1 --2- 172.21.15.83:0/3532840891 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f074806b4d0 0x7f074806d990 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f07500010e0 tx=0x7f0750007950 comp rx=0 tx=0).stop 2024-03-06T07:27:19.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.474+0000 7f076186b640 1 -- 172.21.15.83:0/3532840891 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f075c0cf200 msgr2=0x7f075c1f4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.474+0000 7f076186b640 1 --2- 172.21.15.83:0/3532840891 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f075c0cf200 0x7f075c1f4290 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f074c00c760 tx=0x7f074c00cc20 comp rx=0 tx=0).stop 2024-03-06T07:27:19.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.474+0000 7f076186b640 1 -- 172.21.15.83:0/3532840891 shutdown_connections 2024-03-06T07:27:19.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.474+0000 7f076186b640 1 --2- 172.21.15.83:0/3532840891 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f074806b4d0 0x7f074806d990 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.474+0000 7f076186b640 1 --2- 172.21.15.83:0/3532840891 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f075c15fc30 0x7f075c153be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.474+0000 7f076186b640 1 --2- 172.21.15.83:0/3532840891 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f075c0d06e0 0x7f075c1f7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.474+0000 7f076186b640 1 --2- 172.21.15.83:0/3532840891 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f075c0cf200 0x7f075c1f4290 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.474+0000 7f076186b640 1 -- 172.21.15.83:0/3532840891 >> 172.21.15.83:0/3532840891 conn(0x7f075c0d3cd0 msgr2=0x7f075c161dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:19.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.474+0000 7f076186b640 1 -- 172.21.15.83:0/3532840891 shutdown_connections 2024-03-06T07:27:19.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.474+0000 7f076186b640 1 -- 172.21.15.83:0/3532840891 wait complete. 2024-03-06T07:27:19.487 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:19.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.603+0000 7f02d9615640 1 Processor -- start 2024-03-06T07:27:19.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.603+0000 7f02d9615640 1 -- start start 2024-03-06T07:27:19.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.604+0000 7f02d9615640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f02d40d4760 0x7f02d40b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.604+0000 7f02d9615640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02d40d25d0 con 0x7f02d40b4170 2024-03-06T07:27:19.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.604+0000 7f02d9615640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02d40d2870 con 0x7f02d40d0a80 2024-03-06T07:27:19.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.604+0000 7f02d9615640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02d40d4bf0 con 0x7f02d40d4760 2024-03-06T07:27:19.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.604+0000 7f02d2ffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f02d40b4170 0x7f02d40d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57090/0 (socket says 172.21.15.83:57090) 2024-03-06T07:27:19.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.604+0000 7f02d2ffd640 1 -- 172.21.15.83:0/365519139 learned_addr learned my addr 172.21.15.83:0/365519139 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:27:19.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.604+0000 7f02d27fc640 1 --2- 172.21.15.83:0/365519139 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f02d40d4760 0x7f02d40b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.604+0000 7f02d1ffb640 1 -- 172.21.15.83:0/365519139 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1837191861 0 0) 0x7f02d40d25d0 con 0x7f02d40b4170 2024-03-06T07:27:19.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.604+0000 7f02d1ffb640 1 -- 172.21.15.83:0/365519139 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02bc003520 con 0x7f02d40b4170 2024-03-06T07:27:19.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.604+0000 7f02d1ffb640 1 -- 172.21.15.83:0/365519139 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3238100802 0 0) 0x7f02bc003520 con 0x7f02d40b4170 2024-03-06T07:27:19.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.604+0000 7f02d1ffb640 1 -- 172.21.15.83:0/365519139 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f02d40d4760 msgr2=0x7f02d40b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:19.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.604+0000 7f02d1ffb640 1 --2- 172.21.15.83:0/365519139 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f02d40d4760 0x7f02d40b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.604+0000 7f02d1ffb640 1 -- 172.21.15.83:0/365519139 >> v1:172.21.15.183:6789/0 conn(0x7f02d40d0a80 legacy=0x7f02d40cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.605+0000 7f02d1ffb640 1 -- 172.21.15.83:0/365519139 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d40d2b50 con 0x7f02d40b4170 2024-03-06T07:27:19.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.605+0000 7f02d1ffb640 1 -- 172.21.15.83:0/365519139 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f02c8001670 con 0x7f02d40b4170 2024-03-06T07:27:19.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.605+0000 7f02d1ffb640 1 -- 172.21.15.83:0/365519139 >> v1:172.21.15.83:6789/0 conn(0x7f02d40b4170 legacy=0x7f02d40d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.605+0000 7f02d1ffb640 1 --2- 172.21.15.83:0/365519139 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f02bc0039e0 0x7f02bc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.605+0000 7f02d1ffb640 1 --2- 172.21.15.83:0/365519139 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f02bc004390 0x7f02bc004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.605+0000 7f02d1ffb640 1 --2- 172.21.15.83:0/365519139 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02bc004f80 0x7f02bc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.605+0000 7f02d1ffb640 1 -- 172.21.15.83:0/365519139 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d40d4bf0 con 0x7f02bc004f80 2024-03-06T07:27:19.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.605+0000 7f02d1ffb640 1 -- 172.21.15.83:0/365519139 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02bc00bf00 con 0x7f02bc0039e0 2024-03-06T07:27:19.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.605+0000 7f02d1ffb640 1 -- 172.21.15.83:0/365519139 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02bc00c080 con 0x7f02bc004390 2024-03-06T07:27:19.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.605+0000 7f02d37fe640 1 --2- 172.21.15.83:0/365519139 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02bc004f80 0x7f02bc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.605+0000 7f02d2ffd640 1 --2- 172.21.15.83:0/365519139 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f02bc0039e0 0x7f02bc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.605+0000 7f02d27fc640 1 --2- 172.21.15.83:0/365519139 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f02bc004390 0x7f02bc004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.605+0000 7f02d37fe640 1 -- 172.21.15.83:0/365519139 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f02bc004390 msgr2=0x7f02bc004820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:19.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.605+0000 7f02d37fe640 1 --2- 172.21.15.83:0/365519139 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f02bc004390 0x7f02bc004820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.605+0000 7f02d37fe640 1 -- 172.21.15.83:0/365519139 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f02bc0039e0 msgr2=0x7f02bc003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:19.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.605+0000 7f02d37fe640 1 --2- 172.21.15.83:0/365519139 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f02bc0039e0 0x7f02bc003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.605+0000 7f02d37fe640 1 -- 172.21.15.83:0/365519139 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02bc003520 con 0x7f02bc004f80 2024-03-06T07:27:19.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.606+0000 7f02d37fe640 1 --2- 172.21.15.83:0/365519139 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02bc004f80 0x7f02bc00b9c0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f02cc001f60 tx=0x7f02bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a0368ea23ae6ac0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:19.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.606+0000 7f02d1ffb640 1 -- 172.21.15.83:0/365519139 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02cc001b30 con 0x7f02bc004f80 2024-03-06T07:27:19.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.606+0000 7f02d1ffb640 1 -- 172.21.15.83:0/365519139 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02cc02d6e0 con 0x7f02bc004f80 2024-03-06T07:27:19.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.606+0000 7f02d1ffb640 1 -- 172.21.15.83:0/365519139 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02cc02dad0 con 0x7f02bc004f80 2024-03-06T07:27:19.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.606+0000 7f02d9615640 1 -- 172.21.15.83:0/365519139 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02bc004f80 msgr2=0x7f02bc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.606+0000 7f02d9615640 1 --2- 172.21.15.83:0/365519139 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02bc004f80 0x7f02bc00b9c0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f02cc001f60 tx=0x7f02bc005830 comp rx=0 tx=0).stop 2024-03-06T07:27:19.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.606+0000 7f02d37fe640 1 -- 172.21.15.83:0/365519139 reap_dead start 2024-03-06T07:27:19.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.606+0000 7f02d9615640 1 -- 172.21.15.83:0/365519139 shutdown_connections 2024-03-06T07:27:19.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.606+0000 7f02d9615640 1 -- 172.21.15.83:0/365519139 >> 172.21.15.83:0/365519139 conn(0x7f02d40d3c40 msgr2=0x7f02d40d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:19.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.606+0000 7f02d9615640 1 -- 172.21.15.83:0/365519139 shutdown_connections 2024-03-06T07:27:19.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.606+0000 7f02d9615640 1 -- 172.21.15.83:0/365519139 wait complete. 2024-03-06T07:27:19.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.608+0000 7f02d9615640 1 Processor -- start 2024-03-06T07:27:19.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.608+0000 7f02d9615640 1 -- start start 2024-03-06T07:27:19.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.608+0000 7f02d9615640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02d40b33e0 0x7f02d40c9560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.608+0000 7f02d9615640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f02d40b37c0 0x7f02d40c7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.608+0000 7f02d9615640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f02d41f3570 0x7f02d41f5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.608+0000 7f02d9615640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d40b4dd0 con 0x7f02d40b33e0 2024-03-06T07:27:19.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.608+0000 7f02d9615640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d40ce9e0 con 0x7f02d41f3570 2024-03-06T07:27:19.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.608+0000 7f02d9615640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02d40b4f50 con 0x7f02d40b37c0 2024-03-06T07:27:19.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.609+0000 7f02d2ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02d40b33e0 0x7f02d40c9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.609+0000 7f02d2ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02d40b33e0 0x7f02d40c9560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41448/0 (socket says 172.21.15.83:41448) 2024-03-06T07:27:19.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.609+0000 7f02d2ffd640 1 -- 172.21.15.83:0/79329797 learned_addr learned my addr 172.21.15.83:0/79329797 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:19.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.609+0000 7f02d37fe640 1 --2- 172.21.15.83:0/79329797 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f02d41f3570 0x7f02d41f5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.609+0000 7f02d27fc640 1 --2- 172.21.15.83:0/79329797 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f02d40b37c0 0x7f02d40c7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.609+0000 7f02d2ffd640 1 -- 172.21.15.83:0/79329797 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f02d40b37c0 msgr2=0x7f02d40c7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.609+0000 7f02d2ffd640 1 --2- 172.21.15.83:0/79329797 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f02d40b37c0 0x7f02d40c7b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.609+0000 7f02d2ffd640 1 -- 172.21.15.83:0/79329797 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f02d41f3570 msgr2=0x7f02d41f5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.609+0000 7f02d2ffd640 1 --2- 172.21.15.83:0/79329797 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f02d41f3570 0x7f02d41f5f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.609+0000 7f02d2ffd640 1 -- 172.21.15.83:0/79329797 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d40c80a0 con 0x7f02d40b33e0 2024-03-06T07:27:19.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.609+0000 7f02d37fe640 1 --2- 172.21.15.83:0/79329797 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f02d41f3570 0x7f02d41f5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:19.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.609+0000 7f02d27fc640 1 --2- 172.21.15.83:0/79329797 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f02d40b37c0 0x7f02d40c7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:19.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.609+0000 7f02d2ffd640 1 --2- 172.21.15.83:0/79329797 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02d40b33e0 0x7f02d40c9560 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f02c80011e0 tx=0x7f02c8007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:19.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.609+0000 7f02bb7fe640 1 -- 172.21.15.83:0/79329797 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c800f940 con 0x7f02d40b33e0 2024-03-06T07:27:19.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.609+0000 7f02bb7fe640 1 -- 172.21.15.83:0/79329797 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02c8018070 con 0x7f02d40b33e0 2024-03-06T07:27:19.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.609+0000 7f02d9615640 1 -- 172.21.15.83:0/79329797 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02d40c8340 con 0x7f02d40b33e0 2024-03-06T07:27:19.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.609+0000 7f02bb7fe640 1 -- 172.21.15.83:0/79329797 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c800fd40 con 0x7f02d40b33e0 2024-03-06T07:27:19.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.609+0000 7f02d9615640 1 -- 172.21.15.83:0/79329797 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02d40b0ce0 con 0x7f02d40b33e0 2024-03-06T07:27:19.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.610+0000 7f02d9615640 1 -- 172.21.15.83:0/79329797 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02d40b07f0 con 0x7f02d40b33e0 2024-03-06T07:27:19.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.611+0000 7f02bb7fe640 1 -- 172.21.15.83:0/79329797 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f02c801d460 con 0x7f02d40b33e0 2024-03-06T07:27:19.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.611+0000 7f02bb7fe640 1 --2- 172.21.15.83:0/79329797 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f02c006b500 0x7f02c006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.611+0000 7f02bb7fe640 1 -- 172.21.15.83:0/79329797 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f02c808a230 con 0x7f02d40b33e0 2024-03-06T07:27:19.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.611+0000 7f02d27fc640 1 --2- 172.21.15.83:0/79329797 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f02c006b500 0x7f02c006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.611+0000 7f02d27fc640 1 --2- 172.21.15.83:0/79329797 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f02c006b500 0x7f02c006d9c0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f02d40c8880 tx=0x7f02c4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:19.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.615+0000 7f02bb7fe640 1 -- 172.21.15.83:0/79329797 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f02c8059140 con 0x7f02d40b33e0 2024-03-06T07:27:19.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.823+0000 7f02d9615640 1 -- 172.21.15.83:0/79329797 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02d404f330 con 0x7f02d40b33e0 2024-03-06T07:27:19.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.824+0000 7f02bb7fe640 1 -- 172.21.15.83:0/79329797 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f02c8058a40 con 0x7f02d40b33e0 2024-03-06T07:27:19.826 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:19.826 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":59,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6200,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/2154269119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":2154269119},{"type":"v1","addr":"172.21.15.83:6837","nonce":2154269119}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6206,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3898960075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":3898960075},{"type":"v1","addr":"172.21.15.183:6835","nonce":3898960075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6838,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/1383294888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":1383294888},{"type":"v1","addr":"172.21.15.183:6834","nonce":1383294888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.491022+0000","modified":"2024-03-06T07:27:12.512838+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7887},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7887":{"gid":7887,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6836/855387870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":855387870},{"type":"v1","addr":"172.21.15.83:6836","nonce":855387870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7887,"qdb_cluster":[7887]},"id":8}]} 2024-03-06T07:27:19.826 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 59 2024-03-06T07:27:19.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.825+0000 7f02d9615640 1 -- 172.21.15.83:0/79329797 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f02c006b500 msgr2=0x7f02c006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.825+0000 7f02d9615640 1 --2- 172.21.15.83:0/79329797 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f02c006b500 0x7f02c006d9c0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f02d40c8880 tx=0x7f02c4008040 comp rx=0 tx=0).stop 2024-03-06T07:27:19.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.825+0000 7f02d9615640 1 -- 172.21.15.83:0/79329797 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02d40b33e0 msgr2=0x7f02d40c9560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.825+0000 7f02d9615640 1 --2- 172.21.15.83:0/79329797 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02d40b33e0 0x7f02d40c9560 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f02c80011e0 tx=0x7f02c8007720 comp rx=0 tx=0).stop 2024-03-06T07:27:19.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.826+0000 7f02d9615640 1 -- 172.21.15.83:0/79329797 shutdown_connections 2024-03-06T07:27:19.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.826+0000 7f02d9615640 1 --2- 172.21.15.83:0/79329797 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f02c006b500 0x7f02c006d9c0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.826+0000 7f02d9615640 1 --2- 172.21.15.83:0/79329797 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f02d41f3570 0x7f02d41f5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.826+0000 7f02d9615640 1 --2- 172.21.15.83:0/79329797 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f02d40b37c0 0x7f02d40c7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.826+0000 7f02d9615640 1 --2- 172.21.15.83:0/79329797 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f02d40b33e0 0x7f02d40c9560 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.826+0000 7f02d9615640 1 -- 172.21.15.83:0/79329797 >> 172.21.15.83:0/79329797 conn(0x7f02d40d3c40 msgr2=0x7f02d40b7130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:19.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.826+0000 7f02d9615640 1 -- 172.21.15.83:0/79329797 shutdown_connections 2024-03-06T07:27:19.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:19.826+0000 7f02d9615640 1 -- 172.21.15.83:0/79329797 wait complete. 2024-03-06T07:27:20.241 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 59, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:10.491022+0000', 'modified': '2024-03-06T07:27:12.512838+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7887}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7887': {'gid': 7887, 'name': 'a', 'rank': 0, 'incarnation': 57, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.83:6836/855387870', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 855387870}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 855387870}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7887, 'qdb_cluster': [7887]} 2024-03-06T07:27:20.241 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:27:20.241 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:27:20.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.346+0000 7f16874b5640 1 Processor -- start 2024-03-06T07:27:20.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.346+0000 7f16874b5640 1 -- start start 2024-03-06T07:27:20.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.347+0000 7f16874b5640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f1680162400 0x7f16800d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.347+0000 7f16874b5640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16800d22c0 con 0x7f1680162970 2024-03-06T07:27:20.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.347+0000 7f16874b5640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16800d25a0 con 0x7f16800b4460 2024-03-06T07:27:20.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.347+0000 7f16874b5640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1680161b00 con 0x7f1680162400 2024-03-06T07:27:20.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.347+0000 7f1684a50640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f1680162970 0x7f1680161cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57098/0 (socket says 172.21.15.83:57098) 2024-03-06T07:27:20.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.347+0000 7f1684a50640 1 -- 172.21.15.83:0/3273340741 learned_addr learned my addr 172.21.15.83:0/3273340741 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:27:20.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.347+0000 7f1677fff640 1 -- 172.21.15.83:0/3273340741 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1113808056 0 0) 0x7f16800d22c0 con 0x7f1680162970 2024-03-06T07:27:20.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.347+0000 7f1677fff640 1 -- 172.21.15.83:0/3273340741 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f166c003520 con 0x7f1680162970 2024-03-06T07:27:20.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.347+0000 7f1685a52640 1 --2- 172.21.15.83:0/3273340741 >> v2:172.21.15.183:3300/0 conn(0x7f1680162400 0x7f16800d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.347+0000 7f1677fff640 1 -- 172.21.15.83:0/3273340741 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 786565972 0 0) 0x7f166c003520 con 0x7f1680162970 2024-03-06T07:27:20.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.348+0000 7f1677fff640 1 -- 172.21.15.83:0/3273340741 >> v2:172.21.15.183:3300/0 conn(0x7f1680162400 msgr2=0x7f16800d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:20.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.348+0000 7f1677fff640 1 --2- 172.21.15.83:0/3273340741 >> v2:172.21.15.183:3300/0 conn(0x7f1680162400 0x7f16800d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.348+0000 7f1677fff640 1 -- 172.21.15.83:0/3273340741 >> v1:172.21.15.183:6789/0 conn(0x7f16800b4460 legacy=0x7f16800b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.348+0000 7f1677fff640 1 -- 172.21.15.83:0/3273340741 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16800d2880 con 0x7f1680162970 2024-03-06T07:27:20.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.348+0000 7f1677fff640 1 -- 172.21.15.83:0/3273340741 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f1670001670 con 0x7f1680162970 2024-03-06T07:27:20.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.348+0000 7f1677fff640 1 -- 172.21.15.83:0/3273340741 >> v1:172.21.15.83:6789/0 conn(0x7f1680162970 legacy=0x7f1680161cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.348+0000 7f1677fff640 1 --2- 172.21.15.83:0/3273340741 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f166c0039b0 0x7f166c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.348+0000 7f1677fff640 1 --2- 172.21.15.83:0/3273340741 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f166c004800 0x7f166c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.348+0000 7f1677fff640 1 --2- 172.21.15.83:0/3273340741 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f166c0051c0 0x7f166c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.348+0000 7f1677fff640 1 -- 172.21.15.83:0/3273340741 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1680161b00 con 0x7f166c0051c0 2024-03-06T07:27:20.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.348+0000 7f1677fff640 1 -- 172.21.15.83:0/3273340741 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f166c00c110 con 0x7f166c0039b0 2024-03-06T07:27:20.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.348+0000 7f1677fff640 1 -- 172.21.15.83:0/3273340741 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f166c00c310 con 0x7f166c004800 2024-03-06T07:27:20.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.348+0000 7f1685a52640 1 --2- 172.21.15.83:0/3273340741 >> v2:172.21.15.183:3300/0 conn(0x7f1680162400 0x7f16800d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:20.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.348+0000 7f1685a52640 1 --2- 172.21.15.83:0/3273340741 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f166c0051c0 0x7f166c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.348+0000 7f1684a50640 1 --2- 172.21.15.83:0/3273340741 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f166c004800 0x7f166c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.348+0000 7f1685251640 1 --2- 172.21.15.83:0/3273340741 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f166c0039b0 0x7f166c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.348+0000 7f1685a52640 1 -- 172.21.15.83:0/3273340741 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f166c004800 msgr2=0x7f166c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:20.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.348+0000 7f1685a52640 1 --2- 172.21.15.83:0/3273340741 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f166c004800 0x7f166c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.348+0000 7f1685a52640 1 -- 172.21.15.83:0/3273340741 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f166c0039b0 msgr2=0x7f166c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:20.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.348+0000 7f1685a52640 1 --2- 172.21.15.83:0/3273340741 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f166c0039b0 0x7f166c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.348+0000 7f1685a52640 1 -- 172.21.15.83:0/3273340741 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f166c003520 con 0x7f166c0051c0 2024-03-06T07:27:20.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.349+0000 7f1685a52640 1 -- 172.21.15.83:0/3273340741 reap_dead start 2024-03-06T07:27:20.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.349+0000 7f1685a52640 1 --2- 172.21.15.83:0/3273340741 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f166c0051c0 0x7f166c00b9c0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f16780073b0 tx=0x7f166c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32995bde9a582104 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:20.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.349+0000 7f1677fff640 1 -- 172.21.15.83:0/3273340741 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1678039070 con 0x7f166c0051c0 2024-03-06T07:27:20.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.349+0000 7f1677fff640 1 -- 172.21.15.83:0/3273340741 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f167802cdb0 con 0x7f166c0051c0 2024-03-06T07:27:20.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.349+0000 7f1677fff640 1 -- 172.21.15.83:0/3273340741 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1678009c70 con 0x7f166c0051c0 2024-03-06T07:27:20.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.350+0000 7f16874b5640 1 -- 172.21.15.83:0/3273340741 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f166c0051c0 msgr2=0x7f166c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.350+0000 7f16874b5640 1 --2- 172.21.15.83:0/3273340741 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f166c0051c0 0x7f166c00b9c0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f16780073b0 tx=0x7f166c005830 comp rx=0 tx=0).stop 2024-03-06T07:27:20.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.350+0000 7f16874b5640 1 -- 172.21.15.83:0/3273340741 shutdown_connections 2024-03-06T07:27:20.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.350+0000 7f16874b5640 1 --2- 172.21.15.83:0/3273340741 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f166c0051c0 0x7f166c00b9c0 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.350+0000 7f16874b5640 1 -- 172.21.15.83:0/3273340741 >> 172.21.15.83:0/3273340741 conn(0x7f16800d3ce0 msgr2=0x7f16800d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:20.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.350+0000 7f16874b5640 1 -- 172.21.15.83:0/3273340741 shutdown_connections 2024-03-06T07:27:20.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.350+0000 7f16874b5640 1 -- 172.21.15.83:0/3273340741 wait complete. 2024-03-06T07:27:20.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.351+0000 7f16874b5640 1 Processor -- start 2024-03-06T07:27:20.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.351+0000 7f16874b5640 1 -- start start 2024-03-06T07:27:20.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.352+0000 7f16874b5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f166c0051c0 0x7f16800b7d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.352+0000 7f16874b5640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16801560f0 0x7f16800c9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.352+0000 7f16874b5640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16800ca3f0 0x7f16800d03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.352+0000 7f16874b5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16800ce950 con 0x7f166c0051c0 2024-03-06T07:27:20.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.352+0000 7f16874b5640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16800cecc0 con 0x7f16800ca3f0 2024-03-06T07:27:20.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.352+0000 7f16874b5640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16800b4840 con 0x7f16801560f0 2024-03-06T07:27:20.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.352+0000 7f1685251640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f166c0051c0 0x7f16800b7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.352+0000 7f1685251640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f166c0051c0 0x7f16800b7d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41462/0 (socket says 172.21.15.83:41462) 2024-03-06T07:27:20.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.352+0000 7f1685251640 1 -- 172.21.15.83:0/2065669377 learned_addr learned my addr 172.21.15.83:0/2065669377 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:20.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.352+0000 7f1685a52640 1 --2- 172.21.15.83:0/2065669377 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16800ca3f0 0x7f16800d03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.352+0000 7f1684a50640 1 --2- 172.21.15.83:0/2065669377 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16801560f0 0x7f16800c9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.352+0000 7f1685251640 1 -- 172.21.15.83:0/2065669377 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16801560f0 msgr2=0x7f16800c9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.352+0000 7f1685251640 1 --2- 172.21.15.83:0/2065669377 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16801560f0 0x7f16800c9eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.352+0000 7f1685251640 1 -- 172.21.15.83:0/2065669377 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16800ca3f0 msgr2=0x7f16800d03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.352+0000 7f1685251640 1 --2- 172.21.15.83:0/2065669377 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16800ca3f0 0x7f16800d03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.352+0000 7f1685251640 1 -- 172.21.15.83:0/2065669377 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16800b6ca0 con 0x7f166c0051c0 2024-03-06T07:27:20.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.352+0000 7f1685a52640 1 --2- 172.21.15.83:0/2065669377 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16800ca3f0 0x7f16800d03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:20.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.352+0000 7f1684a50640 1 --2- 172.21.15.83:0/2065669377 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16801560f0 0x7f16800c9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:20.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.352+0000 7f1685251640 1 --2- 172.21.15.83:0/2065669377 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f166c0051c0 0x7f16800b7d40 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f167c00f620 tx=0x7f167c00fae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:20.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.352+0000 7f1675ffb640 1 -- 172.21.15.83:0/2065669377 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f167c0053a0 con 0x7f166c0051c0 2024-03-06T07:27:20.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.352+0000 7f1675ffb640 1 -- 172.21.15.83:0/2065669377 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f167c018070 con 0x7f166c0051c0 2024-03-06T07:27:20.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.352+0000 7f16874b5640 1 -- 172.21.15.83:0/2065669377 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16800b6f80 con 0x7f166c0051c0 2024-03-06T07:27:20.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.352+0000 7f1675ffb640 1 -- 172.21.15.83:0/2065669377 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f167c003830 con 0x7f166c0051c0 2024-03-06T07:27:20.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.353+0000 7f16874b5640 1 -- 172.21.15.83:0/2065669377 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16800b7540 con 0x7f166c0051c0 2024-03-06T07:27:20.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.354+0000 7f1675ffb640 1 -- 172.21.15.83:0/2065669377 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f167c0039d0 con 0x7f166c0051c0 2024-03-06T07:27:20.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.355+0000 7f1675ffb640 1 --2- 172.21.15.83:0/2065669377 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f166406b4d0 0x7f166406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.355+0000 7f1684a50640 1 --2- 172.21.15.83:0/2065669377 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f166406b4d0 0x7f166406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.355+0000 7f1675ffb640 1 -- 172.21.15.83:0/2065669377 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f167c089fb0 con 0x7f166c0051c0 2024-03-06T07:27:20.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.355+0000 7f1684a50640 1 --2- 172.21.15.83:0/2065669377 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f166406b4d0 0x7f166406d990 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f16800c7560 tx=0x7f1670009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:20.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.356+0000 7f168a17f740 1 Processor -- start 2024-03-06T07:27:20.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.356+0000 7f168a17f740 1 -- start start 2024-03-06T07:27:20.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.356+0000 7f168a17f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5605d64b6870 0x5605d64b6c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.356+0000 7f168a17f740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x5605d64ba8c0 0x5605d64bc360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.356+0000 7f168a17f740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5605d64e6600 0x5605d64e6a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.356+0000 7f168a17f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605d64206b0 con 0x5605d64b6870 2024-03-06T07:27:20.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.356+0000 7f168a17f740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605d63c9a60 con 0x5605d64e6600 2024-03-06T07:27:20.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.356+0000 7f168a17f740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x5605d641c750 con 0x5605d64ba8c0 2024-03-06T07:27:20.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.357+0000 7f1685251640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5605d64b6870 0x5605d64b6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.357+0000 7f1685251640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5605d64b6870 0x5605d64b6c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41478/0 (socket says 172.21.15.83:41478) 2024-03-06T07:27:20.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.357+0000 7f1685251640 1 -- 172.21.15.83:0/3170144992 learned_addr learned my addr 172.21.15.83:0/3170144992 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:20.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.357+0000 7f1684a50640 1 --2- 172.21.15.83:0/3170144992 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x5605d64ba8c0 0x5605d64bc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.357+0000 7f1685a52640 1 --2- 172.21.15.83:0/3170144992 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5605d64e6600 0x5605d64e6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.357+0000 7f1685251640 1 -- 172.21.15.83:0/3170144992 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x5605d64ba8c0 msgr2=0x5605d64bc360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:20.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.357+0000 7f1685251640 1 --2- 172.21.15.83:0/3170144992 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x5605d64ba8c0 0x5605d64bc360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.357+0000 7f1685251640 1 -- 172.21.15.83:0/3170144992 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5605d64e6600 msgr2=0x5605d64e6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:20.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.357+0000 7f1685251640 1 --2- 172.21.15.83:0/3170144992 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5605d64e6600 0x5605d64e6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.357+0000 7f1685251640 1 -- 172.21.15.83:0/3170144992 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5605d63e7f30 con 0x5605d64b6870 2024-03-06T07:27:20.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.357+0000 7f1685251640 1 --2- 172.21.15.83:0/3170144992 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5605d64b6870 0x5605d64b6c50 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f167c0050a0 tx=0x7f167c007980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18f6ce6b82f908d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:20.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.357+0000 7f166affd640 1 -- 172.21.15.83:0/3170144992 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f167c05c880 con 0x5605d64b6870 2024-03-06T07:27:20.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.357+0000 7f166affd640 1 -- 172.21.15.83:0/3170144992 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f167c060e40 con 0x5605d64b6870 2024-03-06T07:27:20.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.357+0000 7f166affd640 1 -- 172.21.15.83:0/3170144992 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f167c070200 con 0x5605d64b6870 2024-03-06T07:27:20.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.358+0000 7f168a17f740 1 -- 172.21.15.83:0/3170144992 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5605d64b6870 msgr2=0x5605d64b6c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.358+0000 7f168a17f740 1 --2- 172.21.15.83:0/3170144992 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5605d64b6870 0x5605d64b6c50 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f167c0050a0 tx=0x7f167c007980 comp rx=0 tx=0).stop 2024-03-06T07:27:20.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.358+0000 7f168a17f740 1 -- 172.21.15.83:0/3170144992 shutdown_connections 2024-03-06T07:27:20.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.358+0000 7f168a17f740 1 --2- 172.21.15.83:0/3170144992 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5605d64e6600 0x5605d64e6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.358+0000 7f168a17f740 1 --2- 172.21.15.83:0/3170144992 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x5605d64ba8c0 0x5605d64bc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.358+0000 7f168a17f740 1 --2- 172.21.15.83:0/3170144992 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5605d64b6870 0x5605d64b6c50 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.358+0000 7f168a17f740 1 -- 172.21.15.83:0/3170144992 >> 172.21.15.83:0/3170144992 conn(0x5605d64b88f0 msgr2=0x5605d64ba1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:20.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.358+0000 7f168a17f740 1 -- 172.21.15.83:0/3170144992 shutdown_connections 2024-03-06T07:27:20.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.358+0000 7f168a17f740 1 -- 172.21.15.83:0/3170144992 wait complete. 2024-03-06T07:27:20.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.358+0000 7f168a17f740 1 -- start start 2024-03-06T07:27:20.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.358+0000 7f168a17f740 1 Processor -- start 2024-03-06T07:27:20.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.359+0000 7f168a17f740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5605d64e6600 0x5605d64cfd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.359+0000 7f168a17f740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x5605d64ff1b0 0x5605d64ff590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.359+0000 7f168a17f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5605d64e8310 0x5605d64e8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.359+0000 7f168a17f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605d64bad50 con 0x5605d64e8310 2024-03-06T07:27:20.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.359+0000 7f168a17f740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605d63dcea0 con 0x5605d64e6600 2024-03-06T07:27:20.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.359+0000 7f168a17f740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x5605d64bb030 con 0x5605d64ff1b0 2024-03-06T07:27:20.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.359+0000 7f168a17f740 10 client.0 _ll_register_callbacks cb 0x5605d63dc870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:27:20.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.359+0000 7f1685a52640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5605d64e8310 0x5605d64e8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.360+0000 7f1685a52640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5605d64e8310 0x5605d64e8790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41486/0 (socket says 172.21.15.83:41486) 2024-03-06T07:27:20.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.360+0000 7f1685a52640 1 -- 172.21.15.83:0/1709168873 learned_addr learned my addr 172.21.15.83:0/1709168873 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:20.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.360+0000 7f1685251640 1 --2- 172.21.15.83:0/1709168873 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5605d64e6600 0x5605d64cfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.360+0000 7f1684a50640 1 --2- 172.21.15.83:0/1709168873 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x5605d64ff1b0 0x5605d64ff590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.360+0000 7f1685a52640 1 -- 172.21.15.83:0/1709168873 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x5605d64ff1b0 msgr2=0x5605d64ff590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:20.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.360+0000 7f1685a52640 1 --2- 172.21.15.83:0/1709168873 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x5605d64ff1b0 0x5605d64ff590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.360+0000 7f1685a52640 1 -- 172.21.15.83:0/1709168873 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5605d64e6600 msgr2=0x5605d64cfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:20.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.360+0000 7f1685a52640 1 --2- 172.21.15.83:0/1709168873 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5605d64e6600 0x5605d64cfd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.360+0000 7f1685a52640 1 -- 172.21.15.83:0/1709168873 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5605d63dd270 con 0x5605d64e8310 2024-03-06T07:27:20.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.360+0000 7f1685a52640 1 --2- 172.21.15.83:0/1709168873 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5605d64e8310 0x5605d64e8790 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f167802c800 tx=0x7f1678004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=845d5b5d08bf5333 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:20.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.360+0000 7f165ffff640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-06T07:27:20.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.360+0000 7f165ffff640 1 -- 172.21.15.83:0/1709168873 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1678039070 con 0x5605d64e8310 2024-03-06T07:27:20.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.360+0000 7f165ffff640 1 -- 172.21.15.83:0/1709168873 <== mon.0 v2:172.21.15.83:3300/0 2 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1678034990 con 0x5605d64e8310 2024-03-06T07:27:20.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.360+0000 7f168a17f740 1 -- 172.21.15.83:0/1709168873 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f167c060e40 con 0x5605d64e8310 2024-03-06T07:27:20.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.360+0000 7f165ffff640 1 -- 172.21.15.83:0/1709168873 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1678034b90 con 0x5605d64e8310 2024-03-06T07:27:20.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.360+0000 7f168a17f740 10 client.8052 fetch_fsmap learned FSMap version 59 2024-03-06T07:27:20.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.360+0000 7f168a17f740 1 -- 172.21.15.83:0/1709168873 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=2+}) v3 -- 0x5605d61352d0 con 0x5605d64e8310 2024-03-06T07:27:20.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.360+0000 7f165ffff640 1 -- 172.21.15.83:0/1709168873 <== mon.0 v2:172.21.15.83:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f167c060e40 con 0x5605d64e8310 2024-03-06T07:27:20.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.361+0000 7f165ffff640 1 -- 172.21.15.83:0/1709168873 <== mon.0 v2:172.21.15.83:3300/0 5 ==== fsmap(e 59) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f167803f3f0 con 0x5605d64e8310 2024-03-06T07:27:20.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.361+0000 7f168a17f740 10 client.8052 fetch_fsmap finished waiting for FSMap version 59 2024-03-06T07:27:20.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.361+0000 7f168a17f740 10 client.8052 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:27:20.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.361+0000 7f168a17f740 20 client.8052 populate_metadata read hostname 'smithi083' 2024-03-06T07:27:20.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.361+0000 7f168a17f740 1 --2- 172.21.15.83:0/1709168873 >> [v2:172.21.15.83:6834/855387870,v1:172.21.15.83:6836/855387870] conn(0x5605d6501ff0 0x5605d65209d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.361+0000 7f168a17f740 4 client.8052 mds_command: new command op to 7887 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:27:20.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.361+0000 7f168a17f740 1 -- 172.21.15.83:0/1709168873 --> [v2:172.21.15.83:6834/855387870,v1:172.21.15.83:6836/855387870] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5605d642dbf0 con 0x5605d6501ff0 2024-03-06T07:27:20.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.361+0000 7f1685251640 1 --2- 172.21.15.83:0/1709168873 >> [v2:172.21.15.83:6834/855387870,v1:172.21.15.83:6836/855387870] conn(0x5605d6501ff0 0x5605d65209d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.361+0000 7f165ffff640 10 client.8052 ms_handle_connect on v2:172.21.15.83:6834/855387870 2024-03-06T07:27:20.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.361+0000 7f1685251640 1 --2- 172.21.15.83:0/1709168873 >> [v2:172.21.15.83:6834/855387870,v1:172.21.15.83:6836/855387870] conn(0x5605d6501ff0 0x5605d65209d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9adcd6c1c5bdc479 server_cookie=d29ccb2e9d4eef77 in_seq=0 out_seq=0 2024-03-06T07:27:20.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.362+0000 7f165ffff640 1 -- 172.21.15.83:0/1709168873 <== mds.0 v2:172.21.15.83:6834/855387870 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5605d642dbf0 con 0x5605d6501ff0 2024-03-06T07:27:20.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.362+0000 7f165ffff640 10 client.8052 handle_command_reply: tid=0 2024-03-06T07:27:20.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.362+0000 7f168a17f740 1 client.8052 shutdown 2024-03-06T07:27:20.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.362+0000 7f168a17f740 1 -- 172.21.15.83:0/1709168873 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5605d64e8310 msgr2=0x5605d64e8790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.362+0000 7f168a17f740 1 --2- 172.21.15.83:0/1709168873 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5605d64e8310 0x5605d64e8790 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f167802c800 tx=0x7f1678004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:20.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.362+0000 7f168a17f740 1 -- 172.21.15.83:0/1709168873 shutdown_connections 2024-03-06T07:27:20.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.362+0000 7f168a17f740 1 --2- 172.21.15.83:0/1709168873 >> [v2:172.21.15.83:6834/855387870,v1:172.21.15.83:6836/855387870] conn(0x5605d6501ff0 0x5605d65209d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.362+0000 7f165ffff640 0 client.8052 ms_handle_reset on v2:172.21.15.83:6834/855387870 2024-03-06T07:27:20.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.362+0000 7f168a17f740 1 --2- 172.21.15.83:0/1709168873 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5605d64e8310 0x5605d64e8790 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.362+0000 7f168a17f740 1 --2- 172.21.15.83:0/1709168873 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x5605d64ff1b0 0x5605d64ff590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.362+0000 7f168a17f740 1 --2- 172.21.15.83:0/1709168873 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5605d64e6600 0x5605d64cfd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.362+0000 7f168a17f740 1 -- 172.21.15.83:0/1709168873 >> 172.21.15.83:0/1709168873 conn(0x5605d64b6870 msgr2=0x5605d64e45c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:20.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.362+0000 7f168a17f740 1 -- 172.21.15.83:0/1709168873 shutdown_connections 2024-03-06T07:27:20.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.362+0000 7f168a17f740 1 -- 172.21.15.83:0/1709168873 wait complete. 2024-03-06T07:27:20.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.362+0000 7f168a17f740 20 client.8052 trim_cache size 0 max 16384 2024-03-06T07:27:20.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.379+0000 7f168a17f740 1 Processor -- start 2024-03-06T07:27:20.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.379+0000 7f168a17f740 1 -- start start 2024-03-06T07:27:20.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.380+0000 7f168a17f740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x5605d64e6600 0x5605d64e4600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.380+0000 7f168a17f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5605d64e8310 0x5605d64b5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.380+0000 7f168a17f740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5605d64ff1b0 0x5605d64b5a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.380+0000 7f168a17f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16801ff9c0 con 0x5605d64e8310 2024-03-06T07:27:20.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.380+0000 7f168a17f740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16800cb7b0 con 0x5605d64ff1b0 2024-03-06T07:27:20.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.380+0000 7f168a17f740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16800c8360 con 0x5605d64e6600 2024-03-06T07:27:20.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.380+0000 7f1684a50640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5605d64e8310 0x5605d64b5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.380+0000 7f1684a50640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5605d64e8310 0x5605d64b5510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41500/0 (socket says 172.21.15.83:41500) 2024-03-06T07:27:20.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.380+0000 7f1684a50640 1 -- 172.21.15.83:0/982852124 learned_addr learned my addr 172.21.15.83:0/982852124 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:20.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.380+0000 7f1685251640 1 --2- 172.21.15.83:0/982852124 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x5605d64e6600 0x5605d64e4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.380+0000 7f1685a52640 1 --2- 172.21.15.83:0/982852124 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5605d64ff1b0 0x5605d64b5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.380+0000 7f1684a50640 1 -- 172.21.15.83:0/982852124 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x5605d64e6600 msgr2=0x5605d64e4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:20.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.380+0000 7f1684a50640 1 --2- 172.21.15.83:0/982852124 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x5605d64e6600 0x5605d64e4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.380+0000 7f1684a50640 1 -- 172.21.15.83:0/982852124 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5605d64ff1b0 msgr2=0x5605d64b5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:20.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.380+0000 7f1684a50640 1 --2- 172.21.15.83:0/982852124 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5605d64ff1b0 0x5605d64b5a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.380+0000 7f1684a50640 1 -- 172.21.15.83:0/982852124 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5605d61ff6b0 con 0x5605d64e8310 2024-03-06T07:27:20.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.380+0000 7f1684a50640 1 --2- 172.21.15.83:0/982852124 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5605d64e8310 0x5605d64b5510 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f1670007e10 tx=0x7f1670013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee5d61f63022f9bf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:20.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.381+0000 7f166a7fc640 1 -- 172.21.15.83:0/982852124 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1670022600 con 0x5605d64e8310 2024-03-06T07:27:20.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.381+0000 7f166a7fc640 1 -- 172.21.15.83:0/982852124 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1670027070 con 0x5605d64e8310 2024-03-06T07:27:20.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.381+0000 7f166a7fc640 1 -- 172.21.15.83:0/982852124 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16700229a0 con 0x5605d64e8310 2024-03-06T07:27:20.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.381+0000 7f168a17f740 1 -- 172.21.15.83:0/982852124 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5605d64e8310 msgr2=0x5605d64b5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.381+0000 7f168a17f740 1 --2- 172.21.15.83:0/982852124 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5605d64e8310 0x5605d64b5510 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f1670007e10 tx=0x7f1670013000 comp rx=0 tx=0).stop 2024-03-06T07:27:20.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.381+0000 7f168a17f740 1 -- 172.21.15.83:0/982852124 shutdown_connections 2024-03-06T07:27:20.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.381+0000 7f168a17f740 1 --2- 172.21.15.83:0/982852124 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5605d64ff1b0 0x5605d64b5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.381+0000 7f168a17f740 1 --2- 172.21.15.83:0/982852124 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5605d64e8310 0x5605d64b5510 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.381+0000 7f168a17f740 1 --2- 172.21.15.83:0/982852124 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x5605d64e6600 0x5605d64e4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.381+0000 7f168a17f740 1 -- 172.21.15.83:0/982852124 >> 172.21.15.83:0/982852124 conn(0x5605d64b6870 msgr2=0x5605d64b1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:20.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.381+0000 7f168a17f740 1 -- 172.21.15.83:0/982852124 shutdown_connections 2024-03-06T07:27:20.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.381+0000 7f168a17f740 1 -- 172.21.15.83:0/982852124 wait complete. 2024-03-06T07:27:20.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.382+0000 7f168a17f740 1 -- start start 2024-03-06T07:27:20.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.382+0000 7f168a17f740 1 Processor -- start 2024-03-06T07:27:20.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.383+0000 7f168a17f740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x5605d64e6600 0x5605d64ce430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.383+0000 7f168a17f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5605d64e8310 0x5605d64ce970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.383+0000 7f168a17f740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5605d64ff1b0 0x5605d64e9d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.383+0000 7f168a17f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605d64fde60 con 0x5605d64e8310 2024-03-06T07:27:20.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.383+0000 7f168a17f740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605d64fd8a0 con 0x5605d64ff1b0 2024-03-06T07:27:20.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.383+0000 7f168a17f740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x5605d64fd030 con 0x5605d64e6600 2024-03-06T07:27:20.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.383+0000 7f168a17f740 10 client.0 _ll_register_callbacks cb 0x7f166c0043d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:27:20.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.383+0000 7f1684a50640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5605d64e8310 0x5605d64ce970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.383+0000 7f1684a50640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5605d64e8310 0x5605d64ce970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41502/0 (socket says 172.21.15.83:41502) 2024-03-06T07:27:20.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.383+0000 7f1684a50640 1 -- 172.21.15.83:0/625907347 learned_addr learned my addr 172.21.15.83:0/625907347 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:20.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.383+0000 7f1685a52640 1 --2- 172.21.15.83:0/625907347 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5605d64ff1b0 0x5605d64e9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.383+0000 7f1685251640 1 --2- 172.21.15.83:0/625907347 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x5605d64e6600 0x5605d64ce430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.383+0000 7f1684a50640 1 -- 172.21.15.83:0/625907347 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x5605d64e6600 msgr2=0x5605d64ce430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:20.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.383+0000 7f1684a50640 1 --2- 172.21.15.83:0/625907347 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x5605d64e6600 0x5605d64ce430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.383+0000 7f1684a50640 1 -- 172.21.15.83:0/625907347 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5605d64ff1b0 msgr2=0x5605d64e9d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:20.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.383+0000 7f1684a50640 1 --2- 172.21.15.83:0/625907347 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5605d64ff1b0 0x5605d64e9d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.383+0000 7f1684a50640 1 -- 172.21.15.83:0/625907347 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5605d64e9210 con 0x5605d64e8310 2024-03-06T07:27:20.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.383+0000 7f165ffff640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-06T07:27:20.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.383+0000 7f1684a50640 1 --2- 172.21.15.83:0/625907347 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5605d64e8310 0x5605d64ce970 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f1670015cc0 tx=0x7f16800b3c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=807ae8b3089d2c13 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:20.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.384+0000 7f165ffff640 1 -- 172.21.15.83:0/625907347 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1670035620 con 0x5605d64e8310 2024-03-06T07:27:20.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.384+0000 7f165ffff640 1 -- 172.21.15.83:0/625907347 <== mon.0 v2:172.21.15.83:3300/0 2 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f16700357c0 con 0x5605d64e8310 2024-03-06T07:27:20.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.384+0000 7f168a17f740 1 -- 172.21.15.83:0/625907347 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1670027070 con 0x5605d64e8310 2024-03-06T07:27:20.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.384+0000 7f165ffff640 1 -- 172.21.15.83:0/625907347 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) v2 ==== 24+0+0 (secure 0 0 0) 0x7f167003c050 con 0x5605d64e8310 2024-03-06T07:27:20.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.384+0000 7f168a17f740 10 client.8058 fetch_fsmap learned FSMap version 59 2024-03-06T07:27:20.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.384+0000 7f168a17f740 1 -- 172.21.15.83:0/625907347 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=2+}) v3 -- 0x5605d64e4df0 con 0x5605d64e8310 2024-03-06T07:27:20.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.384+0000 7f165ffff640 1 -- 172.21.15.83:0/625907347 <== mon.0 v2:172.21.15.83:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1670027070 con 0x5605d64e8310 2024-03-06T07:27:20.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.384+0000 7f165ffff640 1 -- 172.21.15.83:0/625907347 <== mon.0 v2:172.21.15.83:3300/0 5 ==== fsmap(e 59) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f167003f020 con 0x5605d64e8310 2024-03-06T07:27:20.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.384+0000 7f168a17f740 10 client.8058 fetch_fsmap finished waiting for FSMap version 59 2024-03-06T07:27:20.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.384+0000 7f168a17f740 10 client.8058 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:27:20.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.384+0000 7f168a17f740 20 client.8058 populate_metadata read hostname 'smithi083' 2024-03-06T07:27:20.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.384+0000 7f168a17f740 1 --2- 172.21.15.83:0/625907347 >> [v2:172.21.15.83:6834/855387870,v1:172.21.15.83:6836/855387870] conn(0x5605d6572160 0x5605d64bb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.384+0000 7f168a17f740 4 client.8058 mds_command: new command op to 7887 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:27:20.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.384+0000 7f168a17f740 1 -- 172.21.15.83:0/625907347 --> [v2:172.21.15.83:6834/855387870,v1:172.21.15.83:6836/855387870] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5605d6427f40 con 0x5605d6572160 2024-03-06T07:27:20.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.384+0000 7f1685251640 1 --2- 172.21.15.83:0/625907347 >> [v2:172.21.15.83:6834/855387870,v1:172.21.15.83:6836/855387870] conn(0x5605d6572160 0x5605d64bb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.384+0000 7f165ffff640 10 client.8058 ms_handle_connect on v2:172.21.15.83:6834/855387870 2024-03-06T07:27:20.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.384+0000 7f1685251640 1 --2- 172.21.15.83:0/625907347 >> [v2:172.21.15.83:6834/855387870,v1:172.21.15.83:6836/855387870] conn(0x5605d6572160 0x5605d64bb500 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ff5934ec760610c server_cookie=8bf19a8e95eff3c8 in_seq=0 out_seq=0 2024-03-06T07:27:20.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.385+0000 7f165ffff640 1 -- 172.21.15.83:0/625907347 <== mds.0 v2:172.21.15.83:6834/855387870 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x5605d6427f40 con 0x5605d6572160 2024-03-06T07:27:20.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.385+0000 7f165ffff640 10 client.8058 handle_command_reply: tid=0 2024-03-06T07:27:20.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.385+0000 7f168a17f740 1 client.8058 shutdown 2024-03-06T07:27:20.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.385+0000 7f168a17f740 1 -- 172.21.15.83:0/625907347 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5605d64e8310 msgr2=0x5605d64ce970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.385+0000 7f168a17f740 1 --2- 172.21.15.83:0/625907347 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5605d64e8310 0x5605d64ce970 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f1670015cc0 tx=0x7f16800b3c90 comp rx=0 tx=0).stop 2024-03-06T07:27:20.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.385+0000 7f168a17f740 1 -- 172.21.15.83:0/625907347 shutdown_connections 2024-03-06T07:27:20.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.385+0000 7f168a17f740 1 --2- 172.21.15.83:0/625907347 >> [v2:172.21.15.83:6834/855387870,v1:172.21.15.83:6836/855387870] conn(0x5605d6572160 0x5605d64bb500 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.385+0000 7f168a17f740 1 --2- 172.21.15.83:0/625907347 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5605d64ff1b0 0x5605d64e9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.385+0000 7f165ffff640 0 client.8058 ms_handle_reset on v2:172.21.15.83:6834/855387870 2024-03-06T07:27:20.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.385+0000 7f168a17f740 1 --2- 172.21.15.83:0/625907347 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5605d64e8310 0x5605d64ce970 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.385+0000 7f168a17f740 1 --2- 172.21.15.83:0/625907347 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x5605d64e6600 0x5605d64ce430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.385+0000 7f168a17f740 1 -- 172.21.15.83:0/625907347 >> 172.21.15.83:0/625907347 conn(0x5605d64b6870 msgr2=0x5605d64ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:20.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.385+0000 7f168a17f740 1 -- 172.21.15.83:0/625907347 shutdown_connections 2024-03-06T07:27:20.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.385+0000 7f168a17f740 1 -- 172.21.15.83:0/625907347 wait complete. 2024-03-06T07:27:20.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.385+0000 7f168a17f740 20 client.8058 trim_cache size 0 max 16384 2024-03-06T07:27:20.394 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:27:20.394 INFO:teuthology.orchestra.run.smithi083.stdout: "cluster_fsid": "039877dd-d2ce-45d2-8e77-ac5eb7e1d55a", 2024-03-06T07:27:20.394 INFO:teuthology.orchestra.run.smithi083.stdout: "whoami": 0, 2024-03-06T07:27:20.394 INFO:teuthology.orchestra.run.smithi083.stdout: "id": 7887, 2024-03-06T07:27:20.394 INFO:teuthology.orchestra.run.smithi083.stdout: "want_state": "up:active", 2024-03-06T07:27:20.394 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "up:active", 2024-03-06T07:27:20.394 INFO:teuthology.orchestra.run.smithi083.stdout: "fs_name": "cephfs", 2024-03-06T07:27:20.394 INFO:teuthology.orchestra.run.smithi083.stdout: "rank_uptime": 9.2473898979999998, 2024-03-06T07:27:20.394 INFO:teuthology.orchestra.run.smithi083.stdout: "mdsmap_epoch": 59, 2024-03-06T07:27:20.395 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_epoch": 110, 2024-03-06T07:27:20.395 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_epoch_barrier": 110, 2024-03-06T07:27:20.395 INFO:teuthology.orchestra.run.smithi083.stdout: "uptime": 14.315238011 2024-03-06T07:27:20.395 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:27:20.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.386+0000 7f16874b5640 1 -- 172.21.15.83:0/2065669377 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f166406b4d0 msgr2=0x7f166406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.386+0000 7f16874b5640 1 --2- 172.21.15.83:0/2065669377 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f166406b4d0 0x7f166406d990 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f16800c7560 tx=0x7f1670009040 comp rx=0 tx=0).stop 2024-03-06T07:27:20.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.386+0000 7f16874b5640 1 -- 172.21.15.83:0/2065669377 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f166c0051c0 msgr2=0x7f16800b7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.386+0000 7f16874b5640 1 --2- 172.21.15.83:0/2065669377 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f166c0051c0 0x7f16800b7d40 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f167c00f620 tx=0x7f167c00fae0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.386+0000 7f16874b5640 1 -- 172.21.15.83:0/2065669377 shutdown_connections 2024-03-06T07:27:20.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.386+0000 7f16874b5640 1 --2- 172.21.15.83:0/2065669377 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f166406b4d0 0x7f166406d990 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.386+0000 7f16874b5640 1 --2- 172.21.15.83:0/2065669377 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16800ca3f0 0x7f16800d03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.386+0000 7f16874b5640 1 --2- 172.21.15.83:0/2065669377 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16801560f0 0x7f16800c9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.386+0000 7f16874b5640 1 --2- 172.21.15.83:0/2065669377 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f166c0051c0 0x7f16800b7d40 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.386+0000 7f16874b5640 1 -- 172.21.15.83:0/2065669377 >> 172.21.15.83:0/2065669377 conn(0x7f16800d3ce0 msgr2=0x7f168015fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:20.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.386+0000 7f16874b5640 1 -- 172.21.15.83:0/2065669377 shutdown_connections 2024-03-06T07:27:20.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.386+0000 7f16874b5640 1 -- 172.21.15.83:0/2065669377 wait complete. 2024-03-06T07:27:20.398 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:27:20.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.498+0000 7ff9dddbc640 1 Processor -- start 2024-03-06T07:27:20.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.498+0000 7ff9dddbc640 1 -- start start 2024-03-06T07:27:20.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.498+0000 7ff9dddbc640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ff9d80d22b0 0x7ff9d80d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.498+0000 7ff9dddbc640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9d80b41f0 0x7ff9d80b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.498+0000 7ff9dddbc640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9d80b3430 con 0x7ff9d815dfe0 2024-03-06T07:27:20.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.498+0000 7ff9dddbc640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9d80d2c50 con 0x7ff9d80d22b0 2024-03-06T07:27:20.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.498+0000 7ff9dddbc640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d80b31e0 con 0x7ff9d80b41f0 2024-03-06T07:27:20.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.499+0000 7ff9d7fff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9d80b41f0 0x7ff9d80b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.499+0000 7ff9d6ffd640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ff9d80d22b0 0x7ff9d80d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.499+0000 7ff9d77fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7ff9d815dfe0 0x7ff9d80cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:45944/0 (socket says 172.21.15.83:45944) 2024-03-06T07:27:20.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.499+0000 7ff9d77fe640 1 -- 172.21.15.83:0/786116897 learned_addr learned my addr 172.21.15.83:0/786116897 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:27:20.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.499+0000 7ff9d6ffd640 1 -- 172.21.15.83:0/786116897 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9d80b41f0 msgr2=0x7ff9d80b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:20.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.499+0000 7ff9d6ffd640 1 --2- 172.21.15.83:0/786116897 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9d80b41f0 0x7ff9d80b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.499+0000 7ff9d6ffd640 1 -- 172.21.15.83:0/786116897 >> v1:172.21.15.183:6789/0 conn(0x7ff9d815dfe0 legacy=0x7ff9d80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.499+0000 7ff9d6ffd640 1 -- 172.21.15.83:0/786116897 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d80b3710 con 0x7ff9d80d22b0 2024-03-06T07:27:20.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.500+0000 7ff9d6ffd640 1 --2- 172.21.15.83:0/786116897 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff9d80d22b0 0x7ff9d80d2710 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc0097a0 tx=0x7ff9cc02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3237f9c92903d9c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:20.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.500+0000 7ff9d67fc640 1 -- 172.21.15.83:0/786116897 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9cc035020 con 0x7ff9d80d22b0 2024-03-06T07:27:20.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.500+0000 7ff9d67fc640 1 -- 172.21.15.83:0/786116897 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9cc031830 con 0x7ff9d80d22b0 2024-03-06T07:27:20.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.500+0000 7ff9d67fc640 1 -- 172.21.15.83:0/786116897 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9cc031c00 con 0x7ff9d80d22b0 2024-03-06T07:27:20.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.501+0000 7ff9dddbc640 1 -- 172.21.15.83:0/786116897 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff9d80d22b0 msgr2=0x7ff9d80d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.501+0000 7ff9dddbc640 1 --2- 172.21.15.83:0/786116897 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff9d80d22b0 0x7ff9d80d2710 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc0097a0 tx=0x7ff9cc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.501+0000 7ff9dddbc640 1 -- 172.21.15.83:0/786116897 shutdown_connections 2024-03-06T07:27:20.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.501+0000 7ff9dddbc640 1 --2- 172.21.15.83:0/786116897 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9d80b41f0 0x7ff9d80b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.501+0000 7ff9dddbc640 1 --2- 172.21.15.83:0/786116897 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff9d80d22b0 0x7ff9d80d2710 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.501+0000 7ff9dddbc640 1 -- 172.21.15.83:0/786116897 >> 172.21.15.83:0/786116897 conn(0x7ff9d80d3cd0 msgr2=0x7ff9d80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:20.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.501+0000 7ff9dddbc640 1 -- 172.21.15.83:0/786116897 shutdown_connections 2024-03-06T07:27:20.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.501+0000 7ff9dddbc640 1 -- 172.21.15.83:0/786116897 wait complete. 2024-03-06T07:27:20.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.503+0000 7ff9dddbc640 1 Processor -- start 2024-03-06T07:27:20.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.503+0000 7ff9dddbc640 1 -- start start 2024-03-06T07:27:20.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.504+0000 7ff9dddbc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9d80b41f0 0x7ff9d80b6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.504+0000 7ff9dddbc640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff9d80d22b0 0x7ff9d81f3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.504+0000 7ff9dddbc640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9d815dfe0 0x7ff9d81f3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.504+0000 7ff9dddbc640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d80cfa00 con 0x7ff9d80b41f0 2024-03-06T07:27:20.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.504+0000 7ff9dddbc640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d80b3d60 con 0x7ff9d80d22b0 2024-03-06T07:27:20.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.504+0000 7ff9dddbc640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d80d1c70 con 0x7ff9d815dfe0 2024-03-06T07:27:20.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.504+0000 7ff9d77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9d80b41f0 0x7ff9d80b6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.504+0000 7ff9d77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9d80b41f0 0x7ff9d80b6910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41506/0 (socket says 172.21.15.83:41506) 2024-03-06T07:27:20.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.504+0000 7ff9d77fe640 1 -- 172.21.15.83:0/3665782110 learned_addr learned my addr 172.21.15.83:0/3665782110 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:20.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.504+0000 7ff9d6ffd640 1 --2- 172.21.15.83:0/3665782110 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff9d80d22b0 0x7ff9d81f3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.504+0000 7ff9d7fff640 1 --2- 172.21.15.83:0/3665782110 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9d815dfe0 0x7ff9d81f3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.504+0000 7ff9d77fe640 1 -- 172.21.15.83:0/3665782110 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9d815dfe0 msgr2=0x7ff9d81f3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.504+0000 7ff9d77fe640 1 --2- 172.21.15.83:0/3665782110 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9d815dfe0 0x7ff9d81f3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.504+0000 7ff9d77fe640 1 -- 172.21.15.83:0/3665782110 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff9d80d22b0 msgr2=0x7ff9d81f3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.505+0000 7ff9d77fe640 1 --2- 172.21.15.83:0/3665782110 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff9d80d22b0 0x7ff9d81f3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.505+0000 7ff9d77fe640 1 -- 172.21.15.83:0/3665782110 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d80b9490 con 0x7ff9d80b41f0 2024-03-06T07:27:20.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.505+0000 7ff9d77fe640 1 --2- 172.21.15.83:0/3665782110 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9d80b41f0 0x7ff9d80b6910 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7ff9c800c3f0 tx=0x7ff9c800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:20.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.505+0000 7ff9bffff640 1 -- 172.21.15.83:0/3665782110 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c8017070 con 0x7ff9d80b41f0 2024-03-06T07:27:20.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.505+0000 7ff9bffff640 1 -- 172.21.15.83:0/3665782110 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9c8005ce0 con 0x7ff9d80b41f0 2024-03-06T07:27:20.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.505+0000 7ff9dddbc640 1 -- 172.21.15.83:0/3665782110 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9d80b9770 con 0x7ff9d80b41f0 2024-03-06T07:27:20.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.505+0000 7ff9bffff640 1 -- 172.21.15.83:0/3665782110 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c80072f0 con 0x7ff9d80b41f0 2024-03-06T07:27:20.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.505+0000 7ff9dddbc640 1 -- 172.21.15.83:0/3665782110 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9d80b1130 con 0x7ff9d80b41f0 2024-03-06T07:27:20.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.506+0000 7ff9dddbc640 1 -- 172.21.15.83:0/3665782110 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9d804f330 con 0x7ff9d80b41f0 2024-03-06T07:27:20.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.507+0000 7ff9bffff640 1 -- 172.21.15.83:0/3665782110 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff9c801a070 con 0x7ff9d80b41f0 2024-03-06T07:27:20.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.507+0000 7ff9bffff640 1 --2- 172.21.15.83:0/3665782110 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff9c406b4d0 0x7ff9c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.507+0000 7ff9bffff640 1 -- 172.21.15.83:0/3665782110 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff9c8089460 con 0x7ff9d80b41f0 2024-03-06T07:27:20.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.507+0000 7ff9d6ffd640 1 --2- 172.21.15.83:0/3665782110 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff9c406b4d0 0x7ff9c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.507+0000 7ff9d6ffd640 1 --2- 172.21.15.83:0/3665782110 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff9c406b4d0 0x7ff9c406d990 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc036000 tx=0x7ff9cc008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:20.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.511+0000 7ff9bffff640 1 -- 172.21.15.83:0/3665782110 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff9c8057d20 con 0x7ff9d80b41f0 2024-03-06T07:27:20.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.725+0000 7ff9dddbc640 1 -- 172.21.15.83:0/3665782110 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff9d80b2f40 con 0x7ff9d80b41f0 2024-03-06T07:27:20.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.725+0000 7ff9bffff640 1 -- 172.21.15.83:0/3665782110 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v59) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff9c805bdc0 con 0x7ff9d80b41f0 2024-03-06T07:27:20.727 INFO:teuthology.orchestra.run.smithi083.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:27:20.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.727+0000 7ff9dddbc640 1 -- 172.21.15.83:0/3665782110 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff9c406b4d0 msgr2=0x7ff9c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.727+0000 7ff9dddbc640 1 --2- 172.21.15.83:0/3665782110 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff9c406b4d0 0x7ff9c406d990 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc036000 tx=0x7ff9cc008030 comp rx=0 tx=0).stop 2024-03-06T07:27:20.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.727+0000 7ff9dddbc640 1 -- 172.21.15.83:0/3665782110 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9d80b41f0 msgr2=0x7ff9d80b6910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.727+0000 7ff9dddbc640 1 --2- 172.21.15.83:0/3665782110 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9d80b41f0 0x7ff9d80b6910 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7ff9c800c3f0 tx=0x7ff9c800c8b0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.727+0000 7ff9dddbc640 1 -- 172.21.15.83:0/3665782110 shutdown_connections 2024-03-06T07:27:20.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.727+0000 7ff9dddbc640 1 --2- 172.21.15.83:0/3665782110 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff9c406b4d0 0x7ff9c406d990 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.727+0000 7ff9dddbc640 1 --2- 172.21.15.83:0/3665782110 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9d815dfe0 0x7ff9d81f3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.727+0000 7ff9dddbc640 1 --2- 172.21.15.83:0/3665782110 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff9d80d22b0 0x7ff9d81f3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.727+0000 7ff9dddbc640 1 --2- 172.21.15.83:0/3665782110 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9d80b41f0 0x7ff9d80b6910 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.727+0000 7ff9dddbc640 1 -- 172.21.15.83:0/3665782110 >> 172.21.15.83:0/3665782110 conn(0x7ff9d80d3cd0 msgr2=0x7ff9d806cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:20.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.727+0000 7ff9dddbc640 1 -- 172.21.15.83:0/3665782110 shutdown_connections 2024-03-06T07:27:20.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:20.728+0000 7ff9dddbc640 1 -- 172.21.15.83:0/3665782110 wait complete. 2024-03-06T07:27:20.741 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:27:20.741 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:27:20.741 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi083.front.sepia.ceph.com 2024-03-06T07:27:20.742 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:27:20.742 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:20.742 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:27:20.742 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:27:20.742 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:27:20.742 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:27:20.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:20.743 DEBUG:teuthology.orchestra.run.smithi083:> ip addr 2024-03-06T07:27:20.759 INFO:teuthology.orchestra.run.smithi083.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:27:20.759 INFO:teuthology.orchestra.run.smithi083.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:27:20.760 INFO:teuthology.orchestra.run.smithi083.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:27:20.760 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:20.760 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 ::1/128 scope host 2024-03-06T07:27:20.760 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:20.760 INFO:teuthology.orchestra.run.smithi083.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:20.760 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:d8:da:1e brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:20.760 INFO:teuthology.orchestra.run.smithi083.stdout: altname enp7s0f0 2024-03-06T07:27:20.760 INFO:teuthology.orchestra.run.smithi083.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:20.760 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:d8:da:1f brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:20.761 INFO:teuthology.orchestra.run.smithi083.stdout: altname enp7s0f1 2024-03-06T07:27:20.761 INFO:teuthology.orchestra.run.smithi083.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:20.761 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:8f:ce:90 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:20.761 INFO:teuthology.orchestra.run.smithi083.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:27:20.761 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:8f:ce:91 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:20.761 INFO:teuthology.orchestra.run.smithi083.stdout: inet 172.21.15.83/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:27:20.761 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft 42415sec preferred_lft 42415sec 2024-03-06T07:27:20.761 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::ec4:7aff:fe8f:ce91/64 scope link 2024-03-06T07:27:20.761 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:20.761 INFO:teuthology.orchestra.run.smithi083.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:27:20.761 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 1e:29:46:c9:e7:c8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:20.761 INFO:teuthology.orchestra.run.smithi083.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:27:20.761 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:20.761 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::60c5:d7ff:fe70:2b1c/64 scope link 2024-03-06T07:27:20.762 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:20.762 INFO:teuthology.orchestra.run.smithi083.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:27:20.762 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 1e:29:46:c9:e7:c8 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:20.762 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::1c29:46ff:fec9:e7c8/64 scope link 2024-03-06T07:27:20.762 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:20.762 INFO:teuthology.orchestra.run.smithi083.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:27:20.762 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 3e:00:2f:23:d5:45 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:20.762 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::3c00:2fff:fe23:d545/64 scope link 2024-03-06T07:27:20.762 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:20.762 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:20.763 DEBUG:teuthology.orchestra.run.smithi083:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:27:20.825 INFO:teuthology.orchestra.run.smithi083.stdout:1 2024-03-06T07:27:20.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:20.827 DEBUG:teuthology.orchestra.run.smithi083:> route 2024-03-06T07:27:20.882 INFO:teuthology.orchestra.run.smithi083.stdout:Kernel IP routing table 2024-03-06T07:27:20.882 INFO:teuthology.orchestra.run.smithi083.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:27:20.882 INFO:teuthology.orchestra.run.smithi083.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:27:20.883 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:27:20.883 INFO:teuthology.orchestra.run.smithi083.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:27:20.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:20.883 DEBUG:teuthology.orchestra.run.smithi083:> (cd / && exec bash -c ' 2024-03-06T07:27:20.883 DEBUG:teuthology.orchestra.run.smithi083:> set -e 2024-03-06T07:27:20.883 DEBUG:teuthology.orchestra.run.smithi083:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:27:20.883 DEBUG:teuthology.orchestra.run.smithi083:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:27:20.883 DEBUG:teuthology.orchestra.run.smithi083:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:27:20.883 DEBUG:teuthology.orchestra.run.smithi083:> ') 2024-03-06T07:27:20.977 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:20.977 DEBUG:teuthology.orchestra.run.smithi083:> ip netns list 2024-03-06T07:27:20.991 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:27:20.991 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:27:20.991 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:27:20.992 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:20.992 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip link set brx.0 up 2024-03-06T07:27:21.058 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:27:21.058 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:27:21.059 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:21.113 INFO:teuthology.orchestra.run.smithi083.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:27:21.113 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:27:21.113 DEBUG:teuthology.orchestra.run.smithi083:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:21.168 DEBUG:teuthology.orchestra.run.smithi083:> sudo modprobe fuse 2024-03-06T07:27:21.235 DEBUG:teuthology.orchestra.run.smithi083:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:27:21.291 INFO:teuthology.orchestra.run.smithi083.stdout:/proc 2024-03-06T07:27:21.291 INFO:teuthology.orchestra.run.smithi083.stdout:/sys 2024-03-06T07:27:21.291 INFO:teuthology.orchestra.run.smithi083.stdout:/dev 2024-03-06T07:27:21.291 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/security 2024-03-06T07:27:21.291 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/shm 2024-03-06T07:27:21.291 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/pts 2024-03-06T07:27:21.291 INFO:teuthology.orchestra.run.smithi083.stdout:/run 2024-03-06T07:27:21.291 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/cgroup 2024-03-06T07:27:21.292 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/pstore 2024-03-06T07:27:21.292 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/bpf 2024-03-06T07:27:21.292 INFO:teuthology.orchestra.run.smithi083.stdout:/ 2024-03-06T07:27:21.292 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/selinux 2024-03-06T07:27:21.292 INFO:teuthology.orchestra.run.smithi083.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:27:21.292 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/mqueue 2024-03-06T07:27:21.292 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/hugepages 2024-03-06T07:27:21.292 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/debug 2024-03-06T07:27:21.292 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/tracing 2024-03-06T07:27:21.292 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:27:21.292 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/fuse/connections 2024-03-06T07:27:21.292 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/config 2024-03-06T07:27:21.292 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:27:21.292 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:27:21.293 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:27:21.293 INFO:teuthology.orchestra.run.smithi083.stdout:/run/user/1000 2024-03-06T07:27:21.293 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph 2024-03-06T07:27:21.293 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:27:21.293 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:27:21.293 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:27:21.293 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:27:21.293 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns 2024-03-06T07:27:21.293 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:21.293 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:21.293 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:21.293 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:21.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:21.294 DEBUG:teuthology.orchestra.run.smithi083:> ls /sys/fs/fuse/connections 2024-03-06T07:27:21.347 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:27:21.347 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:27:21.388 DEBUG:teuthology.orchestra.run.smithi083:> sudo modprobe fuse 2024-03-06T07:27:21.415 DEBUG:teuthology.orchestra.run.smithi083:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:27:21.470 INFO:teuthology.orchestra.run.smithi083.stdout:/proc 2024-03-06T07:27:21.470 INFO:teuthology.orchestra.run.smithi083.stdout:/sys 2024-03-06T07:27:21.470 INFO:teuthology.orchestra.run.smithi083.stdout:/dev 2024-03-06T07:27:21.470 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/security 2024-03-06T07:27:21.470 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/shm 2024-03-06T07:27:21.470 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/pts 2024-03-06T07:27:21.471 INFO:teuthology.orchestra.run.smithi083.stdout:/run 2024-03-06T07:27:21.471 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/cgroup 2024-03-06T07:27:21.471 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/pstore 2024-03-06T07:27:21.471 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/bpf 2024-03-06T07:27:21.471 INFO:teuthology.orchestra.run.smithi083.stdout:/ 2024-03-06T07:27:21.471 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/selinux 2024-03-06T07:27:21.471 INFO:teuthology.orchestra.run.smithi083.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:27:21.471 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/mqueue 2024-03-06T07:27:21.471 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/hugepages 2024-03-06T07:27:21.471 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/debug 2024-03-06T07:27:21.471 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/tracing 2024-03-06T07:27:21.471 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:27:21.471 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/fuse/connections 2024-03-06T07:27:21.471 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/config 2024-03-06T07:27:21.471 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:27:21.472 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:27:21.472 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:27:21.472 INFO:teuthology.orchestra.run.smithi083.stdout:/run/user/1000 2024-03-06T07:27:21.472 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph 2024-03-06T07:27:21.472 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:27:21.472 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:27:21.472 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:27:21.472 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:27:21.472 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns 2024-03-06T07:27:21.472 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:21.472 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:21.472 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:21.472 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:21.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:21.473 DEBUG:teuthology.orchestra.run.smithi083:> ls /sys/fs/fuse/connections 2024-03-06T07:27:21.477 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:2024-03-06T07:27:21.475+0000 7f658ae4c480 -1 init, newargv = 0x55c04fde1680 newargc=15 2024-03-06T07:27:21.477 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:ceph-fuse[73136]: starting ceph client 2024-03-06T07:27:21.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:ceph-fuse[73136]: starting fuse 2024-03-06T07:27:21.492 INFO:teuthology.orchestra.run.smithi083.stdout:41 2024-03-06T07:27:21.493 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-06T07:27:21.493 DEBUG:teuthology.orchestra.run.smithi083:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:27:21.493 DEBUG:teuthology.orchestra.run.smithi083:> import glob 2024-03-06T07:27:21.493 DEBUG:teuthology.orchestra.run.smithi083:> import re 2024-03-06T07:27:21.493 DEBUG:teuthology.orchestra.run.smithi083:> import os 2024-03-06T07:27:21.493 DEBUG:teuthology.orchestra.run.smithi083:> import subprocess 2024-03-06T07:27:21.493 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:27:21.493 DEBUG:teuthology.orchestra.run.smithi083:> def _find_admin_socket(client_name): 2024-03-06T07:27:21.493 DEBUG:teuthology.orchestra.run.smithi083:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:27:21.493 DEBUG:teuthology.orchestra.run.smithi083:> files = glob.glob(asok_path) 2024-03-06T07:27:21.493 DEBUG:teuthology.orchestra.run.smithi083:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:27:21.493 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:27:21.493 DEBUG:teuthology.orchestra.run.smithi083:> # Given a non-glob path, it better be there 2024-03-06T07:27:21.493 DEBUG:teuthology.orchestra.run.smithi083:> if "*" not in asok_path: 2024-03-06T07:27:21.493 DEBUG:teuthology.orchestra.run.smithi083:> assert(len(files) == 1) 2024-03-06T07:27:21.494 DEBUG:teuthology.orchestra.run.smithi083:> return files[0] 2024-03-06T07:27:21.494 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:27:21.494 DEBUG:teuthology.orchestra.run.smithi083:> for f in files: 2024-03-06T07:27:21.494 DEBUG:teuthology.orchestra.run.smithi083:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:27:21.494 DEBUG:teuthology.orchestra.run.smithi083:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:27:21.494 DEBUG:teuthology.orchestra.run.smithi083:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:27:21.494 DEBUG:teuthology.orchestra.run.smithi083:> contents = proc_f.read() 2024-03-06T07:27:21.494 DEBUG:teuthology.orchestra.run.smithi083:> if mountpoint in contents: 2024-03-06T07:27:21.494 DEBUG:teuthology.orchestra.run.smithi083:> return f 2024-03-06T07:27:21.494 DEBUG:teuthology.orchestra.run.smithi083:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:27:21.494 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:27:21.494 DEBUG:teuthology.orchestra.run.smithi083:> print(_find_admin_socket("client.0")) 2024-03-06T07:27:21.494 DEBUG:teuthology.orchestra.run.smithi083:> ' 2024-03-06T07:27:21.612 INFO:teuthology.orchestra.run.smithi083.stdout:/var/run/ceph/ceph-client.0.73136.asok 2024-03-06T07:27:21.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:27:21.622 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.73136.asok 2024-03-06T07:27:21.622 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:21.622 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.73136.asok status 2024-03-06T07:27:21.759 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:27:21.759 INFO:teuthology.orchestra.run.smithi083.stdout: "metadata": { 2024-03-06T07:27:21.759 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:27:21.759 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:27:21.760 INFO:teuthology.orchestra.run.smithi083.stdout: "entity_id": "0", 2024-03-06T07:27:21.760 INFO:teuthology.orchestra.run.smithi083.stdout: "hostname": "smithi083", 2024-03-06T07:27:21.760 INFO:teuthology.orchestra.run.smithi083.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:27:21.760 INFO:teuthology.orchestra.run.smithi083.stdout: "pid": "73136", 2024-03-06T07:27:21.760 INFO:teuthology.orchestra.run.smithi083.stdout: "root": "/" 2024-03-06T07:27:21.760 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:27:21.760 INFO:teuthology.orchestra.run.smithi083.stdout: "dentry_count": 0, 2024-03-06T07:27:21.760 INFO:teuthology.orchestra.run.smithi083.stdout: "dentry_pinned_count": 0, 2024-03-06T07:27:21.760 INFO:teuthology.orchestra.run.smithi083.stdout: "id": 8067, 2024-03-06T07:27:21.760 INFO:teuthology.orchestra.run.smithi083.stdout: "inst": { 2024-03-06T07:27:21.760 INFO:teuthology.orchestra.run.smithi083.stdout: "name": { 2024-03-06T07:27:21.760 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "client", 2024-03-06T07:27:21.760 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 8067 2024-03-06T07:27:21.760 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:27:21.761 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": { 2024-03-06T07:27:21.761 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-03-06T07:27:21.761 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:27:21.761 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 1574564363 2024-03-06T07:27:21.761 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:27:21.761 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:27:21.761 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": { 2024-03-06T07:27:21.761 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-03-06T07:27:21.761 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:27:21.761 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 1574564363 2024-03-06T07:27:21.761 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:27:21.761 INFO:teuthology.orchestra.run.smithi083.stdout: "inst_str": "client.8067 192.168.0.1:0/1574564363", 2024-03-06T07:27:21.761 INFO:teuthology.orchestra.run.smithi083.stdout: "addr_str": "192.168.0.1:0/1574564363", 2024-03-06T07:27:21.762 INFO:teuthology.orchestra.run.smithi083.stdout: "inode_count": 1, 2024-03-06T07:27:21.762 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_epoch": 59, 2024-03-06T07:27:21.762 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_epoch": 110, 2024-03-06T07:27:21.762 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:27:21.762 INFO:teuthology.orchestra.run.smithi083.stdout: "blocklisted": false, 2024-03-06T07:27:21.762 INFO:teuthology.orchestra.run.smithi083.stdout: "fs_name": "cephfs" 2024-03-06T07:27:21.762 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:27:21.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:21.768 DEBUG:teuthology.orchestra.run.smithi083:> stat --file-system '--printf=%T 2024-03-06T07:27:21.768 DEBUG:teuthology.orchestra.run.smithi083:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:21.784 INFO:teuthology.orchestra.run.smithi083.stdout:fuseblk 2024-03-06T07:27:21.784 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:21.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:21.784 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:21.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:21.852 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir_1) 2024-03-06T07:27:21.912 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:27:21.912 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:21.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:21.968 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=dir_1/file_a bs=1M conv=fdatasync count=8 seek=0) 2024-03-06T07:27:22.085 INFO:teuthology.orchestra.run.smithi083.stderr:8+0 records in 2024-03-06T07:27:22.085 INFO:teuthology.orchestra.run.smithi083.stderr:8+0 records out 2024-03-06T07:27:22.085 INFO:teuthology.orchestra.run.smithi083.stderr:8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0610604 s, 137 MB/s 2024-03-06T07:27:22.086 DEBUG:teuthology.orchestra.run.smithi083:> stdin-killer -- python3 -c ' 2024-03-06T07:27:22.086 DEBUG:teuthology.orchestra.run.smithi083:> import os 2024-03-06T07:27:22.086 DEBUG:teuthology.orchestra.run.smithi083:> import stat 2024-03-06T07:27:22.087 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:27:22.087 DEBUG:teuthology.orchestra.run.smithi083:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir_1/file_a").st_ino) 2024-03-06T07:27:22.087 DEBUG:teuthology.orchestra.run.smithi083:> ' 2024-03-06T07:27:22.152 INFO:teuthology.orchestra.run.smithi083.stdout:1099511627777 2024-03-06T07:27:22.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:27:22.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:22.161 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir_2) 2024-03-06T07:27:22.217 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:22.217 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln dir_1/file_a dir_2/file_b) 2024-03-06T07:27:22.277 DEBUG:teuthology.orchestra.run.smithi083:> stdin-killer -- python3 -c ' 2024-03-06T07:27:22.278 DEBUG:teuthology.orchestra.run.smithi083:> import os 2024-03-06T07:27:22.278 DEBUG:teuthology.orchestra.run.smithi083:> import stat 2024-03-06T07:27:22.278 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:27:22.278 DEBUG:teuthology.orchestra.run.smithi083:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir_2/file_b").st_ino) 2024-03-06T07:27:22.278 DEBUG:teuthology.orchestra.run.smithi083:> ' 2024-03-06T07:27:22.382 INFO:teuthology.orchestra.run.smithi083.stdout:1099511627777 2024-03-06T07:27:22.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:27:22.391 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:22.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.530+0000 7f6bdb365640 1 Processor -- start 2024-03-06T07:27:22.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.531+0000 7f6bdb365640 1 -- start start 2024-03-06T07:27:22.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.531+0000 7f6bdb365640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6bd40d2710 0x7f6bd40b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.531+0000 7f6bdb365640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd40b3430 con 0x7f6bd415dfe0 2024-03-06T07:27:22.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.531+0000 7f6bdb365640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd40b3710 con 0x7f6bd40d22b0 2024-03-06T07:27:22.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.531+0000 7f6bdb365640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bd40b31e0 con 0x7f6bd40d2710 2024-03-06T07:27:22.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.531+0000 7f6bd9902640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6bd40d2710 0x7f6bd40b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.531+0000 7f6bd9101640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f6bd415dfe0 0x7f6bd40cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57102/0 (socket says 172.21.15.83:57102) 2024-03-06T07:27:22.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.531+0000 7f6bd9101640 1 -- 172.21.15.83:0/3258468738 learned_addr learned my addr 172.21.15.83:0/3258468738 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:27:22.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.532+0000 7f6bcbfff640 1 -- 172.21.15.83:0/3258468738 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2316508645 0 0) 0x7f6bd40b3430 con 0x7f6bd415dfe0 2024-03-06T07:27:22.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.532+0000 7f6bcbfff640 1 -- 172.21.15.83:0/3258468738 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bc0003520 con 0x7f6bd415dfe0 2024-03-06T07:27:22.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.532+0000 7f6bd9902640 1 -- 172.21.15.83:0/3258468738 >> v1:172.21.15.183:6789/0 conn(0x7f6bd40d22b0 legacy=0x7f6bd40b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.532+0000 7f6bd9902640 1 -- 172.21.15.83:0/3258468738 >> v1:172.21.15.83:6789/0 conn(0x7f6bd415dfe0 legacy=0x7f6bd40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.532+0000 7f6bd9902640 1 -- 172.21.15.83:0/3258468738 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd40b39f0 con 0x7f6bd40d2710 2024-03-06T07:27:22.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.533+0000 7f6bd9902640 1 --2- 172.21.15.83:0/3258468738 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6bd40d2710 0x7f6bd40b4900 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f6bd0009dc0 tx=0x7f6bd002bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6cf1276ce51041b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:22.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.533+0000 7f6bcbfff640 1 -- 172.21.15.83:0/3258468738 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd0039070 con 0x7f6bd40d2710 2024-03-06T07:27:22.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.533+0000 7f6bcbfff640 1 -- 172.21.15.83:0/3258468738 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd002ccd0 con 0x7f6bd40d2710 2024-03-06T07:27:22.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.533+0000 7f6bcbfff640 1 -- 172.21.15.83:0/3258468738 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd0034660 con 0x7f6bd40d2710 2024-03-06T07:27:22.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.533+0000 7f6bdb365640 1 -- 172.21.15.83:0/3258468738 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6bd40d2710 msgr2=0x7f6bd40b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.533+0000 7f6bdb365640 1 --2- 172.21.15.83:0/3258468738 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6bd40d2710 0x7f6bd40b4900 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f6bd0009dc0 tx=0x7f6bd002bf20 comp rx=0 tx=0).stop 2024-03-06T07:27:22.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.534+0000 7f6bdb365640 1 -- 172.21.15.83:0/3258468738 shutdown_connections 2024-03-06T07:27:22.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.534+0000 7f6bdb365640 1 --2- 172.21.15.83:0/3258468738 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6bd40d2710 0x7f6bd40b4900 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.534+0000 7f6bdb365640 1 -- 172.21.15.83:0/3258468738 >> 172.21.15.83:0/3258468738 conn(0x7f6bd40d3cd0 msgr2=0x7f6bd40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:22.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.534+0000 7f6bdb365640 1 -- 172.21.15.83:0/3258468738 shutdown_connections 2024-03-06T07:27:22.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.534+0000 7f6bdb365640 1 -- 172.21.15.83:0/3258468738 wait complete. 2024-03-06T07:27:22.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.535+0000 7f6bdb365640 1 Processor -- start 2024-03-06T07:27:22.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.535+0000 7f6bdb365640 1 -- start start 2024-03-06T07:27:22.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.535+0000 7f6bdb365640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6bd40d22b0 0x7f6bd414c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.535+0000 7f6bdb365640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6bd40d2710 0x7f6bd4161000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.535+0000 7f6bdb365640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6bd415dfe0 0x7f6bd40b6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.536+0000 7f6bdb365640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd40cfa00 con 0x7f6bd40d22b0 2024-03-06T07:27:22.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.536+0000 7f6bdb365640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd40b3d60 con 0x7f6bd415dfe0 2024-03-06T07:27:22.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.536+0000 7f6bdb365640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd40d1c70 con 0x7f6bd40d2710 2024-03-06T07:27:22.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.536+0000 7f6bd9101640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6bd40d22b0 0x7f6bd414c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.536+0000 7f6bd9101640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6bd40d22b0 0x7f6bd414c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41526/0 (socket says 172.21.15.83:41526) 2024-03-06T07:27:22.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.536+0000 7f6bd9101640 1 -- 172.21.15.83:0/2421765993 learned_addr learned my addr 172.21.15.83:0/2421765993 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:22.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.536+0000 7f6bd9902640 1 --2- 172.21.15.83:0/2421765993 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6bd415dfe0 0x7f6bd40b6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.536+0000 7f6bd8900640 1 --2- 172.21.15.83:0/2421765993 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6bd40d2710 0x7f6bd4161000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.536+0000 7f6bd9101640 1 -- 172.21.15.83:0/2421765993 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6bd40d2710 msgr2=0x7f6bd4161000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.536+0000 7f6bd9101640 1 --2- 172.21.15.83:0/2421765993 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6bd40d2710 0x7f6bd4161000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.536+0000 7f6bd9101640 1 -- 172.21.15.83:0/2421765993 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6bd415dfe0 msgr2=0x7f6bd40b6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.536+0000 7f6bd9101640 1 --2- 172.21.15.83:0/2421765993 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6bd415dfe0 0x7f6bd40b6bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.536+0000 7f6bd9101640 1 -- 172.21.15.83:0/2421765993 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd414c630 con 0x7f6bd40d22b0 2024-03-06T07:27:22.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.536+0000 7f6bd9902640 1 --2- 172.21.15.83:0/2421765993 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6bd415dfe0 0x7f6bd40b6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:22.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.536+0000 7f6bd9101640 1 --2- 172.21.15.83:0/2421765993 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6bd40d22b0 0x7f6bd414c0f0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc00c440 tx=0x7f6bcc00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:22.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.536+0000 7f6bc9ffb640 1 -- 172.21.15.83:0/2421765993 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc017070 con 0x7f6bd40d22b0 2024-03-06T07:27:22.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.536+0000 7f6bc9ffb640 1 -- 172.21.15.83:0/2421765993 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bcc005ce0 con 0x7f6bd40d22b0 2024-03-06T07:27:22.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.536+0000 7f6bc9ffb640 1 -- 172.21.15.83:0/2421765993 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc0072b0 con 0x7f6bd40d22b0 2024-03-06T07:27:22.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.536+0000 7f6bdb365640 1 -- 172.21.15.83:0/2421765993 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bd415ff60 con 0x7f6bd40d22b0 2024-03-06T07:27:22.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.537+0000 7f6bdb365640 1 -- 172.21.15.83:0/2421765993 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bd4160520 con 0x7f6bd40d22b0 2024-03-06T07:27:22.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.537+0000 7f6bdb365640 1 -- 172.21.15.83:0/2421765993 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bd404f330 con 0x7f6bd40d22b0 2024-03-06T07:27:22.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.538+0000 7f6bc9ffb640 1 -- 172.21.15.83:0/2421765993 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6bcc01a070 con 0x7f6bd40d22b0 2024-03-06T07:27:22.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.538+0000 7f6bc9ffb640 1 --2- 172.21.15.83:0/2421765993 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6bb806b4d0 0x7f6bb806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.538+0000 7f6bc9ffb640 1 -- 172.21.15.83:0/2421765993 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6bcc089480 con 0x7f6bd40d22b0 2024-03-06T07:27:22.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.538+0000 7f6bd8900640 1 --2- 172.21.15.83:0/2421765993 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6bb806b4d0 0x7f6bb806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.539+0000 7f6bd8900640 1 --2- 172.21.15.83:0/2421765993 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6bb806b4d0 0x7f6bb806d990 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f6bd41f6b70 tx=0x7f6bc4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:22.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.543+0000 7f6bc9ffb640 1 -- 172.21.15.83:0/2421765993 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6bd414c630 con 0x7f6bd40d22b0 2024-03-06T07:27:22.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.812+0000 7f6bdb365640 1 -- 172.21.15.83:0/2421765993 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6bd40747a0 con 0x7f6bd40d22b0 2024-03-06T07:27:22.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.813+0000 7f6bc9ffb640 1 -- 172.21.15.83:0/2421765993 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f6bcc058450 con 0x7f6bd40d22b0 2024-03-06T07:27:22.815 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:22.815 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":59,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6200,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/2154269119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":2154269119},{"type":"v1","addr":"172.21.15.83:6837","nonce":2154269119}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6206,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3898960075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":3898960075},{"type":"v1","addr":"172.21.15.183:6835","nonce":3898960075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6838,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/1383294888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":1383294888},{"type":"v1","addr":"172.21.15.183:6834","nonce":1383294888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.491022+0000","modified":"2024-03-06T07:27:12.512838+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7887},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7887":{"gid":7887,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6836/855387870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":855387870},{"type":"v1","addr":"172.21.15.83:6836","nonce":855387870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7887,"qdb_cluster":[7887]},"id":8}]} 2024-03-06T07:27:22.815 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 59 2024-03-06T07:27:22.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.815+0000 7f6bdb365640 1 -- 172.21.15.83:0/2421765993 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6bb806b4d0 msgr2=0x7f6bb806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.815+0000 7f6bdb365640 1 --2- 172.21.15.83:0/2421765993 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6bb806b4d0 0x7f6bb806d990 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f6bd41f6b70 tx=0x7f6bc4009040 comp rx=0 tx=0).stop 2024-03-06T07:27:22.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.815+0000 7f6bdb365640 1 -- 172.21.15.83:0/2421765993 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6bd40d22b0 msgr2=0x7f6bd414c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.815+0000 7f6bdb365640 1 --2- 172.21.15.83:0/2421765993 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6bd40d22b0 0x7f6bd414c0f0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc00c440 tx=0x7f6bcc00c900 comp rx=0 tx=0).stop 2024-03-06T07:27:22.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.815+0000 7f6bdb365640 1 -- 172.21.15.83:0/2421765993 shutdown_connections 2024-03-06T07:27:22.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.815+0000 7f6bdb365640 1 --2- 172.21.15.83:0/2421765993 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6bb806b4d0 0x7f6bb806d990 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.815+0000 7f6bdb365640 1 --2- 172.21.15.83:0/2421765993 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6bd415dfe0 0x7f6bd40b6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.815+0000 7f6bdb365640 1 --2- 172.21.15.83:0/2421765993 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6bd40d2710 0x7f6bd4161000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.815+0000 7f6bdb365640 1 --2- 172.21.15.83:0/2421765993 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6bd40d22b0 0x7f6bd414c0f0 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.815+0000 7f6bdb365640 1 -- 172.21.15.83:0/2421765993 >> 172.21.15.83:0/2421765993 conn(0x7f6bd40d3cd0 msgr2=0x7f6bd406c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:22.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.815+0000 7f6bdb365640 1 -- 172.21.15.83:0/2421765993 shutdown_connections 2024-03-06T07:27:22.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.815+0000 7f6bdb365640 1 -- 172.21.15.83:0/2421765993 wait complete. 2024-03-06T07:27:22.828 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:27:22.953 INFO:teuthology.orchestra.run.smithi083.stdout:{"message":"","return_code":0} 2024-03-06T07:27:22.953 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:27:22.953 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-06T07:27:22.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.993+0000 7f3fbde0ff40 1 Processor -- start 2024-03-06T07:27:22.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.993+0000 7f3fbde0ff40 1 -- start start 2024-03-06T07:27:22.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.994+0000 7f3fbde0ff40 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3fa4002900 0x55a04c5c7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.994+0000 7f3fbde0ff40 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a04c69b5e0 0x55a04c5c5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.994+0000 7f3fbde0ff40 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55a04c5c6460 0x55a04c5f4090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.994+0000 7f3fbde0ff40 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a04c56bb70 con 0x55a04c69b5e0 2024-03-06T07:27:22.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.994+0000 7f3fbde0ff40 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a04c56fad0 con 0x7f3fa4002900 2024-03-06T07:27:22.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.994+0000 7f3fbde0ff40 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a04c67f9f0 con 0x55a04c5c6460 2024-03-06T07:27:22.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.994+0000 7f3fba639640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a04c69b5e0 0x55a04c5c5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.994+0000 7f3fba639640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a04c69b5e0 0x55a04c5c5f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41544/0 (socket says 172.21.15.83:41544) 2024-03-06T07:27:22.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.994+0000 7f3fba639640 1 -- 172.21.15.83:0/1083730209 learned_addr learned my addr 172.21.15.83:0/1083730209 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:22.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.994+0000 7f3fba639640 1 -- 172.21.15.83:0/1083730209 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55a04c5c6460 msgr2=0x55a04c5f4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:22.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.994+0000 7f3fba639640 1 --2- 172.21.15.83:0/1083730209 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55a04c5c6460 0x55a04c5f4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.994+0000 7f3fba639640 1 -- 172.21.15.83:0/1083730209 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3fa4002900 msgr2=0x55a04c5c7fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:22.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.994+0000 7f3fba639640 1 --2- 172.21.15.83:0/1083730209 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3fa4002900 0x55a04c5c7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.994+0000 7f3fba639640 1 -- 172.21.15.83:0/1083730209 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a04c69acb0 con 0x55a04c69b5e0 2024-03-06T07:27:22.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.995+0000 7f3fba639640 1 --2- 172.21.15.83:0/1083730209 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a04c69b5e0 0x55a04c5c5f20 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f3fb001d1f0 tx=0x7f3fb004d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2a6b384320fd8b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:22.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.995+0000 7f3fb9637640 1 -- 172.21.15.83:0/1083730209 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb0059040 con 0x55a04c69b5e0 2024-03-06T07:27:22.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.995+0000 7f3fb9637640 1 -- 172.21.15.83:0/1083730209 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb005f650 con 0x55a04c69b5e0 2024-03-06T07:27:22.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.995+0000 7f3fb9637640 1 -- 172.21.15.83:0/1083730209 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb005fa40 con 0x55a04c69b5e0 2024-03-06T07:27:22.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.995+0000 7f3fbde0ff40 1 -- 172.21.15.83:0/1083730209 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a04c69b5e0 msgr2=0x55a04c5c5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.995+0000 7f3fbde0ff40 1 --2- 172.21.15.83:0/1083730209 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a04c69b5e0 0x55a04c5c5f20 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f3fb001d1f0 tx=0x7f3fb004d000 comp rx=0 tx=0).stop 2024-03-06T07:27:22.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.995+0000 7f3fbde0ff40 1 -- 172.21.15.83:0/1083730209 shutdown_connections 2024-03-06T07:27:22.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.995+0000 7f3fbde0ff40 1 --2- 172.21.15.83:0/1083730209 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55a04c5c6460 0x55a04c5f4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.995+0000 7f3fbde0ff40 1 --2- 172.21.15.83:0/1083730209 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a04c69b5e0 0x55a04c5c5f20 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.995+0000 7f3fbde0ff40 1 --2- 172.21.15.83:0/1083730209 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3fa4002900 0x55a04c5c7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.995+0000 7f3fbde0ff40 1 -- 172.21.15.83:0/1083730209 >> 172.21.15.83:0/1083730209 conn(0x55a04c5c38b0 msgr2=0x55a04c5f1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:22.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.995+0000 7f3fbde0ff40 1 -- 172.21.15.83:0/1083730209 shutdown_connections 2024-03-06T07:27:23.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.995+0000 7f3fbde0ff40 1 -- 172.21.15.83:0/1083730209 wait complete. 2024-03-06T07:27:23.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.996+0000 7f3fbde0ff40 1 Processor -- start 2024-03-06T07:27:23.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.996+0000 7f3fbde0ff40 1 -- start start 2024-03-06T07:27:23.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.996+0000 7f3fbde0ff40 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55a04c5c6460 0x55a04c67bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.996+0000 7f3fbde0ff40 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a04c69b5e0 0x55a04c5c7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.996+0000 7f3fbde0ff40 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3fa4002900 0x55a04c699110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.997+0000 7f3fbde0ff40 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a04c697310 con 0x7f3fa4002900 2024-03-06T07:27:23.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.997+0000 7f3fbde0ff40 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a04c5f3440 con 0x55a04c69b5e0 2024-03-06T07:27:23.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.997+0000 7f3fbde0ff40 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a04c69b250 con 0x55a04c5c6460 2024-03-06T07:27:23.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.997+0000 7f3fbb63b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3fa4002900 0x55a04c699110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.997+0000 7f3fbb63b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3fa4002900 0x55a04c699110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41554/0 (socket says 172.21.15.83:41554) 2024-03-06T07:27:23.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.997+0000 7f3fbb63b640 1 -- 172.21.15.83:0/3185809812 learned_addr learned my addr 172.21.15.83:0/3185809812 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:23.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.997+0000 7f3fba639640 1 --2- 172.21.15.83:0/3185809812 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a04c69b5e0 0x55a04c5c7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.997+0000 7f3fbae3a640 1 --2- 172.21.15.83:0/3185809812 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55a04c5c6460 0x55a04c67bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.997+0000 7f3fbb63b640 1 -- 172.21.15.83:0/3185809812 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55a04c5c6460 msgr2=0x55a04c67bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.997+0000 7f3fbb63b640 1 --2- 172.21.15.83:0/3185809812 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55a04c5c6460 0x55a04c67bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.997+0000 7f3fbb63b640 1 -- 172.21.15.83:0/3185809812 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a04c69b5e0 msgr2=0x55a04c5c7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.997+0000 7f3fbb63b640 1 --2- 172.21.15.83:0/3185809812 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a04c69b5e0 0x55a04c5c7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.997+0000 7f3fbb63b640 1 -- 172.21.15.83:0/3185809812 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a04c67ab30 con 0x7f3fa4002900 2024-03-06T07:27:23.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.997+0000 7f3fbb63b640 1 --2- 172.21.15.83:0/3185809812 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3fa4002900 0x55a04c699110 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x55a04c60b860 tx=0x7f3fb4003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.997+0000 7f3fa3fff640 1 -- 172.21.15.83:0/3185809812 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb4019070 con 0x7f3fa4002900 2024-03-06T07:27:23.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.997+0000 7f3fa3fff640 1 -- 172.21.15.83:0/3185809812 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb4011350 con 0x7f3fa4002900 2024-03-06T07:27:23.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.997+0000 7f3fa3fff640 1 -- 172.21.15.83:0/3185809812 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb4011700 con 0x7f3fa4002900 2024-03-06T07:27:23.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.997+0000 7f3fbde0ff40 1 -- 172.21.15.83:0/3185809812 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55a04c67ae10 con 0x7f3fa4002900 2024-03-06T07:27:23.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.997+0000 7f3fbde0ff40 1 -- 172.21.15.83:0/3185809812 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a04c67b3d0 con 0x7f3fa4002900 2024-03-06T07:27:23.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.999+0000 7f3fa3fff640 1 -- 172.21.15.83:0/3185809812 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3fb401c070 con 0x7f3fa4002900 2024-03-06T07:27:23.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.999+0000 7f3fa3fff640 1 --2- 172.21.15.83:0/3185809812 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3fa8069ee0 0x7f3fa806c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.999+0000 7f3fa3fff640 1 -- 172.21.15.83:0/3185809812 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3fb4088770 con 0x7f3fa4002900 2024-03-06T07:27:23.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.999+0000 7f3fbae3a640 1 --2- 172.21.15.83:0/3185809812 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3fa8069ee0 0x7f3fa806c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.999+0000 7f3fbde0ff40 1 --2- 172.21.15.83:0/3185809812 >> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] conn(0x55a04c6989f0 0x55a04c673bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.999+0000 7f3fbde0ff40 1 -- 172.21.15.83:0/3185809812 --> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e110) v9 -- 0x55a04c674130 con 0x55a04c6989f0 2024-03-06T07:27:23.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:22.999+0000 7f3fba639640 1 --2- 172.21.15.83:0/3185809812 >> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] conn(0x55a04c6989f0 0x55a04c673bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.000+0000 7f3fbae3a640 1 --2- 172.21.15.83:0/3185809812 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3fa8069ee0 0x7f3fa806c3a0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f3fac0011d0 tx=0x7f3fac0013a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.000+0000 7f3fba639640 1 --2- 172.21.15.83:0/3185809812 >> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] conn(0x55a04c6989f0 0x55a04c673bf0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.000+0000 7f3fba639640 1 -- 172.21.15.83:0/3185809812 <== osd.0 v2:172.21.15.83:6808/1635427425 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (93) out=93b] v0'0 uv3 ondisk = 0) v8 ==== 164+0+93 (crc 0 0 0) 0x7f3fb005f800 con 0x55a04c6989f0 2024-03-06T07:27:23.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.000+0000 7f3fbde0ff40 1 -- 172.21.15.83:0/3185809812 >> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] conn(0x55a04c6989f0 msgr2=0x55a04c673bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.000+0000 7f3fbde0ff40 1 --2- 172.21.15.83:0/3185809812 >> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] conn(0x55a04c6989f0 0x55a04c673bf0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.000+0000 7f3fbde0ff40 1 -- 172.21.15.83:0/3185809812 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3fa8069ee0 msgr2=0x7f3fa806c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.000+0000 7f3fbde0ff40 1 --2- 172.21.15.83:0/3185809812 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3fa8069ee0 0x7f3fa806c3a0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f3fac0011d0 tx=0x7f3fac0013a0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.000+0000 7f3fbde0ff40 1 -- 172.21.15.83:0/3185809812 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3fa4002900 msgr2=0x55a04c699110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.000+0000 7f3fbde0ff40 1 --2- 172.21.15.83:0/3185809812 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3fa4002900 0x55a04c699110 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x55a04c60b860 tx=0x7f3fb4003820 comp rx=0 tx=0).stop 2024-03-06T07:27:23.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.000+0000 7f3fbb63b640 1 -- 172.21.15.83:0/3185809812 reap_dead start 2024-03-06T07:27:23.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.000+0000 7f3fbde0ff40 1 -- 172.21.15.83:0/3185809812 shutdown_connections 2024-03-06T07:27:23.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.000+0000 7f3fbde0ff40 1 -- 172.21.15.83:0/3185809812 >> 172.21.15.83:0/3185809812 conn(0x55a04c5c38b0 msgr2=0x55a04c5f3e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:23.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.000+0000 7f3fbde0ff40 1 -- 172.21.15.83:0/3185809812 shutdown_connections 2024-03-06T07:27:23.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.000+0000 7f3fbde0ff40 1 -- 172.21.15.83:0/3185809812 wait complete. 2024-03-06T07:27:23.008 INFO:teuthology.orchestra.run.smithi083.stdout:Wfile_a 2024-03-06T07:27:23.008 INFO:teuthology.orchestra.run.smithi083.stdout:dir_1 2024-03-06T07:27:23.008 DEBUG:teuthology.orchestra.run.smithi083:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-06T07:27:23.148 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:27:23.148 INFO:teuthology.orchestra.run.smithi083.stdout: "ino": 1099511627777, 2024-03-06T07:27:23.148 INFO:teuthology.orchestra.run.smithi083.stdout: "ancestors": [ 2024-03-06T07:27:23.148 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:27:23.148 INFO:teuthology.orchestra.run.smithi083.stdout: "dirino": 1099511627776, 2024-03-06T07:27:23.148 INFO:teuthology.orchestra.run.smithi083.stdout: "dname": "file_a", 2024-03-06T07:27:23.148 INFO:teuthology.orchestra.run.smithi083.stdout: "version": 10 2024-03-06T07:27:23.148 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:27:23.149 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:27:23.149 INFO:teuthology.orchestra.run.smithi083.stdout: "dirino": 1, 2024-03-06T07:27:23.149 INFO:teuthology.orchestra.run.smithi083.stdout: "dname": "dir_1", 2024-03-06T07:27:23.149 INFO:teuthology.orchestra.run.smithi083.stdout: "version": 17 2024-03-06T07:27:23.149 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:27:23.149 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2024-03-06T07:27:23.149 INFO:teuthology.orchestra.run.smithi083.stdout: "pool": 17, 2024-03-06T07:27:23.149 INFO:teuthology.orchestra.run.smithi083.stdout: "old_pools": [] 2024-03-06T07:27:23.149 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:27:23.149 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:23.164 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:27:23.164 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:23.180 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:23.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.319+0000 7fd591678640 1 Processor -- start 2024-03-06T07:27:23.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.320+0000 7fd591678640 1 -- start start 2024-03-06T07:27:23.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.320+0000 7fd591678640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd58c1424f0 0x7fd58c0cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.320+0000 7fd591678640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fd58c06da40 0x7fd58c15cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.320+0000 7fd591678640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd58c0cc980 con 0x7fd58c0cd7a0 2024-03-06T07:27:23.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.320+0000 7fd591678640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd58c0cd220 con 0x7fd58c1424f0 2024-03-06T07:27:23.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.320+0000 7fd591678640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd58c0ccc60 con 0x7fd58c06da40 2024-03-06T07:27:23.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.320+0000 7fd58affd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd58c1424f0 0x7fd58c0cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.321+0000 7fd58a7fc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fd58c0cd7a0 0x7fd58c06d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57126/0 (socket says 172.21.15.83:57126) 2024-03-06T07:27:23.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.321+0000 7fd58a7fc640 1 -- 172.21.15.83:0/39856633 learned_addr learned my addr 172.21.15.83:0/39856633 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:27:23.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.321+0000 7fd58affd640 1 -- 172.21.15.83:0/39856633 >> v2:172.21.15.183:3300/0 conn(0x7fd58c06da40 msgr2=0x7fd58c15cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:23.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.321+0000 7fd58b7fe640 1 --2- 172.21.15.83:0/39856633 >> v2:172.21.15.183:3300/0 conn(0x7fd58c06da40 0x7fd58c15cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.321+0000 7fd58affd640 1 --2- 172.21.15.83:0/39856633 >> v2:172.21.15.183:3300/0 conn(0x7fd58c06da40 0x7fd58c15cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.321+0000 7fd58affd640 1 -- 172.21.15.83:0/39856633 >> v1:172.21.15.83:6789/0 conn(0x7fd58c0cd7a0 legacy=0x7fd58c06d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.321+0000 7fd58affd640 1 -- 172.21.15.83:0/39856633 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd58c0ccf40 con 0x7fd58c1424f0 2024-03-06T07:27:23.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.321+0000 7fd589ffb640 1 -- 172.21.15.83:0/39856633 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1922590510 0 0) 0x7fd58c0cc980 con 0x7fd58c0cd7a0 2024-03-06T07:27:23.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.322+0000 7fd58affd640 1 --2- 172.21.15.83:0/39856633 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd58c1424f0 0x7fd58c0cc2f0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7fd58c068700 tx=0x7fd58002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0e9d916cdb34ee7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.322+0000 7fd589ffb640 1 -- 172.21.15.83:0/39856633 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd580035020 con 0x7fd58c1424f0 2024-03-06T07:27:23.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.322+0000 7fd589ffb640 1 -- 172.21.15.83:0/39856633 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd580031830 con 0x7fd58c1424f0 2024-03-06T07:27:23.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.322+0000 7fd589ffb640 1 -- 172.21.15.83:0/39856633 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd580031c00 con 0x7fd58c1424f0 2024-03-06T07:27:23.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.322+0000 7fd591678640 1 -- 172.21.15.83:0/39856633 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd58c1424f0 msgr2=0x7fd58c0cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.322+0000 7fd591678640 1 --2- 172.21.15.83:0/39856633 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd58c1424f0 0x7fd58c0cc2f0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7fd58c068700 tx=0x7fd58002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.322+0000 7fd591678640 1 -- 172.21.15.83:0/39856633 shutdown_connections 2024-03-06T07:27:23.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.322+0000 7fd591678640 1 --2- 172.21.15.83:0/39856633 >> v2:172.21.15.183:3300/0 conn(0x7fd58c06da40 0x7fd58c15cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.322+0000 7fd591678640 1 --2- 172.21.15.83:0/39856633 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd58c1424f0 0x7fd58c0cc2f0 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.322+0000 7fd591678640 1 -- 172.21.15.83:0/39856633 >> 172.21.15.83:0/39856633 conn(0x7fd58c0cf8a0 msgr2=0x7fd58c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:23.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.322+0000 7fd591678640 1 -- 172.21.15.83:0/39856633 shutdown_connections 2024-03-06T07:27:23.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.322+0000 7fd591678640 1 -- 172.21.15.83:0/39856633 wait complete. 2024-03-06T07:27:23.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.324+0000 7fd591678640 1 Processor -- start 2024-03-06T07:27:23.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.324+0000 7fd591678640 1 -- start start 2024-03-06T07:27:23.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.325+0000 7fd591678640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd58c06da40 0x7fd58c15ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.325+0000 7fd591678640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd58c0cd7a0 0x7fd58c1f4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.325+0000 7fd591678640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd58c1424f0 0x7fd58c1f54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.325+0000 7fd591678640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd58c071ce0 con 0x7fd58c0cd7a0 2024-03-06T07:27:23.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.325+0000 7fd591678640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd58c071b60 con 0x7fd58c1424f0 2024-03-06T07:27:23.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.325+0000 7fd591678640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd58c071e60 con 0x7fd58c06da40 2024-03-06T07:27:23.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.325+0000 7fd58a7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd58c0cd7a0 0x7fd58c1f4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.325+0000 7fd58a7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd58c0cd7a0 0x7fd58c1f4f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41570/0 (socket says 172.21.15.83:41570) 2024-03-06T07:27:23.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.325+0000 7fd58a7fc640 1 -- 172.21.15.83:0/1043553068 learned_addr learned my addr 172.21.15.83:0/1043553068 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:23.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.325+0000 7fd58b7fe640 1 --2- 172.21.15.83:0/1043553068 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd58c1424f0 0x7fd58c1f54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.325+0000 7fd58affd640 1 --2- 172.21.15.83:0/1043553068 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd58c06da40 0x7fd58c15ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.325+0000 7fd58a7fc640 1 -- 172.21.15.83:0/1043553068 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd58c06da40 msgr2=0x7fd58c15ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.325+0000 7fd58a7fc640 1 --2- 172.21.15.83:0/1043553068 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd58c06da40 0x7fd58c15ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.325+0000 7fd58a7fc640 1 -- 172.21.15.83:0/1043553068 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd58c1424f0 msgr2=0x7fd58c1f54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.325+0000 7fd58a7fc640 1 --2- 172.21.15.83:0/1043553068 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd58c1424f0 0x7fd58c1f54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.325+0000 7fd58a7fc640 1 -- 172.21.15.83:0/1043553068 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd58c158250 con 0x7fd58c0cd7a0 2024-03-06T07:27:23.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.325+0000 7fd58affd640 1 --2- 172.21.15.83:0/1043553068 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd58c06da40 0x7fd58c15ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:23.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.325+0000 7fd58b7fe640 1 --2- 172.21.15.83:0/1043553068 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd58c1424f0 0x7fd58c1f54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:23.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.325+0000 7fd58a7fc640 1 --2- 172.21.15.83:0/1043553068 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd58c0cd7a0 0x7fd58c1f4f60 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7fd57c00c4d0 tx=0x7fd57c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.326+0000 7fd5737fe640 1 -- 172.21.15.83:0/1043553068 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57c018070 con 0x7fd58c0cd7a0 2024-03-06T07:27:23.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.326+0000 7fd5737fe640 1 -- 172.21.15.83:0/1043553068 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd57c007500 con 0x7fd58c0cd7a0 2024-03-06T07:27:23.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.326+0000 7fd591678640 1 -- 172.21.15.83:0/1043553068 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd58c158530 con 0x7fd58c0cd7a0 2024-03-06T07:27:23.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.326+0000 7fd5737fe640 1 -- 172.21.15.83:0/1043553068 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57c004a20 con 0x7fd58c0cd7a0 2024-03-06T07:27:23.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.326+0000 7fd591678640 1 -- 172.21.15.83:0/1043553068 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd58c0a9e90 con 0x7fd58c0cd7a0 2024-03-06T07:27:23.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.327+0000 7fd591678640 1 -- 172.21.15.83:0/1043553068 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd58c158920 con 0x7fd58c0cd7a0 2024-03-06T07:27:23.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.327+0000 7fd5737fe640 1 -- 172.21.15.83:0/1043553068 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd57c01b070 con 0x7fd58c0cd7a0 2024-03-06T07:27:23.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.328+0000 7fd5737fe640 1 --2- 172.21.15.83:0/1043553068 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd57806b4d0 0x7fd57806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.328+0000 7fd58affd640 1 --2- 172.21.15.83:0/1043553068 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd57806b4d0 0x7fd57806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.328+0000 7fd5737fe640 1 -- 172.21.15.83:0/1043553068 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd57c0894b0 con 0x7fd58c0cd7a0 2024-03-06T07:27:23.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.328+0000 7fd58affd640 1 --2- 172.21.15.83:0/1043553068 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd57806b4d0 0x7fd57806d990 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fd580036000 tx=0x7fd580008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.332+0000 7fd5737fe640 1 -- 172.21.15.83:0/1043553068 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd57c0583c0 con 0x7fd58c0cd7a0 2024-03-06T07:27:23.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.537+0000 7fd591678640 1 -- 172.21.15.83:0/1043553068 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd58c04f330 con 0x7fd58c0cd7a0 2024-03-06T07:27:23.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.538+0000 7fd5737fe640 1 -- 172.21.15.83:0/1043553068 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+16639 (secure 0 0 0) 0x7fd57c057cc0 con 0x7fd58c0cd7a0 2024-03-06T07:27:23.540 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:23.541 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":110,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:27:12.146137+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:08.709967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:09.456518+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:23.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.540+0000 7fd591678640 1 -- 172.21.15.83:0/1043553068 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd57806b4d0 msgr2=0x7fd57806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.540+0000 7fd591678640 1 --2- 172.21.15.83:0/1043553068 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd57806b4d0 0x7fd57806d990 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fd580036000 tx=0x7fd580008030 comp rx=0 tx=0).stop 2024-03-06T07:27:23.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.540+0000 7fd591678640 1 -- 172.21.15.83:0/1043553068 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd58c0cd7a0 msgr2=0x7fd58c1f4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.540+0000 7fd591678640 1 --2- 172.21.15.83:0/1043553068 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd58c0cd7a0 0x7fd58c1f4f60 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7fd57c00c4d0 tx=0x7fd57c00c990 comp rx=0 tx=0).stop 2024-03-06T07:27:23.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.540+0000 7fd591678640 1 -- 172.21.15.83:0/1043553068 shutdown_connections 2024-03-06T07:27:23.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.540+0000 7fd591678640 1 --2- 172.21.15.83:0/1043553068 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd57806b4d0 0x7fd57806d990 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.540+0000 7fd591678640 1 --2- 172.21.15.83:0/1043553068 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd58c1424f0 0x7fd58c1f54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.540+0000 7fd591678640 1 --2- 172.21.15.83:0/1043553068 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd58c0cd7a0 0x7fd58c1f4f60 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.540+0000 7fd591678640 1 --2- 172.21.15.83:0/1043553068 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd58c06da40 0x7fd58c15ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.540+0000 7fd591678640 1 -- 172.21.15.83:0/1043553068 >> 172.21.15.83:0/1043553068 conn(0x7fd58c0cf8a0 msgr2=0x7fd58c0cf0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:23.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.540+0000 7fd591678640 1 -- 172.21.15.83:0/1043553068 shutdown_connections 2024-03-06T07:27:23.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.540+0000 7fd591678640 1 -- 172.21.15.83:0/1043553068 wait complete. 2024-03-06T07:27:23.553 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:08.709967+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:09.456518+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:27:23.553 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:27:23.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.655+0000 7fc0ac781640 1 Processor -- start 2024-03-06T07:27:23.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.656+0000 7fc0ac781640 1 -- start start 2024-03-06T07:27:23.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.656+0000 7fc0ac781640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc0a40d4860 0x7fc0a40d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.656+0000 7fc0ac781640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fc0a40b0e30 0x7fc0a40b1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.656+0000 7fc0ac781640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0a41618b0 con 0x7fc0a41614d0 2024-03-06T07:27:23.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.656+0000 7fc0ac781640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0a40d4cf0 con 0x7fc0a40d4860 2024-03-06T07:27:23.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.656+0000 7fc0ac781640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0a40cf400 con 0x7fc0a40b0e30 2024-03-06T07:27:23.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.657+0000 7fc0a9d1c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc0a40d4860 0x7fc0a40d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.657+0000 7fc0a9d1c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc0a40d4860 0x7fc0a40d2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41580/0 (socket says 172.21.15.83:41580) 2024-03-06T07:27:23.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.657+0000 7fc0a9d1c640 1 -- 172.21.15.83:0/3019272472 learned_addr learned my addr 172.21.15.83:0/3019272472 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:23.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.657+0000 7fc0aad1e640 1 --2- 172.21.15.83:0/3019272472 >> v2:172.21.15.183:3300/0 conn(0x7fc0a40b0e30 0x7fc0a40b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.657+0000 7fc0a9d1c640 1 -- 172.21.15.83:0/3019272472 >> v2:172.21.15.183:3300/0 conn(0x7fc0a40b0e30 msgr2=0x7fc0a40b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:23.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.657+0000 7fc0a9d1c640 1 --2- 172.21.15.83:0/3019272472 >> v2:172.21.15.183:3300/0 conn(0x7fc0a40b0e30 0x7fc0a40b1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.657+0000 7fc0a9d1c640 1 -- 172.21.15.83:0/3019272472 >> v1:172.21.15.183:6789/0 conn(0x7fc0a41614d0 legacy=0x7fc0a40d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:23.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.657+0000 7fc0a9d1c640 1 -- 172.21.15.83:0/3019272472 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0a40cf6a0 con 0x7fc0a40d4860 2024-03-06T07:27:23.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.658+0000 7fc0a9d1c640 1 --2- 172.21.15.83:0/3019272472 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0a40d4860 0x7fc0a40d2020 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7fc0a40ce550 tx=0x7fc09402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59b147e5250a9b71 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.658+0000 7fc0a951b640 1 -- 172.21.15.83:0/3019272472 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc094035020 con 0x7fc0a40d4860 2024-03-06T07:27:23.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.658+0000 7fc0a951b640 1 -- 172.21.15.83:0/3019272472 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc094031640 con 0x7fc0a40d4860 2024-03-06T07:27:23.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.658+0000 7fc0a951b640 1 -- 172.21.15.83:0/3019272472 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc09402d040 con 0x7fc0a40d4860 2024-03-06T07:27:23.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.658+0000 7fc0ac781640 1 -- 172.21.15.83:0/3019272472 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0a40d4860 msgr2=0x7fc0a40d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.658+0000 7fc0ac781640 1 --2- 172.21.15.83:0/3019272472 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0a40d4860 0x7fc0a40d2020 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7fc0a40ce550 tx=0x7fc09402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.658+0000 7fc0ac781640 1 -- 172.21.15.83:0/3019272472 shutdown_connections 2024-03-06T07:27:23.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.658+0000 7fc0ac781640 1 --2- 172.21.15.83:0/3019272472 >> v2:172.21.15.183:3300/0 conn(0x7fc0a40b0e30 0x7fc0a40b1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.658+0000 7fc0ac781640 1 --2- 172.21.15.83:0/3019272472 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0a40d4860 0x7fc0a40d2020 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.658+0000 7fc0ac781640 1 -- 172.21.15.83:0/3019272472 >> 172.21.15.83:0/3019272472 conn(0x7fc0a40d3d40 msgr2=0x7fc0a40d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:23.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.658+0000 7fc0ac781640 1 -- 172.21.15.83:0/3019272472 shutdown_connections 2024-03-06T07:27:23.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.658+0000 7fc0ac781640 1 -- 172.21.15.83:0/3019272472 wait complete. 2024-03-06T07:27:23.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.659+0000 7fc0ac781640 1 Processor -- start 2024-03-06T07:27:23.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.659+0000 7fc0ac781640 1 -- start start 2024-03-06T07:27:23.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.660+0000 7fc0ac781640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc0a40b0e30 0x7fc0a40c7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.660+0000 7fc0ac781640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0a40d4860 0x7fc0a40ca6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.660+0000 7fc0ac781640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0a41614d0 0x7fc0a40cd8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.660+0000 7fc0ac781640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a40b0910 con 0x7fc0a40d4860 2024-03-06T07:27:23.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.660+0000 7fc0ac781640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a40cfc60 con 0x7fc0a41614d0 2024-03-06T07:27:23.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.660+0000 7fc0ac781640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a40d2560 con 0x7fc0a40b0e30 2024-03-06T07:27:23.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.660+0000 7fc0a9d1c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0a40d4860 0x7fc0a40ca6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.660+0000 7fc0a9d1c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0a40d4860 0x7fc0a40ca6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41582/0 (socket says 172.21.15.83:41582) 2024-03-06T07:27:23.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.660+0000 7fc0a9d1c640 1 -- 172.21.15.83:0/1574832385 learned_addr learned my addr 172.21.15.83:0/1574832385 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:23.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.660+0000 7fc0aad1e640 1 --2- 172.21.15.83:0/1574832385 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0a41614d0 0x7fc0a40cd8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.660+0000 7fc0a9d1c640 1 -- 172.21.15.83:0/1574832385 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc0a40b0e30 msgr2=0x7fc0a40c7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.660+0000 7fc0aa51d640 1 --2- 172.21.15.83:0/1574832385 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc0a40b0e30 0x7fc0a40c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.660+0000 7fc0a9d1c640 1 --2- 172.21.15.83:0/1574832385 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc0a40b0e30 0x7fc0a40c7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.660+0000 7fc0a9d1c640 1 -- 172.21.15.83:0/1574832385 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0a41614d0 msgr2=0x7fc0a40cd8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.660+0000 7fc0a9d1c640 1 --2- 172.21.15.83:0/1574832385 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0a41614d0 0x7fc0a40cd8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.661+0000 7fc0a9d1c640 1 -- 172.21.15.83:0/1574832385 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0a40b4760 con 0x7fc0a40d4860 2024-03-06T07:27:23.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.661+0000 7fc0aad1e640 1 --2- 172.21.15.83:0/1574832385 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0a41614d0 0x7fc0a40cd8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:23.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.661+0000 7fc0aa51d640 1 --2- 172.21.15.83:0/1574832385 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc0a40b0e30 0x7fc0a40c7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:23.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.661+0000 7fc0a9d1c640 1 --2- 172.21.15.83:0/1574832385 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0a40d4860 0x7fc0a40ca6f0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7fc094000c00 tx=0x7fc0940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.661+0000 7fc09affd640 1 -- 172.21.15.83:0/1574832385 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc094040070 con 0x7fc0a40d4860 2024-03-06T07:27:23.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.661+0000 7fc09affd640 1 -- 172.21.15.83:0/1574832385 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0940399f0 con 0x7fc0a40d4860 2024-03-06T07:27:23.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.661+0000 7fc09affd640 1 -- 172.21.15.83:0/1574832385 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc094035040 con 0x7fc0a40d4860 2024-03-06T07:27:23.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.661+0000 7fc0ac781640 1 -- 172.21.15.83:0/1574832385 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0a40b4a40 con 0x7fc0a40d4860 2024-03-06T07:27:23.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.661+0000 7fc0ac781640 1 -- 172.21.15.83:0/1574832385 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0a40b5000 con 0x7fc0a40d4860 2024-03-06T07:27:23.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.662+0000 7fc0ac781640 1 -- 172.21.15.83:0/1574832385 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0a40b51d0 con 0x7fc0a40d4860 2024-03-06T07:27:23.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.662+0000 7fc09affd640 1 -- 172.21.15.83:0/1574832385 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc09404b060 con 0x7fc0a40d4860 2024-03-06T07:27:23.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.663+0000 7fc09affd640 1 --2- 172.21.15.83:0/1574832385 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc08806b4d0 0x7fc08806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.663+0000 7fc09affd640 1 -- 172.21.15.83:0/1574832385 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc0940abba0 con 0x7fc0a40d4860 2024-03-06T07:27:23.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.663+0000 7fc0aa51d640 1 --2- 172.21.15.83:0/1574832385 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc08806b4d0 0x7fc08806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.663+0000 7fc0aa51d640 1 --2- 172.21.15.83:0/1574832385 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc08806b4d0 0x7fc08806d990 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fc0a000a830 tx=0x7fc0a0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.667+0000 7fc09affd640 1 -- 172.21.15.83:0/1574832385 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc09402f300 con 0x7fc0a40d4860 2024-03-06T07:27:23.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.838+0000 7fc0ac781640 1 -- 172.21.15.83:0/1574832385 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc0a40b5f90 con 0x7fc0a40d4860 2024-03-06T07:27:23.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.838+0000 7fc09affd640 1 -- 172.21.15.83:0/1574832385 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v110) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc09407aab0 con 0x7fc0a40d4860 2024-03-06T07:27:23.840 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:27:23.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.840+0000 7fc0ac781640 1 -- 172.21.15.83:0/1574832385 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc08806b4d0 msgr2=0x7fc08806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.840+0000 7fc0ac781640 1 --2- 172.21.15.83:0/1574832385 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc08806b4d0 0x7fc08806d990 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fc0a000a830 tx=0x7fc0a0008040 comp rx=0 tx=0).stop 2024-03-06T07:27:23.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.840+0000 7fc0ac781640 1 -- 172.21.15.83:0/1574832385 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0a40d4860 msgr2=0x7fc0a40ca6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.840+0000 7fc0ac781640 1 --2- 172.21.15.83:0/1574832385 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0a40d4860 0x7fc0a40ca6f0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7fc094000c00 tx=0x7fc0940046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.840+0000 7fc0ac781640 1 -- 172.21.15.83:0/1574832385 shutdown_connections 2024-03-06T07:27:23.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.840+0000 7fc0ac781640 1 --2- 172.21.15.83:0/1574832385 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc08806b4d0 0x7fc08806d990 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.840+0000 7fc0ac781640 1 --2- 172.21.15.83:0/1574832385 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0a41614d0 0x7fc0a40cd8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.840+0000 7fc0ac781640 1 --2- 172.21.15.83:0/1574832385 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0a40d4860 0x7fc0a40ca6f0 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.840+0000 7fc0ac781640 1 --2- 172.21.15.83:0/1574832385 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc0a40b0e30 0x7fc0a40c7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.840+0000 7fc0ac781640 1 -- 172.21.15.83:0/1574832385 >> 172.21.15.83:0/1574832385 conn(0x7fc0a40d3d40 msgr2=0x7fc0a40d0f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:23.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.840+0000 7fc0ac781640 1 -- 172.21.15.83:0/1574832385 shutdown_connections 2024-03-06T07:27:23.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.840+0000 7fc0ac781640 1 -- 172.21.15.83:0/1574832385 wait complete. 2024-03-06T07:27:23.853 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:27:23.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.955+0000 7f1ed739a640 1 Processor -- start 2024-03-06T07:27:23.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.955+0000 7f1ed739a640 1 -- start start 2024-03-06T07:27:23.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.956+0000 7f1ed739a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1ed015a5b0 0x7f1ed00af430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.957+0000 7f1ed739a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ed015e0c0 con 0x7f1ed00b2480 2024-03-06T07:27:23.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.957+0000 7f1ed739a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ed015e360 con 0x7f1ed00af970 2024-03-06T07:27:23.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.957+0000 7f1ed739a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed015e640 con 0x7f1ed015a5b0 2024-03-06T07:27:23.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.957+0000 7f1ed5937640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f1ed00b2480 0x7f1ed00b0990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57138/0 (socket says 172.21.15.83:57138) 2024-03-06T07:27:23.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.957+0000 7f1ed5937640 1 -- 172.21.15.83:0/3485378808 learned_addr learned my addr 172.21.15.83:0/3485378808 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:27:23.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.957+0000 7f1ed5136640 1 --2- 172.21.15.83:0/3485378808 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1ed015a5b0 0x7f1ed00af430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.957+0000 7f1ec7fff640 1 -- 172.21.15.83:0/3485378808 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2076668884 0 0) 0x7f1ed015e0c0 con 0x7f1ed00b2480 2024-03-06T07:27:23.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.957+0000 7f1ec7fff640 1 -- 172.21.15.83:0/3485378808 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ebc003520 con 0x7f1ed00b2480 2024-03-06T07:27:23.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.957+0000 7f1ec7fff640 1 -- 172.21.15.83:0/3485378808 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3524786803 0 0) 0x7f1ebc003520 con 0x7f1ed00b2480 2024-03-06T07:27:23.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.957+0000 7f1ec7fff640 1 -- 172.21.15.83:0/3485378808 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1ed015a5b0 msgr2=0x7f1ed00af430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:23.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.957+0000 7f1ec7fff640 1 --2- 172.21.15.83:0/3485378808 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1ed015a5b0 0x7f1ed00af430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.957+0000 7f1ec7fff640 1 -- 172.21.15.83:0/3485378808 >> v1:172.21.15.183:6789/0 conn(0x7f1ed00af970 legacy=0x7f1ed00b1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.957+0000 7f1ec7fff640 1 -- 172.21.15.83:0/3485378808 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ed015e920 con 0x7f1ed00b2480 2024-03-06T07:27:23.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.957+0000 7f1ec7fff640 1 -- 172.21.15.83:0/3485378808 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f1ec80019f0 con 0x7f1ed00b2480 2024-03-06T07:27:23.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.957+0000 7f1ec7fff640 1 -- 172.21.15.83:0/3485378808 >> v1:172.21.15.83:6789/0 conn(0x7f1ed00b2480 legacy=0x7f1ed00b0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.957+0000 7f1ec7fff640 1 --2- 172.21.15.83:0/3485378808 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1ebc0039e0 0x7f1ebc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.957+0000 7f1ec7fff640 1 --2- 172.21.15.83:0/3485378808 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1ed015a5b0 0x7f1ebc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.957+0000 7f1ec7fff640 1 --2- 172.21.15.83:0/3485378808 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ebc004d40 0x7f1ebc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.957+0000 7f1ec7fff640 1 -- 172.21.15.83:0/3485378808 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed015e640 con 0x7f1ebc004d40 2024-03-06T07:27:23.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.957+0000 7f1ec7fff640 1 -- 172.21.15.83:0/3485378808 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc00ba70 con 0x7f1ebc0039e0 2024-03-06T07:27:23.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.957+0000 7f1ec7fff640 1 -- 172.21.15.83:0/3485378808 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc00de80 con 0x7f1ed015a5b0 2024-03-06T07:27:23.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.957+0000 7f1ed5937640 1 --2- 172.21.15.83:0/3485378808 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ebc004d40 0x7f1ebc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.957+0000 7f1ed5136640 1 --2- 172.21.15.83:0/3485378808 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1ebc0039e0 0x7f1ebc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.958+0000 7f1ed4935640 1 --2- 172.21.15.83:0/3485378808 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1ed015a5b0 0x7f1ebc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.958+0000 7f1ed5937640 1 -- 172.21.15.83:0/3485378808 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1ed015a5b0 msgr2=0x7f1ebc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:23.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.958+0000 7f1ed5937640 1 --2- 172.21.15.83:0/3485378808 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1ed015a5b0 0x7f1ebc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.958+0000 7f1ed5937640 1 -- 172.21.15.83:0/3485378808 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1ebc0039e0 msgr2=0x7f1ebc003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:23.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.958+0000 7f1ed5937640 1 --2- 172.21.15.83:0/3485378808 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1ebc0039e0 0x7f1ebc003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.958+0000 7f1ed5937640 1 -- 172.21.15.83:0/3485378808 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ebc003520 con 0x7f1ebc004d40 2024-03-06T07:27:23.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.958+0000 7f1ed5937640 1 --2- 172.21.15.83:0/3485378808 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ebc004d40 0x7f1ebc0051c0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f1ec8002230 tx=0x7f1ebc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f9290a2606409e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.959+0000 7f1ec7fff640 1 -- 172.21.15.83:0/3485378808 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ec800a4a0 con 0x7f1ebc004d40 2024-03-06T07:27:23.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.959+0000 7f1ec7fff640 1 -- 172.21.15.83:0/3485378808 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ec8034650 con 0x7f1ebc004d40 2024-03-06T07:27:23.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.959+0000 7f1ec7fff640 1 -- 172.21.15.83:0/3485378808 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ec802ce60 con 0x7f1ebc004d40 2024-03-06T07:27:23.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.959+0000 7f1ed739a640 1 -- 172.21.15.83:0/3485378808 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ebc004d40 msgr2=0x7f1ebc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.959+0000 7f1ed739a640 1 --2- 172.21.15.83:0/3485378808 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ebc004d40 0x7f1ebc0051c0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f1ec8002230 tx=0x7f1ebc005830 comp rx=0 tx=0).stop 2024-03-06T07:27:23.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.959+0000 7f1ed5937640 1 -- 172.21.15.83:0/3485378808 reap_dead start 2024-03-06T07:27:23.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.959+0000 7f1ed739a640 1 -- 172.21.15.83:0/3485378808 shutdown_connections 2024-03-06T07:27:23.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.959+0000 7f1ed739a640 1 -- 172.21.15.83:0/3485378808 >> 172.21.15.83:0/3485378808 conn(0x7f1ed00b3c70 msgr2=0x7f1ed00b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:23.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.959+0000 7f1ed739a640 1 -- 172.21.15.83:0/3485378808 shutdown_connections 2024-03-06T07:27:23.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.959+0000 7f1ed739a640 1 -- 172.21.15.83:0/3485378808 wait complete. 2024-03-06T07:27:23.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.960+0000 7f1ed739a640 1 Processor -- start 2024-03-06T07:27:23.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.960+0000 7f1ed739a640 1 -- start start 2024-03-06T07:27:23.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.961+0000 7f1ed739a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ed009d8e0 0x7f1ed01f0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.961+0000 7f1ed739a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1ed009dcc0 0x7f1ed00d0500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.961+0000 7f1ed739a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1ed00d0a40 0x7f1ed01566e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.961+0000 7f1ed739a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed0071960 con 0x7f1ed009d8e0 2024-03-06T07:27:23.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.961+0000 7f1ed739a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed00717e0 con 0x7f1ed009dcc0 2024-03-06T07:27:23.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.961+0000 7f1ed739a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed00af430 con 0x7f1ed00d0a40 2024-03-06T07:27:23.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.961+0000 7f1ed5136640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ed009d8e0 0x7f1ed01f0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.961+0000 7f1ed5136640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ed009d8e0 0x7f1ed01f0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41598/0 (socket says 172.21.15.83:41598) 2024-03-06T07:27:23.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.961+0000 7f1ed5136640 1 -- 172.21.15.83:0/1887471296 learned_addr learned my addr 172.21.15.83:0/1887471296 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:23.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.961+0000 7f1ed5937640 1 --2- 172.21.15.83:0/1887471296 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1ed00d0a40 0x7f1ed01566e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.961+0000 7f1ed4935640 1 --2- 172.21.15.83:0/1887471296 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1ed009dcc0 0x7f1ed00d0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.961+0000 7f1ed5136640 1 -- 172.21.15.83:0/1887471296 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1ed00d0a40 msgr2=0x7f1ed01566e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.961+0000 7f1ed5136640 1 --2- 172.21.15.83:0/1887471296 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1ed00d0a40 0x7f1ed01566e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.961+0000 7f1ed5136640 1 -- 172.21.15.83:0/1887471296 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1ed009dcc0 msgr2=0x7f1ed00d0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.961+0000 7f1ed5136640 1 --2- 172.21.15.83:0/1887471296 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1ed009dcc0 0x7f1ed00d0500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.961+0000 7f1ed5136640 1 -- 172.21.15.83:0/1887471296 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ed015bb10 con 0x7f1ed009d8e0 2024-03-06T07:27:23.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.961+0000 7f1ed5937640 1 --2- 172.21.15.83:0/1887471296 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1ed00d0a40 0x7f1ed01566e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:23.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.962+0000 7f1ed4935640 1 --2- 172.21.15.83:0/1887471296 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1ed009dcc0 0x7f1ed00d0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:23.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.962+0000 7f1ed5136640 1 --2- 172.21.15.83:0/1887471296 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ed009d8e0 0x7f1ed01f0420 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc0119d0 tx=0x7f1ecc011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.962+0000 7f1ec5ffb640 1 -- 172.21.15.83:0/1887471296 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ecc00faf0 con 0x7f1ed009d8e0 2024-03-06T07:27:23.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.962+0000 7f1ec5ffb640 1 -- 172.21.15.83:0/1887471296 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ecc00fc90 con 0x7f1ed009d8e0 2024-03-06T07:27:23.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.962+0000 7f1ed739a640 1 -- 172.21.15.83:0/1887471296 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ed015bdf0 con 0x7f1ed009d8e0 2024-03-06T07:27:23.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.962+0000 7f1ec5ffb640 1 -- 172.21.15.83:0/1887471296 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ecc014440 con 0x7f1ed009d8e0 2024-03-06T07:27:23.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.962+0000 7f1ed739a640 1 -- 172.21.15.83:0/1887471296 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ed015c3b0 con 0x7f1ed009d8e0 2024-03-06T07:27:23.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.963+0000 7f1ed739a640 1 -- 172.21.15.83:0/1887471296 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ed009e3d0 con 0x7f1ed009d8e0 2024-03-06T07:27:23.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.963+0000 7f1ec5ffb640 1 -- 172.21.15.83:0/1887471296 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1ecc012040 con 0x7f1ed009d8e0 2024-03-06T07:27:23.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.964+0000 7f1ec5ffb640 1 --2- 172.21.15.83:0/1887471296 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1eb406b4d0 0x7f1eb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.964+0000 7f1ed4935640 1 --2- 172.21.15.83:0/1887471296 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1eb406b4d0 0x7f1eb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.964+0000 7f1ec5ffb640 1 -- 172.21.15.83:0/1887471296 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1ecc08a870 con 0x7f1ed009d8e0 2024-03-06T07:27:23.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.964+0000 7f1ed4935640 1 --2- 172.21.15.83:0/1887471296 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1eb406b4d0 0x7f1eb406d990 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f1ed00b2250 tx=0x7f1ec0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:23.968+0000 7f1ec5ffb640 1 -- 172.21.15.83:0/1887471296 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1ecc019030 con 0x7f1ed009d8e0 2024-03-06T07:27:24.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.141+0000 7f1ed739a640 1 -- 172.21.15.83:0/1887471296 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1ed00d1050 con 0x7f1ed009d8e0 2024-03-06T07:27:24.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.141+0000 7f1ec5ffb640 1 -- 172.21.15.83:0/1887471296 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v110) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1ecc0596f0 con 0x7f1ed009d8e0 2024-03-06T07:27:24.143 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-03-06T07:27:24.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.143+0000 7f1ed739a640 1 -- 172.21.15.83:0/1887471296 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1eb406b4d0 msgr2=0x7f1eb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.143+0000 7f1ed739a640 1 --2- 172.21.15.83:0/1887471296 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1eb406b4d0 0x7f1eb406d990 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f1ed00b2250 tx=0x7f1ec0009040 comp rx=0 tx=0).stop 2024-03-06T07:27:24.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.143+0000 7f1ed739a640 1 -- 172.21.15.83:0/1887471296 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ed009d8e0 msgr2=0x7f1ed01f0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.143+0000 7f1ed739a640 1 --2- 172.21.15.83:0/1887471296 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ed009d8e0 0x7f1ed01f0420 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc0119d0 tx=0x7f1ecc011e90 comp rx=0 tx=0).stop 2024-03-06T07:27:24.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.143+0000 7f1ed739a640 1 -- 172.21.15.83:0/1887471296 shutdown_connections 2024-03-06T07:27:24.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.143+0000 7f1ed739a640 1 --2- 172.21.15.83:0/1887471296 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1eb406b4d0 0x7f1eb406d990 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.143+0000 7f1ed739a640 1 --2- 172.21.15.83:0/1887471296 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1ed00d0a40 0x7f1ed01566e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.143+0000 7f1ed739a640 1 --2- 172.21.15.83:0/1887471296 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1ed009dcc0 0x7f1ed00d0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.143+0000 7f1ed739a640 1 --2- 172.21.15.83:0/1887471296 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ed009d8e0 0x7f1ed01f0420 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.143+0000 7f1ed739a640 1 -- 172.21.15.83:0/1887471296 >> 172.21.15.83:0/1887471296 conn(0x7f1ed00b3c70 msgr2=0x7f1ed00ad710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.143+0000 7f1ed739a640 1 -- 172.21.15.83:0/1887471296 shutdown_connections 2024-03-06T07:27:24.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.143+0000 7f1ed739a640 1 -- 172.21.15.83:0/1887471296 wait complete. 2024-03-06T07:27:24.156 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:27:24.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.259+0000 7fc7fe0c2640 1 Processor -- start 2024-03-06T07:27:24.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.259+0000 7fc7fe0c2640 1 -- start start 2024-03-06T07:27:24.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.260+0000 7fc7fe0c2640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fc7f80719e0 0x7fc7f80cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.260+0000 7fc7fe0c2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc7f806c9f0 0x7fc7f806ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.260+0000 7fc7fe0c2640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7f80b3320 con 0x7fc7f815a220 2024-03-06T07:27:24.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.260+0000 7fc7fe0c2640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7f80cf610 con 0x7fc7f806c9f0 2024-03-06T07:27:24.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.260+0000 7fc7fe0c2640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7f80728d0 con 0x7fc7f80719e0 2024-03-06T07:27:24.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.260+0000 7fc7f7fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc7f806c9f0 0x7fc7f806ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.260+0000 7fc7f77fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fc7f815a220 0x7fc7f80712d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57142/0 (socket says 172.21.15.83:57142) 2024-03-06T07:27:24.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.260+0000 7fc7f7fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc7f806c9f0 0x7fc7f806ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41610/0 (socket says 172.21.15.83:41610) 2024-03-06T07:27:24.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.260+0000 7fc7f77fe640 1 -- 172.21.15.83:0/2795638463 learned_addr learned my addr 172.21.15.83:0/2795638463 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:27:24.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.260+0000 7fc7f6ffd640 1 --2- 172.21.15.83:0/2795638463 >> v2:172.21.15.183:3300/0 conn(0x7fc7f80719e0 0x7fc7f80cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.260+0000 7fc7f7fff640 1 -- 172.21.15.83:0/2795638463 >> v2:172.21.15.183:3300/0 conn(0x7fc7f80719e0 msgr2=0x7fc7f80cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:24.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.260+0000 7fc7f7fff640 1 --2- 172.21.15.83:0/2795638463 >> v2:172.21.15.183:3300/0 conn(0x7fc7f80719e0 0x7fc7f80cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.260+0000 7fc7f7fff640 1 -- 172.21.15.83:0/2795638463 >> v1:172.21.15.83:6789/0 conn(0x7fc7f815a220 legacy=0x7fc7f80712d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.260+0000 7fc7f7fff640 1 -- 172.21.15.83:0/2795638463 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f80b3590 con 0x7fc7f806c9f0 2024-03-06T07:27:24.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.260+0000 7fc7f67fc640 1 -- 172.21.15.83:0/2795638463 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2641650032 0 0) 0x7fc7f80b3320 con 0x7fc7f815a220 2024-03-06T07:27:24.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.261+0000 7fc7f7fff640 1 --2- 172.21.15.83:0/2795638463 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7f806c9f0 0x7fc7f806ce40 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fc7f80d0980 tx=0x7fc7f002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ee2af1b2d227441 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.261+0000 7fc7f67fc640 1 -- 172.21.15.83:0/2795638463 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f0039070 con 0x7fc7f806c9f0 2024-03-06T07:27:24.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.261+0000 7fc7f67fc640 1 -- 172.21.15.83:0/2795638463 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7f002cbf0 con 0x7fc7f806c9f0 2024-03-06T07:27:24.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.261+0000 7fc7f67fc640 1 -- 172.21.15.83:0/2795638463 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f0034440 con 0x7fc7f806c9f0 2024-03-06T07:27:24.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.261+0000 7fc7fe0c2640 1 -- 172.21.15.83:0/2795638463 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7f806c9f0 msgr2=0x7fc7f806ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.261+0000 7fc7fe0c2640 1 --2- 172.21.15.83:0/2795638463 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7f806c9f0 0x7fc7f806ce40 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fc7f80d0980 tx=0x7fc7f002be40 comp rx=0 tx=0).stop 2024-03-06T07:27:24.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.261+0000 7fc7fe0c2640 1 -- 172.21.15.83:0/2795638463 shutdown_connections 2024-03-06T07:27:24.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.261+0000 7fc7fe0c2640 1 --2- 172.21.15.83:0/2795638463 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7f806c9f0 0x7fc7f806ce40 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.261+0000 7fc7fe0c2640 1 --2- 172.21.15.83:0/2795638463 >> v2:172.21.15.183:3300/0 conn(0x7fc7f80719e0 0x7fc7f80cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.261+0000 7fc7fe0c2640 1 -- 172.21.15.83:0/2795638463 >> 172.21.15.83:0/2795638463 conn(0x7fc7f806b190 msgr2=0x7fc7f806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.262+0000 7fc7fe0c2640 1 -- 172.21.15.83:0/2795638463 shutdown_connections 2024-03-06T07:27:24.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.262+0000 7fc7fe0c2640 1 -- 172.21.15.83:0/2795638463 wait complete. 2024-03-06T07:27:24.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.263+0000 7fc7fe0c2640 1 Processor -- start 2024-03-06T07:27:24.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.263+0000 7fc7fe0c2640 1 -- start start 2024-03-06T07:27:24.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.263+0000 7fc7fe0c2640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7f806c9f0 0x7fc7f814e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.263+0000 7fc7fe0c2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7f80719e0 0x7fc7f80b0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.263+0000 7fc7fe0c2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc7f815a220 0x7fc7f80b1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.263+0000 7fc7fe0c2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f8072530 con 0x7fc7f80719e0 2024-03-06T07:27:24.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.263+0000 7fc7fe0c2640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f8072030 con 0x7fc7f806c9f0 2024-03-06T07:27:24.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.263+0000 7fc7fe0c2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f80726b0 con 0x7fc7f815a220 2024-03-06T07:27:24.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.264+0000 7fc7f6ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7f80719e0 0x7fc7f80b0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.264+0000 7fc7f6ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7f80719e0 0x7fc7f80b0be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41624/0 (socket says 172.21.15.83:41624) 2024-03-06T07:27:24.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.264+0000 7fc7f6ffd640 1 -- 172.21.15.83:0/2286494211 learned_addr learned my addr 172.21.15.83:0/2286494211 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:24.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.264+0000 7fc7f77fe640 1 --2- 172.21.15.83:0/2286494211 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7f806c9f0 0x7fc7f814e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.264+0000 7fc7f7fff640 1 --2- 172.21.15.83:0/2286494211 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc7f815a220 0x7fc7f80b1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.264+0000 7fc7f6ffd640 1 -- 172.21.15.83:0/2286494211 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc7f815a220 msgr2=0x7fc7f80b1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.264+0000 7fc7f6ffd640 1 --2- 172.21.15.83:0/2286494211 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc7f815a220 0x7fc7f80b1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.264+0000 7fc7f6ffd640 1 -- 172.21.15.83:0/2286494211 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7f806c9f0 msgr2=0x7fc7f814e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.264+0000 7fc7f6ffd640 1 --2- 172.21.15.83:0/2286494211 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7f806c9f0 0x7fc7f814e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.264+0000 7fc7f6ffd640 1 -- 172.21.15.83:0/2286494211 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f8157970 con 0x7fc7f80719e0 2024-03-06T07:27:24.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.264+0000 7fc7f6ffd640 1 --2- 172.21.15.83:0/2286494211 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7f80719e0 0x7fc7f80b0be0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fc7ec00e910 tx=0x7fc7ec00edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.264+0000 7fc7dffff640 1 -- 172.21.15.83:0/2286494211 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7ec016070 con 0x7fc7f80719e0 2024-03-06T07:27:24.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.264+0000 7fc7dffff640 1 -- 172.21.15.83:0/2286494211 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7ec00cbe0 con 0x7fc7f80719e0 2024-03-06T07:27:24.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.264+0000 7fc7fe0c2640 1 -- 172.21.15.83:0/2286494211 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7f8157c50 con 0x7fc7f80719e0 2024-03-06T07:27:24.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.264+0000 7fc7dffff640 1 -- 172.21.15.83:0/2286494211 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7ec00b040 con 0x7fc7f80719e0 2024-03-06T07:27:24.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.264+0000 7fc7fe0c2640 1 -- 172.21.15.83:0/2286494211 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7f8158210 con 0x7fc7f80719e0 2024-03-06T07:27:24.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.265+0000 7fc7fe0c2640 1 -- 172.21.15.83:0/2286494211 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7f80732b0 con 0x7fc7f80719e0 2024-03-06T07:27:24.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.266+0000 7fc7dffff640 1 -- 172.21.15.83:0/2286494211 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc7ec019070 con 0x7fc7f80719e0 2024-03-06T07:27:24.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.266+0000 7fc7dffff640 1 --2- 172.21.15.83:0/2286494211 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc7e406b4d0 0x7fc7e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.266+0000 7fc7dffff640 1 -- 172.21.15.83:0/2286494211 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc7ec089f60 con 0x7fc7f80719e0 2024-03-06T07:27:24.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.266+0000 7fc7f77fe640 1 --2- 172.21.15.83:0/2286494211 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc7e406b4d0 0x7fc7e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.267+0000 7fc7f77fe640 1 --2- 172.21.15.83:0/2286494211 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc7e406b4d0 0x7fc7e406d990 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fc7e8007fd0 tx=0x7fc7e8007de0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.271+0000 7fc7dffff640 1 -- 172.21.15.83:0/2286494211 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc7ec058820 con 0x7fc7f80719e0 2024-03-06T07:27:24.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.515+0000 7fc7fe0c2640 1 -- 172.21.15.83:0/2286494211 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc7f8152520 con 0x7fc7f80719e0 2024-03-06T07:27:24.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.515+0000 7fc7dffff640 1 -- 172.21.15.83:0/2286494211 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v110) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc7ec05c8c0 con 0x7fc7f80719e0 2024-03-06T07:27:24.517 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 64 2024-03-06T07:27:24.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.517+0000 7fc7fe0c2640 1 -- 172.21.15.83:0/2286494211 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc7e406b4d0 msgr2=0x7fc7e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.517+0000 7fc7fe0c2640 1 --2- 172.21.15.83:0/2286494211 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc7e406b4d0 0x7fc7e406d990 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fc7e8007fd0 tx=0x7fc7e8007de0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.517+0000 7fc7fe0c2640 1 -- 172.21.15.83:0/2286494211 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7f80719e0 msgr2=0x7fc7f80b0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.517+0000 7fc7fe0c2640 1 --2- 172.21.15.83:0/2286494211 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7f80719e0 0x7fc7f80b0be0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fc7ec00e910 tx=0x7fc7ec00edd0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.517+0000 7fc7fe0c2640 1 -- 172.21.15.83:0/2286494211 shutdown_connections 2024-03-06T07:27:24.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.517+0000 7fc7fe0c2640 1 --2- 172.21.15.83:0/2286494211 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc7e406b4d0 0x7fc7e406d990 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.517+0000 7fc7fe0c2640 1 --2- 172.21.15.83:0/2286494211 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc7f815a220 0x7fc7f80b1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.517+0000 7fc7fe0c2640 1 --2- 172.21.15.83:0/2286494211 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7f80719e0 0x7fc7f80b0be0 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.517+0000 7fc7fe0c2640 1 --2- 172.21.15.83:0/2286494211 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7f806c9f0 0x7fc7f814e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.517+0000 7fc7fe0c2640 1 -- 172.21.15.83:0/2286494211 >> 172.21.15.83:0/2286494211 conn(0x7fc7f806b190 msgr2=0x7fc7f806cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.517+0000 7fc7fe0c2640 1 -- 172.21.15.83:0/2286494211 shutdown_connections 2024-03-06T07:27:24.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.517+0000 7fc7fe0c2640 1 -- 172.21.15.83:0/2286494211 wait complete. 2024-03-06T07:27:24.531 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:27:24.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.629+0000 7f170aff7640 1 Processor -- start 2024-03-06T07:27:24.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.630+0000 7f170aff7640 1 -- start start 2024-03-06T07:27:24.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.630+0000 7f170aff7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1704162970 0x7f17040d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.630+0000 7f170aff7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17040d22c0 con 0x7f1704161cf0 2024-03-06T07:27:24.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.630+0000 7f170aff7640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17040d25a0 con 0x7f17040b4460 2024-03-06T07:27:24.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.630+0000 7f170aff7640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1704161b00 con 0x7f1704162970 2024-03-06T07:27:24.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.630+0000 7f1703fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1704162970 0x7f17040d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.630+0000 7f1709594640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f1704161cf0 0x7f17040cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57146/0 (socket says 172.21.15.83:57146) 2024-03-06T07:27:24.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.630+0000 7f1703fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1704162970 0x7f17040d4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41638/0 (socket says 172.21.15.83:41638) 2024-03-06T07:27:24.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.630+0000 7f1709594640 1 -- 172.21.15.83:0/1779547900 learned_addr learned my addr 172.21.15.83:0/1779547900 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:27:24.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.631+0000 7f1703fff640 1 -- 172.21.15.83:0/1779547900 >> v1:172.21.15.183:6789/0 conn(0x7f17040b4460 legacy=0x7f17040b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.631+0000 7f1703fff640 1 -- 172.21.15.83:0/1779547900 >> v1:172.21.15.83:6789/0 conn(0x7f1704161cf0 legacy=0x7f17040cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.631+0000 7f1703fff640 1 -- 172.21.15.83:0/1779547900 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17040d2880 con 0x7f1704162970 2024-03-06T07:27:24.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.631+0000 7f17037fe640 1 -- 172.21.15.83:0/1779547900 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1384170602 0 0) 0x7f17040d22c0 con 0x7f1704161cf0 2024-03-06T07:27:24.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.632+0000 7f1703fff640 1 --2- 172.21.15.83:0/1779547900 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1704162970 0x7f17040d4800 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f16ec0097a0 tx=0x7f16ec02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5af01a2e26cec0d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.632+0000 7f17037fe640 1 -- 172.21.15.83:0/1779547900 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ec034070 con 0x7f1704162970 2024-03-06T07:27:24.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.632+0000 7f17037fe640 1 -- 172.21.15.83:0/1779547900 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16ec030610 con 0x7f1704162970 2024-03-06T07:27:24.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.632+0000 7f17037fe640 1 -- 172.21.15.83:0/1779547900 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ec0309e0 con 0x7f1704162970 2024-03-06T07:27:24.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.632+0000 7f170aff7640 1 -- 172.21.15.83:0/1779547900 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1704162970 msgr2=0x7f17040d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.632+0000 7f170aff7640 1 --2- 172.21.15.83:0/1779547900 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1704162970 0x7f17040d4800 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f16ec0097a0 tx=0x7f16ec02b960 comp rx=0 tx=0).stop 2024-03-06T07:27:24.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.632+0000 7f170aff7640 1 -- 172.21.15.83:0/1779547900 shutdown_connections 2024-03-06T07:27:24.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.632+0000 7f170aff7640 1 --2- 172.21.15.83:0/1779547900 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1704162970 0x7f17040d4800 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.632+0000 7f170aff7640 1 -- 172.21.15.83:0/1779547900 >> 172.21.15.83:0/1779547900 conn(0x7f17040d3ce0 msgr2=0x7f17040d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.632+0000 7f170aff7640 1 -- 172.21.15.83:0/1779547900 shutdown_connections 2024-03-06T07:27:24.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.632+0000 7f170aff7640 1 -- 172.21.15.83:0/1779547900 wait complete. 2024-03-06T07:27:24.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.633+0000 7f170aff7640 1 Processor -- start 2024-03-06T07:27:24.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.633+0000 7f170aff7640 1 -- start start 2024-03-06T07:27:24.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.634+0000 7f170aff7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17040b4460 0x7f17040c8710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.634+0000 7f170aff7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1704161cf0 0x7f17040cadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.634+0000 7f170aff7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1704162970 0x7f17040b1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.634+0000 7f170aff7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17040ce950 con 0x7f1704161cf0 2024-03-06T07:27:24.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.634+0000 7f170aff7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17040cecc0 con 0x7f17040b4460 2024-03-06T07:27:24.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.634+0000 7f170aff7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17040b4840 con 0x7f1704162970 2024-03-06T07:27:24.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.634+0000 7f1703fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1704161cf0 0x7f17040cadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.634+0000 7f1703fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1704161cf0 0x7f17040cadb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41646/0 (socket says 172.21.15.83:41646) 2024-03-06T07:27:24.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.634+0000 7f1703fff640 1 -- 172.21.15.83:0/14849728 learned_addr learned my addr 172.21.15.83:0/14849728 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:24.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.635+0000 7f1709594640 1 --2- 172.21.15.83:0/14849728 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1704162970 0x7f17040b1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.635+0000 7f1708d93640 1 --2- 172.21.15.83:0/14849728 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17040b4460 0x7f17040c8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.635+0000 7f1703fff640 1 -- 172.21.15.83:0/14849728 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1704162970 msgr2=0x7f17040b1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.635+0000 7f1703fff640 1 --2- 172.21.15.83:0/14849728 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1704162970 0x7f17040b1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.635+0000 7f1703fff640 1 -- 172.21.15.83:0/14849728 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17040b4460 msgr2=0x7f17040c8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.635+0000 7f1703fff640 1 --2- 172.21.15.83:0/14849728 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17040b4460 0x7f17040c8710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.635+0000 7f1703fff640 1 -- 172.21.15.83:0/14849728 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17040c9010 con 0x7f1704161cf0 2024-03-06T07:27:24.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.635+0000 7f1703fff640 1 --2- 172.21.15.83:0/14849728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1704161cf0 0x7f17040cadb0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f16ec0098c0 tx=0x7f16ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.635+0000 7f17017fa640 1 -- 172.21.15.83:0/14849728 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ec034040 con 0x7f1704161cf0 2024-03-06T07:27:24.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.635+0000 7f17017fa640 1 -- 172.21.15.83:0/14849728 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16ec039a10 con 0x7f1704161cf0 2024-03-06T07:27:24.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.635+0000 7f170aff7640 1 -- 172.21.15.83:0/14849728 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17040b8240 con 0x7f1704161cf0 2024-03-06T07:27:24.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.635+0000 7f17017fa640 1 -- 172.21.15.83:0/14849728 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ec039de0 con 0x7f1704161cf0 2024-03-06T07:27:24.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.635+0000 7f170aff7640 1 -- 172.21.15.83:0/14849728 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17040b8800 con 0x7f1704161cf0 2024-03-06T07:27:24.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.636+0000 7f170aff7640 1 -- 172.21.15.83:0/14849728 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f170404f330 con 0x7f1704161cf0 2024-03-06T07:27:24.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.636+0000 7f17017fa640 1 -- 172.21.15.83:0/14849728 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f16ec037070 con 0x7f1704161cf0 2024-03-06T07:27:24.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.637+0000 7f17017fa640 1 --2- 172.21.15.83:0/14849728 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f16e406b4d0 0x7f16e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.637+0000 7f1708d93640 1 --2- 172.21.15.83:0/14849728 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f16e406b4d0 0x7f16e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.637+0000 7f17017fa640 1 -- 172.21.15.83:0/14849728 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f16ec0ac340 con 0x7f1704161cf0 2024-03-06T07:27:24.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.637+0000 7f1708d93640 1 --2- 172.21.15.83:0/14849728 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f16e406b4d0 0x7f16e406d990 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f17040d1200 tx=0x7f16f4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.641+0000 7f17017fa640 1 -- 172.21.15.83:0/14849728 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f16ec07b1c0 con 0x7f1704161cf0 2024-03-06T07:27:24.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.825+0000 7f170aff7640 1 -- 172.21.15.83:0/14849728 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f170406b180 con 0x7f1704161cf0 2024-03-06T07:27:24.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.826+0000 7f17017fa640 1 -- 172.21.15.83:0/14849728 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v110) v1 ==== 80+0+0 (secure 0 0 0) 0x7f16ec07aac0 con 0x7f1704161cf0 2024-03-06T07:27:24.827 INFO:teuthology.orchestra.run.smithi083.stderr:listed 0 entries 2024-03-06T07:27:24.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.827+0000 7f170aff7640 1 -- 172.21.15.83:0/14849728 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f16e406b4d0 msgr2=0x7f16e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.827+0000 7f170aff7640 1 --2- 172.21.15.83:0/14849728 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f16e406b4d0 0x7f16e406d990 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f17040d1200 tx=0x7f16f4009040 comp rx=0 tx=0).stop 2024-03-06T07:27:24.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.827+0000 7f170aff7640 1 -- 172.21.15.83:0/14849728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1704161cf0 msgr2=0x7f17040cadb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.827+0000 7f170aff7640 1 --2- 172.21.15.83:0/14849728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1704161cf0 0x7f17040cadb0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f16ec0098c0 tx=0x7f16ec0046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.827+0000 7f170aff7640 1 -- 172.21.15.83:0/14849728 shutdown_connections 2024-03-06T07:27:24.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.827+0000 7f170aff7640 1 --2- 172.21.15.83:0/14849728 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f16e406b4d0 0x7f16e406d990 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.827+0000 7f170aff7640 1 --2- 172.21.15.83:0/14849728 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1704162970 0x7f17040b1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.827+0000 7f170aff7640 1 --2- 172.21.15.83:0/14849728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1704161cf0 0x7f17040cadb0 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.827+0000 7f170aff7640 1 --2- 172.21.15.83:0/14849728 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17040b4460 0x7f17040c8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.827+0000 7f170aff7640 1 -- 172.21.15.83:0/14849728 >> 172.21.15.83:0/14849728 conn(0x7f17040d3ce0 msgr2=0x7f17040d40c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.828+0000 7f170aff7640 1 -- 172.21.15.83:0/14849728 shutdown_connections 2024-03-06T07:27:24.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.828+0000 7f170aff7640 1 -- 172.21.15.83:0/14849728 wait complete. 2024-03-06T07:27:24.840 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:27:24.840 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:24.856 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:24.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.994+0000 7fdfca0b5640 1 Processor -- start 2024-03-06T07:27:24.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.995+0000 7fdfca0b5640 1 -- start start 2024-03-06T07:27:24.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.995+0000 7fdfca0b5640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdfc415bb20 0x7fdfc40d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.995+0000 7fdfca0b5640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fdfc40cd8c0 0x7fdfc40d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.995+0000 7fdfca0b5640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfc4161980 con 0x7fdfc40cfff0 2024-03-06T07:27:24.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.995+0000 7fdfca0b5640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfc4161c60 con 0x7fdfc40cd8c0 2024-03-06T07:27:24.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.995+0000 7fdfca0b5640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc4161f40 con 0x7fdfc415bb20 2024-03-06T07:27:24.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.995+0000 7fdfc3fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fdfc40cd8c0 0x7fdfc40d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.995+0000 7fdfc3fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fdfc40cd8c0 0x7fdfc40d0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41662/0 (socket says 172.21.15.83:41662) 2024-03-06T07:27:24.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.995+0000 7fdfc2ffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fdfc40cfff0 0x7fdfc40ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57156/0 (socket says 172.21.15.83:57156) 2024-03-06T07:27:24.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.996+0000 7fdfc3fff640 1 -- 172.21.15.83:0/399038931 learned_addr learned my addr 172.21.15.83:0/399038931 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:24.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.996+0000 7fdfc37fe640 1 --2- 172.21.15.83:0/399038931 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdfc415bb20 0x7fdfc40d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.996+0000 7fdfc3fff640 1 -- 172.21.15.83:0/399038931 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdfc415bb20 msgr2=0x7fdfc40d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:24.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.996+0000 7fdfc3fff640 1 --2- 172.21.15.83:0/399038931 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdfc415bb20 0x7fdfc40d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.996+0000 7fdfc3fff640 1 -- 172.21.15.83:0/399038931 >> v1:172.21.15.83:6789/0 conn(0x7fdfc40cfff0 legacy=0x7fdfc40ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.996+0000 7fdfc3fff640 1 -- 172.21.15.83:0/399038931 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc4162220 con 0x7fdfc40cd8c0 2024-03-06T07:27:24.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.996+0000 7fdfc27fc640 1 -- 172.21.15.83:0/399038931 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4245738820 0 0) 0x7fdfc4161980 con 0x7fdfc40cfff0 2024-03-06T07:27:24.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.996+0000 7fdfc37fe640 1 --2- 172.21.15.83:0/399038931 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdfc415bb20 0x7fdfc40d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:25.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.996+0000 7fdfc3fff640 1 --2- 172.21.15.83:0/399038931 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdfc40cd8c0 0x7fdfc40d0470 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc009e50 tx=0x7fdfbc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5f78f2127d10090 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.997+0000 7fdfc27fc640 1 -- 172.21.15.83:0/399038931 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfbc02ca70 con 0x7fdfc40cd8c0 2024-03-06T07:27:25.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.997+0000 7fdfc27fc640 1 -- 172.21.15.83:0/399038931 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfbc02cc10 con 0x7fdfc40cd8c0 2024-03-06T07:27:25.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.997+0000 7fdfc27fc640 1 -- 172.21.15.83:0/399038931 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfbc0343f0 con 0x7fdfc40cd8c0 2024-03-06T07:27:25.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.997+0000 7fdfca0b5640 1 -- 172.21.15.83:0/399038931 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdfc40cd8c0 msgr2=0x7fdfc40d0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.997+0000 7fdfca0b5640 1 --2- 172.21.15.83:0/399038931 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdfc40cd8c0 0x7fdfc40d0470 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc009e50 tx=0x7fdfbc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.997+0000 7fdfca0b5640 1 -- 172.21.15.83:0/399038931 shutdown_connections 2024-03-06T07:27:25.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.997+0000 7fdfca0b5640 1 --2- 172.21.15.83:0/399038931 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdfc40cd8c0 0x7fdfc40d0470 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.997+0000 7fdfca0b5640 1 --2- 172.21.15.83:0/399038931 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdfc415bb20 0x7fdfc40d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.997+0000 7fdfca0b5640 1 -- 172.21.15.83:0/399038931 >> 172.21.15.83:0/399038931 conn(0x7fdfc409a9f0 msgr2=0x7fdfc409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:25.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.997+0000 7fdfca0b5640 1 -- 172.21.15.83:0/399038931 shutdown_connections 2024-03-06T07:27:25.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.997+0000 7fdfca0b5640 1 -- 172.21.15.83:0/399038931 wait complete. 2024-03-06T07:27:25.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.999+0000 7fdfca0b5640 1 Processor -- start 2024-03-06T07:27:25.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:24.999+0000 7fdfca0b5640 1 -- start start 2024-03-06T07:27:25.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.000+0000 7fdfca0b5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdfc40cd8c0 0x7fdfc41f3370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.000+0000 7fdfca0b5640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdfc40cfff0 0x7fdfc415bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.000+0000 7fdfca0b5640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdfc415bb20 0x7fdfc406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.000+0000 7fdfca0b5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc40d24b0 con 0x7fdfc40cd8c0 2024-03-06T07:27:25.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.000+0000 7fdfca0b5640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc40d2330 con 0x7fdfc40cfff0 2024-03-06T07:27:25.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.000+0000 7fdfca0b5640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc40d1980 con 0x7fdfc415bb20 2024-03-06T07:27:25.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.000+0000 7fdfc37fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdfc40cd8c0 0x7fdfc41f3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.000+0000 7fdfc37fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdfc40cd8c0 0x7fdfc41f3370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41670/0 (socket says 172.21.15.83:41670) 2024-03-06T07:27:25.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.000+0000 7fdfc37fe640 1 -- 172.21.15.83:0/2349068101 learned_addr learned my addr 172.21.15.83:0/2349068101 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:25.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.000+0000 7fdfc2ffd640 1 --2- 172.21.15.83:0/2349068101 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdfc40cfff0 0x7fdfc415bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.000+0000 7fdfc3fff640 1 --2- 172.21.15.83:0/2349068101 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdfc415bb20 0x7fdfc406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.000+0000 7fdfc37fe640 1 -- 172.21.15.83:0/2349068101 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdfc415bb20 msgr2=0x7fdfc406d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.000+0000 7fdfc37fe640 1 --2- 172.21.15.83:0/2349068101 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdfc415bb20 0x7fdfc406d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.000+0000 7fdfc37fe640 1 -- 172.21.15.83:0/2349068101 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdfc40cfff0 msgr2=0x7fdfc415bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.000+0000 7fdfc37fe640 1 --2- 172.21.15.83:0/2349068101 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdfc40cfff0 0x7fdfc415bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.000+0000 7fdfc37fe640 1 -- 172.21.15.83:0/2349068101 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc415c600 con 0x7fdfc40cd8c0 2024-03-06T07:27:25.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.000+0000 7fdfc37fe640 1 --2- 172.21.15.83:0/2349068101 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdfc40cd8c0 0x7fdfc41f3370 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7fdfb400c820 tx=0x7fdfb400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.000+0000 7fdfabfff640 1 -- 172.21.15.83:0/2349068101 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfb4015070 con 0x7fdfc40cd8c0 2024-03-06T07:27:25.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.000+0000 7fdfabfff640 1 -- 172.21.15.83:0/2349068101 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfb40079d0 con 0x7fdfc40cd8c0 2024-03-06T07:27:25.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.000+0000 7fdfabfff640 1 -- 172.21.15.83:0/2349068101 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfb4007d80 con 0x7fdfc40cd8c0 2024-03-06T07:27:25.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.000+0000 7fdfca0b5640 1 -- 172.21.15.83:0/2349068101 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfc415c8e0 con 0x7fdfc40cd8c0 2024-03-06T07:27:25.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.000+0000 7fdfca0b5640 1 -- 172.21.15.83:0/2349068101 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfc415cea0 con 0x7fdfc40cd8c0 2024-03-06T07:27:25.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.002+0000 7fdfca0b5640 1 -- 172.21.15.83:0/2349068101 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfc415a400 con 0x7fdfc40cd8c0 2024-03-06T07:27:25.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.002+0000 7fdfabfff640 1 -- 172.21.15.83:0/2349068101 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdfb4018070 con 0x7fdfc40cd8c0 2024-03-06T07:27:25.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.002+0000 7fdfabfff640 1 --2- 172.21.15.83:0/2349068101 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdfb006b4d0 0x7fdfb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.003+0000 7fdfc2ffd640 1 --2- 172.21.15.83:0/2349068101 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdfb006b4d0 0x7fdfb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.003+0000 7fdfabfff640 1 -- 172.21.15.83:0/2349068101 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdfb408a050 con 0x7fdfc40cd8c0 2024-03-06T07:27:25.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.003+0000 7fdfc2ffd640 1 --2- 172.21.15.83:0/2349068101 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdfb006b4d0 0x7fdfb006d990 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fdfc40b2a90 tx=0x7fdfb8007b50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.007+0000 7fdfabfff640 1 -- 172.21.15.83:0/2349068101 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdfc415a400 con 0x7fdfc40cd8c0 2024-03-06T07:27:25.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.185+0000 7fdfca0b5640 1 -- 172.21.15.83:0/2349068101 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdfc404f330 con 0x7fdfc40cd8c0 2024-03-06T07:27:25.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.188+0000 7fdfabfff640 1 -- 172.21.15.83:0/2349068101 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+16639 (secure 0 0 0) 0x7fdfb4058f60 con 0x7fdfc40cd8c0 2024-03-06T07:27:25.191 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:25.191 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":110,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:27:12.146137+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:08.709967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:09.456518+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:25.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.190+0000 7fdfca0b5640 1 -- 172.21.15.83:0/2349068101 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdfb006b4d0 msgr2=0x7fdfb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.190+0000 7fdfca0b5640 1 --2- 172.21.15.83:0/2349068101 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdfb006b4d0 0x7fdfb006d990 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fdfc40b2a90 tx=0x7fdfb8007b50 comp rx=0 tx=0).stop 2024-03-06T07:27:25.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.191+0000 7fdfca0b5640 1 -- 172.21.15.83:0/2349068101 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdfc40cd8c0 msgr2=0x7fdfc41f3370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.191+0000 7fdfca0b5640 1 --2- 172.21.15.83:0/2349068101 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdfc40cd8c0 0x7fdfc41f3370 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7fdfb400c820 tx=0x7fdfb400cce0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.191+0000 7fdfca0b5640 1 -- 172.21.15.83:0/2349068101 shutdown_connections 2024-03-06T07:27:25.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.191+0000 7fdfca0b5640 1 --2- 172.21.15.83:0/2349068101 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdfb006b4d0 0x7fdfb006d990 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.191+0000 7fdfca0b5640 1 --2- 172.21.15.83:0/2349068101 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdfc415bb20 0x7fdfc406d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.191+0000 7fdfca0b5640 1 --2- 172.21.15.83:0/2349068101 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdfc40cfff0 0x7fdfc415bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.191+0000 7fdfca0b5640 1 --2- 172.21.15.83:0/2349068101 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdfc40cd8c0 0x7fdfc41f3370 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.191+0000 7fdfca0b5640 1 -- 172.21.15.83:0/2349068101 >> 172.21.15.83:0/2349068101 conn(0x7fdfc409a9f0 msgr2=0x7fdfc415a650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:25.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.191+0000 7fdfca0b5640 1 -- 172.21.15.83:0/2349068101 shutdown_connections 2024-03-06T07:27:25.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.191+0000 7fdfca0b5640 1 -- 172.21.15.83:0/2349068101 wait complete. 2024-03-06T07:27:25.207 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:08.709967+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:09.456518+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:27:25.208 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:27:25.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.309+0000 7f2cc7afb640 1 Processor -- start 2024-03-06T07:27:25.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.309+0000 7f2cc7afb640 1 -- start start 2024-03-06T07:27:25.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.310+0000 7f2cc7afb640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f2cc01614d0 0x7f2cc00d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.310+0000 7f2cc7afb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2cc00d24b0 0x7f2cc00b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.310+0000 7f2cc7afb640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc01618b0 con 0x7f2cc00d1910 2024-03-06T07:27:25.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.310+0000 7f2cc7afb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cc00cf400 con 0x7f2cc00d24b0 2024-03-06T07:27:25.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.310+0000 7f2cc7afb640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cc00cf6a0 con 0x7f2cc01614d0 2024-03-06T07:27:25.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.310+0000 7f2cc6098640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2cc00d24b0 0x7f2cc00b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.310+0000 7f2cc6098640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2cc00d24b0 0x7f2cc00b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41680/0 (socket says 172.21.15.83:41680) 2024-03-06T07:27:25.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.310+0000 7f2cc6098640 1 -- 172.21.15.83:0/472836990 learned_addr learned my addr 172.21.15.83:0/472836990 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:25.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.310+0000 7f2cc6098640 1 -- 172.21.15.83:0/472836990 >> v2:172.21.15.183:3300/0 conn(0x7f2cc01614d0 msgr2=0x7f2cc00d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:25.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.310+0000 7f2cc6098640 1 --2- 172.21.15.83:0/472836990 >> v2:172.21.15.183:3300/0 conn(0x7f2cc01614d0 0x7f2cc00d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.310+0000 7f2cc6098640 1 -- 172.21.15.83:0/472836990 >> v1:172.21.15.183:6789/0 conn(0x7f2cc00d1910 legacy=0x7f2cc00d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:25.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.310+0000 7f2cc6098640 1 -- 172.21.15.83:0/472836990 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc00cf980 con 0x7f2cc00d24b0 2024-03-06T07:27:25.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.311+0000 7f2cc6098640 1 --2- 172.21.15.83:0/472836990 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cc00d24b0 0x7f2cc00b0e30 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f2cc008c470 tx=0x7f2cb802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5c0adce42a602e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.311+0000 7f2cc4895640 1 -- 172.21.15.83:0/472836990 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb802ca80 con 0x7f2cc00d24b0 2024-03-06T07:27:25.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.311+0000 7f2cc4895640 1 -- 172.21.15.83:0/472836990 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cb802cc20 con 0x7f2cc00d24b0 2024-03-06T07:27:25.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.311+0000 7f2cc4895640 1 -- 172.21.15.83:0/472836990 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb80343f0 con 0x7f2cc00d24b0 2024-03-06T07:27:25.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.312+0000 7f2cc7afb640 1 -- 172.21.15.83:0/472836990 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cc00d24b0 msgr2=0x7f2cc00b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.312+0000 7f2cc7afb640 1 --2- 172.21.15.83:0/472836990 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cc00d24b0 0x7f2cc00b0e30 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f2cc008c470 tx=0x7f2cb802bee0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.312+0000 7f2cc7afb640 1 -- 172.21.15.83:0/472836990 shutdown_connections 2024-03-06T07:27:25.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.312+0000 7f2cc7afb640 1 --2- 172.21.15.83:0/472836990 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cc00d24b0 0x7f2cc00b0e30 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.312+0000 7f2cc7afb640 1 --2- 172.21.15.83:0/472836990 >> v2:172.21.15.183:3300/0 conn(0x7f2cc01614d0 0x7f2cc00d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.312+0000 7f2cc7afb640 1 -- 172.21.15.83:0/472836990 >> 172.21.15.83:0/472836990 conn(0x7f2cc00d3d40 msgr2=0x7f2cc00d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:25.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.312+0000 7f2cc7afb640 1 -- 172.21.15.83:0/472836990 shutdown_connections 2024-03-06T07:27:25.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.312+0000 7f2cc7afb640 1 -- 172.21.15.83:0/472836990 wait complete. 2024-03-06T07:27:25.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.313+0000 7f2cc7afb640 1 Processor -- start 2024-03-06T07:27:25.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.313+0000 7f2cc7afb640 1 -- start start 2024-03-06T07:27:25.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.314+0000 7f2cc7afb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cc00d1910 0x7f2cc014e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.314+0000 7f2cc7afb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2cc00d24b0 0x7f2cc00b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.314+0000 7f2cc7afb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2cc01614d0 0x7f2cc00c7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.314+0000 7f2cc7afb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc00b1370 con 0x7f2cc00d1910 2024-03-06T07:27:25.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.314+0000 7f2cc7afb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc00b0910 con 0x7f2cc00d24b0 2024-03-06T07:27:25.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.314+0000 7f2cc7afb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc00b14f0 con 0x7f2cc01614d0 2024-03-06T07:27:25.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.314+0000 7f2cc5897640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cc00d1910 0x7f2cc014e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.314+0000 7f2cc5897640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cc00d1910 0x7f2cc014e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41684/0 (socket says 172.21.15.83:41684) 2024-03-06T07:27:25.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.314+0000 7f2cc5897640 1 -- 172.21.15.83:0/3427824562 learned_addr learned my addr 172.21.15.83:0/3427824562 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:25.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.314+0000 7f2cc5897640 1 -- 172.21.15.83:0/3427824562 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2cc01614d0 msgr2=0x7f2cc00c7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.314+0000 7f2cc5897640 1 --2- 172.21.15.83:0/3427824562 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2cc01614d0 0x7f2cc00c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.314+0000 7f2cc5096640 1 --2- 172.21.15.83:0/3427824562 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2cc00d24b0 0x7f2cc00b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.314+0000 7f2cc5897640 1 -- 172.21.15.83:0/3427824562 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2cc00d24b0 msgr2=0x7f2cc00b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.314+0000 7f2cc5897640 1 --2- 172.21.15.83:0/3427824562 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2cc00d24b0 0x7f2cc00b8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.314+0000 7f2cc5897640 1 -- 172.21.15.83:0/3427824562 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc00c8310 con 0x7f2cc00d1910 2024-03-06T07:27:25.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.314+0000 7f2cc5897640 1 --2- 172.21.15.83:0/3427824562 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cc00d1910 0x7f2cc014e9a0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc00b440 tx=0x7f2cbc00b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.314+0000 7f2cb67fc640 1 -- 172.21.15.83:0/3427824562 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc015020 con 0x7f2cc00d1910 2024-03-06T07:27:25.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.314+0000 7f2cb67fc640 1 -- 172.21.15.83:0/3427824562 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cbc011890 con 0x7f2cc00d1910 2024-03-06T07:27:25.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.314+0000 7f2cc7afb640 1 -- 172.21.15.83:0/3427824562 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cc00c85f0 con 0x7f2cc00d1910 2024-03-06T07:27:25.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.314+0000 7f2cb67fc640 1 -- 172.21.15.83:0/3427824562 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc011c80 con 0x7f2cc00d1910 2024-03-06T07:27:25.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.314+0000 7f2cc7afb640 1 -- 172.21.15.83:0/3427824562 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cc00ccae0 con 0x7f2cc00d1910 2024-03-06T07:27:25.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.315+0000 7f2cc7afb640 1 -- 172.21.15.83:0/3427824562 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cc00d1450 con 0x7f2cc00d1910 2024-03-06T07:27:25.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.316+0000 7f2cb67fc640 1 -- 172.21.15.83:0/3427824562 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2cbc018020 con 0x7f2cc00d1910 2024-03-06T07:27:25.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.316+0000 7f2cb67fc640 1 --2- 172.21.15.83:0/3427824562 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2ca406b500 0x7f2ca406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.316+0000 7f2cb67fc640 1 -- 172.21.15.83:0/3427824562 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2cbc08a3c0 con 0x7f2cc00d1910 2024-03-06T07:27:25.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.316+0000 7f2cc5096640 1 --2- 172.21.15.83:0/3427824562 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2ca406b500 0x7f2ca406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.317+0000 7f2cc5096640 1 --2- 172.21.15.83:0/3427824562 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2ca406b500 0x7f2ca406d9c0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f2cc0151ac0 tx=0x7f2cb0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.321+0000 7f2cb67fc640 1 -- 172.21.15.83:0/3427824562 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2cc00d1450 con 0x7f2cc00d1910 2024-03-06T07:27:25.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.544+0000 7f2cc7afb640 1 -- 172.21.15.83:0/3427824562 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2cc00c87c0 con 0x7f2cc00d1910 2024-03-06T07:27:25.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.544+0000 7f2cb67fc640 1 -- 172.21.15.83:0/3427824562 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v110) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2cbc01f020 con 0x7f2cc00d1910 2024-03-06T07:27:25.546 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:27:25.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.546+0000 7f2cc7afb640 1 -- 172.21.15.83:0/3427824562 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2ca406b500 msgr2=0x7f2ca406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.546+0000 7f2cc7afb640 1 --2- 172.21.15.83:0/3427824562 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2ca406b500 0x7f2ca406d9c0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f2cc0151ac0 tx=0x7f2cb0009040 comp rx=0 tx=0).stop 2024-03-06T07:27:25.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.546+0000 7f2cc7afb640 1 -- 172.21.15.83:0/3427824562 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cc00d1910 msgr2=0x7f2cc014e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.546+0000 7f2cc7afb640 1 --2- 172.21.15.83:0/3427824562 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cc00d1910 0x7f2cc014e9a0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc00b440 tx=0x7f2cbc00b900 comp rx=0 tx=0).stop 2024-03-06T07:27:25.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.546+0000 7f2cc7afb640 1 -- 172.21.15.83:0/3427824562 shutdown_connections 2024-03-06T07:27:25.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.546+0000 7f2cc7afb640 1 --2- 172.21.15.83:0/3427824562 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2ca406b500 0x7f2ca406d9c0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.546+0000 7f2cc7afb640 1 --2- 172.21.15.83:0/3427824562 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2cc01614d0 0x7f2cc00c7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.546+0000 7f2cc7afb640 1 --2- 172.21.15.83:0/3427824562 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2cc00d24b0 0x7f2cc00b8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.546+0000 7f2cc7afb640 1 --2- 172.21.15.83:0/3427824562 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cc00d1910 0x7f2cc014e9a0 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.546+0000 7f2cc7afb640 1 -- 172.21.15.83:0/3427824562 >> 172.21.15.83:0/3427824562 conn(0x7f2cc00d3d40 msgr2=0x7f2cc00d1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:25.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.546+0000 7f2cc7afb640 1 -- 172.21.15.83:0/3427824562 shutdown_connections 2024-03-06T07:27:25.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.546+0000 7f2cc7afb640 1 -- 172.21.15.83:0/3427824562 wait complete. 2024-03-06T07:27:25.559 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:27:25.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.659+0000 7f83d44fb640 1 Processor -- start 2024-03-06T07:27:25.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.659+0000 7f83d44fb640 1 -- start start 2024-03-06T07:27:25.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.660+0000 7f83d44fb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f83cc1614d0 0x7f83cc0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.660+0000 7f83d44fb640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f83cc0d2480 0x7f83cc0b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.660+0000 7f83d44fb640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83cc1618b0 con 0x7f83cc0d1910 2024-03-06T07:27:25.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.660+0000 7f83d44fb640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83cc0cf400 con 0x7f83cc0d2480 2024-03-06T07:27:25.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.660+0000 7f83d44fb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83cc0cf6a0 con 0x7f83cc1614d0 2024-03-06T07:27:25.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.661+0000 7f83d2297640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f83cc1614d0 0x7f83cc0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.661+0000 7f83d2a98640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f83cc0d2480 0x7f83cc0b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.661+0000 7f83d1a96640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f83cc0d1910 0x7f83cc0d1d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:46008/0 (socket says 172.21.15.83:46008) 2024-03-06T07:27:25.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.661+0000 7f83d1a96640 1 -- 172.21.15.83:0/2129669166 learned_addr learned my addr 172.21.15.83:0/2129669166 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:27:25.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.661+0000 7f83d2a98640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f83cc0d2480 0x7f83cc0b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:42582/0 (socket says 172.21.15.83:42582) 2024-03-06T07:27:25.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.661+0000 7f83d2297640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f83cc1614d0 0x7f83cc0d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:43272/0 (socket says 172.21.15.83:43272) 2024-03-06T07:27:25.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.661+0000 7f83d1295640 1 -- 172.21.15.83:0/2129669166 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 294632265 0 0) 0x7f83cc1618b0 con 0x7f83cc0d1910 2024-03-06T07:27:25.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.661+0000 7f83d1295640 1 -- 172.21.15.83:0/2129669166 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83b8003520 con 0x7f83cc0d1910 2024-03-06T07:27:25.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.661+0000 7f83d2297640 1 -- 172.21.15.83:0/2129669166 >> v2:172.21.15.183:3300/0 conn(0x7f83cc0d2480 msgr2=0x7f83cc0b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:25.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.661+0000 7f83d2297640 1 --2- 172.21.15.83:0/2129669166 >> v2:172.21.15.183:3300/0 conn(0x7f83cc0d2480 0x7f83cc0b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.661+0000 7f83d2297640 1 -- 172.21.15.83:0/2129669166 >> v1:172.21.15.183:6789/0 conn(0x7f83cc0d1910 legacy=0x7f83cc0d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.661+0000 7f83d2297640 1 -- 172.21.15.83:0/2129669166 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83cc0cf980 con 0x7f83cc1614d0 2024-03-06T07:27:25.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.661+0000 7f83d2a98640 1 --2- 172.21.15.83:0/2129669166 >> v2:172.21.15.183:3300/0 conn(0x7f83cc0d2480 0x7f83cc0b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:25.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.662+0000 7f83d2297640 1 --2- 172.21.15.83:0/2129669166 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f83cc1614d0 0x7f83cc0d4860 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f83c00097a0 tx=0x7f83c002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b190f88683176db5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.662+0000 7f83d1295640 1 -- 172.21.15.83:0/2129669166 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83c0035020 con 0x7f83cc1614d0 2024-03-06T07:27:25.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.662+0000 7f83d1295640 1 -- 172.21.15.83:0/2129669166 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83c0031830 con 0x7f83cc1614d0 2024-03-06T07:27:25.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.662+0000 7f83d1295640 1 -- 172.21.15.83:0/2129669166 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83c0031be0 con 0x7f83cc1614d0 2024-03-06T07:27:25.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.663+0000 7f83d44fb640 1 -- 172.21.15.83:0/2129669166 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f83cc1614d0 msgr2=0x7f83cc0d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.663+0000 7f83d44fb640 1 --2- 172.21.15.83:0/2129669166 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f83cc1614d0 0x7f83cc0d4860 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f83c00097a0 tx=0x7f83c002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.663+0000 7f83d44fb640 1 -- 172.21.15.83:0/2129669166 shutdown_connections 2024-03-06T07:27:25.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.663+0000 7f83d44fb640 1 --2- 172.21.15.83:0/2129669166 >> v2:172.21.15.183:3300/0 conn(0x7f83cc0d2480 0x7f83cc0b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.663+0000 7f83d44fb640 1 --2- 172.21.15.83:0/2129669166 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f83cc1614d0 0x7f83cc0d4860 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.663+0000 7f83d44fb640 1 -- 172.21.15.83:0/2129669166 >> 172.21.15.83:0/2129669166 conn(0x7f83cc0d3d40 msgr2=0x7f83cc0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:25.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.663+0000 7f83d44fb640 1 -- 172.21.15.83:0/2129669166 shutdown_connections 2024-03-06T07:27:25.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.663+0000 7f83d44fb640 1 -- 172.21.15.83:0/2129669166 wait complete. 2024-03-06T07:27:25.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.664+0000 7f83d44fb640 1 Processor -- start 2024-03-06T07:27:25.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.664+0000 7f83d44fb640 1 -- start start 2024-03-06T07:27:25.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.665+0000 7f83d44fb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f83cc0d1910 0x7f83cc0cbac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.665+0000 7f83d44fb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83cc0d2480 0x7f83cc1f38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.665+0000 7f83d44fb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f83cc1614d0 0x7f83cc14f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.665+0000 7f83d44fb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83cc0b1370 con 0x7f83cc0d2480 2024-03-06T07:27:25.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.665+0000 7f83d44fb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83cc0b0910 con 0x7f83cc0d1910 2024-03-06T07:27:25.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.665+0000 7f83d44fb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83cc0b14f0 con 0x7f83cc1614d0 2024-03-06T07:27:25.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.665+0000 7f83d1a96640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83cc0d2480 0x7f83cc1f38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.665+0000 7f83d1a96640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83cc0d2480 0x7f83cc1f38a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41696/0 (socket says 172.21.15.83:41696) 2024-03-06T07:27:25.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.665+0000 7f83d1a96640 1 -- 172.21.15.83:0/3888289234 learned_addr learned my addr 172.21.15.83:0/3888289234 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:25.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.665+0000 7f83d2297640 1 --2- 172.21.15.83:0/3888289234 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f83cc0d1910 0x7f83cc0cbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.665+0000 7f83d2a98640 1 --2- 172.21.15.83:0/3888289234 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f83cc1614d0 0x7f83cc14f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.665+0000 7f83d1a96640 1 -- 172.21.15.83:0/3888289234 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f83cc1614d0 msgr2=0x7f83cc14f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.665+0000 7f83d1a96640 1 --2- 172.21.15.83:0/3888289234 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f83cc1614d0 0x7f83cc14f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.665+0000 7f83d1a96640 1 -- 172.21.15.83:0/3888289234 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f83cc0d1910 msgr2=0x7f83cc0cbac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.665+0000 7f83d1a96640 1 --2- 172.21.15.83:0/3888289234 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f83cc0d1910 0x7f83cc0cbac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.665+0000 7f83d1a96640 1 -- 172.21.15.83:0/3888289234 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83cc0b6b20 con 0x7f83cc0d2480 2024-03-06T07:27:25.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.665+0000 7f83d1a96640 1 --2- 172.21.15.83:0/3888289234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83cc0d2480 0x7f83cc1f38a0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f83c800c430 tx=0x7f83c800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.665+0000 7f83beffd640 1 -- 172.21.15.83:0/3888289234 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83c8017070 con 0x7f83cc0d2480 2024-03-06T07:27:25.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.665+0000 7f83beffd640 1 -- 172.21.15.83:0/3888289234 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83c8005ce0 con 0x7f83cc0d2480 2024-03-06T07:27:25.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.665+0000 7f83d44fb640 1 -- 172.21.15.83:0/3888289234 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83cc0b6e00 con 0x7f83cc0d2480 2024-03-06T07:27:25.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.665+0000 7f83beffd640 1 -- 172.21.15.83:0/3888289234 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83c80072f0 con 0x7f83cc0d2480 2024-03-06T07:27:25.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.665+0000 7f83d44fb640 1 -- 172.21.15.83:0/3888289234 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83cc0b73c0 con 0x7f83cc0d2480 2024-03-06T07:27:25.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.666+0000 7f83d44fb640 1 -- 172.21.15.83:0/3888289234 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83cc0b75c0 con 0x7f83cc0d2480 2024-03-06T07:27:25.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.667+0000 7f83beffd640 1 -- 172.21.15.83:0/3888289234 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f83c801a070 con 0x7f83cc0d2480 2024-03-06T07:27:25.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.667+0000 7f83beffd640 1 --2- 172.21.15.83:0/3888289234 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f83b006b4d0 0x7f83b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.667+0000 7f83beffd640 1 -- 172.21.15.83:0/3888289234 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f83c8089620 con 0x7f83cc0d2480 2024-03-06T07:27:25.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.667+0000 7f83d2297640 1 --2- 172.21.15.83:0/3888289234 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f83b006b4d0 0x7f83b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.668+0000 7f83d2297640 1 --2- 172.21.15.83:0/3888289234 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f83b006b4d0 0x7f83b006d990 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f83c0036000 tx=0x7f83c003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.676+0000 7f83beffd640 1 -- 172.21.15.83:0/3888289234 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f83cc0b75c0 con 0x7f83cc0d2480 2024-03-06T07:27:25.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.846+0000 7f83d44fb640 1 -- 172.21.15.83:0/3888289234 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f83cc068ff0 con 0x7f83cc0d2480 2024-03-06T07:27:25.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.847+0000 7f83beffd640 1 -- 172.21.15.83:0/3888289234 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v110) v1 ==== 104+0+11 (secure 0 0 0) 0x7f83c8058530 con 0x7f83cc0d2480 2024-03-06T07:27:25.849 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-03-06T07:27:25.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.849+0000 7f83d44fb640 1 -- 172.21.15.83:0/3888289234 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f83b006b4d0 msgr2=0x7f83b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.849+0000 7f83d44fb640 1 --2- 172.21.15.83:0/3888289234 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f83b006b4d0 0x7f83b006d990 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f83c0036000 tx=0x7f83c003c040 comp rx=0 tx=0).stop 2024-03-06T07:27:25.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.849+0000 7f83d44fb640 1 -- 172.21.15.83:0/3888289234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83cc0d2480 msgr2=0x7f83cc1f38a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.849+0000 7f83d44fb640 1 --2- 172.21.15.83:0/3888289234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83cc0d2480 0x7f83cc1f38a0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f83c800c430 tx=0x7f83c800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.849+0000 7f83d44fb640 1 -- 172.21.15.83:0/3888289234 shutdown_connections 2024-03-06T07:27:25.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.849+0000 7f83d44fb640 1 --2- 172.21.15.83:0/3888289234 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f83b006b4d0 0x7f83b006d990 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.849+0000 7f83d44fb640 1 --2- 172.21.15.83:0/3888289234 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f83cc1614d0 0x7f83cc14f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.849+0000 7f83d44fb640 1 --2- 172.21.15.83:0/3888289234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83cc0d2480 0x7f83cc1f38a0 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.849+0000 7f83d44fb640 1 --2- 172.21.15.83:0/3888289234 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f83cc0d1910 0x7f83cc0cbac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.849+0000 7f83d44fb640 1 -- 172.21.15.83:0/3888289234 >> 172.21.15.83:0/3888289234 conn(0x7f83cc0d3d40 msgr2=0x7f83cc0cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:25.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.849+0000 7f83d44fb640 1 -- 172.21.15.83:0/3888289234 shutdown_connections 2024-03-06T07:27:25.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.849+0000 7f83d44fb640 1 -- 172.21.15.83:0/3888289234 wait complete. 2024-03-06T07:27:25.862 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:27:25.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.983+0000 7f371e313640 1 Processor -- start 2024-03-06T07:27:25.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.983+0000 7f371e313640 1 -- start start 2024-03-06T07:27:25.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.984+0000 7f371e313640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f37181614d0 0x7f37180d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.984+0000 7f371e313640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f37180d1910 0x7f37180d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.984+0000 7f371e313640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f37180d22e0 0x7f37180d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.984+0000 7f371e313640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37180d2750 con 0x7f37181614d0 2024-03-06T07:27:25.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.984+0000 7f371e313640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37181618b0 con 0x7f37180d1910 2024-03-06T07:27:25.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.984+0000 7f371e313640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37180cf400 con 0x7f37180d22e0 2024-03-06T07:27:25.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.984+0000 7f3717fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f37181614d0 0x7f37180d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.984+0000 7f3717fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f37181614d0 0x7f37180d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41708/0 (socket says 172.21.15.83:41708) 2024-03-06T07:27:25.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.984+0000 7f3717fff640 1 -- 172.21.15.83:0/834123574 learned_addr learned my addr 172.21.15.83:0/834123574 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:25.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.985+0000 7f37177fe640 1 --2- 172.21.15.83:0/834123574 >> v2:172.21.15.183:3300/0 conn(0x7f37180d1910 0x7f37180d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.985+0000 7f3717fff640 1 -- 172.21.15.83:0/834123574 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f37180d22e0 msgr2=0x7f37180d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:25.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.985+0000 7f3717fff640 1 --2- 172.21.15.83:0/834123574 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f37180d22e0 0x7f37180d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.985+0000 7f3717fff640 1 -- 172.21.15.83:0/834123574 >> v2:172.21.15.183:3300/0 conn(0x7f37180d1910 msgr2=0x7f37180d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:25.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.985+0000 7f3717fff640 1 --2- 172.21.15.83:0/834123574 >> v2:172.21.15.183:3300/0 conn(0x7f37180d1910 0x7f37180d1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.985+0000 7f3717fff640 1 -- 172.21.15.83:0/834123574 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37180cf6a0 con 0x7f37181614d0 2024-03-06T07:27:25.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.985+0000 7f37177fe640 1 --2- 172.21.15.83:0/834123574 >> v2:172.21.15.183:3300/0 conn(0x7f37180d1910 0x7f37180d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:25.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.986+0000 7f3717fff640 1 --2- 172.21.15.83:0/834123574 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37181614d0 0x7f37180d4860 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f37180ce550 tx=0x7f370802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcd8c48a033f4362 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.986+0000 7f3716ffd640 1 -- 172.21.15.83:0/834123574 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3708035020 con 0x7f37181614d0 2024-03-06T07:27:25.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.986+0000 7f3716ffd640 1 -- 172.21.15.83:0/834123574 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3708031870 con 0x7f37181614d0 2024-03-06T07:27:25.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.986+0000 7f3716ffd640 1 -- 172.21.15.83:0/834123574 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3708031c40 con 0x7f37181614d0 2024-03-06T07:27:25.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.986+0000 7f371e313640 1 -- 172.21.15.83:0/834123574 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37181614d0 msgr2=0x7f37180d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.986+0000 7f371e313640 1 --2- 172.21.15.83:0/834123574 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37181614d0 0x7f37180d4860 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f37180ce550 tx=0x7f370802b910 comp rx=0 tx=0).stop 2024-03-06T07:27:25.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.986+0000 7f371e313640 1 -- 172.21.15.83:0/834123574 shutdown_connections 2024-03-06T07:27:25.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.986+0000 7f371e313640 1 --2- 172.21.15.83:0/834123574 >> v2:172.21.15.183:3300/0 conn(0x7f37180d1910 0x7f37180d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.986+0000 7f371e313640 1 --2- 172.21.15.83:0/834123574 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f37180d22e0 0x7f37180d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.986+0000 7f371e313640 1 --2- 172.21.15.83:0/834123574 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37181614d0 0x7f37180d4860 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.986+0000 7f371e313640 1 -- 172.21.15.83:0/834123574 >> 172.21.15.83:0/834123574 conn(0x7f37180d3d40 msgr2=0x7f37180d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:25.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.987+0000 7f371e313640 1 -- 172.21.15.83:0/834123574 shutdown_connections 2024-03-06T07:27:25.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.987+0000 7f371e313640 1 -- 172.21.15.83:0/834123574 wait complete. 2024-03-06T07:27:25.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.988+0000 7f371e313640 1 Processor -- start 2024-03-06T07:27:25.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.988+0000 7f371e313640 1 -- start start 2024-03-06T07:27:25.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.988+0000 7f371e313640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37180d1910 0x7f37180b7360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.988+0000 7f371e313640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f37180d22e0 0x7f3718159950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.988+0000 7f371e313640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f37181614d0 0x7f37180b42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.988+0000 7f371e313640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37180b0910 con 0x7f37180d1910 2024-03-06T07:27:25.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.988+0000 7f371e313640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37180cfc60 con 0x7f37180d22e0 2024-03-06T07:27:25.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.988+0000 7f371e313640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37180d0be0 con 0x7f37181614d0 2024-03-06T07:27:25.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.989+0000 7f3717fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37180d1910 0x7f37180b7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.989+0000 7f3717fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37180d1910 0x7f37180b7360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41718/0 (socket says 172.21.15.83:41718) 2024-03-06T07:27:25.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.989+0000 7f3717fff640 1 -- 172.21.15.83:0/3470176118 learned_addr learned my addr 172.21.15.83:0/3470176118 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:25.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.989+0000 7f37177fe640 1 --2- 172.21.15.83:0/3470176118 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f37180d22e0 0x7f3718159950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.989+0000 7f371c8b0640 1 --2- 172.21.15.83:0/3470176118 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f37181614d0 0x7f37180b42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.989+0000 7f3717fff640 1 -- 172.21.15.83:0/3470176118 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f37181614d0 msgr2=0x7f37180b42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.989+0000 7f3717fff640 1 --2- 172.21.15.83:0/3470176118 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f37181614d0 0x7f37180b42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.989+0000 7f3717fff640 1 -- 172.21.15.83:0/3470176118 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f37180d22e0 msgr2=0x7f3718159950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.989+0000 7f3717fff640 1 --2- 172.21.15.83:0/3470176118 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f37180d22e0 0x7f3718159950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.989+0000 7f3717fff640 1 -- 172.21.15.83:0/3470176118 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37180b95f0 con 0x7f37180d1910 2024-03-06T07:27:25.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.989+0000 7f37177fe640 1 --2- 172.21.15.83:0/3470176118 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f37180d22e0 0x7f3718159950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:25.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.989+0000 7f371c8b0640 1 --2- 172.21.15.83:0/3470176118 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f37181614d0 0x7f37180b42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:25.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.989+0000 7f3717fff640 1 --2- 172.21.15.83:0/3470176118 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37180d1910 0x7f37180b7360 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f3708000c00 tx=0x7f37080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.989+0000 7f3714ff9640 1 -- 172.21.15.83:0/3470176118 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3708040070 con 0x7f37180d1910 2024-03-06T07:27:25.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.989+0000 7f3714ff9640 1 -- 172.21.15.83:0/3470176118 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f370803a420 con 0x7f37180d1910 2024-03-06T07:27:25.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.989+0000 7f3714ff9640 1 -- 172.21.15.83:0/3470176118 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3708035040 con 0x7f37180d1910 2024-03-06T07:27:25.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.989+0000 7f371e313640 1 -- 172.21.15.83:0/3470176118 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f371815a710 con 0x7f37180d1910 2024-03-06T07:27:25.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.989+0000 7f371e313640 1 -- 172.21.15.83:0/3470176118 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f371815acd0 con 0x7f37180d1910 2024-03-06T07:27:25.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.990+0000 7f371e313640 1 -- 172.21.15.83:0/3470176118 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37180b9960 con 0x7f37180d1910 2024-03-06T07:27:25.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.991+0000 7f3714ff9640 1 -- 172.21.15.83:0/3470176118 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f370804b060 con 0x7f37180d1910 2024-03-06T07:27:25.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.991+0000 7f3714ff9640 1 --2- 172.21.15.83:0/3470176118 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f370406b500 0x7f370406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.992+0000 7f3714ff9640 1 -- 172.21.15.83:0/3470176118 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f37080ac090 con 0x7f37180d1910 2024-03-06T07:27:25.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.992+0000 7f37177fe640 1 --2- 172.21.15.83:0/3470176118 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f370406b500 0x7f370406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.992+0000 7f37177fe640 1 --2- 172.21.15.83:0/3470176118 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f370406b500 0x7f370406d9c0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f370c006fd0 tx=0x7f370c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:25.996+0000 7f3714ff9640 1 -- 172.21.15.83:0/3470176118 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f370807afa0 con 0x7f37180d1910 2024-03-06T07:27:26.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.179+0000 7f371e313640 1 -- 172.21.15.83:0/3470176118 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f37180d0590 con 0x7f37180d1910 2024-03-06T07:27:26.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.179+0000 7f3714ff9640 1 -- 172.21.15.83:0/3470176118 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v110) v1 ==== 100+0+11 (secure 0 0 0) 0x7f370807a8a0 con 0x7f37180d1910 2024-03-06T07:27:26.181 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 64 2024-03-06T07:27:26.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.181+0000 7f371e313640 1 -- 172.21.15.83:0/3470176118 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f370406b500 msgr2=0x7f370406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.181+0000 7f371e313640 1 --2- 172.21.15.83:0/3470176118 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f370406b500 0x7f370406d9c0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f370c006fd0 tx=0x7f370c008040 comp rx=0 tx=0).stop 2024-03-06T07:27:26.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.181+0000 7f371e313640 1 -- 172.21.15.83:0/3470176118 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37180d1910 msgr2=0x7f37180b7360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.181+0000 7f371e313640 1 --2- 172.21.15.83:0/3470176118 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37180d1910 0x7f37180b7360 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f3708000c00 tx=0x7f37080046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.181+0000 7f371e313640 1 -- 172.21.15.83:0/3470176118 shutdown_connections 2024-03-06T07:27:26.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.181+0000 7f371e313640 1 --2- 172.21.15.83:0/3470176118 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f370406b500 0x7f370406d9c0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.181+0000 7f371e313640 1 --2- 172.21.15.83:0/3470176118 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f37181614d0 0x7f37180b42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.181+0000 7f371e313640 1 --2- 172.21.15.83:0/3470176118 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f37180d22e0 0x7f3718159950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.181+0000 7f371e313640 1 --2- 172.21.15.83:0/3470176118 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37180d1910 0x7f37180b7360 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.181+0000 7f371e313640 1 -- 172.21.15.83:0/3470176118 >> 172.21.15.83:0/3470176118 conn(0x7f37180d3d40 msgr2=0x7f37180d2920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:26.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.181+0000 7f371e313640 1 -- 172.21.15.83:0/3470176118 shutdown_connections 2024-03-06T07:27:26.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.181+0000 7f371e313640 1 -- 172.21.15.83:0/3470176118 wait complete. 2024-03-06T07:27:26.194 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:27:26.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.294+0000 7f9249e9a640 1 Processor -- start 2024-03-06T07:27:26.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.294+0000 7f9249e9a640 1 -- start start 2024-03-06T07:27:26.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.294+0000 7f9249e9a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f92440b4460 0x7f92440d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.294+0000 7f9249e9a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f9244162970 0x7f92440b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.294+0000 7f9249e9a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92440d22c0 con 0x7f9244161cf0 2024-03-06T07:27:26.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.294+0000 7f9249e9a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9244161b00 con 0x7f92440b4460 2024-03-06T07:27:26.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.294+0000 7f9249e9a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92440cecc0 con 0x7f9244162970 2024-03-06T07:27:26.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.294+0000 7f92437fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f92440b4460 0x7f92440d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.295+0000 7f9243fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f9244161cf0 0x7f92440cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57168/0 (socket says 172.21.15.83:57168) 2024-03-06T07:27:26.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.295+0000 7f9243fff640 1 -- 172.21.15.83:0/2105861560 learned_addr learned my addr 172.21.15.83:0/2105861560 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:27:26.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.295+0000 7f923bfff640 1 --2- 172.21.15.83:0/2105861560 >> v2:172.21.15.183:3300/0 conn(0x7f9244162970 0x7f92440b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.295+0000 7f92437fe640 1 -- 172.21.15.83:0/2105861560 >> v2:172.21.15.183:3300/0 conn(0x7f9244162970 msgr2=0x7f92440b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:26.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.295+0000 7f92437fe640 1 --2- 172.21.15.83:0/2105861560 >> v2:172.21.15.183:3300/0 conn(0x7f9244162970 0x7f92440b3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.295+0000 7f92437fe640 1 -- 172.21.15.83:0/2105861560 >> v1:172.21.15.83:6789/0 conn(0x7f9244161cf0 legacy=0x7f92440cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.295+0000 7f92437fe640 1 -- 172.21.15.83:0/2105861560 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92440d25a0 con 0x7f92440b4460 2024-03-06T07:27:26.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.295+0000 7f9242ffd640 1 -- 172.21.15.83:0/2105861560 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 559210841 0 0) 0x7f92440d22c0 con 0x7f9244161cf0 2024-03-06T07:27:26.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.296+0000 7f92437fe640 1 --2- 172.21.15.83:0/2105861560 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92440b4460 0x7f92440d4800 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f92440d17e0 tx=0x7f923402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6639599c65706c31 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.296+0000 7f9242ffd640 1 -- 172.21.15.83:0/2105861560 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9234035020 con 0x7f92440b4460 2024-03-06T07:27:26.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.296+0000 7f9242ffd640 1 -- 172.21.15.83:0/2105861560 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9234031830 con 0x7f92440b4460 2024-03-06T07:27:26.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.296+0000 7f9242ffd640 1 -- 172.21.15.83:0/2105861560 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9234031c00 con 0x7f92440b4460 2024-03-06T07:27:26.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.296+0000 7f9249e9a640 1 -- 172.21.15.83:0/2105861560 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92440b4460 msgr2=0x7f92440d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.296+0000 7f9249e9a640 1 --2- 172.21.15.83:0/2105861560 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92440b4460 0x7f92440d4800 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f92440d17e0 tx=0x7f923402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.296+0000 7f9249e9a640 1 -- 172.21.15.83:0/2105861560 shutdown_connections 2024-03-06T07:27:26.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.296+0000 7f9249e9a640 1 --2- 172.21.15.83:0/2105861560 >> v2:172.21.15.183:3300/0 conn(0x7f9244162970 0x7f92440b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.296+0000 7f9249e9a640 1 --2- 172.21.15.83:0/2105861560 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92440b4460 0x7f92440d4800 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.296+0000 7f9249e9a640 1 -- 172.21.15.83:0/2105861560 >> 172.21.15.83:0/2105861560 conn(0x7f92440d3ce0 msgr2=0x7f92440d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:26.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.296+0000 7f9249e9a640 1 -- 172.21.15.83:0/2105861560 shutdown_connections 2024-03-06T07:27:26.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.296+0000 7f9249e9a640 1 -- 172.21.15.83:0/2105861560 wait complete. 2024-03-06T07:27:26.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.297+0000 7f9249e9a640 1 Processor -- start 2024-03-06T07:27:26.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.298+0000 7f9249e9a640 1 -- start start 2024-03-06T07:27:26.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.298+0000 7f9249e9a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92440b4460 0x7f92440cc450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.298+0000 7f9249e9a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9244161cf0 0x7f92440c8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.298+0000 7f9249e9a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9244162970 0x7f92440b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.298+0000 7f9249e9a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92440ce950 con 0x7f9244162970 2024-03-06T07:27:26.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.298+0000 7f9249e9a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92440d2b60 con 0x7f9244161cf0 2024-03-06T07:27:26.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.298+0000 7f9249e9a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92440b3ac0 con 0x7f92440b4460 2024-03-06T07:27:26.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.298+0000 7f9243fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9244162970 0x7f92440b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.298+0000 7f9243fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9244162970 0x7f92440b2170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41736/0 (socket says 172.21.15.83:41736) 2024-03-06T07:27:26.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.298+0000 7f9243fff640 1 -- 172.21.15.83:0/4264649286 learned_addr learned my addr 172.21.15.83:0/4264649286 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:26.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.299+0000 7f923bfff640 1 --2- 172.21.15.83:0/4264649286 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9244161cf0 0x7f92440c8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.299+0000 7f92437fe640 1 --2- 172.21.15.83:0/4264649286 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92440b4460 0x7f92440cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.299+0000 7f9243fff640 1 -- 172.21.15.83:0/4264649286 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92440b4460 msgr2=0x7f92440cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.299+0000 7f9243fff640 1 --2- 172.21.15.83:0/4264649286 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92440b4460 0x7f92440cc450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.299+0000 7f9243fff640 1 -- 172.21.15.83:0/4264649286 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9244161cf0 msgr2=0x7f92440c8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.299+0000 7f9243fff640 1 --2- 172.21.15.83:0/4264649286 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9244161cf0 0x7f92440c8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.299+0000 7f9243fff640 1 -- 172.21.15.83:0/4264649286 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92440c9610 con 0x7f9244162970 2024-03-06T07:27:26.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.299+0000 7f9243fff640 1 --2- 172.21.15.83:0/4264649286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9244162970 0x7f92440b2170 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f923c00c7a0 tx=0x7f923c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.299+0000 7f9240ff9640 1 -- 172.21.15.83:0/4264649286 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f923c018070 con 0x7f9244162970 2024-03-06T07:27:26.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.299+0000 7f9240ff9640 1 -- 172.21.15.83:0/4264649286 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f923c00a640 con 0x7f9244162970 2024-03-06T07:27:26.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.299+0000 7f9249e9a640 1 -- 172.21.15.83:0/4264649286 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92440c98f0 con 0x7f9244162970 2024-03-06T07:27:26.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.299+0000 7f9240ff9640 1 -- 172.21.15.83:0/4264649286 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f923c00aab0 con 0x7f9244162970 2024-03-06T07:27:26.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.299+0000 7f9249e9a640 1 -- 172.21.15.83:0/4264649286 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92440c9eb0 con 0x7f9244162970 2024-03-06T07:27:26.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.300+0000 7f9249e9a640 1 -- 172.21.15.83:0/4264649286 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f924404f330 con 0x7f9244162970 2024-03-06T07:27:26.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.300+0000 7f9240ff9640 1 -- 172.21.15.83:0/4264649286 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f923c01b070 con 0x7f9244162970 2024-03-06T07:27:26.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.301+0000 7f9240ff9640 1 --2- 172.21.15.83:0/4264649286 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f922406b4d0 0x7f922406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.301+0000 7f92437fe640 1 --2- 172.21.15.83:0/4264649286 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f922406b4d0 0x7f922406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.301+0000 7f9240ff9640 1 -- 172.21.15.83:0/4264649286 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f923c089840 con 0x7f9244162970 2024-03-06T07:27:26.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.301+0000 7f92437fe640 1 --2- 172.21.15.83:0/4264649286 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f922406b4d0 0x7f922406d990 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f9234036000 tx=0x7f923403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.305+0000 7f9240ff9640 1 -- 172.21.15.83:0/4264649286 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f923c0586c0 con 0x7f9244162970 2024-03-06T07:27:26.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.493+0000 7f9249e9a640 1 -- 172.21.15.83:0/4264649286 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f924406b180 con 0x7f9244162970 2024-03-06T07:27:26.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.493+0000 7f9240ff9640 1 -- 172.21.15.83:0/4264649286 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v110) v1 ==== 80+0+0 (secure 0 0 0) 0x7f923c057fc0 con 0x7f9244162970 2024-03-06T07:27:26.495 INFO:teuthology.orchestra.run.smithi083.stderr:listed 0 entries 2024-03-06T07:27:26.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.495+0000 7f9249e9a640 1 -- 172.21.15.83:0/4264649286 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f922406b4d0 msgr2=0x7f922406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.495+0000 7f9249e9a640 1 --2- 172.21.15.83:0/4264649286 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f922406b4d0 0x7f922406d990 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f9234036000 tx=0x7f923403c040 comp rx=0 tx=0).stop 2024-03-06T07:27:26.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.495+0000 7f9249e9a640 1 -- 172.21.15.83:0/4264649286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9244162970 msgr2=0x7f92440b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.495+0000 7f9249e9a640 1 --2- 172.21.15.83:0/4264649286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9244162970 0x7f92440b2170 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f923c00c7a0 tx=0x7f923c00cc60 comp rx=0 tx=0).stop 2024-03-06T07:27:26.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.495+0000 7f9249e9a640 1 -- 172.21.15.83:0/4264649286 shutdown_connections 2024-03-06T07:27:26.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.495+0000 7f9249e9a640 1 --2- 172.21.15.83:0/4264649286 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f922406b4d0 0x7f922406d990 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.495+0000 7f9249e9a640 1 --2- 172.21.15.83:0/4264649286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9244162970 0x7f92440b2170 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.495+0000 7f9249e9a640 1 --2- 172.21.15.83:0/4264649286 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9244161cf0 0x7f92440c8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.495+0000 7f9249e9a640 1 --2- 172.21.15.83:0/4264649286 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92440b4460 0x7f92440cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.495+0000 7f9249e9a640 1 -- 172.21.15.83:0/4264649286 >> 172.21.15.83:0/4264649286 conn(0x7f92440d3ce0 msgr2=0x7f92440d0120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:26.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.495+0000 7f9249e9a640 1 -- 172.21.15.83:0/4264649286 shutdown_connections 2024-03-06T07:27:26.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:26.495+0000 7f9249e9a640 1 -- 172.21.15.83:0/4264649286 wait complete. 2024-03-06T07:27:26.508 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi083.front.sepia.ceph.com... 2024-03-06T07:27:26.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:26.508 DEBUG:teuthology.orchestra.run.smithi083:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:26.533 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:ceph-fuse[73136]: fuse finished with error 0 and tester_r 0 2024-03-06T07:27:26.535 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:29.790 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:29.793 DEBUG:teuthology.orchestra.run.smithi183:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:32.637 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi083.front.sepia.ceph.com 2024-03-06T07:27:32.637 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:32.637 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:27:32.652 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:32.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.791+0000 7f70bc17a640 1 Processor -- start 2024-03-06T07:27:32.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.791+0000 7f70bc17a640 1 -- start start 2024-03-06T07:27:32.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.792+0000 7f70bc17a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f70b415dfe0 0x7f70b40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.792+0000 7f70bc17a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f70b40cf830 0x7f70b40d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.792+0000 7f70bc17a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70b40b3430 con 0x7f70b40d27f0 2024-03-06T07:27:32.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.792+0000 7f70bc17a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70b40b31e0 con 0x7f70b40cf830 2024-03-06T07:27:32.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.792+0000 7f70bc17a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70b40b3d60 con 0x7f70b415dfe0 2024-03-06T07:27:32.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.792+0000 7f70b9715640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f70b40cf830 0x7f70b40d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.792+0000 7f70ba717640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f70b40d27f0 0x7f70b40d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:55970/0 (socket says 172.21.15.83:55970) 2024-03-06T07:27:32.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.792+0000 7f70ba717640 1 -- 172.21.15.83:0/3187337218 learned_addr learned my addr 172.21.15.83:0/3187337218 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:27:32.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.793+0000 7f70b9f16640 1 --2- 172.21.15.83:0/3187337218 >> v2:172.21.15.183:3300/0 conn(0x7f70b415dfe0 0x7f70b40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.793+0000 7f70b9715640 1 -- 172.21.15.83:0/3187337218 >> v2:172.21.15.183:3300/0 conn(0x7f70b415dfe0 msgr2=0x7f70b40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:32.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.793+0000 7f70b9715640 1 --2- 172.21.15.83:0/3187337218 >> v2:172.21.15.183:3300/0 conn(0x7f70b415dfe0 0x7f70b40cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.793+0000 7f70b9715640 1 -- 172.21.15.83:0/3187337218 >> v1:172.21.15.83:6789/0 conn(0x7f70b40d27f0 legacy=0x7f70b40d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.793+0000 7f70b9715640 1 -- 172.21.15.83:0/3187337218 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b40b3710 con 0x7f70b40cf830 2024-03-06T07:27:32.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.793+0000 7f70b9715640 1 --2- 172.21.15.83:0/3187337218 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f70b40cf830 0x7f70b40d22b0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f70a40097a0 tx=0x7f70a402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4987725210a853b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:32.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.793+0000 7f70b8f14640 1 -- 172.21.15.83:0/3187337218 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a4035020 con 0x7f70b40cf830 2024-03-06T07:27:32.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.794+0000 7f70b8f14640 1 -- 172.21.15.83:0/3187337218 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70a4031830 con 0x7f70b40cf830 2024-03-06T07:27:32.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.794+0000 7f70b8f14640 1 -- 172.21.15.83:0/3187337218 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a4031c00 con 0x7f70b40cf830 2024-03-06T07:27:32.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.794+0000 7f70bc17a640 1 -- 172.21.15.83:0/3187337218 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f70b40cf830 msgr2=0x7f70b40d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.794+0000 7f70bc17a640 1 --2- 172.21.15.83:0/3187337218 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f70b40cf830 0x7f70b40d22b0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f70a40097a0 tx=0x7f70a402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.794+0000 7f70bc17a640 1 -- 172.21.15.83:0/3187337218 shutdown_connections 2024-03-06T07:27:32.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.794+0000 7f70bc17a640 1 --2- 172.21.15.83:0/3187337218 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f70b40cf830 0x7f70b40d22b0 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.794+0000 7f70bc17a640 1 --2- 172.21.15.83:0/3187337218 >> v2:172.21.15.183:3300/0 conn(0x7f70b415dfe0 0x7f70b40cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.794+0000 7f70bc17a640 1 -- 172.21.15.83:0/3187337218 >> 172.21.15.83:0/3187337218 conn(0x7f70b40d3cd0 msgr2=0x7f70b40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:32.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.794+0000 7f70bc17a640 1 -- 172.21.15.83:0/3187337218 shutdown_connections 2024-03-06T07:27:32.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.794+0000 7f70bc17a640 1 -- 172.21.15.83:0/3187337218 wait complete. 2024-03-06T07:27:32.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.795+0000 7f70bc17a640 1 Processor -- start 2024-03-06T07:27:32.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.795+0000 7f70bc17a640 1 -- start start 2024-03-06T07:27:32.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.796+0000 7f70bc17a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f70b40cf830 0x7f70b4159b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.796+0000 7f70bc17a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f70b40d27f0 0x7f70b415a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.796+0000 7f70bc17a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f70b415dfe0 0x7f70b40b72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.796+0000 7f70bc17a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70b40d1df0 con 0x7f70b40cf830 2024-03-06T07:27:32.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.796+0000 7f70bc17a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70b40d1c70 con 0x7f70b40d27f0 2024-03-06T07:27:32.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.796+0000 7f70bc17a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70b415e3c0 con 0x7f70b415dfe0 2024-03-06T07:27:32.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.796+0000 7f70b9f16640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f70b40cf830 0x7f70b4159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.796+0000 7f70b9f16640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f70b40cf830 0x7f70b4159b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53926/0 (socket says 172.21.15.83:53926) 2024-03-06T07:27:32.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.796+0000 7f70b9f16640 1 -- 172.21.15.83:0/2233168714 learned_addr learned my addr 172.21.15.83:0/2233168714 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:32.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.796+0000 7f70b9715640 1 --2- 172.21.15.83:0/2233168714 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f70b40d27f0 0x7f70b415a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.796+0000 7f70b9f16640 1 -- 172.21.15.83:0/2233168714 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f70b415dfe0 msgr2=0x7f70b40b72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.796+0000 7f70b9f16640 1 --2- 172.21.15.83:0/2233168714 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f70b415dfe0 0x7f70b40b72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.796+0000 7f70b9f16640 1 -- 172.21.15.83:0/2233168714 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f70b40d27f0 msgr2=0x7f70b415a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.796+0000 7f70b9f16640 1 --2- 172.21.15.83:0/2233168714 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f70b40d27f0 0x7f70b415a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.796+0000 7f70b9f16640 1 -- 172.21.15.83:0/2233168714 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b415e790 con 0x7f70b40cf830 2024-03-06T07:27:32.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.797+0000 7f70b9f16640 1 --2- 172.21.15.83:0/2233168714 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f70b40cf830 0x7f70b4159b50 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f70b000d800 tx=0x7f70b000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:32.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.797+0000 7f70aa7fc640 1 -- 172.21.15.83:0/2233168714 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70b000b6c0 con 0x7f70b40cf830 2024-03-06T07:27:32.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.797+0000 7f70aa7fc640 1 -- 172.21.15.83:0/2233168714 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70b0019070 con 0x7f70b40cf830 2024-03-06T07:27:32.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.797+0000 7f70aa7fc640 1 -- 172.21.15.83:0/2233168714 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70b000ba80 con 0x7f70b40cf830 2024-03-06T07:27:32.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.797+0000 7f70bc17a640 1 -- 172.21.15.83:0/2233168714 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70b415ea70 con 0x7f70b40cf830 2024-03-06T07:27:32.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.797+0000 7f70bc17a640 1 -- 172.21.15.83:0/2233168714 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70b415f030 con 0x7f70b40cf830 2024-03-06T07:27:32.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.798+0000 7f70bc17a640 1 -- 172.21.15.83:0/2233168714 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70b404f330 con 0x7f70b40cf830 2024-03-06T07:27:32.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.798+0000 7f70aa7fc640 1 -- 172.21.15.83:0/2233168714 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f70b000bc20 con 0x7f70b40cf830 2024-03-06T07:27:32.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.799+0000 7f70aa7fc640 1 --2- 172.21.15.83:0/2233168714 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f709806b4d0 0x7f709806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.799+0000 7f70aa7fc640 1 -- 172.21.15.83:0/2233168714 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f70b008a6b0 con 0x7f70b40cf830 2024-03-06T07:27:32.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.799+0000 7f70b9715640 1 --2- 172.21.15.83:0/2233168714 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f709806b4d0 0x7f709806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.799+0000 7f70b9715640 1 --2- 172.21.15.83:0/2233168714 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f709806b4d0 0x7f709806d990 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f70a4036000 tx=0x7f70a4008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:32.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:32.803+0000 7f70aa7fc640 1 -- 172.21.15.83:0/2233168714 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f70b0059530 con 0x7f70b40cf830 2024-03-06T07:27:33.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.018+0000 7f70bc17a640 1 -- 172.21.15.83:0/2233168714 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70b40b51f0 con 0x7f70b40cf830 2024-03-06T07:27:33.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.018+0000 7f70aa7fc640 1 -- 172.21.15.83:0/2233168714 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f70b0058e30 con 0x7f70b40cf830 2024-03-06T07:27:33.020 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:33.021 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":59,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6200,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/2154269119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":2154269119},{"type":"v1","addr":"172.21.15.83:6837","nonce":2154269119}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6206,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3898960075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6833","nonce":3898960075},{"type":"v1","addr":"172.21.15.183:6835","nonce":3898960075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6838,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6834/1383294888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":1383294888},{"type":"v1","addr":"172.21.15.183:6834","nonce":1383294888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.491022+0000","modified":"2024-03-06T07:27:12.512838+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7887},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7887":{"gid":7887,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6836/855387870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":855387870},{"type":"v1","addr":"172.21.15.83:6836","nonce":855387870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7887,"qdb_cluster":[7887]},"id":8}]} 2024-03-06T07:27:33.021 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 59 2024-03-06T07:27:33.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.021+0000 7f70bc17a640 1 -- 172.21.15.83:0/2233168714 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f709806b4d0 msgr2=0x7f709806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.021+0000 7f70bc17a640 1 --2- 172.21.15.83:0/2233168714 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f709806b4d0 0x7f709806d990 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f70a4036000 tx=0x7f70a4008030 comp rx=0 tx=0).stop 2024-03-06T07:27:33.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.021+0000 7f70bc17a640 1 -- 172.21.15.83:0/2233168714 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f70b40cf830 msgr2=0x7f70b4159b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.021+0000 7f70bc17a640 1 --2- 172.21.15.83:0/2233168714 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f70b40cf830 0x7f70b4159b50 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f70b000d800 tx=0x7f70b000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.021+0000 7f70bc17a640 1 -- 172.21.15.83:0/2233168714 shutdown_connections 2024-03-06T07:27:33.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.021+0000 7f70bc17a640 1 --2- 172.21.15.83:0/2233168714 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f709806b4d0 0x7f709806d990 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.021+0000 7f70bc17a640 1 --2- 172.21.15.83:0/2233168714 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f70b415dfe0 0x7f70b40b72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.021+0000 7f70bc17a640 1 --2- 172.21.15.83:0/2233168714 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f70b40d27f0 0x7f70b415a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.021+0000 7f70bc17a640 1 --2- 172.21.15.83:0/2233168714 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f70b40cf830 0x7f70b4159b50 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.021+0000 7f70bc17a640 1 -- 172.21.15.83:0/2233168714 >> 172.21.15.83:0/2233168714 conn(0x7f70b40d3cd0 msgr2=0x7f70b406c810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.021+0000 7f70bc17a640 1 -- 172.21.15.83:0/2233168714 shutdown_connections 2024-03-06T07:27:33.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.021+0000 7f70bc17a640 1 -- 172.21.15.83:0/2233168714 wait complete. 2024-03-06T07:27:33.038 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:27:33.158 INFO:teuthology.orchestra.run.smithi083.stdout:{"message":"","return_code":0} 2024-03-06T07:27:33.158 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:27:33.159 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:27:33.159 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:33.159 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:27:33.159 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:33.159 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:27:33.160 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:33.160 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:27:33.160 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:33.168 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:27:33.168 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail c 2024-03-06T07:27:33.169 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:27:33.169 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2024-03-06T07:27:33.171 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:27:33.171 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-03-06T07:27:33.172 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:27:33.172 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-03-06T07:27:33.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.313+0000 7fc6b4b81640 1 Processor -- start 2024-03-06T07:27:33.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.314+0000 7fc6b4b81640 1 -- start start 2024-03-06T07:27:33.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.314+0000 7ff0764a4640 1 Processor -- start 2024-03-06T07:27:33.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.314+0000 7f7c18464640 1 Processor -- start 2024-03-06T07:27:33.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.314+0000 7f42455f6640 1 Processor -- start 2024-03-06T07:27:33.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.314+0000 7ff0764a4640 1 -- start start 2024-03-06T07:27:33.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.314+0000 7f42455f6640 1 -- start start 2024-03-06T07:27:33.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.314+0000 7f7c18464640 1 -- start start 2024-03-06T07:27:33.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.315+0000 7fc6b4b81640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fc6b006e000 0x7fc6b015eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.315+0000 7fc6ae59c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fc6b00cc920 0x7fc6b0072070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:55976/0 (socket says 172.21.15.83:55976) 2024-03-06T07:27:33.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.315+0000 7fc6ae59c640 1 -- 172.21.15.83:0/1976304957 learned_addr learned my addr 172.21.15.83:0/1976304957 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:27:33.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.315+0000 7fc6b4b81640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6b006e770 con 0x7fc6b00cc920 2024-03-06T07:27:33.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.315+0000 7fc6b4b81640 1 -- 172.21.15.83:0/1976304957 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6b006e490 con 0x7fc6b00b4a50 2024-03-06T07:27:33.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.315+0000 7fc6b4b81640 1 -- 172.21.15.83:0/1976304957 --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6b015e660 con 0x7fc6b006e000 2024-03-06T07:27:33.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.315+0000 7f42455f6640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f42400af7d0 0x7f42400cd430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.315+0000 7ff0764a4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff07015bbf0 0x7ff070071e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.315+0000 7fc6ad59a640 1 -- 172.21.15.83:0/1976304957 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1074157428 0 0) 0x7fc6b006e770 con 0x7fc6b00cc920 2024-03-06T07:27:33.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.315+0000 7fc6add9b640 1 --2- 172.21.15.83:0/1976304957 >> v2:172.21.15.183:3300/0 conn(0x7fc6b006e000 0x7fc6b015eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.315+0000 7f42455f6640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f4240066d00 0x7f42400d0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.315+0000 7f42455f6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f424006f140 con 0x7f4240067160 2024-03-06T07:27:33.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.315+0000 7f42455f6640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f424015dd50 con 0x7f42400af7d0 2024-03-06T07:27:33.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.315+0000 7f42455f6640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f424006c340 con 0x7f4240066d00 2024-03-06T07:27:33.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.315+0000 7ff0764a4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff0700cd7c0 0x7ff0700cdc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.315+0000 7fc6ad59a640 1 -- 172.21.15.83:0/1976304957 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc698005730 con 0x7fc6b00cc920 2024-03-06T07:27:33.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.315+0000 7f423effd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f42400af7d0 0x7f42400cd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.315+0000 7f423effd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f42400af7d0 0x7f42400cd430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53940/0 (socket says 172.21.15.83:53940) 2024-03-06T07:27:33.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.315+0000 7f423effd640 1 -- 172.21.15.83:0/1786034883 learned_addr learned my addr 172.21.15.83:0/1786034883 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:33.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.315+0000 7ff0764a4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff070072490 con 0x7ff0700cebc0 2024-03-06T07:27:33.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.315+0000 7ff0764a4640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0700ce170 con 0x7ff07015bbf0 2024-03-06T07:27:33.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.315+0000 7ff06ffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff07015bbf0 0x7ff070071e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.315+0000 7fc6ad59a640 1 -- 172.21.15.83:0/1976304957 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1840056604 0 0) 0x7fc6b006e490 con 0x7fc6b00b4a50 2024-03-06T07:27:33.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.315+0000 7ff0764a4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0700d0090 con 0x7ff0700cd7c0 2024-03-06T07:27:33.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.315+0000 7ff06ffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff07015bbf0 0x7ff070071e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53950/0 (socket says 172.21.15.83:53950) 2024-03-06T07:27:33.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.315+0000 7f7c18464640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7c100b4460 0x7f7c100d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.315+0000 7ff06ffff640 1 -- 172.21.15.83:0/344488214 learned_addr learned my addr 172.21.15.83:0/344488214 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:33.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.315+0000 7f7c18464640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f7c10161cf0 0x7f7c10162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.315+0000 7fc6ad59a640 1 -- 172.21.15.83:0/1976304957 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6b006e770 con 0x7fc6b00b4a50 2024-03-06T07:27:33.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.316+0000 7f7c18464640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c100d22c0 con 0x7f7c10162970 2024-03-06T07:27:33.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.316+0000 7f7c18464640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c10161b00 con 0x7f7c100b4460 2024-03-06T07:27:33.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.316+0000 7f7c18464640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c100cecc0 con 0x7f7c10161cf0 2024-03-06T07:27:33.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.316+0000 7fc6add9b640 1 -- 172.21.15.83:0/1976304957 >> v1:172.21.15.183:6789/0 conn(0x7fc6b00b4a50 legacy=0x7fc6b0150b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.316+0000 7ff074a41640 1 --2- 172.21.15.83:0/344488214 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff0700cd7c0 0x7ff0700cdc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.316+0000 7f423e7fc640 1 --2- 172.21.15.83:0/1786034883 >> v2:172.21.15.183:3300/0 conn(0x7f4240066d00 0x7f42400d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.316+0000 7fc6ad59a640 1 -- 172.21.15.83:0/1976304957 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3444950649 0 0) 0x7fc698005730 con 0x7fc6b00cc920 2024-03-06T07:27:33.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.316+0000 7fc6add9b640 1 -- 172.21.15.83:0/1976304957 >> v1:172.21.15.83:6789/0 conn(0x7fc6b00cc920 legacy=0x7fc6b0072070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.316+0000 7f423effd640 1 -- 172.21.15.83:0/1786034883 >> v2:172.21.15.183:3300/0 conn(0x7f4240066d00 msgr2=0x7f42400d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:33.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.316+0000 7f423effd640 1 --2- 172.21.15.83:0/1786034883 >> v2:172.21.15.183:3300/0 conn(0x7f4240066d00 0x7f42400d0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.316+0000 7f423effd640 1 -- 172.21.15.83:0/1786034883 >> v1:172.21.15.83:6789/0 conn(0x7f4240067160 legacy=0x7f42400d0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.316+0000 7fc6add9b640 1 -- 172.21.15.83:0/1976304957 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b015e010 con 0x7fc6b006e000 2024-03-06T07:27:33.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.316+0000 7f423effd640 1 -- 172.21.15.83:0/1786034883 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f424006b730 con 0x7f42400af7d0 2024-03-06T07:27:33.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.316+0000 7f7c16a01640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f7c10161cf0 0x7f7c10162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.316+0000 7ff06ffff640 1 -- 172.21.15.83:0/344488214 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff0700cd7c0 msgr2=0x7ff0700cdc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:33.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.316+0000 7f7c16a01640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f7c10161cf0 0x7f7c10162180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:55460/0 (socket says 172.21.15.83:55460) 2024-03-06T07:27:33.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.316+0000 7ff06ffff640 1 --2- 172.21.15.83:0/344488214 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff0700cd7c0 0x7ff0700cdc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.316+0000 7f7c16a01640 1 -- 172.21.15.83:0/3521857370 learned_addr learned my addr 172.21.15.83:0/3521857370 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:33.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.316+0000 7ff06ffff640 1 -- 172.21.15.83:0/344488214 >> v1:172.21.15.83:6789/0 conn(0x7ff0700cebc0 legacy=0x7ff07015bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:33.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.316+0000 7f7c16200640 1 --2- 172.21.15.83:0/3521857370 >> v2:172.21.15.83:3300/0 conn(0x7f7c100b4460 0x7f7c100d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.316+0000 7ff06ffff640 1 -- 172.21.15.83:0/344488214 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0700d0370 con 0x7ff07015bbf0 2024-03-06T07:27:33.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.316+0000 7f7c16200640 1 -- 172.21.15.83:0/3521857370 >> v2:172.21.15.183:3300/0 conn(0x7f7c10161cf0 msgr2=0x7f7c10162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:33.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.316+0000 7f7c16200640 1 --2- 172.21.15.83:0/3521857370 >> v2:172.21.15.183:3300/0 conn(0x7f7c10161cf0 0x7f7c10162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.316+0000 7f7c16200640 1 -- 172.21.15.83:0/3521857370 >> v1:172.21.15.83:6789/0 conn(0x7f7c10162970 legacy=0x7f7c100b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.316+0000 7f7c16200640 1 -- 172.21.15.83:0/3521857370 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c100d25a0 con 0x7f7c100b4460 2024-03-06T07:27:33.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.317+0000 7f423effd640 1 --2- 172.21.15.83:0/1786034883 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42400af7d0 0x7f42400cd430 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f424006da50 tx=0x7f423002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0cb4dd6c39c99c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.317+0000 7f423dffb640 1 -- 172.21.15.83:0/1786034883 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4230034070 con 0x7f42400af7d0 2024-03-06T07:27:33.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.317+0000 7fc6add9b640 1 --2- 172.21.15.83:0/1976304957 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc6b006e000 0x7fc6b015eaf0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fc6b015e7e0 tx=0x7fc6a402fb20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62c147757371345c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.317+0000 7f423dffb640 1 -- 172.21.15.83:0/1786034883 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4230030860 con 0x7f42400af7d0 2024-03-06T07:27:33.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.317+0000 7f423dffb640 1 -- 172.21.15.83:0/1786034883 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4230030c30 con 0x7f42400af7d0 2024-03-06T07:27:33.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.317+0000 7ff06ffff640 1 --2- 172.21.15.83:0/344488214 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff07015bbf0 0x7ff070071e00 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7ff06000dc90 tx=0x7ff06002fd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fe5c4a3cc0c2c5f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.317+0000 7fc6ad59a640 1 -- 172.21.15.83:0/1976304957 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a4037070 con 0x7fc6b006e000 2024-03-06T07:27:33.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.317+0000 7fc6ad59a640 1 -- 172.21.15.83:0/1976304957 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6a400b450 con 0x7fc6b006e000 2024-03-06T07:27:33.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.317+0000 7fc6ad59a640 1 -- 172.21.15.83:0/1976304957 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a00015f0 con 0x7fc6b006e000 2024-03-06T07:27:33.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.317+0000 7f7c16200640 1 --2- 172.21.15.83:0/3521857370 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c100b4460 0x7f7c100d4800 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f7c1008c900 tx=0x7f7c0002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c44f1477365b3d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.318+0000 7ff06effd640 1 -- 172.21.15.83:0/344488214 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff060030930 con 0x7ff07015bbf0 2024-03-06T07:27:33.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.318+0000 7ff06effd640 1 -- 172.21.15.83:0/344488214 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff060030ad0 con 0x7ff07015bbf0 2024-03-06T07:27:33.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.318+0000 7ff06effd640 1 -- 172.21.15.83:0/344488214 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06000b450 con 0x7ff07015bbf0 2024-03-06T07:27:33.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.318+0000 7f7c151fe640 1 -- 172.21.15.83:0/3521857370 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c00035020 con 0x7f7c100b4460 2024-03-06T07:27:33.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.318+0000 7f7c151fe640 1 -- 172.21.15.83:0/3521857370 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c00031830 con 0x7f7c100b4460 2024-03-06T07:27:33.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.318+0000 7f7c151fe640 1 -- 172.21.15.83:0/3521857370 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c00031c00 con 0x7f7c100b4460 2024-03-06T07:27:33.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.318+0000 7f42455f6640 1 -- 172.21.15.83:0/1786034883 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42400af7d0 msgr2=0x7f42400cd430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.318+0000 7f42455f6640 1 --2- 172.21.15.83:0/1786034883 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42400af7d0 0x7f42400cd430 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f424006da50 tx=0x7f423002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.318+0000 7f42455f6640 1 -- 172.21.15.83:0/1786034883 shutdown_connections 2024-03-06T07:27:33.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.318+0000 7f42455f6640 1 --2- 172.21.15.83:0/1786034883 >> v2:172.21.15.183:3300/0 conn(0x7f4240066d00 0x7f42400d0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.318+0000 7f42455f6640 1 --2- 172.21.15.83:0/1786034883 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42400af7d0 0x7f42400cd430 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.318+0000 7f42455f6640 1 -- 172.21.15.83:0/1786034883 >> 172.21.15.83:0/1786034883 conn(0x7f4240072520 msgr2=0x7f4240072930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.318+0000 7f42455f6640 1 -- 172.21.15.83:0/1786034883 shutdown_connections 2024-03-06T07:27:33.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.318+0000 7f42455f6640 1 -- 172.21.15.83:0/1786034883 wait complete. 2024-03-06T07:27:33.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.318+0000 7fc6b4b81640 1 -- 172.21.15.83:0/1976304957 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc6b006e000 msgr2=0x7fc6b015eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.318+0000 7ff0764a4640 1 -- 172.21.15.83:0/344488214 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff07015bbf0 msgr2=0x7ff070071e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.318+0000 7ff0764a4640 1 --2- 172.21.15.83:0/344488214 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff07015bbf0 0x7ff070071e00 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7ff06000dc90 tx=0x7ff06002fd90 comp rx=0 tx=0).stop 2024-03-06T07:27:33.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.318+0000 7fc6b4b81640 1 --2- 172.21.15.83:0/1976304957 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc6b006e000 0x7fc6b015eaf0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fc6b015e7e0 tx=0x7fc6a402fb20 comp rx=0 tx=0).stop 2024-03-06T07:27:33.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.319+0000 7ff0764a4640 1 -- 172.21.15.83:0/344488214 shutdown_connections 2024-03-06T07:27:33.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.319+0000 7ff0764a4640 1 --2- 172.21.15.83:0/344488214 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff0700cd7c0 0x7ff0700cdc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.319+0000 7ff0764a4640 1 --2- 172.21.15.83:0/344488214 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff07015bbf0 0x7ff070071e00 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.319+0000 7ff0764a4640 1 -- 172.21.15.83:0/344488214 >> 172.21.15.83:0/344488214 conn(0x7ff0700b1c80 msgr2=0x7ff0700cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.319+0000 7fc6b4b81640 1 -- 172.21.15.83:0/1976304957 shutdown_connections 2024-03-06T07:27:33.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.319+0000 7ff0764a4640 1 -- 172.21.15.83:0/344488214 shutdown_connections 2024-03-06T07:27:33.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.319+0000 7ff0764a4640 1 -- 172.21.15.83:0/344488214 wait complete. 2024-03-06T07:27:33.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.319+0000 7fc6b4b81640 1 --2- 172.21.15.83:0/1976304957 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc6b006e000 0x7fc6b015eaf0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.319+0000 7f7c18464640 1 -- 172.21.15.83:0/3521857370 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c100b4460 msgr2=0x7f7c100d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.319+0000 7fc6b4b81640 1 -- 172.21.15.83:0/1976304957 >> 172.21.15.83:0/1976304957 conn(0x7fc6b00b1c80 msgr2=0x7fc6b00cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.319+0000 7f7c18464640 1 --2- 172.21.15.83:0/3521857370 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c100b4460 0x7f7c100d4800 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f7c1008c900 tx=0x7f7c0002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.319+0000 7fc6b4b81640 1 -- 172.21.15.83:0/1976304957 shutdown_connections 2024-03-06T07:27:33.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.319+0000 7fc6b4b81640 1 -- 172.21.15.83:0/1976304957 wait complete. 2024-03-06T07:27:33.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.319+0000 7f7c18464640 1 -- 172.21.15.83:0/3521857370 shutdown_connections 2024-03-06T07:27:33.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.319+0000 7f7c18464640 1 --2- 172.21.15.83:0/3521857370 >> v2:172.21.15.183:3300/0 conn(0x7f7c10161cf0 0x7f7c10162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.319+0000 7f7c18464640 1 --2- 172.21.15.83:0/3521857370 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c100b4460 0x7f7c100d4800 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.319+0000 7f7c18464640 1 -- 172.21.15.83:0/3521857370 >> 172.21.15.83:0/3521857370 conn(0x7f7c100d3ce0 msgr2=0x7f7c100d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.319+0000 7f7c18464640 1 -- 172.21.15.83:0/3521857370 shutdown_connections 2024-03-06T07:27:33.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.319+0000 7f7c18464640 1 -- 172.21.15.83:0/3521857370 wait complete. 2024-03-06T07:27:33.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.320+0000 7f42455f6640 1 Processor -- start 2024-03-06T07:27:33.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.320+0000 7f42455f6640 1 -- start start 2024-03-06T07:27:33.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.320+0000 7ff0764a4640 1 Processor -- start 2024-03-06T07:27:33.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.320+0000 7ff0764a4640 1 -- start start 2024-03-06T07:27:33.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.320+0000 7f42455f6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4240066d00 0x7f42401f3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.320+0000 7f42455f6640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4240067160 0x7f424006e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.320+0000 7f42455f6640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f42400af7d0 0x7f42400cbe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.321+0000 7f42455f6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f424006bfd0 con 0x7f4240066d00 2024-03-06T07:27:33.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.321+0000 7f42455f6640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f424006bcf0 con 0x7f4240067160 2024-03-06T07:27:33.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.321+0000 7f42455f6640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42400cd970 con 0x7f42400af7d0 2024-03-06T07:27:33.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.321+0000 7f423effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4240066d00 0x7f42401f3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.321+0000 7f423effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4240066d00 0x7f42401f3cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53958/0 (socket says 172.21.15.83:53958) 2024-03-06T07:27:33.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.321+0000 7f423effd640 1 -- 172.21.15.83:0/4099814569 learned_addr learned my addr 172.21.15.83:0/4099814569 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:33.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.321+0000 7f423f7fe640 1 --2- 172.21.15.83:0/4099814569 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f42400af7d0 0x7f42400cbe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.321+0000 7f423e7fc640 1 --2- 172.21.15.83:0/4099814569 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4240067160 0x7f424006e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.321+0000 7ff0764a4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0700cd7c0 0x7ff07014b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.321+0000 7ff0764a4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff0700cebc0 0x7ff0701f28f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.321+0000 7f423effd640 1 -- 172.21.15.83:0/4099814569 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f42400af7d0 msgr2=0x7f42400cbe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.321+0000 7ff0764a4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff07015bbf0 0x7ff07006cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.321+0000 7ff06ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0700cd7c0 0x7ff07014b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.321+0000 7f7c18464640 1 Processor -- start 2024-03-06T07:27:33.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.321+0000 7fc6b4b81640 1 Processor -- start 2024-03-06T07:27:33.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.321+0000 7ff06ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0700cd7c0 0x7ff07014b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53972/0 (socket says 172.21.15.83:53972) 2024-03-06T07:27:33.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.321+0000 7f7c18464640 1 -- start start 2024-03-06T07:27:33.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.321+0000 7fc6b4b81640 1 -- start start 2024-03-06T07:27:33.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.321+0000 7ff0764a4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0700cf010 con 0x7ff0700cd7c0 2024-03-06T07:27:33.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.321+0000 7ff0764a4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07015c6e0 con 0x7ff07015bbf0 2024-03-06T07:27:33.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.321+0000 7ff0764a4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0700b3680 con 0x7ff0700cebc0 2024-03-06T07:27:33.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.321+0000 7ff06ffff640 1 -- 172.21.15.83:0/2477841688 learned_addr learned my addr 172.21.15.83:0/2477841688 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:33.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.321+0000 7ff074a41640 1 --2- 172.21.15.83:0/2477841688 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff07015bbf0 0x7ff07006cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.321+0000 7ff06f7fe640 1 --2- 172.21.15.83:0/2477841688 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff0700cebc0 0x7ff0701f28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.321+0000 7f423effd640 1 --2- 172.21.15.83:0/4099814569 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f42400af7d0 0x7f42400cbe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.321+0000 7f423effd640 1 -- 172.21.15.83:0/4099814569 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4240067160 msgr2=0x7f424006e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.321+0000 7f423effd640 1 --2- 172.21.15.83:0/4099814569 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4240067160 0x7f424006e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.321+0000 7f423effd640 1 -- 172.21.15.83:0/4099814569 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42400cb680 con 0x7f4240066d00 2024-03-06T07:27:33.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.321+0000 7f423f7fe640 1 --2- 172.21.15.83:0/4099814569 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f42400af7d0 0x7f42400cbe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:33.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.321+0000 7f423e7fc640 1 --2- 172.21.15.83:0/4099814569 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4240067160 0x7f424006e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:33.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.321+0000 7f423effd640 1 --2- 172.21.15.83:0/4099814569 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4240066d00 0x7f42401f3cb0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f423002b9e0 tx=0x7f42300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.322+0000 7f42277fe640 1 -- 172.21.15.83:0/4099814569 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4230034040 con 0x7f4240066d00 2024-03-06T07:27:33.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.322+0000 7f42277fe640 1 -- 172.21.15.83:0/4099814569 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42300433f0 con 0x7f4240066d00 2024-03-06T07:27:33.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.322+0000 7f42455f6640 1 -- 172.21.15.83:0/4099814569 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42401f41f0 con 0x7f4240066d00 2024-03-06T07:27:33.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.322+0000 7f42455f6640 1 -- 172.21.15.83:0/4099814569 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42400b27b0 con 0x7f4240066d00 2024-03-06T07:27:33.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.322+0000 7ff074a41640 1 -- 172.21.15.83:0/2477841688 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff0700cebc0 msgr2=0x7ff0701f28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.322+0000 7ff074a41640 1 --2- 172.21.15.83:0/2477841688 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff0700cebc0 0x7ff0701f28f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.322+0000 7ff074a41640 1 -- 172.21.15.83:0/2477841688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0700cd7c0 msgr2=0x7ff07014b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.322+0000 7ff074a41640 1 --2- 172.21.15.83:0/2477841688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0700cd7c0 0x7ff07014b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.322+0000 7ff074a41640 1 -- 172.21.15.83:0/2477841688 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0701f2e30 con 0x7ff07015bbf0 2024-03-06T07:27:33.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.322+0000 7ff06ffff640 1 --2- 172.21.15.83:0/2477841688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0700cd7c0 0x7ff07014b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:33.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.322+0000 7f42277fe640 1 -- 172.21.15.83:0/4099814569 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42300437e0 con 0x7f4240066d00 2024-03-06T07:27:33.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.322+0000 7ff074a41640 1 --2- 172.21.15.83:0/2477841688 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff07015bbf0 0x7ff07006cf80 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7ff06800efc0 tx=0x7ff06800c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.322+0000 7fc6b4b81640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6b00b4a50 0x7fc6b0149250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.322+0000 7fc6b4b81640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc6b00cc920 0x7fc6b0157bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.322+0000 7f7c18464640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c100b4460 0x7f7c101f4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.322+0000 7f7c18464640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7c10161cf0 0x7f7c1014dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.322+0000 7fc6b4b81640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc6b0158100 0x7fc6b01595f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.322+0000 7fc6b4b81640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b00cce80 con 0x7fc6b00b4a50 2024-03-06T07:27:33.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.322+0000 7fc6b4b81640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b00ccd00 con 0x7fc6b00cc920 2024-03-06T07:27:33.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.322+0000 7fc6b4b81640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b0070940 con 0x7fc6b0158100 2024-03-06T07:27:33.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.322+0000 7f7c18464640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7c10162970 0x7f7c100cd540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.322+0000 7f7c18464640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c100ce950 con 0x7f7c100b4460 2024-03-06T07:27:33.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.322+0000 7f7c18464640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c100d2b60 con 0x7f7c10161cf0 2024-03-06T07:27:33.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.322+0000 7f7c18464640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c100b4840 con 0x7f7c10162970 2024-03-06T07:27:33.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.322+0000 7fc6ae59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6b00b4a50 0x7fc6b0149250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.322+0000 7fc6ae59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6b00b4a50 0x7fc6b0149250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53974/0 (socket says 172.21.15.83:53974) 2024-03-06T07:27:33.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.322+0000 7fc6ae59c640 1 -- 172.21.15.83:0/2208176825 learned_addr learned my addr 172.21.15.83:0/2208176825 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:33.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.322+0000 7f7c16a01640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7c10162970 0x7f7c100cd540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.322+0000 7f7c159ff640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7c10161cf0 0x7f7c1014dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.323+0000 7f7c159ff640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7c10161cf0 0x7f7c1014dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:55472/0 (socket says 172.21.15.83:55472) 2024-03-06T07:27:33.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.323+0000 7f7c159ff640 1 -- 172.21.15.83:0/1538754687 learned_addr learned my addr 172.21.15.83:0/1538754687 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:33.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.323+0000 7f7c16a01640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7c10162970 0x7f7c100cd540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:54496/0 (socket says 172.21.15.83:54496) 2024-03-06T07:27:33.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.323+0000 7f7c16200640 1 --2- 172.21.15.83:0/1538754687 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c100b4460 0x7f7c101f4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.323+0000 7f7c159ff640 1 -- 172.21.15.83:0/1538754687 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7c10162970 msgr2=0x7f7c100cd540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.323+0000 7f7c159ff640 1 --2- 172.21.15.83:0/1538754687 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7c10162970 0x7f7c100cd540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.323+0000 7f7c159ff640 1 -- 172.21.15.83:0/1538754687 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c100b4460 msgr2=0x7f7c101f4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.323+0000 7f7c159ff640 1 --2- 172.21.15.83:0/1538754687 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c100b4460 0x7f7c101f4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.323+0000 7f7c159ff640 1 -- 172.21.15.83:0/1538754687 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c10150890 con 0x7f7c10161cf0 2024-03-06T07:27:33.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.323+0000 7fc6ae59c640 1 -- 172.21.15.83:0/2208176825 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc6b0158100 msgr2=0x7fc6b01595f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.323+0000 7fc6ae59c640 1 --2- 172.21.15.83:0/2208176825 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc6b0158100 0x7fc6b01595f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.323+0000 7fc6ae59c640 1 -- 172.21.15.83:0/2208176825 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc6b00cc920 msgr2=0x7fc6b0157bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.323+0000 7fc6ae59c640 1 --2- 172.21.15.83:0/2208176825 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc6b00cc920 0x7fc6b0157bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.323+0000 7fc6ae59c640 1 -- 172.21.15.83:0/2208176825 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b00b0240 con 0x7fc6b00b4a50 2024-03-06T07:27:33.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.322+0000 7ff06cff9640 1 -- 172.21.15.83:0/2477841688 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff068018020 con 0x7ff07015bbf0 2024-03-06T07:27:33.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.322+0000 7ff06cff9640 1 -- 172.21.15.83:0/2477841688 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff068014480 con 0x7ff07015bbf0 2024-03-06T07:27:33.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.322+0000 7ff0764a4640 1 -- 172.21.15.83:0/2477841688 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff07015aa30 con 0x7ff07015bbf0 2024-03-06T07:27:33.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.322+0000 7ff0764a4640 1 -- 172.21.15.83:0/2477841688 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0701362e0 con 0x7ff07015bbf0 2024-03-06T07:27:33.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.323+0000 7ff06cff9640 1 -- 172.21.15.83:0/2477841688 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff068014830 con 0x7ff07015bbf0 2024-03-06T07:27:33.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.323+0000 7f42455f6640 1 -- 172.21.15.83:0/4099814569 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f424006b1e0 con 0x7f4240066d00 2024-03-06T07:27:33.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.323+0000 7fc6ae59c640 1 --2- 172.21.15.83:0/2208176825 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6b00b4a50 0x7fc6b0149250 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fc6a000c480 tx=0x7fc6a000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.323+0000 7f7c159ff640 1 --2- 172.21.15.83:0/1538754687 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7c10161cf0 0x7f7c1014dcd0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c00c440 tx=0x7f7c0c001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.324+0000 7fc69effd640 1 -- 172.21.15.83:0/2208176825 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a0018070 con 0x7fc6b00b4a50 2024-03-06T07:27:33.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.324+0000 7fc69effd640 1 -- 172.21.15.83:0/2208176825 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6a00075a0 con 0x7fc6b00b4a50 2024-03-06T07:27:33.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.324+0000 7fc6b4b81640 1 -- 172.21.15.83:0/2208176825 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6b01570e0 con 0x7fc6b00b4a50 2024-03-06T07:27:33.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.324+0000 7fc6b4b81640 1 -- 172.21.15.83:0/2208176825 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6b0149790 con 0x7fc6b00b4a50 2024-03-06T07:27:33.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.324+0000 7ff0764a4640 1 -- 172.21.15.83:0/2477841688 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff07004f330 con 0x7ff07015bbf0 2024-03-06T07:27:33.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.324+0000 7fc69effd640 1 -- 172.21.15.83:0/2208176825 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a0018070 con 0x7fc6b00b4a50 2024-03-06T07:27:33.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.324+0000 7f42277fe640 1 -- 172.21.15.83:0/4099814569 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4230037070 con 0x7f4240066d00 2024-03-06T07:27:33.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.324+0000 7f7c06ffd640 1 -- 172.21.15.83:0/1538754687 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c0052d0 con 0x7f7c10161cf0 2024-03-06T07:27:33.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.324+0000 7f7c06ffd640 1 -- 172.21.15.83:0/1538754687 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c0c005470 con 0x7f7c10161cf0 2024-03-06T07:27:33.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.324+0000 7f7c06ffd640 1 -- 172.21.15.83:0/1538754687 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c007260 con 0x7f7c10161cf0 2024-03-06T07:27:33.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.324+0000 7f7c18464640 1 -- 172.21.15.83:0/1538754687 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c10150b70 con 0x7f7c10161cf0 2024-03-06T07:27:33.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.324+0000 7f7c18464640 1 -- 172.21.15.83:0/1538754687 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c10151130 con 0x7f7c10161cf0 2024-03-06T07:27:33.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.325+0000 7ff06cff9640 1 -- 172.21.15.83:0/2477841688 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff068005d10 con 0x7ff07015bbf0 2024-03-06T07:27:33.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.325+0000 7fc6b4b81640 1 -- 172.21.15.83:0/2208176825 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6b004f330 con 0x7fc6b00b4a50 2024-03-06T07:27:33.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.325+0000 7f42277fe640 1 --2- 172.21.15.83:0/4099814569 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f422c06b520 0x7f422c06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.325+0000 7f42277fe640 1 -- 172.21.15.83:0/4099814569 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f42300ac480 con 0x7f4240066d00 2024-03-06T07:27:33.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.325+0000 7f423e7fc640 1 --2- 172.21.15.83:0/4099814569 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f422c06b520 0x7f422c06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.325+0000 7ff06cff9640 1 --2- 172.21.15.83:0/2477841688 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff05c06b520 0x7ff05c06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.326+0000 7ff06cff9640 1 -- 172.21.15.83:0/2477841688 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff06808b490 con 0x7ff07015bbf0 2024-03-06T07:27:33.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.326+0000 7ff06ffff640 1 --2- 172.21.15.83:0/2477841688 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff05c06b520 0x7ff05c06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.326+0000 7f7c18464640 1 -- 172.21.15.83:0/1538754687 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c1004f330 con 0x7f7c10161cf0 2024-03-06T07:27:33.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.326+0000 7fc69effd640 1 -- 172.21.15.83:0/2208176825 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc6a0012400 con 0x7fc6b00b4a50 2024-03-06T07:27:33.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.326+0000 7f423e7fc640 1 --2- 172.21.15.83:0/4099814569 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f422c06b520 0x7f422c06d9e0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f4240153b00 tx=0x7f4234008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.326+0000 7ff06ffff640 1 --2- 172.21.15.83:0/2477841688 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff05c06b520 0x7ff05c06d9e0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7ff060000c00 tx=0x7ff060008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.326+0000 7fc69effd640 1 --2- 172.21.15.83:0/2208176825 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc69006b900 0x7fc69006ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.326+0000 7fc6add9b640 1 --2- 172.21.15.83:0/2208176825 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc69006b900 0x7fc69006ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.326+0000 7fc69effd640 1 -- 172.21.15.83:0/2208176825 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc6a00073a0 con 0x7fc6b00b4a50 2024-03-06T07:27:33.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.327+0000 7fc6add9b640 1 --2- 172.21.15.83:0/2208176825 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc69006b900 0x7fc69006ddc0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4039000 tx=0x7fc6a4034000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.329+0000 7ff06cff9640 1 -- 172.21.15.83:0/2477841688 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff068022030 con 0x7ff07015bbf0 2024-03-06T07:27:33.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.330+0000 7f42277fe640 1 -- 172.21.15.83:0/4099814569 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f424006b1e0 con 0x7f4240066d00 2024-03-06T07:27:33.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.331+0000 7f7c06ffd640 1 -- 172.21.15.83:0/1538754687 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7c0c007400 con 0x7f7c10161cf0 2024-03-06T07:27:33.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.331+0000 7f7c06ffd640 1 --2- 172.21.15.83:0/1538754687 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7bf406b520 0x7f7bf406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.331+0000 7f7c06ffd640 1 -- 172.21.15.83:0/1538754687 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7c0c089b20 con 0x7f7c10161cf0 2024-03-06T07:27:33.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.331+0000 7f7c16200640 1 --2- 172.21.15.83:0/1538754687 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7bf406b520 0x7f7bf406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.332+0000 7f7c16200640 1 --2- 172.21.15.83:0/1538754687 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7bf406b520 0x7f7bf406d9e0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f7c00036000 tx=0x7f7c0003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.334+0000 7f7c06ffd640 1 -- 172.21.15.83:0/1538754687 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7c0c0583c0 con 0x7f7c10161cf0 2024-03-06T07:27:33.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.336+0000 7fc69effd640 1 -- 172.21.15.83:0/2208176825 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc6a0057df0 con 0x7fc6b00b4a50 2024-03-06T07:27:33.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.585+0000 7f7c18464640 1 -- 172.21.15.83:0/1538754687 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) v1 -- 0x7f7c1006b180 con 0x7f7c10161cf0 2024-03-06T07:27:33.587 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-03-06T07:27:33.586+0000 7fb0dae11640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:27:33.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.586+0000 7f42455f6640 1 -- 172.21.15.83:0/4099814569 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7f424006a030 con 0x7f4240066d00 2024-03-06T07:27:33.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.591+0000 7ff0764a4640 1 -- 172.21.15.83:0/2477841688 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "c"} v 0) v1 -- 0x7ff0700b4650 con 0x7ff07015bbf0 2024-03-06T07:27:33.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.593+0000 7f7c06ffd640 1 -- 172.21.15.83:0/1538754687 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 7887 v60) v1 ==== 95+0+0 (secure 0 0 0) 0x7f7c0c05c460 con 0x7f7c10161cf0 2024-03-06T07:27:33.595 INFO:teuthology.orchestra.run.smithi083.stderr:failed mds gid 7887 2024-03-06T07:27:33.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.596+0000 7f42277fe640 1 -- 172.21.15.83:0/4099814569 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 6206 v61) v1 ==== 95+0+0 (secure 0 0 0) 0x7f424006a030 con 0x7f4240066d00 2024-03-06T07:27:33.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.596+0000 7f7c18464640 1 -- 172.21.15.83:0/1538754687 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7bf406b520 msgr2=0x7f7bf406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.596+0000 7f7c18464640 1 --2- 172.21.15.83:0/1538754687 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7bf406b520 0x7f7bf406d9e0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f7c00036000 tx=0x7f7c0003c040 comp rx=0 tx=0).stop 2024-03-06T07:27:33.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.596+0000 7f7c18464640 1 -- 172.21.15.83:0/1538754687 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7c10161cf0 msgr2=0x7f7c1014dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.596+0000 7f7c18464640 1 --2- 172.21.15.83:0/1538754687 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7c10161cf0 0x7f7c1014dcd0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c00c440 tx=0x7f7c0c001e70 comp rx=0 tx=0).stop 2024-03-06T07:27:33.598 INFO:teuthology.orchestra.run.smithi083.stderr:failed mds gid 6206 2024-03-06T07:27:33.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.596+0000 7f7c18464640 1 -- 172.21.15.83:0/1538754687 shutdown_connections 2024-03-06T07:27:33.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.596+0000 7f7c18464640 1 --2- 172.21.15.83:0/1538754687 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7bf406b520 0x7f7bf406d9e0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.596+0000 7f7c18464640 1 --2- 172.21.15.83:0/1538754687 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7c10162970 0x7f7c100cd540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.596+0000 7f7c18464640 1 --2- 172.21.15.83:0/1538754687 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7c10161cf0 0x7f7c1014dcd0 secure :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c00c440 tx=0x7f7c0c001e70 comp rx=0 tx=0).stop 2024-03-06T07:27:33.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.596+0000 7f7c18464640 1 --2- 172.21.15.83:0/1538754687 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c100b4460 0x7f7c101f4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.596+0000 7f7c18464640 1 -- 172.21.15.83:0/1538754687 >> 172.21.15.83:0/1538754687 conn(0x7f7c100d3ce0 msgr2=0x7f7c100d1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.596+0000 7f7c18464640 1 -- 172.21.15.83:0/1538754687 shutdown_connections 2024-03-06T07:27:33.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.596+0000 7f7c18464640 1 -- 172.21.15.83:0/1538754687 wait complete. 2024-03-06T07:27:33.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.598+0000 7f42455f6640 1 -- 172.21.15.83:0/4099814569 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f422c06b520 msgr2=0x7f422c06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.598+0000 7f42455f6640 1 --2- 172.21.15.83:0/4099814569 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f422c06b520 0x7f422c06d9e0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f4240153b00 tx=0x7f4234008040 comp rx=0 tx=0).stop 2024-03-06T07:27:33.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.598+0000 7f42455f6640 1 -- 172.21.15.83:0/4099814569 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4240066d00 msgr2=0x7f42401f3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.598+0000 7f42455f6640 1 --2- 172.21.15.83:0/4099814569 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4240066d00 0x7f42401f3cb0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f423002b9e0 tx=0x7f42300046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.599+0000 7f42455f6640 1 -- 172.21.15.83:0/4099814569 shutdown_connections 2024-03-06T07:27:33.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.599+0000 7f42455f6640 1 --2- 172.21.15.83:0/4099814569 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f422c06b520 0x7f422c06d9e0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.599+0000 7f42455f6640 1 --2- 172.21.15.83:0/4099814569 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f42400af7d0 0x7f42400cbe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.599+0000 7f42455f6640 1 --2- 172.21.15.83:0/4099814569 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4240067160 0x7f424006e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.599+0000 7f42455f6640 1 --2- 172.21.15.83:0/4099814569 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4240066d00 0x7f42401f3cb0 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.599+0000 7f42455f6640 1 -- 172.21.15.83:0/4099814569 >> 172.21.15.83:0/4099814569 conn(0x7f4240072520 msgr2=0x7f42401587f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.599+0000 7f42455f6640 1 -- 172.21.15.83:0/4099814569 shutdown_connections 2024-03-06T07:27:33.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.599+0000 7f42455f6640 1 -- 172.21.15.83:0/4099814569 wait complete. 2024-03-06T07:27:33.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.603+0000 7fc6b4b81640 1 -- 172.21.15.83:0/2208176825 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7fc6b00cbdf0 con 0x7fc6b00b4a50 2024-03-06T07:27:33.606 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-03-06T07:27:33.604+0000 7fb0dae11640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:27:33.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.606+0000 7fc69effd640 1 -- 172.21.15.83:0/2208176825 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 6838 v62) v1 ==== 95+0+0 (secure 0 0 0) 0x7fc6a00576f0 con 0x7fc6b00b4a50 2024-03-06T07:27:33.608 INFO:teuthology.orchestra.run.smithi083.stderr:failed mds gid 6838 2024-03-06T07:27:33.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.610+0000 7ff06cff9640 1 -- 172.21.15.83:0/2477841688 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "c"}]=0 failed mds gid 6200 v62) v1 ==== 95+0+0 (secure 0 0 0) 0x7ff068010040 con 0x7ff07015bbf0 2024-03-06T07:27:33.611 INFO:teuthology.orchestra.run.smithi083.stderr:failed mds gid 6200 2024-03-06T07:27:33.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.610+0000 7fc6b4b81640 1 -- 172.21.15.83:0/2208176825 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc69006b900 msgr2=0x7fc69006ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.610+0000 7fc6b4b81640 1 --2- 172.21.15.83:0/2208176825 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc69006b900 0x7fc69006ddc0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4039000 tx=0x7fc6a4034000 comp rx=0 tx=0).stop 2024-03-06T07:27:33.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.610+0000 7fc6b4b81640 1 -- 172.21.15.83:0/2208176825 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6b00b4a50 msgr2=0x7fc6b0149250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.610+0000 7fc6b4b81640 1 --2- 172.21.15.83:0/2208176825 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6b00b4a50 0x7fc6b0149250 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fc6a000c480 tx=0x7fc6a000c940 comp rx=0 tx=0).stop 2024-03-06T07:27:33.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.610+0000 7fc6b4b81640 1 -- 172.21.15.83:0/2208176825 shutdown_connections 2024-03-06T07:27:33.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.610+0000 7fc6b4b81640 1 --2- 172.21.15.83:0/2208176825 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc69006b900 0x7fc69006ddc0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.610+0000 7fc6b4b81640 1 --2- 172.21.15.83:0/2208176825 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc6b0158100 0x7fc6b01595f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.610+0000 7fc6b4b81640 1 --2- 172.21.15.83:0/2208176825 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc6b00cc920 0x7fc6b0157bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.610+0000 7fc6b4b81640 1 --2- 172.21.15.83:0/2208176825 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6b00b4a50 0x7fc6b0149250 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.610+0000 7fc6b4b81640 1 -- 172.21.15.83:0/2208176825 >> 172.21.15.83:0/2208176825 conn(0x7fc6b00b1c80 msgr2=0x7fc6b006d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.610+0000 7fc6b4b81640 1 -- 172.21.15.83:0/2208176825 shutdown_connections 2024-03-06T07:27:33.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.610+0000 7fc6b4b81640 1 -- 172.21.15.83:0/2208176825 wait complete. 2024-03-06T07:27:33.614 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:27:33.614 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:27:33.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.612+0000 7ff0764a4640 1 -- 172.21.15.83:0/2477841688 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff05c06b520 msgr2=0x7ff05c06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.612+0000 7ff0764a4640 1 --2- 172.21.15.83:0/2477841688 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff05c06b520 0x7ff05c06d9e0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7ff060000c00 tx=0x7ff060008040 comp rx=0 tx=0).stop 2024-03-06T07:27:33.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.612+0000 7ff0764a4640 1 -- 172.21.15.83:0/2477841688 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff07015bbf0 msgr2=0x7ff07006cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.612+0000 7ff0764a4640 1 --2- 172.21.15.83:0/2477841688 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff07015bbf0 0x7ff07006cf80 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7ff06800efc0 tx=0x7ff06800c490 comp rx=0 tx=0).stop 2024-03-06T07:27:33.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.612+0000 7ff0764a4640 1 -- 172.21.15.83:0/2477841688 shutdown_connections 2024-03-06T07:27:33.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.612+0000 7ff0764a4640 1 --2- 172.21.15.83:0/2477841688 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff05c06b520 0x7ff05c06d9e0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.612+0000 7ff0764a4640 1 --2- 172.21.15.83:0/2477841688 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff07015bbf0 0x7ff07006cf80 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.612+0000 7ff0764a4640 1 --2- 172.21.15.83:0/2477841688 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff0700cebc0 0x7ff0701f28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.612+0000 7ff0764a4640 1 --2- 172.21.15.83:0/2477841688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0700cd7c0 0x7ff07014b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.612+0000 7ff0764a4640 1 -- 172.21.15.83:0/2477841688 >> 172.21.15.83:0/2477841688 conn(0x7ff0700b1c80 msgr2=0x7ff070151b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.612+0000 7ff0764a4640 1 -- 172.21.15.83:0/2477841688 shutdown_connections 2024-03-06T07:27:33.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.612+0000 7ff0764a4640 1 -- 172.21.15.83:0/2477841688 wait complete. 2024-03-06T07:27:33.617 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:27:33.617 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:27:33.618 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:27:33.618 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:33.620 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:27:33.621 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:33.626 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:27:33.626 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:27:33.627 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:27:33.627 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:27:33.628 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:27:33.629 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:33.667 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:27:33.668 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:33.668 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:33.692 INFO:tasks.ceph.mds.a.smithi083.stdout:starting mds.a at 2024-03-06T07:27:33.702 INFO:tasks.ceph.mds.b.smithi183.stdout:starting mds.b at 2024-03-06T07:27:33.708 INFO:tasks.ceph.mds.d.smithi183.stdout:starting mds.d at 2024-03-06T07:27:33.744 INFO:tasks.ceph.mds.c.smithi083.stdout:starting mds.c at 2024-03-06T07:27:33.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.768+0000 7f9de1b7a640 1 Processor -- start 2024-03-06T07:27:33.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.768+0000 7f9de1b7a640 1 -- start start 2024-03-06T07:27:33.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.769+0000 7f9de1b7a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f9ddc15bb20 0x7f9ddc0d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.769+0000 7f9de1b7a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ddc0cfff0 0x7f9ddc0d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.769+0000 7f9de1b7a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ddc161980 con 0x7f9ddc0ce4b0 2024-03-06T07:27:33.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.769+0000 7f9de1b7a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ddc161c60 con 0x7f9ddc15bb20 2024-03-06T07:27:33.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.769+0000 7f9de1b7a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ddc161f40 con 0x7f9ddc0cfff0 2024-03-06T07:27:33.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.769+0000 7f9ddaffd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ddc0cfff0 0x7f9ddc0d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.769+0000 7f9ddb7fe640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f9ddc15bb20 0x7f9ddc0d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.769+0000 7f9ddaffd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ddc0cfff0 0x7f9ddc0d0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:54562/0 (socket says 172.21.15.83:54562) 2024-03-06T07:27:33.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.769+0000 7f9ddaffd640 1 -- 172.21.15.83:0/3513599322 learned_addr learned my addr 172.21.15.83:0/3513599322 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:33.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.770+0000 7f9ddaffd640 1 -- 172.21.15.83:0/3513599322 >> v2:172.21.15.183:3300/0 conn(0x7f9ddc15bb20 msgr2=0x7f9ddc0d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:33.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.770+0000 7f9ddaffd640 1 --2- 172.21.15.83:0/3513599322 >> v2:172.21.15.183:3300/0 conn(0x7f9ddc15bb20 0x7f9ddc0d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.770+0000 7f9ddaffd640 1 -- 172.21.15.83:0/3513599322 >> v1:172.21.15.183:6789/0 conn(0x7f9ddc0ce4b0 legacy=0x7f9ddc0ce940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.770+0000 7f9ddaffd640 1 -- 172.21.15.83:0/3513599322 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ddc162220 con 0x7f9ddc0cfff0 2024-03-06T07:27:33.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.770+0000 7f9ddaffd640 1 --2- 172.21.15.83:0/3513599322 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ddc0cfff0 0x7f9ddc0d0470 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0009870 tx=0x7f9dd002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46ad3a0512c4a38a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.771+0000 7f9dda7fc640 1 -- 172.21.15.83:0/3513599322 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd0034070 con 0x7f9ddc0cfff0 2024-03-06T07:27:33.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.771+0000 7f9dda7fc640 1 -- 172.21.15.83:0/3513599322 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dd0030840 con 0x7f9ddc0cfff0 2024-03-06T07:27:33.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.771+0000 7f9dda7fc640 1 -- 172.21.15.83:0/3513599322 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd0030c10 con 0x7f9ddc0cfff0 2024-03-06T07:27:33.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.771+0000 7f9de1b7a640 1 -- 172.21.15.83:0/3513599322 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ddc0cfff0 msgr2=0x7f9ddc0d0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.771+0000 7f9de1b7a640 1 --2- 172.21.15.83:0/3513599322 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ddc0cfff0 0x7f9ddc0d0470 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0009870 tx=0x7f9dd002b980 comp rx=0 tx=0).stop 2024-03-06T07:27:33.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.771+0000 7f9de1b7a640 1 -- 172.21.15.83:0/3513599322 shutdown_connections 2024-03-06T07:27:33.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.771+0000 7f9de1b7a640 1 --2- 172.21.15.83:0/3513599322 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ddc0cfff0 0x7f9ddc0d0470 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.771+0000 7f9de1b7a640 1 --2- 172.21.15.83:0/3513599322 >> v2:172.21.15.183:3300/0 conn(0x7f9ddc15bb20 0x7f9ddc0d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.771+0000 7f9de1b7a640 1 -- 172.21.15.83:0/3513599322 >> 172.21.15.83:0/3513599322 conn(0x7f9ddc09a9f0 msgr2=0x7f9ddc09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.771+0000 7f9de1b7a640 1 -- 172.21.15.83:0/3513599322 shutdown_connections 2024-03-06T07:27:33.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.771+0000 7f9de1b7a640 1 -- 172.21.15.83:0/3513599322 wait complete. 2024-03-06T07:27:33.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.772+0000 7f9de1b7a640 1 Processor -- start 2024-03-06T07:27:33.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.772+0000 7f9de1b7a640 1 -- start start 2024-03-06T07:27:33.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.773+0000 7f9de1b7a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ddc0ce4b0 0x7f9ddc15aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.773+0000 7f9de1b7a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ddc0cfff0 0x7f9ddc06ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.773+0000 7f9de1b7a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9ddc15bb20 0x7f9ddc06d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.773+0000 7f9de1b7a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ddc0d24b0 con 0x7f9ddc0ce4b0 2024-03-06T07:27:33.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.773+0000 7f9de1b7a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ddc0d2330 con 0x7f9ddc15bb20 2024-03-06T07:27:33.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.773+0000 7f9de1b7a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ddc0d1980 con 0x7f9ddc0cfff0 2024-03-06T07:27:33.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.773+0000 7f9ddb7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ddc0ce4b0 0x7f9ddc15aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.773+0000 7f9ddb7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ddc0ce4b0 0x7f9ddc15aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54022/0 (socket says 172.21.15.83:54022) 2024-03-06T07:27:33.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.773+0000 7f9ddb7fe640 1 -- 172.21.15.83:0/2546026095 learned_addr learned my addr 172.21.15.83:0/2546026095 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:33.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.773+0000 7f9ddaffd640 1 --2- 172.21.15.83:0/2546026095 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ddc0cfff0 0x7f9ddc06ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.773+0000 7f9ddbfff640 1 --2- 172.21.15.83:0/2546026095 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9ddc15bb20 0x7f9ddc06d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.774+0000 7f9ddb7fe640 1 -- 172.21.15.83:0/2546026095 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ddc0cfff0 msgr2=0x7f9ddc06ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.774+0000 7f9ddb7fe640 1 --2- 172.21.15.83:0/2546026095 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ddc0cfff0 0x7f9ddc06ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.774+0000 7f9ddb7fe640 1 -- 172.21.15.83:0/2546026095 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9ddc15bb20 msgr2=0x7f9ddc06d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.774+0000 7f9ddb7fe640 1 --2- 172.21.15.83:0/2546026095 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9ddc15bb20 0x7f9ddc06d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.774+0000 7f9ddb7fe640 1 -- 172.21.15.83:0/2546026095 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ddc15ef70 con 0x7f9ddc0ce4b0 2024-03-06T07:27:33.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.774+0000 7f9ddb7fe640 1 --2- 172.21.15.83:0/2546026095 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ddc0ce4b0 0x7f9ddc15aba0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc00e870 tx=0x7f9dcc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.774+0000 7f9dc3fff640 1 -- 172.21.15.83:0/2546026095 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dcc017070 con 0x7f9ddc0ce4b0 2024-03-06T07:27:33.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.774+0000 7f9dc3fff640 1 -- 172.21.15.83:0/2546026095 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dcc00c940 con 0x7f9ddc0ce4b0 2024-03-06T07:27:33.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.774+0000 7f9de1b7a640 1 -- 172.21.15.83:0/2546026095 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ddc15f250 con 0x7f9ddc0ce4b0 2024-03-06T07:27:33.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.774+0000 7f9dc3fff640 1 -- 172.21.15.83:0/2546026095 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dcc00cd30 con 0x7f9ddc0ce4b0 2024-03-06T07:27:33.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.774+0000 7f9de1b7a640 1 -- 172.21.15.83:0/2546026095 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ddc15f810 con 0x7f9ddc0ce4b0 2024-03-06T07:27:33.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.775+0000 7f9de1b7a640 1 -- 172.21.15.83:0/2546026095 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ddc04f330 con 0x7f9ddc0ce4b0 2024-03-06T07:27:33.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.775+0000 7f9dc3fff640 1 -- 172.21.15.83:0/2546026095 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9dcc01a070 con 0x7f9ddc0ce4b0 2024-03-06T07:27:33.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.776+0000 7f9dc3fff640 1 --2- 172.21.15.83:0/2546026095 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9dc806b4d0 0x7f9dc806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.776+0000 7f9dc3fff640 1 -- 172.21.15.83:0/2546026095 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f9dcc08a3f0 con 0x7f9ddc0ce4b0 2024-03-06T07:27:33.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.776+0000 7f9ddaffd640 1 --2- 172.21.15.83:0/2546026095 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9dc806b4d0 0x7f9dc806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.776+0000 7f9ddaffd640 1 --2- 172.21.15.83:0/2546026095 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9dc806b4d0 0x7f9dc806d990 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0036000 tx=0x7f9dd0008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.780+0000 7f9dc3fff640 1 -- 172.21.15.83:0/2546026095 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9dcc058c00 con 0x7f9ddc0ce4b0 2024-03-06T07:27:33.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.989+0000 7f9de1b7a640 1 -- 172.21.15.83:0/2546026095 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ddc0b1820 con 0x7f9ddc0ce4b0 2024-03-06T07:27:33.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.989+0000 7f9dc3fff640 1 -- 172.21.15.83:0/2546026095 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+1802 (secure 0 0 0) 0x7f9dcc05cca0 con 0x7f9ddc0ce4b0 2024-03-06T07:27:33.991 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:33.992 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":62,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.491022+0000","modified":"2024-03-06T07:27:33.605187+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":8}]} 2024-03-06T07:27:33.992 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 62 2024-03-06T07:27:33.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.991+0000 7f9de1b7a640 1 -- 172.21.15.83:0/2546026095 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9dc806b4d0 msgr2=0x7f9dc806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.991+0000 7f9de1b7a640 1 --2- 172.21.15.83:0/2546026095 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9dc806b4d0 0x7f9dc806d990 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0036000 tx=0x7f9dd0008030 comp rx=0 tx=0).stop 2024-03-06T07:27:33.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.991+0000 7f9de1b7a640 1 -- 172.21.15.83:0/2546026095 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ddc0ce4b0 msgr2=0x7f9ddc15aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.991+0000 7f9de1b7a640 1 --2- 172.21.15.83:0/2546026095 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ddc0ce4b0 0x7f9ddc15aba0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc00e870 tx=0x7f9dcc00ed30 comp rx=0 tx=0).stop 2024-03-06T07:27:33.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.992+0000 7f9de1b7a640 1 -- 172.21.15.83:0/2546026095 shutdown_connections 2024-03-06T07:27:33.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.992+0000 7f9de1b7a640 1 --2- 172.21.15.83:0/2546026095 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9dc806b4d0 0x7f9dc806d990 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.992+0000 7f9de1b7a640 1 --2- 172.21.15.83:0/2546026095 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9ddc15bb20 0x7f9ddc06d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.992+0000 7f9de1b7a640 1 --2- 172.21.15.83:0/2546026095 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ddc0cfff0 0x7f9ddc06ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.992+0000 7f9de1b7a640 1 --2- 172.21.15.83:0/2546026095 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ddc0ce4b0 0x7f9ddc15aba0 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.992+0000 7f9de1b7a640 1 -- 172.21.15.83:0/2546026095 >> 172.21.15.83:0/2546026095 conn(0x7f9ddc09a9f0 msgr2=0x7f9ddc0cd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.992+0000 7f9de1b7a640 1 -- 172.21.15.83:0/2546026095 shutdown_connections 2024-03-06T07:27:33.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:33.992+0000 7f9de1b7a640 1 -- 172.21.15.83:0/2546026095 wait complete. 2024-03-06T07:27:34.409 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 62, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:10.491022+0000', 'modified': '2024-03-06T07:27:33.605187+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 114, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:27:34.409 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-03-06T07:27:35.410 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:35.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.519+0000 7fe1933bc640 1 Processor -- start 2024-03-06T07:27:35.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.520+0000 7fe1933bc640 1 -- start start 2024-03-06T07:27:35.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.520+0000 7fe1933bc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe18c0d22b0 0x7fe18c0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.520+0000 7fe1933bc640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fe18c0b41f0 0x7fe18c0b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.520+0000 7fe1933bc640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe18c0b3430 con 0x7fe18c15dfe0 2024-03-06T07:27:35.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.520+0000 7fe1933bc640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe18c0d2c50 con 0x7fe18c0d22b0 2024-03-06T07:27:35.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.520+0000 7fe1933bc640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe18c0b31e0 con 0x7fe18c0b41f0 2024-03-06T07:27:35.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.521+0000 7fe190957640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe18c0d22b0 0x7fe18c0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.521+0000 7fe190957640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe18c0d22b0 0x7fe18c0d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54038/0 (socket says 172.21.15.83:54038) 2024-03-06T07:27:35.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.521+0000 7fe191158640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fe18c15dfe0 0x7fe18c0cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:56020/0 (socket says 172.21.15.83:56020) 2024-03-06T07:27:35.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.521+0000 7fe190957640 1 -- 172.21.15.83:0/1238698475 learned_addr learned my addr 172.21.15.83:0/1238698475 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:35.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.521+0000 7fe191959640 1 --2- 172.21.15.83:0/1238698475 >> v2:172.21.15.183:3300/0 conn(0x7fe18c0b41f0 0x7fe18c0b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.521+0000 7fe190957640 1 -- 172.21.15.83:0/1238698475 >> v2:172.21.15.183:3300/0 conn(0x7fe18c0b41f0 msgr2=0x7fe18c0b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:35.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.521+0000 7fe190957640 1 --2- 172.21.15.83:0/1238698475 >> v2:172.21.15.183:3300/0 conn(0x7fe18c0b41f0 0x7fe18c0b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.521+0000 7fe190957640 1 -- 172.21.15.83:0/1238698475 >> v1:172.21.15.83:6789/0 conn(0x7fe18c15dfe0 legacy=0x7fe18c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.521+0000 7fe190957640 1 -- 172.21.15.83:0/1238698475 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe18c0b3710 con 0x7fe18c0d22b0 2024-03-06T07:27:35.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.521+0000 7fe183fff640 1 -- 172.21.15.83:0/1238698475 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 156196425 0 0) 0x7fe18c0b3430 con 0x7fe18c15dfe0 2024-03-06T07:27:35.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.521+0000 7fe191959640 1 --2- 172.21.15.83:0/1238698475 >> v2:172.21.15.183:3300/0 conn(0x7fe18c0b41f0 0x7fe18c0b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:35.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.522+0000 7fe190957640 1 --2- 172.21.15.83:0/1238698475 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe18c0d22b0 0x7fe18c0d2710 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7fe17c009870 tx=0x7fe17c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a4fb106c2a0d233 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:35.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.522+0000 7fe183fff640 1 -- 172.21.15.83:0/1238698475 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe17c034070 con 0x7fe18c0d22b0 2024-03-06T07:27:35.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.522+0000 7fe183fff640 1 -- 172.21.15.83:0/1238698475 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe17c030860 con 0x7fe18c0d22b0 2024-03-06T07:27:35.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.522+0000 7fe183fff640 1 -- 172.21.15.83:0/1238698475 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe17c030c30 con 0x7fe18c0d22b0 2024-03-06T07:27:35.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.522+0000 7fe1933bc640 1 -- 172.21.15.83:0/1238698475 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe18c0d22b0 msgr2=0x7fe18c0d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.522+0000 7fe1933bc640 1 --2- 172.21.15.83:0/1238698475 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe18c0d22b0 0x7fe18c0d2710 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7fe17c009870 tx=0x7fe17c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.522+0000 7fe1933bc640 1 -- 172.21.15.83:0/1238698475 shutdown_connections 2024-03-06T07:27:35.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.522+0000 7fe1933bc640 1 --2- 172.21.15.83:0/1238698475 >> v2:172.21.15.183:3300/0 conn(0x7fe18c0b41f0 0x7fe18c0b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.522+0000 7fe1933bc640 1 --2- 172.21.15.83:0/1238698475 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe18c0d22b0 0x7fe18c0d2710 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.522+0000 7fe1933bc640 1 -- 172.21.15.83:0/1238698475 >> 172.21.15.83:0/1238698475 conn(0x7fe18c0d3cd0 msgr2=0x7fe18c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:35.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.523+0000 7fe1933bc640 1 -- 172.21.15.83:0/1238698475 shutdown_connections 2024-03-06T07:27:35.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.523+0000 7fe1933bc640 1 -- 172.21.15.83:0/1238698475 wait complete. 2024-03-06T07:27:35.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.524+0000 7fe1933bc640 1 Processor -- start 2024-03-06T07:27:35.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.524+0000 7fe1933bc640 1 -- start start 2024-03-06T07:27:35.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.524+0000 7fe1933bc640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe18c0b41f0 0x7fe18c15e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.524+0000 7fe1933bc640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe18c0d22b0 0x7fe18c0b6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.524+0000 7fe1933bc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe18c15dfe0 0x7fe18c0b7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.524+0000 7fe1933bc640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe18c0cfa00 con 0x7fe18c15dfe0 2024-03-06T07:27:35.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.524+0000 7fe1933bc640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe18c0b3d60 con 0x7fe18c0d22b0 2024-03-06T07:27:35.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.524+0000 7fe1933bc640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe18c0d1c70 con 0x7fe18c0b41f0 2024-03-06T07:27:35.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.524+0000 7fe191959640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe18c15dfe0 0x7fe18c0b7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.525+0000 7fe191959640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe18c15dfe0 0x7fe18c0b7050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54050/0 (socket says 172.21.15.83:54050) 2024-03-06T07:27:35.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.525+0000 7fe191959640 1 -- 172.21.15.83:0/1656223965 learned_addr learned my addr 172.21.15.83:0/1656223965 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:35.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.525+0000 7fe191158640 1 --2- 172.21.15.83:0/1656223965 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe18c0b41f0 0x7fe18c15e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.525+0000 7fe190957640 1 --2- 172.21.15.83:0/1656223965 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe18c0d22b0 0x7fe18c0b6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.525+0000 7fe191959640 1 -- 172.21.15.83:0/1656223965 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe18c0b41f0 msgr2=0x7fe18c15e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.525+0000 7fe191959640 1 --2- 172.21.15.83:0/1656223965 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe18c0b41f0 0x7fe18c15e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.525+0000 7fe191959640 1 -- 172.21.15.83:0/1656223965 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe18c0d22b0 msgr2=0x7fe18c0b6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.525+0000 7fe191959640 1 --2- 172.21.15.83:0/1656223965 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe18c0d22b0 0x7fe18c0b6b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.525+0000 7fe191959640 1 -- 172.21.15.83:0/1656223965 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe18c15e900 con 0x7fe18c15dfe0 2024-03-06T07:27:35.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.525+0000 7fe191158640 1 --2- 172.21.15.83:0/1656223965 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe18c0b41f0 0x7fe18c15e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:35.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.525+0000 7fe191959640 1 --2- 172.21.15.83:0/1656223965 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe18c15dfe0 0x7fe18c0b7050 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7fe18400cef0 tx=0x7fe18400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:35.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.525+0000 7fe181ffb640 1 -- 172.21.15.83:0/1656223965 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe18401a070 con 0x7fe18c15dfe0 2024-03-06T07:27:35.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.525+0000 7fe181ffb640 1 -- 172.21.15.83:0/1656223965 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe18400d860 con 0x7fe18c15dfe0 2024-03-06T07:27:35.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.525+0000 7fe1933bc640 1 -- 172.21.15.83:0/1656223965 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe18c15ec30 con 0x7fe18c15dfe0 2024-03-06T07:27:35.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.525+0000 7fe181ffb640 1 -- 172.21.15.83:0/1656223965 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe18400dc50 con 0x7fe18c15dfe0 2024-03-06T07:27:35.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.525+0000 7fe1933bc640 1 -- 172.21.15.83:0/1656223965 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe18c15f470 con 0x7fe18c15dfe0 2024-03-06T07:27:35.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.526+0000 7fe1933bc640 1 -- 172.21.15.83:0/1656223965 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe18c04f330 con 0x7fe18c15dfe0 2024-03-06T07:27:35.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.527+0000 7fe181ffb640 1 -- 172.21.15.83:0/1656223965 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe18401d070 con 0x7fe18c15dfe0 2024-03-06T07:27:35.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.527+0000 7fe181ffb640 1 --2- 172.21.15.83:0/1656223965 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe17006b4d0 0x7fe17006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.527+0000 7fe181ffb640 1 -- 172.21.15.83:0/1656223965 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fe18408aaf0 con 0x7fe18c15dfe0 2024-03-06T07:27:35.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.527+0000 7fe191158640 1 --2- 172.21.15.83:0/1656223965 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe17006b4d0 0x7fe17006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.527+0000 7fe191158640 1 --2- 172.21.15.83:0/1656223965 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe17006b4d0 0x7fe17006d990 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fe188000f60 tx=0x7fe188007ca0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:35.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.531+0000 7fe181ffb640 1 -- 172.21.15.83:0/1656223965 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe18c15ec30 con 0x7fe18c15dfe0 2024-03-06T07:27:35.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.739+0000 7fe1933bc640 1 -- 172.21.15.83:0/1656223965 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe18c0b1e60 con 0x7fe18c15dfe0 2024-03-06T07:27:35.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.740+0000 7fe181ffb640 1 -- 172.21.15.83:0/1656223965 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+4980 (secure 0 0 0) 0x7fe184059a10 con 0x7fe18c15dfe0 2024-03-06T07:27:35.742 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:35.742 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":64,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7051,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/101642171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":101642171},{"type":"v1","addr":"172.21.15.183:6833","nonce":101642171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7057,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3146884900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":3146884900},{"type":"v1","addr":"172.21.15.183:6835","nonce":3146884900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8208,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1922041621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1922041621},{"type":"v1","addr":"172.21.15.83:6835","nonce":1922041621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.491022+0000","modified":"2024-03-06T07:27:34.011802+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8229":{"gid":8229,"name":"c","rank":0,"incarnation":63,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.83:6837/2003687153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2003687153},{"type":"v1","addr":"172.21.15.83:6837","nonce":2003687153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":8}]} 2024-03-06T07:27:35.742 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 64 2024-03-06T07:27:35.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.742+0000 7fe1933bc640 1 -- 172.21.15.83:0/1656223965 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe17006b4d0 msgr2=0x7fe17006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.742+0000 7fe1933bc640 1 --2- 172.21.15.83:0/1656223965 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe17006b4d0 0x7fe17006d990 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fe188000f60 tx=0x7fe188007ca0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.742+0000 7fe1933bc640 1 -- 172.21.15.83:0/1656223965 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe18c15dfe0 msgr2=0x7fe18c0b7050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.742+0000 7fe1933bc640 1 --2- 172.21.15.83:0/1656223965 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe18c15dfe0 0x7fe18c0b7050 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7fe18400cef0 tx=0x7fe18400a710 comp rx=0 tx=0).stop 2024-03-06T07:27:35.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.742+0000 7fe1933bc640 1 -- 172.21.15.83:0/1656223965 shutdown_connections 2024-03-06T07:27:35.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.742+0000 7fe1933bc640 1 --2- 172.21.15.83:0/1656223965 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe17006b4d0 0x7fe17006d990 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.742+0000 7fe1933bc640 1 --2- 172.21.15.83:0/1656223965 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe18c15dfe0 0x7fe18c0b7050 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.742+0000 7fe1933bc640 1 --2- 172.21.15.83:0/1656223965 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe18c0d22b0 0x7fe18c0b6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.742+0000 7fe1933bc640 1 --2- 172.21.15.83:0/1656223965 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe18c0b41f0 0x7fe18c15e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.742+0000 7fe1933bc640 1 -- 172.21.15.83:0/1656223965 >> 172.21.15.83:0/1656223965 conn(0x7fe18c0d3cd0 msgr2=0x7fe18c06cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:35.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.742+0000 7fe1933bc640 1 -- 172.21.15.83:0/1656223965 shutdown_connections 2024-03-06T07:27:35.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:35.742+0000 7fe1933bc640 1 -- 172.21.15.83:0/1656223965 wait complete. 2024-03-06T07:27:36.158 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:10.491022+0000', 'modified': '2024-03-06T07:27:34.011802+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 114, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8229}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8229': {'gid': 8229, 'name': 'c', 'rank': 0, 'incarnation': 63, 'state': 'up:reconnect', 'state_seq': 2, 'addr': '172.21.15.83:6837/2003687153', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6836', 'nonce': 2003687153}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 2003687153}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:27:36.158 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8229:up:reconnect 2024-03-06T07:27:37.159 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:37.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.258+0000 7f98dc09a640 1 Processor -- start 2024-03-06T07:27:37.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.259+0000 7f98dc09a640 1 -- start start 2024-03-06T07:27:37.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.259+0000 7f98dc09a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f98d40d2710 0x7f98d40b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.259+0000 7f98dc09a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d40b3430 con 0x7f98d415dfe0 2024-03-06T07:27:37.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.259+0000 7f98dc09a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d40b3710 con 0x7f98d40d22b0 2024-03-06T07:27:37.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.259+0000 7f98dc09a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98d40b31e0 con 0x7f98d40d2710 2024-03-06T07:27:37.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.259+0000 7f98d9e36640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f98d415dfe0 0x7f98d40cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43938/0 (socket says 172.21.15.83:43938) 2024-03-06T07:27:37.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.259+0000 7f98d9e36640 1 -- 172.21.15.83:0/2186469001 learned_addr learned my addr 172.21.15.83:0/2186469001 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:27:37.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.260+0000 7f98da637640 1 --2- 172.21.15.83:0/2186469001 >> v2:172.21.15.183:3300/0 conn(0x7f98d40d2710 0x7f98d40b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.260+0000 7f98d8e34640 1 -- 172.21.15.83:0/2186469001 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2407981800 0 0) 0x7f98d40b3430 con 0x7f98d415dfe0 2024-03-06T07:27:37.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.260+0000 7f98d8e34640 1 -- 172.21.15.83:0/2186469001 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98c0003520 con 0x7f98d415dfe0 2024-03-06T07:27:37.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.260+0000 7f98d8e34640 1 -- 172.21.15.83:0/2186469001 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3025207564 0 0) 0x7f98c0003520 con 0x7f98d415dfe0 2024-03-06T07:27:37.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.260+0000 7f98d8e34640 1 -- 172.21.15.83:0/2186469001 >> v2:172.21.15.183:3300/0 conn(0x7f98d40d2710 msgr2=0x7f98d40b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:37.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.260+0000 7f98d8e34640 1 --2- 172.21.15.83:0/2186469001 >> v2:172.21.15.183:3300/0 conn(0x7f98d40d2710 0x7f98d40b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.260+0000 7f98d8e34640 1 -- 172.21.15.83:0/2186469001 >> v1:172.21.15.183:6789/0 conn(0x7f98d40d22b0 legacy=0x7f98d40b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.260+0000 7f98d8e34640 1 -- 172.21.15.83:0/2186469001 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d40b39f0 con 0x7f98d415dfe0 2024-03-06T07:27:37.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.260+0000 7f98d8e34640 1 -- 172.21.15.83:0/2186469001 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f98c4001670 con 0x7f98d415dfe0 2024-03-06T07:27:37.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.260+0000 7f98d8e34640 1 -- 172.21.15.83:0/2186469001 >> v1:172.21.15.83:6789/0 conn(0x7f98d415dfe0 legacy=0x7f98d40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.260+0000 7f98d8e34640 1 --2- 172.21.15.83:0/2186469001 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98c00039e0 0x7f98c0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.260+0000 7f98d8e34640 1 --2- 172.21.15.83:0/2186469001 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98c0004800 0x7f98c0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.260+0000 7f98d8e34640 1 --2- 172.21.15.83:0/2186469001 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98c00051c0 0x7f98c000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.260+0000 7f98d8e34640 1 -- 172.21.15.83:0/2186469001 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d40b31e0 con 0x7f98c00039e0 2024-03-06T07:27:37.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.260+0000 7f98d8e34640 1 -- 172.21.15.83:0/2186469001 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c000c090 con 0x7f98c0004800 2024-03-06T07:27:37.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.260+0000 7f98d8e34640 1 -- 172.21.15.83:0/2186469001 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98c000c290 con 0x7f98c00051c0 2024-03-06T07:27:37.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.260+0000 7f98d9e36640 1 --2- 172.21.15.83:0/2186469001 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98c00039e0 0x7f98c0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.260+0000 7f98d9635640 1 --2- 172.21.15.83:0/2186469001 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98c0004800 0x7f98c0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.260+0000 7f98da637640 1 --2- 172.21.15.83:0/2186469001 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98c00051c0 0x7f98c000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.260+0000 7f98d9e36640 1 -- 172.21.15.83:0/2186469001 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98c00051c0 msgr2=0x7f98c000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:37.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.260+0000 7f98d9e36640 1 --2- 172.21.15.83:0/2186469001 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98c00051c0 0x7f98c000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.260+0000 7f98d9e36640 1 -- 172.21.15.83:0/2186469001 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98c0004800 msgr2=0x7f98c0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:37.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.260+0000 7f98d9e36640 1 --2- 172.21.15.83:0/2186469001 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98c0004800 0x7f98c0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.261+0000 7f98d9e36640 1 -- 172.21.15.83:0/2186469001 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c0003520 con 0x7f98c00039e0 2024-03-06T07:27:37.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.261+0000 7f98da637640 1 -- 172.21.15.83:0/2186469001 reap_dead start 2024-03-06T07:27:37.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.261+0000 7f98d9635640 1 --2- 172.21.15.83:0/2186469001 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98c0004800 0x7f98c0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:37.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.261+0000 7f98d9e36640 1 --2- 172.21.15.83:0/2186469001 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98c00039e0 0x7f98c0003e00 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f98c400c240 tx=0x7f98c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8ee3c5c9dc478e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.261+0000 7f98d8e34640 1 -- 172.21.15.83:0/2186469001 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c4007d10 con 0x7f98c00039e0 2024-03-06T07:27:37.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.261+0000 7f98d8e34640 1 -- 172.21.15.83:0/2186469001 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98c4030070 con 0x7f98c00039e0 2024-03-06T07:27:37.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.261+0000 7f98d8e34640 1 -- 172.21.15.83:0/2186469001 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c4005d90 con 0x7f98c00039e0 2024-03-06T07:27:37.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.262+0000 7f98dc09a640 1 -- 172.21.15.83:0/2186469001 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98c00039e0 msgr2=0x7f98c0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.262+0000 7f98dc09a640 1 --2- 172.21.15.83:0/2186469001 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98c00039e0 0x7f98c0003e00 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f98c400c240 tx=0x7f98c0005830 comp rx=0 tx=0).stop 2024-03-06T07:27:37.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.262+0000 7f98dc09a640 1 -- 172.21.15.83:0/2186469001 shutdown_connections 2024-03-06T07:27:37.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.262+0000 7f98dc09a640 1 --2- 172.21.15.83:0/2186469001 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98c00039e0 0x7f98c0003e00 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.262+0000 7f98dc09a640 1 -- 172.21.15.83:0/2186469001 >> 172.21.15.83:0/2186469001 conn(0x7f98d40d3cd0 msgr2=0x7f98d40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:37.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.262+0000 7f98dc09a640 1 -- 172.21.15.83:0/2186469001 shutdown_connections 2024-03-06T07:27:37.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.262+0000 7f98dc09a640 1 -- 172.21.15.83:0/2186469001 wait complete. 2024-03-06T07:27:37.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.263+0000 7f98dc09a640 1 Processor -- start 2024-03-06T07:27:37.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.263+0000 7f98dc09a640 1 -- start start 2024-03-06T07:27:37.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.264+0000 7f98dc09a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98c00039e0 0x7f98d414c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.264+0000 7f98dc09a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98d414eb80 0x7f98d414d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.264+0000 7f98dc09a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98d414d5d0 0x7f98d414f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.264+0000 7f98dc09a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d40d1c70 con 0x7f98d414d5d0 2024-03-06T07:27:37.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.264+0000 7f98dc09a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d40b3d60 con 0x7f98d414eb80 2024-03-06T07:27:37.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.264+0000 7f98dc09a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98d40d1df0 con 0x7f98c00039e0 2024-03-06T07:27:37.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.264+0000 7f98da637640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98d414d5d0 0x7f98d414f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.264+0000 7f98da637640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98d414d5d0 0x7f98d414f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46578/0 (socket says 172.21.15.83:46578) 2024-03-06T07:27:37.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.264+0000 7f98da637640 1 -- 172.21.15.83:0/1365751499 learned_addr learned my addr 172.21.15.83:0/1365751499 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:37.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.264+0000 7f98d9e36640 1 --2- 172.21.15.83:0/1365751499 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98c00039e0 0x7f98d414c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.264+0000 7f98d9635640 1 --2- 172.21.15.83:0/1365751499 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98d414eb80 0x7f98d414d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.264+0000 7f98da637640 1 -- 172.21.15.83:0/1365751499 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98c00039e0 msgr2=0x7f98d414c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.264+0000 7f98da637640 1 --2- 172.21.15.83:0/1365751499 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98c00039e0 0x7f98d414c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.264+0000 7f98da637640 1 -- 172.21.15.83:0/1365751499 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98d414eb80 msgr2=0x7f98d414d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.264+0000 7f98da637640 1 --2- 172.21.15.83:0/1365751499 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98d414eb80 0x7f98d414d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.264+0000 7f98da637640 1 -- 172.21.15.83:0/1365751499 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d414dae0 con 0x7f98d414d5d0 2024-03-06T07:27:37.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.264+0000 7f98d9e36640 1 --2- 172.21.15.83:0/1365751499 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98c00039e0 0x7f98d414c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:37.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.265+0000 7f98da637640 1 --2- 172.21.15.83:0/1365751499 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98d414d5d0 0x7f98d414f120 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f98d4161bf0 tx=0x7f98cc0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.265+0000 7f98ca7fc640 1 -- 172.21.15.83:0/1365751499 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98cc011de0 con 0x7f98d414d5d0 2024-03-06T07:27:37.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.265+0000 7f98ca7fc640 1 -- 172.21.15.83:0/1365751499 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98cc01b070 con 0x7f98d414d5d0 2024-03-06T07:27:37.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.265+0000 7f98dc09a640 1 -- 172.21.15.83:0/1365751499 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98d414ddc0 con 0x7f98d414d5d0 2024-03-06T07:27:37.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.265+0000 7f98ca7fc640 1 -- 172.21.15.83:0/1365751499 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98cc003990 con 0x7f98d414d5d0 2024-03-06T07:27:37.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.265+0000 7f98dc09a640 1 -- 172.21.15.83:0/1365751499 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98d414e380 con 0x7f98d414d5d0 2024-03-06T07:27:37.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.266+0000 7f98dc09a640 1 -- 172.21.15.83:0/1365751499 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98d404f330 con 0x7f98d414d5d0 2024-03-06T07:27:37.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.267+0000 7f98ca7fc640 1 -- 172.21.15.83:0/1365751499 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f98cc003b30 con 0x7f98d414d5d0 2024-03-06T07:27:37.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.267+0000 7f98ca7fc640 1 --2- 172.21.15.83:0/1365751499 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f98b806b4d0 0x7f98b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.267+0000 7f98d9e36640 1 --2- 172.21.15.83:0/1365751499 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f98b806b4d0 0x7f98b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.267+0000 7f98ca7fc640 1 -- 172.21.15.83:0/1365751499 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f98cc08aa00 con 0x7f98d414d5d0 2024-03-06T07:27:37.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.268+0000 7f98d9e36640 1 --2- 172.21.15.83:0/1365751499 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f98b806b4d0 0x7f98b806d990 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f98c4034000 tx=0x7f98c4039040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.272+0000 7f98ca7fc640 1 -- 172.21.15.83:0/1365751499 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f98cc003e20 con 0x7f98d414d5d0 2024-03-06T07:27:37.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.485+0000 7f98dc09a640 1 -- 172.21.15.83:0/1365751499 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98d40b5550 con 0x7f98d414d5d0 2024-03-06T07:27:37.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.486+0000 7f98ca7fc640 1 -- 172.21.15.83:0/1365751499 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f98cc059860 con 0x7f98d414d5d0 2024-03-06T07:27:37.488 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:37.488 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7051,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/101642171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":101642171},{"type":"v1","addr":"172.21.15.183:6833","nonce":101642171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7057,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3146884900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":3146884900},{"type":"v1","addr":"172.21.15.183:6835","nonce":3146884900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8208,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1922041621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1922041621},{"type":"v1","addr":"172.21.15.83:6835","nonce":1922041621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.491022+0000","modified":"2024-03-06T07:27:37.006649+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8229":{"gid":8229,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/2003687153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2003687153},{"type":"v1","addr":"172.21.15.83:6837","nonce":2003687153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8229,"qdb_cluster":[8229]},"id":8}]} 2024-03-06T07:27:37.488 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 66 2024-03-06T07:27:37.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.487+0000 7f98dc09a640 1 -- 172.21.15.83:0/1365751499 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f98b806b4d0 msgr2=0x7f98b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.487+0000 7f98dc09a640 1 --2- 172.21.15.83:0/1365751499 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f98b806b4d0 0x7f98b806d990 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f98c4034000 tx=0x7f98c4039040 comp rx=0 tx=0).stop 2024-03-06T07:27:37.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.487+0000 7f98dc09a640 1 -- 172.21.15.83:0/1365751499 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98d414d5d0 msgr2=0x7f98d414f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.487+0000 7f98dc09a640 1 --2- 172.21.15.83:0/1365751499 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98d414d5d0 0x7f98d414f120 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f98d4161bf0 tx=0x7f98cc0115a0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.488+0000 7f98dc09a640 1 -- 172.21.15.83:0/1365751499 shutdown_connections 2024-03-06T07:27:37.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.488+0000 7f98dc09a640 1 --2- 172.21.15.83:0/1365751499 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f98b806b4d0 0x7f98b806d990 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.488+0000 7f98dc09a640 1 --2- 172.21.15.83:0/1365751499 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98d414d5d0 0x7f98d414f120 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.488+0000 7f98dc09a640 1 --2- 172.21.15.83:0/1365751499 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98d414eb80 0x7f98d414d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.488+0000 7f98dc09a640 1 --2- 172.21.15.83:0/1365751499 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f98c00039e0 0x7f98d414c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.488+0000 7f98dc09a640 1 -- 172.21.15.83:0/1365751499 >> 172.21.15.83:0/1365751499 conn(0x7f98d40d3cd0 msgr2=0x7f98d40b40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:37.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.488+0000 7f98dc09a640 1 -- 172.21.15.83:0/1365751499 shutdown_connections 2024-03-06T07:27:37.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:37.488+0000 7f98dc09a640 1 -- 172.21.15.83:0/1365751499 wait complete. 2024-03-06T07:27:37.902 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 66, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:10.491022+0000', 'modified': '2024-03-06T07:27:37.006649+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 114, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8229}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8229': {'gid': 8229, 'name': 'c', 'rank': 0, 'incarnation': 63, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/2003687153', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6836', 'nonce': 2003687153}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 2003687153}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8229, 'qdb_cluster': [8229]} 2024-03-06T07:27:37.903 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:27:37.903 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:27:38.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.004+0000 7f75f9ac9640 1 Processor -- start 2024-03-06T07:27:38.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.004+0000 7f75f9ac9640 1 -- start start 2024-03-06T07:27:38.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.004+0000 7f75f9ac9640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f75f4162970 0x7f75f40d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.004+0000 7f75f9ac9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f75f4161cf0 0x7f75f4162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.004+0000 7f75f9ac9640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75f40d22c0 con 0x7f75f40b4460 2024-03-06T07:27:38.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.004+0000 7f75f9ac9640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75f4161b00 con 0x7f75f4161cf0 2024-03-06T07:27:38.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.004+0000 7f75f9ac9640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75f40cecc0 con 0x7f75f4162970 2024-03-06T07:27:38.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.005+0000 7f75f3fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f75f4161cf0 0x7f75f4162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.005+0000 7f75f3fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f75f4161cf0 0x7f75f4162180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46586/0 (socket says 172.21.15.83:46586) 2024-03-06T07:27:38.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.005+0000 7f75f37fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f75f40b4460 0x7f75f40b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43942/0 (socket says 172.21.15.83:43942) 2024-03-06T07:27:38.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.005+0000 7f75f3fff640 1 -- 172.21.15.83:0/3516212554 learned_addr learned my addr 172.21.15.83:0/3516212554 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:38.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.005+0000 7f75f2ffd640 1 --2- 172.21.15.83:0/3516212554 >> v2:172.21.15.183:3300/0 conn(0x7f75f4162970 0x7f75f40d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.005+0000 7f75f3fff640 1 -- 172.21.15.83:0/3516212554 >> v2:172.21.15.183:3300/0 conn(0x7f75f4162970 msgr2=0x7f75f40d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:38.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.005+0000 7f75f3fff640 1 --2- 172.21.15.83:0/3516212554 >> v2:172.21.15.183:3300/0 conn(0x7f75f4162970 0x7f75f40d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.005+0000 7f75f3fff640 1 -- 172.21.15.83:0/3516212554 >> v1:172.21.15.83:6789/0 conn(0x7f75f40b4460 legacy=0x7f75f40b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.005+0000 7f75f27fc640 1 -- 172.21.15.83:0/3516212554 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 553757232 0 0) 0x7f75f40d22c0 con 0x7f75f40b4460 2024-03-06T07:27:38.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.005+0000 7f75f3fff640 1 -- 172.21.15.83:0/3516212554 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f40d25a0 con 0x7f75f4161cf0 2024-03-06T07:27:38.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.006+0000 7f75f3fff640 1 --2- 172.21.15.83:0/3516212554 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f75f4161cf0 0x7f75f4162180 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f75f40d17e0 tx=0x7f75ec02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbac02e8053900e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:38.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.006+0000 7f75f27fc640 1 -- 172.21.15.83:0/3516212554 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ec02cb20 con 0x7f75f4161cf0 2024-03-06T07:27:38.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.006+0000 7f75f27fc640 1 -- 172.21.15.83:0/3516212554 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75ec02ccc0 con 0x7f75f4161cf0 2024-03-06T07:27:38.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.006+0000 7f75f27fc640 1 -- 172.21.15.83:0/3516212554 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ec034610 con 0x7f75f4161cf0 2024-03-06T07:27:38.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.006+0000 7f75f9ac9640 1 -- 172.21.15.83:0/3516212554 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f75f4161cf0 msgr2=0x7f75f4162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.006+0000 7f75f9ac9640 1 --2- 172.21.15.83:0/3516212554 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f75f4161cf0 0x7f75f4162180 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f75f40d17e0 tx=0x7f75ec02bf80 comp rx=0 tx=0).stop 2024-03-06T07:27:38.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.006+0000 7f75f9ac9640 1 -- 172.21.15.83:0/3516212554 shutdown_connections 2024-03-06T07:27:38.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.006+0000 7f75f9ac9640 1 --2- 172.21.15.83:0/3516212554 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f75f4161cf0 0x7f75f4162180 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.006+0000 7f75f9ac9640 1 --2- 172.21.15.83:0/3516212554 >> v2:172.21.15.183:3300/0 conn(0x7f75f4162970 0x7f75f40d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.006+0000 7f75f9ac9640 1 -- 172.21.15.83:0/3516212554 >> 172.21.15.83:0/3516212554 conn(0x7f75f40d3ce0 msgr2=0x7f75f40d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:38.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.007+0000 7f75f9ac9640 1 -- 172.21.15.83:0/3516212554 shutdown_connections 2024-03-06T07:27:38.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.007+0000 7f75f9ac9640 1 -- 172.21.15.83:0/3516212554 wait complete. 2024-03-06T07:27:38.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.008+0000 7f75f9ac9640 1 Processor -- start 2024-03-06T07:27:38.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.008+0000 7f75f9ac9640 1 -- start start 2024-03-06T07:27:38.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.009+0000 7f75f9ac9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f75f40b4460 0x7f75f40ca250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.009+0000 7f75f9ac9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f75f4161cf0 0x7f75f40b7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.009+0000 7f75f9ac9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f75f4162970 0x7f75f40b1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.009+0000 7f75f9ac9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f40ce950 con 0x7f75f40b4460 2024-03-06T07:27:38.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.009+0000 7f75f9ac9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f40d2b60 con 0x7f75f4161cf0 2024-03-06T07:27:38.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.009+0000 7f75f9ac9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75f40b4840 con 0x7f75f4162970 2024-03-06T07:27:38.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.009+0000 7f75f37fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f75f40b4460 0x7f75f40ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.009+0000 7f75f37fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f75f40b4460 0x7f75f40ca250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46588/0 (socket says 172.21.15.83:46588) 2024-03-06T07:27:38.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.009+0000 7f75f37fe640 1 -- 172.21.15.83:0/3231049582 learned_addr learned my addr 172.21.15.83:0/3231049582 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:38.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.009+0000 7f75f2ffd640 1 --2- 172.21.15.83:0/3231049582 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f75f4161cf0 0x7f75f40b7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.009+0000 7f75f3fff640 1 --2- 172.21.15.83:0/3231049582 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f75f4162970 0x7f75f40b1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.009+0000 7f75f37fe640 1 -- 172.21.15.83:0/3231049582 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f75f4162970 msgr2=0x7f75f40b1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.009+0000 7f75f37fe640 1 --2- 172.21.15.83:0/3231049582 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f75f4162970 0x7f75f40b1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.009+0000 7f75f37fe640 1 -- 172.21.15.83:0/3231049582 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f75f4161cf0 msgr2=0x7f75f40b7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.009+0000 7f75f37fe640 1 --2- 172.21.15.83:0/3231049582 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f75f4161cf0 0x7f75f40b7ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.009+0000 7f75f37fe640 1 -- 172.21.15.83:0/3231049582 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f40b88b0 con 0x7f75f40b4460 2024-03-06T07:27:38.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.009+0000 7f75f2ffd640 1 --2- 172.21.15.83:0/3231049582 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f75f4161cf0 0x7f75f40b7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:38.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.009+0000 7f75f37fe640 1 --2- 172.21.15.83:0/3231049582 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f75f40b4460 0x7f75f40ca250 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f75e400c460 tx=0x7f75e400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:38.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.011+0000 7f75dbfff640 1 -- 172.21.15.83:0/3231049582 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e4018070 con 0x7f75f40b4460 2024-03-06T07:27:38.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.011+0000 7f75dbfff640 1 -- 172.21.15.83:0/3231049582 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75e4007500 con 0x7f75f40b4460 2024-03-06T07:27:38.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.011+0000 7f75f9ac9640 1 -- 172.21.15.83:0/3231049582 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75f40b8b90 con 0x7f75f40b4460 2024-03-06T07:27:38.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.011+0000 7f75dbfff640 1 -- 172.21.15.83:0/3231049582 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e4004a20 con 0x7f75f40b4460 2024-03-06T07:27:38.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.011+0000 7f75f9ac9640 1 -- 172.21.15.83:0/3231049582 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75f40b9150 con 0x7f75f40b4460 2024-03-06T07:27:38.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.013+0000 7f75dbfff640 1 -- 172.21.15.83:0/3231049582 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f75e401b070 con 0x7f75f40b4460 2024-03-06T07:27:38.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.013+0000 7f75dbfff640 1 --2- 172.21.15.83:0/3231049582 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f75e006b4d0 0x7f75e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.013+0000 7f75dbfff640 1 -- 172.21.15.83:0/3231049582 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f75e40895d0 con 0x7f75f40b4460 2024-03-06T07:27:38.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.013+0000 7f75f2ffd640 1 --2- 172.21.15.83:0/3231049582 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f75e006b4d0 0x7f75e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.014+0000 7f75f2ffd640 1 --2- 172.21.15.83:0/3231049582 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f75e006b4d0 0x7f75e006d990 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f75f40c7b00 tx=0x7f75e8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:38.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.014+0000 7f75fc790740 1 Processor -- start 2024-03-06T07:27:38.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.014+0000 7f75fc790740 1 -- start start 2024-03-06T07:27:38.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.015+0000 7f75fc790740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55aff4fb4870 0x55aff4fb4c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.015+0000 7f75fc790740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aff4fb88c0 0x55aff4fba360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.015+0000 7f75fc790740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55aff4fe4600 0x55aff4fe4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.015+0000 7f75fc790740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aff4f1e6b0 con 0x55aff4fb88c0 2024-03-06T07:27:38.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.015+0000 7f75fc790740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aff4ec7a60 con 0x55aff4fb4870 2024-03-06T07:27:38.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.015+0000 7f75fc790740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x55aff4f1a750 con 0x55aff4fe4600 2024-03-06T07:27:38.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.015+0000 7f75f2ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aff4fb88c0 0x55aff4fba360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.015+0000 7f75f2ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aff4fb88c0 0x55aff4fba360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46590/0 (socket says 172.21.15.83:46590) 2024-03-06T07:27:38.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.015+0000 7f75f2ffd640 1 -- 172.21.15.83:0/815075622 learned_addr learned my addr 172.21.15.83:0/815075622 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:38.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.015+0000 7f75f37fe640 1 --2- 172.21.15.83:0/815075622 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55aff4fb4870 0x55aff4fb4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.015+0000 7f75f3fff640 1 --2- 172.21.15.83:0/815075622 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55aff4fe4600 0x55aff4fe4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.016+0000 7f75f2ffd640 1 -- 172.21.15.83:0/815075622 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55aff4fe4600 msgr2=0x55aff4fe4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:38.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.016+0000 7f75f2ffd640 1 --2- 172.21.15.83:0/815075622 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55aff4fe4600 0x55aff4fe4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.016+0000 7f75f2ffd640 1 -- 172.21.15.83:0/815075622 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55aff4fb4870 msgr2=0x55aff4fb4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:38.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.016+0000 7f75f2ffd640 1 --2- 172.21.15.83:0/815075622 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55aff4fb4870 0x55aff4fb4c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.016+0000 7f75f2ffd640 1 -- 172.21.15.83:0/815075622 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aff4ee5f30 con 0x55aff4fb88c0 2024-03-06T07:27:38.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.016+0000 7f75f37fe640 1 --2- 172.21.15.83:0/815075622 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55aff4fb4870 0x55aff4fb4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:38.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.016+0000 7f75f2ffd640 1 --2- 172.21.15.83:0/815075622 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aff4fb88c0 0x55aff4fba360 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f75e8011da0 tx=0x7f75e8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b795fd51d1fef935 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:38.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.016+0000 7f75d97fa640 1 -- 172.21.15.83:0/815075622 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e8011dd0 con 0x55aff4fb88c0 2024-03-06T07:27:38.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.016+0000 7f75d97fa640 1 -- 172.21.15.83:0/815075622 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75e8026070 con 0x55aff4fb88c0 2024-03-06T07:27:38.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.016+0000 7f75d97fa640 1 -- 172.21.15.83:0/815075622 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e8021640 con 0x55aff4fb88c0 2024-03-06T07:27:38.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.017+0000 7f75fc790740 1 -- 172.21.15.83:0/815075622 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aff4fb88c0 msgr2=0x55aff4fba360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.017+0000 7f75fc790740 1 --2- 172.21.15.83:0/815075622 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aff4fb88c0 0x55aff4fba360 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f75e8011da0 tx=0x7f75e8014000 comp rx=0 tx=0).stop 2024-03-06T07:27:38.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.017+0000 7f75fc790740 1 -- 172.21.15.83:0/815075622 shutdown_connections 2024-03-06T07:27:38.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.017+0000 7f75fc790740 1 --2- 172.21.15.83:0/815075622 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55aff4fe4600 0x55aff4fe4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.017+0000 7f75fc790740 1 --2- 172.21.15.83:0/815075622 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aff4fb88c0 0x55aff4fba360 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.017+0000 7f75fc790740 1 --2- 172.21.15.83:0/815075622 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55aff4fb4870 0x55aff4fb4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.017+0000 7f75fc790740 1 -- 172.21.15.83:0/815075622 >> 172.21.15.83:0/815075622 conn(0x55aff4fb68f0 msgr2=0x55aff4fb81b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:38.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.017+0000 7f75fc790740 1 -- 172.21.15.83:0/815075622 shutdown_connections 2024-03-06T07:27:38.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.017+0000 7f75fc790740 1 -- 172.21.15.83:0/815075622 wait complete. 2024-03-06T07:27:38.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.017+0000 7f75fc790740 1 -- start start 2024-03-06T07:27:38.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.017+0000 7f75fc790740 1 Processor -- start 2024-03-06T07:27:38.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.018+0000 7f75fc790740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55aff4fe4600 0x55aff4ffeac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.018+0000 7f75fc790740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aff4fdf6b0 0x55aff4fdeb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.018+0000 7f75fc790740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55aff4fe7c60 0x55aff4fe80e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.018+0000 7f75fc790740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aff4fb8d50 con 0x55aff4fdf6b0 2024-03-06T07:27:38.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.018+0000 7f75fc790740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aff4edaea0 con 0x55aff4fe4600 2024-03-06T07:27:38.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.018+0000 7f75fc790740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x55aff4fb9030 con 0x55aff4fe7c60 2024-03-06T07:27:38.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.018+0000 7f75fc790740 10 client.0 _ll_register_callbacks cb 0x55aff4eda870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:27:38.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.018+0000 7f75f2ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aff4fdf6b0 0x55aff4fdeb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.018+0000 7f75f2ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aff4fdf6b0 0x55aff4fdeb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46604/0 (socket says 172.21.15.83:46604) 2024-03-06T07:27:38.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.018+0000 7f75f2ffd640 1 -- 172.21.15.83:0/2521530674 learned_addr learned my addr 172.21.15.83:0/2521530674 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:38.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.018+0000 7f75f37fe640 1 --2- 172.21.15.83:0/2521530674 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55aff4fe4600 0x55aff4ffeac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.018+0000 7f75f3fff640 1 --2- 172.21.15.83:0/2521530674 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55aff4fe7c60 0x55aff4fe80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.019+0000 7f75f2ffd640 1 -- 172.21.15.83:0/2521530674 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55aff4fe7c60 msgr2=0x55aff4fe80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:38.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.019+0000 7f75f2ffd640 1 --2- 172.21.15.83:0/2521530674 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55aff4fe7c60 0x55aff4fe80e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.019+0000 7f75f2ffd640 1 -- 172.21.15.83:0/2521530674 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55aff4fe4600 msgr2=0x55aff4ffeac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:38.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.019+0000 7f75f2ffd640 1 --2- 172.21.15.83:0/2521530674 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55aff4fe4600 0x55aff4ffeac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.019+0000 7f75f2ffd640 1 -- 172.21.15.83:0/2521530674 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aff4edb270 con 0x55aff4fdf6b0 2024-03-06T07:27:38.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.019+0000 7f75f2ffd640 1 --2- 172.21.15.83:0/2521530674 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aff4fdf6b0 0x55aff4fdeb40 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f75e8022000 tx=0x7f75e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e71fcb661449bcb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:38.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.019+0000 7f75d27fc640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-06T07:27:38.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.019+0000 7f75d27fc640 1 -- 172.21.15.83:0/2521530674 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e8011a30 con 0x55aff4fdf6b0 2024-03-06T07:27:38.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.019+0000 7f75d27fc640 1 -- 172.21.15.83:0/2521530674 <== mon.0 v2:172.21.15.83:3300/0 2 ==== osd_map(114..114 src has 1..114) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f75e8021b10 con 0x55aff4fdf6b0 2024-03-06T07:27:38.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.019+0000 7f75fc790740 1 -- 172.21.15.83:0/2521530674 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f75e8026070 con 0x55aff4fdf6b0 2024-03-06T07:27:38.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.019+0000 7f75d27fc640 1 -- 172.21.15.83:0/2521530674 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) v2 ==== 24+0+0 (secure 0 0 0) 0x7f75e803e050 con 0x55aff4fdf6b0 2024-03-06T07:27:38.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.019+0000 7f75fc790740 10 client.8262 fetch_fsmap learned FSMap version 66 2024-03-06T07:27:38.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.019+0000 7f75fc790740 1 -- 172.21.15.83:0/2521530674 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=2+}) v3 -- 0x55aff4c332d0 con 0x55aff4fdf6b0 2024-03-06T07:27:38.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.019+0000 7f75d27fc640 1 -- 172.21.15.83:0/2521530674 <== mon.0 v2:172.21.15.83:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75e8026070 con 0x55aff4fdf6b0 2024-03-06T07:27:38.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.019+0000 7f75d27fc640 1 -- 172.21.15.83:0/2521530674 <== mon.0 v2:172.21.15.83:3300/0 5 ==== fsmap(e 66) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f75e8041020 con 0x55aff4fdf6b0 2024-03-06T07:27:38.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.019+0000 7f75fc790740 10 client.8262 fetch_fsmap finished waiting for FSMap version 66 2024-03-06T07:27:38.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.019+0000 7f75fc790740 10 client.8262 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:27:38.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.019+0000 7f75fc790740 20 client.8262 populate_metadata read hostname 'smithi083' 2024-03-06T07:27:38.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.020+0000 7f75fc790740 1 --2- 172.21.15.83:0/2521530674 >> [v2:172.21.15.83:6836/2003687153,v1:172.21.15.83:6837/2003687153] conn(0x55aff4fe63a0 0x55aff501e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.020+0000 7f75fc790740 4 client.8262 mds_command: new command op to 8229 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:27:38.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.020+0000 7f75fc790740 1 -- 172.21.15.83:0/2521530674 --> [v2:172.21.15.83:6836/2003687153,v1:172.21.15.83:6837/2003687153] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55aff4f2bbf0 con 0x55aff4fe63a0 2024-03-06T07:27:38.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.020+0000 7f75f37fe640 1 --2- 172.21.15.83:0/2521530674 >> [v2:172.21.15.83:6836/2003687153,v1:172.21.15.83:6837/2003687153] conn(0x55aff4fe63a0 0x55aff501e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.020+0000 7f75f37fe640 1 --2- 172.21.15.83:0/2521530674 >> [v2:172.21.15.83:6836/2003687153,v1:172.21.15.83:6837/2003687153] conn(0x55aff4fe63a0 0x55aff501e8f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51f0048581f0c8 server_cookie=c1066ed61696e258 in_seq=0 out_seq=0 2024-03-06T07:27:38.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.020+0000 7f75d27fc640 10 client.8262 ms_handle_connect on v2:172.21.15.83:6836/2003687153 2024-03-06T07:27:38.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.021+0000 7f75d27fc640 1 -- 172.21.15.83:0/2521530674 <== mds.0 v2:172.21.15.83:6836/2003687153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55aff4f2bbf0 con 0x55aff4fe63a0 2024-03-06T07:27:38.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.021+0000 7f75d27fc640 10 client.8262 handle_command_reply: tid=0 2024-03-06T07:27:38.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.021+0000 7f75fc790740 1 client.8262 shutdown 2024-03-06T07:27:38.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.021+0000 7f75fc790740 1 -- 172.21.15.83:0/2521530674 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aff4fdf6b0 msgr2=0x55aff4fdeb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.021+0000 7f75fc790740 1 --2- 172.21.15.83:0/2521530674 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aff4fdf6b0 0x55aff4fdeb40 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f75e8022000 tx=0x7f75e80046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.021+0000 7f75fc790740 1 -- 172.21.15.83:0/2521530674 shutdown_connections 2024-03-06T07:27:38.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.021+0000 7f75fc790740 1 --2- 172.21.15.83:0/2521530674 >> [v2:172.21.15.83:6836/2003687153,v1:172.21.15.83:6837/2003687153] conn(0x55aff4fe63a0 0x55aff501e8f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.021+0000 7f75d27fc640 0 client.8262 ms_handle_reset on v2:172.21.15.83:6836/2003687153 2024-03-06T07:27:38.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.021+0000 7f75fc790740 1 --2- 172.21.15.83:0/2521530674 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55aff4fe7c60 0x55aff4fe80e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.021+0000 7f75fc790740 1 --2- 172.21.15.83:0/2521530674 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aff4fdf6b0 0x55aff4fdeb40 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.021+0000 7f75fc790740 1 --2- 172.21.15.83:0/2521530674 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55aff4fe4600 0x55aff4ffeac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.021+0000 7f75fc790740 1 -- 172.21.15.83:0/2521530674 >> 172.21.15.83:0/2521530674 conn(0x55aff4fb4870 msgr2=0x55aff4fb3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:38.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.021+0000 7f75fc790740 1 -- 172.21.15.83:0/2521530674 shutdown_connections 2024-03-06T07:27:38.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.021+0000 7f75fc790740 1 -- 172.21.15.83:0/2521530674 wait complete. 2024-03-06T07:27:38.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.021+0000 7f75fc790740 20 client.8262 trim_cache size 0 max 16384 2024-03-06T07:27:38.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.053+0000 7f75fc790740 1 Processor -- start 2024-03-06T07:27:38.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.053+0000 7f75fc790740 1 -- start start 2024-03-06T07:27:38.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.055+0000 7f75fc790740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55aff4fdf6b0 0x55aff4fb8080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.055+0000 7f75fc790740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55aff4fe4600 0x55aff4fe7570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.055+0000 7f75fc790740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aff4fe7c60 0x55aff4fb3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.055+0000 7f75fc790740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f41fff90 con 0x55aff4fe7c60 2024-03-06T07:27:38.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.055+0000 7f75fc790740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f41ff2a0 con 0x55aff4fe4600 2024-03-06T07:27:38.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.055+0000 7f75fc790740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75f41f7c00 con 0x55aff4fdf6b0 2024-03-06T07:27:38.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.055+0000 7f75f3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aff4fe7c60 0x55aff4fb3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.055+0000 7f75f3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aff4fe7c60 0x55aff4fb3510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46620/0 (socket says 172.21.15.83:46620) 2024-03-06T07:27:38.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.055+0000 7f75f3fff640 1 -- 172.21.15.83:0/39405167 learned_addr learned my addr 172.21.15.83:0/39405167 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:38.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.055+0000 7f75f37fe640 1 --2- 172.21.15.83:0/39405167 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55aff4fdf6b0 0x55aff4fb8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.055+0000 7f75f2ffd640 1 --2- 172.21.15.83:0/39405167 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55aff4fe4600 0x55aff4fe7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.055+0000 7f75f3fff640 1 -- 172.21.15.83:0/39405167 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55aff4fdf6b0 msgr2=0x55aff4fb8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:38.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.055+0000 7f75f3fff640 1 --2- 172.21.15.83:0/39405167 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55aff4fdf6b0 0x55aff4fb8080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.055+0000 7f75f3fff640 1 -- 172.21.15.83:0/39405167 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55aff4fe4600 msgr2=0x55aff4fe7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:38.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.055+0000 7f75f3fff640 1 --2- 172.21.15.83:0/39405167 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55aff4fe4600 0x55aff4fe7570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.055+0000 7f75f3fff640 1 -- 172.21.15.83:0/39405167 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aff4cfd6b0 con 0x55aff4fe7c60 2024-03-06T07:27:38.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.055+0000 7f75f2ffd640 1 --2- 172.21.15.83:0/39405167 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55aff4fe4600 0x55aff4fe7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:38.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.055+0000 7f75f3fff640 1 --2- 172.21.15.83:0/39405167 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aff4fe7c60 0x55aff4fb3510 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f75ec009f40 tx=0x7f75ec02e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=817b1544e97883a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:38.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.056+0000 7f75d8ff9640 1 -- 172.21.15.83:0/39405167 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ec02c9e0 con 0x55aff4fe7c60 2024-03-06T07:27:38.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.056+0000 7f75d8ff9640 1 -- 172.21.15.83:0/39405167 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75ec02cb80 con 0x55aff4fe7c60 2024-03-06T07:27:38.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.056+0000 7f75d8ff9640 1 -- 172.21.15.83:0/39405167 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ec047650 con 0x55aff4fe7c60 2024-03-06T07:27:38.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.056+0000 7f75fc790740 1 -- 172.21.15.83:0/39405167 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aff4fe7c60 msgr2=0x55aff4fb3510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.056+0000 7f75fc790740 1 --2- 172.21.15.83:0/39405167 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aff4fe7c60 0x55aff4fb3510 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f75ec009f40 tx=0x7f75ec02e000 comp rx=0 tx=0).stop 2024-03-06T07:27:38.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.056+0000 7f75fc790740 1 -- 172.21.15.83:0/39405167 shutdown_connections 2024-03-06T07:27:38.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.056+0000 7f75fc790740 1 --2- 172.21.15.83:0/39405167 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aff4fe7c60 0x55aff4fb3510 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.056+0000 7f75fc790740 1 --2- 172.21.15.83:0/39405167 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55aff4fe4600 0x55aff4fe7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.056+0000 7f75fc790740 1 --2- 172.21.15.83:0/39405167 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55aff4fdf6b0 0x55aff4fb8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.056+0000 7f75fc790740 1 -- 172.21.15.83:0/39405167 >> 172.21.15.83:0/39405167 conn(0x55aff4fb4870 msgr2=0x55aff4fe1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:38.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.056+0000 7f75fc790740 1 -- 172.21.15.83:0/39405167 shutdown_connections 2024-03-06T07:27:38.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.056+0000 7f75fc790740 1 -- 172.21.15.83:0/39405167 wait complete. 2024-03-06T07:27:38.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.057+0000 7f75fc790740 1 -- start start 2024-03-06T07:27:38.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.057+0000 7f75fc790740 1 Processor -- start 2024-03-06T07:27:38.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.058+0000 7f75fc790740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55aff4fdf6b0 0x55aff4fe49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.058+0000 7f75fc790740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aff4fe4600 0x55aff4fe4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.058+0000 7f75fc790740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55aff4fe7c60 0x55aff4fe0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.058+0000 7f75fc790740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aff4fe6eb0 con 0x55aff4fe4600 2024-03-06T07:27:38.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.058+0000 7f75fc790740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f40c94e0 con 0x55aff4fe7c60 2024-03-06T07:27:38.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.058+0000 7f75fc790740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x55aff4fdf3d0 con 0x55aff4fdf6b0 2024-03-06T07:27:38.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.058+0000 7f75fc790740 10 client.0 _ll_register_callbacks cb 0x7f75e8021820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:27:38.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.058+0000 7f75f2ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aff4fe4600 0x55aff4fe4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.058+0000 7f75f2ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aff4fe4600 0x55aff4fe4f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46636/0 (socket says 172.21.15.83:46636) 2024-03-06T07:27:38.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.058+0000 7f75f2ffd640 1 -- 172.21.15.83:0/2997830151 learned_addr learned my addr 172.21.15.83:0/2997830151 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:38.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.058+0000 7f75f37fe640 1 --2- 172.21.15.83:0/2997830151 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55aff4fdf6b0 0x55aff4fe49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.058+0000 7f75f3fff640 1 --2- 172.21.15.83:0/2997830151 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55aff4fe7c60 0x55aff4fe0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.058+0000 7f75f2ffd640 1 -- 172.21.15.83:0/2997830151 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55aff4fdf6b0 msgr2=0x55aff4fe49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:38.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.058+0000 7f75f2ffd640 1 --2- 172.21.15.83:0/2997830151 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55aff4fdf6b0 0x55aff4fe49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.058+0000 7f75f2ffd640 1 -- 172.21.15.83:0/2997830151 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55aff4fe7c60 msgr2=0x55aff4fe0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:38.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.058+0000 7f75f2ffd640 1 --2- 172.21.15.83:0/2997830151 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55aff4fe7c60 0x55aff4fe0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.058+0000 7f75f2ffd640 1 -- 172.21.15.83:0/2997830151 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aff4fdba40 con 0x55aff4fe4600 2024-03-06T07:27:38.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.058+0000 7f75f2ffd640 1 --2- 172.21.15.83:0/2997830151 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aff4fe4600 0x55aff4fe4f20 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f75e80094e0 tx=0x7f75e8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d30cc01e88164c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:38.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.058+0000 7f75d27fc640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-06T07:27:38.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.058+0000 7f75d27fc640 1 -- 172.21.15.83:0/2997830151 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e8011a30 con 0x55aff4fe4600 2024-03-06T07:27:38.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.059+0000 7f75d27fc640 1 -- 172.21.15.83:0/2997830151 <== mon.0 v2:172.21.15.83:3300/0 2 ==== osd_map(114..114 src has 1..114) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f75e8036690 con 0x55aff4fe4600 2024-03-06T07:27:38.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.059+0000 7f75fc790740 1 -- 172.21.15.83:0/2997830151 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f75ec02cb80 con 0x55aff4fe4600 2024-03-06T07:27:38.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.059+0000 7f75d27fc640 1 -- 172.21.15.83:0/2997830151 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) v2 ==== 24+0+0 (secure 0 0 0) 0x7f75e8055050 con 0x55aff4fe4600 2024-03-06T07:27:38.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.059+0000 7f75fc790740 10 client.8268 fetch_fsmap learned FSMap version 66 2024-03-06T07:27:38.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.059+0000 7f75fc790740 1 -- 172.21.15.83:0/2997830151 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=2+}) v3 -- 0x55aff4fdc340 con 0x55aff4fe4600 2024-03-06T07:27:38.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.059+0000 7f75d27fc640 1 -- 172.21.15.83:0/2997830151 <== mon.0 v2:172.21.15.83:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75ec02cb80 con 0x55aff4fe4600 2024-03-06T07:27:38.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.059+0000 7f75d27fc640 1 -- 172.21.15.83:0/2997830151 <== mon.0 v2:172.21.15.83:3300/0 5 ==== fsmap(e 66) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f75e8058020 con 0x55aff4fe4600 2024-03-06T07:27:38.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.059+0000 7f75fc790740 10 client.8268 fetch_fsmap finished waiting for FSMap version 66 2024-03-06T07:27:38.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.059+0000 7f75fc790740 10 client.8268 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:27:38.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.059+0000 7f75fc790740 20 client.8268 populate_metadata read hostname 'smithi083' 2024-03-06T07:27:38.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.059+0000 7f75fc790740 1 --2- 172.21.15.83:0/2997830151 >> [v2:172.21.15.83:6836/2003687153,v1:172.21.15.83:6837/2003687153] conn(0x55aff4fe0930 0x55aff5012070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.059+0000 7f75fc790740 4 client.8268 mds_command: new command op to 8229 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:27:38.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.059+0000 7f75fc790740 1 -- 172.21.15.83:0/2997830151 --> [v2:172.21.15.83:6836/2003687153,v1:172.21.15.83:6837/2003687153] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55aff4f25f40 con 0x55aff4fe0930 2024-03-06T07:27:38.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.059+0000 7f75f37fe640 1 --2- 172.21.15.83:0/2997830151 >> [v2:172.21.15.83:6836/2003687153,v1:172.21.15.83:6837/2003687153] conn(0x55aff4fe0930 0x55aff5012070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.059+0000 7f75f37fe640 1 --2- 172.21.15.83:0/2997830151 >> [v2:172.21.15.83:6836/2003687153,v1:172.21.15.83:6837/2003687153] conn(0x55aff4fe0930 0x55aff5012070 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6379653b5fe62255 server_cookie=a155ea9d99e71cb5 in_seq=0 out_seq=0 2024-03-06T07:27:38.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.059+0000 7f75d27fc640 10 client.8268 ms_handle_connect on v2:172.21.15.83:6836/2003687153 2024-03-06T07:27:38.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.059+0000 7f75d27fc640 1 -- 172.21.15.83:0/2997830151 <== mds.0 v2:172.21.15.83:6836/2003687153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x55aff4f25f40 con 0x55aff4fe0930 2024-03-06T07:27:38.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.059+0000 7f75d27fc640 10 client.8268 handle_command_reply: tid=0 2024-03-06T07:27:38.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.060+0000 7f75fc790740 1 client.8268 shutdown 2024-03-06T07:27:38.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.060+0000 7f75fc790740 1 -- 172.21.15.83:0/2997830151 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aff4fe4600 msgr2=0x55aff4fe4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.060+0000 7f75fc790740 1 --2- 172.21.15.83:0/2997830151 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aff4fe4600 0x55aff4fe4f20 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f75e80094e0 tx=0x7f75e8017000 comp rx=0 tx=0).stop 2024-03-06T07:27:38.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.060+0000 7f75fc790740 1 -- 172.21.15.83:0/2997830151 shutdown_connections 2024-03-06T07:27:38.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.060+0000 7f75fc790740 1 --2- 172.21.15.83:0/2997830151 >> [v2:172.21.15.83:6836/2003687153,v1:172.21.15.83:6837/2003687153] conn(0x55aff4fe0930 0x55aff5012070 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.060+0000 7f75fc790740 1 --2- 172.21.15.83:0/2997830151 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55aff4fe7c60 0x55aff4fe0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.060+0000 7f75fc790740 1 --2- 172.21.15.83:0/2997830151 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55aff4fe4600 0x55aff4fe4f20 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.060+0000 7f75fc790740 1 --2- 172.21.15.83:0/2997830151 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55aff4fdf6b0 0x55aff4fe49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.060+0000 7f75d27fc640 0 client.8268 ms_handle_reset on v2:172.21.15.83:6836/2003687153 2024-03-06T07:27:38.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.060+0000 7f75fc790740 1 -- 172.21.15.83:0/2997830151 >> 172.21.15.83:0/2997830151 conn(0x55aff4fb4870 msgr2=0x55aff4fb37c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:38.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.060+0000 7f75fc790740 1 -- 172.21.15.83:0/2997830151 shutdown_connections 2024-03-06T07:27:38.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.060+0000 7f75fc790740 1 -- 172.21.15.83:0/2997830151 wait complete. 2024-03-06T07:27:38.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.060+0000 7f75fc790740 20 client.8268 trim_cache size 0 max 16384 2024-03-06T07:27:38.068 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:27:38.068 INFO:teuthology.orchestra.run.smithi083.stdout: "cluster_fsid": "039877dd-d2ce-45d2-8e77-ac5eb7e1d55a", 2024-03-06T07:27:38.068 INFO:teuthology.orchestra.run.smithi083.stdout: "whoami": 0, 2024-03-06T07:27:38.068 INFO:teuthology.orchestra.run.smithi083.stdout: "id": 8229, 2024-03-06T07:27:38.068 INFO:teuthology.orchestra.run.smithi083.stdout: "want_state": "up:active", 2024-03-06T07:27:38.068 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "up:active", 2024-03-06T07:27:38.068 INFO:teuthology.orchestra.run.smithi083.stdout: "fs_name": "cephfs", 2024-03-06T07:27:38.068 INFO:teuthology.orchestra.run.smithi083.stdout: "rank_uptime": 4.0638563870000004, 2024-03-06T07:27:38.069 INFO:teuthology.orchestra.run.smithi083.stdout: "mdsmap_epoch": 66, 2024-03-06T07:27:38.069 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_epoch": 114, 2024-03-06T07:27:38.069 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_epoch_barrier": 114, 2024-03-06T07:27:38.069 INFO:teuthology.orchestra.run.smithi083.stdout: "uptime": 4.3161288210000004 2024-03-06T07:27:38.069 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:27:38.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.060+0000 7f75f9ac9640 1 -- 172.21.15.83:0/3231049582 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f75e006b4d0 msgr2=0x7f75e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.060+0000 7f75f9ac9640 1 --2- 172.21.15.83:0/3231049582 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f75e006b4d0 0x7f75e006d990 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f75f40c7b00 tx=0x7f75e8008040 comp rx=0 tx=0).stop 2024-03-06T07:27:38.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.060+0000 7f75f9ac9640 1 -- 172.21.15.83:0/3231049582 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f75f40b4460 msgr2=0x7f75f40ca250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.060+0000 7f75f9ac9640 1 --2- 172.21.15.83:0/3231049582 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f75f40b4460 0x7f75f40ca250 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f75e400c460 tx=0x7f75e400c920 comp rx=0 tx=0).stop 2024-03-06T07:27:38.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.060+0000 7f75f9ac9640 1 -- 172.21.15.83:0/3231049582 shutdown_connections 2024-03-06T07:27:38.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.060+0000 7f75f9ac9640 1 --2- 172.21.15.83:0/3231049582 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f75e006b4d0 0x7f75e006d990 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.060+0000 7f75f9ac9640 1 --2- 172.21.15.83:0/3231049582 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f75f4162970 0x7f75f40b1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.060+0000 7f75f9ac9640 1 --2- 172.21.15.83:0/3231049582 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f75f4161cf0 0x7f75f40b7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.060+0000 7f75f9ac9640 1 --2- 172.21.15.83:0/3231049582 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f75f40b4460 0x7f75f40ca250 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.060+0000 7f75f9ac9640 1 -- 172.21.15.83:0/3231049582 >> 172.21.15.83:0/3231049582 conn(0x7f75f40d3ce0 msgr2=0x7f75f4161040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:38.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.060+0000 7f75f9ac9640 1 -- 172.21.15.83:0/3231049582 shutdown_connections 2024-03-06T07:27:38.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.060+0000 7f75f9ac9640 1 -- 172.21.15.83:0/3231049582 wait complete. 2024-03-06T07:27:38.073 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:27:38.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.173+0000 7f0be0e84640 1 Processor -- start 2024-03-06T07:27:38.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.173+0000 7f0be0e84640 1 -- start start 2024-03-06T07:27:38.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.173+0000 7f0be0e84640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0bdc0d22b0 0x7f0bdc0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.173+0000 7f0be0e84640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0bdc0b41f0 0x7f0bdc0b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.173+0000 7f0be0e84640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bdc0b3430 con 0x7f0bdc15dfe0 2024-03-06T07:27:38.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.173+0000 7f0be0e84640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bdc0d2c50 con 0x7f0bdc0d22b0 2024-03-06T07:27:38.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.173+0000 7f0be0e84640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bdc0b31e0 con 0x7f0bdc0b41f0 2024-03-06T07:27:38.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.174+0000 7f0bd1d9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0bdc0d22b0 0x7f0bdc0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.174+0000 7f0bda59c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f0bdc15dfe0 0x7f0bdc0cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43948/0 (socket says 172.21.15.83:43948) 2024-03-06T07:27:38.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.174+0000 7f0bda59c640 1 -- 172.21.15.83:0/2898010217 learned_addr learned my addr 172.21.15.83:0/2898010217 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:27:38.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.174+0000 7f0bdad9d640 1 --2- 172.21.15.83:0/2898010217 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0bdc0b41f0 0x7f0bdc0b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.174+0000 7f0bd1d9b640 1 -- 172.21.15.83:0/2898010217 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0bdc0b41f0 msgr2=0x7f0bdc0b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:38.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.174+0000 7f0bd1d9b640 1 --2- 172.21.15.83:0/2898010217 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0bdc0b41f0 0x7f0bdc0b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.174+0000 7f0bd1d9b640 1 -- 172.21.15.83:0/2898010217 >> v1:172.21.15.83:6789/0 conn(0x7f0bdc15dfe0 legacy=0x7f0bdc0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.174+0000 7f0bd1d9b640 1 -- 172.21.15.83:0/2898010217 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bdc0b3710 con 0x7f0bdc0d22b0 2024-03-06T07:27:38.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.174+0000 7f0bd9d9b640 1 -- 172.21.15.83:0/2898010217 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1520277607 0 0) 0x7f0bdc0b3430 con 0x7f0bdc15dfe0 2024-03-06T07:27:38.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.174+0000 7f0bdad9d640 1 --2- 172.21.15.83:0/2898010217 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0bdc0b41f0 0x7f0bdc0b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:38.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.175+0000 7f0bd1d9b640 1 --2- 172.21.15.83:0/2898010217 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0bdc0d22b0 0x7f0bdc0d2710 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f0bc40097a0 tx=0x7f0bc402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bfbe0191a7c42a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:38.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.175+0000 7f0bd9d9b640 1 -- 172.21.15.83:0/2898010217 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc4035020 con 0x7f0bdc0d22b0 2024-03-06T07:27:38.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.175+0000 7f0bd9d9b640 1 -- 172.21.15.83:0/2898010217 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bc4031830 con 0x7f0bdc0d22b0 2024-03-06T07:27:38.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.175+0000 7f0bd9d9b640 1 -- 172.21.15.83:0/2898010217 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc4031c00 con 0x7f0bdc0d22b0 2024-03-06T07:27:38.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.175+0000 7f0be0e84640 1 -- 172.21.15.83:0/2898010217 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0bdc0d22b0 msgr2=0x7f0bdc0d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.175+0000 7f0be0e84640 1 --2- 172.21.15.83:0/2898010217 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0bdc0d22b0 0x7f0bdc0d2710 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f0bc40097a0 tx=0x7f0bc402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.175+0000 7f0be0e84640 1 -- 172.21.15.83:0/2898010217 shutdown_connections 2024-03-06T07:27:38.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.175+0000 7f0be0e84640 1 --2- 172.21.15.83:0/2898010217 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0bdc0b41f0 0x7f0bdc0b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.175+0000 7f0be0e84640 1 --2- 172.21.15.83:0/2898010217 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0bdc0d22b0 0x7f0bdc0d2710 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.175+0000 7f0be0e84640 1 -- 172.21.15.83:0/2898010217 >> 172.21.15.83:0/2898010217 conn(0x7f0bdc0d3cd0 msgr2=0x7f0bdc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:38.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.175+0000 7f0be0e84640 1 -- 172.21.15.83:0/2898010217 shutdown_connections 2024-03-06T07:27:38.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.175+0000 7f0be0e84640 1 -- 172.21.15.83:0/2898010217 wait complete. 2024-03-06T07:27:38.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.177+0000 7f0be0e84640 1 Processor -- start 2024-03-06T07:27:38.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.177+0000 7f0be0e84640 1 -- start start 2024-03-06T07:27:38.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.177+0000 7f0be0e84640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0bdc0b41f0 0x7f0bdc0cc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.177+0000 7f0be0e84640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0bdc0d22b0 0x7f0bdc160640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.177+0000 7f0be0e84640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0bdc15dfe0 0x7f0bdc159610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.178+0000 7f0be0e84640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bdc0cfa00 con 0x7f0bdc0b41f0 2024-03-06T07:27:38.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.178+0000 7f0be0e84640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bdc0b3d60 con 0x7f0bdc0d22b0 2024-03-06T07:27:38.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.178+0000 7f0be0e84640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bdc0d1c70 con 0x7f0bdc15dfe0 2024-03-06T07:27:38.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.178+0000 7f0bda59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0bdc0b41f0 0x7f0bdc0cc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.178+0000 7f0bda59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0bdc0b41f0 0x7f0bdc0cc180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46654/0 (socket says 172.21.15.83:46654) 2024-03-06T07:27:38.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.178+0000 7f0bda59c640 1 -- 172.21.15.83:0/3545705970 learned_addr learned my addr 172.21.15.83:0/3545705970 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:38.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.178+0000 7f0bdad9d640 1 --2- 172.21.15.83:0/3545705970 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0bdc15dfe0 0x7f0bdc159610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.178+0000 7f0bda59c640 1 -- 172.21.15.83:0/3545705970 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0bdc15dfe0 msgr2=0x7f0bdc159610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.178+0000 7f0bd1d9b640 1 --2- 172.21.15.83:0/3545705970 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0bdc0d22b0 0x7f0bdc160640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.178+0000 7f0bda59c640 1 --2- 172.21.15.83:0/3545705970 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0bdc15dfe0 0x7f0bdc159610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.178+0000 7f0bda59c640 1 -- 172.21.15.83:0/3545705970 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0bdc0d22b0 msgr2=0x7f0bdc160640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.178+0000 7f0bda59c640 1 --2- 172.21.15.83:0/3545705970 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0bdc0d22b0 0x7f0bdc160640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.178+0000 7f0bda59c640 1 -- 172.21.15.83:0/3545705970 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bdc0cc9e0 con 0x7f0bdc0b41f0 2024-03-06T07:27:38.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.178+0000 7f0bdad9d640 1 --2- 172.21.15.83:0/3545705970 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0bdc15dfe0 0x7f0bdc159610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:38.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.178+0000 7f0bd1d9b640 1 --2- 172.21.15.83:0/3545705970 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0bdc0d22b0 0x7f0bdc160640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:38.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.178+0000 7f0bda59c640 1 --2- 172.21.15.83:0/3545705970 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0bdc0b41f0 0x7f0bdc0cc180 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc00c430 tx=0x7f0bcc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:38.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.178+0000 7f0bd37fe640 1 -- 172.21.15.83:0/3545705970 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bcc0074e0 con 0x7f0bdc0b41f0 2024-03-06T07:27:38.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.178+0000 7f0bd37fe640 1 -- 172.21.15.83:0/3545705970 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bcc0047e0 con 0x7f0bdc0b41f0 2024-03-06T07:27:38.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.178+0000 7f0bd37fe640 1 -- 172.21.15.83:0/3545705970 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bcc004bd0 con 0x7f0bdc0b41f0 2024-03-06T07:27:38.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.178+0000 7f0be0e84640 1 -- 172.21.15.83:0/3545705970 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bdc0ccf70 con 0x7f0bdc0b41f0 2024-03-06T07:27:38.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.178+0000 7f0be0e84640 1 -- 172.21.15.83:0/3545705970 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bdc0cd530 con 0x7f0bdc0b41f0 2024-03-06T07:27:38.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.179+0000 7f0be0e84640 1 -- 172.21.15.83:0/3545705970 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bdc04f330 con 0x7f0bdc0b41f0 2024-03-06T07:27:38.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.180+0000 7f0bd37fe640 1 -- 172.21.15.83:0/3545705970 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0bcc011070 con 0x7f0bdc0b41f0 2024-03-06T07:27:38.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.180+0000 7f0bd37fe640 1 --2- 172.21.15.83:0/3545705970 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0bbc06b4d0 0x7f0bbc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:38.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.180+0000 7f0bd37fe640 1 -- 172.21.15.83:0/3545705970 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f0bcc089ed0 con 0x7f0bdc0b41f0 2024-03-06T07:27:38.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.180+0000 7f0bd1d9b640 1 --2- 172.21.15.83:0/3545705970 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0bbc06b4d0 0x7f0bbc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:38.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.181+0000 7f0bd1d9b640 1 --2- 172.21.15.83:0/3545705970 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0bbc06b4d0 0x7f0bbc06d990 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f0bc4036000 tx=0x7f0bc4008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:38.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.188+0000 7f0bd37fe640 1 -- 172.21.15.83:0/3545705970 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0bcc058ca0 con 0x7f0bdc0b41f0 2024-03-06T07:27:38.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.399+0000 7f0be0e84640 1 -- 172.21.15.83:0/3545705970 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f0bdc06d150 con 0x7f0bdc0b41f0 2024-03-06T07:27:38.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.400+0000 7f0bd37fe640 1 -- 172.21.15.83:0/3545705970 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v66) v1 ==== 53+0+73 (secure 0 0 0) 0x7f0bcc0585a0 con 0x7f0bdc0b41f0 2024-03-06T07:27:38.402 INFO:teuthology.orchestra.run.smithi083.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:27:38.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.402+0000 7f0be0e84640 1 -- 172.21.15.83:0/3545705970 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0bbc06b4d0 msgr2=0x7f0bbc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.402+0000 7f0be0e84640 1 --2- 172.21.15.83:0/3545705970 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0bbc06b4d0 0x7f0bbc06d990 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f0bc4036000 tx=0x7f0bc4008030 comp rx=0 tx=0).stop 2024-03-06T07:27:38.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.402+0000 7f0be0e84640 1 -- 172.21.15.83:0/3545705970 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0bdc0b41f0 msgr2=0x7f0bdc0cc180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.402+0000 7f0be0e84640 1 --2- 172.21.15.83:0/3545705970 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0bdc0b41f0 0x7f0bdc0cc180 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc00c430 tx=0x7f0bcc00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.402+0000 7f0be0e84640 1 -- 172.21.15.83:0/3545705970 shutdown_connections 2024-03-06T07:27:38.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.402+0000 7f0be0e84640 1 --2- 172.21.15.83:0/3545705970 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0bbc06b4d0 0x7f0bbc06d990 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.402+0000 7f0be0e84640 1 --2- 172.21.15.83:0/3545705970 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0bdc15dfe0 0x7f0bdc159610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.402+0000 7f0be0e84640 1 --2- 172.21.15.83:0/3545705970 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0bdc0d22b0 0x7f0bdc160640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.402+0000 7f0be0e84640 1 --2- 172.21.15.83:0/3545705970 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0bdc0b41f0 0x7f0bdc0cc180 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.402+0000 7f0be0e84640 1 -- 172.21.15.83:0/3545705970 >> 172.21.15.83:0/3545705970 conn(0x7f0bdc0d3cd0 msgr2=0x7f0bdc06ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:38.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.402+0000 7f0be0e84640 1 -- 172.21.15.83:0/3545705970 shutdown_connections 2024-03-06T07:27:38.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:38.402+0000 7f0be0e84640 1 -- 172.21.15.83:0/3545705970 wait complete. 2024-03-06T07:27:38.415 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:27:38.415 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:27:38.415 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi083.front.sepia.ceph.com 2024-03-06T07:27:38.415 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:27:38.415 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:38.415 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:27:38.415 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:27:38.415 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:27:38.415 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:27:38.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:38.415 DEBUG:teuthology.orchestra.run.smithi083:> ip addr 2024-03-06T07:27:38.430 INFO:teuthology.orchestra.run.smithi083.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:27:38.430 INFO:teuthology.orchestra.run.smithi083.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:27:38.430 INFO:teuthology.orchestra.run.smithi083.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:27:38.430 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:38.430 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 ::1/128 scope host 2024-03-06T07:27:38.430 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:38.430 INFO:teuthology.orchestra.run.smithi083.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:38.431 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:d8:da:1e brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:38.431 INFO:teuthology.orchestra.run.smithi083.stdout: altname enp7s0f0 2024-03-06T07:27:38.431 INFO:teuthology.orchestra.run.smithi083.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:38.431 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:d8:da:1f brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:38.431 INFO:teuthology.orchestra.run.smithi083.stdout: altname enp7s0f1 2024-03-06T07:27:38.431 INFO:teuthology.orchestra.run.smithi083.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:38.431 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:8f:ce:90 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:38.431 INFO:teuthology.orchestra.run.smithi083.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:27:38.431 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:8f:ce:91 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:38.431 INFO:teuthology.orchestra.run.smithi083.stdout: inet 172.21.15.83/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:27:38.431 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft 42397sec preferred_lft 42397sec 2024-03-06T07:27:38.431 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::ec4:7aff:fe8f:ce91/64 scope link 2024-03-06T07:27:38.431 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:38.431 INFO:teuthology.orchestra.run.smithi083.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:27:38.431 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 1e:29:46:c9:e7:c8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:38.432 INFO:teuthology.orchestra.run.smithi083.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:27:38.432 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:38.432 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::60c5:d7ff:fe70:2b1c/64 scope link 2024-03-06T07:27:38.432 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:38.432 INFO:teuthology.orchestra.run.smithi083.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:27:38.432 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 1e:29:46:c9:e7:c8 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:38.432 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::1c29:46ff:fec9:e7c8/64 scope link 2024-03-06T07:27:38.432 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:38.432 INFO:teuthology.orchestra.run.smithi083.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:27:38.432 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 3e:00:2f:23:d5:45 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:38.432 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::3c00:2fff:fe23:d545/64 scope link 2024-03-06T07:27:38.432 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:38.432 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:38.433 DEBUG:teuthology.orchestra.run.smithi083:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:27:38.496 INFO:teuthology.orchestra.run.smithi083.stdout:1 2024-03-06T07:27:38.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:38.498 DEBUG:teuthology.orchestra.run.smithi083:> route 2024-03-06T07:27:38.554 INFO:teuthology.orchestra.run.smithi083.stdout:Kernel IP routing table 2024-03-06T07:27:38.554 INFO:teuthology.orchestra.run.smithi083.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:27:38.554 INFO:teuthology.orchestra.run.smithi083.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:27:38.554 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:27:38.554 INFO:teuthology.orchestra.run.smithi083.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:27:38.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:38.554 DEBUG:teuthology.orchestra.run.smithi083:> (cd / && exec bash -c ' 2024-03-06T07:27:38.554 DEBUG:teuthology.orchestra.run.smithi083:> set -e 2024-03-06T07:27:38.554 DEBUG:teuthology.orchestra.run.smithi083:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:27:38.555 DEBUG:teuthology.orchestra.run.smithi083:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:27:38.555 DEBUG:teuthology.orchestra.run.smithi083:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:27:38.555 DEBUG:teuthology.orchestra.run.smithi083:> ') 2024-03-06T07:27:38.649 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:38.649 DEBUG:teuthology.orchestra.run.smithi083:> ip netns list 2024-03-06T07:27:38.664 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:27:38.665 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:27:38.665 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:27:38.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:38.665 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip link set brx.0 up 2024-03-06T07:27:38.733 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:27:38.733 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:27:38.734 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:38.788 INFO:teuthology.orchestra.run.smithi083.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:27:38.788 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:27:38.789 DEBUG:teuthology.orchestra.run.smithi083:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:38.843 DEBUG:teuthology.orchestra.run.smithi083:> sudo modprobe fuse 2024-03-06T07:27:38.911 DEBUG:teuthology.orchestra.run.smithi083:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:27:38.967 INFO:teuthology.orchestra.run.smithi083.stdout:/proc 2024-03-06T07:27:38.967 INFO:teuthology.orchestra.run.smithi083.stdout:/sys 2024-03-06T07:27:38.967 INFO:teuthology.orchestra.run.smithi083.stdout:/dev 2024-03-06T07:27:38.967 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/security 2024-03-06T07:27:38.967 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/shm 2024-03-06T07:27:38.967 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/pts 2024-03-06T07:27:38.967 INFO:teuthology.orchestra.run.smithi083.stdout:/run 2024-03-06T07:27:38.967 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/cgroup 2024-03-06T07:27:38.967 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/pstore 2024-03-06T07:27:38.967 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/bpf 2024-03-06T07:27:38.967 INFO:teuthology.orchestra.run.smithi083.stdout:/ 2024-03-06T07:27:38.967 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/selinux 2024-03-06T07:27:38.968 INFO:teuthology.orchestra.run.smithi083.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:27:38.968 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/mqueue 2024-03-06T07:27:38.968 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/hugepages 2024-03-06T07:27:38.968 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/debug 2024-03-06T07:27:38.968 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/tracing 2024-03-06T07:27:38.968 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:27:38.968 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/fuse/connections 2024-03-06T07:27:38.968 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/config 2024-03-06T07:27:38.968 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:27:38.968 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:27:38.968 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:27:38.968 INFO:teuthology.orchestra.run.smithi083.stdout:/run/user/1000 2024-03-06T07:27:38.968 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph 2024-03-06T07:27:38.968 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:27:38.968 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:27:38.969 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:27:38.969 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:27:38.969 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns 2024-03-06T07:27:38.969 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:38.969 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:38.969 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:38.969 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:38.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:38.969 DEBUG:teuthology.orchestra.run.smithi083:> ls /sys/fs/fuse/connections 2024-03-06T07:27:39.005 INFO:tasks.ceph.mds.c.smithi083.stderr:2024-03-06T07:27:39.003+0000 7fe908279640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:27:39.020 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:27:39.020 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:27:39.062 DEBUG:teuthology.orchestra.run.smithi083:> sudo modprobe fuse 2024-03-06T07:27:39.091 DEBUG:teuthology.orchestra.run.smithi083:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:27:39.146 INFO:teuthology.orchestra.run.smithi083.stdout:/proc 2024-03-06T07:27:39.147 INFO:teuthology.orchestra.run.smithi083.stdout:/sys 2024-03-06T07:27:39.147 INFO:teuthology.orchestra.run.smithi083.stdout:/dev 2024-03-06T07:27:39.147 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/security 2024-03-06T07:27:39.147 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/shm 2024-03-06T07:27:39.147 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/pts 2024-03-06T07:27:39.147 INFO:teuthology.orchestra.run.smithi083.stdout:/run 2024-03-06T07:27:39.147 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/cgroup 2024-03-06T07:27:39.147 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/pstore 2024-03-06T07:27:39.147 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/bpf 2024-03-06T07:27:39.147 INFO:teuthology.orchestra.run.smithi083.stdout:/ 2024-03-06T07:27:39.147 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/selinux 2024-03-06T07:27:39.147 INFO:teuthology.orchestra.run.smithi083.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:27:39.147 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/mqueue 2024-03-06T07:27:39.148 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/hugepages 2024-03-06T07:27:39.148 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/debug 2024-03-06T07:27:39.148 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/tracing 2024-03-06T07:27:39.148 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:27:39.148 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/fuse/connections 2024-03-06T07:27:39.148 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/config 2024-03-06T07:27:39.148 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:27:39.148 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:27:39.148 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:27:39.148 INFO:teuthology.orchestra.run.smithi083.stdout:/run/user/1000 2024-03-06T07:27:39.148 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph 2024-03-06T07:27:39.148 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:27:39.148 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:27:39.148 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:27:39.149 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:27:39.149 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns 2024-03-06T07:27:39.149 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:39.149 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:39.149 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:39.149 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:39.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:39.150 DEBUG:teuthology.orchestra.run.smithi083:> ls /sys/fs/fuse/connections 2024-03-06T07:27:39.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:2024-03-06T07:27:39.146+0000 7f5927a36480 -1 init, newargv = 0x5630afec64e0 newargc=15 2024-03-06T07:27:39.151 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:ceph-fuse[74809]: starting ceph client 2024-03-06T07:27:39.152 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:ceph-fuse[74809]: starting fuse 2024-03-06T07:27:39.167 INFO:teuthology.orchestra.run.smithi083.stdout:41 2024-03-06T07:27:39.167 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-06T07:27:39.167 DEBUG:teuthology.orchestra.run.smithi083:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:27:39.167 DEBUG:teuthology.orchestra.run.smithi083:> import glob 2024-03-06T07:27:39.168 DEBUG:teuthology.orchestra.run.smithi083:> import re 2024-03-06T07:27:39.168 DEBUG:teuthology.orchestra.run.smithi083:> import os 2024-03-06T07:27:39.168 DEBUG:teuthology.orchestra.run.smithi083:> import subprocess 2024-03-06T07:27:39.168 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:27:39.168 DEBUG:teuthology.orchestra.run.smithi083:> def _find_admin_socket(client_name): 2024-03-06T07:27:39.168 DEBUG:teuthology.orchestra.run.smithi083:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:27:39.168 DEBUG:teuthology.orchestra.run.smithi083:> files = glob.glob(asok_path) 2024-03-06T07:27:39.168 DEBUG:teuthology.orchestra.run.smithi083:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:27:39.168 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:27:39.168 DEBUG:teuthology.orchestra.run.smithi083:> # Given a non-glob path, it better be there 2024-03-06T07:27:39.168 DEBUG:teuthology.orchestra.run.smithi083:> if "*" not in asok_path: 2024-03-06T07:27:39.168 DEBUG:teuthology.orchestra.run.smithi083:> assert(len(files) == 1) 2024-03-06T07:27:39.168 DEBUG:teuthology.orchestra.run.smithi083:> return files[0] 2024-03-06T07:27:39.169 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:27:39.169 DEBUG:teuthology.orchestra.run.smithi083:> for f in files: 2024-03-06T07:27:39.169 DEBUG:teuthology.orchestra.run.smithi083:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:27:39.169 DEBUG:teuthology.orchestra.run.smithi083:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:27:39.169 DEBUG:teuthology.orchestra.run.smithi083:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:27:39.169 DEBUG:teuthology.orchestra.run.smithi083:> contents = proc_f.read() 2024-03-06T07:27:39.169 DEBUG:teuthology.orchestra.run.smithi083:> if mountpoint in contents: 2024-03-06T07:27:39.169 DEBUG:teuthology.orchestra.run.smithi083:> return f 2024-03-06T07:27:39.169 DEBUG:teuthology.orchestra.run.smithi083:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:27:39.169 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:27:39.169 DEBUG:teuthology.orchestra.run.smithi083:> print(_find_admin_socket("client.0")) 2024-03-06T07:27:39.169 DEBUG:teuthology.orchestra.run.smithi083:> ' 2024-03-06T07:27:39.301 INFO:teuthology.orchestra.run.smithi083.stdout:/var/run/ceph/ceph-client.0.74809.asok 2024-03-06T07:27:39.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:27:39.313 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.74809.asok 2024-03-06T07:27:39.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:39.314 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.74809.asok status 2024-03-06T07:27:39.450 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:27:39.450 INFO:teuthology.orchestra.run.smithi083.stdout: "metadata": { 2024-03-06T07:27:39.450 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:27:39.450 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:27:39.450 INFO:teuthology.orchestra.run.smithi083.stdout: "entity_id": "0", 2024-03-06T07:27:39.450 INFO:teuthology.orchestra.run.smithi083.stdout: "hostname": "smithi083", 2024-03-06T07:27:39.450 INFO:teuthology.orchestra.run.smithi083.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:27:39.450 INFO:teuthology.orchestra.run.smithi083.stdout: "pid": "74809", 2024-03-06T07:27:39.451 INFO:teuthology.orchestra.run.smithi083.stdout: "root": "/" 2024-03-06T07:27:39.451 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:27:39.451 INFO:teuthology.orchestra.run.smithi083.stdout: "dentry_count": 0, 2024-03-06T07:27:39.451 INFO:teuthology.orchestra.run.smithi083.stdout: "dentry_pinned_count": 0, 2024-03-06T07:27:39.451 INFO:teuthology.orchestra.run.smithi083.stdout: "id": 8286, 2024-03-06T07:27:39.451 INFO:teuthology.orchestra.run.smithi083.stdout: "inst": { 2024-03-06T07:27:39.451 INFO:teuthology.orchestra.run.smithi083.stdout: "name": { 2024-03-06T07:27:39.451 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "client", 2024-03-06T07:27:39.451 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 8286 2024-03-06T07:27:39.451 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:27:39.451 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": { 2024-03-06T07:27:39.451 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-03-06T07:27:39.452 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:27:39.452 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 4179157882 2024-03-06T07:27:39.452 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:27:39.452 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:27:39.452 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": { 2024-03-06T07:27:39.452 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-03-06T07:27:39.452 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:27:39.452 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 4179157882 2024-03-06T07:27:39.452 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:27:39.452 INFO:teuthology.orchestra.run.smithi083.stdout: "inst_str": "client.8286 192.168.0.1:0/4179157882", 2024-03-06T07:27:39.452 INFO:teuthology.orchestra.run.smithi083.stdout: "addr_str": "192.168.0.1:0/4179157882", 2024-03-06T07:27:39.453 INFO:teuthology.orchestra.run.smithi083.stdout: "inode_count": 1, 2024-03-06T07:27:39.453 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_epoch": 66, 2024-03-06T07:27:39.453 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_epoch": 114, 2024-03-06T07:27:39.453 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:27:39.453 INFO:teuthology.orchestra.run.smithi083.stdout: "blocklisted": false, 2024-03-06T07:27:39.453 INFO:teuthology.orchestra.run.smithi083.stdout: "fs_name": "cephfs" 2024-03-06T07:27:39.453 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:27:39.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:39.458 DEBUG:teuthology.orchestra.run.smithi083:> stat --file-system '--printf=%T 2024-03-06T07:27:39.459 DEBUG:teuthology.orchestra.run.smithi083:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:39.474 INFO:teuthology.orchestra.run.smithi083.stdout:fuseblk 2024-03-06T07:27:39.474 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:39.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:39.474 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:39.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:39.541 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_1/file_a) 2024-03-06T07:27:39.599 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:39.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.739+0000 7f6c212f4640 1 Processor -- start 2024-03-06T07:27:39.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.739+0000 7f6c212f4640 1 -- start start 2024-03-06T07:27:39.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.740+0000 7f6c212f4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6c1c15fa10 0x7f6c1c0ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.740+0000 7f6c212f4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c1c0cebe0 con 0x7f6c1c15bdd0 2024-03-06T07:27:39.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.740+0000 7f6c212f4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c1c0ceec0 con 0x7f6c1c0d16e0 2024-03-06T07:27:39.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.740+0000 7f6c212f4640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c1c0cf1a0 con 0x7f6c1c15fa10 2024-03-06T07:27:39.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.740+0000 7f6c13fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6c1c15fa10 0x7f6c1c0ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.740+0000 7f6c1affd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f6c1c15bdd0 0x7f6c1c0c5610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43956/0 (socket says 172.21.15.83:43956) 2024-03-06T07:27:39.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.740+0000 7f6c13fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6c1c15fa10 0x7f6c1c0ce270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46668/0 (socket says 172.21.15.83:46668) 2024-03-06T07:27:39.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.740+0000 7f6c1affd640 1 -- 172.21.15.83:0/1756999045 learned_addr learned my addr 172.21.15.83:0/1756999045 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:27:39.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.740+0000 7f6c13fff640 1 -- 172.21.15.83:0/1756999045 >> v1:172.21.15.183:6789/0 conn(0x7f6c1c0d16e0 legacy=0x7f6c1c0d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.740+0000 7f6c13fff640 1 -- 172.21.15.83:0/1756999045 >> v1:172.21.15.83:6789/0 conn(0x7f6c1c15bdd0 legacy=0x7f6c1c0c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.740+0000 7f6c13fff640 1 -- 172.21.15.83:0/1756999045 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c1c0cc8e0 con 0x7f6c1c15fa10 2024-03-06T07:27:39.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.740+0000 7f6c1a7fc640 1 -- 172.21.15.83:0/1756999045 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3814333009 0 0) 0x7f6c1c0cebe0 con 0x7f6c1c15bdd0 2024-03-06T07:27:39.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.741+0000 7f6c13fff640 1 --2- 172.21.15.83:0/1756999045 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c1c15fa10 0x7f6c1c0ce270 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f6c040097a0 tx=0x7f6c0402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fd8122be5c27918 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:39.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.741+0000 7f6c1a7fc640 1 -- 172.21.15.83:0/1756999045 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c04035020 con 0x7f6c1c15fa10 2024-03-06T07:27:39.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.741+0000 7f6c1a7fc640 1 -- 172.21.15.83:0/1756999045 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c04031830 con 0x7f6c1c15fa10 2024-03-06T07:27:39.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.741+0000 7f6c1a7fc640 1 -- 172.21.15.83:0/1756999045 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c04031c00 con 0x7f6c1c15fa10 2024-03-06T07:27:39.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.742+0000 7f6c212f4640 1 -- 172.21.15.83:0/1756999045 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c1c15fa10 msgr2=0x7f6c1c0ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.742+0000 7f6c212f4640 1 --2- 172.21.15.83:0/1756999045 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c1c15fa10 0x7f6c1c0ce270 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f6c040097a0 tx=0x7f6c0402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.742+0000 7f6c212f4640 1 -- 172.21.15.83:0/1756999045 shutdown_connections 2024-03-06T07:27:39.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.742+0000 7f6c212f4640 1 --2- 172.21.15.83:0/1756999045 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c1c15fa10 0x7f6c1c0ce270 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.742+0000 7f6c212f4640 1 -- 172.21.15.83:0/1756999045 >> 172.21.15.83:0/1756999045 conn(0x7f6c1c09a9f0 msgr2=0x7f6c1c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:39.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.742+0000 7f6c212f4640 1 -- 172.21.15.83:0/1756999045 shutdown_connections 2024-03-06T07:27:39.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.742+0000 7f6c212f4640 1 -- 172.21.15.83:0/1756999045 wait complete. 2024-03-06T07:27:39.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.743+0000 7f6c212f4640 1 Processor -- start 2024-03-06T07:27:39.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.743+0000 7f6c212f4640 1 -- start start 2024-03-06T07:27:39.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.744+0000 7f6c212f4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c1c0d16e0 0x7f6c1c15e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.744+0000 7f6c212f4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6c1c15bdd0 0x7f6c1c1f11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.744+0000 7f6c212f4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c1c15fa10 0x7f6c1c0b6f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.744+0000 7f6c212f4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1c0ce7b0 con 0x7f6c1c0d16e0 2024-03-06T07:27:39.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.744+0000 7f6c212f4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1c15c1b0 con 0x7f6c1c15fa10 2024-03-06T07:27:39.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.744+0000 7f6c212f4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1c0ce930 con 0x7f6c1c15bdd0 2024-03-06T07:27:39.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.744+0000 7f6c1affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c1c0d16e0 0x7f6c1c15e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.744+0000 7f6c1affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c1c0d16e0 0x7f6c1c15e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46672/0 (socket says 172.21.15.83:46672) 2024-03-06T07:27:39.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.744+0000 7f6c1affd640 1 -- 172.21.15.83:0/27481817 learned_addr learned my addr 172.21.15.83:0/27481817 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:39.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.744+0000 7f6c13fff640 1 --2- 172.21.15.83:0/27481817 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6c1c15bdd0 0x7f6c1c1f11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.744+0000 7f6c1b7fe640 1 --2- 172.21.15.83:0/27481817 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c1c15fa10 0x7f6c1c0b6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.744+0000 7f6c1affd640 1 -- 172.21.15.83:0/27481817 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6c1c15bdd0 msgr2=0x7f6c1c1f11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.744+0000 7f6c1affd640 1 --2- 172.21.15.83:0/27481817 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6c1c15bdd0 0x7f6c1c1f11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.744+0000 7f6c1affd640 1 -- 172.21.15.83:0/27481817 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c1c15fa10 msgr2=0x7f6c1c0b6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.744+0000 7f6c1affd640 1 --2- 172.21.15.83:0/27481817 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c1c15fa10 0x7f6c1c0b6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.744+0000 7f6c1affd640 1 -- 172.21.15.83:0/27481817 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c1c15eb50 con 0x7f6c1c0d16e0 2024-03-06T07:27:39.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.744+0000 7f6c13fff640 1 --2- 172.21.15.83:0/27481817 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6c1c15bdd0 0x7f6c1c1f11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:39.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.744+0000 7f6c1affd640 1 --2- 172.21.15.83:0/27481817 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c1c0d16e0 0x7f6c1c15e4e0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c00c460 tx=0x7f6c0c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:39.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.745+0000 7f6c137fe640 1 -- 172.21.15.83:0/27481817 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c0c005ce0 con 0x7f6c1c0d16e0 2024-03-06T07:27:39.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.745+0000 7f6c137fe640 1 -- 172.21.15.83:0/27481817 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c0c007070 con 0x7f6c1c0d16e0 2024-03-06T07:27:39.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.745+0000 7f6c137fe640 1 -- 172.21.15.83:0/27481817 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c0c007460 con 0x7f6c1c0d16e0 2024-03-06T07:27:39.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.745+0000 7f6c212f4640 1 -- 172.21.15.83:0/27481817 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c1c15ee30 con 0x7f6c1c0d16e0 2024-03-06T07:27:39.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.745+0000 7f6c212f4640 1 -- 172.21.15.83:0/27481817 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c1c0c8cc0 con 0x7f6c1c0d16e0 2024-03-06T07:27:39.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.746+0000 7f6c212f4640 1 -- 172.21.15.83:0/27481817 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c1c04f330 con 0x7f6c1c0d16e0 2024-03-06T07:27:39.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.746+0000 7f6c137fe640 1 -- 172.21.15.83:0/27481817 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6c0c007600 con 0x7f6c1c0d16e0 2024-03-06T07:27:39.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.747+0000 7f6c137fe640 1 --2- 172.21.15.83:0/27481817 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6bfc06b4d0 0x7f6bfc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.747+0000 7f6c137fe640 1 -- 172.21.15.83:0/27481817 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f6c0c089f00 con 0x7f6c1c0d16e0 2024-03-06T07:27:39.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.747+0000 7f6c13fff640 1 --2- 172.21.15.83:0/27481817 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6bfc06b4d0 0x7f6bfc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.747+0000 7f6c13fff640 1 --2- 172.21.15.83:0/27481817 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6bfc06b4d0 0x7f6bfc06d990 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f6c04036000 tx=0x7f6c04008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:39.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.751+0000 7f6c137fe640 1 -- 172.21.15.83:0/27481817 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6c0c058d60 con 0x7f6c1c0d16e0 2024-03-06T07:27:39.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.966+0000 7f6c212f4640 1 -- 172.21.15.83:0/27481817 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c1c0b7a40 con 0x7f6c1c0d16e0 2024-03-06T07:27:39.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.966+0000 7f6c137fe640 1 -- 172.21.15.83:0/27481817 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f6c0c058660 con 0x7f6c1c0d16e0 2024-03-06T07:27:39.968 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:39.969 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7051,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/101642171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":101642171},{"type":"v1","addr":"172.21.15.183:6833","nonce":101642171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7057,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3146884900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":3146884900},{"type":"v1","addr":"172.21.15.183:6835","nonce":3146884900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8208,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1922041621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1922041621},{"type":"v1","addr":"172.21.15.83:6835","nonce":1922041621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.491022+0000","modified":"2024-03-06T07:27:37.006649+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8229":{"gid":8229,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/2003687153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2003687153},{"type":"v1","addr":"172.21.15.83:6837","nonce":2003687153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8229,"qdb_cluster":[8229]},"id":8}]} 2024-03-06T07:27:39.969 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 66 2024-03-06T07:27:39.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.968+0000 7f6c212f4640 1 -- 172.21.15.83:0/27481817 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6bfc06b4d0 msgr2=0x7f6bfc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.968+0000 7f6c212f4640 1 --2- 172.21.15.83:0/27481817 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6bfc06b4d0 0x7f6bfc06d990 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f6c04036000 tx=0x7f6c04008030 comp rx=0 tx=0).stop 2024-03-06T07:27:39.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.968+0000 7f6c212f4640 1 -- 172.21.15.83:0/27481817 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c1c0d16e0 msgr2=0x7f6c1c15e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.968+0000 7f6c212f4640 1 --2- 172.21.15.83:0/27481817 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c1c0d16e0 0x7f6c1c15e4e0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c00c460 tx=0x7f6c0c00c920 comp rx=0 tx=0).stop 2024-03-06T07:27:39.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.968+0000 7f6c212f4640 1 -- 172.21.15.83:0/27481817 shutdown_connections 2024-03-06T07:27:39.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.968+0000 7f6c212f4640 1 --2- 172.21.15.83:0/27481817 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6bfc06b4d0 0x7f6bfc06d990 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.968+0000 7f6c212f4640 1 --2- 172.21.15.83:0/27481817 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c1c15fa10 0x7f6c1c0b6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.968+0000 7f6c212f4640 1 --2- 172.21.15.83:0/27481817 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6c1c15bdd0 0x7f6c1c1f11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.968+0000 7f6c212f4640 1 --2- 172.21.15.83:0/27481817 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c1c0d16e0 0x7f6c1c15e4e0 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.968+0000 7f6c212f4640 1 -- 172.21.15.83:0/27481817 >> 172.21.15.83:0/27481817 conn(0x7f6c1c09a9f0 msgr2=0x7f6c1c09add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:39.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.968+0000 7f6c212f4640 1 -- 172.21.15.83:0/27481817 shutdown_connections 2024-03-06T07:27:39.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:39.968+0000 7f6c212f4640 1 -- 172.21.15.83:0/27481817 wait complete. 2024-03-06T07:27:39.982 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:27:40.103 INFO:teuthology.orchestra.run.smithi083.stdout:{"mds_cache":{"num_strays":1}} 2024-03-06T07:27:40.103 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 1 } } 2024-03-06T07:27:40.103 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:40.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.216+0000 7fcb73cfa640 1 Processor -- start 2024-03-06T07:27:40.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.216+0000 7fcb73cfa640 1 -- start start 2024-03-06T07:27:40.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.217+0000 7fcb73cfa640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcb6c0b4170 0x7fcb6c0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.217+0000 7fcb73cfa640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb6c0d25d0 con 0x7fcb6c0d0a80 2024-03-06T07:27:40.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.217+0000 7fcb73cfa640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb6c0d2870 con 0x7fcb6c0d4760 2024-03-06T07:27:40.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.217+0000 7fcb73cfa640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb6c0d4bf0 con 0x7fcb6c0b4170 2024-03-06T07:27:40.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.217+0000 7fcb71a96640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcb6c0b4170 0x7fcb6c0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:40.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.217+0000 7fcb71a96640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcb6c0b4170 0x7fcb6c0b6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46676/0 (socket says 172.21.15.83:46676) 2024-03-06T07:27:40.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.217+0000 7fcb71a96640 1 -- 172.21.15.83:0/3049671446 learned_addr learned my addr 172.21.15.83:0/3049671446 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:40.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.217+0000 7fcb72297640 1 --1- 172.21.15.83:0/3049671446 >> v1:172.21.15.83:6789/0 conn(0x7fcb6c0d0a80 0x7fcb6c0cee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43958/0 (socket says 172.21.15.83:43958) 2024-03-06T07:27:40.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.218+0000 7fcb71a96640 1 -- 172.21.15.83:0/3049671446 >> v1:172.21.15.183:6789/0 conn(0x7fcb6c0d4760 legacy=0x7fcb6c0d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:40.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.218+0000 7fcb71a96640 1 -- 172.21.15.83:0/3049671446 >> v1:172.21.15.83:6789/0 conn(0x7fcb6c0d0a80 legacy=0x7fcb6c0cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:40.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.218+0000 7fcb71a96640 1 -- 172.21.15.83:0/3049671446 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb6c0d2b50 con 0x7fcb6c0b4170 2024-03-06T07:27:40.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.218+0000 7fcb70a94640 1 -- 172.21.15.83:0/3049671446 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2477937571 0 0) 0x7fcb6c0d25d0 con 0x7fcb6c0d0a80 2024-03-06T07:27:40.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.218+0000 7fcb71a96640 1 --2- 172.21.15.83:0/3049671446 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcb6c0b4170 0x7fcb6c0b6a90 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7fcb64009870 tx=0x7fcb6402ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=689bb53eeac354c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:40.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.218+0000 7fcb70a94640 1 -- 172.21.15.83:0/3049671446 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb6402f840 con 0x7fcb6c0b4170 2024-03-06T07:27:40.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.218+0000 7fcb70a94640 1 -- 172.21.15.83:0/3049671446 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb6402f9e0 con 0x7fcb6c0b4170 2024-03-06T07:27:40.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.218+0000 7fcb70a94640 1 -- 172.21.15.83:0/3049671446 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb6402fdb0 con 0x7fcb6c0b4170 2024-03-06T07:27:40.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.219+0000 7fcb73cfa640 1 -- 172.21.15.83:0/3049671446 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcb6c0b4170 msgr2=0x7fcb6c0b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:40.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.219+0000 7fcb73cfa640 1 --2- 172.21.15.83:0/3049671446 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcb6c0b4170 0x7fcb6c0b6a90 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7fcb64009870 tx=0x7fcb6402ba20 comp rx=0 tx=0).stop 2024-03-06T07:27:40.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.219+0000 7fcb73cfa640 1 -- 172.21.15.83:0/3049671446 shutdown_connections 2024-03-06T07:27:40.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.219+0000 7fcb73cfa640 1 --2- 172.21.15.83:0/3049671446 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcb6c0b4170 0x7fcb6c0b6a90 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.219+0000 7fcb73cfa640 1 -- 172.21.15.83:0/3049671446 >> 172.21.15.83:0/3049671446 conn(0x7fcb6c0d3c40 msgr2=0x7fcb6c0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:40.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.219+0000 7fcb73cfa640 1 -- 172.21.15.83:0/3049671446 shutdown_connections 2024-03-06T07:27:40.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.219+0000 7fcb73cfa640 1 -- 172.21.15.83:0/3049671446 wait complete. 2024-03-06T07:27:40.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.220+0000 7fcb73cfa640 1 Processor -- start 2024-03-06T07:27:40.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.220+0000 7fcb73cfa640 1 -- start start 2024-03-06T07:27:40.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.221+0000 7fcb73cfa640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcb6c0b4170 0x7fcb6c0ca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.221+0000 7fcb73cfa640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcb6c0d0a80 0x7fcb6c1f4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.221+0000 7fcb73cfa640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcb6c0cd860 0x7fcb6c156e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.221+0000 7fcb73cfa640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6c0b4dd0 con 0x7fcb6c0d0a80 2024-03-06T07:27:40.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.221+0000 7fcb73cfa640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6c0ce9e0 con 0x7fcb6c0b4170 2024-03-06T07:27:40.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.221+0000 7fcb73cfa640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6c0b4f50 con 0x7fcb6c0cd860 2024-03-06T07:27:40.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.221+0000 7fcb71295640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcb6c0d0a80 0x7fcb6c1f4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:40.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.221+0000 7fcb71295640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcb6c0d0a80 0x7fcb6c1f4050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46678/0 (socket says 172.21.15.83:46678) 2024-03-06T07:27:40.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.221+0000 7fcb71295640 1 -- 172.21.15.83:0/3219035684 learned_addr learned my addr 172.21.15.83:0/3219035684 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:27:40.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.221+0000 7fcb71a96640 1 --2- 172.21.15.83:0/3219035684 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcb6c0b4170 0x7fcb6c0ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:40.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.221+0000 7fcb72297640 1 --2- 172.21.15.83:0/3219035684 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcb6c0cd860 0x7fcb6c156e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:40.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.221+0000 7fcb71295640 1 -- 172.21.15.83:0/3219035684 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcb6c0cd860 msgr2=0x7fcb6c156e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:40.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.221+0000 7fcb71295640 1 --2- 172.21.15.83:0/3219035684 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcb6c0cd860 0x7fcb6c156e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.221+0000 7fcb71295640 1 -- 172.21.15.83:0/3219035684 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcb6c0b4170 msgr2=0x7fcb6c0ca2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:40.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.221+0000 7fcb71295640 1 --2- 172.21.15.83:0/3219035684 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcb6c0b4170 0x7fcb6c0ca2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.221+0000 7fcb71295640 1 -- 172.21.15.83:0/3219035684 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb6c0cdcf0 con 0x7fcb6c0d0a80 2024-03-06T07:27:40.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.222+0000 7fcb71295640 1 --2- 172.21.15.83:0/3219035684 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcb6c0d0a80 0x7fcb6c1f4050 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c00c430 tx=0x7fcb5c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:40.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.222+0000 7fcb627fc640 1 -- 172.21.15.83:0/3219035684 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c017070 con 0x7fcb6c0d0a80 2024-03-06T07:27:40.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.222+0000 7fcb627fc640 1 -- 172.21.15.83:0/3219035684 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb5c0052d0 con 0x7fcb6c0d0a80 2024-03-06T07:27:40.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.222+0000 7fcb627fc640 1 -- 172.21.15.83:0/3219035684 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c003830 con 0x7fcb6c0d0a80 2024-03-06T07:27:40.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.222+0000 7fcb73cfa640 1 -- 172.21.15.83:0/3219035684 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb6c1590e0 con 0x7fcb6c0d0a80 2024-03-06T07:27:40.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.222+0000 7fcb73cfa640 1 -- 172.21.15.83:0/3219035684 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb6c1596a0 con 0x7fcb6c0d0a80 2024-03-06T07:27:40.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.223+0000 7fcb73cfa640 1 -- 172.21.15.83:0/3219035684 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb6c04f330 con 0x7fcb6c0d0a80 2024-03-06T07:27:40.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.224+0000 7fcb627fc640 1 -- 172.21.15.83:0/3219035684 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcb5c01a070 con 0x7fcb6c0d0a80 2024-03-06T07:27:40.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.224+0000 7fcb627fc640 1 --2- 172.21.15.83:0/3219035684 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcb5006b4d0 0x7fcb5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.224+0000 7fcb627fc640 1 -- 172.21.15.83:0/3219035684 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fcb5c0894e0 con 0x7fcb6c0d0a80 2024-03-06T07:27:40.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.224+0000 7fcb71a96640 1 --2- 172.21.15.83:0/3219035684 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcb5006b4d0 0x7fcb5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:40.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.225+0000 7fcb71a96640 1 --2- 172.21.15.83:0/3219035684 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcb5006b4d0 0x7fcb5006d990 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fcb64036000 tx=0x7fcb64008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:40.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.228+0000 7fcb627fc640 1 -- 172.21.15.83:0/3219035684 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcb5c057cb0 con 0x7fcb6c0d0a80 2024-03-06T07:27:40.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.443+0000 7fcb73cfa640 1 -- 172.21.15.83:0/3219035684 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb6c0733b0 con 0x7fcb6c0d0a80 2024-03-06T07:27:40.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.444+0000 7fcb627fc640 1 -- 172.21.15.83:0/3219035684 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4984 (secure 0 0 0) 0x7fcb5c05bd50 con 0x7fcb6c0d0a80 2024-03-06T07:27:40.446 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:27:40.446 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7051,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/101642171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":101642171},{"type":"v1","addr":"172.21.15.183:6833","nonce":101642171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7057,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3146884900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":3146884900},{"type":"v1","addr":"172.21.15.183:6835","nonce":3146884900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8208,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1922041621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1922041621},{"type":"v1","addr":"172.21.15.83:6835","nonce":1922041621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.491022+0000","modified":"2024-03-06T07:27:37.006649+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8229":{"gid":8229,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/2003687153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2003687153},{"type":"v1","addr":"172.21.15.83:6837","nonce":2003687153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8229,"qdb_cluster":[8229]},"id":8}]} 2024-03-06T07:27:40.446 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 66 2024-03-06T07:27:40.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.445+0000 7fcb73cfa640 1 -- 172.21.15.83:0/3219035684 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcb5006b4d0 msgr2=0x7fcb5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:40.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.445+0000 7fcb73cfa640 1 --2- 172.21.15.83:0/3219035684 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcb5006b4d0 0x7fcb5006d990 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fcb64036000 tx=0x7fcb64008030 comp rx=0 tx=0).stop 2024-03-06T07:27:40.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.445+0000 7fcb73cfa640 1 -- 172.21.15.83:0/3219035684 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcb6c0d0a80 msgr2=0x7fcb6c1f4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:40.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.445+0000 7fcb73cfa640 1 --2- 172.21.15.83:0/3219035684 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcb6c0d0a80 0x7fcb6c1f4050 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c00c430 tx=0x7fcb5c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.446+0000 7fcb73cfa640 1 -- 172.21.15.83:0/3219035684 shutdown_connections 2024-03-06T07:27:40.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.446+0000 7fcb73cfa640 1 --2- 172.21.15.83:0/3219035684 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcb5006b4d0 0x7fcb5006d990 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.446+0000 7fcb73cfa640 1 --2- 172.21.15.83:0/3219035684 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcb6c0cd860 0x7fcb6c156e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.446+0000 7fcb73cfa640 1 --2- 172.21.15.83:0/3219035684 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcb6c0d0a80 0x7fcb6c1f4050 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.446+0000 7fcb73cfa640 1 --2- 172.21.15.83:0/3219035684 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcb6c0b4170 0x7fcb6c0ca2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.446+0000 7fcb73cfa640 1 -- 172.21.15.83:0/3219035684 >> 172.21.15.83:0/3219035684 conn(0x7fcb6c0d3c40 msgr2=0x7fcb6c0b75d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:40.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.446+0000 7fcb73cfa640 1 -- 172.21.15.83:0/3219035684 shutdown_connections 2024-03-06T07:27:40.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:27:40.446+0000 7fcb73cfa640 1 -- 172.21.15.83:0/3219035684 wait complete. 2024-03-06T07:27:40.459 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:27:40.575 INFO:teuthology.orchestra.run.smithi083.stdout:{"mds_cache":{"strays_created":1}} 2024-03-06T07:27:40.575 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1 } } 2024-03-06T07:27:59.942 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:59.945 DEBUG:teuthology.orchestra.run.smithi183:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:10.576 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data ls 2024-03-06T07:28:10.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.614+0000 7f92d9648f40 1 Processor -- start 2024-03-06T07:28:10.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.614+0000 7f92d9648f40 1 -- start start 2024-03-06T07:28:10.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.614+0000 7f92d9648f40 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55d3061c5810 0x55d30627f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.614+0000 7f92d9648f40 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d30620cf70 0x55d30620aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.614+0000 7f92d9648f40 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d306282690 0x55d30620b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.614+0000 7f92d9648f40 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3061f3b60 con 0x55d306282690 2024-03-06T07:28:10.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.614+0000 7f92d9648f40 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d30616ab70 con 0x55d30620cf70 2024-03-06T07:28:10.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.614+0000 7f92d9648f40 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d3061c5bf0 con 0x55d3061c5810 2024-03-06T07:28:10.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.614+0000 7f92d70ae640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d306282690 0x55d30620b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.614+0000 7f92d70ae640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d306282690 0x55d30620b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51186/0 (socket says 172.21.15.83:51186) 2024-03-06T07:28:10.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.614+0000 7f92d70ae640 1 -- 172.21.15.83:0/572136581 learned_addr learned my addr 172.21.15.83:0/572136581 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:10.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.614+0000 7f92d68ad640 1 --2- 172.21.15.83:0/572136581 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55d3061c5810 0x55d30627f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.615+0000 7f92d60ac640 1 --2- 172.21.15.83:0/572136581 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d30620cf70 0x55d30620aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.615+0000 7f92d70ae640 1 -- 172.21.15.83:0/572136581 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55d3061c5810 msgr2=0x55d30627f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:10.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.615+0000 7f92d70ae640 1 --2- 172.21.15.83:0/572136581 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55d3061c5810 0x55d30627f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.615+0000 7f92d70ae640 1 -- 172.21.15.83:0/572136581 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d30620cf70 msgr2=0x55d30620aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:10.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.615+0000 7f92d70ae640 1 --2- 172.21.15.83:0/572136581 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d30620cf70 0x55d30620aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.615+0000 7f92d70ae640 1 -- 172.21.15.83:0/572136581 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d306280380 con 0x55d306282690 2024-03-06T07:28:10.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.615+0000 7f92d68ad640 1 --2- 172.21.15.83:0/572136581 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55d3061c5810 0x55d30627f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:10.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.615+0000 7f92d70ae640 1 --2- 172.21.15.83:0/572136581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d306282690 0x55d30620b400 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f92d0046fc0 tx=0x7f92d004f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c614197c8add6f28 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:10.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.615+0000 7f92d50aa640 1 -- 172.21.15.83:0/572136581 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d005a040 con 0x55d306282690 2024-03-06T07:28:10.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.615+0000 7f92d50aa640 1 -- 172.21.15.83:0/572136581 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92d0064070 con 0x55d306282690 2024-03-06T07:28:10.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.615+0000 7f92d50aa640 1 -- 172.21.15.83:0/572136581 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d004dd00 con 0x55d306282690 2024-03-06T07:28:10.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.615+0000 7f92d9648f40 1 -- 172.21.15.83:0/572136581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d306282690 msgr2=0x55d30620b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.615+0000 7f92d9648f40 1 --2- 172.21.15.83:0/572136581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d306282690 0x55d30620b400 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f92d0046fc0 tx=0x7f92d004f000 comp rx=0 tx=0).stop 2024-03-06T07:28:10.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.615+0000 7f92d9648f40 1 -- 172.21.15.83:0/572136581 shutdown_connections 2024-03-06T07:28:10.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.615+0000 7f92d9648f40 1 --2- 172.21.15.83:0/572136581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d306282690 0x55d30620b400 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.615+0000 7f92d9648f40 1 --2- 172.21.15.83:0/572136581 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d30620cf70 0x55d30620aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.615+0000 7f92d9648f40 1 --2- 172.21.15.83:0/572136581 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55d3061c5810 0x55d30627f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.615+0000 7f92d9648f40 1 -- 172.21.15.83:0/572136581 >> 172.21.15.83:0/572136581 conn(0x55d3061ef340 msgr2=0x55d306203a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:10.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.616+0000 7f92d9648f40 1 -- 172.21.15.83:0/572136581 shutdown_connections 2024-03-06T07:28:10.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.616+0000 7f92d9648f40 1 -- 172.21.15.83:0/572136581 wait complete. 2024-03-06T07:28:10.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.616+0000 7f92d9648f40 1 Processor -- start 2024-03-06T07:28:10.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.616+0000 7f92d9648f40 1 -- start start 2024-03-06T07:28:10.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.617+0000 7f92d9648f40 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55d3061c5810 0x55d3061f2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.617+0000 7f92d9648f40 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d30620cf70 0x55d3063340d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.617+0000 7f92d9648f40 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d306282690 0x55d306334610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.617+0000 7f92d9648f40 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d306204c60 con 0x55d30620cf70 2024-03-06T07:28:10.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.617+0000 7f92d9648f40 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d306204910 con 0x55d306282690 2024-03-06T07:28:10.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.617+0000 7f92d9648f40 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d306298760 con 0x55d3061c5810 2024-03-06T07:28:10.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.617+0000 7f92d60ac640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d30620cf70 0x55d3063340d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.617+0000 7f92d60ac640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d30620cf70 0x55d3063340d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51188/0 (socket says 172.21.15.83:51188) 2024-03-06T07:28:10.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.617+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 learned_addr learned my addr 172.21.15.83:0/1482419184 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:10.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.617+0000 7f92d68ad640 1 --2- 172.21.15.83:0/1482419184 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55d3061c5810 0x55d3061f2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.617+0000 7f92d70ae640 1 --2- 172.21.15.83:0/1482419184 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d306282690 0x55d306334610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.617+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55d3061c5810 msgr2=0x55d3061f2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.617+0000 7f92d60ac640 1 --2- 172.21.15.83:0/1482419184 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55d3061c5810 0x55d3061f2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.617+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d306282690 msgr2=0x55d306334610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.617+0000 7f92d60ac640 1 --2- 172.21.15.83:0/1482419184 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d306282690 0x55d306334610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.617+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d306298a00 con 0x55d30620cf70 2024-03-06T07:28:10.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.617+0000 7f92d60ac640 1 --2- 172.21.15.83:0/1482419184 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d30620cf70 0x55d3063340d0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f92cc00df70 tx=0x7f92cc00b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:10.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.618+0000 7f92bffff640 1 -- 172.21.15.83:0/1482419184 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92cc0032d0 con 0x55d30620cf70 2024-03-06T07:28:10.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.618+0000 7f92bffff640 1 -- 172.21.15.83:0/1482419184 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92cc016070 con 0x55d30620cf70 2024-03-06T07:28:10.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.618+0000 7f92d9648f40 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55d306298ce0 con 0x55d30620cf70 2024-03-06T07:28:10.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.618+0000 7f92bffff640 1 -- 172.21.15.83:0/1482419184 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92cc0017f0 con 0x55d30620cf70 2024-03-06T07:28:10.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.618+0000 7f92d9648f40 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d3062967b0 con 0x55d30620cf70 2024-03-06T07:28:10.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.619+0000 7f92bffff640 1 -- 172.21.15.83:0/1482419184 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f92cc001990 con 0x55d30620cf70 2024-03-06T07:28:10.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.619+0000 7f92bffff640 1 --2- 172.21.15.83:0/1482419184 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f92c4069ee0 0x7f92c406c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.620+0000 7f92bffff640 1 -- 172.21.15.83:0/1482419184 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f92cc088de0 con 0x55d30620cf70 2024-03-06T07:28:10.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.620+0000 7f92d68ad640 1 --2- 172.21.15.83:0/1482419184 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f92c4069ee0 0x7f92c406c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.620+0000 7f92d9648f40 1 --2- 172.21.15.83:0/1482419184 >> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] conn(0x55d306267a00 0x55d306269b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.620+0000 7f92d9648f40 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] -- osd_op(unknown.0.0:1 17.0 17:00000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x55d306267a00 2024-03-06T07:28:10.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.620+0000 7f92d70ae640 1 --2- 172.21.15.83:0/1482419184 >> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] conn(0x55d306267a00 0x55d306269b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.620+0000 7f92d68ad640 1 --2- 172.21.15.83:0/1482419184 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f92c4069ee0 0x7f92c406c3a0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x55d3061c46b0 tx=0x7f92c80053c0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:10.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.620+0000 7f92d70ae640 1 --2- 172.21.15.83:0/1482419184 >> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] conn(0x55d306267a00 0x55d306269b80 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:10.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.621+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 <== osd.7 v2:172.21.15.183:6820/3336566695 1 ==== osd_op_reply(1 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92d005d040 con 0x55d306267a00 2024-03-06T07:28:10.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.621+0000 7f92d70ae640 1 --2- 172.21.15.83:0/1482419184 >> [v2:172.21.15.83:6814/1822925240,v1:172.21.15.83:6817/1822925240] conn(0x7f92d0055030 0x7f92d006a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.621+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.83:6814/1822925240,v1:172.21.15.83:6817/1822925240] -- osd_op(unknown.0.0:2 17.20 17:04000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92d0055030 2024-03-06T07:28:10.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.621+0000 7f92d68ad640 1 --2- 172.21.15.83:0/1482419184 >> [v2:172.21.15.83:6814/1822925240,v1:172.21.15.83:6817/1822925240] conn(0x7f92d0055030 0x7f92d006a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.621+0000 7f92d68ad640 1 --2- 172.21.15.83:0/1482419184 >> [v2:172.21.15.83:6814/1822925240,v1:172.21.15.83:6817/1822925240] conn(0x7f92d0055030 0x7f92d006a410 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:10.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.621+0000 7f92d68ad640 1 -- 172.21.15.83:0/1482419184 <== osd.1 v2:172.21.15.83:6814/1822925240 1 ==== osd_op_reply(2 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92c80168c0 con 0x7f92d0055030 2024-03-06T07:28:10.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.622+0000 7f92d68ad640 1 --2- 172.21.15.83:0/1482419184 >> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] conn(0x7f92c8008910 0x7f92c80173f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.622+0000 7f92d68ad640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] -- osd_op(unknown.0.0:3 17.10 17:08000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92c8008910 2024-03-06T07:28:10.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.622+0000 7f92d60ac640 1 --2- 172.21.15.83:0/1482419184 >> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] conn(0x7f92c8008910 0x7f92c80173f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.622+0000 7f92d60ac640 1 --2- 172.21.15.83:0/1482419184 >> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] conn(0x7f92c8008910 0x7f92c80173f0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:10.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.622+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 <== osd.3 v2:172.21.15.83:6815/2656563756 1 ==== osd_op_reply(3 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92cc00e040 con 0x7f92c8008910 2024-03-06T07:28:10.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.622+0000 7f92d60ac640 1 --2- 172.21.15.83:0/1482419184 >> [v2:172.21.15.183:6808/2957306877,v1:172.21.15.183:6809/2957306877] conn(0x7f92cc057810 0x7f92cc056470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.622+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.183:6808/2957306877,v1:172.21.15.183:6809/2957306877] -- osd_op(unknown.0.0:4 17.30 17:0c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92cc057810 2024-03-06T07:28:10.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.622+0000 7f92d70ae640 1 --2- 172.21.15.83:0/1482419184 >> [v2:172.21.15.183:6808/2957306877,v1:172.21.15.183:6809/2957306877] conn(0x7f92cc057810 0x7f92cc056470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.623+0000 7f92d70ae640 1 --2- 172.21.15.83:0/1482419184 >> [v2:172.21.15.183:6808/2957306877,v1:172.21.15.183:6809/2957306877] conn(0x7f92cc057810 0x7f92cc056470 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:10.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.623+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 <== osd.5 v2:172.21.15.183:6808/2957306877 1 ==== osd_op_reply(4 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92d005d040 con 0x7f92cc057810 2024-03-06T07:28:10.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.623+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] -- osd_op(unknown.0.0:5 17.8 17:10000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x55d306267a00 2024-03-06T07:28:10.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.624+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 <== osd.7 v2:172.21.15.183:6820/3336566695 2 ==== osd_op_reply(5 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92d005d040 con 0x55d306267a00 2024-03-06T07:28:10.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.624+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] -- osd_op(unknown.0.0:6 17.28 17:14000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92c8008910 2024-03-06T07:28:10.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.624+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 <== osd.3 v2:172.21.15.83:6815/2656563756 2 ==== osd_op_reply(6 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92cc00e040 con 0x7f92c8008910 2024-03-06T07:28:10.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.624+0000 7f92d60ac640 1 --2- 172.21.15.83:0/1482419184 >> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] conn(0x7f92cc05f780 0x7f92cc058830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.624+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] -- osd_op(unknown.0.0:7 17.18 17:18000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92cc05f780 2024-03-06T07:28:10.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.624+0000 7f92d68ad640 1 --2- 172.21.15.83:0/1482419184 >> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] conn(0x7f92cc05f780 0x7f92cc058830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.624+0000 7f92d68ad640 1 --2- 172.21.15.83:0/1482419184 >> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] conn(0x7f92cc05f780 0x7f92cc058830 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:10.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.625+0000 7f92d68ad640 1 -- 172.21.15.83:0/1482419184 <== osd.0 v2:172.21.15.83:6808/1635427425 1 ==== osd_op_reply(7 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92c80168c0 con 0x7f92cc05f780 2024-03-06T07:28:10.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.625+0000 7f92d68ad640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] -- osd_op(unknown.0.0:8 17.38 17:1c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x55d306267a00 2024-03-06T07:28:10.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.625+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 <== osd.7 v2:172.21.15.183:6820/3336566695 3 ==== osd_op_reply(8 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92d005d040 con 0x55d306267a00 2024-03-06T07:28:10.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.625+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] -- osd_op(unknown.0.0:9 17.4 17:20000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92cc05f780 2024-03-06T07:28:10.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.625+0000 7f92d68ad640 1 -- 172.21.15.83:0/1482419184 <== osd.0 v2:172.21.15.83:6808/1635427425 2 ==== osd_op_reply(9 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92c80168c0 con 0x7f92cc05f780 2024-03-06T07:28:10.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.625+0000 7f92d68ad640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] -- osd_op(unknown.0.0:10 17.24 17:24000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92c8008910 2024-03-06T07:28:10.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.625+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 <== osd.3 v2:172.21.15.83:6815/2656563756 3 ==== osd_op_reply(10 [pgnls start_epoch 116 out=49b] v110'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92cc00e040 con 0x7f92c8008910 2024-03-06T07:28:10.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.625+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] -- osd_op(unknown.0.0:11 17.14 17:28000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92c8008910 2024-03-06T07:28:10.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.626+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 <== osd.3 v2:172.21.15.83:6815/2656563756 4 ==== osd_op_reply(11 [pgnls start_epoch 116 out=49b] v110'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92cc00e040 con 0x7f92c8008910 2024-03-06T07:28:10.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.626+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] -- osd_op(unknown.0.0:12 17.34 17:2c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92cc05f780 2024-03-06T07:28:10.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.626+0000 7f92d68ad640 1 -- 172.21.15.83:0/1482419184 <== osd.0 v2:172.21.15.83:6808/1635427425 3 ==== osd_op_reply(12 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92c80168c0 con 0x7f92cc05f780 2024-03-06T07:28:10.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.626+0000 7f92d68ad640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] -- osd_op(unknown.0.0:13 17.c 17:30000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x55d306267a00 2024-03-06T07:28:10.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.626+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 <== osd.7 v2:172.21.15.183:6820/3336566695 4 ==== osd_op_reply(13 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92d005d040 con 0x55d306267a00 2024-03-06T07:28:10.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.626+0000 7f92d70ae640 1 --2- 172.21.15.83:0/1482419184 >> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] conn(0x7f92d0055920 0x7f92d00783f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.626+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] -- osd_op(unknown.0.0:14 17.2c 17:34000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92d0055920 2024-03-06T07:28:10.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.626+0000 7f92d60ac640 1 --2- 172.21.15.83:0/1482419184 >> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] conn(0x7f92d0055920 0x7f92d00783f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.627+0000 7f92d60ac640 1 --2- 172.21.15.83:0/1482419184 >> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] conn(0x7f92d0055920 0x7f92d00783f0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:10.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.627+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 <== osd.4 v2:172.21.15.183:6816/3256618830 1 ==== osd_op_reply(14 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92cc00e040 con 0x7f92d0055920 2024-03-06T07:28:10.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.627+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.83:6814/1822925240,v1:172.21.15.83:6817/1822925240] -- osd_op(unknown.0.0:15 17.1c 17:38000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92d0055030 2024-03-06T07:28:10.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.627+0000 7f92d68ad640 1 -- 172.21.15.83:0/1482419184 <== osd.1 v2:172.21.15.83:6814/1822925240 2 ==== osd_op_reply(15 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92c80168c0 con 0x7f92d0055030 2024-03-06T07:28:10.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.627+0000 7f92d68ad640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] -- osd_op(unknown.0.0:16 17.3c 17:3c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92cc05f780 2024-03-06T07:28:10.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.628+0000 7f92d68ad640 1 -- 172.21.15.83:0/1482419184 <== osd.0 v2:172.21.15.83:6808/1635427425 4 ==== osd_op_reply(16 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92c80168c0 con 0x7f92cc05f780 2024-03-06T07:28:10.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.628+0000 7f92d68ad640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] -- osd_op(unknown.0.0:17 17.2 17:40000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92d0055920 2024-03-06T07:28:10.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.628+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 <== osd.4 v2:172.21.15.183:6816/3256618830 2 ==== osd_op_reply(17 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92cc00e040 con 0x7f92d0055920 2024-03-06T07:28:10.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.628+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.183:6808/2957306877,v1:172.21.15.183:6809/2957306877] -- osd_op(unknown.0.0:18 17.22 17:44000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92cc057810 2024-03-06T07:28:10.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.628+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 <== osd.5 v2:172.21.15.183:6808/2957306877 2 ==== osd_op_reply(18 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92d005d040 con 0x7f92cc057810 2024-03-06T07:28:10.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.628+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] -- osd_op(unknown.0.0:19 17.12 17:48000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92c8008910 2024-03-06T07:28:10.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.629+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 <== osd.3 v2:172.21.15.83:6815/2656563756 5 ==== osd_op_reply(19 [pgnls start_epoch 116 out=49b] v110'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92cc00e040 con 0x7f92c8008910 2024-03-06T07:28:10.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.629+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] -- osd_op(unknown.0.0:20 17.32 17:4c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x55d306267a00 2024-03-06T07:28:10.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.629+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 <== osd.7 v2:172.21.15.183:6820/3336566695 5 ==== osd_op_reply(20 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92d005d040 con 0x55d306267a00 2024-03-06T07:28:10.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.629+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] -- osd_op(unknown.0.0:21 17.a 17:50000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x55d306267a00 2024-03-06T07:28:10.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.629+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 <== osd.7 v2:172.21.15.183:6820/3336566695 6 ==== osd_op_reply(21 [pgnls start_epoch 116 out=49b] v110'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92d005d040 con 0x55d306267a00 2024-03-06T07:28:10.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.629+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] -- osd_op(unknown.0.0:22 17.2a 17:54000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92c8008910 2024-03-06T07:28:10.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.629+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 <== osd.3 v2:172.21.15.83:6815/2656563756 6 ==== osd_op_reply(22 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92cc00e040 con 0x7f92c8008910 2024-03-06T07:28:10.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.629+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] -- osd_op(unknown.0.0:23 17.1a 17:58000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92cc05f780 2024-03-06T07:28:10.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.630+0000 7f92d68ad640 1 -- 172.21.15.83:0/1482419184 <== osd.0 v2:172.21.15.83:6808/1635427425 5 ==== osd_op_reply(23 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92c80168c0 con 0x7f92cc05f780 2024-03-06T07:28:10.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.630+0000 7f92d68ad640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.83:6814/1822925240,v1:172.21.15.83:6817/1822925240] -- osd_op(unknown.0.0:24 17.3a 17:5c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92d0055030 2024-03-06T07:28:10.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.630+0000 7f92d68ad640 1 -- 172.21.15.83:0/1482419184 <== osd.1 v2:172.21.15.83:6814/1822925240 3 ==== osd_op_reply(24 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92c80168c0 con 0x7f92d0055030 2024-03-06T07:28:10.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.630+0000 7f92d68ad640 1 --2- 172.21.15.83:0/1482419184 >> [v2:172.21.15.183:6800/2649185225,v1:172.21.15.183:6801/2649185225] conn(0x7f92c8027960 0x7f92c8025430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.630+0000 7f92d68ad640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.183:6800/2649185225,v1:172.21.15.183:6801/2649185225] -- osd_op(unknown.0.0:25 17.6 17:60000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92c8027960 2024-03-06T07:28:10.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.630+0000 7f92d70ae640 1 --2- 172.21.15.83:0/1482419184 >> [v2:172.21.15.183:6800/2649185225,v1:172.21.15.183:6801/2649185225] conn(0x7f92c8027960 0x7f92c8025430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.631+0000 7f92d70ae640 1 --2- 172.21.15.83:0/1482419184 >> [v2:172.21.15.183:6800/2649185225,v1:172.21.15.183:6801/2649185225] conn(0x7f92c8027960 0x7f92c8025430 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:10.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.631+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 <== osd.6 v2:172.21.15.183:6800/2649185225 1 ==== osd_op_reply(25 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92d005d040 con 0x7f92c8027960 2024-03-06T07:28:10.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.631+0000 7f92d70ae640 1 --2- 172.21.15.83:0/1482419184 >> [v2:172.21.15.83:6800/2591342133,v1:172.21.15.83:6801/2591342133] conn(0x7f92d0088400 0x7f92d0088840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.631+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.83:6800/2591342133,v1:172.21.15.83:6801/2591342133] -- osd_op(unknown.0.0:26 17.26 17:64000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92d0088400 2024-03-06T07:28:10.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.631+0000 7f92d68ad640 1 --2- 172.21.15.83:0/1482419184 >> [v2:172.21.15.83:6800/2591342133,v1:172.21.15.83:6801/2591342133] conn(0x7f92d0088400 0x7f92d0088840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.631+0000 7f92d68ad640 1 --2- 172.21.15.83:0/1482419184 >> [v2:172.21.15.83:6800/2591342133,v1:172.21.15.83:6801/2591342133] conn(0x7f92d0088400 0x7f92d0088840 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:10.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.632+0000 7f92d68ad640 1 -- 172.21.15.83:0/1482419184 <== osd.2 v2:172.21.15.83:6800/2591342133 1 ==== osd_op_reply(26 [pgnls start_epoch 116 out=49b] v110'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92c80168c0 con 0x7f92d0088400 2024-03-06T07:28:10.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.632+0000 7f92d68ad640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.183:6808/2957306877,v1:172.21.15.183:6809/2957306877] -- osd_op(unknown.0.0:27 17.16 17:68000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92cc057810 2024-03-06T07:28:10.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.632+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 <== osd.5 v2:172.21.15.183:6808/2957306877 3 ==== osd_op_reply(27 [pgnls start_epoch 116 out=49b] v110'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92d005d040 con 0x7f92cc057810 2024-03-06T07:28:10.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.632+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] -- osd_op(unknown.0.0:28 17.36 17:6c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92cc05f780 2024-03-06T07:28:10.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.632+0000 7f92d68ad640 1 -- 172.21.15.83:0/1482419184 <== osd.0 v2:172.21.15.83:6808/1635427425 6 ==== osd_op_reply(28 [pgnls start_epoch 116 out=49b] v110'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92c80168c0 con 0x7f92cc05f780 2024-03-06T07:28:10.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.632+0000 7f92d68ad640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.83:6800/2591342133,v1:172.21.15.83:6801/2591342133] -- osd_op(unknown.0.0:29 17.e 17:70000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92d0088400 2024-03-06T07:28:10.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.633+0000 7f92d68ad640 1 -- 172.21.15.83:0/1482419184 <== osd.2 v2:172.21.15.83:6800/2591342133 2 ==== osd_op_reply(29 [pgnls start_epoch 116 out=49b] v110'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92c80168c0 con 0x7f92d0088400 2024-03-06T07:28:10.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.633+0000 7f92d68ad640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.183:6800/2649185225,v1:172.21.15.183:6801/2649185225] -- osd_op(unknown.0.0:30 17.2e 17:74000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92c8027960 2024-03-06T07:28:10.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.633+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 <== osd.6 v2:172.21.15.183:6800/2649185225 2 ==== osd_op_reply(30 [pgnls start_epoch 116 out=49b] v110'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92d005d040 con 0x7f92c8027960 2024-03-06T07:28:10.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.633+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.83:6814/1822925240,v1:172.21.15.83:6817/1822925240] -- osd_op(unknown.0.0:31 17.1e 17:78000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92d0055030 2024-03-06T07:28:10.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.633+0000 7f92d68ad640 1 -- 172.21.15.83:0/1482419184 <== osd.1 v2:172.21.15.83:6814/1822925240 4 ==== osd_op_reply(31 [pgnls start_epoch 116 out=49b] v110'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92c80168c0 con 0x7f92d0055030 2024-03-06T07:28:10.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.633+0000 7f92d68ad640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] -- osd_op(unknown.0.0:32 17.3e 17:7c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92d0055920 2024-03-06T07:28:10.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.634+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 <== osd.4 v2:172.21.15.183:6816/3256618830 3 ==== osd_op_reply(32 [pgnls start_epoch 116 out=49b] v110'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92cc00e040 con 0x7f92d0055920 2024-03-06T07:28:10.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.634+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] -- osd_op(unknown.0.0:33 17.1 17:80000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92cc05f780 2024-03-06T07:28:10.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.634+0000 7f92d68ad640 1 -- 172.21.15.83:0/1482419184 <== osd.0 v2:172.21.15.83:6808/1635427425 7 ==== osd_op_reply(33 [pgnls start_epoch 116 out=49b] v110'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92c80168c0 con 0x7f92cc05f780 2024-03-06T07:28:10.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.634+0000 7f92d68ad640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] -- osd_op(unknown.0.0:34 17.21 17:84000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92cc05f780 2024-03-06T07:28:10.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.634+0000 7f92d68ad640 1 -- 172.21.15.83:0/1482419184 <== osd.0 v2:172.21.15.83:6808/1635427425 8 ==== osd_op_reply(34 [pgnls start_epoch 116 out=49b] v110'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92c80168c0 con 0x7f92cc05f780 2024-03-06T07:28:10.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.634+0000 7f92d68ad640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] -- osd_op(unknown.0.0:35 17.11 17:88000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92c8008910 2024-03-06T07:28:10.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.634+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 <== osd.3 v2:172.21.15.83:6815/2656563756 7 ==== osd_op_reply(35 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92cc00e040 con 0x7f92c8008910 2024-03-06T07:28:10.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.634+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.83:6814/1822925240,v1:172.21.15.83:6817/1822925240] -- osd_op(unknown.0.0:36 17.31 17:8c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92d0055030 2024-03-06T07:28:10.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.635+0000 7f92d68ad640 1 -- 172.21.15.83:0/1482419184 <== osd.1 v2:172.21.15.83:6814/1822925240 5 ==== osd_op_reply(36 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92c80168c0 con 0x7f92d0055030 2024-03-06T07:28:10.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.635+0000 7f92d68ad640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] -- osd_op(unknown.0.0:37 17.9 17:90000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92c8008910 2024-03-06T07:28:10.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.635+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 <== osd.3 v2:172.21.15.83:6815/2656563756 8 ==== osd_op_reply(37 [pgnls start_epoch 116 out=49b] v110'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92cc00e040 con 0x7f92c8008910 2024-03-06T07:28:10.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.635+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] -- osd_op(unknown.0.0:38 17.29 17:94000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92d0055920 2024-03-06T07:28:10.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.635+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 <== osd.4 v2:172.21.15.183:6816/3256618830 4 ==== osd_op_reply(38 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92cc00e040 con 0x7f92d0055920 2024-03-06T07:28:10.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.635+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.183:6808/2957306877,v1:172.21.15.183:6809/2957306877] -- osd_op(unknown.0.0:39 17.19 17:98000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92cc057810 2024-03-06T07:28:10.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.636+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 <== osd.5 v2:172.21.15.183:6808/2957306877 4 ==== osd_op_reply(39 [pgnls start_epoch 116 out=49b] v110'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92d005d040 con 0x7f92cc057810 2024-03-06T07:28:10.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.636+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.183:6808/2957306877,v1:172.21.15.183:6809/2957306877] -- osd_op(unknown.0.0:40 17.39 17:9c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92cc057810 2024-03-06T07:28:10.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.636+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 <== osd.5 v2:172.21.15.183:6808/2957306877 5 ==== osd_op_reply(40 [pgnls start_epoch 116 out=49b] v110'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92d005d040 con 0x7f92cc057810 2024-03-06T07:28:10.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.636+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.183:6800/2649185225,v1:172.21.15.183:6801/2649185225] -- osd_op(unknown.0.0:41 17.5 17:a0000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92c8027960 2024-03-06T07:28:10.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.636+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 <== osd.6 v2:172.21.15.183:6800/2649185225 3 ==== osd_op_reply(41 [pgnls start_epoch 116 out=49b] v110'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92d005d040 con 0x7f92c8027960 2024-03-06T07:28:10.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.636+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.83:6814/1822925240,v1:172.21.15.83:6817/1822925240] -- osd_op(unknown.0.0:42 17.25 17:a4000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92d0055030 2024-03-06T07:28:10.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.637+0000 7f92d68ad640 1 -- 172.21.15.83:0/1482419184 <== osd.1 v2:172.21.15.83:6814/1822925240 6 ==== osd_op_reply(42 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92c80168c0 con 0x7f92d0055030 2024-03-06T07:28:10.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.637+0000 7f92d68ad640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.183:6808/2957306877,v1:172.21.15.183:6809/2957306877] -- osd_op(unknown.0.0:43 17.15 17:a8000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92cc057810 2024-03-06T07:28:10.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.637+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 <== osd.5 v2:172.21.15.183:6808/2957306877 6 ==== osd_op_reply(43 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92d005d040 con 0x7f92cc057810 2024-03-06T07:28:10.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.637+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] -- osd_op(unknown.0.0:44 17.35 17:ac000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92d0055920 2024-03-06T07:28:10.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.637+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 <== osd.4 v2:172.21.15.183:6816/3256618830 5 ==== osd_op_reply(44 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92cc00e040 con 0x7f92d0055920 2024-03-06T07:28:10.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.637+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] -- osd_op(unknown.0.0:45 17.d 17:b0000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x55d306267a00 2024-03-06T07:28:10.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.638+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 <== osd.7 v2:172.21.15.183:6820/3336566695 7 ==== osd_op_reply(45 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92d005d040 con 0x55d306267a00 2024-03-06T07:28:10.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.638+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] -- osd_op(unknown.0.0:46 17.2d 17:b4000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x55d306267a00 2024-03-06T07:28:10.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.638+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 <== osd.7 v2:172.21.15.183:6820/3336566695 8 ==== osd_op_reply(46 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92d005d040 con 0x55d306267a00 2024-03-06T07:28:10.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.638+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] -- osd_op(unknown.0.0:47 17.1d 17:b8000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x55d306267a00 2024-03-06T07:28:10.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.638+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 <== osd.7 v2:172.21.15.183:6820/3336566695 9 ==== osd_op_reply(47 [pgnls start_epoch 116 out=49b] v110'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92d005d040 con 0x55d306267a00 2024-03-06T07:28:10.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.638+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] -- osd_op(unknown.0.0:48 17.3d 17:bc000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x55d306267a00 2024-03-06T07:28:10.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.638+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 <== osd.7 v2:172.21.15.183:6820/3336566695 10 ==== osd_op_reply(48 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92d005d040 con 0x55d306267a00 2024-03-06T07:28:10.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.638+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] -- osd_op(unknown.0.0:49 17.3 17:c0000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92d0055920 2024-03-06T07:28:10.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.639+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 <== osd.4 v2:172.21.15.183:6816/3256618830 6 ==== osd_op_reply(49 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92cc00e040 con 0x7f92d0055920 2024-03-06T07:28:10.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.639+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] -- osd_op(unknown.0.0:50 17.23 17:c4000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92d0055920 2024-03-06T07:28:10.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.639+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 <== osd.4 v2:172.21.15.183:6816/3256618830 7 ==== osd_op_reply(50 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92cc00e040 con 0x7f92d0055920 2024-03-06T07:28:10.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.639+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] -- osd_op(unknown.0.0:51 17.13 17:c8000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92c8008910 2024-03-06T07:28:10.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.639+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 <== osd.3 v2:172.21.15.83:6815/2656563756 9 ==== osd_op_reply(51 [pgnls start_epoch 116 out=49b] v110'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92cc00e040 con 0x7f92c8008910 2024-03-06T07:28:10.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.639+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] -- osd_op(unknown.0.0:52 17.33 17:cc000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x55d306267a00 2024-03-06T07:28:10.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.640+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 <== osd.7 v2:172.21.15.183:6820/3336566695 11 ==== osd_op_reply(52 [pgnls start_epoch 116 out=49b] v110'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92d005d040 con 0x55d306267a00 2024-03-06T07:28:10.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.640+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] -- osd_op(unknown.0.0:53 17.b 17:d0000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92cc05f780 2024-03-06T07:28:10.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.640+0000 7f92d68ad640 1 -- 172.21.15.83:0/1482419184 <== osd.0 v2:172.21.15.83:6808/1635427425 9 ==== osd_op_reply(53 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92c80168c0 con 0x7f92cc05f780 2024-03-06T07:28:10.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.640+0000 7f92d68ad640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] -- osd_op(unknown.0.0:54 17.2b 17:d4000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92d0055920 2024-03-06T07:28:10.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.640+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 <== osd.4 v2:172.21.15.183:6816/3256618830 8 ==== osd_op_reply(54 [pgnls start_epoch 116 out=49b] v110'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92cc00e040 con 0x7f92d0055920 2024-03-06T07:28:10.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.640+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] -- osd_op(unknown.0.0:55 17.1b 17:d8000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92d0055920 2024-03-06T07:28:10.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.641+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 <== osd.4 v2:172.21.15.183:6816/3256618830 9 ==== osd_op_reply(55 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92cc00e040 con 0x7f92d0055920 2024-03-06T07:28:10.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.641+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.83:6800/2591342133,v1:172.21.15.83:6801/2591342133] -- osd_op(unknown.0.0:56 17.3b 17:dc000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92d0088400 2024-03-06T07:28:10.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.641+0000 7f92d68ad640 1 -- 172.21.15.83:0/1482419184 <== osd.2 v2:172.21.15.83:6800/2591342133 3 ==== osd_op_reply(56 [pgnls start_epoch 116 out=49b] v110'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92c80168c0 con 0x7f92d0088400 2024-03-06T07:28:10.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.641+0000 7f92d68ad640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] -- osd_op(unknown.0.0:57 17.7 17:e0000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92cc05f780 2024-03-06T07:28:10.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.641+0000 7f92d68ad640 1 -- 172.21.15.83:0/1482419184 <== osd.0 v2:172.21.15.83:6808/1635427425 10 ==== osd_op_reply(57 [pgnls start_epoch 116 out=81b] v110'8 uv3 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x7f92c80168c0 con 0x7f92cc05f780 2024-03-06T07:28:10.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.641+0000 7f92d68ad640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] -- osd_op(unknown.0.0:58 17.27 17:e4000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92d0055920 2024-03-06T07:28:10.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.642+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 <== osd.4 v2:172.21.15.183:6816/3256618830 10 ==== osd_op_reply(58 [pgnls start_epoch 116 out=49b] v110'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92cc00e040 con 0x7f92d0055920 2024-03-06T07:28:10.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.642+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.183:6800/2649185225,v1:172.21.15.183:6801/2649185225] -- osd_op(unknown.0.0:59 17.17 17:e8000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92c8027960 2024-03-06T07:28:10.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.642+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 <== osd.6 v2:172.21.15.183:6800/2649185225 4 ==== osd_op_reply(59 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92d005d040 con 0x7f92c8027960 2024-03-06T07:28:10.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.642+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] -- osd_op(unknown.0.0:60 17.37 17:ec000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92cc05f780 2024-03-06T07:28:10.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.642+0000 7f92d68ad640 1 -- 172.21.15.83:0/1482419184 <== osd.0 v2:172.21.15.83:6808/1635427425 11 ==== osd_op_reply(60 [pgnls start_epoch 116 out=49b] v110'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92c80168c0 con 0x7f92cc05f780 2024-03-06T07:28:10.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.642+0000 7f92d68ad640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.83:6814/1822925240,v1:172.21.15.83:6817/1822925240] -- osd_op(unknown.0.0:61 17.f 17:f0000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92d0055030 2024-03-06T07:28:10.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.643+0000 7f92d68ad640 1 -- 172.21.15.83:0/1482419184 <== osd.1 v2:172.21.15.83:6814/1822925240 7 ==== osd_op_reply(61 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92c80168c0 con 0x7f92d0055030 2024-03-06T07:28:10.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.643+0000 7f92d68ad640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] -- osd_op(unknown.0.0:62 17.2f 17:f4000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92d0055920 2024-03-06T07:28:10.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.643+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 <== osd.4 v2:172.21.15.183:6816/3256618830 11 ==== osd_op_reply(62 [pgnls start_epoch 116 out=81b] v110'9 uv1 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x7f92cc00e040 con 0x7f92d0055920 2024-03-06T07:28:10.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.643+0000 7f92d60ac640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.183:6800/2649185225,v1:172.21.15.183:6801/2649185225] -- osd_op(unknown.0.0:63 17.1f 17:f8000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92c8027960 2024-03-06T07:28:10.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.643+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 <== osd.6 v2:172.21.15.183:6800/2649185225 5 ==== osd_op_reply(63 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92d005d040 con 0x7f92c8027960 2024-03-06T07:28:10.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.643+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 --> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] -- osd_op(unknown.0.0:64 17.3f 17:fc000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55d30626a0c0 con 0x7f92cc05f780 2024-03-06T07:28:10.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.644+0000 7f92d68ad640 1 -- 172.21.15.83:0/1482419184 <== osd.0 v2:172.21.15.83:6808/1635427425 12 ==== osd_op_reply(64 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92c80168c0 con 0x7f92cc05f780 2024-03-06T07:28:10.651 INFO:teuthology.orchestra.run.smithi083.stdout:10000000001.00000000 2024-03-06T07:28:10.651 INFO:teuthology.orchestra.run.smithi083.stdout:10000000001.00000001 2024-03-06T07:28:10.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.644+0000 7f92d9648f40 1 -- 172.21.15.83:0/1482419184 >> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] conn(0x7f92cc05f780 msgr2=0x7f92cc058830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.644+0000 7f92d9648f40 1 --2- 172.21.15.83:0/1482419184 >> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] conn(0x7f92cc05f780 0x7f92cc058830 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.644+0000 7f92d9648f40 1 -- 172.21.15.83:0/1482419184 >> [v2:172.21.15.83:6814/1822925240,v1:172.21.15.83:6817/1822925240] conn(0x7f92d0055030 msgr2=0x7f92d006a410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.644+0000 7f92d9648f40 1 --2- 172.21.15.83:0/1482419184 >> [v2:172.21.15.83:6814/1822925240,v1:172.21.15.83:6817/1822925240] conn(0x7f92d0055030 0x7f92d006a410 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.644+0000 7f92d9648f40 1 -- 172.21.15.83:0/1482419184 >> [v2:172.21.15.83:6800/2591342133,v1:172.21.15.83:6801/2591342133] conn(0x7f92d0088400 msgr2=0x7f92d0088840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.644+0000 7f92d9648f40 1 --2- 172.21.15.83:0/1482419184 >> [v2:172.21.15.83:6800/2591342133,v1:172.21.15.83:6801/2591342133] conn(0x7f92d0088400 0x7f92d0088840 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.644+0000 7f92d9648f40 1 -- 172.21.15.83:0/1482419184 >> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] conn(0x7f92c8008910 msgr2=0x7f92c80173f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.644+0000 7f92d9648f40 1 --2- 172.21.15.83:0/1482419184 >> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] conn(0x7f92c8008910 0x7f92c80173f0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.644+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 reap_dead start 2024-03-06T07:28:10.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.644+0000 7f92d9648f40 1 -- 172.21.15.83:0/1482419184 >> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] conn(0x7f92d0055920 msgr2=0x7f92d00783f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.644+0000 7f92d9648f40 1 --2- 172.21.15.83:0/1482419184 >> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] conn(0x7f92d0055920 0x7f92d00783f0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.644+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 reap_dead start 2024-03-06T07:28:10.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.644+0000 7f92d9648f40 1 -- 172.21.15.83:0/1482419184 >> [v2:172.21.15.183:6808/2957306877,v1:172.21.15.183:6809/2957306877] conn(0x7f92cc057810 msgr2=0x7f92cc056470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.644+0000 7f92d9648f40 1 --2- 172.21.15.83:0/1482419184 >> [v2:172.21.15.183:6808/2957306877,v1:172.21.15.183:6809/2957306877] conn(0x7f92cc057810 0x7f92cc056470 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.644+0000 7f92d9648f40 1 -- 172.21.15.83:0/1482419184 >> [v2:172.21.15.183:6800/2649185225,v1:172.21.15.183:6801/2649185225] conn(0x7f92c8027960 msgr2=0x7f92c8025430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.644+0000 7f92d9648f40 1 --2- 172.21.15.83:0/1482419184 >> [v2:172.21.15.183:6800/2649185225,v1:172.21.15.183:6801/2649185225] conn(0x7f92c8027960 0x7f92c8025430 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.644+0000 7f92d9648f40 1 -- 172.21.15.83:0/1482419184 >> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] conn(0x55d306267a00 msgr2=0x55d306269b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.644+0000 7f92d9648f40 1 --2- 172.21.15.83:0/1482419184 >> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] conn(0x55d306267a00 0x55d306269b80 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.644+0000 7f92d9648f40 1 -- 172.21.15.83:0/1482419184 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f92c4069ee0 msgr2=0x7f92c406c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.644+0000 7f92d9648f40 1 --2- 172.21.15.83:0/1482419184 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f92c4069ee0 0x7f92c406c3a0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x55d3061c46b0 tx=0x7f92c80053c0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.644+0000 7f92d9648f40 1 -- 172.21.15.83:0/1482419184 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d30620cf70 msgr2=0x55d3063340d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.644+0000 7f92d70ae640 1 -- 172.21.15.83:0/1482419184 reap_dead start 2024-03-06T07:28:10.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.644+0000 7f92d9648f40 1 --2- 172.21.15.83:0/1482419184 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d30620cf70 0x55d3063340d0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f92cc00df70 tx=0x7f92cc00b5b0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.644+0000 7f92d9648f40 1 -- 172.21.15.83:0/1482419184 shutdown_connections 2024-03-06T07:28:10.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.644+0000 7f92d9648f40 1 --2- 172.21.15.83:0/1482419184 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d30620cf70 0x55d3063340d0 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.644+0000 7f92d9648f40 1 -- 172.21.15.83:0/1482419184 >> 172.21.15.83:0/1482419184 conn(0x55d3061ef340 msgr2=0x55d30627fa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:10.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.644+0000 7f92d9648f40 1 -- 172.21.15.83:0/1482419184 shutdown_connections 2024-03-06T07:28:10.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.644+0000 7f92d9648f40 1 -- 172.21.15.83:0/1482419184 wait complete. 2024-03-06T07:28:10.655 DEBUG:tasks.cephfs.filesystem:All objects for ino 1099511627777 size 8388608 found 2024-03-06T07:28:10.655 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:10.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.791+0000 7f9589484640 1 Processor -- start 2024-03-06T07:28:10.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.791+0000 7f9589484640 1 -- start start 2024-03-06T07:28:10.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.792+0000 7f9589484640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f95840d0a80 0x7f95840b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.792+0000 7f9589484640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95840d25d0 con 0x7f95840d4760 2024-03-06T07:28:10.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.792+0000 7f9589484640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95840d2870 con 0x7f95840b4170 2024-03-06T07:28:10.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.792+0000 7f9589484640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95840d4bf0 con 0x7f95840d0a80 2024-03-06T07:28:10.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.792+0000 7f95827fc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f95840d4760 0x7f95840cee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:54788/0 (socket says 172.21.15.83:54788) 2024-03-06T07:28:10.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.792+0000 7f95827fc640 1 -- 172.21.15.83:0/3415066061 learned_addr learned my addr 172.21.15.83:0/3415066061 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:28:10.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.792+0000 7f9581ffb640 1 -- 172.21.15.83:0/3415066061 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2901411372 0 0) 0x7f95840d25d0 con 0x7f95840d4760 2024-03-06T07:28:10.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.792+0000 7f95837fe640 1 --2- 172.21.15.83:0/3415066061 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f95840d0a80 0x7f95840b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.792+0000 7f9581ffb640 1 -- 172.21.15.83:0/3415066061 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f956c003520 con 0x7f95840d4760 2024-03-06T07:28:10.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.792+0000 7f9581ffb640 1 -- 172.21.15.83:0/3415066061 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 504590749 0 0) 0x7f956c003520 con 0x7f95840d4760 2024-03-06T07:28:10.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.792+0000 7f9581ffb640 1 -- 172.21.15.83:0/3415066061 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f95840d0a80 msgr2=0x7f95840b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:10.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.792+0000 7f9581ffb640 1 --2- 172.21.15.83:0/3415066061 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f95840d0a80 0x7f95840b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.792+0000 7f9581ffb640 1 -- 172.21.15.83:0/3415066061 >> v1:172.21.15.183:6789/0 conn(0x7f95840b4170 legacy=0x7f95840d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.792+0000 7f9581ffb640 1 -- 172.21.15.83:0/3415066061 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95840d2b50 con 0x7f95840d4760 2024-03-06T07:28:10.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.792+0000 7f9581ffb640 1 -- 172.21.15.83:0/3415066061 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f95780019f0 con 0x7f95840d4760 2024-03-06T07:28:10.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.792+0000 7f9581ffb640 1 -- 172.21.15.83:0/3415066061 >> v1:172.21.15.83:6789/0 conn(0x7f95840d4760 legacy=0x7f95840cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.793+0000 7f9581ffb640 1 --2- 172.21.15.83:0/3415066061 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f956c0039e0 0x7f956c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.793+0000 7f9581ffb640 1 --2- 172.21.15.83:0/3415066061 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f956c004800 0x7f956c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.793+0000 7f9581ffb640 1 --2- 172.21.15.83:0/3415066061 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f95840d0a80 0x7f956c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.793+0000 7f9581ffb640 1 -- 172.21.15.83:0/3415066061 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95840d4bf0 con 0x7f956c0039e0 2024-03-06T07:28:10.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.793+0000 7f9581ffb640 1 -- 172.21.15.83:0/3415066061 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956c00baa0 con 0x7f956c004800 2024-03-06T07:28:10.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.793+0000 7f9581ffb640 1 -- 172.21.15.83:0/3415066061 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f956c00bca0 con 0x7f95840d0a80 2024-03-06T07:28:10.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.793+0000 7f9582ffd640 1 --2- 172.21.15.83:0/3415066061 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f956c0039e0 0x7f956c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.793+0000 7f95827fc640 1 --2- 172.21.15.83:0/3415066061 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f956c004800 0x7f956c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.793+0000 7f95837fe640 1 --2- 172.21.15.83:0/3415066061 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f95840d0a80 0x7f956c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.793+0000 7f9582ffd640 1 -- 172.21.15.83:0/3415066061 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f95840d0a80 msgr2=0x7f956c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:10.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.793+0000 7f9582ffd640 1 --2- 172.21.15.83:0/3415066061 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f95840d0a80 0x7f956c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.793+0000 7f9582ffd640 1 -- 172.21.15.83:0/3415066061 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f956c004800 msgr2=0x7f956c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:10.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.793+0000 7f9582ffd640 1 --2- 172.21.15.83:0/3415066061 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f956c004800 0x7f956c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.793+0000 7f9582ffd640 1 -- 172.21.15.83:0/3415066061 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f956c003520 con 0x7f956c0039e0 2024-03-06T07:28:10.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.794+0000 7f9582ffd640 1 --2- 172.21.15.83:0/3415066061 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f956c0039e0 0x7f956c003e00 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f957c001750 tx=0x7f956c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dced6663582366c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:10.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.794+0000 7f9581ffb640 1 -- 172.21.15.83:0/3415066061 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957c001370 con 0x7f956c0039e0 2024-03-06T07:28:10.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.794+0000 7f9581ffb640 1 -- 172.21.15.83:0/3415066061 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f957c0052d0 con 0x7f956c0039e0 2024-03-06T07:28:10.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.794+0000 7f9581ffb640 1 -- 172.21.15.83:0/3415066061 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957c003830 con 0x7f956c0039e0 2024-03-06T07:28:10.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.795+0000 7f9589484640 1 -- 172.21.15.83:0/3415066061 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f956c0039e0 msgr2=0x7f956c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.795+0000 7f9589484640 1 --2- 172.21.15.83:0/3415066061 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f956c0039e0 0x7f956c003e00 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f957c001750 tx=0x7f956c005830 comp rx=0 tx=0).stop 2024-03-06T07:28:10.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.795+0000 7f95837fe640 1 -- 172.21.15.83:0/3415066061 reap_dead start 2024-03-06T07:28:10.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.795+0000 7f9589484640 1 -- 172.21.15.83:0/3415066061 shutdown_connections 2024-03-06T07:28:10.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.795+0000 7f9589484640 1 -- 172.21.15.83:0/3415066061 >> 172.21.15.83:0/3415066061 conn(0x7f95840d3c40 msgr2=0x7f95840d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:10.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.795+0000 7f9589484640 1 -- 172.21.15.83:0/3415066061 shutdown_connections 2024-03-06T07:28:10.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.795+0000 7f9589484640 1 -- 172.21.15.83:0/3415066061 wait complete. 2024-03-06T07:28:10.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.797+0000 7f9589484640 1 Processor -- start 2024-03-06T07:28:10.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.797+0000 7f9589484640 1 -- start start 2024-03-06T07:28:10.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.798+0000 7f9589484640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f956c0039e0 0x7f95840b7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.798+0000 7f9589484640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f95840b7560 0x7f95841f3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.798+0000 7f9589484640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f95841f3d90 0x7f95841f4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.798+0000 7f9589484640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95840b4dd0 con 0x7f95841f3d90 2024-03-06T07:28:10.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.798+0000 7f9589484640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95840ce9e0 con 0x7f95840b7560 2024-03-06T07:28:10.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.798+0000 7f9589484640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95840b4f50 con 0x7f956c0039e0 2024-03-06T07:28:10.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.798+0000 7f95837fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f95841f3d90 0x7f95841f4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.798+0000 7f95837fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f95841f3d90 0x7f95841f4210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51206/0 (socket says 172.21.15.83:51206) 2024-03-06T07:28:10.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.798+0000 7f95837fe640 1 -- 172.21.15.83:0/2158539251 learned_addr learned my addr 172.21.15.83:0/2158539251 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:10.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.798+0000 7f9582ffd640 1 --2- 172.21.15.83:0/2158539251 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f956c0039e0 0x7f95840b7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.798+0000 7f95837fe640 1 -- 172.21.15.83:0/2158539251 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f956c0039e0 msgr2=0x7f95840b7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.798+0000 7f95827fc640 1 --2- 172.21.15.83:0/2158539251 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f95840b7560 0x7f95841f3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.798+0000 7f95837fe640 1 --2- 172.21.15.83:0/2158539251 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f956c0039e0 0x7f95840b7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.798+0000 7f95837fe640 1 -- 172.21.15.83:0/2158539251 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f95840b7560 msgr2=0x7f95841f3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.798+0000 7f95837fe640 1 --2- 172.21.15.83:0/2158539251 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f95840b7560 0x7f95841f3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.798+0000 7f95837fe640 1 -- 172.21.15.83:0/2158539251 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95840b7d90 con 0x7f95841f3d90 2024-03-06T07:28:10.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.798+0000 7f9582ffd640 1 --2- 172.21.15.83:0/2158539251 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f956c0039e0 0x7f95840b7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:10.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.798+0000 7f95827fc640 1 --2- 172.21.15.83:0/2158539251 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f95840b7560 0x7f95841f3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:10.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.799+0000 7f95837fe640 1 --2- 172.21.15.83:0/2158539251 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f95841f3d90 0x7f95841f4210 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f95840b67e0 tx=0x7f9574011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:10.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.799+0000 7f956b7fe640 1 -- 172.21.15.83:0/2158539251 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9574004240 con 0x7f95841f3d90 2024-03-06T07:28:10.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.799+0000 7f956b7fe640 1 -- 172.21.15.83:0/2158539251 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f957401b070 con 0x7f95841f3d90 2024-03-06T07:28:10.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.799+0000 7f9589484640 1 -- 172.21.15.83:0/2158539251 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95840b8070 con 0x7f95841f3d90 2024-03-06T07:28:10.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.799+0000 7f956b7fe640 1 -- 172.21.15.83:0/2158539251 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957400f7a0 con 0x7f95841f3d90 2024-03-06T07:28:10.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.799+0000 7f9589484640 1 -- 172.21.15.83:0/2158539251 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95840b8630 con 0x7f95841f3d90 2024-03-06T07:28:10.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.800+0000 7f956b7fe640 1 -- 172.21.15.83:0/2158539251 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9574012040 con 0x7f95841f3d90 2024-03-06T07:28:10.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.800+0000 7f9589484640 1 -- 172.21.15.83:0/2158539251 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95840d4580 con 0x7f95841f3d90 2024-03-06T07:28:10.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.801+0000 7f956b7fe640 1 --2- 172.21.15.83:0/2158539251 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f957006b500 0x7f957006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.801+0000 7f9582ffd640 1 --2- 172.21.15.83:0/2158539251 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f957006b500 0x7f957006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.801+0000 7f956b7fe640 1 -- 172.21.15.83:0/2158539251 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f957408a020 con 0x7f95841f3d90 2024-03-06T07:28:10.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.801+0000 7f9582ffd640 1 --2- 172.21.15.83:0/2158539251 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f957006b500 0x7f957006d9c0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f957c033000 tx=0x7f957c009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:10.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:10.805+0000 7f956b7fe640 1 -- 172.21.15.83:0/2158539251 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f95840d4580 con 0x7f95841f3d90 2024-03-06T07:28:11.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.035+0000 7f9589484640 1 -- 172.21.15.83:0/2158539251 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f958404f330 con 0x7f95841f3d90 2024-03-06T07:28:11.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.035+0000 7f956b7fe640 1 -- 172.21.15.83:0/2158539251 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f9574058db0 con 0x7f95841f3d90 2024-03-06T07:28:11.038 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:28:11.038 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7051,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/101642171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":101642171},{"type":"v1","addr":"172.21.15.183:6833","nonce":101642171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7057,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3146884900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":3146884900},{"type":"v1","addr":"172.21.15.183:6835","nonce":3146884900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8208,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1922041621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1922041621},{"type":"v1","addr":"172.21.15.83:6835","nonce":1922041621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.491022+0000","modified":"2024-03-06T07:27:37.006649+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8229":{"gid":8229,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/2003687153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2003687153},{"type":"v1","addr":"172.21.15.83:6837","nonce":2003687153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8229,"qdb_cluster":[8229]},"id":8}]} 2024-03-06T07:28:11.038 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 66 2024-03-06T07:28:11.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.037+0000 7f9589484640 1 -- 172.21.15.83:0/2158539251 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f957006b500 msgr2=0x7f957006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.037+0000 7f9589484640 1 --2- 172.21.15.83:0/2158539251 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f957006b500 0x7f957006d9c0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f957c033000 tx=0x7f957c009030 comp rx=0 tx=0).stop 2024-03-06T07:28:11.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.037+0000 7f9589484640 1 -- 172.21.15.83:0/2158539251 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f95841f3d90 msgr2=0x7f95841f4210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.037+0000 7f9589484640 1 --2- 172.21.15.83:0/2158539251 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f95841f3d90 0x7f95841f4210 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f95840b67e0 tx=0x7f9574011cf0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.037+0000 7f9589484640 1 -- 172.21.15.83:0/2158539251 shutdown_connections 2024-03-06T07:28:11.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.037+0000 7f9589484640 1 --2- 172.21.15.83:0/2158539251 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f957006b500 0x7f957006d9c0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.037+0000 7f9589484640 1 --2- 172.21.15.83:0/2158539251 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f95841f3d90 0x7f95841f4210 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.037+0000 7f9589484640 1 --2- 172.21.15.83:0/2158539251 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f95840b7560 0x7f95841f3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.037+0000 7f9589484640 1 --2- 172.21.15.83:0/2158539251 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f956c0039e0 0x7f95840b7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.037+0000 7f9589484640 1 -- 172.21.15.83:0/2158539251 >> 172.21.15.83:0/2158539251 conn(0x7f95840d3c40 msgr2=0x7f95840ced70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:11.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.037+0000 7f9589484640 1 -- 172.21.15.83:0/2158539251 shutdown_connections 2024-03-06T07:28:11.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.038+0000 7f9589484640 1 -- 172.21.15.83:0/2158539251 wait complete. 2024-03-06T07:28:11.051 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:28:11.168 INFO:teuthology.orchestra.run.smithi083.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-06T07:28:11.168 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-06T07:28:11.168 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:11.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.268+0000 7f1ebdfb6640 1 Processor -- start 2024-03-06T07:28:11.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.268+0000 7f1ebdfb6640 1 -- start start 2024-03-06T07:28:11.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.269+0000 7f1ebdfb6640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1eb813dc10 0x7f1eb80b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.269+0000 7f1ebdfb6640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1eb815bf30 0x7f1eb80b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.269+0000 7f1ebdfb6640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb815f580 con 0x7f1eb815e850 2024-03-06T07:28:11.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.269+0000 7f1ebdfb6640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eb80cf3f0 con 0x7f1eb815bf30 2024-03-06T07:28:11.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.269+0000 7f1ebdfb6640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb815b210 con 0x7f1eb813dc10 2024-03-06T07:28:11.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.269+0000 7f1eaffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1eb815bf30 0x7f1eb80b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.269+0000 7f1eaffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1eb815bf30 0x7f1eb80b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51220/0 (socket says 172.21.15.83:51220) 2024-03-06T07:28:11.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.269+0000 7f1eaffff640 1 -- 172.21.15.83:0/158969503 learned_addr learned my addr 172.21.15.83:0/158969503 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:11.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.269+0000 7f1eb77fe640 1 --2- 172.21.15.83:0/158969503 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1eb813dc10 0x7f1eb80b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.269+0000 7f1eaffff640 1 -- 172.21.15.83:0/158969503 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1eb813dc10 msgr2=0x7f1eb80b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:11.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.269+0000 7f1eaffff640 1 --2- 172.21.15.83:0/158969503 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1eb813dc10 0x7f1eb80b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.269+0000 7f1eaffff640 1 -- 172.21.15.83:0/158969503 >> v1:172.21.15.183:6789/0 conn(0x7f1eb815e850 legacy=0x7f1eb815eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.269+0000 7f1eaffff640 1 -- 172.21.15.83:0/158969503 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb815b450 con 0x7f1eb815bf30 2024-03-06T07:28:11.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.270+0000 7f1eaffff640 1 --2- 172.21.15.83:0/158969503 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1eb815bf30 0x7f1eb80b2e90 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f1ea0009870 tx=0x7f1ea002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffae627124cc61b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:11.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.270+0000 7f1eb6ffd640 1 -- 172.21.15.83:0/158969503 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea0035020 con 0x7f1eb815bf30 2024-03-06T07:28:11.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.270+0000 7f1eb6ffd640 1 -- 172.21.15.83:0/158969503 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ea0031640 con 0x7f1eb815bf30 2024-03-06T07:28:11.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.270+0000 7f1eb6ffd640 1 -- 172.21.15.83:0/158969503 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea002d040 con 0x7f1eb815bf30 2024-03-06T07:28:11.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.271+0000 7f1ebdfb6640 1 -- 172.21.15.83:0/158969503 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1eb815bf30 msgr2=0x7f1eb80b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.271+0000 7f1ebdfb6640 1 --2- 172.21.15.83:0/158969503 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1eb815bf30 0x7f1eb80b2e90 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f1ea0009870 tx=0x7f1ea002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.271+0000 7f1ebdfb6640 1 -- 172.21.15.83:0/158969503 shutdown_connections 2024-03-06T07:28:11.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.271+0000 7f1ebdfb6640 1 --2- 172.21.15.83:0/158969503 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1eb815bf30 0x7f1eb80b2e90 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.271+0000 7f1ebdfb6640 1 --2- 172.21.15.83:0/158969503 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1eb813dc10 0x7f1eb80b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.271+0000 7f1ebdfb6640 1 -- 172.21.15.83:0/158969503 >> 172.21.15.83:0/158969503 conn(0x7f1eb80cf8a0 msgr2=0x7f1eb80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:11.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.271+0000 7f1ebdfb6640 1 -- 172.21.15.83:0/158969503 shutdown_connections 2024-03-06T07:28:11.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.271+0000 7f1ebdfb6640 1 -- 172.21.15.83:0/158969503 wait complete. 2024-03-06T07:28:11.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.272+0000 7f1ebdfb6640 1 Processor -- start 2024-03-06T07:28:11.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.272+0000 7f1ebdfb6640 1 -- start start 2024-03-06T07:28:11.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.273+0000 7f1ebdfb6640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1eb813dc10 0x7f1eb81517e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.273+0000 7f1ebdfb6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1eb815bf30 0x7f1eb8071ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.273+0000 7f1ebdfb6640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1eb815e850 0x7f1eb8072430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.273+0000 7f1ebdfb6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb80cd7a0 con 0x7f1eb815bf30 2024-03-06T07:28:11.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.273+0000 7f1ebdfb6640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb813dff0 con 0x7f1eb815e850 2024-03-06T07:28:11.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.273+0000 7f1ebdfb6640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb80cd920 con 0x7f1eb813dc10 2024-03-06T07:28:11.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.273+0000 7f1eaffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1eb815bf30 0x7f1eb8071ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.273+0000 7f1eaffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1eb815bf30 0x7f1eb8071ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51224/0 (socket says 172.21.15.83:51224) 2024-03-06T07:28:11.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.273+0000 7f1eaffff640 1 -- 172.21.15.83:0/1622365257 learned_addr learned my addr 172.21.15.83:0/1622365257 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:11.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.274+0000 7f1eb77fe640 1 --2- 172.21.15.83:0/1622365257 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1eb813dc10 0x7f1eb81517e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.274+0000 7f1eb7fff640 1 --2- 172.21.15.83:0/1622365257 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1eb815e850 0x7f1eb8072430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.274+0000 7f1eaffff640 1 -- 172.21.15.83:0/1622365257 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1eb813dc10 msgr2=0x7f1eb81517e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.274+0000 7f1eaffff640 1 --2- 172.21.15.83:0/1622365257 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1eb813dc10 0x7f1eb81517e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.274+0000 7f1eaffff640 1 -- 172.21.15.83:0/1622365257 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1eb815e850 msgr2=0x7f1eb8072430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.274+0000 7f1eaffff640 1 --2- 172.21.15.83:0/1622365257 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1eb815e850 0x7f1eb8072430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.274+0000 7f1eaffff640 1 -- 172.21.15.83:0/1622365257 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb813f8b0 con 0x7f1eb815bf30 2024-03-06T07:28:11.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.274+0000 7f1eaffff640 1 --2- 172.21.15.83:0/1622365257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1eb815bf30 0x7f1eb8071ef0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f1ea002b930 tx=0x7f1ea00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:11.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.274+0000 7f1eb4ff9640 1 -- 172.21.15.83:0/1622365257 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea0040070 con 0x7f1eb815bf30 2024-03-06T07:28:11.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.274+0000 7f1eb4ff9640 1 -- 172.21.15.83:0/1622365257 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ea003e070 con 0x7f1eb815bf30 2024-03-06T07:28:11.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.274+0000 7f1eb4ff9640 1 -- 172.21.15.83:0/1622365257 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea0035040 con 0x7f1eb815bf30 2024-03-06T07:28:11.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.274+0000 7f1ebdfb6640 1 -- 172.21.15.83:0/1622365257 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eb813fb90 con 0x7f1eb815bf30 2024-03-06T07:28:11.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.274+0000 7f1ebdfb6640 1 -- 172.21.15.83:0/1622365257 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eb8140150 con 0x7f1eb815bf30 2024-03-06T07:28:11.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.275+0000 7f1ebdfb6640 1 -- 172.21.15.83:0/1622365257 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eb804f330 con 0x7f1eb815bf30 2024-03-06T07:28:11.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.276+0000 7f1eb4ff9640 1 -- 172.21.15.83:0/1622365257 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1ea0038020 con 0x7f1eb815bf30 2024-03-06T07:28:11.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.276+0000 7f1eb4ff9640 1 --2- 172.21.15.83:0/1622365257 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1e9806b500 0x7f1e9806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.277+0000 7f1eb4ff9640 1 -- 172.21.15.83:0/1622365257 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f1ea00abba0 con 0x7f1eb815bf30 2024-03-06T07:28:11.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.277+0000 7f1eb77fe640 1 --2- 172.21.15.83:0/1622365257 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1e9806b500 0x7f1e9806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.277+0000 7f1eb77fe640 1 --2- 172.21.15.83:0/1622365257 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1e9806b500 0x7f1e9806d9c0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f1ea8009b40 tx=0x7f1ea8009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:11.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.281+0000 7f1eb4ff9640 1 -- 172.21.15.83:0/1622365257 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1ea007a250 con 0x7f1eb815bf30 2024-03-06T07:28:11.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.495+0000 7f1ebdfb6640 1 -- 172.21.15.83:0/1622365257 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1eb806eb00 con 0x7f1eb815bf30 2024-03-06T07:28:11.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.495+0000 7f1eb4ff9640 1 -- 172.21.15.83:0/1622365257 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f1ea007e2f0 con 0x7f1eb815bf30 2024-03-06T07:28:11.497 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:28:11.497 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7051,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/101642171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":101642171},{"type":"v1","addr":"172.21.15.183:6833","nonce":101642171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7057,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3146884900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":3146884900},{"type":"v1","addr":"172.21.15.183:6835","nonce":3146884900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8208,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1922041621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1922041621},{"type":"v1","addr":"172.21.15.83:6835","nonce":1922041621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.491022+0000","modified":"2024-03-06T07:27:37.006649+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8229":{"gid":8229,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/2003687153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2003687153},{"type":"v1","addr":"172.21.15.83:6837","nonce":2003687153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8229,"qdb_cluster":[8229]},"id":8}]} 2024-03-06T07:28:11.497 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 66 2024-03-06T07:28:11.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.497+0000 7f1ebdfb6640 1 -- 172.21.15.83:0/1622365257 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1e9806b500 msgr2=0x7f1e9806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.497+0000 7f1ebdfb6640 1 --2- 172.21.15.83:0/1622365257 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1e9806b500 0x7f1e9806d9c0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f1ea8009b40 tx=0x7f1ea8009190 comp rx=0 tx=0).stop 2024-03-06T07:28:11.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.497+0000 7f1ebdfb6640 1 -- 172.21.15.83:0/1622365257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1eb815bf30 msgr2=0x7f1eb8071ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.497+0000 7f1ebdfb6640 1 --2- 172.21.15.83:0/1622365257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1eb815bf30 0x7f1eb8071ef0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f1ea002b930 tx=0x7f1ea00046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.497+0000 7f1ebdfb6640 1 -- 172.21.15.83:0/1622365257 shutdown_connections 2024-03-06T07:28:11.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.497+0000 7f1ebdfb6640 1 --2- 172.21.15.83:0/1622365257 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1e9806b500 0x7f1e9806d9c0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.497+0000 7f1ebdfb6640 1 --2- 172.21.15.83:0/1622365257 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1eb815e850 0x7f1eb8072430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.497+0000 7f1ebdfb6640 1 --2- 172.21.15.83:0/1622365257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1eb815bf30 0x7f1eb8071ef0 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.497+0000 7f1ebdfb6640 1 --2- 172.21.15.83:0/1622365257 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1eb813dc10 0x7f1eb81517e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.497+0000 7f1ebdfb6640 1 -- 172.21.15.83:0/1622365257 >> 172.21.15.83:0/1622365257 conn(0x7f1eb80cf8a0 msgr2=0x7f1eb80b2740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:11.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.497+0000 7f1ebdfb6640 1 -- 172.21.15.83:0/1622365257 shutdown_connections 2024-03-06T07:28:11.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.497+0000 7f1ebdfb6640 1 -- 172.21.15.83:0/1622365257 wait complete. 2024-03-06T07:28:11.510 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:28:11.637 INFO:teuthology.orchestra.run.smithi083.stdout:{"message":"","return_code":0} 2024-03-06T07:28:11.638 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:28:11.638 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-06T07:28:11.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.678+0000 7fa4a7c3bf40 1 Processor -- start 2024-03-06T07:28:11.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.678+0000 7fa4a7c3bf40 1 -- start start 2024-03-06T07:28:11.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.678+0000 7fa4a7c3bf40 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bf585b41a0 0x55bf585ac410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.678+0000 7fa4a7c3bf40 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bf585aa5a0 0x55bf585b4580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.678+0000 7fa4a7c3bf40 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55bf58510c30 0x55bf58511030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.679+0000 7fa4a7c3bf40 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf58528b80 con 0x55bf585b41a0 2024-03-06T07:28:11.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.679+0000 7fa4a7c3bf40 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf58488b70 con 0x55bf585aa5a0 2024-03-06T07:28:11.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.679+0000 7fa4a7c3bf40 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bf585b8210 con 0x55bf58510c30 2024-03-06T07:28:11.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.679+0000 7fa4a4f06640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bf585b41a0 0x55bf585ac410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.679+0000 7fa4a4f06640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bf585b41a0 0x55bf585ac410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51248/0 (socket says 172.21.15.83:51248) 2024-03-06T07:28:11.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.679+0000 7fa4a4f06640 1 -- 172.21.15.83:0/514358398 learned_addr learned my addr 172.21.15.83:0/514358398 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:11.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.679+0000 7fa49ffff640 1 --2- 172.21.15.83:0/514358398 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bf585aa5a0 0x55bf585b4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.679+0000 7fa4a5707640 1 --2- 172.21.15.83:0/514358398 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55bf58510c30 0x55bf58511030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.679+0000 7fa4a4f06640 1 -- 172.21.15.83:0/514358398 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55bf58510c30 msgr2=0x55bf58511030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:11.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.679+0000 7fa4a4f06640 1 --2- 172.21.15.83:0/514358398 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55bf58510c30 0x55bf58511030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.679+0000 7fa4a4f06640 1 -- 172.21.15.83:0/514358398 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bf585aa5a0 msgr2=0x55bf585b4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:11.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.679+0000 7fa4a4f06640 1 --2- 172.21.15.83:0/514358398 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bf585aa5a0 0x55bf585b4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.679+0000 7fa4a4f06640 1 -- 172.21.15.83:0/514358398 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bf585ab000 con 0x55bf585b41a0 2024-03-06T07:28:11.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.679+0000 7fa4a4f06640 1 --2- 172.21.15.83:0/514358398 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bf585b41a0 0x55bf585ac410 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7fa498009730 tx=0x7fa498005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b462ba8b2a01d366 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:11.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.679+0000 7fa49effd640 1 -- 172.21.15.83:0/514358398 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4980078f0 con 0x55bf585b41a0 2024-03-06T07:28:11.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.679+0000 7fa49effd640 1 -- 172.21.15.83:0/514358398 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa498007a90 con 0x55bf585b41a0 2024-03-06T07:28:11.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.679+0000 7fa49effd640 1 -- 172.21.15.83:0/514358398 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa498007e40 con 0x55bf585b41a0 2024-03-06T07:28:11.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.680+0000 7fa4a7c3bf40 1 -- 172.21.15.83:0/514358398 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bf585b41a0 msgr2=0x55bf585ac410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.680+0000 7fa4a7c3bf40 1 --2- 172.21.15.83:0/514358398 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bf585b41a0 0x55bf585ac410 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7fa498009730 tx=0x7fa498005a00 comp rx=0 tx=0).stop 2024-03-06T07:28:11.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.680+0000 7fa4a7c3bf40 1 -- 172.21.15.83:0/514358398 shutdown_connections 2024-03-06T07:28:11.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.680+0000 7fa4a7c3bf40 1 --2- 172.21.15.83:0/514358398 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55bf58510c30 0x55bf58511030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.680+0000 7fa4a7c3bf40 1 --2- 172.21.15.83:0/514358398 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bf585aa5a0 0x55bf585b4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.680+0000 7fa4a7c3bf40 1 --2- 172.21.15.83:0/514358398 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bf585b41a0 0x55bf585ac410 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.680+0000 7fa4a7c3bf40 1 -- 172.21.15.83:0/514358398 >> 172.21.15.83:0/514358398 conn(0x55bf5850dbd0 msgr2=0x55bf585b2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:11.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.680+0000 7fa4a7c3bf40 1 -- 172.21.15.83:0/514358398 shutdown_connections 2024-03-06T07:28:11.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.680+0000 7fa4a7c3bf40 1 -- 172.21.15.83:0/514358398 wait complete. 2024-03-06T07:28:11.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.680+0000 7fa4a7c3bf40 1 Processor -- start 2024-03-06T07:28:11.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.680+0000 7fa4a7c3bf40 1 -- start start 2024-03-06T07:28:11.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.681+0000 7fa4a7c3bf40 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bf58510c30 0x55bf5864a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.681+0000 7fa4a7c3bf40 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55bf585aa5a0 0x55bf5859f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.681+0000 7fa4a7c3bf40 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bf585b41a0 0x55bf585a0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.681+0000 7fa4a7c3bf40 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf585b5940 con 0x55bf58510c30 2024-03-06T07:28:11.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.681+0000 7fa4a7c3bf40 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf5852b7b0 con 0x55bf585b41a0 2024-03-06T07:28:11.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.681+0000 7fa4a7c3bf40 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bf5852ba50 con 0x55bf585aa5a0 2024-03-06T07:28:11.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.681+0000 7fa4a4f06640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bf58510c30 0x55bf5864a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.681+0000 7fa4a4f06640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bf58510c30 0x55bf5864a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51264/0 (socket says 172.21.15.83:51264) 2024-03-06T07:28:11.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.681+0000 7fa4a4f06640 1 -- 172.21.15.83:0/3292737744 learned_addr learned my addr 172.21.15.83:0/3292737744 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:11.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.681+0000 7fa4a5707640 1 --2- 172.21.15.83:0/3292737744 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bf585b41a0 0x55bf585a0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.681+0000 7fa49ffff640 1 --2- 172.21.15.83:0/3292737744 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55bf585aa5a0 0x55bf5859f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.681+0000 7fa4a4f06640 1 -- 172.21.15.83:0/3292737744 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55bf585aa5a0 msgr2=0x55bf5859f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.681+0000 7fa4a4f06640 1 --2- 172.21.15.83:0/3292737744 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55bf585aa5a0 0x55bf5859f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.681+0000 7fa4a4f06640 1 -- 172.21.15.83:0/3292737744 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bf585b41a0 msgr2=0x55bf585a0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.681+0000 7fa4a4f06640 1 --2- 172.21.15.83:0/3292737744 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bf585b41a0 0x55bf585a0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.681+0000 7fa4a4f06640 1 -- 172.21.15.83:0/3292737744 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bf5859fbd0 con 0x55bf58510c30 2024-03-06T07:28:11.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.681+0000 7fa4a5707640 1 --2- 172.21.15.83:0/3292737744 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bf585b41a0 0x55bf585a0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:11.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.681+0000 7fa49ffff640 1 --2- 172.21.15.83:0/3292737744 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55bf585aa5a0 0x55bf5859f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:11.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.681+0000 7fa4a4f06640 1 --2- 172.21.15.83:0/3292737744 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bf58510c30 0x55bf5864a800 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x55bf585aa520 tx=0x7fa498002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:11.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.682+0000 7fa49dffb640 1 -- 172.21.15.83:0/3292737744 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4980168d0 con 0x55bf58510c30 2024-03-06T07:28:11.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.682+0000 7fa49dffb640 1 -- 172.21.15.83:0/3292737744 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa498024070 con 0x55bf58510c30 2024-03-06T07:28:11.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.682+0000 7fa4a7c3bf40 1 -- 172.21.15.83:0/3292737744 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55bf5859feb0 con 0x55bf58510c30 2024-03-06T07:28:11.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.682+0000 7fa49dffb640 1 -- 172.21.15.83:0/3292737744 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa498016cd0 con 0x55bf58510c30 2024-03-06T07:28:11.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.682+0000 7fa4a7c3bf40 1 -- 172.21.15.83:0/3292737744 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bf585a4dc0 con 0x55bf58510c30 2024-03-06T07:28:11.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.683+0000 7fa49dffb640 1 -- 172.21.15.83:0/3292737744 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa49801d040 con 0x55bf58510c30 2024-03-06T07:28:11.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.683+0000 7fa49dffb640 1 --2- 172.21.15.83:0/3292737744 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa488069ee0 0x7fa48806c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.684+0000 7fa49dffb640 1 -- 172.21.15.83:0/3292737744 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fa49808aa60 con 0x55bf58510c30 2024-03-06T07:28:11.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.684+0000 7fa49ffff640 1 --2- 172.21.15.83:0/3292737744 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa488069ee0 0x7fa48806c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.684+0000 7fa4a7c3bf40 1 --2- 172.21.15.83:0/3292737744 >> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] conn(0x55bf5859f710 0x55bf58658cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.684+0000 7fa4a7c3bf40 1 -- 172.21.15.83:0/3292737744 --> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e116) v9 -- 0x55bf585a16a0 con 0x55bf5859f710 2024-03-06T07:28:11.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.684+0000 7fa4a5707640 1 --2- 172.21.15.83:0/3292737744 >> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] conn(0x55bf5859f710 0x55bf58658cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.684+0000 7fa49ffff640 1 --2- 172.21.15.83:0/3292737744 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa488069ee0 0x7fa48806c3a0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x55bf585b2550 tx=0x7fa49000c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:11.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.684+0000 7fa4a5707640 1 --2- 172.21.15.83:0/3292737744 >> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] conn(0x55bf5859f710 0x55bf58658cb0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:11.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.685+0000 7fa4a5707640 1 -- 172.21.15.83:0/3292737744 <== osd.0 v2:172.21.15.83:6808/1635427425 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (99) out=99b] v0'0 uv9 ondisk = 0) v8 ==== 164+0+99 (crc 0 0 0) 0x7fa4a0001730 con 0x55bf5859f710 2024-03-06T07:28:11.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.685+0000 7fa4a7c3bf40 1 -- 172.21.15.83:0/3292737744 >> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] conn(0x55bf5859f710 msgr2=0x55bf58658cb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.685+0000 7fa4a7c3bf40 1 --2- 172.21.15.83:0/3292737744 >> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] conn(0x55bf5859f710 0x55bf58658cb0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.685+0000 7fa4a7c3bf40 1 -- 172.21.15.83:0/3292737744 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa488069ee0 msgr2=0x7fa48806c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.685+0000 7fa4a7c3bf40 1 --2- 172.21.15.83:0/3292737744 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa488069ee0 0x7fa48806c3a0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x55bf585b2550 tx=0x7fa49000c040 comp rx=0 tx=0).stop 2024-03-06T07:28:11.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.685+0000 7fa4a7c3bf40 1 -- 172.21.15.83:0/3292737744 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bf58510c30 msgr2=0x55bf5864a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.685+0000 7fa4a7c3bf40 1 --2- 172.21.15.83:0/3292737744 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bf58510c30 0x55bf5864a800 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x55bf585aa520 tx=0x7fa498002fb0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.685+0000 7fa4a5707640 1 -- 172.21.15.83:0/3292737744 reap_dead start 2024-03-06T07:28:11.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.685+0000 7fa4a7c3bf40 1 -- 172.21.15.83:0/3292737744 shutdown_connections 2024-03-06T07:28:11.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.685+0000 7fa4a7c3bf40 1 -- 172.21.15.83:0/3292737744 >> 172.21.15.83:0/3292737744 conn(0x55bf5850dbd0 msgr2=0x55bf58511010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:11.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.685+0000 7fa4a7c3bf40 1 -- 172.21.15.83:0/3292737744 shutdown_connections 2024-03-06T07:28:11.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.685+0000 7fa4a7c3bf40 1 -- 172.21.15.83:0/3292737744 wait complete. 2024-03-06T07:28:11.694 INFO:teuthology.orchestra.run.smithi083.stdout:] 10000000001 stray0 2024-03-06T07:28:11.694 DEBUG:teuthology.orchestra.run.smithi083:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-06T07:28:11.780 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:28:11.780 INFO:teuthology.orchestra.run.smithi083.stdout: "ino": 1099511627777, 2024-03-06T07:28:11.780 INFO:teuthology.orchestra.run.smithi083.stdout: "ancestors": [ 2024-03-06T07:28:11.780 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:28:11.780 INFO:teuthology.orchestra.run.smithi083.stdout: "dirino": 1536, 2024-03-06T07:28:11.780 INFO:teuthology.orchestra.run.smithi083.stdout: "dname": "10000000001", 2024-03-06T07:28:11.780 INFO:teuthology.orchestra.run.smithi083.stdout: "version": 11 2024-03-06T07:28:11.781 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:28:11.781 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:28:11.781 INFO:teuthology.orchestra.run.smithi083.stdout: "dirino": 256, 2024-03-06T07:28:11.781 INFO:teuthology.orchestra.run.smithi083.stdout: "dname": "stray0", 2024-03-06T07:28:11.781 INFO:teuthology.orchestra.run.smithi083.stdout: "version": 5 2024-03-06T07:28:11.781 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:28:11.781 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2024-03-06T07:28:11.781 INFO:teuthology.orchestra.run.smithi083.stdout: "pool": 17, 2024-03-06T07:28:11.781 INFO:teuthology.orchestra.run.smithi083.stdout: "old_pools": [] 2024-03-06T07:28:11.781 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:28:11.781 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:28:11.782 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:11.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.922+0000 7fbf5129d640 1 Processor -- start 2024-03-06T07:28:11.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.922+0000 7fbf5129d640 1 -- start start 2024-03-06T07:28:11.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.923+0000 7fbf5129d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbf4c15bdd0 0x7fbf4c0c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.923+0000 7fbf5129d640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fbf4c15fa10 0x7fbf4c0ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.923+0000 7fbf5129d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf4c0cebe0 con 0x7fbf4c0d16e0 2024-03-06T07:28:11.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.923+0000 7fbf5129d640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf4c0ceec0 con 0x7fbf4c15bdd0 2024-03-06T07:28:11.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.923+0000 7fbf5129d640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf4c0cf1a0 con 0x7fbf4c15fa10 2024-03-06T07:28:11.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.923+0000 7fbf4affd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbf4c15bdd0 0x7fbf4c0c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.923+0000 7fbf4affd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbf4c15bdd0 0x7fbf4c0c5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51280/0 (socket says 172.21.15.83:51280) 2024-03-06T07:28:11.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.923+0000 7fbf4affd640 1 -- 172.21.15.83:0/3606454071 learned_addr learned my addr 172.21.15.83:0/3606454071 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:11.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.924+0000 7fbf4a7fc640 1 --2- 172.21.15.83:0/3606454071 >> v2:172.21.15.183:3300/0 conn(0x7fbf4c15fa10 0x7fbf4c0ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.924+0000 7fbf4affd640 1 -- 172.21.15.83:0/3606454071 >> v2:172.21.15.183:3300/0 conn(0x7fbf4c15fa10 msgr2=0x7fbf4c0ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:11.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.924+0000 7fbf4affd640 1 --2- 172.21.15.83:0/3606454071 >> v2:172.21.15.183:3300/0 conn(0x7fbf4c15fa10 0x7fbf4c0ce270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.924+0000 7fbf4affd640 1 -- 172.21.15.83:0/3606454071 >> v1:172.21.15.183:6789/0 conn(0x7fbf4c0d16e0 legacy=0x7fbf4c0d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:11.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.924+0000 7fbf4affd640 1 -- 172.21.15.83:0/3606454071 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf4c0cc8e0 con 0x7fbf4c15bdd0 2024-03-06T07:28:11.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.924+0000 7fbf4affd640 1 --2- 172.21.15.83:0/3606454071 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf4c15bdd0 0x7fbf4c0c5610 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c06e3c0 tx=0x7fbf3c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2be040d77001b519 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:11.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.925+0000 7fbf49ffb640 1 -- 172.21.15.83:0/3606454071 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3c0317c0 con 0x7fbf4c15bdd0 2024-03-06T07:28:11.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.925+0000 7fbf49ffb640 1 -- 172.21.15.83:0/3606454071 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf3c037070 con 0x7fbf4c15bdd0 2024-03-06T07:28:11.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.925+0000 7fbf49ffb640 1 -- 172.21.15.83:0/3606454071 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3c031b80 con 0x7fbf4c15bdd0 2024-03-06T07:28:11.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.925+0000 7fbf5129d640 1 -- 172.21.15.83:0/3606454071 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf4c15bdd0 msgr2=0x7fbf4c0c5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.925+0000 7fbf5129d640 1 --2- 172.21.15.83:0/3606454071 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf4c15bdd0 0x7fbf4c0c5610 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c06e3c0 tx=0x7fbf3c02b860 comp rx=0 tx=0).stop 2024-03-06T07:28:11.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.925+0000 7fbf5129d640 1 -- 172.21.15.83:0/3606454071 shutdown_connections 2024-03-06T07:28:11.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.925+0000 7fbf5129d640 1 --2- 172.21.15.83:0/3606454071 >> v2:172.21.15.183:3300/0 conn(0x7fbf4c15fa10 0x7fbf4c0ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.925+0000 7fbf5129d640 1 --2- 172.21.15.83:0/3606454071 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf4c15bdd0 0x7fbf4c0c5610 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.925+0000 7fbf5129d640 1 -- 172.21.15.83:0/3606454071 >> 172.21.15.83:0/3606454071 conn(0x7fbf4c09a9f0 msgr2=0x7fbf4c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:11.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.925+0000 7fbf5129d640 1 -- 172.21.15.83:0/3606454071 shutdown_connections 2024-03-06T07:28:11.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.925+0000 7fbf5129d640 1 -- 172.21.15.83:0/3606454071 wait complete. 2024-03-06T07:28:11.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.926+0000 7fbf5129d640 1 Processor -- start 2024-03-06T07:28:11.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.926+0000 7fbf5129d640 1 -- start start 2024-03-06T07:28:11.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.927+0000 7fbf5129d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbf4c0d16e0 0x7fbf4c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.927+0000 7fbf5129d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf4c15bdd0 0x7fbf4c1e5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.927+0000 7fbf5129d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbf4c15fa10 0x7fbf4c154e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.927+0000 7fbf5129d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4c15c1b0 con 0x7fbf4c15bdd0 2024-03-06T07:28:11.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.927+0000 7fbf5129d640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4c0cc5d0 con 0x7fbf4c0d16e0 2024-03-06T07:28:11.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.927+0000 7fbf5129d640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4c0ce7b0 con 0x7fbf4c15fa10 2024-03-06T07:28:11.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.927+0000 7fbf4a7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf4c15bdd0 0x7fbf4c1e5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.927+0000 7fbf4a7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf4c15bdd0 0x7fbf4c1e5730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51288/0 (socket says 172.21.15.83:51288) 2024-03-06T07:28:11.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.927+0000 7fbf4a7fc640 1 -- 172.21.15.83:0/1692933503 learned_addr learned my addr 172.21.15.83:0/1692933503 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:11.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.928+0000 7fbf4b7fe640 1 --2- 172.21.15.83:0/1692933503 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbf4c15fa10 0x7fbf4c154e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.928+0000 7fbf4affd640 1 --2- 172.21.15.83:0/1692933503 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbf4c0d16e0 0x7fbf4c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.928+0000 7fbf4a7fc640 1 -- 172.21.15.83:0/1692933503 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbf4c15fa10 msgr2=0x7fbf4c154e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.928+0000 7fbf4a7fc640 1 --2- 172.21.15.83:0/1692933503 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbf4c15fa10 0x7fbf4c154e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.928+0000 7fbf4a7fc640 1 -- 172.21.15.83:0/1692933503 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbf4c0d16e0 msgr2=0x7fbf4c06d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.928+0000 7fbf4a7fc640 1 --2- 172.21.15.83:0/1692933503 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbf4c0d16e0 0x7fbf4c06d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.928+0000 7fbf4a7fc640 1 -- 172.21.15.83:0/1692933503 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf4c1e61f0 con 0x7fbf4c15bdd0 2024-03-06T07:28:11.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.928+0000 7fbf4affd640 1 --2- 172.21.15.83:0/1692933503 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbf4c0d16e0 0x7fbf4c06d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:11.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.928+0000 7fbf4a7fc640 1 --2- 172.21.15.83:0/1692933503 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf4c15bdd0 0x7fbf4c1e5730 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7fbf4000d7d0 tx=0x7fbf4000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:11.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.928+0000 7fbf337fe640 1 -- 172.21.15.83:0/1692933503 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf40004020 con 0x7fbf4c15bdd0 2024-03-06T07:28:11.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.928+0000 7fbf337fe640 1 -- 172.21.15.83:0/1692933503 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf400041e0 con 0x7fbf4c15bdd0 2024-03-06T07:28:11.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.928+0000 7fbf5129d640 1 -- 172.21.15.83:0/1692933503 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf4c1e64d0 con 0x7fbf4c15bdd0 2024-03-06T07:28:11.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.928+0000 7fbf337fe640 1 -- 172.21.15.83:0/1692933503 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf400043d0 con 0x7fbf4c15bdd0 2024-03-06T07:28:11.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.928+0000 7fbf5129d640 1 -- 172.21.15.83:0/1692933503 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf4c15d9a0 con 0x7fbf4c15bdd0 2024-03-06T07:28:11.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.929+0000 7fbf5129d640 1 -- 172.21.15.83:0/1692933503 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf4c04f330 con 0x7fbf4c15bdd0 2024-03-06T07:28:11.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.930+0000 7fbf337fe640 1 -- 172.21.15.83:0/1692933503 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbf4001b020 con 0x7fbf4c15bdd0 2024-03-06T07:28:11.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.930+0000 7fbf337fe640 1 --2- 172.21.15.83:0/1692933503 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbf3806b4d0 0x7fbf3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.931+0000 7fbf4affd640 1 --2- 172.21.15.83:0/1692933503 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbf3806b4d0 0x7fbf3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.931+0000 7fbf337fe640 1 -- 172.21.15.83:0/1692933503 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fbf4008a0a0 con 0x7fbf4c15bdd0 2024-03-06T07:28:11.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.931+0000 7fbf4affd640 1 --2- 172.21.15.83:0/1692933503 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbf3806b4d0 0x7fbf3806d990 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c036000 tx=0x7fbf3c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:11.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:11.935+0000 7fbf337fe640 1 -- 172.21.15.83:0/1692933503 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbf40058ec0 con 0x7fbf4c15bdd0 2024-03-06T07:28:12.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.151+0000 7fbf5129d640 1 -- 172.21.15.83:0/1692933503 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf4c15b560 con 0x7fbf4c15bdd0 2024-03-06T07:28:12.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.152+0000 7fbf337fe640 1 -- 172.21.15.83:0/1692933503 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4984 (secure 0 0 0) 0x7fbf400587c0 con 0x7fbf4c15bdd0 2024-03-06T07:28:12.154 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:28:12.154 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7051,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/101642171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":101642171},{"type":"v1","addr":"172.21.15.183:6833","nonce":101642171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7057,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3146884900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":3146884900},{"type":"v1","addr":"172.21.15.183:6835","nonce":3146884900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8208,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1922041621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1922041621},{"type":"v1","addr":"172.21.15.83:6835","nonce":1922041621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.491022+0000","modified":"2024-03-06T07:27:37.006649+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8229":{"gid":8229,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/2003687153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2003687153},{"type":"v1","addr":"172.21.15.83:6837","nonce":2003687153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8229,"qdb_cluster":[8229]},"id":8}]} 2024-03-06T07:28:12.154 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 66 2024-03-06T07:28:12.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.154+0000 7fbf5129d640 1 -- 172.21.15.83:0/1692933503 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbf3806b4d0 msgr2=0x7fbf3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.154+0000 7fbf5129d640 1 --2- 172.21.15.83:0/1692933503 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbf3806b4d0 0x7fbf3806d990 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c036000 tx=0x7fbf3c008030 comp rx=0 tx=0).stop 2024-03-06T07:28:12.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.154+0000 7fbf5129d640 1 -- 172.21.15.83:0/1692933503 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf4c15bdd0 msgr2=0x7fbf4c1e5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.154+0000 7fbf5129d640 1 --2- 172.21.15.83:0/1692933503 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf4c15bdd0 0x7fbf4c1e5730 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7fbf4000d7d0 tx=0x7fbf4000dc90 comp rx=0 tx=0).stop 2024-03-06T07:28:12.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.154+0000 7fbf5129d640 1 -- 172.21.15.83:0/1692933503 shutdown_connections 2024-03-06T07:28:12.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.154+0000 7fbf5129d640 1 --2- 172.21.15.83:0/1692933503 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbf3806b4d0 0x7fbf3806d990 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.154+0000 7fbf5129d640 1 --2- 172.21.15.83:0/1692933503 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbf4c15fa10 0x7fbf4c154e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.154+0000 7fbf5129d640 1 --2- 172.21.15.83:0/1692933503 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf4c15bdd0 0x7fbf4c1e5730 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.154+0000 7fbf5129d640 1 --2- 172.21.15.83:0/1692933503 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbf4c0d16e0 0x7fbf4c06d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.154+0000 7fbf5129d640 1 -- 172.21.15.83:0/1692933503 >> 172.21.15.83:0/1692933503 conn(0x7fbf4c09a9f0 msgr2=0x7fbf4c0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:12.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.154+0000 7fbf5129d640 1 -- 172.21.15.83:0/1692933503 shutdown_connections 2024-03-06T07:28:12.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.154+0000 7fbf5129d640 1 -- 172.21.15.83:0/1692933503 wait complete. 2024-03-06T07:28:12.166 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_reintegrated 2024-03-06T07:28:12.283 INFO:teuthology.orchestra.run.smithi083.stdout:{"mds_cache":{"strays_reintegrated":0}} 2024-03-06T07:28:12.283 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 0 } } 2024-03-06T07:28:12.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:12.283 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest/mnt.0 && exec mv dir_2/file_b dir_2/file_c) 2024-03-06T07:28:12.307 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:12.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.446+0000 7fb9cf21c640 1 Processor -- start 2024-03-06T07:28:12.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.447+0000 7fb9cf21c640 1 -- start start 2024-03-06T07:28:12.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.447+0000 7fb9cf21c640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fb9c80d22b0 0x7fb9c80d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.447+0000 7fb9cf21c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb9c80b41f0 0x7fb9c80b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.447+0000 7fb9cf21c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9c80b3430 con 0x7fb9c815dfe0 2024-03-06T07:28:12.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.447+0000 7fb9cf21c640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9c80d2c50 con 0x7fb9c80d22b0 2024-03-06T07:28:12.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.447+0000 7fb9cf21c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c80b31e0 con 0x7fb9c80b41f0 2024-03-06T07:28:12.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.447+0000 7fb9ccfb8640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fb9c815dfe0 0x7fb9c80cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:54794/0 (socket says 172.21.15.83:54794) 2024-03-06T07:28:12.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.447+0000 7fb9ccfb8640 1 -- 172.21.15.83:0/1815022679 learned_addr learned my addr 172.21.15.83:0/1815022679 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:28:12.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.448+0000 7fb9cd7b9640 1 --2- 172.21.15.83:0/1815022679 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb9c80b41f0 0x7fb9c80b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.448+0000 7fb9c7fff640 1 --2- 172.21.15.83:0/1815022679 >> v2:172.21.15.183:3300/0 conn(0x7fb9c80d22b0 0x7fb9c80d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.448+0000 7fb9c77fe640 1 -- 172.21.15.83:0/1815022679 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2010701105 0 0) 0x7fb9c80b3430 con 0x7fb9c815dfe0 2024-03-06T07:28:12.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.448+0000 7fb9c77fe640 1 -- 172.21.15.83:0/1815022679 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9b8003520 con 0x7fb9c815dfe0 2024-03-06T07:28:12.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.448+0000 7fb9c77fe640 1 -- 172.21.15.83:0/1815022679 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1305415 0 0) 0x7fb9b8003520 con 0x7fb9c815dfe0 2024-03-06T07:28:12.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.448+0000 7fb9c77fe640 1 -- 172.21.15.83:0/1815022679 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb9c80b41f0 msgr2=0x7fb9c80b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:12.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.448+0000 7fb9c77fe640 1 --2- 172.21.15.83:0/1815022679 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb9c80b41f0 0x7fb9c80b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.448+0000 7fb9c77fe640 1 -- 172.21.15.83:0/1815022679 >> v2:172.21.15.183:3300/0 conn(0x7fb9c80d22b0 msgr2=0x7fb9c80d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:12.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.448+0000 7fb9c77fe640 1 --2- 172.21.15.83:0/1815022679 >> v2:172.21.15.183:3300/0 conn(0x7fb9c80d22b0 0x7fb9c80d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.448+0000 7fb9c77fe640 1 -- 172.21.15.83:0/1815022679 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c80b3710 con 0x7fb9c815dfe0 2024-03-06T07:28:12.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.448+0000 7fb9c77fe640 1 -- 172.21.15.83:0/1815022679 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7fb9bc001670 con 0x7fb9c815dfe0 2024-03-06T07:28:12.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.448+0000 7fb9cd7b9640 1 --2- 172.21.15.83:0/1815022679 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb9c80b41f0 0x7fb9c80b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:12.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.448+0000 7fb9c7fff640 1 --2- 172.21.15.83:0/1815022679 >> v2:172.21.15.183:3300/0 conn(0x7fb9c80d22b0 0x7fb9c80d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:12.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.448+0000 7fb9c77fe640 1 -- 172.21.15.83:0/1815022679 >> v1:172.21.15.83:6789/0 conn(0x7fb9c815dfe0 legacy=0x7fb9c80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.448+0000 7fb9c77fe640 1 --2- 172.21.15.83:0/1815022679 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9b80039b0 0x7fb9b8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.448+0000 7fb9c77fe640 1 --2- 172.21.15.83:0/1815022679 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9b8004800 0x7fb9b8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.448+0000 7fb9c77fe640 1 --2- 172.21.15.83:0/1815022679 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb9c80b41f0 0x7fb9b80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.448+0000 7fb9c77fe640 1 -- 172.21.15.83:0/1815022679 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c80b31e0 con 0x7fb9b80039b0 2024-03-06T07:28:12.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.448+0000 7fb9c77fe640 1 -- 172.21.15.83:0/1815022679 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b800b9c0 con 0x7fb9b8004800 2024-03-06T07:28:12.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.448+0000 7fb9c77fe640 1 -- 172.21.15.83:0/1815022679 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b800bb40 con 0x7fb9c80b41f0 2024-03-06T07:28:12.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.448+0000 7fb9ccfb8640 1 --2- 172.21.15.83:0/1815022679 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9b80039b0 0x7fb9b8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.448+0000 7fb9cd7b9640 1 --2- 172.21.15.83:0/1815022679 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb9c80b41f0 0x7fb9b80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.448+0000 7fb9c7fff640 1 --2- 172.21.15.83:0/1815022679 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9b8004800 0x7fb9b8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.448+0000 7fb9ccfb8640 1 -- 172.21.15.83:0/1815022679 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb9c80b41f0 msgr2=0x7fb9b80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:12.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.448+0000 7fb9ccfb8640 1 --2- 172.21.15.83:0/1815022679 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb9c80b41f0 0x7fb9b80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.448+0000 7fb9ccfb8640 1 -- 172.21.15.83:0/1815022679 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9b8004800 msgr2=0x7fb9b8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:12.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.448+0000 7fb9ccfb8640 1 --2- 172.21.15.83:0/1815022679 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9b8004800 0x7fb9b8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.448+0000 7fb9ccfb8640 1 -- 172.21.15.83:0/1815022679 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b8003520 con 0x7fb9b80039b0 2024-03-06T07:28:12.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.449+0000 7fb9ccfb8640 1 --2- 172.21.15.83:0/1815022679 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9b80039b0 0x7fb9b8003dd0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc00c290 tx=0x7fb9b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b94818a6f3776084 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:12.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.449+0000 7fb9c77fe640 1 -- 172.21.15.83:0/1815022679 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9bc007d50 con 0x7fb9b80039b0 2024-03-06T07:28:12.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.449+0000 7fb9c77fe640 1 -- 172.21.15.83:0/1815022679 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9bc030070 con 0x7fb9b80039b0 2024-03-06T07:28:12.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.449+0000 7fb9c77fe640 1 -- 172.21.15.83:0/1815022679 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9bc02cd80 con 0x7fb9b80039b0 2024-03-06T07:28:12.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.450+0000 7fb9cf21c640 1 -- 172.21.15.83:0/1815022679 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9b80039b0 msgr2=0x7fb9b8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.450+0000 7fb9cf21c640 1 --2- 172.21.15.83:0/1815022679 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9b80039b0 0x7fb9b8003dd0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc00c290 tx=0x7fb9b8005830 comp rx=0 tx=0).stop 2024-03-06T07:28:12.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.450+0000 7fb9cd7b9640 1 -- 172.21.15.83:0/1815022679 reap_dead start 2024-03-06T07:28:12.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.450+0000 7fb9cf21c640 1 -- 172.21.15.83:0/1815022679 shutdown_connections 2024-03-06T07:28:12.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.450+0000 7fb9cf21c640 1 -- 172.21.15.83:0/1815022679 >> 172.21.15.83:0/1815022679 conn(0x7fb9c80d3cd0 msgr2=0x7fb9c80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:12.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.450+0000 7fb9cf21c640 1 -- 172.21.15.83:0/1815022679 shutdown_connections 2024-03-06T07:28:12.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.450+0000 7fb9cf21c640 1 -- 172.21.15.83:0/1815022679 wait complete. 2024-03-06T07:28:12.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.451+0000 7fb9cf21c640 1 Processor -- start 2024-03-06T07:28:12.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.451+0000 7fb9cf21c640 1 -- start start 2024-03-06T07:28:12.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.452+0000 7fb9cf21c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9b80039b0 0x7fb9c815edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.452+0000 7fb9cf21c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9c8160490 0x7fb9c80ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.452+0000 7fb9cf21c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb9c815c5f0 0x7fb9c806bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.452+0000 7fb9cf21c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c80cfa00 con 0x7fb9c8160490 2024-03-06T07:28:12.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.452+0000 7fb9cf21c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c80b3d60 con 0x7fb9b80039b0 2024-03-06T07:28:12.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.452+0000 7fb9cf21c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c80d1c70 con 0x7fb9c815c5f0 2024-03-06T07:28:12.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.452+0000 7fb9c7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9c8160490 0x7fb9c80ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.452+0000 7fb9c7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9c8160490 0x7fb9c80ccf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51306/0 (socket says 172.21.15.83:51306) 2024-03-06T07:28:12.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.452+0000 7fb9c7fff640 1 -- 172.21.15.83:0/1440585285 learned_addr learned my addr 172.21.15.83:0/1440585285 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:12.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.452+0000 7fb9cd7b9640 1 --2- 172.21.15.83:0/1440585285 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb9c815c5f0 0x7fb9c806bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.452+0000 7fb9ccfb8640 1 --2- 172.21.15.83:0/1440585285 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9b80039b0 0x7fb9c815edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.452+0000 7fb9c7fff640 1 -- 172.21.15.83:0/1440585285 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb9c815c5f0 msgr2=0x7fb9c806bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.452+0000 7fb9c7fff640 1 --2- 172.21.15.83:0/1440585285 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb9c815c5f0 0x7fb9c806bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.452+0000 7fb9c7fff640 1 -- 172.21.15.83:0/1440585285 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9b80039b0 msgr2=0x7fb9c815edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.452+0000 7fb9c7fff640 1 --2- 172.21.15.83:0/1440585285 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9b80039b0 0x7fb9c815edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.452+0000 7fb9c7fff640 1 -- 172.21.15.83:0/1440585285 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c80cd450 con 0x7fb9c8160490 2024-03-06T07:28:12.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.452+0000 7fb9cd7b9640 1 --2- 172.21.15.83:0/1440585285 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb9c815c5f0 0x7fb9c806bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:12.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.452+0000 7fb9c7fff640 1 --2- 172.21.15.83:0/1440585285 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9c8160490 0x7fb9c80ccf10 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fb9b4011d00 tx=0x7fb9b400f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:12.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.452+0000 7fb9c57fa640 1 -- 172.21.15.83:0/1440585285 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b400fd20 con 0x7fb9c8160490 2024-03-06T07:28:12.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.453+0000 7fb9c57fa640 1 -- 172.21.15.83:0/1440585285 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9b4004030 con 0x7fb9c8160490 2024-03-06T07:28:12.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.453+0000 7fb9cf21c640 1 -- 172.21.15.83:0/1440585285 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9c815f3f0 con 0x7fb9c8160490 2024-03-06T07:28:12.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.453+0000 7fb9c57fa640 1 -- 172.21.15.83:0/1440585285 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b4004420 con 0x7fb9c8160490 2024-03-06T07:28:12.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.453+0000 7fb9cf21c640 1 -- 172.21.15.83:0/1440585285 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9c815fc90 con 0x7fb9c8160490 2024-03-06T07:28:12.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.453+0000 7fb9cf21c640 1 -- 172.21.15.83:0/1440585285 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9c804f330 con 0x7fb9c8160490 2024-03-06T07:28:12.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.454+0000 7fb9c57fa640 1 -- 172.21.15.83:0/1440585285 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb9b4012040 con 0x7fb9c8160490 2024-03-06T07:28:12.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.454+0000 7fb9c57fa640 1 --2- 172.21.15.83:0/1440585285 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb9ac06b500 0x7fb9ac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.454+0000 7fb9c57fa640 1 -- 172.21.15.83:0/1440585285 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fb9b408a7d0 con 0x7fb9c8160490 2024-03-06T07:28:12.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.454+0000 7fb9ccfb8640 1 --2- 172.21.15.83:0/1440585285 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb9ac06b500 0x7fb9ac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.455+0000 7fb9ccfb8640 1 --2- 172.21.15.83:0/1440585285 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb9ac06b500 0x7fb9ac06d9c0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc034000 tx=0x7fb9bc007000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:12.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.459+0000 7fb9c57fa640 1 -- 172.21.15.83:0/1440585285 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb9b4019070 con 0x7fb9c8160490 2024-03-06T07:28:12.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.671+0000 7fb9cf21c640 1 -- 172.21.15.83:0/1440585285 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9c80d0e30 con 0x7fb9c8160490 2024-03-06T07:28:12.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.672+0000 7fb9c57fa640 1 -- 172.21.15.83:0/1440585285 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4984 (secure 0 0 0) 0x7fb9b40594d0 con 0x7fb9c8160490 2024-03-06T07:28:12.675 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:28:12.675 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7051,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/101642171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":101642171},{"type":"v1","addr":"172.21.15.183:6833","nonce":101642171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7057,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3146884900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":3146884900},{"type":"v1","addr":"172.21.15.183:6835","nonce":3146884900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8208,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1922041621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1922041621},{"type":"v1","addr":"172.21.15.83:6835","nonce":1922041621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.491022+0000","modified":"2024-03-06T07:27:37.006649+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8229":{"gid":8229,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/2003687153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2003687153},{"type":"v1","addr":"172.21.15.83:6837","nonce":2003687153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8229,"qdb_cluster":[8229]},"id":8}]} 2024-03-06T07:28:12.675 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 66 2024-03-06T07:28:12.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.674+0000 7fb9cf21c640 1 -- 172.21.15.83:0/1440585285 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb9ac06b500 msgr2=0x7fb9ac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.674+0000 7fb9cf21c640 1 --2- 172.21.15.83:0/1440585285 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb9ac06b500 0x7fb9ac06d9c0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc034000 tx=0x7fb9bc007000 comp rx=0 tx=0).stop 2024-03-06T07:28:12.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.674+0000 7fb9cf21c640 1 -- 172.21.15.83:0/1440585285 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9c8160490 msgr2=0x7fb9c80ccf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.674+0000 7fb9cf21c640 1 --2- 172.21.15.83:0/1440585285 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9c8160490 0x7fb9c80ccf10 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fb9b4011d00 tx=0x7fb9b400f610 comp rx=0 tx=0).stop 2024-03-06T07:28:12.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.674+0000 7fb9cf21c640 1 -- 172.21.15.83:0/1440585285 shutdown_connections 2024-03-06T07:28:12.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.674+0000 7fb9cf21c640 1 --2- 172.21.15.83:0/1440585285 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb9ac06b500 0x7fb9ac06d9c0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.674+0000 7fb9cf21c640 1 --2- 172.21.15.83:0/1440585285 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb9c815c5f0 0x7fb9c806bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.674+0000 7fb9cf21c640 1 --2- 172.21.15.83:0/1440585285 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9c8160490 0x7fb9c80ccf10 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.674+0000 7fb9cf21c640 1 --2- 172.21.15.83:0/1440585285 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9b80039b0 0x7fb9c815edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.674+0000 7fb9cf21c640 1 -- 172.21.15.83:0/1440585285 >> 172.21.15.83:0/1440585285 conn(0x7fb9c80d3cd0 msgr2=0x7fb9c80cf060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:12.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.674+0000 7fb9cf21c640 1 -- 172.21.15.83:0/1440585285 shutdown_connections 2024-03-06T07:28:12.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.674+0000 7fb9cf21c640 1 -- 172.21.15.83:0/1440585285 wait complete. 2024-03-06T07:28:12.687 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:28:12.841 INFO:teuthology.orchestra.run.smithi083.stdout:{"mds_cache":{"num_strays":0}} 2024-03-06T07:28:12.842 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 0 } } 2024-03-06T07:28:12.842 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:12.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.940+0000 7fe3feecf640 1 Processor -- start 2024-03-06T07:28:12.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.940+0000 7fe3feecf640 1 -- start start 2024-03-06T07:28:12.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.940+0000 7fe3feecf640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fe3f80d2710 0x7fe3f80b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.940+0000 7fe3feecf640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3f80b3430 con 0x7fe3f80d22b0 2024-03-06T07:28:12.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.940+0000 7fe3feecf640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3f80b3710 con 0x7fe3f815dfe0 2024-03-06T07:28:12.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.940+0000 7fe3feecf640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3f80b31e0 con 0x7fe3f80d2710 2024-03-06T07:28:12.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.941+0000 7fe3f7fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fe3f80d22b0 0x7fe3f80b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:54800/0 (socket says 172.21.15.83:54800) 2024-03-06T07:28:12.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.941+0000 7fe3f7fff640 1 -- 172.21.15.83:0/1733454810 learned_addr learned my addr 172.21.15.83:0/1733454810 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:28:12.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.941+0000 7fe3fd46c640 1 --2- 172.21.15.83:0/1733454810 >> v2:172.21.15.183:3300/0 conn(0x7fe3f80d2710 0x7fe3f80b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.941+0000 7fe3f77fe640 1 -- 172.21.15.83:0/1733454810 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1699845916 0 0) 0x7fe3f80b3430 con 0x7fe3f80d22b0 2024-03-06T07:28:12.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.941+0000 7fe3f77fe640 1 -- 172.21.15.83:0/1733454810 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3e8003520 con 0x7fe3f80d22b0 2024-03-06T07:28:12.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.941+0000 7fe3fd46c640 1 -- 172.21.15.83:0/1733454810 >> v1:172.21.15.183:6789/0 conn(0x7fe3f815dfe0 legacy=0x7fe3f80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.941+0000 7fe3fd46c640 1 -- 172.21.15.83:0/1733454810 >> v1:172.21.15.83:6789/0 conn(0x7fe3f80d22b0 legacy=0x7fe3f80b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.941+0000 7fe3fd46c640 1 -- 172.21.15.83:0/1733454810 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f80b39f0 con 0x7fe3f80d2710 2024-03-06T07:28:12.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.942+0000 7fe3fd46c640 1 --2- 172.21.15.83:0/1733454810 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3f80d2710 0x7fe3f80b4900 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe3f0009dc0 tx=0x7fe3f002bf20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f827938ffbf7284e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:12.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.942+0000 7fe3f77fe640 1 -- 172.21.15.83:0/1733454810 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f0039070 con 0x7fe3f80d2710 2024-03-06T07:28:12.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.942+0000 7fe3f77fe640 1 -- 172.21.15.83:0/1733454810 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3f002ccd0 con 0x7fe3f80d2710 2024-03-06T07:28:12.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.942+0000 7fe3f77fe640 1 -- 172.21.15.83:0/1733454810 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f0034660 con 0x7fe3f80d2710 2024-03-06T07:28:12.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.943+0000 7fe3feecf640 1 -- 172.21.15.83:0/1733454810 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3f80d2710 msgr2=0x7fe3f80b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.943+0000 7fe3feecf640 1 --2- 172.21.15.83:0/1733454810 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3f80d2710 0x7fe3f80b4900 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe3f0009dc0 tx=0x7fe3f002bf20 comp rx=0 tx=0).stop 2024-03-06T07:28:12.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.943+0000 7fe3feecf640 1 -- 172.21.15.83:0/1733454810 shutdown_connections 2024-03-06T07:28:12.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.943+0000 7fe3feecf640 1 --2- 172.21.15.83:0/1733454810 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3f80d2710 0x7fe3f80b4900 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.943+0000 7fe3feecf640 1 -- 172.21.15.83:0/1733454810 >> 172.21.15.83:0/1733454810 conn(0x7fe3f80d3cd0 msgr2=0x7fe3f80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:12.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.943+0000 7fe3feecf640 1 -- 172.21.15.83:0/1733454810 shutdown_connections 2024-03-06T07:28:12.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.943+0000 7fe3feecf640 1 -- 172.21.15.83:0/1733454810 wait complete. 2024-03-06T07:28:12.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.944+0000 7fe3feecf640 1 Processor -- start 2024-03-06T07:28:12.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.944+0000 7fe3feecf640 1 -- start start 2024-03-06T07:28:12.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.945+0000 7fe3feecf640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe3f80d22b0 0x7fe3f814f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.945+0000 7fe3feecf640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3f80d2710 0x7fe3f80b2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.945+0000 7fe3feecf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3f815dfe0 0x7fe3f80b25a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.945+0000 7fe3feecf640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f80cfa00 con 0x7fe3f815dfe0 2024-03-06T07:28:12.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.945+0000 7fe3feecf640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f80b3d60 con 0x7fe3f80d2710 2024-03-06T07:28:12.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.945+0000 7fe3feecf640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f80d1c70 con 0x7fe3f80d22b0 2024-03-06T07:28:12.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.945+0000 7fe3fd46c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3f815dfe0 0x7fe3f80b25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.945+0000 7fe3fd46c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3f815dfe0 0x7fe3f80b25a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51322/0 (socket says 172.21.15.83:51322) 2024-03-06T07:28:12.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.945+0000 7fe3fd46c640 1 -- 172.21.15.83:0/1951894200 learned_addr learned my addr 172.21.15.83:0/1951894200 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:12.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.945+0000 7fe3f7fff640 1 --2- 172.21.15.83:0/1951894200 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3f80d2710 0x7fe3f80b2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.945+0000 7fe3fcc6b640 1 --2- 172.21.15.83:0/1951894200 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe3f80d22b0 0x7fe3f814f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.945+0000 7fe3fd46c640 1 -- 172.21.15.83:0/1951894200 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe3f80d22b0 msgr2=0x7fe3f814f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.945+0000 7fe3fd46c640 1 --2- 172.21.15.83:0/1951894200 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe3f80d22b0 0x7fe3f814f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.945+0000 7fe3fd46c640 1 -- 172.21.15.83:0/1951894200 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3f80d2710 msgr2=0x7fe3f80b2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.945+0000 7fe3fd46c640 1 --2- 172.21.15.83:0/1951894200 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3f80d2710 0x7fe3f80b2060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.945+0000 7fe3fd46c640 1 -- 172.21.15.83:0/1951894200 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f814f760 con 0x7fe3f815dfe0 2024-03-06T07:28:12.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.945+0000 7fe3f7fff640 1 --2- 172.21.15.83:0/1951894200 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3f80d2710 0x7fe3f80b2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:12.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.945+0000 7fe3fd46c640 1 --2- 172.21.15.83:0/1951894200 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3f815dfe0 0x7fe3f80b25a0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7fe3f0037040 tx=0x7fe3f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:12.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.945+0000 7fe3f57fa640 1 -- 172.21.15.83:0/1951894200 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f0039050 con 0x7fe3f815dfe0 2024-03-06T07:28:12.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.945+0000 7fe3f57fa640 1 -- 172.21.15.83:0/1951894200 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3f0045070 con 0x7fe3f815dfe0 2024-03-06T07:28:12.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.946+0000 7fe3feecf640 1 -- 172.21.15.83:0/1951894200 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3f80b7830 con 0x7fe3f815dfe0 2024-03-06T07:28:12.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.946+0000 7fe3f57fa640 1 -- 172.21.15.83:0/1951894200 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f003e850 con 0x7fe3f815dfe0 2024-03-06T07:28:12.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.946+0000 7fe3feecf640 1 -- 172.21.15.83:0/1951894200 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3f80b6d50 con 0x7fe3f815dfe0 2024-03-06T07:28:12.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.947+0000 7fe3feecf640 1 -- 172.21.15.83:0/1951894200 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3f804f330 con 0x7fe3f815dfe0 2024-03-06T07:28:12.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.947+0000 7fe3f57fa640 1 -- 172.21.15.83:0/1951894200 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe3f003e9f0 con 0x7fe3f815dfe0 2024-03-06T07:28:12.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.947+0000 7fe3f57fa640 1 --2- 172.21.15.83:0/1951894200 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe3dc06b4d0 0x7fe3dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.948+0000 7fe3f57fa640 1 -- 172.21.15.83:0/1951894200 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fe3f00ad0a0 con 0x7fe3f815dfe0 2024-03-06T07:28:12.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.948+0000 7fe3fcc6b640 1 --2- 172.21.15.83:0/1951894200 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe3dc06b4d0 0x7fe3dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.948+0000 7fe3fcc6b640 1 --2- 172.21.15.83:0/1951894200 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe3dc06b4d0 0x7fe3dc06d990 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec0016e0 tx=0x7fe3ec009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:12.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:12.952+0000 7fe3f57fa640 1 -- 172.21.15.83:0/1951894200 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe3f814f760 con 0x7fe3f815dfe0 2024-03-06T07:28:13.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.179+0000 7fe3feecf640 1 -- 172.21.15.83:0/1951894200 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe3f80b0af0 con 0x7fe3f815dfe0 2024-03-06T07:28:13.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.180+0000 7fe3f57fa640 1 -- 172.21.15.83:0/1951894200 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4984 (secure 0 0 0) 0x7fe3f007bf60 con 0x7fe3f815dfe0 2024-03-06T07:28:13.183 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:28:13.183 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7051,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/101642171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":101642171},{"type":"v1","addr":"172.21.15.183:6833","nonce":101642171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7057,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3146884900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":3146884900},{"type":"v1","addr":"172.21.15.183:6835","nonce":3146884900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8208,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1922041621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1922041621},{"type":"v1","addr":"172.21.15.83:6835","nonce":1922041621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.491022+0000","modified":"2024-03-06T07:27:37.006649+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8229":{"gid":8229,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/2003687153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2003687153},{"type":"v1","addr":"172.21.15.83:6837","nonce":2003687153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8229,"qdb_cluster":[8229]},"id":8}]} 2024-03-06T07:28:13.183 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 66 2024-03-06T07:28:13.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.182+0000 7fe3feecf640 1 -- 172.21.15.83:0/1951894200 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe3dc06b4d0 msgr2=0x7fe3dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.182+0000 7fe3feecf640 1 --2- 172.21.15.83:0/1951894200 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe3dc06b4d0 0x7fe3dc06d990 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec0016e0 tx=0x7fe3ec009040 comp rx=0 tx=0).stop 2024-03-06T07:28:13.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.182+0000 7fe3feecf640 1 -- 172.21.15.83:0/1951894200 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3f815dfe0 msgr2=0x7fe3f80b25a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.182+0000 7fe3feecf640 1 --2- 172.21.15.83:0/1951894200 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3f815dfe0 0x7fe3f80b25a0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7fe3f0037040 tx=0x7fe3f0004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:13.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.182+0000 7fe3feecf640 1 -- 172.21.15.83:0/1951894200 shutdown_connections 2024-03-06T07:28:13.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.182+0000 7fe3feecf640 1 --2- 172.21.15.83:0/1951894200 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe3dc06b4d0 0x7fe3dc06d990 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.182+0000 7fe3feecf640 1 --2- 172.21.15.83:0/1951894200 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3f815dfe0 0x7fe3f80b25a0 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.182+0000 7fe3feecf640 1 --2- 172.21.15.83:0/1951894200 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3f80d2710 0x7fe3f80b2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.182+0000 7fe3feecf640 1 --2- 172.21.15.83:0/1951894200 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe3f80d22b0 0x7fe3f814f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.182+0000 7fe3feecf640 1 -- 172.21.15.83:0/1951894200 >> 172.21.15.83:0/1951894200 conn(0x7fe3f80d3cd0 msgr2=0x7fe3f80cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:13.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.182+0000 7fe3feecf640 1 -- 172.21.15.83:0/1951894200 shutdown_connections 2024-03-06T07:28:13.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.182+0000 7fe3feecf640 1 -- 172.21.15.83:0/1951894200 wait complete. 2024-03-06T07:28:13.199 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_reintegrated 2024-03-06T07:28:13.315 INFO:teuthology.orchestra.run.smithi083.stdout:{"mds_cache":{"strays_reintegrated":1}} 2024-03-06T07:28:13.315 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 1 } } 2024-03-06T07:28:13.315 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:13.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.414+0000 7f059e116640 1 Processor -- start 2024-03-06T07:28:13.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.414+0000 7f059e116640 1 -- start start 2024-03-06T07:28:13.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.415+0000 7f059e116640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f05980d22b0 0x7f05980d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.415+0000 7f059e116640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f05980b41f0 0x7f05980b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.415+0000 7f059e116640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05980b3430 con 0x7f059815dfe0 2024-03-06T07:28:13.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.415+0000 7f059e116640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05980d2c50 con 0x7f05980b41f0 2024-03-06T07:28:13.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.415+0000 7f059e116640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05980b31e0 con 0x7f05980d22b0 2024-03-06T07:28:13.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.415+0000 7f0597fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f05980b41f0 0x7f05980b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.415+0000 7f0597fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f05980b41f0 0x7f05980b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51334/0 (socket says 172.21.15.83:51334) 2024-03-06T07:28:13.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.415+0000 7f0597fff640 1 -- 172.21.15.83:0/2178504407 learned_addr learned my addr 172.21.15.83:0/2178504407 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:13.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.415+0000 7f0597fff640 1 -- 172.21.15.83:0/2178504407 >> v2:172.21.15.183:3300/0 conn(0x7f05980d22b0 msgr2=0x7f05980d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:13.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.415+0000 7f0596ffd640 1 --2- 172.21.15.83:0/2178504407 >> v2:172.21.15.183:3300/0 conn(0x7f05980d22b0 0x7f05980d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.415+0000 7f0597fff640 1 --2- 172.21.15.83:0/2178504407 >> v2:172.21.15.183:3300/0 conn(0x7f05980d22b0 0x7f05980d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.415+0000 7f0597fff640 1 -- 172.21.15.83:0/2178504407 >> v1:172.21.15.183:6789/0 conn(0x7f059815dfe0 legacy=0x7f05980cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:13.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.415+0000 7f0597fff640 1 -- 172.21.15.83:0/2178504407 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05980b3710 con 0x7f05980b41f0 2024-03-06T07:28:13.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.416+0000 7f0597fff640 1 --2- 172.21.15.83:0/2178504407 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f05980b41f0 0x7f05980b4680 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f0590009e50 tx=0x7f059002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec7bae2da5e5665c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:13.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.416+0000 7f05967fc640 1 -- 172.21.15.83:0/2178504407 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059002ca80 con 0x7f05980b41f0 2024-03-06T07:28:13.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.416+0000 7f05967fc640 1 -- 172.21.15.83:0/2178504407 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f059002cc20 con 0x7f05980b41f0 2024-03-06T07:28:13.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.416+0000 7f05967fc640 1 -- 172.21.15.83:0/2178504407 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05900343f0 con 0x7f05980b41f0 2024-03-06T07:28:13.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.417+0000 7f059e116640 1 -- 172.21.15.83:0/2178504407 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f05980b41f0 msgr2=0x7f05980b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.417+0000 7f059e116640 1 --2- 172.21.15.83:0/2178504407 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f05980b41f0 0x7f05980b4680 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f0590009e50 tx=0x7f059002bee0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.417+0000 7f059e116640 1 -- 172.21.15.83:0/2178504407 shutdown_connections 2024-03-06T07:28:13.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.417+0000 7f059e116640 1 --2- 172.21.15.83:0/2178504407 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f05980b41f0 0x7f05980b4680 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.417+0000 7f059e116640 1 --2- 172.21.15.83:0/2178504407 >> v2:172.21.15.183:3300/0 conn(0x7f05980d22b0 0x7f05980d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.417+0000 7f059e116640 1 -- 172.21.15.83:0/2178504407 >> 172.21.15.83:0/2178504407 conn(0x7f05980d3cd0 msgr2=0x7f05980d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:13.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.417+0000 7f059e116640 1 -- 172.21.15.83:0/2178504407 shutdown_connections 2024-03-06T07:28:13.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.417+0000 7f059e116640 1 -- 172.21.15.83:0/2178504407 wait complete. 2024-03-06T07:28:13.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.418+0000 7f059e116640 1 Processor -- start 2024-03-06T07:28:13.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.418+0000 7f059e116640 1 -- start start 2024-03-06T07:28:13.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.419+0000 7f059e116640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f05980b41f0 0x7f05981f3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.419+0000 7f059e116640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f05980d22b0 0x7f059815bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.419+0000 7f059e116640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f059815dfe0 0x7f05981600c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.419+0000 7f059e116640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05980cfa00 con 0x7f05980b41f0 2024-03-06T07:28:13.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.419+0000 7f059e116640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05980b3d60 con 0x7f059815dfe0 2024-03-06T07:28:13.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.419+0000 7f059e116640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05980d1c70 con 0x7f05980d22b0 2024-03-06T07:28:13.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.419+0000 7f05977fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f05980b41f0 0x7f05981f3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.419+0000 7f05977fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f05980b41f0 0x7f05981f3b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51338/0 (socket says 172.21.15.83:51338) 2024-03-06T07:28:13.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.419+0000 7f05977fe640 1 -- 172.21.15.83:0/3910359143 learned_addr learned my addr 172.21.15.83:0/3910359143 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:13.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.419+0000 7f0597fff640 1 --2- 172.21.15.83:0/3910359143 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f059815dfe0 0x7f05981600c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.419+0000 7f0596ffd640 1 --2- 172.21.15.83:0/3910359143 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f05980d22b0 0x7f059815bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.419+0000 7f05977fe640 1 -- 172.21.15.83:0/3910359143 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f05980d22b0 msgr2=0x7f059815bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.419+0000 7f05977fe640 1 --2- 172.21.15.83:0/3910359143 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f05980d22b0 0x7f059815bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.419+0000 7f05977fe640 1 -- 172.21.15.83:0/3910359143 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f059815dfe0 msgr2=0x7f05981600c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.419+0000 7f05977fe640 1 --2- 172.21.15.83:0/3910359143 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f059815dfe0 0x7f05981600c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.419+0000 7f05977fe640 1 -- 172.21.15.83:0/3910359143 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0598160600 con 0x7f05980b41f0 2024-03-06T07:28:13.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.419+0000 7f0597fff640 1 --2- 172.21.15.83:0/3910359143 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f059815dfe0 0x7f05981600c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:13.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.419+0000 7f0596ffd640 1 --2- 172.21.15.83:0/3910359143 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f05980d22b0 0x7f059815bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:13.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.419+0000 7f05977fe640 1 --2- 172.21.15.83:0/3910359143 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f05980b41f0 0x7f05981f3b10 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f058800b3e0 tx=0x7f058800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:13.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.419+0000 7f057ffff640 1 -- 172.21.15.83:0/3910359143 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0588015020 con 0x7f05980b41f0 2024-03-06T07:28:13.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.419+0000 7f057ffff640 1 -- 172.21.15.83:0/3910359143 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0588011850 con 0x7f05980b41f0 2024-03-06T07:28:13.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.419+0000 7f059e116640 1 -- 172.21.15.83:0/3910359143 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05981f4140 con 0x7f05980b41f0 2024-03-06T07:28:13.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.419+0000 7f057ffff640 1 -- 172.21.15.83:0/3910359143 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0588011c40 con 0x7f05980b41f0 2024-03-06T07:28:13.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.420+0000 7f059e116640 1 -- 172.21.15.83:0/3910359143 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05981f4700 con 0x7f05980b41f0 2024-03-06T07:28:13.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.420+0000 7f059e116640 1 -- 172.21.15.83:0/3910359143 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05981f4a40 con 0x7f05980b41f0 2024-03-06T07:28:13.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.421+0000 7f057ffff640 1 -- 172.21.15.83:0/3910359143 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0588018020 con 0x7f05980b41f0 2024-03-06T07:28:13.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.421+0000 7f057ffff640 1 --2- 172.21.15.83:0/3910359143 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f058406b4d0 0x7f058406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.422+0000 7f057ffff640 1 -- 172.21.15.83:0/3910359143 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f058808a210 con 0x7f05980b41f0 2024-03-06T07:28:13.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.422+0000 7f0596ffd640 1 --2- 172.21.15.83:0/3910359143 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f058406b4d0 0x7f058406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.422+0000 7f0596ffd640 1 --2- 172.21.15.83:0/3910359143 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f058406b4d0 0x7f058406d990 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f05980a13f0 tx=0x7f058c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:13.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.426+0000 7f057ffff640 1 -- 172.21.15.83:0/3910359143 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f05981f4a40 con 0x7f05980b41f0 2024-03-06T07:28:13.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.641+0000 7f059e116640 1 -- 172.21.15.83:0/3910359143 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f059815d2b0 con 0x7f05980b41f0 2024-03-06T07:28:13.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.641+0000 7f057ffff640 1 -- 172.21.15.83:0/3910359143 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f058801f020 con 0x7f05980b41f0 2024-03-06T07:28:13.644 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:28:13.644 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7051,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/101642171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":101642171},{"type":"v1","addr":"172.21.15.183:6833","nonce":101642171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7057,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3146884900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":3146884900},{"type":"v1","addr":"172.21.15.183:6835","nonce":3146884900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8208,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1922041621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1922041621},{"type":"v1","addr":"172.21.15.83:6835","nonce":1922041621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.491022+0000","modified":"2024-03-06T07:27:37.006649+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8229":{"gid":8229,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/2003687153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2003687153},{"type":"v1","addr":"172.21.15.83:6837","nonce":2003687153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8229,"qdb_cluster":[8229]},"id":8}]} 2024-03-06T07:28:13.645 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 66 2024-03-06T07:28:13.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.643+0000 7f059e116640 1 -- 172.21.15.83:0/3910359143 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f058406b4d0 msgr2=0x7f058406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.643+0000 7f059e116640 1 --2- 172.21.15.83:0/3910359143 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f058406b4d0 0x7f058406d990 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f05980a13f0 tx=0x7f058c008040 comp rx=0 tx=0).stop 2024-03-06T07:28:13.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.643+0000 7f059e116640 1 -- 172.21.15.83:0/3910359143 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f05980b41f0 msgr2=0x7f05981f3b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.643+0000 7f059e116640 1 --2- 172.21.15.83:0/3910359143 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f05980b41f0 0x7f05981f3b10 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f058800b3e0 tx=0x7f058800b8a0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.643+0000 7f059e116640 1 -- 172.21.15.83:0/3910359143 shutdown_connections 2024-03-06T07:28:13.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.643+0000 7f059e116640 1 --2- 172.21.15.83:0/3910359143 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f058406b4d0 0x7f058406d990 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.643+0000 7f059e116640 1 --2- 172.21.15.83:0/3910359143 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f059815dfe0 0x7f05981600c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.643+0000 7f059e116640 1 --2- 172.21.15.83:0/3910359143 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f05980d22b0 0x7f059815bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.643+0000 7f059e116640 1 --2- 172.21.15.83:0/3910359143 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f05980b41f0 0x7f05981f3b10 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.643+0000 7f059e116640 1 -- 172.21.15.83:0/3910359143 >> 172.21.15.83:0/3910359143 conn(0x7f05980d3cd0 msgr2=0x7f059806ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:13.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.643+0000 7f059e116640 1 -- 172.21.15.83:0/3910359143 shutdown_connections 2024-03-06T07:28:13.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.643+0000 7f059e116640 1 -- 172.21.15.83:0/3910359143 wait complete. 2024-03-06T07:28:13.658 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:28:13.785 INFO:teuthology.orchestra.run.smithi083.stdout:{"message":"","return_code":0} 2024-03-06T07:28:13.785 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:28:13.785 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-06T07:28:13.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.825+0000 7f7a919faf40 1 Processor -- start 2024-03-06T07:28:13.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.825+0000 7f7a919faf40 1 -- start start 2024-03-06T07:28:13.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.825+0000 7f7a919faf40 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a78003ab0 0x5608bcab6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.825+0000 7f7a919faf40 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5608bcb76810 0x5608bcc1ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.825+0000 7f7a919faf40 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x5608bcb70840 0x5608bcc20500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.825+0000 7f7a919faf40 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608bca5fb70 con 0x7f7a78003ab0 2024-03-06T07:28:13.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.825+0000 7f7a919faf40 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608bca63ad0 con 0x5608bcb76810 2024-03-06T07:28:13.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.825+0000 7f7a919faf40 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x5608bcae8b70 con 0x5608bcb70840 2024-03-06T07:28:13.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.825+0000 7f7a8f4fb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a78003ab0 0x5608bcab6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.825+0000 7f7a8f4fb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a78003ab0 0x5608bcab6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51344/0 (socket says 172.21.15.83:51344) 2024-03-06T07:28:13.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.825+0000 7f7a8f4fb640 1 -- 172.21.15.83:0/1586327268 learned_addr learned my addr 172.21.15.83:0/1586327268 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:13.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.826+0000 7f7a8ecfa640 1 --2- 172.21.15.83:0/1586327268 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5608bcb76810 0x5608bcc1ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.826+0000 7f7a8fcfc640 1 --2- 172.21.15.83:0/1586327268 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x5608bcb70840 0x5608bcc20500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.826+0000 7f7a8f4fb640 1 -- 172.21.15.83:0/1586327268 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x5608bcb70840 msgr2=0x5608bcc20500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:13.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.826+0000 7f7a8f4fb640 1 --2- 172.21.15.83:0/1586327268 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x5608bcb70840 0x5608bcc20500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.826+0000 7f7a8f4fb640 1 -- 172.21.15.83:0/1586327268 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5608bcb76810 msgr2=0x5608bcc1ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:13.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.826+0000 7f7a8f4fb640 1 --2- 172.21.15.83:0/1586327268 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5608bcb76810 0x5608bcc1ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.826+0000 7f7a8f4fb640 1 -- 172.21.15.83:0/1586327268 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5608bcb62f90 con 0x7f7a78003ab0 2024-03-06T07:28:13.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.826+0000 7f7a8ecfa640 1 --2- 172.21.15.83:0/1586327268 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5608bcb76810 0x5608bcc1ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:13.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.826+0000 7f7a8f4fb640 1 --2- 172.21.15.83:0/1586327268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a78003ab0 0x5608bcab6e10 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f7a80002ec0 tx=0x7f7a8000e6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2896157a886091d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:13.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.826+0000 7f7a8dcf8640 1 -- 172.21.15.83:0/1586327268 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a80015020 con 0x7f7a78003ab0 2024-03-06T07:28:13.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.826+0000 7f7a8dcf8640 1 -- 172.21.15.83:0/1586327268 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a8000ee70 con 0x7f7a78003ab0 2024-03-06T07:28:13.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.826+0000 7f7a8dcf8640 1 -- 172.21.15.83:0/1586327268 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a80011650 con 0x7f7a78003ab0 2024-03-06T07:28:13.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.826+0000 7f7a919faf40 1 -- 172.21.15.83:0/1586327268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a78003ab0 msgr2=0x5608bcab6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.826+0000 7f7a919faf40 1 --2- 172.21.15.83:0/1586327268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a78003ab0 0x5608bcab6e10 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f7a80002ec0 tx=0x7f7a8000e6a0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.826+0000 7f7a919faf40 1 -- 172.21.15.83:0/1586327268 shutdown_connections 2024-03-06T07:28:13.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.826+0000 7f7a919faf40 1 --2- 172.21.15.83:0/1586327268 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x5608bcb70840 0x5608bcc20500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.826+0000 7f7a919faf40 1 --2- 172.21.15.83:0/1586327268 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5608bcb76810 0x5608bcc1ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.826+0000 7f7a919faf40 1 --2- 172.21.15.83:0/1586327268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a78003ab0 0x5608bcab6e10 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.826+0000 7f7a919faf40 1 -- 172.21.15.83:0/1586327268 >> 172.21.15.83:0/1586327268 conn(0x5608bcae4350 msgr2=0x5608bcb007b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:13.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.827+0000 7f7a919faf40 1 -- 172.21.15.83:0/1586327268 shutdown_connections 2024-03-06T07:28:13.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.827+0000 7f7a919faf40 1 -- 172.21.15.83:0/1586327268 wait complete. 2024-03-06T07:28:13.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.827+0000 7f7a919faf40 1 Processor -- start 2024-03-06T07:28:13.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.827+0000 7f7a919faf40 1 -- start start 2024-03-06T07:28:13.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.828+0000 7f7a919faf40 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5608bcb70840 0x5608bcb8d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.828+0000 7f7a919faf40 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x5608bcb76810 0x5608bcafab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.828+0000 7f7a919faf40 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7a78003ab0 0x5608bcafb060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.828+0000 7f7a919faf40 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608bcaf9580 con 0x5608bcb70840 2024-03-06T07:28:13.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.828+0000 7f7a919faf40 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608bcaf9820 con 0x7f7a78003ab0 2024-03-06T07:28:13.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.828+0000 7f7a919faf40 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x5608bcaf9ae0 con 0x5608bcb76810 2024-03-06T07:28:13.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.828+0000 7f7a8f4fb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5608bcb70840 0x5608bcb8d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.828+0000 7f7a8f4fb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5608bcb70840 0x5608bcb8d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51350/0 (socket says 172.21.15.83:51350) 2024-03-06T07:28:13.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.828+0000 7f7a8f4fb640 1 -- 172.21.15.83:0/2479634028 learned_addr learned my addr 172.21.15.83:0/2479634028 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:13.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.828+0000 7f7a8fcfc640 1 --2- 172.21.15.83:0/2479634028 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7a78003ab0 0x5608bcafb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.828+0000 7f7a8ecfa640 1 --2- 172.21.15.83:0/2479634028 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x5608bcb76810 0x5608bcafab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.828+0000 7f7a8f4fb640 1 -- 172.21.15.83:0/2479634028 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x5608bcb76810 msgr2=0x5608bcafab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.828+0000 7f7a8f4fb640 1 --2- 172.21.15.83:0/2479634028 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x5608bcb76810 0x5608bcafab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.828+0000 7f7a8f4fb640 1 -- 172.21.15.83:0/2479634028 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7a78003ab0 msgr2=0x5608bcafb060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.828+0000 7f7a8f4fb640 1 --2- 172.21.15.83:0/2479634028 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7a78003ab0 0x5608bcafb060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.828+0000 7f7a8f4fb640 1 -- 172.21.15.83:0/2479634028 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5608bcae6d10 con 0x5608bcb70840 2024-03-06T07:28:13.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.828+0000 7f7a8f4fb640 1 --2- 172.21.15.83:0/2479634028 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5608bcb70840 0x5608bcb8d6d0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f7a80002e60 tx=0x7f7a80001190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:13.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.828+0000 7f7a8ccf6640 1 -- 172.21.15.83:0/2479634028 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8002a070 con 0x5608bcb70840 2024-03-06T07:28:13.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.829+0000 7f7a8ccf6640 1 -- 172.21.15.83:0/2479634028 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a80000f50 con 0x5608bcb70840 2024-03-06T07:28:13.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.829+0000 7f7a919faf40 1 -- 172.21.15.83:0/2479634028 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5608bcb717f0 con 0x5608bcb70840 2024-03-06T07:28:13.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.829+0000 7f7a8ccf6640 1 -- 172.21.15.83:0/2479634028 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a80015040 con 0x5608bcb70840 2024-03-06T07:28:13.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.829+0000 7f7a919faf40 1 -- 172.21.15.83:0/2479634028 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5608bcb71db0 con 0x5608bcb70840 2024-03-06T07:28:13.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.830+0000 7f7a8ccf6640 1 -- 172.21.15.83:0/2479634028 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7a80018020 con 0x5608bcb70840 2024-03-06T07:28:13.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.830+0000 7f7a8ccf6640 1 --2- 172.21.15.83:0/2479634028 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7a7c069eb0 0x7f7a7c06c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.830+0000 7f7a8ccf6640 1 -- 172.21.15.83:0/2479634028 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f7a8000b020 con 0x5608bcb70840 2024-03-06T07:28:13.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.830+0000 7f7a8ecfa640 1 --2- 172.21.15.83:0/2479634028 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7a7c069eb0 0x7f7a7c06c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.830+0000 7f7a919faf40 1 --2- 172.21.15.83:0/2479634028 >> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] conn(0x5608bcafcfd0 0x5608bcafd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.830+0000 7f7a919faf40 1 -- 172.21.15.83:0/2479634028 --> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e116) v9 -- 0x5608bcae68c0 con 0x5608bcafcfd0 2024-03-06T07:28:13.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.831+0000 7f7a8fcfc640 1 --2- 172.21.15.83:0/2479634028 >> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] conn(0x5608bcafcfd0 0x5608bcafd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.831+0000 7f7a8ecfa640 1 --2- 172.21.15.83:0/2479634028 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7a7c069eb0 0x7f7a7c06c370 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f7a840017e0 tx=0x7f7a84001e10 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:13.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.831+0000 7f7a8fcfc640 1 --2- 172.21.15.83:0/2479634028 >> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] conn(0x5608bcafcfd0 0x5608bcafd460 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:13.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.831+0000 7f7a8fcfc640 1 -- 172.21.15.83:0/2479634028 <== osd.0 v2:172.21.15.83:6808/1635427425 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (93) out=93b] v0'0 uv10 ondisk = 0) v8 ==== 164+0+93 (crc 0 0 0) 0x7f7a88005040 con 0x5608bcafcfd0 2024-03-06T07:28:13.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.831+0000 7f7a919faf40 1 -- 172.21.15.83:0/2479634028 >> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] conn(0x5608bcafcfd0 msgr2=0x5608bcafd460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.831+0000 7f7a919faf40 1 --2- 172.21.15.83:0/2479634028 >> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] conn(0x5608bcafcfd0 0x5608bcafd460 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.831+0000 7f7a919faf40 1 -- 172.21.15.83:0/2479634028 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7a7c069eb0 msgr2=0x7f7a7c06c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.831+0000 7f7a919faf40 1 --2- 172.21.15.83:0/2479634028 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7a7c069eb0 0x7f7a7c06c370 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f7a840017e0 tx=0x7f7a84001e10 comp rx=0 tx=0).stop 2024-03-06T07:28:13.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.831+0000 7f7a919faf40 1 -- 172.21.15.83:0/2479634028 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5608bcb70840 msgr2=0x5608bcb8d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.831+0000 7f7a919faf40 1 --2- 172.21.15.83:0/2479634028 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5608bcb70840 0x5608bcb8d6d0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f7a80002e60 tx=0x7f7a80001190 comp rx=0 tx=0).stop 2024-03-06T07:28:13.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.831+0000 7f7a8fcfc640 1 -- 172.21.15.83:0/2479634028 reap_dead start 2024-03-06T07:28:13.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.832+0000 7f7a919faf40 1 -- 172.21.15.83:0/2479634028 shutdown_connections 2024-03-06T07:28:13.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.832+0000 7f7a919faf40 1 -- 172.21.15.83:0/2479634028 >> 172.21.15.83:0/2479634028 conn(0x5608bcae4350 msgr2=0x5608bcafec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:13.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.832+0000 7f7a919faf40 1 -- 172.21.15.83:0/2479634028 shutdown_connections 2024-03-06T07:28:13.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:13.832+0000 7f7a919faf40 1 -- 172.21.15.83:0/2479634028 wait complete. 2024-03-06T07:28:13.841 INFO:teuthology.orchestra.run.smithi083.stdout:W�file_cdir_2 2024-03-06T07:28:13.841 DEBUG:teuthology.orchestra.run.smithi083:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-06T07:28:13.926 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:28:13.926 INFO:teuthology.orchestra.run.smithi083.stdout: "ino": 1099511627777, 2024-03-06T07:28:13.926 INFO:teuthology.orchestra.run.smithi083.stdout: "ancestors": [ 2024-03-06T07:28:13.926 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:28:13.927 INFO:teuthology.orchestra.run.smithi083.stdout: "dirino": 1099511628278, 2024-03-06T07:28:13.927 INFO:teuthology.orchestra.run.smithi083.stdout: "dname": "file_c", 2024-03-06T07:28:13.927 INFO:teuthology.orchestra.run.smithi083.stdout: "version": 15 2024-03-06T07:28:13.927 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:28:13.927 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:28:13.927 INFO:teuthology.orchestra.run.smithi083.stdout: "dirino": 1, 2024-03-06T07:28:13.927 INFO:teuthology.orchestra.run.smithi083.stdout: "dname": "dir_2", 2024-03-06T07:28:13.927 INFO:teuthology.orchestra.run.smithi083.stdout: "version": 31 2024-03-06T07:28:13.927 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:28:13.927 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2024-03-06T07:28:13.927 INFO:teuthology.orchestra.run.smithi083.stdout: "pool": 17, 2024-03-06T07:28:13.927 INFO:teuthology.orchestra.run.smithi083.stdout: "old_pools": [] 2024-03-06T07:28:13.927 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:28:13.927 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:28:13.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:13.929 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln dir_2/file_c dir_2/file_d) 2024-03-06T07:28:13.987 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:13.987 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_2/file_c) 2024-03-06T07:28:14.046 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:14.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.185+0000 7fef2335e640 1 Processor -- start 2024-03-06T07:28:14.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.185+0000 7fef2335e640 1 -- start start 2024-03-06T07:28:14.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.186+0000 7fef2335e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fef1c15dfe0 0x7fef1c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.186+0000 7fef2335e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fef1c0d29c0 0x7fef1c0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.186+0000 7fef2335e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef1c0b3430 con 0x7fef1c0cf830 2024-03-06T07:28:14.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.186+0000 7fef2335e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef1c0b31e0 con 0x7fef1c15dfe0 2024-03-06T07:28:14.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.186+0000 7fef2335e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef1c0b3d60 con 0x7fef1c0d29c0 2024-03-06T07:28:14.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.187+0000 7fef210fa640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fef1c15dfe0 0x7fef1c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.187+0000 7fef218fb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fef1c0d29c0 0x7fef1c0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.187+0000 7fef218fb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fef1c0d29c0 0x7fef1c0b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:54342/0 (socket says 172.21.15.83:54342) 2024-03-06T07:28:14.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.187+0000 7fef218fb640 1 -- 172.21.15.83:0/409369270 learned_addr learned my addr 172.21.15.83:0/409369270 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:14.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.187+0000 7fef210fa640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fef1c15dfe0 0x7fef1c0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:45320/0 (socket says 172.21.15.83:45320) 2024-03-06T07:28:14.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.187+0000 7fef208f9640 1 --1- 172.21.15.83:0/409369270 >> v1:172.21.15.183:6789/0 conn(0x7fef1c0cf830 0x7fef1c0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:54632/0 (socket says 172.21.15.83:54632) 2024-03-06T07:28:14.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.187+0000 7fef218fb640 1 -- 172.21.15.83:0/409369270 >> v2:172.21.15.183:3300/0 conn(0x7fef1c15dfe0 msgr2=0x7fef1c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:14.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.187+0000 7fef218fb640 1 --2- 172.21.15.83:0/409369270 >> v2:172.21.15.183:3300/0 conn(0x7fef1c15dfe0 0x7fef1c0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.187+0000 7fef218fb640 1 -- 172.21.15.83:0/409369270 >> v1:172.21.15.183:6789/0 conn(0x7fef1c0cf830 legacy=0x7fef1c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.187+0000 7fef218fb640 1 -- 172.21.15.83:0/409369270 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef1c0b3710 con 0x7fef1c0d29c0 2024-03-06T07:28:14.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.187+0000 7fef210fa640 1 --2- 172.21.15.83:0/409369270 >> v2:172.21.15.183:3300/0 conn(0x7fef1c15dfe0 0x7fef1c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:14.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.188+0000 7fef218fb640 1 --2- 172.21.15.83:0/409369270 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fef1c0d29c0 0x7fef1c0b41f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fef14009e50 tx=0x7fef1402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db6dbb4a1492b49b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:14.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.188+0000 7fef13fff640 1 -- 172.21.15.83:0/409369270 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef1402cb00 con 0x7fef1c0d29c0 2024-03-06T07:28:14.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.188+0000 7fef13fff640 1 -- 172.21.15.83:0/409369270 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef1402cca0 con 0x7fef1c0d29c0 2024-03-06T07:28:14.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.188+0000 7fef13fff640 1 -- 172.21.15.83:0/409369270 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef14034610 con 0x7fef1c0d29c0 2024-03-06T07:28:14.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.188+0000 7fef2335e640 1 -- 172.21.15.83:0/409369270 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fef1c0d29c0 msgr2=0x7fef1c0b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.188+0000 7fef2335e640 1 --2- 172.21.15.83:0/409369270 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fef1c0d29c0 0x7fef1c0b41f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fef14009e50 tx=0x7fef1402bf60 comp rx=0 tx=0).stop 2024-03-06T07:28:14.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.188+0000 7fef2335e640 1 -- 172.21.15.83:0/409369270 shutdown_connections 2024-03-06T07:28:14.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.188+0000 7fef2335e640 1 --2- 172.21.15.83:0/409369270 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fef1c0d29c0 0x7fef1c0b41f0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.188+0000 7fef2335e640 1 --2- 172.21.15.83:0/409369270 >> v2:172.21.15.183:3300/0 conn(0x7fef1c15dfe0 0x7fef1c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.188+0000 7fef2335e640 1 -- 172.21.15.83:0/409369270 >> 172.21.15.83:0/409369270 conn(0x7fef1c0d3cd0 msgr2=0x7fef1c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:14.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.189+0000 7fef2335e640 1 -- 172.21.15.83:0/409369270 shutdown_connections 2024-03-06T07:28:14.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.189+0000 7fef2335e640 1 -- 172.21.15.83:0/409369270 wait complete. 2024-03-06T07:28:14.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.190+0000 7fef2335e640 1 Processor -- start 2024-03-06T07:28:14.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.190+0000 7fef2335e640 1 -- start start 2024-03-06T07:28:14.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.190+0000 7fef2335e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef1c0cf830 0x7fef1c06ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.190+0000 7fef2335e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fef1c0d29c0 0x7fef1c0b9120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.190+0000 7fef2335e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef1c15dfe0 0x7fef1c14ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.190+0000 7fef2335e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef1c0d1df0 con 0x7fef1c15dfe0 2024-03-06T07:28:14.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.190+0000 7fef2335e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef1c0d1c70 con 0x7fef1c0cf830 2024-03-06T07:28:14.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.190+0000 7fef2335e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef1c15e3c0 con 0x7fef1c0d29c0 2024-03-06T07:28:14.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.190+0000 7fef218fb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef1c15dfe0 0x7fef1c14ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.190+0000 7fef218fb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef1c15dfe0 0x7fef1c14ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51362/0 (socket says 172.21.15.83:51362) 2024-03-06T07:28:14.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.190+0000 7fef218fb640 1 -- 172.21.15.83:0/3385767910 learned_addr learned my addr 172.21.15.83:0/3385767910 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:14.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.191+0000 7fef208f9640 1 --2- 172.21.15.83:0/3385767910 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fef1c0d29c0 0x7fef1c0b9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.191+0000 7fef210fa640 1 --2- 172.21.15.83:0/3385767910 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef1c0cf830 0x7fef1c06ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.191+0000 7fef218fb640 1 -- 172.21.15.83:0/3385767910 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fef1c0d29c0 msgr2=0x7fef1c0b9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.191+0000 7fef218fb640 1 --2- 172.21.15.83:0/3385767910 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fef1c0d29c0 0x7fef1c0b9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.191+0000 7fef218fb640 1 -- 172.21.15.83:0/3385767910 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef1c0cf830 msgr2=0x7fef1c06ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.191+0000 7fef218fb640 1 --2- 172.21.15.83:0/3385767910 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef1c0cf830 0x7fef1c06ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.191+0000 7fef218fb640 1 -- 172.21.15.83:0/3385767910 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef1c150ba0 con 0x7fef1c15dfe0 2024-03-06T07:28:14.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.191+0000 7fef218fb640 1 --2- 172.21.15.83:0/3385767910 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef1c15dfe0 0x7fef1c14ed10 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7fef14036040 tx=0x7fef14004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:14.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.191+0000 7fef11ffb640 1 -- 172.21.15.83:0/3385767910 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef14043020 con 0x7fef1c15dfe0 2024-03-06T07:28:14.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.191+0000 7fef11ffb640 1 -- 172.21.15.83:0/3385767910 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef1402ce50 con 0x7fef1c15dfe0 2024-03-06T07:28:14.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.191+0000 7fef11ffb640 1 -- 172.21.15.83:0/3385767910 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef14009510 con 0x7fef1c15dfe0 2024-03-06T07:28:14.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.191+0000 7fef2335e640 1 -- 172.21.15.83:0/3385767910 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef1c150e80 con 0x7fef1c15dfe0 2024-03-06T07:28:14.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.191+0000 7fef2335e640 1 -- 172.21.15.83:0/3385767910 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef1c06d350 con 0x7fef1c15dfe0 2024-03-06T07:28:14.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.192+0000 7fef2335e640 1 -- 172.21.15.83:0/3385767910 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef1c06bf00 con 0x7fef1c15dfe0 2024-03-06T07:28:14.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.193+0000 7fef11ffb640 1 -- 172.21.15.83:0/3385767910 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fef14046020 con 0x7fef1c15dfe0 2024-03-06T07:28:14.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.193+0000 7fef11ffb640 1 --2- 172.21.15.83:0/3385767910 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fef0006b4d0 0x7fef0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.193+0000 7fef11ffb640 1 -- 172.21.15.83:0/3385767910 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fef140ac8f0 con 0x7fef1c15dfe0 2024-03-06T07:28:14.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.193+0000 7fef210fa640 1 --2- 172.21.15.83:0/3385767910 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fef0006b4d0 0x7fef0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.194+0000 7fef210fa640 1 --2- 172.21.15.83:0/3385767910 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fef0006b4d0 0x7fef0006d990 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fef18009560 tx=0x7fef18009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:14.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.198+0000 7fef11ffb640 1 -- 172.21.15.83:0/3385767910 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fef1c06bf00 con 0x7fef1c15dfe0 2024-03-06T07:28:14.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.443+0000 7fef2335e640 1 -- 172.21.15.83:0/3385767910 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef1c06d520 con 0x7fef1c15dfe0 2024-03-06T07:28:14.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.443+0000 7fef11ffb640 1 -- 172.21.15.83:0/3385767910 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4984 (secure 0 0 0) 0x7fef1404d030 con 0x7fef1c15dfe0 2024-03-06T07:28:14.446 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:28:14.447 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7051,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/101642171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":101642171},{"type":"v1","addr":"172.21.15.183:6833","nonce":101642171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7057,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3146884900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":3146884900},{"type":"v1","addr":"172.21.15.183:6835","nonce":3146884900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8208,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1922041621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1922041621},{"type":"v1","addr":"172.21.15.83:6835","nonce":1922041621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.491022+0000","modified":"2024-03-06T07:27:37.006649+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8229":{"gid":8229,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/2003687153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2003687153},{"type":"v1","addr":"172.21.15.83:6837","nonce":2003687153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8229,"qdb_cluster":[8229]},"id":8}]} 2024-03-06T07:28:14.447 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 66 2024-03-06T07:28:14.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.445+0000 7fef2335e640 1 -- 172.21.15.83:0/3385767910 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fef0006b4d0 msgr2=0x7fef0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.445+0000 7fef2335e640 1 --2- 172.21.15.83:0/3385767910 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fef0006b4d0 0x7fef0006d990 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fef18009560 tx=0x7fef18009190 comp rx=0 tx=0).stop 2024-03-06T07:28:14.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.445+0000 7fef2335e640 1 -- 172.21.15.83:0/3385767910 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef1c15dfe0 msgr2=0x7fef1c14ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.445+0000 7fef2335e640 1 --2- 172.21.15.83:0/3385767910 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef1c15dfe0 0x7fef1c14ed10 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7fef14036040 tx=0x7fef14004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:14.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.445+0000 7fef2335e640 1 -- 172.21.15.83:0/3385767910 shutdown_connections 2024-03-06T07:28:14.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.445+0000 7fef2335e640 1 --2- 172.21.15.83:0/3385767910 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fef0006b4d0 0x7fef0006d990 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.445+0000 7fef2335e640 1 --2- 172.21.15.83:0/3385767910 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef1c15dfe0 0x7fef1c14ed10 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.445+0000 7fef2335e640 1 --2- 172.21.15.83:0/3385767910 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fef1c0d29c0 0x7fef1c0b9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.445+0000 7fef2335e640 1 --2- 172.21.15.83:0/3385767910 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef1c0cf830 0x7fef1c06ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.445+0000 7fef2335e640 1 -- 172.21.15.83:0/3385767910 >> 172.21.15.83:0/3385767910 conn(0x7fef1c0d3cd0 msgr2=0x7fef1c06b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:14.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.445+0000 7fef2335e640 1 -- 172.21.15.83:0/3385767910 shutdown_connections 2024-03-06T07:28:14.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:14.446+0000 7fef2335e640 1 -- 172.21.15.83:0/3385767910 wait complete. 2024-03-06T07:28:14.459 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:28:14.576 INFO:teuthology.orchestra.run.smithi083.stdout:{"mds_cache":{"num_strays":1}} 2024-03-06T07:28:14.576 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 1 } } 2024-03-06T07:28:14.577 DEBUG:tasks.ceph_test_case:wait_until_equal: 1 != 0, waiting (timeout=60)... 2024-03-06T07:28:19.578 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:19.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.677+0000 7f94f5a0e640 1 Processor -- start 2024-03-06T07:28:19.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.677+0000 7f94f5a0e640 1 -- start start 2024-03-06T07:28:19.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.677+0000 7f94f5a0e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f94f00b3060 0x7f94f015e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.677+0000 7f94f5a0e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f94f015bf30 0x7f94f015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.677+0000 7f94f5a0e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94f015f2d0 con 0x7f94f013dc10 2024-03-06T07:28:19.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.678+0000 7f94f5a0e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94f00cf3f0 con 0x7f94f015bf30 2024-03-06T07:28:19.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.678+0000 7f94f5a0e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94f015b210 con 0x7f94f00b3060 2024-03-06T07:28:19.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.678+0000 7f94ef7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f94f015bf30 0x7f94f015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.678+0000 7f94ef7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f94f015bf30 0x7f94f015ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33740/0 (socket says 172.21.15.83:33740) 2024-03-06T07:28:19.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.678+0000 7f94eeffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f94f013dc10 0x7f94f00b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:58240/0 (socket says 172.21.15.83:58240) 2024-03-06T07:28:19.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.678+0000 7f94ef7fe640 1 -- 172.21.15.83:0/3326433845 learned_addr learned my addr 172.21.15.83:0/3326433845 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:19.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.678+0000 7f94ef7fe640 1 -- 172.21.15.83:0/3326433845 >> v2:172.21.15.183:3300/0 conn(0x7f94f00b3060 msgr2=0x7f94f015e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:19.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.678+0000 7f94ef7fe640 1 --2- 172.21.15.83:0/3326433845 >> v2:172.21.15.183:3300/0 conn(0x7f94f00b3060 0x7f94f015e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.678+0000 7f94ef7fe640 1 -- 172.21.15.83:0/3326433845 >> v1:172.21.15.83:6789/0 conn(0x7f94f013dc10 legacy=0x7f94f00b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.678+0000 7f94ef7fe640 1 -- 172.21.15.83:0/3326433845 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f015f580 con 0x7f94f015bf30 2024-03-06T07:28:19.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.679+0000 7f94ef7fe640 1 --2- 172.21.15.83:0/3326433845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94f015bf30 0x7f94f015ed90 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f94f0068700 tx=0x7f94e802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e385a539285861f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.679+0000 7f94ee7fc640 1 -- 172.21.15.83:0/3326433845 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e802ca80 con 0x7f94f015bf30 2024-03-06T07:28:19.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.679+0000 7f94ee7fc640 1 -- 172.21.15.83:0/3326433845 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94e802cc20 con 0x7f94f015bf30 2024-03-06T07:28:19.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.679+0000 7f94ee7fc640 1 -- 172.21.15.83:0/3326433845 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e80343f0 con 0x7f94f015bf30 2024-03-06T07:28:19.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.679+0000 7f94f5a0e640 1 -- 172.21.15.83:0/3326433845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94f015bf30 msgr2=0x7f94f015ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.679+0000 7f94f5a0e640 1 --2- 172.21.15.83:0/3326433845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94f015bf30 0x7f94f015ed90 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f94f0068700 tx=0x7f94e802bee0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.679+0000 7f94f5a0e640 1 -- 172.21.15.83:0/3326433845 shutdown_connections 2024-03-06T07:28:19.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.679+0000 7f94f5a0e640 1 --2- 172.21.15.83:0/3326433845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94f015bf30 0x7f94f015ed90 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.679+0000 7f94f5a0e640 1 --2- 172.21.15.83:0/3326433845 >> v2:172.21.15.183:3300/0 conn(0x7f94f00b3060 0x7f94f015e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.679+0000 7f94f5a0e640 1 -- 172.21.15.83:0/3326433845 >> 172.21.15.83:0/3326433845 conn(0x7f94f00cf8a0 msgr2=0x7f94f00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:19.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.679+0000 7f94f5a0e640 1 -- 172.21.15.83:0/3326433845 shutdown_connections 2024-03-06T07:28:19.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.679+0000 7f94f5a0e640 1 -- 172.21.15.83:0/3326433845 wait complete. 2024-03-06T07:28:19.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.681+0000 7f94f5a0e640 1 Processor -- start 2024-03-06T07:28:19.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.681+0000 7f94f5a0e640 1 -- start start 2024-03-06T07:28:19.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.682+0000 7f94f5a0e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f94f00b3060 0x7f94f006db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.682+0000 7f94f5a0e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f94f013dc10 0x7f94f01511e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.682+0000 7f94f5a0e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94f015bf30 0x7f94f0151720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.682+0000 7f94f5a0e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f013dff0 con 0x7f94f015bf30 2024-03-06T07:28:19.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.682+0000 7f94f5a0e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f015b730 con 0x7f94f00b3060 2024-03-06T07:28:19.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.682+0000 7f94f5a0e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94f00708d0 con 0x7f94f013dc10 2024-03-06T07:28:19.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.682+0000 7f94ef7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94f015bf30 0x7f94f0151720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.682+0000 7f94ef7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94f015bf30 0x7f94f0151720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33748/0 (socket says 172.21.15.83:33748) 2024-03-06T07:28:19.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.682+0000 7f94ef7fe640 1 -- 172.21.15.83:0/3525702287 learned_addr learned my addr 172.21.15.83:0/3525702287 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:19.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.682+0000 7f94e7fff640 1 --2- 172.21.15.83:0/3525702287 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f94f013dc10 0x7f94f01511e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.682+0000 7f94eeffd640 1 --2- 172.21.15.83:0/3525702287 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f94f00b3060 0x7f94f006db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.682+0000 7f94ef7fe640 1 -- 172.21.15.83:0/3525702287 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f94f013dc10 msgr2=0x7f94f01511e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.682+0000 7f94ef7fe640 1 --2- 172.21.15.83:0/3525702287 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f94f013dc10 0x7f94f01511e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.682+0000 7f94ef7fe640 1 -- 172.21.15.83:0/3525702287 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f94f00b3060 msgr2=0x7f94f006db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.682+0000 7f94ef7fe640 1 --2- 172.21.15.83:0/3525702287 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f94f00b3060 0x7f94f006db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.682+0000 7f94ef7fe640 1 -- 172.21.15.83:0/3525702287 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f0154070 con 0x7f94f015bf30 2024-03-06T07:28:19.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.683+0000 7f94ef7fe640 1 --2- 172.21.15.83:0/3525702287 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94f015bf30 0x7f94f0151720 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f94e8036040 tx=0x7f94e8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.683+0000 7f94ecff9640 1 -- 172.21.15.83:0/3525702287 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e8043020 con 0x7f94f015bf30 2024-03-06T07:28:19.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.683+0000 7f94ecff9640 1 -- 172.21.15.83:0/3525702287 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94e803e450 con 0x7f94f015bf30 2024-03-06T07:28:19.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.683+0000 7f94f5a0e640 1 -- 172.21.15.83:0/3525702287 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94f0154350 con 0x7f94f015bf30 2024-03-06T07:28:19.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.683+0000 7f94ecff9640 1 -- 172.21.15.83:0/3525702287 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e803e860 con 0x7f94f015bf30 2024-03-06T07:28:19.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.683+0000 7f94f5a0e640 1 -- 172.21.15.83:0/3525702287 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94f0154bf0 con 0x7f94f015bf30 2024-03-06T07:28:19.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.684+0000 7f94f5a0e640 1 -- 172.21.15.83:0/3525702287 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94f004f330 con 0x7f94f015bf30 2024-03-06T07:28:19.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.684+0000 7f94ecff9640 1 -- 172.21.15.83:0/3525702287 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f94e8046020 con 0x7f94f015bf30 2024-03-06T07:28:19.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.685+0000 7f94ecff9640 1 --2- 172.21.15.83:0/3525702287 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f94d006b4d0 0x7f94d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.685+0000 7f94ecff9640 1 -- 172.21.15.83:0/3525702287 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f94e80acc60 con 0x7f94f015bf30 2024-03-06T07:28:19.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.685+0000 7f94eeffd640 1 --2- 172.21.15.83:0/3525702287 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f94d006b4d0 0x7f94d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.685+0000 7f94eeffd640 1 --2- 172.21.15.83:0/3525702287 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f94d006b4d0 0x7f94d006d990 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f94f006e3d0 tx=0x7f94e0007ce0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.693+0000 7f94ecff9640 1 -- 172.21.15.83:0/3525702287 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f94e804d030 con 0x7f94f015bf30 2024-03-06T07:28:19.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.909+0000 7f94f5a0e640 1 -- 172.21.15.83:0/3525702287 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94f006eb00 con 0x7f94f015bf30 2024-03-06T07:28:19.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.909+0000 7f94ecff9640 1 -- 172.21.15.83:0/3525702287 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f94e807b9f0 con 0x7f94f015bf30 2024-03-06T07:28:19.911 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:28:19.911 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7051,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/101642171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":101642171},{"type":"v1","addr":"172.21.15.183:6833","nonce":101642171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7057,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3146884900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":3146884900},{"type":"v1","addr":"172.21.15.183:6835","nonce":3146884900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8208,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1922041621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1922041621},{"type":"v1","addr":"172.21.15.83:6835","nonce":1922041621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.491022+0000","modified":"2024-03-06T07:27:37.006649+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8229":{"gid":8229,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/2003687153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2003687153},{"type":"v1","addr":"172.21.15.83:6837","nonce":2003687153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8229,"qdb_cluster":[8229]},"id":8}]} 2024-03-06T07:28:19.912 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 66 2024-03-06T07:28:19.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.912+0000 7f94f5a0e640 1 -- 172.21.15.83:0/3525702287 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f94d006b4d0 msgr2=0x7f94d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.912+0000 7f94f5a0e640 1 --2- 172.21.15.83:0/3525702287 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f94d006b4d0 0x7f94d006d990 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f94f006e3d0 tx=0x7f94e0007ce0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.912+0000 7f94f5a0e640 1 -- 172.21.15.83:0/3525702287 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94f015bf30 msgr2=0x7f94f0151720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.912+0000 7f94f5a0e640 1 --2- 172.21.15.83:0/3525702287 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94f015bf30 0x7f94f0151720 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f94e8036040 tx=0x7f94e8004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:19.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.912+0000 7f94f5a0e640 1 -- 172.21.15.83:0/3525702287 shutdown_connections 2024-03-06T07:28:19.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.912+0000 7f94f5a0e640 1 --2- 172.21.15.83:0/3525702287 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f94d006b4d0 0x7f94d006d990 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.912+0000 7f94f5a0e640 1 --2- 172.21.15.83:0/3525702287 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94f015bf30 0x7f94f0151720 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.912+0000 7f94f5a0e640 1 --2- 172.21.15.83:0/3525702287 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f94f013dc10 0x7f94f01511e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.912+0000 7f94f5a0e640 1 --2- 172.21.15.83:0/3525702287 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f94f00b3060 0x7f94f006db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.912+0000 7f94f5a0e640 1 -- 172.21.15.83:0/3525702287 >> 172.21.15.83:0/3525702287 conn(0x7f94f00cf8a0 msgr2=0x7f94f00b0b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:19.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.912+0000 7f94f5a0e640 1 -- 172.21.15.83:0/3525702287 shutdown_connections 2024-03-06T07:28:19.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:19.912+0000 7f94f5a0e640 1 -- 172.21.15.83:0/3525702287 wait complete. 2024-03-06T07:28:19.925 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:28:20.041 INFO:teuthology.orchestra.run.smithi083.stdout:{"mds_cache":{"num_strays":0}} 2024-03-06T07:28:20.041 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 0 } } 2024-03-06T07:28:20.041 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:20.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.140+0000 7f2484cc9640 1 Processor -- start 2024-03-06T07:28:20.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.140+0000 7f2484cc9640 1 -- start start 2024-03-06T07:28:20.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.142+0000 7f2484cc9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f248015dfe0 0x7f24800cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.142+0000 7f2484cc9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f24800cf830 0x7f24800d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.142+0000 7f2484cc9640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f24800d27f0 0x7f24800d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.142+0000 7f2484cc9640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24800d2c50 con 0x7f24800cf830 2024-03-06T07:28:20.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.142+0000 7f2484cc9640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24800b31e0 con 0x7f24800d27f0 2024-03-06T07:28:20.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.142+0000 7f2484cc9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24800b3d60 con 0x7f248015dfe0 2024-03-06T07:28:20.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.142+0000 7f2475d9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f24800cf830 0x7f24800d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.142+0000 7f2475d9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f24800cf830 0x7f24800d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33756/0 (socket says 172.21.15.83:33756) 2024-03-06T07:28:20.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.142+0000 7f2475d9b640 1 -- 172.21.15.83:0/3434673583 learned_addr learned my addr 172.21.15.83:0/3434673583 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:20.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.142+0000 7f247e59c640 1 --2- 172.21.15.83:0/3434673583 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f248015dfe0 0x7f24800cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.142+0000 7f247ed9d640 1 --2- 172.21.15.83:0/3434673583 >> v2:172.21.15.183:3300/0 conn(0x7f24800d27f0 0x7f24800d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.142+0000 7f2475d9b640 1 -- 172.21.15.83:0/3434673583 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f248015dfe0 msgr2=0x7f24800cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:20.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.142+0000 7f2475d9b640 1 --2- 172.21.15.83:0/3434673583 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f248015dfe0 0x7f24800cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.142+0000 7f2475d9b640 1 -- 172.21.15.83:0/3434673583 >> v2:172.21.15.183:3300/0 conn(0x7f24800d27f0 msgr2=0x7f24800d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:20.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.142+0000 7f2475d9b640 1 --2- 172.21.15.83:0/3434673583 >> v2:172.21.15.183:3300/0 conn(0x7f24800d27f0 0x7f24800d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.142+0000 7f2475d9b640 1 -- 172.21.15.83:0/3434673583 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24800b3430 con 0x7f24800cf830 2024-03-06T07:28:20.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.143+0000 7f2475d9b640 1 --2- 172.21.15.83:0/3434673583 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f24800cf830 0x7f24800d22b0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f2480067ad0 tx=0x7f246802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1e80c426b5cfa86 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.143+0000 7f247dd9b640 1 -- 172.21.15.83:0/3434673583 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2468035020 con 0x7f24800cf830 2024-03-06T07:28:20.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.143+0000 7f247dd9b640 1 -- 172.21.15.83:0/3434673583 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2468031870 con 0x7f24800cf830 2024-03-06T07:28:20.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.143+0000 7f247dd9b640 1 -- 172.21.15.83:0/3434673583 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2468031c40 con 0x7f24800cf830 2024-03-06T07:28:20.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.144+0000 7f2484cc9640 1 -- 172.21.15.83:0/3434673583 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f24800cf830 msgr2=0x7f24800d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.144+0000 7f2484cc9640 1 --2- 172.21.15.83:0/3434673583 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f24800cf830 0x7f24800d22b0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f2480067ad0 tx=0x7f246802b910 comp rx=0 tx=0).stop 2024-03-06T07:28:20.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.144+0000 7f2484cc9640 1 -- 172.21.15.83:0/3434673583 shutdown_connections 2024-03-06T07:28:20.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.144+0000 7f2484cc9640 1 --2- 172.21.15.83:0/3434673583 >> v2:172.21.15.183:3300/0 conn(0x7f24800d27f0 0x7f24800d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.144+0000 7f2484cc9640 1 --2- 172.21.15.83:0/3434673583 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f24800cf830 0x7f24800d22b0 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.144+0000 7f2484cc9640 1 --2- 172.21.15.83:0/3434673583 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f248015dfe0 0x7f24800cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.144+0000 7f2484cc9640 1 -- 172.21.15.83:0/3434673583 >> 172.21.15.83:0/3434673583 conn(0x7f24800d3cd0 msgr2=0x7f24800d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:20.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.144+0000 7f2484cc9640 1 -- 172.21.15.83:0/3434673583 shutdown_connections 2024-03-06T07:28:20.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.144+0000 7f2484cc9640 1 -- 172.21.15.83:0/3434673583 wait complete. 2024-03-06T07:28:20.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.145+0000 7f2484cc9640 1 Processor -- start 2024-03-06T07:28:20.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.145+0000 7f2484cc9640 1 -- start start 2024-03-06T07:28:20.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.146+0000 7f2484cc9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f24800cf830 0x7f24801f2830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.146+0000 7f2484cc9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f24800d27f0 0x7f24801f2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.146+0000 7f2484cc9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f248015dfe0 0x7f2480149cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.146+0000 7f2484cc9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24800d1c70 con 0x7f24800d27f0 2024-03-06T07:28:20.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.146+0000 7f2484cc9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24800b39f0 con 0x7f24800cf830 2024-03-06T07:28:20.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.146+0000 7f2484cc9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24800d1df0 con 0x7f248015dfe0 2024-03-06T07:28:20.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.146+0000 7f2475d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f24800d27f0 0x7f24801f2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.146+0000 7f2475d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f24800d27f0 0x7f24801f2d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33766/0 (socket says 172.21.15.83:33766) 2024-03-06T07:28:20.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.146+0000 7f2475d9b640 1 -- 172.21.15.83:0/3782644234 learned_addr learned my addr 172.21.15.83:0/3782644234 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:20.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.146+0000 7f247ed9d640 1 --2- 172.21.15.83:0/3782644234 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f248015dfe0 0x7f2480149cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.146+0000 7f247e59c640 1 --2- 172.21.15.83:0/3782644234 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f24800cf830 0x7f24801f2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.146+0000 7f2475d9b640 1 -- 172.21.15.83:0/3782644234 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f248015dfe0 msgr2=0x7f2480149cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.146+0000 7f2475d9b640 1 --2- 172.21.15.83:0/3782644234 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f248015dfe0 0x7f2480149cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.146+0000 7f2475d9b640 1 -- 172.21.15.83:0/3782644234 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f24800cf830 msgr2=0x7f24801f2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.146+0000 7f2475d9b640 1 --2- 172.21.15.83:0/3782644234 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f24800cf830 0x7f24801f2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.146+0000 7f2475d9b640 1 -- 172.21.15.83:0/3782644234 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2480161450 con 0x7f24800d27f0 2024-03-06T07:28:20.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.146+0000 7f247ed9d640 1 --2- 172.21.15.83:0/3782644234 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f248015dfe0 0x7f2480149cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:20.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.146+0000 7f247e59c640 1 --2- 172.21.15.83:0/3782644234 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f24800cf830 0x7f24801f2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:20.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.147+0000 7f2475d9b640 1 --2- 172.21.15.83:0/3782644234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f24800d27f0 0x7f24801f2d70 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f2468000c00 tx=0x7f24680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.147+0000 7f24777fe640 1 -- 172.21.15.83:0/3782644234 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2468035040 con 0x7f24800d27f0 2024-03-06T07:28:20.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.147+0000 7f24777fe640 1 -- 172.21.15.83:0/3782644234 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2468030d00 con 0x7f24800d27f0 2024-03-06T07:28:20.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.147+0000 7f24777fe640 1 -- 172.21.15.83:0/3782644234 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24680456a0 con 0x7f24800d27f0 2024-03-06T07:28:20.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.147+0000 7f2484cc9640 1 -- 172.21.15.83:0/3782644234 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24800cc980 con 0x7f24800d27f0 2024-03-06T07:28:20.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.147+0000 7f2484cc9640 1 -- 172.21.15.83:0/3782644234 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f248014dcc0 con 0x7f24800d27f0 2024-03-06T07:28:20.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.148+0000 7f2484cc9640 1 -- 172.21.15.83:0/3782644234 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24800cc7a0 con 0x7f24800d27f0 2024-03-06T07:28:20.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.148+0000 7f24777fe640 1 -- 172.21.15.83:0/3782644234 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2468038020 con 0x7f24800d27f0 2024-03-06T07:28:20.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.149+0000 7f24777fe640 1 --2- 172.21.15.83:0/3782644234 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f246406b4d0 0x7f246406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.149+0000 7f247e59c640 1 --2- 172.21.15.83:0/3782644234 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f246406b4d0 0x7f246406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.149+0000 7f24777fe640 1 -- 172.21.15.83:0/3782644234 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f24680ac5e0 con 0x7f24800d27f0 2024-03-06T07:28:20.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.149+0000 7f247e59c640 1 --2- 172.21.15.83:0/3782644234 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f246406b4d0 0x7f246406d990 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f2470006fd0 tx=0x7f2470008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.153+0000 7f24777fe640 1 -- 172.21.15.83:0/3782644234 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f24800cc7a0 con 0x7f24800d27f0 2024-03-06T07:28:20.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.364+0000 7f2484cc9640 1 -- 172.21.15.83:0/3782644234 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f248004f330 con 0x7f24800d27f0 2024-03-06T07:28:20.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.364+0000 7f24777fe640 1 -- 172.21.15.83:0/3782644234 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f246807b400 con 0x7f24800d27f0 2024-03-06T07:28:20.366 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:28:20.366 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7051,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/101642171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":101642171},{"type":"v1","addr":"172.21.15.183:6833","nonce":101642171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7057,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3146884900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":3146884900},{"type":"v1","addr":"172.21.15.183:6835","nonce":3146884900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8208,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1922041621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1922041621},{"type":"v1","addr":"172.21.15.83:6835","nonce":1922041621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.491022+0000","modified":"2024-03-06T07:27:37.006649+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8229":{"gid":8229,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/2003687153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2003687153},{"type":"v1","addr":"172.21.15.83:6837","nonce":2003687153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8229,"qdb_cluster":[8229]},"id":8}]} 2024-03-06T07:28:20.367 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 66 2024-03-06T07:28:20.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.366+0000 7f2484cc9640 1 -- 172.21.15.83:0/3782644234 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f246406b4d0 msgr2=0x7f246406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.366+0000 7f2484cc9640 1 --2- 172.21.15.83:0/3782644234 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f246406b4d0 0x7f246406d990 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f2470006fd0 tx=0x7f2470008040 comp rx=0 tx=0).stop 2024-03-06T07:28:20.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.366+0000 7f2484cc9640 1 -- 172.21.15.83:0/3782644234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f24800d27f0 msgr2=0x7f24801f2d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.366+0000 7f2484cc9640 1 --2- 172.21.15.83:0/3782644234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f24800d27f0 0x7f24801f2d70 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f2468000c00 tx=0x7f24680046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.366+0000 7f2484cc9640 1 -- 172.21.15.83:0/3782644234 shutdown_connections 2024-03-06T07:28:20.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.366+0000 7f2484cc9640 1 --2- 172.21.15.83:0/3782644234 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f246406b4d0 0x7f246406d990 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.366+0000 7f2484cc9640 1 --2- 172.21.15.83:0/3782644234 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f248015dfe0 0x7f2480149cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.366+0000 7f2484cc9640 1 --2- 172.21.15.83:0/3782644234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f24800d27f0 0x7f24801f2d70 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.366+0000 7f2484cc9640 1 --2- 172.21.15.83:0/3782644234 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f24800cf830 0x7f24801f2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.366+0000 7f2484cc9640 1 -- 172.21.15.83:0/3782644234 >> 172.21.15.83:0/3782644234 conn(0x7f24800d3cd0 msgr2=0x7f248006ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:20.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.366+0000 7f2484cc9640 1 -- 172.21.15.83:0/3782644234 shutdown_connections 2024-03-06T07:28:20.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.366+0000 7f2484cc9640 1 -- 172.21.15.83:0/3782644234 wait complete. 2024-03-06T07:28:20.379 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_reintegrated 2024-03-06T07:28:20.497 INFO:teuthology.orchestra.run.smithi083.stdout:{"mds_cache":{"strays_reintegrated":2}} 2024-03-06T07:28:20.498 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 2 } } 2024-03-06T07:28:20.498 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:20.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.598+0000 7f663762a640 1 Processor -- start 2024-03-06T07:28:20.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.599+0000 7f663762a640 1 -- start start 2024-03-06T07:28:20.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.599+0000 7f663762a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f663015dfe0 0x7f66300cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.599+0000 7f663762a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f66300cf830 0x7f66300d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.599+0000 7f663762a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66300b3430 con 0x7f66300d27f0 2024-03-06T07:28:20.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.599+0000 7f663762a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66300d2c50 con 0x7f663015dfe0 2024-03-06T07:28:20.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.599+0000 7f663762a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66300b31e0 con 0x7f66300cf830 2024-03-06T07:28:20.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.600+0000 7f6635bc7640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f66300d27f0 0x7f66300d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:58256/0 (socket says 172.21.15.83:58256) 2024-03-06T07:28:20.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.600+0000 7f6635bc7640 1 -- 172.21.15.83:0/3719168431 learned_addr learned my addr 172.21.15.83:0/3719168431 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:28:20.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.600+0000 7f6634bc5640 1 --2- 172.21.15.83:0/3719168431 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f66300cf830 0x7f66300d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.600+0000 7f6627fff640 1 -- 172.21.15.83:0/3719168431 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1320659617 0 0) 0x7f66300b3430 con 0x7f66300d27f0 2024-03-06T07:28:20.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.600+0000 7f66353c6640 1 --2- 172.21.15.83:0/3719168431 >> v2:172.21.15.183:3300/0 conn(0x7f663015dfe0 0x7f66300cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.600+0000 7f6627fff640 1 -- 172.21.15.83:0/3719168431 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f661c003520 con 0x7f66300d27f0 2024-03-06T07:28:20.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.600+0000 7f6627fff640 1 -- 172.21.15.83:0/3719168431 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 503740282 0 0) 0x7f661c003520 con 0x7f66300d27f0 2024-03-06T07:28:20.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.600+0000 7f6627fff640 1 -- 172.21.15.83:0/3719168431 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f66300cf830 msgr2=0x7f66300d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:20.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.600+0000 7f6627fff640 1 --2- 172.21.15.83:0/3719168431 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f66300cf830 0x7f66300d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.600+0000 7f6627fff640 1 -- 172.21.15.83:0/3719168431 >> v2:172.21.15.183:3300/0 conn(0x7f663015dfe0 msgr2=0x7f66300cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:20.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.600+0000 7f6627fff640 1 --2- 172.21.15.83:0/3719168431 >> v2:172.21.15.183:3300/0 conn(0x7f663015dfe0 0x7f66300cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.600+0000 7f6627fff640 1 -- 172.21.15.83:0/3719168431 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66300b3710 con 0x7f66300d27f0 2024-03-06T07:28:20.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.600+0000 7f6627fff640 1 -- 172.21.15.83:0/3719168431 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f662c0019f0 con 0x7f66300d27f0 2024-03-06T07:28:20.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.600+0000 7f6627fff640 1 -- 172.21.15.83:0/3719168431 >> v1:172.21.15.83:6789/0 conn(0x7f66300d27f0 legacy=0x7f66300d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.600+0000 7f6627fff640 1 --2- 172.21.15.83:0/3719168431 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f661c0039b0 0x7f661c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.600+0000 7f6627fff640 1 --2- 172.21.15.83:0/3719168431 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f661c004800 0x7f661c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.600+0000 7f6627fff640 1 --2- 172.21.15.83:0/3719168431 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f66300cf830 0x7f661c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.600+0000 7f6627fff640 1 -- 172.21.15.83:0/3719168431 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66300b31e0 con 0x7f661c004800 2024-03-06T07:28:20.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.600+0000 7f6627fff640 1 -- 172.21.15.83:0/3719168431 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661c00b9c0 con 0x7f661c0039b0 2024-03-06T07:28:20.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.600+0000 7f6627fff640 1 -- 172.21.15.83:0/3719168431 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f661c00bb40 con 0x7f66300cf830 2024-03-06T07:28:20.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.600+0000 7f6634bc5640 1 --2- 172.21.15.83:0/3719168431 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f661c004800 0x7f661c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.600+0000 7f66353c6640 1 --2- 172.21.15.83:0/3719168431 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f661c0039b0 0x7f661c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.601+0000 7f6635bc7640 1 --2- 172.21.15.83:0/3719168431 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f66300cf830 0x7f661c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.601+0000 7f6634bc5640 1 -- 172.21.15.83:0/3719168431 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f66300cf830 msgr2=0x7f661c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:20.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.601+0000 7f6634bc5640 1 --2- 172.21.15.83:0/3719168431 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f66300cf830 0x7f661c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.601+0000 7f6634bc5640 1 -- 172.21.15.83:0/3719168431 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f661c0039b0 msgr2=0x7f661c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:20.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.601+0000 7f6634bc5640 1 --2- 172.21.15.83:0/3719168431 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f661c0039b0 0x7f661c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.601+0000 7f6634bc5640 1 -- 172.21.15.83:0/3719168431 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f661c003520 con 0x7f661c004800 2024-03-06T07:28:20.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.601+0000 7f6634bc5640 1 --2- 172.21.15.83:0/3719168431 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f661c004800 0x7f661c004c80 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f6620009a90 tx=0x7f661c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=117f071b8f5b740c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.602+0000 7f6627fff640 1 -- 172.21.15.83:0/3719168431 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6620039070 con 0x7f661c004800 2024-03-06T07:28:20.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.602+0000 7f6627fff640 1 -- 172.21.15.83:0/3719168431 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f662002c680 con 0x7f661c004800 2024-03-06T07:28:20.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.602+0000 7f6627fff640 1 -- 172.21.15.83:0/3719168431 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6620009690 con 0x7f661c004800 2024-03-06T07:28:20.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.602+0000 7f663762a640 1 -- 172.21.15.83:0/3719168431 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f661c004800 msgr2=0x7f661c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.602+0000 7f663762a640 1 --2- 172.21.15.83:0/3719168431 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f661c004800 0x7f661c004c80 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f6620009a90 tx=0x7f661c005830 comp rx=0 tx=0).stop 2024-03-06T07:28:20.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.602+0000 7f6635bc7640 1 -- 172.21.15.83:0/3719168431 reap_dead start 2024-03-06T07:28:20.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.602+0000 7f663762a640 1 -- 172.21.15.83:0/3719168431 shutdown_connections 2024-03-06T07:28:20.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.602+0000 7f663762a640 1 -- 172.21.15.83:0/3719168431 >> 172.21.15.83:0/3719168431 conn(0x7f66300d3cd0 msgr2=0x7f66300d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:20.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.602+0000 7f663762a640 1 -- 172.21.15.83:0/3719168431 shutdown_connections 2024-03-06T07:28:20.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.602+0000 7f663762a640 1 -- 172.21.15.83:0/3719168431 wait complete. 2024-03-06T07:28:20.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.604+0000 7f663762a640 1 Processor -- start 2024-03-06T07:28:20.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.604+0000 7f663762a640 1 -- start start 2024-03-06T07:28:20.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.604+0000 7f663762a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f661c004800 0x7f663014c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.604+0000 7f663762a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f66301f7e40 0x7f66300cbd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.604+0000 7f663762a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66301f7210 0x7f66301f7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.604+0000 7f663762a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66300d1c70 con 0x7f66301f7210 2024-03-06T07:28:20.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.604+0000 7f663762a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66300b3d60 con 0x7f661c004800 2024-03-06T07:28:20.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.604+0000 7f663762a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66300d1df0 con 0x7f66301f7e40 2024-03-06T07:28:20.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.604+0000 7f6635bc7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66301f7210 0x7f66301f7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.604+0000 7f6635bc7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66301f7210 0x7f66301f7690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33778/0 (socket says 172.21.15.83:33778) 2024-03-06T07:28:20.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.604+0000 7f6635bc7640 1 -- 172.21.15.83:0/674554576 learned_addr learned my addr 172.21.15.83:0/674554576 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:20.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.605+0000 7f6634bc5640 1 --2- 172.21.15.83:0/674554576 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f66301f7e40 0x7f66300cbd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.605+0000 7f66353c6640 1 --2- 172.21.15.83:0/674554576 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f661c004800 0x7f663014c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.605+0000 7f6635bc7640 1 -- 172.21.15.83:0/674554576 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f66301f7e40 msgr2=0x7f66300cbd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.605+0000 7f6635bc7640 1 --2- 172.21.15.83:0/674554576 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f66301f7e40 0x7f66300cbd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.605+0000 7f6635bc7640 1 -- 172.21.15.83:0/674554576 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f661c004800 msgr2=0x7f663014c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.605+0000 7f6635bc7640 1 --2- 172.21.15.83:0/674554576 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f661c004800 0x7f663014c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.605+0000 7f6635bc7640 1 -- 172.21.15.83:0/674554576 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f663014cd50 con 0x7f66301f7210 2024-03-06T07:28:20.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.605+0000 7f66353c6640 1 --2- 172.21.15.83:0/674554576 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f661c004800 0x7f663014c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:20.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.605+0000 7f6635bc7640 1 --2- 172.21.15.83:0/674554576 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66301f7210 0x7f66301f7690 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f663008c940 tx=0x7f662c00a750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.605+0000 7f6625ffb640 1 -- 172.21.15.83:0/674554576 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f662c017020 con 0x7f66301f7210 2024-03-06T07:28:20.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.605+0000 7f6625ffb640 1 -- 172.21.15.83:0/674554576 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f662c00ad50 con 0x7f66301f7210 2024-03-06T07:28:20.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.605+0000 7f6625ffb640 1 -- 172.21.15.83:0/674554576 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f662c013630 con 0x7f66301f7210 2024-03-06T07:28:20.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.605+0000 7f663762a640 1 -- 172.21.15.83:0/674554576 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66300cc620 con 0x7f66301f7210 2024-03-06T07:28:20.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.605+0000 7f663762a640 1 -- 172.21.15.83:0/674554576 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66300ccbe0 con 0x7f66301f7210 2024-03-06T07:28:20.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.606+0000 7f663762a640 1 -- 172.21.15.83:0/674554576 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f663004f330 con 0x7f66301f7210 2024-03-06T07:28:20.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.607+0000 7f6625ffb640 1 -- 172.21.15.83:0/674554576 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f662c01a020 con 0x7f66301f7210 2024-03-06T07:28:20.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.607+0000 7f6625ffb640 1 --2- 172.21.15.83:0/674554576 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f661406b500 0x7f661406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.607+0000 7f6625ffb640 1 -- 172.21.15.83:0/674554576 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f662c08aaf0 con 0x7f66301f7210 2024-03-06T07:28:20.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.607+0000 7f66353c6640 1 --2- 172.21.15.83:0/674554576 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f661406b500 0x7f661406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.607+0000 7f66353c6640 1 --2- 172.21.15.83:0/674554576 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f661406b500 0x7f661406d9c0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f6628009bc0 tx=0x7f6628008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.611+0000 7f6625ffb640 1 -- 172.21.15.83:0/674554576 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f662c021020 con 0x7f66301f7210 2024-03-06T07:28:20.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.827+0000 7f663762a640 1 -- 172.21.15.83:0/674554576 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f663006a190 con 0x7f66301f7210 2024-03-06T07:28:20.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.827+0000 7f6625ffb640 1 -- 172.21.15.83:0/674554576 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f662c059890 con 0x7f66301f7210 2024-03-06T07:28:20.829 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:28:20.829 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7051,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/101642171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":101642171},{"type":"v1","addr":"172.21.15.183:6833","nonce":101642171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7057,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3146884900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":3146884900},{"type":"v1","addr":"172.21.15.183:6835","nonce":3146884900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8208,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1922041621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1922041621},{"type":"v1","addr":"172.21.15.83:6835","nonce":1922041621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.491022+0000","modified":"2024-03-06T07:27:37.006649+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8229":{"gid":8229,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/2003687153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2003687153},{"type":"v1","addr":"172.21.15.83:6837","nonce":2003687153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8229,"qdb_cluster":[8229]},"id":8}]} 2024-03-06T07:28:20.830 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 66 2024-03-06T07:28:20.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.829+0000 7f663762a640 1 -- 172.21.15.83:0/674554576 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f661406b500 msgr2=0x7f661406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.829+0000 7f663762a640 1 --2- 172.21.15.83:0/674554576 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f661406b500 0x7f661406d9c0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f6628009bc0 tx=0x7f6628008040 comp rx=0 tx=0).stop 2024-03-06T07:28:20.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.829+0000 7f663762a640 1 -- 172.21.15.83:0/674554576 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66301f7210 msgr2=0x7f66301f7690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.829+0000 7f663762a640 1 --2- 172.21.15.83:0/674554576 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66301f7210 0x7f66301f7690 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f663008c940 tx=0x7f662c00a750 comp rx=0 tx=0).stop 2024-03-06T07:28:20.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.829+0000 7f663762a640 1 -- 172.21.15.83:0/674554576 shutdown_connections 2024-03-06T07:28:20.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.829+0000 7f663762a640 1 --2- 172.21.15.83:0/674554576 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f661406b500 0x7f661406d9c0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.829+0000 7f663762a640 1 --2- 172.21.15.83:0/674554576 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66301f7210 0x7f66301f7690 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.829+0000 7f663762a640 1 --2- 172.21.15.83:0/674554576 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f66301f7e40 0x7f66300cbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.829+0000 7f663762a640 1 --2- 172.21.15.83:0/674554576 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f661c004800 0x7f663014c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.830+0000 7f663762a640 1 -- 172.21.15.83:0/674554576 >> 172.21.15.83:0/674554576 conn(0x7f66300d3cd0 msgr2=0x7f663006ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:20.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.830+0000 7f663762a640 1 -- 172.21.15.83:0/674554576 shutdown_connections 2024-03-06T07:28:20.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:20.830+0000 7f663762a640 1 -- 172.21.15.83:0/674554576 wait complete. 2024-03-06T07:28:20.846 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:28:20.972 INFO:teuthology.orchestra.run.smithi083.stdout:{"message":"","return_code":0} 2024-03-06T07:28:20.973 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:28:20.973 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-06T07:28:21.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.013+0000 7f90b437df40 1 Processor -- start 2024-03-06T07:28:21.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.013+0000 7f90b437df40 1 -- start start 2024-03-06T07:28:21.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.014+0000 7f90b437df40 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5579e378c500 0x5579e385d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.014+0000 7f90b437df40 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5579e37d1d60 0x5579e3789450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.014+0000 7f90b437df40 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x5579e38608b0 0x5579e384fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.014+0000 7f90b437df40 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579e385fc30 con 0x5579e37d1d60 2024-03-06T07:28:21.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.014+0000 7f90b437df40 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579e3730b70 con 0x5579e378c500 2024-03-06T07:28:21.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.014+0000 7f90b437df40 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x5579e385fed0 con 0x5579e38608b0 2024-03-06T07:28:21.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.014+0000 7f90abfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5579e37d1d60 0x5579e3789450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.014+0000 7f90abfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5579e37d1d60 0x5579e3789450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33792/0 (socket says 172.21.15.83:33792) 2024-03-06T07:28:21.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.014+0000 7f90abfff640 1 -- 172.21.15.83:0/79549201 learned_addr learned my addr 172.21.15.83:0/79549201 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:21.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.014+0000 7f90b1e7e640 1 --2- 172.21.15.83:0/79549201 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5579e378c500 0x5579e385d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.014+0000 7f90b267f640 1 --2- 172.21.15.83:0/79549201 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x5579e38608b0 0x5579e384fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.015+0000 7f90abfff640 1 -- 172.21.15.83:0/79549201 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x5579e38608b0 msgr2=0x5579e384fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:21.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.015+0000 7f90abfff640 1 --2- 172.21.15.83:0/79549201 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x5579e38608b0 0x5579e384fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.015+0000 7f90abfff640 1 -- 172.21.15.83:0/79549201 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5579e378c500 msgr2=0x5579e385d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:21.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.015+0000 7f90abfff640 1 --2- 172.21.15.83:0/79549201 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5579e378c500 0x5579e385d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.015+0000 7f90abfff640 1 -- 172.21.15.83:0/79549201 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5579e37882c0 con 0x5579e37d1d60 2024-03-06T07:28:21.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.015+0000 7f90abfff640 1 --2- 172.21.15.83:0/79549201 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5579e37d1d60 0x5579e3789450 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f909c00b2f0 tx=0x7f909c00b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a593b60c72f3050 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.015+0000 7f90b0e7c640 1 -- 172.21.15.83:0/79549201 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f909c008070 con 0x5579e37d1d60 2024-03-06T07:28:21.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.015+0000 7f90b0e7c640 1 -- 172.21.15.83:0/79549201 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f909c00d500 con 0x5579e37d1d60 2024-03-06T07:28:21.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.015+0000 7f90b0e7c640 1 -- 172.21.15.83:0/79549201 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f909c00d8f0 con 0x5579e37d1d60 2024-03-06T07:28:21.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.015+0000 7f90b437df40 1 -- 172.21.15.83:0/79549201 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5579e37d1d60 msgr2=0x5579e3789450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.015+0000 7f90b437df40 1 --2- 172.21.15.83:0/79549201 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5579e37d1d60 0x5579e3789450 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f909c00b2f0 tx=0x7f909c00b7b0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.015+0000 7f90b437df40 1 -- 172.21.15.83:0/79549201 shutdown_connections 2024-03-06T07:28:21.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.015+0000 7f90b437df40 1 --2- 172.21.15.83:0/79549201 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x5579e38608b0 0x5579e384fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.015+0000 7f90b437df40 1 --2- 172.21.15.83:0/79549201 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5579e37d1d60 0x5579e3789450 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.015+0000 7f90b437df40 1 --2- 172.21.15.83:0/79549201 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5579e378c500 0x5579e385d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.015+0000 7f90b437df40 1 -- 172.21.15.83:0/79549201 >> 172.21.15.83:0/79549201 conn(0x5579e37b5350 msgr2=0x5579e3788550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:21.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.015+0000 7f90b437df40 1 -- 172.21.15.83:0/79549201 shutdown_connections 2024-03-06T07:28:21.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.015+0000 7f90b437df40 1 -- 172.21.15.83:0/79549201 wait complete. 2024-03-06T07:28:21.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.016+0000 7f90b437df40 1 Processor -- start 2024-03-06T07:28:21.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.016+0000 7f90b437df40 1 -- start start 2024-03-06T07:28:21.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.017+0000 7f90b437df40 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5579e378c500 0x5579e37b69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.017+0000 7f90b437df40 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5579e37d1d60 0x5579e384c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.017+0000 7f90b437df40 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x5579e38608b0 0x5579e3846230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.017+0000 7f90b437df40 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579e3787e30 con 0x5579e37d1d60 2024-03-06T07:28:21.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.017+0000 7f90b437df40 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579e384ce40 con 0x5579e378c500 2024-03-06T07:28:21.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.017+0000 7f90b437df40 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x5579e384cba0 con 0x5579e38608b0 2024-03-06T07:28:21.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.017+0000 7f90abfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5579e37d1d60 0x5579e384c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.017+0000 7f90abfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5579e37d1d60 0x5579e384c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33804/0 (socket says 172.21.15.83:33804) 2024-03-06T07:28:21.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.017+0000 7f90abfff640 1 -- 172.21.15.83:0/3269217728 learned_addr learned my addr 172.21.15.83:0/3269217728 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:21.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.017+0000 7f90b1e7e640 1 --2- 172.21.15.83:0/3269217728 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5579e378c500 0x5579e37b69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.017+0000 7f90b267f640 1 --2- 172.21.15.83:0/3269217728 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x5579e38608b0 0x5579e3846230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.017+0000 7f90abfff640 1 -- 172.21.15.83:0/3269217728 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x5579e38608b0 msgr2=0x5579e3846230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.017+0000 7f90abfff640 1 --2- 172.21.15.83:0/3269217728 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x5579e38608b0 0x5579e3846230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.017+0000 7f90abfff640 1 -- 172.21.15.83:0/3269217728 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5579e378c500 msgr2=0x5579e37b69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.017+0000 7f90abfff640 1 --2- 172.21.15.83:0/3269217728 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5579e378c500 0x5579e37b69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.017+0000 7f90abfff640 1 -- 172.21.15.83:0/3269217728 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5579e3852220 con 0x5579e37d1d60 2024-03-06T07:28:21.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.017+0000 7f90abfff640 1 --2- 172.21.15.83:0/3269217728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5579e37d1d60 0x5579e384c660 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f909c00b7f0 tx=0x7f909c000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.017+0000 7f90aaffd640 1 -- 172.21.15.83:0/3269217728 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f909c021030 con 0x5579e37d1d60 2024-03-06T07:28:21.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.017+0000 7f90aaffd640 1 -- 172.21.15.83:0/3269217728 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f909c01c440 con 0x5579e37d1d60 2024-03-06T07:28:21.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.017+0000 7f90b437df40 1 -- 172.21.15.83:0/3269217728 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5579e38509b0 con 0x5579e37d1d60 2024-03-06T07:28:21.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.017+0000 7f90aaffd640 1 -- 172.21.15.83:0/3269217728 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f909c01c610 con 0x5579e37d1d60 2024-03-06T07:28:21.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.017+0000 7f90b437df40 1 -- 172.21.15.83:0/3269217728 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5579e3850f70 con 0x5579e37d1d60 2024-03-06T07:28:21.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.019+0000 7f90aaffd640 1 -- 172.21.15.83:0/3269217728 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f909c024070 con 0x5579e37d1d60 2024-03-06T07:28:21.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.020+0000 7f90aaffd640 1 --2- 172.21.15.83:0/3269217728 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9094069eb0 0x7f909406c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.020+0000 7f90aaffd640 1 -- 172.21.15.83:0/3269217728 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f909c089ed0 con 0x5579e37d1d60 2024-03-06T07:28:21.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.020+0000 7f90b1e7e640 1 --2- 172.21.15.83:0/3269217728 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9094069eb0 0x7f909406c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.020+0000 7f90b437df40 1 --2- 172.21.15.83:0/3269217728 >> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] conn(0x5579e384c1d0 0x5579e3846f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.020+0000 7f90b437df40 1 -- 172.21.15.83:0/3269217728 --> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e116) v9 -- 0x5579e38474b0 con 0x5579e384c1d0 2024-03-06T07:28:21.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.020+0000 7f90b267f640 1 --2- 172.21.15.83:0/3269217728 >> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] conn(0x5579e384c1d0 0x5579e3846f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.020+0000 7f90b1e7e640 1 --2- 172.21.15.83:0/3269217728 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9094069eb0 0x7f909406c370 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f90a4058000 tx=0x7f90a4053000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.020+0000 7f90b267f640 1 --2- 172.21.15.83:0/3269217728 >> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] conn(0x5579e384c1d0 0x5579e3846f70 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.021+0000 7f90b267f640 1 -- 172.21.15.83:0/3269217728 <== osd.0 v2:172.21.15.83:6808/1635427425 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (93) out=93b] v0'0 uv11 ondisk = 0) v8 ==== 164+0+93 (crc 0 0 0) 0x7f90ac00ac10 con 0x5579e384c1d0 2024-03-06T07:28:21.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.021+0000 7f90b437df40 1 -- 172.21.15.83:0/3269217728 >> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] conn(0x5579e384c1d0 msgr2=0x5579e3846f70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.021+0000 7f90b437df40 1 --2- 172.21.15.83:0/3269217728 >> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] conn(0x5579e384c1d0 0x5579e3846f70 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.021+0000 7f90b437df40 1 -- 172.21.15.83:0/3269217728 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9094069eb0 msgr2=0x7f909406c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.021+0000 7f90b437df40 1 --2- 172.21.15.83:0/3269217728 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9094069eb0 0x7f909406c370 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f90a4058000 tx=0x7f90a4053000 comp rx=0 tx=0).stop 2024-03-06T07:28:21.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.021+0000 7f90b437df40 1 -- 172.21.15.83:0/3269217728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5579e37d1d60 msgr2=0x5579e384c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.021+0000 7f90b437df40 1 --2- 172.21.15.83:0/3269217728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5579e37d1d60 0x5579e384c660 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f909c00b7f0 tx=0x7f909c000ed0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.021+0000 7f90b267f640 1 -- 172.21.15.83:0/3269217728 reap_dead start 2024-03-06T07:28:21.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.021+0000 7f90b437df40 1 -- 172.21.15.83:0/3269217728 shutdown_connections 2024-03-06T07:28:21.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.021+0000 7f90b437df40 1 -- 172.21.15.83:0/3269217728 >> 172.21.15.83:0/3269217728 conn(0x5579e37b5350 msgr2=0x5579e385d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:21.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.021+0000 7f90b437df40 1 -- 172.21.15.83:0/3269217728 shutdown_connections 2024-03-06T07:28:21.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.021+0000 7f90b437df40 1 -- 172.21.15.83:0/3269217728 wait complete. 2024-03-06T07:28:21.028 INFO:teuthology.orchestra.run.smithi083.stdout:W�file_ddir_2* 2024-03-06T07:28:21.028 DEBUG:teuthology.orchestra.run.smithi083:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-06T07:28:21.128 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:28:21.128 INFO:teuthology.orchestra.run.smithi083.stdout: "ino": 1099511627777, 2024-03-06T07:28:21.128 INFO:teuthology.orchestra.run.smithi083.stdout: "ancestors": [ 2024-03-06T07:28:21.128 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:28:21.128 INFO:teuthology.orchestra.run.smithi083.stdout: "dirino": 1099511628278, 2024-03-06T07:28:21.128 INFO:teuthology.orchestra.run.smithi083.stdout: "dname": "file_d", 2024-03-06T07:28:21.128 INFO:teuthology.orchestra.run.smithi083.stdout: "version": 25 2024-03-06T07:28:21.129 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:28:21.129 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:28:21.129 INFO:teuthology.orchestra.run.smithi083.stdout: "dirino": 1, 2024-03-06T07:28:21.129 INFO:teuthology.orchestra.run.smithi083.stdout: "dname": "dir_2", 2024-03-06T07:28:21.129 INFO:teuthology.orchestra.run.smithi083.stdout: "version": 42 2024-03-06T07:28:21.129 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:28:21.129 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2024-03-06T07:28:21.129 INFO:teuthology.orchestra.run.smithi083.stdout: "pool": 17, 2024-03-06T07:28:21.129 INFO:teuthology.orchestra.run.smithi083.stdout: "old_pools": [] 2024-03-06T07:28:21.129 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:28:21.129 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:28:21.131 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:21.132 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_2/file_d) 2024-03-06T07:28:21.190 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:21.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.328+0000 7f77b91a0640 1 Processor -- start 2024-03-06T07:28:21.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.328+0000 7f77b91a0640 1 -- start start 2024-03-06T07:28:21.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.329+0000 7f77b91a0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f77b415dfe0 0x7f77b40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.329+0000 7f77b91a0640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f77b40cf830 0x7f77b40d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.329+0000 7f77b91a0640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b40b3430 con 0x7f77b40d27f0 2024-03-06T07:28:21.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.329+0000 7f77b91a0640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77b40d2c50 con 0x7f77b40cf830 2024-03-06T07:28:21.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.329+0000 7f77b91a0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77b40b31e0 con 0x7f77b415dfe0 2024-03-06T07:28:21.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.329+0000 7f77b359e640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f77b40d27f0 0x7f77b40d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:58272/0 (socket says 172.21.15.83:58272) 2024-03-06T07:28:21.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.329+0000 7f77b359e640 1 -- 172.21.15.83:0/2204178590 learned_addr learned my addr 172.21.15.83:0/2204178590 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:28:21.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.329+0000 7f77b2d9d640 1 --2- 172.21.15.83:0/2204178590 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f77b415dfe0 0x7f77b40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.329+0000 7f77b259c640 1 --2- 172.21.15.83:0/2204178590 >> v2:172.21.15.183:3300/0 conn(0x7f77b40cf830 0x7f77b40d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.329+0000 7f77b1d9b640 1 -- 172.21.15.83:0/2204178590 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2048737482 0 0) 0x7f77b40b3430 con 0x7f77b40d27f0 2024-03-06T07:28:21.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.329+0000 7f77b1d9b640 1 -- 172.21.15.83:0/2204178590 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f779c003520 con 0x7f77b40d27f0 2024-03-06T07:28:21.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.329+0000 7f77b1d9b640 1 -- 172.21.15.83:0/2204178590 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2523088138 0 0) 0x7f779c003520 con 0x7f77b40d27f0 2024-03-06T07:28:21.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.329+0000 7f77b1d9b640 1 -- 172.21.15.83:0/2204178590 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f77b415dfe0 msgr2=0x7f77b40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:21.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.329+0000 7f77b1d9b640 1 --2- 172.21.15.83:0/2204178590 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f77b415dfe0 0x7f77b40cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.329+0000 7f77b1d9b640 1 -- 172.21.15.83:0/2204178590 >> v2:172.21.15.183:3300/0 conn(0x7f77b40cf830 msgr2=0x7f77b40d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:21.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.329+0000 7f77b1d9b640 1 --2- 172.21.15.83:0/2204178590 >> v2:172.21.15.183:3300/0 conn(0x7f77b40cf830 0x7f77b40d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.329+0000 7f77b1d9b640 1 -- 172.21.15.83:0/2204178590 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77b40b3710 con 0x7f77b40d27f0 2024-03-06T07:28:21.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.329+0000 7f77b1d9b640 1 -- 172.21.15.83:0/2204178590 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f77ac0019f0 con 0x7f77b40d27f0 2024-03-06T07:28:21.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.329+0000 7f77b1d9b640 1 -- 172.21.15.83:0/2204178590 >> v1:172.21.15.83:6789/0 conn(0x7f77b40d27f0 legacy=0x7f77b40d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.329+0000 7f77b1d9b640 1 --2- 172.21.15.83:0/2204178590 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f779c0039b0 0x7f779c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.329+0000 7f77b1d9b640 1 --2- 172.21.15.83:0/2204178590 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f77b415dfe0 0x7f779c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.329+0000 7f77b1d9b640 1 --2- 172.21.15.83:0/2204178590 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f779c004d40 0x7f779c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.329+0000 7f77b1d9b640 1 -- 172.21.15.83:0/2204178590 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b40b31e0 con 0x7f779c0039b0 2024-03-06T07:28:21.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.329+0000 7f77b1d9b640 1 -- 172.21.15.83:0/2204178590 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f779c00b9c0 con 0x7f779c004d40 2024-03-06T07:28:21.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.329+0000 7f77b1d9b640 1 -- 172.21.15.83:0/2204178590 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f779c00bb40 con 0x7f77b415dfe0 2024-03-06T07:28:21.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.329+0000 7f77b2d9d640 1 --2- 172.21.15.83:0/2204178590 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f779c0039b0 0x7f779c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.330+0000 7f77b259c640 1 --2- 172.21.15.83:0/2204178590 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f77b415dfe0 0x7f779c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.330+0000 7f77b359e640 1 --2- 172.21.15.83:0/2204178590 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f779c004d40 0x7f779c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.330+0000 7f77b2d9d640 1 -- 172.21.15.83:0/2204178590 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f77b415dfe0 msgr2=0x7f779c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:21.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.330+0000 7f77b2d9d640 1 --2- 172.21.15.83:0/2204178590 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f77b415dfe0 0x7f779c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.330+0000 7f77b2d9d640 1 -- 172.21.15.83:0/2204178590 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f779c004d40 msgr2=0x7f779c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:21.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.330+0000 7f77b2d9d640 1 --2- 172.21.15.83:0/2204178590 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f779c004d40 0x7f779c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.330+0000 7f77b2d9d640 1 -- 172.21.15.83:0/2204178590 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f779c003520 con 0x7f779c0039b0 2024-03-06T07:28:21.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.330+0000 7f77b2d9d640 1 --2- 172.21.15.83:0/2204178590 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f779c0039b0 0x7f779c003dd0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f77a4009ac0 tx=0x7f779c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75df4b1bd5fccb3f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.331+0000 7f77b1d9b640 1 -- 172.21.15.83:0/2204178590 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a4039070 con 0x7f779c0039b0 2024-03-06T07:28:21.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.331+0000 7f77b1d9b640 1 -- 172.21.15.83:0/2204178590 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77a402c6b0 con 0x7f779c0039b0 2024-03-06T07:28:21.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.331+0000 7f77b1d9b640 1 -- 172.21.15.83:0/2204178590 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a4009690 con 0x7f779c0039b0 2024-03-06T07:28:21.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.331+0000 7f77b91a0640 1 -- 172.21.15.83:0/2204178590 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f779c0039b0 msgr2=0x7f779c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.331+0000 7f77b91a0640 1 --2- 172.21.15.83:0/2204178590 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f779c0039b0 0x7f779c003dd0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f77a4009ac0 tx=0x7f779c005830 comp rx=0 tx=0).stop 2024-03-06T07:28:21.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.331+0000 7f77b359e640 1 -- 172.21.15.83:0/2204178590 reap_dead start 2024-03-06T07:28:21.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.331+0000 7f77b91a0640 1 -- 172.21.15.83:0/2204178590 shutdown_connections 2024-03-06T07:28:21.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.331+0000 7f77b91a0640 1 -- 172.21.15.83:0/2204178590 >> 172.21.15.83:0/2204178590 conn(0x7f77b40d3cd0 msgr2=0x7f77b40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:21.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.331+0000 7f77b91a0640 1 -- 172.21.15.83:0/2204178590 shutdown_connections 2024-03-06T07:28:21.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.331+0000 7f77b91a0640 1 -- 172.21.15.83:0/2204178590 wait complete. 2024-03-06T07:28:21.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.333+0000 7f77b91a0640 1 Processor -- start 2024-03-06T07:28:21.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.333+0000 7f77b91a0640 1 -- start start 2024-03-06T07:28:21.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.333+0000 7f77b91a0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f779c0039b0 0x7f77b406ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.333+0000 7f77b91a0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f77b40ccdf0 0x7f77b40cd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.333+0000 7f77b91a0640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f77b4156050 0x7f77b41564d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.333+0000 7f77b91a0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b40d1c70 con 0x7f779c0039b0 2024-03-06T07:28:21.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.333+0000 7f77b91a0640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b40b3d60 con 0x7f77b4156050 2024-03-06T07:28:21.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.333+0000 7f77b91a0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77b40d1df0 con 0x7f77b40ccdf0 2024-03-06T07:28:21.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.333+0000 7f77b2d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f779c0039b0 0x7f77b406ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.333+0000 7f77b2d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f779c0039b0 0x7f77b406ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33822/0 (socket says 172.21.15.83:33822) 2024-03-06T07:28:21.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.333+0000 7f77b2d9d640 1 -- 172.21.15.83:0/2503626302 learned_addr learned my addr 172.21.15.83:0/2503626302 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:21.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.334+0000 7f77b259c640 1 --2- 172.21.15.83:0/2503626302 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f77b40ccdf0 0x7f77b40cd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.334+0000 7f77b359e640 1 --2- 172.21.15.83:0/2503626302 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f77b4156050 0x7f77b41564d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.334+0000 7f77b2d9d640 1 -- 172.21.15.83:0/2503626302 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f77b40ccdf0 msgr2=0x7f77b40cd270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.334+0000 7f77b2d9d640 1 --2- 172.21.15.83:0/2503626302 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f77b40ccdf0 0x7f77b40cd270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.334+0000 7f77b2d9d640 1 -- 172.21.15.83:0/2503626302 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f77b4156050 msgr2=0x7f77b41564d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.334+0000 7f77b2d9d640 1 --2- 172.21.15.83:0/2503626302 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f77b4156050 0x7f77b41564d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.334+0000 7f77b2d9d640 1 -- 172.21.15.83:0/2503626302 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77b40cda00 con 0x7f779c0039b0 2024-03-06T07:28:21.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.334+0000 7f77b2d9d640 1 --2- 172.21.15.83:0/2503626302 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f779c0039b0 0x7f77b406ccc0 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f77a4037040 tx=0x7f77a4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.334+0000 7f779b7fe640 1 -- 172.21.15.83:0/2503626302 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a4042020 con 0x7f779c0039b0 2024-03-06T07:28:21.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.334+0000 7f779b7fe640 1 -- 172.21.15.83:0/2503626302 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77a403a040 con 0x7f779c0039b0 2024-03-06T07:28:21.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.334+0000 7f77b91a0640 1 -- 172.21.15.83:0/2503626302 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77b40cdce0 con 0x7f779c0039b0 2024-03-06T07:28:21.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.334+0000 7f779b7fe640 1 -- 172.21.15.83:0/2503626302 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a4039070 con 0x7f779c0039b0 2024-03-06T07:28:21.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.334+0000 7f77b91a0640 1 -- 172.21.15.83:0/2503626302 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77b40b6e00 con 0x7f779c0039b0 2024-03-06T07:28:21.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.335+0000 7f77b91a0640 1 -- 172.21.15.83:0/2503626302 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77b40cf5f0 con 0x7f779c0039b0 2024-03-06T07:28:21.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.336+0000 7f779b7fe640 1 -- 172.21.15.83:0/2503626302 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f77a4045020 con 0x7f779c0039b0 2024-03-06T07:28:21.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.336+0000 7f779b7fe640 1 --2- 172.21.15.83:0/2503626302 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f77a006b4d0 0x7f77a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.336+0000 7f77b259c640 1 --2- 172.21.15.83:0/2503626302 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f77a006b4d0 0x7f77a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.336+0000 7f779b7fe640 1 -- 172.21.15.83:0/2503626302 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f77a40ab630 con 0x7f779c0039b0 2024-03-06T07:28:21.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.337+0000 7f77b259c640 1 --2- 172.21.15.83:0/2503626302 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f77a006b4d0 0x7f77a006d990 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f77b40b9d80 tx=0x7f77a8009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.341+0000 7f779b7fe640 1 -- 172.21.15.83:0/2503626302 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f77b40cf5f0 con 0x7f779c0039b0 2024-03-06T07:28:21.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.555+0000 7f77b91a0640 1 -- 172.21.15.83:0/2503626302 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77b404f330 con 0x7f779c0039b0 2024-03-06T07:28:21.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.556+0000 7f779b7fe640 1 -- 172.21.15.83:0/2503626302 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f77a404c070 con 0x7f779c0039b0 2024-03-06T07:28:21.558 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:28:21.558 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7051,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/101642171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":101642171},{"type":"v1","addr":"172.21.15.183:6833","nonce":101642171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7057,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3146884900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":3146884900},{"type":"v1","addr":"172.21.15.183:6835","nonce":3146884900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8208,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1922041621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1922041621},{"type":"v1","addr":"172.21.15.83:6835","nonce":1922041621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.491022+0000","modified":"2024-03-06T07:27:37.006649+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8229":{"gid":8229,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/2003687153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2003687153},{"type":"v1","addr":"172.21.15.83:6837","nonce":2003687153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8229,"qdb_cluster":[8229]},"id":8}]} 2024-03-06T07:28:21.558 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 66 2024-03-06T07:28:21.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.557+0000 7f77b91a0640 1 -- 172.21.15.83:0/2503626302 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f77a006b4d0 msgr2=0x7f77a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.557+0000 7f77b91a0640 1 --2- 172.21.15.83:0/2503626302 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f77a006b4d0 0x7f77a006d990 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f77b40b9d80 tx=0x7f77a8009190 comp rx=0 tx=0).stop 2024-03-06T07:28:21.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.557+0000 7f77b91a0640 1 -- 172.21.15.83:0/2503626302 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f779c0039b0 msgr2=0x7f77b406ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.557+0000 7f77b91a0640 1 --2- 172.21.15.83:0/2503626302 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f779c0039b0 0x7f77b406ccc0 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f77a4037040 tx=0x7f77a4004730 comp rx=0 tx=0).stop 2024-03-06T07:28:21.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.558+0000 7f77b91a0640 1 -- 172.21.15.83:0/2503626302 shutdown_connections 2024-03-06T07:28:21.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.558+0000 7f77b91a0640 1 --2- 172.21.15.83:0/2503626302 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f77a006b4d0 0x7f77a006d990 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.558+0000 7f77b91a0640 1 --2- 172.21.15.83:0/2503626302 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f77b4156050 0x7f77b41564d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.558+0000 7f77b91a0640 1 --2- 172.21.15.83:0/2503626302 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f77b40ccdf0 0x7f77b40cd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.558+0000 7f77b91a0640 1 --2- 172.21.15.83:0/2503626302 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f779c0039b0 0x7f77b406ccc0 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.558+0000 7f77b91a0640 1 -- 172.21.15.83:0/2503626302 >> 172.21.15.83:0/2503626302 conn(0x7f77b40d3cd0 msgr2=0x7f77b40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:21.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.558+0000 7f77b91a0640 1 -- 172.21.15.83:0/2503626302 shutdown_connections 2024-03-06T07:28:21.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:21.558+0000 7f77b91a0640 1 -- 172.21.15.83:0/2503626302 wait complete. 2024-03-06T07:28:21.571 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:28:21.687 INFO:teuthology.orchestra.run.smithi083.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-06T07:28:21.688 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-06T07:28:21.688 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 1, waiting (timeout=60)... 2024-03-06T07:28:26.689 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:26.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.789+0000 7f724a066640 1 Processor -- start 2024-03-06T07:28:26.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.789+0000 7f724a066640 1 -- start start 2024-03-06T07:28:26.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.789+0000 7f724a066640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f72440b3060 0x7f724415e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.789+0000 7f724a066640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f724415bf30 0x7f724415ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.789+0000 7f724a066640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f724415f2d0 con 0x7f724413dc10 2024-03-06T07:28:26.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.789+0000 7f724a066640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72440cf3f0 con 0x7f72440b3060 2024-03-06T07:28:26.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.789+0000 7f724a066640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f724415b210 con 0x7f724415bf30 2024-03-06T07:28:26.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.790+0000 7f7242ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f72440b3060 0x7f724415e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.790+0000 7f7242ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f72440b3060 0x7f724415e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52218/0 (socket says 172.21.15.83:52218) 2024-03-06T07:28:26.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.790+0000 7f7242ffd640 1 -- 172.21.15.83:0/2602842388 learned_addr learned my addr 172.21.15.83:0/2602842388 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:26.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.790+0000 7f7243fff640 1 --2- 172.21.15.83:0/2602842388 >> v2:172.21.15.183:3300/0 conn(0x7f724415bf30 0x7f724415ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.790+0000 7f7242ffd640 1 -- 172.21.15.83:0/2602842388 >> v2:172.21.15.183:3300/0 conn(0x7f724415bf30 msgr2=0x7f724415ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.790+0000 7f7242ffd640 1 --2- 172.21.15.83:0/2602842388 >> v2:172.21.15.183:3300/0 conn(0x7f724415bf30 0x7f724415ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.790+0000 7f7242ffd640 1 -- 172.21.15.83:0/2602842388 >> v1:172.21.15.183:6789/0 conn(0x7f724413dc10 legacy=0x7f72440b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.790+0000 7f7242ffd640 1 -- 172.21.15.83:0/2602842388 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f724415f580 con 0x7f72440b3060 2024-03-06T07:28:26.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.791+0000 7f7242ffd640 1 --2- 172.21.15.83:0/2602842388 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72440b3060 0x7f724415e850 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f72440b4e00 tx=0x7f723802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e827e51b05c8cb13 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.791+0000 7f72427fc640 1 -- 172.21.15.83:0/2602842388 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f723802e070 con 0x7f72440b3060 2024-03-06T07:28:26.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.791+0000 7f72427fc640 1 -- 172.21.15.83:0/2602842388 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7238037070 con 0x7f72440b3060 2024-03-06T07:28:26.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.791+0000 7f72427fc640 1 -- 172.21.15.83:0/2602842388 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7238031a00 con 0x7f72440b3060 2024-03-06T07:28:26.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.791+0000 7f724a066640 1 -- 172.21.15.83:0/2602842388 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72440b3060 msgr2=0x7f724415e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.791+0000 7f724a066640 1 --2- 172.21.15.83:0/2602842388 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72440b3060 0x7f724415e850 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f72440b4e00 tx=0x7f723802b880 comp rx=0 tx=0).stop 2024-03-06T07:28:26.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.791+0000 7f724a066640 1 -- 172.21.15.83:0/2602842388 shutdown_connections 2024-03-06T07:28:26.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.791+0000 7f724a066640 1 --2- 172.21.15.83:0/2602842388 >> v2:172.21.15.183:3300/0 conn(0x7f724415bf30 0x7f724415ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.791+0000 7f724a066640 1 --2- 172.21.15.83:0/2602842388 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72440b3060 0x7f724415e850 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.791+0000 7f724a066640 1 -- 172.21.15.83:0/2602842388 >> 172.21.15.83:0/2602842388 conn(0x7f72440cf8a0 msgr2=0x7f72440cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.791+0000 7f724a066640 1 -- 172.21.15.83:0/2602842388 shutdown_connections 2024-03-06T07:28:26.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.791+0000 7f724a066640 1 -- 172.21.15.83:0/2602842388 wait complete. 2024-03-06T07:28:26.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.792+0000 7f724a066640 1 Processor -- start 2024-03-06T07:28:26.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.793+0000 7f724a066640 1 -- start start 2024-03-06T07:28:26.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.793+0000 7f724a066640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72440b3060 0x7f7244144190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.793+0000 7f724a066640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f724413dc10 0x7f7244139f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.793+0000 7f724a066640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f724415bf30 0x7f7244153970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.793+0000 7f724a066640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f724413dff0 con 0x7f72440b3060 2024-03-06T07:28:26.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.793+0000 7f724a066640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f724415b730 con 0x7f724415bf30 2024-03-06T07:28:26.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.793+0000 7f724a066640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72440708d0 con 0x7f724413dc10 2024-03-06T07:28:26.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.793+0000 7f72437fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72440b3060 0x7f7244144190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.793+0000 7f72437fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72440b3060 0x7f7244144190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52234/0 (socket says 172.21.15.83:52234) 2024-03-06T07:28:26.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.793+0000 7f72437fe640 1 -- 172.21.15.83:0/4023417685 learned_addr learned my addr 172.21.15.83:0/4023417685 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:26.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.794+0000 7f7243fff640 1 --2- 172.21.15.83:0/4023417685 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f724415bf30 0x7f7244153970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.794+0000 7f7242ffd640 1 --2- 172.21.15.83:0/4023417685 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f724413dc10 0x7f7244139f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.794+0000 7f72437fe640 1 -- 172.21.15.83:0/4023417685 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f724413dc10 msgr2=0x7f7244139f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.794+0000 7f72437fe640 1 --2- 172.21.15.83:0/4023417685 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f724413dc10 0x7f7244139f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.794+0000 7f72437fe640 1 -- 172.21.15.83:0/4023417685 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f724415bf30 msgr2=0x7f7244153970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.794+0000 7f72437fe640 1 --2- 172.21.15.83:0/4023417685 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f724415bf30 0x7f7244153970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.794+0000 7f72437fe640 1 -- 172.21.15.83:0/4023417685 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72441436d0 con 0x7f72440b3060 2024-03-06T07:28:26.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.794+0000 7f72437fe640 1 --2- 172.21.15.83:0/4023417685 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72440b3060 0x7f7244144190 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f723400c460 tx=0x7f723400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.794+0000 7f722bfff640 1 -- 172.21.15.83:0/4023417685 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7234016020 con 0x7f72440b3060 2024-03-06T07:28:26.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.794+0000 7f722bfff640 1 -- 172.21.15.83:0/4023417685 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7234012870 con 0x7f72440b3060 2024-03-06T07:28:26.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.794+0000 7f724a066640 1 -- 172.21.15.83:0/4023417685 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72441439b0 con 0x7f72440b3060 2024-03-06T07:28:26.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.794+0000 7f722bfff640 1 -- 172.21.15.83:0/4023417685 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7234012c60 con 0x7f72440b3060 2024-03-06T07:28:26.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.794+0000 7f724a066640 1 -- 172.21.15.83:0/4023417685 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f724413a460 con 0x7f72440b3060 2024-03-06T07:28:26.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.795+0000 7f724a066640 1 -- 172.21.15.83:0/4023417685 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72441434c0 con 0x7f72440b3060 2024-03-06T07:28:26.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.796+0000 7f722bfff640 1 -- 172.21.15.83:0/4023417685 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7234019020 con 0x7f72440b3060 2024-03-06T07:28:26.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.796+0000 7f722bfff640 1 --2- 172.21.15.83:0/4023417685 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f723006b4d0 0x7f723006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.796+0000 7f7242ffd640 1 --2- 172.21.15.83:0/4023417685 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f723006b4d0 0x7f723006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.796+0000 7f722bfff640 1 -- 172.21.15.83:0/4023417685 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f7234089e30 con 0x7f72440b3060 2024-03-06T07:28:26.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.797+0000 7f7242ffd640 1 --2- 172.21.15.83:0/4023417685 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f723006b4d0 0x7f723006d990 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f7238036000 tx=0x7f723803d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:26.801+0000 7f722bfff640 1 -- 172.21.15.83:0/4023417685 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7234020030 con 0x7f72440b3060 2024-03-06T07:28:27.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.040+0000 7f724a066640 1 -- 172.21.15.83:0/4023417685 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f724404f330 con 0x7f72440b3060 2024-03-06T07:28:27.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.040+0000 7f722bfff640 1 -- 172.21.15.83:0/4023417685 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f7234058600 con 0x7f72440b3060 2024-03-06T07:28:27.042 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:28:27.042 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7051,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/101642171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":101642171},{"type":"v1","addr":"172.21.15.183:6833","nonce":101642171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7057,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3146884900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":3146884900},{"type":"v1","addr":"172.21.15.183:6835","nonce":3146884900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8208,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1922041621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1922041621},{"type":"v1","addr":"172.21.15.83:6835","nonce":1922041621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.491022+0000","modified":"2024-03-06T07:27:37.006649+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8229":{"gid":8229,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/2003687153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2003687153},{"type":"v1","addr":"172.21.15.83:6837","nonce":2003687153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8229,"qdb_cluster":[8229]},"id":8}]} 2024-03-06T07:28:27.043 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 66 2024-03-06T07:28:27.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.043+0000 7f724a066640 1 -- 172.21.15.83:0/4023417685 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f723006b4d0 msgr2=0x7f723006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.043+0000 7f724a066640 1 --2- 172.21.15.83:0/4023417685 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f723006b4d0 0x7f723006d990 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f7238036000 tx=0x7f723803d040 comp rx=0 tx=0).stop 2024-03-06T07:28:27.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.043+0000 7f724a066640 1 -- 172.21.15.83:0/4023417685 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72440b3060 msgr2=0x7f7244144190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.043+0000 7f724a066640 1 --2- 172.21.15.83:0/4023417685 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72440b3060 0x7f7244144190 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f723400c460 tx=0x7f723400c920 comp rx=0 tx=0).stop 2024-03-06T07:28:27.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.043+0000 7f724a066640 1 -- 172.21.15.83:0/4023417685 shutdown_connections 2024-03-06T07:28:27.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.043+0000 7f724a066640 1 --2- 172.21.15.83:0/4023417685 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f723006b4d0 0x7f723006d990 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.043+0000 7f724a066640 1 --2- 172.21.15.83:0/4023417685 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f724415bf30 0x7f7244153970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.043+0000 7f724a066640 1 --2- 172.21.15.83:0/4023417685 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f724413dc10 0x7f7244139f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.043+0000 7f724a066640 1 --2- 172.21.15.83:0/4023417685 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72440b3060 0x7f7244144190 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.043+0000 7f724a066640 1 -- 172.21.15.83:0/4023417685 >> 172.21.15.83:0/4023417685 conn(0x7f72440cf8a0 msgr2=0x7f72440cd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.043+0000 7f724a066640 1 -- 172.21.15.83:0/4023417685 shutdown_connections 2024-03-06T07:28:27.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.043+0000 7f724a066640 1 -- 172.21.15.83:0/4023417685 wait complete. 2024-03-06T07:28:27.059 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:28:27.176 INFO:teuthology.orchestra.run.smithi083.stdout:{"mds_cache":{"strays_enqueued":1}} 2024-03-06T07:28:27.176 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1 } } 2024-03-06T07:28:27.176 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:27.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.274+0000 7f62b2639640 1 Processor -- start 2024-03-06T07:28:27.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.274+0000 7f62b2639640 1 -- start start 2024-03-06T07:28:27.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.275+0000 7f62b2639640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f62ac15dfe0 0x7f62ac0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.275+0000 7f62b2639640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f62ac0cf830 0x7f62ac0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.275+0000 7f62b2639640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62ac0b3430 con 0x7f62ac0d27f0 2024-03-06T07:28:27.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.275+0000 7f62b2639640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62ac0d2c50 con 0x7f62ac0cf830 2024-03-06T07:28:27.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.275+0000 7f62b2639640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62ac0b31e0 con 0x7f62ac15dfe0 2024-03-06T07:28:27.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.276+0000 7f62b0bd6640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f62ac0d27f0 0x7f62ac0d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49078/0 (socket says 172.21.15.83:49078) 2024-03-06T07:28:27.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.276+0000 7f62b0bd6640 1 -- 172.21.15.83:0/1607225400 learned_addr learned my addr 172.21.15.83:0/1607225400 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:28:27.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.276+0000 7f62abfff640 1 --2- 172.21.15.83:0/1607225400 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f62ac15dfe0 0x7f62ac0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.276+0000 7f62ab7fe640 1 --2- 172.21.15.83:0/1607225400 >> v2:172.21.15.183:3300/0 conn(0x7f62ac0cf830 0x7f62ac0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.276+0000 7f62aaffd640 1 -- 172.21.15.83:0/1607225400 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3904121730 0 0) 0x7f62ac0b3430 con 0x7f62ac0d27f0 2024-03-06T07:28:27.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.276+0000 7f62aaffd640 1 -- 172.21.15.83:0/1607225400 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6294003520 con 0x7f62ac0d27f0 2024-03-06T07:28:27.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.276+0000 7f62abfff640 1 -- 172.21.15.83:0/1607225400 >> v2:172.21.15.183:3300/0 conn(0x7f62ac0cf830 msgr2=0x7f62ac0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:27.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.276+0000 7f62aaffd640 1 -- 172.21.15.83:0/1607225400 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1374571079 0 0) 0x7f6294003520 con 0x7f62ac0d27f0 2024-03-06T07:28:27.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.276+0000 7f62abfff640 1 --2- 172.21.15.83:0/1607225400 >> v2:172.21.15.183:3300/0 conn(0x7f62ac0cf830 0x7f62ac0d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.276+0000 7f62abfff640 1 -- 172.21.15.83:0/1607225400 >> v1:172.21.15.83:6789/0 conn(0x7f62ac0d27f0 legacy=0x7f62ac0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.276+0000 7f62abfff640 1 -- 172.21.15.83:0/1607225400 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62ac0b3710 con 0x7f62ac15dfe0 2024-03-06T07:28:27.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.276+0000 7f62ab7fe640 1 --2- 172.21.15.83:0/1607225400 >> v2:172.21.15.183:3300/0 conn(0x7f62ac0cf830 0x7f62ac0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:27.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.277+0000 7f62abfff640 1 --2- 172.21.15.83:0/1607225400 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f62ac15dfe0 0x7f62ac0cf2f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f629c0097a0 tx=0x7f629c02b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42e53bcc23ddf82f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.277+0000 7f62aaffd640 1 -- 172.21.15.83:0/1607225400 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f629c037070 con 0x7f62ac15dfe0 2024-03-06T07:28:27.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.277+0000 7f62aaffd640 1 -- 172.21.15.83:0/1607225400 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f629c030510 con 0x7f62ac15dfe0 2024-03-06T07:28:27.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.277+0000 7f62aaffd640 1 -- 172.21.15.83:0/1607225400 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a40019f0 con 0x7f62ac15dfe0 2024-03-06T07:28:27.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.277+0000 7f62b2639640 1 -- 172.21.15.83:0/1607225400 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f62ac15dfe0 msgr2=0x7f62ac0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.278+0000 7f62b2639640 1 --2- 172.21.15.83:0/1607225400 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f62ac15dfe0 0x7f62ac0cf2f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f629c0097a0 tx=0x7f629c02b790 comp rx=0 tx=0).stop 2024-03-06T07:28:27.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.278+0000 7f62b2639640 1 -- 172.21.15.83:0/1607225400 shutdown_connections 2024-03-06T07:28:27.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.278+0000 7f62b2639640 1 --2- 172.21.15.83:0/1607225400 >> v2:172.21.15.183:3300/0 conn(0x7f62ac0cf830 0x7f62ac0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.278+0000 7f62b2639640 1 --2- 172.21.15.83:0/1607225400 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f62ac15dfe0 0x7f62ac0cf2f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.278+0000 7f62b2639640 1 -- 172.21.15.83:0/1607225400 >> 172.21.15.83:0/1607225400 conn(0x7f62ac0d3cd0 msgr2=0x7f62ac0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.278+0000 7f62b2639640 1 -- 172.21.15.83:0/1607225400 shutdown_connections 2024-03-06T07:28:27.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.278+0000 7f62b2639640 1 -- 172.21.15.83:0/1607225400 wait complete. 2024-03-06T07:28:27.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.279+0000 7f62b2639640 1 Processor -- start 2024-03-06T07:28:27.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.279+0000 7f62b2639640 1 -- start start 2024-03-06T07:28:27.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.280+0000 7f62b2639640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f62ac0cf830 0x7f62ac1f3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.280+0000 7f62b2639640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62ac0d27f0 0x7f62ac1f68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.280+0000 7f62b2639640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f62ac15dfe0 0x7f62ac15eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.280+0000 7f62b2639640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ac0d1c70 con 0x7f62ac0d27f0 2024-03-06T07:28:27.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.280+0000 7f62b2639640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ac0b3d60 con 0x7f62ac15dfe0 2024-03-06T07:28:27.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.280+0000 7f62b2639640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62ac0d1df0 con 0x7f62ac0cf830 2024-03-06T07:28:27.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.280+0000 7f62ab7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62ac0d27f0 0x7f62ac1f68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.280+0000 7f62ab7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62ac0d27f0 0x7f62ac1f68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52240/0 (socket says 172.21.15.83:52240) 2024-03-06T07:28:27.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.280+0000 7f62ab7fe640 1 -- 172.21.15.83:0/3561315870 learned_addr learned my addr 172.21.15.83:0/3561315870 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:27.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.280+0000 7f62abfff640 1 --2- 172.21.15.83:0/3561315870 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f62ac0cf830 0x7f62ac1f3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.280+0000 7f62b0bd6640 1 --2- 172.21.15.83:0/3561315870 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f62ac15dfe0 0x7f62ac15eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.280+0000 7f62ab7fe640 1 -- 172.21.15.83:0/3561315870 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f62ac0cf830 msgr2=0x7f62ac1f3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.280+0000 7f62ab7fe640 1 --2- 172.21.15.83:0/3561315870 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f62ac0cf830 0x7f62ac1f3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.280+0000 7f62ab7fe640 1 -- 172.21.15.83:0/3561315870 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f62ac15dfe0 msgr2=0x7f62ac15eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.280+0000 7f62ab7fe640 1 --2- 172.21.15.83:0/3561315870 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f62ac15dfe0 0x7f62ac15eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.280+0000 7f62ab7fe640 1 -- 172.21.15.83:0/3561315870 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62ac15f860 con 0x7f62ac0d27f0 2024-03-06T07:28:27.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.280+0000 7f62ab7fe640 1 --2- 172.21.15.83:0/3561315870 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62ac0d27f0 0x7f62ac1f68d0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f62a000e9b0 tx=0x7f62a000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.280+0000 7f62a8ff9640 1 -- 172.21.15.83:0/3561315870 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a0017070 con 0x7f62ac0d27f0 2024-03-06T07:28:27.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.281+0000 7f62a8ff9640 1 -- 172.21.15.83:0/3561315870 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62a000ca00 con 0x7f62ac0d27f0 2024-03-06T07:28:27.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.281+0000 7f62b2639640 1 -- 172.21.15.83:0/3561315870 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62ac0b68d0 con 0x7f62ac0d27f0 2024-03-06T07:28:27.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.281+0000 7f62a8ff9640 1 -- 172.21.15.83:0/3561315870 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a000cdf0 con 0x7f62ac0d27f0 2024-03-06T07:28:27.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.281+0000 7f62b2639640 1 -- 172.21.15.83:0/3561315870 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62ac0b6e90 con 0x7f62ac0d27f0 2024-03-06T07:28:27.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.282+0000 7f62b2639640 1 -- 172.21.15.83:0/3561315870 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62ac04f330 con 0x7f62ac0d27f0 2024-03-06T07:28:27.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.282+0000 7f62a8ff9640 1 -- 172.21.15.83:0/3561315870 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f62a001a070 con 0x7f62ac0d27f0 2024-03-06T07:28:27.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.283+0000 7f62a8ff9640 1 --2- 172.21.15.83:0/3561315870 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f629806b4d0 0x7f629806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.283+0000 7f62abfff640 1 --2- 172.21.15.83:0/3561315870 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f629806b4d0 0x7f629806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.283+0000 7f62a8ff9640 1 -- 172.21.15.83:0/3561315870 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f62a0089eb0 con 0x7f62ac0d27f0 2024-03-06T07:28:27.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.283+0000 7f62abfff640 1 --2- 172.21.15.83:0/3561315870 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f629806b4d0 0x7f629806d990 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f629c034000 tx=0x7f629c02f000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.287+0000 7f62a8ff9640 1 -- 172.21.15.83:0/3561315870 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f62a0058c40 con 0x7f62ac0d27f0 2024-03-06T07:28:27.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.616+0000 7f62b2639640 1 -- 172.21.15.83:0/3561315870 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62ac15d630 con 0x7f62ac0d27f0 2024-03-06T07:28:27.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.617+0000 7f62a8ff9640 1 -- 172.21.15.83:0/3561315870 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f62a0058540 con 0x7f62ac0d27f0 2024-03-06T07:28:27.619 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:28:27.619 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7051,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/101642171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":101642171},{"type":"v1","addr":"172.21.15.183:6833","nonce":101642171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7057,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3146884900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":3146884900},{"type":"v1","addr":"172.21.15.183:6835","nonce":3146884900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8208,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1922041621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1922041621},{"type":"v1","addr":"172.21.15.83:6835","nonce":1922041621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.491022+0000","modified":"2024-03-06T07:27:37.006649+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8229":{"gid":8229,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/2003687153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2003687153},{"type":"v1","addr":"172.21.15.83:6837","nonce":2003687153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8229,"qdb_cluster":[8229]},"id":8}]} 2024-03-06T07:28:27.619 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 66 2024-03-06T07:28:27.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.619+0000 7f62b2639640 1 -- 172.21.15.83:0/3561315870 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f629806b4d0 msgr2=0x7f629806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.619+0000 7f62b2639640 1 --2- 172.21.15.83:0/3561315870 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f629806b4d0 0x7f629806d990 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f629c034000 tx=0x7f629c02f000 comp rx=0 tx=0).stop 2024-03-06T07:28:27.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.619+0000 7f62b2639640 1 -- 172.21.15.83:0/3561315870 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62ac0d27f0 msgr2=0x7f62ac1f68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.619+0000 7f62b2639640 1 --2- 172.21.15.83:0/3561315870 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62ac0d27f0 0x7f62ac1f68d0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f62a000e9b0 tx=0x7f62a000ee70 comp rx=0 tx=0).stop 2024-03-06T07:28:27.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.619+0000 7f62b2639640 1 -- 172.21.15.83:0/3561315870 shutdown_connections 2024-03-06T07:28:27.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.619+0000 7f62b2639640 1 --2- 172.21.15.83:0/3561315870 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f629806b4d0 0x7f629806d990 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.619+0000 7f62b2639640 1 --2- 172.21.15.83:0/3561315870 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f62ac15dfe0 0x7f62ac15eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.619+0000 7f62b2639640 1 --2- 172.21.15.83:0/3561315870 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62ac0d27f0 0x7f62ac1f68d0 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.619+0000 7f62b2639640 1 --2- 172.21.15.83:0/3561315870 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f62ac0cf830 0x7f62ac1f3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.619+0000 7f62b2639640 1 -- 172.21.15.83:0/3561315870 >> 172.21.15.83:0/3561315870 conn(0x7f62ac0d3cd0 msgr2=0x7f62ac0cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.619+0000 7f62b2639640 1 -- 172.21.15.83:0/3561315870 shutdown_connections 2024-03-06T07:28:27.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.619+0000 7f62b2639640 1 -- 172.21.15.83:0/3561315870 wait complete. 2024-03-06T07:28:27.636 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump purge_queue pq_executed 2024-03-06T07:28:27.752 INFO:teuthology.orchestra.run.smithi083.stdout:{"purge_queue":{"pq_executed":1}} 2024-03-06T07:28:27.752 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1 } } 2024-03-06T07:28:27.752 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:27.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.851+0000 7fa96149a640 1 Processor -- start 2024-03-06T07:28:27.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.851+0000 7fa96149a640 1 -- start start 2024-03-06T07:28:27.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.852+0000 7fa96149a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fa95c0d4760 0x7fa95c0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.852+0000 7fa96149a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa95c0d0a80 0x7fa95c0cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.852+0000 7fa96149a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa95c0d25d0 con 0x7fa95c0b4170 2024-03-06T07:28:27.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.852+0000 7fa96149a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa95c0d4bf0 con 0x7fa95c0d0a80 2024-03-06T07:28:27.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.852+0000 7fa96149a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa95c0ce9e0 con 0x7fa95c0d4760 2024-03-06T07:28:27.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.852+0000 7fa95b7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa95c0d0a80 0x7fa95c0cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.852+0000 7fa95b7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa95c0d0a80 0x7fa95c0cee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52250/0 (socket says 172.21.15.83:52250) 2024-03-06T07:28:27.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.852+0000 7fa95b7fe640 1 -- 172.21.15.83:0/2731372215 learned_addr learned my addr 172.21.15.83:0/2731372215 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:27.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.852+0000 7fa953fff640 1 --2- 172.21.15.83:0/2731372215 >> v2:172.21.15.183:3300/0 conn(0x7fa95c0d4760 0x7fa95c0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.852+0000 7fa95b7fe640 1 -- 172.21.15.83:0/2731372215 >> v2:172.21.15.183:3300/0 conn(0x7fa95c0d4760 msgr2=0x7fa95c0b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:27.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.852+0000 7fa95b7fe640 1 --2- 172.21.15.83:0/2731372215 >> v2:172.21.15.183:3300/0 conn(0x7fa95c0d4760 0x7fa95c0b6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.852+0000 7fa95b7fe640 1 -- 172.21.15.83:0/2731372215 >> v1:172.21.15.183:6789/0 conn(0x7fa95c0b4170 legacy=0x7fa95c0d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:27.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.853+0000 7fa95b7fe640 1 -- 172.21.15.83:0/2731372215 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa95c0d2870 con 0x7fa95c0d0a80 2024-03-06T07:28:27.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.854+0000 7fa95b7fe640 1 --2- 172.21.15.83:0/2731372215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa95c0d0a80 0x7fa95c0cee70 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fa95c08df30 tx=0x7fa95402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b75d68f1cdb6f53b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.854+0000 7fa95a7fc640 1 -- 172.21.15.83:0/2731372215 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa95402ca50 con 0x7fa95c0d0a80 2024-03-06T07:28:27.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.854+0000 7fa95a7fc640 1 -- 172.21.15.83:0/2731372215 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa95402cbf0 con 0x7fa95c0d0a80 2024-03-06T07:28:27.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.854+0000 7fa95a7fc640 1 -- 172.21.15.83:0/2731372215 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9540343f0 con 0x7fa95c0d0a80 2024-03-06T07:28:27.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.854+0000 7fa96149a640 1 -- 172.21.15.83:0/2731372215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa95c0d0a80 msgr2=0x7fa95c0cee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.854+0000 7fa96149a640 1 --2- 172.21.15.83:0/2731372215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa95c0d0a80 0x7fa95c0cee70 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fa95c08df30 tx=0x7fa95402beb0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.854+0000 7fa96149a640 1 -- 172.21.15.83:0/2731372215 shutdown_connections 2024-03-06T07:28:27.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.854+0000 7fa96149a640 1 --2- 172.21.15.83:0/2731372215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa95c0d0a80 0x7fa95c0cee70 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.854+0000 7fa96149a640 1 --2- 172.21.15.83:0/2731372215 >> v2:172.21.15.183:3300/0 conn(0x7fa95c0d4760 0x7fa95c0b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.854+0000 7fa96149a640 1 -- 172.21.15.83:0/2731372215 >> 172.21.15.83:0/2731372215 conn(0x7fa95c0d3c40 msgr2=0x7fa95c0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.854+0000 7fa96149a640 1 -- 172.21.15.83:0/2731372215 shutdown_connections 2024-03-06T07:28:27.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.854+0000 7fa96149a640 1 -- 172.21.15.83:0/2731372215 wait complete. 2024-03-06T07:28:27.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.856+0000 7fa96149a640 1 Processor -- start 2024-03-06T07:28:27.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.856+0000 7fa96149a640 1 -- start start 2024-03-06T07:28:27.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.857+0000 7fa96149a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa95c0b4170 0x7fa95c1541e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.857+0000 7fa96149a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa95c0d0a80 0x7fa95c0b0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.857+0000 7fa96149a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa95c09f150 0x7fa95c09f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.857+0000 7fa96149a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95c0b4dd0 con 0x7fa95c0b4170 2024-03-06T07:28:27.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.857+0000 7fa96149a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95c0b4550 con 0x7fa95c0d0a80 2024-03-06T07:28:27.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.857+0000 7fa96149a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa95c0b4f50 con 0x7fa95c09f150 2024-03-06T07:28:27.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.857+0000 7fa95affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa95c0b4170 0x7fa95c1541e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.858+0000 7fa95affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa95c0b4170 0x7fa95c1541e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52262/0 (socket says 172.21.15.83:52262) 2024-03-06T07:28:27.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.858+0000 7fa95affd640 1 -- 172.21.15.83:0/511053923 learned_addr learned my addr 172.21.15.83:0/511053923 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:27.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.858+0000 7fa95b7fe640 1 --2- 172.21.15.83:0/511053923 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa95c09f150 0x7fa95c09f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.858+0000 7fa953fff640 1 --2- 172.21.15.83:0/511053923 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa95c0d0a80 0x7fa95c0b0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.858+0000 7fa95affd640 1 -- 172.21.15.83:0/511053923 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa95c09f150 msgr2=0x7fa95c09f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.858+0000 7fa95affd640 1 --2- 172.21.15.83:0/511053923 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa95c09f150 0x7fa95c09f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.858+0000 7fa95affd640 1 -- 172.21.15.83:0/511053923 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa95c0d0a80 msgr2=0x7fa95c0b0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.858+0000 7fa95affd640 1 --2- 172.21.15.83:0/511053923 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa95c0d0a80 0x7fa95c0b0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.858+0000 7fa95affd640 1 -- 172.21.15.83:0/511053923 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa95c0b2340 con 0x7fa95c0b4170 2024-03-06T07:28:27.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.858+0000 7fa95affd640 1 --2- 172.21.15.83:0/511053923 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa95c0b4170 0x7fa95c1541e0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fa94c00c460 tx=0x7fa94c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.858+0000 7fa9537fe640 1 -- 172.21.15.83:0/511053923 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa94c016020 con 0x7fa95c0b4170 2024-03-06T07:28:27.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.858+0000 7fa9537fe640 1 -- 172.21.15.83:0/511053923 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa94c012870 con 0x7fa95c0b4170 2024-03-06T07:28:27.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.858+0000 7fa96149a640 1 -- 172.21.15.83:0/511053923 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa95c0b2620 con 0x7fa95c0b4170 2024-03-06T07:28:27.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.858+0000 7fa9537fe640 1 -- 172.21.15.83:0/511053923 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa94c012c60 con 0x7fa95c0b4170 2024-03-06T07:28:27.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.858+0000 7fa96149a640 1 -- 172.21.15.83:0/511053923 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa95c0b2be0 con 0x7fa95c0b4170 2024-03-06T07:28:27.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.859+0000 7fa96149a640 1 -- 172.21.15.83:0/511053923 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa95c08dc90 con 0x7fa95c0b4170 2024-03-06T07:28:27.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.860+0000 7fa9537fe640 1 -- 172.21.15.83:0/511053923 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa94c019020 con 0x7fa95c0b4170 2024-03-06T07:28:27.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.860+0000 7fa9537fe640 1 --2- 172.21.15.83:0/511053923 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa93c06b500 0x7fa93c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.860+0000 7fa9537fe640 1 -- 172.21.15.83:0/511053923 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fa94c08a0f0 con 0x7fa95c0b4170 2024-03-06T07:28:27.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.860+0000 7fa953fff640 1 --2- 172.21.15.83:0/511053923 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa93c06b500 0x7fa93c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.861+0000 7fa953fff640 1 --2- 172.21.15.83:0/511053923 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa93c06b500 0x7fa93c06d9c0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fa95c160870 tx=0x7fa944008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:27.865+0000 7fa9537fe640 1 -- 172.21.15.83:0/511053923 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa95c08dc90 con 0x7fa95c0b4170 2024-03-06T07:28:28.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.081+0000 7fa96149a640 1 -- 172.21.15.83:0/511053923 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa95c04f330 con 0x7fa95c0b4170 2024-03-06T07:28:28.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.081+0000 7fa9537fe640 1 -- 172.21.15.83:0/511053923 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4984 (secure 0 0 0) 0x7fa94c020020 con 0x7fa95c0b4170 2024-03-06T07:28:28.084 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:28:28.084 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7051,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/101642171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":101642171},{"type":"v1","addr":"172.21.15.183:6833","nonce":101642171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7057,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3146884900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":3146884900},{"type":"v1","addr":"172.21.15.183:6835","nonce":3146884900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8208,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1922041621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1922041621},{"type":"v1","addr":"172.21.15.83:6835","nonce":1922041621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.491022+0000","modified":"2024-03-06T07:27:37.006649+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8229":{"gid":8229,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/2003687153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2003687153},{"type":"v1","addr":"172.21.15.83:6837","nonce":2003687153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8229,"qdb_cluster":[8229]},"id":8}]} 2024-03-06T07:28:28.084 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 66 2024-03-06T07:28:28.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.083+0000 7fa96149a640 1 -- 172.21.15.83:0/511053923 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa93c06b500 msgr2=0x7fa93c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.083+0000 7fa96149a640 1 --2- 172.21.15.83:0/511053923 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa93c06b500 0x7fa93c06d9c0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fa95c160870 tx=0x7fa944008040 comp rx=0 tx=0).stop 2024-03-06T07:28:28.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.083+0000 7fa96149a640 1 -- 172.21.15.83:0/511053923 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa95c0b4170 msgr2=0x7fa95c1541e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.083+0000 7fa96149a640 1 --2- 172.21.15.83:0/511053923 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa95c0b4170 0x7fa95c1541e0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fa94c00c460 tx=0x7fa94c00c920 comp rx=0 tx=0).stop 2024-03-06T07:28:28.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.083+0000 7fa96149a640 1 -- 172.21.15.83:0/511053923 shutdown_connections 2024-03-06T07:28:28.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.083+0000 7fa96149a640 1 --2- 172.21.15.83:0/511053923 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa93c06b500 0x7fa93c06d9c0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.083+0000 7fa96149a640 1 --2- 172.21.15.83:0/511053923 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa95c09f150 0x7fa95c09f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.083+0000 7fa96149a640 1 --2- 172.21.15.83:0/511053923 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa95c0d0a80 0x7fa95c0b0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.083+0000 7fa96149a640 1 --2- 172.21.15.83:0/511053923 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa95c0b4170 0x7fa95c1541e0 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.083+0000 7fa96149a640 1 -- 172.21.15.83:0/511053923 >> 172.21.15.83:0/511053923 conn(0x7fa95c0d3c40 msgr2=0x7fa95c0b7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:28.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.084+0000 7fa96149a640 1 -- 172.21.15.83:0/511053923 shutdown_connections 2024-03-06T07:28:28.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.084+0000 7fa96149a640 1 -- 172.21.15.83:0/511053923 wait complete. 2024-03-06T07:28:28.096 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache 2024-03-06T07:28:28.211 INFO:teuthology.orchestra.run.smithi083.stdout:{"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3,"strays_enqueued":1,"strays_reintegrated":2,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0}} 2024-03-06T07:28:28.212 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3, "strays_enqueued": 1, "strays_migrated": 0, "strays_reintegrated": 2 } } 2024-03-06T07:28:28.212 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:28.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.310+0000 7f248ca9a640 1 Processor -- start 2024-03-06T07:28:28.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.310+0000 7f248ca9a640 1 -- start start 2024-03-06T07:28:28.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.311+0000 7f248ca9a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f248815dfe0 0x7f24880cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.311+0000 7f248ca9a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f24880d29c0 0x7f24880b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.311+0000 7f248ca9a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24880b3430 con 0x7f24880cf830 2024-03-06T07:28:28.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.311+0000 7f248ca9a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24880b31e0 con 0x7f248815dfe0 2024-03-06T07:28:28.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.311+0000 7f248ca9a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24880b3d60 con 0x7f24880d29c0 2024-03-06T07:28:28.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.311+0000 7f248659c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f248815dfe0 0x7f24880cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.311+0000 7f2485d9b640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f24880cf830 0x7f24880d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49082/0 (socket says 172.21.15.83:49082) 2024-03-06T07:28:28.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.311+0000 7f2485d9b640 1 -- 172.21.15.83:0/586488892 learned_addr learned my addr 172.21.15.83:0/586488892 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:28:28.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.311+0000 7f2486d9d640 1 --2- 172.21.15.83:0/586488892 >> v2:172.21.15.183:3300/0 conn(0x7f24880d29c0 0x7f24880b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.311+0000 7f248659c640 1 -- 172.21.15.83:0/586488892 >> v2:172.21.15.183:3300/0 conn(0x7f24880d29c0 msgr2=0x7f24880b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:28.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.311+0000 7f248659c640 1 --2- 172.21.15.83:0/586488892 >> v2:172.21.15.183:3300/0 conn(0x7f24880d29c0 0x7f24880b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.311+0000 7f248659c640 1 -- 172.21.15.83:0/586488892 >> v1:172.21.15.83:6789/0 conn(0x7f24880cf830 legacy=0x7f24880d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.311+0000 7f248559a640 1 -- 172.21.15.83:0/586488892 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2019473565 0 0) 0x7f24880b3430 con 0x7f24880cf830 2024-03-06T07:28:28.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.311+0000 7f248659c640 1 -- 172.21.15.83:0/586488892 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24880b3710 con 0x7f248815dfe0 2024-03-06T07:28:28.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.311+0000 7f2486d9d640 1 --2- 172.21.15.83:0/586488892 >> v2:172.21.15.183:3300/0 conn(0x7f24880d29c0 0x7f24880b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:28.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.312+0000 7f248659c640 1 --2- 172.21.15.83:0/586488892 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f248815dfe0 0x7f24880cf2f0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f24880b3cd0 tx=0x7f247802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c5221fd17239f04 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.312+0000 7f248559a640 1 -- 172.21.15.83:0/586488892 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2478035020 con 0x7f248815dfe0 2024-03-06T07:28:28.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.312+0000 7f248559a640 1 -- 172.21.15.83:0/586488892 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2478031830 con 0x7f248815dfe0 2024-03-06T07:28:28.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.312+0000 7f248559a640 1 -- 172.21.15.83:0/586488892 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2478031c00 con 0x7f248815dfe0 2024-03-06T07:28:28.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.313+0000 7f248ca9a640 1 -- 172.21.15.83:0/586488892 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f248815dfe0 msgr2=0x7f24880cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.313+0000 7f248ca9a640 1 --2- 172.21.15.83:0/586488892 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f248815dfe0 0x7f24880cf2f0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f24880b3cd0 tx=0x7f247802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.313+0000 7f248ca9a640 1 -- 172.21.15.83:0/586488892 shutdown_connections 2024-03-06T07:28:28.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.313+0000 7f248ca9a640 1 --2- 172.21.15.83:0/586488892 >> v2:172.21.15.183:3300/0 conn(0x7f24880d29c0 0x7f24880b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.313+0000 7f248ca9a640 1 --2- 172.21.15.83:0/586488892 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f248815dfe0 0x7f24880cf2f0 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.313+0000 7f248ca9a640 1 -- 172.21.15.83:0/586488892 >> 172.21.15.83:0/586488892 conn(0x7f24880d3cd0 msgr2=0x7f24880d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:28.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.313+0000 7f248ca9a640 1 -- 172.21.15.83:0/586488892 shutdown_connections 2024-03-06T07:28:28.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.313+0000 7f248ca9a640 1 -- 172.21.15.83:0/586488892 wait complete. 2024-03-06T07:28:28.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.314+0000 7f248ca9a640 1 Processor -- start 2024-03-06T07:28:28.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.314+0000 7f248ca9a640 1 -- start start 2024-03-06T07:28:28.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.315+0000 7f248ca9a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f24880cf830 0x7f2488160600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.315+0000 7f248ca9a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f24880d29c0 0x7f24880cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.315+0000 7f248ca9a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f248815dfe0 0x7f24880cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.315+0000 7f248ca9a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24880d1df0 con 0x7f248815dfe0 2024-03-06T07:28:28.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.315+0000 7f248ca9a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24880d1c70 con 0x7f24880d29c0 2024-03-06T07:28:28.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.315+0000 7f248ca9a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f248815e3c0 con 0x7f24880cf830 2024-03-06T07:28:28.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.315+0000 7f2486d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f248815dfe0 0x7f24880cb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.315+0000 7f2486d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f248815dfe0 0x7f24880cb2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52282/0 (socket says 172.21.15.83:52282) 2024-03-06T07:28:28.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.315+0000 7f2486d9d640 1 -- 172.21.15.83:0/3038254838 learned_addr learned my addr 172.21.15.83:0/3038254838 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:28.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.315+0000 7f248659c640 1 --2- 172.21.15.83:0/3038254838 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f24880cf830 0x7f2488160600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.315+0000 7f2485d9b640 1 --2- 172.21.15.83:0/3038254838 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f24880d29c0 0x7f24880cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.315+0000 7f2486d9d640 1 -- 172.21.15.83:0/3038254838 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f24880cf830 msgr2=0x7f2488160600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.315+0000 7f2486d9d640 1 --2- 172.21.15.83:0/3038254838 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f24880cf830 0x7f2488160600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.315+0000 7f2486d9d640 1 -- 172.21.15.83:0/3038254838 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f24880d29c0 msgr2=0x7f24880cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.315+0000 7f2486d9d640 1 --2- 172.21.15.83:0/3038254838 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f24880d29c0 0x7f24880cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.315+0000 7f2486d9d640 1 -- 172.21.15.83:0/3038254838 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f248815f560 con 0x7f248815dfe0 2024-03-06T07:28:28.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.315+0000 7f2486d9d640 1 --2- 172.21.15.83:0/3038254838 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f248815dfe0 0x7f24880cb2e0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f248000ce80 tx=0x7f248000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.316+0000 7f2476ffd640 1 -- 172.21.15.83:0/3038254838 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248001a070 con 0x7f248815dfe0 2024-03-06T07:28:28.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.316+0000 7f2476ffd640 1 -- 172.21.15.83:0/3038254838 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f248000d880 con 0x7f248815dfe0 2024-03-06T07:28:28.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.316+0000 7f248ca9a640 1 -- 172.21.15.83:0/3038254838 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f248815f840 con 0x7f248815dfe0 2024-03-06T07:28:28.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.316+0000 7f2476ffd640 1 -- 172.21.15.83:0/3038254838 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248000dc70 con 0x7f248815dfe0 2024-03-06T07:28:28.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.316+0000 7f248ca9a640 1 -- 172.21.15.83:0/3038254838 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f248815fe00 con 0x7f248815dfe0 2024-03-06T07:28:28.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.317+0000 7f248ca9a640 1 -- 172.21.15.83:0/3038254838 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f248804f330 con 0x7f248815dfe0 2024-03-06T07:28:28.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.317+0000 7f2476ffd640 1 -- 172.21.15.83:0/3038254838 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f248001d070 con 0x7f248815dfe0 2024-03-06T07:28:28.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.317+0000 7f2476ffd640 1 --2- 172.21.15.83:0/3038254838 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f246806b4d0 0x7f246806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.318+0000 7f2476ffd640 1 -- 172.21.15.83:0/3038254838 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f248008aa70 con 0x7f248815dfe0 2024-03-06T07:28:28.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.318+0000 7f248659c640 1 --2- 172.21.15.83:0/3038254838 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f246806b4d0 0x7f246806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.318+0000 7f248659c640 1 --2- 172.21.15.83:0/3038254838 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f246806b4d0 0x7f246806d990 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f2478036000 tx=0x7f247803c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.322+0000 7f2476ffd640 1 -- 172.21.15.83:0/3038254838 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f24800591b0 con 0x7f248815dfe0 2024-03-06T07:28:28.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.536+0000 7f248ca9a640 1 -- 172.21.15.83:0/3038254838 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24880b48e0 con 0x7f248815dfe0 2024-03-06T07:28:28.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.536+0000 7f2476ffd640 1 -- 172.21.15.83:0/3038254838 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f248005d250 con 0x7f248815dfe0 2024-03-06T07:28:28.538 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:28:28.538 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7051,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/101642171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":101642171},{"type":"v1","addr":"172.21.15.183:6833","nonce":101642171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7057,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3146884900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":3146884900},{"type":"v1","addr":"172.21.15.183:6835","nonce":3146884900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8208,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1922041621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1922041621},{"type":"v1","addr":"172.21.15.83:6835","nonce":1922041621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.491022+0000","modified":"2024-03-06T07:27:37.006649+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8229":{"gid":8229,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/2003687153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2003687153},{"type":"v1","addr":"172.21.15.83:6837","nonce":2003687153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8229,"qdb_cluster":[8229]},"id":8}]} 2024-03-06T07:28:28.539 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 66 2024-03-06T07:28:28.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.538+0000 7f248ca9a640 1 -- 172.21.15.83:0/3038254838 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f246806b4d0 msgr2=0x7f246806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.538+0000 7f248ca9a640 1 --2- 172.21.15.83:0/3038254838 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f246806b4d0 0x7f246806d990 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f2478036000 tx=0x7f247803c040 comp rx=0 tx=0).stop 2024-03-06T07:28:28.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.538+0000 7f248ca9a640 1 -- 172.21.15.83:0/3038254838 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f248815dfe0 msgr2=0x7f24880cb2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.538+0000 7f248ca9a640 1 --2- 172.21.15.83:0/3038254838 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f248815dfe0 0x7f24880cb2e0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f248000ce80 tx=0x7f248000a710 comp rx=0 tx=0).stop 2024-03-06T07:28:28.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.538+0000 7f248ca9a640 1 -- 172.21.15.83:0/3038254838 shutdown_connections 2024-03-06T07:28:28.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.538+0000 7f248ca9a640 1 --2- 172.21.15.83:0/3038254838 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f246806b4d0 0x7f246806d990 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.538+0000 7f248ca9a640 1 --2- 172.21.15.83:0/3038254838 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f248815dfe0 0x7f24880cb2e0 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.538+0000 7f248ca9a640 1 --2- 172.21.15.83:0/3038254838 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f24880d29c0 0x7f24880cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.538+0000 7f248ca9a640 1 --2- 172.21.15.83:0/3038254838 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f24880cf830 0x7f2488160600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.538+0000 7f248ca9a640 1 -- 172.21.15.83:0/3038254838 >> 172.21.15.83:0/3038254838 conn(0x7f24880d3cd0 msgr2=0x7f24880b4e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:28.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.538+0000 7f248ca9a640 1 -- 172.21.15.83:0/3038254838 shutdown_connections 2024-03-06T07:28:28.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.538+0000 7f248ca9a640 1 -- 172.21.15.83:0/3038254838 wait complete. 2024-03-06T07:28:28.551 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump purge_queue 2024-03-06T07:28:28.666 INFO:teuthology.orchestra.run.smithi083.stdout:{"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":2,"pq_executing":0,"pq_executing_high_water":1,"pq_executed_ops":2,"pq_executed":1,"pq_item_in_journal":0}} 2024-03-06T07:28:28.667 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1, "pq_executed_ops": 2, "pq_executing": 0, "pq_executing_high_water": 1, "pq_executing_ops": 0, "pq_executing_ops_high_water": 2, "pq_item_in_journal": 0 } } 2024-03-06T07:28:28.667 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data ls 2024-03-06T07:28:28.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.705+0000 7f9178036f40 1 Processor -- start 2024-03-06T07:28:28.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.705+0000 7f9178036f40 1 -- start start 2024-03-06T07:28:28.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.705+0000 7f9178036f40 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c7ca75d640 0x55c7ca713b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.705+0000 7f9178036f40 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55c7ca75df90 0x55c7ca7162a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.705+0000 7f9178036f40 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7ca7ec8a0 0x55c7ca715290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.705+0000 7f9178036f40 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7ca7ebc20 con 0x55c7ca7ec8a0 2024-03-06T07:28:28.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.705+0000 7f9178036f40 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7ca6bcb70 con 0x55c7ca75d640 2024-03-06T07:28:28.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.705+0000 7f9178036f40 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c7ca7ebec0 con 0x55c7ca75df90 2024-03-06T07:28:28.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.706+0000 7f9175bcf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7ca7ec8a0 0x55c7ca715290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.706+0000 7f9175bcf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7ca7ec8a0 0x55c7ca715290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52294/0 (socket says 172.21.15.83:52294) 2024-03-06T07:28:28.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.706+0000 7f9175bcf640 1 -- 172.21.15.83:0/894381834 learned_addr learned my addr 172.21.15.83:0/894381834 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:28.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.706+0000 7f916ffff640 1 --2- 172.21.15.83:0/894381834 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55c7ca75df90 0x55c7ca7162a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.706+0000 7f91753ce640 1 --2- 172.21.15.83:0/894381834 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c7ca75d640 0x55c7ca713b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.706+0000 7f9175bcf640 1 -- 172.21.15.83:0/894381834 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55c7ca75df90 msgr2=0x55c7ca7162a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:28.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.706+0000 7f9175bcf640 1 --2- 172.21.15.83:0/894381834 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55c7ca75df90 0x55c7ca7162a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.706+0000 7f9175bcf640 1 -- 172.21.15.83:0/894381834 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c7ca75d640 msgr2=0x55c7ca713b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:28.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.706+0000 7f9175bcf640 1 --2- 172.21.15.83:0/894381834 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c7ca75d640 0x55c7ca713b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.706+0000 7f9175bcf640 1 -- 172.21.15.83:0/894381834 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c7ca75b100 con 0x55c7ca7ec8a0 2024-03-06T07:28:28.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.706+0000 7f9175bcf640 1 --2- 172.21.15.83:0/894381834 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7ca7ec8a0 0x55c7ca715290 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f917000a8b0 tx=0x7f9170005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac208c2d69268b67 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.706+0000 7f916f7fe640 1 -- 172.21.15.83:0/894381834 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917000d960 con 0x55c7ca7ec8a0 2024-03-06T07:28:28.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.706+0000 7f916f7fe640 1 -- 172.21.15.83:0/894381834 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f917000db00 con 0x55c7ca7ec8a0 2024-03-06T07:28:28.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.706+0000 7f916f7fe640 1 -- 172.21.15.83:0/894381834 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9170012450 con 0x55c7ca7ec8a0 2024-03-06T07:28:28.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.707+0000 7f9178036f40 1 -- 172.21.15.83:0/894381834 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7ca7ec8a0 msgr2=0x55c7ca715290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.707+0000 7f9178036f40 1 --2- 172.21.15.83:0/894381834 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7ca7ec8a0 0x55c7ca715290 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f917000a8b0 tx=0x7f9170005d80 comp rx=0 tx=0).stop 2024-03-06T07:28:28.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.707+0000 7f9178036f40 1 -- 172.21.15.83:0/894381834 shutdown_connections 2024-03-06T07:28:28.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.707+0000 7f9178036f40 1 --2- 172.21.15.83:0/894381834 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7ca7ec8a0 0x55c7ca715290 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.707+0000 7f9178036f40 1 --2- 172.21.15.83:0/894381834 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55c7ca75df90 0x55c7ca7162a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.707+0000 7f9178036f40 1 --2- 172.21.15.83:0/894381834 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c7ca75d640 0x55c7ca713b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.707+0000 7f9178036f40 1 -- 172.21.15.83:0/894381834 >> 172.21.15.83:0/894381834 conn(0x55c7ca741340 msgr2=0x55c7ca7e89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:28.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.707+0000 7f9178036f40 1 -- 172.21.15.83:0/894381834 shutdown_connections 2024-03-06T07:28:28.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.707+0000 7f9178036f40 1 -- 172.21.15.83:0/894381834 wait complete. 2024-03-06T07:28:28.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.707+0000 7f9178036f40 1 Processor -- start 2024-03-06T07:28:28.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.707+0000 7f9178036f40 1 -- start start 2024-03-06T07:28:28.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.708+0000 7f9178036f40 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c7ca75d640 0x55c7ca7df8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.708+0000 7f9178036f40 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55c7ca75df90 0x55c7ca7e8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.708+0000 7f9178036f40 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7ca7ec8a0 0x55c7ca714310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.708+0000 7f9178036f40 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7ca7e45c0 con 0x55c7ca7ec8a0 2024-03-06T07:28:28.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.708+0000 7f9178036f40 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7ca7de840 con 0x55c7ca75d640 2024-03-06T07:28:28.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.708+0000 7f9178036f40 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c7ca7deb20 con 0x55c7ca75df90 2024-03-06T07:28:28.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.708+0000 7f9175bcf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7ca7ec8a0 0x55c7ca714310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.708+0000 7f9175bcf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7ca7ec8a0 0x55c7ca714310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52300/0 (socket says 172.21.15.83:52300) 2024-03-06T07:28:28.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.708+0000 7f9175bcf640 1 -- 172.21.15.83:0/2816044254 learned_addr learned my addr 172.21.15.83:0/2816044254 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:28.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.708+0000 7f916ffff640 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55c7ca75df90 0x55c7ca7e8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.708+0000 7f91753ce640 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c7ca75d640 0x55c7ca7df8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.708+0000 7f9175bcf640 1 -- 172.21.15.83:0/2816044254 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55c7ca75df90 msgr2=0x55c7ca7e8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.708+0000 7f9175bcf640 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55c7ca75df90 0x55c7ca7e8b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.708+0000 7f9175bcf640 1 -- 172.21.15.83:0/2816044254 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c7ca75d640 msgr2=0x55c7ca7df8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.708+0000 7f9175bcf640 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c7ca75d640 0x55c7ca7df8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.708+0000 7f9175bcf640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c7ca7dee00 con 0x55c7ca7ec8a0 2024-03-06T07:28:28.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.708+0000 7f916ffff640 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55c7ca75df90 0x55c7ca7e8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:28.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.708+0000 7f91753ce640 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c7ca75d640 0x55c7ca7df8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:28.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.708+0000 7f9175bcf640 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7ca7ec8a0 0x55c7ca714310 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f9170005dc0 tx=0x7f91700033a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.709+0000 7f916e7fc640 1 -- 172.21.15.83:0/2816044254 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9170021070 con 0x55c7ca7ec8a0 2024-03-06T07:28:28.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.709+0000 7f916e7fc640 1 -- 172.21.15.83:0/2816044254 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9170026680 con 0x55c7ca7ec8a0 2024-03-06T07:28:28.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.709+0000 7f9178036f40 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55c7ca7df0e0 con 0x55c7ca7ec8a0 2024-03-06T07:28:28.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.709+0000 7f916e7fc640 1 -- 172.21.15.83:0/2816044254 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917000d960 con 0x55c7ca7ec8a0 2024-03-06T07:28:28.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.709+0000 7f9178036f40 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c7ca714850 con 0x55c7ca7ec8a0 2024-03-06T07:28:28.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.710+0000 7f916e7fc640 1 -- 172.21.15.83:0/2816044254 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f917001d440 con 0x55c7ca7ec8a0 2024-03-06T07:28:28.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.710+0000 7f916e7fc640 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9158069ee0 0x7f915806c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.711+0000 7f916e7fc640 1 -- 172.21.15.83:0/2816044254 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f917008ba00 con 0x55c7ca7ec8a0 2024-03-06T07:28:28.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.711+0000 7f91753ce640 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9158069ee0 0x7f915806c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.711+0000 7f9178036f40 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] conn(0x55c7ca887e40 0x55c7ca7e68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.711+0000 7f9178036f40 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] -- osd_op(unknown.0.0:1 17.0 17:00000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55c7ca7e6de0 con 0x55c7ca887e40 2024-03-06T07:28:28.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.711+0000 7f91753ce640 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9158069ee0 0x7f915806c3a0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f9168000f60 tx=0x7f9168001770 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.711+0000 7f916ffff640 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] conn(0x55c7ca887e40 0x55c7ca7e68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.711+0000 7f916ffff640 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] conn(0x55c7ca887e40 0x55c7ca7e68a0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.712+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 <== osd.7 v2:172.21.15.183:6820/3336566695 1 ==== osd_op_reply(1 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f916004daa0 con 0x55c7ca887e40 2024-03-06T07:28:28.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.712+0000 7f916ffff640 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.83:6814/1822925240,v1:172.21.15.83:6817/1822925240] conn(0x7f9160061430 0x7f9160061890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.712+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.83:6814/1822925240,v1:172.21.15.83:6817/1822925240] -- osd_op(unknown.0.0:2 17.20 17:04000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55c7ca7e6de0 con 0x7f9160061430 2024-03-06T07:28:28.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.712+0000 7f91753ce640 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.83:6814/1822925240,v1:172.21.15.83:6817/1822925240] conn(0x7f9160061430 0x7f9160061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.712+0000 7f91753ce640 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.83:6814/1822925240,v1:172.21.15.83:6817/1822925240] conn(0x7f9160061430 0x7f9160061890 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.713+0000 7f91753ce640 1 -- 172.21.15.83:0/2816044254 <== osd.1 v2:172.21.15.83:6814/1822925240 1 ==== osd_op_reply(2 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f916801a040 con 0x7f9160061430 2024-03-06T07:28:28.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.713+0000 7f91753ce640 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] conn(0x7f916801d8d0 0x7f91680103a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.713+0000 7f91753ce640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] -- osd_op(unknown.0.0:3 17.10 17:08000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55c7ca7e6de0 con 0x7f916801d8d0 2024-03-06T07:28:28.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.713+0000 7f916ffff640 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] conn(0x7f916801d8d0 0x7f91680103a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.713+0000 7f916ffff640 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] conn(0x7f916801d8d0 0x7f91680103a0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.713+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 <== osd.3 v2:172.21.15.83:6815/2656563756 1 ==== osd_op_reply(3 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f916004daa0 con 0x7f916801d8d0 2024-03-06T07:28:28.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.713+0000 7f916ffff640 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.183:6808/2957306877,v1:172.21.15.183:6809/2957306877] conn(0x7f9160065ad0 0x7f91600643e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.713+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.183:6808/2957306877,v1:172.21.15.183:6809/2957306877] -- osd_op(unknown.0.0:4 17.30 17:0c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55c7ca7e6de0 con 0x7f9160065ad0 2024-03-06T07:28:28.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.714+0000 7f9175bcf640 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.183:6808/2957306877,v1:172.21.15.183:6809/2957306877] conn(0x7f9160065ad0 0x7f91600643e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.714+0000 7f9175bcf640 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.183:6808/2957306877,v1:172.21.15.183:6809/2957306877] conn(0x7f9160065ad0 0x7f91600643e0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.714+0000 7f9175bcf640 1 -- 172.21.15.83:0/2816044254 <== osd.5 v2:172.21.15.183:6808/2957306877 1 ==== osd_op_reply(4 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f917008b040 con 0x7f9160065ad0 2024-03-06T07:28:28.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.714+0000 7f9175bcf640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] -- osd_op(unknown.0.0:5 17.8 17:10000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55c7ca7e6de0 con 0x55c7ca887e40 2024-03-06T07:28:28.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.715+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 <== osd.7 v2:172.21.15.183:6820/3336566695 2 ==== osd_op_reply(5 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f916004daa0 con 0x55c7ca887e40 2024-03-06T07:28:28.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.715+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] -- osd_op(unknown.0.0:6 17.28 17:14000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55c7ca7e6de0 con 0x7f916801d8d0 2024-03-06T07:28:28.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.715+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 <== osd.3 v2:172.21.15.83:6815/2656563756 2 ==== osd_op_reply(6 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f916004daa0 con 0x7f916801d8d0 2024-03-06T07:28:28.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.715+0000 7f916ffff640 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] conn(0x7f91600603c0 0x7f9160060840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.715+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] -- osd_op(unknown.0.0:7 17.18 17:18000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f91600603c0 2024-03-06T07:28:28.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.715+0000 7f91753ce640 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] conn(0x7f91600603c0 0x7f9160060840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.715+0000 7f91753ce640 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] conn(0x7f91600603c0 0x7f9160060840 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.716+0000 7f91753ce640 1 -- 172.21.15.83:0/2816044254 <== osd.0 v2:172.21.15.83:6808/1635427425 1 ==== osd_op_reply(7 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f916801a040 con 0x7f91600603c0 2024-03-06T07:28:28.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.716+0000 7f91753ce640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] -- osd_op(unknown.0.0:8 17.38 17:1c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x55c7ca887e40 2024-03-06T07:28:28.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.716+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 <== osd.7 v2:172.21.15.183:6820/3336566695 3 ==== osd_op_reply(8 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f916004daa0 con 0x55c7ca887e40 2024-03-06T07:28:28.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.716+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] -- osd_op(unknown.0.0:9 17.4 17:20000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f91600603c0 2024-03-06T07:28:28.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.716+0000 7f91753ce640 1 -- 172.21.15.83:0/2816044254 <== osd.0 v2:172.21.15.83:6808/1635427425 2 ==== osd_op_reply(9 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f916801a040 con 0x7f91600603c0 2024-03-06T07:28:28.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.716+0000 7f91753ce640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] -- osd_op(unknown.0.0:10 17.24 17:24000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f916801d8d0 2024-03-06T07:28:28.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.717+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 <== osd.3 v2:172.21.15.83:6815/2656563756 3 ==== osd_op_reply(10 [pgnls start_epoch 116 out=49b] v110'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f916004daa0 con 0x7f916801d8d0 2024-03-06T07:28:28.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.717+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] -- osd_op(unknown.0.0:11 17.14 17:28000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f916801d8d0 2024-03-06T07:28:28.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.717+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 <== osd.3 v2:172.21.15.83:6815/2656563756 4 ==== osd_op_reply(11 [pgnls start_epoch 116 out=49b] v110'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f916004daa0 con 0x7f916801d8d0 2024-03-06T07:28:28.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.717+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] -- osd_op(unknown.0.0:12 17.34 17:2c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f91600603c0 2024-03-06T07:28:28.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.717+0000 7f91753ce640 1 -- 172.21.15.83:0/2816044254 <== osd.0 v2:172.21.15.83:6808/1635427425 3 ==== osd_op_reply(12 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f916801a040 con 0x7f91600603c0 2024-03-06T07:28:28.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.717+0000 7f91753ce640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] -- osd_op(unknown.0.0:13 17.c 17:30000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x55c7ca887e40 2024-03-06T07:28:28.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.717+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 <== osd.7 v2:172.21.15.183:6820/3336566695 4 ==== osd_op_reply(13 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f916004daa0 con 0x55c7ca887e40 2024-03-06T07:28:28.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.717+0000 7f916ffff640 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] conn(0x7f9160074900 0x7f9160071370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.717+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] -- osd_op(unknown.0.0:14 17.2c 17:34000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f9160074900 2024-03-06T07:28:28.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.718+0000 7f916ffff640 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] conn(0x7f9160074900 0x7f9160071370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.718+0000 7f916ffff640 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] conn(0x7f9160074900 0x7f9160071370 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.718+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 <== osd.4 v2:172.21.15.183:6816/3256618830 1 ==== osd_op_reply(14 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f916004daa0 con 0x7f9160074900 2024-03-06T07:28:28.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.718+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.83:6814/1822925240,v1:172.21.15.83:6817/1822925240] -- osd_op(unknown.0.0:15 17.1c 17:38000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f9160061430 2024-03-06T07:28:28.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.719+0000 7f91753ce640 1 -- 172.21.15.83:0/2816044254 <== osd.1 v2:172.21.15.83:6814/1822925240 2 ==== osd_op_reply(15 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f916801a040 con 0x7f9160061430 2024-03-06T07:28:28.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.719+0000 7f91753ce640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] -- osd_op(unknown.0.0:16 17.3c 17:3c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f91600603c0 2024-03-06T07:28:28.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.719+0000 7f91753ce640 1 -- 172.21.15.83:0/2816044254 <== osd.0 v2:172.21.15.83:6808/1635427425 4 ==== osd_op_reply(16 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f916801a040 con 0x7f91600603c0 2024-03-06T07:28:28.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.719+0000 7f91753ce640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] -- osd_op(unknown.0.0:17 17.2 17:40000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f9160074900 2024-03-06T07:28:28.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.719+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 <== osd.4 v2:172.21.15.183:6816/3256618830 2 ==== osd_op_reply(17 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f916004daa0 con 0x7f9160074900 2024-03-06T07:28:28.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.719+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.183:6808/2957306877,v1:172.21.15.183:6809/2957306877] -- osd_op(unknown.0.0:18 17.22 17:44000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f9160065ad0 2024-03-06T07:28:28.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.720+0000 7f9175bcf640 1 -- 172.21.15.83:0/2816044254 <== osd.5 v2:172.21.15.183:6808/2957306877 2 ==== osd_op_reply(18 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f917008b040 con 0x7f9160065ad0 2024-03-06T07:28:28.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.720+0000 7f9175bcf640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] -- osd_op(unknown.0.0:19 17.12 17:48000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f916801d8d0 2024-03-06T07:28:28.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.720+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 <== osd.3 v2:172.21.15.83:6815/2656563756 5 ==== osd_op_reply(19 [pgnls start_epoch 116 out=49b] v110'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f916004daa0 con 0x7f916801d8d0 2024-03-06T07:28:28.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.720+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] -- osd_op(unknown.0.0:20 17.32 17:4c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x55c7ca887e40 2024-03-06T07:28:28.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.721+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 <== osd.7 v2:172.21.15.183:6820/3336566695 5 ==== osd_op_reply(20 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f916004daa0 con 0x55c7ca887e40 2024-03-06T07:28:28.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.721+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] -- osd_op(unknown.0.0:21 17.a 17:50000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x55c7ca887e40 2024-03-06T07:28:28.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.721+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 <== osd.7 v2:172.21.15.183:6820/3336566695 6 ==== osd_op_reply(21 [pgnls start_epoch 116 out=49b] v110'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f916004daa0 con 0x55c7ca887e40 2024-03-06T07:28:28.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.721+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] -- osd_op(unknown.0.0:22 17.2a 17:54000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f916801d8d0 2024-03-06T07:28:28.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.721+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 <== osd.3 v2:172.21.15.83:6815/2656563756 6 ==== osd_op_reply(22 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f916004daa0 con 0x7f916801d8d0 2024-03-06T07:28:28.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.721+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] -- osd_op(unknown.0.0:23 17.1a 17:58000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f91600603c0 2024-03-06T07:28:28.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.721+0000 7f91753ce640 1 -- 172.21.15.83:0/2816044254 <== osd.0 v2:172.21.15.83:6808/1635427425 5 ==== osd_op_reply(23 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f916801a040 con 0x7f91600603c0 2024-03-06T07:28:28.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.721+0000 7f91753ce640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.83:6814/1822925240,v1:172.21.15.83:6817/1822925240] -- osd_op(unknown.0.0:24 17.3a 17:5c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f9160061430 2024-03-06T07:28:28.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.722+0000 7f91753ce640 1 -- 172.21.15.83:0/2816044254 <== osd.1 v2:172.21.15.83:6814/1822925240 3 ==== osd_op_reply(24 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f916801a040 con 0x7f9160061430 2024-03-06T07:28:28.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.722+0000 7f91753ce640 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.183:6800/2649185225,v1:172.21.15.183:6801/2649185225] conn(0x7f9168020950 0x7f9168033440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.722+0000 7f91753ce640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.183:6800/2649185225,v1:172.21.15.183:6801/2649185225] -- osd_op(unknown.0.0:25 17.6 17:60000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f9168020950 2024-03-06T07:28:28.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.722+0000 7f9175bcf640 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.183:6800/2649185225,v1:172.21.15.183:6801/2649185225] conn(0x7f9168020950 0x7f9168033440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.722+0000 7f9175bcf640 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.183:6800/2649185225,v1:172.21.15.183:6801/2649185225] conn(0x7f9168020950 0x7f9168033440 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.723+0000 7f9175bcf640 1 -- 172.21.15.83:0/2816044254 <== osd.6 v2:172.21.15.183:6800/2649185225 1 ==== osd_op_reply(25 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f917008b040 con 0x7f9168020950 2024-03-06T07:28:28.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.723+0000 7f9175bcf640 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.83:6800/2591342133,v1:172.21.15.83:6801/2591342133] conn(0x7f9170012070 0x7f9170058fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.723+0000 7f9175bcf640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.83:6800/2591342133,v1:172.21.15.83:6801/2591342133] -- osd_op(unknown.0.0:26 17.26 17:64000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f9170012070 2024-03-06T07:28:28.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.723+0000 7f91753ce640 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.83:6800/2591342133,v1:172.21.15.83:6801/2591342133] conn(0x7f9170012070 0x7f9170058fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.723+0000 7f91753ce640 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.83:6800/2591342133,v1:172.21.15.83:6801/2591342133] conn(0x7f9170012070 0x7f9170058fe0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.723+0000 7f91753ce640 1 -- 172.21.15.83:0/2816044254 <== osd.2 v2:172.21.15.83:6800/2591342133 1 ==== osd_op_reply(26 [pgnls start_epoch 116 out=49b] v110'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f916801a040 con 0x7f9170012070 2024-03-06T07:28:28.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.723+0000 7f91753ce640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.183:6808/2957306877,v1:172.21.15.183:6809/2957306877] -- osd_op(unknown.0.0:27 17.16 17:68000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f9160065ad0 2024-03-06T07:28:28.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.724+0000 7f9175bcf640 1 -- 172.21.15.83:0/2816044254 <== osd.5 v2:172.21.15.183:6808/2957306877 3 ==== osd_op_reply(27 [pgnls start_epoch 116 out=49b] v110'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f917008b040 con 0x7f9160065ad0 2024-03-06T07:28:28.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.724+0000 7f9175bcf640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] -- osd_op(unknown.0.0:28 17.36 17:6c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f91600603c0 2024-03-06T07:28:28.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.724+0000 7f91753ce640 1 -- 172.21.15.83:0/2816044254 <== osd.0 v2:172.21.15.83:6808/1635427425 6 ==== osd_op_reply(28 [pgnls start_epoch 116 out=49b] v110'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f916801a040 con 0x7f91600603c0 2024-03-06T07:28:28.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.724+0000 7f91753ce640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.83:6800/2591342133,v1:172.21.15.83:6801/2591342133] -- osd_op(unknown.0.0:29 17.e 17:70000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f9170012070 2024-03-06T07:28:28.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.724+0000 7f91753ce640 1 -- 172.21.15.83:0/2816044254 <== osd.2 v2:172.21.15.83:6800/2591342133 2 ==== osd_op_reply(29 [pgnls start_epoch 116 out=49b] v110'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9168043040 con 0x7f9170012070 2024-03-06T07:28:28.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.724+0000 7f91753ce640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.183:6800/2649185225,v1:172.21.15.183:6801/2649185225] -- osd_op(unknown.0.0:30 17.2e 17:74000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f9168020950 2024-03-06T07:28:28.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.725+0000 7f9175bcf640 1 -- 172.21.15.83:0/2816044254 <== osd.6 v2:172.21.15.183:6800/2649185225 2 ==== osd_op_reply(30 [pgnls start_epoch 116 out=49b] v110'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f917008b040 con 0x7f9168020950 2024-03-06T07:28:28.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.725+0000 7f9175bcf640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.83:6814/1822925240,v1:172.21.15.83:6817/1822925240] -- osd_op(unknown.0.0:31 17.1e 17:78000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f9160061430 2024-03-06T07:28:28.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.725+0000 7f91753ce640 1 -- 172.21.15.83:0/2816044254 <== osd.1 v2:172.21.15.83:6814/1822925240 4 ==== osd_op_reply(31 [pgnls start_epoch 116 out=49b] v110'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9168043040 con 0x7f9160061430 2024-03-06T07:28:28.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.725+0000 7f91753ce640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] -- osd_op(unknown.0.0:32 17.3e 17:7c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f9160074900 2024-03-06T07:28:28.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.725+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 <== osd.4 v2:172.21.15.183:6816/3256618830 3 ==== osd_op_reply(32 [pgnls start_epoch 116 out=49b] v110'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f916004daa0 con 0x7f9160074900 2024-03-06T07:28:28.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.725+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] -- osd_op(unknown.0.0:33 17.1 17:80000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f91600603c0 2024-03-06T07:28:28.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.726+0000 7f91753ce640 1 -- 172.21.15.83:0/2816044254 <== osd.0 v2:172.21.15.83:6808/1635427425 7 ==== osd_op_reply(33 [pgnls start_epoch 116 out=49b] v110'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9168043040 con 0x7f91600603c0 2024-03-06T07:28:28.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.726+0000 7f91753ce640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] -- osd_op(unknown.0.0:34 17.21 17:84000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f91600603c0 2024-03-06T07:28:28.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.726+0000 7f91753ce640 1 -- 172.21.15.83:0/2816044254 <== osd.0 v2:172.21.15.83:6808/1635427425 8 ==== osd_op_reply(34 [pgnls start_epoch 116 out=49b] v110'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9168043040 con 0x7f91600603c0 2024-03-06T07:28:28.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.726+0000 7f91753ce640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] -- osd_op(unknown.0.0:35 17.11 17:88000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f916801d8d0 2024-03-06T07:28:28.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.726+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 <== osd.3 v2:172.21.15.83:6815/2656563756 7 ==== osd_op_reply(35 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f916004daa0 con 0x7f916801d8d0 2024-03-06T07:28:28.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.726+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.83:6814/1822925240,v1:172.21.15.83:6817/1822925240] -- osd_op(unknown.0.0:36 17.31 17:8c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f9160061430 2024-03-06T07:28:28.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.726+0000 7f91753ce640 1 -- 172.21.15.83:0/2816044254 <== osd.1 v2:172.21.15.83:6814/1822925240 5 ==== osd_op_reply(36 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9168043040 con 0x7f9160061430 2024-03-06T07:28:28.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.726+0000 7f91753ce640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] -- osd_op(unknown.0.0:37 17.9 17:90000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f916801d8d0 2024-03-06T07:28:28.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.727+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 <== osd.3 v2:172.21.15.83:6815/2656563756 8 ==== osd_op_reply(37 [pgnls start_epoch 116 out=49b] v110'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f916004daa0 con 0x7f916801d8d0 2024-03-06T07:28:28.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.727+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] -- osd_op(unknown.0.0:38 17.29 17:94000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f9160074900 2024-03-06T07:28:28.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.727+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 <== osd.4 v2:172.21.15.183:6816/3256618830 4 ==== osd_op_reply(38 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f916004daa0 con 0x7f9160074900 2024-03-06T07:28:28.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.727+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.183:6808/2957306877,v1:172.21.15.183:6809/2957306877] -- osd_op(unknown.0.0:39 17.19 17:98000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f9160065ad0 2024-03-06T07:28:28.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.727+0000 7f9175bcf640 1 -- 172.21.15.83:0/2816044254 <== osd.5 v2:172.21.15.183:6808/2957306877 4 ==== osd_op_reply(39 [pgnls start_epoch 116 out=49b] v110'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f917008b040 con 0x7f9160065ad0 2024-03-06T07:28:28.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.727+0000 7f9175bcf640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.183:6808/2957306877,v1:172.21.15.183:6809/2957306877] -- osd_op(unknown.0.0:40 17.39 17:9c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f9160065ad0 2024-03-06T07:28:28.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.728+0000 7f9175bcf640 1 -- 172.21.15.83:0/2816044254 <== osd.5 v2:172.21.15.183:6808/2957306877 5 ==== osd_op_reply(40 [pgnls start_epoch 116 out=49b] v110'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f917008b040 con 0x7f9160065ad0 2024-03-06T07:28:28.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.728+0000 7f9175bcf640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.183:6800/2649185225,v1:172.21.15.183:6801/2649185225] -- osd_op(unknown.0.0:41 17.5 17:a0000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f9168020950 2024-03-06T07:28:28.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.728+0000 7f9175bcf640 1 -- 172.21.15.83:0/2816044254 <== osd.6 v2:172.21.15.183:6800/2649185225 3 ==== osd_op_reply(41 [pgnls start_epoch 116 out=49b] v110'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f917008b040 con 0x7f9168020950 2024-03-06T07:28:28.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.728+0000 7f9175bcf640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.83:6814/1822925240,v1:172.21.15.83:6817/1822925240] -- osd_op(unknown.0.0:42 17.25 17:a4000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f9160061430 2024-03-06T07:28:28.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.728+0000 7f91753ce640 1 -- 172.21.15.83:0/2816044254 <== osd.1 v2:172.21.15.83:6814/1822925240 6 ==== osd_op_reply(42 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9168043040 con 0x7f9160061430 2024-03-06T07:28:28.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.728+0000 7f91753ce640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.183:6808/2957306877,v1:172.21.15.183:6809/2957306877] -- osd_op(unknown.0.0:43 17.15 17:a8000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f9160065ad0 2024-03-06T07:28:28.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.729+0000 7f9175bcf640 1 -- 172.21.15.83:0/2816044254 <== osd.5 v2:172.21.15.183:6808/2957306877 6 ==== osd_op_reply(43 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f917008b040 con 0x7f9160065ad0 2024-03-06T07:28:28.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.729+0000 7f9175bcf640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] -- osd_op(unknown.0.0:44 17.35 17:ac000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f9160074900 2024-03-06T07:28:28.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.729+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 <== osd.4 v2:172.21.15.183:6816/3256618830 5 ==== osd_op_reply(44 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f916004daa0 con 0x7f9160074900 2024-03-06T07:28:28.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.729+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] -- osd_op(unknown.0.0:45 17.d 17:b0000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x55c7ca887e40 2024-03-06T07:28:28.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.729+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 <== osd.7 v2:172.21.15.183:6820/3336566695 7 ==== osd_op_reply(45 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f916004daa0 con 0x55c7ca887e40 2024-03-06T07:28:28.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.729+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] -- osd_op(unknown.0.0:46 17.2d 17:b4000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x55c7ca887e40 2024-03-06T07:28:28.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.730+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 <== osd.7 v2:172.21.15.183:6820/3336566695 8 ==== osd_op_reply(46 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f916004daa0 con 0x55c7ca887e40 2024-03-06T07:28:28.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.730+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] -- osd_op(unknown.0.0:47 17.1d 17:b8000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x55c7ca887e40 2024-03-06T07:28:28.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.730+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 <== osd.7 v2:172.21.15.183:6820/3336566695 9 ==== osd_op_reply(47 [pgnls start_epoch 116 out=49b] v110'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f916004daa0 con 0x55c7ca887e40 2024-03-06T07:28:28.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.730+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] -- osd_op(unknown.0.0:48 17.3d 17:bc000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x55c7ca887e40 2024-03-06T07:28:28.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.730+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 <== osd.7 v2:172.21.15.183:6820/3336566695 10 ==== osd_op_reply(48 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f916004daa0 con 0x55c7ca887e40 2024-03-06T07:28:28.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.730+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] -- osd_op(unknown.0.0:49 17.3 17:c0000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f9160074900 2024-03-06T07:28:28.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.730+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 <== osd.4 v2:172.21.15.183:6816/3256618830 6 ==== osd_op_reply(49 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f916004daa0 con 0x7f9160074900 2024-03-06T07:28:28.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.730+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] -- osd_op(unknown.0.0:50 17.23 17:c4000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f9160074900 2024-03-06T07:28:28.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.731+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 <== osd.4 v2:172.21.15.183:6816/3256618830 7 ==== osd_op_reply(50 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f916004daa0 con 0x7f9160074900 2024-03-06T07:28:28.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.731+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] -- osd_op(unknown.0.0:51 17.13 17:c8000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f916801d8d0 2024-03-06T07:28:28.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.731+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 <== osd.3 v2:172.21.15.83:6815/2656563756 9 ==== osd_op_reply(51 [pgnls start_epoch 116 out=49b] v110'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f916004daa0 con 0x7f916801d8d0 2024-03-06T07:28:28.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.731+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] -- osd_op(unknown.0.0:52 17.33 17:cc000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x55c7ca887e40 2024-03-06T07:28:28.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.731+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 <== osd.7 v2:172.21.15.183:6820/3336566695 11 ==== osd_op_reply(52 [pgnls start_epoch 116 out=49b] v110'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f916004daa0 con 0x55c7ca887e40 2024-03-06T07:28:28.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.731+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] -- osd_op(unknown.0.0:53 17.b 17:d0000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f91600603c0 2024-03-06T07:28:28.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.732+0000 7f91753ce640 1 -- 172.21.15.83:0/2816044254 <== osd.0 v2:172.21.15.83:6808/1635427425 9 ==== osd_op_reply(53 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9168043040 con 0x7f91600603c0 2024-03-06T07:28:28.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.732+0000 7f91753ce640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] -- osd_op(unknown.0.0:54 17.2b 17:d4000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f9160074900 2024-03-06T07:28:28.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.732+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 <== osd.4 v2:172.21.15.183:6816/3256618830 8 ==== osd_op_reply(54 [pgnls start_epoch 116 out=49b] v110'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f916004daa0 con 0x7f9160074900 2024-03-06T07:28:28.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.732+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] -- osd_op(unknown.0.0:55 17.1b 17:d8000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f9160074900 2024-03-06T07:28:28.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.732+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 <== osd.4 v2:172.21.15.183:6816/3256618830 9 ==== osd_op_reply(55 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f916004daa0 con 0x7f9160074900 2024-03-06T07:28:28.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.732+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.83:6800/2591342133,v1:172.21.15.83:6801/2591342133] -- osd_op(unknown.0.0:56 17.3b 17:dc000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f9170012070 2024-03-06T07:28:28.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.733+0000 7f91753ce640 1 -- 172.21.15.83:0/2816044254 <== osd.2 v2:172.21.15.83:6800/2591342133 3 ==== osd_op_reply(56 [pgnls start_epoch 116 out=49b] v110'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9168043040 con 0x7f9170012070 2024-03-06T07:28:28.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.733+0000 7f91753ce640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] -- osd_op(unknown.0.0:57 17.7 17:e0000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f91600603c0 2024-03-06T07:28:28.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.733+0000 7f91753ce640 1 -- 172.21.15.83:0/2816044254 <== osd.0 v2:172.21.15.83:6808/1635427425 10 ==== osd_op_reply(57 [pgnls start_epoch 116 out=49b] v116'12 uv12 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9168043040 con 0x7f91600603c0 2024-03-06T07:28:28.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.733+0000 7f91753ce640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] -- osd_op(unknown.0.0:58 17.27 17:e4000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f9160074900 2024-03-06T07:28:28.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.733+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 <== osd.4 v2:172.21.15.183:6816/3256618830 10 ==== osd_op_reply(58 [pgnls start_epoch 116 out=49b] v110'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f916004daa0 con 0x7f9160074900 2024-03-06T07:28:28.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.733+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.183:6800/2649185225,v1:172.21.15.183:6801/2649185225] -- osd_op(unknown.0.0:59 17.17 17:e8000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f9168020950 2024-03-06T07:28:28.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.734+0000 7f9175bcf640 1 -- 172.21.15.83:0/2816044254 <== osd.6 v2:172.21.15.183:6800/2649185225 4 ==== osd_op_reply(59 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f917008b040 con 0x7f9168020950 2024-03-06T07:28:28.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.734+0000 7f9175bcf640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] -- osd_op(unknown.0.0:60 17.37 17:ec000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f91600603c0 2024-03-06T07:28:28.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.734+0000 7f91753ce640 1 -- 172.21.15.83:0/2816044254 <== osd.0 v2:172.21.15.83:6808/1635427425 11 ==== osd_op_reply(60 [pgnls start_epoch 116 out=49b] v110'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9168043040 con 0x7f91600603c0 2024-03-06T07:28:28.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.734+0000 7f91753ce640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.83:6814/1822925240,v1:172.21.15.83:6817/1822925240] -- osd_op(unknown.0.0:61 17.f 17:f0000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f9160061430 2024-03-06T07:28:28.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.734+0000 7f91753ce640 1 -- 172.21.15.83:0/2816044254 <== osd.1 v2:172.21.15.83:6814/1822925240 7 ==== osd_op_reply(61 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9168043040 con 0x7f9160061430 2024-03-06T07:28:28.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.734+0000 7f91753ce640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] -- osd_op(unknown.0.0:62 17.2f 17:f4000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f9160074900 2024-03-06T07:28:28.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.734+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 <== osd.4 v2:172.21.15.183:6816/3256618830 11 ==== osd_op_reply(62 [pgnls start_epoch 116 out=49b] v116'10 uv10 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f916004daa0 con 0x7f9160074900 2024-03-06T07:28:28.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.734+0000 7f916ffff640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.183:6800/2649185225,v1:172.21.15.183:6801/2649185225] -- osd_op(unknown.0.0:63 17.1f 17:f8000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f9168020950 2024-03-06T07:28:28.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.735+0000 7f9175bcf640 1 -- 172.21.15.83:0/2816044254 <== osd.6 v2:172.21.15.183:6800/2649185225 5 ==== osd_op_reply(63 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f917008b040 con 0x7f9168020950 2024-03-06T07:28:28.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.735+0000 7f9175bcf640 1 -- 172.21.15.83:0/2816044254 --> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] -- osd_op(unknown.0.0:64 17.3f 17:fc000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x7f9160072040 con 0x7f91600603c0 2024-03-06T07:28:28.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.735+0000 7f91753ce640 1 -- 172.21.15.83:0/2816044254 <== osd.0 v2:172.21.15.83:6808/1635427425 12 ==== osd_op_reply(64 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9168043040 con 0x7f91600603c0 2024-03-06T07:28:28.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.735+0000 7f9178036f40 1 -- 172.21.15.83:0/2816044254 >> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] conn(0x7f91600603c0 msgr2=0x7f9160060840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.735+0000 7f9178036f40 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] conn(0x7f91600603c0 0x7f9160060840 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.735+0000 7f9178036f40 1 -- 172.21.15.83:0/2816044254 >> [v2:172.21.15.83:6814/1822925240,v1:172.21.15.83:6817/1822925240] conn(0x7f9160061430 msgr2=0x7f9160061890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.735+0000 7f9178036f40 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.83:6814/1822925240,v1:172.21.15.83:6817/1822925240] conn(0x7f9160061430 0x7f9160061890 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.735+0000 7f9178036f40 1 -- 172.21.15.83:0/2816044254 >> [v2:172.21.15.83:6800/2591342133,v1:172.21.15.83:6801/2591342133] conn(0x7f9170012070 msgr2=0x7f9170058fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.735+0000 7f9178036f40 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.83:6800/2591342133,v1:172.21.15.83:6801/2591342133] conn(0x7f9170012070 0x7f9170058fe0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.735+0000 7f9178036f40 1 -- 172.21.15.83:0/2816044254 >> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] conn(0x7f916801d8d0 msgr2=0x7f91680103a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.735+0000 7f9178036f40 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] conn(0x7f916801d8d0 0x7f91680103a0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.735+0000 7f9175bcf640 1 -- 172.21.15.83:0/2816044254 reap_dead start 2024-03-06T07:28:28.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.735+0000 7f9178036f40 1 -- 172.21.15.83:0/2816044254 >> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] conn(0x7f9160074900 msgr2=0x7f9160071370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.735+0000 7f9178036f40 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] conn(0x7f9160074900 0x7f9160071370 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.735+0000 7f9175bcf640 1 -- 172.21.15.83:0/2816044254 reap_dead start 2024-03-06T07:28:28.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.735+0000 7f9178036f40 1 -- 172.21.15.83:0/2816044254 >> [v2:172.21.15.183:6808/2957306877,v1:172.21.15.183:6809/2957306877] conn(0x7f9160065ad0 msgr2=0x7f91600643e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.735+0000 7f9178036f40 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.183:6808/2957306877,v1:172.21.15.183:6809/2957306877] conn(0x7f9160065ad0 0x7f91600643e0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.735+0000 7f9178036f40 1 -- 172.21.15.83:0/2816044254 >> [v2:172.21.15.183:6800/2649185225,v1:172.21.15.183:6801/2649185225] conn(0x7f9168020950 msgr2=0x7f9168033440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.735+0000 7f9178036f40 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.183:6800/2649185225,v1:172.21.15.183:6801/2649185225] conn(0x7f9168020950 0x7f9168033440 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.735+0000 7f9178036f40 1 -- 172.21.15.83:0/2816044254 >> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] conn(0x55c7ca887e40 msgr2=0x55c7ca7e68a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.735+0000 7f9178036f40 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] conn(0x55c7ca887e40 0x55c7ca7e68a0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.735+0000 7f9178036f40 1 -- 172.21.15.83:0/2816044254 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9158069ee0 msgr2=0x7f915806c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.735+0000 7f9178036f40 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9158069ee0 0x7f915806c3a0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f9168000f60 tx=0x7f9168001770 comp rx=0 tx=0).stop 2024-03-06T07:28:28.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.735+0000 7f9178036f40 1 -- 172.21.15.83:0/2816044254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7ca7ec8a0 msgr2=0x55c7ca714310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.735+0000 7f9178036f40 1 --2- 172.21.15.83:0/2816044254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c7ca7ec8a0 0x55c7ca714310 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f9170005dc0 tx=0x7f91700033a0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.735+0000 7f9175bcf640 1 -- 172.21.15.83:0/2816044254 reap_dead start 2024-03-06T07:28:28.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.735+0000 7f9178036f40 1 -- 172.21.15.83:0/2816044254 shutdown_connections 2024-03-06T07:28:28.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.735+0000 7f9178036f40 1 -- 172.21.15.83:0/2816044254 >> 172.21.15.83:0/2816044254 conn(0x55c7ca741340 msgr2=0x55c7ca7eb240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:28.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.735+0000 7f9178036f40 1 -- 172.21.15.83:0/2816044254 shutdown_connections 2024-03-06T07:28:28.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.735+0000 7f9178036f40 1 -- 172.21.15.83:0/2816044254 wait complete. 2024-03-06T07:28:28.745 DEBUG:tasks.cephfs.filesystem:All objects for ino 1099511627777 size 8388608 are absent 2024-03-06T07:28:28.745 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:28.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.879+0000 7f21c7fd7640 1 Processor -- start 2024-03-06T07:28:28.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.880+0000 7f21c7fd7640 1 -- start start 2024-03-06T07:28:28.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.880+0000 7f21c7fd7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f21c015bb20 0x7f21c00d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.880+0000 7f21c7fd7640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f21c00cfff0 0x7f21c00d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.880+0000 7f21c7fd7640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21c0161980 con 0x7f21c00ce4b0 2024-03-06T07:28:28.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.880+0000 7f21c7fd7640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21c0161c60 con 0x7f21c015bb20 2024-03-06T07:28:28.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.880+0000 7f21c7fd7640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21c0161f40 con 0x7f21c00cfff0 2024-03-06T07:28:28.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.881+0000 7f21c5d73640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f21c015bb20 0x7f21c00d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.881+0000 7f21c5d73640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f21c015bb20 0x7f21c00d1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52302/0 (socket says 172.21.15.83:52302) 2024-03-06T07:28:28.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.881+0000 7f21c5d73640 1 -- 172.21.15.83:0/3915857767 learned_addr learned my addr 172.21.15.83:0/3915857767 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:28.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.881+0000 7f21c5572640 1 --2- 172.21.15.83:0/3915857767 >> v2:172.21.15.183:3300/0 conn(0x7f21c00cfff0 0x7f21c00d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.881+0000 7f21c5d73640 1 -- 172.21.15.83:0/3915857767 >> v2:172.21.15.183:3300/0 conn(0x7f21c00cfff0 msgr2=0x7f21c00d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:28.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.881+0000 7f21c5d73640 1 --2- 172.21.15.83:0/3915857767 >> v2:172.21.15.183:3300/0 conn(0x7f21c00cfff0 0x7f21c00d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.881+0000 7f21c5d73640 1 -- 172.21.15.83:0/3915857767 >> v1:172.21.15.183:6789/0 conn(0x7f21c00ce4b0 legacy=0x7f21c00ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:28.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.881+0000 7f21c5d73640 1 -- 172.21.15.83:0/3915857767 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c0162220 con 0x7f21c015bb20 2024-03-06T07:28:28.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.882+0000 7f21c5d73640 1 --2- 172.21.15.83:0/3915857767 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21c015bb20 0x7f21c00d1df0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f21c0071290 tx=0x7f21bc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f84f1b6941e8b9d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.882+0000 7f21c4d71640 1 -- 172.21.15.83:0/3915857767 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21bc035020 con 0x7f21c015bb20 2024-03-06T07:28:28.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.882+0000 7f21c4d71640 1 -- 172.21.15.83:0/3915857767 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21bc031870 con 0x7f21c015bb20 2024-03-06T07:28:28.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.882+0000 7f21c4d71640 1 -- 172.21.15.83:0/3915857767 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21bc031c40 con 0x7f21c015bb20 2024-03-06T07:28:28.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.882+0000 7f21c7fd7640 1 -- 172.21.15.83:0/3915857767 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21c015bb20 msgr2=0x7f21c00d1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.882+0000 7f21c7fd7640 1 --2- 172.21.15.83:0/3915857767 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21c015bb20 0x7f21c00d1df0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f21c0071290 tx=0x7f21bc02b910 comp rx=0 tx=0).stop 2024-03-06T07:28:28.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.882+0000 7f21c7fd7640 1 -- 172.21.15.83:0/3915857767 shutdown_connections 2024-03-06T07:28:28.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.882+0000 7f21c7fd7640 1 --2- 172.21.15.83:0/3915857767 >> v2:172.21.15.183:3300/0 conn(0x7f21c00cfff0 0x7f21c00d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.882+0000 7f21c7fd7640 1 --2- 172.21.15.83:0/3915857767 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21c015bb20 0x7f21c00d1df0 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.882+0000 7f21c7fd7640 1 -- 172.21.15.83:0/3915857767 >> 172.21.15.83:0/3915857767 conn(0x7f21c009a9f0 msgr2=0x7f21c009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:28.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.882+0000 7f21c7fd7640 1 -- 172.21.15.83:0/3915857767 shutdown_connections 2024-03-06T07:28:28.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.882+0000 7f21c7fd7640 1 -- 172.21.15.83:0/3915857767 wait complete. 2024-03-06T07:28:28.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.884+0000 7f21c7fd7640 1 Processor -- start 2024-03-06T07:28:28.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.884+0000 7f21c7fd7640 1 -- start start 2024-03-06T07:28:28.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.884+0000 7f21c7fd7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21c00ce4b0 0x7f21c0158c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.884+0000 7f21c7fd7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f21c00cfff0 0x7f21c00ca8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.884+0000 7f21c7fd7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21c015bb20 0x7f21c00c9f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.884+0000 7f21c7fd7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c00d24b0 con 0x7f21c015bb20 2024-03-06T07:28:28.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.884+0000 7f21c7fd7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c00d2330 con 0x7f21c00ce4b0 2024-03-06T07:28:28.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.884+0000 7f21c7fd7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21c00d1980 con 0x7f21c00cfff0 2024-03-06T07:28:28.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.884+0000 7f21c6574640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21c015bb20 0x7f21c00c9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.884+0000 7f21c6574640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21c015bb20 0x7f21c00c9f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52310/0 (socket says 172.21.15.83:52310) 2024-03-06T07:28:28.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.884+0000 7f21c6574640 1 -- 172.21.15.83:0/2752083751 learned_addr learned my addr 172.21.15.83:0/2752083751 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:28.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.885+0000 7f21c5572640 1 --2- 172.21.15.83:0/2752083751 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f21c00cfff0 0x7f21c00ca8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.885+0000 7f21c5d73640 1 --2- 172.21.15.83:0/2752083751 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21c00ce4b0 0x7f21c0158c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.885+0000 7f21c6574640 1 -- 172.21.15.83:0/2752083751 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f21c00cfff0 msgr2=0x7f21c00ca8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.885+0000 7f21c6574640 1 --2- 172.21.15.83:0/2752083751 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f21c00cfff0 0x7f21c00ca8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.885+0000 7f21c6574640 1 -- 172.21.15.83:0/2752083751 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21c00ce4b0 msgr2=0x7f21c0158c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.885+0000 7f21c6574640 1 --2- 172.21.15.83:0/2752083751 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21c00ce4b0 0x7f21c0158c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.885+0000 7f21c6574640 1 -- 172.21.15.83:0/2752083751 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c00ca490 con 0x7f21c015bb20 2024-03-06T07:28:28.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.885+0000 7f21c5d73640 1 --2- 172.21.15.83:0/2752083751 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21c00ce4b0 0x7f21c0158c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:28.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.885+0000 7f21c5572640 1 --2- 172.21.15.83:0/2752083751 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f21c00cfff0 0x7f21c00ca8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:28.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.885+0000 7f21c6574640 1 --2- 172.21.15.83:0/2752083751 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21c015bb20 0x7f21c00c9f50 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f21b800c780 tx=0x7f21b800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.885+0000 7f21b67fc640 1 -- 172.21.15.83:0/2752083751 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b8004b60 con 0x7f21c015bb20 2024-03-06T07:28:28.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.885+0000 7f21b67fc640 1 -- 172.21.15.83:0/2752083751 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21b8004d00 con 0x7f21c015bb20 2024-03-06T07:28:28.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.885+0000 7f21c7fd7640 1 -- 172.21.15.83:0/2752083751 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21c00b5370 con 0x7f21c015bb20 2024-03-06T07:28:28.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.885+0000 7f21b67fc640 1 -- 172.21.15.83:0/2752083751 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b800aa90 con 0x7f21c015bb20 2024-03-06T07:28:28.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.885+0000 7f21c7fd7640 1 -- 172.21.15.83:0/2752083751 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21c00b5930 con 0x7f21c015bb20 2024-03-06T07:28:28.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.886+0000 7f21c7fd7640 1 -- 172.21.15.83:0/2752083751 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21c004f330 con 0x7f21c015bb20 2024-03-06T07:28:28.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.886+0000 7f21b67fc640 1 -- 172.21.15.83:0/2752083751 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f21b800ac30 con 0x7f21c015bb20 2024-03-06T07:28:28.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.887+0000 7f21b67fc640 1 --2- 172.21.15.83:0/2752083751 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f21a406b4d0 0x7f21a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.887+0000 7f21b67fc640 1 -- 172.21.15.83:0/2752083751 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f21b808a7e0 con 0x7f21c015bb20 2024-03-06T07:28:28.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.887+0000 7f21c5d73640 1 --2- 172.21.15.83:0/2752083751 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f21a406b4d0 0x7f21a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.887+0000 7f21c5d73640 1 --2- 172.21.15.83:0/2752083751 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f21a406b4d0 0x7f21a406d990 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f21bc036000 tx=0x7f21bc03c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:28.891+0000 7f21b67fc640 1 -- 172.21.15.83:0/2752083751 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f21c00ca490 con 0x7f21c015bb20 2024-03-06T07:28:29.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.200+0000 7f21c7fd7640 1 -- 172.21.15.83:0/2752083751 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f21c00ca6c0 con 0x7f21c015bb20 2024-03-06T07:28:29.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.201+0000 7f21b67fc640 1 -- 172.21.15.83:0/2752083751 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f21b8059600 con 0x7f21c015bb20 2024-03-06T07:28:29.203 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:28:29.203 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7051,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/101642171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":101642171},{"type":"v1","addr":"172.21.15.183:6833","nonce":101642171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7057,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3146884900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":3146884900},{"type":"v1","addr":"172.21.15.183:6835","nonce":3146884900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8208,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1922041621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1922041621},{"type":"v1","addr":"172.21.15.83:6835","nonce":1922041621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.491022+0000","modified":"2024-03-06T07:27:37.006649+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8229":{"gid":8229,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/2003687153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2003687153},{"type":"v1","addr":"172.21.15.83:6837","nonce":2003687153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8229,"qdb_cluster":[8229]},"id":8}]} 2024-03-06T07:28:29.203 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 66 2024-03-06T07:28:29.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.202+0000 7f21c7fd7640 1 -- 172.21.15.83:0/2752083751 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f21a406b4d0 msgr2=0x7f21a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.202+0000 7f21c7fd7640 1 --2- 172.21.15.83:0/2752083751 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f21a406b4d0 0x7f21a406d990 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f21bc036000 tx=0x7f21bc03c040 comp rx=0 tx=0).stop 2024-03-06T07:28:29.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.202+0000 7f21c7fd7640 1 -- 172.21.15.83:0/2752083751 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21c015bb20 msgr2=0x7f21c00c9f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.202+0000 7f21c7fd7640 1 --2- 172.21.15.83:0/2752083751 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21c015bb20 0x7f21c00c9f50 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f21b800c780 tx=0x7f21b800cc40 comp rx=0 tx=0).stop 2024-03-06T07:28:29.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.203+0000 7f21c7fd7640 1 -- 172.21.15.83:0/2752083751 shutdown_connections 2024-03-06T07:28:29.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.203+0000 7f21c7fd7640 1 --2- 172.21.15.83:0/2752083751 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f21a406b4d0 0x7f21a406d990 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.203+0000 7f21c7fd7640 1 --2- 172.21.15.83:0/2752083751 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21c015bb20 0x7f21c00c9f50 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.203+0000 7f21c7fd7640 1 --2- 172.21.15.83:0/2752083751 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f21c00cfff0 0x7f21c00ca8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.203+0000 7f21c7fd7640 1 --2- 172.21.15.83:0/2752083751 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21c00ce4b0 0x7f21c0158c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.203+0000 7f21c7fd7640 1 -- 172.21.15.83:0/2752083751 >> 172.21.15.83:0/2752083751 conn(0x7f21c009a9f0 msgr2=0x7f21c015a050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:29.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.203+0000 7f21c7fd7640 1 -- 172.21.15.83:0/2752083751 shutdown_connections 2024-03-06T07:28:29.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.203+0000 7f21c7fd7640 1 -- 172.21.15.83:0/2752083751 wait complete. 2024-03-06T07:28:29.215 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:28:29.332 INFO:teuthology.orchestra.run.smithi083.stdout:{"mds_cache":{"strays_created":3}} 2024-03-06T07:28:29.333 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 3 } } 2024-03-06T07:28:29.333 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:29.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.432+0000 7fef46d0f640 1 Processor -- start 2024-03-06T07:28:29.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.432+0000 7fef46d0f640 1 -- start start 2024-03-06T07:28:29.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.433+0000 7fef46d0f640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fef400b4170 0x7fef400b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.433+0000 7fef46d0f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fef400d0a80 0x7fef400cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.433+0000 7fef46d0f640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef400d25d0 con 0x7fef400d4760 2024-03-06T07:28:29.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.433+0000 7fef46d0f640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef400d4bf0 con 0x7fef400d0a80 2024-03-06T07:28:29.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.433+0000 7fef46d0f640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef400ce9e0 con 0x7fef400b4170 2024-03-06T07:28:29.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.433+0000 7fef452ac640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fef400d0a80 0x7fef400cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.433+0000 7fef452ac640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fef400d0a80 0x7fef400cee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52324/0 (socket says 172.21.15.83:52324) 2024-03-06T07:28:29.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.433+0000 7fef452ac640 1 -- 172.21.15.83:0/2941341903 learned_addr learned my addr 172.21.15.83:0/2941341903 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:29.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.433+0000 7fef44aab640 1 --2- 172.21.15.83:0/2941341903 >> v2:172.21.15.183:3300/0 conn(0x7fef400b4170 0x7fef400b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.434+0000 7fef452ac640 1 -- 172.21.15.83:0/2941341903 >> v2:172.21.15.183:3300/0 conn(0x7fef400b4170 msgr2=0x7fef400b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:29.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.434+0000 7fef452ac640 1 --2- 172.21.15.83:0/2941341903 >> v2:172.21.15.183:3300/0 conn(0x7fef400b4170 0x7fef400b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.434+0000 7fef452ac640 1 -- 172.21.15.83:0/2941341903 >> v1:172.21.15.183:6789/0 conn(0x7fef400d4760 legacy=0x7fef400d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.434+0000 7fef452ac640 1 -- 172.21.15.83:0/2941341903 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef400d2870 con 0x7fef400d0a80 2024-03-06T07:28:29.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.434+0000 7fef452ac640 1 --2- 172.21.15.83:0/2941341903 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef400d0a80 0x7fef400cee70 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fef4006e030 tx=0x7fef3802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17a420f9d251d02b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.434+0000 7fef3f7fe640 1 -- 172.21.15.83:0/2941341903 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3802cb20 con 0x7fef400d0a80 2024-03-06T07:28:29.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.434+0000 7fef3f7fe640 1 -- 172.21.15.83:0/2941341903 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef3802ccc0 con 0x7fef400d0a80 2024-03-06T07:28:29.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.434+0000 7fef3f7fe640 1 -- 172.21.15.83:0/2941341903 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef38034610 con 0x7fef400d0a80 2024-03-06T07:28:29.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.435+0000 7fef46d0f640 1 -- 172.21.15.83:0/2941341903 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef400d0a80 msgr2=0x7fef400cee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.435+0000 7fef46d0f640 1 --2- 172.21.15.83:0/2941341903 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef400d0a80 0x7fef400cee70 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fef4006e030 tx=0x7fef3802bf80 comp rx=0 tx=0).stop 2024-03-06T07:28:29.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.435+0000 7fef46d0f640 1 -- 172.21.15.83:0/2941341903 shutdown_connections 2024-03-06T07:28:29.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.435+0000 7fef46d0f640 1 --2- 172.21.15.83:0/2941341903 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef400d0a80 0x7fef400cee70 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.435+0000 7fef46d0f640 1 --2- 172.21.15.83:0/2941341903 >> v2:172.21.15.183:3300/0 conn(0x7fef400b4170 0x7fef400b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.435+0000 7fef46d0f640 1 -- 172.21.15.83:0/2941341903 >> 172.21.15.83:0/2941341903 conn(0x7fef400d3c40 msgr2=0x7fef400d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:29.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.435+0000 7fef46d0f640 1 -- 172.21.15.83:0/2941341903 shutdown_connections 2024-03-06T07:28:29.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.435+0000 7fef46d0f640 1 -- 172.21.15.83:0/2941341903 wait complete. 2024-03-06T07:28:29.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.436+0000 7fef46d0f640 1 Processor -- start 2024-03-06T07:28:29.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.436+0000 7fef46d0f640 1 -- start start 2024-03-06T07:28:29.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.437+0000 7fef46d0f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fef400b4170 0x7fef400cad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.437+0000 7fef46d0f640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef400d0a80 0x7fef400b2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.437+0000 7fef46d0f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef400b2f40 0x7fef400c9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.437+0000 7fef46d0f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef400b4dd0 con 0x7fef400b2f40 2024-03-06T07:28:29.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.437+0000 7fef46d0f640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef400b4550 con 0x7fef400d0a80 2024-03-06T07:28:29.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.437+0000 7fef46d0f640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef400b4f50 con 0x7fef400b4170 2024-03-06T07:28:29.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.437+0000 7fef452ac640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef400b2f40 0x7fef400c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.437+0000 7fef452ac640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef400b2f40 0x7fef400c9150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52332/0 (socket says 172.21.15.83:52332) 2024-03-06T07:28:29.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.437+0000 7fef452ac640 1 -- 172.21.15.83:0/4274369977 learned_addr learned my addr 172.21.15.83:0/4274369977 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:29.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.437+0000 7fef44aab640 1 --2- 172.21.15.83:0/4274369977 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fef400b4170 0x7fef400cad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.437+0000 7fef3ffff640 1 --2- 172.21.15.83:0/4274369977 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef400d0a80 0x7fef400b2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.437+0000 7fef452ac640 1 -- 172.21.15.83:0/4274369977 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fef400b4170 msgr2=0x7fef400cad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.437+0000 7fef452ac640 1 --2- 172.21.15.83:0/4274369977 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fef400b4170 0x7fef400cad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.437+0000 7fef452ac640 1 -- 172.21.15.83:0/4274369977 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef400d0a80 msgr2=0x7fef400b2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.437+0000 7fef452ac640 1 --2- 172.21.15.83:0/4274369977 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef400d0a80 0x7fef400b2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.437+0000 7fef452ac640 1 -- 172.21.15.83:0/4274369977 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef400b3550 con 0x7fef400b2f40 2024-03-06T07:28:29.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.438+0000 7fef452ac640 1 --2- 172.21.15.83:0/4274369977 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef400b2f40 0x7fef400c9150 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fef38009f40 tx=0x7fef38004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.438+0000 7fef3d7fa640 1 -- 172.21.15.83:0/4274369977 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef38043020 con 0x7fef400b2f40 2024-03-06T07:28:29.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.438+0000 7fef3d7fa640 1 -- 172.21.15.83:0/4274369977 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef3803e450 con 0x7fef400b2f40 2024-03-06T07:28:29.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.438+0000 7fef46d0f640 1 -- 172.21.15.83:0/4274369977 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef400c8390 con 0x7fef400b2f40 2024-03-06T07:28:29.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.438+0000 7fef3d7fa640 1 -- 172.21.15.83:0/4274369977 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef38009510 con 0x7fef400b2f40 2024-03-06T07:28:29.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.438+0000 7fef46d0f640 1 -- 172.21.15.83:0/4274369977 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef400c8950 con 0x7fef400b2f40 2024-03-06T07:28:29.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.439+0000 7fef46d0f640 1 -- 172.21.15.83:0/4274369977 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef4004f330 con 0x7fef400b2f40 2024-03-06T07:28:29.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.439+0000 7fef3d7fa640 1 -- 172.21.15.83:0/4274369977 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fef38046020 con 0x7fef400b2f40 2024-03-06T07:28:29.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.440+0000 7fef3d7fa640 1 --2- 172.21.15.83:0/4274369977 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fef2c06b4d0 0x7fef2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.440+0000 7fef3d7fa640 1 -- 172.21.15.83:0/4274369977 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fef380ac900 con 0x7fef400b2f40 2024-03-06T07:28:29.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.440+0000 7fef44aab640 1 --2- 172.21.15.83:0/4274369977 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fef2c06b4d0 0x7fef2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.440+0000 7fef44aab640 1 --2- 172.21.15.83:0/4274369977 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fef2c06b4d0 0x7fef2c06d990 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fef400b7b40 tx=0x7fef30009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.444+0000 7fef3d7fa640 1 -- 172.21.15.83:0/4274369977 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fef3804d030 con 0x7fef400b2f40 2024-03-06T07:28:29.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.660+0000 7fef46d0f640 1 -- 172.21.15.83:0/4274369977 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef400cfa30 con 0x7fef400b2f40 2024-03-06T07:28:29.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.661+0000 7fef3d7fa640 1 -- 172.21.15.83:0/4274369977 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4984 (secure 0 0 0) 0x7fef3807b040 con 0x7fef400b2f40 2024-03-06T07:28:29.663 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:28:29.663 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7051,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/101642171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":101642171},{"type":"v1","addr":"172.21.15.183:6833","nonce":101642171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7057,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3146884900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":3146884900},{"type":"v1","addr":"172.21.15.183:6835","nonce":3146884900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8208,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1922041621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1922041621},{"type":"v1","addr":"172.21.15.83:6835","nonce":1922041621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.491022+0000","modified":"2024-03-06T07:27:37.006649+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8229":{"gid":8229,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/2003687153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2003687153},{"type":"v1","addr":"172.21.15.83:6837","nonce":2003687153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8229,"qdb_cluster":[8229]},"id":8}]} 2024-03-06T07:28:29.663 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 66 2024-03-06T07:28:29.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.663+0000 7fef46d0f640 1 -- 172.21.15.83:0/4274369977 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fef2c06b4d0 msgr2=0x7fef2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.663+0000 7fef46d0f640 1 --2- 172.21.15.83:0/4274369977 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fef2c06b4d0 0x7fef2c06d990 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fef400b7b40 tx=0x7fef30009190 comp rx=0 tx=0).stop 2024-03-06T07:28:29.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.663+0000 7fef46d0f640 1 -- 172.21.15.83:0/4274369977 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef400b2f40 msgr2=0x7fef400c9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.663+0000 7fef46d0f640 1 --2- 172.21.15.83:0/4274369977 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef400b2f40 0x7fef400c9150 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fef38009f40 tx=0x7fef38004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:29.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.663+0000 7fef46d0f640 1 -- 172.21.15.83:0/4274369977 shutdown_connections 2024-03-06T07:28:29.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.663+0000 7fef46d0f640 1 --2- 172.21.15.83:0/4274369977 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fef2c06b4d0 0x7fef2c06d990 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.663+0000 7fef46d0f640 1 --2- 172.21.15.83:0/4274369977 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef400b2f40 0x7fef400c9150 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.663+0000 7fef46d0f640 1 --2- 172.21.15.83:0/4274369977 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef400d0a80 0x7fef400b2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.663+0000 7fef46d0f640 1 --2- 172.21.15.83:0/4274369977 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fef400b4170 0x7fef400cad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.663+0000 7fef46d0f640 1 -- 172.21.15.83:0/4274369977 >> 172.21.15.83:0/4274369977 conn(0x7fef400d3c40 msgr2=0x7fef4006b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:29.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.663+0000 7fef46d0f640 1 -- 172.21.15.83:0/4274369977 shutdown_connections 2024-03-06T07:28:29.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:29.663+0000 7fef46d0f640 1 -- 172.21.15.83:0/4274369977 wait complete. 2024-03-06T07:28:29.676 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:28:29.791 INFO:teuthology.orchestra.run.smithi083.stdout:{"mds_cache":{"strays_enqueued":1}} 2024-03-06T07:28:29.792 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1 } } 2024-03-06T07:28:29.792 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:28:29.792 DEBUG:teuthology.orchestra.run.smithi083:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:28:29.833 DEBUG:teuthology.orchestra.run.smithi183:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:28:29.887 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:28:29.888 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:28:29.888 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:29.904 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:30.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.043+0000 7fef654df640 1 Processor -- start 2024-03-06T07:28:30.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.043+0000 7fef654df640 1 -- start start 2024-03-06T07:28:30.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.044+0000 7fef654df640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fef6015dfe0 0x7fef600cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.044+0000 7fef654df640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fef600cf830 0x7fef600d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.044+0000 7fef654df640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fef600d27f0 0x7fef600d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.044+0000 7fef654df640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef600d2c50 con 0x7fef6015dfe0 2024-03-06T07:28:30.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.044+0000 7fef654df640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef600b31e0 con 0x7fef600d27f0 2024-03-06T07:28:30.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.044+0000 7fef654df640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef600b3d60 con 0x7fef600cf830 2024-03-06T07:28:30.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.044+0000 7fef5effd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fef6015dfe0 0x7fef600cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.044+0000 7fef5effd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fef6015dfe0 0x7fef600cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52348/0 (socket says 172.21.15.83:52348) 2024-03-06T07:28:30.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.044+0000 7fef5effd640 1 -- 172.21.15.83:0/2617610982 learned_addr learned my addr 172.21.15.83:0/2617610982 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:30.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.044+0000 7fef5e7fc640 1 --2- 172.21.15.83:0/2617610982 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fef600cf830 0x7fef600d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.044+0000 7fef5f7fe640 1 --2- 172.21.15.83:0/2617610982 >> v2:172.21.15.183:3300/0 conn(0x7fef600d27f0 0x7fef600d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.044+0000 7fef5effd640 1 -- 172.21.15.83:0/2617610982 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fef600cf830 msgr2=0x7fef600d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:30.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.044+0000 7fef5effd640 1 --2- 172.21.15.83:0/2617610982 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fef600cf830 0x7fef600d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.044+0000 7fef5effd640 1 -- 172.21.15.83:0/2617610982 >> v2:172.21.15.183:3300/0 conn(0x7fef600d27f0 msgr2=0x7fef600d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:30.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.044+0000 7fef5effd640 1 --2- 172.21.15.83:0/2617610982 >> v2:172.21.15.183:3300/0 conn(0x7fef600d27f0 0x7fef600d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.044+0000 7fef5effd640 1 -- 172.21.15.83:0/2617610982 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef600b3430 con 0x7fef6015dfe0 2024-03-06T07:28:30.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.044+0000 7fef5f7fe640 1 --2- 172.21.15.83:0/2617610982 >> v2:172.21.15.183:3300/0 conn(0x7fef600d27f0 0x7fef600d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:30.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.045+0000 7fef5effd640 1 --2- 172.21.15.83:0/2617610982 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef6015dfe0 0x7fef600cf2f0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7fef60067ad0 tx=0x7fef5002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fef583c3556d966 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.045+0000 7fef5dffb640 1 -- 172.21.15.83:0/2617610982 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef50035020 con 0x7fef6015dfe0 2024-03-06T07:28:30.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.045+0000 7fef5dffb640 1 -- 172.21.15.83:0/2617610982 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef50031870 con 0x7fef6015dfe0 2024-03-06T07:28:30.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.045+0000 7fef5dffb640 1 -- 172.21.15.83:0/2617610982 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef50031c40 con 0x7fef6015dfe0 2024-03-06T07:28:30.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.045+0000 7fef654df640 1 -- 172.21.15.83:0/2617610982 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef6015dfe0 msgr2=0x7fef600cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.045+0000 7fef654df640 1 --2- 172.21.15.83:0/2617610982 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef6015dfe0 0x7fef600cf2f0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7fef60067ad0 tx=0x7fef5002b910 comp rx=0 tx=0).stop 2024-03-06T07:28:30.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.045+0000 7fef654df640 1 -- 172.21.15.83:0/2617610982 shutdown_connections 2024-03-06T07:28:30.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.045+0000 7fef654df640 1 --2- 172.21.15.83:0/2617610982 >> v2:172.21.15.183:3300/0 conn(0x7fef600d27f0 0x7fef600d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.045+0000 7fef654df640 1 --2- 172.21.15.83:0/2617610982 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fef600cf830 0x7fef600d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.045+0000 7fef654df640 1 --2- 172.21.15.83:0/2617610982 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef6015dfe0 0x7fef600cf2f0 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.045+0000 7fef654df640 1 -- 172.21.15.83:0/2617610982 >> 172.21.15.83:0/2617610982 conn(0x7fef600d3cd0 msgr2=0x7fef600d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:30.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.046+0000 7fef654df640 1 -- 172.21.15.83:0/2617610982 shutdown_connections 2024-03-06T07:28:30.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.046+0000 7fef654df640 1 -- 172.21.15.83:0/2617610982 wait complete. 2024-03-06T07:28:30.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.047+0000 7fef654df640 1 Processor -- start 2024-03-06T07:28:30.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.048+0000 7fef654df640 1 -- start start 2024-03-06T07:28:30.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.049+0000 7fef654df640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef600cf830 0x7fef600b6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.049+0000 7fef654df640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fef600d27f0 0x7fef60157a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.049+0000 7fef654df640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef6015dfe0 0x7fef600cd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.049+0000 7fef654df640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef600d1c70 con 0x7fef6015dfe0 2024-03-06T07:28:30.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.049+0000 7fef654df640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef600b39f0 con 0x7fef600cf830 2024-03-06T07:28:30.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.049+0000 7fef654df640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef600d1df0 con 0x7fef600d27f0 2024-03-06T07:28:30.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.049+0000 7fef5f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef6015dfe0 0x7fef600cd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.049+0000 7fef5f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef6015dfe0 0x7fef600cd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52356/0 (socket says 172.21.15.83:52356) 2024-03-06T07:28:30.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.049+0000 7fef5f7fe640 1 -- 172.21.15.83:0/2461295581 learned_addr learned my addr 172.21.15.83:0/2461295581 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:30.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.049+0000 7fef5e7fc640 1 --2- 172.21.15.83:0/2461295581 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fef600d27f0 0x7fef60157a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.049+0000 7fef5effd640 1 --2- 172.21.15.83:0/2461295581 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef600cf830 0x7fef600b6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.049+0000 7fef5f7fe640 1 -- 172.21.15.83:0/2461295581 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fef600d27f0 msgr2=0x7fef60157a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.049+0000 7fef5f7fe640 1 --2- 172.21.15.83:0/2461295581 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fef600d27f0 0x7fef60157a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.049+0000 7fef5f7fe640 1 -- 172.21.15.83:0/2461295581 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef600cf830 msgr2=0x7fef600b6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.049+0000 7fef5f7fe640 1 --2- 172.21.15.83:0/2461295581 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef600cf830 0x7fef600b6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.049+0000 7fef5f7fe640 1 -- 172.21.15.83:0/2461295581 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef600b9af0 con 0x7fef6015dfe0 2024-03-06T07:28:30.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.049+0000 7fef5effd640 1 --2- 172.21.15.83:0/2461295581 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef600cf830 0x7fef600b6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:30.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.049+0000 7fef5f7fe640 1 --2- 172.21.15.83:0/2461295581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef6015dfe0 0x7fef600cd4e0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7fef5800cf20 tx=0x7fef5800a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.049+0000 7fef477fe640 1 -- 172.21.15.83:0/2461295581 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef5801a070 con 0x7fef6015dfe0 2024-03-06T07:28:30.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.049+0000 7fef477fe640 1 -- 172.21.15.83:0/2461295581 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef5800d840 con 0x7fef6015dfe0 2024-03-06T07:28:30.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.049+0000 7fef477fe640 1 -- 172.21.15.83:0/2461295581 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef5800dc30 con 0x7fef6015dfe0 2024-03-06T07:28:30.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.049+0000 7fef654df640 1 -- 172.21.15.83:0/2461295581 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef600cc720 con 0x7fef6015dfe0 2024-03-06T07:28:30.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.049+0000 7fef654df640 1 -- 172.21.15.83:0/2461295581 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef600ccce0 con 0x7fef6015dfe0 2024-03-06T07:28:30.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.050+0000 7fef654df640 1 -- 172.21.15.83:0/2461295581 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef6004f330 con 0x7fef6015dfe0 2024-03-06T07:28:30.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.051+0000 7fef477fe640 1 -- 172.21.15.83:0/2461295581 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fef5801d070 con 0x7fef6015dfe0 2024-03-06T07:28:30.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.051+0000 7fef477fe640 1 --2- 172.21.15.83:0/2461295581 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fef4c06b4d0 0x7fef4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.051+0000 7fef477fe640 1 -- 172.21.15.83:0/2461295581 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fef5808abb0 con 0x7fef6015dfe0 2024-03-06T07:28:30.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.051+0000 7fef5effd640 1 --2- 172.21.15.83:0/2461295581 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fef4c06b4d0 0x7fef4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.052+0000 7fef5effd640 1 --2- 172.21.15.83:0/2461295581 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fef4c06b4d0 0x7fef4c06d990 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fef50036000 tx=0x7fef5003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.056+0000 7fef477fe640 1 -- 172.21.15.83:0/2461295581 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fef58059940 con 0x7fef6015dfe0 2024-03-06T07:28:30.104 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:30.106 DEBUG:teuthology.orchestra.run.smithi183:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:30.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.241+0000 7fef654df640 1 -- 172.21.15.83:0/2461295581 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fef600ceac0 con 0x7fef6015dfe0 2024-03-06T07:28:30.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.242+0000 7fef477fe640 1 -- 172.21.15.83:0/2461295581 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+16992 (secure 0 0 0) 0x7fef58059240 con 0x7fef6015dfe0 2024-03-06T07:28:30.244 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:28:30.244 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":116,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:27:41.021144+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:08.709967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:09.456518+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.26","mappings":[{"from":0,"to":2}]},{"pgid":"17.30","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.183:6834/1383294888":"2024-03-07T07:27:33.605177+0000","172.21.15.183:6832/1383294888":"2024-03-07T07:27:33.605177+0000","172.21.15.83:6836/855387870":"2024-03-07T07:27:33.587133+0000","172.21.15.83:6834/855387870":"2024-03-07T07:27:33.587133+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:30.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.244+0000 7fef654df640 1 -- 172.21.15.83:0/2461295581 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fef4c06b4d0 msgr2=0x7fef4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.244+0000 7fef654df640 1 --2- 172.21.15.83:0/2461295581 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fef4c06b4d0 0x7fef4c06d990 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fef50036000 tx=0x7fef5003c040 comp rx=0 tx=0).stop 2024-03-06T07:28:30.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.244+0000 7fef654df640 1 -- 172.21.15.83:0/2461295581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef6015dfe0 msgr2=0x7fef600cd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.244+0000 7fef654df640 1 --2- 172.21.15.83:0/2461295581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef6015dfe0 0x7fef600cd4e0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7fef5800cf20 tx=0x7fef5800a7d0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.244+0000 7fef654df640 1 -- 172.21.15.83:0/2461295581 shutdown_connections 2024-03-06T07:28:30.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.244+0000 7fef654df640 1 --2- 172.21.15.83:0/2461295581 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fef4c06b4d0 0x7fef4c06d990 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.244+0000 7fef654df640 1 --2- 172.21.15.83:0/2461295581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef6015dfe0 0x7fef600cd4e0 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.244+0000 7fef654df640 1 --2- 172.21.15.83:0/2461295581 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fef600d27f0 0x7fef60157a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.244+0000 7fef654df640 1 --2- 172.21.15.83:0/2461295581 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef600cf830 0x7fef600b6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.244+0000 7fef654df640 1 -- 172.21.15.83:0/2461295581 >> 172.21.15.83:0/2461295581 conn(0x7fef600d3cd0 msgr2=0x7fef6006ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:30.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.244+0000 7fef654df640 1 -- 172.21.15.83:0/2461295581 shutdown_connections 2024-03-06T07:28:30.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.244+0000 7fef654df640 1 -- 172.21.15.83:0/2461295581 wait complete. 2024-03-06T07:28:30.257 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:08.709967+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:09.456518+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:30.258 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:30.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.359+0000 7f997b6f4640 1 Processor -- start 2024-03-06T07:28:30.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.360+0000 7f997b6f4640 1 -- start start 2024-03-06T07:28:30.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.361+0000 7f997b6f4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f99741614d0 0x7f99740d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.361+0000 7f997b6f4640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f99740d1910 0x7f99740d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.361+0000 7f997b6f4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99740d2740 con 0x7f99740d22b0 2024-03-06T07:28:30.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.361+0000 7f997b6f4640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99741618b0 con 0x7f99740d1910 2024-03-06T07:28:30.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.361+0000 7f997b6f4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99740cf400 con 0x7f99741614d0 2024-03-06T07:28:30.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.361+0000 7f9979490640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f99741614d0 0x7f99740d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.361+0000 7f9978c8f640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f99740d1910 0x7f99740d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.361+0000 7f9979490640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f99741614d0 0x7f99740d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:42950/0 (socket says 172.21.15.83:42950) 2024-03-06T07:28:30.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.361+0000 7f9979490640 1 -- 172.21.15.83:0/741708697 learned_addr learned my addr 172.21.15.83:0/741708697 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:30.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.362+0000 7f9978c8f640 1 -- 172.21.15.83:0/741708697 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f99741614d0 msgr2=0x7f99740d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:30.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.362+0000 7f9978c8f640 1 --2- 172.21.15.83:0/741708697 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f99741614d0 0x7f99740d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.362+0000 7f9978c8f640 1 -- 172.21.15.83:0/741708697 >> v1:172.21.15.183:6789/0 conn(0x7f99740d22b0 legacy=0x7f99740d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.362+0000 7f9978c8f640 1 -- 172.21.15.83:0/741708697 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99740cf6a0 con 0x7f99740d1910 2024-03-06T07:28:30.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.362+0000 7f996bfff640 1 -- 172.21.15.83:0/741708697 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2833943474 0 0) 0x7f996c001490 con 0x7f99740d22b0 2024-03-06T07:28:30.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.362+0000 7f9979490640 1 --2- 172.21.15.83:0/741708697 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f99741614d0 0x7f99740d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:30.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.362+0000 7f9978c8f640 1 --2- 172.21.15.83:0/741708697 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f99740d1910 0x7f99740d1d70 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f99640097a0 tx=0x7f996402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ee2a5e4e86a91b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.363+0000 7f996bfff640 1 -- 172.21.15.83:0/741708697 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9964035020 con 0x7f99740d1910 2024-03-06T07:28:30.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.363+0000 7f996bfff640 1 -- 172.21.15.83:0/741708697 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9964031830 con 0x7f99740d1910 2024-03-06T07:28:30.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.363+0000 7f996bfff640 1 -- 172.21.15.83:0/741708697 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9964031c00 con 0x7f99740d1910 2024-03-06T07:28:30.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.363+0000 7f997b6f4640 1 -- 172.21.15.83:0/741708697 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f99740d1910 msgr2=0x7f99740d1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.363+0000 7f997b6f4640 1 --2- 172.21.15.83:0/741708697 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f99740d1910 0x7f99740d1d70 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f99640097a0 tx=0x7f996402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.363+0000 7f997b6f4640 1 -- 172.21.15.83:0/741708697 shutdown_connections 2024-03-06T07:28:30.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.363+0000 7f997b6f4640 1 --2- 172.21.15.83:0/741708697 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f99740d1910 0x7f99740d1d70 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.363+0000 7f997b6f4640 1 --2- 172.21.15.83:0/741708697 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f99741614d0 0x7f99740d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.363+0000 7f997b6f4640 1 -- 172.21.15.83:0/741708697 >> 172.21.15.83:0/741708697 conn(0x7f99740d3d40 msgr2=0x7f99740d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:30.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.363+0000 7f997b6f4640 1 -- 172.21.15.83:0/741708697 shutdown_connections 2024-03-06T07:28:30.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.363+0000 7f997b6f4640 1 -- 172.21.15.83:0/741708697 wait complete. 2024-03-06T07:28:30.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.365+0000 7f997b6f4640 1 Processor -- start 2024-03-06T07:28:30.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.365+0000 7f997b6f4640 1 -- start start 2024-03-06T07:28:30.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.366+0000 7f997b6f4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f99740d1910 0x7f99740d0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.366+0000 7f997b6f4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99740d22b0 0x7f99740cd300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.366+0000 7f997b6f4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f99741614d0 0x7f99740c8db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.366+0000 7f997b6f4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99740b0910 con 0x7f99740d22b0 2024-03-06T07:28:30.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.366+0000 7f997b6f4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99740cfc60 con 0x7f99740d1910 2024-03-06T07:28:30.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.366+0000 7f997b6f4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99740b1e40 con 0x7f99741614d0 2024-03-06T07:28:30.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.366+0000 7f9978c8f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99740d22b0 0x7f99740cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.366+0000 7f9978c8f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99740d22b0 0x7f99740cd300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52370/0 (socket says 172.21.15.83:52370) 2024-03-06T07:28:30.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.366+0000 7f9978c8f640 1 -- 172.21.15.83:0/844108758 learned_addr learned my addr 172.21.15.83:0/844108758 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:30.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.366+0000 7f9979490640 1 --2- 172.21.15.83:0/844108758 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f99740d1910 0x7f99740d0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.366+0000 7f9979c91640 1 --2- 172.21.15.83:0/844108758 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f99741614d0 0x7f99740c8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.366+0000 7f9978c8f640 1 -- 172.21.15.83:0/844108758 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f99741614d0 msgr2=0x7f99740c8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.366+0000 7f9978c8f640 1 --2- 172.21.15.83:0/844108758 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f99741614d0 0x7f99740c8db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.366+0000 7f9978c8f640 1 -- 172.21.15.83:0/844108758 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f99740d1910 msgr2=0x7f99740d0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.366+0000 7f9978c8f640 1 --2- 172.21.15.83:0/844108758 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f99740d1910 0x7f99740d0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.366+0000 7f9978c8f640 1 -- 172.21.15.83:0/844108758 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99740cdc70 con 0x7f99740d22b0 2024-03-06T07:28:30.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.367+0000 7f9978c8f640 1 --2- 172.21.15.83:0/844108758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99740d22b0 0x7f99740cd300 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f9964000c00 tx=0x7f99640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.367+0000 7f9969ffb640 1 -- 172.21.15.83:0/844108758 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9964035070 con 0x7f99740d22b0 2024-03-06T07:28:30.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.367+0000 7f9969ffb640 1 -- 172.21.15.83:0/844108758 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9964004230 con 0x7f99740d22b0 2024-03-06T07:28:30.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.367+0000 7f997b6f4640 1 -- 172.21.15.83:0/844108758 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f997414d2a0 con 0x7f99740d22b0 2024-03-06T07:28:30.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.367+0000 7f9969ffb640 1 -- 172.21.15.83:0/844108758 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996403a630 con 0x7f99740d22b0 2024-03-06T07:28:30.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.367+0000 7f997b6f4640 1 -- 172.21.15.83:0/844108758 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f997414d860 con 0x7f99740d22b0 2024-03-06T07:28:30.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.368+0000 7f997b6f4640 1 -- 172.21.15.83:0/844108758 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9974069e80 con 0x7f99740d22b0 2024-03-06T07:28:30.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.369+0000 7f9969ffb640 1 -- 172.21.15.83:0/844108758 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9964038020 con 0x7f99740d22b0 2024-03-06T07:28:30.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.369+0000 7f9969ffb640 1 --2- 172.21.15.83:0/844108758 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f995806b4d0 0x7f995806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.369+0000 7f9979490640 1 --2- 172.21.15.83:0/844108758 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f995806b4d0 0x7f995806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.369+0000 7f9969ffb640 1 -- 172.21.15.83:0/844108758 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f99640abe20 con 0x7f99740d22b0 2024-03-06T07:28:30.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.370+0000 7f9979490640 1 --2- 172.21.15.83:0/844108758 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f995806b4d0 0x7f995806d990 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f9970006ca0 tx=0x7f9970009240 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.377+0000 7f9969ffb640 1 -- 172.21.15.83:0/844108758 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9974069e80 con 0x7f99740d22b0 2024-03-06T07:28:30.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.554+0000 7f997b6f4640 1 -- 172.21.15.83:0/844108758 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f99740d32b0 con 0x7f99740d22b0 2024-03-06T07:28:30.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.554+0000 7f9969ffb640 1 -- 172.21.15.83:0/844108758 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v116) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9964040070 con 0x7f99740d22b0 2024-03-06T07:28:30.556 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:28:30.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.556+0000 7f997b6f4640 1 -- 172.21.15.83:0/844108758 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f995806b4d0 msgr2=0x7f995806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.556+0000 7f997b6f4640 1 --2- 172.21.15.83:0/844108758 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f995806b4d0 0x7f995806d990 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f9970006ca0 tx=0x7f9970009240 comp rx=0 tx=0).stop 2024-03-06T07:28:30.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.556+0000 7f997b6f4640 1 -- 172.21.15.83:0/844108758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99740d22b0 msgr2=0x7f99740cd300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.556+0000 7f997b6f4640 1 --2- 172.21.15.83:0/844108758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99740d22b0 0x7f99740cd300 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f9964000c00 tx=0x7f99640046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.556+0000 7f997b6f4640 1 -- 172.21.15.83:0/844108758 shutdown_connections 2024-03-06T07:28:30.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.556+0000 7f997b6f4640 1 --2- 172.21.15.83:0/844108758 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f995806b4d0 0x7f995806d990 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.556+0000 7f997b6f4640 1 --2- 172.21.15.83:0/844108758 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f99741614d0 0x7f99740c8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.556+0000 7f997b6f4640 1 --2- 172.21.15.83:0/844108758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99740d22b0 0x7f99740cd300 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.556+0000 7f997b6f4640 1 --2- 172.21.15.83:0/844108758 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f99740d1910 0x7f99740d0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.556+0000 7f997b6f4640 1 -- 172.21.15.83:0/844108758 >> 172.21.15.83:0/844108758 conn(0x7f99740d3d40 msgr2=0x7f997406bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:30.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.556+0000 7f997b6f4640 1 -- 172.21.15.83:0/844108758 shutdown_connections 2024-03-06T07:28:30.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.556+0000 7f997b6f4640 1 -- 172.21.15.83:0/844108758 wait complete. 2024-03-06T07:28:30.569 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:30.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.672+0000 7fd7414a4640 1 Processor -- start 2024-03-06T07:28:30.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.672+0000 7fd7414a4640 1 -- start start 2024-03-06T07:28:30.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.673+0000 7fd7414a4640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fd73c1614d0 0x7fd73c0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.673+0000 7fd7414a4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd73c0d1910 0x7fd73c0d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.673+0000 7fd7414a4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd73c0d22b0 0x7fd73c0d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.673+0000 7fd7414a4640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd73c0d2740 con 0x7fd73c0d22b0 2024-03-06T07:28:30.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.673+0000 7fd7414a4640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd73c1618b0 con 0x7fd73c1614d0 2024-03-06T07:28:30.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.673+0000 7fd7414a4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd73c0cf400 con 0x7fd73c0d1910 2024-03-06T07:28:30.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.673+0000 7fd73b7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd73c0d22b0 0x7fd73c0d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.673+0000 7fd73b7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd73c0d22b0 0x7fd73c0d06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52386/0 (socket says 172.21.15.83:52386) 2024-03-06T07:28:30.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.673+0000 7fd73b7fe640 1 -- 172.21.15.83:0/676709890 learned_addr learned my addr 172.21.15.83:0/676709890 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:30.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.673+0000 7fd73affd640 1 --2- 172.21.15.83:0/676709890 >> v2:172.21.15.183:3300/0 conn(0x7fd73c1614d0 0x7fd73c0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.673+0000 7fd7327fc640 1 --2- 172.21.15.83:0/676709890 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd73c0d1910 0x7fd73c0d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.673+0000 7fd73b7fe640 1 -- 172.21.15.83:0/676709890 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd73c0d1910 msgr2=0x7fd73c0d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:30.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.673+0000 7fd73b7fe640 1 --2- 172.21.15.83:0/676709890 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd73c0d1910 0x7fd73c0d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.673+0000 7fd73b7fe640 1 -- 172.21.15.83:0/676709890 >> v2:172.21.15.183:3300/0 conn(0x7fd73c1614d0 msgr2=0x7fd73c0d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:30.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.673+0000 7fd73b7fe640 1 --2- 172.21.15.83:0/676709890 >> v2:172.21.15.183:3300/0 conn(0x7fd73c1614d0 0x7fd73c0d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.673+0000 7fd73b7fe640 1 -- 172.21.15.83:0/676709890 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd73c0cf6a0 con 0x7fd73c0d22b0 2024-03-06T07:28:30.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.674+0000 7fd73b7fe640 1 --2- 172.21.15.83:0/676709890 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd73c0d22b0 0x7fd73c0d06a0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fd73c08c470 tx=0x7fd73402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f457d1299174239 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.674+0000 7fd73a7fc640 1 -- 172.21.15.83:0/676709890 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd73402ca70 con 0x7fd73c0d22b0 2024-03-06T07:28:30.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.674+0000 7fd73a7fc640 1 -- 172.21.15.83:0/676709890 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd73402cc10 con 0x7fd73c0d22b0 2024-03-06T07:28:30.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.674+0000 7fd73a7fc640 1 -- 172.21.15.83:0/676709890 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7340343f0 con 0x7fd73c0d22b0 2024-03-06T07:28:30.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.675+0000 7fd7414a4640 1 -- 172.21.15.83:0/676709890 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd73c0d22b0 msgr2=0x7fd73c0d06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.675+0000 7fd7414a4640 1 --2- 172.21.15.83:0/676709890 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd73c0d22b0 0x7fd73c0d06a0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fd73c08c470 tx=0x7fd73402bed0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.675+0000 7fd7414a4640 1 -- 172.21.15.83:0/676709890 shutdown_connections 2024-03-06T07:28:30.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.675+0000 7fd7414a4640 1 --2- 172.21.15.83:0/676709890 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd73c0d22b0 0x7fd73c0d06a0 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.675+0000 7fd7414a4640 1 --2- 172.21.15.83:0/676709890 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd73c0d1910 0x7fd73c0d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.675+0000 7fd7414a4640 1 --2- 172.21.15.83:0/676709890 >> v2:172.21.15.183:3300/0 conn(0x7fd73c1614d0 0x7fd73c0d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.675+0000 7fd7414a4640 1 -- 172.21.15.83:0/676709890 >> 172.21.15.83:0/676709890 conn(0x7fd73c0d3d40 msgr2=0x7fd73c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:30.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.675+0000 7fd7414a4640 1 -- 172.21.15.83:0/676709890 shutdown_connections 2024-03-06T07:28:30.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.675+0000 7fd7414a4640 1 -- 172.21.15.83:0/676709890 wait complete. 2024-03-06T07:28:30.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.676+0000 7fd7414a4640 1 Processor -- start 2024-03-06T07:28:30.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.676+0000 7fd7414a4640 1 -- start start 2024-03-06T07:28:30.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.678+0000 7fd7414a4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd73c0d1910 0x7fd73c15c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.678+0000 7fd7414a4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd73c0d22b0 0x7fd73c0d2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.678+0000 7fd7414a4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd73c1614d0 0x7fd73c0d2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.678+0000 7fd7414a4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd73c0b0910 con 0x7fd73c0d1910 2024-03-06T07:28:30.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.678+0000 7fd7414a4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd73c0cfc60 con 0x7fd73c1614d0 2024-03-06T07:28:30.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.678+0000 7fd7414a4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd73c0d0be0 con 0x7fd73c0d22b0 2024-03-06T07:28:30.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.678+0000 7fd73affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd73c0d1910 0x7fd73c15c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.678+0000 7fd73affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd73c0d1910 0x7fd73c15c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52398/0 (socket says 172.21.15.83:52398) 2024-03-06T07:28:30.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.678+0000 7fd73affd640 1 -- 172.21.15.83:0/2050427803 learned_addr learned my addr 172.21.15.83:0/2050427803 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:30.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.678+0000 7fd7327fc640 1 --2- 172.21.15.83:0/2050427803 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd73c0d22b0 0x7fd73c0d2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.678+0000 7fd73b7fe640 1 --2- 172.21.15.83:0/2050427803 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd73c1614d0 0x7fd73c0d2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.678+0000 7fd73affd640 1 -- 172.21.15.83:0/2050427803 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd73c0d22b0 msgr2=0x7fd73c0d2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.678+0000 7fd73affd640 1 --2- 172.21.15.83:0/2050427803 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd73c0d22b0 0x7fd73c0d2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.678+0000 7fd73affd640 1 -- 172.21.15.83:0/2050427803 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd73c1614d0 msgr2=0x7fd73c0d2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.678+0000 7fd73affd640 1 --2- 172.21.15.83:0/2050427803 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd73c1614d0 0x7fd73c0d2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.678+0000 7fd73affd640 1 -- 172.21.15.83:0/2050427803 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd73c0ccae0 con 0x7fd73c0d1910 2024-03-06T07:28:30.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.678+0000 7fd73affd640 1 --2- 172.21.15.83:0/2050427803 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd73c0d1910 0x7fd73c15c1d0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fd72400e870 tx=0x7fd72400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.678+0000 7fd733fff640 1 -- 172.21.15.83:0/2050427803 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd724017070 con 0x7fd73c0d1910 2024-03-06T07:28:30.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.678+0000 7fd733fff640 1 -- 172.21.15.83:0/2050427803 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd72400c940 con 0x7fd73c0d1910 2024-03-06T07:28:30.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.678+0000 7fd7414a4640 1 -- 172.21.15.83:0/2050427803 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd73c0ccdc0 con 0x7fd73c0d1910 2024-03-06T07:28:30.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.678+0000 7fd733fff640 1 -- 172.21.15.83:0/2050427803 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd72400cd30 con 0x7fd73c0d1910 2024-03-06T07:28:30.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.679+0000 7fd7414a4640 1 -- 172.21.15.83:0/2050427803 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd73c0cd380 con 0x7fd73c0d1910 2024-03-06T07:28:30.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.679+0000 7fd7414a4640 1 -- 172.21.15.83:0/2050427803 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd73c0c6fb0 con 0x7fd73c0d1910 2024-03-06T07:28:30.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.680+0000 7fd733fff640 1 -- 172.21.15.83:0/2050427803 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd72401a070 con 0x7fd73c0d1910 2024-03-06T07:28:30.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.681+0000 7fd733fff640 1 --2- 172.21.15.83:0/2050427803 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd72006b4d0 0x7fd72006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.681+0000 7fd733fff640 1 -- 172.21.15.83:0/2050427803 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fd72408a120 con 0x7fd73c0d1910 2024-03-06T07:28:30.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.681+0000 7fd7327fc640 1 --2- 172.21.15.83:0/2050427803 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd72006b4d0 0x7fd72006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.681+0000 7fd7327fc640 1 --2- 172.21.15.83:0/2050427803 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd72006b4d0 0x7fd72006d990 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fd73c15f460 tx=0x7fd72c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.685+0000 7fd733fff640 1 -- 172.21.15.83:0/2050427803 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd724058f40 con 0x7fd73c0d1910 2024-03-06T07:28:30.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.889+0000 7fd7414a4640 1 -- 172.21.15.83:0/2050427803 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd73c068ff0 con 0x7fd73c0d1910 2024-03-06T07:28:30.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.890+0000 7fd733fff640 1 -- 172.21.15.83:0/2050427803 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v116) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd724058840 con 0x7fd73c0d1910 2024-03-06T07:28:30.892 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-03-06T07:28:30.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.892+0000 7fd7414a4640 1 -- 172.21.15.83:0/2050427803 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd72006b4d0 msgr2=0x7fd72006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.892+0000 7fd7414a4640 1 --2- 172.21.15.83:0/2050427803 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd72006b4d0 0x7fd72006d990 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fd73c15f460 tx=0x7fd72c008040 comp rx=0 tx=0).stop 2024-03-06T07:28:30.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.892+0000 7fd7414a4640 1 -- 172.21.15.83:0/2050427803 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd73c0d1910 msgr2=0x7fd73c15c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.892+0000 7fd7414a4640 1 --2- 172.21.15.83:0/2050427803 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd73c0d1910 0x7fd73c15c1d0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fd72400e870 tx=0x7fd72400ed30 comp rx=0 tx=0).stop 2024-03-06T07:28:30.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.892+0000 7fd7414a4640 1 -- 172.21.15.83:0/2050427803 shutdown_connections 2024-03-06T07:28:30.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.892+0000 7fd7414a4640 1 --2- 172.21.15.83:0/2050427803 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd72006b4d0 0x7fd72006d990 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.892+0000 7fd7414a4640 1 --2- 172.21.15.83:0/2050427803 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd73c1614d0 0x7fd73c0d2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.892+0000 7fd7414a4640 1 --2- 172.21.15.83:0/2050427803 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd73c0d22b0 0x7fd73c0d2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.892+0000 7fd7414a4640 1 --2- 172.21.15.83:0/2050427803 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd73c0d1910 0x7fd73c15c1d0 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.892+0000 7fd7414a4640 1 -- 172.21.15.83:0/2050427803 >> 172.21.15.83:0/2050427803 conn(0x7fd73c0d3d40 msgr2=0x7fd73c160650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:30.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.892+0000 7fd7414a4640 1 -- 172.21.15.83:0/2050427803 shutdown_connections 2024-03-06T07:28:30.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:30.892+0000 7fd7414a4640 1 -- 172.21.15.83:0/2050427803 wait complete. 2024-03-06T07:28:30.905 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:31.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.007+0000 7ffb27fe9640 1 Processor -- start 2024-03-06T07:28:31.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.007+0000 7ffb27fe9640 1 -- start start 2024-03-06T07:28:31.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.008+0000 7ffb27fe9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffb200d10c0 0x7ffb200d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.008+0000 7ffb27fe9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ffb200d14a0 0x7ffb2015e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.008+0000 7ffb27fe9640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ffb2015eb00 0x7ffb200cae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.008+0000 7ffb27fe9640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb2015c050 con 0x7ffb200d14a0 2024-03-06T07:28:31.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.008+0000 7ffb27fe9640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb200d0060 con 0x7ffb2015eb00 2024-03-06T07:28:31.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.008+0000 7ffb27fe9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb200cfa50 con 0x7ffb200d10c0 2024-03-06T07:28:31.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.008+0000 7ffb25584640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ffb200d14a0 0x7ffb2015e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.009+0000 7ffb25584640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ffb200d14a0 0x7ffb2015e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52410/0 (socket says 172.21.15.83:52410) 2024-03-06T07:28:31.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.009+0000 7ffb25584640 1 -- 172.21.15.83:0/2316285228 learned_addr learned my addr 172.21.15.83:0/2316285228 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:31.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.009+0000 7ffb25d85640 1 --2- 172.21.15.83:0/2316285228 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffb200d10c0 0x7ffb200d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.009+0000 7ffb26586640 1 --2- 172.21.15.83:0/2316285228 >> v2:172.21.15.183:3300/0 conn(0x7ffb2015eb00 0x7ffb200cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.009+0000 7ffb25584640 1 -- 172.21.15.83:0/2316285228 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffb200d10c0 msgr2=0x7ffb200d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:31.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.009+0000 7ffb25584640 1 --2- 172.21.15.83:0/2316285228 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffb200d10c0 0x7ffb200d04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.009+0000 7ffb25584640 1 -- 172.21.15.83:0/2316285228 >> v2:172.21.15.183:3300/0 conn(0x7ffb2015eb00 msgr2=0x7ffb200cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:31.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.009+0000 7ffb25584640 1 --2- 172.21.15.83:0/2316285228 >> v2:172.21.15.183:3300/0 conn(0x7ffb2015eb00 0x7ffb200cae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.009+0000 7ffb25584640 1 -- 172.21.15.83:0/2316285228 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb200d0a30 con 0x7ffb200d14a0 2024-03-06T07:28:31.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.009+0000 7ffb25584640 1 --2- 172.21.15.83:0/2316285228 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb200d14a0 0x7ffb2015e5c0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c009870 tx=0x7ffb1c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd8fa4044c778d07 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.010+0000 7ffb24d83640 1 -- 172.21.15.83:0/2316285228 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb1c035020 con 0x7ffb200d14a0 2024-03-06T07:28:31.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.010+0000 7ffb24d83640 1 -- 172.21.15.83:0/2316285228 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb1c031640 con 0x7ffb200d14a0 2024-03-06T07:28:31.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.010+0000 7ffb24d83640 1 -- 172.21.15.83:0/2316285228 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb1c02d040 con 0x7ffb200d14a0 2024-03-06T07:28:31.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.010+0000 7ffb27fe9640 1 -- 172.21.15.83:0/2316285228 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb200d14a0 msgr2=0x7ffb2015e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.010+0000 7ffb27fe9640 1 --2- 172.21.15.83:0/2316285228 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb200d14a0 0x7ffb2015e5c0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c009870 tx=0x7ffb1c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.010+0000 7ffb27fe9640 1 -- 172.21.15.83:0/2316285228 shutdown_connections 2024-03-06T07:28:31.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.010+0000 7ffb27fe9640 1 --2- 172.21.15.83:0/2316285228 >> v2:172.21.15.183:3300/0 conn(0x7ffb2015eb00 0x7ffb200cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.010+0000 7ffb27fe9640 1 --2- 172.21.15.83:0/2316285228 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb200d14a0 0x7ffb2015e5c0 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.010+0000 7ffb27fe9640 1 --2- 172.21.15.83:0/2316285228 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffb200d10c0 0x7ffb200d04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.010+0000 7ffb27fe9640 1 -- 172.21.15.83:0/2316285228 >> 172.21.15.83:0/2316285228 conn(0x7ffb200b3800 msgr2=0x7ffb200b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:31.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.011+0000 7ffb27fe9640 1 -- 172.21.15.83:0/2316285228 shutdown_connections 2024-03-06T07:28:31.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.011+0000 7ffb27fe9640 1 -- 172.21.15.83:0/2316285228 wait complete. 2024-03-06T07:28:31.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.012+0000 7ffb27fe9640 1 Processor -- start 2024-03-06T07:28:31.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.012+0000 7ffb27fe9640 1 -- start start 2024-03-06T07:28:31.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.012+0000 7ffb27fe9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffb200d10c0 0x7ffb20152600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.012+0000 7ffb27fe9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffb200d14a0 0x7ffb200b0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.012+0000 7ffb27fe9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb2015eb00 0x7ffb200b0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.012+0000 7ffb27fe9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb2015f440 con 0x7ffb2015eb00 2024-03-06T07:28:31.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.012+0000 7ffb27fe9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb2015f160 con 0x7ffb200d10c0 2024-03-06T07:28:31.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.012+0000 7ffb27fe9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb200cfcf0 con 0x7ffb200d14a0 2024-03-06T07:28:31.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.013+0000 7ffb26586640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb2015eb00 0x7ffb200b0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.013+0000 7ffb26586640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb2015eb00 0x7ffb200b0970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52426/0 (socket says 172.21.15.83:52426) 2024-03-06T07:28:31.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.013+0000 7ffb26586640 1 -- 172.21.15.83:0/1523207456 learned_addr learned my addr 172.21.15.83:0/1523207456 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:31.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.013+0000 7ffb25d85640 1 --2- 172.21.15.83:0/1523207456 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffb200d10c0 0x7ffb20152600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.013+0000 7ffb25584640 1 --2- 172.21.15.83:0/1523207456 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffb200d14a0 0x7ffb200b0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.013+0000 7ffb26586640 1 -- 172.21.15.83:0/1523207456 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffb200d14a0 msgr2=0x7ffb200b0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.013+0000 7ffb26586640 1 --2- 172.21.15.83:0/1523207456 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffb200d14a0 0x7ffb200b0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.013+0000 7ffb26586640 1 -- 172.21.15.83:0/1523207456 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffb200d10c0 msgr2=0x7ffb20152600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.013+0000 7ffb26586640 1 --2- 172.21.15.83:0/1523207456 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffb200d10c0 0x7ffb20152600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.013+0000 7ffb26586640 1 -- 172.21.15.83:0/1523207456 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb200b5850 con 0x7ffb2015eb00 2024-03-06T07:28:31.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.013+0000 7ffb25d85640 1 --2- 172.21.15.83:0/1523207456 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffb200d10c0 0x7ffb20152600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:31.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.013+0000 7ffb26586640 1 --2- 172.21.15.83:0/1523207456 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb2015eb00 0x7ffb200b0970 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7ffb18007e70 tx=0x7ffb1800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.013+0000 7ffb167fc640 1 -- 172.21.15.83:0/1523207456 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb18014620 con 0x7ffb2015eb00 2024-03-06T07:28:31.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.013+0000 7ffb167fc640 1 -- 172.21.15.83:0/1523207456 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb18019070 con 0x7ffb2015eb00 2024-03-06T07:28:31.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.013+0000 7ffb167fc640 1 -- 172.21.15.83:0/1523207456 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb18014a20 con 0x7ffb2015eb00 2024-03-06T07:28:31.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.013+0000 7ffb27fe9640 1 -- 172.21.15.83:0/1523207456 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb200b5b30 con 0x7ffb2015eb00 2024-03-06T07:28:31.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.013+0000 7ffb27fe9640 1 -- 172.21.15.83:0/1523207456 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb200b60f0 con 0x7ffb2015eb00 2024-03-06T07:28:31.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.014+0000 7ffb27fe9640 1 -- 172.21.15.83:0/1523207456 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb2009fe60 con 0x7ffb2015eb00 2024-03-06T07:28:31.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.015+0000 7ffb167fc640 1 -- 172.21.15.83:0/1523207456 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffb18014bc0 con 0x7ffb2015eb00 2024-03-06T07:28:31.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.015+0000 7ffb167fc640 1 --2- 172.21.15.83:0/1523207456 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffb0406b4d0 0x7ffb0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.015+0000 7ffb167fc640 1 -- 172.21.15.83:0/1523207456 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5939+0+0 (secure 0 0 0) 0x7ffb1808ab60 con 0x7ffb2015eb00 2024-03-06T07:28:31.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.015+0000 7ffb25d85640 1 --2- 172.21.15.83:0/1523207456 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffb0406b4d0 0x7ffb0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.015+0000 7ffb25d85640 1 --2- 172.21.15.83:0/1523207456 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffb0406b4d0 0x7ffb0406d990 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7ffb1000ba50 tx=0x7ffb10008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.019+0000 7ffb167fc640 1 -- 172.21.15.83:0/1523207456 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffb18059980 con 0x7ffb2015eb00 2024-03-06T07:28:31.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.200+0000 7ffb27fe9640 1 -- 172.21.15.83:0/1523207456 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ffb200d0c30 con 0x7ffb2015eb00 2024-03-06T07:28:31.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.201+0000 7ffb167fc640 1 -- 172.21.15.83:0/1523207456 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v116) v1 ==== 100+0+11 (secure 0 0 0) 0x7ffb18059280 con 0x7ffb2015eb00 2024-03-06T07:28:31.203 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 64 2024-03-06T07:28:31.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.203+0000 7ffb27fe9640 1 -- 172.21.15.83:0/1523207456 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffb0406b4d0 msgr2=0x7ffb0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.203+0000 7ffb27fe9640 1 --2- 172.21.15.83:0/1523207456 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffb0406b4d0 0x7ffb0406d990 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7ffb1000ba50 tx=0x7ffb10008040 comp rx=0 tx=0).stop 2024-03-06T07:28:31.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.203+0000 7ffb27fe9640 1 -- 172.21.15.83:0/1523207456 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb2015eb00 msgr2=0x7ffb200b0970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.203+0000 7ffb27fe9640 1 --2- 172.21.15.83:0/1523207456 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb2015eb00 0x7ffb200b0970 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7ffb18007e70 tx=0x7ffb1800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.203+0000 7ffb27fe9640 1 -- 172.21.15.83:0/1523207456 shutdown_connections 2024-03-06T07:28:31.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.203+0000 7ffb27fe9640 1 --2- 172.21.15.83:0/1523207456 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffb0406b4d0 0x7ffb0406d990 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.203+0000 7ffb27fe9640 1 --2- 172.21.15.83:0/1523207456 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb2015eb00 0x7ffb200b0970 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.203+0000 7ffb27fe9640 1 --2- 172.21.15.83:0/1523207456 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffb200d14a0 0x7ffb200b0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.203+0000 7ffb27fe9640 1 --2- 172.21.15.83:0/1523207456 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffb200d10c0 0x7ffb20152600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.203+0000 7ffb27fe9640 1 -- 172.21.15.83:0/1523207456 >> 172.21.15.83:0/1523207456 conn(0x7ffb200b3800 msgr2=0x7ffb2015f760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:31.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.203+0000 7ffb27fe9640 1 -- 172.21.15.83:0/1523207456 shutdown_connections 2024-03-06T07:28:31.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.203+0000 7ffb27fe9640 1 -- 172.21.15.83:0/1523207456 wait complete. 2024-03-06T07:28:31.216 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:28:31.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.314+0000 7fe64b640640 1 Processor -- start 2024-03-06T07:28:31.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.314+0000 7fe64b640640 1 -- start start 2024-03-06T07:28:31.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.315+0000 7fe64b640640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe6440b4460 0x7fe6440d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.315+0000 7fe64b640640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe644161cf0 0x7fe644162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.315+0000 7fe64b640640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6440d22c0 con 0x7fe644162970 2024-03-06T07:28:31.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.315+0000 7fe64b640640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe644161b00 con 0x7fe644161cf0 2024-03-06T07:28:31.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.315+0000 7fe64b640640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6440cecc0 con 0x7fe6440b4460 2024-03-06T07:28:31.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.315+0000 7fe649bdd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe644161cf0 0x7fe644162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.315+0000 7fe648bdb640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fe644162970 0x7fe6440b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49094/0 (socket says 172.21.15.83:49094) 2024-03-06T07:28:31.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.315+0000 7fe648bdb640 1 -- 172.21.15.83:0/693573546 learned_addr learned my addr 172.21.15.83:0/693573546 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:28:31.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.315+0000 7fe649bdd640 1 -- 172.21.15.83:0/693573546 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe6440b4460 msgr2=0x7fe6440d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:31.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.315+0000 7fe6493dc640 1 --2- 172.21.15.83:0/693573546 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe6440b4460 0x7fe6440d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.316+0000 7fe649bdd640 1 --2- 172.21.15.83:0/693573546 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe6440b4460 0x7fe6440d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.316+0000 7fe63bfff640 1 -- 172.21.15.83:0/693573546 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 617717647 0 0) 0x7fe6440d22c0 con 0x7fe644162970 2024-03-06T07:28:31.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.316+0000 7fe649bdd640 1 -- 172.21.15.83:0/693573546 >> v1:172.21.15.83:6789/0 conn(0x7fe644162970 legacy=0x7fe6440b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.316+0000 7fe649bdd640 1 -- 172.21.15.83:0/693573546 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6440d25a0 con 0x7fe644161cf0 2024-03-06T07:28:31.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.316+0000 7fe6493dc640 1 --2- 172.21.15.83:0/693573546 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe6440b4460 0x7fe6440d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:31.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.316+0000 7fe649bdd640 1 --2- 172.21.15.83:0/693573546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe644161cf0 0x7fe644162180 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fe63c009d90 tx=0x7fe63c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=998c9746d40f7892 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.316+0000 7fe63bfff640 1 -- 172.21.15.83:0/693573546 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe63c039070 con 0x7fe644161cf0 2024-03-06T07:28:31.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.316+0000 7fe63bfff640 1 -- 172.21.15.83:0/693573546 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe63c02cc50 con 0x7fe644161cf0 2024-03-06T07:28:31.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.316+0000 7fe63bfff640 1 -- 172.21.15.83:0/693573546 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe63c034440 con 0x7fe644161cf0 2024-03-06T07:28:31.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.317+0000 7fe64b640640 1 -- 172.21.15.83:0/693573546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe644161cf0 msgr2=0x7fe644162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.317+0000 7fe64b640640 1 --2- 172.21.15.83:0/693573546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe644161cf0 0x7fe644162180 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fe63c009d90 tx=0x7fe63c02bea0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.317+0000 7fe64b640640 1 -- 172.21.15.83:0/693573546 shutdown_connections 2024-03-06T07:28:31.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.317+0000 7fe64b640640 1 --2- 172.21.15.83:0/693573546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe644161cf0 0x7fe644162180 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.317+0000 7fe64b640640 1 --2- 172.21.15.83:0/693573546 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe6440b4460 0x7fe6440d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.317+0000 7fe64b640640 1 -- 172.21.15.83:0/693573546 >> 172.21.15.83:0/693573546 conn(0x7fe6440d3ce0 msgr2=0x7fe6440d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:31.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.317+0000 7fe64b640640 1 -- 172.21.15.83:0/693573546 shutdown_connections 2024-03-06T07:28:31.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.317+0000 7fe64b640640 1 -- 172.21.15.83:0/693573546 wait complete. 2024-03-06T07:28:31.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.318+0000 7fe64b640640 1 Processor -- start 2024-03-06T07:28:31.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.318+0000 7fe64b640640 1 -- start start 2024-03-06T07:28:31.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.319+0000 7fe64b640640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6440b4460 0x7fe644156110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.319+0000 7fe64b640640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe644161cf0 0x7fe6440b1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.319+0000 7fe64b640640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe644162970 0x7fe6440b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.319+0000 7fe64b640640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6440ce950 con 0x7fe6440b4460 2024-03-06T07:28:31.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.319+0000 7fe64b640640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6440d2b60 con 0x7fe644161cf0 2024-03-06T07:28:31.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.319+0000 7fe64b640640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6440b4840 con 0x7fe644162970 2024-03-06T07:28:31.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.319+0000 7fe6493dc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6440b4460 0x7fe644156110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.319+0000 7fe6493dc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6440b4460 0x7fe644156110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52448/0 (socket says 172.21.15.83:52448) 2024-03-06T07:28:31.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.319+0000 7fe6493dc640 1 -- 172.21.15.83:0/863383877 learned_addr learned my addr 172.21.15.83:0/863383877 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:31.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.319+0000 7fe649bdd640 1 --2- 172.21.15.83:0/863383877 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe644162970 0x7fe6440b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.319+0000 7fe648bdb640 1 --2- 172.21.15.83:0/863383877 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe644161cf0 0x7fe6440b1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.319+0000 7fe6493dc640 1 -- 172.21.15.83:0/863383877 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe644162970 msgr2=0x7fe6440b1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.319+0000 7fe6493dc640 1 --2- 172.21.15.83:0/863383877 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe644162970 0x7fe6440b1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.319+0000 7fe6493dc640 1 -- 172.21.15.83:0/863383877 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe644161cf0 msgr2=0x7fe6440b1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.319+0000 7fe6493dc640 1 --2- 172.21.15.83:0/863383877 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe644161cf0 0x7fe6440b1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.319+0000 7fe6493dc640 1 -- 172.21.15.83:0/863383877 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6440b76e0 con 0x7fe6440b4460 2024-03-06T07:28:31.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.320+0000 7fe6493dc640 1 --2- 172.21.15.83:0/863383877 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6440b4460 0x7fe644156110 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fe63400c7f0 tx=0x7fe63400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.320+0000 7fe639ffb640 1 -- 172.21.15.83:0/863383877 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe634015070 con 0x7fe6440b4460 2024-03-06T07:28:31.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.320+0000 7fe639ffb640 1 -- 172.21.15.83:0/863383877 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe634007820 con 0x7fe6440b4460 2024-03-06T07:28:31.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.320+0000 7fe64b640640 1 -- 172.21.15.83:0/863383877 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6440c9a70 con 0x7fe6440b4460 2024-03-06T07:28:31.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.320+0000 7fe639ffb640 1 -- 172.21.15.83:0/863383877 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe634007bd0 con 0x7fe6440b4460 2024-03-06T07:28:31.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.320+0000 7fe64b640640 1 -- 172.21.15.83:0/863383877 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6440ca030 con 0x7fe6440b4460 2024-03-06T07:28:31.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.321+0000 7fe64b640640 1 -- 172.21.15.83:0/863383877 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe64404f330 con 0x7fe6440b4460 2024-03-06T07:28:31.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.321+0000 7fe639ffb640 1 -- 172.21.15.83:0/863383877 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe634018070 con 0x7fe6440b4460 2024-03-06T07:28:31.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.322+0000 7fe639ffb640 1 --2- 172.21.15.83:0/863383877 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe62806b4d0 0x7fe62806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.322+0000 7fe639ffb640 1 -- 172.21.15.83:0/863383877 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fe63408a2e0 con 0x7fe6440b4460 2024-03-06T07:28:31.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.322+0000 7fe648bdb640 1 --2- 172.21.15.83:0/863383877 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe62806b4d0 0x7fe62806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.322+0000 7fe648bdb640 1 --2- 172.21.15.83:0/863383877 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe62806b4d0 0x7fe62806d990 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fe6440a1bf0 tx=0x7fe640007d50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.326+0000 7fe639ffb640 1 -- 172.21.15.83:0/863383877 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe6440b76e0 con 0x7fe6440b4460 2024-03-06T07:28:31.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.517+0000 7fe64b640640 1 -- 172.21.15.83:0/863383877 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe6440b9b70 con 0x7fe6440b4460 2024-03-06T07:28:31.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.517+0000 7fe639ffb640 1 -- 172.21.15.83:0/863383877 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 4 entries v116) v1 ==== 80+0+244 (secure 0 0 0) 0x7fe634058ab0 con 0x7fe6440b4460 2024-03-06T07:28:31.519 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.183:6834/1383294888 2024-03-07T07:27:33.605177+0000 2024-03-06T07:28:31.520 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.183:6832/1383294888 2024-03-07T07:27:33.605177+0000 2024-03-06T07:28:31.520 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6836/855387870 2024-03-07T07:27:33.587133+0000 2024-03-06T07:28:31.520 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83:6834/855387870 2024-03-07T07:27:33.587133+0000 2024-03-06T07:28:31.520 INFO:teuthology.orchestra.run.smithi083.stderr:listed 4 entries 2024-03-06T07:28:31.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.519+0000 7fe64b640640 1 -- 172.21.15.83:0/863383877 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe62806b4d0 msgr2=0x7fe62806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.519+0000 7fe64b640640 1 --2- 172.21.15.83:0/863383877 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe62806b4d0 0x7fe62806d990 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fe6440a1bf0 tx=0x7fe640007d50 comp rx=0 tx=0).stop 2024-03-06T07:28:31.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.519+0000 7fe64b640640 1 -- 172.21.15.83:0/863383877 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6440b4460 msgr2=0x7fe644156110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.519+0000 7fe64b640640 1 --2- 172.21.15.83:0/863383877 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6440b4460 0x7fe644156110 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fe63400c7f0 tx=0x7fe63400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.520+0000 7fe64b640640 1 -- 172.21.15.83:0/863383877 shutdown_connections 2024-03-06T07:28:31.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.520+0000 7fe64b640640 1 --2- 172.21.15.83:0/863383877 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe62806b4d0 0x7fe62806d990 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.520+0000 7fe64b640640 1 --2- 172.21.15.83:0/863383877 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe644162970 0x7fe6440b1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.520+0000 7fe64b640640 1 --2- 172.21.15.83:0/863383877 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe644161cf0 0x7fe6440b1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.520+0000 7fe64b640640 1 --2- 172.21.15.83:0/863383877 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6440b4460 0x7fe644156110 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.520+0000 7fe64b640640 1 -- 172.21.15.83:0/863383877 >> 172.21.15.83:0/863383877 conn(0x7fe6440d3ce0 msgr2=0x7fe6440d0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:31.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.520+0000 7fe64b640640 1 -- 172.21.15.83:0/863383877 shutdown_connections 2024-03-06T07:28:31.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.520+0000 7fe64b640640 1 -- 172.21.15.83:0/863383877 wait complete. 2024-03-06T07:28:31.532 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi083.front.sepia.ceph.com... 2024-03-06T07:28:31.532 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:31.532 DEBUG:teuthology.orchestra.run.smithi083:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:31.561 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:ceph-fuse[74809]: fuse finished with error 0 and tester_r 0 2024-03-06T07:28:31.563 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi083.front.sepia.ceph.com 2024-03-06T07:28:31.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:31.563 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:28:31.647 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:28:31.647 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:31.665 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=287 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:28:31.665 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi183.front.sepia.ceph.com 2024-03-06T07:28:31.666 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:31.666 DEBUG:teuthology.orchestra.run.smithi183:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:28:31.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:28:31.723 INFO:teuthology.orchestra.run.smithi183.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:28:31.724 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:28:31.724 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:31.738 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1257 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:28:31.739 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi083.front.sepia.ceph.com 2024-03-06T07:28:31.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:31.739 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:28:31.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:28:31.793 INFO:teuthology.orchestra.run.smithi083.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:28:31.793 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:28:31.793 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:31.811 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=287 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:28:31.811 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi183.front.sepia.ceph.com 2024-03-06T07:28:31.811 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:31.811 DEBUG:teuthology.orchestra.run.smithi183:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:28:31.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:28:31.868 INFO:teuthology.orchestra.run.smithi183.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:28:31.868 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:31.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.966+0000 7f5b35023640 1 Processor -- start 2024-03-06T07:28:31.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.966+0000 7f5b35023640 1 -- start start 2024-03-06T07:28:31.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.967+0000 7f5b35023640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5b300d22b0 0x7f5b300d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.967+0000 7f5b35023640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b300b3430 con 0x7f5b3015dfe0 2024-03-06T07:28:31.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.967+0000 7f5b35023640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b300b3710 con 0x7f5b300b41f0 2024-03-06T07:28:31.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.967+0000 7f5b35023640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b300d2c50 con 0x7f5b300d22b0 2024-03-06T07:28:31.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.967+0000 7f5b2ed9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f5b3015dfe0 0x7f5b300cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49102/0 (socket says 172.21.15.83:49102) 2024-03-06T07:28:31.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.967+0000 7f5b2ed9d640 1 -- 172.21.15.83:0/3709982662 learned_addr learned my addr 172.21.15.83:0/3709982662 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:28:31.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.967+0000 7f5b2dd9b640 1 -- 172.21.15.83:0/3709982662 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 805309987 0 0) 0x7f5b300b3430 con 0x7f5b3015dfe0 2024-03-06T07:28:31.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.967+0000 7f5b2e59c640 1 --2- 172.21.15.83:0/3709982662 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5b300d22b0 0x7f5b300d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.967+0000 7f5b2dd9b640 1 -- 172.21.15.83:0/3709982662 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b18003520 con 0x7f5b3015dfe0 2024-03-06T07:28:31.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.968+0000 7f5b2dd9b640 1 -- 172.21.15.83:0/3709982662 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 441000609 0 0) 0x7f5b18003520 con 0x7f5b3015dfe0 2024-03-06T07:28:31.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.968+0000 7f5b2dd9b640 1 -- 172.21.15.83:0/3709982662 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5b300d22b0 msgr2=0x7f5b300d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:31.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.968+0000 7f5b2dd9b640 1 --2- 172.21.15.83:0/3709982662 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5b300d22b0 0x7f5b300d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.968+0000 7f5b2dd9b640 1 -- 172.21.15.83:0/3709982662 >> v1:172.21.15.183:6789/0 conn(0x7f5b300b41f0 legacy=0x7f5b300b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.968+0000 7f5b2dd9b640 1 -- 172.21.15.83:0/3709982662 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b300b39f0 con 0x7f5b3015dfe0 2024-03-06T07:28:31.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.968+0000 7f5b2dd9b640 1 -- 172.21.15.83:0/3709982662 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f5b28001670 con 0x7f5b3015dfe0 2024-03-06T07:28:31.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.968+0000 7f5b2dd9b640 1 -- 172.21.15.83:0/3709982662 >> v1:172.21.15.83:6789/0 conn(0x7f5b3015dfe0 legacy=0x7f5b300cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.968+0000 7f5b2dd9b640 1 --2- 172.21.15.83:0/3709982662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5b180039e0 0x7f5b18003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.968+0000 7f5b2dd9b640 1 --2- 172.21.15.83:0/3709982662 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5b300d22b0 0x7f5b18004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.968+0000 7f5b2dd9b640 1 --2- 172.21.15.83:0/3709982662 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b18004d40 0x7f5b180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.968+0000 7f5b2dd9b640 1 -- 172.21.15.83:0/3709982662 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1800ba70 con 0x7f5b180039e0 2024-03-06T07:28:31.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.968+0000 7f5b2dd9b640 1 -- 172.21.15.83:0/3709982662 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1800bbf0 con 0x7f5b18004d40 2024-03-06T07:28:31.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.968+0000 7f5b2dd9b640 1 -- 172.21.15.83:0/3709982662 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1800bdf0 con 0x7f5b300d22b0 2024-03-06T07:28:31.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.968+0000 7f5b2ed9d640 1 --2- 172.21.15.83:0/3709982662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5b180039e0 0x7f5b18003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.968+0000 7f5b2e59c640 1 --2- 172.21.15.83:0/3709982662 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5b300d22b0 0x7f5b18004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.968+0000 7f5b2f59e640 1 --2- 172.21.15.83:0/3709982662 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b18004d40 0x7f5b180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.968+0000 7f5b2ed9d640 1 -- 172.21.15.83:0/3709982662 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5b300d22b0 msgr2=0x7f5b18004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:31.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.968+0000 7f5b2ed9d640 1 --2- 172.21.15.83:0/3709982662 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5b300d22b0 0x7f5b18004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.968+0000 7f5b2ed9d640 1 -- 172.21.15.83:0/3709982662 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b18004d40 msgr2=0x7f5b180051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:31.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.968+0000 7f5b2ed9d640 1 --2- 172.21.15.83:0/3709982662 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b18004d40 0x7f5b180051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.968+0000 7f5b2ed9d640 1 -- 172.21.15.83:0/3709982662 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b18003520 con 0x7f5b180039e0 2024-03-06T07:28:31.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.969+0000 7f5b2ed9d640 1 --2- 172.21.15.83:0/3709982662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5b180039e0 0x7f5b18003e00 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f5b2800c210 tx=0x7f5b18005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=161992e9f9928df6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.969+0000 7f5b2dd9b640 1 -- 172.21.15.83:0/3709982662 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b28007d10 con 0x7f5b180039e0 2024-03-06T07:28:31.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.969+0000 7f5b2dd9b640 1 -- 172.21.15.83:0/3709982662 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b28030070 con 0x7f5b180039e0 2024-03-06T07:28:31.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.969+0000 7f5b2dd9b640 1 -- 172.21.15.83:0/3709982662 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b28005d50 con 0x7f5b180039e0 2024-03-06T07:28:31.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.969+0000 7f5b35023640 1 -- 172.21.15.83:0/3709982662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5b180039e0 msgr2=0x7f5b18003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.969+0000 7f5b35023640 1 --2- 172.21.15.83:0/3709982662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5b180039e0 0x7f5b18003e00 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f5b2800c210 tx=0x7f5b18005830 comp rx=0 tx=0).stop 2024-03-06T07:28:31.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.969+0000 7f5b2f59e640 1 -- 172.21.15.83:0/3709982662 reap_dead start 2024-03-06T07:28:31.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.969+0000 7f5b35023640 1 -- 172.21.15.83:0/3709982662 shutdown_connections 2024-03-06T07:28:31.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.969+0000 7f5b35023640 1 -- 172.21.15.83:0/3709982662 >> 172.21.15.83:0/3709982662 conn(0x7f5b300d3cd0 msgr2=0x7f5b300d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:31.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.969+0000 7f5b35023640 1 -- 172.21.15.83:0/3709982662 shutdown_connections 2024-03-06T07:28:31.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.969+0000 7f5b35023640 1 -- 172.21.15.83:0/3709982662 wait complete. 2024-03-06T07:28:31.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.972+0000 7f5b35023640 1 Processor -- start 2024-03-06T07:28:31.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.972+0000 7f5b35023640 1 -- start start 2024-03-06T07:28:31.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.972+0000 7f5b35023640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5b180039e0 0x7f5b300b75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.972+0000 7f5b35023640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b300b8490 0x7f5b300b4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.972+0000 7f5b35023640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5b3014dce0 0x7f5b3014e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.972+0000 7f5b35023640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b300b3d60 con 0x7f5b3014dce0 2024-03-06T07:28:31.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.972+0000 7f5b35023640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b300b31e0 con 0x7f5b300b8490 2024-03-06T07:28:31.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.972+0000 7f5b35023640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b300d1c70 con 0x7f5b180039e0 2024-03-06T07:28:31.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.973+0000 7f5b2f59e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5b3014dce0 0x7f5b3014e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.973+0000 7f5b2f59e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5b3014dce0 0x7f5b3014e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52472/0 (socket says 172.21.15.83:52472) 2024-03-06T07:28:31.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.973+0000 7f5b2f59e640 1 -- 172.21.15.83:0/1480211090 learned_addr learned my addr 172.21.15.83:0/1480211090 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:31.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.973+0000 7f5b2ed9d640 1 --2- 172.21.15.83:0/1480211090 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5b180039e0 0x7f5b300b75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.973+0000 7f5b2e59c640 1 --2- 172.21.15.83:0/1480211090 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b300b8490 0x7f5b300b4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.973+0000 7f5b2f59e640 1 -- 172.21.15.83:0/1480211090 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5b180039e0 msgr2=0x7f5b300b75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.973+0000 7f5b2f59e640 1 --2- 172.21.15.83:0/1480211090 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5b180039e0 0x7f5b300b75c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.973+0000 7f5b2f59e640 1 -- 172.21.15.83:0/1480211090 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b300b8490 msgr2=0x7f5b300b4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.973+0000 7f5b2f59e640 1 --2- 172.21.15.83:0/1480211090 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b300b8490 0x7f5b300b4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.973+0000 7f5b2f59e640 1 -- 172.21.15.83:0/1480211090 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b300b8ca0 con 0x7f5b3014dce0 2024-03-06T07:28:31.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.973+0000 7f5b2f59e640 1 --2- 172.21.15.83:0/1480211090 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5b3014dce0 0x7f5b3014e190 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f5b20000c00 tx=0x7f5b2000e8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.973+0000 7f5b177fe640 1 -- 172.21.15.83:0/1480211090 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b20019070 con 0x7f5b3014dce0 2024-03-06T07:28:31.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.973+0000 7f5b177fe640 1 -- 172.21.15.83:0/1480211090 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b200052e0 con 0x7f5b3014dce0 2024-03-06T07:28:31.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.973+0000 7f5b177fe640 1 -- 172.21.15.83:0/1480211090 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b20003830 con 0x7f5b3014dce0 2024-03-06T07:28:31.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.973+0000 7f5b35023640 1 -- 172.21.15.83:0/1480211090 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b300b8f80 con 0x7f5b3014dce0 2024-03-06T07:28:31.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.973+0000 7f5b35023640 1 -- 172.21.15.83:0/1480211090 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b300b9540 con 0x7f5b3014dce0 2024-03-06T07:28:31.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.974+0000 7f5b35023640 1 -- 172.21.15.83:0/1480211090 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b3004f330 con 0x7f5b3014dce0 2024-03-06T07:28:31.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.975+0000 7f5b177fe640 1 -- 172.21.15.83:0/1480211090 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5b2001c070 con 0x7f5b3014dce0 2024-03-06T07:28:31.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.975+0000 7f5b177fe640 1 --2- 172.21.15.83:0/1480211090 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5b1c06b500 0x7f5b1c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.975+0000 7f5b2ed9d640 1 --2- 172.21.15.83:0/1480211090 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5b1c06b500 0x7f5b1c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.975+0000 7f5b177fe640 1 -- 172.21.15.83:0/1480211090 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f5b20089620 con 0x7f5b3014dce0 2024-03-06T07:28:31.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.976+0000 7f5b2ed9d640 1 --2- 172.21.15.83:0/1480211090 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5b1c06b500 0x7f5b1c06d9c0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f5b28034000 tx=0x7f5b28009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:31.979+0000 7f5b177fe640 1 -- 172.21.15.83:0/1480211090 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5b20057d60 con 0x7f5b3014dce0 2024-03-06T07:28:32.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.193+0000 7f5b35023640 1 -- 172.21.15.83:0/1480211090 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b300d00c0 con 0x7f5b3014dce0 2024-03-06T07:28:32.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.194+0000 7f5b177fe640 1 -- 172.21.15.83:0/1480211090 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f5b2005be00 con 0x7f5b3014dce0 2024-03-06T07:28:32.196 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:28:32.196 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7051,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/101642171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":101642171},{"type":"v1","addr":"172.21.15.183:6833","nonce":101642171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7057,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3146884900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":3146884900},{"type":"v1","addr":"172.21.15.183:6835","nonce":3146884900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8208,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1922041621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1922041621},{"type":"v1","addr":"172.21.15.83:6835","nonce":1922041621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.491022+0000","modified":"2024-03-06T07:27:37.006649+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8229":{"gid":8229,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/2003687153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2003687153},{"type":"v1","addr":"172.21.15.83:6837","nonce":2003687153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8229,"qdb_cluster":[8229]},"id":8}]} 2024-03-06T07:28:32.196 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 66 2024-03-06T07:28:32.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.196+0000 7f5b35023640 1 -- 172.21.15.83:0/1480211090 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5b1c06b500 msgr2=0x7f5b1c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.196+0000 7f5b35023640 1 --2- 172.21.15.83:0/1480211090 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5b1c06b500 0x7f5b1c06d9c0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f5b28034000 tx=0x7f5b28009030 comp rx=0 tx=0).stop 2024-03-06T07:28:32.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.197+0000 7f5b35023640 1 -- 172.21.15.83:0/1480211090 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5b3014dce0 msgr2=0x7f5b3014e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.197+0000 7f5b35023640 1 --2- 172.21.15.83:0/1480211090 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5b3014dce0 0x7f5b3014e190 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f5b20000c00 tx=0x7f5b2000e8a0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.197+0000 7f5b35023640 1 -- 172.21.15.83:0/1480211090 shutdown_connections 2024-03-06T07:28:32.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.197+0000 7f5b35023640 1 --2- 172.21.15.83:0/1480211090 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5b1c06b500 0x7f5b1c06d9c0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.197+0000 7f5b35023640 1 --2- 172.21.15.83:0/1480211090 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5b3014dce0 0x7f5b3014e190 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.197+0000 7f5b35023640 1 --2- 172.21.15.83:0/1480211090 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b300b8490 0x7f5b300b4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.197+0000 7f5b35023640 1 --2- 172.21.15.83:0/1480211090 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5b180039e0 0x7f5b300b75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.197+0000 7f5b35023640 1 -- 172.21.15.83:0/1480211090 >> 172.21.15.83:0/1480211090 conn(0x7f5b300d3cd0 msgr2=0x7f5b300d08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:32.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.197+0000 7f5b35023640 1 -- 172.21.15.83:0/1480211090 shutdown_connections 2024-03-06T07:28:32.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.197+0000 7f5b35023640 1 -- 172.21.15.83:0/1480211090 wait complete. 2024-03-06T07:28:32.213 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:32.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.313+0000 7f9ea9ca4640 1 Processor -- start 2024-03-06T07:28:32.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.313+0000 7f9ea9ca4640 1 -- start start 2024-03-06T07:28:32.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.313+0000 7f9ea9ca4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ea40d22b0 0x7f9ea40d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.313+0000 7f9ea9ca4640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f9ea40b41f0 0x7f9ea40b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.313+0000 7f9ea9ca4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ea40b3430 con 0x7f9ea415dfe0 2024-03-06T07:28:32.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.313+0000 7f9ea9ca4640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ea40d2c50 con 0x7f9ea40b41f0 2024-03-06T07:28:32.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.313+0000 7f9ea9ca4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea40b31e0 con 0x7f9ea40d22b0 2024-03-06T07:28:32.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.314+0000 7f9e9bfff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ea40d22b0 0x7f9ea40d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.314+0000 7f9ea3fff640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f9ea40b41f0 0x7f9ea40b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.314+0000 7f9e9bfff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ea40d22b0 0x7f9ea40d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:43060/0 (socket says 172.21.15.83:43060) 2024-03-06T07:28:32.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.314+0000 7f9ea3fff640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f9ea40b41f0 0x7f9ea40b4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:56822/0 (socket says 172.21.15.83:56822) 2024-03-06T07:28:32.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.314+0000 7f9e9bfff640 1 -- 172.21.15.83:0/3723038823 learned_addr learned my addr 172.21.15.83:0/3723038823 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:32.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.314+0000 7f9ea37fe640 1 --1- 172.21.15.83:0/3723038823 >> v1:172.21.15.183:6789/0 conn(0x7f9ea415dfe0 0x7f9ea40cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:33390/0 (socket says 172.21.15.83:33390) 2024-03-06T07:28:32.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.314+0000 7f9e9bfff640 1 -- 172.21.15.83:0/3723038823 >> v2:172.21.15.183:3300/0 conn(0x7f9ea40b41f0 msgr2=0x7f9ea40b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:32.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.314+0000 7f9e9bfff640 1 --2- 172.21.15.83:0/3723038823 >> v2:172.21.15.183:3300/0 conn(0x7f9ea40b41f0 0x7f9ea40b4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.314+0000 7f9e9bfff640 1 -- 172.21.15.83:0/3723038823 >> v1:172.21.15.183:6789/0 conn(0x7f9ea415dfe0 legacy=0x7f9ea40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.314+0000 7f9e9bfff640 1 -- 172.21.15.83:0/3723038823 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ea40b3710 con 0x7f9ea40d22b0 2024-03-06T07:28:32.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.314+0000 7f9ea3fff640 1 --2- 172.21.15.83:0/3723038823 >> v2:172.21.15.183:3300/0 conn(0x7f9ea40b41f0 0x7f9ea40b4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:32.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.315+0000 7f9e9bfff640 1 --2- 172.21.15.83:0/3723038823 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ea40d22b0 0x7f9ea40d2710 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c009870 tx=0x7f9e8c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ff2fadd729e3bba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.315+0000 7f9ea2ffd640 1 -- 172.21.15.83:0/3723038823 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c034070 con 0x7f9ea40d22b0 2024-03-06T07:28:32.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.315+0000 7f9ea2ffd640 1 -- 172.21.15.83:0/3723038823 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e8c030840 con 0x7f9ea40d22b0 2024-03-06T07:28:32.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.315+0000 7f9ea2ffd640 1 -- 172.21.15.83:0/3723038823 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c030c10 con 0x7f9ea40d22b0 2024-03-06T07:28:32.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.316+0000 7f9ea9ca4640 1 -- 172.21.15.83:0/3723038823 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ea40d22b0 msgr2=0x7f9ea40d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.316+0000 7f9ea9ca4640 1 --2- 172.21.15.83:0/3723038823 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ea40d22b0 0x7f9ea40d2710 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c009870 tx=0x7f9e8c02b980 comp rx=0 tx=0).stop 2024-03-06T07:28:32.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.316+0000 7f9ea9ca4640 1 -- 172.21.15.83:0/3723038823 shutdown_connections 2024-03-06T07:28:32.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.316+0000 7f9ea9ca4640 1 --2- 172.21.15.83:0/3723038823 >> v2:172.21.15.183:3300/0 conn(0x7f9ea40b41f0 0x7f9ea40b4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.316+0000 7f9ea9ca4640 1 --2- 172.21.15.83:0/3723038823 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ea40d22b0 0x7f9ea40d2710 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.316+0000 7f9ea9ca4640 1 -- 172.21.15.83:0/3723038823 >> 172.21.15.83:0/3723038823 conn(0x7f9ea40d3cd0 msgr2=0x7f9ea40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:32.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.316+0000 7f9ea9ca4640 1 -- 172.21.15.83:0/3723038823 shutdown_connections 2024-03-06T07:28:32.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.316+0000 7f9ea9ca4640 1 -- 172.21.15.83:0/3723038823 wait complete. 2024-03-06T07:28:32.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.317+0000 7f9ea9ca4640 1 Processor -- start 2024-03-06T07:28:32.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.317+0000 7f9ea9ca4640 1 -- start start 2024-03-06T07:28:32.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.317+0000 7f9ea9ca4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ea40b41f0 0x7f9ea40b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.318+0000 7f9ea9ca4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ea40d22b0 0x7f9ea40b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.318+0000 7f9ea9ca4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9ea415dfe0 0x7f9ea40ccf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.318+0000 7f9ea9ca4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea40cfa00 con 0x7f9ea40d22b0 2024-03-06T07:28:32.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.318+0000 7f9ea9ca4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea40b3d60 con 0x7f9ea415dfe0 2024-03-06T07:28:32.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.318+0000 7f9ea9ca4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea40d1c70 con 0x7f9ea40b41f0 2024-03-06T07:28:32.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.318+0000 7f9e9bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ea40d22b0 0x7f9ea40b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.318+0000 7f9e9bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ea40d22b0 0x7f9ea40b68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52486/0 (socket says 172.21.15.83:52486) 2024-03-06T07:28:32.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.318+0000 7f9e9bfff640 1 -- 172.21.15.83:0/1758913182 learned_addr learned my addr 172.21.15.83:0/1758913182 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:32.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.318+0000 7f9e9bfff640 1 -- 172.21.15.83:0/1758913182 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ea40b41f0 msgr2=0x7f9ea40b2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.318+0000 7f9ea37fe640 1 --2- 172.21.15.83:0/1758913182 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ea40b41f0 0x7f9ea40b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.318+0000 7f9ea3fff640 1 --2- 172.21.15.83:0/1758913182 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9ea415dfe0 0x7f9ea40ccf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.318+0000 7f9e9bfff640 1 --2- 172.21.15.83:0/1758913182 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ea40b41f0 0x7f9ea40b2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.318+0000 7f9e9bfff640 1 -- 172.21.15.83:0/1758913182 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9ea415dfe0 msgr2=0x7f9ea40ccf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.318+0000 7f9e9bfff640 1 --2- 172.21.15.83:0/1758913182 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9ea415dfe0 0x7f9ea40ccf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.318+0000 7f9e9bfff640 1 -- 172.21.15.83:0/1758913182 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ea40b90e0 con 0x7f9ea40d22b0 2024-03-06T07:28:32.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.318+0000 7f9ea3fff640 1 --2- 172.21.15.83:0/1758913182 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9ea415dfe0 0x7f9ea40ccf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:32.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.318+0000 7f9ea37fe640 1 --2- 172.21.15.83:0/1758913182 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ea40b41f0 0x7f9ea40b2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:32.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.318+0000 7f9e9bfff640 1 --2- 172.21.15.83:0/1758913182 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ea40d22b0 0x7f9ea40b68a0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c000c00 tx=0x7f9e8c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.318+0000 7f9ea0ff9640 1 -- 172.21.15.83:0/1758913182 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c048070 con 0x7f9ea40d22b0 2024-03-06T07:28:32.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.318+0000 7f9ea0ff9640 1 -- 172.21.15.83:0/1758913182 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e8c030640 con 0x7f9ea40d22b0 2024-03-06T07:28:32.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.318+0000 7f9ea9ca4640 1 -- 172.21.15.83:0/1758913182 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ea40b93c0 con 0x7f9ea40d22b0 2024-03-06T07:28:32.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.318+0000 7f9ea0ff9640 1 -- 172.21.15.83:0/1758913182 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c034040 con 0x7f9ea40d22b0 2024-03-06T07:28:32.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.318+0000 7f9ea9ca4640 1 -- 172.21.15.83:0/1758913182 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ea40b8cd0 con 0x7f9ea40d22b0 2024-03-06T07:28:32.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.319+0000 7f9ea9ca4640 1 -- 172.21.15.83:0/1758913182 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ea404f330 con 0x7f9ea40d22b0 2024-03-06T07:28:32.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.320+0000 7f9ea0ff9640 1 -- 172.21.15.83:0/1758913182 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9e8c04b070 con 0x7f9ea40d22b0 2024-03-06T07:28:32.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.320+0000 7f9ea0ff9640 1 --2- 172.21.15.83:0/1758913182 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9e8406b4d0 0x7f9e8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.320+0000 7f9ea0ff9640 1 -- 172.21.15.83:0/1758913182 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f9e8c0ac0d0 con 0x7f9ea40d22b0 2024-03-06T07:28:32.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.321+0000 7f9ea37fe640 1 --2- 172.21.15.83:0/1758913182 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9e8406b4d0 0x7f9e8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.321+0000 7f9ea37fe640 1 --2- 172.21.15.83:0/1758913182 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9e8406b4d0 0x7f9e8406d990 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f9e94000f60 tx=0x7f9e94009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.325+0000 7f9ea0ff9640 1 -- 172.21.15.83:0/1758913182 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9e8c07aef0 con 0x7f9ea40d22b0 2024-03-06T07:28:32.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.503+0000 7f9ea9ca4640 1 -- 172.21.15.83:0/1758913182 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ea40b9b90 con 0x7f9ea40d22b0 2024-03-06T07:28:32.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.504+0000 7f9ea0ff9640 1 -- 172.21.15.83:0/1758913182 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+16992 (secure 0 0 0) 0x7f9e8c07a7f0 con 0x7f9ea40d22b0 2024-03-06T07:28:32.506 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:28:32.506 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":116,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:27:41.021144+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:08.709967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:09.456518+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.26","mappings":[{"from":0,"to":2}]},{"pgid":"17.30","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.183:6834/1383294888":"2024-03-07T07:27:33.605177+0000","172.21.15.183:6832/1383294888":"2024-03-07T07:27:33.605177+0000","172.21.15.83:6836/855387870":"2024-03-07T07:27:33.587133+0000","172.21.15.83:6834/855387870":"2024-03-07T07:27:33.587133+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:32.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.506+0000 7f9ea9ca4640 1 -- 172.21.15.83:0/1758913182 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9e8406b4d0 msgr2=0x7f9e8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.506+0000 7f9ea9ca4640 1 --2- 172.21.15.83:0/1758913182 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9e8406b4d0 0x7f9e8406d990 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f9e94000f60 tx=0x7f9e94009040 comp rx=0 tx=0).stop 2024-03-06T07:28:32.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.506+0000 7f9ea9ca4640 1 -- 172.21.15.83:0/1758913182 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ea40d22b0 msgr2=0x7f9ea40b68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.506+0000 7f9ea9ca4640 1 --2- 172.21.15.83:0/1758913182 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ea40d22b0 0x7f9ea40b68a0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c000c00 tx=0x7f9e8c004730 comp rx=0 tx=0).stop 2024-03-06T07:28:32.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.506+0000 7f9ea9ca4640 1 -- 172.21.15.83:0/1758913182 shutdown_connections 2024-03-06T07:28:32.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.506+0000 7f9ea9ca4640 1 --2- 172.21.15.83:0/1758913182 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9e8406b4d0 0x7f9e8406d990 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.506+0000 7f9ea9ca4640 1 --2- 172.21.15.83:0/1758913182 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9ea415dfe0 0x7f9ea40ccf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.506+0000 7f9ea9ca4640 1 --2- 172.21.15.83:0/1758913182 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ea40d22b0 0x7f9ea40b68a0 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.506+0000 7f9ea9ca4640 1 --2- 172.21.15.83:0/1758913182 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ea40b41f0 0x7f9ea40b2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.506+0000 7f9ea9ca4640 1 -- 172.21.15.83:0/1758913182 >> 172.21.15.83:0/1758913182 conn(0x7f9ea40d3cd0 msgr2=0x7f9ea40b4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:32.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.506+0000 7f9ea9ca4640 1 -- 172.21.15.83:0/1758913182 shutdown_connections 2024-03-06T07:28:32.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.506+0000 7f9ea9ca4640 1 -- 172.21.15.83:0/1758913182 wait complete. 2024-03-06T07:28:32.519 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:08.709967+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:09.456518+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:32.520 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:32.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.621+0000 7fd0c0725640 1 Processor -- start 2024-03-06T07:28:32.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.621+0000 7fd0c0725640 1 -- start start 2024-03-06T07:28:32.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.622+0000 7fd0c0725640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0b80d4860 0x7fd0b80d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.622+0000 7fd0c0725640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd0b80b0e30 0x7fd0b80b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.622+0000 7fd0c0725640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0b81618b0 con 0x7fd0b81614d0 2024-03-06T07:28:32.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.622+0000 7fd0c0725640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0b80d4cc0 con 0x7fd0b80b0e30 2024-03-06T07:28:32.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.622+0000 7fd0c0725640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b80cf400 con 0x7fd0b80d4860 2024-03-06T07:28:32.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.622+0000 7fd0becc2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd0b80b0e30 0x7fd0b80b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.622+0000 7fd0becc2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd0b80b0e30 0x7fd0b80b12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52488/0 (socket says 172.21.15.83:52488) 2024-03-06T07:28:32.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.622+0000 7fd0becc2640 1 -- 172.21.15.83:0/1579244771 learned_addr learned my addr 172.21.15.83:0/1579244771 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:32.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.622+0000 7fd0bdcc0640 1 --2- 172.21.15.83:0/1579244771 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0b80d4860 0x7fd0b80d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.622+0000 7fd0becc2640 1 -- 172.21.15.83:0/1579244771 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0b80d4860 msgr2=0x7fd0b80d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:32.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.622+0000 7fd0becc2640 1 --2- 172.21.15.83:0/1579244771 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0b80d4860 0x7fd0b80d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.622+0000 7fd0becc2640 1 -- 172.21.15.83:0/1579244771 >> v1:172.21.15.183:6789/0 conn(0x7fd0b81614d0 legacy=0x7fd0b80d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.622+0000 7fd0becc2640 1 -- 172.21.15.83:0/1579244771 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b80cf6a0 con 0x7fd0b80b0e30 2024-03-06T07:28:32.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.623+0000 7fd0becc2640 1 --2- 172.21.15.83:0/1579244771 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0b80b0e30 0x7fd0b80b12c0 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0009e50 tx=0x7fd0b002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e2493e64e430ae2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.623+0000 7fd0bd4bf640 1 -- 172.21.15.83:0/1579244771 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b002cb00 con 0x7fd0b80b0e30 2024-03-06T07:28:32.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.623+0000 7fd0bd4bf640 1 -- 172.21.15.83:0/1579244771 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0b002cca0 con 0x7fd0b80b0e30 2024-03-06T07:28:32.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.623+0000 7fd0bd4bf640 1 -- 172.21.15.83:0/1579244771 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b0034610 con 0x7fd0b80b0e30 2024-03-06T07:28:32.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.623+0000 7fd0c0725640 1 -- 172.21.15.83:0/1579244771 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0b80b0e30 msgr2=0x7fd0b80b12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.623+0000 7fd0c0725640 1 --2- 172.21.15.83:0/1579244771 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0b80b0e30 0x7fd0b80b12c0 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0009e50 tx=0x7fd0b002bf60 comp rx=0 tx=0).stop 2024-03-06T07:28:32.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.624+0000 7fd0c0725640 1 -- 172.21.15.83:0/1579244771 shutdown_connections 2024-03-06T07:28:32.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.624+0000 7fd0c0725640 1 --2- 172.21.15.83:0/1579244771 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0b80b0e30 0x7fd0b80b12c0 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.624+0000 7fd0c0725640 1 --2- 172.21.15.83:0/1579244771 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0b80d4860 0x7fd0b80d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.624+0000 7fd0c0725640 1 -- 172.21.15.83:0/1579244771 >> 172.21.15.83:0/1579244771 conn(0x7fd0b80d3d40 msgr2=0x7fd0b80d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:32.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.624+0000 7fd0c0725640 1 -- 172.21.15.83:0/1579244771 shutdown_connections 2024-03-06T07:28:32.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.624+0000 7fd0c0725640 1 -- 172.21.15.83:0/1579244771 wait complete. 2024-03-06T07:28:32.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.625+0000 7fd0c0725640 1 Processor -- start 2024-03-06T07:28:32.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.625+0000 7fd0c0725640 1 -- start start 2024-03-06T07:28:32.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.625+0000 7fd0c0725640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0b80b0e30 0x7fd0b815ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.625+0000 7fd0c0725640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0b80d4860 0x7fd0b80c89b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.625+0000 7fd0c0725640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd0b81614d0 0x7fd0b80cce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.625+0000 7fd0c0725640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b80b0910 con 0x7fd0b80b0e30 2024-03-06T07:28:32.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.625+0000 7fd0c0725640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b80cfc60 con 0x7fd0b81614d0 2024-03-06T07:28:32.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.625+0000 7fd0c0725640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b80d2560 con 0x7fd0b80d4860 2024-03-06T07:28:32.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.626+0000 7fd0be4c1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0b80b0e30 0x7fd0b815ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.626+0000 7fd0be4c1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0b80b0e30 0x7fd0b815ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52494/0 (socket says 172.21.15.83:52494) 2024-03-06T07:28:32.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.626+0000 7fd0be4c1640 1 -- 172.21.15.83:0/4258125514 learned_addr learned my addr 172.21.15.83:0/4258125514 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:32.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.626+0000 7fd0bdcc0640 1 --2- 172.21.15.83:0/4258125514 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0b80d4860 0x7fd0b80c89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.626+0000 7fd0becc2640 1 --2- 172.21.15.83:0/4258125514 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd0b81614d0 0x7fd0b80cce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.626+0000 7fd0be4c1640 1 -- 172.21.15.83:0/4258125514 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0b80d4860 msgr2=0x7fd0b80c89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.626+0000 7fd0be4c1640 1 --2- 172.21.15.83:0/4258125514 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0b80d4860 0x7fd0b80c89b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.626+0000 7fd0be4c1640 1 -- 172.21.15.83:0/4258125514 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd0b81614d0 msgr2=0x7fd0b80cce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.626+0000 7fd0be4c1640 1 --2- 172.21.15.83:0/4258125514 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd0b81614d0 0x7fd0b80cce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.626+0000 7fd0be4c1640 1 -- 172.21.15.83:0/4258125514 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b815f380 con 0x7fd0b80b0e30 2024-03-06T07:28:32.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.626+0000 7fd0bdcc0640 1 --2- 172.21.15.83:0/4258125514 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0b80d4860 0x7fd0b80c89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:32.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.626+0000 7fd0becc2640 1 --2- 172.21.15.83:0/4258125514 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd0b81614d0 0x7fd0b80cce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:32.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.626+0000 7fd0be4c1640 1 --2- 172.21.15.83:0/4258125514 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0b80b0e30 0x7fd0b815ee40 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7fd0b400c3c0 tx=0x7fd0b400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.626+0000 7fd0aeffd640 1 -- 172.21.15.83:0/4258125514 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b40052d0 con 0x7fd0b80b0e30 2024-03-06T07:28:32.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.626+0000 7fd0aeffd640 1 -- 172.21.15.83:0/4258125514 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0b4005490 con 0x7fd0b80b0e30 2024-03-06T07:28:32.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.626+0000 7fd0aeffd640 1 -- 172.21.15.83:0/4258125514 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b4007260 con 0x7fd0b80b0e30 2024-03-06T07:28:32.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.626+0000 7fd0c0725640 1 -- 172.21.15.83:0/4258125514 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0b80c7910 con 0x7fd0b80b0e30 2024-03-06T07:28:32.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.626+0000 7fd0c0725640 1 -- 172.21.15.83:0/4258125514 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0b80c7ed0 con 0x7fd0b80b0e30 2024-03-06T07:28:32.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.627+0000 7fd0c0725640 1 -- 172.21.15.83:0/4258125514 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0b80b4d50 con 0x7fd0b80b0e30 2024-03-06T07:28:32.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.628+0000 7fd0aeffd640 1 -- 172.21.15.83:0/4258125514 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd0b4007400 con 0x7fd0b80b0e30 2024-03-06T07:28:32.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.628+0000 7fd0aeffd640 1 --2- 172.21.15.83:0/4258125514 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd09c06b500 0x7fd09c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.628+0000 7fd0aeffd640 1 -- 172.21.15.83:0/4258125514 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fd0b4089740 con 0x7fd0b80b0e30 2024-03-06T07:28:32.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.628+0000 7fd0bdcc0640 1 --2- 172.21.15.83:0/4258125514 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd09c06b500 0x7fd09c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.629+0000 7fd0bdcc0640 1 --2- 172.21.15.83:0/4258125514 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd09c06b500 0x7fd09c06d9c0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fd0b80cde10 tx=0x7fd0a8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.632+0000 7fd0aeffd640 1 -- 172.21.15.83:0/4258125514 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd0b40584d0 con 0x7fd0b80b0e30 2024-03-06T07:28:32.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.809+0000 7fd0c0725640 1 -- 172.21.15.83:0/4258125514 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd0b80d0040 con 0x7fd0b80b0e30 2024-03-06T07:28:32.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.809+0000 7fd0aeffd640 1 -- 172.21.15.83:0/4258125514 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v116) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd0b4057dd0 con 0x7fd0b80b0e30 2024-03-06T07:28:32.811 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:28:32.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.811+0000 7fd0c0725640 1 -- 172.21.15.83:0/4258125514 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd09c06b500 msgr2=0x7fd09c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.811+0000 7fd0c0725640 1 --2- 172.21.15.83:0/4258125514 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd09c06b500 0x7fd09c06d9c0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fd0b80cde10 tx=0x7fd0a8008040 comp rx=0 tx=0).stop 2024-03-06T07:28:32.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.811+0000 7fd0c0725640 1 -- 172.21.15.83:0/4258125514 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0b80b0e30 msgr2=0x7fd0b815ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.811+0000 7fd0c0725640 1 --2- 172.21.15.83:0/4258125514 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0b80b0e30 0x7fd0b815ee40 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7fd0b400c3c0 tx=0x7fd0b400c880 comp rx=0 tx=0).stop 2024-03-06T07:28:32.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.811+0000 7fd0c0725640 1 -- 172.21.15.83:0/4258125514 shutdown_connections 2024-03-06T07:28:32.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.811+0000 7fd0c0725640 1 --2- 172.21.15.83:0/4258125514 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd09c06b500 0x7fd09c06d9c0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.811+0000 7fd0c0725640 1 --2- 172.21.15.83:0/4258125514 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd0b81614d0 0x7fd0b80cce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.811+0000 7fd0c0725640 1 --2- 172.21.15.83:0/4258125514 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0b80d4860 0x7fd0b80c89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.811+0000 7fd0c0725640 1 --2- 172.21.15.83:0/4258125514 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0b80b0e30 0x7fd0b815ee40 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.811+0000 7fd0c0725640 1 -- 172.21.15.83:0/4258125514 >> 172.21.15.83:0/4258125514 conn(0x7fd0b80d3d40 msgr2=0x7fd0b80d0840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:32.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.811+0000 7fd0c0725640 1 -- 172.21.15.83:0/4258125514 shutdown_connections 2024-03-06T07:28:32.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.811+0000 7fd0c0725640 1 -- 172.21.15.83:0/4258125514 wait complete. 2024-03-06T07:28:32.827 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:32.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.928+0000 7fa99741c640 1 Processor -- start 2024-03-06T07:28:32.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.929+0000 7fa99741c640 1 -- start start 2024-03-06T07:28:32.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.929+0000 7fa99741c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa9901614d0 0x7fa9900d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.929+0000 7fa99741c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa9900d1910 0x7fa9900d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.929+0000 7fa99741c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9900d2740 con 0x7fa9900d22b0 2024-03-06T07:28:32.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.929+0000 7fa99741c640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9901618b0 con 0x7fa9900d1910 2024-03-06T07:28:32.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.929+0000 7fa99741c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9900cf400 con 0x7fa9901614d0 2024-03-06T07:28:32.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.929+0000 7fa9949b7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa9900d1910 0x7fa9900d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.929+0000 7fa9949b7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa9900d1910 0x7fa9900d1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52510/0 (socket says 172.21.15.83:52510) 2024-03-06T07:28:32.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.930+0000 7fa9949b7640 1 -- 172.21.15.83:0/4057322281 learned_addr learned my addr 172.21.15.83:0/4057322281 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:32.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.930+0000 7fa9951b8640 1 --2- 172.21.15.83:0/4057322281 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa9901614d0 0x7fa9900d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.930+0000 7fa9949b7640 1 -- 172.21.15.83:0/4057322281 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa9901614d0 msgr2=0x7fa9900d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:32.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.930+0000 7fa9949b7640 1 --2- 172.21.15.83:0/4057322281 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa9901614d0 0x7fa9900d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.930+0000 7fa9949b7640 1 -- 172.21.15.83:0/4057322281 >> v1:172.21.15.183:6789/0 conn(0x7fa9900d22b0 legacy=0x7fa9900d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:32.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.930+0000 7fa9949b7640 1 -- 172.21.15.83:0/4057322281 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9900cf6a0 con 0x7fa9900d1910 2024-03-06T07:28:32.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.930+0000 7fa9951b8640 1 --2- 172.21.15.83:0/4057322281 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa9901614d0 0x7fa9900d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:32.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.930+0000 7fa9949b7640 1 --2- 172.21.15.83:0/4057322281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9900d1910 0x7fa9900d1d70 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7fa984009870 tx=0x7fa98402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=418ba460e77c2f39 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.931+0000 7fa98ffff640 1 -- 172.21.15.83:0/4057322281 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa984035020 con 0x7fa9900d1910 2024-03-06T07:28:32.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.931+0000 7fa98ffff640 1 -- 172.21.15.83:0/4057322281 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa984031640 con 0x7fa9900d1910 2024-03-06T07:28:32.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.931+0000 7fa98ffff640 1 -- 172.21.15.83:0/4057322281 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98402d040 con 0x7fa9900d1910 2024-03-06T07:28:32.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.931+0000 7fa99741c640 1 -- 172.21.15.83:0/4057322281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9900d1910 msgr2=0x7fa9900d1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.931+0000 7fa99741c640 1 --2- 172.21.15.83:0/4057322281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9900d1910 0x7fa9900d1d70 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7fa984009870 tx=0x7fa98402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.931+0000 7fa99741c640 1 -- 172.21.15.83:0/4057322281 shutdown_connections 2024-03-06T07:28:32.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.931+0000 7fa99741c640 1 --2- 172.21.15.83:0/4057322281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9900d1910 0x7fa9900d1d70 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.931+0000 7fa99741c640 1 --2- 172.21.15.83:0/4057322281 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa9901614d0 0x7fa9900d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.931+0000 7fa99741c640 1 -- 172.21.15.83:0/4057322281 >> 172.21.15.83:0/4057322281 conn(0x7fa9900d3d40 msgr2=0x7fa9900d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:32.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.931+0000 7fa99741c640 1 -- 172.21.15.83:0/4057322281 shutdown_connections 2024-03-06T07:28:32.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.931+0000 7fa99741c640 1 -- 172.21.15.83:0/4057322281 wait complete. 2024-03-06T07:28:32.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.932+0000 7fa99741c640 1 Processor -- start 2024-03-06T07:28:32.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.932+0000 7fa99741c640 1 -- start start 2024-03-06T07:28:32.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.933+0000 7fa99741c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9900d1910 0x7fa99015c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.933+0000 7fa99741c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa9900d22b0 0x7fa9900cd670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.933+0000 7fa99741c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa9901614d0 0x7fa99015b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.933+0000 7fa99741c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9900b0910 con 0x7fa9900d1910 2024-03-06T07:28:32.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.933+0000 7fa99741c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9900cfc60 con 0x7fa9900d22b0 2024-03-06T07:28:32.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.933+0000 7fa99741c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9900b1e40 con 0x7fa9901614d0 2024-03-06T07:28:32.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.933+0000 7fa9951b8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9900d1910 0x7fa99015c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.933+0000 7fa9951b8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9900d1910 0x7fa99015c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52518/0 (socket says 172.21.15.83:52518) 2024-03-06T07:28:32.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.933+0000 7fa9951b8640 1 -- 172.21.15.83:0/776682253 learned_addr learned my addr 172.21.15.83:0/776682253 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:32.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.933+0000 7fa9949b7640 1 --2- 172.21.15.83:0/776682253 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa9900d22b0 0x7fa9900cd670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.933+0000 7fa9951b8640 1 -- 172.21.15.83:0/776682253 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa9901614d0 msgr2=0x7fa99015b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.933+0000 7fa9959b9640 1 --2- 172.21.15.83:0/776682253 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa9901614d0 0x7fa99015b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.934+0000 7fa9951b8640 1 --2- 172.21.15.83:0/776682253 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa9901614d0 0x7fa99015b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.934+0000 7fa9951b8640 1 -- 172.21.15.83:0/776682253 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa9900d22b0 msgr2=0x7fa9900cd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.934+0000 7fa9951b8640 1 --2- 172.21.15.83:0/776682253 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa9900d22b0 0x7fa9900cd670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.934+0000 7fa9951b8640 1 -- 172.21.15.83:0/776682253 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9900b8810 con 0x7fa9900d1910 2024-03-06T07:28:32.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.934+0000 7fa9949b7640 1 --2- 172.21.15.83:0/776682253 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa9900d22b0 0x7fa9900cd670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:32.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.934+0000 7fa9951b8640 1 --2- 172.21.15.83:0/776682253 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9900d1910 0x7fa99015c6a0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7fa98000c7f0 tx=0x7fa98000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.934+0000 7fa98dffb640 1 -- 172.21.15.83:0/776682253 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa980015070 con 0x7fa9900d1910 2024-03-06T07:28:32.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.934+0000 7fa98dffb640 1 -- 172.21.15.83:0/776682253 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa980007820 con 0x7fa9900d1910 2024-03-06T07:28:32.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.934+0000 7fa99741c640 1 -- 172.21.15.83:0/776682253 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa99015ae40 con 0x7fa9900d1910 2024-03-06T07:28:32.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.934+0000 7fa98dffb640 1 -- 172.21.15.83:0/776682253 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa980007bd0 con 0x7fa9900d1910 2024-03-06T07:28:32.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.934+0000 7fa99741c640 1 -- 172.21.15.83:0/776682253 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa99015b400 con 0x7fa9900d1910 2024-03-06T07:28:32.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.935+0000 7fa99741c640 1 -- 172.21.15.83:0/776682253 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9900caba0 con 0x7fa9900d1910 2024-03-06T07:28:32.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.935+0000 7fa98dffb640 1 -- 172.21.15.83:0/776682253 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa980018070 con 0x7fa9900d1910 2024-03-06T07:28:32.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.936+0000 7fa98dffb640 1 --2- 172.21.15.83:0/776682253 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa97c06b4d0 0x7fa97c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.936+0000 7fa98dffb640 1 -- 172.21.15.83:0/776682253 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fa98008a460 con 0x7fa9900d1910 2024-03-06T07:28:32.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.936+0000 7fa9949b7640 1 --2- 172.21.15.83:0/776682253 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa97c06b4d0 0x7fa97c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.936+0000 7fa9949b7640 1 --2- 172.21.15.83:0/776682253 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa97c06b4d0 0x7fa97c06d990 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fa984036000 tx=0x7fa984008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:32.940+0000 7fa98dffb640 1 -- 172.21.15.83:0/776682253 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa9800591f0 con 0x7fa9900d1910 2024-03-06T07:28:33.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.259+0000 7fa99741c640 1 -- 172.21.15.83:0/776682253 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa990068ff0 con 0x7fa9900d1910 2024-03-06T07:28:33.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.260+0000 7fa98dffb640 1 -- 172.21.15.83:0/776682253 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v116) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa980058af0 con 0x7fa9900d1910 2024-03-06T07:28:33.262 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-03-06T07:28:33.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.262+0000 7fa99741c640 1 -- 172.21.15.83:0/776682253 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa97c06b4d0 msgr2=0x7fa97c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.263+0000 7fa99741c640 1 --2- 172.21.15.83:0/776682253 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa97c06b4d0 0x7fa97c06d990 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fa984036000 tx=0x7fa984008030 comp rx=0 tx=0).stop 2024-03-06T07:28:33.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.263+0000 7fa99741c640 1 -- 172.21.15.83:0/776682253 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9900d1910 msgr2=0x7fa99015c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.263+0000 7fa99741c640 1 --2- 172.21.15.83:0/776682253 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9900d1910 0x7fa99015c6a0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7fa98000c7f0 tx=0x7fa98000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.263+0000 7fa99741c640 1 -- 172.21.15.83:0/776682253 shutdown_connections 2024-03-06T07:28:33.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.263+0000 7fa99741c640 1 --2- 172.21.15.83:0/776682253 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa97c06b4d0 0x7fa97c06d990 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.263+0000 7fa99741c640 1 --2- 172.21.15.83:0/776682253 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa9901614d0 0x7fa99015b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.263+0000 7fa99741c640 1 --2- 172.21.15.83:0/776682253 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa9900d22b0 0x7fa9900cd670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.263+0000 7fa99741c640 1 --2- 172.21.15.83:0/776682253 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9900d1910 0x7fa99015c6a0 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.263+0000 7fa99741c640 1 -- 172.21.15.83:0/776682253 >> 172.21.15.83:0/776682253 conn(0x7fa9900d3d40 msgr2=0x7fa99015fad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:33.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.263+0000 7fa99741c640 1 -- 172.21.15.83:0/776682253 shutdown_connections 2024-03-06T07:28:33.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.263+0000 7fa99741c640 1 -- 172.21.15.83:0/776682253 wait complete. 2024-03-06T07:28:33.275 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:33.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.377+0000 7fa91f3b6640 1 Processor -- start 2024-03-06T07:28:33.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.378+0000 7fa91f3b6640 1 -- start start 2024-03-06T07:28:33.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.379+0000 7fa91f3b6640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa9180d4860 0x7fa9180d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.379+0000 7fa91f3b6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9181618b0 con 0x7fa9181614d0 2024-03-06T07:28:33.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.379+0000 7fa91f3b6640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9180cf400 con 0x7fa9180b0e30 2024-03-06T07:28:33.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.379+0000 7fa91f3b6640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9180d4cf0 con 0x7fa9180d4860 2024-03-06T07:28:33.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.379+0000 7fa91c951640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa9180d4860 0x7fa9180d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.379+0000 7fa91d152640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fa9181614d0 0x7fa9180d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49118/0 (socket says 172.21.15.83:49118) 2024-03-06T07:28:33.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.379+0000 7fa91d152640 1 -- 172.21.15.83:0/904582104 learned_addr learned my addr 172.21.15.83:0/904582104 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:28:33.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.379+0000 7fa91c951640 1 -- 172.21.15.83:0/904582104 >> v1:172.21.15.183:6789/0 conn(0x7fa9180b0e30 legacy=0x7fa9180b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.379+0000 7fa91c951640 1 -- 172.21.15.83:0/904582104 >> v1:172.21.15.83:6789/0 conn(0x7fa9181614d0 legacy=0x7fa9180d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.379+0000 7fa91c951640 1 -- 172.21.15.83:0/904582104 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9180cf6a0 con 0x7fa9180d4860 2024-03-06T07:28:33.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.379+0000 7fa90bfff640 1 -- 172.21.15.83:0/904582104 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4168336727 0 0) 0x7fa9181618b0 con 0x7fa9181614d0 2024-03-06T07:28:33.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.380+0000 7fa91c951640 1 --2- 172.21.15.83:0/904582104 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9180d4860 0x7fa9180d2020 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7fa90c0097a0 tx=0x7fa90c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c38ee000a3f8d12 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.380+0000 7fa90bfff640 1 -- 172.21.15.83:0/904582104 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa90c034070 con 0x7fa9180d4860 2024-03-06T07:28:33.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.380+0000 7fa90bfff640 1 -- 172.21.15.83:0/904582104 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa90c030610 con 0x7fa9180d4860 2024-03-06T07:28:33.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.380+0000 7fa90bfff640 1 -- 172.21.15.83:0/904582104 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa90c0309e0 con 0x7fa9180d4860 2024-03-06T07:28:33.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.381+0000 7fa91f3b6640 1 -- 172.21.15.83:0/904582104 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9180d4860 msgr2=0x7fa9180d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.381+0000 7fa91f3b6640 1 --2- 172.21.15.83:0/904582104 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9180d4860 0x7fa9180d2020 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7fa90c0097a0 tx=0x7fa90c02b960 comp rx=0 tx=0).stop 2024-03-06T07:28:33.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.381+0000 7fa91f3b6640 1 -- 172.21.15.83:0/904582104 shutdown_connections 2024-03-06T07:28:33.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.381+0000 7fa91f3b6640 1 --2- 172.21.15.83:0/904582104 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9180d4860 0x7fa9180d2020 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.381+0000 7fa91f3b6640 1 -- 172.21.15.83:0/904582104 >> 172.21.15.83:0/904582104 conn(0x7fa9180d3d40 msgr2=0x7fa9180d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:33.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.381+0000 7fa91f3b6640 1 -- 172.21.15.83:0/904582104 shutdown_connections 2024-03-06T07:28:33.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.381+0000 7fa91f3b6640 1 -- 172.21.15.83:0/904582104 wait complete. 2024-03-06T07:28:33.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.382+0000 7fa91f3b6640 1 Processor -- start 2024-03-06T07:28:33.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.382+0000 7fa91f3b6640 1 -- start start 2024-03-06T07:28:33.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.383+0000 7fa91f3b6640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa9180b0e30 0x7fa9180cd410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.383+0000 7fa91f3b6640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa9180d4860 0x7fa9180b6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.383+0000 7fa91f3b6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9181614d0 0x7fa91815e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.383+0000 7fa91f3b6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9180b0910 con 0x7fa9181614d0 2024-03-06T07:28:33.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.383+0000 7fa91f3b6640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9180cfc60 con 0x7fa9180b0e30 2024-03-06T07:28:33.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.383+0000 7fa91f3b6640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9180d2560 con 0x7fa9180d4860 2024-03-06T07:28:33.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.383+0000 7fa91d953640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9181614d0 0x7fa91815e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.383+0000 7fa91d953640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9181614d0 0x7fa91815e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52536/0 (socket says 172.21.15.83:52536) 2024-03-06T07:28:33.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.383+0000 7fa91d953640 1 -- 172.21.15.83:0/1264283400 learned_addr learned my addr 172.21.15.83:0/1264283400 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:33.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.383+0000 7fa91c951640 1 --2- 172.21.15.83:0/1264283400 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa9180d4860 0x7fa9180b6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.383+0000 7fa91d152640 1 --2- 172.21.15.83:0/1264283400 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa9180b0e30 0x7fa9180cd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.383+0000 7fa91d953640 1 -- 172.21.15.83:0/1264283400 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa9180d4860 msgr2=0x7fa9180b6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.383+0000 7fa91d953640 1 --2- 172.21.15.83:0/1264283400 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa9180d4860 0x7fa9180b6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.383+0000 7fa91d953640 1 -- 172.21.15.83:0/1264283400 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa9180b0e30 msgr2=0x7fa9180cd410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.383+0000 7fa91d953640 1 --2- 172.21.15.83:0/1264283400 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa9180b0e30 0x7fa9180cd410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.383+0000 7fa91d953640 1 -- 172.21.15.83:0/1264283400 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9180cdc60 con 0x7fa9181614d0 2024-03-06T07:28:33.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.383+0000 7fa91c951640 1 --2- 172.21.15.83:0/1264283400 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa9180d4860 0x7fa9180b6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:33.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.384+0000 7fa91d953640 1 --2- 172.21.15.83:0/1264283400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9181614d0 0x7fa91815e310 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fa91000c7a0 tx=0x7fa91000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.384+0000 7fa909ffb640 1 -- 172.21.15.83:0/1264283400 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa910003b70 con 0x7fa9181614d0 2024-03-06T07:28:33.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.384+0000 7fa909ffb640 1 -- 172.21.15.83:0/1264283400 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa910003d30 con 0x7fa9181614d0 2024-03-06T07:28:33.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.384+0000 7fa909ffb640 1 -- 172.21.15.83:0/1264283400 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91000aa90 con 0x7fa9181614d0 2024-03-06T07:28:33.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.384+0000 7fa91f3b6640 1 -- 172.21.15.83:0/1264283400 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa91815d550 con 0x7fa9181614d0 2024-03-06T07:28:33.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.384+0000 7fa91f3b6640 1 -- 172.21.15.83:0/1264283400 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa91815db10 con 0x7fa9181614d0 2024-03-06T07:28:33.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.385+0000 7fa91f3b6640 1 -- 172.21.15.83:0/1264283400 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa918068ff0 con 0x7fa9181614d0 2024-03-06T07:28:33.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.385+0000 7fa909ffb640 1 -- 172.21.15.83:0/1264283400 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa91000ac30 con 0x7fa9181614d0 2024-03-06T07:28:33.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.386+0000 7fa909ffb640 1 --2- 172.21.15.83:0/1264283400 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa8fc06b4d0 0x7fa8fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.386+0000 7fa909ffb640 1 -- 172.21.15.83:0/1264283400 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fa91008a110 con 0x7fa9181614d0 2024-03-06T07:28:33.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.386+0000 7fa91d152640 1 --2- 172.21.15.83:0/1264283400 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa8fc06b4d0 0x7fa8fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.386+0000 7fa91d152640 1 --2- 172.21.15.83:0/1264283400 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa8fc06b4d0 0x7fa8fc06d990 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fa914007c30 tx=0x7fa914007b80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.390+0000 7fa909ffb640 1 -- 172.21.15.83:0/1264283400 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa910058f30 con 0x7fa9181614d0 2024-03-06T07:28:33.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.567+0000 7fa91f3b6640 1 -- 172.21.15.83:0/1264283400 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa9180a1c10 con 0x7fa9181614d0 2024-03-06T07:28:33.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.567+0000 7fa909ffb640 1 -- 172.21.15.83:0/1264283400 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v116) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa910058830 con 0x7fa9181614d0 2024-03-06T07:28:33.569 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 64 2024-03-06T07:28:33.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.569+0000 7fa91f3b6640 1 -- 172.21.15.83:0/1264283400 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa8fc06b4d0 msgr2=0x7fa8fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.569+0000 7fa91f3b6640 1 --2- 172.21.15.83:0/1264283400 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa8fc06b4d0 0x7fa8fc06d990 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fa914007c30 tx=0x7fa914007b80 comp rx=0 tx=0).stop 2024-03-06T07:28:33.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.569+0000 7fa91f3b6640 1 -- 172.21.15.83:0/1264283400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9181614d0 msgr2=0x7fa91815e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.569+0000 7fa91f3b6640 1 --2- 172.21.15.83:0/1264283400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9181614d0 0x7fa91815e310 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fa91000c7a0 tx=0x7fa91000cc60 comp rx=0 tx=0).stop 2024-03-06T07:28:33.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.569+0000 7fa91f3b6640 1 -- 172.21.15.83:0/1264283400 shutdown_connections 2024-03-06T07:28:33.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.569+0000 7fa91f3b6640 1 --2- 172.21.15.83:0/1264283400 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa8fc06b4d0 0x7fa8fc06d990 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.569+0000 7fa91f3b6640 1 --2- 172.21.15.83:0/1264283400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9181614d0 0x7fa91815e310 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.569+0000 7fa91f3b6640 1 --2- 172.21.15.83:0/1264283400 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa9180d4860 0x7fa9180b6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.569+0000 7fa91f3b6640 1 --2- 172.21.15.83:0/1264283400 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa9180b0e30 0x7fa9180cd410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.569+0000 7fa91f3b6640 1 -- 172.21.15.83:0/1264283400 >> 172.21.15.83:0/1264283400 conn(0x7fa9180d3d40 msgr2=0x7fa9180d0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:33.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.569+0000 7fa91f3b6640 1 -- 172.21.15.83:0/1264283400 shutdown_connections 2024-03-06T07:28:33.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.569+0000 7fa91f3b6640 1 -- 172.21.15.83:0/1264283400 wait complete. 2024-03-06T07:28:33.582 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:33.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.682+0000 7f6c3978a640 1 Processor -- start 2024-03-06T07:28:33.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.682+0000 7f6c3978a640 1 -- start start 2024-03-06T07:28:33.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.682+0000 7f6c3978a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6c340d2710 0x7f6c340b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.682+0000 7f6c3978a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c340b3430 con 0x7f6c340d22b0 2024-03-06T07:28:33.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.682+0000 7f6c3978a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c340b3710 con 0x7f6c3415dfe0 2024-03-06T07:28:33.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.682+0000 7f6c3978a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c340b31e0 con 0x7f6c340d2710 2024-03-06T07:28:33.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.682+0000 7f6c2bfff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f6c340d22b0 0x7f6c340b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49134/0 (socket says 172.21.15.83:49134) 2024-03-06T07:28:33.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.682+0000 7f6c2bfff640 1 -- 172.21.15.83:0/3782802308 learned_addr learned my addr 172.21.15.83:0/3782802308 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:28:33.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.683+0000 7f6c337fe640 1 --2- 172.21.15.83:0/3782802308 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6c340d2710 0x7f6c340b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.683+0000 7f6c327fc640 1 -- 172.21.15.83:0/3782802308 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3323946448 0 0) 0x7f6c340b3430 con 0x7f6c340d22b0 2024-03-06T07:28:33.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.683+0000 7f6c327fc640 1 -- 172.21.15.83:0/3782802308 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c20003520 con 0x7f6c340d22b0 2024-03-06T07:28:33.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.683+0000 7f6c327fc640 1 -- 172.21.15.83:0/3782802308 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 568083550 0 0) 0x7f6c340b3710 con 0x7f6c3415dfe0 2024-03-06T07:28:33.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.683+0000 7f6c337fe640 1 -- 172.21.15.83:0/3782802308 >> v1:172.21.15.183:6789/0 conn(0x7f6c3415dfe0 legacy=0x7f6c340cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.683+0000 7f6c337fe640 1 -- 172.21.15.83:0/3782802308 >> v1:172.21.15.83:6789/0 conn(0x7f6c340d22b0 legacy=0x7f6c340b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.683+0000 7f6c337fe640 1 -- 172.21.15.83:0/3782802308 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c340b39f0 con 0x7f6c340d2710 2024-03-06T07:28:33.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.684+0000 7f6c337fe640 1 --2- 172.21.15.83:0/3782802308 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6c340d2710 0x7f6c340b4900 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6c340b3360 tx=0x7f6c2c02bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ee1a984e6fa70c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.684+0000 7f6c327fc640 1 -- 172.21.15.83:0/3782802308 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c2c039070 con 0x7f6c340d2710 2024-03-06T07:28:33.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.684+0000 7f6c327fc640 1 -- 172.21.15.83:0/3782802308 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c2c02ccc0 con 0x7f6c340d2710 2024-03-06T07:28:33.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.684+0000 7f6c327fc640 1 -- 172.21.15.83:0/3782802308 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c2c034640 con 0x7f6c340d2710 2024-03-06T07:28:33.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.684+0000 7f6c3978a640 1 -- 172.21.15.83:0/3782802308 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6c340d2710 msgr2=0x7f6c340b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.684+0000 7f6c3978a640 1 --2- 172.21.15.83:0/3782802308 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6c340d2710 0x7f6c340b4900 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6c340b3360 tx=0x7f6c2c02bf10 comp rx=0 tx=0).stop 2024-03-06T07:28:33.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.685+0000 7f6c3978a640 1 -- 172.21.15.83:0/3782802308 shutdown_connections 2024-03-06T07:28:33.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.685+0000 7f6c3978a640 1 --2- 172.21.15.83:0/3782802308 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6c340d2710 0x7f6c340b4900 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.685+0000 7f6c3978a640 1 -- 172.21.15.83:0/3782802308 >> 172.21.15.83:0/3782802308 conn(0x7f6c340d3cd0 msgr2=0x7f6c340d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:33.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.685+0000 7f6c3978a640 1 -- 172.21.15.83:0/3782802308 shutdown_connections 2024-03-06T07:28:33.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.685+0000 7f6c3978a640 1 -- 172.21.15.83:0/3782802308 wait complete. 2024-03-06T07:28:33.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.686+0000 7f6c3978a640 1 Processor -- start 2024-03-06T07:28:33.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.686+0000 7f6c3978a640 1 -- start start 2024-03-06T07:28:33.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.686+0000 7f6c3978a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c340d22b0 0x7f6c34159b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.686+0000 7f6c3978a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c340d2710 0x7f6c340b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.686+0000 7f6c3978a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6c3415dfe0 0x7f6c340b8840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.686+0000 7f6c3978a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c340cfa00 con 0x7f6c340d2710 2024-03-06T07:28:33.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.686+0000 7f6c3978a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c340b3d60 con 0x7f6c340d22b0 2024-03-06T07:28:33.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.686+0000 7f6c3978a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c340d1c70 con 0x7f6c3415dfe0 2024-03-06T07:28:33.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.686+0000 7f6c2bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c340d2710 0x7f6c340b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.687+0000 7f6c2bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c340d2710 0x7f6c340b68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52548/0 (socket says 172.21.15.83:52548) 2024-03-06T07:28:33.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.687+0000 7f6c2bfff640 1 -- 172.21.15.83:0/2583043555 learned_addr learned my addr 172.21.15.83:0/2583043555 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:33.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.687+0000 7f6c337fe640 1 --2- 172.21.15.83:0/2583043555 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6c3415dfe0 0x7f6c340b8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.687+0000 7f6c32ffd640 1 --2- 172.21.15.83:0/2583043555 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c340d22b0 0x7f6c34159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.687+0000 7f6c2bfff640 1 -- 172.21.15.83:0/2583043555 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6c3415dfe0 msgr2=0x7f6c340b8840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.687+0000 7f6c2bfff640 1 --2- 172.21.15.83:0/2583043555 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6c3415dfe0 0x7f6c340b8840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.687+0000 7f6c2bfff640 1 -- 172.21.15.83:0/2583043555 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c340d22b0 msgr2=0x7f6c34159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.687+0000 7f6c2bfff640 1 --2- 172.21.15.83:0/2583043555 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c340d22b0 0x7f6c34159b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.687+0000 7f6c2bfff640 1 -- 172.21.15.83:0/2583043555 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c341f3530 con 0x7f6c340d2710 2024-03-06T07:28:33.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.687+0000 7f6c337fe640 1 --2- 172.21.15.83:0/2583043555 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6c3415dfe0 0x7f6c340b8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:33.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.687+0000 7f6c32ffd640 1 --2- 172.21.15.83:0/2583043555 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c340d22b0 0x7f6c34159b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:33.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.687+0000 7f6c2bfff640 1 --2- 172.21.15.83:0/2583043555 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c340d2710 0x7f6c340b68d0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c00c460 tx=0x7f6c1c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.687+0000 7f6c2b7fe640 1 -- 172.21.15.83:0/2583043555 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c1c017070 con 0x7f6c340d2710 2024-03-06T07:28:33.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.687+0000 7f6c2b7fe640 1 -- 172.21.15.83:0/2583043555 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c1c005ce0 con 0x7f6c340d2710 2024-03-06T07:28:33.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.687+0000 7f6c3978a640 1 -- 172.21.15.83:0/2583043555 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c340b7390 con 0x7f6c340d2710 2024-03-06T07:28:33.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.687+0000 7f6c2b7fe640 1 -- 172.21.15.83:0/2583043555 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c1c0072f0 con 0x7f6c340d2710 2024-03-06T07:28:33.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.687+0000 7f6c3978a640 1 -- 172.21.15.83:0/2583043555 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c340caed0 con 0x7f6c340d2710 2024-03-06T07:28:33.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.688+0000 7f6c3978a640 1 -- 172.21.15.83:0/2583043555 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c3404f330 con 0x7f6c340d2710 2024-03-06T07:28:33.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.689+0000 7f6c2b7fe640 1 -- 172.21.15.83:0/2583043555 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6c1c01a070 con 0x7f6c340d2710 2024-03-06T07:28:33.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.689+0000 7f6c2b7fe640 1 --2- 172.21.15.83:0/2583043555 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6c1406b4d0 0x7f6c1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.689+0000 7f6c2b7fe640 1 -- 172.21.15.83:0/2583043555 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f6c1c0896c0 con 0x7f6c340d2710 2024-03-06T07:28:33.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.689+0000 7f6c32ffd640 1 --2- 172.21.15.83:0/2583043555 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6c1406b4d0 0x7f6c1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.689+0000 7f6c32ffd640 1 --2- 172.21.15.83:0/2583043555 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6c1406b4d0 0x7f6c1406d990 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f6c24000f60 tx=0x7f6c24009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.693+0000 7f6c2b7fe640 1 -- 172.21.15.83:0/2583043555 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6c341f3530 con 0x7f6c340d2710 2024-03-06T07:28:33.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.908+0000 7f6c3978a640 1 -- 172.21.15.83:0/2583043555 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c340b5260 con 0x7f6c340d2710 2024-03-06T07:28:33.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.908+0000 7f6c2b7fe640 1 -- 172.21.15.83:0/2583043555 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f6c1c0584c0 con 0x7f6c340d2710 2024-03-06T07:28:33.910 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:28:33.910 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7051,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/101642171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":101642171},{"type":"v1","addr":"172.21.15.183:6833","nonce":101642171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7057,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3146884900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":3146884900},{"type":"v1","addr":"172.21.15.183:6835","nonce":3146884900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8208,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1922041621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1922041621},{"type":"v1","addr":"172.21.15.83:6835","nonce":1922041621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.491022+0000","modified":"2024-03-06T07:27:37.006649+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8229":{"gid":8229,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/2003687153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2003687153},{"type":"v1","addr":"172.21.15.83:6837","nonce":2003687153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8229,"qdb_cluster":[8229]},"id":8}]} 2024-03-06T07:28:33.911 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 66 2024-03-06T07:28:33.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.910+0000 7f6c3978a640 1 -- 172.21.15.83:0/2583043555 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6c1406b4d0 msgr2=0x7f6c1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.910+0000 7f6c3978a640 1 --2- 172.21.15.83:0/2583043555 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6c1406b4d0 0x7f6c1406d990 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f6c24000f60 tx=0x7f6c24009040 comp rx=0 tx=0).stop 2024-03-06T07:28:33.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.910+0000 7f6c3978a640 1 -- 172.21.15.83:0/2583043555 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c340d2710 msgr2=0x7f6c340b68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.910+0000 7f6c3978a640 1 --2- 172.21.15.83:0/2583043555 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c340d2710 0x7f6c340b68d0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c00c460 tx=0x7f6c1c00c920 comp rx=0 tx=0).stop 2024-03-06T07:28:33.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.910+0000 7f6c3978a640 1 -- 172.21.15.83:0/2583043555 shutdown_connections 2024-03-06T07:28:33.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.910+0000 7f6c3978a640 1 --2- 172.21.15.83:0/2583043555 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6c1406b4d0 0x7f6c1406d990 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.910+0000 7f6c3978a640 1 --2- 172.21.15.83:0/2583043555 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6c3415dfe0 0x7f6c340b8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.910+0000 7f6c3978a640 1 --2- 172.21.15.83:0/2583043555 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c340d2710 0x7f6c340b68d0 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.910+0000 7f6c3978a640 1 --2- 172.21.15.83:0/2583043555 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c340d22b0 0x7f6c34159b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.910+0000 7f6c3978a640 1 -- 172.21.15.83:0/2583043555 >> 172.21.15.83:0/2583043555 conn(0x7f6c340d3cd0 msgr2=0x7f6c340d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:33.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.910+0000 7f6c3978a640 1 -- 172.21.15.83:0/2583043555 shutdown_connections 2024-03-06T07:28:33.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:33.910+0000 7f6c3978a640 1 -- 172.21.15.83:0/2583043555 wait complete. 2024-03-06T07:28:33.923 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:34.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.022+0000 7f1095d2b640 1 Processor -- start 2024-03-06T07:28:34.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.022+0000 7f1095d2b640 1 -- start start 2024-03-06T07:28:34.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.023+0000 7f1095d2b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f109015bb20 0x7f10900d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.023+0000 7f1095d2b640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f10900cfff0 0x7f10900d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.023+0000 7f1095d2b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1090161980 con 0x7f10900ce4b0 2024-03-06T07:28:34.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.023+0000 7f1095d2b640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1090161c60 con 0x7f109015bb20 2024-03-06T07:28:34.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.023+0000 7f1095d2b640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1090161f40 con 0x7f10900cfff0 2024-03-06T07:28:34.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.023+0000 7f108f7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f109015bb20 0x7f10900d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.023+0000 7f108f7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f109015bb20 0x7f10900d1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52564/0 (socket says 172.21.15.83:52564) 2024-03-06T07:28:34.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.023+0000 7f108ffff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f10900ce4b0 0x7f10900ce8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49144/0 (socket says 172.21.15.83:49144) 2024-03-06T07:28:34.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.023+0000 7f108f7fe640 1 -- 172.21.15.83:0/3322720894 learned_addr learned my addr 172.21.15.83:0/3322720894 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:34.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.024+0000 7f1086ffd640 1 --2- 172.21.15.83:0/3322720894 >> v2:172.21.15.183:3300/0 conn(0x7f10900cfff0 0x7f10900d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.024+0000 7f108f7fe640 1 -- 172.21.15.83:0/3322720894 >> v2:172.21.15.183:3300/0 conn(0x7f10900cfff0 msgr2=0x7f10900d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:34.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.024+0000 7f108f7fe640 1 --2- 172.21.15.83:0/3322720894 >> v2:172.21.15.183:3300/0 conn(0x7f10900cfff0 0x7f10900d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.024+0000 7f108f7fe640 1 -- 172.21.15.83:0/3322720894 >> v1:172.21.15.83:6789/0 conn(0x7f10900ce4b0 legacy=0x7f10900ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.024+0000 7f108f7fe640 1 -- 172.21.15.83:0/3322720894 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1090162220 con 0x7f109015bb20 2024-03-06T07:28:34.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.024+0000 7f108effd640 1 -- 172.21.15.83:0/3322720894 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 591754902 0 0) 0x7f1090161980 con 0x7f10900ce4b0 2024-03-06T07:28:34.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.024+0000 7f108f7fe640 1 --2- 172.21.15.83:0/3322720894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f109015bb20 0x7f10900d1df0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f1090071290 tx=0x7f108002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e4266ced9cf5e2e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.025+0000 7f108effd640 1 -- 172.21.15.83:0/3322720894 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1080034070 con 0x7f109015bb20 2024-03-06T07:28:34.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.025+0000 7f108effd640 1 -- 172.21.15.83:0/3322720894 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1080030860 con 0x7f109015bb20 2024-03-06T07:28:34.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.025+0000 7f108effd640 1 -- 172.21.15.83:0/3322720894 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1080030c30 con 0x7f109015bb20 2024-03-06T07:28:34.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.025+0000 7f1095d2b640 1 -- 172.21.15.83:0/3322720894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f109015bb20 msgr2=0x7f10900d1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.025+0000 7f1095d2b640 1 --2- 172.21.15.83:0/3322720894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f109015bb20 0x7f10900d1df0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f1090071290 tx=0x7f108002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.025+0000 7f1095d2b640 1 -- 172.21.15.83:0/3322720894 shutdown_connections 2024-03-06T07:28:34.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.025+0000 7f1095d2b640 1 --2- 172.21.15.83:0/3322720894 >> v2:172.21.15.183:3300/0 conn(0x7f10900cfff0 0x7f10900d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.025+0000 7f1095d2b640 1 --2- 172.21.15.83:0/3322720894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f109015bb20 0x7f10900d1df0 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.025+0000 7f1095d2b640 1 -- 172.21.15.83:0/3322720894 >> 172.21.15.83:0/3322720894 conn(0x7f109009a9f0 msgr2=0x7f109009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:34.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.025+0000 7f1095d2b640 1 -- 172.21.15.83:0/3322720894 shutdown_connections 2024-03-06T07:28:34.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.025+0000 7f1095d2b640 1 -- 172.21.15.83:0/3322720894 wait complete. 2024-03-06T07:28:34.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.027+0000 7f1095d2b640 1 Processor -- start 2024-03-06T07:28:34.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.027+0000 7f1095d2b640 1 -- start start 2024-03-06T07:28:34.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.028+0000 7f1095d2b640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10900ce4b0 0x7f10900c56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.028+0000 7f1095d2b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10900cfff0 0x7f1090150ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.028+0000 7f1095d2b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f109015bb20 0x7f109006ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.028+0000 7f1095d2b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10900d24b0 con 0x7f10900cfff0 2024-03-06T07:28:34.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.028+0000 7f1095d2b640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10900d2330 con 0x7f10900ce4b0 2024-03-06T07:28:34.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.028+0000 7f1095d2b640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10900d1980 con 0x7f109015bb20 2024-03-06T07:28:34.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.029+0000 7f1086ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10900cfff0 0x7f1090150ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.029+0000 7f1086ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10900cfff0 0x7f1090150ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52580/0 (socket says 172.21.15.83:52580) 2024-03-06T07:28:34.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.029+0000 7f1086ffd640 1 -- 172.21.15.83:0/828502623 learned_addr learned my addr 172.21.15.83:0/828502623 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:34.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.029+0000 7f108f7fe640 1 --2- 172.21.15.83:0/828502623 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10900ce4b0 0x7f10900c56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.029+0000 7f108ffff640 1 --2- 172.21.15.83:0/828502623 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f109015bb20 0x7f109006ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.029+0000 7f1086ffd640 1 -- 172.21.15.83:0/828502623 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f109015bb20 msgr2=0x7f109006ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.029+0000 7f1086ffd640 1 --2- 172.21.15.83:0/828502623 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f109015bb20 0x7f109006ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.029+0000 7f1086ffd640 1 -- 172.21.15.83:0/828502623 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10900ce4b0 msgr2=0x7f10900c56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.029+0000 7f1086ffd640 1 --2- 172.21.15.83:0/828502623 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10900ce4b0 0x7f10900c56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.029+0000 7f1086ffd640 1 -- 172.21.15.83:0/828502623 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10900ca600 con 0x7f10900cfff0 2024-03-06T07:28:34.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.029+0000 7f108ffff640 1 --2- 172.21.15.83:0/828502623 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f109015bb20 0x7f109006ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:34.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.029+0000 7f1086ffd640 1 --2- 172.21.15.83:0/828502623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10900cfff0 0x7f1090150ec0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f107800e9e0 tx=0x7f107800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.029+0000 7f108cff9640 1 -- 172.21.15.83:0/828502623 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1078017070 con 0x7f10900cfff0 2024-03-06T07:28:34.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.029+0000 7f108cff9640 1 -- 172.21.15.83:0/828502623 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f107800c970 con 0x7f10900cfff0 2024-03-06T07:28:34.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.029+0000 7f1095d2b640 1 -- 172.21.15.83:0/828502623 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10900ca8e0 con 0x7f10900cfff0 2024-03-06T07:28:34.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.029+0000 7f108cff9640 1 -- 172.21.15.83:0/828502623 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f107800cd60 con 0x7f10900cfff0 2024-03-06T07:28:34.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.029+0000 7f1095d2b640 1 -- 172.21.15.83:0/828502623 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10900caea0 con 0x7f10900cfff0 2024-03-06T07:28:34.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.030+0000 7f1095d2b640 1 -- 172.21.15.83:0/828502623 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f109004f330 con 0x7f10900cfff0 2024-03-06T07:28:34.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.031+0000 7f108cff9640 1 -- 172.21.15.83:0/828502623 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f107801a070 con 0x7f10900cfff0 2024-03-06T07:28:34.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.031+0000 7f108cff9640 1 --2- 172.21.15.83:0/828502623 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f107006b4d0 0x7f107006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.031+0000 7f108cff9640 1 -- 172.21.15.83:0/828502623 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f107808a5b0 con 0x7f10900cfff0 2024-03-06T07:28:34.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.031+0000 7f108f7fe640 1 --2- 172.21.15.83:0/828502623 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f107006b4d0 0x7f107006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.032+0000 7f108f7fe640 1 --2- 172.21.15.83:0/828502623 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f107006b4d0 0x7f107006d990 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f1080036000 tx=0x7f108003b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.035+0000 7f108cff9640 1 -- 172.21.15.83:0/828502623 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1078058cf0 con 0x7f10900cfff0 2024-03-06T07:28:34.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.216+0000 7f1095d2b640 1 -- 172.21.15.83:0/828502623 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f10900b1820 con 0x7f10900cfff0 2024-03-06T07:28:34.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.217+0000 7f108cff9640 1 -- 172.21.15.83:0/828502623 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+16992 (secure 0 0 0) 0x7f107805cd90 con 0x7f10900cfff0 2024-03-06T07:28:34.219 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:28:34.220 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":116,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:27:41.021144+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:08.709967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:09.456518+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.26","mappings":[{"from":0,"to":2}]},{"pgid":"17.30","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.183:6834/1383294888":"2024-03-07T07:27:33.605177+0000","172.21.15.183:6832/1383294888":"2024-03-07T07:27:33.605177+0000","172.21.15.83:6836/855387870":"2024-03-07T07:27:33.587133+0000","172.21.15.83:6834/855387870":"2024-03-07T07:27:33.587133+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:34.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.219+0000 7f1095d2b640 1 -- 172.21.15.83:0/828502623 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f107006b4d0 msgr2=0x7f107006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.219+0000 7f1095d2b640 1 --2- 172.21.15.83:0/828502623 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f107006b4d0 0x7f107006d990 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f1080036000 tx=0x7f108003b040 comp rx=0 tx=0).stop 2024-03-06T07:28:34.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.219+0000 7f1095d2b640 1 -- 172.21.15.83:0/828502623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10900cfff0 msgr2=0x7f1090150ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.219+0000 7f1095d2b640 1 --2- 172.21.15.83:0/828502623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10900cfff0 0x7f1090150ec0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f107800e9e0 tx=0x7f107800eea0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.219+0000 7f1095d2b640 1 -- 172.21.15.83:0/828502623 shutdown_connections 2024-03-06T07:28:34.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.219+0000 7f1095d2b640 1 --2- 172.21.15.83:0/828502623 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f107006b4d0 0x7f107006d990 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.219+0000 7f1095d2b640 1 --2- 172.21.15.83:0/828502623 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f109015bb20 0x7f109006ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.219+0000 7f1095d2b640 1 --2- 172.21.15.83:0/828502623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10900cfff0 0x7f1090150ec0 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.219+0000 7f1095d2b640 1 --2- 172.21.15.83:0/828502623 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10900ce4b0 0x7f10900c56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.219+0000 7f1095d2b640 1 -- 172.21.15.83:0/828502623 >> 172.21.15.83:0/828502623 conn(0x7f109009a9f0 msgr2=0x7f10900b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:34.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.220+0000 7f1095d2b640 1 -- 172.21.15.83:0/828502623 shutdown_connections 2024-03-06T07:28:34.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.220+0000 7f1095d2b640 1 -- 172.21.15.83:0/828502623 wait complete. 2024-03-06T07:28:34.233 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:28:34.233 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:34.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.331+0000 7f2850c2a640 1 Processor -- start 2024-03-06T07:28:34.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.331+0000 7f2850c2a640 1 -- start start 2024-03-06T07:28:34.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.332+0000 7f2850c2a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f284c15dfe0 0x7f284c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.332+0000 7f2850c2a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f284c0cf830 0x7f284c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.332+0000 7f2850c2a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f284c0b3430 con 0x7f284c0d27f0 2024-03-06T07:28:34.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.332+0000 7f2850c2a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f284c0b31e0 con 0x7f284c15dfe0 2024-03-06T07:28:34.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.332+0000 7f2850c2a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f284c0b3d60 con 0x7f284c0cf830 2024-03-06T07:28:34.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.332+0000 7f284a59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f284c15dfe0 0x7f284c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.332+0000 7f284a59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f284c15dfe0 0x7f284c0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52584/0 (socket says 172.21.15.83:52584) 2024-03-06T07:28:34.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.332+0000 7f284a59c640 1 -- 172.21.15.83:0/4070138405 learned_addr learned my addr 172.21.15.83:0/4070138405 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:34.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.332+0000 7f2849d9b640 1 --2- 172.21.15.83:0/4070138405 >> v2:172.21.15.183:3300/0 conn(0x7f284c0cf830 0x7f284c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.332+0000 7f284a59c640 1 -- 172.21.15.83:0/4070138405 >> v2:172.21.15.183:3300/0 conn(0x7f284c0cf830 msgr2=0x7f284c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:34.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.332+0000 7f284a59c640 1 --2- 172.21.15.83:0/4070138405 >> v2:172.21.15.183:3300/0 conn(0x7f284c0cf830 0x7f284c0d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.332+0000 7f284a59c640 1 -- 172.21.15.83:0/4070138405 >> v1:172.21.15.183:6789/0 conn(0x7f284c0d27f0 legacy=0x7f284c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:34.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.332+0000 7f284a59c640 1 -- 172.21.15.83:0/4070138405 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f284c0b3710 con 0x7f284c15dfe0 2024-03-06T07:28:34.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.333+0000 7f284a59c640 1 --2- 172.21.15.83:0/4070138405 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f284c15dfe0 0x7f284c0cf2f0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f2844009870 tx=0x7f284402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90a9d50b391e961c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.333+0000 7f284959a640 1 -- 172.21.15.83:0/4070138405 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2844035020 con 0x7f284c15dfe0 2024-03-06T07:28:34.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.333+0000 7f284959a640 1 -- 172.21.15.83:0/4070138405 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2844031870 con 0x7f284c15dfe0 2024-03-06T07:28:34.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.333+0000 7f284959a640 1 -- 172.21.15.83:0/4070138405 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2844031c40 con 0x7f284c15dfe0 2024-03-06T07:28:34.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.334+0000 7f2850c2a640 1 -- 172.21.15.83:0/4070138405 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f284c15dfe0 msgr2=0x7f284c0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.334+0000 7f2850c2a640 1 --2- 172.21.15.83:0/4070138405 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f284c15dfe0 0x7f284c0cf2f0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f2844009870 tx=0x7f284402b910 comp rx=0 tx=0).stop 2024-03-06T07:28:34.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.334+0000 7f2850c2a640 1 -- 172.21.15.83:0/4070138405 shutdown_connections 2024-03-06T07:28:34.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.334+0000 7f2850c2a640 1 --2- 172.21.15.83:0/4070138405 >> v2:172.21.15.183:3300/0 conn(0x7f284c0cf830 0x7f284c0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.334+0000 7f2850c2a640 1 --2- 172.21.15.83:0/4070138405 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f284c15dfe0 0x7f284c0cf2f0 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.334+0000 7f2850c2a640 1 -- 172.21.15.83:0/4070138405 >> 172.21.15.83:0/4070138405 conn(0x7f284c0d3cd0 msgr2=0x7f284c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:34.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.334+0000 7f2850c2a640 1 -- 172.21.15.83:0/4070138405 shutdown_connections 2024-03-06T07:28:34.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.334+0000 7f2850c2a640 1 -- 172.21.15.83:0/4070138405 wait complete. 2024-03-06T07:28:34.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.335+0000 7f2850c2a640 1 Processor -- start 2024-03-06T07:28:34.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.335+0000 7f2850c2a640 1 -- start start 2024-03-06T07:28:34.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.336+0000 7f2850c2a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f284c0cf830 0x7f284c1610a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.336+0000 7f2850c2a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f284c0d27f0 0x7f284c149f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.336+0000 7f2850c2a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f284c15dfe0 0x7f284c14a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.336+0000 7f2850c2a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f284c0d1df0 con 0x7f284c15dfe0 2024-03-06T07:28:34.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.336+0000 7f2850c2a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f284c0d1c70 con 0x7f284c0d27f0 2024-03-06T07:28:34.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.336+0000 7f2850c2a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f284c15e3c0 con 0x7f284c0cf830 2024-03-06T07:28:34.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.336+0000 7f284ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f284c15dfe0 0x7f284c14a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.336+0000 7f284ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f284c15dfe0 0x7f284c14a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52590/0 (socket says 172.21.15.83:52590) 2024-03-06T07:28:34.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.336+0000 7f284ad9d640 1 -- 172.21.15.83:0/1763384246 learned_addr learned my addr 172.21.15.83:0/1763384246 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:34.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.336+0000 7f284a59c640 1 --2- 172.21.15.83:0/1763384246 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f284c0cf830 0x7f284c1610a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.336+0000 7f2849d9b640 1 --2- 172.21.15.83:0/1763384246 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f284c0d27f0 0x7f284c149f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.336+0000 7f284ad9d640 1 -- 172.21.15.83:0/1763384246 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f284c0cf830 msgr2=0x7f284c1610a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.336+0000 7f284ad9d640 1 --2- 172.21.15.83:0/1763384246 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f284c0cf830 0x7f284c1610a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.336+0000 7f284ad9d640 1 -- 172.21.15.83:0/1763384246 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f284c0d27f0 msgr2=0x7f284c149f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.336+0000 7f284ad9d640 1 --2- 172.21.15.83:0/1763384246 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f284c0d27f0 0x7f284c149f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.336+0000 7f284ad9d640 1 -- 172.21.15.83:0/1763384246 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f284c0ccbd0 con 0x7f284c15dfe0 2024-03-06T07:28:34.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.336+0000 7f284a59c640 1 --2- 172.21.15.83:0/1763384246 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f284c0cf830 0x7f284c1610a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:34.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.336+0000 7f2849d9b640 1 --2- 172.21.15.83:0/1763384246 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f284c0d27f0 0x7f284c149f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:34.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.336+0000 7f284ad9d640 1 --2- 172.21.15.83:0/1763384246 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f284c15dfe0 0x7f284c14a480 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f283c00c7b0 tx=0x7f283c00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.336+0000 7f283affd640 1 -- 172.21.15.83:0/1763384246 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f283c004b60 con 0x7f284c15dfe0 2024-03-06T07:28:34.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.336+0000 7f283affd640 1 -- 172.21.15.83:0/1763384246 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f283c004d00 con 0x7f284c15dfe0 2024-03-06T07:28:34.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.337+0000 7f2850c2a640 1 -- 172.21.15.83:0/1763384246 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f284c0cceb0 con 0x7f284c15dfe0 2024-03-06T07:28:34.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.337+0000 7f283affd640 1 -- 172.21.15.83:0/1763384246 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f283c00aa90 con 0x7f284c15dfe0 2024-03-06T07:28:34.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.337+0000 7f2850c2a640 1 -- 172.21.15.83:0/1763384246 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f284c0cd470 con 0x7f284c15dfe0 2024-03-06T07:28:34.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.338+0000 7f2850c2a640 1 -- 172.21.15.83:0/1763384246 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f284c04f330 con 0x7f284c15dfe0 2024-03-06T07:28:34.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.338+0000 7f283affd640 1 -- 172.21.15.83:0/1763384246 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f283c00ac30 con 0x7f284c15dfe0 2024-03-06T07:28:34.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.338+0000 7f283affd640 1 --2- 172.21.15.83:0/1763384246 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f282c06b4d0 0x7f282c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.338+0000 7f283affd640 1 -- 172.21.15.83:0/1763384246 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f283c08a7e0 con 0x7f284c15dfe0 2024-03-06T07:28:34.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.338+0000 7f284a59c640 1 --2- 172.21.15.83:0/1763384246 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f282c06b4d0 0x7f282c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.339+0000 7f284a59c640 1 --2- 172.21.15.83:0/1763384246 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f282c06b4d0 0x7f282c06d990 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f2844036000 tx=0x7f284403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.343+0000 7f283affd640 1 -- 172.21.15.83:0/1763384246 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f283c059600 con 0x7f284c15dfe0 2024-03-06T07:28:34.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.557+0000 7f2850c2a640 1 -- 172.21.15.83:0/1763384246 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f284c1615e0 con 0x7f284c15dfe0 2024-03-06T07:28:34.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.558+0000 7f283affd640 1 -- 172.21.15.83:0/1763384246 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f283c058f00 con 0x7f284c15dfe0 2024-03-06T07:28:34.560 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:28:34.560 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7051,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/101642171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":101642171},{"type":"v1","addr":"172.21.15.183:6833","nonce":101642171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7057,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3146884900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":3146884900},{"type":"v1","addr":"172.21.15.183:6835","nonce":3146884900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8208,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1922041621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1922041621},{"type":"v1","addr":"172.21.15.83:6835","nonce":1922041621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.491022+0000","modified":"2024-03-06T07:27:37.006649+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8229":{"gid":8229,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/2003687153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2003687153},{"type":"v1","addr":"172.21.15.83:6837","nonce":2003687153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8229,"qdb_cluster":[8229]},"id":8}]} 2024-03-06T07:28:34.560 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 66 2024-03-06T07:28:34.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.560+0000 7f2850c2a640 1 -- 172.21.15.83:0/1763384246 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f282c06b4d0 msgr2=0x7f282c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.560+0000 7f2850c2a640 1 --2- 172.21.15.83:0/1763384246 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f282c06b4d0 0x7f282c06d990 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f2844036000 tx=0x7f284403c040 comp rx=0 tx=0).stop 2024-03-06T07:28:34.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.561+0000 7f2850c2a640 1 -- 172.21.15.83:0/1763384246 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f284c15dfe0 msgr2=0x7f284c14a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.561+0000 7f2850c2a640 1 --2- 172.21.15.83:0/1763384246 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f284c15dfe0 0x7f284c14a480 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f283c00c7b0 tx=0x7f283c00cc70 comp rx=0 tx=0).stop 2024-03-06T07:28:34.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.561+0000 7f2850c2a640 1 -- 172.21.15.83:0/1763384246 shutdown_connections 2024-03-06T07:28:34.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.561+0000 7f2850c2a640 1 --2- 172.21.15.83:0/1763384246 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f282c06b4d0 0x7f282c06d990 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.561+0000 7f2850c2a640 1 --2- 172.21.15.83:0/1763384246 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f284c15dfe0 0x7f284c14a480 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.561+0000 7f2850c2a640 1 --2- 172.21.15.83:0/1763384246 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f284c0d27f0 0x7f284c149f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.561+0000 7f2850c2a640 1 --2- 172.21.15.83:0/1763384246 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f284c0cf830 0x7f284c1610a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.561+0000 7f2850c2a640 1 -- 172.21.15.83:0/1763384246 >> 172.21.15.83:0/1763384246 conn(0x7f284c0d3cd0 msgr2=0x7f284c0cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:34.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.561+0000 7f2850c2a640 1 -- 172.21.15.83:0/1763384246 shutdown_connections 2024-03-06T07:28:34.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.561+0000 7f2850c2a640 1 -- 172.21.15.83:0/1763384246 wait complete. 2024-03-06T07:28:34.574 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:34.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.672+0000 7f9c72cc9640 1 Processor -- start 2024-03-06T07:28:34.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.672+0000 7f9c72cc9640 1 -- start start 2024-03-06T07:28:34.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.673+0000 7f9c72cc9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9c6c15fa10 0x7f9c6c0ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.673+0000 7f9c72cc9640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c6c0cebe0 con 0x7f9c6c0d16e0 2024-03-06T07:28:34.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.673+0000 7f9c72cc9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c6c0ceec0 con 0x7f9c6c15bdd0 2024-03-06T07:28:34.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.673+0000 7f9c72cc9640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c6c0cf1a0 con 0x7f9c6c15fa10 2024-03-06T07:28:34.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.673+0000 7f9c6bfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9c6c15fa10 0x7f9c6c0ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.673+0000 7f9c71266640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f9c6c0d16e0 0x7f9c6c0d1af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49152/0 (socket says 172.21.15.83:49152) 2024-03-06T07:28:34.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.673+0000 7f9c71266640 1 -- 172.21.15.83:0/637730017 learned_addr learned my addr 172.21.15.83:0/637730017 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:28:34.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.673+0000 7f9c6bfff640 1 -- 172.21.15.83:0/637730017 >> v1:172.21.15.183:6789/0 conn(0x7f9c6c15bdd0 legacy=0x7f9c6c0c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:34.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.673+0000 7f9c6bfff640 1 -- 172.21.15.83:0/637730017 >> v1:172.21.15.83:6789/0 conn(0x7f9c6c0d16e0 legacy=0x7f9c6c0d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.673+0000 7f9c6bfff640 1 -- 172.21.15.83:0/637730017 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c6c0cc8e0 con 0x7f9c6c15fa10 2024-03-06T07:28:34.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.673+0000 7f9c6b7fe640 1 -- 172.21.15.83:0/637730017 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2427847233 0 0) 0x7f9c6c0cebe0 con 0x7f9c6c0d16e0 2024-03-06T07:28:34.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.674+0000 7f9c6bfff640 1 --2- 172.21.15.83:0/637730017 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c6c15fa10 0x7f9c6c0ce270 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f9c580097a0 tx=0x7f9c5802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74402786eec74183 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.674+0000 7f9c6b7fe640 1 -- 172.21.15.83:0/637730017 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c580315a0 con 0x7f9c6c15fa10 2024-03-06T07:28:34.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.674+0000 7f9c6b7fe640 1 -- 172.21.15.83:0/637730017 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c58037070 con 0x7f9c6c15fa10 2024-03-06T07:28:34.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.674+0000 7f9c6b7fe640 1 -- 172.21.15.83:0/637730017 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c58031960 con 0x7f9c6c15fa10 2024-03-06T07:28:34.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.675+0000 7f9c72cc9640 1 -- 172.21.15.83:0/637730017 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c6c15fa10 msgr2=0x7f9c6c0ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.675+0000 7f9c72cc9640 1 --2- 172.21.15.83:0/637730017 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c6c15fa10 0x7f9c6c0ce270 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f9c580097a0 tx=0x7f9c5802b840 comp rx=0 tx=0).stop 2024-03-06T07:28:34.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.675+0000 7f9c72cc9640 1 -- 172.21.15.83:0/637730017 shutdown_connections 2024-03-06T07:28:34.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.675+0000 7f9c72cc9640 1 --2- 172.21.15.83:0/637730017 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c6c15fa10 0x7f9c6c0ce270 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.675+0000 7f9c72cc9640 1 -- 172.21.15.83:0/637730017 >> 172.21.15.83:0/637730017 conn(0x7f9c6c09a9f0 msgr2=0x7f9c6c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:34.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.675+0000 7f9c72cc9640 1 -- 172.21.15.83:0/637730017 shutdown_connections 2024-03-06T07:28:34.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.675+0000 7f9c72cc9640 1 -- 172.21.15.83:0/637730017 wait complete. 2024-03-06T07:28:34.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.676+0000 7f9c72cc9640 1 Processor -- start 2024-03-06T07:28:34.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.676+0000 7f9c72cc9640 1 -- start start 2024-03-06T07:28:34.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.677+0000 7f9c72cc9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c6c0d16e0 0x7f9c6c0b60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.677+0000 7f9c72cc9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c6c15bdd0 0x7f9c6c148da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.677+0000 7f9c72cc9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c6c15fa10 0x7f9c6c0c8f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.677+0000 7f9c72cc9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6c0ce7b0 con 0x7f9c6c15fa10 2024-03-06T07:28:34.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.677+0000 7f9c72cc9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6c15c1b0 con 0x7f9c6c0d16e0 2024-03-06T07:28:34.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.677+0000 7f9c72cc9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6c0ce930 con 0x7f9c6c15bdd0 2024-03-06T07:28:34.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.677+0000 7f9c71266640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c6c15fa10 0x7f9c6c0c8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.677+0000 7f9c71266640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c6c15fa10 0x7f9c6c0c8f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52620/0 (socket says 172.21.15.83:52620) 2024-03-06T07:28:34.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.677+0000 7f9c71266640 1 -- 172.21.15.83:0/313231606 learned_addr learned my addr 172.21.15.83:0/313231606 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:34.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.677+0000 7f9c6bfff640 1 --2- 172.21.15.83:0/313231606 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c6c15bdd0 0x7f9c6c148da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.677+0000 7f9c70a65640 1 --2- 172.21.15.83:0/313231606 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c6c0d16e0 0x7f9c6c0b60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.677+0000 7f9c71266640 1 -- 172.21.15.83:0/313231606 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c6c15bdd0 msgr2=0x7f9c6c148da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.677+0000 7f9c71266640 1 --2- 172.21.15.83:0/313231606 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c6c15bdd0 0x7f9c6c148da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.677+0000 7f9c71266640 1 -- 172.21.15.83:0/313231606 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c6c0d16e0 msgr2=0x7f9c6c0b60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.677+0000 7f9c71266640 1 --2- 172.21.15.83:0/313231606 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c6c0d16e0 0x7f9c6c0b60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.677+0000 7f9c71266640 1 -- 172.21.15.83:0/313231606 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c6c158690 con 0x7f9c6c15fa10 2024-03-06T07:28:34.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.677+0000 7f9c70a65640 1 --2- 172.21.15.83:0/313231606 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c6c0d16e0 0x7f9c6c0b60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:34.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.677+0000 7f9c6bfff640 1 --2- 172.21.15.83:0/313231606 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c6c15bdd0 0x7f9c6c148da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:34.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.677+0000 7f9c71266640 1 --2- 172.21.15.83:0/313231606 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c6c15fa10 0x7f9c6c0c8f20 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f9c6400caf0 tx=0x7f9c640021c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.678+0000 7f9c697fa640 1 -- 172.21.15.83:0/313231606 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c6400ac00 con 0x7f9c6c15fa10 2024-03-06T07:28:34.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.678+0000 7f9c697fa640 1 -- 172.21.15.83:0/313231606 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c6400ada0 con 0x7f9c6c15fa10 2024-03-06T07:28:34.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.678+0000 7f9c72cc9640 1 -- 172.21.15.83:0/313231606 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c6c06c550 con 0x7f9c6c15fa10 2024-03-06T07:28:34.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.678+0000 7f9c697fa640 1 -- 172.21.15.83:0/313231606 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c64012640 con 0x7f9c6c15fa10 2024-03-06T07:28:34.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.678+0000 7f9c72cc9640 1 -- 172.21.15.83:0/313231606 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c6c06cb10 con 0x7f9c6c15fa10 2024-03-06T07:28:34.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.679+0000 7f9c72cc9640 1 -- 172.21.15.83:0/313231606 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c6c04f330 con 0x7f9c6c15fa10 2024-03-06T07:28:34.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.679+0000 7f9c697fa640 1 -- 172.21.15.83:0/313231606 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9c640127e0 con 0x7f9c6c15fa10 2024-03-06T07:28:34.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.679+0000 7f9c697fa640 1 --2- 172.21.15.83:0/313231606 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9c5006b4d0 0x7f9c5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.680+0000 7f9c697fa640 1 -- 172.21.15.83:0/313231606 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f9c6408a8a0 con 0x7f9c6c15fa10 2024-03-06T07:28:34.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.680+0000 7f9c70a65640 1 --2- 172.21.15.83:0/313231606 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9c5006b4d0 0x7f9c5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.680+0000 7f9c70a65640 1 --2- 172.21.15.83:0/313231606 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9c5006b4d0 0x7f9c5006d990 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f9c60001750 tx=0x7f9c60009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.684+0000 7f9c697fa640 1 -- 172.21.15.83:0/313231606 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9c64059630 con 0x7f9c6c15fa10 2024-03-06T07:28:34.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.899+0000 7f9c72cc9640 1 -- 172.21.15.83:0/313231606 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c6c06d270 con 0x7f9c6c15fa10 2024-03-06T07:28:34.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.900+0000 7f9c697fa640 1 -- 172.21.15.83:0/313231606 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f9c64058f30 con 0x7f9c6c15fa10 2024-03-06T07:28:34.931 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:28:34.931 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7051,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/101642171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":101642171},{"type":"v1","addr":"172.21.15.183:6833","nonce":101642171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7057,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3146884900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":3146884900},{"type":"v1","addr":"172.21.15.183:6835","nonce":3146884900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8208,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1922041621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1922041621},{"type":"v1","addr":"172.21.15.83:6835","nonce":1922041621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:10.491022+0000","modified":"2024-03-06T07:27:37.006649+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8229":{"gid":8229,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/2003687153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2003687153},{"type":"v1","addr":"172.21.15.83:6837","nonce":2003687153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8229,"qdb_cluster":[8229]},"id":8}]} 2024-03-06T07:28:34.932 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 66 2024-03-06T07:28:34.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.902+0000 7f9c72cc9640 1 -- 172.21.15.83:0/313231606 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9c5006b4d0 msgr2=0x7f9c5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.902+0000 7f9c72cc9640 1 --2- 172.21.15.83:0/313231606 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9c5006b4d0 0x7f9c5006d990 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f9c60001750 tx=0x7f9c60009040 comp rx=0 tx=0).stop 2024-03-06T07:28:34.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.902+0000 7f9c72cc9640 1 -- 172.21.15.83:0/313231606 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c6c15fa10 msgr2=0x7f9c6c0c8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.902+0000 7f9c72cc9640 1 --2- 172.21.15.83:0/313231606 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c6c15fa10 0x7f9c6c0c8f20 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f9c6400caf0 tx=0x7f9c640021c0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.902+0000 7f9c72cc9640 1 -- 172.21.15.83:0/313231606 shutdown_connections 2024-03-06T07:28:34.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.902+0000 7f9c72cc9640 1 --2- 172.21.15.83:0/313231606 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9c5006b4d0 0x7f9c5006d990 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.902+0000 7f9c72cc9640 1 --2- 172.21.15.83:0/313231606 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c6c15fa10 0x7f9c6c0c8f20 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.902+0000 7f9c72cc9640 1 --2- 172.21.15.83:0/313231606 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c6c15bdd0 0x7f9c6c148da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.902+0000 7f9c72cc9640 1 --2- 172.21.15.83:0/313231606 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c6c0d16e0 0x7f9c6c0b60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.902+0000 7f9c72cc9640 1 -- 172.21.15.83:0/313231606 >> 172.21.15.83:0/313231606 conn(0x7f9c6c09a9f0 msgr2=0x7f9c6c0c6220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:34.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.902+0000 7f9c72cc9640 1 -- 172.21.15.83:0/313231606 shutdown_connections 2024-03-06T07:28:34.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:34.902+0000 7f9c72cc9640 1 -- 172.21.15.83:0/313231606 wait complete. 2024-03-06T07:28:34.934 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:35.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.033+0000 7fe54615c640 1 Processor -- start 2024-03-06T07:28:35.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.033+0000 7fe54615c640 1 -- start start 2024-03-06T07:28:35.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.034+0000 7fe54615c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe54015bb20 0x7fe5400d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.034+0000 7fe54615c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe540161980 con 0x7fe5400cfff0 2024-03-06T07:28:35.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.034+0000 7fe54615c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe540161c60 con 0x7fe5400cd8c0 2024-03-06T07:28:35.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.034+0000 7fe54615c640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe540161f40 con 0x7fe54015bb20 2024-03-06T07:28:35.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.034+0000 7fe53f7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe54015bb20 0x7fe5400d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.034+0000 7fe536ffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fe5400cfff0 0x7fe5400ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49162/0 (socket says 172.21.15.83:49162) 2024-03-06T07:28:35.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.034+0000 7fe536ffd640 1 -- 172.21.15.83:0/3280748111 learned_addr learned my addr 172.21.15.83:0/3280748111 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:28:35.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.035+0000 7fe53f7fe640 1 -- 172.21.15.83:0/3280748111 >> v1:172.21.15.183:6789/0 conn(0x7fe5400cd8c0 legacy=0x7fe5400cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.035+0000 7fe53f7fe640 1 -- 172.21.15.83:0/3280748111 >> v1:172.21.15.83:6789/0 conn(0x7fe5400cfff0 legacy=0x7fe5400ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.035+0000 7fe53effd640 1 -- 172.21.15.83:0/3280748111 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2900094298 0 0) 0x7fe540161980 con 0x7fe5400cfff0 2024-03-06T07:28:35.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.035+0000 7fe53f7fe640 1 -- 172.21.15.83:0/3280748111 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe540162220 con 0x7fe54015bb20 2024-03-06T07:28:35.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.036+0000 7fe53f7fe640 1 --2- 172.21.15.83:0/3280748111 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe54015bb20 0x7fe5400d1df0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7fe5300097a0 tx=0x7fe53002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aaff553416701aa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.036+0000 7fe53effd640 1 -- 172.21.15.83:0/3280748111 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe530034070 con 0x7fe54015bb20 2024-03-06T07:28:35.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.036+0000 7fe53effd640 1 -- 172.21.15.83:0/3280748111 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe530030610 con 0x7fe54015bb20 2024-03-06T07:28:35.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.036+0000 7fe53effd640 1 -- 172.21.15.83:0/3280748111 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5300309e0 con 0x7fe54015bb20 2024-03-06T07:28:35.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.036+0000 7fe54615c640 1 -- 172.21.15.83:0/3280748111 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe54015bb20 msgr2=0x7fe5400d1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.036+0000 7fe54615c640 1 --2- 172.21.15.83:0/3280748111 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe54015bb20 0x7fe5400d1df0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7fe5300097a0 tx=0x7fe53002b960 comp rx=0 tx=0).stop 2024-03-06T07:28:35.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.036+0000 7fe54615c640 1 -- 172.21.15.83:0/3280748111 shutdown_connections 2024-03-06T07:28:35.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.036+0000 7fe54615c640 1 --2- 172.21.15.83:0/3280748111 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe54015bb20 0x7fe5400d1df0 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.036+0000 7fe54615c640 1 -- 172.21.15.83:0/3280748111 >> 172.21.15.83:0/3280748111 conn(0x7fe54009a9f0 msgr2=0x7fe54009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.036+0000 7fe54615c640 1 -- 172.21.15.83:0/3280748111 shutdown_connections 2024-03-06T07:28:35.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.036+0000 7fe54615c640 1 -- 172.21.15.83:0/3280748111 wait complete. 2024-03-06T07:28:35.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.037+0000 7fe54615c640 1 Processor -- start 2024-03-06T07:28:35.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.038+0000 7fe54615c640 1 -- start start 2024-03-06T07:28:35.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.038+0000 7fe54615c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe5400cd8c0 0x7fe5401f7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.038+0000 7fe54615c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe5400cfff0 0x7fe5400c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.038+0000 7fe54615c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe54015bb20 0x7fe54006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.038+0000 7fe54615c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5400d2330 con 0x7fe54015bb20 2024-03-06T07:28:35.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.038+0000 7fe54615c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5400cf2d0 con 0x7fe5400cd8c0 2024-03-06T07:28:35.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.038+0000 7fe54615c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5400d24b0 con 0x7fe5400cfff0 2024-03-06T07:28:35.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.038+0000 7fe53ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe54015bb20 0x7fe54006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.039+0000 7fe53ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe54015bb20 0x7fe54006d150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52638/0 (socket says 172.21.15.83:52638) 2024-03-06T07:28:35.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.039+0000 7fe53ffff640 1 -- 172.21.15.83:0/4121600311 learned_addr learned my addr 172.21.15.83:0/4121600311 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:35.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.039+0000 7fe53ffff640 1 -- 172.21.15.83:0/4121600311 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe5400cfff0 msgr2=0x7fe5400c8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.039+0000 7fe53ffff640 1 --2- 172.21.15.83:0/4121600311 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe5400cfff0 0x7fe5400c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.039+0000 7fe53ffff640 1 -- 172.21.15.83:0/4121600311 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe5400cd8c0 msgr2=0x7fe5401f7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.039+0000 7fe53f7fe640 1 --2- 172.21.15.83:0/4121600311 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe5400cd8c0 0x7fe5401f7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.039+0000 7fe53ffff640 1 --2- 172.21.15.83:0/4121600311 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe5400cd8c0 0x7fe5401f7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.039+0000 7fe53ffff640 1 -- 172.21.15.83:0/4121600311 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5401f7ae0 con 0x7fe54015bb20 2024-03-06T07:28:35.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.039+0000 7fe53ffff640 1 --2- 172.21.15.83:0/4121600311 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe54015bb20 0x7fe54006d150 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fe53800c6b0 tx=0x7fe53800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.039+0000 7fe53cff9640 1 -- 172.21.15.83:0/4121600311 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe538003b70 con 0x7fe54015bb20 2024-03-06T07:28:35.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.039+0000 7fe53cff9640 1 -- 172.21.15.83:0/4121600311 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe538003d30 con 0x7fe54015bb20 2024-03-06T07:28:35.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.039+0000 7fe54615c640 1 -- 172.21.15.83:0/4121600311 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5401f7dc0 con 0x7fe54015bb20 2024-03-06T07:28:35.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.039+0000 7fe53cff9640 1 -- 172.21.15.83:0/4121600311 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe538005310 con 0x7fe54015bb20 2024-03-06T07:28:35.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.039+0000 7fe54615c640 1 -- 172.21.15.83:0/4121600311 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe54015adb0 con 0x7fe54015bb20 2024-03-06T07:28:35.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.040+0000 7fe54615c640 1 -- 172.21.15.83:0/4121600311 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe54004f330 con 0x7fe54015bb20 2024-03-06T07:28:35.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.041+0000 7fe53cff9640 1 -- 172.21.15.83:0/4121600311 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe5380054b0 con 0x7fe54015bb20 2024-03-06T07:28:35.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.041+0000 7fe53cff9640 1 --2- 172.21.15.83:0/4121600311 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe52006b4d0 0x7fe52006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.041+0000 7fe53f7fe640 1 --2- 172.21.15.83:0/4121600311 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe52006b4d0 0x7fe52006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.041+0000 7fe53cff9640 1 -- 172.21.15.83:0/4121600311 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fe53808a0c0 con 0x7fe54015bb20 2024-03-06T07:28:35.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.042+0000 7fe53f7fe640 1 --2- 172.21.15.83:0/4121600311 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe52006b4d0 0x7fe52006d990 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fe530036000 tx=0x7fe530008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.045+0000 7fe53cff9640 1 -- 172.21.15.83:0/4121600311 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe538058ee0 con 0x7fe54015bb20 2024-03-06T07:28:35.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.241+0000 7fe54615c640 1 -- 172.21.15.83:0/4121600311 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5400b1820 con 0x7fe54015bb20 2024-03-06T07:28:35.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.243+0000 7fe53cff9640 1 -- 172.21.15.83:0/4121600311 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+16992 (secure 0 0 0) 0x7fe5380587e0 con 0x7fe54015bb20 2024-03-06T07:28:35.245 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:28:35.245 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":116,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:27:41.021144+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:08.709967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:09.456518+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.26","mappings":[{"from":0,"to":2}]},{"pgid":"17.30","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.183:6834/1383294888":"2024-03-07T07:27:33.605177+0000","172.21.15.183:6832/1383294888":"2024-03-07T07:27:33.605177+0000","172.21.15.83:6836/855387870":"2024-03-07T07:27:33.587133+0000","172.21.15.83:6834/855387870":"2024-03-07T07:27:33.587133+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:35.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.244+0000 7fe54615c640 1 -- 172.21.15.83:0/4121600311 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe52006b4d0 msgr2=0x7fe52006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.244+0000 7fe54615c640 1 --2- 172.21.15.83:0/4121600311 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe52006b4d0 0x7fe52006d990 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fe530036000 tx=0x7fe530008030 comp rx=0 tx=0).stop 2024-03-06T07:28:35.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.244+0000 7fe54615c640 1 -- 172.21.15.83:0/4121600311 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe54015bb20 msgr2=0x7fe54006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.244+0000 7fe54615c640 1 --2- 172.21.15.83:0/4121600311 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe54015bb20 0x7fe54006d150 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fe53800c6b0 tx=0x7fe53800cb70 comp rx=0 tx=0).stop 2024-03-06T07:28:35.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.245+0000 7fe54615c640 1 -- 172.21.15.83:0/4121600311 shutdown_connections 2024-03-06T07:28:35.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.245+0000 7fe54615c640 1 --2- 172.21.15.83:0/4121600311 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe52006b4d0 0x7fe52006d990 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.245+0000 7fe54615c640 1 --2- 172.21.15.83:0/4121600311 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe54015bb20 0x7fe54006d150 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.245+0000 7fe54615c640 1 --2- 172.21.15.83:0/4121600311 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe5400cfff0 0x7fe5400c8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.245+0000 7fe54615c640 1 --2- 172.21.15.83:0/4121600311 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe5400cd8c0 0x7fe5401f7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.245+0000 7fe54615c640 1 -- 172.21.15.83:0/4121600311 >> 172.21.15.83:0/4121600311 conn(0x7fe54009a9f0 msgr2=0x7fe54006ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.245+0000 7fe54615c640 1 -- 172.21.15.83:0/4121600311 shutdown_connections 2024-03-06T07:28:35.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.245+0000 7fe54615c640 1 -- 172.21.15.83:0/4121600311 wait complete. 2024-03-06T07:28:35.258 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:28:35.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.360+0000 7ff98f6d4640 1 Processor -- start 2024-03-06T07:28:35.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.360+0000 7ff98f6d4640 1 -- start start 2024-03-06T07:28:35.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.361+0000 7ff98f6d4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9880b4460 0x7ff9880d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.361+0000 7ff98f6d4640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ff988162970 0x7ff9880b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.361+0000 7ff98f6d4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9880d22c0 con 0x7ff988161cf0 2024-03-06T07:28:35.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.361+0000 7ff98f6d4640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff988161b00 con 0x7ff988162970 2024-03-06T07:28:35.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.361+0000 7ff98f6d4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9880cecc0 con 0x7ff9880b4460 2024-03-06T07:28:35.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.361+0000 7ff98dc71640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff988161cf0 0x7ff9880cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49164/0 (socket says 172.21.15.83:49164) 2024-03-06T07:28:35.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.361+0000 7ff98dc71640 1 -- 172.21.15.83:0/2886857868 learned_addr learned my addr 172.21.15.83:0/2886857868 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:28:35.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.361+0000 7ff97ffff640 1 -- 172.21.15.83:0/2886857868 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 976649209 0 0) 0x7ff9880d22c0 con 0x7ff988161cf0 2024-03-06T07:28:35.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.361+0000 7ff97ffff640 1 -- 172.21.15.83:0/2886857868 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff974003520 con 0x7ff988161cf0 2024-03-06T07:28:35.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.361+0000 7ff98cc6f640 1 --2- 172.21.15.83:0/2886857868 >> v2:172.21.15.183:3300/0 conn(0x7ff988162970 0x7ff9880b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.361+0000 7ff98d470640 1 --2- 172.21.15.83:0/2886857868 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9880b4460 0x7ff9880d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.361+0000 7ff97ffff640 1 -- 172.21.15.83:0/2886857868 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1043386317 0 0) 0x7ff974003520 con 0x7ff988161cf0 2024-03-06T07:28:35.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.361+0000 7ff97ffff640 1 -- 172.21.15.83:0/2886857868 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9880b4460 msgr2=0x7ff9880d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:35.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.361+0000 7ff97ffff640 1 --2- 172.21.15.83:0/2886857868 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9880b4460 0x7ff9880d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.361+0000 7ff97ffff640 1 -- 172.21.15.83:0/2886857868 >> v2:172.21.15.183:3300/0 conn(0x7ff988162970 msgr2=0x7ff9880b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:35.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.361+0000 7ff97ffff640 1 --2- 172.21.15.83:0/2886857868 >> v2:172.21.15.183:3300/0 conn(0x7ff988162970 0x7ff9880b3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.361+0000 7ff97ffff640 1 -- 172.21.15.83:0/2886857868 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9880d25a0 con 0x7ff988161cf0 2024-03-06T07:28:35.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.361+0000 7ff97ffff640 1 -- 172.21.15.83:0/2886857868 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7ff9800019f0 con 0x7ff988161cf0 2024-03-06T07:28:35.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.361+0000 7ff97ffff640 1 -- 172.21.15.83:0/2886857868 >> v1:172.21.15.83:6789/0 conn(0x7ff988161cf0 legacy=0x7ff9880cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.362+0000 7ff97ffff640 1 --2- 172.21.15.83:0/2886857868 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff9740039b0 0x7ff974003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.362+0000 7ff97ffff640 1 --2- 172.21.15.83:0/2886857868 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9880b4460 0x7ff974004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.362+0000 7ff97ffff640 1 --2- 172.21.15.83:0/2886857868 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff974004d40 0x7ff9740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.362+0000 7ff97ffff640 1 -- 172.21.15.83:0/2886857868 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff988161b00 con 0x7ff974004d40 2024-03-06T07:28:35.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.362+0000 7ff97ffff640 1 -- 172.21.15.83:0/2886857868 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9880cecc0 con 0x7ff9740039b0 2024-03-06T07:28:35.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.362+0000 7ff97ffff640 1 -- 172.21.15.83:0/2886857868 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff97400b9c0 con 0x7ff9880b4460 2024-03-06T07:28:35.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.362+0000 7ff98dc71640 1 --2- 172.21.15.83:0/2886857868 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff974004d40 0x7ff9740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.362+0000 7ff98d470640 1 --2- 172.21.15.83:0/2886857868 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff9740039b0 0x7ff974003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.362+0000 7ff98cc6f640 1 --2- 172.21.15.83:0/2886857868 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9880b4460 0x7ff974004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.362+0000 7ff98dc71640 1 -- 172.21.15.83:0/2886857868 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9880b4460 msgr2=0x7ff974004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:35.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.362+0000 7ff98dc71640 1 --2- 172.21.15.83:0/2886857868 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9880b4460 0x7ff974004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.362+0000 7ff98dc71640 1 -- 172.21.15.83:0/2886857868 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff9740039b0 msgr2=0x7ff974003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:35.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.362+0000 7ff98dc71640 1 --2- 172.21.15.83:0/2886857868 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff9740039b0 0x7ff974003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.362+0000 7ff98dc71640 1 -- 172.21.15.83:0/2886857868 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff974003520 con 0x7ff974004d40 2024-03-06T07:28:35.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.363+0000 7ff98dc71640 1 --2- 172.21.15.83:0/2886857868 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff974004d40 0x7ff9740051c0 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7ff9800013d0 tx=0x7ff974005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec9009c716a76d50 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.363+0000 7ff97ffff640 1 -- 172.21.15.83:0/2886857868 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98000a4e0 con 0x7ff974004d40 2024-03-06T07:28:35.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.363+0000 7ff97ffff640 1 -- 172.21.15.83:0/2886857868 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff980034650 con 0x7ff974004d40 2024-03-06T07:28:35.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.363+0000 7ff97ffff640 1 -- 172.21.15.83:0/2886857868 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff980034a40 con 0x7ff974004d40 2024-03-06T07:28:35.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.363+0000 7ff98f6d4640 1 -- 172.21.15.83:0/2886857868 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff974004d40 msgr2=0x7ff9740051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.363+0000 7ff98f6d4640 1 --2- 172.21.15.83:0/2886857868 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff974004d40 0x7ff9740051c0 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7ff9800013d0 tx=0x7ff974005830 comp rx=0 tx=0).stop 2024-03-06T07:28:35.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.363+0000 7ff98dc71640 1 -- 172.21.15.83:0/2886857868 reap_dead start 2024-03-06T07:28:35.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.363+0000 7ff98f6d4640 1 -- 172.21.15.83:0/2886857868 shutdown_connections 2024-03-06T07:28:35.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.363+0000 7ff98f6d4640 1 -- 172.21.15.83:0/2886857868 >> 172.21.15.83:0/2886857868 conn(0x7ff9880d3ce0 msgr2=0x7ff9880d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.363+0000 7ff98f6d4640 1 -- 172.21.15.83:0/2886857868 shutdown_connections 2024-03-06T07:28:35.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.364+0000 7ff98f6d4640 1 -- 172.21.15.83:0/2886857868 wait complete. 2024-03-06T07:28:35.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.365+0000 7ff98f6d4640 1 Processor -- start 2024-03-06T07:28:35.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.365+0000 7ff98f6d4640 1 -- start start 2024-03-06T07:28:35.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.365+0000 7ff98f6d4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff974004d40 0x7ff9880ca470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.365+0000 7ff98f6d4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9880c8f30 0x7ff9880b9730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.365+0000 7ff98f6d4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff9880b1880 0x7ff9881588c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.365+0000 7ff98f6d4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9880ce950 con 0x7ff974004d40 2024-03-06T07:28:35.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.365+0000 7ff98f6d4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9880d2b60 con 0x7ff9880b1880 2024-03-06T07:28:35.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.365+0000 7ff98f6d4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9880b3ac0 con 0x7ff9880c8f30 2024-03-06T07:28:35.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.365+0000 7ff98d470640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff974004d40 0x7ff9880ca470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.365+0000 7ff98d470640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff974004d40 0x7ff9880ca470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52656/0 (socket says 172.21.15.83:52656) 2024-03-06T07:28:35.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.365+0000 7ff98d470640 1 -- 172.21.15.83:0/364531396 learned_addr learned my addr 172.21.15.83:0/364531396 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:35.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.366+0000 7ff98cc6f640 1 --2- 172.21.15.83:0/364531396 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9880c8f30 0x7ff9880b9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.366+0000 7ff98dc71640 1 --2- 172.21.15.83:0/364531396 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff9880b1880 0x7ff9881588c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.366+0000 7ff98d470640 1 -- 172.21.15.83:0/364531396 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9880c8f30 msgr2=0x7ff9880b9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.366+0000 7ff98d470640 1 --2- 172.21.15.83:0/364531396 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9880c8f30 0x7ff9880b9730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.366+0000 7ff98d470640 1 -- 172.21.15.83:0/364531396 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff9880b1880 msgr2=0x7ff9881588c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.366+0000 7ff98d470640 1 --2- 172.21.15.83:0/364531396 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff9880b1880 0x7ff9881588c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.366+0000 7ff98d470640 1 -- 172.21.15.83:0/364531396 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9880c93d0 con 0x7ff974004d40 2024-03-06T07:28:35.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.366+0000 7ff98d470640 1 --2- 172.21.15.83:0/364531396 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff974004d40 0x7ff9880ca470 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7ff9840119a0 tx=0x7ff984011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.366+0000 7ff97dffb640 1 -- 172.21.15.83:0/364531396 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98400faf0 con 0x7ff974004d40 2024-03-06T07:28:35.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.366+0000 7ff97dffb640 1 -- 172.21.15.83:0/364531396 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98400fc90 con 0x7ff974004d40 2024-03-06T07:28:35.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.366+0000 7ff98f6d4640 1 -- 172.21.15.83:0/364531396 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9880c96b0 con 0x7ff974004d40 2024-03-06T07:28:35.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.366+0000 7ff97dffb640 1 -- 172.21.15.83:0/364531396 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984014440 con 0x7ff974004d40 2024-03-06T07:28:35.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.366+0000 7ff98f6d4640 1 -- 172.21.15.83:0/364531396 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9880c9c70 con 0x7ff974004d40 2024-03-06T07:28:35.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.367+0000 7ff98f6d4640 1 -- 172.21.15.83:0/364531396 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff98804f330 con 0x7ff974004d40 2024-03-06T07:28:35.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.368+0000 7ff97dffb640 1 -- 172.21.15.83:0/364531396 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff984012040 con 0x7ff974004d40 2024-03-06T07:28:35.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.368+0000 7ff97dffb640 1 --2- 172.21.15.83:0/364531396 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff96c06b4d0 0x7ff96c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.368+0000 7ff98cc6f640 1 --2- 172.21.15.83:0/364531396 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff96c06b4d0 0x7ff96c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.368+0000 7ff97dffb640 1 -- 172.21.15.83:0/364531396 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5939+0+0 (secure 0 0 0) 0x7ff98408a9e0 con 0x7ff974004d40 2024-03-06T07:28:35.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.369+0000 7ff98cc6f640 1 --2- 172.21.15.83:0/364531396 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff96c06b4d0 0x7ff96c06d990 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7ff988162db0 tx=0x7ff978008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.373+0000 7ff97dffb640 1 -- 172.21.15.83:0/364531396 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff984019030 con 0x7ff974004d40 2024-03-06T07:28:35.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.589+0000 7ff98f6d4640 1 -- 172.21.15.83:0/364531396 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7ff98806b180 con 0x7ff974004d40 2024-03-06T07:28:35.591 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-03-06T07:28:35.589+0000 7fb0dae11640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:28:35.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.591+0000 7ff97dffb640 1 -- 172.21.15.83:0/364531396 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v67) v1 ==== 161+0+0 (secure 0 0 0) 0x7ff984059120 con 0x7ff974004d40 2024-03-06T07:28:35.592 INFO:teuthology.orchestra.run.smithi083.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:28:35.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.594+0000 7ff98f6d4640 1 -- 172.21.15.83:0/364531396 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff96c06b4d0 msgr2=0x7ff96c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.594+0000 7ff98f6d4640 1 --2- 172.21.15.83:0/364531396 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff96c06b4d0 0x7ff96c06d990 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7ff988162db0 tx=0x7ff978008040 comp rx=0 tx=0).stop 2024-03-06T07:28:35.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.594+0000 7ff98f6d4640 1 -- 172.21.15.83:0/364531396 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff974004d40 msgr2=0x7ff9880ca470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.594+0000 7ff98f6d4640 1 --2- 172.21.15.83:0/364531396 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff974004d40 0x7ff9880ca470 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7ff9840119a0 tx=0x7ff984011e60 comp rx=0 tx=0).stop 2024-03-06T07:28:35.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.594+0000 7ff98f6d4640 1 -- 172.21.15.83:0/364531396 shutdown_connections 2024-03-06T07:28:35.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.594+0000 7ff98f6d4640 1 --2- 172.21.15.83:0/364531396 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff96c06b4d0 0x7ff96c06d990 secure :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7ff988162db0 tx=0x7ff978008040 comp rx=0 tx=0).stop 2024-03-06T07:28:35.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.594+0000 7ff98f6d4640 1 --2- 172.21.15.83:0/364531396 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff9880b1880 0x7ff9881588c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.594+0000 7ff98f6d4640 1 --2- 172.21.15.83:0/364531396 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9880c8f30 0x7ff9880b9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.594+0000 7ff98f6d4640 1 --2- 172.21.15.83:0/364531396 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff974004d40 0x7ff9880ca470 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.594+0000 7ff98f6d4640 1 -- 172.21.15.83:0/364531396 >> 172.21.15.83:0/364531396 conn(0x7ff9880d3ce0 msgr2=0x7ff988073f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.594+0000 7ff98f6d4640 1 -- 172.21.15.83:0/364531396 shutdown_connections 2024-03-06T07:28:35.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.594+0000 7ff98f6d4640 1 -- 172.21.15.83:0/364531396 wait complete. 2024-03-06T07:28:35.599 INFO:tasks.ceph.mds.c.smithi083.stderr: -8737> 2024-03-06T07:27:39.003+0000 7fe908279640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:28:35.609 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:28:35.631 INFO:tasks.ceph.mds.c.smithi083.stdout:starting mds.c at 2024-03-06T07:28:35.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.709+0000 7f031233a640 1 Processor -- start 2024-03-06T07:28:35.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.709+0000 7f031233a640 1 -- start start 2024-03-06T07:28:35.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.710+0000 7f031233a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f030c1586e0 0x7f030c0aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.710+0000 7f031233a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f030c0aef80 0x7f030c0cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.710+0000 7f031233a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f030c158ac0 con 0x7f030c0cffd0 2024-03-06T07:28:35.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.710+0000 7f031233a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f030c15b2a0 con 0x7f030c0aef80 2024-03-06T07:28:35.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.710+0000 7f031233a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f030c15b540 con 0x7f030c1586e0 2024-03-06T07:28:35.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.710+0000 7f030b7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f030c0aef80 0x7f030c0cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.710+0000 7f030b7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f030c0aef80 0x7f030c0cfa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52696/0 (socket says 172.21.15.83:52696) 2024-03-06T07:28:35.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.710+0000 7f030b7fe640 1 -- 172.21.15.83:0/1365142974 learned_addr learned my addr 172.21.15.83:0/1365142974 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:35.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.710+0000 7f030bfff640 1 --2- 172.21.15.83:0/1365142974 >> v2:172.21.15.183:3300/0 conn(0x7f030c1586e0 0x7f030c0aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.710+0000 7f030b7fe640 1 -- 172.21.15.83:0/1365142974 >> v2:172.21.15.183:3300/0 conn(0x7f030c1586e0 msgr2=0x7f030c0aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:35.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.710+0000 7f030b7fe640 1 --2- 172.21.15.83:0/1365142974 >> v2:172.21.15.183:3300/0 conn(0x7f030c1586e0 0x7f030c0aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.710+0000 7f030b7fe640 1 -- 172.21.15.83:0/1365142974 >> v1:172.21.15.183:6789/0 conn(0x7f030c0cffd0 legacy=0x7f030c06da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.710+0000 7f030b7fe640 1 -- 172.21.15.83:0/1365142974 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f030c15b820 con 0x7f030c0aef80 2024-03-06T07:28:35.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.710+0000 7f030bfff640 1 --2- 172.21.15.83:0/1365142974 >> v2:172.21.15.183:3300/0 conn(0x7f030c1586e0 0x7f030c0aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:35.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.711+0000 7f030b7fe640 1 --2- 172.21.15.83:0/1365142974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f030c0aef80 0x7f030c0cfa90 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f030c070d10 tx=0x7f02fc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54913581405c1947 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.711+0000 7f030affd640 1 -- 172.21.15.83:0/1365142974 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02fc034070 con 0x7f030c0aef80 2024-03-06T07:28:35.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.711+0000 7f030affd640 1 -- 172.21.15.83:0/1365142974 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02fc030860 con 0x7f030c0aef80 2024-03-06T07:28:35.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.711+0000 7f030affd640 1 -- 172.21.15.83:0/1365142974 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02fc030c30 con 0x7f030c0aef80 2024-03-06T07:28:35.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.712+0000 7f031233a640 1 -- 172.21.15.83:0/1365142974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f030c0aef80 msgr2=0x7f030c0cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.712+0000 7f031233a640 1 --2- 172.21.15.83:0/1365142974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f030c0aef80 0x7f030c0cfa90 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f030c070d10 tx=0x7f02fc02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.712+0000 7f031233a640 1 -- 172.21.15.83:0/1365142974 shutdown_connections 2024-03-06T07:28:35.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.712+0000 7f031233a640 1 --2- 172.21.15.83:0/1365142974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f030c0aef80 0x7f030c0cfa90 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.712+0000 7f031233a640 1 --2- 172.21.15.83:0/1365142974 >> v2:172.21.15.183:3300/0 conn(0x7f030c1586e0 0x7f030c0aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.712+0000 7f031233a640 1 -- 172.21.15.83:0/1365142974 >> 172.21.15.83:0/1365142974 conn(0x7f030c06b190 msgr2=0x7f030c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.712+0000 7f031233a640 1 -- 172.21.15.83:0/1365142974 shutdown_connections 2024-03-06T07:28:35.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.712+0000 7f031233a640 1 -- 172.21.15.83:0/1365142974 wait complete. 2024-03-06T07:28:35.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.713+0000 7f031233a640 1 Processor -- start 2024-03-06T07:28:35.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.713+0000 7f031233a640 1 -- start start 2024-03-06T07:28:35.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.714+0000 7f031233a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f030c0aef80 0x7f030c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.714+0000 7f031233a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f030c0cffd0 0x7f030c14b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.714+0000 7f031233a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f030c1586e0 0x7f030c0b0950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.714+0000 7f031233a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f030c071350 con 0x7f030c0cffd0 2024-03-06T07:28:35.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.714+0000 7f031233a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f030c15bde0 con 0x7f030c0aef80 2024-03-06T07:28:35.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.714+0000 7f031233a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f030c0714d0 con 0x7f030c1586e0 2024-03-06T07:28:35.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.714+0000 7f030b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f030c0cffd0 0x7f030c14b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.714+0000 7f030b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f030c0cffd0 0x7f030c14b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52700/0 (socket says 172.21.15.83:52700) 2024-03-06T07:28:35.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.714+0000 7f030b7fe640 1 -- 172.21.15.83:0/2052888179 learned_addr learned my addr 172.21.15.83:0/2052888179 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:35.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.714+0000 7f030b7fe640 1 -- 172.21.15.83:0/2052888179 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f030c1586e0 msgr2=0x7f030c0b0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.714+0000 7f030b7fe640 1 --2- 172.21.15.83:0/2052888179 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f030c1586e0 0x7f030c0b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.714+0000 7f030bfff640 1 --2- 172.21.15.83:0/2052888179 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f030c0aef80 0x7f030c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.714+0000 7f030b7fe640 1 -- 172.21.15.83:0/2052888179 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f030c0aef80 msgr2=0x7f030c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.714+0000 7f030b7fe640 1 --2- 172.21.15.83:0/2052888179 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f030c0aef80 0x7f030c0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.714+0000 7f030b7fe640 1 -- 172.21.15.83:0/2052888179 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f030c14b810 con 0x7f030c0cffd0 2024-03-06T07:28:35.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.714+0000 7f030bfff640 1 --2- 172.21.15.83:0/2052888179 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f030c0aef80 0x7f030c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:35.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.714+0000 7f030b7fe640 1 --2- 172.21.15.83:0/2052888179 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f030c0cffd0 0x7f030c14b2d0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f02fc02b9e0 tx=0x7f02fc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.714+0000 7f0308ff9640 1 -- 172.21.15.83:0/2052888179 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02fc048070 con 0x7f030c0cffd0 2024-03-06T07:28:35.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.714+0000 7f0308ff9640 1 -- 172.21.15.83:0/2052888179 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02fc030660 con 0x7f030c0cffd0 2024-03-06T07:28:35.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.714+0000 7f0308ff9640 1 -- 172.21.15.83:0/2052888179 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02fc034040 con 0x7f030c0cffd0 2024-03-06T07:28:35.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.714+0000 7f031233a640 1 -- 172.21.15.83:0/2052888179 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f030c14a830 con 0x7f030c0cffd0 2024-03-06T07:28:35.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.714+0000 7f031233a640 1 -- 172.21.15.83:0/2052888179 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f030c14aad0 con 0x7f030c0cffd0 2024-03-06T07:28:35.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.715+0000 7f031233a640 1 -- 172.21.15.83:0/2052888179 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f030c0681e0 con 0x7f030c0cffd0 2024-03-06T07:28:35.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.716+0000 7f0308ff9640 1 -- 172.21.15.83:0/2052888179 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f02fc037070 con 0x7f030c0cffd0 2024-03-06T07:28:35.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.716+0000 7f0308ff9640 1 --2- 172.21.15.83:0/2052888179 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f02ec06b4d0 0x7f02ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.716+0000 7f0308ff9640 1 -- 172.21.15.83:0/2052888179 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f02fc0ac1e0 con 0x7f030c0cffd0 2024-03-06T07:28:35.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.716+0000 7f030bfff640 1 --2- 172.21.15.83:0/2052888179 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f02ec06b4d0 0x7f02ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.717+0000 7f030bfff640 1 --2- 172.21.15.83:0/2052888179 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f02ec06b4d0 0x7f02ec06d990 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f02f4006fd0 tx=0x7f02f4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.721+0000 7f0308ff9640 1 -- 172.21.15.83:0/2052888179 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f030c0681e0 con 0x7f030c0cffd0 2024-03-06T07:28:35.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:35.926+0000 7f031233a640 1 -- 172.21.15.83:0/2052888179 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f030c144c80 con 0x7f030c0cffd0 2024-03-06T07:28:36.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.592+0000 7f0308ff9640 1 -- 172.21.15.83:0/2052888179 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v68) v1 ==== 104+0+0 (secure 0 0 0) 0x7f030c144c80 con 0x7f030c0cffd0 2024-03-06T07:28:36.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.595+0000 7f031233a640 1 -- 172.21.15.83:0/2052888179 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f02ec06b4d0 msgr2=0x7f02ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.595+0000 7f031233a640 1 --2- 172.21.15.83:0/2052888179 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f02ec06b4d0 0x7f02ec06d990 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f02f4006fd0 tx=0x7f02f4008040 comp rx=0 tx=0).stop 2024-03-06T07:28:36.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.595+0000 7f031233a640 1 -- 172.21.15.83:0/2052888179 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f030c0cffd0 msgr2=0x7f030c14b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.595+0000 7f031233a640 1 --2- 172.21.15.83:0/2052888179 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f030c0cffd0 0x7f030c14b2d0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f02fc02b9e0 tx=0x7f02fc004730 comp rx=0 tx=0).stop 2024-03-06T07:28:36.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.595+0000 7f031233a640 1 -- 172.21.15.83:0/2052888179 shutdown_connections 2024-03-06T07:28:36.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.595+0000 7f031233a640 1 --2- 172.21.15.83:0/2052888179 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f02ec06b4d0 0x7f02ec06d990 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.595+0000 7f031233a640 1 --2- 172.21.15.83:0/2052888179 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f030c1586e0 0x7f030c0b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.595+0000 7f031233a640 1 --2- 172.21.15.83:0/2052888179 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f030c0cffd0 0x7f030c14b2d0 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.595+0000 7f031233a640 1 --2- 172.21.15.83:0/2052888179 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f030c0aef80 0x7f030c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.595+0000 7f031233a640 1 -- 172.21.15.83:0/2052888179 >> 172.21.15.83:0/2052888179 conn(0x7f030c06b190 msgr2=0x7f030c15a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:36.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.595+0000 7f031233a640 1 -- 172.21.15.83:0/2052888179 shutdown_connections 2024-03-06T07:28:36.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.595+0000 7f031233a640 1 -- 172.21.15.83:0/2052888179 wait complete. 2024-03-06T07:28:36.609 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:28:36.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.753+0000 7f74b19c9640 1 Processor -- start 2024-03-06T07:28:36.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.753+0000 7f74b19c9640 1 -- start start 2024-03-06T07:28:36.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.753+0000 7f74b19c9640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f74ac0d4630 0x7f74ac0d22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.753+0000 7f74b19c9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f74ac0d2820 0x7f74ac0cb190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.753+0000 7f74b19c9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74ac15e380 con 0x7f74ac15dfa0 2024-03-06T07:28:36.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.754+0000 7f74b19c9640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74ac0d2100 con 0x7f74ac0d4630 2024-03-06T07:28:36.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.754+0000 7f74b19c9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74ac0cf470 con 0x7f74ac0d2820 2024-03-06T07:28:36.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.754+0000 7f74ab7fe640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f74ac0d2820 0x7f74ac0cb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.754+0000 7f74a3fff640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f74ac0d4630 0x7f74ac0d22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.754+0000 7f74ab7fe640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f74ac0d2820 0x7f74ac0cb190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:33504/0 (socket says 172.21.15.83:33504) 2024-03-06T07:28:36.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.754+0000 7f74ab7fe640 1 -- 172.21.15.83:0/1631556653 learned_addr learned my addr 172.21.15.83:0/1631556653 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:36.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.754+0000 7f74aaffd640 1 --1- 172.21.15.83:0/1631556653 >> v1:172.21.15.183:6789/0 conn(0x7f74ac15dfa0 0x7f74ac0d18f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:38056/0 (socket says 172.21.15.83:38056) 2024-03-06T07:28:36.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.754+0000 7f74ab7fe640 1 -- 172.21.15.83:0/1631556653 >> v2:172.21.15.183:3300/0 conn(0x7f74ac0d4630 msgr2=0x7f74ac0d22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:36.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.754+0000 7f74ab7fe640 1 --2- 172.21.15.83:0/1631556653 >> v2:172.21.15.183:3300/0 conn(0x7f74ac0d4630 0x7f74ac0d22e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.754+0000 7f74ab7fe640 1 -- 172.21.15.83:0/1631556653 >> v1:172.21.15.183:6789/0 conn(0x7f74ac15dfa0 legacy=0x7f74ac0d18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.754+0000 7f74ab7fe640 1 -- 172.21.15.83:0/1631556653 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74ac0cf750 con 0x7f74ac0d2820 2024-03-06T07:28:36.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.754+0000 7f74a3fff640 1 --2- 172.21.15.83:0/1631556653 >> v2:172.21.15.183:3300/0 conn(0x7f74ac0d4630 0x7f74ac0d22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:36.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.755+0000 7f74ab7fe640 1 --2- 172.21.15.83:0/1631556653 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f74ac0d2820 0x7f74ac0cb190 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f74a4009e50 tx=0x7f74a402bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50b14d028da9e397 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.755+0000 7f74aa7fc640 1 -- 172.21.15.83:0/1631556653 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a402ca70 con 0x7f74ac0d2820 2024-03-06T07:28:36.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.755+0000 7f74aa7fc640 1 -- 172.21.15.83:0/1631556653 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74a402cc10 con 0x7f74ac0d2820 2024-03-06T07:28:36.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.755+0000 7f74aa7fc640 1 -- 172.21.15.83:0/1631556653 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a40343f0 con 0x7f74ac0d2820 2024-03-06T07:28:36.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.756+0000 7f74b19c9640 1 -- 172.21.15.83:0/1631556653 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f74ac0d2820 msgr2=0x7f74ac0cb190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.756+0000 7f74b19c9640 1 --2- 172.21.15.83:0/1631556653 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f74ac0d2820 0x7f74ac0cb190 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f74a4009e50 tx=0x7f74a402bed0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.756+0000 7f74b19c9640 1 -- 172.21.15.83:0/1631556653 shutdown_connections 2024-03-06T07:28:36.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.756+0000 7f74b19c9640 1 --2- 172.21.15.83:0/1631556653 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f74ac0d2820 0x7f74ac0cb190 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.756+0000 7f74b19c9640 1 --2- 172.21.15.83:0/1631556653 >> v2:172.21.15.183:3300/0 conn(0x7f74ac0d4630 0x7f74ac0d22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.756+0000 7f74b19c9640 1 -- 172.21.15.83:0/1631556653 >> 172.21.15.83:0/1631556653 conn(0x7f74ac09d000 msgr2=0x7f74ac0d3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:36.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.756+0000 7f74b19c9640 1 -- 172.21.15.83:0/1631556653 shutdown_connections 2024-03-06T07:28:36.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.756+0000 7f74b19c9640 1 -- 172.21.15.83:0/1631556653 wait complete. 2024-03-06T07:28:36.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.757+0000 7f74b19c9640 1 Processor -- start 2024-03-06T07:28:36.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.757+0000 7f74b19c9640 1 -- start start 2024-03-06T07:28:36.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.758+0000 7f74b19c9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f74ac0d2820 0x7f74ac14ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.758+0000 7f74b19c9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74ac0d4630 0x7f74ac14c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.758+0000 7f74b19c9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f74ac15dfa0 0x7f74ac14d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.758+0000 7f74b19c9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74ac0cb850 con 0x7f74ac0d4630 2024-03-06T07:28:36.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.758+0000 7f74b19c9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74ac0cb6d0 con 0x7f74ac0d2820 2024-03-06T07:28:36.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.758+0000 7f74b19c9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74ac0cb9d0 con 0x7f74ac15dfa0 2024-03-06T07:28:36.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.758+0000 7f74a3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74ac0d4630 0x7f74ac14c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.758+0000 7f74a3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74ac0d4630 0x7f74ac14c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46130/0 (socket says 172.21.15.83:46130) 2024-03-06T07:28:36.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.758+0000 7f74a3fff640 1 -- 172.21.15.83:0/2438328518 learned_addr learned my addr 172.21.15.83:0/2438328518 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:36.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.758+0000 7f74ab7fe640 1 --2- 172.21.15.83:0/2438328518 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f74ac15dfa0 0x7f74ac14d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.758+0000 7f74a3fff640 1 -- 172.21.15.83:0/2438328518 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f74ac15dfa0 msgr2=0x7f74ac14d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.758+0000 7f74a3fff640 1 --2- 172.21.15.83:0/2438328518 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f74ac15dfa0 0x7f74ac14d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.758+0000 7f74a3fff640 1 -- 172.21.15.83:0/2438328518 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f74ac0d2820 msgr2=0x7f74ac14ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.758+0000 7f74aaffd640 1 --2- 172.21.15.83:0/2438328518 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f74ac0d2820 0x7f74ac14ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.758+0000 7f74a3fff640 1 --2- 172.21.15.83:0/2438328518 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f74ac0d2820 0x7f74ac14ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.758+0000 7f74a3fff640 1 -- 172.21.15.83:0/2438328518 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74ac14cda0 con 0x7f74ac0d4630 2024-03-06T07:28:36.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.758+0000 7f74aaffd640 1 --2- 172.21.15.83:0/2438328518 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f74ac0d2820 0x7f74ac14ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:36.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.758+0000 7f74a3fff640 1 --2- 172.21.15.83:0/2438328518 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74ac0d4630 0x7f74ac14c860 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f749400e9e0 tx=0x7f749400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.759+0000 7f74a8ff9640 1 -- 172.21.15.83:0/2438328518 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7494017070 con 0x7f74ac0d4630 2024-03-06T07:28:36.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.759+0000 7f74a8ff9640 1 -- 172.21.15.83:0/2438328518 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f749400ca30 con 0x7f74ac0d4630 2024-03-06T07:28:36.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.759+0000 7f74b19c9640 1 -- 172.21.15.83:0/2438328518 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74ac14d980 con 0x7f74ac0d4630 2024-03-06T07:28:36.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.759+0000 7f74a8ff9640 1 -- 172.21.15.83:0/2438328518 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f749400ce20 con 0x7f74ac0d4630 2024-03-06T07:28:36.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.759+0000 7f74b19c9640 1 -- 172.21.15.83:0/2438328518 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74ac14df40 con 0x7f74ac0d4630 2024-03-06T07:28:36.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.760+0000 7f74b19c9640 1 -- 172.21.15.83:0/2438328518 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74ac04f330 con 0x7f74ac0d4630 2024-03-06T07:28:36.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.760+0000 7f74a8ff9640 1 -- 172.21.15.83:0/2438328518 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f749401a070 con 0x7f74ac0d4630 2024-03-06T07:28:36.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.760+0000 7f74a8ff9640 1 --2- 172.21.15.83:0/2438328518 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f748c06b4d0 0x7f748c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.760+0000 7f74a8ff9640 1 -- 172.21.15.83:0/2438328518 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f7494089ef0 con 0x7f74ac0d4630 2024-03-06T07:28:36.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.760+0000 7f74aaffd640 1 --2- 172.21.15.83:0/2438328518 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f748c06b4d0 0x7f748c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.761+0000 7f74aaffd640 1 --2- 172.21.15.83:0/2438328518 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f748c06b4d0 0x7f748c06d990 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f749c000f60 tx=0x7f749c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.765+0000 7f74a8ff9640 1 -- 172.21.15.83:0/2438328518 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7494058c30 con 0x7f74ac0d4630 2024-03-06T07:28:36.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:36.939+0000 7f74b19c9640 1 -- 172.21.15.83:0/2438328518 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f74ac06c1f0 con 0x7f74ac0d4630 2024-03-06T07:28:37.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.593+0000 7f74a8ff9640 1 -- 172.21.15.83:0/2438328518 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v118) v1 ==== 181+0+0 (secure 0 0 0) 0x7f7494058530 con 0x7f74ac0d4630 2024-03-06T07:28:37.595 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:28:37.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.597+0000 7f74b19c9640 1 -- 172.21.15.83:0/2438328518 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f748c06b4d0 msgr2=0x7f748c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.597+0000 7f74b19c9640 1 --2- 172.21.15.83:0/2438328518 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f748c06b4d0 0x7f748c06d990 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f749c000f60 tx=0x7f749c009040 comp rx=0 tx=0).stop 2024-03-06T07:28:37.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.597+0000 7f74b19c9640 1 -- 172.21.15.83:0/2438328518 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74ac0d4630 msgr2=0x7f74ac14c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.597+0000 7f74b19c9640 1 --2- 172.21.15.83:0/2438328518 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74ac0d4630 0x7f74ac14c860 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f749400e9e0 tx=0x7f749400eea0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.597+0000 7f74b19c9640 1 -- 172.21.15.83:0/2438328518 shutdown_connections 2024-03-06T07:28:37.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.597+0000 7f74b19c9640 1 --2- 172.21.15.83:0/2438328518 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f748c06b4d0 0x7f748c06d990 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.597+0000 7f74b19c9640 1 --2- 172.21.15.83:0/2438328518 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f74ac15dfa0 0x7f74ac14d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.597+0000 7f74b19c9640 1 --2- 172.21.15.83:0/2438328518 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f74ac0d4630 0x7f74ac14c860 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.597+0000 7f74b19c9640 1 --2- 172.21.15.83:0/2438328518 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f74ac0d2820 0x7f74ac14ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.597+0000 7f74b19c9640 1 -- 172.21.15.83:0/2438328518 >> 172.21.15.83:0/2438328518 conn(0x7f74ac09d000 msgr2=0x7f74ac15d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:37.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.597+0000 7f74b19c9640 1 -- 172.21.15.83:0/2438328518 shutdown_connections 2024-03-06T07:28:37.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.597+0000 7f74b19c9640 1 -- 172.21.15.83:0/2438328518 wait complete. 2024-03-06T07:28:37.613 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:28:37.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.719+0000 7f9ba4165640 1 Processor -- start 2024-03-06T07:28:37.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.719+0000 7f9ba4165640 1 -- start start 2024-03-06T07:28:37.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.720+0000 7f9ba4165640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9b9c0708b0 0x7f9b9c070d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.720+0000 7f9ba4165640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b9c15afe0 con 0x7f9b9c0ce0a0 2024-03-06T07:28:37.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.720+0000 7f9ba4165640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b9c0cd000 con 0x7f9b9c15ac00 2024-03-06T07:28:37.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.720+0000 7f9ba4165640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b9c071250 con 0x7f9b9c0708b0 2024-03-06T07:28:37.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.720+0000 7f9ba1700640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9b9c0708b0 0x7f9b9c070d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.720+0000 7f9ba2702640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f9b9c0ce0a0 0x7f9b9c0ce550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:34870/0 (socket says 172.21.15.83:34870) 2024-03-06T07:28:37.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.720+0000 7f9ba2702640 1 -- 172.21.15.83:0/3856535485 learned_addr learned my addr 172.21.15.83:0/3856535485 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:28:37.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.720+0000 7f9ba1700640 1 -- 172.21.15.83:0/3856535485 >> v1:172.21.15.183:6789/0 conn(0x7f9b9c15ac00 legacy=0x7f9b9c06c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:37.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.720+0000 7f9ba1700640 1 -- 172.21.15.83:0/3856535485 >> v1:172.21.15.83:6789/0 conn(0x7f9b9c0ce0a0 legacy=0x7f9b9c0ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.720+0000 7f9ba1700640 1 -- 172.21.15.83:0/3856535485 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b9c0cd2e0 con 0x7f9b9c0708b0 2024-03-06T07:28:37.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.720+0000 7f9ba0eff640 1 -- 172.21.15.83:0/3856535485 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4020434233 0 0) 0x7f9b9c15afe0 con 0x7f9b9c0ce0a0 2024-03-06T07:28:37.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.721+0000 7f9ba1700640 1 --2- 172.21.15.83:0/3856535485 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b9c0708b0 0x7f9b9c070d10 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c0097a0 tx=0x7f9b8c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63926b0d351c4723 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.721+0000 7f9ba0eff640 1 -- 172.21.15.83:0/3856535485 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c0315a0 con 0x7f9b9c0708b0 2024-03-06T07:28:37.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.721+0000 7f9ba0eff640 1 -- 172.21.15.83:0/3856535485 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b8c037070 con 0x7f9b9c0708b0 2024-03-06T07:28:37.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.721+0000 7f9ba0eff640 1 -- 172.21.15.83:0/3856535485 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c031960 con 0x7f9b9c0708b0 2024-03-06T07:28:37.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.721+0000 7f9ba4165640 1 -- 172.21.15.83:0/3856535485 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b9c0708b0 msgr2=0x7f9b9c070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.721+0000 7f9ba4165640 1 --2- 172.21.15.83:0/3856535485 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b9c0708b0 0x7f9b9c070d10 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c0097a0 tx=0x7f9b8c02b840 comp rx=0 tx=0).stop 2024-03-06T07:28:37.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.721+0000 7f9ba4165640 1 -- 172.21.15.83:0/3856535485 shutdown_connections 2024-03-06T07:28:37.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.721+0000 7f9ba4165640 1 --2- 172.21.15.83:0/3856535485 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b9c0708b0 0x7f9b9c070d10 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.721+0000 7f9ba4165640 1 -- 172.21.15.83:0/3856535485 >> 172.21.15.83:0/3856535485 conn(0x7f9b9c0b1650 msgr2=0x7f9b9c0b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:37.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.722+0000 7f9ba4165640 1 -- 172.21.15.83:0/3856535485 shutdown_connections 2024-03-06T07:28:37.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.722+0000 7f9ba4165640 1 -- 172.21.15.83:0/3856535485 wait complete. 2024-03-06T07:28:37.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.723+0000 7f9ba4165640 1 Processor -- start 2024-03-06T07:28:37.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.723+0000 7f9ba4165640 1 -- start start 2024-03-06T07:28:37.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.723+0000 7f9ba4165640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9b9c0708b0 0x7f9b9c149400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.723+0000 7f9ba4165640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b9c0ce0a0 0x7f9b9c15cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.723+0000 7f9ba4165640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9b9c15ac00 0x7f9b9c15d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.723+0000 7f9ba4165640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9c0cd8a0 con 0x7f9b9c0ce0a0 2024-03-06T07:28:37.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.723+0000 7f9ba4165640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9c0cdc10 con 0x7f9b9c15ac00 2024-03-06T07:28:37.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.723+0000 7f9ba4165640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9c0cf5d0 con 0x7f9b9c0708b0 2024-03-06T07:28:37.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.723+0000 7f9ba1700640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b9c0ce0a0 0x7f9b9c15cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.723+0000 7f9ba1700640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b9c0ce0a0 0x7f9b9c15cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46146/0 (socket says 172.21.15.83:46146) 2024-03-06T07:28:37.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.723+0000 7f9ba1700640 1 -- 172.21.15.83:0/3029597948 learned_addr learned my addr 172.21.15.83:0/3029597948 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:37.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.724+0000 7f9ba1f01640 1 --2- 172.21.15.83:0/3029597948 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9b9c0708b0 0x7f9b9c149400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.724+0000 7f9ba1700640 1 -- 172.21.15.83:0/3029597948 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9b9c0708b0 msgr2=0x7f9b9c149400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.724+0000 7f9ba1700640 1 --2- 172.21.15.83:0/3029597948 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9b9c0708b0 0x7f9b9c149400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.724+0000 7f9ba1700640 1 -- 172.21.15.83:0/3029597948 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9b9c15ac00 msgr2=0x7f9b9c15d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.724+0000 7f9ba1700640 1 --2- 172.21.15.83:0/3029597948 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9b9c15ac00 0x7f9b9c15d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.724+0000 7f9ba1700640 1 -- 172.21.15.83:0/3029597948 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b9c146b90 con 0x7f9b9c0ce0a0 2024-03-06T07:28:37.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.724+0000 7f9ba1700640 1 --2- 172.21.15.83:0/3029597948 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b9c0ce0a0 0x7f9b9c15cff0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c0097a0 tx=0x7f9b8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.724+0000 7f9b927fc640 1 -- 172.21.15.83:0/3029597948 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c004030 con 0x7f9b9c0ce0a0 2024-03-06T07:28:37.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.724+0000 7f9b927fc640 1 -- 172.21.15.83:0/3029597948 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b8c0041f0 con 0x7f9b9c0ce0a0 2024-03-06T07:28:37.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.724+0000 7f9b927fc640 1 -- 172.21.15.83:0/3029597948 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c032640 con 0x7f9b9c0ce0a0 2024-03-06T07:28:37.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.724+0000 7f9ba4165640 1 -- 172.21.15.83:0/3029597948 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b9c0b2620 con 0x7f9b9c0ce0a0 2024-03-06T07:28:37.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.724+0000 7f9ba4165640 1 -- 172.21.15.83:0/3029597948 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b9c0b2be0 con 0x7f9b9c0ce0a0 2024-03-06T07:28:37.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.725+0000 7f9ba4165640 1 -- 172.21.15.83:0/3029597948 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b9c0687d0 con 0x7f9b9c0ce0a0 2024-03-06T07:28:37.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.726+0000 7f9b927fc640 1 -- 172.21.15.83:0/3029597948 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9b8c0327e0 con 0x7f9b9c0ce0a0 2024-03-06T07:28:37.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.726+0000 7f9b927fc640 1 --2- 172.21.15.83:0/3029597948 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9b8006b4d0 0x7f9b8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.726+0000 7f9b927fc640 1 -- 172.21.15.83:0/3029597948 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5587+0+0 (secure 0 0 0) 0x7f9b8c02f030 con 0x7f9b9c0ce0a0 2024-03-06T07:28:37.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.726+0000 7f9ba1f01640 1 --2- 172.21.15.83:0/3029597948 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9b8006b4d0 0x7f9b8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.726+0000 7f9ba1f01640 1 --2- 172.21.15.83:0/3029597948 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9b8006b4d0 0x7f9b8006d990 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f9b980015d0 tx=0x7f9b98009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.730+0000 7f9b927fc640 1 -- 172.21.15.83:0/3029597948 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9b8c041070 con 0x7f9b9c0ce0a0 2024-03-06T07:28:37.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:37.903+0000 7f9ba4165640 1 -- 172.21.15.83:0/3029597948 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9b9c156400 con 0x7f9b9c0ce0a0 2024-03-06T07:28:38.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.595+0000 7f9b927fc640 1 -- 172.21.15.83:0/3029597948 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v119) v1 ==== 169+0+0 (secure 0 0 0) 0x7f9b8c07b400 con 0x7f9b9c0ce0a0 2024-03-06T07:28:38.597 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'cephfs_data' removed 2024-03-06T07:28:38.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.598+0000 7f9ba4165640 1 -- 172.21.15.83:0/3029597948 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9b8006b4d0 msgr2=0x7f9b8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.598+0000 7f9ba4165640 1 --2- 172.21.15.83:0/3029597948 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9b8006b4d0 0x7f9b8006d990 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f9b980015d0 tx=0x7f9b98009040 comp rx=0 tx=0).stop 2024-03-06T07:28:38.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.598+0000 7f9ba4165640 1 -- 172.21.15.83:0/3029597948 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b9c0ce0a0 msgr2=0x7f9b9c15cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.598+0000 7f9ba4165640 1 --2- 172.21.15.83:0/3029597948 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b9c0ce0a0 0x7f9b9c15cff0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c0097a0 tx=0x7f9b8c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.598+0000 7f9ba4165640 1 -- 172.21.15.83:0/3029597948 shutdown_connections 2024-03-06T07:28:38.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.598+0000 7f9ba4165640 1 --2- 172.21.15.83:0/3029597948 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9b8006b4d0 0x7f9b8006d990 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.598+0000 7f9ba4165640 1 --2- 172.21.15.83:0/3029597948 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9b9c15ac00 0x7f9b9c15d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.598+0000 7f9ba4165640 1 --2- 172.21.15.83:0/3029597948 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b9c0ce0a0 0x7f9b9c15cff0 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.598+0000 7f9ba4165640 1 --2- 172.21.15.83:0/3029597948 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9b9c0708b0 0x7f9b9c149400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.598+0000 7f9ba4165640 1 -- 172.21.15.83:0/3029597948 >> 172.21.15.83:0/3029597948 conn(0x7f9b9c0b1650 msgr2=0x7f9b9c0b4720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:38.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.598+0000 7f9ba4165640 1 -- 172.21.15.83:0/3029597948 shutdown_connections 2024-03-06T07:28:38.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.598+0000 7f9ba4165640 1 -- 172.21.15.83:0/3029597948 wait complete. 2024-03-06T07:28:38.613 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:28:38.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.714+0000 7fa89bceb640 1 Processor -- start 2024-03-06T07:28:38.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.714+0000 7fa89bceb640 1 -- start start 2024-03-06T07:28:38.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.715+0000 7fa89bceb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa8941614d0 0x7fa8940d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.715+0000 7fa89bceb640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fa8940d1910 0x7fa8940d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.715+0000 7fa89bceb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8940d22e0 0x7fa8940d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.715+0000 7fa89bceb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8940d2750 con 0x7fa8941614d0 2024-03-06T07:28:38.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.715+0000 7fa89bceb640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8941618b0 con 0x7fa8940d1910 2024-03-06T07:28:38.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.715+0000 7fa89bceb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8940cf400 con 0x7fa8940d22e0 2024-03-06T07:28:38.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.715+0000 7fa899a87640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa8941614d0 0x7fa8940d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.715+0000 7fa899a87640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa8941614d0 0x7fa8940d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46158/0 (socket says 172.21.15.83:46158) 2024-03-06T07:28:38.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.715+0000 7fa899a87640 1 -- 172.21.15.83:0/1298207307 learned_addr learned my addr 172.21.15.83:0/1298207307 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:38.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.716+0000 7fa899a87640 1 -- 172.21.15.83:0/1298207307 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8940d22e0 msgr2=0x7fa8940d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:38.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.716+0000 7fa89a288640 1 --2- 172.21.15.83:0/1298207307 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8940d22e0 0x7fa8940d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.716+0000 7fa899286640 1 --2- 172.21.15.83:0/1298207307 >> v2:172.21.15.183:3300/0 conn(0x7fa8940d1910 0x7fa8940d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.716+0000 7fa899a87640 1 --2- 172.21.15.83:0/1298207307 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8940d22e0 0x7fa8940d06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.716+0000 7fa899a87640 1 -- 172.21.15.83:0/1298207307 >> v2:172.21.15.183:3300/0 conn(0x7fa8940d1910 msgr2=0x7fa8940d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:38.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.716+0000 7fa899a87640 1 --2- 172.21.15.83:0/1298207307 >> v2:172.21.15.183:3300/0 conn(0x7fa8940d1910 0x7fa8940d1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.716+0000 7fa899a87640 1 -- 172.21.15.83:0/1298207307 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8940cf6a0 con 0x7fa8941614d0 2024-03-06T07:28:38.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.716+0000 7fa899a87640 1 --2- 172.21.15.83:0/1298207307 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8941614d0 0x7fa8940d4860 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7fa8940ce550 tx=0x7fa89002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1efd5af09af04e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.717+0000 7fa898a85640 1 -- 172.21.15.83:0/1298207307 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa89002ca70 con 0x7fa8941614d0 2024-03-06T07:28:38.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.717+0000 7fa898a85640 1 -- 172.21.15.83:0/1298207307 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa89002cc10 con 0x7fa8941614d0 2024-03-06T07:28:38.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.717+0000 7fa898a85640 1 -- 172.21.15.83:0/1298207307 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8900343f0 con 0x7fa8941614d0 2024-03-06T07:28:38.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.717+0000 7fa89bceb640 1 -- 172.21.15.83:0/1298207307 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8941614d0 msgr2=0x7fa8940d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.717+0000 7fa89bceb640 1 --2- 172.21.15.83:0/1298207307 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8941614d0 0x7fa8940d4860 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7fa8940ce550 tx=0x7fa89002bed0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.717+0000 7fa89bceb640 1 -- 172.21.15.83:0/1298207307 shutdown_connections 2024-03-06T07:28:38.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.717+0000 7fa89bceb640 1 --2- 172.21.15.83:0/1298207307 >> v2:172.21.15.183:3300/0 conn(0x7fa8940d1910 0x7fa8940d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.717+0000 7fa89bceb640 1 --2- 172.21.15.83:0/1298207307 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8940d22e0 0x7fa8940d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.717+0000 7fa89bceb640 1 --2- 172.21.15.83:0/1298207307 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8941614d0 0x7fa8940d4860 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.717+0000 7fa89bceb640 1 -- 172.21.15.83:0/1298207307 >> 172.21.15.83:0/1298207307 conn(0x7fa8940d3d40 msgr2=0x7fa8940d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:38.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.717+0000 7fa89bceb640 1 -- 172.21.15.83:0/1298207307 shutdown_connections 2024-03-06T07:28:38.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.717+0000 7fa89bceb640 1 -- 172.21.15.83:0/1298207307 wait complete. 2024-03-06T07:28:38.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.718+0000 7fa89bceb640 1 Processor -- start 2024-03-06T07:28:38.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.718+0000 7fa89bceb640 1 -- start start 2024-03-06T07:28:38.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.719+0000 7fa89bceb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8940d1910 0x7fa8940b1f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.719+0000 7fa89bceb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8940d22e0 0x7fa8940cc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.719+0000 7fa89bceb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8941614d0 0x7fa894158140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.719+0000 7fa89bceb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8940b0910 con 0x7fa8940d22e0 2024-03-06T07:28:38.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.719+0000 7fa89bceb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8940cfc60 con 0x7fa8940d1910 2024-03-06T07:28:38.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.719+0000 7fa89bceb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8940d0be0 con 0x7fa8941614d0 2024-03-06T07:28:38.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.719+0000 7fa899286640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8940d22e0 0x7fa8940cc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.719+0000 7fa899286640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8940d22e0 0x7fa8940cc5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46160/0 (socket says 172.21.15.83:46160) 2024-03-06T07:28:38.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.719+0000 7fa899286640 1 -- 172.21.15.83:0/453139411 learned_addr learned my addr 172.21.15.83:0/453139411 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:38.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.719+0000 7fa899a87640 1 --2- 172.21.15.83:0/453139411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8940d1910 0x7fa8940b1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.719+0000 7fa89a288640 1 --2- 172.21.15.83:0/453139411 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8941614d0 0x7fa894158140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.719+0000 7fa899286640 1 -- 172.21.15.83:0/453139411 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8941614d0 msgr2=0x7fa894158140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.719+0000 7fa899286640 1 --2- 172.21.15.83:0/453139411 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8941614d0 0x7fa894158140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.719+0000 7fa899286640 1 -- 172.21.15.83:0/453139411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8940d1910 msgr2=0x7fa8940b1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.719+0000 7fa899286640 1 --2- 172.21.15.83:0/453139411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8940d1910 0x7fa8940b1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.720+0000 7fa899286640 1 -- 172.21.15.83:0/453139411 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8940c85f0 con 0x7fa8940d22e0 2024-03-06T07:28:38.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.720+0000 7fa899286640 1 --2- 172.21.15.83:0/453139411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8940d22e0 0x7fa8940cc5d0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7fa88400c820 tx=0x7fa88400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.720+0000 7fa88a7fc640 1 -- 172.21.15.83:0/453139411 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa884015070 con 0x7fa8940d22e0 2024-03-06T07:28:38.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.720+0000 7fa88a7fc640 1 -- 172.21.15.83:0/453139411 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa884004240 con 0x7fa8940d22e0 2024-03-06T07:28:38.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.720+0000 7fa89bceb640 1 -- 172.21.15.83:0/453139411 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8940c88d0 con 0x7fa8940d22e0 2024-03-06T07:28:38.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.720+0000 7fa88a7fc640 1 -- 172.21.15.83:0/453139411 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa88400ce10 con 0x7fa8940d22e0 2024-03-06T07:28:38.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.720+0000 7fa89bceb640 1 -- 172.21.15.83:0/453139411 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8940c8e90 con 0x7fa8940d22e0 2024-03-06T07:28:38.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.721+0000 7fa89bceb640 1 -- 172.21.15.83:0/453139411 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa894068ff0 con 0x7fa8940d22e0 2024-03-06T07:28:38.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.722+0000 7fa88a7fc640 1 -- 172.21.15.83:0/453139411 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa884007830 con 0x7fa8940d22e0 2024-03-06T07:28:38.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.722+0000 7fa88a7fc640 1 --2- 172.21.15.83:0/453139411 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa87806b500 0x7fa87806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.722+0000 7fa899a87640 1 --2- 172.21.15.83:0/453139411 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa87806b500 0x7fa87806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.722+0000 7fa88a7fc640 1 -- 172.21.15.83:0/453139411 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fa884089e50 con 0x7fa8940d22e0 2024-03-06T07:28:38.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.722+0000 7fa899a87640 1 --2- 172.21.15.83:0/453139411 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa87806b500 0x7fa87806d9c0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fa890035000 tx=0x7fa890030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.726+0000 7fa88a7fc640 1 -- 172.21.15.83:0/453139411 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa884058840 con 0x7fa8940d22e0 2024-03-06T07:28:38.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.908+0000 7fa89bceb640 1 -- 172.21.15.83:0/453139411 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fa8940b3de0 con 0x7fa8940d22e0 2024-03-06T07:28:38.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.912+0000 7fa88a7fc640 1 -- 172.21.15.83:0/453139411 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v31) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa88405c8e0 con 0x7fa8940d22e0 2024-03-06T07:28:38.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.914+0000 7fa89bceb640 1 -- 172.21.15.83:0/453139411 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa87806b500 msgr2=0x7fa87806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.914+0000 7fa89bceb640 1 --2- 172.21.15.83:0/453139411 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa87806b500 0x7fa87806d9c0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fa890035000 tx=0x7fa890030000 comp rx=0 tx=0).stop 2024-03-06T07:28:38.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.914+0000 7fa89bceb640 1 -- 172.21.15.83:0/453139411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8940d22e0 msgr2=0x7fa8940cc5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.914+0000 7fa89bceb640 1 --2- 172.21.15.83:0/453139411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8940d22e0 0x7fa8940cc5d0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7fa88400c820 tx=0x7fa88400cce0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.914+0000 7fa89bceb640 1 -- 172.21.15.83:0/453139411 shutdown_connections 2024-03-06T07:28:38.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.914+0000 7fa89bceb640 1 --2- 172.21.15.83:0/453139411 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa87806b500 0x7fa87806d9c0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.914+0000 7fa89bceb640 1 --2- 172.21.15.83:0/453139411 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8941614d0 0x7fa894158140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.914+0000 7fa89bceb640 1 --2- 172.21.15.83:0/453139411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8940d22e0 0x7fa8940cc5d0 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.914+0000 7fa89bceb640 1 --2- 172.21.15.83:0/453139411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8940d1910 0x7fa8940b1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.914+0000 7fa89bceb640 1 -- 172.21.15.83:0/453139411 >> 172.21.15.83:0/453139411 conn(0x7fa8940d3d40 msgr2=0x7fa89406b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:38.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.914+0000 7fa89bceb640 1 -- 172.21.15.83:0/453139411 shutdown_connections 2024-03-06T07:28:38.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:38.914+0000 7fa89bceb640 1 -- 172.21.15.83:0/453139411 wait complete. 2024-03-06T07:28:38.927 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration' 2024-03-06T07:28:39.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.031+0000 7f6e39864640 1 Processor -- start 2024-03-06T07:28:39.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.031+0000 7f6e39864640 1 -- start start 2024-03-06T07:28:39.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.032+0000 7f6e39864640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f6e340b3060 0x7f6e3415e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.032+0000 7f6e39864640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6e3415bf30 0x7f6e3415ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.032+0000 7f6e39864640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e3415f2d0 con 0x7f6e3413dc10 2024-03-06T07:28:39.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.032+0000 7f6e39864640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e340cf3f0 con 0x7f6e340b3060 2024-03-06T07:28:39.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.032+0000 7f6e39864640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3415b210 con 0x7f6e3415bf30 2024-03-06T07:28:39.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.032+0000 7f6e337fe640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6e3415bf30 0x7f6e3415ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.032+0000 7f6e327fc640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f6e340b3060 0x7f6e3415e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.032+0000 7f6e337fe640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6e3415bf30 0x7f6e3415ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:33556/0 (socket says 172.21.15.83:33556) 2024-03-06T07:28:39.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.032+0000 7f6e337fe640 1 -- 172.21.15.83:0/3562442944 learned_addr learned my addr 172.21.15.83:0/3562442944 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:39.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.033+0000 7f6e327fc640 1 -- 172.21.15.83:0/3562442944 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6e3415bf30 msgr2=0x7f6e3415ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:39.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.033+0000 7f6e327fc640 1 --2- 172.21.15.83:0/3562442944 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6e3415bf30 0x7f6e3415ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.033+0000 7f6e327fc640 1 -- 172.21.15.83:0/3562442944 >> v1:172.21.15.183:6789/0 conn(0x7f6e3413dc10 legacy=0x7f6e340b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.033+0000 7f6e327fc640 1 -- 172.21.15.83:0/3562442944 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e3415f580 con 0x7f6e340b3060 2024-03-06T07:28:39.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.033+0000 7f6e337fe640 1 --2- 172.21.15.83:0/3562442944 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6e3415bf30 0x7f6e3415ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:39.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.033+0000 7f6e327fc640 1 --2- 172.21.15.83:0/3562442944 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6e340b3060 0x7f6e3415e850 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f6e280097a0 tx=0x7f6e2802b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83069cedc2393718 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.034+0000 7f6e31ffb640 1 -- 172.21.15.83:0/3562442944 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e28031780 con 0x7f6e340b3060 2024-03-06T07:28:39.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.034+0000 7f6e31ffb640 1 -- 172.21.15.83:0/3562442944 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e28037070 con 0x7f6e340b3060 2024-03-06T07:28:39.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.034+0000 7f6e31ffb640 1 -- 172.21.15.83:0/3562442944 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e28031b40 con 0x7f6e340b3060 2024-03-06T07:28:39.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.034+0000 7f6e39864640 1 -- 172.21.15.83:0/3562442944 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6e340b3060 msgr2=0x7f6e3415e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.034+0000 7f6e39864640 1 --2- 172.21.15.83:0/3562442944 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6e340b3060 0x7f6e3415e850 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f6e280097a0 tx=0x7f6e2802b820 comp rx=0 tx=0).stop 2024-03-06T07:28:39.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.034+0000 7f6e39864640 1 -- 172.21.15.83:0/3562442944 shutdown_connections 2024-03-06T07:28:39.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.034+0000 7f6e39864640 1 --2- 172.21.15.83:0/3562442944 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6e3415bf30 0x7f6e3415ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.034+0000 7f6e39864640 1 --2- 172.21.15.83:0/3562442944 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6e340b3060 0x7f6e3415e850 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.034+0000 7f6e39864640 1 -- 172.21.15.83:0/3562442944 >> 172.21.15.83:0/3562442944 conn(0x7f6e340cf8a0 msgr2=0x7f6e340cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:39.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.034+0000 7f6e39864640 1 -- 172.21.15.83:0/3562442944 shutdown_connections 2024-03-06T07:28:39.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.034+0000 7f6e39864640 1 -- 172.21.15.83:0/3562442944 wait complete. 2024-03-06T07:28:39.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.035+0000 7f6e39864640 1 Processor -- start 2024-03-06T07:28:39.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.036+0000 7f6e39864640 1 -- start start 2024-03-06T07:28:39.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.036+0000 7f6e39864640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6e340b3060 0x7f6e341f6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.036+0000 7f6e39864640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6e3413dc10 0x7f6e341f1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.036+0000 7f6e39864640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6e3415bf30 0x7f6e3412e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.036+0000 7f6e39864640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3413dff0 con 0x7f6e3413dc10 2024-03-06T07:28:39.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.036+0000 7f6e39864640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3415b730 con 0x7f6e3415bf30 2024-03-06T07:28:39.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.036+0000 7f6e39864640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e340708d0 con 0x7f6e340b3060 2024-03-06T07:28:39.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.036+0000 7f6e327fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6e3413dc10 0x7f6e341f1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.036+0000 7f6e327fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6e3413dc10 0x7f6e341f1c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46174/0 (socket says 172.21.15.83:46174) 2024-03-06T07:28:39.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.036+0000 7f6e327fc640 1 -- 172.21.15.83:0/2605553630 learned_addr learned my addr 172.21.15.83:0/2605553630 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:39.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.036+0000 7f6e337fe640 1 --2- 172.21.15.83:0/2605553630 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6e3415bf30 0x7f6e3412e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.036+0000 7f6e32ffd640 1 --2- 172.21.15.83:0/2605553630 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6e340b3060 0x7f6e341f6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.037+0000 7f6e327fc640 1 -- 172.21.15.83:0/2605553630 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6e340b3060 msgr2=0x7f6e341f6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.037+0000 7f6e327fc640 1 --2- 172.21.15.83:0/2605553630 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6e340b3060 0x7f6e341f6310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.037+0000 7f6e327fc640 1 -- 172.21.15.83:0/2605553630 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6e3415bf30 msgr2=0x7f6e3412e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.037+0000 7f6e327fc640 1 --2- 172.21.15.83:0/2605553630 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6e3415bf30 0x7f6e3412e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.037+0000 7f6e327fc640 1 -- 172.21.15.83:0/2605553630 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e34142be0 con 0x7f6e3413dc10 2024-03-06T07:28:39.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.037+0000 7f6e327fc640 1 --2- 172.21.15.83:0/2605553630 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6e3413dc10 0x7f6e341f1c60 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f6e2802b860 tx=0x7f6e280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.037+0000 7f6e1bfff640 1 -- 172.21.15.83:0/2605553630 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e28032610 con 0x7f6e3413dc10 2024-03-06T07:28:39.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.037+0000 7f6e1bfff640 1 -- 172.21.15.83:0/2605553630 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e280327d0 con 0x7f6e3413dc10 2024-03-06T07:28:39.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.037+0000 7f6e39864640 1 -- 172.21.15.83:0/2605553630 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e34142ec0 con 0x7f6e3413dc10 2024-03-06T07:28:39.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.037+0000 7f6e1bfff640 1 -- 172.21.15.83:0/2605553630 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e28032bc0 con 0x7f6e3413dc10 2024-03-06T07:28:39.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.037+0000 7f6e39864640 1 -- 172.21.15.83:0/2605553630 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e34143480 con 0x7f6e3413dc10 2024-03-06T07:28:39.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.038+0000 7f6e39864640 1 -- 172.21.15.83:0/2605553630 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e3404f330 con 0x7f6e3413dc10 2024-03-06T07:28:39.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.038+0000 7f6e1bfff640 1 -- 172.21.15.83:0/2605553630 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6e2804c060 con 0x7f6e3413dc10 2024-03-06T07:28:39.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.039+0000 7f6e1bfff640 1 --2- 172.21.15.83:0/2605553630 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6e2006b4d0 0x7f6e2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.039+0000 7f6e1bfff640 1 -- 172.21.15.83:0/2605553630 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f6e280ab7e0 con 0x7f6e3413dc10 2024-03-06T07:28:39.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.039+0000 7f6e32ffd640 1 --2- 172.21.15.83:0/2605553630 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6e2006b4d0 0x7f6e2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.039+0000 7f6e32ffd640 1 --2- 172.21.15.83:0/2605553630 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6e2006b4d0 0x7f6e2006d990 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f6e24001f10 tx=0x7f6e24009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.043+0000 7f6e1bfff640 1 -- 172.21.15.83:0/2605553630 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6e28040070 con 0x7f6e3413dc10 2024-03-06T07:28:39.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.283+0000 7f6e39864640 1 -- 172.21.15.83:0/2605553630 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]} v 0) v1 -- 0x7f6e3406eb00 con 0x7f6e3413dc10 2024-03-06T07:28:39.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.596+0000 7f6e1bfff640 1 -- 172.21.15.83:0/2605553630 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]}]=0 v362) v1 ==== 142+0+0 (secure 0 0 0) 0x7f6e2802f050 con 0x7f6e3413dc10 2024-03-06T07:28:39.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.598+0000 7f6e39864640 1 -- 172.21.15.83:0/2605553630 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6e2006b4d0 msgr2=0x7f6e2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.598+0000 7f6e39864640 1 --2- 172.21.15.83:0/2605553630 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6e2006b4d0 0x7f6e2006d990 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f6e24001f10 tx=0x7f6e24009040 comp rx=0 tx=0).stop 2024-03-06T07:28:39.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.598+0000 7f6e39864640 1 -- 172.21.15.83:0/2605553630 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6e3413dc10 msgr2=0x7f6e341f1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.598+0000 7f6e39864640 1 --2- 172.21.15.83:0/2605553630 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6e3413dc10 0x7f6e341f1c60 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f6e2802b860 tx=0x7f6e280046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.598+0000 7f6e39864640 1 -- 172.21.15.83:0/2605553630 shutdown_connections 2024-03-06T07:28:39.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.598+0000 7f6e39864640 1 --2- 172.21.15.83:0/2605553630 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6e2006b4d0 0x7f6e2006d990 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.598+0000 7f6e39864640 1 --2- 172.21.15.83:0/2605553630 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6e3415bf30 0x7f6e3412e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.598+0000 7f6e39864640 1 --2- 172.21.15.83:0/2605553630 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6e3413dc10 0x7f6e341f1c60 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.598+0000 7f6e39864640 1 --2- 172.21.15.83:0/2605553630 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6e340b3060 0x7f6e341f6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.599+0000 7f6e39864640 1 -- 172.21.15.83:0/2605553630 >> 172.21.15.83:0/2605553630 conn(0x7f6e340cf8a0 msgr2=0x7f6e340af5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:39.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.599+0000 7f6e39864640 1 -- 172.21.15.83:0/2605553630 shutdown_connections 2024-03-06T07:28:39.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.599+0000 7f6e39864640 1 -- 172.21.15.83:0/2605553630 wait complete. 2024-03-06T07:28:39.612 INFO:tasks.cephfs_test_runner:That removal of primary dentry of hardlinked inode results ... ok 2024-03-06T07:28:39.612 INFO:tasks.cephfs_test_runner:Starting test: test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays) Reproduce https://tracker.ceph.com/issues/53597 2024-03-06T07:28:39.612 INFO:tasks.cephfs_test_runner:test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:28:39.613 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:39.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.712+0000 7f7e79313640 1 Processor -- start 2024-03-06T07:28:39.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.712+0000 7f7e79313640 1 -- start start 2024-03-06T07:28:39.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.712+0000 7f7e79313640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7e740d22b0 0x7f7e740d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.712+0000 7f7e79313640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e740b3430 con 0x7f7e740b41f0 2024-03-06T07:28:39.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.712+0000 7f7e79313640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e740b3710 con 0x7f7e7415dfe0 2024-03-06T07:28:39.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.712+0000 7f7e79313640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e740d2c50 con 0x7f7e740d22b0 2024-03-06T07:28:39.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.713+0000 7f7e727fc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7e740d22b0 0x7f7e740d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.713+0000 7f7e737fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f7e740b41f0 0x7f7e740b4680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:34872/0 (socket says 172.21.15.83:34872) 2024-03-06T07:28:39.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.713+0000 7f7e737fe640 1 -- 172.21.15.83:0/3354117319 learned_addr learned my addr 172.21.15.83:0/3354117319 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:28:39.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.713+0000 7f7e727fc640 1 -- 172.21.15.83:0/3354117319 >> v1:172.21.15.183:6789/0 conn(0x7f7e7415dfe0 legacy=0x7f7e740cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:39.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.713+0000 7f7e71ffb640 1 -- 172.21.15.83:0/3354117319 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1186902190 0 0) 0x7f7e740b3430 con 0x7f7e740b41f0 2024-03-06T07:28:39.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.713+0000 7f7e727fc640 1 -- 172.21.15.83:0/3354117319 >> v1:172.21.15.83:6789/0 conn(0x7f7e740b41f0 legacy=0x7f7e740b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.713+0000 7f7e727fc640 1 -- 172.21.15.83:0/3354117319 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e740b39f0 con 0x7f7e740d22b0 2024-03-06T07:28:39.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.714+0000 7f7e727fc640 1 --2- 172.21.15.83:0/3354117319 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e740d22b0 0x7f7e740d2710 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f7e640097d0 tx=0x7f7e6402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ba005ebbf4a8cbe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.714+0000 7f7e71ffb640 1 -- 172.21.15.83:0/3354117319 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e64035020 con 0x7f7e740d22b0 2024-03-06T07:28:39.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.714+0000 7f7e71ffb640 1 -- 172.21.15.83:0/3354117319 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e64031830 con 0x7f7e740d22b0 2024-03-06T07:28:39.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.714+0000 7f7e71ffb640 1 -- 172.21.15.83:0/3354117319 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e64031c00 con 0x7f7e740d22b0 2024-03-06T07:28:39.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.714+0000 7f7e79313640 1 -- 172.21.15.83:0/3354117319 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e740d22b0 msgr2=0x7f7e740d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.714+0000 7f7e79313640 1 --2- 172.21.15.83:0/3354117319 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e740d22b0 0x7f7e740d2710 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f7e640097d0 tx=0x7f7e6402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.714+0000 7f7e79313640 1 -- 172.21.15.83:0/3354117319 shutdown_connections 2024-03-06T07:28:39.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.714+0000 7f7e79313640 1 --2- 172.21.15.83:0/3354117319 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e740d22b0 0x7f7e740d2710 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.714+0000 7f7e79313640 1 -- 172.21.15.83:0/3354117319 >> 172.21.15.83:0/3354117319 conn(0x7f7e740d3cd0 msgr2=0x7f7e740d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:39.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.714+0000 7f7e79313640 1 -- 172.21.15.83:0/3354117319 shutdown_connections 2024-03-06T07:28:39.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.714+0000 7f7e79313640 1 -- 172.21.15.83:0/3354117319 wait complete. 2024-03-06T07:28:39.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.716+0000 7f7e79313640 1 Processor -- start 2024-03-06T07:28:39.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.716+0000 7f7e79313640 1 -- start start 2024-03-06T07:28:39.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.716+0000 7f7e79313640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e740b41f0 0x7f7e741610c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.716+0000 7f7e79313640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e740d22b0 0x7f7e740b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.716+0000 7f7e79313640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e7415dfe0 0x7f7e74149e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.716+0000 7f7e79313640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e740b3d60 con 0x7f7e740d22b0 2024-03-06T07:28:39.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.716+0000 7f7e79313640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e740b31e0 con 0x7f7e740b41f0 2024-03-06T07:28:39.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.716+0000 7f7e79313640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e740d1c70 con 0x7f7e7415dfe0 2024-03-06T07:28:39.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.716+0000 7f7e727fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e740d22b0 0x7f7e740b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.716+0000 7f7e727fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e740d22b0 0x7f7e740b68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46192/0 (socket says 172.21.15.83:46192) 2024-03-06T07:28:39.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.716+0000 7f7e727fc640 1 -- 172.21.15.83:0/3206677036 learned_addr learned my addr 172.21.15.83:0/3206677036 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:39.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.716+0000 7f7e72ffd640 1 --2- 172.21.15.83:0/3206677036 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e740b41f0 0x7f7e741610c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.716+0000 7f7e737fe640 1 --2- 172.21.15.83:0/3206677036 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e7415dfe0 0x7f7e74149e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.717+0000 7f7e727fc640 1 -- 172.21.15.83:0/3206677036 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e7415dfe0 msgr2=0x7f7e74149e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.717+0000 7f7e727fc640 1 --2- 172.21.15.83:0/3206677036 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e7415dfe0 0x7f7e74149e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.717+0000 7f7e727fc640 1 -- 172.21.15.83:0/3206677036 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e740b41f0 msgr2=0x7f7e741610c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.717+0000 7f7e727fc640 1 --2- 172.21.15.83:0/3206677036 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e740b41f0 0x7f7e741610c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.717+0000 7f7e727fc640 1 -- 172.21.15.83:0/3206677036 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e740b9070 con 0x7f7e740d22b0 2024-03-06T07:28:39.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.717+0000 7f7e737fe640 1 --2- 172.21.15.83:0/3206677036 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e7415dfe0 0x7f7e74149e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:39.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.717+0000 7f7e727fc640 1 --2- 172.21.15.83:0/3206677036 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e740d22b0 0x7f7e740b68d0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f7e640098f0 tx=0x7f7e640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.717+0000 7f7e5b7fe640 1 -- 172.21.15.83:0/3206677036 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6403ba80 con 0x7f7e740d22b0 2024-03-06T07:28:39.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.717+0000 7f7e5b7fe640 1 -- 172.21.15.83:0/3206677036 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e6403bc20 con 0x7f7e740d22b0 2024-03-06T07:28:39.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.717+0000 7f7e5b7fe640 1 -- 172.21.15.83:0/3206677036 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e640453f0 con 0x7f7e740d22b0 2024-03-06T07:28:39.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.717+0000 7f7e79313640 1 -- 172.21.15.83:0/3206677036 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e740b9350 con 0x7f7e740d22b0 2024-03-06T07:28:39.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.717+0000 7f7e79313640 1 -- 172.21.15.83:0/3206677036 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e740b9910 con 0x7f7e740d22b0 2024-03-06T07:28:39.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.718+0000 7f7e79313640 1 -- 172.21.15.83:0/3206677036 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e7404f330 con 0x7f7e740d22b0 2024-03-06T07:28:39.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.719+0000 7f7e5b7fe640 1 -- 172.21.15.83:0/3206677036 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7e6404c060 con 0x7f7e740d22b0 2024-03-06T07:28:39.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.719+0000 7f7e5b7fe640 1 --2- 172.21.15.83:0/3206677036 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7e6006b4d0 0x7f7e6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.719+0000 7f7e5b7fe640 1 -- 172.21.15.83:0/3206677036 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f7e640abc30 con 0x7f7e740d22b0 2024-03-06T07:28:39.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.719+0000 7f7e72ffd640 1 --2- 172.21.15.83:0/3206677036 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7e6006b4d0 0x7f7e6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.720+0000 7f7e72ffd640 1 --2- 172.21.15.83:0/3206677036 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7e6006b4d0 0x7f7e6006d990 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f7e68006fd0 tx=0x7f7e68009060 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.724+0000 7f7e5b7fe640 1 -- 172.21.15.83:0/3206677036 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7e64040070 con 0x7f7e740d22b0 2024-03-06T07:28:39.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.896+0000 7f7e79313640 1 -- 172.21.15.83:0/3206677036 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e7415d4c0 con 0x7f7e740d22b0 2024-03-06T07:28:39.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.897+0000 7f7e5b7fe640 1 -- 172.21.15.83:0/3206677036 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v119) v1 ==== 74+0+13302 (secure 0 0 0) 0x7f7e6407ad90 con 0x7f7e740d22b0 2024-03-06T07:28:39.899 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:28:39.899 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":119,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:28:38.594571+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.183:6834/1383294888":"2024-03-07T07:27:33.605177+0000","172.21.15.83:6836/2003687153":"2024-03-07T07:28:35.590368+0000","172.21.15.183:6832/1383294888":"2024-03-07T07:27:33.605177+0000","172.21.15.83:6837/2003687153":"2024-03-07T07:28:35.590368+0000","172.21.15.83:6836/855387870":"2024-03-07T07:27:33.587133+0000","172.21.15.83:6834/855387870":"2024-03-07T07:27:33.587133+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:39.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.899+0000 7f7e79313640 1 -- 172.21.15.83:0/3206677036 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7e6006b4d0 msgr2=0x7f7e6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.899+0000 7f7e79313640 1 --2- 172.21.15.83:0/3206677036 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7e6006b4d0 0x7f7e6006d990 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f7e68006fd0 tx=0x7f7e68009060 comp rx=0 tx=0).stop 2024-03-06T07:28:39.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.899+0000 7f7e79313640 1 -- 172.21.15.83:0/3206677036 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e740d22b0 msgr2=0x7f7e740b68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.899+0000 7f7e79313640 1 --2- 172.21.15.83:0/3206677036 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e740d22b0 0x7f7e740b68d0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f7e640098f0 tx=0x7f7e640046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.899+0000 7f7e79313640 1 -- 172.21.15.83:0/3206677036 shutdown_connections 2024-03-06T07:28:39.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.899+0000 7f7e79313640 1 --2- 172.21.15.83:0/3206677036 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7e6006b4d0 0x7f7e6006d990 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.899+0000 7f7e79313640 1 --2- 172.21.15.83:0/3206677036 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e7415dfe0 0x7f7e74149e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.899+0000 7f7e79313640 1 --2- 172.21.15.83:0/3206677036 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e740d22b0 0x7f7e740b68d0 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.899+0000 7f7e79313640 1 --2- 172.21.15.83:0/3206677036 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e740b41f0 0x7f7e741610c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.899+0000 7f7e79313640 1 -- 172.21.15.83:0/3206677036 >> 172.21.15.83:0/3206677036 conn(0x7f7e740d3cd0 msgr2=0x7f7e740b5750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:39.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.899+0000 7f7e79313640 1 -- 172.21.15.83:0/3206677036 shutdown_connections 2024-03-06T07:28:39.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:39.899+0000 7f7e79313640 1 -- 172.21.15.83:0/3206677036 wait complete. 2024-03-06T07:28:39.912 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:39.912 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:40.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.013+0000 7f667e0eb640 1 Processor -- start 2024-03-06T07:28:40.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.013+0000 7f667e0eb640 1 -- start start 2024-03-06T07:28:40.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.014+0000 7f667e0eb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f66780719e0 0x7f66780cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.014+0000 7f667e0eb640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f667806c9f0 0x7f667806ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.014+0000 7f667e0eb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66780b3320 con 0x7f667815a220 2024-03-06T07:28:40.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.014+0000 7f667e0eb640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66780cf610 con 0x7f667806c9f0 2024-03-06T07:28:40.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.014+0000 7f667e0eb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66780728d0 con 0x7f66780719e0 2024-03-06T07:28:40.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.014+0000 7f66777fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f667815a220 0x7f66780712d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:34882/0 (socket says 172.21.15.83:34882) 2024-03-06T07:28:40.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.014+0000 7f66777fe640 1 -- 172.21.15.83:0/306373233 learned_addr learned my addr 172.21.15.83:0/306373233 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:28:40.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.014+0000 7f66767fc640 1 -- 172.21.15.83:0/306373233 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1177476834 0 0) 0x7f66780b3320 con 0x7f667815a220 2024-03-06T07:28:40.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.014+0000 7f66767fc640 1 -- 172.21.15.83:0/306373233 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6660003520 con 0x7f667815a220 2024-03-06T07:28:40.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.014+0000 7f6677fff640 1 --2- 172.21.15.83:0/306373233 >> v2:172.21.15.183:3300/0 conn(0x7f667806c9f0 0x7f667806ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.014+0000 7f6676ffd640 1 --2- 172.21.15.83:0/306373233 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f66780719e0 0x7f66780cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.014+0000 7f66767fc640 1 -- 172.21.15.83:0/306373233 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2403126061 0 0) 0x7f6660003520 con 0x7f667815a220 2024-03-06T07:28:40.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.014+0000 7f66767fc640 1 -- 172.21.15.83:0/306373233 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f66780719e0 msgr2=0x7f66780cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:40.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.014+0000 7f66767fc640 1 --2- 172.21.15.83:0/306373233 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f66780719e0 0x7f66780cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.014+0000 7f66767fc640 1 -- 172.21.15.83:0/306373233 >> v2:172.21.15.183:3300/0 conn(0x7f667806c9f0 msgr2=0x7f667806ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:40.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.014+0000 7f66767fc640 1 --2- 172.21.15.83:0/306373233 >> v2:172.21.15.183:3300/0 conn(0x7f667806c9f0 0x7f667806ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.014+0000 7f66767fc640 1 -- 172.21.15.83:0/306373233 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66780b3590 con 0x7f667815a220 2024-03-06T07:28:40.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.014+0000 7f66767fc640 1 -- 172.21.15.83:0/306373233 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f6668001670 con 0x7f667815a220 2024-03-06T07:28:40.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.014+0000 7f66767fc640 1 -- 172.21.15.83:0/306373233 >> v1:172.21.15.83:6789/0 conn(0x7f667815a220 legacy=0x7f66780712d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.014+0000 7f66767fc640 1 --2- 172.21.15.83:0/306373233 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f66780719e0 0x7f66600039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.014+0000 7f66767fc640 1 --2- 172.21.15.83:0/306373233 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6660003f30 0x7f6660004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.014+0000 7f66767fc640 1 --2- 172.21.15.83:0/306373233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6660004d40 0x7f66600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.014+0000 7f66767fc640 1 -- 172.21.15.83:0/306373233 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66780cf610 con 0x7f6660004d40 2024-03-06T07:28:40.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.014+0000 7f66767fc640 1 -- 172.21.15.83:0/306373233 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66780728d0 con 0x7f6660003f30 2024-03-06T07:28:40.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.014+0000 7f66767fc640 1 -- 172.21.15.83:0/306373233 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f666000bac0 con 0x7f66780719e0 2024-03-06T07:28:40.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.014+0000 7f6677fff640 1 --2- 172.21.15.83:0/306373233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6660004d40 0x7f66600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.015+0000 7f66777fe640 1 --2- 172.21.15.83:0/306373233 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f66780719e0 0x7f66600039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.015+0000 7f6676ffd640 1 --2- 172.21.15.83:0/306373233 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6660003f30 0x7f6660004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.015+0000 7f6677fff640 1 -- 172.21.15.83:0/306373233 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f66780719e0 msgr2=0x7f66600039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:40.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.015+0000 7f6677fff640 1 --2- 172.21.15.83:0/306373233 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f66780719e0 0x7f66600039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.015+0000 7f6677fff640 1 -- 172.21.15.83:0/306373233 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6660003f30 msgr2=0x7f6660004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:40.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.015+0000 7f6677fff640 1 --2- 172.21.15.83:0/306373233 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6660003f30 0x7f6660004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.015+0000 7f6677fff640 1 -- 172.21.15.83:0/306373233 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6660003520 con 0x7f6660004d40 2024-03-06T07:28:40.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.015+0000 7f6677fff640 1 --2- 172.21.15.83:0/306373233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6660004d40 0x7f66600051c0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f667000e170 tx=0x7f6660005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d24c63c561264fcb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.016+0000 7f66767fc640 1 -- 172.21.15.83:0/306373233 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6670039070 con 0x7f6660004d40 2024-03-06T07:28:40.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.016+0000 7f66767fc640 1 -- 172.21.15.83:0/306373233 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f667002cc20 con 0x7f6660004d40 2024-03-06T07:28:40.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.016+0000 7f66767fc640 1 -- 172.21.15.83:0/306373233 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6670009c10 con 0x7f6660004d40 2024-03-06T07:28:40.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.016+0000 7f667e0eb640 1 -- 172.21.15.83:0/306373233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6660004d40 msgr2=0x7f66600051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.016+0000 7f667e0eb640 1 --2- 172.21.15.83:0/306373233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6660004d40 0x7f66600051c0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f667000e170 tx=0x7f6660005830 comp rx=0 tx=0).stop 2024-03-06T07:28:40.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.016+0000 7f6677fff640 1 -- 172.21.15.83:0/306373233 reap_dead start 2024-03-06T07:28:40.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.016+0000 7f667e0eb640 1 -- 172.21.15.83:0/306373233 shutdown_connections 2024-03-06T07:28:40.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.016+0000 7f667e0eb640 1 -- 172.21.15.83:0/306373233 >> 172.21.15.83:0/306373233 conn(0x7f667806b190 msgr2=0x7f667806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:40.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.016+0000 7f667e0eb640 1 -- 172.21.15.83:0/306373233 shutdown_connections 2024-03-06T07:28:40.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.016+0000 7f667e0eb640 1 -- 172.21.15.83:0/306373233 wait complete. 2024-03-06T07:28:40.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.017+0000 7f667e0eb640 1 Processor -- start 2024-03-06T07:28:40.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.017+0000 7f667e0eb640 1 -- start start 2024-03-06T07:28:40.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.018+0000 7f667e0eb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6678153690 0x7f66780b1400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.018+0000 7f667e0eb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6678153a70 0x7f667815ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.018+0000 7f667e0eb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f667815b010 0x7f6678154bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.018+0000 7f667e0eb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6678072530 con 0x7f667815b010 2024-03-06T07:28:40.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.018+0000 7f667e0eb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6678072030 con 0x7f6678153690 2024-03-06T07:28:40.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.018+0000 7f667e0eb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66780726b0 con 0x7f6678153a70 2024-03-06T07:28:40.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.018+0000 7f6677fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f667815b010 0x7f6678154bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.018+0000 7f6677fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f667815b010 0x7f6678154bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46206/0 (socket says 172.21.15.83:46206) 2024-03-06T07:28:40.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.018+0000 7f6677fff640 1 -- 172.21.15.83:0/1931161297 learned_addr learned my addr 172.21.15.83:0/1931161297 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:40.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.018+0000 7f6676ffd640 1 --2- 172.21.15.83:0/1931161297 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6678153a70 0x7f667815ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.018+0000 7f6677fff640 1 -- 172.21.15.83:0/1931161297 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6678153a70 msgr2=0x7f667815ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.018+0000 7f66777fe640 1 --2- 172.21.15.83:0/1931161297 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6678153690 0x7f66780b1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.019+0000 7f6677fff640 1 --2- 172.21.15.83:0/1931161297 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6678153a70 0x7f667815ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.019+0000 7f6677fff640 1 -- 172.21.15.83:0/1931161297 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6678153690 msgr2=0x7f66780b1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.019+0000 7f6677fff640 1 --2- 172.21.15.83:0/1931161297 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6678153690 0x7f66780b1400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.019+0000 7f6677fff640 1 -- 172.21.15.83:0/1931161297 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f667815c080 con 0x7f667815b010 2024-03-06T07:28:40.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.019+0000 7f66777fe640 1 --2- 172.21.15.83:0/1931161297 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6678153690 0x7f66780b1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:40.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.019+0000 7f6677fff640 1 --2- 172.21.15.83:0/1931161297 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f667815b010 0x7f6678154bb0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f6670037040 tx=0x7f6670009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.019+0000 7f665ffff640 1 -- 172.21.15.83:0/1931161297 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6670039070 con 0x7f667815b010 2024-03-06T07:28:40.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.019+0000 7f665ffff640 1 -- 172.21.15.83:0/1931161297 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f667003a040 con 0x7f667815b010 2024-03-06T07:28:40.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.019+0000 7f665ffff640 1 -- 172.21.15.83:0/1931161297 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6670009c10 con 0x7f667815b010 2024-03-06T07:28:40.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.019+0000 7f667e0eb640 1 -- 172.21.15.83:0/1931161297 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f667815c360 con 0x7f667815b010 2024-03-06T07:28:40.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.019+0000 7f667e0eb640 1 -- 172.21.15.83:0/1931161297 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f667815c920 con 0x7f667815b010 2024-03-06T07:28:40.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.020+0000 7f667e0eb640 1 -- 172.21.15.83:0/1931161297 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66780732b0 con 0x7f667815b010 2024-03-06T07:28:40.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.020+0000 7f665ffff640 1 -- 172.21.15.83:0/1931161297 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6670046020 con 0x7f667815b010 2024-03-06T07:28:40.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.021+0000 7f665ffff640 1 --2- 172.21.15.83:0/1931161297 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f666406b4d0 0x7f666406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.021+0000 7f665ffff640 1 -- 172.21.15.83:0/1931161297 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f66700ab940 con 0x7f667815b010 2024-03-06T07:28:40.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.021+0000 7f66777fe640 1 --2- 172.21.15.83:0/1931161297 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f666406b4d0 0x7f666406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.021+0000 7f66777fe640 1 --2- 172.21.15.83:0/1931161297 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f666406b4d0 0x7f666406d990 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f66680017e0 tx=0x7f6668009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.025+0000 7f665ffff640 1 -- 172.21.15.83:0/1931161297 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f667004d020 con 0x7f667815b010 2024-03-06T07:28:40.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.199+0000 7f667e0eb640 1 -- 172.21.15.83:0/1931161297 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6678068ff0 con 0x7f667815b010 2024-03-06T07:28:40.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.199+0000 7f665ffff640 1 -- 172.21.15.83:0/1931161297 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v119) v1 ==== 93+0+10 (secure 0 0 0) 0x7f667007aa10 con 0x7f667815b010 2024-03-06T07:28:40.201 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:28:40.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.201+0000 7f667e0eb640 1 -- 172.21.15.83:0/1931161297 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f666406b4d0 msgr2=0x7f666406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.201+0000 7f667e0eb640 1 --2- 172.21.15.83:0/1931161297 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f666406b4d0 0x7f666406d990 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f66680017e0 tx=0x7f6668009040 comp rx=0 tx=0).stop 2024-03-06T07:28:40.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.201+0000 7f667e0eb640 1 -- 172.21.15.83:0/1931161297 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f667815b010 msgr2=0x7f6678154bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.201+0000 7f667e0eb640 1 --2- 172.21.15.83:0/1931161297 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f667815b010 0x7f6678154bb0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f6670037040 tx=0x7f6670009750 comp rx=0 tx=0).stop 2024-03-06T07:28:40.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.201+0000 7f667e0eb640 1 -- 172.21.15.83:0/1931161297 shutdown_connections 2024-03-06T07:28:40.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.201+0000 7f667e0eb640 1 --2- 172.21.15.83:0/1931161297 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f666406b4d0 0x7f666406d990 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.201+0000 7f667e0eb640 1 --2- 172.21.15.83:0/1931161297 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f667815b010 0x7f6678154bb0 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.201+0000 7f667e0eb640 1 --2- 172.21.15.83:0/1931161297 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6678153a70 0x7f667815ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.201+0000 7f667e0eb640 1 --2- 172.21.15.83:0/1931161297 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6678153690 0x7f66780b1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.201+0000 7f667e0eb640 1 -- 172.21.15.83:0/1931161297 >> 172.21.15.83:0/1931161297 conn(0x7f667806b190 msgr2=0x7f66780cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:40.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.201+0000 7f667e0eb640 1 -- 172.21.15.83:0/1931161297 shutdown_connections 2024-03-06T07:28:40.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.201+0000 7f667e0eb640 1 -- 172.21.15.83:0/1931161297 wait complete. 2024-03-06T07:28:40.214 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir' 2024-03-06T07:28:40.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.312+0000 7fd5a36eb640 1 Processor -- start 2024-03-06T07:28:40.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.312+0000 7fd5a36eb640 1 -- start start 2024-03-06T07:28:40.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.313+0000 7fd5a36eb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd59c06e170 0x7fd59c166bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.313+0000 7fd5a36eb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd59c06d3b0 con 0x7fd59c0b50a0 2024-03-06T07:28:40.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.313+0000 7fd5a36eb640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd59c06d690 con 0x7fd59c15bed0 2024-03-06T07:28:40.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.313+0000 7fd5a36eb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd59c0ccaa0 con 0x7fd59c06e170 2024-03-06T07:28:40.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.313+0000 7fd5a1487640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fd59c0b50a0 0x7fd59c0b5480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:34898/0 (socket says 172.21.15.83:34898) 2024-03-06T07:28:40.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.313+0000 7fd5a1487640 1 -- 172.21.15.83:0/2369398494 learned_addr learned my addr 172.21.15.83:0/2369398494 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:28:40.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.313+0000 7fd5a1c88640 1 --2- 172.21.15.83:0/2369398494 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd59c06e170 0x7fd59c166bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.313+0000 7fd59bfff640 1 -- 172.21.15.83:0/2369398494 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4180954360 0 0) 0x7fd59c06d3b0 con 0x7fd59c0b50a0 2024-03-06T07:28:40.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.313+0000 7fd59bfff640 1 -- 172.21.15.83:0/2369398494 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd584003520 con 0x7fd59c0b50a0 2024-03-06T07:28:40.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.314+0000 7fd59bfff640 1 -- 172.21.15.83:0/2369398494 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3507629272 0 0) 0x7fd584003520 con 0x7fd59c0b50a0 2024-03-06T07:28:40.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.314+0000 7fd59bfff640 1 -- 172.21.15.83:0/2369398494 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd59c06e170 msgr2=0x7fd59c166bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:40.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.314+0000 7fd59bfff640 1 --2- 172.21.15.83:0/2369398494 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd59c06e170 0x7fd59c166bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.314+0000 7fd59bfff640 1 -- 172.21.15.83:0/2369398494 >> v1:172.21.15.183:6789/0 conn(0x7fd59c15bed0 legacy=0x7fd59c0726a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.314+0000 7fd59bfff640 1 -- 172.21.15.83:0/2369398494 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd59c06d970 con 0x7fd59c0b50a0 2024-03-06T07:28:40.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.314+0000 7fd59bfff640 1 -- 172.21.15.83:0/2369398494 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7fd58c0019f0 con 0x7fd59c0b50a0 2024-03-06T07:28:40.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.314+0000 7fd59bfff640 1 -- 172.21.15.83:0/2369398494 >> v1:172.21.15.83:6789/0 conn(0x7fd59c0b50a0 legacy=0x7fd59c0b5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.314+0000 7fd59bfff640 1 --2- 172.21.15.83:0/2369398494 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd59c06e170 0x7fd5840039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.314+0000 7fd59bfff640 1 --2- 172.21.15.83:0/2369398494 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd584003f60 0x7fd584004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.314+0000 7fd59bfff640 1 --2- 172.21.15.83:0/2369398494 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd584004d40 0x7fd5840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.314+0000 7fd59bfff640 1 -- 172.21.15.83:0/2369398494 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd58400ba70 con 0x7fd584003f60 2024-03-06T07:28:40.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.314+0000 7fd59bfff640 1 -- 172.21.15.83:0/2369398494 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd58400bc20 con 0x7fd584004d40 2024-03-06T07:28:40.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.314+0000 7fd59bfff640 1 -- 172.21.15.83:0/2369398494 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd58400be20 con 0x7fd59c06e170 2024-03-06T07:28:40.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.314+0000 7fd5a0c86640 1 --2- 172.21.15.83:0/2369398494 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd584003f60 0x7fd584004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.314+0000 7fd5a1487640 1 --2- 172.21.15.83:0/2369398494 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd59c06e170 0x7fd5840039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.314+0000 7fd5a1c88640 1 --2- 172.21.15.83:0/2369398494 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd584004d40 0x7fd5840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.314+0000 7fd5a0c86640 1 -- 172.21.15.83:0/2369398494 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd59c06e170 msgr2=0x7fd5840039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:40.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.314+0000 7fd5a0c86640 1 --2- 172.21.15.83:0/2369398494 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd59c06e170 0x7fd5840039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.314+0000 7fd5a0c86640 1 -- 172.21.15.83:0/2369398494 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd584004d40 msgr2=0x7fd5840051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:40.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.314+0000 7fd5a0c86640 1 --2- 172.21.15.83:0/2369398494 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd584004d40 0x7fd5840051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.314+0000 7fd5a0c86640 1 -- 172.21.15.83:0/2369398494 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd584003520 con 0x7fd584003f60 2024-03-06T07:28:40.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.315+0000 7fd5a0c86640 1 --2- 172.21.15.83:0/2369398494 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd584003f60 0x7fd584004800 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7fd590001720 tx=0x7fd584005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc4c6ba1892e961d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.315+0000 7fd59bfff640 1 -- 172.21.15.83:0/2369398494 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd590001370 con 0x7fd584003f60 2024-03-06T07:28:40.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.315+0000 7fd59bfff640 1 -- 172.21.15.83:0/2369398494 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5900052d0 con 0x7fd584003f60 2024-03-06T07:28:40.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.315+0000 7fd59bfff640 1 -- 172.21.15.83:0/2369398494 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd590003830 con 0x7fd584003f60 2024-03-06T07:28:40.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.315+0000 7fd5a36eb640 1 -- 172.21.15.83:0/2369398494 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd584003f60 msgr2=0x7fd584004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.315+0000 7fd5a36eb640 1 --2- 172.21.15.83:0/2369398494 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd584003f60 0x7fd584004800 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7fd590001720 tx=0x7fd584005830 comp rx=0 tx=0).stop 2024-03-06T07:28:40.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.315+0000 7fd5a1c88640 1 -- 172.21.15.83:0/2369398494 reap_dead start 2024-03-06T07:28:40.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.315+0000 7fd5a36eb640 1 -- 172.21.15.83:0/2369398494 shutdown_connections 2024-03-06T07:28:40.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.315+0000 7fd5a36eb640 1 -- 172.21.15.83:0/2369398494 >> 172.21.15.83:0/2369398494 conn(0x7fd59c0b1a90 msgr2=0x7fd59c0b1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:40.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.316+0000 7fd5a36eb640 1 -- 172.21.15.83:0/2369398494 shutdown_connections 2024-03-06T07:28:40.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.316+0000 7fd5a36eb640 1 -- 172.21.15.83:0/2369398494 wait complete. 2024-03-06T07:28:40.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.317+0000 7fd5a36eb640 1 Processor -- start 2024-03-06T07:28:40.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.317+0000 7fd5a36eb640 1 -- start start 2024-03-06T07:28:40.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.317+0000 7fd5a36eb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd584003f60 0x7fd59c154d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.317+0000 7fd5a36eb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd59c141e70 0x7fd59c1422f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.317+0000 7fd5a36eb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd59c15ca90 0x7fd59c15cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.317+0000 7fd5a36eb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59c06f730 con 0x7fd584003f60 2024-03-06T07:28:40.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.317+0000 7fd5a36eb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59c06f5b0 con 0x7fd59c141e70 2024-03-06T07:28:40.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.317+0000 7fd5a36eb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd59c06f0b0 con 0x7fd59c15ca90 2024-03-06T07:28:40.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.318+0000 7fd5a1487640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd584003f60 0x7fd59c154d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.318+0000 7fd5a1487640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd584003f60 0x7fd59c154d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46218/0 (socket says 172.21.15.83:46218) 2024-03-06T07:28:40.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.318+0000 7fd5a1487640 1 -- 172.21.15.83:0/4104363703 learned_addr learned my addr 172.21.15.83:0/4104363703 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:40.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.318+0000 7fd5a0c86640 1 --2- 172.21.15.83:0/4104363703 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd59c141e70 0x7fd59c1422f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.318+0000 7fd5a1c88640 1 --2- 172.21.15.83:0/4104363703 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd59c15ca90 0x7fd59c15cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.318+0000 7fd5a1487640 1 -- 172.21.15.83:0/4104363703 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd59c15ca90 msgr2=0x7fd59c15cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.318+0000 7fd5a1487640 1 --2- 172.21.15.83:0/4104363703 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd59c15ca90 0x7fd59c15cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.318+0000 7fd5a1487640 1 -- 172.21.15.83:0/4104363703 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd59c141e70 msgr2=0x7fd59c1422f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.318+0000 7fd5a1487640 1 --2- 172.21.15.83:0/4104363703 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd59c141e70 0x7fd59c1422f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.318+0000 7fd5a1487640 1 -- 172.21.15.83:0/4104363703 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd59c15d670 con 0x7fd584003f60 2024-03-06T07:28:40.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.318+0000 7fd5a0c86640 1 --2- 172.21.15.83:0/4104363703 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd59c141e70 0x7fd59c1422f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:40.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.318+0000 7fd5a1487640 1 --2- 172.21.15.83:0/4104363703 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd584003f60 0x7fd59c154d80 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7fd58c0077f0 tx=0x7fd58c00a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.318+0000 7fd599ffb640 1 -- 172.21.15.83:0/4104363703 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58c017020 con 0x7fd584003f60 2024-03-06T07:28:40.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.318+0000 7fd599ffb640 1 -- 172.21.15.83:0/4104363703 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd58c0133f0 con 0x7fd584003f60 2024-03-06T07:28:40.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.318+0000 7fd5a36eb640 1 -- 172.21.15.83:0/4104363703 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd59c15d950 con 0x7fd584003f60 2024-03-06T07:28:40.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.318+0000 7fd599ffb640 1 -- 172.21.15.83:0/4104363703 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58c0137c0 con 0x7fd584003f60 2024-03-06T07:28:40.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.318+0000 7fd5a36eb640 1 -- 172.21.15.83:0/4104363703 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd59c15df10 con 0x7fd584003f60 2024-03-06T07:28:40.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.319+0000 7fd5a36eb640 1 -- 172.21.15.83:0/4104363703 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd59c04f330 con 0x7fd584003f60 2024-03-06T07:28:40.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.320+0000 7fd599ffb640 1 -- 172.21.15.83:0/4104363703 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd58c01a020 con 0x7fd584003f60 2024-03-06T07:28:40.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.320+0000 7fd599ffb640 1 --2- 172.21.15.83:0/4104363703 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd58806b500 0x7fd58806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.320+0000 7fd599ffb640 1 -- 172.21.15.83:0/4104363703 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fd58c08a980 con 0x7fd584003f60 2024-03-06T07:28:40.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.320+0000 7fd5a0c86640 1 --2- 172.21.15.83:0/4104363703 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd58806b500 0x7fd58806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.321+0000 7fd5a0c86640 1 --2- 172.21.15.83:0/4104363703 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd58806b500 0x7fd58806d9c0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fd590033000 tx=0x7fd590009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.324+0000 7fd599ffb640 1 -- 172.21.15.83:0/4104363703 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd58c021020 con 0x7fd584003f60 2024-03-06T07:28:40.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.563+0000 7fd5a36eb640 1 -- 172.21.15.83:0/4104363703 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir"]} v 0) v1 -- 0x7fd59c151ee0 con 0x7fd584003f60 2024-03-06T07:28:40.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.603+0000 7fd599ffb640 1 -- 172.21.15.83:0/4104363703 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir"]}]=0 v363) v1 ==== 143+0+0 (secure 0 0 0) 0x7fd58c059ae0 con 0x7fd584003f60 2024-03-06T07:28:40.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.605+0000 7fd5a36eb640 1 -- 172.21.15.83:0/4104363703 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd58806b500 msgr2=0x7fd58806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.605+0000 7fd5a36eb640 1 --2- 172.21.15.83:0/4104363703 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd58806b500 0x7fd58806d9c0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fd590033000 tx=0x7fd590009030 comp rx=0 tx=0).stop 2024-03-06T07:28:40.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.605+0000 7fd5a36eb640 1 -- 172.21.15.83:0/4104363703 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd584003f60 msgr2=0x7fd59c154d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.605+0000 7fd5a36eb640 1 --2- 172.21.15.83:0/4104363703 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd584003f60 0x7fd59c154d80 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7fd58c0077f0 tx=0x7fd58c00a6f0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.605+0000 7fd5a36eb640 1 -- 172.21.15.83:0/4104363703 shutdown_connections 2024-03-06T07:28:40.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.605+0000 7fd5a36eb640 1 --2- 172.21.15.83:0/4104363703 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd58806b500 0x7fd58806d9c0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.605+0000 7fd5a36eb640 1 --2- 172.21.15.83:0/4104363703 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd59c15ca90 0x7fd59c15cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.605+0000 7fd5a36eb640 1 --2- 172.21.15.83:0/4104363703 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd59c141e70 0x7fd59c1422f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.605+0000 7fd5a36eb640 1 --2- 172.21.15.83:0/4104363703 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd584003f60 0x7fd59c154d80 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.605+0000 7fd5a36eb640 1 -- 172.21.15.83:0/4104363703 >> 172.21.15.83:0/4104363703 conn(0x7fd59c0b1a90 msgr2=0x7fd59c0b5480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:40.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.605+0000 7fd5a36eb640 1 -- 172.21.15.83:0/4104363703 shutdown_connections 2024-03-06T07:28:40.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.605+0000 7fd5a36eb640 1 -- 172.21.15.83:0/4104363703 wait complete. 2024-03-06T07:28:40.618 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:28:40.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.720+0000 7f7264208640 1 Processor -- start 2024-03-06T07:28:40.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.720+0000 7f7264208640 1 -- start start 2024-03-06T07:28:40.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.720+0000 7f7264208640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f725c1586e0 0x7f725c0aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.720+0000 7f7264208640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f725c0d01a0 0x7f725c0b33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.721+0000 7f7264208640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f725c158ac0 con 0x7f725c0aef80 2024-03-06T07:28:40.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.721+0000 7f7264208640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f725c15b2a0 con 0x7f725c1586e0 2024-03-06T07:28:40.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.721+0000 7f7264208640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f725c15b540 con 0x7f725c0d01a0 2024-03-06T07:28:40.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.721+0000 7f72627a5640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f725c0d01a0 0x7f725c0b33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.721+0000 7f7261fa4640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f725c1586e0 0x7f725c0aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.721+0000 7f72627a5640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f725c0d01a0 0x7f725c0b33a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:33644/0 (socket says 172.21.15.83:33644) 2024-03-06T07:28:40.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.721+0000 7f72627a5640 1 -- 172.21.15.83:0/4027031671 learned_addr learned my addr 172.21.15.83:0/4027031671 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:40.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.721+0000 7f7261fa4640 1 -- 172.21.15.83:0/4027031671 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f725c0d01a0 msgr2=0x7f725c0b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:40.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.721+0000 7f7261fa4640 1 --2- 172.21.15.83:0/4027031671 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f725c0d01a0 0x7f725c0b33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.721+0000 7f7261fa4640 1 -- 172.21.15.83:0/4027031671 >> v1:172.21.15.183:6789/0 conn(0x7f725c0aef80 legacy=0x7f725c0cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.721+0000 7f7261fa4640 1 -- 172.21.15.83:0/4027031671 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f725c15b820 con 0x7f725c1586e0 2024-03-06T07:28:40.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.721+0000 7f72627a5640 1 --2- 172.21.15.83:0/4027031671 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f725c0d01a0 0x7f725c0b33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:40.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.722+0000 7f7261fa4640 1 --2- 172.21.15.83:0/4027031671 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f725c1586e0 0x7f725c0aea40 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f72580097a0 tx=0x7f725802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe5ea445756ee3a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.722+0000 7f7260fa2640 1 -- 172.21.15.83:0/4027031671 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7258035020 con 0x7f725c1586e0 2024-03-06T07:28:40.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.722+0000 7f7260fa2640 1 -- 172.21.15.83:0/4027031671 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7258031830 con 0x7f725c1586e0 2024-03-06T07:28:40.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.722+0000 7f7260fa2640 1 -- 172.21.15.83:0/4027031671 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7258031c00 con 0x7f725c1586e0 2024-03-06T07:28:40.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.723+0000 7f7264208640 1 -- 172.21.15.83:0/4027031671 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f725c1586e0 msgr2=0x7f725c0aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.723+0000 7f7264208640 1 --2- 172.21.15.83:0/4027031671 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f725c1586e0 0x7f725c0aea40 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f72580097a0 tx=0x7f725802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.723+0000 7f7264208640 1 -- 172.21.15.83:0/4027031671 shutdown_connections 2024-03-06T07:28:40.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.723+0000 7f7264208640 1 --2- 172.21.15.83:0/4027031671 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f725c0d01a0 0x7f725c0b33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.723+0000 7f7264208640 1 --2- 172.21.15.83:0/4027031671 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f725c1586e0 0x7f725c0aea40 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.723+0000 7f7264208640 1 -- 172.21.15.83:0/4027031671 >> 172.21.15.83:0/4027031671 conn(0x7f725c06b190 msgr2=0x7f725c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:40.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.723+0000 7f7264208640 1 -- 172.21.15.83:0/4027031671 shutdown_connections 2024-03-06T07:28:40.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.723+0000 7f7264208640 1 -- 172.21.15.83:0/4027031671 wait complete. 2024-03-06T07:28:40.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.724+0000 7f7264208640 1 Processor -- start 2024-03-06T07:28:40.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.724+0000 7f7264208640 1 -- start start 2024-03-06T07:28:40.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.725+0000 7f7264208640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f725c0aef80 0x7f725c1487a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.725+0000 7f7264208640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f725c0d01a0 0x7f725c0adad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.725+0000 7f7264208640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f725c1586e0 0x7f725c1f70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.725+0000 7f7264208640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f725c071350 con 0x7f725c1586e0 2024-03-06T07:28:40.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.725+0000 7f7264208640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f725c15bde0 con 0x7f725c0d01a0 2024-03-06T07:28:40.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.725+0000 7f7264208640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f725c0714d0 con 0x7f725c0aef80 2024-03-06T07:28:40.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.725+0000 7f72627a5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f725c1586e0 0x7f725c1f70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.725+0000 7f72627a5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f725c1586e0 0x7f725c1f70c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46228/0 (socket says 172.21.15.83:46228) 2024-03-06T07:28:40.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.725+0000 7f72627a5640 1 -- 172.21.15.83:0/2011708316 learned_addr learned my addr 172.21.15.83:0/2011708316 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:40.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.725+0000 7f7261fa4640 1 --2- 172.21.15.83:0/2011708316 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f725c0aef80 0x7f725c1487a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.725+0000 7f72617a3640 1 --2- 172.21.15.83:0/2011708316 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f725c0d01a0 0x7f725c0adad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.725+0000 7f72627a5640 1 -- 172.21.15.83:0/2011708316 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f725c0aef80 msgr2=0x7f725c1487a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.725+0000 7f72627a5640 1 --2- 172.21.15.83:0/2011708316 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f725c0aef80 0x7f725c1487a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.725+0000 7f72627a5640 1 -- 172.21.15.83:0/2011708316 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f725c0d01a0 msgr2=0x7f725c0adad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.725+0000 7f72627a5640 1 --2- 172.21.15.83:0/2011708316 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f725c0d01a0 0x7f725c0adad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.725+0000 7f72627a5640 1 -- 172.21.15.83:0/2011708316 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f725c1f7c30 con 0x7f725c1586e0 2024-03-06T07:28:40.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.725+0000 7f72617a3640 1 --2- 172.21.15.83:0/2011708316 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f725c0d01a0 0x7f725c0adad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:40.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.726+0000 7f72627a5640 1 --2- 172.21.15.83:0/2011708316 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f725c1586e0 0x7f725c1f70c0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f725400efc0 tx=0x7f725400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.726+0000 7f72527fc640 1 -- 172.21.15.83:0/2011708316 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725400cbe0 con 0x7f725c1586e0 2024-03-06T07:28:40.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.726+0000 7f72527fc640 1 -- 172.21.15.83:0/2011708316 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f725400cd80 con 0x7f725c1586e0 2024-03-06T07:28:40.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.726+0000 7f7264208640 1 -- 172.21.15.83:0/2011708316 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f725c147cc0 con 0x7f725c1586e0 2024-03-06T07:28:40.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.726+0000 7f72527fc640 1 -- 172.21.15.83:0/2011708316 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7254003bd0 con 0x7f725c1586e0 2024-03-06T07:28:40.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.726+0000 7f7264208640 1 -- 172.21.15.83:0/2011708316 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f725c14baa0 con 0x7f725c1586e0 2024-03-06T07:28:40.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.727+0000 7f7264208640 1 -- 172.21.15.83:0/2011708316 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f725c1480b0 con 0x7f725c1586e0 2024-03-06T07:28:40.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.728+0000 7f72527fc640 1 -- 172.21.15.83:0/2011708316 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f725401b030 con 0x7f725c1586e0 2024-03-06T07:28:40.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.728+0000 7f72527fc640 1 --2- 172.21.15.83:0/2011708316 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f724006b4d0 0x7f724006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.728+0000 7f72527fc640 1 -- 172.21.15.83:0/2011708316 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f725408a310 con 0x7f725c1586e0 2024-03-06T07:28:40.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.728+0000 7f7261fa4640 1 --2- 172.21.15.83:0/2011708316 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f724006b4d0 0x7f724006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.729+0000 7f7261fa4640 1 --2- 172.21.15.83:0/2011708316 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f724006b4d0 0x7f724006d990 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f7258036000 tx=0x7f725803c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.732+0000 7f72527fc640 1 -- 172.21.15.83:0/2011708316 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7254022020 con 0x7f725c1586e0 2024-03-06T07:28:40.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.906+0000 7f7264208640 1 -- 172.21.15.83:0/2011708316 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f725c147e90 con 0x7f725c1586e0 2024-03-06T07:28:40.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.911+0000 7f72527fc640 1 -- 172.21.15.83:0/2011708316 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v32)=0 v32) v1 ==== 122+0+0 (secure 0 0 0) 0x7f7254010040 con 0x7f725c1586e0 2024-03-06T07:28:40.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.913+0000 7f7264208640 1 -- 172.21.15.83:0/2011708316 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f724006b4d0 msgr2=0x7f724006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.913+0000 7f7264208640 1 --2- 172.21.15.83:0/2011708316 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f724006b4d0 0x7f724006d990 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f7258036000 tx=0x7f725803c040 comp rx=0 tx=0).stop 2024-03-06T07:28:40.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.913+0000 7f7264208640 1 -- 172.21.15.83:0/2011708316 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f725c1586e0 msgr2=0x7f725c1f70c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.913+0000 7f7264208640 1 --2- 172.21.15.83:0/2011708316 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f725c1586e0 0x7f725c1f70c0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f725400efc0 tx=0x7f725400c490 comp rx=0 tx=0).stop 2024-03-06T07:28:40.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.913+0000 7f7264208640 1 -- 172.21.15.83:0/2011708316 shutdown_connections 2024-03-06T07:28:40.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.913+0000 7f7264208640 1 --2- 172.21.15.83:0/2011708316 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f724006b4d0 0x7f724006d990 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.913+0000 7f7264208640 1 --2- 172.21.15.83:0/2011708316 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f725c1586e0 0x7f725c1f70c0 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.913+0000 7f7264208640 1 --2- 172.21.15.83:0/2011708316 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f725c0d01a0 0x7f725c0adad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.913+0000 7f7264208640 1 --2- 172.21.15.83:0/2011708316 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f725c0aef80 0x7f725c1487a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.913+0000 7f7264208640 1 -- 172.21.15.83:0/2011708316 >> 172.21.15.83:0/2011708316 conn(0x7f725c06b190 msgr2=0x7f725c0cef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:40.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.913+0000 7f7264208640 1 -- 172.21.15.83:0/2011708316 shutdown_connections 2024-03-06T07:28:40.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:40.913+0000 7f7264208640 1 -- 172.21.15.83:0/2011708316 wait complete. 2024-03-06T07:28:40.926 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:28:40.926 DEBUG:teuthology.orchestra.run.smithi083:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:28:40.967 DEBUG:teuthology.orchestra.run.smithi183:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:28:41.011 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:28:41.011 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:28:41.011 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:41.026 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1257 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:28:41.026 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:28:41.026 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:41.068 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=287 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:28:41.068 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:28:41.069 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:41.083 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1257 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:28:41.084 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:28:41.084 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:41.125 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=287 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:28:41.125 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:28:41.126 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:41.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.224+0000 7f7106483640 1 Processor -- start 2024-03-06T07:28:41.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.224+0000 7f7106483640 1 -- start start 2024-03-06T07:28:41.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.224+0000 7f7106483640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f71000d22b0 0x7f71000d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.225+0000 7f7106483640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71000b3430 con 0x7f71000b41f0 2024-03-06T07:28:41.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.225+0000 7f7106483640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71000b3710 con 0x7f710015dfe0 2024-03-06T07:28:41.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.225+0000 7f7106483640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71000d2c50 con 0x7f71000d22b0 2024-03-06T07:28:41.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.225+0000 7f70ff7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f71000d22b0 0x7f71000d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.225+0000 7f7104a20640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f71000b41f0 0x7f71000b4680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:34900/0 (socket says 172.21.15.83:34900) 2024-03-06T07:28:41.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.225+0000 7f7104a20640 1 -- 172.21.15.83:0/1387673065 learned_addr learned my addr 172.21.15.83:0/1387673065 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:28:41.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.225+0000 7f70ff7fe640 1 -- 172.21.15.83:0/1387673065 >> v1:172.21.15.183:6789/0 conn(0x7f710015dfe0 legacy=0x7f71000cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:41.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.225+0000 7f70ff7fe640 1 -- 172.21.15.83:0/1387673065 >> v1:172.21.15.83:6789/0 conn(0x7f71000b41f0 legacy=0x7f71000b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.225+0000 7f70ff7fe640 1 -- 172.21.15.83:0/1387673065 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71000b39f0 con 0x7f71000d22b0 2024-03-06T07:28:41.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.225+0000 7f70feffd640 1 -- 172.21.15.83:0/1387673065 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4123045124 0 0) 0x7f71000b3430 con 0x7f71000b41f0 2024-03-06T07:28:41.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.226+0000 7f70ff7fe640 1 --2- 172.21.15.83:0/1387673065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71000d22b0 0x7f71000d2710 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f70f40097d0 tx=0x7f70f402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a46c5eccfbfe74d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.226+0000 7f70feffd640 1 -- 172.21.15.83:0/1387673065 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f4035020 con 0x7f71000d22b0 2024-03-06T07:28:41.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.226+0000 7f70feffd640 1 -- 172.21.15.83:0/1387673065 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70f4031830 con 0x7f71000d22b0 2024-03-06T07:28:41.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.226+0000 7f70feffd640 1 -- 172.21.15.83:0/1387673065 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f4031c00 con 0x7f71000d22b0 2024-03-06T07:28:41.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.226+0000 7f7106483640 1 -- 172.21.15.83:0/1387673065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71000d22b0 msgr2=0x7f71000d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.226+0000 7f7106483640 1 --2- 172.21.15.83:0/1387673065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71000d22b0 0x7f71000d2710 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f70f40097d0 tx=0x7f70f402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.226+0000 7f7106483640 1 -- 172.21.15.83:0/1387673065 shutdown_connections 2024-03-06T07:28:41.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.226+0000 7f7106483640 1 --2- 172.21.15.83:0/1387673065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71000d22b0 0x7f71000d2710 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.226+0000 7f7106483640 1 -- 172.21.15.83:0/1387673065 >> 172.21.15.83:0/1387673065 conn(0x7f71000d3cd0 msgr2=0x7f71000d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:41.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.226+0000 7f7106483640 1 -- 172.21.15.83:0/1387673065 shutdown_connections 2024-03-06T07:28:41.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.226+0000 7f7106483640 1 -- 172.21.15.83:0/1387673065 wait complete. 2024-03-06T07:28:41.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.227+0000 7f7106483640 1 Processor -- start 2024-03-06T07:28:41.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.228+0000 7f7106483640 1 -- start start 2024-03-06T07:28:41.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.228+0000 7f7106483640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71000b41f0 0x7f71001610c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.228+0000 7f7106483640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f71000d22b0 0x7f71000b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.228+0000 7f7106483640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f710015dfe0 0x7f7100149e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.228+0000 7f7106483640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71000b3d60 con 0x7f71000b41f0 2024-03-06T07:28:41.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.228+0000 7f7106483640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71000b31e0 con 0x7f710015dfe0 2024-03-06T07:28:41.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.228+0000 7f7106483640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71000d1c70 con 0x7f71000d22b0 2024-03-06T07:28:41.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.228+0000 7f70fffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71000b41f0 0x7f71001610c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.229+0000 7f70fffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71000b41f0 0x7f71001610c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46236/0 (socket says 172.21.15.83:46236) 2024-03-06T07:28:41.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.229+0000 7f70fffff640 1 -- 172.21.15.83:0/3708833204 learned_addr learned my addr 172.21.15.83:0/3708833204 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:41.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.229+0000 7f70ff7fe640 1 --2- 172.21.15.83:0/3708833204 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f71000d22b0 0x7f71000b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.229+0000 7f7104a20640 1 --2- 172.21.15.83:0/3708833204 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f710015dfe0 0x7f7100149e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.229+0000 7f70fffff640 1 -- 172.21.15.83:0/3708833204 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f71000d22b0 msgr2=0x7f71000b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.229+0000 7f70fffff640 1 --2- 172.21.15.83:0/3708833204 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f71000d22b0 0x7f71000b68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.229+0000 7f70fffff640 1 -- 172.21.15.83:0/3708833204 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f710015dfe0 msgr2=0x7f7100149e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.229+0000 7f70fffff640 1 --2- 172.21.15.83:0/3708833204 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f710015dfe0 0x7f7100149e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.229+0000 7f70fffff640 1 -- 172.21.15.83:0/3708833204 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71000b9070 con 0x7f71000b41f0 2024-03-06T07:28:41.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.229+0000 7f7104a20640 1 --2- 172.21.15.83:0/3708833204 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f710015dfe0 0x7f7100149e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:41.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.229+0000 7f70fffff640 1 --2- 172.21.15.83:0/3708833204 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71000b41f0 0x7f71001610c0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f70f000b3f0 tx=0x7f70f000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.229+0000 7f70fcff9640 1 -- 172.21.15.83:0/3708833204 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f0015020 con 0x7f71000b41f0 2024-03-06T07:28:41.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.229+0000 7f70fcff9640 1 -- 172.21.15.83:0/3708833204 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70f0011830 con 0x7f71000b41f0 2024-03-06T07:28:41.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.229+0000 7f7106483640 1 -- 172.21.15.83:0/3708833204 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71000b9350 con 0x7f71000b41f0 2024-03-06T07:28:41.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.229+0000 7f70fcff9640 1 -- 172.21.15.83:0/3708833204 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f0011c20 con 0x7f71000b41f0 2024-03-06T07:28:41.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.229+0000 7f7106483640 1 -- 172.21.15.83:0/3708833204 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71000b9910 con 0x7f71000b41f0 2024-03-06T07:28:41.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.230+0000 7f7106483640 1 -- 172.21.15.83:0/3708833204 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f710004f330 con 0x7f71000b41f0 2024-03-06T07:28:41.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.231+0000 7f70fcff9640 1 -- 172.21.15.83:0/3708833204 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f70f0018020 con 0x7f71000b41f0 2024-03-06T07:28:41.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.231+0000 7f70fcff9640 1 --2- 172.21.15.83:0/3708833204 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f70ec06b4d0 0x7f70ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.231+0000 7f70fcff9640 1 -- 172.21.15.83:0/3708833204 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f70f0089c00 con 0x7f71000b41f0 2024-03-06T07:28:41.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.231+0000 7f70ff7fe640 1 --2- 172.21.15.83:0/3708833204 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f70ec06b4d0 0x7f70ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.231+0000 7f70ff7fe640 1 --2- 172.21.15.83:0/3708833204 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f70ec06b4d0 0x7f70ec06d990 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f70f4036000 tx=0x7f70f403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.235+0000 7f70fcff9640 1 -- 172.21.15.83:0/3708833204 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f70f0090080 con 0x7f71000b41f0 2024-03-06T07:28:41.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.443+0000 7f7106483640 1 -- 172.21.15.83:0/3708833204 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f710015d4c0 con 0x7f71000b41f0 2024-03-06T07:28:41.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.443+0000 7f70fcff9640 1 -- 172.21.15.83:0/3708833204 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+3699 (secure 0 0 0) 0x7f70f001f030 con 0x7f71000b41f0 2024-03-06T07:28:41.445 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:28:41.446 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":68,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7051,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/101642171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":101642171},{"type":"v1","addr":"172.21.15.183:6833","nonce":101642171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7057,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3146884900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":3146884900},{"type":"v1","addr":"172.21.15.183:6835","nonce":3146884900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8208,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/1922041621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":1922041621},{"type":"v1","addr":"172.21.15.83:6835","nonce":1922041621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":8559,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/882448879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":882448879},{"type":"v1","addr":"172.21.15.83:6837","nonce":882448879}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68}],"filesystems":[]} 2024-03-06T07:28:41.446 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 68 2024-03-06T07:28:41.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.445+0000 7f7106483640 1 -- 172.21.15.83:0/3708833204 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f70ec06b4d0 msgr2=0x7f70ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.445+0000 7f7106483640 1 --2- 172.21.15.83:0/3708833204 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f70ec06b4d0 0x7f70ec06d990 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f70f4036000 tx=0x7f70f403c040 comp rx=0 tx=0).stop 2024-03-06T07:28:41.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.445+0000 7f7106483640 1 -- 172.21.15.83:0/3708833204 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71000b41f0 msgr2=0x7f71001610c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.445+0000 7f7106483640 1 --2- 172.21.15.83:0/3708833204 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71000b41f0 0x7f71001610c0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f70f000b3f0 tx=0x7f70f000b8b0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.445+0000 7f7106483640 1 -- 172.21.15.83:0/3708833204 shutdown_connections 2024-03-06T07:28:41.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.445+0000 7f7106483640 1 --2- 172.21.15.83:0/3708833204 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f70ec06b4d0 0x7f70ec06d990 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.445+0000 7f7106483640 1 --2- 172.21.15.83:0/3708833204 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f710015dfe0 0x7f7100149e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.445+0000 7f7106483640 1 --2- 172.21.15.83:0/3708833204 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f71000d22b0 0x7f71000b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.445+0000 7f7106483640 1 --2- 172.21.15.83:0/3708833204 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71000b41f0 0x7f71001610c0 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.445+0000 7f7106483640 1 -- 172.21.15.83:0/3708833204 >> 172.21.15.83:0/3708833204 conn(0x7f71000d3cd0 msgr2=0x7f71000b5750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:41.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.446+0000 7f7106483640 1 -- 172.21.15.83:0/3708833204 shutdown_connections 2024-03-06T07:28:41.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:41.446+0000 7f7106483640 1 -- 172.21.15.83:0/3708833204 wait complete. 2024-03-06T07:28:41.459 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:28:41.459 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:28:41.459 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:28:41.459 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:41.459 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:28:41.459 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:28:41.460 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:28:41.460 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:41.460 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:28:41.460 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:28:41.460 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:28:41.460 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:41.460 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:28:41.461 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:28:41.461 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:28:41.461 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:41.468 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:28:41.469 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:28:41.469 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:28:41.469 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:28:41.473 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:28:41.473 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:28:41.511 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:28:41.511 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:41.512 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:28:41.512 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:41.516 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:28:41.516 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:41.586 INFO:tasks.ceph.mds.c.smithi083.stdout:starting mds.c at 2024-03-06T07:28:41.586 INFO:tasks.ceph.mds.a.smithi083.stdout:starting mds.a at 2024-03-06T07:28:41.597 INFO:tasks.ceph.mds.d.smithi183.stdout:starting mds.d at 2024-03-06T07:28:43.984 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-03-06T07:28:47.472 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:28:47.473 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:28:47.476 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:28:47.476 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:47.476 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:28:47.560 INFO:tasks.ceph.mds.b.smithi183.stdout:starting mds.b at 2024-03-06T07:28:47.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.588+0000 7f90772c0640 1 Processor -- start 2024-03-06T07:28:47.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.588+0000 7f90772c0640 1 -- start start 2024-03-06T07:28:47.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.589+0000 7f90772c0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f907015ff70 0x7f907006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.589+0000 7f90772c0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f907006ce30 0x7f907006d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.589+0000 7f90772c0640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f907006ded0 con 0x7f907006da60 2024-03-06T07:28:47.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.589+0000 7f90772c0640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f907006d290 con 0x7f907015ff70 2024-03-06T07:28:47.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.589+0000 7f90772c0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90700717e0 con 0x7f907006ce30 2024-03-06T07:28:47.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.589+0000 7f907505c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f907015ff70 0x7f907006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.589+0000 7f907505c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f907015ff70 0x7f907006c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48710/0 (socket says 172.21.15.83:48710) 2024-03-06T07:28:47.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.589+0000 7f907505c640 1 -- 172.21.15.83:0/133683236 learned_addr learned my addr 172.21.15.83:0/133683236 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:47.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.589+0000 7f907485b640 1 --2- 172.21.15.83:0/133683236 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f907006ce30 0x7f907006d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.589+0000 7f907505c640 1 -- 172.21.15.83:0/133683236 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f907006ce30 msgr2=0x7f907006d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:47.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.589+0000 7f907505c640 1 --2- 172.21.15.83:0/133683236 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f907006ce30 0x7f907006d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.589+0000 7f907505c640 1 -- 172.21.15.83:0/133683236 >> v1:172.21.15.183:6789/0 conn(0x7f907006da60 legacy=0x7f90700cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.589+0000 7f907505c640 1 -- 172.21.15.83:0/133683236 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f907006e170 con 0x7f907015ff70 2024-03-06T07:28:47.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.589+0000 7f907485b640 1 --2- 172.21.15.83:0/133683236 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f907006ce30 0x7f907006d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:47.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.590+0000 7f907505c640 1 --2- 172.21.15.83:0/133683236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f907015ff70 0x7f907006c8f0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f9068009e50 tx=0x7f906802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5011a1f52afa888c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:47.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.590+0000 7f906ffff640 1 -- 172.21.15.83:0/133683236 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f906802ca70 con 0x7f907015ff70 2024-03-06T07:28:47.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.590+0000 7f906ffff640 1 -- 172.21.15.83:0/133683236 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f906802cc10 con 0x7f907015ff70 2024-03-06T07:28:47.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.590+0000 7f906ffff640 1 -- 172.21.15.83:0/133683236 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90680343f0 con 0x7f907015ff70 2024-03-06T07:28:47.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.591+0000 7f90772c0640 1 -- 172.21.15.83:0/133683236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f907015ff70 msgr2=0x7f907006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.591+0000 7f90772c0640 1 --2- 172.21.15.83:0/133683236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f907015ff70 0x7f907006c8f0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f9068009e50 tx=0x7f906802bed0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.591+0000 7f90772c0640 1 -- 172.21.15.83:0/133683236 shutdown_connections 2024-03-06T07:28:47.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.591+0000 7f90772c0640 1 --2- 172.21.15.83:0/133683236 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f907006ce30 0x7f907006d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.591+0000 7f90772c0640 1 --2- 172.21.15.83:0/133683236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f907015ff70 0x7f907006c8f0 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.591+0000 7f90772c0640 1 -- 172.21.15.83:0/133683236 >> 172.21.15.83:0/133683236 conn(0x7f90700b1c80 msgr2=0x7f90700cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:47.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.591+0000 7f90772c0640 1 -- 172.21.15.83:0/133683236 shutdown_connections 2024-03-06T07:28:47.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.591+0000 7f90772c0640 1 -- 172.21.15.83:0/133683236 wait complete. 2024-03-06T07:28:47.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.592+0000 7f90772c0640 1 Processor -- start 2024-03-06T07:28:47.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.592+0000 7f90772c0640 1 -- start start 2024-03-06T07:28:47.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.593+0000 7f90772c0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f907006ce30 0x7f9070156e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.593+0000 7f90772c0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f907006da60 0x7f90701585b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.593+0000 7f90772c0640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f907015ff70 0x7f90701607f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.593+0000 7f90772c0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90701665b0 con 0x7f907006ce30 2024-03-06T07:28:47.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.593+0000 7f90772c0640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90701662d0 con 0x7f907015ff70 2024-03-06T07:28:47.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.593+0000 7f90772c0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9070160350 con 0x7f907006da60 2024-03-06T07:28:47.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.593+0000 7f907505c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f907006ce30 0x7f9070156e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.593+0000 7f907505c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f907006ce30 0x7f9070156e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48722/0 (socket says 172.21.15.83:48722) 2024-03-06T07:28:47.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.593+0000 7f907505c640 1 -- 172.21.15.83:0/3837522102 learned_addr learned my addr 172.21.15.83:0/3837522102 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:47.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.593+0000 7f907485b640 1 --2- 172.21.15.83:0/3837522102 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f907006da60 0x7f90701585b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.593+0000 7f907585d640 1 --2- 172.21.15.83:0/3837522102 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f907015ff70 0x7f90701607f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.593+0000 7f907505c640 1 -- 172.21.15.83:0/3837522102 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f907006da60 msgr2=0x7f90701585b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.593+0000 7f907505c640 1 --2- 172.21.15.83:0/3837522102 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f907006da60 0x7f90701585b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.593+0000 7f907505c640 1 -- 172.21.15.83:0/3837522102 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f907015ff70 msgr2=0x7f90701607f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.593+0000 7f907505c640 1 --2- 172.21.15.83:0/3837522102 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f907015ff70 0x7f90701607f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.593+0000 7f907505c640 1 -- 172.21.15.83:0/3837522102 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9070157510 con 0x7f907006ce30 2024-03-06T07:28:47.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.593+0000 7f907505c640 1 --2- 172.21.15.83:0/3837522102 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f907006ce30 0x7f9070156e50 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f9068000f80 tx=0x7f9068004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:47.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.593+0000 7f906dffb640 1 -- 172.21.15.83:0/3837522102 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9068043020 con 0x7f907006ce30 2024-03-06T07:28:47.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.593+0000 7f906dffb640 1 -- 172.21.15.83:0/3837522102 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f906803e450 con 0x7f907006ce30 2024-03-06T07:28:47.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.593+0000 7f90772c0640 1 -- 172.21.15.83:0/3837522102 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90701577f0 con 0x7f907006ce30 2024-03-06T07:28:47.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.593+0000 7f906dffb640 1 -- 172.21.15.83:0/3837522102 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9068009510 con 0x7f907006ce30 2024-03-06T07:28:47.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.594+0000 7f90772c0640 1 -- 172.21.15.83:0/3837522102 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9070157db0 con 0x7f907006ce30 2024-03-06T07:28:47.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.595+0000 7f90772c0640 1 -- 172.21.15.83:0/3837522102 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f907004f330 con 0x7f907006ce30 2024-03-06T07:28:47.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.595+0000 7f906dffb640 1 -- 172.21.15.83:0/3837522102 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9068046020 con 0x7f907006ce30 2024-03-06T07:28:47.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.595+0000 7f906dffb640 1 --2- 172.21.15.83:0/3837522102 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f905c06b4d0 0x7f905c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.596+0000 7f906dffb640 1 -- 172.21.15.83:0/3837522102 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f90680ac710 con 0x7f907006ce30 2024-03-06T07:28:47.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.596+0000 7f907485b640 1 --2- 172.21.15.83:0/3837522102 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f905c06b4d0 0x7f905c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.596+0000 7f907485b640 1 --2- 172.21.15.83:0/3837522102 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f905c06b4d0 0x7f905c06d990 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f9064006fd0 tx=0x7f9064008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:47.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.600+0000 7f906dffb640 1 -- 172.21.15.83:0/3837522102 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f906804d030 con 0x7f907006ce30 2024-03-06T07:28:47.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:47.784+0000 7f90772c0640 1 -- 172.21.15.83:0/3837522102 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f907015d670 con 0x7f907006ce30 2024-03-06T07:28:48.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:48.921+0000 7f906dffb640 1 -- 172.21.15.83:0/3837522102 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v124) v1 ==== 97+0+0 (secure 0 0 0) 0x7f906807b340 con 0x7f907006ce30 2024-03-06T07:28:48.923 INFO:teuthology.orchestra.run.smithi083.stderr: removed all blocklist entries 2024-03-06T07:28:48.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:48.923+0000 7f90772c0640 1 -- 172.21.15.83:0/3837522102 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f905c06b4d0 msgr2=0x7f905c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:48.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:48.923+0000 7f90772c0640 1 --2- 172.21.15.83:0/3837522102 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f905c06b4d0 0x7f905c06d990 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f9064006fd0 tx=0x7f9064008040 comp rx=0 tx=0).stop 2024-03-06T07:28:48.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:48.923+0000 7f90772c0640 1 -- 172.21.15.83:0/3837522102 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f907006ce30 msgr2=0x7f9070156e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:48.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:48.923+0000 7f90772c0640 1 --2- 172.21.15.83:0/3837522102 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f907006ce30 0x7f9070156e50 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f9068000f80 tx=0x7f9068004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:48.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:48.923+0000 7f90772c0640 1 -- 172.21.15.83:0/3837522102 shutdown_connections 2024-03-06T07:28:48.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:48.923+0000 7f90772c0640 1 --2- 172.21.15.83:0/3837522102 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f905c06b4d0 0x7f905c06d990 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:48.923+0000 7f90772c0640 1 --2- 172.21.15.83:0/3837522102 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f907015ff70 0x7f90701607f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:48.923+0000 7f90772c0640 1 --2- 172.21.15.83:0/3837522102 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f907006da60 0x7f90701585b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:48.923+0000 7f90772c0640 1 --2- 172.21.15.83:0/3837522102 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f907006ce30 0x7f9070156e50 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:48.923+0000 7f90772c0640 1 -- 172.21.15.83:0/3837522102 >> 172.21.15.83:0/3837522102 conn(0x7f90700b1c80 msgr2=0x7f90700cbb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:48.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:48.924+0000 7f90772c0640 1 -- 172.21.15.83:0/3837522102 shutdown_connections 2024-03-06T07:28:48.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:48.924+0000 7f90772c0640 1 -- 172.21.15.83:0/3837522102 wait complete. 2024-03-06T07:28:48.939 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:28:49.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.040+0000 7fe8db59e640 1 Processor -- start 2024-03-06T07:28:49.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.040+0000 7fe8db59e640 1 -- start start 2024-03-06T07:28:49.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.041+0000 7fe8db59e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fe8dc0d1e70 0x7fe8dc06c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.041+0000 7fe8db59e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8dc06d090 con 0x7fe8dc06c8a0 2024-03-06T07:28:49.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.041+0000 7fe8db59e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8dc06d370 con 0x7fe8dc15de40 2024-03-06T07:28:49.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.041+0000 7fe8db59e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8dc0cfab0 con 0x7fe8dc0d1e70 2024-03-06T07:28:49.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.041+0000 7fe8dad9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fe8dc06c8a0 0x7fe8dc0d0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52168/0 (socket says 172.21.15.83:52168) 2024-03-06T07:28:49.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.041+0000 7fe8dad9d640 1 -- 172.21.15.83:0/3901986095 learned_addr learned my addr 172.21.15.83:0/3901986095 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:28:49.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.041+0000 7fe8d9d9b640 1 --2- 172.21.15.83:0/3901986095 >> v2:172.21.15.183:3300/0 conn(0x7fe8dc0d1e70 0x7fe8dc06c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.041+0000 7fe8d959a640 1 -- 172.21.15.83:0/3901986095 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1026067517 0 0) 0x7fe8dc06d090 con 0x7fe8dc06c8a0 2024-03-06T07:28:49.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.041+0000 7fe8d959a640 1 -- 172.21.15.83:0/3901986095 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8c4003520 con 0x7fe8dc06c8a0 2024-03-06T07:28:49.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.041+0000 7fe8d959a640 1 -- 172.21.15.83:0/3901986095 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1879562531 0 0) 0x7fe8c4003520 con 0x7fe8dc06c8a0 2024-03-06T07:28:49.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.042+0000 7fe8d959a640 1 -- 172.21.15.83:0/3901986095 >> v2:172.21.15.183:3300/0 conn(0x7fe8dc0d1e70 msgr2=0x7fe8dc06c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:49.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.042+0000 7fe8d959a640 1 --2- 172.21.15.83:0/3901986095 >> v2:172.21.15.183:3300/0 conn(0x7fe8dc0d1e70 0x7fe8dc06c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.042+0000 7fe8d959a640 1 -- 172.21.15.83:0/3901986095 >> v1:172.21.15.183:6789/0 conn(0x7fe8dc15de40 legacy=0x7fe8dc0d1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.042+0000 7fe8d959a640 1 -- 172.21.15.83:0/3901986095 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8dc06d650 con 0x7fe8dc06c8a0 2024-03-06T07:28:49.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.042+0000 7fe8d959a640 1 -- 172.21.15.83:0/3901986095 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7fe8d40019f0 con 0x7fe8dc06c8a0 2024-03-06T07:28:49.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.042+0000 7fe8d959a640 1 -- 172.21.15.83:0/3901986095 >> v1:172.21.15.83:6789/0 conn(0x7fe8dc06c8a0 legacy=0x7fe8dc0d0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.042+0000 7fe8d959a640 1 --2- 172.21.15.83:0/3901986095 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe8c4003a20 0x7fe8c4003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.042+0000 7fe8d959a640 1 --2- 172.21.15.83:0/3901986095 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8c4004800 0x7fe8c4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.042+0000 7fe8d959a640 1 --2- 172.21.15.83:0/3901986095 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe8c4005230 0x7fe8c400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.042+0000 7fe8d959a640 1 -- 172.21.15.83:0/3901986095 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c400c160 con 0x7fe8c4004800 2024-03-06T07:28:49.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.042+0000 7fe8d959a640 1 -- 172.21.15.83:0/3901986095 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c400c310 con 0x7fe8c4005230 2024-03-06T07:28:49.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.042+0000 7fe8d959a640 1 -- 172.21.15.83:0/3901986095 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c400c510 con 0x7fe8c4003a20 2024-03-06T07:28:49.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.042+0000 7fe8d9d9b640 1 --2- 172.21.15.83:0/3901986095 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8c4004800 0x7fe8c4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.042+0000 7fe8dad9d640 1 --2- 172.21.15.83:0/3901986095 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe8c4005230 0x7fe8c400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.042+0000 7fe8da59c640 1 --2- 172.21.15.83:0/3901986095 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe8c4003a20 0x7fe8c4003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.042+0000 7fe8d9d9b640 1 -- 172.21.15.83:0/3901986095 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe8c4003a20 msgr2=0x7fe8c4003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:49.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.042+0000 7fe8d9d9b640 1 --2- 172.21.15.83:0/3901986095 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe8c4003a20 0x7fe8c4003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.042+0000 7fe8d9d9b640 1 -- 172.21.15.83:0/3901986095 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe8c4005230 msgr2=0x7fe8c400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:49.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.042+0000 7fe8d9d9b640 1 --2- 172.21.15.83:0/3901986095 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe8c4005230 0x7fe8c400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.042+0000 7fe8dad9d640 1 -- 172.21.15.83:0/3901986095 reap_dead start 2024-03-06T07:28:49.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.042+0000 7fe8d9d9b640 1 -- 172.21.15.83:0/3901986095 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8c4003520 con 0x7fe8c4004800 2024-03-06T07:28:49.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.043+0000 7fe8d9d9b640 1 --2- 172.21.15.83:0/3901986095 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8c4004800 0x7fe8c4004c80 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fe8d0009ac0 tx=0x7fe8c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbda81133d396a15 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:49.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.043+0000 7fe8d959a640 1 -- 172.21.15.83:0/3901986095 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d0039070 con 0x7fe8c4004800 2024-03-06T07:28:49.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.043+0000 7fe8d959a640 1 -- 172.21.15.83:0/3901986095 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8d002c6b0 con 0x7fe8c4004800 2024-03-06T07:28:49.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.043+0000 7fe8d959a640 1 -- 172.21.15.83:0/3901986095 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d0009690 con 0x7fe8c4004800 2024-03-06T07:28:49.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.043+0000 7fe8db59e640 1 -- 172.21.15.83:0/3901986095 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8c4004800 msgr2=0x7fe8c4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.043+0000 7fe8db59e640 1 --2- 172.21.15.83:0/3901986095 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8c4004800 0x7fe8c4004c80 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fe8d0009ac0 tx=0x7fe8c4005830 comp rx=0 tx=0).stop 2024-03-06T07:28:49.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.043+0000 7fe8db59e640 1 -- 172.21.15.83:0/3901986095 shutdown_connections 2024-03-06T07:28:49.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.043+0000 7fe8db59e640 1 --2- 172.21.15.83:0/3901986095 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8c4004800 0x7fe8c4004c80 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.044+0000 7fe8db59e640 1 -- 172.21.15.83:0/3901986095 >> 172.21.15.83:0/3901986095 conn(0x7fe8dc09a9f0 msgr2=0x7fe8dc09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:49.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.044+0000 7fe8db59e640 1 -- 172.21.15.83:0/3901986095 shutdown_connections 2024-03-06T07:28:49.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.044+0000 7fe8db59e640 1 -- 172.21.15.83:0/3901986095 wait complete. 2024-03-06T07:28:49.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.045+0000 7fe8db59e640 1 Processor -- start 2024-03-06T07:28:49.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.045+0000 7fe8db59e640 1 -- start start 2024-03-06T07:28:49.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.046+0000 7fe8db59e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe8c4004800 0x7fe8dc146f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.046+0000 7fe8db59e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe8dc150fa0 0x7fe8dc15f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.046+0000 7fe8db59e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8dc15f930 0x7fe8dc15fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.046+0000 7fe8db59e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8dc06ccd0 con 0x7fe8dc15f930 2024-03-06T07:28:49.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.046+0000 7fe8db59e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8dc15e220 con 0x7fe8dc150fa0 2024-03-06T07:28:49.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.046+0000 7fe8db59e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8dc06ce50 con 0x7fe8c4004800 2024-03-06T07:28:49.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.046+0000 7fe8dad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8dc15f930 0x7fe8dc15fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.046+0000 7fe8dad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8dc15f930 0x7fe8dc15fdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48746/0 (socket says 172.21.15.83:48746) 2024-03-06T07:28:49.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.046+0000 7fe8dad9d640 1 -- 172.21.15.83:0/3273790346 learned_addr learned my addr 172.21.15.83:0/3273790346 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:49.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.046+0000 7fe8dad9d640 1 -- 172.21.15.83:0/3273790346 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe8c4004800 msgr2=0x7fe8dc146f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.046+0000 7fe8dad9d640 1 --2- 172.21.15.83:0/3273790346 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe8c4004800 0x7fe8dc146f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.046+0000 7fe8dad9d640 1 -- 172.21.15.83:0/3273790346 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe8dc150fa0 msgr2=0x7fe8dc15f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.046+0000 7fe8dad9d640 1 --2- 172.21.15.83:0/3273790346 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe8dc150fa0 0x7fe8dc15f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.046+0000 7fe8dad9d640 1 -- 172.21.15.83:0/3273790346 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8dc1f39a0 con 0x7fe8dc15f930 2024-03-06T07:28:49.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.046+0000 7fe8dad9d640 1 --2- 172.21.15.83:0/3273790346 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8dc15f930 0x7fe8dc15fdb0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc0cbe20 tx=0x7fe8d4007b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:49.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.046+0000 7fe8caffd640 1 -- 172.21.15.83:0/3273790346 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d400abe0 con 0x7fe8dc15f930 2024-03-06T07:28:49.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.046+0000 7fe8caffd640 1 -- 172.21.15.83:0/3273790346 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8d4019070 con 0x7fe8dc15f930 2024-03-06T07:28:49.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.046+0000 7fe8caffd640 1 -- 172.21.15.83:0/3273790346 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d40133f0 con 0x7fe8dc15f930 2024-03-06T07:28:49.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.046+0000 7fe8db59e640 1 -- 172.21.15.83:0/3273790346 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8dc0b39b0 con 0x7fe8dc15f930 2024-03-06T07:28:49.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.046+0000 7fe8db59e640 1 -- 172.21.15.83:0/3273790346 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8dc144980 con 0x7fe8dc15f930 2024-03-06T07:28:49.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.047+0000 7fe8db59e640 1 -- 172.21.15.83:0/3273790346 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8dc068f80 con 0x7fe8dc15f930 2024-03-06T07:28:49.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.048+0000 7fe8caffd640 1 -- 172.21.15.83:0/3273790346 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe8d4013590 con 0x7fe8dc15f930 2024-03-06T07:28:49.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.048+0000 7fe8caffd640 1 --2- 172.21.15.83:0/3273790346 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe8bc06b4d0 0x7fe8bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.048+0000 7fe8caffd640 1 -- 172.21.15.83:0/3273790346 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe8d408a2a0 con 0x7fe8dc15f930 2024-03-06T07:28:49.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.048+0000 7fe8da59c640 1 --2- 172.21.15.83:0/3273790346 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe8bc06b4d0 0x7fe8bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.049+0000 7fe8da59c640 1 --2- 172.21.15.83:0/3273790346 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe8bc06b4d0 0x7fe8bc06d990 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc000c00 tx=0x7fe8cc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:49.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.053+0000 7fe8caffd640 1 -- 172.21.15.83:0/3273790346 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe8d4059500 con 0x7fe8dc15f930 2024-03-06T07:28:49.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.287+0000 7fe8db59e640 1 -- 172.21.15.83:0/3273790346 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fe8dc0b77f0 con 0x7fe8dc15f930 2024-03-06T07:28:49.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.287+0000 7fe8caffd640 1 -- 172.21.15.83:0/3273790346 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fe8d4058e00 con 0x7fe8dc15f930 2024-03-06T07:28:49.289 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:28:49.289 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:28:49.289 INFO:teuthology.orchestra.run.smithi083.stdout: "auth_dump": [ 2024-03-06T07:28:49.290 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:28:49.290 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mds.a", 2024-03-06T07:28:49.290 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlVHaUGhAAVRpLsHYTUkaYDmceq5Rlgw==", 2024-03-06T07:28:49.290 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:28:49.290 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:28:49.290 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow *", 2024-03-06T07:28:49.290 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow *", 2024-03-06T07:28:49.290 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:28:49.290 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:28:49.290 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:28:49.290 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:28:49.290 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mds.b", 2024-03-06T07:28:49.291 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlFlrrJhAAQbY6xU910eYRg1zwgpj9vg==", 2024-03-06T07:28:49.291 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:28:49.291 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:28:49.291 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow *", 2024-03-06T07:28:49.291 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow *", 2024-03-06T07:28:49.291 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:28:49.291 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:28:49.291 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:28:49.291 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:28:49.291 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mds.c", 2024-03-06T07:28:49.291 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlX0jxIRAAxYNi39qxIFXazlL2qNzLCg==", 2024-03-06T07:28:49.291 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:28:49.291 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:28:49.292 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow *", 2024-03-06T07:28:49.292 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow *", 2024-03-06T07:28:49.292 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:28:49.292 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:28:49.292 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:28:49.292 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:28:49.292 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mds.d", 2024-03-06T07:28:49.292 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehl3utCLhAAzEjASLnYpPYrJsWOt3WwpQ==", 2024-03-06T07:28:49.292 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:28:49.292 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:28:49.292 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow *", 2024-03-06T07:28:49.292 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow *", 2024-03-06T07:28:49.292 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:28:49.292 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:28:49.293 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:28:49.293 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:28:49.293 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.0", 2024-03-06T07:28:49.293 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBgGehlkezBExAAZOLMH2ua3CTWAL7H6LGeZw==", 2024-03-06T07:28:49.293 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:28:49.293 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:49.293 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:28:49.293 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:28:49.293 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:28:49.293 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:28:49.293 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:28:49.293 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.1", 2024-03-06T07:28:49.293 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBjGehlC04qEhAAqirTFR6y3NOGw6N+iz3/pg==", 2024-03-06T07:28:49.294 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:28:49.294 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:49.294 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:28:49.294 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:28:49.294 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:28:49.294 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:28:49.294 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:28:49.294 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.2", 2024-03-06T07:28:49.294 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBmGehlfuFWEBAAj/vjZXrv1wVNSJ1R74UmnQ==", 2024-03-06T07:28:49.294 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:28:49.294 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:49.294 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:28:49.294 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:28:49.295 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:28:49.295 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:28:49.295 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:28:49.295 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.3", 2024-03-06T07:28:49.295 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBpGehl5+ItDhAA0YNc83MhKcgb3JgRuP87aw==", 2024-03-06T07:28:49.295 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:28:49.295 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:49.295 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:28:49.295 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:28:49.295 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:28:49.295 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:28:49.295 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:28:49.295 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.4", 2024-03-06T07:28:49.295 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBvGehlYa/TFBAAvGcizpRQ21pvgpEGgSET0w==", 2024-03-06T07:28:49.296 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:28:49.296 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:49.296 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:28:49.296 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:28:49.296 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:28:49.296 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:28:49.296 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:28:49.296 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.5", 2024-03-06T07:28:49.296 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQByGehlE80TExAAFcWDgnLtW8OKlyu4B0jnmg==", 2024-03-06T07:28:49.296 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:28:49.296 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:49.296 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:28:49.296 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:28:49.297 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:28:49.297 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:28:49.297 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:28:49.297 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.6", 2024-03-06T07:28:49.297 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQB1Gehla/LbERAAYfAjTKL11VMVnGUGV5GVcg==", 2024-03-06T07:28:49.297 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:28:49.297 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:49.297 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:28:49.297 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:28:49.297 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:28:49.297 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:28:49.297 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:28:49.297 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "osd.7", 2024-03-06T07:28:49.297 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQB4GehlpSJkEBAA17tUgMf94jfqvXlD51P8DQ==", 2024-03-06T07:28:49.297 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:28:49.298 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:49.298 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile osd", 2024-03-06T07:28:49.298 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:28:49.298 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:28:49.298 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:28:49.298 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:28:49.298 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.0", 2024-03-06T07:28:49.298 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlRwCIMhAA1DdbdQZPplu7LQOTK3BmSg==", 2024-03-06T07:28:49.298 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:28:49.298 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:28:49.298 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow r", 2024-03-06T07:28:49.298 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:28:49.298 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:28:49.298 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:28:49.298 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:28:49.298 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.1", 2024-03-06T07:28:49.298 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBdGehl77spABAAsiEf+tXos3MwFrXToEQ8dQ==", 2024-03-06T07:28:49.299 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:28:49.299 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:28:49.299 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow r", 2024-03-06T07:28:49.299 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow rw", 2024-03-06T07:28:49.299 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow rwx" 2024-03-06T07:28:49.299 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:28:49.299 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:28:49.299 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:28:49.299 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.2", 2024-03-06T07:28:49.299 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehltY5SOBAAI8rcIEVqMZaWQGgNaYKJTA==", 2024-03-06T07:28:49.299 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:28:49.299 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:28:49.299 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow r", 2024-03-06T07:28:49.299 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow rw", 2024-03-06T07:28:49.300 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow rwx" 2024-03-06T07:28:49.300 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:28:49.300 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:28:49.300 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:28:49.300 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.3", 2024-03-06T07:28:49.300 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBdGehl9ZX1BRAAA7fKnRyaQ+zqHoSGs4UvDw==", 2024-03-06T07:28:49.300 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:28:49.300 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow", 2024-03-06T07:28:49.300 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow r", 2024-03-06T07:28:49.300 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow rw", 2024-03-06T07:28:49.300 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow rwx" 2024-03-06T07:28:49.300 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:28:49.300 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:28:49.300 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:28:49.301 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "client.admin", 2024-03-06T07:28:49.301 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBbGehl4gqRORAAobqSWoeyHYA/lXkp6/EcIQ==", 2024-03-06T07:28:49.301 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:28:49.301 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow *", 2024-03-06T07:28:49.301 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": "allow *", 2024-03-06T07:28:49.301 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow *", 2024-03-06T07:28:49.301 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:28:49.301 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:28:49.301 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:28:49.301 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:28:49.301 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mgr.x", 2024-03-06T07:28:49.301 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlfikrFxAAoZkTUtpHlBcUAMpwWABRxg==", 2024-03-06T07:28:49.302 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:28:49.302 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow *", 2024-03-06T07:28:49.302 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile mgr", 2024-03-06T07:28:49.302 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:28:49.302 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:28:49.302 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:28:49.302 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:28:49.302 INFO:teuthology.orchestra.run.smithi083.stdout: "entity": "mgr.y", 2024-03-06T07:28:49.302 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "AQBcGehlvdcdExAAySXlEfJqLnpMySiiYGFkMQ==", 2024-03-06T07:28:49.302 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": { 2024-03-06T07:28:49.302 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": "allow *", 2024-03-06T07:28:49.302 INFO:teuthology.orchestra.run.smithi083.stdout: "mon": "allow profile mgr", 2024-03-06T07:28:49.302 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": "allow *" 2024-03-06T07:28:49.302 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:28:49.302 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:28:49.303 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-03-06T07:28:49.303 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:28:49.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.289+0000 7fe8db59e640 1 -- 172.21.15.83:0/3273790346 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe8bc06b4d0 msgr2=0x7fe8bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.289+0000 7fe8db59e640 1 --2- 172.21.15.83:0/3273790346 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe8bc06b4d0 0x7fe8bc06d990 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc000c00 tx=0x7fe8cc009040 comp rx=0 tx=0).stop 2024-03-06T07:28:49.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.289+0000 7fe8db59e640 1 -- 172.21.15.83:0/3273790346 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8dc15f930 msgr2=0x7fe8dc15fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.289+0000 7fe8db59e640 1 --2- 172.21.15.83:0/3273790346 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8dc15f930 0x7fe8dc15fdb0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc0cbe20 tx=0x7fe8d4007b00 comp rx=0 tx=0).stop 2024-03-06T07:28:49.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.289+0000 7fe8db59e640 1 -- 172.21.15.83:0/3273790346 shutdown_connections 2024-03-06T07:28:49.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.289+0000 7fe8db59e640 1 --2- 172.21.15.83:0/3273790346 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe8bc06b4d0 0x7fe8bc06d990 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.289+0000 7fe8db59e640 1 --2- 172.21.15.83:0/3273790346 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8dc15f930 0x7fe8dc15fdb0 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.289+0000 7fe8db59e640 1 --2- 172.21.15.83:0/3273790346 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe8dc150fa0 0x7fe8dc15f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.289+0000 7fe8db59e640 1 --2- 172.21.15.83:0/3273790346 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe8c4004800 0x7fe8dc146f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.289+0000 7fe8db59e640 1 -- 172.21.15.83:0/3273790346 >> 172.21.15.83:0/3273790346 conn(0x7fe8dc09a9f0 msgr2=0x7fe8dc0af160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:49.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.289+0000 7fe8db59e640 1 -- 172.21.15.83:0/3273790346 shutdown_connections 2024-03-06T07:28:49.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.289+0000 7fe8db59e640 1 -- 172.21.15.83:0/3273790346 wait complete. 2024-03-06T07:28:49.305 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:49.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.403+0000 7efefad9a640 1 Processor -- start 2024-03-06T07:28:49.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.403+0000 7efefad9a640 1 -- start start 2024-03-06T07:28:49.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.404+0000 7efefad9a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7efef40d00c0 0x7efef40d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.404+0000 7efefad9a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efef406cf20 0x7efef406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.404+0000 7efefad9a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efef40d09e0 con 0x7efef40cf850 2024-03-06T07:28:49.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.404+0000 7efefad9a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efef406ca90 con 0x7efef40d00c0 2024-03-06T07:28:49.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.404+0000 7efefad9a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efef415e380 con 0x7efef406cf20 2024-03-06T07:28:49.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.404+0000 7efef8b36640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7efef40d00c0 0x7efef40d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.404+0000 7efef8b36640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7efef40d00c0 0x7efef40d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48752/0 (socket says 172.21.15.83:48752) 2024-03-06T07:28:49.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.404+0000 7efef8b36640 1 -- 172.21.15.83:0/2520395115 learned_addr learned my addr 172.21.15.83:0/2520395115 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:49.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.404+0000 7efef9337640 1 --2- 172.21.15.83:0/2520395115 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efef406cf20 0x7efef406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.405+0000 7efef8b36640 1 -- 172.21.15.83:0/2520395115 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efef406cf20 msgr2=0x7efef406d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:49.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.405+0000 7efef8b36640 1 --2- 172.21.15.83:0/2520395115 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efef406cf20 0x7efef406d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.405+0000 7efef8b36640 1 -- 172.21.15.83:0/2520395115 >> v1:172.21.15.83:6789/0 conn(0x7efef40cf850 legacy=0x7efef40cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.405+0000 7efef8b36640 1 -- 172.21.15.83:0/2520395115 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef40cdc10 con 0x7efef40d00c0 2024-03-06T07:28:49.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.405+0000 7efef37fe640 1 -- 172.21.15.83:0/2520395115 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 990909802 0 0) 0x7efef40d09e0 con 0x7efef40cf850 2024-03-06T07:28:49.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.405+0000 7efef8b36640 1 --2- 172.21.15.83:0/2520395115 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efef40d00c0 0x7efef40d04a0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7efee4009870 tx=0x7efee402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c61cc547fd50f29 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:49.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.405+0000 7efef37fe640 1 -- 172.21.15.83:0/2520395115 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee4034070 con 0x7efef40d00c0 2024-03-06T07:28:49.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.406+0000 7efef37fe640 1 -- 172.21.15.83:0/2520395115 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efee4030840 con 0x7efef40d00c0 2024-03-06T07:28:49.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.406+0000 7efef37fe640 1 -- 172.21.15.83:0/2520395115 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee4030c10 con 0x7efef40d00c0 2024-03-06T07:28:49.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.406+0000 7efefad9a640 1 -- 172.21.15.83:0/2520395115 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efef40d00c0 msgr2=0x7efef40d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.406+0000 7efefad9a640 1 --2- 172.21.15.83:0/2520395115 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efef40d00c0 0x7efef40d04a0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7efee4009870 tx=0x7efee402b980 comp rx=0 tx=0).stop 2024-03-06T07:28:49.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.406+0000 7efefad9a640 1 -- 172.21.15.83:0/2520395115 shutdown_connections 2024-03-06T07:28:49.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.406+0000 7efefad9a640 1 --2- 172.21.15.83:0/2520395115 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efef406cf20 0x7efef406d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.406+0000 7efefad9a640 1 --2- 172.21.15.83:0/2520395115 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efef40d00c0 0x7efef40d04a0 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.406+0000 7efefad9a640 1 -- 172.21.15.83:0/2520395115 >> 172.21.15.83:0/2520395115 conn(0x7efef40b35a0 msgr2=0x7efef40b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:49.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.406+0000 7efefad9a640 1 -- 172.21.15.83:0/2520395115 shutdown_connections 2024-03-06T07:28:49.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.406+0000 7efefad9a640 1 -- 172.21.15.83:0/2520395115 wait complete. 2024-03-06T07:28:49.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.407+0000 7efefad9a640 1 Processor -- start 2024-03-06T07:28:49.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.407+0000 7efefad9a640 1 -- start start 2024-03-06T07:28:49.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.408+0000 7efefad9a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efef406cf20 0x7efef41f47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.408+0000 7efefad9a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efef40cf850 0x7efef41f3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.408+0000 7efefad9a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efef40d00c0 0x7efef415e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.408+0000 7efefad9a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef406c720 con 0x7efef40d00c0 2024-03-06T07:28:49.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.408+0000 7efefad9a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef406c440 con 0x7efef406cf20 2024-03-06T07:28:49.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.408+0000 7efefad9a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efef40b1030 con 0x7efef40cf850 2024-03-06T07:28:49.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.408+0000 7efef9337640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efef40d00c0 0x7efef415e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.408+0000 7efef9337640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efef40d00c0 0x7efef415e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48766/0 (socket says 172.21.15.83:48766) 2024-03-06T07:28:49.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.408+0000 7efef9337640 1 -- 172.21.15.83:0/3423696657 learned_addr learned my addr 172.21.15.83:0/3423696657 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:49.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.408+0000 7efef3fff640 1 --2- 172.21.15.83:0/3423696657 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efef40cf850 0x7efef41f3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.408+0000 7efef9337640 1 -- 172.21.15.83:0/3423696657 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efef40cf850 msgr2=0x7efef41f3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.408+0000 7efef9337640 1 --2- 172.21.15.83:0/3423696657 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efef40cf850 0x7efef41f3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.408+0000 7efef8b36640 1 --2- 172.21.15.83:0/3423696657 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efef406cf20 0x7efef41f47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.408+0000 7efef9337640 1 -- 172.21.15.83:0/3423696657 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efef406cf20 msgr2=0x7efef41f47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.408+0000 7efef9337640 1 --2- 172.21.15.83:0/3423696657 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efef406cf20 0x7efef41f47c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.408+0000 7efef9337640 1 -- 172.21.15.83:0/3423696657 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef415ec90 con 0x7efef40d00c0 2024-03-06T07:28:49.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.409+0000 7efef3fff640 1 --2- 172.21.15.83:0/3423696657 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efef40cf850 0x7efef41f3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:49.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.409+0000 7efef8b36640 1 --2- 172.21.15.83:0/3423696657 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efef406cf20 0x7efef41f47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:49.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.409+0000 7efef9337640 1 --2- 172.21.15.83:0/3423696657 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efef40d00c0 0x7efef415e590 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7efeec007e90 tx=0x7efeec00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:49.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.409+0000 7efef17fa640 1 -- 172.21.15.83:0/3423696657 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeec014420 con 0x7efef40d00c0 2024-03-06T07:28:49.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.409+0000 7efef17fa640 1 -- 172.21.15.83:0/3423696657 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeec019070 con 0x7efef40d00c0 2024-03-06T07:28:49.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.409+0000 7efefad9a640 1 -- 172.21.15.83:0/3423696657 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efef415ef70 con 0x7efef40d00c0 2024-03-06T07:28:49.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.409+0000 7efef17fa640 1 -- 172.21.15.83:0/3423696657 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeec014800 con 0x7efef40d00c0 2024-03-06T07:28:49.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.409+0000 7efefad9a640 1 -- 172.21.15.83:0/3423696657 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efef415f530 con 0x7efef40d00c0 2024-03-06T07:28:49.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.410+0000 7efefad9a640 1 -- 172.21.15.83:0/3423696657 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efef404f330 con 0x7efef40d00c0 2024-03-06T07:28:49.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.410+0000 7efef17fa640 1 -- 172.21.15.83:0/3423696657 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efeec0149a0 con 0x7efef40d00c0 2024-03-06T07:28:49.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.411+0000 7efef17fa640 1 --2- 172.21.15.83:0/3423696657 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efee006b4d0 0x7efee006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.411+0000 7efef8b36640 1 --2- 172.21.15.83:0/3423696657 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efee006b4d0 0x7efee006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.411+0000 7efef17fa640 1 -- 172.21.15.83:0/3423696657 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 4857+0+0 (secure 0 0 0) 0x7efeec08a6e0 con 0x7efef40d00c0 2024-03-06T07:28:49.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.411+0000 7efef8b36640 1 --2- 172.21.15.83:0/3423696657 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efee006b4d0 0x7efee006d990 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7efee4036000 tx=0x7efee403b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:49.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.415+0000 7efef17fa640 1 -- 172.21.15.83:0/3423696657 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efeec0598b0 con 0x7efef40d00c0 2024-03-06T07:28:49.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.588+0000 7efefad9a640 1 -- 172.21.15.83:0/3423696657 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efef40b19f0 con 0x7efef40d00c0 2024-03-06T07:28:49.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.589+0000 7efef17fa640 1 -- 172.21.15.83:0/3423696657 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v124) v1 ==== 74+0+12913 (secure 0 0 0) 0x7efeec0591b0 con 0x7efef40d00c0 2024-03-06T07:28:49.591 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:28:49.591 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":124,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:28:48.920283+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:49.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.591+0000 7efefad9a640 1 -- 172.21.15.83:0/3423696657 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efee006b4d0 msgr2=0x7efee006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.591+0000 7efefad9a640 1 --2- 172.21.15.83:0/3423696657 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efee006b4d0 0x7efee006d990 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7efee4036000 tx=0x7efee403b040 comp rx=0 tx=0).stop 2024-03-06T07:28:49.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.591+0000 7efefad9a640 1 -- 172.21.15.83:0/3423696657 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efef40d00c0 msgr2=0x7efef415e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.591+0000 7efefad9a640 1 --2- 172.21.15.83:0/3423696657 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efef40d00c0 0x7efef415e590 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7efeec007e90 tx=0x7efeec00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.591+0000 7efefad9a640 1 -- 172.21.15.83:0/3423696657 shutdown_connections 2024-03-06T07:28:49.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.591+0000 7efefad9a640 1 --2- 172.21.15.83:0/3423696657 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efee006b4d0 0x7efee006d990 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.591+0000 7efefad9a640 1 --2- 172.21.15.83:0/3423696657 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efef40d00c0 0x7efef415e590 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.591+0000 7efefad9a640 1 --2- 172.21.15.83:0/3423696657 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efef40cf850 0x7efef41f3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.591+0000 7efefad9a640 1 --2- 172.21.15.83:0/3423696657 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efef406cf20 0x7efef41f47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.591+0000 7efefad9a640 1 -- 172.21.15.83:0/3423696657 >> 172.21.15.83:0/3423696657 conn(0x7efef40b35a0 msgr2=0x7efef40cde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:49.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.591+0000 7efefad9a640 1 -- 172.21.15.83:0/3423696657 shutdown_connections 2024-03-06T07:28:49.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.591+0000 7efefad9a640 1 -- 172.21.15.83:0/3423696657 wait complete. 2024-03-06T07:28:49.604 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:49.604 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:49.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.705+0000 7fa3f361c640 1 Processor -- start 2024-03-06T07:28:49.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.705+0000 7fa3f361c640 1 -- start start 2024-03-06T07:28:49.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.706+0000 7fa3f361c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa3ec15bb40 0x7fa3ec0b4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.706+0000 7fa3f361c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3ec15ff90 con 0x7fa3ec0cfa90 2024-03-06T07:28:49.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.706+0000 7fa3f361c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3ec160270 con 0x7fa3ec15f7d0 2024-03-06T07:28:49.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.706+0000 7fa3f361c640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3ec06c910 con 0x7fa3ec15bb40 2024-03-06T07:28:49.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.706+0000 7fa3f13b8640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa3ec15bb40 0x7fa3ec0b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.706+0000 7fa3f13b8640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa3ec15bb40 0x7fa3ec0b4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48774/0 (socket says 172.21.15.83:48774) 2024-03-06T07:28:49.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.706+0000 7fa3f1bb9640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fa3ec0cfa90 0x7fa3ec0cdcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52176/0 (socket says 172.21.15.83:52176) 2024-03-06T07:28:49.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.706+0000 7fa3f13b8640 1 -- 172.21.15.83:0/3381997080 learned_addr learned my addr 172.21.15.83:0/3381997080 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:49.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.706+0000 7fa3f13b8640 1 -- 172.21.15.83:0/3381997080 >> v1:172.21.15.183:6789/0 conn(0x7fa3ec15f7d0 legacy=0x7fa3ec06d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:49.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.706+0000 7fa3f13b8640 1 -- 172.21.15.83:0/3381997080 >> v1:172.21.15.83:6789/0 conn(0x7fa3ec0cfa90 legacy=0x7fa3ec0cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.706+0000 7fa3f13b8640 1 -- 172.21.15.83:0/3381997080 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3ec160550 con 0x7fa3ec15bb40 2024-03-06T07:28:49.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.706+0000 7fa3e3fff640 1 -- 172.21.15.83:0/3381997080 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3818223693 0 0) 0x7fa3ec15ff90 con 0x7fa3ec0cfa90 2024-03-06T07:28:49.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.707+0000 7fa3f13b8640 1 --2- 172.21.15.83:0/3381997080 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3ec15bb40 0x7fa3ec0b4970 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec06ca90 tx=0x7fa3e802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b5a86a03b0f699b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:49.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.707+0000 7fa3e3fff640 1 -- 172.21.15.83:0/3381997080 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e8034070 con 0x7fa3ec15bb40 2024-03-06T07:28:49.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.707+0000 7fa3e3fff640 1 -- 172.21.15.83:0/3381997080 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3e8030840 con 0x7fa3ec15bb40 2024-03-06T07:28:49.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.707+0000 7fa3e3fff640 1 -- 172.21.15.83:0/3381997080 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e8030c10 con 0x7fa3ec15bb40 2024-03-06T07:28:49.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.708+0000 7fa3f361c640 1 -- 172.21.15.83:0/3381997080 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3ec15bb40 msgr2=0x7fa3ec0b4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.708+0000 7fa3f361c640 1 --2- 172.21.15.83:0/3381997080 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3ec15bb40 0x7fa3ec0b4970 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec06ca90 tx=0x7fa3e802b980 comp rx=0 tx=0).stop 2024-03-06T07:28:49.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.708+0000 7fa3f361c640 1 -- 172.21.15.83:0/3381997080 shutdown_connections 2024-03-06T07:28:49.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.708+0000 7fa3f361c640 1 --2- 172.21.15.83:0/3381997080 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3ec15bb40 0x7fa3ec0b4970 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.708+0000 7fa3f361c640 1 -- 172.21.15.83:0/3381997080 >> 172.21.15.83:0/3381997080 conn(0x7fa3ec06b190 msgr2=0x7fa3ec06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:49.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.708+0000 7fa3f361c640 1 -- 172.21.15.83:0/3381997080 shutdown_connections 2024-03-06T07:28:49.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.708+0000 7fa3f361c640 1 -- 172.21.15.83:0/3381997080 wait complete. 2024-03-06T07:28:49.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.709+0000 7fa3f361c640 1 Processor -- start 2024-03-06T07:28:49.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.709+0000 7fa3f361c640 1 -- start start 2024-03-06T07:28:49.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.710+0000 7fa3f361c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa3ec0cfa90 0x7fa3ec143c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.710+0000 7fa3f361c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3ec15bb40 0x7fa3ec157cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.710+0000 7fa3f361c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa3ec15f7d0 0x7fa3ec1581f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.710+0000 7fa3f361c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ec06d790 con 0x7fa3ec15bb40 2024-03-06T07:28:49.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.710+0000 7fa3f361c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ec15fc30 con 0x7fa3ec0cfa90 2024-03-06T07:28:49.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.710+0000 7fa3f361c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ec15bf20 con 0x7fa3ec15f7d0 2024-03-06T07:28:49.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.710+0000 7fa3f0bb7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3ec15bb40 0x7fa3ec157cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.710+0000 7fa3f0bb7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3ec15bb40 0x7fa3ec157cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48776/0 (socket says 172.21.15.83:48776) 2024-03-06T07:28:49.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.710+0000 7fa3f0bb7640 1 -- 172.21.15.83:0/4213323689 learned_addr learned my addr 172.21.15.83:0/4213323689 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:49.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.710+0000 7fa3f0bb7640 1 -- 172.21.15.83:0/4213323689 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa3ec15f7d0 msgr2=0x7fa3ec1581f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.710+0000 7fa3f0bb7640 1 --2- 172.21.15.83:0/4213323689 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa3ec15f7d0 0x7fa3ec1581f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.710+0000 7fa3f0bb7640 1 -- 172.21.15.83:0/4213323689 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa3ec0cfa90 msgr2=0x7fa3ec143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.710+0000 7fa3f0bb7640 1 --2- 172.21.15.83:0/4213323689 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa3ec0cfa90 0x7fa3ec143c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.710+0000 7fa3f0bb7640 1 -- 172.21.15.83:0/4213323689 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3ec15a850 con 0x7fa3ec15bb40 2024-03-06T07:28:49.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.710+0000 7fa3f0bb7640 1 --2- 172.21.15.83:0/4213323689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3ec15bb40 0x7fa3ec157cb0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc00b420 tx=0x7fa3dc00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:49.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.710+0000 7fa3e1ffb640 1 -- 172.21.15.83:0/4213323689 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3dc011830 con 0x7fa3ec15bb40 2024-03-06T07:28:49.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.710+0000 7fa3e1ffb640 1 -- 172.21.15.83:0/4213323689 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3dc016070 con 0x7fa3ec15bb40 2024-03-06T07:28:49.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.710+0000 7fa3f361c640 1 -- 172.21.15.83:0/4213323689 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3ec15ab30 con 0x7fa3ec15bb40 2024-03-06T07:28:49.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.710+0000 7fa3e1ffb640 1 -- 172.21.15.83:0/4213323689 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3dc011c10 con 0x7fa3ec15bb40 2024-03-06T07:28:49.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.711+0000 7fa3f361c640 1 -- 172.21.15.83:0/4213323689 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3ec15b0f0 con 0x7fa3ec15bb40 2024-03-06T07:28:49.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.711+0000 7fa3f361c640 1 -- 172.21.15.83:0/4213323689 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3ec0732b0 con 0x7fa3ec15bb40 2024-03-06T07:28:49.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.712+0000 7fa3e1ffb640 1 -- 172.21.15.83:0/4213323689 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa3dc01b460 con 0x7fa3ec15bb40 2024-03-06T07:28:49.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.712+0000 7fa3e1ffb640 1 --2- 172.21.15.83:0/4213323689 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa3d006b4d0 0x7fa3d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.712+0000 7fa3e1ffb640 1 -- 172.21.15.83:0/4213323689 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa3dc089960 con 0x7fa3ec15bb40 2024-03-06T07:28:49.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.712+0000 7fa3f13b8640 1 --2- 172.21.15.83:0/4213323689 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa3d006b4d0 0x7fa3d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.713+0000 7fa3f13b8640 1 --2- 172.21.15.83:0/4213323689 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa3d006b4d0 0x7fa3d006d990 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fa3e8036000 tx=0x7fa3e8030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:49.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.716+0000 7fa3e1ffb640 1 -- 172.21.15.83:0/4213323689 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa3dc020020 con 0x7fa3ec15bb40 2024-03-06T07:28:49.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.887+0000 7fa3f361c640 1 -- 172.21.15.83:0/4213323689 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa3ec068ff0 con 0x7fa3ec15bb40 2024-03-06T07:28:49.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.887+0000 7fa3e1ffb640 1 -- 172.21.15.83:0/4213323689 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v124) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa3dc058ba0 con 0x7fa3ec15bb40 2024-03-06T07:28:49.889 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:28:49.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.889+0000 7fa3f361c640 1 -- 172.21.15.83:0/4213323689 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa3d006b4d0 msgr2=0x7fa3d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.889+0000 7fa3f361c640 1 --2- 172.21.15.83:0/4213323689 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa3d006b4d0 0x7fa3d006d990 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fa3e8036000 tx=0x7fa3e8030000 comp rx=0 tx=0).stop 2024-03-06T07:28:49.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.889+0000 7fa3f361c640 1 -- 172.21.15.83:0/4213323689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3ec15bb40 msgr2=0x7fa3ec157cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.889+0000 7fa3f361c640 1 --2- 172.21.15.83:0/4213323689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3ec15bb40 0x7fa3ec157cb0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc00b420 tx=0x7fa3dc00b8e0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.889+0000 7fa3f361c640 1 -- 172.21.15.83:0/4213323689 shutdown_connections 2024-03-06T07:28:49.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.889+0000 7fa3f361c640 1 --2- 172.21.15.83:0/4213323689 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa3d006b4d0 0x7fa3d006d990 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.889+0000 7fa3f361c640 1 --2- 172.21.15.83:0/4213323689 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa3ec15f7d0 0x7fa3ec1581f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.889+0000 7fa3f361c640 1 --2- 172.21.15.83:0/4213323689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3ec15bb40 0x7fa3ec157cb0 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.889+0000 7fa3f361c640 1 --2- 172.21.15.83:0/4213323689 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa3ec0cfa90 0x7fa3ec143c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.889+0000 7fa3f361c640 1 -- 172.21.15.83:0/4213323689 >> 172.21.15.83:0/4213323689 conn(0x7fa3ec06b190 msgr2=0x7fa3ec156200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:49.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.889+0000 7fa3f361c640 1 -- 172.21.15.83:0/4213323689 shutdown_connections 2024-03-06T07:28:49.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:49.889+0000 7fa3f361c640 1 -- 172.21.15.83:0/4213323689 wait complete. 2024-03-06T07:28:49.902 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:28:50.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.001+0000 7ff548888640 1 Processor -- start 2024-03-06T07:28:50.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.001+0000 7ff548888640 1 -- start start 2024-03-06T07:28:50.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.001+0000 7ff548888640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff54415e0f0 0x7ff5440cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.002+0000 7ff548888640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ff5440cf850 0x7ff5440cff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.002+0000 7ff548888640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff54415e4d0 con 0x7ff5440d0490 2024-03-06T07:28:50.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.002+0000 7ff548888640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5440b1410 con 0x7ff5440cf850 2024-03-06T07:28:50.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.002+0000 7ff548888640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff54406c8b0 con 0x7ff54415e0f0 2024-03-06T07:28:50.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.002+0000 7ff539d9b640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ff5440cf850 0x7ff5440cff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.002+0000 7ff54259c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff54415e0f0 0x7ff5440cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.002+0000 7ff542d9d640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7ff5440d0490 0x7ff5440b25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:41862/0 (socket says 172.21.15.83:41862) 2024-03-06T07:28:50.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.002+0000 7ff542d9d640 1 -- 172.21.15.83:0/3085615458 learned_addr learned my addr 172.21.15.83:0/3085615458 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:28:50.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.002+0000 7ff54259c640 1 -- 172.21.15.83:0/3085615458 >> v2:172.21.15.183:3300/0 conn(0x7ff5440cf850 msgr2=0x7ff5440cff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:50.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.002+0000 7ff54259c640 1 --2- 172.21.15.83:0/3085615458 >> v2:172.21.15.183:3300/0 conn(0x7ff5440cf850 0x7ff5440cff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.002+0000 7ff54259c640 1 -- 172.21.15.83:0/3085615458 >> v1:172.21.15.183:6789/0 conn(0x7ff5440d0490 legacy=0x7ff5440b25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.002+0000 7ff54259c640 1 -- 172.21.15.83:0/3085615458 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5440cdac0 con 0x7ff54415e0f0 2024-03-06T07:28:50.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.002+0000 7ff539d9b640 1 --2- 172.21.15.83:0/3085615458 >> v2:172.21.15.183:3300/0 conn(0x7ff5440cf850 0x7ff5440cff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:50.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.003+0000 7ff54259c640 1 --2- 172.21.15.83:0/3085615458 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff54415e0f0 0x7ff5440cd3c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff5340097a0 tx=0x7ff53402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d13299d1dd5a53c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:50.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.003+0000 7ff541d9b640 1 -- 172.21.15.83:0/3085615458 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff534035020 con 0x7ff54415e0f0 2024-03-06T07:28:50.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.003+0000 7ff541d9b640 1 -- 172.21.15.83:0/3085615458 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff534031830 con 0x7ff54415e0f0 2024-03-06T07:28:50.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.003+0000 7ff541d9b640 1 -- 172.21.15.83:0/3085615458 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff534031c00 con 0x7ff54415e0f0 2024-03-06T07:28:50.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.004+0000 7ff548888640 1 -- 172.21.15.83:0/3085615458 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff54415e0f0 msgr2=0x7ff5440cd3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.004+0000 7ff548888640 1 --2- 172.21.15.83:0/3085615458 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff54415e0f0 0x7ff5440cd3c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff5340097a0 tx=0x7ff53402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.004+0000 7ff548888640 1 -- 172.21.15.83:0/3085615458 shutdown_connections 2024-03-06T07:28:50.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.004+0000 7ff548888640 1 --2- 172.21.15.83:0/3085615458 >> v2:172.21.15.183:3300/0 conn(0x7ff5440cf850 0x7ff5440cff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.004+0000 7ff548888640 1 --2- 172.21.15.83:0/3085615458 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff54415e0f0 0x7ff5440cd3c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.004+0000 7ff548888640 1 -- 172.21.15.83:0/3085615458 >> 172.21.15.83:0/3085615458 conn(0x7ff5440b35a0 msgr2=0x7ff5440b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:50.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.004+0000 7ff548888640 1 -- 172.21.15.83:0/3085615458 shutdown_connections 2024-03-06T07:28:50.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.004+0000 7ff548888640 1 -- 172.21.15.83:0/3085615458 wait complete. 2024-03-06T07:28:50.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.005+0000 7ff548888640 1 Processor -- start 2024-03-06T07:28:50.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.005+0000 7ff548888640 1 -- start start 2024-03-06T07:28:50.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.006+0000 7ff548888640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5440cf850 0x7ff544156f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.006+0000 7ff548888640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff5440d0490 0x7ff54409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.006+0000 7ff548888640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff5440ca0a0 0x7ff5440ca530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.006+0000 7ff548888640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5440d0cc0 con 0x7ff5440cf850 2024-03-06T07:28:50.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.006+0000 7ff548888640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff54406c540 con 0x7ff5440d0490 2024-03-06T07:28:50.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.006+0000 7ff548888640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5440b2cc0 con 0x7ff5440ca0a0 2024-03-06T07:28:50.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.006+0000 7ff54259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5440cf850 0x7ff544156f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.006+0000 7ff54259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5440cf850 0x7ff544156f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48790/0 (socket says 172.21.15.83:48790) 2024-03-06T07:28:50.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.006+0000 7ff54259c640 1 -- 172.21.15.83:0/2756159334 learned_addr learned my addr 172.21.15.83:0/2756159334 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:50.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.006+0000 7ff542d9d640 1 --2- 172.21.15.83:0/2756159334 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff5440ca0a0 0x7ff5440ca530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.006+0000 7ff539d9b640 1 --2- 172.21.15.83:0/2756159334 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff5440d0490 0x7ff54409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.007+0000 7ff54259c640 1 -- 172.21.15.83:0/2756159334 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff5440ca0a0 msgr2=0x7ff5440ca530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.007+0000 7ff54259c640 1 --2- 172.21.15.83:0/2756159334 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff5440ca0a0 0x7ff5440ca530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.007+0000 7ff54259c640 1 -- 172.21.15.83:0/2756159334 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff5440d0490 msgr2=0x7ff54409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.007+0000 7ff54259c640 1 --2- 172.21.15.83:0/2756159334 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff5440d0490 0x7ff54409f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.007+0000 7ff54259c640 1 -- 172.21.15.83:0/2756159334 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff544157480 con 0x7ff5440cf850 2024-03-06T07:28:50.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.007+0000 7ff539d9b640 1 --2- 172.21.15.83:0/2756159334 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff5440d0490 0x7ff54409f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:50.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.007+0000 7ff54259c640 1 --2- 172.21.15.83:0/2756159334 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5440cf850 0x7ff544156f40 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7ff534000c00 tx=0x7ff5340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:50.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.007+0000 7ff53b7fe640 1 -- 172.21.15.83:0/2756159334 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff53403b9c0 con 0x7ff5440cf850 2024-03-06T07:28:50.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.007+0000 7ff53b7fe640 1 -- 172.21.15.83:0/2756159334 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff53403bb80 con 0x7ff5440cf850 2024-03-06T07:28:50.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.007+0000 7ff548888640 1 -- 172.21.15.83:0/2756159334 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff54409e520 con 0x7ff5440cf850 2024-03-06T07:28:50.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.007+0000 7ff53b7fe640 1 -- 172.21.15.83:0/2756159334 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5340453f0 con 0x7ff5440cf850 2024-03-06T07:28:50.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.007+0000 7ff548888640 1 -- 172.21.15.83:0/2756159334 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff54409eae0 con 0x7ff5440cf850 2024-03-06T07:28:50.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.008+0000 7ff548888640 1 -- 172.21.15.83:0/2756159334 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff54404f330 con 0x7ff5440cf850 2024-03-06T07:28:50.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.009+0000 7ff53b7fe640 1 -- 172.21.15.83:0/2756159334 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff53404c060 con 0x7ff5440cf850 2024-03-06T07:28:50.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.009+0000 7ff53b7fe640 1 --2- 172.21.15.83:0/2756159334 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff52406b4d0 0x7ff52406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.009+0000 7ff53b7fe640 1 -- 172.21.15.83:0/2756159334 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff5340abaa0 con 0x7ff5440cf850 2024-03-06T07:28:50.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.009+0000 7ff539d9b640 1 --2- 172.21.15.83:0/2756159334 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff52406b4d0 0x7ff52406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.009+0000 7ff539d9b640 1 --2- 172.21.15.83:0/2756159334 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff52406b4d0 0x7ff52406d990 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7ff54409e000 tx=0x7ff52c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:50.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.013+0000 7ff53b7fe640 1 -- 172.21.15.83:0/2756159334 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff534040070 con 0x7ff5440cf850 2024-03-06T07:28:50.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.187+0000 7ff548888640 1 -- 172.21.15.83:0/2756159334 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7ff54409e210 con 0x7ff5440cf850 2024-03-06T07:28:50.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.187+0000 7ff53b7fe640 1 -- 172.21.15.83:0/2756159334 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v124) v1 ==== 84+0+65 (secure 0 0 0) 0x7ff53407ac70 con 0x7ff5440cf850 2024-03-06T07:28:50.189 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:28:50.189 INFO:teuthology.orchestra.run.smithi083.stdout:[ 2024-03-06T07:28:50.190 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-03-06T07:28:50.190 INFO:teuthology.orchestra.run.smithi083.stdout: "poolnum": 1, 2024-03-06T07:28:50.190 INFO:teuthology.orchestra.run.smithi083.stdout: "poolname": ".mgr" 2024-03-06T07:28:50.190 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:28:50.190 INFO:teuthology.orchestra.run.smithi083.stdout:] 2024-03-06T07:28:50.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.189+0000 7ff548888640 1 -- 172.21.15.83:0/2756159334 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff52406b4d0 msgr2=0x7ff52406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.189+0000 7ff548888640 1 --2- 172.21.15.83:0/2756159334 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff52406b4d0 0x7ff52406d990 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7ff54409e000 tx=0x7ff52c008040 comp rx=0 tx=0).stop 2024-03-06T07:28:50.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.189+0000 7ff548888640 1 -- 172.21.15.83:0/2756159334 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5440cf850 msgr2=0x7ff544156f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.189+0000 7ff548888640 1 --2- 172.21.15.83:0/2756159334 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5440cf850 0x7ff544156f40 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7ff534000c00 tx=0x7ff5340046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.189+0000 7ff548888640 1 -- 172.21.15.83:0/2756159334 shutdown_connections 2024-03-06T07:28:50.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.189+0000 7ff548888640 1 --2- 172.21.15.83:0/2756159334 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff52406b4d0 0x7ff52406d990 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.189+0000 7ff548888640 1 --2- 172.21.15.83:0/2756159334 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff5440ca0a0 0x7ff5440ca530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.189+0000 7ff548888640 1 --2- 172.21.15.83:0/2756159334 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff5440d0490 0x7ff54409f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.189+0000 7ff548888640 1 --2- 172.21.15.83:0/2756159334 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5440cf850 0x7ff544156f40 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.189+0000 7ff548888640 1 -- 172.21.15.83:0/2756159334 >> 172.21.15.83:0/2756159334 conn(0x7ff5440b35a0 msgr2=0x7ff5440cdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:50.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.189+0000 7ff548888640 1 -- 172.21.15.83:0/2756159334 shutdown_connections 2024-03-06T07:28:50.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.189+0000 7ff548888640 1 -- 172.21.15.83:0/2756159334 wait complete. 2024-03-06T07:28:50.202 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:28:50.202 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:28:50.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.304+0000 7f992bf25640 1 Processor -- start 2024-03-06T07:28:50.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.304+0000 7f992bf25640 1 -- start start 2024-03-06T07:28:50.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.305+0000 7f992bf25640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f99240708b0 0x7f9924070d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.305+0000 7f992bf25640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f99240ce0a0 0x7f99240ce550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.305+0000 7f992bf25640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f992415afe0 con 0x7f992415ac00 2024-03-06T07:28:50.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.305+0000 7f992bf25640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9924071250 con 0x7f99240708b0 2024-03-06T07:28:50.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.305+0000 7f992bf25640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99240cdc10 con 0x7f99240ce0a0 2024-03-06T07:28:50.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.305+0000 7f99294c0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f99240708b0 0x7f9924070d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.305+0000 7f99294c0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f99240708b0 0x7f9924070d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48794/0 (socket says 172.21.15.83:48794) 2024-03-06T07:28:50.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.305+0000 7f99294c0640 1 -- 172.21.15.83:0/51928392 learned_addr learned my addr 172.21.15.83:0/51928392 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:50.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.305+0000 7f992a4c2640 1 --2- 172.21.15.83:0/51928392 >> v2:172.21.15.183:3300/0 conn(0x7f99240ce0a0 0x7f99240ce550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.306+0000 7f99294c0640 1 -- 172.21.15.83:0/51928392 >> v2:172.21.15.183:3300/0 conn(0x7f99240ce0a0 msgr2=0x7f99240ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:50.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.306+0000 7f99294c0640 1 --2- 172.21.15.83:0/51928392 >> v2:172.21.15.183:3300/0 conn(0x7f99240ce0a0 0x7f99240ce550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.306+0000 7f99294c0640 1 -- 172.21.15.83:0/51928392 >> v1:172.21.15.183:6789/0 conn(0x7f992415ac00 legacy=0x7f992406c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.306+0000 7f99294c0640 1 -- 172.21.15.83:0/51928392 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99240cd000 con 0x7f99240708b0 2024-03-06T07:28:50.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.306+0000 7f992a4c2640 1 --2- 172.21.15.83:0/51928392 >> v2:172.21.15.183:3300/0 conn(0x7f99240ce0a0 0x7f99240ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:50.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.306+0000 7f99294c0640 1 --2- 172.21.15.83:0/51928392 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99240708b0 0x7f9924070d10 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f99240cc0a0 tx=0x7f991402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7386b808d51f8c59 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:50.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.306+0000 7f9928cbf640 1 -- 172.21.15.83:0/51928392 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9914035020 con 0x7f99240708b0 2024-03-06T07:28:50.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.306+0000 7f9928cbf640 1 -- 172.21.15.83:0/51928392 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9914031870 con 0x7f99240708b0 2024-03-06T07:28:50.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.306+0000 7f9928cbf640 1 -- 172.21.15.83:0/51928392 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9914031c40 con 0x7f99240708b0 2024-03-06T07:28:50.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.307+0000 7f992bf25640 1 -- 172.21.15.83:0/51928392 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99240708b0 msgr2=0x7f9924070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.307+0000 7f992bf25640 1 --2- 172.21.15.83:0/51928392 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99240708b0 0x7f9924070d10 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f99240cc0a0 tx=0x7f991402b910 comp rx=0 tx=0).stop 2024-03-06T07:28:50.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.307+0000 7f992bf25640 1 -- 172.21.15.83:0/51928392 shutdown_connections 2024-03-06T07:28:50.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.307+0000 7f992bf25640 1 --2- 172.21.15.83:0/51928392 >> v2:172.21.15.183:3300/0 conn(0x7f99240ce0a0 0x7f99240ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.307+0000 7f992bf25640 1 --2- 172.21.15.83:0/51928392 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99240708b0 0x7f9924070d10 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.307+0000 7f992bf25640 1 -- 172.21.15.83:0/51928392 >> 172.21.15.83:0/51928392 conn(0x7f99240b1650 msgr2=0x7f99240b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:50.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.307+0000 7f992bf25640 1 -- 172.21.15.83:0/51928392 shutdown_connections 2024-03-06T07:28:50.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.307+0000 7f992bf25640 1 -- 172.21.15.83:0/51928392 wait complete. 2024-03-06T07:28:50.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.308+0000 7f992bf25640 1 Processor -- start 2024-03-06T07:28:50.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.308+0000 7f992bf25640 1 -- start start 2024-03-06T07:28:50.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.309+0000 7f992bf25640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f99240708b0 0x7f992414f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.309+0000 7f992bf25640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99240ce0a0 0x7f99241f5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.309+0000 7f992bf25640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f992415ac00 0x7f99241f5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.309+0000 7f992bf25640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99240cd8a0 con 0x7f99240ce0a0 2024-03-06T07:28:50.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.309+0000 7f992bf25640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99240cd5c0 con 0x7f99240708b0 2024-03-06T07:28:50.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.309+0000 7f992bf25640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99240cf5d0 con 0x7f992415ac00 2024-03-06T07:28:50.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.309+0000 7f99294c0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99240ce0a0 0x7f99241f5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.309+0000 7f99294c0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99240ce0a0 0x7f99241f5920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48798/0 (socket says 172.21.15.83:48798) 2024-03-06T07:28:50.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.309+0000 7f99294c0640 1 -- 172.21.15.83:0/3843470630 learned_addr learned my addr 172.21.15.83:0/3843470630 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:50.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.309+0000 7f99294c0640 1 -- 172.21.15.83:0/3843470630 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f992415ac00 msgr2=0x7f99241f5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.309+0000 7f99294c0640 1 --2- 172.21.15.83:0/3843470630 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f992415ac00 0x7f99241f5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.309+0000 7f99294c0640 1 -- 172.21.15.83:0/3843470630 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f99240708b0 msgr2=0x7f992414f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.309+0000 7f99294c0640 1 --2- 172.21.15.83:0/3843470630 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f99240708b0 0x7f992414f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.309+0000 7f99294c0640 1 -- 172.21.15.83:0/3843470630 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99240a9f00 con 0x7f99240ce0a0 2024-03-06T07:28:50.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.309+0000 7f99294c0640 1 --2- 172.21.15.83:0/3843470630 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99240ce0a0 0x7f99241f5920 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f991402b950 tx=0x7f99140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:50.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.310+0000 7f991a7fc640 1 -- 172.21.15.83:0/3843470630 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9914040070 con 0x7f99240ce0a0 2024-03-06T07:28:50.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.310+0000 7f991a7fc640 1 -- 172.21.15.83:0/3843470630 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9914030d00 con 0x7f99240ce0a0 2024-03-06T07:28:50.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.310+0000 7f992bf25640 1 -- 172.21.15.83:0/3843470630 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99240aa1e0 con 0x7f99240ce0a0 2024-03-06T07:28:50.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.310+0000 7f991a7fc640 1 -- 172.21.15.83:0/3843470630 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9914035040 con 0x7f99240ce0a0 2024-03-06T07:28:50.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.310+0000 7f992bf25640 1 -- 172.21.15.83:0/3843470630 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99240aa7a0 con 0x7f99240ce0a0 2024-03-06T07:28:50.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.311+0000 7f992bf25640 1 -- 172.21.15.83:0/3843470630 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f992404f330 con 0x7f99240ce0a0 2024-03-06T07:28:50.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.311+0000 7f991a7fc640 1 -- 172.21.15.83:0/3843470630 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9914038020 con 0x7f99240ce0a0 2024-03-06T07:28:50.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.311+0000 7f991a7fc640 1 --2- 172.21.15.83:0/3843470630 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f990806b4d0 0x7f990806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.312+0000 7f991a7fc640 1 -- 172.21.15.83:0/3843470630 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f99140abce0 con 0x7f99240ce0a0 2024-03-06T07:28:50.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.312+0000 7f9929cc1640 1 --2- 172.21.15.83:0/3843470630 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f990806b4d0 0x7f990806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.312+0000 7f9929cc1640 1 --2- 172.21.15.83:0/3843470630 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f990806b4d0 0x7f990806d990 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f99200012e0 tx=0x7f9920007d50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:50.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.316+0000 7f991a7fc640 1 -- 172.21.15.83:0/3843470630 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f991407aeb0 con 0x7f99240ce0a0 2024-03-06T07:28:50.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.489+0000 7f992bf25640 1 -- 172.21.15.83:0/3843470630 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f992415ea20 con 0x7f99240ce0a0 2024-03-06T07:28:50.695 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-03-06T07:28:50.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.938+0000 7f991a7fc640 1 -- 172.21.15.83:0/3843470630 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v125) v1 ==== 138+0+0 (secure 0 0 0) 0x7f991407a7b0 con 0x7f99240ce0a0 2024-03-06T07:28:50.940 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'cephfs_metadata' created 2024-03-06T07:28:50.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.940+0000 7f992bf25640 1 -- 172.21.15.83:0/3843470630 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f990806b4d0 msgr2=0x7f990806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.940+0000 7f992bf25640 1 --2- 172.21.15.83:0/3843470630 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f990806b4d0 0x7f990806d990 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f99200012e0 tx=0x7f9920007d50 comp rx=0 tx=0).stop 2024-03-06T07:28:50.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.940+0000 7f992bf25640 1 -- 172.21.15.83:0/3843470630 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99240ce0a0 msgr2=0x7f99241f5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.940+0000 7f992bf25640 1 --2- 172.21.15.83:0/3843470630 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99240ce0a0 0x7f99241f5920 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f991402b950 tx=0x7f99140046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.941+0000 7f992bf25640 1 -- 172.21.15.83:0/3843470630 shutdown_connections 2024-03-06T07:28:50.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.941+0000 7f992bf25640 1 --2- 172.21.15.83:0/3843470630 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f990806b4d0 0x7f990806d990 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.941+0000 7f992bf25640 1 --2- 172.21.15.83:0/3843470630 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f992415ac00 0x7f99241f5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.941+0000 7f992bf25640 1 --2- 172.21.15.83:0/3843470630 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99240ce0a0 0x7f99241f5920 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.941+0000 7f992bf25640 1 --2- 172.21.15.83:0/3843470630 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f99240708b0 0x7f992414f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.941+0000 7f992bf25640 1 -- 172.21.15.83:0/3843470630 >> 172.21.15.83:0/3843470630 conn(0x7f99240b1650 msgr2=0x7f992415ef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:50.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.941+0000 7f992bf25640 1 -- 172.21.15.83:0/3843470630 shutdown_connections 2024-03-06T07:28:50.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:50.941+0000 7f992bf25640 1 -- 172.21.15.83:0/3843470630 wait complete. 2024-03-06T07:28:50.957 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:28:51.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.063+0000 7f5728815640 1 Processor -- start 2024-03-06T07:28:51.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.063+0000 7f5728815640 1 -- start start 2024-03-06T07:28:51.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.063+0000 7f5728815640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f57240cff40 0x7f57240b3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.063+0000 7f5728815640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5724158ac0 con 0x7f57241586e0 2024-03-06T07:28:51.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.063+0000 7f5728815640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f572415b2a0 con 0x7f57240cfa90 2024-03-06T07:28:51.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.063+0000 7f5728815640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57240d0390 con 0x7f57240cff40 2024-03-06T07:28:51.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.064+0000 7f572259c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f57241586e0 0x7f57240aea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52186/0 (socket says 172.21.15.83:52186) 2024-03-06T07:28:51.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.064+0000 7f572259c640 1 -- 172.21.15.83:0/1326770655 learned_addr learned my addr 172.21.15.83:0/1326770655 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:28:51.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.064+0000 7f5722d9d640 1 --2- 172.21.15.83:0/1326770655 >> v2:172.21.15.183:3300/0 conn(0x7f57240cff40 0x7f57240b3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.064+0000 7f5721d9b640 1 -- 172.21.15.83:0/1326770655 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3575457543 0 0) 0x7f5724158ac0 con 0x7f57241586e0 2024-03-06T07:28:51.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.064+0000 7f5721d9b640 1 -- 172.21.15.83:0/1326770655 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5710003520 con 0x7f57241586e0 2024-03-06T07:28:51.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.064+0000 7f5721d9b640 1 -- 172.21.15.83:0/1326770655 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3946703929 0 0) 0x7f5710003520 con 0x7f57241586e0 2024-03-06T07:28:51.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.064+0000 7f5721d9b640 1 -- 172.21.15.83:0/1326770655 >> v2:172.21.15.183:3300/0 conn(0x7f57240cff40 msgr2=0x7f57240b3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:51.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.064+0000 7f5721d9b640 1 --2- 172.21.15.83:0/1326770655 >> v2:172.21.15.183:3300/0 conn(0x7f57240cff40 0x7f57240b3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.064+0000 7f5721d9b640 1 -- 172.21.15.83:0/1326770655 >> v1:172.21.15.183:6789/0 conn(0x7f57240cfa90 legacy=0x7f57240b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.064+0000 7f5721d9b640 1 -- 172.21.15.83:0/1326770655 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f572415b540 con 0x7f57241586e0 2024-03-06T07:28:51.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.064+0000 7f5721d9b640 1 -- 172.21.15.83:0/1326770655 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f5714001670 con 0x7f57241586e0 2024-03-06T07:28:51.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.064+0000 7f5721d9b640 1 -- 172.21.15.83:0/1326770655 >> v1:172.21.15.83:6789/0 conn(0x7f57241586e0 legacy=0x7f57240aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.064+0000 7f5721d9b640 1 --2- 172.21.15.83:0/1326770655 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f57100039b0 0x7f5710003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.064+0000 7f5721d9b640 1 --2- 172.21.15.83:0/1326770655 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5710004800 0x7f5710004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.064+0000 7f5721d9b640 1 --2- 172.21.15.83:0/1326770655 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f57100051c0 0x7f571000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.064+0000 7f5721d9b640 1 -- 172.21.15.83:0/1326770655 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57240d0390 con 0x7f5710004800 2024-03-06T07:28:51.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.064+0000 7f5721d9b640 1 -- 172.21.15.83:0/1326770655 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f571000c060 con 0x7f57100051c0 2024-03-06T07:28:51.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.064+0000 7f5721d9b640 1 -- 172.21.15.83:0/1326770655 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f571000c260 con 0x7f57100039b0 2024-03-06T07:28:51.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.064+0000 7f5719d9b640 1 --2- 172.21.15.83:0/1326770655 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5710004800 0x7f5710004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.064+0000 7f572259c640 1 --2- 172.21.15.83:0/1326770655 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f57100039b0 0x7f5710003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.064+0000 7f5722d9d640 1 --2- 172.21.15.83:0/1326770655 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f57100051c0 0x7f571000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.064+0000 7f5719d9b640 1 -- 172.21.15.83:0/1326770655 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f57100039b0 msgr2=0x7f5710003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:51.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.064+0000 7f5719d9b640 1 --2- 172.21.15.83:0/1326770655 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f57100039b0 0x7f5710003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.064+0000 7f5719d9b640 1 -- 172.21.15.83:0/1326770655 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f57100051c0 msgr2=0x7f571000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:51.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.064+0000 7f5719d9b640 1 --2- 172.21.15.83:0/1326770655 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f57100051c0 0x7f571000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.065+0000 7f5719d9b640 1 -- 172.21.15.83:0/1326770655 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5710003520 con 0x7f5710004800 2024-03-06T07:28:51.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.065+0000 7f5722d9d640 1 --2- 172.21.15.83:0/1326770655 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f57100051c0 0x7f571000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:51.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.065+0000 7f5722d9d640 1 -- 172.21.15.83:0/1326770655 reap_dead start 2024-03-06T07:28:51.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.065+0000 7f5719d9b640 1 --2- 172.21.15.83:0/1326770655 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5710004800 0x7f5710004c80 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f570c001750 tx=0x7f5710005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65eea8fa17767711 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:51.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.065+0000 7f5721d9b640 1 -- 172.21.15.83:0/1326770655 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570c001370 con 0x7f5710004800 2024-03-06T07:28:51.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.065+0000 7f5721d9b640 1 -- 172.21.15.83:0/1326770655 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f570c0052d0 con 0x7f5710004800 2024-03-06T07:28:51.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.065+0000 7f5721d9b640 1 -- 172.21.15.83:0/1326770655 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570c003830 con 0x7f5710004800 2024-03-06T07:28:51.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.066+0000 7f5728815640 1 -- 172.21.15.83:0/1326770655 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5710004800 msgr2=0x7f5710004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.066+0000 7f5728815640 1 --2- 172.21.15.83:0/1326770655 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5710004800 0x7f5710004c80 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f570c001750 tx=0x7f5710005830 comp rx=0 tx=0).stop 2024-03-06T07:28:51.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.066+0000 7f5728815640 1 -- 172.21.15.83:0/1326770655 shutdown_connections 2024-03-06T07:28:51.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.066+0000 7f5728815640 1 --2- 172.21.15.83:0/1326770655 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5710004800 0x7f5710004c80 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.066+0000 7f5728815640 1 -- 172.21.15.83:0/1326770655 >> 172.21.15.83:0/1326770655 conn(0x7f572406b190 msgr2=0x7f572406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:51.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.066+0000 7f5728815640 1 -- 172.21.15.83:0/1326770655 shutdown_connections 2024-03-06T07:28:51.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.066+0000 7f5728815640 1 -- 172.21.15.83:0/1326770655 wait complete. 2024-03-06T07:28:51.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.067+0000 7f5728815640 1 Processor -- start 2024-03-06T07:28:51.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.067+0000 7f5728815640 1 -- start start 2024-03-06T07:28:51.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.068+0000 7f5728815640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5710004800 0x7f57240af970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.068+0000 7f5728815640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f57240adbb0 0x7f57240b0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.068+0000 7f5728815640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57240b10b0 0x7f572406c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.068+0000 7f5728815640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f572415bde0 con 0x7f57240b10b0 2024-03-06T07:28:51.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.068+0000 7f5728815640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f572415bb00 con 0x7f57240adbb0 2024-03-06T07:28:51.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.068+0000 7f5728815640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5724071350 con 0x7f5710004800 2024-03-06T07:28:51.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.068+0000 7f5722d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57240b10b0 0x7f572406c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.068+0000 7f5722d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57240b10b0 0x7f572406c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48820/0 (socket says 172.21.15.83:48820) 2024-03-06T07:28:51.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.068+0000 7f5722d9d640 1 -- 172.21.15.83:0/3113298046 learned_addr learned my addr 172.21.15.83:0/3113298046 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:51.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.068+0000 7f572259c640 1 --2- 172.21.15.83:0/3113298046 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5710004800 0x7f57240af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.068+0000 7f5722d9d640 1 -- 172.21.15.83:0/3113298046 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5710004800 msgr2=0x7f57240af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.068+0000 7f5719d9b640 1 --2- 172.21.15.83:0/3113298046 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f57240adbb0 0x7f57240b0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.068+0000 7f5722d9d640 1 --2- 172.21.15.83:0/3113298046 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5710004800 0x7f57240af970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.068+0000 7f5722d9d640 1 -- 172.21.15.83:0/3113298046 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f57240adbb0 msgr2=0x7f57240b0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.068+0000 7f5722d9d640 1 --2- 172.21.15.83:0/3113298046 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f57240adbb0 0x7f57240b0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.068+0000 7f5722d9d640 1 -- 172.21.15.83:0/3113298046 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57240afeb0 con 0x7f57240b10b0 2024-03-06T07:28:51.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.069+0000 7f5722d9d640 1 --2- 172.21.15.83:0/3113298046 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57240b10b0 0x7f572406c910 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f5724071130 tx=0x7f571c010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:51.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.069+0000 7f571b7fe640 1 -- 172.21.15.83:0/3113298046 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571c010da0 con 0x7f57240b10b0 2024-03-06T07:28:51.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.069+0000 7f571b7fe640 1 -- 172.21.15.83:0/3113298046 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f571c003990 con 0x7f57240b10b0 2024-03-06T07:28:51.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.069+0000 7f5728815640 1 -- 172.21.15.83:0/3113298046 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57241f4df0 con 0x7f57240b10b0 2024-03-06T07:28:51.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.069+0000 7f571b7fe640 1 -- 172.21.15.83:0/3113298046 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571c003dc0 con 0x7f57240b10b0 2024-03-06T07:28:51.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.069+0000 7f5728815640 1 -- 172.21.15.83:0/3113298046 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57241f53b0 con 0x7f57240b10b0 2024-03-06T07:28:51.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.070+0000 7f5728815640 1 -- 172.21.15.83:0/3113298046 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57240732b0 con 0x7f57240b10b0 2024-03-06T07:28:51.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.070+0000 7f571b7fe640 1 -- 172.21.15.83:0/3113298046 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f571c01e020 con 0x7f57240b10b0 2024-03-06T07:28:51.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.071+0000 7f571b7fe640 1 --2- 172.21.15.83:0/3113298046 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f570406b4d0 0x7f570406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.071+0000 7f571b7fe640 1 -- 172.21.15.83:0/3113298046 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f571c08a620 con 0x7f57240b10b0 2024-03-06T07:28:51.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.071+0000 7f572259c640 1 --2- 172.21.15.83:0/3113298046 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f570406b4d0 0x7f570406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.071+0000 7f572259c640 1 --2- 172.21.15.83:0/3113298046 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f570406b4d0 0x7f570406d990 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f57241f49b0 tx=0x7f5714009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:51.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.075+0000 7f571b7fe640 1 -- 172.21.15.83:0/3113298046 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f571c059680 con 0x7f57240b10b0 2024-03-06T07:28:51.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.247+0000 7f5728815640 1 -- 172.21.15.83:0/3113298046 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f57240ab910 con 0x7f57240b10b0 2024-03-06T07:28:51.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.947+0000 7f571b7fe640 1 -- 172.21.15.83:0/3113298046 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v126) v1 ==== 170+0+0 (secure 0 0 0) 0x7f571c058f80 con 0x7f57240b10b0 2024-03-06T07:28:51.950 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'cephfs_data' created 2024-03-06T07:28:51.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.952+0000 7f5728815640 1 -- 172.21.15.83:0/3113298046 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f570406b4d0 msgr2=0x7f570406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.952+0000 7f5728815640 1 --2- 172.21.15.83:0/3113298046 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f570406b4d0 0x7f570406d990 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f57241f49b0 tx=0x7f5714009040 comp rx=0 tx=0).stop 2024-03-06T07:28:51.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.952+0000 7f5728815640 1 -- 172.21.15.83:0/3113298046 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57240b10b0 msgr2=0x7f572406c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.952+0000 7f5728815640 1 --2- 172.21.15.83:0/3113298046 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57240b10b0 0x7f572406c910 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f5724071130 tx=0x7f571c010590 comp rx=0 tx=0).stop 2024-03-06T07:28:51.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.952+0000 7f5728815640 1 -- 172.21.15.83:0/3113298046 shutdown_connections 2024-03-06T07:28:51.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.952+0000 7f5728815640 1 --2- 172.21.15.83:0/3113298046 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f570406b4d0 0x7f570406d990 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.952+0000 7f5728815640 1 --2- 172.21.15.83:0/3113298046 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57240b10b0 0x7f572406c910 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.952+0000 7f5728815640 1 --2- 172.21.15.83:0/3113298046 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f57240adbb0 0x7f57240b0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.952+0000 7f5728815640 1 --2- 172.21.15.83:0/3113298046 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5710004800 0x7f57240af970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.952+0000 7f5728815640 1 -- 172.21.15.83:0/3113298046 >> 172.21.15.83:0/3113298046 conn(0x7f572406b190 msgr2=0x7f572415a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:51.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.956+0000 7f5728815640 1 -- 172.21.15.83:0/3113298046 shutdown_connections 2024-03-06T07:28:51.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:51.956+0000 7f5728815640 1 -- 172.21.15.83:0/3113298046 wait complete. 2024-03-06T07:28:51.980 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:28:52.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.086+0000 7f2c31a66640 1 Processor -- start 2024-03-06T07:28:52.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.086+0000 7f2c31a66640 1 -- start start 2024-03-06T07:28:52.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.087+0000 7f2c31a66640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2c2c15bc50 0x7f2c2c0cf430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.087+0000 7f2c31a66640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2c2c0cc280 0x7f2c2c0cf970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.087+0000 7f2c31a66640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c2c06d5b0 con 0x7f2c2c0aee20 2024-03-06T07:28:52.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.087+0000 7f2c31a66640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c2c0b4850 con 0x7f2c2c15bc50 2024-03-06T07:28:52.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.087+0000 7f2c31a66640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2c0cc7b0 con 0x7f2c2c0cc280 2024-03-06T07:28:52.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.087+0000 7f2c2b7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2c2c15bc50 0x7f2c2c0cf430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.087+0000 7f2c2b7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2c2c15bc50 0x7f2c2c0cf430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48830/0 (socket says 172.21.15.83:48830) 2024-03-06T07:28:52.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.087+0000 7f2c2b7fe640 1 -- 172.21.15.83:0/2633402313 learned_addr learned my addr 172.21.15.83:0/2633402313 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:52.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.087+0000 7f2c2b7fe640 1 -- 172.21.15.83:0/2633402313 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2c2c0cc280 msgr2=0x7f2c2c0cf970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:52.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.087+0000 7f2c2b7fe640 1 --2- 172.21.15.83:0/2633402313 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2c2c0cc280 0x7f2c2c0cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.087+0000 7f2c2b7fe640 1 -- 172.21.15.83:0/2633402313 >> v1:172.21.15.183:6789/0 conn(0x7f2c2c0aee20 legacy=0x7f2c2c15f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:52.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.087+0000 7f2c2b7fe640 1 -- 172.21.15.83:0/2633402313 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c2c0b4ce0 con 0x7f2c2c15bc50 2024-03-06T07:28:52.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.088+0000 7f2c2b7fe640 1 --2- 172.21.15.83:0/2633402313 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c2c15bc50 0x7f2c2c0cf430 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c009870 tx=0x7f2c1c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad95f5e86a31cdf4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:52.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.088+0000 7f2c2a7fc640 1 -- 172.21.15.83:0/2633402313 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c1c0317c0 con 0x7f2c2c15bc50 2024-03-06T07:28:52.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.088+0000 7f2c2a7fc640 1 -- 172.21.15.83:0/2633402313 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c1c037070 con 0x7f2c2c15bc50 2024-03-06T07:28:52.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.088+0000 7f2c2a7fc640 1 -- 172.21.15.83:0/2633402313 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c1c031b80 con 0x7f2c2c15bc50 2024-03-06T07:28:52.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.089+0000 7f2c31a66640 1 -- 172.21.15.83:0/2633402313 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c2c15bc50 msgr2=0x7f2c2c0cf430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.089+0000 7f2c31a66640 1 --2- 172.21.15.83:0/2633402313 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c2c15bc50 0x7f2c2c0cf430 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c009870 tx=0x7f2c1c02b860 comp rx=0 tx=0).stop 2024-03-06T07:28:52.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.089+0000 7f2c31a66640 1 -- 172.21.15.83:0/2633402313 shutdown_connections 2024-03-06T07:28:52.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.089+0000 7f2c31a66640 1 --2- 172.21.15.83:0/2633402313 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2c2c0cc280 0x7f2c2c0cf970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.089+0000 7f2c31a66640 1 --2- 172.21.15.83:0/2633402313 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c2c15bc50 0x7f2c2c0cf430 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.089+0000 7f2c31a66640 1 -- 172.21.15.83:0/2633402313 >> 172.21.15.83:0/2633402313 conn(0x7f2c2c06afb0 msgr2=0x7f2c2c06b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:52.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.089+0000 7f2c31a66640 1 -- 172.21.15.83:0/2633402313 shutdown_connections 2024-03-06T07:28:52.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.089+0000 7f2c31a66640 1 -- 172.21.15.83:0/2633402313 wait complete. 2024-03-06T07:28:52.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.090+0000 7f2c31a66640 1 Processor -- start 2024-03-06T07:28:52.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.090+0000 7f2c31a66640 1 -- start start 2024-03-06T07:28:52.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.091+0000 7f2c31a66640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c2c0aee20 0x7f2c2c14c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.091+0000 7f2c31a66640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c2c0cc280 0x7f2c2c14f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.091+0000 7f2c31a66640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2c2c15bc50 0x7f2c2c14fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.091+0000 7f2c31a66640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2c15c030 con 0x7f2c2c0cc280 2024-03-06T07:28:52.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.091+0000 7f2c31a66640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2c0ae900 con 0x7f2c2c0aee20 2024-03-06T07:28:52.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.091+0000 7f2c31a66640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2c15c1b0 con 0x7f2c2c15bc50 2024-03-06T07:28:52.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.091+0000 7f2c2affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c2c0cc280 0x7f2c2c14f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.091+0000 7f2c2affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c2c0cc280 0x7f2c2c14f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48842/0 (socket says 172.21.15.83:48842) 2024-03-06T07:28:52.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.091+0000 7f2c2affd640 1 -- 172.21.15.83:0/4019771990 learned_addr learned my addr 172.21.15.83:0/4019771990 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:52.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.091+0000 7f2c2affd640 1 -- 172.21.15.83:0/4019771990 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2c2c15bc50 msgr2=0x7f2c2c14fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.091+0000 7f2c2bfff640 1 --2- 172.21.15.83:0/4019771990 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2c2c15bc50 0x7f2c2c14fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.091+0000 7f2c2b7fe640 1 --2- 172.21.15.83:0/4019771990 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c2c0aee20 0x7f2c2c14c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.091+0000 7f2c2affd640 1 --2- 172.21.15.83:0/4019771990 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2c2c15bc50 0x7f2c2c14fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.091+0000 7f2c2affd640 1 -- 172.21.15.83:0/4019771990 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c2c0aee20 msgr2=0x7f2c2c14c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.091+0000 7f2c2affd640 1 --2- 172.21.15.83:0/4019771990 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c2c0aee20 0x7f2c2c14c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.091+0000 7f2c2affd640 1 -- 172.21.15.83:0/4019771990 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c2c152a70 con 0x7f2c2c0cc280 2024-03-06T07:28:52.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.091+0000 7f2c2bfff640 1 --2- 172.21.15.83:0/4019771990 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2c2c15bc50 0x7f2c2c14fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:52.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.091+0000 7f2c2b7fe640 1 --2- 172.21.15.83:0/4019771990 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c2c0aee20 0x7f2c2c14c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:52.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.091+0000 7f2c2affd640 1 --2- 172.21.15.83:0/4019771990 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c2c0cc280 0x7f2c2c14f8b0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f2c2000b450 tx=0x7f2c2000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:52.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.091+0000 7f2c13fff640 1 -- 172.21.15.83:0/4019771990 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c20015020 con 0x7f2c2c0cc280 2024-03-06T07:28:52.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.091+0000 7f2c13fff640 1 -- 172.21.15.83:0/4019771990 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c20011890 con 0x7f2c2c0cc280 2024-03-06T07:28:52.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.091+0000 7f2c31a66640 1 -- 172.21.15.83:0/4019771990 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c2c152d50 con 0x7f2c2c0cc280 2024-03-06T07:28:52.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.091+0000 7f2c13fff640 1 -- 172.21.15.83:0/4019771990 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c20011c80 con 0x7f2c2c0cc280 2024-03-06T07:28:52.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.091+0000 7f2c31a66640 1 -- 172.21.15.83:0/4019771990 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c2c153310 con 0x7f2c2c0cc280 2024-03-06T07:28:52.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.092+0000 7f2c31a66640 1 -- 172.21.15.83:0/4019771990 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c2c04f330 con 0x7f2c2c0cc280 2024-03-06T07:28:52.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.093+0000 7f2c13fff640 1 -- 172.21.15.83:0/4019771990 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2c20018020 con 0x7f2c2c0cc280 2024-03-06T07:28:52.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.093+0000 7f2c13fff640 1 --2- 172.21.15.83:0/4019771990 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2c1806b4d0 0x7f2c1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.093+0000 7f2c13fff640 1 -- 172.21.15.83:0/4019771990 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f2c2008a390 con 0x7f2c2c0cc280 2024-03-06T07:28:52.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.093+0000 7f2c2b7fe640 1 --2- 172.21.15.83:0/4019771990 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2c1806b4d0 0x7f2c1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.094+0000 7f2c2b7fe640 1 --2- 172.21.15.83:0/4019771990 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2c1806b4d0 0x7f2c1806d990 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c036000 tx=0x7f2c1c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:52.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.098+0000 7f2c13fff640 1 -- 172.21.15.83:0/4019771990 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2c2001f020 con 0x7f2c2c0cc280 2024-03-06T07:28:52.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.305+0000 7f2c31a66640 1 -- 172.21.15.83:0/4019771990 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f2c2c0b0060 con 0x7f2c2c0cc280 2024-03-06T07:28:52.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.961+0000 7f2c13fff640 1 -- 172.21.15.83:0/4019771990 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 18 and data pool 19 v71) v1 ==== 174+0+0 (secure 0 0 0) 0x7f2c2000d040 con 0x7f2c2c0cc280 2024-03-06T07:28:52.965 INFO:teuthology.orchestra.run.smithi083.stderr:new fs with metadata pool 18 and data pool 19 2024-03-06T07:28:52.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.967+0000 7f2c31a66640 1 -- 172.21.15.83:0/4019771990 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2c1806b4d0 msgr2=0x7f2c1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.967+0000 7f2c31a66640 1 --2- 172.21.15.83:0/4019771990 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2c1806b4d0 0x7f2c1806d990 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c036000 tx=0x7f2c1c008030 comp rx=0 tx=0).stop 2024-03-06T07:28:52.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.967+0000 7f2c31a66640 1 -- 172.21.15.83:0/4019771990 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c2c0cc280 msgr2=0x7f2c2c14f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.967+0000 7f2c31a66640 1 --2- 172.21.15.83:0/4019771990 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c2c0cc280 0x7f2c2c14f8b0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f2c2000b450 tx=0x7f2c2000b910 comp rx=0 tx=0).stop 2024-03-06T07:28:52.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.968+0000 7f2c31a66640 1 -- 172.21.15.83:0/4019771990 shutdown_connections 2024-03-06T07:28:52.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.968+0000 7f2c31a66640 1 --2- 172.21.15.83:0/4019771990 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2c1806b4d0 0x7f2c1806d990 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.968+0000 7f2c31a66640 1 --2- 172.21.15.83:0/4019771990 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2c2c15bc50 0x7f2c2c14fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.968+0000 7f2c31a66640 1 --2- 172.21.15.83:0/4019771990 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c2c0cc280 0x7f2c2c14f8b0 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.968+0000 7f2c31a66640 1 --2- 172.21.15.83:0/4019771990 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c2c0aee20 0x7f2c2c14c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.968+0000 7f2c31a66640 1 -- 172.21.15.83:0/4019771990 >> 172.21.15.83:0/4019771990 conn(0x7f2c2c06afb0 msgr2=0x7f2c2c0b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:52.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.969+0000 7f2c31a66640 1 -- 172.21.15.83:0/4019771990 shutdown_connections 2024-03-06T07:28:52.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:52.969+0000 7f2c31a66640 1 -- 172.21.15.83:0/4019771990 wait complete. 2024-03-06T07:28:52.988 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:53.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.087+0000 7fcce3e2a640 1 Processor -- start 2024-03-06T07:28:53.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.087+0000 7fcce3e2a640 1 -- start start 2024-03-06T07:28:53.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.088+0000 7fcce3e2a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fccdc15bf30 0x7fccdc15ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.088+0000 7fcce3e2a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccdc15f580 con 0x7fccdc0b3060 2024-03-06T07:28:53.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.088+0000 7fcce3e2a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccdc15b450 con 0x7fccdc13dc10 2024-03-06T07:28:53.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.088+0000 7fcce3e2a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccdc0cf3f0 con 0x7fccdc15bf30 2024-03-06T07:28:53.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.088+0000 7fcce13c5640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fccdc0b3060 0x7fccdc15e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52188/0 (socket says 172.21.15.83:52188) 2024-03-06T07:28:53.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.088+0000 7fcce13c5640 1 -- 172.21.15.83:0/4177204948 learned_addr learned my addr 172.21.15.83:0/4177204948 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:28:53.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.088+0000 7fcce23c7640 1 --2- 172.21.15.83:0/4177204948 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fccdc15bf30 0x7fccdc15ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.088+0000 7fcce0bc4640 1 -- 172.21.15.83:0/4177204948 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1609981139 0 0) 0x7fccdc15f580 con 0x7fccdc0b3060 2024-03-06T07:28:53.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.088+0000 7fcce0bc4640 1 -- 172.21.15.83:0/4177204948 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccc8003520 con 0x7fccdc0b3060 2024-03-06T07:28:53.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.089+0000 7fcce0bc4640 1 -- 172.21.15.83:0/4177204948 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2894270763 0 0) 0x7fccc8003520 con 0x7fccdc0b3060 2024-03-06T07:28:53.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.089+0000 7fcce0bc4640 1 -- 172.21.15.83:0/4177204948 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fccdc15bf30 msgr2=0x7fccdc15ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:53.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.089+0000 7fcce0bc4640 1 --2- 172.21.15.83:0/4177204948 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fccdc15bf30 0x7fccdc15ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.089+0000 7fcce0bc4640 1 -- 172.21.15.83:0/4177204948 >> v1:172.21.15.183:6789/0 conn(0x7fccdc13dc10 legacy=0x7fccdc0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.089+0000 7fcce0bc4640 1 -- 172.21.15.83:0/4177204948 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccdc15b730 con 0x7fccdc0b3060 2024-03-06T07:28:53.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.089+0000 7fcce0bc4640 1 -- 172.21.15.83:0/4177204948 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7fcccc001670 con 0x7fccdc0b3060 2024-03-06T07:28:53.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.089+0000 7fcce0bc4640 1 -- 172.21.15.83:0/4177204948 >> v1:172.21.15.83:6789/0 conn(0x7fccdc0b3060 legacy=0x7fccdc15e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.089+0000 7fcce0bc4640 1 --2- 172.21.15.83:0/4177204948 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fccdc15bf30 0x7fccc80039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.089+0000 7fcce0bc4640 1 --2- 172.21.15.83:0/4177204948 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fccc8003f60 0x7fccc8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.089+0000 7fcce0bc4640 1 --2- 172.21.15.83:0/4177204948 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fccc8004d40 0x7fccc80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.089+0000 7fcce0bc4640 1 -- 172.21.15.83:0/4177204948 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccdc0cf3f0 con 0x7fccc8003f60 2024-03-06T07:28:53.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.089+0000 7fcce0bc4640 1 -- 172.21.15.83:0/4177204948 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc800dd60 con 0x7fccc8004d40 2024-03-06T07:28:53.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.089+0000 7fcce0bc4640 1 -- 172.21.15.83:0/4177204948 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccc800df60 con 0x7fccdc15bf30 2024-03-06T07:28:53.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.089+0000 7fcce13c5640 1 --2- 172.21.15.83:0/4177204948 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fccc8003f60 0x7fccc8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.089+0000 7fcce23c7640 1 --2- 172.21.15.83:0/4177204948 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fccc8004d40 0x7fccc80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.089+0000 7fcce1bc6640 1 --2- 172.21.15.83:0/4177204948 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fccdc15bf30 0x7fccc80039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.089+0000 7fcce13c5640 1 -- 172.21.15.83:0/4177204948 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fccdc15bf30 msgr2=0x7fccc80039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:53.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.089+0000 7fcce13c5640 1 --2- 172.21.15.83:0/4177204948 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fccdc15bf30 0x7fccc80039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.089+0000 7fcce13c5640 1 -- 172.21.15.83:0/4177204948 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fccc8004d40 msgr2=0x7fccc80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:53.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.089+0000 7fcce13c5640 1 --2- 172.21.15.83:0/4177204948 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fccc8004d40 0x7fccc80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.089+0000 7fcce13c5640 1 -- 172.21.15.83:0/4177204948 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc8003520 con 0x7fccc8003f60 2024-03-06T07:28:53.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.089+0000 7fcce1bc6640 1 --2- 172.21.15.83:0/4177204948 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fccdc15bf30 0x7fccc80039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:53.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.090+0000 7fcce13c5640 1 --2- 172.21.15.83:0/4177204948 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fccc8003f60 0x7fccc8004800 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7fcccc00c240 tx=0x7fccc8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3660c95dbf060a40 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:53.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.090+0000 7fcce0bc4640 1 -- 172.21.15.83:0/4177204948 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcccc007cf0 con 0x7fccc8003f60 2024-03-06T07:28:53.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.090+0000 7fcce0bc4640 1 -- 172.21.15.83:0/4177204948 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcccc030070 con 0x7fccc8003f60 2024-03-06T07:28:53.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.090+0000 7fcce0bc4640 1 -- 172.21.15.83:0/4177204948 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcccc005d30 con 0x7fccc8003f60 2024-03-06T07:28:53.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.090+0000 7fcce3e2a640 1 -- 172.21.15.83:0/4177204948 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fccc8003f60 msgr2=0x7fccc8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.090+0000 7fcce3e2a640 1 --2- 172.21.15.83:0/4177204948 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fccc8003f60 0x7fccc8004800 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7fcccc00c240 tx=0x7fccc8005830 comp rx=0 tx=0).stop 2024-03-06T07:28:53.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.090+0000 7fcce23c7640 1 -- 172.21.15.83:0/4177204948 reap_dead start 2024-03-06T07:28:53.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.090+0000 7fcce3e2a640 1 -- 172.21.15.83:0/4177204948 shutdown_connections 2024-03-06T07:28:53.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.090+0000 7fcce3e2a640 1 -- 172.21.15.83:0/4177204948 >> 172.21.15.83:0/4177204948 conn(0x7fccdc0cf8a0 msgr2=0x7fccdc0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:53.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.090+0000 7fcce3e2a640 1 -- 172.21.15.83:0/4177204948 shutdown_connections 2024-03-06T07:28:53.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.090+0000 7fcce3e2a640 1 -- 172.21.15.83:0/4177204948 wait complete. 2024-03-06T07:28:53.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.092+0000 7fcce3e2a640 1 Processor -- start 2024-03-06T07:28:53.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.092+0000 7fcce3e2a640 1 -- start start 2024-03-06T07:28:53.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.092+0000 7fcce3e2a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fccc8003f60 0x7fccdc15d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.092+0000 7fcce3e2a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fccdc06d2b0 0x7fccdc06d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.092+0000 7fcce3e2a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fccdc12f8c0 0x7fccdc12fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.092+0000 7fcce3e2a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccdc15ef60 con 0x7fccc8003f60 2024-03-06T07:28:53.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.092+0000 7fcce3e2a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccdc15b210 con 0x7fccdc06d2b0 2024-03-06T07:28:53.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.092+0000 7fcce3e2a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccdc15f0e0 con 0x7fccdc12f8c0 2024-03-06T07:28:53.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.092+0000 7fcce1bc6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fccc8003f60 0x7fccdc15d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.092+0000 7fcce1bc6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fccc8003f60 0x7fccdc15d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48862/0 (socket says 172.21.15.83:48862) 2024-03-06T07:28:53.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.092+0000 7fcce1bc6640 1 -- 172.21.15.83:0/1161709527 learned_addr learned my addr 172.21.15.83:0/1161709527 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:53.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.092+0000 7fcce13c5640 1 --2- 172.21.15.83:0/1161709527 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fccdc06d2b0 0x7fccdc06d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.092+0000 7fcce23c7640 1 --2- 172.21.15.83:0/1161709527 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fccdc12f8c0 0x7fccdc12fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.093+0000 7fcce1bc6640 1 -- 172.21.15.83:0/1161709527 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fccdc12f8c0 msgr2=0x7fccdc12fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.093+0000 7fcce1bc6640 1 --2- 172.21.15.83:0/1161709527 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fccdc12f8c0 0x7fccdc12fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.093+0000 7fcce1bc6640 1 -- 172.21.15.83:0/1161709527 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fccdc06d2b0 msgr2=0x7fccdc06d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.093+0000 7fcce1bc6640 1 --2- 172.21.15.83:0/1161709527 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fccdc06d2b0 0x7fccdc06d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.093+0000 7fcce1bc6640 1 -- 172.21.15.83:0/1161709527 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccdc15da50 con 0x7fccc8003f60 2024-03-06T07:28:53.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.093+0000 7fcce23c7640 1 --2- 172.21.15.83:0/1161709527 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fccdc12f8c0 0x7fccdc12fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:53.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.093+0000 7fcce1bc6640 1 --2- 172.21.15.83:0/1161709527 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fccc8003f60 0x7fccdc15d2b0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7fccd800e580 tx=0x7fccd800ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:53.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.093+0000 7fccd27fc640 1 -- 172.21.15.83:0/1161709527 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd80052e0 con 0x7fccc8003f60 2024-03-06T07:28:53.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.093+0000 7fccd27fc640 1 -- 172.21.15.83:0/1161709527 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccd8018070 con 0x7fccc8003f60 2024-03-06T07:28:53.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.093+0000 7fccd27fc640 1 -- 172.21.15.83:0/1161709527 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd8003830 con 0x7fccc8003f60 2024-03-06T07:28:53.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.093+0000 7fcce3e2a640 1 -- 172.21.15.83:0/1161709527 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccdc15dd30 con 0x7fccc8003f60 2024-03-06T07:28:53.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.093+0000 7fcce3e2a640 1 -- 172.21.15.83:0/1161709527 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccdc15e2f0 con 0x7fccc8003f60 2024-03-06T07:28:53.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.094+0000 7fcce3e2a640 1 -- 172.21.15.83:0/1161709527 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccdc06dc50 con 0x7fccc8003f60 2024-03-06T07:28:53.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.094+0000 7fccd27fc640 1 -- 172.21.15.83:0/1161709527 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fccd80039d0 con 0x7fccc8003f60 2024-03-06T07:28:53.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.095+0000 7fccd27fc640 1 --2- 172.21.15.83:0/1161709527 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fccc006b500 0x7fccc006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.095+0000 7fcce13c5640 1 --2- 172.21.15.83:0/1161709527 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fccc006b500 0x7fccc006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.095+0000 7fccd27fc640 1 -- 172.21.15.83:0/1161709527 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fccd8089e20 con 0x7fccc8003f60 2024-03-06T07:28:53.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.095+0000 7fcce13c5640 1 --2- 172.21.15.83:0/1161709527 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fccc006b500 0x7fccc006d9c0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fcccc00c240 tx=0x7fcccc007000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:53.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.099+0000 7fccd27fc640 1 -- 172.21.15.83:0/1161709527 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fccd8058cb0 con 0x7fccc8003f60 2024-03-06T07:28:53.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.273+0000 7fcce3e2a640 1 -- 172.21.15.83:0/1161709527 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fccdc04f330 con 0x7fccc8003f60 2024-03-06T07:28:53.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.274+0000 7fccd27fc640 1 -- 172.21.15.83:0/1161709527 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) v1 ==== 74+0+16704 (secure 0 0 0) 0x7fccd80585b0 con 0x7fccc8003f60 2024-03-06T07:28:53.277 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:28:53.277 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":127,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:28:52.306386+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:50.508389+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:51.266049+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:53.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.276+0000 7fcce3e2a640 1 -- 172.21.15.83:0/1161709527 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fccc006b500 msgr2=0x7fccc006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.276+0000 7fcce3e2a640 1 --2- 172.21.15.83:0/1161709527 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fccc006b500 0x7fccc006d9c0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fcccc00c240 tx=0x7fcccc007000 comp rx=0 tx=0).stop 2024-03-06T07:28:53.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.276+0000 7fcce3e2a640 1 -- 172.21.15.83:0/1161709527 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fccc8003f60 msgr2=0x7fccdc15d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.276+0000 7fcce3e2a640 1 --2- 172.21.15.83:0/1161709527 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fccc8003f60 0x7fccdc15d2b0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7fccd800e580 tx=0x7fccd800ea40 comp rx=0 tx=0).stop 2024-03-06T07:28:53.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.276+0000 7fcce3e2a640 1 -- 172.21.15.83:0/1161709527 shutdown_connections 2024-03-06T07:28:53.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.276+0000 7fcce3e2a640 1 --2- 172.21.15.83:0/1161709527 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fccc006b500 0x7fccc006d9c0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.276+0000 7fcce3e2a640 1 --2- 172.21.15.83:0/1161709527 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fccdc12f8c0 0x7fccdc12fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.276+0000 7fcce3e2a640 1 --2- 172.21.15.83:0/1161709527 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fccdc06d2b0 0x7fccdc06d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.276+0000 7fcce3e2a640 1 --2- 172.21.15.83:0/1161709527 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fccc8003f60 0x7fccdc15d2b0 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.276+0000 7fcce3e2a640 1 -- 172.21.15.83:0/1161709527 >> 172.21.15.83:0/1161709527 conn(0x7fccdc0cf8a0 msgr2=0x7fccdc0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:53.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.277+0000 7fcce3e2a640 1 -- 172.21.15.83:0/1161709527 shutdown_connections 2024-03-06T07:28:53.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.277+0000 7fcce3e2a640 1 -- 172.21.15.83:0/1161709527 wait complete. 2024-03-06T07:28:53.290 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:53.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.389+0000 7f736c408640 1 Processor -- start 2024-03-06T07:28:53.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.389+0000 7f736c408640 1 -- start start 2024-03-06T07:28:53.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.390+0000 7f736c408640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f736413dc10 0x7f73640b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.390+0000 7f736c408640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f736415ef60 0x7f73640b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.390+0000 7f736c408640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f736415f580 con 0x7f736415bf30 2024-03-06T07:28:53.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.390+0000 7f736c408640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73640cf3f0 con 0x7f736415ef60 2024-03-06T07:28:53.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.390+0000 7f736c408640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f736415b210 con 0x7f736413dc10 2024-03-06T07:28:53.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.390+0000 7f736a9a5640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f736415ef60 0x7f73640b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.390+0000 7f736a9a5640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f736415ef60 0x7f73640b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48868/0 (socket says 172.21.15.83:48868) 2024-03-06T07:28:53.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.390+0000 7f736a9a5640 1 -- 172.21.15.83:0/2846580570 learned_addr learned my addr 172.21.15.83:0/2846580570 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:53.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.390+0000 7f736a1a4640 1 --2- 172.21.15.83:0/2846580570 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f736413dc10 0x7f73640b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.390+0000 7f736a9a5640 1 -- 172.21.15.83:0/2846580570 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f736413dc10 msgr2=0x7f73640b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:53.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.390+0000 7f736a9a5640 1 --2- 172.21.15.83:0/2846580570 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f736413dc10 0x7f73640b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.390+0000 7f736a9a5640 1 -- 172.21.15.83:0/2846580570 >> v1:172.21.15.183:6789/0 conn(0x7f736415bf30 legacy=0x7f736415e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:53.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.390+0000 7f736a9a5640 1 -- 172.21.15.83:0/2846580570 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f736415b450 con 0x7f736415ef60 2024-03-06T07:28:53.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.390+0000 7f736a1a4640 1 --2- 172.21.15.83:0/2846580570 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f736413dc10 0x7f73640b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:53.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.391+0000 7f736a9a5640 1 --2- 172.21.15.83:0/2846580570 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f736415ef60 0x7f73640b2e90 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f735c009e50 tx=0x7f735c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f28fdd8dbbf97ffe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:53.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.391+0000 7f73691a2640 1 -- 172.21.15.83:0/2846580570 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735c02c9e0 con 0x7f736415ef60 2024-03-06T07:28:53.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.391+0000 7f73691a2640 1 -- 172.21.15.83:0/2846580570 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f735c02cb80 con 0x7f736415ef60 2024-03-06T07:28:53.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.391+0000 7f73691a2640 1 -- 172.21.15.83:0/2846580570 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735c0343f0 con 0x7f736415ef60 2024-03-06T07:28:53.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.392+0000 7f736c408640 1 -- 172.21.15.83:0/2846580570 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f736415ef60 msgr2=0x7f73640b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.392+0000 7f736c408640 1 --2- 172.21.15.83:0/2846580570 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f736415ef60 0x7f73640b2e90 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f735c009e50 tx=0x7f735c02be40 comp rx=0 tx=0).stop 2024-03-06T07:28:53.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.392+0000 7f736c408640 1 -- 172.21.15.83:0/2846580570 shutdown_connections 2024-03-06T07:28:53.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.392+0000 7f736c408640 1 --2- 172.21.15.83:0/2846580570 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f736415ef60 0x7f73640b2e90 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.392+0000 7f736c408640 1 --2- 172.21.15.83:0/2846580570 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f736413dc10 0x7f73640b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.392+0000 7f736c408640 1 -- 172.21.15.83:0/2846580570 >> 172.21.15.83:0/2846580570 conn(0x7f73640cf8a0 msgr2=0x7f73640cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:53.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.392+0000 7f736c408640 1 -- 172.21.15.83:0/2846580570 shutdown_connections 2024-03-06T07:28:53.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.392+0000 7f736c408640 1 -- 172.21.15.83:0/2846580570 wait complete. 2024-03-06T07:28:53.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.393+0000 7f736c408640 1 Processor -- start 2024-03-06T07:28:53.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.393+0000 7f736c408640 1 -- start start 2024-03-06T07:28:53.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.394+0000 7f736c408640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f736413dc10 0x7f736413ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.394+0000 7f736c408640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f736415bf30 0x7f73641e4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.394+0000 7f736c408640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f736415ef60 0x7f73641e5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.394+0000 7f736c408640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73640708d0 con 0x7f736415ef60 2024-03-06T07:28:53.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.394+0000 7f736c408640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f736413dff0 con 0x7f736415bf30 2024-03-06T07:28:53.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.394+0000 7f736c408640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7364070a50 con 0x7f736413dc10 2024-03-06T07:28:53.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.394+0000 7f736a9a5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f736415ef60 0x7f73641e5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.394+0000 7f736a9a5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f736415ef60 0x7f73641e5f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48878/0 (socket says 172.21.15.83:48878) 2024-03-06T07:28:53.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.394+0000 7f736a9a5640 1 -- 172.21.15.83:0/1035420774 learned_addr learned my addr 172.21.15.83:0/1035420774 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:53.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.394+0000 7f736a1a4640 1 --2- 172.21.15.83:0/1035420774 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f736413dc10 0x7f736413ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.394+0000 7f73699a3640 1 --2- 172.21.15.83:0/1035420774 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f736415bf30 0x7f73641e4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.394+0000 7f736a9a5640 1 -- 172.21.15.83:0/1035420774 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f736413dc10 msgr2=0x7f736413ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.394+0000 7f736a9a5640 1 --2- 172.21.15.83:0/1035420774 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f736413dc10 0x7f736413ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.394+0000 7f736a9a5640 1 -- 172.21.15.83:0/1035420774 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f736415bf30 msgr2=0x7f73641e4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.394+0000 7f736a9a5640 1 --2- 172.21.15.83:0/1035420774 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f736415bf30 0x7f73641e4e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.394+0000 7f736a9a5640 1 -- 172.21.15.83:0/1035420774 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f736413b190 con 0x7f736415ef60 2024-03-06T07:28:53.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.394+0000 7f736a9a5640 1 --2- 172.21.15.83:0/1035420774 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f736415ef60 0x7f73641e5f90 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f735c036040 tx=0x7f735c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:53.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.394+0000 7f735affd640 1 -- 172.21.15.83:0/1035420774 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735c043020 con 0x7f736415ef60 2024-03-06T07:28:53.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.394+0000 7f735affd640 1 -- 172.21.15.83:0/1035420774 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f735c03e450 con 0x7f736415ef60 2024-03-06T07:28:53.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.394+0000 7f736c408640 1 -- 172.21.15.83:0/1035420774 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f736413b430 con 0x7f736415ef60 2024-03-06T07:28:53.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.394+0000 7f735affd640 1 -- 172.21.15.83:0/1035420774 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735c03e860 con 0x7f736415ef60 2024-03-06T07:28:53.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.394+0000 7f736c408640 1 -- 172.21.15.83:0/1035420774 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73641e4600 con 0x7f736415ef60 2024-03-06T07:28:53.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.395+0000 7f736c408640 1 -- 172.21.15.83:0/1035420774 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f736404f330 con 0x7f736415ef60 2024-03-06T07:28:53.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.396+0000 7f735affd640 1 -- 172.21.15.83:0/1035420774 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f735c046020 con 0x7f736415ef60 2024-03-06T07:28:53.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.396+0000 7f735affd640 1 --2- 172.21.15.83:0/1035420774 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f734806b4d0 0x7f734806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.396+0000 7f735affd640 1 -- 172.21.15.83:0/1035420774 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f735c0acab0 con 0x7f736415ef60 2024-03-06T07:28:53.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.396+0000 7f736a1a4640 1 --2- 172.21.15.83:0/1035420774 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f734806b4d0 0x7f734806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.397+0000 7f736a1a4640 1 --2- 172.21.15.83:0/1035420774 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f734806b4d0 0x7f734806d990 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f736000a990 tx=0x7f7360008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:53.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.401+0000 7f735affd640 1 -- 172.21.15.83:0/1035420774 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f735c04d030 con 0x7f736415ef60 2024-03-06T07:28:53.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.571+0000 7f736c408640 1 -- 172.21.15.83:0/1035420774 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f736406eb00 con 0x7f736415ef60 2024-03-06T07:28:53.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.572+0000 7f735affd640 1 -- 172.21.15.83:0/1035420774 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) v1 ==== 74+0+16704 (secure 0 0 0) 0x7f735c07b9c0 con 0x7f736415ef60 2024-03-06T07:28:53.574 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:28:53.575 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":127,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:28:52.306386+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:50.508389+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:51.266049+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:53.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.574+0000 7f736c408640 1 -- 172.21.15.83:0/1035420774 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f734806b4d0 msgr2=0x7f734806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.574+0000 7f736c408640 1 --2- 172.21.15.83:0/1035420774 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f734806b4d0 0x7f734806d990 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f736000a990 tx=0x7f7360008040 comp rx=0 tx=0).stop 2024-03-06T07:28:53.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.574+0000 7f736c408640 1 -- 172.21.15.83:0/1035420774 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f736415ef60 msgr2=0x7f73641e5f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.574+0000 7f736c408640 1 --2- 172.21.15.83:0/1035420774 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f736415ef60 0x7f73641e5f90 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f735c036040 tx=0x7f735c004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:53.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.574+0000 7f736c408640 1 -- 172.21.15.83:0/1035420774 shutdown_connections 2024-03-06T07:28:53.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.574+0000 7f736c408640 1 --2- 172.21.15.83:0/1035420774 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f734806b4d0 0x7f734806d990 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.574+0000 7f736c408640 1 --2- 172.21.15.83:0/1035420774 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f736415ef60 0x7f73641e5f90 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.574+0000 7f736c408640 1 --2- 172.21.15.83:0/1035420774 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f736415bf30 0x7f73641e4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.574+0000 7f736c408640 1 --2- 172.21.15.83:0/1035420774 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f736413dc10 0x7f736413ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.574+0000 7f736c408640 1 -- 172.21.15.83:0/1035420774 >> 172.21.15.83:0/1035420774 conn(0x7f73640cf8a0 msgr2=0x7f736413be40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:53.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.574+0000 7f736c408640 1 -- 172.21.15.83:0/1035420774 shutdown_connections 2024-03-06T07:28:53.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.574+0000 7f736c408640 1 -- 172.21.15.83:0/1035420774 wait complete. 2024-03-06T07:28:53.588 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:28:53.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.689+0000 7ffab7484640 1 Processor -- start 2024-03-06T07:28:53.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.689+0000 7ffab7484640 1 -- start start 2024-03-06T07:28:53.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.690+0000 7ffab7484640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ffab00b0db0 0x7ffab0066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.690+0000 7ffab7484640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffab00ce750 0x7ffab00b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.690+0000 7ffab7484640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffab006b050 con 0x7ffab015f4f0 2024-03-06T07:28:53.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.690+0000 7ffab7484640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffab006b2c0 con 0x7ffab00b0db0 2024-03-06T07:28:53.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.690+0000 7ffab7484640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffab00b0890 con 0x7ffab00ce750 2024-03-06T07:28:53.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.690+0000 7ffab4a1f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ffab00b0db0 0x7ffab0066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.690+0000 7ffab4a1f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ffab00b0db0 0x7ffab0066d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48892/0 (socket says 172.21.15.83:48892) 2024-03-06T07:28:53.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.690+0000 7ffab5220640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ffab015f4f0 0x7ffab00ce040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52194/0 (socket says 172.21.15.83:52194) 2024-03-06T07:28:53.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.690+0000 7ffab4a1f640 1 -- 172.21.15.83:0/2798826200 learned_addr learned my addr 172.21.15.83:0/2798826200 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:53.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.690+0000 7ffab5a21640 1 --2- 172.21.15.83:0/2798826200 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffab00ce750 0x7ffab00b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.690+0000 7ffab4a1f640 1 -- 172.21.15.83:0/2798826200 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffab00ce750 msgr2=0x7ffab00b4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:53.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.690+0000 7ffab4a1f640 1 --2- 172.21.15.83:0/2798826200 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffab00ce750 0x7ffab00b4ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.690+0000 7ffab4a1f640 1 -- 172.21.15.83:0/2798826200 >> v1:172.21.15.83:6789/0 conn(0x7ffab015f4f0 legacy=0x7ffab00ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.690+0000 7ffaa7fff640 1 -- 172.21.15.83:0/2798826200 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 67111660 0 0) 0x7ffab006b050 con 0x7ffab015f4f0 2024-03-06T07:28:53.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.690+0000 7ffab4a1f640 1 -- 172.21.15.83:0/2798826200 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab00672e0 con 0x7ffab00b0db0 2024-03-06T07:28:53.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.691+0000 7ffab4a1f640 1 --2- 172.21.15.83:0/2798826200 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffab00b0db0 0x7ffab0066d00 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0009870 tx=0x7ffaa002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e4f3565cb39023a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:53.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.691+0000 7ffaa7fff640 1 -- 172.21.15.83:0/2798826200 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa0034070 con 0x7ffab00b0db0 2024-03-06T07:28:53.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.691+0000 7ffaa7fff640 1 -- 172.21.15.83:0/2798826200 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaa0030840 con 0x7ffab00b0db0 2024-03-06T07:28:53.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.691+0000 7ffaa7fff640 1 -- 172.21.15.83:0/2798826200 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa0030c10 con 0x7ffab00b0db0 2024-03-06T07:28:53.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.691+0000 7ffab7484640 1 -- 172.21.15.83:0/2798826200 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffab00b0db0 msgr2=0x7ffab0066d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.691+0000 7ffab7484640 1 --2- 172.21.15.83:0/2798826200 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffab00b0db0 0x7ffab0066d00 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0009870 tx=0x7ffaa002b980 comp rx=0 tx=0).stop 2024-03-06T07:28:53.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.691+0000 7ffab7484640 1 -- 172.21.15.83:0/2798826200 shutdown_connections 2024-03-06T07:28:53.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.691+0000 7ffab7484640 1 --2- 172.21.15.83:0/2798826200 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffab00ce750 0x7ffab00b4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.691+0000 7ffab7484640 1 --2- 172.21.15.83:0/2798826200 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffab00b0db0 0x7ffab0066d00 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.692+0000 7ffab7484640 1 -- 172.21.15.83:0/2798826200 >> 172.21.15.83:0/2798826200 conn(0x7ffab00b1d70 msgr2=0x7ffab00b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:53.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.692+0000 7ffab7484640 1 -- 172.21.15.83:0/2798826200 shutdown_connections 2024-03-06T07:28:53.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.692+0000 7ffab7484640 1 -- 172.21.15.83:0/2798826200 wait complete. 2024-03-06T07:28:53.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.693+0000 7ffab7484640 1 Processor -- start 2024-03-06T07:28:53.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.693+0000 7ffab7484640 1 -- start start 2024-03-06T07:28:53.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.693+0000 7ffab7484640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffab00b0db0 0x7ffab0156ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.693+0000 7ffab7484640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffab00ce750 0x7ffab014f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.693+0000 7ffab7484640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffab015f4f0 0x7ffab00c99d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.693+0000 7ffab7484640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab015f8d0 con 0x7ffab00b0db0 2024-03-06T07:28:53.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.693+0000 7ffab7484640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab00cebe0 con 0x7ffab00ce750 2024-03-06T07:28:53.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.693+0000 7ffab7484640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffab00c9540 con 0x7ffab015f4f0 2024-03-06T07:28:53.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.693+0000 7ffab5220640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffab00b0db0 0x7ffab0156ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.693+0000 7ffab5220640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffab00b0db0 0x7ffab0156ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48904/0 (socket says 172.21.15.83:48904) 2024-03-06T07:28:53.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.694+0000 7ffab5220640 1 -- 172.21.15.83:0/2378745870 learned_addr learned my addr 172.21.15.83:0/2378745870 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:53.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.694+0000 7ffab5a21640 1 --2- 172.21.15.83:0/2378745870 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffab015f4f0 0x7ffab00c99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.694+0000 7ffab5220640 1 -- 172.21.15.83:0/2378745870 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffab015f4f0 msgr2=0x7ffab00c99d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.694+0000 7ffab5220640 1 --2- 172.21.15.83:0/2378745870 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffab015f4f0 0x7ffab00c99d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.694+0000 7ffab5220640 1 -- 172.21.15.83:0/2378745870 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffab00ce750 msgr2=0x7ffab014f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.694+0000 7ffab5220640 1 --2- 172.21.15.83:0/2378745870 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffab00ce750 0x7ffab014f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.694+0000 7ffab5220640 1 -- 172.21.15.83:0/2378745870 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab00c8930 con 0x7ffab00b0db0 2024-03-06T07:28:53.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.694+0000 7ffab5220640 1 --2- 172.21.15.83:0/2378745870 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffab00b0db0 0x7ffab0156ff0 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7ffaac00c430 tx=0x7ffaac00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:53.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.694+0000 7ffaa5ffb640 1 -- 172.21.15.83:0/2378745870 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaac0074e0 con 0x7ffab00b0db0 2024-03-06T07:28:53.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.694+0000 7ffaa5ffb640 1 -- 172.21.15.83:0/2378745870 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaac0047e0 con 0x7ffab00b0db0 2024-03-06T07:28:53.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.694+0000 7ffab7484640 1 -- 172.21.15.83:0/2378745870 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffab00c8c10 con 0x7ffab00b0db0 2024-03-06T07:28:53.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.694+0000 7ffaa5ffb640 1 -- 172.21.15.83:0/2378745870 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaac004bd0 con 0x7ffab00b0db0 2024-03-06T07:28:53.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.694+0000 7ffab7484640 1 -- 172.21.15.83:0/2378745870 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffab00c91d0 con 0x7ffab00b0db0 2024-03-06T07:28:53.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.695+0000 7ffab7484640 1 -- 172.21.15.83:0/2378745870 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffab009e3d0 con 0x7ffab00b0db0 2024-03-06T07:28:53.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.696+0000 7ffaa5ffb640 1 -- 172.21.15.83:0/2378745870 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffaac011070 con 0x7ffab00b0db0 2024-03-06T07:28:53.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.696+0000 7ffaa5ffb640 1 --2- 172.21.15.83:0/2378745870 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffa9406b4d0 0x7ffa9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.696+0000 7ffaa5ffb640 1 -- 172.21.15.83:0/2378745870 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffaac089ec0 con 0x7ffab00b0db0 2024-03-06T07:28:53.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.696+0000 7ffab4a1f640 1 --2- 172.21.15.83:0/2378745870 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffa9406b4d0 0x7ffa9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.697+0000 7ffab4a1f640 1 --2- 172.21.15.83:0/2378745870 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffa9406b4d0 0x7ffa9406d990 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0036000 tx=0x7ffaa003b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:53.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.701+0000 7ffaa5ffb640 1 -- 172.21.15.83:0/2378745870 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffaac00e280 con 0x7ffab00b0db0 2024-03-06T07:28:53.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.906+0000 7ffab7484640 1 -- 172.21.15.83:0/2378745870 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7ffab00ca470 con 0x7ffab00b0db0 2024-03-06T07:28:53.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.961+0000 7ffaa5ffb640 1 -- 172.21.15.83:0/2378745870 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v72) v1 ==== 118+0+0 (secure 0 0 0) 0x7ffaac058cb0 con 0x7ffab00b0db0 2024-03-06T07:28:53.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.963+0000 7ffab7484640 1 -- 172.21.15.83:0/2378745870 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffa9406b4d0 msgr2=0x7ffa9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.963+0000 7ffab7484640 1 --2- 172.21.15.83:0/2378745870 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffa9406b4d0 0x7ffa9406d990 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0036000 tx=0x7ffaa003b040 comp rx=0 tx=0).stop 2024-03-06T07:28:53.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.963+0000 7ffab7484640 1 -- 172.21.15.83:0/2378745870 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffab00b0db0 msgr2=0x7ffab0156ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.963+0000 7ffab7484640 1 --2- 172.21.15.83:0/2378745870 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffab00b0db0 0x7ffab0156ff0 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7ffaac00c430 tx=0x7ffaac00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.963+0000 7ffab7484640 1 -- 172.21.15.83:0/2378745870 shutdown_connections 2024-03-06T07:28:53.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.963+0000 7ffab7484640 1 --2- 172.21.15.83:0/2378745870 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffa9406b4d0 0x7ffa9406d990 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.963+0000 7ffab7484640 1 --2- 172.21.15.83:0/2378745870 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffab015f4f0 0x7ffab00c99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.963+0000 7ffab7484640 1 --2- 172.21.15.83:0/2378745870 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffab00ce750 0x7ffab014f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.963+0000 7ffab7484640 1 --2- 172.21.15.83:0/2378745870 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffab00b0db0 0x7ffab0156ff0 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.963+0000 7ffab7484640 1 -- 172.21.15.83:0/2378745870 >> 172.21.15.83:0/2378745870 conn(0x7ffab00b1d70 msgr2=0x7ffab006e9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:53.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.963+0000 7ffab7484640 1 -- 172.21.15.83:0/2378745870 shutdown_connections 2024-03-06T07:28:53.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:53.963+0000 7ffab7484640 1 -- 172.21.15.83:0/2378745870 wait complete. 2024-03-06T07:28:53.979 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:28:53.980 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:28:54.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.099+0000 7ff1c0e15640 1 Processor -- start 2024-03-06T07:28:54.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.099+0000 7ff1c0e15640 1 -- start start 2024-03-06T07:28:54.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.099+0000 7ff1c0e15640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff1bc0cfa90 0x7ff1bc0cff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.099+0000 7ff1c0e15640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ff1bc0b33a0 0x7ff1bc0b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.100+0000 7ff1c0e15640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1bc158ac0 con 0x7ff1bc1586e0 2024-03-06T07:28:54.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.100+0000 7ff1c0e15640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1bc15b2a0 con 0x7ff1bc0b33a0 2024-03-06T07:28:54.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.100+0000 7ff1c0e15640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc15b540 con 0x7ff1bc0cfa90 2024-03-06T07:28:54.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.100+0000 7ff1ba59c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff1bc1586e0 0x7ff1bc0aea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52210/0 (socket says 172.21.15.83:52210) 2024-03-06T07:28:54.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.100+0000 7ff1ba59c640 1 -- 172.21.15.83:0/93593254 learned_addr learned my addr 172.21.15.83:0/93593254 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:28:54.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.100+0000 7ff1b9d9b640 1 -- 172.21.15.83:0/93593254 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 58109983 0 0) 0x7ff1bc158ac0 con 0x7ff1bc1586e0 2024-03-06T07:28:54.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.100+0000 7ff1b9d9b640 1 -- 172.21.15.83:0/93593254 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1a8003520 con 0x7ff1bc1586e0 2024-03-06T07:28:54.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.100+0000 7ff1b3fff640 1 --2- 172.21.15.83:0/93593254 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff1bc0cfa90 0x7ff1bc0cff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.100+0000 7ff1bad9d640 1 --2- 172.21.15.83:0/93593254 >> v2:172.21.15.183:3300/0 conn(0x7ff1bc0b33a0 0x7ff1bc0b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.100+0000 7ff1b9d9b640 1 -- 172.21.15.83:0/93593254 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3351233734 0 0) 0x7ff1a8003520 con 0x7ff1bc1586e0 2024-03-06T07:28:54.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.100+0000 7ff1b9d9b640 1 -- 172.21.15.83:0/93593254 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff1bc0cfa90 msgr2=0x7ff1bc0cff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:54.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.100+0000 7ff1b9d9b640 1 --2- 172.21.15.83:0/93593254 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff1bc0cfa90 0x7ff1bc0cff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.100+0000 7ff1b9d9b640 1 -- 172.21.15.83:0/93593254 >> v2:172.21.15.183:3300/0 conn(0x7ff1bc0b33a0 msgr2=0x7ff1bc0b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:54.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.100+0000 7ff1b9d9b640 1 --2- 172.21.15.83:0/93593254 >> v2:172.21.15.183:3300/0 conn(0x7ff1bc0b33a0 0x7ff1bc0b37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.100+0000 7ff1b9d9b640 1 -- 172.21.15.83:0/93593254 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1bc15b820 con 0x7ff1bc1586e0 2024-03-06T07:28:54.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.100+0000 7ff1b9d9b640 1 -- 172.21.15.83:0/93593254 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7ff1ac001670 con 0x7ff1bc1586e0 2024-03-06T07:28:54.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.100+0000 7ff1b3fff640 1 --2- 172.21.15.83:0/93593254 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff1bc0cfa90 0x7ff1bc0cff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:54.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.100+0000 7ff1b9d9b640 1 -- 172.21.15.83:0/93593254 >> v1:172.21.15.83:6789/0 conn(0x7ff1bc1586e0 legacy=0x7ff1bc0aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.100+0000 7ff1b9d9b640 1 --2- 172.21.15.83:0/93593254 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff1bc0cfa90 0x7ff1a80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.100+0000 7ff1b9d9b640 1 --2- 172.21.15.83:0/93593254 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1a8003ef0 0x7ff1a8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.100+0000 7ff1b9d9b640 1 --2- 172.21.15.83:0/93593254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1a8004d40 0x7ff1a80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.100+0000 7ff1b9d9b640 1 -- 172.21.15.83:0/93593254 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc15b2a0 con 0x7ff1a8004d40 2024-03-06T07:28:54.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.100+0000 7ff1b9d9b640 1 -- 172.21.15.83:0/93593254 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc15b540 con 0x7ff1a8003ef0 2024-03-06T07:28:54.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.100+0000 7ff1b9d9b640 1 -- 172.21.15.83:0/93593254 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a800b9f0 con 0x7ff1bc0cfa90 2024-03-06T07:28:54.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.100+0000 7ff1bad9d640 1 --2- 172.21.15.83:0/93593254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1a8004d40 0x7ff1a80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.101+0000 7ff1ba59c640 1 --2- 172.21.15.83:0/93593254 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff1bc0cfa90 0x7ff1a80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.101+0000 7ff1b3fff640 1 --2- 172.21.15.83:0/93593254 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1a8003ef0 0x7ff1a8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.101+0000 7ff1bad9d640 1 -- 172.21.15.83:0/93593254 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff1bc0cfa90 msgr2=0x7ff1a80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:54.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.101+0000 7ff1bad9d640 1 --2- 172.21.15.83:0/93593254 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff1bc0cfa90 0x7ff1a80039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.101+0000 7ff1bad9d640 1 -- 172.21.15.83:0/93593254 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1a8003ef0 msgr2=0x7ff1a8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:54.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.101+0000 7ff1bad9d640 1 --2- 172.21.15.83:0/93593254 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1a8003ef0 0x7ff1a8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.101+0000 7ff1bad9d640 1 -- 172.21.15.83:0/93593254 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a8003520 con 0x7ff1a8004d40 2024-03-06T07:28:54.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.102+0000 7ff1bad9d640 1 --2- 172.21.15.83:0/93593254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1a8004d40 0x7ff1a80051c0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4009fc0 tx=0x7ff1a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf4b4140ee2884e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:54.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.102+0000 7ff1b9d9b640 1 -- 172.21.15.83:0/93593254 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b4039070 con 0x7ff1a8004d40 2024-03-06T07:28:54.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.102+0000 7ff1b9d9b640 1 -- 172.21.15.83:0/93593254 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1b402cc60 con 0x7ff1a8004d40 2024-03-06T07:28:54.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.102+0000 7ff1b9d9b640 1 -- 172.21.15.83:0/93593254 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b4009c10 con 0x7ff1a8004d40 2024-03-06T07:28:54.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.103+0000 7ff1c0e15640 1 -- 172.21.15.83:0/93593254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1a8004d40 msgr2=0x7ff1a80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.103+0000 7ff1c0e15640 1 --2- 172.21.15.83:0/93593254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1a8004d40 0x7ff1a80051c0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4009fc0 tx=0x7ff1a8005830 comp rx=0 tx=0).stop 2024-03-06T07:28:54.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.103+0000 7ff1bad9d640 1 -- 172.21.15.83:0/93593254 reap_dead start 2024-03-06T07:28:54.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.103+0000 7ff1c0e15640 1 -- 172.21.15.83:0/93593254 shutdown_connections 2024-03-06T07:28:54.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.103+0000 7ff1c0e15640 1 -- 172.21.15.83:0/93593254 >> 172.21.15.83:0/93593254 conn(0x7ff1bc06b190 msgr2=0x7ff1bc06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:54.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.103+0000 7ff1c0e15640 1 -- 172.21.15.83:0/93593254 shutdown_connections 2024-03-06T07:28:54.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.103+0000 7ff1c0e15640 1 -- 172.21.15.83:0/93593254 wait complete. 2024-03-06T07:28:54.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.105+0000 7ff1c0e15640 1 Processor -- start 2024-03-06T07:28:54.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.105+0000 7ff1c0e15640 1 -- start start 2024-03-06T07:28:54.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.106+0000 7ff1c0e15640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1bc0ce770 0x7ff1bc1f7f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.106+0000 7ff1c0e15640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1bc0ceb50 0x7ff1bc1507a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.106+0000 7ff1c0e15640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff1bc1f8450 0x7ff1bc150ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.106+0000 7ff1c0e15640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc0af150 con 0x7ff1bc0ce770 2024-03-06T07:28:54.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.106+0000 7ff1c0e15640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc15bde0 con 0x7ff1bc0ceb50 2024-03-06T07:28:54.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.106+0000 7ff1c0e15640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc071350 con 0x7ff1bc1f8450 2024-03-06T07:28:54.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.107+0000 7ff1ba59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1bc0ce770 0x7ff1bc1f7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.107+0000 7ff1ba59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1bc0ce770 0x7ff1bc1f7f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48916/0 (socket says 172.21.15.83:48916) 2024-03-06T07:28:54.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.107+0000 7ff1ba59c640 1 -- 172.21.15.83:0/189722158 learned_addr learned my addr 172.21.15.83:0/189722158 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:54.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.107+0000 7ff1bad9d640 1 --2- 172.21.15.83:0/189722158 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff1bc1f8450 0x7ff1bc150ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.107+0000 7ff1b3fff640 1 --2- 172.21.15.83:0/189722158 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1bc0ceb50 0x7ff1bc1507a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.107+0000 7ff1ba59c640 1 -- 172.21.15.83:0/189722158 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff1bc1f8450 msgr2=0x7ff1bc150ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.107+0000 7ff1ba59c640 1 --2- 172.21.15.83:0/189722158 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff1bc1f8450 0x7ff1bc150ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.107+0000 7ff1ba59c640 1 -- 172.21.15.83:0/189722158 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1bc0ceb50 msgr2=0x7ff1bc1507a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.107+0000 7ff1ba59c640 1 --2- 172.21.15.83:0/189722158 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1bc0ceb50 0x7ff1bc1507a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.107+0000 7ff1ba59c640 1 -- 172.21.15.83:0/189722158 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1bc1f8a00 con 0x7ff1bc0ce770 2024-03-06T07:28:54.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.107+0000 7ff1ba59c640 1 --2- 172.21.15.83:0/189722158 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1bc0ce770 0x7ff1bc1f7f10 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac00fd10 tx=0x7ff1ac00e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:54.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.107+0000 7ff1b2ffd640 1 -- 172.21.15.83:0/189722158 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1ac0016e0 con 0x7ff1bc0ce770 2024-03-06T07:28:54.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.107+0000 7ff1b2ffd640 1 -- 172.21.15.83:0/189722158 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1ac014030 con 0x7ff1bc0ce770 2024-03-06T07:28:54.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.107+0000 7ff1b2ffd640 1 -- 172.21.15.83:0/189722158 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1ac007390 con 0x7ff1bc0ce770 2024-03-06T07:28:54.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.107+0000 7ff1c0e15640 1 -- 172.21.15.83:0/189722158 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1bc1f8ce0 con 0x7ff1bc0ce770 2024-03-06T07:28:54.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.107+0000 7ff1c0e15640 1 -- 172.21.15.83:0/189722158 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1bc14f830 con 0x7ff1bc0ce770 2024-03-06T07:28:54.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.108+0000 7ff1c0e15640 1 -- 172.21.15.83:0/189722158 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1bc1f8eb0 con 0x7ff1bc0ce770 2024-03-06T07:28:54.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.109+0000 7ff1b2ffd640 1 -- 172.21.15.83:0/189722158 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff1ac007530 con 0x7ff1bc0ce770 2024-03-06T07:28:54.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.109+0000 7ff1b2ffd640 1 --2- 172.21.15.83:0/189722158 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff19c06b6a0 0x7ff19c06db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.110+0000 7ff1b2ffd640 1 -- 172.21.15.83:0/189722158 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff1ac08aff0 con 0x7ff1bc0ce770 2024-03-06T07:28:54.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.110+0000 7ff1b3fff640 1 --2- 172.21.15.83:0/189722158 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff19c06b6a0 0x7ff19c06db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.110+0000 7ff1b3fff640 1 --2- 172.21.15.83:0/189722158 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff19c06b6a0 0x7ff19c06db60 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc0aec70 tx=0x7ff1a4012040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:54.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.118+0000 7ff1b2ffd640 1 -- 172.21.15.83:0/189722158 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff1bc1f8eb0 con 0x7ff1bc0ce770 2024-03-06T07:28:54.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.337+0000 7ff1c0e15640 1 -- 172.21.15.83:0/189722158 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7ff1bc0732b0 con 0x7ff1bc0ce770 2024-03-06T07:28:54.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.972+0000 7ff1b2ffd640 1 -- 172.21.15.83:0/189722158 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v73) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff1ac059820 con 0x7ff1bc0ce770 2024-03-06T07:28:54.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.974+0000 7ff1c0e15640 1 -- 172.21.15.83:0/189722158 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff19c06b6a0 msgr2=0x7ff19c06db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.974+0000 7ff1c0e15640 1 --2- 172.21.15.83:0/189722158 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff19c06b6a0 0x7ff19c06db60 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc0aec70 tx=0x7ff1a4012040 comp rx=0 tx=0).stop 2024-03-06T07:28:54.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.974+0000 7ff1c0e15640 1 -- 172.21.15.83:0/189722158 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1bc0ce770 msgr2=0x7ff1bc1f7f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.974+0000 7ff1c0e15640 1 --2- 172.21.15.83:0/189722158 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1bc0ce770 0x7ff1bc1f7f10 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac00fd10 tx=0x7ff1ac00e720 comp rx=0 tx=0).stop 2024-03-06T07:28:54.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.974+0000 7ff1c0e15640 1 -- 172.21.15.83:0/189722158 shutdown_connections 2024-03-06T07:28:54.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.974+0000 7ff1c0e15640 1 --2- 172.21.15.83:0/189722158 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff19c06b6a0 0x7ff19c06db60 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.974+0000 7ff1c0e15640 1 --2- 172.21.15.83:0/189722158 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff1bc1f8450 0x7ff1bc150ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.974+0000 7ff1c0e15640 1 --2- 172.21.15.83:0/189722158 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1bc0ceb50 0x7ff1bc1507a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.974+0000 7ff1c0e15640 1 --2- 172.21.15.83:0/189722158 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1bc0ce770 0x7ff1bc1f7f10 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.974+0000 7ff1c0e15640 1 -- 172.21.15.83:0/189722158 >> 172.21.15.83:0/189722158 conn(0x7ff1bc06b190 msgr2=0x7ff1bc156ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:54.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.974+0000 7ff1c0e15640 1 -- 172.21.15.83:0/189722158 shutdown_connections 2024-03-06T07:28:54.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:54.974+0000 7ff1c0e15640 1 -- 172.21.15.83:0/189722158 wait complete. 2024-03-06T07:28:54.988 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:28:54.988 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:55.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.089+0000 7f9cad8c3640 1 Processor -- start 2024-03-06T07:28:55.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.089+0000 7f9cad8c3640 1 -- start start 2024-03-06T07:28:55.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.090+0000 7f9cad8c3640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f9ca813dc10 0x7f9ca80b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.090+0000 7f9cad8c3640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ca815f580 con 0x7f9ca815bf30 2024-03-06T07:28:55.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.090+0000 7f9cad8c3640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ca815b450 con 0x7f9ca815ef60 2024-03-06T07:28:55.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.090+0000 7f9cad8c3640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ca80cf3f0 con 0x7f9ca813dc10 2024-03-06T07:28:55.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.090+0000 7f9c9e7fc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f9ca815bf30 0x7f9ca815e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52212/0 (socket says 172.21.15.83:52212) 2024-03-06T07:28:55.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.090+0000 7f9c9e7fc640 1 -- 172.21.15.83:0/38837430 learned_addr learned my addr 172.21.15.83:0/38837430 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:28:55.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.090+0000 7f9ca6ffd640 1 --2- 172.21.15.83:0/38837430 >> v2:172.21.15.183:3300/0 conn(0x7f9ca813dc10 0x7f9ca80b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.090+0000 7f9ca67fc640 1 -- 172.21.15.83:0/38837430 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3271308214 0 0) 0x7f9ca815f580 con 0x7f9ca815bf30 2024-03-06T07:28:55.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.090+0000 7f9ca67fc640 1 -- 172.21.15.83:0/38837430 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c94003520 con 0x7f9ca815bf30 2024-03-06T07:28:55.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.091+0000 7f9ca67fc640 1 -- 172.21.15.83:0/38837430 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2012715741 0 0) 0x7f9c94003520 con 0x7f9ca815bf30 2024-03-06T07:28:55.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.091+0000 7f9ca67fc640 1 -- 172.21.15.83:0/38837430 >> v2:172.21.15.183:3300/0 conn(0x7f9ca813dc10 msgr2=0x7f9ca80b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:55.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.091+0000 7f9ca67fc640 1 --2- 172.21.15.83:0/38837430 >> v2:172.21.15.183:3300/0 conn(0x7f9ca813dc10 0x7f9ca80b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.091+0000 7f9ca67fc640 1 -- 172.21.15.83:0/38837430 >> v1:172.21.15.183:6789/0 conn(0x7f9ca815ef60 legacy=0x7f9ca80708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.091+0000 7f9ca67fc640 1 -- 172.21.15.83:0/38837430 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca815b730 con 0x7f9ca815bf30 2024-03-06T07:28:55.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.091+0000 7f9ca67fc640 1 -- 172.21.15.83:0/38837430 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f9c90001670 con 0x7f9ca815bf30 2024-03-06T07:28:55.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.091+0000 7f9ca67fc640 1 -- 172.21.15.83:0/38837430 >> v1:172.21.15.83:6789/0 conn(0x7f9ca815bf30 legacy=0x7f9ca815e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.091+0000 7f9ca67fc640 1 --2- 172.21.15.83:0/38837430 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c940039b0 0x7f9c94003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.091+0000 7f9ca67fc640 1 --2- 172.21.15.83:0/38837430 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c94004800 0x7f9c94004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.091+0000 7f9ca67fc640 1 --2- 172.21.15.83:0/38837430 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c940051c0 0x7f9c9400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.091+0000 7f9ca67fc640 1 -- 172.21.15.83:0/38837430 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca80cf3f0 con 0x7f9c94004800 2024-03-06T07:28:55.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.091+0000 7f9ca67fc640 1 -- 172.21.15.83:0/38837430 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9400e370 con 0x7f9c940039b0 2024-03-06T07:28:55.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.091+0000 7f9ca67fc640 1 -- 172.21.15.83:0/38837430 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9400e570 con 0x7f9c940051c0 2024-03-06T07:28:55.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.091+0000 7f9c9e7fc640 1 --2- 172.21.15.83:0/38837430 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c94004800 0x7f9c94004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.091+0000 7f9ca6ffd640 1 --2- 172.21.15.83:0/38837430 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c940039b0 0x7f9c94003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.091+0000 7f9ca77fe640 1 --2- 172.21.15.83:0/38837430 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c940051c0 0x7f9c9400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.091+0000 7f9c9e7fc640 1 -- 172.21.15.83:0/38837430 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c940051c0 msgr2=0x7f9c9400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:55.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.091+0000 7f9c9e7fc640 1 --2- 172.21.15.83:0/38837430 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c940051c0 0x7f9c9400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.091+0000 7f9c9e7fc640 1 -- 172.21.15.83:0/38837430 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c940039b0 msgr2=0x7f9c94003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:55.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.091+0000 7f9c9e7fc640 1 --2- 172.21.15.83:0/38837430 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c940039b0 0x7f9c94003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.091+0000 7f9ca77fe640 1 -- 172.21.15.83:0/38837430 reap_dead start 2024-03-06T07:28:55.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.091+0000 7f9c9e7fc640 1 -- 172.21.15.83:0/38837430 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c94003520 con 0x7f9c94004800 2024-03-06T07:28:55.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.092+0000 7f9c9e7fc640 1 --2- 172.21.15.83:0/38837430 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c94004800 0x7f9c94004c80 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f9c9000c240 tx=0x7f9c94005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e35d6fd2093e840f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:55.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.092+0000 7f9ca67fc640 1 -- 172.21.15.83:0/38837430 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c90007cf0 con 0x7f9c94004800 2024-03-06T07:28:55.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.092+0000 7f9ca67fc640 1 -- 172.21.15.83:0/38837430 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c90030070 con 0x7f9c94004800 2024-03-06T07:28:55.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.092+0000 7f9ca67fc640 1 -- 172.21.15.83:0/38837430 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c90005d70 con 0x7f9c94004800 2024-03-06T07:28:55.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.093+0000 7f9cad8c3640 1 -- 172.21.15.83:0/38837430 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c94004800 msgr2=0x7f9c94004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.093+0000 7f9cad8c3640 1 --2- 172.21.15.83:0/38837430 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c94004800 0x7f9c94004c80 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f9c9000c240 tx=0x7f9c94005830 comp rx=0 tx=0).stop 2024-03-06T07:28:55.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.093+0000 7f9cad8c3640 1 -- 172.21.15.83:0/38837430 shutdown_connections 2024-03-06T07:28:55.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.093+0000 7f9cad8c3640 1 --2- 172.21.15.83:0/38837430 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c94004800 0x7f9c94004c80 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.093+0000 7f9cad8c3640 1 -- 172.21.15.83:0/38837430 >> 172.21.15.83:0/38837430 conn(0x7f9ca80cf8a0 msgr2=0x7f9ca80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:55.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.093+0000 7f9cad8c3640 1 -- 172.21.15.83:0/38837430 shutdown_connections 2024-03-06T07:28:55.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.093+0000 7f9cad8c3640 1 -- 172.21.15.83:0/38837430 wait complete. 2024-03-06T07:28:55.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.094+0000 7f9cad8c3640 1 Processor -- start 2024-03-06T07:28:55.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.095+0000 7f9cad8c3640 1 -- start start 2024-03-06T07:28:55.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.096+0000 7f9cad8c3640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c94004800 0x7f9ca8140cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.096+0000 7f9cad8c3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ca81570e0 0x7f9ca8157560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.096+0000 7f9cad8c3640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ca813ecb0 0x7f9ca813f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.096+0000 7f9cad8c3640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca815e850 con 0x7f9ca81570e0 2024-03-06T07:28:55.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.096+0000 7f9cad8c3640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca815b210 con 0x7f9c94004800 2024-03-06T07:28:55.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.096+0000 7f9cad8c3640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca815e9d0 con 0x7f9ca813ecb0 2024-03-06T07:28:55.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.096+0000 7f9c9e7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ca81570e0 0x7f9ca8157560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.096+0000 7f9c9e7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ca81570e0 0x7f9ca8157560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48926/0 (socket says 172.21.15.83:48926) 2024-03-06T07:28:55.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.096+0000 7f9c9e7fc640 1 -- 172.21.15.83:0/3772100709 learned_addr learned my addr 172.21.15.83:0/3772100709 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:55.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.096+0000 7f9ca6ffd640 1 --2- 172.21.15.83:0/3772100709 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c94004800 0x7f9ca8140cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.096+0000 7f9ca77fe640 1 --2- 172.21.15.83:0/3772100709 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ca813ecb0 0x7f9ca813f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.096+0000 7f9c9e7fc640 1 -- 172.21.15.83:0/3772100709 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ca813ecb0 msgr2=0x7f9ca813f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.096+0000 7f9c9e7fc640 1 --2- 172.21.15.83:0/3772100709 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ca813ecb0 0x7f9ca813f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.096+0000 7f9c9e7fc640 1 -- 172.21.15.83:0/3772100709 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c94004800 msgr2=0x7f9ca8140cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.096+0000 7f9c9e7fc640 1 --2- 172.21.15.83:0/3772100709 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c94004800 0x7f9ca8140cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.096+0000 7f9c9e7fc640 1 -- 172.21.15.83:0/3772100709 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca8156040 con 0x7f9ca81570e0 2024-03-06T07:28:55.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.096+0000 7f9ca77fe640 1 --2- 172.21.15.83:0/3772100709 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ca813ecb0 0x7f9ca813f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:55.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.096+0000 7f9ca6ffd640 1 --2- 172.21.15.83:0/3772100709 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c94004800 0x7f9ca8140cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:55.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.096+0000 7f9c9e7fc640 1 --2- 172.21.15.83:0/3772100709 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ca81570e0 0x7f9ca8157560 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f9c9000c240 tx=0x7f9ca815b620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:55.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.097+0000 7f9ca4ff9640 1 -- 172.21.15.83:0/3772100709 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c90046070 con 0x7f9ca81570e0 2024-03-06T07:28:55.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.097+0000 7f9ca4ff9640 1 -- 172.21.15.83:0/3772100709 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c90000ed0 con 0x7f9ca81570e0 2024-03-06T07:28:55.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.097+0000 7f9cad8c3640 1 -- 172.21.15.83:0/3772100709 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ca8156320 con 0x7f9ca81570e0 2024-03-06T07:28:55.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.097+0000 7f9ca4ff9640 1 -- 172.21.15.83:0/3772100709 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c90004e30 con 0x7f9ca81570e0 2024-03-06T07:28:55.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.097+0000 7f9cad8c3640 1 -- 172.21.15.83:0/3772100709 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ca81568e0 con 0x7f9ca81570e0 2024-03-06T07:28:55.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.098+0000 7f9cad8c3640 1 -- 172.21.15.83:0/3772100709 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ca804f330 con 0x7f9ca81570e0 2024-03-06T07:28:55.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.098+0000 7f9ca4ff9640 1 -- 172.21.15.83:0/3772100709 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9c90007070 con 0x7f9ca81570e0 2024-03-06T07:28:55.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.098+0000 7f9ca4ff9640 1 --2- 172.21.15.83:0/3772100709 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9c8806b4d0 0x7f9c8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.099+0000 7f9ca4ff9640 1 -- 172.21.15.83:0/3772100709 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9c900ab410 con 0x7f9ca81570e0 2024-03-06T07:28:55.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.099+0000 7f9ca6ffd640 1 --2- 172.21.15.83:0/3772100709 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9c8806b4d0 0x7f9c8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.099+0000 7f9ca6ffd640 1 --2- 172.21.15.83:0/3772100709 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9c8806b4d0 0x7f9c8806d990 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f9ca80686a0 tx=0x7f9c98008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:55.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.107+0000 7f9ca4ff9640 1 -- 172.21.15.83:0/3772100709 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9c90079c50 con 0x7f9ca81570e0 2024-03-06T07:28:55.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.323+0000 7f9cad8c3640 1 -- 172.21.15.83:0/3772100709 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ca806eb00 con 0x7f9ca81570e0 2024-03-06T07:28:55.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.323+0000 7f9ca4ff9640 1 -- 172.21.15.83:0/3772100709 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f9c9007dcf0 con 0x7f9ca81570e0 2024-03-06T07:28:55.325 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:28:55.325 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":73,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7381,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/3776086145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3776086145},{"type":"v1","addr":"172.21.15.183:6833","nonce":3776086145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":7387,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2457445285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2457445285},{"type":"v1","addr":"172.21.15.183:6835","nonce":2457445285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8628,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:28:54.339089+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:28:55.326 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 73 2024-03-06T07:28:55.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.326+0000 7f9cad8c3640 1 -- 172.21.15.83:0/3772100709 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9c8806b4d0 msgr2=0x7f9c8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.326+0000 7f9cad8c3640 1 --2- 172.21.15.83:0/3772100709 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9c8806b4d0 0x7f9c8806d990 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f9ca80686a0 tx=0x7f9c98008040 comp rx=0 tx=0).stop 2024-03-06T07:28:55.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.326+0000 7f9cad8c3640 1 -- 172.21.15.83:0/3772100709 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ca81570e0 msgr2=0x7f9ca8157560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.326+0000 7f9cad8c3640 1 --2- 172.21.15.83:0/3772100709 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ca81570e0 0x7f9ca8157560 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f9c9000c240 tx=0x7f9ca815b620 comp rx=0 tx=0).stop 2024-03-06T07:28:55.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.326+0000 7f9cad8c3640 1 -- 172.21.15.83:0/3772100709 shutdown_connections 2024-03-06T07:28:55.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.326+0000 7f9cad8c3640 1 --2- 172.21.15.83:0/3772100709 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9c8806b4d0 0x7f9c8806d990 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.326+0000 7f9cad8c3640 1 --2- 172.21.15.83:0/3772100709 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ca813ecb0 0x7f9ca813f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.326+0000 7f9cad8c3640 1 --2- 172.21.15.83:0/3772100709 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ca81570e0 0x7f9ca8157560 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.326+0000 7f9cad8c3640 1 --2- 172.21.15.83:0/3772100709 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c94004800 0x7f9ca8140cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.326+0000 7f9cad8c3640 1 -- 172.21.15.83:0/3772100709 >> 172.21.15.83:0/3772100709 conn(0x7f9ca80cf8a0 msgr2=0x7f9ca815a720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:55.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.326+0000 7f9cad8c3640 1 -- 172.21.15.83:0/3772100709 shutdown_connections 2024-03-06T07:28:55.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.326+0000 7f9cad8c3640 1 -- 172.21.15.83:0/3772100709 wait complete. 2024-03-06T07:28:55.342 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:55.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.442+0000 7f0134f56640 1 Processor -- start 2024-03-06T07:28:55.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.442+0000 7f0134f56640 1 -- start start 2024-03-06T07:28:55.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.443+0000 7f0134f56640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f013015bb20 0x7f01300d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.443+0000 7f0134f56640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0130161980 con 0x7f01300cfff0 2024-03-06T07:28:55.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.443+0000 7f0134f56640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0130161c60 con 0x7f01300cd8c0 2024-03-06T07:28:55.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.443+0000 7f0134f56640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0130161f40 con 0x7f013015bb20 2024-03-06T07:28:55.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.443+0000 7f012e59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f013015bb20 0x7f01300d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.443+0000 7f012e59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f013015bb20 0x7f01300d1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48938/0 (socket says 172.21.15.83:48938) 2024-03-06T07:28:55.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.443+0000 7f0127fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f01300cfff0 0x7f01300ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52216/0 (socket says 172.21.15.83:52216) 2024-03-06T07:28:55.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.443+0000 7f012e59c640 1 -- 172.21.15.83:0/980754165 learned_addr learned my addr 172.21.15.83:0/980754165 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:55.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.443+0000 7f012e59c640 1 -- 172.21.15.83:0/980754165 >> v1:172.21.15.183:6789/0 conn(0x7f01300cd8c0 legacy=0x7f01300cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:55.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.443+0000 7f012e59c640 1 -- 172.21.15.83:0/980754165 >> v1:172.21.15.83:6789/0 conn(0x7f01300cfff0 legacy=0x7f01300ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.443+0000 7f012e59c640 1 -- 172.21.15.83:0/980754165 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0130162220 con 0x7f013015bb20 2024-03-06T07:28:55.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.443+0000 7f012dd9b640 1 -- 172.21.15.83:0/980754165 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2727329508 0 0) 0x7f0130161980 con 0x7f01300cfff0 2024-03-06T07:28:55.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.444+0000 7f012e59c640 1 --2- 172.21.15.83:0/980754165 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f013015bb20 0x7f01300d1df0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f0120009870 tx=0x7f012002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3de3e210008e4b68 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:55.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.444+0000 7f012dd9b640 1 -- 172.21.15.83:0/980754165 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0120034070 con 0x7f013015bb20 2024-03-06T07:28:55.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.444+0000 7f012dd9b640 1 -- 172.21.15.83:0/980754165 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0120030860 con 0x7f013015bb20 2024-03-06T07:28:55.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.444+0000 7f012dd9b640 1 -- 172.21.15.83:0/980754165 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0120030c30 con 0x7f013015bb20 2024-03-06T07:28:55.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.445+0000 7f0134f56640 1 -- 172.21.15.83:0/980754165 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f013015bb20 msgr2=0x7f01300d1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.445+0000 7f0134f56640 1 --2- 172.21.15.83:0/980754165 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f013015bb20 0x7f01300d1df0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f0120009870 tx=0x7f012002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.445+0000 7f0134f56640 1 -- 172.21.15.83:0/980754165 shutdown_connections 2024-03-06T07:28:55.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.445+0000 7f0134f56640 1 --2- 172.21.15.83:0/980754165 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f013015bb20 0x7f01300d1df0 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.445+0000 7f0134f56640 1 -- 172.21.15.83:0/980754165 >> 172.21.15.83:0/980754165 conn(0x7f013009a9f0 msgr2=0x7f013009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:55.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.445+0000 7f0134f56640 1 -- 172.21.15.83:0/980754165 shutdown_connections 2024-03-06T07:28:55.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.445+0000 7f0134f56640 1 -- 172.21.15.83:0/980754165 wait complete. 2024-03-06T07:28:55.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.446+0000 7f0134f56640 1 Processor -- start 2024-03-06T07:28:55.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.446+0000 7f0134f56640 1 -- start start 2024-03-06T07:28:55.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.447+0000 7f0134f56640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01300cd8c0 0x7f013015aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.447+0000 7f0134f56640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f01300cfff0 0x7f01300ca680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.447+0000 7f0134f56640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f013015bb20 0x7f01300cabc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.447+0000 7f0134f56640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01300d2330 con 0x7f01300cd8c0 2024-03-06T07:28:55.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.447+0000 7f0134f56640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01300cf2d0 con 0x7f01300cfff0 2024-03-06T07:28:55.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.447+0000 7f0134f56640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01300d24b0 con 0x7f013015bb20 2024-03-06T07:28:55.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.447+0000 7f012e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01300cd8c0 0x7f013015aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.447+0000 7f012e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01300cd8c0 0x7f013015aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48948/0 (socket says 172.21.15.83:48948) 2024-03-06T07:28:55.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.447+0000 7f012e59c640 1 -- 172.21.15.83:0/2310735849 learned_addr learned my addr 172.21.15.83:0/2310735849 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:55.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.447+0000 7f012e59c640 1 -- 172.21.15.83:0/2310735849 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f013015bb20 msgr2=0x7f01300cabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.447+0000 7f012e59c640 1 --2- 172.21.15.83:0/2310735849 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f013015bb20 0x7f01300cabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.447+0000 7f012e59c640 1 -- 172.21.15.83:0/2310735849 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f01300cfff0 msgr2=0x7f01300ca680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.447+0000 7f012e59c640 1 --2- 172.21.15.83:0/2310735849 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f01300cfff0 0x7f01300ca680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.447+0000 7f012e59c640 1 -- 172.21.15.83:0/2310735849 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01300c63d0 con 0x7f01300cd8c0 2024-03-06T07:28:55.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.447+0000 7f012e59c640 1 --2- 172.21.15.83:0/2310735849 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01300cd8c0 0x7f013015aaf0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f0120000c00 tx=0x7f0120004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:55.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.448+0000 7f0126ffd640 1 -- 172.21.15.83:0/2310735849 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0120048070 con 0x7f01300cd8c0 2024-03-06T07:28:55.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.448+0000 7f0126ffd640 1 -- 172.21.15.83:0/2310735849 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0120030660 con 0x7f01300cd8c0 2024-03-06T07:28:55.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.448+0000 7f0134f56640 1 -- 172.21.15.83:0/2310735849 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01300c66b0 con 0x7f01300cd8c0 2024-03-06T07:28:55.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.448+0000 7f0126ffd640 1 -- 172.21.15.83:0/2310735849 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0120034040 con 0x7f01300cd8c0 2024-03-06T07:28:55.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.448+0000 7f0134f56640 1 -- 172.21.15.83:0/2310735849 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01300c6c70 con 0x7f01300cd8c0 2024-03-06T07:28:55.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.449+0000 7f0134f56640 1 -- 172.21.15.83:0/2310735849 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f013004f330 con 0x7f01300cd8c0 2024-03-06T07:28:55.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.449+0000 7f0126ffd640 1 -- 172.21.15.83:0/2310735849 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f012004b070 con 0x7f01300cd8c0 2024-03-06T07:28:55.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.449+0000 7f0126ffd640 1 --2- 172.21.15.83:0/2310735849 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f011006b4d0 0x7f011006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.450+0000 7f0126ffd640 1 -- 172.21.15.83:0/2310735849 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f01200ac340 con 0x7f01300cd8c0 2024-03-06T07:28:55.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.450+0000 7f0127fff640 1 --2- 172.21.15.83:0/2310735849 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f011006b4d0 0x7f011006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.450+0000 7f0127fff640 1 --2- 172.21.15.83:0/2310735849 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f011006b4d0 0x7f011006d990 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f0118007fd0 tx=0x7f0118007920 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:55.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.458+0000 7f0126ffd640 1 -- 172.21.15.83:0/2310735849 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f012007b2e0 con 0x7f01300cd8c0 2024-03-06T07:28:55.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.634+0000 7f0134f56640 1 -- 172.21.15.83:0/2310735849 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f013015c7f0 con 0x7f01300cd8c0 2024-03-06T07:28:55.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.635+0000 7f0126ffd640 1 -- 172.21.15.83:0/2310735849 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v128) v1 ==== 74+0+16691 (secure 0 0 0) 0x7f012007abe0 con 0x7f01300cd8c0 2024-03-06T07:28:55.638 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:28:55.638 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":128,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:28:53.950043+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:50.508389+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:51.266049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:55.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.638+0000 7f0134f56640 1 -- 172.21.15.83:0/2310735849 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f011006b4d0 msgr2=0x7f011006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.638+0000 7f0134f56640 1 --2- 172.21.15.83:0/2310735849 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f011006b4d0 0x7f011006d990 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f0118007fd0 tx=0x7f0118007920 comp rx=0 tx=0).stop 2024-03-06T07:28:55.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.638+0000 7f0134f56640 1 -- 172.21.15.83:0/2310735849 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01300cd8c0 msgr2=0x7f013015aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.638+0000 7f0134f56640 1 --2- 172.21.15.83:0/2310735849 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01300cd8c0 0x7f013015aaf0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f0120000c00 tx=0x7f0120004730 comp rx=0 tx=0).stop 2024-03-06T07:28:55.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.638+0000 7f0134f56640 1 -- 172.21.15.83:0/2310735849 shutdown_connections 2024-03-06T07:28:55.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.638+0000 7f0134f56640 1 --2- 172.21.15.83:0/2310735849 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f011006b4d0 0x7f011006d990 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.638+0000 7f0134f56640 1 --2- 172.21.15.83:0/2310735849 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f013015bb20 0x7f01300cabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.638+0000 7f0134f56640 1 --2- 172.21.15.83:0/2310735849 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f01300cfff0 0x7f01300ca680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.638+0000 7f0134f56640 1 --2- 172.21.15.83:0/2310735849 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01300cd8c0 0x7f013015aaf0 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.638+0000 7f0134f56640 1 -- 172.21.15.83:0/2310735849 >> 172.21.15.83:0/2310735849 conn(0x7f013009a9f0 msgr2=0x7f013015cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:55.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.638+0000 7f0134f56640 1 -- 172.21.15.83:0/2310735849 shutdown_connections 2024-03-06T07:28:55.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.638+0000 7f0134f56640 1 -- 172.21.15.83:0/2310735849 wait complete. 2024-03-06T07:28:55.652 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:28:55.653 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:28:55.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.754+0000 7f9025cd4640 1 Processor -- start 2024-03-06T07:28:55.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.754+0000 7f9025cd4640 1 -- start start 2024-03-06T07:28:55.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.756+0000 7f9025cd4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f902015bd20 0x7f90200724b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.756+0000 7f9025cd4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f902015f8f0 0x7f90200ce290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.756+0000 7f9025cd4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9020072ad0 con 0x7f902006afb0 2024-03-06T07:28:55.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.756+0000 7f9025cd4640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9020072db0 con 0x7f902015f8f0 2024-03-06T07:28:55.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.756+0000 7f9025cd4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9020073090 con 0x7f902015bd20 2024-03-06T07:28:55.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.756+0000 7f901effd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f902015f8f0 0x7f90200ce290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.756+0000 7f901effd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f902015f8f0 0x7f90200ce290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48960/0 (socket says 172.21.15.83:48960) 2024-03-06T07:28:55.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.756+0000 7f901effd640 1 -- 172.21.15.83:0/3802776104 learned_addr learned my addr 172.21.15.83:0/3802776104 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:55.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.756+0000 7f901f7fe640 1 --2- 172.21.15.83:0/3802776104 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f902015bd20 0x7f90200724b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.756+0000 7f901effd640 1 -- 172.21.15.83:0/3802776104 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f902015bd20 msgr2=0x7f90200724b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:55.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.756+0000 7f901effd640 1 --2- 172.21.15.83:0/3802776104 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f902015bd20 0x7f90200724b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.756+0000 7f901effd640 1 -- 172.21.15.83:0/3802776104 >> v1:172.21.15.183:6789/0 conn(0x7f902006afb0 legacy=0x7f90200ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.756+0000 7f901effd640 1 -- 172.21.15.83:0/3802776104 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9020073370 con 0x7f902015f8f0 2024-03-06T07:28:55.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.757+0000 7f901effd640 1 --2- 172.21.15.83:0/3802776104 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f902015f8f0 0x7f90200ce290 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f9014009870 tx=0x7f901402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e628cb9967fea87 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:55.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.757+0000 7f901e7fc640 1 -- 172.21.15.83:0/3802776104 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9014034070 con 0x7f902015f8f0 2024-03-06T07:28:55.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.757+0000 7f901e7fc640 1 -- 172.21.15.83:0/3802776104 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9014030840 con 0x7f902015f8f0 2024-03-06T07:28:55.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.757+0000 7f901e7fc640 1 -- 172.21.15.83:0/3802776104 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9014030c10 con 0x7f902015f8f0 2024-03-06T07:28:55.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.758+0000 7f9025cd4640 1 -- 172.21.15.83:0/3802776104 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f902015f8f0 msgr2=0x7f90200ce290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.758+0000 7f9025cd4640 1 --2- 172.21.15.83:0/3802776104 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f902015f8f0 0x7f90200ce290 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f9014009870 tx=0x7f901402b980 comp rx=0 tx=0).stop 2024-03-06T07:28:55.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.758+0000 7f9025cd4640 1 -- 172.21.15.83:0/3802776104 shutdown_connections 2024-03-06T07:28:55.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.758+0000 7f9025cd4640 1 --2- 172.21.15.83:0/3802776104 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f902015f8f0 0x7f90200ce290 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.758+0000 7f9025cd4640 1 --2- 172.21.15.83:0/3802776104 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f902015bd20 0x7f90200724b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.758+0000 7f9025cd4640 1 -- 172.21.15.83:0/3802776104 >> 172.21.15.83:0/3802776104 conn(0x7f90200b1d00 msgr2=0x7f90200b2110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:55.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.758+0000 7f9025cd4640 1 -- 172.21.15.83:0/3802776104 shutdown_connections 2024-03-06T07:28:55.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.758+0000 7f9025cd4640 1 -- 172.21.15.83:0/3802776104 wait complete. 2024-03-06T07:28:55.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.759+0000 7f9025cd4640 1 Processor -- start 2024-03-06T07:28:55.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.760+0000 7f9025cd4640 1 -- start start 2024-03-06T07:28:55.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.761+0000 7f9025cd4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f902006afb0 0x7f902009daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.761+0000 7f9025cd4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f902015bd20 0x7f90200c7240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.761+0000 7f9025cd4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f902015f8f0 0x7f902015c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.761+0000 7f9025cd4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f902015c100 con 0x7f902006afb0 2024-03-06T07:28:55.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.761+0000 7f9025cd4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90200672f0 con 0x7f902015f8f0 2024-03-06T07:28:55.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.761+0000 7f9025cd4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90200ce7d0 con 0x7f902015bd20 2024-03-06T07:28:55.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.761+0000 7f901f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f902006afb0 0x7f902009daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.761+0000 7f901f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f902006afb0 0x7f902009daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48966/0 (socket says 172.21.15.83:48966) 2024-03-06T07:28:55.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.761+0000 7f901f7fe640 1 -- 172.21.15.83:0/2595895477 learned_addr learned my addr 172.21.15.83:0/2595895477 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:55.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.761+0000 7f901f7fe640 1 -- 172.21.15.83:0/2595895477 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f902015bd20 msgr2=0x7f90200c7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.761+0000 7f901f7fe640 1 --2- 172.21.15.83:0/2595895477 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f902015bd20 0x7f90200c7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.761+0000 7f901f7fe640 1 -- 172.21.15.83:0/2595895477 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f902015f8f0 msgr2=0x7f902015c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.761+0000 7f901f7fe640 1 --2- 172.21.15.83:0/2595895477 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f902015f8f0 0x7f902015c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.761+0000 7f901f7fe640 1 -- 172.21.15.83:0/2595895477 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f902006d160 con 0x7f902006afb0 2024-03-06T07:28:55.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.761+0000 7f901f7fe640 1 --2- 172.21.15.83:0/2595895477 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f902006afb0 0x7f902009daa0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f901000e9e0 tx=0x7f901000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:55.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.762+0000 7f9007fff640 1 -- 172.21.15.83:0/2595895477 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9010017070 con 0x7f902006afb0 2024-03-06T07:28:55.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.762+0000 7f9007fff640 1 -- 172.21.15.83:0/2595895477 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f901000c9d0 con 0x7f902006afb0 2024-03-06T07:28:55.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.762+0000 7f9025cd4640 1 -- 172.21.15.83:0/2595895477 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90200c31c0 con 0x7f902006afb0 2024-03-06T07:28:55.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.762+0000 7f9007fff640 1 -- 172.21.15.83:0/2595895477 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f901000cdc0 con 0x7f902006afb0 2024-03-06T07:28:55.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.762+0000 7f9025cd4640 1 -- 172.21.15.83:0/2595895477 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f902006d6e0 con 0x7f902006afb0 2024-03-06T07:28:55.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.763+0000 7f9025cd4640 1 -- 172.21.15.83:0/2595895477 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90200aec10 con 0x7f902006afb0 2024-03-06T07:28:55.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.763+0000 7f9007fff640 1 -- 172.21.15.83:0/2595895477 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f901001a070 con 0x7f902006afb0 2024-03-06T07:28:55.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.763+0000 7f9007fff640 1 --2- 172.21.15.83:0/2595895477 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f900c06b500 0x7f900c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.763+0000 7f9007fff640 1 -- 172.21.15.83:0/2595895477 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f901008a1f0 con 0x7f902006afb0 2024-03-06T07:28:55.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.764+0000 7f901effd640 1 --2- 172.21.15.83:0/2595895477 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f900c06b500 0x7f900c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.764+0000 7f901effd640 1 --2- 172.21.15.83:0/2595895477 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f900c06b500 0x7f900c06d9c0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f9014036000 tx=0x7f9014030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:55.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.768+0000 7f9007fff640 1 -- 172.21.15.83:0/2595895477 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f90200aec10 con 0x7f902006afb0 2024-03-06T07:28:55.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.935+0000 7f9025cd4640 1 -- 172.21.15.83:0/2595895477 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f902015c7c0 con 0x7f900c06b500 2024-03-06T07:28:55.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.941+0000 7f9007fff640 1 -- 172.21.15.83:0/2595895477 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+237205 (secure 0 0 0) 0x7f902015c7c0 con 0x7f900c06b500 2024-03-06T07:28:55.947 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:28:55.948 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-06T07:28:55.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.944+0000 7f9025cd4640 1 -- 172.21.15.83:0/2595895477 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f900c06b500 msgr2=0x7f900c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.944+0000 7f9025cd4640 1 --2- 172.21.15.83:0/2595895477 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f900c06b500 0x7f900c06d9c0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f9014036000 tx=0x7f9014030000 comp rx=0 tx=0).stop 2024-03-06T07:28:55.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.944+0000 7f9025cd4640 1 -- 172.21.15.83:0/2595895477 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f902006afb0 msgr2=0x7f902009daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.944+0000 7f9025cd4640 1 --2- 172.21.15.83:0/2595895477 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f902006afb0 0x7f902009daa0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f901000e9e0 tx=0x7f901000eea0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.944+0000 7f9025cd4640 1 -- 172.21.15.83:0/2595895477 shutdown_connections 2024-03-06T07:28:55.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.944+0000 7f9025cd4640 1 --2- 172.21.15.83:0/2595895477 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f900c06b500 0x7f900c06d9c0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.944+0000 7f9025cd4640 1 --2- 172.21.15.83:0/2595895477 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f902015f8f0 0x7f902015c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.944+0000 7f9025cd4640 1 --2- 172.21.15.83:0/2595895477 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f902015bd20 0x7f90200c7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.944+0000 7f9025cd4640 1 --2- 172.21.15.83:0/2595895477 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f902006afb0 0x7f902009daa0 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.944+0000 7f9025cd4640 1 -- 172.21.15.83:0/2595895477 >> 172.21.15.83:0/2595895477 conn(0x7f90200b1d00 msgr2=0x7f90200ae500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:55.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.944+0000 7f9025cd4640 1 -- 172.21.15.83:0/2595895477 shutdown_connections 2024-03-06T07:28:55.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:55.944+0000 7f9025cd4640 1 -- 172.21.15.83:0/2595895477 wait complete. 2024-03-06T07:28:55.958 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":336,"stamp":"2024-03-06T07:28:54.395551+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":329284,"kb_used_data":13256,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754645436,"statfs":{"total":773094113280,"available":772756926464,"internally_reserved":0,"allocated":13574144,"data_stored":6549624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12757,"internal_metadata":323472939},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000729"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":356,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-06T07:28:48.941732+0000","last_change":"2024-03-06T07:21:47.553693+0000","last_active":"2024-03-06T07:28:48.941732+0000","last_peered":"2024-03-06T07:28:48.941732+0000","last_clean":"2024-03-06T07:28:48.941732+0000","last_became_active":"2024-03-06T07:21:47.552664+0000","last_became_peered":"2024-03-06T07:21:47.552664+0000","last_unstale":"2024-03-06T07:28:48.941732+0000","last_undegraded":"2024-03-06T07:28:48.941732+0000","last_fullsized":"2024-03-06T07:28:48.941732+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_clean_scrub_stamp":"2024-03-06T07:21:46.362028+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:02.916426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574938,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41596,"kb_used_data":2092,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330244,"statfs":{"total":96636764160,"available":96594169856,"internally_reserved":0,"allocated":2142208,"data_stored":1261479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":40434121},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574936,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41016,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330824,"statfs":{"total":96636764160,"available":96594763776,"internally_reserved":0,"allocated":1548288,"data_stored":671111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574937,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41016,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330824,"statfs":{"total":96636764160,"available":96594763776,"internally_reserved":0,"allocated":1548288,"data_stored":671111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574938,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41016,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330824,"statfs":{"total":96636764160,"available":96594763776,"internally_reserved":0,"allocated":1548288,"data_stored":671111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574937,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41016,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330824,"statfs":{"total":96636764160,"available":96594763776,"internally_reserved":0,"allocated":1548288,"data_stored":671111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574937,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41016,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330824,"statfs":{"total":96636764160,"available":96594763776,"internally_reserved":0,"allocated":1548288,"data_stored":671111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574938,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41016,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330824,"statfs":{"total":96636764160,"available":96594763776,"internally_reserved":0,"allocated":1548288,"data_stored":671111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574937,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41592,"kb_used_data":2092,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330248,"statfs":{"total":96636764160,"available":96594173952,"internally_reserved":0,"allocated":2142208,"data_stored":1261479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:28:55.962 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:28:56.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.096+0000 7fed7c6ed640 1 Processor -- start 2024-03-06T07:28:56.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.096+0000 7fed7c6ed640 1 -- start start 2024-03-06T07:28:56.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.097+0000 7fed7c6ed640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fed740d22b0 0x7fed740d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.097+0000 7fed7c6ed640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fed740b41f0 0x7fed740b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.097+0000 7fed7c6ed640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed740b3430 con 0x7fed7415dfe0 2024-03-06T07:28:56.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.097+0000 7fed7c6ed640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed740d2c50 con 0x7fed740b41f0 2024-03-06T07:28:56.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.097+0000 7fed7c6ed640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed740b31e0 con 0x7fed740d22b0 2024-03-06T07:28:56.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.097+0000 7fed7ac8a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fed740b41f0 0x7fed740b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.097+0000 7fed7a489640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fed7415dfe0 0x7fed740cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52224/0 (socket says 172.21.15.83:52224) 2024-03-06T07:28:56.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.097+0000 7fed7a489640 1 -- 172.21.15.83:0/1362932804 learned_addr learned my addr 172.21.15.83:0/1362932804 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:28:56.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.098+0000 7fed79c88640 1 --2- 172.21.15.83:0/1362932804 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fed740d22b0 0x7fed740d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.098+0000 7fed7ac8a640 1 -- 172.21.15.83:0/1362932804 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fed740d22b0 msgr2=0x7fed740d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:56.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.098+0000 7fed7ac8a640 1 --2- 172.21.15.83:0/1362932804 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fed740d22b0 0x7fed740d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.098+0000 7fed7ac8a640 1 -- 172.21.15.83:0/1362932804 >> v1:172.21.15.83:6789/0 conn(0x7fed7415dfe0 legacy=0x7fed740cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.098+0000 7fed79487640 1 -- 172.21.15.83:0/1362932804 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 586718997 0 0) 0x7fed740b3430 con 0x7fed7415dfe0 2024-03-06T07:28:56.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.098+0000 7fed7ac8a640 1 -- 172.21.15.83:0/1362932804 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed740b3710 con 0x7fed740b41f0 2024-03-06T07:28:56.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.098+0000 7fed7ac8a640 1 --2- 172.21.15.83:0/1362932804 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed740b41f0 0x7fed740b4650 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fed6c009d90 tx=0x7fed6c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=315d861520a63701 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.099+0000 7fed79487640 1 -- 172.21.15.83:0/1362932804 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6c039070 con 0x7fed740b41f0 2024-03-06T07:28:56.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.099+0000 7fed79487640 1 -- 172.21.15.83:0/1362932804 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed6c02cc50 con 0x7fed740b41f0 2024-03-06T07:28:56.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.099+0000 7fed79487640 1 -- 172.21.15.83:0/1362932804 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6c034440 con 0x7fed740b41f0 2024-03-06T07:28:56.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.099+0000 7fed7c6ed640 1 -- 172.21.15.83:0/1362932804 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed740b41f0 msgr2=0x7fed740b4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.099+0000 7fed7c6ed640 1 --2- 172.21.15.83:0/1362932804 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed740b41f0 0x7fed740b4650 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fed6c009d90 tx=0x7fed6c02bea0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.099+0000 7fed7c6ed640 1 -- 172.21.15.83:0/1362932804 shutdown_connections 2024-03-06T07:28:56.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.099+0000 7fed7c6ed640 1 --2- 172.21.15.83:0/1362932804 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed740b41f0 0x7fed740b4650 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.099+0000 7fed7c6ed640 1 --2- 172.21.15.83:0/1362932804 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fed740d22b0 0x7fed740d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.099+0000 7fed7c6ed640 1 -- 172.21.15.83:0/1362932804 >> 172.21.15.83:0/1362932804 conn(0x7fed740d3cd0 msgr2=0x7fed740d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:56.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.099+0000 7fed7c6ed640 1 -- 172.21.15.83:0/1362932804 shutdown_connections 2024-03-06T07:28:56.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.099+0000 7fed7c6ed640 1 -- 172.21.15.83:0/1362932804 wait complete. 2024-03-06T07:28:56.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.100+0000 7fed7c6ed640 1 Processor -- start 2024-03-06T07:28:56.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.100+0000 7fed7c6ed640 1 -- start start 2024-03-06T07:28:56.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.101+0000 7fed7c6ed640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fed740b41f0 0x7fed740b97b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.101+0000 7fed7c6ed640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed740d22b0 0x7fed741f76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.101+0000 7fed7c6ed640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fed7415dfe0 0x7fed7414a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.101+0000 7fed7c6ed640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed740cfa00 con 0x7fed740d22b0 2024-03-06T07:28:56.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.101+0000 7fed7c6ed640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed740b3d60 con 0x7fed7415dfe0 2024-03-06T07:28:56.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.101+0000 7fed7c6ed640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed740d1c70 con 0x7fed740b41f0 2024-03-06T07:28:56.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.101+0000 7fed79c88640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed740d22b0 0x7fed741f76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.101+0000 7fed79c88640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed740d22b0 0x7fed741f76f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48978/0 (socket says 172.21.15.83:48978) 2024-03-06T07:28:56.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.101+0000 7fed79c88640 1 -- 172.21.15.83:0/4167292628 learned_addr learned my addr 172.21.15.83:0/4167292628 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:56.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.101+0000 7fed7a489640 1 --2- 172.21.15.83:0/4167292628 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fed740b41f0 0x7fed740b97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.101+0000 7fed7ac8a640 1 --2- 172.21.15.83:0/4167292628 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fed7415dfe0 0x7fed7414a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.102+0000 7fed79c88640 1 -- 172.21.15.83:0/4167292628 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fed740b41f0 msgr2=0x7fed740b97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.102+0000 7fed79c88640 1 --2- 172.21.15.83:0/4167292628 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fed740b41f0 0x7fed740b97b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.102+0000 7fed79c88640 1 -- 172.21.15.83:0/4167292628 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fed7415dfe0 msgr2=0x7fed7414a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.102+0000 7fed79c88640 1 --2- 172.21.15.83:0/4167292628 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fed7415dfe0 0x7fed7414a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.102+0000 7fed79c88640 1 -- 172.21.15.83:0/4167292628 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed7414aad0 con 0x7fed740d22b0 2024-03-06T07:28:56.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.102+0000 7fed7a489640 1 --2- 172.21.15.83:0/4167292628 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fed740b41f0 0x7fed740b97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:56.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.102+0000 7fed79c88640 1 --2- 172.21.15.83:0/4167292628 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed740d22b0 0x7fed741f76f0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7fed7000d800 tx=0x7fed7000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.102+0000 7fed6affd640 1 -- 172.21.15.83:0/4167292628 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7000b6c0 con 0x7fed740d22b0 2024-03-06T07:28:56.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.102+0000 7fed6affd640 1 -- 172.21.15.83:0/4167292628 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed70019070 con 0x7fed740d22b0 2024-03-06T07:28:56.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.102+0000 7fed7c6ed640 1 -- 172.21.15.83:0/4167292628 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed7415e4f0 con 0x7fed740d22b0 2024-03-06T07:28:56.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.102+0000 7fed6affd640 1 -- 172.21.15.83:0/4167292628 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7000ba80 con 0x7fed740d22b0 2024-03-06T07:28:56.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.102+0000 7fed7c6ed640 1 -- 172.21.15.83:0/4167292628 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed741f6ef0 con 0x7fed740d22b0 2024-03-06T07:28:56.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.103+0000 7fed7c6ed640 1 -- 172.21.15.83:0/4167292628 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed7404f330 con 0x7fed740d22b0 2024-03-06T07:28:56.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.103+0000 7fed6affd640 1 -- 172.21.15.83:0/4167292628 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fed7000bc20 con 0x7fed740d22b0 2024-03-06T07:28:56.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.104+0000 7fed6affd640 1 --2- 172.21.15.83:0/4167292628 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fed5806b4d0 0x7fed5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.104+0000 7fed7a489640 1 --2- 172.21.15.83:0/4167292628 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fed5806b4d0 0x7fed5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.104+0000 7fed6affd640 1 -- 172.21.15.83:0/4167292628 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fed7008a5d0 con 0x7fed740d22b0 2024-03-06T07:28:56.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.104+0000 7fed7a489640 1 --2- 172.21.15.83:0/4167292628 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fed5806b4d0 0x7fed5806d990 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fed64000f60 tx=0x7fed64007ca0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.108+0000 7fed6affd640 1 -- 172.21.15.83:0/4167292628 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fed7414aad0 con 0x7fed740d22b0 2024-03-06T07:28:56.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.328+0000 7fed7c6ed640 1 -- 172.21.15.83:0/4167292628 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fed740ce840 con 0x7fed5806b4d0 2024-03-06T07:28:56.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.333+0000 7fed6affd640 1 -- 172.21.15.83:0/4167292628 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+237205 (secure 0 0 0) 0x7fed740ce840 con 0x7fed5806b4d0 2024-03-06T07:28:56.338 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:28:56.340 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-06T07:28:56.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.336+0000 7fed7c6ed640 1 -- 172.21.15.83:0/4167292628 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fed5806b4d0 msgr2=0x7fed5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.336+0000 7fed7c6ed640 1 --2- 172.21.15.83:0/4167292628 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fed5806b4d0 0x7fed5806d990 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fed64000f60 tx=0x7fed64007ca0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.336+0000 7fed7c6ed640 1 -- 172.21.15.83:0/4167292628 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed740d22b0 msgr2=0x7fed741f76f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.336+0000 7fed7c6ed640 1 --2- 172.21.15.83:0/4167292628 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed740d22b0 0x7fed741f76f0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7fed7000d800 tx=0x7fed7000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.336+0000 7fed7c6ed640 1 -- 172.21.15.83:0/4167292628 shutdown_connections 2024-03-06T07:28:56.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.336+0000 7fed7c6ed640 1 --2- 172.21.15.83:0/4167292628 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fed5806b4d0 0x7fed5806d990 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.336+0000 7fed7c6ed640 1 --2- 172.21.15.83:0/4167292628 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fed7415dfe0 0x7fed7414a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.336+0000 7fed7c6ed640 1 --2- 172.21.15.83:0/4167292628 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed740d22b0 0x7fed741f76f0 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.336+0000 7fed7c6ed640 1 --2- 172.21.15.83:0/4167292628 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fed740b41f0 0x7fed740b97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.336+0000 7fed7c6ed640 1 -- 172.21.15.83:0/4167292628 >> 172.21.15.83:0/4167292628 conn(0x7fed740d3cd0 msgr2=0x7fed7415d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:56.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.336+0000 7fed7c6ed640 1 -- 172.21.15.83:0/4167292628 shutdown_connections 2024-03-06T07:28:56.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.336+0000 7fed7c6ed640 1 -- 172.21.15.83:0/4167292628 wait complete. 2024-03-06T07:28:56.350 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":336,"stamp":"2024-03-06T07:28:54.395551+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":329284,"kb_used_data":13256,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754645436,"statfs":{"total":773094113280,"available":772756926464,"internally_reserved":0,"allocated":13574144,"data_stored":6549624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12757,"internal_metadata":323472939},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000729"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:51.939703+0000","last_change":"2024-03-06T07:28:51.939703+0000","last_active":"2024-03-06T07:28:51.939703+0000","last_peered":"2024-03-06T07:28:51.939703+0000","last_clean":"2024-03-06T07:28:51.939703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:51.939703+0000","last_undegraded":"2024-03-06T07:28:51.939703+0000","last_fullsized":"2024-03-06T07:28:51.939703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:50.932785+0000","last_change":"2024-03-06T07:28:50.932785+0000","last_active":"2024-03-06T07:28:50.932785+0000","last_peered":"2024-03-06T07:28:50.932785+0000","last_clean":"2024-03-06T07:28:50.932785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:50.932785+0000","last_undegraded":"2024-03-06T07:28:50.932785+0000","last_fullsized":"2024-03-06T07:28:50.932785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":356,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-06T07:28:48.941732+0000","last_change":"2024-03-06T07:21:47.553693+0000","last_active":"2024-03-06T07:28:48.941732+0000","last_peered":"2024-03-06T07:28:48.941732+0000","last_clean":"2024-03-06T07:28:48.941732+0000","last_became_active":"2024-03-06T07:21:47.552664+0000","last_became_peered":"2024-03-06T07:21:47.552664+0000","last_unstale":"2024-03-06T07:28:48.941732+0000","last_undegraded":"2024-03-06T07:28:48.941732+0000","last_fullsized":"2024-03-06T07:28:48.941732+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_clean_scrub_stamp":"2024-03-06T07:21:46.362028+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:02.916426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574938,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41596,"kb_used_data":2092,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330244,"statfs":{"total":96636764160,"available":96594169856,"internally_reserved":0,"allocated":2142208,"data_stored":1261479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":40434121},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574936,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41016,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330824,"statfs":{"total":96636764160,"available":96594763776,"internally_reserved":0,"allocated":1548288,"data_stored":671111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574937,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41016,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330824,"statfs":{"total":96636764160,"available":96594763776,"internally_reserved":0,"allocated":1548288,"data_stored":671111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574938,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41016,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330824,"statfs":{"total":96636764160,"available":96594763776,"internally_reserved":0,"allocated":1548288,"data_stored":671111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574937,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41016,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330824,"statfs":{"total":96636764160,"available":96594763776,"internally_reserved":0,"allocated":1548288,"data_stored":671111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574937,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41016,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330824,"statfs":{"total":96636764160,"available":96594763776,"internally_reserved":0,"allocated":1548288,"data_stored":671111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574938,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41016,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330824,"statfs":{"total":96636764160,"available":96594763776,"internally_reserved":0,"allocated":1548288,"data_stored":671111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574937,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41592,"kb_used_data":2092,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330248,"statfs":{"total":96636764160,"available":96594173952,"internally_reserved":0,"allocated":2142208,"data_stored":1261479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:28:56.356 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:28:56.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.492+0000 7f4b8f488640 1 Processor -- start 2024-03-06T07:28:56.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.493+0000 7f4b8f488640 1 -- start start 2024-03-06T07:28:56.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.493+0000 7f4b8f488640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f4b8815dfe0 0x7f4b880cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.493+0000 7f4b8f488640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4b880cf830 0x7f4b880d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.493+0000 7f4b8f488640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b880b3430 con 0x7f4b880d27f0 2024-03-06T07:28:56.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.493+0000 7f4b8f488640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b880d2c50 con 0x7f4b8815dfe0 2024-03-06T07:28:56.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.493+0000 7f4b8f488640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b880b31e0 con 0x7f4b880cf830 2024-03-06T07:28:56.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.494+0000 7f4b8da25640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f4b880d27f0 0x7f4b880d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52236/0 (socket says 172.21.15.83:52236) 2024-03-06T07:28:56.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.494+0000 7f4b8da25640 1 -- 172.21.15.83:0/668367625 learned_addr learned my addr 172.21.15.83:0/668367625 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:28:56.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.494+0000 7f4b8ca23640 1 --2- 172.21.15.83:0/668367625 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4b880cf830 0x7f4b880d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.494+0000 7f4b7ffff640 1 -- 172.21.15.83:0/668367625 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 552395879 0 0) 0x7f4b880b3430 con 0x7f4b880d27f0 2024-03-06T07:28:56.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.494+0000 7f4b7ffff640 1 -- 172.21.15.83:0/668367625 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b74003520 con 0x7f4b880d27f0 2024-03-06T07:28:56.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.494+0000 7f4b8d224640 1 --2- 172.21.15.83:0/668367625 >> v2:172.21.15.183:3300/0 conn(0x7f4b8815dfe0 0x7f4b880cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.494+0000 7f4b7ffff640 1 -- 172.21.15.83:0/668367625 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1447275476 0 0) 0x7f4b74003520 con 0x7f4b880d27f0 2024-03-06T07:28:56.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.494+0000 7f4b7ffff640 1 -- 172.21.15.83:0/668367625 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4b880cf830 msgr2=0x7f4b880d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:56.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.494+0000 7f4b7ffff640 1 --2- 172.21.15.83:0/668367625 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4b880cf830 0x7f4b880d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.494+0000 7f4b7ffff640 1 -- 172.21.15.83:0/668367625 >> v2:172.21.15.183:3300/0 conn(0x7f4b8815dfe0 msgr2=0x7f4b880cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:56.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.494+0000 7f4b7ffff640 1 --2- 172.21.15.83:0/668367625 >> v2:172.21.15.183:3300/0 conn(0x7f4b8815dfe0 0x7f4b880cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.494+0000 7f4b7ffff640 1 -- 172.21.15.83:0/668367625 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b880b3710 con 0x7f4b880d27f0 2024-03-06T07:28:56.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.494+0000 7f4b7ffff640 1 -- 172.21.15.83:0/668367625 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f4b800019f0 con 0x7f4b880d27f0 2024-03-06T07:28:56.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.494+0000 7f4b7ffff640 1 -- 172.21.15.83:0/668367625 >> v1:172.21.15.83:6789/0 conn(0x7f4b880d27f0 legacy=0x7f4b880d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.494+0000 7f4b7ffff640 1 --2- 172.21.15.83:0/668367625 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4b880cf830 0x7f4b740039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.494+0000 7f4b7ffff640 1 --2- 172.21.15.83:0/668367625 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b74003f30 0x7f4b74004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.494+0000 7f4b7ffff640 1 --2- 172.21.15.83:0/668367625 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4b74004d40 0x7f4b740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.494+0000 7f4b7ffff640 1 -- 172.21.15.83:0/668367625 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b880b31e0 con 0x7f4b74003f30 2024-03-06T07:28:56.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.494+0000 7f4b7ffff640 1 -- 172.21.15.83:0/668367625 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7400b9c0 con 0x7f4b74004d40 2024-03-06T07:28:56.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.494+0000 7f4b7ffff640 1 -- 172.21.15.83:0/668367625 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7400bb40 con 0x7f4b880cf830 2024-03-06T07:28:56.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.494+0000 7f4b8ca23640 1 --2- 172.21.15.83:0/668367625 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b74003f30 0x7f4b74004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.494+0000 7f4b8d224640 1 --2- 172.21.15.83:0/668367625 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4b880cf830 0x7f4b740039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.494+0000 7f4b8da25640 1 --2- 172.21.15.83:0/668367625 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4b74004d40 0x7f4b740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.495+0000 7f4b8ca23640 1 -- 172.21.15.83:0/668367625 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4b880cf830 msgr2=0x7f4b740039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:56.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.495+0000 7f4b8ca23640 1 --2- 172.21.15.83:0/668367625 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4b880cf830 0x7f4b740039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.495+0000 7f4b8ca23640 1 -- 172.21.15.83:0/668367625 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4b74004d40 msgr2=0x7f4b740051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:56.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.495+0000 7f4b8ca23640 1 --2- 172.21.15.83:0/668367625 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4b74004d40 0x7f4b740051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.495+0000 7f4b8ca23640 1 -- 172.21.15.83:0/668367625 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b74003520 con 0x7f4b74003f30 2024-03-06T07:28:56.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.495+0000 7f4b8da25640 1 --2- 172.21.15.83:0/668367625 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4b74004d40 0x7f4b740051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:56.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.495+0000 7f4b8ca23640 1 --2- 172.21.15.83:0/668367625 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b74003f30 0x7f4b74004800 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f4b78009a90 tx=0x7f4b74005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aae4f60690172ddd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.495+0000 7f4b7ffff640 1 -- 172.21.15.83:0/668367625 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b78039070 con 0x7f4b74003f30 2024-03-06T07:28:56.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.495+0000 7f4b7ffff640 1 -- 172.21.15.83:0/668367625 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b7802c6b0 con 0x7f4b74003f30 2024-03-06T07:28:56.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.495+0000 7f4b7ffff640 1 -- 172.21.15.83:0/668367625 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b78009690 con 0x7f4b74003f30 2024-03-06T07:28:56.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.496+0000 7f4b8f488640 1 -- 172.21.15.83:0/668367625 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b74003f30 msgr2=0x7f4b74004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.496+0000 7f4b8f488640 1 --2- 172.21.15.83:0/668367625 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b74003f30 0x7f4b74004800 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f4b78009a90 tx=0x7f4b74005830 comp rx=0 tx=0).stop 2024-03-06T07:28:56.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.496+0000 7f4b8da25640 1 -- 172.21.15.83:0/668367625 reap_dead start 2024-03-06T07:28:56.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.496+0000 7f4b8f488640 1 -- 172.21.15.83:0/668367625 shutdown_connections 2024-03-06T07:28:56.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.496+0000 7f4b8f488640 1 -- 172.21.15.83:0/668367625 >> 172.21.15.83:0/668367625 conn(0x7f4b880d3cd0 msgr2=0x7f4b880d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:56.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.496+0000 7f4b8f488640 1 -- 172.21.15.83:0/668367625 shutdown_connections 2024-03-06T07:28:56.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.496+0000 7f4b8f488640 1 -- 172.21.15.83:0/668367625 wait complete. 2024-03-06T07:28:56.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.497+0000 7f4b8f488640 1 Processor -- start 2024-03-06T07:28:56.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.497+0000 7f4b8f488640 1 -- start start 2024-03-06T07:28:56.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.498+0000 7f4b8f488640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b74003f30 0x7f4b8815c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.498+0000 7f4b8f488640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4b880a0d00 0x7f4b881f39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.498+0000 7f4b8f488640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4b880a1180 0x7f4b881f3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.498+0000 7f4b8f488640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b880d1c70 con 0x7f4b74003f30 2024-03-06T07:28:56.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.498+0000 7f4b8f488640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b880b3d60 con 0x7f4b880a0d00 2024-03-06T07:28:56.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.498+0000 7f4b8f488640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b880d1df0 con 0x7f4b880a1180 2024-03-06T07:28:56.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.498+0000 7f4b8d224640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b74003f30 0x7f4b8815c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.498+0000 7f4b8d224640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b74003f30 0x7f4b8815c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:48994/0 (socket says 172.21.15.83:48994) 2024-03-06T07:28:56.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.498+0000 7f4b8d224640 1 -- 172.21.15.83:0/2933544653 learned_addr learned my addr 172.21.15.83:0/2933544653 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:56.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.498+0000 7f4b8d224640 1 -- 172.21.15.83:0/2933544653 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4b880a1180 msgr2=0x7f4b881f3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.498+0000 7f4b8d224640 1 --2- 172.21.15.83:0/2933544653 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4b880a1180 0x7f4b881f3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.498+0000 7f4b8d224640 1 -- 172.21.15.83:0/2933544653 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4b880a0d00 msgr2=0x7f4b881f39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.498+0000 7f4b8d224640 1 --2- 172.21.15.83:0/2933544653 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4b880a0d00 0x7f4b881f39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.498+0000 7f4b8d224640 1 -- 172.21.15.83:0/2933544653 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b8809def0 con 0x7f4b74003f30 2024-03-06T07:28:56.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.498+0000 7f4b8d224640 1 --2- 172.21.15.83:0/2933544653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b74003f30 0x7f4b8815c1d0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f4b84011a20 tx=0x7f4b84011ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.499+0000 7f4b7dffb640 1 -- 172.21.15.83:0/2933544653 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b8400fbb0 con 0x7f4b74003f30 2024-03-06T07:28:56.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.499+0000 7f4b7dffb640 1 -- 172.21.15.83:0/2933544653 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b8400fd50 con 0x7f4b74003f30 2024-03-06T07:28:56.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.499+0000 7f4b8f488640 1 -- 172.21.15.83:0/2933544653 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b8809e1d0 con 0x7f4b74003f30 2024-03-06T07:28:56.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.499+0000 7f4b7dffb640 1 -- 172.21.15.83:0/2933544653 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b84014660 con 0x7f4b74003f30 2024-03-06T07:28:56.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.499+0000 7f4b8f488640 1 -- 172.21.15.83:0/2933544653 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b8809e790 con 0x7f4b74003f30 2024-03-06T07:28:56.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.500+0000 7f4b8f488640 1 -- 172.21.15.83:0/2933544653 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b880b3460 con 0x7f4b74003f30 2024-03-06T07:28:56.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.501+0000 7f4b7dffb640 1 -- 172.21.15.83:0/2933544653 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4b84012040 con 0x7f4b74003f30 2024-03-06T07:28:56.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.501+0000 7f4b7dffb640 1 --2- 172.21.15.83:0/2933544653 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4b6c06b4d0 0x7f4b6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.501+0000 7f4b7dffb640 1 -- 172.21.15.83:0/2933544653 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4b8408a860 con 0x7f4b74003f30 2024-03-06T07:28:56.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.501+0000 7f4b8ca23640 1 --2- 172.21.15.83:0/2933544653 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4b6c06b4d0 0x7f4b6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.502+0000 7f4b8ca23640 1 --2- 172.21.15.83:0/2933544653 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4b6c06b4d0 0x7f4b6c06d990 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f4b78036000 tx=0x7f4b7802e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.506+0000 7f4b7dffb640 1 -- 172.21.15.83:0/2933544653 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4b880b3460 con 0x7f4b74003f30 2024-03-06T07:28:56.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.680+0000 7f4b8f488640 1 -- 172.21.15.83:0/2933544653 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4b880ce9a0 con 0x7f4b6c06b4d0 2024-03-06T07:28:56.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.685+0000 7f4b7dffb640 1 -- 172.21.15.83:0/2933544653 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252483 (secure 0 0 0) 0x7f4b880ce9a0 con 0x7f4b6c06b4d0 2024-03-06T07:28:56.691 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:28:56.692 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-06T07:28:56.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.689+0000 7f4b8f488640 1 -- 172.21.15.83:0/2933544653 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4b6c06b4d0 msgr2=0x7f4b6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.689+0000 7f4b8f488640 1 --2- 172.21.15.83:0/2933544653 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4b6c06b4d0 0x7f4b6c06d990 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f4b78036000 tx=0x7f4b7802e000 comp rx=0 tx=0).stop 2024-03-06T07:28:56.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.689+0000 7f4b8f488640 1 -- 172.21.15.83:0/2933544653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b74003f30 msgr2=0x7f4b8815c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.689+0000 7f4b8f488640 1 --2- 172.21.15.83:0/2933544653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b74003f30 0x7f4b8815c1d0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f4b84011a20 tx=0x7f4b84011ee0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.689+0000 7f4b8f488640 1 -- 172.21.15.83:0/2933544653 shutdown_connections 2024-03-06T07:28:56.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.689+0000 7f4b8f488640 1 --2- 172.21.15.83:0/2933544653 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4b6c06b4d0 0x7f4b6c06d990 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.689+0000 7f4b8f488640 1 --2- 172.21.15.83:0/2933544653 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4b880a1180 0x7f4b881f3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.689+0000 7f4b8f488640 1 --2- 172.21.15.83:0/2933544653 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4b880a0d00 0x7f4b881f39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.689+0000 7f4b8f488640 1 --2- 172.21.15.83:0/2933544653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b74003f30 0x7f4b8815c1d0 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.689+0000 7f4b8f488640 1 -- 172.21.15.83:0/2933544653 >> 172.21.15.83:0/2933544653 conn(0x7f4b880d3cd0 msgr2=0x7f4b88160460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:56.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.689+0000 7f4b8f488640 1 -- 172.21.15.83:0/2933544653 shutdown_connections 2024-03-06T07:28:56.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.689+0000 7f4b8f488640 1 -- 172.21.15.83:0/2933544653 wait complete. 2024-03-06T07:28:56.702 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":337,"stamp":"2024-03-06T07:28:56.395815+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":329716,"kb_used_data":13704,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754645004,"statfs":{"total":773094113280,"available":772756484096,"internally_reserved":0,"allocated":14032896,"data_stored":6755780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12757,"internal_metadata":323472939},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.456200"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144920+0000","last_change":"2024-03-06T07:28:52.975574+0000","last_active":"2024-03-06T07:28:54.144920+0000","last_peered":"2024-03-06T07:28:54.144920+0000","last_clean":"2024-03-06T07:28:54.144920+0000","last_became_active":"2024-03-06T07:28:52.975235+0000","last_became_peered":"2024-03-06T07:28:52.975235+0000","last_unstale":"2024-03-06T07:28:54.144920+0000","last_undegraded":"2024-03-06T07:28:54.144920+0000","last_fullsized":"2024-03-06T07:28:54.144920+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:05:49.993454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.954337+0000","last_change":"2024-03-06T07:28:52.957309+0000","last_active":"2024-03-06T07:28:53.954337+0000","last_peered":"2024-03-06T07:28:53.954337+0000","last_clean":"2024-03-06T07:28:53.954337+0000","last_became_active":"2024-03-06T07:28:52.957035+0000","last_became_peered":"2024-03-06T07:28:52.957035+0000","last_unstale":"2024-03-06T07:28:53.954337+0000","last_undegraded":"2024-03-06T07:28:53.954337+0000","last_fullsized":"2024-03-06T07:28:53.954337+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:07.633879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.959053+0000","last_change":"2024-03-06T07:28:52.962280+0000","last_active":"2024-03-06T07:28:53.959053+0000","last_peered":"2024-03-06T07:28:53.959053+0000","last_clean":"2024-03-06T07:28:53.959053+0000","last_became_active":"2024-03-06T07:28:52.961976+0000","last_became_peered":"2024-03-06T07:28:52.961976+0000","last_unstale":"2024-03-06T07:28:53.959053+0000","last_undegraded":"2024-03-06T07:28:53.959053+0000","last_fullsized":"2024-03-06T07:28:53.959053+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:26.937939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144687+0000","last_change":"2024-03-06T07:28:52.975310+0000","last_active":"2024-03-06T07:28:54.144687+0000","last_peered":"2024-03-06T07:28:54.144687+0000","last_clean":"2024-03-06T07:28:54.144687+0000","last_became_active":"2024-03-06T07:28:52.975049+0000","last_became_peered":"2024-03-06T07:28:52.975049+0000","last_unstale":"2024-03-06T07:28:54.144687+0000","last_undegraded":"2024-03-06T07:28:54.144687+0000","last_fullsized":"2024-03-06T07:28:54.144687+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:19.865295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143537+0000","last_change":"2024-03-06T07:28:52.971518+0000","last_active":"2024-03-06T07:28:54.143537+0000","last_peered":"2024-03-06T07:28:54.143537+0000","last_clean":"2024-03-06T07:28:54.143537+0000","last_became_active":"2024-03-06T07:28:52.971206+0000","last_became_peered":"2024-03-06T07:28:52.971206+0000","last_unstale":"2024-03-06T07:28:54.143537+0000","last_undegraded":"2024-03-06T07:28:54.143537+0000","last_fullsized":"2024-03-06T07:28:54.143537+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:42:48.555932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.959724+0000","last_change":"2024-03-06T07:28:52.968239+0000","last_active":"2024-03-06T07:28:53.959724+0000","last_peered":"2024-03-06T07:28:53.959724+0000","last_clean":"2024-03-06T07:28:53.959724+0000","last_became_active":"2024-03-06T07:28:52.967906+0000","last_became_peered":"2024-03-06T07:28:52.967906+0000","last_unstale":"2024-03-06T07:28:53.959724+0000","last_undegraded":"2024-03-06T07:28:53.959724+0000","last_fullsized":"2024-03-06T07:28:53.959724+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:26.937939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.959459+0000","last_change":"2024-03-06T07:28:52.961471+0000","last_active":"2024-03-06T07:28:53.959459+0000","last_peered":"2024-03-06T07:28:53.959459+0000","last_clean":"2024-03-06T07:28:53.959459+0000","last_became_active":"2024-03-06T07:28:52.961189+0000","last_became_peered":"2024-03-06T07:28:52.961189+0000","last_unstale":"2024-03-06T07:28:53.959459+0000","last_undegraded":"2024-03-06T07:28:53.959459+0000","last_fullsized":"2024-03-06T07:28:53.959459+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:38:01.777138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.958700+0000","last_change":"2024-03-06T07:28:52.967625+0000","last_active":"2024-03-06T07:28:53.958700+0000","last_peered":"2024-03-06T07:28:53.958700+0000","last_clean":"2024-03-06T07:28:53.958700+0000","last_became_active":"2024-03-06T07:28:52.966298+0000","last_became_peered":"2024-03-06T07:28:52.966298+0000","last_unstale":"2024-03-06T07:28:53.958700+0000","last_undegraded":"2024-03-06T07:28:53.958700+0000","last_fullsized":"2024-03-06T07:28:53.958700+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:22:48.600473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.958933+0000","last_change":"2024-03-06T07:28:52.971582+0000","last_active":"2024-03-06T07:28:53.958933+0000","last_peered":"2024-03-06T07:28:53.958933+0000","last_clean":"2024-03-06T07:28:53.958933+0000","last_became_active":"2024-03-06T07:28:52.971007+0000","last_became_peered":"2024-03-06T07:28:52.971007+0000","last_unstale":"2024-03-06T07:28:53.958933+0000","last_undegraded":"2024-03-06T07:28:53.958933+0000","last_fullsized":"2024-03-06T07:28:53.958933+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:59.012327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.954355+0000","last_change":"2024-03-06T07:28:52.959897+0000","last_active":"2024-03-06T07:28:53.954355+0000","last_peered":"2024-03-06T07:28:53.954355+0000","last_clean":"2024-03-06T07:28:53.954355+0000","last_became_active":"2024-03-06T07:28:52.959650+0000","last_became_peered":"2024-03-06T07:28:52.959650+0000","last_unstale":"2024-03-06T07:28:53.954355+0000","last_undegraded":"2024-03-06T07:28:53.954355+0000","last_fullsized":"2024-03-06T07:28:53.954355+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:11.992255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143285+0000","last_change":"2024-03-06T07:28:52.969085+0000","last_active":"2024-03-06T07:28:54.143285+0000","last_peered":"2024-03-06T07:28:54.143285+0000","last_clean":"2024-03-06T07:28:54.143285+0000","last_became_active":"2024-03-06T07:28:52.968764+0000","last_became_peered":"2024-03-06T07:28:52.968764+0000","last_unstale":"2024-03-06T07:28:54.143285+0000","last_undegraded":"2024-03-06T07:28:54.143285+0000","last_fullsized":"2024-03-06T07:28:54.143285+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:03.772295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.958428+0000","last_change":"2024-03-06T07:28:52.971904+0000","last_active":"2024-03-06T07:28:53.958428+0000","last_peered":"2024-03-06T07:28:53.958428+0000","last_clean":"2024-03-06T07:28:53.958428+0000","last_became_active":"2024-03-06T07:28:52.971206+0000","last_became_peered":"2024-03-06T07:28:52.971206+0000","last_unstale":"2024-03-06T07:28:53.958428+0000","last_undegraded":"2024-03-06T07:28:53.958428+0000","last_fullsized":"2024-03-06T07:28:53.958428+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:34.971658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144568+0000","last_change":"2024-03-06T07:28:52.966612+0000","last_active":"2024-03-06T07:28:54.144568+0000","last_peered":"2024-03-06T07:28:54.144568+0000","last_clean":"2024-03-06T07:28:54.144568+0000","last_became_active":"2024-03-06T07:28:52.966335+0000","last_became_peered":"2024-03-06T07:28:52.966335+0000","last_unstale":"2024-03-06T07:28:54.144568+0000","last_undegraded":"2024-03-06T07:28:54.144568+0000","last_fullsized":"2024-03-06T07:28:54.144568+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:05.116368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.954143+0000","last_change":"2024-03-06T07:28:52.973834+0000","last_active":"2024-03-06T07:28:53.954143+0000","last_peered":"2024-03-06T07:28:53.954143+0000","last_clean":"2024-03-06T07:28:53.954143+0000","last_became_active":"2024-03-06T07:28:52.973562+0000","last_became_peered":"2024-03-06T07:28:52.973562+0000","last_unstale":"2024-03-06T07:28:53.954143+0000","last_undegraded":"2024-03-06T07:28:53.954143+0000","last_fullsized":"2024-03-06T07:28:53.954143+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:05:49.993454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144309+0000","last_change":"2024-03-06T07:28:52.972039+0000","last_active":"2024-03-06T07:28:54.144309+0000","last_peered":"2024-03-06T07:28:54.144309+0000","last_clean":"2024-03-06T07:28:54.144309+0000","last_became_active":"2024-03-06T07:28:52.971242+0000","last_became_peered":"2024-03-06T07:28:52.971242+0000","last_unstale":"2024-03-06T07:28:54.144309+0000","last_undegraded":"2024-03-06T07:28:54.144309+0000","last_fullsized":"2024-03-06T07:28:54.144309+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:08:59.523654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144014+0000","last_change":"2024-03-06T07:28:52.964715+0000","last_active":"2024-03-06T07:28:54.144014+0000","last_peered":"2024-03-06T07:28:54.144014+0000","last_clean":"2024-03-06T07:28:54.144014+0000","last_became_active":"2024-03-06T07:28:52.964122+0000","last_became_peered":"2024-03-06T07:28:52.964122+0000","last_unstale":"2024-03-06T07:28:54.144014+0000","last_undegraded":"2024-03-06T07:28:54.144014+0000","last_fullsized":"2024-03-06T07:28:54.144014+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:28:34.045789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.959741+0000","last_change":"2024-03-06T07:28:52.960814+0000","last_active":"2024-03-06T07:28:53.959741+0000","last_peered":"2024-03-06T07:28:53.959741+0000","last_clean":"2024-03-06T07:28:53.959741+0000","last_became_active":"2024-03-06T07:28:52.960503+0000","last_became_peered":"2024-03-06T07:28:52.960503+0000","last_unstale":"2024-03-06T07:28:53.959741+0000","last_undegraded":"2024-03-06T07:28:53.959741+0000","last_fullsized":"2024-03-06T07:28:53.959741+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:57.517691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143657+0000","last_change":"2024-03-06T07:28:52.967540+0000","last_active":"2024-03-06T07:28:54.143657+0000","last_peered":"2024-03-06T07:28:54.143657+0000","last_clean":"2024-03-06T07:28:54.143657+0000","last_became_active":"2024-03-06T07:28:52.967117+0000","last_became_peered":"2024-03-06T07:28:52.967117+0000","last_unstale":"2024-03-06T07:28:54.143657+0000","last_undegraded":"2024-03-06T07:28:54.143657+0000","last_fullsized":"2024-03-06T07:28:54.143657+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:21.003630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143443+0000","last_change":"2024-03-06T07:28:52.964819+0000","last_active":"2024-03-06T07:28:54.143443+0000","last_peered":"2024-03-06T07:28:54.143443+0000","last_clean":"2024-03-06T07:28:54.143443+0000","last_became_active":"2024-03-06T07:28:52.964364+0000","last_became_peered":"2024-03-06T07:28:52.964364+0000","last_unstale":"2024-03-06T07:28:54.143443+0000","last_undegraded":"2024-03-06T07:28:54.143443+0000","last_fullsized":"2024-03-06T07:28:54.143443+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:39:07.988272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144602+0000","last_change":"2024-03-06T07:28:52.964742+0000","last_active":"2024-03-06T07:28:54.144602+0000","last_peered":"2024-03-06T07:28:54.144602+0000","last_clean":"2024-03-06T07:28:54.144602+0000","last_became_active":"2024-03-06T07:28:52.964468+0000","last_became_peered":"2024-03-06T07:28:52.964468+0000","last_unstale":"2024-03-06T07:28:54.144602+0000","last_undegraded":"2024-03-06T07:28:54.144602+0000","last_fullsized":"2024-03-06T07:28:54.144602+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:56:12.710010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143383+0000","last_change":"2024-03-06T07:28:52.970726+0000","last_active":"2024-03-06T07:28:54.143383+0000","last_peered":"2024-03-06T07:28:54.143383+0000","last_clean":"2024-03-06T07:28:54.143383+0000","last_became_active":"2024-03-06T07:28:52.970460+0000","last_became_peered":"2024-03-06T07:28:52.970460+0000","last_unstale":"2024-03-06T07:28:54.143383+0000","last_undegraded":"2024-03-06T07:28:54.143383+0000","last_fullsized":"2024-03-06T07:28:54.143383+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:06:57.274089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.142927+0000","last_change":"2024-03-06T07:28:52.959645+0000","last_active":"2024-03-06T07:28:54.142927+0000","last_peered":"2024-03-06T07:28:54.142927+0000","last_clean":"2024-03-06T07:28:54.142927+0000","last_became_active":"2024-03-06T07:28:52.959385+0000","last_became_peered":"2024-03-06T07:28:52.959385+0000","last_unstale":"2024-03-06T07:28:54.142927+0000","last_undegraded":"2024-03-06T07:28:54.142927+0000","last_fullsized":"2024-03-06T07:28:54.142927+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:46:55.628556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.958843+0000","last_change":"2024-03-06T07:28:52.965547+0000","last_active":"2024-03-06T07:28:53.958843+0000","last_peered":"2024-03-06T07:28:53.958843+0000","last_clean":"2024-03-06T07:28:53.958843+0000","last_became_active":"2024-03-06T07:28:52.964474+0000","last_became_peered":"2024-03-06T07:28:52.964474+0000","last_unstale":"2024-03-06T07:28:53.958843+0000","last_undegraded":"2024-03-06T07:28:53.958843+0000","last_fullsized":"2024-03-06T07:28:53.958843+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:24.376865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.142661+0000","last_change":"2024-03-06T07:28:52.963889+0000","last_active":"2024-03-06T07:28:54.142661+0000","last_peered":"2024-03-06T07:28:54.142661+0000","last_clean":"2024-03-06T07:28:54.142661+0000","last_became_active":"2024-03-06T07:28:52.963532+0000","last_became_peered":"2024-03-06T07:28:52.963532+0000","last_unstale":"2024-03-06T07:28:54.142661+0000","last_undegraded":"2024-03-06T07:28:54.142661+0000","last_fullsized":"2024-03-06T07:28:54.142661+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:44.151751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.954355+0000","last_change":"2024-03-06T07:28:52.967292+0000","last_active":"2024-03-06T07:28:53.954355+0000","last_peered":"2024-03-06T07:28:53.954355+0000","last_clean":"2024-03-06T07:28:53.954355+0000","last_became_active":"2024-03-06T07:28:52.967026+0000","last_became_peered":"2024-03-06T07:28:52.967026+0000","last_unstale":"2024-03-06T07:28:53.954355+0000","last_undegraded":"2024-03-06T07:28:53.954355+0000","last_fullsized":"2024-03-06T07:28:53.954355+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:35.361294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143793+0000","last_change":"2024-03-06T07:28:52.967696+0000","last_active":"2024-03-06T07:28:54.143793+0000","last_peered":"2024-03-06T07:28:54.143793+0000","last_clean":"2024-03-06T07:28:54.143793+0000","last_became_active":"2024-03-06T07:28:52.967448+0000","last_became_peered":"2024-03-06T07:28:52.967448+0000","last_unstale":"2024-03-06T07:28:54.143793+0000","last_undegraded":"2024-03-06T07:28:54.143793+0000","last_fullsized":"2024-03-06T07:28:54.143793+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:03.904000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144499+0000","last_change":"2024-03-06T07:28:52.972562+0000","last_active":"2024-03-06T07:28:54.144499+0000","last_peered":"2024-03-06T07:28:54.144499+0000","last_clean":"2024-03-06T07:28:54.144499+0000","last_became_active":"2024-03-06T07:28:52.972297+0000","last_became_peered":"2024-03-06T07:28:52.972297+0000","last_unstale":"2024-03-06T07:28:54.144499+0000","last_undegraded":"2024-03-06T07:28:54.144499+0000","last_fullsized":"2024-03-06T07:28:54.144499+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:26.575165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143103+0000","last_change":"2024-03-06T07:28:52.970892+0000","last_active":"2024-03-06T07:28:54.143103+0000","last_peered":"2024-03-06T07:28:54.143103+0000","last_clean":"2024-03-06T07:28:54.143103+0000","last_became_active":"2024-03-06T07:28:52.970576+0000","last_became_peered":"2024-03-06T07:28:52.970576+0000","last_unstale":"2024-03-06T07:28:54.143103+0000","last_undegraded":"2024-03-06T07:28:54.143103+0000","last_fullsized":"2024-03-06T07:28:54.143103+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:38.981799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144695+0000","last_change":"2024-03-06T07:28:52.970302+0000","last_active":"2024-03-06T07:28:54.144695+0000","last_peered":"2024-03-06T07:28:54.144695+0000","last_clean":"2024-03-06T07:28:54.144695+0000","last_became_active":"2024-03-06T07:28:52.970051+0000","last_became_peered":"2024-03-06T07:28:52.970051+0000","last_unstale":"2024-03-06T07:28:54.144695+0000","last_undegraded":"2024-03-06T07:28:54.144695+0000","last_fullsized":"2024-03-06T07:28:54.144695+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:32.804823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144636+0000","last_change":"2024-03-06T07:28:52.973548+0000","last_active":"2024-03-06T07:28:54.144636+0000","last_peered":"2024-03-06T07:28:54.144636+0000","last_clean":"2024-03-06T07:28:54.144636+0000","last_became_active":"2024-03-06T07:28:52.973271+0000","last_became_peered":"2024-03-06T07:28:52.973271+0000","last_unstale":"2024-03-06T07:28:54.144636+0000","last_undegraded":"2024-03-06T07:28:54.144636+0000","last_fullsized":"2024-03-06T07:28:54.144636+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:41.645943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143244+0000","last_change":"2024-03-06T07:28:52.975572+0000","last_active":"2024-03-06T07:28:54.143244+0000","last_peered":"2024-03-06T07:28:54.143244+0000","last_clean":"2024-03-06T07:28:54.143244+0000","last_became_active":"2024-03-06T07:28:52.975203+0000","last_became_peered":"2024-03-06T07:28:52.975203+0000","last_unstale":"2024-03-06T07:28:54.143244+0000","last_undegraded":"2024-03-06T07:28:54.143244+0000","last_fullsized":"2024-03-06T07:28:54.143244+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:16.118770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.959565+0000","last_change":"2024-03-06T07:28:52.960804+0000","last_active":"2024-03-06T07:28:53.959565+0000","last_peered":"2024-03-06T07:28:53.959565+0000","last_clean":"2024-03-06T07:28:53.959565+0000","last_became_active":"2024-03-06T07:28:52.960557+0000","last_became_peered":"2024-03-06T07:28:52.960557+0000","last_unstale":"2024-03-06T07:28:53.959565+0000","last_undegraded":"2024-03-06T07:28:53.959565+0000","last_fullsized":"2024-03-06T07:28:53.959565+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:15.776204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143936+0000","last_change":"2024-03-06T07:28:52.970667+0000","last_active":"2024-03-06T07:28:54.143936+0000","last_peered":"2024-03-06T07:28:54.143936+0000","last_clean":"2024-03-06T07:28:54.143936+0000","last_became_active":"2024-03-06T07:28:52.970408+0000","last_became_peered":"2024-03-06T07:28:52.970408+0000","last_unstale":"2024-03-06T07:28:54.143936+0000","last_undegraded":"2024-03-06T07:28:54.143936+0000","last_fullsized":"2024-03-06T07:28:54.143936+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:04:38.898631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"127'4","reported_seq":18,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143651+0000","last_change":"2024-03-06T07:28:51.970811+0000","last_active":"2024-03-06T07:28:54.143651+0000","last_peered":"2024-03-06T07:28:54.143651+0000","last_clean":"2024-03-06T07:28:54.143651+0000","last_became_active":"2024-03-06T07:28:51.970564+0000","last_became_peered":"2024-03-06T07:28:51.970564+0000","last_unstale":"2024-03-06T07:28:54.143651+0000","last_undegraded":"2024-03-06T07:28:54.143651+0000","last_fullsized":"2024-03-06T07:28:54.143651+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:59.741625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.959001+0000","last_change":"2024-03-06T07:28:52.969220+0000","last_active":"2024-03-06T07:28:53.959001+0000","last_peered":"2024-03-06T07:28:53.959001+0000","last_clean":"2024-03-06T07:28:53.959001+0000","last_became_active":"2024-03-06T07:28:52.968789+0000","last_became_peered":"2024-03-06T07:28:52.968789+0000","last_unstale":"2024-03-06T07:28:53.959001+0000","last_undegraded":"2024-03-06T07:28:53.959001+0000","last_fullsized":"2024-03-06T07:28:53.959001+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:15.776204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"127'1","reported_seq":15,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143089+0000","last_change":"2024-03-06T07:28:51.964480+0000","last_active":"2024-03-06T07:28:54.143089+0000","last_peered":"2024-03-06T07:28:54.143089+0000","last_clean":"2024-03-06T07:28:54.143089+0000","last_became_active":"2024-03-06T07:28:51.964203+0000","last_became_peered":"2024-03-06T07:28:51.964203+0000","last_unstale":"2024-03-06T07:28:54.143089+0000","last_undegraded":"2024-03-06T07:28:54.143089+0000","last_fullsized":"2024-03-06T07:28:54.143089+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:14:32.674292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143485+0000","last_change":"2024-03-06T07:28:52.965670+0000","last_active":"2024-03-06T07:28:54.143485+0000","last_peered":"2024-03-06T07:28:54.143485+0000","last_clean":"2024-03-06T07:28:54.143485+0000","last_became_active":"2024-03-06T07:28:52.965401+0000","last_became_peered":"2024-03-06T07:28:52.965401+0000","last_unstale":"2024-03-06T07:28:54.143485+0000","last_undegraded":"2024-03-06T07:28:54.143485+0000","last_fullsized":"2024-03-06T07:28:54.143485+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:27:10.021510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"127'2","reported_seq":16,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143552+0000","last_change":"2024-03-06T07:28:51.966438+0000","last_active":"2024-03-06T07:28:54.143552+0000","last_peered":"2024-03-06T07:28:54.143552+0000","last_clean":"2024-03-06T07:28:54.143552+0000","last_became_active":"2024-03-06T07:28:51.966143+0000","last_became_peered":"2024-03-06T07:28:51.966143+0000","last_unstale":"2024-03-06T07:28:54.143552+0000","last_undegraded":"2024-03-06T07:28:54.143552+0000","last_fullsized":"2024-03-06T07:28:54.143552+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:15.111853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143227+0000","last_change":"2024-03-06T07:28:52.969430+0000","last_active":"2024-03-06T07:28:54.143227+0000","last_peered":"2024-03-06T07:28:54.143227+0000","last_clean":"2024-03-06T07:28:54.143227+0000","last_became_active":"2024-03-06T07:28:52.968429+0000","last_became_peered":"2024-03-06T07:28:52.968429+0000","last_unstale":"2024-03-06T07:28:54.143227+0000","last_undegraded":"2024-03-06T07:28:54.143227+0000","last_fullsized":"2024-03-06T07:28:54.143227+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:14.242154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"127'2","reported_seq":16,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.954144+0000","last_change":"2024-03-06T07:28:51.964506+0000","last_active":"2024-03-06T07:28:53.954144+0000","last_peered":"2024-03-06T07:28:53.954144+0000","last_clean":"2024-03-06T07:28:53.954144+0000","last_became_active":"2024-03-06T07:28:51.964192+0000","last_became_peered":"2024-03-06T07:28:51.964192+0000","last_unstale":"2024-03-06T07:28:53.954144+0000","last_undegraded":"2024-03-06T07:28:53.954144+0000","last_fullsized":"2024-03-06T07:28:53.954144+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:43:03.523499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143145+0000","last_change":"2024-03-06T07:28:52.968598+0000","last_active":"2024-03-06T07:28:54.143145+0000","last_peered":"2024-03-06T07:28:54.143145+0000","last_clean":"2024-03-06T07:28:54.143145+0000","last_became_active":"2024-03-06T07:28:52.968335+0000","last_became_peered":"2024-03-06T07:28:52.968335+0000","last_unstale":"2024-03-06T07:28:54.143145+0000","last_undegraded":"2024-03-06T07:28:54.143145+0000","last_fullsized":"2024-03-06T07:28:54.143145+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:17:51.856534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"127'1","reported_seq":15,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.954028+0000","last_change":"2024-03-06T07:28:51.966376+0000","last_active":"2024-03-06T07:28:53.954028+0000","last_peered":"2024-03-06T07:28:53.954028+0000","last_clean":"2024-03-06T07:28:53.954028+0000","last_became_active":"2024-03-06T07:28:51.966123+0000","last_became_peered":"2024-03-06T07:28:51.966123+0000","last_unstale":"2024-03-06T07:28:53.954028+0000","last_undegraded":"2024-03-06T07:28:53.954028+0000","last_fullsized":"2024-03-06T07:28:53.954028+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:38:45.555438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.959722+0000","last_change":"2024-03-06T07:28:52.957122+0000","last_active":"2024-03-06T07:28:53.959722+0000","last_peered":"2024-03-06T07:28:53.959722+0000","last_clean":"2024-03-06T07:28:53.959722+0000","last_became_active":"2024-03-06T07:28:52.956828+0000","last_became_peered":"2024-03-06T07:28:52.956828+0000","last_unstale":"2024-03-06T07:28:53.959722+0000","last_undegraded":"2024-03-06T07:28:53.959722+0000","last_fullsized":"2024-03-06T07:28:53.959722+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:42.642239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"127'1","reported_seq":15,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143244+0000","last_change":"2024-03-06T07:28:51.971434+0000","last_active":"2024-03-06T07:28:54.143244+0000","last_peered":"2024-03-06T07:28:54.143244+0000","last_clean":"2024-03-06T07:28:54.143244+0000","last_became_active":"2024-03-06T07:28:51.971151+0000","last_became_peered":"2024-03-06T07:28:51.971151+0000","last_unstale":"2024-03-06T07:28:54.143244+0000","last_undegraded":"2024-03-06T07:28:54.143244+0000","last_fullsized":"2024-03-06T07:28:54.143244+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:38:00.770221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.958957+0000","last_change":"2024-03-06T07:28:52.959003+0000","last_active":"2024-03-06T07:28:53.958957+0000","last_peered":"2024-03-06T07:28:53.958957+0000","last_clean":"2024-03-06T07:28:53.958957+0000","last_became_active":"2024-03-06T07:28:52.958733+0000","last_became_peered":"2024-03-06T07:28:52.958733+0000","last_unstale":"2024-03-06T07:28:53.958957+0000","last_undegraded":"2024-03-06T07:28:53.958957+0000","last_fullsized":"2024-03-06T07:28:53.958957+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:38:01.777138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.959461+0000","last_change":"2024-03-06T07:28:51.965660+0000","last_active":"2024-03-06T07:28:53.959461+0000","last_peered":"2024-03-06T07:28:53.959461+0000","last_clean":"2024-03-06T07:28:53.959461+0000","last_became_active":"2024-03-06T07:28:51.965337+0000","last_became_peered":"2024-03-06T07:28:51.965337+0000","last_unstale":"2024-03-06T07:28:53.959461+0000","last_undegraded":"2024-03-06T07:28:53.959461+0000","last_fullsized":"2024-03-06T07:28:53.959461+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:35:26.827210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.953967+0000","last_change":"2024-03-06T07:28:52.959392+0000","last_active":"2024-03-06T07:28:53.953967+0000","last_peered":"2024-03-06T07:28:53.953967+0000","last_clean":"2024-03-06T07:28:53.953967+0000","last_became_active":"2024-03-06T07:28:52.959152+0000","last_became_peered":"2024-03-06T07:28:52.959152+0000","last_unstale":"2024-03-06T07:28:53.953967+0000","last_undegraded":"2024-03-06T07:28:53.953967+0000","last_fullsized":"2024-03-06T07:28:53.953967+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:50.800577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.954024+0000","last_change":"2024-03-06T07:28:51.961669+0000","last_active":"2024-03-06T07:28:53.954024+0000","last_peered":"2024-03-06T07:28:53.954024+0000","last_clean":"2024-03-06T07:28:53.954024+0000","last_became_active":"2024-03-06T07:28:51.961367+0000","last_became_peered":"2024-03-06T07:28:51.961367+0000","last_unstale":"2024-03-06T07:28:53.954024+0000","last_undegraded":"2024-03-06T07:28:53.954024+0000","last_fullsized":"2024-03-06T07:28:53.954024+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:37.702067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.145049+0000","last_change":"2024-03-06T07:28:52.967292+0000","last_active":"2024-03-06T07:28:54.145049+0000","last_peered":"2024-03-06T07:28:54.145049+0000","last_clean":"2024-03-06T07:28:54.145049+0000","last_became_active":"2024-03-06T07:28:52.967049+0000","last_became_peered":"2024-03-06T07:28:52.967049+0000","last_unstale":"2024-03-06T07:28:54.145049+0000","last_undegraded":"2024-03-06T07:28:54.145049+0000","last_fullsized":"2024-03-06T07:28:54.145049+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:07.633879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143435+0000","last_change":"2024-03-06T07:28:51.963421+0000","last_active":"2024-03-06T07:28:54.143435+0000","last_peered":"2024-03-06T07:28:54.143435+0000","last_clean":"2024-03-06T07:28:54.143435+0000","last_became_active":"2024-03-06T07:28:51.963127+0000","last_became_peered":"2024-03-06T07:28:51.963127+0000","last_unstale":"2024-03-06T07:28:54.143435+0000","last_undegraded":"2024-03-06T07:28:54.143435+0000","last_fullsized":"2024-03-06T07:28:54.143435+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:05:48.986537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143245+0000","last_change":"2024-03-06T07:28:52.973735+0000","last_active":"2024-03-06T07:28:54.143245+0000","last_peered":"2024-03-06T07:28:54.143245+0000","last_clean":"2024-03-06T07:28:54.143245+0000","last_became_active":"2024-03-06T07:28:52.973385+0000","last_became_peered":"2024-03-06T07:28:52.973385+0000","last_unstale":"2024-03-06T07:28:54.143245+0000","last_undegraded":"2024-03-06T07:28:54.143245+0000","last_fullsized":"2024-03-06T07:28:54.143245+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:50:41.441039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144044+0000","last_change":"2024-03-06T07:28:51.970138+0000","last_active":"2024-03-06T07:28:54.144044+0000","last_peered":"2024-03-06T07:28:54.144044+0000","last_clean":"2024-03-06T07:28:54.144044+0000","last_became_active":"2024-03-06T07:28:51.969796+0000","last_became_peered":"2024-03-06T07:28:51.969796+0000","last_unstale":"2024-03-06T07:28:54.144044+0000","last_undegraded":"2024-03-06T07:28:54.144044+0000","last_fullsized":"2024-03-06T07:28:54.144044+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:58.005409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143650+0000","last_change":"2024-03-06T07:28:52.965088+0000","last_active":"2024-03-06T07:28:54.143650+0000","last_peered":"2024-03-06T07:28:54.143650+0000","last_clean":"2024-03-06T07:28:54.143650+0000","last_became_active":"2024-03-06T07:28:52.964815+0000","last_became_peered":"2024-03-06T07:28:52.964815+0000","last_unstale":"2024-03-06T07:28:54.143650+0000","last_undegraded":"2024-03-06T07:28:54.143650+0000","last_fullsized":"2024-03-06T07:28:54.143650+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:44.151751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"127'3","reported_seq":17,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.142845+0000","last_change":"2024-03-06T07:28:51.968492+0000","last_active":"2024-03-06T07:28:54.142845+0000","last_peered":"2024-03-06T07:28:54.142845+0000","last_clean":"2024-03-06T07:28:54.142845+0000","last_became_active":"2024-03-06T07:28:51.968190+0000","last_became_peered":"2024-03-06T07:28:51.968190+0000","last_unstale":"2024-03-06T07:28:54.142845+0000","last_undegraded":"2024-03-06T07:28:54.142845+0000","last_fullsized":"2024-03-06T07:28:54.142845+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:18.858377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143597+0000","last_change":"2024-03-06T07:28:52.964754+0000","last_active":"2024-03-06T07:28:54.143597+0000","last_peered":"2024-03-06T07:28:54.143597+0000","last_clean":"2024-03-06T07:28:54.143597+0000","last_became_active":"2024-03-06T07:28:52.964437+0000","last_became_peered":"2024-03-06T07:28:52.964437+0000","last_unstale":"2024-03-06T07:28:54.143597+0000","last_undegraded":"2024-03-06T07:28:54.143597+0000","last_fullsized":"2024-03-06T07:28:54.143597+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:53.934839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"127'6","reported_seq":20,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.958669+0000","last_change":"2024-03-06T07:28:51.966045+0000","last_active":"2024-03-06T07:28:53.958669+0000","last_peered":"2024-03-06T07:28:53.958669+0000","last_clean":"2024-03-06T07:28:53.958669+0000","last_became_active":"2024-03-06T07:28:51.965790+0000","last_became_peered":"2024-03-06T07:28:51.965790+0000","last_unstale":"2024-03-06T07:28:53.958669+0000","last_undegraded":"2024-03-06T07:28:53.958669+0000","last_fullsized":"2024-03-06T07:28:53.958669+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:06:54.088262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144821+0000","last_change":"2024-03-06T07:28:52.962473+0000","last_active":"2024-03-06T07:28:54.144821+0000","last_peered":"2024-03-06T07:28:54.144821+0000","last_clean":"2024-03-06T07:28:54.144821+0000","last_became_active":"2024-03-06T07:28:52.961570+0000","last_became_peered":"2024-03-06T07:28:52.961570+0000","last_unstale":"2024-03-06T07:28:54.144821+0000","last_undegraded":"2024-03-06T07:28:54.144821+0000","last_fullsized":"2024-03-06T07:28:54.144821+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:50.800577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144881+0000","last_change":"2024-03-06T07:28:51.958418+0000","last_active":"2024-03-06T07:28:54.144881+0000","last_peered":"2024-03-06T07:28:54.144881+0000","last_clean":"2024-03-06T07:28:54.144881+0000","last_became_active":"2024-03-06T07:28:51.957944+0000","last_became_peered":"2024-03-06T07:28:51.957944+0000","last_unstale":"2024-03-06T07:28:54.144881+0000","last_undegraded":"2024-03-06T07:28:54.144881+0000","last_fullsized":"2024-03-06T07:28:54.144881+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:35.191066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.953924+0000","last_change":"2024-03-06T07:28:52.962686+0000","last_active":"2024-03-06T07:28:53.953924+0000","last_peered":"2024-03-06T07:28:53.953924+0000","last_clean":"2024-03-06T07:28:53.953924+0000","last_became_active":"2024-03-06T07:28:52.962407+0000","last_became_peered":"2024-03-06T07:28:52.962407+0000","last_unstale":"2024-03-06T07:28:53.953924+0000","last_undegraded":"2024-03-06T07:28:53.953924+0000","last_fullsized":"2024-03-06T07:28:53.953924+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:19.865295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.954049+0000","last_change":"2024-03-06T07:28:51.969835+0000","last_active":"2024-03-06T07:28:53.954049+0000","last_peered":"2024-03-06T07:28:53.954049+0000","last_clean":"2024-03-06T07:28:53.954049+0000","last_became_active":"2024-03-06T07:28:51.969530+0000","last_became_peered":"2024-03-06T07:28:51.969530+0000","last_unstale":"2024-03-06T07:28:53.954049+0000","last_undegraded":"2024-03-06T07:28:53.954049+0000","last_fullsized":"2024-03-06T07:28:53.954049+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:35.191066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.958376+0000","last_change":"2024-03-06T07:28:52.965364+0000","last_active":"2024-03-06T07:28:53.958376+0000","last_peered":"2024-03-06T07:28:53.958376+0000","last_clean":"2024-03-06T07:28:53.958376+0000","last_became_active":"2024-03-06T07:28:52.964831+0000","last_became_peered":"2024-03-06T07:28:52.964831+0000","last_unstale":"2024-03-06T07:28:53.958376+0000","last_undegraded":"2024-03-06T07:28:53.958376+0000","last_fullsized":"2024-03-06T07:28:53.958376+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:18.486012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"127'8","reported_seq":22,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.958804+0000","last_change":"2024-03-06T07:28:51.968492+0000","last_active":"2024-03-06T07:28:53.958804+0000","last_peered":"2024-03-06T07:28:53.958804+0000","last_clean":"2024-03-06T07:28:53.958804+0000","last_became_active":"2024-03-06T07:28:51.968146+0000","last_became_peered":"2024-03-06T07:28:51.968146+0000","last_unstale":"2024-03-06T07:28:53.958804+0000","last_undegraded":"2024-03-06T07:28:53.958804+0000","last_fullsized":"2024-03-06T07:28:53.958804+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:37.974882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.958778+0000","last_change":"2024-03-06T07:28:52.971128+0000","last_active":"2024-03-06T07:28:53.958778+0000","last_peered":"2024-03-06T07:28:53.958778+0000","last_clean":"2024-03-06T07:28:53.958778+0000","last_became_active":"2024-03-06T07:28:52.970657+0000","last_became_peered":"2024-03-06T07:28:52.970657+0000","last_unstale":"2024-03-06T07:28:53.958778+0000","last_undegraded":"2024-03-06T07:28:53.958778+0000","last_fullsized":"2024-03-06T07:28:53.958778+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:48:26.069964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"127'1","reported_seq":15,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143472+0000","last_change":"2024-03-06T07:28:51.966032+0000","last_active":"2024-03-06T07:28:54.143472+0000","last_peered":"2024-03-06T07:28:54.143472+0000","last_clean":"2024-03-06T07:28:54.143472+0000","last_became_active":"2024-03-06T07:28:51.965778+0000","last_became_peered":"2024-03-06T07:28:51.965778+0000","last_unstale":"2024-03-06T07:28:54.143472+0000","last_undegraded":"2024-03-06T07:28:54.143472+0000","last_fullsized":"2024-03-06T07:28:54.143472+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:01.850038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143257+0000","last_change":"2024-03-06T07:28:52.970288+0000","last_active":"2024-03-06T07:28:54.143257+0000","last_peered":"2024-03-06T07:28:54.143257+0000","last_clean":"2024-03-06T07:28:54.143257+0000","last_became_active":"2024-03-06T07:28:52.969973+0000","last_became_peered":"2024-03-06T07:28:52.969973+0000","last_unstale":"2024-03-06T07:28:54.143257+0000","last_undegraded":"2024-03-06T07:28:54.143257+0000","last_fullsized":"2024-03-06T07:28:54.143257+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:20:25.751101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"127'2","reported_seq":16,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143047+0000","last_change":"2024-03-06T07:28:51.961696+0000","last_active":"2024-03-06T07:28:54.143047+0000","last_peered":"2024-03-06T07:28:54.143047+0000","last_clean":"2024-03-06T07:28:54.143047+0000","last_became_active":"2024-03-06T07:28:51.961441+0000","last_became_peered":"2024-03-06T07:28:51.961441+0000","last_unstale":"2024-03-06T07:28:54.143047+0000","last_undegraded":"2024-03-06T07:28:54.143047+0000","last_fullsized":"2024-03-06T07:28:54.143047+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:42.468283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144735+0000","last_change":"2024-03-06T07:28:52.967362+0000","last_active":"2024-03-06T07:28:54.144735+0000","last_peered":"2024-03-06T07:28:54.144735+0000","last_clean":"2024-03-06T07:28:54.144735+0000","last_became_active":"2024-03-06T07:28:52.967091+0000","last_became_peered":"2024-03-06T07:28:52.967091+0000","last_unstale":"2024-03-06T07:28:54.144735+0000","last_undegraded":"2024-03-06T07:28:54.144735+0000","last_fullsized":"2024-03-06T07:28:54.144735+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:31.346210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"127'1","reported_seq":15,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.959971+0000","last_change":"2024-03-06T07:28:51.966073+0000","last_active":"2024-03-06T07:28:53.959971+0000","last_peered":"2024-03-06T07:28:53.959971+0000","last_clean":"2024-03-06T07:28:53.959971+0000","last_became_active":"2024-03-06T07:28:51.965786+0000","last_became_peered":"2024-03-06T07:28:51.965786+0000","last_unstale":"2024-03-06T07:28:53.959971+0000","last_undegraded":"2024-03-06T07:28:53.959971+0000","last_fullsized":"2024-03-06T07:28:53.959971+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:31:24.030877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.954365+0000","last_change":"2024-03-06T07:28:52.963231+0000","last_active":"2024-03-06T07:28:53.954365+0000","last_peered":"2024-03-06T07:28:53.954365+0000","last_clean":"2024-03-06T07:28:53.954365+0000","last_became_active":"2024-03-06T07:28:52.962463+0000","last_became_peered":"2024-03-06T07:28:52.962463+0000","last_unstale":"2024-03-06T07:28:53.954365+0000","last_undegraded":"2024-03-06T07:28:53.954365+0000","last_fullsized":"2024-03-06T07:28:53.954365+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:31.450403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143020+0000","last_change":"2024-03-06T07:28:51.964810+0000","last_active":"2024-03-06T07:28:54.143020+0000","last_peered":"2024-03-06T07:28:54.143020+0000","last_clean":"2024-03-06T07:28:54.143020+0000","last_became_active":"2024-03-06T07:28:51.964498+0000","last_became_peered":"2024-03-06T07:28:51.964498+0000","last_unstale":"2024-03-06T07:28:54.143020+0000","last_undegraded":"2024-03-06T07:28:54.143020+0000","last_fullsized":"2024-03-06T07:28:54.143020+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:31.797905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144772+0000","last_change":"2024-03-06T07:28:52.975907+0000","last_active":"2024-03-06T07:28:54.144772+0000","last_peered":"2024-03-06T07:28:54.144772+0000","last_clean":"2024-03-06T07:28:54.144772+0000","last_became_active":"2024-03-06T07:28:52.975063+0000","last_became_peered":"2024-03-06T07:28:52.975063+0000","last_unstale":"2024-03-06T07:28:54.144772+0000","last_undegraded":"2024-03-06T07:28:54.144772+0000","last_fullsized":"2024-03-06T07:28:54.144772+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:14:33.681210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"127'2","reported_seq":16,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143077+0000","last_change":"2024-03-06T07:28:51.963111+0000","last_active":"2024-03-06T07:28:54.143077+0000","last_peered":"2024-03-06T07:28:54.143077+0000","last_clean":"2024-03-06T07:28:54.143077+0000","last_became_active":"2024-03-06T07:28:51.962804+0000","last_became_peered":"2024-03-06T07:28:51.962804+0000","last_unstale":"2024-03-06T07:28:54.143077+0000","last_undegraded":"2024-03-06T07:28:54.143077+0000","last_fullsized":"2024-03-06T07:28:54.143077+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:49.793659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.959862+0000","last_change":"2024-03-06T07:28:52.963806+0000","last_active":"2024-03-06T07:28:53.959862+0000","last_peered":"2024-03-06T07:28:53.959862+0000","last_clean":"2024-03-06T07:28:53.959862+0000","last_became_active":"2024-03-06T07:28:52.963507+0000","last_became_peered":"2024-03-06T07:28:52.963507+0000","last_unstale":"2024-03-06T07:28:53.959862+0000","last_undegraded":"2024-03-06T07:28:53.959862+0000","last_fullsized":"2024-03-06T07:28:53.959862+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:54:25.798793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143332+0000","last_change":"2024-03-06T07:28:51.968470+0000","last_active":"2024-03-06T07:28:54.143332+0000","last_peered":"2024-03-06T07:28:54.143332+0000","last_clean":"2024-03-06T07:28:54.143332+0000","last_became_active":"2024-03-06T07:28:51.968177+0000","last_became_peered":"2024-03-06T07:28:51.968177+0000","last_unstale":"2024-03-06T07:28:54.143332+0000","last_undegraded":"2024-03-06T07:28:54.143332+0000","last_fullsized":"2024-03-06T07:28:54.143332+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:48:25.063046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.953928+0000","last_change":"2024-03-06T07:28:52.958145+0000","last_active":"2024-03-06T07:28:53.953928+0000","last_peered":"2024-03-06T07:28:53.953928+0000","last_clean":"2024-03-06T07:28:53.953928+0000","last_became_active":"2024-03-06T07:28:52.957842+0000","last_became_peered":"2024-03-06T07:28:52.957842+0000","last_unstale":"2024-03-06T07:28:53.953928+0000","last_undegraded":"2024-03-06T07:28:53.953928+0000","last_fullsized":"2024-03-06T07:28:53.953928+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:32.804823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.142893+0000","last_change":"2024-03-06T07:28:51.968301+0000","last_active":"2024-03-06T07:28:54.142893+0000","last_peered":"2024-03-06T07:28:54.142893+0000","last_clean":"2024-03-06T07:28:54.142893+0000","last_became_active":"2024-03-06T07:28:51.968026+0000","last_became_peered":"2024-03-06T07:28:51.968026+0000","last_unstale":"2024-03-06T07:28:54.142893+0000","last_undegraded":"2024-03-06T07:28:54.142893+0000","last_fullsized":"2024-03-06T07:28:54.142893+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:23.369947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143009+0000","last_change":"2024-03-06T07:28:52.972374+0000","last_active":"2024-03-06T07:28:54.143009+0000","last_peered":"2024-03-06T07:28:54.143009+0000","last_clean":"2024-03-06T07:28:54.143009+0000","last_became_active":"2024-03-06T07:28:52.972101+0000","last_became_peered":"2024-03-06T07:28:52.972101+0000","last_unstale":"2024-03-06T07:28:54.143009+0000","last_undegraded":"2024-03-06T07:28:54.143009+0000","last_fullsized":"2024-03-06T07:28:54.143009+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:41.237344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.142599+0000","last_change":"2024-03-06T07:28:51.960935+0000","last_active":"2024-03-06T07:28:54.142599+0000","last_peered":"2024-03-06T07:28:54.142599+0000","last_clean":"2024-03-06T07:28:54.142599+0000","last_became_active":"2024-03-06T07:28:51.960458+0000","last_became_peered":"2024-03-06T07:28:51.960458+0000","last_unstale":"2024-03-06T07:28:54.142599+0000","last_undegraded":"2024-03-06T07:28:54.142599+0000","last_fullsized":"2024-03-06T07:28:54.142599+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:40.230426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.959956+0000","last_change":"2024-03-06T07:28:52.969742+0000","last_active":"2024-03-06T07:28:53.959956+0000","last_peered":"2024-03-06T07:28:53.959956+0000","last_clean":"2024-03-06T07:28:53.959956+0000","last_became_active":"2024-03-06T07:28:52.969401+0000","last_became_peered":"2024-03-06T07:28:52.969401+0000","last_unstale":"2024-03-06T07:28:53.959956+0000","last_undegraded":"2024-03-06T07:28:53.959956+0000","last_fullsized":"2024-03-06T07:28:53.959956+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:08.434943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"127'2","reported_seq":16,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.145058+0000","last_change":"2024-03-06T07:28:51.959525+0000","last_active":"2024-03-06T07:28:54.145058+0000","last_peered":"2024-03-06T07:28:54.145058+0000","last_clean":"2024-03-06T07:28:54.145058+0000","last_became_active":"2024-03-06T07:28:51.959247+0000","last_became_peered":"2024-03-06T07:28:51.959247+0000","last_unstale":"2024-03-06T07:28:54.145058+0000","last_undegraded":"2024-03-06T07:28:54.145058+0000","last_fullsized":"2024-03-06T07:28:54.145058+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:37.278665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.145016+0000","last_change":"2024-03-06T07:28:52.972711+0000","last_active":"2024-03-06T07:28:54.145016+0000","last_peered":"2024-03-06T07:28:54.145016+0000","last_clean":"2024-03-06T07:28:54.145016+0000","last_became_active":"2024-03-06T07:28:52.972448+0000","last_became_peered":"2024-03-06T07:28:52.972448+0000","last_unstale":"2024-03-06T07:28:54.145016+0000","last_undegraded":"2024-03-06T07:28:54.145016+0000","last_fullsized":"2024-03-06T07:28:54.145016+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:02.856955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"128'1","reported_seq":15,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.963947+0000","last_change":"2024-03-06T07:28:51.969705+0000","last_active":"2024-03-06T07:28:54.963947+0000","last_peered":"2024-03-06T07:28:54.963947+0000","last_clean":"2024-03-06T07:28:54.963947+0000","last_became_active":"2024-03-06T07:28:51.969395+0000","last_became_peered":"2024-03-06T07:28:51.969395+0000","last_unstale":"2024-03-06T07:28:54.963947+0000","last_undegraded":"2024-03-06T07:28:54.963947+0000","last_fullsized":"2024-03-06T07:28:54.963947+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:59:39.689093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.954215+0000","last_change":"2024-03-06T07:28:52.963942+0000","last_active":"2024-03-06T07:28:53.954215+0000","last_peered":"2024-03-06T07:28:53.954215+0000","last_clean":"2024-03-06T07:28:53.954215+0000","last_became_active":"2024-03-06T07:28:52.963685+0000","last_became_peered":"2024-03-06T07:28:52.963685+0000","last_unstale":"2024-03-06T07:28:53.954215+0000","last_undegraded":"2024-03-06T07:28:53.954215+0000","last_fullsized":"2024-03-06T07:28:53.954215+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:02.856955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.954356+0000","last_change":"2024-03-06T07:28:51.960696+0000","last_active":"2024-03-06T07:28:53.954356+0000","last_peered":"2024-03-06T07:28:53.954356+0000","last_clean":"2024-03-06T07:28:53.954356+0000","last_became_active":"2024-03-06T07:28:51.960438+0000","last_became_peered":"2024-03-06T07:28:51.960438+0000","last_unstale":"2024-03-06T07:28:53.954356+0000","last_undegraded":"2024-03-06T07:28:53.954356+0000","last_fullsized":"2024-03-06T07:28:53.954356+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:14.232718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.959925+0000","last_change":"2024-03-06T07:28:52.967435+0000","last_active":"2024-03-06T07:28:53.959925+0000","last_peered":"2024-03-06T07:28:53.959925+0000","last_clean":"2024-03-06T07:28:53.959925+0000","last_became_active":"2024-03-06T07:28:52.967158+0000","last_became_peered":"2024-03-06T07:28:52.967158+0000","last_unstale":"2024-03-06T07:28:53.959925+0000","last_undegraded":"2024-03-06T07:28:53.959925+0000","last_fullsized":"2024-03-06T07:28:53.959925+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:59:40.696010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143231+0000","last_change":"2024-03-06T07:28:51.963760+0000","last_active":"2024-03-06T07:28:54.143231+0000","last_peered":"2024-03-06T07:28:54.143231+0000","last_clean":"2024-03-06T07:28:54.143231+0000","last_became_active":"2024-03-06T07:28:51.963374+0000","last_became_peered":"2024-03-06T07:28:51.963374+0000","last_unstale":"2024-03-06T07:28:54.143231+0000","last_undegraded":"2024-03-06T07:28:54.143231+0000","last_fullsized":"2024-03-06T07:28:54.143231+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:13.235236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143242+0000","last_change":"2024-03-06T07:28:52.968583+0000","last_active":"2024-03-06T07:28:54.143242+0000","last_peered":"2024-03-06T07:28:54.143242+0000","last_clean":"2024-03-06T07:28:54.143242+0000","last_became_active":"2024-03-06T07:28:52.968266+0000","last_became_peered":"2024-03-06T07:28:52.968266+0000","last_unstale":"2024-03-06T07:28:54.143242+0000","last_undegraded":"2024-03-06T07:28:54.143242+0000","last_fullsized":"2024-03-06T07:28:54.143242+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:43.475200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.954120+0000","last_change":"2024-03-06T07:28:51.966274+0000","last_active":"2024-03-06T07:28:53.954120+0000","last_peered":"2024-03-06T07:28:53.954120+0000","last_clean":"2024-03-06T07:28:53.954120+0000","last_became_active":"2024-03-06T07:28:51.965795+0000","last_became_peered":"2024-03-06T07:28:51.965795+0000","last_unstale":"2024-03-06T07:28:53.954120+0000","last_undegraded":"2024-03-06T07:28:53.954120+0000","last_fullsized":"2024-03-06T07:28:53.954120+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:44:14.025940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143168+0000","last_change":"2024-03-06T07:28:52.978648+0000","last_active":"2024-03-06T07:28:54.143168+0000","last_peered":"2024-03-06T07:28:54.143168+0000","last_clean":"2024-03-06T07:28:54.143168+0000","last_became_active":"2024-03-06T07:28:52.978331+0000","last_became_peered":"2024-03-06T07:28:52.978331+0000","last_unstale":"2024-03-06T07:28:54.143168+0000","last_undegraded":"2024-03-06T07:28:54.143168+0000","last_fullsized":"2024-03-06T07:28:54.143168+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:04.981862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143248+0000","last_change":"2024-03-06T07:28:51.971364+0000","last_active":"2024-03-06T07:28:54.143248+0000","last_peered":"2024-03-06T07:28:54.143248+0000","last_clean":"2024-03-06T07:28:54.143248+0000","last_became_active":"2024-03-06T07:28:51.971030+0000","last_became_peered":"2024-03-06T07:28:51.971030+0000","last_unstale":"2024-03-06T07:28:54.143248+0000","last_undegraded":"2024-03-06T07:28:54.143248+0000","last_fullsized":"2024-03-06T07:28:54.143248+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:07.428026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144153+0000","last_change":"2024-03-06T07:28:52.961576+0000","last_active":"2024-03-06T07:28:54.144153+0000","last_peered":"2024-03-06T07:28:54.144153+0000","last_clean":"2024-03-06T07:28:54.144153+0000","last_became_active":"2024-03-06T07:28:52.961325+0000","last_became_peered":"2024-03-06T07:28:52.961325+0000","last_unstale":"2024-03-06T07:28:54.144153+0000","last_undegraded":"2024-03-06T07:28:54.144153+0000","last_fullsized":"2024-03-06T07:28:54.144153+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:53.405336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.142987+0000","last_change":"2024-03-06T07:28:51.968672+0000","last_active":"2024-03-06T07:28:54.142987+0000","last_peered":"2024-03-06T07:28:54.142987+0000","last_clean":"2024-03-06T07:28:54.142987+0000","last_became_active":"2024-03-06T07:28:51.968340+0000","last_became_peered":"2024-03-06T07:28:51.968340+0000","last_unstale":"2024-03-06T07:28:54.142987+0000","last_undegraded":"2024-03-06T07:28:54.142987+0000","last_fullsized":"2024-03-06T07:28:54.142987+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:22:47.593555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.953957+0000","last_change":"2024-03-06T07:28:52.966628+0000","last_active":"2024-03-06T07:28:53.953957+0000","last_peered":"2024-03-06T07:28:53.953957+0000","last_clean":"2024-03-06T07:28:53.953957+0000","last_became_active":"2024-03-06T07:28:52.966354+0000","last_became_peered":"2024-03-06T07:28:52.966354+0000","last_unstale":"2024-03-06T07:28:53.953957+0000","last_undegraded":"2024-03-06T07:28:53.953957+0000","last_fullsized":"2024-03-06T07:28:53.953957+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:14:33.681210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.142673+0000","last_change":"2024-03-06T07:28:51.960707+0000","last_active":"2024-03-06T07:28:54.142673+0000","last_peered":"2024-03-06T07:28:54.142673+0000","last_clean":"2024-03-06T07:28:54.142673+0000","last_became_active":"2024-03-06T07:28:51.960177+0000","last_became_peered":"2024-03-06T07:28:51.960177+0000","last_unstale":"2024-03-06T07:28:54.142673+0000","last_undegraded":"2024-03-06T07:28:54.142673+0000","last_fullsized":"2024-03-06T07:28:54.142673+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:41.094093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143122+0000","last_change":"2024-03-06T07:28:52.975661+0000","last_active":"2024-03-06T07:28:54.143122+0000","last_peered":"2024-03-06T07:28:54.143122+0000","last_clean":"2024-03-06T07:28:54.143122+0000","last_became_active":"2024-03-06T07:28:52.975355+0000","last_became_peered":"2024-03-06T07:28:52.975355+0000","last_unstale":"2024-03-06T07:28:54.143122+0000","last_undegraded":"2024-03-06T07:28:54.143122+0000","last_fullsized":"2024-03-06T07:28:54.143122+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:42.101011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"127'2","reported_seq":16,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143618+0000","last_change":"2024-03-06T07:28:51.970862+0000","last_active":"2024-03-06T07:28:54.143618+0000","last_peered":"2024-03-06T07:28:54.143618+0000","last_clean":"2024-03-06T07:28:54.143618+0000","last_became_active":"2024-03-06T07:28:51.970609+0000","last_became_peered":"2024-03-06T07:28:51.970609+0000","last_unstale":"2024-03-06T07:28:54.143618+0000","last_undegraded":"2024-03-06T07:28:54.143618+0000","last_fullsized":"2024-03-06T07:28:54.143618+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:40.486032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":364,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.142779+0000","last_change":"2024-03-06T07:21:47.553693+0000","last_active":"2024-03-06T07:28:54.142779+0000","last_peered":"2024-03-06T07:28:54.142779+0000","last_clean":"2024-03-06T07:28:54.142779+0000","last_became_active":"2024-03-06T07:21:47.552664+0000","last_became_peered":"2024-03-06T07:21:47.552664+0000","last_unstale":"2024-03-06T07:28:54.142779+0000","last_undegraded":"2024-03-06T07:28:54.142779+0000","last_fullsized":"2024-03-06T07:28:54.142779+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_clean_scrub_stamp":"2024-03-06T07:21:46.362028+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:02.916426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574939,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41664,"kb_used_data":2164,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330176,"statfs":{"total":96636764160,"available":96594100224,"internally_reserved":0,"allocated":2215936,"data_stored":1288896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":40434121},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574937,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41072,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330768,"statfs":{"total":96636764160,"available":96594706432,"internally_reserved":0,"allocated":1609728,"data_stored":697890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574938,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41068,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330772,"statfs":{"total":96636764160,"available":96594710528,"internally_reserved":0,"allocated":1605632,"data_stored":696390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574939,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41068,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330772,"statfs":{"total":96636764160,"available":96594710528,"internally_reserved":0,"allocated":1601536,"data_stored":696348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574938,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41068,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330772,"statfs":{"total":96636764160,"available":96594710528,"internally_reserved":0,"allocated":1605632,"data_stored":696438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574938,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41068,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330772,"statfs":{"total":96636764160,"available":96594710528,"internally_reserved":0,"allocated":1605632,"data_stored":696894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574939,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41060,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330780,"statfs":{"total":96636764160,"available":96594718720,"internally_reserved":0,"allocated":1597440,"data_stored":696278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574938,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41648,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330192,"statfs":{"total":96636764160,"available":96594116608,"internally_reserved":0,"allocated":2191360,"data_stored":1286646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:28:56.707 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:28:56.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.840+0000 7f32b5466640 1 Processor -- start 2024-03-06T07:28:56.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.840+0000 7f32b5466640 1 -- start start 2024-03-06T07:28:56.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.840+0000 7f32b5466640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f32b00b4460 0x7f32b00d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.840+0000 7f32b5466640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f32b0162970 0x7f32b00b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.840+0000 7f32b5466640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b00d22c0 con 0x7f32b0161cf0 2024-03-06T07:28:56.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.840+0000 7f32b5466640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32b0161b00 con 0x7f32b0162970 2024-03-06T07:28:56.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.840+0000 7f32b5466640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32b00cecc0 con 0x7f32b00b4460 2024-03-06T07:28:56.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.841+0000 7f32af7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f32b0161cf0 0x7f32b00cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49396/0 (socket says 172.21.15.83:49396) 2024-03-06T07:28:56.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.841+0000 7f32af7fe640 1 -- 172.21.15.83:0/3364475417 learned_addr learned my addr 172.21.15.83:0/3364475417 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:28:56.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.841+0000 7f32ae7fc640 1 --2- 172.21.15.83:0/3364475417 >> v2:172.21.15.183:3300/0 conn(0x7f32b0162970 0x7f32b00b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.841+0000 7f32aeffd640 1 --2- 172.21.15.83:0/3364475417 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f32b00b4460 0x7f32b00d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.841+0000 7f32adffb640 1 -- 172.21.15.83:0/3364475417 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3480072446 0 0) 0x7f32b00d22c0 con 0x7f32b0161cf0 2024-03-06T07:28:56.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.841+0000 7f32adffb640 1 -- 172.21.15.83:0/3364475417 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3298003520 con 0x7f32b0161cf0 2024-03-06T07:28:56.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.841+0000 7f32adffb640 1 -- 172.21.15.83:0/3364475417 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2946643140 0 0) 0x7f3298003520 con 0x7f32b0161cf0 2024-03-06T07:28:56.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.841+0000 7f32adffb640 1 -- 172.21.15.83:0/3364475417 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f32b00b4460 msgr2=0x7f32b00d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:56.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.841+0000 7f32adffb640 1 --2- 172.21.15.83:0/3364475417 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f32b00b4460 0x7f32b00d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.841+0000 7f32adffb640 1 -- 172.21.15.83:0/3364475417 >> v2:172.21.15.183:3300/0 conn(0x7f32b0162970 msgr2=0x7f32b00b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:56.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.841+0000 7f32adffb640 1 --2- 172.21.15.83:0/3364475417 >> v2:172.21.15.183:3300/0 conn(0x7f32b0162970 0x7f32b00b3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.841+0000 7f32adffb640 1 -- 172.21.15.83:0/3364475417 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b00d25a0 con 0x7f32b0161cf0 2024-03-06T07:28:56.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.841+0000 7f32adffb640 1 -- 172.21.15.83:0/3364475417 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f32a80019f0 con 0x7f32b0161cf0 2024-03-06T07:28:56.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.841+0000 7f32adffb640 1 -- 172.21.15.83:0/3364475417 >> v1:172.21.15.83:6789/0 conn(0x7f32b0161cf0 legacy=0x7f32b00cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.841+0000 7f32adffb640 1 --2- 172.21.15.83:0/3364475417 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f32980039b0 0x7f3298003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.841+0000 7f32adffb640 1 --2- 172.21.15.83:0/3364475417 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f32b00b4460 0x7f3298004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.841+0000 7f32adffb640 1 --2- 172.21.15.83:0/3364475417 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3298004d40 0x7f32980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.841+0000 7f32adffb640 1 -- 172.21.15.83:0/3364475417 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b0161b00 con 0x7f32980039b0 2024-03-06T07:28:56.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.841+0000 7f32adffb640 1 -- 172.21.15.83:0/3364475417 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b00cecc0 con 0x7f3298004d40 2024-03-06T07:28:56.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.841+0000 7f32adffb640 1 -- 172.21.15.83:0/3364475417 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f329800b9c0 con 0x7f32b00b4460 2024-03-06T07:28:56.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.841+0000 7f32aeffd640 1 --2- 172.21.15.83:0/3364475417 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f32980039b0 0x7f3298003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.841+0000 7f32ae7fc640 1 --2- 172.21.15.83:0/3364475417 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f32b00b4460 0x7f3298004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.841+0000 7f32af7fe640 1 --2- 172.21.15.83:0/3364475417 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3298004d40 0x7f32980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.841+0000 7f32aeffd640 1 -- 172.21.15.83:0/3364475417 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f32b00b4460 msgr2=0x7f3298004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:56.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.841+0000 7f32aeffd640 1 --2- 172.21.15.83:0/3364475417 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f32b00b4460 0x7f3298004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.841+0000 7f32aeffd640 1 -- 172.21.15.83:0/3364475417 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3298004d40 msgr2=0x7f32980051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:56.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.841+0000 7f32aeffd640 1 --2- 172.21.15.83:0/3364475417 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3298004d40 0x7f32980051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.841+0000 7f32aeffd640 1 -- 172.21.15.83:0/3364475417 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3298003520 con 0x7f32980039b0 2024-03-06T07:28:56.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.841+0000 7f32af7fe640 1 --2- 172.21.15.83:0/3364475417 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3298004d40 0x7f32980051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:56.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.842+0000 7f32aeffd640 1 --2- 172.21.15.83:0/3364475417 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f32980039b0 0x7f3298003dd0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f32a0009ac0 tx=0x7f3298005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=239fb5c906310ef1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.842+0000 7f32adffb640 1 -- 172.21.15.83:0/3364475417 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a0039070 con 0x7f32980039b0 2024-03-06T07:28:56.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.842+0000 7f32adffb640 1 -- 172.21.15.83:0/3364475417 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32a002c6b0 con 0x7f32980039b0 2024-03-06T07:28:56.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.842+0000 7f32adffb640 1 -- 172.21.15.83:0/3364475417 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a0009690 con 0x7f32980039b0 2024-03-06T07:28:56.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.843+0000 7f32b5466640 1 -- 172.21.15.83:0/3364475417 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f32980039b0 msgr2=0x7f3298003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.843+0000 7f32b5466640 1 --2- 172.21.15.83:0/3364475417 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f32980039b0 0x7f3298003dd0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f32a0009ac0 tx=0x7f3298005830 comp rx=0 tx=0).stop 2024-03-06T07:28:56.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.843+0000 7f32af7fe640 1 -- 172.21.15.83:0/3364475417 reap_dead start 2024-03-06T07:28:56.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.843+0000 7f32b5466640 1 -- 172.21.15.83:0/3364475417 shutdown_connections 2024-03-06T07:28:56.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.843+0000 7f32b5466640 1 -- 172.21.15.83:0/3364475417 >> 172.21.15.83:0/3364475417 conn(0x7f32b00d3ce0 msgr2=0x7f32b00d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:56.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.843+0000 7f32b5466640 1 -- 172.21.15.83:0/3364475417 shutdown_connections 2024-03-06T07:28:56.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.843+0000 7f32b5466640 1 -- 172.21.15.83:0/3364475417 wait complete. 2024-03-06T07:28:56.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.844+0000 7f32b5466640 1 Processor -- start 2024-03-06T07:28:56.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.845+0000 7f32b5466640 1 -- start start 2024-03-06T07:28:56.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.845+0000 7f32b5466640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f32980039b0 0x7f32b00b7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.845+0000 7f32b5466640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f32b0150140 0x7f32b00cd570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.845+0000 7f32b5466640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f32b0160e40 0x7f32b01612c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.845+0000 7f32b5466640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b00ce950 con 0x7f32980039b0 2024-03-06T07:28:56.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.845+0000 7f32b5466640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b00d2b60 con 0x7f32b0160e40 2024-03-06T07:28:56.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.845+0000 7f32b5466640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32b00b3ac0 con 0x7f32b0150140 2024-03-06T07:28:56.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.845+0000 7f32aeffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f32980039b0 0x7f32b00b7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.845+0000 7f32aeffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f32980039b0 0x7f32b00b7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51992/0 (socket says 172.21.15.83:51992) 2024-03-06T07:28:56.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.845+0000 7f32aeffd640 1 -- 172.21.15.83:0/312402911 learned_addr learned my addr 172.21.15.83:0/312402911 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:56.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.845+0000 7f32ae7fc640 1 --2- 172.21.15.83:0/312402911 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f32b0150140 0x7f32b00cd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.845+0000 7f32af7fe640 1 --2- 172.21.15.83:0/312402911 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f32b0160e40 0x7f32b01612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.846+0000 7f32aeffd640 1 -- 172.21.15.83:0/312402911 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f32b0150140 msgr2=0x7f32b00cd570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.846+0000 7f32aeffd640 1 --2- 172.21.15.83:0/312402911 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f32b0150140 0x7f32b00cd570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.846+0000 7f32aeffd640 1 -- 172.21.15.83:0/312402911 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f32b0160e40 msgr2=0x7f32b01612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.846+0000 7f32aeffd640 1 --2- 172.21.15.83:0/312402911 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f32b0160e40 0x7f32b01612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.846+0000 7f32aeffd640 1 -- 172.21.15.83:0/312402911 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b00b6d50 con 0x7f32980039b0 2024-03-06T07:28:56.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.846+0000 7f32aeffd640 1 --2- 172.21.15.83:0/312402911 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f32980039b0 0x7f32b00b7df0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f32a0037040 tx=0x7f32a0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.846+0000 7f32977fe640 1 -- 172.21.15.83:0/312402911 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a0042020 con 0x7f32980039b0 2024-03-06T07:28:56.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.846+0000 7f32977fe640 1 -- 172.21.15.83:0/312402911 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32a003a040 con 0x7f32980039b0 2024-03-06T07:28:56.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.846+0000 7f32b5466640 1 -- 172.21.15.83:0/312402911 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32b00b7030 con 0x7f32980039b0 2024-03-06T07:28:56.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.846+0000 7f32977fe640 1 -- 172.21.15.83:0/312402911 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a0039070 con 0x7f32980039b0 2024-03-06T07:28:56.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.846+0000 7f32b5466640 1 -- 172.21.15.83:0/312402911 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32b00b75f0 con 0x7f32980039b0 2024-03-06T07:28:56.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.847+0000 7f32b5466640 1 -- 172.21.15.83:0/312402911 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32b004f330 con 0x7f32980039b0 2024-03-06T07:28:56.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.847+0000 7f32977fe640 1 -- 172.21.15.83:0/312402911 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f32a0045020 con 0x7f32980039b0 2024-03-06T07:28:56.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.848+0000 7f32977fe640 1 --2- 172.21.15.83:0/312402911 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f329c06b500 0x7f329c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.848+0000 7f32977fe640 1 -- 172.21.15.83:0/312402911 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f32a00ab590 con 0x7f32980039b0 2024-03-06T07:28:56.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.848+0000 7f32ae7fc640 1 --2- 172.21.15.83:0/312402911 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f329c06b500 0x7f329c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.848+0000 7f32ae7fc640 1 --2- 172.21.15.83:0/312402911 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f329c06b500 0x7f329c06d9c0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f32b00740e0 tx=0x7f32a4009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:56.852+0000 7f32977fe640 1 -- 172.21.15.83:0/312402911 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f32a004c070 con 0x7f32980039b0 2024-03-06T07:28:56.920 INFO:tasks.ceph.mds.a.smithi083.stderr:2024-03-06T07:28:56.917+0000 7fa26e0b8640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:28:57.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.098+0000 7f32b5466640 1 -- 172.21.15.83:0/312402911 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f32b00b3d00 con 0x7f32980039b0 2024-03-06T07:28:57.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.099+0000 7f32977fe640 1 -- 172.21.15.83:0/312402911 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1014 (secure 0 0 0) 0x7f32a007a4d0 con 0x7f32980039b0 2024-03-06T07:28:57.102 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:28:57.102 INFO:teuthology.orchestra.run.smithi083.stdout:{"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":442,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":128,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709705,"num_in_osds":8,"osd_in_since":1709709696,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":24,"data_bytes":592778,"bytes_used":337629184,"bytes_avail":772756484096,"bytes_total":773094113280,"write_bytes_sec":2627,"read_op_per_sec":0,"write_op_per_sec":8},"fsmap":{"epoch":73,"id":9,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":9,"rank":0,"name":"a","status":"up:active","gid":8631}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-03-06T07:28:36.392424+0000","services":{}},"progress_events":{}} 2024-03-06T07:28:57.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.101+0000 7f32b5466640 1 -- 172.21.15.83:0/312402911 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f329c06b500 msgr2=0x7f329c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.101+0000 7f32b5466640 1 --2- 172.21.15.83:0/312402911 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f329c06b500 0x7f329c06d9c0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f32b00740e0 tx=0x7f32a4009190 comp rx=0 tx=0).stop 2024-03-06T07:28:57.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.101+0000 7f32b5466640 1 -- 172.21.15.83:0/312402911 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f32980039b0 msgr2=0x7f32b00b7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.101+0000 7f32b5466640 1 --2- 172.21.15.83:0/312402911 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f32980039b0 0x7f32b00b7df0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f32a0037040 tx=0x7f32a0004730 comp rx=0 tx=0).stop 2024-03-06T07:28:57.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.101+0000 7f32b5466640 1 -- 172.21.15.83:0/312402911 shutdown_connections 2024-03-06T07:28:57.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.101+0000 7f32b5466640 1 --2- 172.21.15.83:0/312402911 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f329c06b500 0x7f329c06d9c0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.101+0000 7f32b5466640 1 --2- 172.21.15.83:0/312402911 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f32b0160e40 0x7f32b01612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.101+0000 7f32b5466640 1 --2- 172.21.15.83:0/312402911 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f32b0150140 0x7f32b00cd570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.101+0000 7f32b5466640 1 --2- 172.21.15.83:0/312402911 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f32980039b0 0x7f32b00b7df0 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.101+0000 7f32b5466640 1 -- 172.21.15.83:0/312402911 >> 172.21.15.83:0/312402911 conn(0x7f32b00d3ce0 msgr2=0x7f32b00d00c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:57.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.101+0000 7f32b5466640 1 -- 172.21.15.83:0/312402911 shutdown_connections 2024-03-06T07:28:57.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.101+0000 7f32b5466640 1 -- 172.21.15.83:0/312402911 wait complete. 2024-03-06T07:28:57.115 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:28:57.115 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:28:57.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.213+0000 7f0ba7a42640 1 Processor -- start 2024-03-06T07:28:57.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.213+0000 7f0ba7a42640 1 -- start start 2024-03-06T07:28:57.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.213+0000 7f0ba7a42640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0ba015dfe0 0x7f0ba00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.213+0000 7f0ba7a42640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0ba00cf830 0x7f0ba00d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.213+0000 7f0ba7a42640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba00b3430 con 0x7f0ba00d27f0 2024-03-06T07:28:57.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.213+0000 7f0ba7a42640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ba00d2c50 con 0x7f0ba015dfe0 2024-03-06T07:28:57.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.213+0000 7f0ba7a42640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba00b31e0 con 0x7f0ba00cf830 2024-03-06T07:28:57.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.213+0000 7f0ba57de640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0ba015dfe0 0x7f0ba00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.214+0000 7f0ba57de640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0ba015dfe0 0x7f0ba00cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52006/0 (socket says 172.21.15.83:52006) 2024-03-06T07:28:57.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.214+0000 7f0ba57de640 1 -- 172.21.15.83:0/3341517576 learned_addr learned my addr 172.21.15.83:0/3341517576 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:57.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.214+0000 7f0ba4fdd640 1 --2- 172.21.15.83:0/3341517576 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0ba00cf830 0x7f0ba00d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.214+0000 7f0ba57de640 1 -- 172.21.15.83:0/3341517576 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0ba00cf830 msgr2=0x7f0ba00d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:57.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.214+0000 7f0ba57de640 1 --2- 172.21.15.83:0/3341517576 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0ba00cf830 0x7f0ba00d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.214+0000 7f0ba57de640 1 -- 172.21.15.83:0/3341517576 >> v1:172.21.15.183:6789/0 conn(0x7f0ba00d27f0 legacy=0x7f0ba00d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:57.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.214+0000 7f0ba57de640 1 -- 172.21.15.83:0/3341517576 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba00b3710 con 0x7f0ba015dfe0 2024-03-06T07:28:57.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.214+0000 7f0ba4fdd640 1 --2- 172.21.15.83:0/3341517576 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0ba00cf830 0x7f0ba00d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:57.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.214+0000 7f0ba57de640 1 --2- 172.21.15.83:0/3341517576 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ba015dfe0 0x7f0ba00cf2f0 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c009870 tx=0x7f0b9c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5518b532bdded7a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.215+0000 7f0b97fff640 1 -- 172.21.15.83:0/3341517576 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b9c035020 con 0x7f0ba015dfe0 2024-03-06T07:28:57.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.215+0000 7f0b97fff640 1 -- 172.21.15.83:0/3341517576 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b9c031640 con 0x7f0ba015dfe0 2024-03-06T07:28:57.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.215+0000 7f0b97fff640 1 -- 172.21.15.83:0/3341517576 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b9c02d040 con 0x7f0ba015dfe0 2024-03-06T07:28:57.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.215+0000 7f0ba7a42640 1 -- 172.21.15.83:0/3341517576 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ba015dfe0 msgr2=0x7f0ba00cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.215+0000 7f0ba7a42640 1 --2- 172.21.15.83:0/3341517576 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ba015dfe0 0x7f0ba00cf2f0 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c009870 tx=0x7f0b9c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.215+0000 7f0ba7a42640 1 -- 172.21.15.83:0/3341517576 shutdown_connections 2024-03-06T07:28:57.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.215+0000 7f0ba7a42640 1 --2- 172.21.15.83:0/3341517576 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0ba00cf830 0x7f0ba00d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.215+0000 7f0ba7a42640 1 --2- 172.21.15.83:0/3341517576 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ba015dfe0 0x7f0ba00cf2f0 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.215+0000 7f0ba7a42640 1 -- 172.21.15.83:0/3341517576 >> 172.21.15.83:0/3341517576 conn(0x7f0ba00d3cd0 msgr2=0x7f0ba00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:57.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.215+0000 7f0ba7a42640 1 -- 172.21.15.83:0/3341517576 shutdown_connections 2024-03-06T07:28:57.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.215+0000 7f0ba7a42640 1 -- 172.21.15.83:0/3341517576 wait complete. 2024-03-06T07:28:57.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.217+0000 7f0ba7a42640 1 Processor -- start 2024-03-06T07:28:57.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.217+0000 7f0ba7a42640 1 -- start start 2024-03-06T07:28:57.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.218+0000 7f0ba7a42640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0ba00cf830 0x7f0ba0159970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.218+0000 7f0ba7a42640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ba00d27f0 0x7f0ba00cbb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.218+0000 7f0ba7a42640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0ba015dfe0 0x7f0ba00cc040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.218+0000 7f0ba7a42640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba00d1c70 con 0x7f0ba00d27f0 2024-03-06T07:28:57.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.218+0000 7f0ba7a42640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba00b3d60 con 0x7f0ba00cf830 2024-03-06T07:28:57.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.218+0000 7f0ba7a42640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba00d1df0 con 0x7f0ba015dfe0 2024-03-06T07:28:57.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.218+0000 7f0ba4fdd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ba00d27f0 0x7f0ba00cbb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.219+0000 7f0ba4fdd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ba00d27f0 0x7f0ba00cbb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52016/0 (socket says 172.21.15.83:52016) 2024-03-06T07:28:57.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.219+0000 7f0ba4fdd640 1 -- 172.21.15.83:0/2839376871 learned_addr learned my addr 172.21.15.83:0/2839376871 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:28:57.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.219+0000 7f0ba57de640 1 --2- 172.21.15.83:0/2839376871 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0ba00cf830 0x7f0ba0159970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.219+0000 7f0ba5fdf640 1 --2- 172.21.15.83:0/2839376871 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0ba015dfe0 0x7f0ba00cc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.219+0000 7f0ba4fdd640 1 -- 172.21.15.83:0/2839376871 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0ba015dfe0 msgr2=0x7f0ba00cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.219+0000 7f0ba4fdd640 1 --2- 172.21.15.83:0/2839376871 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0ba015dfe0 0x7f0ba00cc040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.219+0000 7f0ba4fdd640 1 -- 172.21.15.83:0/2839376871 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0ba00cf830 msgr2=0x7f0ba0159970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.219+0000 7f0ba4fdd640 1 --2- 172.21.15.83:0/2839376871 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0ba00cf830 0x7f0ba0159970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.219+0000 7f0ba4fdd640 1 -- 172.21.15.83:0/2839376871 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba01f6f80 con 0x7f0ba00d27f0 2024-03-06T07:28:57.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.219+0000 7f0ba4fdd640 1 --2- 172.21.15.83:0/2839376871 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ba00d27f0 0x7f0ba00cbb00 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f0b9000c7f0 tx=0x7f0b9000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.219+0000 7f0b95ffb640 1 -- 172.21.15.83:0/2839376871 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b90015070 con 0x7f0ba00d27f0 2024-03-06T07:28:57.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.219+0000 7f0b95ffb640 1 -- 172.21.15.83:0/2839376871 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b90007a30 con 0x7f0ba00d27f0 2024-03-06T07:28:57.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.219+0000 7f0ba7a42640 1 -- 172.21.15.83:0/2839376871 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ba00cc6e0 con 0x7f0ba00d27f0 2024-03-06T07:28:57.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.219+0000 7f0b95ffb640 1 -- 172.21.15.83:0/2839376871 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b90007e00 con 0x7f0ba00d27f0 2024-03-06T07:28:57.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.219+0000 7f0ba7a42640 1 -- 172.21.15.83:0/2839376871 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ba01f7500 con 0x7f0ba00d27f0 2024-03-06T07:28:57.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.221+0000 7f0ba7a42640 1 -- 172.21.15.83:0/2839376871 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ba004f330 con 0x7f0ba00d27f0 2024-03-06T07:28:57.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.221+0000 7f0b95ffb640 1 -- 172.21.15.83:0/2839376871 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0b90018070 con 0x7f0ba00d27f0 2024-03-06T07:28:57.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.221+0000 7f0b95ffb640 1 --2- 172.21.15.83:0/2839376871 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0b8406b500 0x7f0b8406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.221+0000 7f0b95ffb640 1 -- 172.21.15.83:0/2839376871 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0b9008a040 con 0x7f0ba00d27f0 2024-03-06T07:28:57.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.221+0000 7f0ba57de640 1 --2- 172.21.15.83:0/2839376871 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0b8406b500 0x7f0b8406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.222+0000 7f0ba57de640 1 --2- 172.21.15.83:0/2839376871 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0b8406b500 0x7f0b8406d9c0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c036000 tx=0x7f0b9c03c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.229+0000 7f0b95ffb640 1 -- 172.21.15.83:0/2839376871 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0b90058870 con 0x7f0ba00d27f0 2024-03-06T07:28:57.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.408+0000 7f0ba7a42640 1 -- 172.21.15.83:0/2839376871 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0ba00ce9c0 con 0x7f0b8406b500 2024-03-06T07:28:57.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.414+0000 7f0b95ffb640 1 -- 172.21.15.83:0/2839376871 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252483 (secure 0 0 0) 0x7f0ba00ce9c0 con 0x7f0b8406b500 2024-03-06T07:28:57.419 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:28:57.420 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-06T07:28:57.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.416+0000 7f0ba7a42640 1 -- 172.21.15.83:0/2839376871 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0b8406b500 msgr2=0x7f0b8406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.416+0000 7f0ba7a42640 1 --2- 172.21.15.83:0/2839376871 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0b8406b500 0x7f0b8406d9c0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c036000 tx=0x7f0b9c03c040 comp rx=0 tx=0).stop 2024-03-06T07:28:57.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.416+0000 7f0ba7a42640 1 -- 172.21.15.83:0/2839376871 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ba00d27f0 msgr2=0x7f0ba00cbb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.416+0000 7f0ba7a42640 1 --2- 172.21.15.83:0/2839376871 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ba00d27f0 0x7f0ba00cbb00 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f0b9000c7f0 tx=0x7f0b9000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.416+0000 7f0ba7a42640 1 -- 172.21.15.83:0/2839376871 shutdown_connections 2024-03-06T07:28:57.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.416+0000 7f0ba7a42640 1 --2- 172.21.15.83:0/2839376871 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0b8406b500 0x7f0b8406d9c0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.416+0000 7f0ba7a42640 1 --2- 172.21.15.83:0/2839376871 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0ba015dfe0 0x7f0ba00cc040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.416+0000 7f0ba7a42640 1 --2- 172.21.15.83:0/2839376871 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ba00d27f0 0x7f0ba00cbb00 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.416+0000 7f0ba7a42640 1 --2- 172.21.15.83:0/2839376871 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0ba00cf830 0x7f0ba0159970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.416+0000 7f0ba7a42640 1 -- 172.21.15.83:0/2839376871 >> 172.21.15.83:0/2839376871 conn(0x7f0ba00d3cd0 msgr2=0x7f0ba00b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:57.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.417+0000 7f0ba7a42640 1 -- 172.21.15.83:0/2839376871 shutdown_connections 2024-03-06T07:28:57.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:28:57.417+0000 7f0ba7a42640 1 -- 172.21.15.83:0/2839376871 wait complete. 2024-03-06T07:28:57.430 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":337,"stamp":"2024-03-06T07:28:56.395815+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":329716,"kb_used_data":13704,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754645004,"statfs":{"total":773094113280,"available":772756484096,"internally_reserved":0,"allocated":14032896,"data_stored":6755780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12757,"internal_metadata":323472939},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.456200"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144920+0000","last_change":"2024-03-06T07:28:52.975574+0000","last_active":"2024-03-06T07:28:54.144920+0000","last_peered":"2024-03-06T07:28:54.144920+0000","last_clean":"2024-03-06T07:28:54.144920+0000","last_became_active":"2024-03-06T07:28:52.975235+0000","last_became_peered":"2024-03-06T07:28:52.975235+0000","last_unstale":"2024-03-06T07:28:54.144920+0000","last_undegraded":"2024-03-06T07:28:54.144920+0000","last_fullsized":"2024-03-06T07:28:54.144920+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:05:49.993454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.954337+0000","last_change":"2024-03-06T07:28:52.957309+0000","last_active":"2024-03-06T07:28:53.954337+0000","last_peered":"2024-03-06T07:28:53.954337+0000","last_clean":"2024-03-06T07:28:53.954337+0000","last_became_active":"2024-03-06T07:28:52.957035+0000","last_became_peered":"2024-03-06T07:28:52.957035+0000","last_unstale":"2024-03-06T07:28:53.954337+0000","last_undegraded":"2024-03-06T07:28:53.954337+0000","last_fullsized":"2024-03-06T07:28:53.954337+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:07.633879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.959053+0000","last_change":"2024-03-06T07:28:52.962280+0000","last_active":"2024-03-06T07:28:53.959053+0000","last_peered":"2024-03-06T07:28:53.959053+0000","last_clean":"2024-03-06T07:28:53.959053+0000","last_became_active":"2024-03-06T07:28:52.961976+0000","last_became_peered":"2024-03-06T07:28:52.961976+0000","last_unstale":"2024-03-06T07:28:53.959053+0000","last_undegraded":"2024-03-06T07:28:53.959053+0000","last_fullsized":"2024-03-06T07:28:53.959053+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:26.937939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144687+0000","last_change":"2024-03-06T07:28:52.975310+0000","last_active":"2024-03-06T07:28:54.144687+0000","last_peered":"2024-03-06T07:28:54.144687+0000","last_clean":"2024-03-06T07:28:54.144687+0000","last_became_active":"2024-03-06T07:28:52.975049+0000","last_became_peered":"2024-03-06T07:28:52.975049+0000","last_unstale":"2024-03-06T07:28:54.144687+0000","last_undegraded":"2024-03-06T07:28:54.144687+0000","last_fullsized":"2024-03-06T07:28:54.144687+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:19.865295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143537+0000","last_change":"2024-03-06T07:28:52.971518+0000","last_active":"2024-03-06T07:28:54.143537+0000","last_peered":"2024-03-06T07:28:54.143537+0000","last_clean":"2024-03-06T07:28:54.143537+0000","last_became_active":"2024-03-06T07:28:52.971206+0000","last_became_peered":"2024-03-06T07:28:52.971206+0000","last_unstale":"2024-03-06T07:28:54.143537+0000","last_undegraded":"2024-03-06T07:28:54.143537+0000","last_fullsized":"2024-03-06T07:28:54.143537+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:42:48.555932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.959724+0000","last_change":"2024-03-06T07:28:52.968239+0000","last_active":"2024-03-06T07:28:53.959724+0000","last_peered":"2024-03-06T07:28:53.959724+0000","last_clean":"2024-03-06T07:28:53.959724+0000","last_became_active":"2024-03-06T07:28:52.967906+0000","last_became_peered":"2024-03-06T07:28:52.967906+0000","last_unstale":"2024-03-06T07:28:53.959724+0000","last_undegraded":"2024-03-06T07:28:53.959724+0000","last_fullsized":"2024-03-06T07:28:53.959724+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:26.937939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.959459+0000","last_change":"2024-03-06T07:28:52.961471+0000","last_active":"2024-03-06T07:28:53.959459+0000","last_peered":"2024-03-06T07:28:53.959459+0000","last_clean":"2024-03-06T07:28:53.959459+0000","last_became_active":"2024-03-06T07:28:52.961189+0000","last_became_peered":"2024-03-06T07:28:52.961189+0000","last_unstale":"2024-03-06T07:28:53.959459+0000","last_undegraded":"2024-03-06T07:28:53.959459+0000","last_fullsized":"2024-03-06T07:28:53.959459+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:38:01.777138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.958700+0000","last_change":"2024-03-06T07:28:52.967625+0000","last_active":"2024-03-06T07:28:53.958700+0000","last_peered":"2024-03-06T07:28:53.958700+0000","last_clean":"2024-03-06T07:28:53.958700+0000","last_became_active":"2024-03-06T07:28:52.966298+0000","last_became_peered":"2024-03-06T07:28:52.966298+0000","last_unstale":"2024-03-06T07:28:53.958700+0000","last_undegraded":"2024-03-06T07:28:53.958700+0000","last_fullsized":"2024-03-06T07:28:53.958700+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:22:48.600473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.958933+0000","last_change":"2024-03-06T07:28:52.971582+0000","last_active":"2024-03-06T07:28:53.958933+0000","last_peered":"2024-03-06T07:28:53.958933+0000","last_clean":"2024-03-06T07:28:53.958933+0000","last_became_active":"2024-03-06T07:28:52.971007+0000","last_became_peered":"2024-03-06T07:28:52.971007+0000","last_unstale":"2024-03-06T07:28:53.958933+0000","last_undegraded":"2024-03-06T07:28:53.958933+0000","last_fullsized":"2024-03-06T07:28:53.958933+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:59.012327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.954355+0000","last_change":"2024-03-06T07:28:52.959897+0000","last_active":"2024-03-06T07:28:53.954355+0000","last_peered":"2024-03-06T07:28:53.954355+0000","last_clean":"2024-03-06T07:28:53.954355+0000","last_became_active":"2024-03-06T07:28:52.959650+0000","last_became_peered":"2024-03-06T07:28:52.959650+0000","last_unstale":"2024-03-06T07:28:53.954355+0000","last_undegraded":"2024-03-06T07:28:53.954355+0000","last_fullsized":"2024-03-06T07:28:53.954355+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:11.992255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143285+0000","last_change":"2024-03-06T07:28:52.969085+0000","last_active":"2024-03-06T07:28:54.143285+0000","last_peered":"2024-03-06T07:28:54.143285+0000","last_clean":"2024-03-06T07:28:54.143285+0000","last_became_active":"2024-03-06T07:28:52.968764+0000","last_became_peered":"2024-03-06T07:28:52.968764+0000","last_unstale":"2024-03-06T07:28:54.143285+0000","last_undegraded":"2024-03-06T07:28:54.143285+0000","last_fullsized":"2024-03-06T07:28:54.143285+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:03.772295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.958428+0000","last_change":"2024-03-06T07:28:52.971904+0000","last_active":"2024-03-06T07:28:53.958428+0000","last_peered":"2024-03-06T07:28:53.958428+0000","last_clean":"2024-03-06T07:28:53.958428+0000","last_became_active":"2024-03-06T07:28:52.971206+0000","last_became_peered":"2024-03-06T07:28:52.971206+0000","last_unstale":"2024-03-06T07:28:53.958428+0000","last_undegraded":"2024-03-06T07:28:53.958428+0000","last_fullsized":"2024-03-06T07:28:53.958428+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:34.971658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144568+0000","last_change":"2024-03-06T07:28:52.966612+0000","last_active":"2024-03-06T07:28:54.144568+0000","last_peered":"2024-03-06T07:28:54.144568+0000","last_clean":"2024-03-06T07:28:54.144568+0000","last_became_active":"2024-03-06T07:28:52.966335+0000","last_became_peered":"2024-03-06T07:28:52.966335+0000","last_unstale":"2024-03-06T07:28:54.144568+0000","last_undegraded":"2024-03-06T07:28:54.144568+0000","last_fullsized":"2024-03-06T07:28:54.144568+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:05.116368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.954143+0000","last_change":"2024-03-06T07:28:52.973834+0000","last_active":"2024-03-06T07:28:53.954143+0000","last_peered":"2024-03-06T07:28:53.954143+0000","last_clean":"2024-03-06T07:28:53.954143+0000","last_became_active":"2024-03-06T07:28:52.973562+0000","last_became_peered":"2024-03-06T07:28:52.973562+0000","last_unstale":"2024-03-06T07:28:53.954143+0000","last_undegraded":"2024-03-06T07:28:53.954143+0000","last_fullsized":"2024-03-06T07:28:53.954143+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:05:49.993454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144309+0000","last_change":"2024-03-06T07:28:52.972039+0000","last_active":"2024-03-06T07:28:54.144309+0000","last_peered":"2024-03-06T07:28:54.144309+0000","last_clean":"2024-03-06T07:28:54.144309+0000","last_became_active":"2024-03-06T07:28:52.971242+0000","last_became_peered":"2024-03-06T07:28:52.971242+0000","last_unstale":"2024-03-06T07:28:54.144309+0000","last_undegraded":"2024-03-06T07:28:54.144309+0000","last_fullsized":"2024-03-06T07:28:54.144309+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:08:59.523654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144014+0000","last_change":"2024-03-06T07:28:52.964715+0000","last_active":"2024-03-06T07:28:54.144014+0000","last_peered":"2024-03-06T07:28:54.144014+0000","last_clean":"2024-03-06T07:28:54.144014+0000","last_became_active":"2024-03-06T07:28:52.964122+0000","last_became_peered":"2024-03-06T07:28:52.964122+0000","last_unstale":"2024-03-06T07:28:54.144014+0000","last_undegraded":"2024-03-06T07:28:54.144014+0000","last_fullsized":"2024-03-06T07:28:54.144014+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:28:34.045789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.959741+0000","last_change":"2024-03-06T07:28:52.960814+0000","last_active":"2024-03-06T07:28:53.959741+0000","last_peered":"2024-03-06T07:28:53.959741+0000","last_clean":"2024-03-06T07:28:53.959741+0000","last_became_active":"2024-03-06T07:28:52.960503+0000","last_became_peered":"2024-03-06T07:28:52.960503+0000","last_unstale":"2024-03-06T07:28:53.959741+0000","last_undegraded":"2024-03-06T07:28:53.959741+0000","last_fullsized":"2024-03-06T07:28:53.959741+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:57.517691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143657+0000","last_change":"2024-03-06T07:28:52.967540+0000","last_active":"2024-03-06T07:28:54.143657+0000","last_peered":"2024-03-06T07:28:54.143657+0000","last_clean":"2024-03-06T07:28:54.143657+0000","last_became_active":"2024-03-06T07:28:52.967117+0000","last_became_peered":"2024-03-06T07:28:52.967117+0000","last_unstale":"2024-03-06T07:28:54.143657+0000","last_undegraded":"2024-03-06T07:28:54.143657+0000","last_fullsized":"2024-03-06T07:28:54.143657+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:21.003630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143443+0000","last_change":"2024-03-06T07:28:52.964819+0000","last_active":"2024-03-06T07:28:54.143443+0000","last_peered":"2024-03-06T07:28:54.143443+0000","last_clean":"2024-03-06T07:28:54.143443+0000","last_became_active":"2024-03-06T07:28:52.964364+0000","last_became_peered":"2024-03-06T07:28:52.964364+0000","last_unstale":"2024-03-06T07:28:54.143443+0000","last_undegraded":"2024-03-06T07:28:54.143443+0000","last_fullsized":"2024-03-06T07:28:54.143443+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:39:07.988272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144602+0000","last_change":"2024-03-06T07:28:52.964742+0000","last_active":"2024-03-06T07:28:54.144602+0000","last_peered":"2024-03-06T07:28:54.144602+0000","last_clean":"2024-03-06T07:28:54.144602+0000","last_became_active":"2024-03-06T07:28:52.964468+0000","last_became_peered":"2024-03-06T07:28:52.964468+0000","last_unstale":"2024-03-06T07:28:54.144602+0000","last_undegraded":"2024-03-06T07:28:54.144602+0000","last_fullsized":"2024-03-06T07:28:54.144602+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:56:12.710010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143383+0000","last_change":"2024-03-06T07:28:52.970726+0000","last_active":"2024-03-06T07:28:54.143383+0000","last_peered":"2024-03-06T07:28:54.143383+0000","last_clean":"2024-03-06T07:28:54.143383+0000","last_became_active":"2024-03-06T07:28:52.970460+0000","last_became_peered":"2024-03-06T07:28:52.970460+0000","last_unstale":"2024-03-06T07:28:54.143383+0000","last_undegraded":"2024-03-06T07:28:54.143383+0000","last_fullsized":"2024-03-06T07:28:54.143383+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:06:57.274089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.142927+0000","last_change":"2024-03-06T07:28:52.959645+0000","last_active":"2024-03-06T07:28:54.142927+0000","last_peered":"2024-03-06T07:28:54.142927+0000","last_clean":"2024-03-06T07:28:54.142927+0000","last_became_active":"2024-03-06T07:28:52.959385+0000","last_became_peered":"2024-03-06T07:28:52.959385+0000","last_unstale":"2024-03-06T07:28:54.142927+0000","last_undegraded":"2024-03-06T07:28:54.142927+0000","last_fullsized":"2024-03-06T07:28:54.142927+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:46:55.628556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.958843+0000","last_change":"2024-03-06T07:28:52.965547+0000","last_active":"2024-03-06T07:28:53.958843+0000","last_peered":"2024-03-06T07:28:53.958843+0000","last_clean":"2024-03-06T07:28:53.958843+0000","last_became_active":"2024-03-06T07:28:52.964474+0000","last_became_peered":"2024-03-06T07:28:52.964474+0000","last_unstale":"2024-03-06T07:28:53.958843+0000","last_undegraded":"2024-03-06T07:28:53.958843+0000","last_fullsized":"2024-03-06T07:28:53.958843+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:24.376865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.142661+0000","last_change":"2024-03-06T07:28:52.963889+0000","last_active":"2024-03-06T07:28:54.142661+0000","last_peered":"2024-03-06T07:28:54.142661+0000","last_clean":"2024-03-06T07:28:54.142661+0000","last_became_active":"2024-03-06T07:28:52.963532+0000","last_became_peered":"2024-03-06T07:28:52.963532+0000","last_unstale":"2024-03-06T07:28:54.142661+0000","last_undegraded":"2024-03-06T07:28:54.142661+0000","last_fullsized":"2024-03-06T07:28:54.142661+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:44.151751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.954355+0000","last_change":"2024-03-06T07:28:52.967292+0000","last_active":"2024-03-06T07:28:53.954355+0000","last_peered":"2024-03-06T07:28:53.954355+0000","last_clean":"2024-03-06T07:28:53.954355+0000","last_became_active":"2024-03-06T07:28:52.967026+0000","last_became_peered":"2024-03-06T07:28:52.967026+0000","last_unstale":"2024-03-06T07:28:53.954355+0000","last_undegraded":"2024-03-06T07:28:53.954355+0000","last_fullsized":"2024-03-06T07:28:53.954355+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:35.361294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143793+0000","last_change":"2024-03-06T07:28:52.967696+0000","last_active":"2024-03-06T07:28:54.143793+0000","last_peered":"2024-03-06T07:28:54.143793+0000","last_clean":"2024-03-06T07:28:54.143793+0000","last_became_active":"2024-03-06T07:28:52.967448+0000","last_became_peered":"2024-03-06T07:28:52.967448+0000","last_unstale":"2024-03-06T07:28:54.143793+0000","last_undegraded":"2024-03-06T07:28:54.143793+0000","last_fullsized":"2024-03-06T07:28:54.143793+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:03.904000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144499+0000","last_change":"2024-03-06T07:28:52.972562+0000","last_active":"2024-03-06T07:28:54.144499+0000","last_peered":"2024-03-06T07:28:54.144499+0000","last_clean":"2024-03-06T07:28:54.144499+0000","last_became_active":"2024-03-06T07:28:52.972297+0000","last_became_peered":"2024-03-06T07:28:52.972297+0000","last_unstale":"2024-03-06T07:28:54.144499+0000","last_undegraded":"2024-03-06T07:28:54.144499+0000","last_fullsized":"2024-03-06T07:28:54.144499+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:26.575165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143103+0000","last_change":"2024-03-06T07:28:52.970892+0000","last_active":"2024-03-06T07:28:54.143103+0000","last_peered":"2024-03-06T07:28:54.143103+0000","last_clean":"2024-03-06T07:28:54.143103+0000","last_became_active":"2024-03-06T07:28:52.970576+0000","last_became_peered":"2024-03-06T07:28:52.970576+0000","last_unstale":"2024-03-06T07:28:54.143103+0000","last_undegraded":"2024-03-06T07:28:54.143103+0000","last_fullsized":"2024-03-06T07:28:54.143103+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:38.981799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144695+0000","last_change":"2024-03-06T07:28:52.970302+0000","last_active":"2024-03-06T07:28:54.144695+0000","last_peered":"2024-03-06T07:28:54.144695+0000","last_clean":"2024-03-06T07:28:54.144695+0000","last_became_active":"2024-03-06T07:28:52.970051+0000","last_became_peered":"2024-03-06T07:28:52.970051+0000","last_unstale":"2024-03-06T07:28:54.144695+0000","last_undegraded":"2024-03-06T07:28:54.144695+0000","last_fullsized":"2024-03-06T07:28:54.144695+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:32.804823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144636+0000","last_change":"2024-03-06T07:28:52.973548+0000","last_active":"2024-03-06T07:28:54.144636+0000","last_peered":"2024-03-06T07:28:54.144636+0000","last_clean":"2024-03-06T07:28:54.144636+0000","last_became_active":"2024-03-06T07:28:52.973271+0000","last_became_peered":"2024-03-06T07:28:52.973271+0000","last_unstale":"2024-03-06T07:28:54.144636+0000","last_undegraded":"2024-03-06T07:28:54.144636+0000","last_fullsized":"2024-03-06T07:28:54.144636+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:41.645943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143244+0000","last_change":"2024-03-06T07:28:52.975572+0000","last_active":"2024-03-06T07:28:54.143244+0000","last_peered":"2024-03-06T07:28:54.143244+0000","last_clean":"2024-03-06T07:28:54.143244+0000","last_became_active":"2024-03-06T07:28:52.975203+0000","last_became_peered":"2024-03-06T07:28:52.975203+0000","last_unstale":"2024-03-06T07:28:54.143244+0000","last_undegraded":"2024-03-06T07:28:54.143244+0000","last_fullsized":"2024-03-06T07:28:54.143244+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:16.118770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.959565+0000","last_change":"2024-03-06T07:28:52.960804+0000","last_active":"2024-03-06T07:28:53.959565+0000","last_peered":"2024-03-06T07:28:53.959565+0000","last_clean":"2024-03-06T07:28:53.959565+0000","last_became_active":"2024-03-06T07:28:52.960557+0000","last_became_peered":"2024-03-06T07:28:52.960557+0000","last_unstale":"2024-03-06T07:28:53.959565+0000","last_undegraded":"2024-03-06T07:28:53.959565+0000","last_fullsized":"2024-03-06T07:28:53.959565+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:15.776204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143936+0000","last_change":"2024-03-06T07:28:52.970667+0000","last_active":"2024-03-06T07:28:54.143936+0000","last_peered":"2024-03-06T07:28:54.143936+0000","last_clean":"2024-03-06T07:28:54.143936+0000","last_became_active":"2024-03-06T07:28:52.970408+0000","last_became_peered":"2024-03-06T07:28:52.970408+0000","last_unstale":"2024-03-06T07:28:54.143936+0000","last_undegraded":"2024-03-06T07:28:54.143936+0000","last_fullsized":"2024-03-06T07:28:54.143936+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:04:38.898631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"127'4","reported_seq":18,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143651+0000","last_change":"2024-03-06T07:28:51.970811+0000","last_active":"2024-03-06T07:28:54.143651+0000","last_peered":"2024-03-06T07:28:54.143651+0000","last_clean":"2024-03-06T07:28:54.143651+0000","last_became_active":"2024-03-06T07:28:51.970564+0000","last_became_peered":"2024-03-06T07:28:51.970564+0000","last_unstale":"2024-03-06T07:28:54.143651+0000","last_undegraded":"2024-03-06T07:28:54.143651+0000","last_fullsized":"2024-03-06T07:28:54.143651+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:59.741625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.959001+0000","last_change":"2024-03-06T07:28:52.969220+0000","last_active":"2024-03-06T07:28:53.959001+0000","last_peered":"2024-03-06T07:28:53.959001+0000","last_clean":"2024-03-06T07:28:53.959001+0000","last_became_active":"2024-03-06T07:28:52.968789+0000","last_became_peered":"2024-03-06T07:28:52.968789+0000","last_unstale":"2024-03-06T07:28:53.959001+0000","last_undegraded":"2024-03-06T07:28:53.959001+0000","last_fullsized":"2024-03-06T07:28:53.959001+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:15.776204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"127'1","reported_seq":15,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143089+0000","last_change":"2024-03-06T07:28:51.964480+0000","last_active":"2024-03-06T07:28:54.143089+0000","last_peered":"2024-03-06T07:28:54.143089+0000","last_clean":"2024-03-06T07:28:54.143089+0000","last_became_active":"2024-03-06T07:28:51.964203+0000","last_became_peered":"2024-03-06T07:28:51.964203+0000","last_unstale":"2024-03-06T07:28:54.143089+0000","last_undegraded":"2024-03-06T07:28:54.143089+0000","last_fullsized":"2024-03-06T07:28:54.143089+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:14:32.674292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143485+0000","last_change":"2024-03-06T07:28:52.965670+0000","last_active":"2024-03-06T07:28:54.143485+0000","last_peered":"2024-03-06T07:28:54.143485+0000","last_clean":"2024-03-06T07:28:54.143485+0000","last_became_active":"2024-03-06T07:28:52.965401+0000","last_became_peered":"2024-03-06T07:28:52.965401+0000","last_unstale":"2024-03-06T07:28:54.143485+0000","last_undegraded":"2024-03-06T07:28:54.143485+0000","last_fullsized":"2024-03-06T07:28:54.143485+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:27:10.021510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"127'2","reported_seq":16,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143552+0000","last_change":"2024-03-06T07:28:51.966438+0000","last_active":"2024-03-06T07:28:54.143552+0000","last_peered":"2024-03-06T07:28:54.143552+0000","last_clean":"2024-03-06T07:28:54.143552+0000","last_became_active":"2024-03-06T07:28:51.966143+0000","last_became_peered":"2024-03-06T07:28:51.966143+0000","last_unstale":"2024-03-06T07:28:54.143552+0000","last_undegraded":"2024-03-06T07:28:54.143552+0000","last_fullsized":"2024-03-06T07:28:54.143552+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:15.111853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143227+0000","last_change":"2024-03-06T07:28:52.969430+0000","last_active":"2024-03-06T07:28:54.143227+0000","last_peered":"2024-03-06T07:28:54.143227+0000","last_clean":"2024-03-06T07:28:54.143227+0000","last_became_active":"2024-03-06T07:28:52.968429+0000","last_became_peered":"2024-03-06T07:28:52.968429+0000","last_unstale":"2024-03-06T07:28:54.143227+0000","last_undegraded":"2024-03-06T07:28:54.143227+0000","last_fullsized":"2024-03-06T07:28:54.143227+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:14.242154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"127'2","reported_seq":16,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.954144+0000","last_change":"2024-03-06T07:28:51.964506+0000","last_active":"2024-03-06T07:28:53.954144+0000","last_peered":"2024-03-06T07:28:53.954144+0000","last_clean":"2024-03-06T07:28:53.954144+0000","last_became_active":"2024-03-06T07:28:51.964192+0000","last_became_peered":"2024-03-06T07:28:51.964192+0000","last_unstale":"2024-03-06T07:28:53.954144+0000","last_undegraded":"2024-03-06T07:28:53.954144+0000","last_fullsized":"2024-03-06T07:28:53.954144+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:43:03.523499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143145+0000","last_change":"2024-03-06T07:28:52.968598+0000","last_active":"2024-03-06T07:28:54.143145+0000","last_peered":"2024-03-06T07:28:54.143145+0000","last_clean":"2024-03-06T07:28:54.143145+0000","last_became_active":"2024-03-06T07:28:52.968335+0000","last_became_peered":"2024-03-06T07:28:52.968335+0000","last_unstale":"2024-03-06T07:28:54.143145+0000","last_undegraded":"2024-03-06T07:28:54.143145+0000","last_fullsized":"2024-03-06T07:28:54.143145+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:17:51.856534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"127'1","reported_seq":15,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.954028+0000","last_change":"2024-03-06T07:28:51.966376+0000","last_active":"2024-03-06T07:28:53.954028+0000","last_peered":"2024-03-06T07:28:53.954028+0000","last_clean":"2024-03-06T07:28:53.954028+0000","last_became_active":"2024-03-06T07:28:51.966123+0000","last_became_peered":"2024-03-06T07:28:51.966123+0000","last_unstale":"2024-03-06T07:28:53.954028+0000","last_undegraded":"2024-03-06T07:28:53.954028+0000","last_fullsized":"2024-03-06T07:28:53.954028+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:38:45.555438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.959722+0000","last_change":"2024-03-06T07:28:52.957122+0000","last_active":"2024-03-06T07:28:53.959722+0000","last_peered":"2024-03-06T07:28:53.959722+0000","last_clean":"2024-03-06T07:28:53.959722+0000","last_became_active":"2024-03-06T07:28:52.956828+0000","last_became_peered":"2024-03-06T07:28:52.956828+0000","last_unstale":"2024-03-06T07:28:53.959722+0000","last_undegraded":"2024-03-06T07:28:53.959722+0000","last_fullsized":"2024-03-06T07:28:53.959722+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:42.642239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"127'1","reported_seq":15,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143244+0000","last_change":"2024-03-06T07:28:51.971434+0000","last_active":"2024-03-06T07:28:54.143244+0000","last_peered":"2024-03-06T07:28:54.143244+0000","last_clean":"2024-03-06T07:28:54.143244+0000","last_became_active":"2024-03-06T07:28:51.971151+0000","last_became_peered":"2024-03-06T07:28:51.971151+0000","last_unstale":"2024-03-06T07:28:54.143244+0000","last_undegraded":"2024-03-06T07:28:54.143244+0000","last_fullsized":"2024-03-06T07:28:54.143244+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:38:00.770221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.958957+0000","last_change":"2024-03-06T07:28:52.959003+0000","last_active":"2024-03-06T07:28:53.958957+0000","last_peered":"2024-03-06T07:28:53.958957+0000","last_clean":"2024-03-06T07:28:53.958957+0000","last_became_active":"2024-03-06T07:28:52.958733+0000","last_became_peered":"2024-03-06T07:28:52.958733+0000","last_unstale":"2024-03-06T07:28:53.958957+0000","last_undegraded":"2024-03-06T07:28:53.958957+0000","last_fullsized":"2024-03-06T07:28:53.958957+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:38:01.777138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.959461+0000","last_change":"2024-03-06T07:28:51.965660+0000","last_active":"2024-03-06T07:28:53.959461+0000","last_peered":"2024-03-06T07:28:53.959461+0000","last_clean":"2024-03-06T07:28:53.959461+0000","last_became_active":"2024-03-06T07:28:51.965337+0000","last_became_peered":"2024-03-06T07:28:51.965337+0000","last_unstale":"2024-03-06T07:28:53.959461+0000","last_undegraded":"2024-03-06T07:28:53.959461+0000","last_fullsized":"2024-03-06T07:28:53.959461+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:35:26.827210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.953967+0000","last_change":"2024-03-06T07:28:52.959392+0000","last_active":"2024-03-06T07:28:53.953967+0000","last_peered":"2024-03-06T07:28:53.953967+0000","last_clean":"2024-03-06T07:28:53.953967+0000","last_became_active":"2024-03-06T07:28:52.959152+0000","last_became_peered":"2024-03-06T07:28:52.959152+0000","last_unstale":"2024-03-06T07:28:53.953967+0000","last_undegraded":"2024-03-06T07:28:53.953967+0000","last_fullsized":"2024-03-06T07:28:53.953967+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:50.800577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.954024+0000","last_change":"2024-03-06T07:28:51.961669+0000","last_active":"2024-03-06T07:28:53.954024+0000","last_peered":"2024-03-06T07:28:53.954024+0000","last_clean":"2024-03-06T07:28:53.954024+0000","last_became_active":"2024-03-06T07:28:51.961367+0000","last_became_peered":"2024-03-06T07:28:51.961367+0000","last_unstale":"2024-03-06T07:28:53.954024+0000","last_undegraded":"2024-03-06T07:28:53.954024+0000","last_fullsized":"2024-03-06T07:28:53.954024+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:37.702067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.145049+0000","last_change":"2024-03-06T07:28:52.967292+0000","last_active":"2024-03-06T07:28:54.145049+0000","last_peered":"2024-03-06T07:28:54.145049+0000","last_clean":"2024-03-06T07:28:54.145049+0000","last_became_active":"2024-03-06T07:28:52.967049+0000","last_became_peered":"2024-03-06T07:28:52.967049+0000","last_unstale":"2024-03-06T07:28:54.145049+0000","last_undegraded":"2024-03-06T07:28:54.145049+0000","last_fullsized":"2024-03-06T07:28:54.145049+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:07.633879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143435+0000","last_change":"2024-03-06T07:28:51.963421+0000","last_active":"2024-03-06T07:28:54.143435+0000","last_peered":"2024-03-06T07:28:54.143435+0000","last_clean":"2024-03-06T07:28:54.143435+0000","last_became_active":"2024-03-06T07:28:51.963127+0000","last_became_peered":"2024-03-06T07:28:51.963127+0000","last_unstale":"2024-03-06T07:28:54.143435+0000","last_undegraded":"2024-03-06T07:28:54.143435+0000","last_fullsized":"2024-03-06T07:28:54.143435+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:05:48.986537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143245+0000","last_change":"2024-03-06T07:28:52.973735+0000","last_active":"2024-03-06T07:28:54.143245+0000","last_peered":"2024-03-06T07:28:54.143245+0000","last_clean":"2024-03-06T07:28:54.143245+0000","last_became_active":"2024-03-06T07:28:52.973385+0000","last_became_peered":"2024-03-06T07:28:52.973385+0000","last_unstale":"2024-03-06T07:28:54.143245+0000","last_undegraded":"2024-03-06T07:28:54.143245+0000","last_fullsized":"2024-03-06T07:28:54.143245+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:50:41.441039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144044+0000","last_change":"2024-03-06T07:28:51.970138+0000","last_active":"2024-03-06T07:28:54.144044+0000","last_peered":"2024-03-06T07:28:54.144044+0000","last_clean":"2024-03-06T07:28:54.144044+0000","last_became_active":"2024-03-06T07:28:51.969796+0000","last_became_peered":"2024-03-06T07:28:51.969796+0000","last_unstale":"2024-03-06T07:28:54.144044+0000","last_undegraded":"2024-03-06T07:28:54.144044+0000","last_fullsized":"2024-03-06T07:28:54.144044+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:58.005409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143650+0000","last_change":"2024-03-06T07:28:52.965088+0000","last_active":"2024-03-06T07:28:54.143650+0000","last_peered":"2024-03-06T07:28:54.143650+0000","last_clean":"2024-03-06T07:28:54.143650+0000","last_became_active":"2024-03-06T07:28:52.964815+0000","last_became_peered":"2024-03-06T07:28:52.964815+0000","last_unstale":"2024-03-06T07:28:54.143650+0000","last_undegraded":"2024-03-06T07:28:54.143650+0000","last_fullsized":"2024-03-06T07:28:54.143650+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:44.151751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"127'3","reported_seq":17,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.142845+0000","last_change":"2024-03-06T07:28:51.968492+0000","last_active":"2024-03-06T07:28:54.142845+0000","last_peered":"2024-03-06T07:28:54.142845+0000","last_clean":"2024-03-06T07:28:54.142845+0000","last_became_active":"2024-03-06T07:28:51.968190+0000","last_became_peered":"2024-03-06T07:28:51.968190+0000","last_unstale":"2024-03-06T07:28:54.142845+0000","last_undegraded":"2024-03-06T07:28:54.142845+0000","last_fullsized":"2024-03-06T07:28:54.142845+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:18.858377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143597+0000","last_change":"2024-03-06T07:28:52.964754+0000","last_active":"2024-03-06T07:28:54.143597+0000","last_peered":"2024-03-06T07:28:54.143597+0000","last_clean":"2024-03-06T07:28:54.143597+0000","last_became_active":"2024-03-06T07:28:52.964437+0000","last_became_peered":"2024-03-06T07:28:52.964437+0000","last_unstale":"2024-03-06T07:28:54.143597+0000","last_undegraded":"2024-03-06T07:28:54.143597+0000","last_fullsized":"2024-03-06T07:28:54.143597+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:53.934839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"127'6","reported_seq":20,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.958669+0000","last_change":"2024-03-06T07:28:51.966045+0000","last_active":"2024-03-06T07:28:53.958669+0000","last_peered":"2024-03-06T07:28:53.958669+0000","last_clean":"2024-03-06T07:28:53.958669+0000","last_became_active":"2024-03-06T07:28:51.965790+0000","last_became_peered":"2024-03-06T07:28:51.965790+0000","last_unstale":"2024-03-06T07:28:53.958669+0000","last_undegraded":"2024-03-06T07:28:53.958669+0000","last_fullsized":"2024-03-06T07:28:53.958669+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:06:54.088262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144821+0000","last_change":"2024-03-06T07:28:52.962473+0000","last_active":"2024-03-06T07:28:54.144821+0000","last_peered":"2024-03-06T07:28:54.144821+0000","last_clean":"2024-03-06T07:28:54.144821+0000","last_became_active":"2024-03-06T07:28:52.961570+0000","last_became_peered":"2024-03-06T07:28:52.961570+0000","last_unstale":"2024-03-06T07:28:54.144821+0000","last_undegraded":"2024-03-06T07:28:54.144821+0000","last_fullsized":"2024-03-06T07:28:54.144821+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:50.800577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144881+0000","last_change":"2024-03-06T07:28:51.958418+0000","last_active":"2024-03-06T07:28:54.144881+0000","last_peered":"2024-03-06T07:28:54.144881+0000","last_clean":"2024-03-06T07:28:54.144881+0000","last_became_active":"2024-03-06T07:28:51.957944+0000","last_became_peered":"2024-03-06T07:28:51.957944+0000","last_unstale":"2024-03-06T07:28:54.144881+0000","last_undegraded":"2024-03-06T07:28:54.144881+0000","last_fullsized":"2024-03-06T07:28:54.144881+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:35.191066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.953924+0000","last_change":"2024-03-06T07:28:52.962686+0000","last_active":"2024-03-06T07:28:53.953924+0000","last_peered":"2024-03-06T07:28:53.953924+0000","last_clean":"2024-03-06T07:28:53.953924+0000","last_became_active":"2024-03-06T07:28:52.962407+0000","last_became_peered":"2024-03-06T07:28:52.962407+0000","last_unstale":"2024-03-06T07:28:53.953924+0000","last_undegraded":"2024-03-06T07:28:53.953924+0000","last_fullsized":"2024-03-06T07:28:53.953924+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:19.865295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.954049+0000","last_change":"2024-03-06T07:28:51.969835+0000","last_active":"2024-03-06T07:28:53.954049+0000","last_peered":"2024-03-06T07:28:53.954049+0000","last_clean":"2024-03-06T07:28:53.954049+0000","last_became_active":"2024-03-06T07:28:51.969530+0000","last_became_peered":"2024-03-06T07:28:51.969530+0000","last_unstale":"2024-03-06T07:28:53.954049+0000","last_undegraded":"2024-03-06T07:28:53.954049+0000","last_fullsized":"2024-03-06T07:28:53.954049+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:35.191066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.958376+0000","last_change":"2024-03-06T07:28:52.965364+0000","last_active":"2024-03-06T07:28:53.958376+0000","last_peered":"2024-03-06T07:28:53.958376+0000","last_clean":"2024-03-06T07:28:53.958376+0000","last_became_active":"2024-03-06T07:28:52.964831+0000","last_became_peered":"2024-03-06T07:28:52.964831+0000","last_unstale":"2024-03-06T07:28:53.958376+0000","last_undegraded":"2024-03-06T07:28:53.958376+0000","last_fullsized":"2024-03-06T07:28:53.958376+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:18.486012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"127'8","reported_seq":22,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.958804+0000","last_change":"2024-03-06T07:28:51.968492+0000","last_active":"2024-03-06T07:28:53.958804+0000","last_peered":"2024-03-06T07:28:53.958804+0000","last_clean":"2024-03-06T07:28:53.958804+0000","last_became_active":"2024-03-06T07:28:51.968146+0000","last_became_peered":"2024-03-06T07:28:51.968146+0000","last_unstale":"2024-03-06T07:28:53.958804+0000","last_undegraded":"2024-03-06T07:28:53.958804+0000","last_fullsized":"2024-03-06T07:28:53.958804+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:37.974882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.958778+0000","last_change":"2024-03-06T07:28:52.971128+0000","last_active":"2024-03-06T07:28:53.958778+0000","last_peered":"2024-03-06T07:28:53.958778+0000","last_clean":"2024-03-06T07:28:53.958778+0000","last_became_active":"2024-03-06T07:28:52.970657+0000","last_became_peered":"2024-03-06T07:28:52.970657+0000","last_unstale":"2024-03-06T07:28:53.958778+0000","last_undegraded":"2024-03-06T07:28:53.958778+0000","last_fullsized":"2024-03-06T07:28:53.958778+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:48:26.069964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"127'1","reported_seq":15,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143472+0000","last_change":"2024-03-06T07:28:51.966032+0000","last_active":"2024-03-06T07:28:54.143472+0000","last_peered":"2024-03-06T07:28:54.143472+0000","last_clean":"2024-03-06T07:28:54.143472+0000","last_became_active":"2024-03-06T07:28:51.965778+0000","last_became_peered":"2024-03-06T07:28:51.965778+0000","last_unstale":"2024-03-06T07:28:54.143472+0000","last_undegraded":"2024-03-06T07:28:54.143472+0000","last_fullsized":"2024-03-06T07:28:54.143472+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:01.850038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143257+0000","last_change":"2024-03-06T07:28:52.970288+0000","last_active":"2024-03-06T07:28:54.143257+0000","last_peered":"2024-03-06T07:28:54.143257+0000","last_clean":"2024-03-06T07:28:54.143257+0000","last_became_active":"2024-03-06T07:28:52.969973+0000","last_became_peered":"2024-03-06T07:28:52.969973+0000","last_unstale":"2024-03-06T07:28:54.143257+0000","last_undegraded":"2024-03-06T07:28:54.143257+0000","last_fullsized":"2024-03-06T07:28:54.143257+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:20:25.751101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"127'2","reported_seq":16,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143047+0000","last_change":"2024-03-06T07:28:51.961696+0000","last_active":"2024-03-06T07:28:54.143047+0000","last_peered":"2024-03-06T07:28:54.143047+0000","last_clean":"2024-03-06T07:28:54.143047+0000","last_became_active":"2024-03-06T07:28:51.961441+0000","last_became_peered":"2024-03-06T07:28:51.961441+0000","last_unstale":"2024-03-06T07:28:54.143047+0000","last_undegraded":"2024-03-06T07:28:54.143047+0000","last_fullsized":"2024-03-06T07:28:54.143047+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:42.468283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144735+0000","last_change":"2024-03-06T07:28:52.967362+0000","last_active":"2024-03-06T07:28:54.144735+0000","last_peered":"2024-03-06T07:28:54.144735+0000","last_clean":"2024-03-06T07:28:54.144735+0000","last_became_active":"2024-03-06T07:28:52.967091+0000","last_became_peered":"2024-03-06T07:28:52.967091+0000","last_unstale":"2024-03-06T07:28:54.144735+0000","last_undegraded":"2024-03-06T07:28:54.144735+0000","last_fullsized":"2024-03-06T07:28:54.144735+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:31.346210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"127'1","reported_seq":15,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.959971+0000","last_change":"2024-03-06T07:28:51.966073+0000","last_active":"2024-03-06T07:28:53.959971+0000","last_peered":"2024-03-06T07:28:53.959971+0000","last_clean":"2024-03-06T07:28:53.959971+0000","last_became_active":"2024-03-06T07:28:51.965786+0000","last_became_peered":"2024-03-06T07:28:51.965786+0000","last_unstale":"2024-03-06T07:28:53.959971+0000","last_undegraded":"2024-03-06T07:28:53.959971+0000","last_fullsized":"2024-03-06T07:28:53.959971+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:31:24.030877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.954365+0000","last_change":"2024-03-06T07:28:52.963231+0000","last_active":"2024-03-06T07:28:53.954365+0000","last_peered":"2024-03-06T07:28:53.954365+0000","last_clean":"2024-03-06T07:28:53.954365+0000","last_became_active":"2024-03-06T07:28:52.962463+0000","last_became_peered":"2024-03-06T07:28:52.962463+0000","last_unstale":"2024-03-06T07:28:53.954365+0000","last_undegraded":"2024-03-06T07:28:53.954365+0000","last_fullsized":"2024-03-06T07:28:53.954365+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:31.450403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143020+0000","last_change":"2024-03-06T07:28:51.964810+0000","last_active":"2024-03-06T07:28:54.143020+0000","last_peered":"2024-03-06T07:28:54.143020+0000","last_clean":"2024-03-06T07:28:54.143020+0000","last_became_active":"2024-03-06T07:28:51.964498+0000","last_became_peered":"2024-03-06T07:28:51.964498+0000","last_unstale":"2024-03-06T07:28:54.143020+0000","last_undegraded":"2024-03-06T07:28:54.143020+0000","last_fullsized":"2024-03-06T07:28:54.143020+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:31.797905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144772+0000","last_change":"2024-03-06T07:28:52.975907+0000","last_active":"2024-03-06T07:28:54.144772+0000","last_peered":"2024-03-06T07:28:54.144772+0000","last_clean":"2024-03-06T07:28:54.144772+0000","last_became_active":"2024-03-06T07:28:52.975063+0000","last_became_peered":"2024-03-06T07:28:52.975063+0000","last_unstale":"2024-03-06T07:28:54.144772+0000","last_undegraded":"2024-03-06T07:28:54.144772+0000","last_fullsized":"2024-03-06T07:28:54.144772+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:14:33.681210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"127'2","reported_seq":16,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143077+0000","last_change":"2024-03-06T07:28:51.963111+0000","last_active":"2024-03-06T07:28:54.143077+0000","last_peered":"2024-03-06T07:28:54.143077+0000","last_clean":"2024-03-06T07:28:54.143077+0000","last_became_active":"2024-03-06T07:28:51.962804+0000","last_became_peered":"2024-03-06T07:28:51.962804+0000","last_unstale":"2024-03-06T07:28:54.143077+0000","last_undegraded":"2024-03-06T07:28:54.143077+0000","last_fullsized":"2024-03-06T07:28:54.143077+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:49.793659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.959862+0000","last_change":"2024-03-06T07:28:52.963806+0000","last_active":"2024-03-06T07:28:53.959862+0000","last_peered":"2024-03-06T07:28:53.959862+0000","last_clean":"2024-03-06T07:28:53.959862+0000","last_became_active":"2024-03-06T07:28:52.963507+0000","last_became_peered":"2024-03-06T07:28:52.963507+0000","last_unstale":"2024-03-06T07:28:53.959862+0000","last_undegraded":"2024-03-06T07:28:53.959862+0000","last_fullsized":"2024-03-06T07:28:53.959862+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:54:25.798793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143332+0000","last_change":"2024-03-06T07:28:51.968470+0000","last_active":"2024-03-06T07:28:54.143332+0000","last_peered":"2024-03-06T07:28:54.143332+0000","last_clean":"2024-03-06T07:28:54.143332+0000","last_became_active":"2024-03-06T07:28:51.968177+0000","last_became_peered":"2024-03-06T07:28:51.968177+0000","last_unstale":"2024-03-06T07:28:54.143332+0000","last_undegraded":"2024-03-06T07:28:54.143332+0000","last_fullsized":"2024-03-06T07:28:54.143332+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:48:25.063046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.953928+0000","last_change":"2024-03-06T07:28:52.958145+0000","last_active":"2024-03-06T07:28:53.953928+0000","last_peered":"2024-03-06T07:28:53.953928+0000","last_clean":"2024-03-06T07:28:53.953928+0000","last_became_active":"2024-03-06T07:28:52.957842+0000","last_became_peered":"2024-03-06T07:28:52.957842+0000","last_unstale":"2024-03-06T07:28:53.953928+0000","last_undegraded":"2024-03-06T07:28:53.953928+0000","last_fullsized":"2024-03-06T07:28:53.953928+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:32.804823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.142893+0000","last_change":"2024-03-06T07:28:51.968301+0000","last_active":"2024-03-06T07:28:54.142893+0000","last_peered":"2024-03-06T07:28:54.142893+0000","last_clean":"2024-03-06T07:28:54.142893+0000","last_became_active":"2024-03-06T07:28:51.968026+0000","last_became_peered":"2024-03-06T07:28:51.968026+0000","last_unstale":"2024-03-06T07:28:54.142893+0000","last_undegraded":"2024-03-06T07:28:54.142893+0000","last_fullsized":"2024-03-06T07:28:54.142893+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:23.369947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143009+0000","last_change":"2024-03-06T07:28:52.972374+0000","last_active":"2024-03-06T07:28:54.143009+0000","last_peered":"2024-03-06T07:28:54.143009+0000","last_clean":"2024-03-06T07:28:54.143009+0000","last_became_active":"2024-03-06T07:28:52.972101+0000","last_became_peered":"2024-03-06T07:28:52.972101+0000","last_unstale":"2024-03-06T07:28:54.143009+0000","last_undegraded":"2024-03-06T07:28:54.143009+0000","last_fullsized":"2024-03-06T07:28:54.143009+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:41.237344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.142599+0000","last_change":"2024-03-06T07:28:51.960935+0000","last_active":"2024-03-06T07:28:54.142599+0000","last_peered":"2024-03-06T07:28:54.142599+0000","last_clean":"2024-03-06T07:28:54.142599+0000","last_became_active":"2024-03-06T07:28:51.960458+0000","last_became_peered":"2024-03-06T07:28:51.960458+0000","last_unstale":"2024-03-06T07:28:54.142599+0000","last_undegraded":"2024-03-06T07:28:54.142599+0000","last_fullsized":"2024-03-06T07:28:54.142599+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:40.230426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.959956+0000","last_change":"2024-03-06T07:28:52.969742+0000","last_active":"2024-03-06T07:28:53.959956+0000","last_peered":"2024-03-06T07:28:53.959956+0000","last_clean":"2024-03-06T07:28:53.959956+0000","last_became_active":"2024-03-06T07:28:52.969401+0000","last_became_peered":"2024-03-06T07:28:52.969401+0000","last_unstale":"2024-03-06T07:28:53.959956+0000","last_undegraded":"2024-03-06T07:28:53.959956+0000","last_fullsized":"2024-03-06T07:28:53.959956+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:08.434943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"127'2","reported_seq":16,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.145058+0000","last_change":"2024-03-06T07:28:51.959525+0000","last_active":"2024-03-06T07:28:54.145058+0000","last_peered":"2024-03-06T07:28:54.145058+0000","last_clean":"2024-03-06T07:28:54.145058+0000","last_became_active":"2024-03-06T07:28:51.959247+0000","last_became_peered":"2024-03-06T07:28:51.959247+0000","last_unstale":"2024-03-06T07:28:54.145058+0000","last_undegraded":"2024-03-06T07:28:54.145058+0000","last_fullsized":"2024-03-06T07:28:54.145058+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:37.278665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.145016+0000","last_change":"2024-03-06T07:28:52.972711+0000","last_active":"2024-03-06T07:28:54.145016+0000","last_peered":"2024-03-06T07:28:54.145016+0000","last_clean":"2024-03-06T07:28:54.145016+0000","last_became_active":"2024-03-06T07:28:52.972448+0000","last_became_peered":"2024-03-06T07:28:52.972448+0000","last_unstale":"2024-03-06T07:28:54.145016+0000","last_undegraded":"2024-03-06T07:28:54.145016+0000","last_fullsized":"2024-03-06T07:28:54.145016+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:02.856955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"128'1","reported_seq":15,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.963947+0000","last_change":"2024-03-06T07:28:51.969705+0000","last_active":"2024-03-06T07:28:54.963947+0000","last_peered":"2024-03-06T07:28:54.963947+0000","last_clean":"2024-03-06T07:28:54.963947+0000","last_became_active":"2024-03-06T07:28:51.969395+0000","last_became_peered":"2024-03-06T07:28:51.969395+0000","last_unstale":"2024-03-06T07:28:54.963947+0000","last_undegraded":"2024-03-06T07:28:54.963947+0000","last_fullsized":"2024-03-06T07:28:54.963947+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:59:39.689093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.954215+0000","last_change":"2024-03-06T07:28:52.963942+0000","last_active":"2024-03-06T07:28:53.954215+0000","last_peered":"2024-03-06T07:28:53.954215+0000","last_clean":"2024-03-06T07:28:53.954215+0000","last_became_active":"2024-03-06T07:28:52.963685+0000","last_became_peered":"2024-03-06T07:28:52.963685+0000","last_unstale":"2024-03-06T07:28:53.954215+0000","last_undegraded":"2024-03-06T07:28:53.954215+0000","last_fullsized":"2024-03-06T07:28:53.954215+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:02.856955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.954356+0000","last_change":"2024-03-06T07:28:51.960696+0000","last_active":"2024-03-06T07:28:53.954356+0000","last_peered":"2024-03-06T07:28:53.954356+0000","last_clean":"2024-03-06T07:28:53.954356+0000","last_became_active":"2024-03-06T07:28:51.960438+0000","last_became_peered":"2024-03-06T07:28:51.960438+0000","last_unstale":"2024-03-06T07:28:53.954356+0000","last_undegraded":"2024-03-06T07:28:53.954356+0000","last_fullsized":"2024-03-06T07:28:53.954356+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:14.232718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.959925+0000","last_change":"2024-03-06T07:28:52.967435+0000","last_active":"2024-03-06T07:28:53.959925+0000","last_peered":"2024-03-06T07:28:53.959925+0000","last_clean":"2024-03-06T07:28:53.959925+0000","last_became_active":"2024-03-06T07:28:52.967158+0000","last_became_peered":"2024-03-06T07:28:52.967158+0000","last_unstale":"2024-03-06T07:28:53.959925+0000","last_undegraded":"2024-03-06T07:28:53.959925+0000","last_fullsized":"2024-03-06T07:28:53.959925+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:59:40.696010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143231+0000","last_change":"2024-03-06T07:28:51.963760+0000","last_active":"2024-03-06T07:28:54.143231+0000","last_peered":"2024-03-06T07:28:54.143231+0000","last_clean":"2024-03-06T07:28:54.143231+0000","last_became_active":"2024-03-06T07:28:51.963374+0000","last_became_peered":"2024-03-06T07:28:51.963374+0000","last_unstale":"2024-03-06T07:28:54.143231+0000","last_undegraded":"2024-03-06T07:28:54.143231+0000","last_fullsized":"2024-03-06T07:28:54.143231+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:13.235236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143242+0000","last_change":"2024-03-06T07:28:52.968583+0000","last_active":"2024-03-06T07:28:54.143242+0000","last_peered":"2024-03-06T07:28:54.143242+0000","last_clean":"2024-03-06T07:28:54.143242+0000","last_became_active":"2024-03-06T07:28:52.968266+0000","last_became_peered":"2024-03-06T07:28:52.968266+0000","last_unstale":"2024-03-06T07:28:54.143242+0000","last_undegraded":"2024-03-06T07:28:54.143242+0000","last_fullsized":"2024-03-06T07:28:54.143242+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:43.475200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.954120+0000","last_change":"2024-03-06T07:28:51.966274+0000","last_active":"2024-03-06T07:28:53.954120+0000","last_peered":"2024-03-06T07:28:53.954120+0000","last_clean":"2024-03-06T07:28:53.954120+0000","last_became_active":"2024-03-06T07:28:51.965795+0000","last_became_peered":"2024-03-06T07:28:51.965795+0000","last_unstale":"2024-03-06T07:28:53.954120+0000","last_undegraded":"2024-03-06T07:28:53.954120+0000","last_fullsized":"2024-03-06T07:28:53.954120+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:44:14.025940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143168+0000","last_change":"2024-03-06T07:28:52.978648+0000","last_active":"2024-03-06T07:28:54.143168+0000","last_peered":"2024-03-06T07:28:54.143168+0000","last_clean":"2024-03-06T07:28:54.143168+0000","last_became_active":"2024-03-06T07:28:52.978331+0000","last_became_peered":"2024-03-06T07:28:52.978331+0000","last_unstale":"2024-03-06T07:28:54.143168+0000","last_undegraded":"2024-03-06T07:28:54.143168+0000","last_fullsized":"2024-03-06T07:28:54.143168+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:04.981862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143248+0000","last_change":"2024-03-06T07:28:51.971364+0000","last_active":"2024-03-06T07:28:54.143248+0000","last_peered":"2024-03-06T07:28:54.143248+0000","last_clean":"2024-03-06T07:28:54.143248+0000","last_became_active":"2024-03-06T07:28:51.971030+0000","last_became_peered":"2024-03-06T07:28:51.971030+0000","last_unstale":"2024-03-06T07:28:54.143248+0000","last_undegraded":"2024-03-06T07:28:54.143248+0000","last_fullsized":"2024-03-06T07:28:54.143248+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:07.428026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144153+0000","last_change":"2024-03-06T07:28:52.961576+0000","last_active":"2024-03-06T07:28:54.144153+0000","last_peered":"2024-03-06T07:28:54.144153+0000","last_clean":"2024-03-06T07:28:54.144153+0000","last_became_active":"2024-03-06T07:28:52.961325+0000","last_became_peered":"2024-03-06T07:28:52.961325+0000","last_unstale":"2024-03-06T07:28:54.144153+0000","last_undegraded":"2024-03-06T07:28:54.144153+0000","last_fullsized":"2024-03-06T07:28:54.144153+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:53.405336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.142987+0000","last_change":"2024-03-06T07:28:51.968672+0000","last_active":"2024-03-06T07:28:54.142987+0000","last_peered":"2024-03-06T07:28:54.142987+0000","last_clean":"2024-03-06T07:28:54.142987+0000","last_became_active":"2024-03-06T07:28:51.968340+0000","last_became_peered":"2024-03-06T07:28:51.968340+0000","last_unstale":"2024-03-06T07:28:54.142987+0000","last_undegraded":"2024-03-06T07:28:54.142987+0000","last_fullsized":"2024-03-06T07:28:54.142987+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:22:47.593555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.953957+0000","last_change":"2024-03-06T07:28:52.966628+0000","last_active":"2024-03-06T07:28:53.953957+0000","last_peered":"2024-03-06T07:28:53.953957+0000","last_clean":"2024-03-06T07:28:53.953957+0000","last_became_active":"2024-03-06T07:28:52.966354+0000","last_became_peered":"2024-03-06T07:28:52.966354+0000","last_unstale":"2024-03-06T07:28:53.953957+0000","last_undegraded":"2024-03-06T07:28:53.953957+0000","last_fullsized":"2024-03-06T07:28:53.953957+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:14:33.681210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.142673+0000","last_change":"2024-03-06T07:28:51.960707+0000","last_active":"2024-03-06T07:28:54.142673+0000","last_peered":"2024-03-06T07:28:54.142673+0000","last_clean":"2024-03-06T07:28:54.142673+0000","last_became_active":"2024-03-06T07:28:51.960177+0000","last_became_peered":"2024-03-06T07:28:51.960177+0000","last_unstale":"2024-03-06T07:28:54.142673+0000","last_undegraded":"2024-03-06T07:28:54.142673+0000","last_fullsized":"2024-03-06T07:28:54.142673+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:41.094093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143122+0000","last_change":"2024-03-06T07:28:52.975661+0000","last_active":"2024-03-06T07:28:54.143122+0000","last_peered":"2024-03-06T07:28:54.143122+0000","last_clean":"2024-03-06T07:28:54.143122+0000","last_became_active":"2024-03-06T07:28:52.975355+0000","last_became_peered":"2024-03-06T07:28:52.975355+0000","last_unstale":"2024-03-06T07:28:54.143122+0000","last_undegraded":"2024-03-06T07:28:54.143122+0000","last_fullsized":"2024-03-06T07:28:54.143122+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:42.101011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"127'2","reported_seq":16,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143618+0000","last_change":"2024-03-06T07:28:51.970862+0000","last_active":"2024-03-06T07:28:54.143618+0000","last_peered":"2024-03-06T07:28:54.143618+0000","last_clean":"2024-03-06T07:28:54.143618+0000","last_became_active":"2024-03-06T07:28:51.970609+0000","last_became_peered":"2024-03-06T07:28:51.970609+0000","last_unstale":"2024-03-06T07:28:54.143618+0000","last_undegraded":"2024-03-06T07:28:54.143618+0000","last_fullsized":"2024-03-06T07:28:54.143618+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:40.486032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":364,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.142779+0000","last_change":"2024-03-06T07:21:47.553693+0000","last_active":"2024-03-06T07:28:54.142779+0000","last_peered":"2024-03-06T07:28:54.142779+0000","last_clean":"2024-03-06T07:28:54.142779+0000","last_became_active":"2024-03-06T07:21:47.552664+0000","last_became_peered":"2024-03-06T07:21:47.552664+0000","last_unstale":"2024-03-06T07:28:54.142779+0000","last_undegraded":"2024-03-06T07:28:54.142779+0000","last_fullsized":"2024-03-06T07:28:54.142779+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_clean_scrub_stamp":"2024-03-06T07:21:46.362028+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:02.916426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574939,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41664,"kb_used_data":2164,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330176,"statfs":{"total":96636764160,"available":96594100224,"internally_reserved":0,"allocated":2215936,"data_stored":1288896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":40434121},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574937,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41072,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330768,"statfs":{"total":96636764160,"available":96594706432,"internally_reserved":0,"allocated":1609728,"data_stored":697890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574938,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41068,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330772,"statfs":{"total":96636764160,"available":96594710528,"internally_reserved":0,"allocated":1605632,"data_stored":696390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574939,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41068,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330772,"statfs":{"total":96636764160,"available":96594710528,"internally_reserved":0,"allocated":1601536,"data_stored":696348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574938,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41068,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330772,"statfs":{"total":96636764160,"available":96594710528,"internally_reserved":0,"allocated":1605632,"data_stored":696438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574938,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41068,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330772,"statfs":{"total":96636764160,"available":96594710528,"internally_reserved":0,"allocated":1605632,"data_stored":696894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574939,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41060,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330780,"statfs":{"total":96636764160,"available":96594718720,"internally_reserved":0,"allocated":1597440,"data_stored":696278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574938,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41648,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330192,"statfs":{"total":96636764160,"available":96594116608,"internally_reserved":0,"allocated":2191360,"data_stored":1286646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:29:00.167 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:00.169 DEBUG:teuthology.orchestra.run.smithi183:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:00.435 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:29:00.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.535+0000 7f4c5251e640 1 Processor -- start 2024-03-06T07:29:00.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.535+0000 7f4c5251e640 1 -- start start 2024-03-06T07:29:00.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.536+0000 7f4c5251e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f4c4c15dfe0 0x7f4c4c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.536+0000 7f4c5251e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4c4c0cf830 0x7f4c4c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.536+0000 7f4c5251e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c4c0b3430 con 0x7f4c4c0d27f0 2024-03-06T07:29:00.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.536+0000 7f4c5251e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c4c0b31e0 con 0x7f4c4c0cf830 2024-03-06T07:29:00.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.536+0000 7f4c5251e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c4c0b3d60 con 0x7f4c4c15dfe0 2024-03-06T07:29:00.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.536+0000 7f4c4b7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4c4c0cf830 0x7f4c4c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.536+0000 7f4c4b7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4c4c0cf830 0x7f4c4c0d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52026/0 (socket says 172.21.15.83:52026) 2024-03-06T07:29:00.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.536+0000 7f4c4b7fe640 1 -- 172.21.15.83:0/1255935147 learned_addr learned my addr 172.21.15.83:0/1255935147 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:00.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.536+0000 7f4c4b7fe640 1 -- 172.21.15.83:0/1255935147 >> v2:172.21.15.183:3300/0 conn(0x7f4c4c15dfe0 msgr2=0x7f4c4c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:00.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.536+0000 7f4c4b7fe640 1 --2- 172.21.15.83:0/1255935147 >> v2:172.21.15.183:3300/0 conn(0x7f4c4c15dfe0 0x7f4c4c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.536+0000 7f4c4b7fe640 1 -- 172.21.15.83:0/1255935147 >> v1:172.21.15.183:6789/0 conn(0x7f4c4c0d27f0 legacy=0x7f4c4c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:00.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.536+0000 7f4c4b7fe640 1 -- 172.21.15.83:0/1255935147 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c4c0b3710 con 0x7f4c4c0cf830 2024-03-06T07:29:00.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.537+0000 7f4c4b7fe640 1 --2- 172.21.15.83:0/1255935147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c4c0cf830 0x7f4c4c0d22b0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f4c40009870 tx=0x7f4c4002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ed685b8307178b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:00.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.537+0000 7f4c4affd640 1 -- 172.21.15.83:0/1255935147 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c40035020 con 0x7f4c4c0cf830 2024-03-06T07:29:00.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.537+0000 7f4c4affd640 1 -- 172.21.15.83:0/1255935147 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c40031870 con 0x7f4c4c0cf830 2024-03-06T07:29:00.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.537+0000 7f4c4affd640 1 -- 172.21.15.83:0/1255935147 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c40031c40 con 0x7f4c4c0cf830 2024-03-06T07:29:00.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.538+0000 7f4c5251e640 1 -- 172.21.15.83:0/1255935147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c4c0cf830 msgr2=0x7f4c4c0d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.538+0000 7f4c5251e640 1 --2- 172.21.15.83:0/1255935147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c4c0cf830 0x7f4c4c0d22b0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f4c40009870 tx=0x7f4c4002b910 comp rx=0 tx=0).stop 2024-03-06T07:29:00.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.538+0000 7f4c5251e640 1 -- 172.21.15.83:0/1255935147 shutdown_connections 2024-03-06T07:29:00.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.538+0000 7f4c5251e640 1 --2- 172.21.15.83:0/1255935147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c4c0cf830 0x7f4c4c0d22b0 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.538+0000 7f4c5251e640 1 --2- 172.21.15.83:0/1255935147 >> v2:172.21.15.183:3300/0 conn(0x7f4c4c15dfe0 0x7f4c4c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.538+0000 7f4c5251e640 1 -- 172.21.15.83:0/1255935147 >> 172.21.15.83:0/1255935147 conn(0x7f4c4c0d3cd0 msgr2=0x7f4c4c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:00.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.538+0000 7f4c5251e640 1 -- 172.21.15.83:0/1255935147 shutdown_connections 2024-03-06T07:29:00.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.538+0000 7f4c5251e640 1 -- 172.21.15.83:0/1255935147 wait complete. 2024-03-06T07:29:00.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.539+0000 7f4c5251e640 1 Processor -- start 2024-03-06T07:29:00.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.539+0000 7f4c5251e640 1 -- start start 2024-03-06T07:29:00.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.540+0000 7f4c5251e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c4c0cf830 0x7f4c4c14e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.540+0000 7f4c5251e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4c4c0d27f0 0x7f4c4c0b1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.540+0000 7f4c5251e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4c4c15dfe0 0x7f4c4c0b24b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.540+0000 7f4c5251e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4c0d1df0 con 0x7f4c4c0cf830 2024-03-06T07:29:00.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.540+0000 7f4c5251e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4c0d1c70 con 0x7f4c4c0d27f0 2024-03-06T07:29:00.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.540+0000 7f4c5251e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4c15e3c0 con 0x7f4c4c15dfe0 2024-03-06T07:29:00.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.540+0000 7f4c4bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c4c0cf830 0x7f4c4c14e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.540+0000 7f4c4bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c4c0cf830 0x7f4c4c14e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52032/0 (socket says 172.21.15.83:52032) 2024-03-06T07:29:00.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.540+0000 7f4c4bfff640 1 -- 172.21.15.83:0/3922436673 learned_addr learned my addr 172.21.15.83:0/3922436673 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:00.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.540+0000 7f4c4b7fe640 1 --2- 172.21.15.83:0/3922436673 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4c4c0d27f0 0x7f4c4c0b1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.540+0000 7f4c50abb640 1 --2- 172.21.15.83:0/3922436673 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4c4c15dfe0 0x7f4c4c0b24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.540+0000 7f4c4bfff640 1 -- 172.21.15.83:0/3922436673 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4c4c15dfe0 msgr2=0x7f4c4c0b24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.540+0000 7f4c4bfff640 1 --2- 172.21.15.83:0/3922436673 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4c4c15dfe0 0x7f4c4c0b24b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.540+0000 7f4c4bfff640 1 -- 172.21.15.83:0/3922436673 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4c4c0d27f0 msgr2=0x7f4c4c0b1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.540+0000 7f4c4bfff640 1 --2- 172.21.15.83:0/3922436673 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4c4c0d27f0 0x7f4c4c0b1f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.540+0000 7f4c4bfff640 1 -- 172.21.15.83:0/3922436673 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c4c14e7e0 con 0x7f4c4c0cf830 2024-03-06T07:29:00.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.541+0000 7f4c4bfff640 1 --2- 172.21.15.83:0/3922436673 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c4c0cf830 0x7f4c4c14e2a0 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c00b4a0 tx=0x7f4c3c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:00.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.541+0000 7f4c48ff9640 1 -- 172.21.15.83:0/3922436673 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c3c014070 con 0x7f4c4c0cf830 2024-03-06T07:29:00.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.541+0000 7f4c48ff9640 1 -- 172.21.15.83:0/3922436673 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c3c010610 con 0x7f4c4c0cf830 2024-03-06T07:29:00.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.541+0000 7f4c5251e640 1 -- 172.21.15.83:0/3922436673 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c4c15f1b0 con 0x7f4c4c0cf830 2024-03-06T07:29:00.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.541+0000 7f4c48ff9640 1 -- 172.21.15.83:0/3922436673 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c3c010a00 con 0x7f4c4c0cf830 2024-03-06T07:29:00.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.541+0000 7f4c5251e640 1 -- 172.21.15.83:0/3922436673 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c4c15f770 con 0x7f4c4c0cf830 2024-03-06T07:29:00.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.542+0000 7f4c5251e640 1 -- 172.21.15.83:0/3922436673 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c4c04f330 con 0x7f4c4c0cf830 2024-03-06T07:29:00.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.543+0000 7f4c48ff9640 1 -- 172.21.15.83:0/3922436673 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4c3c017070 con 0x7f4c4c0cf830 2024-03-06T07:29:00.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.543+0000 7f4c48ff9640 1 --2- 172.21.15.83:0/3922436673 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4c3806b4d0 0x7f4c3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.543+0000 7f4c48ff9640 1 -- 172.21.15.83:0/3922436673 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4c3c089b40 con 0x7f4c4c0cf830 2024-03-06T07:29:00.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.543+0000 7f4c4b7fe640 1 --2- 172.21.15.83:0/3922436673 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4c3806b4d0 0x7f4c3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.544+0000 7f4c4b7fe640 1 --2- 172.21.15.83:0/3922436673 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4c3806b4d0 0x7f4c3806d990 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f4c40036000 tx=0x7f4c4003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:00.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.548+0000 7f4c48ff9640 1 -- 172.21.15.83:0/3922436673 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4c3c058370 con 0x7f4c4c0cf830 2024-03-06T07:29:00.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.720+0000 7f4c5251e640 1 -- 172.21.15.83:0/3922436673 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4c4c0b2f60 con 0x7f4c3806b4d0 2024-03-06T07:29:00.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.726+0000 7f4c48ff9640 1 -- 172.21.15.83:0/3922436673 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252407 (secure 0 0 0) 0x7f4c4c0b2f60 con 0x7f4c3806b4d0 2024-03-06T07:29:00.731 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:29:00.733 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-06T07:29:00.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.730+0000 7f4c5251e640 1 -- 172.21.15.83:0/3922436673 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4c3806b4d0 msgr2=0x7f4c3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.730+0000 7f4c5251e640 1 --2- 172.21.15.83:0/3922436673 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4c3806b4d0 0x7f4c3806d990 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f4c40036000 tx=0x7f4c4003c040 comp rx=0 tx=0).stop 2024-03-06T07:29:00.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.730+0000 7f4c5251e640 1 -- 172.21.15.83:0/3922436673 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c4c0cf830 msgr2=0x7f4c4c14e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.730+0000 7f4c5251e640 1 --2- 172.21.15.83:0/3922436673 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c4c0cf830 0x7f4c4c14e2a0 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c00b4a0 tx=0x7f4c3c00b960 comp rx=0 tx=0).stop 2024-03-06T07:29:00.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.730+0000 7f4c5251e640 1 -- 172.21.15.83:0/3922436673 shutdown_connections 2024-03-06T07:29:00.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.730+0000 7f4c5251e640 1 --2- 172.21.15.83:0/3922436673 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4c3806b4d0 0x7f4c3806d990 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.730+0000 7f4c5251e640 1 --2- 172.21.15.83:0/3922436673 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4c4c15dfe0 0x7f4c4c0b24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.730+0000 7f4c5251e640 1 --2- 172.21.15.83:0/3922436673 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4c4c0d27f0 0x7f4c4c0b1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.730+0000 7f4c5251e640 1 --2- 172.21.15.83:0/3922436673 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c4c0cf830 0x7f4c4c14e2a0 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.730+0000 7f4c5251e640 1 -- 172.21.15.83:0/3922436673 >> 172.21.15.83:0/3922436673 conn(0x7f4c4c0d3cd0 msgr2=0x7f4c4c0cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:00.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.730+0000 7f4c5251e640 1 -- 172.21.15.83:0/3922436673 shutdown_connections 2024-03-06T07:29:00.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.730+0000 7f4c5251e640 1 -- 172.21.15.83:0/3922436673 wait complete. 2024-03-06T07:29:00.742 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":339,"stamp":"2024-03-06T07:29:00.396552+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":329716,"kb_used_data":13704,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754645004,"statfs":{"total":773094113280,"available":772756484096,"internally_reserved":0,"allocated":14032896,"data_stored":6755780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12757,"internal_metadata":323472939},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001332"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144920+0000","last_change":"2024-03-06T07:28:52.975574+0000","last_active":"2024-03-06T07:28:54.144920+0000","last_peered":"2024-03-06T07:28:54.144920+0000","last_clean":"2024-03-06T07:28:54.144920+0000","last_became_active":"2024-03-06T07:28:52.975235+0000","last_became_peered":"2024-03-06T07:28:52.975235+0000","last_unstale":"2024-03-06T07:28:54.144920+0000","last_undegraded":"2024-03-06T07:28:54.144920+0000","last_fullsized":"2024-03-06T07:28:54.144920+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:05:49.993454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.954337+0000","last_change":"2024-03-06T07:28:52.957309+0000","last_active":"2024-03-06T07:28:53.954337+0000","last_peered":"2024-03-06T07:28:53.954337+0000","last_clean":"2024-03-06T07:28:53.954337+0000","last_became_active":"2024-03-06T07:28:52.957035+0000","last_became_peered":"2024-03-06T07:28:52.957035+0000","last_unstale":"2024-03-06T07:28:53.954337+0000","last_undegraded":"2024-03-06T07:28:53.954337+0000","last_fullsized":"2024-03-06T07:28:53.954337+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:07.633879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.959053+0000","last_change":"2024-03-06T07:28:52.962280+0000","last_active":"2024-03-06T07:28:53.959053+0000","last_peered":"2024-03-06T07:28:53.959053+0000","last_clean":"2024-03-06T07:28:53.959053+0000","last_became_active":"2024-03-06T07:28:52.961976+0000","last_became_peered":"2024-03-06T07:28:52.961976+0000","last_unstale":"2024-03-06T07:28:53.959053+0000","last_undegraded":"2024-03-06T07:28:53.959053+0000","last_fullsized":"2024-03-06T07:28:53.959053+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:26.937939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144687+0000","last_change":"2024-03-06T07:28:52.975310+0000","last_active":"2024-03-06T07:28:54.144687+0000","last_peered":"2024-03-06T07:28:54.144687+0000","last_clean":"2024-03-06T07:28:54.144687+0000","last_became_active":"2024-03-06T07:28:52.975049+0000","last_became_peered":"2024-03-06T07:28:52.975049+0000","last_unstale":"2024-03-06T07:28:54.144687+0000","last_undegraded":"2024-03-06T07:28:54.144687+0000","last_fullsized":"2024-03-06T07:28:54.144687+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:19.865295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143537+0000","last_change":"2024-03-06T07:28:52.971518+0000","last_active":"2024-03-06T07:28:54.143537+0000","last_peered":"2024-03-06T07:28:54.143537+0000","last_clean":"2024-03-06T07:28:54.143537+0000","last_became_active":"2024-03-06T07:28:52.971206+0000","last_became_peered":"2024-03-06T07:28:52.971206+0000","last_unstale":"2024-03-06T07:28:54.143537+0000","last_undegraded":"2024-03-06T07:28:54.143537+0000","last_fullsized":"2024-03-06T07:28:54.143537+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:42:48.555932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.959724+0000","last_change":"2024-03-06T07:28:52.968239+0000","last_active":"2024-03-06T07:28:53.959724+0000","last_peered":"2024-03-06T07:28:53.959724+0000","last_clean":"2024-03-06T07:28:53.959724+0000","last_became_active":"2024-03-06T07:28:52.967906+0000","last_became_peered":"2024-03-06T07:28:52.967906+0000","last_unstale":"2024-03-06T07:28:53.959724+0000","last_undegraded":"2024-03-06T07:28:53.959724+0000","last_fullsized":"2024-03-06T07:28:53.959724+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:26.937939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.959459+0000","last_change":"2024-03-06T07:28:52.961471+0000","last_active":"2024-03-06T07:28:53.959459+0000","last_peered":"2024-03-06T07:28:53.959459+0000","last_clean":"2024-03-06T07:28:53.959459+0000","last_became_active":"2024-03-06T07:28:52.961189+0000","last_became_peered":"2024-03-06T07:28:52.961189+0000","last_unstale":"2024-03-06T07:28:53.959459+0000","last_undegraded":"2024-03-06T07:28:53.959459+0000","last_fullsized":"2024-03-06T07:28:53.959459+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:38:01.777138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.958700+0000","last_change":"2024-03-06T07:28:52.967625+0000","last_active":"2024-03-06T07:28:53.958700+0000","last_peered":"2024-03-06T07:28:53.958700+0000","last_clean":"2024-03-06T07:28:53.958700+0000","last_became_active":"2024-03-06T07:28:52.966298+0000","last_became_peered":"2024-03-06T07:28:52.966298+0000","last_unstale":"2024-03-06T07:28:53.958700+0000","last_undegraded":"2024-03-06T07:28:53.958700+0000","last_fullsized":"2024-03-06T07:28:53.958700+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:22:48.600473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.958933+0000","last_change":"2024-03-06T07:28:52.971582+0000","last_active":"2024-03-06T07:28:53.958933+0000","last_peered":"2024-03-06T07:28:53.958933+0000","last_clean":"2024-03-06T07:28:53.958933+0000","last_became_active":"2024-03-06T07:28:52.971007+0000","last_became_peered":"2024-03-06T07:28:52.971007+0000","last_unstale":"2024-03-06T07:28:53.958933+0000","last_undegraded":"2024-03-06T07:28:53.958933+0000","last_fullsized":"2024-03-06T07:28:53.958933+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:59.012327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.954355+0000","last_change":"2024-03-06T07:28:52.959897+0000","last_active":"2024-03-06T07:28:53.954355+0000","last_peered":"2024-03-06T07:28:53.954355+0000","last_clean":"2024-03-06T07:28:53.954355+0000","last_became_active":"2024-03-06T07:28:52.959650+0000","last_became_peered":"2024-03-06T07:28:52.959650+0000","last_unstale":"2024-03-06T07:28:53.954355+0000","last_undegraded":"2024-03-06T07:28:53.954355+0000","last_fullsized":"2024-03-06T07:28:53.954355+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:11.992255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143285+0000","last_change":"2024-03-06T07:28:52.969085+0000","last_active":"2024-03-06T07:28:54.143285+0000","last_peered":"2024-03-06T07:28:54.143285+0000","last_clean":"2024-03-06T07:28:54.143285+0000","last_became_active":"2024-03-06T07:28:52.968764+0000","last_became_peered":"2024-03-06T07:28:52.968764+0000","last_unstale":"2024-03-06T07:28:54.143285+0000","last_undegraded":"2024-03-06T07:28:54.143285+0000","last_fullsized":"2024-03-06T07:28:54.143285+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:03.772295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.958428+0000","last_change":"2024-03-06T07:28:52.971904+0000","last_active":"2024-03-06T07:28:53.958428+0000","last_peered":"2024-03-06T07:28:53.958428+0000","last_clean":"2024-03-06T07:28:53.958428+0000","last_became_active":"2024-03-06T07:28:52.971206+0000","last_became_peered":"2024-03-06T07:28:52.971206+0000","last_unstale":"2024-03-06T07:28:53.958428+0000","last_undegraded":"2024-03-06T07:28:53.958428+0000","last_fullsized":"2024-03-06T07:28:53.958428+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:34.971658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144568+0000","last_change":"2024-03-06T07:28:52.966612+0000","last_active":"2024-03-06T07:28:54.144568+0000","last_peered":"2024-03-06T07:28:54.144568+0000","last_clean":"2024-03-06T07:28:54.144568+0000","last_became_active":"2024-03-06T07:28:52.966335+0000","last_became_peered":"2024-03-06T07:28:52.966335+0000","last_unstale":"2024-03-06T07:28:54.144568+0000","last_undegraded":"2024-03-06T07:28:54.144568+0000","last_fullsized":"2024-03-06T07:28:54.144568+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:05.116368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.954143+0000","last_change":"2024-03-06T07:28:52.973834+0000","last_active":"2024-03-06T07:28:53.954143+0000","last_peered":"2024-03-06T07:28:53.954143+0000","last_clean":"2024-03-06T07:28:53.954143+0000","last_became_active":"2024-03-06T07:28:52.973562+0000","last_became_peered":"2024-03-06T07:28:52.973562+0000","last_unstale":"2024-03-06T07:28:53.954143+0000","last_undegraded":"2024-03-06T07:28:53.954143+0000","last_fullsized":"2024-03-06T07:28:53.954143+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:05:49.993454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144309+0000","last_change":"2024-03-06T07:28:52.972039+0000","last_active":"2024-03-06T07:28:54.144309+0000","last_peered":"2024-03-06T07:28:54.144309+0000","last_clean":"2024-03-06T07:28:54.144309+0000","last_became_active":"2024-03-06T07:28:52.971242+0000","last_became_peered":"2024-03-06T07:28:52.971242+0000","last_unstale":"2024-03-06T07:28:54.144309+0000","last_undegraded":"2024-03-06T07:28:54.144309+0000","last_fullsized":"2024-03-06T07:28:54.144309+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:08:59.523654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144014+0000","last_change":"2024-03-06T07:28:52.964715+0000","last_active":"2024-03-06T07:28:54.144014+0000","last_peered":"2024-03-06T07:28:54.144014+0000","last_clean":"2024-03-06T07:28:54.144014+0000","last_became_active":"2024-03-06T07:28:52.964122+0000","last_became_peered":"2024-03-06T07:28:52.964122+0000","last_unstale":"2024-03-06T07:28:54.144014+0000","last_undegraded":"2024-03-06T07:28:54.144014+0000","last_fullsized":"2024-03-06T07:28:54.144014+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:28:34.045789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.959741+0000","last_change":"2024-03-06T07:28:52.960814+0000","last_active":"2024-03-06T07:28:53.959741+0000","last_peered":"2024-03-06T07:28:53.959741+0000","last_clean":"2024-03-06T07:28:53.959741+0000","last_became_active":"2024-03-06T07:28:52.960503+0000","last_became_peered":"2024-03-06T07:28:52.960503+0000","last_unstale":"2024-03-06T07:28:53.959741+0000","last_undegraded":"2024-03-06T07:28:53.959741+0000","last_fullsized":"2024-03-06T07:28:53.959741+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:57.517691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143657+0000","last_change":"2024-03-06T07:28:52.967540+0000","last_active":"2024-03-06T07:28:54.143657+0000","last_peered":"2024-03-06T07:28:54.143657+0000","last_clean":"2024-03-06T07:28:54.143657+0000","last_became_active":"2024-03-06T07:28:52.967117+0000","last_became_peered":"2024-03-06T07:28:52.967117+0000","last_unstale":"2024-03-06T07:28:54.143657+0000","last_undegraded":"2024-03-06T07:28:54.143657+0000","last_fullsized":"2024-03-06T07:28:54.143657+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:21.003630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143443+0000","last_change":"2024-03-06T07:28:52.964819+0000","last_active":"2024-03-06T07:28:54.143443+0000","last_peered":"2024-03-06T07:28:54.143443+0000","last_clean":"2024-03-06T07:28:54.143443+0000","last_became_active":"2024-03-06T07:28:52.964364+0000","last_became_peered":"2024-03-06T07:28:52.964364+0000","last_unstale":"2024-03-06T07:28:54.143443+0000","last_undegraded":"2024-03-06T07:28:54.143443+0000","last_fullsized":"2024-03-06T07:28:54.143443+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:39:07.988272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144602+0000","last_change":"2024-03-06T07:28:52.964742+0000","last_active":"2024-03-06T07:28:54.144602+0000","last_peered":"2024-03-06T07:28:54.144602+0000","last_clean":"2024-03-06T07:28:54.144602+0000","last_became_active":"2024-03-06T07:28:52.964468+0000","last_became_peered":"2024-03-06T07:28:52.964468+0000","last_unstale":"2024-03-06T07:28:54.144602+0000","last_undegraded":"2024-03-06T07:28:54.144602+0000","last_fullsized":"2024-03-06T07:28:54.144602+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:56:12.710010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143383+0000","last_change":"2024-03-06T07:28:52.970726+0000","last_active":"2024-03-06T07:28:54.143383+0000","last_peered":"2024-03-06T07:28:54.143383+0000","last_clean":"2024-03-06T07:28:54.143383+0000","last_became_active":"2024-03-06T07:28:52.970460+0000","last_became_peered":"2024-03-06T07:28:52.970460+0000","last_unstale":"2024-03-06T07:28:54.143383+0000","last_undegraded":"2024-03-06T07:28:54.143383+0000","last_fullsized":"2024-03-06T07:28:54.143383+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:06:57.274089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.142927+0000","last_change":"2024-03-06T07:28:52.959645+0000","last_active":"2024-03-06T07:28:54.142927+0000","last_peered":"2024-03-06T07:28:54.142927+0000","last_clean":"2024-03-06T07:28:54.142927+0000","last_became_active":"2024-03-06T07:28:52.959385+0000","last_became_peered":"2024-03-06T07:28:52.959385+0000","last_unstale":"2024-03-06T07:28:54.142927+0000","last_undegraded":"2024-03-06T07:28:54.142927+0000","last_fullsized":"2024-03-06T07:28:54.142927+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:46:55.628556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.958843+0000","last_change":"2024-03-06T07:28:52.965547+0000","last_active":"2024-03-06T07:28:53.958843+0000","last_peered":"2024-03-06T07:28:53.958843+0000","last_clean":"2024-03-06T07:28:53.958843+0000","last_became_active":"2024-03-06T07:28:52.964474+0000","last_became_peered":"2024-03-06T07:28:52.964474+0000","last_unstale":"2024-03-06T07:28:53.958843+0000","last_undegraded":"2024-03-06T07:28:53.958843+0000","last_fullsized":"2024-03-06T07:28:53.958843+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:24.376865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.142661+0000","last_change":"2024-03-06T07:28:52.963889+0000","last_active":"2024-03-06T07:28:54.142661+0000","last_peered":"2024-03-06T07:28:54.142661+0000","last_clean":"2024-03-06T07:28:54.142661+0000","last_became_active":"2024-03-06T07:28:52.963532+0000","last_became_peered":"2024-03-06T07:28:52.963532+0000","last_unstale":"2024-03-06T07:28:54.142661+0000","last_undegraded":"2024-03-06T07:28:54.142661+0000","last_fullsized":"2024-03-06T07:28:54.142661+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:44.151751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.954355+0000","last_change":"2024-03-06T07:28:52.967292+0000","last_active":"2024-03-06T07:28:53.954355+0000","last_peered":"2024-03-06T07:28:53.954355+0000","last_clean":"2024-03-06T07:28:53.954355+0000","last_became_active":"2024-03-06T07:28:52.967026+0000","last_became_peered":"2024-03-06T07:28:52.967026+0000","last_unstale":"2024-03-06T07:28:53.954355+0000","last_undegraded":"2024-03-06T07:28:53.954355+0000","last_fullsized":"2024-03-06T07:28:53.954355+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:35.361294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143793+0000","last_change":"2024-03-06T07:28:52.967696+0000","last_active":"2024-03-06T07:28:54.143793+0000","last_peered":"2024-03-06T07:28:54.143793+0000","last_clean":"2024-03-06T07:28:54.143793+0000","last_became_active":"2024-03-06T07:28:52.967448+0000","last_became_peered":"2024-03-06T07:28:52.967448+0000","last_unstale":"2024-03-06T07:28:54.143793+0000","last_undegraded":"2024-03-06T07:28:54.143793+0000","last_fullsized":"2024-03-06T07:28:54.143793+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:03.904000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144499+0000","last_change":"2024-03-06T07:28:52.972562+0000","last_active":"2024-03-06T07:28:54.144499+0000","last_peered":"2024-03-06T07:28:54.144499+0000","last_clean":"2024-03-06T07:28:54.144499+0000","last_became_active":"2024-03-06T07:28:52.972297+0000","last_became_peered":"2024-03-06T07:28:52.972297+0000","last_unstale":"2024-03-06T07:28:54.144499+0000","last_undegraded":"2024-03-06T07:28:54.144499+0000","last_fullsized":"2024-03-06T07:28:54.144499+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:26.575165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143103+0000","last_change":"2024-03-06T07:28:52.970892+0000","last_active":"2024-03-06T07:28:54.143103+0000","last_peered":"2024-03-06T07:28:54.143103+0000","last_clean":"2024-03-06T07:28:54.143103+0000","last_became_active":"2024-03-06T07:28:52.970576+0000","last_became_peered":"2024-03-06T07:28:52.970576+0000","last_unstale":"2024-03-06T07:28:54.143103+0000","last_undegraded":"2024-03-06T07:28:54.143103+0000","last_fullsized":"2024-03-06T07:28:54.143103+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:38.981799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144695+0000","last_change":"2024-03-06T07:28:52.970302+0000","last_active":"2024-03-06T07:28:54.144695+0000","last_peered":"2024-03-06T07:28:54.144695+0000","last_clean":"2024-03-06T07:28:54.144695+0000","last_became_active":"2024-03-06T07:28:52.970051+0000","last_became_peered":"2024-03-06T07:28:52.970051+0000","last_unstale":"2024-03-06T07:28:54.144695+0000","last_undegraded":"2024-03-06T07:28:54.144695+0000","last_fullsized":"2024-03-06T07:28:54.144695+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:32.804823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144636+0000","last_change":"2024-03-06T07:28:52.973548+0000","last_active":"2024-03-06T07:28:54.144636+0000","last_peered":"2024-03-06T07:28:54.144636+0000","last_clean":"2024-03-06T07:28:54.144636+0000","last_became_active":"2024-03-06T07:28:52.973271+0000","last_became_peered":"2024-03-06T07:28:52.973271+0000","last_unstale":"2024-03-06T07:28:54.144636+0000","last_undegraded":"2024-03-06T07:28:54.144636+0000","last_fullsized":"2024-03-06T07:28:54.144636+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:41.645943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143244+0000","last_change":"2024-03-06T07:28:52.975572+0000","last_active":"2024-03-06T07:28:54.143244+0000","last_peered":"2024-03-06T07:28:54.143244+0000","last_clean":"2024-03-06T07:28:54.143244+0000","last_became_active":"2024-03-06T07:28:52.975203+0000","last_became_peered":"2024-03-06T07:28:52.975203+0000","last_unstale":"2024-03-06T07:28:54.143244+0000","last_undegraded":"2024-03-06T07:28:54.143244+0000","last_fullsized":"2024-03-06T07:28:54.143244+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:16.118770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.959565+0000","last_change":"2024-03-06T07:28:52.960804+0000","last_active":"2024-03-06T07:28:53.959565+0000","last_peered":"2024-03-06T07:28:53.959565+0000","last_clean":"2024-03-06T07:28:53.959565+0000","last_became_active":"2024-03-06T07:28:52.960557+0000","last_became_peered":"2024-03-06T07:28:52.960557+0000","last_unstale":"2024-03-06T07:28:53.959565+0000","last_undegraded":"2024-03-06T07:28:53.959565+0000","last_fullsized":"2024-03-06T07:28:53.959565+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:15.776204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143936+0000","last_change":"2024-03-06T07:28:52.970667+0000","last_active":"2024-03-06T07:28:54.143936+0000","last_peered":"2024-03-06T07:28:54.143936+0000","last_clean":"2024-03-06T07:28:54.143936+0000","last_became_active":"2024-03-06T07:28:52.970408+0000","last_became_peered":"2024-03-06T07:28:52.970408+0000","last_unstale":"2024-03-06T07:28:54.143936+0000","last_undegraded":"2024-03-06T07:28:54.143936+0000","last_fullsized":"2024-03-06T07:28:54.143936+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:04:38.898631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"127'4","reported_seq":18,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143651+0000","last_change":"2024-03-06T07:28:51.970811+0000","last_active":"2024-03-06T07:28:54.143651+0000","last_peered":"2024-03-06T07:28:54.143651+0000","last_clean":"2024-03-06T07:28:54.143651+0000","last_became_active":"2024-03-06T07:28:51.970564+0000","last_became_peered":"2024-03-06T07:28:51.970564+0000","last_unstale":"2024-03-06T07:28:54.143651+0000","last_undegraded":"2024-03-06T07:28:54.143651+0000","last_fullsized":"2024-03-06T07:28:54.143651+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:59.741625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.959001+0000","last_change":"2024-03-06T07:28:52.969220+0000","last_active":"2024-03-06T07:28:53.959001+0000","last_peered":"2024-03-06T07:28:53.959001+0000","last_clean":"2024-03-06T07:28:53.959001+0000","last_became_active":"2024-03-06T07:28:52.968789+0000","last_became_peered":"2024-03-06T07:28:52.968789+0000","last_unstale":"2024-03-06T07:28:53.959001+0000","last_undegraded":"2024-03-06T07:28:53.959001+0000","last_fullsized":"2024-03-06T07:28:53.959001+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:15.776204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"127'1","reported_seq":15,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143089+0000","last_change":"2024-03-06T07:28:51.964480+0000","last_active":"2024-03-06T07:28:54.143089+0000","last_peered":"2024-03-06T07:28:54.143089+0000","last_clean":"2024-03-06T07:28:54.143089+0000","last_became_active":"2024-03-06T07:28:51.964203+0000","last_became_peered":"2024-03-06T07:28:51.964203+0000","last_unstale":"2024-03-06T07:28:54.143089+0000","last_undegraded":"2024-03-06T07:28:54.143089+0000","last_fullsized":"2024-03-06T07:28:54.143089+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:14:32.674292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143485+0000","last_change":"2024-03-06T07:28:52.965670+0000","last_active":"2024-03-06T07:28:54.143485+0000","last_peered":"2024-03-06T07:28:54.143485+0000","last_clean":"2024-03-06T07:28:54.143485+0000","last_became_active":"2024-03-06T07:28:52.965401+0000","last_became_peered":"2024-03-06T07:28:52.965401+0000","last_unstale":"2024-03-06T07:28:54.143485+0000","last_undegraded":"2024-03-06T07:28:54.143485+0000","last_fullsized":"2024-03-06T07:28:54.143485+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:27:10.021510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"127'2","reported_seq":16,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143552+0000","last_change":"2024-03-06T07:28:51.966438+0000","last_active":"2024-03-06T07:28:54.143552+0000","last_peered":"2024-03-06T07:28:54.143552+0000","last_clean":"2024-03-06T07:28:54.143552+0000","last_became_active":"2024-03-06T07:28:51.966143+0000","last_became_peered":"2024-03-06T07:28:51.966143+0000","last_unstale":"2024-03-06T07:28:54.143552+0000","last_undegraded":"2024-03-06T07:28:54.143552+0000","last_fullsized":"2024-03-06T07:28:54.143552+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:15.111853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143227+0000","last_change":"2024-03-06T07:28:52.969430+0000","last_active":"2024-03-06T07:28:54.143227+0000","last_peered":"2024-03-06T07:28:54.143227+0000","last_clean":"2024-03-06T07:28:54.143227+0000","last_became_active":"2024-03-06T07:28:52.968429+0000","last_became_peered":"2024-03-06T07:28:52.968429+0000","last_unstale":"2024-03-06T07:28:54.143227+0000","last_undegraded":"2024-03-06T07:28:54.143227+0000","last_fullsized":"2024-03-06T07:28:54.143227+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:14.242154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"127'2","reported_seq":16,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.954144+0000","last_change":"2024-03-06T07:28:51.964506+0000","last_active":"2024-03-06T07:28:53.954144+0000","last_peered":"2024-03-06T07:28:53.954144+0000","last_clean":"2024-03-06T07:28:53.954144+0000","last_became_active":"2024-03-06T07:28:51.964192+0000","last_became_peered":"2024-03-06T07:28:51.964192+0000","last_unstale":"2024-03-06T07:28:53.954144+0000","last_undegraded":"2024-03-06T07:28:53.954144+0000","last_fullsized":"2024-03-06T07:28:53.954144+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:43:03.523499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143145+0000","last_change":"2024-03-06T07:28:52.968598+0000","last_active":"2024-03-06T07:28:54.143145+0000","last_peered":"2024-03-06T07:28:54.143145+0000","last_clean":"2024-03-06T07:28:54.143145+0000","last_became_active":"2024-03-06T07:28:52.968335+0000","last_became_peered":"2024-03-06T07:28:52.968335+0000","last_unstale":"2024-03-06T07:28:54.143145+0000","last_undegraded":"2024-03-06T07:28:54.143145+0000","last_fullsized":"2024-03-06T07:28:54.143145+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:17:51.856534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"127'1","reported_seq":15,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.954028+0000","last_change":"2024-03-06T07:28:51.966376+0000","last_active":"2024-03-06T07:28:53.954028+0000","last_peered":"2024-03-06T07:28:53.954028+0000","last_clean":"2024-03-06T07:28:53.954028+0000","last_became_active":"2024-03-06T07:28:51.966123+0000","last_became_peered":"2024-03-06T07:28:51.966123+0000","last_unstale":"2024-03-06T07:28:53.954028+0000","last_undegraded":"2024-03-06T07:28:53.954028+0000","last_fullsized":"2024-03-06T07:28:53.954028+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:38:45.555438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.959722+0000","last_change":"2024-03-06T07:28:52.957122+0000","last_active":"2024-03-06T07:28:53.959722+0000","last_peered":"2024-03-06T07:28:53.959722+0000","last_clean":"2024-03-06T07:28:53.959722+0000","last_became_active":"2024-03-06T07:28:52.956828+0000","last_became_peered":"2024-03-06T07:28:52.956828+0000","last_unstale":"2024-03-06T07:28:53.959722+0000","last_undegraded":"2024-03-06T07:28:53.959722+0000","last_fullsized":"2024-03-06T07:28:53.959722+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:42.642239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"127'1","reported_seq":15,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143244+0000","last_change":"2024-03-06T07:28:51.971434+0000","last_active":"2024-03-06T07:28:54.143244+0000","last_peered":"2024-03-06T07:28:54.143244+0000","last_clean":"2024-03-06T07:28:54.143244+0000","last_became_active":"2024-03-06T07:28:51.971151+0000","last_became_peered":"2024-03-06T07:28:51.971151+0000","last_unstale":"2024-03-06T07:28:54.143244+0000","last_undegraded":"2024-03-06T07:28:54.143244+0000","last_fullsized":"2024-03-06T07:28:54.143244+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:38:00.770221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.958957+0000","last_change":"2024-03-06T07:28:52.959003+0000","last_active":"2024-03-06T07:28:53.958957+0000","last_peered":"2024-03-06T07:28:53.958957+0000","last_clean":"2024-03-06T07:28:53.958957+0000","last_became_active":"2024-03-06T07:28:52.958733+0000","last_became_peered":"2024-03-06T07:28:52.958733+0000","last_unstale":"2024-03-06T07:28:53.958957+0000","last_undegraded":"2024-03-06T07:28:53.958957+0000","last_fullsized":"2024-03-06T07:28:53.958957+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:38:01.777138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.959461+0000","last_change":"2024-03-06T07:28:51.965660+0000","last_active":"2024-03-06T07:28:53.959461+0000","last_peered":"2024-03-06T07:28:53.959461+0000","last_clean":"2024-03-06T07:28:53.959461+0000","last_became_active":"2024-03-06T07:28:51.965337+0000","last_became_peered":"2024-03-06T07:28:51.965337+0000","last_unstale":"2024-03-06T07:28:53.959461+0000","last_undegraded":"2024-03-06T07:28:53.959461+0000","last_fullsized":"2024-03-06T07:28:53.959461+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:35:26.827210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.953967+0000","last_change":"2024-03-06T07:28:52.959392+0000","last_active":"2024-03-06T07:28:53.953967+0000","last_peered":"2024-03-06T07:28:53.953967+0000","last_clean":"2024-03-06T07:28:53.953967+0000","last_became_active":"2024-03-06T07:28:52.959152+0000","last_became_peered":"2024-03-06T07:28:52.959152+0000","last_unstale":"2024-03-06T07:28:53.953967+0000","last_undegraded":"2024-03-06T07:28:53.953967+0000","last_fullsized":"2024-03-06T07:28:53.953967+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:50.800577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.954024+0000","last_change":"2024-03-06T07:28:51.961669+0000","last_active":"2024-03-06T07:28:53.954024+0000","last_peered":"2024-03-06T07:28:53.954024+0000","last_clean":"2024-03-06T07:28:53.954024+0000","last_became_active":"2024-03-06T07:28:51.961367+0000","last_became_peered":"2024-03-06T07:28:51.961367+0000","last_unstale":"2024-03-06T07:28:53.954024+0000","last_undegraded":"2024-03-06T07:28:53.954024+0000","last_fullsized":"2024-03-06T07:28:53.954024+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:37.702067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.145049+0000","last_change":"2024-03-06T07:28:52.967292+0000","last_active":"2024-03-06T07:28:54.145049+0000","last_peered":"2024-03-06T07:28:54.145049+0000","last_clean":"2024-03-06T07:28:54.145049+0000","last_became_active":"2024-03-06T07:28:52.967049+0000","last_became_peered":"2024-03-06T07:28:52.967049+0000","last_unstale":"2024-03-06T07:28:54.145049+0000","last_undegraded":"2024-03-06T07:28:54.145049+0000","last_fullsized":"2024-03-06T07:28:54.145049+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:07.633879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143435+0000","last_change":"2024-03-06T07:28:51.963421+0000","last_active":"2024-03-06T07:28:54.143435+0000","last_peered":"2024-03-06T07:28:54.143435+0000","last_clean":"2024-03-06T07:28:54.143435+0000","last_became_active":"2024-03-06T07:28:51.963127+0000","last_became_peered":"2024-03-06T07:28:51.963127+0000","last_unstale":"2024-03-06T07:28:54.143435+0000","last_undegraded":"2024-03-06T07:28:54.143435+0000","last_fullsized":"2024-03-06T07:28:54.143435+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:05:48.986537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143245+0000","last_change":"2024-03-06T07:28:52.973735+0000","last_active":"2024-03-06T07:28:54.143245+0000","last_peered":"2024-03-06T07:28:54.143245+0000","last_clean":"2024-03-06T07:28:54.143245+0000","last_became_active":"2024-03-06T07:28:52.973385+0000","last_became_peered":"2024-03-06T07:28:52.973385+0000","last_unstale":"2024-03-06T07:28:54.143245+0000","last_undegraded":"2024-03-06T07:28:54.143245+0000","last_fullsized":"2024-03-06T07:28:54.143245+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:50:41.441039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144044+0000","last_change":"2024-03-06T07:28:51.970138+0000","last_active":"2024-03-06T07:28:54.144044+0000","last_peered":"2024-03-06T07:28:54.144044+0000","last_clean":"2024-03-06T07:28:54.144044+0000","last_became_active":"2024-03-06T07:28:51.969796+0000","last_became_peered":"2024-03-06T07:28:51.969796+0000","last_unstale":"2024-03-06T07:28:54.144044+0000","last_undegraded":"2024-03-06T07:28:54.144044+0000","last_fullsized":"2024-03-06T07:28:54.144044+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:58.005409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143650+0000","last_change":"2024-03-06T07:28:52.965088+0000","last_active":"2024-03-06T07:28:54.143650+0000","last_peered":"2024-03-06T07:28:54.143650+0000","last_clean":"2024-03-06T07:28:54.143650+0000","last_became_active":"2024-03-06T07:28:52.964815+0000","last_became_peered":"2024-03-06T07:28:52.964815+0000","last_unstale":"2024-03-06T07:28:54.143650+0000","last_undegraded":"2024-03-06T07:28:54.143650+0000","last_fullsized":"2024-03-06T07:28:54.143650+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:44.151751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"127'3","reported_seq":17,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.142845+0000","last_change":"2024-03-06T07:28:51.968492+0000","last_active":"2024-03-06T07:28:54.142845+0000","last_peered":"2024-03-06T07:28:54.142845+0000","last_clean":"2024-03-06T07:28:54.142845+0000","last_became_active":"2024-03-06T07:28:51.968190+0000","last_became_peered":"2024-03-06T07:28:51.968190+0000","last_unstale":"2024-03-06T07:28:54.142845+0000","last_undegraded":"2024-03-06T07:28:54.142845+0000","last_fullsized":"2024-03-06T07:28:54.142845+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:18.858377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143597+0000","last_change":"2024-03-06T07:28:52.964754+0000","last_active":"2024-03-06T07:28:54.143597+0000","last_peered":"2024-03-06T07:28:54.143597+0000","last_clean":"2024-03-06T07:28:54.143597+0000","last_became_active":"2024-03-06T07:28:52.964437+0000","last_became_peered":"2024-03-06T07:28:52.964437+0000","last_unstale":"2024-03-06T07:28:54.143597+0000","last_undegraded":"2024-03-06T07:28:54.143597+0000","last_fullsized":"2024-03-06T07:28:54.143597+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:53.934839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"127'6","reported_seq":20,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.958669+0000","last_change":"2024-03-06T07:28:51.966045+0000","last_active":"2024-03-06T07:28:53.958669+0000","last_peered":"2024-03-06T07:28:53.958669+0000","last_clean":"2024-03-06T07:28:53.958669+0000","last_became_active":"2024-03-06T07:28:51.965790+0000","last_became_peered":"2024-03-06T07:28:51.965790+0000","last_unstale":"2024-03-06T07:28:53.958669+0000","last_undegraded":"2024-03-06T07:28:53.958669+0000","last_fullsized":"2024-03-06T07:28:53.958669+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:06:54.088262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144821+0000","last_change":"2024-03-06T07:28:52.962473+0000","last_active":"2024-03-06T07:28:54.144821+0000","last_peered":"2024-03-06T07:28:54.144821+0000","last_clean":"2024-03-06T07:28:54.144821+0000","last_became_active":"2024-03-06T07:28:52.961570+0000","last_became_peered":"2024-03-06T07:28:52.961570+0000","last_unstale":"2024-03-06T07:28:54.144821+0000","last_undegraded":"2024-03-06T07:28:54.144821+0000","last_fullsized":"2024-03-06T07:28:54.144821+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:50.800577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144881+0000","last_change":"2024-03-06T07:28:51.958418+0000","last_active":"2024-03-06T07:28:54.144881+0000","last_peered":"2024-03-06T07:28:54.144881+0000","last_clean":"2024-03-06T07:28:54.144881+0000","last_became_active":"2024-03-06T07:28:51.957944+0000","last_became_peered":"2024-03-06T07:28:51.957944+0000","last_unstale":"2024-03-06T07:28:54.144881+0000","last_undegraded":"2024-03-06T07:28:54.144881+0000","last_fullsized":"2024-03-06T07:28:54.144881+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:35.191066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.953924+0000","last_change":"2024-03-06T07:28:52.962686+0000","last_active":"2024-03-06T07:28:53.953924+0000","last_peered":"2024-03-06T07:28:53.953924+0000","last_clean":"2024-03-06T07:28:53.953924+0000","last_became_active":"2024-03-06T07:28:52.962407+0000","last_became_peered":"2024-03-06T07:28:52.962407+0000","last_unstale":"2024-03-06T07:28:53.953924+0000","last_undegraded":"2024-03-06T07:28:53.953924+0000","last_fullsized":"2024-03-06T07:28:53.953924+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:19.865295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.954049+0000","last_change":"2024-03-06T07:28:51.969835+0000","last_active":"2024-03-06T07:28:53.954049+0000","last_peered":"2024-03-06T07:28:53.954049+0000","last_clean":"2024-03-06T07:28:53.954049+0000","last_became_active":"2024-03-06T07:28:51.969530+0000","last_became_peered":"2024-03-06T07:28:51.969530+0000","last_unstale":"2024-03-06T07:28:53.954049+0000","last_undegraded":"2024-03-06T07:28:53.954049+0000","last_fullsized":"2024-03-06T07:28:53.954049+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:35.191066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.958376+0000","last_change":"2024-03-06T07:28:52.965364+0000","last_active":"2024-03-06T07:28:53.958376+0000","last_peered":"2024-03-06T07:28:53.958376+0000","last_clean":"2024-03-06T07:28:53.958376+0000","last_became_active":"2024-03-06T07:28:52.964831+0000","last_became_peered":"2024-03-06T07:28:52.964831+0000","last_unstale":"2024-03-06T07:28:53.958376+0000","last_undegraded":"2024-03-06T07:28:53.958376+0000","last_fullsized":"2024-03-06T07:28:53.958376+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:18.486012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"127'8","reported_seq":22,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.958804+0000","last_change":"2024-03-06T07:28:51.968492+0000","last_active":"2024-03-06T07:28:53.958804+0000","last_peered":"2024-03-06T07:28:53.958804+0000","last_clean":"2024-03-06T07:28:53.958804+0000","last_became_active":"2024-03-06T07:28:51.968146+0000","last_became_peered":"2024-03-06T07:28:51.968146+0000","last_unstale":"2024-03-06T07:28:53.958804+0000","last_undegraded":"2024-03-06T07:28:53.958804+0000","last_fullsized":"2024-03-06T07:28:53.958804+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:37.974882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.958778+0000","last_change":"2024-03-06T07:28:52.971128+0000","last_active":"2024-03-06T07:28:53.958778+0000","last_peered":"2024-03-06T07:28:53.958778+0000","last_clean":"2024-03-06T07:28:53.958778+0000","last_became_active":"2024-03-06T07:28:52.970657+0000","last_became_peered":"2024-03-06T07:28:52.970657+0000","last_unstale":"2024-03-06T07:28:53.958778+0000","last_undegraded":"2024-03-06T07:28:53.958778+0000","last_fullsized":"2024-03-06T07:28:53.958778+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:48:26.069964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"127'1","reported_seq":15,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143472+0000","last_change":"2024-03-06T07:28:51.966032+0000","last_active":"2024-03-06T07:28:54.143472+0000","last_peered":"2024-03-06T07:28:54.143472+0000","last_clean":"2024-03-06T07:28:54.143472+0000","last_became_active":"2024-03-06T07:28:51.965778+0000","last_became_peered":"2024-03-06T07:28:51.965778+0000","last_unstale":"2024-03-06T07:28:54.143472+0000","last_undegraded":"2024-03-06T07:28:54.143472+0000","last_fullsized":"2024-03-06T07:28:54.143472+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:01.850038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143257+0000","last_change":"2024-03-06T07:28:52.970288+0000","last_active":"2024-03-06T07:28:54.143257+0000","last_peered":"2024-03-06T07:28:54.143257+0000","last_clean":"2024-03-06T07:28:54.143257+0000","last_became_active":"2024-03-06T07:28:52.969973+0000","last_became_peered":"2024-03-06T07:28:52.969973+0000","last_unstale":"2024-03-06T07:28:54.143257+0000","last_undegraded":"2024-03-06T07:28:54.143257+0000","last_fullsized":"2024-03-06T07:28:54.143257+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:20:25.751101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"127'2","reported_seq":16,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143047+0000","last_change":"2024-03-06T07:28:51.961696+0000","last_active":"2024-03-06T07:28:54.143047+0000","last_peered":"2024-03-06T07:28:54.143047+0000","last_clean":"2024-03-06T07:28:54.143047+0000","last_became_active":"2024-03-06T07:28:51.961441+0000","last_became_peered":"2024-03-06T07:28:51.961441+0000","last_unstale":"2024-03-06T07:28:54.143047+0000","last_undegraded":"2024-03-06T07:28:54.143047+0000","last_fullsized":"2024-03-06T07:28:54.143047+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:42.468283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144735+0000","last_change":"2024-03-06T07:28:52.967362+0000","last_active":"2024-03-06T07:28:54.144735+0000","last_peered":"2024-03-06T07:28:54.144735+0000","last_clean":"2024-03-06T07:28:54.144735+0000","last_became_active":"2024-03-06T07:28:52.967091+0000","last_became_peered":"2024-03-06T07:28:52.967091+0000","last_unstale":"2024-03-06T07:28:54.144735+0000","last_undegraded":"2024-03-06T07:28:54.144735+0000","last_fullsized":"2024-03-06T07:28:54.144735+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:31.346210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"127'1","reported_seq":15,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.959971+0000","last_change":"2024-03-06T07:28:51.966073+0000","last_active":"2024-03-06T07:28:53.959971+0000","last_peered":"2024-03-06T07:28:53.959971+0000","last_clean":"2024-03-06T07:28:53.959971+0000","last_became_active":"2024-03-06T07:28:51.965786+0000","last_became_peered":"2024-03-06T07:28:51.965786+0000","last_unstale":"2024-03-06T07:28:53.959971+0000","last_undegraded":"2024-03-06T07:28:53.959971+0000","last_fullsized":"2024-03-06T07:28:53.959971+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:31:24.030877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.954365+0000","last_change":"2024-03-06T07:28:52.963231+0000","last_active":"2024-03-06T07:28:53.954365+0000","last_peered":"2024-03-06T07:28:53.954365+0000","last_clean":"2024-03-06T07:28:53.954365+0000","last_became_active":"2024-03-06T07:28:52.962463+0000","last_became_peered":"2024-03-06T07:28:52.962463+0000","last_unstale":"2024-03-06T07:28:53.954365+0000","last_undegraded":"2024-03-06T07:28:53.954365+0000","last_fullsized":"2024-03-06T07:28:53.954365+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:31.450403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143020+0000","last_change":"2024-03-06T07:28:51.964810+0000","last_active":"2024-03-06T07:28:54.143020+0000","last_peered":"2024-03-06T07:28:54.143020+0000","last_clean":"2024-03-06T07:28:54.143020+0000","last_became_active":"2024-03-06T07:28:51.964498+0000","last_became_peered":"2024-03-06T07:28:51.964498+0000","last_unstale":"2024-03-06T07:28:54.143020+0000","last_undegraded":"2024-03-06T07:28:54.143020+0000","last_fullsized":"2024-03-06T07:28:54.143020+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:31.797905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144772+0000","last_change":"2024-03-06T07:28:52.975907+0000","last_active":"2024-03-06T07:28:54.144772+0000","last_peered":"2024-03-06T07:28:54.144772+0000","last_clean":"2024-03-06T07:28:54.144772+0000","last_became_active":"2024-03-06T07:28:52.975063+0000","last_became_peered":"2024-03-06T07:28:52.975063+0000","last_unstale":"2024-03-06T07:28:54.144772+0000","last_undegraded":"2024-03-06T07:28:54.144772+0000","last_fullsized":"2024-03-06T07:28:54.144772+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:14:33.681210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"127'2","reported_seq":16,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143077+0000","last_change":"2024-03-06T07:28:51.963111+0000","last_active":"2024-03-06T07:28:54.143077+0000","last_peered":"2024-03-06T07:28:54.143077+0000","last_clean":"2024-03-06T07:28:54.143077+0000","last_became_active":"2024-03-06T07:28:51.962804+0000","last_became_peered":"2024-03-06T07:28:51.962804+0000","last_unstale":"2024-03-06T07:28:54.143077+0000","last_undegraded":"2024-03-06T07:28:54.143077+0000","last_fullsized":"2024-03-06T07:28:54.143077+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:49.793659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.959862+0000","last_change":"2024-03-06T07:28:52.963806+0000","last_active":"2024-03-06T07:28:53.959862+0000","last_peered":"2024-03-06T07:28:53.959862+0000","last_clean":"2024-03-06T07:28:53.959862+0000","last_became_active":"2024-03-06T07:28:52.963507+0000","last_became_peered":"2024-03-06T07:28:52.963507+0000","last_unstale":"2024-03-06T07:28:53.959862+0000","last_undegraded":"2024-03-06T07:28:53.959862+0000","last_fullsized":"2024-03-06T07:28:53.959862+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:54:25.798793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143332+0000","last_change":"2024-03-06T07:28:51.968470+0000","last_active":"2024-03-06T07:28:54.143332+0000","last_peered":"2024-03-06T07:28:54.143332+0000","last_clean":"2024-03-06T07:28:54.143332+0000","last_became_active":"2024-03-06T07:28:51.968177+0000","last_became_peered":"2024-03-06T07:28:51.968177+0000","last_unstale":"2024-03-06T07:28:54.143332+0000","last_undegraded":"2024-03-06T07:28:54.143332+0000","last_fullsized":"2024-03-06T07:28:54.143332+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:48:25.063046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.953928+0000","last_change":"2024-03-06T07:28:52.958145+0000","last_active":"2024-03-06T07:28:53.953928+0000","last_peered":"2024-03-06T07:28:53.953928+0000","last_clean":"2024-03-06T07:28:53.953928+0000","last_became_active":"2024-03-06T07:28:52.957842+0000","last_became_peered":"2024-03-06T07:28:52.957842+0000","last_unstale":"2024-03-06T07:28:53.953928+0000","last_undegraded":"2024-03-06T07:28:53.953928+0000","last_fullsized":"2024-03-06T07:28:53.953928+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:32.804823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.142893+0000","last_change":"2024-03-06T07:28:51.968301+0000","last_active":"2024-03-06T07:28:54.142893+0000","last_peered":"2024-03-06T07:28:54.142893+0000","last_clean":"2024-03-06T07:28:54.142893+0000","last_became_active":"2024-03-06T07:28:51.968026+0000","last_became_peered":"2024-03-06T07:28:51.968026+0000","last_unstale":"2024-03-06T07:28:54.142893+0000","last_undegraded":"2024-03-06T07:28:54.142893+0000","last_fullsized":"2024-03-06T07:28:54.142893+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:23.369947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143009+0000","last_change":"2024-03-06T07:28:52.972374+0000","last_active":"2024-03-06T07:28:54.143009+0000","last_peered":"2024-03-06T07:28:54.143009+0000","last_clean":"2024-03-06T07:28:54.143009+0000","last_became_active":"2024-03-06T07:28:52.972101+0000","last_became_peered":"2024-03-06T07:28:52.972101+0000","last_unstale":"2024-03-06T07:28:54.143009+0000","last_undegraded":"2024-03-06T07:28:54.143009+0000","last_fullsized":"2024-03-06T07:28:54.143009+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:41.237344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.142599+0000","last_change":"2024-03-06T07:28:51.960935+0000","last_active":"2024-03-06T07:28:54.142599+0000","last_peered":"2024-03-06T07:28:54.142599+0000","last_clean":"2024-03-06T07:28:54.142599+0000","last_became_active":"2024-03-06T07:28:51.960458+0000","last_became_peered":"2024-03-06T07:28:51.960458+0000","last_unstale":"2024-03-06T07:28:54.142599+0000","last_undegraded":"2024-03-06T07:28:54.142599+0000","last_fullsized":"2024-03-06T07:28:54.142599+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:40.230426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.959956+0000","last_change":"2024-03-06T07:28:52.969742+0000","last_active":"2024-03-06T07:28:53.959956+0000","last_peered":"2024-03-06T07:28:53.959956+0000","last_clean":"2024-03-06T07:28:53.959956+0000","last_became_active":"2024-03-06T07:28:52.969401+0000","last_became_peered":"2024-03-06T07:28:52.969401+0000","last_unstale":"2024-03-06T07:28:53.959956+0000","last_undegraded":"2024-03-06T07:28:53.959956+0000","last_fullsized":"2024-03-06T07:28:53.959956+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:08.434943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"127'2","reported_seq":16,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.145058+0000","last_change":"2024-03-06T07:28:51.959525+0000","last_active":"2024-03-06T07:28:54.145058+0000","last_peered":"2024-03-06T07:28:54.145058+0000","last_clean":"2024-03-06T07:28:54.145058+0000","last_became_active":"2024-03-06T07:28:51.959247+0000","last_became_peered":"2024-03-06T07:28:51.959247+0000","last_unstale":"2024-03-06T07:28:54.145058+0000","last_undegraded":"2024-03-06T07:28:54.145058+0000","last_fullsized":"2024-03-06T07:28:54.145058+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:37.278665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.145016+0000","last_change":"2024-03-06T07:28:52.972711+0000","last_active":"2024-03-06T07:28:54.145016+0000","last_peered":"2024-03-06T07:28:54.145016+0000","last_clean":"2024-03-06T07:28:54.145016+0000","last_became_active":"2024-03-06T07:28:52.972448+0000","last_became_peered":"2024-03-06T07:28:52.972448+0000","last_unstale":"2024-03-06T07:28:54.145016+0000","last_undegraded":"2024-03-06T07:28:54.145016+0000","last_fullsized":"2024-03-06T07:28:54.145016+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:02.856955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"128'1","reported_seq":15,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.963947+0000","last_change":"2024-03-06T07:28:51.969705+0000","last_active":"2024-03-06T07:28:54.963947+0000","last_peered":"2024-03-06T07:28:54.963947+0000","last_clean":"2024-03-06T07:28:54.963947+0000","last_became_active":"2024-03-06T07:28:51.969395+0000","last_became_peered":"2024-03-06T07:28:51.969395+0000","last_unstale":"2024-03-06T07:28:54.963947+0000","last_undegraded":"2024-03-06T07:28:54.963947+0000","last_fullsized":"2024-03-06T07:28:54.963947+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:59:39.689093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.954215+0000","last_change":"2024-03-06T07:28:52.963942+0000","last_active":"2024-03-06T07:28:53.954215+0000","last_peered":"2024-03-06T07:28:53.954215+0000","last_clean":"2024-03-06T07:28:53.954215+0000","last_became_active":"2024-03-06T07:28:52.963685+0000","last_became_peered":"2024-03-06T07:28:52.963685+0000","last_unstale":"2024-03-06T07:28:53.954215+0000","last_undegraded":"2024-03-06T07:28:53.954215+0000","last_fullsized":"2024-03-06T07:28:53.954215+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:02.856955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.954356+0000","last_change":"2024-03-06T07:28:51.960696+0000","last_active":"2024-03-06T07:28:53.954356+0000","last_peered":"2024-03-06T07:28:53.954356+0000","last_clean":"2024-03-06T07:28:53.954356+0000","last_became_active":"2024-03-06T07:28:51.960438+0000","last_became_peered":"2024-03-06T07:28:51.960438+0000","last_unstale":"2024-03-06T07:28:53.954356+0000","last_undegraded":"2024-03-06T07:28:53.954356+0000","last_fullsized":"2024-03-06T07:28:53.954356+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:14.232718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.959925+0000","last_change":"2024-03-06T07:28:52.967435+0000","last_active":"2024-03-06T07:28:53.959925+0000","last_peered":"2024-03-06T07:28:53.959925+0000","last_clean":"2024-03-06T07:28:53.959925+0000","last_became_active":"2024-03-06T07:28:52.967158+0000","last_became_peered":"2024-03-06T07:28:52.967158+0000","last_unstale":"2024-03-06T07:28:53.959925+0000","last_undegraded":"2024-03-06T07:28:53.959925+0000","last_fullsized":"2024-03-06T07:28:53.959925+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:59:40.696010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143231+0000","last_change":"2024-03-06T07:28:51.963760+0000","last_active":"2024-03-06T07:28:54.143231+0000","last_peered":"2024-03-06T07:28:54.143231+0000","last_clean":"2024-03-06T07:28:54.143231+0000","last_became_active":"2024-03-06T07:28:51.963374+0000","last_became_peered":"2024-03-06T07:28:51.963374+0000","last_unstale":"2024-03-06T07:28:54.143231+0000","last_undegraded":"2024-03-06T07:28:54.143231+0000","last_fullsized":"2024-03-06T07:28:54.143231+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:13.235236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143242+0000","last_change":"2024-03-06T07:28:52.968583+0000","last_active":"2024-03-06T07:28:54.143242+0000","last_peered":"2024-03-06T07:28:54.143242+0000","last_clean":"2024-03-06T07:28:54.143242+0000","last_became_active":"2024-03-06T07:28:52.968266+0000","last_became_peered":"2024-03-06T07:28:52.968266+0000","last_unstale":"2024-03-06T07:28:54.143242+0000","last_undegraded":"2024-03-06T07:28:54.143242+0000","last_fullsized":"2024-03-06T07:28:54.143242+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:43.475200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.954120+0000","last_change":"2024-03-06T07:28:51.966274+0000","last_active":"2024-03-06T07:28:53.954120+0000","last_peered":"2024-03-06T07:28:53.954120+0000","last_clean":"2024-03-06T07:28:53.954120+0000","last_became_active":"2024-03-06T07:28:51.965795+0000","last_became_peered":"2024-03-06T07:28:51.965795+0000","last_unstale":"2024-03-06T07:28:53.954120+0000","last_undegraded":"2024-03-06T07:28:53.954120+0000","last_fullsized":"2024-03-06T07:28:53.954120+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:44:14.025940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143168+0000","last_change":"2024-03-06T07:28:52.978648+0000","last_active":"2024-03-06T07:28:54.143168+0000","last_peered":"2024-03-06T07:28:54.143168+0000","last_clean":"2024-03-06T07:28:54.143168+0000","last_became_active":"2024-03-06T07:28:52.978331+0000","last_became_peered":"2024-03-06T07:28:52.978331+0000","last_unstale":"2024-03-06T07:28:54.143168+0000","last_undegraded":"2024-03-06T07:28:54.143168+0000","last_fullsized":"2024-03-06T07:28:54.143168+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:04.981862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143248+0000","last_change":"2024-03-06T07:28:51.971364+0000","last_active":"2024-03-06T07:28:54.143248+0000","last_peered":"2024-03-06T07:28:54.143248+0000","last_clean":"2024-03-06T07:28:54.143248+0000","last_became_active":"2024-03-06T07:28:51.971030+0000","last_became_peered":"2024-03-06T07:28:51.971030+0000","last_unstale":"2024-03-06T07:28:54.143248+0000","last_undegraded":"2024-03-06T07:28:54.143248+0000","last_fullsized":"2024-03-06T07:28:54.143248+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:07.428026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.144153+0000","last_change":"2024-03-06T07:28:52.961576+0000","last_active":"2024-03-06T07:28:54.144153+0000","last_peered":"2024-03-06T07:28:54.144153+0000","last_clean":"2024-03-06T07:28:54.144153+0000","last_became_active":"2024-03-06T07:28:52.961325+0000","last_became_peered":"2024-03-06T07:28:52.961325+0000","last_unstale":"2024-03-06T07:28:54.144153+0000","last_undegraded":"2024-03-06T07:28:54.144153+0000","last_fullsized":"2024-03-06T07:28:54.144153+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:53.405336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.142987+0000","last_change":"2024-03-06T07:28:51.968672+0000","last_active":"2024-03-06T07:28:54.142987+0000","last_peered":"2024-03-06T07:28:54.142987+0000","last_clean":"2024-03-06T07:28:54.142987+0000","last_became_active":"2024-03-06T07:28:51.968340+0000","last_became_peered":"2024-03-06T07:28:51.968340+0000","last_unstale":"2024-03-06T07:28:54.142987+0000","last_undegraded":"2024-03-06T07:28:54.142987+0000","last_fullsized":"2024-03-06T07:28:54.142987+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:22:47.593555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:53.953957+0000","last_change":"2024-03-06T07:28:52.966628+0000","last_active":"2024-03-06T07:28:53.953957+0000","last_peered":"2024-03-06T07:28:53.953957+0000","last_clean":"2024-03-06T07:28:53.953957+0000","last_became_active":"2024-03-06T07:28:52.966354+0000","last_became_peered":"2024-03-06T07:28:52.966354+0000","last_unstale":"2024-03-06T07:28:53.953957+0000","last_undegraded":"2024-03-06T07:28:53.953957+0000","last_fullsized":"2024-03-06T07:28:53.953957+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:14:33.681210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.142673+0000","last_change":"2024-03-06T07:28:51.960707+0000","last_active":"2024-03-06T07:28:54.142673+0000","last_peered":"2024-03-06T07:28:54.142673+0000","last_clean":"2024-03-06T07:28:54.142673+0000","last_became_active":"2024-03-06T07:28:51.960177+0000","last_became_peered":"2024-03-06T07:28:51.960177+0000","last_unstale":"2024-03-06T07:28:54.142673+0000","last_undegraded":"2024-03-06T07:28:54.142673+0000","last_fullsized":"2024-03-06T07:28:54.142673+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:41.094093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143122+0000","last_change":"2024-03-06T07:28:52.975661+0000","last_active":"2024-03-06T07:28:54.143122+0000","last_peered":"2024-03-06T07:28:54.143122+0000","last_clean":"2024-03-06T07:28:54.143122+0000","last_became_active":"2024-03-06T07:28:52.975355+0000","last_became_peered":"2024-03-06T07:28:52.975355+0000","last_unstale":"2024-03-06T07:28:54.143122+0000","last_undegraded":"2024-03-06T07:28:54.143122+0000","last_fullsized":"2024-03-06T07:28:54.143122+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:51.939703+0000","last_clean_scrub_stamp":"2024-03-06T07:28:51.939703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:42.101011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"127'2","reported_seq":16,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.143618+0000","last_change":"2024-03-06T07:28:51.970862+0000","last_active":"2024-03-06T07:28:54.143618+0000","last_peered":"2024-03-06T07:28:54.143618+0000","last_clean":"2024-03-06T07:28:54.143618+0000","last_became_active":"2024-03-06T07:28:51.970609+0000","last_became_peered":"2024-03-06T07:28:51.970609+0000","last_unstale":"2024-03-06T07:28:54.143618+0000","last_undegraded":"2024-03-06T07:28:54.143618+0000","last_fullsized":"2024-03-06T07:28:54.143618+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:50.932785+0000","last_clean_scrub_stamp":"2024-03-06T07:28:50.932785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:40.486032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":364,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-03-06T07:28:54.142779+0000","last_change":"2024-03-06T07:21:47.553693+0000","last_active":"2024-03-06T07:28:54.142779+0000","last_peered":"2024-03-06T07:28:54.142779+0000","last_clean":"2024-03-06T07:28:54.142779+0000","last_became_active":"2024-03-06T07:21:47.552664+0000","last_became_peered":"2024-03-06T07:21:47.552664+0000","last_unstale":"2024-03-06T07:28:54.142779+0000","last_undegraded":"2024-03-06T07:28:54.142779+0000","last_fullsized":"2024-03-06T07:28:54.142779+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_clean_scrub_stamp":"2024-03-06T07:21:46.362028+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:02.916426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574940,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41664,"kb_used_data":2164,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330176,"statfs":{"total":96636764160,"available":96594100224,"internally_reserved":0,"allocated":2215936,"data_stored":1288896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":40434121},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574938,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41072,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330768,"statfs":{"total":96636764160,"available":96594706432,"internally_reserved":0,"allocated":1609728,"data_stored":697890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574938,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41068,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330772,"statfs":{"total":96636764160,"available":96594710528,"internally_reserved":0,"allocated":1605632,"data_stored":696390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574939,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41068,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330772,"statfs":{"total":96636764160,"available":96594710528,"internally_reserved":0,"allocated":1601536,"data_stored":696348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574939,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41068,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330772,"statfs":{"total":96636764160,"available":96594710528,"internally_reserved":0,"allocated":1605632,"data_stored":696438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574939,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41068,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330772,"statfs":{"total":96636764160,"available":96594710528,"internally_reserved":0,"allocated":1605632,"data_stored":696894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574940,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41060,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330780,"statfs":{"total":96636764160,"available":96594718720,"internally_reserved":0,"allocated":1597440,"data_stored":696278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574939,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41648,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330192,"statfs":{"total":96636764160,"available":96594116608,"internally_reserved":0,"allocated":2191360,"data_stored":1286646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:29:00.746 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:29:00.747 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:29:00.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.890+0000 7f28161bc640 1 Processor -- start 2024-03-06T07:29:00.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.890+0000 7f28161bc640 1 -- start start 2024-03-06T07:29:00.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.891+0000 7f28161bc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f281015ecd0 0x7f28100cae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.891+0000 7f28161bc640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28100cfa50 con 0x7f28100d14a0 2024-03-06T07:29:00.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.891+0000 7f28161bc640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28100b45c0 con 0x7f28100d10c0 2024-03-06T07:29:00.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.891+0000 7f28161bc640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f281015c050 con 0x7f281015ecd0 2024-03-06T07:29:00.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.891+0000 7f280ffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f281015ecd0 0x7f28100cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.891+0000 7f280effd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f28100d14a0 0x7f281015e5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49400/0 (socket says 172.21.15.83:49400) 2024-03-06T07:29:00.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.891+0000 7f280ffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f281015ecd0 0x7f28100cae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52042/0 (socket says 172.21.15.83:52042) 2024-03-06T07:29:00.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.891+0000 7f280effd640 1 -- 172.21.15.83:0/956588624 learned_addr learned my addr 172.21.15.83:0/956588624 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:29:00.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.891+0000 7f280ffff640 1 -- 172.21.15.83:0/956588624 >> v1:172.21.15.183:6789/0 conn(0x7f28100d10c0 legacy=0x7f28100d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:00.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.891+0000 7f280ffff640 1 -- 172.21.15.83:0/956588624 >> v1:172.21.15.83:6789/0 conn(0x7f28100d14a0 legacy=0x7f281015e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.891+0000 7f280ffff640 1 -- 172.21.15.83:0/956588624 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f281015f160 con 0x7f281015ecd0 2024-03-06T07:29:00.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.891+0000 7f280e7fc640 1 -- 172.21.15.83:0/956588624 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3295158182 0 0) 0x7f28100cfa50 con 0x7f28100d14a0 2024-03-06T07:29:00.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.892+0000 7f280ffff640 1 --2- 172.21.15.83:0/956588624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f281015ecd0 0x7f28100cae80 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f2808009d90 tx=0x7f280802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c576f476737ea8cb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:00.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.892+0000 7f280e7fc640 1 -- 172.21.15.83:0/956588624 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2808039070 con 0x7f281015ecd0 2024-03-06T07:29:00.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.892+0000 7f280e7fc640 1 -- 172.21.15.83:0/956588624 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f280802cc60 con 0x7f281015ecd0 2024-03-06T07:29:00.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.892+0000 7f280e7fc640 1 -- 172.21.15.83:0/956588624 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2808034660 con 0x7f281015ecd0 2024-03-06T07:29:00.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.892+0000 7f28161bc640 1 -- 172.21.15.83:0/956588624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f281015ecd0 msgr2=0x7f28100cae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.892+0000 7f28161bc640 1 --2- 172.21.15.83:0/956588624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f281015ecd0 0x7f28100cae80 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f2808009d90 tx=0x7f280802beb0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.893+0000 7f28161bc640 1 -- 172.21.15.83:0/956588624 shutdown_connections 2024-03-06T07:29:00.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.893+0000 7f28161bc640 1 --2- 172.21.15.83:0/956588624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f281015ecd0 0x7f28100cae80 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.893+0000 7f28161bc640 1 -- 172.21.15.83:0/956588624 >> 172.21.15.83:0/956588624 conn(0x7f28100b3800 msgr2=0x7f28100b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:00.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.893+0000 7f28161bc640 1 -- 172.21.15.83:0/956588624 shutdown_connections 2024-03-06T07:29:00.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.893+0000 7f28161bc640 1 -- 172.21.15.83:0/956588624 wait complete. 2024-03-06T07:29:00.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.894+0000 7f28161bc640 1 Processor -- start 2024-03-06T07:29:00.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.894+0000 7f28161bc640 1 -- start start 2024-03-06T07:29:00.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.895+0000 7f28161bc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28100d10c0 0x7f28100b48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.895+0000 7f28161bc640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f28100d14a0 0x7f28100b71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.895+0000 7f28161bc640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f28100b5300 0x7f281014d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.895+0000 7f28161bc640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28100cfcf0 con 0x7f28100d10c0 2024-03-06T07:29:00.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.895+0000 7f28161bc640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28100d0060 con 0x7f28100d14a0 2024-03-06T07:29:00.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.895+0000 7f28161bc640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28100cb3c0 con 0x7f28100b5300 2024-03-06T07:29:00.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.895+0000 7f280f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28100d10c0 0x7f28100b48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.895+0000 7f280f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28100d10c0 0x7f28100b48f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52046/0 (socket says 172.21.15.83:52046) 2024-03-06T07:29:00.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.895+0000 7f280f7fe640 1 -- 172.21.15.83:0/2902935877 learned_addr learned my addr 172.21.15.83:0/2902935877 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:00.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.895+0000 7f280f7fe640 1 -- 172.21.15.83:0/2902935877 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f28100b5300 msgr2=0x7f281014d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.895+0000 7f280f7fe640 1 --2- 172.21.15.83:0/2902935877 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f28100b5300 0x7f281014d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.895+0000 7f280f7fe640 1 -- 172.21.15.83:0/2902935877 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f28100d14a0 msgr2=0x7f28100b71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.895+0000 7f280f7fe640 1 --2- 172.21.15.83:0/2902935877 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f28100d14a0 0x7f28100b71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.895+0000 7f280f7fe640 1 -- 172.21.15.83:0/2902935877 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f281014e330 con 0x7f28100d10c0 2024-03-06T07:29:00.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.895+0000 7f280f7fe640 1 --2- 172.21.15.83:0/2902935877 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28100d10c0 0x7f28100b48f0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f280000c430 tx=0x7f280000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:00.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.895+0000 7f27f7fff640 1 -- 172.21.15.83:0/2902935877 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2800011790 con 0x7f28100d10c0 2024-03-06T07:29:00.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.895+0000 7f27f7fff640 1 -- 172.21.15.83:0/2902935877 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2800017070 con 0x7f28100d10c0 2024-03-06T07:29:00.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.895+0000 7f28161bc640 1 -- 172.21.15.83:0/2902935877 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f281014e610 con 0x7f28100d10c0 2024-03-06T07:29:00.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.895+0000 7f27f7fff640 1 -- 172.21.15.83:0/2902935877 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2800011bb0 con 0x7f28100d10c0 2024-03-06T07:29:00.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.895+0000 7f28161bc640 1 -- 172.21.15.83:0/2902935877 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28100b7710 con 0x7f28100d10c0 2024-03-06T07:29:00.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.896+0000 7f28161bc640 1 -- 172.21.15.83:0/2902935877 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f281009fe60 con 0x7f28100d10c0 2024-03-06T07:29:00.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.897+0000 7f27f7fff640 1 -- 172.21.15.83:0/2902935877 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f280001c460 con 0x7f28100d10c0 2024-03-06T07:29:00.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.897+0000 7f27f7fff640 1 --2- 172.21.15.83:0/2902935877 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f27fc06b500 0x7f27fc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.897+0000 7f280effd640 1 --2- 172.21.15.83:0/2902935877 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f27fc06b500 0x7f27fc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.897+0000 7f27f7fff640 1 -- 172.21.15.83:0/2902935877 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f280008a000 con 0x7f28100d10c0 2024-03-06T07:29:00.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.898+0000 7f280effd640 1 --2- 172.21.15.83:0/2902935877 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f27fc06b500 0x7f27fc06d9c0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f2804005880 tx=0x7f2804007950 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:00.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:00.902+0000 7f27f7fff640 1 -- 172.21.15.83:0/2902935877 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2800058f10 con 0x7f28100d10c0 2024-03-06T07:29:01.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.148+0000 7f28161bc640 1 -- 172.21.15.83:0/2902935877 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f2810068ff0 con 0x7f28100d10c0 2024-03-06T07:29:01.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.152+0000 7f27f7fff640 1 -- 172.21.15.83:0/2902935877 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) v1 ==== 190+0+147 (secure 0 0 0) 0x7f2800058810 con 0x7f28100d10c0 2024-03-06T07:29:01.154 INFO:teuthology.orchestra.run.smithi083.stdout:[client.0] 2024-03-06T07:29:01.154 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBcGehlRwCIMhAA1DdbdQZPplu7LQOTK3BmSg== 2024-03-06T07:29:01.154 INFO:teuthology.orchestra.run.smithi083.stdout: caps mds = "allow" 2024-03-06T07:29:01.155 INFO:teuthology.orchestra.run.smithi083.stdout: caps mon = "allow r" 2024-03-06T07:29:01.155 INFO:teuthology.orchestra.run.smithi083.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:29:01.155 INFO:teuthology.orchestra.run.smithi083.stderr:updated caps for client.0 2024-03-06T07:29:01.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.154+0000 7f28161bc640 1 -- 172.21.15.83:0/2902935877 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f27fc06b500 msgr2=0x7f27fc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.154+0000 7f28161bc640 1 --2- 172.21.15.83:0/2902935877 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f27fc06b500 0x7f27fc06d9c0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f2804005880 tx=0x7f2804007950 comp rx=0 tx=0).stop 2024-03-06T07:29:01.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.154+0000 7f28161bc640 1 -- 172.21.15.83:0/2902935877 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28100d10c0 msgr2=0x7f28100b48f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.154+0000 7f28161bc640 1 --2- 172.21.15.83:0/2902935877 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28100d10c0 0x7f28100b48f0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f280000c430 tx=0x7f280000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.154+0000 7f28161bc640 1 -- 172.21.15.83:0/2902935877 shutdown_connections 2024-03-06T07:29:01.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.154+0000 7f28161bc640 1 --2- 172.21.15.83:0/2902935877 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f27fc06b500 0x7f27fc06d9c0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.154+0000 7f28161bc640 1 --2- 172.21.15.83:0/2902935877 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f28100b5300 0x7f281014d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.154+0000 7f28161bc640 1 --2- 172.21.15.83:0/2902935877 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f28100d14a0 0x7f28100b71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.154+0000 7f28161bc640 1 --2- 172.21.15.83:0/2902935877 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28100d10c0 0x7f28100b48f0 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.154+0000 7f28161bc640 1 -- 172.21.15.83:0/2902935877 >> 172.21.15.83:0/2902935877 conn(0x7f28100b3800 msgr2=0x7f28100b2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:01.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.154+0000 7f28161bc640 1 -- 172.21.15.83:0/2902935877 shutdown_connections 2024-03-06T07:29:01.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.154+0000 7f28161bc640 1 -- 172.21.15.83:0/2902935877 wait complete. 2024-03-06T07:29:01.167 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:01.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.270+0000 7f3ce7dd0640 1 Processor -- start 2024-03-06T07:29:01.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.270+0000 7f3ce7dd0640 1 -- start start 2024-03-06T07:29:01.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.271+0000 7f3ce7dd0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f3ce00d00c0 0x7f3ce00d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.271+0000 7f3ce7dd0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3ce006cf20 0x7f3ce006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.271+0000 7f3ce7dd0640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ce00d09e0 con 0x7f3ce00cf850 2024-03-06T07:29:01.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.271+0000 7f3ce7dd0640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ce006ca90 con 0x7f3ce00d00c0 2024-03-06T07:29:01.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.271+0000 7f3ce7dd0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce015e380 con 0x7f3ce006cf20 2024-03-06T07:29:01.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.271+0000 7f3ce5b6c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f3ce00d00c0 0x7f3ce00d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.271+0000 7f3ce5b6c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f3ce00d00c0 0x7f3ce00d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52058/0 (socket says 172.21.15.83:52058) 2024-03-06T07:29:01.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.271+0000 7f3ce5b6c640 1 -- 172.21.15.83:0/3387512781 learned_addr learned my addr 172.21.15.83:0/3387512781 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:01.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.271+0000 7f3ce636d640 1 --2- 172.21.15.83:0/3387512781 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3ce006cf20 0x7f3ce006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.272+0000 7f3ce5b6c640 1 -- 172.21.15.83:0/3387512781 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3ce006cf20 msgr2=0x7f3ce006d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:01.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.272+0000 7f3ce5b6c640 1 --2- 172.21.15.83:0/3387512781 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3ce006cf20 0x7f3ce006d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.272+0000 7f3ce5b6c640 1 -- 172.21.15.83:0/3387512781 >> v1:172.21.15.183:6789/0 conn(0x7f3ce00cf850 legacy=0x7f3ce00cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:01.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.272+0000 7f3ce5b6c640 1 -- 172.21.15.83:0/3387512781 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce00cdc10 con 0x7f3ce00d00c0 2024-03-06T07:29:01.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.272+0000 7f3ce636d640 1 --2- 172.21.15.83:0/3387512781 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3ce006cf20 0x7f3ce006d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:01.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.272+0000 7f3ce5b6c640 1 --2- 172.21.15.83:0/3387512781 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ce00d00c0 0x7f3ce00d04a0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f3cd0009870 tx=0x7f3cd002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9798576ac6ef2c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:01.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.272+0000 7f3ce4b6a640 1 -- 172.21.15.83:0/3387512781 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd0035020 con 0x7f3ce00d00c0 2024-03-06T07:29:01.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.272+0000 7f3ce4b6a640 1 -- 172.21.15.83:0/3387512781 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cd0031640 con 0x7f3ce00d00c0 2024-03-06T07:29:01.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.272+0000 7f3ce4b6a640 1 -- 172.21.15.83:0/3387512781 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd002d040 con 0x7f3ce00d00c0 2024-03-06T07:29:01.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.273+0000 7f3ce7dd0640 1 -- 172.21.15.83:0/3387512781 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ce00d00c0 msgr2=0x7f3ce00d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.273+0000 7f3ce7dd0640 1 --2- 172.21.15.83:0/3387512781 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ce00d00c0 0x7f3ce00d04a0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f3cd0009870 tx=0x7f3cd002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.273+0000 7f3ce7dd0640 1 -- 172.21.15.83:0/3387512781 shutdown_connections 2024-03-06T07:29:01.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.273+0000 7f3ce7dd0640 1 --2- 172.21.15.83:0/3387512781 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3ce006cf20 0x7f3ce006d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.273+0000 7f3ce7dd0640 1 --2- 172.21.15.83:0/3387512781 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ce00d00c0 0x7f3ce00d04a0 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.273+0000 7f3ce7dd0640 1 -- 172.21.15.83:0/3387512781 >> 172.21.15.83:0/3387512781 conn(0x7f3ce00b35a0 msgr2=0x7f3ce00b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:01.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.273+0000 7f3ce7dd0640 1 -- 172.21.15.83:0/3387512781 shutdown_connections 2024-03-06T07:29:01.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.273+0000 7f3ce7dd0640 1 -- 172.21.15.83:0/3387512781 wait complete. 2024-03-06T07:29:01.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.274+0000 7f3ce7dd0640 1 Processor -- start 2024-03-06T07:29:01.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.274+0000 7f3ce7dd0640 1 -- start start 2024-03-06T07:29:01.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.275+0000 7f3ce7dd0640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3ce006cf20 0x7f3ce0156170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.275+0000 7f3ce7dd0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ce00cf850 0x7f3ce00cb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.275+0000 7f3ce7dd0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3ce00d00c0 0x7f3ce00cb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.275+0000 7f3ce7dd0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce006c720 con 0x7f3ce00cf850 2024-03-06T07:29:01.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.275+0000 7f3ce7dd0640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce006c440 con 0x7f3ce006cf20 2024-03-06T07:29:01.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.275+0000 7f3ce7dd0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce00b1030 con 0x7f3ce00d00c0 2024-03-06T07:29:01.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.275+0000 7f3ce536b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ce00cf850 0x7f3ce00cb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.275+0000 7f3ce536b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ce00cf850 0x7f3ce00cb0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52074/0 (socket says 172.21.15.83:52074) 2024-03-06T07:29:01.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.275+0000 7f3ce536b640 1 -- 172.21.15.83:0/367960561 learned_addr learned my addr 172.21.15.83:0/367960561 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:01.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.275+0000 7f3ce5b6c640 1 --2- 172.21.15.83:0/367960561 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3ce006cf20 0x7f3ce0156170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.275+0000 7f3ce636d640 1 --2- 172.21.15.83:0/367960561 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3ce00d00c0 0x7f3ce00cb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.276+0000 7f3ce536b640 1 -- 172.21.15.83:0/367960561 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3ce00d00c0 msgr2=0x7f3ce00cb5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.276+0000 7f3ce536b640 1 --2- 172.21.15.83:0/367960561 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3ce00d00c0 0x7f3ce00cb5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.276+0000 7f3ce536b640 1 -- 172.21.15.83:0/367960561 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3ce006cf20 msgr2=0x7f3ce0156170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.276+0000 7f3ce536b640 1 --2- 172.21.15.83:0/367960561 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3ce006cf20 0x7f3ce0156170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.276+0000 7f3ce536b640 1 -- 172.21.15.83:0/367960561 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce00b60a0 con 0x7f3ce00cf850 2024-03-06T07:29:01.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.276+0000 7f3ce536b640 1 --2- 172.21.15.83:0/367960561 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ce00cf850 0x7f3ce00cb0b0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc00c430 tx=0x7f3cdc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:01.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.276+0000 7f3cd67fc640 1 -- 172.21.15.83:0/367960561 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cdc011790 con 0x7f3ce00cf850 2024-03-06T07:29:01.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.276+0000 7f3cd67fc640 1 -- 172.21.15.83:0/367960561 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cdc017070 con 0x7f3ce00cf850 2024-03-06T07:29:01.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.276+0000 7f3ce7dd0640 1 -- 172.21.15.83:0/367960561 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ce00b6380 con 0x7f3ce00cf850 2024-03-06T07:29:01.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.276+0000 7f3cd67fc640 1 -- 172.21.15.83:0/367960561 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cdc011bb0 con 0x7f3ce00cf850 2024-03-06T07:29:01.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.276+0000 7f3ce7dd0640 1 -- 172.21.15.83:0/367960561 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ce00b6940 con 0x7f3ce00cf850 2024-03-06T07:29:01.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.277+0000 7f3ce7dd0640 1 -- 172.21.15.83:0/367960561 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ce0071290 con 0x7f3ce00cf850 2024-03-06T07:29:01.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.277+0000 7f3cd67fc640 1 -- 172.21.15.83:0/367960561 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3cdc01c460 con 0x7f3ce00cf850 2024-03-06T07:29:01.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.278+0000 7f3cd67fc640 1 --2- 172.21.15.83:0/367960561 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3cc406b4d0 0x7f3cc406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.278+0000 7f3cd67fc640 1 -- 172.21.15.83:0/367960561 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3cdc089ef0 con 0x7f3ce00cf850 2024-03-06T07:29:01.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.278+0000 7f3ce5b6c640 1 --2- 172.21.15.83:0/367960561 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3cc406b4d0 0x7f3cc406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.278+0000 7f3ce5b6c640 1 --2- 172.21.15.83:0/367960561 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3cc406b4d0 0x7f3cc406d990 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f3cd0036000 tx=0x7f3cd003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:01.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.282+0000 7f3cd67fc640 1 -- 172.21.15.83:0/367960561 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3cdc0587b0 con 0x7f3ce00cf850 2024-03-06T07:29:01.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.492+0000 7f3ce7dd0640 1 -- 172.21.15.83:0/367960561 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ce00cbb70 con 0x7f3ce00cf850 2024-03-06T07:29:01.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.493+0000 7f3cd67fc640 1 -- 172.21.15.83:0/367960561 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f3cdc05c850 con 0x7f3ce00cf850 2024-03-06T07:29:01.495 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:29:01.495 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":73,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7381,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/3776086145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3776086145},{"type":"v1","addr":"172.21.15.183:6833","nonce":3776086145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":7387,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2457445285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2457445285},{"type":"v1","addr":"172.21.15.183:6835","nonce":2457445285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8628,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:28:54.339089+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:29:01.495 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 73 2024-03-06T07:29:01.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.495+0000 7f3ce7dd0640 1 -- 172.21.15.83:0/367960561 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3cc406b4d0 msgr2=0x7f3cc406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.495+0000 7f3ce7dd0640 1 --2- 172.21.15.83:0/367960561 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3cc406b4d0 0x7f3cc406d990 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f3cd0036000 tx=0x7f3cd003c040 comp rx=0 tx=0).stop 2024-03-06T07:29:01.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.495+0000 7f3ce7dd0640 1 -- 172.21.15.83:0/367960561 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ce00cf850 msgr2=0x7f3ce00cb0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.495+0000 7f3ce7dd0640 1 --2- 172.21.15.83:0/367960561 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ce00cf850 0x7f3ce00cb0b0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc00c430 tx=0x7f3cdc00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.495+0000 7f3ce7dd0640 1 -- 172.21.15.83:0/367960561 shutdown_connections 2024-03-06T07:29:01.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.495+0000 7f3ce7dd0640 1 --2- 172.21.15.83:0/367960561 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3cc406b4d0 0x7f3cc406d990 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.495+0000 7f3ce7dd0640 1 --2- 172.21.15.83:0/367960561 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3ce00d00c0 0x7f3ce00cb5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.495+0000 7f3ce7dd0640 1 --2- 172.21.15.83:0/367960561 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ce00cf850 0x7f3ce00cb0b0 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.495+0000 7f3ce7dd0640 1 --2- 172.21.15.83:0/367960561 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3ce006cf20 0x7f3ce0156170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.495+0000 7f3ce7dd0640 1 -- 172.21.15.83:0/367960561 >> 172.21.15.83:0/367960561 conn(0x7f3ce00b35a0 msgr2=0x7f3ce00b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:01.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.495+0000 7f3ce7dd0640 1 -- 172.21.15.83:0/367960561 shutdown_connections 2024-03-06T07:29:01.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:01.495+0000 7f3ce7dd0640 1 -- 172.21.15.83:0/367960561 wait complete. 2024-03-06T07:29:01.914 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:28:54.339089+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8631}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:29:01.914 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:29:01.914 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:29:02.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.015+0000 7f5e38ceb640 1 Processor -- start 2024-03-06T07:29:02.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.015+0000 7f5e38ceb640 1 -- start start 2024-03-06T07:29:02.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.016+0000 7f5e38ceb640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f5e34162970 0x7f5e340d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.016+0000 7f5e38ceb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e340d22c0 con 0x7f5e34161cf0 2024-03-06T07:29:02.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.016+0000 7f5e38ceb640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e340d25a0 con 0x7f5e340b4460 2024-03-06T07:29:02.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.016+0000 7f5e38ceb640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e34161b00 con 0x7f5e34162970 2024-03-06T07:29:02.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.016+0000 7f5e32d9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f5e34161cf0 0x7f5e340cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49404/0 (socket says 172.21.15.83:49404) 2024-03-06T07:29:02.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.016+0000 7f5e32d9d640 1 -- 172.21.15.83:0/1508683208 learned_addr learned my addr 172.21.15.83:0/1508683208 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:29:02.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.017+0000 7f5e31d9b640 1 --2- 172.21.15.83:0/1508683208 >> v2:172.21.15.183:3300/0 conn(0x7f5e34162970 0x7f5e340d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.017+0000 7f5e3159a640 1 -- 172.21.15.83:0/1508683208 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1308757886 0 0) 0x7f5e340d22c0 con 0x7f5e34161cf0 2024-03-06T07:29:02.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.017+0000 7f5e3159a640 1 -- 172.21.15.83:0/1508683208 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e1c003520 con 0x7f5e34161cf0 2024-03-06T07:29:02.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.017+0000 7f5e3159a640 1 -- 172.21.15.83:0/1508683208 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 358489388 0 0) 0x7f5e1c003520 con 0x7f5e34161cf0 2024-03-06T07:29:02.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.017+0000 7f5e3159a640 1 -- 172.21.15.83:0/1508683208 >> v2:172.21.15.183:3300/0 conn(0x7f5e34162970 msgr2=0x7f5e340d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:02.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.017+0000 7f5e3159a640 1 --2- 172.21.15.83:0/1508683208 >> v2:172.21.15.183:3300/0 conn(0x7f5e34162970 0x7f5e340d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.017+0000 7f5e3159a640 1 -- 172.21.15.83:0/1508683208 >> v1:172.21.15.183:6789/0 conn(0x7f5e340b4460 legacy=0x7f5e340b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.017+0000 7f5e3159a640 1 -- 172.21.15.83:0/1508683208 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e340d2880 con 0x7f5e34161cf0 2024-03-06T07:29:02.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.017+0000 7f5e3159a640 1 -- 172.21.15.83:0/1508683208 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f5e2c0019f0 con 0x7f5e34161cf0 2024-03-06T07:29:02.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.017+0000 7f5e3159a640 1 -- 172.21.15.83:0/1508683208 >> v1:172.21.15.83:6789/0 conn(0x7f5e34161cf0 legacy=0x7f5e340cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.017+0000 7f5e3159a640 1 --2- 172.21.15.83:0/1508683208 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5e1c0039b0 0x7f5e1c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.017+0000 7f5e3159a640 1 --2- 172.21.15.83:0/1508683208 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e1c0043b0 0x7f5e1c004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.017+0000 7f5e3159a640 1 --2- 172.21.15.83:0/1508683208 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5e1c005080 0x7f5e1c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.017+0000 7f5e3159a640 1 -- 172.21.15.83:0/1508683208 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e34161b00 con 0x7f5e1c0043b0 2024-03-06T07:29:02.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.017+0000 7f5e3159a640 1 -- 172.21.15.83:0/1508683208 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1c00bfe0 con 0x7f5e1c0039b0 2024-03-06T07:29:02.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.017+0000 7f5e3159a640 1 -- 172.21.15.83:0/1508683208 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1c00c1e0 con 0x7f5e1c005080 2024-03-06T07:29:02.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.017+0000 7f5e31d9b640 1 --2- 172.21.15.83:0/1508683208 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e1c0043b0 0x7f5e1c004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.017+0000 7f5e3259c640 1 --2- 172.21.15.83:0/1508683208 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5e1c0039b0 0x7f5e1c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.017+0000 7f5e32d9d640 1 --2- 172.21.15.83:0/1508683208 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5e1c005080 0x7f5e1c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.017+0000 7f5e31d9b640 1 -- 172.21.15.83:0/1508683208 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5e1c005080 msgr2=0x7f5e1c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:02.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.017+0000 7f5e31d9b640 1 --2- 172.21.15.83:0/1508683208 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5e1c005080 0x7f5e1c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.017+0000 7f5e31d9b640 1 -- 172.21.15.83:0/1508683208 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5e1c0039b0 msgr2=0x7f5e1c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:02.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.017+0000 7f5e31d9b640 1 --2- 172.21.15.83:0/1508683208 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5e1c0039b0 0x7f5e1c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.017+0000 7f5e31d9b640 1 -- 172.21.15.83:0/1508683208 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e1c003520 con 0x7f5e1c0043b0 2024-03-06T07:29:02.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.017+0000 7f5e3259c640 1 --2- 172.21.15.83:0/1508683208 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5e1c0039b0 0x7f5e1c003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:02.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.017+0000 7f5e32d9d640 1 --2- 172.21.15.83:0/1508683208 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5e1c005080 0x7f5e1c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:02.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.017+0000 7f5e32d9d640 1 -- 172.21.15.83:0/1508683208 reap_dead start 2024-03-06T07:29:02.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.018+0000 7f5e31d9b640 1 --2- 172.21.15.83:0/1508683208 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e1c0043b0 0x7f5e1c004880 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f5e28009a90 tx=0x7f5e1c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e066a06961cabe3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:02.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.018+0000 7f5e3159a640 1 -- 172.21.15.83:0/1508683208 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e28039070 con 0x7f5e1c0043b0 2024-03-06T07:29:02.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.018+0000 7f5e3159a640 1 -- 172.21.15.83:0/1508683208 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e2802c680 con 0x7f5e1c0043b0 2024-03-06T07:29:02.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.018+0000 7f5e3159a640 1 -- 172.21.15.83:0/1508683208 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e28009690 con 0x7f5e1c0043b0 2024-03-06T07:29:02.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.019+0000 7f5e38ceb640 1 -- 172.21.15.83:0/1508683208 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e1c0043b0 msgr2=0x7f5e1c004880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.019+0000 7f5e38ceb640 1 --2- 172.21.15.83:0/1508683208 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e1c0043b0 0x7f5e1c004880 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f5e28009a90 tx=0x7f5e1c005830 comp rx=0 tx=0).stop 2024-03-06T07:29:02.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.019+0000 7f5e38ceb640 1 -- 172.21.15.83:0/1508683208 shutdown_connections 2024-03-06T07:29:02.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.019+0000 7f5e38ceb640 1 --2- 172.21.15.83:0/1508683208 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e1c0043b0 0x7f5e1c004880 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.019+0000 7f5e38ceb640 1 -- 172.21.15.83:0/1508683208 >> 172.21.15.83:0/1508683208 conn(0x7f5e340d3ce0 msgr2=0x7f5e340d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:02.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.019+0000 7f5e38ceb640 1 -- 172.21.15.83:0/1508683208 shutdown_connections 2024-03-06T07:29:02.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.019+0000 7f5e38ceb640 1 -- 172.21.15.83:0/1508683208 wait complete. 2024-03-06T07:29:02.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.021+0000 7f5e38ceb640 1 Processor -- start 2024-03-06T07:29:02.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.021+0000 7f5e38ceb640 1 -- start start 2024-03-06T07:29:02.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.022+0000 7f5e38ceb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5e1c0043b0 0x7f5e34159500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.022+0000 7f5e38ceb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e34158fb0 0x7f5e3414f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.022+0000 7f5e38ceb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5e340b1960 0x7f5e340b1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.022+0000 7f5e38ceb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e340ce950 con 0x7f5e34158fb0 2024-03-06T07:29:02.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.022+0000 7f5e38ceb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e340cecc0 con 0x7f5e1c0043b0 2024-03-06T07:29:02.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.022+0000 7f5e38ceb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e340b4840 con 0x7f5e340b1960 2024-03-06T07:29:02.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.022+0000 7f5e31d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e34158fb0 0x7f5e3414f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.022+0000 7f5e31d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e34158fb0 0x7f5e3414f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52102/0 (socket says 172.21.15.83:52102) 2024-03-06T07:29:02.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.022+0000 7f5e31d9b640 1 -- 172.21.15.83:0/4208779527 learned_addr learned my addr 172.21.15.83:0/4208779527 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:02.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.023+0000 7f5e3259c640 1 --2- 172.21.15.83:0/4208779527 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5e1c0043b0 0x7f5e34159500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.023+0000 7f5e32d9d640 1 --2- 172.21.15.83:0/4208779527 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5e340b1960 0x7f5e340b1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.023+0000 7f5e31d9b640 1 -- 172.21.15.83:0/4208779527 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5e340b1960 msgr2=0x7f5e340b1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.023+0000 7f5e31d9b640 1 --2- 172.21.15.83:0/4208779527 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5e340b1960 0x7f5e340b1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.023+0000 7f5e31d9b640 1 -- 172.21.15.83:0/4208779527 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5e1c0043b0 msgr2=0x7f5e34159500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.023+0000 7f5e31d9b640 1 --2- 172.21.15.83:0/4208779527 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5e1c0043b0 0x7f5e34159500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.023+0000 7f5e31d9b640 1 -- 172.21.15.83:0/4208779527 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e3414f540 con 0x7f5e34158fb0 2024-03-06T07:29:02.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.023+0000 7f5e3259c640 1 --2- 172.21.15.83:0/4208779527 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5e1c0043b0 0x7f5e34159500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:02.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.023+0000 7f5e32d9d640 1 --2- 172.21.15.83:0/4208779527 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5e340b1960 0x7f5e340b1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:02.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.023+0000 7f5e31d9b640 1 --2- 172.21.15.83:0/4208779527 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e34158fb0 0x7f5e3414f000 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f5e28037040 tx=0x7f5e28004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:02.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.023+0000 7f5e22ffd640 1 -- 172.21.15.83:0/4208779527 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e28042020 con 0x7f5e34158fb0 2024-03-06T07:29:02.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.023+0000 7f5e22ffd640 1 -- 172.21.15.83:0/4208779527 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e2803a040 con 0x7f5e34158fb0 2024-03-06T07:29:02.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.023+0000 7f5e38ceb640 1 -- 172.21.15.83:0/4208779527 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e341f74a0 con 0x7f5e34158fb0 2024-03-06T07:29:02.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.023+0000 7f5e22ffd640 1 -- 172.21.15.83:0/4208779527 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e28039070 con 0x7f5e34158fb0 2024-03-06T07:29:02.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.023+0000 7f5e38ceb640 1 -- 172.21.15.83:0/4208779527 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e341f7a60 con 0x7f5e34158fb0 2024-03-06T07:29:02.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.025+0000 7f5e22ffd640 1 -- 172.21.15.83:0/4208779527 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5e28045020 con 0x7f5e34158fb0 2024-03-06T07:29:02.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.025+0000 7f5e22ffd640 1 --2- 172.21.15.83:0/4208779527 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5e1406b500 0x7f5e1406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.025+0000 7f5e3259c640 1 --2- 172.21.15.83:0/4208779527 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5e1406b500 0x7f5e1406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.025+0000 7f5e22ffd640 1 -- 172.21.15.83:0/4208779527 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5e280ab880 con 0x7f5e34158fb0 2024-03-06T07:29:02.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.026+0000 7f5e3259c640 1 --2- 172.21.15.83:0/4208779527 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5e1406b500 0x7f5e1406d9c0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f5e24007b40 tx=0x7f5e24009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:02.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.026+0000 7f5e3b9ae740 1 Processor -- start 2024-03-06T07:29:02.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.026+0000 7f5e3b9ae740 1 -- start start 2024-03-06T07:29:02.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.027+0000 7f5e3b9ae740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56150c318870 0x56150c318c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.027+0000 7f5e3b9ae740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56150c31c8c0 0x56150c31e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.027+0000 7f5e3b9ae740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56150c348600 0x56150c348a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.027+0000 7f5e3b9ae740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x56150c2826b0 con 0x56150c31c8c0 2024-03-06T07:29:02.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.027+0000 7f5e3b9ae740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x56150c22ba60 con 0x56150c348600 2024-03-06T07:29:02.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.027+0000 7f5e3b9ae740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x56150c27e750 con 0x56150c318870 2024-03-06T07:29:02.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.027+0000 7f5e31d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56150c31c8c0 0x56150c31e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.027+0000 7f5e31d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56150c31c8c0 0x56150c31e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52116/0 (socket says 172.21.15.83:52116) 2024-03-06T07:29:02.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.027+0000 7f5e31d9b640 1 -- 172.21.15.83:0/1391039653 learned_addr learned my addr 172.21.15.83:0/1391039653 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:02.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.027+0000 7f5e3259c640 1 --2- 172.21.15.83:0/1391039653 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56150c318870 0x56150c318c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.027+0000 7f5e32d9d640 1 --2- 172.21.15.83:0/1391039653 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56150c348600 0x56150c348a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.027+0000 7f5e31d9b640 1 -- 172.21.15.83:0/1391039653 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56150c318870 msgr2=0x56150c318c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:02.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.027+0000 7f5e31d9b640 1 --2- 172.21.15.83:0/1391039653 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56150c318870 0x56150c318c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.027+0000 7f5e31d9b640 1 -- 172.21.15.83:0/1391039653 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56150c348600 msgr2=0x56150c348a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:02.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.027+0000 7f5e31d9b640 1 --2- 172.21.15.83:0/1391039653 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56150c348600 0x56150c348a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.027+0000 7f5e31d9b640 1 -- 172.21.15.83:0/1391039653 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56150c249f30 con 0x56150c31c8c0 2024-03-06T07:29:02.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.027+0000 7f5e31d9b640 1 --2- 172.21.15.83:0/1391039653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56150c31c8c0 0x56150c31e360 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f5e28082270 tx=0x7f5e28079c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=828760db01b461bd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:02.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.027+0000 7f5e13fff640 1 -- 172.21.15.83:0/1391039653 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2807a790 con 0x56150c31c8c0 2024-03-06T07:29:02.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.028+0000 7f5e13fff640 1 -- 172.21.15.83:0/1391039653 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e2807a090 con 0x56150c31c8c0 2024-03-06T07:29:02.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.028+0000 7f5e13fff640 1 -- 172.21.15.83:0/1391039653 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2807e1e0 con 0x56150c31c8c0 2024-03-06T07:29:02.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.028+0000 7f5e3b9ae740 1 -- 172.21.15.83:0/1391039653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56150c31c8c0 msgr2=0x56150c31e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.028+0000 7f5e3b9ae740 1 --2- 172.21.15.83:0/1391039653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56150c31c8c0 0x56150c31e360 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f5e28082270 tx=0x7f5e28079c90 comp rx=0 tx=0).stop 2024-03-06T07:29:02.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.028+0000 7f5e3b9ae740 1 -- 172.21.15.83:0/1391039653 shutdown_connections 2024-03-06T07:29:02.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.028+0000 7f5e3b9ae740 1 --2- 172.21.15.83:0/1391039653 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56150c348600 0x56150c348a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.028+0000 7f5e3b9ae740 1 --2- 172.21.15.83:0/1391039653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56150c31c8c0 0x56150c31e360 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.028+0000 7f5e3b9ae740 1 --2- 172.21.15.83:0/1391039653 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56150c318870 0x56150c318c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.028+0000 7f5e3b9ae740 1 -- 172.21.15.83:0/1391039653 >> 172.21.15.83:0/1391039653 conn(0x56150c31a8f0 msgr2=0x56150c31c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:02.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.028+0000 7f5e3b9ae740 1 -- 172.21.15.83:0/1391039653 shutdown_connections 2024-03-06T07:29:02.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.028+0000 7f5e3b9ae740 1 -- 172.21.15.83:0/1391039653 wait complete. 2024-03-06T07:29:02.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.029+0000 7f5e3b9ae740 1 -- start start 2024-03-06T07:29:02.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.029+0000 7f5e3b9ae740 1 Processor -- start 2024-03-06T07:29:02.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.030+0000 7f5e3b9ae740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56150c348600 0x56150c34b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.030+0000 7f5e3b9ae740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56150c3403b0 0x56150c3343c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.030+0000 7f5e3b9ae740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56150c334900 0x56150c361160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.030+0000 7f5e3b9ae740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x56150c27e750 con 0x56150c334900 2024-03-06T07:29:02.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.030+0000 7f5e3b9ae740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x56150c23eea0 con 0x56150c3403b0 2024-03-06T07:29:02.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.030+0000 7f5e3b9ae740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x56150c2826b0 con 0x56150c348600 2024-03-06T07:29:02.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.030+0000 7f5e32d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56150c334900 0x56150c361160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.030+0000 7f5e3b9ae740 10 client.0 _ll_register_callbacks cb 0x56150c23e870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:29:02.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.030+0000 7f5e32d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56150c334900 0x56150c361160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52128/0 (socket says 172.21.15.83:52128) 2024-03-06T07:29:02.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.030+0000 7f5e32d9d640 1 -- 172.21.15.83:0/3463156071 learned_addr learned my addr 172.21.15.83:0/3463156071 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:02.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.030+0000 7f5e32d9d640 1 -- 172.21.15.83:0/3463156071 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56150c348600 msgr2=0x56150c34b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:02.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.030+0000 7f5e3259c640 1 --2- 172.21.15.83:0/3463156071 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56150c348600 0x56150c34b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.030+0000 7f5e32d9d640 1 --2- 172.21.15.83:0/3463156071 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56150c348600 0x56150c34b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.030+0000 7f5e31d9b640 1 --2- 172.21.15.83:0/3463156071 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56150c3403b0 0x56150c3343c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.030+0000 7f5e32d9d640 1 -- 172.21.15.83:0/3463156071 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56150c3403b0 msgr2=0x56150c3343c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:02.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.030+0000 7f5e32d9d640 1 --2- 172.21.15.83:0/3463156071 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56150c3403b0 0x56150c3343c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.030+0000 7f5e32d9d640 1 -- 172.21.15.83:0/3463156071 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56150c23f270 con 0x56150c334900 2024-03-06T07:29:02.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.030+0000 7f5e3259c640 1 --2- 172.21.15.83:0/3463156071 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56150c348600 0x56150c34b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:02.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.030+0000 7f5e31d9b640 1 --2- 172.21.15.83:0/3463156071 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56150c3403b0 0x56150c3343c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:02.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.031+0000 7f5e32d9d640 1 --2- 172.21.15.83:0/3463156071 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56150c334900 0x56150c361160 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f5e340c7610 tx=0x7f5e2c007790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e0043866ea20add server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:02.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.031+0000 7f5e117fa640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-06T07:29:02.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.031+0000 7f5e117fa640 1 -- 172.21.15.83:0/3463156071 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2c019060 con 0x56150c334900 2024-03-06T07:29:02.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.031+0000 7f5e117fa640 1 -- 172.21.15.83:0/3463156071 <== mon.0 v2:172.21.15.83:3300/0 2 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5e2c010910 con 0x56150c334900 2024-03-06T07:29:02.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.031+0000 7f5e3b9ae740 1 -- 172.21.15.83:0/3463156071 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5e2807a090 con 0x56150c334900 2024-03-06T07:29:02.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.031+0000 7f5e117fa640 1 -- 172.21.15.83:0/3463156071 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e2c01f050 con 0x56150c334900 2024-03-06T07:29:02.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.031+0000 7f5e3b9ae740 10 client.8802 fetch_fsmap learned FSMap version 73 2024-03-06T07:29:02.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.031+0000 7f5e3b9ae740 1 -- 172.21.15.83:0/3463156071 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=2+}) v3 -- 0x56150bf972d0 con 0x56150c334900 2024-03-06T07:29:02.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.031+0000 7f5e117fa640 1 -- 172.21.15.83:0/3463156071 <== mon.0 v2:172.21.15.83:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e2807a090 con 0x56150c334900 2024-03-06T07:29:02.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.032+0000 7f5e117fa640 1 -- 172.21.15.83:0/3463156071 <== mon.0 v2:172.21.15.83:3300/0 5 ==== fsmap(e 73) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f5e2c022020 con 0x56150c334900 2024-03-06T07:29:02.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.032+0000 7f5e3b9ae740 10 client.8802 fetch_fsmap finished waiting for FSMap version 73 2024-03-06T07:29:02.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.032+0000 7f5e3b9ae740 10 client.8802 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:29:02.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.032+0000 7f5e3b9ae740 20 client.8802 populate_metadata read hostname 'smithi083' 2024-03-06T07:29:02.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.032+0000 7f5e3b9ae740 1 --2- 172.21.15.83:0/3463156071 >> [v2:172.21.15.83:6835/1401317546,v1:172.21.15.83:6837/1401317546] conn(0x56150c349d60 0x56150c34a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.032+0000 7f5e3b9ae740 4 client.8802 mds_command: new command op to 8631 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:29:02.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.032+0000 7f5e3b9ae740 1 -- 172.21.15.83:0/3463156071 --> [v2:172.21.15.83:6835/1401317546,v1:172.21.15.83:6837/1401317546] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56150c28fbf0 con 0x56150c349d60 2024-03-06T07:29:02.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.032+0000 7f5e3259c640 1 --2- 172.21.15.83:0/3463156071 >> [v2:172.21.15.83:6835/1401317546,v1:172.21.15.83:6837/1401317546] conn(0x56150c349d60 0x56150c34a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.032+0000 7f5e117fa640 10 client.8802 ms_handle_connect on v2:172.21.15.83:6835/1401317546 2024-03-06T07:29:02.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.032+0000 7f5e3259c640 1 --2- 172.21.15.83:0/3463156071 >> [v2:172.21.15.83:6835/1401317546,v1:172.21.15.83:6837/1401317546] conn(0x56150c349d60 0x56150c34a140 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f7b10e0f33862c2 server_cookie=a566098c86645ef6 in_seq=0 out_seq=0 2024-03-06T07:29:02.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.033+0000 7f5e117fa640 1 -- 172.21.15.83:0/3463156071 <== mds.0 v2:172.21.15.83:6835/1401317546 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56150c28fbf0 con 0x56150c349d60 2024-03-06T07:29:02.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.033+0000 7f5e117fa640 10 client.8802 handle_command_reply: tid=0 2024-03-06T07:29:02.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.033+0000 7f5e3b9ae740 1 client.8802 shutdown 2024-03-06T07:29:02.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.033+0000 7f5e3b9ae740 1 -- 172.21.15.83:0/3463156071 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56150c334900 msgr2=0x56150c361160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.033+0000 7f5e3b9ae740 1 --2- 172.21.15.83:0/3463156071 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56150c334900 0x56150c361160 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f5e340c7610 tx=0x7f5e2c007790 comp rx=0 tx=0).stop 2024-03-06T07:29:02.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.033+0000 7f5e3b9ae740 1 -- 172.21.15.83:0/3463156071 shutdown_connections 2024-03-06T07:29:02.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.033+0000 7f5e3b9ae740 1 --2- 172.21.15.83:0/3463156071 >> [v2:172.21.15.83:6835/1401317546,v1:172.21.15.83:6837/1401317546] conn(0x56150c349d60 0x56150c34a140 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.033+0000 7f5e3b9ae740 1 --2- 172.21.15.83:0/3463156071 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56150c334900 0x56150c361160 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.033+0000 7f5e3b9ae740 1 --2- 172.21.15.83:0/3463156071 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56150c3403b0 0x56150c3343c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.033+0000 7f5e3b9ae740 1 --2- 172.21.15.83:0/3463156071 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56150c348600 0x56150c34b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.033+0000 7f5e3b9ae740 1 -- 172.21.15.83:0/3463156071 >> 172.21.15.83:0/3463156071 conn(0x56150c318870 msgr2=0x56150c317510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:02.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.033+0000 7f5e117fa640 0 client.8802 ms_handle_reset on v2:172.21.15.83:6835/1401317546 2024-03-06T07:29:02.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.033+0000 7f5e3b9ae740 1 -- 172.21.15.83:0/3463156071 shutdown_connections 2024-03-06T07:29:02.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.033+0000 7f5e3b9ae740 1 -- 172.21.15.83:0/3463156071 wait complete. 2024-03-06T07:29:02.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.033+0000 7f5e3b9ae740 20 client.8802 trim_cache size 0 max 16384 2024-03-06T07:29:02.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.050+0000 7f5e3b9ae740 1 Processor -- start 2024-03-06T07:29:02.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.050+0000 7f5e3b9ae740 1 -- start start 2024-03-06T07:29:02.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.050+0000 7f5e3b9ae740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56150c334900 0x56150c3445f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.050+0000 7f5e3b9ae740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56150c3403b0 0x56150c344b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.050+0000 7f5e3b9ae740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56150c348600 0x56150c34a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.050+0000 7f5e3b9ae740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e341ff950 con 0x56150c334900 2024-03-06T07:29:02.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.050+0000 7f5e3b9ae740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e340ccde0 con 0x56150c348600 2024-03-06T07:29:02.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.050+0000 7f5e3b9ae740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e340cd3e0 con 0x56150c3403b0 2024-03-06T07:29:02.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.050+0000 7f5e3259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56150c334900 0x56150c3445f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.050+0000 7f5e3259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56150c334900 0x56150c3445f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52134/0 (socket says 172.21.15.83:52134) 2024-03-06T07:29:02.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.050+0000 7f5e3259c640 1 -- 172.21.15.83:0/1465321734 learned_addr learned my addr 172.21.15.83:0/1465321734 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:02.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.051+0000 7f5e3259c640 1 -- 172.21.15.83:0/1465321734 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56150c3403b0 msgr2=0x56150c344b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:02.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.051+0000 7f5e3259c640 1 --2- 172.21.15.83:0/1465321734 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56150c3403b0 0x56150c344b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.051+0000 7f5e3259c640 1 -- 172.21.15.83:0/1465321734 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56150c348600 msgr2=0x56150c34a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:02.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.051+0000 7f5e3259c640 1 --2- 172.21.15.83:0/1465321734 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56150c348600 0x56150c34a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.051+0000 7f5e3259c640 1 -- 172.21.15.83:0/1465321734 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56150c0616b0 con 0x56150c334900 2024-03-06T07:29:02.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.051+0000 7f5e3259c640 1 --2- 172.21.15.83:0/1465321734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56150c334900 0x56150c3445f0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f5e2400f5a0 tx=0x7f5e24012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=674d623575a1762c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:02.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.051+0000 7f5e137fe640 1 -- 172.21.15.83:0/1465321734 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2402d070 con 0x56150c334900 2024-03-06T07:29:02.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.051+0000 7f5e137fe640 1 -- 172.21.15.83:0/1465321734 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e2401b030 con 0x56150c334900 2024-03-06T07:29:02.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.051+0000 7f5e137fe640 1 -- 172.21.15.83:0/1465321734 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e24027150 con 0x56150c334900 2024-03-06T07:29:02.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.051+0000 7f5e3b9ae740 1 -- 172.21.15.83:0/1465321734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56150c334900 msgr2=0x56150c3445f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.051+0000 7f5e3b9ae740 1 --2- 172.21.15.83:0/1465321734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56150c334900 0x56150c3445f0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f5e2400f5a0 tx=0x7f5e24012000 comp rx=0 tx=0).stop 2024-03-06T07:29:02.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.051+0000 7f5e3b9ae740 1 -- 172.21.15.83:0/1465321734 shutdown_connections 2024-03-06T07:29:02.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.051+0000 7f5e3b9ae740 1 --2- 172.21.15.83:0/1465321734 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56150c348600 0x56150c34a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.051+0000 7f5e3b9ae740 1 --2- 172.21.15.83:0/1465321734 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56150c3403b0 0x56150c344b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.051+0000 7f5e3b9ae740 1 --2- 172.21.15.83:0/1465321734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56150c334900 0x56150c3445f0 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.051+0000 7f5e3b9ae740 1 -- 172.21.15.83:0/1465321734 >> 172.21.15.83:0/1465321734 conn(0x56150c318870 msgr2=0x56150c34aee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:02.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.051+0000 7f5e3b9ae740 1 -- 172.21.15.83:0/1465321734 shutdown_connections 2024-03-06T07:29:02.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.051+0000 7f5e3b9ae740 1 -- 172.21.15.83:0/1465321734 wait complete. 2024-03-06T07:29:02.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.052+0000 7f5e3b9ae740 1 -- start start 2024-03-06T07:29:02.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.052+0000 7f5e3b9ae740 1 Processor -- start 2024-03-06T07:29:02.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.053+0000 7f5e3b9ae740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56150c334900 0x56150c3d2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.053+0000 7f5e3b9ae740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56150c3403b0 0x56150c34a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.053+0000 7f5e3b9ae740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56150c348600 0x56150c34a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.053+0000 7f5e3b9ae740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x56150c345a00 con 0x56150c348600 2024-03-06T07:29:02.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.053+0000 7f5e3b9ae740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x56150c22ba60 con 0x56150c3403b0 2024-03-06T07:29:02.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.053+0000 7f5e3b9ae740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x56150c364610 con 0x56150c334900 2024-03-06T07:29:02.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.053+0000 7f5e3b9ae740 10 client.0 _ll_register_callbacks cb 0x7f5e2c004f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:29:02.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.053+0000 7f5e32d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56150c348600 0x56150c34a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.053+0000 7f5e32d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56150c348600 0x56150c34a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52150/0 (socket says 172.21.15.83:52150) 2024-03-06T07:29:02.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.053+0000 7f5e32d9d640 1 -- 172.21.15.83:0/166762856 learned_addr learned my addr 172.21.15.83:0/166762856 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:02.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.053+0000 7f5e32d9d640 1 -- 172.21.15.83:0/166762856 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56150c334900 msgr2=0x56150c3d2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:02.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.053+0000 7f5e32d9d640 1 --2- 172.21.15.83:0/166762856 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56150c334900 0x56150c3d2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.053+0000 7f5e32d9d640 1 -- 172.21.15.83:0/166762856 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56150c3403b0 msgr2=0x56150c34a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:02.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.053+0000 7f5e32d9d640 1 --2- 172.21.15.83:0/166762856 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56150c3403b0 0x56150c34a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.053+0000 7f5e32d9d640 1 -- 172.21.15.83:0/166762856 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56150c362e70 con 0x56150c348600 2024-03-06T07:29:02.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.053+0000 7f5e32d9d640 1 --2- 172.21.15.83:0/166762856 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56150c348600 0x56150c34a680 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c0077d0 tx=0x7f5e340cec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5af5aa428124a29f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:02.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.053+0000 7f5e117fa640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-06T07:29:02.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.054+0000 7f5e117fa640 1 -- 172.21.15.83:0/166762856 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2c00d030 con 0x56150c348600 2024-03-06T07:29:02.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.054+0000 7f5e117fa640 1 -- 172.21.15.83:0/166762856 <== mon.0 v2:172.21.15.83:3300/0 2 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5e2c014810 con 0x56150c348600 2024-03-06T07:29:02.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.054+0000 7f5e3b9ae740 1 -- 172.21.15.83:0/166762856 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5e2401b030 con 0x56150c348600 2024-03-06T07:29:02.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.054+0000 7f5e117fa640 1 -- 172.21.15.83:0/166762856 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e2c033050 con 0x56150c348600 2024-03-06T07:29:02.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.054+0000 7f5e3b9ae740 10 client.8808 fetch_fsmap learned FSMap version 73 2024-03-06T07:29:02.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.054+0000 7f5e3b9ae740 1 -- 172.21.15.83:0/166762856 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=2+}) v3 -- 0x56150c344650 con 0x56150c348600 2024-03-06T07:29:02.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.054+0000 7f5e117fa640 1 -- 172.21.15.83:0/166762856 <== mon.0 v2:172.21.15.83:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e2401b030 con 0x56150c348600 2024-03-06T07:29:02.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.054+0000 7f5e117fa640 1 -- 172.21.15.83:0/166762856 <== mon.0 v2:172.21.15.83:3300/0 5 ==== fsmap(e 73) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f5e2c036020 con 0x56150c348600 2024-03-06T07:29:02.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.054+0000 7f5e3b9ae740 10 client.8808 fetch_fsmap finished waiting for FSMap version 73 2024-03-06T07:29:02.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.054+0000 7f5e3b9ae740 10 client.8808 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:29:02.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.054+0000 7f5e3b9ae740 20 client.8808 populate_metadata read hostname 'smithi083' 2024-03-06T07:29:02.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.054+0000 7f5e3b9ae740 1 --2- 172.21.15.83:0/166762856 >> [v2:172.21.15.83:6835/1401317546,v1:172.21.15.83:6837/1401317546] conn(0x56150c3874b0 0x56150c337000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.054+0000 7f5e3b9ae740 4 client.8808 mds_command: new command op to 8631 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:29:02.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.054+0000 7f5e3b9ae740 1 -- 172.21.15.83:0/166762856 --> [v2:172.21.15.83:6835/1401317546,v1:172.21.15.83:6837/1401317546] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56150c289f40 con 0x56150c3874b0 2024-03-06T07:29:02.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.054+0000 7f5e3259c640 1 --2- 172.21.15.83:0/166762856 >> [v2:172.21.15.83:6835/1401317546,v1:172.21.15.83:6837/1401317546] conn(0x56150c3874b0 0x56150c337000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.054+0000 7f5e117fa640 10 client.8808 ms_handle_connect on v2:172.21.15.83:6835/1401317546 2024-03-06T07:29:02.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.055+0000 7f5e3259c640 1 --2- 172.21.15.83:0/166762856 >> [v2:172.21.15.83:6835/1401317546,v1:172.21.15.83:6837/1401317546] conn(0x56150c3874b0 0x56150c337000 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2ac273d7ddb38dd server_cookie=c5308fded9abcc92 in_seq=0 out_seq=0 2024-03-06T07:29:02.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.055+0000 7f5e117fa640 1 -- 172.21.15.83:0/166762856 <== mds.0 v2:172.21.15.83:6835/1401317546 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x56150c289f40 con 0x56150c3874b0 2024-03-06T07:29:02.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.055+0000 7f5e117fa640 10 client.8808 handle_command_reply: tid=0 2024-03-06T07:29:02.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.055+0000 7f5e3b9ae740 1 client.8808 shutdown 2024-03-06T07:29:02.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.055+0000 7f5e3b9ae740 1 -- 172.21.15.83:0/166762856 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56150c348600 msgr2=0x56150c34a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.055+0000 7f5e3b9ae740 1 --2- 172.21.15.83:0/166762856 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56150c348600 0x56150c34a680 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c0077d0 tx=0x7f5e340cec30 comp rx=0 tx=0).stop 2024-03-06T07:29:02.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.055+0000 7f5e3b9ae740 1 -- 172.21.15.83:0/166762856 shutdown_connections 2024-03-06T07:29:02.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.055+0000 7f5e3b9ae740 1 --2- 172.21.15.83:0/166762856 >> [v2:172.21.15.83:6835/1401317546,v1:172.21.15.83:6837/1401317546] conn(0x56150c3874b0 0x56150c337000 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.055+0000 7f5e3b9ae740 1 --2- 172.21.15.83:0/166762856 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56150c348600 0x56150c34a680 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.055+0000 7f5e3b9ae740 1 --2- 172.21.15.83:0/166762856 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56150c3403b0 0x56150c34a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.055+0000 7f5e117fa640 0 client.8808 ms_handle_reset on v2:172.21.15.83:6835/1401317546 2024-03-06T07:29:02.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.055+0000 7f5e3b9ae740 1 --2- 172.21.15.83:0/166762856 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x56150c334900 0x56150c3d2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.055+0000 7f5e3b9ae740 1 -- 172.21.15.83:0/166762856 >> 172.21.15.83:0/166762856 conn(0x56150c318870 msgr2=0x56150c31c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:02.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.055+0000 7f5e3b9ae740 1 -- 172.21.15.83:0/166762856 shutdown_connections 2024-03-06T07:29:02.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.055+0000 7f5e3b9ae740 1 -- 172.21.15.83:0/166762856 wait complete. 2024-03-06T07:29:02.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.055+0000 7f5e3b9ae740 20 client.8808 trim_cache size 0 max 16384 2024-03-06T07:29:02.063 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:29:02.063 INFO:teuthology.orchestra.run.smithi083.stdout: "cluster_fsid": "039877dd-d2ce-45d2-8e77-ac5eb7e1d55a", 2024-03-06T07:29:02.063 INFO:teuthology.orchestra.run.smithi083.stdout: "whoami": 0, 2024-03-06T07:29:02.063 INFO:teuthology.orchestra.run.smithi083.stdout: "id": 8631, 2024-03-06T07:29:02.063 INFO:teuthology.orchestra.run.smithi083.stdout: "want_state": "up:active", 2024-03-06T07:29:02.063 INFO:teuthology.orchestra.run.smithi083.stdout: "state": "up:active", 2024-03-06T07:29:02.064 INFO:teuthology.orchestra.run.smithi083.stdout: "fs_name": "cephfs", 2024-03-06T07:29:02.064 INFO:teuthology.orchestra.run.smithi083.stdout: "rank_uptime": 9.0964928129999993, 2024-03-06T07:29:02.064 INFO:teuthology.orchestra.run.smithi083.stdout: "mdsmap_epoch": 73, 2024-03-06T07:29:02.064 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_epoch": 128, 2024-03-06T07:29:02.064 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_epoch_barrier": 128, 2024-03-06T07:29:02.064 INFO:teuthology.orchestra.run.smithi083.stdout: "uptime": 20.469228682000001 2024-03-06T07:29:02.064 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:29:02.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.056+0000 7f5e38ceb640 1 -- 172.21.15.83:0/4208779527 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5e1406b500 msgr2=0x7f5e1406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.056+0000 7f5e38ceb640 1 --2- 172.21.15.83:0/4208779527 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5e1406b500 0x7f5e1406d9c0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f5e24007b40 tx=0x7f5e24009040 comp rx=0 tx=0).stop 2024-03-06T07:29:02.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.056+0000 7f5e38ceb640 1 -- 172.21.15.83:0/4208779527 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e34158fb0 msgr2=0x7f5e3414f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.056+0000 7f5e38ceb640 1 --2- 172.21.15.83:0/4208779527 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e34158fb0 0x7f5e3414f000 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f5e28037040 tx=0x7f5e28004730 comp rx=0 tx=0).stop 2024-03-06T07:29:02.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.056+0000 7f5e38ceb640 1 -- 172.21.15.83:0/4208779527 shutdown_connections 2024-03-06T07:29:02.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.056+0000 7f5e38ceb640 1 --2- 172.21.15.83:0/4208779527 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5e1406b500 0x7f5e1406d9c0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.056+0000 7f5e38ceb640 1 --2- 172.21.15.83:0/4208779527 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5e340b1960 0x7f5e340b1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.056+0000 7f5e38ceb640 1 --2- 172.21.15.83:0/4208779527 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e34158fb0 0x7f5e3414f000 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.056+0000 7f5e38ceb640 1 --2- 172.21.15.83:0/4208779527 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5e1c0043b0 0x7f5e34159500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.056+0000 7f5e38ceb640 1 -- 172.21.15.83:0/4208779527 >> 172.21.15.83:0/4208779527 conn(0x7f5e340d3ce0 msgr2=0x7f5e340d02b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:02.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.056+0000 7f5e38ceb640 1 -- 172.21.15.83:0/4208779527 shutdown_connections 2024-03-06T07:29:02.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.056+0000 7f5e38ceb640 1 -- 172.21.15.83:0/4208779527 wait complete. 2024-03-06T07:29:02.068 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:29:02.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.167+0000 7f085790c640 1 Processor -- start 2024-03-06T07:29:02.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.167+0000 7f085790c640 1 -- start start 2024-03-06T07:29:02.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.168+0000 7f085790c640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f085015bd20 0x7f08500724b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.168+0000 7f085790c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f085015f8f0 0x7f08500ce290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.168+0000 7f085790c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0850072ad0 con 0x7f085006afb0 2024-03-06T07:29:02.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.168+0000 7f085790c640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0850072db0 con 0x7f085015bd20 2024-03-06T07:29:02.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.168+0000 7f085790c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0850073090 con 0x7f085015f8f0 2024-03-06T07:29:02.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.169+0000 7f0855ea9640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f085006afb0 0x7f08500ccfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49416/0 (socket says 172.21.15.83:49416) 2024-03-06T07:29:02.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.169+0000 7f0855ea9640 1 -- 172.21.15.83:0/154360519 learned_addr learned my addr 172.21.15.83:0/154360519 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:29:02.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.169+0000 7f0854ea7640 1 --2- 172.21.15.83:0/154360519 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f085015f8f0 0x7f08500ce290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.169+0000 7f0847fff640 1 -- 172.21.15.83:0/154360519 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2565498076 0 0) 0x7f0850072ad0 con 0x7f085006afb0 2024-03-06T07:29:02.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.169+0000 7f0847fff640 1 -- 172.21.15.83:0/154360519 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f083c003520 con 0x7f085006afb0 2024-03-06T07:29:02.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.169+0000 7f08556a8640 1 --2- 172.21.15.83:0/154360519 >> v2:172.21.15.183:3300/0 conn(0x7f085015bd20 0x7f08500724b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.169+0000 7f0847fff640 1 -- 172.21.15.83:0/154360519 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2337328134 0 0) 0x7f083c003520 con 0x7f085006afb0 2024-03-06T07:29:02.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.169+0000 7f0847fff640 1 -- 172.21.15.83:0/154360519 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f085015f8f0 msgr2=0x7f08500ce290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:02.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.169+0000 7f0847fff640 1 --2- 172.21.15.83:0/154360519 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f085015f8f0 0x7f08500ce290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.169+0000 7f0847fff640 1 -- 172.21.15.83:0/154360519 >> v2:172.21.15.183:3300/0 conn(0x7f085015bd20 msgr2=0x7f08500724b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:02.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.169+0000 7f0847fff640 1 --2- 172.21.15.83:0/154360519 >> v2:172.21.15.183:3300/0 conn(0x7f085015bd20 0x7f08500724b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.169+0000 7f0847fff640 1 -- 172.21.15.83:0/154360519 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0850073370 con 0x7f085006afb0 2024-03-06T07:29:02.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.169+0000 7f0847fff640 1 -- 172.21.15.83:0/154360519 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f08480019f0 con 0x7f085006afb0 2024-03-06T07:29:02.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.169+0000 7f0847fff640 1 -- 172.21.15.83:0/154360519 >> v1:172.21.15.83:6789/0 conn(0x7f085006afb0 legacy=0x7f08500ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.169+0000 7f0847fff640 1 --2- 172.21.15.83:0/154360519 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f085015f8f0 0x7f083c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.169+0000 7f0847fff640 1 --2- 172.21.15.83:0/154360519 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f083c003f30 0x7f083c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.169+0000 7f0847fff640 1 --2- 172.21.15.83:0/154360519 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f083c004d40 0x7f083c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.169+0000 7f0847fff640 1 -- 172.21.15.83:0/154360519 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0850072db0 con 0x7f083c003f30 2024-03-06T07:29:02.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.169+0000 7f0847fff640 1 -- 172.21.15.83:0/154360519 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0850073090 con 0x7f083c004d40 2024-03-06T07:29:02.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.169+0000 7f0847fff640 1 -- 172.21.15.83:0/154360519 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f083c00b9c0 con 0x7f085015f8f0 2024-03-06T07:29:02.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.169+0000 7f0854ea7640 1 --2- 172.21.15.83:0/154360519 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f083c003f30 0x7f083c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.169+0000 7f08556a8640 1 --2- 172.21.15.83:0/154360519 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f085015f8f0 0x7f083c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.169+0000 7f0855ea9640 1 --2- 172.21.15.83:0/154360519 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f083c004d40 0x7f083c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.170+0000 7f0854ea7640 1 -- 172.21.15.83:0/154360519 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f085015f8f0 msgr2=0x7f083c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:02.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.170+0000 7f0854ea7640 1 --2- 172.21.15.83:0/154360519 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f085015f8f0 0x7f083c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.170+0000 7f0854ea7640 1 -- 172.21.15.83:0/154360519 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f083c004d40 msgr2=0x7f083c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:02.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.170+0000 7f0854ea7640 1 --2- 172.21.15.83:0/154360519 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f083c004d40 0x7f083c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.170+0000 7f0854ea7640 1 -- 172.21.15.83:0/154360519 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f083c003520 con 0x7f083c003f30 2024-03-06T07:29:02.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.170+0000 7f0854ea7640 1 --2- 172.21.15.83:0/154360519 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f083c003f30 0x7f083c004800 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f084c009ac0 tx=0x7f083c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59fab39d4c6328a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:02.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.170+0000 7f0847fff640 1 -- 172.21.15.83:0/154360519 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f084c039070 con 0x7f083c003f30 2024-03-06T07:29:02.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.170+0000 7f0847fff640 1 -- 172.21.15.83:0/154360519 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f084c02c6b0 con 0x7f083c003f30 2024-03-06T07:29:02.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.170+0000 7f0847fff640 1 -- 172.21.15.83:0/154360519 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f084c009690 con 0x7f083c003f30 2024-03-06T07:29:02.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.171+0000 7f085790c640 1 -- 172.21.15.83:0/154360519 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f083c003f30 msgr2=0x7f083c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.171+0000 7f085790c640 1 --2- 172.21.15.83:0/154360519 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f083c003f30 0x7f083c004800 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f084c009ac0 tx=0x7f083c005830 comp rx=0 tx=0).stop 2024-03-06T07:29:02.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.171+0000 7f0855ea9640 1 -- 172.21.15.83:0/154360519 reap_dead start 2024-03-06T07:29:02.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.171+0000 7f085790c640 1 -- 172.21.15.83:0/154360519 shutdown_connections 2024-03-06T07:29:02.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.171+0000 7f085790c640 1 -- 172.21.15.83:0/154360519 >> 172.21.15.83:0/154360519 conn(0x7f08500b1d00 msgr2=0x7f08500b2110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:02.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.171+0000 7f085790c640 1 -- 172.21.15.83:0/154360519 shutdown_connections 2024-03-06T07:29:02.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.171+0000 7f085790c640 1 -- 172.21.15.83:0/154360519 wait complete. 2024-03-06T07:29:02.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.172+0000 7f085790c640 1 Processor -- start 2024-03-06T07:29:02.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.172+0000 7f085790c640 1 -- start start 2024-03-06T07:29:02.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.173+0000 7f085790c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f083c003f30 0x7f08500c3820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.173+0000 7f085790c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08500c8a60 0x7f08501f49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.173+0000 7f085790c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f08500ae3a0 0x7f08500ae820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.173+0000 7f085790c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f085015c100 con 0x7f08500c8a60 2024-03-06T07:29:02.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.173+0000 7f085790c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08500672f0 con 0x7f08500ae3a0 2024-03-06T07:29:02.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.173+0000 7f085790c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08500cd6c0 con 0x7f083c003f30 2024-03-06T07:29:02.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.173+0000 7f0854ea7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08500c8a60 0x7f08501f49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.173+0000 7f0854ea7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08500c8a60 0x7f08501f49e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52164/0 (socket says 172.21.15.83:52164) 2024-03-06T07:29:02.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.173+0000 7f0854ea7640 1 -- 172.21.15.83:0/2490126737 learned_addr learned my addr 172.21.15.83:0/2490126737 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:02.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.173+0000 7f08556a8640 1 --2- 172.21.15.83:0/2490126737 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f083c003f30 0x7f08500c3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.173+0000 7f0855ea9640 1 --2- 172.21.15.83:0/2490126737 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f08500ae3a0 0x7f08500ae820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.173+0000 7f0854ea7640 1 -- 172.21.15.83:0/2490126737 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f083c003f30 msgr2=0x7f08500c3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.173+0000 7f0854ea7640 1 --2- 172.21.15.83:0/2490126737 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f083c003f30 0x7f08500c3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.173+0000 7f0854ea7640 1 -- 172.21.15.83:0/2490126737 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f08500ae3a0 msgr2=0x7f08500ae820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.173+0000 7f0854ea7640 1 --2- 172.21.15.83:0/2490126737 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f08500ae3a0 0x7f08500ae820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.173+0000 7f0854ea7640 1 -- 172.21.15.83:0/2490126737 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08501f5280 con 0x7f08500c8a60 2024-03-06T07:29:02.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.173+0000 7f0855ea9640 1 --2- 172.21.15.83:0/2490126737 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f08500ae3a0 0x7f08500ae820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:02.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.173+0000 7f08556a8640 1 --2- 172.21.15.83:0/2490126737 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f083c003f30 0x7f08500c3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:02.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.174+0000 7f0854ea7640 1 --2- 172.21.15.83:0/2490126737 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08500c8a60 0x7f08501f49e0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f084c037040 tx=0x7f084c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:02.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.174+0000 7f0845ffb640 1 -- 172.21.15.83:0/2490126737 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f084c042020 con 0x7f08500c8a60 2024-03-06T07:29:02.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.174+0000 7f0845ffb640 1 -- 172.21.15.83:0/2490126737 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f084c03a040 con 0x7f08500c8a60 2024-03-06T07:29:02.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.174+0000 7f085790c640 1 -- 172.21.15.83:0/2490126737 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08501f5520 con 0x7f08500c8a60 2024-03-06T07:29:02.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.174+0000 7f0845ffb640 1 -- 172.21.15.83:0/2490126737 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f084c039070 con 0x7f08500c8a60 2024-03-06T07:29:02.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.174+0000 7f085790c640 1 -- 172.21.15.83:0/2490126737 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08501f5ae0 con 0x7f08500c8a60 2024-03-06T07:29:02.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.175+0000 7f085790c640 1 -- 172.21.15.83:0/2490126737 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f085004f330 con 0x7f08500c8a60 2024-03-06T07:29:02.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.175+0000 7f0845ffb640 1 -- 172.21.15.83:0/2490126737 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f084c045020 con 0x7f08500c8a60 2024-03-06T07:29:02.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.176+0000 7f0845ffb640 1 --2- 172.21.15.83:0/2490126737 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f083406b500 0x7f083406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.176+0000 7f08556a8640 1 --2- 172.21.15.83:0/2490126737 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f083406b500 0x7f083406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.176+0000 7f0845ffb640 1 -- 172.21.15.83:0/2490126737 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f084c0ab520 con 0x7f08500c8a60 2024-03-06T07:29:02.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.176+0000 7f08556a8640 1 --2- 172.21.15.83:0/2490126737 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f083406b500 0x7f083406d9c0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f0840009c40 tx=0x7f0840008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:02.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.184+0000 7f0845ffb640 1 -- 172.21.15.83:0/2490126737 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f084c04c070 con 0x7f08500c8a60 2024-03-06T07:29:02.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.398+0000 7f085790c640 1 -- 172.21.15.83:0/2490126737 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f0850074580 con 0x7f08500c8a60 2024-03-06T07:29:02.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.398+0000 7f0845ffb640 1 -- 172.21.15.83:0/2490126737 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v73) v1 ==== 53+0+73 (secure 0 0 0) 0x7f084c07a430 con 0x7f08500c8a60 2024-03-06T07:29:02.400 INFO:teuthology.orchestra.run.smithi083.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:29:02.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.400+0000 7f085790c640 1 -- 172.21.15.83:0/2490126737 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f083406b500 msgr2=0x7f083406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.400+0000 7f085790c640 1 --2- 172.21.15.83:0/2490126737 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f083406b500 0x7f083406d9c0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f0840009c40 tx=0x7f0840008040 comp rx=0 tx=0).stop 2024-03-06T07:29:02.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.400+0000 7f085790c640 1 -- 172.21.15.83:0/2490126737 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08500c8a60 msgr2=0x7f08501f49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.400+0000 7f085790c640 1 --2- 172.21.15.83:0/2490126737 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08500c8a60 0x7f08501f49e0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f084c037040 tx=0x7f084c004730 comp rx=0 tx=0).stop 2024-03-06T07:29:02.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.400+0000 7f085790c640 1 -- 172.21.15.83:0/2490126737 shutdown_connections 2024-03-06T07:29:02.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.400+0000 7f085790c640 1 --2- 172.21.15.83:0/2490126737 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f083406b500 0x7f083406d9c0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.400+0000 7f085790c640 1 --2- 172.21.15.83:0/2490126737 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f08500ae3a0 0x7f08500ae820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.400+0000 7f085790c640 1 --2- 172.21.15.83:0/2490126737 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08500c8a60 0x7f08501f49e0 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.400+0000 7f085790c640 1 --2- 172.21.15.83:0/2490126737 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f083c003f30 0x7f08500c3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.400+0000 7f085790c640 1 -- 172.21.15.83:0/2490126737 >> 172.21.15.83:0/2490126737 conn(0x7f08500b1d00 msgr2=0x7f08500cb4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:02.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.400+0000 7f085790c640 1 -- 172.21.15.83:0/2490126737 shutdown_connections 2024-03-06T07:29:02.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:02.400+0000 7f085790c640 1 -- 172.21.15.83:0/2490126737 wait complete. 2024-03-06T07:29:02.413 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:29:02.413 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:29:02.413 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi083.front.sepia.ceph.com 2024-03-06T07:29:02.413 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:29:02.414 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:29:02.414 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:29:02.414 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:29:02.414 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:29:02.414 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:29:02.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:02.414 DEBUG:teuthology.orchestra.run.smithi083:> ip addr 2024-03-06T07:29:02.429 INFO:teuthology.orchestra.run.smithi083.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:29:02.429 INFO:teuthology.orchestra.run.smithi083.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:29:02.429 INFO:teuthology.orchestra.run.smithi083.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:29:02.429 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:02.429 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 ::1/128 scope host 2024-03-06T07:29:02.429 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:02.429 INFO:teuthology.orchestra.run.smithi083.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:29:02.429 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:d8:da:1e brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:02.429 INFO:teuthology.orchestra.run.smithi083.stdout: altname enp7s0f0 2024-03-06T07:29:02.429 INFO:teuthology.orchestra.run.smithi083.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:29:02.429 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:d8:da:1f brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:02.429 INFO:teuthology.orchestra.run.smithi083.stdout: altname enp7s0f1 2024-03-06T07:29:02.429 INFO:teuthology.orchestra.run.smithi083.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:29:02.430 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:8f:ce:90 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:02.430 INFO:teuthology.orchestra.run.smithi083.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:29:02.430 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 0c:c4:7a:8f:ce:91 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:02.430 INFO:teuthology.orchestra.run.smithi083.stdout: inet 172.21.15.83/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:29:02.430 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft 42313sec preferred_lft 42313sec 2024-03-06T07:29:02.430 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::ec4:7aff:fe8f:ce91/64 scope link 2024-03-06T07:29:02.430 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:02.430 INFO:teuthology.orchestra.run.smithi083.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:29:02.430 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 1e:29:46:c9:e7:c8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:02.430 INFO:teuthology.orchestra.run.smithi083.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:29:02.430 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:02.430 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::60c5:d7ff:fe70:2b1c/64 scope link 2024-03-06T07:29:02.430 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:02.430 INFO:teuthology.orchestra.run.smithi083.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:29:02.430 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 1e:29:46:c9:e7:c8 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:29:02.431 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::1c29:46ff:fec9:e7c8/64 scope link 2024-03-06T07:29:02.431 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:02.431 INFO:teuthology.orchestra.run.smithi083.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:29:02.431 INFO:teuthology.orchestra.run.smithi083.stdout: link/ether 3e:00:2f:23:d5:45 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:29:02.431 INFO:teuthology.orchestra.run.smithi083.stdout: inet6 fe80::3c00:2fff:fe23:d545/64 scope link 2024-03-06T07:29:02.431 INFO:teuthology.orchestra.run.smithi083.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:02.431 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:02.431 DEBUG:teuthology.orchestra.run.smithi083:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:29:02.494 INFO:teuthology.orchestra.run.smithi083.stdout:1 2024-03-06T07:29:02.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:02.496 DEBUG:teuthology.orchestra.run.smithi083:> route 2024-03-06T07:29:02.551 INFO:teuthology.orchestra.run.smithi083.stdout:Kernel IP routing table 2024-03-06T07:29:02.551 INFO:teuthology.orchestra.run.smithi083.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:29:02.551 INFO:teuthology.orchestra.run.smithi083.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:29:02.551 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:29:02.551 INFO:teuthology.orchestra.run.smithi083.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:29:02.551 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:02.551 DEBUG:teuthology.orchestra.run.smithi083:> (cd / && exec bash -c ' 2024-03-06T07:29:02.551 DEBUG:teuthology.orchestra.run.smithi083:> set -e 2024-03-06T07:29:02.551 DEBUG:teuthology.orchestra.run.smithi083:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:29:02.552 DEBUG:teuthology.orchestra.run.smithi083:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:29:02.552 DEBUG:teuthology.orchestra.run.smithi083:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:29:02.552 DEBUG:teuthology.orchestra.run.smithi083:> ') 2024-03-06T07:29:02.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:02.645 DEBUG:teuthology.orchestra.run.smithi083:> ip netns list 2024-03-06T07:29:02.659 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:29:02.659 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:29:02.660 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:29:02.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:02.660 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip link set brx.0 up 2024-03-06T07:29:02.726 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:29:02.726 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:29:02.726 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:29:02.781 INFO:teuthology.orchestra.run.smithi083.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:29:02.781 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:29:02.781 DEBUG:teuthology.orchestra.run.smithi083:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:29:02.835 DEBUG:teuthology.orchestra.run.smithi083:> sudo modprobe fuse 2024-03-06T07:29:02.902 DEBUG:teuthology.orchestra.run.smithi083:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:29:02.958 INFO:teuthology.orchestra.run.smithi083.stdout:/proc 2024-03-06T07:29:02.958 INFO:teuthology.orchestra.run.smithi083.stdout:/sys 2024-03-06T07:29:02.958 INFO:teuthology.orchestra.run.smithi083.stdout:/dev 2024-03-06T07:29:02.958 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/security 2024-03-06T07:29:02.958 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/shm 2024-03-06T07:29:02.958 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/pts 2024-03-06T07:29:02.958 INFO:teuthology.orchestra.run.smithi083.stdout:/run 2024-03-06T07:29:02.958 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/cgroup 2024-03-06T07:29:02.959 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/pstore 2024-03-06T07:29:02.959 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/bpf 2024-03-06T07:29:02.959 INFO:teuthology.orchestra.run.smithi083.stdout:/ 2024-03-06T07:29:02.959 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/selinux 2024-03-06T07:29:02.959 INFO:teuthology.orchestra.run.smithi083.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:29:02.959 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/mqueue 2024-03-06T07:29:02.959 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/hugepages 2024-03-06T07:29:02.959 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/debug 2024-03-06T07:29:02.959 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/tracing 2024-03-06T07:29:02.959 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:29:02.960 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/fuse/connections 2024-03-06T07:29:02.960 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/config 2024-03-06T07:29:02.960 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:29:02.960 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:29:02.960 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:29:02.960 INFO:teuthology.orchestra.run.smithi083.stdout:/run/user/1000 2024-03-06T07:29:02.960 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph 2024-03-06T07:29:02.960 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:29:02.960 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:29:02.960 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:29:02.960 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:29:02.960 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns 2024-03-06T07:29:02.960 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:29:02.961 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:29:02.961 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:29:02.961 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:29:02.961 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:02.961 DEBUG:teuthology.orchestra.run.smithi083:> ls /sys/fs/fuse/connections 2024-03-06T07:29:03.013 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:29:03.013 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:29:03.056 DEBUG:teuthology.orchestra.run.smithi083:> sudo modprobe fuse 2024-03-06T07:29:03.082 DEBUG:teuthology.orchestra.run.smithi083:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:29:03.138 INFO:teuthology.orchestra.run.smithi083.stdout:/proc 2024-03-06T07:29:03.138 INFO:teuthology.orchestra.run.smithi083.stdout:/sys 2024-03-06T07:29:03.138 INFO:teuthology.orchestra.run.smithi083.stdout:/dev 2024-03-06T07:29:03.139 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/security 2024-03-06T07:29:03.139 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/shm 2024-03-06T07:29:03.139 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/pts 2024-03-06T07:29:03.139 INFO:teuthology.orchestra.run.smithi083.stdout:/run 2024-03-06T07:29:03.139 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/cgroup 2024-03-06T07:29:03.139 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/pstore 2024-03-06T07:29:03.139 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/bpf 2024-03-06T07:29:03.139 INFO:teuthology.orchestra.run.smithi083.stdout:/ 2024-03-06T07:29:03.139 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/selinux 2024-03-06T07:29:03.139 INFO:teuthology.orchestra.run.smithi083.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:29:03.139 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/mqueue 2024-03-06T07:29:03.139 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/hugepages 2024-03-06T07:29:03.139 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/debug 2024-03-06T07:29:03.140 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/tracing 2024-03-06T07:29:03.140 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:29:03.140 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/fs/fuse/connections 2024-03-06T07:29:03.140 INFO:teuthology.orchestra.run.smithi083.stdout:/sys/kernel/config 2024-03-06T07:29:03.140 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:29:03.140 INFO:teuthology.orchestra.run.smithi083.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:29:03.140 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:29:03.140 INFO:teuthology.orchestra.run.smithi083.stdout:/run/user/1000 2024-03-06T07:29:03.140 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph 2024-03-06T07:29:03.140 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:29:03.140 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:29:03.141 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:29:03.141 INFO:teuthology.orchestra.run.smithi083.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:29:03.141 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns 2024-03-06T07:29:03.141 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:29:03.141 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:29:03.141 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:29:03.141 INFO:teuthology.orchestra.run.smithi083.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:29:03.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:03.141 DEBUG:teuthology.orchestra.run.smithi083:> ls /sys/fs/fuse/connections 2024-03-06T07:29:03.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:2024-03-06T07:29:03.141+0000 7fbd9722a480 -1 init, newargv = 0x55f4ef35dc20 newargc=15 2024-03-06T07:29:03.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:ceph-fuse[79136]: starting ceph client 2024-03-06T07:29:03.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:ceph-fuse[79136]: starting fuse 2024-03-06T07:29:03.157 INFO:teuthology.orchestra.run.smithi083.stdout:41 2024-03-06T07:29:03.157 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-06T07:29:03.157 DEBUG:teuthology.orchestra.run.smithi083:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:29:03.157 DEBUG:teuthology.orchestra.run.smithi083:> import glob 2024-03-06T07:29:03.157 DEBUG:teuthology.orchestra.run.smithi083:> import re 2024-03-06T07:29:03.157 DEBUG:teuthology.orchestra.run.smithi083:> import os 2024-03-06T07:29:03.158 DEBUG:teuthology.orchestra.run.smithi083:> import subprocess 2024-03-06T07:29:03.158 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:29:03.158 DEBUG:teuthology.orchestra.run.smithi083:> def _find_admin_socket(client_name): 2024-03-06T07:29:03.158 DEBUG:teuthology.orchestra.run.smithi083:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:29:03.158 DEBUG:teuthology.orchestra.run.smithi083:> files = glob.glob(asok_path) 2024-03-06T07:29:03.158 DEBUG:teuthology.orchestra.run.smithi083:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:29:03.158 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:29:03.158 DEBUG:teuthology.orchestra.run.smithi083:> # Given a non-glob path, it better be there 2024-03-06T07:29:03.158 DEBUG:teuthology.orchestra.run.smithi083:> if "*" not in asok_path: 2024-03-06T07:29:03.158 DEBUG:teuthology.orchestra.run.smithi083:> assert(len(files) == 1) 2024-03-06T07:29:03.158 DEBUG:teuthology.orchestra.run.smithi083:> return files[0] 2024-03-06T07:29:03.158 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:29:03.158 DEBUG:teuthology.orchestra.run.smithi083:> for f in files: 2024-03-06T07:29:03.158 DEBUG:teuthology.orchestra.run.smithi083:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:29:03.158 DEBUG:teuthology.orchestra.run.smithi083:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:29:03.158 DEBUG:teuthology.orchestra.run.smithi083:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:29:03.158 DEBUG:teuthology.orchestra.run.smithi083:> contents = proc_f.read() 2024-03-06T07:29:03.159 DEBUG:teuthology.orchestra.run.smithi083:> if mountpoint in contents: 2024-03-06T07:29:03.159 DEBUG:teuthology.orchestra.run.smithi083:> return f 2024-03-06T07:29:03.159 DEBUG:teuthology.orchestra.run.smithi083:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:29:03.159 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:29:03.159 DEBUG:teuthology.orchestra.run.smithi083:> print(_find_admin_socket("client.0")) 2024-03-06T07:29:03.159 DEBUG:teuthology.orchestra.run.smithi083:> ' 2024-03-06T07:29:03.281 INFO:teuthology.orchestra.run.smithi083.stdout:/var/run/ceph/ceph-client.0.79136.asok 2024-03-06T07:29:03.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:29:03.291 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.79136.asok 2024-03-06T07:29:03.292 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:03.292 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.79136.asok status 2024-03-06T07:29:03.428 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:29:03.428 INFO:teuthology.orchestra.run.smithi083.stdout: "metadata": { 2024-03-06T07:29:03.428 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:29:03.428 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:29:03.428 INFO:teuthology.orchestra.run.smithi083.stdout: "entity_id": "0", 2024-03-06T07:29:03.428 INFO:teuthology.orchestra.run.smithi083.stdout: "hostname": "smithi083", 2024-03-06T07:29:03.429 INFO:teuthology.orchestra.run.smithi083.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:29:03.429 INFO:teuthology.orchestra.run.smithi083.stdout: "pid": "79136", 2024-03-06T07:29:03.429 INFO:teuthology.orchestra.run.smithi083.stdout: "root": "/" 2024-03-06T07:29:03.429 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:29:03.429 INFO:teuthology.orchestra.run.smithi083.stdout: "dentry_count": 0, 2024-03-06T07:29:03.429 INFO:teuthology.orchestra.run.smithi083.stdout: "dentry_pinned_count": 0, 2024-03-06T07:29:03.429 INFO:teuthology.orchestra.run.smithi083.stdout: "id": 8820, 2024-03-06T07:29:03.429 INFO:teuthology.orchestra.run.smithi083.stdout: "inst": { 2024-03-06T07:29:03.429 INFO:teuthology.orchestra.run.smithi083.stdout: "name": { 2024-03-06T07:29:03.429 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "client", 2024-03-06T07:29:03.429 INFO:teuthology.orchestra.run.smithi083.stdout: "num": 8820 2024-03-06T07:29:03.430 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:29:03.430 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": { 2024-03-06T07:29:03.430 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-03-06T07:29:03.430 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:29:03.430 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 3824777380 2024-03-06T07:29:03.430 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-06T07:29:03.430 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:29:03.430 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": { 2024-03-06T07:29:03.430 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-03-06T07:29:03.430 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:29:03.431 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 3824777380 2024-03-06T07:29:03.431 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-06T07:29:03.431 INFO:teuthology.orchestra.run.smithi083.stdout: "inst_str": "client.8820 192.168.0.1:0/3824777380", 2024-03-06T07:29:03.431 INFO:teuthology.orchestra.run.smithi083.stdout: "addr_str": "192.168.0.1:0/3824777380", 2024-03-06T07:29:03.431 INFO:teuthology.orchestra.run.smithi083.stdout: "inode_count": 1, 2024-03-06T07:29:03.431 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_epoch": 73, 2024-03-06T07:29:03.431 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_epoch": 128, 2024-03-06T07:29:03.431 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:29:03.431 INFO:teuthology.orchestra.run.smithi083.stdout: "blocklisted": false, 2024-03-06T07:29:03.431 INFO:teuthology.orchestra.run.smithi083.stdout: "fs_name": "cephfs" 2024-03-06T07:29:03.431 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:29:03.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:03.437 DEBUG:teuthology.orchestra.run.smithi083:> stat --file-system '--printf=%T 2024-03-06T07:29:03.437 DEBUG:teuthology.orchestra.run.smithi083:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:29:03.452 INFO:teuthology.orchestra.run.smithi083.stdout:fuseblk 2024-03-06T07:29:03.452 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:29:03.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:03.453 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:29:03.520 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2024-03-06T07:29:03.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.661+0000 7fc9505d6640 1 Processor -- start 2024-03-06T07:29:03.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.661+0000 7fc9505d6640 1 -- start start 2024-03-06T07:29:03.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.662+0000 7fc9505d6640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fc94815aec0 0x7fc9480ad720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:03.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.662+0000 7fc9505d6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9480b0f70 con 0x7fc9480b0930 2024-03-06T07:29:03.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.662+0000 7fc9505d6640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9480adc60 con 0x7fc94806c5b0 2024-03-06T07:29:03.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.662+0000 7fc9505d6640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc94815c520 con 0x7fc94815aec0 2024-03-06T07:29:03.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.662+0000 7fc94eb73640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fc9480b0930 0x7fc94806b670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49418/0 (socket says 172.21.15.83:49418) 2024-03-06T07:29:03.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.662+0000 7fc94eb73640 1 -- 172.21.15.83:0/4270838650 learned_addr learned my addr 172.21.15.83:0/4270838650 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:29:03.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.662+0000 7fc94d370640 1 -- 172.21.15.83:0/4270838650 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 667666214 0 0) 0x7fc9480b0f70 con 0x7fc9480b0930 2024-03-06T07:29:03.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.662+0000 7fc94e372640 1 --2- 172.21.15.83:0/4270838650 >> v2:172.21.15.183:3300/0 conn(0x7fc94815aec0 0x7fc9480ad720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:03.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.662+0000 7fc94d370640 1 -- 172.21.15.83:0/4270838650 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc934003520 con 0x7fc9480b0930 2024-03-06T07:29:03.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.662+0000 7fc94d370640 1 -- 172.21.15.83:0/4270838650 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 342373624 0 0) 0x7fc934003520 con 0x7fc9480b0930 2024-03-06T07:29:03.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.662+0000 7fc94d370640 1 -- 172.21.15.83:0/4270838650 >> v2:172.21.15.183:3300/0 conn(0x7fc94815aec0 msgr2=0x7fc9480ad720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:03.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.662+0000 7fc94d370640 1 --2- 172.21.15.83:0/4270838650 >> v2:172.21.15.183:3300/0 conn(0x7fc94815aec0 0x7fc9480ad720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.662+0000 7fc94d370640 1 -- 172.21.15.83:0/4270838650 >> v1:172.21.15.183:6789/0 conn(0x7fc94806c5b0 legacy=0x7fc9480b0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:03.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.662+0000 7fc94d370640 1 -- 172.21.15.83:0/4270838650 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc94815bf60 con 0x7fc9480b0930 2024-03-06T07:29:03.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.662+0000 7fc94d370640 1 -- 172.21.15.83:0/4270838650 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7fc9400019f0 con 0x7fc9480b0930 2024-03-06T07:29:03.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.662+0000 7fc94d370640 1 -- 172.21.15.83:0/4270838650 >> v1:172.21.15.83:6789/0 conn(0x7fc9480b0930 legacy=0x7fc94806b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:03.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.663+0000 7fc94d370640 1 --2- 172.21.15.83:0/4270838650 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc9340039b0 0x7fc934003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:03.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.663+0000 7fc94d370640 1 --2- 172.21.15.83:0/4270838650 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc934004800 0x7fc934004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:03.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.663+0000 7fc94d370640 1 --2- 172.21.15.83:0/4270838650 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc9340051c0 0x7fc93400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:03.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.663+0000 7fc94d370640 1 -- 172.21.15.83:0/4270838650 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc93400c030 con 0x7fc934004800 2024-03-06T07:29:03.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.663+0000 7fc94d370640 1 -- 172.21.15.83:0/4270838650 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc93400c1e0 con 0x7fc9340039b0 2024-03-06T07:29:03.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.663+0000 7fc94d370640 1 -- 172.21.15.83:0/4270838650 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc93400c3e0 con 0x7fc9340051c0 2024-03-06T07:29:03.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.663+0000 7fc94db71640 1 --2- 172.21.15.83:0/4270838650 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc934004800 0x7fc934004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:03.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.663+0000 7fc94e372640 1 --2- 172.21.15.83:0/4270838650 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc9340039b0 0x7fc934003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:03.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.663+0000 7fc94eb73640 1 --2- 172.21.15.83:0/4270838650 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc9340051c0 0x7fc93400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:03.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.663+0000 7fc94db71640 1 -- 172.21.15.83:0/4270838650 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc9340051c0 msgr2=0x7fc93400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:03.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.663+0000 7fc94db71640 1 --2- 172.21.15.83:0/4270838650 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc9340051c0 0x7fc93400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.663+0000 7fc94db71640 1 -- 172.21.15.83:0/4270838650 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc9340039b0 msgr2=0x7fc934003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:03.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.663+0000 7fc94db71640 1 --2- 172.21.15.83:0/4270838650 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc9340039b0 0x7fc934003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.663+0000 7fc94db71640 1 -- 172.21.15.83:0/4270838650 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc934003520 con 0x7fc934004800 2024-03-06T07:29:03.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.663+0000 7fc94eb73640 1 -- 172.21.15.83:0/4270838650 reap_dead start 2024-03-06T07:29:03.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.664+0000 7fc94db71640 1 --2- 172.21.15.83:0/4270838650 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc934004800 0x7fc934004c80 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fc938001750 tx=0x7fc934005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cfb2c4a484600f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:03.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.664+0000 7fc94d370640 1 -- 172.21.15.83:0/4270838650 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc938001370 con 0x7fc934004800 2024-03-06T07:29:03.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.664+0000 7fc94d370640 1 -- 172.21.15.83:0/4270838650 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9380052d0 con 0x7fc934004800 2024-03-06T07:29:03.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.664+0000 7fc94d370640 1 -- 172.21.15.83:0/4270838650 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc938003830 con 0x7fc934004800 2024-03-06T07:29:03.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.664+0000 7fc9505d6640 1 -- 172.21.15.83:0/4270838650 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc934004800 msgr2=0x7fc934004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:03.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.664+0000 7fc9505d6640 1 --2- 172.21.15.83:0/4270838650 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc934004800 0x7fc934004c80 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fc938001750 tx=0x7fc934005830 comp rx=0 tx=0).stop 2024-03-06T07:29:03.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.664+0000 7fc9505d6640 1 -- 172.21.15.83:0/4270838650 shutdown_connections 2024-03-06T07:29:03.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.664+0000 7fc9505d6640 1 --2- 172.21.15.83:0/4270838650 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc934004800 0x7fc934004c80 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.664+0000 7fc9505d6640 1 -- 172.21.15.83:0/4270838650 >> 172.21.15.83:0/4270838650 conn(0x7fc9480b4fe0 msgr2=0x7fc9480b56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:03.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.664+0000 7fc9505d6640 1 -- 172.21.15.83:0/4270838650 shutdown_connections 2024-03-06T07:29:03.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.664+0000 7fc9505d6640 1 -- 172.21.15.83:0/4270838650 wait complete. 2024-03-06T07:29:03.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.665+0000 7fc9505d6640 1 Processor -- start 2024-03-06T07:29:03.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.666+0000 7fc9505d6640 1 -- start start 2024-03-06T07:29:03.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.666+0000 7fc9505d6640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc934004800 0x7fc94814f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:03.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.666+0000 7fc9505d6640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc9481532f0 0x7fc948155390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:03.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.666+0000 7fc9505d6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc948153770 0x7fc9481558d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:03.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.666+0000 7fc9505d6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9480b5db0 con 0x7fc948153770 2024-03-06T07:29:03.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.666+0000 7fc9505d6640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc94806b150 con 0x7fc9481532f0 2024-03-06T07:29:03.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.666+0000 7fc9505d6640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9480b5f30 con 0x7fc934004800 2024-03-06T07:29:03.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.666+0000 7fc94eb73640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc948153770 0x7fc9481558d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:03.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.666+0000 7fc94eb73640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc948153770 0x7fc9481558d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52186/0 (socket says 172.21.15.83:52186) 2024-03-06T07:29:03.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.666+0000 7fc94eb73640 1 -- 172.21.15.83:0/3620127558 learned_addr learned my addr 172.21.15.83:0/3620127558 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:03.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.667+0000 7fc94e372640 1 --2- 172.21.15.83:0/3620127558 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc934004800 0x7fc94814f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:03.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.667+0000 7fc94db71640 1 --2- 172.21.15.83:0/3620127558 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc9481532f0 0x7fc948155390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:03.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.667+0000 7fc94eb73640 1 -- 172.21.15.83:0/3620127558 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc934004800 msgr2=0x7fc94814f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:03.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.667+0000 7fc94eb73640 1 --2- 172.21.15.83:0/3620127558 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc934004800 0x7fc94814f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.667+0000 7fc94eb73640 1 -- 172.21.15.83:0/3620127558 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc9481532f0 msgr2=0x7fc948155390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:03.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.667+0000 7fc94eb73640 1 --2- 172.21.15.83:0/3620127558 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc9481532f0 0x7fc948155390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.667+0000 7fc94eb73640 1 -- 172.21.15.83:0/3620127558 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc948153df0 con 0x7fc948153770 2024-03-06T07:29:03.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.667+0000 7fc94db71640 1 --2- 172.21.15.83:0/3620127558 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc9481532f0 0x7fc948155390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:03.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.667+0000 7fc94eb73640 1 --2- 172.21.15.83:0/3620127558 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc948153770 0x7fc9481558d0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7fc9480b5510 tx=0x7fc940007b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:03.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.667+0000 7fc93effd640 1 -- 172.21.15.83:0/3620127558 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc94000ab60 con 0x7fc948153770 2024-03-06T07:29:03.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.667+0000 7fc93effd640 1 -- 172.21.15.83:0/3620127558 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc940018070 con 0x7fc948153770 2024-03-06T07:29:03.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.667+0000 7fc9505d6640 1 -- 172.21.15.83:0/3620127558 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9481540d0 con 0x7fc948153770 2024-03-06T07:29:03.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.667+0000 7fc93effd640 1 -- 172.21.15.83:0/3620127558 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9400133f0 con 0x7fc948153770 2024-03-06T07:29:03.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.667+0000 7fc9505d6640 1 -- 172.21.15.83:0/3620127558 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc948154690 con 0x7fc948153770 2024-03-06T07:29:03.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.668+0000 7fc9505d6640 1 -- 172.21.15.83:0/3620127558 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc94809e4e0 con 0x7fc948153770 2024-03-06T07:29:03.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.669+0000 7fc93effd640 1 -- 172.21.15.83:0/3620127558 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc940013590 con 0x7fc948153770 2024-03-06T07:29:03.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.669+0000 7fc93effd640 1 --2- 172.21.15.83:0/3620127558 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc92c06b500 0x7fc92c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:03.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.669+0000 7fc93effd640 1 -- 172.21.15.83:0/3620127558 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc94008a6d0 con 0x7fc948153770 2024-03-06T07:29:03.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.669+0000 7fc94e372640 1 --2- 172.21.15.83:0/3620127558 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc92c06b500 0x7fc92c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:03.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.669+0000 7fc94e372640 1 --2- 172.21.15.83:0/3620127558 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc92c06b500 0x7fc92c06d9c0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fc9480cf400 tx=0x7fc944008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:03.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.678+0000 7fc93effd640 1 -- 172.21.15.83:0/3620127558 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc9400595e0 con 0x7fc948153770 2024-03-06T07:29:03.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.888+0000 7fc9505d6640 1 -- 172.21.15.83:0/3620127558 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7fc948068ff0 con 0x7fc948153770 2024-03-06T07:29:03.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.996+0000 7fc93effd640 1 -- 172.21.15.83:0/3620127558 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v74) v1 ==== 105+0+0 (secure 0 0 0) 0x7fc940058ee0 con 0x7fc948153770 2024-03-06T07:29:04.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.998+0000 7fc9505d6640 1 -- 172.21.15.83:0/3620127558 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc92c06b500 msgr2=0x7fc92c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.998+0000 7fc9505d6640 1 --2- 172.21.15.83:0/3620127558 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc92c06b500 0x7fc92c06d9c0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fc9480cf400 tx=0x7fc944008040 comp rx=0 tx=0).stop 2024-03-06T07:29:04.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.998+0000 7fc9505d6640 1 -- 172.21.15.83:0/3620127558 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc948153770 msgr2=0x7fc9481558d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.998+0000 7fc9505d6640 1 --2- 172.21.15.83:0/3620127558 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc948153770 0x7fc9481558d0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7fc9480b5510 tx=0x7fc940007b00 comp rx=0 tx=0).stop 2024-03-06T07:29:04.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.998+0000 7fc9505d6640 1 -- 172.21.15.83:0/3620127558 shutdown_connections 2024-03-06T07:29:04.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.998+0000 7fc9505d6640 1 --2- 172.21.15.83:0/3620127558 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc92c06b500 0x7fc92c06d9c0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.998+0000 7fc9505d6640 1 --2- 172.21.15.83:0/3620127558 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc948153770 0x7fc9481558d0 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.998+0000 7fc9505d6640 1 --2- 172.21.15.83:0/3620127558 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc9481532f0 0x7fc948155390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.998+0000 7fc9505d6640 1 --2- 172.21.15.83:0/3620127558 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc934004800 0x7fc94814f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.998+0000 7fc9505d6640 1 -- 172.21.15.83:0/3620127558 >> 172.21.15.83:0/3620127558 conn(0x7fc9480b4fe0 msgr2=0x7fc9480b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:04.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.998+0000 7fc9505d6640 1 -- 172.21.15.83:0/3620127558 shutdown_connections 2024-03-06T07:29:04.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:03.998+0000 7fc9505d6640 1 -- 172.21.15.83:0/3620127558 wait complete. 2024-03-06T07:29:04.015 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:04.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.116+0000 7fd18ceb5640 1 Processor -- start 2024-03-06T07:29:04.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.117+0000 7fd18ceb5640 1 -- start start 2024-03-06T07:29:04.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.117+0000 7fd18ceb5640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd18815dfe0 0x7fd1880cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.117+0000 7fd18ceb5640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd1880d29c0 0x7fd1880b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.117+0000 7fd18ceb5640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1880b3430 con 0x7fd1880cf830 2024-03-06T07:29:04.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.117+0000 7fd18ceb5640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1880b31e0 con 0x7fd18815dfe0 2024-03-06T07:29:04.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.117+0000 7fd18ceb5640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1880b3d60 con 0x7fd1880d29c0 2024-03-06T07:29:04.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.118+0000 7fd18659c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd18815dfe0 0x7fd1880cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.118+0000 7fd185d9b640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fd1880cf830 0x7fd1880d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49426/0 (socket says 172.21.15.83:49426) 2024-03-06T07:29:04.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.118+0000 7fd185d9b640 1 -- 172.21.15.83:0/1634578824 learned_addr learned my addr 172.21.15.83:0/1634578824 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:29:04.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.118+0000 7fd18659c640 1 -- 172.21.15.83:0/1634578824 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd1880d29c0 msgr2=0x7fd1880b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:04.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.118+0000 7fd186d9d640 1 --2- 172.21.15.83:0/1634578824 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd1880d29c0 0x7fd1880b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.118+0000 7fd18659c640 1 --2- 172.21.15.83:0/1634578824 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd1880d29c0 0x7fd1880b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.118+0000 7fd18559a640 1 -- 172.21.15.83:0/1634578824 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3054411816 0 0) 0x7fd1880b3430 con 0x7fd1880cf830 2024-03-06T07:29:04.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.118+0000 7fd18659c640 1 -- 172.21.15.83:0/1634578824 >> v1:172.21.15.83:6789/0 conn(0x7fd1880cf830 legacy=0x7fd1880d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.118+0000 7fd18659c640 1 -- 172.21.15.83:0/1634578824 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1880b3710 con 0x7fd18815dfe0 2024-03-06T07:29:04.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.118+0000 7fd186d9d640 1 --2- 172.21.15.83:0/1634578824 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd1880d29c0 0x7fd1880b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:04.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.119+0000 7fd18659c640 1 --2- 172.21.15.83:0/1634578824 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd18815dfe0 0x7fd1880cf2f0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7fd180009d90 tx=0x7fd18002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23b6dd4066145c6e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:04.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.119+0000 7fd18559a640 1 -- 172.21.15.83:0/1634578824 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd180039070 con 0x7fd18815dfe0 2024-03-06T07:29:04.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.119+0000 7fd18559a640 1 -- 172.21.15.83:0/1634578824 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd18002cc50 con 0x7fd18815dfe0 2024-03-06T07:29:04.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.119+0000 7fd18559a640 1 -- 172.21.15.83:0/1634578824 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd180034440 con 0x7fd18815dfe0 2024-03-06T07:29:04.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.119+0000 7fd18ceb5640 1 -- 172.21.15.83:0/1634578824 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd18815dfe0 msgr2=0x7fd1880cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.119+0000 7fd18ceb5640 1 --2- 172.21.15.83:0/1634578824 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd18815dfe0 0x7fd1880cf2f0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7fd180009d90 tx=0x7fd18002bea0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.119+0000 7fd18ceb5640 1 -- 172.21.15.83:0/1634578824 shutdown_connections 2024-03-06T07:29:04.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.119+0000 7fd18ceb5640 1 --2- 172.21.15.83:0/1634578824 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd1880d29c0 0x7fd1880b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.119+0000 7fd18ceb5640 1 --2- 172.21.15.83:0/1634578824 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd18815dfe0 0x7fd1880cf2f0 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.119+0000 7fd18ceb5640 1 -- 172.21.15.83:0/1634578824 >> 172.21.15.83:0/1634578824 conn(0x7fd1880d3cd0 msgr2=0x7fd1880d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:04.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.119+0000 7fd18ceb5640 1 -- 172.21.15.83:0/1634578824 shutdown_connections 2024-03-06T07:29:04.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.119+0000 7fd18ceb5640 1 -- 172.21.15.83:0/1634578824 wait complete. 2024-03-06T07:29:04.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.121+0000 7fd18ceb5640 1 Processor -- start 2024-03-06T07:29:04.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.121+0000 7fd18ceb5640 1 -- start start 2024-03-06T07:29:04.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.121+0000 7fd18ceb5640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd1880cf830 0x7fd18815a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.121+0000 7fd18ceb5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1880d29c0 0x7fd18809eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.121+0000 7fd18ceb5640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd18815dfe0 0x7fd1881f3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.121+0000 7fd18ceb5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1880d1df0 con 0x7fd1880d29c0 2024-03-06T07:29:04.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.121+0000 7fd18ceb5640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1880d1c70 con 0x7fd1880cf830 2024-03-06T07:29:04.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.121+0000 7fd18ceb5640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd18815e3c0 con 0x7fd18815dfe0 2024-03-06T07:29:04.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.122+0000 7fd185d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1880d29c0 0x7fd18809eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.122+0000 7fd185d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1880d29c0 0x7fd18809eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52214/0 (socket says 172.21.15.83:52214) 2024-03-06T07:29:04.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.122+0000 7fd185d9b640 1 -- 172.21.15.83:0/3639732560 learned_addr learned my addr 172.21.15.83:0/3639732560 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:04.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.122+0000 7fd186d9d640 1 --2- 172.21.15.83:0/3639732560 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd18815dfe0 0x7fd1881f3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.122+0000 7fd18659c640 1 --2- 172.21.15.83:0/3639732560 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd1880cf830 0x7fd18815a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.122+0000 7fd185d9b640 1 -- 172.21.15.83:0/3639732560 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd18815dfe0 msgr2=0x7fd1881f3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.122+0000 7fd185d9b640 1 --2- 172.21.15.83:0/3639732560 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd18815dfe0 0x7fd1881f3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.122+0000 7fd185d9b640 1 -- 172.21.15.83:0/3639732560 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd1880cf830 msgr2=0x7fd18815a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.122+0000 7fd185d9b640 1 --2- 172.21.15.83:0/3639732560 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd1880cf830 0x7fd18815a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.122+0000 7fd185d9b640 1 -- 172.21.15.83:0/3639732560 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd188161650 con 0x7fd1880d29c0 2024-03-06T07:29:04.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.122+0000 7fd18659c640 1 --2- 172.21.15.83:0/3639732560 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd1880cf830 0x7fd18815a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:04.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.122+0000 7fd185d9b640 1 --2- 172.21.15.83:0/3639732560 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1880d29c0 0x7fd18809eb60 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7fd17c00c4b0 tx=0x7fd17c00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:04.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.122+0000 7fd176ffd640 1 -- 172.21.15.83:0/3639732560 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17c018070 con 0x7fd1880d29c0 2024-03-06T07:29:04.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.122+0000 7fd176ffd640 1 -- 172.21.15.83:0/3639732560 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd17c007500 con 0x7fd1880d29c0 2024-03-06T07:29:04.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.122+0000 7fd18ceb5640 1 -- 172.21.15.83:0/3639732560 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd18815ac90 con 0x7fd1880d29c0 2024-03-06T07:29:04.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.122+0000 7fd176ffd640 1 -- 172.21.15.83:0/3639732560 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17c004a20 con 0x7fd1880d29c0 2024-03-06T07:29:04.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.122+0000 7fd18ceb5640 1 -- 172.21.15.83:0/3639732560 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd18815b530 con 0x7fd1880d29c0 2024-03-06T07:29:04.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.123+0000 7fd18ceb5640 1 -- 172.21.15.83:0/3639732560 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd18815b080 con 0x7fd1880d29c0 2024-03-06T07:29:04.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.124+0000 7fd176ffd640 1 -- 172.21.15.83:0/3639732560 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd17c01b070 con 0x7fd1880d29c0 2024-03-06T07:29:04.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.125+0000 7fd176ffd640 1 --2- 172.21.15.83:0/3639732560 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd16806b4d0 0x7fd16806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.125+0000 7fd176ffd640 1 -- 172.21.15.83:0/3639732560 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd17c0893f0 con 0x7fd1880d29c0 2024-03-06T07:29:04.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.125+0000 7fd18659c640 1 --2- 172.21.15.83:0/3639732560 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd16806b4d0 0x7fd16806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.125+0000 7fd18659c640 1 --2- 172.21.15.83:0/3639732560 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd16806b4d0 0x7fd16806d990 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fd180036000 tx=0x7fd180030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:04.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.129+0000 7fd176ffd640 1 -- 172.21.15.83:0/3639732560 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd17c057d70 con 0x7fd1880d29c0 2024-03-06T07:29:04.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.343+0000 7fd18ceb5640 1 -- 172.21.15.83:0/3639732560 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1880d1010 con 0x7fd1880d29c0 2024-03-06T07:29:04.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.344+0000 7fd176ffd640 1 -- 172.21.15.83:0/3639732560 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+5001 (secure 0 0 0) 0x7fd17c05be10 con 0x7fd1880d29c0 2024-03-06T07:29:04.346 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:29:04.346 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":75,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7381,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/3776086145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3776086145},{"type":"v1","addr":"172.21.15.183:6833","nonce":3776086145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":7387,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2457445285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2457445285},{"type":"v1","addr":"172.21.15.183:6835","nonce":2457445285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:03.999598+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:creating","state_seq":1,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:29:04.346 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 75 2024-03-06T07:29:04.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.346+0000 7fd18ceb5640 1 -- 172.21.15.83:0/3639732560 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd16806b4d0 msgr2=0x7fd16806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.346+0000 7fd18ceb5640 1 --2- 172.21.15.83:0/3639732560 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd16806b4d0 0x7fd16806d990 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fd180036000 tx=0x7fd180030000 comp rx=0 tx=0).stop 2024-03-06T07:29:04.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.346+0000 7fd18ceb5640 1 -- 172.21.15.83:0/3639732560 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1880d29c0 msgr2=0x7fd18809eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.346+0000 7fd18ceb5640 1 --2- 172.21.15.83:0/3639732560 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1880d29c0 0x7fd18809eb60 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7fd17c00c4b0 tx=0x7fd17c00c970 comp rx=0 tx=0).stop 2024-03-06T07:29:04.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.346+0000 7fd18ceb5640 1 -- 172.21.15.83:0/3639732560 shutdown_connections 2024-03-06T07:29:04.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.346+0000 7fd18ceb5640 1 --2- 172.21.15.83:0/3639732560 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd16806b4d0 0x7fd16806d990 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.346+0000 7fd18ceb5640 1 --2- 172.21.15.83:0/3639732560 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd18815dfe0 0x7fd1881f3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.346+0000 7fd18ceb5640 1 --2- 172.21.15.83:0/3639732560 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1880d29c0 0x7fd18809eb60 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.346+0000 7fd18ceb5640 1 --2- 172.21.15.83:0/3639732560 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd1880cf830 0x7fd18815a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.346+0000 7fd18ceb5640 1 -- 172.21.15.83:0/3639732560 >> 172.21.15.83:0/3639732560 conn(0x7fd1880d3cd0 msgr2=0x7fd1880cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:04.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.346+0000 7fd18ceb5640 1 -- 172.21.15.83:0/3639732560 shutdown_connections 2024-03-06T07:29:04.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:04.346+0000 7fd18ceb5640 1 -- 172.21.15.83:0/3639732560 wait complete. 2024-03-06T07:29:04.359 DEBUG:tasks.ceph_test_case:wait_until_equal: 1 != 2, waiting (timeout=30)... 2024-03-06T07:29:06.920 INFO:tasks.ceph.mds.a.smithi083.stderr:2024-03-06T07:29:06.918+0000 7fa26e0b8640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-06T07:29:09.361 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:09.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.498+0000 7f7fc8864640 1 Processor -- start 2024-03-06T07:29:09.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.498+0000 7f7fc8864640 1 -- start start 2024-03-06T07:29:09.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.499+0000 7f7fc8864640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f7fc40d2270 0x7f7fc40d2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.499+0000 7f7fc8864640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc40b3840 con 0x7f7fc40b4600 2024-03-06T07:29:09.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.499+0000 7f7fc8864640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc40b3b20 con 0x7f7fc41617e0 2024-03-06T07:29:09.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.499+0000 7f7fc8864640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fc40b35f0 con 0x7f7fc40d2270 2024-03-06T07:29:09.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.499+0000 7f7fc2d9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f7fc40b4600 0x7f7fc40b4a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:51124/0 (socket says 172.21.15.83:51124) 2024-03-06T07:29:09.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.499+0000 7f7fc2d9d640 1 -- 172.21.15.83:0/4067442584 learned_addr learned my addr 172.21.15.83:0/4067442584 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:29:09.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.500+0000 7f7fc1d9b640 1 --2- 172.21.15.83:0/4067442584 >> v2:172.21.15.183:3300/0 conn(0x7f7fc40d2270 0x7f7fc40d2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.500+0000 7f7fc159a640 1 -- 172.21.15.83:0/4067442584 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2555839496 0 0) 0x7f7fc40b3840 con 0x7f7fc40b4600 2024-03-06T07:29:09.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.500+0000 7f7fc159a640 1 -- 172.21.15.83:0/4067442584 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fac003520 con 0x7f7fc40b4600 2024-03-06T07:29:09.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.500+0000 7f7fc159a640 1 -- 172.21.15.83:0/4067442584 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 692963868 0 0) 0x7f7fac003520 con 0x7f7fc40b4600 2024-03-06T07:29:09.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.500+0000 7f7fc159a640 1 -- 172.21.15.83:0/4067442584 >> v2:172.21.15.183:3300/0 conn(0x7f7fc40d2270 msgr2=0x7f7fc40d2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:09.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.500+0000 7f7fc159a640 1 --2- 172.21.15.83:0/4067442584 >> v2:172.21.15.183:3300/0 conn(0x7f7fc40d2270 0x7f7fc40d2700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.500+0000 7f7fc159a640 1 -- 172.21.15.83:0/4067442584 >> v1:172.21.15.183:6789/0 conn(0x7f7fc41617e0 legacy=0x7f7fc40cf460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.500+0000 7f7fc159a640 1 -- 172.21.15.83:0/4067442584 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc40b3e00 con 0x7f7fc40b4600 2024-03-06T07:29:09.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.500+0000 7f7fc159a640 1 -- 172.21.15.83:0/4067442584 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f7fbc0019f0 con 0x7f7fc40b4600 2024-03-06T07:29:09.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.500+0000 7f7fc159a640 1 -- 172.21.15.83:0/4067442584 >> v1:172.21.15.83:6789/0 conn(0x7f7fc40b4600 legacy=0x7f7fc40b4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.500+0000 7f7fc159a640 1 --2- 172.21.15.83:0/4067442584 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7fac0039b0 0x7f7fac003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.500+0000 7f7fc159a640 1 --2- 172.21.15.83:0/4067442584 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7fac004800 0x7f7fac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.500+0000 7f7fc159a640 1 --2- 172.21.15.83:0/4067442584 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fac0051c0 0x7f7fac00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.500+0000 7f7fc159a640 1 -- 172.21.15.83:0/4067442584 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc40b35f0 con 0x7f7fac0051c0 2024-03-06T07:29:09.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.500+0000 7f7fc159a640 1 -- 172.21.15.83:0/4067442584 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac00c160 con 0x7f7fac0039b0 2024-03-06T07:29:09.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.500+0000 7f7fc159a640 1 -- 172.21.15.83:0/4067442584 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac00c360 con 0x7f7fac004800 2024-03-06T07:29:09.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.500+0000 7f7fc2d9d640 1 --2- 172.21.15.83:0/4067442584 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fac0051c0 0x7f7fac00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.500+0000 7f7fc259c640 1 --2- 172.21.15.83:0/4067442584 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7fac0039b0 0x7f7fac003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.500+0000 7f7fc1d9b640 1 --2- 172.21.15.83:0/4067442584 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7fac004800 0x7f7fac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.500+0000 7f7fc2d9d640 1 -- 172.21.15.83:0/4067442584 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7fac004800 msgr2=0x7f7fac004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:09.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.500+0000 7f7fc2d9d640 1 --2- 172.21.15.83:0/4067442584 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7fac004800 0x7f7fac004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.500+0000 7f7fc2d9d640 1 -- 172.21.15.83:0/4067442584 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7fac0039b0 msgr2=0x7f7fac003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:09.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.500+0000 7f7fc2d9d640 1 --2- 172.21.15.83:0/4067442584 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7fac0039b0 0x7f7fac003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.500+0000 7f7fc2d9d640 1 -- 172.21.15.83:0/4067442584 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fac003520 con 0x7f7fac0051c0 2024-03-06T07:29:09.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.501+0000 7f7fc2d9d640 1 -- 172.21.15.83:0/4067442584 reap_dead start 2024-03-06T07:29:09.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.502+0000 7f7fc2d9d640 1 --2- 172.21.15.83:0/4067442584 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fac0051c0 0x7f7fac00b9c0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc00a860 tx=0x7f7fac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9488cb5b6f9cfeda server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:09.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.502+0000 7f7fc159a640 1 -- 172.21.15.83:0/4067442584 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fbc00a480 con 0x7f7fac0051c0 2024-03-06T07:29:09.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.502+0000 7f7fc159a640 1 -- 172.21.15.83:0/4067442584 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fbc034630 con 0x7f7fac0051c0 2024-03-06T07:29:09.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.502+0000 7f7fc159a640 1 -- 172.21.15.83:0/4067442584 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fbc034a20 con 0x7f7fac0051c0 2024-03-06T07:29:09.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.502+0000 7f7fc8864640 1 -- 172.21.15.83:0/4067442584 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fac0051c0 msgr2=0x7f7fac00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.502+0000 7f7fc8864640 1 --2- 172.21.15.83:0/4067442584 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fac0051c0 0x7f7fac00b9c0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc00a860 tx=0x7f7fac005830 comp rx=0 tx=0).stop 2024-03-06T07:29:09.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.502+0000 7f7fc8864640 1 -- 172.21.15.83:0/4067442584 shutdown_connections 2024-03-06T07:29:09.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.502+0000 7f7fc8864640 1 --2- 172.21.15.83:0/4067442584 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fac0051c0 0x7f7fac00b9c0 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.502+0000 7f7fc8864640 1 -- 172.21.15.83:0/4067442584 >> 172.21.15.83:0/4067442584 conn(0x7f7fc40d3c90 msgr2=0x7f7fc40d40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:09.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.502+0000 7f7fc8864640 1 -- 172.21.15.83:0/4067442584 shutdown_connections 2024-03-06T07:29:09.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.502+0000 7f7fc8864640 1 -- 172.21.15.83:0/4067442584 wait complete. 2024-03-06T07:29:09.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.504+0000 7f7fc8864640 1 Processor -- start 2024-03-06T07:29:09.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.504+0000 7f7fc8864640 1 -- start start 2024-03-06T07:29:09.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.504+0000 7f7fc8864640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fac0051c0 0x7f7fc4155d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.504+0000 7f7fc8864640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7fc41f4980 0x7f7fc41603a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.504+0000 7f7fc8864640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7fc41608e0 0x7f7fc415bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.504+0000 7f7fc8864640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc40b0860 con 0x7f7fac0051c0 2024-03-06T07:29:09.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.504+0000 7f7fc8864640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc40b4170 con 0x7f7fc41f4980 2024-03-06T07:29:09.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.504+0000 7f7fc8864640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc40b09e0 con 0x7f7fc41608e0 2024-03-06T07:29:09.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.505+0000 7f7fc259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fac0051c0 0x7f7fc4155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.505+0000 7f7fc259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fac0051c0 0x7f7fc4155d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60178/0 (socket says 172.21.15.83:60178) 2024-03-06T07:29:09.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.505+0000 7f7fc259c640 1 -- 172.21.15.83:0/2514053131 learned_addr learned my addr 172.21.15.83:0/2514053131 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:09.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.505+0000 7f7fc1d9b640 1 --2- 172.21.15.83:0/2514053131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7fc41f4980 0x7f7fc41603a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.505+0000 7f7fc2d9d640 1 --2- 172.21.15.83:0/2514053131 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7fc41608e0 0x7f7fc415bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.505+0000 7f7fc259c640 1 -- 172.21.15.83:0/2514053131 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7fc41608e0 msgr2=0x7f7fc415bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.505+0000 7f7fc259c640 1 --2- 172.21.15.83:0/2514053131 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7fc41608e0 0x7f7fc415bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.505+0000 7f7fc259c640 1 -- 172.21.15.83:0/2514053131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7fc41f4980 msgr2=0x7f7fc41603a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.505+0000 7f7fc259c640 1 --2- 172.21.15.83:0/2514053131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7fc41f4980 0x7f7fc41603a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.505+0000 7f7fc259c640 1 -- 172.21.15.83:0/2514053131 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc40ccb00 con 0x7f7fac0051c0 2024-03-06T07:29:09.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.505+0000 7f7fc2d9d640 1 --2- 172.21.15.83:0/2514053131 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7fc41608e0 0x7f7fc415bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:09.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.505+0000 7f7fc1d9b640 1 --2- 172.21.15.83:0/2514053131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7fc41f4980 0x7f7fc41603a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:09.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.505+0000 7f7fc259c640 1 --2- 172.21.15.83:0/2514053131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fac0051c0 0x7f7fc4155d00 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f7fb400f620 tx=0x7f7fb400fae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:09.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.505+0000 7f7fb2ffd640 1 -- 172.21.15.83:0/2514053131 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb40052e0 con 0x7f7fac0051c0 2024-03-06T07:29:09.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.505+0000 7f7fb2ffd640 1 -- 172.21.15.83:0/2514053131 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fb4018070 con 0x7f7fac0051c0 2024-03-06T07:29:09.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.505+0000 7f7fc8864640 1 -- 172.21.15.83:0/2514053131 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fc40ccde0 con 0x7f7fac0051c0 2024-03-06T07:29:09.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.505+0000 7f7fb2ffd640 1 -- 172.21.15.83:0/2514053131 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb4003810 con 0x7f7fac0051c0 2024-03-06T07:29:09.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.505+0000 7f7fc8864640 1 -- 172.21.15.83:0/2514053131 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fc40cd3a0 con 0x7f7fac0051c0 2024-03-06T07:29:09.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.506+0000 7f7fc8864640 1 -- 172.21.15.83:0/2514053131 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fc408cce0 con 0x7f7fac0051c0 2024-03-06T07:29:09.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.507+0000 7f7fb2ffd640 1 -- 172.21.15.83:0/2514053131 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7fb40039b0 con 0x7f7fac0051c0 2024-03-06T07:29:09.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.507+0000 7f7fb2ffd640 1 --2- 172.21.15.83:0/2514053131 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7fa406b500 0x7f7fa406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.507+0000 7f7fb2ffd640 1 -- 172.21.15.83:0/2514053131 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7fb4089e60 con 0x7f7fac0051c0 2024-03-06T07:29:09.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.507+0000 7f7fc1d9b640 1 --2- 172.21.15.83:0/2514053131 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7fa406b500 0x7f7fa406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.508+0000 7f7fc1d9b640 1 --2- 172.21.15.83:0/2514053131 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7fa406b500 0x7f7fa406d9c0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f7fb80093d0 tx=0x7f7fb8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:09.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.511+0000 7f7fb2ffd640 1 -- 172.21.15.83:0/2514053131 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7fc408cce0 con 0x7f7fac0051c0 2024-03-06T07:29:09.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.840+0000 7f7fc8864640 1 -- 172.21.15.83:0/2514053131 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7fc404f330 con 0x7f7fac0051c0 2024-03-06T07:29:09.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.841+0000 7f7fb2ffd640 1 -- 172.21.15.83:0/2514053131 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+5004 (secure 0 0 0) 0x7f7fb4058d70 con 0x7f7fac0051c0 2024-03-06T07:29:09.843 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:29:09.843 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":76,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7381,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/3776086145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3776086145},{"type":"v1","addr":"172.21.15.183:6833","nonce":3776086145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":7387,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2457445285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2457445285},{"type":"v1","addr":"172.21.15.183:6835","nonce":2457445285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:05.001149+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8628,8631]},"id":9}]} 2024-03-06T07:29:09.843 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 76 2024-03-06T07:29:09.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.843+0000 7f7fc8864640 1 -- 172.21.15.83:0/2514053131 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7fa406b500 msgr2=0x7f7fa406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.843+0000 7f7fc8864640 1 --2- 172.21.15.83:0/2514053131 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7fa406b500 0x7f7fa406d9c0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f7fb80093d0 tx=0x7f7fb8008040 comp rx=0 tx=0).stop 2024-03-06T07:29:09.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.843+0000 7f7fc8864640 1 -- 172.21.15.83:0/2514053131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fac0051c0 msgr2=0x7f7fc4155d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.843+0000 7f7fc8864640 1 --2- 172.21.15.83:0/2514053131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fac0051c0 0x7f7fc4155d00 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f7fb400f620 tx=0x7f7fb400fae0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.843+0000 7f7fc8864640 1 -- 172.21.15.83:0/2514053131 shutdown_connections 2024-03-06T07:29:09.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.843+0000 7f7fc8864640 1 --2- 172.21.15.83:0/2514053131 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7fa406b500 0x7f7fa406d9c0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.843+0000 7f7fc8864640 1 --2- 172.21.15.83:0/2514053131 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7fc41608e0 0x7f7fc415bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.843+0000 7f7fc8864640 1 --2- 172.21.15.83:0/2514053131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7fc41f4980 0x7f7fc41603a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.843+0000 7f7fc8864640 1 --2- 172.21.15.83:0/2514053131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fac0051c0 0x7f7fc4155d00 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.843+0000 7f7fc8864640 1 -- 172.21.15.83:0/2514053131 >> 172.21.15.83:0/2514053131 conn(0x7f7fc40d3c90 msgr2=0x7f7fc40cea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:09.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.843+0000 7f7fc8864640 1 -- 172.21.15.83:0/2514053131 shutdown_connections 2024-03-06T07:29:09.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.843+0000 7f7fc8864640 1 -- 172.21.15.83:0/2514053131 wait complete. 2024-03-06T07:29:09.856 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:09.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.956+0000 7f465ae1c640 1 Processor -- start 2024-03-06T07:29:09.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.956+0000 7f465ae1c640 1 -- start start 2024-03-06T07:29:09.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.957+0000 7f465ae1c640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f46540d22b0 0x7f46540d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.957+0000 7f465ae1c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f46540b41f0 0x7f46540b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.957+0000 7f465ae1c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46540b3430 con 0x7f465415dfe0 2024-03-06T07:29:09.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.957+0000 7f465ae1c640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46540d2c50 con 0x7f46540b41f0 2024-03-06T07:29:09.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.957+0000 7f465ae1c640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46540b31e0 con 0x7f46540d22b0 2024-03-06T07:29:09.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.957+0000 7f46593b9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f46540b41f0 0x7f46540b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.957+0000 7f46593b9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f46540b41f0 0x7f46540b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60182/0 (socket says 172.21.15.83:60182) 2024-03-06T07:29:09.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.957+0000 7f46593b9640 1 -- 172.21.15.83:0/1259228876 learned_addr learned my addr 172.21.15.83:0/1259228876 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:09.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.957+0000 7f4653fff640 1 --2- 172.21.15.83:0/1259228876 >> v2:172.21.15.183:3300/0 conn(0x7f46540d22b0 0x7f46540d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.957+0000 7f46593b9640 1 -- 172.21.15.83:0/1259228876 >> v2:172.21.15.183:3300/0 conn(0x7f46540d22b0 msgr2=0x7f46540d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:09.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.957+0000 7f46593b9640 1 --2- 172.21.15.83:0/1259228876 >> v2:172.21.15.183:3300/0 conn(0x7f46540d22b0 0x7f46540d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.957+0000 7f46593b9640 1 -- 172.21.15.83:0/1259228876 >> v1:172.21.15.183:6789/0 conn(0x7f465415dfe0 legacy=0x7f46540cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.957+0000 7f46593b9640 1 -- 172.21.15.83:0/1259228876 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46540b3710 con 0x7f46540b41f0 2024-03-06T07:29:09.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.958+0000 7f46593b9640 1 --2- 172.21.15.83:0/1259228876 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46540b41f0 0x7f46540b4680 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f464c009e50 tx=0x7f464c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae4ed8e23bc91750 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:09.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.958+0000 7f46537fe640 1 -- 172.21.15.83:0/1259228876 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f464c02cb20 con 0x7f46540b41f0 2024-03-06T07:29:09.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.958+0000 7f46537fe640 1 -- 172.21.15.83:0/1259228876 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f464c02ccc0 con 0x7f46540b41f0 2024-03-06T07:29:09.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.958+0000 7f46537fe640 1 -- 172.21.15.83:0/1259228876 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f464c034610 con 0x7f46540b41f0 2024-03-06T07:29:09.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.959+0000 7f465ae1c640 1 -- 172.21.15.83:0/1259228876 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46540b41f0 msgr2=0x7f46540b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.959+0000 7f465ae1c640 1 --2- 172.21.15.83:0/1259228876 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46540b41f0 0x7f46540b4680 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f464c009e50 tx=0x7f464c02bf80 comp rx=0 tx=0).stop 2024-03-06T07:29:09.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.959+0000 7f465ae1c640 1 -- 172.21.15.83:0/1259228876 shutdown_connections 2024-03-06T07:29:09.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.959+0000 7f465ae1c640 1 --2- 172.21.15.83:0/1259228876 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46540b41f0 0x7f46540b4680 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.959+0000 7f465ae1c640 1 --2- 172.21.15.83:0/1259228876 >> v2:172.21.15.183:3300/0 conn(0x7f46540d22b0 0x7f46540d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.959+0000 7f465ae1c640 1 -- 172.21.15.83:0/1259228876 >> 172.21.15.83:0/1259228876 conn(0x7f46540d3cd0 msgr2=0x7f46540d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:09.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.959+0000 7f465ae1c640 1 -- 172.21.15.83:0/1259228876 shutdown_connections 2024-03-06T07:29:09.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.959+0000 7f465ae1c640 1 -- 172.21.15.83:0/1259228876 wait complete. 2024-03-06T07:29:09.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.961+0000 7f465ae1c640 1 Processor -- start 2024-03-06T07:29:09.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.961+0000 7f465ae1c640 1 -- start start 2024-03-06T07:29:09.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.962+0000 7f465ae1c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46540b41f0 0x7f46540b6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.962+0000 7f465ae1c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46540d22b0 0x7f46541f3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.962+0000 7f465ae1c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f465415dfe0 0x7f46540cc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.962+0000 7f465ae1c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46540cfa00 con 0x7f46540d22b0 2024-03-06T07:29:09.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.962+0000 7f465ae1c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46540b3d60 con 0x7f465415dfe0 2024-03-06T07:29:09.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.962+0000 7f465ae1c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46540d1c70 con 0x7f46540b41f0 2024-03-06T07:29:09.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.962+0000 7f4653fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46540d22b0 0x7f46541f3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.962+0000 7f4653fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46540d22b0 0x7f46541f3af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60190/0 (socket says 172.21.15.83:60190) 2024-03-06T07:29:09.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.962+0000 7f4653fff640 1 -- 172.21.15.83:0/2267869230 learned_addr learned my addr 172.21.15.83:0/2267869230 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:09.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.962+0000 7f4658bb8640 1 --2- 172.21.15.83:0/2267869230 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46540b41f0 0x7f46540b6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.963+0000 7f46593b9640 1 --2- 172.21.15.83:0/2267869230 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f465415dfe0 0x7f46540cc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.963+0000 7f4653fff640 1 -- 172.21.15.83:0/2267869230 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46540b41f0 msgr2=0x7f46540b6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.963+0000 7f4653fff640 1 --2- 172.21.15.83:0/2267869230 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46540b41f0 0x7f46540b6f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.963+0000 7f4653fff640 1 -- 172.21.15.83:0/2267869230 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f465415dfe0 msgr2=0x7f46540cc9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.963+0000 7f4653fff640 1 --2- 172.21.15.83:0/2267869230 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f465415dfe0 0x7f46540cc9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.963+0000 7f4653fff640 1 -- 172.21.15.83:0/2267869230 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46541f4370 con 0x7f46540d22b0 2024-03-06T07:29:09.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.963+0000 7f4653fff640 1 --2- 172.21.15.83:0/2267869230 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46540d22b0 0x7f46541f3af0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f464000d800 tx=0x7f464000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:09.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.963+0000 7f46517fa640 1 -- 172.21.15.83:0/2267869230 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f464000b6c0 con 0x7f46540d22b0 2024-03-06T07:29:09.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.963+0000 7f46517fa640 1 -- 172.21.15.83:0/2267869230 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4640019070 con 0x7f46540d22b0 2024-03-06T07:29:09.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.963+0000 7f465ae1c640 1 -- 172.21.15.83:0/2267869230 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46541f4650 con 0x7f46540d22b0 2024-03-06T07:29:09.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.963+0000 7f46517fa640 1 -- 172.21.15.83:0/2267869230 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f464000bac0 con 0x7f46540d22b0 2024-03-06T07:29:09.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.963+0000 7f465ae1c640 1 -- 172.21.15.83:0/2267869230 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46541f4030 con 0x7f46540d22b0 2024-03-06T07:29:09.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.964+0000 7f465ae1c640 1 -- 172.21.15.83:0/2267869230 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f465415e820 con 0x7f46540d22b0 2024-03-06T07:29:09.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.965+0000 7f46517fa640 1 -- 172.21.15.83:0/2267869230 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f464000bc60 con 0x7f46540d22b0 2024-03-06T07:29:09.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.965+0000 7f46517fa640 1 --2- 172.21.15.83:0/2267869230 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f463806b4d0 0x7f463806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.965+0000 7f46517fa640 1 -- 172.21.15.83:0/2267869230 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f464008a4e0 con 0x7f46540d22b0 2024-03-06T07:29:09.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.965+0000 7f4658bb8640 1 --2- 172.21.15.83:0/2267869230 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f463806b4d0 0x7f463806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.966+0000 7f4658bb8640 1 --2- 172.21.15.83:0/2267869230 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f463806b4d0 0x7f463806d990 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f46480016e0 tx=0x7f4648009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:09.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:09.970+0000 7f46517fa640 1 -- 172.21.15.83:0/2267869230 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4640058da0 con 0x7f46540d22b0 2024-03-06T07:29:10.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.188+0000 7f465ae1c640 1 -- 172.21.15.83:0/2267869230 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f465415f610 con 0x7f46540d22b0 2024-03-06T07:29:10.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.188+0000 7f46517fa640 1 -- 172.21.15.83:0/2267869230 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+5004 (secure 0 0 0) 0x7f464005ce40 con 0x7f46540d22b0 2024-03-06T07:29:10.190 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:29:10.190 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":76,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7381,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/3776086145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3776086145},{"type":"v1","addr":"172.21.15.183:6833","nonce":3776086145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":7387,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2457445285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2457445285},{"type":"v1","addr":"172.21.15.183:6835","nonce":2457445285}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:05.001149+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8628,8631]},"id":9}]} 2024-03-06T07:29:10.190 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 76 2024-03-06T07:29:10.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.191+0000 7f465ae1c640 1 -- 172.21.15.83:0/2267869230 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f463806b4d0 msgr2=0x7f463806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.191+0000 7f465ae1c640 1 --2- 172.21.15.83:0/2267869230 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f463806b4d0 0x7f463806d990 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f46480016e0 tx=0x7f4648009040 comp rx=0 tx=0).stop 2024-03-06T07:29:10.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.191+0000 7f465ae1c640 1 -- 172.21.15.83:0/2267869230 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46540d22b0 msgr2=0x7f46541f3af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.191+0000 7f465ae1c640 1 --2- 172.21.15.83:0/2267869230 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46540d22b0 0x7f46541f3af0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f464000d800 tx=0x7f464000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.191+0000 7f465ae1c640 1 -- 172.21.15.83:0/2267869230 shutdown_connections 2024-03-06T07:29:10.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.191+0000 7f465ae1c640 1 --2- 172.21.15.83:0/2267869230 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f463806b4d0 0x7f463806d990 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.191+0000 7f465ae1c640 1 --2- 172.21.15.83:0/2267869230 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f465415dfe0 0x7f46540cc9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.191+0000 7f465ae1c640 1 --2- 172.21.15.83:0/2267869230 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46540d22b0 0x7f46541f3af0 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.191+0000 7f465ae1c640 1 --2- 172.21.15.83:0/2267869230 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46540b41f0 0x7f46540b6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.191+0000 7f465ae1c640 1 -- 172.21.15.83:0/2267869230 >> 172.21.15.83:0/2267869230 conn(0x7f46540d3cd0 msgr2=0x7f46540b45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:10.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.191+0000 7f465ae1c640 1 -- 172.21.15.83:0/2267869230 shutdown_connections 2024-03-06T07:29:10.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.191+0000 7f465ae1c640 1 -- 172.21.15.83:0/2267869230 wait complete. 2024-03-06T07:29:10.207 INFO:tasks.cephfs.test_strays:Ranks 0 and 1 are a and c 2024-03-06T07:29:10.207 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:29:10.207 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:29:10.218 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:29:10.219 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-03-06T07:29:10.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.319+0000 7fe2ccdb6640 1 Processor -- start 2024-03-06T07:29:10.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.319+0000 7fe2ccdb6640 1 -- start start 2024-03-06T07:29:10.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.320+0000 7fe2ccdb6640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fe2c8162970 0x7fe2c80d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.320+0000 7fe2ccdb6640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe2c8161cf0 0x7fe2c8162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.320+0000 7fe2ccdb6640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2c80d22c0 con 0x7fe2c80b4460 2024-03-06T07:29:10.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.320+0000 7fe2ccdb6640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2c8161b00 con 0x7fe2c8162970 2024-03-06T07:29:10.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.320+0000 7fe2ccdb6640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c80cecc0 con 0x7fe2c8161cf0 2024-03-06T07:29:10.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.320+0000 7fe2c6d9d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe2c8161cf0 0x7fe2c8162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.320+0000 7fe2c5d9b640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fe2c8162970 0x7fe2c80d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.320+0000 7fe2c5d9b640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fe2c8162970 0x7fe2c80d4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:39152/0 (socket says 172.21.15.83:39152) 2024-03-06T07:29:10.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.320+0000 7fe2c6d9d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe2c8161cf0 0x7fe2c8162180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:46694/0 (socket says 172.21.15.83:46694) 2024-03-06T07:29:10.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.320+0000 7fe2c5d9b640 1 -- 172.21.15.83:0/1039032364 learned_addr learned my addr 172.21.15.83:0/1039032364 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:10.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.320+0000 7fe2c659c640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fe2c80b4460 0x7fe2c80b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:60070/0 (socket says 172.21.15.83:60070) 2024-03-06T07:29:10.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.321+0000 7fe2c6d9d640 1 -- 172.21.15.83:0/1039032364 >> v2:172.21.15.183:3300/0 conn(0x7fe2c8162970 msgr2=0x7fe2c80d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:10.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.321+0000 7fe2c6d9d640 1 --2- 172.21.15.83:0/1039032364 >> v2:172.21.15.183:3300/0 conn(0x7fe2c8162970 0x7fe2c80d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.321+0000 7fe2c6d9d640 1 -- 172.21.15.83:0/1039032364 >> v1:172.21.15.183:6789/0 conn(0x7fe2c80b4460 legacy=0x7fe2c80b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.321+0000 7fe2c6d9d640 1 -- 172.21.15.83:0/1039032364 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c80d25a0 con 0x7fe2c8161cf0 2024-03-06T07:29:10.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.321+0000 7fe2c5d9b640 1 --2- 172.21.15.83:0/1039032364 >> v2:172.21.15.183:3300/0 conn(0x7fe2c8162970 0x7fe2c80d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:10.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.322+0000 7fe2c6d9d640 1 --2- 172.21.15.83:0/1039032364 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe2c8161cf0 0x7fe2c8162180 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe2b80097a0 tx=0x7fe2b802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ec61cb0a5f3f3eb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.322+0000 7fe2c559a640 1 -- 172.21.15.83:0/1039032364 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b8035020 con 0x7fe2c8161cf0 2024-03-06T07:29:10.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.322+0000 7fe2c559a640 1 -- 172.21.15.83:0/1039032364 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b8031830 con 0x7fe2c8161cf0 2024-03-06T07:29:10.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.322+0000 7fe2c559a640 1 -- 172.21.15.83:0/1039032364 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b8031c00 con 0x7fe2c8161cf0 2024-03-06T07:29:10.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.323+0000 7fe2ccdb6640 1 -- 172.21.15.83:0/1039032364 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe2c8161cf0 msgr2=0x7fe2c8162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.323+0000 7fe2ccdb6640 1 --2- 172.21.15.83:0/1039032364 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe2c8161cf0 0x7fe2c8162180 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe2b80097a0 tx=0x7fe2b802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.323+0000 7fe2ccdb6640 1 -- 172.21.15.83:0/1039032364 shutdown_connections 2024-03-06T07:29:10.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.323+0000 7fe2ccdb6640 1 --2- 172.21.15.83:0/1039032364 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe2c8161cf0 0x7fe2c8162180 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.323+0000 7fe2ccdb6640 1 --2- 172.21.15.83:0/1039032364 >> v2:172.21.15.183:3300/0 conn(0x7fe2c8162970 0x7fe2c80d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.323+0000 7fe2ccdb6640 1 -- 172.21.15.83:0/1039032364 >> 172.21.15.83:0/1039032364 conn(0x7fe2c80d3ce0 msgr2=0x7fe2c80d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:10.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.323+0000 7fe2ccdb6640 1 -- 172.21.15.83:0/1039032364 shutdown_connections 2024-03-06T07:29:10.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.323+0000 7fe2ccdb6640 1 -- 172.21.15.83:0/1039032364 wait complete. 2024-03-06T07:29:10.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.324+0000 7fe2ccdb6640 1 Processor -- start 2024-03-06T07:29:10.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.324+0000 7fe2ccdb6640 1 -- start start 2024-03-06T07:29:10.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.325+0000 7fe2ccdb6640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe2c80b4460 0x7fe2c80cca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.325+0000 7fe2ccdb6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe2c8161cf0 0x7fe2c80cd4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.325+0000 7fe2ccdb6640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe2c8162970 0x7fe2c80b1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.325+0000 7fe2ccdb6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c80ce950 con 0x7fe2c8161cf0 2024-03-06T07:29:10.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.325+0000 7fe2ccdb6640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c80d2b60 con 0x7fe2c80b4460 2024-03-06T07:29:10.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.325+0000 7fe2ccdb6640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c80b4840 con 0x7fe2c8162970 2024-03-06T07:29:10.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.325+0000 7fe2c5d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe2c8161cf0 0x7fe2c80cd4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.325+0000 7fe2c5d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe2c8161cf0 0x7fe2c80cd4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60206/0 (socket says 172.21.15.83:60206) 2024-03-06T07:29:10.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.325+0000 7fe2c5d9b640 1 -- 172.21.15.83:0/3040153360 learned_addr learned my addr 172.21.15.83:0/3040153360 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:10.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.325+0000 7fe2c6d9d640 1 --2- 172.21.15.83:0/3040153360 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe2c8162970 0x7fe2c80b1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.325+0000 7fe2c659c640 1 --2- 172.21.15.83:0/3040153360 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe2c80b4460 0x7fe2c80cca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.325+0000 7fe2c5d9b640 1 -- 172.21.15.83:0/3040153360 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe2c8162970 msgr2=0x7fe2c80b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.325+0000 7fe2c5d9b640 1 --2- 172.21.15.83:0/3040153360 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe2c8162970 0x7fe2c80b1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.325+0000 7fe2c5d9b640 1 -- 172.21.15.83:0/3040153360 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe2c80b4460 msgr2=0x7fe2c80cca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.325+0000 7fe2c5d9b640 1 --2- 172.21.15.83:0/3040153360 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe2c80b4460 0x7fe2c80cca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.325+0000 7fe2c5d9b640 1 -- 172.21.15.83:0/3040153360 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c80cb300 con 0x7fe2c8161cf0 2024-03-06T07:29:10.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.325+0000 7fe2c5d9b640 1 --2- 172.21.15.83:0/3040153360 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe2c8161cf0 0x7fe2c80cd4c0 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc009f00 tx=0x7fe2bc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.326+0000 7fe2b6ffd640 1 -- 172.21.15.83:0/3040153360 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2bc014480 con 0x7fe2c8161cf0 2024-03-06T07:29:10.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.326+0000 7fe2b6ffd640 1 -- 172.21.15.83:0/3040153360 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2bc019070 con 0x7fe2c8161cf0 2024-03-06T07:29:10.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.326+0000 7fe2ccdb6640 1 -- 172.21.15.83:0/3040153360 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2c80cb5e0 con 0x7fe2c8161cf0 2024-03-06T07:29:10.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.326+0000 7fe2b6ffd640 1 -- 172.21.15.83:0/3040153360 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2bc014860 con 0x7fe2c8161cf0 2024-03-06T07:29:10.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.326+0000 7fe2ccdb6640 1 -- 172.21.15.83:0/3040153360 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2c80cbba0 con 0x7fe2c8161cf0 2024-03-06T07:29:10.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.327+0000 7fe2ccdb6640 1 -- 172.21.15.83:0/3040153360 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2c804f330 con 0x7fe2c8161cf0 2024-03-06T07:29:10.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.327+0000 7fe2b6ffd640 1 -- 172.21.15.83:0/3040153360 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe2bc005d10 con 0x7fe2c8161cf0 2024-03-06T07:29:10.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.328+0000 7fe2b6ffd640 1 --2- 172.21.15.83:0/3040153360 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe2a806b4d0 0x7fe2a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.328+0000 7fe2b6ffd640 1 -- 172.21.15.83:0/3040153360 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe2bc08a7b0 con 0x7fe2c8161cf0 2024-03-06T07:29:10.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.328+0000 7fe2c659c640 1 --2- 172.21.15.83:0/3040153360 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe2a806b4d0 0x7fe2a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.328+0000 7fe2c659c640 1 --2- 172.21.15.83:0/3040153360 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe2a806b4d0 0x7fe2a806d990 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fe2c0007fd0 tx=0x7fe2c0007e80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.332+0000 7fe2b6ffd640 1 -- 172.21.15.83:0/3040153360 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe2bc023020 con 0x7fe2c8161cf0 2024-03-06T07:29:10.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:10.684+0000 7fe2ccdb6640 1 -- 172.21.15.83:0/3040153360 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7fe2c806b180 con 0x7fe2c8161cf0 2024-03-06T07:29:11.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.023+0000 7fe2b6ffd640 1 -- 172.21.15.83:0/3040153360 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 7387 v77) v1 ==== 95+0+0 (secure 0 0 0) 0x7fe2bc010040 con 0x7fe2c8161cf0 2024-03-06T07:29:11.025 INFO:teuthology.orchestra.run.smithi083.stderr:failed mds gid 7387 2024-03-06T07:29:11.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.025+0000 7fe2ccdb6640 1 -- 172.21.15.83:0/3040153360 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe2a806b4d0 msgr2=0x7fe2a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.025+0000 7fe2ccdb6640 1 --2- 172.21.15.83:0/3040153360 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe2a806b4d0 0x7fe2a806d990 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fe2c0007fd0 tx=0x7fe2c0007e80 comp rx=0 tx=0).stop 2024-03-06T07:29:11.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.025+0000 7fe2ccdb6640 1 -- 172.21.15.83:0/3040153360 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe2c8161cf0 msgr2=0x7fe2c80cd4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.025+0000 7fe2ccdb6640 1 --2- 172.21.15.83:0/3040153360 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe2c8161cf0 0x7fe2c80cd4c0 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc009f00 tx=0x7fe2bc00c490 comp rx=0 tx=0).stop 2024-03-06T07:29:11.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.025+0000 7fe2ccdb6640 1 -- 172.21.15.83:0/3040153360 shutdown_connections 2024-03-06T07:29:11.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.025+0000 7fe2ccdb6640 1 --2- 172.21.15.83:0/3040153360 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe2a806b4d0 0x7fe2a806d990 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.025+0000 7fe2ccdb6640 1 --2- 172.21.15.83:0/3040153360 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe2c8162970 0x7fe2c80b1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.025+0000 7fe2ccdb6640 1 --2- 172.21.15.83:0/3040153360 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe2c8161cf0 0x7fe2c80cd4c0 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.025+0000 7fe2ccdb6640 1 --2- 172.21.15.83:0/3040153360 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe2c80b4460 0x7fe2c80cca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.025+0000 7fe2ccdb6640 1 -- 172.21.15.83:0/3040153360 >> 172.21.15.83:0/3040153360 conn(0x7fe2c80d3ce0 msgr2=0x7fe2c8160560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:11.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.025+0000 7fe2ccdb6640 1 -- 172.21.15.83:0/3040153360 shutdown_connections 2024-03-06T07:29:11.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.025+0000 7fe2ccdb6640 1 -- 172.21.15.83:0/3040153360 wait complete. 2024-03-06T07:29:11.038 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:29:11.038 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:29:11.049 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:29:11.049 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-03-06T07:29:11.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.153+0000 7f1e2d84e640 1 Processor -- start 2024-03-06T07:29:11.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.154+0000 7f1e2d84e640 1 -- start start 2024-03-06T07:29:11.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.155+0000 7f1e2d84e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1e281617d0 0x7f1e280cd130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.155+0000 7f1e2d84e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f1e280b5320 0x7f1e2806d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.155+0000 7f1e2d84e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e280b5ef0 con 0x7f1e281613f0 2024-03-06T07:29:11.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.155+0000 7f1e2d84e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e2806c0a0 con 0x7f1e280b5320 2024-03-06T07:29:11.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.155+0000 7f1e2d84e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e280d0b50 con 0x7f1e281617d0 2024-03-06T07:29:11.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.155+0000 7f1e26ffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f1e281613f0 0x7f1e280cca20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:51138/0 (socket says 172.21.15.83:51138) 2024-03-06T07:29:11.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.155+0000 7f1e26ffd640 1 -- 172.21.15.83:0/2687988235 learned_addr learned my addr 172.21.15.83:0/2687988235 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:29:11.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.155+0000 7f1e277fe640 1 --2- 172.21.15.83:0/2687988235 >> v2:172.21.15.183:3300/0 conn(0x7f1e280b5320 0x7f1e2806d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.155+0000 7f1e267fc640 1 --2- 172.21.15.83:0/2687988235 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1e281617d0 0x7f1e280cd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.155+0000 7f1e25ffb640 1 -- 172.21.15.83:0/2687988235 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3989417529 0 0) 0x7f1e280b5ef0 con 0x7f1e281613f0 2024-03-06T07:29:11.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.155+0000 7f1e25ffb640 1 -- 172.21.15.83:0/2687988235 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e10003520 con 0x7f1e281613f0 2024-03-06T07:29:11.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.155+0000 7f1e25ffb640 1 -- 172.21.15.83:0/2687988235 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2332850131 0 0) 0x7f1e10003520 con 0x7f1e281613f0 2024-03-06T07:29:11.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.155+0000 7f1e25ffb640 1 -- 172.21.15.83:0/2687988235 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1e281617d0 msgr2=0x7f1e280cd130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:11.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.155+0000 7f1e25ffb640 1 --2- 172.21.15.83:0/2687988235 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1e281617d0 0x7f1e280cd130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.155+0000 7f1e25ffb640 1 -- 172.21.15.83:0/2687988235 >> v2:172.21.15.183:3300/0 conn(0x7f1e280b5320 msgr2=0x7f1e2806d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:11.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.155+0000 7f1e25ffb640 1 --2- 172.21.15.83:0/2687988235 >> v2:172.21.15.183:3300/0 conn(0x7f1e280b5320 0x7f1e2806d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.155+0000 7f1e25ffb640 1 -- 172.21.15.83:0/2687988235 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e280b5bf0 con 0x7f1e281613f0 2024-03-06T07:29:11.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.155+0000 7f1e25ffb640 1 -- 172.21.15.83:0/2687988235 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f1e1c001670 con 0x7f1e281613f0 2024-03-06T07:29:11.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.155+0000 7f1e25ffb640 1 -- 172.21.15.83:0/2687988235 >> v1:172.21.15.83:6789/0 conn(0x7f1e281613f0 legacy=0x7f1e280cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.155+0000 7f1e25ffb640 1 --2- 172.21.15.83:0/2687988235 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e100039b0 0x7f1e10003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.156+0000 7f1e25ffb640 1 --2- 172.21.15.83:0/2687988235 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e10004800 0x7f1e10004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.156+0000 7f1e25ffb640 1 --2- 172.21.15.83:0/2687988235 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1e281617d0 0x7f1e100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.156+0000 7f1e25ffb640 1 -- 172.21.15.83:0/2687988235 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e280d0b50 con 0x7f1e100039b0 2024-03-06T07:29:11.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.156+0000 7f1e25ffb640 1 -- 172.21.15.83:0/2687988235 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1000b9c0 con 0x7f1e10004800 2024-03-06T07:29:11.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.156+0000 7f1e25ffb640 1 -- 172.21.15.83:0/2687988235 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1000bb40 con 0x7f1e281617d0 2024-03-06T07:29:11.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.156+0000 7f1e26ffd640 1 --2- 172.21.15.83:0/2687988235 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e100039b0 0x7f1e10003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.156+0000 7f1e267fc640 1 --2- 172.21.15.83:0/2687988235 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e10004800 0x7f1e10004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.156+0000 7f1e277fe640 1 --2- 172.21.15.83:0/2687988235 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1e281617d0 0x7f1e100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.156+0000 7f1e26ffd640 1 -- 172.21.15.83:0/2687988235 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1e281617d0 msgr2=0x7f1e100051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:11.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.156+0000 7f1e26ffd640 1 --2- 172.21.15.83:0/2687988235 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1e281617d0 0x7f1e100051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.156+0000 7f1e26ffd640 1 -- 172.21.15.83:0/2687988235 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e10004800 msgr2=0x7f1e10004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:11.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.156+0000 7f1e26ffd640 1 --2- 172.21.15.83:0/2687988235 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e10004800 0x7f1e10004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.156+0000 7f1e26ffd640 1 -- 172.21.15.83:0/2687988235 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e10003520 con 0x7f1e100039b0 2024-03-06T07:29:11.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.157+0000 7f1e26ffd640 1 --2- 172.21.15.83:0/2687988235 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e100039b0 0x7f1e10003dd0 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c001050 tx=0x7f1e10005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db51dbd8492cc78a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:11.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.157+0000 7f1e25ffb640 1 -- 172.21.15.83:0/2687988235 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e1c007d70 con 0x7f1e100039b0 2024-03-06T07:29:11.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.157+0000 7f1e25ffb640 1 -- 172.21.15.83:0/2687988235 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e1c030070 con 0x7f1e100039b0 2024-03-06T07:29:11.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.157+0000 7f1e25ffb640 1 -- 172.21.15.83:0/2687988235 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e1c02cd80 con 0x7f1e100039b0 2024-03-06T07:29:11.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.158+0000 7f1e2d84e640 1 -- 172.21.15.83:0/2687988235 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e100039b0 msgr2=0x7f1e10003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.158+0000 7f1e2d84e640 1 --2- 172.21.15.83:0/2687988235 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e100039b0 0x7f1e10003dd0 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c001050 tx=0x7f1e10005830 comp rx=0 tx=0).stop 2024-03-06T07:29:11.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.158+0000 7f1e277fe640 1 -- 172.21.15.83:0/2687988235 reap_dead start 2024-03-06T07:29:11.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.158+0000 7f1e2d84e640 1 -- 172.21.15.83:0/2687988235 shutdown_connections 2024-03-06T07:29:11.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.158+0000 7f1e2d84e640 1 -- 172.21.15.83:0/2687988235 >> 172.21.15.83:0/2687988235 conn(0x7f1e28072520 msgr2=0x7f1e28072930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:11.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.158+0000 7f1e2d84e640 1 -- 172.21.15.83:0/2687988235 shutdown_connections 2024-03-06T07:29:11.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.158+0000 7f1e2d84e640 1 -- 172.21.15.83:0/2687988235 wait complete. 2024-03-06T07:29:11.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.159+0000 7f1e2d84e640 1 Processor -- start 2024-03-06T07:29:11.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.159+0000 7f1e2d84e640 1 -- start start 2024-03-06T07:29:11.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.160+0000 7f1e2d84e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e100039b0 0x7f1e2815c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.160+0000 7f1e2d84e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e280c3cb0 0x7f1e280c4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.160+0000 7f1e2d84e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1e280c5c90 0x7f1e280c6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.160+0000 7f1e2d84e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e280cd2b0 con 0x7f1e280c3cb0 2024-03-06T07:29:11.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.160+0000 7f1e2d84e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e280cd130 con 0x7f1e100039b0 2024-03-06T07:29:11.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.160+0000 7f1e2d84e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e280cd430 con 0x7f1e280c5c90 2024-03-06T07:29:11.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.160+0000 7f1e267fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e280c3cb0 0x7f1e280c4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.160+0000 7f1e267fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e280c3cb0 0x7f1e280c4b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60226/0 (socket says 172.21.15.83:60226) 2024-03-06T07:29:11.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.160+0000 7f1e267fc640 1 -- 172.21.15.83:0/2355398129 learned_addr learned my addr 172.21.15.83:0/2355398129 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:11.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.160+0000 7f1e26ffd640 1 --2- 172.21.15.83:0/2355398129 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e100039b0 0x7f1e2815c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.160+0000 7f1e277fe640 1 --2- 172.21.15.83:0/2355398129 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1e280c5c90 0x7f1e280c6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.160+0000 7f1e267fc640 1 -- 172.21.15.83:0/2355398129 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1e280c5c90 msgr2=0x7f1e280c6110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.160+0000 7f1e267fc640 1 --2- 172.21.15.83:0/2355398129 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1e280c5c90 0x7f1e280c6110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.160+0000 7f1e267fc640 1 -- 172.21.15.83:0/2355398129 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e100039b0 msgr2=0x7f1e2815c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.160+0000 7f1e267fc640 1 --2- 172.21.15.83:0/2355398129 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e100039b0 0x7f1e2815c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.160+0000 7f1e267fc640 1 -- 172.21.15.83:0/2355398129 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e280c6650 con 0x7f1e280c3cb0 2024-03-06T07:29:11.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.160+0000 7f1e26ffd640 1 --2- 172.21.15.83:0/2355398129 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e100039b0 0x7f1e2815c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:11.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.160+0000 7f1e267fc640 1 --2- 172.21.15.83:0/2355398129 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e280c3cb0 0x7f1e280c4b40 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f1e18011940 tx=0x7f1e18011e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:11.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.161+0000 7f1e0ffff640 1 -- 172.21.15.83:0/2355398129 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e1800faf0 con 0x7f1e280c3cb0 2024-03-06T07:29:11.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.161+0000 7f1e0ffff640 1 -- 172.21.15.83:0/2355398129 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e1800fc90 con 0x7f1e280c3cb0 2024-03-06T07:29:11.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.161+0000 7f1e2d84e640 1 -- 172.21.15.83:0/2355398129 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e2815c850 con 0x7f1e280c3cb0 2024-03-06T07:29:11.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.161+0000 7f1e0ffff640 1 -- 172.21.15.83:0/2355398129 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e18014440 con 0x7f1e280c3cb0 2024-03-06T07:29:11.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.161+0000 7f1e2d84e640 1 -- 172.21.15.83:0/2355398129 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e280c5790 con 0x7f1e280c3cb0 2024-03-06T07:29:11.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.162+0000 7f1e2d84e640 1 -- 172.21.15.83:0/2355398129 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e280c5990 con 0x7f1e280c3cb0 2024-03-06T07:29:11.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.163+0000 7f1e0ffff640 1 -- 172.21.15.83:0/2355398129 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1e18012040 con 0x7f1e280c3cb0 2024-03-06T07:29:11.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.163+0000 7f1e0ffff640 1 --2- 172.21.15.83:0/2355398129 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1e1406b500 0x7f1e1406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.163+0000 7f1e0ffff640 1 -- 172.21.15.83:0/2355398129 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1e1808a860 con 0x7f1e280c3cb0 2024-03-06T07:29:11.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.163+0000 7f1e26ffd640 1 --2- 172.21.15.83:0/2355398129 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1e1406b500 0x7f1e1406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.164+0000 7f1e26ffd640 1 --2- 172.21.15.83:0/2355398129 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1e1406b500 0x7f1e1406d9c0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c034000 tx=0x7f1e1c007000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:11.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.168+0000 7f1e0ffff640 1 -- 172.21.15.83:0/2355398129 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1e280c6650 con 0x7f1e280c3cb0 2024-03-06T07:29:11.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:11.448+0000 7f1e2d84e640 1 -- 172.21.15.83:0/2355398129 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7f1e2815e9d0 con 0x7f1e280c3cb0 2024-03-06T07:29:12.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.033+0000 7f1e0ffff640 1 -- 172.21.15.83:0/2355398129 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 7381 v78) v1 ==== 95+0+0 (secure 0 0 0) 0x7f1e1808a030 con 0x7f1e280c3cb0 2024-03-06T07:29:12.035 INFO:teuthology.orchestra.run.smithi083.stderr:failed mds gid 7381 2024-03-06T07:29:12.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.035+0000 7f1e2d84e640 1 -- 172.21.15.83:0/2355398129 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1e1406b500 msgr2=0x7f1e1406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.035+0000 7f1e2d84e640 1 --2- 172.21.15.83:0/2355398129 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1e1406b500 0x7f1e1406d9c0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c034000 tx=0x7f1e1c007000 comp rx=0 tx=0).stop 2024-03-06T07:29:12.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.035+0000 7f1e2d84e640 1 -- 172.21.15.83:0/2355398129 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e280c3cb0 msgr2=0x7f1e280c4b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.035+0000 7f1e2d84e640 1 --2- 172.21.15.83:0/2355398129 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e280c3cb0 0x7f1e280c4b40 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f1e18011940 tx=0x7f1e18011e00 comp rx=0 tx=0).stop 2024-03-06T07:29:12.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.035+0000 7f1e2d84e640 1 -- 172.21.15.83:0/2355398129 shutdown_connections 2024-03-06T07:29:12.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.035+0000 7f1e2d84e640 1 --2- 172.21.15.83:0/2355398129 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1e1406b500 0x7f1e1406d9c0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.035+0000 7f1e2d84e640 1 --2- 172.21.15.83:0/2355398129 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1e280c5c90 0x7f1e280c6110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.035+0000 7f1e2d84e640 1 --2- 172.21.15.83:0/2355398129 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e280c3cb0 0x7f1e280c4b40 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.035+0000 7f1e2d84e640 1 --2- 172.21.15.83:0/2355398129 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e100039b0 0x7f1e2815c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.035+0000 7f1e2d84e640 1 -- 172.21.15.83:0/2355398129 >> 172.21.15.83:0/2355398129 conn(0x7f1e28072520 msgr2=0x7f1e280ad5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:12.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.035+0000 7f1e2d84e640 1 -- 172.21.15.83:0/2355398129 shutdown_connections 2024-03-06T07:29:12.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.035+0000 7f1e2d84e640 1 -- 172.21.15.83:0/2355398129 wait complete. 2024-03-06T07:29:12.048 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:12.049 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-03-06T07:29:12.049 DEBUG:teuthology.orchestra.run.smithi083:> mkdir pin 2024-03-06T07:29:12.049 DEBUG:teuthology.orchestra.run.smithi083:> touch pin/placeholder 2024-03-06T07:29:12.049 DEBUG:teuthology.orchestra.run.smithi083:> ') 2024-03-06T07:29:12.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:12.074 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 pin) 2024-03-06T07:29:12.131 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:12.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.271+0000 7fd39e66f640 1 Processor -- start 2024-03-06T07:29:12.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.271+0000 7fd39e66f640 1 -- start start 2024-03-06T07:29:12.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.272+0000 7fd39e66f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd3980d16e0 0x7fd3980d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.272+0000 7fd39e66f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3980cebe0 con 0x7fd39815bdd0 2024-03-06T07:29:12.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.272+0000 7fd39e66f640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3980ceec0 con 0x7fd39815fa10 2024-03-06T07:29:12.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.272+0000 7fd39e66f640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3980ce980 con 0x7fd3980d16e0 2024-03-06T07:29:12.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.272+0000 7fd39cc0c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd3980d16e0 0x7fd3980d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.272+0000 7fd397fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fd39815bdd0 0x7fd3980c5610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:51154/0 (socket says 172.21.15.83:51154) 2024-03-06T07:29:12.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.272+0000 7fd397fff640 1 -- 172.21.15.83:0/3942429697 learned_addr learned my addr 172.21.15.83:0/3942429697 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:29:12.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.272+0000 7fd39cc0c640 1 -- 172.21.15.83:0/3942429697 >> v1:172.21.15.183:6789/0 conn(0x7fd39815fa10 legacy=0x7fd3980ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:12.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.272+0000 7fd39cc0c640 1 -- 172.21.15.83:0/3942429697 >> v1:172.21.15.83:6789/0 conn(0x7fd39815bdd0 legacy=0x7fd3980c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.272+0000 7fd39cc0c640 1 -- 172.21.15.83:0/3942429697 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3980cf1a0 con 0x7fd3980d16e0 2024-03-06T07:29:12.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.273+0000 7fd39cc0c640 1 --2- 172.21.15.83:0/3942429697 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd3980d16e0 0x7fd3980d1af0 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7fd390009d90 tx=0x7fd39002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea6c565733833cbe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:12.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.273+0000 7fd396ffd640 1 -- 172.21.15.83:0/3942429697 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd390039070 con 0x7fd3980d16e0 2024-03-06T07:29:12.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.273+0000 7fd396ffd640 1 -- 172.21.15.83:0/3942429697 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd39002cbd0 con 0x7fd3980d16e0 2024-03-06T07:29:12.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.273+0000 7fd396ffd640 1 -- 172.21.15.83:0/3942429697 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd390034440 con 0x7fd3980d16e0 2024-03-06T07:29:12.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.274+0000 7fd39e66f640 1 -- 172.21.15.83:0/3942429697 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd3980d16e0 msgr2=0x7fd3980d1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.274+0000 7fd39e66f640 1 --2- 172.21.15.83:0/3942429697 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd3980d16e0 0x7fd3980d1af0 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7fd390009d90 tx=0x7fd39002be20 comp rx=0 tx=0).stop 2024-03-06T07:29:12.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.274+0000 7fd39e66f640 1 -- 172.21.15.83:0/3942429697 shutdown_connections 2024-03-06T07:29:12.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.274+0000 7fd39e66f640 1 --2- 172.21.15.83:0/3942429697 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd3980d16e0 0x7fd3980d1af0 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.274+0000 7fd39e66f640 1 -- 172.21.15.83:0/3942429697 >> 172.21.15.83:0/3942429697 conn(0x7fd39809a9f0 msgr2=0x7fd39809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:12.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.274+0000 7fd39e66f640 1 -- 172.21.15.83:0/3942429697 shutdown_connections 2024-03-06T07:29:12.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.274+0000 7fd39e66f640 1 -- 172.21.15.83:0/3942429697 wait complete. 2024-03-06T07:29:12.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.275+0000 7fd39e66f640 1 Processor -- start 2024-03-06T07:29:12.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.276+0000 7fd39e66f640 1 -- start start 2024-03-06T07:29:12.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.277+0000 7fd39e66f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd3980d16e0 0x7fd3980c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.277+0000 7fd39e66f640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd39815bdd0 0x7fd3981f1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.277+0000 7fd39e66f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd39815fa10 0x7fd3980b5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.277+0000 7fd39e66f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd39815c1b0 con 0x7fd3980d16e0 2024-03-06T07:29:12.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.277+0000 7fd39e66f640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3980cc5d0 con 0x7fd39815bdd0 2024-03-06T07:29:12.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.277+0000 7fd39e66f640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3980d2030 con 0x7fd39815fa10 2024-03-06T07:29:12.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.277+0000 7fd397fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd3980d16e0 0x7fd3980c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.277+0000 7fd397fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd3980d16e0 0x7fd3980c8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60246/0 (socket says 172.21.15.83:60246) 2024-03-06T07:29:12.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.277+0000 7fd397fff640 1 -- 172.21.15.83:0/3982460568 learned_addr learned my addr 172.21.15.83:0/3982460568 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:12.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.277+0000 7fd3977fe640 1 --2- 172.21.15.83:0/3982460568 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd39815bdd0 0x7fd3981f1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.277+0000 7fd397fff640 1 -- 172.21.15.83:0/3982460568 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd39815fa10 msgr2=0x7fd3980b5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.277+0000 7fd397fff640 1 --2- 172.21.15.83:0/3982460568 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd39815fa10 0x7fd3980b5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.277+0000 7fd397fff640 1 -- 172.21.15.83:0/3982460568 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd39815bdd0 msgr2=0x7fd3981f1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.277+0000 7fd397fff640 1 --2- 172.21.15.83:0/3982460568 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd39815bdd0 0x7fd3981f1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.277+0000 7fd397fff640 1 -- 172.21.15.83:0/3982460568 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3980c9470 con 0x7fd3980d16e0 2024-03-06T07:29:12.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.277+0000 7fd397fff640 1 --2- 172.21.15.83:0/3982460568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd3980d16e0 0x7fd3980c8b90 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7fd38800c400 tx=0x7fd38800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:12.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.278+0000 7fd394ff9640 1 -- 172.21.15.83:0/3982460568 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd388017070 con 0x7fd3980d16e0 2024-03-06T07:29:12.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.278+0000 7fd394ff9640 1 -- 172.21.15.83:0/3982460568 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd388005ce0 con 0x7fd3980d16e0 2024-03-06T07:29:12.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.278+0000 7fd39e66f640 1 -- 172.21.15.83:0/3982460568 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3980c9750 con 0x7fd3980d16e0 2024-03-06T07:29:12.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.278+0000 7fd394ff9640 1 -- 172.21.15.83:0/3982460568 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd388007310 con 0x7fd3980d16e0 2024-03-06T07:29:12.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.278+0000 7fd39e66f640 1 -- 172.21.15.83:0/3982460568 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3980c9d10 con 0x7fd3980d16e0 2024-03-06T07:29:12.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.279+0000 7fd39e66f640 1 -- 172.21.15.83:0/3982460568 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd39804f330 con 0x7fd3980d16e0 2024-03-06T07:29:12.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.279+0000 7fd394ff9640 1 -- 172.21.15.83:0/3982460568 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd38801a070 con 0x7fd3980d16e0 2024-03-06T07:29:12.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.280+0000 7fd394ff9640 1 --2- 172.21.15.83:0/3982460568 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd38406b4d0 0x7fd38406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.280+0000 7fd394ff9640 1 -- 172.21.15.83:0/3982460568 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd388089590 con 0x7fd3980d16e0 2024-03-06T07:29:12.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.280+0000 7fd3977fe640 1 --2- 172.21.15.83:0/3982460568 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd38406b4d0 0x7fd38406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.280+0000 7fd3977fe640 1 --2- 172.21.15.83:0/3982460568 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd38406b4d0 0x7fd38406d990 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fd39809aff0 tx=0x7fd38c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:12.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.284+0000 7fd394ff9640 1 -- 172.21.15.83:0/3982460568 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd3880584a0 con 0x7fd3980d16e0 2024-03-06T07:29:12.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.524+0000 7fd39e66f640 1 -- 172.21.15.83:0/3982460568 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3980b7a40 con 0x7fd3980d16e0 2024-03-06T07:29:12.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.525+0000 7fd394ff9640 1 -- 172.21.15.83:0/3982460568 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+3417 (secure 0 0 0) 0x7fd388057da0 con 0x7fd3980d16e0 2024-03-06T07:29:12.527 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:29:12.527 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":78,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:05.001149+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8628,8631]},"id":9}]} 2024-03-06T07:29:12.527 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 78 2024-03-06T07:29:12.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.527+0000 7fd39e66f640 1 -- 172.21.15.83:0/3982460568 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd38406b4d0 msgr2=0x7fd38406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.527+0000 7fd39e66f640 1 --2- 172.21.15.83:0/3982460568 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd38406b4d0 0x7fd38406d990 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fd39809aff0 tx=0x7fd38c008040 comp rx=0 tx=0).stop 2024-03-06T07:29:12.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.527+0000 7fd39e66f640 1 -- 172.21.15.83:0/3982460568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd3980d16e0 msgr2=0x7fd3980c8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.527+0000 7fd39e66f640 1 --2- 172.21.15.83:0/3982460568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd3980d16e0 0x7fd3980c8b90 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7fd38800c400 tx=0x7fd38800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.527+0000 7fd39e66f640 1 -- 172.21.15.83:0/3982460568 shutdown_connections 2024-03-06T07:29:12.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.527+0000 7fd39e66f640 1 --2- 172.21.15.83:0/3982460568 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd38406b4d0 0x7fd38406d990 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.527+0000 7fd39e66f640 1 --2- 172.21.15.83:0/3982460568 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd39815fa10 0x7fd3980b5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.527+0000 7fd39e66f640 1 --2- 172.21.15.83:0/3982460568 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd39815bdd0 0x7fd3981f1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.527+0000 7fd39e66f640 1 --2- 172.21.15.83:0/3982460568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd3980d16e0 0x7fd3980c8b90 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.527+0000 7fd39e66f640 1 -- 172.21.15.83:0/3982460568 >> 172.21.15.83:0/3982460568 conn(0x7fd39809a9f0 msgr2=0x7fd3980b3a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:12.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.527+0000 7fd39e66f640 1 -- 172.21.15.83:0/3982460568 shutdown_connections 2024-03-06T07:29:12.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:12.527+0000 7fd39e66f640 1 -- 172.21.15.83:0/3982460568 wait complete. 2024-03-06T07:29:12.540 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:29:12.656 INFO:teuthology.orchestra.run.smithi083.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:29:12.657 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:29:12.657 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/pin', 1)] 2024-03-06T07:29:14.658 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:14.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.758+0000 7f717e6af640 1 Processor -- start 2024-03-06T07:29:14.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.758+0000 7f717e6af640 1 -- start start 2024-03-06T07:29:14.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.758+0000 7f717e6af640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f717815dfe0 0x7f71780cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:14.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.758+0000 7f717e6af640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f71780cf830 0x7f71780d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:14.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.758+0000 7f717e6af640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71780b3430 con 0x7f71780d27f0 2024-03-06T07:29:14.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.758+0000 7f717e6af640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71780b31e0 con 0x7f71780cf830 2024-03-06T07:29:14.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.758+0000 7f717e6af640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71780b3d60 con 0x7f717815dfe0 2024-03-06T07:29:14.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.758+0000 7f71777fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f71780cf830 0x7f71780d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:14.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.759+0000 7f71777fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f71780cf830 0x7f71780d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60262/0 (socket says 172.21.15.83:60262) 2024-03-06T07:29:14.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.759+0000 7f71777fe640 1 -- 172.21.15.83:0/258817322 learned_addr learned my addr 172.21.15.83:0/258817322 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:14.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.759+0000 7f7177fff640 1 --2- 172.21.15.83:0/258817322 >> v2:172.21.15.183:3300/0 conn(0x7f717815dfe0 0x7f71780cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:14.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.759+0000 7f71777fe640 1 -- 172.21.15.83:0/258817322 >> v2:172.21.15.183:3300/0 conn(0x7f717815dfe0 msgr2=0x7f71780cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:14.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.759+0000 7f71777fe640 1 --2- 172.21.15.83:0/258817322 >> v2:172.21.15.183:3300/0 conn(0x7f717815dfe0 0x7f71780cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.759+0000 7f71777fe640 1 -- 172.21.15.83:0/258817322 >> v1:172.21.15.183:6789/0 conn(0x7f71780d27f0 legacy=0x7f71780d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:14.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.759+0000 7f71777fe640 1 -- 172.21.15.83:0/258817322 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71780b3710 con 0x7f71780cf830 2024-03-06T07:29:14.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.760+0000 7f71777fe640 1 --2- 172.21.15.83:0/258817322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71780cf830 0x7f71780d22b0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f716c009870 tx=0x7f716c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=469f3851e3d94b18 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:14.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.760+0000 7f7176ffd640 1 -- 172.21.15.83:0/258817322 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f716c034070 con 0x7f71780cf830 2024-03-06T07:29:14.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.760+0000 7f7176ffd640 1 -- 172.21.15.83:0/258817322 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f716c030860 con 0x7f71780cf830 2024-03-06T07:29:14.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.760+0000 7f7176ffd640 1 -- 172.21.15.83:0/258817322 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f716c030c30 con 0x7f71780cf830 2024-03-06T07:29:14.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.760+0000 7f717e6af640 1 -- 172.21.15.83:0/258817322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71780cf830 msgr2=0x7f71780d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:14.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.760+0000 7f717e6af640 1 --2- 172.21.15.83:0/258817322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71780cf830 0x7f71780d22b0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f716c009870 tx=0x7f716c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.761+0000 7f717e6af640 1 -- 172.21.15.83:0/258817322 shutdown_connections 2024-03-06T07:29:14.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.761+0000 7f717e6af640 1 --2- 172.21.15.83:0/258817322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71780cf830 0x7f71780d22b0 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.761+0000 7f717e6af640 1 --2- 172.21.15.83:0/258817322 >> v2:172.21.15.183:3300/0 conn(0x7f717815dfe0 0x7f71780cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.761+0000 7f717e6af640 1 -- 172.21.15.83:0/258817322 >> 172.21.15.83:0/258817322 conn(0x7f71780d3cd0 msgr2=0x7f71780d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:14.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.761+0000 7f717e6af640 1 -- 172.21.15.83:0/258817322 shutdown_connections 2024-03-06T07:29:14.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.761+0000 7f717e6af640 1 -- 172.21.15.83:0/258817322 wait complete. 2024-03-06T07:29:14.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.762+0000 7f717e6af640 1 Processor -- start 2024-03-06T07:29:14.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.763+0000 7f717e6af640 1 -- start start 2024-03-06T07:29:14.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.763+0000 7f717e6af640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f71780cf830 0x7f71781f42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:14.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.763+0000 7f717e6af640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f71780d27f0 0x7f71781f2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:14.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.763+0000 7f717e6af640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f717815dfe0 0x7f71780ccc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:14.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.763+0000 7f717e6af640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71780d1df0 con 0x7f717815dfe0 2024-03-06T07:29:14.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.763+0000 7f717e6af640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71780d1c70 con 0x7f71780d27f0 2024-03-06T07:29:14.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.763+0000 7f717e6af640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f717815e3c0 con 0x7f71780cf830 2024-03-06T07:29:14.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.764+0000 7f717cc4c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f717815dfe0 0x7f71780ccc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:14.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.764+0000 7f717cc4c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f717815dfe0 0x7f71780ccc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60264/0 (socket says 172.21.15.83:60264) 2024-03-06T07:29:14.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.764+0000 7f717cc4c640 1 -- 172.21.15.83:0/2580070722 learned_addr learned my addr 172.21.15.83:0/2580070722 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:14.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.764+0000 7f71777fe640 1 --2- 172.21.15.83:0/2580070722 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f71780d27f0 0x7f71781f2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:14.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.764+0000 7f7177fff640 1 --2- 172.21.15.83:0/2580070722 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f71780cf830 0x7f71781f42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:14.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.764+0000 7f717cc4c640 1 -- 172.21.15.83:0/2580070722 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f71780cf830 msgr2=0x7f71781f42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:14.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.764+0000 7f717cc4c640 1 --2- 172.21.15.83:0/2580070722 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f71780cf830 0x7f71781f42e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.764+0000 7f717cc4c640 1 -- 172.21.15.83:0/2580070722 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f71780d27f0 msgr2=0x7f71781f2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:14.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.764+0000 7f717cc4c640 1 --2- 172.21.15.83:0/2580070722 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f71780d27f0 0x7f71781f2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.764+0000 7f717cc4c640 1 -- 172.21.15.83:0/2580070722 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71781f6eb0 con 0x7f717815dfe0 2024-03-06T07:29:14.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.764+0000 7f7177fff640 1 --2- 172.21.15.83:0/2580070722 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f71780cf830 0x7f71781f42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:14.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.764+0000 7f717cc4c640 1 --2- 172.21.15.83:0/2580070722 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f717815dfe0 0x7f71780ccc10 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f717000c7a0 tx=0x7f717000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:14.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.764+0000 7f7174ff9640 1 -- 172.21.15.83:0/2580070722 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7170003b70 con 0x7f717815dfe0 2024-03-06T07:29:14.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.764+0000 7f7174ff9640 1 -- 172.21.15.83:0/2580070722 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7170003d30 con 0x7f717815dfe0 2024-03-06T07:29:14.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.764+0000 7f717e6af640 1 -- 172.21.15.83:0/2580070722 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71781f7190 con 0x7f717815dfe0 2024-03-06T07:29:14.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.764+0000 7f7174ff9640 1 -- 172.21.15.83:0/2580070722 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f717000aa90 con 0x7f717815dfe0 2024-03-06T07:29:14.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.764+0000 7f717e6af640 1 -- 172.21.15.83:0/2580070722 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71781f7750 con 0x7f717815dfe0 2024-03-06T07:29:14.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.766+0000 7f717e6af640 1 -- 172.21.15.83:0/2580070722 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f717804f330 con 0x7f717815dfe0 2024-03-06T07:29:14.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.766+0000 7f7174ff9640 1 -- 172.21.15.83:0/2580070722 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f717000ac30 con 0x7f717815dfe0 2024-03-06T07:29:14.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.766+0000 7f7174ff9640 1 --2- 172.21.15.83:0/2580070722 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f716406b500 0x7f716406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:14.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.766+0000 7f7177fff640 1 --2- 172.21.15.83:0/2580070722 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f716406b500 0x7f716406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:14.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.766+0000 7f7174ff9640 1 -- 172.21.15.83:0/2580070722 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f717008a130 con 0x7f717815dfe0 2024-03-06T07:29:14.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.767+0000 7f7177fff640 1 --2- 172.21.15.83:0/2580070722 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f716406b500 0x7f716406d9c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f7168006fd0 tx=0x7f7168008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:14.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:14.771+0000 7f7174ff9640 1 -- 172.21.15.83:0/2580070722 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7170058fb0 con 0x7f717815dfe0 2024-03-06T07:29:15.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:15.042+0000 7f717e6af640 1 -- 172.21.15.83:0/2580070722 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71781f7be0 con 0x7f717815dfe0 2024-03-06T07:29:15.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:15.042+0000 7f7174ff9640 1 -- 172.21.15.83:0/2580070722 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+3417 (secure 0 0 0) 0x7f71700588b0 con 0x7f717815dfe0 2024-03-06T07:29:15.044 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:29:15.044 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":78,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:05.001149+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8628,8631]},"id":9}]} 2024-03-06T07:29:15.045 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 78 2024-03-06T07:29:15.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:15.044+0000 7f717e6af640 1 -- 172.21.15.83:0/2580070722 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f716406b500 msgr2=0x7f716406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:15.044+0000 7f717e6af640 1 --2- 172.21.15.83:0/2580070722 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f716406b500 0x7f716406d9c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f7168006fd0 tx=0x7f7168008040 comp rx=0 tx=0).stop 2024-03-06T07:29:15.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:15.044+0000 7f717e6af640 1 -- 172.21.15.83:0/2580070722 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f717815dfe0 msgr2=0x7f71780ccc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:15.044+0000 7f717e6af640 1 --2- 172.21.15.83:0/2580070722 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f717815dfe0 0x7f71780ccc10 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f717000c7a0 tx=0x7f717000cc60 comp rx=0 tx=0).stop 2024-03-06T07:29:15.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:15.044+0000 7f717e6af640 1 -- 172.21.15.83:0/2580070722 shutdown_connections 2024-03-06T07:29:15.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:15.044+0000 7f717e6af640 1 --2- 172.21.15.83:0/2580070722 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f716406b500 0x7f716406d9c0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:15.044+0000 7f717e6af640 1 --2- 172.21.15.83:0/2580070722 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f717815dfe0 0x7f71780ccc10 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:15.044+0000 7f717e6af640 1 --2- 172.21.15.83:0/2580070722 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f71780d27f0 0x7f71781f2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:15.044+0000 7f717e6af640 1 --2- 172.21.15.83:0/2580070722 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f71780cf830 0x7f71781f42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:15.044+0000 7f717e6af640 1 -- 172.21.15.83:0/2580070722 >> 172.21.15.83:0/2580070722 conn(0x7f71780d3cd0 msgr2=0x7f71780b4f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:15.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:15.044+0000 7f717e6af640 1 -- 172.21.15.83:0/2580070722 shutdown_connections 2024-03-06T07:29:15.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:15.045+0000 7f717e6af640 1 -- 172.21.15.83:0/2580070722 wait complete. 2024-03-06T07:29:15.061 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:29:15.176 INFO:teuthology.orchestra.run.smithi083.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:29:15.176 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:29:15.176 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/pin', 1)] 2024-03-06T07:29:17.177 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:17.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.310+0000 7fb2afd56640 1 Processor -- start 2024-03-06T07:29:17.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.310+0000 7fb2afd56640 1 -- start start 2024-03-06T07:29:17.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.311+0000 7fb2afd56640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb2a80d22b0 0x7fb2a80d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.311+0000 7fb2afd56640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fb2a80b41f0 0x7fb2a80b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.311+0000 7fb2afd56640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2a80b3430 con 0x7fb2a815dfe0 2024-03-06T07:29:17.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.311+0000 7fb2afd56640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2a80d2c50 con 0x7fb2a80d22b0 2024-03-06T07:29:17.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.311+0000 7fb2afd56640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2a80b31e0 con 0x7fb2a80b41f0 2024-03-06T07:29:17.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.311+0000 7fb2ad2f1640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb2a80d22b0 0x7fb2a80d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.311+0000 7fb2ad2f1640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb2a80d22b0 0x7fb2a80d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57152/0 (socket says 172.21.15.83:57152) 2024-03-06T07:29:17.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.311+0000 7fb2ad2f1640 1 -- 172.21.15.83:0/62796786 learned_addr learned my addr 172.21.15.83:0/62796786 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:17.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.312+0000 7fb2ae2f3640 1 --2- 172.21.15.83:0/62796786 >> v2:172.21.15.183:3300/0 conn(0x7fb2a80b41f0 0x7fb2a80b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.312+0000 7fb2ad2f1640 1 -- 172.21.15.83:0/62796786 >> v2:172.21.15.183:3300/0 conn(0x7fb2a80b41f0 msgr2=0x7fb2a80b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:17.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.312+0000 7fb2ad2f1640 1 --2- 172.21.15.83:0/62796786 >> v2:172.21.15.183:3300/0 conn(0x7fb2a80b41f0 0x7fb2a80b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.312+0000 7fb2ad2f1640 1 -- 172.21.15.83:0/62796786 >> v1:172.21.15.183:6789/0 conn(0x7fb2a815dfe0 legacy=0x7fb2a80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:17.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.312+0000 7fb2ad2f1640 1 -- 172.21.15.83:0/62796786 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2a80b3710 con 0x7fb2a80d22b0 2024-03-06T07:29:17.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.312+0000 7fb2ae2f3640 1 --2- 172.21.15.83:0/62796786 >> v2:172.21.15.183:3300/0 conn(0x7fb2a80b41f0 0x7fb2a80b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:17.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.312+0000 7fb2ad2f1640 1 --2- 172.21.15.83:0/62796786 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2a80d22b0 0x7fb2a80d2710 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7fb298009870 tx=0x7fb29802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36f15b0e635d7959 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:17.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.313+0000 7fb2acaf0640 1 -- 172.21.15.83:0/62796786 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb298035020 con 0x7fb2a80d22b0 2024-03-06T07:29:17.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.313+0000 7fb2acaf0640 1 -- 172.21.15.83:0/62796786 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb29802d040 con 0x7fb2a80d22b0 2024-03-06T07:29:17.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.313+0000 7fb2acaf0640 1 -- 172.21.15.83:0/62796786 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb298031a80 con 0x7fb2a80d22b0 2024-03-06T07:29:17.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.313+0000 7fb2afd56640 1 -- 172.21.15.83:0/62796786 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2a80d22b0 msgr2=0x7fb2a80d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.313+0000 7fb2afd56640 1 --2- 172.21.15.83:0/62796786 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2a80d22b0 0x7fb2a80d2710 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7fb298009870 tx=0x7fb29802b900 comp rx=0 tx=0).stop 2024-03-06T07:29:17.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.313+0000 7fb2afd56640 1 -- 172.21.15.83:0/62796786 shutdown_connections 2024-03-06T07:29:17.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.313+0000 7fb2afd56640 1 --2- 172.21.15.83:0/62796786 >> v2:172.21.15.183:3300/0 conn(0x7fb2a80b41f0 0x7fb2a80b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.313+0000 7fb2afd56640 1 --2- 172.21.15.83:0/62796786 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2a80d22b0 0x7fb2a80d2710 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.313+0000 7fb2afd56640 1 -- 172.21.15.83:0/62796786 >> 172.21.15.83:0/62796786 conn(0x7fb2a80d3cd0 msgr2=0x7fb2a80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:17.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.313+0000 7fb2afd56640 1 -- 172.21.15.83:0/62796786 shutdown_connections 2024-03-06T07:29:17.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.313+0000 7fb2afd56640 1 -- 172.21.15.83:0/62796786 wait complete. 2024-03-06T07:29:17.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.315+0000 7fb2afd56640 1 Processor -- start 2024-03-06T07:29:17.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.315+0000 7fb2afd56640 1 -- start start 2024-03-06T07:29:17.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.316+0000 7fb2afd56640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2a80b41f0 0x7fb2a80cd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.316+0000 7fb2afd56640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb2a80d22b0 0x7fb2a80b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.316+0000 7fb2afd56640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb2a815dfe0 0x7fb2a80b6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.316+0000 7fb2afd56640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a80cfa00 con 0x7fb2a80b41f0 2024-03-06T07:29:17.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.316+0000 7fb2afd56640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a80b3d60 con 0x7fb2a80d22b0 2024-03-06T07:29:17.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.316+0000 7fb2afd56640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a80d1c70 con 0x7fb2a815dfe0 2024-03-06T07:29:17.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.316+0000 7fb2adaf2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2a80b41f0 0x7fb2a80cd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.316+0000 7fb2adaf2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2a80b41f0 0x7fb2a80cd270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57156/0 (socket says 172.21.15.83:57156) 2024-03-06T07:29:17.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.316+0000 7fb2adaf2640 1 -- 172.21.15.83:0/2145292708 learned_addr learned my addr 172.21.15.83:0/2145292708 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:17.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.316+0000 7fb2ae2f3640 1 --2- 172.21.15.83:0/2145292708 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb2a815dfe0 0x7fb2a80b6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.316+0000 7fb2ad2f1640 1 --2- 172.21.15.83:0/2145292708 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb2a80d22b0 0x7fb2a80b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.316+0000 7fb2adaf2640 1 -- 172.21.15.83:0/2145292708 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb2a815dfe0 msgr2=0x7fb2a80b6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.316+0000 7fb2adaf2640 1 --2- 172.21.15.83:0/2145292708 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb2a815dfe0 0x7fb2a80b6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.316+0000 7fb2adaf2640 1 -- 172.21.15.83:0/2145292708 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb2a80d22b0 msgr2=0x7fb2a80b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.316+0000 7fb2adaf2640 1 --2- 172.21.15.83:0/2145292708 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb2a80d22b0 0x7fb2a80b68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.316+0000 7fb2adaf2640 1 -- 172.21.15.83:0/2145292708 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2a80cd7b0 con 0x7fb2a80b41f0 2024-03-06T07:29:17.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.316+0000 7fb2adaf2640 1 --2- 172.21.15.83:0/2145292708 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2a80b41f0 0x7fb2a80cd270 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7fb2a400b3f0 tx=0x7fb2a400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:17.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.316+0000 7fb29e7fc640 1 -- 172.21.15.83:0/2145292708 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2a4015020 con 0x7fb2a80b41f0 2024-03-06T07:29:17.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.317+0000 7fb29e7fc640 1 -- 172.21.15.83:0/2145292708 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2a4011830 con 0x7fb2a80b41f0 2024-03-06T07:29:17.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.317+0000 7fb2afd56640 1 -- 172.21.15.83:0/2145292708 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2a80cc1d0 con 0x7fb2a80b41f0 2024-03-06T07:29:17.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.317+0000 7fb29e7fc640 1 -- 172.21.15.83:0/2145292708 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2a4011c20 con 0x7fb2a80b41f0 2024-03-06T07:29:17.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.317+0000 7fb2afd56640 1 -- 172.21.15.83:0/2145292708 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2a80cc790 con 0x7fb2a80b41f0 2024-03-06T07:29:17.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.318+0000 7fb2afd56640 1 -- 172.21.15.83:0/2145292708 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2a804f330 con 0x7fb2a80b41f0 2024-03-06T07:29:17.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.318+0000 7fb29e7fc640 1 -- 172.21.15.83:0/2145292708 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb2a4018020 con 0x7fb2a80b41f0 2024-03-06T07:29:17.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.318+0000 7fb29e7fc640 1 --2- 172.21.15.83:0/2145292708 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb28c06b4d0 0x7fb28c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.318+0000 7fb29e7fc640 1 -- 172.21.15.83:0/2145292708 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb2a4089d90 con 0x7fb2a80b41f0 2024-03-06T07:29:17.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.318+0000 7fb2ad2f1640 1 --2- 172.21.15.83:0/2145292708 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb28c06b4d0 0x7fb28c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.319+0000 7fb2ad2f1640 1 --2- 172.21.15.83:0/2145292708 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb28c06b4d0 0x7fb28c06d990 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fb298036000 tx=0x7fb29803c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:17.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.327+0000 7fb29e7fc640 1 -- 172.21.15.83:0/2145292708 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb2a401f030 con 0x7fb2a80b41f0 2024-03-06T07:29:17.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.546+0000 7fb2afd56640 1 -- 172.21.15.83:0/2145292708 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2a80b0af0 con 0x7fb2a80b41f0 2024-03-06T07:29:17.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.547+0000 7fb29e7fc640 1 -- 172.21.15.83:0/2145292708 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+3417 (secure 0 0 0) 0x7fb2a40172c0 con 0x7fb2a80b41f0 2024-03-06T07:29:17.549 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:29:17.549 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":78,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:05.001149+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8628,8631]},"id":9}]} 2024-03-06T07:29:17.549 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 78 2024-03-06T07:29:17.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.549+0000 7fb2afd56640 1 -- 172.21.15.83:0/2145292708 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb28c06b4d0 msgr2=0x7fb28c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.549+0000 7fb2afd56640 1 --2- 172.21.15.83:0/2145292708 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb28c06b4d0 0x7fb28c06d990 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fb298036000 tx=0x7fb29803c040 comp rx=0 tx=0).stop 2024-03-06T07:29:17.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.549+0000 7fb2afd56640 1 -- 172.21.15.83:0/2145292708 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2a80b41f0 msgr2=0x7fb2a80cd270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.549+0000 7fb2afd56640 1 --2- 172.21.15.83:0/2145292708 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2a80b41f0 0x7fb2a80cd270 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7fb2a400b3f0 tx=0x7fb2a400b8b0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.549+0000 7fb2afd56640 1 -- 172.21.15.83:0/2145292708 shutdown_connections 2024-03-06T07:29:17.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.549+0000 7fb2afd56640 1 --2- 172.21.15.83:0/2145292708 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb28c06b4d0 0x7fb28c06d990 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.549+0000 7fb2afd56640 1 --2- 172.21.15.83:0/2145292708 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb2a815dfe0 0x7fb2a80b6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.549+0000 7fb2afd56640 1 --2- 172.21.15.83:0/2145292708 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb2a80d22b0 0x7fb2a80b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.549+0000 7fb2afd56640 1 --2- 172.21.15.83:0/2145292708 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2a80b41f0 0x7fb2a80cd270 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.549+0000 7fb2afd56640 1 -- 172.21.15.83:0/2145292708 >> 172.21.15.83:0/2145292708 conn(0x7fb2a80d3cd0 msgr2=0x7fb2a80cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:17.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.549+0000 7fb2afd56640 1 -- 172.21.15.83:0/2145292708 shutdown_connections 2024-03-06T07:29:17.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:17.549+0000 7fb2afd56640 1 -- 172.21.15.83:0/2145292708 wait complete. 2024-03-06T07:29:17.561 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:29:17.708 INFO:teuthology.orchestra.run.smithi083.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:29:17.709 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:29:17.709 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/pin', 1)] 2024-03-06T07:29:19.710 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:19.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.809+0000 7f966858c640 1 Processor -- start 2024-03-06T07:29:19.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.809+0000 7f966858c640 1 -- start start 2024-03-06T07:29:19.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.810+0000 7f966858c640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f966015ab00 0x7f96600b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.810+0000 7f966858c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96600afdd0 0x7f966015bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.810+0000 7f966858c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96600b4e30 con 0x7f96600b48f0 2024-03-06T07:29:19.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.810+0000 7f966858c640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96600b5110 con 0x7f966015ab00 2024-03-06T07:29:19.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.810+0000 7f966858c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f966015b310 con 0x7f96600afdd0 2024-03-06T07:29:19.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.810+0000 7f9666328640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f966015ab00 0x7f96600b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.810+0000 7f9665b27640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f96600b48f0 0x7f96600af6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:51070/0 (socket says 172.21.15.83:51070) 2024-03-06T07:29:19.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.810+0000 7f9665b27640 1 -- 172.21.15.83:0/1625801467 learned_addr learned my addr 172.21.15.83:0/1625801467 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:29:19.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.810+0000 7f9666b29640 1 --2- 172.21.15.83:0/1625801467 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96600afdd0 0x7f966015bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.811+0000 7f9666b29640 1 -- 172.21.15.83:0/1625801467 >> v2:172.21.15.183:3300/0 conn(0x7f966015ab00 msgr2=0x7f96600b43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:19.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.811+0000 7f9666b29640 1 --2- 172.21.15.83:0/1625801467 >> v2:172.21.15.183:3300/0 conn(0x7f966015ab00 0x7f96600b43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.811+0000 7f9666b29640 1 -- 172.21.15.83:0/1625801467 >> v1:172.21.15.183:6789/0 conn(0x7f96600b48f0 legacy=0x7f96600af6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.811+0000 7f9666b29640 1 -- 172.21.15.83:0/1625801467 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f966015b5f0 con 0x7f96600afdd0 2024-03-06T07:29:19.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.811+0000 7f9665326640 1 -- 172.21.15.83:0/1625801467 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1363674040 0 0) 0x7f96600b4e30 con 0x7f96600b48f0 2024-03-06T07:29:19.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.811+0000 7f9666328640 1 --2- 172.21.15.83:0/1625801467 >> v2:172.21.15.183:3300/0 conn(0x7f966015ab00 0x7f96600b43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:19.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.811+0000 7f9666b29640 1 --2- 172.21.15.83:0/1625801467 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96600afdd0 0x7f966015bb10 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9658009d90 tx=0x7f965802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a6f4f06399e9f27 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:19.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.812+0000 7f9665326640 1 -- 172.21.15.83:0/1625801467 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9658039070 con 0x7f96600afdd0 2024-03-06T07:29:19.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.812+0000 7f9665326640 1 -- 172.21.15.83:0/1625801467 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f965802cc50 con 0x7f96600afdd0 2024-03-06T07:29:19.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.812+0000 7f9665326640 1 -- 172.21.15.83:0/1625801467 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9658034440 con 0x7f96600afdd0 2024-03-06T07:29:19.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.812+0000 7f966858c640 1 -- 172.21.15.83:0/1625801467 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96600afdd0 msgr2=0x7f966015bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.812+0000 7f966858c640 1 --2- 172.21.15.83:0/1625801467 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96600afdd0 0x7f966015bb10 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9658009d90 tx=0x7f965802bea0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.812+0000 7f966858c640 1 -- 172.21.15.83:0/1625801467 shutdown_connections 2024-03-06T07:29:19.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.812+0000 7f966858c640 1 --2- 172.21.15.83:0/1625801467 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96600afdd0 0x7f966015bb10 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.812+0000 7f966858c640 1 --2- 172.21.15.83:0/1625801467 >> v2:172.21.15.183:3300/0 conn(0x7f966015ab00 0x7f96600b43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.812+0000 7f966858c640 1 -- 172.21.15.83:0/1625801467 >> 172.21.15.83:0/1625801467 conn(0x7f966006c0c0 msgr2=0x7f966006c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:19.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.812+0000 7f966858c640 1 -- 172.21.15.83:0/1625801467 shutdown_connections 2024-03-06T07:29:19.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.812+0000 7f966858c640 1 -- 172.21.15.83:0/1625801467 wait complete. 2024-03-06T07:29:19.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.813+0000 7f966858c640 1 Processor -- start 2024-03-06T07:29:19.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.813+0000 7f966858c640 1 -- start start 2024-03-06T07:29:19.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.814+0000 7f966858c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96600afdd0 0x7f96600cf250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.814+0000 7f966858c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96600b48f0 0x7f96600b3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.814+0000 7f966858c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f966015ab00 0x7f9660149020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.814+0000 7f966858c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f966015b060 con 0x7f966015ab00 2024-03-06T07:29:19.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.814+0000 7f966858c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f966015aee0 con 0x7f96600b48f0 2024-03-06T07:29:19.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.814+0000 7f966858c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f966015c050 con 0x7f96600afdd0 2024-03-06T07:29:19.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.814+0000 7f9666b29640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f966015ab00 0x7f9660149020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.814+0000 7f9666b29640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f966015ab00 0x7f9660149020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57162/0 (socket says 172.21.15.83:57162) 2024-03-06T07:29:19.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.814+0000 7f9666b29640 1 -- 172.21.15.83:0/3691992908 learned_addr learned my addr 172.21.15.83:0/3691992908 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:19.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.814+0000 7f9666328640 1 --2- 172.21.15.83:0/3691992908 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96600afdd0 0x7f96600cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.814+0000 7f9665b27640 1 --2- 172.21.15.83:0/3691992908 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96600b48f0 0x7f96600b3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.814+0000 7f9666b29640 1 -- 172.21.15.83:0/3691992908 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96600afdd0 msgr2=0x7f96600cf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.814+0000 7f9666b29640 1 --2- 172.21.15.83:0/3691992908 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96600afdd0 0x7f96600cf250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.814+0000 7f9666b29640 1 -- 172.21.15.83:0/3691992908 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96600b48f0 msgr2=0x7f96600b3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.814+0000 7f9666b29640 1 --2- 172.21.15.83:0/3691992908 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96600b48f0 0x7f96600b3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.814+0000 7f9666b29640 1 -- 172.21.15.83:0/3691992908 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96600d0a60 con 0x7f966015ab00 2024-03-06T07:29:19.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.815+0000 7f9666b29640 1 --2- 172.21.15.83:0/3691992908 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f966015ab00 0x7f9660149020 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f9658009d90 tx=0x7f9658004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:19.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.815+0000 7f9656ffd640 1 -- 172.21.15.83:0/3691992908 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9658039050 con 0x7f966015ab00 2024-03-06T07:29:19.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.815+0000 7f9656ffd640 1 -- 172.21.15.83:0/3691992908 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9658045070 con 0x7f966015ab00 2024-03-06T07:29:19.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.815+0000 7f966858c640 1 -- 172.21.15.83:0/3691992908 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96600cf9c0 con 0x7f966015ab00 2024-03-06T07:29:19.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.815+0000 7f9656ffd640 1 -- 172.21.15.83:0/3691992908 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9658009450 con 0x7f966015ab00 2024-03-06T07:29:19.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.815+0000 7f966858c640 1 -- 172.21.15.83:0/3691992908 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96600cff80 con 0x7f966015ab00 2024-03-06T07:29:19.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.816+0000 7f966858c640 1 -- 172.21.15.83:0/3691992908 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f966004f330 con 0x7f966015ab00 2024-03-06T07:29:19.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.816+0000 7f9656ffd640 1 -- 172.21.15.83:0/3691992908 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f965803e610 con 0x7f966015ab00 2024-03-06T07:29:19.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.817+0000 7f9656ffd640 1 --2- 172.21.15.83:0/3691992908 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f964406b500 0x7f964406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.817+0000 7f9656ffd640 1 -- 172.21.15.83:0/3691992908 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f96580aca30 con 0x7f966015ab00 2024-03-06T07:29:19.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.817+0000 7f9666328640 1 --2- 172.21.15.83:0/3691992908 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f964406b500 0x7f964406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.817+0000 7f9666328640 1 --2- 172.21.15.83:0/3691992908 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f964406b500 0x7f964406d9c0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f96600ccf60 tx=0x7f965c009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:19.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:19.821+0000 7f9656ffd640 1 -- 172.21.15.83:0/3691992908 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f965807b260 con 0x7f966015ab00 2024-03-06T07:29:20.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:20.093+0000 7f966858c640 1 -- 172.21.15.83:0/3691992908 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f966006f550 con 0x7f966015ab00 2024-03-06T07:29:20.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:20.093+0000 7f9656ffd640 1 -- 172.21.15.83:0/3691992908 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+3417 (secure 0 0 0) 0x7f965807f300 con 0x7f966015ab00 2024-03-06T07:29:20.096 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:29:20.096 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":78,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:05.001149+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8628,8631]},"id":9}]} 2024-03-06T07:29:20.096 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 78 2024-03-06T07:29:20.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:20.095+0000 7f966858c640 1 -- 172.21.15.83:0/3691992908 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f964406b500 msgr2=0x7f964406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:20.095+0000 7f966858c640 1 --2- 172.21.15.83:0/3691992908 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f964406b500 0x7f964406d9c0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f96600ccf60 tx=0x7f965c009190 comp rx=0 tx=0).stop 2024-03-06T07:29:20.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:20.095+0000 7f966858c640 1 -- 172.21.15.83:0/3691992908 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f966015ab00 msgr2=0x7f9660149020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:20.095+0000 7f966858c640 1 --2- 172.21.15.83:0/3691992908 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f966015ab00 0x7f9660149020 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f9658009d90 tx=0x7f9658004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:20.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:20.095+0000 7f966858c640 1 -- 172.21.15.83:0/3691992908 shutdown_connections 2024-03-06T07:29:20.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:20.095+0000 7f966858c640 1 --2- 172.21.15.83:0/3691992908 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f964406b500 0x7f964406d9c0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:20.095+0000 7f966858c640 1 --2- 172.21.15.83:0/3691992908 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f966015ab00 0x7f9660149020 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:20.095+0000 7f966858c640 1 --2- 172.21.15.83:0/3691992908 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96600b48f0 0x7f96600b3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:20.095+0000 7f966858c640 1 --2- 172.21.15.83:0/3691992908 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96600afdd0 0x7f96600cf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:20.095+0000 7f966858c640 1 -- 172.21.15.83:0/3691992908 >> 172.21.15.83:0/3691992908 conn(0x7f966006c0c0 msgr2=0x7f9660071d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:20.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:20.095+0000 7f966858c640 1 -- 172.21.15.83:0/3691992908 shutdown_connections 2024-03-06T07:29:20.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:20.095+0000 7f966858c640 1 -- 172.21.15.83:0/3691992908 wait complete. 2024-03-06T07:29:20.112 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:29:20.259 INFO:teuthology.orchestra.run.smithi083.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:29:20.259 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:29:20.259 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/pin', 1)] 2024-03-06T07:29:22.260 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:22.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.360+0000 7f734973e640 1 Processor -- start 2024-03-06T07:29:22.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.360+0000 7f734973e640 1 -- start start 2024-03-06T07:29:22.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.362+0000 7f734973e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f73440d4760 0x7f73440b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.362+0000 7f734973e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f73440d0a80 0x7f73440cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.362+0000 7f734973e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73440d25d0 con 0x7f73440b4170 2024-03-06T07:29:22.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.362+0000 7f734973e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73440d4bf0 con 0x7f73440d0a80 2024-03-06T07:29:22.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.362+0000 7f734973e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73440ce9e0 con 0x7f73440d4760 2024-03-06T07:29:22.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.362+0000 7f73437fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f73440d0a80 0x7f73440cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.362+0000 7f7342ffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f73440b4170 0x7f73440d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48396/0 (socket says 172.21.15.83:48396) 2024-03-06T07:29:22.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.362+0000 7f7342ffd640 1 -- 172.21.15.83:0/1898895009 learned_addr learned my addr 172.21.15.83:0/1898895009 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:29:22.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.362+0000 7f73427fc640 1 --2- 172.21.15.83:0/1898895009 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f73440d4760 0x7f73440b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.362+0000 7f73437fe640 1 -- 172.21.15.83:0/1898895009 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f73440d4760 msgr2=0x7f73440b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:22.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.362+0000 7f73437fe640 1 --2- 172.21.15.83:0/1898895009 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f73440d4760 0x7f73440b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.362+0000 7f73437fe640 1 -- 172.21.15.83:0/1898895009 >> v1:172.21.15.83:6789/0 conn(0x7f73440b4170 legacy=0x7f73440d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.362+0000 7f7341ffb640 1 -- 172.21.15.83:0/1898895009 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2482874312 0 0) 0x7f73440d25d0 con 0x7f73440b4170 2024-03-06T07:29:22.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.362+0000 7f73437fe640 1 -- 172.21.15.83:0/1898895009 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73440d2870 con 0x7f73440d0a80 2024-03-06T07:29:22.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.362+0000 7f73427fc640 1 --2- 172.21.15.83:0/1898895009 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f73440d4760 0x7f73440b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:22.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.363+0000 7f73437fe640 1 --2- 172.21.15.83:0/1898895009 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73440d0a80 0x7f73440cee70 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f733c009d90 tx=0x7f733c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26886e3873d770a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:22.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.363+0000 7f7341ffb640 1 -- 172.21.15.83:0/1898895009 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f733c039070 con 0x7f73440d0a80 2024-03-06T07:29:22.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.363+0000 7f7341ffb640 1 -- 172.21.15.83:0/1898895009 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f733c02cc50 con 0x7f73440d0a80 2024-03-06T07:29:22.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.363+0000 7f7341ffb640 1 -- 172.21.15.83:0/1898895009 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f733c034440 con 0x7f73440d0a80 2024-03-06T07:29:22.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.364+0000 7f734973e640 1 -- 172.21.15.83:0/1898895009 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73440d0a80 msgr2=0x7f73440cee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.364+0000 7f734973e640 1 --2- 172.21.15.83:0/1898895009 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73440d0a80 0x7f73440cee70 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f733c009d90 tx=0x7f733c02bea0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.364+0000 7f734973e640 1 -- 172.21.15.83:0/1898895009 shutdown_connections 2024-03-06T07:29:22.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.364+0000 7f734973e640 1 --2- 172.21.15.83:0/1898895009 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73440d0a80 0x7f73440cee70 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.364+0000 7f734973e640 1 --2- 172.21.15.83:0/1898895009 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f73440d4760 0x7f73440b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.364+0000 7f734973e640 1 -- 172.21.15.83:0/1898895009 >> 172.21.15.83:0/1898895009 conn(0x7f73440d3c40 msgr2=0x7f73440d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:22.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.364+0000 7f734973e640 1 -- 172.21.15.83:0/1898895009 shutdown_connections 2024-03-06T07:29:22.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.364+0000 7f734973e640 1 -- 172.21.15.83:0/1898895009 wait complete. 2024-03-06T07:29:22.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.365+0000 7f734973e640 1 Processor -- start 2024-03-06T07:29:22.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.365+0000 7f734973e640 1 -- start start 2024-03-06T07:29:22.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.366+0000 7f734973e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f73440b4170 0x7f73440ccb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.366+0000 7f734973e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73440d0a80 0x7f7344157110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.366+0000 7f734973e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73440b6860 0x7f73441f6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.366+0000 7f734973e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73440b4dd0 con 0x7f73440b6860 2024-03-06T07:29:22.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.366+0000 7f734973e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73440b4550 con 0x7f73440d0a80 2024-03-06T07:29:22.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.366+0000 7f734973e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73440b4f50 con 0x7f73440b4170 2024-03-06T07:29:22.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.366+0000 7f73437fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73440b6860 0x7f73441f6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.366+0000 7f73437fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73440b6860 0x7f73441f6eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57168/0 (socket says 172.21.15.83:57168) 2024-03-06T07:29:22.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.366+0000 7f73437fe640 1 -- 172.21.15.83:0/1767743168 learned_addr learned my addr 172.21.15.83:0/1767743168 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:22.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.366+0000 7f73437fe640 1 -- 172.21.15.83:0/1767743168 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f73440b4170 msgr2=0x7f73440ccb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.366+0000 7f73437fe640 1 --2- 172.21.15.83:0/1767743168 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f73440b4170 0x7f73440ccb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.366+0000 7f73437fe640 1 -- 172.21.15.83:0/1767743168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73440d0a80 msgr2=0x7f7344157110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.366+0000 7f73437fe640 1 --2- 172.21.15.83:0/1767743168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73440d0a80 0x7f7344157110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.366+0000 7f73437fe640 1 -- 172.21.15.83:0/1767743168 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73441f7c90 con 0x7f73440b6860 2024-03-06T07:29:22.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.366+0000 7f73437fe640 1 --2- 172.21.15.83:0/1767743168 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73440b6860 0x7f73441f6eb0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f733c009d90 tx=0x7f733c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:22.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.367+0000 7f732b7fe640 1 -- 172.21.15.83:0/1767743168 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f733c039050 con 0x7f73440b6860 2024-03-06T07:29:22.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.367+0000 7f732b7fe640 1 -- 172.21.15.83:0/1767743168 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f733c045070 con 0x7f73440b6860 2024-03-06T07:29:22.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.367+0000 7f734973e640 1 -- 172.21.15.83:0/1767743168 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73440b6d60 con 0x7f73440b6860 2024-03-06T07:29:22.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.367+0000 7f732b7fe640 1 -- 172.21.15.83:0/1767743168 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f733c03e610 con 0x7f73440b6860 2024-03-06T07:29:22.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.367+0000 7f734973e640 1 -- 172.21.15.83:0/1767743168 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73440cc380 con 0x7f73440b6860 2024-03-06T07:29:22.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.368+0000 7f734973e640 1 -- 172.21.15.83:0/1767743168 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f734404f330 con 0x7f73440b6860 2024-03-06T07:29:22.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.368+0000 7f732b7fe640 1 -- 172.21.15.83:0/1767743168 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f733c03e7b0 con 0x7f73440b6860 2024-03-06T07:29:22.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.368+0000 7f732b7fe640 1 --2- 172.21.15.83:0/1767743168 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f733006b4d0 0x7f733006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.369+0000 7f732b7fe640 1 -- 172.21.15.83:0/1767743168 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f733c0acc90 con 0x7f73440b6860 2024-03-06T07:29:22.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.369+0000 7f7342ffd640 1 --2- 172.21.15.83:0/1767743168 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f733006b4d0 0x7f733006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.369+0000 7f7342ffd640 1 --2- 172.21.15.83:0/1767743168 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f733006b4d0 0x7f733006d990 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f7334007fd0 tx=0x7f7334007920 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:22.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.377+0000 7f732b7fe640 1 -- 172.21.15.83:0/1767743168 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f733c032e20 con 0x7f73440b6860 2024-03-06T07:29:22.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.697+0000 7f734973e640 1 -- 172.21.15.83:0/1767743168 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73440733b0 con 0x7f73440b6860 2024-03-06T07:29:22.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.698+0000 7f732b7fe640 1 -- 172.21.15.83:0/1767743168 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+3418 (secure 0 0 0) 0x7f733c07bcc0 con 0x7f73440b6860 2024-03-06T07:29:22.700 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:29:22.700 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":79,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:21.589667+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:active","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631,8628]},"id":9}]} 2024-03-06T07:29:22.700 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 79 2024-03-06T07:29:22.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.701+0000 7f734973e640 1 -- 172.21.15.83:0/1767743168 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f733006b4d0 msgr2=0x7f733006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.701+0000 7f734973e640 1 --2- 172.21.15.83:0/1767743168 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f733006b4d0 0x7f733006d990 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f7334007fd0 tx=0x7f7334007920 comp rx=0 tx=0).stop 2024-03-06T07:29:22.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.701+0000 7f734973e640 1 -- 172.21.15.83:0/1767743168 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73440b6860 msgr2=0x7f73441f6eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.701+0000 7f734973e640 1 --2- 172.21.15.83:0/1767743168 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73440b6860 0x7f73441f6eb0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f733c009d90 tx=0x7f733c004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:22.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.701+0000 7f734973e640 1 -- 172.21.15.83:0/1767743168 shutdown_connections 2024-03-06T07:29:22.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.701+0000 7f734973e640 1 --2- 172.21.15.83:0/1767743168 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f733006b4d0 0x7f733006d990 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.701+0000 7f734973e640 1 --2- 172.21.15.83:0/1767743168 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73440b6860 0x7f73441f6eb0 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.701+0000 7f734973e640 1 --2- 172.21.15.83:0/1767743168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73440d0a80 0x7f7344157110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.701+0000 7f734973e640 1 --2- 172.21.15.83:0/1767743168 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f73440b4170 0x7f73440ccb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.701+0000 7f734973e640 1 -- 172.21.15.83:0/1767743168 >> 172.21.15.83:0/1767743168 conn(0x7f73440d3c40 msgr2=0x7f73440b55c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:22.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.701+0000 7f734973e640 1 -- 172.21.15.83:0/1767743168 shutdown_connections 2024-03-06T07:29:22.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:22.701+0000 7f734973e640 1 -- 172.21.15.83:0/1767743168 wait complete. 2024-03-06T07:29:22.717 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:29:22.834 INFO:teuthology.orchestra.run.smithi083.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"10","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/pin","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"5","version":"5","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2024-03-06T07:29:22.835 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "10" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/pin", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "5", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:29:22.835 INFO:tasks.cephfs.cephfs_test_case:[('/pin', 1)] =?= [('/pin', 1)] 2024-03-06T07:29:22.835 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:29:22.835 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:22.851 DEBUG:teuthology.orchestra.run.smithi083:> stdin-killer -- python3 -c ' 2024-03-06T07:29:22.851 DEBUG:teuthology.orchestra.run.smithi083:> import fcntl 2024-03-06T07:29:22.851 DEBUG:teuthology.orchestra.run.smithi083:> import sys 2024-03-06T07:29:22.851 DEBUG:teuthology.orchestra.run.smithi083:> import time 2024-03-06T07:29:22.851 DEBUG:teuthology.orchestra.run.smithi083:> import os 2024-03-06T07:29:22.851 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:29:22.851 DEBUG:teuthology.orchestra.run.smithi083:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-06T07:29:22.851 DEBUG:teuthology.orchestra.run.smithi083:> 2024-03-06T07:29:22.851 DEBUG:teuthology.orchestra.run.smithi083:> os.mkdir("/home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked") 2024-03-06T07:29:22.851 DEBUG:teuthology.orchestra.run.smithi083:> fd = os.open("/home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked", os.O_RDONLY) 2024-03-06T07:29:22.852 DEBUG:teuthology.orchestra.run.smithi083:> while True: 2024-03-06T07:29:22.852 DEBUG:teuthology.orchestra.run.smithi083:> print("open_dir_background: keeping dir open", file=sys.stderr) 2024-03-06T07:29:22.852 DEBUG:teuthology.orchestra.run.smithi083:> try: 2024-03-06T07:29:22.852 DEBUG:teuthology.orchestra.run.smithi083:> if os.read(0, 4096) == b"": 2024-03-06T07:29:22.852 DEBUG:teuthology.orchestra.run.smithi083:> break 2024-03-06T07:29:22.852 DEBUG:teuthology.orchestra.run.smithi083:> except BlockingIOError: 2024-03-06T07:29:22.852 DEBUG:teuthology.orchestra.run.smithi083:> pass 2024-03-06T07:29:22.852 DEBUG:teuthology.orchestra.run.smithi083:> time.sleep(2) 2024-03-06T07:29:22.852 DEBUG:teuthology.orchestra.run.smithi083:> ' 2024-03-06T07:29:22.892 DEBUG:teuthology.orchestra.run.smithi083:> stat /home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked 2024-03-06T07:29:22.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:29:22.910 INFO:teuthology.orchestra.run.smithi083.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked': No such file or directory 2024-03-06T07:29:22.966 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:23.911 DEBUG:teuthology.orchestra.run.smithi083:> stat /home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked 2024-03-06T07:29:23.926 INFO:teuthology.orchestra.run.smithi083.stdout: File: /home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked 2024-03-06T07:29:23.927 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 0 Blocks: 1 IO Block: 4096 directory 2024-03-06T07:29:23.927 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 29h/41d Inode: 2199023255552 Links: 2 2024-03-06T07:29:23.927 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0755/drwxr-xr-x) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-06T07:29:23.927 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-06T07:29:23.927 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-03-06 07:29:22.958770457 +0000 2024-03-06T07:29:23.927 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-03-06 07:29:22.958770457 +0000 2024-03-06T07:29:23.927 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-03-06 07:29:22.958770457 +0000 2024-03-06T07:29:23.927 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: - 2024-03-06T07:29:23.927 INFO:tasks.cephfs.mount:File pin/to-be-unlinked became visible from 0 after 1s 2024-03-06T07:29:23.928 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:23.928 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir pin/to-be-unlinked) 2024-03-06T07:29:23.985 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:29:24.163 INFO:teuthology.orchestra.run.smithi083.stdout:{"mds_cache":{"num_strays":1}} 2024-03-06T07:29:24.164 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 1 } } 2024-03-06T07:29:24.164 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:29:24.336 INFO:teuthology.orchestra.run.smithi083.stdout:{"mds_cache":{"strays_created":1}} 2024-03-06T07:29:24.337 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1 } } 2024-03-06T07:29:24.337 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:29:24.455 INFO:teuthology.orchestra.run.smithi083.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-06T07:29:24.456 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-06T07:29:24.456 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:29:24.585 INFO:teuthology.orchestra.run.smithi083.stdout:{"message":"","return_code":0} 2024-03-06T07:29:24.585 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:29:24.585 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json cache drop 2024-03-06T07:29:24.707 INFO:teuthology.orchestra.run.smithi083.stdout:{"client_recall":{"return_code":0,"message":"(0) Success","recalled":0},"flush_journal":{"return_code":0,"message":""},"trim_cache":{"trimmed":0},"cache":{"pool":{"items":323,"bytes":64695}},"duration":0.0065343900000000002} 2024-03-06T07:29:24.708 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cache": { "pool": { "bytes": 64695, "items": 323 } }, "client_recall": { "message": "(0) Success", "recalled": 0, "return_code": 0 }, "duration": 0.00653439, "flush_journal": { "message": "", "return_code": 0 }, "trim_cache": { "trimmed": 0 } } 2024-03-06T07:29:24.708 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2024-03-06T07:29:24.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.810+0000 7fc761c6b640 1 Processor -- start 2024-03-06T07:29:24.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.810+0000 7fc761c6b640 1 -- start start 2024-03-06T07:29:24.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.811+0000 7fc761c6b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc75c0d4860 0x7fc75c0d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.811+0000 7fc761c6b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc75c0b0e30 0x7fc75c0b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.811+0000 7fc761c6b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc75c1618b0 con 0x7fc75c1614d0 2024-03-06T07:29:24.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.811+0000 7fc761c6b640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc75c0d4cf0 con 0x7fc75c0d4860 2024-03-06T07:29:24.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.811+0000 7fc761c6b640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc75c0cf400 con 0x7fc75c0b0e30 2024-03-06T07:29:24.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.811+0000 7fc75affd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc75c0d4860 0x7fc75c0d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.811+0000 7fc75affd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc75c0d4860 0x7fc75c0d2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57170/0 (socket says 172.21.15.83:57170) 2024-03-06T07:29:24.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.811+0000 7fc75affd640 1 -- 172.21.15.83:0/1302070610 learned_addr learned my addr 172.21.15.83:0/1302070610 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:24.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.811+0000 7fc75bfff640 1 --2- 172.21.15.83:0/1302070610 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc75c0b0e30 0x7fc75c0b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.811+0000 7fc75affd640 1 -- 172.21.15.83:0/1302070610 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc75c0b0e30 msgr2=0x7fc75c0b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:24.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.811+0000 7fc75affd640 1 --2- 172.21.15.83:0/1302070610 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc75c0b0e30 0x7fc75c0b12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.811+0000 7fc75affd640 1 -- 172.21.15.83:0/1302070610 >> v1:172.21.15.183:6789/0 conn(0x7fc75c1614d0 legacy=0x7fc75c0d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.811+0000 7fc75affd640 1 -- 172.21.15.83:0/1302070610 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc75c0cf6a0 con 0x7fc75c0d4860 2024-03-06T07:29:24.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.812+0000 7fc75affd640 1 --2- 172.21.15.83:0/1302070610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc75c0d4860 0x7fc75c0d2020 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7fc750009870 tx=0x7fc75002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3233e9d113dd972 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:24.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.812+0000 7fc75a7fc640 1 -- 172.21.15.83:0/1302070610 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc750034070 con 0x7fc75c0d4860 2024-03-06T07:29:24.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.812+0000 7fc75a7fc640 1 -- 172.21.15.83:0/1302070610 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc750030840 con 0x7fc75c0d4860 2024-03-06T07:29:24.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.812+0000 7fc75a7fc640 1 -- 172.21.15.83:0/1302070610 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc750030c10 con 0x7fc75c0d4860 2024-03-06T07:29:24.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.813+0000 7fc761c6b640 1 -- 172.21.15.83:0/1302070610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc75c0d4860 msgr2=0x7fc75c0d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.813+0000 7fc761c6b640 1 --2- 172.21.15.83:0/1302070610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc75c0d4860 0x7fc75c0d2020 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7fc750009870 tx=0x7fc75002b980 comp rx=0 tx=0).stop 2024-03-06T07:29:24.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.813+0000 7fc761c6b640 1 -- 172.21.15.83:0/1302070610 shutdown_connections 2024-03-06T07:29:24.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.813+0000 7fc761c6b640 1 --2- 172.21.15.83:0/1302070610 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc75c0b0e30 0x7fc75c0b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.813+0000 7fc761c6b640 1 --2- 172.21.15.83:0/1302070610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc75c0d4860 0x7fc75c0d2020 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.813+0000 7fc761c6b640 1 -- 172.21.15.83:0/1302070610 >> 172.21.15.83:0/1302070610 conn(0x7fc75c0d3d40 msgr2=0x7fc75c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:24.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.813+0000 7fc761c6b640 1 -- 172.21.15.83:0/1302070610 shutdown_connections 2024-03-06T07:29:24.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.813+0000 7fc761c6b640 1 -- 172.21.15.83:0/1302070610 wait complete. 2024-03-06T07:29:24.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.815+0000 7fc761c6b640 1 Processor -- start 2024-03-06T07:29:24.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.815+0000 7fc761c6b640 1 -- start start 2024-03-06T07:29:24.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.816+0000 7fc761c6b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc75c0b0e30 0x7fc75c1572a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.816+0000 7fc761c6b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc75c0d4860 0x7fc75c0b6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.816+0000 7fc761c6b640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc75c1614d0 0x7fc75c0b6e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.816+0000 7fc761c6b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc75c0b0910 con 0x7fc75c0d4860 2024-03-06T07:29:24.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.816+0000 7fc761c6b640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc75c0cfc60 con 0x7fc75c1614d0 2024-03-06T07:29:24.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.816+0000 7fc761c6b640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc75c0d2560 con 0x7fc75c0b0e30 2024-03-06T07:29:24.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.816+0000 7fc75affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc75c0d4860 0x7fc75c0b6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.816+0000 7fc75affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc75c0d4860 0x7fc75c0b6930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57174/0 (socket says 172.21.15.83:57174) 2024-03-06T07:29:24.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.816+0000 7fc75affd640 1 -- 172.21.15.83:0/2169759212 learned_addr learned my addr 172.21.15.83:0/2169759212 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:24.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.816+0000 7fc75affd640 1 -- 172.21.15.83:0/2169759212 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc75c0b0e30 msgr2=0x7fc75c1572a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.816+0000 7fc75bfff640 1 --2- 172.21.15.83:0/2169759212 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc75c1614d0 0x7fc75c0b6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.816+0000 7fc75b7fe640 1 --2- 172.21.15.83:0/2169759212 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc75c0b0e30 0x7fc75c1572a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.817+0000 7fc75affd640 1 --2- 172.21.15.83:0/2169759212 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc75c0b0e30 0x7fc75c1572a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.817+0000 7fc75affd640 1 -- 172.21.15.83:0/2169759212 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc75c1614d0 msgr2=0x7fc75c0b6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.817+0000 7fc75affd640 1 --2- 172.21.15.83:0/2169759212 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc75c1614d0 0x7fc75c0b6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.817+0000 7fc75affd640 1 -- 172.21.15.83:0/2169759212 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc75c0b75d0 con 0x7fc75c0d4860 2024-03-06T07:29:24.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.817+0000 7fc75bfff640 1 --2- 172.21.15.83:0/2169759212 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc75c1614d0 0x7fc75c0b6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:24.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.817+0000 7fc75b7fe640 1 --2- 172.21.15.83:0/2169759212 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc75c0b0e30 0x7fc75c1572a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:24.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.817+0000 7fc75affd640 1 --2- 172.21.15.83:0/2169759212 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc75c0d4860 0x7fc75c0b6930 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7fc75002b9c0 tx=0x7fc750004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:24.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.817+0000 7fc743fff640 1 -- 172.21.15.83:0/2169759212 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc750048070 con 0x7fc75c0d4860 2024-03-06T07:29:24.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.817+0000 7fc743fff640 1 -- 172.21.15.83:0/2169759212 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc750030640 con 0x7fc75c0d4860 2024-03-06T07:29:24.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.817+0000 7fc761c6b640 1 -- 172.21.15.83:0/2169759212 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc75c0b78b0 con 0x7fc75c0d4860 2024-03-06T07:29:24.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.817+0000 7fc743fff640 1 -- 172.21.15.83:0/2169759212 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc750034040 con 0x7fc75c0d4860 2024-03-06T07:29:24.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.817+0000 7fc761c6b640 1 -- 172.21.15.83:0/2169759212 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc75c0b7e70 con 0x7fc75c0d4860 2024-03-06T07:29:24.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.818+0000 7fc761c6b640 1 -- 172.21.15.83:0/2169759212 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc75c068ff0 con 0x7fc75c0d4860 2024-03-06T07:29:24.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.818+0000 7fc743fff640 1 -- 172.21.15.83:0/2169759212 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc75004b070 con 0x7fc75c0d4860 2024-03-06T07:29:24.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.819+0000 7fc743fff640 1 --2- 172.21.15.83:0/2169759212 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc74806b4d0 0x7fc74806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.819+0000 7fc743fff640 1 -- 172.21.15.83:0/2169759212 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc7500ac320 con 0x7fc75c0d4860 2024-03-06T07:29:24.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.819+0000 7fc75b7fe640 1 --2- 172.21.15.83:0/2169759212 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc74806b4d0 0x7fc74806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.819+0000 7fc75b7fe640 1 --2- 172.21.15.83:0/2169759212 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc74806b4d0 0x7fc74806d990 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fc74c000f60 tx=0x7fc74c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:24.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:24.823+0000 7fc743fff640 1 -- 172.21.15.83:0/2169759212 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc75007b220 con 0x7fc75c0d4860 2024-03-06T07:29:24.967 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:25.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:25.109+0000 7fc761c6b640 1 -- 172.21.15.83:0/2169759212 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7fc75c0d1910 con 0x7fc75c0d4860 2024-03-06T07:29:26.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.070+0000 7fc743fff640 1 -- 172.21.15.83:0/2169759212 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v80) v1 ==== 105+0+0 (secure 0 0 0) 0x7fc75007ab20 con 0x7fc75c0d4860 2024-03-06T07:29:26.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.073+0000 7fc761c6b640 1 -- 172.21.15.83:0/2169759212 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc74806b4d0 msgr2=0x7fc74806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.073+0000 7fc761c6b640 1 --2- 172.21.15.83:0/2169759212 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc74806b4d0 0x7fc74806d990 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fc74c000f60 tx=0x7fc74c009040 comp rx=0 tx=0).stop 2024-03-06T07:29:26.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.073+0000 7fc761c6b640 1 -- 172.21.15.83:0/2169759212 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc75c0d4860 msgr2=0x7fc75c0b6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.073+0000 7fc761c6b640 1 --2- 172.21.15.83:0/2169759212 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc75c0d4860 0x7fc75c0b6930 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7fc75002b9c0 tx=0x7fc750004730 comp rx=0 tx=0).stop 2024-03-06T07:29:26.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.073+0000 7fc761c6b640 1 -- 172.21.15.83:0/2169759212 shutdown_connections 2024-03-06T07:29:26.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.073+0000 7fc761c6b640 1 --2- 172.21.15.83:0/2169759212 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc74806b4d0 0x7fc74806d990 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.073+0000 7fc761c6b640 1 --2- 172.21.15.83:0/2169759212 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc75c1614d0 0x7fc75c0b6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.073+0000 7fc761c6b640 1 --2- 172.21.15.83:0/2169759212 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc75c0d4860 0x7fc75c0b6930 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.073+0000 7fc761c6b640 1 --2- 172.21.15.83:0/2169759212 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc75c0b0e30 0x7fc75c1572a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.073+0000 7fc761c6b640 1 -- 172.21.15.83:0/2169759212 >> 172.21.15.83:0/2169759212 conn(0x7fc75c0d3d40 msgr2=0x7fc75c0b1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:26.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.073+0000 7fc761c6b640 1 -- 172.21.15.83:0/2169759212 shutdown_connections 2024-03-06T07:29:26.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.073+0000 7fc761c6b640 1 -- 172.21.15.83:0/2169759212 wait complete. 2024-03-06T07:29:26.086 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:26.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.186+0000 7fbf9e325640 1 Processor -- start 2024-03-06T07:29:26.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.186+0000 7fbf9e325640 1 -- start start 2024-03-06T07:29:26.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.187+0000 7fbf9e325640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbf9815dfe0 0x7fbf980cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.187+0000 7fbf9e325640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fbf980cf830 0x7fbf980d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.187+0000 7fbf9e325640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbf980d27f0 0x7fbf980d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.187+0000 7fbf9e325640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf980d2c50 con 0x7fbf980d27f0 2024-03-06T07:29:26.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.187+0000 7fbf9e325640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf980b31e0 con 0x7fbf980cf830 2024-03-06T07:29:26.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.187+0000 7fbf9e325640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf980b3d60 con 0x7fbf9815dfe0 2024-03-06T07:29:26.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.187+0000 7fbf9c8c2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbf980d27f0 0x7fbf980d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.187+0000 7fbf9c8c2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbf980d27f0 0x7fbf980d0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57178/0 (socket says 172.21.15.83:57178) 2024-03-06T07:29:26.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.187+0000 7fbf9c8c2640 1 -- 172.21.15.83:0/1776284001 learned_addr learned my addr 172.21.15.83:0/1776284001 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:26.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.187+0000 7fbf97fff640 1 --2- 172.21.15.83:0/1776284001 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbf9815dfe0 0x7fbf980cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.187+0000 7fbf9c8c2640 1 -- 172.21.15.83:0/1776284001 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbf9815dfe0 msgr2=0x7fbf980cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:26.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.187+0000 7fbf977fe640 1 --2- 172.21.15.83:0/1776284001 >> v2:172.21.15.183:3300/0 conn(0x7fbf980cf830 0x7fbf980d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.187+0000 7fbf9c8c2640 1 --2- 172.21.15.83:0/1776284001 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbf9815dfe0 0x7fbf980cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.187+0000 7fbf9c8c2640 1 -- 172.21.15.83:0/1776284001 >> v2:172.21.15.183:3300/0 conn(0x7fbf980cf830 msgr2=0x7fbf980d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:26.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.187+0000 7fbf9c8c2640 1 --2- 172.21.15.83:0/1776284001 >> v2:172.21.15.183:3300/0 conn(0x7fbf980cf830 0x7fbf980d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.187+0000 7fbf9c8c2640 1 -- 172.21.15.83:0/1776284001 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf980b3430 con 0x7fbf980d27f0 2024-03-06T07:29:26.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.187+0000 7fbf97fff640 1 --2- 172.21.15.83:0/1776284001 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbf9815dfe0 0x7fbf980cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:26.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.188+0000 7fbf977fe640 1 --2- 172.21.15.83:0/1776284001 >> v2:172.21.15.183:3300/0 conn(0x7fbf980cf830 0x7fbf980d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:26.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.188+0000 7fbf9c8c2640 1 --2- 172.21.15.83:0/1776284001 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf980d27f0 0x7fbf980d0300 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7fbf9808cba0 tx=0x7fbf9002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dadd169a667d0926 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:26.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.188+0000 7fbf96ffd640 1 -- 172.21.15.83:0/1776284001 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9002ca70 con 0x7fbf980d27f0 2024-03-06T07:29:26.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.188+0000 7fbf96ffd640 1 -- 172.21.15.83:0/1776284001 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf9002cc10 con 0x7fbf980d27f0 2024-03-06T07:29:26.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.188+0000 7fbf96ffd640 1 -- 172.21.15.83:0/1776284001 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf900343f0 con 0x7fbf980d27f0 2024-03-06T07:29:26.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.189+0000 7fbf9e325640 1 -- 172.21.15.83:0/1776284001 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf980d27f0 msgr2=0x7fbf980d0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.189+0000 7fbf9e325640 1 --2- 172.21.15.83:0/1776284001 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf980d27f0 0x7fbf980d0300 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7fbf9808cba0 tx=0x7fbf9002bed0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.189+0000 7fbf9e325640 1 -- 172.21.15.83:0/1776284001 shutdown_connections 2024-03-06T07:29:26.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.189+0000 7fbf9e325640 1 --2- 172.21.15.83:0/1776284001 >> v2:172.21.15.183:3300/0 conn(0x7fbf980cf830 0x7fbf980d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.189+0000 7fbf9e325640 1 --2- 172.21.15.83:0/1776284001 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf980d27f0 0x7fbf980d0300 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.189+0000 7fbf9e325640 1 --2- 172.21.15.83:0/1776284001 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbf9815dfe0 0x7fbf980cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.189+0000 7fbf9e325640 1 -- 172.21.15.83:0/1776284001 >> 172.21.15.83:0/1776284001 conn(0x7fbf980d3cd0 msgr2=0x7fbf980d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:26.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.189+0000 7fbf9e325640 1 -- 172.21.15.83:0/1776284001 shutdown_connections 2024-03-06T07:29:26.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.189+0000 7fbf9e325640 1 -- 172.21.15.83:0/1776284001 wait complete. 2024-03-06T07:29:26.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.190+0000 7fbf9e325640 1 Processor -- start 2024-03-06T07:29:26.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.190+0000 7fbf9e325640 1 -- start start 2024-03-06T07:29:26.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.191+0000 7fbf9e325640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbf980cf830 0x7fbf981567e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.191+0000 7fbf9e325640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbf980d27f0 0x7fbf980b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.191+0000 7fbf9e325640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf9815dfe0 0x7fbf980b83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.191+0000 7fbf9e325640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf980d1c70 con 0x7fbf9815dfe0 2024-03-06T07:29:26.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.191+0000 7fbf9e325640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf980b39f0 con 0x7fbf980cf830 2024-03-06T07:29:26.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.191+0000 7fbf9e325640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf980d1df0 con 0x7fbf980d27f0 2024-03-06T07:29:26.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.191+0000 7fbf9c8c2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf9815dfe0 0x7fbf980b83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.191+0000 7fbf9c8c2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf9815dfe0 0x7fbf980b83f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:57194/0 (socket says 172.21.15.83:57194) 2024-03-06T07:29:26.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.191+0000 7fbf9c8c2640 1 -- 172.21.15.83:0/1317402860 learned_addr learned my addr 172.21.15.83:0/1317402860 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:26.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.191+0000 7fbf97fff640 1 --2- 172.21.15.83:0/1317402860 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbf980cf830 0x7fbf981567e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.191+0000 7fbf977fe640 1 --2- 172.21.15.83:0/1317402860 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbf980d27f0 0x7fbf980b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.191+0000 7fbf9c8c2640 1 -- 172.21.15.83:0/1317402860 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbf980d27f0 msgr2=0x7fbf980b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.191+0000 7fbf9c8c2640 1 --2- 172.21.15.83:0/1317402860 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbf980d27f0 0x7fbf980b68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.191+0000 7fbf9c8c2640 1 -- 172.21.15.83:0/1317402860 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbf980cf830 msgr2=0x7fbf981567e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.191+0000 7fbf9c8c2640 1 --2- 172.21.15.83:0/1317402860 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbf980cf830 0x7fbf981567e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.191+0000 7fbf9c8c2640 1 -- 172.21.15.83:0/1317402860 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf980b90c0 con 0x7fbf9815dfe0 2024-03-06T07:29:26.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.192+0000 7fbf9c8c2640 1 --2- 172.21.15.83:0/1317402860 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf9815dfe0 0x7fbf980b83f0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7fbf90009f40 tx=0x7fbf90004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:26.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.192+0000 7fbf94ff9640 1 -- 172.21.15.83:0/1317402860 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf90043020 con 0x7fbf9815dfe0 2024-03-06T07:29:26.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.192+0000 7fbf94ff9640 1 -- 172.21.15.83:0/1317402860 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf9003e450 con 0x7fbf9815dfe0 2024-03-06T07:29:26.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.192+0000 7fbf9e325640 1 -- 172.21.15.83:0/1317402860 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf980b93a0 con 0x7fbf9815dfe0 2024-03-06T07:29:26.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.192+0000 7fbf94ff9640 1 -- 172.21.15.83:0/1317402860 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf90009510 con 0x7fbf9815dfe0 2024-03-06T07:29:26.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.192+0000 7fbf9e325640 1 -- 172.21.15.83:0/1317402860 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf980b9960 con 0x7fbf9815dfe0 2024-03-06T07:29:26.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.193+0000 7fbf9e325640 1 -- 172.21.15.83:0/1317402860 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf9804f330 con 0x7fbf9815dfe0 2024-03-06T07:29:26.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.193+0000 7fbf94ff9640 1 -- 172.21.15.83:0/1317402860 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbf90046020 con 0x7fbf9815dfe0 2024-03-06T07:29:26.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.194+0000 7fbf94ff9640 1 --2- 172.21.15.83:0/1317402860 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbf7806b4d0 0x7fbf7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.194+0000 7fbf97fff640 1 --2- 172.21.15.83:0/1317402860 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbf7806b4d0 0x7fbf7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.194+0000 7fbf94ff9640 1 -- 172.21.15.83:0/1317402860 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbf900acb80 con 0x7fbf9815dfe0 2024-03-06T07:29:26.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.194+0000 7fbf97fff640 1 --2- 172.21.15.83:0/1317402860 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbf7806b4d0 0x7fbf7806d990 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fbf8000a950 tx=0x7fbf80008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:26.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.198+0000 7fbf94ff9640 1 -- 172.21.15.83:0/1317402860 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbf9004d030 con 0x7fbf9815dfe0 2024-03-06T07:29:26.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.413+0000 7fbf9e325640 1 -- 172.21.15.83:0/1317402860 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf9815d4c0 con 0x7fbf9815dfe0 2024-03-06T07:29:26.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.414+0000 7fbf94ff9640 1 -- 172.21.15.83:0/1317402860 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+3415 (secure 0 0 0) 0x7fbf9007b970 con 0x7fbf9815dfe0 2024-03-06T07:29:26.416 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:29:26.416 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":81,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:26.073593+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:29:26.416 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 81 2024-03-06T07:29:26.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.416+0000 7fbf9e325640 1 -- 172.21.15.83:0/1317402860 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbf7806b4d0 msgr2=0x7fbf7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.416+0000 7fbf9e325640 1 --2- 172.21.15.83:0/1317402860 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbf7806b4d0 0x7fbf7806d990 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fbf8000a950 tx=0x7fbf80008040 comp rx=0 tx=0).stop 2024-03-06T07:29:26.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.416+0000 7fbf9e325640 1 -- 172.21.15.83:0/1317402860 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf9815dfe0 msgr2=0x7fbf980b83f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.416+0000 7fbf9e325640 1 --2- 172.21.15.83:0/1317402860 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf9815dfe0 0x7fbf980b83f0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7fbf90009f40 tx=0x7fbf90004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:26.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.416+0000 7fbf9e325640 1 -- 172.21.15.83:0/1317402860 shutdown_connections 2024-03-06T07:29:26.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.416+0000 7fbf9e325640 1 --2- 172.21.15.83:0/1317402860 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbf7806b4d0 0x7fbf7806d990 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.416+0000 7fbf9e325640 1 --2- 172.21.15.83:0/1317402860 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf9815dfe0 0x7fbf980b83f0 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.416+0000 7fbf9e325640 1 --2- 172.21.15.83:0/1317402860 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbf980d27f0 0x7fbf980b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.416+0000 7fbf9e325640 1 --2- 172.21.15.83:0/1317402860 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbf980cf830 0x7fbf981567e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.416+0000 7fbf9e325640 1 -- 172.21.15.83:0/1317402860 >> 172.21.15.83:0/1317402860 conn(0x7fbf980d3cd0 msgr2=0x7fbf9806ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:26.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.416+0000 7fbf9e325640 1 -- 172.21.15.83:0/1317402860 shutdown_connections 2024-03-06T07:29:26.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:26.416+0000 7fbf9e325640 1 -- 172.21.15.83:0/1317402860 wait complete. 2024-03-06T07:29:26.630 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:26.073593+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:29:26.630 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:29:26.969 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:27.631 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:27.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.730+0000 7f84b7e6b640 1 Processor -- start 2024-03-06T07:29:27.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.731+0000 7f84b7e6b640 1 -- start start 2024-03-06T07:29:27.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.731+0000 7f84b7e6b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f84b00d00c0 0x7f84b00d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.731+0000 7f84b7e6b640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f84b00cf850 0x7f84b00cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.731+0000 7f84b7e6b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f84b00ce1b0 0x7f84b00ce640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.731+0000 7f84b7e6b640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84b006ca90 con 0x7f84b00ce1b0 2024-03-06T07:29:27.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.731+0000 7f84b7e6b640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84b00d09e0 con 0x7f84b00cf850 2024-03-06T07:29:27.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.731+0000 7f84b7e6b640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84b015e380 con 0x7f84b00d00c0 2024-03-06T07:29:27.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.732+0000 7f84b6408640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f84b00ce1b0 0x7f84b00ce640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.732+0000 7f84b6408640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f84b00ce1b0 0x7f84b00ce640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35038/0 (socket says 172.21.15.83:35038) 2024-03-06T07:29:27.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.732+0000 7f84b6408640 1 -- 172.21.15.83:0/3662448061 learned_addr learned my addr 172.21.15.83:0/3662448061 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:27.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.732+0000 7f84b6408640 1 -- 172.21.15.83:0/3662448061 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f84b00d00c0 msgr2=0x7f84b00d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:27.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.732+0000 7f84b5406640 1 --2- 172.21.15.83:0/3662448061 >> v2:172.21.15.183:3300/0 conn(0x7f84b00cf850 0x7f84b00cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.732+0000 7f84b5c07640 1 --2- 172.21.15.83:0/3662448061 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f84b00d00c0 0x7f84b00d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.732+0000 7f84b6408640 1 --2- 172.21.15.83:0/3662448061 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f84b00d00c0 0x7f84b00d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.732+0000 7f84b6408640 1 -- 172.21.15.83:0/3662448061 >> v2:172.21.15.183:3300/0 conn(0x7f84b00cf850 msgr2=0x7f84b00cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:27.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.732+0000 7f84b6408640 1 --2- 172.21.15.83:0/3662448061 >> v2:172.21.15.183:3300/0 conn(0x7f84b00cf850 0x7f84b00cd3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.732+0000 7f84b6408640 1 -- 172.21.15.83:0/3662448061 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b00cdc10 con 0x7f84b00ce1b0 2024-03-06T07:29:27.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.732+0000 7f84b5406640 1 --2- 172.21.15.83:0/3662448061 >> v2:172.21.15.183:3300/0 conn(0x7f84b00cf850 0x7f84b00cd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:27.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.732+0000 7f84b5c07640 1 --2- 172.21.15.83:0/3662448061 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f84b00d00c0 0x7f84b00d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:27.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.733+0000 7f84b6408640 1 --2- 172.21.15.83:0/3662448061 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84b00ce1b0 0x7f84b00ce640 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f84b00722b0 tx=0x7f84a802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7793350be4b43c5e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:27.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.733+0000 7f84b4c05640 1 -- 172.21.15.83:0/3662448061 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a8035020 con 0x7f84b00ce1b0 2024-03-06T07:29:27.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.733+0000 7f84b4c05640 1 -- 172.21.15.83:0/3662448061 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84a8031870 con 0x7f84b00ce1b0 2024-03-06T07:29:27.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.733+0000 7f84b4c05640 1 -- 172.21.15.83:0/3662448061 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a8031c40 con 0x7f84b00ce1b0 2024-03-06T07:29:27.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.733+0000 7f84b7e6b640 1 -- 172.21.15.83:0/3662448061 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84b00ce1b0 msgr2=0x7f84b00ce640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.733+0000 7f84b7e6b640 1 --2- 172.21.15.83:0/3662448061 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84b00ce1b0 0x7f84b00ce640 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f84b00722b0 tx=0x7f84a802b910 comp rx=0 tx=0).stop 2024-03-06T07:29:27.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.733+0000 7f84b7e6b640 1 -- 172.21.15.83:0/3662448061 shutdown_connections 2024-03-06T07:29:27.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.733+0000 7f84b7e6b640 1 --2- 172.21.15.83:0/3662448061 >> v2:172.21.15.183:3300/0 conn(0x7f84b00cf850 0x7f84b00cd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.733+0000 7f84b7e6b640 1 --2- 172.21.15.83:0/3662448061 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84b00ce1b0 0x7f84b00ce640 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.733+0000 7f84b7e6b640 1 --2- 172.21.15.83:0/3662448061 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f84b00d00c0 0x7f84b00d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.733+0000 7f84b7e6b640 1 -- 172.21.15.83:0/3662448061 >> 172.21.15.83:0/3662448061 conn(0x7f84b00b35a0 msgr2=0x7f84b00b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:27.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.734+0000 7f84b7e6b640 1 -- 172.21.15.83:0/3662448061 shutdown_connections 2024-03-06T07:29:27.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.734+0000 7f84b7e6b640 1 -- 172.21.15.83:0/3662448061 wait complete. 2024-03-06T07:29:27.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.735+0000 7f84b7e6b640 1 Processor -- start 2024-03-06T07:29:27.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.735+0000 7f84b7e6b640 1 -- start start 2024-03-06T07:29:27.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.735+0000 7f84b7e6b640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84b00ce1b0 0x7f84b0157dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.735+0000 7f84b7e6b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84b00cf850 0x7f84b00b7100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.735+0000 7f84b7e6b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f84b00d00c0 0x7f84b00caca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.735+0000 7f84b7e6b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b006c720 con 0x7f84b00cf850 2024-03-06T07:29:27.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.735+0000 7f84b7e6b640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b006c440 con 0x7f84b00ce1b0 2024-03-06T07:29:27.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.735+0000 7f84b7e6b640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84b00cd900 con 0x7f84b00d00c0 2024-03-06T07:29:27.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.735+0000 7f84b5406640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84b00cf850 0x7f84b00b7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.736+0000 7f84b5406640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84b00cf850 0x7f84b00b7100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35050/0 (socket says 172.21.15.83:35050) 2024-03-06T07:29:27.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.736+0000 7f84b5406640 1 -- 172.21.15.83:0/2286628889 learned_addr learned my addr 172.21.15.83:0/2286628889 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:27.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.736+0000 7f84b5c07640 1 --2- 172.21.15.83:0/2286628889 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84b00ce1b0 0x7f84b0157dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.736+0000 7f84b6408640 1 --2- 172.21.15.83:0/2286628889 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f84b00d00c0 0x7f84b00caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.736+0000 7f84b5406640 1 -- 172.21.15.83:0/2286628889 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f84b00d00c0 msgr2=0x7f84b00caca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.736+0000 7f84b5406640 1 --2- 172.21.15.83:0/2286628889 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f84b00d00c0 0x7f84b00caca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.736+0000 7f84b5406640 1 -- 172.21.15.83:0/2286628889 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84b00ce1b0 msgr2=0x7f84b0157dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.736+0000 7f84b5406640 1 --2- 172.21.15.83:0/2286628889 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84b00ce1b0 0x7f84b0157dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.736+0000 7f84b5406640 1 -- 172.21.15.83:0/2286628889 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b00b6220 con 0x7f84b00cf850 2024-03-06T07:29:27.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.736+0000 7f84b5c07640 1 --2- 172.21.15.83:0/2286628889 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84b00ce1b0 0x7f84b0157dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:27.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.736+0000 7f84b5406640 1 --2- 172.21.15.83:0/2286628889 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84b00cf850 0x7f84b00b7100 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f84a400c7f0 tx=0x7f84a400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:27.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.736+0000 7f84ae7fc640 1 -- 172.21.15.83:0/2286628889 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a4015070 con 0x7f84b00cf850 2024-03-06T07:29:27.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.736+0000 7f84ae7fc640 1 -- 172.21.15.83:0/2286628889 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84a4007a30 con 0x7f84b00cf850 2024-03-06T07:29:27.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.736+0000 7f84b7e6b640 1 -- 172.21.15.83:0/2286628889 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84b00b6500 con 0x7f84b00cf850 2024-03-06T07:29:27.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.736+0000 7f84ae7fc640 1 -- 172.21.15.83:0/2286628889 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a4007e00 con 0x7f84b00cf850 2024-03-06T07:29:27.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.736+0000 7f84b7e6b640 1 -- 172.21.15.83:0/2286628889 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84b00b7640 con 0x7f84b00cf850 2024-03-06T07:29:27.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.737+0000 7f84b7e6b640 1 -- 172.21.15.83:0/2286628889 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84b004f330 con 0x7f84b00cf850 2024-03-06T07:29:27.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.738+0000 7f84ae7fc640 1 -- 172.21.15.83:0/2286628889 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f84a4018070 con 0x7f84b00cf850 2024-03-06T07:29:27.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.738+0000 7f84ae7fc640 1 --2- 172.21.15.83:0/2286628889 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f849c06b4d0 0x7f849c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.738+0000 7f84ae7fc640 1 -- 172.21.15.83:0/2286628889 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f84a408a230 con 0x7f84b00cf850 2024-03-06T07:29:27.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.738+0000 7f84b5c07640 1 --2- 172.21.15.83:0/2286628889 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f849c06b4d0 0x7f849c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.739+0000 7f84b5c07640 1 --2- 172.21.15.83:0/2286628889 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f849c06b4d0 0x7f849c06d990 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f84a0007130 tx=0x7f84a0007080 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:27.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.743+0000 7f84ae7fc640 1 -- 172.21.15.83:0/2286628889 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f84a4059030 con 0x7f84b00cf850 2024-03-06T07:29:27.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.958+0000 7f84b7e6b640 1 -- 172.21.15.83:0/2286628889 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84b006fba0 con 0x7f84b00cf850 2024-03-06T07:29:27.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.959+0000 7f84ae7fc640 1 -- 172.21.15.83:0/2286628889 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+3415 (secure 0 0 0) 0x7f84a4058930 con 0x7f84b00cf850 2024-03-06T07:29:27.961 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:29:27.961 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":81,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:26.073593+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:29:27.961 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 81 2024-03-06T07:29:27.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.961+0000 7f84b7e6b640 1 -- 172.21.15.83:0/2286628889 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f849c06b4d0 msgr2=0x7f849c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.961+0000 7f84b7e6b640 1 --2- 172.21.15.83:0/2286628889 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f849c06b4d0 0x7f849c06d990 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f84a0007130 tx=0x7f84a0007080 comp rx=0 tx=0).stop 2024-03-06T07:29:27.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.961+0000 7f84b7e6b640 1 -- 172.21.15.83:0/2286628889 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84b00cf850 msgr2=0x7f84b00b7100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.961+0000 7f84b7e6b640 1 --2- 172.21.15.83:0/2286628889 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84b00cf850 0x7f84b00b7100 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f84a400c7f0 tx=0x7f84a400cbf0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.961+0000 7f84b7e6b640 1 -- 172.21.15.83:0/2286628889 shutdown_connections 2024-03-06T07:29:27.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.961+0000 7f84b7e6b640 1 --2- 172.21.15.83:0/2286628889 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f849c06b4d0 0x7f849c06d990 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.961+0000 7f84b7e6b640 1 --2- 172.21.15.83:0/2286628889 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f84b00d00c0 0x7f84b00caca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.961+0000 7f84b7e6b640 1 --2- 172.21.15.83:0/2286628889 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84b00cf850 0x7f84b00b7100 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.961+0000 7f84b7e6b640 1 --2- 172.21.15.83:0/2286628889 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84b00ce1b0 0x7f84b0157dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.961+0000 7f84b7e6b640 1 -- 172.21.15.83:0/2286628889 >> 172.21.15.83:0/2286628889 conn(0x7f84b00b35a0 msgr2=0x7f84b00b0820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:27.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.961+0000 7f84b7e6b640 1 -- 172.21.15.83:0/2286628889 shutdown_connections 2024-03-06T07:29:27.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:27.961+0000 7f84b7e6b640 1 -- 172.21.15.83:0/2286628889 wait complete. 2024-03-06T07:29:28.178 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:26.073593+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:29:28.178 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:29:28.971 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:29.179 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:29.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.280+0000 7fc39680f640 1 Processor -- start 2024-03-06T07:29:29.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.280+0000 7fc39680f640 1 -- start start 2024-03-06T07:29:29.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.281+0000 7fc39680f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc39015dfe0 0x7fc3900cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.281+0000 7fc39680f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc3900d29c0 0x7fc3900b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.281+0000 7fc39680f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3900b3430 con 0x7fc3900cf830 2024-03-06T07:29:29.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.281+0000 7fc39680f640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3900b31e0 con 0x7fc3900d29c0 2024-03-06T07:29:29.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.281+0000 7fc39680f640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3900b3d60 con 0x7fc39015dfe0 2024-03-06T07:29:29.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.281+0000 7fc394dac640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc3900d29c0 0x7fc3900b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.281+0000 7fc394dac640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc3900d29c0 0x7fc3900b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35062/0 (socket says 172.21.15.83:35062) 2024-03-06T07:29:29.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.281+0000 7fc394dac640 1 -- 172.21.15.83:0/2411934406 learned_addr learned my addr 172.21.15.83:0/2411934406 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:29.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.281+0000 7fc38f7fe640 1 --1- 172.21.15.83:0/2411934406 >> v1:172.21.15.83:6789/0 conn(0x7fc3900cf830 0x7fc3900d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48918/0 (socket says 172.21.15.83:48918) 2024-03-06T07:29:29.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.281+0000 7fc394dac640 1 -- 172.21.15.83:0/2411934406 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc39015dfe0 msgr2=0x7fc3900cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:29.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.281+0000 7fc394dac640 1 --2- 172.21.15.83:0/2411934406 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc39015dfe0 0x7fc3900cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.281+0000 7fc394dac640 1 -- 172.21.15.83:0/2411934406 >> v1:172.21.15.83:6789/0 conn(0x7fc3900cf830 legacy=0x7fc3900d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.281+0000 7fc394dac640 1 -- 172.21.15.83:0/2411934406 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3900b3710 con 0x7fc3900d29c0 2024-03-06T07:29:29.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.282+0000 7fc394dac640 1 --2- 172.21.15.83:0/2411934406 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc3900d29c0 0x7fc3900b41f0 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7fc388009e50 tx=0x7fc38802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54493476d3036c74 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:29.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.282+0000 7fc38effd640 1 -- 172.21.15.83:0/2411934406 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc38802cb00 con 0x7fc3900d29c0 2024-03-06T07:29:29.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.282+0000 7fc38effd640 1 -- 172.21.15.83:0/2411934406 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc38802cca0 con 0x7fc3900d29c0 2024-03-06T07:29:29.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.282+0000 7fc38effd640 1 -- 172.21.15.83:0/2411934406 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc388034610 con 0x7fc3900d29c0 2024-03-06T07:29:29.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.283+0000 7fc39680f640 1 -- 172.21.15.83:0/2411934406 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc3900d29c0 msgr2=0x7fc3900b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.283+0000 7fc39680f640 1 --2- 172.21.15.83:0/2411934406 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc3900d29c0 0x7fc3900b41f0 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7fc388009e50 tx=0x7fc38802bf60 comp rx=0 tx=0).stop 2024-03-06T07:29:29.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.283+0000 7fc39680f640 1 -- 172.21.15.83:0/2411934406 shutdown_connections 2024-03-06T07:29:29.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.283+0000 7fc39680f640 1 --2- 172.21.15.83:0/2411934406 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc3900d29c0 0x7fc3900b41f0 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.283+0000 7fc39680f640 1 --2- 172.21.15.83:0/2411934406 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc39015dfe0 0x7fc3900cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.283+0000 7fc39680f640 1 -- 172.21.15.83:0/2411934406 >> 172.21.15.83:0/2411934406 conn(0x7fc3900d3cd0 msgr2=0x7fc3900d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:29.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.283+0000 7fc39680f640 1 -- 172.21.15.83:0/2411934406 shutdown_connections 2024-03-06T07:29:29.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.283+0000 7fc39680f640 1 -- 172.21.15.83:0/2411934406 wait complete. 2024-03-06T07:29:29.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.284+0000 7fc39680f640 1 Processor -- start 2024-03-06T07:29:29.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.284+0000 7fc39680f640 1 -- start start 2024-03-06T07:29:29.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.285+0000 7fc39680f640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc3900cf830 0x7fc39015c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.285+0000 7fc39680f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc3900d29c0 0x7fc3900b7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.285+0000 7fc39680f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc39015dfe0 0x7fc3901f4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.285+0000 7fc39680f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3900d1df0 con 0x7fc3900d29c0 2024-03-06T07:29:29.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.285+0000 7fc39680f640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3900d1c70 con 0x7fc3900cf830 2024-03-06T07:29:29.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.285+0000 7fc39680f640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc39015e3c0 con 0x7fc39015dfe0 2024-03-06T07:29:29.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.285+0000 7fc38f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc3900d29c0 0x7fc3900b7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.285+0000 7fc38f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc3900d29c0 0x7fc3900b7b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35076/0 (socket says 172.21.15.83:35076) 2024-03-06T07:29:29.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.285+0000 7fc38f7fe640 1 -- 172.21.15.83:0/1486273689 learned_addr learned my addr 172.21.15.83:0/1486273689 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:29.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.285+0000 7fc394dac640 1 --2- 172.21.15.83:0/1486273689 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc39015dfe0 0x7fc3901f4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.285+0000 7fc38ffff640 1 --2- 172.21.15.83:0/1486273689 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc3900cf830 0x7fc39015c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.285+0000 7fc38f7fe640 1 -- 172.21.15.83:0/1486273689 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc39015dfe0 msgr2=0x7fc3901f4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.285+0000 7fc38f7fe640 1 --2- 172.21.15.83:0/1486273689 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc39015dfe0 0x7fc3901f4fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.285+0000 7fc38f7fe640 1 -- 172.21.15.83:0/1486273689 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc3900cf830 msgr2=0x7fc39015c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.285+0000 7fc38f7fe640 1 --2- 172.21.15.83:0/1486273689 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc3900cf830 0x7fc39015c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.285+0000 7fc38f7fe640 1 -- 172.21.15.83:0/1486273689 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3900b9b60 con 0x7fc3900d29c0 2024-03-06T07:29:29.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.285+0000 7fc38f7fe640 1 --2- 172.21.15.83:0/1486273689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc3900d29c0 0x7fc3900b7b60 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fc38400c440 tx=0x7fc384001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:29.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.285+0000 7fc38cff9640 1 -- 172.21.15.83:0/1486273689 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3840074e0 con 0x7fc3900d29c0 2024-03-06T07:29:29.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.285+0000 7fc38cff9640 1 -- 172.21.15.83:0/1486273689 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc384007680 con 0x7fc3900d29c0 2024-03-06T07:29:29.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.285+0000 7fc38cff9640 1 -- 172.21.15.83:0/1486273689 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc384004a20 con 0x7fc3900d29c0 2024-03-06T07:29:29.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.285+0000 7fc39680f640 1 -- 172.21.15.83:0/1486273689 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3900b95a0 con 0x7fc3900d29c0 2024-03-06T07:29:29.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.286+0000 7fc39680f640 1 -- 172.21.15.83:0/1486273689 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3900b7380 con 0x7fc3900d29c0 2024-03-06T07:29:29.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.286+0000 7fc39680f640 1 -- 172.21.15.83:0/1486273689 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc39008d4b0 con 0x7fc3900d29c0 2024-03-06T07:29:29.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.287+0000 7fc38cff9640 1 -- 172.21.15.83:0/1486273689 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc38401a020 con 0x7fc3900d29c0 2024-03-06T07:29:29.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.287+0000 7fc38cff9640 1 --2- 172.21.15.83:0/1486273689 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc37c06b4d0 0x7fc37c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.287+0000 7fc38cff9640 1 -- 172.21.15.83:0/1486273689 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc384089a90 con 0x7fc3900d29c0 2024-03-06T07:29:29.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.287+0000 7fc38ffff640 1 --2- 172.21.15.83:0/1486273689 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc37c06b4d0 0x7fc37c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.288+0000 7fc38ffff640 1 --2- 172.21.15.83:0/1486273689 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc37c06b4d0 0x7fc37c06d990 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fc3900b9530 tx=0x7fc380008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:29.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.296+0000 7fc38cff9640 1 -- 172.21.15.83:0/1486273689 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc39008d4b0 con 0x7fc3900d29c0 2024-03-06T07:29:29.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.508+0000 7fc39680f640 1 -- 172.21.15.83:0/1486273689 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc39004f330 con 0x7fc3900d29c0 2024-03-06T07:29:29.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.509+0000 7fc38cff9640 1 -- 172.21.15.83:0/1486273689 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+3415 (secure 0 0 0) 0x7fc384058870 con 0x7fc3900d29c0 2024-03-06T07:29:29.511 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:29:29.511 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":81,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:26.073593+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:29:29.511 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 81 2024-03-06T07:29:29.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.511+0000 7fc39680f640 1 -- 172.21.15.83:0/1486273689 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc37c06b4d0 msgr2=0x7fc37c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.511+0000 7fc39680f640 1 --2- 172.21.15.83:0/1486273689 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc37c06b4d0 0x7fc37c06d990 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fc3900b9530 tx=0x7fc380008040 comp rx=0 tx=0).stop 2024-03-06T07:29:29.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.511+0000 7fc39680f640 1 -- 172.21.15.83:0/1486273689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc3900d29c0 msgr2=0x7fc3900b7b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.511+0000 7fc39680f640 1 --2- 172.21.15.83:0/1486273689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc3900d29c0 0x7fc3900b7b60 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fc38400c440 tx=0x7fc384001e70 comp rx=0 tx=0).stop 2024-03-06T07:29:29.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.511+0000 7fc39680f640 1 -- 172.21.15.83:0/1486273689 shutdown_connections 2024-03-06T07:29:29.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.511+0000 7fc39680f640 1 --2- 172.21.15.83:0/1486273689 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc37c06b4d0 0x7fc37c06d990 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.511+0000 7fc39680f640 1 --2- 172.21.15.83:0/1486273689 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc39015dfe0 0x7fc3901f4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.511+0000 7fc39680f640 1 --2- 172.21.15.83:0/1486273689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc3900d29c0 0x7fc3900b7b60 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.511+0000 7fc39680f640 1 --2- 172.21.15.83:0/1486273689 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc3900cf830 0x7fc39015c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.511+0000 7fc39680f640 1 -- 172.21.15.83:0/1486273689 >> 172.21.15.83:0/1486273689 conn(0x7fc3900d3cd0 msgr2=0x7fc39006c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:29.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.511+0000 7fc39680f640 1 -- 172.21.15.83:0/1486273689 shutdown_connections 2024-03-06T07:29:29.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:29.511+0000 7fc39680f640 1 -- 172.21.15.83:0/1486273689 wait complete. 2024-03-06T07:29:29.726 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:26.073593+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:29:29.726 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:29:30.322 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:30.324 DEBUG:teuthology.orchestra.run.smithi183:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:30.726 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:30.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.825+0000 7fa99c3eb640 1 Processor -- start 2024-03-06T07:29:30.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.825+0000 7fa99c3eb640 1 -- start start 2024-03-06T07:29:30.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.826+0000 7fa99c3eb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa9940d22b0 0x7fa9940d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.826+0000 7fa99c3eb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa9940b41f0 0x7fa9940b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.826+0000 7fa99c3eb640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9940b3430 con 0x7fa99415dfe0 2024-03-06T07:29:30.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.826+0000 7fa99c3eb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9940d2c50 con 0x7fa9940d22b0 2024-03-06T07:29:30.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.826+0000 7fa99c3eb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9940b31e0 con 0x7fa9940b41f0 2024-03-06T07:29:30.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.826+0000 7fa999986640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa9940d22b0 0x7fa9940d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.827+0000 7fa999986640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa9940d22b0 0x7fa9940d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35086/0 (socket says 172.21.15.83:35086) 2024-03-06T07:29:30.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.827+0000 7fa999986640 1 -- 172.21.15.83:0/1300143576 learned_addr learned my addr 172.21.15.83:0/1300143576 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:30.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.827+0000 7fa99a988640 1 --2- 172.21.15.83:0/1300143576 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa9940b41f0 0x7fa9940b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.827+0000 7fa999986640 1 -- 172.21.15.83:0/1300143576 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa9940b41f0 msgr2=0x7fa9940b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:30.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.827+0000 7fa999986640 1 --2- 172.21.15.83:0/1300143576 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa9940b41f0 0x7fa9940b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.827+0000 7fa999986640 1 -- 172.21.15.83:0/1300143576 >> v1:172.21.15.183:6789/0 conn(0x7fa99415dfe0 legacy=0x7fa9940cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:30.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.827+0000 7fa999986640 1 -- 172.21.15.83:0/1300143576 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9940b3710 con 0x7fa9940d22b0 2024-03-06T07:29:30.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.827+0000 7fa999986640 1 --2- 172.21.15.83:0/1300143576 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9940d22b0 0x7fa9940d2710 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7fa990009870 tx=0x7fa99002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1c998580d4023e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:30.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.828+0000 7fa999185640 1 -- 172.21.15.83:0/1300143576 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa990035020 con 0x7fa9940d22b0 2024-03-06T07:29:30.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.828+0000 7fa999185640 1 -- 172.21.15.83:0/1300143576 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa990031640 con 0x7fa9940d22b0 2024-03-06T07:29:30.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.828+0000 7fa999185640 1 -- 172.21.15.83:0/1300143576 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99002d040 con 0x7fa9940d22b0 2024-03-06T07:29:30.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.828+0000 7fa99c3eb640 1 -- 172.21.15.83:0/1300143576 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9940d22b0 msgr2=0x7fa9940d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.828+0000 7fa99c3eb640 1 --2- 172.21.15.83:0/1300143576 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9940d22b0 0x7fa9940d2710 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7fa990009870 tx=0x7fa99002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.828+0000 7fa99c3eb640 1 -- 172.21.15.83:0/1300143576 shutdown_connections 2024-03-06T07:29:30.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.828+0000 7fa99c3eb640 1 --2- 172.21.15.83:0/1300143576 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa9940b41f0 0x7fa9940b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.828+0000 7fa99c3eb640 1 --2- 172.21.15.83:0/1300143576 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9940d22b0 0x7fa9940d2710 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.828+0000 7fa99c3eb640 1 -- 172.21.15.83:0/1300143576 >> 172.21.15.83:0/1300143576 conn(0x7fa9940d3cd0 msgr2=0x7fa9940d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:30.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.828+0000 7fa99c3eb640 1 -- 172.21.15.83:0/1300143576 shutdown_connections 2024-03-06T07:29:30.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.828+0000 7fa99c3eb640 1 -- 172.21.15.83:0/1300143576 wait complete. 2024-03-06T07:29:30.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.829+0000 7fa99c3eb640 1 Processor -- start 2024-03-06T07:29:30.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.829+0000 7fa99c3eb640 1 -- start start 2024-03-06T07:29:30.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.830+0000 7fa99c3eb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9940b41f0 0x7fa99415ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.830+0000 7fa99c3eb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa9940d22b0 0x7fa994147ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.830+0000 7fa99c3eb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa99415dfe0 0x7fa99414a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.830+0000 7fa99c3eb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9940cfa00 con 0x7fa9940b41f0 2024-03-06T07:29:30.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.830+0000 7fa99c3eb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9940b3d60 con 0x7fa9940d22b0 2024-03-06T07:29:30.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.830+0000 7fa99c3eb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9940d1c70 con 0x7fa99415dfe0 2024-03-06T07:29:30.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.830+0000 7fa99a187640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9940b41f0 0x7fa99415ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.830+0000 7fa99a187640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9940b41f0 0x7fa99415ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35090/0 (socket says 172.21.15.83:35090) 2024-03-06T07:29:30.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.830+0000 7fa99a187640 1 -- 172.21.15.83:0/808456099 learned_addr learned my addr 172.21.15.83:0/808456099 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:30.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.830+0000 7fa999986640 1 --2- 172.21.15.83:0/808456099 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa9940d22b0 0x7fa994147ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.830+0000 7fa99a988640 1 --2- 172.21.15.83:0/808456099 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa99415dfe0 0x7fa99414a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.831+0000 7fa99a187640 1 -- 172.21.15.83:0/808456099 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa99415dfe0 msgr2=0x7fa99414a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.831+0000 7fa99a187640 1 --2- 172.21.15.83:0/808456099 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa99415dfe0 0x7fa99414a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.831+0000 7fa99a187640 1 -- 172.21.15.83:0/808456099 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa9940d22b0 msgr2=0x7fa994147ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.831+0000 7fa99a187640 1 --2- 172.21.15.83:0/808456099 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa9940d22b0 0x7fa994147ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.831+0000 7fa99a187640 1 -- 172.21.15.83:0/808456099 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa994148430 con 0x7fa9940b41f0 2024-03-06T07:29:30.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.831+0000 7fa999986640 1 --2- 172.21.15.83:0/808456099 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa9940d22b0 0x7fa994147ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:30.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.831+0000 7fa99a187640 1 --2- 172.21.15.83:0/808456099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9940b41f0 0x7fa99415ead0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7fa98400b420 tx=0x7fa98400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:30.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.831+0000 7fa98affd640 1 -- 172.21.15.83:0/808456099 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa984011830 con 0x7fa9940b41f0 2024-03-06T07:29:30.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.831+0000 7fa98affd640 1 -- 172.21.15.83:0/808456099 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa984016070 con 0x7fa9940b41f0 2024-03-06T07:29:30.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.831+0000 7fa99c3eb640 1 -- 172.21.15.83:0/808456099 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9940b9b90 con 0x7fa9940b41f0 2024-03-06T07:29:30.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.831+0000 7fa98affd640 1 -- 172.21.15.83:0/808456099 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa984011c10 con 0x7fa9940b41f0 2024-03-06T07:29:30.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.831+0000 7fa99c3eb640 1 -- 172.21.15.83:0/808456099 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9940b6c20 con 0x7fa9940b41f0 2024-03-06T07:29:30.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.832+0000 7fa99c3eb640 1 -- 172.21.15.83:0/808456099 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa99404f330 con 0x7fa9940b41f0 2024-03-06T07:29:30.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.832+0000 7fa98affd640 1 -- 172.21.15.83:0/808456099 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa98401b460 con 0x7fa9940b41f0 2024-03-06T07:29:30.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.833+0000 7fa98affd640 1 --2- 172.21.15.83:0/808456099 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa97806b4d0 0x7fa97806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.833+0000 7fa999986640 1 --2- 172.21.15.83:0/808456099 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa97806b4d0 0x7fa97806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.833+0000 7fa98affd640 1 -- 172.21.15.83:0/808456099 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa984089cf0 con 0x7fa9940b41f0 2024-03-06T07:29:30.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.833+0000 7fa999986640 1 --2- 172.21.15.83:0/808456099 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa97806b4d0 0x7fa97806d990 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fa990036000 tx=0x7fa99003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:30.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:30.837+0000 7fa98affd640 1 -- 172.21.15.83:0/808456099 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa98408e070 con 0x7fa9940b41f0 2024-03-06T07:29:30.973 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:31.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:31.046+0000 7fa99c3eb640 1 -- 172.21.15.83:0/808456099 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9940cea80 con 0x7fa9940b41f0 2024-03-06T07:29:31.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:31.047+0000 7fa98affd640 1 -- 172.21.15.83:0/808456099 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+3415 (secure 0 0 0) 0x7fa984020020 con 0x7fa9940b41f0 2024-03-06T07:29:31.049 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:29:31.049 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":81,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:26.073593+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:29:31.049 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 81 2024-03-06T07:29:31.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:31.049+0000 7fa99c3eb640 1 -- 172.21.15.83:0/808456099 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa97806b4d0 msgr2=0x7fa97806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:31.049+0000 7fa99c3eb640 1 --2- 172.21.15.83:0/808456099 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa97806b4d0 0x7fa97806d990 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fa990036000 tx=0x7fa99003c040 comp rx=0 tx=0).stop 2024-03-06T07:29:31.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:31.049+0000 7fa99c3eb640 1 -- 172.21.15.83:0/808456099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9940b41f0 msgr2=0x7fa99415ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:31.049+0000 7fa99c3eb640 1 --2- 172.21.15.83:0/808456099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9940b41f0 0x7fa99415ead0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7fa98400b420 tx=0x7fa98400b8e0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:31.049+0000 7fa99c3eb640 1 -- 172.21.15.83:0/808456099 shutdown_connections 2024-03-06T07:29:31.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:31.049+0000 7fa99c3eb640 1 --2- 172.21.15.83:0/808456099 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa97806b4d0 0x7fa97806d990 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:31.049+0000 7fa99c3eb640 1 --2- 172.21.15.83:0/808456099 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa99415dfe0 0x7fa99414a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:31.049+0000 7fa99c3eb640 1 --2- 172.21.15.83:0/808456099 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa9940d22b0 0x7fa994147ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:31.049+0000 7fa99c3eb640 1 --2- 172.21.15.83:0/808456099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9940b41f0 0x7fa99415ead0 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:31.049+0000 7fa99c3eb640 1 -- 172.21.15.83:0/808456099 >> 172.21.15.83:0/808456099 conn(0x7fa9940d3cd0 msgr2=0x7fa9940b4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:31.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:31.049+0000 7fa99c3eb640 1 -- 172.21.15.83:0/808456099 shutdown_connections 2024-03-06T07:29:31.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:31.049+0000 7fa99c3eb640 1 -- 172.21.15.83:0/808456099 wait complete. 2024-03-06T07:29:31.263 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:26.073593+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:29:31.263 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:29:32.264 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:32.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.364+0000 7fdecd781640 1 Processor -- start 2024-03-06T07:29:32.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.364+0000 7fdecd781640 1 -- start start 2024-03-06T07:29:32.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.365+0000 7fdecd781640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fdec80b4170 0x7fdec80b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.365+0000 7fdecd781640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdec80d0a80 0x7fdec80cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.365+0000 7fdecd781640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdec80d25d0 con 0x7fdec80d4760 2024-03-06T07:29:32.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.365+0000 7fdecd781640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdec80d4bf0 con 0x7fdec80b4170 2024-03-06T07:29:32.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.365+0000 7fdecd781640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdec80ce9e0 con 0x7fdec80d0a80 2024-03-06T07:29:32.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.365+0000 7fdebffff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fdec80d4760 0x7fdec80d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48922/0 (socket says 172.21.15.83:48922) 2024-03-06T07:29:32.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.365+0000 7fdebffff640 1 -- 172.21.15.83:0/2033126708 learned_addr learned my addr 172.21.15.83:0/2033126708 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:29:32.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.365+0000 7fdec77fe640 1 --2- 172.21.15.83:0/2033126708 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdec80d0a80 0x7fdec80cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.365+0000 7fdec67fc640 1 -- 172.21.15.83:0/2033126708 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1821843296 0 0) 0x7fdec80d25d0 con 0x7fdec80d4760 2024-03-06T07:29:32.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.365+0000 7fdec6ffd640 1 --2- 172.21.15.83:0/2033126708 >> v2:172.21.15.183:3300/0 conn(0x7fdec80b4170 0x7fdec80b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.365+0000 7fdec67fc640 1 -- 172.21.15.83:0/2033126708 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdeb4003520 con 0x7fdec80d4760 2024-03-06T07:29:32.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.365+0000 7fdec67fc640 1 -- 172.21.15.83:0/2033126708 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 458424172 0 0) 0x7fdeb4003520 con 0x7fdec80d4760 2024-03-06T07:29:32.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.365+0000 7fdec67fc640 1 -- 172.21.15.83:0/2033126708 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdec80d0a80 msgr2=0x7fdec80cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:32.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.365+0000 7fdec67fc640 1 --2- 172.21.15.83:0/2033126708 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdec80d0a80 0x7fdec80cee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.365+0000 7fdec67fc640 1 -- 172.21.15.83:0/2033126708 >> v2:172.21.15.183:3300/0 conn(0x7fdec80b4170 msgr2=0x7fdec80b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:32.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.365+0000 7fdec67fc640 1 --2- 172.21.15.83:0/2033126708 >> v2:172.21.15.183:3300/0 conn(0x7fdec80b4170 0x7fdec80b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.365+0000 7fdec67fc640 1 -- 172.21.15.83:0/2033126708 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec80d2870 con 0x7fdec80d4760 2024-03-06T07:29:32.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.365+0000 7fdec67fc640 1 -- 172.21.15.83:0/2033126708 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7fdeb0001690 con 0x7fdec80d4760 2024-03-06T07:29:32.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.365+0000 7fdec67fc640 1 -- 172.21.15.83:0/2033126708 >> v1:172.21.15.83:6789/0 conn(0x7fdec80d4760 legacy=0x7fdec80d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.365+0000 7fdec67fc640 1 --2- 172.21.15.83:0/2033126708 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdec80d0a80 0x7fdeb40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.366+0000 7fdec67fc640 1 --2- 172.21.15.83:0/2033126708 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdeb4003f30 0x7fdeb4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.366+0000 7fdec67fc640 1 --2- 172.21.15.83:0/2033126708 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdeb4004d40 0x7fdeb40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.366+0000 7fdec67fc640 1 -- 172.21.15.83:0/2033126708 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec80d4bf0 con 0x7fdeb4004d40 2024-03-06T07:29:32.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.366+0000 7fdec67fc640 1 -- 172.21.15.83:0/2033126708 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec80ce9e0 con 0x7fdeb4003f30 2024-03-06T07:29:32.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.366+0000 7fdec67fc640 1 -- 172.21.15.83:0/2033126708 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb400b9c0 con 0x7fdec80d0a80 2024-03-06T07:29:32.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.366+0000 7fdec77fe640 1 --2- 172.21.15.83:0/2033126708 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdeb4004d40 0x7fdeb40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.366+0000 7fdec6ffd640 1 --2- 172.21.15.83:0/2033126708 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdec80d0a80 0x7fdeb40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.366+0000 7fdebffff640 1 --2- 172.21.15.83:0/2033126708 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdeb4003f30 0x7fdeb4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.366+0000 7fdec77fe640 1 -- 172.21.15.83:0/2033126708 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdec80d0a80 msgr2=0x7fdeb40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:32.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.366+0000 7fdec77fe640 1 --2- 172.21.15.83:0/2033126708 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdec80d0a80 0x7fdeb40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.366+0000 7fdec77fe640 1 -- 172.21.15.83:0/2033126708 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdeb4003f30 msgr2=0x7fdeb4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:32.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.366+0000 7fdec77fe640 1 --2- 172.21.15.83:0/2033126708 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdeb4003f30 0x7fdeb4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.366+0000 7fdec77fe640 1 -- 172.21.15.83:0/2033126708 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb4003520 con 0x7fdeb4004d40 2024-03-06T07:29:32.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.367+0000 7fdec77fe640 1 --2- 172.21.15.83:0/2033126708 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdeb4004d40 0x7fdeb40051c0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7fdec0009f90 tx=0x7fdeb4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7a45af2d00fbede server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:32.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.367+0000 7fdec67fc640 1 -- 172.21.15.83:0/2033126708 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec0039070 con 0x7fdeb4004d40 2024-03-06T07:29:32.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.367+0000 7fdec67fc640 1 -- 172.21.15.83:0/2033126708 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdec002cbc0 con 0x7fdeb4004d40 2024-03-06T07:29:32.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.367+0000 7fdec67fc640 1 -- 172.21.15.83:0/2033126708 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec0009c10 con 0x7fdeb4004d40 2024-03-06T07:29:32.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.367+0000 7fdecd781640 1 -- 172.21.15.83:0/2033126708 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdeb4004d40 msgr2=0x7fdeb40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.367+0000 7fdecd781640 1 --2- 172.21.15.83:0/2033126708 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdeb4004d40 0x7fdeb40051c0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7fdec0009f90 tx=0x7fdeb4005830 comp rx=0 tx=0).stop 2024-03-06T07:29:32.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.367+0000 7fdec77fe640 1 -- 172.21.15.83:0/2033126708 reap_dead start 2024-03-06T07:29:32.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.367+0000 7fdecd781640 1 -- 172.21.15.83:0/2033126708 shutdown_connections 2024-03-06T07:29:32.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.367+0000 7fdecd781640 1 -- 172.21.15.83:0/2033126708 >> 172.21.15.83:0/2033126708 conn(0x7fdec80d3c40 msgr2=0x7fdec80d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:32.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.367+0000 7fdecd781640 1 -- 172.21.15.83:0/2033126708 shutdown_connections 2024-03-06T07:29:32.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.367+0000 7fdecd781640 1 -- 172.21.15.83:0/2033126708 wait complete. 2024-03-06T07:29:32.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.368+0000 7fdecd781640 1 Processor -- start 2024-03-06T07:29:32.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.369+0000 7fdecd781640 1 -- start start 2024-03-06T07:29:32.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.369+0000 7fdecd781640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdec8155fd0 0x7fdec81f4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.369+0000 7fdecd781640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdec81563b0 0x7fdec81600e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.369+0000 7fdecd781640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdec8160620 0x7fdec80ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.369+0000 7fdecd781640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec80b4dd0 con 0x7fdec81563b0 2024-03-06T07:29:32.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.369+0000 7fdecd781640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec80b4550 con 0x7fdec8155fd0 2024-03-06T07:29:32.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.369+0000 7fdecd781640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdec80b4f50 con 0x7fdec8160620 2024-03-06T07:29:32.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.369+0000 7fdebffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdec81563b0 0x7fdec81600e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.369+0000 7fdebffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdec81563b0 0x7fdec81600e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35114/0 (socket says 172.21.15.83:35114) 2024-03-06T07:29:32.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.369+0000 7fdebffff640 1 -- 172.21.15.83:0/3614628867 learned_addr learned my addr 172.21.15.83:0/3614628867 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:32.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.370+0000 7fdec77fe640 1 --2- 172.21.15.83:0/3614628867 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdec8160620 0x7fdec80ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.370+0000 7fdec6ffd640 1 --2- 172.21.15.83:0/3614628867 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdec8155fd0 0x7fdec81f4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.370+0000 7fdebffff640 1 -- 172.21.15.83:0/3614628867 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdec8160620 msgr2=0x7fdec80ccca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.370+0000 7fdebffff640 1 --2- 172.21.15.83:0/3614628867 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdec8160620 0x7fdec80ccca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.370+0000 7fdebffff640 1 -- 172.21.15.83:0/3614628867 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdec8155fd0 msgr2=0x7fdec81f4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.370+0000 7fdebffff640 1 --2- 172.21.15.83:0/3614628867 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdec8155fd0 0x7fdec81f4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.370+0000 7fdebffff640 1 -- 172.21.15.83:0/3614628867 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec80cd9d0 con 0x7fdec81563b0 2024-03-06T07:29:32.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.370+0000 7fdec77fe640 1 --2- 172.21.15.83:0/3614628867 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdec8160620 0x7fdec80ccca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:32.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.370+0000 7fdebffff640 1 --2- 172.21.15.83:0/3614628867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdec81563b0 0x7fdec81600e0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7fdeb0001b00 tx=0x7fdeb0007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:32.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.370+0000 7fdebf7fe640 1 -- 172.21.15.83:0/3614628867 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb000f910 con 0x7fdec81563b0 2024-03-06T07:29:32.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.370+0000 7fdebf7fe640 1 -- 172.21.15.83:0/3614628867 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeb0018070 con 0x7fdec81563b0 2024-03-06T07:29:32.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.370+0000 7fdebf7fe640 1 -- 172.21.15.83:0/3614628867 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb000fd30 con 0x7fdec81563b0 2024-03-06T07:29:32.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.370+0000 7fdecd781640 1 -- 172.21.15.83:0/3614628867 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdec81f3f20 con 0x7fdec81563b0 2024-03-06T07:29:32.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.370+0000 7fdecd781640 1 -- 172.21.15.83:0/3614628867 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdec81f44e0 con 0x7fdec81563b0 2024-03-06T07:29:32.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.371+0000 7fdecd781640 1 -- 172.21.15.83:0/3614628867 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdec8160a30 con 0x7fdec81563b0 2024-03-06T07:29:32.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.371+0000 7fdebf7fe640 1 -- 172.21.15.83:0/3614628867 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdeb001d460 con 0x7fdec81563b0 2024-03-06T07:29:32.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.372+0000 7fdebf7fe640 1 --2- 172.21.15.83:0/3614628867 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdea806b500 0x7fdea806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.372+0000 7fdec6ffd640 1 --2- 172.21.15.83:0/3614628867 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdea806b500 0x7fdea806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.372+0000 7fdebf7fe640 1 -- 172.21.15.83:0/3614628867 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdeb008a430 con 0x7fdec81563b0 2024-03-06T07:29:32.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.372+0000 7fdec6ffd640 1 --2- 172.21.15.83:0/3614628867 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdea806b500 0x7fdea806d9c0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8009a60 tx=0x7fdeb8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:32.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.376+0000 7fdebf7fe640 1 -- 172.21.15.83:0/3614628867 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdec8160a30 con 0x7fdec81563b0 2024-03-06T07:29:32.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.586+0000 7fdecd781640 1 -- 172.21.15.83:0/3614628867 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdec81f3d10 con 0x7fdec81563b0 2024-03-06T07:29:32.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.587+0000 7fdebf7fe640 1 -- 172.21.15.83:0/3614628867 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+3415 (secure 0 0 0) 0x7fdeb00592b0 con 0x7fdec81563b0 2024-03-06T07:29:32.589 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:29:32.589 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":81,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:26.073593+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:29:32.589 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 81 2024-03-06T07:29:32.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.589+0000 7fdecd781640 1 -- 172.21.15.83:0/3614628867 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdea806b500 msgr2=0x7fdea806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.589+0000 7fdecd781640 1 --2- 172.21.15.83:0/3614628867 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdea806b500 0x7fdea806d9c0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8009a60 tx=0x7fdeb8008040 comp rx=0 tx=0).stop 2024-03-06T07:29:32.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.589+0000 7fdecd781640 1 -- 172.21.15.83:0/3614628867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdec81563b0 msgr2=0x7fdec81600e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.589+0000 7fdecd781640 1 --2- 172.21.15.83:0/3614628867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdec81563b0 0x7fdec81600e0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7fdeb0001b00 tx=0x7fdeb0007720 comp rx=0 tx=0).stop 2024-03-06T07:29:32.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.589+0000 7fdecd781640 1 -- 172.21.15.83:0/3614628867 shutdown_connections 2024-03-06T07:29:32.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.589+0000 7fdecd781640 1 --2- 172.21.15.83:0/3614628867 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdea806b500 0x7fdea806d9c0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.589+0000 7fdecd781640 1 --2- 172.21.15.83:0/3614628867 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdec8160620 0x7fdec80ccca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.589+0000 7fdecd781640 1 --2- 172.21.15.83:0/3614628867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdec81563b0 0x7fdec81600e0 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.589+0000 7fdecd781640 1 --2- 172.21.15.83:0/3614628867 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdec8155fd0 0x7fdec81f4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.589+0000 7fdecd781640 1 -- 172.21.15.83:0/3614628867 >> 172.21.15.83:0/3614628867 conn(0x7fdec80d3c40 msgr2=0x7fdec806cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:32.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.589+0000 7fdecd781640 1 -- 172.21.15.83:0/3614628867 shutdown_connections 2024-03-06T07:29:32.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:32.589+0000 7fdecd781640 1 -- 172.21.15.83:0/3614628867 wait complete. 2024-03-06T07:29:32.803 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:26.073593+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:29:32.803 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:29:32.975 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:33.805 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:33.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.904+0000 7f41b535e640 1 Processor -- start 2024-03-06T07:29:33.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.904+0000 7f41b535e640 1 -- start start 2024-03-06T07:29:33.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.904+0000 7f41b535e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f41b015bdd0 0x7f41b00c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.904+0000 7f41b535e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f41b00d16e0 0x7f41b00d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.904+0000 7f41b535e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41b00cebe0 con 0x7f41b015fa10 2024-03-06T07:29:33.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.904+0000 7f41b535e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41b00ce980 con 0x7f41b00d16e0 2024-03-06T07:29:33.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.904+0000 7f41b535e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41b00ceec0 con 0x7f41b015bdd0 2024-03-06T07:29:33.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.904+0000 7f41af7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f41b00d16e0 0x7f41b00d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.904+0000 7f41a7fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f41b015fa10 0x7f41b00ce270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48930/0 (socket says 172.21.15.83:48930) 2024-03-06T07:29:33.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.905+0000 7f41a7fff640 1 -- 172.21.15.83:0/10800950 learned_addr learned my addr 172.21.15.83:0/10800950 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:29:33.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.905+0000 7f41aeffd640 1 --2- 172.21.15.83:0/10800950 >> v2:172.21.15.183:3300/0 conn(0x7f41b015bdd0 0x7f41b00c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.905+0000 7f41af7fe640 1 -- 172.21.15.83:0/10800950 >> v2:172.21.15.183:3300/0 conn(0x7f41b015bdd0 msgr2=0x7f41b00c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:33.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.905+0000 7f41af7fe640 1 --2- 172.21.15.83:0/10800950 >> v2:172.21.15.183:3300/0 conn(0x7f41b015bdd0 0x7f41b00c5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.905+0000 7f41af7fe640 1 -- 172.21.15.83:0/10800950 >> v1:172.21.15.83:6789/0 conn(0x7f41b015fa10 legacy=0x7f41b00ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.905+0000 7f41af7fe640 1 -- 172.21.15.83:0/10800950 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b00cf1a0 con 0x7f41b00d16e0 2024-03-06T07:29:33.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.905+0000 7f41ae7fc640 1 -- 172.21.15.83:0/10800950 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4173032546 0 0) 0x7f41b00cebe0 con 0x7f41b015fa10 2024-03-06T07:29:33.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.905+0000 7f41aeffd640 1 --2- 172.21.15.83:0/10800950 >> v2:172.21.15.183:3300/0 conn(0x7f41b015bdd0 0x7f41b00c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:33.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.906+0000 7f41af7fe640 1 --2- 172.21.15.83:0/10800950 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41b00d16e0 0x7f41b00d1af0 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f41b006e3c0 tx=0x7f41a802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef17e42056a85a48 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.906+0000 7f41ae7fc640 1 -- 172.21.15.83:0/10800950 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a8039070 con 0x7f41b00d16e0 2024-03-06T07:29:33.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.906+0000 7f41ae7fc640 1 -- 172.21.15.83:0/10800950 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41a802cbf0 con 0x7f41b00d16e0 2024-03-06T07:29:33.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.906+0000 7f41ae7fc640 1 -- 172.21.15.83:0/10800950 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a8034440 con 0x7f41b00d16e0 2024-03-06T07:29:33.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.906+0000 7f41b535e640 1 -- 172.21.15.83:0/10800950 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41b00d16e0 msgr2=0x7f41b00d1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.906+0000 7f41b535e640 1 --2- 172.21.15.83:0/10800950 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41b00d16e0 0x7f41b00d1af0 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f41b006e3c0 tx=0x7f41a802be40 comp rx=0 tx=0).stop 2024-03-06T07:29:33.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.906+0000 7f41b535e640 1 -- 172.21.15.83:0/10800950 shutdown_connections 2024-03-06T07:29:33.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.906+0000 7f41b535e640 1 --2- 172.21.15.83:0/10800950 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41b00d16e0 0x7f41b00d1af0 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.906+0000 7f41b535e640 1 --2- 172.21.15.83:0/10800950 >> v2:172.21.15.183:3300/0 conn(0x7f41b015bdd0 0x7f41b00c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.906+0000 7f41b535e640 1 -- 172.21.15.83:0/10800950 >> 172.21.15.83:0/10800950 conn(0x7f41b009a9f0 msgr2=0x7f41b009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:33.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.906+0000 7f41b535e640 1 -- 172.21.15.83:0/10800950 shutdown_connections 2024-03-06T07:29:33.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.906+0000 7f41b535e640 1 -- 172.21.15.83:0/10800950 wait complete. 2024-03-06T07:29:33.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.907+0000 7f41b535e640 1 Processor -- start 2024-03-06T07:29:33.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.908+0000 7f41b535e640 1 -- start start 2024-03-06T07:29:33.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.908+0000 7f41b535e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41b00d16e0 0x7f41b01f0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.908+0000 7f41b535e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f41b015bdd0 0x7f41b00ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.908+0000 7f41b535e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f41b015fa10 0x7f41b00cd240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.908+0000 7f41b535e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b00cc5d0 con 0x7f41b00d16e0 2024-03-06T07:29:33.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.908+0000 7f41b535e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b00c5b50 con 0x7f41b015bdd0 2024-03-06T07:29:33.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.908+0000 7f41b535e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41b015c1b0 con 0x7f41b015fa10 2024-03-06T07:29:33.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.908+0000 7f41aeffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41b00d16e0 0x7f41b01f0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.908+0000 7f41aeffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41b00d16e0 0x7f41b01f0fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35130/0 (socket says 172.21.15.83:35130) 2024-03-06T07:29:33.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.908+0000 7f41aeffd640 1 -- 172.21.15.83:0/1704139524 learned_addr learned my addr 172.21.15.83:0/1704139524 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:33.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.908+0000 7f41a7fff640 1 --2- 172.21.15.83:0/1704139524 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f41b015bdd0 0x7f41b00ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.909+0000 7f41af7fe640 1 --2- 172.21.15.83:0/1704139524 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f41b015fa10 0x7f41b00cd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.909+0000 7f41aeffd640 1 -- 172.21.15.83:0/1704139524 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f41b015fa10 msgr2=0x7f41b00cd240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.909+0000 7f41aeffd640 1 --2- 172.21.15.83:0/1704139524 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f41b015fa10 0x7f41b00cd240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.909+0000 7f41aeffd640 1 -- 172.21.15.83:0/1704139524 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f41b015bdd0 msgr2=0x7f41b00ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.909+0000 7f41aeffd640 1 --2- 172.21.15.83:0/1704139524 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f41b015bdd0 0x7f41b00ccd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.909+0000 7f41aeffd640 1 -- 172.21.15.83:0/1704139524 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b0149bd0 con 0x7f41b00d16e0 2024-03-06T07:29:33.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.909+0000 7f41a7fff640 1 --2- 172.21.15.83:0/1704139524 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f41b015bdd0 0x7f41b00ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:33.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.909+0000 7f41aeffd640 1 --2- 172.21.15.83:0/1704139524 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41b00d16e0 0x7f41b01f0fa0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f41a000d7d0 tx=0x7f41a000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.909+0000 7f41a77fe640 1 -- 172.21.15.83:0/1704139524 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a0004030 con 0x7f41b00d16e0 2024-03-06T07:29:33.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.909+0000 7f41a77fe640 1 -- 172.21.15.83:0/1704139524 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41a00041d0 con 0x7f41b00d16e0 2024-03-06T07:29:33.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.909+0000 7f41b535e640 1 -- 172.21.15.83:0/1704139524 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41b0149eb0 con 0x7f41b00d16e0 2024-03-06T07:29:33.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.909+0000 7f41a77fe640 1 -- 172.21.15.83:0/1704139524 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a000b6c0 con 0x7f41b00d16e0 2024-03-06T07:29:33.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.909+0000 7f41b535e640 1 -- 172.21.15.83:0/1704139524 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41b014a470 con 0x7f41b00d16e0 2024-03-06T07:29:33.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.910+0000 7f41b535e640 1 -- 172.21.15.83:0/1704139524 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41b004f330 con 0x7f41b00d16e0 2024-03-06T07:29:33.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.911+0000 7f41a77fe640 1 -- 172.21.15.83:0/1704139524 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f41a000b860 con 0x7f41b00d16e0 2024-03-06T07:29:33.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.911+0000 7f41a77fe640 1 --2- 172.21.15.83:0/1704139524 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f419006b500 0x7f419006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.911+0000 7f41a77fe640 1 -- 172.21.15.83:0/1704139524 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f41a008a420 con 0x7f41b00d16e0 2024-03-06T07:29:33.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.912+0000 7f41a7fff640 1 --2- 172.21.15.83:0/1704139524 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f419006b500 0x7f419006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.912+0000 7f41a7fff640 1 --2- 172.21.15.83:0/1704139524 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f419006b500 0x7f419006d9c0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f41b01f4a10 tx=0x7f4198009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:33.916+0000 7f41a77fe640 1 -- 172.21.15.83:0/1704139524 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f41a0059330 con 0x7f41b00d16e0 2024-03-06T07:29:34.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:34.124+0000 7f41b535e640 1 -- 172.21.15.83:0/1704139524 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41b00b7a40 con 0x7f41b00d16e0 2024-03-06T07:29:34.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:34.125+0000 7f41a77fe640 1 -- 172.21.15.83:0/1704139524 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+3415 (secure 0 0 0) 0x7f41a0058c30 con 0x7f41b00d16e0 2024-03-06T07:29:34.127 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:29:34.127 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":81,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:26.073593+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:29:34.127 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 81 2024-03-06T07:29:34.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:34.127+0000 7f41b535e640 1 -- 172.21.15.83:0/1704139524 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f419006b500 msgr2=0x7f419006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:34.127+0000 7f41b535e640 1 --2- 172.21.15.83:0/1704139524 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f419006b500 0x7f419006d9c0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f41b01f4a10 tx=0x7f4198009040 comp rx=0 tx=0).stop 2024-03-06T07:29:34.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:34.127+0000 7f41b535e640 1 -- 172.21.15.83:0/1704139524 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41b00d16e0 msgr2=0x7f41b01f0fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:34.127+0000 7f41b535e640 1 --2- 172.21.15.83:0/1704139524 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41b00d16e0 0x7f41b01f0fa0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f41a000d7d0 tx=0x7f41a000dc90 comp rx=0 tx=0).stop 2024-03-06T07:29:34.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:34.127+0000 7f41b535e640 1 -- 172.21.15.83:0/1704139524 shutdown_connections 2024-03-06T07:29:34.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:34.127+0000 7f41b535e640 1 --2- 172.21.15.83:0/1704139524 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f419006b500 0x7f419006d9c0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:34.127+0000 7f41b535e640 1 --2- 172.21.15.83:0/1704139524 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f41b015fa10 0x7f41b00cd240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:34.127+0000 7f41b535e640 1 --2- 172.21.15.83:0/1704139524 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f41b015bdd0 0x7f41b00ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:34.127+0000 7f41b535e640 1 --2- 172.21.15.83:0/1704139524 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41b00d16e0 0x7f41b01f0fa0 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:34.127+0000 7f41b535e640 1 -- 172.21.15.83:0/1704139524 >> 172.21.15.83:0/1704139524 conn(0x7f41b009a9f0 msgr2=0x7f41b009add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:34.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:34.127+0000 7f41b535e640 1 -- 172.21.15.83:0/1704139524 shutdown_connections 2024-03-06T07:29:34.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:34.127+0000 7f41b535e640 1 -- 172.21.15.83:0/1704139524 wait complete. 2024-03-06T07:29:34.340 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:26.073593+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:29:34.341 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:29:34.977 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:35.342 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:35.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.441+0000 7f00b1afb640 1 Processor -- start 2024-03-06T07:29:35.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.441+0000 7f00b1afb640 1 -- start start 2024-03-06T07:29:35.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.442+0000 7f00b1afb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f00ac0d22b0 0x7f00ac0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.442+0000 7f00b1afb640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f00ac0b41f0 0x7f00ac0b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.442+0000 7f00b1afb640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00ac0b3430 con 0x7f00ac15dfe0 2024-03-06T07:29:35.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.442+0000 7f00b1afb640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00ac0d2c50 con 0x7f00ac0b41f0 2024-03-06T07:29:35.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.442+0000 7f00b1afb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac0b31e0 con 0x7f00ac0d22b0 2024-03-06T07:29:35.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.442+0000 7f00aaffd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f00ac0d22b0 0x7f00ac0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.442+0000 7f00abfff640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f00ac0b41f0 0x7f00ac0b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.442+0000 7f00aaffd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f00ac0d22b0 0x7f00ac0d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:36820/0 (socket says 172.21.15.83:36820) 2024-03-06T07:29:35.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.442+0000 7f00aaffd640 1 -- 172.21.15.83:0/1436431386 learned_addr learned my addr 172.21.15.83:0/1436431386 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:35.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.442+0000 7f00ab7fe640 1 --1- 172.21.15.83:0/1436431386 >> v1:172.21.15.183:6789/0 conn(0x7f00ac15dfe0 0x7f00ac0cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:56096/0 (socket says 172.21.15.83:56096) 2024-03-06T07:29:35.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.442+0000 7f00aaffd640 1 -- 172.21.15.83:0/1436431386 >> v2:172.21.15.183:3300/0 conn(0x7f00ac0b41f0 msgr2=0x7f00ac0b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:35.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.442+0000 7f00aaffd640 1 --2- 172.21.15.83:0/1436431386 >> v2:172.21.15.183:3300/0 conn(0x7f00ac0b41f0 0x7f00ac0b4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.442+0000 7f00aaffd640 1 -- 172.21.15.83:0/1436431386 >> v1:172.21.15.183:6789/0 conn(0x7f00ac15dfe0 legacy=0x7f00ac0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.442+0000 7f00aaffd640 1 -- 172.21.15.83:0/1436431386 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ac0b3710 con 0x7f00ac0d22b0 2024-03-06T07:29:35.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.442+0000 7f00abfff640 1 --2- 172.21.15.83:0/1436431386 >> v2:172.21.15.183:3300/0 conn(0x7f00ac0b41f0 0x7f00ac0b4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:35.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.443+0000 7f00aaffd640 1 --2- 172.21.15.83:0/1436431386 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f00ac0d22b0 0x7f00ac0d2710 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f00a0009870 tx=0x7f00a002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a16ca7e04352d985 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:35.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.443+0000 7f00aa7fc640 1 -- 172.21.15.83:0/1436431386 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a0034070 con 0x7f00ac0d22b0 2024-03-06T07:29:35.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.443+0000 7f00aa7fc640 1 -- 172.21.15.83:0/1436431386 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00a0030840 con 0x7f00ac0d22b0 2024-03-06T07:29:35.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.443+0000 7f00aa7fc640 1 -- 172.21.15.83:0/1436431386 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a0030c10 con 0x7f00ac0d22b0 2024-03-06T07:29:35.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.444+0000 7f00b1afb640 1 -- 172.21.15.83:0/1436431386 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f00ac0d22b0 msgr2=0x7f00ac0d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.444+0000 7f00b1afb640 1 --2- 172.21.15.83:0/1436431386 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f00ac0d22b0 0x7f00ac0d2710 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f00a0009870 tx=0x7f00a002b980 comp rx=0 tx=0).stop 2024-03-06T07:29:35.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.444+0000 7f00b1afb640 1 -- 172.21.15.83:0/1436431386 shutdown_connections 2024-03-06T07:29:35.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.444+0000 7f00b1afb640 1 --2- 172.21.15.83:0/1436431386 >> v2:172.21.15.183:3300/0 conn(0x7f00ac0b41f0 0x7f00ac0b4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.444+0000 7f00b1afb640 1 --2- 172.21.15.83:0/1436431386 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f00ac0d22b0 0x7f00ac0d2710 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.444+0000 7f00b1afb640 1 -- 172.21.15.83:0/1436431386 >> 172.21.15.83:0/1436431386 conn(0x7f00ac0d3cd0 msgr2=0x7f00ac0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:35.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.444+0000 7f00b1afb640 1 -- 172.21.15.83:0/1436431386 shutdown_connections 2024-03-06T07:29:35.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.444+0000 7f00b1afb640 1 -- 172.21.15.83:0/1436431386 wait complete. 2024-03-06T07:29:35.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.446+0000 7f00b1afb640 1 Processor -- start 2024-03-06T07:29:35.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.446+0000 7f00b1afb640 1 -- start start 2024-03-06T07:29:35.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.447+0000 7f00b1afb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f00ac0b41f0 0x7f00ac0cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.447+0000 7f00b1afb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f00ac0d22b0 0x7f00ac150f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.447+0000 7f00b1afb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f00ac15dfe0 0x7f00ac0cc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.447+0000 7f00b1afb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac0cfa00 con 0x7f00ac15dfe0 2024-03-06T07:29:35.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.447+0000 7f00b1afb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac0b3d60 con 0x7f00ac0d22b0 2024-03-06T07:29:35.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.447+0000 7f00b1afb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac0d1c70 con 0x7f00ac0b41f0 2024-03-06T07:29:35.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.447+0000 7f00abfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f00ac15dfe0 0x7f00ac0cc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.447+0000 7f00abfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f00ac15dfe0 0x7f00ac0cc9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35142/0 (socket says 172.21.15.83:35142) 2024-03-06T07:29:35.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.447+0000 7f00abfff640 1 -- 172.21.15.83:0/1398255876 learned_addr learned my addr 172.21.15.83:0/1398255876 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:35.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.447+0000 7f00ab7fe640 1 --2- 172.21.15.83:0/1398255876 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f00ac0b41f0 0x7f00ac0cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.447+0000 7f00aaffd640 1 --2- 172.21.15.83:0/1398255876 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f00ac0d22b0 0x7f00ac150f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.447+0000 7f00abfff640 1 -- 172.21.15.83:0/1398255876 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f00ac0b41f0 msgr2=0x7f00ac0cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.447+0000 7f00abfff640 1 --2- 172.21.15.83:0/1398255876 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f00ac0b41f0 0x7f00ac0cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.447+0000 7f00abfff640 1 -- 172.21.15.83:0/1398255876 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f00ac0d22b0 msgr2=0x7f00ac150f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.448+0000 7f00abfff640 1 --2- 172.21.15.83:0/1398255876 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f00ac0d22b0 0x7f00ac150f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.448+0000 7f00abfff640 1 -- 172.21.15.83:0/1398255876 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ac0cd820 con 0x7f00ac15dfe0 2024-03-06T07:29:35.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.448+0000 7f00abfff640 1 --2- 172.21.15.83:0/1398255876 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f00ac15dfe0 0x7f00ac0cc9a0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f00a400efc0 tx=0x7f00a4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:35.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.448+0000 7f0093fff640 1 -- 172.21.15.83:0/1398255876 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a4018030 con 0x7f00ac15dfe0 2024-03-06T07:29:35.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.448+0000 7f0093fff640 1 -- 172.21.15.83:0/1398255876 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00a400cd80 con 0x7f00ac15dfe0 2024-03-06T07:29:35.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.448+0000 7f00b1afb640 1 -- 172.21.15.83:0/1398255876 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00ac0cdb00 con 0x7f00ac15dfe0 2024-03-06T07:29:35.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.448+0000 7f0093fff640 1 -- 172.21.15.83:0/1398255876 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a4003bb0 con 0x7f00ac15dfe0 2024-03-06T07:29:35.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.448+0000 7f00b1afb640 1 -- 172.21.15.83:0/1398255876 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00ac0cb2e0 con 0x7f00ac15dfe0 2024-03-06T07:29:35.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.449+0000 7f00b1afb640 1 -- 172.21.15.83:0/1398255876 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00ac04f330 con 0x7f00ac15dfe0 2024-03-06T07:29:35.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.449+0000 7f0093fff640 1 -- 172.21.15.83:0/1398255876 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f00a4005d10 con 0x7f00ac15dfe0 2024-03-06T07:29:35.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.450+0000 7f0093fff640 1 --2- 172.21.15.83:0/1398255876 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f009806b500 0x7f009806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.450+0000 7f0093fff640 1 -- 172.21.15.83:0/1398255876 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f00a408aa30 con 0x7f00ac15dfe0 2024-03-06T07:29:35.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.450+0000 7f00ab7fe640 1 --2- 172.21.15.83:0/1398255876 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f009806b500 0x7f009806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.450+0000 7f00ab7fe640 1 --2- 172.21.15.83:0/1398255876 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f009806b500 0x7f009806d9c0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f009c007fd0 tx=0x7f009c007e80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:35.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.454+0000 7f0093fff640 1 -- 172.21.15.83:0/1398255876 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f00a4022070 con 0x7f00ac15dfe0 2024-03-06T07:29:35.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.664+0000 7f00b1afb640 1 -- 172.21.15.83:0/1398255876 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00ac0d0db0 con 0x7f00ac15dfe0 2024-03-06T07:29:35.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.665+0000 7f0093fff640 1 -- 172.21.15.83:0/1398255876 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+3415 (secure 0 0 0) 0x7f00a4010040 con 0x7f00ac15dfe0 2024-03-06T07:29:35.667 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:29:35.667 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":81,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:26.073593+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:29:35.667 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 81 2024-03-06T07:29:35.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.667+0000 7f00b1afb640 1 -- 172.21.15.83:0/1398255876 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f009806b500 msgr2=0x7f009806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.667+0000 7f00b1afb640 1 --2- 172.21.15.83:0/1398255876 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f009806b500 0x7f009806d9c0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f009c007fd0 tx=0x7f009c007e80 comp rx=0 tx=0).stop 2024-03-06T07:29:35.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.667+0000 7f00b1afb640 1 -- 172.21.15.83:0/1398255876 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f00ac15dfe0 msgr2=0x7f00ac0cc9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.667+0000 7f00b1afb640 1 --2- 172.21.15.83:0/1398255876 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f00ac15dfe0 0x7f00ac0cc9a0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f00a400efc0 tx=0x7f00a4009f90 comp rx=0 tx=0).stop 2024-03-06T07:29:35.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.667+0000 7f00b1afb640 1 -- 172.21.15.83:0/1398255876 shutdown_connections 2024-03-06T07:29:35.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.667+0000 7f00b1afb640 1 --2- 172.21.15.83:0/1398255876 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f009806b500 0x7f009806d9c0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.667+0000 7f00b1afb640 1 --2- 172.21.15.83:0/1398255876 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f00ac15dfe0 0x7f00ac0cc9a0 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.667+0000 7f00b1afb640 1 --2- 172.21.15.83:0/1398255876 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f00ac0d22b0 0x7f00ac150f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.667+0000 7f00b1afb640 1 --2- 172.21.15.83:0/1398255876 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f00ac0b41f0 0x7f00ac0cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.667+0000 7f00b1afb640 1 -- 172.21.15.83:0/1398255876 >> 172.21.15.83:0/1398255876 conn(0x7f00ac0d3cd0 msgr2=0x7f00ac0b2a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:35.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.667+0000 7f00b1afb640 1 -- 172.21.15.83:0/1398255876 shutdown_connections 2024-03-06T07:29:35.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:35.667+0000 7f00b1afb640 1 -- 172.21.15.83:0/1398255876 wait complete. 2024-03-06T07:29:35.881 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:26.073593+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:29:35.881 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:29:36.882 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:36.979 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:36.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.981+0000 7ff03865c640 1 Processor -- start 2024-03-06T07:29:36.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.981+0000 7ff03865c640 1 -- start start 2024-03-06T07:29:36.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.982+0000 7ff03865c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff0300d4760 0x7ff0300b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.982+0000 7ff03865c640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ff0300d0a80 0x7ff0300cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.982+0000 7ff03865c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0300d25d0 con 0x7ff0300b4170 2024-03-06T07:29:36.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.982+0000 7ff03865c640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0300d4bf0 con 0x7ff0300d0a80 2024-03-06T07:29:36.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.982+0000 7ff03865c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0300ce9e0 con 0x7ff0300d4760 2024-03-06T07:29:36.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.982+0000 7ff0363f8640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff0300b4170 0x7ff0300d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:47772/0 (socket says 172.21.15.83:47772) 2024-03-06T07:29:36.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.982+0000 7ff0363f8640 1 -- 172.21.15.83:0/3017866817 learned_addr learned my addr 172.21.15.83:0/3017866817 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:29:36.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.982+0000 7ff0353f6640 1 -- 172.21.15.83:0/3017866817 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3483456769 0 0) 0x7ff0300d25d0 con 0x7ff0300b4170 2024-03-06T07:29:36.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.983+0000 7ff0353f6640 1 -- 172.21.15.83:0/3017866817 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff01c003520 con 0x7ff0300b4170 2024-03-06T07:29:36.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.983+0000 7ff035bf7640 1 --2- 172.21.15.83:0/3017866817 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff0300d4760 0x7ff0300b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.983+0000 7ff036bf9640 1 --2- 172.21.15.83:0/3017866817 >> v2:172.21.15.183:3300/0 conn(0x7ff0300d0a80 0x7ff0300cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.983+0000 7ff0353f6640 1 -- 172.21.15.83:0/3017866817 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 446034866 0 0) 0x7ff01c003520 con 0x7ff0300b4170 2024-03-06T07:29:36.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.983+0000 7ff0353f6640 1 -- 172.21.15.83:0/3017866817 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff0300d4760 msgr2=0x7ff0300b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:36.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.983+0000 7ff0353f6640 1 --2- 172.21.15.83:0/3017866817 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff0300d4760 0x7ff0300b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.983+0000 7ff0353f6640 1 -- 172.21.15.83:0/3017866817 >> v2:172.21.15.183:3300/0 conn(0x7ff0300d0a80 msgr2=0x7ff0300cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:36.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.983+0000 7ff0353f6640 1 --2- 172.21.15.83:0/3017866817 >> v2:172.21.15.183:3300/0 conn(0x7ff0300d0a80 0x7ff0300cee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.983+0000 7ff0353f6640 1 -- 172.21.15.83:0/3017866817 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0300d2870 con 0x7ff0300b4170 2024-03-06T07:29:36.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.983+0000 7ff0353f6640 1 -- 172.21.15.83:0/3017866817 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7ff02c001670 con 0x7ff0300b4170 2024-03-06T07:29:36.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.983+0000 7ff036bf9640 1 --2- 172.21.15.83:0/3017866817 >> v2:172.21.15.183:3300/0 conn(0x7ff0300d0a80 0x7ff0300cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:36.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.983+0000 7ff0353f6640 1 -- 172.21.15.83:0/3017866817 >> v1:172.21.15.83:6789/0 conn(0x7ff0300b4170 legacy=0x7ff0300d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.983+0000 7ff0353f6640 1 --2- 172.21.15.83:0/3017866817 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff01c0039b0 0x7ff01c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.983+0000 7ff0353f6640 1 --2- 172.21.15.83:0/3017866817 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff01c004800 0x7ff01c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.983+0000 7ff0353f6640 1 --2- 172.21.15.83:0/3017866817 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff01c0051c0 0x7ff01c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.983+0000 7ff0353f6640 1 -- 172.21.15.83:0/3017866817 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0300d4bf0 con 0x7ff01c004800 2024-03-06T07:29:36.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.983+0000 7ff0353f6640 1 -- 172.21.15.83:0/3017866817 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0300ce9e0 con 0x7ff01c0039b0 2024-03-06T07:29:36.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.983+0000 7ff0353f6640 1 -- 172.21.15.83:0/3017866817 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff01c00bf00 con 0x7ff01c0051c0 2024-03-06T07:29:36.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.983+0000 7ff035bf7640 1 --2- 172.21.15.83:0/3017866817 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff01c004800 0x7ff01c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.983+0000 7ff0363f8640 1 --2- 172.21.15.83:0/3017866817 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff01c0039b0 0x7ff01c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.983+0000 7ff036bf9640 1 --2- 172.21.15.83:0/3017866817 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff01c0051c0 0x7ff01c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.983+0000 7ff035bf7640 1 -- 172.21.15.83:0/3017866817 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff01c0051c0 msgr2=0x7ff01c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:36.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.983+0000 7ff035bf7640 1 --2- 172.21.15.83:0/3017866817 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff01c0051c0 0x7ff01c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.983+0000 7ff035bf7640 1 -- 172.21.15.83:0/3017866817 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff01c0039b0 msgr2=0x7ff01c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:36.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.983+0000 7ff035bf7640 1 --2- 172.21.15.83:0/3017866817 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff01c0039b0 0x7ff01c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.983+0000 7ff035bf7640 1 -- 172.21.15.83:0/3017866817 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff01c003520 con 0x7ff01c004800 2024-03-06T07:29:36.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.984+0000 7ff035bf7640 1 --2- 172.21.15.83:0/3017866817 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff01c004800 0x7ff01c004c80 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7ff020009ac0 tx=0x7ff01c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a0f181141af4c89 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:36.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.984+0000 7ff0353f6640 1 -- 172.21.15.83:0/3017866817 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff020039070 con 0x7ff01c004800 2024-03-06T07:29:36.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.984+0000 7ff0353f6640 1 -- 172.21.15.83:0/3017866817 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff02002c6b0 con 0x7ff01c004800 2024-03-06T07:29:36.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.984+0000 7ff0353f6640 1 -- 172.21.15.83:0/3017866817 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff020009690 con 0x7ff01c004800 2024-03-06T07:29:36.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.985+0000 7ff03865c640 1 -- 172.21.15.83:0/3017866817 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff01c004800 msgr2=0x7ff01c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.985+0000 7ff03865c640 1 --2- 172.21.15.83:0/3017866817 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff01c004800 0x7ff01c004c80 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7ff020009ac0 tx=0x7ff01c005830 comp rx=0 tx=0).stop 2024-03-06T07:29:36.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.985+0000 7ff036bf9640 1 -- 172.21.15.83:0/3017866817 reap_dead start 2024-03-06T07:29:36.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.985+0000 7ff03865c640 1 -- 172.21.15.83:0/3017866817 shutdown_connections 2024-03-06T07:29:36.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.985+0000 7ff03865c640 1 -- 172.21.15.83:0/3017866817 >> 172.21.15.83:0/3017866817 conn(0x7ff0300d3c40 msgr2=0x7ff0300d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:36.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.985+0000 7ff03865c640 1 -- 172.21.15.83:0/3017866817 shutdown_connections 2024-03-06T07:29:36.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.985+0000 7ff03865c640 1 -- 172.21.15.83:0/3017866817 wait complete. 2024-03-06T07:29:36.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.986+0000 7ff03865c640 1 Processor -- start 2024-03-06T07:29:36.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.986+0000 7ff03865c640 1 -- start start 2024-03-06T07:29:36.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.987+0000 7ff03865c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff01c004800 0x7ff030157180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.987+0000 7ff03865c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff0300ca3a0 0x7ff0300b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.987+0000 7ff03865c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff03015ae40 0x7ff03015b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.987+0000 7ff03865c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0300b4dd0 con 0x7ff01c004800 2024-03-06T07:29:36.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.987+0000 7ff03865c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0300b4550 con 0x7ff0300ca3a0 2024-03-06T07:29:36.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.987+0000 7ff03865c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0300b4f50 con 0x7ff03015ae40 2024-03-06T07:29:36.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.987+0000 7ff0363f8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff01c004800 0x7ff030157180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.987+0000 7ff0363f8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff01c004800 0x7ff030157180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34150/0 (socket says 172.21.15.83:34150) 2024-03-06T07:29:36.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.987+0000 7ff0363f8640 1 -- 172.21.15.83:0/621322503 learned_addr learned my addr 172.21.15.83:0/621322503 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:36.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.987+0000 7ff036bf9640 1 --2- 172.21.15.83:0/621322503 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff03015ae40 0x7ff03015b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.987+0000 7ff035bf7640 1 --2- 172.21.15.83:0/621322503 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff0300ca3a0 0x7ff0300b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.987+0000 7ff0363f8640 1 -- 172.21.15.83:0/621322503 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff03015ae40 msgr2=0x7ff03015b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.987+0000 7ff0363f8640 1 --2- 172.21.15.83:0/621322503 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff03015ae40 0x7ff03015b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.987+0000 7ff0363f8640 1 -- 172.21.15.83:0/621322503 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff0300ca3a0 msgr2=0x7ff0300b71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.987+0000 7ff0363f8640 1 --2- 172.21.15.83:0/621322503 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff0300ca3a0 0x7ff0300b71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.987+0000 7ff0363f8640 1 -- 172.21.15.83:0/621322503 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0301576c0 con 0x7ff01c004800 2024-03-06T07:29:36.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.987+0000 7ff036bf9640 1 --2- 172.21.15.83:0/621322503 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff03015ae40 0x7ff03015b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:36.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.987+0000 7ff0363f8640 1 --2- 172.21.15.83:0/621322503 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff01c004800 0x7ff030157180 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7ff02c007340 tx=0x7ff02c007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:36.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.987+0000 7ff026ffd640 1 -- 172.21.15.83:0/621322503 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff02c00f9c0 con 0x7ff01c004800 2024-03-06T07:29:36.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.987+0000 7ff026ffd640 1 -- 172.21.15.83:0/621322503 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff02c018070 con 0x7ff01c004800 2024-03-06T07:29:36.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.988+0000 7ff03865c640 1 -- 172.21.15.83:0/621322503 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0300cc110 con 0x7ff01c004800 2024-03-06T07:29:36.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.988+0000 7ff026ffd640 1 -- 172.21.15.83:0/621322503 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff02c00fda0 con 0x7ff01c004800 2024-03-06T07:29:36.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.988+0000 7ff03865c640 1 -- 172.21.15.83:0/621322503 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0300cc6d0 con 0x7ff01c004800 2024-03-06T07:29:36.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.988+0000 7ff03865c640 1 -- 172.21.15.83:0/621322503 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0300d4760 con 0x7ff01c004800 2024-03-06T07:29:36.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.989+0000 7ff026ffd640 1 -- 172.21.15.83:0/621322503 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff02c01d460 con 0x7ff01c004800 2024-03-06T07:29:36.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.989+0000 7ff026ffd640 1 --2- 172.21.15.83:0/621322503 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff01406b4d0 0x7ff01406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.989+0000 7ff026ffd640 1 -- 172.21.15.83:0/621322503 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff02c08a360 con 0x7ff01c004800 2024-03-06T07:29:36.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.989+0000 7ff035bf7640 1 --2- 172.21.15.83:0/621322503 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff01406b4d0 0x7ff01406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.990+0000 7ff035bf7640 1 --2- 172.21.15.83:0/621322503 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff01406b4d0 0x7ff01406d990 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7ff020036000 tx=0x7ff020030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:36.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:36.997+0000 7ff026ffd640 1 -- 172.21.15.83:0/621322503 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff0300d4760 con 0x7ff01c004800 2024-03-06T07:29:37.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:37.211+0000 7ff03865c640 1 -- 172.21.15.83:0/621322503 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff03004f330 con 0x7ff01c004800 2024-03-06T07:29:37.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:37.211+0000 7ff026ffd640 1 -- 172.21.15.83:0/621322503 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+3415 (secure 0 0 0) 0x7ff0301576c0 con 0x7ff01c004800 2024-03-06T07:29:37.213 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:29:37.213 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":81,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:26.073593+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:29:37.214 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 81 2024-03-06T07:29:37.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:37.213+0000 7ff03865c640 1 -- 172.21.15.83:0/621322503 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff01406b4d0 msgr2=0x7ff01406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:37.213+0000 7ff03865c640 1 --2- 172.21.15.83:0/621322503 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff01406b4d0 0x7ff01406d990 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7ff020036000 tx=0x7ff020030000 comp rx=0 tx=0).stop 2024-03-06T07:29:37.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:37.213+0000 7ff03865c640 1 -- 172.21.15.83:0/621322503 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff01c004800 msgr2=0x7ff030157180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:37.213+0000 7ff03865c640 1 --2- 172.21.15.83:0/621322503 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff01c004800 0x7ff030157180 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7ff02c007340 tx=0x7ff02c007800 comp rx=0 tx=0).stop 2024-03-06T07:29:37.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:37.213+0000 7ff03865c640 1 -- 172.21.15.83:0/621322503 shutdown_connections 2024-03-06T07:29:37.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:37.213+0000 7ff03865c640 1 --2- 172.21.15.83:0/621322503 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff01406b4d0 0x7ff01406d990 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:37.213+0000 7ff03865c640 1 --2- 172.21.15.83:0/621322503 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff03015ae40 0x7ff03015b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:37.213+0000 7ff03865c640 1 --2- 172.21.15.83:0/621322503 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff0300ca3a0 0x7ff0300b71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:37.213+0000 7ff03865c640 1 --2- 172.21.15.83:0/621322503 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff01c004800 0x7ff030157180 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:37.213+0000 7ff03865c640 1 -- 172.21.15.83:0/621322503 >> 172.21.15.83:0/621322503 conn(0x7ff0300d3c40 msgr2=0x7ff03006cf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:37.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:37.213+0000 7ff03865c640 1 -- 172.21.15.83:0/621322503 shutdown_connections 2024-03-06T07:29:37.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:37.213+0000 7ff03865c640 1 -- 172.21.15.83:0/621322503 wait complete. 2024-03-06T07:29:37.427 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:26.073593+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:29:37.427 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:29:38.428 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:38.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.527+0000 7f86680b5640 1 Processor -- start 2024-03-06T07:29:38.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.527+0000 7f86680b5640 1 -- start start 2024-03-06T07:29:38.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.528+0000 7f86680b5640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f86600d22b0 0x7f86600d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.528+0000 7f86680b5640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86600b3430 con 0x7f866015dfe0 2024-03-06T07:29:38.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.528+0000 7f86680b5640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86600b3710 con 0x7f86600b41f0 2024-03-06T07:29:38.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.528+0000 7f86680b5640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86600d2c50 con 0x7f86600d22b0 2024-03-06T07:29:38.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.528+0000 7f8665e51640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f866015dfe0 0x7f86600cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:47782/0 (socket says 172.21.15.83:47782) 2024-03-06T07:29:38.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.528+0000 7f8665e51640 1 -- 172.21.15.83:0/562564423 learned_addr learned my addr 172.21.15.83:0/562564423 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:29:38.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.528+0000 7f8665650640 1 --2- 172.21.15.83:0/562564423 >> v2:172.21.15.183:3300/0 conn(0x7f86600d22b0 0x7f86600d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.528+0000 7f8664e4f640 1 -- 172.21.15.83:0/562564423 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3727570257 0 0) 0x7f86600b3430 con 0x7f866015dfe0 2024-03-06T07:29:38.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.528+0000 7f8664e4f640 1 -- 172.21.15.83:0/562564423 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f864c003520 con 0x7f866015dfe0 2024-03-06T07:29:38.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.528+0000 7f8664e4f640 1 -- 172.21.15.83:0/562564423 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3484924509 0 0) 0x7f864c003520 con 0x7f866015dfe0 2024-03-06T07:29:38.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.528+0000 7f8664e4f640 1 -- 172.21.15.83:0/562564423 >> v2:172.21.15.183:3300/0 conn(0x7f86600d22b0 msgr2=0x7f86600d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:38.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.528+0000 7f8664e4f640 1 --2- 172.21.15.83:0/562564423 >> v2:172.21.15.183:3300/0 conn(0x7f86600d22b0 0x7f86600d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.528+0000 7f8664e4f640 1 -- 172.21.15.83:0/562564423 >> v1:172.21.15.183:6789/0 conn(0x7f86600b41f0 legacy=0x7f86600b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.528+0000 7f8664e4f640 1 -- 172.21.15.83:0/562564423 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86600b39f0 con 0x7f866015dfe0 2024-03-06T07:29:38.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.528+0000 7f8664e4f640 1 -- 172.21.15.83:0/562564423 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f865c001670 con 0x7f866015dfe0 2024-03-06T07:29:38.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.528+0000 7f8664e4f640 1 -- 172.21.15.83:0/562564423 >> v1:172.21.15.83:6789/0 conn(0x7f866015dfe0 legacy=0x7f86600cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.529+0000 7f8664e4f640 1 --2- 172.21.15.83:0/562564423 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f864c0039b0 0x7f864c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.529+0000 7f8664e4f640 1 --2- 172.21.15.83:0/562564423 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f864c004800 0x7f864c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.529+0000 7f8664e4f640 1 --2- 172.21.15.83:0/562564423 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f864c0051c0 0x7f864c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.529+0000 7f8664e4f640 1 -- 172.21.15.83:0/562564423 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864c00c030 con 0x7f864c0051c0 2024-03-06T07:29:38.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.529+0000 7f8664e4f640 1 -- 172.21.15.83:0/562564423 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864c00c1e0 con 0x7f864c004800 2024-03-06T07:29:38.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.529+0000 7f8664e4f640 1 -- 172.21.15.83:0/562564423 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f864c00c3e0 con 0x7f864c0039b0 2024-03-06T07:29:38.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.529+0000 7f8666652640 1 --2- 172.21.15.83:0/562564423 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f864c0051c0 0x7f864c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.529+0000 7f8665650640 1 --2- 172.21.15.83:0/562564423 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f864c004800 0x7f864c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.529+0000 7f8665e51640 1 --2- 172.21.15.83:0/562564423 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f864c0039b0 0x7f864c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.529+0000 7f8666652640 1 -- 172.21.15.83:0/562564423 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f864c0039b0 msgr2=0x7f864c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:38.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.529+0000 7f8666652640 1 --2- 172.21.15.83:0/562564423 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f864c0039b0 0x7f864c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.529+0000 7f8666652640 1 -- 172.21.15.83:0/562564423 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f864c004800 msgr2=0x7f864c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:38.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.529+0000 7f8666652640 1 --2- 172.21.15.83:0/562564423 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f864c004800 0x7f864c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.529+0000 7f8666652640 1 -- 172.21.15.83:0/562564423 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f864c003520 con 0x7f864c0051c0 2024-03-06T07:29:38.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.530+0000 7f8666652640 1 -- 172.21.15.83:0/562564423 reap_dead start 2024-03-06T07:29:38.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.530+0000 7f8666652640 1 --2- 172.21.15.83:0/562564423 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f864c0051c0 0x7f864c00b9c0 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f8658001e50 tx=0x7f864c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7b9efb5a16be433 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.530+0000 7f8664e4f640 1 -- 172.21.15.83:0/562564423 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8658001a70 con 0x7f864c0051c0 2024-03-06T07:29:38.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.530+0000 7f8664e4f640 1 -- 172.21.15.83:0/562564423 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8658003bb0 con 0x7f864c0051c0 2024-03-06T07:29:38.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.530+0000 7f8664e4f640 1 -- 172.21.15.83:0/562564423 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86580050d0 con 0x7f864c0051c0 2024-03-06T07:29:38.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.530+0000 7f86680b5640 1 -- 172.21.15.83:0/562564423 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f864c0051c0 msgr2=0x7f864c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.530+0000 7f86680b5640 1 --2- 172.21.15.83:0/562564423 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f864c0051c0 0x7f864c00b9c0 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f8658001e50 tx=0x7f864c005830 comp rx=0 tx=0).stop 2024-03-06T07:29:38.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.530+0000 7f86680b5640 1 -- 172.21.15.83:0/562564423 shutdown_connections 2024-03-06T07:29:38.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.530+0000 7f86680b5640 1 --2- 172.21.15.83:0/562564423 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f864c0051c0 0x7f864c00b9c0 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.530+0000 7f86680b5640 1 -- 172.21.15.83:0/562564423 >> 172.21.15.83:0/562564423 conn(0x7f86600d3cd0 msgr2=0x7f86600d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:38.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.530+0000 7f86680b5640 1 -- 172.21.15.83:0/562564423 shutdown_connections 2024-03-06T07:29:38.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.530+0000 7f86680b5640 1 -- 172.21.15.83:0/562564423 wait complete. 2024-03-06T07:29:38.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.532+0000 7f86680b5640 1 Processor -- start 2024-03-06T07:29:38.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.532+0000 7f86680b5640 1 -- start start 2024-03-06T07:29:38.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.532+0000 7f86680b5640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f864c0051c0 0x7f86601f3410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.532+0000 7f86680b5640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f86600cc8f0 0x7f8660156560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.532+0000 7f86680b5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f866015b650 0x7f866015bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.532+0000 7f86680b5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86600b3d60 con 0x7f866015b650 2024-03-06T07:29:38.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.532+0000 7f86680b5640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86600b31e0 con 0x7f864c0051c0 2024-03-06T07:29:38.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.532+0000 7f86680b5640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f866015e3c0 con 0x7f86600cc8f0 2024-03-06T07:29:38.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.533+0000 7f8666652640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f866015b650 0x7f866015bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.533+0000 7f8666652640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f866015b650 0x7f866015bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34168/0 (socket says 172.21.15.83:34168) 2024-03-06T07:29:38.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.533+0000 7f8666652640 1 -- 172.21.15.83:0/2782988625 learned_addr learned my addr 172.21.15.83:0/2782988625 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:38.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.533+0000 7f8665650640 1 --2- 172.21.15.83:0/2782988625 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f86600cc8f0 0x7f8660156560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.533+0000 7f8665e51640 1 --2- 172.21.15.83:0/2782988625 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f864c0051c0 0x7f86601f3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.533+0000 7f8666652640 1 -- 172.21.15.83:0/2782988625 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f86600cc8f0 msgr2=0x7f8660156560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.533+0000 7f8666652640 1 --2- 172.21.15.83:0/2782988625 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f86600cc8f0 0x7f8660156560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.533+0000 7f8666652640 1 -- 172.21.15.83:0/2782988625 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f864c0051c0 msgr2=0x7f86601f3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.533+0000 7f8666652640 1 --2- 172.21.15.83:0/2782988625 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f864c0051c0 0x7f86601f3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.533+0000 7f8666652640 1 -- 172.21.15.83:0/2782988625 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86600cce90 con 0x7f866015b650 2024-03-06T07:29:38.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.533+0000 7f8666652640 1 --2- 172.21.15.83:0/2782988625 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f866015b650 0x7f866015bad0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f865802c060 tx=0x7f86580014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.533+0000 7f86567fc640 1 -- 172.21.15.83:0/2782988625 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8658007db0 con 0x7f866015b650 2024-03-06T07:29:38.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.533+0000 7f86567fc640 1 -- 172.21.15.83:0/2782988625 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8658006d60 con 0x7f866015b650 2024-03-06T07:29:38.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.533+0000 7f86567fc640 1 -- 172.21.15.83:0/2782988625 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8658007230 con 0x7f866015b650 2024-03-06T07:29:38.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.533+0000 7f86680b5640 1 -- 172.21.15.83:0/2782988625 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f866015eb60 con 0x7f866015b650 2024-03-06T07:29:38.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.533+0000 7f86680b5640 1 -- 172.21.15.83:0/2782988625 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86601f3950 con 0x7f866015b650 2024-03-06T07:29:38.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.534+0000 7f86680b5640 1 -- 172.21.15.83:0/2782988625 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f866004f330 con 0x7f866015b650 2024-03-06T07:29:38.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.535+0000 7f86567fc640 1 -- 172.21.15.83:0/2782988625 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f86580073d0 con 0x7f866015b650 2024-03-06T07:29:38.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.535+0000 7f86567fc640 1 --2- 172.21.15.83:0/2782988625 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f864406b500 0x7f864406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.535+0000 7f86567fc640 1 -- 172.21.15.83:0/2782988625 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f86580ac1a0 con 0x7f866015b650 2024-03-06T07:29:38.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.535+0000 7f8665e51640 1 --2- 172.21.15.83:0/2782988625 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f864406b500 0x7f864406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.536+0000 7f8665e51640 1 --2- 172.21.15.83:0/2782988625 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f864406b500 0x7f864406d9c0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f865c000f60 tx=0x7f865c000ed0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.539+0000 7f86567fc640 1 -- 172.21.15.83:0/2782988625 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8658049020 con 0x7f866015b650 2024-03-06T07:29:38.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.753+0000 7f86680b5640 1 -- 172.21.15.83:0/2782988625 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86601e8940 con 0x7f866015b650 2024-03-06T07:29:38.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.754+0000 7f86567fc640 1 -- 172.21.15.83:0/2782988625 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+3415 (secure 0 0 0) 0x7f8658033030 con 0x7f866015b650 2024-03-06T07:29:38.757 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:29:38.757 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":81,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:26.073593+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:29:38.757 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 81 2024-03-06T07:29:38.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.756+0000 7f86680b5640 1 -- 172.21.15.83:0/2782988625 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f864406b500 msgr2=0x7f864406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.756+0000 7f86680b5640 1 --2- 172.21.15.83:0/2782988625 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f864406b500 0x7f864406d9c0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f865c000f60 tx=0x7f865c000ed0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.756+0000 7f86680b5640 1 -- 172.21.15.83:0/2782988625 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f866015b650 msgr2=0x7f866015bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.756+0000 7f86680b5640 1 --2- 172.21.15.83:0/2782988625 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f866015b650 0x7f866015bad0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f865802c060 tx=0x7f86580014a0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.756+0000 7f86680b5640 1 -- 172.21.15.83:0/2782988625 shutdown_connections 2024-03-06T07:29:38.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.756+0000 7f86680b5640 1 --2- 172.21.15.83:0/2782988625 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f864406b500 0x7f864406d9c0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.756+0000 7f86680b5640 1 --2- 172.21.15.83:0/2782988625 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f866015b650 0x7f866015bad0 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.756+0000 7f86680b5640 1 --2- 172.21.15.83:0/2782988625 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f86600cc8f0 0x7f8660156560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.756+0000 7f86680b5640 1 --2- 172.21.15.83:0/2782988625 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f864c0051c0 0x7f86601f3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.756+0000 7f86680b5640 1 -- 172.21.15.83:0/2782988625 >> 172.21.15.83:0/2782988625 conn(0x7f86600d3cd0 msgr2=0x7f86600b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:38.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.756+0000 7f86680b5640 1 -- 172.21.15.83:0/2782988625 shutdown_connections 2024-03-06T07:29:38.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:38.756+0000 7f86680b5640 1 -- 172.21.15.83:0/2782988625 wait complete. 2024-03-06T07:29:38.971 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:26.073593+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:29:38.971 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:29:38.981 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:39.972 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:40.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.086+0000 7fb16713a640 1 Processor -- start 2024-03-06T07:29:40.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.086+0000 7fb16713a640 1 -- start start 2024-03-06T07:29:40.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.087+0000 7fb16713a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb1601515e0 0x7fb1600cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.087+0000 7fb16713a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fb16015b430 0x7fb160071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.087+0000 7fb16713a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb16015b8e0 con 0x7fb1600cd7a0 2024-03-06T07:29:40.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.087+0000 7fb16713a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb160070a70 con 0x7fb1601515e0 2024-03-06T07:29:40.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.087+0000 7fb16713a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb160070d50 con 0x7fb16015b430 2024-03-06T07:29:40.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.087+0000 7fb164ed6640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb1601515e0 0x7fb1600cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.087+0000 7fb164ed6640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb1601515e0 0x7fb1600cce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34182/0 (socket says 172.21.15.83:34182) 2024-03-06T07:29:40.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.087+0000 7fb164ed6640 1 -- 172.21.15.83:0/4183445783 learned_addr learned my addr 172.21.15.83:0/4183445783 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:40.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.087+0000 7fb15ffff640 1 --2- 172.21.15.83:0/4183445783 >> v2:172.21.15.183:3300/0 conn(0x7fb16015b430 0x7fb160071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.087+0000 7fb164ed6640 1 -- 172.21.15.83:0/4183445783 >> v2:172.21.15.183:3300/0 conn(0x7fb16015b430 msgr2=0x7fb160071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:40.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.087+0000 7fb164ed6640 1 --2- 172.21.15.83:0/4183445783 >> v2:172.21.15.183:3300/0 conn(0x7fb16015b430 0x7fb160071830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.087+0000 7fb164ed6640 1 -- 172.21.15.83:0/4183445783 >> v1:172.21.15.183:6789/0 conn(0x7fb1600cd7a0 legacy=0x7fb1600cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.087+0000 7fb164ed6640 1 -- 172.21.15.83:0/4183445783 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb160071030 con 0x7fb1601515e0 2024-03-06T07:29:40.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.087+0000 7fb15ffff640 1 --2- 172.21.15.83:0/4183445783 >> v2:172.21.15.183:3300/0 conn(0x7fb16015b430 0x7fb160071830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:40.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.088+0000 7fb164ed6640 1 --2- 172.21.15.83:0/4183445783 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1601515e0 0x7fb1600cce40 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7fb1600cc110 tx=0x7fb15002e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc24f98b6e65e6b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:40.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.088+0000 7fb15f7fe640 1 -- 172.21.15.83:0/4183445783 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15003b070 con 0x7fb1601515e0 2024-03-06T07:29:40.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.088+0000 7fb15f7fe640 1 -- 172.21.15.83:0/4183445783 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb150036440 con 0x7fb1601515e0 2024-03-06T07:29:40.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.088+0000 7fb15f7fe640 1 -- 172.21.15.83:0/4183445783 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb150036810 con 0x7fb1601515e0 2024-03-06T07:29:40.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.089+0000 7fb16713a640 1 -- 172.21.15.83:0/4183445783 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1601515e0 msgr2=0x7fb1600cce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.089+0000 7fb16713a640 1 --2- 172.21.15.83:0/4183445783 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1601515e0 0x7fb1600cce40 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7fb1600cc110 tx=0x7fb15002e170 comp rx=0 tx=0).stop 2024-03-06T07:29:40.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.089+0000 7fb16713a640 1 -- 172.21.15.83:0/4183445783 shutdown_connections 2024-03-06T07:29:40.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.089+0000 7fb16713a640 1 --2- 172.21.15.83:0/4183445783 >> v2:172.21.15.183:3300/0 conn(0x7fb16015b430 0x7fb160071830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.089+0000 7fb16713a640 1 --2- 172.21.15.83:0/4183445783 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1601515e0 0x7fb1600cce40 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.089+0000 7fb16713a640 1 -- 172.21.15.83:0/4183445783 >> 172.21.15.83:0/4183445783 conn(0x7fb1600cf8a0 msgr2=0x7fb1600cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:40.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.089+0000 7fb16713a640 1 -- 172.21.15.83:0/4183445783 shutdown_connections 2024-03-06T07:29:40.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.089+0000 7fb16713a640 1 -- 172.21.15.83:0/4183445783 wait complete. 2024-03-06T07:29:40.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.090+0000 7fb16713a640 1 Processor -- start 2024-03-06T07:29:40.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.090+0000 7fb16713a640 1 -- start start 2024-03-06T07:29:40.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.092+0000 7fb16713a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1600cd7a0 0x7fb16006dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.092+0000 7fb16713a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1601515e0 0x7fb16006e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.092+0000 7fb16713a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb16015b430 0x7fb160158dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.092+0000 7fb16713a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb160071ef0 con 0x7fb1600cd7a0 2024-03-06T07:29:40.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.092+0000 7fb16713a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb160071d70 con 0x7fb1601515e0 2024-03-06T07:29:40.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.092+0000 7fb16713a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1600cc3a0 con 0x7fb16015b430 2024-03-06T07:29:40.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.092+0000 7fb164ed6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1600cd7a0 0x7fb16006dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.092+0000 7fb164ed6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1600cd7a0 0x7fb16006dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34194/0 (socket says 172.21.15.83:34194) 2024-03-06T07:29:40.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.092+0000 7fb164ed6640 1 -- 172.21.15.83:0/3135177532 learned_addr learned my addr 172.21.15.83:0/3135177532 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:40.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.092+0000 7fb15ffff640 1 --2- 172.21.15.83:0/3135177532 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1601515e0 0x7fb16006e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.092+0000 7fb1656d7640 1 --2- 172.21.15.83:0/3135177532 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb16015b430 0x7fb160158dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.092+0000 7fb164ed6640 1 -- 172.21.15.83:0/3135177532 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb16015b430 msgr2=0x7fb160158dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.092+0000 7fb164ed6640 1 --2- 172.21.15.83:0/3135177532 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb16015b430 0x7fb160158dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.092+0000 7fb164ed6640 1 -- 172.21.15.83:0/3135177532 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1601515e0 msgr2=0x7fb16006e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.092+0000 7fb164ed6640 1 --2- 172.21.15.83:0/3135177532 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1601515e0 0x7fb16006e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.092+0000 7fb164ed6640 1 -- 172.21.15.83:0/3135177532 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1600ca930 con 0x7fb1600cd7a0 2024-03-06T07:29:40.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.092+0000 7fb1656d7640 1 --2- 172.21.15.83:0/3135177532 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb16015b430 0x7fb160158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:40.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.092+0000 7fb164ed6640 1 --2- 172.21.15.83:0/3135177532 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1600cd7a0 0x7fb16006dc40 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7fb15002e1b0 tx=0x7fb1500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:40.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.092+0000 7fb15d7fa640 1 -- 172.21.15.83:0/3135177532 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb150045020 con 0x7fb1600cd7a0 2024-03-06T07:29:40.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.093+0000 7fb15d7fa640 1 -- 172.21.15.83:0/3135177532 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1500346e0 con 0x7fb1600cd7a0 2024-03-06T07:29:40.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.093+0000 7fb15d7fa640 1 -- 172.21.15.83:0/3135177532 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15003b040 con 0x7fb1600cd7a0 2024-03-06T07:29:40.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.093+0000 7fb16713a640 1 -- 172.21.15.83:0/3135177532 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1600cabd0 con 0x7fb1600cd7a0 2024-03-06T07:29:40.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.093+0000 7fb16713a640 1 -- 172.21.15.83:0/3135177532 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1600cb470 con 0x7fb1600cd7a0 2024-03-06T07:29:40.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.093+0000 7fb16713a640 1 -- 172.21.15.83:0/3135177532 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1600cada0 con 0x7fb1600cd7a0 2024-03-06T07:29:40.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.094+0000 7fb15d7fa640 1 -- 172.21.15.83:0/3135177532 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb150048020 con 0x7fb1600cd7a0 2024-03-06T07:29:40.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.094+0000 7fb15d7fa640 1 --2- 172.21.15.83:0/3135177532 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb14006b4d0 0x7fb14006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.094+0000 7fb15ffff640 1 --2- 172.21.15.83:0/3135177532 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb14006b4d0 0x7fb14006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.094+0000 7fb15d7fa640 1 -- 172.21.15.83:0/3135177532 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb1500ae980 con 0x7fb1600cd7a0 2024-03-06T07:29:40.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.095+0000 7fb15ffff640 1 --2- 172.21.15.83:0/3135177532 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb14006b4d0 0x7fb14006d990 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fb16014ecd0 tx=0x7fb148008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:40.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.099+0000 7fb15d7fa640 1 -- 172.21.15.83:0/3135177532 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb15004f070 con 0x7fb1600cd7a0 2024-03-06T07:29:40.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.323+0000 7fb16713a640 1 -- 172.21.15.83:0/3135177532 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb160068840 con 0x7fb1600cd7a0 2024-03-06T07:29:40.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.324+0000 7fb15d7fa640 1 -- 172.21.15.83:0/3135177532 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+3415 (secure 0 0 0) 0x7fb15007d810 con 0x7fb1600cd7a0 2024-03-06T07:29:40.327 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:29:40.327 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":81,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:26.073593+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:29:40.327 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 81 2024-03-06T07:29:40.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.326+0000 7fb16713a640 1 -- 172.21.15.83:0/3135177532 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb14006b4d0 msgr2=0x7fb14006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.326+0000 7fb16713a640 1 --2- 172.21.15.83:0/3135177532 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb14006b4d0 0x7fb14006d990 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fb16014ecd0 tx=0x7fb148008040 comp rx=0 tx=0).stop 2024-03-06T07:29:40.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.326+0000 7fb16713a640 1 -- 172.21.15.83:0/3135177532 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1600cd7a0 msgr2=0x7fb16006dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.326+0000 7fb16713a640 1 --2- 172.21.15.83:0/3135177532 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1600cd7a0 0x7fb16006dc40 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7fb15002e1b0 tx=0x7fb1500046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.326+0000 7fb16713a640 1 -- 172.21.15.83:0/3135177532 shutdown_connections 2024-03-06T07:29:40.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.326+0000 7fb16713a640 1 --2- 172.21.15.83:0/3135177532 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb14006b4d0 0x7fb14006d990 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.326+0000 7fb16713a640 1 --2- 172.21.15.83:0/3135177532 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb16015b430 0x7fb160158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.326+0000 7fb16713a640 1 --2- 172.21.15.83:0/3135177532 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1601515e0 0x7fb16006e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.326+0000 7fb16713a640 1 --2- 172.21.15.83:0/3135177532 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1600cd7a0 0x7fb16006dc40 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.326+0000 7fb16713a640 1 -- 172.21.15.83:0/3135177532 >> 172.21.15.83:0/3135177532 conn(0x7fb1600cf8a0 msgr2=0x7fb16014ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:40.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.326+0000 7fb16713a640 1 -- 172.21.15.83:0/3135177532 shutdown_connections 2024-03-06T07:29:40.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:40.326+0000 7fb16713a640 1 -- 172.21.15.83:0/3135177532 wait complete. 2024-03-06T07:29:40.542 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:26.073593+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:29:40.543 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:29:40.983 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:41.543 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:41.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.643+0000 7f971a084640 1 Processor -- start 2024-03-06T07:29:41.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.643+0000 7f971a084640 1 -- start start 2024-03-06T07:29:41.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.644+0000 7f971a084640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f971415dfe0 0x7f97140cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.644+0000 7f971a084640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f97140cf830 0x7f97140d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.644+0000 7f971a084640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97140b3430 con 0x7f97140d27f0 2024-03-06T07:29:41.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.644+0000 7f971a084640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97140d2c50 con 0x7f97140cf830 2024-03-06T07:29:41.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.644+0000 7f971a084640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97140b31e0 con 0x7f971415dfe0 2024-03-06T07:29:41.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.644+0000 7f9713fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f97140d27f0 0x7f97140d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:47788/0 (socket says 172.21.15.83:47788) 2024-03-06T07:29:41.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.644+0000 7f9713fff640 1 -- 172.21.15.83:0/169964197 learned_addr learned my addr 172.21.15.83:0/169964197 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:29:41.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.644+0000 7f9712ffd640 1 -- 172.21.15.83:0/169964197 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3844444194 0 0) 0x7f97140b3430 con 0x7f97140d27f0 2024-03-06T07:29:41.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.644+0000 7f9712ffd640 1 -- 172.21.15.83:0/169964197 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9700003520 con 0x7f97140d27f0 2024-03-06T07:29:41.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.644+0000 7f970affd640 1 --2- 172.21.15.83:0/169964197 >> v2:172.21.15.183:3300/0 conn(0x7f97140cf830 0x7f97140d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.644+0000 7f97137fe640 1 --2- 172.21.15.83:0/169964197 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f971415dfe0 0x7f97140cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.644+0000 7f9712ffd640 1 -- 172.21.15.83:0/169964197 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2476528501 0 0) 0x7f9700003520 con 0x7f97140d27f0 2024-03-06T07:29:41.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.644+0000 7f9712ffd640 1 -- 172.21.15.83:0/169964197 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f971415dfe0 msgr2=0x7f97140cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:41.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.644+0000 7f9712ffd640 1 --2- 172.21.15.83:0/169964197 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f971415dfe0 0x7f97140cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.644+0000 7f9712ffd640 1 -- 172.21.15.83:0/169964197 >> v2:172.21.15.183:3300/0 conn(0x7f97140cf830 msgr2=0x7f97140d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:41.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.644+0000 7f9712ffd640 1 --2- 172.21.15.83:0/169964197 >> v2:172.21.15.183:3300/0 conn(0x7f97140cf830 0x7f97140d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.644+0000 7f9712ffd640 1 -- 172.21.15.83:0/169964197 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97140b3710 con 0x7f97140d27f0 2024-03-06T07:29:41.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.644+0000 7f9712ffd640 1 -- 172.21.15.83:0/169964197 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f970c0019f0 con 0x7f97140d27f0 2024-03-06T07:29:41.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.644+0000 7f97137fe640 1 --2- 172.21.15.83:0/169964197 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f971415dfe0 0x7f97140cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:41.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.644+0000 7f970affd640 1 --2- 172.21.15.83:0/169964197 >> v2:172.21.15.183:3300/0 conn(0x7f97140cf830 0x7f97140d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:41.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.644+0000 7f9712ffd640 1 -- 172.21.15.83:0/169964197 >> v1:172.21.15.83:6789/0 conn(0x7f97140d27f0 legacy=0x7f97140d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.644+0000 7f9712ffd640 1 --2- 172.21.15.83:0/169964197 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97000039b0 0x7f9700003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.644+0000 7f9712ffd640 1 --2- 172.21.15.83:0/169964197 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9700004800 0x7f9700004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.645+0000 7f9712ffd640 1 --2- 172.21.15.83:0/169964197 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f971415dfe0 0x7f97000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.645+0000 7f9712ffd640 1 -- 172.21.15.83:0/169964197 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97140b31e0 con 0x7f97000039b0 2024-03-06T07:29:41.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.645+0000 7f9712ffd640 1 -- 172.21.15.83:0/169964197 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f970000ba70 con 0x7f9700004800 2024-03-06T07:29:41.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.645+0000 7f9712ffd640 1 -- 172.21.15.83:0/169964197 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f970000bc20 con 0x7f971415dfe0 2024-03-06T07:29:41.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.645+0000 7f97137fe640 1 --2- 172.21.15.83:0/169964197 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97000039b0 0x7f9700003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.645+0000 7f9713fff640 1 --2- 172.21.15.83:0/169964197 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f971415dfe0 0x7f97000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.645+0000 7f970affd640 1 --2- 172.21.15.83:0/169964197 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9700004800 0x7f9700004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.645+0000 7f97137fe640 1 -- 172.21.15.83:0/169964197 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f971415dfe0 msgr2=0x7f97000051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:41.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.645+0000 7f97137fe640 1 --2- 172.21.15.83:0/169964197 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f971415dfe0 0x7f97000051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.645+0000 7f97137fe640 1 -- 172.21.15.83:0/169964197 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9700004800 msgr2=0x7f9700004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:41.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.645+0000 7f97137fe640 1 --2- 172.21.15.83:0/169964197 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9700004800 0x7f9700004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.645+0000 7f97137fe640 1 -- 172.21.15.83:0/169964197 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9700003520 con 0x7f97000039b0 2024-03-06T07:29:41.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.645+0000 7f970affd640 1 --2- 172.21.15.83:0/169964197 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9700004800 0x7f9700004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:41.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.646+0000 7f97137fe640 1 --2- 172.21.15.83:0/169964197 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97000039b0 0x7f9700003dd0 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f9704006fd0 tx=0x7f9700005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3507639f9c1e244c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:41.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.646+0000 7f9712ffd640 1 -- 172.21.15.83:0/169964197 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f970400f040 con 0x7f97000039b0 2024-03-06T07:29:41.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.646+0000 7f9712ffd640 1 -- 172.21.15.83:0/169964197 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9704004230 con 0x7f97000039b0 2024-03-06T07:29:41.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.646+0000 7f9712ffd640 1 -- 172.21.15.83:0/169964197 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9704003610 con 0x7f97000039b0 2024-03-06T07:29:41.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.646+0000 7f971a084640 1 -- 172.21.15.83:0/169964197 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97000039b0 msgr2=0x7f9700003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.646+0000 7f971a084640 1 --2- 172.21.15.83:0/169964197 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97000039b0 0x7f9700003dd0 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f9704006fd0 tx=0x7f9700005830 comp rx=0 tx=0).stop 2024-03-06T07:29:41.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.646+0000 7f9713fff640 1 -- 172.21.15.83:0/169964197 reap_dead start 2024-03-06T07:29:41.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.646+0000 7f971a084640 1 -- 172.21.15.83:0/169964197 shutdown_connections 2024-03-06T07:29:41.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.646+0000 7f971a084640 1 -- 172.21.15.83:0/169964197 >> 172.21.15.83:0/169964197 conn(0x7f97140d3cd0 msgr2=0x7f97140d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:41.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.646+0000 7f971a084640 1 -- 172.21.15.83:0/169964197 shutdown_connections 2024-03-06T07:29:41.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.646+0000 7f971a084640 1 -- 172.21.15.83:0/169964197 wait complete. 2024-03-06T07:29:41.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.647+0000 7f971a084640 1 Processor -- start 2024-03-06T07:29:41.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.648+0000 7f971a084640 1 -- start start 2024-03-06T07:29:41.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.648+0000 7f971a084640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97000039b0 0x7f97140cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.648+0000 7f971a084640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97141f7170 0x7f97141f75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.648+0000 7f971a084640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97141f3b80 0x7f97141f4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.648+0000 7f971a084640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97140d1c70 con 0x7f97141f3b80 2024-03-06T07:29:41.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.648+0000 7f971a084640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97140b3d60 con 0x7f97141f7170 2024-03-06T07:29:41.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.648+0000 7f971a084640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97140d1df0 con 0x7f97000039b0 2024-03-06T07:29:41.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.648+0000 7f9713fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97141f3b80 0x7f97141f4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.648+0000 7f9713fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97141f3b80 0x7f97141f4030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34204/0 (socket says 172.21.15.83:34204) 2024-03-06T07:29:41.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.649+0000 7f9713fff640 1 -- 172.21.15.83:0/697061292 learned_addr learned my addr 172.21.15.83:0/697061292 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:41.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.649+0000 7f97137fe640 1 --2- 172.21.15.83:0/697061292 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97000039b0 0x7f97140cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.649+0000 7f970affd640 1 --2- 172.21.15.83:0/697061292 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97141f7170 0x7f97141f75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.649+0000 7f9713fff640 1 -- 172.21.15.83:0/697061292 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97000039b0 msgr2=0x7f97140cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.649+0000 7f9713fff640 1 --2- 172.21.15.83:0/697061292 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97000039b0 0x7f97140cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.649+0000 7f9713fff640 1 -- 172.21.15.83:0/697061292 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97141f7170 msgr2=0x7f97141f75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.649+0000 7f9713fff640 1 --2- 172.21.15.83:0/697061292 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97141f7170 0x7f97141f75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.649+0000 7f9713fff640 1 -- 172.21.15.83:0/697061292 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f971415cdb0 con 0x7f97141f3b80 2024-03-06T07:29:41.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.649+0000 7f970affd640 1 --2- 172.21.15.83:0/697061292 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97141f7170 0x7f97141f75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:41.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.649+0000 7f9713fff640 1 --2- 172.21.15.83:0/697061292 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97141f3b80 0x7f97141f4030 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f971408cb30 tx=0x7f970c007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:41.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.649+0000 7f9710ff9640 1 -- 172.21.15.83:0/697061292 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f970c017020 con 0x7f97141f3b80 2024-03-06T07:29:41.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.649+0000 7f9710ff9640 1 -- 172.21.15.83:0/697061292 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f970c00ab90 con 0x7f97141f3b80 2024-03-06T07:29:41.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.649+0000 7f9710ff9640 1 -- 172.21.15.83:0/697061292 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f970c0133f0 con 0x7f97141f3b80 2024-03-06T07:29:41.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.649+0000 7f971a084640 1 -- 172.21.15.83:0/697061292 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97140cb710 con 0x7f97141f3b80 2024-03-06T07:29:41.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.649+0000 7f971a084640 1 -- 172.21.15.83:0/697061292 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97140cbcd0 con 0x7f97141f3b80 2024-03-06T07:29:41.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.650+0000 7f971a084640 1 -- 172.21.15.83:0/697061292 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97140b6650 con 0x7f97141f3b80 2024-03-06T07:29:41.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.650+0000 7f9710ff9640 1 -- 172.21.15.83:0/697061292 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f970c01a020 con 0x7f97141f3b80 2024-03-06T07:29:41.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.651+0000 7f9710ff9640 1 --2- 172.21.15.83:0/697061292 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f96f406b500 0x7f96f406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.651+0000 7f97137fe640 1 --2- 172.21.15.83:0/697061292 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f96f406b500 0x7f96f406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.651+0000 7f9710ff9640 1 -- 172.21.15.83:0/697061292 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f970c08a930 con 0x7f97141f3b80 2024-03-06T07:29:41.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.651+0000 7f97137fe640 1 --2- 172.21.15.83:0/697061292 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f96f406b500 0x7f96f406d9c0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f9704035000 tx=0x7f9704031000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:41.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:41.655+0000 7f9710ff9640 1 -- 172.21.15.83:0/697061292 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f97140b6650 con 0x7f97141f3b80 2024-03-06T07:29:42.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:42.064+0000 7f971a084640 1 -- 172.21.15.83:0/697061292 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f971404f330 con 0x7f97141f3b80 2024-03-06T07:29:42.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:42.064+0000 7f9710ff9640 1 -- 172.21.15.83:0/697061292 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+3415 (secure 0 0 0) 0x7f970c021020 con 0x7f97141f3b80 2024-03-06T07:29:42.066 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:29:42.067 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":81,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:26.073593+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:29:42.067 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 81 2024-03-06T07:29:42.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:42.067+0000 7f971a084640 1 -- 172.21.15.83:0/697061292 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f96f406b500 msgr2=0x7f96f406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:42.067+0000 7f971a084640 1 --2- 172.21.15.83:0/697061292 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f96f406b500 0x7f96f406d9c0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f9704035000 tx=0x7f9704031000 comp rx=0 tx=0).stop 2024-03-06T07:29:42.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:42.067+0000 7f971a084640 1 -- 172.21.15.83:0/697061292 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97141f3b80 msgr2=0x7f97141f4030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:42.067+0000 7f971a084640 1 --2- 172.21.15.83:0/697061292 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97141f3b80 0x7f97141f4030 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f971408cb30 tx=0x7f970c007bd0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:42.067+0000 7f971a084640 1 -- 172.21.15.83:0/697061292 shutdown_connections 2024-03-06T07:29:42.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:42.067+0000 7f971a084640 1 --2- 172.21.15.83:0/697061292 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f96f406b500 0x7f96f406d9c0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:42.067+0000 7f971a084640 1 --2- 172.21.15.83:0/697061292 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97141f3b80 0x7f97141f4030 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:42.067+0000 7f971a084640 1 --2- 172.21.15.83:0/697061292 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97141f7170 0x7f97141f75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:42.067+0000 7f971a084640 1 --2- 172.21.15.83:0/697061292 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97000039b0 0x7f97140cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:42.067+0000 7f971a084640 1 -- 172.21.15.83:0/697061292 >> 172.21.15.83:0/697061292 conn(0x7f97140d3cd0 msgr2=0x7f97140b48b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:42.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:42.067+0000 7f971a084640 1 -- 172.21.15.83:0/697061292 shutdown_connections 2024-03-06T07:29:42.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:42.067+0000 7f971a084640 1 -- 172.21.15.83:0/697061292 wait complete. 2024-03-06T07:29:42.281 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:26.073593+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:29:42.281 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:29:42.985 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:43.281 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:43.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.382+0000 7f38cbf1e640 1 Processor -- start 2024-03-06T07:29:43.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.382+0000 7f38cbf1e640 1 -- start start 2024-03-06T07:29:43.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.383+0000 7f38cbf1e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f38c40cdeb0 0x7f38c415b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.383+0000 7f38cbf1e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c4071f40 con 0x7f38c41515e0 2024-03-06T07:29:43.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.383+0000 7f38cbf1e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c4070a70 con 0x7f38c40cce40 2024-03-06T07:29:43.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.383+0000 7f38cbf1e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38c40cd2f0 con 0x7f38c40cdeb0 2024-03-06T07:29:43.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.383+0000 7f38c9cba640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f38c41515e0 0x7f38c4071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:47796/0 (socket says 172.21.15.83:47796) 2024-03-06T07:29:43.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.383+0000 7f38c9cba640 1 -- 172.21.15.83:0/2249403553 learned_addr learned my addr 172.21.15.83:0/2249403553 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:29:43.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.383+0000 7f38ca4bb640 1 --2- 172.21.15.83:0/2249403553 >> v2:172.21.15.183:3300/0 conn(0x7f38c40cdeb0 0x7f38c415b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.383+0000 7f38c8cb8640 1 -- 172.21.15.83:0/2249403553 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1268506779 0 0) 0x7f38c4071f40 con 0x7f38c41515e0 2024-03-06T07:29:43.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.383+0000 7f38c8cb8640 1 -- 172.21.15.83:0/2249403553 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38b0003520 con 0x7f38c41515e0 2024-03-06T07:29:43.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.383+0000 7f38c8cb8640 1 -- 172.21.15.83:0/2249403553 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1446284252 0 0) 0x7f38b0003520 con 0x7f38c41515e0 2024-03-06T07:29:43.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.383+0000 7f38c8cb8640 1 -- 172.21.15.83:0/2249403553 >> v2:172.21.15.183:3300/0 conn(0x7f38c40cdeb0 msgr2=0x7f38c415b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:43.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.384+0000 7f38c8cb8640 1 --2- 172.21.15.83:0/2249403553 >> v2:172.21.15.183:3300/0 conn(0x7f38c40cdeb0 0x7f38c415b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.384+0000 7f38c8cb8640 1 -- 172.21.15.83:0/2249403553 >> v1:172.21.15.183:6789/0 conn(0x7f38c40cce40 legacy=0x7f38c40cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.384+0000 7f38c8cb8640 1 -- 172.21.15.83:0/2249403553 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c4070d50 con 0x7f38c41515e0 2024-03-06T07:29:43.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.384+0000 7f38c8cb8640 1 -- 172.21.15.83:0/2249403553 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f38b4001690 con 0x7f38c41515e0 2024-03-06T07:29:43.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.384+0000 7f38c8cb8640 1 -- 172.21.15.83:0/2249403553 >> v1:172.21.15.83:6789/0 conn(0x7f38c41515e0 legacy=0x7f38c4071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.384+0000 7f38c8cb8640 1 --2- 172.21.15.83:0/2249403553 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f38b00039b0 0x7f38b0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.384+0000 7f38c8cb8640 1 --2- 172.21.15.83:0/2249403553 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f38b0004800 0x7f38b0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.384+0000 7f38c8cb8640 1 --2- 172.21.15.83:0/2249403553 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f38b00051c0 0x7f38b000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.384+0000 7f38c8cb8640 1 -- 172.21.15.83:0/2249403553 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c40cd2f0 con 0x7f38b00039b0 2024-03-06T07:29:43.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.384+0000 7f38c8cb8640 1 -- 172.21.15.83:0/2249403553 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b000c060 con 0x7f38b0004800 2024-03-06T07:29:43.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.384+0000 7f38c8cb8640 1 -- 172.21.15.83:0/2249403553 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38b000c260 con 0x7f38b00051c0 2024-03-06T07:29:43.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.384+0000 7f38c9cba640 1 --2- 172.21.15.83:0/2249403553 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f38b00039b0 0x7f38b0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.384+0000 7f38ca4bb640 1 --2- 172.21.15.83:0/2249403553 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f38b00051c0 0x7f38b000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.384+0000 7f38c94b9640 1 --2- 172.21.15.83:0/2249403553 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f38b0004800 0x7f38b0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.384+0000 7f38c9cba640 1 -- 172.21.15.83:0/2249403553 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f38b00051c0 msgr2=0x7f38b000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:43.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.384+0000 7f38c9cba640 1 --2- 172.21.15.83:0/2249403553 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f38b00051c0 0x7f38b000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.384+0000 7f38c9cba640 1 -- 172.21.15.83:0/2249403553 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f38b0004800 msgr2=0x7f38b0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:43.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.384+0000 7f38c9cba640 1 --2- 172.21.15.83:0/2249403553 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f38b0004800 0x7f38b0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.384+0000 7f38c9cba640 1 -- 172.21.15.83:0/2249403553 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38b0003520 con 0x7f38b00039b0 2024-03-06T07:29:43.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.384+0000 7f38ca4bb640 1 -- 172.21.15.83:0/2249403553 reap_dead start 2024-03-06T07:29:43.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.385+0000 7f38c9cba640 1 --2- 172.21.15.83:0/2249403553 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f38b00039b0 0x7f38b0003dd0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f38b400c210 tx=0x7f38b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82e474bebdc98c5f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.385+0000 7f38c8cb8640 1 -- 172.21.15.83:0/2249403553 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b4007d10 con 0x7f38b00039b0 2024-03-06T07:29:43.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.385+0000 7f38c8cb8640 1 -- 172.21.15.83:0/2249403553 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38b4030070 con 0x7f38b00039b0 2024-03-06T07:29:43.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.385+0000 7f38c8cb8640 1 -- 172.21.15.83:0/2249403553 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b4005d70 con 0x7f38b00039b0 2024-03-06T07:29:43.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.385+0000 7f38cbf1e640 1 -- 172.21.15.83:0/2249403553 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f38b00039b0 msgr2=0x7f38b0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.385+0000 7f38cbf1e640 1 --2- 172.21.15.83:0/2249403553 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f38b00039b0 0x7f38b0003dd0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f38b400c210 tx=0x7f38b0005830 comp rx=0 tx=0).stop 2024-03-06T07:29:43.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.385+0000 7f38cbf1e640 1 -- 172.21.15.83:0/2249403553 shutdown_connections 2024-03-06T07:29:43.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.385+0000 7f38cbf1e640 1 --2- 172.21.15.83:0/2249403553 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f38b00039b0 0x7f38b0003dd0 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.385+0000 7f38cbf1e640 1 -- 172.21.15.83:0/2249403553 >> 172.21.15.83:0/2249403553 conn(0x7f38c40cf8a0 msgr2=0x7f38c40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:43.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.385+0000 7f38cbf1e640 1 -- 172.21.15.83:0/2249403553 shutdown_connections 2024-03-06T07:29:43.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.385+0000 7f38cbf1e640 1 -- 172.21.15.83:0/2249403553 wait complete. 2024-03-06T07:29:43.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.387+0000 7f38cbf1e640 1 Processor -- start 2024-03-06T07:29:43.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.387+0000 7f38cbf1e640 1 -- start start 2024-03-06T07:29:43.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.387+0000 7f38cbf1e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f38b00039b0 0x7f38c415b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.387+0000 7f38cbf1e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f38c413ae10 0x7f38c406d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.387+0000 7f38cbf1e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f38c413b270 0x7f38c41593c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.387+0000 7f38cbf1e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c415b430 con 0x7f38c413ae10 2024-03-06T07:29:43.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.387+0000 7f38cbf1e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c4071310 con 0x7f38b00039b0 2024-03-06T07:29:43.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.387+0000 7f38cbf1e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38c415b5b0 con 0x7f38c413b270 2024-03-06T07:29:43.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.387+0000 7f38c94b9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f38c413ae10 0x7f38c406d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.387+0000 7f38c94b9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f38c413ae10 0x7f38c406d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34220/0 (socket says 172.21.15.83:34220) 2024-03-06T07:29:43.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.388+0000 7f38c94b9640 1 -- 172.21.15.83:0/3935785609 learned_addr learned my addr 172.21.15.83:0/3935785609 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:43.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.388+0000 7f38ca4bb640 1 --2- 172.21.15.83:0/3935785609 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f38c413b270 0x7f38c41593c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.388+0000 7f38c9cba640 1 --2- 172.21.15.83:0/3935785609 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f38b00039b0 0x7f38c415b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.388+0000 7f38c94b9640 1 -- 172.21.15.83:0/3935785609 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f38c413b270 msgr2=0x7f38c41593c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.388+0000 7f38c94b9640 1 --2- 172.21.15.83:0/3935785609 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f38c413b270 0x7f38c41593c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.388+0000 7f38c94b9640 1 -- 172.21.15.83:0/3935785609 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f38b00039b0 msgr2=0x7f38c415b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.388+0000 7f38c94b9640 1 --2- 172.21.15.83:0/3935785609 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f38b00039b0 0x7f38c415b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.388+0000 7f38c94b9640 1 -- 172.21.15.83:0/3935785609 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c413b960 con 0x7f38c413ae10 2024-03-06T07:29:43.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.388+0000 7f38c9cba640 1 --2- 172.21.15.83:0/3935785609 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f38b00039b0 0x7f38c415b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:43.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.388+0000 7f38c94b9640 1 --2- 172.21.15.83:0/3935785609 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f38c413ae10 0x7f38c406d4f0 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f38bc00e4b0 tx=0x7f38bc00e970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.388+0000 7f38ba7fc640 1 -- 172.21.15.83:0/3935785609 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38bc0052e0 con 0x7f38c413ae10 2024-03-06T07:29:43.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.388+0000 7f38ba7fc640 1 -- 172.21.15.83:0/3935785609 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38bc018070 con 0x7f38c413ae10 2024-03-06T07:29:43.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.388+0000 7f38ba7fc640 1 -- 172.21.15.83:0/3935785609 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38bc003830 con 0x7f38c413ae10 2024-03-06T07:29:43.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.388+0000 7f38cbf1e640 1 -- 172.21.15.83:0/3935785609 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38c4153020 con 0x7f38c413ae10 2024-03-06T07:29:43.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.388+0000 7f38cbf1e640 1 -- 172.21.15.83:0/3935785609 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38c41535e0 con 0x7f38c413ae10 2024-03-06T07:29:43.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.389+0000 7f38cbf1e640 1 -- 172.21.15.83:0/3935785609 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38c4068840 con 0x7f38c413ae10 2024-03-06T07:29:43.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.390+0000 7f38ba7fc640 1 -- 172.21.15.83:0/3935785609 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f38bc0039d0 con 0x7f38c413ae10 2024-03-06T07:29:43.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.390+0000 7f38ba7fc640 1 --2- 172.21.15.83:0/3935785609 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f38a806b4d0 0x7f38a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.390+0000 7f38c9cba640 1 --2- 172.21.15.83:0/3935785609 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f38a806b4d0 0x7f38a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.390+0000 7f38ba7fc640 1 -- 172.21.15.83:0/3935785609 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f38bc089e60 con 0x7f38c413ae10 2024-03-06T07:29:43.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.391+0000 7f38c9cba640 1 --2- 172.21.15.83:0/3935785609 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f38a806b4d0 0x7f38a806d990 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f38b4034000 tx=0x7f38b4039040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.395+0000 7f38ba7fc640 1 -- 172.21.15.83:0/3935785609 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f38bc058ca0 con 0x7f38c413ae10 2024-03-06T07:29:43.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.608+0000 7f38cbf1e640 1 -- 172.21.15.83:0/3935785609 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38c404f330 con 0x7f38c413ae10 2024-03-06T07:29:43.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.609+0000 7f38ba7fc640 1 -- 172.21.15.83:0/3935785609 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+3415 (secure 0 0 0) 0x7f38bc0585a0 con 0x7f38c413ae10 2024-03-06T07:29:43.611 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:29:43.611 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":81,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:26.073593+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:29:43.611 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 81 2024-03-06T07:29:43.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.610+0000 7f38cbf1e640 1 -- 172.21.15.83:0/3935785609 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f38a806b4d0 msgr2=0x7f38a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.610+0000 7f38cbf1e640 1 --2- 172.21.15.83:0/3935785609 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f38a806b4d0 0x7f38a806d990 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f38b4034000 tx=0x7f38b4039040 comp rx=0 tx=0).stop 2024-03-06T07:29:43.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.610+0000 7f38cbf1e640 1 -- 172.21.15.83:0/3935785609 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f38c413ae10 msgr2=0x7f38c406d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.610+0000 7f38cbf1e640 1 --2- 172.21.15.83:0/3935785609 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f38c413ae10 0x7f38c406d4f0 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f38bc00e4b0 tx=0x7f38bc00e970 comp rx=0 tx=0).stop 2024-03-06T07:29:43.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.611+0000 7f38cbf1e640 1 -- 172.21.15.83:0/3935785609 shutdown_connections 2024-03-06T07:29:43.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.611+0000 7f38cbf1e640 1 --2- 172.21.15.83:0/3935785609 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f38a806b4d0 0x7f38a806d990 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.611+0000 7f38cbf1e640 1 --2- 172.21.15.83:0/3935785609 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f38c413b270 0x7f38c41593c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.611+0000 7f38cbf1e640 1 --2- 172.21.15.83:0/3935785609 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f38c413ae10 0x7f38c406d4f0 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.611+0000 7f38cbf1e640 1 --2- 172.21.15.83:0/3935785609 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f38b00039b0 0x7f38c415b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.611+0000 7f38cbf1e640 1 -- 172.21.15.83:0/3935785609 >> 172.21.15.83:0/3935785609 conn(0x7f38c40cf8a0 msgr2=0x7f38c40725c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:43.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.611+0000 7f38cbf1e640 1 -- 172.21.15.83:0/3935785609 shutdown_connections 2024-03-06T07:29:43.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:43.611+0000 7f38cbf1e640 1 -- 172.21.15.83:0/3935785609 wait complete. 2024-03-06T07:29:43.824 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:26.073593+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:29:43.824 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:29:44.825 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:44.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.945+0000 7f1f7d35e640 1 Processor -- start 2024-03-06T07:29:44.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.945+0000 7f1f7d35e640 1 -- start start 2024-03-06T07:29:44.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.946+0000 7f1f7d35e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1f780d04a0 0x7f1f780ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.946+0000 7f1f7d35e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f780d0910 con 0x7f1f7815bb20 2024-03-06T07:29:44.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.946+0000 7f1f7d35e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f78161980 con 0x7f1f780cfff0 2024-03-06T07:29:44.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.946+0000 7f1f7d35e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f780d2500 con 0x7f1f780d04a0 2024-03-06T07:29:44.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.946+0000 7f1f76ffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f1f7815bb20 0x7f1f780d1df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:47802/0 (socket says 172.21.15.83:47802) 2024-03-06T07:29:44.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.946+0000 7f1f76ffd640 1 -- 172.21.15.83:0/3294429634 learned_addr learned my addr 172.21.15.83:0/3294429634 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:29:44.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.947+0000 7f1f75ffb640 1 -- 172.21.15.83:0/3294429634 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 568421078 0 0) 0x7f1f780d0910 con 0x7f1f7815bb20 2024-03-06T07:29:44.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.947+0000 7f1f75ffb640 1 -- 172.21.15.83:0/3294429634 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f60003520 con 0x7f1f7815bb20 2024-03-06T07:29:44.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.947+0000 7f1f777fe640 1 --2- 172.21.15.83:0/3294429634 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1f780d04a0 0x7f1f780ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.947+0000 7f1f75ffb640 1 -- 172.21.15.83:0/3294429634 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3665367632 0 0) 0x7f1f60003520 con 0x7f1f7815bb20 2024-03-06T07:29:44.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.947+0000 7f1f75ffb640 1 -- 172.21.15.83:0/3294429634 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1f780d04a0 msgr2=0x7f1f780ccfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:44.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.947+0000 7f1f75ffb640 1 --2- 172.21.15.83:0/3294429634 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1f780d04a0 0x7f1f780ccfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.947+0000 7f1f75ffb640 1 -- 172.21.15.83:0/3294429634 >> v1:172.21.15.183:6789/0 conn(0x7f1f780cfff0 legacy=0x7f1f780cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.947+0000 7f1f75ffb640 1 -- 172.21.15.83:0/3294429634 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f78161c60 con 0x7f1f7815bb20 2024-03-06T07:29:44.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.947+0000 7f1f75ffb640 1 -- 172.21.15.83:0/3294429634 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f1f6c001670 con 0x7f1f7815bb20 2024-03-06T07:29:44.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.947+0000 7f1f75ffb640 1 -- 172.21.15.83:0/3294429634 >> v1:172.21.15.83:6789/0 conn(0x7f1f7815bb20 legacy=0x7f1f780d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.947+0000 7f1f75ffb640 1 --2- 172.21.15.83:0/3294429634 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f600039e0 0x7f1f60003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.947+0000 7f1f75ffb640 1 --2- 172.21.15.83:0/3294429634 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1f780d04a0 0x7f1f60004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.947+0000 7f1f75ffb640 1 --2- 172.21.15.83:0/3294429634 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f60004d40 0x7f1f600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.947+0000 7f1f75ffb640 1 -- 172.21.15.83:0/3294429634 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f780d2500 con 0x7f1f60004d40 2024-03-06T07:29:44.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.947+0000 7f1f75ffb640 1 -- 172.21.15.83:0/3294429634 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6000bb50 con 0x7f1f600039e0 2024-03-06T07:29:44.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.947+0000 7f1f75ffb640 1 -- 172.21.15.83:0/3294429634 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6000bd20 con 0x7f1f780d04a0 2024-03-06T07:29:44.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.947+0000 7f1f777fe640 1 --2- 172.21.15.83:0/3294429634 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f60004d40 0x7f1f600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.947+0000 7f1f76ffd640 1 --2- 172.21.15.83:0/3294429634 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f600039e0 0x7f1f60003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.947+0000 7f1f767fc640 1 --2- 172.21.15.83:0/3294429634 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1f780d04a0 0x7f1f60004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.947+0000 7f1f777fe640 1 -- 172.21.15.83:0/3294429634 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1f780d04a0 msgr2=0x7f1f60004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:44.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.947+0000 7f1f777fe640 1 --2- 172.21.15.83:0/3294429634 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1f780d04a0 0x7f1f60004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.947+0000 7f1f777fe640 1 -- 172.21.15.83:0/3294429634 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f600039e0 msgr2=0x7f1f60003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:44.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.947+0000 7f1f777fe640 1 --2- 172.21.15.83:0/3294429634 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f600039e0 0x7f1f60003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.947+0000 7f1f777fe640 1 -- 172.21.15.83:0/3294429634 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f60003520 con 0x7f1f60004d40 2024-03-06T07:29:44.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.948+0000 7f1f777fe640 1 --2- 172.21.15.83:0/3294429634 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f60004d40 0x7f1f600051c0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f1f70007390 tx=0x7f1f60005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59ec6e34fabe5d28 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:44.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.948+0000 7f1f75ffb640 1 -- 172.21.15.83:0/3294429634 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f70039070 con 0x7f1f60004d40 2024-03-06T07:29:44.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.948+0000 7f1f75ffb640 1 -- 172.21.15.83:0/3294429634 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f7002ccb0 con 0x7f1f60004d40 2024-03-06T07:29:44.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.948+0000 7f1f75ffb640 1 -- 172.21.15.83:0/3294429634 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f70009be0 con 0x7f1f60004d40 2024-03-06T07:29:44.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.949+0000 7f1f7d35e640 1 -- 172.21.15.83:0/3294429634 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f60004d40 msgr2=0x7f1f600051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.949+0000 7f1f7d35e640 1 --2- 172.21.15.83:0/3294429634 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f60004d40 0x7f1f600051c0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f1f70007390 tx=0x7f1f60005830 comp rx=0 tx=0).stop 2024-03-06T07:29:44.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.949+0000 7f1f777fe640 1 -- 172.21.15.83:0/3294429634 reap_dead start 2024-03-06T07:29:44.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.949+0000 7f1f7d35e640 1 -- 172.21.15.83:0/3294429634 shutdown_connections 2024-03-06T07:29:44.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.949+0000 7f1f7d35e640 1 -- 172.21.15.83:0/3294429634 >> 172.21.15.83:0/3294429634 conn(0x7f1f7809a9f0 msgr2=0x7f1f7809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:44.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.949+0000 7f1f7d35e640 1 -- 172.21.15.83:0/3294429634 shutdown_connections 2024-03-06T07:29:44.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.949+0000 7f1f7d35e640 1 -- 172.21.15.83:0/3294429634 wait complete. 2024-03-06T07:29:44.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.951+0000 7f1f7d35e640 1 Processor -- start 2024-03-06T07:29:44.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.951+0000 7f1f7d35e640 1 -- start start 2024-03-06T07:29:44.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.952+0000 7f1f7d35e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f78158b80 0x7f1f780b4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.952+0000 7f1f7d35e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f780c63c0 0x7f1f780af940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.952+0000 7f1f7d35e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1f780c6840 0x7f1f780afe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.952+0000 7f1f7d35e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f78162500 con 0x7f1f780c63c0 2024-03-06T07:29:44.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.952+0000 7f1f7d35e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f78162220 con 0x7f1f78158b80 2024-03-06T07:29:44.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.952+0000 7f1f7d35e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f780d1980 con 0x7f1f780c6840 2024-03-06T07:29:44.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.952+0000 7f1f767fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f780c63c0 0x7f1f780af940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.952+0000 7f1f767fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f780c63c0 0x7f1f780af940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34242/0 (socket says 172.21.15.83:34242) 2024-03-06T07:29:44.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.952+0000 7f1f767fc640 1 -- 172.21.15.83:0/2153051687 learned_addr learned my addr 172.21.15.83:0/2153051687 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:44.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.952+0000 7f1f777fe640 1 --2- 172.21.15.83:0/2153051687 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1f780c6840 0x7f1f780afe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.952+0000 7f1f76ffd640 1 --2- 172.21.15.83:0/2153051687 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f78158b80 0x7f1f780b4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.952+0000 7f1f767fc640 1 -- 172.21.15.83:0/2153051687 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1f780c6840 msgr2=0x7f1f780afe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.952+0000 7f1f767fc640 1 --2- 172.21.15.83:0/2153051687 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1f780c6840 0x7f1f780afe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.952+0000 7f1f767fc640 1 -- 172.21.15.83:0/2153051687 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f78158b80 msgr2=0x7f1f780b4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.952+0000 7f1f767fc640 1 --2- 172.21.15.83:0/2153051687 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f78158b80 0x7f1f780b4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.952+0000 7f1f767fc640 1 -- 172.21.15.83:0/2153051687 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f780c6da0 con 0x7f1f780c63c0 2024-03-06T07:29:44.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.953+0000 7f1f767fc640 1 --2- 172.21.15.83:0/2153051687 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f780c63c0 0x7f1f780af940 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f1f6800f650 tx=0x7f1f6800fb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:44.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.953+0000 7f1f5f7fe640 1 -- 172.21.15.83:0/2153051687 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f680050d0 con 0x7f1f780c63c0 2024-03-06T07:29:44.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.953+0000 7f1f5f7fe640 1 -- 172.21.15.83:0/2153051687 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f68018070 con 0x7f1f780c63c0 2024-03-06T07:29:44.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.953+0000 7f1f7d35e640 1 -- 172.21.15.83:0/2153051687 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f780ca2e0 con 0x7f1f780c63c0 2024-03-06T07:29:44.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.953+0000 7f1f5f7fe640 1 -- 172.21.15.83:0/2153051687 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6800d730 con 0x7f1f780c63c0 2024-03-06T07:29:44.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.953+0000 7f1f7d35e640 1 -- 172.21.15.83:0/2153051687 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f780ca8a0 con 0x7f1f780c63c0 2024-03-06T07:29:44.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.954+0000 7f1f7d35e640 1 -- 172.21.15.83:0/2153051687 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f780b6a80 con 0x7f1f780c63c0 2024-03-06T07:29:44.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.954+0000 7f1f5f7fe640 1 -- 172.21.15.83:0/2153051687 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1f6800d8d0 con 0x7f1f780c63c0 2024-03-06T07:29:44.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.955+0000 7f1f5f7fe640 1 --2- 172.21.15.83:0/2153051687 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1f6406b4d0 0x7f1f6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.955+0000 7f1f5f7fe640 1 -- 172.21.15.83:0/2153051687 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1f68089f80 con 0x7f1f780c63c0 2024-03-06T07:29:44.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.955+0000 7f1f76ffd640 1 --2- 172.21.15.83:0/2153051687 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1f6406b4d0 0x7f1f6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.955+0000 7f1f76ffd640 1 --2- 172.21.15.83:0/2153051687 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1f6406b4d0 0x7f1f6406d990 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c004840 tx=0x7f1f6c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:44.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:44.959+0000 7f1f5f7fe640 1 -- 172.21.15.83:0/2153051687 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1f780b6a80 con 0x7f1f780c63c0 2024-03-06T07:29:44.987 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:45.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:45.174+0000 7f1f7d35e640 1 -- 172.21.15.83:0/2153051687 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f7804f330 con 0x7f1f780c63c0 2024-03-06T07:29:45.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:45.174+0000 7f1f5f7fe640 1 -- 172.21.15.83:0/2153051687 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+3415 (secure 0 0 0) 0x7f1f68058e50 con 0x7f1f780c63c0 2024-03-06T07:29:45.176 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:29:45.177 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":81,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:26.073593+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:29:45.177 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 81 2024-03-06T07:29:45.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:45.176+0000 7f1f7d35e640 1 -- 172.21.15.83:0/2153051687 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1f6406b4d0 msgr2=0x7f1f6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:45.176+0000 7f1f7d35e640 1 --2- 172.21.15.83:0/2153051687 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1f6406b4d0 0x7f1f6406d990 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c004840 tx=0x7f1f6c009040 comp rx=0 tx=0).stop 2024-03-06T07:29:45.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:45.176+0000 7f1f7d35e640 1 -- 172.21.15.83:0/2153051687 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f780c63c0 msgr2=0x7f1f780af940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:45.176+0000 7f1f7d35e640 1 --2- 172.21.15.83:0/2153051687 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f780c63c0 0x7f1f780af940 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f1f6800f650 tx=0x7f1f6800fb10 comp rx=0 tx=0).stop 2024-03-06T07:29:45.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:45.176+0000 7f1f7d35e640 1 -- 172.21.15.83:0/2153051687 shutdown_connections 2024-03-06T07:29:45.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:45.176+0000 7f1f7d35e640 1 --2- 172.21.15.83:0/2153051687 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1f6406b4d0 0x7f1f6406d990 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:45.176+0000 7f1f7d35e640 1 --2- 172.21.15.83:0/2153051687 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1f780c6840 0x7f1f780afe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:45.176+0000 7f1f7d35e640 1 --2- 172.21.15.83:0/2153051687 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f780c63c0 0x7f1f780af940 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:45.176+0000 7f1f7d35e640 1 --2- 172.21.15.83:0/2153051687 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f78158b80 0x7f1f780b4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:45.176+0000 7f1f7d35e640 1 -- 172.21.15.83:0/2153051687 >> 172.21.15.83:0/2153051687 conn(0x7f1f7809a9f0 msgr2=0x7f1f78159f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:45.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:45.176+0000 7f1f7d35e640 1 -- 172.21.15.83:0/2153051687 shutdown_connections 2024-03-06T07:29:45.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:45.176+0000 7f1f7d35e640 1 -- 172.21.15.83:0/2153051687 wait complete. 2024-03-06T07:29:45.390 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:26.073593+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:29:45.390 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:29:46.391 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:46.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.491+0000 7fd19550c640 1 Processor -- start 2024-03-06T07:29:46.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.491+0000 7fd19550c640 1 -- start start 2024-03-06T07:29:46.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.492+0000 7fd19550c640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fd1900d2710 0x7fd1900b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.492+0000 7fd19550c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1900b3430 con 0x7fd1900d22b0 2024-03-06T07:29:46.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.492+0000 7fd19550c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1900b3710 con 0x7fd19015dfe0 2024-03-06T07:29:46.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.492+0000 7fd19550c640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1900b31e0 con 0x7fd1900d2710 2024-03-06T07:29:46.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.492+0000 7fd18e7fc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fd1900d22b0 0x7fd1900b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:47818/0 (socket says 172.21.15.83:47818) 2024-03-06T07:29:46.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.492+0000 7fd18e7fc640 1 -- 172.21.15.83:0/698810368 learned_addr learned my addr 172.21.15.83:0/698810368 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:29:46.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.493+0000 7fd18f7fe640 1 --2- 172.21.15.83:0/698810368 >> v2:172.21.15.183:3300/0 conn(0x7fd1900d2710 0x7fd1900b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.493+0000 7fd18dffb640 1 -- 172.21.15.83:0/698810368 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3936127763 0 0) 0x7fd1900b3430 con 0x7fd1900d22b0 2024-03-06T07:29:46.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.493+0000 7fd18dffb640 1 -- 172.21.15.83:0/698810368 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd17c003520 con 0x7fd1900d22b0 2024-03-06T07:29:46.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.493+0000 7fd18f7fe640 1 -- 172.21.15.83:0/698810368 >> v1:172.21.15.183:6789/0 conn(0x7fd19015dfe0 legacy=0x7fd1900cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.493+0000 7fd18f7fe640 1 -- 172.21.15.83:0/698810368 >> v1:172.21.15.83:6789/0 conn(0x7fd1900d22b0 legacy=0x7fd1900b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.493+0000 7fd18dffb640 1 -- 172.21.15.83:0/698810368 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3112532098 0 0) 0x7fd17c003520 con 0x7fd1900d22b0 2024-03-06T07:29:46.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.493+0000 7fd18f7fe640 1 -- 172.21.15.83:0/698810368 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1900b39f0 con 0x7fd1900d2710 2024-03-06T07:29:46.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.494+0000 7fd18f7fe640 1 --2- 172.21.15.83:0/698810368 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd1900d2710 0x7fd1900b4900 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fd1800097d0 tx=0x7fd18002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8997ac1a62eba7d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:46.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.494+0000 7fd18dffb640 1 -- 172.21.15.83:0/698810368 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd180038070 con 0x7fd1900d2710 2024-03-06T07:29:46.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.494+0000 7fd18dffb640 1 -- 172.21.15.83:0/698810368 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd180031500 con 0x7fd1900d2710 2024-03-06T07:29:46.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.494+0000 7fd18dffb640 1 -- 172.21.15.83:0/698810368 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd178001670 con 0x7fd1900d2710 2024-03-06T07:29:46.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.495+0000 7fd19550c640 1 -- 172.21.15.83:0/698810368 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd1900d2710 msgr2=0x7fd1900b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.495+0000 7fd19550c640 1 --2- 172.21.15.83:0/698810368 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd1900d2710 0x7fd1900b4900 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fd1800097d0 tx=0x7fd18002b820 comp rx=0 tx=0).stop 2024-03-06T07:29:46.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.495+0000 7fd19550c640 1 -- 172.21.15.83:0/698810368 shutdown_connections 2024-03-06T07:29:46.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.495+0000 7fd19550c640 1 --2- 172.21.15.83:0/698810368 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd1900d2710 0x7fd1900b4900 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.495+0000 7fd19550c640 1 -- 172.21.15.83:0/698810368 >> 172.21.15.83:0/698810368 conn(0x7fd1900d3cd0 msgr2=0x7fd1900d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:46.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.495+0000 7fd19550c640 1 -- 172.21.15.83:0/698810368 shutdown_connections 2024-03-06T07:29:46.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.495+0000 7fd19550c640 1 -- 172.21.15.83:0/698810368 wait complete. 2024-03-06T07:29:46.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.496+0000 7fd19550c640 1 Processor -- start 2024-03-06T07:29:46.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.496+0000 7fd19550c640 1 -- start start 2024-03-06T07:29:46.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.497+0000 7fd19550c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd1900d22b0 0x7fd1900b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.497+0000 7fd19550c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1900d2710 0x7fd1900cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.497+0000 7fd19550c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd19015dfe0 0x7fd1900ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.497+0000 7fd19550c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1900cfa00 con 0x7fd1900d2710 2024-03-06T07:29:46.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.497+0000 7fd19550c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1900b3d60 con 0x7fd1900d22b0 2024-03-06T07:29:46.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.497+0000 7fd19550c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1900d1c70 con 0x7fd19015dfe0 2024-03-06T07:29:46.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.497+0000 7fd18e7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1900d2710 0x7fd1900cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.497+0000 7fd18e7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1900d2710 0x7fd1900cc9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34258/0 (socket says 172.21.15.83:34258) 2024-03-06T07:29:46.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.497+0000 7fd18e7fc640 1 -- 172.21.15.83:0/1524319360 learned_addr learned my addr 172.21.15.83:0/1524319360 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:46.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.497+0000 7fd18effd640 1 --2- 172.21.15.83:0/1524319360 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd1900d22b0 0x7fd1900b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.497+0000 7fd18f7fe640 1 --2- 172.21.15.83:0/1524319360 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd19015dfe0 0x7fd1900ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.497+0000 7fd18e7fc640 1 -- 172.21.15.83:0/1524319360 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd19015dfe0 msgr2=0x7fd1900ccf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.497+0000 7fd18e7fc640 1 --2- 172.21.15.83:0/1524319360 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd19015dfe0 0x7fd1900ccf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.497+0000 7fd18e7fc640 1 -- 172.21.15.83:0/1524319360 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd1900d22b0 msgr2=0x7fd1900b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.497+0000 7fd18e7fc640 1 --2- 172.21.15.83:0/1524319360 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd1900d22b0 0x7fd1900b68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.497+0000 7fd18e7fc640 1 -- 172.21.15.83:0/1524319360 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1900b71b0 con 0x7fd1900d2710 2024-03-06T07:29:46.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.497+0000 7fd18e7fc640 1 --2- 172.21.15.83:0/1524319360 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1900d2710 0x7fd1900cc9d0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7fd17800c730 tx=0x7fd17800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:46.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.497+0000 7fd1877fe640 1 -- 172.21.15.83:0/1524319360 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd178017070 con 0x7fd1900d2710 2024-03-06T07:29:46.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.497+0000 7fd1877fe640 1 -- 172.21.15.83:0/1524319360 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd178005af0 con 0x7fd1900d2710 2024-03-06T07:29:46.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.497+0000 7fd19550c640 1 -- 172.21.15.83:0/1524319360 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1900b7490 con 0x7fd1900d2710 2024-03-06T07:29:46.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.497+0000 7fd1877fe640 1 -- 172.21.15.83:0/1524319360 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd178007070 con 0x7fd1900d2710 2024-03-06T07:29:46.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.497+0000 7fd19550c640 1 -- 172.21.15.83:0/1524319360 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1900b7a50 con 0x7fd1900d2710 2024-03-06T07:29:46.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.499+0000 7fd1877fe640 1 -- 172.21.15.83:0/1524319360 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd17801a070 con 0x7fd1900d2710 2024-03-06T07:29:46.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.499+0000 7fd19550c640 1 -- 172.21.15.83:0/1524319360 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd19004f330 con 0x7fd1900d2710 2024-03-06T07:29:46.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.499+0000 7fd1877fe640 1 --2- 172.21.15.83:0/1524319360 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd17006b4d0 0x7fd17006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.499+0000 7fd18effd640 1 --2- 172.21.15.83:0/1524319360 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd17006b4d0 0x7fd17006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.499+0000 7fd1877fe640 1 -- 172.21.15.83:0/1524319360 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd178089870 con 0x7fd1900d2710 2024-03-06T07:29:46.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.500+0000 7fd18effd640 1 --2- 172.21.15.83:0/1524319360 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd17006b4d0 0x7fd17006d990 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fd1880012e0 tx=0x7fd188009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:46.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.503+0000 7fd1877fe640 1 -- 172.21.15.83:0/1524319360 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd178058630 con 0x7fd1900d2710 2024-03-06T07:29:46.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.717+0000 7fd19550c640 1 -- 172.21.15.83:0/1524319360 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd19015d250 con 0x7fd1900d2710 2024-03-06T07:29:46.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.717+0000 7fd1877fe640 1 -- 172.21.15.83:0/1524319360 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+3415 (secure 0 0 0) 0x7fd178057f30 con 0x7fd1900d2710 2024-03-06T07:29:46.719 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:29:46.720 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":81,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:26.073593+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:29:46.720 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 81 2024-03-06T07:29:46.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.719+0000 7fd19550c640 1 -- 172.21.15.83:0/1524319360 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd17006b4d0 msgr2=0x7fd17006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.719+0000 7fd19550c640 1 --2- 172.21.15.83:0/1524319360 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd17006b4d0 0x7fd17006d990 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fd1880012e0 tx=0x7fd188009040 comp rx=0 tx=0).stop 2024-03-06T07:29:46.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.719+0000 7fd19550c640 1 -- 172.21.15.83:0/1524319360 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1900d2710 msgr2=0x7fd1900cc9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.719+0000 7fd19550c640 1 --2- 172.21.15.83:0/1524319360 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1900d2710 0x7fd1900cc9d0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7fd17800c730 tx=0x7fd17800cbf0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.719+0000 7fd19550c640 1 -- 172.21.15.83:0/1524319360 shutdown_connections 2024-03-06T07:29:46.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.719+0000 7fd19550c640 1 --2- 172.21.15.83:0/1524319360 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd17006b4d0 0x7fd17006d990 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.719+0000 7fd19550c640 1 --2- 172.21.15.83:0/1524319360 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd19015dfe0 0x7fd1900ccf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.719+0000 7fd19550c640 1 --2- 172.21.15.83:0/1524319360 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1900d2710 0x7fd1900cc9d0 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.719+0000 7fd19550c640 1 --2- 172.21.15.83:0/1524319360 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd1900d22b0 0x7fd1900b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.719+0000 7fd19550c640 1 -- 172.21.15.83:0/1524319360 >> 172.21.15.83:0/1524319360 conn(0x7fd1900d3cd0 msgr2=0x7fd1900b40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:46.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.719+0000 7fd19550c640 1 -- 172.21.15.83:0/1524319360 shutdown_connections 2024-03-06T07:29:46.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:46.719+0000 7fd19550c640 1 -- 172.21.15.83:0/1524319360 wait complete. 2024-03-06T07:29:46.933 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:26.073593+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:29:46.933 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:29:46.989 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:47.934 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:48.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.036+0000 7f1d0c876640 1 Processor -- start 2024-03-06T07:29:48.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.036+0000 7f1d0c876640 1 -- start start 2024-03-06T07:29:48.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.037+0000 7f1d0c876640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1d0815dfe0 0x7f1d080cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.037+0000 7f1d0c876640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1d080cf830 0x7f1d080d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.037+0000 7f1d0c876640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d080b3430 con 0x7f1d080d27f0 2024-03-06T07:29:48.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.037+0000 7f1d0c876640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d080d2c50 con 0x7f1d080cf830 2024-03-06T07:29:48.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.037+0000 7f1d0c876640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d080b31e0 con 0x7f1d0815dfe0 2024-03-06T07:29:48.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.037+0000 7f1cfdd9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1d080cf830 0x7f1d080d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.037+0000 7f1cfdd9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1d080cf830 0x7f1d080d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:32846/0 (socket says 172.21.15.83:32846) 2024-03-06T07:29:48.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.037+0000 7f1cfdd9b640 1 -- 172.21.15.83:0/4056966483 learned_addr learned my addr 172.21.15.83:0/4056966483 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:48.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.037+0000 7f1d06d9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f1d080d27f0 0x7f1d080d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:42760/0 (socket says 172.21.15.83:42760) 2024-03-06T07:29:48.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.037+0000 7f1d0659c640 1 --2- 172.21.15.83:0/4056966483 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1d0815dfe0 0x7f1d080cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.038+0000 7f1cfdd9b640 1 -- 172.21.15.83:0/4056966483 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1d0815dfe0 msgr2=0x7f1d080cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:48.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.038+0000 7f1cfdd9b640 1 --2- 172.21.15.83:0/4056966483 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1d0815dfe0 0x7f1d080cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.038+0000 7f1cfdd9b640 1 -- 172.21.15.83:0/4056966483 >> v1:172.21.15.83:6789/0 conn(0x7f1d080d27f0 legacy=0x7f1d080d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.038+0000 7f1cfdd9b640 1 -- 172.21.15.83:0/4056966483 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d080b3710 con 0x7f1d080cf830 2024-03-06T07:29:48.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.038+0000 7f1d05d9b640 1 -- 172.21.15.83:0/4056966483 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 928071085 0 0) 0x7f1d080b3430 con 0x7f1d080d27f0 2024-03-06T07:29:48.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.038+0000 7f1cfdd9b640 1 --2- 172.21.15.83:0/4056966483 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d080cf830 0x7f1d080d22b0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f1cf0009870 tx=0x7f1cf002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=759b6025811ebc38 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.038+0000 7f1d05d9b640 1 -- 172.21.15.83:0/4056966483 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf0034070 con 0x7f1d080cf830 2024-03-06T07:29:48.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.038+0000 7f1d05d9b640 1 -- 172.21.15.83:0/4056966483 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cf0030840 con 0x7f1d080cf830 2024-03-06T07:29:48.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.039+0000 7f1d05d9b640 1 -- 172.21.15.83:0/4056966483 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf0030c10 con 0x7f1d080cf830 2024-03-06T07:29:48.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.039+0000 7f1d0c876640 1 -- 172.21.15.83:0/4056966483 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d080cf830 msgr2=0x7f1d080d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.039+0000 7f1d0c876640 1 --2- 172.21.15.83:0/4056966483 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d080cf830 0x7f1d080d22b0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f1cf0009870 tx=0x7f1cf002b980 comp rx=0 tx=0).stop 2024-03-06T07:29:48.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.039+0000 7f1d0c876640 1 -- 172.21.15.83:0/4056966483 shutdown_connections 2024-03-06T07:29:48.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.039+0000 7f1d0c876640 1 --2- 172.21.15.83:0/4056966483 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d080cf830 0x7f1d080d22b0 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.039+0000 7f1d0c876640 1 --2- 172.21.15.83:0/4056966483 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1d0815dfe0 0x7f1d080cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.039+0000 7f1d0c876640 1 -- 172.21.15.83:0/4056966483 >> 172.21.15.83:0/4056966483 conn(0x7f1d080d3cd0 msgr2=0x7f1d080d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:48.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.039+0000 7f1d0c876640 1 -- 172.21.15.83:0/4056966483 shutdown_connections 2024-03-06T07:29:48.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.039+0000 7f1d0c876640 1 -- 172.21.15.83:0/4056966483 wait complete. 2024-03-06T07:29:48.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.040+0000 7f1d0c876640 1 Processor -- start 2024-03-06T07:29:48.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.041+0000 7f1d0c876640 1 -- start start 2024-03-06T07:29:48.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.041+0000 7f1d0c876640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1d080cf830 0x7f1d080b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.041+0000 7f1d0c876640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1d080d27f0 0x7f1d080b8680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.041+0000 7f1d0c876640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d0815dfe0 0x7f1d08150c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.041+0000 7f1d0c876640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d080d1c70 con 0x7f1d0815dfe0 2024-03-06T07:29:48.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.041+0000 7f1d0c876640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d080b3d60 con 0x7f1d080cf830 2024-03-06T07:29:48.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.041+0000 7f1d0c876640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d080d1df0 con 0x7f1d080d27f0 2024-03-06T07:29:48.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.041+0000 7f1d06d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d0815dfe0 0x7f1d08150c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.041+0000 7f1d06d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d0815dfe0 0x7f1d08150c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:32854/0 (socket says 172.21.15.83:32854) 2024-03-06T07:29:48.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.041+0000 7f1d06d9d640 1 -- 172.21.15.83:0/1023970065 learned_addr learned my addr 172.21.15.83:0/1023970065 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:48.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.041+0000 7f1cfdd9b640 1 --2- 172.21.15.83:0/1023970065 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1d080d27f0 0x7f1d080b8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.042+0000 7f1d0659c640 1 --2- 172.21.15.83:0/1023970065 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1d080cf830 0x7f1d080b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.042+0000 7f1d06d9d640 1 -- 172.21.15.83:0/1023970065 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1d080d27f0 msgr2=0x7f1d080b8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.042+0000 7f1d06d9d640 1 --2- 172.21.15.83:0/1023970065 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1d080d27f0 0x7f1d080b8680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.042+0000 7f1d06d9d640 1 -- 172.21.15.83:0/1023970065 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1d080cf830 msgr2=0x7f1d080b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.042+0000 7f1d06d9d640 1 --2- 172.21.15.83:0/1023970065 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1d080cf830 0x7f1d080b68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.042+0000 7f1d06d9d640 1 -- 172.21.15.83:0/1023970065 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d0815a570 con 0x7f1d0815dfe0 2024-03-06T07:29:48.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.042+0000 7f1d06d9d640 1 --2- 172.21.15.83:0/1023970065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d0815dfe0 0x7f1d08150c30 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f1d0000c7a0 tx=0x7f1d0000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.042+0000 7f1cff7fe640 1 -- 172.21.15.83:0/1023970065 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d00018070 con 0x7f1d0815dfe0 2024-03-06T07:29:48.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.042+0000 7f1cff7fe640 1 -- 172.21.15.83:0/1023970065 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d0000a640 con 0x7f1d0815dfe0 2024-03-06T07:29:48.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.042+0000 7f1cff7fe640 1 -- 172.21.15.83:0/1023970065 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0000aab0 con 0x7f1d0815dfe0 2024-03-06T07:29:48.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.042+0000 7f1d0c876640 1 -- 172.21.15.83:0/1023970065 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d080cdc80 con 0x7f1d0815dfe0 2024-03-06T07:29:48.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.042+0000 7f1d0c876640 1 -- 172.21.15.83:0/1023970065 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d080b95a0 con 0x7f1d0815dfe0 2024-03-06T07:29:48.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.043+0000 7f1d0c876640 1 -- 172.21.15.83:0/1023970065 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d0804f330 con 0x7f1d0815dfe0 2024-03-06T07:29:48.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.044+0000 7f1cff7fe640 1 -- 172.21.15.83:0/1023970065 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1d0000ac50 con 0x7f1d0815dfe0 2024-03-06T07:29:48.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.044+0000 7f1cff7fe640 1 --2- 172.21.15.83:0/1023970065 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1ce806b4d0 0x7f1ce806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.044+0000 7f1cff7fe640 1 -- 172.21.15.83:0/1023970065 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1d00089810 con 0x7f1d0815dfe0 2024-03-06T07:29:48.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.044+0000 7f1d0659c640 1 --2- 172.21.15.83:0/1023970065 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1ce806b4d0 0x7f1ce806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.045+0000 7f1d0659c640 1 --2- 172.21.15.83:0/1023970065 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1ce806b4d0 0x7f1ce806d990 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f1cf80079a0 tx=0x7f1cf8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.049+0000 7f1cff7fe640 1 -- 172.21.15.83:0/1023970065 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1d080cdc80 con 0x7f1d0815dfe0 2024-03-06T07:29:48.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.263+0000 7f1d0c876640 1 -- 172.21.15.83:0/1023970065 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d080b09c0 con 0x7f1d0815dfe0 2024-03-06T07:29:48.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.264+0000 7f1cff7fe640 1 -- 172.21.15.83:0/1023970065 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+3415 (secure 0 0 0) 0x7f1d00058000 con 0x7f1d0815dfe0 2024-03-06T07:29:48.266 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:29:48.266 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":81,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:26.073593+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:29:48.266 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 81 2024-03-06T07:29:48.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.266+0000 7f1d0c876640 1 -- 172.21.15.83:0/1023970065 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1ce806b4d0 msgr2=0x7f1ce806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.266+0000 7f1d0c876640 1 --2- 172.21.15.83:0/1023970065 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1ce806b4d0 0x7f1ce806d990 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f1cf80079a0 tx=0x7f1cf8008040 comp rx=0 tx=0).stop 2024-03-06T07:29:48.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.266+0000 7f1d0c876640 1 -- 172.21.15.83:0/1023970065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d0815dfe0 msgr2=0x7f1d08150c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.266+0000 7f1d0c876640 1 --2- 172.21.15.83:0/1023970065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d0815dfe0 0x7f1d08150c30 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f1d0000c7a0 tx=0x7f1d0000cc60 comp rx=0 tx=0).stop 2024-03-06T07:29:48.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.266+0000 7f1d0c876640 1 -- 172.21.15.83:0/1023970065 shutdown_connections 2024-03-06T07:29:48.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.266+0000 7f1d0c876640 1 --2- 172.21.15.83:0/1023970065 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1ce806b4d0 0x7f1ce806d990 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.266+0000 7f1d0c876640 1 --2- 172.21.15.83:0/1023970065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d0815dfe0 0x7f1d08150c30 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.266+0000 7f1d0c876640 1 --2- 172.21.15.83:0/1023970065 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1d080d27f0 0x7f1d080b8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.266+0000 7f1d0c876640 1 --2- 172.21.15.83:0/1023970065 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1d080cf830 0x7f1d080b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.266+0000 7f1d0c876640 1 -- 172.21.15.83:0/1023970065 >> 172.21.15.83:0/1023970065 conn(0x7f1d080d3cd0 msgr2=0x7f1d080d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:48.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.266+0000 7f1d0c876640 1 -- 172.21.15.83:0/1023970065 shutdown_connections 2024-03-06T07:29:48.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:48.266+0000 7f1d0c876640 1 -- 172.21.15.83:0/1023970065 wait complete. 2024-03-06T07:29:48.480 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:26.073593+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:29:48.480 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:29:48.991 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:49.481 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:49.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.580+0000 7f462e1d0640 1 Processor -- start 2024-03-06T07:29:49.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.580+0000 7f462e1d0640 1 -- start start 2024-03-06T07:29:49.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.581+0000 7f462e1d0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46280cce40 0x7f462815b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.581+0000 7f462e1d0640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4628071f40 con 0x7f46281515e0 2024-03-06T07:29:49.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.581+0000 7f462e1d0640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46280cd2c0 con 0x7f46280cd7a0 2024-03-06T07:29:49.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.581+0000 7f462e1d0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4628070a70 con 0x7f46280cce40 2024-03-06T07:29:49.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.581+0000 7f46277fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f46281515e0 0x7f4628071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:42772/0 (socket says 172.21.15.83:42772) 2024-03-06T07:29:49.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.581+0000 7f46277fe640 1 -- 172.21.15.83:0/4087544916 learned_addr learned my addr 172.21.15.83:0/4087544916 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:29:49.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.582+0000 7f46267fc640 1 -- 172.21.15.83:0/4087544916 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3578811267 0 0) 0x7f4628071f40 con 0x7f46281515e0 2024-03-06T07:29:49.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.582+0000 7f4626ffd640 1 --2- 172.21.15.83:0/4087544916 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46280cce40 0x7f462815b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.582+0000 7f46267fc640 1 -- 172.21.15.83:0/4087544916 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4610003440 con 0x7f46281515e0 2024-03-06T07:29:49.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.582+0000 7f46267fc640 1 -- 172.21.15.83:0/4087544916 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2810344544 0 0) 0x7f4610003440 con 0x7f46281515e0 2024-03-06T07:29:49.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.582+0000 7f46267fc640 1 -- 172.21.15.83:0/4087544916 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46280cce40 msgr2=0x7f462815b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:49.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.582+0000 7f46267fc640 1 --2- 172.21.15.83:0/4087544916 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46280cce40 0x7f462815b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.582+0000 7f46267fc640 1 -- 172.21.15.83:0/4087544916 >> v1:172.21.15.183:6789/0 conn(0x7f46280cd7a0 legacy=0x7f46280cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.582+0000 7f46267fc640 1 -- 172.21.15.83:0/4087544916 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4628070d50 con 0x7f46281515e0 2024-03-06T07:29:49.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.582+0000 7f46267fc640 1 -- 172.21.15.83:0/4087544916 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f46180019f0 con 0x7f46281515e0 2024-03-06T07:29:49.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.582+0000 7f46267fc640 1 -- 172.21.15.83:0/4087544916 >> v1:172.21.15.83:6789/0 conn(0x7f46281515e0 legacy=0x7f4628071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.582+0000 7f46267fc640 1 --2- 172.21.15.83:0/4087544916 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46280cce40 0x7f4610003ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.582+0000 7f46267fc640 1 --2- 172.21.15.83:0/4087544916 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4610004050 0x7f4610008650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.582+0000 7f46267fc640 1 --2- 172.21.15.83:0/4087544916 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4610008de0 0x7f461000b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.582+0000 7f46267fc640 1 -- 172.21.15.83:0/4087544916 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4628070a70 con 0x7f4610004050 2024-03-06T07:29:49.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.582+0000 7f46267fc640 1 -- 172.21.15.83:0/4087544916 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f461000bab0 con 0x7f4610008de0 2024-03-06T07:29:49.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.582+0000 7f46267fc640 1 -- 172.21.15.83:0/4087544916 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f461000bcb0 con 0x7f46280cce40 2024-03-06T07:29:49.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.582+0000 7f4626ffd640 1 --2- 172.21.15.83:0/4087544916 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4610004050 0x7f4610008650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.582+0000 7f46277fe640 1 --2- 172.21.15.83:0/4087544916 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46280cce40 0x7f4610003ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.582+0000 7f4627fff640 1 --2- 172.21.15.83:0/4087544916 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4610008de0 0x7f461000b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.582+0000 7f4626ffd640 1 -- 172.21.15.83:0/4087544916 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46280cce40 msgr2=0x7f4610003ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:49.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.582+0000 7f4626ffd640 1 --2- 172.21.15.83:0/4087544916 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46280cce40 0x7f4610003ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.582+0000 7f4626ffd640 1 -- 172.21.15.83:0/4087544916 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4610008de0 msgr2=0x7f461000b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:49.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.582+0000 7f4626ffd640 1 --2- 172.21.15.83:0/4087544916 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4610008de0 0x7f461000b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.582+0000 7f4626ffd640 1 -- 172.21.15.83:0/4087544916 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4610003440 con 0x7f4610004050 2024-03-06T07:29:49.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.583+0000 7f4626ffd640 1 --2- 172.21.15.83:0/4087544916 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4610004050 0x7f4610008650 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f461c006ad0 tx=0x7f4610003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecde70bccc00d117 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:49.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.583+0000 7f46267fc640 1 -- 172.21.15.83:0/4087544916 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f461c032070 con 0x7f4610004050 2024-03-06T07:29:49.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.583+0000 7f46267fc640 1 -- 172.21.15.83:0/4087544916 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f461c02c970 con 0x7f4610004050 2024-03-06T07:29:49.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.583+0000 7f46267fc640 1 -- 172.21.15.83:0/4087544916 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f461c00bd60 con 0x7f4610004050 2024-03-06T07:29:49.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.584+0000 7f462e1d0640 1 -- 172.21.15.83:0/4087544916 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4610004050 msgr2=0x7f4610008650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.584+0000 7f462e1d0640 1 --2- 172.21.15.83:0/4087544916 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4610004050 0x7f4610008650 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f461c006ad0 tx=0x7f4610003730 comp rx=0 tx=0).stop 2024-03-06T07:29:49.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.584+0000 7f4627fff640 1 -- 172.21.15.83:0/4087544916 reap_dead start 2024-03-06T07:29:49.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.584+0000 7f462e1d0640 1 -- 172.21.15.83:0/4087544916 shutdown_connections 2024-03-06T07:29:49.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.584+0000 7f462e1d0640 1 -- 172.21.15.83:0/4087544916 >> 172.21.15.83:0/4087544916 conn(0x7f46280cf8a0 msgr2=0x7f46280cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:49.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.584+0000 7f462e1d0640 1 -- 172.21.15.83:0/4087544916 shutdown_connections 2024-03-06T07:29:49.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.584+0000 7f462e1d0640 1 -- 172.21.15.83:0/4087544916 wait complete. 2024-03-06T07:29:49.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.585+0000 7f462e1d0640 1 Processor -- start 2024-03-06T07:29:49.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.585+0000 7f462e1d0640 1 -- start start 2024-03-06T07:29:49.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.586+0000 7f462e1d0640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4610004050 0x7f462814b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.586+0000 7f462e1d0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4628152f60 0x7f462815d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.586+0000 7f462e1d0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46281533e0 0x7f462815de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.586+0000 7f462e1d0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f462815b430 con 0x7f46281533e0 2024-03-06T07:29:49.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.586+0000 7f462e1d0640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4628071310 con 0x7f4610004050 2024-03-06T07:29:49.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.586+0000 7f462e1d0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f462815b5b0 con 0x7f4628152f60 2024-03-06T07:29:49.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.586+0000 7f4627fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46281533e0 0x7f462815de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.586+0000 7f4627fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46281533e0 0x7f462815de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:32868/0 (socket says 172.21.15.83:32868) 2024-03-06T07:29:49.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.586+0000 7f4627fff640 1 -- 172.21.15.83:0/4142907171 learned_addr learned my addr 172.21.15.83:0/4142907171 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:49.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.586+0000 7f4626ffd640 1 --2- 172.21.15.83:0/4142907171 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4628152f60 0x7f462815d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.586+0000 7f46277fe640 1 --2- 172.21.15.83:0/4142907171 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4610004050 0x7f462814b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.586+0000 7f4627fff640 1 -- 172.21.15.83:0/4142907171 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4628152f60 msgr2=0x7f462815d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.586+0000 7f4627fff640 1 --2- 172.21.15.83:0/4142907171 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4628152f60 0x7f462815d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.586+0000 7f4627fff640 1 -- 172.21.15.83:0/4142907171 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4610004050 msgr2=0x7f462814b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.586+0000 7f4627fff640 1 --2- 172.21.15.83:0/4142907171 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4610004050 0x7f462814b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.586+0000 7f4627fff640 1 -- 172.21.15.83:0/4142907171 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46281539b0 con 0x7f46281533e0 2024-03-06T07:29:49.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.587+0000 7f4627fff640 1 --2- 172.21.15.83:0/4142907171 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46281533e0 0x7f462815de80 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f46280731f0 tx=0x7f462000f920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:49.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.587+0000 7f460ffff640 1 -- 172.21.15.83:0/4142907171 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46200052e0 con 0x7f46281533e0 2024-03-06T07:29:49.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.587+0000 7f460ffff640 1 -- 172.21.15.83:0/4142907171 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4620018070 con 0x7f46281533e0 2024-03-06T07:29:49.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.587+0000 7f462e1d0640 1 -- 172.21.15.83:0/4142907171 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4628153c90 con 0x7f46281533e0 2024-03-06T07:29:49.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.587+0000 7f460ffff640 1 -- 172.21.15.83:0/4142907171 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4620003830 con 0x7f46281533e0 2024-03-06T07:29:49.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.587+0000 7f462e1d0640 1 -- 172.21.15.83:0/4142907171 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4628154250 con 0x7f46281533e0 2024-03-06T07:29:49.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.588+0000 7f462e1d0640 1 -- 172.21.15.83:0/4142907171 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4628154450 con 0x7f46281533e0 2024-03-06T07:29:49.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.588+0000 7f460ffff640 1 -- 172.21.15.83:0/4142907171 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f46200039d0 con 0x7f46281533e0 2024-03-06T07:29:49.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.589+0000 7f460ffff640 1 --2- 172.21.15.83:0/4142907171 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f461406b500 0x7f461406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.589+0000 7f46277fe640 1 --2- 172.21.15.83:0/4142907171 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f461406b500 0x7f461406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.589+0000 7f460ffff640 1 -- 172.21.15.83:0/4142907171 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4620089be0 con 0x7f46281533e0 2024-03-06T07:29:49.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.589+0000 7f46277fe640 1 --2- 172.21.15.83:0/4142907171 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f461406b500 0x7f461406d9c0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f462815a000 tx=0x7f4618006040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:49.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.593+0000 7f460ffff640 1 -- 172.21.15.83:0/4142907171 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f46280cd2c0 con 0x7f46281533e0 2024-03-06T07:29:49.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.820+0000 7f462e1d0640 1 -- 172.21.15.83:0/4142907171 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4628068840 con 0x7f46281533e0 2024-03-06T07:29:49.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.821+0000 7f460ffff640 1 -- 172.21.15.83:0/4142907171 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+3415 (secure 0 0 0) 0x7f46200583d0 con 0x7f46281533e0 2024-03-06T07:29:49.824 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:29:49.824 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":81,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:26.073593+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:29:49.826 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 81 2024-03-06T07:29:49.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.822+0000 7f462e1d0640 1 -- 172.21.15.83:0/4142907171 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f461406b500 msgr2=0x7f461406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.822+0000 7f462e1d0640 1 --2- 172.21.15.83:0/4142907171 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f461406b500 0x7f461406d9c0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f462815a000 tx=0x7f4618006040 comp rx=0 tx=0).stop 2024-03-06T07:29:49.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.822+0000 7f462e1d0640 1 -- 172.21.15.83:0/4142907171 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46281533e0 msgr2=0x7f462815de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.822+0000 7f462e1d0640 1 --2- 172.21.15.83:0/4142907171 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46281533e0 0x7f462815de80 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f46280731f0 tx=0x7f462000f920 comp rx=0 tx=0).stop 2024-03-06T07:29:49.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.823+0000 7f462e1d0640 1 -- 172.21.15.83:0/4142907171 shutdown_connections 2024-03-06T07:29:49.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.823+0000 7f462e1d0640 1 --2- 172.21.15.83:0/4142907171 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f461406b500 0x7f461406d9c0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.823+0000 7f462e1d0640 1 --2- 172.21.15.83:0/4142907171 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46281533e0 0x7f462815de80 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.823+0000 7f462e1d0640 1 --2- 172.21.15.83:0/4142907171 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4628152f60 0x7f462815d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.823+0000 7f462e1d0640 1 --2- 172.21.15.83:0/4142907171 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4610004050 0x7f462814b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.823+0000 7f462e1d0640 1 -- 172.21.15.83:0/4142907171 >> 172.21.15.83:0/4142907171 conn(0x7f46280cf8a0 msgr2=0x7f462814c280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:49.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.823+0000 7f462e1d0640 1 -- 172.21.15.83:0/4142907171 shutdown_connections 2024-03-06T07:29:49.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:49.823+0000 7f462e1d0640 1 -- 172.21.15.83:0/4142907171 wait complete. 2024-03-06T07:29:50.036 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:26.073593+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:29:50.036 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:29:50.993 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:51.037 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:51.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.135+0000 7f330de8d640 1 Processor -- start 2024-03-06T07:29:51.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.135+0000 7f330de8d640 1 -- start start 2024-03-06T07:29:51.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.136+0000 7f330de8d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f330815bdd0 0x7f33080c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.136+0000 7f330de8d640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f33080d16e0 0x7f33080d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.137+0000 7f330de8d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33080cebe0 con 0x7f330815fa10 2024-03-06T07:29:51.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.137+0000 7f330de8d640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33080ce980 con 0x7f330815bdd0 2024-03-06T07:29:51.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.137+0000 7f330de8d640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33080ceec0 con 0x7f33080d16e0 2024-03-06T07:29:51.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.137+0000 7f33077fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f330815bdd0 0x7f33080c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.137+0000 7f33077fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f330815bdd0 0x7f33080c5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:32872/0 (socket says 172.21.15.83:32872) 2024-03-06T07:29:51.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.137+0000 7f33077fe640 1 -- 172.21.15.83:0/3604466144 learned_addr learned my addr 172.21.15.83:0/3604466144 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:51.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.137+0000 7f33077fe640 1 -- 172.21.15.83:0/3604466144 >> v2:172.21.15.183:3300/0 conn(0x7f33080d16e0 msgr2=0x7f33080d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:51.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.137+0000 7f3307fff640 1 --2- 172.21.15.83:0/3604466144 >> v2:172.21.15.183:3300/0 conn(0x7f33080d16e0 0x7f33080d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.137+0000 7f33077fe640 1 --2- 172.21.15.83:0/3604466144 >> v2:172.21.15.183:3300/0 conn(0x7f33080d16e0 0x7f33080d1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.137+0000 7f33077fe640 1 -- 172.21.15.83:0/3604466144 >> v1:172.21.15.183:6789/0 conn(0x7f330815fa10 legacy=0x7f33080ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:51.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.137+0000 7f33077fe640 1 -- 172.21.15.83:0/3604466144 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33080cf1a0 con 0x7f330815bdd0 2024-03-06T07:29:51.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.138+0000 7f33077fe640 1 --2- 172.21.15.83:0/3604466144 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f330815bdd0 0x7f33080c5610 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f330806e3c0 tx=0x7f32f802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fcb5d230172db80 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:51.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.138+0000 7f3306ffd640 1 -- 172.21.15.83:0/3604466144 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f802e070 con 0x7f330815bdd0 2024-03-06T07:29:51.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.138+0000 7f3306ffd640 1 -- 172.21.15.83:0/3604466144 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32f8037070 con 0x7f330815bdd0 2024-03-06T07:29:51.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.138+0000 7f3306ffd640 1 -- 172.21.15.83:0/3604466144 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f8031a00 con 0x7f330815bdd0 2024-03-06T07:29:51.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.139+0000 7f330de8d640 1 -- 172.21.15.83:0/3604466144 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f330815bdd0 msgr2=0x7f33080c5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.139+0000 7f330de8d640 1 --2- 172.21.15.83:0/3604466144 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f330815bdd0 0x7f33080c5610 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f330806e3c0 tx=0x7f32f802b880 comp rx=0 tx=0).stop 2024-03-06T07:29:51.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.139+0000 7f330de8d640 1 -- 172.21.15.83:0/3604466144 shutdown_connections 2024-03-06T07:29:51.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.139+0000 7f330de8d640 1 --2- 172.21.15.83:0/3604466144 >> v2:172.21.15.183:3300/0 conn(0x7f33080d16e0 0x7f33080d1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.139+0000 7f330de8d640 1 --2- 172.21.15.83:0/3604466144 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f330815bdd0 0x7f33080c5610 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.139+0000 7f330de8d640 1 -- 172.21.15.83:0/3604466144 >> 172.21.15.83:0/3604466144 conn(0x7f330809a9f0 msgr2=0x7f330809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:51.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.139+0000 7f330de8d640 1 -- 172.21.15.83:0/3604466144 shutdown_connections 2024-03-06T07:29:51.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.139+0000 7f330de8d640 1 -- 172.21.15.83:0/3604466144 wait complete. 2024-03-06T07:29:51.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.140+0000 7f330de8d640 1 Processor -- start 2024-03-06T07:29:51.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.140+0000 7f330de8d640 1 -- start start 2024-03-06T07:29:51.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.141+0000 7f330de8d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f33080d16e0 0x7f330815efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.141+0000 7f330de8d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f330815bdd0 0x7f33080c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.141+0000 7f330de8d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f330815fa10 0x7f33081f4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.141+0000 7f330de8d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33080cc5d0 con 0x7f330815fa10 2024-03-06T07:29:51.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.141+0000 7f330de8d640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33080c5b50 con 0x7f330815bdd0 2024-03-06T07:29:51.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.141+0000 7f330de8d640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f330815c1b0 con 0x7f33080d16e0 2024-03-06T07:29:51.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.141+0000 7f3307fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f330815fa10 0x7f33081f4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.141+0000 7f3307fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f330815fa10 0x7f33081f4e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:32884/0 (socket says 172.21.15.83:32884) 2024-03-06T07:29:51.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.141+0000 7f3307fff640 1 -- 172.21.15.83:0/2122838863 learned_addr learned my addr 172.21.15.83:0/2122838863 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:51.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.141+0000 7f33077fe640 1 --2- 172.21.15.83:0/2122838863 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f33080d16e0 0x7f330815efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.141+0000 7f3307fff640 1 -- 172.21.15.83:0/2122838863 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f33080d16e0 msgr2=0x7f330815efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.141+0000 7f32fffff640 1 --2- 172.21.15.83:0/2122838863 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f330815bdd0 0x7f33080c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.141+0000 7f3307fff640 1 --2- 172.21.15.83:0/2122838863 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f33080d16e0 0x7f330815efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.141+0000 7f3307fff640 1 -- 172.21.15.83:0/2122838863 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f330815bdd0 msgr2=0x7f33080c9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.141+0000 7f3307fff640 1 --2- 172.21.15.83:0/2122838863 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f330815bdd0 0x7f33080c9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.141+0000 7f3307fff640 1 -- 172.21.15.83:0/2122838863 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33080ca120 con 0x7f330815fa10 2024-03-06T07:29:51.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.141+0000 7f33077fe640 1 --2- 172.21.15.83:0/2122838863 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f33080d16e0 0x7f330815efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:51.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.141+0000 7f32fffff640 1 --2- 172.21.15.83:0/2122838863 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f330815bdd0 0x7f33080c9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:51.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.141+0000 7f3307fff640 1 --2- 172.21.15.83:0/2122838863 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f330815fa10 0x7f33081f4e70 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f330000cc30 tx=0x7f330000a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:51.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.141+0000 7f3304ff9640 1 -- 172.21.15.83:0/2122838863 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f330000ce40 con 0x7f330815fa10 2024-03-06T07:29:51.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.141+0000 7f3304ff9640 1 -- 172.21.15.83:0/2122838863 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3300003990 con 0x7f330815fa10 2024-03-06T07:29:51.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.141+0000 7f330de8d640 1 -- 172.21.15.83:0/2122838863 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33081f53b0 con 0x7f330815fa10 2024-03-06T07:29:51.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.141+0000 7f3304ff9640 1 -- 172.21.15.83:0/2122838863 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3300003d40 con 0x7f330815fa10 2024-03-06T07:29:51.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.142+0000 7f330de8d640 1 -- 172.21.15.83:0/2122838863 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33080ca680 con 0x7f330815fa10 2024-03-06T07:29:51.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.143+0000 7f330de8d640 1 -- 172.21.15.83:0/2122838863 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f330804f330 con 0x7f330815fa10 2024-03-06T07:29:51.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.143+0000 7f3304ff9640 1 -- 172.21.15.83:0/2122838863 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3300004000 con 0x7f330815fa10 2024-03-06T07:29:51.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.143+0000 7f3304ff9640 1 --2- 172.21.15.83:0/2122838863 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f32e806b4d0 0x7f32e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.144+0000 7f3304ff9640 1 -- 172.21.15.83:0/2122838863 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f330008a6e0 con 0x7f330815fa10 2024-03-06T07:29:51.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.144+0000 7f33077fe640 1 --2- 172.21.15.83:0/2122838863 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f32e806b4d0 0x7f32e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.144+0000 7f33077fe640 1 --2- 172.21.15.83:0/2122838863 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f32e806b4d0 0x7f32e806d990 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f32f8036000 tx=0x7f32f8008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:51.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.152+0000 7f3304ff9640 1 -- 172.21.15.83:0/2122838863 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f33080ca120 con 0x7f330815fa10 2024-03-06T07:29:51.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.361+0000 7f330de8d640 1 -- 172.21.15.83:0/2122838863 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f33080b7a40 con 0x7f330815fa10 2024-03-06T07:29:51.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.362+0000 7f3304ff9640 1 -- 172.21.15.83:0/2122838863 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+3415 (secure 0 0 0) 0x7f3300014030 con 0x7f330815fa10 2024-03-06T07:29:51.364 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:29:51.364 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":81,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:26.073593+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:29:51.364 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 81 2024-03-06T07:29:51.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.365+0000 7f330de8d640 1 -- 172.21.15.83:0/2122838863 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f32e806b4d0 msgr2=0x7f32e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.365+0000 7f330de8d640 1 --2- 172.21.15.83:0/2122838863 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f32e806b4d0 0x7f32e806d990 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f32f8036000 tx=0x7f32f8008030 comp rx=0 tx=0).stop 2024-03-06T07:29:51.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.365+0000 7f330de8d640 1 -- 172.21.15.83:0/2122838863 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f330815fa10 msgr2=0x7f33081f4e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.365+0000 7f330de8d640 1 --2- 172.21.15.83:0/2122838863 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f330815fa10 0x7f33081f4e70 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f330000cc30 tx=0x7f330000a5c0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.365+0000 7f330de8d640 1 -- 172.21.15.83:0/2122838863 shutdown_connections 2024-03-06T07:29:51.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.365+0000 7f330de8d640 1 --2- 172.21.15.83:0/2122838863 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f32e806b4d0 0x7f32e806d990 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.365+0000 7f330de8d640 1 --2- 172.21.15.83:0/2122838863 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f330815fa10 0x7f33081f4e70 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.365+0000 7f330de8d640 1 --2- 172.21.15.83:0/2122838863 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f330815bdd0 0x7f33080c9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.365+0000 7f330de8d640 1 --2- 172.21.15.83:0/2122838863 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f33080d16e0 0x7f330815efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.365+0000 7f330de8d640 1 -- 172.21.15.83:0/2122838863 >> 172.21.15.83:0/2122838863 conn(0x7f330809a9f0 msgr2=0x7f330806ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:51.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.365+0000 7f330de8d640 1 -- 172.21.15.83:0/2122838863 shutdown_connections 2024-03-06T07:29:51.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:51.365+0000 7f330de8d640 1 -- 172.21.15.83:0/2122838863 wait complete. 2024-03-06T07:29:51.582 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:26.073593+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:29:51.582 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:29:52.583 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:52.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.683+0000 7fe31c4fb640 1 Processor -- start 2024-03-06T07:29:52.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.683+0000 7fe31c4fb640 1 -- start start 2024-03-06T07:29:52.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.685+0000 7fe31c4fb640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fe3140d22b0 0x7fe3140d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.685+0000 7fe31c4fb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3140b3430 con 0x7fe31415dfe0 2024-03-06T07:29:52.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.685+0000 7fe31c4fb640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3140b3710 con 0x7fe3140b41f0 2024-03-06T07:29:52.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.685+0000 7fe31c4fb640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3140d2c50 con 0x7fe3140d22b0 2024-03-06T07:29:52.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.685+0000 7fe31a297640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fe31415dfe0 0x7fe3140cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:42778/0 (socket says 172.21.15.83:42778) 2024-03-06T07:29:52.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.685+0000 7fe31a297640 1 -- 172.21.15.83:0/3832240179 learned_addr learned my addr 172.21.15.83:0/3832240179 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:29:52.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.685+0000 7fe319295640 1 -- 172.21.15.83:0/3832240179 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1401132321 0 0) 0x7fe3140b3430 con 0x7fe31415dfe0 2024-03-06T07:29:52.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.685+0000 7fe319295640 1 -- 172.21.15.83:0/3832240179 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe300003520 con 0x7fe31415dfe0 2024-03-06T07:29:52.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.685+0000 7fe319a96640 1 --2- 172.21.15.83:0/3832240179 >> v2:172.21.15.183:3300/0 conn(0x7fe3140d22b0 0x7fe3140d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.685+0000 7fe319295640 1 -- 172.21.15.83:0/3832240179 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 745218435 0 0) 0x7fe300003520 con 0x7fe31415dfe0 2024-03-06T07:29:52.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.685+0000 7fe319295640 1 -- 172.21.15.83:0/3832240179 >> v2:172.21.15.183:3300/0 conn(0x7fe3140d22b0 msgr2=0x7fe3140d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:52.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.685+0000 7fe319295640 1 --2- 172.21.15.83:0/3832240179 >> v2:172.21.15.183:3300/0 conn(0x7fe3140d22b0 0x7fe3140d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.685+0000 7fe319295640 1 -- 172.21.15.83:0/3832240179 >> v1:172.21.15.183:6789/0 conn(0x7fe3140b41f0 legacy=0x7fe3140b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.685+0000 7fe319295640 1 -- 172.21.15.83:0/3832240179 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3140b39f0 con 0x7fe31415dfe0 2024-03-06T07:29:52.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.685+0000 7fe319295640 1 -- 172.21.15.83:0/3832240179 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7fe304001670 con 0x7fe31415dfe0 2024-03-06T07:29:52.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.685+0000 7fe319a96640 1 --2- 172.21.15.83:0/3832240179 >> v2:172.21.15.183:3300/0 conn(0x7fe3140d22b0 0x7fe3140d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:52.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.686+0000 7fe319295640 1 -- 172.21.15.83:0/3832240179 >> v1:172.21.15.83:6789/0 conn(0x7fe31415dfe0 legacy=0x7fe3140cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.686+0000 7fe319295640 1 --2- 172.21.15.83:0/3832240179 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe3000039b0 0x7fe300003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.686+0000 7fe319295640 1 --2- 172.21.15.83:0/3832240179 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe300004800 0x7fe300004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.686+0000 7fe319295640 1 --2- 172.21.15.83:0/3832240179 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3000051c0 0x7fe30000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.686+0000 7fe319295640 1 -- 172.21.15.83:0/3832240179 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe30000c0e0 con 0x7fe300004800 2024-03-06T07:29:52.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.686+0000 7fe319295640 1 -- 172.21.15.83:0/3832240179 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe30000c290 con 0x7fe3000051c0 2024-03-06T07:29:52.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.686+0000 7fe319295640 1 -- 172.21.15.83:0/3832240179 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe30000c490 con 0x7fe3000039b0 2024-03-06T07:29:52.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.686+0000 7fe319a96640 1 --2- 172.21.15.83:0/3832240179 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe300004800 0x7fe300004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.686+0000 7fe31a297640 1 --2- 172.21.15.83:0/3832240179 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe3000039b0 0x7fe300003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.686+0000 7fe31aa98640 1 --2- 172.21.15.83:0/3832240179 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3000051c0 0x7fe30000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.686+0000 7fe319a96640 1 -- 172.21.15.83:0/3832240179 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe3000039b0 msgr2=0x7fe300003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:52.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.686+0000 7fe319a96640 1 --2- 172.21.15.83:0/3832240179 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe3000039b0 0x7fe300003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.686+0000 7fe319a96640 1 -- 172.21.15.83:0/3832240179 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3000051c0 msgr2=0x7fe30000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:52.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.686+0000 7fe319a96640 1 --2- 172.21.15.83:0/3832240179 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3000051c0 0x7fe30000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.686+0000 7fe319a96640 1 -- 172.21.15.83:0/3832240179 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe300003520 con 0x7fe300004800 2024-03-06T07:29:52.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.686+0000 7fe31aa98640 1 -- 172.21.15.83:0/3832240179 reap_dead start 2024-03-06T07:29:52.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.687+0000 7fe319a96640 1 --2- 172.21.15.83:0/3832240179 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe300004800 0x7fe300004c80 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fe3100098a0 tx=0x7fe300005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3913be9e1a40b184 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.687+0000 7fe319295640 1 -- 172.21.15.83:0/3832240179 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe31000f040 con 0x7fe300004800 2024-03-06T07:29:52.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.687+0000 7fe319295640 1 -- 172.21.15.83:0/3832240179 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe310004230 con 0x7fe300004800 2024-03-06T07:29:52.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.687+0000 7fe319295640 1 -- 172.21.15.83:0/3832240179 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe310003610 con 0x7fe300004800 2024-03-06T07:29:52.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.688+0000 7fe31c4fb640 1 -- 172.21.15.83:0/3832240179 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe300004800 msgr2=0x7fe300004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.688+0000 7fe31c4fb640 1 --2- 172.21.15.83:0/3832240179 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe300004800 0x7fe300004c80 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fe3100098a0 tx=0x7fe300005830 comp rx=0 tx=0).stop 2024-03-06T07:29:52.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.688+0000 7fe31c4fb640 1 -- 172.21.15.83:0/3832240179 shutdown_connections 2024-03-06T07:29:52.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.688+0000 7fe31c4fb640 1 --2- 172.21.15.83:0/3832240179 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe300004800 0x7fe300004c80 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.688+0000 7fe31c4fb640 1 -- 172.21.15.83:0/3832240179 >> 172.21.15.83:0/3832240179 conn(0x7fe3140d3cd0 msgr2=0x7fe3140d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:52.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.688+0000 7fe31c4fb640 1 -- 172.21.15.83:0/3832240179 shutdown_connections 2024-03-06T07:29:52.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.688+0000 7fe31c4fb640 1 -- 172.21.15.83:0/3832240179 wait complete. 2024-03-06T07:29:52.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.689+0000 7fe31c4fb640 1 Processor -- start 2024-03-06T07:29:52.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.689+0000 7fe31c4fb640 1 -- start start 2024-03-06T07:29:52.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.690+0000 7fe31c4fb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe300004800 0x7fe31415bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.690+0000 7fe31c4fb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3141601f0 0x7fe3140b83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.690+0000 7fe31c4fb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe314160670 0x7fe3140b8900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.690+0000 7fe31c4fb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3140b3d60 con 0x7fe314160670 2024-03-06T07:29:52.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.690+0000 7fe31c4fb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3140b31e0 con 0x7fe3141601f0 2024-03-06T07:29:52.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.690+0000 7fe31c4fb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe31415e3c0 con 0x7fe300004800 2024-03-06T07:29:52.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.690+0000 7fe31aa98640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe314160670 0x7fe3140b8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.690+0000 7fe31aa98640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe314160670 0x7fe3140b8900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:32900/0 (socket says 172.21.15.83:32900) 2024-03-06T07:29:52.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.690+0000 7fe31aa98640 1 -- 172.21.15.83:0/289218643 learned_addr learned my addr 172.21.15.83:0/289218643 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:52.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.691+0000 7fe31a297640 1 --2- 172.21.15.83:0/289218643 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe300004800 0x7fe31415bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.691+0000 7fe319a96640 1 --2- 172.21.15.83:0/289218643 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3141601f0 0x7fe3140b83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.691+0000 7fe31aa98640 1 -- 172.21.15.83:0/289218643 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe300004800 msgr2=0x7fe31415bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.691+0000 7fe31aa98640 1 --2- 172.21.15.83:0/289218643 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe300004800 0x7fe31415bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.691+0000 7fe31aa98640 1 -- 172.21.15.83:0/289218643 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3141601f0 msgr2=0x7fe3140b83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.691+0000 7fe31aa98640 1 --2- 172.21.15.83:0/289218643 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3141601f0 0x7fe3140b83c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.691+0000 7fe31aa98640 1 -- 172.21.15.83:0/289218643 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3141f3c90 con 0x7fe314160670 2024-03-06T07:29:52.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.691+0000 7fe31aa98640 1 --2- 172.21.15.83:0/289218643 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe314160670 0x7fe3140b8900 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7fe3140cfd40 tx=0x7fe30c00fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.691+0000 7fe30affd640 1 -- 172.21.15.83:0/289218643 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30c01a070 con 0x7fe314160670 2024-03-06T07:29:52.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.691+0000 7fe30affd640 1 -- 172.21.15.83:0/289218643 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe30c006d60 con 0x7fe314160670 2024-03-06T07:29:52.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.691+0000 7fe31c4fb640 1 -- 172.21.15.83:0/289218643 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3141f3f30 con 0x7fe314160670 2024-03-06T07:29:52.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.691+0000 7fe30affd640 1 -- 172.21.15.83:0/289218643 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30c003bb0 con 0x7fe314160670 2024-03-06T07:29:52.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.691+0000 7fe31c4fb640 1 -- 172.21.15.83:0/289218643 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3141f44f0 con 0x7fe314160670 2024-03-06T07:29:52.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.692+0000 7fe31c4fb640 1 -- 172.21.15.83:0/289218643 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe31406c970 con 0x7fe314160670 2024-03-06T07:29:52.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.693+0000 7fe30affd640 1 -- 172.21.15.83:0/289218643 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe30c01d070 con 0x7fe314160670 2024-03-06T07:29:52.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.693+0000 7fe30affd640 1 --2- 172.21.15.83:0/289218643 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe2f806b4d0 0x7fe2f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.693+0000 7fe31a297640 1 --2- 172.21.15.83:0/289218643 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe2f806b4d0 0x7fe2f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.693+0000 7fe30affd640 1 -- 172.21.15.83:0/289218643 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe30c0896a0 con 0x7fe314160670 2024-03-06T07:29:52.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.693+0000 7fe31a297640 1 --2- 172.21.15.83:0/289218643 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe2f806b4d0 0x7fe2f806d990 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fe3040011e0 tx=0x7fe304000ed0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.697+0000 7fe30affd640 1 -- 172.21.15.83:0/289218643 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe31406c970 con 0x7fe314160670 2024-03-06T07:29:52.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.904+0000 7fe31c4fb640 1 -- 172.21.15.83:0/289218643 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe31404f330 con 0x7fe314160670 2024-03-06T07:29:52.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.905+0000 7fe30affd640 1 -- 172.21.15.83:0/289218643 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+3415 (secure 0 0 0) 0x7fe30c057e90 con 0x7fe314160670 2024-03-06T07:29:52.907 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:29:52.907 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":81,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:26.073593+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:29:52.907 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 81 2024-03-06T07:29:52.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.907+0000 7fe31c4fb640 1 -- 172.21.15.83:0/289218643 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe2f806b4d0 msgr2=0x7fe2f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.907+0000 7fe31c4fb640 1 --2- 172.21.15.83:0/289218643 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe2f806b4d0 0x7fe2f806d990 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fe3040011e0 tx=0x7fe304000ed0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.907+0000 7fe31c4fb640 1 -- 172.21.15.83:0/289218643 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe314160670 msgr2=0x7fe3140b8900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.907+0000 7fe31c4fb640 1 --2- 172.21.15.83:0/289218643 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe314160670 0x7fe3140b8900 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7fe3140cfd40 tx=0x7fe30c00fbe0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.907+0000 7fe31c4fb640 1 -- 172.21.15.83:0/289218643 shutdown_connections 2024-03-06T07:29:52.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.907+0000 7fe31c4fb640 1 --2- 172.21.15.83:0/289218643 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe2f806b4d0 0x7fe2f806d990 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.907+0000 7fe31c4fb640 1 --2- 172.21.15.83:0/289218643 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe314160670 0x7fe3140b8900 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.907+0000 7fe31c4fb640 1 --2- 172.21.15.83:0/289218643 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3141601f0 0x7fe3140b83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.907+0000 7fe31c4fb640 1 --2- 172.21.15.83:0/289218643 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe300004800 0x7fe31415bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.907+0000 7fe31c4fb640 1 -- 172.21.15.83:0/289218643 >> 172.21.15.83:0/289218643 conn(0x7fe3140d3cd0 msgr2=0x7fe31415d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:52.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.907+0000 7fe31c4fb640 1 -- 172.21.15.83:0/289218643 shutdown_connections 2024-03-06T07:29:52.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:52.907+0000 7fe31c4fb640 1 -- 172.21.15.83:0/289218643 wait complete. 2024-03-06T07:29:52.995 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:53.120 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:26.073593+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:29:53.121 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:29:54.121 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:54.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.223+0000 7f349115c640 1 Processor -- start 2024-03-06T07:29:54.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.223+0000 7f349115c640 1 -- start start 2024-03-06T07:29:54.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.224+0000 7f349115c640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f348c0d4760 0x7f348c0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.224+0000 7f349115c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f348c0d25d0 con 0x7f348c0b4170 2024-03-06T07:29:54.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.224+0000 7f349115c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f348c0d2870 con 0x7f348c0d0a80 2024-03-06T07:29:54.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.224+0000 7f349115c640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f348c0d4bf0 con 0x7f348c0d4760 2024-03-06T07:29:54.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.224+0000 7f348ad9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f348c0b4170 0x7f348c0d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:42784/0 (socket says 172.21.15.83:42784) 2024-03-06T07:29:54.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.224+0000 7f348ad9d640 1 -- 172.21.15.83:0/4254505816 learned_addr learned my addr 172.21.15.83:0/4254505816 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:29:54.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.224+0000 7f348a59c640 1 -- 172.21.15.83:0/4254505816 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3656225585 0 0) 0x7f348c0d25d0 con 0x7f348c0b4170 2024-03-06T07:29:54.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.224+0000 7f348a59c640 1 -- 172.21.15.83:0/4254505816 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3478003520 con 0x7f348c0b4170 2024-03-06T07:29:54.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.224+0000 7f3483fff640 1 --2- 172.21.15.83:0/4254505816 >> v2:172.21.15.183:3300/0 conn(0x7f348c0d4760 0x7f348c0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.225+0000 7f348a59c640 1 -- 172.21.15.83:0/4254505816 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1541303848 0 0) 0x7f3478003520 con 0x7f348c0b4170 2024-03-06T07:29:54.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.225+0000 7f348a59c640 1 -- 172.21.15.83:0/4254505816 >> v2:172.21.15.183:3300/0 conn(0x7f348c0d4760 msgr2=0x7f348c0b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:54.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.225+0000 7f348a59c640 1 --2- 172.21.15.83:0/4254505816 >> v2:172.21.15.183:3300/0 conn(0x7f348c0d4760 0x7f348c0b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.225+0000 7f348a59c640 1 -- 172.21.15.83:0/4254505816 >> v1:172.21.15.183:6789/0 conn(0x7f348c0d0a80 legacy=0x7f348c0cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.225+0000 7f348a59c640 1 -- 172.21.15.83:0/4254505816 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f348c0d2b50 con 0x7f348c0b4170 2024-03-06T07:29:54.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.225+0000 7f348a59c640 1 -- 172.21.15.83:0/4254505816 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f347c001690 con 0x7f348c0b4170 2024-03-06T07:29:54.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.225+0000 7f348a59c640 1 -- 172.21.15.83:0/4254505816 >> v1:172.21.15.83:6789/0 conn(0x7f348c0b4170 legacy=0x7f348c0d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.225+0000 7f348a59c640 1 --2- 172.21.15.83:0/4254505816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34780039b0 0x7f3478003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.225+0000 7f348a59c640 1 --2- 172.21.15.83:0/4254505816 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3478004800 0x7f3478004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.225+0000 7f348a59c640 1 --2- 172.21.15.83:0/4254505816 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f34780051c0 0x7f347800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.225+0000 7f348a59c640 1 -- 172.21.15.83:0/4254505816 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f348c0d4bf0 con 0x7f34780039b0 2024-03-06T07:29:54.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.225+0000 7f348a59c640 1 -- 172.21.15.83:0/4254505816 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f347800c060 con 0x7f3478004800 2024-03-06T07:29:54.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.225+0000 7f348a59c640 1 -- 172.21.15.83:0/4254505816 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f347800c260 con 0x7f34780051c0 2024-03-06T07:29:54.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.225+0000 7f348ad9d640 1 --2- 172.21.15.83:0/4254505816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34780039b0 0x7f3478003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.225+0000 7f3483fff640 1 --2- 172.21.15.83:0/4254505816 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3478004800 0x7f3478004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.225+0000 7f348b59e640 1 --2- 172.21.15.83:0/4254505816 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f34780051c0 0x7f347800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.225+0000 7f348ad9d640 1 -- 172.21.15.83:0/4254505816 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f34780051c0 msgr2=0x7f347800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:54.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.225+0000 7f348ad9d640 1 --2- 172.21.15.83:0/4254505816 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f34780051c0 0x7f347800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.225+0000 7f348ad9d640 1 -- 172.21.15.83:0/4254505816 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3478004800 msgr2=0x7f3478004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:54.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.225+0000 7f348ad9d640 1 --2- 172.21.15.83:0/4254505816 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3478004800 0x7f3478004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.225+0000 7f348ad9d640 1 -- 172.21.15.83:0/4254505816 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3478003520 con 0x7f34780039b0 2024-03-06T07:29:54.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.225+0000 7f348b59e640 1 -- 172.21.15.83:0/4254505816 reap_dead start 2024-03-06T07:29:54.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.226+0000 7f348ad9d640 1 --2- 172.21.15.83:0/4254505816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34780039b0 0x7f3478003dd0 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f347c007fd0 tx=0x7f3478005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2137636171de0268 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.226+0000 7f348a59c640 1 -- 172.21.15.83:0/4254505816 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f347c007d50 con 0x7f34780039b0 2024-03-06T07:29:54.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.226+0000 7f348a59c640 1 -- 172.21.15.83:0/4254505816 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f347c030070 con 0x7f34780039b0 2024-03-06T07:29:54.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.226+0000 7f348a59c640 1 -- 172.21.15.83:0/4254505816 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f347c005d30 con 0x7f34780039b0 2024-03-06T07:29:54.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.227+0000 7f349115c640 1 -- 172.21.15.83:0/4254505816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34780039b0 msgr2=0x7f3478003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.227+0000 7f349115c640 1 --2- 172.21.15.83:0/4254505816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34780039b0 0x7f3478003dd0 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f347c007fd0 tx=0x7f3478005830 comp rx=0 tx=0).stop 2024-03-06T07:29:54.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.227+0000 7f349115c640 1 -- 172.21.15.83:0/4254505816 shutdown_connections 2024-03-06T07:29:54.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.227+0000 7f349115c640 1 --2- 172.21.15.83:0/4254505816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34780039b0 0x7f3478003dd0 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.227+0000 7f349115c640 1 -- 172.21.15.83:0/4254505816 >> 172.21.15.83:0/4254505816 conn(0x7f348c0d3c40 msgr2=0x7f348c0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:54.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.227+0000 7f349115c640 1 -- 172.21.15.83:0/4254505816 shutdown_connections 2024-03-06T07:29:54.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.227+0000 7f349115c640 1 -- 172.21.15.83:0/4254505816 wait complete. 2024-03-06T07:29:54.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.228+0000 7f349115c640 1 Processor -- start 2024-03-06T07:29:54.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.228+0000 7f349115c640 1 -- start start 2024-03-06T07:29:54.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.229+0000 7f349115c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34780039b0 0x7f348c09e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.229+0000 7f349115c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f348c0b2ca0 0x7f348c0ca460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.229+0000 7f349115c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f348c0a0580 0x7f348c0a0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.229+0000 7f349115c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f348c0b4dd0 con 0x7f34780039b0 2024-03-06T07:29:54.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.229+0000 7f349115c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f348c0ce9e0 con 0x7f348c0b2ca0 2024-03-06T07:29:54.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.229+0000 7f349115c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f348c0b4f50 con 0x7f348c0a0580 2024-03-06T07:29:54.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.229+0000 7f348ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34780039b0 0x7f348c09e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.229+0000 7f348ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34780039b0 0x7f348c09e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:32918/0 (socket says 172.21.15.83:32918) 2024-03-06T07:29:54.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.229+0000 7f348ad9d640 1 -- 172.21.15.83:0/1170900264 learned_addr learned my addr 172.21.15.83:0/1170900264 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:54.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.229+0000 7f3483fff640 1 --2- 172.21.15.83:0/1170900264 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f348c0b2ca0 0x7f348c0ca460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.229+0000 7f348b59e640 1 --2- 172.21.15.83:0/1170900264 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f348c0a0580 0x7f348c0a0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.229+0000 7f348ad9d640 1 -- 172.21.15.83:0/1170900264 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f348c0a0580 msgr2=0x7f348c0a0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.229+0000 7f348ad9d640 1 --2- 172.21.15.83:0/1170900264 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f348c0a0580 0x7f348c0a0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.229+0000 7f348ad9d640 1 -- 172.21.15.83:0/1170900264 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f348c0b2ca0 msgr2=0x7f348c0ca460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.229+0000 7f348ad9d640 1 --2- 172.21.15.83:0/1170900264 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f348c0b2ca0 0x7f348c0ca460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.229+0000 7f348ad9d640 1 -- 172.21.15.83:0/1170900264 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f348c0b3120 con 0x7f34780039b0 2024-03-06T07:29:54.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.229+0000 7f348ad9d640 1 --2- 172.21.15.83:0/1170900264 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34780039b0 0x7f348c09e970 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f347c00c380 tx=0x7f347c02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.230+0000 7f34837fe640 1 -- 172.21.15.83:0/1170900264 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f347c007d50 con 0x7f34780039b0 2024-03-06T07:29:54.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.230+0000 7f34837fe640 1 -- 172.21.15.83:0/1170900264 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f347c004bf0 con 0x7f34780039b0 2024-03-06T07:29:54.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.230+0000 7f349115c640 1 -- 172.21.15.83:0/1170900264 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f348c09f0a0 con 0x7f34780039b0 2024-03-06T07:29:54.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.230+0000 7f34837fe640 1 -- 172.21.15.83:0/1170900264 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f347c007070 con 0x7f34780039b0 2024-03-06T07:29:54.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.230+0000 7f349115c640 1 -- 172.21.15.83:0/1170900264 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f348c09f660 con 0x7f34780039b0 2024-03-06T07:29:54.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.231+0000 7f349115c640 1 -- 172.21.15.83:0/1170900264 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f348c04f330 con 0x7f34780039b0 2024-03-06T07:29:54.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.231+0000 7f34837fe640 1 -- 172.21.15.83:0/1170900264 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f347c048020 con 0x7f34780039b0 2024-03-06T07:29:54.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.232+0000 7f34837fe640 1 --2- 172.21.15.83:0/1170900264 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f346c06b4d0 0x7f346c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.232+0000 7f3483fff640 1 --2- 172.21.15.83:0/1170900264 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f346c06b4d0 0x7f346c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.232+0000 7f34837fe640 1 -- 172.21.15.83:0/1170900264 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f347c0ab5d0 con 0x7f34780039b0 2024-03-06T07:29:54.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.232+0000 7f3483fff640 1 --2- 172.21.15.83:0/1170900264 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f346c06b4d0 0x7f346c06d990 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f348c1f72e0 tx=0x7f3474008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.236+0000 7f34837fe640 1 -- 172.21.15.83:0/1170900264 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f347c07a4a0 con 0x7f34780039b0 2024-03-06T07:29:54.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.451+0000 7f349115c640 1 -- 172.21.15.83:0/1170900264 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f348c0733b0 con 0x7f34780039b0 2024-03-06T07:29:54.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.452+0000 7f34837fe640 1 -- 172.21.15.83:0/1170900264 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f347c034e80 con 0x7f34780039b0 2024-03-06T07:29:54.454 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:29:54.454 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:29:54.455 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:29:54.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.454+0000 7f349115c640 1 -- 172.21.15.83:0/1170900264 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f346c06b4d0 msgr2=0x7f346c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.454+0000 7f349115c640 1 --2- 172.21.15.83:0/1170900264 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f346c06b4d0 0x7f346c06d990 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f348c1f72e0 tx=0x7f3474008040 comp rx=0 tx=0).stop 2024-03-06T07:29:54.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.454+0000 7f349115c640 1 -- 172.21.15.83:0/1170900264 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34780039b0 msgr2=0x7f348c09e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.454+0000 7f349115c640 1 --2- 172.21.15.83:0/1170900264 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34780039b0 0x7f348c09e970 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f347c00c380 tx=0x7f347c02e070 comp rx=0 tx=0).stop 2024-03-06T07:29:54.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.454+0000 7f349115c640 1 -- 172.21.15.83:0/1170900264 shutdown_connections 2024-03-06T07:29:54.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.454+0000 7f349115c640 1 --2- 172.21.15.83:0/1170900264 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f346c06b4d0 0x7f346c06d990 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.454+0000 7f349115c640 1 --2- 172.21.15.83:0/1170900264 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f348c0a0580 0x7f348c0a0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.454+0000 7f349115c640 1 --2- 172.21.15.83:0/1170900264 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f348c0b2ca0 0x7f348c0ca460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.454+0000 7f349115c640 1 --2- 172.21.15.83:0/1170900264 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34780039b0 0x7f348c09e970 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.454+0000 7f349115c640 1 -- 172.21.15.83:0/1170900264 >> 172.21.15.83:0/1170900264 conn(0x7f348c0d3c40 msgr2=0x7f348c06bb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:54.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.454+0000 7f349115c640 1 -- 172.21.15.83:0/1170900264 shutdown_connections 2024-03-06T07:29:54.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:54.454+0000 7f349115c640 1 -- 172.21.15.83:0/1170900264 wait complete. 2024-03-06T07:29:54.668 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:29:54.669 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:29:54.997 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:55.669 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:55.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.768+0000 7effcb1c7640 1 Processor -- start 2024-03-06T07:29:55.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.768+0000 7effcb1c7640 1 -- start start 2024-03-06T07:29:55.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.769+0000 7effcb1c7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7effc415dfe0 0x7effc40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.769+0000 7effcb1c7640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7effc40cf830 0x7effc40d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.769+0000 7effcb1c7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7effc40d27f0 0x7effc40d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.769+0000 7effcb1c7640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effc40d2c50 con 0x7effc40d27f0 2024-03-06T07:29:55.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.769+0000 7effcb1c7640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effc40b31e0 con 0x7effc40cf830 2024-03-06T07:29:55.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.769+0000 7effcb1c7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effc40b3d60 con 0x7effc415dfe0 2024-03-06T07:29:55.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.769+0000 7effc9764640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7effc40d27f0 0x7effc40d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.769+0000 7effc9764640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7effc40d27f0 0x7effc40d0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:32928/0 (socket says 172.21.15.83:32928) 2024-03-06T07:29:55.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.769+0000 7effc9764640 1 -- 172.21.15.83:0/2117143343 learned_addr learned my addr 172.21.15.83:0/2117143343 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:55.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.770+0000 7effc8f63640 1 --2- 172.21.15.83:0/2117143343 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7effc415dfe0 0x7effc40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.770+0000 7effc3fff640 1 --2- 172.21.15.83:0/2117143343 >> v2:172.21.15.183:3300/0 conn(0x7effc40cf830 0x7effc40d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.770+0000 7effc9764640 1 -- 172.21.15.83:0/2117143343 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7effc415dfe0 msgr2=0x7effc40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:55.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.770+0000 7effc9764640 1 --2- 172.21.15.83:0/2117143343 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7effc415dfe0 0x7effc40cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.770+0000 7effc9764640 1 -- 172.21.15.83:0/2117143343 >> v2:172.21.15.183:3300/0 conn(0x7effc40cf830 msgr2=0x7effc40d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:55.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.770+0000 7effc9764640 1 --2- 172.21.15.83:0/2117143343 >> v2:172.21.15.183:3300/0 conn(0x7effc40cf830 0x7effc40d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.770+0000 7effc9764640 1 -- 172.21.15.83:0/2117143343 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc40b3430 con 0x7effc40d27f0 2024-03-06T07:29:55.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.770+0000 7effc9764640 1 --2- 172.21.15.83:0/2117143343 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effc40d27f0 0x7effc40d0300 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7effc408cba0 tx=0x7effbc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc65446547498adf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.771+0000 7effc37fe640 1 -- 172.21.15.83:0/2117143343 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effbc02ca70 con 0x7effc40d27f0 2024-03-06T07:29:55.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.771+0000 7effc37fe640 1 -- 172.21.15.83:0/2117143343 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effbc02cc10 con 0x7effc40d27f0 2024-03-06T07:29:55.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.771+0000 7effc37fe640 1 -- 172.21.15.83:0/2117143343 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effbc0343f0 con 0x7effc40d27f0 2024-03-06T07:29:55.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.771+0000 7effcb1c7640 1 -- 172.21.15.83:0/2117143343 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effc40d27f0 msgr2=0x7effc40d0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.771+0000 7effcb1c7640 1 --2- 172.21.15.83:0/2117143343 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effc40d27f0 0x7effc40d0300 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7effc408cba0 tx=0x7effbc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.771+0000 7effcb1c7640 1 -- 172.21.15.83:0/2117143343 shutdown_connections 2024-03-06T07:29:55.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.771+0000 7effcb1c7640 1 --2- 172.21.15.83:0/2117143343 >> v2:172.21.15.183:3300/0 conn(0x7effc40cf830 0x7effc40d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.771+0000 7effcb1c7640 1 --2- 172.21.15.83:0/2117143343 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effc40d27f0 0x7effc40d0300 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.771+0000 7effcb1c7640 1 --2- 172.21.15.83:0/2117143343 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7effc415dfe0 0x7effc40cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.771+0000 7effcb1c7640 1 -- 172.21.15.83:0/2117143343 >> 172.21.15.83:0/2117143343 conn(0x7effc40d3cd0 msgr2=0x7effc40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:55.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.771+0000 7effcb1c7640 1 -- 172.21.15.83:0/2117143343 shutdown_connections 2024-03-06T07:29:55.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.771+0000 7effcb1c7640 1 -- 172.21.15.83:0/2117143343 wait complete. 2024-03-06T07:29:55.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.773+0000 7effcb1c7640 1 Processor -- start 2024-03-06T07:29:55.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.773+0000 7effcb1c7640 1 -- start start 2024-03-06T07:29:55.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.774+0000 7effcb1c7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7effc40cf830 0x7effc41f71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.775+0000 7effcb1c7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effc40d27f0 0x7effc40cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.775+0000 7effcb1c7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7effc415dfe0 0x7effc40b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.775+0000 7effcb1c7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc40d1c70 con 0x7effc40d27f0 2024-03-06T07:29:55.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.775+0000 7effcb1c7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc40b39f0 con 0x7effc40cf830 2024-03-06T07:29:55.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.775+0000 7effcb1c7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effc40d1df0 con 0x7effc415dfe0 2024-03-06T07:29:55.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.775+0000 7effc3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effc40d27f0 0x7effc40cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.775+0000 7effc3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effc40d27f0 0x7effc40cbdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:32942/0 (socket says 172.21.15.83:32942) 2024-03-06T07:29:55.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.775+0000 7effc3fff640 1 -- 172.21.15.83:0/1141060057 learned_addr learned my addr 172.21.15.83:0/1141060057 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:55.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.775+0000 7effc9764640 1 --2- 172.21.15.83:0/1141060057 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7effc415dfe0 0x7effc40b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.775+0000 7effc8f63640 1 --2- 172.21.15.83:0/1141060057 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7effc40cf830 0x7effc41f71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.775+0000 7effc3fff640 1 -- 172.21.15.83:0/1141060057 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7effc415dfe0 msgr2=0x7effc40b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.775+0000 7effc3fff640 1 --2- 172.21.15.83:0/1141060057 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7effc415dfe0 0x7effc40b68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.775+0000 7effc3fff640 1 -- 172.21.15.83:0/1141060057 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7effc40cf830 msgr2=0x7effc41f71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.775+0000 7effc3fff640 1 --2- 172.21.15.83:0/1141060057 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7effc40cf830 0x7effc41f71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.775+0000 7effc3fff640 1 -- 172.21.15.83:0/1141060057 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc40b8970 con 0x7effc40d27f0 2024-03-06T07:29:55.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.775+0000 7effc8f63640 1 --2- 172.21.15.83:0/1141060057 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7effc40cf830 0x7effc41f71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:55.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.775+0000 7effc3fff640 1 --2- 172.21.15.83:0/1141060057 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effc40d27f0 0x7effc40cbdf0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7effb000d7d0 tx=0x7effb000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.775+0000 7effc17fa640 1 -- 172.21.15.83:0/1141060057 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb00096a0 con 0x7effc40d27f0 2024-03-06T07:29:55.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.775+0000 7effc17fa640 1 -- 172.21.15.83:0/1141060057 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effb0019070 con 0x7effc40d27f0 2024-03-06T07:29:55.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.775+0000 7effc17fa640 1 -- 172.21.15.83:0/1141060057 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb0009a60 con 0x7effc40d27f0 2024-03-06T07:29:55.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.775+0000 7effcb1c7640 1 -- 172.21.15.83:0/1141060057 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effc41f7910 con 0x7effc40d27f0 2024-03-06T07:29:55.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.775+0000 7effcb1c7640 1 -- 172.21.15.83:0/1141060057 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effc41f7ed0 con 0x7effc40d27f0 2024-03-06T07:29:55.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.776+0000 7effcb1c7640 1 -- 172.21.15.83:0/1141060057 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effc404f330 con 0x7effc40d27f0 2024-03-06T07:29:55.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.777+0000 7effc17fa640 1 -- 172.21.15.83:0/1141060057 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7effb0009c00 con 0x7effc40d27f0 2024-03-06T07:29:55.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.777+0000 7effc17fa640 1 --2- 172.21.15.83:0/1141060057 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7effa806b500 0x7effa806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.777+0000 7effc17fa640 1 -- 172.21.15.83:0/1141060057 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7effb008a5b0 con 0x7effc40d27f0 2024-03-06T07:29:55.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.777+0000 7effc8f63640 1 --2- 172.21.15.83:0/1141060057 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7effa806b500 0x7effa806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.778+0000 7effc8f63640 1 --2- 172.21.15.83:0/1141060057 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7effa806b500 0x7effa806d9c0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7effb8006fd0 tx=0x7effb8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.782+0000 7effc17fa640 1 -- 172.21.15.83:0/1141060057 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7effc40b8970 con 0x7effc40d27f0 2024-03-06T07:29:55.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.993+0000 7effcb1c7640 1 -- 172.21.15.83:0/1141060057 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7effc40ce870 con 0x7effc40d27f0 2024-03-06T07:29:55.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.993+0000 7effc17fa640 1 -- 172.21.15.83:0/1141060057 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7effb0059480 con 0x7effc40d27f0 2024-03-06T07:29:55.996 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:29:55.996 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:29:55.996 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:29:55.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.995+0000 7effcb1c7640 1 -- 172.21.15.83:0/1141060057 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7effa806b500 msgr2=0x7effa806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.995+0000 7effcb1c7640 1 --2- 172.21.15.83:0/1141060057 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7effa806b500 0x7effa806d9c0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7effb8006fd0 tx=0x7effb8008040 comp rx=0 tx=0).stop 2024-03-06T07:29:55.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.995+0000 7effcb1c7640 1 -- 172.21.15.83:0/1141060057 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effc40d27f0 msgr2=0x7effc40cbdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.995+0000 7effcb1c7640 1 --2- 172.21.15.83:0/1141060057 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effc40d27f0 0x7effc40cbdf0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7effb000d7d0 tx=0x7effb000dc90 comp rx=0 tx=0).stop 2024-03-06T07:29:55.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.996+0000 7effcb1c7640 1 -- 172.21.15.83:0/1141060057 shutdown_connections 2024-03-06T07:29:55.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.996+0000 7effcb1c7640 1 --2- 172.21.15.83:0/1141060057 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7effa806b500 0x7effa806d9c0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.996+0000 7effcb1c7640 1 --2- 172.21.15.83:0/1141060057 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7effc415dfe0 0x7effc40b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.996+0000 7effcb1c7640 1 --2- 172.21.15.83:0/1141060057 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effc40d27f0 0x7effc40cbdf0 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.996+0000 7effcb1c7640 1 --2- 172.21.15.83:0/1141060057 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7effc40cf830 0x7effc41f71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.996+0000 7effcb1c7640 1 -- 172.21.15.83:0/1141060057 >> 172.21.15.83:0/1141060057 conn(0x7effc40d3cd0 msgr2=0x7effc40cff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:55.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.996+0000 7effcb1c7640 1 -- 172.21.15.83:0/1141060057 shutdown_connections 2024-03-06T07:29:55.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:55.996+0000 7effcb1c7640 1 -- 172.21.15.83:0/1141060057 wait complete. 2024-03-06T07:29:56.210 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:29:56.210 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:29:56.999 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:57.211 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:57.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.340+0000 7f4acb93e640 1 Processor -- start 2024-03-06T07:29:57.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.340+0000 7f4acb93e640 1 -- start start 2024-03-06T07:29:57.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.341+0000 7f4acb93e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4ac40d22b0 0x7f4ac40d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.341+0000 7f4acb93e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4ac40b41f0 0x7f4ac40b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.341+0000 7f4acb93e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ac40b3430 con 0x7f4ac415dfe0 2024-03-06T07:29:57.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.341+0000 7f4acb93e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ac40d2c50 con 0x7f4ac40b41f0 2024-03-06T07:29:57.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.341+0000 7f4acb93e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac40b31e0 con 0x7f4ac40d22b0 2024-03-06T07:29:57.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.341+0000 7f4ac9edb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4ac40b41f0 0x7f4ac40b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.341+0000 7f4ac9edb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4ac40b41f0 0x7f4ac40b4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59572/0 (socket says 172.21.15.83:59572) 2024-03-06T07:29:57.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.341+0000 7f4ac9edb640 1 -- 172.21.15.83:0/177455 learned_addr learned my addr 172.21.15.83:0/177455 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:57.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.342+0000 7f4ac8ed9640 1 --2- 172.21.15.83:0/177455 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4ac40d22b0 0x7f4ac40d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.342+0000 7f4ac9edb640 1 -- 172.21.15.83:0/177455 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4ac40d22b0 msgr2=0x7f4ac40d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:57.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.342+0000 7f4ac9edb640 1 --2- 172.21.15.83:0/177455 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4ac40d22b0 0x7f4ac40d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.342+0000 7f4ac9edb640 1 -- 172.21.15.83:0/177455 >> v1:172.21.15.183:6789/0 conn(0x7f4ac415dfe0 legacy=0x7f4ac40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.342+0000 7f4ac9edb640 1 -- 172.21.15.83:0/177455 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ac40b3710 con 0x7f4ac40b41f0 2024-03-06T07:29:57.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.342+0000 7f4ac8ed9640 1 --2- 172.21.15.83:0/177455 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4ac40d22b0 0x7f4ac40d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:57.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.342+0000 7f4ac9edb640 1 --2- 172.21.15.83:0/177455 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ac40b41f0 0x7f4ac40b4650 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f4abc009e50 tx=0x7f4abc02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adcff6c92968f094 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.343+0000 7f4abbfff640 1 -- 172.21.15.83:0/177455 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4abc02cb00 con 0x7f4ac40b41f0 2024-03-06T07:29:57.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.343+0000 7f4abbfff640 1 -- 172.21.15.83:0/177455 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4abc02cca0 con 0x7f4ac40b41f0 2024-03-06T07:29:57.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.343+0000 7f4abbfff640 1 -- 172.21.15.83:0/177455 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4abc034610 con 0x7f4ac40b41f0 2024-03-06T07:29:57.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.343+0000 7f4acb93e640 1 -- 172.21.15.83:0/177455 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ac40b41f0 msgr2=0x7f4ac40b4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.343+0000 7f4acb93e640 1 --2- 172.21.15.83:0/177455 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ac40b41f0 0x7f4ac40b4650 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f4abc009e50 tx=0x7f4abc02bf60 comp rx=0 tx=0).stop 2024-03-06T07:29:57.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.343+0000 7f4acb93e640 1 -- 172.21.15.83:0/177455 shutdown_connections 2024-03-06T07:29:57.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.343+0000 7f4acb93e640 1 --2- 172.21.15.83:0/177455 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ac40b41f0 0x7f4ac40b4650 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.343+0000 7f4acb93e640 1 --2- 172.21.15.83:0/177455 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4ac40d22b0 0x7f4ac40d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.343+0000 7f4acb93e640 1 -- 172.21.15.83:0/177455 >> 172.21.15.83:0/177455 conn(0x7f4ac40d3cd0 msgr2=0x7f4ac40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:57.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.343+0000 7f4acb93e640 1 -- 172.21.15.83:0/177455 shutdown_connections 2024-03-06T07:29:57.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.343+0000 7f4acb93e640 1 -- 172.21.15.83:0/177455 wait complete. 2024-03-06T07:29:57.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.344+0000 7f4acb93e640 1 Processor -- start 2024-03-06T07:29:57.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.344+0000 7f4acb93e640 1 -- start start 2024-03-06T07:29:57.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.345+0000 7f4acb93e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4ac40b41f0 0x7f4ac40caef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.345+0000 7f4acb93e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4ac40d22b0 0x7f4ac40cd9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.345+0000 7f4acb93e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ac415dfe0 0x7f4ac41f6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.345+0000 7f4acb93e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac40cfa00 con 0x7f4ac415dfe0 2024-03-06T07:29:57.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.345+0000 7f4acb93e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac40b3d60 con 0x7f4ac40d22b0 2024-03-06T07:29:57.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.345+0000 7f4acb93e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac40d1c70 con 0x7f4ac40b41f0 2024-03-06T07:29:57.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.345+0000 7f4ac9edb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ac415dfe0 0x7f4ac41f6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.345+0000 7f4ac9edb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ac415dfe0 0x7f4ac41f6ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59580/0 (socket says 172.21.15.83:59580) 2024-03-06T07:29:57.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.345+0000 7f4ac9edb640 1 -- 172.21.15.83:0/353096810 learned_addr learned my addr 172.21.15.83:0/353096810 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:57.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.345+0000 7f4ac96da640 1 --2- 172.21.15.83:0/353096810 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4ac40b41f0 0x7f4ac40caef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.345+0000 7f4ac8ed9640 1 --2- 172.21.15.83:0/353096810 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4ac40d22b0 0x7f4ac40cd9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.346+0000 7f4ac9edb640 1 -- 172.21.15.83:0/353096810 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4ac40b41f0 msgr2=0x7f4ac40caef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.346+0000 7f4ac9edb640 1 --2- 172.21.15.83:0/353096810 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4ac40b41f0 0x7f4ac40caef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.346+0000 7f4ac9edb640 1 -- 172.21.15.83:0/353096810 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4ac40d22b0 msgr2=0x7f4ac40cd9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.346+0000 7f4ac9edb640 1 --2- 172.21.15.83:0/353096810 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4ac40d22b0 0x7f4ac40cd9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.346+0000 7f4ac9edb640 1 -- 172.21.15.83:0/353096810 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ac40cc930 con 0x7f4ac415dfe0 2024-03-06T07:29:57.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.346+0000 7f4ac96da640 1 --2- 172.21.15.83:0/353096810 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4ac40b41f0 0x7f4ac40caef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:57.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.346+0000 7f4ac8ed9640 1 --2- 172.21.15.83:0/353096810 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4ac40d22b0 0x7f4ac40cd9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:57.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.346+0000 7f4ac9edb640 1 --2- 172.21.15.83:0/353096810 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ac415dfe0 0x7f4ac41f6ff0 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f4abc009e50 tx=0x7f4abc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.346+0000 7f4ab9ffb640 1 -- 172.21.15.83:0/353096810 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4abc043020 con 0x7f4ac415dfe0 2024-03-06T07:29:57.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.346+0000 7f4ab9ffb640 1 -- 172.21.15.83:0/353096810 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4abc02ce50 con 0x7f4ac415dfe0 2024-03-06T07:29:57.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.346+0000 7f4ab9ffb640 1 -- 172.21.15.83:0/353096810 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4abc009510 con 0x7f4ac415dfe0 2024-03-06T07:29:57.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.346+0000 7f4acb93e640 1 -- 172.21.15.83:0/353096810 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ac40ccc10 con 0x7f4ac415dfe0 2024-03-06T07:29:57.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.346+0000 7f4acb93e640 1 -- 172.21.15.83:0/353096810 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ac40cd1d0 con 0x7f4ac415dfe0 2024-03-06T07:29:57.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.347+0000 7f4acb93e640 1 -- 172.21.15.83:0/353096810 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ac404f330 con 0x7f4ac415dfe0 2024-03-06T07:29:57.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.348+0000 7f4ab9ffb640 1 -- 172.21.15.83:0/353096810 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4abc046020 con 0x7f4ac415dfe0 2024-03-06T07:29:57.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.348+0000 7f4ab9ffb640 1 --2- 172.21.15.83:0/353096810 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4aa806b4d0 0x7f4aa806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.348+0000 7f4ab9ffb640 1 -- 172.21.15.83:0/353096810 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4abc0ac7d0 con 0x7f4ac415dfe0 2024-03-06T07:29:57.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.348+0000 7f4ac96da640 1 --2- 172.21.15.83:0/353096810 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4aa806b4d0 0x7f4aa806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.348+0000 7f4ac96da640 1 --2- 172.21.15.83:0/353096810 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4aa806b4d0 0x7f4aa806d990 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f4ac0007fd0 tx=0x7f4ac0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.352+0000 7f4ab9ffb640 1 -- 172.21.15.83:0/353096810 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4abc04d030 con 0x7f4ac415dfe0 2024-03-06T07:29:57.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.560+0000 7f4acb93e640 1 -- 172.21.15.83:0/353096810 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ac40cea10 con 0x7f4ac415dfe0 2024-03-06T07:29:57.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.560+0000 7f4ab9ffb640 1 -- 172.21.15.83:0/353096810 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f4abc07b6a0 con 0x7f4ac415dfe0 2024-03-06T07:29:57.562 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:29:57.563 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:29:57.563 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:29:57.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.563+0000 7f4acb93e640 1 -- 172.21.15.83:0/353096810 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4aa806b4d0 msgr2=0x7f4aa806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.563+0000 7f4acb93e640 1 --2- 172.21.15.83:0/353096810 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4aa806b4d0 0x7f4aa806d990 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f4ac0007fd0 tx=0x7f4ac0009040 comp rx=0 tx=0).stop 2024-03-06T07:29:57.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.563+0000 7f4acb93e640 1 -- 172.21.15.83:0/353096810 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ac415dfe0 msgr2=0x7f4ac41f6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.563+0000 7f4acb93e640 1 --2- 172.21.15.83:0/353096810 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ac415dfe0 0x7f4ac41f6ff0 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f4abc009e50 tx=0x7f4abc004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:57.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.563+0000 7f4acb93e640 1 -- 172.21.15.83:0/353096810 shutdown_connections 2024-03-06T07:29:57.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.563+0000 7f4acb93e640 1 --2- 172.21.15.83:0/353096810 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4aa806b4d0 0x7f4aa806d990 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.563+0000 7f4acb93e640 1 --2- 172.21.15.83:0/353096810 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ac415dfe0 0x7f4ac41f6ff0 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.563+0000 7f4acb93e640 1 --2- 172.21.15.83:0/353096810 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4ac40d22b0 0x7f4ac40cd9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.563+0000 7f4acb93e640 1 --2- 172.21.15.83:0/353096810 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4ac40b41f0 0x7f4ac40caef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.563+0000 7f4acb93e640 1 -- 172.21.15.83:0/353096810 >> 172.21.15.83:0/353096810 conn(0x7f4ac40d3cd0 msgr2=0x7f4ac40cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:57.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.563+0000 7f4acb93e640 1 -- 172.21.15.83:0/353096810 shutdown_connections 2024-03-06T07:29:57.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:57.563+0000 7f4acb93e640 1 -- 172.21.15.83:0/353096810 wait complete. 2024-03-06T07:29:57.777 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:29:57.777 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:29:58.778 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:58.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.878+0000 7ff55d28a640 1 Processor -- start 2024-03-06T07:29:58.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.878+0000 7ff55d28a640 1 -- start start 2024-03-06T07:29:58.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.878+0000 7ff55d28a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff55815dfe0 0x7ff5580cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.878+0000 7ff55d28a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ff5580cf830 0x7ff5580d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.878+0000 7ff55d28a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5580b3430 con 0x7ff5580d27f0 2024-03-06T07:29:58.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.878+0000 7ff55d28a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5580d2c50 con 0x7ff5580cf830 2024-03-06T07:29:58.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.878+0000 7ff55d28a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5580b31e0 con 0x7ff55815dfe0 2024-03-06T07:29:58.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.878+0000 7ff5577fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff5580d27f0 0x7ff5580d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49868/0 (socket says 172.21.15.83:49868) 2024-03-06T07:29:58.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.878+0000 7ff5577fe640 1 -- 172.21.15.83:0/2397748947 learned_addr learned my addr 172.21.15.83:0/2397748947 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:29:58.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.879+0000 7ff54ffff640 1 --2- 172.21.15.83:0/2397748947 >> v2:172.21.15.183:3300/0 conn(0x7ff5580cf830 0x7ff5580d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.879+0000 7ff556ffd640 1 --2- 172.21.15.83:0/2397748947 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff55815dfe0 0x7ff5580cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.879+0000 7ff5567fc640 1 -- 172.21.15.83:0/2397748947 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 110037444 0 0) 0x7ff5580b3430 con 0x7ff5580d27f0 2024-03-06T07:29:58.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.879+0000 7ff5567fc640 1 -- 172.21.15.83:0/2397748947 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff544003520 con 0x7ff5580d27f0 2024-03-06T07:29:58.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.879+0000 7ff5567fc640 1 -- 172.21.15.83:0/2397748947 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4143892854 0 0) 0x7ff544003520 con 0x7ff5580d27f0 2024-03-06T07:29:58.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.879+0000 7ff5567fc640 1 -- 172.21.15.83:0/2397748947 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff55815dfe0 msgr2=0x7ff5580cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:58.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.879+0000 7ff5567fc640 1 --2- 172.21.15.83:0/2397748947 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff55815dfe0 0x7ff5580cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.879+0000 7ff5567fc640 1 -- 172.21.15.83:0/2397748947 >> v2:172.21.15.183:3300/0 conn(0x7ff5580cf830 msgr2=0x7ff5580d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:58.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.879+0000 7ff5567fc640 1 --2- 172.21.15.83:0/2397748947 >> v2:172.21.15.183:3300/0 conn(0x7ff5580cf830 0x7ff5580d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.879+0000 7ff5567fc640 1 -- 172.21.15.83:0/2397748947 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5580b3710 con 0x7ff5580d27f0 2024-03-06T07:29:58.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.879+0000 7ff5567fc640 1 -- 172.21.15.83:0/2397748947 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7ff5500019f0 con 0x7ff5580d27f0 2024-03-06T07:29:58.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.879+0000 7ff5567fc640 1 -- 172.21.15.83:0/2397748947 >> v1:172.21.15.83:6789/0 conn(0x7ff5580d27f0 legacy=0x7ff5580d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.879+0000 7ff5567fc640 1 --2- 172.21.15.83:0/2397748947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5440039b0 0x7ff544003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.879+0000 7ff5567fc640 1 --2- 172.21.15.83:0/2397748947 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff55815dfe0 0x7ff544004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.879+0000 7ff5567fc640 1 --2- 172.21.15.83:0/2397748947 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff544004d40 0x7ff5440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.879+0000 7ff5567fc640 1 -- 172.21.15.83:0/2397748947 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5580b31e0 con 0x7ff5440039b0 2024-03-06T07:29:58.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.879+0000 7ff5567fc640 1 -- 172.21.15.83:0/2397748947 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff54400b9c0 con 0x7ff544004d40 2024-03-06T07:29:58.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.879+0000 7ff5567fc640 1 -- 172.21.15.83:0/2397748947 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff54400bb40 con 0x7ff55815dfe0 2024-03-06T07:29:58.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.879+0000 7ff556ffd640 1 --2- 172.21.15.83:0/2397748947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5440039b0 0x7ff544003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.879+0000 7ff54ffff640 1 --2- 172.21.15.83:0/2397748947 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff55815dfe0 0x7ff544004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.879+0000 7ff5577fe640 1 --2- 172.21.15.83:0/2397748947 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff544004d40 0x7ff5440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.879+0000 7ff556ffd640 1 -- 172.21.15.83:0/2397748947 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff55815dfe0 msgr2=0x7ff544004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:58.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.879+0000 7ff556ffd640 1 --2- 172.21.15.83:0/2397748947 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff55815dfe0 0x7ff544004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.879+0000 7ff556ffd640 1 -- 172.21.15.83:0/2397748947 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff544004d40 msgr2=0x7ff5440051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:58.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.879+0000 7ff556ffd640 1 --2- 172.21.15.83:0/2397748947 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff544004d40 0x7ff5440051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.879+0000 7ff556ffd640 1 -- 172.21.15.83:0/2397748947 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff544003520 con 0x7ff5440039b0 2024-03-06T07:29:58.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.880+0000 7ff54ffff640 1 --2- 172.21.15.83:0/2397748947 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff55815dfe0 0x7ff544004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:58.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.880+0000 7ff5577fe640 1 --2- 172.21.15.83:0/2397748947 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff544004d40 0x7ff5440051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:58.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.880+0000 7ff556ffd640 1 --2- 172.21.15.83:0/2397748947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5440039b0 0x7ff544003dd0 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7ff548009ac0 tx=0x7ff544005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdd888caae9a3b9b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.880+0000 7ff5567fc640 1 -- 172.21.15.83:0/2397748947 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff548039070 con 0x7ff5440039b0 2024-03-06T07:29:58.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.880+0000 7ff5567fc640 1 -- 172.21.15.83:0/2397748947 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff54802c6b0 con 0x7ff5440039b0 2024-03-06T07:29:58.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.880+0000 7ff5567fc640 1 -- 172.21.15.83:0/2397748947 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff548009690 con 0x7ff5440039b0 2024-03-06T07:29:58.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.881+0000 7ff55d28a640 1 -- 172.21.15.83:0/2397748947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5440039b0 msgr2=0x7ff544003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.881+0000 7ff55d28a640 1 --2- 172.21.15.83:0/2397748947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5440039b0 0x7ff544003dd0 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7ff548009ac0 tx=0x7ff544005830 comp rx=0 tx=0).stop 2024-03-06T07:29:58.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.881+0000 7ff5577fe640 1 -- 172.21.15.83:0/2397748947 reap_dead start 2024-03-06T07:29:58.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.881+0000 7ff55d28a640 1 -- 172.21.15.83:0/2397748947 shutdown_connections 2024-03-06T07:29:58.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.881+0000 7ff55d28a640 1 -- 172.21.15.83:0/2397748947 >> 172.21.15.83:0/2397748947 conn(0x7ff5580d3cd0 msgr2=0x7ff5580d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:58.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.881+0000 7ff55d28a640 1 -- 172.21.15.83:0/2397748947 shutdown_connections 2024-03-06T07:29:58.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.881+0000 7ff55d28a640 1 -- 172.21.15.83:0/2397748947 wait complete. 2024-03-06T07:29:58.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.882+0000 7ff55d28a640 1 Processor -- start 2024-03-06T07:29:58.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.882+0000 7ff55d28a640 1 -- start start 2024-03-06T07:29:58.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.883+0000 7ff55d28a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff5440039b0 0x7ff5580cc160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.883+0000 7ff55d28a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5580cb270 0x7ff5580b1520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.883+0000 7ff55d28a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff5580b1a60 0x7ff55815f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.883+0000 7ff55d28a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5580d1c70 con 0x7ff5580cb270 2024-03-06T07:29:58.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.883+0000 7ff55d28a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5580b3d60 con 0x7ff5440039b0 2024-03-06T07:29:58.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.883+0000 7ff55d28a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5580d1df0 con 0x7ff5580b1a60 2024-03-06T07:29:58.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.883+0000 7ff54ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5580cb270 0x7ff5580b1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.883+0000 7ff54ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5580cb270 0x7ff5580b1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59590/0 (socket says 172.21.15.83:59590) 2024-03-06T07:29:58.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.883+0000 7ff54ffff640 1 -- 172.21.15.83:0/2902322849 learned_addr learned my addr 172.21.15.83:0/2902322849 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:29:58.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.883+0000 7ff5577fe640 1 --2- 172.21.15.83:0/2902322849 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff5580b1a60 0x7ff55815f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.883+0000 7ff54ffff640 1 -- 172.21.15.83:0/2902322849 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff5580b1a60 msgr2=0x7ff55815f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.883+0000 7ff556ffd640 1 --2- 172.21.15.83:0/2902322849 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff5440039b0 0x7ff5580cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.883+0000 7ff54ffff640 1 --2- 172.21.15.83:0/2902322849 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff5580b1a60 0x7ff55815f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.883+0000 7ff54ffff640 1 -- 172.21.15.83:0/2902322849 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff5440039b0 msgr2=0x7ff5580cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.883+0000 7ff54ffff640 1 --2- 172.21.15.83:0/2902322849 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff5440039b0 0x7ff5580cc160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.883+0000 7ff54ffff640 1 -- 172.21.15.83:0/2902322849 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5580cc9e0 con 0x7ff5580cb270 2024-03-06T07:29:58.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.883+0000 7ff54ffff640 1 --2- 172.21.15.83:0/2902322849 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5580cb270 0x7ff5580b1520 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7ff540011660 tx=0x7ff540011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.884+0000 7ff54f7fe640 1 -- 172.21.15.83:0/2902322849 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54000f9c0 con 0x7ff5580cb270 2024-03-06T07:29:58.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.884+0000 7ff54f7fe640 1 -- 172.21.15.83:0/2902322849 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff54000fb60 con 0x7ff5580cb270 2024-03-06T07:29:58.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.884+0000 7ff55d28a640 1 -- 172.21.15.83:0/2902322849 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5580cccc0 con 0x7ff5580cb270 2024-03-06T07:29:58.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.884+0000 7ff54f7fe640 1 -- 172.21.15.83:0/2902322849 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff540015460 con 0x7ff5580cb270 2024-03-06T07:29:58.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.884+0000 7ff55d28a640 1 -- 172.21.15.83:0/2902322849 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5580cd280 con 0x7ff5580cb270 2024-03-06T07:29:58.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.885+0000 7ff55d28a640 1 -- 172.21.15.83:0/2902322849 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5580b6650 con 0x7ff5580cb270 2024-03-06T07:29:58.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.885+0000 7ff54f7fe640 1 -- 172.21.15.83:0/2902322849 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff540015600 con 0x7ff5580cb270 2024-03-06T07:29:58.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.885+0000 7ff54f7fe640 1 --2- 172.21.15.83:0/2902322849 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff53806b4d0 0x7ff53806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.886+0000 7ff54f7fe640 1 -- 172.21.15.83:0/2902322849 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff54008ac30 con 0x7ff5580cb270 2024-03-06T07:29:58.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.886+0000 7ff556ffd640 1 --2- 172.21.15.83:0/2902322849 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff53806b4d0 0x7ff53806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.886+0000 7ff556ffd640 1 --2- 172.21.15.83:0/2902322849 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff53806b4d0 0x7ff53806d990 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7ff548036000 tx=0x7ff548030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:58.890+0000 7ff54f7fe640 1 -- 172.21.15.83:0/2902322849 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff5580b6650 con 0x7ff5580cb270 2024-03-06T07:29:59.001 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:59.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:59.101+0000 7ff55d28a640 1 -- 172.21.15.83:0/2902322849 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff55804f330 con 0x7ff5580cb270 2024-03-06T07:29:59.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:59.101+0000 7ff54f7fe640 1 -- 172.21.15.83:0/2902322849 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7ff540019030 con 0x7ff5580cb270 2024-03-06T07:29:59.103 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:29:59.103 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:29:59.104 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:29:59.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:59.103+0000 7ff55d28a640 1 -- 172.21.15.83:0/2902322849 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff53806b4d0 msgr2=0x7ff53806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:59.103+0000 7ff55d28a640 1 --2- 172.21.15.83:0/2902322849 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff53806b4d0 0x7ff53806d990 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7ff548036000 tx=0x7ff548030000 comp rx=0 tx=0).stop 2024-03-06T07:29:59.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:59.103+0000 7ff55d28a640 1 -- 172.21.15.83:0/2902322849 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5580cb270 msgr2=0x7ff5580b1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:59.103+0000 7ff55d28a640 1 --2- 172.21.15.83:0/2902322849 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5580cb270 0x7ff5580b1520 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7ff540011660 tx=0x7ff540011b20 comp rx=0 tx=0).stop 2024-03-06T07:29:59.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:59.103+0000 7ff55d28a640 1 -- 172.21.15.83:0/2902322849 shutdown_connections 2024-03-06T07:29:59.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:59.103+0000 7ff55d28a640 1 --2- 172.21.15.83:0/2902322849 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff53806b4d0 0x7ff53806d990 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:59.103+0000 7ff55d28a640 1 --2- 172.21.15.83:0/2902322849 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff5580b1a60 0x7ff55815f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:59.103+0000 7ff55d28a640 1 --2- 172.21.15.83:0/2902322849 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff5580cb270 0x7ff5580b1520 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:59.103+0000 7ff55d28a640 1 --2- 172.21.15.83:0/2902322849 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff5440039b0 0x7ff5580cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:59.103+0000 7ff55d28a640 1 -- 172.21.15.83:0/2902322849 >> 172.21.15.83:0/2902322849 conn(0x7ff5580d3cd0 msgr2=0x7ff5580b47b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:59.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:59.103+0000 7ff55d28a640 1 -- 172.21.15.83:0/2902322849 shutdown_connections 2024-03-06T07:29:59.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:29:59.103+0000 7ff55d28a640 1 -- 172.21.15.83:0/2902322849 wait complete. 2024-03-06T07:29:59.317 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:29:59.317 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:30:00.318 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:00.389 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:00.392 DEBUG:teuthology.orchestra.run.smithi183:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:00.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.421+0000 7f170baa4640 1 Processor -- start 2024-03-06T07:30:00.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.421+0000 7f170baa4640 1 -- start start 2024-03-06T07:30:00.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.422+0000 7f170baa4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f17040cd610 0x7f17040d1060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.422+0000 7f170baa4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f17040d15a0 0x7f170406afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.422+0000 7f170baa4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17040afd40 con 0x7f170415bb90 2024-03-06T07:30:00.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.422+0000 7f170baa4640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17040d0bd0 con 0x7f17040d15a0 2024-03-06T07:30:00.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.422+0000 7f170baa4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17040b0020 con 0x7f17040cd610 2024-03-06T07:30:00.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.422+0000 7f170a041640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f17040d15a0 0x7f170406afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.422+0000 7f170a041640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f17040d15a0 0x7f170406afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59598/0 (socket says 172.21.15.83:59598) 2024-03-06T07:30:00.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.422+0000 7f170a041640 1 -- 172.21.15.83:0/2788905311 learned_addr learned my addr 172.21.15.83:0/2788905311 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:00.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.422+0000 7f170903f640 1 --2- 172.21.15.83:0/2788905311 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f17040cd610 0x7f17040d1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.422+0000 7f170a041640 1 -- 172.21.15.83:0/2788905311 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f17040cd610 msgr2=0x7f17040d1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:00.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.422+0000 7f170a041640 1 --2- 172.21.15.83:0/2788905311 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f17040cd610 0x7f17040d1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.422+0000 7f170a041640 1 -- 172.21.15.83:0/2788905311 >> v1:172.21.15.183:6789/0 conn(0x7f170415bb90 legacy=0x7f170406f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.422+0000 7f170a041640 1 -- 172.21.15.83:0/2788905311 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17040d0860 con 0x7f17040d15a0 2024-03-06T07:30:00.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.423+0000 7f170a041640 1 --2- 172.21.15.83:0/2788905311 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17040d15a0 0x7f170406afb0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f16fc009e50 tx=0x7f16fc02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7ae837145b78ada server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.423+0000 7f170883e640 1 -- 172.21.15.83:0/2788905311 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16fc02cb00 con 0x7f17040d15a0 2024-03-06T07:30:00.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.423+0000 7f170883e640 1 -- 172.21.15.83:0/2788905311 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16fc02cca0 con 0x7f17040d15a0 2024-03-06T07:30:00.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.423+0000 7f170883e640 1 -- 172.21.15.83:0/2788905311 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16fc034610 con 0x7f17040d15a0 2024-03-06T07:30:00.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.424+0000 7f170baa4640 1 -- 172.21.15.83:0/2788905311 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17040d15a0 msgr2=0x7f170406afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.424+0000 7f170baa4640 1 --2- 172.21.15.83:0/2788905311 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17040d15a0 0x7f170406afb0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f16fc009e50 tx=0x7f16fc02bf60 comp rx=0 tx=0).stop 2024-03-06T07:30:00.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.424+0000 7f170baa4640 1 -- 172.21.15.83:0/2788905311 shutdown_connections 2024-03-06T07:30:00.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.424+0000 7f170baa4640 1 --2- 172.21.15.83:0/2788905311 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17040d15a0 0x7f170406afb0 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.424+0000 7f170baa4640 1 --2- 172.21.15.83:0/2788905311 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f17040cd610 0x7f17040d1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.424+0000 7f170baa4640 1 -- 172.21.15.83:0/2788905311 >> 172.21.15.83:0/2788905311 conn(0x7f170406c0c0 msgr2=0x7f170406c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:00.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.424+0000 7f170baa4640 1 -- 172.21.15.83:0/2788905311 shutdown_connections 2024-03-06T07:30:00.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.424+0000 7f170baa4640 1 -- 172.21.15.83:0/2788905311 wait complete. 2024-03-06T07:30:00.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.425+0000 7f170baa4640 1 Processor -- start 2024-03-06T07:30:00.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.425+0000 7f170baa4640 1 -- start start 2024-03-06T07:30:00.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.426+0000 7f170baa4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17040cd610 0x7f17041496d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.426+0000 7f170baa4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f170415bb90 0x7f17041f42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.426+0000 7f170baa4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f17040b4d60 0x7f17040b5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.426+0000 7f170baa4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f170406b670 con 0x7f170415bb90 2024-03-06T07:30:00.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.426+0000 7f170baa4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f170406b4f0 con 0x7f17040cd610 2024-03-06T07:30:00.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.426+0000 7f170baa4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f170406b7f0 con 0x7f17040b4d60 2024-03-06T07:30:00.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.426+0000 7f170903f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f170415bb90 0x7f17041f42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.426+0000 7f170903f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f170415bb90 0x7f17041f42c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59606/0 (socket says 172.21.15.83:59606) 2024-03-06T07:30:00.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.426+0000 7f170903f640 1 -- 172.21.15.83:0/3866237412 learned_addr learned my addr 172.21.15.83:0/3866237412 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:00.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.426+0000 7f170a041640 1 --2- 172.21.15.83:0/3866237412 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f17040b4d60 0x7f17040b5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.426+0000 7f1709840640 1 --2- 172.21.15.83:0/3866237412 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17040cd610 0x7f17041496d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.426+0000 7f170903f640 1 -- 172.21.15.83:0/3866237412 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f17040b4d60 msgr2=0x7f17040b5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.426+0000 7f170903f640 1 --2- 172.21.15.83:0/3866237412 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f17040b4d60 0x7f17040b5210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.426+0000 7f170903f640 1 -- 172.21.15.83:0/3866237412 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17040cd610 msgr2=0x7f17041496d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.426+0000 7f170903f640 1 --2- 172.21.15.83:0/3866237412 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17040cd610 0x7f17041496d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.426+0000 7f170903f640 1 -- 172.21.15.83:0/3866237412 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17040ad990 con 0x7f170415bb90 2024-03-06T07:30:00.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.426+0000 7f1709840640 1 --2- 172.21.15.83:0/3866237412 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17040cd610 0x7f17041496d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:00.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.426+0000 7f170903f640 1 --2- 172.21.15.83:0/3866237412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f170415bb90 0x7f17041f42c0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f16f400d800 tx=0x7f16f400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.426+0000 7f16fa7fc640 1 -- 172.21.15.83:0/3866237412 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f400b6c0 con 0x7f170415bb90 2024-03-06T07:30:00.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.426+0000 7f16fa7fc640 1 -- 172.21.15.83:0/3866237412 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16f4019070 con 0x7f170415bb90 2024-03-06T07:30:00.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.427+0000 7f170baa4640 1 -- 172.21.15.83:0/3866237412 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17040adc70 con 0x7f170415bb90 2024-03-06T07:30:00.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.427+0000 7f16fa7fc640 1 -- 172.21.15.83:0/3866237412 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f400ba80 con 0x7f170415bb90 2024-03-06T07:30:00.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.427+0000 7f170baa4640 1 -- 172.21.15.83:0/3866237412 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17040ae230 con 0x7f170415bb90 2024-03-06T07:30:00.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.428+0000 7f170baa4640 1 -- 172.21.15.83:0/3866237412 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f170404f330 con 0x7f170415bb90 2024-03-06T07:30:00.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.428+0000 7f16fa7fc640 1 -- 172.21.15.83:0/3866237412 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f16f400bc20 con 0x7f170415bb90 2024-03-06T07:30:00.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.428+0000 7f16fa7fc640 1 --2- 172.21.15.83:0/3866237412 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f16e806b500 0x7f16e806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.428+0000 7f16fa7fc640 1 -- 172.21.15.83:0/3866237412 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f16f408a5c0 con 0x7f170415bb90 2024-03-06T07:30:00.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.428+0000 7f1709840640 1 --2- 172.21.15.83:0/3866237412 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f16e806b500 0x7f16e806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.429+0000 7f1709840640 1 --2- 172.21.15.83:0/3866237412 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f16e806b500 0x7f16e806d9c0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f17040d03e0 tx=0x7f1700009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.433+0000 7f16fa7fc640 1 -- 172.21.15.83:0/3866237412 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f16f4059490 con 0x7f170415bb90 2024-03-06T07:30:00.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.663+0000 7f170baa4640 1 -- 172.21.15.83:0/3866237412 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1704071f30 con 0x7f170415bb90 2024-03-06T07:30:00.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.664+0000 7f16fa7fc640 1 -- 172.21.15.83:0/3866237412 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f16f4058d90 con 0x7f170415bb90 2024-03-06T07:30:00.666 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:30:00.666 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:30:00.667 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:30:00.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.666+0000 7f170baa4640 1 -- 172.21.15.83:0/3866237412 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f16e806b500 msgr2=0x7f16e806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.666+0000 7f170baa4640 1 --2- 172.21.15.83:0/3866237412 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f16e806b500 0x7f16e806d9c0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f17040d03e0 tx=0x7f1700009040 comp rx=0 tx=0).stop 2024-03-06T07:30:00.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.666+0000 7f170baa4640 1 -- 172.21.15.83:0/3866237412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f170415bb90 msgr2=0x7f17041f42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.666+0000 7f170baa4640 1 --2- 172.21.15.83:0/3866237412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f170415bb90 0x7f17041f42c0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f16f400d800 tx=0x7f16f400dcc0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.666+0000 7f170baa4640 1 -- 172.21.15.83:0/3866237412 shutdown_connections 2024-03-06T07:30:00.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.666+0000 7f170baa4640 1 --2- 172.21.15.83:0/3866237412 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f16e806b500 0x7f16e806d9c0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.666+0000 7f170baa4640 1 --2- 172.21.15.83:0/3866237412 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f17040b4d60 0x7f17040b5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.666+0000 7f170baa4640 1 --2- 172.21.15.83:0/3866237412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f170415bb90 0x7f17041f42c0 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.666+0000 7f170baa4640 1 --2- 172.21.15.83:0/3866237412 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17040cd610 0x7f17041496d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.666+0000 7f170baa4640 1 -- 172.21.15.83:0/3866237412 >> 172.21.15.83:0/3866237412 conn(0x7f170406c0c0 msgr2=0x7f170406e670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:00.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.666+0000 7f170baa4640 1 -- 172.21.15.83:0/3866237412 shutdown_connections 2024-03-06T07:30:00.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:00.666+0000 7f170baa4640 1 -- 172.21.15.83:0/3866237412 wait complete. 2024-03-06T07:30:00.882 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:30:00.883 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:30:01.004 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:01.884 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:01.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.984+0000 7f824578a640 1 Processor -- start 2024-03-06T07:30:01.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.984+0000 7f824578a640 1 -- start start 2024-03-06T07:30:01.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.985+0000 7f824578a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f82401424f0 0x7f82400cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.985+0000 7f824578a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f824006da40 0x7f824015cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.985+0000 7f824578a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82400cc980 con 0x7f82400cd7a0 2024-03-06T07:30:01.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.985+0000 7f824578a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82400cd220 con 0x7f824006da40 2024-03-06T07:30:01.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.985+0000 7f824578a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82400ccc60 con 0x7f82401424f0 2024-03-06T07:30:01.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.985+0000 7f823e7fc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f82400cd7a0 0x7f824006d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49870/0 (socket says 172.21.15.83:49870) 2024-03-06T07:30:01.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.985+0000 7f823e7fc640 1 -- 172.21.15.83:0/1279407134 learned_addr learned my addr 172.21.15.83:0/1279407134 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:30:01.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.985+0000 7f823effd640 1 --2- 172.21.15.83:0/1279407134 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f82401424f0 0x7f82400cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.986+0000 7f823f7fe640 1 --2- 172.21.15.83:0/1279407134 >> v2:172.21.15.183:3300/0 conn(0x7f824006da40 0x7f824015cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.986+0000 7f823dffb640 1 -- 172.21.15.83:0/1279407134 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2917639423 0 0) 0x7f82400cc980 con 0x7f82400cd7a0 2024-03-06T07:30:01.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.986+0000 7f823dffb640 1 -- 172.21.15.83:0/1279407134 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8228003520 con 0x7f82400cd7a0 2024-03-06T07:30:01.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.986+0000 7f823effd640 1 -- 172.21.15.83:0/1279407134 >> v2:172.21.15.183:3300/0 conn(0x7f824006da40 msgr2=0x7f824015cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:01.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.986+0000 7f823effd640 1 --2- 172.21.15.83:0/1279407134 >> v2:172.21.15.183:3300/0 conn(0x7f824006da40 0x7f824015cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.986+0000 7f823effd640 1 -- 172.21.15.83:0/1279407134 >> v1:172.21.15.83:6789/0 conn(0x7f82400cd7a0 legacy=0x7f824006d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.986+0000 7f823effd640 1 -- 172.21.15.83:0/1279407134 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82400ccf40 con 0x7f82401424f0 2024-03-06T07:30:01.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.986+0000 7f823f7fe640 1 --2- 172.21.15.83:0/1279407134 >> v2:172.21.15.183:3300/0 conn(0x7f824006da40 0x7f824015cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:01.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.987+0000 7f823effd640 1 --2- 172.21.15.83:0/1279407134 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f82401424f0 0x7f82400cc2f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f82300097a0 tx=0x7f823002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52f189a19340c8dc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.987+0000 7f823dffb640 1 -- 172.21.15.83:0/1279407134 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8230035020 con 0x7f82401424f0 2024-03-06T07:30:01.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.987+0000 7f823dffb640 1 -- 172.21.15.83:0/1279407134 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8230031810 con 0x7f82401424f0 2024-03-06T07:30:01.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.987+0000 7f823dffb640 1 -- 172.21.15.83:0/1279407134 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8230031bc0 con 0x7f82401424f0 2024-03-06T07:30:01.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.987+0000 7f824578a640 1 -- 172.21.15.83:0/1279407134 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f82401424f0 msgr2=0x7f82400cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.987+0000 7f824578a640 1 --2- 172.21.15.83:0/1279407134 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f82401424f0 0x7f82400cc2f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f82300097a0 tx=0x7f823002b8b0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.987+0000 7f824578a640 1 -- 172.21.15.83:0/1279407134 shutdown_connections 2024-03-06T07:30:01.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.987+0000 7f824578a640 1 --2- 172.21.15.83:0/1279407134 >> v2:172.21.15.183:3300/0 conn(0x7f824006da40 0x7f824015cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.987+0000 7f824578a640 1 --2- 172.21.15.83:0/1279407134 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f82401424f0 0x7f82400cc2f0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.987+0000 7f824578a640 1 -- 172.21.15.83:0/1279407134 >> 172.21.15.83:0/1279407134 conn(0x7f82400cf8a0 msgr2=0x7f82400cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:01.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.987+0000 7f824578a640 1 -- 172.21.15.83:0/1279407134 shutdown_connections 2024-03-06T07:30:01.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.988+0000 7f824578a640 1 -- 172.21.15.83:0/1279407134 wait complete. 2024-03-06T07:30:01.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.989+0000 7f824578a640 1 Processor -- start 2024-03-06T07:30:01.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.989+0000 7f824578a640 1 -- start start 2024-03-06T07:30:01.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.990+0000 7f824578a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f824006da40 0x7f82400aee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.990+0000 7f824578a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82400cd7a0 0x7f8240144ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.990+0000 7f824578a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82401424f0 0x7f82400ac410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.990+0000 7f824578a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8240071ce0 con 0x7f82401424f0 2024-03-06T07:30:01.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.990+0000 7f824578a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8240071b60 con 0x7f82400cd7a0 2024-03-06T07:30:01.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.990+0000 7f824578a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8240071e60 con 0x7f824006da40 2024-03-06T07:30:01.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.990+0000 7f823f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82401424f0 0x7f82400ac410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.990+0000 7f823f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82401424f0 0x7f82400ac410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59608/0 (socket says 172.21.15.83:59608) 2024-03-06T07:30:01.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.990+0000 7f823f7fe640 1 -- 172.21.15.83:0/1990003173 learned_addr learned my addr 172.21.15.83:0/1990003173 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:01.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.990+0000 7f823effd640 1 --2- 172.21.15.83:0/1990003173 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f824006da40 0x7f82400aee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.990+0000 7f823e7fc640 1 --2- 172.21.15.83:0/1990003173 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82400cd7a0 0x7f8240144ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.990+0000 7f823f7fe640 1 -- 172.21.15.83:0/1990003173 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f824006da40 msgr2=0x7f82400aee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.990+0000 7f823f7fe640 1 --2- 172.21.15.83:0/1990003173 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f824006da40 0x7f82400aee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.990+0000 7f823f7fe640 1 -- 172.21.15.83:0/1990003173 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82400cd7a0 msgr2=0x7f8240144ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.990+0000 7f823f7fe640 1 --2- 172.21.15.83:0/1990003173 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82400cd7a0 0x7f8240144ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.990+0000 7f823f7fe640 1 -- 172.21.15.83:0/1990003173 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8240143920 con 0x7f82401424f0 2024-03-06T07:30:01.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.990+0000 7f823effd640 1 --2- 172.21.15.83:0/1990003173 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f824006da40 0x7f82400aee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:01.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.990+0000 7f823e7fc640 1 --2- 172.21.15.83:0/1990003173 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82400cd7a0 0x7f8240144ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:01.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.990+0000 7f823f7fe640 1 --2- 172.21.15.83:0/1990003173 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82401424f0 0x7f82400ac410 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f8238009f00 tx=0x7f823800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.990+0000 7f82277fe640 1 -- 172.21.15.83:0/1990003173 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8238018030 con 0x7f82401424f0 2024-03-06T07:30:01.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.990+0000 7f82277fe640 1 -- 172.21.15.83:0/1990003173 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f823800ce50 con 0x7f82401424f0 2024-03-06T07:30:01.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.990+0000 7f824578a640 1 -- 172.21.15.83:0/1990003173 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8240143c00 con 0x7f82401424f0 2024-03-06T07:30:01.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.990+0000 7f82277fe640 1 -- 172.21.15.83:0/1990003173 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8238003bb0 con 0x7f82401424f0 2024-03-06T07:30:01.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.990+0000 7f824578a640 1 -- 172.21.15.83:0/1990003173 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82401441c0 con 0x7f82401424f0 2024-03-06T07:30:01.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.991+0000 7f824578a640 1 -- 172.21.15.83:0/1990003173 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f824004f330 con 0x7f82401424f0 2024-03-06T07:30:01.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.992+0000 7f82277fe640 1 -- 172.21.15.83:0/1990003173 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f823801b070 con 0x7f82401424f0 2024-03-06T07:30:01.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.992+0000 7f82277fe640 1 --2- 172.21.15.83:0/1990003173 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f822c06b4d0 0x7f822c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.992+0000 7f82277fe640 1 -- 172.21.15.83:0/1990003173 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f823808acc0 con 0x7f82401424f0 2024-03-06T07:30:01.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.992+0000 7f823effd640 1 --2- 172.21.15.83:0/1990003173 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f822c06b4d0 0x7f822c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.993+0000 7f823effd640 1 --2- 172.21.15.83:0/1990003173 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f822c06b4d0 0x7f822c06d990 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f8230036000 tx=0x7f823003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:01.996+0000 7f82277fe640 1 -- 172.21.15.83:0/1990003173 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8238010040 con 0x7f82401424f0 2024-03-06T07:30:02.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:02.211+0000 7f824578a640 1 -- 172.21.15.83:0/1990003173 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82400d0620 con 0x7f82401424f0 2024-03-06T07:30:02.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:02.212+0000 7f82277fe640 1 -- 172.21.15.83:0/1990003173 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f8238012340 con 0x7f82401424f0 2024-03-06T07:30:02.214 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:30:02.214 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:30:02.214 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:30:02.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:02.214+0000 7f824578a640 1 -- 172.21.15.83:0/1990003173 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f822c06b4d0 msgr2=0x7f822c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:02.214+0000 7f824578a640 1 --2- 172.21.15.83:0/1990003173 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f822c06b4d0 0x7f822c06d990 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f8230036000 tx=0x7f823003c040 comp rx=0 tx=0).stop 2024-03-06T07:30:02.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:02.214+0000 7f824578a640 1 -- 172.21.15.83:0/1990003173 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82401424f0 msgr2=0x7f82400ac410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:02.214+0000 7f824578a640 1 --2- 172.21.15.83:0/1990003173 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82401424f0 0x7f82400ac410 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f8238009f00 tx=0x7f823800c490 comp rx=0 tx=0).stop 2024-03-06T07:30:02.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:02.214+0000 7f824578a640 1 -- 172.21.15.83:0/1990003173 shutdown_connections 2024-03-06T07:30:02.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:02.214+0000 7f824578a640 1 --2- 172.21.15.83:0/1990003173 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f822c06b4d0 0x7f822c06d990 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:02.214+0000 7f824578a640 1 --2- 172.21.15.83:0/1990003173 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82401424f0 0x7f82400ac410 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:02.214+0000 7f824578a640 1 --2- 172.21.15.83:0/1990003173 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82400cd7a0 0x7f8240144ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:02.214+0000 7f824578a640 1 --2- 172.21.15.83:0/1990003173 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f824006da40 0x7f82400aee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:02.214+0000 7f824578a640 1 -- 172.21.15.83:0/1990003173 >> 172.21.15.83:0/1990003173 conn(0x7f82400cf8a0 msgr2=0x7f824015ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:02.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:02.214+0000 7f824578a640 1 -- 172.21.15.83:0/1990003173 shutdown_connections 2024-03-06T07:30:02.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:02.214+0000 7f824578a640 1 -- 172.21.15.83:0/1990003173 wait complete. 2024-03-06T07:30:02.428 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:30:02.428 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:30:03.006 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:03.429 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:03.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.528+0000 7f4970aa4640 1 Processor -- start 2024-03-06T07:30:03.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.528+0000 7f4970aa4640 1 -- start start 2024-03-06T07:30:03.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.529+0000 7f4970aa4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f496c15dfe0 0x7f496c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.529+0000 7f4970aa4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f496c0b3430 con 0x7f496c0d29c0 2024-03-06T07:30:03.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.529+0000 7f4970aa4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f496c0b3710 con 0x7f496c0cf830 2024-03-06T07:30:03.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.529+0000 7f4970aa4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f496c0b31e0 con 0x7f496c15dfe0 2024-03-06T07:30:03.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.529+0000 7f496ad9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f496c0d29c0 0x7f496c0b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49886/0 (socket says 172.21.15.83:49886) 2024-03-06T07:30:03.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.529+0000 7f496ad9d640 1 -- 172.21.15.83:0/1163687761 learned_addr learned my addr 172.21.15.83:0/1163687761 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:30:03.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.529+0000 7f496a59c640 1 --2- 172.21.15.83:0/1163687761 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f496c15dfe0 0x7f496c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.529+0000 7f496959a640 1 -- 172.21.15.83:0/1163687761 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2963584470 0 0) 0x7f496c0b3430 con 0x7f496c0d29c0 2024-03-06T07:30:03.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.529+0000 7f496959a640 1 -- 172.21.15.83:0/1163687761 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4954003520 con 0x7f496c0d29c0 2024-03-06T07:30:03.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.530+0000 7f496959a640 1 -- 172.21.15.83:0/1163687761 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1425928311 0 0) 0x7f4954003520 con 0x7f496c0d29c0 2024-03-06T07:30:03.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.530+0000 7f496959a640 1 -- 172.21.15.83:0/1163687761 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f496c15dfe0 msgr2=0x7f496c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:03.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.530+0000 7f496959a640 1 --2- 172.21.15.83:0/1163687761 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f496c15dfe0 0x7f496c0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.530+0000 7f496959a640 1 -- 172.21.15.83:0/1163687761 >> v1:172.21.15.183:6789/0 conn(0x7f496c0cf830 legacy=0x7f496c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.530+0000 7f496959a640 1 -- 172.21.15.83:0/1163687761 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f496c0b39f0 con 0x7f496c0d29c0 2024-03-06T07:30:03.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.530+0000 7f496959a640 1 -- 172.21.15.83:0/1163687761 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f49640019f0 con 0x7f496c0d29c0 2024-03-06T07:30:03.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.530+0000 7f496959a640 1 -- 172.21.15.83:0/1163687761 >> v1:172.21.15.83:6789/0 conn(0x7f496c0d29c0 legacy=0x7f496c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.530+0000 7f496959a640 1 --2- 172.21.15.83:0/1163687761 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49540039e0 0x7f4954003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.530+0000 7f496959a640 1 --2- 172.21.15.83:0/1163687761 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4954004800 0x7f4954004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.530+0000 7f496959a640 1 --2- 172.21.15.83:0/1163687761 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f496c15dfe0 0x7f49540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.530+0000 7f496959a640 1 -- 172.21.15.83:0/1163687761 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f496c0b31e0 con 0x7f49540039e0 2024-03-06T07:30:03.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.530+0000 7f496959a640 1 -- 172.21.15.83:0/1163687761 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f495400ba70 con 0x7f4954004800 2024-03-06T07:30:03.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.530+0000 7f496959a640 1 -- 172.21.15.83:0/1163687761 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f495400bc70 con 0x7f496c15dfe0 2024-03-06T07:30:03.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.530+0000 7f496a59c640 1 --2- 172.21.15.83:0/1163687761 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49540039e0 0x7f4954003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.530+0000 7f496ad9d640 1 --2- 172.21.15.83:0/1163687761 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f496c15dfe0 0x7f49540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.530+0000 7f4969d9b640 1 --2- 172.21.15.83:0/1163687761 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4954004800 0x7f4954004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.530+0000 7f496a59c640 1 -- 172.21.15.83:0/1163687761 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f496c15dfe0 msgr2=0x7f49540051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:03.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.530+0000 7f496a59c640 1 --2- 172.21.15.83:0/1163687761 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f496c15dfe0 0x7f49540051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.530+0000 7f496a59c640 1 -- 172.21.15.83:0/1163687761 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4954004800 msgr2=0x7f4954004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:03.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.530+0000 7f496a59c640 1 --2- 172.21.15.83:0/1163687761 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4954004800 0x7f4954004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.530+0000 7f496a59c640 1 -- 172.21.15.83:0/1163687761 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4954003520 con 0x7f49540039e0 2024-03-06T07:30:03.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.531+0000 7f496a59c640 1 --2- 172.21.15.83:0/1163687761 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49540039e0 0x7f4954003e00 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f495c009a90 tx=0x7f4954005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be6d50d5fbf9b061 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.531+0000 7f496959a640 1 -- 172.21.15.83:0/1163687761 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f495c039070 con 0x7f49540039e0 2024-03-06T07:30:03.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.531+0000 7f496959a640 1 -- 172.21.15.83:0/1163687761 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f495c02c680 con 0x7f49540039e0 2024-03-06T07:30:03.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.531+0000 7f496959a640 1 -- 172.21.15.83:0/1163687761 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f495c009690 con 0x7f49540039e0 2024-03-06T07:30:03.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.532+0000 7f4970aa4640 1 -- 172.21.15.83:0/1163687761 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49540039e0 msgr2=0x7f4954003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.532+0000 7f4970aa4640 1 --2- 172.21.15.83:0/1163687761 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49540039e0 0x7f4954003e00 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f495c009a90 tx=0x7f4954005830 comp rx=0 tx=0).stop 2024-03-06T07:30:03.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.532+0000 7f496ad9d640 1 -- 172.21.15.83:0/1163687761 reap_dead start 2024-03-06T07:30:03.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.532+0000 7f4970aa4640 1 -- 172.21.15.83:0/1163687761 shutdown_connections 2024-03-06T07:30:03.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.532+0000 7f4970aa4640 1 -- 172.21.15.83:0/1163687761 >> 172.21.15.83:0/1163687761 conn(0x7f496c0d3cd0 msgr2=0x7f496c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:03.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.532+0000 7f4970aa4640 1 -- 172.21.15.83:0/1163687761 shutdown_connections 2024-03-06T07:30:03.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.532+0000 7f4970aa4640 1 -- 172.21.15.83:0/1163687761 wait complete. 2024-03-06T07:30:03.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.534+0000 7f4970aa4640 1 Processor -- start 2024-03-06T07:30:03.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.534+0000 7f4970aa4640 1 -- start start 2024-03-06T07:30:03.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.535+0000 7f4970aa4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f49540039e0 0x7f496c0cba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.535+0000 7f4970aa4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f496c0cc7e0 0x7f496c0b88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.535+0000 7f4970aa4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f496c0ccc60 0x7f496c0b8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.535+0000 7f4970aa4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f496c15e3c0 con 0x7f496c0cc7e0 2024-03-06T07:30:03.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.535+0000 7f4970aa4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f496c0b3d60 con 0x7f49540039e0 2024-03-06T07:30:03.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.535+0000 7f4970aa4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f496c15e540 con 0x7f496c0ccc60 2024-03-06T07:30:03.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.535+0000 7f4969d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f496c0cc7e0 0x7f496c0b88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.535+0000 7f4969d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f496c0cc7e0 0x7f496c0b88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59630/0 (socket says 172.21.15.83:59630) 2024-03-06T07:30:03.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.535+0000 7f4969d9b640 1 -- 172.21.15.83:0/158078800 learned_addr learned my addr 172.21.15.83:0/158078800 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:03.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.535+0000 7f496a59c640 1 --2- 172.21.15.83:0/158078800 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f49540039e0 0x7f496c0cba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.535+0000 7f4969d9b640 1 -- 172.21.15.83:0/158078800 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f496c0ccc60 msgr2=0x7f496c0b8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.535+0000 7f496ad9d640 1 --2- 172.21.15.83:0/158078800 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f496c0ccc60 0x7f496c0b8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.535+0000 7f4969d9b640 1 --2- 172.21.15.83:0/158078800 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f496c0ccc60 0x7f496c0b8e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.535+0000 7f4969d9b640 1 -- 172.21.15.83:0/158078800 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f49540039e0 msgr2=0x7f496c0cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.535+0000 7f4969d9b640 1 --2- 172.21.15.83:0/158078800 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f49540039e0 0x7f496c0cba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.535+0000 7f4969d9b640 1 -- 172.21.15.83:0/158078800 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f496c0cd1f0 con 0x7f496c0cc7e0 2024-03-06T07:30:03.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.535+0000 7f496a59c640 1 --2- 172.21.15.83:0/158078800 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f49540039e0 0x7f496c0cba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:03.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.536+0000 7f4969d9b640 1 --2- 172.21.15.83:0/158078800 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f496c0cc7e0 0x7f496c0b88e0 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f496000e5e0 tx=0x7f496000eaa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.536+0000 7f495affd640 1 -- 172.21.15.83:0/158078800 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49600052e0 con 0x7f496c0cc7e0 2024-03-06T07:30:03.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.536+0000 7f495affd640 1 -- 172.21.15.83:0/158078800 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4960018070 con 0x7f496c0cc7e0 2024-03-06T07:30:03.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.536+0000 7f4970aa4640 1 -- 172.21.15.83:0/158078800 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f496c0caf40 con 0x7f496c0cc7e0 2024-03-06T07:30:03.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.536+0000 7f495affd640 1 -- 172.21.15.83:0/158078800 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4960003810 con 0x7f496c0cc7e0 2024-03-06T07:30:03.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.536+0000 7f4970aa4640 1 -- 172.21.15.83:0/158078800 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f496c0cb500 con 0x7f496c0cc7e0 2024-03-06T07:30:03.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.537+0000 7f4970aa4640 1 -- 172.21.15.83:0/158078800 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f496c04f330 con 0x7f496c0cc7e0 2024-03-06T07:30:03.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.537+0000 7f495affd640 1 -- 172.21.15.83:0/158078800 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f49600039b0 con 0x7f496c0cc7e0 2024-03-06T07:30:03.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.538+0000 7f495affd640 1 --2- 172.21.15.83:0/158078800 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f494c06b4d0 0x7f494c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.538+0000 7f496a59c640 1 --2- 172.21.15.83:0/158078800 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f494c06b4d0 0x7f494c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.538+0000 7f495affd640 1 -- 172.21.15.83:0/158078800 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4960089f00 con 0x7f496c0cc7e0 2024-03-06T07:30:03.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.538+0000 7f496a59c640 1 --2- 172.21.15.83:0/158078800 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f494c06b4d0 0x7f494c06d990 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f495c036000 tx=0x7f495c030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.542+0000 7f495affd640 1 -- 172.21.15.83:0/158078800 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4960058d50 con 0x7f496c0cc7e0 2024-03-06T07:30:03.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.801+0000 7f4970aa4640 1 -- 172.21.15.83:0/158078800 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f496c0b2640 con 0x7f496c0cc7e0 2024-03-06T07:30:03.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.802+0000 7f495affd640 1 -- 172.21.15.83:0/158078800 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f4960075e20 con 0x7f496c0cc7e0 2024-03-06T07:30:03.804 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:30:03.804 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:30:03.804 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:30:03.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.804+0000 7f4970aa4640 1 -- 172.21.15.83:0/158078800 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f494c06b4d0 msgr2=0x7f494c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.804+0000 7f4970aa4640 1 --2- 172.21.15.83:0/158078800 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f494c06b4d0 0x7f494c06d990 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f495c036000 tx=0x7f495c030000 comp rx=0 tx=0).stop 2024-03-06T07:30:03.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.804+0000 7f4970aa4640 1 -- 172.21.15.83:0/158078800 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f496c0cc7e0 msgr2=0x7f496c0b88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.804+0000 7f4970aa4640 1 --2- 172.21.15.83:0/158078800 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f496c0cc7e0 0x7f496c0b88e0 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f496000e5e0 tx=0x7f496000eaa0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.804+0000 7f4970aa4640 1 -- 172.21.15.83:0/158078800 shutdown_connections 2024-03-06T07:30:03.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.804+0000 7f4970aa4640 1 --2- 172.21.15.83:0/158078800 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f494c06b4d0 0x7f494c06d990 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.804+0000 7f4970aa4640 1 --2- 172.21.15.83:0/158078800 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f496c0ccc60 0x7f496c0b8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.804+0000 7f4970aa4640 1 --2- 172.21.15.83:0/158078800 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f496c0cc7e0 0x7f496c0b88e0 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.804+0000 7f4970aa4640 1 --2- 172.21.15.83:0/158078800 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f49540039e0 0x7f496c0cba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.804+0000 7f4970aa4640 1 -- 172.21.15.83:0/158078800 >> 172.21.15.83:0/158078800 conn(0x7f496c0d3cd0 msgr2=0x7f496c06cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:03.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.804+0000 7f4970aa4640 1 -- 172.21.15.83:0/158078800 shutdown_connections 2024-03-06T07:30:03.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:03.804+0000 7f4970aa4640 1 -- 172.21.15.83:0/158078800 wait complete. 2024-03-06T07:30:04.022 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:30:04.022 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:30:05.008 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:05.022 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:05.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.122+0000 7f0eed09d640 1 Processor -- start 2024-03-06T07:30:05.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.122+0000 7f0eed09d640 1 -- start start 2024-03-06T07:30:05.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.122+0000 7f0eed09d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0ee815dfe0 0x7f0ee80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.122+0000 7f0eed09d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0ee80cf830 0x7f0ee80d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.122+0000 7f0eed09d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ee80b3430 con 0x7f0ee80d27f0 2024-03-06T07:30:05.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.122+0000 7f0eed09d640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ee80d2c50 con 0x7f0ee80cf830 2024-03-06T07:30:05.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.122+0000 7f0eed09d640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee80b31e0 con 0x7f0ee815dfe0 2024-03-06T07:30:05.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.123+0000 7f0edffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0ee80cf830 0x7f0ee80d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.123+0000 7f0edffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0ee80cf830 0x7f0ee80d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59638/0 (socket says 172.21.15.83:59638) 2024-03-06T07:30:05.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.123+0000 7f0edffff640 1 -- 172.21.15.83:0/349311801 learned_addr learned my addr 172.21.15.83:0/349311801 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:05.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.123+0000 7f0ee6d9d640 1 --2- 172.21.15.83:0/349311801 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0ee815dfe0 0x7f0ee80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.123+0000 7f0edffff640 1 -- 172.21.15.83:0/349311801 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0ee815dfe0 msgr2=0x7f0ee80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:05.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.123+0000 7f0edffff640 1 --2- 172.21.15.83:0/349311801 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0ee815dfe0 0x7f0ee80cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.123+0000 7f0edffff640 1 -- 172.21.15.83:0/349311801 >> v1:172.21.15.183:6789/0 conn(0x7f0ee80d27f0 legacy=0x7f0ee80d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.123+0000 7f0edffff640 1 -- 172.21.15.83:0/349311801 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee80b3710 con 0x7f0ee80cf830 2024-03-06T07:30:05.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.123+0000 7f0ee6d9d640 1 --2- 172.21.15.83:0/349311801 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0ee815dfe0 0x7f0ee80cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:05.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.124+0000 7f0edffff640 1 --2- 172.21.15.83:0/349311801 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ee80cf830 0x7f0ee80d22b0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0009870 tx=0x7f0ed002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd8a5f2538b2d304 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:05.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.124+0000 7f0ee659c640 1 -- 172.21.15.83:0/349311801 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed0034070 con 0x7f0ee80cf830 2024-03-06T07:30:05.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.124+0000 7f0ee659c640 1 -- 172.21.15.83:0/349311801 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ed0030840 con 0x7f0ee80cf830 2024-03-06T07:30:05.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.124+0000 7f0ee659c640 1 -- 172.21.15.83:0/349311801 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed0030c10 con 0x7f0ee80cf830 2024-03-06T07:30:05.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.124+0000 7f0eed09d640 1 -- 172.21.15.83:0/349311801 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ee80cf830 msgr2=0x7f0ee80d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.124+0000 7f0eed09d640 1 --2- 172.21.15.83:0/349311801 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ee80cf830 0x7f0ee80d22b0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0009870 tx=0x7f0ed002b980 comp rx=0 tx=0).stop 2024-03-06T07:30:05.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.124+0000 7f0eed09d640 1 -- 172.21.15.83:0/349311801 shutdown_connections 2024-03-06T07:30:05.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.124+0000 7f0eed09d640 1 --2- 172.21.15.83:0/349311801 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ee80cf830 0x7f0ee80d22b0 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.124+0000 7f0eed09d640 1 --2- 172.21.15.83:0/349311801 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0ee815dfe0 0x7f0ee80cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.124+0000 7f0eed09d640 1 -- 172.21.15.83:0/349311801 >> 172.21.15.83:0/349311801 conn(0x7f0ee80d3cd0 msgr2=0x7f0ee80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:05.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.124+0000 7f0eed09d640 1 -- 172.21.15.83:0/349311801 shutdown_connections 2024-03-06T07:30:05.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.124+0000 7f0eed09d640 1 -- 172.21.15.83:0/349311801 wait complete. 2024-03-06T07:30:05.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.126+0000 7f0eed09d640 1 Processor -- start 2024-03-06T07:30:05.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.126+0000 7f0eed09d640 1 -- start start 2024-03-06T07:30:05.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.126+0000 7f0eed09d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0ee80cf830 0x7f0ee81f7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.126+0000 7f0eed09d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ee80d27f0 0x7f0ee8155210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.126+0000 7f0eed09d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0ee815dfe0 0x7f0ee80b2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.126+0000 7f0eed09d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee80d1c70 con 0x7f0ee80d27f0 2024-03-06T07:30:05.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.126+0000 7f0eed09d640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee80b3d60 con 0x7f0ee815dfe0 2024-03-06T07:30:05.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.126+0000 7f0eed09d640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee80d1df0 con 0x7f0ee80cf830 2024-03-06T07:30:05.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.126+0000 7f0edffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ee80d27f0 0x7f0ee8155210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.126+0000 7f0edffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ee80d27f0 0x7f0ee8155210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59644/0 (socket says 172.21.15.83:59644) 2024-03-06T07:30:05.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.127+0000 7f0edffff640 1 -- 172.21.15.83:0/2926285397 learned_addr learned my addr 172.21.15.83:0/2926285397 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:05.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.127+0000 7f0ee6d9d640 1 --2- 172.21.15.83:0/2926285397 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0ee80cf830 0x7f0ee81f7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.127+0000 7f0ee759e640 1 --2- 172.21.15.83:0/2926285397 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0ee815dfe0 0x7f0ee80b2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.127+0000 7f0edffff640 1 -- 172.21.15.83:0/2926285397 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0ee80cf830 msgr2=0x7f0ee81f7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.127+0000 7f0edffff640 1 --2- 172.21.15.83:0/2926285397 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0ee80cf830 0x7f0ee81f7930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.127+0000 7f0edffff640 1 -- 172.21.15.83:0/2926285397 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0ee815dfe0 msgr2=0x7f0ee80b2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.127+0000 7f0edffff640 1 --2- 172.21.15.83:0/2926285397 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0ee815dfe0 0x7f0ee80b2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.127+0000 7f0edffff640 1 -- 172.21.15.83:0/2926285397 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee81f7e70 con 0x7f0ee80d27f0 2024-03-06T07:30:05.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.127+0000 7f0edffff640 1 --2- 172.21.15.83:0/2926285397 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ee80d27f0 0x7f0ee8155210 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f0ed002b9c0 tx=0x7f0ed0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:05.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.127+0000 7f0edf7fe640 1 -- 172.21.15.83:0/2926285397 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed0048070 con 0x7f0ee80d27f0 2024-03-06T07:30:05.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.127+0000 7f0edf7fe640 1 -- 172.21.15.83:0/2926285397 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ed0030640 con 0x7f0ee80d27f0 2024-03-06T07:30:05.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.127+0000 7f0eed09d640 1 -- 172.21.15.83:0/2926285397 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ee8160620 con 0x7f0ee80d27f0 2024-03-06T07:30:05.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.127+0000 7f0edf7fe640 1 -- 172.21.15.83:0/2926285397 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed0034040 con 0x7f0ee80d27f0 2024-03-06T07:30:05.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.127+0000 7f0eed09d640 1 -- 172.21.15.83:0/2926285397 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ee80b6ce0 con 0x7f0ee80d27f0 2024-03-06T07:30:05.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.128+0000 7f0eed09d640 1 -- 172.21.15.83:0/2926285397 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ee804f330 con 0x7f0ee80d27f0 2024-03-06T07:30:05.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.129+0000 7f0edf7fe640 1 -- 172.21.15.83:0/2926285397 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0ed004b070 con 0x7f0ee80d27f0 2024-03-06T07:30:05.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.129+0000 7f0edf7fe640 1 --2- 172.21.15.83:0/2926285397 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0ec806b4d0 0x7f0ec806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.129+0000 7f0edf7fe640 1 -- 172.21.15.83:0/2926285397 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0ed00ac360 con 0x7f0ee80d27f0 2024-03-06T07:30:05.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.129+0000 7f0ee6d9d640 1 --2- 172.21.15.83:0/2926285397 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0ec806b4d0 0x7f0ec806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.130+0000 7f0ee6d9d640 1 --2- 172.21.15.83:0/2926285397 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0ec806b4d0 0x7f0ec806d990 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f0ed8006fd0 tx=0x7f0ed8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:05.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.134+0000 7f0edf7fe640 1 -- 172.21.15.83:0/2926285397 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0ee81f7e70 con 0x7f0ee80d27f0 2024-03-06T07:30:05.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.349+0000 7f0eed09d640 1 -- 172.21.15.83:0/2926285397 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ee8155750 con 0x7f0ee80d27f0 2024-03-06T07:30:05.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.349+0000 7f0edf7fe640 1 -- 172.21.15.83:0/2926285397 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f0ed007ad00 con 0x7f0ee80d27f0 2024-03-06T07:30:05.351 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:30:05.352 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:30:05.352 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:30:05.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.351+0000 7f0eed09d640 1 -- 172.21.15.83:0/2926285397 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0ec806b4d0 msgr2=0x7f0ec806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.351+0000 7f0eed09d640 1 --2- 172.21.15.83:0/2926285397 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0ec806b4d0 0x7f0ec806d990 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f0ed8006fd0 tx=0x7f0ed8008040 comp rx=0 tx=0).stop 2024-03-06T07:30:05.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.351+0000 7f0eed09d640 1 -- 172.21.15.83:0/2926285397 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ee80d27f0 msgr2=0x7f0ee8155210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.351+0000 7f0eed09d640 1 --2- 172.21.15.83:0/2926285397 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ee80d27f0 0x7f0ee8155210 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f0ed002b9c0 tx=0x7f0ed0004730 comp rx=0 tx=0).stop 2024-03-06T07:30:05.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.351+0000 7f0eed09d640 1 -- 172.21.15.83:0/2926285397 shutdown_connections 2024-03-06T07:30:05.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.351+0000 7f0eed09d640 1 --2- 172.21.15.83:0/2926285397 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0ec806b4d0 0x7f0ec806d990 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.351+0000 7f0eed09d640 1 --2- 172.21.15.83:0/2926285397 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0ee815dfe0 0x7f0ee80b2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.351+0000 7f0eed09d640 1 --2- 172.21.15.83:0/2926285397 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ee80d27f0 0x7f0ee8155210 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.351+0000 7f0eed09d640 1 --2- 172.21.15.83:0/2926285397 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0ee80cf830 0x7f0ee81f7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.351+0000 7f0eed09d640 1 -- 172.21.15.83:0/2926285397 >> 172.21.15.83:0/2926285397 conn(0x7f0ee80d3cd0 msgr2=0x7f0ee80cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:05.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.351+0000 7f0eed09d640 1 -- 172.21.15.83:0/2926285397 shutdown_connections 2024-03-06T07:30:05.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:05.351+0000 7f0eed09d640 1 -- 172.21.15.83:0/2926285397 wait complete. 2024-03-06T07:30:05.565 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:30:05.565 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:30:06.566 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:06.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.668+0000 7fdc2333a640 1 Processor -- start 2024-03-06T07:30:06.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.669+0000 7fdc2333a640 1 -- start start 2024-03-06T07:30:06.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.669+0000 7fdc2333a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdc1c0d00c0 0x7fdc1c0d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.669+0000 7fdc2333a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fdc1c06cf20 0x7fdc1c06d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.669+0000 7fdc2333a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc1c0d09e0 con 0x7fdc1c0cf850 2024-03-06T07:30:06.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.669+0000 7fdc2333a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc1c06ca90 con 0x7fdc1c06cf20 2024-03-06T07:30:06.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.669+0000 7fdc2333a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1c15e380 con 0x7fdc1c0d00c0 2024-03-06T07:30:06.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.670+0000 7fdc218d7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fdc1c06cf20 0x7fdc1c06d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.670+0000 7fdc208d5640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fdc1c0cf850 0x7fdc1c0cd3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:36602/0 (socket says 172.21.15.83:36602) 2024-03-06T07:30:06.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.670+0000 7fdc208d5640 1 -- 172.21.15.83:0/2280616008 learned_addr learned my addr 172.21.15.83:0/2280616008 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:30:06.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.670+0000 7fdc218d7640 1 -- 172.21.15.83:0/2280616008 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdc1c0d00c0 msgr2=0x7fdc1c0d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:06.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.670+0000 7fdc210d6640 1 --2- 172.21.15.83:0/2280616008 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdc1c0d00c0 0x7fdc1c0d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.670+0000 7fdc13fff640 1 -- 172.21.15.83:0/2280616008 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2680606870 0 0) 0x7fdc1c0d09e0 con 0x7fdc1c0cf850 2024-03-06T07:30:06.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.670+0000 7fdc218d7640 1 --2- 172.21.15.83:0/2280616008 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdc1c0d00c0 0x7fdc1c0d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.670+0000 7fdc218d7640 1 -- 172.21.15.83:0/2280616008 >> v1:172.21.15.83:6789/0 conn(0x7fdc1c0cf850 legacy=0x7fdc1c0cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.670+0000 7fdc218d7640 1 -- 172.21.15.83:0/2280616008 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc1c0cdc10 con 0x7fdc1c06cf20 2024-03-06T07:30:06.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.670+0000 7fdc210d6640 1 --2- 172.21.15.83:0/2280616008 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdc1c0d00c0 0x7fdc1c0d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:06.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.671+0000 7fdc218d7640 1 --2- 172.21.15.83:0/2280616008 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc1c06cf20 0x7fdc1c06d3b0 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7fdc14009d90 tx=0x7fdc1402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29351165913ea568 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.671+0000 7fdc13fff640 1 -- 172.21.15.83:0/2280616008 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc14039070 con 0x7fdc1c06cf20 2024-03-06T07:30:06.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.671+0000 7fdc13fff640 1 -- 172.21.15.83:0/2280616008 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc1402cc50 con 0x7fdc1c06cf20 2024-03-06T07:30:06.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.671+0000 7fdc13fff640 1 -- 172.21.15.83:0/2280616008 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc14034440 con 0x7fdc1c06cf20 2024-03-06T07:30:06.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.671+0000 7fdc2333a640 1 -- 172.21.15.83:0/2280616008 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc1c06cf20 msgr2=0x7fdc1c06d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.671+0000 7fdc2333a640 1 --2- 172.21.15.83:0/2280616008 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc1c06cf20 0x7fdc1c06d3b0 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7fdc14009d90 tx=0x7fdc1402bea0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.671+0000 7fdc2333a640 1 -- 172.21.15.83:0/2280616008 shutdown_connections 2024-03-06T07:30:06.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.671+0000 7fdc2333a640 1 --2- 172.21.15.83:0/2280616008 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc1c06cf20 0x7fdc1c06d3b0 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.671+0000 7fdc2333a640 1 --2- 172.21.15.83:0/2280616008 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdc1c0d00c0 0x7fdc1c0d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.671+0000 7fdc2333a640 1 -- 172.21.15.83:0/2280616008 >> 172.21.15.83:0/2280616008 conn(0x7fdc1c0b35a0 msgr2=0x7fdc1c0b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:06.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.671+0000 7fdc2333a640 1 -- 172.21.15.83:0/2280616008 shutdown_connections 2024-03-06T07:30:06.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.671+0000 7fdc2333a640 1 -- 172.21.15.83:0/2280616008 wait complete. 2024-03-06T07:30:06.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.673+0000 7fdc2333a640 1 Processor -- start 2024-03-06T07:30:06.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.673+0000 7fdc2333a640 1 -- start start 2024-03-06T07:30:06.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.673+0000 7fdc2333a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdc1c06cf20 0x7fdc1c153fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.673+0000 7fdc2333a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdc1c0cf850 0x7fdc1c1f7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.673+0000 7fdc2333a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc1c0d00c0 0x7fdc1c0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.673+0000 7fdc2333a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1c06c720 con 0x7fdc1c0d00c0 2024-03-06T07:30:06.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.673+0000 7fdc2333a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1c06c440 con 0x7fdc1c06cf20 2024-03-06T07:30:06.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.673+0000 7fdc2333a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1c0b1030 con 0x7fdc1c0cf850 2024-03-06T07:30:06.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.673+0000 7fdc218d7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc1c0d00c0 0x7fdc1c0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.673+0000 7fdc218d7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc1c0d00c0 0x7fdc1c0b4f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37318/0 (socket says 172.21.15.83:37318) 2024-03-06T07:30:06.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.673+0000 7fdc218d7640 1 -- 172.21.15.83:0/512718651 learned_addr learned my addr 172.21.15.83:0/512718651 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:06.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.674+0000 7fdc208d5640 1 --2- 172.21.15.83:0/512718651 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdc1c0cf850 0x7fdc1c1f7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.674+0000 7fdc210d6640 1 --2- 172.21.15.83:0/512718651 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdc1c06cf20 0x7fdc1c153fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.674+0000 7fdc218d7640 1 -- 172.21.15.83:0/512718651 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdc1c0cf850 msgr2=0x7fdc1c1f7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.674+0000 7fdc218d7640 1 --2- 172.21.15.83:0/512718651 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdc1c0cf850 0x7fdc1c1f7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.674+0000 7fdc218d7640 1 -- 172.21.15.83:0/512718651 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdc1c06cf20 msgr2=0x7fdc1c153fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.674+0000 7fdc218d7640 1 --2- 172.21.15.83:0/512718651 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdc1c06cf20 0x7fdc1c153fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.674+0000 7fdc218d7640 1 -- 172.21.15.83:0/512718651 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc1c1f7b40 con 0x7fdc1c0d00c0 2024-03-06T07:30:06.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.674+0000 7fdc210d6640 1 --2- 172.21.15.83:0/512718651 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdc1c06cf20 0x7fdc1c153fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:06.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.674+0000 7fdc218d7640 1 --2- 172.21.15.83:0/512718651 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc1c0d00c0 0x7fdc1c0b4f30 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7fdc14037040 tx=0x7fdc14004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.674+0000 7fdc11ffb640 1 -- 172.21.15.83:0/512718651 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc14044020 con 0x7fdc1c0d00c0 2024-03-06T07:30:06.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.674+0000 7fdc11ffb640 1 -- 172.21.15.83:0/512718651 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc1403a040 con 0x7fdc1c0d00c0 2024-03-06T07:30:06.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.674+0000 7fdc2333a640 1 -- 172.21.15.83:0/512718651 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc1c1f7e20 con 0x7fdc1c0d00c0 2024-03-06T07:30:06.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.674+0000 7fdc11ffb640 1 -- 172.21.15.83:0/512718651 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc14039050 con 0x7fdc1c0d00c0 2024-03-06T07:30:06.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.674+0000 7fdc2333a640 1 -- 172.21.15.83:0/512718651 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc1c0b4730 con 0x7fdc1c0d00c0 2024-03-06T07:30:06.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.675+0000 7fdc2333a640 1 -- 172.21.15.83:0/512718651 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc1c068940 con 0x7fdc1c0d00c0 2024-03-06T07:30:06.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.676+0000 7fdc11ffb640 1 -- 172.21.15.83:0/512718651 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdc14003990 con 0x7fdc1c0d00c0 2024-03-06T07:30:06.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.676+0000 7fdc11ffb640 1 --2- 172.21.15.83:0/512718651 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdc0006b4d0 0x7fdc0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.676+0000 7fdc11ffb640 1 -- 172.21.15.83:0/512718651 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fdc140ac680 con 0x7fdc1c0d00c0 2024-03-06T07:30:06.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.676+0000 7fdc210d6640 1 --2- 172.21.15.83:0/512718651 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdc0006b4d0 0x7fdc0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.677+0000 7fdc210d6640 1 --2- 172.21.15.83:0/512718651 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdc0006b4d0 0x7fdc0006d990 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fdc1800aa30 tx=0x7fdc18008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.680+0000 7fdc11ffb640 1 -- 172.21.15.83:0/512718651 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdc1c068940 con 0x7fdc1c0d00c0 2024-03-06T07:30:06.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.896+0000 7fdc2333a640 1 -- 172.21.15.83:0/512718651 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc1c04f330 con 0x7fdc1c0d00c0 2024-03-06T07:30:06.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.896+0000 7fdc11ffb640 1 -- 172.21.15.83:0/512718651 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7fdc1404e060 con 0x7fdc1c0d00c0 2024-03-06T07:30:06.898 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:30:06.898 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:30:06.899 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:30:06.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.899+0000 7fdc2333a640 1 -- 172.21.15.83:0/512718651 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdc0006b4d0 msgr2=0x7fdc0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.899+0000 7fdc2333a640 1 --2- 172.21.15.83:0/512718651 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdc0006b4d0 0x7fdc0006d990 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fdc1800aa30 tx=0x7fdc18008040 comp rx=0 tx=0).stop 2024-03-06T07:30:06.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.899+0000 7fdc2333a640 1 -- 172.21.15.83:0/512718651 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc1c0d00c0 msgr2=0x7fdc1c0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.899+0000 7fdc2333a640 1 --2- 172.21.15.83:0/512718651 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc1c0d00c0 0x7fdc1c0b4f30 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7fdc14037040 tx=0x7fdc14004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:06.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.899+0000 7fdc2333a640 1 -- 172.21.15.83:0/512718651 shutdown_connections 2024-03-06T07:30:06.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.899+0000 7fdc2333a640 1 --2- 172.21.15.83:0/512718651 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdc0006b4d0 0x7fdc0006d990 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.899+0000 7fdc2333a640 1 --2- 172.21.15.83:0/512718651 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc1c0d00c0 0x7fdc1c0b4f30 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.899+0000 7fdc2333a640 1 --2- 172.21.15.83:0/512718651 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdc1c0cf850 0x7fdc1c1f7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.899+0000 7fdc2333a640 1 --2- 172.21.15.83:0/512718651 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdc1c06cf20 0x7fdc1c153fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.899+0000 7fdc2333a640 1 -- 172.21.15.83:0/512718651 >> 172.21.15.83:0/512718651 conn(0x7fdc1c0b35a0 msgr2=0x7fdc1c06f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:06.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.899+0000 7fdc2333a640 1 -- 172.21.15.83:0/512718651 shutdown_connections 2024-03-06T07:30:06.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:06.899+0000 7fdc2333a640 1 -- 172.21.15.83:0/512718651 wait complete. 2024-03-06T07:30:07.009 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:07.113 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:30:07.113 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:30:08.113 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:08.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.213+0000 7f1155f20640 1 Processor -- start 2024-03-06T07:30:08.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.213+0000 7f1155f20640 1 -- start start 2024-03-06T07:30:08.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.214+0000 7f1155f20640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f11500d00c0 0x7f11500d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.214+0000 7f1155f20640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f11500cf850 0x7f11500cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.214+0000 7f1155f20640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11500d09e0 con 0x7f11500ce1b0 2024-03-06T07:30:08.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.214+0000 7f1155f20640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f115006ca90 con 0x7f11500cf850 2024-03-06T07:30:08.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.214+0000 7f1155f20640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f115006df30 con 0x7f11500d00c0 2024-03-06T07:30:08.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.214+0000 7f1147fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f11500cf850 0x7f11500cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.214+0000 7f1147fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f11500cf850 0x7f11500cd3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37334/0 (socket says 172.21.15.83:37334) 2024-03-06T07:30:08.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.214+0000 7f114ffff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f11500ce1b0 0x7f11500ce640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:36606/0 (socket says 172.21.15.83:36606) 2024-03-06T07:30:08.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.214+0000 7f1147fff640 1 -- 172.21.15.83:0/947851105 learned_addr learned my addr 172.21.15.83:0/947851105 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:08.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.214+0000 7f114f7fe640 1 --2- 172.21.15.83:0/947851105 >> v2:172.21.15.183:3300/0 conn(0x7f11500d00c0 0x7f11500d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.214+0000 7f1147fff640 1 -- 172.21.15.83:0/947851105 >> v2:172.21.15.183:3300/0 conn(0x7f11500d00c0 msgr2=0x7f11500d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:08.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.214+0000 7f1147fff640 1 --2- 172.21.15.83:0/947851105 >> v2:172.21.15.183:3300/0 conn(0x7f11500d00c0 0x7f11500d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.214+0000 7f1147fff640 1 -- 172.21.15.83:0/947851105 >> v1:172.21.15.83:6789/0 conn(0x7f11500ce1b0 legacy=0x7f11500ce640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.214+0000 7f114effd640 1 -- 172.21.15.83:0/947851105 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1804718667 0 0) 0x7f11500d09e0 con 0x7f11500ce1b0 2024-03-06T07:30:08.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.215+0000 7f1147fff640 1 -- 172.21.15.83:0/947851105 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f115015e380 con 0x7f11500cf850 2024-03-06T07:30:08.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.215+0000 7f1147fff640 1 --2- 172.21.15.83:0/947851105 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11500cf850 0x7f11500cd3c0 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f11500d1a40 tx=0x7f113802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee2941a0f48621f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.215+0000 7f114effd640 1 -- 172.21.15.83:0/947851105 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1138034070 con 0x7f11500cf850 2024-03-06T07:30:08.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.215+0000 7f114effd640 1 -- 172.21.15.83:0/947851105 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1138030860 con 0x7f11500cf850 2024-03-06T07:30:08.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.215+0000 7f114effd640 1 -- 172.21.15.83:0/947851105 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1138030c30 con 0x7f11500cf850 2024-03-06T07:30:08.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.216+0000 7f1155f20640 1 -- 172.21.15.83:0/947851105 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11500cf850 msgr2=0x7f11500cd3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.216+0000 7f1155f20640 1 --2- 172.21.15.83:0/947851105 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11500cf850 0x7f11500cd3c0 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f11500d1a40 tx=0x7f113802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.216+0000 7f1155f20640 1 -- 172.21.15.83:0/947851105 shutdown_connections 2024-03-06T07:30:08.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.216+0000 7f1155f20640 1 --2- 172.21.15.83:0/947851105 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11500cf850 0x7f11500cd3c0 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.216+0000 7f1155f20640 1 --2- 172.21.15.83:0/947851105 >> v2:172.21.15.183:3300/0 conn(0x7f11500d00c0 0x7f11500d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.216+0000 7f1155f20640 1 -- 172.21.15.83:0/947851105 >> 172.21.15.83:0/947851105 conn(0x7f11500b35a0 msgr2=0x7f11500b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:08.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.216+0000 7f1155f20640 1 -- 172.21.15.83:0/947851105 shutdown_connections 2024-03-06T07:30:08.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.216+0000 7f1155f20640 1 -- 172.21.15.83:0/947851105 wait complete. 2024-03-06T07:30:08.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.217+0000 7f1155f20640 1 Processor -- start 2024-03-06T07:30:08.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.217+0000 7f1155f20640 1 -- start start 2024-03-06T07:30:08.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.218+0000 7f1155f20640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11500ce1b0 0x7f11500b6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.218+0000 7f1155f20640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11500cf850 0x7f11500c70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.218+0000 7f1155f20640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11500d00c0 0x7f11500c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.218+0000 7f1155f20640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f115006c440 con 0x7f11500d00c0 2024-03-06T07:30:08.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.218+0000 7f1155f20640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f115006c160 con 0x7f11500cf850 2024-03-06T07:30:08.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.218+0000 7f1155f20640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f115006c720 con 0x7f11500ce1b0 2024-03-06T07:30:08.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.218+0000 7f114ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11500d00c0 0x7f11500c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.218+0000 7f114ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11500d00c0 0x7f11500c79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37338/0 (socket says 172.21.15.83:37338) 2024-03-06T07:30:08.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.218+0000 7f114ffff640 1 -- 172.21.15.83:0/2212745785 learned_addr learned my addr 172.21.15.83:0/2212745785 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:08.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.218+0000 7f114f7fe640 1 --2- 172.21.15.83:0/2212745785 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11500ce1b0 0x7f11500b6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.218+0000 7f1147fff640 1 --2- 172.21.15.83:0/2212745785 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11500cf850 0x7f11500c70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.218+0000 7f114ffff640 1 -- 172.21.15.83:0/2212745785 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11500ce1b0 msgr2=0x7f11500b6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.218+0000 7f114ffff640 1 --2- 172.21.15.83:0/2212745785 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11500ce1b0 0x7f11500b6dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.218+0000 7f114ffff640 1 -- 172.21.15.83:0/2212745785 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11500cf850 msgr2=0x7f11500c70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.218+0000 7f114ffff640 1 --2- 172.21.15.83:0/2212745785 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11500cf850 0x7f11500c70b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.218+0000 7f114ffff640 1 -- 172.21.15.83:0/2212745785 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11500c56d0 con 0x7f11500d00c0 2024-03-06T07:30:08.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.218+0000 7f1147fff640 1 --2- 172.21.15.83:0/2212745785 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11500cf850 0x7f11500c70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:08.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.218+0000 7f114ffff640 1 --2- 172.21.15.83:0/2212745785 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11500d00c0 0x7f11500c79e0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f114800c810 tx=0x7f114800ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.219+0000 7f114cff9640 1 -- 172.21.15.83:0/2212745785 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1148018070 con 0x7f11500d00c0 2024-03-06T07:30:08.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.219+0000 7f114cff9640 1 -- 172.21.15.83:0/2212745785 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f114800a640 con 0x7f11500d00c0 2024-03-06T07:30:08.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.219+0000 7f1155f20640 1 -- 172.21.15.83:0/2212745785 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11500c59b0 con 0x7f11500d00c0 2024-03-06T07:30:08.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.219+0000 7f114cff9640 1 -- 172.21.15.83:0/2212745785 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f114800aa30 con 0x7f11500d00c0 2024-03-06T07:30:08.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.219+0000 7f1155f20640 1 -- 172.21.15.83:0/2212745785 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11500c5f70 con 0x7f11500d00c0 2024-03-06T07:30:08.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.220+0000 7f1155f20640 1 -- 172.21.15.83:0/2212745785 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f115004f330 con 0x7f11500d00c0 2024-03-06T07:30:08.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.220+0000 7f114cff9640 1 -- 172.21.15.83:0/2212745785 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f114801b070 con 0x7f11500d00c0 2024-03-06T07:30:08.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.221+0000 7f114cff9640 1 --2- 172.21.15.83:0/2212745785 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f113006b500 0x7f113006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.221+0000 7f114f7fe640 1 --2- 172.21.15.83:0/2212745785 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f113006b500 0x7f113006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.221+0000 7f114cff9640 1 -- 172.21.15.83:0/2212745785 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f11480897a0 con 0x7f11500d00c0 2024-03-06T07:30:08.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.221+0000 7f114f7fe640 1 --2- 172.21.15.83:0/2212745785 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f113006b500 0x7f113006d9c0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f11400079a0 tx=0x7f1140008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.229+0000 7f114cff9640 1 -- 172.21.15.83:0/2212745785 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1148058020 con 0x7f11500d00c0 2024-03-06T07:30:08.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.439+0000 7f1155f20640 1 -- 172.21.15.83:0/2212745785 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f115015dba0 con 0x7f11500d00c0 2024-03-06T07:30:08.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.440+0000 7f114cff9640 1 -- 172.21.15.83:0/2212745785 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f114805c0c0 con 0x7f11500d00c0 2024-03-06T07:30:08.442 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:30:08.442 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:30:08.442 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:30:08.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.442+0000 7f1155f20640 1 -- 172.21.15.83:0/2212745785 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f113006b500 msgr2=0x7f113006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.442+0000 7f1155f20640 1 --2- 172.21.15.83:0/2212745785 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f113006b500 0x7f113006d9c0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f11400079a0 tx=0x7f1140008040 comp rx=0 tx=0).stop 2024-03-06T07:30:08.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.442+0000 7f1155f20640 1 -- 172.21.15.83:0/2212745785 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11500d00c0 msgr2=0x7f11500c79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.442+0000 7f1155f20640 1 --2- 172.21.15.83:0/2212745785 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11500d00c0 0x7f11500c79e0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f114800c810 tx=0x7f114800ccd0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.442+0000 7f1155f20640 1 -- 172.21.15.83:0/2212745785 shutdown_connections 2024-03-06T07:30:08.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.442+0000 7f1155f20640 1 --2- 172.21.15.83:0/2212745785 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f113006b500 0x7f113006d9c0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.442+0000 7f1155f20640 1 --2- 172.21.15.83:0/2212745785 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11500d00c0 0x7f11500c79e0 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.442+0000 7f1155f20640 1 --2- 172.21.15.83:0/2212745785 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11500cf850 0x7f11500c70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.442+0000 7f1155f20640 1 --2- 172.21.15.83:0/2212745785 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11500ce1b0 0x7f11500b6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.442+0000 7f1155f20640 1 -- 172.21.15.83:0/2212745785 >> 172.21.15.83:0/2212745785 conn(0x7f11500b35a0 msgr2=0x7f11500d0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:08.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.442+0000 7f1155f20640 1 -- 172.21.15.83:0/2212745785 shutdown_connections 2024-03-06T07:30:08.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:08.442+0000 7f1155f20640 1 -- 172.21.15.83:0/2212745785 wait complete. 2024-03-06T07:30:08.656 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:30:08.656 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:30:09.011 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:09.658 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:09.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.759+0000 7fab7d11e640 1 Processor -- start 2024-03-06T07:30:09.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.759+0000 7fab7d11e640 1 -- start start 2024-03-06T07:30:09.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.760+0000 7fab7d11e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fab7815dfe0 0x7fab780cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.760+0000 7fab7d11e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fab780cf830 0x7fab780d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.760+0000 7fab7d11e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab780b3430 con 0x7fab780d27f0 2024-03-06T07:30:09.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.760+0000 7fab7d11e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab780d2c50 con 0x7fab7815dfe0 2024-03-06T07:30:09.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.760+0000 7fab7d11e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab780b31e0 con 0x7fab780cf830 2024-03-06T07:30:09.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.760+0000 7fab76d9d640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fab7815dfe0 0x7fab780cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.760+0000 7fab6ffff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fab780cf830 0x7fab780d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.760+0000 7fab7759e640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fab780d27f0 0x7fab780d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:47730/0 (socket says 172.21.15.83:47730) 2024-03-06T07:30:09.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.760+0000 7fab76d9d640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fab7815dfe0 0x7fab780cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:33434/0 (socket says 172.21.15.83:33434) 2024-03-06T07:30:09.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.760+0000 7fab7759e640 1 -- 172.21.15.83:0/3728336641 learned_addr learned my addr 172.21.15.83:0/3728336641 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:30:09.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.760+0000 7fab76d9d640 1 -- 172.21.15.83:0/3728336641 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fab780cf830 msgr2=0x7fab780d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:09.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.760+0000 7fab76d9d640 1 --2- 172.21.15.83:0/3728336641 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fab780cf830 0x7fab780d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.760+0000 7fab76d9d640 1 -- 172.21.15.83:0/3728336641 >> v1:172.21.15.183:6789/0 conn(0x7fab780d27f0 legacy=0x7fab780d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.760+0000 7fab76d9d640 1 -- 172.21.15.83:0/3728336641 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab780b3710 con 0x7fab7815dfe0 2024-03-06T07:30:09.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.760+0000 7fab6ffff640 1 --2- 172.21.15.83:0/3728336641 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fab780cf830 0x7fab780d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:09.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.761+0000 7fab76d9d640 1 --2- 172.21.15.83:0/3728336641 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fab7815dfe0 0x7fab780cf2f0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fab680097a0 tx=0x7fab6802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d64ba276a603060e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:09.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.761+0000 7fab7659c640 1 -- 172.21.15.83:0/3728336641 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab68035020 con 0x7fab7815dfe0 2024-03-06T07:30:09.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.761+0000 7fab7659c640 1 -- 172.21.15.83:0/3728336641 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab68031830 con 0x7fab7815dfe0 2024-03-06T07:30:09.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.761+0000 7fab7659c640 1 -- 172.21.15.83:0/3728336641 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab68031c00 con 0x7fab7815dfe0 2024-03-06T07:30:09.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.762+0000 7fab7d11e640 1 -- 172.21.15.83:0/3728336641 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fab7815dfe0 msgr2=0x7fab780cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.762+0000 7fab7d11e640 1 --2- 172.21.15.83:0/3728336641 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fab7815dfe0 0x7fab780cf2f0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fab680097a0 tx=0x7fab6802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.762+0000 7fab7d11e640 1 -- 172.21.15.83:0/3728336641 shutdown_connections 2024-03-06T07:30:09.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.762+0000 7fab7d11e640 1 --2- 172.21.15.83:0/3728336641 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fab780cf830 0x7fab780d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.762+0000 7fab7d11e640 1 --2- 172.21.15.83:0/3728336641 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fab7815dfe0 0x7fab780cf2f0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.762+0000 7fab7d11e640 1 -- 172.21.15.83:0/3728336641 >> 172.21.15.83:0/3728336641 conn(0x7fab780d3cd0 msgr2=0x7fab780d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:09.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.762+0000 7fab7d11e640 1 -- 172.21.15.83:0/3728336641 shutdown_connections 2024-03-06T07:30:09.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.762+0000 7fab7d11e640 1 -- 172.21.15.83:0/3728336641 wait complete. 2024-03-06T07:30:09.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.764+0000 7fab7d11e640 1 Processor -- start 2024-03-06T07:30:09.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.764+0000 7fab7d11e640 1 -- start start 2024-03-06T07:30:09.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.765+0000 7fab7d11e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab780cf830 0x7fab7814e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.765+0000 7fab7d11e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fab780d27f0 0x7fab780b7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.765+0000 7fab7d11e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fab7815dfe0 0x7fab780ccf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.765+0000 7fab7d11e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab780d1c70 con 0x7fab780cf830 2024-03-06T07:30:09.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.765+0000 7fab7d11e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab780b3d60 con 0x7fab780d27f0 2024-03-06T07:30:09.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.765+0000 7fab7d11e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab780d1df0 con 0x7fab7815dfe0 2024-03-06T07:30:09.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.766+0000 7fab76d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab780cf830 0x7fab7814e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.766+0000 7fab76d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab780cf830 0x7fab7814e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37340/0 (socket says 172.21.15.83:37340) 2024-03-06T07:30:09.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.766+0000 7fab76d9d640 1 -- 172.21.15.83:0/3452159346 learned_addr learned my addr 172.21.15.83:0/3452159346 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:09.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.766+0000 7fab7759e640 1 --2- 172.21.15.83:0/3452159346 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fab7815dfe0 0x7fab780ccf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.766+0000 7fab6ffff640 1 --2- 172.21.15.83:0/3452159346 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fab780d27f0 0x7fab780b7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.766+0000 7fab76d9d640 1 -- 172.21.15.83:0/3452159346 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fab7815dfe0 msgr2=0x7fab780ccf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.766+0000 7fab76d9d640 1 --2- 172.21.15.83:0/3452159346 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fab7815dfe0 0x7fab780ccf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.766+0000 7fab76d9d640 1 -- 172.21.15.83:0/3452159346 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fab780d27f0 msgr2=0x7fab780b7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.766+0000 7fab76d9d640 1 --2- 172.21.15.83:0/3452159346 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fab780d27f0 0x7fab780b7dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.766+0000 7fab76d9d640 1 -- 172.21.15.83:0/3452159346 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab781f6fe0 con 0x7fab780cf830 2024-03-06T07:30:09.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.766+0000 7fab76d9d640 1 --2- 172.21.15.83:0/3452159346 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab780cf830 0x7fab7814e3c0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7fab6802b910 tx=0x7fab680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:09.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.766+0000 7fab6f7fe640 1 -- 172.21.15.83:0/3452159346 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab68040070 con 0x7fab780cf830 2024-03-06T07:30:09.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.766+0000 7fab6f7fe640 1 -- 172.21.15.83:0/3452159346 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab6803a930 con 0x7fab780cf830 2024-03-06T07:30:09.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.766+0000 7fab7d11e640 1 -- 172.21.15.83:0/3452159346 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab781f7280 con 0x7fab780cf830 2024-03-06T07:30:09.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.766+0000 7fab6f7fe640 1 -- 172.21.15.83:0/3452159346 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab68035040 con 0x7fab780cf830 2024-03-06T07:30:09.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.766+0000 7fab7d11e640 1 -- 172.21.15.83:0/3452159346 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab781f7840 con 0x7fab780cf830 2024-03-06T07:30:09.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.767+0000 7fab7d11e640 1 -- 172.21.15.83:0/3452159346 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab7804f330 con 0x7fab780cf830 2024-03-06T07:30:09.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.768+0000 7fab6f7fe640 1 -- 172.21.15.83:0/3452159346 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fab6804b060 con 0x7fab780cf830 2024-03-06T07:30:09.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.768+0000 7fab6f7fe640 1 --2- 172.21.15.83:0/3452159346 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fab5806b4d0 0x7fab5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.768+0000 7fab6f7fe640 1 -- 172.21.15.83:0/3452159346 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fab680ab950 con 0x7fab780cf830 2024-03-06T07:30:09.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.768+0000 7fab6ffff640 1 --2- 172.21.15.83:0/3452159346 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fab5806b4d0 0x7fab5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.769+0000 7fab6ffff640 1 --2- 172.21.15.83:0/3452159346 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fab5806b4d0 0x7fab5806d990 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fab7815d270 tx=0x7fab60009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:09.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.774+0000 7fab6f7fe640 1 -- 172.21.15.83:0/3452159346 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fab68031800 con 0x7fab780cf830 2024-03-06T07:30:09.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.985+0000 7fab7d11e640 1 -- 172.21.15.83:0/3452159346 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab7815d5e0 con 0x7fab780cf830 2024-03-06T07:30:09.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.986+0000 7fab6f7fe640 1 -- 172.21.15.83:0/3452159346 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7fab6807a140 con 0x7fab780cf830 2024-03-06T07:30:09.988 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:30:09.988 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:30:09.988 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:30:09.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.988+0000 7fab7d11e640 1 -- 172.21.15.83:0/3452159346 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fab5806b4d0 msgr2=0x7fab5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.988+0000 7fab7d11e640 1 --2- 172.21.15.83:0/3452159346 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fab5806b4d0 0x7fab5806d990 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fab7815d270 tx=0x7fab60009190 comp rx=0 tx=0).stop 2024-03-06T07:30:09.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.988+0000 7fab7d11e640 1 -- 172.21.15.83:0/3452159346 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab780cf830 msgr2=0x7fab7814e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.988+0000 7fab7d11e640 1 --2- 172.21.15.83:0/3452159346 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab780cf830 0x7fab7814e3c0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7fab6802b910 tx=0x7fab680046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.988+0000 7fab7d11e640 1 -- 172.21.15.83:0/3452159346 shutdown_connections 2024-03-06T07:30:09.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.988+0000 7fab7d11e640 1 --2- 172.21.15.83:0/3452159346 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fab5806b4d0 0x7fab5806d990 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.988+0000 7fab7d11e640 1 --2- 172.21.15.83:0/3452159346 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fab7815dfe0 0x7fab780ccf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.988+0000 7fab7d11e640 1 --2- 172.21.15.83:0/3452159346 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fab780d27f0 0x7fab780b7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.988+0000 7fab7d11e640 1 --2- 172.21.15.83:0/3452159346 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab780cf830 0x7fab7814e3c0 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.988+0000 7fab7d11e640 1 -- 172.21.15.83:0/3452159346 >> 172.21.15.83:0/3452159346 conn(0x7fab780d3cd0 msgr2=0x7fab7806cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:09.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.988+0000 7fab7d11e640 1 -- 172.21.15.83:0/3452159346 shutdown_connections 2024-03-06T07:30:09.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:09.988+0000 7fab7d11e640 1 -- 172.21.15.83:0/3452159346 wait complete. 2024-03-06T07:30:10.202 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:30:10.202 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:30:11.013 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:11.203 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:11.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.302+0000 7fa22c3d0640 1 Processor -- start 2024-03-06T07:30:11.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.302+0000 7fa22c3d0640 1 -- start start 2024-03-06T07:30:11.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.303+0000 7fa22c3d0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa2240d2710 0x7fa2240b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.303+0000 7fa22c3d0640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2240b3430 con 0x7fa22415dfe0 2024-03-06T07:30:11.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.303+0000 7fa22c3d0640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2240b3710 con 0x7fa2240d22b0 2024-03-06T07:30:11.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.303+0000 7fa22c3d0640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2240b31e0 con 0x7fa2240d2710 2024-03-06T07:30:11.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.303+0000 7fa22a96d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa2240d2710 0x7fa2240b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.303+0000 7fa22a16c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fa22415dfe0 0x7fa2240cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:36608/0 (socket says 172.21.15.83:36608) 2024-03-06T07:30:11.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.303+0000 7fa22a16c640 1 -- 172.21.15.83:0/2776803066 learned_addr learned my addr 172.21.15.83:0/2776803066 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:30:11.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.303+0000 7fa22a96d640 1 -- 172.21.15.83:0/2776803066 >> v1:172.21.15.183:6789/0 conn(0x7fa2240d22b0 legacy=0x7fa2240b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:11.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.303+0000 7fa22916a640 1 -- 172.21.15.83:0/2776803066 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2774252303 0 0) 0x7fa2240b3430 con 0x7fa22415dfe0 2024-03-06T07:30:11.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.303+0000 7fa22a96d640 1 -- 172.21.15.83:0/2776803066 >> v1:172.21.15.83:6789/0 conn(0x7fa22415dfe0 legacy=0x7fa2240cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.303+0000 7fa22a96d640 1 -- 172.21.15.83:0/2776803066 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2240b39f0 con 0x7fa2240d2710 2024-03-06T07:30:11.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.304+0000 7fa22a96d640 1 --2- 172.21.15.83:0/2776803066 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2240d2710 0x7fa2240b4900 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7fa21c009dc0 tx=0x7fa21c02bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf2453fcf77e2319 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:11.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.304+0000 7fa22916a640 1 -- 172.21.15.83:0/2776803066 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21c039070 con 0x7fa2240d2710 2024-03-06T07:30:11.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.304+0000 7fa22916a640 1 -- 172.21.15.83:0/2776803066 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa21c02ccd0 con 0x7fa2240d2710 2024-03-06T07:30:11.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.304+0000 7fa22916a640 1 -- 172.21.15.83:0/2776803066 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21c034660 con 0x7fa2240d2710 2024-03-06T07:30:11.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.305+0000 7fa22c3d0640 1 -- 172.21.15.83:0/2776803066 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2240d2710 msgr2=0x7fa2240b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.305+0000 7fa22c3d0640 1 --2- 172.21.15.83:0/2776803066 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2240d2710 0x7fa2240b4900 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7fa21c009dc0 tx=0x7fa21c02bf20 comp rx=0 tx=0).stop 2024-03-06T07:30:11.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.305+0000 7fa22c3d0640 1 -- 172.21.15.83:0/2776803066 shutdown_connections 2024-03-06T07:30:11.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.305+0000 7fa22c3d0640 1 --2- 172.21.15.83:0/2776803066 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2240d2710 0x7fa2240b4900 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.305+0000 7fa22c3d0640 1 -- 172.21.15.83:0/2776803066 >> 172.21.15.83:0/2776803066 conn(0x7fa2240d3cd0 msgr2=0x7fa2240d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:11.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.305+0000 7fa22c3d0640 1 -- 172.21.15.83:0/2776803066 shutdown_connections 2024-03-06T07:30:11.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.305+0000 7fa22c3d0640 1 -- 172.21.15.83:0/2776803066 wait complete. 2024-03-06T07:30:11.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.306+0000 7fa22c3d0640 1 Processor -- start 2024-03-06T07:30:11.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.306+0000 7fa22c3d0640 1 -- start start 2024-03-06T07:30:11.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.307+0000 7fa22c3d0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa2240d22b0 0x7fa224157b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.307+0000 7fa22c3d0640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa2240d2710 0x7fa2240b1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.307+0000 7fa22c3d0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa22415dfe0 0x7fa2240b1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.307+0000 7fa22c3d0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2240cfa00 con 0x7fa22415dfe0 2024-03-06T07:30:11.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.307+0000 7fa22c3d0640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2240b3d60 con 0x7fa2240d2710 2024-03-06T07:30:11.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.307+0000 7fa22c3d0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2240d1c70 con 0x7fa2240d22b0 2024-03-06T07:30:11.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.308+0000 7fa22a96d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa22415dfe0 0x7fa2240b1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.308+0000 7fa22a96d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa22415dfe0 0x7fa2240b1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37344/0 (socket says 172.21.15.83:37344) 2024-03-06T07:30:11.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.308+0000 7fa22a96d640 1 -- 172.21.15.83:0/2245038624 learned_addr learned my addr 172.21.15.83:0/2245038624 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:11.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.308+0000 7fa22a16c640 1 --2- 172.21.15.83:0/2245038624 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa2240d22b0 0x7fa224157b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.308+0000 7fa22996b640 1 --2- 172.21.15.83:0/2245038624 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa2240d2710 0x7fa2240b1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.308+0000 7fa22a96d640 1 -- 172.21.15.83:0/2245038624 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa2240d22b0 msgr2=0x7fa224157b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.308+0000 7fa22a96d640 1 --2- 172.21.15.83:0/2245038624 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa2240d22b0 0x7fa224157b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.308+0000 7fa22a96d640 1 -- 172.21.15.83:0/2245038624 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa2240d2710 msgr2=0x7fa2240b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.308+0000 7fa22a96d640 1 --2- 172.21.15.83:0/2245038624 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa2240d2710 0x7fa2240b1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.308+0000 7fa22a96d640 1 -- 172.21.15.83:0/2245038624 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2240b8bb0 con 0x7fa22415dfe0 2024-03-06T07:30:11.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.308+0000 7fa22996b640 1 --2- 172.21.15.83:0/2245038624 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa2240d2710 0x7fa2240b1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:11.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.308+0000 7fa22a96d640 1 --2- 172.21.15.83:0/2245038624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa22415dfe0 0x7fa2240b1550 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7fa21c009ee0 tx=0x7fa21c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:11.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.308+0000 7fa21affd640 1 -- 172.21.15.83:0/2245038624 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21c044020 con 0x7fa22415dfe0 2024-03-06T07:30:11.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.308+0000 7fa21affd640 1 -- 172.21.15.83:0/2245038624 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa21c03a040 con 0x7fa22415dfe0 2024-03-06T07:30:11.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.308+0000 7fa22c3d0640 1 -- 172.21.15.83:0/2245038624 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2240b8e90 con 0x7fa22415dfe0 2024-03-06T07:30:11.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.308+0000 7fa21affd640 1 -- 172.21.15.83:0/2245038624 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21c039050 con 0x7fa22415dfe0 2024-03-06T07:30:11.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.308+0000 7fa22c3d0640 1 -- 172.21.15.83:0/2245038624 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2240b9450 con 0x7fa22415dfe0 2024-03-06T07:30:11.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.309+0000 7fa22c3d0640 1 -- 172.21.15.83:0/2245038624 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa22404f330 con 0x7fa22415dfe0 2024-03-06T07:30:11.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.310+0000 7fa21affd640 1 -- 172.21.15.83:0/2245038624 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa21c02cb90 con 0x7fa22415dfe0 2024-03-06T07:30:11.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.310+0000 7fa21affd640 1 --2- 172.21.15.83:0/2245038624 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa20806b4d0 0x7fa20806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.310+0000 7fa21affd640 1 -- 172.21.15.83:0/2245038624 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa21c0ac7c0 con 0x7fa22415dfe0 2024-03-06T07:30:11.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.310+0000 7fa22a16c640 1 --2- 172.21.15.83:0/2245038624 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa20806b4d0 0x7fa20806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.311+0000 7fa22a16c640 1 --2- 172.21.15.83:0/2245038624 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa20806b4d0 0x7fa20806d990 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fa220000f60 tx=0x7fa220007e40 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:11.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.319+0000 7fa21affd640 1 -- 172.21.15.83:0/2245038624 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa21c04e060 con 0x7fa22415dfe0 2024-03-06T07:30:11.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.530+0000 7fa22c3d0640 1 -- 172.21.15.83:0/2245038624 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2240ce960 con 0x7fa22415dfe0 2024-03-06T07:30:11.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.531+0000 7fa21affd640 1 -- 172.21.15.83:0/2245038624 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7fa21c07b690 con 0x7fa22415dfe0 2024-03-06T07:30:11.533 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:30:11.533 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:30:11.533 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:30:11.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.532+0000 7fa22c3d0640 1 -- 172.21.15.83:0/2245038624 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa20806b4d0 msgr2=0x7fa20806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.532+0000 7fa22c3d0640 1 --2- 172.21.15.83:0/2245038624 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa20806b4d0 0x7fa20806d990 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fa220000f60 tx=0x7fa220007e40 comp rx=0 tx=0).stop 2024-03-06T07:30:11.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.533+0000 7fa22c3d0640 1 -- 172.21.15.83:0/2245038624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa22415dfe0 msgr2=0x7fa2240b1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.533+0000 7fa22c3d0640 1 --2- 172.21.15.83:0/2245038624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa22415dfe0 0x7fa2240b1550 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7fa21c009ee0 tx=0x7fa21c004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:11.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.533+0000 7fa22c3d0640 1 -- 172.21.15.83:0/2245038624 shutdown_connections 2024-03-06T07:30:11.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.533+0000 7fa22c3d0640 1 --2- 172.21.15.83:0/2245038624 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa20806b4d0 0x7fa20806d990 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.533+0000 7fa22c3d0640 1 --2- 172.21.15.83:0/2245038624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa22415dfe0 0x7fa2240b1550 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.533+0000 7fa22c3d0640 1 --2- 172.21.15.83:0/2245038624 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa2240d2710 0x7fa2240b1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.533+0000 7fa22c3d0640 1 --2- 172.21.15.83:0/2245038624 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa2240d22b0 0x7fa224157b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.533+0000 7fa22c3d0640 1 -- 172.21.15.83:0/2245038624 >> 172.21.15.83:0/2245038624 conn(0x7fa2240d3cd0 msgr2=0x7fa2240ccdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:11.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.533+0000 7fa22c3d0640 1 -- 172.21.15.83:0/2245038624 shutdown_connections 2024-03-06T07:30:11.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:11.533+0000 7fa22c3d0640 1 -- 172.21.15.83:0/2245038624 wait complete. 2024-03-06T07:30:11.746 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:30:11.746 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:30:12.747 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:12.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.847+0000 7f502073e640 1 Processor -- start 2024-03-06T07:30:12.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.847+0000 7f502073e640 1 -- start start 2024-03-06T07:30:12.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.848+0000 7f502073e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f501815dfe0 0x7f50180cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.848+0000 7f502073e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f50180cf830 0x7f50180d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.848+0000 7f502073e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f50180d27f0 0x7f50180d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.848+0000 7f502073e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50180d2c50 con 0x7f501815dfe0 2024-03-06T07:30:12.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.848+0000 7f502073e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50180b31e0 con 0x7f50180d27f0 2024-03-06T07:30:12.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.848+0000 7f502073e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50180b3d60 con 0x7f50180cf830 2024-03-06T07:30:12.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.848+0000 7f501e4da640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f501815dfe0 0x7f50180cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.848+0000 7f501e4da640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f501815dfe0 0x7f50180cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37354/0 (socket says 172.21.15.83:37354) 2024-03-06T07:30:12.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.848+0000 7f501e4da640 1 -- 172.21.15.83:0/737155715 learned_addr learned my addr 172.21.15.83:0/737155715 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:12.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.848+0000 7f501dcd9640 1 --2- 172.21.15.83:0/737155715 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f50180cf830 0x7f50180d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.848+0000 7f501ecdb640 1 --2- 172.21.15.83:0/737155715 >> v2:172.21.15.183:3300/0 conn(0x7f50180d27f0 0x7f50180d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.848+0000 7f501e4da640 1 -- 172.21.15.83:0/737155715 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f50180cf830 msgr2=0x7f50180d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:12.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.848+0000 7f501e4da640 1 --2- 172.21.15.83:0/737155715 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f50180cf830 0x7f50180d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.848+0000 7f501e4da640 1 -- 172.21.15.83:0/737155715 >> v2:172.21.15.183:3300/0 conn(0x7f50180d27f0 msgr2=0x7f50180d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:12.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.848+0000 7f501e4da640 1 --2- 172.21.15.83:0/737155715 >> v2:172.21.15.183:3300/0 conn(0x7f50180d27f0 0x7f50180d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.848+0000 7f501e4da640 1 -- 172.21.15.83:0/737155715 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50180b3430 con 0x7f501815dfe0 2024-03-06T07:30:12.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.849+0000 7f501e4da640 1 --2- 172.21.15.83:0/737155715 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f501815dfe0 0x7f50180cf2f0 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f5018067ad0 tx=0x7f501402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f7f1c8026566dcb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:12.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.849+0000 7f501d4d8640 1 -- 172.21.15.83:0/737155715 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5014035020 con 0x7f501815dfe0 2024-03-06T07:30:12.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.849+0000 7f501d4d8640 1 -- 172.21.15.83:0/737155715 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5014031870 con 0x7f501815dfe0 2024-03-06T07:30:12.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.849+0000 7f501d4d8640 1 -- 172.21.15.83:0/737155715 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5014031c40 con 0x7f501815dfe0 2024-03-06T07:30:12.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.849+0000 7f502073e640 1 -- 172.21.15.83:0/737155715 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f501815dfe0 msgr2=0x7f50180cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.849+0000 7f502073e640 1 --2- 172.21.15.83:0/737155715 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f501815dfe0 0x7f50180cf2f0 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f5018067ad0 tx=0x7f501402b910 comp rx=0 tx=0).stop 2024-03-06T07:30:12.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.850+0000 7f502073e640 1 -- 172.21.15.83:0/737155715 shutdown_connections 2024-03-06T07:30:12.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.850+0000 7f502073e640 1 --2- 172.21.15.83:0/737155715 >> v2:172.21.15.183:3300/0 conn(0x7f50180d27f0 0x7f50180d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.850+0000 7f502073e640 1 --2- 172.21.15.83:0/737155715 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f50180cf830 0x7f50180d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.850+0000 7f502073e640 1 --2- 172.21.15.83:0/737155715 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f501815dfe0 0x7f50180cf2f0 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.850+0000 7f502073e640 1 -- 172.21.15.83:0/737155715 >> 172.21.15.83:0/737155715 conn(0x7f50180d3cd0 msgr2=0x7f50180d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:12.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.850+0000 7f502073e640 1 -- 172.21.15.83:0/737155715 shutdown_connections 2024-03-06T07:30:12.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.850+0000 7f502073e640 1 -- 172.21.15.83:0/737155715 wait complete. 2024-03-06T07:30:12.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.851+0000 7f502073e640 1 Processor -- start 2024-03-06T07:30:12.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.851+0000 7f502073e640 1 -- start start 2024-03-06T07:30:12.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.851+0000 7f502073e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f50180cf830 0x7f50181f4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.851+0000 7f502073e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50180d27f0 0x7f5018160ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.851+0000 7f502073e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f501815dfe0 0x7f501815fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.851+0000 7f502073e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50180d1c70 con 0x7f50180d27f0 2024-03-06T07:30:12.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.851+0000 7f502073e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50180b39f0 con 0x7f50180cf830 2024-03-06T07:30:12.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.851+0000 7f502073e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50180d1df0 con 0x7f501815dfe0 2024-03-06T07:30:12.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.851+0000 7f501dcd9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50180d27f0 0x7f5018160ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.851+0000 7f501dcd9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50180d27f0 0x7f5018160ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37362/0 (socket says 172.21.15.83:37362) 2024-03-06T07:30:12.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.852+0000 7f501dcd9640 1 -- 172.21.15.83:0/587699976 learned_addr learned my addr 172.21.15.83:0/587699976 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:12.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.852+0000 7f501e4da640 1 --2- 172.21.15.83:0/587699976 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f50180cf830 0x7f50181f4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.852+0000 7f501ecdb640 1 --2- 172.21.15.83:0/587699976 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f501815dfe0 0x7f501815fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.852+0000 7f501dcd9640 1 -- 172.21.15.83:0/587699976 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f501815dfe0 msgr2=0x7f501815fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.852+0000 7f501dcd9640 1 --2- 172.21.15.83:0/587699976 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f501815dfe0 0x7f501815fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.852+0000 7f501dcd9640 1 -- 172.21.15.83:0/587699976 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f50180cf830 msgr2=0x7f50181f4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.852+0000 7f501dcd9640 1 --2- 172.21.15.83:0/587699976 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f50180cf830 0x7f50181f4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.852+0000 7f501dcd9640 1 -- 172.21.15.83:0/587699976 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50180b9740 con 0x7f50180d27f0 2024-03-06T07:30:12.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.852+0000 7f501dcd9640 1 --2- 172.21.15.83:0/587699976 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50180d27f0 0x7f5018160ad0 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f500800d7d0 tx=0x7f500800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:12.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.852+0000 7f500effd640 1 -- 172.21.15.83:0/587699976 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50080096a0 con 0x7f50180d27f0 2024-03-06T07:30:12.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.852+0000 7f500effd640 1 -- 172.21.15.83:0/587699976 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5008019070 con 0x7f50180d27f0 2024-03-06T07:30:12.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.852+0000 7f502073e640 1 -- 172.21.15.83:0/587699976 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50180b9a20 con 0x7f50180d27f0 2024-03-06T07:30:12.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.852+0000 7f500effd640 1 -- 172.21.15.83:0/587699976 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5008009a60 con 0x7f50180d27f0 2024-03-06T07:30:12.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.852+0000 7f502073e640 1 -- 172.21.15.83:0/587699976 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50180cceb0 con 0x7f50180d27f0 2024-03-06T07:30:12.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.853+0000 7f502073e640 1 -- 172.21.15.83:0/587699976 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50180b5200 con 0x7f50180d27f0 2024-03-06T07:30:12.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.854+0000 7f500effd640 1 -- 172.21.15.83:0/587699976 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5008009c00 con 0x7f50180d27f0 2024-03-06T07:30:12.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.854+0000 7f500effd640 1 --2- 172.21.15.83:0/587699976 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4ffc06b4d0 0x7f4ffc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.855+0000 7f500effd640 1 -- 172.21.15.83:0/587699976 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f500808a500 con 0x7f50180d27f0 2024-03-06T07:30:12.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.855+0000 7f501e4da640 1 --2- 172.21.15.83:0/587699976 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4ffc06b4d0 0x7f4ffc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.855+0000 7f501e4da640 1 --2- 172.21.15.83:0/587699976 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4ffc06b4d0 0x7f4ffc06d990 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f5014036000 tx=0x7f501403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:12.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:12.863+0000 7f500effd640 1 -- 172.21.15.83:0/587699976 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f50180b5200 con 0x7f50180d27f0 2024-03-06T07:30:13.015 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:13.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:13.080+0000 7f502073e640 1 -- 172.21.15.83:0/587699976 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f501804f330 con 0x7f50180d27f0 2024-03-06T07:30:13.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:13.081+0000 7f500effd640 1 -- 172.21.15.83:0/587699976 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f5008058cf0 con 0x7f50180d27f0 2024-03-06T07:30:13.083 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:30:13.083 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:30:13.083 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:30:13.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:13.083+0000 7f502073e640 1 -- 172.21.15.83:0/587699976 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4ffc06b4d0 msgr2=0x7f4ffc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:13.083+0000 7f502073e640 1 --2- 172.21.15.83:0/587699976 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4ffc06b4d0 0x7f4ffc06d990 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f5014036000 tx=0x7f501403c040 comp rx=0 tx=0).stop 2024-03-06T07:30:13.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:13.083+0000 7f502073e640 1 -- 172.21.15.83:0/587699976 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50180d27f0 msgr2=0x7f5018160ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:13.083+0000 7f502073e640 1 --2- 172.21.15.83:0/587699976 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50180d27f0 0x7f5018160ad0 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f500800d7d0 tx=0x7f500800dc90 comp rx=0 tx=0).stop 2024-03-06T07:30:13.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:13.083+0000 7f502073e640 1 -- 172.21.15.83:0/587699976 shutdown_connections 2024-03-06T07:30:13.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:13.083+0000 7f502073e640 1 --2- 172.21.15.83:0/587699976 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4ffc06b4d0 0x7f4ffc06d990 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:13.083+0000 7f502073e640 1 --2- 172.21.15.83:0/587699976 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f501815dfe0 0x7f501815fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:13.083+0000 7f502073e640 1 --2- 172.21.15.83:0/587699976 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50180d27f0 0x7f5018160ad0 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:13.083+0000 7f502073e640 1 --2- 172.21.15.83:0/587699976 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f50180cf830 0x7f50181f4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:13.083+0000 7f502073e640 1 -- 172.21.15.83:0/587699976 >> 172.21.15.83:0/587699976 conn(0x7f50180d3cd0 msgr2=0x7f501806cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:13.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:13.083+0000 7f502073e640 1 -- 172.21.15.83:0/587699976 shutdown_connections 2024-03-06T07:30:13.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:13.083+0000 7f502073e640 1 -- 172.21.15.83:0/587699976 wait complete. 2024-03-06T07:30:13.296 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:30:13.296 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:30:14.298 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:14.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.397+0000 7f19a40f4640 1 Processor -- start 2024-03-06T07:30:14.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.397+0000 7f19a40f4640 1 -- start start 2024-03-06T07:30:14.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.398+0000 7f19a40f4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f199c0d4760 0x7f199c0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.398+0000 7f19a40f4640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f199c0d0a80 0x7f199c0cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.398+0000 7f19a40f4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f199c0d25d0 con 0x7f199c0b4170 2024-03-06T07:30:14.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.398+0000 7f19a40f4640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f199c0d4bf0 con 0x7f199c0d4760 2024-03-06T07:30:14.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.398+0000 7f19a40f4640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f199c0ce9e0 con 0x7f199c0d0a80 2024-03-06T07:30:14.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.398+0000 7f19a168f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f199c0d4760 0x7f199c0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.398+0000 7f19a168f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f199c0d4760 0x7f199c0b6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37368/0 (socket says 172.21.15.83:37368) 2024-03-06T07:30:14.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.398+0000 7f19a168f640 1 -- 172.21.15.83:0/2806776828 learned_addr learned my addr 172.21.15.83:0/2806776828 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:14.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.398+0000 7f19a2691640 1 --2- 172.21.15.83:0/2806776828 >> v2:172.21.15.183:3300/0 conn(0x7f199c0d0a80 0x7f199c0cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.398+0000 7f19a168f640 1 -- 172.21.15.83:0/2806776828 >> v2:172.21.15.183:3300/0 conn(0x7f199c0d0a80 msgr2=0x7f199c0cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:14.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.398+0000 7f19a168f640 1 --2- 172.21.15.83:0/2806776828 >> v2:172.21.15.183:3300/0 conn(0x7f199c0d0a80 0x7f199c0cee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.398+0000 7f19a168f640 1 -- 172.21.15.83:0/2806776828 >> v1:172.21.15.183:6789/0 conn(0x7f199c0b4170 legacy=0x7f199c0d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:14.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.398+0000 7f19a168f640 1 -- 172.21.15.83:0/2806776828 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f199c0d2870 con 0x7f199c0d4760 2024-03-06T07:30:14.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.399+0000 7f19a168f640 1 --2- 172.21.15.83:0/2806776828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f199c0d4760 0x7f199c0b6a90 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f199c08df30 tx=0x7f198c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a32974af60b3fc23 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:14.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.399+0000 7f19a0e8e640 1 -- 172.21.15.83:0/2806776828 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f198c034070 con 0x7f199c0d4760 2024-03-06T07:30:14.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.399+0000 7f19a0e8e640 1 -- 172.21.15.83:0/2806776828 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f198c030860 con 0x7f199c0d4760 2024-03-06T07:30:14.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.399+0000 7f19a0e8e640 1 -- 172.21.15.83:0/2806776828 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f198c030c30 con 0x7f199c0d4760 2024-03-06T07:30:14.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.400+0000 7f19a40f4640 1 -- 172.21.15.83:0/2806776828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f199c0d4760 msgr2=0x7f199c0b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.400+0000 7f19a40f4640 1 --2- 172.21.15.83:0/2806776828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f199c0d4760 0x7f199c0b6a90 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f199c08df30 tx=0x7f198c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.400+0000 7f19a40f4640 1 -- 172.21.15.83:0/2806776828 shutdown_connections 2024-03-06T07:30:14.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.400+0000 7f19a40f4640 1 --2- 172.21.15.83:0/2806776828 >> v2:172.21.15.183:3300/0 conn(0x7f199c0d0a80 0x7f199c0cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.400+0000 7f19a40f4640 1 --2- 172.21.15.83:0/2806776828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f199c0d4760 0x7f199c0b6a90 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.400+0000 7f19a40f4640 1 -- 172.21.15.83:0/2806776828 >> 172.21.15.83:0/2806776828 conn(0x7f199c0d3c40 msgr2=0x7f199c0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:14.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.400+0000 7f19a40f4640 1 -- 172.21.15.83:0/2806776828 shutdown_connections 2024-03-06T07:30:14.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.400+0000 7f19a40f4640 1 -- 172.21.15.83:0/2806776828 wait complete. 2024-03-06T07:30:14.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.401+0000 7f19a40f4640 1 Processor -- start 2024-03-06T07:30:14.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.401+0000 7f19a40f4640 1 -- start start 2024-03-06T07:30:14.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.402+0000 7f19a40f4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f199c0b4170 0x7f199c156cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.402+0000 7f19a40f4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f199c0d0a80 0x7f199c158950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.402+0000 7f19a40f4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f199c0cb820 0x7f199c1f3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.402+0000 7f19a40f4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f199c0b4dd0 con 0x7f199c0cb820 2024-03-06T07:30:14.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.402+0000 7f19a40f4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f199c0b4550 con 0x7f199c0d0a80 2024-03-06T07:30:14.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.402+0000 7f19a40f4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f199c0b4f50 con 0x7f199c0b4170 2024-03-06T07:30:14.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.402+0000 7f19a2691640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f199c0cb820 0x7f199c1f3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.402+0000 7f19a2691640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f199c0cb820 0x7f199c1f3f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37370/0 (socket says 172.21.15.83:37370) 2024-03-06T07:30:14.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.402+0000 7f19a2691640 1 -- 172.21.15.83:0/919277777 learned_addr learned my addr 172.21.15.83:0/919277777 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:14.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.402+0000 7f19a168f640 1 --2- 172.21.15.83:0/919277777 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f199c0d0a80 0x7f199c158950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.402+0000 7f19a1e90640 1 --2- 172.21.15.83:0/919277777 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f199c0b4170 0x7f199c156cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.403+0000 7f19a2691640 1 -- 172.21.15.83:0/919277777 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f199c0b4170 msgr2=0x7f199c156cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.403+0000 7f19a2691640 1 --2- 172.21.15.83:0/919277777 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f199c0b4170 0x7f199c156cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.403+0000 7f19a2691640 1 -- 172.21.15.83:0/919277777 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f199c0d0a80 msgr2=0x7f199c158950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.403+0000 7f19a2691640 1 --2- 172.21.15.83:0/919277777 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f199c0d0a80 0x7f199c158950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.403+0000 7f19a2691640 1 -- 172.21.15.83:0/919277777 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f199c158fe0 con 0x7f199c0cb820 2024-03-06T07:30:14.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.403+0000 7f19a1e90640 1 --2- 172.21.15.83:0/919277777 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f199c0b4170 0x7f199c156cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:14.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.403+0000 7f19a2691640 1 --2- 172.21.15.83:0/919277777 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f199c0cb820 0x7f199c1f3f90 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f1994007e40 tx=0x7f199400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:14.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.403+0000 7f19927fc640 1 -- 172.21.15.83:0/919277777 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1994014620 con 0x7f199c0cb820 2024-03-06T07:30:14.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.403+0000 7f19927fc640 1 -- 172.21.15.83:0/919277777 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1994019070 con 0x7f199c0cb820 2024-03-06T07:30:14.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.403+0000 7f19927fc640 1 -- 172.21.15.83:0/919277777 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1994014a20 con 0x7f199c0cb820 2024-03-06T07:30:14.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.403+0000 7f19a40f4640 1 -- 172.21.15.83:0/919277777 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f199c15c510 con 0x7f199c0cb820 2024-03-06T07:30:14.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.403+0000 7f19a40f4640 1 -- 172.21.15.83:0/919277777 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f199c15ca90 con 0x7f199c0cb820 2024-03-06T07:30:14.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.404+0000 7f19a40f4640 1 -- 172.21.15.83:0/919277777 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f199c1f72c0 con 0x7f199c0cb820 2024-03-06T07:30:14.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.404+0000 7f19927fc640 1 -- 172.21.15.83:0/919277777 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1994014bc0 con 0x7f199c0cb820 2024-03-06T07:30:14.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.405+0000 7f19927fc640 1 --2- 172.21.15.83:0/919277777 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f198006b500 0x7f198006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.405+0000 7f19927fc640 1 -- 172.21.15.83:0/919277777 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f199408aa00 con 0x7f199c0cb820 2024-03-06T07:30:14.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.405+0000 7f19a1e90640 1 --2- 172.21.15.83:0/919277777 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f198006b500 0x7f198006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.405+0000 7f19a1e90640 1 --2- 172.21.15.83:0/919277777 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f198006b500 0x7f198006d9c0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f1998007fd0 tx=0x7f1998009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:14.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.409+0000 7f19927fc640 1 -- 172.21.15.83:0/919277777 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f19940598d0 con 0x7f199c0cb820 2024-03-06T07:30:14.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.624+0000 7f19a40f4640 1 -- 172.21.15.83:0/919277777 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f199c04f330 con 0x7f199c0cb820 2024-03-06T07:30:14.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.625+0000 7f19927fc640 1 -- 172.21.15.83:0/919277777 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f19940591d0 con 0x7f199c0cb820 2024-03-06T07:30:14.627 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:30:14.627 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:30:14.627 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:30:14.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.626+0000 7f19a40f4640 1 -- 172.21.15.83:0/919277777 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f198006b500 msgr2=0x7f198006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.626+0000 7f19a40f4640 1 --2- 172.21.15.83:0/919277777 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f198006b500 0x7f198006d9c0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f1998007fd0 tx=0x7f1998009040 comp rx=0 tx=0).stop 2024-03-06T07:30:14.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.626+0000 7f19a40f4640 1 -- 172.21.15.83:0/919277777 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f199c0cb820 msgr2=0x7f199c1f3f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.626+0000 7f19a40f4640 1 --2- 172.21.15.83:0/919277777 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f199c0cb820 0x7f199c1f3f90 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f1994007e40 tx=0x7f199400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.627+0000 7f19a40f4640 1 -- 172.21.15.83:0/919277777 shutdown_connections 2024-03-06T07:30:14.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.627+0000 7f19a40f4640 1 --2- 172.21.15.83:0/919277777 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f198006b500 0x7f198006d9c0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.627+0000 7f19a40f4640 1 --2- 172.21.15.83:0/919277777 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f199c0cb820 0x7f199c1f3f90 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.627+0000 7f19a40f4640 1 --2- 172.21.15.83:0/919277777 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f199c0d0a80 0x7f199c158950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.627+0000 7f19a40f4640 1 --2- 172.21.15.83:0/919277777 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f199c0b4170 0x7f199c156cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.627+0000 7f19a40f4640 1 -- 172.21.15.83:0/919277777 >> 172.21.15.83:0/919277777 conn(0x7f199c0d3c40 msgr2=0x7f199c0b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:14.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.627+0000 7f19a40f4640 1 -- 172.21.15.83:0/919277777 shutdown_connections 2024-03-06T07:30:14.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:14.627+0000 7f19a40f4640 1 -- 172.21.15.83:0/919277777 wait complete. 2024-03-06T07:30:14.841 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:30:14.841 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:30:15.017 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:15.842 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:15.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.945+0000 7f90dd3e9640 1 Processor -- start 2024-03-06T07:30:15.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.945+0000 7f90dd3e9640 1 -- start start 2024-03-06T07:30:15.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.945+0000 7f90dd3e9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f90d815bb20 0x7f90d80d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.945+0000 7f90dd3e9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f90d80cd8c0 0x7f90d80d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.945+0000 7f90dd3e9640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90d8161980 con 0x7f90d80cfff0 2024-03-06T07:30:15.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.945+0000 7f90dd3e9640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90d8161c60 con 0x7f90d815bb20 2024-03-06T07:30:15.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.945+0000 7f90dd3e9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90d8161f40 con 0x7f90d80cd8c0 2024-03-06T07:30:15.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.946+0000 7f90d6ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f90d815bb20 0x7f90d80d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.946+0000 7f90cffff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f90d80cfff0 0x7f90d80ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:36624/0 (socket says 172.21.15.83:36624) 2024-03-06T07:30:15.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.946+0000 7f90cffff640 1 -- 172.21.15.83:0/1300057850 learned_addr learned my addr 172.21.15.83:0/1300057850 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:30:15.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.946+0000 7f90d77fe640 1 --2- 172.21.15.83:0/1300057850 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f90d80cd8c0 0x7f90d80d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.946+0000 7f90d6ffd640 1 -- 172.21.15.83:0/1300057850 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f90d80cd8c0 msgr2=0x7f90d80d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:15.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.946+0000 7f90d6ffd640 1 --2- 172.21.15.83:0/1300057850 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f90d80cd8c0 0x7f90d80d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.946+0000 7f90d6ffd640 1 -- 172.21.15.83:0/1300057850 >> v1:172.21.15.83:6789/0 conn(0x7f90d80cfff0 legacy=0x7f90d80ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.946+0000 7f90d6ffd640 1 -- 172.21.15.83:0/1300057850 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d8162220 con 0x7f90d815bb20 2024-03-06T07:30:15.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.946+0000 7f90d67fc640 1 -- 172.21.15.83:0/1300057850 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3941131497 0 0) 0x7f90d8161980 con 0x7f90d80cfff0 2024-03-06T07:30:15.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.946+0000 7f90d77fe640 1 --2- 172.21.15.83:0/1300057850 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f90d80cd8c0 0x7f90d80d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:15.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.947+0000 7f90d6ffd640 1 --2- 172.21.15.83:0/1300057850 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90d815bb20 0x7f90d80d1df0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f90c80097a0 tx=0x7f90c802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f578489b820043a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.947+0000 7f90d67fc640 1 -- 172.21.15.83:0/1300057850 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c8035020 con 0x7f90d815bb20 2024-03-06T07:30:15.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.947+0000 7f90d67fc640 1 -- 172.21.15.83:0/1300057850 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90c8031830 con 0x7f90d815bb20 2024-03-06T07:30:15.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.947+0000 7f90d67fc640 1 -- 172.21.15.83:0/1300057850 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c8031c00 con 0x7f90d815bb20 2024-03-06T07:30:15.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.947+0000 7f90dd3e9640 1 -- 172.21.15.83:0/1300057850 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90d815bb20 msgr2=0x7f90d80d1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.947+0000 7f90dd3e9640 1 --2- 172.21.15.83:0/1300057850 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90d815bb20 0x7f90d80d1df0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f90c80097a0 tx=0x7f90c802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.947+0000 7f90dd3e9640 1 -- 172.21.15.83:0/1300057850 shutdown_connections 2024-03-06T07:30:15.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.947+0000 7f90dd3e9640 1 --2- 172.21.15.83:0/1300057850 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f90d80cd8c0 0x7f90d80d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.947+0000 7f90dd3e9640 1 --2- 172.21.15.83:0/1300057850 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90d815bb20 0x7f90d80d1df0 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.947+0000 7f90dd3e9640 1 -- 172.21.15.83:0/1300057850 >> 172.21.15.83:0/1300057850 conn(0x7f90d809a9f0 msgr2=0x7f90d809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:15.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.947+0000 7f90dd3e9640 1 -- 172.21.15.83:0/1300057850 shutdown_connections 2024-03-06T07:30:15.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.947+0000 7f90dd3e9640 1 -- 172.21.15.83:0/1300057850 wait complete. 2024-03-06T07:30:15.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.948+0000 7f90dd3e9640 1 Processor -- start 2024-03-06T07:30:15.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.949+0000 7f90dd3e9640 1 -- start start 2024-03-06T07:30:15.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.949+0000 7f90dd3e9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90d80cd8c0 0x7f90d80c6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.949+0000 7f90dd3e9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90d80cfff0 0x7f90d80b2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.949+0000 7f90dd3e9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f90d815bb20 0x7f90d8157160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.949+0000 7f90dd3e9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d80d24b0 con 0x7f90d80cfff0 2024-03-06T07:30:15.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.949+0000 7f90dd3e9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d80d2330 con 0x7f90d80cd8c0 2024-03-06T07:30:15.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.949+0000 7f90dd3e9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90d80d1980 con 0x7f90d815bb20 2024-03-06T07:30:15.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.949+0000 7f90cffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90d80cfff0 0x7f90d80b2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.949+0000 7f90cffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90d80cfff0 0x7f90d80b2320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:37388/0 (socket says 172.21.15.83:37388) 2024-03-06T07:30:15.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.949+0000 7f90cffff640 1 -- 172.21.15.83:0/92250897 learned_addr learned my addr 172.21.15.83:0/92250897 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:15.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.950+0000 7f90d77fe640 1 --2- 172.21.15.83:0/92250897 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f90d815bb20 0x7f90d8157160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.950+0000 7f90d6ffd640 1 --2- 172.21.15.83:0/92250897 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90d80cd8c0 0x7f90d80c6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.950+0000 7f90cffff640 1 -- 172.21.15.83:0/92250897 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f90d815bb20 msgr2=0x7f90d8157160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.950+0000 7f90cffff640 1 --2- 172.21.15.83:0/92250897 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f90d815bb20 0x7f90d8157160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.950+0000 7f90cffff640 1 -- 172.21.15.83:0/92250897 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90d80cd8c0 msgr2=0x7f90d80c6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.950+0000 7f90cffff640 1 --2- 172.21.15.83:0/92250897 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90d80cd8c0 0x7f90d80c6310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.950+0000 7f90cffff640 1 -- 172.21.15.83:0/92250897 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d80c6b40 con 0x7f90d80cfff0 2024-03-06T07:30:15.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.950+0000 7f90d77fe640 1 --2- 172.21.15.83:0/92250897 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f90d815bb20 0x7f90d8157160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:15.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.950+0000 7f90d6ffd640 1 --2- 172.21.15.83:0/92250897 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90d80cd8c0 0x7f90d80c6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:15.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.950+0000 7f90cffff640 1 --2- 172.21.15.83:0/92250897 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90d80cfff0 0x7f90d80b2320 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f90c000c4d0 tx=0x7f90c000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.950+0000 7f90cf7fe640 1 -- 172.21.15.83:0/92250897 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c0018070 con 0x7f90d80cfff0 2024-03-06T07:30:15.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.950+0000 7f90cf7fe640 1 -- 172.21.15.83:0/92250897 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90c0007500 con 0x7f90d80cfff0 2024-03-06T07:30:15.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.950+0000 7f90cf7fe640 1 -- 172.21.15.83:0/92250897 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c0004a20 con 0x7f90d80cfff0 2024-03-06T07:30:15.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.950+0000 7f90dd3e9640 1 -- 172.21.15.83:0/92250897 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90d80c6e20 con 0x7f90d80cfff0 2024-03-06T07:30:15.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.950+0000 7f90dd3e9640 1 -- 172.21.15.83:0/92250897 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90d80c6850 con 0x7f90d80cfff0 2024-03-06T07:30:15.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.951+0000 7f90dd3e9640 1 -- 172.21.15.83:0/92250897 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90d804f330 con 0x7f90d80cfff0 2024-03-06T07:30:15.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.952+0000 7f90cf7fe640 1 -- 172.21.15.83:0/92250897 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f90c0004bc0 con 0x7f90d80cfff0 2024-03-06T07:30:15.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.952+0000 7f90cf7fe640 1 --2- 172.21.15.83:0/92250897 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f90b806b4d0 0x7f90b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.953+0000 7f90cf7fe640 1 -- 172.21.15.83:0/92250897 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f90c00894c0 con 0x7f90d80cfff0 2024-03-06T07:30:15.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.953+0000 7f90d6ffd640 1 --2- 172.21.15.83:0/92250897 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f90b806b4d0 0x7f90b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.953+0000 7f90d6ffd640 1 --2- 172.21.15.83:0/92250897 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f90b806b4d0 0x7f90b806d990 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f90c8036000 tx=0x7f90c8008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:15.957+0000 7f90cf7fe640 1 -- 172.21.15.83:0/92250897 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f90c0058390 con 0x7f90d80cfff0 2024-03-06T07:30:16.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:16.175+0000 7f90dd3e9640 1 -- 172.21.15.83:0/92250897 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90d815e150 con 0x7f90d80cfff0 2024-03-06T07:30:16.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:16.175+0000 7f90cf7fe640 1 -- 172.21.15.83:0/92250897 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f90c0006e20 con 0x7f90d80cfff0 2024-03-06T07:30:16.177 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:30:16.177 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:30:16.178 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:30:16.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:16.177+0000 7f90dd3e9640 1 -- 172.21.15.83:0/92250897 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f90b806b4d0 msgr2=0x7f90b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:16.177+0000 7f90dd3e9640 1 --2- 172.21.15.83:0/92250897 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f90b806b4d0 0x7f90b806d990 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f90c8036000 tx=0x7f90c8008030 comp rx=0 tx=0).stop 2024-03-06T07:30:16.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:16.177+0000 7f90dd3e9640 1 -- 172.21.15.83:0/92250897 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90d80cfff0 msgr2=0x7f90d80b2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:16.177+0000 7f90dd3e9640 1 --2- 172.21.15.83:0/92250897 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90d80cfff0 0x7f90d80b2320 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f90c000c4d0 tx=0x7f90c000c990 comp rx=0 tx=0).stop 2024-03-06T07:30:16.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:16.177+0000 7f90dd3e9640 1 -- 172.21.15.83:0/92250897 shutdown_connections 2024-03-06T07:30:16.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:16.177+0000 7f90dd3e9640 1 --2- 172.21.15.83:0/92250897 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f90b806b4d0 0x7f90b806d990 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:16.177+0000 7f90dd3e9640 1 --2- 172.21.15.83:0/92250897 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f90d815bb20 0x7f90d8157160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:16.177+0000 7f90dd3e9640 1 --2- 172.21.15.83:0/92250897 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90d80cfff0 0x7f90d80b2320 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:16.177+0000 7f90dd3e9640 1 --2- 172.21.15.83:0/92250897 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90d80cd8c0 0x7f90d80c6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:16.177+0000 7f90dd3e9640 1 -- 172.21.15.83:0/92250897 >> 172.21.15.83:0/92250897 conn(0x7f90d809a9f0 msgr2=0x7f90d809add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:16.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:16.177+0000 7f90dd3e9640 1 -- 172.21.15.83:0/92250897 shutdown_connections 2024-03-06T07:30:16.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:16.177+0000 7f90dd3e9640 1 -- 172.21.15.83:0/92250897 wait complete. 2024-03-06T07:30:16.391 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:30:16.391 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:30:17.019 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:17.393 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:17.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.491+0000 7f2a3b8c3640 1 Processor -- start 2024-03-06T07:30:17.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.492+0000 7f2a3b8c3640 1 -- start start 2024-03-06T07:30:17.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.492+0000 7f2a3b8c3640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f2a3415bb20 0x7f2a340d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.492+0000 7f2a3b8c3640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2a340cfff0 0x7f2a340d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.492+0000 7f2a3b8c3640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a34161980 con 0x7f2a340ce4b0 2024-03-06T07:30:17.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.492+0000 7f2a3b8c3640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a34161c60 con 0x7f2a340cfff0 2024-03-06T07:30:17.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.492+0000 7f2a3b8c3640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a34161f40 con 0x7f2a3415bb20 2024-03-06T07:30:17.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.493+0000 7f2a38e5e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2a340cfff0 0x7f2a340d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.493+0000 7f2a38e5e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2a340cfff0 0x7f2a340d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50386/0 (socket says 172.21.15.83:50386) 2024-03-06T07:30:17.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.493+0000 7f2a38e5e640 1 -- 172.21.15.83:0/1837241460 learned_addr learned my addr 172.21.15.83:0/1837241460 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:17.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.493+0000 7f2a3965f640 1 --2- 172.21.15.83:0/1837241460 >> v2:172.21.15.183:3300/0 conn(0x7f2a3415bb20 0x7f2a340d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.493+0000 7f2a38e5e640 1 -- 172.21.15.83:0/1837241460 >> v2:172.21.15.183:3300/0 conn(0x7f2a3415bb20 msgr2=0x7f2a340d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:17.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.493+0000 7f2a38e5e640 1 --2- 172.21.15.83:0/1837241460 >> v2:172.21.15.183:3300/0 conn(0x7f2a3415bb20 0x7f2a340d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.493+0000 7f2a38e5e640 1 -- 172.21.15.83:0/1837241460 >> v1:172.21.15.183:6789/0 conn(0x7f2a340ce4b0 legacy=0x7f2a340ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.493+0000 7f2a38e5e640 1 -- 172.21.15.83:0/1837241460 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a34162220 con 0x7f2a340cfff0 2024-03-06T07:30:17.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.494+0000 7f2a38e5e640 1 --2- 172.21.15.83:0/1837241460 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a340cfff0 0x7f2a340d04a0 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f2a340d2bc0 tx=0x7f2a2402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87075037c62bec97 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:17.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.494+0000 7f2a2bfff640 1 -- 172.21.15.83:0/1837241460 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a24034070 con 0x7f2a340cfff0 2024-03-06T07:30:17.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.494+0000 7f2a2bfff640 1 -- 172.21.15.83:0/1837241460 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a24030860 con 0x7f2a340cfff0 2024-03-06T07:30:17.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.494+0000 7f2a2bfff640 1 -- 172.21.15.83:0/1837241460 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a24030c30 con 0x7f2a340cfff0 2024-03-06T07:30:17.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.495+0000 7f2a3b8c3640 1 -- 172.21.15.83:0/1837241460 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a340cfff0 msgr2=0x7f2a340d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.495+0000 7f2a3b8c3640 1 --2- 172.21.15.83:0/1837241460 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a340cfff0 0x7f2a340d04a0 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f2a340d2bc0 tx=0x7f2a2402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.495+0000 7f2a3b8c3640 1 -- 172.21.15.83:0/1837241460 shutdown_connections 2024-03-06T07:30:17.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.495+0000 7f2a3b8c3640 1 --2- 172.21.15.83:0/1837241460 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a340cfff0 0x7f2a340d04a0 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.495+0000 7f2a3b8c3640 1 --2- 172.21.15.83:0/1837241460 >> v2:172.21.15.183:3300/0 conn(0x7f2a3415bb20 0x7f2a340d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.495+0000 7f2a3b8c3640 1 -- 172.21.15.83:0/1837241460 >> 172.21.15.83:0/1837241460 conn(0x7f2a3409a9f0 msgr2=0x7f2a3409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:17.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.495+0000 7f2a3b8c3640 1 -- 172.21.15.83:0/1837241460 shutdown_connections 2024-03-06T07:30:17.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.495+0000 7f2a3b8c3640 1 -- 172.21.15.83:0/1837241460 wait complete. 2024-03-06T07:30:17.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.496+0000 7f2a3b8c3640 1 Processor -- start 2024-03-06T07:30:17.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.496+0000 7f2a3b8c3640 1 -- start start 2024-03-06T07:30:17.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.497+0000 7f2a3b8c3640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2a340ce4b0 0x7f2a34160b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.497+0000 7f2a3b8c3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a340cfff0 0x7f2a341f4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.497+0000 7f2a3b8c3640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2a3415bb20 0x7f2a341f4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.497+0000 7f2a3b8c3640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a340d24b0 con 0x7f2a340cfff0 2024-03-06T07:30:17.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.497+0000 7f2a3b8c3640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a340d2330 con 0x7f2a3415bb20 2024-03-06T07:30:17.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.497+0000 7f2a3b8c3640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a340d1980 con 0x7f2a340ce4b0 2024-03-06T07:30:17.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.497+0000 7f2a38e5e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a340cfff0 0x7f2a341f4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.497+0000 7f2a38e5e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a340cfff0 0x7f2a341f4120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50400/0 (socket says 172.21.15.83:50400) 2024-03-06T07:30:17.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.497+0000 7f2a38e5e640 1 -- 172.21.15.83:0/2145966288 learned_addr learned my addr 172.21.15.83:0/2145966288 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:17.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.497+0000 7f2a3965f640 1 --2- 172.21.15.83:0/2145966288 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2a340ce4b0 0x7f2a34160b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.497+0000 7f2a39e60640 1 --2- 172.21.15.83:0/2145966288 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2a3415bb20 0x7f2a341f4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.497+0000 7f2a38e5e640 1 -- 172.21.15.83:0/2145966288 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2a340ce4b0 msgr2=0x7f2a34160b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.497+0000 7f2a38e5e640 1 --2- 172.21.15.83:0/2145966288 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2a340ce4b0 0x7f2a34160b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.497+0000 7f2a38e5e640 1 -- 172.21.15.83:0/2145966288 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2a3415bb20 msgr2=0x7f2a341f4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.497+0000 7f2a38e5e640 1 --2- 172.21.15.83:0/2145966288 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2a3415bb20 0x7f2a341f4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.497+0000 7f2a38e5e640 1 -- 172.21.15.83:0/2145966288 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a34155860 con 0x7f2a340cfff0 2024-03-06T07:30:17.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.497+0000 7f2a38e5e640 1 --2- 172.21.15.83:0/2145966288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a340cfff0 0x7f2a341f4120 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f2a2402b9e0 tx=0x7f2a24004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:17.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.497+0000 7f2a29ffb640 1 -- 172.21.15.83:0/2145966288 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a24048070 con 0x7f2a340cfff0 2024-03-06T07:30:17.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.498+0000 7f2a29ffb640 1 -- 172.21.15.83:0/2145966288 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a24030660 con 0x7f2a340cfff0 2024-03-06T07:30:17.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.498+0000 7f2a3b8c3640 1 -- 172.21.15.83:0/2145966288 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a3415e320 con 0x7f2a340cfff0 2024-03-06T07:30:17.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.498+0000 7f2a29ffb640 1 -- 172.21.15.83:0/2145966288 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a24034040 con 0x7f2a340cfff0 2024-03-06T07:30:17.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.498+0000 7f2a3b8c3640 1 -- 172.21.15.83:0/2145966288 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a3415e8c0 con 0x7f2a340cfff0 2024-03-06T07:30:17.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.499+0000 7f2a3b8c3640 1 -- 172.21.15.83:0/2145966288 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a3404f330 con 0x7f2a340cfff0 2024-03-06T07:30:17.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.499+0000 7f2a29ffb640 1 -- 172.21.15.83:0/2145966288 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2a2404b070 con 0x7f2a340cfff0 2024-03-06T07:30:17.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.500+0000 7f2a29ffb640 1 --2- 172.21.15.83:0/2145966288 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2a1806b500 0x7f2a1806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.500+0000 7f2a3965f640 1 --2- 172.21.15.83:0/2145966288 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2a1806b500 0x7f2a1806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.500+0000 7f2a29ffb640 1 -- 172.21.15.83:0/2145966288 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2a240ac0f0 con 0x7f2a340cfff0 2024-03-06T07:30:17.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.500+0000 7f2a3965f640 1 --2- 172.21.15.83:0/2145966288 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2a1806b500 0x7f2a1806d9c0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f2a30009aa0 tx=0x7f2a30009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:17.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.504+0000 7f2a29ffb640 1 -- 172.21.15.83:0/2145966288 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2a2407a8e0 con 0x7f2a340cfff0 2024-03-06T07:30:17.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.756+0000 7f2a3b8c3640 1 -- 172.21.15.83:0/2145966288 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a3415df90 con 0x7f2a340cfff0 2024-03-06T07:30:17.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.757+0000 7f2a29ffb640 1 -- 172.21.15.83:0/2145966288 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f2a2407e980 con 0x7f2a340cfff0 2024-03-06T07:30:17.759 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:30:17.760 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:30:17.760 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:30:17.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.759+0000 7f2a3b8c3640 1 -- 172.21.15.83:0/2145966288 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2a1806b500 msgr2=0x7f2a1806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.759+0000 7f2a3b8c3640 1 --2- 172.21.15.83:0/2145966288 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2a1806b500 0x7f2a1806d9c0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f2a30009aa0 tx=0x7f2a30009190 comp rx=0 tx=0).stop 2024-03-06T07:30:17.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.759+0000 7f2a3b8c3640 1 -- 172.21.15.83:0/2145966288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a340cfff0 msgr2=0x7f2a341f4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.759+0000 7f2a3b8c3640 1 --2- 172.21.15.83:0/2145966288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a340cfff0 0x7f2a341f4120 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f2a2402b9e0 tx=0x7f2a24004730 comp rx=0 tx=0).stop 2024-03-06T07:30:17.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.759+0000 7f2a3b8c3640 1 -- 172.21.15.83:0/2145966288 shutdown_connections 2024-03-06T07:30:17.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.759+0000 7f2a3b8c3640 1 --2- 172.21.15.83:0/2145966288 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2a1806b500 0x7f2a1806d9c0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.759+0000 7f2a3b8c3640 1 --2- 172.21.15.83:0/2145966288 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2a3415bb20 0x7f2a341f4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.759+0000 7f2a3b8c3640 1 --2- 172.21.15.83:0/2145966288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a340cfff0 0x7f2a341f4120 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.759+0000 7f2a3b8c3640 1 --2- 172.21.15.83:0/2145966288 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2a340ce4b0 0x7f2a34160b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.759+0000 7f2a3b8c3640 1 -- 172.21.15.83:0/2145966288 >> 172.21.15.83:0/2145966288 conn(0x7f2a3409a9f0 msgr2=0x7f2a3406eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:17.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.759+0000 7f2a3b8c3640 1 -- 172.21.15.83:0/2145966288 shutdown_connections 2024-03-06T07:30:17.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:17.759+0000 7f2a3b8c3640 1 -- 172.21.15.83:0/2145966288 wait complete. 2024-03-06T07:30:17.973 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:30:17.973 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:30:18.974 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:19.021 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:19.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.075+0000 7ff239f3e640 1 Processor -- start 2024-03-06T07:30:19.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.075+0000 7ff239f3e640 1 -- start start 2024-03-06T07:30:19.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.075+0000 7ff239f3e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff2340d2710 0x7ff2340b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.075+0000 7ff239f3e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2340b3430 con 0x7ff2340d22b0 2024-03-06T07:30:19.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.075+0000 7ff239f3e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2340b3710 con 0x7ff23415dfe0 2024-03-06T07:30:19.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.075+0000 7ff239f3e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2340b31e0 con 0x7ff2340d2710 2024-03-06T07:30:19.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.076+0000 7ff233fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff2340d2710 0x7ff2340b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.076+0000 7ff233fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff2340d2710 0x7ff2340b4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50416/0 (socket says 172.21.15.83:50416) 2024-03-06T07:30:19.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.076+0000 7ff232ffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff2340d22b0 0x7ff2340b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:37126/0 (socket says 172.21.15.83:37126) 2024-03-06T07:30:19.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.076+0000 7ff233fff640 1 -- 172.21.15.83:0/4193410486 learned_addr learned my addr 172.21.15.83:0/4193410486 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:19.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.076+0000 7ff233fff640 1 -- 172.21.15.83:0/4193410486 >> v1:172.21.15.183:6789/0 conn(0x7ff23415dfe0 legacy=0x7ff2340cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:19.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.076+0000 7ff233fff640 1 -- 172.21.15.83:0/4193410486 >> v1:172.21.15.83:6789/0 conn(0x7ff2340d22b0 legacy=0x7ff2340b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.076+0000 7ff233fff640 1 -- 172.21.15.83:0/4193410486 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2340b39f0 con 0x7ff2340d2710 2024-03-06T07:30:19.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.077+0000 7ff233fff640 1 --2- 172.21.15.83:0/4193410486 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2340d2710 0x7ff2340b4900 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7ff22c009e80 tx=0x7ff22c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ed0f619f6582b6e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:19.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.077+0000 7ff2327fc640 1 -- 172.21.15.83:0/4193410486 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22c02caf0 con 0x7ff2340d2710 2024-03-06T07:30:19.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.077+0000 7ff2327fc640 1 -- 172.21.15.83:0/4193410486 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff22c02cc90 con 0x7ff2340d2710 2024-03-06T07:30:19.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.077+0000 7ff2327fc640 1 -- 172.21.15.83:0/4193410486 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22c034610 con 0x7ff2340d2710 2024-03-06T07:30:19.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.078+0000 7ff239f3e640 1 -- 172.21.15.83:0/4193410486 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2340d2710 msgr2=0x7ff2340b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.078+0000 7ff239f3e640 1 --2- 172.21.15.83:0/4193410486 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2340d2710 0x7ff2340b4900 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7ff22c009e80 tx=0x7ff22c02bf50 comp rx=0 tx=0).stop 2024-03-06T07:30:19.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.078+0000 7ff239f3e640 1 -- 172.21.15.83:0/4193410486 shutdown_connections 2024-03-06T07:30:19.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.078+0000 7ff239f3e640 1 --2- 172.21.15.83:0/4193410486 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2340d2710 0x7ff2340b4900 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.078+0000 7ff239f3e640 1 -- 172.21.15.83:0/4193410486 >> 172.21.15.83:0/4193410486 conn(0x7ff2340d3cd0 msgr2=0x7ff2340d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:19.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.078+0000 7ff239f3e640 1 -- 172.21.15.83:0/4193410486 shutdown_connections 2024-03-06T07:30:19.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.078+0000 7ff239f3e640 1 -- 172.21.15.83:0/4193410486 wait complete. 2024-03-06T07:30:19.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.079+0000 7ff239f3e640 1 Processor -- start 2024-03-06T07:30:19.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.079+0000 7ff239f3e640 1 -- start start 2024-03-06T07:30:19.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.080+0000 7ff239f3e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2340d22b0 0x7ff23414a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.080+0000 7ff239f3e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff2340d2710 0x7ff2340cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.080+0000 7ff239f3e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff23415dfe0 0x7ff2340b9300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.080+0000 7ff239f3e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2340cfa00 con 0x7ff2340d22b0 2024-03-06T07:30:19.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.080+0000 7ff239f3e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2340b3d60 con 0x7ff23415dfe0 2024-03-06T07:30:19.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.080+0000 7ff239f3e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2340d1c70 con 0x7ff2340d2710 2024-03-06T07:30:19.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.080+0000 7ff2337fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2340d22b0 0x7ff23414a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.080+0000 7ff2337fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2340d22b0 0x7ff23414a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50420/0 (socket says 172.21.15.83:50420) 2024-03-06T07:30:19.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.080+0000 7ff2337fe640 1 -- 172.21.15.83:0/3870920673 learned_addr learned my addr 172.21.15.83:0/3870920673 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:19.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.080+0000 7ff232ffd640 1 --2- 172.21.15.83:0/3870920673 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff2340d2710 0x7ff2340cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.080+0000 7ff233fff640 1 --2- 172.21.15.83:0/3870920673 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff23415dfe0 0x7ff2340b9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.080+0000 7ff2337fe640 1 -- 172.21.15.83:0/3870920673 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff2340d2710 msgr2=0x7ff2340cbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.080+0000 7ff2337fe640 1 --2- 172.21.15.83:0/3870920673 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff2340d2710 0x7ff2340cbdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.080+0000 7ff2337fe640 1 -- 172.21.15.83:0/3870920673 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff23415dfe0 msgr2=0x7ff2340b9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.080+0000 7ff2337fe640 1 --2- 172.21.15.83:0/3870920673 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff23415dfe0 0x7ff2340b9300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.080+0000 7ff2337fe640 1 -- 172.21.15.83:0/3870920673 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2340b8880 con 0x7ff2340d22b0 2024-03-06T07:30:19.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.080+0000 7ff2337fe640 1 --2- 172.21.15.83:0/3870920673 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2340d22b0 0x7ff23414a2a0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7ff22400b3f0 tx=0x7ff22400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:19.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.080+0000 7ff21bfff640 1 -- 172.21.15.83:0/3870920673 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff224015020 con 0x7ff2340d22b0 2024-03-06T07:30:19.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.080+0000 7ff21bfff640 1 -- 172.21.15.83:0/3870920673 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff224011830 con 0x7ff2340d22b0 2024-03-06T07:30:19.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.080+0000 7ff239f3e640 1 -- 172.21.15.83:0/3870920673 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff23414ac80 con 0x7ff2340d22b0 2024-03-06T07:30:19.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.080+0000 7ff21bfff640 1 -- 172.21.15.83:0/3870920673 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff224011c20 con 0x7ff2340d22b0 2024-03-06T07:30:19.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.080+0000 7ff239f3e640 1 -- 172.21.15.83:0/3870920673 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2340b8b00 con 0x7ff2340d22b0 2024-03-06T07:30:19.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.082+0000 7ff239f3e640 1 -- 172.21.15.83:0/3870920673 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff23404f330 con 0x7ff2340d22b0 2024-03-06T07:30:19.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.082+0000 7ff21bfff640 1 -- 172.21.15.83:0/3870920673 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff224018020 con 0x7ff2340d22b0 2024-03-06T07:30:19.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.082+0000 7ff21bfff640 1 --2- 172.21.15.83:0/3870920673 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff22006b4d0 0x7ff22006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.083+0000 7ff232ffd640 1 --2- 172.21.15.83:0/3870920673 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff22006b4d0 0x7ff22006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.083+0000 7ff21bfff640 1 -- 172.21.15.83:0/3870920673 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff224089e60 con 0x7ff2340d22b0 2024-03-06T07:30:19.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.083+0000 7ff232ffd640 1 --2- 172.21.15.83:0/3870920673 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff22006b4d0 0x7ff22006d990 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7ff23414b3a0 tx=0x7ff228007cc0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:19.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.087+0000 7ff21bfff640 1 -- 172.21.15.83:0/3870920673 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff22401f030 con 0x7ff2340d22b0 2024-03-06T07:30:19.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.377+0000 7ff239f3e640 1 -- 172.21.15.83:0/3870920673 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2340ce840 con 0x7ff2340d22b0 2024-03-06T07:30:19.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.378+0000 7ff21bfff640 1 -- 172.21.15.83:0/3870920673 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7ff2340b8880 con 0x7ff2340d22b0 2024-03-06T07:30:19.380 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:30:19.380 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:30:19.380 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:30:19.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.380+0000 7ff239f3e640 1 -- 172.21.15.83:0/3870920673 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff22006b4d0 msgr2=0x7ff22006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.380+0000 7ff239f3e640 1 --2- 172.21.15.83:0/3870920673 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff22006b4d0 0x7ff22006d990 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7ff23414b3a0 tx=0x7ff228007cc0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.380+0000 7ff239f3e640 1 -- 172.21.15.83:0/3870920673 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2340d22b0 msgr2=0x7ff23414a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.380+0000 7ff239f3e640 1 --2- 172.21.15.83:0/3870920673 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2340d22b0 0x7ff23414a2a0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7ff22400b3f0 tx=0x7ff22400b8b0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.380+0000 7ff239f3e640 1 -- 172.21.15.83:0/3870920673 shutdown_connections 2024-03-06T07:30:19.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.380+0000 7ff239f3e640 1 --2- 172.21.15.83:0/3870920673 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff22006b4d0 0x7ff22006d990 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.380+0000 7ff239f3e640 1 --2- 172.21.15.83:0/3870920673 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff23415dfe0 0x7ff2340b9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.380+0000 7ff239f3e640 1 --2- 172.21.15.83:0/3870920673 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff2340d2710 0x7ff2340cbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.380+0000 7ff239f3e640 1 --2- 172.21.15.83:0/3870920673 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2340d22b0 0x7ff23414a2a0 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.380+0000 7ff239f3e640 1 -- 172.21.15.83:0/3870920673 >> 172.21.15.83:0/3870920673 conn(0x7ff2340d3cd0 msgr2=0x7ff2340cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:19.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.380+0000 7ff239f3e640 1 -- 172.21.15.83:0/3870920673 shutdown_connections 2024-03-06T07:30:19.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:19.380+0000 7ff239f3e640 1 -- 172.21.15.83:0/3870920673 wait complete. 2024-03-06T07:30:19.594 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:30:19.594 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:30:20.595 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:20.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.694+0000 7f93e819a640 1 Processor -- start 2024-03-06T07:30:20.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.694+0000 7f93e819a640 1 -- start start 2024-03-06T07:30:20.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.695+0000 7f93e819a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f93e015dfe0 0x7f93e00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.695+0000 7f93e819a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f93e00cf830 0x7f93e00d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.695+0000 7f93e819a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f93e00d27f0 0x7f93e00d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.695+0000 7f93e819a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93e00d2c50 con 0x7f93e00cf830 2024-03-06T07:30:20.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.695+0000 7f93e819a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93e00b31e0 con 0x7f93e00d27f0 2024-03-06T07:30:20.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.695+0000 7f93e819a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93e00b3d60 con 0x7f93e015dfe0 2024-03-06T07:30:20.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.696+0000 7f93e5735640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f93e00cf830 0x7f93e00d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.696+0000 7f93e5735640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f93e00cf830 0x7f93e00d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50434/0 (socket says 172.21.15.83:50434) 2024-03-06T07:30:20.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.696+0000 7f93e5735640 1 -- 172.21.15.83:0/2157127332 learned_addr learned my addr 172.21.15.83:0/2157127332 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:20.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.696+0000 7f93e5f36640 1 --2- 172.21.15.83:0/2157127332 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f93e015dfe0 0x7f93e00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.696+0000 7f93e6737640 1 --2- 172.21.15.83:0/2157127332 >> v2:172.21.15.183:3300/0 conn(0x7f93e00d27f0 0x7f93e00d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.696+0000 7f93e5735640 1 -- 172.21.15.83:0/2157127332 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f93e015dfe0 msgr2=0x7f93e00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:20.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.696+0000 7f93e5735640 1 --2- 172.21.15.83:0/2157127332 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f93e015dfe0 0x7f93e00cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.696+0000 7f93e5735640 1 -- 172.21.15.83:0/2157127332 >> v2:172.21.15.183:3300/0 conn(0x7f93e00d27f0 msgr2=0x7f93e00d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:20.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.696+0000 7f93e5735640 1 --2- 172.21.15.83:0/2157127332 >> v2:172.21.15.183:3300/0 conn(0x7f93e00d27f0 0x7f93e00d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.696+0000 7f93e5735640 1 -- 172.21.15.83:0/2157127332 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93e00b3430 con 0x7f93e00cf830 2024-03-06T07:30:20.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.697+0000 7f93e5735640 1 --2- 172.21.15.83:0/2157127332 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93e00cf830 0x7f93e00d22b0 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f93e0067ad0 tx=0x7f93d002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25f69147259dfdc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:20.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.697+0000 7f93e4f34640 1 -- 172.21.15.83:0/2157127332 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d0035020 con 0x7f93e00cf830 2024-03-06T07:30:20.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.697+0000 7f93e4f34640 1 -- 172.21.15.83:0/2157127332 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93d0031870 con 0x7f93e00cf830 2024-03-06T07:30:20.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.697+0000 7f93e4f34640 1 -- 172.21.15.83:0/2157127332 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d0031c40 con 0x7f93e00cf830 2024-03-06T07:30:20.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.697+0000 7f93e819a640 1 -- 172.21.15.83:0/2157127332 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93e00cf830 msgr2=0x7f93e00d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.697+0000 7f93e819a640 1 --2- 172.21.15.83:0/2157127332 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93e00cf830 0x7f93e00d22b0 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f93e0067ad0 tx=0x7f93d002b910 comp rx=0 tx=0).stop 2024-03-06T07:30:20.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.697+0000 7f93e819a640 1 -- 172.21.15.83:0/2157127332 shutdown_connections 2024-03-06T07:30:20.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.697+0000 7f93e819a640 1 --2- 172.21.15.83:0/2157127332 >> v2:172.21.15.183:3300/0 conn(0x7f93e00d27f0 0x7f93e00d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.697+0000 7f93e819a640 1 --2- 172.21.15.83:0/2157127332 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93e00cf830 0x7f93e00d22b0 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.697+0000 7f93e819a640 1 --2- 172.21.15.83:0/2157127332 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f93e015dfe0 0x7f93e00cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.697+0000 7f93e819a640 1 -- 172.21.15.83:0/2157127332 >> 172.21.15.83:0/2157127332 conn(0x7f93e00d3cd0 msgr2=0x7f93e00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:20.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.697+0000 7f93e819a640 1 -- 172.21.15.83:0/2157127332 shutdown_connections 2024-03-06T07:30:20.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.697+0000 7f93e819a640 1 -- 172.21.15.83:0/2157127332 wait complete. 2024-03-06T07:30:20.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.698+0000 7f93e819a640 1 Processor -- start 2024-03-06T07:30:20.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.699+0000 7f93e819a640 1 -- start start 2024-03-06T07:30:20.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.699+0000 7f93e819a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f93e00cf830 0x7f93e00b1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.699+0000 7f93e819a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f93e00d27f0 0x7f93e0160620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.699+0000 7f93e819a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93e015dfe0 0x7f93e0160b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.699+0000 7f93e819a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93e00d1c70 con 0x7f93e015dfe0 2024-03-06T07:30:20.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.699+0000 7f93e819a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93e00b39f0 con 0x7f93e00cf830 2024-03-06T07:30:20.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.699+0000 7f93e819a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93e00d1df0 con 0x7f93e00d27f0 2024-03-06T07:30:20.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.699+0000 7f93e6737640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93e015dfe0 0x7f93e0160b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.699+0000 7f93e6737640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93e015dfe0 0x7f93e0160b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50442/0 (socket says 172.21.15.83:50442) 2024-03-06T07:30:20.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.699+0000 7f93e6737640 1 -- 172.21.15.83:0/285825569 learned_addr learned my addr 172.21.15.83:0/285825569 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:20.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.699+0000 7f93e5f36640 1 --2- 172.21.15.83:0/285825569 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f93e00cf830 0x7f93e00b1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.699+0000 7f93e5735640 1 --2- 172.21.15.83:0/285825569 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f93e00d27f0 0x7f93e0160620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.700+0000 7f93e6737640 1 -- 172.21.15.83:0/285825569 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f93e00d27f0 msgr2=0x7f93e0160620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.700+0000 7f93e6737640 1 --2- 172.21.15.83:0/285825569 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f93e00d27f0 0x7f93e0160620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.700+0000 7f93e6737640 1 -- 172.21.15.83:0/285825569 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f93e00cf830 msgr2=0x7f93e00b1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.700+0000 7f93e6737640 1 --2- 172.21.15.83:0/285825569 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f93e00cf830 0x7f93e00b1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.700+0000 7f93e6737640 1 -- 172.21.15.83:0/285825569 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93e014de50 con 0x7f93e015dfe0 2024-03-06T07:30:20.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.700+0000 7f93e6737640 1 --2- 172.21.15.83:0/285825569 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93e015dfe0 0x7f93e0160b60 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f93d8007e90 tx=0x7f93d800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:20.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.700+0000 7f93d67fc640 1 -- 172.21.15.83:0/285825569 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d8014420 con 0x7f93e015dfe0 2024-03-06T07:30:20.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.700+0000 7f93d67fc640 1 -- 172.21.15.83:0/285825569 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93d8019070 con 0x7f93e015dfe0 2024-03-06T07:30:20.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.700+0000 7f93e819a640 1 -- 172.21.15.83:0/285825569 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93e00cc480 con 0x7f93e015dfe0 2024-03-06T07:30:20.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.700+0000 7f93d67fc640 1 -- 172.21.15.83:0/285825569 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d8014820 con 0x7f93e015dfe0 2024-03-06T07:30:20.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.700+0000 7f93e819a640 1 -- 172.21.15.83:0/285825569 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93e00cca40 con 0x7f93e015dfe0 2024-03-06T07:30:20.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.701+0000 7f93e819a640 1 -- 172.21.15.83:0/285825569 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93e004f330 con 0x7f93e015dfe0 2024-03-06T07:30:20.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.701+0000 7f93d67fc640 1 -- 172.21.15.83:0/285825569 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f93d80149c0 con 0x7f93e015dfe0 2024-03-06T07:30:20.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.702+0000 7f93d67fc640 1 --2- 172.21.15.83:0/285825569 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f93c406b4d0 0x7f93c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.702+0000 7f93e5f36640 1 --2- 172.21.15.83:0/285825569 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f93c406b4d0 0x7f93c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.702+0000 7f93d67fc640 1 -- 172.21.15.83:0/285825569 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f93d808a8f0 con 0x7f93e015dfe0 2024-03-06T07:30:20.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.702+0000 7f93e5f36640 1 --2- 172.21.15.83:0/285825569 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f93c406b4d0 0x7f93c406d990 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f93dc0079a0 tx=0x7f93dc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:20.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.710+0000 7f93d67fc640 1 -- 172.21.15.83:0/285825569 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f93d8059170 con 0x7f93e015dfe0 2024-03-06T07:30:20.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.965+0000 7f93e819a640 1 -- 172.21.15.83:0/285825569 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93e00d0ab0 con 0x7f93e015dfe0 2024-03-06T07:30:20.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.965+0000 7f93d67fc640 1 -- 172.21.15.83:0/285825569 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f93d805d210 con 0x7f93e015dfe0 2024-03-06T07:30:20.967 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:30:20.967 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:30:20.968 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:30:20.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.967+0000 7f93e819a640 1 -- 172.21.15.83:0/285825569 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f93c406b4d0 msgr2=0x7f93c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.967+0000 7f93e819a640 1 --2- 172.21.15.83:0/285825569 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f93c406b4d0 0x7f93c406d990 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f93dc0079a0 tx=0x7f93dc008040 comp rx=0 tx=0).stop 2024-03-06T07:30:20.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.967+0000 7f93e819a640 1 -- 172.21.15.83:0/285825569 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93e015dfe0 msgr2=0x7f93e0160b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.967+0000 7f93e819a640 1 --2- 172.21.15.83:0/285825569 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93e015dfe0 0x7f93e0160b60 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f93d8007e90 tx=0x7f93d800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.967+0000 7f93e819a640 1 -- 172.21.15.83:0/285825569 shutdown_connections 2024-03-06T07:30:20.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.967+0000 7f93e819a640 1 --2- 172.21.15.83:0/285825569 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f93c406b4d0 0x7f93c406d990 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.967+0000 7f93e819a640 1 --2- 172.21.15.83:0/285825569 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93e015dfe0 0x7f93e0160b60 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.967+0000 7f93e819a640 1 --2- 172.21.15.83:0/285825569 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f93e00d27f0 0x7f93e0160620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.967+0000 7f93e819a640 1 --2- 172.21.15.83:0/285825569 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f93e00cf830 0x7f93e00b1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.967+0000 7f93e819a640 1 -- 172.21.15.83:0/285825569 >> 172.21.15.83:0/285825569 conn(0x7f93e00d3cd0 msgr2=0x7f93e006bbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:20.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.968+0000 7f93e819a640 1 -- 172.21.15.83:0/285825569 shutdown_connections 2024-03-06T07:30:20.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:20.968+0000 7f93e819a640 1 -- 172.21.15.83:0/285825569 wait complete. 2024-03-06T07:30:21.023 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:21.183 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:30:21.183 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:30:22.184 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:22.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.283+0000 7f4ccbe9a640 1 Processor -- start 2024-03-06T07:30:22.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.283+0000 7f4ccbe9a640 1 -- start start 2024-03-06T07:30:22.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.284+0000 7f4ccbe9a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4cc415dfe0 0x7f4cc40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:22.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.284+0000 7f4ccbe9a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4cc40d29c0 0x7f4cc40b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:22.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.284+0000 7f4ccbe9a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cc40b3430 con 0x7f4cc40cf830 2024-03-06T07:30:22.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.284+0000 7f4ccbe9a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cc40b31e0 con 0x7f4cc40d29c0 2024-03-06T07:30:22.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.284+0000 7f4ccbe9a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc40b3d60 con 0x7f4cc415dfe0 2024-03-06T07:30:22.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.284+0000 7f4cca437640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4cc40d29c0 0x7f4cc40b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:22.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.284+0000 7f4cca437640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4cc40d29c0 0x7f4cc40b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50450/0 (socket says 172.21.15.83:50450) 2024-03-06T07:30:22.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.284+0000 7f4cca437640 1 -- 172.21.15.83:0/3361237519 learned_addr learned my addr 172.21.15.83:0/3361237519 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:22.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.285+0000 7f4cc9c36640 1 --2- 172.21.15.83:0/3361237519 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4cc415dfe0 0x7f4cc40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:22.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.285+0000 7f4cca437640 1 -- 172.21.15.83:0/3361237519 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4cc415dfe0 msgr2=0x7f4cc40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:22.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.285+0000 7f4cca437640 1 --2- 172.21.15.83:0/3361237519 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4cc415dfe0 0x7f4cc40cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.285+0000 7f4cca437640 1 -- 172.21.15.83:0/3361237519 >> v1:172.21.15.183:6789/0 conn(0x7f4cc40cf830 legacy=0x7f4cc40d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:22.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.285+0000 7f4cca437640 1 -- 172.21.15.83:0/3361237519 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc40b3710 con 0x7f4cc40d29c0 2024-03-06T07:30:22.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.285+0000 7f4cca437640 1 --2- 172.21.15.83:0/3361237519 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cc40d29c0 0x7f4cc40b41f0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f4cc0009870 tx=0x7f4cc002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c453b50395ecc6d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:22.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.285+0000 7f4cc8c34640 1 -- 172.21.15.83:0/3361237519 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc0035020 con 0x7f4cc40d29c0 2024-03-06T07:30:22.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.285+0000 7f4cc8c34640 1 -- 172.21.15.83:0/3361237519 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cc0031640 con 0x7f4cc40d29c0 2024-03-06T07:30:22.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.285+0000 7f4cc8c34640 1 -- 172.21.15.83:0/3361237519 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc002d040 con 0x7f4cc40d29c0 2024-03-06T07:30:22.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.286+0000 7f4ccbe9a640 1 -- 172.21.15.83:0/3361237519 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cc40d29c0 msgr2=0x7f4cc40b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.286+0000 7f4ccbe9a640 1 --2- 172.21.15.83:0/3361237519 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cc40d29c0 0x7f4cc40b41f0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f4cc0009870 tx=0x7f4cc002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.286+0000 7f4ccbe9a640 1 -- 172.21.15.83:0/3361237519 shutdown_connections 2024-03-06T07:30:22.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.286+0000 7f4ccbe9a640 1 --2- 172.21.15.83:0/3361237519 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cc40d29c0 0x7f4cc40b41f0 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.286+0000 7f4ccbe9a640 1 --2- 172.21.15.83:0/3361237519 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4cc415dfe0 0x7f4cc40cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.286+0000 7f4ccbe9a640 1 -- 172.21.15.83:0/3361237519 >> 172.21.15.83:0/3361237519 conn(0x7f4cc40d3cd0 msgr2=0x7f4cc40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:22.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.286+0000 7f4ccbe9a640 1 -- 172.21.15.83:0/3361237519 shutdown_connections 2024-03-06T07:30:22.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.286+0000 7f4ccbe9a640 1 -- 172.21.15.83:0/3361237519 wait complete. 2024-03-06T07:30:22.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.287+0000 7f4ccbe9a640 1 Processor -- start 2024-03-06T07:30:22.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.287+0000 7f4ccbe9a640 1 -- start start 2024-03-06T07:30:22.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.288+0000 7f4ccbe9a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4cc40cf830 0x7f4cc40b9730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:22.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.288+0000 7f4ccbe9a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cc40d29c0 0x7f4cc414eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:22.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.288+0000 7f4ccbe9a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4cc415dfe0 0x7f4cc40b1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:22.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.288+0000 7f4ccbe9a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc40d1df0 con 0x7f4cc40d29c0 2024-03-06T07:30:22.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.288+0000 7f4ccbe9a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc40d1c70 con 0x7f4cc40cf830 2024-03-06T07:30:22.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.288+0000 7f4ccbe9a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc415e3c0 con 0x7f4cc415dfe0 2024-03-06T07:30:22.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.288+0000 7f4cc9435640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cc40d29c0 0x7f4cc414eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:22.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.288+0000 7f4cc9435640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cc40d29c0 0x7f4cc414eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50466/0 (socket says 172.21.15.83:50466) 2024-03-06T07:30:22.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.288+0000 7f4cc9435640 1 -- 172.21.15.83:0/4059939825 learned_addr learned my addr 172.21.15.83:0/4059939825 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:22.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.288+0000 7f4cc9435640 1 -- 172.21.15.83:0/4059939825 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4cc415dfe0 msgr2=0x7f4cc40b1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.288+0000 7f4cc9435640 1 --2- 172.21.15.83:0/4059939825 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4cc415dfe0 0x7f4cc40b1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.288+0000 7f4cc9435640 1 -- 172.21.15.83:0/4059939825 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4cc40cf830 msgr2=0x7f4cc40b9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.288+0000 7f4cc9435640 1 --2- 172.21.15.83:0/4059939825 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4cc40cf830 0x7f4cc40b9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.288+0000 7f4cc9435640 1 -- 172.21.15.83:0/4059939825 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc40b8690 con 0x7f4cc40d29c0 2024-03-06T07:30:22.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.288+0000 7f4cc9435640 1 --2- 172.21.15.83:0/4059939825 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cc40d29c0 0x7f4cc414eff0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f4cb800c400 tx=0x7f4cb800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:22.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.289+0000 7f4cb67fc640 1 -- 172.21.15.83:0/4059939825 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb8011790 con 0x7f4cc40d29c0 2024-03-06T07:30:22.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.289+0000 7f4cb67fc640 1 -- 172.21.15.83:0/4059939825 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cb8017070 con 0x7f4cc40d29c0 2024-03-06T07:30:22.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.289+0000 7f4ccbe9a640 1 -- 172.21.15.83:0/4059939825 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cc40b8970 con 0x7f4cc40d29c0 2024-03-06T07:30:22.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.289+0000 7f4cb67fc640 1 -- 172.21.15.83:0/4059939825 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb8011bb0 con 0x7f4cc40d29c0 2024-03-06T07:30:22.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.289+0000 7f4ccbe9a640 1 -- 172.21.15.83:0/4059939825 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cc40b8f30 con 0x7f4cc40d29c0 2024-03-06T07:30:22.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.290+0000 7f4ccbe9a640 1 -- 172.21.15.83:0/4059939825 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cc40ce7e0 con 0x7f4cc40d29c0 2024-03-06T07:30:22.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.290+0000 7f4cb67fc640 1 -- 172.21.15.83:0/4059939825 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4cb801c460 con 0x7f4cc40d29c0 2024-03-06T07:30:22.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.290+0000 7f4cb67fc640 1 --2- 172.21.15.83:0/4059939825 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4ca806b4d0 0x7f4ca806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:22.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.291+0000 7f4cb67fc640 1 -- 172.21.15.83:0/4059939825 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4cb808a090 con 0x7f4cc40d29c0 2024-03-06T07:30:22.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.291+0000 7f4cc9c36640 1 --2- 172.21.15.83:0/4059939825 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4ca806b4d0 0x7f4ca806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:22.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.291+0000 7f4cc9c36640 1 --2- 172.21.15.83:0/4059939825 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4ca806b4d0 0x7f4ca806d990 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc007dc0 tx=0x7f4cbc0072f0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:22.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.299+0000 7f4cb67fc640 1 -- 172.21.15.83:0/4059939825 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4cc40ce7e0 con 0x7f4cc40d29c0 2024-03-06T07:30:22.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.515+0000 7f4ccbe9a640 1 -- 172.21.15.83:0/4059939825 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4cc40b4d30 con 0x7f4cc40d29c0 2024-03-06T07:30:22.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.515+0000 7f4cb67fc640 1 -- 172.21.15.83:0/4059939825 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f4cb8058ed0 con 0x7f4cc40d29c0 2024-03-06T07:30:22.518 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:30:22.519 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:30:22.519 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:30:22.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.517+0000 7f4ccbe9a640 1 -- 172.21.15.83:0/4059939825 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4ca806b4d0 msgr2=0x7f4ca806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.517+0000 7f4ccbe9a640 1 --2- 172.21.15.83:0/4059939825 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4ca806b4d0 0x7f4ca806d990 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc007dc0 tx=0x7f4cbc0072f0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.517+0000 7f4ccbe9a640 1 -- 172.21.15.83:0/4059939825 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cc40d29c0 msgr2=0x7f4cc414eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.517+0000 7f4ccbe9a640 1 --2- 172.21.15.83:0/4059939825 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cc40d29c0 0x7f4cc414eff0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f4cb800c400 tx=0x7f4cb800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.517+0000 7f4ccbe9a640 1 -- 172.21.15.83:0/4059939825 shutdown_connections 2024-03-06T07:30:22.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.517+0000 7f4ccbe9a640 1 --2- 172.21.15.83:0/4059939825 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4ca806b4d0 0x7f4ca806d990 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.517+0000 7f4ccbe9a640 1 --2- 172.21.15.83:0/4059939825 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4cc415dfe0 0x7f4cc40b1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.517+0000 7f4ccbe9a640 1 --2- 172.21.15.83:0/4059939825 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4cc40d29c0 0x7f4cc414eff0 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.517+0000 7f4ccbe9a640 1 --2- 172.21.15.83:0/4059939825 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4cc40cf830 0x7f4cc40b9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.517+0000 7f4ccbe9a640 1 -- 172.21.15.83:0/4059939825 >> 172.21.15.83:0/4059939825 conn(0x7f4cc40d3cd0 msgr2=0x7f4cc406b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:22.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.517+0000 7f4ccbe9a640 1 -- 172.21.15.83:0/4059939825 shutdown_connections 2024-03-06T07:30:22.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:22.517+0000 7f4ccbe9a640 1 -- 172.21.15.83:0/4059939825 wait complete. 2024-03-06T07:30:22.732 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:30:22.732 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:30:23.025 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:23.733 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:23.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.831+0000 7fcafd30c640 1 Processor -- start 2024-03-06T07:30:23.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.831+0000 7fcafd30c640 1 -- start start 2024-03-06T07:30:23.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.833+0000 7fcafd30c640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fcaf815bdd0 0x7fcaf80c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.833+0000 7fcafd30c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcaf815fa10 0x7fcaf80ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.833+0000 7fcafd30c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaf80cebe0 con 0x7fcaf80d16e0 2024-03-06T07:30:23.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.833+0000 7fcafd30c640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcaf80ceec0 con 0x7fcaf815fa10 2024-03-06T07:30:23.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.833+0000 7fcafd30c640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcaf80cf1a0 con 0x7fcaf815bdd0 2024-03-06T07:30:23.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.833+0000 7fcaee7fc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcaf815fa10 0x7fcaf80ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.833+0000 7fcaee7fc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcaf815fa10 0x7fcaf80ce270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50482/0 (socket says 172.21.15.83:50482) 2024-03-06T07:30:23.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.833+0000 7fcaee7fc640 1 -- 172.21.15.83:0/4108558030 learned_addr learned my addr 172.21.15.83:0/4108558030 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:23.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.833+0000 7fcaf6ffd640 1 --2- 172.21.15.83:0/4108558030 >> v2:172.21.15.183:3300/0 conn(0x7fcaf815bdd0 0x7fcaf80c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.833+0000 7fcaee7fc640 1 -- 172.21.15.83:0/4108558030 >> v2:172.21.15.183:3300/0 conn(0x7fcaf815bdd0 msgr2=0x7fcaf80c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:23.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.833+0000 7fcaee7fc640 1 --2- 172.21.15.83:0/4108558030 >> v2:172.21.15.183:3300/0 conn(0x7fcaf815bdd0 0x7fcaf80c5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.833+0000 7fcaee7fc640 1 -- 172.21.15.83:0/4108558030 >> v1:172.21.15.183:6789/0 conn(0x7fcaf80d16e0 legacy=0x7fcaf80d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.833+0000 7fcaee7fc640 1 -- 172.21.15.83:0/4108558030 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf80cc8e0 con 0x7fcaf815fa10 2024-03-06T07:30:23.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.833+0000 7fcaf6ffd640 1 --2- 172.21.15.83:0/4108558030 >> v2:172.21.15.183:3300/0 conn(0x7fcaf815bdd0 0x7fcaf80c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:23.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.834+0000 7fcaee7fc640 1 --2- 172.21.15.83:0/4108558030 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcaf815fa10 0x7fcaf80ce270 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7fcaf806e3c0 tx=0x7fcae002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2a1f65d43e67813 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:23.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.834+0000 7fcaf67fc640 1 -- 172.21.15.83:0/4108558030 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae0035020 con 0x7fcaf815fa10 2024-03-06T07:30:23.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.834+0000 7fcaf67fc640 1 -- 172.21.15.83:0/4108558030 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcae0031870 con 0x7fcaf815fa10 2024-03-06T07:30:23.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.834+0000 7fcaf67fc640 1 -- 172.21.15.83:0/4108558030 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae0031c40 con 0x7fcaf815fa10 2024-03-06T07:30:23.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.835+0000 7fcafd30c640 1 -- 172.21.15.83:0/4108558030 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcaf815fa10 msgr2=0x7fcaf80ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.835+0000 7fcafd30c640 1 --2- 172.21.15.83:0/4108558030 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcaf815fa10 0x7fcaf80ce270 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7fcaf806e3c0 tx=0x7fcae002b910 comp rx=0 tx=0).stop 2024-03-06T07:30:23.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.835+0000 7fcafd30c640 1 -- 172.21.15.83:0/4108558030 shutdown_connections 2024-03-06T07:30:23.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.835+0000 7fcafd30c640 1 --2- 172.21.15.83:0/4108558030 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcaf815fa10 0x7fcaf80ce270 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.835+0000 7fcafd30c640 1 --2- 172.21.15.83:0/4108558030 >> v2:172.21.15.183:3300/0 conn(0x7fcaf815bdd0 0x7fcaf80c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.835+0000 7fcafd30c640 1 -- 172.21.15.83:0/4108558030 >> 172.21.15.83:0/4108558030 conn(0x7fcaf809a9f0 msgr2=0x7fcaf809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:23.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.835+0000 7fcafd30c640 1 -- 172.21.15.83:0/4108558030 shutdown_connections 2024-03-06T07:30:23.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.835+0000 7fcafd30c640 1 -- 172.21.15.83:0/4108558030 wait complete. 2024-03-06T07:30:23.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.837+0000 7fcafd30c640 1 Processor -- start 2024-03-06T07:30:23.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.837+0000 7fcafd30c640 1 -- start start 2024-03-06T07:30:23.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.838+0000 7fcafd30c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcaf80d16e0 0x7fcaf80cb6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.838+0000 7fcafd30c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcaf815bdd0 0x7fcaf80ca610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.838+0000 7fcafd30c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcaf815fa10 0x7fcaf80cab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.838+0000 7fcafd30c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf815c1b0 con 0x7fcaf815bdd0 2024-03-06T07:30:23.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.838+0000 7fcafd30c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf80cc5d0 con 0x7fcaf80d16e0 2024-03-06T07:30:23.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.838+0000 7fcafd30c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf80ce7b0 con 0x7fcaf815fa10 2024-03-06T07:30:23.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.838+0000 7fcaee7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcaf815bdd0 0x7fcaf80ca610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.838+0000 7fcaee7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcaf815bdd0 0x7fcaf80ca610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50490/0 (socket says 172.21.15.83:50490) 2024-03-06T07:30:23.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.838+0000 7fcaee7fc640 1 -- 172.21.15.83:0/1987401285 learned_addr learned my addr 172.21.15.83:0/1987401285 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:23.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.838+0000 7fcaf6ffd640 1 --2- 172.21.15.83:0/1987401285 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcaf80d16e0 0x7fcaf80cb6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.838+0000 7fcaee7fc640 1 -- 172.21.15.83:0/1987401285 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcaf815fa10 msgr2=0x7fcaf80cab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.838+0000 7fcaf77fe640 1 --2- 172.21.15.83:0/1987401285 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcaf815fa10 0x7fcaf80cab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.838+0000 7fcaee7fc640 1 --2- 172.21.15.83:0/1987401285 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcaf815fa10 0x7fcaf80cab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.838+0000 7fcaee7fc640 1 -- 172.21.15.83:0/1987401285 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcaf80d16e0 msgr2=0x7fcaf80cb6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.838+0000 7fcaee7fc640 1 --2- 172.21.15.83:0/1987401285 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcaf80d16e0 0x7fcaf80cb6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.838+0000 7fcaee7fc640 1 -- 172.21.15.83:0/1987401285 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf815c6a0 con 0x7fcaf815bdd0 2024-03-06T07:30:23.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.838+0000 7fcaf6ffd640 1 --2- 172.21.15.83:0/1987401285 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcaf80d16e0 0x7fcaf80cb6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:23.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.838+0000 7fcaf77fe640 1 --2- 172.21.15.83:0/1987401285 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcaf815fa10 0x7fcaf80cab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:23.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.839+0000 7fcaee7fc640 1 --2- 172.21.15.83:0/1987401285 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcaf815bdd0 0x7fcaf80ca610 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7fcae0000c00 tx=0x7fcae00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:23.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.839+0000 7fcaeffff640 1 -- 172.21.15.83:0/1987401285 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae0035040 con 0x7fcaf815bdd0 2024-03-06T07:30:23.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.839+0000 7fcaeffff640 1 -- 172.21.15.83:0/1987401285 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcae0030d00 con 0x7fcaf815bdd0 2024-03-06T07:30:23.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.839+0000 7fcafd30c640 1 -- 172.21.15.83:0/1987401285 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaf815c980 con 0x7fcaf815bdd0 2024-03-06T07:30:23.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.839+0000 7fcaeffff640 1 -- 172.21.15.83:0/1987401285 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae00456a0 con 0x7fcaf815bdd0 2024-03-06T07:30:23.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.839+0000 7fcafd30c640 1 -- 172.21.15.83:0/1987401285 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaf815cf40 con 0x7fcaf815bdd0 2024-03-06T07:30:23.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.840+0000 7fcafd30c640 1 -- 172.21.15.83:0/1987401285 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaf804f330 con 0x7fcaf815bdd0 2024-03-06T07:30:23.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.840+0000 7fcaeffff640 1 -- 172.21.15.83:0/1987401285 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcae0038020 con 0x7fcaf815bdd0 2024-03-06T07:30:23.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.841+0000 7fcaeffff640 1 --2- 172.21.15.83:0/1987401285 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcadc06b4d0 0x7fcadc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.841+0000 7fcaeffff640 1 -- 172.21.15.83:0/1987401285 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fcae00ac570 con 0x7fcaf815bdd0 2024-03-06T07:30:23.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.841+0000 7fcaf6ffd640 1 --2- 172.21.15.83:0/1987401285 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcadc06b4d0 0x7fcadc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.841+0000 7fcaf6ffd640 1 --2- 172.21.15.83:0/1987401285 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcadc06b4d0 0x7fcadc06d990 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fcae8006fd0 tx=0x7fcae8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:23.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:23.845+0000 7fcaeffff640 1 -- 172.21.15.83:0/1987401285 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcae002f320 con 0x7fcaf815bdd0 2024-03-06T07:30:24.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:24.082+0000 7fcafd30c640 1 -- 172.21.15.83:0/1987401285 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcaf80d1ac0 con 0x7fcaf815bdd0 2024-03-06T07:30:24.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:24.082+0000 7fcaeffff640 1 -- 172.21.15.83:0/1987401285 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7fcae007b440 con 0x7fcaf815bdd0 2024-03-06T07:30:24.086 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:30:24.086 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:30:24.086 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:30:24.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:24.086+0000 7fcafd30c640 1 -- 172.21.15.83:0/1987401285 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcadc06b4d0 msgr2=0x7fcadc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:24.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:24.086+0000 7fcafd30c640 1 --2- 172.21.15.83:0/1987401285 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcadc06b4d0 0x7fcadc06d990 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fcae8006fd0 tx=0x7fcae8008040 comp rx=0 tx=0).stop 2024-03-06T07:30:24.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:24.086+0000 7fcafd30c640 1 -- 172.21.15.83:0/1987401285 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcaf815bdd0 msgr2=0x7fcaf80ca610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:24.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:24.086+0000 7fcafd30c640 1 --2- 172.21.15.83:0/1987401285 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcaf815bdd0 0x7fcaf80ca610 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7fcae0000c00 tx=0x7fcae00046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:24.086+0000 7fcafd30c640 1 -- 172.21.15.83:0/1987401285 shutdown_connections 2024-03-06T07:30:24.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:24.086+0000 7fcafd30c640 1 --2- 172.21.15.83:0/1987401285 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcadc06b4d0 0x7fcadc06d990 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:24.086+0000 7fcafd30c640 1 --2- 172.21.15.83:0/1987401285 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcaf815fa10 0x7fcaf80cab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:24.086+0000 7fcafd30c640 1 --2- 172.21.15.83:0/1987401285 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcaf815bdd0 0x7fcaf80ca610 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:24.086+0000 7fcafd30c640 1 --2- 172.21.15.83:0/1987401285 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcaf80d16e0 0x7fcaf80cb6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:24.086+0000 7fcafd30c640 1 -- 172.21.15.83:0/1987401285 >> 172.21.15.83:0/1987401285 conn(0x7fcaf809a9f0 msgr2=0x7fcaf8070390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:24.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:24.086+0000 7fcafd30c640 1 -- 172.21.15.83:0/1987401285 shutdown_connections 2024-03-06T07:30:24.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:24.086+0000 7fcafd30c640 1 -- 172.21.15.83:0/1987401285 wait complete. 2024-03-06T07:30:24.304 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:30:24.304 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:30:25.026 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:25.305 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:25.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.405+0000 7f94f8965640 1 Processor -- start 2024-03-06T07:30:25.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.405+0000 7f94f8965640 1 -- start start 2024-03-06T07:30:25.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.406+0000 7f94f8965640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f94f415dfe0 0x7f94f40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.406+0000 7f94f8965640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f94f40cf830 0x7f94f40d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.406+0000 7f94f8965640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94f40b3430 con 0x7f94f40d27f0 2024-03-06T07:30:25.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.406+0000 7f94f8965640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94f40b31e0 con 0x7f94f40cf830 2024-03-06T07:30:25.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.406+0000 7f94f8965640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94f40b3d60 con 0x7f94f415dfe0 2024-03-06T07:30:25.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.406+0000 7f94ebfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f94f40cf830 0x7f94f40d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.406+0000 7f94f2d9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f94f40d27f0 0x7f94f40d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:37130/0 (socket says 172.21.15.83:37130) 2024-03-06T07:30:25.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.406+0000 7f94f2d9d640 1 -- 172.21.15.83:0/175775930 learned_addr learned my addr 172.21.15.83:0/175775930 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:30:25.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.406+0000 7f94ebfff640 1 -- 172.21.15.83:0/175775930 >> v2:172.21.15.183:3300/0 conn(0x7f94f415dfe0 msgr2=0x7f94f40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:25.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.406+0000 7f94ebfff640 1 --2- 172.21.15.83:0/175775930 >> v2:172.21.15.183:3300/0 conn(0x7f94f415dfe0 0x7f94f40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.406+0000 7f94ebfff640 1 -- 172.21.15.83:0/175775930 >> v1:172.21.15.83:6789/0 conn(0x7f94f40d27f0 legacy=0x7f94f40d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.406+0000 7f94ebfff640 1 -- 172.21.15.83:0/175775930 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f40b3710 con 0x7f94f40cf830 2024-03-06T07:30:25.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.407+0000 7f94ebfff640 1 --2- 172.21.15.83:0/175775930 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94f40cf830 0x7f94f40d22b0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f94dc0097a0 tx=0x7f94dc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e6e2de30d56ca42 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:25.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.407+0000 7f94f1d9b640 1 -- 172.21.15.83:0/175775930 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94dc035020 con 0x7f94f40cf830 2024-03-06T07:30:25.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.407+0000 7f94f1d9b640 1 -- 172.21.15.83:0/175775930 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94dc031830 con 0x7f94f40cf830 2024-03-06T07:30:25.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.407+0000 7f94f1d9b640 1 -- 172.21.15.83:0/175775930 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94dc031c00 con 0x7f94f40cf830 2024-03-06T07:30:25.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.408+0000 7f94f8965640 1 -- 172.21.15.83:0/175775930 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94f40cf830 msgr2=0x7f94f40d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.408+0000 7f94f8965640 1 --2- 172.21.15.83:0/175775930 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94f40cf830 0x7f94f40d22b0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f94dc0097a0 tx=0x7f94dc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.408+0000 7f94f8965640 1 -- 172.21.15.83:0/175775930 shutdown_connections 2024-03-06T07:30:25.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.408+0000 7f94f8965640 1 --2- 172.21.15.83:0/175775930 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94f40cf830 0x7f94f40d22b0 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.408+0000 7f94f8965640 1 --2- 172.21.15.83:0/175775930 >> v2:172.21.15.183:3300/0 conn(0x7f94f415dfe0 0x7f94f40cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.408+0000 7f94f8965640 1 -- 172.21.15.83:0/175775930 >> 172.21.15.83:0/175775930 conn(0x7f94f40d3cd0 msgr2=0x7f94f40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:25.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.408+0000 7f94f8965640 1 -- 172.21.15.83:0/175775930 shutdown_connections 2024-03-06T07:30:25.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.408+0000 7f94f8965640 1 -- 172.21.15.83:0/175775930 wait complete. 2024-03-06T07:30:25.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.410+0000 7f94f8965640 1 Processor -- start 2024-03-06T07:30:25.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.410+0000 7f94f8965640 1 -- start start 2024-03-06T07:30:25.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.411+0000 7f94f8965640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f94f40cf830 0x7f94f4156ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.411+0000 7f94f8965640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f94f40d27f0 0x7f94f4147f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.411+0000 7f94f8965640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94f415dfe0 0x7f94f415a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.411+0000 7f94f8965640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f40d1df0 con 0x7f94f415dfe0 2024-03-06T07:30:25.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.411+0000 7f94f8965640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f40d1c70 con 0x7f94f40cf830 2024-03-06T07:30:25.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.411+0000 7f94f8965640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94f415e3c0 con 0x7f94f40d27f0 2024-03-06T07:30:25.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.411+0000 7f94f2d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94f415dfe0 0x7f94f415a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.411+0000 7f94f2d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94f415dfe0 0x7f94f415a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50514/0 (socket says 172.21.15.83:50514) 2024-03-06T07:30:25.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.411+0000 7f94f2d9d640 1 -- 172.21.15.83:0/3528975132 learned_addr learned my addr 172.21.15.83:0/3528975132 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:25.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.411+0000 7f94f259c640 1 --2- 172.21.15.83:0/3528975132 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f94f40cf830 0x7f94f4156ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.411+0000 7f94ebfff640 1 --2- 172.21.15.83:0/3528975132 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f94f40d27f0 0x7f94f4147f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.411+0000 7f94f2d9d640 1 -- 172.21.15.83:0/3528975132 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f94f40d27f0 msgr2=0x7f94f4147f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.412+0000 7f94f2d9d640 1 --2- 172.21.15.83:0/3528975132 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f94f40d27f0 0x7f94f4147f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.412+0000 7f94f2d9d640 1 -- 172.21.15.83:0/3528975132 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f94f40cf830 msgr2=0x7f94f4156ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.412+0000 7f94f2d9d640 1 --2- 172.21.15.83:0/3528975132 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f94f40cf830 0x7f94f4156ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.412+0000 7f94f2d9d640 1 -- 172.21.15.83:0/3528975132 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f415aeb0 con 0x7f94f415dfe0 2024-03-06T07:30:25.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.412+0000 7f94f259c640 1 --2- 172.21.15.83:0/3528975132 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f94f40cf830 0x7f94f4156ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:25.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.412+0000 7f94ebfff640 1 --2- 172.21.15.83:0/3528975132 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f94f40d27f0 0x7f94f4147f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:25.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.412+0000 7f94f2d9d640 1 --2- 172.21.15.83:0/3528975132 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94f415dfe0 0x7f94f415a340 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f94ec00cbf0 tx=0x7f94ec00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:25.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.412+0000 7f94eaffd640 1 -- 172.21.15.83:0/3528975132 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94ec017070 con 0x7f94f415dfe0 2024-03-06T07:30:25.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.412+0000 7f94eaffd640 1 -- 172.21.15.83:0/3528975132 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94ec00ad40 con 0x7f94f415dfe0 2024-03-06T07:30:25.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.412+0000 7f94eaffd640 1 -- 172.21.15.83:0/3528975132 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94ec012600 con 0x7f94f415dfe0 2024-03-06T07:30:25.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.412+0000 7f94f8965640 1 -- 172.21.15.83:0/3528975132 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94f415f870 con 0x7f94f415dfe0 2024-03-06T07:30:25.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.412+0000 7f94f8965640 1 -- 172.21.15.83:0/3528975132 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94f415fe30 con 0x7f94f415dfe0 2024-03-06T07:30:25.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.413+0000 7f94f8965640 1 -- 172.21.15.83:0/3528975132 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94f404f330 con 0x7f94f415dfe0 2024-03-06T07:30:25.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.413+0000 7f94eaffd640 1 -- 172.21.15.83:0/3528975132 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f94ec01a070 con 0x7f94f415dfe0 2024-03-06T07:30:25.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.414+0000 7f94eaffd640 1 --2- 172.21.15.83:0/3528975132 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f94d406b4d0 0x7f94d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.414+0000 7f94eaffd640 1 -- 172.21.15.83:0/3528975132 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f94ec089e00 con 0x7f94f415dfe0 2024-03-06T07:30:25.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.414+0000 7f94f259c640 1 --2- 172.21.15.83:0/3528975132 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f94d406b4d0 0x7f94d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.414+0000 7f94f259c640 1 --2- 172.21.15.83:0/3528975132 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f94d406b4d0 0x7f94d406d990 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f94f406d2d0 tx=0x7f94e4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:25.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.418+0000 7f94eaffd640 1 -- 172.21.15.83:0/3528975132 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f94ec058680 con 0x7f94f415dfe0 2024-03-06T07:30:25.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.633+0000 7f94f8965640 1 -- 172.21.15.83:0/3528975132 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94f415d600 con 0x7f94f415dfe0 2024-03-06T07:30:25.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.634+0000 7f94eaffd640 1 -- 172.21.15.83:0/3528975132 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f94ec05c720 con 0x7f94f415dfe0 2024-03-06T07:30:25.636 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:30:25.636 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:30:25.636 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:30:25.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.636+0000 7f94f8965640 1 -- 172.21.15.83:0/3528975132 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f94d406b4d0 msgr2=0x7f94d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.636+0000 7f94f8965640 1 --2- 172.21.15.83:0/3528975132 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f94d406b4d0 0x7f94d406d990 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f94f406d2d0 tx=0x7f94e4008040 comp rx=0 tx=0).stop 2024-03-06T07:30:25.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.636+0000 7f94f8965640 1 -- 172.21.15.83:0/3528975132 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94f415dfe0 msgr2=0x7f94f415a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.636+0000 7f94f8965640 1 --2- 172.21.15.83:0/3528975132 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94f415dfe0 0x7f94f415a340 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f94ec00cbf0 tx=0x7f94ec00a430 comp rx=0 tx=0).stop 2024-03-06T07:30:25.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.636+0000 7f94f8965640 1 -- 172.21.15.83:0/3528975132 shutdown_connections 2024-03-06T07:30:25.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.636+0000 7f94f8965640 1 --2- 172.21.15.83:0/3528975132 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f94d406b4d0 0x7f94d406d990 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.636+0000 7f94f8965640 1 --2- 172.21.15.83:0/3528975132 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94f415dfe0 0x7f94f415a340 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.636+0000 7f94f8965640 1 --2- 172.21.15.83:0/3528975132 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f94f40d27f0 0x7f94f4147f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.636+0000 7f94f8965640 1 --2- 172.21.15.83:0/3528975132 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f94f40cf830 0x7f94f4156ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.636+0000 7f94f8965640 1 -- 172.21.15.83:0/3528975132 >> 172.21.15.83:0/3528975132 conn(0x7f94f40d3cd0 msgr2=0x7f94f406bb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:25.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.636+0000 7f94f8965640 1 -- 172.21.15.83:0/3528975132 shutdown_connections 2024-03-06T07:30:25.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:25.636+0000 7f94f8965640 1 -- 172.21.15.83:0/3528975132 wait complete. 2024-03-06T07:30:25.850 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:30:25.850 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:30:26.851 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:26.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.950+0000 7f137671e640 1 Processor -- start 2024-03-06T07:30:26.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.950+0000 7f137671e640 1 -- start start 2024-03-06T07:30:26.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.951+0000 7f137671e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f13700d2710 0x7f13700b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.951+0000 7f137671e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13700b3430 con 0x7f13700d22b0 2024-03-06T07:30:26.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.951+0000 7f137671e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13700b3710 con 0x7f137015dfe0 2024-03-06T07:30:26.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.951+0000 7f137671e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13700b31e0 con 0x7f13700d2710 2024-03-06T07:30:26.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.951+0000 7f1374cbb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f13700d2710 0x7f13700b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.951+0000 7f136f7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f13700d22b0 0x7f13700b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41590/0 (socket says 172.21.15.83:41590) 2024-03-06T07:30:26.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.951+0000 7f136f7fe640 1 -- 172.21.15.83:0/3617606207 learned_addr learned my addr 172.21.15.83:0/3617606207 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:30:26.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.951+0000 7f1374cbb640 1 -- 172.21.15.83:0/3617606207 >> v1:172.21.15.183:6789/0 conn(0x7f137015dfe0 legacy=0x7f13700cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:26.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.951+0000 7f1374cbb640 1 -- 172.21.15.83:0/3617606207 >> v1:172.21.15.83:6789/0 conn(0x7f13700d22b0 legacy=0x7f13700b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.951+0000 7f136effd640 1 -- 172.21.15.83:0/3617606207 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4182811163 0 0) 0x7f13700b3430 con 0x7f13700d22b0 2024-03-06T07:30:26.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.951+0000 7f1374cbb640 1 -- 172.21.15.83:0/3617606207 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13700b39f0 con 0x7f13700d2710 2024-03-06T07:30:26.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.952+0000 7f1374cbb640 1 --2- 172.21.15.83:0/3617606207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13700d2710 0x7f13700b4900 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f1368009dc0 tx=0x7f136802be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=246a6a0f7b093fa2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.952+0000 7f136effd640 1 -- 172.21.15.83:0/3617606207 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1368039070 con 0x7f13700d2710 2024-03-06T07:30:26.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.952+0000 7f136effd640 1 -- 172.21.15.83:0/3617606207 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f136802cc40 con 0x7f13700d2710 2024-03-06T07:30:26.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.952+0000 7f136effd640 1 -- 172.21.15.83:0/3617606207 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1368034440 con 0x7f13700d2710 2024-03-06T07:30:26.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.952+0000 7f137671e640 1 -- 172.21.15.83:0/3617606207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13700d2710 msgr2=0x7f13700b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.952+0000 7f137671e640 1 --2- 172.21.15.83:0/3617606207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13700d2710 0x7f13700b4900 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f1368009dc0 tx=0x7f136802be90 comp rx=0 tx=0).stop 2024-03-06T07:30:26.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.952+0000 7f137671e640 1 -- 172.21.15.83:0/3617606207 shutdown_connections 2024-03-06T07:30:26.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.952+0000 7f137671e640 1 --2- 172.21.15.83:0/3617606207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13700d2710 0x7f13700b4900 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.952+0000 7f137671e640 1 -- 172.21.15.83:0/3617606207 >> 172.21.15.83:0/3617606207 conn(0x7f13700d3cd0 msgr2=0x7f13700d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:26.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.953+0000 7f137671e640 1 -- 172.21.15.83:0/3617606207 shutdown_connections 2024-03-06T07:30:26.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.953+0000 7f137671e640 1 -- 172.21.15.83:0/3617606207 wait complete. 2024-03-06T07:30:26.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.954+0000 7f137671e640 1 Processor -- start 2024-03-06T07:30:26.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.954+0000 7f137671e640 1 -- start start 2024-03-06T07:30:26.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.954+0000 7f137671e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13700d22b0 0x7f13700b8db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.955+0000 7f137671e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f13700d2710 0x7f13701f6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.955+0000 7f137671e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f137015dfe0 0x7f1370159510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.955+0000 7f137671e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13700cfa00 con 0x7f13700d22b0 2024-03-06T07:30:26.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.955+0000 7f137671e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13700b3d60 con 0x7f137015dfe0 2024-03-06T07:30:26.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.955+0000 7f137671e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13700d1c70 con 0x7f13700d2710 2024-03-06T07:30:26.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.955+0000 7f136ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13700d22b0 0x7f13700b8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.955+0000 7f136ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13700d22b0 0x7f13700b8db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59108/0 (socket says 172.21.15.83:59108) 2024-03-06T07:30:26.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.955+0000 7f136ffff640 1 -- 172.21.15.83:0/78410131 learned_addr learned my addr 172.21.15.83:0/78410131 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:26.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.955+0000 7f136f7fe640 1 --2- 172.21.15.83:0/78410131 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f13700d2710 0x7f13701f6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.955+0000 7f1374cbb640 1 --2- 172.21.15.83:0/78410131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f137015dfe0 0x7f1370159510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.955+0000 7f136ffff640 1 -- 172.21.15.83:0/78410131 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f13700d2710 msgr2=0x7f13701f6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.955+0000 7f136ffff640 1 --2- 172.21.15.83:0/78410131 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f13700d2710 0x7f13701f6f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.955+0000 7f136ffff640 1 -- 172.21.15.83:0/78410131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f137015dfe0 msgr2=0x7f1370159510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.955+0000 7f136ffff640 1 --2- 172.21.15.83:0/78410131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f137015dfe0 0x7f1370159510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.955+0000 7f136ffff640 1 -- 172.21.15.83:0/78410131 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13700b68d0 con 0x7f13700d22b0 2024-03-06T07:30:26.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.955+0000 7f136f7fe640 1 --2- 172.21.15.83:0/78410131 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f13700d2710 0x7f13701f6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:26.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.955+0000 7f1374cbb640 1 --2- 172.21.15.83:0/78410131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f137015dfe0 0x7f1370159510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:26.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.955+0000 7f136ffff640 1 --2- 172.21.15.83:0/78410131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13700d22b0 0x7f13700b8db0 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f136000b3f0 tx=0x7f136000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.955+0000 7f136cff9640 1 -- 172.21.15.83:0/78410131 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1360015020 con 0x7f13700d22b0 2024-03-06T07:30:26.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.955+0000 7f136cff9640 1 -- 172.21.15.83:0/78410131 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1360011830 con 0x7f13700d22b0 2024-03-06T07:30:26.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.955+0000 7f136cff9640 1 -- 172.21.15.83:0/78410131 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1360011c20 con 0x7f13700d22b0 2024-03-06T07:30:26.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.955+0000 7f137671e640 1 -- 172.21.15.83:0/78410131 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13701f7a90 con 0x7f13700d22b0 2024-03-06T07:30:26.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.955+0000 7f137671e640 1 -- 172.21.15.83:0/78410131 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13700b6e50 con 0x7f13700d22b0 2024-03-06T07:30:26.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.956+0000 7f137671e640 1 -- 172.21.15.83:0/78410131 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13700b7050 con 0x7f13700d22b0 2024-03-06T07:30:26.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.957+0000 7f136cff9640 1 -- 172.21.15.83:0/78410131 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1360018020 con 0x7f13700d22b0 2024-03-06T07:30:26.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.957+0000 7f136cff9640 1 --2- 172.21.15.83:0/78410131 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f135c06b4d0 0x7f135c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.957+0000 7f136cff9640 1 -- 172.21.15.83:0/78410131 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1360089e60 con 0x7f13700d22b0 2024-03-06T07:30:26.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.957+0000 7f136f7fe640 1 --2- 172.21.15.83:0/78410131 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f135c06b4d0 0x7f135c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.958+0000 7f136f7fe640 1 --2- 172.21.15.83:0/78410131 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f135c06b4d0 0x7f135c06d990 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f137015c9e0 tx=0x7f1364009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:26.961+0000 7f136cff9640 1 -- 172.21.15.83:0/78410131 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f13700b7050 con 0x7f13700d22b0 2024-03-06T07:30:27.028 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:27.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:27.172+0000 7f137671e640 1 -- 172.21.15.83:0/78410131 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13700cec80 con 0x7f13700d22b0 2024-03-06T07:30:27.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:27.172+0000 7f136cff9640 1 -- 172.21.15.83:0/78410131 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f136001f030 con 0x7f13700d22b0 2024-03-06T07:30:27.174 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:30:27.174 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:30:27.174 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:30:27.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:27.174+0000 7f137671e640 1 -- 172.21.15.83:0/78410131 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f135c06b4d0 msgr2=0x7f135c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:27.174+0000 7f137671e640 1 --2- 172.21.15.83:0/78410131 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f135c06b4d0 0x7f135c06d990 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f137015c9e0 tx=0x7f1364009040 comp rx=0 tx=0).stop 2024-03-06T07:30:27.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:27.174+0000 7f137671e640 1 -- 172.21.15.83:0/78410131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13700d22b0 msgr2=0x7f13700b8db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:27.174+0000 7f137671e640 1 --2- 172.21.15.83:0/78410131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13700d22b0 0x7f13700b8db0 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f136000b3f0 tx=0x7f136000b8b0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:27.174+0000 7f137671e640 1 -- 172.21.15.83:0/78410131 shutdown_connections 2024-03-06T07:30:27.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:27.174+0000 7f137671e640 1 --2- 172.21.15.83:0/78410131 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f135c06b4d0 0x7f135c06d990 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:27.174+0000 7f137671e640 1 --2- 172.21.15.83:0/78410131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f137015dfe0 0x7f1370159510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:27.174+0000 7f137671e640 1 --2- 172.21.15.83:0/78410131 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f13700d2710 0x7f13701f6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:27.174+0000 7f137671e640 1 --2- 172.21.15.83:0/78410131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13700d22b0 0x7f13700b8db0 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:27.174+0000 7f137671e640 1 -- 172.21.15.83:0/78410131 >> 172.21.15.83:0/78410131 conn(0x7f13700d3cd0 msgr2=0x7f13700cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:27.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:27.174+0000 7f137671e640 1 -- 172.21.15.83:0/78410131 shutdown_connections 2024-03-06T07:30:27.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:27.174+0000 7f137671e640 1 -- 172.21.15.83:0/78410131 wait complete. 2024-03-06T07:30:27.388 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:30:27.388 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:30:28.389 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:28.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.490+0000 7f53bebee640 1 Processor -- start 2024-03-06T07:30:28.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.490+0000 7f53bebee640 1 -- start start 2024-03-06T07:30:28.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.491+0000 7f53bebee640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f53b80d22b0 0x7f53b80d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.491+0000 7f53bebee640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f53b80b41f0 0x7f53b80b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.491+0000 7f53bebee640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b80b3430 con 0x7f53b815dfe0 2024-03-06T07:30:28.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.491+0000 7f53bebee640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53b80d2c50 con 0x7f53b80b41f0 2024-03-06T07:30:28.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.491+0000 7f53bebee640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53b80b31e0 con 0x7f53b80d22b0 2024-03-06T07:30:28.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.491+0000 7f53bd18b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f53b80b41f0 0x7f53b80b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.491+0000 7f53bc98a640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f53b815dfe0 0x7f53b80cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41598/0 (socket says 172.21.15.83:41598) 2024-03-06T07:30:28.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.491+0000 7f53bc98a640 1 -- 172.21.15.83:0/2308520012 learned_addr learned my addr 172.21.15.83:0/2308520012 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:30:28.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.491+0000 7f53bd18b640 1 -- 172.21.15.83:0/2308520012 >> v2:172.21.15.183:3300/0 conn(0x7f53b80d22b0 msgr2=0x7f53b80d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:28.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.491+0000 7f53b7fff640 1 --2- 172.21.15.83:0/2308520012 >> v2:172.21.15.183:3300/0 conn(0x7f53b80d22b0 0x7f53b80d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.491+0000 7f53b77fe640 1 -- 172.21.15.83:0/2308520012 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1473838060 0 0) 0x7f53b80b3430 con 0x7f53b815dfe0 2024-03-06T07:30:28.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.491+0000 7f53bd18b640 1 --2- 172.21.15.83:0/2308520012 >> v2:172.21.15.183:3300/0 conn(0x7f53b80d22b0 0x7f53b80d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.491+0000 7f53bd18b640 1 -- 172.21.15.83:0/2308520012 >> v1:172.21.15.83:6789/0 conn(0x7f53b815dfe0 legacy=0x7f53b80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.491+0000 7f53bd18b640 1 -- 172.21.15.83:0/2308520012 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b80b3710 con 0x7f53b80b41f0 2024-03-06T07:30:28.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.491+0000 7f53b7fff640 1 --2- 172.21.15.83:0/2308520012 >> v2:172.21.15.183:3300/0 conn(0x7f53b80d22b0 0x7f53b80d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:28.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.492+0000 7f53bd18b640 1 --2- 172.21.15.83:0/2308520012 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53b80b41f0 0x7f53b80b4680 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f53b0009d90 tx=0x7f53b002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1056d15d263b6e4b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.492+0000 7f53b77fe640 1 -- 172.21.15.83:0/2308520012 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b0039070 con 0x7f53b80b41f0 2024-03-06T07:30:28.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.492+0000 7f53b77fe640 1 -- 172.21.15.83:0/2308520012 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53b002cc60 con 0x7f53b80b41f0 2024-03-06T07:30:28.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.492+0000 7f53b77fe640 1 -- 172.21.15.83:0/2308520012 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b0034660 con 0x7f53b80b41f0 2024-03-06T07:30:28.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.492+0000 7f53bebee640 1 -- 172.21.15.83:0/2308520012 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53b80b41f0 msgr2=0x7f53b80b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.492+0000 7f53bebee640 1 --2- 172.21.15.83:0/2308520012 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53b80b41f0 0x7f53b80b4680 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f53b0009d90 tx=0x7f53b002beb0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.492+0000 7f53bebee640 1 -- 172.21.15.83:0/2308520012 shutdown_connections 2024-03-06T07:30:28.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.492+0000 7f53bebee640 1 --2- 172.21.15.83:0/2308520012 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53b80b41f0 0x7f53b80b4680 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.492+0000 7f53bebee640 1 --2- 172.21.15.83:0/2308520012 >> v2:172.21.15.183:3300/0 conn(0x7f53b80d22b0 0x7f53b80d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.492+0000 7f53bebee640 1 -- 172.21.15.83:0/2308520012 >> 172.21.15.83:0/2308520012 conn(0x7f53b80d3cd0 msgr2=0x7f53b80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:28.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.492+0000 7f53bebee640 1 -- 172.21.15.83:0/2308520012 shutdown_connections 2024-03-06T07:30:28.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.492+0000 7f53bebee640 1 -- 172.21.15.83:0/2308520012 wait complete. 2024-03-06T07:30:28.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.494+0000 7f53bebee640 1 Processor -- start 2024-03-06T07:30:28.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.494+0000 7f53bebee640 1 -- start start 2024-03-06T07:30:28.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.494+0000 7f53bebee640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53b80b41f0 0x7f53b80b6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.494+0000 7f53bebee640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f53b80d22b0 0x7f53b815ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.494+0000 7f53bebee640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53b815dfe0 0x7f53b80b93c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.494+0000 7f53bebee640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b80cfa00 con 0x7f53b80b41f0 2024-03-06T07:30:28.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.494+0000 7f53bebee640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b80b3d60 con 0x7f53b815dfe0 2024-03-06T07:30:28.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.494+0000 7f53bebee640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53b80d1c70 con 0x7f53b80d22b0 2024-03-06T07:30:28.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.495+0000 7f53bc98a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53b80b41f0 0x7f53b80b6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.495+0000 7f53bc98a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53b80b41f0 0x7f53b80b6bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59118/0 (socket says 172.21.15.83:59118) 2024-03-06T07:30:28.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.495+0000 7f53bc98a640 1 -- 172.21.15.83:0/3662846952 learned_addr learned my addr 172.21.15.83:0/3662846952 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:28.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.495+0000 7f53bd18b640 1 --2- 172.21.15.83:0/3662846952 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53b815dfe0 0x7f53b80b93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.495+0000 7f53b7fff640 1 --2- 172.21.15.83:0/3662846952 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f53b80d22b0 0x7f53b815ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.495+0000 7f53bc98a640 1 -- 172.21.15.83:0/3662846952 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f53b80d22b0 msgr2=0x7f53b815ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.495+0000 7f53bc98a640 1 --2- 172.21.15.83:0/3662846952 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f53b80d22b0 0x7f53b815ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.495+0000 7f53bc98a640 1 -- 172.21.15.83:0/3662846952 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53b815dfe0 msgr2=0x7f53b80b93c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.495+0000 7f53bc98a640 1 --2- 172.21.15.83:0/3662846952 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53b815dfe0 0x7f53b80b93c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.495+0000 7f53bc98a640 1 -- 172.21.15.83:0/3662846952 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b80cbd00 con 0x7f53b80b41f0 2024-03-06T07:30:28.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.495+0000 7f53bc98a640 1 --2- 172.21.15.83:0/3662846952 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53b80b41f0 0x7f53b80b6bc0 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f53ac00c460 tx=0x7f53ac00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.495+0000 7f53b57fa640 1 -- 172.21.15.83:0/3662846952 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ac018070 con 0x7f53b80b41f0 2024-03-06T07:30:28.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.495+0000 7f53b57fa640 1 -- 172.21.15.83:0/3662846952 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53ac007500 con 0x7f53b80b41f0 2024-03-06T07:30:28.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.495+0000 7f53bebee640 1 -- 172.21.15.83:0/3662846952 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53b80caf40 con 0x7f53b80b41f0 2024-03-06T07:30:28.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.495+0000 7f53b57fa640 1 -- 172.21.15.83:0/3662846952 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ac004a20 con 0x7f53b80b41f0 2024-03-06T07:30:28.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.495+0000 7f53bebee640 1 -- 172.21.15.83:0/3662846952 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53b80cb500 con 0x7f53b80b41f0 2024-03-06T07:30:28.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.496+0000 7f53bebee640 1 -- 172.21.15.83:0/3662846952 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53b804f330 con 0x7f53b80b41f0 2024-03-06T07:30:28.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.497+0000 7f53b57fa640 1 -- 172.21.15.83:0/3662846952 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f53ac01b070 con 0x7f53b80b41f0 2024-03-06T07:30:28.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.497+0000 7f53b57fa640 1 --2- 172.21.15.83:0/3662846952 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f539c06b4d0 0x7f539c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.498+0000 7f53b57fa640 1 -- 172.21.15.83:0/3662846952 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f53ac089510 con 0x7f53b80b41f0 2024-03-06T07:30:28.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.498+0000 7f53b7fff640 1 --2- 172.21.15.83:0/3662846952 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f539c06b4d0 0x7f539c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.498+0000 7f53b7fff640 1 --2- 172.21.15.83:0/3662846952 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f539c06b4d0 0x7f539c06d990 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f53b80b7100 tx=0x7f53a4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.502+0000 7f53b57fa640 1 -- 172.21.15.83:0/3662846952 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f53ac058350 con 0x7f53b80b41f0 2024-03-06T07:30:28.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.713+0000 7f53bebee640 1 -- 172.21.15.83:0/3662846952 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53b80cdcd0 con 0x7f53b80b41f0 2024-03-06T07:30:28.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.714+0000 7f53b57fa640 1 -- 172.21.15.83:0/3662846952 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f53ac087e20 con 0x7f53b80b41f0 2024-03-06T07:30:28.716 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:30:28.716 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:30:28.716 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:30:28.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.715+0000 7f53bebee640 1 -- 172.21.15.83:0/3662846952 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f539c06b4d0 msgr2=0x7f539c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.715+0000 7f53bebee640 1 --2- 172.21.15.83:0/3662846952 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f539c06b4d0 0x7f539c06d990 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f53b80b7100 tx=0x7f53a4008040 comp rx=0 tx=0).stop 2024-03-06T07:30:28.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.715+0000 7f53bebee640 1 -- 172.21.15.83:0/3662846952 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53b80b41f0 msgr2=0x7f53b80b6bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.715+0000 7f53bebee640 1 --2- 172.21.15.83:0/3662846952 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53b80b41f0 0x7f53b80b6bc0 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f53ac00c460 tx=0x7f53ac00c920 comp rx=0 tx=0).stop 2024-03-06T07:30:28.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.715+0000 7f53bebee640 1 -- 172.21.15.83:0/3662846952 shutdown_connections 2024-03-06T07:30:28.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.715+0000 7f53bebee640 1 --2- 172.21.15.83:0/3662846952 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f539c06b4d0 0x7f539c06d990 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.715+0000 7f53bebee640 1 --2- 172.21.15.83:0/3662846952 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53b815dfe0 0x7f53b80b93c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.715+0000 7f53bebee640 1 --2- 172.21.15.83:0/3662846952 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f53b80d22b0 0x7f53b815ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.715+0000 7f53bebee640 1 --2- 172.21.15.83:0/3662846952 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f53b80b41f0 0x7f53b80b6bc0 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.715+0000 7f53bebee640 1 -- 172.21.15.83:0/3662846952 >> 172.21.15.83:0/3662846952 conn(0x7f53b80d3cd0 msgr2=0x7f53b806cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:28.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.716+0000 7f53bebee640 1 -- 172.21.15.83:0/3662846952 shutdown_connections 2024-03-06T07:30:28.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:28.716+0000 7f53bebee640 1 -- 172.21.15.83:0/3662846952 wait complete. 2024-03-06T07:30:28.930 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:30:28.930 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:30:29.030 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:29.931 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:30.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.030+0000 7faf1c0c2640 1 Processor -- start 2024-03-06T07:30:30.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.031+0000 7faf1c0c2640 1 -- start start 2024-03-06T07:30:30.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.031+0000 7faf1c0c2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7faf1415dfe0 0x7faf140cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:30.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.031+0000 7faf1c0c2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf140d29c0 0x7faf140b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:30.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.031+0000 7faf1c0c2640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf140b3430 con 0x7faf140cf830 2024-03-06T07:30:30.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.031+0000 7faf1c0c2640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf140b31e0 con 0x7faf1415dfe0 2024-03-06T07:30:30.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.031+0000 7faf1c0c2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf140b3d60 con 0x7faf140d29c0 2024-03-06T07:30:30.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.031+0000 7faf19e5e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7faf1415dfe0 0x7faf140cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:30.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.031+0000 7faf19e5e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7faf1415dfe0 0x7faf140cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59132/0 (socket says 172.21.15.83:59132) 2024-03-06T07:30:30.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.031+0000 7faf19e5e640 1 -- 172.21.15.83:0/649452979 learned_addr learned my addr 172.21.15.83:0/649452979 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:30.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.032+0000 7faf1a65f640 1 --2- 172.21.15.83:0/649452979 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf140d29c0 0x7faf140b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:30.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.032+0000 7faf19e5e640 1 -- 172.21.15.83:0/649452979 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf140d29c0 msgr2=0x7faf140b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:30.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.032+0000 7faf19e5e640 1 --2- 172.21.15.83:0/649452979 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf140d29c0 0x7faf140b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.032+0000 7faf19e5e640 1 -- 172.21.15.83:0/649452979 >> v1:172.21.15.183:6789/0 conn(0x7faf140cf830 legacy=0x7faf140d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:30.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.032+0000 7faf19e5e640 1 -- 172.21.15.83:0/649452979 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf140b3710 con 0x7faf1415dfe0 2024-03-06T07:30:30.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.032+0000 7faf1a65f640 1 --2- 172.21.15.83:0/649452979 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf140d29c0 0x7faf140b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:30.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.032+0000 7faf19e5e640 1 --2- 172.21.15.83:0/649452979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf1415dfe0 0x7faf140cf2f0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7faf10009870 tx=0x7faf1002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e7f1ef0fd95fe40 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:30.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.033+0000 7faf18e5c640 1 -- 172.21.15.83:0/649452979 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf10034070 con 0x7faf1415dfe0 2024-03-06T07:30:30.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.033+0000 7faf18e5c640 1 -- 172.21.15.83:0/649452979 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf10030840 con 0x7faf1415dfe0 2024-03-06T07:30:30.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.033+0000 7faf18e5c640 1 -- 172.21.15.83:0/649452979 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf10030c10 con 0x7faf1415dfe0 2024-03-06T07:30:30.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.033+0000 7faf1c0c2640 1 -- 172.21.15.83:0/649452979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf1415dfe0 msgr2=0x7faf140cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:30.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.033+0000 7faf1c0c2640 1 --2- 172.21.15.83:0/649452979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf1415dfe0 0x7faf140cf2f0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7faf10009870 tx=0x7faf1002b980 comp rx=0 tx=0).stop 2024-03-06T07:30:30.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.033+0000 7faf1c0c2640 1 -- 172.21.15.83:0/649452979 shutdown_connections 2024-03-06T07:30:30.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.033+0000 7faf1c0c2640 1 --2- 172.21.15.83:0/649452979 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf140d29c0 0x7faf140b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.033+0000 7faf1c0c2640 1 --2- 172.21.15.83:0/649452979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf1415dfe0 0x7faf140cf2f0 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.033+0000 7faf1c0c2640 1 -- 172.21.15.83:0/649452979 >> 172.21.15.83:0/649452979 conn(0x7faf140d3cd0 msgr2=0x7faf140d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:30.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.033+0000 7faf1c0c2640 1 -- 172.21.15.83:0/649452979 shutdown_connections 2024-03-06T07:30:30.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.033+0000 7faf1c0c2640 1 -- 172.21.15.83:0/649452979 wait complete. 2024-03-06T07:30:30.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.034+0000 7faf1c0c2640 1 Processor -- start 2024-03-06T07:30:30.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.035+0000 7faf1c0c2640 1 -- start start 2024-03-06T07:30:30.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.035+0000 7faf1c0c2640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf140cf830 0x7faf14150000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:30.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.035+0000 7faf1c0c2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf140d29c0 0x7faf1409f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:30.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.035+0000 7faf1c0c2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf1415dfe0 0x7faf140cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:30.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.035+0000 7faf1c0c2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf140d1df0 con 0x7faf1415dfe0 2024-03-06T07:30:30.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.035+0000 7faf1c0c2640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf140d1c70 con 0x7faf140cf830 2024-03-06T07:30:30.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.035+0000 7faf1c0c2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf1415e3c0 con 0x7faf140d29c0 2024-03-06T07:30:30.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.035+0000 7faf1a65f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf1415dfe0 0x7faf140cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:30.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.035+0000 7faf1a65f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf1415dfe0 0x7faf140cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59148/0 (socket says 172.21.15.83:59148) 2024-03-06T07:30:30.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.035+0000 7faf1a65f640 1 -- 172.21.15.83:0/4062536402 learned_addr learned my addr 172.21.15.83:0/4062536402 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:30.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.035+0000 7faf1965d640 1 --2- 172.21.15.83:0/4062536402 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf140d29c0 0x7faf1409f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:30.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.036+0000 7faf19e5e640 1 --2- 172.21.15.83:0/4062536402 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf140cf830 0x7faf14150000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:30.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.036+0000 7faf1a65f640 1 -- 172.21.15.83:0/4062536402 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf140d29c0 msgr2=0x7faf1409f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:30.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.036+0000 7faf1a65f640 1 --2- 172.21.15.83:0/4062536402 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf140d29c0 0x7faf1409f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.036+0000 7faf1a65f640 1 -- 172.21.15.83:0/4062536402 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf140cf830 msgr2=0x7faf14150000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:30.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.036+0000 7faf1a65f640 1 --2- 172.21.15.83:0/4062536402 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf140cf830 0x7faf14150000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.036+0000 7faf1a65f640 1 -- 172.21.15.83:0/4062536402 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf140cb9b0 con 0x7faf1415dfe0 2024-03-06T07:30:30.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.036+0000 7faf19e5e640 1 --2- 172.21.15.83:0/4062536402 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf140cf830 0x7faf14150000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:30.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.036+0000 7faf1965d640 1 --2- 172.21.15.83:0/4062536402 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf140d29c0 0x7faf1409f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:30.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.036+0000 7faf1a65f640 1 --2- 172.21.15.83:0/4062536402 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf1415dfe0 0x7faf140cada0 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7faf0c007ef0 tx=0x7faf0c00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:30.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.036+0000 7faf0a7fc640 1 -- 172.21.15.83:0/4062536402 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0c01a070 con 0x7faf1415dfe0 2024-03-06T07:30:30.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.036+0000 7faf0a7fc640 1 -- 172.21.15.83:0/4062536402 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf0c003b90 con 0x7faf1415dfe0 2024-03-06T07:30:30.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.036+0000 7faf0a7fc640 1 -- 172.21.15.83:0/4062536402 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0c004030 con 0x7faf1415dfe0 2024-03-06T07:30:30.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.036+0000 7faf1c0c2640 1 -- 172.21.15.83:0/4062536402 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf140cbc90 con 0x7faf1415dfe0 2024-03-06T07:30:30.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.036+0000 7faf1c0c2640 1 -- 172.21.15.83:0/4062536402 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf140cc250 con 0x7faf1415dfe0 2024-03-06T07:30:30.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.037+0000 7faf1c0c2640 1 -- 172.21.15.83:0/4062536402 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf1404f330 con 0x7faf1415dfe0 2024-03-06T07:30:30.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.037+0000 7faf0a7fc640 1 -- 172.21.15.83:0/4062536402 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faf0c01d070 con 0x7faf1415dfe0 2024-03-06T07:30:30.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.038+0000 7faf0a7fc640 1 --2- 172.21.15.83:0/4062536402 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7faef806b500 0x7faef806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:30.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.038+0000 7faf19e5e640 1 --2- 172.21.15.83:0/4062536402 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7faef806b500 0x7faef806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:30.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.038+0000 7faf0a7fc640 1 -- 172.21.15.83:0/4062536402 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7faf0c08a450 con 0x7faf1415dfe0 2024-03-06T07:30:30.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.038+0000 7faf19e5e640 1 --2- 172.21.15.83:0/4062536402 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7faef806b500 0x7faef806d9c0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7faf10036000 tx=0x7faf1003b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:30.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.046+0000 7faf0a7fc640 1 -- 172.21.15.83:0/4062536402 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faf0c010070 con 0x7faf1415dfe0 2024-03-06T07:30:30.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.262+0000 7faf1c0c2640 1 -- 172.21.15.83:0/4062536402 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf140b5460 con 0x7faf1415dfe0 2024-03-06T07:30:30.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.264+0000 7faf0a7fc640 1 -- 172.21.15.83:0/4062536402 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7faf0c058bb0 con 0x7faf1415dfe0 2024-03-06T07:30:30.266 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:30:30.267 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:30:30.267 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:30:30.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.266+0000 7faf1c0c2640 1 -- 172.21.15.83:0/4062536402 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7faef806b500 msgr2=0x7faef806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:30.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.266+0000 7faf1c0c2640 1 --2- 172.21.15.83:0/4062536402 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7faef806b500 0x7faef806d9c0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7faf10036000 tx=0x7faf1003b040 comp rx=0 tx=0).stop 2024-03-06T07:30:30.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.266+0000 7faf1c0c2640 1 -- 172.21.15.83:0/4062536402 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf1415dfe0 msgr2=0x7faf140cada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:30.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.266+0000 7faf1c0c2640 1 --2- 172.21.15.83:0/4062536402 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf1415dfe0 0x7faf140cada0 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7faf0c007ef0 tx=0x7faf0c00d4a0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.266+0000 7faf1c0c2640 1 -- 172.21.15.83:0/4062536402 shutdown_connections 2024-03-06T07:30:30.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.266+0000 7faf1c0c2640 1 --2- 172.21.15.83:0/4062536402 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7faef806b500 0x7faef806d9c0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.266+0000 7faf1c0c2640 1 --2- 172.21.15.83:0/4062536402 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf1415dfe0 0x7faf140cada0 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.266+0000 7faf1c0c2640 1 --2- 172.21.15.83:0/4062536402 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf140d29c0 0x7faf1409f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.266+0000 7faf1c0c2640 1 --2- 172.21.15.83:0/4062536402 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf140cf830 0x7faf14150000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.267+0000 7faf1c0c2640 1 -- 172.21.15.83:0/4062536402 >> 172.21.15.83:0/4062536402 conn(0x7faf140d3cd0 msgr2=0x7faf140b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:30.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.267+0000 7faf1c0c2640 1 -- 172.21.15.83:0/4062536402 shutdown_connections 2024-03-06T07:30:30.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:30.267+0000 7faf1c0c2640 1 -- 172.21.15.83:0/4062536402 wait complete. 2024-03-06T07:30:30.456 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:30.459 DEBUG:teuthology.orchestra.run.smithi183:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:30.481 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:30:30.481 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:30:31.032 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:31.482 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:31.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.582+0000 7f5eb793a640 1 Processor -- start 2024-03-06T07:30:31.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.582+0000 7f5eb793a640 1 -- start start 2024-03-06T07:30:31.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.583+0000 7f5eb793a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5eb015dfe0 0x7f5eb00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.583+0000 7f5eb793a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f5eb00d29c0 0x7f5eb00b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.583+0000 7f5eb793a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb00b3430 con 0x7f5eb00cf830 2024-03-06T07:30:31.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.583+0000 7f5eb793a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5eb00b31e0 con 0x7f5eb015dfe0 2024-03-06T07:30:31.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.583+0000 7f5eb793a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5eb00b3d60 con 0x7f5eb00d29c0 2024-03-06T07:30:31.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.583+0000 7f5eb56d6640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5eb015dfe0 0x7f5eb00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.583+0000 7f5eb4ed5640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f5eb00cf830 0x7f5eb00d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41612/0 (socket says 172.21.15.83:41612) 2024-03-06T07:30:31.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.583+0000 7f5eb4ed5640 1 -- 172.21.15.83:0/3292767146 learned_addr learned my addr 172.21.15.83:0/3292767146 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:30:31.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.583+0000 7f5eb56d6640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5eb015dfe0 0x7f5eb00cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59160/0 (socket says 172.21.15.83:59160) 2024-03-06T07:30:31.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.584+0000 7f5eb5ed7640 1 --2- 172.21.15.83:0/3292767146 >> v2:172.21.15.183:3300/0 conn(0x7f5eb00d29c0 0x7f5eb00b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.584+0000 7f5eb56d6640 1 -- 172.21.15.83:0/3292767146 >> v2:172.21.15.183:3300/0 conn(0x7f5eb00d29c0 msgr2=0x7f5eb00b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:31.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.584+0000 7f5eb56d6640 1 --2- 172.21.15.83:0/3292767146 >> v2:172.21.15.183:3300/0 conn(0x7f5eb00d29c0 0x7f5eb00b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.584+0000 7f5eb56d6640 1 -- 172.21.15.83:0/3292767146 >> v1:172.21.15.83:6789/0 conn(0x7f5eb00cf830 legacy=0x7f5eb00d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.584+0000 7f5ea7fff640 1 -- 172.21.15.83:0/3292767146 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2389644536 0 0) 0x7f5eb00b3430 con 0x7f5eb00cf830 2024-03-06T07:30:31.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.584+0000 7f5eb56d6640 1 -- 172.21.15.83:0/3292767146 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb00b3710 con 0x7f5eb015dfe0 2024-03-06T07:30:31.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.584+0000 7f5eb5ed7640 1 --2- 172.21.15.83:0/3292767146 >> v2:172.21.15.183:3300/0 conn(0x7f5eb00d29c0 0x7f5eb00b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:31.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.584+0000 7f5eb56d6640 1 --2- 172.21.15.83:0/3292767146 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5eb015dfe0 0x7f5eb00cf2f0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f5eb00b3cd0 tx=0x7f5eac02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=462d053352ade852 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:31.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.584+0000 7f5ea7fff640 1 -- 172.21.15.83:0/3292767146 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eac035020 con 0x7f5eb015dfe0 2024-03-06T07:30:31.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.585+0000 7f5ea7fff640 1 -- 172.21.15.83:0/3292767146 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eac031830 con 0x7f5eb015dfe0 2024-03-06T07:30:31.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.585+0000 7f5ea7fff640 1 -- 172.21.15.83:0/3292767146 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eac031c00 con 0x7f5eb015dfe0 2024-03-06T07:30:31.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.585+0000 7f5eb793a640 1 -- 172.21.15.83:0/3292767146 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5eb015dfe0 msgr2=0x7f5eb00cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.585+0000 7f5eb793a640 1 --2- 172.21.15.83:0/3292767146 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5eb015dfe0 0x7f5eb00cf2f0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f5eb00b3cd0 tx=0x7f5eac02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.585+0000 7f5eb793a640 1 -- 172.21.15.83:0/3292767146 shutdown_connections 2024-03-06T07:30:31.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.585+0000 7f5eb793a640 1 --2- 172.21.15.83:0/3292767146 >> v2:172.21.15.183:3300/0 conn(0x7f5eb00d29c0 0x7f5eb00b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.585+0000 7f5eb793a640 1 --2- 172.21.15.83:0/3292767146 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5eb015dfe0 0x7f5eb00cf2f0 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.585+0000 7f5eb793a640 1 -- 172.21.15.83:0/3292767146 >> 172.21.15.83:0/3292767146 conn(0x7f5eb00d3cd0 msgr2=0x7f5eb00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:31.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.585+0000 7f5eb793a640 1 -- 172.21.15.83:0/3292767146 shutdown_connections 2024-03-06T07:30:31.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.585+0000 7f5eb793a640 1 -- 172.21.15.83:0/3292767146 wait complete. 2024-03-06T07:30:31.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.586+0000 7f5eb793a640 1 Processor -- start 2024-03-06T07:30:31.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.586+0000 7f5eb793a640 1 -- start start 2024-03-06T07:30:31.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.587+0000 7f5eb793a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5eb00cf830 0x7f5eb0160600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.587+0000 7f5eb793a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5eb00d29c0 0x7f5eb00cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.587+0000 7f5eb793a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5eb015dfe0 0x7f5eb00cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.587+0000 7f5eb793a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb00d1df0 con 0x7f5eb015dfe0 2024-03-06T07:30:31.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.587+0000 7f5eb793a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb00d1c70 con 0x7f5eb00cf830 2024-03-06T07:30:31.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.587+0000 7f5eb793a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb015e3c0 con 0x7f5eb00d29c0 2024-03-06T07:30:31.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.587+0000 7f5eb5ed7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5eb015dfe0 0x7f5eb00cb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.587+0000 7f5eb5ed7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5eb015dfe0 0x7f5eb00cb2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59176/0 (socket says 172.21.15.83:59176) 2024-03-06T07:30:31.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.587+0000 7f5eb5ed7640 1 -- 172.21.15.83:0/3721413253 learned_addr learned my addr 172.21.15.83:0/3721413253 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:31.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.587+0000 7f5eb56d6640 1 --2- 172.21.15.83:0/3721413253 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5eb00cf830 0x7f5eb0160600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.587+0000 7f5eb4ed5640 1 --2- 172.21.15.83:0/3721413253 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5eb00d29c0 0x7f5eb00cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.587+0000 7f5eb5ed7640 1 -- 172.21.15.83:0/3721413253 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5eb00d29c0 msgr2=0x7f5eb00cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.587+0000 7f5eb5ed7640 1 --2- 172.21.15.83:0/3721413253 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5eb00d29c0 0x7f5eb00cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.587+0000 7f5eb5ed7640 1 -- 172.21.15.83:0/3721413253 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5eb00cf830 msgr2=0x7f5eb0160600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.587+0000 7f5eb5ed7640 1 --2- 172.21.15.83:0/3721413253 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5eb00cf830 0x7f5eb0160600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.587+0000 7f5eb5ed7640 1 -- 172.21.15.83:0/3721413253 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb015f560 con 0x7f5eb015dfe0 2024-03-06T07:30:31.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.588+0000 7f5eb5ed7640 1 --2- 172.21.15.83:0/3721413253 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5eb015dfe0 0x7f5eb00cb2e0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f5ea800ce80 tx=0x7f5ea800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:31.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.588+0000 7f5ea5ffb640 1 -- 172.21.15.83:0/3721413253 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea801a070 con 0x7f5eb015dfe0 2024-03-06T07:30:31.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.588+0000 7f5ea5ffb640 1 -- 172.21.15.83:0/3721413253 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ea800d880 con 0x7f5eb015dfe0 2024-03-06T07:30:31.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.588+0000 7f5eb793a640 1 -- 172.21.15.83:0/3721413253 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eb015f840 con 0x7f5eb015dfe0 2024-03-06T07:30:31.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.588+0000 7f5ea5ffb640 1 -- 172.21.15.83:0/3721413253 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea800dc70 con 0x7f5eb015dfe0 2024-03-06T07:30:31.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.588+0000 7f5eb793a640 1 -- 172.21.15.83:0/3721413253 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eb015fe00 con 0x7f5eb015dfe0 2024-03-06T07:30:31.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.589+0000 7f5eb793a640 1 -- 172.21.15.83:0/3721413253 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5eb004f330 con 0x7f5eb015dfe0 2024-03-06T07:30:31.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.589+0000 7f5ea5ffb640 1 -- 172.21.15.83:0/3721413253 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5ea801d070 con 0x7f5eb015dfe0 2024-03-06T07:30:31.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.590+0000 7f5ea5ffb640 1 --2- 172.21.15.83:0/3721413253 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5e9406b4d0 0x7f5e9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.590+0000 7f5eb56d6640 1 --2- 172.21.15.83:0/3721413253 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5e9406b4d0 0x7f5e9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.590+0000 7f5ea5ffb640 1 -- 172.21.15.83:0/3721413253 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5ea808a9c0 con 0x7f5eb015dfe0 2024-03-06T07:30:31.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.590+0000 7f5eb56d6640 1 --2- 172.21.15.83:0/3721413253 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5e9406b4d0 0x7f5e9406d990 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f5eac036000 tx=0x7f5eac008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:31.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.594+0000 7f5ea5ffb640 1 -- 172.21.15.83:0/3721413253 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5ea808ce20 con 0x7f5eb015dfe0 2024-03-06T07:30:31.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.806+0000 7f5eb793a640 1 -- 172.21.15.83:0/3721413253 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5eb00b48e0 con 0x7f5eb015dfe0 2024-03-06T07:30:31.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.806+0000 7f5ea5ffb640 1 -- 172.21.15.83:0/3721413253 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f5ea80591b0 con 0x7f5eb015dfe0 2024-03-06T07:30:31.808 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:30:31.808 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:30:31.809 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:30:31.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.808+0000 7f5eb793a640 1 -- 172.21.15.83:0/3721413253 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5e9406b4d0 msgr2=0x7f5e9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.808+0000 7f5eb793a640 1 --2- 172.21.15.83:0/3721413253 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5e9406b4d0 0x7f5e9406d990 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f5eac036000 tx=0x7f5eac008030 comp rx=0 tx=0).stop 2024-03-06T07:30:31.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.808+0000 7f5eb793a640 1 -- 172.21.15.83:0/3721413253 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5eb015dfe0 msgr2=0x7f5eb00cb2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.808+0000 7f5eb793a640 1 --2- 172.21.15.83:0/3721413253 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5eb015dfe0 0x7f5eb00cb2e0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f5ea800ce80 tx=0x7f5ea800a710 comp rx=0 tx=0).stop 2024-03-06T07:30:31.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.808+0000 7f5eb793a640 1 -- 172.21.15.83:0/3721413253 shutdown_connections 2024-03-06T07:30:31.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.808+0000 7f5eb793a640 1 --2- 172.21.15.83:0/3721413253 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5e9406b4d0 0x7f5e9406d990 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.808+0000 7f5eb793a640 1 --2- 172.21.15.83:0/3721413253 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5eb015dfe0 0x7f5eb00cb2e0 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.808+0000 7f5eb793a640 1 --2- 172.21.15.83:0/3721413253 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5eb00d29c0 0x7f5eb00cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.808+0000 7f5eb793a640 1 --2- 172.21.15.83:0/3721413253 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5eb00cf830 0x7f5eb0160600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.808+0000 7f5eb793a640 1 -- 172.21.15.83:0/3721413253 >> 172.21.15.83:0/3721413253 conn(0x7f5eb00d3cd0 msgr2=0x7f5eb00b4e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:31.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.808+0000 7f5eb793a640 1 -- 172.21.15.83:0/3721413253 shutdown_connections 2024-03-06T07:30:31.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:31.808+0000 7f5eb793a640 1 -- 172.21.15.83:0/3721413253 wait complete. 2024-03-06T07:30:32.022 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:30:32.023 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:30:33.024 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:33.034 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:33.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.127+0000 7efc4c98a640 1 Processor -- start 2024-03-06T07:30:33.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.127+0000 7efc4c98a640 1 -- start start 2024-03-06T07:30:33.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.128+0000 7efc4c98a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efc4815dfe0 0x7efc480cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.128+0000 7efc4c98a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7efc480cf830 0x7efc480d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.128+0000 7efc4c98a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7efc480d27f0 0x7efc480d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.128+0000 7efc4c98a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc480d2c50 con 0x7efc480d27f0 2024-03-06T07:30:33.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.128+0000 7efc4c98a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc480b31e0 con 0x7efc480cf830 2024-03-06T07:30:33.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.128+0000 7efc4c98a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc480b3d60 con 0x7efc4815dfe0 2024-03-06T07:30:33.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.128+0000 7efc46d9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7efc480d27f0 0x7efc480d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:33.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.128+0000 7efc46d9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7efc480d27f0 0x7efc480d0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59182/0 (socket says 172.21.15.83:59182) 2024-03-06T07:30:33.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.128+0000 7efc46d9d640 1 -- 172.21.15.83:0/1355813770 learned_addr learned my addr 172.21.15.83:0/1355813770 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:33.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.129+0000 7efc4659c640 1 --2- 172.21.15.83:0/1355813770 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efc4815dfe0 0x7efc480cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:33.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.129+0000 7efc45d9b640 1 --2- 172.21.15.83:0/1355813770 >> v2:172.21.15.183:3300/0 conn(0x7efc480cf830 0x7efc480d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:33.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.129+0000 7efc46d9d640 1 -- 172.21.15.83:0/1355813770 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efc4815dfe0 msgr2=0x7efc480cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:33.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.129+0000 7efc46d9d640 1 --2- 172.21.15.83:0/1355813770 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efc4815dfe0 0x7efc480cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.129+0000 7efc46d9d640 1 -- 172.21.15.83:0/1355813770 >> v2:172.21.15.183:3300/0 conn(0x7efc480cf830 msgr2=0x7efc480d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:33.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.129+0000 7efc46d9d640 1 --2- 172.21.15.83:0/1355813770 >> v2:172.21.15.183:3300/0 conn(0x7efc480cf830 0x7efc480d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.129+0000 7efc46d9d640 1 -- 172.21.15.83:0/1355813770 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc480b3430 con 0x7efc480d27f0 2024-03-06T07:30:33.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.129+0000 7efc46d9d640 1 --2- 172.21.15.83:0/1355813770 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc480d27f0 0x7efc480d0300 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7efc4808cba0 tx=0x7efc3802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f489f31b474b256c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:33.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.130+0000 7efc4559a640 1 -- 172.21.15.83:0/1355813770 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc38035020 con 0x7efc480d27f0 2024-03-06T07:30:33.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.130+0000 7efc4559a640 1 -- 172.21.15.83:0/1355813770 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc38031870 con 0x7efc480d27f0 2024-03-06T07:30:33.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.130+0000 7efc4559a640 1 -- 172.21.15.83:0/1355813770 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc38031c40 con 0x7efc480d27f0 2024-03-06T07:30:33.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.130+0000 7efc4c98a640 1 -- 172.21.15.83:0/1355813770 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc480d27f0 msgr2=0x7efc480d0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:33.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.130+0000 7efc4c98a640 1 --2- 172.21.15.83:0/1355813770 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc480d27f0 0x7efc480d0300 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7efc4808cba0 tx=0x7efc3802b910 comp rx=0 tx=0).stop 2024-03-06T07:30:33.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.130+0000 7efc4c98a640 1 -- 172.21.15.83:0/1355813770 shutdown_connections 2024-03-06T07:30:33.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.130+0000 7efc4c98a640 1 --2- 172.21.15.83:0/1355813770 >> v2:172.21.15.183:3300/0 conn(0x7efc480cf830 0x7efc480d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.130+0000 7efc4c98a640 1 --2- 172.21.15.83:0/1355813770 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc480d27f0 0x7efc480d0300 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.130+0000 7efc4c98a640 1 --2- 172.21.15.83:0/1355813770 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efc4815dfe0 0x7efc480cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.130+0000 7efc4c98a640 1 -- 172.21.15.83:0/1355813770 >> 172.21.15.83:0/1355813770 conn(0x7efc480d3cd0 msgr2=0x7efc480d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:33.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.130+0000 7efc4c98a640 1 -- 172.21.15.83:0/1355813770 shutdown_connections 2024-03-06T07:30:33.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.131+0000 7efc4c98a640 1 -- 172.21.15.83:0/1355813770 wait complete. 2024-03-06T07:30:33.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.132+0000 7efc4c98a640 1 Processor -- start 2024-03-06T07:30:33.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.132+0000 7efc4c98a640 1 -- start start 2024-03-06T07:30:33.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.132+0000 7efc4c98a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc480cf830 0x7efc48161190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.132+0000 7efc4c98a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efc480d27f0 0x7efc480b8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.132+0000 7efc4c98a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efc4815dfe0 0x7efc48149f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.132+0000 7efc4c98a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc480d1c70 con 0x7efc480cf830 2024-03-06T07:30:33.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.133+0000 7efc4c98a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc480b39f0 con 0x7efc480d27f0 2024-03-06T07:30:33.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.133+0000 7efc4c98a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc480d1df0 con 0x7efc4815dfe0 2024-03-06T07:30:33.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.133+0000 7efc4659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc480cf830 0x7efc48161190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:33.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.133+0000 7efc4659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc480cf830 0x7efc48161190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59184/0 (socket says 172.21.15.83:59184) 2024-03-06T07:30:33.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.133+0000 7efc4659c640 1 -- 172.21.15.83:0/4211772606 learned_addr learned my addr 172.21.15.83:0/4211772606 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:33.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.133+0000 7efc46d9d640 1 --2- 172.21.15.83:0/4211772606 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efc4815dfe0 0x7efc48149f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:33.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.133+0000 7efc45d9b640 1 --2- 172.21.15.83:0/4211772606 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efc480d27f0 0x7efc480b8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:33.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.133+0000 7efc4659c640 1 -- 172.21.15.83:0/4211772606 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efc4815dfe0 msgr2=0x7efc48149f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:33.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.133+0000 7efc4659c640 1 --2- 172.21.15.83:0/4211772606 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efc4815dfe0 0x7efc48149f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.133+0000 7efc4659c640 1 -- 172.21.15.83:0/4211772606 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efc480d27f0 msgr2=0x7efc480b8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:33.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.133+0000 7efc4659c640 1 --2- 172.21.15.83:0/4211772606 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efc480d27f0 0x7efc480b8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.133+0000 7efc4659c640 1 -- 172.21.15.83:0/4211772606 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc480b8ee0 con 0x7efc480cf830 2024-03-06T07:30:33.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.133+0000 7efc4659c640 1 --2- 172.21.15.83:0/4211772606 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc480cf830 0x7efc48161190 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7efc40007e40 tx=0x7efc4000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:33.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.133+0000 7efc36ffd640 1 -- 172.21.15.83:0/4211772606 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc40014620 con 0x7efc480cf830 2024-03-06T07:30:33.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.133+0000 7efc36ffd640 1 -- 172.21.15.83:0/4211772606 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc40019070 con 0x7efc480cf830 2024-03-06T07:30:33.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.133+0000 7efc4c98a640 1 -- 172.21.15.83:0/4211772606 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc480b91c0 con 0x7efc480cf830 2024-03-06T07:30:33.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.133+0000 7efc36ffd640 1 -- 172.21.15.83:0/4211772606 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc40014a00 con 0x7efc480cf830 2024-03-06T07:30:33.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.133+0000 7efc4c98a640 1 -- 172.21.15.83:0/4211772606 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc480b9780 con 0x7efc480cf830 2024-03-06T07:30:33.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.134+0000 7efc4c98a640 1 -- 172.21.15.83:0/4211772606 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc4804f330 con 0x7efc480cf830 2024-03-06T07:30:33.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.135+0000 7efc36ffd640 1 -- 172.21.15.83:0/4211772606 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efc40014ba0 con 0x7efc480cf830 2024-03-06T07:30:33.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.135+0000 7efc36ffd640 1 --2- 172.21.15.83:0/4211772606 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efc2806b4d0 0x7efc2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.135+0000 7efc45d9b640 1 --2- 172.21.15.83:0/4211772606 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efc2806b4d0 0x7efc2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:33.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.135+0000 7efc36ffd640 1 -- 172.21.15.83:0/4211772606 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7efc4008a990 con 0x7efc480cf830 2024-03-06T07:30:33.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.136+0000 7efc45d9b640 1 --2- 172.21.15.83:0/4211772606 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efc2806b4d0 0x7efc2806d990 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7efc4814aff0 tx=0x7efc3c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:33.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.140+0000 7efc36ffd640 1 -- 172.21.15.83:0/4211772606 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efc40059180 con 0x7efc480cf830 2024-03-06T07:30:33.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.355+0000 7efc4c98a640 1 -- 172.21.15.83:0/4211772606 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc4815d4c0 con 0x7efc480cf830 2024-03-06T07:30:33.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.355+0000 7efc36ffd640 1 -- 172.21.15.83:0/4211772606 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7efc4005d220 con 0x7efc480cf830 2024-03-06T07:30:33.357 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:30:33.357 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:30:33.357 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:30:33.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.357+0000 7efc4c98a640 1 -- 172.21.15.83:0/4211772606 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efc2806b4d0 msgr2=0x7efc2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:33.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.357+0000 7efc4c98a640 1 --2- 172.21.15.83:0/4211772606 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efc2806b4d0 0x7efc2806d990 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7efc4814aff0 tx=0x7efc3c008040 comp rx=0 tx=0).stop 2024-03-06T07:30:33.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.357+0000 7efc4c98a640 1 -- 172.21.15.83:0/4211772606 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc480cf830 msgr2=0x7efc48161190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:33.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.357+0000 7efc4c98a640 1 --2- 172.21.15.83:0/4211772606 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc480cf830 0x7efc48161190 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7efc40007e40 tx=0x7efc4000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.357+0000 7efc4c98a640 1 -- 172.21.15.83:0/4211772606 shutdown_connections 2024-03-06T07:30:33.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.357+0000 7efc4c98a640 1 --2- 172.21.15.83:0/4211772606 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efc2806b4d0 0x7efc2806d990 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.357+0000 7efc4c98a640 1 --2- 172.21.15.83:0/4211772606 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efc4815dfe0 0x7efc48149f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.357+0000 7efc4c98a640 1 --2- 172.21.15.83:0/4211772606 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efc480d27f0 0x7efc480b8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.357+0000 7efc4c98a640 1 --2- 172.21.15.83:0/4211772606 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc480cf830 0x7efc48161190 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.357+0000 7efc4c98a640 1 -- 172.21.15.83:0/4211772606 >> 172.21.15.83:0/4211772606 conn(0x7efc480d3cd0 msgr2=0x7efc4806cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:33.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.357+0000 7efc4c98a640 1 -- 172.21.15.83:0/4211772606 shutdown_connections 2024-03-06T07:30:33.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:33.357+0000 7efc4c98a640 1 -- 172.21.15.83:0/4211772606 wait complete. 2024-03-06T07:30:33.574 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:30:33.574 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:30:34.576 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:34.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.681+0000 7f16962f4640 1 Processor -- start 2024-03-06T07:30:34.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.681+0000 7f16962f4640 1 -- start start 2024-03-06T07:30:34.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.682+0000 7f16962f4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f169015dfe0 0x7f16900cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.682+0000 7f16962f4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f16900cf830 0x7f16900d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.682+0000 7f16962f4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16900b3430 con 0x7f16900d27f0 2024-03-06T07:30:34.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.682+0000 7f16962f4640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16900d2c50 con 0x7f16900cf830 2024-03-06T07:30:34.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.682+0000 7f16962f4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16900b31e0 con 0x7f169015dfe0 2024-03-06T07:30:34.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.682+0000 7f168f7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f16900cf830 0x7f16900d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.682+0000 7f1694891640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f16900d27f0 0x7f16900d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41624/0 (socket says 172.21.15.83:41624) 2024-03-06T07:30:34.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.682+0000 7f1694891640 1 -- 172.21.15.83:0/982919356 learned_addr learned my addr 172.21.15.83:0/982919356 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:30:34.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.683+0000 7f168ffff640 1 --2- 172.21.15.83:0/982919356 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f169015dfe0 0x7f16900cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.683+0000 7f168f7fe640 1 -- 172.21.15.83:0/982919356 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f169015dfe0 msgr2=0x7f16900cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:34.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.683+0000 7f168f7fe640 1 --2- 172.21.15.83:0/982919356 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f169015dfe0 0x7f16900cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.683+0000 7f168f7fe640 1 -- 172.21.15.83:0/982919356 >> v1:172.21.15.83:6789/0 conn(0x7f16900d27f0 legacy=0x7f16900d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.683+0000 7f168f7fe640 1 -- 172.21.15.83:0/982919356 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16900b3710 con 0x7f16900cf830 2024-03-06T07:30:34.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.683+0000 7f168effd640 1 -- 172.21.15.83:0/982919356 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2514681967 0 0) 0x7f16900b3430 con 0x7f16900d27f0 2024-03-06T07:30:34.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.683+0000 7f168f7fe640 1 --2- 172.21.15.83:0/982919356 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16900cf830 0x7f16900d22b0 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f16840097a0 tx=0x7f168402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0190c6158968c92 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:34.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.684+0000 7f168effd640 1 -- 172.21.15.83:0/982919356 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1684035020 con 0x7f16900cf830 2024-03-06T07:30:34.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.684+0000 7f168effd640 1 -- 172.21.15.83:0/982919356 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1684031830 con 0x7f16900cf830 2024-03-06T07:30:34.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.684+0000 7f168effd640 1 -- 172.21.15.83:0/982919356 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1684031c00 con 0x7f16900cf830 2024-03-06T07:30:34.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.684+0000 7f16962f4640 1 -- 172.21.15.83:0/982919356 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16900cf830 msgr2=0x7f16900d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.684+0000 7f16962f4640 1 --2- 172.21.15.83:0/982919356 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16900cf830 0x7f16900d22b0 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f16840097a0 tx=0x7f168402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.684+0000 7f16962f4640 1 -- 172.21.15.83:0/982919356 shutdown_connections 2024-03-06T07:30:34.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.684+0000 7f16962f4640 1 --2- 172.21.15.83:0/982919356 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16900cf830 0x7f16900d22b0 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.684+0000 7f16962f4640 1 --2- 172.21.15.83:0/982919356 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f169015dfe0 0x7f16900cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.684+0000 7f16962f4640 1 -- 172.21.15.83:0/982919356 >> 172.21.15.83:0/982919356 conn(0x7f16900d3cd0 msgr2=0x7f16900d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:34.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.684+0000 7f16962f4640 1 -- 172.21.15.83:0/982919356 shutdown_connections 2024-03-06T07:30:34.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.684+0000 7f16962f4640 1 -- 172.21.15.83:0/982919356 wait complete. 2024-03-06T07:30:34.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.685+0000 7f16962f4640 1 Processor -- start 2024-03-06T07:30:34.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.685+0000 7f16962f4640 1 -- start start 2024-03-06T07:30:34.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.686+0000 7f16962f4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16900cf830 0x7f16900b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.686+0000 7f16962f4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16900d27f0 0x7f16900b8680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.686+0000 7f16962f4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f169015dfe0 0x7f1690150c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.686+0000 7f16962f4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16900d1c70 con 0x7f169015dfe0 2024-03-06T07:30:34.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.686+0000 7f16962f4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16900b3d60 con 0x7f16900d27f0 2024-03-06T07:30:34.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.686+0000 7f16962f4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16900d1df0 con 0x7f16900cf830 2024-03-06T07:30:34.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.686+0000 7f1694891640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f169015dfe0 0x7f1690150c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.686+0000 7f1694891640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f169015dfe0 0x7f1690150c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59212/0 (socket says 172.21.15.83:59212) 2024-03-06T07:30:34.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.686+0000 7f1694891640 1 -- 172.21.15.83:0/937005432 learned_addr learned my addr 172.21.15.83:0/937005432 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:34.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.686+0000 7f168ffff640 1 --2- 172.21.15.83:0/937005432 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16900cf830 0x7f16900b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.686+0000 7f168f7fe640 1 --2- 172.21.15.83:0/937005432 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16900d27f0 0x7f16900b8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.687+0000 7f1694891640 1 -- 172.21.15.83:0/937005432 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16900cf830 msgr2=0x7f16900b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.687+0000 7f1694891640 1 --2- 172.21.15.83:0/937005432 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16900cf830 0x7f16900b68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.687+0000 7f1694891640 1 -- 172.21.15.83:0/937005432 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16900d27f0 msgr2=0x7f16900b8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.687+0000 7f1694891640 1 --2- 172.21.15.83:0/937005432 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16900d27f0 0x7f16900b8680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.687+0000 7f1694891640 1 -- 172.21.15.83:0/937005432 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f169015a570 con 0x7f169015dfe0 2024-03-06T07:30:34.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.687+0000 7f168f7fe640 1 --2- 172.21.15.83:0/937005432 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16900d27f0 0x7f16900b8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:34.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.687+0000 7f1694891640 1 --2- 172.21.15.83:0/937005432 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f169015dfe0 0x7f1690150c30 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f168800c7a0 tx=0x7f168800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:34.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.687+0000 7f168cff9640 1 -- 172.21.15.83:0/937005432 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1688018070 con 0x7f169015dfe0 2024-03-06T07:30:34.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.687+0000 7f168cff9640 1 -- 172.21.15.83:0/937005432 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f168800a640 con 0x7f169015dfe0 2024-03-06T07:30:34.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.687+0000 7f16962f4640 1 -- 172.21.15.83:0/937005432 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16900cdc80 con 0x7f169015dfe0 2024-03-06T07:30:34.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.687+0000 7f168cff9640 1 -- 172.21.15.83:0/937005432 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f168800aab0 con 0x7f169015dfe0 2024-03-06T07:30:34.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.687+0000 7f16962f4640 1 -- 172.21.15.83:0/937005432 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16900b95a0 con 0x7f169015dfe0 2024-03-06T07:30:34.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.688+0000 7f16962f4640 1 -- 172.21.15.83:0/937005432 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f169004f330 con 0x7f169015dfe0 2024-03-06T07:30:34.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.689+0000 7f168cff9640 1 -- 172.21.15.83:0/937005432 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f168801b070 con 0x7f169015dfe0 2024-03-06T07:30:34.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.689+0000 7f168cff9640 1 --2- 172.21.15.83:0/937005432 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f167c06b4d0 0x7f167c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.690+0000 7f168ffff640 1 --2- 172.21.15.83:0/937005432 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f167c06b4d0 0x7f167c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.690+0000 7f168cff9640 1 -- 172.21.15.83:0/937005432 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1688089830 con 0x7f169015dfe0 2024-03-06T07:30:34.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.690+0000 7f168ffff640 1 --2- 172.21.15.83:0/937005432 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f167c06b4d0 0x7f167c06d990 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f1680006fd0 tx=0x7f1680008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:34.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.694+0000 7f168cff9640 1 -- 172.21.15.83:0/937005432 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f16900cdc80 con 0x7f169015dfe0 2024-03-06T07:30:34.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.910+0000 7f16962f4640 1 -- 172.21.15.83:0/937005432 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16900b09c0 con 0x7f169015dfe0 2024-03-06T07:30:34.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.910+0000 7f168cff9640 1 -- 172.21.15.83:0/937005432 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f1688058670 con 0x7f169015dfe0 2024-03-06T07:30:34.912 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:30:34.912 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:30:34.913 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:30:34.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.912+0000 7f16962f4640 1 -- 172.21.15.83:0/937005432 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f167c06b4d0 msgr2=0x7f167c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.912+0000 7f16962f4640 1 --2- 172.21.15.83:0/937005432 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f167c06b4d0 0x7f167c06d990 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f1680006fd0 tx=0x7f1680008040 comp rx=0 tx=0).stop 2024-03-06T07:30:34.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.912+0000 7f16962f4640 1 -- 172.21.15.83:0/937005432 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f169015dfe0 msgr2=0x7f1690150c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.912+0000 7f16962f4640 1 --2- 172.21.15.83:0/937005432 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f169015dfe0 0x7f1690150c30 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f168800c7a0 tx=0x7f168800cc60 comp rx=0 tx=0).stop 2024-03-06T07:30:34.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.912+0000 7f16962f4640 1 -- 172.21.15.83:0/937005432 shutdown_connections 2024-03-06T07:30:34.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.912+0000 7f16962f4640 1 --2- 172.21.15.83:0/937005432 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f167c06b4d0 0x7f167c06d990 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.912+0000 7f16962f4640 1 --2- 172.21.15.83:0/937005432 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f169015dfe0 0x7f1690150c30 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.912+0000 7f16962f4640 1 --2- 172.21.15.83:0/937005432 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16900d27f0 0x7f16900b8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.912+0000 7f16962f4640 1 --2- 172.21.15.83:0/937005432 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16900cf830 0x7f16900b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.912+0000 7f16962f4640 1 -- 172.21.15.83:0/937005432 >> 172.21.15.83:0/937005432 conn(0x7f16900d3cd0 msgr2=0x7f16900d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:34.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.912+0000 7f16962f4640 1 -- 172.21.15.83:0/937005432 shutdown_connections 2024-03-06T07:30:34.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:34.912+0000 7f16962f4640 1 -- 172.21.15.83:0/937005432 wait complete. 2024-03-06T07:30:35.036 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:35.127 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:30:35.127 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:30:36.128 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:36.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.226+0000 7fedf78d4640 1 Processor -- start 2024-03-06T07:30:36.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.227+0000 7fedf78d4640 1 -- start start 2024-03-06T07:30:36.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.227+0000 7fedf78d4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fedf00cf790 0x7fedf00ad860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.227+0000 7fedf78d4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedf00ae240 con 0x7fedf00adda0 2024-03-06T07:30:36.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.227+0000 7fedf78d4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedf00ae520 con 0x7fedf00723f0 2024-03-06T07:30:36.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.227+0000 7fedf78d4640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedf006cf70 con 0x7fedf00cf790 2024-03-06T07:30:36.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.228+0000 7fedf5670640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fedf00cf790 0x7fedf00ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.228+0000 7fedf4e6f640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fedf00adda0 0x7fedf0071ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41636/0 (socket says 172.21.15.83:41636) 2024-03-06T07:30:36.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.228+0000 7fedf5670640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fedf00cf790 0x7fedf00ad860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59218/0 (socket says 172.21.15.83:59218) 2024-03-06T07:30:36.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.228+0000 7fedf4e6f640 1 -- 172.21.15.83:0/2607467914 learned_addr learned my addr 172.21.15.83:0/2607467914 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:30:36.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.228+0000 7fedf5670640 1 -- 172.21.15.83:0/2607467914 >> v1:172.21.15.183:6789/0 conn(0x7fedf00723f0 legacy=0x7fedf006d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:36.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.228+0000 7fedf5670640 1 -- 172.21.15.83:0/2607467914 >> v1:172.21.15.83:6789/0 conn(0x7fedf00adda0 legacy=0x7fedf0071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.228+0000 7fede7fff640 1 -- 172.21.15.83:0/2607467914 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1874731736 0 0) 0x7fedf00ae240 con 0x7fedf00adda0 2024-03-06T07:30:36.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.228+0000 7fedf5670640 1 -- 172.21.15.83:0/2607467914 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf00ae800 con 0x7fedf00cf790 2024-03-06T07:30:36.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.229+0000 7fedf5670640 1 --2- 172.21.15.83:0/2607467914 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedf00cf790 0x7fedf00ad860 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7fede00097a0 tx=0x7fede002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f6bf063f1386371 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:36.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.229+0000 7fede7fff640 1 -- 172.21.15.83:0/2607467914 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede0035020 con 0x7fedf00cf790 2024-03-06T07:30:36.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.229+0000 7fede7fff640 1 -- 172.21.15.83:0/2607467914 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fede0031830 con 0x7fedf00cf790 2024-03-06T07:30:36.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.229+0000 7fede7fff640 1 -- 172.21.15.83:0/2607467914 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede0031c00 con 0x7fedf00cf790 2024-03-06T07:30:36.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.230+0000 7fedf78d4640 1 -- 172.21.15.83:0/2607467914 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedf00cf790 msgr2=0x7fedf00ad860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.230+0000 7fedf78d4640 1 --2- 172.21.15.83:0/2607467914 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedf00cf790 0x7fedf00ad860 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7fede00097a0 tx=0x7fede002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.230+0000 7fedf78d4640 1 -- 172.21.15.83:0/2607467914 shutdown_connections 2024-03-06T07:30:36.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.230+0000 7fedf78d4640 1 --2- 172.21.15.83:0/2607467914 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedf00cf790 0x7fedf00ad860 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.230+0000 7fedf78d4640 1 -- 172.21.15.83:0/2607467914 >> 172.21.15.83:0/2607467914 conn(0x7fedf00b1a90 msgr2=0x7fedf00b1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:36.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.230+0000 7fedf78d4640 1 -- 172.21.15.83:0/2607467914 shutdown_connections 2024-03-06T07:30:36.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.230+0000 7fedf78d4640 1 -- 172.21.15.83:0/2607467914 wait complete. 2024-03-06T07:30:36.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.232+0000 7fedf78d4640 1 Processor -- start 2024-03-06T07:30:36.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.232+0000 7fedf78d4640 1 -- start start 2024-03-06T07:30:36.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.233+0000 7fedf78d4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fedf00723f0 0x7fedf015bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.233+0000 7fedf78d4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fedf00adda0 0x7fedf00c4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.233+0000 7fedf78d4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedf00cf790 0x7fedf01f44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.233+0000 7fedf78d4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf015e700 con 0x7fedf00cf790 2024-03-06T07:30:36.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.233+0000 7fedf78d4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf015e580 con 0x7fedf00adda0 2024-03-06T07:30:36.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.233+0000 7fedf78d4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedf006db10 con 0x7fedf00723f0 2024-03-06T07:30:36.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.233+0000 7fedf5e71640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedf00cf790 0x7fedf01f44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.233+0000 7fedf5e71640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedf00cf790 0x7fedf01f44f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:59220/0 (socket says 172.21.15.83:59220) 2024-03-06T07:30:36.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.233+0000 7fedf5e71640 1 -- 172.21.15.83:0/3120897129 learned_addr learned my addr 172.21.15.83:0/3120897129 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:36.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.233+0000 7fedf5e71640 1 -- 172.21.15.83:0/3120897129 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fedf00723f0 msgr2=0x7fedf015bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.233+0000 7fedf5e71640 1 --2- 172.21.15.83:0/3120897129 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fedf00723f0 0x7fedf015bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.233+0000 7fedf5e71640 1 -- 172.21.15.83:0/3120897129 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fedf00adda0 msgr2=0x7fedf00c4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.233+0000 7fedf5e71640 1 --2- 172.21.15.83:0/3120897129 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fedf00adda0 0x7fedf00c4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.233+0000 7fedf5e71640 1 -- 172.21.15.83:0/3120897129 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf00c5410 con 0x7fedf00cf790 2024-03-06T07:30:36.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.233+0000 7fedf5e71640 1 --2- 172.21.15.83:0/3120897129 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedf00cf790 0x7fedf01f44f0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7fede800c6d0 tx=0x7fede800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:36.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.233+0000 7fede5ffb640 1 -- 172.21.15.83:0/3120897129 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede8016020 con 0x7fedf00cf790 2024-03-06T07:30:36.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.233+0000 7fede5ffb640 1 -- 172.21.15.83:0/3120897129 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fede800a840 con 0x7fedf00cf790 2024-03-06T07:30:36.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.234+0000 7fedf78d4640 1 -- 172.21.15.83:0/3120897129 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedf0159c50 con 0x7fedf00cf790 2024-03-06T07:30:36.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.234+0000 7fede5ffb640 1 -- 172.21.15.83:0/3120897129 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede800ac70 con 0x7fedf00cf790 2024-03-06T07:30:36.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.234+0000 7fedf78d4640 1 -- 172.21.15.83:0/3120897129 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedf015a210 con 0x7fedf00cf790 2024-03-06T07:30:36.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.235+0000 7fedf78d4640 1 -- 172.21.15.83:0/3120897129 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedf015a410 con 0x7fedf00cf790 2024-03-06T07:30:36.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.235+0000 7fede5ffb640 1 -- 172.21.15.83:0/3120897129 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fede8019020 con 0x7fedf00cf790 2024-03-06T07:30:36.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.235+0000 7fede5ffb640 1 --2- 172.21.15.83:0/3120897129 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fedd406b4d0 0x7fedd406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.236+0000 7fede5ffb640 1 -- 172.21.15.83:0/3120897129 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fede808a730 con 0x7fedf00cf790 2024-03-06T07:30:36.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.236+0000 7fedf5670640 1 --2- 172.21.15.83:0/3120897129 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fedd406b4d0 0x7fedd406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.236+0000 7fedf5670640 1 --2- 172.21.15.83:0/3120897129 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fedd406b4d0 0x7fedd406d990 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fede0036000 tx=0x7fede0030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:36.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.240+0000 7fede5ffb640 1 -- 172.21.15.83:0/3120897129 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fedf015a410 con 0x7fedf00cf790 2024-03-06T07:30:36.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.453+0000 7fedf78d4640 1 -- 172.21.15.83:0/3120897129 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fedf0159a40 con 0x7fedf00cf790 2024-03-06T07:30:36.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.453+0000 7fede5ffb640 1 -- 172.21.15.83:0/3120897129 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7fede8020020 con 0x7fedf00cf790 2024-03-06T07:30:36.455 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:30:36.455 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:30:36.455 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:30:36.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.456+0000 7fedf78d4640 1 -- 172.21.15.83:0/3120897129 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fedd406b4d0 msgr2=0x7fedd406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.456+0000 7fedf78d4640 1 --2- 172.21.15.83:0/3120897129 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fedd406b4d0 0x7fedd406d990 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fede0036000 tx=0x7fede0030000 comp rx=0 tx=0).stop 2024-03-06T07:30:36.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.456+0000 7fedf78d4640 1 -- 172.21.15.83:0/3120897129 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedf00cf790 msgr2=0x7fedf01f44f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.456+0000 7fedf78d4640 1 --2- 172.21.15.83:0/3120897129 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedf00cf790 0x7fedf01f44f0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7fede800c6d0 tx=0x7fede800cb90 comp rx=0 tx=0).stop 2024-03-06T07:30:36.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.456+0000 7fedf78d4640 1 -- 172.21.15.83:0/3120897129 shutdown_connections 2024-03-06T07:30:36.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.456+0000 7fedf78d4640 1 --2- 172.21.15.83:0/3120897129 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fedd406b4d0 0x7fedd406d990 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.456+0000 7fedf78d4640 1 --2- 172.21.15.83:0/3120897129 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedf00cf790 0x7fedf01f44f0 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.456+0000 7fedf78d4640 1 --2- 172.21.15.83:0/3120897129 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fedf00adda0 0x7fedf00c4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.456+0000 7fedf78d4640 1 --2- 172.21.15.83:0/3120897129 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fedf00723f0 0x7fedf015bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.456+0000 7fedf78d4640 1 -- 172.21.15.83:0/3120897129 >> 172.21.15.83:0/3120897129 conn(0x7fedf00b1a90 msgr2=0x7fedf015e910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:36.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.456+0000 7fedf78d4640 1 -- 172.21.15.83:0/3120897129 shutdown_connections 2024-03-06T07:30:36.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:36.456+0000 7fedf78d4640 1 -- 172.21.15.83:0/3120897129 wait complete. 2024-03-06T07:30:36.673 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:30:36.673 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:30:37.038 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:37.674 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:37.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.774+0000 7fcad6f81640 1 Processor -- start 2024-03-06T07:30:37.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.774+0000 7fcad6f81640 1 -- start start 2024-03-06T07:30:37.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.775+0000 7fcad6f81640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcad015bb20 0x7fcad00d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.775+0000 7fcad6f81640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcad00cd8c0 0x7fcad00d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.775+0000 7fcad6f81640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcad0161980 con 0x7fcad00cfff0 2024-03-06T07:30:37.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.775+0000 7fcad6f81640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcad0161c60 con 0x7fcad015bb20 2024-03-06T07:30:37.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.775+0000 7fcad6f81640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcad0161f40 con 0x7fcad00cd8c0 2024-03-06T07:30:37.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.775+0000 7fcad4d1d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcad015bb20 0x7fcad00d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.775+0000 7fcacffff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fcad00cfff0 0x7fcad00ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49316/0 (socket says 172.21.15.83:49316) 2024-03-06T07:30:37.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.775+0000 7fcacffff640 1 -- 172.21.15.83:0/217713723 learned_addr learned my addr 172.21.15.83:0/217713723 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:30:37.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.776+0000 7fcad551e640 1 --2- 172.21.15.83:0/217713723 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcad00cd8c0 0x7fcad00d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.776+0000 7fcad4d1d640 1 -- 172.21.15.83:0/217713723 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcad00cd8c0 msgr2=0x7fcad00d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:37.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.776+0000 7fcad4d1d640 1 --2- 172.21.15.83:0/217713723 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcad00cd8c0 0x7fcad00d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.776+0000 7fcad4d1d640 1 -- 172.21.15.83:0/217713723 >> v1:172.21.15.83:6789/0 conn(0x7fcad00cfff0 legacy=0x7fcad00ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.776+0000 7fcad4d1d640 1 -- 172.21.15.83:0/217713723 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcad0162220 con 0x7fcad015bb20 2024-03-06T07:30:37.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.776+0000 7fcacf7fe640 1 -- 172.21.15.83:0/217713723 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2767388571 0 0) 0x7fcad0161980 con 0x7fcad00cfff0 2024-03-06T07:30:37.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.776+0000 7fcad4d1d640 1 --2- 172.21.15.83:0/217713723 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcad015bb20 0x7fcad00d1df0 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7fcac40097a0 tx=0x7fcac402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=641318cd3337c127 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:37.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.777+0000 7fcacf7fe640 1 -- 172.21.15.83:0/217713723 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac4035020 con 0x7fcad015bb20 2024-03-06T07:30:37.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.777+0000 7fcacf7fe640 1 -- 172.21.15.83:0/217713723 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcac4031830 con 0x7fcad015bb20 2024-03-06T07:30:37.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.777+0000 7fcacf7fe640 1 -- 172.21.15.83:0/217713723 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac4031c00 con 0x7fcad015bb20 2024-03-06T07:30:37.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.777+0000 7fcad6f81640 1 -- 172.21.15.83:0/217713723 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcad015bb20 msgr2=0x7fcad00d1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.777+0000 7fcad6f81640 1 --2- 172.21.15.83:0/217713723 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcad015bb20 0x7fcad00d1df0 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7fcac40097a0 tx=0x7fcac402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.777+0000 7fcad6f81640 1 -- 172.21.15.83:0/217713723 shutdown_connections 2024-03-06T07:30:37.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.777+0000 7fcad6f81640 1 --2- 172.21.15.83:0/217713723 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcad00cd8c0 0x7fcad00d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.777+0000 7fcad6f81640 1 --2- 172.21.15.83:0/217713723 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcad015bb20 0x7fcad00d1df0 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.777+0000 7fcad6f81640 1 -- 172.21.15.83:0/217713723 >> 172.21.15.83:0/217713723 conn(0x7fcad009a9f0 msgr2=0x7fcad009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:37.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.777+0000 7fcad6f81640 1 -- 172.21.15.83:0/217713723 shutdown_connections 2024-03-06T07:30:37.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.777+0000 7fcad6f81640 1 -- 172.21.15.83:0/217713723 wait complete. 2024-03-06T07:30:37.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.778+0000 7fcad6f81640 1 Processor -- start 2024-03-06T07:30:37.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.778+0000 7fcad6f81640 1 -- start start 2024-03-06T07:30:37.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.779+0000 7fcad6f81640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcad00cd8c0 0x7fcad01f4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.779+0000 7fcad6f81640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcad00cfff0 0x7fcad00c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.779+0000 7fcad6f81640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcad015bb20 0x7fcad01f0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.779+0000 7fcad6f81640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad00d24b0 con 0x7fcad00cfff0 2024-03-06T07:30:37.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.779+0000 7fcad6f81640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad00d2330 con 0x7fcad015bb20 2024-03-06T07:30:37.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.779+0000 7fcad6f81640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcad00d1980 con 0x7fcad00cd8c0 2024-03-06T07:30:37.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.779+0000 7fcacffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcad00cfff0 0x7fcad00c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.779+0000 7fcacffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcad00cfff0 0x7fcad00c5eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52712/0 (socket says 172.21.15.83:52712) 2024-03-06T07:30:37.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.779+0000 7fcacffff640 1 -- 172.21.15.83:0/4275266350 learned_addr learned my addr 172.21.15.83:0/4275266350 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:37.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.779+0000 7fcad4d1d640 1 --2- 172.21.15.83:0/4275266350 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcad00cd8c0 0x7fcad01f4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.779+0000 7fcad551e640 1 --2- 172.21.15.83:0/4275266350 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcad015bb20 0x7fcad01f0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.779+0000 7fcacffff640 1 -- 172.21.15.83:0/4275266350 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcad00cd8c0 msgr2=0x7fcad01f4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.779+0000 7fcacffff640 1 --2- 172.21.15.83:0/4275266350 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcad00cd8c0 0x7fcad01f4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.779+0000 7fcacffff640 1 -- 172.21.15.83:0/4275266350 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcad015bb20 msgr2=0x7fcad01f0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.779+0000 7fcacffff640 1 --2- 172.21.15.83:0/4275266350 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcad015bb20 0x7fcad01f0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.779+0000 7fcacffff640 1 -- 172.21.15.83:0/4275266350 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcad00c65e0 con 0x7fcad00cfff0 2024-03-06T07:30:37.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.779+0000 7fcad551e640 1 --2- 172.21.15.83:0/4275266350 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcad015bb20 0x7fcad01f0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:37.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.780+0000 7fcacffff640 1 --2- 172.21.15.83:0/4275266350 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcad00cfff0 0x7fcad00c5eb0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7fcabc00c4d0 tx=0x7fcabc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:37.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.780+0000 7fcacd7fa640 1 -- 172.21.15.83:0/4275266350 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcabc018070 con 0x7fcad00cfff0 2024-03-06T07:30:37.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.780+0000 7fcacd7fa640 1 -- 172.21.15.83:0/4275266350 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcabc007500 con 0x7fcad00cfff0 2024-03-06T07:30:37.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.780+0000 7fcad6f81640 1 -- 172.21.15.83:0/4275266350 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcad00c68c0 con 0x7fcad00cfff0 2024-03-06T07:30:37.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.780+0000 7fcacd7fa640 1 -- 172.21.15.83:0/4275266350 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcabc004a20 con 0x7fcad00cfff0 2024-03-06T07:30:37.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.780+0000 7fcad6f81640 1 -- 172.21.15.83:0/4275266350 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcad0158fd0 con 0x7fcad00cfff0 2024-03-06T07:30:37.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.781+0000 7fcad6f81640 1 -- 172.21.15.83:0/4275266350 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcad00c6c00 con 0x7fcad00cfff0 2024-03-06T07:30:37.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.782+0000 7fcacd7fa640 1 -- 172.21.15.83:0/4275266350 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcabc01b070 con 0x7fcad00cfff0 2024-03-06T07:30:37.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.782+0000 7fcacd7fa640 1 --2- 172.21.15.83:0/4275266350 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcab406b4d0 0x7fcab406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.782+0000 7fcacd7fa640 1 -- 172.21.15.83:0/4275266350 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fcabc0894f0 con 0x7fcad00cfff0 2024-03-06T07:30:37.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.782+0000 7fcad4d1d640 1 --2- 172.21.15.83:0/4275266350 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcab406b4d0 0x7fcab406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.783+0000 7fcad4d1d640 1 --2- 172.21.15.83:0/4275266350 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcab406b4d0 0x7fcab406d990 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fcac4036000 tx=0x7fcac4008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:37.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.786+0000 7fcacd7fa640 1 -- 172.21.15.83:0/4275266350 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcabc0583c0 con 0x7fcad00cfff0 2024-03-06T07:30:38.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:37.999+0000 7fcad6f81640 1 -- 172.21.15.83:0/4275266350 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcad004f330 con 0x7fcad00cfff0 2024-03-06T07:30:38.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:38.000+0000 7fcacd7fa640 1 -- 172.21.15.83:0/4275266350 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7fcabc006e20 con 0x7fcad00cfff0 2024-03-06T07:30:38.002 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:30:38.002 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:30:38.002 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:30:38.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:38.002+0000 7fcad6f81640 1 -- 172.21.15.83:0/4275266350 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcab406b4d0 msgr2=0x7fcab406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:38.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:38.002+0000 7fcad6f81640 1 --2- 172.21.15.83:0/4275266350 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcab406b4d0 0x7fcab406d990 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fcac4036000 tx=0x7fcac4008030 comp rx=0 tx=0).stop 2024-03-06T07:30:38.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:38.002+0000 7fcad6f81640 1 -- 172.21.15.83:0/4275266350 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcad00cfff0 msgr2=0x7fcad00c5eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:38.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:38.002+0000 7fcad6f81640 1 --2- 172.21.15.83:0/4275266350 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcad00cfff0 0x7fcad00c5eb0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7fcabc00c4d0 tx=0x7fcabc00c990 comp rx=0 tx=0).stop 2024-03-06T07:30:38.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:38.002+0000 7fcad6f81640 1 -- 172.21.15.83:0/4275266350 shutdown_connections 2024-03-06T07:30:38.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:38.002+0000 7fcad6f81640 1 --2- 172.21.15.83:0/4275266350 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcab406b4d0 0x7fcab406d990 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:38.002+0000 7fcad6f81640 1 --2- 172.21.15.83:0/4275266350 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcad015bb20 0x7fcad01f0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:38.002+0000 7fcad6f81640 1 --2- 172.21.15.83:0/4275266350 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcad00cfff0 0x7fcad00c5eb0 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:38.002+0000 7fcad6f81640 1 --2- 172.21.15.83:0/4275266350 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcad00cd8c0 0x7fcad01f4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:38.002+0000 7fcad6f81640 1 -- 172.21.15.83:0/4275266350 >> 172.21.15.83:0/4275266350 conn(0x7fcad009a9f0 msgr2=0x7fcad009add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:38.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:38.002+0000 7fcad6f81640 1 -- 172.21.15.83:0/4275266350 shutdown_connections 2024-03-06T07:30:38.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:38.002+0000 7fcad6f81640 1 -- 172.21.15.83:0/4275266350 wait complete. 2024-03-06T07:30:38.216 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:30:38.216 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:30:39.040 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:39.217 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:39.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.316+0000 7f1b68a22640 1 Processor -- start 2024-03-06T07:30:39.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.317+0000 7f1b68a22640 1 -- start start 2024-03-06T07:30:39.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.317+0000 7f1b68a22640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1b6415c040 0x7f1b641603a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.317+0000 7f1b68a22640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b640cfc90 con 0x7f1b640d05e0 2024-03-06T07:30:39.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.317+0000 7f1b68a22640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b6415c530 con 0x7f1b640cf6f0 2024-03-06T07:30:39.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.317+0000 7f1b68a22640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b641608e0 con 0x7f1b6415c040 2024-03-06T07:30:39.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.317+0000 7f1b62d9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f1b640d05e0 0x7f1b640cc5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49332/0 (socket says 172.21.15.83:49332) 2024-03-06T07:30:39.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.317+0000 7f1b62d9d640 1 -- 172.21.15.83:0/3671486751 learned_addr learned my addr 172.21.15.83:0/3671486751 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:30:39.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.318+0000 7f1b6259c640 1 --2- 172.21.15.83:0/3671486751 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1b6415c040 0x7f1b641603a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.318+0000 7f1b61d9b640 1 -- 172.21.15.83:0/3671486751 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 423474362 0 0) 0x7f1b640cfc90 con 0x7f1b640d05e0 2024-03-06T07:30:39.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.318+0000 7f1b61d9b640 1 -- 172.21.15.83:0/3671486751 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b50003520 con 0x7f1b640d05e0 2024-03-06T07:30:39.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.318+0000 7f1b61d9b640 1 -- 172.21.15.83:0/3671486751 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3311555876 0 0) 0x7f1b50003520 con 0x7f1b640d05e0 2024-03-06T07:30:39.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.318+0000 7f1b61d9b640 1 -- 172.21.15.83:0/3671486751 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1b6415c040 msgr2=0x7f1b641603a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:39.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.318+0000 7f1b61d9b640 1 --2- 172.21.15.83:0/3671486751 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1b6415c040 0x7f1b641603a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.318+0000 7f1b61d9b640 1 -- 172.21.15.83:0/3671486751 >> v1:172.21.15.183:6789/0 conn(0x7f1b640cf6f0 legacy=0x7f1b640cfed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.318+0000 7f1b61d9b640 1 -- 172.21.15.83:0/3671486751 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b6415c810 con 0x7f1b640d05e0 2024-03-06T07:30:39.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.318+0000 7f1b61d9b640 1 -- 172.21.15.83:0/3671486751 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f1b5c0019f0 con 0x7f1b640d05e0 2024-03-06T07:30:39.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.318+0000 7f1b61d9b640 1 -- 172.21.15.83:0/3671486751 >> v1:172.21.15.83:6789/0 conn(0x7f1b640d05e0 legacy=0x7f1b640cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.318+0000 7f1b61d9b640 1 --2- 172.21.15.83:0/3671486751 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1b500039e0 0x7f1b50003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.318+0000 7f1b61d9b640 1 --2- 172.21.15.83:0/3671486751 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1b6415c040 0x7f1b50004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.318+0000 7f1b61d9b640 1 --2- 172.21.15.83:0/3671486751 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b50004d40 0x7f1b500051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.318+0000 7f1b61d9b640 1 -- 172.21.15.83:0/3671486751 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b641608e0 con 0x7f1b50004d40 2024-03-06T07:30:39.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.318+0000 7f1b61d9b640 1 -- 172.21.15.83:0/3671486751 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5000b9c0 con 0x7f1b500039e0 2024-03-06T07:30:39.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.318+0000 7f1b61d9b640 1 -- 172.21.15.83:0/3671486751 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5000bbc0 con 0x7f1b6415c040 2024-03-06T07:30:39.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.318+0000 7f1b62d9d640 1 --2- 172.21.15.83:0/3671486751 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b50004d40 0x7f1b500051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.318+0000 7f1b6259c640 1 --2- 172.21.15.83:0/3671486751 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1b500039e0 0x7f1b50003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.318+0000 7f1b59d9b640 1 --2- 172.21.15.83:0/3671486751 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1b6415c040 0x7f1b50004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.318+0000 7f1b62d9d640 1 -- 172.21.15.83:0/3671486751 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1b6415c040 msgr2=0x7f1b50004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:39.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.318+0000 7f1b62d9d640 1 --2- 172.21.15.83:0/3671486751 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1b6415c040 0x7f1b50004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.318+0000 7f1b62d9d640 1 -- 172.21.15.83:0/3671486751 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1b500039e0 msgr2=0x7f1b50003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:39.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.318+0000 7f1b62d9d640 1 --2- 172.21.15.83:0/3671486751 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1b500039e0 0x7f1b50003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.319+0000 7f1b62d9d640 1 -- 172.21.15.83:0/3671486751 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b50003520 con 0x7f1b50004d40 2024-03-06T07:30:39.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.320+0000 7f1b62d9d640 1 --2- 172.21.15.83:0/3671486751 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b50004d40 0x7f1b500051c0 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c00a920 tx=0x7f1b50005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae5f760cc103452a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:39.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.320+0000 7f1b61d9b640 1 -- 172.21.15.83:0/3671486751 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b5c00a4c0 con 0x7f1b50004d40 2024-03-06T07:30:39.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.320+0000 7f1b61d9b640 1 -- 172.21.15.83:0/3671486751 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b5c034650 con 0x7f1b50004d40 2024-03-06T07:30:39.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.320+0000 7f1b61d9b640 1 -- 172.21.15.83:0/3671486751 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b5c034a40 con 0x7f1b50004d40 2024-03-06T07:30:39.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.320+0000 7f1b68a22640 1 -- 172.21.15.83:0/3671486751 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b50004d40 msgr2=0x7f1b500051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.320+0000 7f1b68a22640 1 --2- 172.21.15.83:0/3671486751 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b50004d40 0x7f1b500051c0 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c00a920 tx=0x7f1b50005830 comp rx=0 tx=0).stop 2024-03-06T07:30:39.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.320+0000 7f1b62d9d640 1 -- 172.21.15.83:0/3671486751 reap_dead start 2024-03-06T07:30:39.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.320+0000 7f1b68a22640 1 -- 172.21.15.83:0/3671486751 shutdown_connections 2024-03-06T07:30:39.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.320+0000 7f1b68a22640 1 -- 172.21.15.83:0/3671486751 >> 172.21.15.83:0/3671486751 conn(0x7f1b640d1b20 msgr2=0x7f1b640d1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:39.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.320+0000 7f1b68a22640 1 -- 172.21.15.83:0/3671486751 shutdown_connections 2024-03-06T07:30:39.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.320+0000 7f1b68a22640 1 -- 172.21.15.83:0/3671486751 wait complete. 2024-03-06T07:30:39.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.322+0000 7f1b68a22640 1 Processor -- start 2024-03-06T07:30:39.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.322+0000 7f1b68a22640 1 -- start start 2024-03-06T07:30:39.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.322+0000 7f1b68a22640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1b64156c50 0x7f1b640b4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.322+0000 7f1b68a22640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b64157030 0x7f1b640caa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.322+0000 7f1b68a22640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1b640caf50 0x7f1b6406c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.322+0000 7f1b68a22640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6415d2f0 con 0x7f1b64157030 2024-03-06T07:30:39.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.323+0000 7f1b68a22640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6415cdd0 con 0x7f1b640caf50 2024-03-06T07:30:39.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.323+0000 7f1b68a22640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b641603a0 con 0x7f1b64156c50 2024-03-06T07:30:39.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.323+0000 7f1b59d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b64157030 0x7f1b640caa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.323+0000 7f1b59d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b64157030 0x7f1b640caa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52720/0 (socket says 172.21.15.83:52720) 2024-03-06T07:30:39.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.323+0000 7f1b59d9b640 1 -- 172.21.15.83:0/289724497 learned_addr learned my addr 172.21.15.83:0/289724497 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:39.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.323+0000 7f1b62d9d640 1 --2- 172.21.15.83:0/289724497 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1b640caf50 0x7f1b6406c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.323+0000 7f1b6259c640 1 --2- 172.21.15.83:0/289724497 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1b64156c50 0x7f1b640b4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.323+0000 7f1b59d9b640 1 -- 172.21.15.83:0/289724497 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1b64156c50 msgr2=0x7f1b640b4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.323+0000 7f1b59d9b640 1 --2- 172.21.15.83:0/289724497 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1b64156c50 0x7f1b640b4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.323+0000 7f1b59d9b640 1 -- 172.21.15.83:0/289724497 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1b640caf50 msgr2=0x7f1b6406c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.323+0000 7f1b59d9b640 1 --2- 172.21.15.83:0/289724497 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1b640caf50 0x7f1b6406c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.323+0000 7f1b59d9b640 1 -- 172.21.15.83:0/289724497 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b640c9830 con 0x7f1b64157030 2024-03-06T07:30:39.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.323+0000 7f1b59d9b640 1 --2- 172.21.15.83:0/289724497 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b64157030 0x7f1b640caa10 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c00f7c0 tx=0x7f1b4c00fc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:39.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.323+0000 7f1b5b7fe640 1 -- 172.21.15.83:0/289724497 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c0052e0 con 0x7f1b64157030 2024-03-06T07:30:39.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.323+0000 7f1b5b7fe640 1 -- 172.21.15.83:0/289724497 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b4c018070 con 0x7f1b64157030 2024-03-06T07:30:39.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.323+0000 7f1b68a22640 1 -- 172.21.15.83:0/289724497 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b640ca5d0 con 0x7f1b64157030 2024-03-06T07:30:39.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.323+0000 7f1b5b7fe640 1 -- 172.21.15.83:0/289724497 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c00d970 con 0x7f1b64157030 2024-03-06T07:30:39.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.323+0000 7f1b68a22640 1 -- 172.21.15.83:0/289724497 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b640c9af0 con 0x7f1b64157030 2024-03-06T07:30:39.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.324+0000 7f1b68a22640 1 -- 172.21.15.83:0/289724497 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b640cb360 con 0x7f1b64157030 2024-03-06T07:30:39.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.325+0000 7f1b5b7fe640 1 -- 172.21.15.83:0/289724497 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1b4c00db10 con 0x7f1b64157030 2024-03-06T07:30:39.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.325+0000 7f1b5b7fe640 1 --2- 172.21.15.83:0/289724497 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1b4406b500 0x7f1b4406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.325+0000 7f1b5b7fe640 1 -- 172.21.15.83:0/289724497 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1b4c08a1a0 con 0x7f1b64157030 2024-03-06T07:30:39.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.325+0000 7f1b6259c640 1 --2- 172.21.15.83:0/289724497 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1b4406b500 0x7f1b4406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.326+0000 7f1b6259c640 1 --2- 172.21.15.83:0/289724497 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1b4406b500 0x7f1b4406d9c0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f1b640b6130 tx=0x7f1b54008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:39.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.330+0000 7f1b5b7fe640 1 -- 172.21.15.83:0/289724497 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1b640cb360 con 0x7f1b64157030 2024-03-06T07:30:39.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.545+0000 7f1b68a22640 1 -- 172.21.15.83:0/289724497 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b6406e850 con 0x7f1b64157030 2024-03-06T07:30:39.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.545+0000 7f1b5b7fe640 1 -- 172.21.15.83:0/289724497 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f1b4c01a310 con 0x7f1b64157030 2024-03-06T07:30:39.547 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:30:39.547 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:30:39.547 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:30:39.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.547+0000 7f1b68a22640 1 -- 172.21.15.83:0/289724497 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1b4406b500 msgr2=0x7f1b4406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.547+0000 7f1b68a22640 1 --2- 172.21.15.83:0/289724497 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1b4406b500 0x7f1b4406d9c0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f1b640b6130 tx=0x7f1b54008040 comp rx=0 tx=0).stop 2024-03-06T07:30:39.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.547+0000 7f1b68a22640 1 -- 172.21.15.83:0/289724497 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b64157030 msgr2=0x7f1b640caa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.547+0000 7f1b68a22640 1 --2- 172.21.15.83:0/289724497 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b64157030 0x7f1b640caa10 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c00f7c0 tx=0x7f1b4c00fc80 comp rx=0 tx=0).stop 2024-03-06T07:30:39.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.547+0000 7f1b68a22640 1 -- 172.21.15.83:0/289724497 shutdown_connections 2024-03-06T07:30:39.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.547+0000 7f1b68a22640 1 --2- 172.21.15.83:0/289724497 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1b4406b500 0x7f1b4406d9c0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.547+0000 7f1b68a22640 1 --2- 172.21.15.83:0/289724497 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1b640caf50 0x7f1b6406c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.547+0000 7f1b68a22640 1 --2- 172.21.15.83:0/289724497 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b64157030 0x7f1b640caa10 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.547+0000 7f1b68a22640 1 --2- 172.21.15.83:0/289724497 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1b64156c50 0x7f1b640b4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.547+0000 7f1b68a22640 1 -- 172.21.15.83:0/289724497 >> 172.21.15.83:0/289724497 conn(0x7f1b640d1b20 msgr2=0x7f1b640cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:39.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.547+0000 7f1b68a22640 1 -- 172.21.15.83:0/289724497 shutdown_connections 2024-03-06T07:30:39.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:39.547+0000 7f1b68a22640 1 -- 172.21.15.83:0/289724497 wait complete. 2024-03-06T07:30:39.761 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:30:39.761 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:30:40.762 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:40.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.862+0000 7f58f969a640 1 Processor -- start 2024-03-06T07:30:40.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.862+0000 7f58f969a640 1 -- start start 2024-03-06T07:30:40.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.863+0000 7f58f969a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f58f40b4170 0x7f58f40b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:40.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.863+0000 7f58f969a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f58f40d0a80 0x7f58f40cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:40.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.863+0000 7f58f969a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58f40d25d0 con 0x7f58f40d4760 2024-03-06T07:30:40.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.863+0000 7f58f969a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58f40d4bf0 con 0x7f58f40b4170 2024-03-06T07:30:40.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.863+0000 7f58f969a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58f40ce9e0 con 0x7f58f40d0a80 2024-03-06T07:30:40.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.863+0000 7f58f27fc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f58f40d4760 0x7f58f40d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49334/0 (socket says 172.21.15.83:49334) 2024-03-06T07:30:40.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.863+0000 7f58f27fc640 1 -- 172.21.15.83:0/2300727523 learned_addr learned my addr 172.21.15.83:0/2300727523 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:30:40.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.864+0000 7f58f37fe640 1 --2- 172.21.15.83:0/2300727523 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f58f40d0a80 0x7f58f40cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:40.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.864+0000 7f58f2ffd640 1 --2- 172.21.15.83:0/2300727523 >> v2:172.21.15.183:3300/0 conn(0x7f58f40b4170 0x7f58f40b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:40.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.864+0000 7f58f1ffb640 1 -- 172.21.15.83:0/2300727523 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3031336084 0 0) 0x7f58f40d25d0 con 0x7f58f40d4760 2024-03-06T07:30:40.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.864+0000 7f58f1ffb640 1 -- 172.21.15.83:0/2300727523 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58dc003520 con 0x7f58f40d4760 2024-03-06T07:30:40.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.864+0000 7f58f1ffb640 1 -- 172.21.15.83:0/2300727523 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 557122117 0 0) 0x7f58dc003520 con 0x7f58f40d4760 2024-03-06T07:30:40.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.864+0000 7f58f1ffb640 1 -- 172.21.15.83:0/2300727523 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f58f40d0a80 msgr2=0x7f58f40cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:40.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.864+0000 7f58f1ffb640 1 --2- 172.21.15.83:0/2300727523 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f58f40d0a80 0x7f58f40cee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.864+0000 7f58f1ffb640 1 -- 172.21.15.83:0/2300727523 >> v2:172.21.15.183:3300/0 conn(0x7f58f40b4170 msgr2=0x7f58f40b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:40.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.864+0000 7f58f1ffb640 1 --2- 172.21.15.83:0/2300727523 >> v2:172.21.15.183:3300/0 conn(0x7f58f40b4170 0x7f58f40b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.864+0000 7f58f1ffb640 1 -- 172.21.15.83:0/2300727523 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58f40d2870 con 0x7f58f40d4760 2024-03-06T07:30:40.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.864+0000 7f58f1ffb640 1 -- 172.21.15.83:0/2300727523 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f58e8001690 con 0x7f58f40d4760 2024-03-06T07:30:40.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.864+0000 7f58f1ffb640 1 -- 172.21.15.83:0/2300727523 >> v1:172.21.15.83:6789/0 conn(0x7f58f40d4760 legacy=0x7f58f40d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:40.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.864+0000 7f58f1ffb640 1 --2- 172.21.15.83:0/2300727523 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f58f40d0a80 0x7f58dc0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:40.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.864+0000 7f58f1ffb640 1 --2- 172.21.15.83:0/2300727523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58dc003f30 0x7f58dc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:40.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.864+0000 7f58f1ffb640 1 --2- 172.21.15.83:0/2300727523 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f58dc004d40 0x7f58dc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:40.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.864+0000 7f58f1ffb640 1 -- 172.21.15.83:0/2300727523 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f40d4bf0 con 0x7f58dc003f30 2024-03-06T07:30:40.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.864+0000 7f58f1ffb640 1 -- 172.21.15.83:0/2300727523 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f40ce9e0 con 0x7f58dc004d40 2024-03-06T07:30:40.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.864+0000 7f58f1ffb640 1 -- 172.21.15.83:0/2300727523 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58dc00b9c0 con 0x7f58f40d0a80 2024-03-06T07:30:40.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.864+0000 7f58f27fc640 1 --2- 172.21.15.83:0/2300727523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58dc003f30 0x7f58dc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:40.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.864+0000 7f58f2ffd640 1 --2- 172.21.15.83:0/2300727523 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f58f40d0a80 0x7f58dc0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:40.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.864+0000 7f58f37fe640 1 --2- 172.21.15.83:0/2300727523 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f58dc004d40 0x7f58dc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:40.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.864+0000 7f58f27fc640 1 -- 172.21.15.83:0/2300727523 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f58f40d0a80 msgr2=0x7f58dc0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:40.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.864+0000 7f58f27fc640 1 --2- 172.21.15.83:0/2300727523 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f58f40d0a80 0x7f58dc0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.864+0000 7f58f27fc640 1 -- 172.21.15.83:0/2300727523 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f58dc004d40 msgr2=0x7f58dc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:40.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.864+0000 7f58f27fc640 1 --2- 172.21.15.83:0/2300727523 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f58dc004d40 0x7f58dc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.864+0000 7f58f27fc640 1 -- 172.21.15.83:0/2300727523 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58dc003520 con 0x7f58dc003f30 2024-03-06T07:30:40.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.864+0000 7f58f2ffd640 1 --2- 172.21.15.83:0/2300727523 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f58f40d0a80 0x7f58dc0039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:40.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.865+0000 7f58f27fc640 1 --2- 172.21.15.83:0/2300727523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58dc003f30 0x7f58dc004800 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f58e800c210 tx=0x7f58dc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93e65ef7a3327d4b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:40.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.865+0000 7f58f1ffb640 1 -- 172.21.15.83:0/2300727523 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e8007d70 con 0x7f58dc003f30 2024-03-06T07:30:40.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.865+0000 7f58f1ffb640 1 -- 172.21.15.83:0/2300727523 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58e8030070 con 0x7f58dc003f30 2024-03-06T07:30:40.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.865+0000 7f58f1ffb640 1 -- 172.21.15.83:0/2300727523 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e802cd40 con 0x7f58dc003f30 2024-03-06T07:30:40.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.866+0000 7f58f969a640 1 -- 172.21.15.83:0/2300727523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58dc003f30 msgr2=0x7f58dc004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:40.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.866+0000 7f58f969a640 1 --2- 172.21.15.83:0/2300727523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58dc003f30 0x7f58dc004800 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f58e800c210 tx=0x7f58dc005830 comp rx=0 tx=0).stop 2024-03-06T07:30:40.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.866+0000 7f58f37fe640 1 -- 172.21.15.83:0/2300727523 reap_dead start 2024-03-06T07:30:40.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.866+0000 7f58f969a640 1 -- 172.21.15.83:0/2300727523 shutdown_connections 2024-03-06T07:30:40.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.866+0000 7f58f969a640 1 -- 172.21.15.83:0/2300727523 >> 172.21.15.83:0/2300727523 conn(0x7f58f40d3c40 msgr2=0x7f58f40d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:40.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.866+0000 7f58f969a640 1 -- 172.21.15.83:0/2300727523 shutdown_connections 2024-03-06T07:30:40.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.866+0000 7f58f969a640 1 -- 172.21.15.83:0/2300727523 wait complete. 2024-03-06T07:30:40.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.867+0000 7f58f969a640 1 Processor -- start 2024-03-06T07:30:40.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.867+0000 7f58f969a640 1 -- start start 2024-03-06T07:30:40.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.868+0000 7f58f969a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58dc003f30 0x7f58f4157110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:40.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.868+0000 7f58f969a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f58f40a04c0 0x7f58f40a0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:40.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.868+0000 7f58f969a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f58f41f3f60 0x7f58f41f43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:40.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.868+0000 7f58f969a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f40b4dd0 con 0x7f58dc003f30 2024-03-06T07:30:40.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.868+0000 7f58f969a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f40b4550 con 0x7f58f41f3f60 2024-03-06T07:30:40.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.868+0000 7f58f969a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58f40b4f50 con 0x7f58f40a04c0 2024-03-06T07:30:40.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.868+0000 7f58f2ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58dc003f30 0x7f58f4157110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:40.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.868+0000 7f58f2ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58dc003f30 0x7f58f4157110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52742/0 (socket says 172.21.15.83:52742) 2024-03-06T07:30:40.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.868+0000 7f58f2ffd640 1 -- 172.21.15.83:0/2217448148 learned_addr learned my addr 172.21.15.83:0/2217448148 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:40.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.868+0000 7f58f27fc640 1 --2- 172.21.15.83:0/2217448148 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f58f40a04c0 0x7f58f40a0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:40.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.868+0000 7f58f37fe640 1 --2- 172.21.15.83:0/2217448148 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f58f41f3f60 0x7f58f41f43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:40.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.868+0000 7f58f2ffd640 1 -- 172.21.15.83:0/2217448148 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f58f40a04c0 msgr2=0x7f58f40a0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:40.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.868+0000 7f58f2ffd640 1 --2- 172.21.15.83:0/2217448148 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f58f40a04c0 0x7f58f40a0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.868+0000 7f58f2ffd640 1 -- 172.21.15.83:0/2217448148 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f58f41f3f60 msgr2=0x7f58f41f43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:40.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.868+0000 7f58f2ffd640 1 --2- 172.21.15.83:0/2217448148 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f58f41f3f60 0x7f58f41f43e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.868+0000 7f58f2ffd640 1 -- 172.21.15.83:0/2217448148 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58f409e0d0 con 0x7f58dc003f30 2024-03-06T07:30:40.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.868+0000 7f58f2ffd640 1 --2- 172.21.15.83:0/2217448148 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58dc003f30 0x7f58f4157110 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f58e4011600 tx=0x7f58e4011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:40.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.869+0000 7f58db7fe640 1 -- 172.21.15.83:0/2217448148 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e400f920 con 0x7f58dc003f30 2024-03-06T07:30:40.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.869+0000 7f58db7fe640 1 -- 172.21.15.83:0/2217448148 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58e400fac0 con 0x7f58dc003f30 2024-03-06T07:30:40.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.869+0000 7f58f969a640 1 -- 172.21.15.83:0/2217448148 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58f4157650 con 0x7f58dc003f30 2024-03-06T07:30:40.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.869+0000 7f58db7fe640 1 -- 172.21.15.83:0/2217448148 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e400fe90 con 0x7f58dc003f30 2024-03-06T07:30:40.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.869+0000 7f58f969a640 1 -- 172.21.15.83:0/2217448148 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58f40b2610 con 0x7f58dc003f30 2024-03-06T07:30:40.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.870+0000 7f58f969a640 1 -- 172.21.15.83:0/2217448148 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58f4069e80 con 0x7f58dc003f30 2024-03-06T07:30:40.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.870+0000 7f58db7fe640 1 -- 172.21.15.83:0/2217448148 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f58e4015460 con 0x7f58dc003f30 2024-03-06T07:30:40.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.870+0000 7f58db7fe640 1 --2- 172.21.15.83:0/2217448148 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f58e006b6a0 0x7f58e006db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:40.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.870+0000 7f58db7fe640 1 -- 172.21.15.83:0/2217448148 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f58e408ab70 con 0x7f58dc003f30 2024-03-06T07:30:40.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.870+0000 7f58f27fc640 1 --2- 172.21.15.83:0/2217448148 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f58e006b6a0 0x7f58e006db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:40.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.871+0000 7f58f27fc640 1 --2- 172.21.15.83:0/2217448148 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f58e006b6a0 0x7f58e006db60 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f58e8034000 tx=0x7f58e8037040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:40.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:40.876+0000 7f58db7fe640 1 -- 172.21.15.83:0/2217448148 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f58f4069e80 con 0x7f58dc003f30 2024-03-06T07:30:41.042 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:41.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:41.110+0000 7f58f969a640 1 -- 172.21.15.83:0/2217448148 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58f404f330 con 0x7f58dc003f30 2024-03-06T07:30:41.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:41.111+0000 7f58db7fe640 1 -- 172.21.15.83:0/2217448148 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f58f4157650 con 0x7f58dc003f30 2024-03-06T07:30:41.113 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:30:41.113 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:30:41.113 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:30:41.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:41.113+0000 7f58f969a640 1 -- 172.21.15.83:0/2217448148 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f58e006b6a0 msgr2=0x7f58e006db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:41.113+0000 7f58f969a640 1 --2- 172.21.15.83:0/2217448148 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f58e006b6a0 0x7f58e006db60 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f58e8034000 tx=0x7f58e8037040 comp rx=0 tx=0).stop 2024-03-06T07:30:41.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:41.113+0000 7f58f969a640 1 -- 172.21.15.83:0/2217448148 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58dc003f30 msgr2=0x7f58f4157110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:41.113+0000 7f58f969a640 1 --2- 172.21.15.83:0/2217448148 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58dc003f30 0x7f58f4157110 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f58e4011600 tx=0x7f58e4011ac0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:41.113+0000 7f58f969a640 1 -- 172.21.15.83:0/2217448148 shutdown_connections 2024-03-06T07:30:41.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:41.113+0000 7f58f969a640 1 --2- 172.21.15.83:0/2217448148 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f58e006b6a0 0x7f58e006db60 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:41.113+0000 7f58f969a640 1 --2- 172.21.15.83:0/2217448148 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f58f41f3f60 0x7f58f41f43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:41.113+0000 7f58f969a640 1 --2- 172.21.15.83:0/2217448148 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f58f40a04c0 0x7f58f40a0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:41.113+0000 7f58f969a640 1 --2- 172.21.15.83:0/2217448148 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58dc003f30 0x7f58f4157110 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:41.113+0000 7f58f969a640 1 -- 172.21.15.83:0/2217448148 >> 172.21.15.83:0/2217448148 conn(0x7f58f40d3c40 msgr2=0x7f58f406b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:41.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:41.113+0000 7f58f969a640 1 -- 172.21.15.83:0/2217448148 shutdown_connections 2024-03-06T07:30:41.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:41.113+0000 7f58f969a640 1 -- 172.21.15.83:0/2217448148 wait complete. 2024-03-06T07:30:41.327 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:30:41.327 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:30:42.328 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:42.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.428+0000 7f65994b5640 1 Processor -- start 2024-03-06T07:30:42.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.428+0000 7f65994b5640 1 -- start start 2024-03-06T07:30:42.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.429+0000 7f65994b5640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f659413dc10 0x7f65940b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.429+0000 7f65994b5640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f659415bf30 0x7f65940b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.429+0000 7f65994b5640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f659415e850 0x7f659415eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.429+0000 7f65994b5640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65940cf3f0 con 0x7f659413dc10 2024-03-06T07:30:42.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.429+0000 7f65994b5640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f659415b210 con 0x7f659415bf30 2024-03-06T07:30:42.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.429+0000 7f65994b5640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f659415f580 con 0x7f659415e850 2024-03-06T07:30:42.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.429+0000 7f6592ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f659413dc10 0x7f65940b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.429+0000 7f6592ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f659413dc10 0x7f65940b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52746/0 (socket says 172.21.15.83:52746) 2024-03-06T07:30:42.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.429+0000 7f6592ffd640 1 -- 172.21.15.83:0/2290832190 learned_addr learned my addr 172.21.15.83:0/2290832190 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:42.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.430+0000 7f65937fe640 1 --2- 172.21.15.83:0/2290832190 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f659415e850 0x7f659415eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.430+0000 7f658bfff640 1 --2- 172.21.15.83:0/2290832190 >> v2:172.21.15.183:3300/0 conn(0x7f659415bf30 0x7f65940b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.430+0000 7f6592ffd640 1 -- 172.21.15.83:0/2290832190 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f659415e850 msgr2=0x7f659415eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:42.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.430+0000 7f6592ffd640 1 --2- 172.21.15.83:0/2290832190 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f659415e850 0x7f659415eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.430+0000 7f6592ffd640 1 -- 172.21.15.83:0/2290832190 >> v2:172.21.15.183:3300/0 conn(0x7f659415bf30 msgr2=0x7f65940b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:42.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.430+0000 7f6592ffd640 1 --2- 172.21.15.83:0/2290832190 >> v2:172.21.15.183:3300/0 conn(0x7f659415bf30 0x7f65940b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.430+0000 7f6592ffd640 1 -- 172.21.15.83:0/2290832190 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f659415b450 con 0x7f659413dc10 2024-03-06T07:30:42.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.431+0000 7f6592ffd640 1 --2- 172.21.15.83:0/2290832190 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f659413dc10 0x7f65940b2950 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f659406c790 tx=0x7f658402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f1d22de67465e92 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.431+0000 7f65927fc640 1 -- 172.21.15.83:0/2290832190 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f658402e070 con 0x7f659413dc10 2024-03-06T07:30:42.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.431+0000 7f65927fc640 1 -- 172.21.15.83:0/2290832190 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6584037070 con 0x7f659413dc10 2024-03-06T07:30:42.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.431+0000 7f65927fc640 1 -- 172.21.15.83:0/2290832190 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6584031a00 con 0x7f659413dc10 2024-03-06T07:30:42.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.432+0000 7f65994b5640 1 -- 172.21.15.83:0/2290832190 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f659413dc10 msgr2=0x7f65940b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.432+0000 7f65994b5640 1 --2- 172.21.15.83:0/2290832190 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f659413dc10 0x7f65940b2950 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f659406c790 tx=0x7f658402b880 comp rx=0 tx=0).stop 2024-03-06T07:30:42.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.432+0000 7f65994b5640 1 -- 172.21.15.83:0/2290832190 shutdown_connections 2024-03-06T07:30:42.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.432+0000 7f65994b5640 1 --2- 172.21.15.83:0/2290832190 >> v2:172.21.15.183:3300/0 conn(0x7f659415bf30 0x7f65940b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.432+0000 7f65994b5640 1 --2- 172.21.15.83:0/2290832190 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f659415e850 0x7f659415eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.432+0000 7f65994b5640 1 --2- 172.21.15.83:0/2290832190 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f659413dc10 0x7f65940b2950 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.432+0000 7f65994b5640 1 -- 172.21.15.83:0/2290832190 >> 172.21.15.83:0/2290832190 conn(0x7f65940cf8a0 msgr2=0x7f65940cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:42.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.432+0000 7f65994b5640 1 -- 172.21.15.83:0/2290832190 shutdown_connections 2024-03-06T07:30:42.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.432+0000 7f65994b5640 1 -- 172.21.15.83:0/2290832190 wait complete. 2024-03-06T07:30:42.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.433+0000 7f65994b5640 1 Processor -- start 2024-03-06T07:30:42.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.434+0000 7f65994b5640 1 -- start start 2024-03-06T07:30:42.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.435+0000 7f65994b5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f659413dc10 0x7f65941547a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.435+0000 7f65994b5640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f659415bf30 0x7f659413f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.435+0000 7f65994b5640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f659415e850 0x7f659406ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.435+0000 7f65994b5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659415f1e0 con 0x7f659413dc10 2024-03-06T07:30:42.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.435+0000 7f65994b5640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659413dff0 con 0x7f659415e850 2024-03-06T07:30:42.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.435+0000 7f65994b5640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f659415f360 con 0x7f659415bf30 2024-03-06T07:30:42.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.435+0000 7f6592ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f659413dc10 0x7f65941547a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.435+0000 7f6592ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f659413dc10 0x7f65941547a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52756/0 (socket says 172.21.15.83:52756) 2024-03-06T07:30:42.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.435+0000 7f6592ffd640 1 -- 172.21.15.83:0/2965094967 learned_addr learned my addr 172.21.15.83:0/2965094967 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:42.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.435+0000 7f6592ffd640 1 -- 172.21.15.83:0/2965094967 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f659415bf30 msgr2=0x7f659413f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.435+0000 7f6592ffd640 1 --2- 172.21.15.83:0/2965094967 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f659415bf30 0x7f659413f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.435+0000 7f6592ffd640 1 -- 172.21.15.83:0/2965094967 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f659415e850 msgr2=0x7f659406ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.435+0000 7f6592ffd640 1 --2- 172.21.15.83:0/2965094967 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f659415e850 0x7f659406ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.435+0000 7f6592ffd640 1 -- 172.21.15.83:0/2965094967 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f659413a390 con 0x7f659413dc10 2024-03-06T07:30:42.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.435+0000 7f6592ffd640 1 --2- 172.21.15.83:0/2965094967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f659413dc10 0x7f65941547a0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f658402b8c0 tx=0x7f65840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.435+0000 7f658b7fe640 1 -- 172.21.15.83:0/2965094967 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6584032880 con 0x7f659413dc10 2024-03-06T07:30:42.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.435+0000 7f658b7fe640 1 -- 172.21.15.83:0/2965094967 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6584032a40 con 0x7f659413dc10 2024-03-06T07:30:42.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.435+0000 7f658b7fe640 1 -- 172.21.15.83:0/2965094967 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6584032e30 con 0x7f659413dc10 2024-03-06T07:30:42.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.435+0000 7f65994b5640 1 -- 172.21.15.83:0/2965094967 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6594139dd0 con 0x7f659413dc10 2024-03-06T07:30:42.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.436+0000 7f65994b5640 1 -- 172.21.15.83:0/2965094967 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65941415b0 con 0x7f659413dc10 2024-03-06T07:30:42.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.437+0000 7f65994b5640 1 -- 172.21.15.83:0/2965094967 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f659404f330 con 0x7f659413dc10 2024-03-06T07:30:42.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.437+0000 7f658b7fe640 1 -- 172.21.15.83:0/2965094967 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f65840453f0 con 0x7f659413dc10 2024-03-06T07:30:42.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.437+0000 7f658b7fe640 1 --2- 172.21.15.83:0/2965094967 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f657406b4d0 0x7f657406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.437+0000 7f658b7fe640 1 -- 172.21.15.83:0/2965094967 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f65840abfa0 con 0x7f659413dc10 2024-03-06T07:30:42.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.437+0000 7f658bfff640 1 --2- 172.21.15.83:0/2965094967 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f657406b4d0 0x7f657406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.438+0000 7f658bfff640 1 --2- 172.21.15.83:0/2965094967 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f657406b4d0 0x7f657406d990 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f65940691b0 tx=0x7f657c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.442+0000 7f658b7fe640 1 -- 172.21.15.83:0/2965094967 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6584040070 con 0x7f659413dc10 2024-03-06T07:30:42.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.655+0000 7f65994b5640 1 -- 172.21.15.83:0/2965094967 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f659406eb00 con 0x7f659413dc10 2024-03-06T07:30:42.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.655+0000 7f658b7fe640 1 -- 172.21.15.83:0/2965094967 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f658407af90 con 0x7f659413dc10 2024-03-06T07:30:42.657 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:30:42.658 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:30:42.658 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:30:42.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.657+0000 7f65994b5640 1 -- 172.21.15.83:0/2965094967 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f657406b4d0 msgr2=0x7f657406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.657+0000 7f65994b5640 1 --2- 172.21.15.83:0/2965094967 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f657406b4d0 0x7f657406d990 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f65940691b0 tx=0x7f657c008040 comp rx=0 tx=0).stop 2024-03-06T07:30:42.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.657+0000 7f65994b5640 1 -- 172.21.15.83:0/2965094967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f659413dc10 msgr2=0x7f65941547a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.657+0000 7f65994b5640 1 --2- 172.21.15.83:0/2965094967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f659413dc10 0x7f65941547a0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f658402b8c0 tx=0x7f65840046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.657+0000 7f65994b5640 1 -- 172.21.15.83:0/2965094967 shutdown_connections 2024-03-06T07:30:42.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.657+0000 7f65994b5640 1 --2- 172.21.15.83:0/2965094967 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f657406b4d0 0x7f657406d990 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.657+0000 7f65994b5640 1 --2- 172.21.15.83:0/2965094967 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f659415e850 0x7f659406ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.657+0000 7f65994b5640 1 --2- 172.21.15.83:0/2965094967 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f659415bf30 0x7f659413f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.657+0000 7f65994b5640 1 --2- 172.21.15.83:0/2965094967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f659413dc10 0x7f65941547a0 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.657+0000 7f65994b5640 1 -- 172.21.15.83:0/2965094967 >> 172.21.15.83:0/2965094967 conn(0x7f65940cf8a0 msgr2=0x7f65940d0140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:42.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.657+0000 7f65994b5640 1 -- 172.21.15.83:0/2965094967 shutdown_connections 2024-03-06T07:30:42.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:42.657+0000 7f65994b5640 1 -- 172.21.15.83:0/2965094967 wait complete. 2024-03-06T07:30:42.871 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:30:42.871 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:30:43.044 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:43.872 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:43.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.977+0000 7fa2d48ec640 1 Processor -- start 2024-03-06T07:30:43.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.977+0000 7fa2d48ec640 1 -- start start 2024-03-06T07:30:43.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.978+0000 7fa2d48ec640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa2d00d22b0 0x7fa2d00d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.978+0000 7fa2d48ec640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa2d00b41f0 0x7fa2d00b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.978+0000 7fa2d48ec640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2d00b3430 con 0x7fa2d015dfe0 2024-03-06T07:30:43.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.978+0000 7fa2d48ec640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2d00d2c50 con 0x7fa2d00d22b0 2024-03-06T07:30:43.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.978+0000 7fa2d48ec640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d00b31e0 con 0x7fa2d00b41f0 2024-03-06T07:30:43.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.978+0000 7fa2c7fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa2d00d22b0 0x7fa2d00d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.978+0000 7fa2ce59c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fa2d015dfe0 0x7fa2d00cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49348/0 (socket says 172.21.15.83:49348) 2024-03-06T07:30:43.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.978+0000 7fa2c7fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa2d00d22b0 0x7fa2d00d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52768/0 (socket says 172.21.15.83:52768) 2024-03-06T07:30:43.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.978+0000 7fa2ce59c640 1 -- 172.21.15.83:0/1557058420 learned_addr learned my addr 172.21.15.83:0/1557058420 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:30:43.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.978+0000 7fa2ced9d640 1 --2- 172.21.15.83:0/1557058420 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa2d00b41f0 0x7fa2d00b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.978+0000 7fa2c7fff640 1 -- 172.21.15.83:0/1557058420 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa2d00b41f0 msgr2=0x7fa2d00b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:43.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.978+0000 7fa2c7fff640 1 --2- 172.21.15.83:0/1557058420 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa2d00b41f0 0x7fa2d00b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.978+0000 7fa2c7fff640 1 -- 172.21.15.83:0/1557058420 >> v1:172.21.15.83:6789/0 conn(0x7fa2d015dfe0 legacy=0x7fa2d00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.978+0000 7fa2cdd9b640 1 -- 172.21.15.83:0/1557058420 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 344255220 0 0) 0x7fa2d00b3430 con 0x7fa2d015dfe0 2024-03-06T07:30:43.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.978+0000 7fa2c7fff640 1 -- 172.21.15.83:0/1557058420 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d00b3710 con 0x7fa2d00d22b0 2024-03-06T07:30:43.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.979+0000 7fa2c7fff640 1 --2- 172.21.15.83:0/1557058420 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2d00d22b0 0x7fa2d00d2710 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7fa2b80097a0 tx=0x7fa2b802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0ad596106dbdf94 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:43.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.979+0000 7fa2cdd9b640 1 -- 172.21.15.83:0/1557058420 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b8035020 con 0x7fa2d00d22b0 2024-03-06T07:30:43.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.979+0000 7fa2cdd9b640 1 -- 172.21.15.83:0/1557058420 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2b8031830 con 0x7fa2d00d22b0 2024-03-06T07:30:43.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.979+0000 7fa2cdd9b640 1 -- 172.21.15.83:0/1557058420 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b8031c00 con 0x7fa2d00d22b0 2024-03-06T07:30:43.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.980+0000 7fa2d48ec640 1 -- 172.21.15.83:0/1557058420 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2d00d22b0 msgr2=0x7fa2d00d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.980+0000 7fa2d48ec640 1 --2- 172.21.15.83:0/1557058420 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2d00d22b0 0x7fa2d00d2710 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7fa2b80097a0 tx=0x7fa2b802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.980+0000 7fa2d48ec640 1 -- 172.21.15.83:0/1557058420 shutdown_connections 2024-03-06T07:30:43.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.980+0000 7fa2d48ec640 1 --2- 172.21.15.83:0/1557058420 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa2d00b41f0 0x7fa2d00b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.980+0000 7fa2d48ec640 1 --2- 172.21.15.83:0/1557058420 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2d00d22b0 0x7fa2d00d2710 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.980+0000 7fa2d48ec640 1 -- 172.21.15.83:0/1557058420 >> 172.21.15.83:0/1557058420 conn(0x7fa2d00d3cd0 msgr2=0x7fa2d00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:43.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.980+0000 7fa2d48ec640 1 -- 172.21.15.83:0/1557058420 shutdown_connections 2024-03-06T07:30:43.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.980+0000 7fa2d48ec640 1 -- 172.21.15.83:0/1557058420 wait complete. 2024-03-06T07:30:43.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.981+0000 7fa2d48ec640 1 Processor -- start 2024-03-06T07:30:43.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.981+0000 7fa2d48ec640 1 -- start start 2024-03-06T07:30:43.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.982+0000 7fa2d48ec640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa2d00b41f0 0x7fa2d009ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.982+0000 7fa2d48ec640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa2d00d22b0 0x7fa2d009f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.982+0000 7fa2d48ec640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2d015dfe0 0x7fa2d00cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.982+0000 7fa2d48ec640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d00cfa00 con 0x7fa2d015dfe0 2024-03-06T07:30:43.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.982+0000 7fa2d48ec640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d00b3d60 con 0x7fa2d00d22b0 2024-03-06T07:30:43.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.982+0000 7fa2d48ec640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d00d1c70 con 0x7fa2d00b41f0 2024-03-06T07:30:43.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.982+0000 7fa2ced9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2d015dfe0 0x7fa2d00cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.982+0000 7fa2ced9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2d015dfe0 0x7fa2d00cbdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52782/0 (socket says 172.21.15.83:52782) 2024-03-06T07:30:43.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.982+0000 7fa2ced9d640 1 -- 172.21.15.83:0/3407965156 learned_addr learned my addr 172.21.15.83:0/3407965156 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:43.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.982+0000 7fa2ce59c640 1 --2- 172.21.15.83:0/3407965156 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa2d00b41f0 0x7fa2d009ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.982+0000 7fa2c7fff640 1 --2- 172.21.15.83:0/3407965156 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa2d00d22b0 0x7fa2d009f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.982+0000 7fa2ced9d640 1 -- 172.21.15.83:0/3407965156 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa2d00b41f0 msgr2=0x7fa2d009ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.982+0000 7fa2ced9d640 1 --2- 172.21.15.83:0/3407965156 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa2d00b41f0 0x7fa2d009ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.982+0000 7fa2ced9d640 1 -- 172.21.15.83:0/3407965156 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa2d00d22b0 msgr2=0x7fa2d009f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.982+0000 7fa2ced9d640 1 --2- 172.21.15.83:0/3407965156 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa2d00d22b0 0x7fa2d009f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.982+0000 7fa2ced9d640 1 -- 172.21.15.83:0/3407965156 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d009fa30 con 0x7fa2d015dfe0 2024-03-06T07:30:43.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.982+0000 7fa2ced9d640 1 --2- 172.21.15.83:0/3407965156 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2d015dfe0 0x7fa2d00cbdf0 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8007e40 tx=0x7fa2c800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:43.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.982+0000 7fa2c6ffd640 1 -- 172.21.15.83:0/3407965156 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c8014620 con 0x7fa2d015dfe0 2024-03-06T07:30:43.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.982+0000 7fa2c6ffd640 1 -- 172.21.15.83:0/3407965156 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2c8019070 con 0x7fa2d015dfe0 2024-03-06T07:30:43.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.983+0000 7fa2d48ec640 1 -- 172.21.15.83:0/3407965156 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2d009fd10 con 0x7fa2d015dfe0 2024-03-06T07:30:43.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.983+0000 7fa2c6ffd640 1 -- 172.21.15.83:0/3407965156 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c8014a20 con 0x7fa2d015dfe0 2024-03-06T07:30:43.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.983+0000 7fa2d48ec640 1 -- 172.21.15.83:0/3407965156 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2d00a02d0 con 0x7fa2d015dfe0 2024-03-06T07:30:43.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.984+0000 7fa2d48ec640 1 -- 172.21.15.83:0/3407965156 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2d004f330 con 0x7fa2d015dfe0 2024-03-06T07:30:43.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.984+0000 7fa2c6ffd640 1 -- 172.21.15.83:0/3407965156 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa2c8014bc0 con 0x7fa2d015dfe0 2024-03-06T07:30:43.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.984+0000 7fa2c6ffd640 1 --2- 172.21.15.83:0/3407965156 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa2b006b4d0 0x7fa2b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.984+0000 7fa2c6ffd640 1 -- 172.21.15.83:0/3407965156 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa2c808aa00 con 0x7fa2d015dfe0 2024-03-06T07:30:43.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.984+0000 7fa2ce59c640 1 --2- 172.21.15.83:0/3407965156 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa2b006b4d0 0x7fa2b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.985+0000 7fa2ce59c640 1 --2- 172.21.15.83:0/3407965156 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa2b006b4d0 0x7fa2b006d990 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fa2c0007fd0 tx=0x7fa2c0007de0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:43.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:43.989+0000 7fa2c6ffd640 1 -- 172.21.15.83:0/3407965156 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa2c80591f0 con 0x7fa2d015dfe0 2024-03-06T07:30:44.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:44.225+0000 7fa2d48ec640 1 -- 172.21.15.83:0/3407965156 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2d00b2c70 con 0x7fa2d015dfe0 2024-03-06T07:30:44.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:44.225+0000 7fa2c6ffd640 1 -- 172.21.15.83:0/3407965156 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7fa2c805f070 con 0x7fa2d015dfe0 2024-03-06T07:30:44.227 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:30:44.228 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:30:44.228 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:30:44.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:44.228+0000 7fa2d48ec640 1 -- 172.21.15.83:0/3407965156 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa2b006b4d0 msgr2=0x7fa2b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:44.228+0000 7fa2d48ec640 1 --2- 172.21.15.83:0/3407965156 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa2b006b4d0 0x7fa2b006d990 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fa2c0007fd0 tx=0x7fa2c0007de0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:44.228+0000 7fa2d48ec640 1 -- 172.21.15.83:0/3407965156 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2d015dfe0 msgr2=0x7fa2d00cbdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:44.228+0000 7fa2d48ec640 1 --2- 172.21.15.83:0/3407965156 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2d015dfe0 0x7fa2d00cbdf0 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8007e40 tx=0x7fa2c800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:44.228+0000 7fa2d48ec640 1 -- 172.21.15.83:0/3407965156 shutdown_connections 2024-03-06T07:30:44.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:44.228+0000 7fa2d48ec640 1 --2- 172.21.15.83:0/3407965156 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa2b006b4d0 0x7fa2b006d990 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:44.228+0000 7fa2d48ec640 1 --2- 172.21.15.83:0/3407965156 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa2d015dfe0 0x7fa2d00cbdf0 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:44.228+0000 7fa2d48ec640 1 --2- 172.21.15.83:0/3407965156 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa2d00d22b0 0x7fa2d009f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:44.228+0000 7fa2d48ec640 1 --2- 172.21.15.83:0/3407965156 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa2d00b41f0 0x7fa2d009ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:44.228+0000 7fa2d48ec640 1 -- 172.21.15.83:0/3407965156 >> 172.21.15.83:0/3407965156 conn(0x7fa2d00d3cd0 msgr2=0x7fa2d006c940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:44.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:44.228+0000 7fa2d48ec640 1 -- 172.21.15.83:0/3407965156 shutdown_connections 2024-03-06T07:30:44.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:44.228+0000 7fa2d48ec640 1 -- 172.21.15.83:0/3407965156 wait complete. 2024-03-06T07:30:44.442 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:30:44.442 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:30:45.046 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:45.444 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:45.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.573+0000 7f2a15f93640 1 Processor -- start 2024-03-06T07:30:45.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.573+0000 7f2a15f93640 1 -- start start 2024-03-06T07:30:45.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.573+0000 7f2a15f93640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2a100d22b0 0x7f2a100d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.573+0000 7f2a15f93640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f2a100b41f0 0x7f2a100b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.573+0000 7f2a15f93640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a100b3430 con 0x7f2a1015dfe0 2024-03-06T07:30:45.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.573+0000 7f2a15f93640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a100d2c50 con 0x7f2a100d22b0 2024-03-06T07:30:45.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.573+0000 7f2a15f93640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a100b31e0 con 0x7f2a100b41f0 2024-03-06T07:30:45.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.574+0000 7f2a0effd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2a100d22b0 0x7f2a100d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.574+0000 7f2a0effd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2a100d22b0 0x7f2a100d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52788/0 (socket says 172.21.15.83:52788) 2024-03-06T07:30:45.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.574+0000 7f2a0effd640 1 -- 172.21.15.83:0/3260379274 learned_addr learned my addr 172.21.15.83:0/3260379274 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:45.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.574+0000 7f2a0ffff640 1 --2- 172.21.15.83:0/3260379274 >> v2:172.21.15.183:3300/0 conn(0x7f2a100b41f0 0x7f2a100b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.574+0000 7f2a0effd640 1 -- 172.21.15.83:0/3260379274 >> v2:172.21.15.183:3300/0 conn(0x7f2a100b41f0 msgr2=0x7f2a100b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:45.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.574+0000 7f2a0effd640 1 --2- 172.21.15.83:0/3260379274 >> v2:172.21.15.183:3300/0 conn(0x7f2a100b41f0 0x7f2a100b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.574+0000 7f2a0effd640 1 -- 172.21.15.83:0/3260379274 >> v1:172.21.15.183:6789/0 conn(0x7f2a1015dfe0 legacy=0x7f2a100cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:45.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.574+0000 7f2a0effd640 1 -- 172.21.15.83:0/3260379274 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a100b3710 con 0x7f2a100d22b0 2024-03-06T07:30:45.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.575+0000 7f2a0effd640 1 --2- 172.21.15.83:0/3260379274 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a100d22b0 0x7f2a100d2710 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f29f8009870 tx=0x7f29f802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51851e1f2ee5ec2d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:45.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.575+0000 7f2a0e7fc640 1 -- 172.21.15.83:0/3260379274 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f8034070 con 0x7f2a100d22b0 2024-03-06T07:30:45.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.575+0000 7f2a0e7fc640 1 -- 172.21.15.83:0/3260379274 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29f8030860 con 0x7f2a100d22b0 2024-03-06T07:30:45.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.575+0000 7f2a0e7fc640 1 -- 172.21.15.83:0/3260379274 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f8030c30 con 0x7f2a100d22b0 2024-03-06T07:30:45.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.575+0000 7f2a15f93640 1 -- 172.21.15.83:0/3260379274 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a100d22b0 msgr2=0x7f2a100d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.575+0000 7f2a15f93640 1 --2- 172.21.15.83:0/3260379274 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a100d22b0 0x7f2a100d2710 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f29f8009870 tx=0x7f29f802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.575+0000 7f2a15f93640 1 -- 172.21.15.83:0/3260379274 shutdown_connections 2024-03-06T07:30:45.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.575+0000 7f2a15f93640 1 --2- 172.21.15.83:0/3260379274 >> v2:172.21.15.183:3300/0 conn(0x7f2a100b41f0 0x7f2a100b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.575+0000 7f2a15f93640 1 --2- 172.21.15.83:0/3260379274 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a100d22b0 0x7f2a100d2710 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.575+0000 7f2a15f93640 1 -- 172.21.15.83:0/3260379274 >> 172.21.15.83:0/3260379274 conn(0x7f2a100d3cd0 msgr2=0x7f2a100d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:45.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.575+0000 7f2a15f93640 1 -- 172.21.15.83:0/3260379274 shutdown_connections 2024-03-06T07:30:45.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.575+0000 7f2a15f93640 1 -- 172.21.15.83:0/3260379274 wait complete. 2024-03-06T07:30:45.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.576+0000 7f2a15f93640 1 Processor -- start 2024-03-06T07:30:45.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.577+0000 7f2a15f93640 1 -- start start 2024-03-06T07:30:45.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.577+0000 7f2a15f93640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2a100b41f0 0x7f2a1014b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.577+0000 7f2a15f93640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a100d22b0 0x7f2a101f6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.577+0000 7f2a15f93640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2a1015dfe0 0x7f2a10160320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.577+0000 7f2a15f93640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a100cfa00 con 0x7f2a100d22b0 2024-03-06T07:30:45.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.577+0000 7f2a15f93640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a100b3d60 con 0x7f2a100b41f0 2024-03-06T07:30:45.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.577+0000 7f2a15f93640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a100d1c70 con 0x7f2a1015dfe0 2024-03-06T07:30:45.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.577+0000 7f2a0effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a100d22b0 0x7f2a101f6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.578+0000 7f2a0effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a100d22b0 0x7f2a101f6a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52792/0 (socket says 172.21.15.83:52792) 2024-03-06T07:30:45.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.578+0000 7f2a0effd640 1 -- 172.21.15.83:0/2786505799 learned_addr learned my addr 172.21.15.83:0/2786505799 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:45.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.578+0000 7f2a0f7fe640 1 --2- 172.21.15.83:0/2786505799 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2a100b41f0 0x7f2a1014b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.578+0000 7f2a0ffff640 1 --2- 172.21.15.83:0/2786505799 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2a1015dfe0 0x7f2a10160320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.578+0000 7f2a0effd640 1 -- 172.21.15.83:0/2786505799 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2a1015dfe0 msgr2=0x7f2a10160320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.578+0000 7f2a0effd640 1 --2- 172.21.15.83:0/2786505799 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2a1015dfe0 0x7f2a10160320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.578+0000 7f2a0effd640 1 -- 172.21.15.83:0/2786505799 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2a100b41f0 msgr2=0x7f2a1014b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.578+0000 7f2a0effd640 1 --2- 172.21.15.83:0/2786505799 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2a100b41f0 0x7f2a1014b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.578+0000 7f2a0effd640 1 -- 172.21.15.83:0/2786505799 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a1015a640 con 0x7f2a100d22b0 2024-03-06T07:30:45.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.578+0000 7f2a0effd640 1 --2- 172.21.15.83:0/2786505799 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a100d22b0 0x7f2a101f6a30 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f29f8000c00 tx=0x7f29f8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:45.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.578+0000 7f2a07fff640 1 -- 172.21.15.83:0/2786505799 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f8048070 con 0x7f2a100d22b0 2024-03-06T07:30:45.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.578+0000 7f2a07fff640 1 -- 172.21.15.83:0/2786505799 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29f8030660 con 0x7f2a100d22b0 2024-03-06T07:30:45.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.578+0000 7f2a07fff640 1 -- 172.21.15.83:0/2786505799 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f8034040 con 0x7f2a100d22b0 2024-03-06T07:30:45.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.578+0000 7f2a15f93640 1 -- 172.21.15.83:0/2786505799 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a100cae40 con 0x7f2a100d22b0 2024-03-06T07:30:45.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.578+0000 7f2a15f93640 1 -- 172.21.15.83:0/2786505799 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a1015fdc0 con 0x7f2a100d22b0 2024-03-06T07:30:45.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.579+0000 7f2a15f93640 1 -- 172.21.15.83:0/2786505799 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a1004f330 con 0x7f2a100d22b0 2024-03-06T07:30:45.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.580+0000 7f2a07fff640 1 -- 172.21.15.83:0/2786505799 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f29f804b070 con 0x7f2a100d22b0 2024-03-06T07:30:45.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.580+0000 7f2a07fff640 1 --2- 172.21.15.83:0/2786505799 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f29f006b500 0x7f29f006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.581+0000 7f2a07fff640 1 -- 172.21.15.83:0/2786505799 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f29f80ac090 con 0x7f2a100d22b0 2024-03-06T07:30:45.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.581+0000 7f2a0f7fe640 1 --2- 172.21.15.83:0/2786505799 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f29f006b500 0x7f29f006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.581+0000 7f2a0f7fe640 1 --2- 172.21.15.83:0/2786505799 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f29f006b500 0x7f29f006d9c0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f2a100cf6a0 tx=0x7f2a08009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:45.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.589+0000 7f2a07fff640 1 -- 172.21.15.83:0/2786505799 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f29f8030830 con 0x7f2a100d22b0 2024-03-06T07:30:45.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.800+0000 7f2a15f93640 1 -- 172.21.15.83:0/2786505799 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a100cb1d0 con 0x7f2a100d22b0 2024-03-06T07:30:45.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.800+0000 7f2a07fff640 1 -- 172.21.15.83:0/2786505799 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f29f807aa30 con 0x7f2a100d22b0 2024-03-06T07:30:45.802 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:30:45.802 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:30:45.803 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:30:45.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.802+0000 7f2a15f93640 1 -- 172.21.15.83:0/2786505799 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f29f006b500 msgr2=0x7f29f006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.802+0000 7f2a15f93640 1 --2- 172.21.15.83:0/2786505799 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f29f006b500 0x7f29f006d9c0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f2a100cf6a0 tx=0x7f2a08009040 comp rx=0 tx=0).stop 2024-03-06T07:30:45.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.802+0000 7f2a15f93640 1 -- 172.21.15.83:0/2786505799 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a100d22b0 msgr2=0x7f2a101f6a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.802+0000 7f2a15f93640 1 --2- 172.21.15.83:0/2786505799 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a100d22b0 0x7f2a101f6a30 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f29f8000c00 tx=0x7f29f8004730 comp rx=0 tx=0).stop 2024-03-06T07:30:45.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.802+0000 7f2a15f93640 1 -- 172.21.15.83:0/2786505799 shutdown_connections 2024-03-06T07:30:45.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.802+0000 7f2a15f93640 1 --2- 172.21.15.83:0/2786505799 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f29f006b500 0x7f29f006d9c0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.802+0000 7f2a15f93640 1 --2- 172.21.15.83:0/2786505799 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2a1015dfe0 0x7f2a10160320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.802+0000 7f2a15f93640 1 --2- 172.21.15.83:0/2786505799 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a100d22b0 0x7f2a101f6a30 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.802+0000 7f2a15f93640 1 --2- 172.21.15.83:0/2786505799 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2a100b41f0 0x7f2a1014b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.802+0000 7f2a15f93640 1 -- 172.21.15.83:0/2786505799 >> 172.21.15.83:0/2786505799 conn(0x7f2a100d3cd0 msgr2=0x7f2a100d08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:45.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.802+0000 7f2a15f93640 1 -- 172.21.15.83:0/2786505799 shutdown_connections 2024-03-06T07:30:45.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:45.802+0000 7f2a15f93640 1 -- 172.21.15.83:0/2786505799 wait complete. 2024-03-06T07:30:46.020 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:30:46.020 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:30:47.021 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:47.048 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:47.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.121+0000 7f70109f7640 1 Processor -- start 2024-03-06T07:30:47.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.121+0000 7f70109f7640 1 -- start start 2024-03-06T07:30:47.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.122+0000 7f70109f7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f700c15bb20 0x7f700c0d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.122+0000 7f70109f7640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f700c0cfff0 0x7f700c0d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.122+0000 7f70109f7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f700c161980 con 0x7f700c0ce4b0 2024-03-06T07:30:47.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.122+0000 7f70109f7640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f700c161c60 con 0x7f700c15bb20 2024-03-06T07:30:47.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.122+0000 7f70109f7640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f700c161f40 con 0x7f700c0cfff0 2024-03-06T07:30:47.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.122+0000 7f700a59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f700c15bb20 0x7f700c0d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.122+0000 7f700a59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f700c15bb20 0x7f700c0d1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:42188/0 (socket says 172.21.15.83:42188) 2024-03-06T07:30:47.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.122+0000 7f700ad9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f700c0ce4b0 0x7f700c0ce8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43258/0 (socket says 172.21.15.83:43258) 2024-03-06T07:30:47.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.122+0000 7f700a59c640 1 -- 172.21.15.83:0/559015264 learned_addr learned my addr 172.21.15.83:0/559015264 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:47.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.123+0000 7f700a59c640 1 -- 172.21.15.83:0/559015264 >> v2:172.21.15.183:3300/0 conn(0x7f700c0cfff0 msgr2=0x7f700c0d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:47.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.123+0000 7f700a59c640 1 --2- 172.21.15.83:0/559015264 >> v2:172.21.15.183:3300/0 conn(0x7f700c0cfff0 0x7f700c0d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.123+0000 7f700a59c640 1 -- 172.21.15.83:0/559015264 >> v1:172.21.15.83:6789/0 conn(0x7f700c0ce4b0 legacy=0x7f700c0ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.123+0000 7f700a59c640 1 -- 172.21.15.83:0/559015264 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f700c162220 con 0x7f700c15bb20 2024-03-06T07:30:47.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.123+0000 7f700a59c640 1 --2- 172.21.15.83:0/559015264 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f700c15bb20 0x7f700c0d1df0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f700c071290 tx=0x7f6ffc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a946a7421628bb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:47.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.123+0000 7f700959a640 1 -- 172.21.15.83:0/559015264 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ffc034070 con 0x7f700c15bb20 2024-03-06T07:30:47.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.124+0000 7f700959a640 1 -- 172.21.15.83:0/559015264 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ffc030860 con 0x7f700c15bb20 2024-03-06T07:30:47.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.124+0000 7f700959a640 1 -- 172.21.15.83:0/559015264 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ffc030c30 con 0x7f700c15bb20 2024-03-06T07:30:47.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.124+0000 7f70109f7640 1 -- 172.21.15.83:0/559015264 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f700c15bb20 msgr2=0x7f700c0d1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.124+0000 7f70109f7640 1 --2- 172.21.15.83:0/559015264 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f700c15bb20 0x7f700c0d1df0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f700c071290 tx=0x7f6ffc02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.124+0000 7f70109f7640 1 -- 172.21.15.83:0/559015264 shutdown_connections 2024-03-06T07:30:47.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.124+0000 7f70109f7640 1 --2- 172.21.15.83:0/559015264 >> v2:172.21.15.183:3300/0 conn(0x7f700c0cfff0 0x7f700c0d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.124+0000 7f70109f7640 1 --2- 172.21.15.83:0/559015264 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f700c15bb20 0x7f700c0d1df0 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.124+0000 7f70109f7640 1 -- 172.21.15.83:0/559015264 >> 172.21.15.83:0/559015264 conn(0x7f700c09a9f0 msgr2=0x7f700c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:47.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.124+0000 7f70109f7640 1 -- 172.21.15.83:0/559015264 shutdown_connections 2024-03-06T07:30:47.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.124+0000 7f70109f7640 1 -- 172.21.15.83:0/559015264 wait complete. 2024-03-06T07:30:47.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.125+0000 7f70109f7640 1 Processor -- start 2024-03-06T07:30:47.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.125+0000 7f70109f7640 1 -- start start 2024-03-06T07:30:47.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.126+0000 7f70109f7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f700c0ce4b0 0x7f700c160750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.126+0000 7f70109f7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f700c0cfff0 0x7f700c0c7c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.126+0000 7f70109f7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f700c15bb20 0x7f700c15d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.126+0000 7f70109f7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f700c0d24b0 con 0x7f700c15bb20 2024-03-06T07:30:47.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.126+0000 7f70109f7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f700c0d2330 con 0x7f700c0ce4b0 2024-03-06T07:30:47.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.126+0000 7f70109f7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f700c0d1980 con 0x7f700c0cfff0 2024-03-06T07:30:47.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.126+0000 7f700ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f700c15bb20 0x7f700c15d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.126+0000 7f700ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f700c15bb20 0x7f700c15d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:42194/0 (socket says 172.21.15.83:42194) 2024-03-06T07:30:47.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.126+0000 7f700ad9d640 1 -- 172.21.15.83:0/2433162608 learned_addr learned my addr 172.21.15.83:0/2433162608 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:47.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.126+0000 7f700a59c640 1 --2- 172.21.15.83:0/2433162608 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f700c0ce4b0 0x7f700c160750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.126+0000 7f7009d9b640 1 --2- 172.21.15.83:0/2433162608 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f700c0cfff0 0x7f700c0c7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.126+0000 7f700ad9d640 1 -- 172.21.15.83:0/2433162608 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f700c0cfff0 msgr2=0x7f700c0c7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.126+0000 7f700ad9d640 1 --2- 172.21.15.83:0/2433162608 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f700c0cfff0 0x7f700c0c7c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.126+0000 7f700ad9d640 1 -- 172.21.15.83:0/2433162608 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f700c0ce4b0 msgr2=0x7f700c160750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.126+0000 7f700ad9d640 1 --2- 172.21.15.83:0/2433162608 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f700c0ce4b0 0x7f700c160750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.126+0000 7f700ad9d640 1 -- 172.21.15.83:0/2433162608 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f700c1f4190 con 0x7f700c15bb20 2024-03-06T07:30:47.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.127+0000 7f700ad9d640 1 --2- 172.21.15.83:0/2433162608 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f700c15bb20 0x7f700c15d2f0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f700400c7b0 tx=0x7f700400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:47.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.127+0000 7f6ffaffd640 1 -- 172.21.15.83:0/2433162608 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7004017070 con 0x7f700c15bb20 2024-03-06T07:30:47.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.127+0000 7f6ffaffd640 1 -- 172.21.15.83:0/2433162608 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f700400a640 con 0x7f700c15bb20 2024-03-06T07:30:47.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.127+0000 7f70109f7640 1 -- 172.21.15.83:0/2433162608 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f700c1f4470 con 0x7f700c15bb20 2024-03-06T07:30:47.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.127+0000 7f6ffaffd640 1 -- 172.21.15.83:0/2433162608 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700400a9f0 con 0x7f700c15bb20 2024-03-06T07:30:47.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.127+0000 7f70109f7640 1 -- 172.21.15.83:0/2433162608 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f700c1f4a30 con 0x7f700c15bb20 2024-03-06T07:30:47.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.128+0000 7f70109f7640 1 -- 172.21.15.83:0/2433162608 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f700c04f330 con 0x7f700c15bb20 2024-03-06T07:30:47.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.128+0000 7f6ffaffd640 1 -- 172.21.15.83:0/2433162608 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f700401a070 con 0x7f700c15bb20 2024-03-06T07:30:47.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.129+0000 7f6ffaffd640 1 --2- 172.21.15.83:0/2433162608 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6fec06b4d0 0x7f6fec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.129+0000 7f6ffaffd640 1 -- 172.21.15.83:0/2433162608 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7004089710 con 0x7f700c15bb20 2024-03-06T07:30:47.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.129+0000 7f700a59c640 1 --2- 172.21.15.83:0/2433162608 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6fec06b4d0 0x7f6fec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.129+0000 7f700a59c640 1 --2- 172.21.15.83:0/2433162608 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6fec06b4d0 0x7f6fec06d990 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc036000 tx=0x7f6ffc03b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:47.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.133+0000 7f6ffaffd640 1 -- 172.21.15.83:0/2433162608 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7004057f90 con 0x7f700c15bb20 2024-03-06T07:30:47.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.345+0000 7f70109f7640 1 -- 172.21.15.83:0/2433162608 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f700c15a760 con 0x7f700c15bb20 2024-03-06T07:30:47.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.345+0000 7f6ffaffd640 1 -- 172.21.15.83:0/2433162608 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f700405c030 con 0x7f700c15bb20 2024-03-06T07:30:47.347 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:30:47.347 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:30:47.347 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:30:47.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.347+0000 7f70109f7640 1 -- 172.21.15.83:0/2433162608 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6fec06b4d0 msgr2=0x7f6fec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.347+0000 7f70109f7640 1 --2- 172.21.15.83:0/2433162608 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6fec06b4d0 0x7f6fec06d990 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc036000 tx=0x7f6ffc03b040 comp rx=0 tx=0).stop 2024-03-06T07:30:47.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.347+0000 7f70109f7640 1 -- 172.21.15.83:0/2433162608 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f700c15bb20 msgr2=0x7f700c15d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.347+0000 7f70109f7640 1 --2- 172.21.15.83:0/2433162608 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f700c15bb20 0x7f700c15d2f0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f700400c7b0 tx=0x7f700400cc70 comp rx=0 tx=0).stop 2024-03-06T07:30:47.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.347+0000 7f70109f7640 1 -- 172.21.15.83:0/2433162608 shutdown_connections 2024-03-06T07:30:47.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.347+0000 7f70109f7640 1 --2- 172.21.15.83:0/2433162608 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f6fec06b4d0 0x7f6fec06d990 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.347+0000 7f70109f7640 1 --2- 172.21.15.83:0/2433162608 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f700c15bb20 0x7f700c15d2f0 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.347+0000 7f70109f7640 1 --2- 172.21.15.83:0/2433162608 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f700c0cfff0 0x7f700c0c7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.347+0000 7f70109f7640 1 --2- 172.21.15.83:0/2433162608 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f700c0ce4b0 0x7f700c160750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.347+0000 7f70109f7640 1 -- 172.21.15.83:0/2433162608 >> 172.21.15.83:0/2433162608 conn(0x7f700c09a9f0 msgr2=0x7f700c06f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:47.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.347+0000 7f70109f7640 1 -- 172.21.15.83:0/2433162608 shutdown_connections 2024-03-06T07:30:47.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:47.347+0000 7f70109f7640 1 -- 172.21.15.83:0/2433162608 wait complete. 2024-03-06T07:30:47.561 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:30:47.561 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:30:48.562 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:48.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.661+0000 7fc33efd7640 1 Processor -- start 2024-03-06T07:30:48.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.661+0000 7fc33efd7640 1 -- start start 2024-03-06T07:30:48.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.662+0000 7fc33efd7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc3380b4170 0x7fc3380b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.662+0000 7fc33efd7640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fc3380d4760 0x7fc3380d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.662+0000 7fc33efd7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc3380b4dd0 0x7fc3380d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.662+0000 7fc33efd7640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3380d4bf0 con 0x7fc3380b4170 2024-03-06T07:30:48.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.662+0000 7fc33efd7640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3380ce9e0 con 0x7fc3380d4760 2024-03-06T07:30:48.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.662+0000 7fc33efd7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3380d25d0 con 0x7fc3380b4dd0 2024-03-06T07:30:48.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.662+0000 7fc33cd73640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc3380b4170 0x7fc3380b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.662+0000 7fc33cd73640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc3380b4170 0x7fc3380b6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:42198/0 (socket says 172.21.15.83:42198) 2024-03-06T07:30:48.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.662+0000 7fc33cd73640 1 -- 172.21.15.83:0/69097832 learned_addr learned my addr 172.21.15.83:0/69097832 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:48.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.663+0000 7fc33d574640 1 --2- 172.21.15.83:0/69097832 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc3380b4dd0 0x7fc3380d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.663+0000 7fc33cd73640 1 -- 172.21.15.83:0/69097832 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc3380b4dd0 msgr2=0x7fc3380d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:48.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.663+0000 7fc337fff640 1 --2- 172.21.15.83:0/69097832 >> v2:172.21.15.183:3300/0 conn(0x7fc3380d4760 0x7fc3380d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.663+0000 7fc33cd73640 1 --2- 172.21.15.83:0/69097832 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc3380b4dd0 0x7fc3380d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.663+0000 7fc33cd73640 1 -- 172.21.15.83:0/69097832 >> v2:172.21.15.183:3300/0 conn(0x7fc3380d4760 msgr2=0x7fc3380d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:48.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.663+0000 7fc33cd73640 1 --2- 172.21.15.83:0/69097832 >> v2:172.21.15.183:3300/0 conn(0x7fc3380d4760 0x7fc3380d0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.663+0000 7fc33cd73640 1 -- 172.21.15.83:0/69097832 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3380d2870 con 0x7fc3380b4170 2024-03-06T07:30:48.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.663+0000 7fc33d574640 1 --2- 172.21.15.83:0/69097832 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc3380b4dd0 0x7fc3380d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:48.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.663+0000 7fc337fff640 1 --2- 172.21.15.83:0/69097832 >> v2:172.21.15.183:3300/0 conn(0x7fc3380d4760 0x7fc3380d0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:48.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.663+0000 7fc33cd73640 1 --2- 172.21.15.83:0/69097832 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc3380b4170 0x7fc3380b6a90 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7fc33808d0c0 tx=0x7fc33002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4141dadc10092121 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:48.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.664+0000 7fc3377fe640 1 -- 172.21.15.83:0/69097832 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc33002ca70 con 0x7fc3380b4170 2024-03-06T07:30:48.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.664+0000 7fc3377fe640 1 -- 172.21.15.83:0/69097832 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc33002cc10 con 0x7fc3380b4170 2024-03-06T07:30:48.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.664+0000 7fc3377fe640 1 -- 172.21.15.83:0/69097832 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3300343f0 con 0x7fc3380b4170 2024-03-06T07:30:48.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.664+0000 7fc33efd7640 1 -- 172.21.15.83:0/69097832 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc3380b4170 msgr2=0x7fc3380b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.664+0000 7fc33efd7640 1 --2- 172.21.15.83:0/69097832 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc3380b4170 0x7fc3380b6a90 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7fc33808d0c0 tx=0x7fc33002bed0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.664+0000 7fc33efd7640 1 -- 172.21.15.83:0/69097832 shutdown_connections 2024-03-06T07:30:48.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.664+0000 7fc33efd7640 1 --2- 172.21.15.83:0/69097832 >> v2:172.21.15.183:3300/0 conn(0x7fc3380d4760 0x7fc3380d0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.664+0000 7fc33efd7640 1 --2- 172.21.15.83:0/69097832 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc3380b4dd0 0x7fc3380d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.664+0000 7fc33efd7640 1 --2- 172.21.15.83:0/69097832 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc3380b4170 0x7fc3380b6a90 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.664+0000 7fc33efd7640 1 -- 172.21.15.83:0/69097832 >> 172.21.15.83:0/69097832 conn(0x7fc3380d3c40 msgr2=0x7fc3380d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:48.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.664+0000 7fc33efd7640 1 -- 172.21.15.83:0/69097832 shutdown_connections 2024-03-06T07:30:48.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.664+0000 7fc33efd7640 1 -- 172.21.15.83:0/69097832 wait complete. 2024-03-06T07:30:48.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.665+0000 7fc33efd7640 1 Processor -- start 2024-03-06T07:30:48.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.665+0000 7fc33efd7640 1 -- start start 2024-03-06T07:30:48.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.666+0000 7fc33efd7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc3380b4170 0x7fc338158e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.666+0000 7fc33efd7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc3380b4dd0 0x7fc33815b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.666+0000 7fc33efd7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc338147eb0 0x7fc338148360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.666+0000 7fc33efd7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3380b47d0 con 0x7fc3380b4dd0 2024-03-06T07:30:48.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.666+0000 7fc33efd7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3380b4550 con 0x7fc338147eb0 2024-03-06T07:30:48.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.666+0000 7fc33efd7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3380b4950 con 0x7fc3380b4170 2024-03-06T07:30:48.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.666+0000 7fc337fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc3380b4dd0 0x7fc33815b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.666+0000 7fc337fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc3380b4dd0 0x7fc33815b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:42212/0 (socket says 172.21.15.83:42212) 2024-03-06T07:30:48.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.666+0000 7fc337fff640 1 -- 172.21.15.83:0/1741373831 learned_addr learned my addr 172.21.15.83:0/1741373831 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:48.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.666+0000 7fc33d574640 1 --2- 172.21.15.83:0/1741373831 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc338147eb0 0x7fc338148360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.666+0000 7fc33cd73640 1 --2- 172.21.15.83:0/1741373831 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc3380b4170 0x7fc338158e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.667+0000 7fc337fff640 1 -- 172.21.15.83:0/1741373831 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc3380b4170 msgr2=0x7fc338158e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.667+0000 7fc337fff640 1 --2- 172.21.15.83:0/1741373831 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc3380b4170 0x7fc338158e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.667+0000 7fc337fff640 1 -- 172.21.15.83:0/1741373831 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc338147eb0 msgr2=0x7fc338148360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.667+0000 7fc337fff640 1 --2- 172.21.15.83:0/1741373831 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc338147eb0 0x7fc338148360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.667+0000 7fc337fff640 1 -- 172.21.15.83:0/1741373831 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc338159720 con 0x7fc3380b4dd0 2024-03-06T07:30:48.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.667+0000 7fc337fff640 1 --2- 172.21.15.83:0/1741373831 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc3380b4dd0 0x7fc33815b5e0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7fc32000c820 tx=0x7fc32000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:48.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.667+0000 7fc3357fa640 1 -- 172.21.15.83:0/1741373831 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc320015070 con 0x7fc3380b4dd0 2024-03-06T07:30:48.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.667+0000 7fc3357fa640 1 -- 172.21.15.83:0/1741373831 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3200079d0 con 0x7fc3380b4dd0 2024-03-06T07:30:48.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.667+0000 7fc33efd7640 1 -- 172.21.15.83:0/1741373831 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc338159a00 con 0x7fc3380b4dd0 2024-03-06T07:30:48.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.667+0000 7fc3357fa640 1 -- 172.21.15.83:0/1741373831 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc320007da0 con 0x7fc3380b4dd0 2024-03-06T07:30:48.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.667+0000 7fc33efd7640 1 -- 172.21.15.83:0/1741373831 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc338159fc0 con 0x7fc3380b4dd0 2024-03-06T07:30:48.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.668+0000 7fc33efd7640 1 -- 172.21.15.83:0/1741373831 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc33804f330 con 0x7fc3380b4dd0 2024-03-06T07:30:48.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.669+0000 7fc3357fa640 1 -- 172.21.15.83:0/1741373831 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc320018070 con 0x7fc3380b4dd0 2024-03-06T07:30:48.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.669+0000 7fc3357fa640 1 --2- 172.21.15.83:0/1741373831 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc31806b4d0 0x7fc31806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.669+0000 7fc3357fa640 1 -- 172.21.15.83:0/1741373831 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc32008a040 con 0x7fc3380b4dd0 2024-03-06T07:30:48.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.669+0000 7fc33cd73640 1 --2- 172.21.15.83:0/1741373831 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc31806b4d0 0x7fc31806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.669+0000 7fc33cd73640 1 --2- 172.21.15.83:0/1741373831 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc31806b4d0 0x7fc31806d990 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fc330035000 tx=0x7fc330030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:48.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.673+0000 7fc3357fa640 1 -- 172.21.15.83:0/1741373831 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc3200588c0 con 0x7fc3380b4dd0 2024-03-06T07:30:48.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.880+0000 7fc33efd7640 1 -- 172.21.15.83:0/1741373831 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3380733b0 con 0x7fc3380b4dd0 2024-03-06T07:30:48.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.880+0000 7fc3357fa640 1 -- 172.21.15.83:0/1741373831 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7fc32005c960 con 0x7fc3380b4dd0 2024-03-06T07:30:48.882 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:30:48.882 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:30:48.882 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:30:48.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.882+0000 7fc33efd7640 1 -- 172.21.15.83:0/1741373831 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc31806b4d0 msgr2=0x7fc31806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.882+0000 7fc33efd7640 1 --2- 172.21.15.83:0/1741373831 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc31806b4d0 0x7fc31806d990 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fc330035000 tx=0x7fc330030000 comp rx=0 tx=0).stop 2024-03-06T07:30:48.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.882+0000 7fc33efd7640 1 -- 172.21.15.83:0/1741373831 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc3380b4dd0 msgr2=0x7fc33815b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.882+0000 7fc33efd7640 1 --2- 172.21.15.83:0/1741373831 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc3380b4dd0 0x7fc33815b5e0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7fc32000c820 tx=0x7fc32000cc20 comp rx=0 tx=0).stop 2024-03-06T07:30:48.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.882+0000 7fc33efd7640 1 -- 172.21.15.83:0/1741373831 shutdown_connections 2024-03-06T07:30:48.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.882+0000 7fc33efd7640 1 --2- 172.21.15.83:0/1741373831 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc31806b4d0 0x7fc31806d990 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.882+0000 7fc33efd7640 1 --2- 172.21.15.83:0/1741373831 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc338147eb0 0x7fc338148360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.882+0000 7fc33efd7640 1 --2- 172.21.15.83:0/1741373831 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc3380b4dd0 0x7fc33815b5e0 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.882+0000 7fc33efd7640 1 --2- 172.21.15.83:0/1741373831 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc3380b4170 0x7fc338158e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.882+0000 7fc33efd7640 1 -- 172.21.15.83:0/1741373831 >> 172.21.15.83:0/1741373831 conn(0x7fc3380d3c40 msgr2=0x7fc3380cee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:48.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.882+0000 7fc33efd7640 1 -- 172.21.15.83:0/1741373831 shutdown_connections 2024-03-06T07:30:48.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:48.882+0000 7fc33efd7640 1 -- 172.21.15.83:0/1741373831 wait complete. 2024-03-06T07:30:49.050 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:49.097 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:30:49.097 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:30:50.098 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:50.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.196+0000 7fa824d32640 1 Processor -- start 2024-03-06T07:30:50.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.196+0000 7fa824d32640 1 -- start start 2024-03-06T07:30:50.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.197+0000 7fa824d32640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa82015bdd0 0x7fa8200c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.197+0000 7fa824d32640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fa8200d16e0 0x7fa8200d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.197+0000 7fa824d32640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8200cebe0 con 0x7fa82015fa10 2024-03-06T07:30:50.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.197+0000 7fa824d32640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8200ce980 con 0x7fa82015bdd0 2024-03-06T07:30:50.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.197+0000 7fa824d32640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8200ceec0 con 0x7fa8200d16e0 2024-03-06T07:30:50.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.197+0000 7fa81e59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa82015bdd0 0x7fa8200c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.197+0000 7fa81e59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa82015bdd0 0x7fa8200c5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:42222/0 (socket says 172.21.15.83:42222) 2024-03-06T07:30:50.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.197+0000 7fa81e59c640 1 -- 172.21.15.83:0/2150489001 learned_addr learned my addr 172.21.15.83:0/2150489001 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:50.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.197+0000 7fa817fff640 1 --1- 172.21.15.83:0/2150489001 >> v1:172.21.15.83:6789/0 conn(0x7fa82015fa10 0x7fa8200ce270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43262/0 (socket says 172.21.15.83:43262) 2024-03-06T07:30:50.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.197+0000 7fa81e59c640 1 -- 172.21.15.83:0/2150489001 >> v2:172.21.15.183:3300/0 conn(0x7fa8200d16e0 msgr2=0x7fa8200d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:50.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.197+0000 7fa81ed9d640 1 --2- 172.21.15.83:0/2150489001 >> v2:172.21.15.183:3300/0 conn(0x7fa8200d16e0 0x7fa8200d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.197+0000 7fa81dd9b640 1 -- 172.21.15.83:0/2150489001 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3285722191 0 0) 0x7fa8200cebe0 con 0x7fa82015fa10 2024-03-06T07:30:50.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.197+0000 7fa81e59c640 1 --2- 172.21.15.83:0/2150489001 >> v2:172.21.15.183:3300/0 conn(0x7fa8200d16e0 0x7fa8200d1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.197+0000 7fa81e59c640 1 -- 172.21.15.83:0/2150489001 >> v1:172.21.15.83:6789/0 conn(0x7fa82015fa10 legacy=0x7fa8200ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.197+0000 7fa81e59c640 1 -- 172.21.15.83:0/2150489001 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8200cf1a0 con 0x7fa82015bdd0 2024-03-06T07:30:50.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.198+0000 7fa81e59c640 1 --2- 172.21.15.83:0/2150489001 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa82015bdd0 0x7fa8200c5610 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7fa8200c5520 tx=0x7fa81002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b9d04d0b4dc7085 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:50.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.198+0000 7fa81dd9b640 1 -- 172.21.15.83:0/2150489001 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa810035020 con 0x7fa82015bdd0 2024-03-06T07:30:50.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.198+0000 7fa81dd9b640 1 -- 172.21.15.83:0/2150489001 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa810031870 con 0x7fa82015bdd0 2024-03-06T07:30:50.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.198+0000 7fa81dd9b640 1 -- 172.21.15.83:0/2150489001 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa810031c40 con 0x7fa82015bdd0 2024-03-06T07:30:50.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.198+0000 7fa824d32640 1 -- 172.21.15.83:0/2150489001 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa82015bdd0 msgr2=0x7fa8200c5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.198+0000 7fa824d32640 1 --2- 172.21.15.83:0/2150489001 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa82015bdd0 0x7fa8200c5610 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7fa8200c5520 tx=0x7fa81002b910 comp rx=0 tx=0).stop 2024-03-06T07:30:50.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.198+0000 7fa824d32640 1 -- 172.21.15.83:0/2150489001 shutdown_connections 2024-03-06T07:30:50.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.198+0000 7fa824d32640 1 --2- 172.21.15.83:0/2150489001 >> v2:172.21.15.183:3300/0 conn(0x7fa8200d16e0 0x7fa8200d1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.198+0000 7fa824d32640 1 --2- 172.21.15.83:0/2150489001 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa82015bdd0 0x7fa8200c5610 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.199+0000 7fa824d32640 1 -- 172.21.15.83:0/2150489001 >> 172.21.15.83:0/2150489001 conn(0x7fa82009a9f0 msgr2=0x7fa82009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:50.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.199+0000 7fa824d32640 1 -- 172.21.15.83:0/2150489001 shutdown_connections 2024-03-06T07:30:50.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.199+0000 7fa824d32640 1 -- 172.21.15.83:0/2150489001 wait complete. 2024-03-06T07:30:50.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.200+0000 7fa824d32640 1 Processor -- start 2024-03-06T07:30:50.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.200+0000 7fa824d32640 1 -- start start 2024-03-06T07:30:50.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.200+0000 7fa824d32640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8200d16e0 0x7fa8200ca0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.200+0000 7fa824d32640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa82015bdd0 0x7fa8200cb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.201+0000 7fa824d32640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa82015fa10 0x7fa8200cb730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.201+0000 7fa824d32640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8200cc5d0 con 0x7fa82015fa10 2024-03-06T07:30:50.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.201+0000 7fa824d32640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8200c5b50 con 0x7fa82015bdd0 2024-03-06T07:30:50.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.201+0000 7fa824d32640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa82015c1b0 con 0x7fa8200d16e0 2024-03-06T07:30:50.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.201+0000 7fa81ed9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa82015fa10 0x7fa8200cb730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.201+0000 7fa81ed9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa82015fa10 0x7fa8200cb730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:42226/0 (socket says 172.21.15.83:42226) 2024-03-06T07:30:50.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.201+0000 7fa81ed9d640 1 -- 172.21.15.83:0/2599881505 learned_addr learned my addr 172.21.15.83:0/2599881505 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:50.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.201+0000 7fa817fff640 1 --2- 172.21.15.83:0/2599881505 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa82015bdd0 0x7fa8200cb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.201+0000 7fa81e59c640 1 --2- 172.21.15.83:0/2599881505 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8200d16e0 0x7fa8200ca0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.201+0000 7fa81ed9d640 1 -- 172.21.15.83:0/2599881505 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8200d16e0 msgr2=0x7fa8200ca0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.201+0000 7fa81ed9d640 1 --2- 172.21.15.83:0/2599881505 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8200d16e0 0x7fa8200ca0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.201+0000 7fa81ed9d640 1 -- 172.21.15.83:0/2599881505 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa82015bdd0 msgr2=0x7fa8200cb1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.201+0000 7fa81ed9d640 1 --2- 172.21.15.83:0/2599881505 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa82015bdd0 0x7fa8200cb1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.201+0000 7fa81ed9d640 1 -- 172.21.15.83:0/2599881505 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8200c9050 con 0x7fa82015fa10 2024-03-06T07:30:50.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.201+0000 7fa817fff640 1 --2- 172.21.15.83:0/2599881505 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa82015bdd0 0x7fa8200cb1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:50.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.201+0000 7fa81ed9d640 1 --2- 172.21.15.83:0/2599881505 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa82015fa10 0x7fa8200cb730 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7fa81800cc30 tx=0x7fa81800a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:50.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.201+0000 7fa816ffd640 1 -- 172.21.15.83:0/2599881505 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa81800ce40 con 0x7fa82015fa10 2024-03-06T07:30:50.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.201+0000 7fa816ffd640 1 -- 172.21.15.83:0/2599881505 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa818003990 con 0x7fa82015fa10 2024-03-06T07:30:50.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.201+0000 7fa824d32640 1 -- 172.21.15.83:0/2599881505 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8200c9330 con 0x7fa82015fa10 2024-03-06T07:30:50.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.201+0000 7fa816ffd640 1 -- 172.21.15.83:0/2599881505 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa818003d40 con 0x7fa82015fa10 2024-03-06T07:30:50.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.201+0000 7fa824d32640 1 -- 172.21.15.83:0/2599881505 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8200c98f0 con 0x7fa82015fa10 2024-03-06T07:30:50.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.202+0000 7fa824d32640 1 -- 172.21.15.83:0/2599881505 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa82004f330 con 0x7fa82015fa10 2024-03-06T07:30:50.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.203+0000 7fa816ffd640 1 -- 172.21.15.83:0/2599881505 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa818004000 con 0x7fa82015fa10 2024-03-06T07:30:50.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.204+0000 7fa816ffd640 1 --2- 172.21.15.83:0/2599881505 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa80006b4d0 0x7fa80006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.204+0000 7fa816ffd640 1 -- 172.21.15.83:0/2599881505 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa81808a6f0 con 0x7fa82015fa10 2024-03-06T07:30:50.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.204+0000 7fa81e59c640 1 --2- 172.21.15.83:0/2599881505 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa80006b4d0 0x7fa80006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.204+0000 7fa81e59c640 1 --2- 172.21.15.83:0/2599881505 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa80006b4d0 0x7fa80006d990 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fa810036000 tx=0x7fa81003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:50.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.208+0000 7fa816ffd640 1 -- 172.21.15.83:0/2599881505 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa818014030 con 0x7fa82015fa10 2024-03-06T07:30:50.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.423+0000 7fa824d32640 1 -- 172.21.15.83:0/2599881505 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8200b7a40 con 0x7fa82015fa10 2024-03-06T07:30:50.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.423+0000 7fa816ffd640 1 -- 172.21.15.83:0/2599881505 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7fa818059530 con 0x7fa82015fa10 2024-03-06T07:30:50.425 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:30:50.426 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:30:50.426 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:30:50.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.426+0000 7fa824d32640 1 -- 172.21.15.83:0/2599881505 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa80006b4d0 msgr2=0x7fa80006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.426+0000 7fa824d32640 1 --2- 172.21.15.83:0/2599881505 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa80006b4d0 0x7fa80006d990 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fa810036000 tx=0x7fa81003c040 comp rx=0 tx=0).stop 2024-03-06T07:30:50.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.426+0000 7fa824d32640 1 -- 172.21.15.83:0/2599881505 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa82015fa10 msgr2=0x7fa8200cb730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.426+0000 7fa824d32640 1 --2- 172.21.15.83:0/2599881505 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa82015fa10 0x7fa8200cb730 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7fa81800cc30 tx=0x7fa81800a5c0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.426+0000 7fa824d32640 1 -- 172.21.15.83:0/2599881505 shutdown_connections 2024-03-06T07:30:50.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.426+0000 7fa824d32640 1 --2- 172.21.15.83:0/2599881505 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa80006b4d0 0x7fa80006d990 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.426+0000 7fa824d32640 1 --2- 172.21.15.83:0/2599881505 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa82015fa10 0x7fa8200cb730 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.426+0000 7fa824d32640 1 --2- 172.21.15.83:0/2599881505 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa82015bdd0 0x7fa8200cb1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.426+0000 7fa824d32640 1 --2- 172.21.15.83:0/2599881505 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8200d16e0 0x7fa8200ca0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.426+0000 7fa824d32640 1 -- 172.21.15.83:0/2599881505 >> 172.21.15.83:0/2599881505 conn(0x7fa82009a9f0 msgr2=0x7fa82015fe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:50.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.426+0000 7fa824d32640 1 -- 172.21.15.83:0/2599881505 shutdown_connections 2024-03-06T07:30:50.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:50.426+0000 7fa824d32640 1 -- 172.21.15.83:0/2599881505 wait complete. 2024-03-06T07:30:50.639 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:30:50.640 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:30:51.052 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:51.641 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:51.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.741+0000 7f5011d3e640 1 Processor -- start 2024-03-06T07:30:51.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.741+0000 7f5011d3e640 1 -- start start 2024-03-06T07:30:51.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.742+0000 7f5011d3e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f500c0d04a0 0x7f500c0ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.742+0000 7f5011d3e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f500c161980 con 0x7f500c0cfff0 2024-03-06T07:30:51.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.742+0000 7f5011d3e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f500c161c60 con 0x7f500c15bb20 2024-03-06T07:30:51.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.742+0000 7f5011d3e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f500c0d2500 con 0x7f500c0d04a0 2024-03-06T07:30:51.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.742+0000 7f500bfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f500c0d04a0 0x7f500c0ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.742+0000 7f500affd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f500c0cfff0 0x7f500c0cc8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43276/0 (socket says 172.21.15.83:43276) 2024-03-06T07:30:51.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.742+0000 7f500affd640 1 -- 172.21.15.83:0/4112741147 learned_addr learned my addr 172.21.15.83:0/4112741147 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:30:51.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.742+0000 7f500bfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f500c0d04a0 0x7f500c0ccfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:42238/0 (socket says 172.21.15.83:42238) 2024-03-06T07:30:51.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.743+0000 7f500bfff640 1 -- 172.21.15.83:0/4112741147 >> v1:172.21.15.183:6789/0 conn(0x7f500c15bb20 legacy=0x7f500c0d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.743+0000 7f500bfff640 1 -- 172.21.15.83:0/4112741147 >> v1:172.21.15.83:6789/0 conn(0x7f500c0cfff0 legacy=0x7f500c0cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.743+0000 7f500bfff640 1 -- 172.21.15.83:0/4112741147 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f500c161f40 con 0x7f500c0d04a0 2024-03-06T07:30:51.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.743+0000 7f500bfff640 1 --2- 172.21.15.83:0/4112741147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f500c0d04a0 0x7f500c0ccfc0 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f5004009d90 tx=0x7f500402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d006563f5ffad2ba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:51.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.744+0000 7f500a7fc640 1 -- 172.21.15.83:0/4112741147 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5004039070 con 0x7f500c0d04a0 2024-03-06T07:30:51.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.744+0000 7f500a7fc640 1 -- 172.21.15.83:0/4112741147 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f500402cd00 con 0x7f500c0d04a0 2024-03-06T07:30:51.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.744+0000 7f500a7fc640 1 -- 172.21.15.83:0/4112741147 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5004034660 con 0x7f500c0d04a0 2024-03-06T07:30:51.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.744+0000 7f5011d3e640 1 -- 172.21.15.83:0/4112741147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f500c0d04a0 msgr2=0x7f500c0ccfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.744+0000 7f5011d3e640 1 --2- 172.21.15.83:0/4112741147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f500c0d04a0 0x7f500c0ccfc0 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f5004009d90 tx=0x7f500402bf50 comp rx=0 tx=0).stop 2024-03-06T07:30:51.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.744+0000 7f5011d3e640 1 -- 172.21.15.83:0/4112741147 shutdown_connections 2024-03-06T07:30:51.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.744+0000 7f5011d3e640 1 --2- 172.21.15.83:0/4112741147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f500c0d04a0 0x7f500c0ccfc0 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.744+0000 7f5011d3e640 1 -- 172.21.15.83:0/4112741147 >> 172.21.15.83:0/4112741147 conn(0x7f500c09a9f0 msgr2=0x7f500c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:51.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.744+0000 7f5011d3e640 1 -- 172.21.15.83:0/4112741147 shutdown_connections 2024-03-06T07:30:51.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.744+0000 7f5011d3e640 1 -- 172.21.15.83:0/4112741147 wait complete. 2024-03-06T07:30:51.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.745+0000 7f5011d3e640 1 Processor -- start 2024-03-06T07:30:51.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.746+0000 7f5011d3e640 1 -- start start 2024-03-06T07:30:51.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.746+0000 7f5011d3e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f500c0cfff0 0x7f500c0c8080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.746+0000 7f5011d3e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f500c0d04a0 0x7f500c06fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.746+0000 7f5011d3e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f500c15bb20 0x7f500c070200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.746+0000 7f5011d3e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f500c0d08d0 con 0x7f500c0d04a0 2024-03-06T07:30:51.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.746+0000 7f5011d3e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f500c162500 con 0x7f500c0cfff0 2024-03-06T07:30:51.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.746+0000 7f5011d3e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f500c0d1980 con 0x7f500c15bb20 2024-03-06T07:30:51.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.746+0000 7f500affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f500c0d04a0 0x7f500c06fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.746+0000 7f500affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f500c0d04a0 0x7f500c06fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:42240/0 (socket says 172.21.15.83:42240) 2024-03-06T07:30:51.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.746+0000 7f500affd640 1 -- 172.21.15.83:0/411277142 learned_addr learned my addr 172.21.15.83:0/411277142 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:51.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.746+0000 7f500b7fe640 1 --2- 172.21.15.83:0/411277142 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f500c0cfff0 0x7f500c0c8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.747+0000 7f500bfff640 1 --2- 172.21.15.83:0/411277142 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f500c15bb20 0x7f500c070200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.747+0000 7f500affd640 1 -- 172.21.15.83:0/411277142 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f500c15bb20 msgr2=0x7f500c070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.747+0000 7f500affd640 1 --2- 172.21.15.83:0/411277142 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f500c15bb20 0x7f500c070200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.747+0000 7f500affd640 1 -- 172.21.15.83:0/411277142 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f500c0cfff0 msgr2=0x7f500c0c8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.747+0000 7f500affd640 1 --2- 172.21.15.83:0/411277142 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f500c0cfff0 0x7f500c0c8080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.747+0000 7f500affd640 1 -- 172.21.15.83:0/411277142 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f500c0b4cb0 con 0x7f500c0d04a0 2024-03-06T07:30:51.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.747+0000 7f500affd640 1 --2- 172.21.15.83:0/411277142 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f500c0d04a0 0x7f500c06fcc0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f500000c480 tx=0x7f500000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:51.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.747+0000 7f4ff3fff640 1 -- 172.21.15.83:0/411277142 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5000017070 con 0x7f500c0d04a0 2024-03-06T07:30:51.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.747+0000 7f4ff3fff640 1 -- 172.21.15.83:0/411277142 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5000005ce0 con 0x7f500c0d04a0 2024-03-06T07:30:51.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.747+0000 7f4ff3fff640 1 -- 172.21.15.83:0/411277142 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50000072f0 con 0x7f500c0d04a0 2024-03-06T07:30:51.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.747+0000 7f5011d3e640 1 -- 172.21.15.83:0/411277142 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f500c0b4f90 con 0x7f500c0d04a0 2024-03-06T07:30:51.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.747+0000 7f5011d3e640 1 -- 172.21.15.83:0/411277142 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f500c0b5550 con 0x7f500c0d04a0 2024-03-06T07:30:51.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.748+0000 7f5011d3e640 1 -- 172.21.15.83:0/411277142 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f500c04f330 con 0x7f500c0d04a0 2024-03-06T07:30:51.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.749+0000 7f4ff3fff640 1 -- 172.21.15.83:0/411277142 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f500001a070 con 0x7f500c0d04a0 2024-03-06T07:30:51.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.749+0000 7f4ff3fff640 1 --2- 172.21.15.83:0/411277142 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4ff806b4d0 0x7f4ff806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.749+0000 7f500b7fe640 1 --2- 172.21.15.83:0/411277142 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4ff806b4d0 0x7f4ff806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.749+0000 7f4ff3fff640 1 -- 172.21.15.83:0/411277142 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5000089490 con 0x7f500c0d04a0 2024-03-06T07:30:51.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.750+0000 7f500b7fe640 1 --2- 172.21.15.83:0/411277142 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4ff806b4d0 0x7f4ff806d990 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc001780 tx=0x7f4ffc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:51.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.754+0000 7f4ff3fff640 1 -- 172.21.15.83:0/411277142 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5000057d10 con 0x7f500c0d04a0 2024-03-06T07:30:51.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.973+0000 7f5011d3e640 1 -- 172.21.15.83:0/411277142 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f500c0ccc00 con 0x7f500c0d04a0 2024-03-06T07:30:51.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.974+0000 7f4ff3fff640 1 -- 172.21.15.83:0/411277142 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f500005bdb0 con 0x7f500c0d04a0 2024-03-06T07:30:51.976 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:30:51.976 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:30:51.977 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:30:51.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.977+0000 7f5011d3e640 1 -- 172.21.15.83:0/411277142 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4ff806b4d0 msgr2=0x7f4ff806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.977+0000 7f5011d3e640 1 --2- 172.21.15.83:0/411277142 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4ff806b4d0 0x7f4ff806d990 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc001780 tx=0x7f4ffc009040 comp rx=0 tx=0).stop 2024-03-06T07:30:51.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.977+0000 7f5011d3e640 1 -- 172.21.15.83:0/411277142 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f500c0d04a0 msgr2=0x7f500c06fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.977+0000 7f5011d3e640 1 --2- 172.21.15.83:0/411277142 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f500c0d04a0 0x7f500c06fcc0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f500000c480 tx=0x7f500000c940 comp rx=0 tx=0).stop 2024-03-06T07:30:51.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.977+0000 7f5011d3e640 1 -- 172.21.15.83:0/411277142 shutdown_connections 2024-03-06T07:30:51.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.977+0000 7f5011d3e640 1 --2- 172.21.15.83:0/411277142 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4ff806b4d0 0x7f4ff806d990 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.977+0000 7f5011d3e640 1 --2- 172.21.15.83:0/411277142 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f500c15bb20 0x7f500c070200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.977+0000 7f5011d3e640 1 --2- 172.21.15.83:0/411277142 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f500c0d04a0 0x7f500c06fcc0 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.977+0000 7f5011d3e640 1 --2- 172.21.15.83:0/411277142 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f500c0cfff0 0x7f500c0c8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.977+0000 7f5011d3e640 1 -- 172.21.15.83:0/411277142 >> 172.21.15.83:0/411277142 conn(0x7f500c09a9f0 msgr2=0x7f500c06f070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:51.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.977+0000 7f5011d3e640 1 -- 172.21.15.83:0/411277142 shutdown_connections 2024-03-06T07:30:51.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:51.977+0000 7f5011d3e640 1 -- 172.21.15.83:0/411277142 wait complete. 2024-03-06T07:30:52.191 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:30:52.191 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:30:53.054 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:53.192 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:53.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.291+0000 7f4eff579640 1 Processor -- start 2024-03-06T07:30:53.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.291+0000 7f4eff579640 1 -- start start 2024-03-06T07:30:53.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.292+0000 7f4eff579640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4ef80d16e0 0x7f4ef80d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.292+0000 7f4eff579640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef80cebe0 con 0x7f4ef815fa10 2024-03-06T07:30:53.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.292+0000 7f4eff579640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef80ceec0 con 0x7f4ef815bdd0 2024-03-06T07:30:53.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.292+0000 7f4eff579640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ef80ce980 con 0x7f4ef80d16e0 2024-03-06T07:30:53.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.292+0000 7f4efcb14640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f4ef815fa10 0x7f4ef80ce270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43284/0 (socket says 172.21.15.83:43284) 2024-03-06T07:30:53.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.292+0000 7f4efcb14640 1 -- 172.21.15.83:0/151338899 learned_addr learned my addr 172.21.15.83:0/151338899 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:30:53.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.292+0000 7f4efdb16640 1 --2- 172.21.15.83:0/151338899 >> v2:172.21.15.83:3300/0 conn(0x7f4ef80d16e0 0x7f4ef80d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.293+0000 7f4eeffff640 1 -- 172.21.15.83:0/151338899 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 296191249 0 0) 0x7f4ef80cebe0 con 0x7f4ef815fa10 2024-03-06T07:30:53.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.293+0000 7f4eeffff640 1 -- 172.21.15.83:0/151338899 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ee4003520 con 0x7f4ef815fa10 2024-03-06T07:30:53.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.293+0000 7f4efdb16640 1 -- 172.21.15.83:0/151338899 >> v1:172.21.15.183:6789/0 conn(0x7f4ef815bdd0 legacy=0x7f4ef80c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.293+0000 7f4efdb16640 1 -- 172.21.15.83:0/151338899 >> v1:172.21.15.83:6789/0 conn(0x7f4ef815fa10 legacy=0x7f4ef80ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.293+0000 7f4efdb16640 1 -- 172.21.15.83:0/151338899 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef80cf1a0 con 0x7f4ef80d16e0 2024-03-06T07:30:53.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.293+0000 7f4efdb16640 1 --2- 172.21.15.83:0/151338899 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ef80d16e0 0x7f4ef80d1af0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f4ef0009d90 tx=0x7f4ef002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99ecc7d7a8b7b5df server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:53.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.294+0000 7f4eeffff640 1 -- 172.21.15.83:0/151338899 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef0039070 con 0x7f4ef80d16e0 2024-03-06T07:30:53.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.294+0000 7f4eeffff640 1 -- 172.21.15.83:0/151338899 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ef002cc60 con 0x7f4ef80d16e0 2024-03-06T07:30:53.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.294+0000 7f4eeffff640 1 -- 172.21.15.83:0/151338899 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef0034660 con 0x7f4ef80d16e0 2024-03-06T07:30:53.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.294+0000 7f4eff579640 1 -- 172.21.15.83:0/151338899 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ef80d16e0 msgr2=0x7f4ef80d1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.294+0000 7f4eff579640 1 --2- 172.21.15.83:0/151338899 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ef80d16e0 0x7f4ef80d1af0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f4ef0009d90 tx=0x7f4ef002beb0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.294+0000 7f4eff579640 1 -- 172.21.15.83:0/151338899 shutdown_connections 2024-03-06T07:30:53.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.294+0000 7f4eff579640 1 --2- 172.21.15.83:0/151338899 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ef80d16e0 0x7f4ef80d1af0 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.294+0000 7f4eff579640 1 -- 172.21.15.83:0/151338899 >> 172.21.15.83:0/151338899 conn(0x7f4ef809a9f0 msgr2=0x7f4ef809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:53.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.294+0000 7f4eff579640 1 -- 172.21.15.83:0/151338899 shutdown_connections 2024-03-06T07:30:53.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.294+0000 7f4eff579640 1 -- 172.21.15.83:0/151338899 wait complete. 2024-03-06T07:30:53.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.296+0000 7f4eff579640 1 Processor -- start 2024-03-06T07:30:53.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.296+0000 7f4eff579640 1 -- start start 2024-03-06T07:30:53.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.297+0000 7f4eff579640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4ef80d16e0 0x7f4ef80c9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.297+0000 7f4eff579640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4ef815bdd0 0x7f4ef81f1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.297+0000 7f4eff579640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ef815fa10 0x7f4ef81f2360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.297+0000 7f4eff579640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef815c1b0 con 0x7f4ef815fa10 2024-03-06T07:30:53.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.297+0000 7f4eff579640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef80cc5d0 con 0x7f4ef815bdd0 2024-03-06T07:30:53.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.297+0000 7f4eff579640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef80d2030 con 0x7f4ef80d16e0 2024-03-06T07:30:53.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.298+0000 7f4efdb16640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ef815fa10 0x7f4ef81f2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.298+0000 7f4efdb16640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ef815fa10 0x7f4ef81f2360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:42254/0 (socket says 172.21.15.83:42254) 2024-03-06T07:30:53.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.298+0000 7f4efdb16640 1 -- 172.21.15.83:0/1635330234 learned_addr learned my addr 172.21.15.83:0/1635330234 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:53.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.298+0000 7f4efcb14640 1 --2- 172.21.15.83:0/1635330234 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4ef815bdd0 0x7f4ef81f1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.298+0000 7f4efd315640 1 --2- 172.21.15.83:0/1635330234 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4ef80d16e0 0x7f4ef80c9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.298+0000 7f4efdb16640 1 -- 172.21.15.83:0/1635330234 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4ef80d16e0 msgr2=0x7f4ef80c9ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.298+0000 7f4efdb16640 1 --2- 172.21.15.83:0/1635330234 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4ef80d16e0 0x7f4ef80c9ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.298+0000 7f4efdb16640 1 -- 172.21.15.83:0/1635330234 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4ef815bdd0 msgr2=0x7f4ef81f1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.298+0000 7f4efdb16640 1 --2- 172.21.15.83:0/1635330234 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4ef815bdd0 0x7f4ef81f1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.298+0000 7f4efdb16640 1 -- 172.21.15.83:0/1635330234 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef80ca4e0 con 0x7f4ef815fa10 2024-03-06T07:30:53.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.298+0000 7f4efcb14640 1 --2- 172.21.15.83:0/1635330234 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4ef815bdd0 0x7f4ef81f1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:53.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.298+0000 7f4efdb16640 1 --2- 172.21.15.83:0/1635330234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ef815fa10 0x7f4ef81f2360 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f4ef0009eb0 tx=0x7f4ef0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:53.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.298+0000 7f4eedffb640 1 -- 172.21.15.83:0/1635330234 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef0039050 con 0x7f4ef815fa10 2024-03-06T07:30:53.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.298+0000 7f4eedffb640 1 -- 172.21.15.83:0/1635330234 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ef0045070 con 0x7f4ef815fa10 2024-03-06T07:30:53.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.298+0000 7f4eedffb640 1 -- 172.21.15.83:0/1635330234 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef0009450 con 0x7f4ef815fa10 2024-03-06T07:30:53.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.298+0000 7f4eff579640 1 -- 172.21.15.83:0/1635330234 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ef81f4fc0 con 0x7f4ef815fa10 2024-03-06T07:30:53.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.298+0000 7f4eff579640 1 -- 172.21.15.83:0/1635330234 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ef81f5aa0 con 0x7f4ef815fa10 2024-03-06T07:30:53.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.299+0000 7f4eff579640 1 -- 172.21.15.83:0/1635330234 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ef804f330 con 0x7f4ef815fa10 2024-03-06T07:30:53.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.300+0000 7f4eedffb640 1 -- 172.21.15.83:0/1635330234 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4ef003e850 con 0x7f4ef815fa10 2024-03-06T07:30:53.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.300+0000 7f4eedffb640 1 --2- 172.21.15.83:0/1635330234 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4edc06b4d0 0x7f4edc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.300+0000 7f4eedffb640 1 -- 172.21.15.83:0/1635330234 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4ef00ad180 con 0x7f4ef815fa10 2024-03-06T07:30:53.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.300+0000 7f4efd315640 1 --2- 172.21.15.83:0/1635330234 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4edc06b4d0 0x7f4edc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.301+0000 7f4efd315640 1 --2- 172.21.15.83:0/1635330234 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4edc06b4d0 0x7f4edc06d990 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f4ef80cb940 tx=0x7f4ef4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:53.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.305+0000 7f4eedffb640 1 -- 172.21.15.83:0/1635330234 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4ef0032e20 con 0x7f4ef815fa10 2024-03-06T07:30:53.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.519+0000 7f4eff579640 1 -- 172.21.15.83:0/1635330234 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ef815f1c0 con 0x7f4ef815fa10 2024-03-06T07:30:53.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.520+0000 7f4eedffb640 1 -- 172.21.15.83:0/1635330234 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f4ef007c170 con 0x7f4ef815fa10 2024-03-06T07:30:53.522 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:30:53.522 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:30:53.522 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:30:53.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.522+0000 7f4eff579640 1 -- 172.21.15.83:0/1635330234 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4edc06b4d0 msgr2=0x7f4edc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.522+0000 7f4eff579640 1 --2- 172.21.15.83:0/1635330234 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4edc06b4d0 0x7f4edc06d990 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f4ef80cb940 tx=0x7f4ef4009040 comp rx=0 tx=0).stop 2024-03-06T07:30:53.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.522+0000 7f4eff579640 1 -- 172.21.15.83:0/1635330234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ef815fa10 msgr2=0x7f4ef81f2360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.522+0000 7f4eff579640 1 --2- 172.21.15.83:0/1635330234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ef815fa10 0x7f4ef81f2360 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f4ef0009eb0 tx=0x7f4ef0004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:53.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.522+0000 7f4eff579640 1 -- 172.21.15.83:0/1635330234 shutdown_connections 2024-03-06T07:30:53.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.522+0000 7f4eff579640 1 --2- 172.21.15.83:0/1635330234 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4edc06b4d0 0x7f4edc06d990 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.522+0000 7f4eff579640 1 --2- 172.21.15.83:0/1635330234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ef815fa10 0x7f4ef81f2360 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.522+0000 7f4eff579640 1 --2- 172.21.15.83:0/1635330234 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4ef815bdd0 0x7f4ef81f1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.522+0000 7f4eff579640 1 --2- 172.21.15.83:0/1635330234 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4ef80d16e0 0x7f4ef80c9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.522+0000 7f4eff579640 1 -- 172.21.15.83:0/1635330234 >> 172.21.15.83:0/1635330234 conn(0x7f4ef809a9f0 msgr2=0x7f4ef809add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:53.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.522+0000 7f4eff579640 1 -- 172.21.15.83:0/1635330234 shutdown_connections 2024-03-06T07:30:53.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:53.522+0000 7f4eff579640 1 -- 172.21.15.83:0/1635330234 wait complete. 2024-03-06T07:30:53.736 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:30:53.736 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:30:54.737 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:54.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.836+0000 7f5f50f08640 1 Processor -- start 2024-03-06T07:30:54.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.836+0000 7f5f50f08640 1 -- start start 2024-03-06T07:30:54.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.837+0000 7f5f50f08640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5f4c0d22b0 0x7f5f4c0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.837+0000 7f5f50f08640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f4c0b3430 con 0x7f5f4c0b41f0 2024-03-06T07:30:54.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.837+0000 7f5f50f08640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f4c0b3710 con 0x7f5f4c15dfe0 2024-03-06T07:30:54.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.837+0000 7f5f50f08640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f4c0d2c50 con 0x7f5f4c0d22b0 2024-03-06T07:30:54.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.837+0000 7f5f41d9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5f4c0d22b0 0x7f5f4c0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.837+0000 7f5f4ad9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f5f4c0b41f0 0x7f5f4c0b4680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43298/0 (socket says 172.21.15.83:43298) 2024-03-06T07:30:54.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.837+0000 7f5f4ad9d640 1 -- 172.21.15.83:0/3838969056 learned_addr learned my addr 172.21.15.83:0/3838969056 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:30:54.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.837+0000 7f5f41d9b640 1 -- 172.21.15.83:0/3838969056 >> v1:172.21.15.183:6789/0 conn(0x7f5f4c15dfe0 legacy=0x7f5f4c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:54.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.837+0000 7f5f41d9b640 1 -- 172.21.15.83:0/3838969056 >> v1:172.21.15.83:6789/0 conn(0x7f5f4c0b41f0 legacy=0x7f5f4c0b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.837+0000 7f5f41d9b640 1 -- 172.21.15.83:0/3838969056 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f4c0b39f0 con 0x7f5f4c0d22b0 2024-03-06T07:30:54.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.837+0000 7f5f49d9b640 1 -- 172.21.15.83:0/3838969056 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3882728598 0 0) 0x7f5f4c0b3430 con 0x7f5f4c0b41f0 2024-03-06T07:30:54.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.838+0000 7f5f41d9b640 1 --2- 172.21.15.83:0/3838969056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f4c0d22b0 0x7f5f4c0d2710 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f5f340097d0 tx=0x7f5f3402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9d32039a0887f1a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:54.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.838+0000 7f5f49d9b640 1 -- 172.21.15.83:0/3838969056 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f34035020 con 0x7f5f4c0d22b0 2024-03-06T07:30:54.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.838+0000 7f5f49d9b640 1 -- 172.21.15.83:0/3838969056 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f34031830 con 0x7f5f4c0d22b0 2024-03-06T07:30:54.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.838+0000 7f5f49d9b640 1 -- 172.21.15.83:0/3838969056 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f34031c00 con 0x7f5f4c0d22b0 2024-03-06T07:30:54.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.838+0000 7f5f50f08640 1 -- 172.21.15.83:0/3838969056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f4c0d22b0 msgr2=0x7f5f4c0d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.838+0000 7f5f50f08640 1 --2- 172.21.15.83:0/3838969056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f4c0d22b0 0x7f5f4c0d2710 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f5f340097d0 tx=0x7f5f3402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.839+0000 7f5f50f08640 1 -- 172.21.15.83:0/3838969056 shutdown_connections 2024-03-06T07:30:54.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.839+0000 7f5f50f08640 1 --2- 172.21.15.83:0/3838969056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f4c0d22b0 0x7f5f4c0d2710 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.839+0000 7f5f50f08640 1 -- 172.21.15.83:0/3838969056 >> 172.21.15.83:0/3838969056 conn(0x7f5f4c0d3cd0 msgr2=0x7f5f4c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:54.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.839+0000 7f5f50f08640 1 -- 172.21.15.83:0/3838969056 shutdown_connections 2024-03-06T07:30:54.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.839+0000 7f5f50f08640 1 -- 172.21.15.83:0/3838969056 wait complete. 2024-03-06T07:30:54.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.840+0000 7f5f50f08640 1 Processor -- start 2024-03-06T07:30:54.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.840+0000 7f5f50f08640 1 -- start start 2024-03-06T07:30:54.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.841+0000 7f5f50f08640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f4c0b41f0 0x7f5f4c161200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.841+0000 7f5f50f08640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f4c0d22b0 0x7f5f4c0b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.841+0000 7f5f50f08640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f4c15dfe0 0x7f5f4c15bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.841+0000 7f5f50f08640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4c0b3d60 con 0x7f5f4c0d22b0 2024-03-06T07:30:54.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.841+0000 7f5f50f08640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4c0b31e0 con 0x7f5f4c0b41f0 2024-03-06T07:30:54.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.841+0000 7f5f50f08640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4c0d1c70 con 0x7f5f4c15dfe0 2024-03-06T07:30:54.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.841+0000 7f5f41d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f4c0d22b0 0x7f5f4c0b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.841+0000 7f5f41d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f4c0d22b0 0x7f5f4c0b68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:42276/0 (socket says 172.21.15.83:42276) 2024-03-06T07:30:54.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.841+0000 7f5f41d9b640 1 -- 172.21.15.83:0/1884677713 learned_addr learned my addr 172.21.15.83:0/1884677713 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:54.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.841+0000 7f5f4a59c640 1 --2- 172.21.15.83:0/1884677713 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f4c0b41f0 0x7f5f4c161200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.841+0000 7f5f4ad9d640 1 --2- 172.21.15.83:0/1884677713 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f4c15dfe0 0x7f5f4c15bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.841+0000 7f5f41d9b640 1 -- 172.21.15.83:0/1884677713 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f4c15dfe0 msgr2=0x7f5f4c15bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.841+0000 7f5f41d9b640 1 --2- 172.21.15.83:0/1884677713 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f4c15dfe0 0x7f5f4c15bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.841+0000 7f5f41d9b640 1 -- 172.21.15.83:0/1884677713 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f4c0b41f0 msgr2=0x7f5f4c161200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.841+0000 7f5f41d9b640 1 --2- 172.21.15.83:0/1884677713 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f4c0b41f0 0x7f5f4c161200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.841+0000 7f5f41d9b640 1 -- 172.21.15.83:0/1884677713 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f4c0b7000 con 0x7f5f4c0d22b0 2024-03-06T07:30:54.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.841+0000 7f5f4a59c640 1 --2- 172.21.15.83:0/1884677713 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f4c0b41f0 0x7f5f4c161200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:54.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.841+0000 7f5f41d9b640 1 --2- 172.21.15.83:0/1884677713 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f4c0d22b0 0x7f5f4c0b68d0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f5f340098f0 tx=0x7f5f340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:54.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.841+0000 7f5f437fe640 1 -- 172.21.15.83:0/1884677713 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f3403ba80 con 0x7f5f4c0d22b0 2024-03-06T07:30:54.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.842+0000 7f5f437fe640 1 -- 172.21.15.83:0/1884677713 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f3403bc20 con 0x7f5f4c0d22b0 2024-03-06T07:30:54.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.842+0000 7f5f50f08640 1 -- 172.21.15.83:0/1884677713 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f4c0b72e0 con 0x7f5f4c0d22b0 2024-03-06T07:30:54.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.842+0000 7f5f437fe640 1 -- 172.21.15.83:0/1884677713 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f340453f0 con 0x7f5f4c0d22b0 2024-03-06T07:30:54.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.842+0000 7f5f50f08640 1 -- 172.21.15.83:0/1884677713 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f4c0b78a0 con 0x7f5f4c0d22b0 2024-03-06T07:30:54.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.842+0000 7f5f50f08640 1 -- 172.21.15.83:0/1884677713 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f4c04f330 con 0x7f5f4c0d22b0 2024-03-06T07:30:54.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.843+0000 7f5f437fe640 1 -- 172.21.15.83:0/1884677713 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5f3404c060 con 0x7f5f4c0d22b0 2024-03-06T07:30:54.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.843+0000 7f5f437fe640 1 --2- 172.21.15.83:0/1884677713 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5f2c06b4d0 0x7f5f2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.843+0000 7f5f437fe640 1 -- 172.21.15.83:0/1884677713 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5f340abfc0 con 0x7f5f4c0d22b0 2024-03-06T07:30:54.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.843+0000 7f5f4a59c640 1 --2- 172.21.15.83:0/1884677713 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5f2c06b4d0 0x7f5f2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.844+0000 7f5f4a59c640 1 --2- 172.21.15.83:0/1884677713 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5f2c06b4d0 0x7f5f2c06d990 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c00a810 tx=0x7f5f3c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:54.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:54.848+0000 7f5f437fe640 1 -- 172.21.15.83:0/1884677713 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5f34040070 con 0x7f5f4c0d22b0 2024-03-06T07:30:55.056 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:55.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:55.059+0000 7f5f50f08640 1 -- 172.21.15.83:0/1884677713 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f4c15c090 con 0x7f5f4c0d22b0 2024-03-06T07:30:55.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:55.060+0000 7f5f437fe640 1 -- 172.21.15.83:0/1884677713 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f5f3402f2e0 con 0x7f5f4c0d22b0 2024-03-06T07:30:55.062 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:30:55.062 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:30:55.062 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:30:55.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:55.062+0000 7f5f50f08640 1 -- 172.21.15.83:0/1884677713 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5f2c06b4d0 msgr2=0x7f5f2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:55.062+0000 7f5f50f08640 1 --2- 172.21.15.83:0/1884677713 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5f2c06b4d0 0x7f5f2c06d990 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c00a810 tx=0x7f5f3c008040 comp rx=0 tx=0).stop 2024-03-06T07:30:55.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:55.062+0000 7f5f50f08640 1 -- 172.21.15.83:0/1884677713 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f4c0d22b0 msgr2=0x7f5f4c0b68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:55.062+0000 7f5f50f08640 1 --2- 172.21.15.83:0/1884677713 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f4c0d22b0 0x7f5f4c0b68d0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f5f340098f0 tx=0x7f5f340046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:55.062+0000 7f5f50f08640 1 -- 172.21.15.83:0/1884677713 shutdown_connections 2024-03-06T07:30:55.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:55.062+0000 7f5f50f08640 1 --2- 172.21.15.83:0/1884677713 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5f2c06b4d0 0x7f5f2c06d990 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:55.062+0000 7f5f50f08640 1 --2- 172.21.15.83:0/1884677713 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f4c15dfe0 0x7f5f4c15bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:55.062+0000 7f5f50f08640 1 --2- 172.21.15.83:0/1884677713 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f4c0d22b0 0x7f5f4c0b68d0 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:55.062+0000 7f5f50f08640 1 --2- 172.21.15.83:0/1884677713 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f4c0b41f0 0x7f5f4c161200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:55.062+0000 7f5f50f08640 1 -- 172.21.15.83:0/1884677713 >> 172.21.15.83:0/1884677713 conn(0x7f5f4c0d3cd0 msgr2=0x7f5f4c06ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:55.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:55.062+0000 7f5f50f08640 1 -- 172.21.15.83:0/1884677713 shutdown_connections 2024-03-06T07:30:55.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:55.062+0000 7f5f50f08640 1 -- 172.21.15.83:0/1884677713 wait complete. 2024-03-06T07:30:55.276 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:30:55.276 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:30:56.277 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:56.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.377+0000 7f88d7633640 1 Processor -- start 2024-03-06T07:30:56.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.377+0000 7f88d7633640 1 -- start start 2024-03-06T07:30:56.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.378+0000 7f88d7633640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f88d00b4170 0x7f88d00b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.378+0000 7f88d7633640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f88d00d0a80 0x7f88d00cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.378+0000 7f88d7633640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88d00d25d0 con 0x7f88d00d4760 2024-03-06T07:30:56.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.378+0000 7f88d7633640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88d00d4bf0 con 0x7f88d00b4170 2024-03-06T07:30:56.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.378+0000 7f88d7633640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88d00ce9e0 con 0x7f88d00d0a80 2024-03-06T07:30:56.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.378+0000 7f88d53cf640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f88d00b4170 0x7f88d00b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.378+0000 7f88d53cf640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f88d00b4170 0x7f88d00b6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:42284/0 (socket says 172.21.15.83:42284) 2024-03-06T07:30:56.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.378+0000 7f88d53cf640 1 -- 172.21.15.83:0/1567499273 learned_addr learned my addr 172.21.15.83:0/1567499273 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:56.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.378+0000 7f88d5bd0640 1 --2- 172.21.15.83:0/1567499273 >> v2:172.21.15.183:3300/0 conn(0x7f88d00d0a80 0x7f88d00cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.378+0000 7f88d53cf640 1 -- 172.21.15.83:0/1567499273 >> v2:172.21.15.183:3300/0 conn(0x7f88d00d0a80 msgr2=0x7f88d00cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:56.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.378+0000 7f88d53cf640 1 --2- 172.21.15.83:0/1567499273 >> v2:172.21.15.183:3300/0 conn(0x7f88d00d0a80 0x7f88d00cee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.379+0000 7f88d53cf640 1 -- 172.21.15.83:0/1567499273 >> v1:172.21.15.183:6789/0 conn(0x7f88d00d4760 legacy=0x7f88d00d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:56.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.379+0000 7f88d53cf640 1 -- 172.21.15.83:0/1567499273 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88d00d2870 con 0x7f88d00b4170 2024-03-06T07:30:56.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.379+0000 7f88d53cf640 1 --2- 172.21.15.83:0/1567499273 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f88d00b4170 0x7f88d00b6a90 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f88d008df00 tx=0x7f88c802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffa30076ce5edfc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:56.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.379+0000 7f88c7fff640 1 -- 172.21.15.83:0/1567499273 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c8034070 con 0x7f88d00b4170 2024-03-06T07:30:56.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.379+0000 7f88c7fff640 1 -- 172.21.15.83:0/1567499273 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88c8030860 con 0x7f88d00b4170 2024-03-06T07:30:56.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.379+0000 7f88c7fff640 1 -- 172.21.15.83:0/1567499273 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c8030c30 con 0x7f88d00b4170 2024-03-06T07:30:56.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.380+0000 7f88d7633640 1 -- 172.21.15.83:0/1567499273 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f88d00b4170 msgr2=0x7f88d00b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.380+0000 7f88d7633640 1 --2- 172.21.15.83:0/1567499273 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f88d00b4170 0x7f88d00b6a90 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f88d008df00 tx=0x7f88c802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.380+0000 7f88d7633640 1 -- 172.21.15.83:0/1567499273 shutdown_connections 2024-03-06T07:30:56.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.380+0000 7f88d7633640 1 --2- 172.21.15.83:0/1567499273 >> v2:172.21.15.183:3300/0 conn(0x7f88d00d0a80 0x7f88d00cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.380+0000 7f88d7633640 1 --2- 172.21.15.83:0/1567499273 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f88d00b4170 0x7f88d00b6a90 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.380+0000 7f88d7633640 1 -- 172.21.15.83:0/1567499273 >> 172.21.15.83:0/1567499273 conn(0x7f88d00d3c40 msgr2=0x7f88d00d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:56.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.380+0000 7f88d7633640 1 -- 172.21.15.83:0/1567499273 shutdown_connections 2024-03-06T07:30:56.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.380+0000 7f88d7633640 1 -- 172.21.15.83:0/1567499273 wait complete. 2024-03-06T07:30:56.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.382+0000 7f88d7633640 1 Processor -- start 2024-03-06T07:30:56.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.382+0000 7f88d7633640 1 -- start start 2024-03-06T07:30:56.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.383+0000 7f88d7633640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f88d00b4170 0x7f88d015d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.383+0000 7f88d7633640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f88d00d0a80 0x7f88d009f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.383+0000 7f88d7633640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f88d009fb70 0x7f88d00b3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.383+0000 7f88d7633640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d00b4dd0 con 0x7f88d00b4170 2024-03-06T07:30:56.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.383+0000 7f88d7633640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d00b4550 con 0x7f88d00d0a80 2024-03-06T07:30:56.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.383+0000 7f88d7633640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88d00b4f50 con 0x7f88d009fb70 2024-03-06T07:30:56.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.383+0000 7f88d53cf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f88d00b4170 0x7f88d015d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.383+0000 7f88d53cf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f88d00b4170 0x7f88d015d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:42286/0 (socket says 172.21.15.83:42286) 2024-03-06T07:30:56.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.383+0000 7f88d53cf640 1 -- 172.21.15.83:0/843480385 learned_addr learned my addr 172.21.15.83:0/843480385 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:56.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.383+0000 7f88d5bd0640 1 --2- 172.21.15.83:0/843480385 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f88d009fb70 0x7f88d00b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.383+0000 7f88d4bce640 1 --2- 172.21.15.83:0/843480385 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f88d00d0a80 0x7f88d009f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.384+0000 7f88d53cf640 1 -- 172.21.15.83:0/843480385 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f88d009fb70 msgr2=0x7f88d00b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.384+0000 7f88d53cf640 1 --2- 172.21.15.83:0/843480385 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f88d009fb70 0x7f88d00b3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.384+0000 7f88d53cf640 1 -- 172.21.15.83:0/843480385 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f88d00d0a80 msgr2=0x7f88d009f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.384+0000 7f88d53cf640 1 --2- 172.21.15.83:0/843480385 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f88d00d0a80 0x7f88d009f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.384+0000 7f88d53cf640 1 -- 172.21.15.83:0/843480385 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88d00b2160 con 0x7f88d00b4170 2024-03-06T07:30:56.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.384+0000 7f88d4bce640 1 --2- 172.21.15.83:0/843480385 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f88d00d0a80 0x7f88d009f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:56.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.384+0000 7f88d53cf640 1 --2- 172.21.15.83:0/843480385 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f88d00b4170 0x7f88d015d7c0 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f88c802b9e0 tx=0x7f88c8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:56.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.384+0000 7f88c5ffb640 1 -- 172.21.15.83:0/843480385 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c8048070 con 0x7f88d00b4170 2024-03-06T07:30:56.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.384+0000 7f88c5ffb640 1 -- 172.21.15.83:0/843480385 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88c8030660 con 0x7f88d00b4170 2024-03-06T07:30:56.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.384+0000 7f88d7633640 1 -- 172.21.15.83:0/843480385 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88d00a01a0 con 0x7f88d00b4170 2024-03-06T07:30:56.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.384+0000 7f88c5ffb640 1 -- 172.21.15.83:0/843480385 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c8034040 con 0x7f88d00b4170 2024-03-06T07:30:56.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.384+0000 7f88d7633640 1 -- 172.21.15.83:0/843480385 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88d00b1680 con 0x7f88d00b4170 2024-03-06T07:30:56.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.385+0000 7f88d7633640 1 -- 172.21.15.83:0/843480385 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88d008dd30 con 0x7f88d00b4170 2024-03-06T07:30:56.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.386+0000 7f88c5ffb640 1 -- 172.21.15.83:0/843480385 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f88c804b070 con 0x7f88d00b4170 2024-03-06T07:30:56.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.386+0000 7f88c5ffb640 1 --2- 172.21.15.83:0/843480385 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f88b406b500 0x7f88b406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.386+0000 7f88c5ffb640 1 -- 172.21.15.83:0/843480385 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f88c80ac290 con 0x7f88d00b4170 2024-03-06T07:30:56.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.386+0000 7f88d4bce640 1 --2- 172.21.15.83:0/843480385 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f88b406b500 0x7f88b406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.386+0000 7f88d4bce640 1 --2- 172.21.15.83:0/843480385 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f88b406b500 0x7f88b406d9c0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f88d006d340 tx=0x7f88c0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:56.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.390+0000 7f88c5ffb640 1 -- 172.21.15.83:0/843480385 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f88d008dd30 con 0x7f88d00b4170 2024-03-06T07:30:56.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.605+0000 7f88d7633640 1 -- 172.21.15.83:0/843480385 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88d004f330 con 0x7f88d00b4170 2024-03-06T07:30:56.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.605+0000 7f88c5ffb640 1 -- 172.21.15.83:0/843480385 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f88c807b280 con 0x7f88d00b4170 2024-03-06T07:30:56.607 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:30:56.607 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:30:56.607 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:30:56.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.607+0000 7f88d7633640 1 -- 172.21.15.83:0/843480385 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f88b406b500 msgr2=0x7f88b406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.607+0000 7f88d7633640 1 --2- 172.21.15.83:0/843480385 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f88b406b500 0x7f88b406d9c0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f88d006d340 tx=0x7f88c0009040 comp rx=0 tx=0).stop 2024-03-06T07:30:56.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.607+0000 7f88d7633640 1 -- 172.21.15.83:0/843480385 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f88d00b4170 msgr2=0x7f88d015d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.607+0000 7f88d7633640 1 --2- 172.21.15.83:0/843480385 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f88d00b4170 0x7f88d015d7c0 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f88c802b9e0 tx=0x7f88c8004730 comp rx=0 tx=0).stop 2024-03-06T07:30:56.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.607+0000 7f88d7633640 1 -- 172.21.15.83:0/843480385 shutdown_connections 2024-03-06T07:30:56.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.607+0000 7f88d7633640 1 --2- 172.21.15.83:0/843480385 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f88b406b500 0x7f88b406d9c0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.607+0000 7f88d7633640 1 --2- 172.21.15.83:0/843480385 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f88d009fb70 0x7f88d00b3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.607+0000 7f88d7633640 1 --2- 172.21.15.83:0/843480385 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f88d00d0a80 0x7f88d009f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.607+0000 7f88d7633640 1 --2- 172.21.15.83:0/843480385 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f88d00b4170 0x7f88d015d7c0 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.607+0000 7f88d7633640 1 -- 172.21.15.83:0/843480385 >> 172.21.15.83:0/843480385 conn(0x7f88d00d3c40 msgr2=0x7f88d00b7060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:56.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.608+0000 7f88d7633640 1 -- 172.21.15.83:0/843480385 shutdown_connections 2024-03-06T07:30:56.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:56.608+0000 7f88d7633640 1 -- 172.21.15.83:0/843480385 wait complete. 2024-03-06T07:30:56.821 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:30:56.821 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:30:57.058 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:57.822 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:57.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.922+0000 7f971190c640 1 Processor -- start 2024-03-06T07:30:57.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.922+0000 7f971190c640 1 -- start start 2024-03-06T07:30:57.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.923+0000 7f971190c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f970c0d22b0 0x7f970c0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.923+0000 7f971190c640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f970c0b41f0 0x7f970c0b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.923+0000 7f971190c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f970c0b3430 con 0x7f970c15dfe0 2024-03-06T07:30:57.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.923+0000 7f971190c640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f970c0d2c50 con 0x7f970c0b41f0 2024-03-06T07:30:57.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.923+0000 7f971190c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f970c0b31e0 con 0x7f970c0d22b0 2024-03-06T07:30:57.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.923+0000 7f970b7fe640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f970c0b41f0 0x7f970c0b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.923+0000 7f97027fc640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f970c0d22b0 0x7f970c0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.923+0000 7f970affd640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f970c15dfe0 0x7f970c0cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:60862/0 (socket says 172.21.15.83:60862) 2024-03-06T07:30:57.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.923+0000 7f970affd640 1 -- 172.21.15.83:0/3321042564 learned_addr learned my addr 172.21.15.83:0/3321042564 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:30:57.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.924+0000 7f970b7fe640 1 -- 172.21.15.83:0/3321042564 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f970c0d22b0 msgr2=0x7f970c0d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:57.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.924+0000 7f970b7fe640 1 --2- 172.21.15.83:0/3321042564 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f970c0d22b0 0x7f970c0d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.924+0000 7f970b7fe640 1 -- 172.21.15.83:0/3321042564 >> v1:172.21.15.183:6789/0 conn(0x7f970c15dfe0 legacy=0x7f970c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.924+0000 7f970a7fc640 1 -- 172.21.15.83:0/3321042564 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 252952925 0 0) 0x7f970c0b3430 con 0x7f970c15dfe0 2024-03-06T07:30:57.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.924+0000 7f970b7fe640 1 -- 172.21.15.83:0/3321042564 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f970c0b3710 con 0x7f970c0b41f0 2024-03-06T07:30:57.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.924+0000 7f97027fc640 1 --2- 172.21.15.83:0/3321042564 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f970c0d22b0 0x7f970c0d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:57.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.925+0000 7f970b7fe640 1 --2- 172.21.15.83:0/3321042564 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f970c0b41f0 0x7f970c0b4650 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f9704009d90 tx=0x7f970402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f442fdd2e35f951 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:57.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.925+0000 7f970a7fc640 1 -- 172.21.15.83:0/3321042564 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9704039070 con 0x7f970c0b41f0 2024-03-06T07:30:57.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.925+0000 7f970a7fc640 1 -- 172.21.15.83:0/3321042564 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f970402cc50 con 0x7f970c0b41f0 2024-03-06T07:30:57.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.925+0000 7f970a7fc640 1 -- 172.21.15.83:0/3321042564 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9704034440 con 0x7f970c0b41f0 2024-03-06T07:30:57.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.925+0000 7f971190c640 1 -- 172.21.15.83:0/3321042564 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f970c0b41f0 msgr2=0x7f970c0b4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.925+0000 7f971190c640 1 --2- 172.21.15.83:0/3321042564 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f970c0b41f0 0x7f970c0b4650 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f9704009d90 tx=0x7f970402bea0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.925+0000 7f971190c640 1 -- 172.21.15.83:0/3321042564 shutdown_connections 2024-03-06T07:30:57.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.925+0000 7f971190c640 1 --2- 172.21.15.83:0/3321042564 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f970c0b41f0 0x7f970c0b4650 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.925+0000 7f971190c640 1 --2- 172.21.15.83:0/3321042564 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f970c0d22b0 0x7f970c0d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.925+0000 7f971190c640 1 -- 172.21.15.83:0/3321042564 >> 172.21.15.83:0/3321042564 conn(0x7f970c0d3cd0 msgr2=0x7f970c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:57.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.925+0000 7f971190c640 1 -- 172.21.15.83:0/3321042564 shutdown_connections 2024-03-06T07:30:57.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.925+0000 7f971190c640 1 -- 172.21.15.83:0/3321042564 wait complete. 2024-03-06T07:30:57.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.926+0000 7f971190c640 1 Processor -- start 2024-03-06T07:30:57.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.927+0000 7f971190c640 1 -- start start 2024-03-06T07:30:57.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.927+0000 7f971190c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f970c0b41f0 0x7f970c15ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.927+0000 7f971190c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f970c0d22b0 0x7f970c0b87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.927+0000 7f971190c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f970c15dfe0 0x7f970c15c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.927+0000 7f971190c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f970c0cfa00 con 0x7f970c0d22b0 2024-03-06T07:30:57.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.927+0000 7f971190c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f970c0b3d60 con 0x7f970c15dfe0 2024-03-06T07:30:57.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.927+0000 7f971190c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f970c0d1c70 con 0x7f970c0b41f0 2024-03-06T07:30:57.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.927+0000 7f97027fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f970c0d22b0 0x7f970c0b87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.927+0000 7f97027fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f970c0d22b0 0x7f970c0b87d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52482/0 (socket says 172.21.15.83:52482) 2024-03-06T07:30:57.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.927+0000 7f97027fc640 1 -- 172.21.15.83:0/703995168 learned_addr learned my addr 172.21.15.83:0/703995168 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:57.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.928+0000 7f970b7fe640 1 --2- 172.21.15.83:0/703995168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f970c15dfe0 0x7f970c15c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.928+0000 7f970affd640 1 --2- 172.21.15.83:0/703995168 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f970c0b41f0 0x7f970c15ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.928+0000 7f97027fc640 1 -- 172.21.15.83:0/703995168 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f970c0b41f0 msgr2=0x7f970c15ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.928+0000 7f97027fc640 1 --2- 172.21.15.83:0/703995168 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f970c0b41f0 0x7f970c15ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.928+0000 7f97027fc640 1 -- 172.21.15.83:0/703995168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f970c15dfe0 msgr2=0x7f970c15c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.928+0000 7f97027fc640 1 --2- 172.21.15.83:0/703995168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f970c15dfe0 0x7f970c15c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.928+0000 7f97027fc640 1 -- 172.21.15.83:0/703995168 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f970c1f7da0 con 0x7f970c0d22b0 2024-03-06T07:30:57.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.928+0000 7f970affd640 1 --2- 172.21.15.83:0/703995168 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f970c0b41f0 0x7f970c15ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:57.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.928+0000 7f97027fc640 1 --2- 172.21.15.83:0/703995168 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f970c0d22b0 0x7f970c0b87d0 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f96f400ebe0 tx=0x7f96f400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:57.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.928+0000 7f9708ff9640 1 -- 172.21.15.83:0/703995168 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f4017070 con 0x7f970c0d22b0 2024-03-06T07:30:57.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.928+0000 7f9708ff9640 1 -- 172.21.15.83:0/703995168 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96f400cc30 con 0x7f970c0d22b0 2024-03-06T07:30:57.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.928+0000 7f971190c640 1 -- 172.21.15.83:0/703995168 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f970c15e5d0 con 0x7f970c0d22b0 2024-03-06T07:30:57.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.928+0000 7f9708ff9640 1 -- 172.21.15.83:0/703995168 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f4013470 con 0x7f970c0d22b0 2024-03-06T07:30:57.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.928+0000 7f971190c640 1 -- 172.21.15.83:0/703995168 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f970c15eb90 con 0x7f970c0d22b0 2024-03-06T07:30:57.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.929+0000 7f971190c640 1 -- 172.21.15.83:0/703995168 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f970c15e3c0 con 0x7f970c0d22b0 2024-03-06T07:30:57.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.929+0000 7f9708ff9640 1 -- 172.21.15.83:0/703995168 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f96f401a070 con 0x7f970c0d22b0 2024-03-06T07:30:57.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.930+0000 7f9708ff9640 1 --2- 172.21.15.83:0/703995168 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f96ec06b4d0 0x7f96ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.930+0000 7f970affd640 1 --2- 172.21.15.83:0/703995168 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f96ec06b4d0 0x7f96ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.930+0000 7f9708ff9640 1 -- 172.21.15.83:0/703995168 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f96f408a690 con 0x7f970c0d22b0 2024-03-06T07:30:57.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.930+0000 7f970affd640 1 --2- 172.21.15.83:0/703995168 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f96ec06b4d0 0x7f96ec06d990 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f96fc000f60 tx=0x7f96fc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:57.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:57.934+0000 7f9708ff9640 1 -- 172.21.15.83:0/703995168 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f96f40593b0 con 0x7f970c0d22b0 2024-03-06T07:30:58.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:58.178+0000 7f971190c640 1 -- 172.21.15.83:0/703995168 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f970c04f330 con 0x7f970c0d22b0 2024-03-06T07:30:58.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:58.179+0000 7f9708ff9640 1 -- 172.21.15.83:0/703995168 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f96f4058cb0 con 0x7f970c0d22b0 2024-03-06T07:30:58.181 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:30:58.181 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:30:58.181 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:30:58.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:58.181+0000 7f971190c640 1 -- 172.21.15.83:0/703995168 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f96ec06b4d0 msgr2=0x7f96ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:58.181+0000 7f971190c640 1 --2- 172.21.15.83:0/703995168 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f96ec06b4d0 0x7f96ec06d990 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f96fc000f60 tx=0x7f96fc009040 comp rx=0 tx=0).stop 2024-03-06T07:30:58.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:58.181+0000 7f971190c640 1 -- 172.21.15.83:0/703995168 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f970c0d22b0 msgr2=0x7f970c0b87d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:58.181+0000 7f971190c640 1 --2- 172.21.15.83:0/703995168 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f970c0d22b0 0x7f970c0b87d0 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f96f400ebe0 tx=0x7f96f400c520 comp rx=0 tx=0).stop 2024-03-06T07:30:58.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:58.181+0000 7f971190c640 1 -- 172.21.15.83:0/703995168 shutdown_connections 2024-03-06T07:30:58.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:58.181+0000 7f971190c640 1 --2- 172.21.15.83:0/703995168 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f96ec06b4d0 0x7f96ec06d990 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:58.181+0000 7f971190c640 1 --2- 172.21.15.83:0/703995168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f970c15dfe0 0x7f970c15c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:58.181+0000 7f971190c640 1 --2- 172.21.15.83:0/703995168 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f970c0d22b0 0x7f970c0b87d0 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:58.181+0000 7f971190c640 1 --2- 172.21.15.83:0/703995168 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f970c0b41f0 0x7f970c15ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:58.181+0000 7f971190c640 1 -- 172.21.15.83:0/703995168 >> 172.21.15.83:0/703995168 conn(0x7f970c0d3cd0 msgr2=0x7f970c0d06e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:58.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:58.181+0000 7f971190c640 1 -- 172.21.15.83:0/703995168 shutdown_connections 2024-03-06T07:30:58.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:58.181+0000 7f971190c640 1 -- 172.21.15.83:0/703995168 wait complete. 2024-03-06T07:30:58.395 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:30:58.395 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:30:59.060 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:59.396 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:59.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.517+0000 7fd6afe1c640 1 Processor -- start 2024-03-06T07:30:59.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.517+0000 7fd6afe1c640 1 -- start start 2024-03-06T07:30:59.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.518+0000 7fd6afe1c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd6a815dfe0 0x7fd6a80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.518+0000 7fd6afe1c640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fd6a80d29c0 0x7fd6a80b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.518+0000 7fd6afe1c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6a80b3430 con 0x7fd6a80cf830 2024-03-06T07:30:59.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.518+0000 7fd6afe1c640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6a80b31e0 con 0x7fd6a815dfe0 2024-03-06T07:30:59.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.518+0000 7fd6afe1c640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6a80b3d60 con 0x7fd6a80d29c0 2024-03-06T07:30:59.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.518+0000 7fd6adbb8640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd6a815dfe0 0x7fd6a80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.518+0000 7fd6adbb8640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd6a815dfe0 0x7fd6a80cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52492/0 (socket says 172.21.15.83:52492) 2024-03-06T07:30:59.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.518+0000 7fd6adbb8640 1 -- 172.21.15.83:0/381256838 learned_addr learned my addr 172.21.15.83:0/381256838 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:59.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.518+0000 7fd6ae3b9640 1 --2- 172.21.15.83:0/381256838 >> v2:172.21.15.183:3300/0 conn(0x7fd6a80d29c0 0x7fd6a80b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.518+0000 7fd6adbb8640 1 -- 172.21.15.83:0/381256838 >> v2:172.21.15.183:3300/0 conn(0x7fd6a80d29c0 msgr2=0x7fd6a80b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:59.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.518+0000 7fd6adbb8640 1 --2- 172.21.15.83:0/381256838 >> v2:172.21.15.183:3300/0 conn(0x7fd6a80d29c0 0x7fd6a80b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.518+0000 7fd6adbb8640 1 -- 172.21.15.83:0/381256838 >> v1:172.21.15.183:6789/0 conn(0x7fd6a80cf830 legacy=0x7fd6a80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:59.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.518+0000 7fd6adbb8640 1 -- 172.21.15.83:0/381256838 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a80b3710 con 0x7fd6a815dfe0 2024-03-06T07:30:59.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.519+0000 7fd6adbb8640 1 --2- 172.21.15.83:0/381256838 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6a815dfe0 0x7fd6a80cf2f0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7fd6a80b3cd0 tx=0x7fd6a402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b794aa8ea44343d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:59.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.519+0000 7fd6acbb6640 1 -- 172.21.15.83:0/381256838 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a4035020 con 0x7fd6a815dfe0 2024-03-06T07:30:59.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.519+0000 7fd6acbb6640 1 -- 172.21.15.83:0/381256838 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6a4031870 con 0x7fd6a815dfe0 2024-03-06T07:30:59.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.519+0000 7fd6acbb6640 1 -- 172.21.15.83:0/381256838 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a4031c40 con 0x7fd6a815dfe0 2024-03-06T07:30:59.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.520+0000 7fd6afe1c640 1 -- 172.21.15.83:0/381256838 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6a815dfe0 msgr2=0x7fd6a80cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.520+0000 7fd6afe1c640 1 --2- 172.21.15.83:0/381256838 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6a815dfe0 0x7fd6a80cf2f0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7fd6a80b3cd0 tx=0x7fd6a402b910 comp rx=0 tx=0).stop 2024-03-06T07:30:59.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.520+0000 7fd6afe1c640 1 -- 172.21.15.83:0/381256838 shutdown_connections 2024-03-06T07:30:59.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.520+0000 7fd6afe1c640 1 --2- 172.21.15.83:0/381256838 >> v2:172.21.15.183:3300/0 conn(0x7fd6a80d29c0 0x7fd6a80b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.520+0000 7fd6afe1c640 1 --2- 172.21.15.83:0/381256838 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6a815dfe0 0x7fd6a80cf2f0 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.520+0000 7fd6afe1c640 1 -- 172.21.15.83:0/381256838 >> 172.21.15.83:0/381256838 conn(0x7fd6a80d3cd0 msgr2=0x7fd6a80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:59.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.520+0000 7fd6afe1c640 1 -- 172.21.15.83:0/381256838 shutdown_connections 2024-03-06T07:30:59.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.520+0000 7fd6afe1c640 1 -- 172.21.15.83:0/381256838 wait complete. 2024-03-06T07:30:59.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.522+0000 7fd6afe1c640 1 Processor -- start 2024-03-06T07:30:59.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.522+0000 7fd6afe1c640 1 -- start start 2024-03-06T07:30:59.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.523+0000 7fd6afe1c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd6a80cf830 0x7fd6a8154070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.523+0000 7fd6afe1c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd6a80d29c0 0x7fd6a80cd500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.523+0000 7fd6afe1c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6a815dfe0 0x7fd6a80cda40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.523+0000 7fd6afe1c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a80d1df0 con 0x7fd6a815dfe0 2024-03-06T07:30:59.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.523+0000 7fd6afe1c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a80d1c70 con 0x7fd6a80d29c0 2024-03-06T07:30:59.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.523+0000 7fd6afe1c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a815e3c0 con 0x7fd6a80cf830 2024-03-06T07:30:59.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.523+0000 7fd6ae3b9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6a815dfe0 0x7fd6a80cda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.523+0000 7fd6ae3b9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6a815dfe0 0x7fd6a80cda40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52506/0 (socket says 172.21.15.83:52506) 2024-03-06T07:30:59.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.523+0000 7fd6ae3b9640 1 -- 172.21.15.83:0/2347587045 learned_addr learned my addr 172.21.15.83:0/2347587045 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:30:59.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.523+0000 7fd6ad3b7640 1 --2- 172.21.15.83:0/2347587045 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd6a80d29c0 0x7fd6a80cd500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.523+0000 7fd6adbb8640 1 --2- 172.21.15.83:0/2347587045 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd6a80cf830 0x7fd6a8154070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.523+0000 7fd6ae3b9640 1 -- 172.21.15.83:0/2347587045 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd6a80cf830 msgr2=0x7fd6a8154070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.523+0000 7fd6ae3b9640 1 --2- 172.21.15.83:0/2347587045 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd6a80cf830 0x7fd6a8154070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.523+0000 7fd6ae3b9640 1 -- 172.21.15.83:0/2347587045 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd6a80d29c0 msgr2=0x7fd6a80cd500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.523+0000 7fd6ae3b9640 1 --2- 172.21.15.83:0/2347587045 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd6a80d29c0 0x7fd6a80cd500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.524+0000 7fd6ae3b9640 1 -- 172.21.15.83:0/2347587045 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a80cbdf0 con 0x7fd6a815dfe0 2024-03-06T07:30:59.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.524+0000 7fd6adbb8640 1 --2- 172.21.15.83:0/2347587045 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd6a80cf830 0x7fd6a8154070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:59.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.524+0000 7fd6ae3b9640 1 --2- 172.21.15.83:0/2347587045 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6a815dfe0 0x7fd6a80cda40 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7fd6a000cd40 tx=0x7fd6a000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:59.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.524+0000 7fd69e7fc640 1 -- 172.21.15.83:0/2347587045 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a0003b90 con 0x7fd6a815dfe0 2024-03-06T07:30:59.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.524+0000 7fd69e7fc640 1 -- 172.21.15.83:0/2347587045 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6a0003d30 con 0x7fd6a815dfe0 2024-03-06T07:30:59.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.524+0000 7fd6afe1c640 1 -- 172.21.15.83:0/2347587045 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6a80cc0d0 con 0x7fd6a815dfe0 2024-03-06T07:30:59.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.524+0000 7fd69e7fc640 1 -- 172.21.15.83:0/2347587045 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a0004240 con 0x7fd6a815dfe0 2024-03-06T07:30:59.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.524+0000 7fd6afe1c640 1 -- 172.21.15.83:0/2347587045 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6a80cc690 con 0x7fd6a815dfe0 2024-03-06T07:30:59.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.525+0000 7fd6afe1c640 1 -- 172.21.15.83:0/2347587045 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6a804f330 con 0x7fd6a815dfe0 2024-03-06T07:30:59.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.526+0000 7fd69e7fc640 1 -- 172.21.15.83:0/2347587045 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd6a00043e0 con 0x7fd6a815dfe0 2024-03-06T07:30:59.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.526+0000 7fd69e7fc640 1 --2- 172.21.15.83:0/2347587045 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd68c06b4d0 0x7fd68c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.526+0000 7fd69e7fc640 1 -- 172.21.15.83:0/2347587045 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd6a008aca0 con 0x7fd6a815dfe0 2024-03-06T07:30:59.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.526+0000 7fd6adbb8640 1 --2- 172.21.15.83:0/2347587045 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd68c06b4d0 0x7fd68c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.527+0000 7fd6adbb8640 1 --2- 172.21.15.83:0/2347587045 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd68c06b4d0 0x7fd68c06d990 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fd6a4036000 tx=0x7fd6a403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:59.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.531+0000 7fd69e7fc640 1 -- 172.21.15.83:0/2347587045 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd6a0059ad0 con 0x7fd6a815dfe0 2024-03-06T07:30:59.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.745+0000 7fd6afe1c640 1 -- 172.21.15.83:0/2347587045 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6a80b0de0 con 0x7fd6a815dfe0 2024-03-06T07:30:59.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.746+0000 7fd69e7fc640 1 -- 172.21.15.83:0/2347587045 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7fd6a00593d0 con 0x7fd6a815dfe0 2024-03-06T07:30:59.748 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:30:59.748 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:30:59.748 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:30:59.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.747+0000 7fd6afe1c640 1 -- 172.21.15.83:0/2347587045 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd68c06b4d0 msgr2=0x7fd68c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.747+0000 7fd6afe1c640 1 --2- 172.21.15.83:0/2347587045 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd68c06b4d0 0x7fd68c06d990 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fd6a4036000 tx=0x7fd6a403c040 comp rx=0 tx=0).stop 2024-03-06T07:30:59.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.747+0000 7fd6afe1c640 1 -- 172.21.15.83:0/2347587045 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6a815dfe0 msgr2=0x7fd6a80cda40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.747+0000 7fd6afe1c640 1 --2- 172.21.15.83:0/2347587045 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6a815dfe0 0x7fd6a80cda40 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7fd6a000cd40 tx=0x7fd6a000a660 comp rx=0 tx=0).stop 2024-03-06T07:30:59.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.747+0000 7fd6afe1c640 1 -- 172.21.15.83:0/2347587045 shutdown_connections 2024-03-06T07:30:59.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.747+0000 7fd6afe1c640 1 --2- 172.21.15.83:0/2347587045 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd68c06b4d0 0x7fd68c06d990 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.747+0000 7fd6afe1c640 1 --2- 172.21.15.83:0/2347587045 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6a815dfe0 0x7fd6a80cda40 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.747+0000 7fd6afe1c640 1 --2- 172.21.15.83:0/2347587045 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd6a80d29c0 0x7fd6a80cd500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.747+0000 7fd6afe1c640 1 --2- 172.21.15.83:0/2347587045 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd6a80cf830 0x7fd6a8154070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.747+0000 7fd6afe1c640 1 -- 172.21.15.83:0/2347587045 >> 172.21.15.83:0/2347587045 conn(0x7fd6a80d3cd0 msgr2=0x7fd6a80b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:59.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.748+0000 7fd6afe1c640 1 -- 172.21.15.83:0/2347587045 shutdown_connections 2024-03-06T07:30:59.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:30:59.748+0000 7fd6afe1c640 1 -- 172.21.15.83:0/2347587045 wait complete. 2024-03-06T07:30:59.961 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:30:59.962 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:31:00.581 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:00.584 DEBUG:teuthology.orchestra.run.smithi183:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:00.963 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:01.062 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:01.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.064+0000 7f1a643f7640 1 Processor -- start 2024-03-06T07:31:01.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.064+0000 7f1a643f7640 1 -- start start 2024-03-06T07:31:01.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.065+0000 7f1a643f7640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f1a5c0d22b0 0x7f1a5c0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.065+0000 7f1a643f7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a5c0b3430 con 0x7f1a5c15dfe0 2024-03-06T07:31:01.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.065+0000 7f1a643f7640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a5c0b3710 con 0x7f1a5c0b41f0 2024-03-06T07:31:01.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.065+0000 7f1a643f7640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a5c0d2c50 con 0x7f1a5c0d22b0 2024-03-06T07:31:01.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.065+0000 7f1a62193640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f1a5c15dfe0 0x7f1a5c0cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60662/0 (socket says 172.21.15.83:60662) 2024-03-06T07:31:01.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.065+0000 7f1a62193640 1 -- 172.21.15.83:0/3128001671 learned_addr learned my addr 172.21.15.83:0/3128001671 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:31:01.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.065+0000 7f1a61191640 1 -- 172.21.15.83:0/3128001671 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2436047543 0 0) 0x7f1a5c0b3430 con 0x7f1a5c15dfe0 2024-03-06T07:31:01.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.066+0000 7f1a61191640 1 -- 172.21.15.83:0/3128001671 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a48003520 con 0x7f1a5c15dfe0 2024-03-06T07:31:01.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.066+0000 7f1a61992640 1 --2- 172.21.15.83:0/3128001671 >> v2:172.21.15.183:3300/0 conn(0x7f1a5c0d22b0 0x7f1a5c0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.066+0000 7f1a61191640 1 -- 172.21.15.83:0/3128001671 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4024678743 0 0) 0x7f1a48003520 con 0x7f1a5c15dfe0 2024-03-06T07:31:01.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.066+0000 7f1a61191640 1 -- 172.21.15.83:0/3128001671 >> v2:172.21.15.183:3300/0 conn(0x7f1a5c0d22b0 msgr2=0x7f1a5c0d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:01.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.066+0000 7f1a61191640 1 --2- 172.21.15.83:0/3128001671 >> v2:172.21.15.183:3300/0 conn(0x7f1a5c0d22b0 0x7f1a5c0d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.066+0000 7f1a61191640 1 -- 172.21.15.83:0/3128001671 >> v1:172.21.15.183:6789/0 conn(0x7f1a5c0b41f0 legacy=0x7f1a5c0b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:01.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.066+0000 7f1a61191640 1 -- 172.21.15.83:0/3128001671 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a5c0b39f0 con 0x7f1a5c15dfe0 2024-03-06T07:31:01.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.066+0000 7f1a61191640 1 -- 172.21.15.83:0/3128001671 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f1a58001670 con 0x7f1a5c15dfe0 2024-03-06T07:31:01.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.066+0000 7f1a61191640 1 -- 172.21.15.83:0/3128001671 >> v1:172.21.15.83:6789/0 conn(0x7f1a5c15dfe0 legacy=0x7f1a5c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.066+0000 7f1a61191640 1 --2- 172.21.15.83:0/3128001671 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1a480039b0 0x7f1a48003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.066+0000 7f1a61191640 1 --2- 172.21.15.83:0/3128001671 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a48004800 0x7f1a48004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.066+0000 7f1a61191640 1 --2- 172.21.15.83:0/3128001671 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a480051c0 0x7f1a4800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.066+0000 7f1a61191640 1 -- 172.21.15.83:0/3128001671 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4800c0c0 con 0x7f1a48004800 2024-03-06T07:31:01.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.066+0000 7f1a61191640 1 -- 172.21.15.83:0/3128001671 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4800c270 con 0x7f1a480051c0 2024-03-06T07:31:01.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.066+0000 7f1a61191640 1 -- 172.21.15.83:0/3128001671 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4800c470 con 0x7f1a480039b0 2024-03-06T07:31:01.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.066+0000 7f1a61992640 1 --2- 172.21.15.83:0/3128001671 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a48004800 0x7f1a48004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.066+0000 7f1a62994640 1 --2- 172.21.15.83:0/3128001671 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a480051c0 0x7f1a4800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.066+0000 7f1a62193640 1 --2- 172.21.15.83:0/3128001671 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1a480039b0 0x7f1a48003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.066+0000 7f1a61992640 1 -- 172.21.15.83:0/3128001671 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1a480039b0 msgr2=0x7f1a48003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:01.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.066+0000 7f1a61992640 1 --2- 172.21.15.83:0/3128001671 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1a480039b0 0x7f1a48003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.066+0000 7f1a61992640 1 -- 172.21.15.83:0/3128001671 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a480051c0 msgr2=0x7f1a4800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:01.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.066+0000 7f1a61992640 1 --2- 172.21.15.83:0/3128001671 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a480051c0 0x7f1a4800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.066+0000 7f1a61992640 1 -- 172.21.15.83:0/3128001671 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a48003520 con 0x7f1a48004800 2024-03-06T07:31:01.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.066+0000 7f1a62994640 1 --2- 172.21.15.83:0/3128001671 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a480051c0 0x7f1a4800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:01.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.066+0000 7f1a62994640 1 -- 172.21.15.83:0/3128001671 reap_dead start 2024-03-06T07:31:01.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.067+0000 7f1a61992640 1 --2- 172.21.15.83:0/3128001671 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a48004800 0x7f1a48004c80 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c00ba00 tx=0x7f1a48005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f071e04dc6b68952 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:01.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.067+0000 7f1a61191640 1 -- 172.21.15.83:0/3128001671 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c036020 con 0x7f1a48004800 2024-03-06T07:31:01.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.067+0000 7f1a61191640 1 -- 172.21.15.83:0/3128001671 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a4c032620 con 0x7f1a48004800 2024-03-06T07:31:01.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.067+0000 7f1a61191640 1 -- 172.21.15.83:0/3128001671 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c00b600 con 0x7f1a48004800 2024-03-06T07:31:01.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.068+0000 7f1a643f7640 1 -- 172.21.15.83:0/3128001671 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a48004800 msgr2=0x7f1a48004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.068+0000 7f1a643f7640 1 --2- 172.21.15.83:0/3128001671 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a48004800 0x7f1a48004c80 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c00ba00 tx=0x7f1a48005830 comp rx=0 tx=0).stop 2024-03-06T07:31:01.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.068+0000 7f1a643f7640 1 -- 172.21.15.83:0/3128001671 shutdown_connections 2024-03-06T07:31:01.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.068+0000 7f1a643f7640 1 --2- 172.21.15.83:0/3128001671 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a48004800 0x7f1a48004c80 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.068+0000 7f1a643f7640 1 -- 172.21.15.83:0/3128001671 >> 172.21.15.83:0/3128001671 conn(0x7f1a5c0d3cd0 msgr2=0x7f1a5c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:01.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.068+0000 7f1a643f7640 1 -- 172.21.15.83:0/3128001671 shutdown_connections 2024-03-06T07:31:01.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.068+0000 7f1a643f7640 1 -- 172.21.15.83:0/3128001671 wait complete. 2024-03-06T07:31:01.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.069+0000 7f1a643f7640 1 Processor -- start 2024-03-06T07:31:01.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.069+0000 7f1a643f7640 1 -- start start 2024-03-06T07:31:01.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.070+0000 7f1a643f7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1a48004800 0x7f1a5c160db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.070+0000 7f1a643f7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a5c0b7630 0x7f1a5c0cd340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.070+0000 7f1a643f7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a5c0cb010 0x7f1a5c1f2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.070+0000 7f1a643f7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c0b3d60 con 0x7f1a5c0cb010 2024-03-06T07:31:01.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.070+0000 7f1a643f7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c0b31e0 con 0x7f1a5c0b7630 2024-03-06T07:31:01.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.070+0000 7f1a643f7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c15e3c0 con 0x7f1a48004800 2024-03-06T07:31:01.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.070+0000 7f1a62994640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a5c0cb010 0x7f1a5c1f2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.070+0000 7f1a62994640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a5c0cb010 0x7f1a5c1f2c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52526/0 (socket says 172.21.15.83:52526) 2024-03-06T07:31:01.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.070+0000 7f1a62994640 1 -- 172.21.15.83:0/281102999 learned_addr learned my addr 172.21.15.83:0/281102999 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:01.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.070+0000 7f1a62193640 1 --2- 172.21.15.83:0/281102999 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1a48004800 0x7f1a5c160db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.070+0000 7f1a61992640 1 --2- 172.21.15.83:0/281102999 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a5c0b7630 0x7f1a5c0cd340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.070+0000 7f1a62994640 1 -- 172.21.15.83:0/281102999 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1a48004800 msgr2=0x7f1a5c160db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.070+0000 7f1a62994640 1 --2- 172.21.15.83:0/281102999 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1a48004800 0x7f1a5c160db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.070+0000 7f1a62994640 1 -- 172.21.15.83:0/281102999 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a5c0b7630 msgr2=0x7f1a5c0cd340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.070+0000 7f1a62994640 1 --2- 172.21.15.83:0/281102999 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a5c0b7630 0x7f1a5c0cd340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.070+0000 7f1a62994640 1 -- 172.21.15.83:0/281102999 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a5c0cd880 con 0x7f1a5c0cb010 2024-03-06T07:31:01.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.070+0000 7f1a61992640 1 --2- 172.21.15.83:0/281102999 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a5c0b7630 0x7f1a5c0cd340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:01.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.070+0000 7f1a62994640 1 --2- 172.21.15.83:0/281102999 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a5c0cb010 0x7f1a5c1f2c20 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c08c940 tx=0x7f1a5400eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:01.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.070+0000 7f1a52ffd640 1 -- 172.21.15.83:0/281102999 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a54006b50 con 0x7f1a5c0cb010 2024-03-06T07:31:01.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.070+0000 7f1a52ffd640 1 -- 172.21.15.83:0/281102999 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a54006cf0 con 0x7f1a5c0cb010 2024-03-06T07:31:01.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.070+0000 7f1a643f7640 1 -- 172.21.15.83:0/281102999 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a5c1614b0 con 0x7f1a5c0cb010 2024-03-06T07:31:01.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.070+0000 7f1a52ffd640 1 -- 172.21.15.83:0/281102999 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a54004b00 con 0x7f1a5c0cb010 2024-03-06T07:31:01.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.070+0000 7f1a643f7640 1 -- 172.21.15.83:0/281102999 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a5c15ea30 con 0x7f1a5c0cb010 2024-03-06T07:31:01.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.071+0000 7f1a643f7640 1 -- 172.21.15.83:0/281102999 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a5c04f330 con 0x7f1a5c0cb010 2024-03-06T07:31:01.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.072+0000 7f1a52ffd640 1 -- 172.21.15.83:0/281102999 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1a54004ca0 con 0x7f1a5c0cb010 2024-03-06T07:31:01.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.072+0000 7f1a52ffd640 1 --2- 172.21.15.83:0/281102999 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1a4006b500 0x7f1a4006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.072+0000 7f1a62193640 1 --2- 172.21.15.83:0/281102999 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1a4006b500 0x7f1a4006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.072+0000 7f1a52ffd640 1 -- 172.21.15.83:0/281102999 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1a54089b20 con 0x7f1a5c0cb010 2024-03-06T07:31:01.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.073+0000 7f1a62193640 1 --2- 172.21.15.83:0/281102999 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1a4006b500 0x7f1a4006d9c0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f1a58001ce0 tx=0x7f1a58001db0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:01.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.077+0000 7f1a52ffd640 1 -- 172.21.15.83:0/281102999 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1a54058960 con 0x7f1a5c0cb010 2024-03-06T07:31:01.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.285+0000 7f1a643f7640 1 -- 172.21.15.83:0/281102999 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a5c0d03f0 con 0x7f1a5c0cb010 2024-03-06T07:31:01.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.286+0000 7f1a52ffd640 1 -- 172.21.15.83:0/281102999 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f1a54058260 con 0x7f1a5c0cb010 2024-03-06T07:31:01.289 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:31:01.289 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:31:01.290 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:31:01.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.288+0000 7f1a643f7640 1 -- 172.21.15.83:0/281102999 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1a4006b500 msgr2=0x7f1a4006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.288+0000 7f1a643f7640 1 --2- 172.21.15.83:0/281102999 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1a4006b500 0x7f1a4006d9c0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f1a58001ce0 tx=0x7f1a58001db0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.288+0000 7f1a643f7640 1 -- 172.21.15.83:0/281102999 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a5c0cb010 msgr2=0x7f1a5c1f2c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.288+0000 7f1a643f7640 1 --2- 172.21.15.83:0/281102999 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a5c0cb010 0x7f1a5c1f2c20 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c08c940 tx=0x7f1a5400eb90 comp rx=0 tx=0).stop 2024-03-06T07:31:01.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.288+0000 7f1a643f7640 1 -- 172.21.15.83:0/281102999 shutdown_connections 2024-03-06T07:31:01.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.288+0000 7f1a643f7640 1 --2- 172.21.15.83:0/281102999 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1a4006b500 0x7f1a4006d9c0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.288+0000 7f1a643f7640 1 --2- 172.21.15.83:0/281102999 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1a5c0cb010 0x7f1a5c1f2c20 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.288+0000 7f1a643f7640 1 --2- 172.21.15.83:0/281102999 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a5c0b7630 0x7f1a5c0cd340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.288+0000 7f1a643f7640 1 --2- 172.21.15.83:0/281102999 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1a48004800 0x7f1a5c160db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.288+0000 7f1a643f7640 1 -- 172.21.15.83:0/281102999 >> 172.21.15.83:0/281102999 conn(0x7f1a5c0d3cd0 msgr2=0x7f1a5c0d0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:01.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.288+0000 7f1a643f7640 1 -- 172.21.15.83:0/281102999 shutdown_connections 2024-03-06T07:31:01.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:01.288+0000 7f1a643f7640 1 -- 172.21.15.83:0/281102999 wait complete. 2024-03-06T07:31:01.503 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:31:01.503 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:31:02.504 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:02.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.614+0000 7fc5c1d7a640 1 Processor -- start 2024-03-06T07:31:02.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.614+0000 7fc5c1d7a640 1 -- start start 2024-03-06T07:31:02.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.614+0000 7fc5c1d7a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc5bc15dfe0 0x7fc5bc0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.614+0000 7fc5c1d7a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fc5bc0cf830 0x7fc5bc0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.614+0000 7fc5c1d7a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5bc0b3430 con 0x7fc5bc0d27f0 2024-03-06T07:31:02.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.615+0000 7fc5c1d7a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5bc0b31e0 con 0x7fc5bc15dfe0 2024-03-06T07:31:02.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.615+0000 7fc5c1d7a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5bc0b3d60 con 0x7fc5bc0cf830 2024-03-06T07:31:02.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.615+0000 7fc5bb7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc5bc15dfe0 0x7fc5bc0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.615+0000 7fc5bbfff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fc5bc0d27f0 0x7fc5bc0d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60672/0 (socket says 172.21.15.83:60672) 2024-03-06T07:31:02.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.615+0000 7fc5bbfff640 1 -- 172.21.15.83:0/3130719538 learned_addr learned my addr 172.21.15.83:0/3130719538 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:31:02.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.615+0000 7fc5baffd640 1 --2- 172.21.15.83:0/3130719538 >> v2:172.21.15.183:3300/0 conn(0x7fc5bc0cf830 0x7fc5bc0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.615+0000 7fc5bb7fe640 1 -- 172.21.15.83:0/3130719538 >> v2:172.21.15.183:3300/0 conn(0x7fc5bc0cf830 msgr2=0x7fc5bc0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:02.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.615+0000 7fc5bb7fe640 1 --2- 172.21.15.83:0/3130719538 >> v2:172.21.15.183:3300/0 conn(0x7fc5bc0cf830 0x7fc5bc0d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.615+0000 7fc5ba7fc640 1 -- 172.21.15.83:0/3130719538 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3785550840 0 0) 0x7fc5bc0b3430 con 0x7fc5bc0d27f0 2024-03-06T07:31:02.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.615+0000 7fc5bb7fe640 1 -- 172.21.15.83:0/3130719538 >> v1:172.21.15.83:6789/0 conn(0x7fc5bc0d27f0 legacy=0x7fc5bc0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.615+0000 7fc5bb7fe640 1 -- 172.21.15.83:0/3130719538 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5bc0b3710 con 0x7fc5bc15dfe0 2024-03-06T07:31:02.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.615+0000 7fc5baffd640 1 --2- 172.21.15.83:0/3130719538 >> v2:172.21.15.183:3300/0 conn(0x7fc5bc0cf830 0x7fc5bc0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:02.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.616+0000 7fc5bb7fe640 1 --2- 172.21.15.83:0/3130719538 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5bc15dfe0 0x7fc5bc0cf2f0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac0097a0 tx=0x7fc5ac02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc533d0c28502b8b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:02.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.616+0000 7fc5ba7fc640 1 -- 172.21.15.83:0/3130719538 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac035020 con 0x7fc5bc15dfe0 2024-03-06T07:31:02.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.616+0000 7fc5ba7fc640 1 -- 172.21.15.83:0/3130719538 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5ac031830 con 0x7fc5bc15dfe0 2024-03-06T07:31:02.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.616+0000 7fc5ba7fc640 1 -- 172.21.15.83:0/3130719538 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac031c00 con 0x7fc5bc15dfe0 2024-03-06T07:31:02.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.616+0000 7fc5c1d7a640 1 -- 172.21.15.83:0/3130719538 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5bc15dfe0 msgr2=0x7fc5bc0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.616+0000 7fc5c1d7a640 1 --2- 172.21.15.83:0/3130719538 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5bc15dfe0 0x7fc5bc0cf2f0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac0097a0 tx=0x7fc5ac02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.616+0000 7fc5c1d7a640 1 -- 172.21.15.83:0/3130719538 shutdown_connections 2024-03-06T07:31:02.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.616+0000 7fc5c1d7a640 1 --2- 172.21.15.83:0/3130719538 >> v2:172.21.15.183:3300/0 conn(0x7fc5bc0cf830 0x7fc5bc0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.616+0000 7fc5c1d7a640 1 --2- 172.21.15.83:0/3130719538 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5bc15dfe0 0x7fc5bc0cf2f0 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.616+0000 7fc5c1d7a640 1 -- 172.21.15.83:0/3130719538 >> 172.21.15.83:0/3130719538 conn(0x7fc5bc0d3cd0 msgr2=0x7fc5bc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:02.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.616+0000 7fc5c1d7a640 1 -- 172.21.15.83:0/3130719538 shutdown_connections 2024-03-06T07:31:02.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.616+0000 7fc5c1d7a640 1 -- 172.21.15.83:0/3130719538 wait complete. 2024-03-06T07:31:02.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.618+0000 7fc5c1d7a640 1 Processor -- start 2024-03-06T07:31:02.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.618+0000 7fc5c1d7a640 1 -- start start 2024-03-06T07:31:02.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.618+0000 7fc5c1d7a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5bc0cf830 0x7fc5bc0b8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.618+0000 7fc5c1d7a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5bc0d27f0 0x7fc5bc1f3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.618+0000 7fc5c1d7a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc5bc15dfe0 0x7fc5bc15b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.618+0000 7fc5c1d7a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5bc0d1df0 con 0x7fc5bc0d27f0 2024-03-06T07:31:02.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.618+0000 7fc5c1d7a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5bc0d1c70 con 0x7fc5bc0cf830 2024-03-06T07:31:02.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.618+0000 7fc5c1d7a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5bc15e3c0 con 0x7fc5bc15dfe0 2024-03-06T07:31:02.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.619+0000 7fc5baffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5bc0d27f0 0x7fc5bc1f3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.619+0000 7fc5baffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5bc0d27f0 0x7fc5bc1f3d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52532/0 (socket says 172.21.15.83:52532) 2024-03-06T07:31:02.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.619+0000 7fc5baffd640 1 -- 172.21.15.83:0/1506599214 learned_addr learned my addr 172.21.15.83:0/1506599214 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:02.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.619+0000 7fc5bbfff640 1 --2- 172.21.15.83:0/1506599214 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc5bc15dfe0 0x7fc5bc15b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.619+0000 7fc5bb7fe640 1 --2- 172.21.15.83:0/1506599214 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5bc0cf830 0x7fc5bc0b8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.619+0000 7fc5baffd640 1 -- 172.21.15.83:0/1506599214 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc5bc15dfe0 msgr2=0x7fc5bc15b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.619+0000 7fc5baffd640 1 --2- 172.21.15.83:0/1506599214 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc5bc15dfe0 0x7fc5bc15b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.619+0000 7fc5baffd640 1 -- 172.21.15.83:0/1506599214 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5bc0cf830 msgr2=0x7fc5bc0b8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.619+0000 7fc5baffd640 1 --2- 172.21.15.83:0/1506599214 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5bc0cf830 0x7fc5bc0b8bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.619+0000 7fc5baffd640 1 -- 172.21.15.83:0/1506599214 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5bc0b95a0 con 0x7fc5bc0d27f0 2024-03-06T07:31:02.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.619+0000 7fc5bb7fe640 1 --2- 172.21.15.83:0/1506599214 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5bc0cf830 0x7fc5bc0b8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:02.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.619+0000 7fc5baffd640 1 --2- 172.21.15.83:0/1506599214 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5bc0d27f0 0x7fc5bc1f3d50 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7fc5b000d800 tx=0x7fc5b000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:02.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.619+0000 7fc5a3fff640 1 -- 172.21.15.83:0/1506599214 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0004030 con 0x7fc5bc0d27f0 2024-03-06T07:31:02.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.619+0000 7fc5a3fff640 1 -- 172.21.15.83:0/1506599214 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b00041d0 con 0x7fc5bc0d27f0 2024-03-06T07:31:02.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.619+0000 7fc5c1d7a640 1 -- 172.21.15.83:0/1506599214 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5bc1f6d60 con 0x7fc5bc0d27f0 2024-03-06T07:31:02.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.619+0000 7fc5a3fff640 1 -- 172.21.15.83:0/1506599214 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b00096a0 con 0x7fc5bc0d27f0 2024-03-06T07:31:02.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.619+0000 7fc5c1d7a640 1 -- 172.21.15.83:0/1506599214 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5bc1f7320 con 0x7fc5bc0d27f0 2024-03-06T07:31:02.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.620+0000 7fc5c1d7a640 1 -- 172.21.15.83:0/1506599214 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5bc04f330 con 0x7fc5bc0d27f0 2024-03-06T07:31:02.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.621+0000 7fc5a3fff640 1 -- 172.21.15.83:0/1506599214 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc5b0009840 con 0x7fc5bc0d27f0 2024-03-06T07:31:02.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.621+0000 7fc5a3fff640 1 --2- 172.21.15.83:0/1506599214 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc5a806b4d0 0x7fc5a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.621+0000 7fc5bb7fe640 1 --2- 172.21.15.83:0/1506599214 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc5a806b4d0 0x7fc5a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.621+0000 7fc5a3fff640 1 -- 172.21.15.83:0/1506599214 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc5b008a480 con 0x7fc5bc0d27f0 2024-03-06T07:31:02.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.621+0000 7fc5bb7fe640 1 --2- 172.21.15.83:0/1506599214 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc5a806b4d0 0x7fc5a806d990 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac036000 tx=0x7fc5ac008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:02.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.630+0000 7fc5a3fff640 1 -- 172.21.15.83:0/1506599214 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc5b0059350 con 0x7fc5bc0d27f0 2024-03-06T07:31:02.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.842+0000 7fc5c1d7a640 1 -- 172.21.15.83:0/1506599214 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5bc0747a0 con 0x7fc5bc0d27f0 2024-03-06T07:31:02.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.843+0000 7fc5a3fff640 1 -- 172.21.15.83:0/1506599214 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7fc5b0058c50 con 0x7fc5bc0d27f0 2024-03-06T07:31:02.848 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:31:02.848 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:31:02.849 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:31:02.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.845+0000 7fc5c1d7a640 1 -- 172.21.15.83:0/1506599214 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc5a806b4d0 msgr2=0x7fc5a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.845+0000 7fc5c1d7a640 1 --2- 172.21.15.83:0/1506599214 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc5a806b4d0 0x7fc5a806d990 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac036000 tx=0x7fc5ac008030 comp rx=0 tx=0).stop 2024-03-06T07:31:02.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.845+0000 7fc5c1d7a640 1 -- 172.21.15.83:0/1506599214 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5bc0d27f0 msgr2=0x7fc5bc1f3d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.845+0000 7fc5c1d7a640 1 --2- 172.21.15.83:0/1506599214 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5bc0d27f0 0x7fc5bc1f3d50 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7fc5b000d800 tx=0x7fc5b000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.846+0000 7fc5c1d7a640 1 -- 172.21.15.83:0/1506599214 shutdown_connections 2024-03-06T07:31:02.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.846+0000 7fc5c1d7a640 1 --2- 172.21.15.83:0/1506599214 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc5a806b4d0 0x7fc5a806d990 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.846+0000 7fc5c1d7a640 1 --2- 172.21.15.83:0/1506599214 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc5bc15dfe0 0x7fc5bc15b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.846+0000 7fc5c1d7a640 1 --2- 172.21.15.83:0/1506599214 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5bc0d27f0 0x7fc5bc1f3d50 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.846+0000 7fc5c1d7a640 1 --2- 172.21.15.83:0/1506599214 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5bc0cf830 0x7fc5bc0b8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.846+0000 7fc5c1d7a640 1 -- 172.21.15.83:0/1506599214 >> 172.21.15.83:0/1506599214 conn(0x7fc5bc0d3cd0 msgr2=0x7fc5bc0d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:02.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.846+0000 7fc5c1d7a640 1 -- 172.21.15.83:0/1506599214 shutdown_connections 2024-03-06T07:31:02.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:02.846+0000 7fc5c1d7a640 1 -- 172.21.15.83:0/1506599214 wait complete. 2024-03-06T07:31:03.060 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:31:03.060 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:31:03.064 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:04.061 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:04.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.160+0000 7f8062de2640 1 Processor -- start 2024-03-06T07:31:04.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.161+0000 7f8062de2640 1 -- start start 2024-03-06T07:31:04.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.162+0000 7f8062de2640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f805c15dfe0 0x7f805c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.162+0000 7f8062de2640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f805c0b3430 con 0x7f805c0cf830 2024-03-06T07:31:04.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.162+0000 7f8062de2640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f805c0b3710 con 0x7f805c0d29c0 2024-03-06T07:31:04.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.162+0000 7f8062de2640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f805c0b31e0 con 0x7f805c15dfe0 2024-03-06T07:31:04.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.162+0000 7f805bfff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f805c0cf830 0x7f805c0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60682/0 (socket says 172.21.15.83:60682) 2024-03-06T07:31:04.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.162+0000 7f805bfff640 1 -- 172.21.15.83:0/3462170887 learned_addr learned my addr 172.21.15.83:0/3462170887 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:31:04.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.162+0000 7f805b7fe640 1 -- 172.21.15.83:0/3462170887 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1523835988 0 0) 0x7f805c0b3430 con 0x7f805c0cf830 2024-03-06T07:31:04.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.162+0000 7f805b7fe640 1 -- 172.21.15.83:0/3462170887 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8048003520 con 0x7f805c0cf830 2024-03-06T07:31:04.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.162+0000 7f8060b7e640 1 --2- 172.21.15.83:0/3462170887 >> v2:172.21.15.183:3300/0 conn(0x7f805c15dfe0 0x7f805c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.163+0000 7f805b7fe640 1 -- 172.21.15.83:0/3462170887 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 541294193 0 0) 0x7f8048003520 con 0x7f805c0cf830 2024-03-06T07:31:04.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.163+0000 7f805b7fe640 1 -- 172.21.15.83:0/3462170887 >> v2:172.21.15.183:3300/0 conn(0x7f805c15dfe0 msgr2=0x7f805c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:04.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.163+0000 7f805b7fe640 1 --2- 172.21.15.83:0/3462170887 >> v2:172.21.15.183:3300/0 conn(0x7f805c15dfe0 0x7f805c0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.163+0000 7f805b7fe640 1 -- 172.21.15.83:0/3462170887 >> v1:172.21.15.183:6789/0 conn(0x7f805c0d29c0 legacy=0x7f805c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.163+0000 7f805b7fe640 1 -- 172.21.15.83:0/3462170887 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f805c0b39f0 con 0x7f805c0cf830 2024-03-06T07:31:04.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.163+0000 7f805b7fe640 1 -- 172.21.15.83:0/3462170887 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f804c0019f0 con 0x7f805c0cf830 2024-03-06T07:31:04.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.163+0000 7f805b7fe640 1 -- 172.21.15.83:0/3462170887 >> v1:172.21.15.83:6789/0 conn(0x7f805c0cf830 legacy=0x7f805c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.163+0000 7f805b7fe640 1 --2- 172.21.15.83:0/3462170887 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f80480039b0 0x7f8048003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.163+0000 7f805b7fe640 1 --2- 172.21.15.83:0/3462170887 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8048004800 0x7f8048004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.163+0000 7f805b7fe640 1 --2- 172.21.15.83:0/3462170887 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f80480051c0 0x7f804800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.163+0000 7f805b7fe640 1 -- 172.21.15.83:0/3462170887 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f805c0b31e0 con 0x7f8048004800 2024-03-06T07:31:04.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.163+0000 7f805b7fe640 1 -- 172.21.15.83:0/3462170887 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f804800c090 con 0x7f80480051c0 2024-03-06T07:31:04.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.163+0000 7f805b7fe640 1 -- 172.21.15.83:0/3462170887 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f804800c290 con 0x7f80480039b0 2024-03-06T07:31:04.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.163+0000 7f805bfff640 1 --2- 172.21.15.83:0/3462170887 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8048004800 0x7f8048004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.163+0000 7f806137f640 1 --2- 172.21.15.83:0/3462170887 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f80480051c0 0x7f804800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.163+0000 7f8060b7e640 1 --2- 172.21.15.83:0/3462170887 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f80480039b0 0x7f8048003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.163+0000 7f805bfff640 1 -- 172.21.15.83:0/3462170887 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f80480039b0 msgr2=0x7f8048003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:04.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.163+0000 7f805bfff640 1 --2- 172.21.15.83:0/3462170887 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f80480039b0 0x7f8048003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.163+0000 7f805bfff640 1 -- 172.21.15.83:0/3462170887 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f80480051c0 msgr2=0x7f804800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:04.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.163+0000 7f805bfff640 1 --2- 172.21.15.83:0/3462170887 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f80480051c0 0x7f804800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.163+0000 7f805bfff640 1 -- 172.21.15.83:0/3462170887 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8048003520 con 0x7f8048004800 2024-03-06T07:31:04.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.163+0000 7f806137f640 1 --2- 172.21.15.83:0/3462170887 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f80480051c0 0x7f804800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:04.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.163+0000 7f806137f640 1 -- 172.21.15.83:0/3462170887 reap_dead start 2024-03-06T07:31:04.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.164+0000 7f805bfff640 1 --2- 172.21.15.83:0/3462170887 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8048004800 0x7f8048004c80 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f805c0cfc70 tx=0x7f8048005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fa3a77de4044687 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.164+0000 7f805b7fe640 1 -- 172.21.15.83:0/3462170887 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f804c00a480 con 0x7f8048004800 2024-03-06T07:31:04.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.164+0000 7f805b7fe640 1 -- 172.21.15.83:0/3462170887 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f804c034650 con 0x7f8048004800 2024-03-06T07:31:04.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.164+0000 7f805b7fe640 1 -- 172.21.15.83:0/3462170887 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f804c034a40 con 0x7f8048004800 2024-03-06T07:31:04.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.164+0000 7f8062de2640 1 -- 172.21.15.83:0/3462170887 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8048004800 msgr2=0x7f8048004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.164+0000 7f8062de2640 1 --2- 172.21.15.83:0/3462170887 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8048004800 0x7f8048004c80 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f805c0cfc70 tx=0x7f8048005830 comp rx=0 tx=0).stop 2024-03-06T07:31:04.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.164+0000 7f8062de2640 1 -- 172.21.15.83:0/3462170887 shutdown_connections 2024-03-06T07:31:04.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.164+0000 7f8062de2640 1 --2- 172.21.15.83:0/3462170887 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8048004800 0x7f8048004c80 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.164+0000 7f8062de2640 1 -- 172.21.15.83:0/3462170887 >> 172.21.15.83:0/3462170887 conn(0x7f805c0d3cd0 msgr2=0x7f805c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:04.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.164+0000 7f8062de2640 1 -- 172.21.15.83:0/3462170887 shutdown_connections 2024-03-06T07:31:04.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.164+0000 7f8062de2640 1 -- 172.21.15.83:0/3462170887 wait complete. 2024-03-06T07:31:04.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.166+0000 7f8062de2640 1 Processor -- start 2024-03-06T07:31:04.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.166+0000 7f8062de2640 1 -- start start 2024-03-06T07:31:04.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.168+0000 7f8062de2640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8048004800 0x7f805c0b7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.168+0000 7f8062de2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f805c0cbef0 0x7f805c0b9370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.168+0000 7f8062de2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f805c0cc370 0x7f805c14fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.168+0000 7f8062de2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f805c15e3c0 con 0x7f805c0cc370 2024-03-06T07:31:04.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.168+0000 7f8062de2640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f805c0b3d60 con 0x7f8048004800 2024-03-06T07:31:04.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.168+0000 7f8062de2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f805c15e540 con 0x7f805c0cbef0 2024-03-06T07:31:04.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.168+0000 7f806137f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f805c0cc370 0x7f805c14fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.168+0000 7f806137f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f805c0cc370 0x7f805c14fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52562/0 (socket says 172.21.15.83:52562) 2024-03-06T07:31:04.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.168+0000 7f806137f640 1 -- 172.21.15.83:0/594126522 learned_addr learned my addr 172.21.15.83:0/594126522 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:04.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.168+0000 7f805bfff640 1 --2- 172.21.15.83:0/594126522 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f805c0cbef0 0x7f805c0b9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.168+0000 7f8060b7e640 1 --2- 172.21.15.83:0/594126522 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8048004800 0x7f805c0b7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.168+0000 7f806137f640 1 -- 172.21.15.83:0/594126522 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f805c0cbef0 msgr2=0x7f805c0b9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.168+0000 7f806137f640 1 --2- 172.21.15.83:0/594126522 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f805c0cbef0 0x7f805c0b9370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.168+0000 7f806137f640 1 -- 172.21.15.83:0/594126522 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8048004800 msgr2=0x7f805c0b7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.168+0000 7f806137f640 1 --2- 172.21.15.83:0/594126522 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8048004800 0x7f805c0b7d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.168+0000 7f806137f640 1 -- 172.21.15.83:0/594126522 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f805c0b6cb0 con 0x7f805c0cc370 2024-03-06T07:31:04.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.168+0000 7f806137f640 1 --2- 172.21.15.83:0/594126522 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f805c0cc370 0x7f805c14fd20 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f805c0cff60 tx=0x7f805400e820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.168+0000 7f80597fa640 1 -- 172.21.15.83:0/594126522 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8054019070 con 0x7f805c0cc370 2024-03-06T07:31:04.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.168+0000 7f80597fa640 1 -- 172.21.15.83:0/594126522 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80540050d0 con 0x7f805c0cc370 2024-03-06T07:31:04.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.168+0000 7f80597fa640 1 -- 172.21.15.83:0/594126522 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80540035f0 con 0x7f805c0cc370 2024-03-06T07:31:04.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.168+0000 7f8062de2640 1 -- 172.21.15.83:0/594126522 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f805c0b6f90 con 0x7f805c0cc370 2024-03-06T07:31:04.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.169+0000 7f8062de2640 1 -- 172.21.15.83:0/594126522 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f805c0b7550 con 0x7f805c0cc370 2024-03-06T07:31:04.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.169+0000 7f8062de2640 1 -- 172.21.15.83:0/594126522 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f805c04f330 con 0x7f805c0cc370 2024-03-06T07:31:04.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.170+0000 7f80597fa640 1 -- 172.21.15.83:0/594126522 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f805401c070 con 0x7f805c0cc370 2024-03-06T07:31:04.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.170+0000 7f80597fa640 1 --2- 172.21.15.83:0/594126522 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f803c06b4d0 0x7f803c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.170+0000 7f80597fa640 1 -- 172.21.15.83:0/594126522 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8054089440 con 0x7f805c0cc370 2024-03-06T07:31:04.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.170+0000 7f8060b7e640 1 --2- 172.21.15.83:0/594126522 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f803c06b4d0 0x7f803c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.171+0000 7f8060b7e640 1 --2- 172.21.15.83:0/594126522 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f803c06b4d0 0x7f803c06d990 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f8044009b40 tx=0x7f8044008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.175+0000 7f80597fa640 1 -- 172.21.15.83:0/594126522 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8054057cc0 con 0x7f805c0cc370 2024-03-06T07:31:04.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.583+0000 7f8062de2640 1 -- 172.21.15.83:0/594126522 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f805c15d800 con 0x7f805c0cc370 2024-03-06T07:31:04.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.583+0000 7f80597fa640 1 -- 172.21.15.83:0/594126522 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f805405bd60 con 0x7f805c0cc370 2024-03-06T07:31:04.585 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:31:04.585 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:31:04.586 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:31:04.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.585+0000 7f8062de2640 1 -- 172.21.15.83:0/594126522 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f803c06b4d0 msgr2=0x7f803c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.585+0000 7f8062de2640 1 --2- 172.21.15.83:0/594126522 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f803c06b4d0 0x7f803c06d990 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f8044009b40 tx=0x7f8044008040 comp rx=0 tx=0).stop 2024-03-06T07:31:04.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.585+0000 7f8062de2640 1 -- 172.21.15.83:0/594126522 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f805c0cc370 msgr2=0x7f805c14fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.585+0000 7f8062de2640 1 --2- 172.21.15.83:0/594126522 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f805c0cc370 0x7f805c14fd20 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f805c0cff60 tx=0x7f805400e820 comp rx=0 tx=0).stop 2024-03-06T07:31:04.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.585+0000 7f8062de2640 1 -- 172.21.15.83:0/594126522 shutdown_connections 2024-03-06T07:31:04.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.585+0000 7f8062de2640 1 --2- 172.21.15.83:0/594126522 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f803c06b4d0 0x7f803c06d990 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.585+0000 7f8062de2640 1 --2- 172.21.15.83:0/594126522 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f805c0cc370 0x7f805c14fd20 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.585+0000 7f8062de2640 1 --2- 172.21.15.83:0/594126522 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f805c0cbef0 0x7f805c0b9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.585+0000 7f8062de2640 1 --2- 172.21.15.83:0/594126522 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8048004800 0x7f805c0b7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.585+0000 7f8062de2640 1 -- 172.21.15.83:0/594126522 >> 172.21.15.83:0/594126522 conn(0x7f805c0d3cd0 msgr2=0x7f805c06b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:04.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.585+0000 7f8062de2640 1 -- 172.21.15.83:0/594126522 shutdown_connections 2024-03-06T07:31:04.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:04.586+0000 7f8062de2640 1 -- 172.21.15.83:0/594126522 wait complete. 2024-03-06T07:31:04.799 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:31:04.799 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:31:05.066 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:05.801 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:05.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.899+0000 7f2b64239640 1 Processor -- start 2024-03-06T07:31:05.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.900+0000 7f2b64239640 1 -- start start 2024-03-06T07:31:05.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.900+0000 7f2b64239640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2b5c15dfe0 0x7f2b5c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.900+0000 7f2b64239640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2b5c0cf830 0x7f2b5c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.900+0000 7f2b64239640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b5c0b3430 con 0x7f2b5c0d27f0 2024-03-06T07:31:05.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.900+0000 7f2b64239640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b5c0d2c50 con 0x7f2b5c0cf830 2024-03-06T07:31:05.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.900+0000 7f2b64239640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5c0b31e0 con 0x7f2b5c15dfe0 2024-03-06T07:31:05.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.900+0000 7f2b617d4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2b5c0cf830 0x7f2b5c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.900+0000 7f2b617d4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2b5c0cf830 0x7f2b5c0d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52564/0 (socket says 172.21.15.83:52564) 2024-03-06T07:31:05.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.900+0000 7f2b617d4640 1 -- 172.21.15.83:0/3346917152 learned_addr learned my addr 172.21.15.83:0/3346917152 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:05.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.901+0000 7f2b617d4640 1 -- 172.21.15.83:0/3346917152 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2b5c15dfe0 msgr2=0x7f2b5c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:05.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.901+0000 7f2b617d4640 1 --2- 172.21.15.83:0/3346917152 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2b5c15dfe0 0x7f2b5c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.901+0000 7f2b617d4640 1 -- 172.21.15.83:0/3346917152 >> v1:172.21.15.183:6789/0 conn(0x7f2b5c0d27f0 legacy=0x7f2b5c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:05.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.901+0000 7f2b617d4640 1 -- 172.21.15.83:0/3346917152 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b5c0b3710 con 0x7f2b5c0cf830 2024-03-06T07:31:05.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.901+0000 7f2b617d4640 1 --2- 172.21.15.83:0/3346917152 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b5c0cf830 0x7f2b5c0d22b0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c009870 tx=0x7f2b4c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1463232e1fa90bdc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.902+0000 7f2b60fd3640 1 -- 172.21.15.83:0/3346917152 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b4c035020 con 0x7f2b5c0cf830 2024-03-06T07:31:05.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.902+0000 7f2b60fd3640 1 -- 172.21.15.83:0/3346917152 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b4c031640 con 0x7f2b5c0cf830 2024-03-06T07:31:05.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.902+0000 7f2b60fd3640 1 -- 172.21.15.83:0/3346917152 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b4c02d040 con 0x7f2b5c0cf830 2024-03-06T07:31:05.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.902+0000 7f2b64239640 1 -- 172.21.15.83:0/3346917152 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b5c0cf830 msgr2=0x7f2b5c0d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.902+0000 7f2b64239640 1 --2- 172.21.15.83:0/3346917152 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b5c0cf830 0x7f2b5c0d22b0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c009870 tx=0x7f2b4c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.902+0000 7f2b64239640 1 -- 172.21.15.83:0/3346917152 shutdown_connections 2024-03-06T07:31:05.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.902+0000 7f2b64239640 1 --2- 172.21.15.83:0/3346917152 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b5c0cf830 0x7f2b5c0d22b0 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.902+0000 7f2b64239640 1 --2- 172.21.15.83:0/3346917152 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2b5c15dfe0 0x7f2b5c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.902+0000 7f2b64239640 1 -- 172.21.15.83:0/3346917152 >> 172.21.15.83:0/3346917152 conn(0x7f2b5c0d3cd0 msgr2=0x7f2b5c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:05.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.902+0000 7f2b64239640 1 -- 172.21.15.83:0/3346917152 shutdown_connections 2024-03-06T07:31:05.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.902+0000 7f2b64239640 1 -- 172.21.15.83:0/3346917152 wait complete. 2024-03-06T07:31:05.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.903+0000 7f2b64239640 1 Processor -- start 2024-03-06T07:31:05.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.903+0000 7f2b64239640 1 -- start start 2024-03-06T07:31:05.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.904+0000 7f2b64239640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b5c0cf830 0x7f2b5c0cafb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.904+0000 7f2b64239640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b5c0d27f0 0x7f2b5c159d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.904+0000 7f2b64239640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2b5c15dfe0 0x7f2b5c0b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.904+0000 7f2b64239640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5c0d1c70 con 0x7f2b5c0cf830 2024-03-06T07:31:05.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.904+0000 7f2b64239640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5c0b3d60 con 0x7f2b5c0d27f0 2024-03-06T07:31:05.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.904+0000 7f2b64239640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5c0d1df0 con 0x7f2b5c15dfe0 2024-03-06T07:31:05.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.904+0000 7f2b61fd5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b5c0cf830 0x7f2b5c0cafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.904+0000 7f2b61fd5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b5c0cf830 0x7f2b5c0cafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52566/0 (socket says 172.21.15.83:52566) 2024-03-06T07:31:05.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.904+0000 7f2b61fd5640 1 -- 172.21.15.83:0/3667484919 learned_addr learned my addr 172.21.15.83:0/3667484919 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:05.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.904+0000 7f2b627d6640 1 --2- 172.21.15.83:0/3667484919 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2b5c15dfe0 0x7f2b5c0b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.904+0000 7f2b61fd5640 1 -- 172.21.15.83:0/3667484919 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2b5c15dfe0 msgr2=0x7f2b5c0b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.904+0000 7f2b617d4640 1 --2- 172.21.15.83:0/3667484919 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b5c0d27f0 0x7f2b5c159d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.904+0000 7f2b61fd5640 1 --2- 172.21.15.83:0/3667484919 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2b5c15dfe0 0x7f2b5c0b68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.905+0000 7f2b61fd5640 1 -- 172.21.15.83:0/3667484919 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b5c0d27f0 msgr2=0x7f2b5c159d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.905+0000 7f2b61fd5640 1 --2- 172.21.15.83:0/3667484919 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b5c0d27f0 0x7f2b5c159d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.905+0000 7f2b61fd5640 1 -- 172.21.15.83:0/3667484919 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b5c0cc3f0 con 0x7f2b5c0cf830 2024-03-06T07:31:05.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.905+0000 7f2b627d6640 1 --2- 172.21.15.83:0/3667484919 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2b5c15dfe0 0x7f2b5c0b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:05.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.905+0000 7f2b617d4640 1 --2- 172.21.15.83:0/3667484919 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b5c0d27f0 0x7f2b5c159d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:05.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.905+0000 7f2b61fd5640 1 --2- 172.21.15.83:0/3667484919 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b5c0cf830 0x7f2b5c0cafb0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f2b5800b440 tx=0x7f2b5800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.905+0000 7f2b527fc640 1 -- 172.21.15.83:0/3667484919 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b58015020 con 0x7f2b5c0cf830 2024-03-06T07:31:05.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.905+0000 7f2b527fc640 1 -- 172.21.15.83:0/3667484919 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b58011890 con 0x7f2b5c0cf830 2024-03-06T07:31:05.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.905+0000 7f2b527fc640 1 -- 172.21.15.83:0/3667484919 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b58011c80 con 0x7f2b5c0cf830 2024-03-06T07:31:05.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.905+0000 7f2b64239640 1 -- 172.21.15.83:0/3667484919 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b5c0cc6d0 con 0x7f2b5c0cf830 2024-03-06T07:31:05.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.905+0000 7f2b64239640 1 -- 172.21.15.83:0/3667484919 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b5c15a2a0 con 0x7f2b5c0cf830 2024-03-06T07:31:05.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.906+0000 7f2b64239640 1 -- 172.21.15.83:0/3667484919 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b5c04f330 con 0x7f2b5c0cf830 2024-03-06T07:31:05.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.906+0000 7f2b527fc640 1 -- 172.21.15.83:0/3667484919 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2b58018020 con 0x7f2b5c0cf830 2024-03-06T07:31:05.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.907+0000 7f2b527fc640 1 --2- 172.21.15.83:0/3667484919 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2b4006b4d0 0x7f2b4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.907+0000 7f2b527fc640 1 -- 172.21.15.83:0/3667484919 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2b5808a300 con 0x7f2b5c0cf830 2024-03-06T07:31:05.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.907+0000 7f2b617d4640 1 --2- 172.21.15.83:0/3667484919 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2b4006b4d0 0x7f2b4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.907+0000 7f2b617d4640 1 --2- 172.21.15.83:0/3667484919 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2b4006b4d0 0x7f2b4006d990 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c036000 tx=0x7f2b4c03c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:05.911+0000 7f2b527fc640 1 -- 172.21.15.83:0/3667484919 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2b5801f020 con 0x7f2b5c0cf830 2024-03-06T07:31:06.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:06.161+0000 7f2b64239640 1 -- 172.21.15.83:0/3667484919 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b5c0cea30 con 0x7f2b5c0cf830 2024-03-06T07:31:06.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:06.161+0000 7f2b527fc640 1 -- 172.21.15.83:0/3667484919 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f2b5800d040 con 0x7f2b5c0cf830 2024-03-06T07:31:06.164 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:31:06.164 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:31:06.165 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:31:06.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:06.163+0000 7f2b64239640 1 -- 172.21.15.83:0/3667484919 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2b4006b4d0 msgr2=0x7f2b4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:06.163+0000 7f2b64239640 1 --2- 172.21.15.83:0/3667484919 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2b4006b4d0 0x7f2b4006d990 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c036000 tx=0x7f2b4c03c040 comp rx=0 tx=0).stop 2024-03-06T07:31:06.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:06.163+0000 7f2b64239640 1 -- 172.21.15.83:0/3667484919 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b5c0cf830 msgr2=0x7f2b5c0cafb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:06.163+0000 7f2b64239640 1 --2- 172.21.15.83:0/3667484919 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b5c0cf830 0x7f2b5c0cafb0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f2b5800b440 tx=0x7f2b5800b900 comp rx=0 tx=0).stop 2024-03-06T07:31:06.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:06.163+0000 7f2b64239640 1 -- 172.21.15.83:0/3667484919 shutdown_connections 2024-03-06T07:31:06.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:06.163+0000 7f2b64239640 1 --2- 172.21.15.83:0/3667484919 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2b4006b4d0 0x7f2b4006d990 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:06.163+0000 7f2b64239640 1 --2- 172.21.15.83:0/3667484919 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2b5c15dfe0 0x7f2b5c0b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:06.163+0000 7f2b64239640 1 --2- 172.21.15.83:0/3667484919 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b5c0d27f0 0x7f2b5c159d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:06.163+0000 7f2b64239640 1 --2- 172.21.15.83:0/3667484919 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2b5c0cf830 0x7f2b5c0cafb0 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:06.163+0000 7f2b64239640 1 -- 172.21.15.83:0/3667484919 >> 172.21.15.83:0/3667484919 conn(0x7f2b5c0d3cd0 msgr2=0x7f2b5c06cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:06.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:06.163+0000 7f2b64239640 1 -- 172.21.15.83:0/3667484919 shutdown_connections 2024-03-06T07:31:06.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:06.163+0000 7f2b64239640 1 -- 172.21.15.83:0/3667484919 wait complete. 2024-03-06T07:31:06.378 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:31:06.378 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:31:07.068 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:07.379 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:07.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.478+0000 7fe773888640 1 Processor -- start 2024-03-06T07:31:07.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.478+0000 7fe773888640 1 -- start start 2024-03-06T07:31:07.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.480+0000 7fe773888640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fe76c0d2710 0x7fe76c0b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.480+0000 7fe773888640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe76c0b3430 con 0x7fe76c15dfe0 2024-03-06T07:31:07.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.480+0000 7fe773888640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe76c0b3710 con 0x7fe76c0d22b0 2024-03-06T07:31:07.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.480+0000 7fe773888640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe76c0b31e0 con 0x7fe76c0d2710 2024-03-06T07:31:07.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.480+0000 7fe771624640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fe76c15dfe0 0x7fe76c0cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:54036/0 (socket says 172.21.15.83:54036) 2024-03-06T07:31:07.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.480+0000 7fe771624640 1 -- 172.21.15.83:0/3365632429 learned_addr learned my addr 172.21.15.83:0/3365632429 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:31:07.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.480+0000 7fe771e25640 1 --2- 172.21.15.83:0/3365632429 >> v2:172.21.15.183:3300/0 conn(0x7fe76c0d2710 0x7fe76c0b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.480+0000 7fe763fff640 1 -- 172.21.15.83:0/3365632429 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3998869522 0 0) 0x7fe76c0b3430 con 0x7fe76c15dfe0 2024-03-06T07:31:07.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.480+0000 7fe763fff640 1 -- 172.21.15.83:0/3365632429 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe758003520 con 0x7fe76c15dfe0 2024-03-06T07:31:07.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.480+0000 7fe763fff640 1 -- 172.21.15.83:0/3365632429 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2473897018 0 0) 0x7fe758003520 con 0x7fe76c15dfe0 2024-03-06T07:31:07.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.480+0000 7fe763fff640 1 -- 172.21.15.83:0/3365632429 >> v2:172.21.15.183:3300/0 conn(0x7fe76c0d2710 msgr2=0x7fe76c0b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:07.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.480+0000 7fe763fff640 1 --2- 172.21.15.83:0/3365632429 >> v2:172.21.15.183:3300/0 conn(0x7fe76c0d2710 0x7fe76c0b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.480+0000 7fe763fff640 1 -- 172.21.15.83:0/3365632429 >> v1:172.21.15.183:6789/0 conn(0x7fe76c0d22b0 legacy=0x7fe76c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.480+0000 7fe763fff640 1 -- 172.21.15.83:0/3365632429 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe76c0b39f0 con 0x7fe76c15dfe0 2024-03-06T07:31:07.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.480+0000 7fe763fff640 1 -- 172.21.15.83:0/3365632429 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7fe768001670 con 0x7fe76c15dfe0 2024-03-06T07:31:07.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.480+0000 7fe763fff640 1 -- 172.21.15.83:0/3365632429 >> v1:172.21.15.83:6789/0 conn(0x7fe76c15dfe0 legacy=0x7fe76c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.480+0000 7fe763fff640 1 --2- 172.21.15.83:0/3365632429 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7580039e0 0x7fe758003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.480+0000 7fe763fff640 1 --2- 172.21.15.83:0/3365632429 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe758004800 0x7fe758004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.480+0000 7fe763fff640 1 --2- 172.21.15.83:0/3365632429 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe7580051c0 0x7fe75800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.480+0000 7fe763fff640 1 -- 172.21.15.83:0/3365632429 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c0b31e0 con 0x7fe7580039e0 2024-03-06T07:31:07.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.480+0000 7fe763fff640 1 -- 172.21.15.83:0/3365632429 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75800c010 con 0x7fe758004800 2024-03-06T07:31:07.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.480+0000 7fe763fff640 1 -- 172.21.15.83:0/3365632429 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe75800c210 con 0x7fe7580051c0 2024-03-06T07:31:07.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.480+0000 7fe771624640 1 --2- 172.21.15.83:0/3365632429 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7580039e0 0x7fe758003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.480+0000 7fe770e23640 1 --2- 172.21.15.83:0/3365632429 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe758004800 0x7fe758004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.480+0000 7fe771e25640 1 --2- 172.21.15.83:0/3365632429 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe7580051c0 0x7fe75800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.480+0000 7fe771624640 1 -- 172.21.15.83:0/3365632429 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe7580051c0 msgr2=0x7fe75800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:07.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.480+0000 7fe771624640 1 --2- 172.21.15.83:0/3365632429 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe7580051c0 0x7fe75800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.480+0000 7fe771624640 1 -- 172.21.15.83:0/3365632429 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe758004800 msgr2=0x7fe758004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:07.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.480+0000 7fe771624640 1 --2- 172.21.15.83:0/3365632429 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe758004800 0x7fe758004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.480+0000 7fe771624640 1 -- 172.21.15.83:0/3365632429 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe758003520 con 0x7fe7580039e0 2024-03-06T07:31:07.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.480+0000 7fe771e25640 1 -- 172.21.15.83:0/3365632429 reap_dead start 2024-03-06T07:31:07.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.481+0000 7fe771624640 1 --2- 172.21.15.83:0/3365632429 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7580039e0 0x7fe758003dc0 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7fe76800c240 tx=0x7fe758005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee6314b51c4061fb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:07.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.481+0000 7fe763fff640 1 -- 172.21.15.83:0/3365632429 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe768007d30 con 0x7fe7580039e0 2024-03-06T07:31:07.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.481+0000 7fe763fff640 1 -- 172.21.15.83:0/3365632429 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe768030070 con 0x7fe7580039e0 2024-03-06T07:31:07.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.481+0000 7fe763fff640 1 -- 172.21.15.83:0/3365632429 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe768005d30 con 0x7fe7580039e0 2024-03-06T07:31:07.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.481+0000 7fe773888640 1 -- 172.21.15.83:0/3365632429 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7580039e0 msgr2=0x7fe758003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.481+0000 7fe773888640 1 --2- 172.21.15.83:0/3365632429 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7580039e0 0x7fe758003dc0 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7fe76800c240 tx=0x7fe758005830 comp rx=0 tx=0).stop 2024-03-06T07:31:07.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.482+0000 7fe773888640 1 -- 172.21.15.83:0/3365632429 shutdown_connections 2024-03-06T07:31:07.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.482+0000 7fe773888640 1 --2- 172.21.15.83:0/3365632429 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7580039e0 0x7fe758003dc0 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.482+0000 7fe773888640 1 -- 172.21.15.83:0/3365632429 >> 172.21.15.83:0/3365632429 conn(0x7fe76c0d3cd0 msgr2=0x7fe76c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:07.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.482+0000 7fe773888640 1 -- 172.21.15.83:0/3365632429 shutdown_connections 2024-03-06T07:31:07.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.482+0000 7fe773888640 1 -- 172.21.15.83:0/3365632429 wait complete. 2024-03-06T07:31:07.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.484+0000 7fe773888640 1 Processor -- start 2024-03-06T07:31:07.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.484+0000 7fe773888640 1 -- start start 2024-03-06T07:31:07.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.484+0000 7fe773888640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe7580039e0 0x7fe76c0cb410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.484+0000 7fe773888640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe76c0cca90 0x7fe76c09ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.484+0000 7fe773888640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe76c1f4960 0x7fe76c1f4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.484+0000 7fe773888640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c0d1c70 con 0x7fe76c1f4960 2024-03-06T07:31:07.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.484+0000 7fe773888640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c0b3d60 con 0x7fe7580039e0 2024-03-06T07:31:07.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.484+0000 7fe773888640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c0d1df0 con 0x7fe76c0cca90 2024-03-06T07:31:07.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.484+0000 7fe771e25640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe76c1f4960 0x7fe76c1f4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.484+0000 7fe771e25640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe76c1f4960 0x7fe76c1f4de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45922/0 (socket says 172.21.15.83:45922) 2024-03-06T07:31:07.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.484+0000 7fe771e25640 1 -- 172.21.15.83:0/3372660577 learned_addr learned my addr 172.21.15.83:0/3372660577 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:07.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.484+0000 7fe770e23640 1 --2- 172.21.15.83:0/3372660577 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe76c0cca90 0x7fe76c09ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.484+0000 7fe771624640 1 --2- 172.21.15.83:0/3372660577 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe7580039e0 0x7fe76c0cb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.484+0000 7fe771e25640 1 -- 172.21.15.83:0/3372660577 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe76c0cca90 msgr2=0x7fe76c09ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.484+0000 7fe771e25640 1 --2- 172.21.15.83:0/3372660577 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe76c0cca90 0x7fe76c09ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.484+0000 7fe771e25640 1 -- 172.21.15.83:0/3372660577 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe7580039e0 msgr2=0x7fe76c0cb410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.484+0000 7fe771e25640 1 --2- 172.21.15.83:0/3372660577 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe7580039e0 0x7fe76c0cb410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.484+0000 7fe771e25640 1 -- 172.21.15.83:0/3372660577 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe76c15b810 con 0x7fe76c1f4960 2024-03-06T07:31:07.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.484+0000 7fe771624640 1 --2- 172.21.15.83:0/3372660577 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe7580039e0 0x7fe76c0cb410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:07.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.484+0000 7fe771e25640 1 --2- 172.21.15.83:0/3372660577 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe76c1f4960 0x7fe76c1f4de0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7fe76c161bf0 tx=0x7fe7640115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:07.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.484+0000 7fe761ffb640 1 -- 172.21.15.83:0/3372660577 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe764011e40 con 0x7fe76c1f4960 2024-03-06T07:31:07.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.484+0000 7fe761ffb640 1 -- 172.21.15.83:0/3372660577 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe76401b070 con 0x7fe76c1f4960 2024-03-06T07:31:07.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.484+0000 7fe773888640 1 -- 172.21.15.83:0/3372660577 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe76c0cb9f0 con 0x7fe76c1f4960 2024-03-06T07:31:07.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.484+0000 7fe761ffb640 1 -- 172.21.15.83:0/3372660577 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe764003990 con 0x7fe76c1f4960 2024-03-06T07:31:07.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.484+0000 7fe773888640 1 -- 172.21.15.83:0/3372660577 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe76c0cbfb0 con 0x7fe76c1f4960 2024-03-06T07:31:07.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.486+0000 7fe773888640 1 -- 172.21.15.83:0/3372660577 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe76c04f330 con 0x7fe76c1f4960 2024-03-06T07:31:07.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.486+0000 7fe761ffb640 1 -- 172.21.15.83:0/3372660577 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe764005d10 con 0x7fe76c1f4960 2024-03-06T07:31:07.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.486+0000 7fe761ffb640 1 --2- 172.21.15.83:0/3372660577 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe75006b500 0x7fe75006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.486+0000 7fe771624640 1 --2- 172.21.15.83:0/3372660577 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe75006b500 0x7fe75006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.486+0000 7fe761ffb640 1 -- 172.21.15.83:0/3372660577 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe76408a750 con 0x7fe76c1f4960 2024-03-06T07:31:07.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.488+0000 7fe771624640 1 --2- 172.21.15.83:0/3372660577 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe75006b500 0x7fe75006d9c0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fe768034000 tx=0x7fe768039040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:07.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.490+0000 7fe761ffb640 1 -- 172.21.15.83:0/3372660577 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe764025020 con 0x7fe76c1f4960 2024-03-06T07:31:07.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.706+0000 7fe773888640 1 -- 172.21.15.83:0/3372660577 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe76c06cc80 con 0x7fe76c1f4960 2024-03-06T07:31:07.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.707+0000 7fe761ffb640 1 -- 172.21.15.83:0/3372660577 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7fe764014350 con 0x7fe76c1f4960 2024-03-06T07:31:07.709 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:31:07.709 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:31:07.709 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:31:07.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.709+0000 7fe773888640 1 -- 172.21.15.83:0/3372660577 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe75006b500 msgr2=0x7fe75006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.709+0000 7fe773888640 1 --2- 172.21.15.83:0/3372660577 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe75006b500 0x7fe75006d9c0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fe768034000 tx=0x7fe768039040 comp rx=0 tx=0).stop 2024-03-06T07:31:07.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.709+0000 7fe773888640 1 -- 172.21.15.83:0/3372660577 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe76c1f4960 msgr2=0x7fe76c1f4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.709+0000 7fe773888640 1 --2- 172.21.15.83:0/3372660577 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe76c1f4960 0x7fe76c1f4de0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7fe76c161bf0 tx=0x7fe7640115a0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.709+0000 7fe773888640 1 -- 172.21.15.83:0/3372660577 shutdown_connections 2024-03-06T07:31:07.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.709+0000 7fe773888640 1 --2- 172.21.15.83:0/3372660577 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe75006b500 0x7fe75006d9c0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.709+0000 7fe773888640 1 --2- 172.21.15.83:0/3372660577 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe76c1f4960 0x7fe76c1f4de0 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.709+0000 7fe773888640 1 --2- 172.21.15.83:0/3372660577 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe76c0cca90 0x7fe76c09ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.709+0000 7fe773888640 1 --2- 172.21.15.83:0/3372660577 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe7580039e0 0x7fe76c0cb410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.709+0000 7fe773888640 1 -- 172.21.15.83:0/3372660577 >> 172.21.15.83:0/3372660577 conn(0x7fe76c0d3cd0 msgr2=0x7fe76c15d180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:07.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.709+0000 7fe773888640 1 -- 172.21.15.83:0/3372660577 shutdown_connections 2024-03-06T07:31:07.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:07.709+0000 7fe773888640 1 -- 172.21.15.83:0/3372660577 wait complete. 2024-03-06T07:31:07.923 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:31:07.924 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:31:08.924 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:09.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.027+0000 7fd48b081640 1 Processor -- start 2024-03-06T07:31:09.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.027+0000 7fd48b081640 1 -- start start 2024-03-06T07:31:09.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.028+0000 7fd48b081640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fd48415dfe0 0x7fd4840cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.028+0000 7fd48b081640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd4840d29c0 0x7fd4840b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.028+0000 7fd48b081640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4840b3430 con 0x7fd4840cf830 2024-03-06T07:31:09.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.028+0000 7fd48b081640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4840b31e0 con 0x7fd4840d29c0 2024-03-06T07:31:09.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.028+0000 7fd48b081640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4840b3d60 con 0x7fd48415dfe0 2024-03-06T07:31:09.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.028+0000 7fd48961e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd4840d29c0 0x7fd4840b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.028+0000 7fd48961e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd4840d29c0 0x7fd4840b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45928/0 (socket says 172.21.15.83:45928) 2024-03-06T07:31:09.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.028+0000 7fd48961e640 1 -- 172.21.15.83:0/3635130749 learned_addr learned my addr 172.21.15.83:0/3635130749 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:09.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.028+0000 7fd483fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fd4840cf830 0x7fd4840d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:54038/0 (socket says 172.21.15.83:54038) 2024-03-06T07:31:09.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.029+0000 7fd48961e640 1 -- 172.21.15.83:0/3635130749 >> v2:172.21.15.183:3300/0 conn(0x7fd48415dfe0 msgr2=0x7fd4840cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:09.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.029+0000 7fd48961e640 1 --2- 172.21.15.83:0/3635130749 >> v2:172.21.15.183:3300/0 conn(0x7fd48415dfe0 0x7fd4840cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.029+0000 7fd48961e640 1 -- 172.21.15.83:0/3635130749 >> v1:172.21.15.83:6789/0 conn(0x7fd4840cf830 legacy=0x7fd4840d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.029+0000 7fd48961e640 1 -- 172.21.15.83:0/3635130749 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4840b3710 con 0x7fd4840d29c0 2024-03-06T07:31:09.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.029+0000 7fd4837fe640 1 -- 172.21.15.83:0/3635130749 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2918206896 0 0) 0x7fd4840b3430 con 0x7fd4840cf830 2024-03-06T07:31:09.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.029+0000 7fd48961e640 1 --2- 172.21.15.83:0/3635130749 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd4840d29c0 0x7fd4840b41f0 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7fd484067990 tx=0x7fd47c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32ba2523ead269b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:09.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.029+0000 7fd4837fe640 1 -- 172.21.15.83:0/3635130749 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd47c02cb20 con 0x7fd4840d29c0 2024-03-06T07:31:09.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.029+0000 7fd4837fe640 1 -- 172.21.15.83:0/3635130749 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd47c02ccc0 con 0x7fd4840d29c0 2024-03-06T07:31:09.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.029+0000 7fd4837fe640 1 -- 172.21.15.83:0/3635130749 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd47c034610 con 0x7fd4840d29c0 2024-03-06T07:31:09.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.030+0000 7fd48b081640 1 -- 172.21.15.83:0/3635130749 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd4840d29c0 msgr2=0x7fd4840b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.030+0000 7fd48b081640 1 --2- 172.21.15.83:0/3635130749 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd4840d29c0 0x7fd4840b41f0 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7fd484067990 tx=0x7fd47c02bf80 comp rx=0 tx=0).stop 2024-03-06T07:31:09.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.030+0000 7fd48b081640 1 -- 172.21.15.83:0/3635130749 shutdown_connections 2024-03-06T07:31:09.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.030+0000 7fd48b081640 1 --2- 172.21.15.83:0/3635130749 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd4840d29c0 0x7fd4840b41f0 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.030+0000 7fd48b081640 1 --2- 172.21.15.83:0/3635130749 >> v2:172.21.15.183:3300/0 conn(0x7fd48415dfe0 0x7fd4840cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.030+0000 7fd48b081640 1 -- 172.21.15.83:0/3635130749 >> 172.21.15.83:0/3635130749 conn(0x7fd4840d3cd0 msgr2=0x7fd4840d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:09.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.030+0000 7fd48b081640 1 -- 172.21.15.83:0/3635130749 shutdown_connections 2024-03-06T07:31:09.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.030+0000 7fd48b081640 1 -- 172.21.15.83:0/3635130749 wait complete. 2024-03-06T07:31:09.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.031+0000 7fd48b081640 1 Processor -- start 2024-03-06T07:31:09.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.031+0000 7fd48b081640 1 -- start start 2024-03-06T07:31:09.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.032+0000 7fd48b081640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd4840cf830 0x7fd4841f31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.032+0000 7fd48b081640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd4840d29c0 0x7fd4841f6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.032+0000 7fd48b081640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd48415dfe0 0x7fd4841f7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.032+0000 7fd48b081640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4840d1df0 con 0x7fd4840cf830 2024-03-06T07:31:09.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.032+0000 7fd48b081640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4840d1c70 con 0x7fd48415dfe0 2024-03-06T07:31:09.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.032+0000 7fd48b081640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd48415e3c0 con 0x7fd4840d29c0 2024-03-06T07:31:09.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.032+0000 7fd488e1d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd4840cf830 0x7fd4841f31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.032+0000 7fd488e1d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd4840cf830 0x7fd4841f31b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45936/0 (socket says 172.21.15.83:45936) 2024-03-06T07:31:09.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.032+0000 7fd488e1d640 1 -- 172.21.15.83:0/1386609945 learned_addr learned my addr 172.21.15.83:0/1386609945 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:09.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.032+0000 7fd483fff640 1 --2- 172.21.15.83:0/1386609945 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd4840d29c0 0x7fd4841f6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.032+0000 7fd48961e640 1 --2- 172.21.15.83:0/1386609945 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd48415dfe0 0x7fd4841f7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.032+0000 7fd488e1d640 1 -- 172.21.15.83:0/1386609945 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd4840d29c0 msgr2=0x7fd4841f6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.032+0000 7fd488e1d640 1 --2- 172.21.15.83:0/1386609945 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd4840d29c0 0x7fd4841f6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.032+0000 7fd488e1d640 1 -- 172.21.15.83:0/1386609945 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd48415dfe0 msgr2=0x7fd4841f7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.032+0000 7fd488e1d640 1 --2- 172.21.15.83:0/1386609945 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd48415dfe0 0x7fd4841f7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.032+0000 7fd488e1d640 1 -- 172.21.15.83:0/1386609945 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4841f36f0 con 0x7fd4840cf830 2024-03-06T07:31:09.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.033+0000 7fd488e1d640 1 --2- 172.21.15.83:0/1386609945 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd4840cf830 0x7fd4841f31b0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7fd47800b440 tx=0x7fd47800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:09.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.033+0000 7fd4817fa640 1 -- 172.21.15.83:0/1386609945 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd478015020 con 0x7fd4840cf830 2024-03-06T07:31:09.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.033+0000 7fd4817fa640 1 -- 172.21.15.83:0/1386609945 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd478011890 con 0x7fd4840cf830 2024-03-06T07:31:09.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.033+0000 7fd48b081640 1 -- 172.21.15.83:0/1386609945 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd484161220 con 0x7fd4840cf830 2024-03-06T07:31:09.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.033+0000 7fd4817fa640 1 -- 172.21.15.83:0/1386609945 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd478011c80 con 0x7fd4840cf830 2024-03-06T07:31:09.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.033+0000 7fd48b081640 1 -- 172.21.15.83:0/1386609945 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4840b8d00 con 0x7fd4840cf830 2024-03-06T07:31:09.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.034+0000 7fd48b081640 1 -- 172.21.15.83:0/1386609945 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd48404f330 con 0x7fd4840cf830 2024-03-06T07:31:09.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.034+0000 7fd4817fa640 1 -- 172.21.15.83:0/1386609945 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd478018020 con 0x7fd4840cf830 2024-03-06T07:31:09.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.035+0000 7fd4817fa640 1 --2- 172.21.15.83:0/1386609945 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd46806b4d0 0x7fd46806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.035+0000 7fd483fff640 1 --2- 172.21.15.83:0/1386609945 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd46806b4d0 0x7fd46806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.035+0000 7fd4817fa640 1 -- 172.21.15.83:0/1386609945 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd47808a2a0 con 0x7fd4840cf830 2024-03-06T07:31:09.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.035+0000 7fd483fff640 1 --2- 172.21.15.83:0/1386609945 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd46806b4d0 0x7fd46806d990 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fd4841f6d50 tx=0x7fd470007b50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:09.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.039+0000 7fd4817fa640 1 -- 172.21.15.83:0/1386609945 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd47801f020 con 0x7fd4840cf830 2024-03-06T07:31:09.070 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:09.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.257+0000 7fd48b081640 1 -- 172.21.15.83:0/1386609945 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4840747a0 con 0x7fd4840cf830 2024-03-06T07:31:09.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.257+0000 7fd4817fa640 1 -- 172.21.15.83:0/1386609945 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7fd4841f36f0 con 0x7fd4840cf830 2024-03-06T07:31:09.259 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:31:09.259 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:31:09.260 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:31:09.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.259+0000 7fd48b081640 1 -- 172.21.15.83:0/1386609945 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd46806b4d0 msgr2=0x7fd46806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.259+0000 7fd48b081640 1 --2- 172.21.15.83:0/1386609945 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd46806b4d0 0x7fd46806d990 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fd4841f6d50 tx=0x7fd470007b50 comp rx=0 tx=0).stop 2024-03-06T07:31:09.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.259+0000 7fd48b081640 1 -- 172.21.15.83:0/1386609945 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd4840cf830 msgr2=0x7fd4841f31b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.259+0000 7fd48b081640 1 --2- 172.21.15.83:0/1386609945 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd4840cf830 0x7fd4841f31b0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7fd47800b440 tx=0x7fd47800b900 comp rx=0 tx=0).stop 2024-03-06T07:31:09.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.259+0000 7fd48b081640 1 -- 172.21.15.83:0/1386609945 shutdown_connections 2024-03-06T07:31:09.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.259+0000 7fd48b081640 1 --2- 172.21.15.83:0/1386609945 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd46806b4d0 0x7fd46806d990 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.259+0000 7fd48b081640 1 --2- 172.21.15.83:0/1386609945 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd48415dfe0 0x7fd4841f7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.259+0000 7fd48b081640 1 --2- 172.21.15.83:0/1386609945 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd4840d29c0 0x7fd4841f6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.259+0000 7fd48b081640 1 --2- 172.21.15.83:0/1386609945 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd4840cf830 0x7fd4841f31b0 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.259+0000 7fd48b081640 1 -- 172.21.15.83:0/1386609945 >> 172.21.15.83:0/1386609945 conn(0x7fd4840d3cd0 msgr2=0x7fd48406ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:09.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.259+0000 7fd48b081640 1 -- 172.21.15.83:0/1386609945 shutdown_connections 2024-03-06T07:31:09.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:09.259+0000 7fd48b081640 1 -- 172.21.15.83:0/1386609945 wait complete. 2024-03-06T07:31:09.473 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:31:09.473 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:31:10.474 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:10.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.572+0000 7f760e26f640 1 Processor -- start 2024-03-06T07:31:10.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.572+0000 7f760e26f640 1 -- start start 2024-03-06T07:31:10.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.573+0000 7f760e26f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f76080d22b0 0x7f76080d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.573+0000 7f760e26f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76080b3430 con 0x7f760815dfe0 2024-03-06T07:31:10.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.573+0000 7f760e26f640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76080b3710 con 0x7f76080b41f0 2024-03-06T07:31:10.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.573+0000 7f760e26f640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76080d2c50 con 0x7f76080d22b0 2024-03-06T07:31:10.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.573+0000 7f7607fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f760815dfe0 0x7f76080cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:54040/0 (socket says 172.21.15.83:54040) 2024-03-06T07:31:10.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.573+0000 7f76077fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f76080d22b0 0x7f76080d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.573+0000 7f7607fff640 1 -- 172.21.15.83:0/703930625 learned_addr learned my addr 172.21.15.83:0/703930625 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:31:10.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.573+0000 7f7606ffd640 1 -- 172.21.15.83:0/703930625 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3491395114 0 0) 0x7f76080b3430 con 0x7f760815dfe0 2024-03-06T07:31:10.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.573+0000 7f7606ffd640 1 -- 172.21.15.83:0/703930625 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75f0003520 con 0x7f760815dfe0 2024-03-06T07:31:10.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.574+0000 7f76077fe640 1 -- 172.21.15.83:0/703930625 >> v1:172.21.15.183:6789/0 conn(0x7f76080b41f0 legacy=0x7f76080b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:10.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.574+0000 7f76077fe640 1 -- 172.21.15.83:0/703930625 >> v1:172.21.15.83:6789/0 conn(0x7f760815dfe0 legacy=0x7f76080cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.574+0000 7f76077fe640 1 -- 172.21.15.83:0/703930625 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76080b39f0 con 0x7f76080d22b0 2024-03-06T07:31:10.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.574+0000 7f76077fe640 1 --2- 172.21.15.83:0/703930625 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f76080d22b0 0x7f76080d2710 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f75fc0097d0 tx=0x7f75fc02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=472e72a7d81a2a87 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:10.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.574+0000 7f7606ffd640 1 -- 172.21.15.83:0/703930625 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75fc034070 con 0x7f76080d22b0 2024-03-06T07:31:10.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.575+0000 7f7606ffd640 1 -- 172.21.15.83:0/703930625 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75fc030610 con 0x7f76080d22b0 2024-03-06T07:31:10.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.575+0000 7f7606ffd640 1 -- 172.21.15.83:0/703930625 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75fc0309e0 con 0x7f76080d22b0 2024-03-06T07:31:10.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.575+0000 7f760e26f640 1 -- 172.21.15.83:0/703930625 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f76080d22b0 msgr2=0x7f76080d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.575+0000 7f760e26f640 1 --2- 172.21.15.83:0/703930625 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f76080d22b0 0x7f76080d2710 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f75fc0097d0 tx=0x7f75fc02b960 comp rx=0 tx=0).stop 2024-03-06T07:31:10.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.575+0000 7f760e26f640 1 -- 172.21.15.83:0/703930625 shutdown_connections 2024-03-06T07:31:10.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.575+0000 7f760e26f640 1 --2- 172.21.15.83:0/703930625 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f76080d22b0 0x7f76080d2710 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.575+0000 7f760e26f640 1 -- 172.21.15.83:0/703930625 >> 172.21.15.83:0/703930625 conn(0x7f76080d3cd0 msgr2=0x7f76080d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:10.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.575+0000 7f760e26f640 1 -- 172.21.15.83:0/703930625 shutdown_connections 2024-03-06T07:31:10.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.575+0000 7f760e26f640 1 -- 172.21.15.83:0/703930625 wait complete. 2024-03-06T07:31:10.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.576+0000 7f760e26f640 1 Processor -- start 2024-03-06T07:31:10.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.576+0000 7f760e26f640 1 -- start start 2024-03-06T07:31:10.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.577+0000 7f760e26f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f76080b41f0 0x7f76081f4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.577+0000 7f760e26f640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f76080d22b0 0x7f7608160640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.577+0000 7f760e26f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f760815dfe0 0x7f7608160b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.577+0000 7f760e26f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76080b3d60 con 0x7f760815dfe0 2024-03-06T07:31:10.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.577+0000 7f760e26f640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76080b31e0 con 0x7f76080d22b0 2024-03-06T07:31:10.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.577+0000 7f760e26f640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76080d1c70 con 0x7f76080b41f0 2024-03-06T07:31:10.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.577+0000 7f760c80c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f760815dfe0 0x7f7608160b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.577+0000 7f760c80c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f760815dfe0 0x7f7608160b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45946/0 (socket says 172.21.15.83:45946) 2024-03-06T07:31:10.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.577+0000 7f760c80c640 1 -- 172.21.15.83:0/2469655359 learned_addr learned my addr 172.21.15.83:0/2469655359 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:10.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.577+0000 7f76077fe640 1 --2- 172.21.15.83:0/2469655359 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f76080d22b0 0x7f7608160640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.577+0000 7f7607fff640 1 --2- 172.21.15.83:0/2469655359 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f76080b41f0 0x7f76081f4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.577+0000 7f760c80c640 1 -- 172.21.15.83:0/2469655359 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f76080b41f0 msgr2=0x7f76081f4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.577+0000 7f760c80c640 1 --2- 172.21.15.83:0/2469655359 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f76080b41f0 0x7f76081f4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.577+0000 7f760c80c640 1 -- 172.21.15.83:0/2469655359 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f76080d22b0 msgr2=0x7f7608160640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.577+0000 7f760c80c640 1 --2- 172.21.15.83:0/2469655359 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f76080d22b0 0x7f7608160640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.577+0000 7f760c80c640 1 -- 172.21.15.83:0/2469655359 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76081f4700 con 0x7f760815dfe0 2024-03-06T07:31:10.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.578+0000 7f760c80c640 1 --2- 172.21.15.83:0/2469655359 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f760815dfe0 0x7f7608160b80 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f760000c7d0 tx=0x7f760000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:10.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.578+0000 7f7604ff9640 1 -- 172.21.15.83:0/2469655359 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7600003b70 con 0x7f760815dfe0 2024-03-06T07:31:10.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.578+0000 7f7604ff9640 1 -- 172.21.15.83:0/2469655359 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7600003d30 con 0x7f760815dfe0 2024-03-06T07:31:10.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.578+0000 7f760e26f640 1 -- 172.21.15.83:0/2469655359 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76080caef0 con 0x7f760815dfe0 2024-03-06T07:31:10.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.578+0000 7f7604ff9640 1 -- 172.21.15.83:0/2469655359 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f760000aa90 con 0x7f760815dfe0 2024-03-06T07:31:10.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.578+0000 7f760e26f640 1 -- 172.21.15.83:0/2469655359 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76080cb4b0 con 0x7f760815dfe0 2024-03-06T07:31:10.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.579+0000 7f760e26f640 1 -- 172.21.15.83:0/2469655359 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f760804f330 con 0x7f760815dfe0 2024-03-06T07:31:10.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.579+0000 7f7604ff9640 1 -- 172.21.15.83:0/2469655359 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f760000ac30 con 0x7f760815dfe0 2024-03-06T07:31:10.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.580+0000 7f7604ff9640 1 --2- 172.21.15.83:0/2469655359 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f75f406b4d0 0x7f75f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.580+0000 7f7607fff640 1 --2- 172.21.15.83:0/2469655359 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f75f406b4d0 0x7f75f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.580+0000 7f7604ff9640 1 -- 172.21.15.83:0/2469655359 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f760008a170 con 0x7f760815dfe0 2024-03-06T07:31:10.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.580+0000 7f7607fff640 1 --2- 172.21.15.83:0/2469655359 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f75f406b4d0 0x7f75f406d990 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f75f80011e0 tx=0x7f75f8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:10.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.584+0000 7f7604ff9640 1 -- 172.21.15.83:0/2469655359 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7600058fb0 con 0x7f760815dfe0 2024-03-06T07:31:10.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.799+0000 7f760e26f640 1 -- 172.21.15.83:0/2469655359 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76080b09c0 con 0x7f760815dfe0 2024-03-06T07:31:10.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.800+0000 7f7604ff9640 1 -- 172.21.15.83:0/2469655359 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f76000588b0 con 0x7f760815dfe0 2024-03-06T07:31:10.802 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:31:10.802 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:31:10.802 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:31:10.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.802+0000 7f760e26f640 1 -- 172.21.15.83:0/2469655359 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f75f406b4d0 msgr2=0x7f75f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.802+0000 7f760e26f640 1 --2- 172.21.15.83:0/2469655359 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f75f406b4d0 0x7f75f406d990 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f75f80011e0 tx=0x7f75f8009040 comp rx=0 tx=0).stop 2024-03-06T07:31:10.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.802+0000 7f760e26f640 1 -- 172.21.15.83:0/2469655359 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f760815dfe0 msgr2=0x7f7608160b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.802+0000 7f760e26f640 1 --2- 172.21.15.83:0/2469655359 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f760815dfe0 0x7f7608160b80 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f760000c7d0 tx=0x7f760000cc90 comp rx=0 tx=0).stop 2024-03-06T07:31:10.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.802+0000 7f760e26f640 1 -- 172.21.15.83:0/2469655359 shutdown_connections 2024-03-06T07:31:10.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.802+0000 7f760e26f640 1 --2- 172.21.15.83:0/2469655359 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f75f406b4d0 0x7f75f406d990 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.802+0000 7f760e26f640 1 --2- 172.21.15.83:0/2469655359 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f760815dfe0 0x7f7608160b80 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.802+0000 7f760e26f640 1 --2- 172.21.15.83:0/2469655359 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f76080d22b0 0x7f7608160640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.802+0000 7f760e26f640 1 --2- 172.21.15.83:0/2469655359 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f76080b41f0 0x7f76081f4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.802+0000 7f760e26f640 1 -- 172.21.15.83:0/2469655359 >> 172.21.15.83:0/2469655359 conn(0x7f76080d3cd0 msgr2=0x7f76080d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:10.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.802+0000 7f760e26f640 1 -- 172.21.15.83:0/2469655359 shutdown_connections 2024-03-06T07:31:10.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:10.802+0000 7f760e26f640 1 -- 172.21.15.83:0/2469655359 wait complete. 2024-03-06T07:31:11.016 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:31:11.016 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:31:11.073 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:12.017 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:12.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.117+0000 7fad5c666640 1 Processor -- start 2024-03-06T07:31:12.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.118+0000 7fad5c666640 1 -- start start 2024-03-06T07:31:12.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.119+0000 7fad5c666640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fad5415fa10 0x7fad540ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.119+0000 7fad5c666640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad540cebe0 con 0x7fad540d16e0 2024-03-06T07:31:12.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.119+0000 7fad5c666640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad540ceec0 con 0x7fad5415bdd0 2024-03-06T07:31:12.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.119+0000 7fad5c666640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad540cf1a0 con 0x7fad5415fa10 2024-03-06T07:31:12.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.119+0000 7fad5ac03640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fad540d16e0 0x7fad540d1b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:54056/0 (socket says 172.21.15.83:54056) 2024-03-06T07:31:12.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.119+0000 7fad5ac03640 1 -- 172.21.15.83:0/3059993767 learned_addr learned my addr 172.21.15.83:0/3059993767 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:31:12.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.119+0000 7fad59c01640 1 --2- 172.21.15.83:0/3059993767 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fad5415fa10 0x7fad540ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.120+0000 7fad59400640 1 -- 172.21.15.83:0/3059993767 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3767214623 0 0) 0x7fad540cebe0 con 0x7fad540d16e0 2024-03-06T07:31:12.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.120+0000 7fad59400640 1 -- 172.21.15.83:0/3059993767 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad40003520 con 0x7fad540d16e0 2024-03-06T07:31:12.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.120+0000 7fad59400640 1 -- 172.21.15.83:0/3059993767 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1033086256 0 0) 0x7fad540ceec0 con 0x7fad5415bdd0 2024-03-06T07:31:12.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.120+0000 7fad59400640 1 -- 172.21.15.83:0/3059993767 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad540cebe0 con 0x7fad5415bdd0 2024-03-06T07:31:12.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.120+0000 7fad59400640 1 -- 172.21.15.83:0/3059993767 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2158530998 0 0) 0x7fad40003520 con 0x7fad540d16e0 2024-03-06T07:31:12.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.120+0000 7fad59c01640 1 -- 172.21.15.83:0/3059993767 >> v1:172.21.15.183:6789/0 conn(0x7fad5415bdd0 legacy=0x7fad540c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.120+0000 7fad59c01640 1 -- 172.21.15.83:0/3059993767 >> v1:172.21.15.83:6789/0 conn(0x7fad540d16e0 legacy=0x7fad540d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.120+0000 7fad59c01640 1 -- 172.21.15.83:0/3059993767 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad540cc8e0 con 0x7fad5415fa10 2024-03-06T07:31:12.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.121+0000 7fad59c01640 1 --2- 172.21.15.83:0/3059993767 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fad5415fa10 0x7fad540ce270 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fad44009750 tx=0x7fad4402b660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5bd3a3add7f0bf16 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:12.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.121+0000 7fad59400640 1 -- 172.21.15.83:0/3059993767 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad44037070 con 0x7fad5415fa10 2024-03-06T07:31:12.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.121+0000 7fad59400640 1 -- 172.21.15.83:0/3059993767 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad44030510 con 0x7fad5415fa10 2024-03-06T07:31:12.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.121+0000 7fad59400640 1 -- 172.21.15.83:0/3059993767 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4402be60 con 0x7fad5415fa10 2024-03-06T07:31:12.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.122+0000 7fad5c666640 1 -- 172.21.15.83:0/3059993767 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fad5415fa10 msgr2=0x7fad540ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.122+0000 7fad5c666640 1 --2- 172.21.15.83:0/3059993767 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fad5415fa10 0x7fad540ce270 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fad44009750 tx=0x7fad4402b660 comp rx=0 tx=0).stop 2024-03-06T07:31:12.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.122+0000 7fad5c666640 1 -- 172.21.15.83:0/3059993767 shutdown_connections 2024-03-06T07:31:12.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.122+0000 7fad5c666640 1 --2- 172.21.15.83:0/3059993767 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fad5415fa10 0x7fad540ce270 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.122+0000 7fad5c666640 1 -- 172.21.15.83:0/3059993767 >> 172.21.15.83:0/3059993767 conn(0x7fad5409a9f0 msgr2=0x7fad5409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:12.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.122+0000 7fad5c666640 1 -- 172.21.15.83:0/3059993767 shutdown_connections 2024-03-06T07:31:12.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.122+0000 7fad5c666640 1 -- 172.21.15.83:0/3059993767 wait complete. 2024-03-06T07:31:12.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.123+0000 7fad5c666640 1 Processor -- start 2024-03-06T07:31:12.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.123+0000 7fad5c666640 1 -- start start 2024-03-06T07:31:12.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.124+0000 7fad5c666640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad540d16e0 0x7fad541553b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.124+0000 7fad5c666640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fad5415bdd0 0x7fad540c9c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.124+0000 7fad5c666640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fad5415fa10 0x7fad540b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.124+0000 7fad5c666640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad540ce7b0 con 0x7fad540d16e0 2024-03-06T07:31:12.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.124+0000 7fad5c666640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad5415c1b0 con 0x7fad5415fa10 2024-03-06T07:31:12.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.124+0000 7fad5c666640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad540ce930 con 0x7fad5415bdd0 2024-03-06T07:31:12.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.124+0000 7fad5a402640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad540d16e0 0x7fad541553b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.124+0000 7fad5a402640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad540d16e0 0x7fad541553b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45960/0 (socket says 172.21.15.83:45960) 2024-03-06T07:31:12.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.124+0000 7fad5a402640 1 -- 172.21.15.83:0/298807917 learned_addr learned my addr 172.21.15.83:0/298807917 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:12.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.124+0000 7fad59c01640 1 --2- 172.21.15.83:0/298807917 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fad5415bdd0 0x7fad540c9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.124+0000 7fad5ac03640 1 --2- 172.21.15.83:0/298807917 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fad5415fa10 0x7fad540b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.125+0000 7fad5a402640 1 -- 172.21.15.83:0/298807917 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fad5415bdd0 msgr2=0x7fad540c9c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.125+0000 7fad5a402640 1 --2- 172.21.15.83:0/298807917 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fad5415bdd0 0x7fad540c9c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.125+0000 7fad5a402640 1 -- 172.21.15.83:0/298807917 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fad5415fa10 msgr2=0x7fad540b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.125+0000 7fad5a402640 1 --2- 172.21.15.83:0/298807917 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fad5415fa10 0x7fad540b68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.125+0000 7fad5a402640 1 -- 172.21.15.83:0/298807917 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad540c9050 con 0x7fad540d16e0 2024-03-06T07:31:12.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.125+0000 7fad5a402640 1 --2- 172.21.15.83:0/298807917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad540d16e0 0x7fad541553b0 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7fad500015c0 tx=0x7fad5000cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:12.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.125+0000 7fad4affd640 1 -- 172.21.15.83:0/298807917 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad50018070 con 0x7fad540d16e0 2024-03-06T07:31:12.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.125+0000 7fad4affd640 1 -- 172.21.15.83:0/298807917 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad50007050 con 0x7fad540d16e0 2024-03-06T07:31:12.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.125+0000 7fad4affd640 1 -- 172.21.15.83:0/298807917 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad500074c0 con 0x7fad540d16e0 2024-03-06T07:31:12.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.125+0000 7fad5c666640 1 -- 172.21.15.83:0/298807917 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad5415daa0 con 0x7fad540d16e0 2024-03-06T07:31:12.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.125+0000 7fad5c666640 1 -- 172.21.15.83:0/298807917 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad5415e060 con 0x7fad540d16e0 2024-03-06T07:31:12.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.126+0000 7fad5c666640 1 -- 172.21.15.83:0/298807917 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad5404f330 con 0x7fad540d16e0 2024-03-06T07:31:12.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.126+0000 7fad4affd640 1 -- 172.21.15.83:0/298807917 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fad5001b070 con 0x7fad540d16e0 2024-03-06T07:31:12.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.127+0000 7fad4affd640 1 --2- 172.21.15.83:0/298807917 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fad3806b4d0 0x7fad3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.127+0000 7fad59c01640 1 --2- 172.21.15.83:0/298807917 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fad3806b4d0 0x7fad3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.127+0000 7fad4affd640 1 -- 172.21.15.83:0/298807917 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fad50089920 con 0x7fad540d16e0 2024-03-06T07:31:12.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.127+0000 7fad59c01640 1 --2- 172.21.15.83:0/298807917 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fad3806b4d0 0x7fad3806d990 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fad44034000 tx=0x7fad44008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:12.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.131+0000 7fad4affd640 1 -- 172.21.15.83:0/298807917 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fad50007660 con 0x7fad540d16e0 2024-03-06T07:31:12.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.343+0000 7fad5c666640 1 -- 172.21.15.83:0/298807917 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad540c7bb0 con 0x7fad540d16e0 2024-03-06T07:31:12.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.344+0000 7fad4affd640 1 -- 172.21.15.83:0/298807917 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7fad50058760 con 0x7fad540d16e0 2024-03-06T07:31:12.346 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:31:12.346 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:31:12.346 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:31:12.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.346+0000 7fad5c666640 1 -- 172.21.15.83:0/298807917 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fad3806b4d0 msgr2=0x7fad3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.346+0000 7fad5c666640 1 --2- 172.21.15.83:0/298807917 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fad3806b4d0 0x7fad3806d990 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fad44034000 tx=0x7fad44008030 comp rx=0 tx=0).stop 2024-03-06T07:31:12.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.346+0000 7fad5c666640 1 -- 172.21.15.83:0/298807917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad540d16e0 msgr2=0x7fad541553b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.346+0000 7fad5c666640 1 --2- 172.21.15.83:0/298807917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad540d16e0 0x7fad541553b0 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7fad500015c0 tx=0x7fad5000cd20 comp rx=0 tx=0).stop 2024-03-06T07:31:12.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.346+0000 7fad5c666640 1 -- 172.21.15.83:0/298807917 shutdown_connections 2024-03-06T07:31:12.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.346+0000 7fad5c666640 1 --2- 172.21.15.83:0/298807917 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fad3806b4d0 0x7fad3806d990 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.346+0000 7fad5c666640 1 --2- 172.21.15.83:0/298807917 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fad5415fa10 0x7fad540b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.346+0000 7fad5c666640 1 --2- 172.21.15.83:0/298807917 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fad5415bdd0 0x7fad540c9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.346+0000 7fad5c666640 1 --2- 172.21.15.83:0/298807917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad540d16e0 0x7fad541553b0 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.346+0000 7fad5c666640 1 -- 172.21.15.83:0/298807917 >> 172.21.15.83:0/298807917 conn(0x7fad5409a9f0 msgr2=0x7fad540c55f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:12.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.346+0000 7fad5c666640 1 -- 172.21.15.83:0/298807917 shutdown_connections 2024-03-06T07:31:12.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:12.346+0000 7fad5c666640 1 -- 172.21.15.83:0/298807917 wait complete. 2024-03-06T07:31:12.560 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:31:12.560 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:31:13.075 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:13.561 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:13.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.665+0000 7fbfc8d3a640 1 Processor -- start 2024-03-06T07:31:13.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.665+0000 7fbfc8d3a640 1 -- start start 2024-03-06T07:31:13.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.666+0000 7fbfc8d3a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fbfc41515e0 0x7fbfc40cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.666+0000 7fbfc8d3a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbfc415b430 0x7fbfc4071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.666+0000 7fbfc8d3a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfc415b8b0 con 0x7fbfc40cd7a0 2024-03-06T07:31:13.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.666+0000 7fbfc8d3a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfc4070a70 con 0x7fbfc41515e0 2024-03-06T07:31:13.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.666+0000 7fbfc8d3a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc4070d50 con 0x7fbfc415b430 2024-03-06T07:31:13.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.666+0000 7fbfc2d9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fbfc40cd7a0 0x7fbfc40cdbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:54062/0 (socket says 172.21.15.83:54062) 2024-03-06T07:31:13.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.666+0000 7fbfc2d9d640 1 -- 172.21.15.83:0/650053042 learned_addr learned my addr 172.21.15.83:0/650053042 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:31:13.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.666+0000 7fbfc1d9b640 1 --2- 172.21.15.83:0/650053042 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbfc415b430 0x7fbfc4071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.666+0000 7fbfc259c640 1 --2- 172.21.15.83:0/650053042 >> v2:172.21.15.183:3300/0 conn(0x7fbfc41515e0 0x7fbfc40cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.666+0000 7fbfc159a640 1 -- 172.21.15.83:0/650053042 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1779069286 0 0) 0x7fbfc415b8b0 con 0x7fbfc40cd7a0 2024-03-06T07:31:13.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.666+0000 7fbfc159a640 1 -- 172.21.15.83:0/650053042 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfac003520 con 0x7fbfc40cd7a0 2024-03-06T07:31:13.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.666+0000 7fbfc159a640 1 -- 172.21.15.83:0/650053042 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 49478341 0 0) 0x7fbfac003520 con 0x7fbfc40cd7a0 2024-03-06T07:31:13.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.666+0000 7fbfc259c640 1 -- 172.21.15.83:0/650053042 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbfc415b430 msgr2=0x7fbfc4071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:13.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.666+0000 7fbfc259c640 1 --2- 172.21.15.83:0/650053042 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbfc415b430 0x7fbfc4071830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.666+0000 7fbfc259c640 1 -- 172.21.15.83:0/650053042 >> v1:172.21.15.83:6789/0 conn(0x7fbfc40cd7a0 legacy=0x7fbfc40cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.666+0000 7fbfc259c640 1 -- 172.21.15.83:0/650053042 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc4071030 con 0x7fbfc41515e0 2024-03-06T07:31:13.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.667+0000 7fbfc259c640 1 --2- 172.21.15.83:0/650053042 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbfc41515e0 0x7fbfc40cce40 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fbfb40097a0 tx=0x7fbfb402b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2827547bbb9a4aaf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:13.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.667+0000 7fbfc159a640 1 -- 172.21.15.83:0/650053042 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfb4037070 con 0x7fbfc41515e0 2024-03-06T07:31:13.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.667+0000 7fbfc159a640 1 -- 172.21.15.83:0/650053042 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfb4030510 con 0x7fbfc41515e0 2024-03-06T07:31:13.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.667+0000 7fbfc159a640 1 -- 172.21.15.83:0/650053042 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfbc0019f0 con 0x7fbfc41515e0 2024-03-06T07:31:13.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.668+0000 7fbfc8d3a640 1 -- 172.21.15.83:0/650053042 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbfc41515e0 msgr2=0x7fbfc40cce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.668+0000 7fbfc8d3a640 1 --2- 172.21.15.83:0/650053042 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbfc41515e0 0x7fbfc40cce40 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fbfb40097a0 tx=0x7fbfb402b790 comp rx=0 tx=0).stop 2024-03-06T07:31:13.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.668+0000 7fbfc8d3a640 1 -- 172.21.15.83:0/650053042 shutdown_connections 2024-03-06T07:31:13.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.668+0000 7fbfc8d3a640 1 --2- 172.21.15.83:0/650053042 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbfc415b430 0x7fbfc4071830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.668+0000 7fbfc8d3a640 1 --2- 172.21.15.83:0/650053042 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbfc41515e0 0x7fbfc40cce40 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.668+0000 7fbfc8d3a640 1 -- 172.21.15.83:0/650053042 >> 172.21.15.83:0/650053042 conn(0x7fbfc40cf8a0 msgr2=0x7fbfc40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:13.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.668+0000 7fbfc8d3a640 1 -- 172.21.15.83:0/650053042 shutdown_connections 2024-03-06T07:31:13.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.668+0000 7fbfc8d3a640 1 -- 172.21.15.83:0/650053042 wait complete. 2024-03-06T07:31:13.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.669+0000 7fbfc8d3a640 1 Processor -- start 2024-03-06T07:31:13.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.669+0000 7fbfc8d3a640 1 -- start start 2024-03-06T07:31:13.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.670+0000 7fbfc8d3a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbfc40cd7a0 0x7fbfc415e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.670+0000 7fbfc8d3a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbfc41515e0 0x7fbfc414b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.670+0000 7fbfc8d3a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbfc415b430 0x7fbfc41f3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.670+0000 7fbfc8d3a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc4071ef0 con 0x7fbfc415b430 2024-03-06T07:31:13.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.670+0000 7fbfc8d3a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc4071d70 con 0x7fbfc40cd7a0 2024-03-06T07:31:13.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.670+0000 7fbfc8d3a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc40cc3a0 con 0x7fbfc41515e0 2024-03-06T07:31:13.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.670+0000 7fbfc2d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbfc415b430 0x7fbfc41f3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.670+0000 7fbfc2d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbfc415b430 0x7fbfc41f3cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45968/0 (socket says 172.21.15.83:45968) 2024-03-06T07:31:13.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.670+0000 7fbfc2d9d640 1 -- 172.21.15.83:0/1049402563 learned_addr learned my addr 172.21.15.83:0/1049402563 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:13.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.670+0000 7fbfc259c640 1 --2- 172.21.15.83:0/1049402563 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbfc40cd7a0 0x7fbfc415e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.670+0000 7fbfc1d9b640 1 --2- 172.21.15.83:0/1049402563 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbfc41515e0 0x7fbfc414b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.670+0000 7fbfc2d9d640 1 -- 172.21.15.83:0/1049402563 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbfc41515e0 msgr2=0x7fbfc414b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.670+0000 7fbfc2d9d640 1 --2- 172.21.15.83:0/1049402563 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbfc41515e0 0x7fbfc414b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.670+0000 7fbfc2d9d640 1 -- 172.21.15.83:0/1049402563 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbfc40cd7a0 msgr2=0x7fbfc415e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.670+0000 7fbfc2d9d640 1 --2- 172.21.15.83:0/1049402563 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbfc40cd7a0 0x7fbfc415e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.670+0000 7fbfc2d9d640 1 -- 172.21.15.83:0/1049402563 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc414b600 con 0x7fbfc415b430 2024-03-06T07:31:13.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.670+0000 7fbfc259c640 1 --2- 172.21.15.83:0/1049402563 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbfc40cd7a0 0x7fbfc415e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:13.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.670+0000 7fbfc2d9d640 1 --2- 172.21.15.83:0/1049402563 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbfc415b430 0x7fbfc41f3cf0 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc001890 tx=0x7fbfbc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:13.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.671+0000 7fbfb2ffd640 1 -- 172.21.15.83:0/1049402563 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfbc012400 con 0x7fbfc415b430 2024-03-06T07:31:13.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.671+0000 7fbfb2ffd640 1 -- 172.21.15.83:0/1049402563 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfbc0125c0 con 0x7fbfc415b430 2024-03-06T07:31:13.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.671+0000 7fbfc8d3a640 1 -- 172.21.15.83:0/1049402563 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfc4151e30 con 0x7fbfc415b430 2024-03-06T07:31:13.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.671+0000 7fbfb2ffd640 1 -- 172.21.15.83:0/1049402563 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfbc0129b0 con 0x7fbfc415b430 2024-03-06T07:31:13.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.671+0000 7fbfc8d3a640 1 -- 172.21.15.83:0/1049402563 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfc41523f0 con 0x7fbfc415b430 2024-03-06T07:31:13.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.672+0000 7fbfb2ffd640 1 -- 172.21.15.83:0/1049402563 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbfbc012b50 con 0x7fbfc415b430 2024-03-06T07:31:13.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.672+0000 7fbfc8d3a640 1 -- 172.21.15.83:0/1049402563 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfc4068840 con 0x7fbfc415b430 2024-03-06T07:31:13.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.672+0000 7fbfb2ffd640 1 --2- 172.21.15.83:0/1049402563 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbfa406b500 0x7fbfa406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.672+0000 7fbfb2ffd640 1 -- 172.21.15.83:0/1049402563 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fbfbc08aae0 con 0x7fbfc415b430 2024-03-06T07:31:13.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.672+0000 7fbfc259c640 1 --2- 172.21.15.83:0/1049402563 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbfa406b500 0x7fbfa406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.673+0000 7fbfc259c640 1 --2- 172.21.15.83:0/1049402563 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbfa406b500 0x7fbfa406d9c0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fbfb4034000 tx=0x7fbfb402f000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:13.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.677+0000 7fbfb2ffd640 1 -- 172.21.15.83:0/1049402563 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbfbc0182b0 con 0x7fbfc415b430 2024-03-06T07:31:13.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.891+0000 7fbfc8d3a640 1 -- 172.21.15.83:0/1049402563 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbfc404f330 con 0x7fbfc415b430 2024-03-06T07:31:13.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.892+0000 7fbfb2ffd640 1 -- 172.21.15.83:0/1049402563 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7fbfbc059920 con 0x7fbfc415b430 2024-03-06T07:31:13.894 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:31:13.894 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:31:13.894 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:31:13.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.894+0000 7fbfc8d3a640 1 -- 172.21.15.83:0/1049402563 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbfa406b500 msgr2=0x7fbfa406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.894+0000 7fbfc8d3a640 1 --2- 172.21.15.83:0/1049402563 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbfa406b500 0x7fbfa406d9c0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fbfb4034000 tx=0x7fbfb402f000 comp rx=0 tx=0).stop 2024-03-06T07:31:13.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.894+0000 7fbfc8d3a640 1 -- 172.21.15.83:0/1049402563 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbfc415b430 msgr2=0x7fbfc41f3cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.894+0000 7fbfc8d3a640 1 --2- 172.21.15.83:0/1049402563 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbfc415b430 0x7fbfc41f3cf0 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc001890 tx=0x7fbfbc00a430 comp rx=0 tx=0).stop 2024-03-06T07:31:13.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.894+0000 7fbfc8d3a640 1 -- 172.21.15.83:0/1049402563 shutdown_connections 2024-03-06T07:31:13.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.894+0000 7fbfc8d3a640 1 --2- 172.21.15.83:0/1049402563 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbfa406b500 0x7fbfa406d9c0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.894+0000 7fbfc8d3a640 1 --2- 172.21.15.83:0/1049402563 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbfc415b430 0x7fbfc41f3cf0 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.894+0000 7fbfc8d3a640 1 --2- 172.21.15.83:0/1049402563 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbfc41515e0 0x7fbfc414b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.894+0000 7fbfc8d3a640 1 --2- 172.21.15.83:0/1049402563 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbfc40cd7a0 0x7fbfc415e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.894+0000 7fbfc8d3a640 1 -- 172.21.15.83:0/1049402563 >> 172.21.15.83:0/1049402563 conn(0x7fbfc40cf8a0 msgr2=0x7fbfc414f650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:13.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.894+0000 7fbfc8d3a640 1 -- 172.21.15.83:0/1049402563 shutdown_connections 2024-03-06T07:31:13.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:13.894+0000 7fbfc8d3a640 1 -- 172.21.15.83:0/1049402563 wait complete. 2024-03-06T07:31:14.109 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:31:14.109 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:31:15.077 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:15.110 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:15.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.208+0000 7f0925720640 1 Processor -- start 2024-03-06T07:31:15.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.209+0000 7f0925720640 1 -- start start 2024-03-06T07:31:15.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.210+0000 7f0925720640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f092015bb20 0x7f09200d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.210+0000 7f0925720640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f09200cfff0 0x7f09200d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.210+0000 7f0925720640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0920161980 con 0x7f09200ce4b0 2024-03-06T07:31:15.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.210+0000 7f0925720640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0920161c60 con 0x7f09200cfff0 2024-03-06T07:31:15.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.210+0000 7f0925720640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0920161f40 con 0x7f092015bb20 2024-03-06T07:31:15.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.210+0000 7f0917fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f09200cfff0 0x7f09200d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.210+0000 7f0917fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f09200cfff0 0x7f09200d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:45984/0 (socket says 172.21.15.83:45984) 2024-03-06T07:31:15.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.210+0000 7f0917fff640 1 -- 172.21.15.83:0/1377132625 learned_addr learned my addr 172.21.15.83:0/1377132625 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:15.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.210+0000 7f091effd640 1 --2- 172.21.15.83:0/1377132625 >> v2:172.21.15.183:3300/0 conn(0x7f092015bb20 0x7f09200d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.210+0000 7f0917fff640 1 -- 172.21.15.83:0/1377132625 >> v2:172.21.15.183:3300/0 conn(0x7f092015bb20 msgr2=0x7f09200d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:15.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.210+0000 7f0917fff640 1 --2- 172.21.15.83:0/1377132625 >> v2:172.21.15.183:3300/0 conn(0x7f092015bb20 0x7f09200d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.210+0000 7f0917fff640 1 -- 172.21.15.83:0/1377132625 >> v1:172.21.15.183:6789/0 conn(0x7f09200ce4b0 legacy=0x7f09200ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.210+0000 7f0917fff640 1 -- 172.21.15.83:0/1377132625 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0920162220 con 0x7f09200cfff0 2024-03-06T07:31:15.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.211+0000 7f0917fff640 1 --2- 172.21.15.83:0/1377132625 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09200cfff0 0x7f09200d04a0 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f09200d2bc0 tx=0x7f090802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d24521a9a838dfab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:15.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.212+0000 7f091e7fc640 1 -- 172.21.15.83:0/1377132625 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0908034070 con 0x7f09200cfff0 2024-03-06T07:31:15.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.212+0000 7f091e7fc640 1 -- 172.21.15.83:0/1377132625 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0908030860 con 0x7f09200cfff0 2024-03-06T07:31:15.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.212+0000 7f091e7fc640 1 -- 172.21.15.83:0/1377132625 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0908030c30 con 0x7f09200cfff0 2024-03-06T07:31:15.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.212+0000 7f0925720640 1 -- 172.21.15.83:0/1377132625 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09200cfff0 msgr2=0x7f09200d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.212+0000 7f0925720640 1 --2- 172.21.15.83:0/1377132625 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09200cfff0 0x7f09200d04a0 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f09200d2bc0 tx=0x7f090802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.212+0000 7f0925720640 1 -- 172.21.15.83:0/1377132625 shutdown_connections 2024-03-06T07:31:15.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.212+0000 7f0925720640 1 --2- 172.21.15.83:0/1377132625 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09200cfff0 0x7f09200d04a0 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.212+0000 7f0925720640 1 --2- 172.21.15.83:0/1377132625 >> v2:172.21.15.183:3300/0 conn(0x7f092015bb20 0x7f09200d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.212+0000 7f0925720640 1 -- 172.21.15.83:0/1377132625 >> 172.21.15.83:0/1377132625 conn(0x7f092009a9f0 msgr2=0x7f092009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:15.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.212+0000 7f0925720640 1 -- 172.21.15.83:0/1377132625 shutdown_connections 2024-03-06T07:31:15.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.212+0000 7f0925720640 1 -- 172.21.15.83:0/1377132625 wait complete. 2024-03-06T07:31:15.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.213+0000 7f0925720640 1 Processor -- start 2024-03-06T07:31:15.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.213+0000 7f0925720640 1 -- start start 2024-03-06T07:31:15.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.214+0000 7f0925720640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f09200ce4b0 0x7f0920160b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.214+0000 7f0925720640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09200cfff0 0x7f09201f4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.214+0000 7f0925720640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f092015bb20 0x7f09201f4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.214+0000 7f0925720640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09200d24b0 con 0x7f09200cfff0 2024-03-06T07:31:15.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.214+0000 7f0925720640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09200d2330 con 0x7f092015bb20 2024-03-06T07:31:15.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.214+0000 7f0925720640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09200d1980 con 0x7f09200ce4b0 2024-03-06T07:31:15.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.214+0000 7f0917fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09200cfff0 0x7f09201f4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.214+0000 7f0917fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09200cfff0 0x7f09201f4120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46000/0 (socket says 172.21.15.83:46000) 2024-03-06T07:31:15.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.214+0000 7f0917fff640 1 -- 172.21.15.83:0/2956040596 learned_addr learned my addr 172.21.15.83:0/2956040596 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:15.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.214+0000 7f091f7fe640 1 --2- 172.21.15.83:0/2956040596 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f092015bb20 0x7f09201f4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.214+0000 7f091effd640 1 --2- 172.21.15.83:0/2956040596 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f09200ce4b0 0x7f0920160b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.214+0000 7f0917fff640 1 -- 172.21.15.83:0/2956040596 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f09200ce4b0 msgr2=0x7f0920160b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.214+0000 7f0917fff640 1 --2- 172.21.15.83:0/2956040596 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f09200ce4b0 0x7f0920160b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.214+0000 7f0917fff640 1 -- 172.21.15.83:0/2956040596 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f092015bb20 msgr2=0x7f09201f4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.214+0000 7f0917fff640 1 --2- 172.21.15.83:0/2956040596 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f092015bb20 0x7f09201f4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.214+0000 7f0917fff640 1 -- 172.21.15.83:0/2956040596 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0920155860 con 0x7f09200cfff0 2024-03-06T07:31:15.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.214+0000 7f091f7fe640 1 --2- 172.21.15.83:0/2956040596 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f092015bb20 0x7f09201f4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:15.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.215+0000 7f0917fff640 1 --2- 172.21.15.83:0/2956040596 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09200cfff0 0x7f09201f4120 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f090802b9e0 tx=0x7f0908004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:15.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.215+0000 7f09177fe640 1 -- 172.21.15.83:0/2956040596 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0908048070 con 0x7f09200cfff0 2024-03-06T07:31:15.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.215+0000 7f09177fe640 1 -- 172.21.15.83:0/2956040596 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0908030660 con 0x7f09200cfff0 2024-03-06T07:31:15.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.215+0000 7f0925720640 1 -- 172.21.15.83:0/2956040596 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f092015e320 con 0x7f09200cfff0 2024-03-06T07:31:15.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.215+0000 7f09177fe640 1 -- 172.21.15.83:0/2956040596 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0908034040 con 0x7f09200cfff0 2024-03-06T07:31:15.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.215+0000 7f0925720640 1 -- 172.21.15.83:0/2956040596 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f092015e8c0 con 0x7f09200cfff0 2024-03-06T07:31:15.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.216+0000 7f0925720640 1 -- 172.21.15.83:0/2956040596 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f092004f330 con 0x7f09200cfff0 2024-03-06T07:31:15.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.217+0000 7f09177fe640 1 -- 172.21.15.83:0/2956040596 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f090804b070 con 0x7f09200cfff0 2024-03-06T07:31:15.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.217+0000 7f09177fe640 1 --2- 172.21.15.83:0/2956040596 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f090006b500 0x7f090006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.217+0000 7f09177fe640 1 -- 172.21.15.83:0/2956040596 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f09080ac240 con 0x7f09200cfff0 2024-03-06T07:31:15.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.217+0000 7f091effd640 1 --2- 172.21.15.83:0/2956040596 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f090006b500 0x7f090006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.218+0000 7f091effd640 1 --2- 172.21.15.83:0/2956040596 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f090006b500 0x7f090006d9c0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f0910006fd0 tx=0x7f0910008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:15.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.222+0000 7f09177fe640 1 -- 172.21.15.83:0/2956040596 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f090807b080 con 0x7f09200cfff0 2024-03-06T07:31:15.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.436+0000 7f0925720640 1 -- 172.21.15.83:0/2956040596 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f092015df90 con 0x7f09200cfff0 2024-03-06T07:31:15.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.437+0000 7f09177fe640 1 -- 172.21.15.83:0/2956040596 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f090807a980 con 0x7f09200cfff0 2024-03-06T07:31:15.439 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:31:15.439 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:31:15.439 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:31:15.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.439+0000 7f0925720640 1 -- 172.21.15.83:0/2956040596 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f090006b500 msgr2=0x7f090006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.439+0000 7f0925720640 1 --2- 172.21.15.83:0/2956040596 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f090006b500 0x7f090006d9c0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f0910006fd0 tx=0x7f0910008040 comp rx=0 tx=0).stop 2024-03-06T07:31:15.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.439+0000 7f0925720640 1 -- 172.21.15.83:0/2956040596 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09200cfff0 msgr2=0x7f09201f4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.439+0000 7f0925720640 1 --2- 172.21.15.83:0/2956040596 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09200cfff0 0x7f09201f4120 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f090802b9e0 tx=0x7f0908004730 comp rx=0 tx=0).stop 2024-03-06T07:31:15.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.439+0000 7f0925720640 1 -- 172.21.15.83:0/2956040596 shutdown_connections 2024-03-06T07:31:15.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.439+0000 7f0925720640 1 --2- 172.21.15.83:0/2956040596 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f090006b500 0x7f090006d9c0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.439+0000 7f0925720640 1 --2- 172.21.15.83:0/2956040596 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f092015bb20 0x7f09201f4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.439+0000 7f0925720640 1 --2- 172.21.15.83:0/2956040596 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09200cfff0 0x7f09201f4120 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.439+0000 7f0925720640 1 --2- 172.21.15.83:0/2956040596 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f09200ce4b0 0x7f0920160b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.439+0000 7f0925720640 1 -- 172.21.15.83:0/2956040596 >> 172.21.15.83:0/2956040596 conn(0x7f092009a9f0 msgr2=0x7f092006eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:15.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.439+0000 7f0925720640 1 -- 172.21.15.83:0/2956040596 shutdown_connections 2024-03-06T07:31:15.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:15.439+0000 7f0925720640 1 -- 172.21.15.83:0/2956040596 wait complete. 2024-03-06T07:31:15.652 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:31:15.653 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:31:16.654 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:16.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.754+0000 7f0ab1833640 1 Processor -- start 2024-03-06T07:31:16.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.754+0000 7f0ab1833640 1 -- start start 2024-03-06T07:31:16.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.755+0000 7f0ab1833640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0aac0d0a80 0x7f0aac0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.755+0000 7f0ab1833640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0aac0d25d0 con 0x7f0aac0d4760 2024-03-06T07:31:16.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.755+0000 7f0ab1833640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0aac0d2870 con 0x7f0aac0b4170 2024-03-06T07:31:16.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.755+0000 7f0ab1833640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aac0d4bf0 con 0x7f0aac0d0a80 2024-03-06T07:31:16.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.755+0000 7f0aaa7fc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f0aac0d4760 0x7f0aac0cee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48264/0 (socket says 172.21.15.83:48264) 2024-03-06T07:31:16.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.755+0000 7f0aaa7fc640 1 -- 172.21.15.83:0/85484064 learned_addr learned my addr 172.21.15.83:0/85484064 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:31:16.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.756+0000 7f0aa9ffb640 1 -- 172.21.15.83:0/85484064 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1543676380 0 0) 0x7f0aac0d25d0 con 0x7f0aac0d4760 2024-03-06T07:31:16.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.756+0000 7f0aab7fe640 1 --2- 172.21.15.83:0/85484064 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0aac0d0a80 0x7f0aac0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.756+0000 7f0aa9ffb640 1 -- 172.21.15.83:0/85484064 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a94003520 con 0x7f0aac0d4760 2024-03-06T07:31:16.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.756+0000 7f0aa9ffb640 1 -- 172.21.15.83:0/85484064 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3801750055 0 0) 0x7f0a94003520 con 0x7f0aac0d4760 2024-03-06T07:31:16.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.756+0000 7f0aa9ffb640 1 -- 172.21.15.83:0/85484064 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0aac0d0a80 msgr2=0x7f0aac0b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:16.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.756+0000 7f0aa9ffb640 1 --2- 172.21.15.83:0/85484064 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0aac0d0a80 0x7f0aac0b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.756+0000 7f0aa9ffb640 1 -- 172.21.15.83:0/85484064 >> v1:172.21.15.183:6789/0 conn(0x7f0aac0b4170 legacy=0x7f0aac0d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.756+0000 7f0aa9ffb640 1 -- 172.21.15.83:0/85484064 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aac0d2b50 con 0x7f0aac0d4760 2024-03-06T07:31:16.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.756+0000 7f0aa9ffb640 1 -- 172.21.15.83:0/85484064 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f0aa0001670 con 0x7f0aac0d4760 2024-03-06T07:31:16.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.756+0000 7f0aa9ffb640 1 -- 172.21.15.83:0/85484064 >> v1:172.21.15.83:6789/0 conn(0x7f0aac0d4760 legacy=0x7f0aac0cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.756+0000 7f0aa9ffb640 1 --2- 172.21.15.83:0/85484064 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a940039e0 0x7f0a94003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.756+0000 7f0aa9ffb640 1 --2- 172.21.15.83:0/85484064 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a94004800 0x7f0a94004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.756+0000 7f0aa9ffb640 1 --2- 172.21.15.83:0/85484064 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0aac0d0a80 0x7f0a940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.756+0000 7f0aa9ffb640 1 -- 172.21.15.83:0/85484064 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aac0d4bf0 con 0x7f0a94004800 2024-03-06T07:31:16.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.756+0000 7f0aa9ffb640 1 -- 172.21.15.83:0/85484064 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9400baa0 con 0x7f0a940039e0 2024-03-06T07:31:16.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.756+0000 7f0aa9ffb640 1 -- 172.21.15.83:0/85484064 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9400bca0 con 0x7f0aac0d0a80 2024-03-06T07:31:16.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.756+0000 7f0aaa7fc640 1 --2- 172.21.15.83:0/85484064 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a94004800 0x7f0a94004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.756+0000 7f0aaaffd640 1 --2- 172.21.15.83:0/85484064 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a940039e0 0x7f0a94003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.756+0000 7f0aab7fe640 1 --2- 172.21.15.83:0/85484064 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0aac0d0a80 0x7f0a940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.756+0000 7f0aaa7fc640 1 -- 172.21.15.83:0/85484064 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0aac0d0a80 msgr2=0x7f0a940051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:16.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.756+0000 7f0aaa7fc640 1 --2- 172.21.15.83:0/85484064 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0aac0d0a80 0x7f0a940051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.756+0000 7f0aaa7fc640 1 -- 172.21.15.83:0/85484064 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a940039e0 msgr2=0x7f0a94003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:16.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.756+0000 7f0aaa7fc640 1 --2- 172.21.15.83:0/85484064 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a940039e0 0x7f0a94003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.756+0000 7f0aaa7fc640 1 -- 172.21.15.83:0/85484064 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a94003520 con 0x7f0a94004800 2024-03-06T07:31:16.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.756+0000 7f0aab7fe640 1 --2- 172.21.15.83:0/85484064 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0aac0d0a80 0x7f0a940051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:16.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.756+0000 7f0aaaffd640 1 --2- 172.21.15.83:0/85484064 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a940039e0 0x7f0a94003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:16.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.757+0000 7f0aaa7fc640 1 --2- 172.21.15.83:0/85484064 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a94004800 0x7f0a94004c80 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f0aac06e080 tx=0x7f0a94005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12c64d98e70b63a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.757+0000 7f0aa9ffb640 1 -- 172.21.15.83:0/85484064 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa0007d30 con 0x7f0a94004800 2024-03-06T07:31:16.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.757+0000 7f0aa9ffb640 1 -- 172.21.15.83:0/85484064 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aa0030070 con 0x7f0a94004800 2024-03-06T07:31:16.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.757+0000 7f0aa9ffb640 1 -- 172.21.15.83:0/85484064 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa0005d30 con 0x7f0a94004800 2024-03-06T07:31:16.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.758+0000 7f0ab1833640 1 -- 172.21.15.83:0/85484064 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a94004800 msgr2=0x7f0a94004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.758+0000 7f0ab1833640 1 --2- 172.21.15.83:0/85484064 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a94004800 0x7f0a94004c80 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f0aac06e080 tx=0x7f0a94005830 comp rx=0 tx=0).stop 2024-03-06T07:31:16.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.758+0000 7f0aab7fe640 1 -- 172.21.15.83:0/85484064 reap_dead start 2024-03-06T07:31:16.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.758+0000 7f0ab1833640 1 -- 172.21.15.83:0/85484064 shutdown_connections 2024-03-06T07:31:16.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.758+0000 7f0ab1833640 1 -- 172.21.15.83:0/85484064 >> 172.21.15.83:0/85484064 conn(0x7f0aac0d3c40 msgr2=0x7f0aac0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:16.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.758+0000 7f0ab1833640 1 -- 172.21.15.83:0/85484064 shutdown_connections 2024-03-06T07:31:16.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.758+0000 7f0ab1833640 1 -- 172.21.15.83:0/85484064 wait complete. 2024-03-06T07:31:16.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.760+0000 7f0ab1833640 1 Processor -- start 2024-03-06T07:31:16.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.760+0000 7f0ab1833640 1 -- start start 2024-03-06T07:31:16.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.761+0000 7f0ab1833640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0a94004800 0x7f0aac156ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.761+0000 7f0ab1833640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0aac0a06c0 0x7f0aac0a0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.761+0000 7f0ab1833640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0aac09e010 0x7f0aac09e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.761+0000 7f0ab1833640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aac0b4dd0 con 0x7f0aac0a06c0 2024-03-06T07:31:16.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.761+0000 7f0ab1833640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aac0ce9e0 con 0x7f0aac09e010 2024-03-06T07:31:16.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.761+0000 7f0ab1833640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aac0b4f50 con 0x7f0a94004800 2024-03-06T07:31:16.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.761+0000 7f0aaa7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0aac0a06c0 0x7f0aac0a0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.761+0000 7f0aaa7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0aac0a06c0 0x7f0aac0a0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56218/0 (socket says 172.21.15.83:56218) 2024-03-06T07:31:16.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.761+0000 7f0aaa7fc640 1 -- 172.21.15.83:0/306631310 learned_addr learned my addr 172.21.15.83:0/306631310 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:16.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.761+0000 7f0aab7fe640 1 --2- 172.21.15.83:0/306631310 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0aac09e010 0x7f0aac09e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.761+0000 7f0aaaffd640 1 --2- 172.21.15.83:0/306631310 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0a94004800 0x7f0aac156ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.762+0000 7f0aaa7fc640 1 -- 172.21.15.83:0/306631310 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0a94004800 msgr2=0x7f0aac156ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.762+0000 7f0aaa7fc640 1 --2- 172.21.15.83:0/306631310 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0a94004800 0x7f0aac156ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.762+0000 7f0aaa7fc640 1 -- 172.21.15.83:0/306631310 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0aac09e010 msgr2=0x7f0aac09e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.762+0000 7f0aaa7fc640 1 --2- 172.21.15.83:0/306631310 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0aac09e010 0x7f0aac09e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.762+0000 7f0aaa7fc640 1 -- 172.21.15.83:0/306631310 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aac09e9d0 con 0x7f0aac0a06c0 2024-03-06T07:31:16.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.762+0000 7f0aaa7fc640 1 --2- 172.21.15.83:0/306631310 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0aac0a06c0 0x7f0aac0a0b20 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f0aa000c2d0 tx=0x7f0aa0001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.762+0000 7f0a93fff640 1 -- 172.21.15.83:0/306631310 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa0045070 con 0x7f0aac0a06c0 2024-03-06T07:31:16.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.762+0000 7f0a93fff640 1 -- 172.21.15.83:0/306631310 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aa0000ed0 con 0x7f0aac0a06c0 2024-03-06T07:31:16.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.762+0000 7f0ab1833640 1 -- 172.21.15.83:0/306631310 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0aac09ec90 con 0x7f0aac0a06c0 2024-03-06T07:31:16.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.762+0000 7f0a93fff640 1 -- 172.21.15.83:0/306631310 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa0004e30 con 0x7f0aac0a06c0 2024-03-06T07:31:16.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.762+0000 7f0ab1833640 1 -- 172.21.15.83:0/306631310 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0aac09f250 con 0x7f0aac0a06c0 2024-03-06T07:31:16.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.763+0000 7f0ab1833640 1 -- 172.21.15.83:0/306631310 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0aac04f330 con 0x7f0aac0a06c0 2024-03-06T07:31:16.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.764+0000 7f0a93fff640 1 -- 172.21.15.83:0/306631310 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0aa0007070 con 0x7f0aac0a06c0 2024-03-06T07:31:16.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.764+0000 7f0a93fff640 1 --2- 172.21.15.83:0/306631310 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0a9806b500 0x7f0a9806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.764+0000 7f0aaaffd640 1 --2- 172.21.15.83:0/306631310 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0a9806b500 0x7f0a9806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.764+0000 7f0a93fff640 1 -- 172.21.15.83:0/306631310 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0aa00ab370 con 0x7f0aac0a06c0 2024-03-06T07:31:16.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.765+0000 7f0aaaffd640 1 --2- 172.21.15.83:0/306631310 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0a9806b500 0x7f0a9806d9c0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c001e50 tx=0x7f0a9c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.769+0000 7f0a93fff640 1 -- 172.21.15.83:0/306631310 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0aa0079bb0 con 0x7f0aac0a06c0 2024-03-06T07:31:16.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.986+0000 7f0ab1833640 1 -- 172.21.15.83:0/306631310 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0aac0733b0 con 0x7f0aac0a06c0 2024-03-06T07:31:16.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.987+0000 7f0a93fff640 1 -- 172.21.15.83:0/306631310 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f0aa007dc50 con 0x7f0aac0a06c0 2024-03-06T07:31:16.989 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:31:16.989 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:31:16.989 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:31:16.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.989+0000 7f0ab1833640 1 -- 172.21.15.83:0/306631310 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0a9806b500 msgr2=0x7f0a9806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.989+0000 7f0ab1833640 1 --2- 172.21.15.83:0/306631310 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0a9806b500 0x7f0a9806d9c0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c001e50 tx=0x7f0a9c009040 comp rx=0 tx=0).stop 2024-03-06T07:31:16.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.989+0000 7f0ab1833640 1 -- 172.21.15.83:0/306631310 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0aac0a06c0 msgr2=0x7f0aac0a0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.989+0000 7f0ab1833640 1 --2- 172.21.15.83:0/306631310 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0aac0a06c0 0x7f0aac0a0b20 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f0aa000c2d0 tx=0x7f0aa0001a40 comp rx=0 tx=0).stop 2024-03-06T07:31:16.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.989+0000 7f0ab1833640 1 -- 172.21.15.83:0/306631310 shutdown_connections 2024-03-06T07:31:16.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.989+0000 7f0ab1833640 1 --2- 172.21.15.83:0/306631310 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0a9806b500 0x7f0a9806d9c0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.989+0000 7f0ab1833640 1 --2- 172.21.15.83:0/306631310 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0aac09e010 0x7f0aac09e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.989+0000 7f0ab1833640 1 --2- 172.21.15.83:0/306631310 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0aac0a06c0 0x7f0aac0a0b20 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.989+0000 7f0ab1833640 1 --2- 172.21.15.83:0/306631310 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0a94004800 0x7f0aac156ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.989+0000 7f0ab1833640 1 -- 172.21.15.83:0/306631310 >> 172.21.15.83:0/306631310 conn(0x7f0aac0d3c40 msgr2=0x7f0aac0b2340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:16.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.989+0000 7f0ab1833640 1 -- 172.21.15.83:0/306631310 shutdown_connections 2024-03-06T07:31:16.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:16.989+0000 7f0ab1833640 1 -- 172.21.15.83:0/306631310 wait complete. 2024-03-06T07:31:17.079 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:17.203 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:31:17.203 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:31:18.204 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:18.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.307+0000 7fb528288640 1 Processor -- start 2024-03-06T07:31:18.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.307+0000 7fb528288640 1 -- start start 2024-03-06T07:31:18.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.308+0000 7fb528288640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb52015aec0 0x7fb5200cd170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.308+0000 7fb528288640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fb5200ce890 0x7fb5200cd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.308+0000 7fb528288640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5200cee00 con 0x7fb5200cdbf0 2024-03-06T07:31:18.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.308+0000 7fb528288640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5200723e0 con 0x7fb52015aec0 2024-03-06T07:31:18.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.308+0000 7fb528288640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5200af860 con 0x7fb5200ce890 2024-03-06T07:31:18.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.308+0000 7fb526024640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb52015aec0 0x7fb5200cd170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.308+0000 7fb526024640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb52015aec0 0x7fb5200cd170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56226/0 (socket says 172.21.15.83:56226) 2024-03-06T07:31:18.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.308+0000 7fb526024640 1 -- 172.21.15.83:0/1804533841 learned_addr learned my addr 172.21.15.83:0/1804533841 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:18.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.308+0000 7fb526825640 1 --1- 172.21.15.83:0/1804533841 >> v1:172.21.15.83:6789/0 conn(0x7fb5200cdbf0 0x7fb52006da70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48266/0 (socket says 172.21.15.83:48266) 2024-03-06T07:31:18.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.308+0000 7fb525823640 1 --2- 172.21.15.83:0/1804533841 >> v2:172.21.15.183:3300/0 conn(0x7fb5200ce890 0x7fb5200cd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.308+0000 7fb526024640 1 -- 172.21.15.83:0/1804533841 >> v2:172.21.15.183:3300/0 conn(0x7fb5200ce890 msgr2=0x7fb5200cd6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:18.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.308+0000 7fb526024640 1 --2- 172.21.15.83:0/1804533841 >> v2:172.21.15.183:3300/0 conn(0x7fb5200ce890 0x7fb5200cd6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.308+0000 7fb526024640 1 -- 172.21.15.83:0/1804533841 >> v1:172.21.15.83:6789/0 conn(0x7fb5200cdbf0 legacy=0x7fb52006da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.308+0000 7fb525022640 1 -- 172.21.15.83:0/1804533841 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3578837671 0 0) 0x7fb518001490 con 0x7fb5200cdbf0 2024-03-06T07:31:18.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.308+0000 7fb526024640 1 -- 172.21.15.83:0/1804533841 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb520072680 con 0x7fb52015aec0 2024-03-06T07:31:18.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.309+0000 7fb526024640 1 --2- 172.21.15.83:0/1804533841 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb52015aec0 0x7fb5200cd170 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7fb520067ce0 tx=0x7fb51c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ace207ee2367d352 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:18.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.309+0000 7fb525022640 1 -- 172.21.15.83:0/1804533841 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb51c034070 con 0x7fb52015aec0 2024-03-06T07:31:18.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.309+0000 7fb525022640 1 -- 172.21.15.83:0/1804533841 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb51c030860 con 0x7fb52015aec0 2024-03-06T07:31:18.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.309+0000 7fb525022640 1 -- 172.21.15.83:0/1804533841 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb51c030c30 con 0x7fb52015aec0 2024-03-06T07:31:18.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.310+0000 7fb528288640 1 -- 172.21.15.83:0/1804533841 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb52015aec0 msgr2=0x7fb5200cd170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.310+0000 7fb528288640 1 --2- 172.21.15.83:0/1804533841 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb52015aec0 0x7fb5200cd170 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7fb520067ce0 tx=0x7fb51c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.310+0000 7fb528288640 1 -- 172.21.15.83:0/1804533841 shutdown_connections 2024-03-06T07:31:18.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.310+0000 7fb528288640 1 --2- 172.21.15.83:0/1804533841 >> v2:172.21.15.183:3300/0 conn(0x7fb5200ce890 0x7fb5200cd6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.310+0000 7fb528288640 1 --2- 172.21.15.83:0/1804533841 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb52015aec0 0x7fb5200cd170 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.310+0000 7fb528288640 1 -- 172.21.15.83:0/1804533841 >> 172.21.15.83:0/1804533841 conn(0x7fb5200d0990 msgr2=0x7fb5200d0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:18.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.310+0000 7fb528288640 1 -- 172.21.15.83:0/1804533841 shutdown_connections 2024-03-06T07:31:18.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.310+0000 7fb528288640 1 -- 172.21.15.83:0/1804533841 wait complete. 2024-03-06T07:31:18.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.311+0000 7fb528288640 1 Processor -- start 2024-03-06T07:31:18.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.311+0000 7fb528288640 1 -- start start 2024-03-06T07:31:18.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.311+0000 7fb528288640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb5200cdbf0 0x7fb5200b4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.311+0000 7fb528288640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb5200ce890 0x7fb52014c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.312+0000 7fb528288640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb52014cb50 0x7fb52014b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.312+0000 7fb528288640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb52006e180 con 0x7fb5200ce890 2024-03-06T07:31:18.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.312+0000 7fb528288640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5200afff0 con 0x7fb5200cdbf0 2024-03-06T07:31:18.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.312+0000 7fb528288640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb52006e300 con 0x7fb52014cb50 2024-03-06T07:31:18.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.312+0000 7fb525823640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb5200ce890 0x7fb52014c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.312+0000 7fb525823640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb5200ce890 0x7fb52014c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56238/0 (socket says 172.21.15.83:56238) 2024-03-06T07:31:18.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.312+0000 7fb525823640 1 -- 172.21.15.83:0/4084102232 learned_addr learned my addr 172.21.15.83:0/4084102232 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:18.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.312+0000 7fb526825640 1 --2- 172.21.15.83:0/4084102232 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb52014cb50 0x7fb52014b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.312+0000 7fb526024640 1 --2- 172.21.15.83:0/4084102232 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb5200cdbf0 0x7fb5200b4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.312+0000 7fb525823640 1 -- 172.21.15.83:0/4084102232 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb52014cb50 msgr2=0x7fb52014b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.312+0000 7fb525823640 1 --2- 172.21.15.83:0/4084102232 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb52014cb50 0x7fb52014b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.312+0000 7fb525823640 1 -- 172.21.15.83:0/4084102232 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb5200cdbf0 msgr2=0x7fb5200b4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.312+0000 7fb525823640 1 --2- 172.21.15.83:0/4084102232 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb5200cdbf0 0x7fb5200b4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.312+0000 7fb525823640 1 -- 172.21.15.83:0/4084102232 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb52015c3f0 con 0x7fb5200ce890 2024-03-06T07:31:18.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.312+0000 7fb526024640 1 --2- 172.21.15.83:0/4084102232 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb5200cdbf0 0x7fb5200b4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:18.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.312+0000 7fb525823640 1 --2- 172.21.15.83:0/4084102232 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb5200ce890 0x7fb52014c610 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7fb51000c7f0 tx=0x7fb51000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:18.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.312+0000 7fb516ffd640 1 -- 172.21.15.83:0/4084102232 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb510015070 con 0x7fb5200ce890 2024-03-06T07:31:18.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.312+0000 7fb516ffd640 1 -- 172.21.15.83:0/4084102232 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb510004240 con 0x7fb5200ce890 2024-03-06T07:31:18.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.312+0000 7fb528288640 1 -- 172.21.15.83:0/4084102232 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb52015b350 con 0x7fb5200ce890 2024-03-06T07:31:18.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.312+0000 7fb516ffd640 1 -- 172.21.15.83:0/4084102232 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb510007620 con 0x7fb5200ce890 2024-03-06T07:31:18.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.312+0000 7fb528288640 1 -- 172.21.15.83:0/4084102232 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb52015b910 con 0x7fb5200ce890 2024-03-06T07:31:18.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.313+0000 7fb528288640 1 -- 172.21.15.83:0/4084102232 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb52004f330 con 0x7fb5200ce890 2024-03-06T07:31:18.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.314+0000 7fb516ffd640 1 -- 172.21.15.83:0/4084102232 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb5100079d0 con 0x7fb5200ce890 2024-03-06T07:31:18.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.314+0000 7fb516ffd640 1 --2- 172.21.15.83:0/4084102232 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb50406b4d0 0x7fb50406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.314+0000 7fb526024640 1 --2- 172.21.15.83:0/4084102232 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb50406b4d0 0x7fb50406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.314+0000 7fb516ffd640 1 -- 172.21.15.83:0/4084102232 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb51008a6c0 con 0x7fb5200ce890 2024-03-06T07:31:18.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.315+0000 7fb526024640 1 --2- 172.21.15.83:0/4084102232 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb50406b4d0 0x7fb50406d990 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fb51c036000 tx=0x7fb51c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:18.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.319+0000 7fb516ffd640 1 -- 172.21.15.83:0/4084102232 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb510059500 con 0x7fb5200ce890 2024-03-06T07:31:18.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.534+0000 7fb528288640 1 -- 172.21.15.83:0/4084102232 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5200ce1f0 con 0x7fb5200ce890 2024-03-06T07:31:18.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.535+0000 7fb516ffd640 1 -- 172.21.15.83:0/4084102232 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7fb510058e00 con 0x7fb5200ce890 2024-03-06T07:31:18.537 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:31:18.537 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:31:18.537 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:31:18.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.536+0000 7fb528288640 1 -- 172.21.15.83:0/4084102232 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb50406b4d0 msgr2=0x7fb50406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.536+0000 7fb528288640 1 --2- 172.21.15.83:0/4084102232 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb50406b4d0 0x7fb50406d990 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fb51c036000 tx=0x7fb51c008030 comp rx=0 tx=0).stop 2024-03-06T07:31:18.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.536+0000 7fb528288640 1 -- 172.21.15.83:0/4084102232 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb5200ce890 msgr2=0x7fb52014c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.536+0000 7fb528288640 1 --2- 172.21.15.83:0/4084102232 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb5200ce890 0x7fb52014c610 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7fb51000c7f0 tx=0x7fb51000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.537+0000 7fb528288640 1 -- 172.21.15.83:0/4084102232 shutdown_connections 2024-03-06T07:31:18.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.537+0000 7fb528288640 1 --2- 172.21.15.83:0/4084102232 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb50406b4d0 0x7fb50406d990 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.537+0000 7fb528288640 1 --2- 172.21.15.83:0/4084102232 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb52014cb50 0x7fb52014b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.537+0000 7fb528288640 1 --2- 172.21.15.83:0/4084102232 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb5200ce890 0x7fb52014c610 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.537+0000 7fb528288640 1 --2- 172.21.15.83:0/4084102232 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb5200cdbf0 0x7fb5200b4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.537+0000 7fb528288640 1 -- 172.21.15.83:0/4084102232 >> 172.21.15.83:0/4084102232 conn(0x7fb5200d0990 msgr2=0x7fb52006e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:18.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.537+0000 7fb528288640 1 -- 172.21.15.83:0/4084102232 shutdown_connections 2024-03-06T07:31:18.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:18.537+0000 7fb528288640 1 -- 172.21.15.83:0/4084102232 wait complete. 2024-03-06T07:31:18.750 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:31:18.750 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:31:19.081 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:19.751 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:19.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.855+0000 7f08a8313640 1 Processor -- start 2024-03-06T07:31:19.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.855+0000 7f08a8313640 1 -- start start 2024-03-06T07:31:19.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.856+0000 7f08a8313640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f08a013dc10 0x7f08a00b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.856+0000 7f08a8313640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f08a015bf30 0x7f08a00b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.856+0000 7f08a8313640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08a015f580 con 0x7f08a015e850 2024-03-06T07:31:19.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.856+0000 7f08a8313640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08a00cf3f0 con 0x7f08a013dc10 2024-03-06T07:31:19.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.856+0000 7f08a8313640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08a015b210 con 0x7f08a015bf30 2024-03-06T07:31:19.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.856+0000 7f08a68b0640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f08a015e850 0x7f08a015eca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48274/0 (socket says 172.21.15.83:48274) 2024-03-06T07:31:19.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.856+0000 7f08a68b0640 1 -- 172.21.15.83:0/921808770 learned_addr learned my addr 172.21.15.83:0/921808770 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:31:19.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.856+0000 7f08a58ae640 1 --2- 172.21.15.83:0/921808770 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f08a015bf30 0x7f08a00b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.856+0000 7f08a60af640 1 --2- 172.21.15.83:0/921808770 >> v2:172.21.15.183:3300/0 conn(0x7f08a013dc10 0x7f08a00b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.856+0000 7f08a50ad640 1 -- 172.21.15.83:0/921808770 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3359764830 0 0) 0x7f08a015f580 con 0x7f08a015e850 2024-03-06T07:31:19.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.856+0000 7f08a50ad640 1 -- 172.21.15.83:0/921808770 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f088c003520 con 0x7f08a015e850 2024-03-06T07:31:19.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.857+0000 7f08a58ae640 1 -- 172.21.15.83:0/921808770 >> v2:172.21.15.183:3300/0 conn(0x7f08a013dc10 msgr2=0x7f08a00b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:19.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.857+0000 7f08a58ae640 1 --2- 172.21.15.83:0/921808770 >> v2:172.21.15.183:3300/0 conn(0x7f08a013dc10 0x7f08a00b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.857+0000 7f08a58ae640 1 -- 172.21.15.83:0/921808770 >> v1:172.21.15.83:6789/0 conn(0x7f08a015e850 legacy=0x7f08a015eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.857+0000 7f08a50ad640 1 -- 172.21.15.83:0/921808770 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4239063558 0 0) 0x7f088c003520 con 0x7f08a015e850 2024-03-06T07:31:19.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.857+0000 7f08a58ae640 1 -- 172.21.15.83:0/921808770 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08a015b450 con 0x7f08a015bf30 2024-03-06T07:31:19.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.857+0000 7f08a58ae640 1 --2- 172.21.15.83:0/921808770 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f08a015bf30 0x7f08a00b2e90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f089c0097a0 tx=0x7f089c02b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f62dbc9890be5c3c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:19.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.858+0000 7f08a50ad640 1 -- 172.21.15.83:0/921808770 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f089c037070 con 0x7f08a015bf30 2024-03-06T07:31:19.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.858+0000 7f08a50ad640 1 -- 172.21.15.83:0/921808770 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f089c030510 con 0x7f08a015bf30 2024-03-06T07:31:19.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.858+0000 7f08a50ad640 1 -- 172.21.15.83:0/921808770 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08980019f0 con 0x7f08a015bf30 2024-03-06T07:31:19.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.858+0000 7f08a8313640 1 -- 172.21.15.83:0/921808770 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f08a015bf30 msgr2=0x7f08a00b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.858+0000 7f08a8313640 1 --2- 172.21.15.83:0/921808770 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f08a015bf30 0x7f08a00b2e90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f089c0097a0 tx=0x7f089c02b710 comp rx=0 tx=0).stop 2024-03-06T07:31:19.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.858+0000 7f08a8313640 1 -- 172.21.15.83:0/921808770 shutdown_connections 2024-03-06T07:31:19.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.858+0000 7f08a8313640 1 --2- 172.21.15.83:0/921808770 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f08a015bf30 0x7f08a00b2e90 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.858+0000 7f08a8313640 1 --2- 172.21.15.83:0/921808770 >> v2:172.21.15.183:3300/0 conn(0x7f08a013dc10 0x7f08a00b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.858+0000 7f08a8313640 1 -- 172.21.15.83:0/921808770 >> 172.21.15.83:0/921808770 conn(0x7f08a00cf8a0 msgr2=0x7f08a00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:19.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.858+0000 7f08a8313640 1 -- 172.21.15.83:0/921808770 shutdown_connections 2024-03-06T07:31:19.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.858+0000 7f08a8313640 1 -- 172.21.15.83:0/921808770 wait complete. 2024-03-06T07:31:19.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.860+0000 7f08a8313640 1 Processor -- start 2024-03-06T07:31:19.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.860+0000 7f08a8313640 1 -- start start 2024-03-06T07:31:19.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.861+0000 7f08a8313640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f08a013dc10 0x7f08a01f1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.861+0000 7f08a8313640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08a015bf30 0x7f08a0151b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.861+0000 7f08a8313640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f08a015e850 0x7f08a0154e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.861+0000 7f08a8313640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a00708d0 con 0x7f08a015bf30 2024-03-06T07:31:19.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.861+0000 7f08a8313640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a013dff0 con 0x7f08a013dc10 2024-03-06T07:31:19.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.861+0000 7f08a8313640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08a0070a50 con 0x7f08a015e850 2024-03-06T07:31:19.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.861+0000 7f08a58ae640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08a015bf30 0x7f08a0151b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.861+0000 7f08a58ae640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08a015bf30 0x7f08a0151b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56248/0 (socket says 172.21.15.83:56248) 2024-03-06T07:31:19.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.861+0000 7f08a58ae640 1 -- 172.21.15.83:0/1938000823 learned_addr learned my addr 172.21.15.83:0/1938000823 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:19.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.862+0000 7f08a68b0640 1 --2- 172.21.15.83:0/1938000823 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f08a015e850 0x7f08a0154e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.862+0000 7f08a60af640 1 --2- 172.21.15.83:0/1938000823 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f08a013dc10 0x7f08a01f1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.862+0000 7f08a58ae640 1 -- 172.21.15.83:0/1938000823 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f08a015e850 msgr2=0x7f08a0154e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.862+0000 7f08a58ae640 1 --2- 172.21.15.83:0/1938000823 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f08a015e850 0x7f08a0154e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.862+0000 7f08a58ae640 1 -- 172.21.15.83:0/1938000823 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f08a013dc10 msgr2=0x7f08a01f1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.862+0000 7f08a58ae640 1 --2- 172.21.15.83:0/1938000823 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f08a013dc10 0x7f08a01f1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.862+0000 7f08a58ae640 1 -- 172.21.15.83:0/1938000823 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08a01f20b0 con 0x7f08a015bf30 2024-03-06T07:31:19.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.862+0000 7f08a60af640 1 --2- 172.21.15.83:0/1938000823 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f08a013dc10 0x7f08a01f1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:19.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.862+0000 7f08a58ae640 1 --2- 172.21.15.83:0/1938000823 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08a015bf30 0x7f08a0151b00 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f089c000c00 tx=0x7f089c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:19.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.862+0000 7f0896ffd640 1 -- 172.21.15.83:0/1938000823 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f089c041020 con 0x7f08a015bf30 2024-03-06T07:31:19.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.862+0000 7f0896ffd640 1 -- 172.21.15.83:0/1938000823 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f089c03c870 con 0x7f08a015bf30 2024-03-06T07:31:19.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.862+0000 7f08a8313640 1 -- 172.21.15.83:0/1938000823 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08a01416d0 con 0x7f08a015bf30 2024-03-06T07:31:19.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.862+0000 7f0896ffd640 1 -- 172.21.15.83:0/1938000823 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f089c037040 con 0x7f08a015bf30 2024-03-06T07:31:19.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.862+0000 7f08a8313640 1 -- 172.21.15.83:0/1938000823 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08a0141c90 con 0x7f08a015bf30 2024-03-06T07:31:19.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.863+0000 7f08a8313640 1 -- 172.21.15.83:0/1938000823 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08a004f330 con 0x7f08a015bf30 2024-03-06T07:31:19.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.864+0000 7f0896ffd640 1 -- 172.21.15.83:0/1938000823 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f089c044020 con 0x7f08a015bf30 2024-03-06T07:31:19.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.864+0000 7f0896ffd640 1 --2- 172.21.15.83:0/1938000823 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f088406b4d0 0x7f088406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.864+0000 7f0896ffd640 1 -- 172.21.15.83:0/1938000823 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f089c0abc40 con 0x7f08a015bf30 2024-03-06T07:31:19.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.864+0000 7f08a60af640 1 --2- 172.21.15.83:0/1938000823 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f088406b4d0 0x7f088406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.865+0000 7f08a60af640 1 --2- 172.21.15.83:0/1938000823 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f088406b4d0 0x7f088406d990 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f0890009ac0 tx=0x7f0890008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:19.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:19.873+0000 7f0896ffd640 1 -- 172.21.15.83:0/1938000823 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f089c04b030 con 0x7f08a015bf30 2024-03-06T07:31:20.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:20.085+0000 7f08a8313640 1 -- 172.21.15.83:0/1938000823 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08a006eb00 con 0x7f08a015bf30 2024-03-06T07:31:20.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:20.086+0000 7f0896ffd640 1 -- 172.21.15.83:0/1938000823 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f089c07aa80 con 0x7f08a015bf30 2024-03-06T07:31:20.088 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:31:20.088 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:31:20.088 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:31:20.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:20.088+0000 7f08a8313640 1 -- 172.21.15.83:0/1938000823 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f088406b4d0 msgr2=0x7f088406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:20.088+0000 7f08a8313640 1 --2- 172.21.15.83:0/1938000823 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f088406b4d0 0x7f088406d990 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f0890009ac0 tx=0x7f0890008040 comp rx=0 tx=0).stop 2024-03-06T07:31:20.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:20.088+0000 7f08a8313640 1 -- 172.21.15.83:0/1938000823 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08a015bf30 msgr2=0x7f08a0151b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:20.088+0000 7f08a8313640 1 --2- 172.21.15.83:0/1938000823 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08a015bf30 0x7f08a0151b00 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f089c000c00 tx=0x7f089c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:20.088+0000 7f08a8313640 1 -- 172.21.15.83:0/1938000823 shutdown_connections 2024-03-06T07:31:20.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:20.088+0000 7f08a8313640 1 --2- 172.21.15.83:0/1938000823 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f088406b4d0 0x7f088406d990 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:20.088+0000 7f08a8313640 1 --2- 172.21.15.83:0/1938000823 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f08a015e850 0x7f08a0154e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:20.088+0000 7f08a8313640 1 --2- 172.21.15.83:0/1938000823 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08a015bf30 0x7f08a0151b00 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:20.088+0000 7f08a8313640 1 --2- 172.21.15.83:0/1938000823 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f08a013dc10 0x7f08a01f1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:20.088+0000 7f08a8313640 1 -- 172.21.15.83:0/1938000823 >> 172.21.15.83:0/1938000823 conn(0x7f08a00cf8a0 msgr2=0x7f08a00b0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:20.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:20.088+0000 7f08a8313640 1 -- 172.21.15.83:0/1938000823 shutdown_connections 2024-03-06T07:31:20.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:20.088+0000 7f08a8313640 1 -- 172.21.15.83:0/1938000823 wait complete. 2024-03-06T07:31:20.302 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:31:20.302 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:31:21.083 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:21.303 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:21.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.428+0000 7fc50d8aa640 1 Processor -- start 2024-03-06T07:31:21.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.428+0000 7fc50d8aa640 1 -- start start 2024-03-06T07:31:21.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.429+0000 7fc50d8aa640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fc5080b4170 0x7fc5080b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.429+0000 7fc50d8aa640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc5080d0a80 0x7fc5080cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.429+0000 7fc50d8aa640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5080d25d0 con 0x7fc5080d4760 2024-03-06T07:31:21.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.429+0000 7fc50d8aa640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5080d4bf0 con 0x7fc5080b4170 2024-03-06T07:31:21.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.429+0000 7fc50d8aa640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5080ce9e0 con 0x7fc5080d0a80 2024-03-06T07:31:21.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.430+0000 7fc5077fe640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc5080d0a80 0x7fc5080cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.430+0000 7fc5077fe640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc5080d0a80 0x7fc5080cee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:45022/0 (socket says 172.21.15.83:45022) 2024-03-06T07:31:21.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.430+0000 7fc5077fe640 1 -- 172.21.15.83:0/2991453201 learned_addr learned my addr 172.21.15.83:0/2991453201 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:21.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.430+0000 7fc506ffd640 1 --2- 172.21.15.83:0/2991453201 >> v2:172.21.15.183:3300/0 conn(0x7fc5080b4170 0x7fc5080b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.430+0000 7fc5077fe640 1 -- 172.21.15.83:0/2991453201 >> v2:172.21.15.183:3300/0 conn(0x7fc5080b4170 msgr2=0x7fc5080b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:21.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.430+0000 7fc5077fe640 1 --2- 172.21.15.83:0/2991453201 >> v2:172.21.15.183:3300/0 conn(0x7fc5080b4170 0x7fc5080b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.430+0000 7fc5077fe640 1 -- 172.21.15.83:0/2991453201 >> v1:172.21.15.183:6789/0 conn(0x7fc5080d4760 legacy=0x7fc5080d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.430+0000 7fc5077fe640 1 -- 172.21.15.83:0/2991453201 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5080d2870 con 0x7fc5080d0a80 2024-03-06T07:31:21.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.430+0000 7fc506ffd640 1 --2- 172.21.15.83:0/2991453201 >> v2:172.21.15.183:3300/0 conn(0x7fc5080b4170 0x7fc5080b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:21.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.431+0000 7fc5077fe640 1 --2- 172.21.15.83:0/2991453201 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc5080d0a80 0x7fc5080cee70 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8009870 tx=0x7fc4f802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d03a81de217aabd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:21.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.431+0000 7fc505ffb640 1 -- 172.21.15.83:0/2991453201 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4f8034070 con 0x7fc5080d0a80 2024-03-06T07:31:21.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.431+0000 7fc505ffb640 1 -- 172.21.15.83:0/2991453201 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4f8030840 con 0x7fc5080d0a80 2024-03-06T07:31:21.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.431+0000 7fc505ffb640 1 -- 172.21.15.83:0/2991453201 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4f8030c10 con 0x7fc5080d0a80 2024-03-06T07:31:21.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.431+0000 7fc50d8aa640 1 -- 172.21.15.83:0/2991453201 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc5080d0a80 msgr2=0x7fc5080cee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.431+0000 7fc50d8aa640 1 --2- 172.21.15.83:0/2991453201 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc5080d0a80 0x7fc5080cee70 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8009870 tx=0x7fc4f802b980 comp rx=0 tx=0).stop 2024-03-06T07:31:21.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.431+0000 7fc50d8aa640 1 -- 172.21.15.83:0/2991453201 shutdown_connections 2024-03-06T07:31:21.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.431+0000 7fc50d8aa640 1 --2- 172.21.15.83:0/2991453201 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc5080d0a80 0x7fc5080cee70 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.432+0000 7fc50d8aa640 1 --2- 172.21.15.83:0/2991453201 >> v2:172.21.15.183:3300/0 conn(0x7fc5080b4170 0x7fc5080b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.432+0000 7fc50d8aa640 1 -- 172.21.15.83:0/2991453201 >> 172.21.15.83:0/2991453201 conn(0x7fc5080d3c40 msgr2=0x7fc5080d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:21.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.432+0000 7fc50d8aa640 1 -- 172.21.15.83:0/2991453201 shutdown_connections 2024-03-06T07:31:21.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.432+0000 7fc50d8aa640 1 -- 172.21.15.83:0/2991453201 wait complete. 2024-03-06T07:31:21.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.433+0000 7fc50d8aa640 1 Processor -- start 2024-03-06T07:31:21.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.433+0000 7fc50d8aa640 1 -- start start 2024-03-06T07:31:21.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.433+0000 7fc50d8aa640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5080b4170 0x7fc508156cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.433+0000 7fc50d8aa640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc5080d0a80 0x7fc50815eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.434+0000 7fc50d8aa640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc50815f050 0x7fc5080b0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.434+0000 7fc50d8aa640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5080b4dd0 con 0x7fc50815f050 2024-03-06T07:31:21.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.434+0000 7fc50d8aa640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5080b4550 con 0x7fc5080b4170 2024-03-06T07:31:21.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.434+0000 7fc50d8aa640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5080b4f50 con 0x7fc5080d0a80 2024-03-06T07:31:21.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.434+0000 7fc5077fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc50815f050 0x7fc5080b0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.434+0000 7fc5077fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc50815f050 0x7fc5080b0b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56260/0 (socket says 172.21.15.83:56260) 2024-03-06T07:31:21.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.434+0000 7fc5077fe640 1 -- 172.21.15.83:0/3614823458 learned_addr learned my addr 172.21.15.83:0/3614823458 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:21.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.434+0000 7fc506ffd640 1 --2- 172.21.15.83:0/3614823458 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5080b4170 0x7fc508156cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.434+0000 7fc5067fc640 1 --2- 172.21.15.83:0/3614823458 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc5080d0a80 0x7fc50815eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.434+0000 7fc5077fe640 1 -- 172.21.15.83:0/3614823458 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc5080d0a80 msgr2=0x7fc50815eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.434+0000 7fc5077fe640 1 --2- 172.21.15.83:0/3614823458 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc5080d0a80 0x7fc50815eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.434+0000 7fc5077fe640 1 -- 172.21.15.83:0/3614823458 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5080b4170 msgr2=0x7fc508156cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.434+0000 7fc5077fe640 1 --2- 172.21.15.83:0/3614823458 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5080b4170 0x7fc508156cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.434+0000 7fc5077fe640 1 -- 172.21.15.83:0/3614823458 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5080b2eb0 con 0x7fc50815f050 2024-03-06T07:31:21.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.434+0000 7fc5077fe640 1 --2- 172.21.15.83:0/3614823458 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc50815f050 0x7fc5080b0b30 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8009870 tx=0x7fc4f8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:21.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.434+0000 7fc4effff640 1 -- 172.21.15.83:0/3614823458 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4f8048070 con 0x7fc50815f050 2024-03-06T07:31:21.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.434+0000 7fc4effff640 1 -- 172.21.15.83:0/3614823458 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4f8030640 con 0x7fc50815f050 2024-03-06T07:31:21.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.434+0000 7fc50d8aa640 1 -- 172.21.15.83:0/3614823458 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5080b3190 con 0x7fc50815f050 2024-03-06T07:31:21.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.434+0000 7fc4effff640 1 -- 172.21.15.83:0/3614823458 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4f8034040 con 0x7fc50815f050 2024-03-06T07:31:21.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.434+0000 7fc50d8aa640 1 -- 172.21.15.83:0/3614823458 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc508157200 con 0x7fc50815f050 2024-03-06T07:31:21.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.435+0000 7fc50d8aa640 1 -- 172.21.15.83:0/3614823458 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc50804f330 con 0x7fc50815f050 2024-03-06T07:31:21.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.436+0000 7fc4effff640 1 -- 172.21.15.83:0/3614823458 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc4f804b070 con 0x7fc50815f050 2024-03-06T07:31:21.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.436+0000 7fc4effff640 1 --2- 172.21.15.83:0/3614823458 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc4f406b4d0 0x7fc4f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.437+0000 7fc4effff640 1 -- 172.21.15.83:0/3614823458 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc4f80abf90 con 0x7fc50815f050 2024-03-06T07:31:21.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.437+0000 7fc506ffd640 1 --2- 172.21.15.83:0/3614823458 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc4f406b4d0 0x7fc4f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.437+0000 7fc506ffd640 1 --2- 172.21.15.83:0/3614823458 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc4f406b4d0 0x7fc4f406d990 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fc5080c8280 tx=0x7fc500009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:21.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.441+0000 7fc4effff640 1 -- 172.21.15.83:0/3614823458 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc4f807a930 con 0x7fc50815f050 2024-03-06T07:31:21.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.656+0000 7fc50d8aa640 1 -- 172.21.15.83:0/3614823458 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5080b3bf0 con 0x7fc50815f050 2024-03-06T07:31:21.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.656+0000 7fc4effff640 1 -- 172.21.15.83:0/3614823458 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7fc4f807e9d0 con 0x7fc50815f050 2024-03-06T07:31:21.658 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:31:21.659 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:31:21.659 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:31:21.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.658+0000 7fc50d8aa640 1 -- 172.21.15.83:0/3614823458 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc4f406b4d0 msgr2=0x7fc4f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.658+0000 7fc50d8aa640 1 --2- 172.21.15.83:0/3614823458 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc4f406b4d0 0x7fc4f406d990 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fc5080c8280 tx=0x7fc500009190 comp rx=0 tx=0).stop 2024-03-06T07:31:21.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.658+0000 7fc50d8aa640 1 -- 172.21.15.83:0/3614823458 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc50815f050 msgr2=0x7fc5080b0b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.658+0000 7fc50d8aa640 1 --2- 172.21.15.83:0/3614823458 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc50815f050 0x7fc5080b0b30 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8009870 tx=0x7fc4f8004730 comp rx=0 tx=0).stop 2024-03-06T07:31:21.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.658+0000 7fc50d8aa640 1 -- 172.21.15.83:0/3614823458 shutdown_connections 2024-03-06T07:31:21.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.658+0000 7fc50d8aa640 1 --2- 172.21.15.83:0/3614823458 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc4f406b4d0 0x7fc4f406d990 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.658+0000 7fc50d8aa640 1 --2- 172.21.15.83:0/3614823458 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc50815f050 0x7fc5080b0b30 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.658+0000 7fc50d8aa640 1 --2- 172.21.15.83:0/3614823458 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc5080d0a80 0x7fc50815eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.658+0000 7fc50d8aa640 1 --2- 172.21.15.83:0/3614823458 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5080b4170 0x7fc508156cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.658+0000 7fc50d8aa640 1 -- 172.21.15.83:0/3614823458 >> 172.21.15.83:0/3614823458 conn(0x7fc5080d3c40 msgr2=0x7fc50806b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:21.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.659+0000 7fc50d8aa640 1 -- 172.21.15.83:0/3614823458 shutdown_connections 2024-03-06T07:31:21.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:21.659+0000 7fc50d8aa640 1 -- 172.21.15.83:0/3614823458 wait complete. 2024-03-06T07:31:21.872 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:31:21.873 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:31:22.875 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:22.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.975+0000 7fa780fc7640 1 Processor -- start 2024-03-06T07:31:22.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.975+0000 7fa780fc7640 1 -- start start 2024-03-06T07:31:22.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.976+0000 7fa780fc7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa77c0d22b0 0x7fa77c0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.976+0000 7fa780fc7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa77c0b3430 con 0x7fa77c15dfe0 2024-03-06T07:31:22.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.976+0000 7fa780fc7640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa77c0b3710 con 0x7fa77c0b41f0 2024-03-06T07:31:22.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.976+0000 7fa780fc7640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa77c0d2c50 con 0x7fa77c0d22b0 2024-03-06T07:31:22.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.976+0000 7fa779d9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa77c0d22b0 0x7fa77c0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.976+0000 7fa77a59c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fa77c15dfe0 0x7fa77c0cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48286/0 (socket says 172.21.15.83:48286) 2024-03-06T07:31:22.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.976+0000 7fa77a59c640 1 -- 172.21.15.83:0/2894248182 learned_addr learned my addr 172.21.15.83:0/2894248182 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:31:22.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.976+0000 7fa779d9b640 1 -- 172.21.15.83:0/2894248182 >> v1:172.21.15.183:6789/0 conn(0x7fa77c0b41f0 legacy=0x7fa77c0b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:22.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.976+0000 7fa779d9b640 1 -- 172.21.15.83:0/2894248182 >> v1:172.21.15.83:6789/0 conn(0x7fa77c15dfe0 legacy=0x7fa77c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.976+0000 7fa779d9b640 1 -- 172.21.15.83:0/2894248182 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa77c0b39f0 con 0x7fa77c0d22b0 2024-03-06T07:31:22.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.977+0000 7fa779d9b640 1 --2- 172.21.15.83:0/2894248182 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa77c0d22b0 0x7fa77c0d2710 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7fa7700097d0 tx=0x7fa77002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=611fa445443676d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:22.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.977+0000 7fa77959a640 1 -- 172.21.15.83:0/2894248182 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa770035020 con 0x7fa77c0d22b0 2024-03-06T07:31:22.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.977+0000 7fa77959a640 1 -- 172.21.15.83:0/2894248182 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa770031830 con 0x7fa77c0d22b0 2024-03-06T07:31:22.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.977+0000 7fa77959a640 1 -- 172.21.15.83:0/2894248182 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa770031c00 con 0x7fa77c0d22b0 2024-03-06T07:31:22.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.978+0000 7fa780fc7640 1 -- 172.21.15.83:0/2894248182 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa77c0d22b0 msgr2=0x7fa77c0d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.978+0000 7fa780fc7640 1 --2- 172.21.15.83:0/2894248182 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa77c0d22b0 0x7fa77c0d2710 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7fa7700097d0 tx=0x7fa77002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.978+0000 7fa780fc7640 1 -- 172.21.15.83:0/2894248182 shutdown_connections 2024-03-06T07:31:22.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.978+0000 7fa780fc7640 1 --2- 172.21.15.83:0/2894248182 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa77c0d22b0 0x7fa77c0d2710 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.978+0000 7fa780fc7640 1 -- 172.21.15.83:0/2894248182 >> 172.21.15.83:0/2894248182 conn(0x7fa77c0d3cd0 msgr2=0x7fa77c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:22.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.978+0000 7fa780fc7640 1 -- 172.21.15.83:0/2894248182 shutdown_connections 2024-03-06T07:31:22.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.978+0000 7fa780fc7640 1 -- 172.21.15.83:0/2894248182 wait complete. 2024-03-06T07:31:22.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.979+0000 7fa780fc7640 1 Processor -- start 2024-03-06T07:31:22.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.979+0000 7fa780fc7640 1 -- start start 2024-03-06T07:31:22.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.980+0000 7fa780fc7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa77c0b41f0 0x7fa77c14d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.980+0000 7fa780fc7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa77c0d22b0 0x7fa77c0cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.980+0000 7fa780fc7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa77c15dfe0 0x7fa77c1605f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.980+0000 7fa780fc7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c0b3d60 con 0x7fa77c0b41f0 2024-03-06T07:31:22.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.980+0000 7fa780fc7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c0b31e0 con 0x7fa77c15dfe0 2024-03-06T07:31:22.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.980+0000 7fa780fc7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c0d1c70 con 0x7fa77c0d22b0 2024-03-06T07:31:22.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.980+0000 7fa77a59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa77c0b41f0 0x7fa77c14d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.980+0000 7fa77a59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa77c0b41f0 0x7fa77c14d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56280/0 (socket says 172.21.15.83:56280) 2024-03-06T07:31:22.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.980+0000 7fa77a59c640 1 -- 172.21.15.83:0/1633892802 learned_addr learned my addr 172.21.15.83:0/1633892802 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:22.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.980+0000 7fa77a59c640 1 -- 172.21.15.83:0/1633892802 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa77c0d22b0 msgr2=0x7fa77c0cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.980+0000 7fa77a59c640 1 --2- 172.21.15.83:0/1633892802 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa77c0d22b0 0x7fa77c0cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.980+0000 7fa77a59c640 1 -- 172.21.15.83:0/1633892802 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa77c15dfe0 msgr2=0x7fa77c1605f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.980+0000 7fa77a59c640 1 --2- 172.21.15.83:0/1633892802 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa77c15dfe0 0x7fa77c1605f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.980+0000 7fa77a59c640 1 -- 172.21.15.83:0/1633892802 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa77c0cb760 con 0x7fa77c0b41f0 2024-03-06T07:31:22.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.980+0000 7fa77a59c640 1 --2- 172.21.15.83:0/1633892802 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa77c0b41f0 0x7fa77c14d320 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7fa76c00c3f0 tx=0x7fa76c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:22.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.981+0000 7fa76affd640 1 -- 172.21.15.83:0/1633892802 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76c017070 con 0x7fa77c0b41f0 2024-03-06T07:31:22.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.981+0000 7fa76affd640 1 -- 172.21.15.83:0/1633892802 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa76c005ce0 con 0x7fa77c0b41f0 2024-03-06T07:31:22.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.981+0000 7fa780fc7640 1 -- 172.21.15.83:0/1633892802 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa77c0cba40 con 0x7fa77c0b41f0 2024-03-06T07:31:22.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.981+0000 7fa76affd640 1 -- 172.21.15.83:0/1633892802 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76c0072f0 con 0x7fa77c0b41f0 2024-03-06T07:31:22.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.981+0000 7fa780fc7640 1 -- 172.21.15.83:0/1633892802 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa77c0cc000 con 0x7fa77c0b41f0 2024-03-06T07:31:22.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.982+0000 7fa780fc7640 1 -- 172.21.15.83:0/1633892802 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa77c04f330 con 0x7fa77c0b41f0 2024-03-06T07:31:22.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.982+0000 7fa76affd640 1 -- 172.21.15.83:0/1633892802 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa76c01a070 con 0x7fa77c0b41f0 2024-03-06T07:31:22.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.982+0000 7fa76affd640 1 --2- 172.21.15.83:0/1633892802 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa75c06b4d0 0x7fa75c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.983+0000 7fa76affd640 1 -- 172.21.15.83:0/1633892802 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa76c089650 con 0x7fa77c0b41f0 2024-03-06T07:31:22.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.983+0000 7fa779d9b640 1 --2- 172.21.15.83:0/1633892802 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa75c06b4d0 0x7fa75c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.983+0000 7fa779d9b640 1 --2- 172.21.15.83:0/1633892802 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa75c06b4d0 0x7fa75c06d990 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fa770036000 tx=0x7fa770030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:22.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:22.987+0000 7fa76affd640 1 -- 172.21.15.83:0/1633892802 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa76c058410 con 0x7fa77c0b41f0 2024-03-06T07:31:23.085 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:23.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:23.195+0000 7fa780fc7640 1 -- 172.21.15.83:0/1633892802 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa77c0747a0 con 0x7fa77c0b41f0 2024-03-06T07:31:23.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:23.195+0000 7fa76affd640 1 -- 172.21.15.83:0/1633892802 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7fa76c057d10 con 0x7fa77c0b41f0 2024-03-06T07:31:23.197 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:31:23.197 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:31:23.198 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:31:23.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:23.197+0000 7fa780fc7640 1 -- 172.21.15.83:0/1633892802 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa75c06b4d0 msgr2=0x7fa75c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:23.197+0000 7fa780fc7640 1 --2- 172.21.15.83:0/1633892802 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa75c06b4d0 0x7fa75c06d990 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fa770036000 tx=0x7fa770030000 comp rx=0 tx=0).stop 2024-03-06T07:31:23.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:23.197+0000 7fa780fc7640 1 -- 172.21.15.83:0/1633892802 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa77c0b41f0 msgr2=0x7fa77c14d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:23.197+0000 7fa780fc7640 1 --2- 172.21.15.83:0/1633892802 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa77c0b41f0 0x7fa77c14d320 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7fa76c00c3f0 tx=0x7fa76c00c8b0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:23.197+0000 7fa780fc7640 1 -- 172.21.15.83:0/1633892802 shutdown_connections 2024-03-06T07:31:23.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:23.197+0000 7fa780fc7640 1 --2- 172.21.15.83:0/1633892802 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa75c06b4d0 0x7fa75c06d990 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:23.197+0000 7fa780fc7640 1 --2- 172.21.15.83:0/1633892802 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa77c15dfe0 0x7fa77c1605f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:23.197+0000 7fa780fc7640 1 --2- 172.21.15.83:0/1633892802 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa77c0d22b0 0x7fa77c0cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:23.197+0000 7fa780fc7640 1 --2- 172.21.15.83:0/1633892802 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa77c0b41f0 0x7fa77c14d320 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:23.197+0000 7fa780fc7640 1 -- 172.21.15.83:0/1633892802 >> 172.21.15.83:0/1633892802 conn(0x7fa77c0d3cd0 msgr2=0x7fa77c06cc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:23.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:23.197+0000 7fa780fc7640 1 -- 172.21.15.83:0/1633892802 shutdown_connections 2024-03-06T07:31:23.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:23.197+0000 7fa780fc7640 1 -- 172.21.15.83:0/1633892802 wait complete. 2024-03-06T07:31:23.411 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:31:23.412 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:31:24.413 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:24.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.513+0000 7f110b21c640 1 Processor -- start 2024-03-06T07:31:24.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.513+0000 7f110b21c640 1 -- start start 2024-03-06T07:31:24.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.514+0000 7f110b21c640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f11040d22b0 0x7f11040d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.514+0000 7f110b21c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11040b3430 con 0x7f110415dfe0 2024-03-06T07:31:24.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.514+0000 7f110b21c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11040b3710 con 0x7f11040b41f0 2024-03-06T07:31:24.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.514+0000 7f110b21c640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11040d2c50 con 0x7f11040d22b0 2024-03-06T07:31:24.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.514+0000 7f1108fb8640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f110415dfe0 0x7f11040cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48298/0 (socket says 172.21.15.83:48298) 2024-03-06T07:31:24.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.514+0000 7f1108fb8640 1 -- 172.21.15.83:0/3671529360 learned_addr learned my addr 172.21.15.83:0/3671529360 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:31:24.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.514+0000 7f1103fff640 1 --2- 172.21.15.83:0/3671529360 >> v2:172.21.15.183:3300/0 conn(0x7f11040d22b0 0x7f11040d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.514+0000 7f11037fe640 1 -- 172.21.15.83:0/3671529360 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 684481309 0 0) 0x7f11040b3430 con 0x7f110415dfe0 2024-03-06T07:31:24.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.514+0000 7f11037fe640 1 -- 172.21.15.83:0/3671529360 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10f4003520 con 0x7f110415dfe0 2024-03-06T07:31:24.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.514+0000 7f11037fe640 1 -- 172.21.15.83:0/3671529360 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1903886310 0 0) 0x7f10f4003520 con 0x7f110415dfe0 2024-03-06T07:31:24.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.514+0000 7f11037fe640 1 -- 172.21.15.83:0/3671529360 >> v2:172.21.15.183:3300/0 conn(0x7f11040d22b0 msgr2=0x7f11040d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:24.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.514+0000 7f11037fe640 1 --2- 172.21.15.83:0/3671529360 >> v2:172.21.15.183:3300/0 conn(0x7f11040d22b0 0x7f11040d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.514+0000 7f11037fe640 1 -- 172.21.15.83:0/3671529360 >> v1:172.21.15.183:6789/0 conn(0x7f11040b41f0 legacy=0x7f11040b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.514+0000 7f11037fe640 1 -- 172.21.15.83:0/3671529360 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11040b39f0 con 0x7f110415dfe0 2024-03-06T07:31:24.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.514+0000 7f11037fe640 1 -- 172.21.15.83:0/3671529360 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f10f8001670 con 0x7f110415dfe0 2024-03-06T07:31:24.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.514+0000 7f11037fe640 1 -- 172.21.15.83:0/3671529360 >> v1:172.21.15.83:6789/0 conn(0x7f110415dfe0 legacy=0x7f11040cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.514+0000 7f11037fe640 1 --2- 172.21.15.83:0/3671529360 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10f40039b0 0x7f10f4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.514+0000 7f11037fe640 1 --2- 172.21.15.83:0/3671529360 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10f4004800 0x7f10f4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.514+0000 7f11037fe640 1 --2- 172.21.15.83:0/3671529360 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f10f40051c0 0x7f10f400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.514+0000 7f11037fe640 1 -- 172.21.15.83:0/3671529360 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f400c030 con 0x7f10f40039b0 2024-03-06T07:31:24.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.514+0000 7f11037fe640 1 -- 172.21.15.83:0/3671529360 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f400c1e0 con 0x7f10f4004800 2024-03-06T07:31:24.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.514+0000 7f11037fe640 1 -- 172.21.15.83:0/3671529360 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10f400c3e0 con 0x7f10f40051c0 2024-03-06T07:31:24.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.515+0000 7f1108fb8640 1 --2- 172.21.15.83:0/3671529360 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10f40039b0 0x7f10f4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.515+0000 7f1103fff640 1 --2- 172.21.15.83:0/3671529360 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10f4004800 0x7f10f4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.515+0000 7f11097b9640 1 --2- 172.21.15.83:0/3671529360 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f10f40051c0 0x7f10f400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.515+0000 7f1108fb8640 1 -- 172.21.15.83:0/3671529360 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f10f40051c0 msgr2=0x7f10f400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:24.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.515+0000 7f1108fb8640 1 --2- 172.21.15.83:0/3671529360 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f10f40051c0 0x7f10f400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.515+0000 7f1108fb8640 1 -- 172.21.15.83:0/3671529360 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10f4004800 msgr2=0x7f10f4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:24.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.515+0000 7f1108fb8640 1 --2- 172.21.15.83:0/3671529360 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10f4004800 0x7f10f4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.515+0000 7f11097b9640 1 -- 172.21.15.83:0/3671529360 reap_dead start 2024-03-06T07:31:24.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.515+0000 7f1108fb8640 1 -- 172.21.15.83:0/3671529360 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f4003520 con 0x7f10f40039b0 2024-03-06T07:31:24.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.516+0000 7f1108fb8640 1 --2- 172.21.15.83:0/3671529360 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10f40039b0 0x7f10f4003dd0 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f10f800c210 tx=0x7f10f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20a01c255cafb675 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:24.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.516+0000 7f11037fe640 1 -- 172.21.15.83:0/3671529360 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f8007d10 con 0x7f10f40039b0 2024-03-06T07:31:24.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.516+0000 7f11037fe640 1 -- 172.21.15.83:0/3671529360 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f8030070 con 0x7f10f40039b0 2024-03-06T07:31:24.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.516+0000 7f11037fe640 1 -- 172.21.15.83:0/3671529360 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f8005d30 con 0x7f10f40039b0 2024-03-06T07:31:24.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.516+0000 7f110b21c640 1 -- 172.21.15.83:0/3671529360 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10f40039b0 msgr2=0x7f10f4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.516+0000 7f110b21c640 1 --2- 172.21.15.83:0/3671529360 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10f40039b0 0x7f10f4003dd0 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f10f800c210 tx=0x7f10f4005830 comp rx=0 tx=0).stop 2024-03-06T07:31:24.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.516+0000 7f110b21c640 1 -- 172.21.15.83:0/3671529360 shutdown_connections 2024-03-06T07:31:24.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.516+0000 7f110b21c640 1 --2- 172.21.15.83:0/3671529360 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10f40039b0 0x7f10f4003dd0 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.516+0000 7f110b21c640 1 -- 172.21.15.83:0/3671529360 >> 172.21.15.83:0/3671529360 conn(0x7f11040d3cd0 msgr2=0x7f11040d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:24.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.516+0000 7f110b21c640 1 -- 172.21.15.83:0/3671529360 shutdown_connections 2024-03-06T07:31:24.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.516+0000 7f110b21c640 1 -- 172.21.15.83:0/3671529360 wait complete. 2024-03-06T07:31:24.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.518+0000 7f110b21c640 1 Processor -- start 2024-03-06T07:31:24.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.518+0000 7f110b21c640 1 -- start start 2024-03-06T07:31:24.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.518+0000 7f110b21c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10f40039b0 0x7f110414eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.518+0000 7f110b21c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1104160920 0x7f1104160da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.518+0000 7f110b21c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11040b7570 0x7f11040b79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.518+0000 7f110b21c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11040b3d60 con 0x7f1104160920 2024-03-06T07:31:24.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.518+0000 7f110b21c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11040b31e0 con 0x7f10f40039b0 2024-03-06T07:31:24.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.518+0000 7f110b21c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f110415e3c0 con 0x7f11040b7570 2024-03-06T07:31:24.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.519+0000 7f1103fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1104160920 0x7f1104160da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.519+0000 7f1103fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1104160920 0x7f1104160da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56296/0 (socket says 172.21.15.83:56296) 2024-03-06T07:31:24.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.519+0000 7f1103fff640 1 -- 172.21.15.83:0/1550669786 learned_addr learned my addr 172.21.15.83:0/1550669786 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:24.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.519+0000 7f1108fb8640 1 --2- 172.21.15.83:0/1550669786 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10f40039b0 0x7f110414eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.519+0000 7f11097b9640 1 --2- 172.21.15.83:0/1550669786 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11040b7570 0x7f11040b79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.519+0000 7f1103fff640 1 -- 172.21.15.83:0/1550669786 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11040b7570 msgr2=0x7f11040b79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.519+0000 7f1103fff640 1 --2- 172.21.15.83:0/1550669786 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11040b7570 0x7f11040b79f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.519+0000 7f1103fff640 1 -- 172.21.15.83:0/1550669786 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10f40039b0 msgr2=0x7f110414eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.519+0000 7f1103fff640 1 --2- 172.21.15.83:0/1550669786 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10f40039b0 0x7f110414eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.519+0000 7f1103fff640 1 -- 172.21.15.83:0/1550669786 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11040b82e0 con 0x7f1104160920 2024-03-06T07:31:24.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.519+0000 7f1103fff640 1 --2- 172.21.15.83:0/1550669786 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1104160920 0x7f1104160da0 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f10f00118c0 tx=0x7f10f0011d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:24.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.519+0000 7f11017fa640 1 -- 172.21.15.83:0/1550669786 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f0019070 con 0x7f1104160920 2024-03-06T07:31:24.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.519+0000 7f11017fa640 1 -- 172.21.15.83:0/1550669786 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f0004240 con 0x7f1104160920 2024-03-06T07:31:24.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.519+0000 7f110b21c640 1 -- 172.21.15.83:0/1550669786 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11040caf40 con 0x7f1104160920 2024-03-06T07:31:24.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.519+0000 7f11017fa640 1 -- 172.21.15.83:0/1550669786 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f000f820 con 0x7f1104160920 2024-03-06T07:31:24.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.519+0000 7f110b21c640 1 -- 172.21.15.83:0/1550669786 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11040cb500 con 0x7f1104160920 2024-03-06T07:31:24.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.521+0000 7f110b21c640 1 -- 172.21.15.83:0/1550669786 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11041612e0 con 0x7f1104160920 2024-03-06T07:31:24.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.521+0000 7f11017fa640 1 -- 172.21.15.83:0/1550669786 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f10f001c070 con 0x7f1104160920 2024-03-06T07:31:24.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.521+0000 7f11017fa640 1 --2- 172.21.15.83:0/1550669786 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f10e806b4d0 0x7f10e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.521+0000 7f11017fa640 1 -- 172.21.15.83:0/1550669786 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f10f008a440 con 0x7f1104160920 2024-03-06T07:31:24.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.521+0000 7f1108fb8640 1 --2- 172.21.15.83:0/1550669786 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f10e806b4d0 0x7f10e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.522+0000 7f1108fb8640 1 --2- 172.21.15.83:0/1550669786 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f10e806b4d0 0x7f10e806d990 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f10f8034000 tx=0x7f10f8007000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:24.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.530+0000 7f11017fa640 1 -- 172.21.15.83:0/1550669786 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f11040b82e0 con 0x7f1104160920 2024-03-06T07:31:24.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.745+0000 7f110b21c640 1 -- 172.21.15.83:0/1550669786 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f110404f330 con 0x7f1104160920 2024-03-06T07:31:24.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.746+0000 7f11017fa640 1 -- 172.21.15.83:0/1550669786 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f10f0058cc0 con 0x7f1104160920 2024-03-06T07:31:24.748 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:31:24.748 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:31:24.748 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:31:24.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.748+0000 7f110b21c640 1 -- 172.21.15.83:0/1550669786 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f10e806b4d0 msgr2=0x7f10e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.748+0000 7f110b21c640 1 --2- 172.21.15.83:0/1550669786 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f10e806b4d0 0x7f10e806d990 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f10f8034000 tx=0x7f10f8007000 comp rx=0 tx=0).stop 2024-03-06T07:31:24.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.748+0000 7f110b21c640 1 -- 172.21.15.83:0/1550669786 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1104160920 msgr2=0x7f1104160da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.748+0000 7f110b21c640 1 --2- 172.21.15.83:0/1550669786 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1104160920 0x7f1104160da0 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f10f00118c0 tx=0x7f10f0011d80 comp rx=0 tx=0).stop 2024-03-06T07:31:24.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.748+0000 7f110b21c640 1 -- 172.21.15.83:0/1550669786 shutdown_connections 2024-03-06T07:31:24.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.748+0000 7f110b21c640 1 --2- 172.21.15.83:0/1550669786 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f10e806b4d0 0x7f10e806d990 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.748+0000 7f110b21c640 1 --2- 172.21.15.83:0/1550669786 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11040b7570 0x7f11040b79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.748+0000 7f110b21c640 1 --2- 172.21.15.83:0/1550669786 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1104160920 0x7f1104160da0 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.748+0000 7f110b21c640 1 --2- 172.21.15.83:0/1550669786 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10f40039b0 0x7f110414eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.748+0000 7f110b21c640 1 -- 172.21.15.83:0/1550669786 >> 172.21.15.83:0/1550669786 conn(0x7f11040d3cd0 msgr2=0x7f110406b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:24.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.748+0000 7f110b21c640 1 -- 172.21.15.83:0/1550669786 shutdown_connections 2024-03-06T07:31:24.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:24.748+0000 7f110b21c640 1 -- 172.21.15.83:0/1550669786 wait complete. 2024-03-06T07:31:24.962 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:31:24.962 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:31:25.087 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:25.963 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:26.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.063+0000 7f18ea01c640 1 Processor -- start 2024-03-06T07:31:26.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.063+0000 7f18ea01c640 1 -- start start 2024-03-06T07:31:26.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.063+0000 7f18ea01c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f18e40cf850 0x7f18e40cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.064+0000 7f18ea01c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18e415e380 con 0x7f18e40d00c0 2024-03-06T07:31:26.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.064+0000 7f18ea01c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18e40cdc10 con 0x7f18e406cf20 2024-03-06T07:31:26.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.064+0000 7f18ea01c640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18e406ca90 con 0x7f18e40cf850 2024-03-06T07:31:26.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.064+0000 7f18e2ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f18e40cf850 0x7f18e40cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.064+0000 7f18e2ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f18e40cf850 0x7f18e40cd3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56306/0 (socket says 172.21.15.83:56306) 2024-03-06T07:31:26.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.064+0000 7f18e2ffd640 1 -- 172.21.15.83:0/1962342503 learned_addr learned my addr 172.21.15.83:0/1962342503 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:26.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.064+0000 7f18e37fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f18e40d00c0 0x7f18e40d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48306/0 (socket says 172.21.15.83:48306) 2024-03-06T07:31:26.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.064+0000 7f18e2ffd640 1 -- 172.21.15.83:0/1962342503 >> v1:172.21.15.183:6789/0 conn(0x7f18e406cf20 legacy=0x7f18e406d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:26.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.064+0000 7f18e2ffd640 1 -- 172.21.15.83:0/1962342503 >> v1:172.21.15.83:6789/0 conn(0x7f18e40d00c0 legacy=0x7f18e40d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.064+0000 7f18e2ffd640 1 -- 172.21.15.83:0/1962342503 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e406c160 con 0x7f18e40cf850 2024-03-06T07:31:26.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.065+0000 7f18e2ffd640 1 --2- 172.21.15.83:0/1962342503 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18e40cf850 0x7f18e40cd3c0 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f18d8009870 tx=0x7f18d802b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2154e96615fb4ef9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:26.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.065+0000 7f18e27fc640 1 -- 172.21.15.83:0/1962342503 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d8037070 con 0x7f18e40cf850 2024-03-06T07:31:26.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.065+0000 7f18e27fc640 1 -- 172.21.15.83:0/1962342503 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18d8004230 con 0x7f18e40cf850 2024-03-06T07:31:26.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.065+0000 7f18e27fc640 1 -- 172.21.15.83:0/1962342503 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d8003610 con 0x7f18e40cf850 2024-03-06T07:31:26.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.065+0000 7f18ea01c640 1 -- 172.21.15.83:0/1962342503 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18e40cf850 msgr2=0x7f18e40cd3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.065+0000 7f18ea01c640 1 --2- 172.21.15.83:0/1962342503 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18e40cf850 0x7f18e40cd3c0 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f18d8009870 tx=0x7f18d802b7e0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.065+0000 7f18ea01c640 1 -- 172.21.15.83:0/1962342503 shutdown_connections 2024-03-06T07:31:26.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.065+0000 7f18ea01c640 1 --2- 172.21.15.83:0/1962342503 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18e40cf850 0x7f18e40cd3c0 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.065+0000 7f18ea01c640 1 -- 172.21.15.83:0/1962342503 >> 172.21.15.83:0/1962342503 conn(0x7f18e40b35a0 msgr2=0x7f18e40b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:26.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.066+0000 7f18ea01c640 1 -- 172.21.15.83:0/1962342503 shutdown_connections 2024-03-06T07:31:26.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.066+0000 7f18ea01c640 1 -- 172.21.15.83:0/1962342503 wait complete. 2024-03-06T07:31:26.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.067+0000 7f18ea01c640 1 Processor -- start 2024-03-06T07:31:26.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.067+0000 7f18ea01c640 1 -- start start 2024-03-06T07:31:26.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.067+0000 7f18ea01c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18e406cf20 0x7f18e415f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.067+0000 7f18ea01c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18e40cf850 0x7f18e41600e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.067+0000 7f18ea01c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18e40d00c0 0x7f18e4160620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.067+0000 7f18ea01c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e40cd900 con 0x7f18e406cf20 2024-03-06T07:31:26.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.067+0000 7f18ea01c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e406c720 con 0x7f18e40d00c0 2024-03-06T07:31:26.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.067+0000 7f18ea01c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18e40b1030 con 0x7f18e40cf850 2024-03-06T07:31:26.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.068+0000 7f18e37fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18e406cf20 0x7f18e415f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.068+0000 7f18e37fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18e406cf20 0x7f18e415f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56308/0 (socket says 172.21.15.83:56308) 2024-03-06T07:31:26.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.068+0000 7f18e37fe640 1 -- 172.21.15.83:0/3704650491 learned_addr learned my addr 172.21.15.83:0/3704650491 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:26.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.068+0000 7f18e3fff640 1 --2- 172.21.15.83:0/3704650491 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18e40d00c0 0x7f18e4160620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.068+0000 7f18e2ffd640 1 --2- 172.21.15.83:0/3704650491 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18e40cf850 0x7f18e41600e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.068+0000 7f18e37fe640 1 -- 172.21.15.83:0/3704650491 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18e40cf850 msgr2=0x7f18e41600e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.068+0000 7f18e37fe640 1 --2- 172.21.15.83:0/3704650491 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18e40cf850 0x7f18e41600e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.068+0000 7f18e37fe640 1 -- 172.21.15.83:0/3704650491 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18e40d00c0 msgr2=0x7f18e4160620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.068+0000 7f18e37fe640 1 --2- 172.21.15.83:0/3704650491 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18e40d00c0 0x7f18e4160620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.068+0000 7f18e37fe640 1 -- 172.21.15.83:0/3704650491 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e40ca4a0 con 0x7f18e406cf20 2024-03-06T07:31:26.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.068+0000 7f18e2ffd640 1 --2- 172.21.15.83:0/3704650491 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18e40cf850 0x7f18e41600e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:26.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.068+0000 7f18e37fe640 1 --2- 172.21.15.83:0/3704650491 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18e406cf20 0x7f18e415f440 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f18d400c780 tx=0x7f18d400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:26.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.068+0000 7f18cbfff640 1 -- 172.21.15.83:0/3704650491 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d4018070 con 0x7f18e406cf20 2024-03-06T07:31:26.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.068+0000 7f18cbfff640 1 -- 172.21.15.83:0/3704650491 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18d40072d0 con 0x7f18e406cf20 2024-03-06T07:31:26.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.068+0000 7f18ea01c640 1 -- 172.21.15.83:0/3704650491 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18e40ca780 con 0x7f18e406cf20 2024-03-06T07:31:26.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.068+0000 7f18cbfff640 1 -- 172.21.15.83:0/3704650491 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d40047e0 con 0x7f18e406cf20 2024-03-06T07:31:26.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.068+0000 7f18ea01c640 1 -- 172.21.15.83:0/3704650491 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18e40cad40 con 0x7f18e406cf20 2024-03-06T07:31:26.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.069+0000 7f18ea01c640 1 -- 172.21.15.83:0/3704650491 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18e404f330 con 0x7f18e406cf20 2024-03-06T07:31:26.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.070+0000 7f18cbfff640 1 -- 172.21.15.83:0/3704650491 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f18d401b070 con 0x7f18e406cf20 2024-03-06T07:31:26.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.070+0000 7f18cbfff640 1 --2- 172.21.15.83:0/3704650491 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f18d006b4d0 0x7f18d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.070+0000 7f18cbfff640 1 -- 172.21.15.83:0/3704650491 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f18d4089580 con 0x7f18e406cf20 2024-03-06T07:31:26.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.070+0000 7f18e2ffd640 1 --2- 172.21.15.83:0/3704650491 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f18d006b4d0 0x7f18d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.071+0000 7f18e2ffd640 1 --2- 172.21.15.83:0/3704650491 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f18d006b4d0 0x7f18d006d990 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f18d8034000 tx=0x7f18d8008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:26.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.074+0000 7f18cbfff640 1 -- 172.21.15.83:0/3704650491 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f18d4058510 con 0x7f18e406cf20 2024-03-06T07:31:26.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.289+0000 7f18ea01c640 1 -- 172.21.15.83:0/3704650491 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18e40cbf30 con 0x7f18e406cf20 2024-03-06T07:31:26.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.290+0000 7f18cbfff640 1 -- 172.21.15.83:0/3704650491 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f18d4006e20 con 0x7f18e406cf20 2024-03-06T07:31:26.292 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:31:26.292 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:29:54.013138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:31:26.292 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 82 2024-03-06T07:31:26.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.292+0000 7f18ea01c640 1 -- 172.21.15.83:0/3704650491 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f18d006b4d0 msgr2=0x7f18d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.292+0000 7f18ea01c640 1 --2- 172.21.15.83:0/3704650491 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f18d006b4d0 0x7f18d006d990 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f18d8034000 tx=0x7f18d8008030 comp rx=0 tx=0).stop 2024-03-06T07:31:26.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.292+0000 7f18ea01c640 1 -- 172.21.15.83:0/3704650491 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18e406cf20 msgr2=0x7f18e415f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.292+0000 7f18ea01c640 1 --2- 172.21.15.83:0/3704650491 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18e406cf20 0x7f18e415f440 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f18d400c780 tx=0x7f18d400cc40 comp rx=0 tx=0).stop 2024-03-06T07:31:26.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.292+0000 7f18ea01c640 1 -- 172.21.15.83:0/3704650491 shutdown_connections 2024-03-06T07:31:26.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.292+0000 7f18ea01c640 1 --2- 172.21.15.83:0/3704650491 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f18d006b4d0 0x7f18d006d990 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.292+0000 7f18ea01c640 1 --2- 172.21.15.83:0/3704650491 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18e40d00c0 0x7f18e4160620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.292+0000 7f18ea01c640 1 --2- 172.21.15.83:0/3704650491 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18e40cf850 0x7f18e41600e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.292+0000 7f18ea01c640 1 --2- 172.21.15.83:0/3704650491 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18e406cf20 0x7f18e415f440 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.292+0000 7f18ea01c640 1 -- 172.21.15.83:0/3704650491 >> 172.21.15.83:0/3704650491 conn(0x7f18e40b35a0 msgr2=0x7f18e40cdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:26.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.292+0000 7f18ea01c640 1 -- 172.21.15.83:0/3704650491 shutdown_connections 2024-03-06T07:31:26.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:26.292+0000 7f18ea01c640 1 -- 172.21.15.83:0/3704650491 wait complete. 2024-03-06T07:31:26.506 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:29:54.013138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:31:26.506 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:31:27.089 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:27.507 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:27.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.611+0000 7f9dfb113640 1 Processor -- start 2024-03-06T07:31:27.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.611+0000 7f9dfb113640 1 -- start start 2024-03-06T07:31:27.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.612+0000 7f9dfb113640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9df40b3060 0x7f9df415e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.612+0000 7f9dfb113640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9df415f580 con 0x7f9df413dc10 2024-03-06T07:31:27.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.612+0000 7f9dfb113640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9df415b450 con 0x7f9df415bf30 2024-03-06T07:31:27.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.612+0000 7f9dfb113640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9df40cf3f0 con 0x7f9df40b3060 2024-03-06T07:31:27.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.612+0000 7f9df8eaf640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f9df413dc10 0x7f9df40b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:33588/0 (socket says 172.21.15.83:33588) 2024-03-06T07:31:27.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.612+0000 7f9df8eaf640 1 -- 172.21.15.83:0/944565037 learned_addr learned my addr 172.21.15.83:0/944565037 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:31:27.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.612+0000 7f9df3fff640 1 --2- 172.21.15.83:0/944565037 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9df40b3060 0x7f9df415e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.612+0000 7f9df37fe640 1 -- 172.21.15.83:0/944565037 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1021787254 0 0) 0x7f9df415f580 con 0x7f9df413dc10 2024-03-06T07:31:27.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.612+0000 7f9df37fe640 1 -- 172.21.15.83:0/944565037 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ddc003520 con 0x7f9df413dc10 2024-03-06T07:31:27.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.613+0000 7f9df37fe640 1 -- 172.21.15.83:0/944565037 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2948602137 0 0) 0x7f9ddc003520 con 0x7f9df413dc10 2024-03-06T07:31:27.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.613+0000 7f9df37fe640 1 -- 172.21.15.83:0/944565037 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9df40b3060 msgr2=0x7f9df415e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:27.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.613+0000 7f9df37fe640 1 --2- 172.21.15.83:0/944565037 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9df40b3060 0x7f9df415e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.613+0000 7f9df37fe640 1 -- 172.21.15.83:0/944565037 >> v1:172.21.15.183:6789/0 conn(0x7f9df415bf30 legacy=0x7f9df415ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.613+0000 7f9df37fe640 1 -- 172.21.15.83:0/944565037 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df415b730 con 0x7f9df413dc10 2024-03-06T07:31:27.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.613+0000 7f9df37fe640 1 -- 172.21.15.83:0/944565037 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f9dec0019f0 con 0x7f9df413dc10 2024-03-06T07:31:27.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.613+0000 7f9df37fe640 1 -- 172.21.15.83:0/944565037 >> v1:172.21.15.83:6789/0 conn(0x7f9df413dc10 legacy=0x7f9df40b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.613+0000 7f9df37fe640 1 --2- 172.21.15.83:0/944565037 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9ddc0039e0 0x7f9ddc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.613+0000 7f9df37fe640 1 --2- 172.21.15.83:0/944565037 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ddc004800 0x7f9ddc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.613+0000 7f9df37fe640 1 --2- 172.21.15.83:0/944565037 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9df40b3060 0x7f9ddc005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.613+0000 7f9df37fe640 1 -- 172.21.15.83:0/944565037 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df40cf3f0 con 0x7f9ddc004800 2024-03-06T07:31:27.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.613+0000 7f9df37fe640 1 -- 172.21.15.83:0/944565037 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ddc00bb20 con 0x7f9ddc0039e0 2024-03-06T07:31:27.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.613+0000 7f9df37fe640 1 -- 172.21.15.83:0/944565037 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ddc00bd20 con 0x7f9df40b3060 2024-03-06T07:31:27.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.613+0000 7f9df3fff640 1 --2- 172.21.15.83:0/944565037 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ddc004800 0x7f9ddc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.613+0000 7f9df8eaf640 1 --2- 172.21.15.83:0/944565037 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9ddc0039e0 0x7f9ddc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.613+0000 7f9df96b0640 1 --2- 172.21.15.83:0/944565037 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9df40b3060 0x7f9ddc005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.613+0000 7f9df3fff640 1 -- 172.21.15.83:0/944565037 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9df40b3060 msgr2=0x7f9ddc005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:27.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.613+0000 7f9df3fff640 1 --2- 172.21.15.83:0/944565037 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9df40b3060 0x7f9ddc005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.613+0000 7f9df3fff640 1 -- 172.21.15.83:0/944565037 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9ddc0039e0 msgr2=0x7f9ddc003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:27.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.613+0000 7f9df3fff640 1 --2- 172.21.15.83:0/944565037 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9ddc0039e0 0x7f9ddc003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.613+0000 7f9df3fff640 1 -- 172.21.15.83:0/944565037 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ddc003520 con 0x7f9ddc004800 2024-03-06T07:31:27.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.614+0000 7f9df3fff640 1 --2- 172.21.15.83:0/944565037 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ddc004800 0x7f9ddc004c80 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f9de8009ac0 tx=0x7f9ddc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58a31f97985309cf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:27.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.614+0000 7f9df37fe640 1 -- 172.21.15.83:0/944565037 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de8039070 con 0x7f9ddc004800 2024-03-06T07:31:27.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.614+0000 7f9df37fe640 1 -- 172.21.15.83:0/944565037 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9de802c6b0 con 0x7f9ddc004800 2024-03-06T07:31:27.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.614+0000 7f9df37fe640 1 -- 172.21.15.83:0/944565037 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de8009690 con 0x7f9ddc004800 2024-03-06T07:31:27.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.615+0000 7f9dfb113640 1 -- 172.21.15.83:0/944565037 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ddc004800 msgr2=0x7f9ddc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.615+0000 7f9dfb113640 1 --2- 172.21.15.83:0/944565037 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ddc004800 0x7f9ddc004c80 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f9de8009ac0 tx=0x7f9ddc005830 comp rx=0 tx=0).stop 2024-03-06T07:31:27.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.615+0000 7f9df96b0640 1 -- 172.21.15.83:0/944565037 reap_dead start 2024-03-06T07:31:27.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.615+0000 7f9dfb113640 1 -- 172.21.15.83:0/944565037 shutdown_connections 2024-03-06T07:31:27.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.615+0000 7f9dfb113640 1 -- 172.21.15.83:0/944565037 >> 172.21.15.83:0/944565037 conn(0x7f9df40cf8a0 msgr2=0x7f9df40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:27.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.615+0000 7f9dfb113640 1 -- 172.21.15.83:0/944565037 shutdown_connections 2024-03-06T07:31:27.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.615+0000 7f9dfb113640 1 -- 172.21.15.83:0/944565037 wait complete. 2024-03-06T07:31:27.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.616+0000 7f9dfb113640 1 Processor -- start 2024-03-06T07:31:27.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.616+0000 7f9dfb113640 1 -- start start 2024-03-06T07:31:27.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.617+0000 7f9dfb113640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ddc004800 0x7f9df413f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.617+0000 7f9dfb113640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9df41567e0 0x7f9df41f4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.617+0000 7f9dfb113640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9df4153530 0x7f9df41539b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.617+0000 7f9dfb113640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df415e850 con 0x7f9ddc004800 2024-03-06T07:31:27.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.617+0000 7f9dfb113640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df415b210 con 0x7f9df4153530 2024-03-06T07:31:27.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.617+0000 7f9dfb113640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9df415e9d0 con 0x7f9df41567e0 2024-03-06T07:31:27.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.617+0000 7f9df8eaf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ddc004800 0x7f9df413f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.617+0000 7f9df8eaf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ddc004800 0x7f9df413f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36122/0 (socket says 172.21.15.83:36122) 2024-03-06T07:31:27.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.617+0000 7f9df8eaf640 1 -- 172.21.15.83:0/3351138054 learned_addr learned my addr 172.21.15.83:0/3351138054 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:27.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.617+0000 7f9df96b0640 1 --2- 172.21.15.83:0/3351138054 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9df4153530 0x7f9df41539b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.617+0000 7f9df3fff640 1 --2- 172.21.15.83:0/3351138054 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9df41567e0 0x7f9df41f4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.617+0000 7f9df8eaf640 1 -- 172.21.15.83:0/3351138054 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9df41567e0 msgr2=0x7f9df41f4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.617+0000 7f9df8eaf640 1 --2- 172.21.15.83:0/3351138054 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9df41567e0 0x7f9df41f4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.617+0000 7f9df8eaf640 1 -- 172.21.15.83:0/3351138054 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9df4153530 msgr2=0x7f9df41539b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.617+0000 7f9df8eaf640 1 --2- 172.21.15.83:0/3351138054 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9df4153530 0x7f9df41539b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.617+0000 7f9df8eaf640 1 -- 172.21.15.83:0/3351138054 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df4156fd0 con 0x7f9ddc004800 2024-03-06T07:31:27.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.617+0000 7f9df8eaf640 1 --2- 172.21.15.83:0/3351138054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ddc004800 0x7f9df413f400 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f9dec0077c0 tx=0x7f9dec00a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:27.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.617+0000 7f9df17fa640 1 -- 172.21.15.83:0/3351138054 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dec017020 con 0x7f9ddc004800 2024-03-06T07:31:27.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.617+0000 7f9df17fa640 1 -- 172.21.15.83:0/3351138054 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dec007ac0 con 0x7f9ddc004800 2024-03-06T07:31:27.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.617+0000 7f9dfb113640 1 -- 172.21.15.83:0/3351138054 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9df41572b0 con 0x7f9ddc004800 2024-03-06T07:31:27.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.617+0000 7f9df17fa640 1 -- 172.21.15.83:0/3351138054 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dec013610 con 0x7f9ddc004800 2024-03-06T07:31:27.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.617+0000 7f9dfb113640 1 -- 172.21.15.83:0/3351138054 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9df4157870 con 0x7f9ddc004800 2024-03-06T07:31:27.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.619+0000 7f9dfb113640 1 -- 172.21.15.83:0/3351138054 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9df404f330 con 0x7f9ddc004800 2024-03-06T07:31:27.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.619+0000 7f9df17fa640 1 -- 172.21.15.83:0/3351138054 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9dec01a020 con 0x7f9ddc004800 2024-03-06T07:31:27.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.619+0000 7f9df17fa640 1 --2- 172.21.15.83:0/3351138054 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9de006b4d0 0x7f9de006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.619+0000 7f9df17fa640 1 -- 172.21.15.83:0/3351138054 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9dec08ad00 con 0x7f9ddc004800 2024-03-06T07:31:27.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.619+0000 7f9df3fff640 1 --2- 172.21.15.83:0/3351138054 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9de006b4d0 0x7f9de006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.620+0000 7f9df3fff640 1 --2- 172.21.15.83:0/3351138054 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9de006b4d0 0x7f9de006d990 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f9de8036000 tx=0x7f9de8030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:27.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.628+0000 7f9df17fa640 1 -- 172.21.15.83:0/3351138054 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9dec021020 con 0x7f9ddc004800 2024-03-06T07:31:27.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.844+0000 7f9dfb113640 1 -- 172.21.15.83:0/3351138054 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9df406f260 con 0x7f9ddc004800 2024-03-06T07:31:27.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.844+0000 7f9df17fa640 1 -- 172.21.15.83:0/3351138054 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f9dec059ab0 con 0x7f9ddc004800 2024-03-06T07:31:27.846 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:31:27.846 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:31:27.847 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:31:27.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.846+0000 7f9dfb113640 1 -- 172.21.15.83:0/3351138054 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9de006b4d0 msgr2=0x7f9de006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.846+0000 7f9dfb113640 1 --2- 172.21.15.83:0/3351138054 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9de006b4d0 0x7f9de006d990 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f9de8036000 tx=0x7f9de8030000 comp rx=0 tx=0).stop 2024-03-06T07:31:27.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.846+0000 7f9dfb113640 1 -- 172.21.15.83:0/3351138054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ddc004800 msgr2=0x7f9df413f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.846+0000 7f9dfb113640 1 --2- 172.21.15.83:0/3351138054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ddc004800 0x7f9df413f400 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f9dec0077c0 tx=0x7f9dec00a6f0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.846+0000 7f9dfb113640 1 -- 172.21.15.83:0/3351138054 shutdown_connections 2024-03-06T07:31:27.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.846+0000 7f9dfb113640 1 --2- 172.21.15.83:0/3351138054 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9de006b4d0 0x7f9de006d990 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.846+0000 7f9dfb113640 1 --2- 172.21.15.83:0/3351138054 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9df4153530 0x7f9df41539b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.846+0000 7f9dfb113640 1 --2- 172.21.15.83:0/3351138054 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9df41567e0 0x7f9df41f4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.846+0000 7f9dfb113640 1 --2- 172.21.15.83:0/3351138054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ddc004800 0x7f9df413f400 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.846+0000 7f9dfb113640 1 -- 172.21.15.83:0/3351138054 >> 172.21.15.83:0/3351138054 conn(0x7f9df40cf8a0 msgr2=0x7f9df406d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:27.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.846+0000 7f9dfb113640 1 -- 172.21.15.83:0/3351138054 shutdown_connections 2024-03-06T07:31:27.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:27.846+0000 7f9dfb113640 1 -- 172.21.15.83:0/3351138054 wait complete. 2024-03-06T07:31:28.060 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:31:26.591282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:31:28.060 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:31:29.061 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:29.090 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:29.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.159+0000 7f604ba50640 1 Processor -- start 2024-03-06T07:31:29.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.160+0000 7f604ba50640 1 -- start start 2024-03-06T07:31:29.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.160+0000 7f604ba50640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f604415dfe0 0x7f60440cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.160+0000 7f604ba50640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f60440cf830 0x7f60440d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.160+0000 7f604ba50640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f60440d27f0 0x7f60440d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.160+0000 7f604ba50640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60440d2c50 con 0x7f60440d27f0 2024-03-06T07:31:29.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.160+0000 7f604ba50640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60440b31e0 con 0x7f604415dfe0 2024-03-06T07:31:29.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.160+0000 7f604ba50640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60440b3d60 con 0x7f60440cf830 2024-03-06T07:31:29.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.160+0000 7f6049fed640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f60440d27f0 0x7f60440d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.161+0000 7f6049fed640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f60440d27f0 0x7f60440d0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36136/0 (socket says 172.21.15.83:36136) 2024-03-06T07:31:29.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.161+0000 7f6049fed640 1 -- 172.21.15.83:0/2751079233 learned_addr learned my addr 172.21.15.83:0/2751079233 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:29.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.161+0000 7f6049fed640 1 -- 172.21.15.83:0/2751079233 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f60440cf830 msgr2=0x7f60440d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:29.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.161+0000 7f6049fed640 1 --2- 172.21.15.83:0/2751079233 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f60440cf830 0x7f60440d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.161+0000 7f6049fed640 1 -- 172.21.15.83:0/2751079233 >> v2:172.21.15.183:3300/0 conn(0x7f604415dfe0 msgr2=0x7f60440cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:29.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.161+0000 7f6049fed640 1 --2- 172.21.15.83:0/2751079233 >> v2:172.21.15.183:3300/0 conn(0x7f604415dfe0 0x7f60440cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.161+0000 7f6049fed640 1 -- 172.21.15.83:0/2751079233 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60440b3430 con 0x7f60440d27f0 2024-03-06T07:31:29.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.162+0000 7f6049fed640 1 --2- 172.21.15.83:0/2751079233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60440d27f0 0x7f60440d0300 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f6044067990 tx=0x7f603c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b49b73e0413e26d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:29.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.162+0000 7f603bfff640 1 -- 172.21.15.83:0/2751079233 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603c02ca70 con 0x7f60440d27f0 2024-03-06T07:31:29.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.162+0000 7f603bfff640 1 -- 172.21.15.83:0/2751079233 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f603c02cc10 con 0x7f60440d27f0 2024-03-06T07:31:29.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.162+0000 7f603bfff640 1 -- 172.21.15.83:0/2751079233 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603c0343f0 con 0x7f60440d27f0 2024-03-06T07:31:29.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.162+0000 7f604ba50640 1 -- 172.21.15.83:0/2751079233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60440d27f0 msgr2=0x7f60440d0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.162+0000 7f604ba50640 1 --2- 172.21.15.83:0/2751079233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60440d27f0 0x7f60440d0300 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f6044067990 tx=0x7f603c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.162+0000 7f604ba50640 1 -- 172.21.15.83:0/2751079233 shutdown_connections 2024-03-06T07:31:29.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.162+0000 7f604ba50640 1 --2- 172.21.15.83:0/2751079233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60440d27f0 0x7f60440d0300 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.162+0000 7f604ba50640 1 --2- 172.21.15.83:0/2751079233 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f60440cf830 0x7f60440d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.162+0000 7f604ba50640 1 --2- 172.21.15.83:0/2751079233 >> v2:172.21.15.183:3300/0 conn(0x7f604415dfe0 0x7f60440cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.162+0000 7f604ba50640 1 -- 172.21.15.83:0/2751079233 >> 172.21.15.83:0/2751079233 conn(0x7f60440d3cd0 msgr2=0x7f60440d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:29.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.162+0000 7f604ba50640 1 -- 172.21.15.83:0/2751079233 shutdown_connections 2024-03-06T07:31:29.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.162+0000 7f604ba50640 1 -- 172.21.15.83:0/2751079233 wait complete. 2024-03-06T07:31:29.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.164+0000 7f604ba50640 1 Processor -- start 2024-03-06T07:31:29.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.164+0000 7f604ba50640 1 -- start start 2024-03-06T07:31:29.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.164+0000 7f604ba50640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f60440cf830 0x7f6044158140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.164+0000 7f604ba50640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60440d27f0 0x7f604415fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.164+0000 7f604ba50640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f604415dfe0 0x7f604415b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.164+0000 7f604ba50640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60440d1c70 con 0x7f60440d27f0 2024-03-06T07:31:29.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.164+0000 7f604ba50640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60440b39f0 con 0x7f60440cf830 2024-03-06T07:31:29.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.164+0000 7f604ba50640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60440d1df0 con 0x7f604415dfe0 2024-03-06T07:31:29.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.164+0000 7f6048feb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60440d27f0 0x7f604415fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.164+0000 7f6048feb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60440d27f0 0x7f604415fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36148/0 (socket says 172.21.15.83:36148) 2024-03-06T07:31:29.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.164+0000 7f6048feb640 1 -- 172.21.15.83:0/1982924861 learned_addr learned my addr 172.21.15.83:0/1982924861 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:29.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.165+0000 7f6049fed640 1 --2- 172.21.15.83:0/1982924861 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f604415dfe0 0x7f604415b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.165+0000 7f60497ec640 1 --2- 172.21.15.83:0/1982924861 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f60440cf830 0x7f6044158140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.165+0000 7f6048feb640 1 -- 172.21.15.83:0/1982924861 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f604415dfe0 msgr2=0x7f604415b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.165+0000 7f6048feb640 1 --2- 172.21.15.83:0/1982924861 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f604415dfe0 0x7f604415b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.165+0000 7f6048feb640 1 -- 172.21.15.83:0/1982924861 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f60440cf830 msgr2=0x7f6044158140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.165+0000 7f6048feb640 1 --2- 172.21.15.83:0/1982924861 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f60440cf830 0x7f6044158140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.165+0000 7f6048feb640 1 -- 172.21.15.83:0/1982924861 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60440ccbf0 con 0x7f60440d27f0 2024-03-06T07:31:29.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.165+0000 7f6048feb640 1 --2- 172.21.15.83:0/1982924861 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60440d27f0 0x7f604415fea0 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f603400b450 tx=0x7f603400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:29.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.165+0000 7f6039ffb640 1 -- 172.21.15.83:0/1982924861 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6034015020 con 0x7f60440d27f0 2024-03-06T07:31:29.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.165+0000 7f6039ffb640 1 -- 172.21.15.83:0/1982924861 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6034011890 con 0x7f60440d27f0 2024-03-06T07:31:29.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.165+0000 7f604ba50640 1 -- 172.21.15.83:0/1982924861 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60440cbb50 con 0x7f60440d27f0 2024-03-06T07:31:29.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.165+0000 7f6039ffb640 1 -- 172.21.15.83:0/1982924861 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6034011c80 con 0x7f60440d27f0 2024-03-06T07:31:29.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.165+0000 7f604ba50640 1 -- 172.21.15.83:0/1982924861 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60440cc3f0 con 0x7f60440d27f0 2024-03-06T07:31:29.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.166+0000 7f604ba50640 1 -- 172.21.15.83:0/1982924861 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f604404f330 con 0x7f60440d27f0 2024-03-06T07:31:29.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.166+0000 7f6039ffb640 1 -- 172.21.15.83:0/1982924861 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6034018020 con 0x7f60440d27f0 2024-03-06T07:31:29.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.167+0000 7f6039ffb640 1 --2- 172.21.15.83:0/1982924861 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f602806b4d0 0x7f602806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.167+0000 7f60497ec640 1 --2- 172.21.15.83:0/1982924861 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f602806b4d0 0x7f602806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.167+0000 7f6039ffb640 1 -- 172.21.15.83:0/1982924861 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f603408a3a0 con 0x7f60440d27f0 2024-03-06T07:31:29.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.167+0000 7f60497ec640 1 --2- 172.21.15.83:0/1982924861 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f602806b4d0 0x7f602806d990 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f60440ca2e0 tx=0x7f6040008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:29.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.171+0000 7f6039ffb640 1 -- 172.21.15.83:0/1982924861 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f603401f020 con 0x7f60440d27f0 2024-03-06T07:31:29.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.380+0000 7f604ba50640 1 -- 172.21.15.83:0/1982924861 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60440ceb30 con 0x7f60440d27f0 2024-03-06T07:31:29.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.381+0000 7f6039ffb640 1 -- 172.21.15.83:0/1982924861 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f603400d040 con 0x7f60440d27f0 2024-03-06T07:31:29.383 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:31:29.383 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:31:29.383 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:31:29.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.383+0000 7f604ba50640 1 -- 172.21.15.83:0/1982924861 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f602806b4d0 msgr2=0x7f602806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.383+0000 7f604ba50640 1 --2- 172.21.15.83:0/1982924861 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f602806b4d0 0x7f602806d990 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f60440ca2e0 tx=0x7f6040008040 comp rx=0 tx=0).stop 2024-03-06T07:31:29.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.383+0000 7f604ba50640 1 -- 172.21.15.83:0/1982924861 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60440d27f0 msgr2=0x7f604415fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.383+0000 7f604ba50640 1 --2- 172.21.15.83:0/1982924861 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60440d27f0 0x7f604415fea0 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f603400b450 tx=0x7f603400b910 comp rx=0 tx=0).stop 2024-03-06T07:31:29.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.383+0000 7f604ba50640 1 -- 172.21.15.83:0/1982924861 shutdown_connections 2024-03-06T07:31:29.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.383+0000 7f604ba50640 1 --2- 172.21.15.83:0/1982924861 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f602806b4d0 0x7f602806d990 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.383+0000 7f604ba50640 1 --2- 172.21.15.83:0/1982924861 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f604415dfe0 0x7f604415b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.383+0000 7f604ba50640 1 --2- 172.21.15.83:0/1982924861 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f60440d27f0 0x7f604415fea0 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.383+0000 7f604ba50640 1 --2- 172.21.15.83:0/1982924861 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f60440cf830 0x7f6044158140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.383+0000 7f604ba50640 1 -- 172.21.15.83:0/1982924861 >> 172.21.15.83:0/1982924861 conn(0x7f60440d3cd0 msgr2=0x7f60440b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:29.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.383+0000 7f604ba50640 1 -- 172.21.15.83:0/1982924861 shutdown_connections 2024-03-06T07:31:29.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:29.383+0000 7f604ba50640 1 -- 172.21.15.83:0/1982924861 wait complete. 2024-03-06T07:31:29.597 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:31:26.591282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:31:29.597 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:31:30.597 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:30.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.696+0000 7f5744a4e640 1 Processor -- start 2024-03-06T07:31:30.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.696+0000 7f5744a4e640 1 -- start start 2024-03-06T07:31:30.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.697+0000 7f5744a4e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f574015dfe0 0x7f57400cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.697+0000 7f5744a4e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f57400cf830 0x7f57400d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.697+0000 7f5744a4e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57400b3430 con 0x7f57400d27f0 2024-03-06T07:31:30.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.697+0000 7f5744a4e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57400d2c50 con 0x7f574015dfe0 2024-03-06T07:31:30.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.697+0000 7f5744a4e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57400b31e0 con 0x7f57400cf830 2024-03-06T07:31:30.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.697+0000 7f573e59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f574015dfe0 0x7f57400cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.697+0000 7f573ed9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f57400d27f0 0x7f57400d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:33592/0 (socket says 172.21.15.83:33592) 2024-03-06T07:31:30.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.697+0000 7f573e59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f574015dfe0 0x7f57400cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36154/0 (socket says 172.21.15.83:36154) 2024-03-06T07:31:30.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.697+0000 7f573ed9d640 1 -- 172.21.15.83:0/382300900 learned_addr learned my addr 172.21.15.83:0/382300900 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:31:30.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.697+0000 7f573dd9b640 1 --2- 172.21.15.83:0/382300900 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f57400cf830 0x7f57400d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.697+0000 7f573e59c640 1 -- 172.21.15.83:0/382300900 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f57400cf830 msgr2=0x7f57400d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:30.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.697+0000 7f573e59c640 1 --2- 172.21.15.83:0/382300900 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f57400cf830 0x7f57400d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.697+0000 7f573e59c640 1 -- 172.21.15.83:0/382300900 >> v1:172.21.15.83:6789/0 conn(0x7f57400d27f0 legacy=0x7f57400d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.697+0000 7f573e59c640 1 -- 172.21.15.83:0/382300900 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57400b3710 con 0x7f574015dfe0 2024-03-06T07:31:30.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.697+0000 7f573d59a640 1 -- 172.21.15.83:0/382300900 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 22638023 0 0) 0x7f57400b3430 con 0x7f57400d27f0 2024-03-06T07:31:30.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.698+0000 7f573e59c640 1 --2- 172.21.15.83:0/382300900 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f574015dfe0 0x7f57400cf2f0 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f5738009d90 tx=0x7f573802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb80b06e8d7e58f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:30.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.698+0000 7f573d59a640 1 -- 172.21.15.83:0/382300900 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5738039070 con 0x7f574015dfe0 2024-03-06T07:31:30.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.698+0000 7f573d59a640 1 -- 172.21.15.83:0/382300900 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f573802cc50 con 0x7f574015dfe0 2024-03-06T07:31:30.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.698+0000 7f573d59a640 1 -- 172.21.15.83:0/382300900 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5738034440 con 0x7f574015dfe0 2024-03-06T07:31:30.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.699+0000 7f5744a4e640 1 -- 172.21.15.83:0/382300900 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f574015dfe0 msgr2=0x7f57400cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.699+0000 7f5744a4e640 1 --2- 172.21.15.83:0/382300900 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f574015dfe0 0x7f57400cf2f0 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f5738009d90 tx=0x7f573802bea0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.699+0000 7f5744a4e640 1 -- 172.21.15.83:0/382300900 shutdown_connections 2024-03-06T07:31:30.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.699+0000 7f5744a4e640 1 --2- 172.21.15.83:0/382300900 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f57400cf830 0x7f57400d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.699+0000 7f5744a4e640 1 --2- 172.21.15.83:0/382300900 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f574015dfe0 0x7f57400cf2f0 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.699+0000 7f5744a4e640 1 -- 172.21.15.83:0/382300900 >> 172.21.15.83:0/382300900 conn(0x7f57400d3cd0 msgr2=0x7f57400d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:30.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.699+0000 7f5744a4e640 1 -- 172.21.15.83:0/382300900 shutdown_connections 2024-03-06T07:31:30.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.699+0000 7f5744a4e640 1 -- 172.21.15.83:0/382300900 wait complete. 2024-03-06T07:31:30.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.700+0000 7f5744a4e640 1 Processor -- start 2024-03-06T07:31:30.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.700+0000 7f5744a4e640 1 -- start start 2024-03-06T07:31:30.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.701+0000 7f5744a4e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f57400cf830 0x7f57400b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.701+0000 7f5744a4e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f57400d27f0 0x7f57401f3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.701+0000 7f5744a4e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f574015dfe0 0x7f574015f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.701+0000 7f5744a4e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57400d1c70 con 0x7f574015dfe0 2024-03-06T07:31:30.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.701+0000 7f5744a4e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57400b3d60 con 0x7f57400d27f0 2024-03-06T07:31:30.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.701+0000 7f5744a4e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57400d1df0 con 0x7f57400cf830 2024-03-06T07:31:30.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.701+0000 7f573ed9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f574015dfe0 0x7f574015f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.701+0000 7f573ed9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f574015dfe0 0x7f574015f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36168/0 (socket says 172.21.15.83:36168) 2024-03-06T07:31:30.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.701+0000 7f573ed9d640 1 -- 172.21.15.83:0/3120584107 learned_addr learned my addr 172.21.15.83:0/3120584107 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:30.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.701+0000 7f573e59c640 1 --2- 172.21.15.83:0/3120584107 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f57400cf830 0x7f57400b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.701+0000 7f573dd9b640 1 --2- 172.21.15.83:0/3120584107 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f57400d27f0 0x7f57401f3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.701+0000 7f573ed9d640 1 -- 172.21.15.83:0/3120584107 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f57400cf830 msgr2=0x7f57400b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.701+0000 7f573ed9d640 1 --2- 172.21.15.83:0/3120584107 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f57400cf830 0x7f57400b68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.702+0000 7f573ed9d640 1 -- 172.21.15.83:0/3120584107 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f57400d27f0 msgr2=0x7f57401f3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.702+0000 7f573ed9d640 1 --2- 172.21.15.83:0/3120584107 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f57400d27f0 0x7f57401f3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.702+0000 7f573ed9d640 1 -- 172.21.15.83:0/3120584107 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f574014e030 con 0x7f574015dfe0 2024-03-06T07:31:30.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.702+0000 7f573e59c640 1 --2- 172.21.15.83:0/3120584107 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f57400cf830 0x7f57400b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:30.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.702+0000 7f573dd9b640 1 --2- 172.21.15.83:0/3120584107 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f57400d27f0 0x7f57401f3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:30.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.702+0000 7f573ed9d640 1 --2- 172.21.15.83:0/3120584107 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f574015dfe0 0x7f574015f4a0 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f573000c450 tx=0x7f573000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:30.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.702+0000 7f572effd640 1 -- 172.21.15.83:0/3120584107 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57300074e0 con 0x7f574015dfe0 2024-03-06T07:31:30.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.702+0000 7f572effd640 1 -- 172.21.15.83:0/3120584107 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57300047e0 con 0x7f574015dfe0 2024-03-06T07:31:30.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.702+0000 7f572effd640 1 -- 172.21.15.83:0/3120584107 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5730004bd0 con 0x7f574015dfe0 2024-03-06T07:31:30.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.702+0000 7f5744a4e640 1 -- 172.21.15.83:0/3120584107 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57400b8430 con 0x7f574015dfe0 2024-03-06T07:31:30.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.702+0000 7f5744a4e640 1 -- 172.21.15.83:0/3120584107 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57400b89f0 con 0x7f574015dfe0 2024-03-06T07:31:30.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.703+0000 7f5744a4e640 1 -- 172.21.15.83:0/3120584107 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f574004f330 con 0x7f574015dfe0 2024-03-06T07:31:30.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.704+0000 7f572effd640 1 -- 172.21.15.83:0/3120584107 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5730011070 con 0x7f574015dfe0 2024-03-06T07:31:30.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.704+0000 7f572effd640 1 --2- 172.21.15.83:0/3120584107 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f572006b500 0x7f572006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.704+0000 7f572effd640 1 -- 172.21.15.83:0/3120584107 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5730089d60 con 0x7f574015dfe0 2024-03-06T07:31:30.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.704+0000 7f573e59c640 1 --2- 172.21.15.83:0/3120584107 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f572006b500 0x7f572006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.705+0000 7f573e59c640 1 --2- 172.21.15.83:0/3120584107 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f572006b500 0x7f572006d9c0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f5738036000 tx=0x7f5738030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:30.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.708+0000 7f572effd640 1 -- 172.21.15.83:0/3120584107 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f573001a2b0 con 0x7f574015dfe0 2024-03-06T07:31:30.808 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:30.810 DEBUG:teuthology.orchestra.run.smithi183:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:30.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.931+0000 7f5744a4e640 1 -- 172.21.15.83:0/3120584107 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57400ca280 con 0x7f574015dfe0 2024-03-06T07:31:30.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.931+0000 7f572effd640 1 -- 172.21.15.83:0/3120584107 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f5730058ba0 con 0x7f574015dfe0 2024-03-06T07:31:30.933 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:31:30.933 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:31:30.934 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:31:30.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.933+0000 7f5744a4e640 1 -- 172.21.15.83:0/3120584107 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f572006b500 msgr2=0x7f572006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.933+0000 7f5744a4e640 1 --2- 172.21.15.83:0/3120584107 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f572006b500 0x7f572006d9c0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f5738036000 tx=0x7f5738030000 comp rx=0 tx=0).stop 2024-03-06T07:31:30.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.933+0000 7f5744a4e640 1 -- 172.21.15.83:0/3120584107 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f574015dfe0 msgr2=0x7f574015f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.933+0000 7f5744a4e640 1 --2- 172.21.15.83:0/3120584107 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f574015dfe0 0x7f574015f4a0 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f573000c450 tx=0x7f573000c910 comp rx=0 tx=0).stop 2024-03-06T07:31:30.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.933+0000 7f5744a4e640 1 -- 172.21.15.83:0/3120584107 shutdown_connections 2024-03-06T07:31:30.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.933+0000 7f5744a4e640 1 --2- 172.21.15.83:0/3120584107 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f572006b500 0x7f572006d9c0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.933+0000 7f5744a4e640 1 --2- 172.21.15.83:0/3120584107 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f574015dfe0 0x7f574015f4a0 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.933+0000 7f5744a4e640 1 --2- 172.21.15.83:0/3120584107 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f57400d27f0 0x7f57401f3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.934+0000 7f5744a4e640 1 --2- 172.21.15.83:0/3120584107 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f57400cf830 0x7f57400b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.934+0000 7f5744a4e640 1 -- 172.21.15.83:0/3120584107 >> 172.21.15.83:0/3120584107 conn(0x7f57400d3cd0 msgr2=0x7f574006ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:30.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.934+0000 7f5744a4e640 1 -- 172.21.15.83:0/3120584107 shutdown_connections 2024-03-06T07:31:30.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:30.934+0000 7f5744a4e640 1 -- 172.21.15.83:0/3120584107 wait complete. 2024-03-06T07:31:31.092 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:31.151 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:31:26.591282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:31:31.151 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:31:32.152 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:32.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.252+0000 7f93f2484640 1 Processor -- start 2024-03-06T07:31:32.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.252+0000 7f93f2484640 1 -- start start 2024-03-06T07:31:32.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.253+0000 7f93f2484640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f93ec15dfe0 0x7f93ec0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.253+0000 7f93f2484640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f93ec0d29c0 0x7f93ec0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.253+0000 7f93f2484640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93ec0b3430 con 0x7f93ec0cf830 2024-03-06T07:31:32.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.253+0000 7f93f2484640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93ec0b31e0 con 0x7f93ec0d29c0 2024-03-06T07:31:32.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.253+0000 7f93f2484640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93ec0b3d60 con 0x7f93ec15dfe0 2024-03-06T07:31:32.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.253+0000 7f93eb7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f93ec0cf830 0x7f93ec0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:33600/0 (socket says 172.21.15.83:33600) 2024-03-06T07:31:32.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.253+0000 7f93eb7fe640 1 -- 172.21.15.83:0/1259356001 learned_addr learned my addr 172.21.15.83:0/1259356001 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:31:32.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.253+0000 7f93f0a21640 1 --2- 172.21.15.83:0/1259356001 >> v2:172.21.15.183:3300/0 conn(0x7f93ec0d29c0 0x7f93ec0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:32.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.253+0000 7f93ebfff640 1 --2- 172.21.15.83:0/1259356001 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f93ec15dfe0 0x7f93ec0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:32.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.253+0000 7f93eaffd640 1 -- 172.21.15.83:0/1259356001 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3339772445 0 0) 0x7f93ec0b3430 con 0x7f93ec0cf830 2024-03-06T07:31:32.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.253+0000 7f93eaffd640 1 -- 172.21.15.83:0/1259356001 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93d4003520 con 0x7f93ec0cf830 2024-03-06T07:31:32.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.253+0000 7f93eaffd640 1 -- 172.21.15.83:0/1259356001 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3902222093 0 0) 0x7f93d4003520 con 0x7f93ec0cf830 2024-03-06T07:31:32.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.253+0000 7f93eaffd640 1 -- 172.21.15.83:0/1259356001 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f93ec15dfe0 msgr2=0x7f93ec0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:32.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.253+0000 7f93eaffd640 1 --2- 172.21.15.83:0/1259356001 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f93ec15dfe0 0x7f93ec0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.253+0000 7f93eaffd640 1 -- 172.21.15.83:0/1259356001 >> v2:172.21.15.183:3300/0 conn(0x7f93ec0d29c0 msgr2=0x7f93ec0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:32.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.253+0000 7f93eaffd640 1 --2- 172.21.15.83:0/1259356001 >> v2:172.21.15.183:3300/0 conn(0x7f93ec0d29c0 0x7f93ec0b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.253+0000 7f93eaffd640 1 -- 172.21.15.83:0/1259356001 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93ec0b3710 con 0x7f93ec0cf830 2024-03-06T07:31:32.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.253+0000 7f93eaffd640 1 -- 172.21.15.83:0/1259356001 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f93e0001670 con 0x7f93ec0cf830 2024-03-06T07:31:32.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.253+0000 7f93eaffd640 1 -- 172.21.15.83:0/1259356001 >> v1:172.21.15.83:6789/0 conn(0x7f93ec0cf830 legacy=0x7f93ec0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:32.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.253+0000 7f93eaffd640 1 --2- 172.21.15.83:0/1259356001 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f93d40039b0 0x7f93d4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.253+0000 7f93eaffd640 1 --2- 172.21.15.83:0/1259356001 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93d4004800 0x7f93d4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.253+0000 7f93eaffd640 1 --2- 172.21.15.83:0/1259356001 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f93ec15dfe0 0x7f93d40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.253+0000 7f93eaffd640 1 -- 172.21.15.83:0/1259356001 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93ec0b31e0 con 0x7f93d4004800 2024-03-06T07:31:32.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.253+0000 7f93eaffd640 1 -- 172.21.15.83:0/1259356001 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93ec0b3d60 con 0x7f93d40039b0 2024-03-06T07:31:32.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.253+0000 7f93eaffd640 1 -- 172.21.15.83:0/1259356001 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93d400b9c0 con 0x7f93ec15dfe0 2024-03-06T07:31:32.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.254+0000 7f93eb7fe640 1 --2- 172.21.15.83:0/1259356001 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93d4004800 0x7f93d4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:32.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.254+0000 7f93ebfff640 1 --2- 172.21.15.83:0/1259356001 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f93d40039b0 0x7f93d4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:32.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.254+0000 7f93f0a21640 1 --2- 172.21.15.83:0/1259356001 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f93ec15dfe0 0x7f93d40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:32.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.254+0000 7f93eb7fe640 1 -- 172.21.15.83:0/1259356001 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f93ec15dfe0 msgr2=0x7f93d40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:32.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.254+0000 7f93eb7fe640 1 --2- 172.21.15.83:0/1259356001 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f93ec15dfe0 0x7f93d40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.254+0000 7f93eb7fe640 1 -- 172.21.15.83:0/1259356001 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f93d40039b0 msgr2=0x7f93d4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:32.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.254+0000 7f93eb7fe640 1 --2- 172.21.15.83:0/1259356001 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f93d40039b0 0x7f93d4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.254+0000 7f93eb7fe640 1 -- 172.21.15.83:0/1259356001 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d4003520 con 0x7f93d4004800 2024-03-06T07:31:32.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.255+0000 7f93eb7fe640 1 --2- 172.21.15.83:0/1259356001 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93d4004800 0x7f93d4004c80 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f93ec0cfc70 tx=0x7f93d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27ec4a27f44093d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:32.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.255+0000 7f93eaffd640 1 -- 172.21.15.83:0/1259356001 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e0007d50 con 0x7f93d4004800 2024-03-06T07:31:32.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.255+0000 7f93eaffd640 1 -- 172.21.15.83:0/1259356001 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93e0030070 con 0x7f93d4004800 2024-03-06T07:31:32.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.255+0000 7f93eaffd640 1 -- 172.21.15.83:0/1259356001 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e0005d10 con 0x7f93d4004800 2024-03-06T07:31:32.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.255+0000 7f93f2484640 1 -- 172.21.15.83:0/1259356001 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93d4004800 msgr2=0x7f93d4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:32.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.255+0000 7f93f2484640 1 --2- 172.21.15.83:0/1259356001 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93d4004800 0x7f93d4004c80 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f93ec0cfc70 tx=0x7f93d4005830 comp rx=0 tx=0).stop 2024-03-06T07:31:32.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.255+0000 7f93f0a21640 1 -- 172.21.15.83:0/1259356001 reap_dead start 2024-03-06T07:31:32.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.255+0000 7f93f2484640 1 -- 172.21.15.83:0/1259356001 shutdown_connections 2024-03-06T07:31:32.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.255+0000 7f93f2484640 1 -- 172.21.15.83:0/1259356001 >> 172.21.15.83:0/1259356001 conn(0x7f93ec0d3cd0 msgr2=0x7f93ec0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:32.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.255+0000 7f93f2484640 1 -- 172.21.15.83:0/1259356001 shutdown_connections 2024-03-06T07:31:32.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.255+0000 7f93f2484640 1 -- 172.21.15.83:0/1259356001 wait complete. 2024-03-06T07:31:32.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.256+0000 7f93f2484640 1 Processor -- start 2024-03-06T07:31:32.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.257+0000 7f93f2484640 1 -- start start 2024-03-06T07:31:32.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.257+0000 7f93f2484640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93d4004800 0x7f93ec15ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.257+0000 7f93f2484640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f93ec0cd3f0 0x7f93ec0cd870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.257+0000 7f93f2484640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f93ec06cae0 0x7f93ec06cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.257+0000 7f93f2484640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93ec0d1df0 con 0x7f93d4004800 2024-03-06T07:31:32.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.257+0000 7f93f2484640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93ec0d1c70 con 0x7f93ec0cd3f0 2024-03-06T07:31:32.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.257+0000 7f93f2484640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93ec15e3c0 con 0x7f93ec06cae0 2024-03-06T07:31:32.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.257+0000 7f93ebfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93d4004800 0x7f93ec15ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:32.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.257+0000 7f93ebfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93d4004800 0x7f93ec15ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36190/0 (socket says 172.21.15.83:36190) 2024-03-06T07:31:32.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.258+0000 7f93ebfff640 1 -- 172.21.15.83:0/2670470316 learned_addr learned my addr 172.21.15.83:0/2670470316 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:32.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.258+0000 7f93f0a21640 1 --2- 172.21.15.83:0/2670470316 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f93ec06cae0 0x7f93ec06cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:32.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.258+0000 7f93eb7fe640 1 --2- 172.21.15.83:0/2670470316 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f93ec0cd3f0 0x7f93ec0cd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:32.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.258+0000 7f93ebfff640 1 -- 172.21.15.83:0/2670470316 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f93ec06cae0 msgr2=0x7f93ec06cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:32.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.258+0000 7f93ebfff640 1 --2- 172.21.15.83:0/2670470316 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f93ec06cae0 0x7f93ec06cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.258+0000 7f93ebfff640 1 -- 172.21.15.83:0/2670470316 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f93ec0cd3f0 msgr2=0x7f93ec0cd870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:32.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.258+0000 7f93ebfff640 1 --2- 172.21.15.83:0/2670470316 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f93ec0cd3f0 0x7f93ec0cd870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.258+0000 7f93ebfff640 1 -- 172.21.15.83:0/2670470316 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93ec15bf60 con 0x7f93d4004800 2024-03-06T07:31:32.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.258+0000 7f93eb7fe640 1 --2- 172.21.15.83:0/2670470316 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f93ec0cd3f0 0x7f93ec0cd870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:32.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.258+0000 7f93ebfff640 1 --2- 172.21.15.83:0/2670470316 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93d4004800 0x7f93ec15ba20 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f93dc0118e0 tx=0x7f93dc011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:32.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.258+0000 7f93e8ff9640 1 -- 172.21.15.83:0/2670470316 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93dc00faf0 con 0x7f93d4004800 2024-03-06T07:31:32.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.258+0000 7f93e8ff9640 1 -- 172.21.15.83:0/2670470316 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93dc00fc90 con 0x7f93d4004800 2024-03-06T07:31:32.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.258+0000 7f93f2484640 1 -- 172.21.15.83:0/2670470316 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93ec15e7b0 con 0x7f93d4004800 2024-03-06T07:31:32.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.258+0000 7f93e8ff9640 1 -- 172.21.15.83:0/2670470316 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93dc014440 con 0x7f93d4004800 2024-03-06T07:31:32.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.258+0000 7f93f2484640 1 -- 172.21.15.83:0/2670470316 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93ec15ed70 con 0x7f93d4004800 2024-03-06T07:31:32.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.259+0000 7f93f2484640 1 -- 172.21.15.83:0/2670470316 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93ec04f330 con 0x7f93d4004800 2024-03-06T07:31:32.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.260+0000 7f93e8ff9640 1 -- 172.21.15.83:0/2670470316 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f93dc012040 con 0x7f93d4004800 2024-03-06T07:31:32.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.260+0000 7f93e8ff9640 1 --2- 172.21.15.83:0/2670470316 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f93d806b4d0 0x7f93d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.260+0000 7f93e8ff9640 1 -- 172.21.15.83:0/2670470316 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f93dc08a760 con 0x7f93d4004800 2024-03-06T07:31:32.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.260+0000 7f93eb7fe640 1 --2- 172.21.15.83:0/2670470316 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f93d806b4d0 0x7f93d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:32.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.260+0000 7f93eb7fe640 1 --2- 172.21.15.83:0/2670470316 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f93d806b4d0 0x7f93d806d990 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f93e0034000 tx=0x7f93e0007000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:32.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.264+0000 7f93e8ff9640 1 -- 172.21.15.83:0/2670470316 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f93dc019030 con 0x7f93d4004800 2024-03-06T07:31:32.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.480+0000 7f93f2484640 1 -- 172.21.15.83:0/2670470316 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93ec0b2610 con 0x7f93d4004800 2024-03-06T07:31:32.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.480+0000 7f93e8ff9640 1 -- 172.21.15.83:0/2670470316 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f93dc059630 con 0x7f93d4004800 2024-03-06T07:31:32.482 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:31:32.482 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:31:32.483 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:31:32.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.482+0000 7f93f2484640 1 -- 172.21.15.83:0/2670470316 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f93d806b4d0 msgr2=0x7f93d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:32.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.482+0000 7f93f2484640 1 --2- 172.21.15.83:0/2670470316 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f93d806b4d0 0x7f93d806d990 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f93e0034000 tx=0x7f93e0007000 comp rx=0 tx=0).stop 2024-03-06T07:31:32.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.482+0000 7f93f2484640 1 -- 172.21.15.83:0/2670470316 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93d4004800 msgr2=0x7f93ec15ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:32.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.482+0000 7f93f2484640 1 --2- 172.21.15.83:0/2670470316 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93d4004800 0x7f93ec15ba20 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f93dc0118e0 tx=0x7f93dc011da0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.482+0000 7f93f2484640 1 -- 172.21.15.83:0/2670470316 shutdown_connections 2024-03-06T07:31:32.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.482+0000 7f93f2484640 1 --2- 172.21.15.83:0/2670470316 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f93d806b4d0 0x7f93d806d990 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.482+0000 7f93f2484640 1 --2- 172.21.15.83:0/2670470316 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f93ec06cae0 0x7f93ec06cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.482+0000 7f93f2484640 1 --2- 172.21.15.83:0/2670470316 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f93ec0cd3f0 0x7f93ec0cd870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.482+0000 7f93f2484640 1 --2- 172.21.15.83:0/2670470316 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93d4004800 0x7f93ec15ba20 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.483+0000 7f93f2484640 1 -- 172.21.15.83:0/2670470316 >> 172.21.15.83:0/2670470316 conn(0x7f93ec0d3cd0 msgr2=0x7f93ec0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:32.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.483+0000 7f93f2484640 1 -- 172.21.15.83:0/2670470316 shutdown_connections 2024-03-06T07:31:32.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:32.483+0000 7f93f2484640 1 -- 172.21.15.83:0/2670470316 wait complete. 2024-03-06T07:31:32.697 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:31:26.591282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:31:32.697 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:31:33.094 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:33.698 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:33.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.798+0000 7f09ea9b5640 1 Processor -- start 2024-03-06T07:31:33.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.798+0000 7f09ea9b5640 1 -- start start 2024-03-06T07:31:33.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.799+0000 7f09ea9b5640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f09e415dfe0 0x7f09e40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.799+0000 7f09ea9b5640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f09e40cf830 0x7f09e40d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.799+0000 7f09ea9b5640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f09e40d27f0 0x7f09e40d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.799+0000 7f09ea9b5640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09e40b31e0 con 0x7f09e415dfe0 2024-03-06T07:31:33.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.799+0000 7f09ea9b5640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09e40b3d60 con 0x7f09e40cf830 2024-03-06T07:31:33.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.799+0000 7f09ea9b5640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09e40b3430 con 0x7f09e40d27f0 2024-03-06T07:31:33.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.799+0000 7f09e3fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f09e415dfe0 0x7f09e40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.799+0000 7f09e3fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f09e415dfe0 0x7f09e40cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36192/0 (socket says 172.21.15.83:36192) 2024-03-06T07:31:33.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.799+0000 7f09e3fff640 1 -- 172.21.15.83:0/2708229688 learned_addr learned my addr 172.21.15.83:0/2708229688 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:33.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.799+0000 7f09e37fe640 1 --2- 172.21.15.83:0/2708229688 >> v2:172.21.15.183:3300/0 conn(0x7f09e40cf830 0x7f09e40d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.799+0000 7f09e8f52640 1 --2- 172.21.15.83:0/2708229688 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f09e40d27f0 0x7f09e40d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.799+0000 7f09e3fff640 1 -- 172.21.15.83:0/2708229688 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f09e40d27f0 msgr2=0x7f09e40d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:33.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.799+0000 7f09e3fff640 1 --2- 172.21.15.83:0/2708229688 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f09e40d27f0 0x7f09e40d0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.799+0000 7f09e3fff640 1 -- 172.21.15.83:0/2708229688 >> v2:172.21.15.183:3300/0 conn(0x7f09e40cf830 msgr2=0x7f09e40d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:33.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.799+0000 7f09e3fff640 1 --2- 172.21.15.83:0/2708229688 >> v2:172.21.15.183:3300/0 conn(0x7f09e40cf830 0x7f09e40d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.799+0000 7f09e3fff640 1 -- 172.21.15.83:0/2708229688 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e40b3710 con 0x7f09e415dfe0 2024-03-06T07:31:33.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.800+0000 7f09e3fff640 1 --2- 172.21.15.83:0/2708229688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09e415dfe0 0x7f09e40cf2f0 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f09e40d16b0 tx=0x7f09d802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3af3c0efc6315525 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:33.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.800+0000 7f09e2ffd640 1 -- 172.21.15.83:0/2708229688 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d8035020 con 0x7f09e415dfe0 2024-03-06T07:31:33.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.801+0000 7f09e2ffd640 1 -- 172.21.15.83:0/2708229688 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09d8031870 con 0x7f09e415dfe0 2024-03-06T07:31:33.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.801+0000 7f09e2ffd640 1 -- 172.21.15.83:0/2708229688 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d8031c40 con 0x7f09e415dfe0 2024-03-06T07:31:33.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.801+0000 7f09ea9b5640 1 -- 172.21.15.83:0/2708229688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09e415dfe0 msgr2=0x7f09e40cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:33.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.801+0000 7f09ea9b5640 1 --2- 172.21.15.83:0/2708229688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09e415dfe0 0x7f09e40cf2f0 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f09e40d16b0 tx=0x7f09d802b910 comp rx=0 tx=0).stop 2024-03-06T07:31:33.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.801+0000 7f09ea9b5640 1 -- 172.21.15.83:0/2708229688 shutdown_connections 2024-03-06T07:31:33.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.801+0000 7f09ea9b5640 1 --2- 172.21.15.83:0/2708229688 >> v2:172.21.15.183:3300/0 conn(0x7f09e40cf830 0x7f09e40d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.801+0000 7f09ea9b5640 1 --2- 172.21.15.83:0/2708229688 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f09e40d27f0 0x7f09e40d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.801+0000 7f09ea9b5640 1 --2- 172.21.15.83:0/2708229688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09e415dfe0 0x7f09e40cf2f0 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.801+0000 7f09ea9b5640 1 -- 172.21.15.83:0/2708229688 >> 172.21.15.83:0/2708229688 conn(0x7f09e40d3cd0 msgr2=0x7f09e40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:33.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.801+0000 7f09ea9b5640 1 -- 172.21.15.83:0/2708229688 shutdown_connections 2024-03-06T07:31:33.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.801+0000 7f09ea9b5640 1 -- 172.21.15.83:0/2708229688 wait complete. 2024-03-06T07:31:33.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.802+0000 7f09ea9b5640 1 Processor -- start 2024-03-06T07:31:33.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.802+0000 7f09ea9b5640 1 -- start start 2024-03-06T07:31:33.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.803+0000 7f09ea9b5640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f09e40cf830 0x7f09e41f4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.803+0000 7f09ea9b5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09e40d27f0 0x7f09e41f47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.803+0000 7f09ea9b5640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f09e415dfe0 0x7f09e40cd590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.803+0000 7f09ea9b5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e40d1df0 con 0x7f09e40d27f0 2024-03-06T07:31:33.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.803+0000 7f09ea9b5640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e40d1c70 con 0x7f09e40cf830 2024-03-06T07:31:33.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.803+0000 7f09ea9b5640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09e415e3c0 con 0x7f09e415dfe0 2024-03-06T07:31:33.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.803+0000 7f09e37fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09e40d27f0 0x7f09e41f47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.803+0000 7f09e37fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09e40d27f0 0x7f09e41f47d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36196/0 (socket says 172.21.15.83:36196) 2024-03-06T07:31:33.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.803+0000 7f09e37fe640 1 -- 172.21.15.83:0/2006934569 learned_addr learned my addr 172.21.15.83:0/2006934569 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:33.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.803+0000 7f09e3fff640 1 --2- 172.21.15.83:0/2006934569 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f09e40cf830 0x7f09e41f4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.803+0000 7f09e8f52640 1 --2- 172.21.15.83:0/2006934569 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f09e415dfe0 0x7f09e40cd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.804+0000 7f09e37fe640 1 -- 172.21.15.83:0/2006934569 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f09e415dfe0 msgr2=0x7f09e40cd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:33.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.804+0000 7f09e37fe640 1 --2- 172.21.15.83:0/2006934569 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f09e415dfe0 0x7f09e40cd590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.804+0000 7f09e37fe640 1 -- 172.21.15.83:0/2006934569 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f09e40cf830 msgr2=0x7f09e41f4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:33.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.804+0000 7f09e37fe640 1 --2- 172.21.15.83:0/2006934569 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f09e40cf830 0x7f09e41f4260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.804+0000 7f09e37fe640 1 -- 172.21.15.83:0/2006934569 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e40caed0 con 0x7f09e40d27f0 2024-03-06T07:31:33.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.804+0000 7f09e8f52640 1 --2- 172.21.15.83:0/2006934569 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f09e415dfe0 0x7f09e40cd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:33.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.804+0000 7f09e37fe640 1 --2- 172.21.15.83:0/2006934569 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09e40d27f0 0x7f09e41f47d0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f09d400c7f0 tx=0x7f09d400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:33.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.804+0000 7f09e0ff9640 1 -- 172.21.15.83:0/2006934569 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d4015070 con 0x7f09e40d27f0 2024-03-06T07:31:33.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.804+0000 7f09e0ff9640 1 -- 172.21.15.83:0/2006934569 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09d4004220 con 0x7f09e40d27f0 2024-03-06T07:31:33.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.804+0000 7f09ea9b5640 1 -- 172.21.15.83:0/2006934569 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09e40cb1b0 con 0x7f09e40d27f0 2024-03-06T07:31:33.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.804+0000 7f09e0ff9640 1 -- 172.21.15.83:0/2006934569 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d4007620 con 0x7f09e40d27f0 2024-03-06T07:31:33.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.804+0000 7f09ea9b5640 1 -- 172.21.15.83:0/2006934569 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09e41f4d10 con 0x7f09e40d27f0 2024-03-06T07:31:33.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.805+0000 7f09ea9b5640 1 -- 172.21.15.83:0/2006934569 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09e404f330 con 0x7f09e40d27f0 2024-03-06T07:31:33.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.806+0000 7f09e0ff9640 1 -- 172.21.15.83:0/2006934569 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f09d40077c0 con 0x7f09e40d27f0 2024-03-06T07:31:33.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.806+0000 7f09e0ff9640 1 --2- 172.21.15.83:0/2006934569 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f09d006b4d0 0x7f09d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.806+0000 7f09e0ff9640 1 -- 172.21.15.83:0/2006934569 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f09d4089fc0 con 0x7f09e40d27f0 2024-03-06T07:31:33.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.806+0000 7f09e3fff640 1 --2- 172.21.15.83:0/2006934569 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f09d006b4d0 0x7f09d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.807+0000 7f09e3fff640 1 --2- 172.21.15.83:0/2006934569 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f09d006b4d0 0x7f09d006d990 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f09d8036000 tx=0x7f09d803c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:33.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:33.810+0000 7f09e0ff9640 1 -- 172.21.15.83:0/2006934569 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f09d40587b0 con 0x7f09e40d27f0 2024-03-06T07:31:34.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:34.025+0000 7f09ea9b5640 1 -- 172.21.15.83:0/2006934569 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09e40747a0 con 0x7f09e40d27f0 2024-03-06T07:31:34.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:34.025+0000 7f09e0ff9640 1 -- 172.21.15.83:0/2006934569 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f09d405c850 con 0x7f09e40d27f0 2024-03-06T07:31:34.027 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:31:34.028 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:31:34.028 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:31:34.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:34.028+0000 7f09ea9b5640 1 -- 172.21.15.83:0/2006934569 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f09d006b4d0 msgr2=0x7f09d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:34.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:34.028+0000 7f09ea9b5640 1 --2- 172.21.15.83:0/2006934569 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f09d006b4d0 0x7f09d006d990 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f09d8036000 tx=0x7f09d803c040 comp rx=0 tx=0).stop 2024-03-06T07:31:34.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:34.028+0000 7f09ea9b5640 1 -- 172.21.15.83:0/2006934569 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09e40d27f0 msgr2=0x7f09e41f47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:34.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:34.028+0000 7f09ea9b5640 1 --2- 172.21.15.83:0/2006934569 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09e40d27f0 0x7f09e41f47d0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f09d400c7f0 tx=0x7f09d400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:34.028+0000 7f09ea9b5640 1 -- 172.21.15.83:0/2006934569 shutdown_connections 2024-03-06T07:31:34.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:34.028+0000 7f09ea9b5640 1 --2- 172.21.15.83:0/2006934569 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f09d006b4d0 0x7f09d006d990 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:34.028+0000 7f09ea9b5640 1 --2- 172.21.15.83:0/2006934569 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f09e415dfe0 0x7f09e40cd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:34.028+0000 7f09ea9b5640 1 --2- 172.21.15.83:0/2006934569 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09e40d27f0 0x7f09e41f47d0 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:34.028+0000 7f09ea9b5640 1 --2- 172.21.15.83:0/2006934569 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f09e40cf830 0x7f09e41f4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:34.028+0000 7f09ea9b5640 1 -- 172.21.15.83:0/2006934569 >> 172.21.15.83:0/2006934569 conn(0x7f09e40d3cd0 msgr2=0x7f09e40d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:34.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:34.028+0000 7f09ea9b5640 1 -- 172.21.15.83:0/2006934569 shutdown_connections 2024-03-06T07:31:34.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:34.028+0000 7f09ea9b5640 1 -- 172.21.15.83:0/2006934569 wait complete. 2024-03-06T07:31:34.245 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:31:26.591282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:31:34.246 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:31:35.096 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:35.246 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:35.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.348+0000 7f8c1cff7640 1 Processor -- start 2024-03-06T07:31:35.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.348+0000 7f8c1cff7640 1 -- start start 2024-03-06T07:31:35.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.349+0000 7f8c1cff7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8c1815dfe0 0x7f8c180cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.349+0000 7f8c1cff7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f8c180cf830 0x7f8c180d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.349+0000 7f8c1cff7640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f8c180d27f0 0x7f8c180d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.349+0000 7f8c1cff7640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c180d2c50 con 0x7f8c180cf830 2024-03-06T07:31:35.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.349+0000 7f8c1cff7640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c180b31e0 con 0x7f8c180d27f0 2024-03-06T07:31:35.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.349+0000 7f8c1cff7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c180b3d60 con 0x7f8c1815dfe0 2024-03-06T07:31:35.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.349+0000 7f8c15d9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f8c180cf830 0x7f8c180d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.349+0000 7f8c15d9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f8c180cf830 0x7f8c180d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36200/0 (socket says 172.21.15.83:36200) 2024-03-06T07:31:35.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.349+0000 7f8c15d9b640 1 -- 172.21.15.83:0/2807910926 learned_addr learned my addr 172.21.15.83:0/2807910926 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:35.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.349+0000 7f8c15d9b640 1 -- 172.21.15.83:0/2807910926 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8c1815dfe0 msgr2=0x7f8c180cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:35.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.349+0000 7f8c15d9b640 1 --2- 172.21.15.83:0/2807910926 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8c1815dfe0 0x7f8c180cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.349+0000 7f8c15d9b640 1 -- 172.21.15.83:0/2807910926 >> v2:172.21.15.183:3300/0 conn(0x7f8c180d27f0 msgr2=0x7f8c180d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:35.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.349+0000 7f8c15d9b640 1 --2- 172.21.15.83:0/2807910926 >> v2:172.21.15.183:3300/0 conn(0x7f8c180d27f0 0x7f8c180d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.349+0000 7f8c15d9b640 1 -- 172.21.15.83:0/2807910926 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c180b3430 con 0x7f8c180cf830 2024-03-06T07:31:35.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.350+0000 7f8c15d9b640 1 --2- 172.21.15.83:0/2807910926 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8c180cf830 0x7f8c180d22b0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f8c18067ad0 tx=0x7f8c0c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfb9456f5a19864b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:35.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.350+0000 7f8c1559a640 1 -- 172.21.15.83:0/2807910926 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c0c035020 con 0x7f8c180cf830 2024-03-06T07:31:35.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.350+0000 7f8c1559a640 1 -- 172.21.15.83:0/2807910926 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c0c031870 con 0x7f8c180cf830 2024-03-06T07:31:35.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.350+0000 7f8c1559a640 1 -- 172.21.15.83:0/2807910926 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c0c031c40 con 0x7f8c180cf830 2024-03-06T07:31:35.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.351+0000 7f8c1cff7640 1 -- 172.21.15.83:0/2807910926 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8c180cf830 msgr2=0x7f8c180d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.351+0000 7f8c1cff7640 1 --2- 172.21.15.83:0/2807910926 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8c180cf830 0x7f8c180d22b0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f8c18067ad0 tx=0x7f8c0c02b910 comp rx=0 tx=0).stop 2024-03-06T07:31:35.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.351+0000 7f8c1cff7640 1 -- 172.21.15.83:0/2807910926 shutdown_connections 2024-03-06T07:31:35.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.351+0000 7f8c1cff7640 1 --2- 172.21.15.83:0/2807910926 >> v2:172.21.15.183:3300/0 conn(0x7f8c180d27f0 0x7f8c180d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.351+0000 7f8c1cff7640 1 --2- 172.21.15.83:0/2807910926 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8c180cf830 0x7f8c180d22b0 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.351+0000 7f8c1cff7640 1 --2- 172.21.15.83:0/2807910926 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8c1815dfe0 0x7f8c180cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.351+0000 7f8c1cff7640 1 -- 172.21.15.83:0/2807910926 >> 172.21.15.83:0/2807910926 conn(0x7f8c180d3cd0 msgr2=0x7f8c180d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:35.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.351+0000 7f8c1cff7640 1 -- 172.21.15.83:0/2807910926 shutdown_connections 2024-03-06T07:31:35.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.351+0000 7f8c1cff7640 1 -- 172.21.15.83:0/2807910926 wait complete. 2024-03-06T07:31:35.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.352+0000 7f8c1cff7640 1 Processor -- start 2024-03-06T07:31:35.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.352+0000 7f8c1cff7640 1 -- start start 2024-03-06T07:31:35.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.353+0000 7f8c1cff7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8c180cf830 0x7f8c18155310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.353+0000 7f8c1cff7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8c180d27f0 0x7f8c181f39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.353+0000 7f8c1cff7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c1815dfe0 0x7f8c1815feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.353+0000 7f8c1cff7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c180d1c70 con 0x7f8c180d27f0 2024-03-06T07:31:35.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.353+0000 7f8c1cff7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c180b39f0 con 0x7f8c1815dfe0 2024-03-06T07:31:35.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.353+0000 7f8c1cff7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c180d1df0 con 0x7f8c180cf830 2024-03-06T07:31:35.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.353+0000 7f8c15d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8c180d27f0 0x7f8c181f39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.353+0000 7f8c15d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8c180d27f0 0x7f8c181f39b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36206/0 (socket says 172.21.15.83:36206) 2024-03-06T07:31:35.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.353+0000 7f8c15d9b640 1 -- 172.21.15.83:0/3534643940 learned_addr learned my addr 172.21.15.83:0/3534643940 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:35.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.353+0000 7f8c16d9d640 1 --2- 172.21.15.83:0/3534643940 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c1815dfe0 0x7f8c1815feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.353+0000 7f8c1659c640 1 --2- 172.21.15.83:0/3534643940 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8c180cf830 0x7f8c18155310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.354+0000 7f8c15d9b640 1 -- 172.21.15.83:0/3534643940 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8c180cf830 msgr2=0x7f8c18155310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.354+0000 7f8c15d9b640 1 --2- 172.21.15.83:0/3534643940 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8c180cf830 0x7f8c18155310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.354+0000 7f8c15d9b640 1 -- 172.21.15.83:0/3534643940 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c1815dfe0 msgr2=0x7f8c1815feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.354+0000 7f8c15d9b640 1 --2- 172.21.15.83:0/3534643940 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c1815dfe0 0x7f8c1815feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.354+0000 7f8c15d9b640 1 -- 172.21.15.83:0/3534643940 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c180caff0 con 0x7f8c180d27f0 2024-03-06T07:31:35.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.354+0000 7f8c16d9d640 1 --2- 172.21.15.83:0/3534643940 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c1815dfe0 0x7f8c1815feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:35.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.354+0000 7f8c15d9b640 1 --2- 172.21.15.83:0/3534643940 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8c180d27f0 0x7f8c181f39b0 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c000c00 tx=0x7f8c0c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:35.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.354+0000 7f8c06ffd640 1 -- 172.21.15.83:0/3534643940 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c0c03ad60 con 0x7f8c180d27f0 2024-03-06T07:31:35.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.354+0000 7f8c06ffd640 1 -- 172.21.15.83:0/3534643940 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c0c0443f0 con 0x7f8c180d27f0 2024-03-06T07:31:35.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.354+0000 7f8c1cff7640 1 -- 172.21.15.83:0/3534643940 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c180cb2d0 con 0x7f8c180d27f0 2024-03-06T07:31:35.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.354+0000 7f8c06ffd640 1 -- 172.21.15.83:0/3534643940 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c0c0447e0 con 0x7f8c180d27f0 2024-03-06T07:31:35.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.354+0000 7f8c1cff7640 1 -- 172.21.15.83:0/3534643940 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c180cb890 con 0x7f8c180d27f0 2024-03-06T07:31:35.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.355+0000 7f8c1cff7640 1 -- 172.21.15.83:0/3534643940 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c1804f330 con 0x7f8c180d27f0 2024-03-06T07:31:35.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.355+0000 7f8c06ffd640 1 -- 172.21.15.83:0/3534643940 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8c0c038020 con 0x7f8c180d27f0 2024-03-06T07:31:35.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.356+0000 7f8c06ffd640 1 --2- 172.21.15.83:0/3534643940 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8bf806b4d0 0x7f8bf806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.356+0000 7f8c1659c640 1 --2- 172.21.15.83:0/3534643940 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8bf806b4d0 0x7f8bf806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.356+0000 7f8c06ffd640 1 -- 172.21.15.83:0/3534643940 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8c0c0ac4f0 con 0x7f8c180d27f0 2024-03-06T07:31:35.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.357+0000 7f8c1659c640 1 --2- 172.21.15.83:0/3534643940 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8bf806b4d0 0x7f8bf806d990 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f8c180cecc0 tx=0x7f8c08008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:35.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.360+0000 7f8c06ffd640 1 -- 172.21.15.83:0/3534643940 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8c0c03f070 con 0x7f8c180d27f0 2024-03-06T07:31:35.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.567+0000 7f8c1cff7640 1 -- 172.21.15.83:0/3534643940 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c180cad20 con 0x7f8c180d27f0 2024-03-06T07:31:35.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.568+0000 7f8c06ffd640 1 -- 172.21.15.83:0/3534643940 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f8c0c07b3c0 con 0x7f8c180d27f0 2024-03-06T07:31:35.570 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:31:35.570 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:31:35.570 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:31:35.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.570+0000 7f8c1cff7640 1 -- 172.21.15.83:0/3534643940 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8bf806b4d0 msgr2=0x7f8bf806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.570+0000 7f8c1cff7640 1 --2- 172.21.15.83:0/3534643940 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8bf806b4d0 0x7f8bf806d990 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f8c180cecc0 tx=0x7f8c08008040 comp rx=0 tx=0).stop 2024-03-06T07:31:35.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.570+0000 7f8c1cff7640 1 -- 172.21.15.83:0/3534643940 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8c180d27f0 msgr2=0x7f8c181f39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.570+0000 7f8c1cff7640 1 --2- 172.21.15.83:0/3534643940 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8c180d27f0 0x7f8c181f39b0 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c000c00 tx=0x7f8c0c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.570+0000 7f8c1cff7640 1 -- 172.21.15.83:0/3534643940 shutdown_connections 2024-03-06T07:31:35.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.570+0000 7f8c1cff7640 1 --2- 172.21.15.83:0/3534643940 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8bf806b4d0 0x7f8bf806d990 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.570+0000 7f8c1cff7640 1 --2- 172.21.15.83:0/3534643940 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c1815dfe0 0x7f8c1815feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.570+0000 7f8c1cff7640 1 --2- 172.21.15.83:0/3534643940 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8c180d27f0 0x7f8c181f39b0 unknown :-1 s=CLOSED pgs=1879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.570+0000 7f8c1cff7640 1 --2- 172.21.15.83:0/3534643940 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8c180cf830 0x7f8c18155310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.570+0000 7f8c1cff7640 1 -- 172.21.15.83:0/3534643940 >> 172.21.15.83:0/3534643940 conn(0x7f8c180d3cd0 msgr2=0x7f8c180b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:35.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.570+0000 7f8c1cff7640 1 -- 172.21.15.83:0/3534643940 shutdown_connections 2024-03-06T07:31:35.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:35.570+0000 7f8c1cff7640 1 -- 172.21.15.83:0/3534643940 wait complete. 2024-03-06T07:31:35.783 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:31:26.591282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:31:35.784 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:31:36.785 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:36.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.885+0000 7fedb75f7640 1 Processor -- start 2024-03-06T07:31:36.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.885+0000 7fedb75f7640 1 -- start start 2024-03-06T07:31:36.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.886+0000 7fedb75f7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fedb013dc10 0x7fedb00b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.886+0000 7fedb75f7640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fedb015bf30 0x7fedb00b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.887+0000 7fedb75f7640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedb015f580 con 0x7fedb015e850 2024-03-06T07:31:36.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.887+0000 7fedb75f7640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedb00cf3f0 con 0x7fedb015bf30 2024-03-06T07:31:36.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.887+0000 7fedb75f7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedb015b210 con 0x7fedb013dc10 2024-03-06T07:31:36.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.887+0000 7fedb5393640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fedb013dc10 0x7fedb00b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.887+0000 7fedb4b92640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fedb015bf30 0x7fedb00b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.887+0000 7fedb5b94640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fedb015e850 0x7fedb015eca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:58892/0 (socket says 172.21.15.83:58892) 2024-03-06T07:31:36.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.887+0000 7fedb5b94640 1 -- 172.21.15.83:0/2592952429 learned_addr learned my addr 172.21.15.83:0/2592952429 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:31:36.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.887+0000 7fedb4b92640 1 -- 172.21.15.83:0/2592952429 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fedb013dc10 msgr2=0x7fedb00b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:36.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.887+0000 7fedb4b92640 1 --2- 172.21.15.83:0/2592952429 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fedb013dc10 0x7fedb00b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.887+0000 7fedb4b92640 1 -- 172.21.15.83:0/2592952429 >> v1:172.21.15.183:6789/0 conn(0x7fedb015e850 legacy=0x7fedb015eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.887+0000 7fedb4b92640 1 -- 172.21.15.83:0/2592952429 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedb015b450 con 0x7fedb015bf30 2024-03-06T07:31:36.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.887+0000 7feda3fff640 1 -- 172.21.15.83:0/2592952429 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 43943493 0 0) 0x7fedb015f580 con 0x7fedb015e850 2024-03-06T07:31:36.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.887+0000 7fedb5393640 1 --2- 172.21.15.83:0/2592952429 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fedb013dc10 0x7fedb00b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:36.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.888+0000 7fedb4b92640 1 --2- 172.21.15.83:0/2592952429 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fedb015bf30 0x7fedb00b2e90 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fedac0097a0 tx=0x7fedac02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa733e7a3e4c3ff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:36.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.888+0000 7feda3fff640 1 -- 172.21.15.83:0/2592952429 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedac0315a0 con 0x7fedb015bf30 2024-03-06T07:31:36.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.888+0000 7feda3fff640 1 -- 172.21.15.83:0/2592952429 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedac037070 con 0x7fedb015bf30 2024-03-06T07:31:36.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.888+0000 7feda3fff640 1 -- 172.21.15.83:0/2592952429 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedac031960 con 0x7fedb015bf30 2024-03-06T07:31:36.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.889+0000 7fedb75f7640 1 -- 172.21.15.83:0/2592952429 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fedb015bf30 msgr2=0x7fedb00b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.889+0000 7fedb75f7640 1 --2- 172.21.15.83:0/2592952429 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fedb015bf30 0x7fedb00b2e90 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fedac0097a0 tx=0x7fedac02b840 comp rx=0 tx=0).stop 2024-03-06T07:31:36.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.889+0000 7fedb75f7640 1 -- 172.21.15.83:0/2592952429 shutdown_connections 2024-03-06T07:31:36.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.889+0000 7fedb75f7640 1 --2- 172.21.15.83:0/2592952429 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fedb015bf30 0x7fedb00b2e90 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.889+0000 7fedb75f7640 1 --2- 172.21.15.83:0/2592952429 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fedb013dc10 0x7fedb00b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.889+0000 7fedb75f7640 1 -- 172.21.15.83:0/2592952429 >> 172.21.15.83:0/2592952429 conn(0x7fedb00cf8a0 msgr2=0x7fedb00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:36.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.889+0000 7fedb75f7640 1 -- 172.21.15.83:0/2592952429 shutdown_connections 2024-03-06T07:31:36.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.889+0000 7fedb75f7640 1 -- 172.21.15.83:0/2592952429 wait complete. 2024-03-06T07:31:36.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.890+0000 7fedb75f7640 1 Processor -- start 2024-03-06T07:31:36.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.890+0000 7fedb75f7640 1 -- start start 2024-03-06T07:31:36.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.891+0000 7fedb75f7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedb013dc10 0x7fedb012ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.891+0000 7fedb75f7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fedb015bf30 0x7fedb012e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.891+0000 7fedb75f7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fedb015e850 0x7fedb0159260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.891+0000 7fedb75f7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb00708d0 con 0x7fedb013dc10 2024-03-06T07:31:36.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.891+0000 7fedb75f7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb013dff0 con 0x7fedb015bf30 2024-03-06T07:31:36.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.891+0000 7fedb75f7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedb0070a50 con 0x7fedb015e850 2024-03-06T07:31:36.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.891+0000 7fedb5393640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedb013dc10 0x7fedb012ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.891+0000 7fedb5393640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedb013dc10 0x7fedb012ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52318/0 (socket says 172.21.15.83:52318) 2024-03-06T07:31:36.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.891+0000 7fedb5393640 1 -- 172.21.15.83:0/2757633261 learned_addr learned my addr 172.21.15.83:0/2757633261 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:36.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.891+0000 7fedb5b94640 1 --2- 172.21.15.83:0/2757633261 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fedb015e850 0x7fedb0159260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.891+0000 7fedb5393640 1 -- 172.21.15.83:0/2757633261 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fedb015e850 msgr2=0x7fedb0159260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.891+0000 7fedb5393640 1 --2- 172.21.15.83:0/2757633261 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fedb015e850 0x7fedb0159260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.891+0000 7fedb5393640 1 -- 172.21.15.83:0/2757633261 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fedb015bf30 msgr2=0x7fedb012e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.891+0000 7fedb5393640 1 --2- 172.21.15.83:0/2757633261 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fedb015bf30 0x7fedb012e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.891+0000 7fedb5393640 1 -- 172.21.15.83:0/2757633261 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedb012d3b0 con 0x7fedb013dc10 2024-03-06T07:31:36.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.891+0000 7fedb5393640 1 --2- 172.21.15.83:0/2757633261 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedb013dc10 0x7fedb012ce70 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7feda400ebe0 tx=0x7feda400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:36.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.892+0000 7feda1ffb640 1 -- 172.21.15.83:0/2757633261 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feda4017070 con 0x7fedb013dc10 2024-03-06T07:31:36.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.892+0000 7feda1ffb640 1 -- 172.21.15.83:0/2757633261 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feda400cc30 con 0x7fedb013dc10 2024-03-06T07:31:36.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.892+0000 7fedb75f7640 1 -- 172.21.15.83:0/2757633261 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedb012ec60 con 0x7fedb013dc10 2024-03-06T07:31:36.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.892+0000 7feda1ffb640 1 -- 172.21.15.83:0/2757633261 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feda4013470 con 0x7fedb013dc10 2024-03-06T07:31:36.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.892+0000 7fedb75f7640 1 -- 172.21.15.83:0/2757633261 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedb012d960 con 0x7fedb013dc10 2024-03-06T07:31:36.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.893+0000 7fedb75f7640 1 -- 172.21.15.83:0/2757633261 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedb004f330 con 0x7fedb013dc10 2024-03-06T07:31:36.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.894+0000 7feda1ffb640 1 -- 172.21.15.83:0/2757633261 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feda401a070 con 0x7fedb013dc10 2024-03-06T07:31:36.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.894+0000 7feda1ffb640 1 --2- 172.21.15.83:0/2757633261 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fed9406b4d0 0x7fed9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.894+0000 7feda1ffb640 1 -- 172.21.15.83:0/2757633261 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7feda408a280 con 0x7fedb013dc10 2024-03-06T07:31:36.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.894+0000 7fedb4b92640 1 --2- 172.21.15.83:0/2757633261 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fed9406b4d0 0x7fed9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.894+0000 7fedb4b92640 1 --2- 172.21.15.83:0/2757633261 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fed9406b4d0 0x7fed9406d990 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fedac036000 tx=0x7fedac030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:36.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:36.898+0000 7feda1ffb640 1 -- 172.21.15.83:0/2757633261 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feda4059040 con 0x7fedb013dc10 2024-03-06T07:31:37.098 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:37.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:37.109+0000 7fedb75f7640 1 -- 172.21.15.83:0/2757633261 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fedb0072430 con 0x7fedb013dc10 2024-03-06T07:31:37.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:37.109+0000 7feda1ffb640 1 -- 172.21.15.83:0/2757633261 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7feda4058940 con 0x7fedb013dc10 2024-03-06T07:31:37.111 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:31:37.111 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:31:37.112 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:31:37.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:37.111+0000 7fedb75f7640 1 -- 172.21.15.83:0/2757633261 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fed9406b4d0 msgr2=0x7fed9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:37.111+0000 7fedb75f7640 1 --2- 172.21.15.83:0/2757633261 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fed9406b4d0 0x7fed9406d990 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fedac036000 tx=0x7fedac030000 comp rx=0 tx=0).stop 2024-03-06T07:31:37.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:37.111+0000 7fedb75f7640 1 -- 172.21.15.83:0/2757633261 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedb013dc10 msgr2=0x7fedb012ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:37.111+0000 7fedb75f7640 1 --2- 172.21.15.83:0/2757633261 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedb013dc10 0x7fedb012ce70 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7feda400ebe0 tx=0x7feda400c520 comp rx=0 tx=0).stop 2024-03-06T07:31:37.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:37.111+0000 7fedb75f7640 1 -- 172.21.15.83:0/2757633261 shutdown_connections 2024-03-06T07:31:37.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:37.111+0000 7fedb75f7640 1 --2- 172.21.15.83:0/2757633261 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fed9406b4d0 0x7fed9406d990 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:37.111+0000 7fedb75f7640 1 --2- 172.21.15.83:0/2757633261 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fedb015e850 0x7fedb0159260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:37.111+0000 7fedb75f7640 1 --2- 172.21.15.83:0/2757633261 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fedb015bf30 0x7fedb012e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:37.111+0000 7fedb75f7640 1 --2- 172.21.15.83:0/2757633261 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedb013dc10 0x7fedb012ce70 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:37.111+0000 7fedb75f7640 1 -- 172.21.15.83:0/2757633261 >> 172.21.15.83:0/2757633261 conn(0x7fedb00cf8a0 msgr2=0x7fedb015cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:37.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:37.111+0000 7fedb75f7640 1 -- 172.21.15.83:0/2757633261 shutdown_connections 2024-03-06T07:31:37.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:37.111+0000 7fedb75f7640 1 -- 172.21.15.83:0/2757633261 wait complete. 2024-03-06T07:31:37.328 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:31:26.591282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:31:37.328 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:31:38.329 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:38.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.428+0000 7f71dc75e640 1 Processor -- start 2024-03-06T07:31:38.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.428+0000 7f71dc75e640 1 -- start start 2024-03-06T07:31:38.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.429+0000 7f71dc75e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f71d415fa10 0x7f71d40ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.429+0000 7f71dc75e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71d40cebe0 con 0x7f71d40d16e0 2024-03-06T07:31:38.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.429+0000 7f71dc75e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71d40ceec0 con 0x7f71d415bdd0 2024-03-06T07:31:38.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.429+0000 7f71dc75e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71d40cf1a0 con 0x7f71d415fa10 2024-03-06T07:31:38.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.429+0000 7f71dacfb640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f71d40d16e0 0x7f71d40d1b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45198/0 (socket says 172.21.15.83:45198) 2024-03-06T07:31:38.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.429+0000 7f71dacfb640 1 -- 172.21.15.83:0/3729912030 learned_addr learned my addr 172.21.15.83:0/3729912030 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:31:38.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.429+0000 7f71d94f8640 1 -- 172.21.15.83:0/3729912030 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3642914525 0 0) 0x7f71d40cebe0 con 0x7f71d40d16e0 2024-03-06T07:31:38.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.429+0000 7f71d9cf9640 1 --2- 172.21.15.83:0/3729912030 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f71d415fa10 0x7f71d40ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:38.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.429+0000 7f71d94f8640 1 -- 172.21.15.83:0/3729912030 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71c0003520 con 0x7f71d40d16e0 2024-03-06T07:31:38.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.430+0000 7f71d94f8640 1 -- 172.21.15.83:0/3729912030 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1832464088 0 0) 0x7f71c0003520 con 0x7f71d40d16e0 2024-03-06T07:31:38.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.430+0000 7f71d94f8640 1 -- 172.21.15.83:0/3729912030 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f71d415fa10 msgr2=0x7f71d40ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:38.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.430+0000 7f71d94f8640 1 --2- 172.21.15.83:0/3729912030 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f71d415fa10 0x7f71d40ce270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.430+0000 7f71d94f8640 1 -- 172.21.15.83:0/3729912030 >> v1:172.21.15.183:6789/0 conn(0x7f71d415bdd0 legacy=0x7f71d40c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.430+0000 7f71d94f8640 1 -- 172.21.15.83:0/3729912030 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d40cc8e0 con 0x7f71d40d16e0 2024-03-06T07:31:38.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.430+0000 7f71d94f8640 1 -- 172.21.15.83:0/3729912030 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f71cc0019f0 con 0x7f71d40d16e0 2024-03-06T07:31:38.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.430+0000 7f71d94f8640 1 -- 172.21.15.83:0/3729912030 >> v1:172.21.15.83:6789/0 conn(0x7f71d40d16e0 legacy=0x7f71d40d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.430+0000 7f71d94f8640 1 --2- 172.21.15.83:0/3729912030 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71c00039e0 0x7f71c0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.430+0000 7f71d94f8640 1 --2- 172.21.15.83:0/3729912030 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f71d415fa10 0x7f71c0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.430+0000 7f71d94f8640 1 --2- 172.21.15.83:0/3729912030 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f71c0004d40 0x7f71c00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.430+0000 7f71d94f8640 1 -- 172.21.15.83:0/3729912030 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d40cf1a0 con 0x7f71c00039e0 2024-03-06T07:31:38.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.430+0000 7f71d94f8640 1 -- 172.21.15.83:0/3729912030 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c000bb20 con 0x7f71c0004d40 2024-03-06T07:31:38.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.430+0000 7f71d94f8640 1 -- 172.21.15.83:0/3729912030 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71c000bd20 con 0x7f71d415fa10 2024-03-06T07:31:38.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.430+0000 7f71da4fa640 1 --2- 172.21.15.83:0/3729912030 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71c00039e0 0x7f71c0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:38.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.430+0000 7f71dacfb640 1 --2- 172.21.15.83:0/3729912030 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f71c0004d40 0x7f71c00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:38.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.430+0000 7f71d9cf9640 1 --2- 172.21.15.83:0/3729912030 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f71d415fa10 0x7f71c0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:38.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.430+0000 7f71da4fa640 1 -- 172.21.15.83:0/3729912030 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f71d415fa10 msgr2=0x7f71c0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:38.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.430+0000 7f71da4fa640 1 --2- 172.21.15.83:0/3729912030 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f71d415fa10 0x7f71c0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.430+0000 7f71da4fa640 1 -- 172.21.15.83:0/3729912030 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f71c0004d40 msgr2=0x7f71c00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:38.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.430+0000 7f71da4fa640 1 --2- 172.21.15.83:0/3729912030 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f71c0004d40 0x7f71c00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.430+0000 7f71da4fa640 1 -- 172.21.15.83:0/3729912030 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c0003520 con 0x7f71c00039e0 2024-03-06T07:31:38.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.430+0000 7f71d9cf9640 1 --2- 172.21.15.83:0/3729912030 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f71d415fa10 0x7f71c0004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:38.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.431+0000 7f71da4fa640 1 --2- 172.21.15.83:0/3729912030 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71c00039e0 0x7f71c0003e00 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f71c4001750 tx=0x7f71c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95e5bb7be35b0b25 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:38.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.431+0000 7f71d94f8640 1 -- 172.21.15.83:0/3729912030 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c4001370 con 0x7f71c00039e0 2024-03-06T07:31:38.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.431+0000 7f71d94f8640 1 -- 172.21.15.83:0/3729912030 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71c40052d0 con 0x7f71c00039e0 2024-03-06T07:31:38.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.431+0000 7f71d94f8640 1 -- 172.21.15.83:0/3729912030 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c4003830 con 0x7f71c00039e0 2024-03-06T07:31:38.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.432+0000 7f71dc75e640 1 -- 172.21.15.83:0/3729912030 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71c00039e0 msgr2=0x7f71c0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.432+0000 7f71dc75e640 1 --2- 172.21.15.83:0/3729912030 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71c00039e0 0x7f71c0003e00 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f71c4001750 tx=0x7f71c0005830 comp rx=0 tx=0).stop 2024-03-06T07:31:38.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.432+0000 7f71dacfb640 1 -- 172.21.15.83:0/3729912030 reap_dead start 2024-03-06T07:31:38.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.432+0000 7f71dc75e640 1 -- 172.21.15.83:0/3729912030 shutdown_connections 2024-03-06T07:31:38.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.432+0000 7f71dc75e640 1 -- 172.21.15.83:0/3729912030 >> 172.21.15.83:0/3729912030 conn(0x7f71d409a9f0 msgr2=0x7f71d409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:38.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.432+0000 7f71dc75e640 1 -- 172.21.15.83:0/3729912030 shutdown_connections 2024-03-06T07:31:38.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.432+0000 7f71dc75e640 1 -- 172.21.15.83:0/3729912030 wait complete. 2024-03-06T07:31:38.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.433+0000 7f71dc75e640 1 Processor -- start 2024-03-06T07:31:38.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.433+0000 7f71dc75e640 1 -- start start 2024-03-06T07:31:38.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.433+0000 7f71dc75e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f71c00039e0 0x7f71d4157b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.433+0000 7f71dc75e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f71d415c330 0x7f71d40ca180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.433+0000 7f71dc75e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71d4158200 0x7f71d40ca6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.434+0000 7f71dc75e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d40ce270 con 0x7f71d4158200 2024-03-06T07:31:38.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.434+0000 7f71dc75e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d415c1b0 con 0x7f71c00039e0 2024-03-06T07:31:38.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.434+0000 7f71dc75e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71d40ce3f0 con 0x7f71d415c330 2024-03-06T07:31:38.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.434+0000 7f71dacfb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71d4158200 0x7f71d40ca6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:38.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.434+0000 7f71dacfb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71d4158200 0x7f71d40ca6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52340/0 (socket says 172.21.15.83:52340) 2024-03-06T07:31:38.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.434+0000 7f71dacfb640 1 -- 172.21.15.83:0/2405217288 learned_addr learned my addr 172.21.15.83:0/2405217288 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:38.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.434+0000 7f71da4fa640 1 --2- 172.21.15.83:0/2405217288 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f71c00039e0 0x7f71d4157b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:38.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.434+0000 7f71d9cf9640 1 --2- 172.21.15.83:0/2405217288 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f71d415c330 0x7f71d40ca180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:38.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.434+0000 7f71dacfb640 1 -- 172.21.15.83:0/2405217288 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f71d415c330 msgr2=0x7f71d40ca180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.434+0000 7f71dacfb640 1 --2- 172.21.15.83:0/2405217288 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f71d415c330 0x7f71d40ca180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.434+0000 7f71dacfb640 1 -- 172.21.15.83:0/2405217288 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f71c00039e0 msgr2=0x7f71d4157b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.434+0000 7f71dacfb640 1 --2- 172.21.15.83:0/2405217288 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f71c00039e0 0x7f71d4157b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.434+0000 7f71dacfb640 1 -- 172.21.15.83:0/2405217288 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d415c830 con 0x7f71d4158200 2024-03-06T07:31:38.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.434+0000 7f71dacfb640 1 --2- 172.21.15.83:0/2405217288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71d4158200 0x7f71d40ca6c0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f71d40b3450 tx=0x7f71cc007b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:38.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.434+0000 7f71caffd640 1 -- 172.21.15.83:0/2405217288 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71cc00abc0 con 0x7f71d4158200 2024-03-06T07:31:38.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.434+0000 7f71caffd640 1 -- 172.21.15.83:0/2405217288 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71cc018070 con 0x7f71d4158200 2024-03-06T07:31:38.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.434+0000 7f71dc75e640 1 -- 172.21.15.83:0/2405217288 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71d415cb10 con 0x7f71d4158200 2024-03-06T07:31:38.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.434+0000 7f71caffd640 1 -- 172.21.15.83:0/2405217288 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71cc0133f0 con 0x7f71d4158200 2024-03-06T07:31:38.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.434+0000 7f71dc75e640 1 -- 172.21.15.83:0/2405217288 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71d415d0d0 con 0x7f71d4158200 2024-03-06T07:31:38.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.435+0000 7f71dc75e640 1 -- 172.21.15.83:0/2405217288 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71d404f330 con 0x7f71d4158200 2024-03-06T07:31:38.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.436+0000 7f71caffd640 1 -- 172.21.15.83:0/2405217288 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f71cc013590 con 0x7f71d4158200 2024-03-06T07:31:38.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.436+0000 7f71caffd640 1 --2- 172.21.15.83:0/2405217288 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f71b806b4d0 0x7f71b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.436+0000 7f71caffd640 1 -- 172.21.15.83:0/2405217288 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f71cc08a920 con 0x7f71d4158200 2024-03-06T07:31:38.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.436+0000 7f71da4fa640 1 --2- 172.21.15.83:0/2405217288 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f71b806b4d0 0x7f71b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:38.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.437+0000 7f71da4fa640 1 --2- 172.21.15.83:0/2405217288 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f71b806b4d0 0x7f71b806d990 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f71c4033000 tx=0x7f71c4009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:38.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.440+0000 7f71caffd640 1 -- 172.21.15.83:0/2405217288 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f71cc059760 con 0x7f71d4158200 2024-03-06T07:31:38.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.674+0000 7f71dc75e640 1 -- 172.21.15.83:0/2405217288 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71d40b7a40 con 0x7f71d4158200 2024-03-06T07:31:38.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.675+0000 7f71caffd640 1 -- 172.21.15.83:0/2405217288 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f71cc059060 con 0x7f71d4158200 2024-03-06T07:31:38.677 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:31:38.677 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:31:38.677 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:31:38.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.677+0000 7f71dc75e640 1 -- 172.21.15.83:0/2405217288 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f71b806b4d0 msgr2=0x7f71b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.677+0000 7f71dc75e640 1 --2- 172.21.15.83:0/2405217288 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f71b806b4d0 0x7f71b806d990 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f71c4033000 tx=0x7f71c4009030 comp rx=0 tx=0).stop 2024-03-06T07:31:38.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.677+0000 7f71dc75e640 1 -- 172.21.15.83:0/2405217288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71d4158200 msgr2=0x7f71d40ca6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.677+0000 7f71dc75e640 1 --2- 172.21.15.83:0/2405217288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71d4158200 0x7f71d40ca6c0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f71d40b3450 tx=0x7f71cc007b20 comp rx=0 tx=0).stop 2024-03-06T07:31:38.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.677+0000 7f71dc75e640 1 -- 172.21.15.83:0/2405217288 shutdown_connections 2024-03-06T07:31:38.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.677+0000 7f71dc75e640 1 --2- 172.21.15.83:0/2405217288 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f71b806b4d0 0x7f71b806d990 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.677+0000 7f71dc75e640 1 --2- 172.21.15.83:0/2405217288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71d4158200 0x7f71d40ca6c0 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.677+0000 7f71dc75e640 1 --2- 172.21.15.83:0/2405217288 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f71d415c330 0x7f71d40ca180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.677+0000 7f71dc75e640 1 --2- 172.21.15.83:0/2405217288 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f71c00039e0 0x7f71d4157b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.677+0000 7f71dc75e640 1 -- 172.21.15.83:0/2405217288 >> 172.21.15.83:0/2405217288 conn(0x7f71d409a9f0 msgr2=0x7f71d406ffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:38.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.677+0000 7f71dc75e640 1 -- 172.21.15.83:0/2405217288 shutdown_connections 2024-03-06T07:31:38.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:38.677+0000 7f71dc75e640 1 -- 172.21.15.83:0/2405217288 wait complete. 2024-03-06T07:31:38.890 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:31:26.591282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:31:38.890 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:31:39.100 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:39.891 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:39.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.993+0000 7f7318de2640 1 Processor -- start 2024-03-06T07:31:39.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.994+0000 7f7318de2640 1 -- start start 2024-03-06T07:31:39.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.995+0000 7f7318de2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f731415c320 0x7f73140d06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.995+0000 7f7318de2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f73140b40c0 0x7f73140b1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.995+0000 7f7318de2640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73140d0020 con 0x7f73140b2120 2024-03-06T07:31:39.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.995+0000 7f7318de2640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73140b1000 con 0x7f73140b40c0 2024-03-06T07:31:39.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.995+0000 7f7318de2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73140b0d20 con 0x7f731415c320 2024-03-06T07:31:39.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.995+0000 7f7311d9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f73140b40c0 0x7f73140b1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:39.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.995+0000 7f7311d9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f73140b40c0 0x7f73140b1be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52350/0 (socket says 172.21.15.83:52350) 2024-03-06T07:31:39.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.995+0000 7f7311d9b640 1 -- 172.21.15.83:0/1562191581 learned_addr learned my addr 172.21.15.83:0/1562191581 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:39.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.995+0000 7f7311d9b640 1 -- 172.21.15.83:0/1562191581 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f731415c320 msgr2=0x7f73140d06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:39.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.995+0000 7f731259c640 1 --2- 172.21.15.83:0/1562191581 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f731415c320 0x7f73140d06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:39.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.996+0000 7f7311d9b640 1 --2- 172.21.15.83:0/1562191581 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f731415c320 0x7f73140d06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.996+0000 7f7311d9b640 1 -- 172.21.15.83:0/1562191581 >> v1:172.21.15.183:6789/0 conn(0x7f73140b2120 legacy=0x7f73140ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:39.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.996+0000 7f7311d9b640 1 -- 172.21.15.83:0/1562191581 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73140cfa60 con 0x7f73140b40c0 2024-03-06T07:31:39.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.996+0000 7f7311d9b640 1 --2- 172.21.15.83:0/1562191581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73140b40c0 0x7f73140b1be0 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f7308009870 tx=0x7f730802b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51f98dd7e5b9e31f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:39.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.996+0000 7f731159a640 1 -- 172.21.15.83:0/1562191581 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7308038070 con 0x7f73140b40c0 2024-03-06T07:31:39.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.996+0000 7f731159a640 1 -- 172.21.15.83:0/1562191581 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7308031740 con 0x7f73140b40c0 2024-03-06T07:31:39.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.997+0000 7f731159a640 1 -- 172.21.15.83:0/1562191581 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730802f050 con 0x7f73140b40c0 2024-03-06T07:31:40.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.997+0000 7f7318de2640 1 -- 172.21.15.83:0/1562191581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73140b40c0 msgr2=0x7f73140b1be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.997+0000 7f7318de2640 1 --2- 172.21.15.83:0/1562191581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73140b40c0 0x7f73140b1be0 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f7308009870 tx=0x7f730802b7e0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.997+0000 7f7318de2640 1 -- 172.21.15.83:0/1562191581 shutdown_connections 2024-03-06T07:31:40.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.997+0000 7f7318de2640 1 --2- 172.21.15.83:0/1562191581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73140b40c0 0x7f73140b1be0 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.997+0000 7f7318de2640 1 --2- 172.21.15.83:0/1562191581 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f731415c320 0x7f73140d06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.997+0000 7f7318de2640 1 -- 172.21.15.83:0/1562191581 >> 172.21.15.83:0/1562191581 conn(0x7f73140b35a0 msgr2=0x7f73140b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:40.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.997+0000 7f7318de2640 1 -- 172.21.15.83:0/1562191581 shutdown_connections 2024-03-06T07:31:40.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.997+0000 7f7318de2640 1 -- 172.21.15.83:0/1562191581 wait complete. 2024-03-06T07:31:40.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.998+0000 7f7318de2640 1 Processor -- start 2024-03-06T07:31:40.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.998+0000 7f7318de2640 1 -- start start 2024-03-06T07:31:40.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.999+0000 7f7318de2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f73140b2120 0x7f73140c9a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.999+0000 7f7318de2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73140b40c0 0x7f73140c6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.999+0000 7f7318de2640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f731415c320 0x7f73140afc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.999+0000 7f7318de2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73140cda00 con 0x7f73140b40c0 2024-03-06T07:31:40.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.999+0000 7f7318de2640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73140cd880 con 0x7f731415c320 2024-03-06T07:31:40.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.999+0000 7f7318de2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73140b25b0 con 0x7f73140b2120 2024-03-06T07:31:40.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.999+0000 7f7311d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73140b40c0 0x7f73140c6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.999+0000 7f7311d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73140b40c0 0x7f73140c6a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52360/0 (socket says 172.21.15.83:52360) 2024-03-06T07:31:40.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.999+0000 7f7311d9b640 1 -- 172.21.15.83:0/673018879 learned_addr learned my addr 172.21.15.83:0/673018879 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:40.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.999+0000 7f731259c640 1 --2- 172.21.15.83:0/673018879 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f73140b2120 0x7f73140c9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.999+0000 7f7312d9d640 1 --2- 172.21.15.83:0/673018879 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f731415c320 0x7f73140afc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.999+0000 7f7311d9b640 1 -- 172.21.15.83:0/673018879 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f73140b2120 msgr2=0x7f73140c9a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.999+0000 7f7311d9b640 1 --2- 172.21.15.83:0/673018879 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f73140b2120 0x7f73140c9a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.999+0000 7f7311d9b640 1 -- 172.21.15.83:0/673018879 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f731415c320 msgr2=0x7f73140afc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.999+0000 7f7311d9b640 1 --2- 172.21.15.83:0/673018879 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f731415c320 0x7f73140afc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:39.999+0000 7f7311d9b640 1 -- 172.21.15.83:0/673018879 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73140ca0f0 con 0x7f73140b40c0 2024-03-06T07:31:40.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:40.000+0000 7f7311d9b640 1 --2- 172.21.15.83:0/673018879 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73140b40c0 0x7f73140c6a30 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f7308009870 tx=0x7f73080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:40.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:40.000+0000 7f7302ffd640 1 -- 172.21.15.83:0/673018879 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7308043070 con 0x7f73140b40c0 2024-03-06T07:31:40.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:40.000+0000 7f7302ffd640 1 -- 172.21.15.83:0/673018879 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7308033c10 con 0x7f73140b40c0 2024-03-06T07:31:40.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:40.000+0000 7f7318de2640 1 -- 172.21.15.83:0/673018879 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73140ca3d0 con 0x7f73140b40c0 2024-03-06T07:31:40.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:40.000+0000 7f7302ffd640 1 -- 172.21.15.83:0/673018879 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7308038040 con 0x7f73140b40c0 2024-03-06T07:31:40.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:40.000+0000 7f7318de2640 1 -- 172.21.15.83:0/673018879 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73140ca990 con 0x7f73140b40c0 2024-03-06T07:31:40.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:40.001+0000 7f7318de2640 1 -- 172.21.15.83:0/673018879 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f731404f330 con 0x7f73140b40c0 2024-03-06T07:31:40.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:40.001+0000 7f7302ffd640 1 -- 172.21.15.83:0/673018879 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7308046070 con 0x7f73140b40c0 2024-03-06T07:31:40.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:40.002+0000 7f7302ffd640 1 --2- 172.21.15.83:0/673018879 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f72f406b4d0 0x7f72f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:40.002+0000 7f7302ffd640 1 -- 172.21.15.83:0/673018879 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f73080ac060 con 0x7f73140b40c0 2024-03-06T07:31:40.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:40.002+0000 7f731259c640 1 --2- 172.21.15.83:0/673018879 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f72f406b4d0 0x7f72f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:40.002+0000 7f731259c640 1 --2- 172.21.15.83:0/673018879 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f72f406b4d0 0x7f72f406d990 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f73040070b0 tx=0x7f7304007040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:40.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:40.006+0000 7f7302ffd640 1 -- 172.21.15.83:0/673018879 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f730804d030 con 0x7f73140b40c0 2024-03-06T07:31:40.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:40.219+0000 7f7318de2640 1 -- 172.21.15.83:0/673018879 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73140cc780 con 0x7f73140b40c0 2024-03-06T07:31:40.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:40.220+0000 7f7302ffd640 1 -- 172.21.15.83:0/673018879 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f730807af30 con 0x7f73140b40c0 2024-03-06T07:31:40.222 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:31:40.222 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:31:40.222 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:31:40.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:40.222+0000 7f7318de2640 1 -- 172.21.15.83:0/673018879 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f72f406b4d0 msgr2=0x7f72f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:40.222+0000 7f7318de2640 1 --2- 172.21.15.83:0/673018879 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f72f406b4d0 0x7f72f406d990 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f73040070b0 tx=0x7f7304007040 comp rx=0 tx=0).stop 2024-03-06T07:31:40.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:40.222+0000 7f7318de2640 1 -- 172.21.15.83:0/673018879 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73140b40c0 msgr2=0x7f73140c6a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:40.222+0000 7f7318de2640 1 --2- 172.21.15.83:0/673018879 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73140b40c0 0x7f73140c6a30 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f7308009870 tx=0x7f73080046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:40.222+0000 7f7318de2640 1 -- 172.21.15.83:0/673018879 shutdown_connections 2024-03-06T07:31:40.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:40.222+0000 7f7318de2640 1 --2- 172.21.15.83:0/673018879 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f72f406b4d0 0x7f72f406d990 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:40.222+0000 7f7318de2640 1 --2- 172.21.15.83:0/673018879 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f731415c320 0x7f73140afc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:40.222+0000 7f7318de2640 1 --2- 172.21.15.83:0/673018879 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73140b40c0 0x7f73140c6a30 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:40.222+0000 7f7318de2640 1 --2- 172.21.15.83:0/673018879 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f73140b2120 0x7f73140c9a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:40.222+0000 7f7318de2640 1 -- 172.21.15.83:0/673018879 >> 172.21.15.83:0/673018879 conn(0x7f73140b35a0 msgr2=0x7f73140ccc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:40.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:40.222+0000 7f7318de2640 1 -- 172.21.15.83:0/673018879 shutdown_connections 2024-03-06T07:31:40.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:40.222+0000 7f7318de2640 1 -- 172.21.15.83:0/673018879 wait complete. 2024-03-06T07:31:40.437 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:31:26.591282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:31:40.438 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:31:41.103 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:41.439 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:41.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.542+0000 7fdc2466b640 1 Processor -- start 2024-03-06T07:31:41.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.542+0000 7fdc2466b640 1 -- start start 2024-03-06T07:31:41.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.543+0000 7fdc2466b640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fdc1c0d2710 0x7fdc1c0b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.543+0000 7fdc2466b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc1c0b3430 con 0x7fdc1c0d22b0 2024-03-06T07:31:41.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.543+0000 7fdc2466b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc1c0b3710 con 0x7fdc1c15dfe0 2024-03-06T07:31:41.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.543+0000 7fdc2466b640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc1c0b31e0 con 0x7fdc1c0d2710 2024-03-06T07:31:41.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.543+0000 7fdc21c06640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fdc1c0d22b0 0x7fdc1c0b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45204/0 (socket says 172.21.15.83:45204) 2024-03-06T07:31:41.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.543+0000 7fdc21c06640 1 -- 172.21.15.83:0/105870580 learned_addr learned my addr 172.21.15.83:0/105870580 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:31:41.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.544+0000 7fdc22c08640 1 --2- 172.21.15.83:0/105870580 >> v2:172.21.15.183:3300/0 conn(0x7fdc1c0d2710 0x7fdc1c0b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.544+0000 7fdc21405640 1 -- 172.21.15.83:0/105870580 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2462528583 0 0) 0x7fdc1c0b3430 con 0x7fdc1c0d22b0 2024-03-06T07:31:41.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.544+0000 7fdc21405640 1 -- 172.21.15.83:0/105870580 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc08003520 con 0x7fdc1c0d22b0 2024-03-06T07:31:41.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.544+0000 7fdc21405640 1 -- 172.21.15.83:0/105870580 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4218459218 0 0) 0x7fdc1c0b3710 con 0x7fdc1c15dfe0 2024-03-06T07:31:41.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.544+0000 7fdc21405640 1 -- 172.21.15.83:0/105870580 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc1c0b3430 con 0x7fdc1c15dfe0 2024-03-06T07:31:41.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.544+0000 7fdc21405640 1 -- 172.21.15.83:0/105870580 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2778041370 0 0) 0x7fdc08003520 con 0x7fdc1c0d22b0 2024-03-06T07:31:41.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.544+0000 7fdc21405640 1 -- 172.21.15.83:0/105870580 >> v2:172.21.15.183:3300/0 conn(0x7fdc1c0d2710 msgr2=0x7fdc1c0b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:41.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.544+0000 7fdc21405640 1 --2- 172.21.15.83:0/105870580 >> v2:172.21.15.183:3300/0 conn(0x7fdc1c0d2710 0x7fdc1c0b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.544+0000 7fdc21405640 1 -- 172.21.15.83:0/105870580 >> v1:172.21.15.183:6789/0 conn(0x7fdc1c15dfe0 legacy=0x7fdc1c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.544+0000 7fdc21405640 1 -- 172.21.15.83:0/105870580 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc1c0b39f0 con 0x7fdc1c0d22b0 2024-03-06T07:31:41.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.544+0000 7fdc21405640 1 -- 172.21.15.83:0/105870580 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7fdc18001670 con 0x7fdc1c0d22b0 2024-03-06T07:31:41.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.544+0000 7fdc21405640 1 -- 172.21.15.83:0/105870580 >> v1:172.21.15.83:6789/0 conn(0x7fdc1c0d22b0 legacy=0x7fdc1c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.544+0000 7fdc21405640 1 --2- 172.21.15.83:0/105870580 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdc08003b70 0x7fdc08003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.544+0000 7fdc21405640 1 --2- 172.21.15.83:0/105870580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc08004800 0x7fdc08004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.544+0000 7fdc21405640 1 --2- 172.21.15.83:0/105870580 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdc080051c0 0x7fdc0800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.544+0000 7fdc21405640 1 -- 172.21.15.83:0/105870580 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1c0b31e0 con 0x7fdc08004800 2024-03-06T07:31:41.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.544+0000 7fdc21405640 1 -- 172.21.15.83:0/105870580 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0800c060 con 0x7fdc080051c0 2024-03-06T07:31:41.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.544+0000 7fdc21405640 1 -- 172.21.15.83:0/105870580 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0800c260 con 0x7fdc08003b70 2024-03-06T07:31:41.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.544+0000 7fdc21c06640 1 --2- 172.21.15.83:0/105870580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc08004800 0x7fdc08004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.544+0000 7fdc22c08640 1 --2- 172.21.15.83:0/105870580 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdc080051c0 0x7fdc0800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.544+0000 7fdc22407640 1 --2- 172.21.15.83:0/105870580 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdc08003b70 0x7fdc08003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.544+0000 7fdc21c06640 1 -- 172.21.15.83:0/105870580 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdc08003b70 msgr2=0x7fdc08003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:41.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.544+0000 7fdc21c06640 1 --2- 172.21.15.83:0/105870580 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdc08003b70 0x7fdc08003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.544+0000 7fdc21c06640 1 -- 172.21.15.83:0/105870580 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdc080051c0 msgr2=0x7fdc0800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:41.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.544+0000 7fdc21c06640 1 --2- 172.21.15.83:0/105870580 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdc080051c0 0x7fdc0800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.544+0000 7fdc22c08640 1 -- 172.21.15.83:0/105870580 reap_dead start 2024-03-06T07:31:41.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.544+0000 7fdc21c06640 1 -- 172.21.15.83:0/105870580 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc08003520 con 0x7fdc08004800 2024-03-06T07:31:41.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.545+0000 7fdc21c06640 1 --2- 172.21.15.83:0/105870580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc08004800 0x7fdc08004c80 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7fdc1800c190 tx=0x7fdc08005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4280b33b151f115 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:41.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.545+0000 7fdc21405640 1 -- 172.21.15.83:0/105870580 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc18007cc0 con 0x7fdc08004800 2024-03-06T07:31:41.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.545+0000 7fdc21405640 1 -- 172.21.15.83:0/105870580 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc18030070 con 0x7fdc08004800 2024-03-06T07:31:41.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.545+0000 7fdc21405640 1 -- 172.21.15.83:0/105870580 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc18005d70 con 0x7fdc08004800 2024-03-06T07:31:41.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.546+0000 7fdc2466b640 1 -- 172.21.15.83:0/105870580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc08004800 msgr2=0x7fdc08004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.546+0000 7fdc2466b640 1 --2- 172.21.15.83:0/105870580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc08004800 0x7fdc08004c80 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7fdc1800c190 tx=0x7fdc08005830 comp rx=0 tx=0).stop 2024-03-06T07:31:41.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.546+0000 7fdc2466b640 1 -- 172.21.15.83:0/105870580 shutdown_connections 2024-03-06T07:31:41.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.546+0000 7fdc2466b640 1 --2- 172.21.15.83:0/105870580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc08004800 0x7fdc08004c80 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.546+0000 7fdc2466b640 1 -- 172.21.15.83:0/105870580 >> 172.21.15.83:0/105870580 conn(0x7fdc1c0d3cd0 msgr2=0x7fdc1c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:41.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.546+0000 7fdc2466b640 1 -- 172.21.15.83:0/105870580 shutdown_connections 2024-03-06T07:31:41.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.546+0000 7fdc2466b640 1 -- 172.21.15.83:0/105870580 wait complete. 2024-03-06T07:31:41.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.547+0000 7fdc2466b640 1 Processor -- start 2024-03-06T07:31:41.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.547+0000 7fdc2466b640 1 -- start start 2024-03-06T07:31:41.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.548+0000 7fdc2466b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc08004800 0x7fdc1c159d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.548+0000 7fdc2466b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdc1c0b8e30 0x7fdc1c0b92b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.548+0000 7fdc2466b640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdc1c0b6b00 0x7fdc1c0b6f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.548+0000 7fdc2466b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1c0d1c70 con 0x7fdc08004800 2024-03-06T07:31:41.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.548+0000 7fdc2466b640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1c0b3d60 con 0x7fdc1c0b6b00 2024-03-06T07:31:41.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.548+0000 7fdc2466b640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1c0d1df0 con 0x7fdc1c0b8e30 2024-03-06T07:31:41.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.548+0000 7fdc22407640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc08004800 0x7fdc1c159d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.548+0000 7fdc22407640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc08004800 0x7fdc1c159d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52384/0 (socket says 172.21.15.83:52384) 2024-03-06T07:31:41.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.548+0000 7fdc22407640 1 -- 172.21.15.83:0/2005424234 learned_addr learned my addr 172.21.15.83:0/2005424234 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:41.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.548+0000 7fdc22407640 1 -- 172.21.15.83:0/2005424234 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdc1c0b8e30 msgr2=0x7fdc1c0b92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.548+0000 7fdc22407640 1 --2- 172.21.15.83:0/2005424234 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdc1c0b8e30 0x7fdc1c0b92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.548+0000 7fdc22407640 1 -- 172.21.15.83:0/2005424234 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdc1c0b6b00 msgr2=0x7fdc1c0b6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.548+0000 7fdc22407640 1 --2- 172.21.15.83:0/2005424234 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdc1c0b6b00 0x7fdc1c0b6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.548+0000 7fdc22407640 1 -- 172.21.15.83:0/2005424234 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc1c0b7830 con 0x7fdc08004800 2024-03-06T07:31:41.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.548+0000 7fdc22407640 1 --2- 172.21.15.83:0/2005424234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc08004800 0x7fdc1c159d80 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7fdc1000e560 tx=0x7fdc1000ea20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:41.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.549+0000 7fdc0effd640 1 -- 172.21.15.83:0/2005424234 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc100052e0 con 0x7fdc08004800 2024-03-06T07:31:41.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.549+0000 7fdc0effd640 1 -- 172.21.15.83:0/2005424234 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc10018070 con 0x7fdc08004800 2024-03-06T07:31:41.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.549+0000 7fdc2466b640 1 -- 172.21.15.83:0/2005424234 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc1c0b7b10 con 0x7fdc08004800 2024-03-06T07:31:41.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.549+0000 7fdc0effd640 1 -- 172.21.15.83:0/2005424234 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc10003830 con 0x7fdc08004800 2024-03-06T07:31:41.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.549+0000 7fdc2466b640 1 -- 172.21.15.83:0/2005424234 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc1c0b80d0 con 0x7fdc08004800 2024-03-06T07:31:41.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.550+0000 7fdc2466b640 1 -- 172.21.15.83:0/2005424234 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc1c04f330 con 0x7fdc08004800 2024-03-06T07:31:41.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.551+0000 7fdc0effd640 1 -- 172.21.15.83:0/2005424234 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdc100039d0 con 0x7fdc08004800 2024-03-06T07:31:41.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.551+0000 7fdc0effd640 1 --2- 172.21.15.83:0/2005424234 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdc0006b4d0 0x7fdc0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.551+0000 7fdc21c06640 1 --2- 172.21.15.83:0/2005424234 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdc0006b4d0 0x7fdc0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.551+0000 7fdc0effd640 1 -- 172.21.15.83:0/2005424234 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fdc10089fa0 con 0x7fdc08004800 2024-03-06T07:31:41.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.552+0000 7fdc21c06640 1 --2- 172.21.15.83:0/2005424234 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdc0006b4d0 0x7fdc0006d990 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fdc18034000 tx=0x7fdc1802e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:41.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.556+0000 7fdc0effd640 1 -- 172.21.15.83:0/2005424234 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdc10058de0 con 0x7fdc08004800 2024-03-06T07:31:41.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.829+0000 7fdc2466b640 1 -- 172.21.15.83:0/2005424234 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc1c15d590 con 0x7fdc08004800 2024-03-06T07:31:41.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.830+0000 7fdc0effd640 1 -- 172.21.15.83:0/2005424234 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7fdc100586e0 con 0x7fdc08004800 2024-03-06T07:31:41.832 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:31:41.832 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:31:41.832 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:31:41.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.833+0000 7fdc2466b640 1 -- 172.21.15.83:0/2005424234 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdc0006b4d0 msgr2=0x7fdc0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.833+0000 7fdc2466b640 1 --2- 172.21.15.83:0/2005424234 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdc0006b4d0 0x7fdc0006d990 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fdc18034000 tx=0x7fdc1802e000 comp rx=0 tx=0).stop 2024-03-06T07:31:41.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.833+0000 7fdc2466b640 1 -- 172.21.15.83:0/2005424234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc08004800 msgr2=0x7fdc1c159d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.833+0000 7fdc2466b640 1 --2- 172.21.15.83:0/2005424234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc08004800 0x7fdc1c159d80 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7fdc1000e560 tx=0x7fdc1000ea20 comp rx=0 tx=0).stop 2024-03-06T07:31:41.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.833+0000 7fdc2466b640 1 -- 172.21.15.83:0/2005424234 shutdown_connections 2024-03-06T07:31:41.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.833+0000 7fdc2466b640 1 --2- 172.21.15.83:0/2005424234 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fdc0006b4d0 0x7fdc0006d990 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.833+0000 7fdc2466b640 1 --2- 172.21.15.83:0/2005424234 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdc1c0b6b00 0x7fdc1c0b6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.833+0000 7fdc2466b640 1 --2- 172.21.15.83:0/2005424234 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdc1c0b8e30 0x7fdc1c0b92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.833+0000 7fdc2466b640 1 --2- 172.21.15.83:0/2005424234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc08004800 0x7fdc1c159d80 unknown :-1 s=CLOSED pgs=1888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.833+0000 7fdc2466b640 1 -- 172.21.15.83:0/2005424234 >> 172.21.15.83:0/2005424234 conn(0x7fdc1c0d3cd0 msgr2=0x7fdc1c0b40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:41.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.833+0000 7fdc2466b640 1 -- 172.21.15.83:0/2005424234 shutdown_connections 2024-03-06T07:31:41.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:41.833+0000 7fdc2466b640 1 -- 172.21.15.83:0/2005424234 wait complete. 2024-03-06T07:31:42.047 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:31:26.591282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:31:42.047 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:31:43.048 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:43.104 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:43.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.148+0000 7f5eba516640 1 Processor -- start 2024-03-06T07:31:43.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.148+0000 7f5eba516640 1 -- start start 2024-03-06T07:31:43.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.149+0000 7f5eba516640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5eb415bb20 0x7f5eb40d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:43.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.149+0000 7f5eba516640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5eb40cfff0 0x7f5eb40d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:43.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.149+0000 7f5eba516640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb4161980 con 0x7f5eb40ce4b0 2024-03-06T07:31:43.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.149+0000 7f5eba516640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5eb4161c60 con 0x7f5eb40cfff0 2024-03-06T07:31:43.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.149+0000 7f5eba516640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb4161f40 con 0x7f5eb415bb20 2024-03-06T07:31:43.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.149+0000 7f5eb37fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5eb40cfff0 0x7f5eb40d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:43.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.149+0000 7f5eb8ab3640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f5eb40ce4b0 0x7f5eb40ce940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45220/0 (socket says 172.21.15.83:45220) 2024-03-06T07:31:43.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.149+0000 7f5eb8ab3640 1 -- 172.21.15.83:0/3279200286 learned_addr learned my addr 172.21.15.83:0/3279200286 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:31:43.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.149+0000 7f5eb3fff640 1 --2- 172.21.15.83:0/3279200286 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5eb415bb20 0x7f5eb40d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:43.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.149+0000 7f5eb37fe640 1 -- 172.21.15.83:0/3279200286 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5eb415bb20 msgr2=0x7f5eb40d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:43.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.149+0000 7f5eb37fe640 1 --2- 172.21.15.83:0/3279200286 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5eb415bb20 0x7f5eb40d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.149+0000 7f5eb37fe640 1 -- 172.21.15.83:0/3279200286 >> v1:172.21.15.83:6789/0 conn(0x7f5eb40ce4b0 legacy=0x7f5eb40ce940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:43.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.149+0000 7f5eb2ffd640 1 -- 172.21.15.83:0/3279200286 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1096646635 0 0) 0x7f5eb4161980 con 0x7f5eb40ce4b0 2024-03-06T07:31:43.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.149+0000 7f5eb37fe640 1 -- 172.21.15.83:0/3279200286 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb4162220 con 0x7f5eb40cfff0 2024-03-06T07:31:43.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.149+0000 7f5eb3fff640 1 --2- 172.21.15.83:0/3279200286 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5eb415bb20 0x7f5eb40d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:43.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.150+0000 7f5eb37fe640 1 --2- 172.21.15.83:0/3279200286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5eb40cfff0 0x7f5eb40d0470 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f5ea40097a0 tx=0x7f5ea402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7994dc5751ca3cc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:43.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.150+0000 7f5eb2ffd640 1 -- 172.21.15.83:0/3279200286 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea4035020 con 0x7f5eb40cfff0 2024-03-06T07:31:43.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.150+0000 7f5eb2ffd640 1 -- 172.21.15.83:0/3279200286 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ea4031830 con 0x7f5eb40cfff0 2024-03-06T07:31:43.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.150+0000 7f5eb2ffd640 1 -- 172.21.15.83:0/3279200286 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea4031c00 con 0x7f5eb40cfff0 2024-03-06T07:31:43.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.151+0000 7f5eba516640 1 -- 172.21.15.83:0/3279200286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5eb40cfff0 msgr2=0x7f5eb40d0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:43.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.151+0000 7f5eba516640 1 --2- 172.21.15.83:0/3279200286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5eb40cfff0 0x7f5eb40d0470 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f5ea40097a0 tx=0x7f5ea402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.151+0000 7f5eba516640 1 -- 172.21.15.83:0/3279200286 shutdown_connections 2024-03-06T07:31:43.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.151+0000 7f5eba516640 1 --2- 172.21.15.83:0/3279200286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5eb40cfff0 0x7f5eb40d0470 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.151+0000 7f5eba516640 1 --2- 172.21.15.83:0/3279200286 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5eb415bb20 0x7f5eb40d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.151+0000 7f5eba516640 1 -- 172.21.15.83:0/3279200286 >> 172.21.15.83:0/3279200286 conn(0x7f5eb409a9f0 msgr2=0x7f5eb409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:43.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.151+0000 7f5eba516640 1 -- 172.21.15.83:0/3279200286 shutdown_connections 2024-03-06T07:31:43.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.151+0000 7f5eba516640 1 -- 172.21.15.83:0/3279200286 wait complete. 2024-03-06T07:31:43.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.152+0000 7f5eba516640 1 Processor -- start 2024-03-06T07:31:43.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.152+0000 7f5eba516640 1 -- start start 2024-03-06T07:31:43.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.153+0000 7f5eba516640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5eb40ce4b0 0x7f5eb41595e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:43.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.153+0000 7f5eba516640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5eb40cfff0 0x7f5eb41f7200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:43.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.153+0000 7f5eba516640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5eb415bb20 0x7f5eb41f7740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:43.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.153+0000 7f5eba516640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb40d24b0 con 0x7f5eb40ce4b0 2024-03-06T07:31:43.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.153+0000 7f5eba516640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb40d2330 con 0x7f5eb40cfff0 2024-03-06T07:31:43.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.153+0000 7f5eba516640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb40d1980 con 0x7f5eb415bb20 2024-03-06T07:31:43.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.153+0000 7f5eb3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5eb40ce4b0 0x7f5eb41595e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:43.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.153+0000 7f5eb3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5eb40ce4b0 0x7f5eb41595e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52402/0 (socket says 172.21.15.83:52402) 2024-03-06T07:31:43.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.153+0000 7f5eb3fff640 1 -- 172.21.15.83:0/287471745 learned_addr learned my addr 172.21.15.83:0/287471745 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:43.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.153+0000 7f5eb3fff640 1 -- 172.21.15.83:0/287471745 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5eb415bb20 msgr2=0x7f5eb41f7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:43.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.153+0000 7f5eb3fff640 1 --2- 172.21.15.83:0/287471745 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5eb415bb20 0x7f5eb41f7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.153+0000 7f5eb3fff640 1 -- 172.21.15.83:0/287471745 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5eb40cfff0 msgr2=0x7f5eb41f7200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:43.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.153+0000 7f5eb3fff640 1 --2- 172.21.15.83:0/287471745 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5eb40cfff0 0x7f5eb41f7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.153+0000 7f5eb3fff640 1 -- 172.21.15.83:0/287471745 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb4160dc0 con 0x7f5eb40ce4b0 2024-03-06T07:31:43.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.153+0000 7f5eb3fff640 1 --2- 172.21.15.83:0/287471745 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5eb40ce4b0 0x7f5eb41595e0 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c00d7d0 tx=0x7f5e9c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:43.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.153+0000 7f5eb0ff9640 1 -- 172.21.15.83:0/287471745 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e9c004030 con 0x7f5eb40ce4b0 2024-03-06T07:31:43.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.153+0000 7f5eb0ff9640 1 -- 172.21.15.83:0/287471745 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e9c0041d0 con 0x7f5eb40ce4b0 2024-03-06T07:31:43.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.153+0000 7f5eba516640 1 -- 172.21.15.83:0/287471745 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eb41610a0 con 0x7f5eb40ce4b0 2024-03-06T07:31:43.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.153+0000 7f5eb0ff9640 1 -- 172.21.15.83:0/287471745 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e9c0096a0 con 0x7f5eb40ce4b0 2024-03-06T07:31:43.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.153+0000 7f5eba516640 1 -- 172.21.15.83:0/287471745 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eb4157e50 con 0x7f5eb40ce4b0 2024-03-06T07:31:43.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.154+0000 7f5eba516640 1 -- 172.21.15.83:0/287471745 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5eb404f330 con 0x7f5eb40ce4b0 2024-03-06T07:31:43.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.155+0000 7f5eb0ff9640 1 -- 172.21.15.83:0/287471745 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5e9c009840 con 0x7f5eb40ce4b0 2024-03-06T07:31:43.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.155+0000 7f5eb0ff9640 1 --2- 172.21.15.83:0/287471745 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5e9406b4d0 0x7f5e9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:43.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.155+0000 7f5eb0ff9640 1 -- 172.21.15.83:0/287471745 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5e9c08a570 con 0x7f5eb40ce4b0 2024-03-06T07:31:43.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.155+0000 7f5eb37fe640 1 --2- 172.21.15.83:0/287471745 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5e9406b4d0 0x7f5e9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:43.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.156+0000 7f5eb37fe640 1 --2- 172.21.15.83:0/287471745 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5e9406b4d0 0x7f5e9406d990 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f5ea4036000 tx=0x7f5ea4030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:43.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.164+0000 7f5eb0ff9640 1 -- 172.21.15.83:0/287471745 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5e9c0593b0 con 0x7f5eb40ce4b0 2024-03-06T07:31:43.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.379+0000 7f5eba516640 1 -- 172.21.15.83:0/287471745 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5eb40b7940 con 0x7f5eb40ce4b0 2024-03-06T07:31:43.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.380+0000 7f5eb0ff9640 1 -- 172.21.15.83:0/287471745 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f5e9c058cb0 con 0x7f5eb40ce4b0 2024-03-06T07:31:43.382 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:31:43.382 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:31:43.382 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:31:43.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.382+0000 7f5eba516640 1 -- 172.21.15.83:0/287471745 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5e9406b4d0 msgr2=0x7f5e9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:43.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.382+0000 7f5eba516640 1 --2- 172.21.15.83:0/287471745 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5e9406b4d0 0x7f5e9406d990 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f5ea4036000 tx=0x7f5ea4030000 comp rx=0 tx=0).stop 2024-03-06T07:31:43.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.382+0000 7f5eba516640 1 -- 172.21.15.83:0/287471745 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5eb40ce4b0 msgr2=0x7f5eb41595e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:43.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.382+0000 7f5eba516640 1 --2- 172.21.15.83:0/287471745 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5eb40ce4b0 0x7f5eb41595e0 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c00d7d0 tx=0x7f5e9c00dc90 comp rx=0 tx=0).stop 2024-03-06T07:31:43.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.382+0000 7f5eba516640 1 -- 172.21.15.83:0/287471745 shutdown_connections 2024-03-06T07:31:43.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.382+0000 7f5eba516640 1 --2- 172.21.15.83:0/287471745 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5e9406b4d0 0x7f5e9406d990 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.382+0000 7f5eba516640 1 --2- 172.21.15.83:0/287471745 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5eb415bb20 0x7f5eb41f7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.382+0000 7f5eba516640 1 --2- 172.21.15.83:0/287471745 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5eb40cfff0 0x7f5eb41f7200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.382+0000 7f5eba516640 1 --2- 172.21.15.83:0/287471745 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5eb40ce4b0 0x7f5eb41595e0 unknown :-1 s=CLOSED pgs=1891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.382+0000 7f5eba516640 1 -- 172.21.15.83:0/287471745 >> 172.21.15.83:0/287471745 conn(0x7f5eb409a9f0 msgr2=0x7f5eb40ce940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:43.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.382+0000 7f5eba516640 1 -- 172.21.15.83:0/287471745 shutdown_connections 2024-03-06T07:31:43.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:43.382+0000 7f5eba516640 1 -- 172.21.15.83:0/287471745 wait complete. 2024-03-06T07:31:43.599 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:31:26.591282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:31:43.599 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:31:44.600 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:44.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.702+0000 7f5991a9d640 1 Processor -- start 2024-03-06T07:31:44.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.702+0000 7f5991a9d640 1 -- start start 2024-03-06T07:31:44.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.703+0000 7f5991a9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f598c0d2710 0x7f598c0b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.703+0000 7f5991a9d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f598c0b3430 con 0x7f598c0d22b0 2024-03-06T07:31:44.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.703+0000 7f5991a9d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f598c0b3710 con 0x7f598c15dfe0 2024-03-06T07:31:44.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.703+0000 7f5991a9d640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f598c0b31e0 con 0x7f598c0d2710 2024-03-06T07:31:44.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.703+0000 7f598affd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f598c0d22b0 0x7f598c0b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45230/0 (socket says 172.21.15.83:45230) 2024-03-06T07:31:44.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.703+0000 7f598affd640 1 -- 172.21.15.83:0/2324166739 learned_addr learned my addr 172.21.15.83:0/2324166739 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:31:44.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.703+0000 7f598bfff640 1 --2- 172.21.15.83:0/2324166739 >> v2:172.21.15.83:3300/0 conn(0x7f598c0d2710 0x7f598c0b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:44.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.703+0000 7f598bfff640 1 -- 172.21.15.83:0/2324166739 >> v1:172.21.15.183:6789/0 conn(0x7f598c15dfe0 legacy=0x7f598c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:44.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.703+0000 7f598bfff640 1 -- 172.21.15.83:0/2324166739 >> v1:172.21.15.83:6789/0 conn(0x7f598c0d22b0 legacy=0x7f598c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.703+0000 7f598a7fc640 1 -- 172.21.15.83:0/2324166739 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 337793402 0 0) 0x7f598c0b3430 con 0x7f598c0d22b0 2024-03-06T07:31:44.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.703+0000 7f598bfff640 1 -- 172.21.15.83:0/2324166739 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f598c0b39f0 con 0x7f598c0d2710 2024-03-06T07:31:44.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.704+0000 7f598bfff640 1 --2- 172.21.15.83:0/2324166739 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f598c0d2710 0x7f598c0b4900 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f5984009dc0 tx=0x7f598402be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fca62e2a11aced14 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:44.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.704+0000 7f598a7fc640 1 -- 172.21.15.83:0/2324166739 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5984039070 con 0x7f598c0d2710 2024-03-06T07:31:44.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.704+0000 7f598a7fc640 1 -- 172.21.15.83:0/2324166739 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f598402cc40 con 0x7f598c0d2710 2024-03-06T07:31:44.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.704+0000 7f598a7fc640 1 -- 172.21.15.83:0/2324166739 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5984034440 con 0x7f598c0d2710 2024-03-06T07:31:44.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.705+0000 7f5991a9d640 1 -- 172.21.15.83:0/2324166739 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f598c0d2710 msgr2=0x7f598c0b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.705+0000 7f5991a9d640 1 --2- 172.21.15.83:0/2324166739 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f598c0d2710 0x7f598c0b4900 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f5984009dc0 tx=0x7f598402be90 comp rx=0 tx=0).stop 2024-03-06T07:31:44.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.705+0000 7f5991a9d640 1 -- 172.21.15.83:0/2324166739 shutdown_connections 2024-03-06T07:31:44.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.705+0000 7f5991a9d640 1 --2- 172.21.15.83:0/2324166739 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f598c0d2710 0x7f598c0b4900 unknown :-1 s=CLOSED pgs=1893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.705+0000 7f5991a9d640 1 -- 172.21.15.83:0/2324166739 >> 172.21.15.83:0/2324166739 conn(0x7f598c0d3cd0 msgr2=0x7f598c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:44.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.705+0000 7f5991a9d640 1 -- 172.21.15.83:0/2324166739 shutdown_connections 2024-03-06T07:31:44.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.705+0000 7f5991a9d640 1 -- 172.21.15.83:0/2324166739 wait complete. 2024-03-06T07:31:44.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.707+0000 7f5991a9d640 1 Processor -- start 2024-03-06T07:31:44.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.707+0000 7f5991a9d640 1 -- start start 2024-03-06T07:31:44.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.708+0000 7f5991a9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f598c0d22b0 0x7f598c0b8db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.708+0000 7f5991a9d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f598c0d2710 0x7f598c1f6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.708+0000 7f5991a9d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f598c15dfe0 0x7f598c159510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.708+0000 7f5991a9d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598c0cfa00 con 0x7f598c0d22b0 2024-03-06T07:31:44.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.708+0000 7f5991a9d640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598c0b3d60 con 0x7f598c15dfe0 2024-03-06T07:31:44.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.708+0000 7f5991a9d640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f598c0d1c70 con 0x7f598c0d2710 2024-03-06T07:31:44.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.708+0000 7f598b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f598c0d22b0 0x7f598c0b8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:44.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.708+0000 7f598b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f598c0d22b0 0x7f598c0b8db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52420/0 (socket says 172.21.15.83:52420) 2024-03-06T07:31:44.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.708+0000 7f598b7fe640 1 -- 172.21.15.83:0/2889259199 learned_addr learned my addr 172.21.15.83:0/2889259199 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:44.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.708+0000 7f598affd640 1 --2- 172.21.15.83:0/2889259199 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f598c0d2710 0x7f598c1f6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:44.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.708+0000 7f598bfff640 1 --2- 172.21.15.83:0/2889259199 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f598c15dfe0 0x7f598c159510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:44.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.708+0000 7f598b7fe640 1 -- 172.21.15.83:0/2889259199 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f598c0d2710 msgr2=0x7f598c1f6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.708+0000 7f598b7fe640 1 --2- 172.21.15.83:0/2889259199 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f598c0d2710 0x7f598c1f6f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.708+0000 7f598b7fe640 1 -- 172.21.15.83:0/2889259199 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f598c15dfe0 msgr2=0x7f598c159510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.708+0000 7f598b7fe640 1 --2- 172.21.15.83:0/2889259199 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f598c15dfe0 0x7f598c159510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.708+0000 7f598b7fe640 1 -- 172.21.15.83:0/2889259199 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f598c0b68d0 con 0x7f598c0d22b0 2024-03-06T07:31:44.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.709+0000 7f598b7fe640 1 --2- 172.21.15.83:0/2889259199 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f598c0d22b0 0x7f598c0b8db0 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f597c00b3f0 tx=0x7f597c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:44.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.709+0000 7f5973fff640 1 -- 172.21.15.83:0/2889259199 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f597c015020 con 0x7f598c0d22b0 2024-03-06T07:31:44.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.709+0000 7f5973fff640 1 -- 172.21.15.83:0/2889259199 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f597c011830 con 0x7f598c0d22b0 2024-03-06T07:31:44.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.709+0000 7f5973fff640 1 -- 172.21.15.83:0/2889259199 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f597c011c20 con 0x7f598c0d22b0 2024-03-06T07:31:44.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.709+0000 7f5991a9d640 1 -- 172.21.15.83:0/2889259199 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f598c1f7a90 con 0x7f598c0d22b0 2024-03-06T07:31:44.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.709+0000 7f5991a9d640 1 -- 172.21.15.83:0/2889259199 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f598c0b6e50 con 0x7f598c0d22b0 2024-03-06T07:31:44.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.710+0000 7f5991a9d640 1 -- 172.21.15.83:0/2889259199 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f598c0b7050 con 0x7f598c0d22b0 2024-03-06T07:31:44.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.710+0000 7f5973fff640 1 -- 172.21.15.83:0/2889259199 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f597c018020 con 0x7f598c0d22b0 2024-03-06T07:31:44.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.711+0000 7f5973fff640 1 --2- 172.21.15.83:0/2889259199 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f597806b4d0 0x7f597806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.711+0000 7f5973fff640 1 -- 172.21.15.83:0/2889259199 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f597c089e40 con 0x7f598c0d22b0 2024-03-06T07:31:44.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.711+0000 7f598affd640 1 --2- 172.21.15.83:0/2889259199 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f597806b4d0 0x7f597806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:44.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.711+0000 7f598affd640 1 --2- 172.21.15.83:0/2889259199 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f597806b4d0 0x7f597806d990 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f598c15c9e0 tx=0x7f5980007b50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:44.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.715+0000 7f5973fff640 1 -- 172.21.15.83:0/2889259199 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f598c0b7050 con 0x7f598c0d22b0 2024-03-06T07:31:44.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.930+0000 7f5991a9d640 1 -- 172.21.15.83:0/2889259199 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f598c0cec80 con 0x7f598c0d22b0 2024-03-06T07:31:44.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.930+0000 7f5973fff640 1 -- 172.21.15.83:0/2889259199 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f597c01f030 con 0x7f598c0d22b0 2024-03-06T07:31:44.932 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:31:44.932 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:31:44.933 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:31:44.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.932+0000 7f5991a9d640 1 -- 172.21.15.83:0/2889259199 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f597806b4d0 msgr2=0x7f597806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.932+0000 7f5991a9d640 1 --2- 172.21.15.83:0/2889259199 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f597806b4d0 0x7f597806d990 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f598c15c9e0 tx=0x7f5980007b50 comp rx=0 tx=0).stop 2024-03-06T07:31:44.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.932+0000 7f5991a9d640 1 -- 172.21.15.83:0/2889259199 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f598c0d22b0 msgr2=0x7f598c0b8db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.932+0000 7f5991a9d640 1 --2- 172.21.15.83:0/2889259199 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f598c0d22b0 0x7f598c0b8db0 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f597c00b3f0 tx=0x7f597c00b8b0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.932+0000 7f5991a9d640 1 -- 172.21.15.83:0/2889259199 shutdown_connections 2024-03-06T07:31:44.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.932+0000 7f5991a9d640 1 --2- 172.21.15.83:0/2889259199 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f597806b4d0 0x7f597806d990 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.932+0000 7f5991a9d640 1 --2- 172.21.15.83:0/2889259199 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f598c15dfe0 0x7f598c159510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.932+0000 7f5991a9d640 1 --2- 172.21.15.83:0/2889259199 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f598c0d2710 0x7f598c1f6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.932+0000 7f5991a9d640 1 --2- 172.21.15.83:0/2889259199 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f598c0d22b0 0x7f598c0b8db0 unknown :-1 s=CLOSED pgs=1894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.932+0000 7f5991a9d640 1 -- 172.21.15.83:0/2889259199 >> 172.21.15.83:0/2889259199 conn(0x7f598c0d3cd0 msgr2=0x7f598c0cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:44.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.933+0000 7f5991a9d640 1 -- 172.21.15.83:0/2889259199 shutdown_connections 2024-03-06T07:31:44.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:44.933+0000 7f5991a9d640 1 -- 172.21.15.83:0/2889259199 wait complete. 2024-03-06T07:31:45.106 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:45.146 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:31:26.591282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:31:45.147 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:31:46.148 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:46.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.247+0000 7ff6f7d47640 1 Processor -- start 2024-03-06T07:31:46.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.248+0000 7ff6f7d47640 1 -- start start 2024-03-06T07:31:46.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.248+0000 7ff6f7d47640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ff6f015dfe0 0x7ff6f00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.248+0000 7ff6f7d47640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff6f00cf830 0x7ff6f00d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.248+0000 7ff6f7d47640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f00b3430 con 0x7ff6f00d27f0 2024-03-06T07:31:46.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.248+0000 7ff6f7d47640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6f00b31e0 con 0x7ff6f00cf830 2024-03-06T07:31:46.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.248+0000 7ff6f7d47640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6f00b3d60 con 0x7ff6f015dfe0 2024-03-06T07:31:46.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.248+0000 7ff6f52e2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff6f00cf830 0x7ff6f00d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:46.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.248+0000 7ff6f62e4640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff6f00d27f0 0x7ff6f00d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45242/0 (socket says 172.21.15.83:45242) 2024-03-06T07:31:46.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.248+0000 7ff6f62e4640 1 -- 172.21.15.83:0/1070935220 learned_addr learned my addr 172.21.15.83:0/1070935220 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:31:46.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.249+0000 7ff6f5ae3640 1 --2- 172.21.15.83:0/1070935220 >> v2:172.21.15.183:3300/0 conn(0x7ff6f015dfe0 0x7ff6f00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:46.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.249+0000 7ff6f52e2640 1 -- 172.21.15.83:0/1070935220 >> v2:172.21.15.183:3300/0 conn(0x7ff6f015dfe0 msgr2=0x7ff6f00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:46.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.249+0000 7ff6f52e2640 1 --2- 172.21.15.83:0/1070935220 >> v2:172.21.15.183:3300/0 conn(0x7ff6f015dfe0 0x7ff6f00cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.249+0000 7ff6f52e2640 1 -- 172.21.15.83:0/1070935220 >> v1:172.21.15.83:6789/0 conn(0x7ff6f00d27f0 legacy=0x7ff6f00d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:46.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.249+0000 7ff6f52e2640 1 -- 172.21.15.83:0/1070935220 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f00b3710 con 0x7ff6f00cf830 2024-03-06T07:31:46.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.250+0000 7ff6f52e2640 1 --2- 172.21.15.83:0/1070935220 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6f00cf830 0x7ff6f00d22b0 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7ff6e00097a0 tx=0x7ff6e002b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c934858c8988defd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:46.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.250+0000 7ff6f4ae1640 1 -- 172.21.15.83:0/1070935220 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e0037070 con 0x7ff6f00cf830 2024-03-06T07:31:46.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.250+0000 7ff6f4ae1640 1 -- 172.21.15.83:0/1070935220 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6e0030710 con 0x7ff6f00cf830 2024-03-06T07:31:46.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.250+0000 7ff6f4ae1640 1 -- 172.21.15.83:0/1070935220 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e0030ae0 con 0x7ff6f00cf830 2024-03-06T07:31:46.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.251+0000 7ff6f7d47640 1 -- 172.21.15.83:0/1070935220 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6f00cf830 msgr2=0x7ff6f00d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:46.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.251+0000 7ff6f7d47640 1 --2- 172.21.15.83:0/1070935220 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6f00cf830 0x7ff6f00d22b0 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7ff6e00097a0 tx=0x7ff6e002b710 comp rx=0 tx=0).stop 2024-03-06T07:31:46.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.251+0000 7ff6f7d47640 1 -- 172.21.15.83:0/1070935220 shutdown_connections 2024-03-06T07:31:46.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.251+0000 7ff6f7d47640 1 --2- 172.21.15.83:0/1070935220 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6f00cf830 0x7ff6f00d22b0 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.251+0000 7ff6f7d47640 1 --2- 172.21.15.83:0/1070935220 >> v2:172.21.15.183:3300/0 conn(0x7ff6f015dfe0 0x7ff6f00cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.251+0000 7ff6f7d47640 1 -- 172.21.15.83:0/1070935220 >> 172.21.15.83:0/1070935220 conn(0x7ff6f00d3cd0 msgr2=0x7ff6f00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:46.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.251+0000 7ff6f7d47640 1 -- 172.21.15.83:0/1070935220 shutdown_connections 2024-03-06T07:31:46.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.251+0000 7ff6f7d47640 1 -- 172.21.15.83:0/1070935220 wait complete. 2024-03-06T07:31:46.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.252+0000 7ff6f7d47640 1 Processor -- start 2024-03-06T07:31:46.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.252+0000 7ff6f7d47640 1 -- start start 2024-03-06T07:31:46.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.253+0000 7ff6f7d47640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6f00cf830 0x7ff6f01f6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.253+0000 7ff6f7d47640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff6f00d27f0 0x7ff6f00b8370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.253+0000 7ff6f7d47640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff6f015dfe0 0x7ff6f00b88b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.253+0000 7ff6f7d47640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f00d1df0 con 0x7ff6f00cf830 2024-03-06T07:31:46.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.253+0000 7ff6f7d47640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f00d1c70 con 0x7ff6f015dfe0 2024-03-06T07:31:46.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.253+0000 7ff6f7d47640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f015e3c0 con 0x7ff6f00d27f0 2024-03-06T07:31:46.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.253+0000 7ff6f5ae3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6f00cf830 0x7ff6f01f6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:46.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.253+0000 7ff6f5ae3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6f00cf830 0x7ff6f01f6df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52440/0 (socket says 172.21.15.83:52440) 2024-03-06T07:31:46.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.253+0000 7ff6f5ae3640 1 -- 172.21.15.83:0/3632023699 learned_addr learned my addr 172.21.15.83:0/3632023699 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:46.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.253+0000 7ff6f52e2640 1 --2- 172.21.15.83:0/3632023699 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff6f00d27f0 0x7ff6f00b8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:46.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.253+0000 7ff6f62e4640 1 --2- 172.21.15.83:0/3632023699 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff6f015dfe0 0x7ff6f00b88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:46.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.253+0000 7ff6f5ae3640 1 -- 172.21.15.83:0/3632023699 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff6f00d27f0 msgr2=0x7ff6f00b8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:46.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.253+0000 7ff6f5ae3640 1 --2- 172.21.15.83:0/3632023699 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff6f00d27f0 0x7ff6f00b8370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.253+0000 7ff6f5ae3640 1 -- 172.21.15.83:0/3632023699 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff6f015dfe0 msgr2=0x7ff6f00b88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:46.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.253+0000 7ff6f5ae3640 1 --2- 172.21.15.83:0/3632023699 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff6f015dfe0 0x7ff6f00b88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.253+0000 7ff6f5ae3640 1 -- 172.21.15.83:0/3632023699 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f01f7330 con 0x7ff6f00cf830 2024-03-06T07:31:46.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.253+0000 7ff6f62e4640 1 --2- 172.21.15.83:0/3632023699 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff6f015dfe0 0x7ff6f00b88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:46.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.253+0000 7ff6f5ae3640 1 --2- 172.21.15.83:0/3632023699 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6f00cf830 0x7ff6f01f6df0 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec00d830 tx=0x7ff6ec00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:46.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.253+0000 7ff6e67fc640 1 -- 172.21.15.83:0/3632023699 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6ec00b6c0 con 0x7ff6f00cf830 2024-03-06T07:31:46.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.253+0000 7ff6e67fc640 1 -- 172.21.15.83:0/3632023699 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6ec019070 con 0x7ff6f00cf830 2024-03-06T07:31:46.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.253+0000 7ff6f7d47640 1 -- 172.21.15.83:0/3632023699 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6f015f1a0 con 0x7ff6f00cf830 2024-03-06T07:31:46.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.253+0000 7ff6e67fc640 1 -- 172.21.15.83:0/3632023699 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6ec00ba80 con 0x7ff6f00cf830 2024-03-06T07:31:46.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.253+0000 7ff6f7d47640 1 -- 172.21.15.83:0/3632023699 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f015f760 con 0x7ff6f00cf830 2024-03-06T07:31:46.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.254+0000 7ff6f7d47640 1 -- 172.21.15.83:0/3632023699 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6f00ce7a0 con 0x7ff6f00cf830 2024-03-06T07:31:46.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.255+0000 7ff6e67fc640 1 -- 172.21.15.83:0/3632023699 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff6ec01e460 con 0x7ff6f00cf830 2024-03-06T07:31:46.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.255+0000 7ff6e67fc640 1 --2- 172.21.15.83:0/3632023699 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff6d406b4d0 0x7ff6d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.255+0000 7ff6e67fc640 1 -- 172.21.15.83:0/3632023699 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff6ec08a830 con 0x7ff6f00cf830 2024-03-06T07:31:46.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.255+0000 7ff6f52e2640 1 --2- 172.21.15.83:0/3632023699 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff6d406b4d0 0x7ff6d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:46.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.256+0000 7ff6f52e2640 1 --2- 172.21.15.83:0/3632023699 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff6d406b4d0 0x7ff6d406d990 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7ff6e0034000 tx=0x7ff6e0008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:46.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.260+0000 7ff6e67fc640 1 -- 172.21.15.83:0/3632023699 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff6f00ce7a0 con 0x7ff6f00cf830 2024-03-06T07:31:46.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.474+0000 7ff6f7d47640 1 -- 172.21.15.83:0/3632023699 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6f004f330 con 0x7ff6f00cf830 2024-03-06T07:31:46.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.475+0000 7ff6e67fc640 1 -- 172.21.15.83:0/3632023699 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7ff6ec059670 con 0x7ff6f00cf830 2024-03-06T07:31:46.477 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:31:46.477 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:31:46.477 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:31:46.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.477+0000 7ff6f7d47640 1 -- 172.21.15.83:0/3632023699 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff6d406b4d0 msgr2=0x7ff6d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:46.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.477+0000 7ff6f7d47640 1 --2- 172.21.15.83:0/3632023699 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff6d406b4d0 0x7ff6d406d990 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7ff6e0034000 tx=0x7ff6e0008030 comp rx=0 tx=0).stop 2024-03-06T07:31:46.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.477+0000 7ff6f7d47640 1 -- 172.21.15.83:0/3632023699 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6f00cf830 msgr2=0x7ff6f01f6df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:46.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.477+0000 7ff6f7d47640 1 --2- 172.21.15.83:0/3632023699 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6f00cf830 0x7ff6f01f6df0 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec00d830 tx=0x7ff6ec00dcf0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.477+0000 7ff6f7d47640 1 -- 172.21.15.83:0/3632023699 shutdown_connections 2024-03-06T07:31:46.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.477+0000 7ff6f7d47640 1 --2- 172.21.15.83:0/3632023699 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff6d406b4d0 0x7ff6d406d990 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.477+0000 7ff6f7d47640 1 --2- 172.21.15.83:0/3632023699 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff6f015dfe0 0x7ff6f00b88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.477+0000 7ff6f7d47640 1 --2- 172.21.15.83:0/3632023699 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff6f00d27f0 0x7ff6f00b8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.477+0000 7ff6f7d47640 1 --2- 172.21.15.83:0/3632023699 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff6f00cf830 0x7ff6f01f6df0 unknown :-1 s=CLOSED pgs=1897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.477+0000 7ff6f7d47640 1 -- 172.21.15.83:0/3632023699 >> 172.21.15.83:0/3632023699 conn(0x7ff6f00d3cd0 msgr2=0x7ff6f00d08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:46.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.477+0000 7ff6f7d47640 1 -- 172.21.15.83:0/3632023699 shutdown_connections 2024-03-06T07:31:46.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:46.477+0000 7ff6f7d47640 1 -- 172.21.15.83:0/3632023699 wait complete. 2024-03-06T07:31:46.691 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:31:26.591282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:31:46.691 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:31:47.107 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:47.692 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:47.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.795+0000 7f52c1b81640 1 Processor -- start 2024-03-06T07:31:47.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.795+0000 7f52c1b81640 1 -- start start 2024-03-06T07:31:47.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.796+0000 7f52c1b81640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f52bc15dfe0 0x7f52bc0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.796+0000 7f52c1b81640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f52bc0d29c0 0x7f52bc0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.796+0000 7f52c1b81640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52bc0b3430 con 0x7f52bc0cf830 2024-03-06T07:31:47.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.796+0000 7f52c1b81640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52bc0b31e0 con 0x7f52bc15dfe0 2024-03-06T07:31:47.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.796+0000 7f52c1b81640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52bc0b3d60 con 0x7f52bc0d29c0 2024-03-06T07:31:47.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.796+0000 7f52bb7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f52bc15dfe0 0x7f52bc0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.796+0000 7f52bb7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f52bc15dfe0 0x7f52bc0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58546/0 (socket says 172.21.15.83:58546) 2024-03-06T07:31:47.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.796+0000 7f52baffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f52bc0cf830 0x7f52bc0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:34504/0 (socket says 172.21.15.83:34504) 2024-03-06T07:31:47.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.796+0000 7f52bb7fe640 1 -- 172.21.15.83:0/1640172567 learned_addr learned my addr 172.21.15.83:0/1640172567 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:47.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.796+0000 7f52bbfff640 1 --2- 172.21.15.83:0/1640172567 >> v2:172.21.15.183:3300/0 conn(0x7f52bc0d29c0 0x7f52bc0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.796+0000 7f52bb7fe640 1 -- 172.21.15.83:0/1640172567 >> v2:172.21.15.183:3300/0 conn(0x7f52bc0d29c0 msgr2=0x7f52bc0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:47.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.796+0000 7f52bb7fe640 1 --2- 172.21.15.83:0/1640172567 >> v2:172.21.15.183:3300/0 conn(0x7f52bc0d29c0 0x7f52bc0b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.796+0000 7f52bb7fe640 1 -- 172.21.15.83:0/1640172567 >> v1:172.21.15.83:6789/0 conn(0x7f52bc0cf830 legacy=0x7f52bc0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.796+0000 7f52bb7fe640 1 -- 172.21.15.83:0/1640172567 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52bc0b3710 con 0x7f52bc15dfe0 2024-03-06T07:31:47.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.796+0000 7f52ba7fc640 1 -- 172.21.15.83:0/1640172567 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2097955644 0 0) 0x7f52bc0b3430 con 0x7f52bc0cf830 2024-03-06T07:31:47.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.797+0000 7f52bb7fe640 1 --2- 172.21.15.83:0/1640172567 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f52bc15dfe0 0x7f52bc0cf2f0 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f52bc0b3cd0 tx=0x7f52b002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3173d2bc90849ad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:47.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.797+0000 7f52ba7fc640 1 -- 172.21.15.83:0/1640172567 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b0034070 con 0x7f52bc15dfe0 2024-03-06T07:31:47.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.797+0000 7f52ba7fc640 1 -- 172.21.15.83:0/1640172567 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52b0030860 con 0x7f52bc15dfe0 2024-03-06T07:31:47.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.797+0000 7f52ba7fc640 1 -- 172.21.15.83:0/1640172567 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b0030c30 con 0x7f52bc15dfe0 2024-03-06T07:31:47.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.798+0000 7f52c1b81640 1 -- 172.21.15.83:0/1640172567 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f52bc15dfe0 msgr2=0x7f52bc0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.798+0000 7f52c1b81640 1 --2- 172.21.15.83:0/1640172567 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f52bc15dfe0 0x7f52bc0cf2f0 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f52bc0b3cd0 tx=0x7f52b002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.798+0000 7f52c1b81640 1 -- 172.21.15.83:0/1640172567 shutdown_connections 2024-03-06T07:31:47.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.798+0000 7f52c1b81640 1 --2- 172.21.15.83:0/1640172567 >> v2:172.21.15.183:3300/0 conn(0x7f52bc0d29c0 0x7f52bc0b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.798+0000 7f52c1b81640 1 --2- 172.21.15.83:0/1640172567 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f52bc15dfe0 0x7f52bc0cf2f0 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.798+0000 7f52c1b81640 1 -- 172.21.15.83:0/1640172567 >> 172.21.15.83:0/1640172567 conn(0x7f52bc0d3cd0 msgr2=0x7f52bc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:47.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.798+0000 7f52c1b81640 1 -- 172.21.15.83:0/1640172567 shutdown_connections 2024-03-06T07:31:47.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.798+0000 7f52c1b81640 1 -- 172.21.15.83:0/1640172567 wait complete. 2024-03-06T07:31:47.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.799+0000 7f52c1b81640 1 Processor -- start 2024-03-06T07:31:47.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.799+0000 7f52c1b81640 1 -- start start 2024-03-06T07:31:47.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.800+0000 7f52c1b81640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f52bc0cf830 0x7f52bc15bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.800+0000 7f52c1b81640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f52bc0d29c0 0x7f52bc1f4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.800+0000 7f52c1b81640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f52bc15dfe0 0x7f52bc0b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.800+0000 7f52c1b81640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52bc0d1df0 con 0x7f52bc15dfe0 2024-03-06T07:31:47.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.800+0000 7f52c1b81640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52bc0d1c70 con 0x7f52bc0cf830 2024-03-06T07:31:47.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.800+0000 7f52c1b81640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52bc15e3c0 con 0x7f52bc0d29c0 2024-03-06T07:31:47.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.800+0000 7f52bbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f52bc15dfe0 0x7f52bc0b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.800+0000 7f52bbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f52bc15dfe0 0x7f52bc0b68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58556/0 (socket says 172.21.15.83:58556) 2024-03-06T07:31:47.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.800+0000 7f52bbfff640 1 -- 172.21.15.83:0/2979497203 learned_addr learned my addr 172.21.15.83:0/2979497203 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:47.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.800+0000 7f52bb7fe640 1 --2- 172.21.15.83:0/2979497203 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f52bc0cf830 0x7f52bc15bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.800+0000 7f52baffd640 1 --2- 172.21.15.83:0/2979497203 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f52bc0d29c0 0x7f52bc1f4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.800+0000 7f52bbfff640 1 -- 172.21.15.83:0/2979497203 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f52bc0d29c0 msgr2=0x7f52bc1f4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.800+0000 7f52bbfff640 1 --2- 172.21.15.83:0/2979497203 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f52bc0d29c0 0x7f52bc1f4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.800+0000 7f52bbfff640 1 -- 172.21.15.83:0/2979497203 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f52bc0cf830 msgr2=0x7f52bc15bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.800+0000 7f52bbfff640 1 --2- 172.21.15.83:0/2979497203 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f52bc0cf830 0x7f52bc15bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.800+0000 7f52bbfff640 1 -- 172.21.15.83:0/2979497203 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52bc1f7280 con 0x7f52bc15dfe0 2024-03-06T07:31:47.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.800+0000 7f52bb7fe640 1 --2- 172.21.15.83:0/2979497203 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f52bc0cf830 0x7f52bc15bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:47.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.801+0000 7f52bbfff640 1 --2- 172.21.15.83:0/2979497203 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f52bc15dfe0 0x7f52bc0b68d0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f52b4007e30 tx=0x7f52b400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:47.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.801+0000 7f52a3fff640 1 -- 172.21.15.83:0/2979497203 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b4014620 con 0x7f52bc15dfe0 2024-03-06T07:31:47.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.801+0000 7f52a3fff640 1 -- 172.21.15.83:0/2979497203 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52b4019070 con 0x7f52bc15dfe0 2024-03-06T07:31:47.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.801+0000 7f52c1b81640 1 -- 172.21.15.83:0/2979497203 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52bc1f7560 con 0x7f52bc15dfe0 2024-03-06T07:31:47.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.801+0000 7f52a3fff640 1 -- 172.21.15.83:0/2979497203 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b4014a00 con 0x7f52bc15dfe0 2024-03-06T07:31:47.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.801+0000 7f52c1b81640 1 -- 172.21.15.83:0/2979497203 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52bc1f7b20 con 0x7f52bc15dfe0 2024-03-06T07:31:47.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.802+0000 7f52c1b81640 1 -- 172.21.15.83:0/2979497203 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52bc04f330 con 0x7f52bc15dfe0 2024-03-06T07:31:47.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.802+0000 7f52a3fff640 1 -- 172.21.15.83:0/2979497203 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f52b4014ba0 con 0x7f52bc15dfe0 2024-03-06T07:31:47.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.803+0000 7f52a3fff640 1 --2- 172.21.15.83:0/2979497203 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f52a806b500 0x7f52a806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.803+0000 7f52a3fff640 1 -- 172.21.15.83:0/2979497203 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f52b408aa10 con 0x7f52bc15dfe0 2024-03-06T07:31:47.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.803+0000 7f52bb7fe640 1 --2- 172.21.15.83:0/2979497203 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f52a806b500 0x7f52a806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.803+0000 7f52bb7fe640 1 --2- 172.21.15.83:0/2979497203 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f52a806b500 0x7f52a806d9c0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f52b0036000 tx=0x7f52b003b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:47.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:47.807+0000 7f52a3fff640 1 -- 172.21.15.83:0/2979497203 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f52b40598e0 con 0x7f52bc15dfe0 2024-03-06T07:31:48.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:48.052+0000 7f52c1b81640 1 -- 172.21.15.83:0/2979497203 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52bc1f7d20 con 0x7f52bc15dfe0 2024-03-06T07:31:48.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:48.053+0000 7f52a3fff640 1 -- 172.21.15.83:0/2979497203 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f52b40591e0 con 0x7f52bc15dfe0 2024-03-06T07:31:48.055 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:31:48.055 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:31:48.055 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:31:48.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:48.055+0000 7f52c1b81640 1 -- 172.21.15.83:0/2979497203 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f52a806b500 msgr2=0x7f52a806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:48.055+0000 7f52c1b81640 1 --2- 172.21.15.83:0/2979497203 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f52a806b500 0x7f52a806d9c0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f52b0036000 tx=0x7f52b003b040 comp rx=0 tx=0).stop 2024-03-06T07:31:48.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:48.055+0000 7f52c1b81640 1 -- 172.21.15.83:0/2979497203 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f52bc15dfe0 msgr2=0x7f52bc0b68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:48.055+0000 7f52c1b81640 1 --2- 172.21.15.83:0/2979497203 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f52bc15dfe0 0x7f52bc0b68d0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f52b4007e30 tx=0x7f52b400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:48.055+0000 7f52c1b81640 1 -- 172.21.15.83:0/2979497203 shutdown_connections 2024-03-06T07:31:48.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:48.055+0000 7f52c1b81640 1 --2- 172.21.15.83:0/2979497203 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f52a806b500 0x7f52a806d9c0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:48.055+0000 7f52c1b81640 1 --2- 172.21.15.83:0/2979497203 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f52bc15dfe0 0x7f52bc0b68d0 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:48.055+0000 7f52c1b81640 1 --2- 172.21.15.83:0/2979497203 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f52bc0d29c0 0x7f52bc1f4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:48.055+0000 7f52c1b81640 1 --2- 172.21.15.83:0/2979497203 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f52bc0cf830 0x7f52bc15bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:48.055+0000 7f52c1b81640 1 -- 172.21.15.83:0/2979497203 >> 172.21.15.83:0/2979497203 conn(0x7f52bc0d3cd0 msgr2=0x7f52bc0ce9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:48.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:48.055+0000 7f52c1b81640 1 -- 172.21.15.83:0/2979497203 shutdown_connections 2024-03-06T07:31:48.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:48.055+0000 7f52c1b81640 1 -- 172.21.15.83:0/2979497203 wait complete. 2024-03-06T07:31:48.269 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:31:26.591282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:31:48.269 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:31:49.108 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:49.269 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:49.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.382+0000 7f11bf3bc640 1 Processor -- start 2024-03-06T07:31:49.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.382+0000 7f11bf3bc640 1 -- start start 2024-03-06T07:31:49.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.383+0000 7f11bf3bc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f11b815dfe0 0x7f11b80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.383+0000 7f11bf3bc640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11b80b3430 con 0x7f11b80cf830 2024-03-06T07:31:49.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.383+0000 7f11bf3bc640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11b80b3710 con 0x7f11b80d29c0 2024-03-06T07:31:49.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.383+0000 7f11bf3bc640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11b80b31e0 con 0x7f11b815dfe0 2024-03-06T07:31:49.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.383+0000 7f11bd158640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f11b815dfe0 0x7f11b80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.383+0000 7f11bc957640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f11b80cf830 0x7f11b80d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:34512/0 (socket says 172.21.15.83:34512) 2024-03-06T07:31:49.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.383+0000 7f11bd158640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f11b815dfe0 0x7f11b80cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58572/0 (socket says 172.21.15.83:58572) 2024-03-06T07:31:49.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.383+0000 7f11bc957640 1 -- 172.21.15.83:0/3944324221 learned_addr learned my addr 172.21.15.83:0/3944324221 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:31:49.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.383+0000 7f11bd158640 1 -- 172.21.15.83:0/3944324221 >> v1:172.21.15.183:6789/0 conn(0x7f11b80d29c0 legacy=0x7f11b80b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:49.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.383+0000 7f11bd158640 1 -- 172.21.15.83:0/3944324221 >> v1:172.21.15.83:6789/0 conn(0x7f11b80cf830 legacy=0x7f11b80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.383+0000 7f11bd158640 1 -- 172.21.15.83:0/3944324221 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b80b39f0 con 0x7f11b815dfe0 2024-03-06T07:31:49.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.384+0000 7f11bd158640 1 --2- 172.21.15.83:0/3944324221 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11b815dfe0 0x7f11b80cf2f0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f11b80b3360 tx=0x7f11a802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92c50ea44376cb25 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:49.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.385+0000 7f11b7fff640 1 -- 172.21.15.83:0/3944324221 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a8035020 con 0x7f11b815dfe0 2024-03-06T07:31:49.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.385+0000 7f11b7fff640 1 -- 172.21.15.83:0/3944324221 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11a8031830 con 0x7f11b815dfe0 2024-03-06T07:31:49.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.385+0000 7f11b7fff640 1 -- 172.21.15.83:0/3944324221 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a8031c00 con 0x7f11b815dfe0 2024-03-06T07:31:49.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.385+0000 7f11bf3bc640 1 -- 172.21.15.83:0/3944324221 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11b815dfe0 msgr2=0x7f11b80cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.385+0000 7f11bf3bc640 1 --2- 172.21.15.83:0/3944324221 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11b815dfe0 0x7f11b80cf2f0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f11b80b3360 tx=0x7f11a802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.385+0000 7f11bf3bc640 1 -- 172.21.15.83:0/3944324221 shutdown_connections 2024-03-06T07:31:49.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.385+0000 7f11bf3bc640 1 --2- 172.21.15.83:0/3944324221 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11b815dfe0 0x7f11b80cf2f0 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.385+0000 7f11bf3bc640 1 -- 172.21.15.83:0/3944324221 >> 172.21.15.83:0/3944324221 conn(0x7f11b80d3cd0 msgr2=0x7f11b80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:49.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.385+0000 7f11bf3bc640 1 -- 172.21.15.83:0/3944324221 shutdown_connections 2024-03-06T07:31:49.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.385+0000 7f11bf3bc640 1 -- 172.21.15.83:0/3944324221 wait complete. 2024-03-06T07:31:49.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.386+0000 7f11bf3bc640 1 Processor -- start 2024-03-06T07:31:49.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.386+0000 7f11bf3bc640 1 -- start start 2024-03-06T07:31:49.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.387+0000 7f11bf3bc640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11b80cf830 0x7f11b8157330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.387+0000 7f11bf3bc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11b80d29c0 0x7f11b81f48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.387+0000 7f11bf3bc640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11b815dfe0 0x7f11b81f5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.387+0000 7f11bf3bc640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b815e3c0 con 0x7f11b80d29c0 2024-03-06T07:31:49.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.387+0000 7f11bf3bc640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b80b3d60 con 0x7f11b80cf830 2024-03-06T07:31:49.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.387+0000 7f11bf3bc640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11b815e540 con 0x7f11b815dfe0 2024-03-06T07:31:49.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.387+0000 7f11bc957640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11b80d29c0 0x7f11b81f48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.387+0000 7f11bc957640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11b80d29c0 0x7f11b81f48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58584/0 (socket says 172.21.15.83:58584) 2024-03-06T07:31:49.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.387+0000 7f11bc957640 1 -- 172.21.15.83:0/1369904660 learned_addr learned my addr 172.21.15.83:0/1369904660 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:49.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.387+0000 7f11bd158640 1 --2- 172.21.15.83:0/1369904660 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11b80cf830 0x7f11b8157330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.387+0000 7f11bc957640 1 -- 172.21.15.83:0/1369904660 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11b815dfe0 msgr2=0x7f11b81f5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.387+0000 7f11bc957640 1 --2- 172.21.15.83:0/1369904660 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11b815dfe0 0x7f11b81f5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.387+0000 7f11bc957640 1 -- 172.21.15.83:0/1369904660 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11b80cf830 msgr2=0x7f11b8157330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.387+0000 7f11bc957640 1 --2- 172.21.15.83:0/1369904660 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11b80cf830 0x7f11b8157330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.387+0000 7f11bc957640 1 -- 172.21.15.83:0/1369904660 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b80b6cd0 con 0x7f11b80d29c0 2024-03-06T07:31:49.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.387+0000 7f11bd158640 1 --2- 172.21.15.83:0/1369904660 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11b80cf830 0x7f11b8157330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:49.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.388+0000 7f11bc957640 1 --2- 172.21.15.83:0/1369904660 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11b80d29c0 0x7f11b81f48b0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f11ac00c440 tx=0x7f11ac001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:49.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.388+0000 7f11b5ffb640 1 -- 172.21.15.83:0/1369904660 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ac017070 con 0x7f11b80d29c0 2024-03-06T07:31:49.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.388+0000 7f11b5ffb640 1 -- 172.21.15.83:0/1369904660 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11ac005ce0 con 0x7f11b80d29c0 2024-03-06T07:31:49.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.388+0000 7f11bf3bc640 1 -- 172.21.15.83:0/1369904660 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11b80b6fb0 con 0x7f11b80d29c0 2024-03-06T07:31:49.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.388+0000 7f11b5ffb640 1 -- 172.21.15.83:0/1369904660 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ac0072f0 con 0x7f11b80d29c0 2024-03-06T07:31:49.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.388+0000 7f11bf3bc640 1 -- 172.21.15.83:0/1369904660 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11b81f4df0 con 0x7f11b80d29c0 2024-03-06T07:31:49.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.389+0000 7f11bf3bc640 1 -- 172.21.15.83:0/1369904660 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11b804f330 con 0x7f11b80d29c0 2024-03-06T07:31:49.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.389+0000 7f11b5ffb640 1 -- 172.21.15.83:0/1369904660 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f11ac01a070 con 0x7f11b80d29c0 2024-03-06T07:31:49.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.390+0000 7f11b5ffb640 1 --2- 172.21.15.83:0/1369904660 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f11a406b4d0 0x7f11a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.390+0000 7f11bd158640 1 --2- 172.21.15.83:0/1369904660 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f11a406b4d0 0x7f11a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.390+0000 7f11b5ffb640 1 -- 172.21.15.83:0/1369904660 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f11ac0895a0 con 0x7f11b80d29c0 2024-03-06T07:31:49.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.390+0000 7f11bd158640 1 --2- 172.21.15.83:0/1369904660 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f11a406b4d0 0x7f11a406d990 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f11a8036000 tx=0x7f11a803c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:49.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.394+0000 7f11b5ffb640 1 -- 172.21.15.83:0/1369904660 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f11ac058470 con 0x7f11b80d29c0 2024-03-06T07:31:49.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.635+0000 7f11bf3bc640 1 -- 172.21.15.83:0/1369904660 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11b806d180 con 0x7f11b80d29c0 2024-03-06T07:31:49.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.635+0000 7f11b5ffb640 1 -- 172.21.15.83:0/1369904660 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f11ac057d70 con 0x7f11b80d29c0 2024-03-06T07:31:49.638 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:31:49.638 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:31:49.639 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:31:49.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.637+0000 7f11bf3bc640 1 -- 172.21.15.83:0/1369904660 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f11a406b4d0 msgr2=0x7f11a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.637+0000 7f11bf3bc640 1 --2- 172.21.15.83:0/1369904660 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f11a406b4d0 0x7f11a406d990 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f11a8036000 tx=0x7f11a803c040 comp rx=0 tx=0).stop 2024-03-06T07:31:49.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.637+0000 7f11bf3bc640 1 -- 172.21.15.83:0/1369904660 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11b80d29c0 msgr2=0x7f11b81f48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.637+0000 7f11bf3bc640 1 --2- 172.21.15.83:0/1369904660 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11b80d29c0 0x7f11b81f48b0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f11ac00c440 tx=0x7f11ac001e70 comp rx=0 tx=0).stop 2024-03-06T07:31:49.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.637+0000 7f11bf3bc640 1 -- 172.21.15.83:0/1369904660 shutdown_connections 2024-03-06T07:31:49.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.637+0000 7f11bf3bc640 1 --2- 172.21.15.83:0/1369904660 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f11a406b4d0 0x7f11a406d990 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.637+0000 7f11bf3bc640 1 --2- 172.21.15.83:0/1369904660 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11b815dfe0 0x7f11b81f5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.637+0000 7f11bf3bc640 1 --2- 172.21.15.83:0/1369904660 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11b80d29c0 0x7f11b81f48b0 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.637+0000 7f11bf3bc640 1 --2- 172.21.15.83:0/1369904660 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11b80cf830 0x7f11b8157330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.637+0000 7f11bf3bc640 1 -- 172.21.15.83:0/1369904660 >> 172.21.15.83:0/1369904660 conn(0x7f11b80d3cd0 msgr2=0x7f11b806ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:49.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.637+0000 7f11bf3bc640 1 -- 172.21.15.83:0/1369904660 shutdown_connections 2024-03-06T07:31:49.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:49.637+0000 7f11bf3bc640 1 -- 172.21.15.83:0/1369904660 wait complete. 2024-03-06T07:31:49.853 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:31:26.591282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:31:49.853 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:31:50.853 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:50.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.956+0000 7f7e21dc9640 1 Processor -- start 2024-03-06T07:31:50.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.956+0000 7f7e21dc9640 1 -- start start 2024-03-06T07:31:50.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.957+0000 7f7e21dc9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e1c0d2710 0x7f7e1c0b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.957+0000 7f7e21dc9640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e1c0b3430 con 0x7f7e1c0d22b0 2024-03-06T07:31:50.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.957+0000 7f7e21dc9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e1c0b3710 con 0x7f7e1c15dfe0 2024-03-06T07:31:50.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.957+0000 7f7e21dc9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1c0b31e0 con 0x7f7e1c0d2710 2024-03-06T07:31:50.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.957+0000 7f7e1affd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f7e1c0d22b0 0x7f7e1c0b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:34524/0 (socket says 172.21.15.83:34524) 2024-03-06T07:31:50.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.957+0000 7f7e1affd640 1 -- 172.21.15.83:0/377242779 learned_addr learned my addr 172.21.15.83:0/377242779 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:31:50.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.958+0000 7f7e1a7fc640 1 -- 172.21.15.83:0/377242779 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1208492595 0 0) 0x7f7e1c0b3430 con 0x7f7e1c0d22b0 2024-03-06T07:31:50.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.958+0000 7f7e1bfff640 1 --2- 172.21.15.83:0/377242779 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e1c0d2710 0x7f7e1c0b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.958+0000 7f7e1a7fc640 1 -- 172.21.15.83:0/377242779 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e04003520 con 0x7f7e1c0d22b0 2024-03-06T07:31:50.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.958+0000 7f7e1a7fc640 1 -- 172.21.15.83:0/377242779 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2091084881 0 0) 0x7f7e04003520 con 0x7f7e1c0d22b0 2024-03-06T07:31:50.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.958+0000 7f7e1a7fc640 1 -- 172.21.15.83:0/377242779 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e1c0d2710 msgr2=0x7f7e1c0b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:50.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.958+0000 7f7e1a7fc640 1 --2- 172.21.15.83:0/377242779 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e1c0d2710 0x7f7e1c0b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.958+0000 7f7e1a7fc640 1 -- 172.21.15.83:0/377242779 >> v1:172.21.15.183:6789/0 conn(0x7f7e1c15dfe0 legacy=0x7f7e1c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.958+0000 7f7e1a7fc640 1 -- 172.21.15.83:0/377242779 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e1c0b39f0 con 0x7f7e1c0d22b0 2024-03-06T07:31:50.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.958+0000 7f7e1a7fc640 1 -- 172.21.15.83:0/377242779 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f7e0c001670 con 0x7f7e1c0d22b0 2024-03-06T07:31:50.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.958+0000 7f7e1a7fc640 1 -- 172.21.15.83:0/377242779 >> v1:172.21.15.83:6789/0 conn(0x7f7e1c0d22b0 legacy=0x7f7e1c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.958+0000 7f7e1a7fc640 1 --2- 172.21.15.83:0/377242779 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e1c0d2710 0x7f7e04003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.958+0000 7f7e1a7fc640 1 --2- 172.21.15.83:0/377242779 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e04003f80 0x7f7e04004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.958+0000 7f7e1a7fc640 1 --2- 172.21.15.83:0/377242779 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e04004d40 0x7f7e040051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.958+0000 7f7e1a7fc640 1 -- 172.21.15.83:0/377242779 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1c0b31e0 con 0x7f7e04003f80 2024-03-06T07:31:50.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.958+0000 7f7e1a7fc640 1 -- 172.21.15.83:0/377242779 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0400bad0 con 0x7f7e04004d40 2024-03-06T07:31:50.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.958+0000 7f7e1a7fc640 1 -- 172.21.15.83:0/377242779 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0400bcd0 con 0x7f7e1c0d2710 2024-03-06T07:31:50.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.958+0000 7f7e1affd640 1 --2- 172.21.15.83:0/377242779 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e04003f80 0x7f7e04004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.958+0000 7f7e1bfff640 1 --2- 172.21.15.83:0/377242779 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e04004d40 0x7f7e040051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.958+0000 7f7e1b7fe640 1 --2- 172.21.15.83:0/377242779 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e1c0d2710 0x7f7e04003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.958+0000 7f7e1affd640 1 -- 172.21.15.83:0/377242779 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e1c0d2710 msgr2=0x7f7e04003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:50.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.958+0000 7f7e1affd640 1 --2- 172.21.15.83:0/377242779 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e1c0d2710 0x7f7e04003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.958+0000 7f7e1affd640 1 -- 172.21.15.83:0/377242779 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e04004d40 msgr2=0x7f7e040051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:50.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.958+0000 7f7e1affd640 1 --2- 172.21.15.83:0/377242779 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e04004d40 0x7f7e040051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.958+0000 7f7e1affd640 1 -- 172.21.15.83:0/377242779 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e04003520 con 0x7f7e04003f80 2024-03-06T07:31:50.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.960+0000 7f7e1affd640 1 --2- 172.21.15.83:0/377242779 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e04003f80 0x7f7e04004800 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c00c210 tx=0x7f7e04005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf129835bf4986a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:50.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.960+0000 7f7e1a7fc640 1 -- 172.21.15.83:0/377242779 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e0c007d10 con 0x7f7e04003f80 2024-03-06T07:31:50.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.960+0000 7f7e1a7fc640 1 -- 172.21.15.83:0/377242779 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e0c030070 con 0x7f7e04003f80 2024-03-06T07:31:50.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.960+0000 7f7e1a7fc640 1 -- 172.21.15.83:0/377242779 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e0c005d90 con 0x7f7e04003f80 2024-03-06T07:31:50.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.960+0000 7f7e21dc9640 1 -- 172.21.15.83:0/377242779 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e04003f80 msgr2=0x7f7e04004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.960+0000 7f7e21dc9640 1 --2- 172.21.15.83:0/377242779 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e04003f80 0x7f7e04004800 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c00c210 tx=0x7f7e04005830 comp rx=0 tx=0).stop 2024-03-06T07:31:50.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.960+0000 7f7e1bfff640 1 -- 172.21.15.83:0/377242779 reap_dead start 2024-03-06T07:31:50.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.960+0000 7f7e21dc9640 1 -- 172.21.15.83:0/377242779 shutdown_connections 2024-03-06T07:31:50.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.960+0000 7f7e21dc9640 1 -- 172.21.15.83:0/377242779 >> 172.21.15.83:0/377242779 conn(0x7f7e1c0d3cd0 msgr2=0x7f7e1c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:50.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.960+0000 7f7e21dc9640 1 -- 172.21.15.83:0/377242779 shutdown_connections 2024-03-06T07:31:50.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.960+0000 7f7e21dc9640 1 -- 172.21.15.83:0/377242779 wait complete. 2024-03-06T07:31:50.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.961+0000 7f7e21dc9640 1 Processor -- start 2024-03-06T07:31:50.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.962+0000 7f7e21dc9640 1 -- start start 2024-03-06T07:31:50.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.962+0000 7f7e21dc9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e04003f80 0x7f7e1c0a1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.962+0000 7f7e21dc9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e1c0b10e0 0x7f7e1c0b4320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.962+0000 7f7e21dc9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e1c0b4860 0x7f7e1c0ccf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.962+0000 7f7e21dc9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1c0cfa00 con 0x7f7e04003f80 2024-03-06T07:31:50.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.962+0000 7f7e21dc9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1c0b3d60 con 0x7f7e1c0b10e0 2024-03-06T07:31:50.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.962+0000 7f7e21dc9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1c0d1c70 con 0x7f7e1c0b4860 2024-03-06T07:31:50.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.962+0000 7f7e1b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e04003f80 0x7f7e1c0a1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.962+0000 7f7e1b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e04003f80 0x7f7e1c0a1480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58590/0 (socket says 172.21.15.83:58590) 2024-03-06T07:31:50.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.962+0000 7f7e1b7fe640 1 -- 172.21.15.83:0/130640185 learned_addr learned my addr 172.21.15.83:0/130640185 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:50.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.963+0000 7f7e1affd640 1 --2- 172.21.15.83:0/130640185 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e1c0b10e0 0x7f7e1c0b4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.963+0000 7f7e1bfff640 1 --2- 172.21.15.83:0/130640185 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e1c0b4860 0x7f7e1c0ccf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.963+0000 7f7e1b7fe640 1 -- 172.21.15.83:0/130640185 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e1c0b4860 msgr2=0x7f7e1c0ccf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.963+0000 7f7e1b7fe640 1 --2- 172.21.15.83:0/130640185 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e1c0b4860 0x7f7e1c0ccf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.963+0000 7f7e1b7fe640 1 -- 172.21.15.83:0/130640185 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e1c0b10e0 msgr2=0x7f7e1c0b4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.963+0000 7f7e1b7fe640 1 --2- 172.21.15.83:0/130640185 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e1c0b10e0 0x7f7e1c0b4320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.963+0000 7f7e1b7fe640 1 -- 172.21.15.83:0/130640185 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e1c0a1b10 con 0x7f7e04003f80 2024-03-06T07:31:50.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.963+0000 7f7e1b7fe640 1 --2- 172.21.15.83:0/130640185 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e04003f80 0x7f7e1c0a1480 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f7e1000f880 tx=0x7f7e1000fd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:50.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.963+0000 7f7e03fff640 1 -- 172.21.15.83:0/130640185 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e10001d20 con 0x7f7e04003f80 2024-03-06T07:31:50.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.963+0000 7f7e03fff640 1 -- 172.21.15.83:0/130640185 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e10018070 con 0x7f7e04003f80 2024-03-06T07:31:50.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.963+0000 7f7e21dc9640 1 -- 172.21.15.83:0/130640185 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e1c1f4480 con 0x7f7e04003f80 2024-03-06T07:31:50.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.963+0000 7f7e03fff640 1 -- 172.21.15.83:0/130640185 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1000d730 con 0x7f7e04003f80 2024-03-06T07:31:50.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.963+0000 7f7e21dc9640 1 -- 172.21.15.83:0/130640185 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e1c1f39a0 con 0x7f7e04003f80 2024-03-06T07:31:50.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.964+0000 7f7e21dc9640 1 -- 172.21.15.83:0/130640185 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e1c04f330 con 0x7f7e04003f80 2024-03-06T07:31:50.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.965+0000 7f7e03fff640 1 -- 172.21.15.83:0/130640185 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7e1000d8d0 con 0x7f7e04003f80 2024-03-06T07:31:50.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.965+0000 7f7e03fff640 1 --2- 172.21.15.83:0/130640185 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7e0806b4d0 0x7f7e0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.965+0000 7f7e03fff640 1 -- 172.21.15.83:0/130640185 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7e10089e80 con 0x7f7e04003f80 2024-03-06T07:31:50.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.965+0000 7f7e1affd640 1 --2- 172.21.15.83:0/130640185 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7e0806b4d0 0x7f7e0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.965+0000 7f7e1affd640 1 --2- 172.21.15.83:0/130640185 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7e0806b4d0 0x7f7e0806d990 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c034000 tx=0x7f7e0c009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:50.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:50.970+0000 7f7e03fff640 1 -- 172.21.15.83:0/130640185 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7e10058700 con 0x7f7e04003f80 2024-03-06T07:31:51.110 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:51.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:51.184+0000 7f7e21dc9640 1 -- 172.21.15.83:0/130640185 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e1c0b4c40 con 0x7f7e04003f80 2024-03-06T07:31:51.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:51.184+0000 7f7e03fff640 1 -- 172.21.15.83:0/130640185 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f7e1005c7a0 con 0x7f7e04003f80 2024-03-06T07:31:51.187 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:31:51.187 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:31:51.187 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:31:51.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:51.186+0000 7f7e21dc9640 1 -- 172.21.15.83:0/130640185 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7e0806b4d0 msgr2=0x7f7e0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:51.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:51.186+0000 7f7e21dc9640 1 --2- 172.21.15.83:0/130640185 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7e0806b4d0 0x7f7e0806d990 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c034000 tx=0x7f7e0c009030 comp rx=0 tx=0).stop 2024-03-06T07:31:51.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:51.186+0000 7f7e21dc9640 1 -- 172.21.15.83:0/130640185 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e04003f80 msgr2=0x7f7e1c0a1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:51.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:51.186+0000 7f7e21dc9640 1 --2- 172.21.15.83:0/130640185 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e04003f80 0x7f7e1c0a1480 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f7e1000f880 tx=0x7f7e1000fd40 comp rx=0 tx=0).stop 2024-03-06T07:31:51.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:51.186+0000 7f7e21dc9640 1 -- 172.21.15.83:0/130640185 shutdown_connections 2024-03-06T07:31:51.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:51.186+0000 7f7e21dc9640 1 --2- 172.21.15.83:0/130640185 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7e0806b4d0 0x7f7e0806d990 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:51.186+0000 7f7e21dc9640 1 --2- 172.21.15.83:0/130640185 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e1c0b4860 0x7f7e1c0ccf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:51.186+0000 7f7e21dc9640 1 --2- 172.21.15.83:0/130640185 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e1c0b10e0 0x7f7e1c0b4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:51.186+0000 7f7e21dc9640 1 --2- 172.21.15.83:0/130640185 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e04003f80 0x7f7e1c0a1480 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:51.186+0000 7f7e21dc9640 1 -- 172.21.15.83:0/130640185 >> 172.21.15.83:0/130640185 conn(0x7f7e1c0d3cd0 msgr2=0x7f7e1c0b2110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:51.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:51.186+0000 7f7e21dc9640 1 -- 172.21.15.83:0/130640185 shutdown_connections 2024-03-06T07:31:51.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:51.186+0000 7f7e21dc9640 1 -- 172.21.15.83:0/130640185 wait complete. 2024-03-06T07:31:51.400 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:31:26.591282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:31:51.400 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:31:52.401 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:52.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.500+0000 7ffaf839a640 1 Processor -- start 2024-03-06T07:31:52.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.500+0000 7ffaf839a640 1 -- start start 2024-03-06T07:31:52.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.502+0000 7ffaf839a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ffaf00d2710 0x7ffaf00b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.502+0000 7ffaf839a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf00b3430 con 0x7ffaf015dfe0 2024-03-06T07:31:52.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.502+0000 7ffaf839a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf00b3710 con 0x7ffaf00d22b0 2024-03-06T07:31:52.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.502+0000 7ffaf839a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaf00b31e0 con 0x7ffaf00d2710 2024-03-06T07:31:52.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.502+0000 7ffaf6937640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ffaf00d2710 0x7ffaf00b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.502+0000 7ffaf6937640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ffaf00d2710 0x7ffaf00b4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58598/0 (socket says 172.21.15.83:58598) 2024-03-06T07:31:52.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.502+0000 7ffaf6937640 1 -- 172.21.15.83:0/1120779339 learned_addr learned my addr 172.21.15.83:0/1120779339 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:52.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.502+0000 7ffaf6136640 1 --1- 172.21.15.83:0/1120779339 >> v1:172.21.15.83:6789/0 conn(0x7ffaf015dfe0 0x7ffaf00cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:34540/0 (socket says 172.21.15.83:34540) 2024-03-06T07:31:52.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.502+0000 7ffaf6937640 1 -- 172.21.15.83:0/1120779339 >> v1:172.21.15.183:6789/0 conn(0x7ffaf00d22b0 legacy=0x7ffaf00b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:52.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.502+0000 7ffaf5134640 1 -- 172.21.15.83:0/1120779339 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1811368339 0 0) 0x7ffaf00b3430 con 0x7ffaf015dfe0 2024-03-06T07:31:52.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.502+0000 7ffaf6937640 1 -- 172.21.15.83:0/1120779339 >> v1:172.21.15.83:6789/0 conn(0x7ffaf015dfe0 legacy=0x7ffaf00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.502+0000 7ffaf6937640 1 -- 172.21.15.83:0/1120779339 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf00b39f0 con 0x7ffaf00d2710 2024-03-06T07:31:52.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.503+0000 7ffaf6937640 1 --2- 172.21.15.83:0/1120779339 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffaf00d2710 0x7ffaf00b4900 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7ffae8009e80 tx=0x7ffae802bfe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71ac3533fb037ee server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:52.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.503+0000 7ffaf5134640 1 -- 172.21.15.83:0/1120779339 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae802cb80 con 0x7ffaf00d2710 2024-03-06T07:31:52.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.503+0000 7ffaf5134640 1 -- 172.21.15.83:0/1120779339 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffae802cd20 con 0x7ffaf00d2710 2024-03-06T07:31:52.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.503+0000 7ffaf5134640 1 -- 172.21.15.83:0/1120779339 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae8034610 con 0x7ffaf00d2710 2024-03-06T07:31:52.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.504+0000 7ffaf839a640 1 -- 172.21.15.83:0/1120779339 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffaf00d2710 msgr2=0x7ffaf00b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.504+0000 7ffaf839a640 1 --2- 172.21.15.83:0/1120779339 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffaf00d2710 0x7ffaf00b4900 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7ffae8009e80 tx=0x7ffae802bfe0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.504+0000 7ffaf839a640 1 -- 172.21.15.83:0/1120779339 shutdown_connections 2024-03-06T07:31:52.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.504+0000 7ffaf839a640 1 --2- 172.21.15.83:0/1120779339 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffaf00d2710 0x7ffaf00b4900 unknown :-1 s=CLOSED pgs=1908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.504+0000 7ffaf839a640 1 -- 172.21.15.83:0/1120779339 >> 172.21.15.83:0/1120779339 conn(0x7ffaf00d3cd0 msgr2=0x7ffaf00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:52.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.504+0000 7ffaf839a640 1 -- 172.21.15.83:0/1120779339 shutdown_connections 2024-03-06T07:31:52.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.504+0000 7ffaf839a640 1 -- 172.21.15.83:0/1120779339 wait complete. 2024-03-06T07:31:52.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.505+0000 7ffaf839a640 1 Processor -- start 2024-03-06T07:31:52.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.505+0000 7ffaf839a640 1 -- start start 2024-03-06T07:31:52.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.506+0000 7ffaf839a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffaf00d22b0 0x7ffaf01611a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.506+0000 7ffaf839a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffaf00d2710 0x7ffaf00b8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.506+0000 7ffaf839a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffaf015dfe0 0x7ffaf00cc390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.506+0000 7ffaf839a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf00cfa00 con 0x7ffaf015dfe0 2024-03-06T07:31:52.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.506+0000 7ffaf839a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf00b3d60 con 0x7ffaf00d2710 2024-03-06T07:31:52.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.506+0000 7ffaf839a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf00d1c70 con 0x7ffaf00d22b0 2024-03-06T07:31:52.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.506+0000 7ffaf6937640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffaf015dfe0 0x7ffaf00cc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.506+0000 7ffaf6937640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffaf015dfe0 0x7ffaf00cc390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58606/0 (socket says 172.21.15.83:58606) 2024-03-06T07:31:52.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.506+0000 7ffaf6937640 1 -- 172.21.15.83:0/2192579599 learned_addr learned my addr 172.21.15.83:0/2192579599 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:52.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.506+0000 7ffaf6136640 1 --2- 172.21.15.83:0/2192579599 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffaf00d22b0 0x7ffaf01611a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.506+0000 7ffaf6937640 1 -- 172.21.15.83:0/2192579599 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffaf00d22b0 msgr2=0x7ffaf01611a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.506+0000 7ffaf5935640 1 --2- 172.21.15.83:0/2192579599 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffaf00d2710 0x7ffaf00b8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.506+0000 7ffaf6937640 1 --2- 172.21.15.83:0/2192579599 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffaf00d22b0 0x7ffaf01611a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.506+0000 7ffaf6937640 1 -- 172.21.15.83:0/2192579599 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffaf00d2710 msgr2=0x7ffaf00b8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.506+0000 7ffaf6937640 1 --2- 172.21.15.83:0/2192579599 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffaf00d2710 0x7ffaf00b8010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.506+0000 7ffaf6937640 1 -- 172.21.15.83:0/2192579599 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf00b8880 con 0x7ffaf015dfe0 2024-03-06T07:31:52.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.506+0000 7ffaf6937640 1 --2- 172.21.15.83:0/2192579599 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffaf015dfe0 0x7ffaf00cc390 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7ffae8009fa0 tx=0x7ffae8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:52.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.506+0000 7ffae6ffd640 1 -- 172.21.15.83:0/2192579599 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae8043020 con 0x7ffaf015dfe0 2024-03-06T07:31:52.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.506+0000 7ffae6ffd640 1 -- 172.21.15.83:0/2192579599 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffae803e450 con 0x7ffaf015dfe0 2024-03-06T07:31:52.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.507+0000 7ffae6ffd640 1 -- 172.21.15.83:0/2192579599 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae8009510 con 0x7ffaf015dfe0 2024-03-06T07:31:52.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.507+0000 7ffaf839a640 1 -- 172.21.15.83:0/2192579599 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaf00b8b60 con 0x7ffaf015dfe0 2024-03-06T07:31:52.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.507+0000 7ffaf839a640 1 -- 172.21.15.83:0/2192579599 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaf00b9400 con 0x7ffaf015dfe0 2024-03-06T07:31:52.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.507+0000 7ffaf839a640 1 -- 172.21.15.83:0/2192579599 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaf004f330 con 0x7ffaf015dfe0 2024-03-06T07:31:52.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.508+0000 7ffae6ffd640 1 -- 172.21.15.83:0/2192579599 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffae8046020 con 0x7ffaf015dfe0 2024-03-06T07:31:52.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.508+0000 7ffae6ffd640 1 --2- 172.21.15.83:0/2192579599 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffad406b500 0x7ffad406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.508+0000 7ffae6ffd640 1 -- 172.21.15.83:0/2192579599 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ffae80ac790 con 0x7ffaf015dfe0 2024-03-06T07:31:52.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.508+0000 7ffaf6136640 1 --2- 172.21.15.83:0/2192579599 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffad406b500 0x7ffad406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.509+0000 7ffaf6136640 1 --2- 172.21.15.83:0/2192579599 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffad406b500 0x7ffad406d9c0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7ffaec000f60 tx=0x7ffaec009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:52.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.513+0000 7ffae6ffd640 1 -- 172.21.15.83:0/2192579599 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffae804d070 con 0x7ffaf015dfe0 2024-03-06T07:31:52.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.726+0000 7ffaf839a640 1 -- 172.21.15.83:0/2192579599 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffaf015d2e0 con 0x7ffaf015dfe0 2024-03-06T07:31:52.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.726+0000 7ffae6ffd640 1 -- 172.21.15.83:0/2192579599 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7ffae807b660 con 0x7ffaf015dfe0 2024-03-06T07:31:52.728 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:31:52.728 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:31:52.729 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:31:52.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.728+0000 7ffaf839a640 1 -- 172.21.15.83:0/2192579599 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffad406b500 msgr2=0x7ffad406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.728+0000 7ffaf839a640 1 --2- 172.21.15.83:0/2192579599 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffad406b500 0x7ffad406d9c0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7ffaec000f60 tx=0x7ffaec009040 comp rx=0 tx=0).stop 2024-03-06T07:31:52.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.728+0000 7ffaf839a640 1 -- 172.21.15.83:0/2192579599 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffaf015dfe0 msgr2=0x7ffaf00cc390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.728+0000 7ffaf839a640 1 --2- 172.21.15.83:0/2192579599 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffaf015dfe0 0x7ffaf00cc390 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7ffae8009fa0 tx=0x7ffae8004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:52.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.728+0000 7ffaf839a640 1 -- 172.21.15.83:0/2192579599 shutdown_connections 2024-03-06T07:31:52.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.728+0000 7ffaf839a640 1 --2- 172.21.15.83:0/2192579599 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ffad406b500 0x7ffad406d9c0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.728+0000 7ffaf839a640 1 --2- 172.21.15.83:0/2192579599 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffaf015dfe0 0x7ffaf00cc390 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.728+0000 7ffaf839a640 1 --2- 172.21.15.83:0/2192579599 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffaf00d2710 0x7ffaf00b8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.728+0000 7ffaf839a640 1 --2- 172.21.15.83:0/2192579599 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffaf00d22b0 0x7ffaf01611a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.728+0000 7ffaf839a640 1 -- 172.21.15.83:0/2192579599 >> 172.21.15.83:0/2192579599 conn(0x7ffaf00d3cd0 msgr2=0x7ffaf006cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:52.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.728+0000 7ffaf839a640 1 -- 172.21.15.83:0/2192579599 shutdown_connections 2024-03-06T07:31:52.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:52.728+0000 7ffaf839a640 1 -- 172.21.15.83:0/2192579599 wait complete. 2024-03-06T07:31:52.942 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:31:26.591282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:31:52.942 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:31:53.112 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:53.943 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:54.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.043+0000 7f450e393640 1 Processor -- start 2024-03-06T07:31:54.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.043+0000 7f450e393640 1 -- start start 2024-03-06T07:31:54.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.044+0000 7f450e393640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f450815bb20 0x7f45080d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.044+0000 7f450e393640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f45080cfff0 0x7f45080d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.044+0000 7f450e393640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f45080ce4b0 0x7f45080ce920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.044+0000 7f450e393640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4508161980 con 0x7f450815bb20 2024-03-06T07:31:54.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.044+0000 7f450e393640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4508161c60 con 0x7f45080cfff0 2024-03-06T07:31:54.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.044+0000 7f450e393640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4508161f40 con 0x7f45080ce4b0 2024-03-06T07:31:54.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.044+0000 7f4507fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f450815bb20 0x7f45080d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.044+0000 7f4507fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f450815bb20 0x7f45080d1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58622/0 (socket says 172.21.15.83:58622) 2024-03-06T07:31:54.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.044+0000 7f4507fff640 1 -- 172.21.15.83:0/1628246319 learned_addr learned my addr 172.21.15.83:0/1628246319 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:54.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.044+0000 7f45077fe640 1 --2- 172.21.15.83:0/1628246319 >> v2:172.21.15.183:3300/0 conn(0x7f45080cfff0 0x7f45080d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.044+0000 7f450c930640 1 --2- 172.21.15.83:0/1628246319 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f45080ce4b0 0x7f45080ce920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.044+0000 7f4507fff640 1 -- 172.21.15.83:0/1628246319 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f45080ce4b0 msgr2=0x7f45080ce920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:54.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.044+0000 7f4507fff640 1 --2- 172.21.15.83:0/1628246319 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f45080ce4b0 0x7f45080ce920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.044+0000 7f4507fff640 1 -- 172.21.15.83:0/1628246319 >> v2:172.21.15.183:3300/0 conn(0x7f45080cfff0 msgr2=0x7f45080d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:54.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.044+0000 7f4507fff640 1 --2- 172.21.15.83:0/1628246319 >> v2:172.21.15.183:3300/0 conn(0x7f45080cfff0 0x7f45080d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.044+0000 7f4507fff640 1 -- 172.21.15.83:0/1628246319 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4508162220 con 0x7f450815bb20 2024-03-06T07:31:54.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.044+0000 7f45077fe640 1 --2- 172.21.15.83:0/1628246319 >> v2:172.21.15.183:3300/0 conn(0x7f45080cfff0 0x7f45080d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:54.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.045+0000 7f4507fff640 1 --2- 172.21.15.83:0/1628246319 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f450815bb20 0x7f45080d1df0 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f4508071290 tx=0x7f44f802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57083136598e7d20 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:54.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.045+0000 7f4506ffd640 1 -- 172.21.15.83:0/1628246319 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44f8035020 con 0x7f450815bb20 2024-03-06T07:31:54.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.045+0000 7f4506ffd640 1 -- 172.21.15.83:0/1628246319 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44f8031870 con 0x7f450815bb20 2024-03-06T07:31:54.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.045+0000 7f4506ffd640 1 -- 172.21.15.83:0/1628246319 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44f8031c40 con 0x7f450815bb20 2024-03-06T07:31:54.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.046+0000 7f450e393640 1 -- 172.21.15.83:0/1628246319 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f450815bb20 msgr2=0x7f45080d1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.046+0000 7f450e393640 1 --2- 172.21.15.83:0/1628246319 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f450815bb20 0x7f45080d1df0 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f4508071290 tx=0x7f44f802b910 comp rx=0 tx=0).stop 2024-03-06T07:31:54.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.046+0000 7f450e393640 1 -- 172.21.15.83:0/1628246319 shutdown_connections 2024-03-06T07:31:54.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.046+0000 7f450e393640 1 --2- 172.21.15.83:0/1628246319 >> v2:172.21.15.183:3300/0 conn(0x7f45080cfff0 0x7f45080d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.046+0000 7f450e393640 1 --2- 172.21.15.83:0/1628246319 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f45080ce4b0 0x7f45080ce920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.046+0000 7f450e393640 1 --2- 172.21.15.83:0/1628246319 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f450815bb20 0x7f45080d1df0 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.046+0000 7f450e393640 1 -- 172.21.15.83:0/1628246319 >> 172.21.15.83:0/1628246319 conn(0x7f450809a9f0 msgr2=0x7f450809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:54.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.046+0000 7f450e393640 1 -- 172.21.15.83:0/1628246319 shutdown_connections 2024-03-06T07:31:54.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.046+0000 7f450e393640 1 -- 172.21.15.83:0/1628246319 wait complete. 2024-03-06T07:31:54.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.047+0000 7f450e393640 1 Processor -- start 2024-03-06T07:31:54.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.047+0000 7f450e393640 1 -- start start 2024-03-06T07:31:54.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.048+0000 7f450e393640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f45080ce4b0 0x7f450806ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.048+0000 7f450e393640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45080cfff0 0x7f45080b6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.048+0000 7f450e393640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f450815bb20 0x7f45080af940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.048+0000 7f450e393640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45080d24b0 con 0x7f45080cfff0 2024-03-06T07:31:54.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.048+0000 7f450e393640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45080d2330 con 0x7f450815bb20 2024-03-06T07:31:54.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.048+0000 7f450e393640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45080d1980 con 0x7f45080ce4b0 2024-03-06T07:31:54.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.048+0000 7f45077fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45080cfff0 0x7f45080b6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.048+0000 7f45077fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45080cfff0 0x7f45080b6ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58638/0 (socket says 172.21.15.83:58638) 2024-03-06T07:31:54.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.048+0000 7f45077fe640 1 -- 172.21.15.83:0/363382825 learned_addr learned my addr 172.21.15.83:0/363382825 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:54.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.048+0000 7f4507fff640 1 --2- 172.21.15.83:0/363382825 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f45080ce4b0 0x7f450806ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.048+0000 7f450c930640 1 --2- 172.21.15.83:0/363382825 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f450815bb20 0x7f45080af940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.048+0000 7f45077fe640 1 -- 172.21.15.83:0/363382825 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f45080ce4b0 msgr2=0x7f450806ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.048+0000 7f45077fe640 1 --2- 172.21.15.83:0/363382825 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f45080ce4b0 0x7f450806ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.048+0000 7f45077fe640 1 -- 172.21.15.83:0/363382825 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f450815bb20 msgr2=0x7f45080af940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.048+0000 7f45077fe640 1 --2- 172.21.15.83:0/363382825 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f450815bb20 0x7f45080af940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.048+0000 7f45077fe640 1 -- 172.21.15.83:0/363382825 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4508149c90 con 0x7f45080cfff0 2024-03-06T07:31:54.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.048+0000 7f450c930640 1 --2- 172.21.15.83:0/363382825 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f450815bb20 0x7f45080af940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:54.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.048+0000 7f45077fe640 1 --2- 172.21.15.83:0/363382825 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45080cfff0 0x7f45080b6ce0 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f44fc00e7e0 tx=0x7f44fc00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:54.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.049+0000 7f4504ff9640 1 -- 172.21.15.83:0/363382825 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44fc00f040 con 0x7f45080cfff0 2024-03-06T07:31:54.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.049+0000 7f4504ff9640 1 -- 172.21.15.83:0/363382825 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44fc004030 con 0x7f45080cfff0 2024-03-06T07:31:54.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.049+0000 7f450e393640 1 -- 172.21.15.83:0/363382825 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4508147930 con 0x7f45080cfff0 2024-03-06T07:31:54.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.049+0000 7f4504ff9640 1 -- 172.21.15.83:0/363382825 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44fc004400 con 0x7f45080cfff0 2024-03-06T07:31:54.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.049+0000 7f450e393640 1 -- 172.21.15.83:0/363382825 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f450806bf70 con 0x7f45080cfff0 2024-03-06T07:31:54.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.050+0000 7f450e393640 1 -- 172.21.15.83:0/363382825 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f450804f330 con 0x7f45080cfff0 2024-03-06T07:31:54.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.050+0000 7f4504ff9640 1 -- 172.21.15.83:0/363382825 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f44fc00c6d0 con 0x7f45080cfff0 2024-03-06T07:31:54.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.051+0000 7f4504ff9640 1 --2- 172.21.15.83:0/363382825 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f44f406b4d0 0x7f44f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.051+0000 7f4507fff640 1 --2- 172.21.15.83:0/363382825 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f44f406b4d0 0x7f44f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.051+0000 7f4504ff9640 1 -- 172.21.15.83:0/363382825 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f44fc089f50 con 0x7f45080cfff0 2024-03-06T07:31:54.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.051+0000 7f4507fff640 1 --2- 172.21.15.83:0/363382825 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f44f406b4d0 0x7f44f406d990 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f44f8036000 tx=0x7f44f803c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:54.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.055+0000 7f4504ff9640 1 -- 172.21.15.83:0/363382825 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f44fc017020 con 0x7f45080cfff0 2024-03-06T07:31:54.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.269+0000 7f450e393640 1 -- 172.21.15.83:0/363382825 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45080747a0 con 0x7f45080cfff0 2024-03-06T07:31:54.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.270+0000 7f4504ff9640 1 -- 172.21.15.83:0/363382825 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f44fc058740 con 0x7f45080cfff0 2024-03-06T07:31:54.272 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:31:54.272 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:31:54.272 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:31:54.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.272+0000 7f450e393640 1 -- 172.21.15.83:0/363382825 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f44f406b4d0 msgr2=0x7f44f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.272+0000 7f450e393640 1 --2- 172.21.15.83:0/363382825 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f44f406b4d0 0x7f44f406d990 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f44f8036000 tx=0x7f44f803c040 comp rx=0 tx=0).stop 2024-03-06T07:31:54.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.272+0000 7f450e393640 1 -- 172.21.15.83:0/363382825 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45080cfff0 msgr2=0x7f45080b6ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.272+0000 7f450e393640 1 --2- 172.21.15.83:0/363382825 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45080cfff0 0x7f45080b6ce0 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f44fc00e7e0 tx=0x7f44fc00eca0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.272+0000 7f450e393640 1 -- 172.21.15.83:0/363382825 shutdown_connections 2024-03-06T07:31:54.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.272+0000 7f450e393640 1 --2- 172.21.15.83:0/363382825 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f44f406b4d0 0x7f44f406d990 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.272+0000 7f450e393640 1 --2- 172.21.15.83:0/363382825 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f450815bb20 0x7f45080af940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.272+0000 7f450e393640 1 --2- 172.21.15.83:0/363382825 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45080cfff0 0x7f45080b6ce0 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.272+0000 7f450e393640 1 --2- 172.21.15.83:0/363382825 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f45080ce4b0 0x7f450806ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.272+0000 7f450e393640 1 -- 172.21.15.83:0/363382825 >> 172.21.15.83:0/363382825 conn(0x7f450809a9f0 msgr2=0x7f450806eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:54.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.272+0000 7f450e393640 1 -- 172.21.15.83:0/363382825 shutdown_connections 2024-03-06T07:31:54.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:54.272+0000 7f450e393640 1 -- 172.21.15.83:0/363382825 wait complete. 2024-03-06T07:31:54.485 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:31:26.591282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:31:54.485 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:31:55.115 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:55.486 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:55.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.586+0000 7f67051a0640 1 Processor -- start 2024-03-06T07:31:55.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.586+0000 7f67051a0640 1 -- start start 2024-03-06T07:31:55.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.587+0000 7f67051a0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f67000ae180 0x7f67001687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.587+0000 7f67051a0640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f670015e180 con 0x7f670006b6c0 2024-03-06T07:31:55.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.587+0000 7f67051a0640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67000ae680 con 0x7f670006f060 2024-03-06T07:31:55.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.587+0000 7f67051a0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f670015e440 con 0x7f67000ae180 2024-03-06T07:31:55.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.587+0000 7f66ff7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f670006b6c0 0x7f67000ada70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:34544/0 (socket says 172.21.15.83:34544) 2024-03-06T07:31:55.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.587+0000 7f66ff7fe640 1 -- 172.21.15.83:0/2670128477 learned_addr learned my addr 172.21.15.83:0/2670128477 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:31:55.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.587+0000 7f66feffd640 1 -- 172.21.15.83:0/2670128477 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 309475298 0 0) 0x7f670015e180 con 0x7f670006b6c0 2024-03-06T07:31:55.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.587+0000 7f66feffd640 1 -- 172.21.15.83:0/2670128477 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66ec003520 con 0x7f670006b6c0 2024-03-06T07:31:55.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.587+0000 7f670499f640 1 --2- 172.21.15.83:0/2670128477 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f67000ae180 0x7f67001687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.588+0000 7f66feffd640 1 -- 172.21.15.83:0/2670128477 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3780168752 0 0) 0x7f66ec003520 con 0x7f670006b6c0 2024-03-06T07:31:55.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.588+0000 7f66feffd640 1 -- 172.21.15.83:0/2670128477 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f67000ae180 msgr2=0x7f67001687b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:55.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.588+0000 7f66feffd640 1 --2- 172.21.15.83:0/2670128477 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f67000ae180 0x7f67001687b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.588+0000 7f66feffd640 1 -- 172.21.15.83:0/2670128477 >> v1:172.21.15.183:6789/0 conn(0x7f670006f060 legacy=0x7f670006afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.588+0000 7f66feffd640 1 -- 172.21.15.83:0/2670128477 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f670015e720 con 0x7f670006b6c0 2024-03-06T07:31:55.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.588+0000 7f66feffd640 1 -- 172.21.15.83:0/2670128477 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f66f0001670 con 0x7f670006b6c0 2024-03-06T07:31:55.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.588+0000 7f66feffd640 1 -- 172.21.15.83:0/2670128477 >> v1:172.21.15.83:6789/0 conn(0x7f670006b6c0 legacy=0x7f67000ada70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.588+0000 7f66feffd640 1 --2- 172.21.15.83:0/2670128477 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66ec003a50 0x7f66ec003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.588+0000 7f66feffd640 1 --2- 172.21.15.83:0/2670128477 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f67000ae180 0x7f66ec004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.588+0000 7f66feffd640 1 --2- 172.21.15.83:0/2670128477 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f66ec004d40 0x7f66ec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.588+0000 7f66feffd640 1 -- 172.21.15.83:0/2670128477 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f670015e440 con 0x7f66ec003a50 2024-03-06T07:31:55.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.588+0000 7f66feffd640 1 -- 172.21.15.83:0/2670128477 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ec00bb50 con 0x7f66ec004d40 2024-03-06T07:31:55.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.588+0000 7f66feffd640 1 -- 172.21.15.83:0/2670128477 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66ec00bd50 con 0x7f67000ae180 2024-03-06T07:31:55.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.588+0000 7f66fffff640 1 --2- 172.21.15.83:0/2670128477 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66ec003a50 0x7f66ec003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.588+0000 7f66ff7fe640 1 --2- 172.21.15.83:0/2670128477 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f67000ae180 0x7f66ec004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.588+0000 7f670499f640 1 --2- 172.21.15.83:0/2670128477 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f66ec004d40 0x7f66ec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.588+0000 7f66fffff640 1 -- 172.21.15.83:0/2670128477 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f67000ae180 msgr2=0x7f66ec004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:55.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.588+0000 7f66fffff640 1 --2- 172.21.15.83:0/2670128477 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f67000ae180 0x7f66ec004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.588+0000 7f66fffff640 1 -- 172.21.15.83:0/2670128477 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f66ec004d40 msgr2=0x7f66ec0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:55.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.588+0000 7f66fffff640 1 --2- 172.21.15.83:0/2670128477 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f66ec004d40 0x7f66ec0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.588+0000 7f66fffff640 1 -- 172.21.15.83:0/2670128477 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66ec003520 con 0x7f66ec003a50 2024-03-06T07:31:55.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.588+0000 7f66ff7fe640 1 --2- 172.21.15.83:0/2670128477 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f67000ae180 0x7f66ec004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:55.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.588+0000 7f670499f640 1 --2- 172.21.15.83:0/2670128477 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f66ec004d40 0x7f66ec0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:55.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.589+0000 7f66fffff640 1 --2- 172.21.15.83:0/2670128477 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66ec003a50 0x7f66ec003e70 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f66e8001750 tx=0x7f66ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=344b856e431cd662 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:55.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.589+0000 7f66feffd640 1 -- 172.21.15.83:0/2670128477 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e8001370 con 0x7f66ec003a50 2024-03-06T07:31:55.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.589+0000 7f66feffd640 1 -- 172.21.15.83:0/2670128477 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66e80052d0 con 0x7f66ec003a50 2024-03-06T07:31:55.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.589+0000 7f66feffd640 1 -- 172.21.15.83:0/2670128477 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e8003830 con 0x7f66ec003a50 2024-03-06T07:31:55.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.589+0000 7f67051a0640 1 -- 172.21.15.83:0/2670128477 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66ec003a50 msgr2=0x7f66ec003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.589+0000 7f67051a0640 1 --2- 172.21.15.83:0/2670128477 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66ec003a50 0x7f66ec003e70 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f66e8001750 tx=0x7f66ec005830 comp rx=0 tx=0).stop 2024-03-06T07:31:55.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.589+0000 7f670499f640 1 -- 172.21.15.83:0/2670128477 reap_dead start 2024-03-06T07:31:55.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.589+0000 7f67051a0640 1 -- 172.21.15.83:0/2670128477 shutdown_connections 2024-03-06T07:31:55.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.589+0000 7f67051a0640 1 -- 172.21.15.83:0/2670128477 >> 172.21.15.83:0/2670128477 conn(0x7f67000d0860 msgr2=0x7f67000d0c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:55.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.590+0000 7f67051a0640 1 -- 172.21.15.83:0/2670128477 shutdown_connections 2024-03-06T07:31:55.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.590+0000 7f67051a0640 1 -- 172.21.15.83:0/2670128477 wait complete. 2024-03-06T07:31:55.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.591+0000 7f67051a0640 1 Processor -- start 2024-03-06T07:31:55.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.592+0000 7f67051a0640 1 -- start start 2024-03-06T07:31:55.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.593+0000 7f67051a0640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f66ec003a50 0x7f6700156150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.593+0000 7f67051a0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6700156690 0x7f6700152ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.593+0000 7f67051a0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67001530e0 0x7f670015a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.593+0000 7f67051a0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f670015f0a0 con 0x7f67001530e0 2024-03-06T07:31:55.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.593+0000 7f67051a0640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f670015ef20 con 0x7f66ec003a50 2024-03-06T07:31:55.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.593+0000 7f67051a0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67001667a0 con 0x7f6700156690 2024-03-06T07:31:55.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.593+0000 7f670499f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67001530e0 0x7f670015a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.593+0000 7f670499f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67001530e0 0x7f670015a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58666/0 (socket says 172.21.15.83:58666) 2024-03-06T07:31:55.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.593+0000 7f670499f640 1 -- 172.21.15.83:0/3981563159 learned_addr learned my addr 172.21.15.83:0/3981563159 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:55.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.593+0000 7f66ff7fe640 1 --2- 172.21.15.83:0/3981563159 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6700156690 0x7f6700152ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.593+0000 7f66fffff640 1 --2- 172.21.15.83:0/3981563159 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f66ec003a50 0x7f6700156150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.593+0000 7f670499f640 1 -- 172.21.15.83:0/3981563159 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6700156690 msgr2=0x7f6700152ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.593+0000 7f670499f640 1 --2- 172.21.15.83:0/3981563159 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6700156690 0x7f6700152ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.593+0000 7f670499f640 1 -- 172.21.15.83:0/3981563159 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f66ec003a50 msgr2=0x7f6700156150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.593+0000 7f670499f640 1 --2- 172.21.15.83:0/3981563159 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f66ec003a50 0x7f6700156150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.593+0000 7f670499f640 1 -- 172.21.15.83:0/3981563159 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67001535d0 con 0x7f67001530e0 2024-03-06T07:31:55.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.593+0000 7f670499f640 1 --2- 172.21.15.83:0/3981563159 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67001530e0 0x7f670015a5d0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f67000b5da0 tx=0x7f66f8010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:55.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.593+0000 7f66fcff9640 1 -- 172.21.15.83:0/3981563159 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f801a030 con 0x7f67001530e0 2024-03-06T07:31:55.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.593+0000 7f66fcff9640 1 -- 172.21.15.83:0/3981563159 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66f8007e00 con 0x7f67001530e0 2024-03-06T07:31:55.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.593+0000 7f67051a0640 1 -- 172.21.15.83:0/3981563159 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67001538b0 con 0x7f67001530e0 2024-03-06T07:31:55.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.593+0000 7f66fcff9640 1 -- 172.21.15.83:0/3981563159 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f8003990 con 0x7f67001530e0 2024-03-06T07:31:55.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.593+0000 7f67051a0640 1 -- 172.21.15.83:0/3981563159 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6700154360 con 0x7f67001530e0 2024-03-06T07:31:55.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.594+0000 7f67051a0640 1 -- 172.21.15.83:0/3981563159 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6700068f80 con 0x7f67001530e0 2024-03-06T07:31:55.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.595+0000 7f66fcff9640 1 -- 172.21.15.83:0/3981563159 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f66f801d070 con 0x7f67001530e0 2024-03-06T07:31:55.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.595+0000 7f66fcff9640 1 --2- 172.21.15.83:0/3981563159 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f66e006b4d0 0x7f66e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.595+0000 7f66fcff9640 1 -- 172.21.15.83:0/3981563159 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f66f808ab20 con 0x7f67001530e0 2024-03-06T07:31:55.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.595+0000 7f66fffff640 1 --2- 172.21.15.83:0/3981563159 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f66e006b4d0 0x7f66e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.596+0000 7f66fffff640 1 --2- 172.21.15.83:0/3981563159 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f66e006b4d0 0x7f66e006d990 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f66e8033000 tx=0x7f66e8009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:55.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.601+0000 7f66fcff9640 1 -- 172.21.15.83:0/3981563159 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f66f80599f0 con 0x7f67001530e0 2024-03-06T07:31:55.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.808+0000 7f67051a0640 1 -- 172.21.15.83:0/3981563159 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f67001546d0 con 0x7f67001530e0 2024-03-06T07:31:55.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.809+0000 7f66fcff9640 1 -- 172.21.15.83:0/3981563159 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f66f80592f0 con 0x7f67001530e0 2024-03-06T07:31:55.811 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:31:55.811 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:31:55.811 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:31:55.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.811+0000 7f67051a0640 1 -- 172.21.15.83:0/3981563159 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f66e006b4d0 msgr2=0x7f66e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.811+0000 7f67051a0640 1 --2- 172.21.15.83:0/3981563159 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f66e006b4d0 0x7f66e006d990 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f66e8033000 tx=0x7f66e8009030 comp rx=0 tx=0).stop 2024-03-06T07:31:55.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.811+0000 7f67051a0640 1 -- 172.21.15.83:0/3981563159 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67001530e0 msgr2=0x7f670015a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.811+0000 7f67051a0640 1 --2- 172.21.15.83:0/3981563159 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67001530e0 0x7f670015a5d0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f67000b5da0 tx=0x7f66f8010590 comp rx=0 tx=0).stop 2024-03-06T07:31:55.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.811+0000 7f67051a0640 1 -- 172.21.15.83:0/3981563159 shutdown_connections 2024-03-06T07:31:55.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.811+0000 7f67051a0640 1 --2- 172.21.15.83:0/3981563159 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f66e006b4d0 0x7f66e006d990 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.811+0000 7f67051a0640 1 --2- 172.21.15.83:0/3981563159 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67001530e0 0x7f670015a5d0 unknown :-1 s=CLOSED pgs=1914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.811+0000 7f67051a0640 1 --2- 172.21.15.83:0/3981563159 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6700156690 0x7f6700152ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.811+0000 7f67051a0640 1 --2- 172.21.15.83:0/3981563159 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f66ec003a50 0x7f6700156150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.811+0000 7f67051a0640 1 -- 172.21.15.83:0/3981563159 >> 172.21.15.83:0/3981563159 conn(0x7f67000d0860 msgr2=0x7f67000b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:55.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.811+0000 7f67051a0640 1 -- 172.21.15.83:0/3981563159 shutdown_connections 2024-03-06T07:31:55.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:55.811+0000 7f67051a0640 1 -- 172.21.15.83:0/3981563159 wait complete. 2024-03-06T07:31:56.026 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:31:26.591282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:31:56.026 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:31:57.027 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:57.115 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:57.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.158+0000 7effb955e640 1 Processor -- start 2024-03-06T07:31:57.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.158+0000 7effb955e640 1 -- start start 2024-03-06T07:31:57.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.158+0000 7effb955e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7effb40d22b0 0x7effb40d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.159+0000 7effb955e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effb40b3430 con 0x7effb415dfe0 2024-03-06T07:31:57.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.159+0000 7effb955e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effb40b3710 con 0x7effb40b41f0 2024-03-06T07:31:57.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.159+0000 7effb955e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effb40d2c50 con 0x7effb40d22b0 2024-03-06T07:31:57.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.159+0000 7effb2ffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7effb415dfe0 0x7effb40cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:38158/0 (socket says 172.21.15.83:38158) 2024-03-06T07:31:57.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.159+0000 7effb2ffd640 1 -- 172.21.15.83:0/3236172060 learned_addr learned my addr 172.21.15.83:0/3236172060 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:31:57.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.159+0000 7effb27fc640 1 -- 172.21.15.83:0/3236172060 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3182005650 0 0) 0x7effb40b3430 con 0x7effb415dfe0 2024-03-06T07:31:57.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.159+0000 7effb27fc640 1 -- 172.21.15.83:0/3236172060 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effa0003520 con 0x7effb415dfe0 2024-03-06T07:31:57.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.159+0000 7effabfff640 1 --2- 172.21.15.83:0/3236172060 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7effb40d22b0 0x7effb40d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.159+0000 7effb27fc640 1 -- 172.21.15.83:0/3236172060 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1740815187 0 0) 0x7effa0003520 con 0x7effb415dfe0 2024-03-06T07:31:57.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.159+0000 7effb27fc640 1 -- 172.21.15.83:0/3236172060 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7effb40d22b0 msgr2=0x7effb40d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:57.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.159+0000 7effb27fc640 1 --2- 172.21.15.83:0/3236172060 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7effb40d22b0 0x7effb40d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.159+0000 7effb27fc640 1 -- 172.21.15.83:0/3236172060 >> v1:172.21.15.183:6789/0 conn(0x7effb40b41f0 legacy=0x7effb40b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.159+0000 7effb27fc640 1 -- 172.21.15.83:0/3236172060 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb40b39f0 con 0x7effb415dfe0 2024-03-06T07:31:57.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.159+0000 7effb27fc640 1 -- 172.21.15.83:0/3236172060 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7effa4001670 con 0x7effb415dfe0 2024-03-06T07:31:57.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.159+0000 7effb27fc640 1 -- 172.21.15.83:0/3236172060 >> v1:172.21.15.83:6789/0 conn(0x7effb415dfe0 legacy=0x7effb40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.159+0000 7effb27fc640 1 --2- 172.21.15.83:0/3236172060 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effa00039e0 0x7effa0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.159+0000 7effb27fc640 1 --2- 172.21.15.83:0/3236172060 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7effa0004800 0x7effa0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.159+0000 7effb27fc640 1 --2- 172.21.15.83:0/3236172060 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7effb40d22b0 0x7effa00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.159+0000 7effb27fc640 1 -- 172.21.15.83:0/3236172060 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa000ba70 con 0x7effa00039e0 2024-03-06T07:31:57.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.159+0000 7effb27fc640 1 -- 172.21.15.83:0/3236172060 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa000bbf0 con 0x7effa0004800 2024-03-06T07:31:57.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.159+0000 7effb27fc640 1 -- 172.21.15.83:0/3236172060 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa000bdf0 con 0x7effb40d22b0 2024-03-06T07:31:57.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.159+0000 7effb2ffd640 1 --2- 172.21.15.83:0/3236172060 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effa00039e0 0x7effa0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.160+0000 7effb37fe640 1 --2- 172.21.15.83:0/3236172060 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7effb40d22b0 0x7effa00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.160+0000 7effabfff640 1 --2- 172.21.15.83:0/3236172060 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7effa0004800 0x7effa0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.160+0000 7effb2ffd640 1 -- 172.21.15.83:0/3236172060 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7effb40d22b0 msgr2=0x7effa00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:57.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.160+0000 7effb2ffd640 1 --2- 172.21.15.83:0/3236172060 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7effb40d22b0 0x7effa00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.160+0000 7effb2ffd640 1 -- 172.21.15.83:0/3236172060 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7effa0004800 msgr2=0x7effa0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:57.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.160+0000 7effb2ffd640 1 --2- 172.21.15.83:0/3236172060 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7effa0004800 0x7effa0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.160+0000 7effb2ffd640 1 -- 172.21.15.83:0/3236172060 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa0003520 con 0x7effa00039e0 2024-03-06T07:31:57.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.160+0000 7effabfff640 1 --2- 172.21.15.83:0/3236172060 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7effa0004800 0x7effa0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:57.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.160+0000 7effb2ffd640 1 --2- 172.21.15.83:0/3236172060 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effa00039e0 0x7effa0003e00 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7effa400c210 tx=0x7effa0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c572f966e8dee626 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:57.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.161+0000 7effb27fc640 1 -- 172.21.15.83:0/3236172060 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa4007d10 con 0x7effa00039e0 2024-03-06T07:31:57.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.161+0000 7effb27fc640 1 -- 172.21.15.83:0/3236172060 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effa4030070 con 0x7effa00039e0 2024-03-06T07:31:57.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.161+0000 7effb27fc640 1 -- 172.21.15.83:0/3236172060 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa4005d50 con 0x7effa00039e0 2024-03-06T07:31:57.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.161+0000 7effb955e640 1 -- 172.21.15.83:0/3236172060 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effa00039e0 msgr2=0x7effa0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.161+0000 7effb955e640 1 --2- 172.21.15.83:0/3236172060 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effa00039e0 0x7effa0003e00 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7effa400c210 tx=0x7effa0005830 comp rx=0 tx=0).stop 2024-03-06T07:31:57.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.161+0000 7effb37fe640 1 -- 172.21.15.83:0/3236172060 reap_dead start 2024-03-06T07:31:57.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.161+0000 7effb955e640 1 -- 172.21.15.83:0/3236172060 shutdown_connections 2024-03-06T07:31:57.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.161+0000 7effb955e640 1 -- 172.21.15.83:0/3236172060 >> 172.21.15.83:0/3236172060 conn(0x7effb40d3cd0 msgr2=0x7effb40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:57.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.161+0000 7effb955e640 1 -- 172.21.15.83:0/3236172060 shutdown_connections 2024-03-06T07:31:57.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.161+0000 7effb955e640 1 -- 172.21.15.83:0/3236172060 wait complete. 2024-03-06T07:31:57.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.162+0000 7effb955e640 1 Processor -- start 2024-03-06T07:31:57.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.162+0000 7effb955e640 1 -- start start 2024-03-06T07:31:57.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.163+0000 7effb955e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7effa00039e0 0x7effb40b6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.163+0000 7effb955e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7effb40cd960 0x7effb4157110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.163+0000 7effb955e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effb41f3920 0x7effb41f3da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.163+0000 7effb955e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb40b3d60 con 0x7effb41f3920 2024-03-06T07:31:57.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.163+0000 7effb955e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb40b31e0 con 0x7effb40cd960 2024-03-06T07:31:57.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.163+0000 7effb955e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effb40d1c70 con 0x7effa00039e0 2024-03-06T07:31:57.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.163+0000 7effb37fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effb41f3920 0x7effb41f3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.163+0000 7effb37fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effb41f3920 0x7effb41f3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40852/0 (socket says 172.21.15.83:40852) 2024-03-06T07:31:57.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.163+0000 7effb37fe640 1 -- 172.21.15.83:0/3786455751 learned_addr learned my addr 172.21.15.83:0/3786455751 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:57.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.163+0000 7effb2ffd640 1 --2- 172.21.15.83:0/3786455751 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7effa00039e0 0x7effb40b6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.163+0000 7effabfff640 1 --2- 172.21.15.83:0/3786455751 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7effb40cd960 0x7effb4157110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.164+0000 7effb37fe640 1 -- 172.21.15.83:0/3786455751 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7effa00039e0 msgr2=0x7effb40b6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.164+0000 7effb37fe640 1 --2- 172.21.15.83:0/3786455751 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7effa00039e0 0x7effb40b6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.164+0000 7effb37fe640 1 -- 172.21.15.83:0/3786455751 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7effb40cd960 msgr2=0x7effb4157110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.164+0000 7effb37fe640 1 --2- 172.21.15.83:0/3786455751 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7effb40cd960 0x7effb4157110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.164+0000 7effb37fe640 1 -- 172.21.15.83:0/3786455751 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb41f42e0 con 0x7effb41f3920 2024-03-06T07:31:57.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.164+0000 7effb37fe640 1 --2- 172.21.15.83:0/3786455751 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effb41f3920 0x7effb41f3da0 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7effb40cff40 tx=0x7effac00fd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:57.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.164+0000 7effab7fe640 1 -- 172.21.15.83:0/3786455751 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effac01a070 con 0x7effb41f3920 2024-03-06T07:31:57.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.164+0000 7effab7fe640 1 -- 172.21.15.83:0/3786455751 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effac006d60 con 0x7effb41f3920 2024-03-06T07:31:57.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.164+0000 7effb955e640 1 -- 172.21.15.83:0/3786455751 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effb40b8d90 con 0x7effb41f3920 2024-03-06T07:31:57.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.164+0000 7effab7fe640 1 -- 172.21.15.83:0/3786455751 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effac003b90 con 0x7effb41f3920 2024-03-06T07:31:57.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.164+0000 7effb955e640 1 -- 172.21.15.83:0/3786455751 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effb40b9310 con 0x7effb41f3920 2024-03-06T07:31:57.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.165+0000 7effb955e640 1 -- 172.21.15.83:0/3786455751 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effb404f330 con 0x7effb41f3920 2024-03-06T07:31:57.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.165+0000 7effab7fe640 1 -- 172.21.15.83:0/3786455751 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7effac01d070 con 0x7effb41f3920 2024-03-06T07:31:57.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.166+0000 7effab7fe640 1 --2- 172.21.15.83:0/3786455751 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7eff9406b4d0 0x7eff9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.166+0000 7effab7fe640 1 -- 172.21.15.83:0/3786455751 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7effac0897c0 con 0x7effb41f3920 2024-03-06T07:31:57.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.166+0000 7effb2ffd640 1 --2- 172.21.15.83:0/3786455751 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7eff9406b4d0 0x7eff9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.166+0000 7effb2ffd640 1 --2- 172.21.15.83:0/3786455751 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7eff9406b4d0 0x7eff9406d990 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7effa4034000 tx=0x7effa4009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:57.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.170+0000 7effab7fe640 1 -- 172.21.15.83:0/3786455751 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7effb41f42e0 con 0x7effb41f3920 2024-03-06T07:31:57.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.381+0000 7effb955e640 1 -- 172.21.15.83:0/3786455751 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7effb40d41e0 con 0x7effb41f3920 2024-03-06T07:31:57.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.382+0000 7effab7fe640 1 -- 172.21.15.83:0/3786455751 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7effac058600 con 0x7effb41f3920 2024-03-06T07:31:57.384 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:31:57.384 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:31:57.384 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:31:57.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.384+0000 7effb955e640 1 -- 172.21.15.83:0/3786455751 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7eff9406b4d0 msgr2=0x7eff9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.384+0000 7effb955e640 1 --2- 172.21.15.83:0/3786455751 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7eff9406b4d0 0x7eff9406d990 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7effa4034000 tx=0x7effa4009030 comp rx=0 tx=0).stop 2024-03-06T07:31:57.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.384+0000 7effb955e640 1 -- 172.21.15.83:0/3786455751 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effb41f3920 msgr2=0x7effb41f3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.384+0000 7effb955e640 1 --2- 172.21.15.83:0/3786455751 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effb41f3920 0x7effb41f3da0 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7effb40cff40 tx=0x7effac00fd40 comp rx=0 tx=0).stop 2024-03-06T07:31:57.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.384+0000 7effb955e640 1 -- 172.21.15.83:0/3786455751 shutdown_connections 2024-03-06T07:31:57.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.384+0000 7effb955e640 1 --2- 172.21.15.83:0/3786455751 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7eff9406b4d0 0x7eff9406d990 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.384+0000 7effb955e640 1 --2- 172.21.15.83:0/3786455751 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effb41f3920 0x7effb41f3da0 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.384+0000 7effb955e640 1 --2- 172.21.15.83:0/3786455751 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7effb40cd960 0x7effb4157110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.384+0000 7effb955e640 1 --2- 172.21.15.83:0/3786455751 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7effa00039e0 0x7effb40b6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.384+0000 7effb955e640 1 -- 172.21.15.83:0/3786455751 >> 172.21.15.83:0/3786455751 conn(0x7effb40d3cd0 msgr2=0x7effb40d0aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:57.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.384+0000 7effb955e640 1 -- 172.21.15.83:0/3786455751 shutdown_connections 2024-03-06T07:31:57.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:57.384+0000 7effb955e640 1 -- 172.21.15.83:0/3786455751 wait complete. 2024-03-06T07:31:57.598 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:31:26.591282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:31:57.598 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:31:58.599 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:58.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.707+0000 7fcbe844e640 1 Processor -- start 2024-03-06T07:31:58.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.707+0000 7fcbe844e640 1 -- start start 2024-03-06T07:31:58.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.707+0000 7fcbe844e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcbe00cf790 0x7fcbe00ad860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.708+0000 7fcbe844e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcbe00723f0 0x7fcbe006d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.708+0000 7fcbe844e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe00ae240 con 0x7fcbe00adda0 2024-03-06T07:31:58.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.708+0000 7fcbe844e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbe006cf70 con 0x7fcbe00723f0 2024-03-06T07:31:58.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.708+0000 7fcbe844e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe00ae520 con 0x7fcbe00cf790 2024-03-06T07:31:58.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.708+0000 7fcbe69eb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcbe00723f0 0x7fcbe006d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.708+0000 7fcbe69eb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fcbe00723f0 0x7fcbe006d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40860/0 (socket says 172.21.15.83:40860) 2024-03-06T07:31:58.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.708+0000 7fcbe69eb640 1 -- 172.21.15.83:0/3545963181 learned_addr learned my addr 172.21.15.83:0/3545963181 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:58.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.708+0000 7fcbe61ea640 1 --2- 172.21.15.83:0/3545963181 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcbe00cf790 0x7fcbe00ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.708+0000 7fcbe69eb640 1 -- 172.21.15.83:0/3545963181 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcbe00cf790 msgr2=0x7fcbe00ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:58.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.708+0000 7fcbe69eb640 1 --2- 172.21.15.83:0/3545963181 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcbe00cf790 0x7fcbe00ad860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.708+0000 7fcbe69eb640 1 -- 172.21.15.83:0/3545963181 >> v1:172.21.15.183:6789/0 conn(0x7fcbe00adda0 legacy=0x7fcbe0071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.708+0000 7fcbe69eb640 1 -- 172.21.15.83:0/3545963181 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe00ae800 con 0x7fcbe00723f0 2024-03-06T07:31:58.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.709+0000 7fcbe69eb640 1 --2- 172.21.15.83:0/3545963181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcbe00723f0 0x7fcbe006d400 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7fcbd8009e50 tx=0x7fcbd802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b80a27f8e566c2f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:58.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.710+0000 7fcbe51e8640 1 -- 172.21.15.83:0/3545963181 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd802cb00 con 0x7fcbe00723f0 2024-03-06T07:31:58.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.710+0000 7fcbe51e8640 1 -- 172.21.15.83:0/3545963181 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbd802cca0 con 0x7fcbe00723f0 2024-03-06T07:31:58.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.710+0000 7fcbe51e8640 1 -- 172.21.15.83:0/3545963181 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd8034610 con 0x7fcbe00723f0 2024-03-06T07:31:58.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.711+0000 7fcbe844e640 1 -- 172.21.15.83:0/3545963181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcbe00723f0 msgr2=0x7fcbe006d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.711+0000 7fcbe844e640 1 --2- 172.21.15.83:0/3545963181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcbe00723f0 0x7fcbe006d400 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7fcbd8009e50 tx=0x7fcbd802bf60 comp rx=0 tx=0).stop 2024-03-06T07:31:58.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.711+0000 7fcbe844e640 1 -- 172.21.15.83:0/3545963181 shutdown_connections 2024-03-06T07:31:58.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.711+0000 7fcbe844e640 1 --2- 172.21.15.83:0/3545963181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcbe00723f0 0x7fcbe006d400 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.711+0000 7fcbe844e640 1 --2- 172.21.15.83:0/3545963181 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcbe00cf790 0x7fcbe00ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.711+0000 7fcbe844e640 1 -- 172.21.15.83:0/3545963181 >> 172.21.15.83:0/3545963181 conn(0x7fcbe00b1a90 msgr2=0x7fcbe00b1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:58.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.711+0000 7fcbe844e640 1 -- 172.21.15.83:0/3545963181 shutdown_connections 2024-03-06T07:31:58.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.711+0000 7fcbe844e640 1 -- 172.21.15.83:0/3545963181 wait complete. 2024-03-06T07:31:58.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.713+0000 7fcbe844e640 1 Processor -- start 2024-03-06T07:31:58.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.713+0000 7fcbe844e640 1 -- start start 2024-03-06T07:31:58.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.714+0000 7fcbe844e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcbe00723f0 0x7fcbe014d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.714+0000 7fcbe844e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcbe00adda0 0x7fcbe00c4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.714+0000 7fcbe844e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcbe00cf790 0x7fcbe015c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.714+0000 7fcbe844e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe015e700 con 0x7fcbe00adda0 2024-03-06T07:31:58.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.714+0000 7fcbe844e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe015e580 con 0x7fcbe00cf790 2024-03-06T07:31:58.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.714+0000 7fcbe844e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe006d940 con 0x7fcbe00723f0 2024-03-06T07:31:58.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.714+0000 7fcbe59e9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcbe00adda0 0x7fcbe00c4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.714+0000 7fcbe59e9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcbe00adda0 0x7fcbe00c4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40868/0 (socket says 172.21.15.83:40868) 2024-03-06T07:31:58.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.714+0000 7fcbe59e9640 1 -- 172.21.15.83:0/2824320472 learned_addr learned my addr 172.21.15.83:0/2824320472 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:31:58.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.714+0000 7fcbe61ea640 1 --2- 172.21.15.83:0/2824320472 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcbe00723f0 0x7fcbe014d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.714+0000 7fcbe69eb640 1 --2- 172.21.15.83:0/2824320472 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcbe00cf790 0x7fcbe015c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.715+0000 7fcbe59e9640 1 -- 172.21.15.83:0/2824320472 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcbe00723f0 msgr2=0x7fcbe014d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.715+0000 7fcbe59e9640 1 --2- 172.21.15.83:0/2824320472 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcbe00723f0 0x7fcbe014d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.715+0000 7fcbe59e9640 1 -- 172.21.15.83:0/2824320472 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcbe00cf790 msgr2=0x7fcbe015c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.715+0000 7fcbe59e9640 1 --2- 172.21.15.83:0/2824320472 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcbe00cf790 0x7fcbe015c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.715+0000 7fcbe59e9640 1 -- 172.21.15.83:0/2824320472 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe014dbe0 con 0x7fcbe00adda0 2024-03-06T07:31:58.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.715+0000 7fcbe69eb640 1 --2- 172.21.15.83:0/2824320472 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcbe00cf790 0x7fcbe015c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:58.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.715+0000 7fcbe59e9640 1 --2- 172.21.15.83:0/2824320472 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcbe00adda0 0x7fcbe00c4d00 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc00c400 tx=0x7fcbdc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:58.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.715+0000 7fcbd6ffd640 1 -- 172.21.15.83:0/2824320472 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbdc0052d0 con 0x7fcbe00adda0 2024-03-06T07:31:58.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.715+0000 7fcbd6ffd640 1 -- 172.21.15.83:0/2824320472 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbdc005490 con 0x7fcbe00adda0 2024-03-06T07:31:58.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.715+0000 7fcbe844e640 1 -- 172.21.15.83:0/2824320472 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbe00c5670 con 0x7fcbe00adda0 2024-03-06T07:31:58.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.715+0000 7fcbd6ffd640 1 -- 172.21.15.83:0/2824320472 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbdc007260 con 0x7fcbe00adda0 2024-03-06T07:31:58.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.715+0000 7fcbe844e640 1 -- 172.21.15.83:0/2824320472 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbe00c5c30 con 0x7fcbe00adda0 2024-03-06T07:31:58.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.716+0000 7fcbd6ffd640 1 -- 172.21.15.83:0/2824320472 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcbdc007400 con 0x7fcbe00adda0 2024-03-06T07:31:58.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.716+0000 7fcbe844e640 1 -- 172.21.15.83:0/2824320472 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbe004f330 con 0x7fcbe00adda0 2024-03-06T07:31:58.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.717+0000 7fcbd6ffd640 1 --2- 172.21.15.83:0/2824320472 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcbc406b4d0 0x7fcbc406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.717+0000 7fcbd6ffd640 1 -- 172.21.15.83:0/2824320472 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fcbdc0896e0 con 0x7fcbe00adda0 2024-03-06T07:31:58.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.717+0000 7fcbe61ea640 1 --2- 172.21.15.83:0/2824320472 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcbc406b4d0 0x7fcbc406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.717+0000 7fcbe61ea640 1 --2- 172.21.15.83:0/2824320472 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcbc406b4d0 0x7fcbc406d990 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fcbd0009aa0 tx=0x7fcbd0009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:58.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.721+0000 7fcbd6ffd640 1 -- 172.21.15.83:0/2824320472 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcbdc057ed0 con 0x7fcbe00adda0 2024-03-06T07:31:58.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.935+0000 7fcbe844e640 1 -- 172.21.15.83:0/2824320472 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcbe0071950 con 0x7fcbe00adda0 2024-03-06T07:31:58.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.936+0000 7fcbd6ffd640 1 -- 172.21.15.83:0/2824320472 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7fcbdc05bf70 con 0x7fcbe00adda0 2024-03-06T07:31:58.938 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:31:58.938 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:31:58.938 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:31:58.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.938+0000 7fcbe844e640 1 -- 172.21.15.83:0/2824320472 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcbc406b4d0 msgr2=0x7fcbc406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.938+0000 7fcbe844e640 1 --2- 172.21.15.83:0/2824320472 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcbc406b4d0 0x7fcbc406d990 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fcbd0009aa0 tx=0x7fcbd0009190 comp rx=0 tx=0).stop 2024-03-06T07:31:58.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.938+0000 7fcbe844e640 1 -- 172.21.15.83:0/2824320472 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcbe00adda0 msgr2=0x7fcbe00c4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.938+0000 7fcbe844e640 1 --2- 172.21.15.83:0/2824320472 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcbe00adda0 0x7fcbe00c4d00 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc00c400 tx=0x7fcbdc00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.938+0000 7fcbe844e640 1 -- 172.21.15.83:0/2824320472 shutdown_connections 2024-03-06T07:31:58.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.938+0000 7fcbe844e640 1 --2- 172.21.15.83:0/2824320472 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fcbc406b4d0 0x7fcbc406d990 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.938+0000 7fcbe844e640 1 --2- 172.21.15.83:0/2824320472 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcbe00cf790 0x7fcbe015c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.938+0000 7fcbe844e640 1 --2- 172.21.15.83:0/2824320472 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcbe00adda0 0x7fcbe00c4d00 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.938+0000 7fcbe844e640 1 --2- 172.21.15.83:0/2824320472 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcbe00723f0 0x7fcbe014d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.938+0000 7fcbe844e640 1 -- 172.21.15.83:0/2824320472 >> 172.21.15.83:0/2824320472 conn(0x7fcbe00b1a90 msgr2=0x7fcbe015e910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:58.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.938+0000 7fcbe844e640 1 -- 172.21.15.83:0/2824320472 shutdown_connections 2024-03-06T07:31:58.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:31:58.938+0000 7fcbe844e640 1 -- 172.21.15.83:0/2824320472 wait complete. 2024-03-06T07:31:59.117 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:59.152 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:31:26.591282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:31:59.152 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:32:00.153 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:00.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.255+0000 7fe6f070f640 1 Processor -- start 2024-03-06T07:32:00.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.255+0000 7fe6f070f640 1 -- start start 2024-03-06T07:32:00.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.256+0000 7fe6f070f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe6e815dfe0 0x7fe6e80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.256+0000 7fe6f070f640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fe6e80d29c0 0x7fe6e80b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.256+0000 7fe6f070f640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6e80b3430 con 0x7fe6e80cf830 2024-03-06T07:32:00.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.256+0000 7fe6f070f640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6e80b31e0 con 0x7fe6e80d29c0 2024-03-06T07:32:00.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.256+0000 7fe6f070f640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e80b3d60 con 0x7fe6e815dfe0 2024-03-06T07:32:00.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.257+0000 7fe6ee4ab640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe6e815dfe0 0x7fe6e80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.257+0000 7fe6eecac640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fe6e80d29c0 0x7fe6e80b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.257+0000 7fe6ee4ab640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe6e815dfe0 0x7fe6e80cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:35040/0 (socket says 172.21.15.83:35040) 2024-03-06T07:32:00.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.257+0000 7fe6ee4ab640 1 -- 172.21.15.83:0/1232866072 learned_addr learned my addr 172.21.15.83:0/1232866072 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:00.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.257+0000 7fe6edcaa640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fe6e80cf830 0x7fe6e80d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:53608/0 (socket says 172.21.15.83:53608) 2024-03-06T07:32:00.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.257+0000 7fe6ee4ab640 1 -- 172.21.15.83:0/1232866072 >> v2:172.21.15.183:3300/0 conn(0x7fe6e80d29c0 msgr2=0x7fe6e80b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:00.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.257+0000 7fe6ee4ab640 1 --2- 172.21.15.83:0/1232866072 >> v2:172.21.15.183:3300/0 conn(0x7fe6e80d29c0 0x7fe6e80b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.257+0000 7fe6ee4ab640 1 -- 172.21.15.83:0/1232866072 >> v1:172.21.15.183:6789/0 conn(0x7fe6e80cf830 legacy=0x7fe6e80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.257+0000 7fe6ee4ab640 1 -- 172.21.15.83:0/1232866072 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e80b3710 con 0x7fe6e815dfe0 2024-03-06T07:32:00.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.257+0000 7fe6ed4a9640 1 -- 172.21.15.83:0/1232866072 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1364873743 0 0) 0x7fe6e80b3430 con 0x7fe6e80cf830 2024-03-06T07:32:00.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.257+0000 7fe6eecac640 1 --2- 172.21.15.83:0/1232866072 >> v2:172.21.15.183:3300/0 conn(0x7fe6e80d29c0 0x7fe6e80b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:00.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.258+0000 7fe6ee4ab640 1 --2- 172.21.15.83:0/1232866072 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe6e815dfe0 0x7fe6e80cf2f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe6e4009870 tx=0x7fe6e402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b35fee4a70adc638 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:00.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.258+0000 7fe6ed4a9640 1 -- 172.21.15.83:0/1232866072 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e4034070 con 0x7fe6e815dfe0 2024-03-06T07:32:00.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.258+0000 7fe6ed4a9640 1 -- 172.21.15.83:0/1232866072 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6e4030840 con 0x7fe6e815dfe0 2024-03-06T07:32:00.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.258+0000 7fe6ed4a9640 1 -- 172.21.15.83:0/1232866072 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e4030c10 con 0x7fe6e815dfe0 2024-03-06T07:32:00.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.258+0000 7fe6f070f640 1 -- 172.21.15.83:0/1232866072 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe6e815dfe0 msgr2=0x7fe6e80cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.258+0000 7fe6f070f640 1 --2- 172.21.15.83:0/1232866072 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe6e815dfe0 0x7fe6e80cf2f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe6e4009870 tx=0x7fe6e402b980 comp rx=0 tx=0).stop 2024-03-06T07:32:00.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.258+0000 7fe6f070f640 1 -- 172.21.15.83:0/1232866072 shutdown_connections 2024-03-06T07:32:00.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.258+0000 7fe6f070f640 1 --2- 172.21.15.83:0/1232866072 >> v2:172.21.15.183:3300/0 conn(0x7fe6e80d29c0 0x7fe6e80b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.258+0000 7fe6f070f640 1 --2- 172.21.15.83:0/1232866072 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe6e815dfe0 0x7fe6e80cf2f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.258+0000 7fe6f070f640 1 -- 172.21.15.83:0/1232866072 >> 172.21.15.83:0/1232866072 conn(0x7fe6e80d3cd0 msgr2=0x7fe6e80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:00.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.258+0000 7fe6f070f640 1 -- 172.21.15.83:0/1232866072 shutdown_connections 2024-03-06T07:32:00.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.258+0000 7fe6f070f640 1 -- 172.21.15.83:0/1232866072 wait complete. 2024-03-06T07:32:00.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.260+0000 7fe6f070f640 1 Processor -- start 2024-03-06T07:32:00.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.260+0000 7fe6f070f640 1 -- start start 2024-03-06T07:32:00.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.260+0000 7fe6f070f640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6e80cf830 0x7fe6e81520c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.260+0000 7fe6f070f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe6e80d29c0 0x7fe6e81f3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.260+0000 7fe6f070f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6e815dfe0 0x7fe6e80cd850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.260+0000 7fe6f070f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e80d1df0 con 0x7fe6e815dfe0 2024-03-06T07:32:00.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.260+0000 7fe6f070f640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e80d1c70 con 0x7fe6e80cf830 2024-03-06T07:32:00.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.260+0000 7fe6f070f640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e815e3c0 con 0x7fe6e80d29c0 2024-03-06T07:32:00.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.260+0000 7fe6eecac640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6e815dfe0 0x7fe6e80cd850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.260+0000 7fe6eecac640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6e815dfe0 0x7fe6e80cd850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40876/0 (socket says 172.21.15.83:40876) 2024-03-06T07:32:00.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.260+0000 7fe6eecac640 1 -- 172.21.15.83:0/574769231 learned_addr learned my addr 172.21.15.83:0/574769231 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:00.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.260+0000 7fe6ee4ab640 1 --2- 172.21.15.83:0/574769231 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6e80cf830 0x7fe6e81520c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.261+0000 7fe6edcaa640 1 --2- 172.21.15.83:0/574769231 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe6e80d29c0 0x7fe6e81f3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.261+0000 7fe6eecac640 1 -- 172.21.15.83:0/574769231 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe6e80d29c0 msgr2=0x7fe6e81f3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.261+0000 7fe6eecac640 1 --2- 172.21.15.83:0/574769231 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe6e80d29c0 0x7fe6e81f3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.261+0000 7fe6eecac640 1 -- 172.21.15.83:0/574769231 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6e80cf830 msgr2=0x7fe6e81520c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.261+0000 7fe6eecac640 1 --2- 172.21.15.83:0/574769231 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6e80cf830 0x7fe6e81520c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.261+0000 7fe6eecac640 1 -- 172.21.15.83:0/574769231 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e815fb30 con 0x7fe6e815dfe0 2024-03-06T07:32:00.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.261+0000 7fe6edcaa640 1 --2- 172.21.15.83:0/574769231 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe6e80d29c0 0x7fe6e81f3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:00.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.261+0000 7fe6eecac640 1 --2- 172.21.15.83:0/574769231 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6e815dfe0 0x7fe6e80cd850 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7fe6e0009f00 tx=0x7fe6e000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:00.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.261+0000 7fe6deffd640 1 -- 172.21.15.83:0/574769231 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e0018030 con 0x7fe6e815dfe0 2024-03-06T07:32:00.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.261+0000 7fe6deffd640 1 -- 172.21.15.83:0/574769231 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6e000cdf0 con 0x7fe6e815dfe0 2024-03-06T07:32:00.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.261+0000 7fe6f070f640 1 -- 172.21.15.83:0/574769231 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6e8152600 con 0x7fe6e815dfe0 2024-03-06T07:32:00.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.261+0000 7fe6deffd640 1 -- 172.21.15.83:0/574769231 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e0003bb0 con 0x7fe6e815dfe0 2024-03-06T07:32:00.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.261+0000 7fe6f070f640 1 -- 172.21.15.83:0/574769231 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6e81600b0 con 0x7fe6e815dfe0 2024-03-06T07:32:00.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.262+0000 7fe6f070f640 1 -- 172.21.15.83:0/574769231 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6e804f330 con 0x7fe6e815dfe0 2024-03-06T07:32:00.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.263+0000 7fe6deffd640 1 -- 172.21.15.83:0/574769231 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe6e0005d10 con 0x7fe6e815dfe0 2024-03-06T07:32:00.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.263+0000 7fe6deffd640 1 --2- 172.21.15.83:0/574769231 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe6cc06b500 0x7fe6cc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.263+0000 7fe6ee4ab640 1 --2- 172.21.15.83:0/574769231 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe6cc06b500 0x7fe6cc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.263+0000 7fe6deffd640 1 -- 172.21.15.83:0/574769231 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe6e008ab40 con 0x7fe6e815dfe0 2024-03-06T07:32:00.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.264+0000 7fe6ee4ab640 1 --2- 172.21.15.83:0/574769231 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe6cc06b500 0x7fe6cc06d9c0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fe6e4036000 tx=0x7fe6e4008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:00.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.268+0000 7fe6deffd640 1 -- 172.21.15.83:0/574769231 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe6e0022070 con 0x7fe6e815dfe0 2024-03-06T07:32:00.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.583+0000 7fe6f070f640 1 -- 172.21.15.83:0/574769231 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6e80d2450 con 0x7fe6e815dfe0 2024-03-06T07:32:00.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.583+0000 7fe6deffd640 1 -- 172.21.15.83:0/574769231 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7fe6e0010040 con 0x7fe6e815dfe0 2024-03-06T07:32:00.585 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:32:00.586 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:32:00.586 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:32:00.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.586+0000 7fe6f070f640 1 -- 172.21.15.83:0/574769231 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe6cc06b500 msgr2=0x7fe6cc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.586+0000 7fe6f070f640 1 --2- 172.21.15.83:0/574769231 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe6cc06b500 0x7fe6cc06d9c0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fe6e4036000 tx=0x7fe6e4008030 comp rx=0 tx=0).stop 2024-03-06T07:32:00.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.586+0000 7fe6f070f640 1 -- 172.21.15.83:0/574769231 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6e815dfe0 msgr2=0x7fe6e80cd850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.586+0000 7fe6f070f640 1 --2- 172.21.15.83:0/574769231 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6e815dfe0 0x7fe6e80cd850 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7fe6e0009f00 tx=0x7fe6e000c490 comp rx=0 tx=0).stop 2024-03-06T07:32:00.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.586+0000 7fe6f070f640 1 -- 172.21.15.83:0/574769231 shutdown_connections 2024-03-06T07:32:00.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.586+0000 7fe6f070f640 1 --2- 172.21.15.83:0/574769231 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe6cc06b500 0x7fe6cc06d9c0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.586+0000 7fe6f070f640 1 --2- 172.21.15.83:0/574769231 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6e815dfe0 0x7fe6e80cd850 unknown :-1 s=CLOSED pgs=1920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.586+0000 7fe6f070f640 1 --2- 172.21.15.83:0/574769231 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe6e80d29c0 0x7fe6e81f3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.586+0000 7fe6f070f640 1 --2- 172.21.15.83:0/574769231 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6e80cf830 0x7fe6e81520c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.586+0000 7fe6f070f640 1 -- 172.21.15.83:0/574769231 >> 172.21.15.83:0/574769231 conn(0x7fe6e80d3cd0 msgr2=0x7fe6e806cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:00.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.586+0000 7fe6f070f640 1 -- 172.21.15.83:0/574769231 shutdown_connections 2024-03-06T07:32:00.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:00.586+0000 7fe6f070f640 1 -- 172.21.15.83:0/574769231 wait complete. 2024-03-06T07:32:00.800 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:31:26.591282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:32:00.800 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:32:00.970 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:00.972 DEBUG:teuthology.orchestra.run.smithi183:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:01.119 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:32:01.802 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:01.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.901+0000 7fe7116c9640 1 Processor -- start 2024-03-06T07:32:01.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.901+0000 7fe7116c9640 1 -- start start 2024-03-06T07:32:01.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.902+0000 7fe7116c9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe70c13dc10 0x7fe70c0b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:01.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.902+0000 7fe7116c9640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fe70c15ef60 0x7fe70c0b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:01.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.902+0000 7fe7116c9640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe70c15f580 con 0x7fe70c15bf30 2024-03-06T07:32:01.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.902+0000 7fe7116c9640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe70c0cf3f0 con 0x7fe70c15ef60 2024-03-06T07:32:01.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.902+0000 7fe7116c9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe70c15b210 con 0x7fe70c13dc10 2024-03-06T07:32:01.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.903+0000 7fe703fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fe70c15bf30 0x7fe70c15e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:38160/0 (socket says 172.21.15.83:38160) 2024-03-06T07:32:01.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.903+0000 7fe703fff640 1 -- 172.21.15.83:0/3474438380 learned_addr learned my addr 172.21.15.83:0/3474438380 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:32:01.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.903+0000 7fe70a7fc640 1 -- 172.21.15.83:0/3474438380 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2698636272 0 0) 0x7fe70c15f580 con 0x7fe70c15bf30 2024-03-06T07:32:01.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.903+0000 7fe70a7fc640 1 -- 172.21.15.83:0/3474438380 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6f8005730 con 0x7fe70c15bf30 2024-03-06T07:32:01.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.903+0000 7fe70affd640 1 --2- 172.21.15.83:0/3474438380 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe70c13dc10 0x7fe70c0b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:01.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.903+0000 7fe70b7fe640 1 --2- 172.21.15.83:0/3474438380 >> v2:172.21.15.183:3300/0 conn(0x7fe70c15ef60 0x7fe70c0b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:01.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.903+0000 7fe70a7fc640 1 -- 172.21.15.83:0/3474438380 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 837611795 0 0) 0x7fe6f8005730 con 0x7fe70c15bf30 2024-03-06T07:32:01.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.903+0000 7fe70a7fc640 1 -- 172.21.15.83:0/3474438380 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe70c13dc10 msgr2=0x7fe70c0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:01.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.903+0000 7fe70a7fc640 1 --2- 172.21.15.83:0/3474438380 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe70c13dc10 0x7fe70c0b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.903+0000 7fe70a7fc640 1 -- 172.21.15.83:0/3474438380 >> v2:172.21.15.183:3300/0 conn(0x7fe70c15ef60 msgr2=0x7fe70c0b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:01.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.903+0000 7fe70a7fc640 1 --2- 172.21.15.83:0/3474438380 >> v2:172.21.15.183:3300/0 conn(0x7fe70c15ef60 0x7fe70c0b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.903+0000 7fe70a7fc640 1 -- 172.21.15.83:0/3474438380 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe70c15b450 con 0x7fe70c15bf30 2024-03-06T07:32:01.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.903+0000 7fe70a7fc640 1 -- 172.21.15.83:0/3474438380 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7fe6f4001670 con 0x7fe70c15bf30 2024-03-06T07:32:01.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.903+0000 7fe70b7fe640 1 --2- 172.21.15.83:0/3474438380 >> v2:172.21.15.183:3300/0 conn(0x7fe70c15ef60 0x7fe70c0b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:01.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.903+0000 7fe70a7fc640 1 -- 172.21.15.83:0/3474438380 >> v1:172.21.15.83:6789/0 conn(0x7fe70c15bf30 legacy=0x7fe70c15e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:01.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.903+0000 7fe70a7fc640 1 --2- 172.21.15.83:0/3474438380 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe70c13dc10 0x7fe6f8005bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:01.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.903+0000 7fe70a7fc640 1 --2- 172.21.15.83:0/3474438380 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6f8006140 0x7fe6f8006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:01.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.903+0000 7fe70a7fc640 1 --2- 172.21.15.83:0/3474438380 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6f8006f50 0x7fe6f80073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:01.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.903+0000 7fe70a7fc640 1 -- 172.21.15.83:0/3474438380 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe70c0cf3f0 con 0x7fe6f8006f50 2024-03-06T07:32:01.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.903+0000 7fe70a7fc640 1 -- 172.21.15.83:0/3474438380 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe70c15b210 con 0x7fe6f8006140 2024-03-06T07:32:01.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.903+0000 7fe70a7fc640 1 -- 172.21.15.83:0/3474438380 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f800dcd0 con 0x7fe70c13dc10 2024-03-06T07:32:01.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.903+0000 7fe70b7fe640 1 --2- 172.21.15.83:0/3474438380 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6f8006f50 0x7fe6f80073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:01.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.903+0000 7fe70affd640 1 --2- 172.21.15.83:0/3474438380 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe70c13dc10 0x7fe6f8005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:01.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.903+0000 7fe703fff640 1 --2- 172.21.15.83:0/3474438380 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6f8006140 0x7fe6f8006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:01.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.904+0000 7fe70b7fe640 1 -- 172.21.15.83:0/3474438380 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe70c13dc10 msgr2=0x7fe6f8005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:01.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.904+0000 7fe70b7fe640 1 --2- 172.21.15.83:0/3474438380 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe70c13dc10 0x7fe6f8005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.904+0000 7fe70b7fe640 1 -- 172.21.15.83:0/3474438380 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6f8006140 msgr2=0x7fe6f8006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:01.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.904+0000 7fe70b7fe640 1 --2- 172.21.15.83:0/3474438380 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6f8006140 0x7fe6f8006a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.904+0000 7fe70b7fe640 1 -- 172.21.15.83:0/3474438380 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f8005730 con 0x7fe6f8006f50 2024-03-06T07:32:01.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.904+0000 7fe70b7fe640 1 --2- 172.21.15.83:0/3474438380 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6f8006f50 0x7fe6f80073d0 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7fe704007410 tx=0x7fe6f8007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8b6e1725a3e41cb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:01.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.904+0000 7fe70a7fc640 1 -- 172.21.15.83:0/3474438380 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe704039070 con 0x7fe6f8006f50 2024-03-06T07:32:01.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.905+0000 7fe70a7fc640 1 -- 172.21.15.83:0/3474438380 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe70402cdc0 con 0x7fe6f8006f50 2024-03-06T07:32:01.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.905+0000 7fe70a7fc640 1 -- 172.21.15.83:0/3474438380 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe704009c80 con 0x7fe6f8006f50 2024-03-06T07:32:01.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.905+0000 7fe7116c9640 1 -- 172.21.15.83:0/3474438380 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6f8006f50 msgr2=0x7fe6f80073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:01.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.905+0000 7fe7116c9640 1 --2- 172.21.15.83:0/3474438380 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6f8006f50 0x7fe6f80073d0 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7fe704007410 tx=0x7fe6f8007a40 comp rx=0 tx=0).stop 2024-03-06T07:32:01.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.905+0000 7fe70b7fe640 1 -- 172.21.15.83:0/3474438380 reap_dead start 2024-03-06T07:32:01.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.905+0000 7fe7116c9640 1 -- 172.21.15.83:0/3474438380 shutdown_connections 2024-03-06T07:32:01.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.905+0000 7fe7116c9640 1 -- 172.21.15.83:0/3474438380 >> 172.21.15.83:0/3474438380 conn(0x7fe70c0cf8a0 msgr2=0x7fe70c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:01.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.905+0000 7fe7116c9640 1 -- 172.21.15.83:0/3474438380 shutdown_connections 2024-03-06T07:32:01.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.905+0000 7fe7116c9640 1 -- 172.21.15.83:0/3474438380 wait complete. 2024-03-06T07:32:01.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.906+0000 7fe7116c9640 1 Processor -- start 2024-03-06T07:32:01.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.906+0000 7fe7116c9640 1 -- start start 2024-03-06T07:32:01.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.907+0000 7fe7116c9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6f8006f50 0x7fe70c158350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:01.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.907+0000 7fe7116c9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe70c1382a0 0x7fe70c138720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:01.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.907+0000 7fe7116c9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe70c1efb60 0x7fe70c1f0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:01.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.907+0000 7fe7116c9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe70c0b2950 con 0x7fe6f8006f50 2024-03-06T07:32:01.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.907+0000 7fe7116c9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe70c13dff0 con 0x7fe70c1382a0 2024-03-06T07:32:01.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.907+0000 7fe7116c9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe70c0b2ad0 con 0x7fe70c1efb60 2024-03-06T07:32:01.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.907+0000 7fe70affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6f8006f50 0x7fe70c158350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:01.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.907+0000 7fe70affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6f8006f50 0x7fe70c158350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40896/0 (socket says 172.21.15.83:40896) 2024-03-06T07:32:01.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.907+0000 7fe70affd640 1 -- 172.21.15.83:0/2791456720 learned_addr learned my addr 172.21.15.83:0/2791456720 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:01.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.907+0000 7fe70affd640 1 -- 172.21.15.83:0/2791456720 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe70c1efb60 msgr2=0x7fe70c1f0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:01.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.907+0000 7fe70affd640 1 --2- 172.21.15.83:0/2791456720 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe70c1efb60 0x7fe70c1f0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.907+0000 7fe70affd640 1 -- 172.21.15.83:0/2791456720 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe70c1382a0 msgr2=0x7fe70c138720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:01.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.907+0000 7fe70affd640 1 --2- 172.21.15.83:0/2791456720 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe70c1382a0 0x7fe70c138720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.907+0000 7fe70affd640 1 -- 172.21.15.83:0/2791456720 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe70c138c60 con 0x7fe6f8006f50 2024-03-06T07:32:01.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.907+0000 7fe70affd640 1 --2- 172.21.15.83:0/2791456720 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6f8006f50 0x7fe70c158350 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc011910 tx=0x7fe6fc011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:01.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.908+0000 7fe7037fe640 1 -- 172.21.15.83:0/2791456720 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc00fa70 con 0x7fe6f8006f50 2024-03-06T07:32:01.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.908+0000 7fe7037fe640 1 -- 172.21.15.83:0/2791456720 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6fc00fc10 con 0x7fe6f8006f50 2024-03-06T07:32:01.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.908+0000 7fe7116c9640 1 -- 172.21.15.83:0/2791456720 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe70c06d210 con 0x7fe6f8006f50 2024-03-06T07:32:01.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.908+0000 7fe7037fe640 1 -- 172.21.15.83:0/2791456720 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc014440 con 0x7fe6f8006f50 2024-03-06T07:32:01.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.908+0000 7fe7116c9640 1 -- 172.21.15.83:0/2791456720 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe70c06dab0 con 0x7fe6f8006f50 2024-03-06T07:32:01.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.909+0000 7fe7116c9640 1 -- 172.21.15.83:0/2791456720 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe70c15e5f0 con 0x7fe6f8006f50 2024-03-06T07:32:01.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.909+0000 7fe7037fe640 1 -- 172.21.15.83:0/2791456720 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe6fc012040 con 0x7fe6f8006f50 2024-03-06T07:32:01.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.910+0000 7fe7037fe640 1 --2- 172.21.15.83:0/2791456720 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe6ec06b4d0 0x7fe6ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:01.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.910+0000 7fe703fff640 1 --2- 172.21.15.83:0/2791456720 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe6ec06b4d0 0x7fe6ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:01.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.910+0000 7fe7037fe640 1 -- 172.21.15.83:0/2791456720 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe6fc08a830 con 0x7fe6f8006f50 2024-03-06T07:32:01.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.910+0000 7fe703fff640 1 --2- 172.21.15.83:0/2791456720 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe6ec06b4d0 0x7fe6ec06d990 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fe6f4001050 tx=0x7fe6f4000fe0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:01.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:01.914+0000 7fe7037fe640 1 -- 172.21.15.83:0/2791456720 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe6fc019030 con 0x7fe6f8006f50 2024-03-06T07:32:02.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:02.127+0000 7fe7116c9640 1 -- 172.21.15.83:0/2791456720 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe70c04f330 con 0x7fe6f8006f50 2024-03-06T07:32:02.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:02.128+0000 7fe7037fe640 1 -- 172.21.15.83:0/2791456720 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7fe6fc059020 con 0x7fe6f8006f50 2024-03-06T07:32:02.130 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:32:02.130 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:32:02.130 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:32:02.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:02.130+0000 7fe7116c9640 1 -- 172.21.15.83:0/2791456720 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe6ec06b4d0 msgr2=0x7fe6ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:02.130+0000 7fe7116c9640 1 --2- 172.21.15.83:0/2791456720 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe6ec06b4d0 0x7fe6ec06d990 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fe6f4001050 tx=0x7fe6f4000fe0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:02.130+0000 7fe7116c9640 1 -- 172.21.15.83:0/2791456720 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6f8006f50 msgr2=0x7fe70c158350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:02.130+0000 7fe7116c9640 1 --2- 172.21.15.83:0/2791456720 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6f8006f50 0x7fe70c158350 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc011910 tx=0x7fe6fc011dd0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:02.130+0000 7fe7116c9640 1 -- 172.21.15.83:0/2791456720 shutdown_connections 2024-03-06T07:32:02.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:02.130+0000 7fe7116c9640 1 --2- 172.21.15.83:0/2791456720 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe6ec06b4d0 0x7fe6ec06d990 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:02.130+0000 7fe7116c9640 1 --2- 172.21.15.83:0/2791456720 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe70c1efb60 0x7fe70c1f0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:02.130+0000 7fe7116c9640 1 --2- 172.21.15.83:0/2791456720 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe70c1382a0 0x7fe70c138720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:02.130+0000 7fe7116c9640 1 --2- 172.21.15.83:0/2791456720 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6f8006f50 0x7fe70c158350 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:02.130+0000 7fe7116c9640 1 -- 172.21.15.83:0/2791456720 >> 172.21.15.83:0/2791456720 conn(0x7fe70c0cf8a0 msgr2=0x7fe70c0ce400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:02.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:02.130+0000 7fe7116c9640 1 -- 172.21.15.83:0/2791456720 shutdown_connections 2024-03-06T07:32:02.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:02.130+0000 7fe7116c9640 1 -- 172.21.15.83:0/2791456720 wait complete. 2024-03-06T07:32:02.347 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:31:26.591282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:32:02.347 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:32:03.122 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:32:03.348 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:03.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.452+0000 7fd7669f7640 1 Processor -- start 2024-03-06T07:32:03.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.452+0000 7fd7669f7640 1 -- start start 2024-03-06T07:32:03.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.453+0000 7fd7669f7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd7600d0a80 0x7fd7600b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.453+0000 7fd7669f7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7600d25d0 con 0x7fd7600d4760 2024-03-06T07:32:03.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.453+0000 7fd7669f7640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7600d2870 con 0x7fd7600b4170 2024-03-06T07:32:03.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.453+0000 7fd7669f7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7600d4bf0 con 0x7fd7600d0a80 2024-03-06T07:32:03.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.453+0000 7fd75f7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fd7600d4760 0x7fd7600cee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:38166/0 (socket says 172.21.15.83:38166) 2024-03-06T07:32:03.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.453+0000 7fd75f7fe640 1 -- 172.21.15.83:0/920954502 learned_addr learned my addr 172.21.15.83:0/920954502 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:32:03.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.454+0000 7fd75effd640 1 -- 172.21.15.83:0/920954502 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2051399989 0 0) 0x7fd7600d25d0 con 0x7fd7600d4760 2024-03-06T07:32:03.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.454+0000 7fd75effd640 1 -- 172.21.15.83:0/920954502 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd748003520 con 0x7fd7600d4760 2024-03-06T07:32:03.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.454+0000 7fd764f94640 1 --2- 172.21.15.83:0/920954502 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd7600d0a80 0x7fd7600b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.454+0000 7fd75effd640 1 -- 172.21.15.83:0/920954502 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 55425981 0 0) 0x7fd748003520 con 0x7fd7600d4760 2024-03-06T07:32:03.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.454+0000 7fd75effd640 1 -- 172.21.15.83:0/920954502 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd7600d0a80 msgr2=0x7fd7600b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:03.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.454+0000 7fd75effd640 1 --2- 172.21.15.83:0/920954502 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd7600d0a80 0x7fd7600b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.454+0000 7fd75effd640 1 -- 172.21.15.83:0/920954502 >> v1:172.21.15.183:6789/0 conn(0x7fd7600b4170 legacy=0x7fd7600d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.454+0000 7fd75effd640 1 -- 172.21.15.83:0/920954502 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7600d2b50 con 0x7fd7600d4760 2024-03-06T07:32:03.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.454+0000 7fd764f94640 1 --2- 172.21.15.83:0/920954502 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd7600d0a80 0x7fd7600b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:03.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.454+0000 7fd75effd640 1 -- 172.21.15.83:0/920954502 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7fd754001670 con 0x7fd7600d4760 2024-03-06T07:32:03.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.454+0000 7fd75effd640 1 -- 172.21.15.83:0/920954502 >> v1:172.21.15.83:6789/0 conn(0x7fd7600d4760 legacy=0x7fd7600cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.454+0000 7fd75effd640 1 --2- 172.21.15.83:0/920954502 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd7480039e0 0x7fd748003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.454+0000 7fd75effd640 1 --2- 172.21.15.83:0/920954502 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd7600d0a80 0x7fd748004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.454+0000 7fd75effd640 1 --2- 172.21.15.83:0/920954502 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd748004d40 0x7fd7480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.454+0000 7fd75effd640 1 -- 172.21.15.83:0/920954502 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7600d4bf0 con 0x7fd748004d40 2024-03-06T07:32:03.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.454+0000 7fd75effd640 1 -- 172.21.15.83:0/920954502 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74800bb20 con 0x7fd7480039e0 2024-03-06T07:32:03.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.454+0000 7fd75effd640 1 -- 172.21.15.83:0/920954502 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd74800bd20 con 0x7fd7600d0a80 2024-03-06T07:32:03.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.454+0000 7fd764f94640 1 --2- 172.21.15.83:0/920954502 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd748004d40 0x7fd7480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.454+0000 7fd75ffff640 1 --2- 172.21.15.83:0/920954502 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd7480039e0 0x7fd748003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.454+0000 7fd75f7fe640 1 --2- 172.21.15.83:0/920954502 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd7600d0a80 0x7fd748004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.454+0000 7fd764f94640 1 -- 172.21.15.83:0/920954502 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd7600d0a80 msgr2=0x7fd748004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:03.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.454+0000 7fd764f94640 1 --2- 172.21.15.83:0/920954502 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd7600d0a80 0x7fd748004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.454+0000 7fd764f94640 1 -- 172.21.15.83:0/920954502 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd7480039e0 msgr2=0x7fd748003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:03.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.454+0000 7fd764f94640 1 --2- 172.21.15.83:0/920954502 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd7480039e0 0x7fd748003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.454+0000 7fd764f94640 1 -- 172.21.15.83:0/920954502 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd748003520 con 0x7fd748004d40 2024-03-06T07:32:03.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.454+0000 7fd75f7fe640 1 --2- 172.21.15.83:0/920954502 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd7600d0a80 0x7fd748004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:03.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.455+0000 7fd764f94640 1 --2- 172.21.15.83:0/920954502 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd748004d40 0x7fd7480051c0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7fd7580073e0 tx=0x7fd748005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c4c7e6e52d07cb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:03.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.455+0000 7fd75effd640 1 -- 172.21.15.83:0/920954502 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd758039070 con 0x7fd748004d40 2024-03-06T07:32:03.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.455+0000 7fd75effd640 1 -- 172.21.15.83:0/920954502 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd75802cd30 con 0x7fd748004d40 2024-03-06T07:32:03.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.455+0000 7fd75effd640 1 -- 172.21.15.83:0/920954502 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd758009c80 con 0x7fd748004d40 2024-03-06T07:32:03.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.456+0000 7fd7669f7640 1 -- 172.21.15.83:0/920954502 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd748004d40 msgr2=0x7fd7480051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.456+0000 7fd7669f7640 1 --2- 172.21.15.83:0/920954502 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd748004d40 0x7fd7480051c0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7fd7580073e0 tx=0x7fd748005830 comp rx=0 tx=0).stop 2024-03-06T07:32:03.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.456+0000 7fd764f94640 1 -- 172.21.15.83:0/920954502 reap_dead start 2024-03-06T07:32:03.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.456+0000 7fd7669f7640 1 -- 172.21.15.83:0/920954502 shutdown_connections 2024-03-06T07:32:03.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.456+0000 7fd7669f7640 1 -- 172.21.15.83:0/920954502 >> 172.21.15.83:0/920954502 conn(0x7fd7600d3c40 msgr2=0x7fd7600d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:03.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.456+0000 7fd7669f7640 1 -- 172.21.15.83:0/920954502 shutdown_connections 2024-03-06T07:32:03.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.456+0000 7fd7669f7640 1 -- 172.21.15.83:0/920954502 wait complete. 2024-03-06T07:32:03.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.457+0000 7fd7669f7640 1 Processor -- start 2024-03-06T07:32:03.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.457+0000 7fd7669f7640 1 -- start start 2024-03-06T07:32:03.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.458+0000 7fd7669f7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd760157180 0x7fd7600b2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.458+0000 7fd7669f7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd7600b30d0 0x7fd760158a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.458+0000 7fd7669f7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd760158fd0 0x7fd7600cbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.458+0000 7fd7669f7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7600b4dd0 con 0x7fd760158fd0 2024-03-06T07:32:03.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.458+0000 7fd7669f7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7600ce9e0 con 0x7fd7600b30d0 2024-03-06T07:32:03.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.458+0000 7fd7669f7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7600b4f50 con 0x7fd760157180 2024-03-06T07:32:03.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.458+0000 7fd764f94640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd760158fd0 0x7fd7600cbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.458+0000 7fd764f94640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd760158fd0 0x7fd7600cbbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40912/0 (socket says 172.21.15.83:40912) 2024-03-06T07:32:03.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.458+0000 7fd764f94640 1 -- 172.21.15.83:0/4078109744 learned_addr learned my addr 172.21.15.83:0/4078109744 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:03.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.459+0000 7fd75f7fe640 1 --2- 172.21.15.83:0/4078109744 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd7600b30d0 0x7fd760158a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.459+0000 7fd75ffff640 1 --2- 172.21.15.83:0/4078109744 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd760157180 0x7fd7600b2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.459+0000 7fd764f94640 1 -- 172.21.15.83:0/4078109744 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd760157180 msgr2=0x7fd7600b2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.459+0000 7fd764f94640 1 --2- 172.21.15.83:0/4078109744 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd760157180 0x7fd7600b2b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.459+0000 7fd764f94640 1 -- 172.21.15.83:0/4078109744 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd7600b30d0 msgr2=0x7fd760158a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.459+0000 7fd764f94640 1 --2- 172.21.15.83:0/4078109744 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd7600b30d0 0x7fd760158a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.459+0000 7fd764f94640 1 -- 172.21.15.83:0/4078109744 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7600b1af0 con 0x7fd760158fd0 2024-03-06T07:32:03.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.459+0000 7fd75f7fe640 1 --2- 172.21.15.83:0/4078109744 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd7600b30d0 0x7fd760158a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:03.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.459+0000 7fd75ffff640 1 --2- 172.21.15.83:0/4078109744 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd760157180 0x7fd7600b2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:03.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.459+0000 7fd764f94640 1 --2- 172.21.15.83:0/4078109744 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd760158fd0 0x7fd7600cbbc0 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7fd758037040 tx=0x7fd758004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:03.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.459+0000 7fd75cff9640 1 -- 172.21.15.83:0/4078109744 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd758039030 con 0x7fd760158fd0 2024-03-06T07:32:03.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.459+0000 7fd75cff9640 1 -- 172.21.15.83:0/4078109744 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd75803a040 con 0x7fd760158fd0 2024-03-06T07:32:03.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.459+0000 7fd75cff9640 1 -- 172.21.15.83:0/4078109744 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd758009c80 con 0x7fd760158fd0 2024-03-06T07:32:03.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.459+0000 7fd7669f7640 1 -- 172.21.15.83:0/4078109744 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7600b1dd0 con 0x7fd760158fd0 2024-03-06T07:32:03.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.459+0000 7fd7669f7640 1 -- 172.21.15.83:0/4078109744 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7600b2390 con 0x7fd760158fd0 2024-03-06T07:32:03.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.460+0000 7fd7669f7640 1 -- 172.21.15.83:0/4078109744 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd76004f330 con 0x7fd760158fd0 2024-03-06T07:32:03.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.461+0000 7fd75cff9640 1 -- 172.21.15.83:0/4078109744 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd758045020 con 0x7fd760158fd0 2024-03-06T07:32:03.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.461+0000 7fd75cff9640 1 --2- 172.21.15.83:0/4078109744 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd74c06b4d0 0x7fd74c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.462+0000 7fd75cff9640 1 -- 172.21.15.83:0/4078109744 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd7580ab5f0 con 0x7fd760158fd0 2024-03-06T07:32:03.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.462+0000 7fd75ffff640 1 --2- 172.21.15.83:0/4078109744 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd74c06b4d0 0x7fd74c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.462+0000 7fd75ffff640 1 --2- 172.21.15.83:0/4078109744 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd74c06b4d0 0x7fd74c06d990 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fd7500055c0 tx=0x7fd750009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:03.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.466+0000 7fd75cff9640 1 -- 172.21.15.83:0/4078109744 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd75804c020 con 0x7fd760158fd0 2024-03-06T07:32:03.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.674+0000 7fd7669f7640 1 -- 172.21.15.83:0/4078109744 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd7600733b0 con 0x7fd760158fd0 2024-03-06T07:32:03.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.675+0000 7fd75cff9640 1 -- 172.21.15.83:0/4078109744 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7fd75807a4c0 con 0x7fd760158fd0 2024-03-06T07:32:03.677 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:32:03.677 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:32:03.677 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:32:03.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.677+0000 7fd7669f7640 1 -- 172.21.15.83:0/4078109744 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd74c06b4d0 msgr2=0x7fd74c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.677+0000 7fd7669f7640 1 --2- 172.21.15.83:0/4078109744 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd74c06b4d0 0x7fd74c06d990 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fd7500055c0 tx=0x7fd750009040 comp rx=0 tx=0).stop 2024-03-06T07:32:03.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.677+0000 7fd7669f7640 1 -- 172.21.15.83:0/4078109744 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd760158fd0 msgr2=0x7fd7600cbbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.677+0000 7fd7669f7640 1 --2- 172.21.15.83:0/4078109744 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd760158fd0 0x7fd7600cbbc0 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7fd758037040 tx=0x7fd758004ab0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.677+0000 7fd7669f7640 1 -- 172.21.15.83:0/4078109744 shutdown_connections 2024-03-06T07:32:03.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.677+0000 7fd7669f7640 1 --2- 172.21.15.83:0/4078109744 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd74c06b4d0 0x7fd74c06d990 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.677+0000 7fd7669f7640 1 --2- 172.21.15.83:0/4078109744 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd760158fd0 0x7fd7600cbbc0 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.677+0000 7fd7669f7640 1 --2- 172.21.15.83:0/4078109744 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd7600b30d0 0x7fd760158a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.677+0000 7fd7669f7640 1 --2- 172.21.15.83:0/4078109744 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd760157180 0x7fd7600b2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.677+0000 7fd7669f7640 1 -- 172.21.15.83:0/4078109744 >> 172.21.15.83:0/4078109744 conn(0x7fd7600d3c40 msgr2=0x7fd7600b73f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:03.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.677+0000 7fd7669f7640 1 -- 172.21.15.83:0/4078109744 shutdown_connections 2024-03-06T07:32:03.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:03.677+0000 7fd7669f7640 1 -- 172.21.15.83:0/4078109744 wait complete. 2024-03-06T07:32:03.891 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:31:26.591282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:32:03.891 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:32:04.892 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:04.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.995+0000 7f3033a2a640 1 Processor -- start 2024-03-06T07:32:04.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.995+0000 7f3033a2a640 1 -- start start 2024-03-06T07:32:05.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.995+0000 7f3033a2a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f302c15bdd0 0x7f302c0c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.995+0000 7f3033a2a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f302c0cebe0 con 0x7f302c0d16e0 2024-03-06T07:32:05.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.995+0000 7f3033a2a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f302c0ceec0 con 0x7f302c15fa10 2024-03-06T07:32:05.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.995+0000 7f3033a2a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f302c0ce980 con 0x7f302c15bdd0 2024-03-06T07:32:05.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.996+0000 7f3031fc7640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f302c0d16e0 0x7f302c0d1b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:38174/0 (socket says 172.21.15.83:38174) 2024-03-06T07:32:05.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.996+0000 7f3031fc7640 1 -- 172.21.15.83:0/3381657891 learned_addr learned my addr 172.21.15.83:0/3381657891 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:32:05.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.996+0000 7f30317c6640 1 --2- 172.21.15.83:0/3381657891 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f302c15bdd0 0x7f302c0c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.996+0000 7f3023fff640 1 -- 172.21.15.83:0/3381657891 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3156372189 0 0) 0x7f302c0cebe0 con 0x7f302c0d16e0 2024-03-06T07:32:05.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.996+0000 7f3023fff640 1 -- 172.21.15.83:0/3381657891 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3018003520 con 0x7f302c0d16e0 2024-03-06T07:32:05.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.996+0000 7f3023fff640 1 -- 172.21.15.83:0/3381657891 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3048433093 0 0) 0x7f3018003520 con 0x7f302c0d16e0 2024-03-06T07:32:05.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.996+0000 7f3023fff640 1 -- 172.21.15.83:0/3381657891 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f302c15bdd0 msgr2=0x7f302c0c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:05.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.996+0000 7f3023fff640 1 --2- 172.21.15.83:0/3381657891 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f302c15bdd0 0x7f302c0c5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.996+0000 7f3023fff640 1 -- 172.21.15.83:0/3381657891 >> v1:172.21.15.183:6789/0 conn(0x7f302c15fa10 legacy=0x7f302c0ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.996+0000 7f3023fff640 1 -- 172.21.15.83:0/3381657891 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f302c0cf1a0 con 0x7f302c0d16e0 2024-03-06T07:32:05.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.996+0000 7f3023fff640 1 -- 172.21.15.83:0/3381657891 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f30240019f0 con 0x7f302c0d16e0 2024-03-06T07:32:05.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.996+0000 7f3023fff640 1 -- 172.21.15.83:0/3381657891 >> v1:172.21.15.83:6789/0 conn(0x7f302c0d16e0 legacy=0x7f302c0d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.996+0000 7f3023fff640 1 --2- 172.21.15.83:0/3381657891 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f30180039e0 0x7f3018003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.996+0000 7f3023fff640 1 --2- 172.21.15.83:0/3381657891 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f302c15bdd0 0x7f3018004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.996+0000 7f3023fff640 1 --2- 172.21.15.83:0/3381657891 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3018004d40 0x7f30180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.996+0000 7f3023fff640 1 -- 172.21.15.83:0/3381657891 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302c0ce980 con 0x7f3018004d40 2024-03-06T07:32:05.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.996+0000 7f3023fff640 1 -- 172.21.15.83:0/3381657891 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f301800baf0 con 0x7f30180039e0 2024-03-06T07:32:05.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.996+0000 7f3023fff640 1 -- 172.21.15.83:0/3381657891 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f301800bcf0 con 0x7f302c15bdd0 2024-03-06T07:32:05.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.996+0000 7f3031fc7640 1 --2- 172.21.15.83:0/3381657891 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3018004d40 0x7f30180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.996+0000 7f30317c6640 1 --2- 172.21.15.83:0/3381657891 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f30180039e0 0x7f3018003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.996+0000 7f3030fc5640 1 --2- 172.21.15.83:0/3381657891 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f302c15bdd0 0x7f3018004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.997+0000 7f3031fc7640 1 -- 172.21.15.83:0/3381657891 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f302c15bdd0 msgr2=0x7f3018004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:05.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.997+0000 7f3031fc7640 1 --2- 172.21.15.83:0/3381657891 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f302c15bdd0 0x7f3018004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.997+0000 7f3031fc7640 1 -- 172.21.15.83:0/3381657891 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f30180039e0 msgr2=0x7f3018003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:05.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.997+0000 7f3031fc7640 1 --2- 172.21.15.83:0/3381657891 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f30180039e0 0x7f3018003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.997+0000 7f3031fc7640 1 -- 172.21.15.83:0/3381657891 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3018003520 con 0x7f3018004d40 2024-03-06T07:32:05.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.997+0000 7f3031fc7640 1 --2- 172.21.15.83:0/3381657891 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3018004d40 0x7f30180051c0 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f302400a850 tx=0x7f3018005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2f7b2a4dc068567 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:05.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.997+0000 7f3023fff640 1 -- 172.21.15.83:0/3381657891 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f302400a4a0 con 0x7f3018004d40 2024-03-06T07:32:05.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.997+0000 7f3023fff640 1 -- 172.21.15.83:0/3381657891 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3024034650 con 0x7f3018004d40 2024-03-06T07:32:05.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.997+0000 7f3023fff640 1 -- 172.21.15.83:0/3381657891 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3024034a40 con 0x7f3018004d40 2024-03-06T07:32:05.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.998+0000 7f3033a2a640 1 -- 172.21.15.83:0/3381657891 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3018004d40 msgr2=0x7f30180051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.998+0000 7f3033a2a640 1 --2- 172.21.15.83:0/3381657891 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3018004d40 0x7f30180051c0 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f302400a850 tx=0x7f3018005830 comp rx=0 tx=0).stop 2024-03-06T07:32:05.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.998+0000 7f3031fc7640 1 -- 172.21.15.83:0/3381657891 reap_dead start 2024-03-06T07:32:05.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.998+0000 7f3033a2a640 1 -- 172.21.15.83:0/3381657891 shutdown_connections 2024-03-06T07:32:05.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.998+0000 7f3033a2a640 1 -- 172.21.15.83:0/3381657891 >> 172.21.15.83:0/3381657891 conn(0x7f302c09a9f0 msgr2=0x7f302c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:05.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.998+0000 7f3033a2a640 1 -- 172.21.15.83:0/3381657891 shutdown_connections 2024-03-06T07:32:05.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.998+0000 7f3033a2a640 1 -- 172.21.15.83:0/3381657891 wait complete. 2024-03-06T07:32:05.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.999+0000 7f3033a2a640 1 Processor -- start 2024-03-06T07:32:05.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:04.999+0000 7f3033a2a640 1 -- start start 2024-03-06T07:32:05.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.000+0000 7f3033a2a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3018004d40 0x7f302c1f8620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.000+0000 7f3033a2a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f302c14daf0 0x7f302c1f74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.000+0000 7f3033a2a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f302c0b2d00 0x7f302c0b3190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.000+0000 7f3033a2a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302c15c1b0 con 0x7f3018004d40 2024-03-06T07:32:05.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.000+0000 7f3033a2a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302c0cc5d0 con 0x7f302c0b2d00 2024-03-06T07:32:05.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.000+0000 7f3033a2a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f302c0c5610 con 0x7f302c14daf0 2024-03-06T07:32:05.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.000+0000 7f30317c6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3018004d40 0x7f302c1f8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.000+0000 7f30317c6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3018004d40 0x7f302c1f8620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40926/0 (socket says 172.21.15.83:40926) 2024-03-06T07:32:05.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.000+0000 7f30317c6640 1 -- 172.21.15.83:0/2856289649 learned_addr learned my addr 172.21.15.83:0/2856289649 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:05.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.000+0000 7f3030fc5640 1 --2- 172.21.15.83:0/2856289649 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f302c14daf0 0x7f302c1f74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.000+0000 7f3031fc7640 1 --2- 172.21.15.83:0/2856289649 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f302c0b2d00 0x7f302c0b3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.000+0000 7f30317c6640 1 -- 172.21.15.83:0/2856289649 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f302c14daf0 msgr2=0x7f302c1f74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.000+0000 7f30317c6640 1 --2- 172.21.15.83:0/2856289649 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f302c14daf0 0x7f302c1f74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.000+0000 7f30317c6640 1 -- 172.21.15.83:0/2856289649 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f302c0b2d00 msgr2=0x7f302c0b3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.000+0000 7f30317c6640 1 --2- 172.21.15.83:0/2856289649 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f302c0b2d00 0x7f302c0b3190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.000+0000 7f30317c6640 1 -- 172.21.15.83:0/2856289649 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f302c1f7a00 con 0x7f3018004d40 2024-03-06T07:32:05.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.000+0000 7f3031fc7640 1 --2- 172.21.15.83:0/2856289649 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f302c0b2d00 0x7f302c0b3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:05.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.000+0000 7f30317c6640 1 --2- 172.21.15.83:0/2856289649 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3018004d40 0x7f302c1f8620 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f30280119a0 tx=0x7f3028011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:05.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.001+0000 7f3021ffb640 1 -- 172.21.15.83:0/2856289649 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f302800faf0 con 0x7f3018004d40 2024-03-06T07:32:05.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.001+0000 7f3021ffb640 1 -- 172.21.15.83:0/2856289649 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f302800fc90 con 0x7f3018004d40 2024-03-06T07:32:05.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.001+0000 7f3033a2a640 1 -- 172.21.15.83:0/2856289649 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f302c0c8e50 con 0x7f3018004d40 2024-03-06T07:32:05.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.001+0000 7f3021ffb640 1 -- 172.21.15.83:0/2856289649 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3028014440 con 0x7f3018004d40 2024-03-06T07:32:05.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.001+0000 7f3033a2a640 1 -- 172.21.15.83:0/2856289649 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f302c14e2b0 con 0x7f3018004d40 2024-03-06T07:32:05.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.002+0000 7f3033a2a640 1 -- 172.21.15.83:0/2856289649 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f302c04f330 con 0x7f3018004d40 2024-03-06T07:32:05.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.002+0000 7f3021ffb640 1 -- 172.21.15.83:0/2856289649 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3028012040 con 0x7f3018004d40 2024-03-06T07:32:05.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.003+0000 7f3021ffb640 1 --2- 172.21.15.83:0/2856289649 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f301006b4d0 0x7f301006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.003+0000 7f3021ffb640 1 -- 172.21.15.83:0/2856289649 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f302808af10 con 0x7f3018004d40 2024-03-06T07:32:05.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.003+0000 7f3030fc5640 1 --2- 172.21.15.83:0/2856289649 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f301006b4d0 0x7f301006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.003+0000 7f3030fc5640 1 --2- 172.21.15.83:0/2856289649 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f301006b4d0 0x7f301006d990 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f302c06e2a0 tx=0x7f301c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:05.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.007+0000 7f3021ffb640 1 -- 172.21.15.83:0/2856289649 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f302802b020 con 0x7f3018004d40 2024-03-06T07:32:05.124 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:32:05.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.218+0000 7f3033a2a640 1 -- 172.21.15.83:0/2856289649 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f302c0d0310 con 0x7f3018004d40 2024-03-06T07:32:05.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.218+0000 7f3021ffb640 1 -- 172.21.15.83:0/2856289649 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f3028019030 con 0x7f3018004d40 2024-03-06T07:32:05.220 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:32:05.220 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:32:05.221 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:32:05.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.220+0000 7f3033a2a640 1 -- 172.21.15.83:0/2856289649 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f301006b4d0 msgr2=0x7f301006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.220+0000 7f3033a2a640 1 --2- 172.21.15.83:0/2856289649 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f301006b4d0 0x7f301006d990 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f302c06e2a0 tx=0x7f301c009040 comp rx=0 tx=0).stop 2024-03-06T07:32:05.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.220+0000 7f3033a2a640 1 -- 172.21.15.83:0/2856289649 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3018004d40 msgr2=0x7f302c1f8620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.220+0000 7f3033a2a640 1 --2- 172.21.15.83:0/2856289649 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3018004d40 0x7f302c1f8620 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f30280119a0 tx=0x7f3028011e60 comp rx=0 tx=0).stop 2024-03-06T07:32:05.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.220+0000 7f3033a2a640 1 -- 172.21.15.83:0/2856289649 shutdown_connections 2024-03-06T07:32:05.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.220+0000 7f3033a2a640 1 --2- 172.21.15.83:0/2856289649 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f301006b4d0 0x7f301006d990 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.220+0000 7f3033a2a640 1 --2- 172.21.15.83:0/2856289649 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f302c0b2d00 0x7f302c0b3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.220+0000 7f3033a2a640 1 --2- 172.21.15.83:0/2856289649 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f302c14daf0 0x7f302c1f74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.220+0000 7f3033a2a640 1 --2- 172.21.15.83:0/2856289649 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3018004d40 0x7f302c1f8620 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.220+0000 7f3033a2a640 1 -- 172.21.15.83:0/2856289649 >> 172.21.15.83:0/2856289649 conn(0x7f302c09a9f0 msgr2=0x7f302c0cfb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:05.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.220+0000 7f3033a2a640 1 -- 172.21.15.83:0/2856289649 shutdown_connections 2024-03-06T07:32:05.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:05.220+0000 7f3033a2a640 1 -- 172.21.15.83:0/2856289649 wait complete. 2024-03-06T07:32:05.434 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:31:26.591282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:32:05.435 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:32:06.436 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:06.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.534+0000 7ff250ac9640 1 Processor -- start 2024-03-06T07:32:06.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.535+0000 7ff250ac9640 1 -- start start 2024-03-06T07:32:06.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.535+0000 7ff250ac9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff24c15dfe0 0x7ff24c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.535+0000 7ff250ac9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff24c0cf830 0x7ff24c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.535+0000 7ff250ac9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff24c0b3430 con 0x7ff24c0d27f0 2024-03-06T07:32:06.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.535+0000 7ff250ac9640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff24c0d2c50 con 0x7ff24c0cf830 2024-03-06T07:32:06.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.535+0000 7ff250ac9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff24c0b31e0 con 0x7ff24c15dfe0 2024-03-06T07:32:06.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.535+0000 7ff243fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff24c0cf830 0x7ff24c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.535+0000 7ff243fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff24c0cf830 0x7ff24c0d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40938/0 (socket says 172.21.15.83:40938) 2024-03-06T07:32:06.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.535+0000 7ff243fff640 1 -- 172.21.15.83:0/2741565001 learned_addr learned my addr 172.21.15.83:0/2741565001 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:06.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.536+0000 7ff24a59c640 1 --2- 172.21.15.83:0/2741565001 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff24c15dfe0 0x7ff24c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.536+0000 7ff243fff640 1 -- 172.21.15.83:0/2741565001 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff24c15dfe0 msgr2=0x7ff24c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:06.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.536+0000 7ff243fff640 1 --2- 172.21.15.83:0/2741565001 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff24c15dfe0 0x7ff24c0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.536+0000 7ff243fff640 1 -- 172.21.15.83:0/2741565001 >> v1:172.21.15.183:6789/0 conn(0x7ff24c0d27f0 legacy=0x7ff24c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.536+0000 7ff243fff640 1 -- 172.21.15.83:0/2741565001 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff24c0b3710 con 0x7ff24c0cf830 2024-03-06T07:32:06.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.536+0000 7ff243fff640 1 --2- 172.21.15.83:0/2741565001 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff24c0cf830 0x7ff24c0d22b0 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7ff234009870 tx=0x7ff23402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20b2486fae9aa391 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:06.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.537+0000 7ff249d9b640 1 -- 172.21.15.83:0/2741565001 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff234034070 con 0x7ff24c0cf830 2024-03-06T07:32:06.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.537+0000 7ff249d9b640 1 -- 172.21.15.83:0/2741565001 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff234030840 con 0x7ff24c0cf830 2024-03-06T07:32:06.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.537+0000 7ff249d9b640 1 -- 172.21.15.83:0/2741565001 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff234030c10 con 0x7ff24c0cf830 2024-03-06T07:32:06.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.537+0000 7ff250ac9640 1 -- 172.21.15.83:0/2741565001 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff24c0cf830 msgr2=0x7ff24c0d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.537+0000 7ff250ac9640 1 --2- 172.21.15.83:0/2741565001 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff24c0cf830 0x7ff24c0d22b0 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7ff234009870 tx=0x7ff23402b980 comp rx=0 tx=0).stop 2024-03-06T07:32:06.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.537+0000 7ff250ac9640 1 -- 172.21.15.83:0/2741565001 shutdown_connections 2024-03-06T07:32:06.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.537+0000 7ff250ac9640 1 --2- 172.21.15.83:0/2741565001 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff24c0cf830 0x7ff24c0d22b0 unknown :-1 s=CLOSED pgs=1930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.537+0000 7ff250ac9640 1 --2- 172.21.15.83:0/2741565001 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff24c15dfe0 0x7ff24c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.537+0000 7ff250ac9640 1 -- 172.21.15.83:0/2741565001 >> 172.21.15.83:0/2741565001 conn(0x7ff24c0d3cd0 msgr2=0x7ff24c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:06.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.537+0000 7ff250ac9640 1 -- 172.21.15.83:0/2741565001 shutdown_connections 2024-03-06T07:32:06.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.537+0000 7ff250ac9640 1 -- 172.21.15.83:0/2741565001 wait complete. 2024-03-06T07:32:06.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.539+0000 7ff250ac9640 1 Processor -- start 2024-03-06T07:32:06.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.539+0000 7ff250ac9640 1 -- start start 2024-03-06T07:32:06.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.539+0000 7ff250ac9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff24c0cf830 0x7ff24c15a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.539+0000 7ff250ac9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff24c0d27f0 0x7ff24c0cc270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.539+0000 7ff250ac9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff24c15dfe0 0x7ff24c160390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.539+0000 7ff250ac9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff24c0d1c70 con 0x7ff24c0d27f0 2024-03-06T07:32:06.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.539+0000 7ff250ac9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff24c0b3d60 con 0x7ff24c15dfe0 2024-03-06T07:32:06.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.539+0000 7ff250ac9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff24c0d1df0 con 0x7ff24c0cf830 2024-03-06T07:32:06.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.539+0000 7ff243fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff24c0d27f0 0x7ff24c0cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.539+0000 7ff243fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff24c0d27f0 0x7ff24c0cc270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40950/0 (socket says 172.21.15.83:40950) 2024-03-06T07:32:06.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.539+0000 7ff243fff640 1 -- 172.21.15.83:0/2672296758 learned_addr learned my addr 172.21.15.83:0/2672296758 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:06.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.540+0000 7ff24a59c640 1 --2- 172.21.15.83:0/2672296758 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff24c0cf830 0x7ff24c15a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.540+0000 7ff24ad9d640 1 --2- 172.21.15.83:0/2672296758 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff24c15dfe0 0x7ff24c160390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.540+0000 7ff243fff640 1 -- 172.21.15.83:0/2672296758 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff24c0cf830 msgr2=0x7ff24c15a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.540+0000 7ff243fff640 1 --2- 172.21.15.83:0/2672296758 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff24c0cf830 0x7ff24c15a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.540+0000 7ff243fff640 1 -- 172.21.15.83:0/2672296758 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff24c15dfe0 msgr2=0x7ff24c160390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.540+0000 7ff243fff640 1 --2- 172.21.15.83:0/2672296758 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff24c15dfe0 0x7ff24c160390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.540+0000 7ff243fff640 1 -- 172.21.15.83:0/2672296758 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff24c0b6c50 con 0x7ff24c0d27f0 2024-03-06T07:32:06.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.540+0000 7ff24ad9d640 1 --2- 172.21.15.83:0/2672296758 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff24c15dfe0 0x7ff24c160390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:06.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.540+0000 7ff243fff640 1 --2- 172.21.15.83:0/2672296758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff24c0d27f0 0x7ff24c0cc270 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7ff23402b9c0 tx=0x7ff234004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:06.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.540+0000 7ff242ffd640 1 -- 172.21.15.83:0/2672296758 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff234048070 con 0x7ff24c0d27f0 2024-03-06T07:32:06.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.540+0000 7ff242ffd640 1 -- 172.21.15.83:0/2672296758 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff234030640 con 0x7ff24c0d27f0 2024-03-06T07:32:06.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.540+0000 7ff250ac9640 1 -- 172.21.15.83:0/2672296758 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff24c0b6f30 con 0x7ff24c0d27f0 2024-03-06T07:32:06.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.540+0000 7ff242ffd640 1 -- 172.21.15.83:0/2672296758 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff234034040 con 0x7ff24c0d27f0 2024-03-06T07:32:06.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.540+0000 7ff250ac9640 1 -- 172.21.15.83:0/2672296758 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff24c15f730 con 0x7ff24c0d27f0 2024-03-06T07:32:06.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.541+0000 7ff250ac9640 1 -- 172.21.15.83:0/2672296758 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff24c04f330 con 0x7ff24c0d27f0 2024-03-06T07:32:06.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.542+0000 7ff242ffd640 1 -- 172.21.15.83:0/2672296758 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff23404b070 con 0x7ff24c0d27f0 2024-03-06T07:32:06.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.542+0000 7ff242ffd640 1 --2- 172.21.15.83:0/2672296758 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff22c06b500 0x7ff22c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.542+0000 7ff242ffd640 1 -- 172.21.15.83:0/2672296758 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff2340ac1f0 con 0x7ff24c0d27f0 2024-03-06T07:32:06.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.542+0000 7ff24a59c640 1 --2- 172.21.15.83:0/2672296758 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff22c06b500 0x7ff22c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.542+0000 7ff24a59c640 1 --2- 172.21.15.83:0/2672296758 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff22c06b500 0x7ff22c06d9c0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7ff23c006fd0 tx=0x7ff23c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:06.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.546+0000 7ff242ffd640 1 -- 172.21.15.83:0/2672296758 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff23407b150 con 0x7ff24c0d27f0 2024-03-06T07:32:06.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.755+0000 7ff250ac9640 1 -- 172.21.15.83:0/2672296758 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff24c0747a0 con 0x7ff24c0d27f0 2024-03-06T07:32:06.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.756+0000 7ff242ffd640 1 -- 172.21.15.83:0/2672296758 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7ff234030810 con 0x7ff24c0d27f0 2024-03-06T07:32:06.757 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:32:06.758 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:32:06.758 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:32:06.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.758+0000 7ff250ac9640 1 -- 172.21.15.83:0/2672296758 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff22c06b500 msgr2=0x7ff22c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.758+0000 7ff250ac9640 1 --2- 172.21.15.83:0/2672296758 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff22c06b500 0x7ff22c06d9c0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7ff23c006fd0 tx=0x7ff23c008040 comp rx=0 tx=0).stop 2024-03-06T07:32:06.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.758+0000 7ff250ac9640 1 -- 172.21.15.83:0/2672296758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff24c0d27f0 msgr2=0x7ff24c0cc270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.758+0000 7ff250ac9640 1 --2- 172.21.15.83:0/2672296758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff24c0d27f0 0x7ff24c0cc270 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7ff23402b9c0 tx=0x7ff234004730 comp rx=0 tx=0).stop 2024-03-06T07:32:06.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.758+0000 7ff250ac9640 1 -- 172.21.15.83:0/2672296758 shutdown_connections 2024-03-06T07:32:06.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.758+0000 7ff250ac9640 1 --2- 172.21.15.83:0/2672296758 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff22c06b500 0x7ff22c06d9c0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.758+0000 7ff250ac9640 1 --2- 172.21.15.83:0/2672296758 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff24c15dfe0 0x7ff24c160390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.758+0000 7ff250ac9640 1 --2- 172.21.15.83:0/2672296758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff24c0d27f0 0x7ff24c0cc270 unknown :-1 s=CLOSED pgs=1931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.758+0000 7ff250ac9640 1 --2- 172.21.15.83:0/2672296758 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff24c0cf830 0x7ff24c15a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.758+0000 7ff250ac9640 1 -- 172.21.15.83:0/2672296758 >> 172.21.15.83:0/2672296758 conn(0x7ff24c0d3cd0 msgr2=0x7ff24c0cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:06.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.758+0000 7ff250ac9640 1 -- 172.21.15.83:0/2672296758 shutdown_connections 2024-03-06T07:32:06.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:06.758+0000 7ff250ac9640 1 -- 172.21.15.83:0/2672296758 wait complete. 2024-03-06T07:32:06.971 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:31:26.591282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:32:06.971 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:32:07.126 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:32:07.972 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:08.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.073+0000 7f4e2302a640 1 Processor -- start 2024-03-06T07:32:08.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.073+0000 7f4e2302a640 1 -- start start 2024-03-06T07:32:08.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.074+0000 7f4e2302a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f4e1c15ac70 0x7f4e1c0cdcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.074+0000 7f4e2302a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4e1c06d0d0 0x7f4e1c0ce1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.074+0000 7f4e2302a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e1c06efb0 con 0x7f4e1c15e650 2024-03-06T07:32:08.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.074+0000 7f4e2302a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e1c06b170 con 0x7f4e1c06d0d0 2024-03-06T07:32:08.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.074+0000 7f4e2302a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e1c06ec60 con 0x7f4e1c15ac70 2024-03-06T07:32:08.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.074+0000 7f4e1bfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4e1c06d0d0 0x7f4e1c0ce1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.074+0000 7f4e215c7640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f4e1c15e650 0x7f4e1c15eac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52182/0 (socket says 172.21.15.83:52182) 2024-03-06T07:32:08.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.074+0000 7f4e215c7640 1 -- 172.21.15.83:0/1375548376 learned_addr learned my addr 172.21.15.83:0/1375548376 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:32:08.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.075+0000 7f4e20dc6640 1 --2- 172.21.15.83:0/1375548376 >> v2:172.21.15.183:3300/0 conn(0x7f4e1c15ac70 0x7f4e1c0cdcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.075+0000 7f4e1bfff640 1 -- 172.21.15.83:0/1375548376 >> v2:172.21.15.183:3300/0 conn(0x7f4e1c15ac70 msgr2=0x7f4e1c0cdcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:08.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.075+0000 7f4e1bfff640 1 --2- 172.21.15.83:0/1375548376 >> v2:172.21.15.183:3300/0 conn(0x7f4e1c15ac70 0x7f4e1c0cdcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.075+0000 7f4e1bfff640 1 -- 172.21.15.83:0/1375548376 >> v1:172.21.15.83:6789/0 conn(0x7f4e1c15e650 legacy=0x7f4e1c15eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.075+0000 7f4e1bfff640 1 -- 172.21.15.83:0/1375548376 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e1c06e6e0 con 0x7f4e1c06d0d0 2024-03-06T07:32:08.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.075+0000 7f4e1b7fe640 1 -- 172.21.15.83:0/1375548376 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3090605860 0 0) 0x7f4e1c06efb0 con 0x7f4e1c15e650 2024-03-06T07:32:08.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.075+0000 7f4e1bfff640 1 --2- 172.21.15.83:0/1375548376 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e1c06d0d0 0x7f4e1c0ce1f0 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c067a90 tx=0x7f4e0802b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41cfbf13d4d5803d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:08.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.076+0000 7f4e1b7fe640 1 -- 172.21.15.83:0/1375548376 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e08038070 con 0x7f4e1c06d0d0 2024-03-06T07:32:08.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.076+0000 7f4e1b7fe640 1 -- 172.21.15.83:0/1375548376 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e08031710 con 0x7f4e1c06d0d0 2024-03-06T07:32:08.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.076+0000 7f4e1b7fe640 1 -- 172.21.15.83:0/1375548376 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e08031ae0 con 0x7f4e1c06d0d0 2024-03-06T07:32:08.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.076+0000 7f4e2302a640 1 -- 172.21.15.83:0/1375548376 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e1c06d0d0 msgr2=0x7f4e1c0ce1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.076+0000 7f4e2302a640 1 --2- 172.21.15.83:0/1375548376 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e1c06d0d0 0x7f4e1c0ce1f0 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c067a90 tx=0x7f4e0802b7b0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.076+0000 7f4e2302a640 1 -- 172.21.15.83:0/1375548376 shutdown_connections 2024-03-06T07:32:08.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.076+0000 7f4e2302a640 1 --2- 172.21.15.83:0/1375548376 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e1c06d0d0 0x7f4e1c0ce1f0 unknown :-1 s=CLOSED pgs=1933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.076+0000 7f4e2302a640 1 --2- 172.21.15.83:0/1375548376 >> v2:172.21.15.183:3300/0 conn(0x7f4e1c15ac70 0x7f4e1c0cdcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.076+0000 7f4e2302a640 1 -- 172.21.15.83:0/1375548376 >> 172.21.15.83:0/1375548376 conn(0x7f4e1c0d00e0 msgr2=0x7f4e1c0d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:08.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.076+0000 7f4e2302a640 1 -- 172.21.15.83:0/1375548376 shutdown_connections 2024-03-06T07:32:08.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.076+0000 7f4e2302a640 1 -- 172.21.15.83:0/1375548376 wait complete. 2024-03-06T07:32:08.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.077+0000 7f4e2302a640 1 Processor -- start 2024-03-06T07:32:08.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.077+0000 7f4e2302a640 1 -- start start 2024-03-06T07:32:08.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.078+0000 7f4e2302a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4e1c06d0d0 0x7f4e1c0b59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.078+0000 7f4e2302a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e1c15ac70 0x7f4e1c149ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.078+0000 7f4e2302a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4e1c15e650 0x7f4e1c14bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.078+0000 7f4e2302a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e1c06e910 con 0x7f4e1c15ac70 2024-03-06T07:32:08.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.078+0000 7f4e2302a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e1c06e400 con 0x7f4e1c06d0d0 2024-03-06T07:32:08.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.078+0000 7f4e2302a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e1c09dec0 con 0x7f4e1c15e650 2024-03-06T07:32:08.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.078+0000 7f4e1bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e1c15ac70 0x7f4e1c149ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.078+0000 7f4e1bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e1c15ac70 0x7f4e1c149ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53256/0 (socket says 172.21.15.83:53256) 2024-03-06T07:32:08.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.078+0000 7f4e1bfff640 1 -- 172.21.15.83:0/2858567176 learned_addr learned my addr 172.21.15.83:0/2858567176 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:08.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.078+0000 7f4e20dc6640 1 --2- 172.21.15.83:0/2858567176 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4e1c06d0d0 0x7f4e1c0b59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.078+0000 7f4e215c7640 1 --2- 172.21.15.83:0/2858567176 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4e1c15e650 0x7f4e1c14bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.079+0000 7f4e1bfff640 1 -- 172.21.15.83:0/2858567176 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4e1c15e650 msgr2=0x7f4e1c14bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.079+0000 7f4e1bfff640 1 --2- 172.21.15.83:0/2858567176 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4e1c15e650 0x7f4e1c14bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.079+0000 7f4e1bfff640 1 -- 172.21.15.83:0/2858567176 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4e1c06d0d0 msgr2=0x7f4e1c0b59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.079+0000 7f4e1bfff640 1 --2- 172.21.15.83:0/2858567176 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4e1c06d0d0 0x7f4e1c0b59c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.079+0000 7f4e1bfff640 1 -- 172.21.15.83:0/2858567176 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e1c09ae60 con 0x7f4e1c15ac70 2024-03-06T07:32:08.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.079+0000 7f4e20dc6640 1 --2- 172.21.15.83:0/2858567176 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4e1c06d0d0 0x7f4e1c0b59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:08.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.079+0000 7f4e1bfff640 1 --2- 172.21.15.83:0/2858567176 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e1c15ac70 0x7f4e1c149ad0 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7f4e08009890 tx=0x7f4e080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:08.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.079+0000 7f4e197fa640 1 -- 172.21.15.83:0/2858567176 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e08042030 con 0x7f4e1c15ac70 2024-03-06T07:32:08.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.079+0000 7f4e197fa640 1 -- 172.21.15.83:0/2858567176 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e08031510 con 0x7f4e1c15ac70 2024-03-06T07:32:08.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.079+0000 7f4e2302a640 1 -- 172.21.15.83:0/2858567176 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e1c09b140 con 0x7f4e1c15ac70 2024-03-06T07:32:08.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.079+0000 7f4e197fa640 1 -- 172.21.15.83:0/2858567176 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e08038040 con 0x7f4e1c15ac70 2024-03-06T07:32:08.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.079+0000 7f4e2302a640 1 -- 172.21.15.83:0/2858567176 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e1c0b3250 con 0x7f4e1c15ac70 2024-03-06T07:32:08.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.080+0000 7f4e2302a640 1 -- 172.21.15.83:0/2858567176 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e1c06c7e0 con 0x7f4e1c15ac70 2024-03-06T07:32:08.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.081+0000 7f4e197fa640 1 -- 172.21.15.83:0/2858567176 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4e08045070 con 0x7f4e1c15ac70 2024-03-06T07:32:08.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.081+0000 7f4e197fa640 1 --2- 172.21.15.83:0/2858567176 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4e0006b4d0 0x7f4e0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.081+0000 7f4e197fa640 1 -- 172.21.15.83:0/2858567176 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4e080abce0 con 0x7f4e1c15ac70 2024-03-06T07:32:08.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.081+0000 7f4e20dc6640 1 --2- 172.21.15.83:0/2858567176 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4e0006b4d0 0x7f4e0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.081+0000 7f4e20dc6640 1 --2- 172.21.15.83:0/2858567176 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4e0006b4d0 0x7f4e0006d990 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c15f5c0 tx=0x7f4e10008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:08.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.085+0000 7f4e197fa640 1 -- 172.21.15.83:0/2858567176 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4e1c06c7e0 con 0x7f4e1c15ac70 2024-03-06T07:32:08.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.302+0000 7f4e2302a640 1 -- 172.21.15.83:0/2858567176 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e1c04f330 con 0x7f4e1c15ac70 2024-03-06T07:32:08.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.302+0000 7f4e197fa640 1 -- 172.21.15.83:0/2858567176 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f4e0807ab20 con 0x7f4e1c15ac70 2024-03-06T07:32:08.305 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:32:08.305 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:32:08.305 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:32:08.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.305+0000 7f4e2302a640 1 -- 172.21.15.83:0/2858567176 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4e0006b4d0 msgr2=0x7f4e0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.305+0000 7f4e2302a640 1 --2- 172.21.15.83:0/2858567176 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4e0006b4d0 0x7f4e0006d990 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c15f5c0 tx=0x7f4e10008040 comp rx=0 tx=0).stop 2024-03-06T07:32:08.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.305+0000 7f4e2302a640 1 -- 172.21.15.83:0/2858567176 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e1c15ac70 msgr2=0x7f4e1c149ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.305+0000 7f4e2302a640 1 --2- 172.21.15.83:0/2858567176 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e1c15ac70 0x7f4e1c149ad0 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7f4e08009890 tx=0x7f4e080046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.305+0000 7f4e2302a640 1 -- 172.21.15.83:0/2858567176 shutdown_connections 2024-03-06T07:32:08.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.305+0000 7f4e2302a640 1 --2- 172.21.15.83:0/2858567176 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4e0006b4d0 0x7f4e0006d990 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.305+0000 7f4e2302a640 1 --2- 172.21.15.83:0/2858567176 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4e1c15e650 0x7f4e1c14bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.305+0000 7f4e2302a640 1 --2- 172.21.15.83:0/2858567176 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e1c15ac70 0x7f4e1c149ad0 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.305+0000 7f4e2302a640 1 --2- 172.21.15.83:0/2858567176 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4e1c06d0d0 0x7f4e1c0b59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.305+0000 7f4e2302a640 1 -- 172.21.15.83:0/2858567176 >> 172.21.15.83:0/2858567176 conn(0x7f4e1c0d00e0 msgr2=0x7f4e1c15eac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:08.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.305+0000 7f4e2302a640 1 -- 172.21.15.83:0/2858567176 shutdown_connections 2024-03-06T07:32:08.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:08.305+0000 7f4e2302a640 1 -- 172.21.15.83:0/2858567176 wait complete. 2024-03-06T07:32:08.519 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:31:26.591282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:32:08.520 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:32:09.127 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:32:09.521 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:09.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.620+0000 7fd4845ac640 1 Processor -- start 2024-03-06T07:32:09.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.620+0000 7fd4845ac640 1 -- start start 2024-03-06T07:32:09.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.621+0000 7fd4845ac640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd47c15bb20 0x7fd47c0d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.621+0000 7fd4845ac640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd47c0cd8c0 0x7fd47c0d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.621+0000 7fd4845ac640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd47c161980 con 0x7fd47c0cfff0 2024-03-06T07:32:09.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.621+0000 7fd4845ac640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd47c161c60 con 0x7fd47c0cd8c0 2024-03-06T07:32:09.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.621+0000 7fd4845ac640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd47c161f40 con 0x7fd47c15bb20 2024-03-06T07:32:09.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.621+0000 7fd482b49640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd47c0cd8c0 0x7fd47c0d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.621+0000 7fd482b49640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd47c0cd8c0 0x7fd47c0d0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53260/0 (socket says 172.21.15.83:53260) 2024-03-06T07:32:09.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.621+0000 7fd482b49640 1 -- 172.21.15.83:0/1572115662 learned_addr learned my addr 172.21.15.83:0/1572115662 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:09.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.621+0000 7fd481b47640 1 --1- 172.21.15.83:0/1572115662 >> v1:172.21.15.83:6789/0 conn(0x7fd47c0cfff0 0x7fd47c0ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52190/0 (socket says 172.21.15.83:52190) 2024-03-06T07:32:09.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.622+0000 7fd482348640 1 --2- 172.21.15.83:0/1572115662 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd47c15bb20 0x7fd47c0d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.622+0000 7fd482b49640 1 -- 172.21.15.83:0/1572115662 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd47c15bb20 msgr2=0x7fd47c0d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:09.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.622+0000 7fd482b49640 1 --2- 172.21.15.83:0/1572115662 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd47c15bb20 0x7fd47c0d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.622+0000 7fd482b49640 1 -- 172.21.15.83:0/1572115662 >> v1:172.21.15.83:6789/0 conn(0x7fd47c0cfff0 legacy=0x7fd47c0ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.622+0000 7fd482b49640 1 -- 172.21.15.83:0/1572115662 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd47c162220 con 0x7fd47c0cd8c0 2024-03-06T07:32:09.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.622+0000 7fd482b49640 1 --2- 172.21.15.83:0/1572115662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd47c0cd8c0 0x7fd47c0d0470 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7fd474009e50 tx=0x7fd47402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fd4cd7db8ab5323 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:09.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.622+0000 7fd481346640 1 -- 172.21.15.83:0/1572115662 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd47402ca70 con 0x7fd47c0cd8c0 2024-03-06T07:32:09.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.622+0000 7fd481346640 1 -- 172.21.15.83:0/1572115662 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd47402cc10 con 0x7fd47c0cd8c0 2024-03-06T07:32:09.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.623+0000 7fd481346640 1 -- 172.21.15.83:0/1572115662 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4740343f0 con 0x7fd47c0cd8c0 2024-03-06T07:32:09.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.623+0000 7fd4845ac640 1 -- 172.21.15.83:0/1572115662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd47c0cd8c0 msgr2=0x7fd47c0d0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.623+0000 7fd4845ac640 1 --2- 172.21.15.83:0/1572115662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd47c0cd8c0 0x7fd47c0d0470 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7fd474009e50 tx=0x7fd47402bed0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.623+0000 7fd4845ac640 1 -- 172.21.15.83:0/1572115662 shutdown_connections 2024-03-06T07:32:09.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.623+0000 7fd4845ac640 1 --2- 172.21.15.83:0/1572115662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd47c0cd8c0 0x7fd47c0d0470 unknown :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.623+0000 7fd4845ac640 1 --2- 172.21.15.83:0/1572115662 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd47c15bb20 0x7fd47c0d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.623+0000 7fd4845ac640 1 -- 172.21.15.83:0/1572115662 >> 172.21.15.83:0/1572115662 conn(0x7fd47c09a9f0 msgr2=0x7fd47c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:09.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.623+0000 7fd4845ac640 1 -- 172.21.15.83:0/1572115662 shutdown_connections 2024-03-06T07:32:09.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.623+0000 7fd4845ac640 1 -- 172.21.15.83:0/1572115662 wait complete. 2024-03-06T07:32:09.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.625+0000 7fd4845ac640 1 Processor -- start 2024-03-06T07:32:09.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.625+0000 7fd4845ac640 1 -- start start 2024-03-06T07:32:09.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.626+0000 7fd4845ac640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd47c0cd8c0 0x7fd47c15af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.626+0000 7fd4845ac640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd47c0cfff0 0x7fd47c0c5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.626+0000 7fd4845ac640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd47c15bb20 0x7fd47c0c6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.626+0000 7fd4845ac640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd47c0d24b0 con 0x7fd47c15bb20 2024-03-06T07:32:09.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.626+0000 7fd4845ac640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd47c0d2330 con 0x7fd47c0cfff0 2024-03-06T07:32:09.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.626+0000 7fd4845ac640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd47c0d1980 con 0x7fd47c0cd8c0 2024-03-06T07:32:09.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.626+0000 7fd482b49640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd47c15bb20 0x7fd47c0c6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.627+0000 7fd482b49640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd47c15bb20 0x7fd47c0c6420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53266/0 (socket says 172.21.15.83:53266) 2024-03-06T07:32:09.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.627+0000 7fd482b49640 1 -- 172.21.15.83:0/1578171661 learned_addr learned my addr 172.21.15.83:0/1578171661 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:09.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.627+0000 7fd482348640 1 --2- 172.21.15.83:0/1578171661 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd47c0cd8c0 0x7fd47c15af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.627+0000 7fd481b47640 1 --2- 172.21.15.83:0/1578171661 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd47c0cfff0 0x7fd47c0c5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.627+0000 7fd482b49640 1 -- 172.21.15.83:0/1578171661 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd47c0cd8c0 msgr2=0x7fd47c15af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.627+0000 7fd482b49640 1 --2- 172.21.15.83:0/1578171661 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd47c0cd8c0 0x7fd47c15af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.627+0000 7fd482b49640 1 -- 172.21.15.83:0/1578171661 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd47c0cfff0 msgr2=0x7fd47c0c5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.627+0000 7fd482b49640 1 --2- 172.21.15.83:0/1578171661 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd47c0cfff0 0x7fd47c0c5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.627+0000 7fd482b49640 1 -- 172.21.15.83:0/1578171661 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd47c160210 con 0x7fd47c15bb20 2024-03-06T07:32:09.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.627+0000 7fd482b49640 1 --2- 172.21.15.83:0/1578171661 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd47c15bb20 0x7fd47c0c6420 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7fd474009e50 tx=0x7fd474004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:09.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.627+0000 7fd472ffd640 1 -- 172.21.15.83:0/1578171661 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd474043020 con 0x7fd47c15bb20 2024-03-06T07:32:09.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.627+0000 7fd472ffd640 1 -- 172.21.15.83:0/1578171661 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd47403e450 con 0x7fd47c15bb20 2024-03-06T07:32:09.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.627+0000 7fd4845ac640 1 -- 172.21.15.83:0/1578171661 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd47c1604f0 con 0x7fd47c15bb20 2024-03-06T07:32:09.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.627+0000 7fd472ffd640 1 -- 172.21.15.83:0/1578171661 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd474009510 con 0x7fd47c15bb20 2024-03-06T07:32:09.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.627+0000 7fd4845ac640 1 -- 172.21.15.83:0/1578171661 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd47c160ab0 con 0x7fd47c15bb20 2024-03-06T07:32:09.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.628+0000 7fd4845ac640 1 -- 172.21.15.83:0/1578171661 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd47c04f330 con 0x7fd47c15bb20 2024-03-06T07:32:09.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.629+0000 7fd472ffd640 1 -- 172.21.15.83:0/1578171661 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd474046020 con 0x7fd47c15bb20 2024-03-06T07:32:09.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.629+0000 7fd472ffd640 1 --2- 172.21.15.83:0/1578171661 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd46006b4d0 0x7fd46006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.629+0000 7fd472ffd640 1 -- 172.21.15.83:0/1578171661 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd4740acdc0 con 0x7fd47c15bb20 2024-03-06T07:32:09.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.629+0000 7fd482348640 1 --2- 172.21.15.83:0/1578171661 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd46006b4d0 0x7fd46006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.630+0000 7fd482348640 1 --2- 172.21.15.83:0/1578171661 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd46006b4d0 0x7fd46006d990 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fd46c006fd0 tx=0x7fd46c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:09.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.633+0000 7fd472ffd640 1 -- 172.21.15.83:0/1578171661 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd47404d030 con 0x7fd47c15bb20 2024-03-06T07:32:09.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.848+0000 7fd4845ac640 1 -- 172.21.15.83:0/1578171661 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd47c09b090 con 0x7fd47c15bb20 2024-03-06T07:32:09.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.849+0000 7fd472ffd640 1 -- 172.21.15.83:0/1578171661 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7fd47407bce0 con 0x7fd47c15bb20 2024-03-06T07:32:09.851 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:32:09.851 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:32:09.851 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:32:09.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.850+0000 7fd4845ac640 1 -- 172.21.15.83:0/1578171661 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd46006b4d0 msgr2=0x7fd46006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.850+0000 7fd4845ac640 1 --2- 172.21.15.83:0/1578171661 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd46006b4d0 0x7fd46006d990 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fd46c006fd0 tx=0x7fd46c008040 comp rx=0 tx=0).stop 2024-03-06T07:32:09.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.850+0000 7fd4845ac640 1 -- 172.21.15.83:0/1578171661 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd47c15bb20 msgr2=0x7fd47c0c6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.850+0000 7fd4845ac640 1 --2- 172.21.15.83:0/1578171661 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd47c15bb20 0x7fd47c0c6420 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7fd474009e50 tx=0x7fd474004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:09.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.851+0000 7fd4845ac640 1 -- 172.21.15.83:0/1578171661 shutdown_connections 2024-03-06T07:32:09.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.851+0000 7fd4845ac640 1 --2- 172.21.15.83:0/1578171661 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd46006b4d0 0x7fd46006d990 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.851+0000 7fd4845ac640 1 --2- 172.21.15.83:0/1578171661 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd47c15bb20 0x7fd47c0c6420 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.851+0000 7fd4845ac640 1 --2- 172.21.15.83:0/1578171661 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd47c0cfff0 0x7fd47c0c5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.851+0000 7fd4845ac640 1 --2- 172.21.15.83:0/1578171661 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd47c0cd8c0 0x7fd47c15af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.851+0000 7fd4845ac640 1 -- 172.21.15.83:0/1578171661 >> 172.21.15.83:0/1578171661 conn(0x7fd47c09a9f0 msgr2=0x7fd47c06eb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:09.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.851+0000 7fd4845ac640 1 -- 172.21.15.83:0/1578171661 shutdown_connections 2024-03-06T07:32:09.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:09.851+0000 7fd4845ac640 1 -- 172.21.15.83:0/1578171661 wait complete. 2024-03-06T07:32:10.064 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:31:26.591282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:32:10.065 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:32:11.066 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:11.129 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:32:11.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.167+0000 7f414ed13640 1 Processor -- start 2024-03-06T07:32:11.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.167+0000 7f414ed13640 1 -- start start 2024-03-06T07:32:11.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.167+0000 7f414ed13640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f414815dfe0 0x7f41480cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.167+0000 7f414ed13640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f41480cf830 0x7f41480d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.167+0000 7f414ed13640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41480b3430 con 0x7f41480d27f0 2024-03-06T07:32:11.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.167+0000 7f414ed13640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41480b31e0 con 0x7f41480cf830 2024-03-06T07:32:11.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.167+0000 7f414ed13640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41480b3d60 con 0x7f414815dfe0 2024-03-06T07:32:11.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.168+0000 7f4147fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f41480cf830 0x7f41480d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.168+0000 7f414d2b0640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f41480d27f0 0x7f41480d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52198/0 (socket says 172.21.15.83:52198) 2024-03-06T07:32:11.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.168+0000 7f414d2b0640 1 -- 172.21.15.83:0/1851580487 learned_addr learned my addr 172.21.15.83:0/1851580487 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:32:11.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.168+0000 7f4147fff640 1 -- 172.21.15.83:0/1851580487 >> v2:172.21.15.183:3300/0 conn(0x7f414815dfe0 msgr2=0x7f41480cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:11.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.168+0000 7f4147fff640 1 --2- 172.21.15.83:0/1851580487 >> v2:172.21.15.183:3300/0 conn(0x7f414815dfe0 0x7f41480cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.168+0000 7f4147fff640 1 -- 172.21.15.83:0/1851580487 >> v1:172.21.15.83:6789/0 conn(0x7f41480d27f0 legacy=0x7f41480d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.168+0000 7f4147fff640 1 -- 172.21.15.83:0/1851580487 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41480b3710 con 0x7f41480cf830 2024-03-06T07:32:11.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.168+0000 7f41477fe640 1 -- 172.21.15.83:0/1851580487 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2765715513 0 0) 0x7f41480b3430 con 0x7f41480d27f0 2024-03-06T07:32:11.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.169+0000 7f4147fff640 1 --2- 172.21.15.83:0/1851580487 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41480cf830 0x7f41480d22b0 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f41340097a0 tx=0x7f413402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f56fe03f321ca8b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:11.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.169+0000 7f41477fe640 1 -- 172.21.15.83:0/1851580487 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4134035020 con 0x7f41480cf830 2024-03-06T07:32:11.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.169+0000 7f41477fe640 1 -- 172.21.15.83:0/1851580487 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4134031830 con 0x7f41480cf830 2024-03-06T07:32:11.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.169+0000 7f41477fe640 1 -- 172.21.15.83:0/1851580487 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4134031c00 con 0x7f41480cf830 2024-03-06T07:32:11.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.169+0000 7f414ed13640 1 -- 172.21.15.83:0/1851580487 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41480cf830 msgr2=0x7f41480d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.169+0000 7f414ed13640 1 --2- 172.21.15.83:0/1851580487 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41480cf830 0x7f41480d22b0 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f41340097a0 tx=0x7f413402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.169+0000 7f414ed13640 1 -- 172.21.15.83:0/1851580487 shutdown_connections 2024-03-06T07:32:11.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.169+0000 7f414ed13640 1 --2- 172.21.15.83:0/1851580487 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41480cf830 0x7f41480d22b0 unknown :-1 s=CLOSED pgs=1939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.169+0000 7f414ed13640 1 --2- 172.21.15.83:0/1851580487 >> v2:172.21.15.183:3300/0 conn(0x7f414815dfe0 0x7f41480cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.169+0000 7f414ed13640 1 -- 172.21.15.83:0/1851580487 >> 172.21.15.83:0/1851580487 conn(0x7f41480d3cd0 msgr2=0x7f41480d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:11.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.169+0000 7f414ed13640 1 -- 172.21.15.83:0/1851580487 shutdown_connections 2024-03-06T07:32:11.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.169+0000 7f414ed13640 1 -- 172.21.15.83:0/1851580487 wait complete. 2024-03-06T07:32:11.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.170+0000 7f414ed13640 1 Processor -- start 2024-03-06T07:32:11.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.170+0000 7f414ed13640 1 -- start start 2024-03-06T07:32:11.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.171+0000 7f414ed13640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f41480cf830 0x7f414815c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.171+0000 7f414ed13640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f41480d27f0 0x7f414806cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.171+0000 7f414ed13640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f414815dfe0 0x7f414806d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.171+0000 7f414ed13640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41480d1df0 con 0x7f414815dfe0 2024-03-06T07:32:11.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.171+0000 7f414ed13640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41480d1c70 con 0x7f41480cf830 2024-03-06T07:32:11.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.171+0000 7f414ed13640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f414815e3c0 con 0x7f41480d27f0 2024-03-06T07:32:11.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.171+0000 7f414d2b0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f414815dfe0 0x7f414806d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.171+0000 7f414d2b0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f414815dfe0 0x7f414806d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53276/0 (socket says 172.21.15.83:53276) 2024-03-06T07:32:11.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.171+0000 7f414d2b0640 1 -- 172.21.15.83:0/2443864296 learned_addr learned my addr 172.21.15.83:0/2443864296 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:11.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.171+0000 7f414caaf640 1 --2- 172.21.15.83:0/2443864296 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f41480cf830 0x7f414815c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.172+0000 7f4147fff640 1 --2- 172.21.15.83:0/2443864296 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f41480d27f0 0x7f414806cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.172+0000 7f414d2b0640 1 -- 172.21.15.83:0/2443864296 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f41480d27f0 msgr2=0x7f414806cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.172+0000 7f414d2b0640 1 --2- 172.21.15.83:0/2443864296 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f41480d27f0 0x7f414806cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.172+0000 7f414d2b0640 1 -- 172.21.15.83:0/2443864296 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f41480cf830 msgr2=0x7f414815c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.172+0000 7f414d2b0640 1 --2- 172.21.15.83:0/2443864296 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f41480cf830 0x7f414815c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.172+0000 7f414d2b0640 1 -- 172.21.15.83:0/2443864296 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41481f7100 con 0x7f414815dfe0 2024-03-06T07:32:11.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.172+0000 7f414d2b0640 1 --2- 172.21.15.83:0/2443864296 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f414815dfe0 0x7f414806d020 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f414000c7a0 tx=0x7f414000cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:11.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.172+0000 7f41457fa640 1 -- 172.21.15.83:0/2443864296 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4140018070 con 0x7f414815dfe0 2024-03-06T07:32:11.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.172+0000 7f41457fa640 1 -- 172.21.15.83:0/2443864296 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4140004d60 con 0x7f414815dfe0 2024-03-06T07:32:11.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.172+0000 7f41457fa640 1 -- 172.21.15.83:0/2443864296 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414000a880 con 0x7f414815dfe0 2024-03-06T07:32:11.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.172+0000 7f414ed13640 1 -- 172.21.15.83:0/2443864296 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41481f73e0 con 0x7f414815dfe0 2024-03-06T07:32:11.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.172+0000 7f414ed13640 1 -- 172.21.15.83:0/2443864296 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41481f79a0 con 0x7f414815dfe0 2024-03-06T07:32:11.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.173+0000 7f414ed13640 1 -- 172.21.15.83:0/2443864296 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f414804f330 con 0x7f414815dfe0 2024-03-06T07:32:11.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.174+0000 7f41457fa640 1 -- 172.21.15.83:0/2443864296 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f414001b070 con 0x7f414815dfe0 2024-03-06T07:32:11.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.174+0000 7f41457fa640 1 --2- 172.21.15.83:0/2443864296 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f412c06b4d0 0x7f412c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.174+0000 7f41457fa640 1 -- 172.21.15.83:0/2443864296 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4140089680 con 0x7f414815dfe0 2024-03-06T07:32:11.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.174+0000 7f414caaf640 1 --2- 172.21.15.83:0/2443864296 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f412c06b4d0 0x7f412c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.174+0000 7f414caaf640 1 --2- 172.21.15.83:0/2443864296 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f412c06b4d0 0x7f412c06d990 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f413c009990 tx=0x7f413c009110 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:11.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.178+0000 7f41457fa640 1 -- 172.21.15.83:0/2443864296 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4140057f00 con 0x7f414815dfe0 2024-03-06T07:32:11.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.467+0000 7f414ed13640 1 -- 172.21.15.83:0/2443864296 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41480b4320 con 0x7f414815dfe0 2024-03-06T07:32:11.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.468+0000 7f41457fa640 1 -- 172.21.15.83:0/2443864296 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f4140006e20 con 0x7f414815dfe0 2024-03-06T07:32:11.470 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:32:11.470 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:32:11.470 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:32:11.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.470+0000 7f414ed13640 1 -- 172.21.15.83:0/2443864296 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f412c06b4d0 msgr2=0x7f412c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.470+0000 7f414ed13640 1 --2- 172.21.15.83:0/2443864296 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f412c06b4d0 0x7f412c06d990 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f413c009990 tx=0x7f413c009110 comp rx=0 tx=0).stop 2024-03-06T07:32:11.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.470+0000 7f414ed13640 1 -- 172.21.15.83:0/2443864296 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f414815dfe0 msgr2=0x7f414806d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.470+0000 7f414ed13640 1 --2- 172.21.15.83:0/2443864296 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f414815dfe0 0x7f414806d020 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f414000c7a0 tx=0x7f414000cba0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.470+0000 7f414ed13640 1 -- 172.21.15.83:0/2443864296 shutdown_connections 2024-03-06T07:32:11.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.470+0000 7f414ed13640 1 --2- 172.21.15.83:0/2443864296 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f412c06b4d0 0x7f412c06d990 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.470+0000 7f414ed13640 1 --2- 172.21.15.83:0/2443864296 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f414815dfe0 0x7f414806d020 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.470+0000 7f414ed13640 1 --2- 172.21.15.83:0/2443864296 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f41480d27f0 0x7f414806cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.470+0000 7f414ed13640 1 --2- 172.21.15.83:0/2443864296 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f41480cf830 0x7f414815c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.470+0000 7f414ed13640 1 -- 172.21.15.83:0/2443864296 >> 172.21.15.83:0/2443864296 conn(0x7f41480d3cd0 msgr2=0x7f41480d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:11.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.470+0000 7f414ed13640 1 -- 172.21.15.83:0/2443864296 shutdown_connections 2024-03-06T07:32:11.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:11.470+0000 7f414ed13640 1 -- 172.21.15.83:0/2443864296 wait complete. 2024-03-06T07:32:11.684 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:31:26.591282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:32:11.684 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:32:12.685 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:12.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.804+0000 7f4f12a84640 1 Processor -- start 2024-03-06T07:32:12.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.804+0000 7f4f12a84640 1 -- start start 2024-03-06T07:32:12.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.805+0000 7f4f12a84640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4f0c1515e0 0x7f4f0c0cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.805+0000 7f4f12a84640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4f0c15b430 0x7f4f0c071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.805+0000 7f4f12a84640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f0c15b8b0 con 0x7f4f0c0cd7a0 2024-03-06T07:32:12.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.805+0000 7f4f12a84640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f0c070a70 con 0x7f4f0c1515e0 2024-03-06T07:32:12.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.805+0000 7f4f12a84640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0c070d50 con 0x7f4f0c15b430 2024-03-06T07:32:12.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.805+0000 7f4f10820640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4f0c1515e0 0x7f4f0c0cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.805+0000 7f4f10820640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4f0c1515e0 0x7f4f0c0cce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53280/0 (socket says 172.21.15.83:53280) 2024-03-06T07:32:12.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.805+0000 7f4f10820640 1 -- 172.21.15.83:0/131595133 learned_addr learned my addr 172.21.15.83:0/131595133 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:12.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.805+0000 7f4f0bfff640 1 --2- 172.21.15.83:0/131595133 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4f0c15b430 0x7f4f0c071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.805+0000 7f4f10820640 1 -- 172.21.15.83:0/131595133 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4f0c15b430 msgr2=0x7f4f0c071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:12.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.805+0000 7f4f10820640 1 --2- 172.21.15.83:0/131595133 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4f0c15b430 0x7f4f0c071830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.805+0000 7f4f10820640 1 -- 172.21.15.83:0/131595133 >> v1:172.21.15.183:6789/0 conn(0x7f4f0c0cd7a0 legacy=0x7f4f0c0cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:12.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.805+0000 7f4f10820640 1 -- 172.21.15.83:0/131595133 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f0c071030 con 0x7f4f0c1515e0 2024-03-06T07:32:12.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.806+0000 7f4f0bfff640 1 --2- 172.21.15.83:0/131595133 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4f0c15b430 0x7f4f0c071830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:12.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.806+0000 7f4f10820640 1 --2- 172.21.15.83:0/131595133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f0c1515e0 0x7f4f0c0cce40 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f4f0000bb10 tx=0x7f4f0002e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46c76f35038f9385 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:12.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.806+0000 7f4f0b7fe640 1 -- 172.21.15.83:0/131595133 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f0003b070 con 0x7f4f0c1515e0 2024-03-06T07:32:12.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.806+0000 7f4f0b7fe640 1 -- 172.21.15.83:0/131595133 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f00036440 con 0x7f4f0c1515e0 2024-03-06T07:32:12.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.806+0000 7f4f0b7fe640 1 -- 172.21.15.83:0/131595133 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f00036810 con 0x7f4f0c1515e0 2024-03-06T07:32:12.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.807+0000 7f4f12a84640 1 -- 172.21.15.83:0/131595133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f0c1515e0 msgr2=0x7f4f0c0cce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:12.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.807+0000 7f4f12a84640 1 --2- 172.21.15.83:0/131595133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f0c1515e0 0x7f4f0c0cce40 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f4f0000bb10 tx=0x7f4f0002e0c0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.807+0000 7f4f12a84640 1 -- 172.21.15.83:0/131595133 shutdown_connections 2024-03-06T07:32:12.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.807+0000 7f4f12a84640 1 --2- 172.21.15.83:0/131595133 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4f0c15b430 0x7f4f0c071830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.807+0000 7f4f12a84640 1 --2- 172.21.15.83:0/131595133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f0c1515e0 0x7f4f0c0cce40 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.807+0000 7f4f12a84640 1 -- 172.21.15.83:0/131595133 >> 172.21.15.83:0/131595133 conn(0x7f4f0c0cf8a0 msgr2=0x7f4f0c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:12.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.807+0000 7f4f12a84640 1 -- 172.21.15.83:0/131595133 shutdown_connections 2024-03-06T07:32:12.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.807+0000 7f4f12a84640 1 -- 172.21.15.83:0/131595133 wait complete. 2024-03-06T07:32:12.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.808+0000 7f4f12a84640 1 Processor -- start 2024-03-06T07:32:12.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.808+0000 7f4f12a84640 1 -- start start 2024-03-06T07:32:12.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.809+0000 7f4f12a84640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4f0c0cd7a0 0x7f4f0c1535c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.809+0000 7f4f12a84640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4f0c1515e0 0x7f4f0c159380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.809+0000 7f4f12a84640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f0c15b430 0x7f4f0c1598c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.809+0000 7f4f12a84640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0c071ef0 con 0x7f4f0c15b430 2024-03-06T07:32:12.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.809+0000 7f4f12a84640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0c071d70 con 0x7f4f0c1515e0 2024-03-06T07:32:12.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.809+0000 7f4f12a84640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0c0cc3a0 con 0x7f4f0c0cd7a0 2024-03-06T07:32:12.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.809+0000 7f4f11021640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f0c15b430 0x7f4f0c1598c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.809+0000 7f4f11021640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f0c15b430 0x7f4f0c1598c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53294/0 (socket says 172.21.15.83:53294) 2024-03-06T07:32:12.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.809+0000 7f4f11021640 1 -- 172.21.15.83:0/2775528517 learned_addr learned my addr 172.21.15.83:0/2775528517 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:12.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.809+0000 7f4f0bfff640 1 --2- 172.21.15.83:0/2775528517 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4f0c1515e0 0x7f4f0c159380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.809+0000 7f4f10820640 1 --2- 172.21.15.83:0/2775528517 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4f0c0cd7a0 0x7f4f0c1535c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.809+0000 7f4f11021640 1 -- 172.21.15.83:0/2775528517 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4f0c0cd7a0 msgr2=0x7f4f0c1535c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:12.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.809+0000 7f4f11021640 1 --2- 172.21.15.83:0/2775528517 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4f0c0cd7a0 0x7f4f0c1535c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.809+0000 7f4f11021640 1 -- 172.21.15.83:0/2775528517 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4f0c1515e0 msgr2=0x7f4f0c159380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:12.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.809+0000 7f4f11021640 1 --2- 172.21.15.83:0/2775528517 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4f0c1515e0 0x7f4f0c159380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.809+0000 7f4f11021640 1 -- 172.21.15.83:0/2775528517 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f0c153cc0 con 0x7f4f0c15b430 2024-03-06T07:32:12.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.809+0000 7f4f0bfff640 1 --2- 172.21.15.83:0/2775528517 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4f0c1515e0 0x7f4f0c159380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:12.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.809+0000 7f4f10820640 1 --2- 172.21.15.83:0/2775528517 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4f0c0cd7a0 0x7f4f0c1535c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:12.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.810+0000 7f4f11021640 1 --2- 172.21.15.83:0/2775528517 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f0c15b430 0x7f4f0c1598c0 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f4f0400c780 tx=0x7f4f0400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:12.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.810+0000 7f4f097fa640 1 -- 172.21.15.83:0/2775528517 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f04004b60 con 0x7f4f0c15b430 2024-03-06T07:32:12.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.810+0000 7f4f097fa640 1 -- 172.21.15.83:0/2775528517 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f04004d00 con 0x7f4f0c15b430 2024-03-06T07:32:12.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.810+0000 7f4f097fa640 1 -- 172.21.15.83:0/2775528517 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f0400aa90 con 0x7f4f0c15b430 2024-03-06T07:32:12.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.810+0000 7f4f12a84640 1 -- 172.21.15.83:0/2775528517 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f0c153fa0 con 0x7f4f0c15b430 2024-03-06T07:32:12.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.810+0000 7f4f12a84640 1 -- 172.21.15.83:0/2775528517 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f0c154560 con 0x7f4f0c15b430 2024-03-06T07:32:12.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.811+0000 7f4f12a84640 1 -- 172.21.15.83:0/2775528517 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f0c068840 con 0x7f4f0c15b430 2024-03-06T07:32:12.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.812+0000 7f4f097fa640 1 -- 172.21.15.83:0/2775528517 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4f0400ac30 con 0x7f4f0c15b430 2024-03-06T07:32:12.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.812+0000 7f4f097fa640 1 --2- 172.21.15.83:0/2775528517 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4ef006b4d0 0x7f4ef006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.812+0000 7f4f097fa640 1 -- 172.21.15.83:0/2775528517 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4f0408a490 con 0x7f4f0c15b430 2024-03-06T07:32:12.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.812+0000 7f4f10820640 1 --2- 172.21.15.83:0/2775528517 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4ef006b4d0 0x7f4ef006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.813+0000 7f4f10820640 1 --2- 172.21.15.83:0/2775528517 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4ef006b4d0 0x7f4ef006d990 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f4f00038000 tx=0x7f4f00008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:12.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:12.817+0000 7f4f097fa640 1 -- 172.21.15.83:0/2775528517 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4f04020070 con 0x7f4f0c15b430 2024-03-06T07:32:13.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:13.035+0000 7f4f12a84640 1 -- 172.21.15.83:0/2775528517 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f0c04f330 con 0x7f4f0c15b430 2024-03-06T07:32:13.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:13.036+0000 7f4f097fa640 1 -- 172.21.15.83:0/2775528517 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f4f04059360 con 0x7f4f0c15b430 2024-03-06T07:32:13.038 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:32:13.038 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:32:13.038 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:32:13.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:13.037+0000 7f4f12a84640 1 -- 172.21.15.83:0/2775528517 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4ef006b4d0 msgr2=0x7f4ef006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:13.037+0000 7f4f12a84640 1 --2- 172.21.15.83:0/2775528517 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4ef006b4d0 0x7f4ef006d990 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f4f00038000 tx=0x7f4f00008040 comp rx=0 tx=0).stop 2024-03-06T07:32:13.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:13.037+0000 7f4f12a84640 1 -- 172.21.15.83:0/2775528517 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f0c15b430 msgr2=0x7f4f0c1598c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:13.037+0000 7f4f12a84640 1 --2- 172.21.15.83:0/2775528517 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f0c15b430 0x7f4f0c1598c0 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f4f0400c780 tx=0x7f4f0400cc40 comp rx=0 tx=0).stop 2024-03-06T07:32:13.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:13.038+0000 7f4f12a84640 1 -- 172.21.15.83:0/2775528517 shutdown_connections 2024-03-06T07:32:13.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:13.038+0000 7f4f12a84640 1 --2- 172.21.15.83:0/2775528517 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4ef006b4d0 0x7f4ef006d990 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:13.038+0000 7f4f12a84640 1 --2- 172.21.15.83:0/2775528517 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f0c15b430 0x7f4f0c1598c0 unknown :-1 s=CLOSED pgs=1942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:13.038+0000 7f4f12a84640 1 --2- 172.21.15.83:0/2775528517 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4f0c1515e0 0x7f4f0c159380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:13.038+0000 7f4f12a84640 1 --2- 172.21.15.83:0/2775528517 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4f0c0cd7a0 0x7f4f0c1535c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:13.038+0000 7f4f12a84640 1 -- 172.21.15.83:0/2775528517 >> 172.21.15.83:0/2775528517 conn(0x7f4f0c0cf8a0 msgr2=0x7f4f0c150080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:13.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:13.038+0000 7f4f12a84640 1 -- 172.21.15.83:0/2775528517 shutdown_connections 2024-03-06T07:32:13.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:13.038+0000 7f4f12a84640 1 -- 172.21.15.83:0/2775528517 wait complete. 2024-03-06T07:32:13.131 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:32:13.252 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:31:26.591282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:32:13.252 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:32:14.252 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:14.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.352+0000 7f0e1759e640 1 Processor -- start 2024-03-06T07:32:14.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.352+0000 7f0e1759e640 1 -- start start 2024-03-06T07:32:14.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.353+0000 7f0e1759e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0e1806f060 0x7f0e1806afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.353+0000 7f0e1759e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f0e1806b4f0 0x7f0e180ada70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.353+0000 7f0e1759e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0e180adfb0 0x7f0e18071850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.353+0000 7f0e1759e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e180ae420 con 0x7f0e180adfb0 2024-03-06T07:32:14.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.353+0000 7f0e1759e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e1815e180 con 0x7f0e1806b4f0 2024-03-06T07:32:14.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.353+0000 7f0e1759e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e180ae680 con 0x7f0e1806f060 2024-03-06T07:32:14.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.353+0000 7f0e16d9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0e180adfb0 0x7f0e18071850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.353+0000 7f0e16d9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0e180adfb0 0x7f0e18071850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53298/0 (socket says 172.21.15.83:53298) 2024-03-06T07:32:14.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.353+0000 7f0e16d9d640 1 -- 172.21.15.83:0/4209253668 learned_addr learned my addr 172.21.15.83:0/4209253668 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:14.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.353+0000 7f0e0dd9b640 1 --2- 172.21.15.83:0/4209253668 >> v2:172.21.15.183:3300/0 conn(0x7f0e1806b4f0 0x7f0e180ada70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.353+0000 7f0e16d9d640 1 -- 172.21.15.83:0/4209253668 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0e1806f060 msgr2=0x7f0e1806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:14.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.353+0000 7f0e1659c640 1 --2- 172.21.15.83:0/4209253668 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0e1806f060 0x7f0e1806afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.353+0000 7f0e16d9d640 1 --2- 172.21.15.83:0/4209253668 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0e1806f060 0x7f0e1806afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.353+0000 7f0e16d9d640 1 -- 172.21.15.83:0/4209253668 >> v2:172.21.15.183:3300/0 conn(0x7f0e1806b4f0 msgr2=0x7f0e180ada70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:14.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.353+0000 7f0e16d9d640 1 --2- 172.21.15.83:0/4209253668 >> v2:172.21.15.183:3300/0 conn(0x7f0e1806b4f0 0x7f0e180ada70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.353+0000 7f0e16d9d640 1 -- 172.21.15.83:0/4209253668 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e1815e440 con 0x7f0e180adfb0 2024-03-06T07:32:14.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.353+0000 7f0e0dd9b640 1 --2- 172.21.15.83:0/4209253668 >> v2:172.21.15.183:3300/0 conn(0x7f0e1806b4f0 0x7f0e180ada70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:14.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.353+0000 7f0e1659c640 1 --2- 172.21.15.83:0/4209253668 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0e1806f060 0x7f0e1806afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:14.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.354+0000 7f0e16d9d640 1 --2- 172.21.15.83:0/4209253668 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0e180adfb0 0x7f0e18071850 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f0e18068cf0 tx=0x7f0e1002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6bf43c412faa17b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:14.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.354+0000 7f0e15d9b640 1 -- 172.21.15.83:0/4209253668 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e1002c9e0 con 0x7f0e180adfb0 2024-03-06T07:32:14.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.354+0000 7f0e15d9b640 1 -- 172.21.15.83:0/4209253668 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e1002cb80 con 0x7f0e180adfb0 2024-03-06T07:32:14.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.354+0000 7f0e15d9b640 1 -- 172.21.15.83:0/4209253668 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e100343f0 con 0x7f0e180adfb0 2024-03-06T07:32:14.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.354+0000 7f0e1759e640 1 -- 172.21.15.83:0/4209253668 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0e180adfb0 msgr2=0x7f0e18071850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.354+0000 7f0e1759e640 1 --2- 172.21.15.83:0/4209253668 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0e180adfb0 0x7f0e18071850 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f0e18068cf0 tx=0x7f0e1002be40 comp rx=0 tx=0).stop 2024-03-06T07:32:14.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.354+0000 7f0e1759e640 1 -- 172.21.15.83:0/4209253668 shutdown_connections 2024-03-06T07:32:14.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.354+0000 7f0e1759e640 1 --2- 172.21.15.83:0/4209253668 >> v2:172.21.15.183:3300/0 conn(0x7f0e1806b4f0 0x7f0e180ada70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.354+0000 7f0e1759e640 1 --2- 172.21.15.83:0/4209253668 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0e180adfb0 0x7f0e18071850 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.354+0000 7f0e1759e640 1 --2- 172.21.15.83:0/4209253668 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0e1806f060 0x7f0e1806afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.354+0000 7f0e1759e640 1 -- 172.21.15.83:0/4209253668 >> 172.21.15.83:0/4209253668 conn(0x7f0e180d0860 msgr2=0x7f0e180d0c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:14.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.354+0000 7f0e1759e640 1 -- 172.21.15.83:0/4209253668 shutdown_connections 2024-03-06T07:32:14.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.354+0000 7f0e1759e640 1 -- 172.21.15.83:0/4209253668 wait complete. 2024-03-06T07:32:14.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.356+0000 7f0e1759e640 1 Processor -- start 2024-03-06T07:32:14.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.356+0000 7f0e1759e640 1 -- start start 2024-03-06T07:32:14.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.357+0000 7f0e1759e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e1806b4f0 0x7f0e180cfd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.357+0000 7f0e1759e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0e1806f060 0x7f0e180cde10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.357+0000 7f0e1759e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0e180adfb0 0x7f0e181f6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.357+0000 7f0e1759e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1815ef20 con 0x7f0e1806f060 2024-03-06T07:32:14.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.357+0000 7f0e1759e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1815ea00 con 0x7f0e1806b4f0 2024-03-06T07:32:14.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.357+0000 7f0e1759e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1815f0a0 con 0x7f0e180adfb0 2024-03-06T07:32:14.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.357+0000 7f0e0dd9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0e1806f060 0x7f0e180cde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.357+0000 7f0e0dd9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0e1806f060 0x7f0e180cde10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53310/0 (socket says 172.21.15.83:53310) 2024-03-06T07:32:14.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.357+0000 7f0e0dd9b640 1 -- 172.21.15.83:0/592472624 learned_addr learned my addr 172.21.15.83:0/592472624 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:14.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.357+0000 7f0e16d9d640 1 --2- 172.21.15.83:0/592472624 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0e180adfb0 0x7f0e181f6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.357+0000 7f0e1659c640 1 --2- 172.21.15.83:0/592472624 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e1806b4f0 0x7f0e180cfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.357+0000 7f0e0dd9b640 1 -- 172.21.15.83:0/592472624 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0e180adfb0 msgr2=0x7f0e181f6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.357+0000 7f0e0dd9b640 1 --2- 172.21.15.83:0/592472624 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0e180adfb0 0x7f0e181f6530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.357+0000 7f0e0dd9b640 1 -- 172.21.15.83:0/592472624 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e1806b4f0 msgr2=0x7f0e180cfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.357+0000 7f0e0dd9b640 1 --2- 172.21.15.83:0/592472624 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e1806b4f0 0x7f0e180cfd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.357+0000 7f0e0dd9b640 1 -- 172.21.15.83:0/592472624 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e180ce580 con 0x7f0e1806f060 2024-03-06T07:32:14.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.357+0000 7f0e1659c640 1 --2- 172.21.15.83:0/592472624 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e1806b4f0 0x7f0e180cfd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:14.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.357+0000 7f0e16d9d640 1 --2- 172.21.15.83:0/592472624 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0e180adfb0 0x7f0e181f6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:14.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.357+0000 7f0e0dd9b640 1 --2- 172.21.15.83:0/592472624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0e1806f060 0x7f0e180cde10 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f0e0000c7f0 tx=0x7f0e0000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:14.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.357+0000 7f0e0f7fe640 1 -- 172.21.15.83:0/592472624 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e00015070 con 0x7f0e1806f060 2024-03-06T07:32:14.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.357+0000 7f0e0f7fe640 1 -- 172.21.15.83:0/592472624 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e000077c0 con 0x7f0e1806f060 2024-03-06T07:32:14.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.357+0000 7f0e1759e640 1 -- 172.21.15.83:0/592472624 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e180ce860 con 0x7f0e1806f060 2024-03-06T07:32:14.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.357+0000 7f0e0f7fe640 1 -- 172.21.15.83:0/592472624 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e00007bb0 con 0x7f0e1806f060 2024-03-06T07:32:14.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.357+0000 7f0e1759e640 1 -- 172.21.15.83:0/592472624 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e180cee20 con 0x7f0e1806f060 2024-03-06T07:32:14.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.358+0000 7f0e1759e640 1 -- 172.21.15.83:0/592472624 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e180af610 con 0x7f0e1806f060 2024-03-06T07:32:14.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.359+0000 7f0e0f7fe640 1 -- 172.21.15.83:0/592472624 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0e00018070 con 0x7f0e1806f060 2024-03-06T07:32:14.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.359+0000 7f0e0f7fe640 1 --2- 172.21.15.83:0/592472624 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0df806b4d0 0x7f0df806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.359+0000 7f0e0f7fe640 1 -- 172.21.15.83:0/592472624 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0e0008a0c0 con 0x7f0e1806f060 2024-03-06T07:32:14.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.359+0000 7f0e1659c640 1 --2- 172.21.15.83:0/592472624 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0df806b4d0 0x7f0df806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.360+0000 7f0e1659c640 1 --2- 172.21.15.83:0/592472624 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0df806b4d0 0x7f0df806d990 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f0e08006fd0 tx=0x7f0e08008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:14.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.364+0000 7f0e0f7fe640 1 -- 172.21.15.83:0/592472624 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0e180af610 con 0x7f0e1806f060 2024-03-06T07:32:14.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.689+0000 7f0e1759e640 1 -- 172.21.15.83:0/592472624 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e180cf260 con 0x7f0e1806f060 2024-03-06T07:32:14.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.689+0000 7f0e0f7fe640 1 -- 172.21.15.83:0/592472624 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f0e00058f10 con 0x7f0e1806f060 2024-03-06T07:32:14.691 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:32:14.691 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:32:14.692 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:32:14.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.691+0000 7f0e1759e640 1 -- 172.21.15.83:0/592472624 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0df806b4d0 msgr2=0x7f0df806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.691+0000 7f0e1759e640 1 --2- 172.21.15.83:0/592472624 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0df806b4d0 0x7f0df806d990 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f0e08006fd0 tx=0x7f0e08008040 comp rx=0 tx=0).stop 2024-03-06T07:32:14.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.691+0000 7f0e1759e640 1 -- 172.21.15.83:0/592472624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0e1806f060 msgr2=0x7f0e180cde10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.691+0000 7f0e1759e640 1 --2- 172.21.15.83:0/592472624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0e1806f060 0x7f0e180cde10 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f0e0000c7f0 tx=0x7f0e0000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.691+0000 7f0e1759e640 1 -- 172.21.15.83:0/592472624 shutdown_connections 2024-03-06T07:32:14.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.691+0000 7f0e1759e640 1 --2- 172.21.15.83:0/592472624 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0df806b4d0 0x7f0df806d990 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.691+0000 7f0e1759e640 1 --2- 172.21.15.83:0/592472624 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0e180adfb0 0x7f0e181f6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.691+0000 7f0e1759e640 1 --2- 172.21.15.83:0/592472624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0e1806f060 0x7f0e180cde10 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.691+0000 7f0e1759e640 1 --2- 172.21.15.83:0/592472624 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e1806b4f0 0x7f0e180cfd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.691+0000 7f0e1759e640 1 -- 172.21.15.83:0/592472624 >> 172.21.15.83:0/592472624 conn(0x7f0e180d0860 msgr2=0x7f0e180b0560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:14.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.691+0000 7f0e1759e640 1 -- 172.21.15.83:0/592472624 shutdown_connections 2024-03-06T07:32:14.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:14.691+0000 7f0e1759e640 1 -- 172.21.15.83:0/592472624 wait complete. 2024-03-06T07:32:14.905 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:31:26.591282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:32:14.905 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:32:15.133 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:32:15.906 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:16.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.011+0000 7f8f80c84640 1 Processor -- start 2024-03-06T07:32:16.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.011+0000 7f8f80c84640 1 -- start start 2024-03-06T07:32:16.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.012+0000 7f8f80c84640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8f7c0d22b0 0x7f8f7c0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.012+0000 7f8f80c84640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f7c0b3430 con 0x7f8f7c15dfe0 2024-03-06T07:32:16.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.012+0000 7f8f80c84640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f7c0b3710 con 0x7f8f7c0b41f0 2024-03-06T07:32:16.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.012+0000 7f8f80c84640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7c0d2c50 con 0x7f8f7c0d22b0 2024-03-06T07:32:16.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.012+0000 7f8f7a59c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f8f7c15dfe0 0x7f8f7c0cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52204/0 (socket says 172.21.15.83:52204) 2024-03-06T07:32:16.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.012+0000 7f8f7a59c640 1 -- 172.21.15.83:0/1783608249 learned_addr learned my addr 172.21.15.83:0/1783608249 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:32:16.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.013+0000 7f8f71d9b640 1 --2- 172.21.15.83:0/1783608249 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8f7c0d22b0 0x7f8f7c0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.013+0000 7f8f79d9b640 1 -- 172.21.15.83:0/1783608249 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 951660838 0 0) 0x7f8f7c0b3430 con 0x7f8f7c15dfe0 2024-03-06T07:32:16.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.013+0000 7f8f79d9b640 1 -- 172.21.15.83:0/1783608249 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f68003520 con 0x7f8f7c15dfe0 2024-03-06T07:32:16.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.013+0000 7f8f79d9b640 1 -- 172.21.15.83:0/1783608249 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3236535639 0 0) 0x7f8f7c0b3710 con 0x7f8f7c0b41f0 2024-03-06T07:32:16.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.013+0000 7f8f79d9b640 1 -- 172.21.15.83:0/1783608249 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f7c0b3430 con 0x7f8f7c0b41f0 2024-03-06T07:32:16.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.013+0000 7f8f79d9b640 1 -- 172.21.15.83:0/1783608249 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1066791744 0 0) 0x7f8f68003520 con 0x7f8f7c15dfe0 2024-03-06T07:32:16.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.013+0000 7f8f71d9b640 1 -- 172.21.15.83:0/1783608249 >> v1:172.21.15.183:6789/0 conn(0x7f8f7c0b41f0 legacy=0x7f8f7c0b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.013+0000 7f8f71d9b640 1 -- 172.21.15.83:0/1783608249 >> v1:172.21.15.83:6789/0 conn(0x7f8f7c15dfe0 legacy=0x7f8f7c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.013+0000 7f8f71d9b640 1 -- 172.21.15.83:0/1783608249 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f7c0b39f0 con 0x7f8f7c0d22b0 2024-03-06T07:32:16.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.014+0000 7f8f71d9b640 1 --2- 172.21.15.83:0/1783608249 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8f7c0d22b0 0x7f8f7c0d2710 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8f640097d0 tx=0x7f8f6402b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=beebd318dd33a141 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:16.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.014+0000 7f8f79d9b640 1 -- 172.21.15.83:0/1783608249 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f64037070 con 0x7f8f7c0d22b0 2024-03-06T07:32:16.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.014+0000 7f8f79d9b640 1 -- 172.21.15.83:0/1783608249 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f64030510 con 0x7f8f7c0d22b0 2024-03-06T07:32:16.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.014+0000 7f8f79d9b640 1 -- 172.21.15.83:0/1783608249 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f6c001670 con 0x7f8f7c0d22b0 2024-03-06T07:32:16.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.014+0000 7f8f80c84640 1 -- 172.21.15.83:0/1783608249 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8f7c0d22b0 msgr2=0x7f8f7c0d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.014+0000 7f8f80c84640 1 --2- 172.21.15.83:0/1783608249 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8f7c0d22b0 0x7f8f7c0d2710 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8f640097d0 tx=0x7f8f6402b800 comp rx=0 tx=0).stop 2024-03-06T07:32:16.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.014+0000 7f8f80c84640 1 -- 172.21.15.83:0/1783608249 shutdown_connections 2024-03-06T07:32:16.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.014+0000 7f8f80c84640 1 --2- 172.21.15.83:0/1783608249 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8f7c0d22b0 0x7f8f7c0d2710 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.014+0000 7f8f80c84640 1 -- 172.21.15.83:0/1783608249 >> 172.21.15.83:0/1783608249 conn(0x7f8f7c0d3cd0 msgr2=0x7f8f7c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:16.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.014+0000 7f8f80c84640 1 -- 172.21.15.83:0/1783608249 shutdown_connections 2024-03-06T07:32:16.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.014+0000 7f8f80c84640 1 -- 172.21.15.83:0/1783608249 wait complete. 2024-03-06T07:32:16.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.016+0000 7f8f80c84640 1 Processor -- start 2024-03-06T07:32:16.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.016+0000 7f8f80c84640 1 -- start start 2024-03-06T07:32:16.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.017+0000 7f8f80c84640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8f7c0b41f0 0x7f8f7c14ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.017+0000 7f8f80c84640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8f7c0d22b0 0x7f8f7c0b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.017+0000 7f8f80c84640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f7c15dfe0 0x7f8f7c1f7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.017+0000 7f8f80c84640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7c0b3d60 con 0x7f8f7c15dfe0 2024-03-06T07:32:16.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.017+0000 7f8f80c84640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7c0b31e0 con 0x7f8f7c0b41f0 2024-03-06T07:32:16.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.017+0000 7f8f80c84640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7c0d1c70 con 0x7f8f7c0d22b0 2024-03-06T07:32:16.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.017+0000 7f8f7ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f7c15dfe0 0x7f8f7c1f7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.017+0000 7f8f7ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f7c15dfe0 0x7f8f7c1f7410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53324/0 (socket says 172.21.15.83:53324) 2024-03-06T07:32:16.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.017+0000 7f8f7ad9d640 1 -- 172.21.15.83:0/834400670 learned_addr learned my addr 172.21.15.83:0/834400670 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:16.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.017+0000 7f8f71d9b640 1 --2- 172.21.15.83:0/834400670 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8f7c0d22b0 0x7f8f7c0b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.017+0000 7f8f7a59c640 1 --2- 172.21.15.83:0/834400670 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8f7c0b41f0 0x7f8f7c14ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.017+0000 7f8f7ad9d640 1 -- 172.21.15.83:0/834400670 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8f7c0d22b0 msgr2=0x7f8f7c0b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.017+0000 7f8f7ad9d640 1 --2- 172.21.15.83:0/834400670 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8f7c0d22b0 0x7f8f7c0b68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.017+0000 7f8f7ad9d640 1 -- 172.21.15.83:0/834400670 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8f7c0b41f0 msgr2=0x7f8f7c14ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.017+0000 7f8f7ad9d640 1 --2- 172.21.15.83:0/834400670 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8f7c0b41f0 0x7f8f7c14ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.017+0000 7f8f7ad9d640 1 -- 172.21.15.83:0/834400670 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f7c0b9430 con 0x7f8f7c15dfe0 2024-03-06T07:32:16.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.017+0000 7f8f7ad9d640 1 --2- 172.21.15.83:0/834400670 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f7c15dfe0 0x7f8f7c1f7410 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f8f7400c790 tx=0x7f8f7400cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:16.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.017+0000 7f8f737fe640 1 -- 172.21.15.83:0/834400670 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f74003b70 con 0x7f8f7c15dfe0 2024-03-06T07:32:16.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.018+0000 7f8f80c84640 1 -- 172.21.15.83:0/834400670 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f7c0b9710 con 0x7f8f7c15dfe0 2024-03-06T07:32:16.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.018+0000 7f8f737fe640 1 -- 172.21.15.83:0/834400670 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f74003d10 con 0x7f8f7c15dfe0 2024-03-06T07:32:16.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.018+0000 7f8f737fe640 1 -- 172.21.15.83:0/834400670 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f7400aa50 con 0x7f8f7c15dfe0 2024-03-06T07:32:16.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.018+0000 7f8f80c84640 1 -- 172.21.15.83:0/834400670 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f7c0cc040 con 0x7f8f7c15dfe0 2024-03-06T07:32:16.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.019+0000 7f8f80c84640 1 -- 172.21.15.83:0/834400670 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f7c0d3740 con 0x7f8f7c15dfe0 2024-03-06T07:32:16.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.019+0000 7f8f737fe640 1 -- 172.21.15.83:0/834400670 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8f7400abf0 con 0x7f8f7c15dfe0 2024-03-06T07:32:16.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.019+0000 7f8f737fe640 1 --2- 172.21.15.83:0/834400670 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8f5c06b4d0 0x7f8f5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.019+0000 7f8f7a59c640 1 --2- 172.21.15.83:0/834400670 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8f5c06b4d0 0x7f8f5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.020+0000 7f8f737fe640 1 -- 172.21.15.83:0/834400670 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8f7408a120 con 0x7f8f7c15dfe0 2024-03-06T07:32:16.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.020+0000 7f8f7a59c640 1 --2- 172.21.15.83:0/834400670 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8f5c06b4d0 0x7f8f5c06d990 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c0011e0 tx=0x7f8f6c001690 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:16.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.024+0000 7f8f737fe640 1 -- 172.21.15.83:0/834400670 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8f7c0d3740 con 0x7f8f7c15dfe0 2024-03-06T07:32:16.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.235+0000 7f8f80c84640 1 -- 172.21.15.83:0/834400670 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f7c04f330 con 0x7f8f7c15dfe0 2024-03-06T07:32:16.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.235+0000 7f8f737fe640 1 -- 172.21.15.83:0/834400670 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f8f74058890 con 0x7f8f7c15dfe0 2024-03-06T07:32:16.237 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:32:16.237 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:32:16.238 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:32:16.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.237+0000 7f8f80c84640 1 -- 172.21.15.83:0/834400670 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8f5c06b4d0 msgr2=0x7f8f5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.237+0000 7f8f80c84640 1 --2- 172.21.15.83:0/834400670 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8f5c06b4d0 0x7f8f5c06d990 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c0011e0 tx=0x7f8f6c001690 comp rx=0 tx=0).stop 2024-03-06T07:32:16.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.237+0000 7f8f80c84640 1 -- 172.21.15.83:0/834400670 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f7c15dfe0 msgr2=0x7f8f7c1f7410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.237+0000 7f8f80c84640 1 --2- 172.21.15.83:0/834400670 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f7c15dfe0 0x7f8f7c1f7410 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f8f7400c790 tx=0x7f8f7400cc50 comp rx=0 tx=0).stop 2024-03-06T07:32:16.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.237+0000 7f8f80c84640 1 -- 172.21.15.83:0/834400670 shutdown_connections 2024-03-06T07:32:16.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.237+0000 7f8f80c84640 1 --2- 172.21.15.83:0/834400670 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8f5c06b4d0 0x7f8f5c06d990 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.237+0000 7f8f80c84640 1 --2- 172.21.15.83:0/834400670 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f7c15dfe0 0x7f8f7c1f7410 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.237+0000 7f8f80c84640 1 --2- 172.21.15.83:0/834400670 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8f7c0d22b0 0x7f8f7c0b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.237+0000 7f8f80c84640 1 --2- 172.21.15.83:0/834400670 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8f7c0b41f0 0x7f8f7c14ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.237+0000 7f8f80c84640 1 -- 172.21.15.83:0/834400670 >> 172.21.15.83:0/834400670 conn(0x7f8f7c0d3cd0 msgr2=0x7f8f7c0d0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:16.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.237+0000 7f8f80c84640 1 -- 172.21.15.83:0/834400670 shutdown_connections 2024-03-06T07:32:16.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:16.237+0000 7f8f80c84640 1 -- 172.21.15.83:0/834400670 wait complete. 2024-03-06T07:32:16.451 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:31:26.591282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:32:16.451 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:32:17.136 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:32:17.452 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:17.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.551+0000 7fb69ad56640 1 Processor -- start 2024-03-06T07:32:17.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.551+0000 7fb69ad56640 1 -- start start 2024-03-06T07:32:17.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.552+0000 7fb69ad56640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fb69415dfe0 0x7fb6940cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.552+0000 7fb69ad56640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb6940cf830 0x7fb6940d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.552+0000 7fb69ad56640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6940b3430 con 0x7fb6940d27f0 2024-03-06T07:32:17.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.552+0000 7fb69ad56640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6940b31e0 con 0x7fb6940cf830 2024-03-06T07:32:17.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.552+0000 7fb69ad56640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6940b3d60 con 0x7fb69415dfe0 2024-03-06T07:32:17.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.552+0000 7fb693fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb6940cf830 0x7fb6940d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.552+0000 7fb693fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb6940cf830 0x7fb6940d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53314/0 (socket says 172.21.15.83:53314) 2024-03-06T07:32:17.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.552+0000 7fb693fff640 1 -- 172.21.15.83:0/742906223 learned_addr learned my addr 172.21.15.83:0/742906223 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:17.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.552+0000 7fb698af2640 1 --2- 172.21.15.83:0/742906223 >> v2:172.21.15.183:3300/0 conn(0x7fb69415dfe0 0x7fb6940cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.552+0000 7fb693fff640 1 -- 172.21.15.83:0/742906223 >> v2:172.21.15.183:3300/0 conn(0x7fb69415dfe0 msgr2=0x7fb6940cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:17.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.552+0000 7fb693fff640 1 --2- 172.21.15.83:0/742906223 >> v2:172.21.15.183:3300/0 conn(0x7fb69415dfe0 0x7fb6940cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.552+0000 7fb693fff640 1 -- 172.21.15.83:0/742906223 >> v1:172.21.15.183:6789/0 conn(0x7fb6940d27f0 legacy=0x7fb6940d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:17.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.552+0000 7fb693fff640 1 -- 172.21.15.83:0/742906223 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6940b3710 con 0x7fb6940cf830 2024-03-06T07:32:17.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.553+0000 7fb693fff640 1 --2- 172.21.15.83:0/742906223 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb6940cf830 0x7fb6940d22b0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7fb688009870 tx=0x7fb68802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4beb11e7bc1df7e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:17.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.553+0000 7fb6937fe640 1 -- 172.21.15.83:0/742906223 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb688035020 con 0x7fb6940cf830 2024-03-06T07:32:17.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.553+0000 7fb6937fe640 1 -- 172.21.15.83:0/742906223 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb688031870 con 0x7fb6940cf830 2024-03-06T07:32:17.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.553+0000 7fb6937fe640 1 -- 172.21.15.83:0/742906223 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb688031c40 con 0x7fb6940cf830 2024-03-06T07:32:17.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.554+0000 7fb69ad56640 1 -- 172.21.15.83:0/742906223 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb6940cf830 msgr2=0x7fb6940d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.554+0000 7fb69ad56640 1 --2- 172.21.15.83:0/742906223 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb6940cf830 0x7fb6940d22b0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7fb688009870 tx=0x7fb68802b910 comp rx=0 tx=0).stop 2024-03-06T07:32:17.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.554+0000 7fb69ad56640 1 -- 172.21.15.83:0/742906223 shutdown_connections 2024-03-06T07:32:17.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.554+0000 7fb69ad56640 1 --2- 172.21.15.83:0/742906223 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb6940cf830 0x7fb6940d22b0 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.554+0000 7fb69ad56640 1 --2- 172.21.15.83:0/742906223 >> v2:172.21.15.183:3300/0 conn(0x7fb69415dfe0 0x7fb6940cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.554+0000 7fb69ad56640 1 -- 172.21.15.83:0/742906223 >> 172.21.15.83:0/742906223 conn(0x7fb6940d3cd0 msgr2=0x7fb6940d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:17.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.554+0000 7fb69ad56640 1 -- 172.21.15.83:0/742906223 shutdown_connections 2024-03-06T07:32:17.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.554+0000 7fb69ad56640 1 -- 172.21.15.83:0/742906223 wait complete. 2024-03-06T07:32:17.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.555+0000 7fb69ad56640 1 Processor -- start 2024-03-06T07:32:17.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.555+0000 7fb69ad56640 1 -- start start 2024-03-06T07:32:17.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.556+0000 7fb69ad56640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb6940cf830 0x7fb694161210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.556+0000 7fb69ad56640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb6940d27f0 0x7fb6940cc800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.556+0000 7fb69ad56640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb69415dfe0 0x7fb6940b9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.556+0000 7fb69ad56640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6940d1df0 con 0x7fb6940d27f0 2024-03-06T07:32:17.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.556+0000 7fb69ad56640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6940d1c70 con 0x7fb69415dfe0 2024-03-06T07:32:17.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.556+0000 7fb69ad56640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb69415e3c0 con 0x7fb6940cf830 2024-03-06T07:32:17.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.556+0000 7fb693fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb6940d27f0 0x7fb6940cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.556+0000 7fb693fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb6940d27f0 0x7fb6940cc800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53316/0 (socket says 172.21.15.83:53316) 2024-03-06T07:32:17.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.556+0000 7fb693fff640 1 -- 172.21.15.83:0/2652152791 learned_addr learned my addr 172.21.15.83:0/2652152791 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:17.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.556+0000 7fb698af2640 1 --2- 172.21.15.83:0/2652152791 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb6940cf830 0x7fb694161210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.556+0000 7fb6992f3640 1 --2- 172.21.15.83:0/2652152791 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb69415dfe0 0x7fb6940b9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.556+0000 7fb693fff640 1 -- 172.21.15.83:0/2652152791 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb6940cf830 msgr2=0x7fb694161210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.556+0000 7fb693fff640 1 --2- 172.21.15.83:0/2652152791 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb6940cf830 0x7fb694161210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.556+0000 7fb693fff640 1 -- 172.21.15.83:0/2652152791 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb69415dfe0 msgr2=0x7fb6940b9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.556+0000 7fb693fff640 1 --2- 172.21.15.83:0/2652152791 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb69415dfe0 0x7fb6940b9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.557+0000 7fb693fff640 1 -- 172.21.15.83:0/2652152791 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6940ccf50 con 0x7fb6940d27f0 2024-03-06T07:32:17.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.557+0000 7fb693fff640 1 --2- 172.21.15.83:0/2652152791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb6940d27f0 0x7fb6940cc800 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7fb688000c00 tx=0x7fb6880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:17.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.557+0000 7fb6917fa640 1 -- 172.21.15.83:0/2652152791 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb688035040 con 0x7fb6940d27f0 2024-03-06T07:32:17.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.557+0000 7fb6917fa640 1 -- 172.21.15.83:0/2652152791 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb688030d00 con 0x7fb6940d27f0 2024-03-06T07:32:17.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.557+0000 7fb69ad56640 1 -- 172.21.15.83:0/2652152791 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6940cd230 con 0x7fb6940d27f0 2024-03-06T07:32:17.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.557+0000 7fb6917fa640 1 -- 172.21.15.83:0/2652152791 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6880456c0 con 0x7fb6940d27f0 2024-03-06T07:32:17.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.557+0000 7fb69ad56640 1 -- 172.21.15.83:0/2652152791 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6940b9690 con 0x7fb6940d27f0 2024-03-06T07:32:17.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.558+0000 7fb69ad56640 1 -- 172.21.15.83:0/2652152791 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6940d2230 con 0x7fb6940d27f0 2024-03-06T07:32:17.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.558+0000 7fb6917fa640 1 -- 172.21.15.83:0/2652152791 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb688038020 con 0x7fb6940d27f0 2024-03-06T07:32:17.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.559+0000 7fb6917fa640 1 --2- 172.21.15.83:0/2652152791 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb68006b4d0 0x7fb68006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.559+0000 7fb6917fa640 1 -- 172.21.15.83:0/2652152791 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb6880ac3b0 con 0x7fb6940d27f0 2024-03-06T07:32:17.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.559+0000 7fb698af2640 1 --2- 172.21.15.83:0/2652152791 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb68006b4d0 0x7fb68006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.559+0000 7fb698af2640 1 --2- 172.21.15.83:0/2652152791 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb68006b4d0 0x7fb68006d990 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fb6940d3250 tx=0x7fb68c0039b0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:17.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.563+0000 7fb6917fa640 1 -- 172.21.15.83:0/2652152791 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb68807b280 con 0x7fb6940d27f0 2024-03-06T07:32:17.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.772+0000 7fb69ad56640 1 -- 172.21.15.83:0/2652152791 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb69404f330 con 0x7fb6940d27f0 2024-03-06T07:32:17.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.772+0000 7fb6917fa640 1 -- 172.21.15.83:0/2652152791 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7fb68807ab80 con 0x7fb6940d27f0 2024-03-06T07:32:17.774 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:32:17.774 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:32:17.775 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:32:17.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.775+0000 7fb69ad56640 1 -- 172.21.15.83:0/2652152791 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb68006b4d0 msgr2=0x7fb68006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.775+0000 7fb69ad56640 1 --2- 172.21.15.83:0/2652152791 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb68006b4d0 0x7fb68006d990 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fb6940d3250 tx=0x7fb68c0039b0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.775+0000 7fb69ad56640 1 -- 172.21.15.83:0/2652152791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb6940d27f0 msgr2=0x7fb6940cc800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.775+0000 7fb69ad56640 1 --2- 172.21.15.83:0/2652152791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb6940d27f0 0x7fb6940cc800 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7fb688000c00 tx=0x7fb6880046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.775+0000 7fb69ad56640 1 -- 172.21.15.83:0/2652152791 shutdown_connections 2024-03-06T07:32:17.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.775+0000 7fb69ad56640 1 --2- 172.21.15.83:0/2652152791 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb68006b4d0 0x7fb68006d990 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.775+0000 7fb69ad56640 1 --2- 172.21.15.83:0/2652152791 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb69415dfe0 0x7fb6940b9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.775+0000 7fb69ad56640 1 --2- 172.21.15.83:0/2652152791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb6940d27f0 0x7fb6940cc800 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.775+0000 7fb69ad56640 1 --2- 172.21.15.83:0/2652152791 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb6940cf830 0x7fb694161210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.775+0000 7fb69ad56640 1 -- 172.21.15.83:0/2652152791 >> 172.21.15.83:0/2652152791 conn(0x7fb6940d3cd0 msgr2=0x7fb6940d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:17.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.775+0000 7fb69ad56640 1 -- 172.21.15.83:0/2652152791 shutdown_connections 2024-03-06T07:32:17.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:17.775+0000 7fb69ad56640 1 -- 172.21.15.83:0/2652152791 wait complete. 2024-03-06T07:32:17.992 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:31:26.591282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:32:17.992 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:32:18.993 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:19.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.094+0000 7fbf10eb1640 1 Processor -- start 2024-03-06T07:32:19.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.094+0000 7fbf10eb1640 1 -- start start 2024-03-06T07:32:19.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.095+0000 7fbf10eb1640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbf0c0b40c0 0x7fbf0c0d06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.095+0000 7fbf10eb1640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf0c0d0020 con 0x7fbf0c0b22f0 2024-03-06T07:32:19.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.095+0000 7fbf10eb1640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf0c0b0d20 con 0x7fbf0c15c320 2024-03-06T07:32:19.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.095+0000 7fbf10eb1640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf0c0b1000 con 0x7fbf0c0b40c0 2024-03-06T07:32:19.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.095+0000 7fbf03fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbf0c0b40c0 0x7fbf0c0d06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.095+0000 7fbf0ad9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fbf0c0b22f0 0x7fbf0c0cc620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:35078/0 (socket says 172.21.15.83:35078) 2024-03-06T07:32:19.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.095+0000 7fbf0ad9d640 1 -- 172.21.15.83:0/3385128435 learned_addr learned my addr 172.21.15.83:0/3385128435 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:32:19.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.096+0000 7fbf03fff640 1 -- 172.21.15.83:0/3385128435 >> v1:172.21.15.183:6789/0 conn(0x7fbf0c15c320 legacy=0x7fbf0c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.096+0000 7fbf03fff640 1 -- 172.21.15.83:0/3385128435 >> v1:172.21.15.83:6789/0 conn(0x7fbf0c0b22f0 legacy=0x7fbf0c0cc620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.096+0000 7fbf03fff640 1 -- 172.21.15.83:0/3385128435 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf0c0cfa60 con 0x7fbf0c0b40c0 2024-03-06T07:32:19.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.096+0000 7fbf03fff640 1 --2- 172.21.15.83:0/3385128435 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf0c0b40c0 0x7fbf0c0d06b0 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7fbef40097a0 tx=0x7fbef402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66591b77aed1c845 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:19.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.096+0000 7fbf09d9b640 1 -- 172.21.15.83:0/3385128435 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef40315a0 con 0x7fbf0c0b40c0 2024-03-06T07:32:19.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.097+0000 7fbf09d9b640 1 -- 172.21.15.83:0/3385128435 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbef4037070 con 0x7fbf0c0b40c0 2024-03-06T07:32:19.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.097+0000 7fbf09d9b640 1 -- 172.21.15.83:0/3385128435 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef4031960 con 0x7fbf0c0b40c0 2024-03-06T07:32:19.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.097+0000 7fbf10eb1640 1 -- 172.21.15.83:0/3385128435 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf0c0b40c0 msgr2=0x7fbf0c0d06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.097+0000 7fbf10eb1640 1 --2- 172.21.15.83:0/3385128435 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf0c0b40c0 0x7fbf0c0d06b0 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7fbef40097a0 tx=0x7fbef402b840 comp rx=0 tx=0).stop 2024-03-06T07:32:19.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.097+0000 7fbf10eb1640 1 -- 172.21.15.83:0/3385128435 shutdown_connections 2024-03-06T07:32:19.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.097+0000 7fbf10eb1640 1 --2- 172.21.15.83:0/3385128435 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf0c0b40c0 0x7fbf0c0d06b0 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.097+0000 7fbf10eb1640 1 -- 172.21.15.83:0/3385128435 >> 172.21.15.83:0/3385128435 conn(0x7fbf0c0b35a0 msgr2=0x7fbf0c0b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:19.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.097+0000 7fbf10eb1640 1 -- 172.21.15.83:0/3385128435 shutdown_connections 2024-03-06T07:32:19.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.097+0000 7fbf10eb1640 1 -- 172.21.15.83:0/3385128435 wait complete. 2024-03-06T07:32:19.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.098+0000 7fbf10eb1640 1 Processor -- start 2024-03-06T07:32:19.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.098+0000 7fbf10eb1640 1 -- start start 2024-03-06T07:32:19.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.099+0000 7fbf10eb1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf0c0b22f0 0x7fbf0c0c74d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.099+0000 7fbf10eb1640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbf0c0b40c0 0x7fbf0c14d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.099+0000 7fbf10eb1640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbf0c15c320 0x7fbf0c14dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.099+0000 7fbf10eb1640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0c0b2920 con 0x7fbf0c0b22f0 2024-03-06T07:32:19.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.099+0000 7fbf10eb1640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0c0b27a0 con 0x7fbf0c0b40c0 2024-03-06T07:32:19.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.099+0000 7fbf10eb1640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0c0cd880 con 0x7fbf0c15c320 2024-03-06T07:32:19.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.099+0000 7fbf0a59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf0c0b22f0 0x7fbf0c0c74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.099+0000 7fbf0a59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf0c0b22f0 0x7fbf0c0c74d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53338/0 (socket says 172.21.15.83:53338) 2024-03-06T07:32:19.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.099+0000 7fbf0a59c640 1 -- 172.21.15.83:0/3195638314 learned_addr learned my addr 172.21.15.83:0/3195638314 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:19.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.099+0000 7fbf03fff640 1 --2- 172.21.15.83:0/3195638314 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbf0c0b40c0 0x7fbf0c14d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.099+0000 7fbf0ad9d640 1 --2- 172.21.15.83:0/3195638314 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbf0c15c320 0x7fbf0c14dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.100+0000 7fbf0a59c640 1 -- 172.21.15.83:0/3195638314 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbf0c15c320 msgr2=0x7fbf0c14dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.100+0000 7fbf0a59c640 1 --2- 172.21.15.83:0/3195638314 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbf0c15c320 0x7fbf0c14dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.100+0000 7fbf0a59c640 1 -- 172.21.15.83:0/3195638314 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbf0c0b40c0 msgr2=0x7fbf0c14d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.100+0000 7fbf0a59c640 1 --2- 172.21.15.83:0/3195638314 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbf0c0b40c0 0x7fbf0c14d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.100+0000 7fbf0a59c640 1 -- 172.21.15.83:0/3195638314 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf0c14e3e0 con 0x7fbf0c0b22f0 2024-03-06T07:32:19.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.100+0000 7fbf03fff640 1 --2- 172.21.15.83:0/3195638314 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbf0c0b40c0 0x7fbf0c14d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:19.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.100+0000 7fbf0ad9d640 1 --2- 172.21.15.83:0/3195638314 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbf0c15c320 0x7fbf0c14dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:19.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.100+0000 7fbf0a59c640 1 --2- 172.21.15.83:0/3195638314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf0c0b22f0 0x7fbf0c0c74d0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7fbefc00c400 tx=0x7fbefc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:19.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.100+0000 7fbf02ffd640 1 -- 172.21.15.83:0/3195638314 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbefc017070 con 0x7fbf0c0b22f0 2024-03-06T07:32:19.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.100+0000 7fbf02ffd640 1 -- 172.21.15.83:0/3195638314 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbefc0052d0 con 0x7fbf0c0b22f0 2024-03-06T07:32:19.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.100+0000 7fbf10eb1640 1 -- 172.21.15.83:0/3195638314 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf0c0b5860 con 0x7fbf0c0b22f0 2024-03-06T07:32:19.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.100+0000 7fbf02ffd640 1 -- 172.21.15.83:0/3195638314 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbefc003830 con 0x7fbf0c0b22f0 2024-03-06T07:32:19.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.100+0000 7fbf10eb1640 1 -- 172.21.15.83:0/3195638314 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf0c0b5e20 con 0x7fbf0c0b22f0 2024-03-06T07:32:19.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.101+0000 7fbf10eb1640 1 -- 172.21.15.83:0/3195638314 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf0c0b0980 con 0x7fbf0c0b22f0 2024-03-06T07:32:19.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.101+0000 7fbf02ffd640 1 -- 172.21.15.83:0/3195638314 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbefc01a070 con 0x7fbf0c0b22f0 2024-03-06T07:32:19.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.102+0000 7fbf02ffd640 1 --2- 172.21.15.83:0/3195638314 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbeec06b4d0 0x7fbeec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.102+0000 7fbf02ffd640 1 -- 172.21.15.83:0/3195638314 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fbefc0895a0 con 0x7fbf0c0b22f0 2024-03-06T07:32:19.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.102+0000 7fbf03fff640 1 --2- 172.21.15.83:0/3195638314 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbeec06b4d0 0x7fbeec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.102+0000 7fbf03fff640 1 --2- 172.21.15.83:0/3195638314 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbeec06b4d0 0x7fbeec06d990 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fbef4036000 tx=0x7fbef4008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:19.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.106+0000 7fbf02ffd640 1 -- 172.21.15.83:0/3195638314 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbf0c0b0980 con 0x7fbf0c0b22f0 2024-03-06T07:32:19.137 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:32:19.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.313+0000 7fbf10eb1640 1 -- 172.21.15.83:0/3195638314 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf0c04f330 con 0x7fbf0c0b22f0 2024-03-06T07:32:19.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.313+0000 7fbf02ffd640 1 -- 172.21.15.83:0/3195638314 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7fbefc0583e0 con 0x7fbf0c0b22f0 2024-03-06T07:32:19.316 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:32:19.316 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:32:19.316 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:32:19.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.315+0000 7fbf10eb1640 1 -- 172.21.15.83:0/3195638314 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbeec06b4d0 msgr2=0x7fbeec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.315+0000 7fbf10eb1640 1 --2- 172.21.15.83:0/3195638314 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbeec06b4d0 0x7fbeec06d990 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fbef4036000 tx=0x7fbef4008030 comp rx=0 tx=0).stop 2024-03-06T07:32:19.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.315+0000 7fbf10eb1640 1 -- 172.21.15.83:0/3195638314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf0c0b22f0 msgr2=0x7fbf0c0c74d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.315+0000 7fbf10eb1640 1 --2- 172.21.15.83:0/3195638314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf0c0b22f0 0x7fbf0c0c74d0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7fbefc00c400 tx=0x7fbefc00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.315+0000 7fbf10eb1640 1 -- 172.21.15.83:0/3195638314 shutdown_connections 2024-03-06T07:32:19.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.315+0000 7fbf10eb1640 1 --2- 172.21.15.83:0/3195638314 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbeec06b4d0 0x7fbeec06d990 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.315+0000 7fbf10eb1640 1 --2- 172.21.15.83:0/3195638314 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbf0c15c320 0x7fbf0c14dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.315+0000 7fbf10eb1640 1 --2- 172.21.15.83:0/3195638314 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbf0c0b40c0 0x7fbf0c14d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.315+0000 7fbf10eb1640 1 --2- 172.21.15.83:0/3195638314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf0c0b22f0 0x7fbf0c0c74d0 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.315+0000 7fbf10eb1640 1 -- 172.21.15.83:0/3195638314 >> 172.21.15.83:0/3195638314 conn(0x7fbf0c0b35a0 msgr2=0x7fbf0c06ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:19.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.316+0000 7fbf10eb1640 1 -- 172.21.15.83:0/3195638314 shutdown_connections 2024-03-06T07:32:19.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:19.316+0000 7fbf10eb1640 1 -- 172.21.15.83:0/3195638314 wait complete. 2024-03-06T07:32:19.529 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:31:26.591282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:32:19.529 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:32:20.531 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:20.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.630+0000 7f97d518a640 1 Processor -- start 2024-03-06T07:32:20.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.630+0000 7f97d518a640 1 -- start start 2024-03-06T07:32:20.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.631+0000 7f97d518a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f97d015fa10 0x7f97d00ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.631+0000 7f97d518a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97d00d16e0 0x7f97d00d1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.631+0000 7f97d518a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97d00cebe0 con 0x7f97d015bdd0 2024-03-06T07:32:20.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.631+0000 7f97d518a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97d00ceec0 con 0x7f97d015fa10 2024-03-06T07:32:20.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.631+0000 7f97d518a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97d00cf1a0 con 0x7f97d00d16e0 2024-03-06T07:32:20.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.631+0000 7f97ce59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f97d015fa10 0x7f97d00ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.631+0000 7f97ce59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f97d015fa10 0x7f97d00ce270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53352/0 (socket says 172.21.15.83:53352) 2024-03-06T07:32:20.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.631+0000 7f97ce59c640 1 -- 172.21.15.83:0/909618986 learned_addr learned my addr 172.21.15.83:0/909618986 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:20.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.631+0000 7f97cf59e640 1 --2- 172.21.15.83:0/909618986 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97d00d16e0 0x7f97d00d1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.631+0000 7f97ce59c640 1 -- 172.21.15.83:0/909618986 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97d00d16e0 msgr2=0x7f97d00d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:20.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.631+0000 7f97ce59c640 1 --2- 172.21.15.83:0/909618986 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97d00d16e0 0x7f97d00d1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.631+0000 7f97ce59c640 1 -- 172.21.15.83:0/909618986 >> v1:172.21.15.183:6789/0 conn(0x7f97d015bdd0 legacy=0x7f97d00c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:20.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.631+0000 7f97ce59c640 1 -- 172.21.15.83:0/909618986 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97d00cc8e0 con 0x7f97d015fa10 2024-03-06T07:32:20.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.632+0000 7f97ce59c640 1 --2- 172.21.15.83:0/909618986 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97d015fa10 0x7f97d00ce270 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f97c4009870 tx=0x7f97c402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5790eb11782f47fa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.632+0000 7f97cdd9b640 1 -- 172.21.15.83:0/909618986 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c40317c0 con 0x7f97d015fa10 2024-03-06T07:32:20.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.632+0000 7f97cdd9b640 1 -- 172.21.15.83:0/909618986 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97c4037070 con 0x7f97d015fa10 2024-03-06T07:32:20.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.632+0000 7f97cdd9b640 1 -- 172.21.15.83:0/909618986 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c4031b80 con 0x7f97d015fa10 2024-03-06T07:32:20.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.633+0000 7f97d518a640 1 -- 172.21.15.83:0/909618986 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97d015fa10 msgr2=0x7f97d00ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.633+0000 7f97d518a640 1 --2- 172.21.15.83:0/909618986 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97d015fa10 0x7f97d00ce270 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f97c4009870 tx=0x7f97c402b860 comp rx=0 tx=0).stop 2024-03-06T07:32:20.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.633+0000 7f97d518a640 1 -- 172.21.15.83:0/909618986 shutdown_connections 2024-03-06T07:32:20.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.633+0000 7f97d518a640 1 --2- 172.21.15.83:0/909618986 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97d00d16e0 0x7f97d00d1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.633+0000 7f97d518a640 1 --2- 172.21.15.83:0/909618986 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97d015fa10 0x7f97d00ce270 unknown :-1 s=CLOSED pgs=1952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.633+0000 7f97d518a640 1 -- 172.21.15.83:0/909618986 >> 172.21.15.83:0/909618986 conn(0x7f97d009a9f0 msgr2=0x7f97d009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:20.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.633+0000 7f97d518a640 1 -- 172.21.15.83:0/909618986 shutdown_connections 2024-03-06T07:32:20.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.633+0000 7f97d518a640 1 -- 172.21.15.83:0/909618986 wait complete. 2024-03-06T07:32:20.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.634+0000 7f97d518a640 1 Processor -- start 2024-03-06T07:32:20.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.634+0000 7f97d518a640 1 -- start start 2024-03-06T07:32:20.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.635+0000 7f97d518a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97d00d16e0 0x7f97d01f4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.635+0000 7f97d518a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97d015bdd0 0x7f97d00b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.635+0000 7f97d518a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97d015fa10 0x7f97d00c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.635+0000 7f97d518a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d015c1b0 con 0x7f97d015bdd0 2024-03-06T07:32:20.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.635+0000 7f97d518a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d00cc5d0 con 0x7f97d015fa10 2024-03-06T07:32:20.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.635+0000 7f97d518a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97d00ce7b0 con 0x7f97d00d16e0 2024-03-06T07:32:20.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.635+0000 7f97ce59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97d015bdd0 0x7f97d00b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.635+0000 7f97ce59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97d015bdd0 0x7f97d00b46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53366/0 (socket says 172.21.15.83:53366) 2024-03-06T07:32:20.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.635+0000 7f97ce59c640 1 -- 172.21.15.83:0/3655043293 learned_addr learned my addr 172.21.15.83:0/3655043293 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:20.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.635+0000 7f97cf59e640 1 --2- 172.21.15.83:0/3655043293 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97d015fa10 0x7f97d00c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.635+0000 7f97ced9d640 1 --2- 172.21.15.83:0/3655043293 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97d00d16e0 0x7f97d01f4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.635+0000 7f97ce59c640 1 -- 172.21.15.83:0/3655043293 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97d00d16e0 msgr2=0x7f97d01f4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.635+0000 7f97ce59c640 1 --2- 172.21.15.83:0/3655043293 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97d00d16e0 0x7f97d01f4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.635+0000 7f97ce59c640 1 -- 172.21.15.83:0/3655043293 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97d015fa10 msgr2=0x7f97d00c8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.635+0000 7f97ce59c640 1 --2- 172.21.15.83:0/3655043293 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97d015fa10 0x7f97d00c8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.635+0000 7f97ce59c640 1 -- 172.21.15.83:0/3655043293 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97d0145290 con 0x7f97d015bdd0 2024-03-06T07:32:20.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.635+0000 7f97ce59c640 1 --2- 172.21.15.83:0/3655043293 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97d015bdd0 0x7f97d00b46d0 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f97c4000c00 tx=0x7f97c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.636+0000 7f97b77fe640 1 -- 172.21.15.83:0/3655043293 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c4032650 con 0x7f97d015bdd0 2024-03-06T07:32:20.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.636+0000 7f97b77fe640 1 -- 172.21.15.83:0/3655043293 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97c40327f0 con 0x7f97d015bdd0 2024-03-06T07:32:20.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.636+0000 7f97b77fe640 1 -- 172.21.15.83:0/3655043293 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c4032be0 con 0x7f97d015bdd0 2024-03-06T07:32:20.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.636+0000 7f97d518a640 1 -- 172.21.15.83:0/3655043293 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97d0145570 con 0x7f97d015bdd0 2024-03-06T07:32:20.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.636+0000 7f97d518a640 1 -- 172.21.15.83:0/3655043293 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97d0145b30 con 0x7f97d015bdd0 2024-03-06T07:32:20.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.637+0000 7f97d518a640 1 -- 172.21.15.83:0/3655043293 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97d00cdc00 con 0x7f97d015bdd0 2024-03-06T07:32:20.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.637+0000 7f97b77fe640 1 -- 172.21.15.83:0/3655043293 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f97c404c060 con 0x7f97d015bdd0 2024-03-06T07:32:20.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.638+0000 7f97b77fe640 1 --2- 172.21.15.83:0/3655043293 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f97bc06b4d0 0x7f97bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.638+0000 7f97ced9d640 1 --2- 172.21.15.83:0/3655043293 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f97bc06b4d0 0x7f97bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.638+0000 7f97b77fe640 1 -- 172.21.15.83:0/3655043293 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f97c40abd00 con 0x7f97d015bdd0 2024-03-06T07:32:20.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.638+0000 7f97ced9d640 1 --2- 172.21.15.83:0/3655043293 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f97bc06b4d0 0x7f97bc06d990 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f97c0004ea0 tx=0x7f97c0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.642+0000 7f97b77fe640 1 -- 172.21.15.83:0/3655043293 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f97d00cdc00 con 0x7f97d015bdd0 2024-03-06T07:32:20.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.854+0000 7f97d518a640 1 -- 172.21.15.83:0/3655043293 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97d004f330 con 0x7f97d015bdd0 2024-03-06T07:32:20.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.855+0000 7f97b77fe640 1 -- 172.21.15.83:0/3655043293 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f97c4040070 con 0x7f97d015bdd0 2024-03-06T07:32:20.857 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:32:20.857 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:32:20.857 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:32:20.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.858+0000 7f97d518a640 1 -- 172.21.15.83:0/3655043293 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f97bc06b4d0 msgr2=0x7f97bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.858+0000 7f97d518a640 1 --2- 172.21.15.83:0/3655043293 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f97bc06b4d0 0x7f97bc06d990 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f97c0004ea0 tx=0x7f97c0009040 comp rx=0 tx=0).stop 2024-03-06T07:32:20.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.858+0000 7f97d518a640 1 -- 172.21.15.83:0/3655043293 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97d015bdd0 msgr2=0x7f97d00b46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.858+0000 7f97d518a640 1 --2- 172.21.15.83:0/3655043293 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97d015bdd0 0x7f97d00b46d0 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f97c4000c00 tx=0x7f97c40046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.858+0000 7f97d518a640 1 -- 172.21.15.83:0/3655043293 shutdown_connections 2024-03-06T07:32:20.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.858+0000 7f97d518a640 1 --2- 172.21.15.83:0/3655043293 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f97bc06b4d0 0x7f97bc06d990 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.858+0000 7f97d518a640 1 --2- 172.21.15.83:0/3655043293 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97d015fa10 0x7f97d00c8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.858+0000 7f97d518a640 1 --2- 172.21.15.83:0/3655043293 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97d015bdd0 0x7f97d00b46d0 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.858+0000 7f97d518a640 1 --2- 172.21.15.83:0/3655043293 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97d00d16e0 0x7f97d01f4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.858+0000 7f97d518a640 1 -- 172.21.15.83:0/3655043293 >> 172.21.15.83:0/3655043293 conn(0x7f97d009a9f0 msgr2=0x7f97d00c6ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:20.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.858+0000 7f97d518a640 1 -- 172.21.15.83:0/3655043293 shutdown_connections 2024-03-06T07:32:20.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:20.858+0000 7f97d518a640 1 -- 172.21.15.83:0/3655043293 wait complete. 2024-03-06T07:32:21.071 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:31:26.591282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:32:21.071 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:32:21.139 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:32:22.072 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:22.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.172+0000 7fa0d0513640 1 Processor -- start 2024-03-06T07:32:22.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.172+0000 7fa0d0513640 1 -- start start 2024-03-06T07:32:22.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.173+0000 7fa0d0513640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa0c815bb20 0x7fa0c80d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.173+0000 7fa0d0513640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0c8161980 con 0x7fa0c80cfff0 2024-03-06T07:32:22.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.173+0000 7fa0d0513640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0c8161c60 con 0x7fa0c80cd8c0 2024-03-06T07:32:22.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.173+0000 7fa0d0513640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0c8161f40 con 0x7fa0c815bb20 2024-03-06T07:32:22.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.173+0000 7fa0ce2af640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa0c815bb20 0x7fa0c80d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.173+0000 7fa0cdaae640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fa0c80cfff0 0x7fa0c80ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:35090/0 (socket says 172.21.15.83:35090) 2024-03-06T07:32:22.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.173+0000 7fa0cdaae640 1 -- 172.21.15.83:0/1213921912 learned_addr learned my addr 172.21.15.83:0/1213921912 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:32:22.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.173+0000 7fa0ce2af640 1 -- 172.21.15.83:0/1213921912 >> v1:172.21.15.183:6789/0 conn(0x7fa0c80cd8c0 legacy=0x7fa0c80cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:22.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.173+0000 7fa0ce2af640 1 -- 172.21.15.83:0/1213921912 >> v1:172.21.15.83:6789/0 conn(0x7fa0c80cfff0 legacy=0x7fa0c80ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.173+0000 7fa0cd2ad640 1 -- 172.21.15.83:0/1213921912 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 514249236 0 0) 0x7fa0c8161980 con 0x7fa0c80cfff0 2024-03-06T07:32:22.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.173+0000 7fa0ce2af640 1 -- 172.21.15.83:0/1213921912 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c8162220 con 0x7fa0c815bb20 2024-03-06T07:32:22.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.174+0000 7fa0ce2af640 1 --2- 172.21.15.83:0/1213921912 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa0c815bb20 0x7fa0c80d1df0 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4009d90 tx=0x7fa0c402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=521643ef9f348c87 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:22.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.174+0000 7fa0cd2ad640 1 -- 172.21.15.83:0/1213921912 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c4039070 con 0x7fa0c815bb20 2024-03-06T07:32:22.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.174+0000 7fa0cd2ad640 1 -- 172.21.15.83:0/1213921912 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0c402cc60 con 0x7fa0c815bb20 2024-03-06T07:32:22.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.174+0000 7fa0cd2ad640 1 -- 172.21.15.83:0/1213921912 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c4034660 con 0x7fa0c815bb20 2024-03-06T07:32:22.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.175+0000 7fa0d0513640 1 -- 172.21.15.83:0/1213921912 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa0c815bb20 msgr2=0x7fa0c80d1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.175+0000 7fa0d0513640 1 --2- 172.21.15.83:0/1213921912 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa0c815bb20 0x7fa0c80d1df0 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4009d90 tx=0x7fa0c402beb0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.175+0000 7fa0d0513640 1 -- 172.21.15.83:0/1213921912 shutdown_connections 2024-03-06T07:32:22.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.175+0000 7fa0d0513640 1 --2- 172.21.15.83:0/1213921912 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa0c815bb20 0x7fa0c80d1df0 unknown :-1 s=CLOSED pgs=1955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.175+0000 7fa0d0513640 1 -- 172.21.15.83:0/1213921912 >> 172.21.15.83:0/1213921912 conn(0x7fa0c809a9f0 msgr2=0x7fa0c809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:22.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.175+0000 7fa0d0513640 1 -- 172.21.15.83:0/1213921912 shutdown_connections 2024-03-06T07:32:22.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.175+0000 7fa0d0513640 1 -- 172.21.15.83:0/1213921912 wait complete. 2024-03-06T07:32:22.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.176+0000 7fa0d0513640 1 Processor -- start 2024-03-06T07:32:22.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.176+0000 7fa0d0513640 1 -- start start 2024-03-06T07:32:22.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.177+0000 7fa0d0513640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa0c80cd8c0 0x7fa0c815b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.177+0000 7fa0d0513640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa0c80cfff0 0x7fa0c80c9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.177+0000 7fa0d0513640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa0c815bb20 0x7fa0c80c55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.177+0000 7fa0d0513640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c80d2330 con 0x7fa0c815bb20 2024-03-06T07:32:22.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.177+0000 7fa0d0513640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c80cf2d0 con 0x7fa0c80cd8c0 2024-03-06T07:32:22.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.177+0000 7fa0d0513640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c80d24b0 con 0x7fa0c80cfff0 2024-03-06T07:32:22.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.177+0000 7fa0ceab0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa0c815bb20 0x7fa0c80c55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.177+0000 7fa0ceab0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa0c815bb20 0x7fa0c80c55f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53386/0 (socket says 172.21.15.83:53386) 2024-03-06T07:32:22.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.177+0000 7fa0ceab0640 1 -- 172.21.15.83:0/1413783330 learned_addr learned my addr 172.21.15.83:0/1413783330 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:22.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.177+0000 7fa0cdaae640 1 --2- 172.21.15.83:0/1413783330 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa0c80cfff0 0x7fa0c80c9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.177+0000 7fa0ce2af640 1 --2- 172.21.15.83:0/1413783330 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa0c80cd8c0 0x7fa0c815b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.177+0000 7fa0ceab0640 1 -- 172.21.15.83:0/1413783330 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa0c80cfff0 msgr2=0x7fa0c80c9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.177+0000 7fa0ceab0640 1 --2- 172.21.15.83:0/1413783330 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa0c80cfff0 0x7fa0c80c9810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.177+0000 7fa0ceab0640 1 -- 172.21.15.83:0/1413783330 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa0c80cd8c0 msgr2=0x7fa0c815b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.177+0000 7fa0ceab0640 1 --2- 172.21.15.83:0/1413783330 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa0c80cd8c0 0x7fa0c815b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.177+0000 7fa0ceab0640 1 -- 172.21.15.83:0/1413783330 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c815b5a0 con 0x7fa0c815bb20 2024-03-06T07:32:22.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.177+0000 7fa0ce2af640 1 --2- 172.21.15.83:0/1413783330 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa0c80cd8c0 0x7fa0c815b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:22.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.177+0000 7fa0ceab0640 1 --2- 172.21.15.83:0/1413783330 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa0c815bb20 0x7fa0c80c55f0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7fa0c000b490 tx=0x7fa0c000b950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:22.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.177+0000 7fa0beffd640 1 -- 172.21.15.83:0/1413783330 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c0015020 con 0x7fa0c815bb20 2024-03-06T07:32:22.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.177+0000 7fa0beffd640 1 -- 172.21.15.83:0/1413783330 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0c0011850 con 0x7fa0c815bb20 2024-03-06T07:32:22.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.177+0000 7fa0beffd640 1 -- 172.21.15.83:0/1413783330 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c0011c40 con 0x7fa0c815bb20 2024-03-06T07:32:22.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.177+0000 7fa0d0513640 1 -- 172.21.15.83:0/1413783330 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0c80c82c0 con 0x7fa0c815bb20 2024-03-06T07:32:22.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.177+0000 7fa0d0513640 1 -- 172.21.15.83:0/1413783330 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0c80c9010 con 0x7fa0c815bb20 2024-03-06T07:32:22.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.178+0000 7fa0d0513640 1 -- 172.21.15.83:0/1413783330 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0c804f330 con 0x7fa0c815bb20 2024-03-06T07:32:22.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.179+0000 7fa0beffd640 1 -- 172.21.15.83:0/1413783330 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa0c0018020 con 0x7fa0c815bb20 2024-03-06T07:32:22.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.179+0000 7fa0beffd640 1 --2- 172.21.15.83:0/1413783330 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa0ac06b500 0x7fa0ac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.179+0000 7fa0beffd640 1 -- 172.21.15.83:0/1413783330 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa0c008a070 con 0x7fa0c815bb20 2024-03-06T07:32:22.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.179+0000 7fa0ce2af640 1 --2- 172.21.15.83:0/1413783330 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa0ac06b500 0x7fa0ac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.180+0000 7fa0ce2af640 1 --2- 172.21.15.83:0/1413783330 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa0ac06b500 0x7fa0ac06d9c0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4036000 tx=0x7fa0c4030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:22.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.183+0000 7fa0beffd640 1 -- 172.21.15.83:0/1413783330 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa0c001f020 con 0x7fa0c815bb20 2024-03-06T07:32:22.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.471+0000 7fa0d0513640 1 -- 172.21.15.83:0/1413783330 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0c80b1820 con 0x7fa0c815bb20 2024-03-06T07:32:22.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.471+0000 7fa0beffd640 1 -- 172.21.15.83:0/1413783330 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7fa0c815b5a0 con 0x7fa0c815bb20 2024-03-06T07:32:22.473 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:32:22.474 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:32:22.474 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:32:22.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.473+0000 7fa0d0513640 1 -- 172.21.15.83:0/1413783330 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa0ac06b500 msgr2=0x7fa0ac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.473+0000 7fa0d0513640 1 --2- 172.21.15.83:0/1413783330 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa0ac06b500 0x7fa0ac06d9c0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4036000 tx=0x7fa0c4030000 comp rx=0 tx=0).stop 2024-03-06T07:32:22.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.473+0000 7fa0d0513640 1 -- 172.21.15.83:0/1413783330 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa0c815bb20 msgr2=0x7fa0c80c55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.473+0000 7fa0d0513640 1 --2- 172.21.15.83:0/1413783330 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa0c815bb20 0x7fa0c80c55f0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7fa0c000b490 tx=0x7fa0c000b950 comp rx=0 tx=0).stop 2024-03-06T07:32:22.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.473+0000 7fa0d0513640 1 -- 172.21.15.83:0/1413783330 shutdown_connections 2024-03-06T07:32:22.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.473+0000 7fa0d0513640 1 --2- 172.21.15.83:0/1413783330 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa0ac06b500 0x7fa0ac06d9c0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.473+0000 7fa0d0513640 1 --2- 172.21.15.83:0/1413783330 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa0c815bb20 0x7fa0c80c55f0 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.473+0000 7fa0d0513640 1 --2- 172.21.15.83:0/1413783330 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa0c80cfff0 0x7fa0c80c9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.473+0000 7fa0d0513640 1 --2- 172.21.15.83:0/1413783330 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa0c80cd8c0 0x7fa0c815b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.473+0000 7fa0d0513640 1 -- 172.21.15.83:0/1413783330 >> 172.21.15.83:0/1413783330 conn(0x7fa0c809a9f0 msgr2=0x7fa0c815cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:22.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.474+0000 7fa0d0513640 1 -- 172.21.15.83:0/1413783330 shutdown_connections 2024-03-06T07:32:22.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:22.474+0000 7fa0d0513640 1 -- 172.21.15.83:0/1413783330 wait complete. 2024-03-06T07:32:22.690 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:31:26.591282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:32:22.690 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:32:23.141 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:32:23.691 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:23.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.801+0000 7f73beec9640 1 Processor -- start 2024-03-06T07:32:23.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.801+0000 7f73beec9640 1 -- start start 2024-03-06T07:32:23.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.802+0000 7f73beec9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f73b80d04a0 0x7f73b80ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.802+0000 7f73beec9640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73b8161980 con 0x7f73b80cfff0 2024-03-06T07:32:23.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.802+0000 7f73beec9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73b8161c60 con 0x7f73b815bb20 2024-03-06T07:32:23.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.802+0000 7f73beec9640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73b80d2500 con 0x7f73b80d04a0 2024-03-06T07:32:23.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.802+0000 7f73bd466640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f73b80d04a0 0x7f73b80ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.802+0000 7f73b7fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f73b80cfff0 0x7f73b80cc8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:35096/0 (socket says 172.21.15.83:35096) 2024-03-06T07:32:23.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.802+0000 7f73b7fff640 1 -- 172.21.15.83:0/477416526 learned_addr learned my addr 172.21.15.83:0/477416526 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:32:23.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.803+0000 7f73bd466640 1 -- 172.21.15.83:0/477416526 >> v1:172.21.15.183:6789/0 conn(0x7f73b815bb20 legacy=0x7f73b80d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:23.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.803+0000 7f73bd466640 1 -- 172.21.15.83:0/477416526 >> v1:172.21.15.83:6789/0 conn(0x7f73b80cfff0 legacy=0x7f73b80cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.803+0000 7f73bd466640 1 -- 172.21.15.83:0/477416526 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b8161f40 con 0x7f73b80d04a0 2024-03-06T07:32:23.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.803+0000 7f73b77fe640 1 -- 172.21.15.83:0/477416526 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4156674680 0 0) 0x7f73b8161980 con 0x7f73b80cfff0 2024-03-06T07:32:23.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.803+0000 7f73bd466640 1 --2- 172.21.15.83:0/477416526 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73b80d04a0 0x7f73b80ccfc0 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7f73a00097a0 tx=0x7f73a002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b1bf94d06a9cae6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:23.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.804+0000 7f73b77fe640 1 -- 172.21.15.83:0/477416526 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a0035020 con 0x7f73b80d04a0 2024-03-06T07:32:23.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.804+0000 7f73b77fe640 1 -- 172.21.15.83:0/477416526 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73a0031830 con 0x7f73b80d04a0 2024-03-06T07:32:23.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.804+0000 7f73b77fe640 1 -- 172.21.15.83:0/477416526 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a0031c00 con 0x7f73b80d04a0 2024-03-06T07:32:23.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.804+0000 7f73beec9640 1 -- 172.21.15.83:0/477416526 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73b80d04a0 msgr2=0x7f73b80ccfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.804+0000 7f73beec9640 1 --2- 172.21.15.83:0/477416526 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73b80d04a0 0x7f73b80ccfc0 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7f73a00097a0 tx=0x7f73a002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.804+0000 7f73beec9640 1 -- 172.21.15.83:0/477416526 shutdown_connections 2024-03-06T07:32:23.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.804+0000 7f73beec9640 1 --2- 172.21.15.83:0/477416526 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73b80d04a0 0x7f73b80ccfc0 unknown :-1 s=CLOSED pgs=1958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.804+0000 7f73beec9640 1 -- 172.21.15.83:0/477416526 >> 172.21.15.83:0/477416526 conn(0x7f73b809a9f0 msgr2=0x7f73b809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:23.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.804+0000 7f73beec9640 1 -- 172.21.15.83:0/477416526 shutdown_connections 2024-03-06T07:32:23.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.804+0000 7f73beec9640 1 -- 172.21.15.83:0/477416526 wait complete. 2024-03-06T07:32:23.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.805+0000 7f73beec9640 1 Processor -- start 2024-03-06T07:32:23.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.805+0000 7f73beec9640 1 -- start start 2024-03-06T07:32:23.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.806+0000 7f73beec9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73b80cfff0 0x7f73b80b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.806+0000 7f73beec9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73b80d04a0 0x7f73b815b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.806+0000 7f73beec9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f73b815bb20 0x7f73b81f4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.806+0000 7f73beec9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b80d08d0 con 0x7f73b80d04a0 2024-03-06T07:32:23.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.806+0000 7f73beec9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b8162500 con 0x7f73b80cfff0 2024-03-06T07:32:23.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.806+0000 7f73beec9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73b80d1980 con 0x7f73b815bb20 2024-03-06T07:32:23.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.806+0000 7f73b7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73b80d04a0 0x7f73b815b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.806+0000 7f73b7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73b80d04a0 0x7f73b815b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53404/0 (socket says 172.21.15.83:53404) 2024-03-06T07:32:23.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.806+0000 7f73b7fff640 1 -- 172.21.15.83:0/941140760 learned_addr learned my addr 172.21.15.83:0/941140760 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:23.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.807+0000 7f73bd466640 1 --2- 172.21.15.83:0/941140760 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f73b815bb20 0x7f73b81f4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.806+0000 7f73bcc65640 1 --2- 172.21.15.83:0/941140760 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73b80cfff0 0x7f73b80b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.807+0000 7f73b7fff640 1 -- 172.21.15.83:0/941140760 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f73b815bb20 msgr2=0x7f73b81f4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.807+0000 7f73b7fff640 1 --2- 172.21.15.83:0/941140760 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f73b815bb20 0x7f73b81f4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.807+0000 7f73b7fff640 1 -- 172.21.15.83:0/941140760 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73b80cfff0 msgr2=0x7f73b80b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.807+0000 7f73b7fff640 1 --2- 172.21.15.83:0/941140760 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73b80cfff0 0x7f73b80b6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.807+0000 7f73b7fff640 1 -- 172.21.15.83:0/941140760 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b815b5a0 con 0x7f73b80d04a0 2024-03-06T07:32:23.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.807+0000 7f73bcc65640 1 --2- 172.21.15.83:0/941140760 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73b80cfff0 0x7f73b80b6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:23.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.807+0000 7f73bd466640 1 --2- 172.21.15.83:0/941140760 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f73b815bb20 0x7f73b81f4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:23.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.807+0000 7f73b7fff640 1 --2- 172.21.15.83:0/941140760 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73b80d04a0 0x7f73b815b060 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f73a800c420 tx=0x7f73a800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:23.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.807+0000 7f73b57fa640 1 -- 172.21.15.83:0/941140760 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a80074e0 con 0x7f73b80d04a0 2024-03-06T07:32:23.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.807+0000 7f73b57fa640 1 -- 172.21.15.83:0/941140760 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73a80047e0 con 0x7f73b80d04a0 2024-03-06T07:32:23.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.807+0000 7f73beec9640 1 -- 172.21.15.83:0/941140760 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73b80b5040 con 0x7f73b80d04a0 2024-03-06T07:32:23.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.807+0000 7f73b57fa640 1 -- 172.21.15.83:0/941140760 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a8004c10 con 0x7f73b80d04a0 2024-03-06T07:32:23.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.807+0000 7f73beec9640 1 -- 172.21.15.83:0/941140760 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73b80b5300 con 0x7f73b80d04a0 2024-03-06T07:32:23.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.808+0000 7f73beec9640 1 -- 172.21.15.83:0/941140760 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73b804f330 con 0x7f73b80d04a0 2024-03-06T07:32:23.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.809+0000 7f73b57fa640 1 -- 172.21.15.83:0/941140760 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f73a8011070 con 0x7f73b80d04a0 2024-03-06T07:32:23.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.809+0000 7f73b57fa640 1 --2- 172.21.15.83:0/941140760 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f739806b500 0x7f739806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.809+0000 7f73b57fa640 1 -- 172.21.15.83:0/941140760 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f73a8089de0 con 0x7f73b80d04a0 2024-03-06T07:32:23.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.809+0000 7f73bcc65640 1 --2- 172.21.15.83:0/941140760 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f739806b500 0x7f739806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.810+0000 7f73bcc65640 1 --2- 172.21.15.83:0/941140760 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f739806b500 0x7f739806d9c0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f73b0001e50 tx=0x7f73b0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:23.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:23.818+0000 7f73b57fa640 1 -- 172.21.15.83:0/941140760 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f73b815b5a0 con 0x7f73b80d04a0 2024-03-06T07:32:24.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:24.034+0000 7f73beec9640 1 -- 172.21.15.83:0/941140760 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73b80b1820 con 0x7f73b80d04a0 2024-03-06T07:32:24.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:24.034+0000 7f73b57fa640 1 -- 172.21.15.83:0/941140760 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f73a800e280 con 0x7f73b80d04a0 2024-03-06T07:32:24.036 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:32:24.036 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:32:24.037 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:32:24.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:24.036+0000 7f73beec9640 1 -- 172.21.15.83:0/941140760 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f739806b500 msgr2=0x7f739806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:24.036+0000 7f73beec9640 1 --2- 172.21.15.83:0/941140760 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f739806b500 0x7f739806d9c0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f73b0001e50 tx=0x7f73b0009040 comp rx=0 tx=0).stop 2024-03-06T07:32:24.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:24.036+0000 7f73beec9640 1 -- 172.21.15.83:0/941140760 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73b80d04a0 msgr2=0x7f73b815b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:24.036+0000 7f73beec9640 1 --2- 172.21.15.83:0/941140760 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73b80d04a0 0x7f73b815b060 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f73a800c420 tx=0x7f73a800c8e0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:24.036+0000 7f73beec9640 1 -- 172.21.15.83:0/941140760 shutdown_connections 2024-03-06T07:32:24.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:24.036+0000 7f73beec9640 1 --2- 172.21.15.83:0/941140760 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f739806b500 0x7f739806d9c0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:24.036+0000 7f73beec9640 1 --2- 172.21.15.83:0/941140760 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f73b815bb20 0x7f73b81f4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:24.036+0000 7f73beec9640 1 --2- 172.21.15.83:0/941140760 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f73b80d04a0 0x7f73b815b060 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:24.036+0000 7f73beec9640 1 --2- 172.21.15.83:0/941140760 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73b80cfff0 0x7f73b80b6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:24.036+0000 7f73beec9640 1 -- 172.21.15.83:0/941140760 >> 172.21.15.83:0/941140760 conn(0x7f73b809a9f0 msgr2=0x7f73b80ae8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:24.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:24.036+0000 7f73beec9640 1 -- 172.21.15.83:0/941140760 shutdown_connections 2024-03-06T07:32:24.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:24.036+0000 7f73beec9640 1 -- 172.21.15.83:0/941140760 wait complete. 2024-03-06T07:32:24.250 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:31:26.591282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:32:24.250 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:32:25.143 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:32:25.251 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:25.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.350+0000 7f423680e640 1 Processor -- start 2024-03-06T07:32:25.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.350+0000 7f423680e640 1 -- start start 2024-03-06T07:32:25.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.351+0000 7f423680e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f42301424f0 0x7f42300cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.351+0000 7f423680e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42300cc980 con 0x7f423006da40 2024-03-06T07:32:25.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.351+0000 7f423680e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42300ccc60 con 0x7f42300cd7a0 2024-03-06T07:32:25.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.351+0000 7f423680e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42300cd220 con 0x7f42301424f0 2024-03-06T07:32:25.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.351+0000 7f4234dab640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f423006da40 0x7f4230071b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:35112/0 (socket says 172.21.15.83:35112) 2024-03-06T07:32:25.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.351+0000 7f4234dab640 1 -- 172.21.15.83:0/2335875044 learned_addr learned my addr 172.21.15.83:0/2335875044 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:32:25.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.352+0000 7f422ffff640 1 --2- 172.21.15.83:0/2335875044 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f42301424f0 0x7f42300cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.352+0000 7f422effd640 1 -- 172.21.15.83:0/2335875044 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3626648539 0 0) 0x7f42300cc980 con 0x7f423006da40 2024-03-06T07:32:25.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.352+0000 7f422effd640 1 -- 172.21.15.83:0/2335875044 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4218003520 con 0x7f423006da40 2024-03-06T07:32:25.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.352+0000 7f422effd640 1 -- 172.21.15.83:0/2335875044 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2409300796 0 0) 0x7f4218003520 con 0x7f423006da40 2024-03-06T07:32:25.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.352+0000 7f422effd640 1 -- 172.21.15.83:0/2335875044 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f42301424f0 msgr2=0x7f42300cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:25.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.352+0000 7f422effd640 1 --2- 172.21.15.83:0/2335875044 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f42301424f0 0x7f42300cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.352+0000 7f422effd640 1 -- 172.21.15.83:0/2335875044 >> v1:172.21.15.183:6789/0 conn(0x7f42300cd7a0 legacy=0x7f423006d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.352+0000 7f422effd640 1 -- 172.21.15.83:0/2335875044 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42300ccf40 con 0x7f423006da40 2024-03-06T07:32:25.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.352+0000 7f422effd640 1 -- 172.21.15.83:0/2335875044 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f42280019f0 con 0x7f423006da40 2024-03-06T07:32:25.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.352+0000 7f422effd640 1 -- 172.21.15.83:0/2335875044 >> v1:172.21.15.83:6789/0 conn(0x7f423006da40 legacy=0x7f4230071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.352+0000 7f422effd640 1 --2- 172.21.15.83:0/2335875044 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42180039e0 0x7f4218003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.352+0000 7f422effd640 1 --2- 172.21.15.83:0/2335875044 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4218004800 0x7f4218004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.352+0000 7f422effd640 1 --2- 172.21.15.83:0/2335875044 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f42301424f0 0x7f42180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.352+0000 7f422effd640 1 -- 172.21.15.83:0/2335875044 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42300cd220 con 0x7f42180039e0 2024-03-06T07:32:25.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.352+0000 7f422effd640 1 -- 172.21.15.83:0/2335875044 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f421800ba70 con 0x7f4218004800 2024-03-06T07:32:25.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.352+0000 7f422effd640 1 -- 172.21.15.83:0/2335875044 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f421800bc70 con 0x7f42301424f0 2024-03-06T07:32:25.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.352+0000 7f422ffff640 1 --2- 172.21.15.83:0/2335875044 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42180039e0 0x7f4218003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.352+0000 7f422f7fe640 1 --2- 172.21.15.83:0/2335875044 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4218004800 0x7f4218004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.352+0000 7f4234dab640 1 --2- 172.21.15.83:0/2335875044 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f42301424f0 0x7f42180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.352+0000 7f422ffff640 1 -- 172.21.15.83:0/2335875044 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f42301424f0 msgr2=0x7f42180051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:25.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.352+0000 7f422ffff640 1 --2- 172.21.15.83:0/2335875044 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f42301424f0 0x7f42180051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.352+0000 7f422ffff640 1 -- 172.21.15.83:0/2335875044 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4218004800 msgr2=0x7f4218004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:25.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.352+0000 7f422ffff640 1 --2- 172.21.15.83:0/2335875044 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4218004800 0x7f4218004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.352+0000 7f422ffff640 1 -- 172.21.15.83:0/2335875044 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4218003520 con 0x7f42180039e0 2024-03-06T07:32:25.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.352+0000 7f422f7fe640 1 --2- 172.21.15.83:0/2335875044 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4218004800 0x7f4218004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:25.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.353+0000 7f422ffff640 1 --2- 172.21.15.83:0/2335875044 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42180039e0 0x7f4218003e00 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f4224009a90 tx=0x7f4218005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=860cadd03c9e032b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:25.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.353+0000 7f422effd640 1 -- 172.21.15.83:0/2335875044 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4224039070 con 0x7f42180039e0 2024-03-06T07:32:25.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.353+0000 7f422effd640 1 -- 172.21.15.83:0/2335875044 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f422402c680 con 0x7f42180039e0 2024-03-06T07:32:25.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.353+0000 7f422effd640 1 -- 172.21.15.83:0/2335875044 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4224009690 con 0x7f42180039e0 2024-03-06T07:32:25.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.354+0000 7f423680e640 1 -- 172.21.15.83:0/2335875044 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42180039e0 msgr2=0x7f4218003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.354+0000 7f423680e640 1 --2- 172.21.15.83:0/2335875044 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42180039e0 0x7f4218003e00 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f4224009a90 tx=0x7f4218005830 comp rx=0 tx=0).stop 2024-03-06T07:32:25.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.354+0000 7f4234dab640 1 -- 172.21.15.83:0/2335875044 reap_dead start 2024-03-06T07:32:25.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.354+0000 7f423680e640 1 -- 172.21.15.83:0/2335875044 shutdown_connections 2024-03-06T07:32:25.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.354+0000 7f423680e640 1 -- 172.21.15.83:0/2335875044 >> 172.21.15.83:0/2335875044 conn(0x7f42300cf8a0 msgr2=0x7f42300cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:25.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.354+0000 7f423680e640 1 -- 172.21.15.83:0/2335875044 shutdown_connections 2024-03-06T07:32:25.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.354+0000 7f423680e640 1 -- 172.21.15.83:0/2335875044 wait complete. 2024-03-06T07:32:25.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.355+0000 7f423680e640 1 Processor -- start 2024-03-06T07:32:25.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.356+0000 7f423680e640 1 -- start start 2024-03-06T07:32:25.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.357+0000 7f423680e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f42180039e0 0x7f4230151570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.357+0000 7f423680e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f423013ba10 0x7f423013d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.357+0000 7f423680e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f423013ad50 0x7f423013b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.357+0000 7f423680e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42300cc470 con 0x7f423013ad50 2024-03-06T07:32:25.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.357+0000 7f423680e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42300cc2f0 con 0x7f42180039e0 2024-03-06T07:32:25.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.357+0000 7f423680e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42300cc5f0 con 0x7f423013ba10 2024-03-06T07:32:25.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.357+0000 7f4234dab640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f423013ad50 0x7f423013b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.357+0000 7f4234dab640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f423013ad50 0x7f423013b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53430/0 (socket says 172.21.15.83:53430) 2024-03-06T07:32:25.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.357+0000 7f4234dab640 1 -- 172.21.15.83:0/2018033689 learned_addr learned my addr 172.21.15.83:0/2018033689 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:25.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.357+0000 7f422f7fe640 1 --2- 172.21.15.83:0/2018033689 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f423013ba10 0x7f423013d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.357+0000 7f422ffff640 1 --2- 172.21.15.83:0/2018033689 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f42180039e0 0x7f4230151570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.357+0000 7f4234dab640 1 -- 172.21.15.83:0/2018033689 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f423013ba10 msgr2=0x7f423013d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.357+0000 7f4234dab640 1 --2- 172.21.15.83:0/2018033689 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f423013ba10 0x7f423013d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.357+0000 7f4234dab640 1 -- 172.21.15.83:0/2018033689 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f42180039e0 msgr2=0x7f4230151570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.357+0000 7f4234dab640 1 --2- 172.21.15.83:0/2018033689 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f42180039e0 0x7f4230151570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.357+0000 7f4234dab640 1 -- 172.21.15.83:0/2018033689 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f423013be70 con 0x7f423013ad50 2024-03-06T07:32:25.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.357+0000 7f422f7fe640 1 --2- 172.21.15.83:0/2018033689 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f423013ba10 0x7f423013d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:25.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.357+0000 7f422ffff640 1 --2- 172.21.15.83:0/2018033689 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f42180039e0 0x7f4230151570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:25.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.357+0000 7f4234dab640 1 --2- 172.21.15.83:0/2018033689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f423013ad50 0x7f423013b190 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f423006c6d0 tx=0x7f422800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:25.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.358+0000 7f422cff9640 1 -- 172.21.15.83:0/2018033689 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f422800eb90 con 0x7f423013ad50 2024-03-06T07:32:25.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.358+0000 7f422cff9640 1 -- 172.21.15.83:0/2018033689 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f422800ed30 con 0x7f423013ad50 2024-03-06T07:32:25.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.358+0000 7f423680e640 1 -- 172.21.15.83:0/2018033689 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f423013c140 con 0x7f423013ad50 2024-03-06T07:32:25.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.358+0000 7f422cff9640 1 -- 172.21.15.83:0/2018033689 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4228014670 con 0x7f423013ad50 2024-03-06T07:32:25.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.358+0000 7f423680e640 1 -- 172.21.15.83:0/2018033689 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f423013c700 con 0x7f423013ad50 2024-03-06T07:32:25.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.359+0000 7f423680e640 1 -- 172.21.15.83:0/2018033689 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f423004f330 con 0x7f423013ad50 2024-03-06T07:32:25.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.360+0000 7f422cff9640 1 -- 172.21.15.83:0/2018033689 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4228014810 con 0x7f423013ad50 2024-03-06T07:32:25.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.360+0000 7f422cff9640 1 --2- 172.21.15.83:0/2018033689 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f421c06b500 0x7f421c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.360+0000 7f422ffff640 1 --2- 172.21.15.83:0/2018033689 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f421c06b500 0x7f421c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.360+0000 7f422cff9640 1 -- 172.21.15.83:0/2018033689 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f422808a010 con 0x7f423013ad50 2024-03-06T07:32:25.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.361+0000 7f422ffff640 1 --2- 172.21.15.83:0/2018033689 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f421c06b500 0x7f421c06d9c0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f4224036000 tx=0x7f4224030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:25.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.365+0000 7f422cff9640 1 -- 172.21.15.83:0/2018033689 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4228058e50 con 0x7f423013ad50 2024-03-06T07:32:25.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.579+0000 7f423680e640 1 -- 172.21.15.83:0/2018033689 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42300d0620 con 0x7f423013ad50 2024-03-06T07:32:25.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.579+0000 7f422cff9640 1 -- 172.21.15.83:0/2018033689 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f4228058750 con 0x7f423013ad50 2024-03-06T07:32:25.581 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:32:25.581 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:32:25.582 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:32:25.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.581+0000 7f423680e640 1 -- 172.21.15.83:0/2018033689 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f421c06b500 msgr2=0x7f421c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.581+0000 7f423680e640 1 --2- 172.21.15.83:0/2018033689 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f421c06b500 0x7f421c06d9c0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f4224036000 tx=0x7f4224030000 comp rx=0 tx=0).stop 2024-03-06T07:32:25.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.581+0000 7f423680e640 1 -- 172.21.15.83:0/2018033689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f423013ad50 msgr2=0x7f423013b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.581+0000 7f423680e640 1 --2- 172.21.15.83:0/2018033689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f423013ad50 0x7f423013b190 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f423006c6d0 tx=0x7f422800e530 comp rx=0 tx=0).stop 2024-03-06T07:32:25.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.581+0000 7f423680e640 1 -- 172.21.15.83:0/2018033689 shutdown_connections 2024-03-06T07:32:25.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.581+0000 7f423680e640 1 --2- 172.21.15.83:0/2018033689 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f421c06b500 0x7f421c06d9c0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.581+0000 7f423680e640 1 --2- 172.21.15.83:0/2018033689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f423013ad50 0x7f423013b190 unknown :-1 s=CLOSED pgs=1962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.581+0000 7f423680e640 1 --2- 172.21.15.83:0/2018033689 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f423013ba10 0x7f423013d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.581+0000 7f423680e640 1 --2- 172.21.15.83:0/2018033689 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f42180039e0 0x7f4230151570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.581+0000 7f423680e640 1 -- 172.21.15.83:0/2018033689 >> 172.21.15.83:0/2018033689 conn(0x7f42300cf8a0 msgr2=0x7f42300af5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:25.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.581+0000 7f423680e640 1 -- 172.21.15.83:0/2018033689 shutdown_connections 2024-03-06T07:32:25.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:25.581+0000 7f423680e640 1 -- 172.21.15.83:0/2018033689 wait complete. 2024-03-06T07:32:25.795 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:31:26.591282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:32:25.795 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:32:26.796 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:26.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.896+0000 7f23b6f81640 1 Processor -- start 2024-03-06T07:32:26.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.896+0000 7f23b6f81640 1 -- start start 2024-03-06T07:32:26.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.897+0000 7f23b6f81640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f23b00d22b0 0x7f23b00d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.897+0000 7f23b6f81640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b00b3430 con 0x7f23b015dfe0 2024-03-06T07:32:26.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.897+0000 7f23b6f81640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b00b3710 con 0x7f23b00b41f0 2024-03-06T07:32:26.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.897+0000 7f23b6f81640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23b00d2c50 con 0x7f23b00d22b0 2024-03-06T07:32:26.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.897+0000 7f23b4d1d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f23b015dfe0 0x7f23b00cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:44482/0 (socket says 172.21.15.83:44482) 2024-03-06T07:32:26.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.897+0000 7f23b4d1d640 1 -- 172.21.15.83:0/395324075 learned_addr learned my addr 172.21.15.83:0/395324075 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:32:26.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.897+0000 7f23affff640 1 --2- 172.21.15.83:0/395324075 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f23b00d22b0 0x7f23b00d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.897+0000 7f23af7fe640 1 -- 172.21.15.83:0/395324075 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1135734957 0 0) 0x7f23b00b3430 con 0x7f23b015dfe0 2024-03-06T07:32:26.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.897+0000 7f23af7fe640 1 -- 172.21.15.83:0/395324075 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23a0003520 con 0x7f23b015dfe0 2024-03-06T07:32:26.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.898+0000 7f23af7fe640 1 -- 172.21.15.83:0/395324075 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2001118377 0 0) 0x7f23a0003520 con 0x7f23b015dfe0 2024-03-06T07:32:26.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.898+0000 7f23af7fe640 1 -- 172.21.15.83:0/395324075 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f23b00d22b0 msgr2=0x7f23b00d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:26.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.898+0000 7f23af7fe640 1 --2- 172.21.15.83:0/395324075 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f23b00d22b0 0x7f23b00d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.898+0000 7f23af7fe640 1 -- 172.21.15.83:0/395324075 >> v1:172.21.15.183:6789/0 conn(0x7f23b00b41f0 legacy=0x7f23b00b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.898+0000 7f23af7fe640 1 -- 172.21.15.83:0/395324075 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b00b39f0 con 0x7f23b015dfe0 2024-03-06T07:32:26.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.898+0000 7f23af7fe640 1 -- 172.21.15.83:0/395324075 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f23a4001670 con 0x7f23b015dfe0 2024-03-06T07:32:26.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.898+0000 7f23af7fe640 1 -- 172.21.15.83:0/395324075 >> v1:172.21.15.83:6789/0 conn(0x7f23b015dfe0 legacy=0x7f23b00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.898+0000 7f23af7fe640 1 --2- 172.21.15.83:0/395324075 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23a00039e0 0x7f23a0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.898+0000 7f23af7fe640 1 --2- 172.21.15.83:0/395324075 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f23a0004800 0x7f23a0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.898+0000 7f23af7fe640 1 --2- 172.21.15.83:0/395324075 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f23b00d22b0 0x7f23a00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.898+0000 7f23af7fe640 1 -- 172.21.15.83:0/395324075 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a000ba70 con 0x7f23a00039e0 2024-03-06T07:32:26.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.898+0000 7f23af7fe640 1 -- 172.21.15.83:0/395324075 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a000bbf0 con 0x7f23a0004800 2024-03-06T07:32:26.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.898+0000 7f23af7fe640 1 -- 172.21.15.83:0/395324075 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23a000bdf0 con 0x7f23b00d22b0 2024-03-06T07:32:26.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.898+0000 7f23b4d1d640 1 --2- 172.21.15.83:0/395324075 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23a00039e0 0x7f23a0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.898+0000 7f23affff640 1 --2- 172.21.15.83:0/395324075 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f23a0004800 0x7f23a0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.898+0000 7f23b551e640 1 --2- 172.21.15.83:0/395324075 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f23b00d22b0 0x7f23a00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.898+0000 7f23b4d1d640 1 -- 172.21.15.83:0/395324075 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f23b00d22b0 msgr2=0x7f23a00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:26.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.898+0000 7f23b4d1d640 1 --2- 172.21.15.83:0/395324075 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f23b00d22b0 0x7f23a00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.898+0000 7f23b4d1d640 1 -- 172.21.15.83:0/395324075 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f23a0004800 msgr2=0x7f23a0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:26.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.898+0000 7f23b4d1d640 1 --2- 172.21.15.83:0/395324075 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f23a0004800 0x7f23a0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.898+0000 7f23b4d1d640 1 -- 172.21.15.83:0/395324075 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a0003520 con 0x7f23a00039e0 2024-03-06T07:32:26.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.898+0000 7f23affff640 1 --2- 172.21.15.83:0/395324075 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f23a0004800 0x7f23a0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:26.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.898+0000 7f23b551e640 1 --2- 172.21.15.83:0/395324075 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f23b00d22b0 0x7f23a00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:26.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.899+0000 7f23b4d1d640 1 --2- 172.21.15.83:0/395324075 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23a00039e0 0x7f23a0003e00 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f23a400c210 tx=0x7f23a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afe2bf47df6fe46b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:26.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.899+0000 7f23af7fe640 1 -- 172.21.15.83:0/395324075 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a4007d10 con 0x7f23a00039e0 2024-03-06T07:32:26.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.899+0000 7f23af7fe640 1 -- 172.21.15.83:0/395324075 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23a4030070 con 0x7f23a00039e0 2024-03-06T07:32:26.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.899+0000 7f23af7fe640 1 -- 172.21.15.83:0/395324075 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a4005d50 con 0x7f23a00039e0 2024-03-06T07:32:26.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.900+0000 7f23b6f81640 1 -- 172.21.15.83:0/395324075 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23a00039e0 msgr2=0x7f23a0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.900+0000 7f23b6f81640 1 --2- 172.21.15.83:0/395324075 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23a00039e0 0x7f23a0003e00 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f23a400c210 tx=0x7f23a0005830 comp rx=0 tx=0).stop 2024-03-06T07:32:26.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.900+0000 7f23b551e640 1 -- 172.21.15.83:0/395324075 reap_dead start 2024-03-06T07:32:26.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.900+0000 7f23b6f81640 1 -- 172.21.15.83:0/395324075 shutdown_connections 2024-03-06T07:32:26.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.900+0000 7f23b6f81640 1 -- 172.21.15.83:0/395324075 >> 172.21.15.83:0/395324075 conn(0x7f23b00d3cd0 msgr2=0x7f23b00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:26.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.900+0000 7f23b6f81640 1 -- 172.21.15.83:0/395324075 shutdown_connections 2024-03-06T07:32:26.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.900+0000 7f23b6f81640 1 -- 172.21.15.83:0/395324075 wait complete. 2024-03-06T07:32:26.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.902+0000 7f23b6f81640 1 Processor -- start 2024-03-06T07:32:26.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.902+0000 7f23b6f81640 1 -- start start 2024-03-06T07:32:26.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.903+0000 7f23b6f81640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f23a00039e0 0x7f23b00b75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.903+0000 7f23b6f81640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23b00b8490 0x7f23b00b4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.903+0000 7f23b6f81640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f23b014dce0 0x7f23b014e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.903+0000 7f23b6f81640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b00b3d60 con 0x7f23b00b8490 2024-03-06T07:32:26.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.903+0000 7f23b6f81640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b00b31e0 con 0x7f23b014dce0 2024-03-06T07:32:26.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.903+0000 7f23b6f81640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23b00d1c70 con 0x7f23a00039e0 2024-03-06T07:32:26.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.903+0000 7f23affff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23b00b8490 0x7f23b00b4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.903+0000 7f23affff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23b00b8490 0x7f23b00b4840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58088/0 (socket says 172.21.15.83:58088) 2024-03-06T07:32:26.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.903+0000 7f23affff640 1 -- 172.21.15.83:0/1691151867 learned_addr learned my addr 172.21.15.83:0/1691151867 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:26.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.903+0000 7f23b4d1d640 1 --2- 172.21.15.83:0/1691151867 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f23a00039e0 0x7f23b00b75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.903+0000 7f23b551e640 1 --2- 172.21.15.83:0/1691151867 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f23b014dce0 0x7f23b014e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.903+0000 7f23affff640 1 -- 172.21.15.83:0/1691151867 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f23a00039e0 msgr2=0x7f23b00b75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.903+0000 7f23affff640 1 --2- 172.21.15.83:0/1691151867 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f23a00039e0 0x7f23b00b75c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.904+0000 7f23affff640 1 -- 172.21.15.83:0/1691151867 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f23b014dce0 msgr2=0x7f23b014e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.904+0000 7f23affff640 1 --2- 172.21.15.83:0/1691151867 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f23b014dce0 0x7f23b014e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.904+0000 7f23affff640 1 -- 172.21.15.83:0/1691151867 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b00b8ca0 con 0x7f23b00b8490 2024-03-06T07:32:26.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.904+0000 7f23affff640 1 --2- 172.21.15.83:0/1691151867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23b00b8490 0x7f23b00b4840 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f239c0116d0 tx=0x7f239c011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:26.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.904+0000 7f23ad7fa640 1 -- 172.21.15.83:0/1691151867 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239c00f9c0 con 0x7f23b00b8490 2024-03-06T07:32:26.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.904+0000 7f23ad7fa640 1 -- 172.21.15.83:0/1691151867 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f239c00fb60 con 0x7f23b00b8490 2024-03-06T07:32:26.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.904+0000 7f23b6f81640 1 -- 172.21.15.83:0/1691151867 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23b00b8f80 con 0x7f23b00b8490 2024-03-06T07:32:26.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.904+0000 7f23ad7fa640 1 -- 172.21.15.83:0/1691151867 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239c015460 con 0x7f23b00b8490 2024-03-06T07:32:26.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.904+0000 7f23b6f81640 1 -- 172.21.15.83:0/1691151867 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23b00b9540 con 0x7f23b00b8490 2024-03-06T07:32:26.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.905+0000 7f23b6f81640 1 -- 172.21.15.83:0/1691151867 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23b004f330 con 0x7f23b00b8490 2024-03-06T07:32:26.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.905+0000 7f23ad7fa640 1 -- 172.21.15.83:0/1691151867 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f239c015600 con 0x7f23b00b8490 2024-03-06T07:32:26.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.906+0000 7f23ad7fa640 1 --2- 172.21.15.83:0/1691151867 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f239406b500 0x7f239406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.906+0000 7f23ad7fa640 1 -- 172.21.15.83:0/1691151867 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f239c08ab20 con 0x7f23b00b8490 2024-03-06T07:32:26.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.906+0000 7f23b4d1d640 1 --2- 172.21.15.83:0/1691151867 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f239406b500 0x7f239406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.906+0000 7f23b4d1d640 1 --2- 172.21.15.83:0/1691151867 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f239406b500 0x7f239406d9c0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f23a4034000 tx=0x7f23a4009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:26.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:26.910+0000 7f23ad7fa640 1 -- 172.21.15.83:0/1691151867 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f239c019030 con 0x7f23b00b8490 2024-03-06T07:32:27.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:27.125+0000 7f23b6f81640 1 -- 172.21.15.83:0/1691151867 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23b015d590 con 0x7f23b00b8490 2024-03-06T07:32:27.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:27.125+0000 7f23ad7fa640 1 -- 172.21.15.83:0/1691151867 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f239c059960 con 0x7f23b00b8490 2024-03-06T07:32:27.127 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:32:27.127 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:32:27.128 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:32:27.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:27.127+0000 7f23b6f81640 1 -- 172.21.15.83:0/1691151867 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f239406b500 msgr2=0x7f239406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:27.127+0000 7f23b6f81640 1 --2- 172.21.15.83:0/1691151867 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f239406b500 0x7f239406d9c0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f23a4034000 tx=0x7f23a4009030 comp rx=0 tx=0).stop 2024-03-06T07:32:27.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:27.127+0000 7f23b6f81640 1 -- 172.21.15.83:0/1691151867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23b00b8490 msgr2=0x7f23b00b4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:27.127+0000 7f23b6f81640 1 --2- 172.21.15.83:0/1691151867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23b00b8490 0x7f23b00b4840 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f239c0116d0 tx=0x7f239c011b90 comp rx=0 tx=0).stop 2024-03-06T07:32:27.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:27.127+0000 7f23b6f81640 1 -- 172.21.15.83:0/1691151867 shutdown_connections 2024-03-06T07:32:27.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:27.127+0000 7f23b6f81640 1 --2- 172.21.15.83:0/1691151867 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f239406b500 0x7f239406d9c0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:27.127+0000 7f23b6f81640 1 --2- 172.21.15.83:0/1691151867 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f23b014dce0 0x7f23b014e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:27.127+0000 7f23b6f81640 1 --2- 172.21.15.83:0/1691151867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23b00b8490 0x7f23b00b4840 unknown :-1 s=CLOSED pgs=1965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:27.127+0000 7f23b6f81640 1 --2- 172.21.15.83:0/1691151867 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f23a00039e0 0x7f23b00b75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:27.127+0000 7f23b6f81640 1 -- 172.21.15.83:0/1691151867 >> 172.21.15.83:0/1691151867 conn(0x7f23b00d3cd0 msgr2=0x7f23b00d08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:27.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:27.127+0000 7f23b6f81640 1 -- 172.21.15.83:0/1691151867 shutdown_connections 2024-03-06T07:32:27.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:27.127+0000 7f23b6f81640 1 -- 172.21.15.83:0/1691151867 wait complete. 2024-03-06T07:32:27.145 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:32:27.342 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:31:26.591282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:32:27.342 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:32:28.342 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:28.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.461+0000 7f56d4f7a640 1 Processor -- start 2024-03-06T07:32:28.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.461+0000 7f56d4f7a640 1 -- start start 2024-03-06T07:32:28.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.462+0000 7f56d4f7a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f56d015bdd0 0x7f56d00c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.462+0000 7f56d4f7a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f56d015fa10 0x7f56d00ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.462+0000 7f56d4f7a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f56d00d16e0 0x7f56d00d1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.462+0000 7f56d4f7a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56d00cebe0 con 0x7f56d015fa10 2024-03-06T07:32:28.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.462+0000 7f56d4f7a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56d00ceec0 con 0x7f56d00d16e0 2024-03-06T07:32:28.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.462+0000 7f56d4f7a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56d00cf1a0 con 0x7f56d015bdd0 2024-03-06T07:32:28.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.462+0000 7f56c5d9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f56d015fa10 0x7f56d00ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.462+0000 7f56c5d9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f56d015fa10 0x7f56d00ce270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58098/0 (socket says 172.21.15.83:58098) 2024-03-06T07:32:28.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.462+0000 7f56c5d9b640 1 -- 172.21.15.83:0/2000653515 learned_addr learned my addr 172.21.15.83:0/2000653515 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:28.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.462+0000 7f56ced9d640 1 --2- 172.21.15.83:0/2000653515 >> v2:172.21.15.183:3300/0 conn(0x7f56d00d16e0 0x7f56d00d1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.462+0000 7f56ce59c640 1 --2- 172.21.15.83:0/2000653515 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f56d015bdd0 0x7f56d00c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.462+0000 7f56c5d9b640 1 -- 172.21.15.83:0/2000653515 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f56d015bdd0 msgr2=0x7f56d00c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:28.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.462+0000 7f56c5d9b640 1 --2- 172.21.15.83:0/2000653515 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f56d015bdd0 0x7f56d00c5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.462+0000 7f56c5d9b640 1 -- 172.21.15.83:0/2000653515 >> v2:172.21.15.183:3300/0 conn(0x7f56d00d16e0 msgr2=0x7f56d00d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:28.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.462+0000 7f56c5d9b640 1 --2- 172.21.15.83:0/2000653515 >> v2:172.21.15.183:3300/0 conn(0x7f56d00d16e0 0x7f56d00d1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.462+0000 7f56c5d9b640 1 -- 172.21.15.83:0/2000653515 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d00cc8e0 con 0x7f56d015fa10 2024-03-06T07:32:28.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.462+0000 7f56ced9d640 1 --2- 172.21.15.83:0/2000653515 >> v2:172.21.15.183:3300/0 conn(0x7f56d00d16e0 0x7f56d00d1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:28.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.463+0000 7f56c5d9b640 1 --2- 172.21.15.83:0/2000653515 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56d015fa10 0x7f56d00ce270 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f56d00729e0 tx=0x7f56b802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9755af8e979d1177 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:28.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.463+0000 7f56cdd9b640 1 -- 172.21.15.83:0/2000653515 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b802e070 con 0x7f56d015fa10 2024-03-06T07:32:28.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.463+0000 7f56cdd9b640 1 -- 172.21.15.83:0/2000653515 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56b8037070 con 0x7f56d015fa10 2024-03-06T07:32:28.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.463+0000 7f56cdd9b640 1 -- 172.21.15.83:0/2000653515 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b8031a00 con 0x7f56d015fa10 2024-03-06T07:32:28.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.464+0000 7f56d4f7a640 1 -- 172.21.15.83:0/2000653515 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56d015fa10 msgr2=0x7f56d00ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.464+0000 7f56d4f7a640 1 --2- 172.21.15.83:0/2000653515 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56d015fa10 0x7f56d00ce270 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f56d00729e0 tx=0x7f56b802b880 comp rx=0 tx=0).stop 2024-03-06T07:32:28.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.464+0000 7f56d4f7a640 1 -- 172.21.15.83:0/2000653515 shutdown_connections 2024-03-06T07:32:28.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.464+0000 7f56d4f7a640 1 --2- 172.21.15.83:0/2000653515 >> v2:172.21.15.183:3300/0 conn(0x7f56d00d16e0 0x7f56d00d1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.464+0000 7f56d4f7a640 1 --2- 172.21.15.83:0/2000653515 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56d015fa10 0x7f56d00ce270 unknown :-1 s=CLOSED pgs=1966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.464+0000 7f56d4f7a640 1 --2- 172.21.15.83:0/2000653515 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f56d015bdd0 0x7f56d00c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.464+0000 7f56d4f7a640 1 -- 172.21.15.83:0/2000653515 >> 172.21.15.83:0/2000653515 conn(0x7f56d009a9f0 msgr2=0x7f56d009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:28.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.464+0000 7f56d4f7a640 1 -- 172.21.15.83:0/2000653515 shutdown_connections 2024-03-06T07:32:28.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.464+0000 7f56d4f7a640 1 -- 172.21.15.83:0/2000653515 wait complete. 2024-03-06T07:32:28.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.465+0000 7f56d4f7a640 1 Processor -- start 2024-03-06T07:32:28.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.465+0000 7f56d4f7a640 1 -- start start 2024-03-06T07:32:28.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.466+0000 7f56d4f7a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f56d00d16e0 0x7f56d009eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.466+0000 7f56d4f7a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f56d015bdd0 0x7f56d00b5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.466+0000 7f56d4f7a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56d015fa10 0x7f56d015edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.466+0000 7f56d4f7a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d00cc5d0 con 0x7f56d015fa10 2024-03-06T07:32:28.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.466+0000 7f56d4f7a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d00d2070 con 0x7f56d00d16e0 2024-03-06T07:32:28.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.466+0000 7f56d4f7a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56d015c1b0 con 0x7f56d015bdd0 2024-03-06T07:32:28.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.466+0000 7f56ced9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56d015fa10 0x7f56d015edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.466+0000 7f56ced9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56d015fa10 0x7f56d015edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58110/0 (socket says 172.21.15.83:58110) 2024-03-06T07:32:28.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.466+0000 7f56ced9d640 1 -- 172.21.15.83:0/2627932075 learned_addr learned my addr 172.21.15.83:0/2627932075 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:28.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.466+0000 7f56c5d9b640 1 --2- 172.21.15.83:0/2627932075 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f56d015bdd0 0x7f56d00b5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.466+0000 7f56ce59c640 1 --2- 172.21.15.83:0/2627932075 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f56d00d16e0 0x7f56d009eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.466+0000 7f56ced9d640 1 -- 172.21.15.83:0/2627932075 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f56d015bdd0 msgr2=0x7f56d00b5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.466+0000 7f56ced9d640 1 --2- 172.21.15.83:0/2627932075 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f56d015bdd0 0x7f56d00b5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.466+0000 7f56ced9d640 1 -- 172.21.15.83:0/2627932075 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f56d00d16e0 msgr2=0x7f56d009eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.466+0000 7f56ced9d640 1 --2- 172.21.15.83:0/2627932075 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f56d00d16e0 0x7f56d009eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.466+0000 7f56ced9d640 1 -- 172.21.15.83:0/2627932075 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d00b6c20 con 0x7f56d015fa10 2024-03-06T07:32:28.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.467+0000 7f56ced9d640 1 --2- 172.21.15.83:0/2627932075 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56d015fa10 0x7f56d015edb0 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7f56c8007ea0 tx=0x7f56c8007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:28.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.467+0000 7f56c77fe640 1 -- 172.21.15.83:0/2627932075 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c801a070 con 0x7f56d015fa10 2024-03-06T07:32:28.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.467+0000 7f56c77fe640 1 -- 172.21.15.83:0/2627932075 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56c8003bb0 con 0x7f56d015fa10 2024-03-06T07:32:28.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.467+0000 7f56d4f7a640 1 -- 172.21.15.83:0/2627932075 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56d015e4a0 con 0x7f56d015fa10 2024-03-06T07:32:28.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.467+0000 7f56c77fe640 1 -- 172.21.15.83:0/2627932075 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c800de70 con 0x7f56d015fa10 2024-03-06T07:32:28.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.467+0000 7f56d4f7a640 1 -- 172.21.15.83:0/2627932075 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56d00b6660 con 0x7f56d015fa10 2024-03-06T07:32:28.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.468+0000 7f56d4f7a640 1 -- 172.21.15.83:0/2627932075 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56d004f330 con 0x7f56d015fa10 2024-03-06T07:32:28.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.469+0000 7f56c77fe640 1 -- 172.21.15.83:0/2627932075 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f56c801d070 con 0x7f56d015fa10 2024-03-06T07:32:28.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.469+0000 7f56c77fe640 1 --2- 172.21.15.83:0/2627932075 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f56b006b4d0 0x7f56b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.469+0000 7f56c77fe640 1 -- 172.21.15.83:0/2627932075 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f56c808a400 con 0x7f56d015fa10 2024-03-06T07:32:28.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.469+0000 7f56ce59c640 1 --2- 172.21.15.83:0/2627932075 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f56b006b4d0 0x7f56b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.470+0000 7f56ce59c640 1 --2- 172.21.15.83:0/2627932075 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f56b006b4d0 0x7f56b006d990 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f56c0006fd0 tx=0x7f56c0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:28.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.478+0000 7f56c77fe640 1 -- 172.21.15.83:0/2627932075 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f56c8010070 con 0x7f56d015fa10 2024-03-06T07:32:28.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.686+0000 7f56d4f7a640 1 -- 172.21.15.83:0/2627932075 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56d00b7a40 con 0x7f56d015fa10 2024-03-06T07:32:28.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.686+0000 7f56c77fe640 1 -- 172.21.15.83:0/2627932075 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f56c80592d0 con 0x7f56d015fa10 2024-03-06T07:32:28.688 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:32:28.689 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:32:28.689 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:32:28.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.688+0000 7f56d4f7a640 1 -- 172.21.15.83:0/2627932075 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f56b006b4d0 msgr2=0x7f56b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.688+0000 7f56d4f7a640 1 --2- 172.21.15.83:0/2627932075 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f56b006b4d0 0x7f56b006d990 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f56c0006fd0 tx=0x7f56c0008040 comp rx=0 tx=0).stop 2024-03-06T07:32:28.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.688+0000 7f56d4f7a640 1 -- 172.21.15.83:0/2627932075 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56d015fa10 msgr2=0x7f56d015edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.688+0000 7f56d4f7a640 1 --2- 172.21.15.83:0/2627932075 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56d015fa10 0x7f56d015edb0 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7f56c8007ea0 tx=0x7f56c8007f90 comp rx=0 tx=0).stop 2024-03-06T07:32:28.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.688+0000 7f56d4f7a640 1 -- 172.21.15.83:0/2627932075 shutdown_connections 2024-03-06T07:32:28.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.688+0000 7f56d4f7a640 1 --2- 172.21.15.83:0/2627932075 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f56b006b4d0 0x7f56b006d990 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.688+0000 7f56d4f7a640 1 --2- 172.21.15.83:0/2627932075 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56d015fa10 0x7f56d015edb0 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.688+0000 7f56d4f7a640 1 --2- 172.21.15.83:0/2627932075 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f56d015bdd0 0x7f56d00b5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.688+0000 7f56d4f7a640 1 --2- 172.21.15.83:0/2627932075 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f56d00d16e0 0x7f56d009eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.688+0000 7f56d4f7a640 1 -- 172.21.15.83:0/2627932075 >> 172.21.15.83:0/2627932075 conn(0x7f56d009a9f0 msgr2=0x7f56d006f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:28.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.688+0000 7f56d4f7a640 1 -- 172.21.15.83:0/2627932075 shutdown_connections 2024-03-06T07:32:28.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:28.688+0000 7f56d4f7a640 1 -- 172.21.15.83:0/2627932075 wait complete. 2024-03-06T07:32:28.906 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:31:26.591282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:32:28.906 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:32:29.147 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:32:29.906 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:30.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.006+0000 7f67e3c9a640 1 Processor -- start 2024-03-06T07:32:30.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.007+0000 7f67e3c9a640 1 -- start start 2024-03-06T07:32:30.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.007+0000 7f67e3c9a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f67dc15dfe0 0x7f67dc0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.007+0000 7f67e3c9a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67dc0b3430 con 0x7f67dc0cf830 2024-03-06T07:32:30.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.007+0000 7f67e3c9a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67dc0b3710 con 0x7f67dc0d29c0 2024-03-06T07:32:30.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.007+0000 7f67e3c9a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67dc0b31e0 con 0x7f67dc15dfe0 2024-03-06T07:32:30.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.007+0000 7f67e1a36640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f67dc15dfe0 0x7f67dc0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.007+0000 7f67e1235640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f67dc0cf830 0x7f67dc0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:44498/0 (socket says 172.21.15.83:44498) 2024-03-06T07:32:30.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.007+0000 7f67e1235640 1 -- 172.21.15.83:0/2865863116 learned_addr learned my addr 172.21.15.83:0/2865863116 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:32:30.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.007+0000 7f67e1a36640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f67dc15dfe0 0x7f67dc0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58122/0 (socket says 172.21.15.83:58122) 2024-03-06T07:32:30.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.008+0000 7f67e1a36640 1 -- 172.21.15.83:0/2865863116 >> v1:172.21.15.183:6789/0 conn(0x7f67dc0d29c0 legacy=0x7f67dc0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.008+0000 7f67e1a36640 1 -- 172.21.15.83:0/2865863116 >> v1:172.21.15.83:6789/0 conn(0x7f67dc0cf830 legacy=0x7f67dc0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.008+0000 7f67e1a36640 1 -- 172.21.15.83:0/2865863116 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67dc0b39f0 con 0x7f67dc15dfe0 2024-03-06T07:32:30.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.008+0000 7f67e0a34640 1 -- 172.21.15.83:0/2865863116 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4016399037 0 0) 0x7f67dc0b3430 con 0x7f67dc0cf830 2024-03-06T07:32:30.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.009+0000 7f67e1a36640 1 --2- 172.21.15.83:0/2865863116 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67dc15dfe0 0x7f67dc0cf2f0 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f67dc0b3360 tx=0x7f67d802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d444bc7c88559bb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:30.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.009+0000 7f67e0a34640 1 -- 172.21.15.83:0/2865863116 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d8034070 con 0x7f67dc15dfe0 2024-03-06T07:32:30.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.009+0000 7f67e0a34640 1 -- 172.21.15.83:0/2865863116 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67d8030610 con 0x7f67dc15dfe0 2024-03-06T07:32:30.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.009+0000 7f67e0a34640 1 -- 172.21.15.83:0/2865863116 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d80309e0 con 0x7f67dc15dfe0 2024-03-06T07:32:30.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.010+0000 7f67e3c9a640 1 -- 172.21.15.83:0/2865863116 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67dc15dfe0 msgr2=0x7f67dc0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.010+0000 7f67e3c9a640 1 --2- 172.21.15.83:0/2865863116 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67dc15dfe0 0x7f67dc0cf2f0 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f67dc0b3360 tx=0x7f67d802b960 comp rx=0 tx=0).stop 2024-03-06T07:32:30.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.010+0000 7f67e3c9a640 1 -- 172.21.15.83:0/2865863116 shutdown_connections 2024-03-06T07:32:30.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.010+0000 7f67e3c9a640 1 --2- 172.21.15.83:0/2865863116 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67dc15dfe0 0x7f67dc0cf2f0 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.010+0000 7f67e3c9a640 1 -- 172.21.15.83:0/2865863116 >> 172.21.15.83:0/2865863116 conn(0x7f67dc0d3cd0 msgr2=0x7f67dc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:30.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.010+0000 7f67e3c9a640 1 -- 172.21.15.83:0/2865863116 shutdown_connections 2024-03-06T07:32:30.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.010+0000 7f67e3c9a640 1 -- 172.21.15.83:0/2865863116 wait complete. 2024-03-06T07:32:30.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.011+0000 7f67e3c9a640 1 Processor -- start 2024-03-06T07:32:30.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.011+0000 7f67e3c9a640 1 -- start start 2024-03-06T07:32:30.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.012+0000 7f67e3c9a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67dc0cf830 0x7f67dc0b8040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.012+0000 7f67e3c9a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f67dc0d29c0 0x7f67dc152130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.012+0000 7f67e3c9a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f67dc15dfe0 0x7f67dc0b1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.012+0000 7f67e3c9a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67dc15e3c0 con 0x7f67dc0cf830 2024-03-06T07:32:30.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.012+0000 7f67e3c9a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67dc0b3d60 con 0x7f67dc0d29c0 2024-03-06T07:32:30.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.012+0000 7f67e3c9a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67dc15e540 con 0x7f67dc15dfe0 2024-03-06T07:32:30.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.012+0000 7f67e1a36640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67dc0cf830 0x7f67dc0b8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.012+0000 7f67e1a36640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67dc0cf830 0x7f67dc0b8040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58126/0 (socket says 172.21.15.83:58126) 2024-03-06T07:32:30.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.012+0000 7f67e1a36640 1 -- 172.21.15.83:0/3957392821 learned_addr learned my addr 172.21.15.83:0/3957392821 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:30.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.012+0000 7f67e1235640 1 --2- 172.21.15.83:0/3957392821 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f67dc0d29c0 0x7f67dc152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.012+0000 7f67e2237640 1 --2- 172.21.15.83:0/3957392821 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f67dc15dfe0 0x7f67dc0b1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.012+0000 7f67e1a36640 1 -- 172.21.15.83:0/3957392821 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f67dc15dfe0 msgr2=0x7f67dc0b1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.012+0000 7f67e1a36640 1 --2- 172.21.15.83:0/3957392821 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f67dc15dfe0 0x7f67dc0b1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.012+0000 7f67e1a36640 1 -- 172.21.15.83:0/3957392821 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f67dc0d29c0 msgr2=0x7f67dc152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.012+0000 7f67e1a36640 1 --2- 172.21.15.83:0/3957392821 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f67dc0d29c0 0x7f67dc152130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.012+0000 7f67e1a36640 1 -- 172.21.15.83:0/3957392821 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67dc1f8030 con 0x7f67dc0cf830 2024-03-06T07:32:30.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.012+0000 7f67e1a36640 1 --2- 172.21.15.83:0/3957392821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67dc0cf830 0x7f67dc0b8040 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f67d8009890 tx=0x7f67d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:30.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.012+0000 7f67d27fc640 1 -- 172.21.15.83:0/3957392821 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d8034040 con 0x7f67dc0cf830 2024-03-06T07:32:30.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.012+0000 7f67d27fc640 1 -- 172.21.15.83:0/3957392821 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67d803ace0 con 0x7f67dc0cf830 2024-03-06T07:32:30.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.012+0000 7f67e3c9a640 1 -- 172.21.15.83:0/3957392821 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67dc1f73b0 con 0x7f67dc0cf830 2024-03-06T07:32:30.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.012+0000 7f67d27fc640 1 -- 172.21.15.83:0/3957392821 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d8044630 con 0x7f67dc0cf830 2024-03-06T07:32:30.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.012+0000 7f67e3c9a640 1 -- 172.21.15.83:0/3957392821 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67dc1f7970 con 0x7f67dc0cf830 2024-03-06T07:32:30.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.013+0000 7f67e3c9a640 1 -- 172.21.15.83:0/3957392821 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67dc04f330 con 0x7f67dc0cf830 2024-03-06T07:32:30.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.014+0000 7f67d27fc640 1 -- 172.21.15.83:0/3957392821 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f67d8037070 con 0x7f67dc0cf830 2024-03-06T07:32:30.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.014+0000 7f67d27fc640 1 --2- 172.21.15.83:0/3957392821 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f67c006b4d0 0x7f67c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.014+0000 7f67d27fc640 1 -- 172.21.15.83:0/3957392821 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f67d80ac1d0 con 0x7f67dc0cf830 2024-03-06T07:32:30.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.014+0000 7f67e1235640 1 --2- 172.21.15.83:0/3957392821 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f67c006b4d0 0x7f67c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.015+0000 7f67e1235640 1 --2- 172.21.15.83:0/3957392821 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f67c006b4d0 0x7f67c006d990 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f67cc000f60 tx=0x7f67cc007b90 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:30.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.018+0000 7f67d27fc640 1 -- 172.21.15.83:0/3957392821 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f67d803f070 con 0x7f67dc0cf830 2024-03-06T07:32:30.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.228+0000 7f67e3c9a640 1 -- 172.21.15.83:0/3957392821 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f67dc06bb70 con 0x7f67dc0cf830 2024-03-06T07:32:30.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.228+0000 7f67d27fc640 1 -- 172.21.15.83:0/3957392821 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f67d807b0a0 con 0x7f67dc0cf830 2024-03-06T07:32:30.230 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:32:30.230 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:32:30.230 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:32:30.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.230+0000 7f67e3c9a640 1 -- 172.21.15.83:0/3957392821 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f67c006b4d0 msgr2=0x7f67c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.230+0000 7f67e3c9a640 1 --2- 172.21.15.83:0/3957392821 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f67c006b4d0 0x7f67c006d990 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f67cc000f60 tx=0x7f67cc007b90 comp rx=0 tx=0).stop 2024-03-06T07:32:30.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.230+0000 7f67e3c9a640 1 -- 172.21.15.83:0/3957392821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67dc0cf830 msgr2=0x7f67dc0b8040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.230+0000 7f67e3c9a640 1 --2- 172.21.15.83:0/3957392821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67dc0cf830 0x7f67dc0b8040 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f67d8009890 tx=0x7f67d80046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.230+0000 7f67e3c9a640 1 -- 172.21.15.83:0/3957392821 shutdown_connections 2024-03-06T07:32:30.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.230+0000 7f67e3c9a640 1 --2- 172.21.15.83:0/3957392821 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f67c006b4d0 0x7f67c006d990 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.230+0000 7f67e3c9a640 1 --2- 172.21.15.83:0/3957392821 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f67dc15dfe0 0x7f67dc0b1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.230+0000 7f67e3c9a640 1 --2- 172.21.15.83:0/3957392821 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f67dc0d29c0 0x7f67dc152130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.230+0000 7f67e3c9a640 1 --2- 172.21.15.83:0/3957392821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67dc0cf830 0x7f67dc0b8040 unknown :-1 s=CLOSED pgs=1970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.230+0000 7f67e3c9a640 1 -- 172.21.15.83:0/3957392821 >> 172.21.15.83:0/3957392821 conn(0x7f67dc0d3cd0 msgr2=0x7f67dc0b4a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:30.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.230+0000 7f67e3c9a640 1 -- 172.21.15.83:0/3957392821 shutdown_connections 2024-03-06T07:32:30.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:30.230+0000 7f67e3c9a640 1 -- 172.21.15.83:0/3957392821 wait complete. 2024-03-06T07:32:30.444 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:31:26.591282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:32:30.444 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:32:31.116 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:31.119 DEBUG:teuthology.orchestra.run.smithi183:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:31.149 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:32:31.445 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:31.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.547+0000 7f2a2ae71640 1 Processor -- start 2024-03-06T07:32:31.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.547+0000 7f2a2ae71640 1 -- start start 2024-03-06T07:32:31.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.548+0000 7f2a2ae71640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2a240d22b0 0x7f2a240d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.549+0000 7f2a2ae71640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a240b3430 con 0x7f2a2415dfe0 2024-03-06T07:32:31.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.549+0000 7f2a2ae71640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a240b3710 con 0x7f2a240b41f0 2024-03-06T07:32:31.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.549+0000 7f2a2ae71640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a240d2c50 con 0x7f2a240d22b0 2024-03-06T07:32:31.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.549+0000 7f2a28c0d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f2a2415dfe0 0x7f2a240cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:44506/0 (socket says 172.21.15.83:44506) 2024-03-06T07:32:31.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.549+0000 7f2a28c0d640 1 -- 172.21.15.83:0/1973576786 learned_addr learned my addr 172.21.15.83:0/1973576786 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:32:31.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.549+0000 7f2a1bfff640 1 --2- 172.21.15.83:0/1973576786 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2a240d22b0 0x7f2a240d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.549+0000 7f2a1b7fe640 1 -- 172.21.15.83:0/1973576786 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3588545016 0 0) 0x7f2a240b3430 con 0x7f2a2415dfe0 2024-03-06T07:32:31.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.549+0000 7f2a1b7fe640 1 -- 172.21.15.83:0/1973576786 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a0c003520 con 0x7f2a2415dfe0 2024-03-06T07:32:31.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.549+0000 7f2a1b7fe640 1 -- 172.21.15.83:0/1973576786 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 367789685 0 0) 0x7f2a0c003520 con 0x7f2a2415dfe0 2024-03-06T07:32:31.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.549+0000 7f2a1b7fe640 1 -- 172.21.15.83:0/1973576786 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2a240d22b0 msgr2=0x7f2a240d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:31.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.549+0000 7f2a1b7fe640 1 --2- 172.21.15.83:0/1973576786 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2a240d22b0 0x7f2a240d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.549+0000 7f2a1b7fe640 1 -- 172.21.15.83:0/1973576786 >> v1:172.21.15.183:6789/0 conn(0x7f2a240b41f0 legacy=0x7f2a240b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.549+0000 7f2a1b7fe640 1 -- 172.21.15.83:0/1973576786 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a240b39f0 con 0x7f2a2415dfe0 2024-03-06T07:32:31.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.549+0000 7f2a1b7fe640 1 -- 172.21.15.83:0/1973576786 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f2a20001670 con 0x7f2a2415dfe0 2024-03-06T07:32:31.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.549+0000 7f2a1b7fe640 1 -- 172.21.15.83:0/1973576786 >> v1:172.21.15.83:6789/0 conn(0x7f2a2415dfe0 legacy=0x7f2a240cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.549+0000 7f2a1b7fe640 1 --2- 172.21.15.83:0/1973576786 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2a0c0039e0 0x7f2a0c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.549+0000 7f2a1b7fe640 1 --2- 172.21.15.83:0/1973576786 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2a240d22b0 0x7f2a0c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.549+0000 7f2a1b7fe640 1 --2- 172.21.15.83:0/1973576786 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a0c004d40 0x7f2a0c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.549+0000 7f2a1b7fe640 1 -- 172.21.15.83:0/1973576786 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0c00b9c0 con 0x7f2a0c004d40 2024-03-06T07:32:31.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.549+0000 7f2a1b7fe640 1 -- 172.21.15.83:0/1973576786 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0c00bb40 con 0x7f2a0c0039e0 2024-03-06T07:32:31.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.549+0000 7f2a1b7fe640 1 -- 172.21.15.83:0/1973576786 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0c00bd40 con 0x7f2a240d22b0 2024-03-06T07:32:31.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.549+0000 7f2a2940e640 1 --2- 172.21.15.83:0/1973576786 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a0c004d40 0x7f2a0c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.550+0000 7f2a1bfff640 1 --2- 172.21.15.83:0/1973576786 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2a240d22b0 0x7f2a0c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.550+0000 7f2a28c0d640 1 --2- 172.21.15.83:0/1973576786 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2a0c0039e0 0x7f2a0c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.550+0000 7f2a2940e640 1 -- 172.21.15.83:0/1973576786 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2a240d22b0 msgr2=0x7f2a0c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:31.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.550+0000 7f2a2940e640 1 --2- 172.21.15.83:0/1973576786 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2a240d22b0 0x7f2a0c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.550+0000 7f2a2940e640 1 -- 172.21.15.83:0/1973576786 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2a0c0039e0 msgr2=0x7f2a0c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:31.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.550+0000 7f2a2940e640 1 --2- 172.21.15.83:0/1973576786 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2a0c0039e0 0x7f2a0c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.550+0000 7f2a2940e640 1 -- 172.21.15.83:0/1973576786 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a0c003520 con 0x7f2a0c004d40 2024-03-06T07:32:31.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.551+0000 7f2a2940e640 1 --2- 172.21.15.83:0/1973576786 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a0c004d40 0x7f2a0c0051c0 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c001f60 tx=0x7f2a0c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f996eb6660228ca server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:31.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.551+0000 7f2a1b7fe640 1 -- 172.21.15.83:0/1973576786 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1c001b30 con 0x7f2a0c004d40 2024-03-06T07:32:31.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.551+0000 7f2a1b7fe640 1 -- 172.21.15.83:0/1973576786 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a1c02d710 con 0x7f2a0c004d40 2024-03-06T07:32:31.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.551+0000 7f2a1b7fe640 1 -- 172.21.15.83:0/1973576786 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1c02db00 con 0x7f2a0c004d40 2024-03-06T07:32:31.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.551+0000 7f2a2ae71640 1 -- 172.21.15.83:0/1973576786 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a0c004d40 msgr2=0x7f2a0c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.551+0000 7f2a2ae71640 1 --2- 172.21.15.83:0/1973576786 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a0c004d40 0x7f2a0c0051c0 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c001f60 tx=0x7f2a0c005830 comp rx=0 tx=0).stop 2024-03-06T07:32:31.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.552+0000 7f2a2940e640 1 -- 172.21.15.83:0/1973576786 reap_dead start 2024-03-06T07:32:31.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.552+0000 7f2a2ae71640 1 -- 172.21.15.83:0/1973576786 shutdown_connections 2024-03-06T07:32:31.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.552+0000 7f2a2ae71640 1 -- 172.21.15.83:0/1973576786 >> 172.21.15.83:0/1973576786 conn(0x7f2a240d3cd0 msgr2=0x7f2a240d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:31.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.552+0000 7f2a2ae71640 1 -- 172.21.15.83:0/1973576786 shutdown_connections 2024-03-06T07:32:31.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.552+0000 7f2a2ae71640 1 -- 172.21.15.83:0/1973576786 wait complete. 2024-03-06T07:32:31.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.553+0000 7f2a2ae71640 1 Processor -- start 2024-03-06T07:32:31.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.553+0000 7f2a2ae71640 1 -- start start 2024-03-06T07:32:31.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.554+0000 7f2a2ae71640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2a240ccf50 0x7f2a240cb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.554+0000 7f2a2ae71640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a240cd330 0x7f2a240cd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.554+0000 7f2a2ae71640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2a24156ec0 0x7f2a24157340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.554+0000 7f2a2ae71640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a240b3d60 con 0x7f2a240cd330 2024-03-06T07:32:31.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.554+0000 7f2a2ae71640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a240b31e0 con 0x7f2a24156ec0 2024-03-06T07:32:31.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.554+0000 7f2a2ae71640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a240d1c70 con 0x7f2a240ccf50 2024-03-06T07:32:31.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.554+0000 7f2a1bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a240cd330 0x7f2a240cd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.554+0000 7f2a1bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a240cd330 0x7f2a240cd7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58140/0 (socket says 172.21.15.83:58140) 2024-03-06T07:32:31.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.554+0000 7f2a1bfff640 1 -- 172.21.15.83:0/932376628 learned_addr learned my addr 172.21.15.83:0/932376628 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:31.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.554+0000 7f2a1bfff640 1 -- 172.21.15.83:0/932376628 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2a240ccf50 msgr2=0x7f2a240cb7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.554+0000 7f2a1bfff640 1 --2- 172.21.15.83:0/932376628 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2a240ccf50 0x7f2a240cb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.554+0000 7f2a1bfff640 1 -- 172.21.15.83:0/932376628 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2a24156ec0 msgr2=0x7f2a24157340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.554+0000 7f2a1bfff640 1 --2- 172.21.15.83:0/932376628 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2a24156ec0 0x7f2a24157340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.554+0000 7f2a1bfff640 1 -- 172.21.15.83:0/932376628 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a240cbeb0 con 0x7f2a240cd330 2024-03-06T07:32:31.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.554+0000 7f2a1bfff640 1 --2- 172.21.15.83:0/932376628 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a240cd330 0x7f2a240cd7b0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f2a140119a0 tx=0x7f2a14011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:31.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.554+0000 7f2a197fa640 1 -- 172.21.15.83:0/932376628 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1400fa70 con 0x7f2a240cd330 2024-03-06T07:32:31.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.554+0000 7f2a197fa640 1 -- 172.21.15.83:0/932376628 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a1400fc10 con 0x7f2a240cd330 2024-03-06T07:32:31.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.554+0000 7f2a197fa640 1 -- 172.21.15.83:0/932376628 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a14014440 con 0x7f2a240cd330 2024-03-06T07:32:31.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.554+0000 7f2a2ae71640 1 -- 172.21.15.83:0/932376628 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a240cc190 con 0x7f2a240cd330 2024-03-06T07:32:31.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.554+0000 7f2a2ae71640 1 -- 172.21.15.83:0/932376628 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a240b7da0 con 0x7f2a240cd330 2024-03-06T07:32:31.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.555+0000 7f2a2ae71640 1 -- 172.21.15.83:0/932376628 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a2404f330 con 0x7f2a240cd330 2024-03-06T07:32:31.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.556+0000 7f2a197fa640 1 -- 172.21.15.83:0/932376628 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2a14012040 con 0x7f2a240cd330 2024-03-06T07:32:31.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.556+0000 7f2a197fa640 1 --2- 172.21.15.83:0/932376628 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2a1006b4d0 0x7f2a1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.556+0000 7f2a197fa640 1 -- 172.21.15.83:0/932376628 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2a1408a910 con 0x7f2a240cd330 2024-03-06T07:32:31.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.556+0000 7f2a28c0d640 1 --2- 172.21.15.83:0/932376628 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2a1006b4d0 0x7f2a1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.557+0000 7f2a28c0d640 1 --2- 172.21.15.83:0/932376628 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2a1006b4d0 0x7f2a1006d990 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f2a20007d90 tx=0x7f2a20001470 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:31.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.561+0000 7f2a197fa640 1 -- 172.21.15.83:0/932376628 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2a14019030 con 0x7f2a240cd330 2024-03-06T07:32:31.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.773+0000 7f2a2ae71640 1 -- 172.21.15.83:0/932376628 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a2406bfe0 con 0x7f2a240cd330 2024-03-06T07:32:31.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.774+0000 7f2a197fa640 1 -- 172.21.15.83:0/932376628 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f2a14059750 con 0x7f2a240cd330 2024-03-06T07:32:31.776 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:32:31.776 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:32:31.776 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:32:31.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.776+0000 7f2a2ae71640 1 -- 172.21.15.83:0/932376628 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2a1006b4d0 msgr2=0x7f2a1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.776+0000 7f2a2ae71640 1 --2- 172.21.15.83:0/932376628 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2a1006b4d0 0x7f2a1006d990 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f2a20007d90 tx=0x7f2a20001470 comp rx=0 tx=0).stop 2024-03-06T07:32:31.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.776+0000 7f2a2ae71640 1 -- 172.21.15.83:0/932376628 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a240cd330 msgr2=0x7f2a240cd7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.776+0000 7f2a2ae71640 1 --2- 172.21.15.83:0/932376628 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a240cd330 0x7f2a240cd7b0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f2a140119a0 tx=0x7f2a14011e60 comp rx=0 tx=0).stop 2024-03-06T07:32:31.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.776+0000 7f2a2ae71640 1 -- 172.21.15.83:0/932376628 shutdown_connections 2024-03-06T07:32:31.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.776+0000 7f2a2ae71640 1 --2- 172.21.15.83:0/932376628 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f2a1006b4d0 0x7f2a1006d990 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.776+0000 7f2a2ae71640 1 --2- 172.21.15.83:0/932376628 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2a24156ec0 0x7f2a24157340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.776+0000 7f2a2ae71640 1 --2- 172.21.15.83:0/932376628 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a240cd330 0x7f2a240cd7b0 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.776+0000 7f2a2ae71640 1 --2- 172.21.15.83:0/932376628 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2a240ccf50 0x7f2a240cb7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.776+0000 7f2a2ae71640 1 -- 172.21.15.83:0/932376628 >> 172.21.15.83:0/932376628 conn(0x7f2a240d3cd0 msgr2=0x7f2a241603d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:31.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.776+0000 7f2a2ae71640 1 -- 172.21.15.83:0/932376628 shutdown_connections 2024-03-06T07:32:31.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:31.776+0000 7f2a2ae71640 1 -- 172.21.15.83:0/932376628 wait complete. 2024-03-06T07:32:31.990 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:31:26.591282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:32:31.990 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:32:32.991 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:33.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.094+0000 7f5f3a7ac640 1 Processor -- start 2024-03-06T07:32:33.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.094+0000 7f5f3a7ac640 1 -- start start 2024-03-06T07:32:33.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.094+0000 7f5f3a7ac640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f3415dfe0 0x7f5f340cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.094+0000 7f5f3a7ac640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f5f340d29c0 0x7f5f340b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.094+0000 7f5f3a7ac640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f340b3430 con 0x7f5f340cf830 2024-03-06T07:32:33.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.094+0000 7f5f3a7ac640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f340b31e0 con 0x7f5f340d29c0 2024-03-06T07:32:33.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.094+0000 7f5f3a7ac640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f340b3d60 con 0x7f5f3415dfe0 2024-03-06T07:32:33.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.095+0000 7f5f337fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f5f340cf830 0x7f5f340d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:44512/0 (socket says 172.21.15.83:44512) 2024-03-06T07:32:33.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.095+0000 7f5f337fe640 1 -- 172.21.15.83:0/4275794440 learned_addr learned my addr 172.21.15.83:0/4275794440 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:32:33.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.095+0000 7f5f33fff640 1 --2- 172.21.15.83:0/4275794440 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f3415dfe0 0x7f5f340cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.095+0000 7f5f38d49640 1 --2- 172.21.15.83:0/4275794440 >> v2:172.21.15.183:3300/0 conn(0x7f5f340d29c0 0x7f5f340b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.095+0000 7f5f32ffd640 1 -- 172.21.15.83:0/4275794440 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1299362524 0 0) 0x7f5f340b3430 con 0x7f5f340cf830 2024-03-06T07:32:33.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.095+0000 7f5f32ffd640 1 -- 172.21.15.83:0/4275794440 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f1c003520 con 0x7f5f340cf830 2024-03-06T07:32:33.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.095+0000 7f5f32ffd640 1 -- 172.21.15.83:0/4275794440 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4138171035 0 0) 0x7f5f1c003520 con 0x7f5f340cf830 2024-03-06T07:32:33.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.095+0000 7f5f32ffd640 1 -- 172.21.15.83:0/4275794440 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f3415dfe0 msgr2=0x7f5f340cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:33.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.095+0000 7f5f32ffd640 1 --2- 172.21.15.83:0/4275794440 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f3415dfe0 0x7f5f340cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.095+0000 7f5f32ffd640 1 -- 172.21.15.83:0/4275794440 >> v2:172.21.15.183:3300/0 conn(0x7f5f340d29c0 msgr2=0x7f5f340b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:33.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.095+0000 7f5f32ffd640 1 --2- 172.21.15.83:0/4275794440 >> v2:172.21.15.183:3300/0 conn(0x7f5f340d29c0 0x7f5f340b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.095+0000 7f5f32ffd640 1 -- 172.21.15.83:0/4275794440 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f340b3710 con 0x7f5f340cf830 2024-03-06T07:32:33.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.095+0000 7f5f32ffd640 1 -- 172.21.15.83:0/4275794440 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f5f28001670 con 0x7f5f340cf830 2024-03-06T07:32:33.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.095+0000 7f5f32ffd640 1 -- 172.21.15.83:0/4275794440 >> v1:172.21.15.83:6789/0 conn(0x7f5f340cf830 legacy=0x7f5f340d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.095+0000 7f5f32ffd640 1 --2- 172.21.15.83:0/4275794440 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f3415dfe0 0x7f5f1c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.095+0000 7f5f32ffd640 1 --2- 172.21.15.83:0/4275794440 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f1c003f30 0x7f5f1c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.095+0000 7f5f32ffd640 1 --2- 172.21.15.83:0/4275794440 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f1c004d40 0x7f5f1c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.095+0000 7f5f32ffd640 1 -- 172.21.15.83:0/4275794440 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f340b31e0 con 0x7f5f1c004d40 2024-03-06T07:32:33.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.095+0000 7f5f32ffd640 1 -- 172.21.15.83:0/4275794440 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f340b3d60 con 0x7f5f1c003f30 2024-03-06T07:32:33.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.095+0000 7f5f32ffd640 1 -- 172.21.15.83:0/4275794440 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1c00b9c0 con 0x7f5f3415dfe0 2024-03-06T07:32:33.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.095+0000 7f5f38d49640 1 --2- 172.21.15.83:0/4275794440 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f1c004d40 0x7f5f1c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.095+0000 7f5f33fff640 1 --2- 172.21.15.83:0/4275794440 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f3415dfe0 0x7f5f1c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.096+0000 7f5f337fe640 1 --2- 172.21.15.83:0/4275794440 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f1c003f30 0x7f5f1c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.096+0000 7f5f38d49640 1 -- 172.21.15.83:0/4275794440 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f3415dfe0 msgr2=0x7f5f1c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:33.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.096+0000 7f5f38d49640 1 --2- 172.21.15.83:0/4275794440 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f3415dfe0 0x7f5f1c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.096+0000 7f5f38d49640 1 -- 172.21.15.83:0/4275794440 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f1c003f30 msgr2=0x7f5f1c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:33.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.096+0000 7f5f38d49640 1 --2- 172.21.15.83:0/4275794440 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f1c003f30 0x7f5f1c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.096+0000 7f5f38d49640 1 -- 172.21.15.83:0/4275794440 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f1c003520 con 0x7f5f1c004d40 2024-03-06T07:32:33.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.096+0000 7f5f33fff640 1 --2- 172.21.15.83:0/4275794440 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f3415dfe0 0x7f5f1c0039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:33.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.096+0000 7f5f38d49640 1 --2- 172.21.15.83:0/4275794440 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f1c004d40 0x7f5f1c0051c0 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c009fc0 tx=0x7f5f1c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4aa8f6710af462d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.097+0000 7f5f32ffd640 1 -- 172.21.15.83:0/4275794440 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c039070 con 0x7f5f1c004d40 2024-03-06T07:32:33.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.097+0000 7f5f32ffd640 1 -- 172.21.15.83:0/4275794440 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f2c02cc40 con 0x7f5f1c004d40 2024-03-06T07:32:33.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.097+0000 7f5f32ffd640 1 -- 172.21.15.83:0/4275794440 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c009c10 con 0x7f5f1c004d40 2024-03-06T07:32:33.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.097+0000 7f5f3a7ac640 1 -- 172.21.15.83:0/4275794440 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f1c004d40 msgr2=0x7f5f1c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.097+0000 7f5f3a7ac640 1 --2- 172.21.15.83:0/4275794440 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f1c004d40 0x7f5f1c0051c0 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c009fc0 tx=0x7f5f1c005830 comp rx=0 tx=0).stop 2024-03-06T07:32:33.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.097+0000 7f5f38d49640 1 -- 172.21.15.83:0/4275794440 reap_dead start 2024-03-06T07:32:33.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.097+0000 7f5f3a7ac640 1 -- 172.21.15.83:0/4275794440 shutdown_connections 2024-03-06T07:32:33.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.097+0000 7f5f3a7ac640 1 -- 172.21.15.83:0/4275794440 >> 172.21.15.83:0/4275794440 conn(0x7f5f340d3cd0 msgr2=0x7f5f340d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:33.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.097+0000 7f5f3a7ac640 1 -- 172.21.15.83:0/4275794440 shutdown_connections 2024-03-06T07:32:33.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.097+0000 7f5f3a7ac640 1 -- 172.21.15.83:0/4275794440 wait complete. 2024-03-06T07:32:33.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.098+0000 7f5f3a7ac640 1 Processor -- start 2024-03-06T07:32:33.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.098+0000 7f5f3a7ac640 1 -- start start 2024-03-06T07:32:33.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.099+0000 7f5f3a7ac640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f3415bc10 0x7f5f340b8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.099+0000 7f5f3a7ac640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f3415c170 0x7f5f340b9200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.099+0000 7f5f3a7ac640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f340b68d0 0x7f5f341f7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.099+0000 7f5f3a7ac640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f340d1df0 con 0x7f5f3415bc10 2024-03-06T07:32:33.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.099+0000 7f5f3a7ac640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f340d1c70 con 0x7f5f3415c170 2024-03-06T07:32:33.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.099+0000 7f5f3a7ac640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3415e3c0 con 0x7f5f340b68d0 2024-03-06T07:32:33.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.099+0000 7f5f33fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f3415bc10 0x7f5f340b8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.099+0000 7f5f33fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f3415bc10 0x7f5f340b8cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58164/0 (socket says 172.21.15.83:58164) 2024-03-06T07:32:33.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.099+0000 7f5f33fff640 1 -- 172.21.15.83:0/3890687352 learned_addr learned my addr 172.21.15.83:0/3890687352 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:33.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.099+0000 7f5f337fe640 1 --2- 172.21.15.83:0/3890687352 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f3415c170 0x7f5f340b9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.099+0000 7f5f38d49640 1 --2- 172.21.15.83:0/3890687352 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f340b68d0 0x7f5f341f7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.099+0000 7f5f33fff640 1 -- 172.21.15.83:0/3890687352 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f340b68d0 msgr2=0x7f5f341f7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.099+0000 7f5f33fff640 1 --2- 172.21.15.83:0/3890687352 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f340b68d0 0x7f5f341f7380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.100+0000 7f5f33fff640 1 -- 172.21.15.83:0/3890687352 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f3415c170 msgr2=0x7f5f340b9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.100+0000 7f5f33fff640 1 --2- 172.21.15.83:0/3890687352 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f3415c170 0x7f5f340b9200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.100+0000 7f5f33fff640 1 -- 172.21.15.83:0/3890687352 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f340b6e90 con 0x7f5f3415bc10 2024-03-06T07:32:33.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.100+0000 7f5f33fff640 1 --2- 172.21.15.83:0/3890687352 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f3415bc10 0x7f5f340b8cc0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f5f24011600 tx=0x7f5f24011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.100+0000 7f5f30ff9640 1 -- 172.21.15.83:0/3890687352 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2400f920 con 0x7f5f3415bc10 2024-03-06T07:32:33.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.100+0000 7f5f30ff9640 1 -- 172.21.15.83:0/3890687352 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f2400fac0 con 0x7f5f3415bc10 2024-03-06T07:32:33.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.100+0000 7f5f3a7ac640 1 -- 172.21.15.83:0/3890687352 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f340b7170 con 0x7f5f3415bc10 2024-03-06T07:32:33.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.100+0000 7f5f30ff9640 1 -- 172.21.15.83:0/3890687352 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2400fe90 con 0x7f5f3415bc10 2024-03-06T07:32:33.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.100+0000 7f5f3a7ac640 1 -- 172.21.15.83:0/3890687352 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f340b7730 con 0x7f5f3415bc10 2024-03-06T07:32:33.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.101+0000 7f5f3a7ac640 1 -- 172.21.15.83:0/3890687352 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f3404f330 con 0x7f5f3415bc10 2024-03-06T07:32:33.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.101+0000 7f5f30ff9640 1 -- 172.21.15.83:0/3890687352 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5f24015460 con 0x7f5f3415bc10 2024-03-06T07:32:33.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.102+0000 7f5f30ff9640 1 --2- 172.21.15.83:0/3890687352 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5f2006b6a0 0x7f5f2006db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.102+0000 7f5f337fe640 1 --2- 172.21.15.83:0/3890687352 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5f2006b6a0 0x7f5f2006db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.102+0000 7f5f30ff9640 1 -- 172.21.15.83:0/3890687352 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5f2408abd0 con 0x7f5f3415bc10 2024-03-06T07:32:33.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.102+0000 7f5f337fe640 1 --2- 172.21.15.83:0/3890687352 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5f2006b6a0 0x7f5f2006db60 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f5f28000fb0 tx=0x7f5f28009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.106+0000 7f5f30ff9640 1 -- 172.21.15.83:0/3890687352 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5f24019030 con 0x7f5f3415bc10 2024-03-06T07:32:33.151 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:32:33.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.320+0000 7f5f3a7ac640 1 -- 172.21.15.83:0/3890687352 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f340b8aa0 con 0x7f5f3415bc10 2024-03-06T07:32:33.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.320+0000 7f5f30ff9640 1 -- 172.21.15.83:0/3890687352 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f5f24059a10 con 0x7f5f3415bc10 2024-03-06T07:32:33.323 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:32:33.323 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:32:33.323 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:32:33.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.322+0000 7f5f3a7ac640 1 -- 172.21.15.83:0/3890687352 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5f2006b6a0 msgr2=0x7f5f2006db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.322+0000 7f5f3a7ac640 1 --2- 172.21.15.83:0/3890687352 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5f2006b6a0 0x7f5f2006db60 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f5f28000fb0 tx=0x7f5f28009040 comp rx=0 tx=0).stop 2024-03-06T07:32:33.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.322+0000 7f5f3a7ac640 1 -- 172.21.15.83:0/3890687352 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f3415bc10 msgr2=0x7f5f340b8cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.322+0000 7f5f3a7ac640 1 --2- 172.21.15.83:0/3890687352 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f3415bc10 0x7f5f340b8cc0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f5f24011600 tx=0x7f5f24011ac0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.322+0000 7f5f3a7ac640 1 -- 172.21.15.83:0/3890687352 shutdown_connections 2024-03-06T07:32:33.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.322+0000 7f5f3a7ac640 1 --2- 172.21.15.83:0/3890687352 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f5f2006b6a0 0x7f5f2006db60 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.322+0000 7f5f3a7ac640 1 --2- 172.21.15.83:0/3890687352 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5f340b68d0 0x7f5f341f7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.322+0000 7f5f3a7ac640 1 --2- 172.21.15.83:0/3890687352 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f3415c170 0x7f5f340b9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.322+0000 7f5f3a7ac640 1 --2- 172.21.15.83:0/3890687352 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f3415bc10 0x7f5f340b8cc0 unknown :-1 s=CLOSED pgs=1976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.322+0000 7f5f3a7ac640 1 -- 172.21.15.83:0/3890687352 >> 172.21.15.83:0/3890687352 conn(0x7f5f340d3cd0 msgr2=0x7f5f3415d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:33.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.322+0000 7f5f3a7ac640 1 -- 172.21.15.83:0/3890687352 shutdown_connections 2024-03-06T07:32:33.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:33.322+0000 7f5f3a7ac640 1 -- 172.21.15.83:0/3890687352 wait complete. 2024-03-06T07:32:33.538 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:52.306698+0000', 'modified': '2024-03-06T07:31:26.591282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8631, 'mds_1': 8628}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8628': {'gid': 8628, 'name': 'c', 'rank': 1, 'incarnation': 75, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.83:6836/3544993654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3544993654}, {'type': 'v1', 'addr': '172.21.15.83:6836', 'nonce': 3544993654}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:54.013138+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8631': {'gid': 8631, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.83:6837/1401317546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6835', 'nonce': 1401317546}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 1401317546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8631, 'qdb_cluster': [8631]} 2024-03-06T07:32:33.538 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8628:up:stopping 2024-03-06T07:32:34.539 DEBUG:tasks.cephfs.filesystem:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 9, "epoch": 83, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 9, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:28:52.306698+0000", "damaged": [], "data_pools": [ 19 ], "enabled": true, "epoch": 83, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_8628": { "addr": "172.21.15.83:6836/3544993654", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6834", "nonce": 3544993654, "type": "v2" }, { "addr": "172.21.15.83:6836", "nonce": 3544993654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8628, "incarnation": 75, "join_fscid": -1, "laggy_since": "2024-03-06T07:29:54.013138+0000", "name": "c", "rank": 1, "state": "up:stopping", "state_seq": 7 }, "gid_8631": { "addr": "172.21.15.83:6837/1401317546", "addrs": { "addrvec": [ { "addr": "172.21.15.83:6835", "nonce": 1401317546, "type": "v2" }, { "addr": "172.21.15.83:6837", "nonce": 1401317546, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8631, "incarnation": 71, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 18, "modified": "2024-03-06T07:31:26.591282+0000", "qdb_cluster": [ 8631 ], "qdb_leader": 8631, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8631, "mds_1": 8628 } } } ], "standbys": [] } 2024-03-06T07:32:34.539 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:32:34.539 DEBUG:teuthology.orchestra.run.smithi083:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:32:34.587 DEBUG:teuthology.orchestra.run.smithi183:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:32:34.639 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:32:34.639 INFO:tasks.cephfs.mount:Terminating background process 2024-03-06T07:32:34.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:34 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-06T07:32:35.153 INFO:teuthology.orchestra.run.smithi083.stderr:open_dir_background: keeping dir open 2024-03-06T07:32:35.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:32:35.162 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:32:35.163 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:35.219 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:35.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.356+0000 7f9c4f5f7640 1 Processor -- start 2024-03-06T07:32:35.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.356+0000 7f9c4f5f7640 1 -- start start 2024-03-06T07:32:35.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.357+0000 7f9c4f5f7640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f9c4815dfe0 0x7f9c480cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.357+0000 7f9c4f5f7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9c480cf830 0x7f9c480d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.357+0000 7f9c4f5f7640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c480b3430 con 0x7f9c480d27f0 2024-03-06T07:32:35.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.357+0000 7f9c4f5f7640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c480b31e0 con 0x7f9c480cf830 2024-03-06T07:32:35.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.357+0000 7f9c4f5f7640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c480b3d60 con 0x7f9c4815dfe0 2024-03-06T07:32:35.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.357+0000 7f9c4cb92640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9c480cf830 0x7f9c480d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.357+0000 7f9c4cb92640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9c480cf830 0x7f9c480d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58180/0 (socket says 172.21.15.83:58180) 2024-03-06T07:32:35.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.357+0000 7f9c4cb92640 1 -- 172.21.15.83:0/2864690559 learned_addr learned my addr 172.21.15.83:0/2864690559 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:35.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.357+0000 7f9c4d393640 1 --2- 172.21.15.83:0/2864690559 >> v2:172.21.15.183:3300/0 conn(0x7f9c4815dfe0 0x7f9c480cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.357+0000 7f9c4cb92640 1 -- 172.21.15.83:0/2864690559 >> v2:172.21.15.183:3300/0 conn(0x7f9c4815dfe0 msgr2=0x7f9c480cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:35.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.357+0000 7f9c4cb92640 1 --2- 172.21.15.83:0/2864690559 >> v2:172.21.15.183:3300/0 conn(0x7f9c4815dfe0 0x7f9c480cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.357+0000 7f9c4cb92640 1 -- 172.21.15.83:0/2864690559 >> v1:172.21.15.183:6789/0 conn(0x7f9c480d27f0 legacy=0x7f9c480d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.357+0000 7f9c4cb92640 1 -- 172.21.15.83:0/2864690559 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c480b3710 con 0x7f9c480cf830 2024-03-06T07:32:35.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.358+0000 7f9c4cb92640 1 --2- 172.21.15.83:0/2864690559 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c480cf830 0x7f9c480d22b0 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f9c38009870 tx=0x7f9c3802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cfa86e813766546 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:35.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.358+0000 7f9c3ffff640 1 -- 172.21.15.83:0/2864690559 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c38035020 con 0x7f9c480cf830 2024-03-06T07:32:35.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.358+0000 7f9c3ffff640 1 -- 172.21.15.83:0/2864690559 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c38031870 con 0x7f9c480cf830 2024-03-06T07:32:35.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.358+0000 7f9c3ffff640 1 -- 172.21.15.83:0/2864690559 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c38031c40 con 0x7f9c480cf830 2024-03-06T07:32:35.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.359+0000 7f9c4f5f7640 1 -- 172.21.15.83:0/2864690559 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c480cf830 msgr2=0x7f9c480d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.359+0000 7f9c4f5f7640 1 --2- 172.21.15.83:0/2864690559 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c480cf830 0x7f9c480d22b0 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f9c38009870 tx=0x7f9c3802b910 comp rx=0 tx=0).stop 2024-03-06T07:32:35.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.359+0000 7f9c4f5f7640 1 -- 172.21.15.83:0/2864690559 shutdown_connections 2024-03-06T07:32:35.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.359+0000 7f9c4f5f7640 1 --2- 172.21.15.83:0/2864690559 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c480cf830 0x7f9c480d22b0 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.359+0000 7f9c4f5f7640 1 --2- 172.21.15.83:0/2864690559 >> v2:172.21.15.183:3300/0 conn(0x7f9c4815dfe0 0x7f9c480cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.359+0000 7f9c4f5f7640 1 -- 172.21.15.83:0/2864690559 >> 172.21.15.83:0/2864690559 conn(0x7f9c480d3cd0 msgr2=0x7f9c480d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:35.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.359+0000 7f9c4f5f7640 1 -- 172.21.15.83:0/2864690559 shutdown_connections 2024-03-06T07:32:35.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.359+0000 7f9c4f5f7640 1 -- 172.21.15.83:0/2864690559 wait complete. 2024-03-06T07:32:35.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.360+0000 7f9c4f5f7640 1 Processor -- start 2024-03-06T07:32:35.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.360+0000 7f9c4f5f7640 1 -- start start 2024-03-06T07:32:35.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.361+0000 7f9c4f5f7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c480cf830 0x7f9c4814df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.361+0000 7f9c4f5f7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c480d27f0 0x7f9c480b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.361+0000 7f9c4f5f7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c4815dfe0 0x7f9c480b8400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.361+0000 7f9c4f5f7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c480d1df0 con 0x7f9c4815dfe0 2024-03-06T07:32:35.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.361+0000 7f9c4f5f7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c480d1c70 con 0x7f9c480cf830 2024-03-06T07:32:35.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.361+0000 7f9c4f5f7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4815e3c0 con 0x7f9c480d27f0 2024-03-06T07:32:35.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.361+0000 7f9c4db94640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c4815dfe0 0x7f9c480b8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.361+0000 7f9c4db94640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c4815dfe0 0x7f9c480b8400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58190/0 (socket says 172.21.15.83:58190) 2024-03-06T07:32:35.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.361+0000 7f9c4db94640 1 -- 172.21.15.83:0/1144668038 learned_addr learned my addr 172.21.15.83:0/1144668038 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:35.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.361+0000 7f9c4cb92640 1 --2- 172.21.15.83:0/1144668038 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c480d27f0 0x7f9c480b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.361+0000 7f9c4d393640 1 --2- 172.21.15.83:0/1144668038 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c480cf830 0x7f9c4814df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.361+0000 7f9c4db94640 1 -- 172.21.15.83:0/1144668038 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c480d27f0 msgr2=0x7f9c480b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.361+0000 7f9c4db94640 1 --2- 172.21.15.83:0/1144668038 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c480d27f0 0x7f9c480b68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.361+0000 7f9c4db94640 1 -- 172.21.15.83:0/1144668038 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c480cf830 msgr2=0x7f9c4814df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.361+0000 7f9c4db94640 1 --2- 172.21.15.83:0/1144668038 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c480cf830 0x7f9c4814df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.361+0000 7f9c4db94640 1 -- 172.21.15.83:0/1144668038 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c4815e9c0 con 0x7f9c4815dfe0 2024-03-06T07:32:35.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.361+0000 7f9c4d393640 1 --2- 172.21.15.83:0/1144668038 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c480cf830 0x7f9c4814df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:35.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.361+0000 7f9c4db94640 1 --2- 172.21.15.83:0/1144668038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c4815dfe0 0x7f9c480b8400 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f9c4000c7d0 tx=0x7f9c4000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:35.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.361+0000 7f9c3dffb640 1 -- 172.21.15.83:0/1144668038 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c40016020 con 0x7f9c4815dfe0 2024-03-06T07:32:35.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.361+0000 7f9c3dffb640 1 -- 172.21.15.83:0/1144668038 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c4000aa50 con 0x7f9c4815dfe0 2024-03-06T07:32:35.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.361+0000 7f9c4f5f7640 1 -- 172.21.15.83:0/1144668038 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c481605b0 con 0x7f9c4815dfe0 2024-03-06T07:32:35.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.361+0000 7f9c3dffb640 1 -- 172.21.15.83:0/1144668038 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c4000ae40 con 0x7f9c4815dfe0 2024-03-06T07:32:35.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.362+0000 7f9c4f5f7640 1 -- 172.21.15.83:0/1144668038 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c4815ef40 con 0x7f9c4815dfe0 2024-03-06T07:32:35.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.362+0000 7f9c4f5f7640 1 -- 172.21.15.83:0/1144668038 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c4804f330 con 0x7f9c4815dfe0 2024-03-06T07:32:35.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.363+0000 7f9c3dffb640 1 -- 172.21.15.83:0/1144668038 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9c40019020 con 0x7f9c4815dfe0 2024-03-06T07:32:35.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.363+0000 7f9c3dffb640 1 --2- 172.21.15.83:0/1144668038 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9c2c06b4d0 0x7f9c2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.364+0000 7f9c4d393640 1 --2- 172.21.15.83:0/1144668038 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9c2c06b4d0 0x7f9c2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.364+0000 7f9c3dffb640 1 -- 172.21.15.83:0/1144668038 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9c40089be0 con 0x7f9c4815dfe0 2024-03-06T07:32:35.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.364+0000 7f9c4d393640 1 --2- 172.21.15.83:0/1144668038 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9c2c06b4d0 0x7f9c2c06d990 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f9c44006fd0 tx=0x7f9c44008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:35.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.368+0000 7f9c3dffb640 1 -- 172.21.15.83:0/1144668038 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9c40020020 con 0x7f9c4815dfe0 2024-03-06T07:32:35.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.547+0000 7f9c4f5f7640 1 -- 172.21.15.83:0/1144668038 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9c480b4290 con 0x7f9c4815dfe0 2024-03-06T07:32:35.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.548+0000 7f9c3dffb640 1 -- 172.21.15.83:0/1144668038 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v132) v1 ==== 74+0+16732 (secure 0 0 0) 0x7f9c40058a20 con 0x7f9c4815dfe0 2024-03-06T07:32:35.550 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:32:35.551 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":132,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:29:41.127522+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:50.508389+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:51.266049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.7","mappings":[{"from":2,"to":0}]},{"pgid":"19.13","mappings":[{"from":2,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:35.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.550+0000 7f9c4f5f7640 1 -- 172.21.15.83:0/1144668038 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9c2c06b4d0 msgr2=0x7f9c2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.550+0000 7f9c4f5f7640 1 --2- 172.21.15.83:0/1144668038 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9c2c06b4d0 0x7f9c2c06d990 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f9c44006fd0 tx=0x7f9c44008040 comp rx=0 tx=0).stop 2024-03-06T07:32:35.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.550+0000 7f9c4f5f7640 1 -- 172.21.15.83:0/1144668038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c4815dfe0 msgr2=0x7f9c480b8400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.550+0000 7f9c4f5f7640 1 --2- 172.21.15.83:0/1144668038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c4815dfe0 0x7f9c480b8400 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f9c4000c7d0 tx=0x7f9c4000cc90 comp rx=0 tx=0).stop 2024-03-06T07:32:35.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.550+0000 7f9c4f5f7640 1 -- 172.21.15.83:0/1144668038 shutdown_connections 2024-03-06T07:32:35.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.550+0000 7f9c4f5f7640 1 --2- 172.21.15.83:0/1144668038 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9c2c06b4d0 0x7f9c2c06d990 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.550+0000 7f9c4f5f7640 1 --2- 172.21.15.83:0/1144668038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c4815dfe0 0x7f9c480b8400 unknown :-1 s=CLOSED pgs=1978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.550+0000 7f9c4f5f7640 1 --2- 172.21.15.83:0/1144668038 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9c480d27f0 0x7f9c480b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.550+0000 7f9c4f5f7640 1 --2- 172.21.15.83:0/1144668038 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c480cf830 0x7f9c4814df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.550+0000 7f9c4f5f7640 1 -- 172.21.15.83:0/1144668038 >> 172.21.15.83:0/1144668038 conn(0x7f9c480d3cd0 msgr2=0x7f9c4806bb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:35.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.550+0000 7f9c4f5f7640 1 -- 172.21.15.83:0/1144668038 shutdown_connections 2024-03-06T07:32:35.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.550+0000 7f9c4f5f7640 1 -- 172.21.15.83:0/1144668038 wait complete. 2024-03-06T07:32:35.563 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:28:50.508389+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '127', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:28:51.266049+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '128', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:32:35.563 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:32:35.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.702+0000 7f7a9039a640 1 Processor -- start 2024-03-06T07:32:35.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.702+0000 7f7a9039a640 1 -- start start 2024-03-06T07:32:35.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.703+0000 7f7a9039a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f7a881614d0 0x7f7a880d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.703+0000 7f7a9039a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7a880d24b0 0x7f7a880b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.703+0000 7f7a9039a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a881618b0 con 0x7f7a880d1910 2024-03-06T07:32:35.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.703+0000 7f7a9039a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a880cf400 con 0x7f7a880d24b0 2024-03-06T07:32:35.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.703+0000 7f7a9039a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a880cf6a0 con 0x7f7a881614d0 2024-03-06T07:32:35.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.703+0000 7f7a8e937640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7a880d24b0 0x7f7a880b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.703+0000 7f7a8e937640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7a880d24b0 0x7f7a880b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58196/0 (socket says 172.21.15.83:58196) 2024-03-06T07:32:35.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.703+0000 7f7a8e937640 1 -- 172.21.15.83:0/854958851 learned_addr learned my addr 172.21.15.83:0/854958851 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:35.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.704+0000 7f7a8e136640 1 --2- 172.21.15.83:0/854958851 >> v2:172.21.15.183:3300/0 conn(0x7f7a881614d0 0x7f7a880d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.704+0000 7f7a8e937640 1 -- 172.21.15.83:0/854958851 >> v2:172.21.15.183:3300/0 conn(0x7f7a881614d0 msgr2=0x7f7a880d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:35.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.704+0000 7f7a8e937640 1 --2- 172.21.15.83:0/854958851 >> v2:172.21.15.183:3300/0 conn(0x7f7a881614d0 0x7f7a880d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.704+0000 7f7a8e937640 1 -- 172.21.15.83:0/854958851 >> v1:172.21.15.183:6789/0 conn(0x7f7a880d1910 legacy=0x7f7a880d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:35.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.704+0000 7f7a8e937640 1 -- 172.21.15.83:0/854958851 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a880cf980 con 0x7f7a880d24b0 2024-03-06T07:32:35.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.704+0000 7f7a8e937640 1 --2- 172.21.15.83:0/854958851 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a880d24b0 0x7f7a880b0e30 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f7a8808c470 tx=0x7f7a8002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6120afba4d60a5a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:35.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.704+0000 7f7a8d134640 1 -- 172.21.15.83:0/854958851 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8002ca80 con 0x7f7a880d24b0 2024-03-06T07:32:35.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.704+0000 7f7a8d134640 1 -- 172.21.15.83:0/854958851 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a8002cc20 con 0x7f7a880d24b0 2024-03-06T07:32:35.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.704+0000 7f7a8d134640 1 -- 172.21.15.83:0/854958851 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a800343f0 con 0x7f7a880d24b0 2024-03-06T07:32:35.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.705+0000 7f7a9039a640 1 -- 172.21.15.83:0/854958851 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a880d24b0 msgr2=0x7f7a880b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.705+0000 7f7a9039a640 1 --2- 172.21.15.83:0/854958851 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a880d24b0 0x7f7a880b0e30 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f7a8808c470 tx=0x7f7a8002bee0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.705+0000 7f7a9039a640 1 -- 172.21.15.83:0/854958851 shutdown_connections 2024-03-06T07:32:35.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.705+0000 7f7a9039a640 1 --2- 172.21.15.83:0/854958851 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a880d24b0 0x7f7a880b0e30 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.705+0000 7f7a9039a640 1 --2- 172.21.15.83:0/854958851 >> v2:172.21.15.183:3300/0 conn(0x7f7a881614d0 0x7f7a880d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.705+0000 7f7a9039a640 1 -- 172.21.15.83:0/854958851 >> 172.21.15.83:0/854958851 conn(0x7f7a880d3d40 msgr2=0x7f7a880d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:35.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.705+0000 7f7a9039a640 1 -- 172.21.15.83:0/854958851 shutdown_connections 2024-03-06T07:32:35.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.705+0000 7f7a9039a640 1 -- 172.21.15.83:0/854958851 wait complete. 2024-03-06T07:32:35.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.706+0000 7f7a9039a640 1 Processor -- start 2024-03-06T07:32:35.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.706+0000 7f7a9039a640 1 -- start start 2024-03-06T07:32:35.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.707+0000 7f7a9039a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a880d1910 0x7f7a8814aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.707+0000 7f7a9039a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7a880d24b0 0x7f7a880c7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.707+0000 7f7a9039a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7a881614d0 0x7f7a8814b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.707+0000 7f7a9039a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a880b1370 con 0x7f7a880d1910 2024-03-06T07:32:35.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.707+0000 7f7a9039a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a880b0910 con 0x7f7a880d24b0 2024-03-06T07:32:35.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.707+0000 7f7a9039a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a880b14f0 con 0x7f7a881614d0 2024-03-06T07:32:35.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.707+0000 7f7a8e136640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a880d1910 0x7f7a8814aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.707+0000 7f7a8e136640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a880d1910 0x7f7a8814aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58212/0 (socket says 172.21.15.83:58212) 2024-03-06T07:32:35.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.707+0000 7f7a8e136640 1 -- 172.21.15.83:0/1207185946 learned_addr learned my addr 172.21.15.83:0/1207185946 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:35.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.707+0000 7f7a8e136640 1 -- 172.21.15.83:0/1207185946 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7a881614d0 msgr2=0x7f7a8814b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.707+0000 7f7a8e136640 1 --2- 172.21.15.83:0/1207185946 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7a881614d0 0x7f7a8814b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.707+0000 7f7a8e136640 1 -- 172.21.15.83:0/1207185946 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7a880d24b0 msgr2=0x7f7a880c7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.707+0000 7f7a8d935640 1 --2- 172.21.15.83:0/1207185946 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7a880d24b0 0x7f7a880c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.707+0000 7f7a8e136640 1 --2- 172.21.15.83:0/1207185946 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7a880d24b0 0x7f7a880c7870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.707+0000 7f7a8e136640 1 -- 172.21.15.83:0/1207185946 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a880c7db0 con 0x7f7a880d1910 2024-03-06T07:32:35.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.707+0000 7f7a8e136640 1 --2- 172.21.15.83:0/1207185946 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a880d1910 0x7f7a8814aa20 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7f7a8400c820 tx=0x7f7a8400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:35.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.708+0000 7f7a7effd640 1 -- 172.21.15.83:0/1207185946 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a84015070 con 0x7f7a880d1910 2024-03-06T07:32:35.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.708+0000 7f7a7effd640 1 -- 172.21.15.83:0/1207185946 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a84004020 con 0x7f7a880d1910 2024-03-06T07:32:35.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.708+0000 7f7a9039a640 1 -- 172.21.15.83:0/1207185946 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a8814ddb0 con 0x7f7a880d1910 2024-03-06T07:32:35.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.708+0000 7f7a7effd640 1 -- 172.21.15.83:0/1207185946 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a840043f0 con 0x7f7a880d1910 2024-03-06T07:32:35.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.708+0000 7f7a9039a640 1 -- 172.21.15.83:0/1207185946 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a880c96c0 con 0x7f7a880d1910 2024-03-06T07:32:35.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.709+0000 7f7a9039a640 1 -- 172.21.15.83:0/1207185946 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a88068ff0 con 0x7f7a880d1910 2024-03-06T07:32:35.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.709+0000 7f7a7effd640 1 -- 172.21.15.83:0/1207185946 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7a84007830 con 0x7f7a880d1910 2024-03-06T07:32:35.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.710+0000 7f7a7effd640 1 --2- 172.21.15.83:0/1207185946 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7a6c06b4d0 0x7f7a6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.710+0000 7f7a7effd640 1 -- 172.21.15.83:0/1207185946 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7a8408a010 con 0x7f7a880d1910 2024-03-06T07:32:35.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.710+0000 7f7a8d935640 1 --2- 172.21.15.83:0/1207185946 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7a6c06b4d0 0x7f7a6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.710+0000 7f7a8d935640 1 --2- 172.21.15.83:0/1207185946 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7a6c06b4d0 0x7f7a6c06d990 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f7a881f71c0 tx=0x7f7a78009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:35.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.714+0000 7f7a7effd640 1 -- 172.21.15.83:0/1207185946 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7a84058e50 con 0x7f7a880d1910 2024-03-06T07:32:35.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.892+0000 7f7a9039a640 1 -- 172.21.15.83:0/1207185946 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7a8814b6b0 con 0x7f7a880d1910 2024-03-06T07:32:35.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.892+0000 7f7a7effd640 1 -- 172.21.15.83:0/1207185946 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v132) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7a84058750 con 0x7f7a880d1910 2024-03-06T07:32:35.894 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:32:35.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.894+0000 7f7a9039a640 1 -- 172.21.15.83:0/1207185946 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7a6c06b4d0 msgr2=0x7f7a6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.894+0000 7f7a9039a640 1 --2- 172.21.15.83:0/1207185946 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7a6c06b4d0 0x7f7a6c06d990 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f7a881f71c0 tx=0x7f7a78009040 comp rx=0 tx=0).stop 2024-03-06T07:32:35.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.894+0000 7f7a9039a640 1 -- 172.21.15.83:0/1207185946 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a880d1910 msgr2=0x7f7a8814aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.894+0000 7f7a9039a640 1 --2- 172.21.15.83:0/1207185946 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a880d1910 0x7f7a8814aa20 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7f7a8400c820 tx=0x7f7a8400cce0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.894+0000 7f7a9039a640 1 -- 172.21.15.83:0/1207185946 shutdown_connections 2024-03-06T07:32:35.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.894+0000 7f7a9039a640 1 --2- 172.21.15.83:0/1207185946 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7a6c06b4d0 0x7f7a6c06d990 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.894+0000 7f7a9039a640 1 --2- 172.21.15.83:0/1207185946 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7a881614d0 0x7f7a8814b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.894+0000 7f7a9039a640 1 --2- 172.21.15.83:0/1207185946 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7a880d24b0 0x7f7a880c7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.894+0000 7f7a9039a640 1 --2- 172.21.15.83:0/1207185946 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a880d1910 0x7f7a8814aa20 unknown :-1 s=CLOSED pgs=1980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.894+0000 7f7a9039a640 1 -- 172.21.15.83:0/1207185946 >> 172.21.15.83:0/1207185946 conn(0x7f7a880d3d40 msgr2=0x7f7a880d2920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:35.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.894+0000 7f7a9039a640 1 -- 172.21.15.83:0/1207185946 shutdown_connections 2024-03-06T07:32:35.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:35.894+0000 7f7a9039a640 1 -- 172.21.15.83:0/1207185946 wait complete. 2024-03-06T07:32:35.907 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:32:36.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.010+0000 7f1117d16640 1 Processor -- start 2024-03-06T07:32:36.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.011+0000 7f1117d16640 1 -- start start 2024-03-06T07:32:36.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.011+0000 7f1117d16640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11101614d0 0x7f11100d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.011+0000 7f1117d16640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f11100d2480 0x7f11100b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.011+0000 7f1117d16640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11101618b0 con 0x7f11100d1910 2024-03-06T07:32:36.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.011+0000 7f1117d16640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11100cf400 con 0x7f11100d2480 2024-03-06T07:32:36.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.011+0000 7f1117d16640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11100cf6a0 con 0x7f11101614d0 2024-03-06T07:32:36.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.012+0000 7f1115ab2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11101614d0 0x7f11100d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.012+0000 7f11162b3640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f11100d2480 0x7f11100b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.012+0000 7f11152b1640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f11100d1910 0x7f11100d1d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:38348/0 (socket says 172.21.15.83:38348) 2024-03-06T07:32:36.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.012+0000 7f11152b1640 1 -- 172.21.15.83:0/2338583451 learned_addr learned my addr 172.21.15.83:0/2338583451 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:32:36.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.012+0000 7f1115ab2640 1 -- 172.21.15.83:0/2338583451 >> v2:172.21.15.183:3300/0 conn(0x7f11100d2480 msgr2=0x7f11100b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:36.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.012+0000 7f1115ab2640 1 --2- 172.21.15.83:0/2338583451 >> v2:172.21.15.183:3300/0 conn(0x7f11100d2480 0x7f11100b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.012+0000 7f1115ab2640 1 -- 172.21.15.83:0/2338583451 >> v1:172.21.15.183:6789/0 conn(0x7f11100d1910 legacy=0x7f11100d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.012+0000 7f1115ab2640 1 -- 172.21.15.83:0/2338583451 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11100cf980 con 0x7f11101614d0 2024-03-06T07:32:36.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.012+0000 7f11162b3640 1 --2- 172.21.15.83:0/2338583451 >> v2:172.21.15.183:3300/0 conn(0x7f11100d2480 0x7f11100b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:36.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.013+0000 7f1115ab2640 1 --2- 172.21.15.83:0/2338583451 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11101614d0 0x7f11100d4860 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f11000097a0 tx=0x7f110002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57553f325dfc7c47 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:36.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.013+0000 7f1114ab0640 1 -- 172.21.15.83:0/2338583451 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1100035020 con 0x7f11101614d0 2024-03-06T07:32:36.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.013+0000 7f1114ab0640 1 -- 172.21.15.83:0/2338583451 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1100031830 con 0x7f11101614d0 2024-03-06T07:32:36.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.013+0000 7f1114ab0640 1 -- 172.21.15.83:0/2338583451 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1100031c00 con 0x7f11101614d0 2024-03-06T07:32:36.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.013+0000 7f1117d16640 1 -- 172.21.15.83:0/2338583451 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11101614d0 msgr2=0x7f11100d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.013+0000 7f1117d16640 1 --2- 172.21.15.83:0/2338583451 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11101614d0 0x7f11100d4860 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f11000097a0 tx=0x7f110002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.013+0000 7f1117d16640 1 -- 172.21.15.83:0/2338583451 shutdown_connections 2024-03-06T07:32:36.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.013+0000 7f1117d16640 1 --2- 172.21.15.83:0/2338583451 >> v2:172.21.15.183:3300/0 conn(0x7f11100d2480 0x7f11100b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.013+0000 7f1117d16640 1 --2- 172.21.15.83:0/2338583451 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11101614d0 0x7f11100d4860 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.013+0000 7f1117d16640 1 -- 172.21.15.83:0/2338583451 >> 172.21.15.83:0/2338583451 conn(0x7f11100d3d40 msgr2=0x7f11100d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:36.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.014+0000 7f1117d16640 1 -- 172.21.15.83:0/2338583451 shutdown_connections 2024-03-06T07:32:36.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.014+0000 7f1117d16640 1 -- 172.21.15.83:0/2338583451 wait complete. 2024-03-06T07:32:36.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.015+0000 7f1117d16640 1 Processor -- start 2024-03-06T07:32:36.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.015+0000 7f1117d16640 1 -- start start 2024-03-06T07:32:36.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.015+0000 7f1117d16640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11100d1910 0x7f11100ca6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.015+0000 7f1117d16640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11100d2480 0x7f11100b7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.015+0000 7f1117d16640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11101614d0 0x7f11100b7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.015+0000 7f1117d16640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11100b1370 con 0x7f11101614d0 2024-03-06T07:32:36.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.015+0000 7f1117d16640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11100b0910 con 0x7f11100d1910 2024-03-06T07:32:36.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.015+0000 7f1117d16640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11100b14f0 con 0x7f11100d2480 2024-03-06T07:32:36.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.015+0000 7f11162b3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11101614d0 0x7f11100b7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.015+0000 7f11162b3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11101614d0 0x7f11100b7a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58226/0 (socket says 172.21.15.83:58226) 2024-03-06T07:32:36.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.015+0000 7f11162b3640 1 -- 172.21.15.83:0/2994350677 learned_addr learned my addr 172.21.15.83:0/2994350677 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:36.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.016+0000 7f11152b1640 1 --2- 172.21.15.83:0/2994350677 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11100d2480 0x7f11100b7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.016+0000 7f1115ab2640 1 --2- 172.21.15.83:0/2994350677 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11100d1910 0x7f11100ca6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.016+0000 7f11162b3640 1 -- 172.21.15.83:0/2994350677 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11100d2480 msgr2=0x7f11100b7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.016+0000 7f11162b3640 1 --2- 172.21.15.83:0/2994350677 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11100d2480 0x7f11100b7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.016+0000 7f11162b3640 1 -- 172.21.15.83:0/2994350677 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11100d1910 msgr2=0x7f11100ca6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.016+0000 7f11162b3640 1 --2- 172.21.15.83:0/2994350677 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11100d1910 0x7f11100ca6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.016+0000 7f11162b3640 1 -- 172.21.15.83:0/2994350677 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11100b80a0 con 0x7f11101614d0 2024-03-06T07:32:36.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.016+0000 7f11162b3640 1 --2- 172.21.15.83:0/2994350677 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11101614d0 0x7f11100b7a80 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7f110800efb0 tx=0x7f110800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:36.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.016+0000 7f11067fc640 1 -- 172.21.15.83:0/2994350677 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1108018020 con 0x7f11101614d0 2024-03-06T07:32:36.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.016+0000 7f11067fc640 1 -- 172.21.15.83:0/2994350677 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1108014480 con 0x7f11101614d0 2024-03-06T07:32:36.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.016+0000 7f1117d16640 1 -- 172.21.15.83:0/2994350677 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11100b8380 con 0x7f11101614d0 2024-03-06T07:32:36.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.016+0000 7f11067fc640 1 -- 172.21.15.83:0/2994350677 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1108014850 con 0x7f11101614d0 2024-03-06T07:32:36.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.016+0000 7f1117d16640 1 -- 172.21.15.83:0/2994350677 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11100b8940 con 0x7f11101614d0 2024-03-06T07:32:36.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.017+0000 7f1117d16640 1 -- 172.21.15.83:0/2994350677 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11100ce170 con 0x7f11101614d0 2024-03-06T07:32:36.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.018+0000 7f11067fc640 1 -- 172.21.15.83:0/2994350677 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1108005d10 con 0x7f11101614d0 2024-03-06T07:32:36.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.018+0000 7f11067fc640 1 --2- 172.21.15.83:0/2994350677 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f10f406b4d0 0x7f10f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.018+0000 7f11067fc640 1 -- 172.21.15.83:0/2994350677 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f110808af10 con 0x7f11101614d0 2024-03-06T07:32:36.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.019+0000 7f1115ab2640 1 --2- 172.21.15.83:0/2994350677 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f10f406b4d0 0x7f10f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.019+0000 7f1115ab2640 1 --2- 172.21.15.83:0/2994350677 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f10f406b4d0 0x7f10f406d990 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f1100036000 tx=0x7f110003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:36.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.023+0000 7f11067fc640 1 -- 172.21.15.83:0/2994350677 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1108022030 con 0x7f11101614d0 2024-03-06T07:32:36.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.200+0000 7f1117d16640 1 -- 172.21.15.83:0/2994350677 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1110160cd0 con 0x7f11101614d0 2024-03-06T07:32:36.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.200+0000 7f11067fc640 1 -- 172.21.15.83:0/2994350677 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v132) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1108010040 con 0x7f11101614d0 2024-03-06T07:32:36.202 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-03-06T07:32:36.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.202+0000 7f1117d16640 1 -- 172.21.15.83:0/2994350677 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f10f406b4d0 msgr2=0x7f10f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.202+0000 7f1117d16640 1 --2- 172.21.15.83:0/2994350677 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f10f406b4d0 0x7f10f406d990 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f1100036000 tx=0x7f110003c040 comp rx=0 tx=0).stop 2024-03-06T07:32:36.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.202+0000 7f1117d16640 1 -- 172.21.15.83:0/2994350677 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11101614d0 msgr2=0x7f11100b7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.202+0000 7f1117d16640 1 --2- 172.21.15.83:0/2994350677 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11101614d0 0x7f11100b7a80 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7f110800efb0 tx=0x7f110800c490 comp rx=0 tx=0).stop 2024-03-06T07:32:36.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.203+0000 7f1117d16640 1 -- 172.21.15.83:0/2994350677 shutdown_connections 2024-03-06T07:32:36.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.203+0000 7f1117d16640 1 --2- 172.21.15.83:0/2994350677 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f10f406b4d0 0x7f10f406d990 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.203+0000 7f1117d16640 1 --2- 172.21.15.83:0/2994350677 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11101614d0 0x7f11100b7a80 unknown :-1 s=CLOSED pgs=1981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.203+0000 7f1117d16640 1 --2- 172.21.15.83:0/2994350677 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11100d2480 0x7f11100b7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.203+0000 7f1117d16640 1 --2- 172.21.15.83:0/2994350677 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11100d1910 0x7f11100ca6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.203+0000 7f1117d16640 1 -- 172.21.15.83:0/2994350677 >> 172.21.15.83:0/2994350677 conn(0x7f11100d3d40 msgr2=0x7f11100cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:36.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.203+0000 7f1117d16640 1 -- 172.21.15.83:0/2994350677 shutdown_connections 2024-03-06T07:32:36.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.203+0000 7f1117d16640 1 -- 172.21.15.83:0/2994350677 wait complete. 2024-03-06T07:32:36.215 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:32:36.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.321+0000 7fd99ef1e640 1 Processor -- start 2024-03-06T07:32:36.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.321+0000 7fd99ef1e640 1 -- start start 2024-03-06T07:32:36.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.321+0000 7fd99ef1e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fd9980d4860 0x7fd9980d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.321+0000 7fd99ef1e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9981618b0 con 0x7fd9980b0e30 2024-03-06T07:32:36.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.321+0000 7fd99ef1e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9980cf400 con 0x7fd9981614d0 2024-03-06T07:32:36.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.321+0000 7fd99ef1e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9980d4cf0 con 0x7fd9980d4860 2024-03-06T07:32:36.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.321+0000 7fd99d4bb640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fd9980b0e30 0x7fd9980b1260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:44518/0 (socket says 172.21.15.83:44518) 2024-03-06T07:32:36.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.321+0000 7fd99d4bb640 1 -- 172.21.15.83:0/218751015 learned_addr learned my addr 172.21.15.83:0/218751015 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:32:36.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.322+0000 7fd997fff640 1 --2- 172.21.15.83:0/218751015 >> v2:172.21.15.183:3300/0 conn(0x7fd9980d4860 0x7fd9980d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.322+0000 7fd9977fe640 1 -- 172.21.15.83:0/218751015 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 648805013 0 0) 0x7fd9981618b0 con 0x7fd9980b0e30 2024-03-06T07:32:36.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.322+0000 7fd9977fe640 1 -- 172.21.15.83:0/218751015 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd988003520 con 0x7fd9980b0e30 2024-03-06T07:32:36.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.322+0000 7fd9977fe640 1 -- 172.21.15.83:0/218751015 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2781932517 0 0) 0x7fd9980cf400 con 0x7fd9981614d0 2024-03-06T07:32:36.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.322+0000 7fd9977fe640 1 -- 172.21.15.83:0/218751015 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9981618b0 con 0x7fd9981614d0 2024-03-06T07:32:36.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.322+0000 7fd997fff640 1 -- 172.21.15.83:0/218751015 >> v1:172.21.15.183:6789/0 conn(0x7fd9981614d0 legacy=0x7fd9980d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.322+0000 7fd997fff640 1 -- 172.21.15.83:0/218751015 >> v1:172.21.15.83:6789/0 conn(0x7fd9980b0e30 legacy=0x7fd9980b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.322+0000 7fd997fff640 1 -- 172.21.15.83:0/218751015 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9980cf6a0 con 0x7fd9980d4860 2024-03-06T07:32:36.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.323+0000 7fd997fff640 1 --2- 172.21.15.83:0/218751015 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9980d4860 0x7fd9980d2020 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fd9840097a0 tx=0x7fd98402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13359799fa14c81 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:36.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.323+0000 7fd9977fe640 1 -- 172.21.15.83:0/218751015 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd984034070 con 0x7fd9980d4860 2024-03-06T07:32:36.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.323+0000 7fd9977fe640 1 -- 172.21.15.83:0/218751015 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd984030610 con 0x7fd9980d4860 2024-03-06T07:32:36.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.323+0000 7fd9977fe640 1 -- 172.21.15.83:0/218751015 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9840309c0 con 0x7fd9980d4860 2024-03-06T07:32:36.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.324+0000 7fd99ef1e640 1 -- 172.21.15.83:0/218751015 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9980d4860 msgr2=0x7fd9980d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.324+0000 7fd99ef1e640 1 --2- 172.21.15.83:0/218751015 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9980d4860 0x7fd9980d2020 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fd9840097a0 tx=0x7fd98402b960 comp rx=0 tx=0).stop 2024-03-06T07:32:36.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.324+0000 7fd99ef1e640 1 -- 172.21.15.83:0/218751015 shutdown_connections 2024-03-06T07:32:36.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.324+0000 7fd99ef1e640 1 --2- 172.21.15.83:0/218751015 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9980d4860 0x7fd9980d2020 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.324+0000 7fd99ef1e640 1 -- 172.21.15.83:0/218751015 >> 172.21.15.83:0/218751015 conn(0x7fd9980d3d40 msgr2=0x7fd9980d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:36.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.324+0000 7fd99ef1e640 1 -- 172.21.15.83:0/218751015 shutdown_connections 2024-03-06T07:32:36.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.324+0000 7fd99ef1e640 1 -- 172.21.15.83:0/218751015 wait complete. 2024-03-06T07:32:36.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.326+0000 7fd99ef1e640 1 Processor -- start 2024-03-06T07:32:36.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.326+0000 7fd99ef1e640 1 -- start start 2024-03-06T07:32:36.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.327+0000 7fd99ef1e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9980b0e30 0x7fd9980c7d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.327+0000 7fd99ef1e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9980d4860 0x7fd9981530a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.327+0000 7fd99ef1e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9981614d0 0x7fd9980b85f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.327+0000 7fd99ef1e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9980b0910 con 0x7fd9980b0e30 2024-03-06T07:32:36.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.327+0000 7fd99ef1e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9980cfc60 con 0x7fd9981614d0 2024-03-06T07:32:36.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.327+0000 7fd99ef1e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9980d2560 con 0x7fd9980d4860 2024-03-06T07:32:36.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.327+0000 7fd99ccba640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9980b0e30 0x7fd9980c7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.327+0000 7fd99ccba640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9980b0e30 0x7fd9980c7d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:58242/0 (socket says 172.21.15.83:58242) 2024-03-06T07:32:36.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.327+0000 7fd99ccba640 1 -- 172.21.15.83:0/792674120 learned_addr learned my addr 172.21.15.83:0/792674120 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:36.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.327+0000 7fd997fff640 1 --2- 172.21.15.83:0/792674120 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9980d4860 0x7fd9981530a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.327+0000 7fd99ccba640 1 -- 172.21.15.83:0/792674120 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9980d4860 msgr2=0x7fd9981530a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.327+0000 7fd99ccba640 1 --2- 172.21.15.83:0/792674120 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9980d4860 0x7fd9981530a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.327+0000 7fd99d4bb640 1 --2- 172.21.15.83:0/792674120 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9981614d0 0x7fd9980b85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.327+0000 7fd99ccba640 1 -- 172.21.15.83:0/792674120 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9981614d0 msgr2=0x7fd9980b85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.327+0000 7fd99ccba640 1 --2- 172.21.15.83:0/792674120 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9981614d0 0x7fd9980b85f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.327+0000 7fd99ccba640 1 -- 172.21.15.83:0/792674120 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9980c82c0 con 0x7fd9980b0e30 2024-03-06T07:32:36.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.327+0000 7fd997fff640 1 --2- 172.21.15.83:0/792674120 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9980d4860 0x7fd9981530a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:36.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.328+0000 7fd99ccba640 1 --2- 172.21.15.83:0/792674120 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9980b0e30 0x7fd9980c7d80 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7fd98c00c430 tx=0x7fd98c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:36.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.328+0000 7fd9957fa640 1 -- 172.21.15.83:0/792674120 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd98c017070 con 0x7fd9980b0e30 2024-03-06T07:32:36.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.328+0000 7fd9957fa640 1 -- 172.21.15.83:0/792674120 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd98c0052f0 con 0x7fd9980b0e30 2024-03-06T07:32:36.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.328+0000 7fd99ef1e640 1 -- 172.21.15.83:0/792674120 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9981535e0 con 0x7fd9980b0e30 2024-03-06T07:32:36.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.328+0000 7fd9957fa640 1 -- 172.21.15.83:0/792674120 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd98c003830 con 0x7fd9980b0e30 2024-03-06T07:32:36.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.328+0000 7fd99ef1e640 1 -- 172.21.15.83:0/792674120 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9980b9390 con 0x7fd9980b0e30 2024-03-06T07:32:36.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.329+0000 7fd99ef1e640 1 -- 172.21.15.83:0/792674120 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd998068ff0 con 0x7fd9980b0e30 2024-03-06T07:32:36.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.329+0000 7fd9957fa640 1 -- 172.21.15.83:0/792674120 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd98c01a070 con 0x7fd9980b0e30 2024-03-06T07:32:36.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.330+0000 7fd9957fa640 1 --2- 172.21.15.83:0/792674120 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd97c06b4d0 0x7fd97c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.330+0000 7fd997fff640 1 --2- 172.21.15.83:0/792674120 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd97c06b4d0 0x7fd97c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.330+0000 7fd9957fa640 1 -- 172.21.15.83:0/792674120 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd98c089580 con 0x7fd9980b0e30 2024-03-06T07:32:36.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.330+0000 7fd997fff640 1 --2- 172.21.15.83:0/792674120 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd97c06b4d0 0x7fd97c06d990 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fd984036000 tx=0x7fd98403b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:36.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.334+0000 7fd9957fa640 1 -- 172.21.15.83:0/792674120 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd9980c82c0 con 0x7fd9980b0e30 2024-03-06T07:32:36.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.507+0000 7fd99ef1e640 1 -- 172.21.15.83:0/792674120 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd9980ce9e0 con 0x7fd9980b0e30 2024-03-06T07:32:36.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.507+0000 7fd9957fa640 1 -- 172.21.15.83:0/792674120 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v132) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd98c058450 con 0x7fd9980b0e30 2024-03-06T07:32:36.509 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 64 2024-03-06T07:32:36.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.509+0000 7fd99ef1e640 1 -- 172.21.15.83:0/792674120 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd97c06b4d0 msgr2=0x7fd97c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.509+0000 7fd99ef1e640 1 --2- 172.21.15.83:0/792674120 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd97c06b4d0 0x7fd97c06d990 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fd984036000 tx=0x7fd98403b040 comp rx=0 tx=0).stop 2024-03-06T07:32:36.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.509+0000 7fd99ef1e640 1 -- 172.21.15.83:0/792674120 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9980b0e30 msgr2=0x7fd9980c7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.509+0000 7fd99ef1e640 1 --2- 172.21.15.83:0/792674120 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9980b0e30 0x7fd9980c7d80 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7fd98c00c430 tx=0x7fd98c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.509+0000 7fd99ef1e640 1 -- 172.21.15.83:0/792674120 shutdown_connections 2024-03-06T07:32:36.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.509+0000 7fd99ef1e640 1 --2- 172.21.15.83:0/792674120 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd97c06b4d0 0x7fd97c06d990 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.509+0000 7fd99ef1e640 1 --2- 172.21.15.83:0/792674120 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9981614d0 0x7fd9980b85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.509+0000 7fd99ef1e640 1 --2- 172.21.15.83:0/792674120 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9980d4860 0x7fd9981530a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.509+0000 7fd99ef1e640 1 --2- 172.21.15.83:0/792674120 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9980b0e30 0x7fd9980c7d80 unknown :-1 s=CLOSED pgs=1983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.509+0000 7fd99ef1e640 1 -- 172.21.15.83:0/792674120 >> 172.21.15.83:0/792674120 conn(0x7fd9980d3d40 msgr2=0x7fd99806c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:36.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.509+0000 7fd99ef1e640 1 -- 172.21.15.83:0/792674120 shutdown_connections 2024-03-06T07:32:36.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.509+0000 7fd99ef1e640 1 -- 172.21.15.83:0/792674120 wait complete. 2024-03-06T07:32:36.522 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:32:36.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.621+0000 7f8f4fb6b640 1 Processor -- start 2024-03-06T07:32:36.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.621+0000 7f8f4fb6b640 1 -- start start 2024-03-06T07:32:36.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.622+0000 7f8f4fb6b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f8f480b3240 0x7f8f480b23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.622+0000 7f8f4fb6b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8f480b2910 0x7f8f480d1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.622+0000 7f8f4fb6b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f4815e3f0 con 0x7f8f4815e010 2024-03-06T07:32:36.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.622+0000 7f8f4fb6b640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f480b1f40 con 0x7f8f480b3240 2024-03-06T07:32:36.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.622+0000 7f8f4fb6b640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f480b1c10 con 0x7f8f480b2910 2024-03-06T07:32:36.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.622+0000 7f8f4d106640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f8f480b3240 0x7f8f480b23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.622+0000 7f8f4d106640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f8f480b3240 0x7f8f480b23d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50460/0 (socket says 172.21.15.83:50460) 2024-03-06T07:32:36.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.622+0000 7f8f4d106640 1 -- 172.21.15.83:0/3033890141 learned_addr learned my addr 172.21.15.83:0/3033890141 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:36.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.622+0000 7f8f4e108640 1 --2- 172.21.15.83:0/3033890141 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8f480b2910 0x7f8f480d1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.623+0000 7f8f4d106640 1 -- 172.21.15.83:0/3033890141 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8f480b2910 msgr2=0x7f8f480d1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:36.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.623+0000 7f8f4d106640 1 --2- 172.21.15.83:0/3033890141 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8f480b2910 0x7f8f480d1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.623+0000 7f8f4d106640 1 -- 172.21.15.83:0/3033890141 >> v1:172.21.15.183:6789/0 conn(0x7f8f4815e010 legacy=0x7f8f480cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.623+0000 7f8f4d106640 1 -- 172.21.15.83:0/3033890141 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f4806d1b0 con 0x7f8f480b3240 2024-03-06T07:32:36.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.623+0000 7f8f4d106640 1 --2- 172.21.15.83:0/3033890141 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f480b3240 0x7f8f480b23d0 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f8f44009870 tx=0x7f8f4402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5367835494219013 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:36.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.623+0000 7f8f4c905640 1 -- 172.21.15.83:0/3033890141 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f44035020 con 0x7f8f480b3240 2024-03-06T07:32:36.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.624+0000 7f8f4c905640 1 -- 172.21.15.83:0/3033890141 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f44031640 con 0x7f8f480b3240 2024-03-06T07:32:36.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.624+0000 7f8f4c905640 1 -- 172.21.15.83:0/3033890141 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f4402d040 con 0x7f8f480b3240 2024-03-06T07:32:36.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.624+0000 7f8f4fb6b640 1 -- 172.21.15.83:0/3033890141 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f480b3240 msgr2=0x7f8f480b23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.624+0000 7f8f4fb6b640 1 --2- 172.21.15.83:0/3033890141 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f480b3240 0x7f8f480b23d0 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f8f44009870 tx=0x7f8f4402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.624+0000 7f8f4fb6b640 1 -- 172.21.15.83:0/3033890141 shutdown_connections 2024-03-06T07:32:36.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.624+0000 7f8f4fb6b640 1 --2- 172.21.15.83:0/3033890141 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8f480b2910 0x7f8f480d1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.624+0000 7f8f4fb6b640 1 --2- 172.21.15.83:0/3033890141 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f480b3240 0x7f8f480b23d0 unknown :-1 s=CLOSED pgs=1984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.624+0000 7f8f4fb6b640 1 -- 172.21.15.83:0/3033890141 >> 172.21.15.83:0/3033890141 conn(0x7f8f4809a9f0 msgr2=0x7f8f4809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:36.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.624+0000 7f8f4fb6b640 1 -- 172.21.15.83:0/3033890141 shutdown_connections 2024-03-06T07:32:36.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.624+0000 7f8f4fb6b640 1 -- 172.21.15.83:0/3033890141 wait complete. 2024-03-06T07:32:36.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.625+0000 7f8f4fb6b640 1 Processor -- start 2024-03-06T07:32:36.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.625+0000 7f8f4fb6b640 1 -- start start 2024-03-06T07:32:36.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.626+0000 7f8f4fb6b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8f4815e010 0x7f8f480b5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.626+0000 7f8f4fb6b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f481576a0 0x7f8f48153d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.626+0000 7f8f4fb6b640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8f481f3690 0x7f8f481f3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.626+0000 7f8f4fb6b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f480d2010 con 0x7f8f481576a0 2024-03-06T07:32:36.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.626+0000 7f8f4fb6b640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f4806ced0 con 0x7f8f481f3690 2024-03-06T07:32:36.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.626+0000 7f8f4fb6b640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f480d2190 con 0x7f8f4815e010 2024-03-06T07:32:36.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.626+0000 7f8f4d106640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f481576a0 0x7f8f48153d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.626+0000 7f8f4d106640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f481576a0 0x7f8f48153d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50476/0 (socket says 172.21.15.83:50476) 2024-03-06T07:32:36.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.626+0000 7f8f4d106640 1 -- 172.21.15.83:0/2588582231 learned_addr learned my addr 172.21.15.83:0/2588582231 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:36.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.626+0000 7f8f4e108640 1 --2- 172.21.15.83:0/2588582231 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8f481f3690 0x7f8f481f3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.626+0000 7f8f4d907640 1 --2- 172.21.15.83:0/2588582231 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8f4815e010 0x7f8f480b5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.626+0000 7f8f4d106640 1 -- 172.21.15.83:0/2588582231 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8f4815e010 msgr2=0x7f8f480b5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.626+0000 7f8f4d106640 1 --2- 172.21.15.83:0/2588582231 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8f4815e010 0x7f8f480b5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.626+0000 7f8f4d106640 1 -- 172.21.15.83:0/2588582231 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8f481f3690 msgr2=0x7f8f481f3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.626+0000 7f8f4d106640 1 --2- 172.21.15.83:0/2588582231 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8f481f3690 0x7f8f481f3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.627+0000 7f8f4d106640 1 -- 172.21.15.83:0/2588582231 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f480b63c0 con 0x7f8f481576a0 2024-03-06T07:32:36.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.627+0000 7f8f4d106640 1 --2- 172.21.15.83:0/2588582231 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f481576a0 0x7f8f48153d00 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f8f44000c00 tx=0x7f8f440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:36.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.627+0000 7f8f3e7fc640 1 -- 172.21.15.83:0/2588582231 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f44040070 con 0x7f8f481576a0 2024-03-06T07:32:36.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.627+0000 7f8f3e7fc640 1 -- 172.21.15.83:0/2588582231 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f440399f0 con 0x7f8f481576a0 2024-03-06T07:32:36.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.627+0000 7f8f4fb6b640 1 -- 172.21.15.83:0/2588582231 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f480b66a0 con 0x7f8f481576a0 2024-03-06T07:32:36.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.627+0000 7f8f3e7fc640 1 -- 172.21.15.83:0/2588582231 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f44035040 con 0x7f8f481576a0 2024-03-06T07:32:36.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.627+0000 7f8f4fb6b640 1 -- 172.21.15.83:0/2588582231 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f48151be0 con 0x7f8f481576a0 2024-03-06T07:32:36.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.628+0000 7f8f4fb6b640 1 -- 172.21.15.83:0/2588582231 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f48151a00 con 0x7f8f481576a0 2024-03-06T07:32:36.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.629+0000 7f8f3e7fc640 1 -- 172.21.15.83:0/2588582231 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8f4404b060 con 0x7f8f481576a0 2024-03-06T07:32:36.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.629+0000 7f8f3e7fc640 1 --2- 172.21.15.83:0/2588582231 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8f2c06b500 0x7f8f2c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.629+0000 7f8f3e7fc640 1 -- 172.21.15.83:0/2588582231 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8f440aba40 con 0x7f8f481576a0 2024-03-06T07:32:36.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.629+0000 7f8f4d907640 1 --2- 172.21.15.83:0/2588582231 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8f2c06b500 0x7f8f2c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.630+0000 7f8f4d907640 1 --2- 172.21.15.83:0/2588582231 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8f2c06b500 0x7f8f2c06d9c0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f8f38007fd0 tx=0x7f8f38009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:36.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.634+0000 7f8f3e7fc640 1 -- 172.21.15.83:0/2588582231 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8f48151a00 con 0x7f8f481576a0 2024-03-06T07:32:36.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.823+0000 7f8f4fb6b640 1 -- 172.21.15.83:0/2588582231 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f8f4804f330 con 0x7f8f481576a0 2024-03-06T07:32:36.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.824+0000 7f8f3e7fc640 1 -- 172.21.15.83:0/2588582231 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v132) v1 ==== 80+0+0 (secure 0 0 0) 0x7f8f4407a870 con 0x7f8f481576a0 2024-03-06T07:32:36.826 INFO:teuthology.orchestra.run.smithi083.stderr:listed 0 entries 2024-03-06T07:32:36.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.826+0000 7f8f4fb6b640 1 -- 172.21.15.83:0/2588582231 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8f2c06b500 msgr2=0x7f8f2c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.826+0000 7f8f4fb6b640 1 --2- 172.21.15.83:0/2588582231 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8f2c06b500 0x7f8f2c06d9c0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f8f38007fd0 tx=0x7f8f38009040 comp rx=0 tx=0).stop 2024-03-06T07:32:36.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.826+0000 7f8f4fb6b640 1 -- 172.21.15.83:0/2588582231 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f481576a0 msgr2=0x7f8f48153d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.826+0000 7f8f4fb6b640 1 --2- 172.21.15.83:0/2588582231 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f481576a0 0x7f8f48153d00 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f8f44000c00 tx=0x7f8f440046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.826+0000 7f8f4fb6b640 1 -- 172.21.15.83:0/2588582231 shutdown_connections 2024-03-06T07:32:36.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.826+0000 7f8f4fb6b640 1 --2- 172.21.15.83:0/2588582231 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8f2c06b500 0x7f8f2c06d9c0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.826+0000 7f8f4fb6b640 1 --2- 172.21.15.83:0/2588582231 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8f481f3690 0x7f8f481f3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.826+0000 7f8f4fb6b640 1 --2- 172.21.15.83:0/2588582231 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f481576a0 0x7f8f48153d00 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.826+0000 7f8f4fb6b640 1 --2- 172.21.15.83:0/2588582231 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8f4815e010 0x7f8f480b5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.826+0000 7f8f4fb6b640 1 -- 172.21.15.83:0/2588582231 >> 172.21.15.83:0/2588582231 conn(0x7f8f4809a9f0 msgr2=0x7f8f480ae890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:36.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.826+0000 7f8f4fb6b640 1 -- 172.21.15.83:0/2588582231 shutdown_connections 2024-03-06T07:32:36.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:36.826+0000 7f8f4fb6b640 1 -- 172.21.15.83:0/2588582231 wait complete. 2024-03-06T07:32:36.838 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi083.front.sepia.ceph.com... 2024-03-06T07:32:36.838 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:36.838 DEBUG:teuthology.orchestra.run.smithi083:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:32:36.863 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi083.stderr:ceph-fuse[79136]: fuse finished with error 0 and tester_r 0 2024-03-06T07:32:36.865 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi083.front.sepia.ceph.com 2024-03-06T07:32:36.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:36.865 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:32:37.032 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:32:37.033 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:37.051 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=287 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:32:37.051 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi183.front.sepia.ceph.com 2024-03-06T07:32:37.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:37.051 DEBUG:teuthology.orchestra.run.smithi183:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:32:37.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:32:37.108 INFO:teuthology.orchestra.run.smithi183.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:32:37.108 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:32:37.108 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:37.123 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1257 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:32:37.123 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi083.front.sepia.ceph.com 2024-03-06T07:32:37.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:37.123 DEBUG:teuthology.orchestra.run.smithi083:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:32:37.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:32:37.178 INFO:teuthology.orchestra.run.smithi083.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:32:37.178 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:32:37.179 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:37.196 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=287 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:32:37.196 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi183.front.sepia.ceph.com 2024-03-06T07:32:37.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:37.196 DEBUG:teuthology.orchestra.run.smithi183:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:32:37.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:32:37.253 INFO:teuthology.orchestra.run.smithi183.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:32:37.254 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:37.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.381+0000 7f84ef042640 1 Processor -- start 2024-03-06T07:32:37.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.381+0000 7f84ef042640 1 -- start start 2024-03-06T07:32:37.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.382+0000 7f84ef042640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f84e80b3060 0x7f84e815e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.382+0000 7f84ef042640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f84e815bf30 0x7f84e815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.382+0000 7f84ef042640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84e815f2d0 con 0x7f84e813dc10 2024-03-06T07:32:37.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.382+0000 7f84ef042640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84e80cf3f0 con 0x7f84e815bf30 2024-03-06T07:32:37.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.382+0000 7f84ef042640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84e815b210 con 0x7f84e80b3060 2024-03-06T07:32:37.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.382+0000 7f84e7fff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f84e80b3060 0x7f84e815e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.382+0000 7f84ed5df640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f84e815bf30 0x7f84e815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.382+0000 7f84ecdde640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f84e813dc10 0x7f84e80b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:46580/0 (socket says 172.21.15.83:46580) 2024-03-06T07:32:37.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.382+0000 7f84ecdde640 1 -- 172.21.15.83:0/883200311 learned_addr learned my addr 172.21.15.83:0/883200311 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:32:37.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.382+0000 7f84e7fff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f84e80b3060 0x7f84e815e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:37080/0 (socket says 172.21.15.83:37080) 2024-03-06T07:32:37.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.382+0000 7f84e77fe640 1 -- 172.21.15.83:0/883200311 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3342510233 0 0) 0x7f84e815f2d0 con 0x7f84e813dc10 2024-03-06T07:32:37.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.382+0000 7f84e77fe640 1 -- 172.21.15.83:0/883200311 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84d8003520 con 0x7f84e813dc10 2024-03-06T07:32:37.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.382+0000 7f84e7fff640 1 -- 172.21.15.83:0/883200311 >> v2:172.21.15.183:3300/0 conn(0x7f84e815bf30 msgr2=0x7f84e815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:37.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.382+0000 7f84e7fff640 1 --2- 172.21.15.83:0/883200311 >> v2:172.21.15.183:3300/0 conn(0x7f84e815bf30 0x7f84e815ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.382+0000 7f84e7fff640 1 -- 172.21.15.83:0/883200311 >> v1:172.21.15.183:6789/0 conn(0x7f84e813dc10 legacy=0x7f84e80b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.382+0000 7f84e7fff640 1 -- 172.21.15.83:0/883200311 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e815f580 con 0x7f84e80b3060 2024-03-06T07:32:37.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.382+0000 7f84ed5df640 1 --2- 172.21.15.83:0/883200311 >> v2:172.21.15.183:3300/0 conn(0x7f84e815bf30 0x7f84e815ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:37.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.383+0000 7f84e7fff640 1 --2- 172.21.15.83:0/883200311 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f84e80b3060 0x7f84e815e850 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f84d4006d10 tx=0x7f84d402fc50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d53cc9a33ec45efb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:37.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.383+0000 7f84e77fe640 1 -- 172.21.15.83:0/883200311 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d403d070 con 0x7f84e80b3060 2024-03-06T07:32:37.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.383+0000 7f84e77fe640 1 -- 172.21.15.83:0/883200311 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d4030a00 con 0x7f84e80b3060 2024-03-06T07:32:37.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.383+0000 7f84e77fe640 1 -- 172.21.15.83:0/883200311 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d4030dd0 con 0x7f84e80b3060 2024-03-06T07:32:37.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.384+0000 7f84ef042640 1 -- 172.21.15.83:0/883200311 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f84e80b3060 msgr2=0x7f84e815e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.384+0000 7f84ef042640 1 --2- 172.21.15.83:0/883200311 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f84e80b3060 0x7f84e815e850 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f84d4006d10 tx=0x7f84d402fc50 comp rx=0 tx=0).stop 2024-03-06T07:32:37.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.384+0000 7f84ef042640 1 -- 172.21.15.83:0/883200311 shutdown_connections 2024-03-06T07:32:37.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.384+0000 7f84ef042640 1 --2- 172.21.15.83:0/883200311 >> v2:172.21.15.183:3300/0 conn(0x7f84e815bf30 0x7f84e815ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.384+0000 7f84ef042640 1 --2- 172.21.15.83:0/883200311 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f84e80b3060 0x7f84e815e850 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.384+0000 7f84ef042640 1 -- 172.21.15.83:0/883200311 >> 172.21.15.83:0/883200311 conn(0x7f84e80cf8a0 msgr2=0x7f84e80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:37.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.384+0000 7f84ef042640 1 -- 172.21.15.83:0/883200311 shutdown_connections 2024-03-06T07:32:37.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.384+0000 7f84ef042640 1 -- 172.21.15.83:0/883200311 wait complete. 2024-03-06T07:32:37.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.385+0000 7f84ef042640 1 Processor -- start 2024-03-06T07:32:37.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.385+0000 7f84ef042640 1 -- start start 2024-03-06T07:32:37.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.386+0000 7f84ef042640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84e80b3060 0x7f84e81399b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.386+0000 7f84ef042640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84e813dc10 0x7f84e81542b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.386+0000 7f84ef042640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f84e815bf30 0x7f84e806d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.386+0000 7f84ef042640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e813dff0 con 0x7f84e80b3060 2024-03-06T07:32:37.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.386+0000 7f84ef042640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e815b730 con 0x7f84e813dc10 2024-03-06T07:32:37.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.386+0000 7f84ef042640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84e80cd7a0 con 0x7f84e815bf30 2024-03-06T07:32:37.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.386+0000 7f84ecdde640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84e80b3060 0x7f84e81399b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.386+0000 7f84ecdde640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84e80b3060 0x7f84e81399b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50492/0 (socket says 172.21.15.83:50492) 2024-03-06T07:32:37.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.386+0000 7f84ecdde640 1 -- 172.21.15.83:0/1622339405 learned_addr learned my addr 172.21.15.83:0/1622339405 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:37.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.386+0000 7f84e7fff640 1 --2- 172.21.15.83:0/1622339405 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84e813dc10 0x7f84e81542b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.386+0000 7f84ed5df640 1 --2- 172.21.15.83:0/1622339405 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f84e815bf30 0x7f84e806d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.387+0000 7f84ecdde640 1 -- 172.21.15.83:0/1622339405 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f84e815bf30 msgr2=0x7f84e806d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.387+0000 7f84ecdde640 1 --2- 172.21.15.83:0/1622339405 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f84e815bf30 0x7f84e806d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.387+0000 7f84ecdde640 1 -- 172.21.15.83:0/1622339405 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84e813dc10 msgr2=0x7f84e81542b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.387+0000 7f84ecdde640 1 --2- 172.21.15.83:0/1622339405 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84e813dc10 0x7f84e81542b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.387+0000 7f84ecdde640 1 -- 172.21.15.83:0/1622339405 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e8154c90 con 0x7f84e80b3060 2024-03-06T07:32:37.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.387+0000 7f84ecdde640 1 --2- 172.21.15.83:0/1622339405 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84e80b3060 0x7f84e81399b0 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f84dc00c460 tx=0x7f84dc00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:37.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.387+0000 7f84e57fa640 1 -- 172.21.15.83:0/1622339405 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84dc017070 con 0x7f84e80b3060 2024-03-06T07:32:37.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.387+0000 7f84e57fa640 1 -- 172.21.15.83:0/1622339405 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84dc005ce0 con 0x7f84e80b3060 2024-03-06T07:32:37.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.387+0000 7f84ef042640 1 -- 172.21.15.83:0/1622339405 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84e8154f70 con 0x7f84e80b3060 2024-03-06T07:32:37.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.387+0000 7f84e57fa640 1 -- 172.21.15.83:0/1622339405 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84dc0072f0 con 0x7f84e80b3060 2024-03-06T07:32:37.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.387+0000 7f84ef042640 1 -- 172.21.15.83:0/1622339405 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84e815c340 con 0x7f84e80b3060 2024-03-06T07:32:37.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.388+0000 7f84ef042640 1 -- 172.21.15.83:0/1622339405 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84e804f330 con 0x7f84e80b3060 2024-03-06T07:32:37.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.388+0000 7f84e57fa640 1 -- 172.21.15.83:0/1622339405 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f84dc01a070 con 0x7f84e80b3060 2024-03-06T07:32:37.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.389+0000 7f84e57fa640 1 --2- 172.21.15.83:0/1622339405 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f84cc06b500 0x7f84cc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.389+0000 7f84e7fff640 1 --2- 172.21.15.83:0/1622339405 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f84cc06b500 0x7f84cc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.389+0000 7f84e57fa640 1 -- 172.21.15.83:0/1622339405 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f84dc0894b0 con 0x7f84e80b3060 2024-03-06T07:32:37.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.389+0000 7f84e7fff640 1 --2- 172.21.15.83:0/1622339405 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f84cc06b500 0x7f84cc06d9c0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f84d403a000 tx=0x7f84d4034000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:37.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.397+0000 7f84e57fa640 1 -- 172.21.15.83:0/1622339405 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f84dc057d30 con 0x7f84e80b3060 2024-03-06T07:32:37.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.611+0000 7f84ef042640 1 -- 172.21.15.83:0/1622339405 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84e8155140 con 0x7f84e80b3060 2024-03-06T07:32:37.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.611+0000 7f84e57fa640 1 -- 172.21.15.83:0/1622339405 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f84dc05bdd0 con 0x7f84e80b3060 2024-03-06T07:32:37.613 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:32:37.613 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:32:37.614 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:32:37.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.613+0000 7f84ef042640 1 -- 172.21.15.83:0/1622339405 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f84cc06b500 msgr2=0x7f84cc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.613+0000 7f84ef042640 1 --2- 172.21.15.83:0/1622339405 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f84cc06b500 0x7f84cc06d9c0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f84d403a000 tx=0x7f84d4034000 comp rx=0 tx=0).stop 2024-03-06T07:32:37.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.613+0000 7f84ef042640 1 -- 172.21.15.83:0/1622339405 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84e80b3060 msgr2=0x7f84e81399b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.613+0000 7f84ef042640 1 --2- 172.21.15.83:0/1622339405 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84e80b3060 0x7f84e81399b0 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f84dc00c460 tx=0x7f84dc00c860 comp rx=0 tx=0).stop 2024-03-06T07:32:37.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.613+0000 7f84ef042640 1 -- 172.21.15.83:0/1622339405 shutdown_connections 2024-03-06T07:32:37.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.613+0000 7f84ef042640 1 --2- 172.21.15.83:0/1622339405 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f84cc06b500 0x7f84cc06d9c0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.613+0000 7f84ef042640 1 --2- 172.21.15.83:0/1622339405 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f84e815bf30 0x7f84e806d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.613+0000 7f84ef042640 1 --2- 172.21.15.83:0/1622339405 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84e813dc10 0x7f84e81542b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.613+0000 7f84ef042640 1 --2- 172.21.15.83:0/1622339405 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84e80b3060 0x7f84e81399b0 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.613+0000 7f84ef042640 1 -- 172.21.15.83:0/1622339405 >> 172.21.15.83:0/1622339405 conn(0x7f84e80cf8a0 msgr2=0x7f84e80cdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:37.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.613+0000 7f84ef042640 1 -- 172.21.15.83:0/1622339405 shutdown_connections 2024-03-06T07:32:37.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.613+0000 7f84ef042640 1 -- 172.21.15.83:0/1622339405 wait complete. 2024-03-06T07:32:37.627 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:37.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.726+0000 7fb1800eb640 1 Processor -- start 2024-03-06T07:32:37.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.726+0000 7fb1800eb640 1 -- start start 2024-03-06T07:32:37.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.727+0000 7fb1800eb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb17815dfe0 0x7fb1780cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.727+0000 7fb1800eb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb1780cf830 0x7fb1780d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.727+0000 7fb1800eb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1780b3430 con 0x7fb1780d27f0 2024-03-06T07:32:37.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.727+0000 7fb1800eb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1780d2c50 con 0x7fb1780cf830 2024-03-06T07:32:37.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.727+0000 7fb1800eb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1780b31e0 con 0x7fb17815dfe0 2024-03-06T07:32:37.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.727+0000 7fb17d686640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb1780cf830 0x7fb1780d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.727+0000 7fb17d686640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb1780cf830 0x7fb1780d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50508/0 (socket says 172.21.15.83:50508) 2024-03-06T07:32:37.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.727+0000 7fb17d686640 1 -- 172.21.15.83:0/244147421 learned_addr learned my addr 172.21.15.83:0/244147421 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:37.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.727+0000 7fb17e688640 1 --1- 172.21.15.83:0/244147421 >> v1:172.21.15.83:6789/0 conn(0x7fb1780d27f0 0x7fb1780d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57584/0 (socket says 172.21.15.83:57584) 2024-03-06T07:32:37.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.728+0000 7fb17de87640 1 --2- 172.21.15.83:0/244147421 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb17815dfe0 0x7fb1780cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.728+0000 7fb17d686640 1 -- 172.21.15.83:0/244147421 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb17815dfe0 msgr2=0x7fb1780cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:37.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.728+0000 7fb17d686640 1 --2- 172.21.15.83:0/244147421 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb17815dfe0 0x7fb1780cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.728+0000 7fb17d686640 1 -- 172.21.15.83:0/244147421 >> v1:172.21.15.83:6789/0 conn(0x7fb1780d27f0 legacy=0x7fb1780d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.728+0000 7fb17d686640 1 -- 172.21.15.83:0/244147421 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1780b3710 con 0x7fb1780cf830 2024-03-06T07:32:37.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.729+0000 7fb17d686640 1 --2- 172.21.15.83:0/244147421 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1780cf830 0x7fb1780d22b0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7fb16c009870 tx=0x7fb16c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f52b17cbeba61296 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:37.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.729+0000 7fb17ce85640 1 -- 172.21.15.83:0/244147421 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb16c034070 con 0x7fb1780cf830 2024-03-06T07:32:37.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.729+0000 7fb17ce85640 1 -- 172.21.15.83:0/244147421 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb16c030840 con 0x7fb1780cf830 2024-03-06T07:32:37.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.729+0000 7fb17ce85640 1 -- 172.21.15.83:0/244147421 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb16c030c10 con 0x7fb1780cf830 2024-03-06T07:32:37.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.729+0000 7fb1800eb640 1 -- 172.21.15.83:0/244147421 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1780cf830 msgr2=0x7fb1780d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.729+0000 7fb1800eb640 1 --2- 172.21.15.83:0/244147421 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1780cf830 0x7fb1780d22b0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7fb16c009870 tx=0x7fb16c02b980 comp rx=0 tx=0).stop 2024-03-06T07:32:37.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.730+0000 7fb1800eb640 1 -- 172.21.15.83:0/244147421 shutdown_connections 2024-03-06T07:32:37.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.730+0000 7fb1800eb640 1 --2- 172.21.15.83:0/244147421 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1780cf830 0x7fb1780d22b0 unknown :-1 s=CLOSED pgs=1988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.730+0000 7fb1800eb640 1 --2- 172.21.15.83:0/244147421 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb17815dfe0 0x7fb1780cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.730+0000 7fb1800eb640 1 -- 172.21.15.83:0/244147421 >> 172.21.15.83:0/244147421 conn(0x7fb1780d3cd0 msgr2=0x7fb1780d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:37.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.730+0000 7fb1800eb640 1 -- 172.21.15.83:0/244147421 shutdown_connections 2024-03-06T07:32:37.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.730+0000 7fb1800eb640 1 -- 172.21.15.83:0/244147421 wait complete. 2024-03-06T07:32:37.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.731+0000 7fb1800eb640 1 Processor -- start 2024-03-06T07:32:37.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.731+0000 7fb1800eb640 1 -- start start 2024-03-06T07:32:37.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.731+0000 7fb1800eb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1780cf830 0x7fb1781f7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.731+0000 7fb1800eb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1780d27f0 0x7fb178155210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.731+0000 7fb1800eb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb17815dfe0 0x7fb1780b2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.731+0000 7fb1800eb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1780d1c70 con 0x7fb1780cf830 2024-03-06T07:32:37.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.731+0000 7fb1800eb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1780b3d60 con 0x7fb1780d27f0 2024-03-06T07:32:37.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.732+0000 7fb1800eb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1780d1df0 con 0x7fb17815dfe0 2024-03-06T07:32:37.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.732+0000 7fb17de87640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1780cf830 0x7fb1781f7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.732+0000 7fb17de87640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1780cf830 0x7fb1781f7930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50514/0 (socket says 172.21.15.83:50514) 2024-03-06T07:32:37.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.732+0000 7fb17de87640 1 -- 172.21.15.83:0/4170287313 learned_addr learned my addr 172.21.15.83:0/4170287313 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:37.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.732+0000 7fb17de87640 1 -- 172.21.15.83:0/4170287313 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb17815dfe0 msgr2=0x7fb1780b2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.732+0000 7fb17de87640 1 --2- 172.21.15.83:0/4170287313 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb17815dfe0 0x7fb1780b2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.732+0000 7fb17de87640 1 -- 172.21.15.83:0/4170287313 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1780d27f0 msgr2=0x7fb178155210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.732+0000 7fb17de87640 1 --2- 172.21.15.83:0/4170287313 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1780d27f0 0x7fb178155210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.732+0000 7fb17de87640 1 -- 172.21.15.83:0/4170287313 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1781f7e70 con 0x7fb1780cf830 2024-03-06T07:32:37.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.732+0000 7fb17de87640 1 --2- 172.21.15.83:0/4170287313 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1780cf830 0x7fb1781f7930 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7fb16800d620 tx=0x7fb16800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:37.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.732+0000 7fb1767fc640 1 -- 172.21.15.83:0/4170287313 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb16800b620 con 0x7fb1780cf830 2024-03-06T07:32:37.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.732+0000 7fb1767fc640 1 -- 172.21.15.83:0/4170287313 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb168019070 con 0x7fb1780cf830 2024-03-06T07:32:37.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.732+0000 7fb1800eb640 1 -- 172.21.15.83:0/4170287313 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb178160620 con 0x7fb1780cf830 2024-03-06T07:32:37.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.732+0000 7fb1767fc640 1 -- 172.21.15.83:0/4170287313 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb16800b9e0 con 0x7fb1780cf830 2024-03-06T07:32:37.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.732+0000 7fb1800eb640 1 -- 172.21.15.83:0/4170287313 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1780b6ce0 con 0x7fb1780cf830 2024-03-06T07:32:37.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.733+0000 7fb1800eb640 1 -- 172.21.15.83:0/4170287313 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb17804f330 con 0x7fb1780cf830 2024-03-06T07:32:37.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.734+0000 7fb1767fc640 1 -- 172.21.15.83:0/4170287313 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb16800bb80 con 0x7fb1780cf830 2024-03-06T07:32:37.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.734+0000 7fb1767fc640 1 --2- 172.21.15.83:0/4170287313 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb16406b4d0 0x7fb16406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.734+0000 7fb17d686640 1 --2- 172.21.15.83:0/4170287313 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb16406b4d0 0x7fb16406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.734+0000 7fb1767fc640 1 -- 172.21.15.83:0/4170287313 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb16808a5e0 con 0x7fb1780cf830 2024-03-06T07:32:37.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.735+0000 7fb17d686640 1 --2- 172.21.15.83:0/4170287313 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb16406b4d0 0x7fb16406d990 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fb16c036000 tx=0x7fb16c03b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:37.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.739+0000 7fb1767fc640 1 -- 172.21.15.83:0/4170287313 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb1781f7e70 con 0x7fb1780cf830 2024-03-06T07:32:37.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.918+0000 7fb1800eb640 1 -- 172.21.15.83:0/4170287313 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb178155750 con 0x7fb1780cf830 2024-03-06T07:32:37.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.920+0000 7fb1767fc640 1 -- 172.21.15.83:0/4170287313 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v132) v1 ==== 74+0+16732 (secure 0 0 0) 0x7fb16805de20 con 0x7fb1780cf830 2024-03-06T07:32:37.922 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:32:37.922 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":132,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:29:41.127522+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:50.508389+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:51.266049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.7","mappings":[{"from":2,"to":0}]},{"pgid":"19.13","mappings":[{"from":2,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:37.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.922+0000 7fb1800eb640 1 -- 172.21.15.83:0/4170287313 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb16406b4d0 msgr2=0x7fb16406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.922+0000 7fb1800eb640 1 --2- 172.21.15.83:0/4170287313 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb16406b4d0 0x7fb16406d990 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fb16c036000 tx=0x7fb16c03b040 comp rx=0 tx=0).stop 2024-03-06T07:32:37.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.922+0000 7fb1800eb640 1 -- 172.21.15.83:0/4170287313 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1780cf830 msgr2=0x7fb1781f7930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.922+0000 7fb1800eb640 1 --2- 172.21.15.83:0/4170287313 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1780cf830 0x7fb1781f7930 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7fb16800d620 tx=0x7fb16800dae0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.922+0000 7fb1800eb640 1 -- 172.21.15.83:0/4170287313 shutdown_connections 2024-03-06T07:32:37.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.922+0000 7fb1800eb640 1 --2- 172.21.15.83:0/4170287313 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb16406b4d0 0x7fb16406d990 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.922+0000 7fb1800eb640 1 --2- 172.21.15.83:0/4170287313 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb17815dfe0 0x7fb1780b2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.922+0000 7fb1800eb640 1 --2- 172.21.15.83:0/4170287313 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1780d27f0 0x7fb178155210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.922+0000 7fb1800eb640 1 --2- 172.21.15.83:0/4170287313 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1780cf830 0x7fb1781f7930 unknown :-1 s=CLOSED pgs=1989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.922+0000 7fb1800eb640 1 -- 172.21.15.83:0/4170287313 >> 172.21.15.83:0/4170287313 conn(0x7fb1780d3cd0 msgr2=0x7fb1780cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:37.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.922+0000 7fb1800eb640 1 -- 172.21.15.83:0/4170287313 shutdown_connections 2024-03-06T07:32:37.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:37.922+0000 7fb1800eb640 1 -- 172.21.15.83:0/4170287313 wait complete. 2024-03-06T07:32:37.935 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:46.361929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:28:50.508389+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '127', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:28:51.266049+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '128', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:32:37.936 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:32:38.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.060+0000 7f909241c640 1 Processor -- start 2024-03-06T07:32:38.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.060+0000 7f909241c640 1 -- start start 2024-03-06T07:32:38.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.061+0000 7f909241c640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f908c1620e0 0x7f908c168d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.061+0000 7f909241c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f908c06cd20 con 0x7f908c1614f0 2024-03-06T07:32:38.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.061+0000 7f909241c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f908c06d000 con 0x7f908c06c860 2024-03-06T07:32:38.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.061+0000 7f909241c640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f908c06d2e0 con 0x7f908c1620e0 2024-03-06T07:32:38.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.061+0000 7f908bfff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f908c1614f0 0x7f908c0c77b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57590/0 (socket says 172.21.15.83:57590) 2024-03-06T07:32:38.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.061+0000 7f908bfff640 1 -- 172.21.15.83:0/3391769094 learned_addr learned my addr 172.21.15.83:0/3391769094 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:32:38.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.062+0000 7f90909b9640 1 --2- 172.21.15.83:0/3391769094 >> v2:172.21.15.183:3300/0 conn(0x7f908c1620e0 0x7f908c168d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.062+0000 7f908affd640 1 -- 172.21.15.83:0/3391769094 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3617932643 0 0) 0x7f908c06cd20 con 0x7f908c1614f0 2024-03-06T07:32:38.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.062+0000 7f908affd640 1 -- 172.21.15.83:0/3391769094 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9074003520 con 0x7f908c1614f0 2024-03-06T07:32:38.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.062+0000 7f908affd640 1 -- 172.21.15.83:0/3391769094 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 885423357 0 0) 0x7f9074003520 con 0x7f908c1614f0 2024-03-06T07:32:38.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.062+0000 7f908affd640 1 -- 172.21.15.83:0/3391769094 >> v2:172.21.15.183:3300/0 conn(0x7f908c1620e0 msgr2=0x7f908c168d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:38.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.062+0000 7f908affd640 1 --2- 172.21.15.83:0/3391769094 >> v2:172.21.15.183:3300/0 conn(0x7f908c1620e0 0x7f908c168d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.062+0000 7f908affd640 1 -- 172.21.15.83:0/3391769094 >> v1:172.21.15.183:6789/0 conn(0x7f908c06c860 legacy=0x7f908c1619d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.062+0000 7f908affd640 1 -- 172.21.15.83:0/3391769094 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f908c06d5c0 con 0x7f908c1614f0 2024-03-06T07:32:38.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.062+0000 7f908affd640 1 -- 172.21.15.83:0/3391769094 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f9080001670 con 0x7f908c1614f0 2024-03-06T07:32:38.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.062+0000 7f908affd640 1 -- 172.21.15.83:0/3391769094 >> v1:172.21.15.83:6789/0 conn(0x7f908c1614f0 legacy=0x7f908c0c77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.062+0000 7f908affd640 1 --2- 172.21.15.83:0/3391769094 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90740039b0 0x7f9074003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.062+0000 7f908affd640 1 --2- 172.21.15.83:0/3391769094 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90740043b0 0x7f9074004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.062+0000 7f908affd640 1 --2- 172.21.15.83:0/3391769094 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9074005120 0x7f907400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.062+0000 7f908affd640 1 -- 172.21.15.83:0/3391769094 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908c06d2e0 con 0x7f90740039b0 2024-03-06T07:32:38.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.062+0000 7f908affd640 1 -- 172.21.15.83:0/3391769094 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f907400c060 con 0x7f90740043b0 2024-03-06T07:32:38.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.062+0000 7f908affd640 1 -- 172.21.15.83:0/3391769094 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f907400c260 con 0x7f9074005120 2024-03-06T07:32:38.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.062+0000 7f908bfff640 1 --2- 172.21.15.83:0/3391769094 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90740039b0 0x7f9074003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.062+0000 7f90909b9640 1 --2- 172.21.15.83:0/3391769094 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9074005120 0x7f907400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.062+0000 7f908b7fe640 1 --2- 172.21.15.83:0/3391769094 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90740043b0 0x7f9074004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.062+0000 7f908bfff640 1 -- 172.21.15.83:0/3391769094 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9074005120 msgr2=0x7f907400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:38.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.062+0000 7f908bfff640 1 --2- 172.21.15.83:0/3391769094 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9074005120 0x7f907400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.062+0000 7f908bfff640 1 -- 172.21.15.83:0/3391769094 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90740043b0 msgr2=0x7f9074004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:38.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.062+0000 7f908bfff640 1 --2- 172.21.15.83:0/3391769094 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90740043b0 0x7f9074004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.062+0000 7f908bfff640 1 -- 172.21.15.83:0/3391769094 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9074003520 con 0x7f90740039b0 2024-03-06T07:32:38.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.062+0000 7f90909b9640 1 -- 172.21.15.83:0/3391769094 reap_dead start 2024-03-06T07:32:38.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.063+0000 7f908bfff640 1 --2- 172.21.15.83:0/3391769094 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90740039b0 0x7f9074003d90 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f908000c290 tx=0x7f9074005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0f4219173c29fcb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:38.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.063+0000 7f908affd640 1 -- 172.21.15.83:0/3391769094 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9080007d50 con 0x7f90740039b0 2024-03-06T07:32:38.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.063+0000 7f908affd640 1 -- 172.21.15.83:0/3391769094 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9080030070 con 0x7f90740039b0 2024-03-06T07:32:38.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.063+0000 7f908affd640 1 -- 172.21.15.83:0/3391769094 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9080005d70 con 0x7f90740039b0 2024-03-06T07:32:38.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.064+0000 7f909241c640 1 -- 172.21.15.83:0/3391769094 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90740039b0 msgr2=0x7f9074003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.064+0000 7f909241c640 1 --2- 172.21.15.83:0/3391769094 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90740039b0 0x7f9074003d90 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f908000c290 tx=0x7f9074005830 comp rx=0 tx=0).stop 2024-03-06T07:32:38.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.064+0000 7f909241c640 1 -- 172.21.15.83:0/3391769094 shutdown_connections 2024-03-06T07:32:38.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.064+0000 7f909241c640 1 --2- 172.21.15.83:0/3391769094 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90740039b0 0x7f9074003d90 unknown :-1 s=CLOSED pgs=1991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.064+0000 7f909241c640 1 -- 172.21.15.83:0/3391769094 >> 172.21.15.83:0/3391769094 conn(0x7f908c0d1b40 msgr2=0x7f908c0d1f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:38.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.064+0000 7f909241c640 1 -- 172.21.15.83:0/3391769094 shutdown_connections 2024-03-06T07:32:38.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.064+0000 7f909241c640 1 -- 172.21.15.83:0/3391769094 wait complete. 2024-03-06T07:32:38.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.065+0000 7f909241c640 1 Processor -- start 2024-03-06T07:32:38.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.065+0000 7f909241c640 1 -- start start 2024-03-06T07:32:38.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.066+0000 7f909241c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90740039b0 0x7f908c0cb600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.066+0000 7f909241c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f908c0c9620 0x7f908c1f77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.066+0000 7f909241c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f908c1f38b0 0x7f908c0cd140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.066+0000 7f909241c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908c1619d0 con 0x7f90740039b0 2024-03-06T07:32:38.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.066+0000 7f909241c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908c0d28f0 con 0x7f908c1f38b0 2024-03-06T07:32:38.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.066+0000 7f909241c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f908c161b50 con 0x7f908c0c9620 2024-03-06T07:32:38.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.066+0000 7f908bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90740039b0 0x7f908c0cb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.066+0000 7f908bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90740039b0 0x7f908c0cb600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50532/0 (socket says 172.21.15.83:50532) 2024-03-06T07:32:38.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.066+0000 7f908bfff640 1 -- 172.21.15.83:0/1608349531 learned_addr learned my addr 172.21.15.83:0/1608349531 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:38.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.066+0000 7f908b7fe640 1 --2- 172.21.15.83:0/1608349531 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f908c0c9620 0x7f908c1f77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.066+0000 7f90909b9640 1 --2- 172.21.15.83:0/1608349531 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f908c1f38b0 0x7f908c0cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.066+0000 7f908bfff640 1 -- 172.21.15.83:0/1608349531 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f908c0c9620 msgr2=0x7f908c1f77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.066+0000 7f908bfff640 1 --2- 172.21.15.83:0/1608349531 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f908c0c9620 0x7f908c1f77d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.066+0000 7f908bfff640 1 -- 172.21.15.83:0/1608349531 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f908c1f38b0 msgr2=0x7f908c0cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.066+0000 7f908bfff640 1 --2- 172.21.15.83:0/1608349531 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f908c1f38b0 0x7f908c0cd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.066+0000 7f908bfff640 1 -- 172.21.15.83:0/1608349531 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f908c1f4220 con 0x7f90740039b0 2024-03-06T07:32:38.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.066+0000 7f908b7fe640 1 --2- 172.21.15.83:0/1608349531 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f908c0c9620 0x7f908c1f77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:38.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.066+0000 7f90909b9640 1 --2- 172.21.15.83:0/1608349531 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f908c1f38b0 0x7f908c0cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:38.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.066+0000 7f908bfff640 1 --2- 172.21.15.83:0/1608349531 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90740039b0 0x7f908c0cb600 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f908000c290 tx=0x7f908002e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:38.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.066+0000 7f9088ff9640 1 -- 172.21.15.83:0/1608349531 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9080045070 con 0x7f90740039b0 2024-03-06T07:32:38.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.066+0000 7f9088ff9640 1 -- 172.21.15.83:0/1608349531 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9080004bf0 con 0x7f90740039b0 2024-03-06T07:32:38.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.066+0000 7f9088ff9640 1 -- 172.21.15.83:0/1608349531 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9080007d50 con 0x7f90740039b0 2024-03-06T07:32:38.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.066+0000 7f909241c640 1 -- 172.21.15.83:0/1608349531 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f908c1f4500 con 0x7f90740039b0 2024-03-06T07:32:38.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.067+0000 7f909241c640 1 -- 172.21.15.83:0/1608349531 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f908c1f7d10 con 0x7f90740039b0 2024-03-06T07:32:38.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.068+0000 7f909241c640 1 -- 172.21.15.83:0/1608349531 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f908c09fdf0 con 0x7f90740039b0 2024-03-06T07:32:38.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.069+0000 7f9088ff9640 1 -- 172.21.15.83:0/1608349531 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9080007070 con 0x7f90740039b0 2024-03-06T07:32:38.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.069+0000 7f9088ff9640 1 --2- 172.21.15.83:0/1608349531 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f907806b500 0x7f907806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.069+0000 7f9088ff9640 1 -- 172.21.15.83:0/1608349531 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f90800aaf40 con 0x7f90740039b0 2024-03-06T07:32:38.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.069+0000 7f908b7fe640 1 --2- 172.21.15.83:0/1608349531 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f907806b500 0x7f907806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.069+0000 7f908b7fe640 1 --2- 172.21.15.83:0/1608349531 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f907806b500 0x7f907806d9c0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f908c0cdd50 tx=0x7f907c007af0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:38.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.073+0000 7f9088ff9640 1 -- 172.21.15.83:0/1608349531 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9080079e10 con 0x7f90740039b0 2024-03-06T07:32:38.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.249+0000 7f909241c640 1 -- 172.21.15.83:0/1608349531 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f908c068ff0 con 0x7f90740039b0 2024-03-06T07:32:38.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.250+0000 7f9088ff9640 1 -- 172.21.15.83:0/1608349531 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v132) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9080079710 con 0x7f90740039b0 2024-03-06T07:32:38.252 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-03-06T07:32:38.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.252+0000 7f909241c640 1 -- 172.21.15.83:0/1608349531 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f907806b500 msgr2=0x7f907806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.252+0000 7f909241c640 1 --2- 172.21.15.83:0/1608349531 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f907806b500 0x7f907806d9c0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f908c0cdd50 tx=0x7f907c007af0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.252+0000 7f909241c640 1 -- 172.21.15.83:0/1608349531 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90740039b0 msgr2=0x7f908c0cb600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.252+0000 7f909241c640 1 --2- 172.21.15.83:0/1608349531 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90740039b0 0x7f908c0cb600 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f908000c290 tx=0x7f908002e070 comp rx=0 tx=0).stop 2024-03-06T07:32:38.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.252+0000 7f909241c640 1 -- 172.21.15.83:0/1608349531 shutdown_connections 2024-03-06T07:32:38.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.252+0000 7f909241c640 1 --2- 172.21.15.83:0/1608349531 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f907806b500 0x7f907806d9c0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.252+0000 7f909241c640 1 --2- 172.21.15.83:0/1608349531 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f908c1f38b0 0x7f908c0cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.252+0000 7f909241c640 1 --2- 172.21.15.83:0/1608349531 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f908c0c9620 0x7f908c1f77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.252+0000 7f909241c640 1 --2- 172.21.15.83:0/1608349531 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90740039b0 0x7f908c0cb600 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.252+0000 7f909241c640 1 -- 172.21.15.83:0/1608349531 >> 172.21.15.83:0/1608349531 conn(0x7f908c0d1b40 msgr2=0x7f908c0cc0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:38.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.252+0000 7f909241c640 1 -- 172.21.15.83:0/1608349531 shutdown_connections 2024-03-06T07:32:38.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.252+0000 7f909241c640 1 -- 172.21.15.83:0/1608349531 wait complete. 2024-03-06T07:32:38.265 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:32:38.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.368+0000 7fb13cb9a640 1 Processor -- start 2024-03-06T07:32:38.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.368+0000 7fb13cb9a640 1 -- start start 2024-03-06T07:32:38.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.369+0000 7fb13cb9a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb1380d10c0 0x7fb1380d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.369+0000 7fb13cb9a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fb1380d14a0 0x7fb13815e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.369+0000 7fb13cb9a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb13815eb00 0x7fb1380cae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.369+0000 7fb13cb9a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb13815c050 con 0x7fb1380d10c0 2024-03-06T07:32:38.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.369+0000 7fb13cb9a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1380d0060 con 0x7fb1380d14a0 2024-03-06T07:32:38.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.369+0000 7fb13cb9a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1380cfa50 con 0x7fb13815eb00 2024-03-06T07:32:38.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.369+0000 7fb13659c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb1380d10c0 0x7fb1380d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.369+0000 7fb13659c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb1380d10c0 0x7fb1380d04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50546/0 (socket says 172.21.15.83:50546) 2024-03-06T07:32:38.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.369+0000 7fb13659c640 1 -- 172.21.15.83:0/4053731063 learned_addr learned my addr 172.21.15.83:0/4053731063 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:38.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.369+0000 7fb136d9d640 1 --2- 172.21.15.83:0/4053731063 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb13815eb00 0x7fb1380cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.369+0000 7fb12ffff640 1 --2- 172.21.15.83:0/4053731063 >> v2:172.21.15.183:3300/0 conn(0x7fb1380d14a0 0x7fb13815e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.369+0000 7fb13659c640 1 -- 172.21.15.83:0/4053731063 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb13815eb00 msgr2=0x7fb1380cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:38.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.369+0000 7fb13659c640 1 --2- 172.21.15.83:0/4053731063 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb13815eb00 0x7fb1380cae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.369+0000 7fb13659c640 1 -- 172.21.15.83:0/4053731063 >> v2:172.21.15.183:3300/0 conn(0x7fb1380d14a0 msgr2=0x7fb13815e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:38.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.369+0000 7fb13659c640 1 --2- 172.21.15.83:0/4053731063 >> v2:172.21.15.183:3300/0 conn(0x7fb1380d14a0 0x7fb13815e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.369+0000 7fb13659c640 1 -- 172.21.15.83:0/4053731063 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1380d0a30 con 0x7fb1380d10c0 2024-03-06T07:32:38.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.371+0000 7fb13659c640 1 --2- 172.21.15.83:0/4053731063 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1380d10c0 0x7fb1380d04f0 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7fb13806f920 tx=0x7fb12802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1588959a1e096441 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:38.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.371+0000 7fb135d9b640 1 -- 172.21.15.83:0/4053731063 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb128035020 con 0x7fb1380d10c0 2024-03-06T07:32:38.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.371+0000 7fb135d9b640 1 -- 172.21.15.83:0/4053731063 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb128031870 con 0x7fb1380d10c0 2024-03-06T07:32:38.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.371+0000 7fb135d9b640 1 -- 172.21.15.83:0/4053731063 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb128031c40 con 0x7fb1380d10c0 2024-03-06T07:32:38.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.371+0000 7fb13cb9a640 1 -- 172.21.15.83:0/4053731063 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1380d10c0 msgr2=0x7fb1380d04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.371+0000 7fb13cb9a640 1 --2- 172.21.15.83:0/4053731063 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1380d10c0 0x7fb1380d04f0 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7fb13806f920 tx=0x7fb12802b910 comp rx=0 tx=0).stop 2024-03-06T07:32:38.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.371+0000 7fb13cb9a640 1 -- 172.21.15.83:0/4053731063 shutdown_connections 2024-03-06T07:32:38.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.371+0000 7fb13cb9a640 1 --2- 172.21.15.83:0/4053731063 >> v2:172.21.15.183:3300/0 conn(0x7fb1380d14a0 0x7fb13815e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.371+0000 7fb13cb9a640 1 --2- 172.21.15.83:0/4053731063 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb13815eb00 0x7fb1380cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.371+0000 7fb13cb9a640 1 --2- 172.21.15.83:0/4053731063 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1380d10c0 0x7fb1380d04f0 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.371+0000 7fb13cb9a640 1 -- 172.21.15.83:0/4053731063 >> 172.21.15.83:0/4053731063 conn(0x7fb1380b3800 msgr2=0x7fb1380b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:38.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.371+0000 7fb13cb9a640 1 -- 172.21.15.83:0/4053731063 shutdown_connections 2024-03-06T07:32:38.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.371+0000 7fb13cb9a640 1 -- 172.21.15.83:0/4053731063 wait complete. 2024-03-06T07:32:38.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.373+0000 7fb13cb9a640 1 Processor -- start 2024-03-06T07:32:38.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.373+0000 7fb13cb9a640 1 -- start start 2024-03-06T07:32:38.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.374+0000 7fb13cb9a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1380d10c0 0x7fb1380cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.374+0000 7fb13cb9a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1380d14a0 0x7fb1380b5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.374+0000 7fb13cb9a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb13815eb00 0x7fb1381f5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.374+0000 7fb13cb9a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13815f440 con 0x7fb1380d14a0 2024-03-06T07:32:38.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.375+0000 7fb13cb9a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13815f160 con 0x7fb1380d10c0 2024-03-06T07:32:38.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.375+0000 7fb13cb9a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1380cfcf0 con 0x7fb13815eb00 2024-03-06T07:32:38.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.375+0000 7fb12ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1380d14a0 0x7fb1380b5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.375+0000 7fb12ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1380d14a0 0x7fb1380b5f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50554/0 (socket says 172.21.15.83:50554) 2024-03-06T07:32:38.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.375+0000 7fb12ffff640 1 -- 172.21.15.83:0/4091572605 learned_addr learned my addr 172.21.15.83:0/4091572605 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:38.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.375+0000 7fb136d9d640 1 --2- 172.21.15.83:0/4091572605 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb13815eb00 0x7fb1381f5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.375+0000 7fb13659c640 1 --2- 172.21.15.83:0/4091572605 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1380d10c0 0x7fb1380cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.375+0000 7fb12ffff640 1 -- 172.21.15.83:0/4091572605 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb13815eb00 msgr2=0x7fb1381f5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.375+0000 7fb12ffff640 1 --2- 172.21.15.83:0/4091572605 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb13815eb00 0x7fb1381f5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.375+0000 7fb12ffff640 1 -- 172.21.15.83:0/4091572605 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1380d10c0 msgr2=0x7fb1380cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.375+0000 7fb12ffff640 1 --2- 172.21.15.83:0/4091572605 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1380d10c0 0x7fb1380cd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.375+0000 7fb12ffff640 1 -- 172.21.15.83:0/4091572605 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1380b64b0 con 0x7fb1380d14a0 2024-03-06T07:32:38.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.375+0000 7fb12ffff640 1 --2- 172.21.15.83:0/4091572605 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1380d14a0 0x7fb1380b5f70 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7fb12000d800 tx=0x7fb12000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:38.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.375+0000 7fb12effd640 1 -- 172.21.15.83:0/4091572605 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb120018030 con 0x7fb1380d14a0 2024-03-06T07:32:38.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.375+0000 7fb12effd640 1 -- 172.21.15.83:0/4091572605 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1200096a0 con 0x7fb1380d14a0 2024-03-06T07:32:38.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.375+0000 7fb12effd640 1 -- 172.21.15.83:0/4091572605 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb120009870 con 0x7fb1380d14a0 2024-03-06T07:32:38.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.375+0000 7fb13cb9a640 1 -- 172.21.15.83:0/4091572605 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1381f5760 con 0x7fb1380d14a0 2024-03-06T07:32:38.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.375+0000 7fb13cb9a640 1 -- 172.21.15.83:0/4091572605 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1380cc9e0 con 0x7fb1380d14a0 2024-03-06T07:32:38.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.376+0000 7fb13cb9a640 1 -- 172.21.15.83:0/4091572605 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1381618a0 con 0x7fb1380d14a0 2024-03-06T07:32:38.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.377+0000 7fb12effd640 1 -- 172.21.15.83:0/4091572605 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb12001b070 con 0x7fb1380d14a0 2024-03-06T07:32:38.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.377+0000 7fb12effd640 1 --2- 172.21.15.83:0/4091572605 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb11806b4d0 0x7fb11806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.377+0000 7fb13659c640 1 --2- 172.21.15.83:0/4091572605 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb11806b4d0 0x7fb11806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.377+0000 7fb12effd640 1 -- 172.21.15.83:0/4091572605 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb12008a330 con 0x7fb1380d14a0 2024-03-06T07:32:38.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.378+0000 7fb13659c640 1 --2- 172.21.15.83:0/4091572605 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb11806b4d0 0x7fb11806d990 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fb128036000 tx=0x7fb12803c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:38.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.381+0000 7fb12effd640 1 -- 172.21.15.83:0/4091572605 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb1381618a0 con 0x7fb1380d14a0 2024-03-06T07:32:38.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.558+0000 7fb13cb9a640 1 -- 172.21.15.83:0/4091572605 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb13809fe60 con 0x7fb1380d14a0 2024-03-06T07:32:38.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.558+0000 7fb12effd640 1 -- 172.21.15.83:0/4091572605 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v132) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb1380b64b0 con 0x7fb1380d14a0 2024-03-06T07:32:38.560 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 32 2024-03-06T07:32:38.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.560+0000 7fb13cb9a640 1 -- 172.21.15.83:0/4091572605 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb11806b4d0 msgr2=0x7fb11806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.560+0000 7fb13cb9a640 1 --2- 172.21.15.83:0/4091572605 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb11806b4d0 0x7fb11806d990 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fb128036000 tx=0x7fb12803c040 comp rx=0 tx=0).stop 2024-03-06T07:32:38.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.560+0000 7fb13cb9a640 1 -- 172.21.15.83:0/4091572605 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1380d14a0 msgr2=0x7fb1380b5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.560+0000 7fb13cb9a640 1 --2- 172.21.15.83:0/4091572605 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1380d14a0 0x7fb1380b5f70 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7fb12000d800 tx=0x7fb12000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.560+0000 7fb13cb9a640 1 -- 172.21.15.83:0/4091572605 shutdown_connections 2024-03-06T07:32:38.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.560+0000 7fb13cb9a640 1 --2- 172.21.15.83:0/4091572605 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb11806b4d0 0x7fb11806d990 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.560+0000 7fb13cb9a640 1 --2- 172.21.15.83:0/4091572605 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb13815eb00 0x7fb1381f5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.560+0000 7fb13cb9a640 1 --2- 172.21.15.83:0/4091572605 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1380d14a0 0x7fb1380b5f70 unknown :-1 s=CLOSED pgs=1994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.560+0000 7fb13cb9a640 1 --2- 172.21.15.83:0/4091572605 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1380d10c0 0x7fb1380cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.560+0000 7fb13cb9a640 1 -- 172.21.15.83:0/4091572605 >> 172.21.15.83:0/4091572605 conn(0x7fb1380b3800 msgr2=0x7fb138070130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:38.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.560+0000 7fb13cb9a640 1 -- 172.21.15.83:0/4091572605 shutdown_connections 2024-03-06T07:32:38.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.560+0000 7fb13cb9a640 1 -- 172.21.15.83:0/4091572605 wait complete. 2024-03-06T07:32:38.573 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:32:38.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.685+0000 7f063fbf7640 1 Processor -- start 2024-03-06T07:32:38.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.686+0000 7f063fbf7640 1 -- start start 2024-03-06T07:32:38.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.686+0000 7f063fbf7640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f06380b1e40 0x7f0638168b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.686+0000 7f063fbf7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06380d2730 con 0x7f06381614d0 2024-03-06T07:32:38.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.686+0000 7f063fbf7640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06381618b0 con 0x7f06380d4860 2024-03-06T07:32:38.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.686+0000 7f063fbf7640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06380d4cf0 con 0x7f06380b1e40 2024-03-06T07:32:38.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.687+0000 7f063d993640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f06381614d0 0x7f06380d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57600/0 (socket says 172.21.15.83:57600) 2024-03-06T07:32:38.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.687+0000 7f063d993640 1 -- 172.21.15.83:0/1410311157 learned_addr learned my addr 172.21.15.83:0/1410311157 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:32:38.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.687+0000 7f063e194640 1 --2- 172.21.15.83:0/1410311157 >> v2:172.21.15.183:3300/0 conn(0x7f06380b1e40 0x7f0638168b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.687+0000 7f063c991640 1 -- 172.21.15.83:0/1410311157 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3913395483 0 0) 0x7f06380d2730 con 0x7f06381614d0 2024-03-06T07:32:38.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.687+0000 7f063c991640 1 -- 172.21.15.83:0/1410311157 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0624003520 con 0x7f06381614d0 2024-03-06T07:32:38.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.687+0000 7f063c991640 1 -- 172.21.15.83:0/1410311157 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2723589159 0 0) 0x7f0624003520 con 0x7f06381614d0 2024-03-06T07:32:38.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.687+0000 7f063c991640 1 -- 172.21.15.83:0/1410311157 >> v2:172.21.15.183:3300/0 conn(0x7f06380b1e40 msgr2=0x7f0638168b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:38.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.687+0000 7f063c991640 1 --2- 172.21.15.83:0/1410311157 >> v2:172.21.15.183:3300/0 conn(0x7f06380b1e40 0x7f0638168b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.687+0000 7f063c991640 1 -- 172.21.15.83:0/1410311157 >> v1:172.21.15.183:6789/0 conn(0x7f06380d4860 legacy=0x7f06380d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.687+0000 7f063c991640 1 -- 172.21.15.83:0/1410311157 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06380cf400 con 0x7f06381614d0 2024-03-06T07:32:38.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.687+0000 7f063c991640 1 -- 172.21.15.83:0/1410311157 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f0630001670 con 0x7f06381614d0 2024-03-06T07:32:38.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.687+0000 7f063c991640 1 -- 172.21.15.83:0/1410311157 >> v1:172.21.15.83:6789/0 conn(0x7f06381614d0 legacy=0x7f06380d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.687+0000 7f063c991640 1 --2- 172.21.15.83:0/1410311157 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06240039b0 0x7f0624003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.687+0000 7f063c991640 1 --2- 172.21.15.83:0/1410311157 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0624004800 0x7f0624004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.687+0000 7f063c991640 1 --2- 172.21.15.83:0/1410311157 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0624005210 0x7f062400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.687+0000 7f063c991640 1 -- 172.21.15.83:0/1410311157 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06380d4cf0 con 0x7f06240039b0 2024-03-06T07:32:38.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.687+0000 7f063c991640 1 -- 172.21.15.83:0/1410311157 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f062400c140 con 0x7f0624005210 2024-03-06T07:32:38.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.687+0000 7f063c991640 1 -- 172.21.15.83:0/1410311157 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f062400c340 con 0x7f0624004800 2024-03-06T07:32:38.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.687+0000 7f063d993640 1 --2- 172.21.15.83:0/1410311157 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06240039b0 0x7f0624003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.687+0000 7f063d192640 1 --2- 172.21.15.83:0/1410311157 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0624004800 0x7f0624004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.687+0000 7f063e194640 1 --2- 172.21.15.83:0/1410311157 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0624005210 0x7f062400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.688+0000 7f063d993640 1 -- 172.21.15.83:0/1410311157 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0624004800 msgr2=0x7f0624004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:38.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.688+0000 7f063d993640 1 --2- 172.21.15.83:0/1410311157 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0624004800 0x7f0624004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.688+0000 7f063d993640 1 -- 172.21.15.83:0/1410311157 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0624005210 msgr2=0x7f062400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:38.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.688+0000 7f063d993640 1 --2- 172.21.15.83:0/1410311157 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0624005210 0x7f062400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.688+0000 7f063d993640 1 -- 172.21.15.83:0/1410311157 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0624003520 con 0x7f06240039b0 2024-03-06T07:32:38.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.688+0000 7f063e194640 1 -- 172.21.15.83:0/1410311157 reap_dead start 2024-03-06T07:32:38.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.689+0000 7f063d993640 1 --2- 172.21.15.83:0/1410311157 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06240039b0 0x7f0624003df0 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f063000c210 tx=0x7f0624005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26cbd158030b3688 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:38.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.689+0000 7f063c991640 1 -- 172.21.15.83:0/1410311157 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0630007d30 con 0x7f06240039b0 2024-03-06T07:32:38.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.689+0000 7f063c991640 1 -- 172.21.15.83:0/1410311157 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0630030070 con 0x7f06240039b0 2024-03-06T07:32:38.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.689+0000 7f063c991640 1 -- 172.21.15.83:0/1410311157 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0630005d70 con 0x7f06240039b0 2024-03-06T07:32:38.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.689+0000 7f063fbf7640 1 -- 172.21.15.83:0/1410311157 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06240039b0 msgr2=0x7f0624003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.689+0000 7f063fbf7640 1 --2- 172.21.15.83:0/1410311157 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06240039b0 0x7f0624003df0 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f063000c210 tx=0x7f0624005830 comp rx=0 tx=0).stop 2024-03-06T07:32:38.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.689+0000 7f063fbf7640 1 -- 172.21.15.83:0/1410311157 shutdown_connections 2024-03-06T07:32:38.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.689+0000 7f063fbf7640 1 --2- 172.21.15.83:0/1410311157 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06240039b0 0x7f0624003df0 unknown :-1 s=CLOSED pgs=1996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.689+0000 7f063fbf7640 1 -- 172.21.15.83:0/1410311157 >> 172.21.15.83:0/1410311157 conn(0x7f06380d3d40 msgr2=0x7f06380d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:38.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.690+0000 7f063fbf7640 1 -- 172.21.15.83:0/1410311157 shutdown_connections 2024-03-06T07:32:38.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.690+0000 7f063fbf7640 1 -- 172.21.15.83:0/1410311157 wait complete. 2024-03-06T07:32:38.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.691+0000 7f063fbf7640 1 Processor -- start 2024-03-06T07:32:38.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.691+0000 7f063fbf7640 1 -- start start 2024-03-06T07:32:38.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.691+0000 7f063fbf7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f06240039b0 0x7f06381f4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.691+0000 7f063fbf7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f063814e160 0x7f06380b6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.691+0000 7f063fbf7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06380b6e70 0x7f06380b8b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.691+0000 7f063fbf7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06380cfc60 con 0x7f06380b6e70 2024-03-06T07:32:38.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.691+0000 7f063fbf7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06380cf980 con 0x7f063814e160 2024-03-06T07:32:38.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.691+0000 7f063fbf7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06380b0910 con 0x7f06240039b0 2024-03-06T07:32:38.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.691+0000 7f063e194640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06380b6e70 0x7f06380b8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.691+0000 7f063e194640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06380b6e70 0x7f06380b8b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50572/0 (socket says 172.21.15.83:50572) 2024-03-06T07:32:38.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.691+0000 7f063e194640 1 -- 172.21.15.83:0/1817104580 learned_addr learned my addr 172.21.15.83:0/1817104580 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:38.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.692+0000 7f063d993640 1 --2- 172.21.15.83:0/1817104580 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f06240039b0 0x7f06381f4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.692+0000 7f063d192640 1 --2- 172.21.15.83:0/1817104580 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f063814e160 0x7f06380b6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.692+0000 7f063e194640 1 -- 172.21.15.83:0/1817104580 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f06240039b0 msgr2=0x7f06381f4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.692+0000 7f063e194640 1 --2- 172.21.15.83:0/1817104580 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f06240039b0 0x7f06381f4790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.692+0000 7f063e194640 1 -- 172.21.15.83:0/1817104580 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f063814e160 msgr2=0x7f06380b6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.692+0000 7f063e194640 1 --2- 172.21.15.83:0/1817104580 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f063814e160 0x7f06380b6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.692+0000 7f063e194640 1 -- 172.21.15.83:0/1817104580 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06380b98a0 con 0x7f06380b6e70 2024-03-06T07:32:38.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.692+0000 7f063e194640 1 --2- 172.21.15.83:0/1817104580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06380b6e70 0x7f06380b8b10 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f06380cadc0 tx=0x7f063400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:38.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.692+0000 7f062e7fc640 1 -- 172.21.15.83:0/1817104580 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f063400fd80 con 0x7f06380b6e70 2024-03-06T07:32:38.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.692+0000 7f062e7fc640 1 -- 172.21.15.83:0/1817104580 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f063401d070 con 0x7f06380b6e70 2024-03-06T07:32:38.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.692+0000 7f062e7fc640 1 -- 172.21.15.83:0/1817104580 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0634015670 con 0x7f06380b6e70 2024-03-06T07:32:38.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.692+0000 7f063fbf7640 1 -- 172.21.15.83:0/1817104580 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06380c8300 con 0x7f06380b6e70 2024-03-06T07:32:38.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.692+0000 7f063fbf7640 1 -- 172.21.15.83:0/1817104580 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06380c88c0 con 0x7f06380b6e70 2024-03-06T07:32:38.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.693+0000 7f063fbf7640 1 -- 172.21.15.83:0/1817104580 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06380c8ac0 con 0x7f06380b6e70 2024-03-06T07:32:38.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.694+0000 7f062e7fc640 1 -- 172.21.15.83:0/1817104580 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0634015810 con 0x7f06380b6e70 2024-03-06T07:32:38.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.694+0000 7f062e7fc640 1 --2- 172.21.15.83:0/1817104580 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f061c06b4d0 0x7f061c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.694+0000 7f062e7fc640 1 -- 172.21.15.83:0/1817104580 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f063408ad90 con 0x7f06380b6e70 2024-03-06T07:32:38.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.694+0000 7f063d993640 1 --2- 172.21.15.83:0/1817104580 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f061c06b4d0 0x7f061c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.694+0000 7f063d993640 1 --2- 172.21.15.83:0/1817104580 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f061c06b4d0 0x7f061c06d990 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f0630034000 tx=0x7f0630009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:38.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.699+0000 7f062e7fc640 1 -- 172.21.15.83:0/1817104580 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0634027020 con 0x7f06380b6e70 2024-03-06T07:32:38.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.945+0000 7f063fbf7640 1 -- 172.21.15.83:0/1817104580 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0638073fa0 con 0x7f06380b6e70 2024-03-06T07:32:38.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.945+0000 7f062e7fc640 1 -- 172.21.15.83:0/1817104580 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v132) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0634010040 con 0x7f06380b6e70 2024-03-06T07:32:38.947 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 64 2024-03-06T07:32:38.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.948+0000 7f063fbf7640 1 -- 172.21.15.83:0/1817104580 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f061c06b4d0 msgr2=0x7f061c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.948+0000 7f063fbf7640 1 --2- 172.21.15.83:0/1817104580 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f061c06b4d0 0x7f061c06d990 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f0630034000 tx=0x7f0630009030 comp rx=0 tx=0).stop 2024-03-06T07:32:38.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.948+0000 7f063fbf7640 1 -- 172.21.15.83:0/1817104580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06380b6e70 msgr2=0x7f06380b8b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.948+0000 7f063fbf7640 1 --2- 172.21.15.83:0/1817104580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06380b6e70 0x7f06380b8b10 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f06380cadc0 tx=0x7f063400f580 comp rx=0 tx=0).stop 2024-03-06T07:32:38.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.948+0000 7f063fbf7640 1 -- 172.21.15.83:0/1817104580 shutdown_connections 2024-03-06T07:32:38.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.948+0000 7f063fbf7640 1 --2- 172.21.15.83:0/1817104580 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f061c06b4d0 0x7f061c06d990 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.948+0000 7f063fbf7640 1 --2- 172.21.15.83:0/1817104580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06380b6e70 0x7f06380b8b10 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.948+0000 7f063fbf7640 1 --2- 172.21.15.83:0/1817104580 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f063814e160 0x7f06380b6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.948+0000 7f063fbf7640 1 --2- 172.21.15.83:0/1817104580 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f06240039b0 0x7f06381f4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.948+0000 7f063fbf7640 1 -- 172.21.15.83:0/1817104580 >> 172.21.15.83:0/1817104580 conn(0x7f06380d3d40 msgr2=0x7f06380d0f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:38.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.948+0000 7f063fbf7640 1 -- 172.21.15.83:0/1817104580 shutdown_connections 2024-03-06T07:32:38.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:38.948+0000 7f063fbf7640 1 -- 172.21.15.83:0/1817104580 wait complete. 2024-03-06T07:32:38.961 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:39.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.060+0000 7fe20e015640 1 Processor -- start 2024-03-06T07:32:39.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.060+0000 7fe20e015640 1 -- start start 2024-03-06T07:32:39.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.061+0000 7fe20e015640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe20815fa10 0x7fe2080ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.061+0000 7fe20e015640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fe2080d16e0 0x7fe2080d1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.061+0000 7fe20e015640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2080cebe0 con 0x7fe20815bdd0 2024-03-06T07:32:39.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.061+0000 7fe20e015640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2080ceec0 con 0x7fe2080d16e0 2024-03-06T07:32:39.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.061+0000 7fe20e015640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2080cf1a0 con 0x7fe20815fa10 2024-03-06T07:32:39.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.062+0000 7fe1fffff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe20815fa10 0x7fe2080ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.062+0000 7fe207fff640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fe2080d16e0 0x7fe2080d1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.062+0000 7fe2077fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fe20815bdd0 0x7fe2080c5610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:46608/0 (socket says 172.21.15.83:46608) 2024-03-06T07:32:39.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.062+0000 7fe1fffff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe20815fa10 0x7fe2080ce270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:37166/0 (socket says 172.21.15.83:37166) 2024-03-06T07:32:39.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.062+0000 7fe2077fe640 1 -- 172.21.15.83:0/2658260303 learned_addr learned my addr 172.21.15.83:0/2658260303 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:32:39.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.062+0000 7fe206ffd640 1 -- 172.21.15.83:0/2658260303 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1634977336 0 0) 0x7fe2080cebe0 con 0x7fe20815bdd0 2024-03-06T07:32:39.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.062+0000 7fe206ffd640 1 -- 172.21.15.83:0/2658260303 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1f4003520 con 0x7fe20815bdd0 2024-03-06T07:32:39.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.062+0000 7fe207fff640 1 -- 172.21.15.83:0/2658260303 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe20815fa10 msgr2=0x7fe2080ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:39.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.062+0000 7fe207fff640 1 --2- 172.21.15.83:0/2658260303 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe20815fa10 0x7fe2080ce270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.062+0000 7fe207fff640 1 -- 172.21.15.83:0/2658260303 >> v1:172.21.15.183:6789/0 conn(0x7fe20815bdd0 legacy=0x7fe2080c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.062+0000 7fe207fff640 1 -- 172.21.15.83:0/2658260303 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2080cc8e0 con 0x7fe2080d16e0 2024-03-06T07:32:39.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.062+0000 7fe1fffff640 1 --2- 172.21.15.83:0/2658260303 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe20815fa10 0x7fe2080ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:39.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.063+0000 7fe207fff640 1 --2- 172.21.15.83:0/2658260303 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe2080d16e0 0x7fe2080d1b30 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fe200009d90 tx=0x7fe20002be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=202e51bf01390285 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:39.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.063+0000 7fe206ffd640 1 -- 172.21.15.83:0/2658260303 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe200039070 con 0x7fe2080d16e0 2024-03-06T07:32:39.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.063+0000 7fe206ffd640 1 -- 172.21.15.83:0/2658260303 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe20002cbc0 con 0x7fe2080d16e0 2024-03-06T07:32:39.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.063+0000 7fe206ffd640 1 -- 172.21.15.83:0/2658260303 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe200034440 con 0x7fe2080d16e0 2024-03-06T07:32:39.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.064+0000 7fe20e015640 1 -- 172.21.15.83:0/2658260303 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe2080d16e0 msgr2=0x7fe2080d1b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.064+0000 7fe20e015640 1 --2- 172.21.15.83:0/2658260303 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe2080d16e0 0x7fe2080d1b30 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fe200009d90 tx=0x7fe20002be10 comp rx=0 tx=0).stop 2024-03-06T07:32:39.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.064+0000 7fe20e015640 1 -- 172.21.15.83:0/2658260303 shutdown_connections 2024-03-06T07:32:39.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.064+0000 7fe20e015640 1 --2- 172.21.15.83:0/2658260303 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe2080d16e0 0x7fe2080d1b30 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.064+0000 7fe20e015640 1 --2- 172.21.15.83:0/2658260303 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe20815fa10 0x7fe2080ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.064+0000 7fe20e015640 1 -- 172.21.15.83:0/2658260303 >> 172.21.15.83:0/2658260303 conn(0x7fe20809a9f0 msgr2=0x7fe20809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:39.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.064+0000 7fe20e015640 1 -- 172.21.15.83:0/2658260303 shutdown_connections 2024-03-06T07:32:39.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.064+0000 7fe20e015640 1 -- 172.21.15.83:0/2658260303 wait complete. 2024-03-06T07:32:39.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.065+0000 7fe20e015640 1 Processor -- start 2024-03-06T07:32:39.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.065+0000 7fe20e015640 1 -- start start 2024-03-06T07:32:39.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.066+0000 7fe20e015640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe2080d16e0 0x7fe20815a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.066+0000 7fe20e015640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe20815bdd0 0x7fe2080cad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.066+0000 7fe20e015640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe20815fa10 0x7fe2080cb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.066+0000 7fe20e015640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe20815c1b0 con 0x7fe20815fa10 2024-03-06T07:32:39.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.066+0000 7fe20e015640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2080cc5d0 con 0x7fe2080d16e0 2024-03-06T07:32:39.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.066+0000 7fe20e015640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2080ce7b0 con 0x7fe20815bdd0 2024-03-06T07:32:39.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.066+0000 7fe207fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe20815fa10 0x7fe2080cb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.066+0000 7fe207fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe20815fa10 0x7fe2080cb2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50580/0 (socket says 172.21.15.83:50580) 2024-03-06T07:32:39.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.066+0000 7fe207fff640 1 -- 172.21.15.83:0/987339292 learned_addr learned my addr 172.21.15.83:0/987339292 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:39.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.066+0000 7fe1fffff640 1 --2- 172.21.15.83:0/987339292 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe20815bdd0 0x7fe2080cad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.066+0000 7fe2077fe640 1 --2- 172.21.15.83:0/987339292 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe2080d16e0 0x7fe20815a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.066+0000 7fe207fff640 1 -- 172.21.15.83:0/987339292 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe20815bdd0 msgr2=0x7fe2080cad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.066+0000 7fe207fff640 1 --2- 172.21.15.83:0/987339292 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe20815bdd0 0x7fe2080cad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.066+0000 7fe207fff640 1 -- 172.21.15.83:0/987339292 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe2080d16e0 msgr2=0x7fe20815a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.066+0000 7fe207fff640 1 --2- 172.21.15.83:0/987339292 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe2080d16e0 0x7fe20815a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.066+0000 7fe207fff640 1 -- 172.21.15.83:0/987339292 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe20806d200 con 0x7fe20815fa10 2024-03-06T07:32:39.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.067+0000 7fe207fff640 1 --2- 172.21.15.83:0/987339292 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe20815fa10 0x7fe2080cb2a0 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7fe200009d90 tx=0x7fe200004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:39.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.067+0000 7fe204ff9640 1 -- 172.21.15.83:0/987339292 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe200039050 con 0x7fe20815fa10 2024-03-06T07:32:39.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.067+0000 7fe204ff9640 1 -- 172.21.15.83:0/987339292 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe200045070 con 0x7fe20815fa10 2024-03-06T07:32:39.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.067+0000 7fe20e015640 1 -- 172.21.15.83:0/987339292 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe20806d4e0 con 0x7fe20815fa10 2024-03-06T07:32:39.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.067+0000 7fe204ff9640 1 -- 172.21.15.83:0/987339292 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe200009450 con 0x7fe20815fa10 2024-03-06T07:32:39.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.067+0000 7fe20e015640 1 -- 172.21.15.83:0/987339292 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe20806daa0 con 0x7fe20815fa10 2024-03-06T07:32:39.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.068+0000 7fe20e015640 1 -- 172.21.15.83:0/987339292 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe20804f330 con 0x7fe20815fa10 2024-03-06T07:32:39.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.068+0000 7fe204ff9640 1 -- 172.21.15.83:0/987339292 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe20003e850 con 0x7fe20815fa10 2024-03-06T07:32:39.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.069+0000 7fe204ff9640 1 --2- 172.21.15.83:0/987339292 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe1e806b4d0 0x7fe1e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.069+0000 7fe2077fe640 1 --2- 172.21.15.83:0/987339292 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe1e806b4d0 0x7fe1e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.069+0000 7fe204ff9640 1 -- 172.21.15.83:0/987339292 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe2000acf90 con 0x7fe20815fa10 2024-03-06T07:32:39.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.069+0000 7fe2077fe640 1 --2- 172.21.15.83:0/987339292 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe1e806b4d0 0x7fe1e806d990 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fe1f8005880 tx=0x7fe1f8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:39.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.074+0000 7fe204ff9640 1 -- 172.21.15.83:0/987339292 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe20007b8a0 con 0x7fe20815fa10 2024-03-06T07:32:39.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.282+0000 7fe20e015640 1 -- 172.21.15.83:0/987339292 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe2080b7a40 con 0x7fe20815fa10 2024-03-06T07:32:39.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.283+0000 7fe204ff9640 1 -- 172.21.15.83:0/987339292 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7fe20007f940 con 0x7fe20815fa10 2024-03-06T07:32:39.285 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:32:39.285 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:32:39.285 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:32:39.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.285+0000 7fe20e015640 1 -- 172.21.15.83:0/987339292 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe1e806b4d0 msgr2=0x7fe1e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.285+0000 7fe20e015640 1 --2- 172.21.15.83:0/987339292 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe1e806b4d0 0x7fe1e806d990 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fe1f8005880 tx=0x7fe1f8009040 comp rx=0 tx=0).stop 2024-03-06T07:32:39.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.285+0000 7fe20e015640 1 -- 172.21.15.83:0/987339292 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe20815fa10 msgr2=0x7fe2080cb2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.285+0000 7fe20e015640 1 --2- 172.21.15.83:0/987339292 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe20815fa10 0x7fe2080cb2a0 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7fe200009d90 tx=0x7fe200004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:39.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.285+0000 7fe20e015640 1 -- 172.21.15.83:0/987339292 shutdown_connections 2024-03-06T07:32:39.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.285+0000 7fe20e015640 1 --2- 172.21.15.83:0/987339292 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fe1e806b4d0 0x7fe1e806d990 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.285+0000 7fe20e015640 1 --2- 172.21.15.83:0/987339292 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe20815fa10 0x7fe2080cb2a0 unknown :-1 s=CLOSED pgs=1998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.285+0000 7fe20e015640 1 --2- 172.21.15.83:0/987339292 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe20815bdd0 0x7fe2080cad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.285+0000 7fe20e015640 1 --2- 172.21.15.83:0/987339292 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe2080d16e0 0x7fe20815a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.285+0000 7fe20e015640 1 -- 172.21.15.83:0/987339292 >> 172.21.15.83:0/987339292 conn(0x7fe20809a9f0 msgr2=0x7fe2080c7630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:39.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.286+0000 7fe20e015640 1 -- 172.21.15.83:0/987339292 shutdown_connections 2024-03-06T07:32:39.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.286+0000 7fe20e015640 1 -- 172.21.15.83:0/987339292 wait complete. 2024-03-06T07:32:39.300 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:39.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.420+0000 7f5830c37640 1 Processor -- start 2024-03-06T07:32:39.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.420+0000 7f5830c37640 1 -- start start 2024-03-06T07:32:39.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.421+0000 7f5830c37640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f582c1424f0 0x7f582c0cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.421+0000 7f5830c37640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f582c0cd7a0 0x7f582c15cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.421+0000 7f5830c37640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f582c0cc980 con 0x7f582c06d330 2024-03-06T07:32:39.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.421+0000 7f5830c37640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f582c0cd220 con 0x7f582c1424f0 2024-03-06T07:32:39.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.421+0000 7f5830c37640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f582c0ccc60 con 0x7f582c0cd7a0 2024-03-06T07:32:39.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.421+0000 7f582a59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f582c1424f0 0x7f582c0cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.421+0000 7f582a59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f582c1424f0 0x7f582c0cc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50594/0 (socket says 172.21.15.83:50594) 2024-03-06T07:32:39.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.421+0000 7f582a59c640 1 -- 172.21.15.83:0/463335114 learned_addr learned my addr 172.21.15.83:0/463335114 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:39.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.421+0000 7f5829d9b640 1 --2- 172.21.15.83:0/463335114 >> v2:172.21.15.183:3300/0 conn(0x7f582c0cd7a0 0x7f582c15cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.421+0000 7f582a59c640 1 -- 172.21.15.83:0/463335114 >> v2:172.21.15.183:3300/0 conn(0x7f582c0cd7a0 msgr2=0x7f582c15cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:39.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.421+0000 7f582a59c640 1 --2- 172.21.15.83:0/463335114 >> v2:172.21.15.183:3300/0 conn(0x7f582c0cd7a0 0x7f582c15cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.421+0000 7f582a59c640 1 -- 172.21.15.83:0/463335114 >> v1:172.21.15.183:6789/0 conn(0x7f582c06d330 legacy=0x7f582c071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:39.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.421+0000 7f582a59c640 1 -- 172.21.15.83:0/463335114 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f582c0ccf40 con 0x7f582c1424f0 2024-03-06T07:32:39.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.422+0000 7f582a59c640 1 --2- 172.21.15.83:0/463335114 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f582c1424f0 0x7f582c0cc2f0 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f582c0b2a00 tx=0x7f582402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3775589f9445f8ca server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:39.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.422+0000 7f582959a640 1 -- 172.21.15.83:0/463335114 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582402ca80 con 0x7f582c1424f0 2024-03-06T07:32:39.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.422+0000 7f582959a640 1 -- 172.21.15.83:0/463335114 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f582402cc20 con 0x7f582c1424f0 2024-03-06T07:32:39.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.422+0000 7f582959a640 1 -- 172.21.15.83:0/463335114 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58240343f0 con 0x7f582c1424f0 2024-03-06T07:32:39.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.423+0000 7f5830c37640 1 -- 172.21.15.83:0/463335114 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f582c1424f0 msgr2=0x7f582c0cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.423+0000 7f5830c37640 1 --2- 172.21.15.83:0/463335114 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f582c1424f0 0x7f582c0cc2f0 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f582c0b2a00 tx=0x7f582402bee0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.423+0000 7f5830c37640 1 -- 172.21.15.83:0/463335114 shutdown_connections 2024-03-06T07:32:39.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.423+0000 7f5830c37640 1 --2- 172.21.15.83:0/463335114 >> v2:172.21.15.183:3300/0 conn(0x7f582c0cd7a0 0x7f582c15cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.423+0000 7f5830c37640 1 --2- 172.21.15.83:0/463335114 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f582c1424f0 0x7f582c0cc2f0 unknown :-1 s=CLOSED pgs=1999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.423+0000 7f5830c37640 1 -- 172.21.15.83:0/463335114 >> 172.21.15.83:0/463335114 conn(0x7f582c0cf8a0 msgr2=0x7f582c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:39.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.423+0000 7f5830c37640 1 -- 172.21.15.83:0/463335114 shutdown_connections 2024-03-06T07:32:39.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.423+0000 7f5830c37640 1 -- 172.21.15.83:0/463335114 wait complete. 2024-03-06T07:32:39.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.424+0000 7f5830c37640 1 Processor -- start 2024-03-06T07:32:39.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.424+0000 7f5830c37640 1 -- start start 2024-03-06T07:32:39.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.425+0000 7f5830c37640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f582c06d330 0x7f582c140e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.425+0000 7f5830c37640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f582c0cd7a0 0x7f582c151670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.425+0000 7f5830c37640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f582c1424f0 0x7f582c154bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.425+0000 7f5830c37640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f582c06d920 con 0x7f582c1424f0 2024-03-06T07:32:39.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.425+0000 7f5830c37640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f582c06d7a0 con 0x7f582c06d330 2024-03-06T07:32:39.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.425+0000 7f5830c37640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f582c06daa0 con 0x7f582c0cd7a0 2024-03-06T07:32:39.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.425+0000 7f582ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f582c1424f0 0x7f582c154bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.425+0000 7f582ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f582c1424f0 0x7f582c154bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50604/0 (socket says 172.21.15.83:50604) 2024-03-06T07:32:39.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.425+0000 7f582ad9d640 1 -- 172.21.15.83:0/1623193570 learned_addr learned my addr 172.21.15.83:0/1623193570 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:39.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.425+0000 7f582a59c640 1 --2- 172.21.15.83:0/1623193570 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f582c06d330 0x7f582c140e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.425+0000 7f5829d9b640 1 --2- 172.21.15.83:0/1623193570 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f582c0cd7a0 0x7f582c151670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.425+0000 7f582ad9d640 1 -- 172.21.15.83:0/1623193570 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f582c0cd7a0 msgr2=0x7f582c151670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.425+0000 7f582ad9d640 1 --2- 172.21.15.83:0/1623193570 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f582c0cd7a0 0x7f582c151670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.425+0000 7f582ad9d640 1 -- 172.21.15.83:0/1623193570 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f582c06d330 msgr2=0x7f582c140e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.425+0000 7f582ad9d640 1 --2- 172.21.15.83:0/1623193570 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f582c06d330 0x7f582c140e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.425+0000 7f582ad9d640 1 -- 172.21.15.83:0/1623193570 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f582c152ec0 con 0x7f582c1424f0 2024-03-06T07:32:39.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.426+0000 7f582ad9d640 1 --2- 172.21.15.83:0/1623193570 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f582c1424f0 0x7f582c154bf0 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f581c00c440 tx=0x7f581c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:39.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.426+0000 7f581affd640 1 -- 172.21.15.83:0/1623193570 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f581c011790 con 0x7f582c1424f0 2024-03-06T07:32:39.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.426+0000 7f581affd640 1 -- 172.21.15.83:0/1623193570 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f581c017070 con 0x7f582c1424f0 2024-03-06T07:32:39.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.426+0000 7f5830c37640 1 -- 172.21.15.83:0/1623193570 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f582c152620 con 0x7f582c1424f0 2024-03-06T07:32:39.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.426+0000 7f581affd640 1 -- 172.21.15.83:0/1623193570 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f581c011b70 con 0x7f582c1424f0 2024-03-06T07:32:39.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.426+0000 7f5830c37640 1 -- 172.21.15.83:0/1623193570 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f582c152be0 con 0x7f582c1424f0 2024-03-06T07:32:39.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.427+0000 7f5830c37640 1 -- 172.21.15.83:0/1623193570 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f582c04f330 con 0x7f582c1424f0 2024-03-06T07:32:39.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.427+0000 7f581affd640 1 -- 172.21.15.83:0/1623193570 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f581c011d10 con 0x7f582c1424f0 2024-03-06T07:32:39.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.428+0000 7f581affd640 1 --2- 172.21.15.83:0/1623193570 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f580c06b500 0x7f580c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.428+0000 7f582a59c640 1 --2- 172.21.15.83:0/1623193570 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f580c06b500 0x7f580c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.428+0000 7f581affd640 1 -- 172.21.15.83:0/1623193570 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f581c089fd0 con 0x7f582c1424f0 2024-03-06T07:32:39.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.428+0000 7f582a59c640 1 --2- 172.21.15.83:0/1623193570 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f580c06b500 0x7f580c06d9c0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f5824035000 tx=0x7f5824030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:39.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.432+0000 7f581affd640 1 -- 172.21.15.83:0/1623193570 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f581c058e10 con 0x7f582c1424f0 2024-03-06T07:32:39.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.605+0000 7f5830c37640 1 -- 172.21.15.83:0/1623193570 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f582c06eb30 con 0x7f582c1424f0 2024-03-06T07:32:39.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.606+0000 7f581affd640 1 -- 172.21.15.83:0/1623193570 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v132) v1 ==== 74+0+16732 (secure 0 0 0) 0x7f581c081e20 con 0x7f582c1424f0 2024-03-06T07:32:39.609 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:32:39.609 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":132,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:29:41.127522+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:50.508389+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:51.266049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.7","mappings":[{"from":2,"to":0}]},{"pgid":"19.13","mappings":[{"from":2,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:39.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.608+0000 7f5830c37640 1 -- 172.21.15.83:0/1623193570 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f580c06b500 msgr2=0x7f580c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.608+0000 7f5830c37640 1 --2- 172.21.15.83:0/1623193570 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f580c06b500 0x7f580c06d9c0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f5824035000 tx=0x7f5824030000 comp rx=0 tx=0).stop 2024-03-06T07:32:39.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.608+0000 7f5830c37640 1 -- 172.21.15.83:0/1623193570 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f582c1424f0 msgr2=0x7f582c154bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.608+0000 7f5830c37640 1 --2- 172.21.15.83:0/1623193570 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f582c1424f0 0x7f582c154bf0 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f581c00c440 tx=0x7f581c00c900 comp rx=0 tx=0).stop 2024-03-06T07:32:39.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.609+0000 7f5830c37640 1 -- 172.21.15.83:0/1623193570 shutdown_connections 2024-03-06T07:32:39.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.609+0000 7f5830c37640 1 --2- 172.21.15.83:0/1623193570 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f580c06b500 0x7f580c06d9c0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.609+0000 7f5830c37640 1 --2- 172.21.15.83:0/1623193570 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f582c1424f0 0x7f582c154bf0 unknown :-1 s=CLOSED pgs=2000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.609+0000 7f5830c37640 1 --2- 172.21.15.83:0/1623193570 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f582c0cd7a0 0x7f582c151670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.609+0000 7f5830c37640 1 --2- 172.21.15.83:0/1623193570 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f582c06d330 0x7f582c140e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.609+0000 7f5830c37640 1 -- 172.21.15.83:0/1623193570 >> 172.21.15.83:0/1623193570 conn(0x7f582c0cf8a0 msgr2=0x7f582c15ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:39.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.609+0000 7f5830c37640 1 -- 172.21.15.83:0/1623193570 shutdown_connections 2024-03-06T07:32:39.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.609+0000 7f5830c37640 1 -- 172.21.15.83:0/1623193570 wait complete. 2024-03-06T07:32:39.622 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:32:39.622 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:39.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.721+0000 7f1d9ee4e640 1 Processor -- start 2024-03-06T07:32:39.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.721+0000 7f1d9ee4e640 1 -- start start 2024-03-06T07:32:39.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.722+0000 7f1d9ee4e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1d980d22b0 0x7f1d980d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.722+0000 7f1d9ee4e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d980b3430 con 0x7f1d9815dfe0 2024-03-06T07:32:39.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.722+0000 7f1d9ee4e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d980b3710 con 0x7f1d980b41f0 2024-03-06T07:32:39.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.722+0000 7f1d9ee4e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d980d2c50 con 0x7f1d980d22b0 2024-03-06T07:32:39.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.722+0000 7f1d97fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1d980d22b0 0x7f1d980d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.722+0000 7f1d9cbea640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f1d9815dfe0 0x7f1d980cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57614/0 (socket says 172.21.15.83:57614) 2024-03-06T07:32:39.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.722+0000 7f1d9cbea640 1 -- 172.21.15.83:0/3463258764 learned_addr learned my addr 172.21.15.83:0/3463258764 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:32:39.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.722+0000 7f1d97fff640 1 -- 172.21.15.83:0/3463258764 >> v1:172.21.15.183:6789/0 conn(0x7f1d980b41f0 legacy=0x7f1d980b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:39.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.722+0000 7f1d977fe640 1 -- 172.21.15.83:0/3463258764 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1979977869 0 0) 0x7f1d980b3430 con 0x7f1d9815dfe0 2024-03-06T07:32:39.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.722+0000 7f1d97fff640 1 -- 172.21.15.83:0/3463258764 >> v1:172.21.15.83:6789/0 conn(0x7f1d9815dfe0 legacy=0x7f1d980cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.722+0000 7f1d97fff640 1 -- 172.21.15.83:0/3463258764 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d980b39f0 con 0x7f1d980d22b0 2024-03-06T07:32:39.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.723+0000 7f1d97fff640 1 --2- 172.21.15.83:0/3463258764 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d980d22b0 0x7f1d980d2710 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f1d840097d0 tx=0x7f1d8402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c8871c931aec59 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:39.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.723+0000 7f1d977fe640 1 -- 172.21.15.83:0/3463258764 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d84034070 con 0x7f1d980d22b0 2024-03-06T07:32:39.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.723+0000 7f1d977fe640 1 -- 172.21.15.83:0/3463258764 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d84030610 con 0x7f1d980d22b0 2024-03-06T07:32:39.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.723+0000 7f1d977fe640 1 -- 172.21.15.83:0/3463258764 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d840309e0 con 0x7f1d980d22b0 2024-03-06T07:32:39.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.724+0000 7f1d9ee4e640 1 -- 172.21.15.83:0/3463258764 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d980d22b0 msgr2=0x7f1d980d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.724+0000 7f1d9ee4e640 1 --2- 172.21.15.83:0/3463258764 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d980d22b0 0x7f1d980d2710 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f1d840097d0 tx=0x7f1d8402b960 comp rx=0 tx=0).stop 2024-03-06T07:32:39.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.724+0000 7f1d9ee4e640 1 -- 172.21.15.83:0/3463258764 shutdown_connections 2024-03-06T07:32:39.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.724+0000 7f1d9ee4e640 1 --2- 172.21.15.83:0/3463258764 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d980d22b0 0x7f1d980d2710 unknown :-1 s=CLOSED pgs=2002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.724+0000 7f1d9ee4e640 1 -- 172.21.15.83:0/3463258764 >> 172.21.15.83:0/3463258764 conn(0x7f1d980d3cd0 msgr2=0x7f1d980d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:39.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.724+0000 7f1d9ee4e640 1 -- 172.21.15.83:0/3463258764 shutdown_connections 2024-03-06T07:32:39.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.724+0000 7f1d9ee4e640 1 -- 172.21.15.83:0/3463258764 wait complete. 2024-03-06T07:32:39.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.725+0000 7f1d9ee4e640 1 Processor -- start 2024-03-06T07:32:39.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.725+0000 7f1d9ee4e640 1 -- start start 2024-03-06T07:32:39.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.726+0000 7f1d9ee4e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1d980b41f0 0x7f1d98154100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.726+0000 7f1d9ee4e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d980d22b0 0x7f1d980b8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.726+0000 7f1d9ee4e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1d9815dfe0 0x7f1d980b9230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.726+0000 7f1d9ee4e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d980b3d60 con 0x7f1d980d22b0 2024-03-06T07:32:39.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.726+0000 7f1d9ee4e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d980b31e0 con 0x7f1d9815dfe0 2024-03-06T07:32:39.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.726+0000 7f1d9ee4e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d980d1c70 con 0x7f1d980b41f0 2024-03-06T07:32:39.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.726+0000 7f1d97fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d980d22b0 0x7f1d980b8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.726+0000 7f1d97fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d980d22b0 0x7f1d980b8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50618/0 (socket says 172.21.15.83:50618) 2024-03-06T07:32:39.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.726+0000 7f1d97fff640 1 -- 172.21.15.83:0/2565322244 learned_addr learned my addr 172.21.15.83:0/2565322244 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:39.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.726+0000 7f1d9cbea640 1 --2- 172.21.15.83:0/2565322244 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1d980b41f0 0x7f1d98154100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.726+0000 7f1d9d3eb640 1 --2- 172.21.15.83:0/2565322244 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1d9815dfe0 0x7f1d980b9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.726+0000 7f1d97fff640 1 -- 172.21.15.83:0/2565322244 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1d980b41f0 msgr2=0x7f1d98154100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.726+0000 7f1d97fff640 1 --2- 172.21.15.83:0/2565322244 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1d980b41f0 0x7f1d98154100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.726+0000 7f1d97fff640 1 -- 172.21.15.83:0/2565322244 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1d9815dfe0 msgr2=0x7f1d980b9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.726+0000 7f1d97fff640 1 --2- 172.21.15.83:0/2565322244 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1d9815dfe0 0x7f1d980b9230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.726+0000 7f1d97fff640 1 -- 172.21.15.83:0/2565322244 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d9815ffa0 con 0x7f1d980d22b0 2024-03-06T07:32:39.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.726+0000 7f1d97fff640 1 --2- 172.21.15.83:0/2565322244 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d980d22b0 0x7f1d980b8cf0 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f1d840098c0 tx=0x7f1d840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:39.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.727+0000 7f1d957fa640 1 -- 172.21.15.83:0/2565322244 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d84034040 con 0x7f1d980d22b0 2024-03-06T07:32:39.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.727+0000 7f1d957fa640 1 -- 172.21.15.83:0/2565322244 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d84039990 con 0x7f1d980d22b0 2024-03-06T07:32:39.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.727+0000 7f1d957fa640 1 -- 172.21.15.83:0/2565322244 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d84039d80 con 0x7f1d980d22b0 2024-03-06T07:32:39.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.727+0000 7f1d9ee4e640 1 -- 172.21.15.83:0/2565322244 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d980b8330 con 0x7f1d980d22b0 2024-03-06T07:32:39.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.727+0000 7f1d9ee4e640 1 -- 172.21.15.83:0/2565322244 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d98160540 con 0x7f1d980d22b0 2024-03-06T07:32:39.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.728+0000 7f1d9ee4e640 1 -- 172.21.15.83:0/2565322244 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d9804f330 con 0x7f1d980d22b0 2024-03-06T07:32:39.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.729+0000 7f1d957fa640 1 -- 172.21.15.83:0/2565322244 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1d84037070 con 0x7f1d980d22b0 2024-03-06T07:32:39.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.729+0000 7f1d957fa640 1 --2- 172.21.15.83:0/2565322244 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1d7c06b4d0 0x7f1d7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.729+0000 7f1d957fa640 1 -- 172.21.15.83:0/2565322244 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1d840ac300 con 0x7f1d980d22b0 2024-03-06T07:32:39.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.729+0000 7f1d9cbea640 1 --2- 172.21.15.83:0/2565322244 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1d7c06b4d0 0x7f1d7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.730+0000 7f1d9cbea640 1 --2- 172.21.15.83:0/2565322244 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1d7c06b4d0 0x7f1d7c06d990 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c000f60 tx=0x7f1d8c007d00 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:39.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.734+0000 7f1d957fa640 1 -- 172.21.15.83:0/2565322244 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1d8407b140 con 0x7f1d980d22b0 2024-03-06T07:32:39.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.945+0000 7f1d9ee4e640 1 -- 172.21.15.83:0/2565322244 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d980b09c0 con 0x7f1d980d22b0 2024-03-06T07:32:39.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.946+0000 7f1d957fa640 1 -- 172.21.15.83:0/2565322244 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f1d8407aa40 con 0x7f1d980d22b0 2024-03-06T07:32:39.948 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:32:39.948 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:32:39.949 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:32:39.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.948+0000 7f1d9ee4e640 1 -- 172.21.15.83:0/2565322244 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1d7c06b4d0 msgr2=0x7f1d7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.948+0000 7f1d9ee4e640 1 --2- 172.21.15.83:0/2565322244 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1d7c06b4d0 0x7f1d7c06d990 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c000f60 tx=0x7f1d8c007d00 comp rx=0 tx=0).stop 2024-03-06T07:32:39.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.948+0000 7f1d9ee4e640 1 -- 172.21.15.83:0/2565322244 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d980d22b0 msgr2=0x7f1d980b8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.948+0000 7f1d9ee4e640 1 --2- 172.21.15.83:0/2565322244 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d980d22b0 0x7f1d980b8cf0 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f1d840098c0 tx=0x7f1d840046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.948+0000 7f1d9ee4e640 1 -- 172.21.15.83:0/2565322244 shutdown_connections 2024-03-06T07:32:39.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.948+0000 7f1d9ee4e640 1 --2- 172.21.15.83:0/2565322244 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1d7c06b4d0 0x7f1d7c06d990 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.948+0000 7f1d9ee4e640 1 --2- 172.21.15.83:0/2565322244 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1d9815dfe0 0x7f1d980b9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.948+0000 7f1d9ee4e640 1 --2- 172.21.15.83:0/2565322244 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1d980d22b0 0x7f1d980b8cf0 unknown :-1 s=CLOSED pgs=2003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.948+0000 7f1d9ee4e640 1 --2- 172.21.15.83:0/2565322244 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1d980b41f0 0x7f1d98154100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.948+0000 7f1d9ee4e640 1 -- 172.21.15.83:0/2565322244 >> 172.21.15.83:0/2565322244 conn(0x7f1d980d3cd0 msgr2=0x7f1d980d0600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:39.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.948+0000 7f1d9ee4e640 1 -- 172.21.15.83:0/2565322244 shutdown_connections 2024-03-06T07:32:39.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:39.948+0000 7f1d9ee4e640 1 -- 172.21.15.83:0/2565322244 wait complete. 2024-03-06T07:32:39.962 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:40.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.061+0000 7ff88a5a0640 1 Processor -- start 2024-03-06T07:32:40.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.061+0000 7ff88a5a0640 1 -- start start 2024-03-06T07:32:40.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.062+0000 7ff88a5a0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8840d16e0 0x7ff8840d1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.062+0000 7ff88a5a0640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8840cebe0 con 0x7ff88415bdd0 2024-03-06T07:32:40.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.062+0000 7ff88a5a0640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8840ceec0 con 0x7ff88415fa10 2024-03-06T07:32:40.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.062+0000 7ff88a5a0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8840ce980 con 0x7ff8840d16e0 2024-03-06T07:32:40.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.062+0000 7ff883fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff88415bdd0 0x7ff8840c5610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57630/0 (socket says 172.21.15.83:57630) 2024-03-06T07:32:40.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.062+0000 7ff883fff640 1 -- 172.21.15.83:0/305317241 learned_addr learned my addr 172.21.15.83:0/305317241 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:32:40.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.062+0000 7ff882ffd640 1 -- 172.21.15.83:0/305317241 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3985562464 0 0) 0x7ff8840cebe0 con 0x7ff88415bdd0 2024-03-06T07:32:40.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.062+0000 7ff888b3d640 1 --2- 172.21.15.83:0/305317241 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8840d16e0 0x7ff8840d1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.062+0000 7ff882ffd640 1 -- 172.21.15.83:0/305317241 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff86c003520 con 0x7ff88415bdd0 2024-03-06T07:32:40.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.063+0000 7ff882ffd640 1 -- 172.21.15.83:0/305317241 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2903782762 0 0) 0x7ff86c003520 con 0x7ff88415bdd0 2024-03-06T07:32:40.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.063+0000 7ff882ffd640 1 -- 172.21.15.83:0/305317241 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8840d16e0 msgr2=0x7ff8840d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:40.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.063+0000 7ff882ffd640 1 --2- 172.21.15.83:0/305317241 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8840d16e0 0x7ff8840d1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.063+0000 7ff882ffd640 1 -- 172.21.15.83:0/305317241 >> v1:172.21.15.183:6789/0 conn(0x7ff88415fa10 legacy=0x7ff8840ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.063+0000 7ff882ffd640 1 -- 172.21.15.83:0/305317241 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8840cf1a0 con 0x7ff88415bdd0 2024-03-06T07:32:40.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.063+0000 7ff882ffd640 1 -- 172.21.15.83:0/305317241 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7ff874001670 con 0x7ff88415bdd0 2024-03-06T07:32:40.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.063+0000 7ff882ffd640 1 -- 172.21.15.83:0/305317241 >> v1:172.21.15.83:6789/0 conn(0x7ff88415bdd0 legacy=0x7ff8840c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.063+0000 7ff882ffd640 1 --2- 172.21.15.83:0/305317241 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff86c0039e0 0x7ff86c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.063+0000 7ff882ffd640 1 --2- 172.21.15.83:0/305317241 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8840d16e0 0x7ff86c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.063+0000 7ff882ffd640 1 --2- 172.21.15.83:0/305317241 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff86c004d40 0x7ff86c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.063+0000 7ff882ffd640 1 -- 172.21.15.83:0/305317241 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8840ce980 con 0x7ff86c004d40 2024-03-06T07:32:40.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.063+0000 7ff882ffd640 1 -- 172.21.15.83:0/305317241 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86c00baf0 con 0x7ff86c0039e0 2024-03-06T07:32:40.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.063+0000 7ff882ffd640 1 -- 172.21.15.83:0/305317241 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff86c00bcf0 con 0x7ff8840d16e0 2024-03-06T07:32:40.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.063+0000 7ff888b3d640 1 --2- 172.21.15.83:0/305317241 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff86c004d40 0x7ff86c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.063+0000 7ff883fff640 1 --2- 172.21.15.83:0/305317241 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff86c0039e0 0x7ff86c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.063+0000 7ff8837fe640 1 --2- 172.21.15.83:0/305317241 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8840d16e0 0x7ff86c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.063+0000 7ff888b3d640 1 -- 172.21.15.83:0/305317241 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8840d16e0 msgr2=0x7ff86c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:40.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.063+0000 7ff888b3d640 1 --2- 172.21.15.83:0/305317241 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8840d16e0 0x7ff86c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.063+0000 7ff888b3d640 1 -- 172.21.15.83:0/305317241 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff86c0039e0 msgr2=0x7ff86c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:40.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.063+0000 7ff888b3d640 1 --2- 172.21.15.83:0/305317241 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff86c0039e0 0x7ff86c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.063+0000 7ff888b3d640 1 -- 172.21.15.83:0/305317241 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff86c003520 con 0x7ff86c004d40 2024-03-06T07:32:40.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.064+0000 7ff888b3d640 1 --2- 172.21.15.83:0/305317241 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff86c004d40 0x7ff86c0051c0 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7ff87c009fc0 tx=0x7ff86c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cb560f81d0e9707 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:40.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.064+0000 7ff882ffd640 1 -- 172.21.15.83:0/305317241 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87c039070 con 0x7ff86c004d40 2024-03-06T07:32:40.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.064+0000 7ff882ffd640 1 -- 172.21.15.83:0/305317241 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff87c02cc20 con 0x7ff86c004d40 2024-03-06T07:32:40.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.064+0000 7ff882ffd640 1 -- 172.21.15.83:0/305317241 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87c009c10 con 0x7ff86c004d40 2024-03-06T07:32:40.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.064+0000 7ff88a5a0640 1 -- 172.21.15.83:0/305317241 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff86c004d40 msgr2=0x7ff86c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.064+0000 7ff88a5a0640 1 --2- 172.21.15.83:0/305317241 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff86c004d40 0x7ff86c0051c0 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7ff87c009fc0 tx=0x7ff86c005830 comp rx=0 tx=0).stop 2024-03-06T07:32:40.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.064+0000 7ff888b3d640 1 -- 172.21.15.83:0/305317241 reap_dead start 2024-03-06T07:32:40.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.064+0000 7ff88a5a0640 1 -- 172.21.15.83:0/305317241 shutdown_connections 2024-03-06T07:32:40.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.064+0000 7ff88a5a0640 1 -- 172.21.15.83:0/305317241 >> 172.21.15.83:0/305317241 conn(0x7ff88409a9f0 msgr2=0x7ff88409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:40.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.065+0000 7ff88a5a0640 1 -- 172.21.15.83:0/305317241 shutdown_connections 2024-03-06T07:32:40.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.065+0000 7ff88a5a0640 1 -- 172.21.15.83:0/305317241 wait complete. 2024-03-06T07:32:40.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.066+0000 7ff88a5a0640 1 Processor -- start 2024-03-06T07:32:40.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.066+0000 7ff88a5a0640 1 -- start start 2024-03-06T07:32:40.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.066+0000 7ff88a5a0640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff88414fde0 0x7ff8840b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.066+0000 7ff88a5a0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff88406de00 0x7ff8840b2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.066+0000 7ff88a5a0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8840b5540 0x7ff8840b3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.066+0000 7ff88a5a0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff88415c1b0 con 0x7ff88406de00 2024-03-06T07:32:40.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.066+0000 7ff88a5a0640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8840cc5d0 con 0x7ff88414fde0 2024-03-06T07:32:40.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.066+0000 7ff88a5a0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8840d1b10 con 0x7ff8840b5540 2024-03-06T07:32:40.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.067+0000 7ff8837fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff88406de00 0x7ff8840b2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.067+0000 7ff8837fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff88406de00 0x7ff8840b2d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50632/0 (socket says 172.21.15.83:50632) 2024-03-06T07:32:40.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.067+0000 7ff8837fe640 1 -- 172.21.15.83:0/1506353237 learned_addr learned my addr 172.21.15.83:0/1506353237 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:40.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.067+0000 7ff888b3d640 1 --2- 172.21.15.83:0/1506353237 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8840b5540 0x7ff8840b3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.067+0000 7ff883fff640 1 --2- 172.21.15.83:0/1506353237 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff88414fde0 0x7ff8840b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.067+0000 7ff8837fe640 1 -- 172.21.15.83:0/1506353237 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8840b5540 msgr2=0x7ff8840b3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.067+0000 7ff8837fe640 1 --2- 172.21.15.83:0/1506353237 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8840b5540 0x7ff8840b3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.067+0000 7ff8837fe640 1 -- 172.21.15.83:0/1506353237 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff88414fde0 msgr2=0x7ff8840b5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.067+0000 7ff8837fe640 1 --2- 172.21.15.83:0/1506353237 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff88414fde0 0x7ff8840b5000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.067+0000 7ff8837fe640 1 -- 172.21.15.83:0/1506353237 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8840b5b00 con 0x7ff88406de00 2024-03-06T07:32:40.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.067+0000 7ff8837fe640 1 --2- 172.21.15.83:0/1506353237 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff88406de00 0x7ff8840b2d00 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7ff87800f820 tx=0x7ff87800fce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:40.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.067+0000 7ff880ff9640 1 -- 172.21.15.83:0/1506353237 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8780050d0 con 0x7ff88406de00 2024-03-06T07:32:40.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.067+0000 7ff880ff9640 1 -- 172.21.15.83:0/1506353237 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff878018070 con 0x7ff88406de00 2024-03-06T07:32:40.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.067+0000 7ff88a5a0640 1 -- 172.21.15.83:0/1506353237 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff88406d320 con 0x7ff88406de00 2024-03-06T07:32:40.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.067+0000 7ff880ff9640 1 -- 172.21.15.83:0/1506353237 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff878005470 con 0x7ff88406de00 2024-03-06T07:32:40.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.067+0000 7ff88a5a0640 1 -- 172.21.15.83:0/1506353237 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8840b7560 con 0x7ff88406de00 2024-03-06T07:32:40.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.068+0000 7ff88a5a0640 1 -- 172.21.15.83:0/1506353237 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8840c5760 con 0x7ff88406de00 2024-03-06T07:32:40.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.069+0000 7ff880ff9640 1 -- 172.21.15.83:0/1506353237 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff87800d730 con 0x7ff88406de00 2024-03-06T07:32:40.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.069+0000 7ff880ff9640 1 --2- 172.21.15.83:0/1506353237 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff87006b500 0x7ff87006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.069+0000 7ff883fff640 1 --2- 172.21.15.83:0/1506353237 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff87006b500 0x7ff87006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.069+0000 7ff880ff9640 1 -- 172.21.15.83:0/1506353237 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff87808a150 con 0x7ff88406de00 2024-03-06T07:32:40.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.070+0000 7ff883fff640 1 --2- 172.21.15.83:0/1506353237 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff87006b500 0x7ff87006d9c0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7ff874004840 tx=0x7ff874009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:40.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.073+0000 7ff880ff9640 1 -- 172.21.15.83:0/1506353237 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff8840c5760 con 0x7ff88406de00 2024-03-06T07:32:40.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.289+0000 7ff88a5a0640 1 -- 172.21.15.83:0/1506353237 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff88404f330 con 0x7ff88406de00 2024-03-06T07:32:40.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.289+0000 7ff880ff9640 1 -- 172.21.15.83:0/1506353237 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3462 (secure 0 0 0) 0x7ff878058f00 con 0x7ff88406de00 2024-03-06T07:32:40.291 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:32:40.291 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:52.306698+0000","modified":"2024-03-06T07:31:26.591282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8631,"mds_1":8628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8628":{"gid":8628,"name":"c","rank":1,"incarnation":75,"state":"up:stopping","state_seq":7,"addr":"172.21.15.83:6836/3544993654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3544993654},{"type":"v1","addr":"172.21.15.83:6836","nonce":3544993654}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:54.013138+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8631":{"gid":8631,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.83:6837/1401317546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":1401317546},{"type":"v1","addr":"172.21.15.83:6837","nonce":1401317546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8631,"qdb_cluster":[8631]},"id":9}]} 2024-03-06T07:32:40.292 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 83 2024-03-06T07:32:40.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.291+0000 7ff88a5a0640 1 -- 172.21.15.83:0/1506353237 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff87006b500 msgr2=0x7ff87006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.291+0000 7ff88a5a0640 1 --2- 172.21.15.83:0/1506353237 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff87006b500 0x7ff87006d9c0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7ff874004840 tx=0x7ff874009040 comp rx=0 tx=0).stop 2024-03-06T07:32:40.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.291+0000 7ff88a5a0640 1 -- 172.21.15.83:0/1506353237 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff88406de00 msgr2=0x7ff8840b2d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.291+0000 7ff88a5a0640 1 --2- 172.21.15.83:0/1506353237 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff88406de00 0x7ff8840b2d00 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7ff87800f820 tx=0x7ff87800fce0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.291+0000 7ff88a5a0640 1 -- 172.21.15.83:0/1506353237 shutdown_connections 2024-03-06T07:32:40.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.291+0000 7ff88a5a0640 1 --2- 172.21.15.83:0/1506353237 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff87006b500 0x7ff87006d9c0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.291+0000 7ff88a5a0640 1 --2- 172.21.15.83:0/1506353237 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8840b5540 0x7ff8840b3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.291+0000 7ff88a5a0640 1 --2- 172.21.15.83:0/1506353237 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff88406de00 0x7ff8840b2d00 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.291+0000 7ff88a5a0640 1 --2- 172.21.15.83:0/1506353237 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff88414fde0 0x7ff8840b5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.291+0000 7ff88a5a0640 1 -- 172.21.15.83:0/1506353237 >> 172.21.15.83:0/1506353237 conn(0x7ff88409a9f0 msgr2=0x7ff8840c6330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:40.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.291+0000 7ff88a5a0640 1 -- 172.21.15.83:0/1506353237 shutdown_connections 2024-03-06T07:32:40.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.291+0000 7ff88a5a0640 1 -- 172.21.15.83:0/1506353237 wait complete. 2024-03-06T07:32:40.304 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:40.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.404+0000 7f7e984df640 1 Processor -- start 2024-03-06T07:32:40.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.404+0000 7f7e984df640 1 -- start start 2024-03-06T07:32:40.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.405+0000 7f7e984df640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7e900b4170 0x7f7e900b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.405+0000 7f7e984df640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f7e900d0a80 0x7f7e900cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.405+0000 7f7e984df640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e900d25d0 con 0x7f7e900d4760 2024-03-06T07:32:40.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.405+0000 7f7e984df640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e900d4bf0 con 0x7f7e900b4170 2024-03-06T07:32:40.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.405+0000 7f7e984df640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e900ce9e0 con 0x7f7e900d0a80 2024-03-06T07:32:40.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.405+0000 7f7e9627b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7e900b4170 0x7f7e900b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.405+0000 7f7e9627b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7e900b4170 0x7f7e900b6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50646/0 (socket says 172.21.15.83:50646) 2024-03-06T07:32:40.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.405+0000 7f7e9627b640 1 -- 172.21.15.83:0/1146300268 learned_addr learned my addr 172.21.15.83:0/1146300268 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:40.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.406+0000 7f7e96a7c640 1 --2- 172.21.15.83:0/1146300268 >> v2:172.21.15.183:3300/0 conn(0x7f7e900d0a80 0x7f7e900cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.406+0000 7f7e9627b640 1 -- 172.21.15.83:0/1146300268 >> v2:172.21.15.183:3300/0 conn(0x7f7e900d0a80 msgr2=0x7f7e900cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:40.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.406+0000 7f7e9627b640 1 --2- 172.21.15.83:0/1146300268 >> v2:172.21.15.183:3300/0 conn(0x7f7e900d0a80 0x7f7e900cee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.406+0000 7f7e9627b640 1 -- 172.21.15.83:0/1146300268 >> v1:172.21.15.183:6789/0 conn(0x7f7e900d4760 legacy=0x7f7e900d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:40.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.406+0000 7f7e9627b640 1 -- 172.21.15.83:0/1146300268 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e900d2870 con 0x7f7e900b4170 2024-03-06T07:32:40.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.406+0000 7f7e9627b640 1 --2- 172.21.15.83:0/1146300268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e900b4170 0x7f7e900b6a90 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f7e9008df00 tx=0x7f7e8c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4ac1afecb9417d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:40.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.407+0000 7f7e95279640 1 -- 172.21.15.83:0/1146300268 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c035020 con 0x7f7e900b4170 2024-03-06T07:32:40.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.407+0000 7f7e95279640 1 -- 172.21.15.83:0/1146300268 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e8c031870 con 0x7f7e900b4170 2024-03-06T07:32:40.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.407+0000 7f7e95279640 1 -- 172.21.15.83:0/1146300268 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c031c40 con 0x7f7e900b4170 2024-03-06T07:32:40.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.407+0000 7f7e984df640 1 -- 172.21.15.83:0/1146300268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e900b4170 msgr2=0x7f7e900b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.407+0000 7f7e984df640 1 --2- 172.21.15.83:0/1146300268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e900b4170 0x7f7e900b6a90 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f7e9008df00 tx=0x7f7e8c02b910 comp rx=0 tx=0).stop 2024-03-06T07:32:40.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.407+0000 7f7e984df640 1 -- 172.21.15.83:0/1146300268 shutdown_connections 2024-03-06T07:32:40.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.407+0000 7f7e984df640 1 --2- 172.21.15.83:0/1146300268 >> v2:172.21.15.183:3300/0 conn(0x7f7e900d0a80 0x7f7e900cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.407+0000 7f7e984df640 1 --2- 172.21.15.83:0/1146300268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e900b4170 0x7f7e900b6a90 unknown :-1 s=CLOSED pgs=2007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.407+0000 7f7e984df640 1 -- 172.21.15.83:0/1146300268 >> 172.21.15.83:0/1146300268 conn(0x7f7e900d3c40 msgr2=0x7f7e900d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:40.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.407+0000 7f7e984df640 1 -- 172.21.15.83:0/1146300268 shutdown_connections 2024-03-06T07:32:40.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.407+0000 7f7e984df640 1 -- 172.21.15.83:0/1146300268 wait complete. 2024-03-06T07:32:40.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.408+0000 7f7e984df640 1 Processor -- start 2024-03-06T07:32:40.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.408+0000 7f7e984df640 1 -- start start 2024-03-06T07:32:40.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.409+0000 7f7e984df640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e900b4170 0x7f7e900cb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.409+0000 7f7e984df640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e900d0a80 0x7f7e900cd830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.409+0000 7f7e984df640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e901587d0 0x7f7e90156a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.409+0000 7f7e984df640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e900b4dd0 con 0x7f7e900d0a80 2024-03-06T07:32:40.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.409+0000 7f7e984df640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e900b4550 con 0x7f7e900b4170 2024-03-06T07:32:40.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.409+0000 7f7e984df640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e900b4f50 con 0x7f7e901587d0 2024-03-06T07:32:40.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.409+0000 7f7e95a7a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e900d0a80 0x7f7e900cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.409+0000 7f7e95a7a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e900d0a80 0x7f7e900cd830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50656/0 (socket says 172.21.15.83:50656) 2024-03-06T07:32:40.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.409+0000 7f7e95a7a640 1 -- 172.21.15.83:0/2646309893 learned_addr learned my addr 172.21.15.83:0/2646309893 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:40.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.409+0000 7f7e96a7c640 1 --2- 172.21.15.83:0/2646309893 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e901587d0 0x7f7e90156a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.409+0000 7f7e9627b640 1 --2- 172.21.15.83:0/2646309893 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e900b4170 0x7f7e900cb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.410+0000 7f7e95a7a640 1 -- 172.21.15.83:0/2646309893 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e901587d0 msgr2=0x7f7e90156a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.410+0000 7f7e95a7a640 1 --2- 172.21.15.83:0/2646309893 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e901587d0 0x7f7e90156a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.410+0000 7f7e95a7a640 1 -- 172.21.15.83:0/2646309893 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e900b4170 msgr2=0x7f7e900cb7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.410+0000 7f7e95a7a640 1 --2- 172.21.15.83:0/2646309893 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e900b4170 0x7f7e900cb7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.410+0000 7f7e95a7a640 1 -- 172.21.15.83:0/2646309893 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e900cbe90 con 0x7f7e900d0a80 2024-03-06T07:32:40.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.410+0000 7f7e95a7a640 1 --2- 172.21.15.83:0/2646309893 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e900d0a80 0x7f7e900cd830 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f7e8000b3f0 tx=0x7f7e8000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:40.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.410+0000 7f7e86ffd640 1 -- 172.21.15.83:0/2646309893 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e80015020 con 0x7f7e900d0a80 2024-03-06T07:32:40.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.410+0000 7f7e86ffd640 1 -- 172.21.15.83:0/2646309893 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e80011830 con 0x7f7e900d0a80 2024-03-06T07:32:40.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.410+0000 7f7e86ffd640 1 -- 172.21.15.83:0/2646309893 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e80011c20 con 0x7f7e900d0a80 2024-03-06T07:32:40.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.410+0000 7f7e984df640 1 -- 172.21.15.83:0/2646309893 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e900cc170 con 0x7f7e900d0a80 2024-03-06T07:32:40.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.410+0000 7f7e984df640 1 -- 172.21.15.83:0/2646309893 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e901582b0 con 0x7f7e900d0a80 2024-03-06T07:32:40.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.411+0000 7f7e984df640 1 -- 172.21.15.83:0/2646309893 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e9004f330 con 0x7f7e900d0a80 2024-03-06T07:32:40.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.411+0000 7f7e86ffd640 1 -- 172.21.15.83:0/2646309893 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7e80018020 con 0x7f7e900d0a80 2024-03-06T07:32:40.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.412+0000 7f7e86ffd640 1 --2- 172.21.15.83:0/2646309893 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7e7406b4d0 0x7f7e7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.412+0000 7f7e86ffd640 1 -- 172.21.15.83:0/2646309893 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7e80089e40 con 0x7f7e900d0a80 2024-03-06T07:32:40.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.412+0000 7f7e9627b640 1 --2- 172.21.15.83:0/2646309893 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7e7406b4d0 0x7f7e7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.412+0000 7f7e9627b640 1 --2- 172.21.15.83:0/2646309893 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7e7406b4d0 0x7f7e7406d990 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c036000 tx=0x7f7e8c03c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:40.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.420+0000 7f7e86ffd640 1 -- 172.21.15.83:0/2646309893 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7e8001f030 con 0x7f7e900d0a80 2024-03-06T07:32:40.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.599+0000 7f7e984df640 1 -- 172.21.15.83:0/2646309893 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e900733b0 con 0x7f7e900d0a80 2024-03-06T07:32:40.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.600+0000 7f7e86ffd640 1 -- 172.21.15.83:0/2646309893 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v132) v1 ==== 74+0+16732 (secure 0 0 0) 0x7f7e800172c0 con 0x7f7e900d0a80 2024-03-06T07:32:40.602 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:32:40.602 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":132,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:29:41.127522+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:50.508389+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:51.266049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.7","mappings":[{"from":2,"to":0}]},{"pgid":"19.13","mappings":[{"from":2,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:40.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.602+0000 7f7e984df640 1 -- 172.21.15.83:0/2646309893 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7e7406b4d0 msgr2=0x7f7e7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.602+0000 7f7e984df640 1 --2- 172.21.15.83:0/2646309893 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7e7406b4d0 0x7f7e7406d990 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c036000 tx=0x7f7e8c03c040 comp rx=0 tx=0).stop 2024-03-06T07:32:40.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.602+0000 7f7e984df640 1 -- 172.21.15.83:0/2646309893 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e900d0a80 msgr2=0x7f7e900cd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.602+0000 7f7e984df640 1 --2- 172.21.15.83:0/2646309893 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e900d0a80 0x7f7e900cd830 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f7e8000b3f0 tx=0x7f7e8000b8b0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.602+0000 7f7e984df640 1 -- 172.21.15.83:0/2646309893 shutdown_connections 2024-03-06T07:32:40.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.602+0000 7f7e984df640 1 --2- 172.21.15.83:0/2646309893 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f7e7406b4d0 0x7f7e7406d990 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.602+0000 7f7e984df640 1 --2- 172.21.15.83:0/2646309893 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e901587d0 0x7f7e90156a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.602+0000 7f7e984df640 1 --2- 172.21.15.83:0/2646309893 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e900d0a80 0x7f7e900cd830 unknown :-1 s=CLOSED pgs=2008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.602+0000 7f7e984df640 1 --2- 172.21.15.83:0/2646309893 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e900b4170 0x7f7e900cb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.602+0000 7f7e984df640 1 -- 172.21.15.83:0/2646309893 >> 172.21.15.83:0/2646309893 conn(0x7f7e900d3c40 msgr2=0x7f7e900d4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:40.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.603+0000 7f7e984df640 1 -- 172.21.15.83:0/2646309893 shutdown_connections 2024-03-06T07:32:40.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.603+0000 7f7e984df640 1 -- 172.21.15.83:0/2646309893 wait complete. 2024-03-06T07:32:40.616 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:32:40.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.753+0000 7f45fa3c7640 1 Processor -- start 2024-03-06T07:32:40.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.753+0000 7f45fa3c7640 1 -- start start 2024-03-06T07:32:40.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.753+0000 7f45fa3c7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f45f40b4460 0x7f45f40d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.753+0000 7f45fa3c7640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f45f4162970 0x7f45f40b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.753+0000 7f45fa3c7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f45f4161cf0 0x7f45f4162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.754+0000 7f45fa3c7640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45f4161b00 con 0x7f45f40b4460 2024-03-06T07:32:40.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.754+0000 7f45fa3c7640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45f40cecc0 con 0x7f45f4162970 2024-03-06T07:32:40.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.754+0000 7f45fa3c7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45f40d22c0 con 0x7f45f4161cf0 2024-03-06T07:32:40.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.754+0000 7f45f3fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f45f40b4460 0x7f45f40d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.754+0000 7f45f3fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f45f40b4460 0x7f45f40d4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50658/0 (socket says 172.21.15.83:50658) 2024-03-06T07:32:40.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.754+0000 7f45f3fff640 1 -- 172.21.15.83:0/598398069 learned_addr learned my addr 172.21.15.83:0/598398069 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:40.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.754+0000 7f45f8964640 1 --2- 172.21.15.83:0/598398069 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f45f4161cf0 0x7f45f4162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.754+0000 7f45f37fe640 1 --2- 172.21.15.83:0/598398069 >> v2:172.21.15.183:3300/0 conn(0x7f45f4162970 0x7f45f40b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.754+0000 7f45f3fff640 1 -- 172.21.15.83:0/598398069 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f45f4161cf0 msgr2=0x7f45f4162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:40.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.754+0000 7f45f3fff640 1 --2- 172.21.15.83:0/598398069 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f45f4161cf0 0x7f45f4162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.754+0000 7f45f3fff640 1 -- 172.21.15.83:0/598398069 >> v2:172.21.15.183:3300/0 conn(0x7f45f4162970 msgr2=0x7f45f40b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:40.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.754+0000 7f45f3fff640 1 --2- 172.21.15.83:0/598398069 >> v2:172.21.15.183:3300/0 conn(0x7f45f4162970 0x7f45f40b3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.754+0000 7f45f3fff640 1 -- 172.21.15.83:0/598398069 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45f40d25a0 con 0x7f45f40b4460 2024-03-06T07:32:40.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.755+0000 7f45f3fff640 1 --2- 172.21.15.83:0/598398069 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45f40b4460 0x7f45f40d4800 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f45f40d17e0 tx=0x7f45e402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82084ec5d6b82683 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:40.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.755+0000 7f45f2ffd640 1 -- 172.21.15.83:0/598398069 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e4035020 con 0x7f45f40b4460 2024-03-06T07:32:40.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.755+0000 7f45f2ffd640 1 -- 172.21.15.83:0/598398069 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45e4031870 con 0x7f45f40b4460 2024-03-06T07:32:40.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.755+0000 7f45f2ffd640 1 -- 172.21.15.83:0/598398069 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e4031c40 con 0x7f45f40b4460 2024-03-06T07:32:40.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.756+0000 7f45fa3c7640 1 -- 172.21.15.83:0/598398069 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45f40b4460 msgr2=0x7f45f40d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.756+0000 7f45fa3c7640 1 --2- 172.21.15.83:0/598398069 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45f40b4460 0x7f45f40d4800 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f45f40d17e0 tx=0x7f45e402b910 comp rx=0 tx=0).stop 2024-03-06T07:32:40.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.756+0000 7f45fa3c7640 1 -- 172.21.15.83:0/598398069 shutdown_connections 2024-03-06T07:32:40.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.756+0000 7f45fa3c7640 1 --2- 172.21.15.83:0/598398069 >> v2:172.21.15.183:3300/0 conn(0x7f45f4162970 0x7f45f40b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.756+0000 7f45fa3c7640 1 --2- 172.21.15.83:0/598398069 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f45f4161cf0 0x7f45f4162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.756+0000 7f45fa3c7640 1 --2- 172.21.15.83:0/598398069 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45f40b4460 0x7f45f40d4800 unknown :-1 s=CLOSED pgs=2009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.756+0000 7f45fa3c7640 1 -- 172.21.15.83:0/598398069 >> 172.21.15.83:0/598398069 conn(0x7f45f40d3ce0 msgr2=0x7f45f40d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:40.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.756+0000 7f45fa3c7640 1 -- 172.21.15.83:0/598398069 shutdown_connections 2024-03-06T07:32:40.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.756+0000 7f45fa3c7640 1 -- 172.21.15.83:0/598398069 wait complete. 2024-03-06T07:32:40.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.757+0000 7f45fa3c7640 1 Processor -- start 2024-03-06T07:32:40.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.757+0000 7f45fa3c7640 1 -- start start 2024-03-06T07:32:40.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.758+0000 7f45fa3c7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f45f40b4460 0x7f45f40b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.758+0000 7f45fa3c7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45f4161cf0 0x7f45f40cc620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.758+0000 7f45fa3c7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f45f4162970 0x7f45f4155d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.758+0000 7f45fa3c7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f40ce950 con 0x7f45f4161cf0 2024-03-06T07:32:40.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.758+0000 7f45fa3c7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f40d2b60 con 0x7f45f40b4460 2024-03-06T07:32:40.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.758+0000 7f45fa3c7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45f40b3ac0 con 0x7f45f4162970 2024-03-06T07:32:40.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.758+0000 7f45f37fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45f4161cf0 0x7f45f40cc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.758+0000 7f45f37fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45f4161cf0 0x7f45f40cc620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50664/0 (socket says 172.21.15.83:50664) 2024-03-06T07:32:40.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.758+0000 7f45f37fe640 1 -- 172.21.15.83:0/4234892311 learned_addr learned my addr 172.21.15.83:0/4234892311 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:40.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.758+0000 7f45f8964640 1 --2- 172.21.15.83:0/4234892311 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f45f4162970 0x7f45f4155d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.758+0000 7f45f3fff640 1 --2- 172.21.15.83:0/4234892311 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f45f40b4460 0x7f45f40b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.758+0000 7f45f37fe640 1 -- 172.21.15.83:0/4234892311 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f45f4162970 msgr2=0x7f45f4155d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.758+0000 7f45f37fe640 1 --2- 172.21.15.83:0/4234892311 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f45f4162970 0x7f45f4155d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.758+0000 7f45f37fe640 1 -- 172.21.15.83:0/4234892311 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f45f40b4460 msgr2=0x7f45f40b21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.758+0000 7f45f37fe640 1 --2- 172.21.15.83:0/4234892311 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f45f40b4460 0x7f45f40b21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.758+0000 7f45f37fe640 1 -- 172.21.15.83:0/4234892311 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45f40cd0a0 con 0x7f45f4161cf0 2024-03-06T07:32:40.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.758+0000 7f45f8964640 1 --2- 172.21.15.83:0/4234892311 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f45f4162970 0x7f45f4155d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:40.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.758+0000 7f45f37fe640 1 --2- 172.21.15.83:0/4234892311 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45f4161cf0 0x7f45f40cc620 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f45e800d800 tx=0x7f45e800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:40.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.759+0000 7f45f0ff9640 1 -- 172.21.15.83:0/4234892311 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e800b6c0 con 0x7f45f4161cf0 2024-03-06T07:32:40.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.759+0000 7f45f0ff9640 1 -- 172.21.15.83:0/4234892311 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45e8019070 con 0x7f45f4161cf0 2024-03-06T07:32:40.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.759+0000 7f45fa3c7640 1 -- 172.21.15.83:0/4234892311 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45f40cd380 con 0x7f45f4161cf0 2024-03-06T07:32:40.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.759+0000 7f45f0ff9640 1 -- 172.21.15.83:0/4234892311 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e800ba80 con 0x7f45f4161cf0 2024-03-06T07:32:40.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.759+0000 7f45fa3c7640 1 -- 172.21.15.83:0/4234892311 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45f40cd940 con 0x7f45f4161cf0 2024-03-06T07:32:40.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.760+0000 7f45fa3c7640 1 -- 172.21.15.83:0/4234892311 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45f404f330 con 0x7f45f4161cf0 2024-03-06T07:32:40.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.760+0000 7f45f0ff9640 1 -- 172.21.15.83:0/4234892311 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f45e800bc20 con 0x7f45f4161cf0 2024-03-06T07:32:40.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.760+0000 7f45f0ff9640 1 --2- 172.21.15.83:0/4234892311 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f45e006b4d0 0x7f45e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.761+0000 7f45f0ff9640 1 -- 172.21.15.83:0/4234892311 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f45e808a5d0 con 0x7f45f4161cf0 2024-03-06T07:32:40.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.761+0000 7f45f3fff640 1 --2- 172.21.15.83:0/4234892311 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f45e006b4d0 0x7f45e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.761+0000 7f45f3fff640 1 --2- 172.21.15.83:0/4234892311 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f45e006b4d0 0x7f45e006d990 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f45e4036000 tx=0x7f45e403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:40.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.765+0000 7f45f0ff9640 1 -- 172.21.15.83:0/4234892311 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f45e80594a0 con 0x7f45f4161cf0 2024-03-06T07:32:40.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.982+0000 7f45fa3c7640 1 -- 172.21.15.83:0/4234892311 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f45f406b180 con 0x7f45f4161cf0 2024-03-06T07:32:40.984 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-03-06T07:32:40.983+0000 7fb0dae11640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:32:40.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.984+0000 7f45f0ff9640 1 -- 172.21.15.83:0/4234892311 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v84) v1 ==== 161+0+0 (secure 0 0 0) 0x7f45e8058da0 con 0x7f45f4161cf0 2024-03-06T07:32:40.986 INFO:teuthology.orchestra.run.smithi083.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:32:40.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.987+0000 7f45fa3c7640 1 -- 172.21.15.83:0/4234892311 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f45e006b4d0 msgr2=0x7f45e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.987+0000 7f45fa3c7640 1 --2- 172.21.15.83:0/4234892311 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f45e006b4d0 0x7f45e006d990 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f45e4036000 tx=0x7f45e403c040 comp rx=0 tx=0).stop 2024-03-06T07:32:40.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.987+0000 7f45fa3c7640 1 -- 172.21.15.83:0/4234892311 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45f4161cf0 msgr2=0x7f45f40cc620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.987+0000 7f45fa3c7640 1 --2- 172.21.15.83:0/4234892311 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45f4161cf0 0x7f45f40cc620 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f45e800d800 tx=0x7f45e800dcc0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.987+0000 7f45fa3c7640 1 -- 172.21.15.83:0/4234892311 shutdown_connections 2024-03-06T07:32:40.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.987+0000 7f45fa3c7640 1 --2- 172.21.15.83:0/4234892311 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f45e006b4d0 0x7f45e006d990 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.987+0000 7f45fa3c7640 1 --2- 172.21.15.83:0/4234892311 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f45f4162970 0x7f45f4155d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.987+0000 7f45fa3c7640 1 --2- 172.21.15.83:0/4234892311 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45f4161cf0 0x7f45f40cc620 unknown :-1 s=CLOSED pgs=2010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.987+0000 7f45fa3c7640 1 --2- 172.21.15.83:0/4234892311 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f45f40b4460 0x7f45f40b21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.987+0000 7f45fa3c7640 1 -- 172.21.15.83:0/4234892311 >> 172.21.15.83:0/4234892311 conn(0x7f45f40d3ce0 msgr2=0x7f45f40d02d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:40.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.987+0000 7f45fa3c7640 1 -- 172.21.15.83:0/4234892311 shutdown_connections 2024-03-06T07:32:40.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:40.987+0000 7f45fa3c7640 1 -- 172.21.15.83:0/4234892311 wait complete. 2024-03-06T07:32:41.004 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:32:41.031 INFO:tasks.ceph.mds.a.smithi083.stdout:starting mds.a at 2024-03-06T07:32:41.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.107+0000 7fc515f25640 1 Processor -- start 2024-03-06T07:32:41.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.107+0000 7fc515f25640 1 -- start start 2024-03-06T07:32:41.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.107+0000 7fc515f25640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc5100b1e40 0x7fc510168b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:41.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.107+0000 7fc515f25640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5100d2730 con 0x7fc5100d4860 2024-03-06T07:32:41.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.107+0000 7fc515f25640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5101618b0 con 0x7fc5101614d0 2024-03-06T07:32:41.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.107+0000 7fc515f25640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5100d4cf0 con 0x7fc5100b1e40 2024-03-06T07:32:41.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.108+0000 7fc50ffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc5100b1e40 0x7fc510168b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:41.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.108+0000 7fc50ffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc5100b1e40 0x7fc510168b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50692/0 (socket says 172.21.15.83:50692) 2024-03-06T07:32:41.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.108+0000 7fc50ffff640 1 -- 172.21.15.83:0/3783949465 learned_addr learned my addr 172.21.15.83:0/3783949465 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:41.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.108+0000 7fc507fff640 1 --1- 172.21.15.83:0/3783949465 >> v1:172.21.15.83:6789/0 conn(0x7fc5100d4860 0x7fc5100d2020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57640/0 (socket says 172.21.15.83:57640) 2024-03-06T07:32:41.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.108+0000 7fc50ffff640 1 -- 172.21.15.83:0/3783949465 >> v1:172.21.15.183:6789/0 conn(0x7fc5101614d0 legacy=0x7fc5100d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:41.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.108+0000 7fc50ffff640 1 -- 172.21.15.83:0/3783949465 >> v1:172.21.15.83:6789/0 conn(0x7fc5100d4860 legacy=0x7fc5100d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:41.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.108+0000 7fc50ffff640 1 -- 172.21.15.83:0/3783949465 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5100cf400 con 0x7fc5100b1e40 2024-03-06T07:32:41.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.109+0000 7fc50ffff640 1 --2- 172.21.15.83:0/3783949465 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5100b1e40 0x7fc510168b60 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7fc5100cf5d0 tx=0x7fc50802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1832cdb77df4b804 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:41.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.109+0000 7fc50effd640 1 -- 172.21.15.83:0/3783949465 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50802ca70 con 0x7fc5100b1e40 2024-03-06T07:32:41.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.109+0000 7fc50effd640 1 -- 172.21.15.83:0/3783949465 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc50802cc10 con 0x7fc5100b1e40 2024-03-06T07:32:41.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.109+0000 7fc50effd640 1 -- 172.21.15.83:0/3783949465 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5080343f0 con 0x7fc5100b1e40 2024-03-06T07:32:41.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.109+0000 7fc515f25640 1 -- 172.21.15.83:0/3783949465 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5100b1e40 msgr2=0x7fc510168b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:41.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.109+0000 7fc515f25640 1 --2- 172.21.15.83:0/3783949465 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5100b1e40 0x7fc510168b60 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7fc5100cf5d0 tx=0x7fc50802bed0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.109+0000 7fc515f25640 1 -- 172.21.15.83:0/3783949465 shutdown_connections 2024-03-06T07:32:41.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.109+0000 7fc515f25640 1 --2- 172.21.15.83:0/3783949465 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5100b1e40 0x7fc510168b60 unknown :-1 s=CLOSED pgs=2014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.109+0000 7fc515f25640 1 -- 172.21.15.83:0/3783949465 >> 172.21.15.83:0/3783949465 conn(0x7fc5100d3d40 msgr2=0x7fc5100d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:41.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.109+0000 7fc515f25640 1 -- 172.21.15.83:0/3783949465 shutdown_connections 2024-03-06T07:32:41.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.109+0000 7fc515f25640 1 -- 172.21.15.83:0/3783949465 wait complete. 2024-03-06T07:32:41.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.111+0000 7fc515f25640 1 Processor -- start 2024-03-06T07:32:41.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.111+0000 7fc515f25640 1 -- start start 2024-03-06T07:32:41.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.111+0000 7fc515f25640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc5100b1e40 0x7fc5100cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:41.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.111+0000 7fc515f25640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5100d4860 0x7fc5100ca300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:41.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.111+0000 7fc515f25640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5101614d0 0x7fc510155db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:41.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.111+0000 7fc515f25640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5100cfc60 con 0x7fc5101614d0 2024-03-06T07:32:41.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.111+0000 7fc515f25640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5100cf980 con 0x7fc5100d4860 2024-03-06T07:32:41.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.111+0000 7fc515f25640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5100b0910 con 0x7fc5100b1e40 2024-03-06T07:32:41.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.111+0000 7fc50ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5101614d0 0x7fc510155db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:41.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.111+0000 7fc50ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5101614d0 0x7fc510155db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50700/0 (socket says 172.21.15.83:50700) 2024-03-06T07:32:41.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.111+0000 7fc50ffff640 1 -- 172.21.15.83:0/819933633 learned_addr learned my addr 172.21.15.83:0/819933633 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:41.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.111+0000 7fc50f7fe640 1 --2- 172.21.15.83:0/819933633 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc5100b1e40 0x7fc5100cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:41.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.111+0000 7fc507fff640 1 --2- 172.21.15.83:0/819933633 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5100d4860 0x7fc5100ca300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:41.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.112+0000 7fc50ffff640 1 -- 172.21.15.83:0/819933633 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc5100b1e40 msgr2=0x7fc5100cd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:41.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.112+0000 7fc50ffff640 1 --2- 172.21.15.83:0/819933633 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc5100b1e40 0x7fc5100cd8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.112+0000 7fc50ffff640 1 -- 172.21.15.83:0/819933633 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5100d4860 msgr2=0x7fc5100ca300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:41.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.112+0000 7fc50ffff640 1 --2- 172.21.15.83:0/819933633 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5100d4860 0x7fc5100ca300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.112+0000 7fc50ffff640 1 -- 172.21.15.83:0/819933633 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5100b7990 con 0x7fc5101614d0 2024-03-06T07:32:41.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.112+0000 7fc50ffff640 1 --2- 172.21.15.83:0/819933633 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5101614d0 0x7fc510155db0 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7fc508036040 tx=0x7fc508004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:41.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.112+0000 7fc50cff9640 1 -- 172.21.15.83:0/819933633 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc508043020 con 0x7fc5101614d0 2024-03-06T07:32:41.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.112+0000 7fc50cff9640 1 -- 172.21.15.83:0/819933633 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc50803e450 con 0x7fc5101614d0 2024-03-06T07:32:41.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.112+0000 7fc515f25640 1 -- 172.21.15.83:0/819933633 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5100c9820 con 0x7fc5101614d0 2024-03-06T07:32:41.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.112+0000 7fc50cff9640 1 -- 172.21.15.83:0/819933633 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50803e860 con 0x7fc5101614d0 2024-03-06T07:32:41.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.112+0000 7fc515f25640 1 -- 172.21.15.83:0/819933633 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5100b7c70 con 0x7fc5101614d0 2024-03-06T07:32:41.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.113+0000 7fc515f25640 1 -- 172.21.15.83:0/819933633 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc510068ff0 con 0x7fc5101614d0 2024-03-06T07:32:41.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.113+0000 7fc50cff9640 1 -- 172.21.15.83:0/819933633 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc508046020 con 0x7fc5101614d0 2024-03-06T07:32:41.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.114+0000 7fc50cff9640 1 --2- 172.21.15.83:0/819933633 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc4f006b4d0 0x7fc4f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:41.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.114+0000 7fc50cff9640 1 -- 172.21.15.83:0/819933633 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fc5080acc60 con 0x7fc5101614d0 2024-03-06T07:32:41.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.114+0000 7fc50f7fe640 1 --2- 172.21.15.83:0/819933633 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc4f006b4d0 0x7fc4f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:41.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.114+0000 7fc50f7fe640 1 --2- 172.21.15.83:0/819933633 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc4f006b4d0 0x7fc4f006d990 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fc500004ea0 tx=0x7fc500009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:41.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.118+0000 7fc50cff9640 1 -- 172.21.15.83:0/819933633 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc50804d030 con 0x7fc5101614d0 2024-03-06T07:32:41.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.328+0000 7fc515f25640 1 -- 172.21.15.83:0/819933633 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc51006d2a0 con 0x7fc5101614d0 2024-03-06T07:32:41.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.985+0000 7fc50cff9640 1 -- 172.21.15.83:0/819933633 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v85) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc50807b9f0 con 0x7fc5101614d0 2024-03-06T07:32:41.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.988+0000 7fc515f25640 1 -- 172.21.15.83:0/819933633 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc4f006b4d0 msgr2=0x7fc4f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:41.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.988+0000 7fc515f25640 1 --2- 172.21.15.83:0/819933633 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc4f006b4d0 0x7fc4f006d990 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fc500004ea0 tx=0x7fc500009040 comp rx=0 tx=0).stop 2024-03-06T07:32:41.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.988+0000 7fc515f25640 1 -- 172.21.15.83:0/819933633 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5101614d0 msgr2=0x7fc510155db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:41.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.988+0000 7fc515f25640 1 --2- 172.21.15.83:0/819933633 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5101614d0 0x7fc510155db0 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7fc508036040 tx=0x7fc508004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:41.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.988+0000 7fc515f25640 1 -- 172.21.15.83:0/819933633 shutdown_connections 2024-03-06T07:32:41.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.988+0000 7fc515f25640 1 --2- 172.21.15.83:0/819933633 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc4f006b4d0 0x7fc4f006d990 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.988+0000 7fc515f25640 1 --2- 172.21.15.83:0/819933633 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5101614d0 0x7fc510155db0 unknown :-1 s=CLOSED pgs=2015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.988+0000 7fc515f25640 1 --2- 172.21.15.83:0/819933633 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5100d4860 0x7fc5100ca300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.988+0000 7fc515f25640 1 --2- 172.21.15.83:0/819933633 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc5100b1e40 0x7fc5100cd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.988+0000 7fc515f25640 1 -- 172.21.15.83:0/819933633 >> 172.21.15.83:0/819933633 conn(0x7fc5100d3d40 msgr2=0x7fc51006c7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:41.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.988+0000 7fc515f25640 1 -- 172.21.15.83:0/819933633 shutdown_connections 2024-03-06T07:32:41.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:41.988+0000 7fc515f25640 1 -- 172.21.15.83:0/819933633 wait complete. 2024-03-06T07:32:42.004 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:32:42.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.151+0000 7ff9dd2a4640 1 Processor -- start 2024-03-06T07:32:42.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.151+0000 7ff9dd2a4640 1 -- start start 2024-03-06T07:32:42.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.152+0000 7ff9dd2a4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff9d815dff0 0x7ff9d80cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:42.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.152+0000 7ff9dd2a4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9d80d1250 0x7ff9d80b3e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:42.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.152+0000 7ff9dd2a4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9d80cf8b0 con 0x7ff9d80d06b0 2024-03-06T07:32:42.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.152+0000 7ff9dd2a4640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9d80cfb30 con 0x7ff9d815dff0 2024-03-06T07:32:42.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.152+0000 7ff9dd2a4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d80b3320 con 0x7ff9d80d1250 2024-03-06T07:32:42.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.152+0000 7ff9d6ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff9d815dff0 0x7ff9d80cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:42.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.152+0000 7ff9d67fc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff9d80d06b0 0x7ff9d80d0b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57652/0 (socket says 172.21.15.83:57652) 2024-03-06T07:32:42.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.153+0000 7ff9d67fc640 1 -- 172.21.15.83:0/1049042322 learned_addr learned my addr 172.21.15.83:0/1049042322 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:32:42.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.153+0000 7ff9d77fe640 1 --2- 172.21.15.83:0/1049042322 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9d80d1250 0x7ff9d80b3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:42.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.153+0000 7ff9d6ffd640 1 -- 172.21.15.83:0/1049042322 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9d80d1250 msgr2=0x7ff9d80b3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:42.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.153+0000 7ff9d6ffd640 1 --2- 172.21.15.83:0/1049042322 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9d80d1250 0x7ff9d80b3e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.153+0000 7ff9d6ffd640 1 -- 172.21.15.83:0/1049042322 >> v1:172.21.15.83:6789/0 conn(0x7ff9d80d06b0 legacy=0x7ff9d80d0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:42.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.153+0000 7ff9d5ffb640 1 -- 172.21.15.83:0/1049042322 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1089538111 0 0) 0x7ff9c0001110 con 0x7ff9d80d06b0 2024-03-06T07:32:42.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.153+0000 7ff9d6ffd640 1 -- 172.21.15.83:0/1049042322 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d80b3600 con 0x7ff9d815dff0 2024-03-06T07:32:42.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.154+0000 7ff9d6ffd640 1 --2- 172.21.15.83:0/1049042322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9d815dff0 0x7ff9d80cf370 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7ff9c8009d90 tx=0x7ff9c802bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c0d02f2a4464634 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:42.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.154+0000 7ff9d5ffb640 1 -- 172.21.15.83:0/1049042322 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c8039070 con 0x7ff9d815dff0 2024-03-06T07:32:42.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.154+0000 7ff9d5ffb640 1 -- 172.21.15.83:0/1049042322 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9c802cce0 con 0x7ff9d815dff0 2024-03-06T07:32:42.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.154+0000 7ff9d5ffb640 1 -- 172.21.15.83:0/1049042322 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c8034660 con 0x7ff9d815dff0 2024-03-06T07:32:42.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.154+0000 7ff9dd2a4640 1 -- 172.21.15.83:0/1049042322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9d815dff0 msgr2=0x7ff9d80cf370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:42.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.154+0000 7ff9dd2a4640 1 --2- 172.21.15.83:0/1049042322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9d815dff0 0x7ff9d80cf370 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7ff9c8009d90 tx=0x7ff9c802bf30 comp rx=0 tx=0).stop 2024-03-06T07:32:42.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.154+0000 7ff9dd2a4640 1 -- 172.21.15.83:0/1049042322 shutdown_connections 2024-03-06T07:32:42.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.154+0000 7ff9dd2a4640 1 --2- 172.21.15.83:0/1049042322 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9d80d1250 0x7ff9d80b3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.154+0000 7ff9dd2a4640 1 --2- 172.21.15.83:0/1049042322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9d815dff0 0x7ff9d80cf370 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.154+0000 7ff9dd2a4640 1 -- 172.21.15.83:0/1049042322 >> 172.21.15.83:0/1049042322 conn(0x7ff9d80d3d50 msgr2=0x7ff9d80d4160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:42.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.154+0000 7ff9dd2a4640 1 -- 172.21.15.83:0/1049042322 shutdown_connections 2024-03-06T07:32:42.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.154+0000 7ff9dd2a4640 1 -- 172.21.15.83:0/1049042322 wait complete. 2024-03-06T07:32:42.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.156+0000 7ff9dd2a4640 1 Processor -- start 2024-03-06T07:32:42.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.156+0000 7ff9dd2a4640 1 -- start start 2024-03-06T07:32:42.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.156+0000 7ff9dd2a4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff9d80d06b0 0x7ff9d8160610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:42.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.156+0000 7ff9dd2a4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9d80d1250 0x7ff9d80b7f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:42.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.156+0000 7ff9dd2a4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9d815dff0 0x7ff9d80b8490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:42.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.156+0000 7ff9dd2a4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d815e550 con 0x7ff9d80d1250 2024-03-06T07:32:42.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.156+0000 7ff9dd2a4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d815e3d0 con 0x7ff9d80d06b0 2024-03-06T07:32:42.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.156+0000 7ff9dd2a4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d80b4340 con 0x7ff9d815dff0 2024-03-06T07:32:42.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.156+0000 7ff9d67fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9d80d1250 0x7ff9d80b7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:42.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.156+0000 7ff9d67fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9d80d1250 0x7ff9d80b7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50730/0 (socket says 172.21.15.83:50730) 2024-03-06T07:32:42.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.156+0000 7ff9d67fc640 1 -- 172.21.15.83:0/2528455538 learned_addr learned my addr 172.21.15.83:0/2528455538 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:42.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.156+0000 7ff9d6ffd640 1 --2- 172.21.15.83:0/2528455538 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff9d80d06b0 0x7ff9d8160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:42.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.157+0000 7ff9d77fe640 1 --2- 172.21.15.83:0/2528455538 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9d815dff0 0x7ff9d80b8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:42.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.157+0000 7ff9d67fc640 1 -- 172.21.15.83:0/2528455538 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9d815dff0 msgr2=0x7ff9d80b8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:42.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.157+0000 7ff9d67fc640 1 --2- 172.21.15.83:0/2528455538 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9d815dff0 0x7ff9d80b8490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.157+0000 7ff9d67fc640 1 -- 172.21.15.83:0/2528455538 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff9d80d06b0 msgr2=0x7ff9d8160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:42.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.157+0000 7ff9d67fc640 1 --2- 172.21.15.83:0/2528455538 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff9d80d06b0 0x7ff9d8160610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.157+0000 7ff9d67fc640 1 -- 172.21.15.83:0/2528455538 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d8160b50 con 0x7ff9d80d1250 2024-03-06T07:32:42.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.157+0000 7ff9d77fe640 1 --2- 172.21.15.83:0/2528455538 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9d815dff0 0x7ff9d80b8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:42.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.157+0000 7ff9d67fc640 1 --2- 172.21.15.83:0/2528455538 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9d80d1250 0x7ff9d80b7f50 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7ff9c000c780 tx=0x7ff9c000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:42.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.157+0000 7ff9cf7fe640 1 -- 172.21.15.83:0/2528455538 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c0018070 con 0x7ff9d80d1250 2024-03-06T07:32:42.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.157+0000 7ff9cf7fe640 1 -- 172.21.15.83:0/2528455538 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9c00072d0 con 0x7ff9d80d1250 2024-03-06T07:32:42.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.157+0000 7ff9dd2a4640 1 -- 172.21.15.83:0/2528455538 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9d80cc6f0 con 0x7ff9d80d1250 2024-03-06T07:32:42.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.157+0000 7ff9cf7fe640 1 -- 172.21.15.83:0/2528455538 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c00047e0 con 0x7ff9d80d1250 2024-03-06T07:32:42.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.157+0000 7ff9dd2a4640 1 -- 172.21.15.83:0/2528455538 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9d80b8f00 con 0x7ff9d80d1250 2024-03-06T07:32:42.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.158+0000 7ff9dd2a4640 1 -- 172.21.15.83:0/2528455538 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9d804f330 con 0x7ff9d80d1250 2024-03-06T07:32:42.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.158+0000 7ff9cf7fe640 1 -- 172.21.15.83:0/2528455538 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff9c001b070 con 0x7ff9d80d1250 2024-03-06T07:32:42.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.159+0000 7ff9cf7fe640 1 --2- 172.21.15.83:0/2528455538 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff9b806b4d0 0x7ff9b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:42.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.159+0000 7ff9cf7fe640 1 -- 172.21.15.83:0/2528455538 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5939+0+0 (secure 0 0 0) 0x7ff9c0089630 con 0x7ff9d80d1250 2024-03-06T07:32:42.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.159+0000 7ff9d6ffd640 1 --2- 172.21.15.83:0/2528455538 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff9b806b4d0 0x7ff9b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:42.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.159+0000 7ff9d6ffd640 1 --2- 172.21.15.83:0/2528455538 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff9b806b4d0 0x7ff9b806d990 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7ff9c8036000 tx=0x7ff9c8030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:42.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.163+0000 7ff9cf7fe640 1 -- 172.21.15.83:0/2528455538 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff9d80cf8b0 con 0x7ff9d80d1250 2024-03-06T07:32:42.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.335+0000 7ff9dd2a4640 1 -- 172.21.15.83:0/2528455538 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff9d815c760 con 0x7ff9d80d1250 2024-03-06T07:32:42.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.990+0000 7ff9cf7fe640 1 -- 172.21.15.83:0/2528455538 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v134) v1 ==== 181+0+0 (secure 0 0 0) 0x7ff9c0057ce0 con 0x7ff9d80d1250 2024-03-06T07:32:42.992 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:32:42.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.993+0000 7ff9dd2a4640 1 -- 172.21.15.83:0/2528455538 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff9b806b4d0 msgr2=0x7ff9b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:42.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.993+0000 7ff9dd2a4640 1 --2- 172.21.15.83:0/2528455538 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff9b806b4d0 0x7ff9b806d990 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7ff9c8036000 tx=0x7ff9c8030000 comp rx=0 tx=0).stop 2024-03-06T07:32:42.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.993+0000 7ff9dd2a4640 1 -- 172.21.15.83:0/2528455538 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9d80d1250 msgr2=0x7ff9d80b7f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:42.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.993+0000 7ff9dd2a4640 1 --2- 172.21.15.83:0/2528455538 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9d80d1250 0x7ff9d80b7f50 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7ff9c000c780 tx=0x7ff9c000cc40 comp rx=0 tx=0).stop 2024-03-06T07:32:42.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.993+0000 7ff9dd2a4640 1 -- 172.21.15.83:0/2528455538 shutdown_connections 2024-03-06T07:32:42.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.993+0000 7ff9dd2a4640 1 --2- 172.21.15.83:0/2528455538 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff9b806b4d0 0x7ff9b806d990 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.993+0000 7ff9dd2a4640 1 --2- 172.21.15.83:0/2528455538 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9d815dff0 0x7ff9d80b8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.993+0000 7ff9dd2a4640 1 --2- 172.21.15.83:0/2528455538 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9d80d1250 0x7ff9d80b7f50 secure :-1 s=CLOSED pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7ff9c000c780 tx=0x7ff9c000cc40 comp rx=0 tx=0).stop 2024-03-06T07:32:42.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.993+0000 7ff9dd2a4640 1 --2- 172.21.15.83:0/2528455538 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff9d80d06b0 0x7ff9d8160610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.993+0000 7ff9dd2a4640 1 -- 172.21.15.83:0/2528455538 >> 172.21.15.83:0/2528455538 conn(0x7ff9d80d3d50 msgr2=0x7ff9d80d4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:42.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.994+0000 7ff9dd2a4640 1 -- 172.21.15.83:0/2528455538 shutdown_connections 2024-03-06T07:32:42.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:42.994+0000 7ff9dd2a4640 1 -- 172.21.15.83:0/2528455538 wait complete. 2024-03-06T07:32:43.008 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:32:43.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.114+0000 7fb4d0415640 1 Processor -- start 2024-03-06T07:32:43.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.114+0000 7fb4d0415640 1 -- start start 2024-03-06T07:32:43.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.115+0000 7fb4d0415640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb4c815df60 0x7fb4c80b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.115+0000 7fb4d0415640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb4c80aea50 0x7fb4c80708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.115+0000 7fb4d0415640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4c80b3570 con 0x7fb4c806cde0 2024-03-06T07:32:43.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.115+0000 7fb4d0415640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4c815e340 con 0x7fb4c80aea50 2024-03-06T07:32:43.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.115+0000 7fb4d0415640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c80b3850 con 0x7fb4c815df60 2024-03-06T07:32:43.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.115+0000 7fb4ce9b2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb4c80aea50 0x7fb4c80708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:43.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.115+0000 7fb4ce9b2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb4c80aea50 0x7fb4c80708b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50736/0 (socket says 172.21.15.83:50736) 2024-03-06T07:32:43.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.115+0000 7fb4ce9b2640 1 -- 172.21.15.83:0/2350198794 learned_addr learned my addr 172.21.15.83:0/2350198794 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:43.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.115+0000 7fb4ce1b1640 1 --2- 172.21.15.83:0/2350198794 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb4c815df60 0x7fb4c80b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:43.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.115+0000 7fb4ce9b2640 1 -- 172.21.15.83:0/2350198794 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb4c815df60 msgr2=0x7fb4c80b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:43.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.115+0000 7fb4ce9b2640 1 --2- 172.21.15.83:0/2350198794 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb4c815df60 0x7fb4c80b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.115+0000 7fb4ce9b2640 1 -- 172.21.15.83:0/2350198794 >> v1:172.21.15.183:6789/0 conn(0x7fb4c806cde0 legacy=0x7fb4c80ae340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:43.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.115+0000 7fb4ce9b2640 1 -- 172.21.15.83:0/2350198794 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c80ad860 con 0x7fb4c80aea50 2024-03-06T07:32:43.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.116+0000 7fb4ce9b2640 1 --2- 172.21.15.83:0/2350198794 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4c80aea50 0x7fb4c80708b0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7fb4c0009e50 tx=0x7fb4c002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=365e9d9a22001795 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:43.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.116+0000 7fb4cd1af640 1 -- 172.21.15.83:0/2350198794 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c002ca70 con 0x7fb4c80aea50 2024-03-06T07:32:43.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.116+0000 7fb4cd1af640 1 -- 172.21.15.83:0/2350198794 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4c002cc10 con 0x7fb4c80aea50 2024-03-06T07:32:43.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.116+0000 7fb4cd1af640 1 -- 172.21.15.83:0/2350198794 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c00343f0 con 0x7fb4c80aea50 2024-03-06T07:32:43.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.116+0000 7fb4d0415640 1 -- 172.21.15.83:0/2350198794 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4c80aea50 msgr2=0x7fb4c80708b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:43.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.116+0000 7fb4d0415640 1 --2- 172.21.15.83:0/2350198794 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4c80aea50 0x7fb4c80708b0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7fb4c0009e50 tx=0x7fb4c002bed0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.117+0000 7fb4d0415640 1 -- 172.21.15.83:0/2350198794 shutdown_connections 2024-03-06T07:32:43.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.117+0000 7fb4d0415640 1 --2- 172.21.15.83:0/2350198794 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4c80aea50 0x7fb4c80708b0 unknown :-1 s=CLOSED pgs=2019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.117+0000 7fb4d0415640 1 --2- 172.21.15.83:0/2350198794 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb4c815df60 0x7fb4c80b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.117+0000 7fb4d0415640 1 -- 172.21.15.83:0/2350198794 >> 172.21.15.83:0/2350198794 conn(0x7fb4c80b1650 msgr2=0x7fb4c80b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:43.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.117+0000 7fb4d0415640 1 -- 172.21.15.83:0/2350198794 shutdown_connections 2024-03-06T07:32:43.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.117+0000 7fb4d0415640 1 -- 172.21.15.83:0/2350198794 wait complete. 2024-03-06T07:32:43.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.118+0000 7fb4d0415640 1 Processor -- start 2024-03-06T07:32:43.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.118+0000 7fb4d0415640 1 -- start start 2024-03-06T07:32:43.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.118+0000 7fb4d0415640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4c806cde0 0x7fb4c815bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.118+0000 7fb4d0415640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb4c80aea50 0x7fb4c8141a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.118+0000 7fb4d0415640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb4c815df60 0x7fb4c8141f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.118+0000 7fb4d0415640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c80aef00 con 0x7fb4c806cde0 2024-03-06T07:32:43.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.118+0000 7fb4d0415640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c80ade20 con 0x7fb4c80aea50 2024-03-06T07:32:43.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.118+0000 7fb4d0415640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c80af080 con 0x7fb4c815df60 2024-03-06T07:32:43.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.119+0000 7fb4ce1b1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4c806cde0 0x7fb4c815bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:43.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.119+0000 7fb4ce1b1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4c806cde0 0x7fb4c815bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50744/0 (socket says 172.21.15.83:50744) 2024-03-06T07:32:43.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.119+0000 7fb4ce1b1640 1 -- 172.21.15.83:0/256454918 learned_addr learned my addr 172.21.15.83:0/256454918 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:43.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.119+0000 7fb4cd9b0640 1 --2- 172.21.15.83:0/256454918 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb4c80aea50 0x7fb4c8141a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:43.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.119+0000 7fb4ce9b2640 1 --2- 172.21.15.83:0/256454918 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb4c815df60 0x7fb4c8141f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:43.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.119+0000 7fb4ce1b1640 1 -- 172.21.15.83:0/256454918 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb4c815df60 msgr2=0x7fb4c8141f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:43.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.119+0000 7fb4ce1b1640 1 --2- 172.21.15.83:0/256454918 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb4c815df60 0x7fb4c8141f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.119+0000 7fb4ce1b1640 1 -- 172.21.15.83:0/256454918 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb4c80aea50 msgr2=0x7fb4c8141a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:43.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.119+0000 7fb4ce1b1640 1 --2- 172.21.15.83:0/256454918 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb4c80aea50 0x7fb4c8141a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.119+0000 7fb4ce1b1640 1 -- 172.21.15.83:0/256454918 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c815c4b0 con 0x7fb4c806cde0 2024-03-06T07:32:43.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.119+0000 7fb4cd9b0640 1 --2- 172.21.15.83:0/256454918 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb4c80aea50 0x7fb4c8141a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:43.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.119+0000 7fb4ce1b1640 1 --2- 172.21.15.83:0/256454918 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4c806cde0 0x7fb4c815bf70 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7fb4c400d5f0 tx=0x7fb4c400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:43.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.119+0000 7fb4beffd640 1 -- 172.21.15.83:0/256454918 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c40125f0 con 0x7fb4c806cde0 2024-03-06T07:32:43.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.119+0000 7fb4beffd640 1 -- 172.21.15.83:0/256454918 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4c4019070 con 0x7fb4c806cde0 2024-03-06T07:32:43.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.119+0000 7fb4beffd640 1 -- 172.21.15.83:0/256454918 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c40129d0 con 0x7fb4c806cde0 2024-03-06T07:32:43.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.119+0000 7fb4d0415640 1 -- 172.21.15.83:0/256454918 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4c815c750 con 0x7fb4c806cde0 2024-03-06T07:32:43.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.119+0000 7fb4d0415640 1 -- 172.21.15.83:0/256454918 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4c8152f10 con 0x7fb4c806cde0 2024-03-06T07:32:43.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.120+0000 7fb4d0415640 1 -- 172.21.15.83:0/256454918 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4c804f330 con 0x7fb4c806cde0 2024-03-06T07:32:43.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.121+0000 7fb4beffd640 1 -- 172.21.15.83:0/256454918 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb4c4012b70 con 0x7fb4c806cde0 2024-03-06T07:32:43.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.121+0000 7fb4beffd640 1 --2- 172.21.15.83:0/256454918 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb4ac06b4d0 0x7fb4ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.121+0000 7fb4cd9b0640 1 --2- 172.21.15.83:0/256454918 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb4ac06b4d0 0x7fb4ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:43.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.121+0000 7fb4beffd640 1 -- 172.21.15.83:0/256454918 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5501+0+0 (secure 0 0 0) 0x7fb4c408a4b0 con 0x7fb4c806cde0 2024-03-06T07:32:43.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.121+0000 7fb4cd9b0640 1 --2- 172.21.15.83:0/256454918 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb4ac06b4d0 0x7fb4ac06d990 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fb4c8148da0 tx=0x7fb4b8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:43.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.125+0000 7fb4beffd640 1 -- 172.21.15.83:0/256454918 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb4c4059490 con 0x7fb4c806cde0 2024-03-06T07:32:43.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.295+0000 7fb4d0415640 1 -- 172.21.15.83:0/256454918 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb4c80cbcf0 con 0x7fb4c806cde0 2024-03-06T07:32:43.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.992+0000 7fb4beffd640 1 -- 172.21.15.83:0/256454918 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v135) v1 ==== 169+0+0 (secure 0 0 0) 0x7fb4c4058d90 con 0x7fb4c806cde0 2024-03-06T07:32:43.994 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'cephfs_data' removed 2024-03-06T07:32:43.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.994+0000 7fb4d0415640 1 -- 172.21.15.83:0/256454918 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb4ac06b4d0 msgr2=0x7fb4ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:43.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.994+0000 7fb4d0415640 1 --2- 172.21.15.83:0/256454918 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb4ac06b4d0 0x7fb4ac06d990 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fb4c8148da0 tx=0x7fb4b8009040 comp rx=0 tx=0).stop 2024-03-06T07:32:43.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.994+0000 7fb4d0415640 1 -- 172.21.15.83:0/256454918 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4c806cde0 msgr2=0x7fb4c815bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:43.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.994+0000 7fb4d0415640 1 --2- 172.21.15.83:0/256454918 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4c806cde0 0x7fb4c815bf70 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7fb4c400d5f0 tx=0x7fb4c400dab0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.994+0000 7fb4d0415640 1 -- 172.21.15.83:0/256454918 shutdown_connections 2024-03-06T07:32:43.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.994+0000 7fb4d0415640 1 --2- 172.21.15.83:0/256454918 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb4ac06b4d0 0x7fb4ac06d990 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.994+0000 7fb4d0415640 1 --2- 172.21.15.83:0/256454918 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb4c815df60 0x7fb4c8141f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.994+0000 7fb4d0415640 1 --2- 172.21.15.83:0/256454918 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb4c80aea50 0x7fb4c8141a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.994+0000 7fb4d0415640 1 --2- 172.21.15.83:0/256454918 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4c806cde0 0x7fb4c815bf70 unknown :-1 s=CLOSED pgs=2020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.994+0000 7fb4d0415640 1 -- 172.21.15.83:0/256454918 >> 172.21.15.83:0/256454918 conn(0x7fb4c80b1650 msgr2=0x7fb4c80af500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:43.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.994+0000 7fb4d0415640 1 -- 172.21.15.83:0/256454918 shutdown_connections 2024-03-06T07:32:43.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:43.995+0000 7fb4d0415640 1 -- 172.21.15.83:0/256454918 wait complete. 2024-03-06T07:32:44.013 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:32:44.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.119+0000 7fc01edf7640 1 Processor -- start 2024-03-06T07:32:44.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.119+0000 7fc01edf7640 1 -- start start 2024-03-06T07:32:44.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.120+0000 7fc01edf7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc0180cf9a0 0x7fc0180b5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.120+0000 7fc01edf7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc0180b55e0 0x7fc0180ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.120+0000 7fc01edf7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc01806b710 con 0x7fc018066d90 2024-03-06T07:32:44.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.120+0000 7fc01edf7640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc01806b150 con 0x7fc0180cf9a0 2024-03-06T07:32:44.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.120+0000 7fc01edf7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc01806b430 con 0x7fc0180b55e0 2024-03-06T07:32:44.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.120+0000 7fc01cb93640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc0180cf9a0 0x7fc0180b5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.120+0000 7fc01cb93640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc0180cf9a0 0x7fc0180b5ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50748/0 (socket says 172.21.15.83:50748) 2024-03-06T07:32:44.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.120+0000 7fc017fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fc018066d90 0x7fc0180b4ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57668/0 (socket says 172.21.15.83:57668) 2024-03-06T07:32:44.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.120+0000 7fc01cb93640 1 -- 172.21.15.83:0/1953605133 learned_addr learned my addr 172.21.15.83:0/1953605133 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:44.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.120+0000 7fc01d394640 1 --2- 172.21.15.83:0/1953605133 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc0180b55e0 0x7fc0180ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.120+0000 7fc01cb93640 1 -- 172.21.15.83:0/1953605133 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc0180b55e0 msgr2=0x7fc0180ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:44.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.120+0000 7fc01cb93640 1 --2- 172.21.15.83:0/1953605133 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc0180b55e0 0x7fc0180ce040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.120+0000 7fc01cb93640 1 -- 172.21.15.83:0/1953605133 >> v1:172.21.15.83:6789/0 conn(0x7fc018066d90 legacy=0x7fc0180b4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.120+0000 7fc01cb93640 1 -- 172.21.15.83:0/1953605133 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0180b0970 con 0x7fc0180cf9a0 2024-03-06T07:32:44.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.120+0000 7fc0177fe640 1 -- 172.21.15.83:0/1953605133 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1769284663 0 0) 0x7fc01806b710 con 0x7fc018066d90 2024-03-06T07:32:44.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.121+0000 7fc01cb93640 1 --2- 172.21.15.83:0/1953605133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0180cf9a0 0x7fc0180b5ab0 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7fc00c009870 tx=0x7fc00c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=417ca1a534bf7277 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:44.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.121+0000 7fc0177fe640 1 -- 172.21.15.83:0/1953605133 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00c034070 con 0x7fc0180cf9a0 2024-03-06T07:32:44.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.121+0000 7fc0177fe640 1 -- 172.21.15.83:0/1953605133 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc00c030840 con 0x7fc0180cf9a0 2024-03-06T07:32:44.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.121+0000 7fc0177fe640 1 -- 172.21.15.83:0/1953605133 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00c030c10 con 0x7fc0180cf9a0 2024-03-06T07:32:44.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.121+0000 7fc01edf7640 1 -- 172.21.15.83:0/1953605133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0180cf9a0 msgr2=0x7fc0180b5ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.121+0000 7fc01edf7640 1 --2- 172.21.15.83:0/1953605133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0180cf9a0 0x7fc0180b5ab0 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7fc00c009870 tx=0x7fc00c02b980 comp rx=0 tx=0).stop 2024-03-06T07:32:44.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.121+0000 7fc01edf7640 1 -- 172.21.15.83:0/1953605133 shutdown_connections 2024-03-06T07:32:44.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.121+0000 7fc01edf7640 1 --2- 172.21.15.83:0/1953605133 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc0180b55e0 0x7fc0180ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.121+0000 7fc01edf7640 1 --2- 172.21.15.83:0/1953605133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0180cf9a0 0x7fc0180b5ab0 unknown :-1 s=CLOSED pgs=2022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.121+0000 7fc01edf7640 1 -- 172.21.15.83:0/1953605133 >> 172.21.15.83:0/1953605133 conn(0x7fc0180b1d70 msgr2=0x7fc0180b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:44.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.122+0000 7fc01edf7640 1 -- 172.21.15.83:0/1953605133 shutdown_connections 2024-03-06T07:32:44.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.122+0000 7fc01edf7640 1 -- 172.21.15.83:0/1953605133 wait complete. 2024-03-06T07:32:44.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.123+0000 7fc01edf7640 1 Processor -- start 2024-03-06T07:32:44.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.123+0000 7fc01edf7640 1 -- start start 2024-03-06T07:32:44.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.124+0000 7fc01edf7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc018066d90 0x7fc0180ae490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.124+0000 7fc01edf7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0180b55e0 0x7fc018157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.124+0000 7fc01edf7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc0180cf9a0 0x7fc018158510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.124+0000 7fc01edf7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0180b6170 con 0x7fc018066d90 2024-03-06T07:32:44.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.124+0000 7fc01edf7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0180b5ff0 con 0x7fc0180b55e0 2024-03-06T07:32:44.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.124+0000 7fc01edf7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0180cfd80 con 0x7fc0180cf9a0 2024-03-06T07:32:44.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.124+0000 7fc01cb93640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc018066d90 0x7fc0180ae490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.124+0000 7fc01cb93640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc018066d90 0x7fc0180ae490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50758/0 (socket says 172.21.15.83:50758) 2024-03-06T07:32:44.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.124+0000 7fc01cb93640 1 -- 172.21.15.83:0/3592951651 learned_addr learned my addr 172.21.15.83:0/3592951651 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:44.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.124+0000 7fc01d394640 1 --2- 172.21.15.83:0/3592951651 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc0180cf9a0 0x7fc018158510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.124+0000 7fc017fff640 1 --2- 172.21.15.83:0/3592951651 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0180b55e0 0x7fc018157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.124+0000 7fc01cb93640 1 -- 172.21.15.83:0/3592951651 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc0180cf9a0 msgr2=0x7fc018158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.124+0000 7fc01cb93640 1 --2- 172.21.15.83:0/3592951651 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc0180cf9a0 0x7fc018158510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.124+0000 7fc01cb93640 1 -- 172.21.15.83:0/3592951651 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0180b55e0 msgr2=0x7fc018157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.124+0000 7fc01cb93640 1 --2- 172.21.15.83:0/3592951651 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0180b55e0 0x7fc018157fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.124+0000 7fc01cb93640 1 -- 172.21.15.83:0/3592951651 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0180ad3f0 con 0x7fc018066d90 2024-03-06T07:32:44.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.124+0000 7fc01cb93640 1 --2- 172.21.15.83:0/3592951651 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc018066d90 0x7fc0180ae490 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7fc00c0090b0 tx=0x7fc00c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:44.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.124+0000 7fc0157fa640 1 -- 172.21.15.83:0/3592951651 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00c034040 con 0x7fc018066d90 2024-03-06T07:32:44.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.124+0000 7fc0157fa640 1 -- 172.21.15.83:0/3592951651 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc00c03ab90 con 0x7fc018066d90 2024-03-06T07:32:44.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.124+0000 7fc0157fa640 1 -- 172.21.15.83:0/3592951651 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00c004030 con 0x7fc018066d90 2024-03-06T07:32:44.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.124+0000 7fc01edf7640 1 -- 172.21.15.83:0/3592951651 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0180ad6d0 con 0x7fc018066d90 2024-03-06T07:32:44.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.124+0000 7fc01edf7640 1 -- 172.21.15.83:0/3592951651 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0180adc90 con 0x7fc018066d90 2024-03-06T07:32:44.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.125+0000 7fc01edf7640 1 -- 172.21.15.83:0/3592951651 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc01809e3d0 con 0x7fc018066d90 2024-03-06T07:32:44.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.126+0000 7fc0157fa640 1 -- 172.21.15.83:0/3592951651 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc00c037070 con 0x7fc018066d90 2024-03-06T07:32:44.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.126+0000 7fc0157fa640 1 --2- 172.21.15.83:0/3592951651 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbffc06b4d0 0x7fbffc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.126+0000 7fc0157fa640 1 -- 172.21.15.83:0/3592951651 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fc00c0abf70 con 0x7fc018066d90 2024-03-06T07:32:44.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.126+0000 7fc017fff640 1 --2- 172.21.15.83:0/3592951651 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbffc06b4d0 0x7fbffc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.127+0000 7fc017fff640 1 --2- 172.21.15.83:0/3592951651 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbffc06b4d0 0x7fbffc06d990 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fc004000f60 tx=0x7fc004007b70 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:44.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.131+0000 7fc0157fa640 1 -- 172.21.15.83:0/3592951651 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc00c03f070 con 0x7fc018066d90 2024-03-06T07:32:44.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.314+0000 7fc01edf7640 1 -- 172.21.15.83:0/3592951651 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fc018068ff0 con 0x7fc018066d90 2024-03-06T07:32:44.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.317+0000 7fc0157fa640 1 -- 172.21.15.83:0/3592951651 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v33) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc00c07abf0 con 0x7fc018066d90 2024-03-06T07:32:44.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.319+0000 7fc01edf7640 1 -- 172.21.15.83:0/3592951651 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbffc06b4d0 msgr2=0x7fbffc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.319+0000 7fc01edf7640 1 --2- 172.21.15.83:0/3592951651 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbffc06b4d0 0x7fbffc06d990 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fc004000f60 tx=0x7fc004007b70 comp rx=0 tx=0).stop 2024-03-06T07:32:44.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.319+0000 7fc01edf7640 1 -- 172.21.15.83:0/3592951651 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc018066d90 msgr2=0x7fc0180ae490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.319+0000 7fc01edf7640 1 --2- 172.21.15.83:0/3592951651 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc018066d90 0x7fc0180ae490 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7fc00c0090b0 tx=0x7fc00c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.319+0000 7fc01edf7640 1 -- 172.21.15.83:0/3592951651 shutdown_connections 2024-03-06T07:32:44.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.319+0000 7fc01edf7640 1 --2- 172.21.15.83:0/3592951651 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fbffc06b4d0 0x7fbffc06d990 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.319+0000 7fc01edf7640 1 --2- 172.21.15.83:0/3592951651 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc0180cf9a0 0x7fc018158510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.319+0000 7fc01edf7640 1 --2- 172.21.15.83:0/3592951651 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0180b55e0 0x7fc018157fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.319+0000 7fc01edf7640 1 --2- 172.21.15.83:0/3592951651 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc018066d90 0x7fc0180ae490 unknown :-1 s=CLOSED pgs=2023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.319+0000 7fc01edf7640 1 -- 172.21.15.83:0/3592951651 >> 172.21.15.83:0/3592951651 conn(0x7fc0180b1d70 msgr2=0x7fc0180998a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:44.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.319+0000 7fc01edf7640 1 -- 172.21.15.83:0/3592951651 shutdown_connections 2024-03-06T07:32:44.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.320+0000 7fc01edf7640 1 -- 172.21.15.83:0/3592951651 wait complete. 2024-03-06T07:32:44.332 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir' 2024-03-06T07:32:44.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.432+0000 7f84ace4e640 1 Processor -- start 2024-03-06T07:32:44.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.432+0000 7f84ace4e640 1 -- start start 2024-03-06T07:32:44.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.433+0000 7f84ace4e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f84a80cd7a0 0x7f84a80cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.433+0000 7f84ace4e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a80cc980 con 0x7f84a806da40 2024-03-06T07:32:44.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.433+0000 7f84ace4e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a80ccc60 con 0x7f84a81424f0 2024-03-06T07:32:44.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.433+0000 7f84ace4e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84a80cd220 con 0x7f84a80cd7a0 2024-03-06T07:32:44.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.433+0000 7f84a6d9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f84a80cd7a0 0x7f84a80cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.433+0000 7f849ffff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f84a806da40 0x7f84a8071b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57674/0 (socket says 172.21.15.83:57674) 2024-03-06T07:32:44.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.433+0000 7f849ffff640 1 -- 172.21.15.83:0/3170298816 learned_addr learned my addr 172.21.15.83:0/3170298816 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:32:44.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.433+0000 7f84a6d9d640 1 -- 172.21.15.83:0/3170298816 >> v1:172.21.15.183:6789/0 conn(0x7f84a81424f0 legacy=0x7f84a806d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:44.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.433+0000 7f84a6d9d640 1 -- 172.21.15.83:0/3170298816 >> v1:172.21.15.83:6789/0 conn(0x7f84a806da40 legacy=0x7f84a8071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.433+0000 7f84a6d9d640 1 -- 172.21.15.83:0/3170298816 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a80ccf40 con 0x7f84a80cd7a0 2024-03-06T07:32:44.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.433+0000 7f84a5d9b640 1 -- 172.21.15.83:0/3170298816 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 888481699 0 0) 0x7f84a80cc980 con 0x7f84a806da40 2024-03-06T07:32:44.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.434+0000 7f84a6d9d640 1 --2- 172.21.15.83:0/3170298816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84a80cd7a0 0x7f84a80cc2f0 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7f84a0009d90 tx=0x7f84a002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=107a1edf41199da9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:44.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.434+0000 7f84a5d9b640 1 -- 172.21.15.83:0/3170298816 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a0039070 con 0x7f84a80cd7a0 2024-03-06T07:32:44.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.434+0000 7f84a5d9b640 1 -- 172.21.15.83:0/3170298816 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84a002cc60 con 0x7f84a80cd7a0 2024-03-06T07:32:44.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.434+0000 7f84a5d9b640 1 -- 172.21.15.83:0/3170298816 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a0034660 con 0x7f84a80cd7a0 2024-03-06T07:32:44.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.434+0000 7f84ace4e640 1 -- 172.21.15.83:0/3170298816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84a80cd7a0 msgr2=0x7f84a80cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.434+0000 7f84ace4e640 1 --2- 172.21.15.83:0/3170298816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84a80cd7a0 0x7f84a80cc2f0 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7f84a0009d90 tx=0x7f84a002beb0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.435+0000 7f84ace4e640 1 -- 172.21.15.83:0/3170298816 shutdown_connections 2024-03-06T07:32:44.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.435+0000 7f84ace4e640 1 --2- 172.21.15.83:0/3170298816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84a80cd7a0 0x7f84a80cc2f0 unknown :-1 s=CLOSED pgs=2025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.435+0000 7f84ace4e640 1 -- 172.21.15.83:0/3170298816 >> 172.21.15.83:0/3170298816 conn(0x7f84a80cf8a0 msgr2=0x7f84a80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:44.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.435+0000 7f84ace4e640 1 -- 172.21.15.83:0/3170298816 shutdown_connections 2024-03-06T07:32:44.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.435+0000 7f84ace4e640 1 -- 172.21.15.83:0/3170298816 wait complete. 2024-03-06T07:32:44.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.436+0000 7f84ace4e640 1 Processor -- start 2024-03-06T07:32:44.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.436+0000 7f84ace4e640 1 -- start start 2024-03-06T07:32:44.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.437+0000 7f84ace4e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f84a806da40 0x7f84a80a9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.437+0000 7f84ace4e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84a80cd7a0 0x7f84a8140510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.437+0000 7f84ace4e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84a81424f0 0x7f84a813f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.437+0000 7f84ace4e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a815d070 con 0x7f84a80cd7a0 2024-03-06T07:32:44.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.437+0000 7f84ace4e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a815cef0 con 0x7f84a81424f0 2024-03-06T07:32:44.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.437+0000 7f84ace4e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84a815d1f0 con 0x7f84a806da40 2024-03-06T07:32:44.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.437+0000 7f849ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84a80cd7a0 0x7f84a8140510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.437+0000 7f849ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84a80cd7a0 0x7f84a8140510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50782/0 (socket says 172.21.15.83:50782) 2024-03-06T07:32:44.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.437+0000 7f849ffff640 1 -- 172.21.15.83:0/1733022996 learned_addr learned my addr 172.21.15.83:0/1733022996 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:44.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.437+0000 7f849ffff640 1 -- 172.21.15.83:0/1733022996 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f84a806da40 msgr2=0x7f84a80a9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.437+0000 7f849ffff640 1 --2- 172.21.15.83:0/1733022996 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f84a806da40 0x7f84a80a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.437+0000 7f849ffff640 1 -- 172.21.15.83:0/1733022996 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84a81424f0 msgr2=0x7f84a813f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.437+0000 7f849ffff640 1 --2- 172.21.15.83:0/1733022996 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84a81424f0 0x7f84a813f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.437+0000 7f849ffff640 1 -- 172.21.15.83:0/1733022996 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a80a9b10 con 0x7f84a80cd7a0 2024-03-06T07:32:44.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.437+0000 7f849ffff640 1 --2- 172.21.15.83:0/1733022996 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84a80cd7a0 0x7f84a8140510 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f849000c4d0 tx=0x7f849000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:44.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.437+0000 7f849effd640 1 -- 172.21.15.83:0/1733022996 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8490018070 con 0x7f84a80cd7a0 2024-03-06T07:32:44.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.437+0000 7f849effd640 1 -- 172.21.15.83:0/1733022996 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8490007500 con 0x7f84a80cd7a0 2024-03-06T07:32:44.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.437+0000 7f84ace4e640 1 -- 172.21.15.83:0/1733022996 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84a80a9df0 con 0x7f84a80cd7a0 2024-03-06T07:32:44.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.437+0000 7f849effd640 1 -- 172.21.15.83:0/1733022996 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8490004a20 con 0x7f84a80cd7a0 2024-03-06T07:32:44.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.437+0000 7f84ace4e640 1 -- 172.21.15.83:0/1733022996 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84a80aa3b0 con 0x7f84a80cd7a0 2024-03-06T07:32:44.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.438+0000 7f84ace4e640 1 -- 172.21.15.83:0/1733022996 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84a804f330 con 0x7f84a80cd7a0 2024-03-06T07:32:44.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.439+0000 7f849effd640 1 -- 172.21.15.83:0/1733022996 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f849001b070 con 0x7f84a80cd7a0 2024-03-06T07:32:44.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.439+0000 7f849effd640 1 --2- 172.21.15.83:0/1733022996 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f848806b4d0 0x7f848806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.439+0000 7f84a659c640 1 --2- 172.21.15.83:0/1733022996 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f848806b4d0 0x7f848806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.439+0000 7f849effd640 1 -- 172.21.15.83:0/1733022996 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f8490089230 con 0x7f84a80cd7a0 2024-03-06T07:32:44.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.440+0000 7f84a659c640 1 --2- 172.21.15.83:0/1733022996 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f848806b4d0 0x7f848806d990 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f8498006fd0 tx=0x7f8498006bc0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:44.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.444+0000 7f849effd640 1 -- 172.21.15.83:0/1733022996 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f84900583c0 con 0x7f84a80cd7a0 2024-03-06T07:32:44.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:44.679+0000 7f84ace4e640 1 -- 172.21.15.83:0/1733022996 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir"]} v 0) v1 -- 0x7f84a815bd70 con 0x7f84a80cd7a0 2024-03-06T07:32:45.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:45.002+0000 7f849effd640 1 -- 172.21.15.83:0/1733022996 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir"]}]=0 v563) v1 ==== 140+0+0 (secure 0 0 0) 0x7f849005e0c0 con 0x7f84a80cd7a0 2024-03-06T07:32:45.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:45.004+0000 7f84ace4e640 1 -- 172.21.15.83:0/1733022996 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f848806b4d0 msgr2=0x7f848806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:45.004+0000 7f84ace4e640 1 --2- 172.21.15.83:0/1733022996 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f848806b4d0 0x7f848806d990 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f8498006fd0 tx=0x7f8498006bc0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:45.004+0000 7f84ace4e640 1 -- 172.21.15.83:0/1733022996 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84a80cd7a0 msgr2=0x7f84a8140510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:45.004+0000 7f84ace4e640 1 --2- 172.21.15.83:0/1733022996 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84a80cd7a0 0x7f84a8140510 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f849000c4d0 tx=0x7f849000c990 comp rx=0 tx=0).stop 2024-03-06T07:32:45.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:45.004+0000 7f84ace4e640 1 -- 172.21.15.83:0/1733022996 shutdown_connections 2024-03-06T07:32:45.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:45.004+0000 7f84ace4e640 1 --2- 172.21.15.83:0/1733022996 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f848806b4d0 0x7f848806d990 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:45.004+0000 7f84ace4e640 1 --2- 172.21.15.83:0/1733022996 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84a81424f0 0x7f84a813f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:45.004+0000 7f84ace4e640 1 --2- 172.21.15.83:0/1733022996 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84a80cd7a0 0x7f84a8140510 unknown :-1 s=CLOSED pgs=2026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:45.004+0000 7f84ace4e640 1 --2- 172.21.15.83:0/1733022996 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f84a806da40 0x7f84a80a9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:45.004+0000 7f84ace4e640 1 -- 172.21.15.83:0/1733022996 >> 172.21.15.83:0/1733022996 conn(0x7f84a80cf8a0 msgr2=0x7f84a815d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:45.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:45.004+0000 7f84ace4e640 1 -- 172.21.15.83:0/1733022996 shutdown_connections 2024-03-06T07:32:45.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:45.004+0000 7f84ace4e640 1 -- 172.21.15.83:0/1733022996 wait complete. 2024-03-06T07:32:45.018 INFO:tasks.cephfs_test_runner:Reproduce https://tracker.ceph.com/issues/53597 ... ERROR 2024-03-06T07:32:45.019 INFO:tasks.cephfs_test_runner: 2024-03-06T07:32:45.019 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-06T07:32:45.019 INFO:tasks.cephfs_test_runner:ERROR: test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:32:45.019 INFO:tasks.cephfs_test_runner:Reproduce https://tracker.ceph.com/issues/53597 2024-03-06T07:32:45.019 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-06T07:32:45.019 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-06T07:32:45.019 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs/test_strays.py", line 734, in test_migrate_unlinked_dir 2024-03-06T07:32:45.019 INFO:tasks.cephfs_test_runner: self.fs.wait_for_daemons(timeout=120) 2024-03-06T07:32:45.019 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs/filesystem.py", line 1240, in wait_for_daemons 2024-03-06T07:32:45.019 INFO:tasks.cephfs_test_runner: raise RuntimeError("Timed out waiting for MDS daemons to become healthy") 2024-03-06T07:32:45.019 INFO:tasks.cephfs_test_runner:RuntimeError: Timed out waiting for MDS daemons to become healthy 2024-03-06T07:32:45.019 INFO:tasks.cephfs_test_runner: 2024-03-06T07:32:45.019 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-06T07:32:45.020 INFO:tasks.cephfs_test_runner:Ran 8 tests in 615.403s 2024-03-06T07:32:45.020 INFO:tasks.cephfs_test_runner: 2024-03-06T07:32:45.020 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2024-03-06T07:32:45.020 INFO:tasks.cephfs_test_runner: 2024-03-06T07:32:45.020 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-06T07:32:45.020 INFO:tasks.cephfs_test_runner:ERROR: test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:32:45.020 INFO:tasks.cephfs_test_runner:Reproduce https://tracker.ceph.com/issues/53597 2024-03-06T07:32:45.020 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-06T07:32:45.020 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-06T07:32:45.020 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs/test_strays.py", line 734, in test_migrate_unlinked_dir 2024-03-06T07:32:45.020 INFO:tasks.cephfs_test_runner: self.fs.wait_for_daemons(timeout=120) 2024-03-06T07:32:45.020 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs/filesystem.py", line 1240, in wait_for_daemons 2024-03-06T07:32:45.020 INFO:tasks.cephfs_test_runner: raise RuntimeError("Timed out waiting for MDS daemons to become healthy") 2024-03-06T07:32:45.020 INFO:tasks.cephfs_test_runner:RuntimeError: Timed out waiting for MDS daemons to become healthy 2024-03-06T07:32:45.021 INFO:tasks.cephfs_test_runner: 2024-03-06T07:32:45.021 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:32:45.222 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=3c11dada7ce94582beb21ab348035e72 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:32:45.224 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-03-06T07:32:45.232 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-03-06T07:32:45.247 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-03-06T07:32:45.247 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:32:45.247 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:45.267 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1257 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:32:45.267 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-06T07:32:45.267 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:45.323 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1257 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:32:45.323 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:32:45.324 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:45.342 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=287 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:32:45.342 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-06T07:32:45.342 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:45.399 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=287 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:32:45.399 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:45.399 DEBUG:teuthology.orchestra.run.smithi183:> ip netns list 2024-03-06T07:32:45.456 INFO:teuthology.orchestra.run.smithi183.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-03-06T07:32:45.457 INFO:teuthology.orchestra.run.smithi183.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-06T07:32:45.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:45.457 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-06T07:32:45.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:45.527 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:32:45.596 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:45.597 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip link delete ceph-brx 2024-03-06T07:32:45.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:45.701 DEBUG:teuthology.orchestra.run.smithi083:> ip netns list 2024-03-06T07:32:45.720 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:32:45.720 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:32:45.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:45.721 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:32:45.793 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:45.793 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:32:45.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:45.972 DEBUG:teuthology.orchestra.run.smithi083:> sudo ip link delete ceph-brx 2024-03-06T07:32:46.028 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-06T07:32:46.100 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-06T07:32:46.100 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:32:46.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.203+0000 7f53438c9640 1 Processor -- start 2024-03-06T07:32:46.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.203+0000 7f53438c9640 1 -- start start 2024-03-06T07:32:46.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.204+0000 7f53438c9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f533c1515e0 0x7f533c0cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.204+0000 7f53438c9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f533c15b430 0x7f533c071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.204+0000 7f53438c9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f533c15b8b0 con 0x7f533c0cd7a0 2024-03-06T07:32:46.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.204+0000 7f53438c9640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f533c070a70 con 0x7f533c15b430 2024-03-06T07:32:46.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.204+0000 7f53438c9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f533c070d50 con 0x7f533c1515e0 2024-03-06T07:32:46.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.204+0000 7f5340e64640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f533c15b430 0x7f533c071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.204+0000 7f5340e64640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f533c15b430 0x7f533c071830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50792/0 (socket says 172.21.15.83:50792) 2024-03-06T07:32:46.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.204+0000 7f5340e64640 1 -- 172.21.15.83:0/1548904589 learned_addr learned my addr 172.21.15.83:0/1548904589 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:46.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.204+0000 7f5340e64640 1 -- 172.21.15.83:0/1548904589 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f533c1515e0 msgr2=0x7f533c0cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:46.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.204+0000 7f5340e64640 1 --2- 172.21.15.83:0/1548904589 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f533c1515e0 0x7f533c0cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.204+0000 7f5340e64640 1 -- 172.21.15.83:0/1548904589 >> v1:172.21.15.183:6789/0 conn(0x7f533c0cd7a0 legacy=0x7f533c0cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:46.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.204+0000 7f5340e64640 1 -- 172.21.15.83:0/1548904589 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f533c071030 con 0x7f533c15b430 2024-03-06T07:32:46.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.205+0000 7f5340e64640 1 --2- 172.21.15.83:0/1548904589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f533c15b430 0x7f533c071830 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f532c009870 tx=0x7f532c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe26156af3fbc8c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:46.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.205+0000 7f5333fff640 1 -- 172.21.15.83:0/1548904589 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f532c035020 con 0x7f533c15b430 2024-03-06T07:32:46.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.205+0000 7f5333fff640 1 -- 172.21.15.83:0/1548904589 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f532c031640 con 0x7f533c15b430 2024-03-06T07:32:46.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.205+0000 7f5333fff640 1 -- 172.21.15.83:0/1548904589 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f532c02d040 con 0x7f533c15b430 2024-03-06T07:32:46.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.206+0000 7f53438c9640 1 -- 172.21.15.83:0/1548904589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f533c15b430 msgr2=0x7f533c071830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.206+0000 7f53438c9640 1 --2- 172.21.15.83:0/1548904589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f533c15b430 0x7f533c071830 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f532c009870 tx=0x7f532c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.206+0000 7f53438c9640 1 -- 172.21.15.83:0/1548904589 shutdown_connections 2024-03-06T07:32:46.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.206+0000 7f53438c9640 1 --2- 172.21.15.83:0/1548904589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f533c15b430 0x7f533c071830 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.206+0000 7f53438c9640 1 --2- 172.21.15.83:0/1548904589 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f533c1515e0 0x7f533c0cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.206+0000 7f53438c9640 1 -- 172.21.15.83:0/1548904589 >> 172.21.15.83:0/1548904589 conn(0x7f533c0cf8a0 msgr2=0x7f533c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:46.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.206+0000 7f53438c9640 1 -- 172.21.15.83:0/1548904589 shutdown_connections 2024-03-06T07:32:46.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.206+0000 7f53438c9640 1 -- 172.21.15.83:0/1548904589 wait complete. 2024-03-06T07:32:46.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.207+0000 7f53438c9640 1 Processor -- start 2024-03-06T07:32:46.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.207+0000 7f53438c9640 1 -- start start 2024-03-06T07:32:46.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.208+0000 7f53438c9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f533c0cd7a0 0x7f533c13fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.208+0000 7f53438c9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f533c1515e0 0x7f533c149000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.208+0000 7f53438c9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f533c15b430 0x7f533c14fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.208+0000 7f53438c9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f533c071ef0 con 0x7f533c1515e0 2024-03-06T07:32:46.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.208+0000 7f53438c9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f533c071d70 con 0x7f533c15b430 2024-03-06T07:32:46.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.208+0000 7f53438c9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f533c0cc3a0 con 0x7f533c0cd7a0 2024-03-06T07:32:46.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.208+0000 7f5340e64640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f533c1515e0 0x7f533c149000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.208+0000 7f5340e64640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f533c1515e0 0x7f533c149000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50808/0 (socket says 172.21.15.83:50808) 2024-03-06T07:32:46.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.208+0000 7f5340e64640 1 -- 172.21.15.83:0/3097178253 learned_addr learned my addr 172.21.15.83:0/3097178253 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:46.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.208+0000 7f5340e64640 1 -- 172.21.15.83:0/3097178253 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f533c0cd7a0 msgr2=0x7f533c13fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.208+0000 7f5340e64640 1 --2- 172.21.15.83:0/3097178253 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f533c0cd7a0 0x7f533c13fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.208+0000 7f5340e64640 1 -- 172.21.15.83:0/3097178253 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f533c15b430 msgr2=0x7f533c14fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.208+0000 7f5340e64640 1 --2- 172.21.15.83:0/3097178253 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f533c15b430 0x7f533c14fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.208+0000 7f5340e64640 1 -- 172.21.15.83:0/3097178253 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f533c1486b0 con 0x7f533c1515e0 2024-03-06T07:32:46.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.208+0000 7f5340e64640 1 --2- 172.21.15.83:0/3097178253 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f533c1515e0 0x7f533c149000 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f532c02b930 tx=0x7f532c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:46.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.208+0000 7f5331ffb640 1 -- 172.21.15.83:0/3097178253 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f532c03ac60 con 0x7f533c1515e0 2024-03-06T07:32:46.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.208+0000 7f5331ffb640 1 -- 172.21.15.83:0/3097178253 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f532c03ae20 con 0x7f533c1515e0 2024-03-06T07:32:46.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.208+0000 7f53438c9640 1 -- 172.21.15.83:0/3097178253 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f533c149930 con 0x7f533c1515e0 2024-03-06T07:32:46.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.208+0000 7f5331ffb640 1 -- 172.21.15.83:0/3097178253 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f532c044630 con 0x7f533c1515e0 2024-03-06T07:32:46.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.209+0000 7f53438c9640 1 -- 172.21.15.83:0/3097178253 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f533c144840 con 0x7f533c1515e0 2024-03-06T07:32:46.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.210+0000 7f53438c9640 1 -- 172.21.15.83:0/3097178253 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f533c068840 con 0x7f533c1515e0 2024-03-06T07:32:46.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.210+0000 7f5331ffb640 1 -- 172.21.15.83:0/3097178253 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f532c04b060 con 0x7f533c1515e0 2024-03-06T07:32:46.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.210+0000 7f5331ffb640 1 --2- 172.21.15.83:0/3097178253 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f532006b4d0 0x7f532006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.211+0000 7f5331ffb640 1 -- 172.21.15.83:0/3097178253 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f532c0abce0 con 0x7f533c1515e0 2024-03-06T07:32:46.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.211+0000 7f5341665640 1 --2- 172.21.15.83:0/3097178253 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f532006b4d0 0x7f532006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.211+0000 7f5341665640 1 --2- 172.21.15.83:0/3097178253 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f532006b4d0 0x7f532006d990 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f533c1f7950 tx=0x7f5338009250 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:46.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.215+0000 7f5331ffb640 1 -- 172.21.15.83:0/3097178253 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f532c03f070 con 0x7f533c1515e0 2024-03-06T07:32:46.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.383+0000 7f53438c9640 1 -- 172.21.15.83:0/3097178253 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f533c068ad0 con 0x7f532006b4d0 2024-03-06T07:32:46.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.384+0000 7f5331ffb640 1 -- 172.21.15.83:0/3097178253 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13511 (secure 0 0 0) 0x7f533c068ad0 con 0x7f532006b4d0 2024-03-06T07:32:46.386 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:32:46.387 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-06T07:32:46.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.386+0000 7f53438c9640 1 -- 172.21.15.83:0/3097178253 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f532006b4d0 msgr2=0x7f532006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.386+0000 7f53438c9640 1 --2- 172.21.15.83:0/3097178253 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f532006b4d0 0x7f532006d990 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f533c1f7950 tx=0x7f5338009250 comp rx=0 tx=0).stop 2024-03-06T07:32:46.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.386+0000 7f53438c9640 1 -- 172.21.15.83:0/3097178253 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f533c1515e0 msgr2=0x7f533c149000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.386+0000 7f53438c9640 1 --2- 172.21.15.83:0/3097178253 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f533c1515e0 0x7f533c149000 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f532c02b930 tx=0x7f532c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.386+0000 7f53438c9640 1 -- 172.21.15.83:0/3097178253 shutdown_connections 2024-03-06T07:32:46.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.386+0000 7f53438c9640 1 --2- 172.21.15.83:0/3097178253 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f532006b4d0 0x7f532006d990 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.386+0000 7f53438c9640 1 --2- 172.21.15.83:0/3097178253 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f533c15b430 0x7f533c14fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.386+0000 7f53438c9640 1 --2- 172.21.15.83:0/3097178253 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f533c1515e0 0x7f533c149000 unknown :-1 s=CLOSED pgs=2028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.386+0000 7f53438c9640 1 --2- 172.21.15.83:0/3097178253 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f533c0cd7a0 0x7f533c13fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.386+0000 7f53438c9640 1 -- 172.21.15.83:0/3097178253 >> 172.21.15.83:0/3097178253 conn(0x7f533c0cf8a0 msgr2=0x7f533c14e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:46.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.386+0000 7f53438c9640 1 -- 172.21.15.83:0/3097178253 shutdown_connections 2024-03-06T07:32:46.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.386+0000 7f53438c9640 1 -- 172.21.15.83:0/3097178253 wait complete. 2024-03-06T07:32:46.399 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":458,"stamp":"2024-03-06T07:32:44.436135+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":330160,"kb_used_data":14192,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754644560,"statfs":{"total":773094113280,"available":772756029440,"internally_reserved":0,"allocated":14532608,"data_stored":7009368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12757,"internal_metadata":323472939},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-33651,"num_objects":-44,"num_object_clones":0,"num_object_copies":-88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-4,"num_read_kb":0,"num_write":-143,"num_write_kb":-80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001147"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":374,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:29:50.389249+0000","last_change":"2024-03-06T07:21:47.553693+0000","last_active":"2024-03-06T07:29:50.389249+0000","last_peered":"2024-03-06T07:29:50.389249+0000","last_clean":"2024-03-06T07:29:50.389249+0000","last_became_active":"2024-03-06T07:21:47.552664+0000","last_became_peered":"2024-03-06T07:21:47.552664+0000","last_unstale":"2024-03-06T07:29:50.389249+0000","last_undegraded":"2024-03-06T07:29:50.389249+0000","last_fullsized":"2024-03-06T07:29:50.389249+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_clean_scrub_stamp":"2024-03-06T07:21:46.362028+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:02.916426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574984,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41724,"kb_used_data":2228,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330116,"statfs":{"total":96636764160,"available":96594038784,"internally_reserved":0,"allocated":2281472,"data_stored":1329833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":40434121},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574982,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41136,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330704,"statfs":{"total":96636764160,"available":96594640896,"internally_reserved":0,"allocated":1679360,"data_stored":738849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574983,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41124,"kb_used_data":1628,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330716,"statfs":{"total":96636764160,"available":96594653184,"internally_reserved":0,"allocated":1667072,"data_stored":720978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574984,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41108,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330732,"statfs":{"total":96636764160,"available":96594669568,"internally_reserved":0,"allocated":1650688,"data_stored":720174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574983,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41140,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330700,"statfs":{"total":96636764160,"available":96594636800,"internally_reserved":0,"allocated":1683456,"data_stored":734531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574983,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41116,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330724,"statfs":{"total":96636764160,"available":96594661376,"internally_reserved":0,"allocated":1658880,"data_stored":720810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574984,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41128,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330712,"statfs":{"total":96636764160,"available":96594649088,"internally_reserved":0,"allocated":1671168,"data_stored":733721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574983,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41684,"kb_used_data":2188,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330156,"statfs":{"total":96636764160,"available":96594079744,"internally_reserved":0,"allocated":2240512,"data_stored":1310472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:32:46.400 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:32:46.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.499+0000 7fd419af4640 1 Processor -- start 2024-03-06T07:32:46.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.499+0000 7fd419af4640 1 -- start start 2024-03-06T07:32:46.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.500+0000 7fd419af4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd4140b3060 0x7fd41415e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.500+0000 7fd419af4640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fd41415bf30 0x7fd41415ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.500+0000 7fd419af4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd41415f2d0 con 0x7fd41413dc10 2024-03-06T07:32:46.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.500+0000 7fd419af4640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4140cf3f0 con 0x7fd4140b3060 2024-03-06T07:32:46.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.500+0000 7fd419af4640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd41415b210 con 0x7fd41415bf30 2024-03-06T07:32:46.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.500+0000 7fd4137fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fd41413dc10 0x7fd4140b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57688/0 (socket says 172.21.15.83:57688) 2024-03-06T07:32:46.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.500+0000 7fd4137fe640 1 -- 172.21.15.83:0/299354181 learned_addr learned my addr 172.21.15.83:0/299354181 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:32:46.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.500+0000 7fd412ffd640 1 --2- 172.21.15.83:0/299354181 >> v2:172.21.15.83:3300/0 conn(0x7fd4140b3060 0x7fd41415e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.500+0000 7fd413fff640 1 --2- 172.21.15.83:0/299354181 >> v2:172.21.15.183:3300/0 conn(0x7fd41415bf30 0x7fd41415ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.500+0000 7fd4127fc640 1 -- 172.21.15.83:0/299354181 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 697681634 0 0) 0x7fd41415f2d0 con 0x7fd41413dc10 2024-03-06T07:32:46.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.500+0000 7fd4127fc640 1 -- 172.21.15.83:0/299354181 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3fc003520 con 0x7fd41413dc10 2024-03-06T07:32:46.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.500+0000 7fd412ffd640 1 -- 172.21.15.83:0/299354181 >> v2:172.21.15.183:3300/0 conn(0x7fd41415bf30 msgr2=0x7fd41415ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:46.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.500+0000 7fd412ffd640 1 --2- 172.21.15.83:0/299354181 >> v2:172.21.15.183:3300/0 conn(0x7fd41415bf30 0x7fd41415ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.500+0000 7fd412ffd640 1 -- 172.21.15.83:0/299354181 >> v1:172.21.15.83:6789/0 conn(0x7fd41413dc10 legacy=0x7fd4140b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.500+0000 7fd412ffd640 1 -- 172.21.15.83:0/299354181 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd41415f580 con 0x7fd4140b3060 2024-03-06T07:32:46.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.501+0000 7fd412ffd640 1 --2- 172.21.15.83:0/299354181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd4140b3060 0x7fd41415e850 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7fd4140b4e00 tx=0x7fd40802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=896e408224e38417 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:46.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.501+0000 7fd4127fc640 1 -- 172.21.15.83:0/299354181 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4080315a0 con 0x7fd4140b3060 2024-03-06T07:32:46.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.501+0000 7fd4127fc640 1 -- 172.21.15.83:0/299354181 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd408037070 con 0x7fd4140b3060 2024-03-06T07:32:46.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.501+0000 7fd4127fc640 1 -- 172.21.15.83:0/299354181 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd408031960 con 0x7fd4140b3060 2024-03-06T07:32:46.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.502+0000 7fd419af4640 1 -- 172.21.15.83:0/299354181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd4140b3060 msgr2=0x7fd41415e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.502+0000 7fd419af4640 1 --2- 172.21.15.83:0/299354181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd4140b3060 0x7fd41415e850 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7fd4140b4e00 tx=0x7fd40802b840 comp rx=0 tx=0).stop 2024-03-06T07:32:46.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.502+0000 7fd419af4640 1 -- 172.21.15.83:0/299354181 shutdown_connections 2024-03-06T07:32:46.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.502+0000 7fd419af4640 1 --2- 172.21.15.83:0/299354181 >> v2:172.21.15.183:3300/0 conn(0x7fd41415bf30 0x7fd41415ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.502+0000 7fd419af4640 1 --2- 172.21.15.83:0/299354181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd4140b3060 0x7fd41415e850 unknown :-1 s=CLOSED pgs=2030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.502+0000 7fd419af4640 1 -- 172.21.15.83:0/299354181 >> 172.21.15.83:0/299354181 conn(0x7fd4140cf8a0 msgr2=0x7fd4140cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:46.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.502+0000 7fd419af4640 1 -- 172.21.15.83:0/299354181 shutdown_connections 2024-03-06T07:32:46.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.502+0000 7fd419af4640 1 -- 172.21.15.83:0/299354181 wait complete. 2024-03-06T07:32:46.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.503+0000 7fd419af4640 1 Processor -- start 2024-03-06T07:32:46.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.503+0000 7fd419af4640 1 -- start start 2024-03-06T07:32:46.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.504+0000 7fd419af4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd4140b3060 0x7fd414143ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.504+0000 7fd419af4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd41413dc10 0x7fd41406c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.504+0000 7fd419af4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd41415bf30 0x7fd41406cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.504+0000 7fd419af4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd41413dff0 con 0x7fd4140b3060 2024-03-06T07:32:46.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.504+0000 7fd419af4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd41415b730 con 0x7fd41415bf30 2024-03-06T07:32:46.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.504+0000 7fd419af4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4140708d0 con 0x7fd41413dc10 2024-03-06T07:32:46.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.504+0000 7fd4137fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd4140b3060 0x7fd414143ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.504+0000 7fd4137fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd4140b3060 0x7fd414143ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:50836/0 (socket says 172.21.15.83:50836) 2024-03-06T07:32:46.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.504+0000 7fd4137fe640 1 -- 172.21.15.83:0/3879345003 learned_addr learned my addr 172.21.15.83:0/3879345003 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:46.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.504+0000 7fd4137fe640 1 -- 172.21.15.83:0/3879345003 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd41413dc10 msgr2=0x7fd41406c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.504+0000 7fd4137fe640 1 --2- 172.21.15.83:0/3879345003 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd41413dc10 0x7fd41406c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.504+0000 7fd4137fe640 1 -- 172.21.15.83:0/3879345003 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd41415bf30 msgr2=0x7fd41406cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.504+0000 7fd4137fe640 1 --2- 172.21.15.83:0/3879345003 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd41415bf30 0x7fd41406cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.504+0000 7fd4137fe640 1 -- 172.21.15.83:0/3879345003 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd41406dcb0 con 0x7fd4140b3060 2024-03-06T07:32:46.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.504+0000 7fd4137fe640 1 --2- 172.21.15.83:0/3879345003 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd4140b3060 0x7fd414143ed0 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7fd40400c460 tx=0x7fd40400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:46.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.505+0000 7fd3fbfff640 1 -- 172.21.15.83:0/3879345003 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd404017070 con 0x7fd4140b3060 2024-03-06T07:32:46.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.505+0000 7fd3fbfff640 1 -- 172.21.15.83:0/3879345003 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd404005ce0 con 0x7fd4140b3060 2024-03-06T07:32:46.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.505+0000 7fd419af4640 1 -- 172.21.15.83:0/3879345003 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd41412dff0 con 0x7fd4140b3060 2024-03-06T07:32:46.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.505+0000 7fd3fbfff640 1 -- 172.21.15.83:0/3879345003 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4040072f0 con 0x7fd4140b3060 2024-03-06T07:32:46.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.505+0000 7fd419af4640 1 -- 172.21.15.83:0/3879345003 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd41412e5b0 con 0x7fd4140b3060 2024-03-06T07:32:46.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.506+0000 7fd419af4640 1 -- 172.21.15.83:0/3879345003 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd41404f330 con 0x7fd4140b3060 2024-03-06T07:32:46.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.506+0000 7fd3fbfff640 1 -- 172.21.15.83:0/3879345003 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd40401a070 con 0x7fd4140b3060 2024-03-06T07:32:46.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.507+0000 7fd3fbfff640 1 --2- 172.21.15.83:0/3879345003 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd40006b4d0 0x7fd40006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.507+0000 7fd3fbfff640 1 -- 172.21.15.83:0/3879345003 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fd4040892c0 con 0x7fd4140b3060 2024-03-06T07:32:46.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.507+0000 7fd412ffd640 1 --2- 172.21.15.83:0/3879345003 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd40006b4d0 0x7fd40006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.507+0000 7fd412ffd640 1 --2- 172.21.15.83:0/3879345003 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd40006b4d0 0x7fd40006d990 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fd408036000 tx=0x7fd408030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:46.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.511+0000 7fd3fbfff640 1 -- 172.21.15.83:0/3879345003 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd404058530 con 0x7fd4140b3060 2024-03-06T07:32:46.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.679+0000 7fd419af4640 1 -- 172.21.15.83:0/3879345003 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd41406eb00 con 0x7fd40006b4d0 2024-03-06T07:32:46.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.679+0000 7fd3fbfff640 1 -- 172.21.15.83:0/3879345003 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13508 (secure 0 0 0) 0x7fd41406eb00 con 0x7fd40006b4d0 2024-03-06T07:32:46.682 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:32:46.682 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-06T07:32:46.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.681+0000 7fd419af4640 1 -- 172.21.15.83:0/3879345003 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd40006b4d0 msgr2=0x7fd40006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.681+0000 7fd419af4640 1 --2- 172.21.15.83:0/3879345003 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd40006b4d0 0x7fd40006d990 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fd408036000 tx=0x7fd408030000 comp rx=0 tx=0).stop 2024-03-06T07:32:46.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.681+0000 7fd419af4640 1 -- 172.21.15.83:0/3879345003 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd4140b3060 msgr2=0x7fd414143ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.681+0000 7fd419af4640 1 --2- 172.21.15.83:0/3879345003 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd4140b3060 0x7fd414143ed0 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7fd40400c460 tx=0x7fd40400c920 comp rx=0 tx=0).stop 2024-03-06T07:32:46.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.681+0000 7fd419af4640 1 -- 172.21.15.83:0/3879345003 shutdown_connections 2024-03-06T07:32:46.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.681+0000 7fd419af4640 1 --2- 172.21.15.83:0/3879345003 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fd40006b4d0 0x7fd40006d990 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.681+0000 7fd419af4640 1 --2- 172.21.15.83:0/3879345003 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd41415bf30 0x7fd41406cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.681+0000 7fd419af4640 1 --2- 172.21.15.83:0/3879345003 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd41413dc10 0x7fd41406c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.681+0000 7fd419af4640 1 --2- 172.21.15.83:0/3879345003 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd4140b3060 0x7fd414143ed0 unknown :-1 s=CLOSED pgs=2031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.681+0000 7fd419af4640 1 -- 172.21.15.83:0/3879345003 >> 172.21.15.83:0/3879345003 conn(0x7fd4140cf8a0 msgr2=0x7fd414070cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:46.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.682+0000 7fd419af4640 1 -- 172.21.15.83:0/3879345003 shutdown_connections 2024-03-06T07:32:46.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.682+0000 7fd419af4640 1 -- 172.21.15.83:0/3879345003 wait complete. 2024-03-06T07:32:46.694 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":459,"stamp":"2024-03-06T07:32:46.436384+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":124,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":330392,"kb_used_data":14376,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754644328,"statfs":{"total":773094113280,"available":772755791872,"internally_reserved":0,"allocated":14721024,"data_stored":7086712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12757,"internal_metadata":323472939},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-33900,"num_objects":-44,"num_object_clones":0,"num_object_copies":-88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-4,"num_read_kb":0,"num_write":-148,"num_write_kb":-83,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001134"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":380,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:32:44.163214+0000","last_change":"2024-03-06T07:21:47.553693+0000","last_active":"2024-03-06T07:32:44.163214+0000","last_peered":"2024-03-06T07:32:44.163214+0000","last_clean":"2024-03-06T07:32:44.163214+0000","last_became_active":"2024-03-06T07:21:47.552664+0000","last_became_peered":"2024-03-06T07:21:47.552664+0000","last_unstale":"2024-03-06T07:32:44.163214+0000","last_undegraded":"2024-03-06T07:32:44.163214+0000","last_fullsized":"2024-03-06T07:32:44.163214+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_clean_scrub_stamp":"2024-03-06T07:21:46.362028+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:02.916426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574985,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41736,"kb_used_data":2232,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330104,"statfs":{"total":96636764160,"available":96594026496,"internally_reserved":0,"allocated":2285568,"data_stored":1328615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":40434121},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574983,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41152,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330688,"statfs":{"total":96636764160,"available":96594624512,"internally_reserved":0,"allocated":1691648,"data_stored":738247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574984,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41152,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330688,"statfs":{"total":96636764160,"available":96594624512,"internally_reserved":0,"allocated":1691648,"data_stored":738247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574985,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41156,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330684,"statfs":{"total":96636764160,"available":96594620416,"internally_reserved":0,"allocated":1691648,"data_stored":738247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574984,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41156,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330684,"statfs":{"total":96636764160,"available":96594620416,"internally_reserved":0,"allocated":1691648,"data_stored":738247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574984,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41156,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330684,"statfs":{"total":96636764160,"available":96594620416,"internally_reserved":0,"allocated":1691648,"data_stored":738247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574985,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41152,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330688,"statfs":{"total":96636764160,"available":96594624512,"internally_reserved":0,"allocated":1691648,"data_stored":738247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574984,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41732,"kb_used_data":2232,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330108,"statfs":{"total":96636764160,"available":96594030592,"internally_reserved":0,"allocated":2285568,"data_stored":1328615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:32:46.695 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-06T07:32:46.695 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:32:46.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.794+0000 7f267cb1e640 1 Processor -- start 2024-03-06T07:32:46.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.794+0000 7f267cb1e640 1 -- start start 2024-03-06T07:32:46.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.795+0000 7f267cb1e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f26780b3060 0x7f267815e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.795+0000 7f267cb1e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f267815f580 con 0x7f267815bf30 2024-03-06T07:32:46.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.795+0000 7f267cb1e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f267815b450 con 0x7f267813dc10 2024-03-06T07:32:46.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.795+0000 7f267cb1e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26780cf3f0 con 0x7f26780b3060 2024-03-06T07:32:46.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.795+0000 7f2676d9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f267815bf30 0x7f267815ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:53816/0 (socket says 172.21.15.83:53816) 2024-03-06T07:32:46.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.795+0000 7f2676d9d640 1 -- 172.21.15.83:0/4189480752 learned_addr learned my addr 172.21.15.83:0/4189480752 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:32:46.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.795+0000 7f2675d9b640 1 --2- 172.21.15.83:0/4189480752 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f26780b3060 0x7f267815e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.795+0000 7f267559a640 1 -- 172.21.15.83:0/4189480752 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1809052018 0 0) 0x7f267815f580 con 0x7f267815bf30 2024-03-06T07:32:46.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.795+0000 7f267559a640 1 -- 172.21.15.83:0/4189480752 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2664005730 con 0x7f267815bf30 2024-03-06T07:32:46.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.795+0000 7f267559a640 1 -- 172.21.15.83:0/4189480752 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2547771437 0 0) 0x7f2664005730 con 0x7f267815bf30 2024-03-06T07:32:46.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.795+0000 7f267559a640 1 -- 172.21.15.83:0/4189480752 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f26780b3060 msgr2=0x7f267815e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:46.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.795+0000 7f267559a640 1 --2- 172.21.15.83:0/4189480752 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f26780b3060 0x7f267815e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.795+0000 7f267559a640 1 -- 172.21.15.83:0/4189480752 >> v1:172.21.15.183:6789/0 conn(0x7f267813dc10 legacy=0x7f26780b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.795+0000 7f267559a640 1 -- 172.21.15.83:0/4189480752 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f267815b730 con 0x7f267815bf30 2024-03-06T07:32:46.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.795+0000 7f267559a640 1 -- 172.21.15.83:0/4189480752 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f26700019f0 con 0x7f267815bf30 2024-03-06T07:32:46.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.795+0000 7f267559a640 1 -- 172.21.15.83:0/4189480752 >> v1:172.21.15.83:6789/0 conn(0x7f267815bf30 legacy=0x7f267815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.795+0000 7f267559a640 1 --2- 172.21.15.83:0/4189480752 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2664005bf0 0x7f2664006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.795+0000 7f267559a640 1 --2- 172.21.15.83:0/4189480752 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f26780b3060 0x7f2664006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.795+0000 7f267559a640 1 --2- 172.21.15.83:0/4189480752 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2664006f50 0x7f26640073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.795+0000 7f267559a640 1 -- 172.21.15.83:0/4189480752 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26780cf3f0 con 0x7f2664006f50 2024-03-06T07:32:46.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.795+0000 7f267559a640 1 -- 172.21.15.83:0/4189480752 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f266400dde0 con 0x7f2664005bf0 2024-03-06T07:32:46.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.795+0000 7f267559a640 1 -- 172.21.15.83:0/4189480752 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f266400dfe0 con 0x7f26780b3060 2024-03-06T07:32:46.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.795+0000 7f2676d9d640 1 --2- 172.21.15.83:0/4189480752 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2664006f50 0x7f26640073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.795+0000 7f267659c640 1 --2- 172.21.15.83:0/4189480752 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2664005bf0 0x7f2664006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.795+0000 7f2675d9b640 1 --2- 172.21.15.83:0/4189480752 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f26780b3060 0x7f2664006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.796+0000 7f2676d9d640 1 -- 172.21.15.83:0/4189480752 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f26780b3060 msgr2=0x7f2664006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:46.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.796+0000 7f2676d9d640 1 --2- 172.21.15.83:0/4189480752 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f26780b3060 0x7f2664006a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.796+0000 7f2676d9d640 1 -- 172.21.15.83:0/4189480752 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2664005bf0 msgr2=0x7f2664006010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:46.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.796+0000 7f2676d9d640 1 --2- 172.21.15.83:0/4189480752 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2664005bf0 0x7f2664006010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.796+0000 7f2676d9d640 1 -- 172.21.15.83:0/4189480752 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2664005730 con 0x7f2664006f50 2024-03-06T07:32:46.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.796+0000 7f2676d9d640 1 --2- 172.21.15.83:0/4189480752 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2664006f50 0x7f26640073d0 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f267000a840 tx=0x7f2664007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84688c0ec5e63b0d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:46.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.796+0000 7f267559a640 1 -- 172.21.15.83:0/4189480752 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f267000a460 con 0x7f2664006f50 2024-03-06T07:32:46.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.796+0000 7f267559a640 1 -- 172.21.15.83:0/4189480752 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2670034650 con 0x7f2664006f50 2024-03-06T07:32:46.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.796+0000 7f267559a640 1 -- 172.21.15.83:0/4189480752 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2670034a40 con 0x7f2664006f50 2024-03-06T07:32:46.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.797+0000 7f267cb1e640 1 -- 172.21.15.83:0/4189480752 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2664006f50 msgr2=0x7f26640073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.797+0000 7f267cb1e640 1 --2- 172.21.15.83:0/4189480752 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2664006f50 0x7f26640073d0 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f267000a840 tx=0x7f2664007a40 comp rx=0 tx=0).stop 2024-03-06T07:32:46.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.797+0000 7f2676d9d640 1 -- 172.21.15.83:0/4189480752 reap_dead start 2024-03-06T07:32:46.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.797+0000 7f267cb1e640 1 -- 172.21.15.83:0/4189480752 shutdown_connections 2024-03-06T07:32:46.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.797+0000 7f267cb1e640 1 -- 172.21.15.83:0/4189480752 >> 172.21.15.83:0/4189480752 conn(0x7f26780cf8a0 msgr2=0x7f26780cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:46.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.797+0000 7f267cb1e640 1 -- 172.21.15.83:0/4189480752 shutdown_connections 2024-03-06T07:32:46.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.797+0000 7f267cb1e640 1 -- 172.21.15.83:0/4189480752 wait complete. 2024-03-06T07:32:46.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.798+0000 7f267cb1e640 1 Processor -- start 2024-03-06T07:32:46.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.798+0000 7f267cb1e640 1 -- start start 2024-03-06T07:32:46.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.799+0000 7f267cb1e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2664006f50 0x7f267813d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.799+0000 7f267cb1e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2678140cd0 0x7f267815d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.799+0000 7f267cb1e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2678141150 0x7f267815d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.799+0000 7f267cb1e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f267815e850 con 0x7f2678140cd0 2024-03-06T07:32:46.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.799+0000 7f267cb1e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f267815b210 con 0x7f2678141150 2024-03-06T07:32:46.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.799+0000 7f267cb1e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f267815e9d0 con 0x7f2664006f50 2024-03-06T07:32:46.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.799+0000 7f2675d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2678140cd0 0x7f267815d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.799+0000 7f2675d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2678140cd0 0x7f267815d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36258/0 (socket says 172.21.15.83:36258) 2024-03-06T07:32:46.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.799+0000 7f2675d9b640 1 -- 172.21.15.83:0/2958343914 learned_addr learned my addr 172.21.15.83:0/2958343914 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:46.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.799+0000 7f267659c640 1 --2- 172.21.15.83:0/2958343914 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2664006f50 0x7f267813d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.799+0000 7f2676d9d640 1 --2- 172.21.15.83:0/2958343914 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2678141150 0x7f267815d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.799+0000 7f2675d9b640 1 -- 172.21.15.83:0/2958343914 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2664006f50 msgr2=0x7f267813d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.799+0000 7f2675d9b640 1 --2- 172.21.15.83:0/2958343914 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2664006f50 0x7f267813d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.799+0000 7f2675d9b640 1 -- 172.21.15.83:0/2958343914 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2678141150 msgr2=0x7f267815d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.799+0000 7f2675d9b640 1 --2- 172.21.15.83:0/2958343914 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2678141150 0x7f267815d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.800+0000 7f2675d9b640 1 -- 172.21.15.83:0/2958343914 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26781efa00 con 0x7f2678140cd0 2024-03-06T07:32:46.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.800+0000 7f2675d9b640 1 --2- 172.21.15.83:0/2958343914 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2678140cd0 0x7f267815d260 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7f266000b820 tx=0x7f266000bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:46.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.800+0000 7f266effd640 1 -- 172.21.15.83:0/2958343914 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2660018070 con 0x7f2678140cd0 2024-03-06T07:32:46.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.800+0000 7f266effd640 1 -- 172.21.15.83:0/2958343914 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f266000be10 con 0x7f2678140cd0 2024-03-06T07:32:46.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.800+0000 7f267cb1e640 1 -- 172.21.15.83:0/2958343914 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26781438a0 con 0x7f2678140cd0 2024-03-06T07:32:46.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.800+0000 7f266effd640 1 -- 172.21.15.83:0/2958343914 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2660010c60 con 0x7f2678140cd0 2024-03-06T07:32:46.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.800+0000 7f267cb1e640 1 -- 172.21.15.83:0/2958343914 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2678143e60 con 0x7f2678140cd0 2024-03-06T07:32:46.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.801+0000 7f267cb1e640 1 -- 172.21.15.83:0/2958343914 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f267804f330 con 0x7f2678140cd0 2024-03-06T07:32:46.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.801+0000 7f266effd640 1 -- 172.21.15.83:0/2958343914 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f266001b070 con 0x7f2678140cd0 2024-03-06T07:32:46.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.802+0000 7f266effd640 1 --2- 172.21.15.83:0/2958343914 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f265806b4d0 0x7f265806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.802+0000 7f267659c640 1 --2- 172.21.15.83:0/2958343914 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f265806b4d0 0x7f265806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.802+0000 7f266effd640 1 -- 172.21.15.83:0/2958343914 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f266008a300 con 0x7f2678140cd0 2024-03-06T07:32:46.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.802+0000 7f267659c640 1 --2- 172.21.15.83:0/2958343914 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f265806b4d0 0x7f265806d990 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f26680035f0 tx=0x7f2668009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:46.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.806+0000 7f266effd640 1 -- 172.21.15.83:0/2958343914 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2660059490 con 0x7f2678140cd0 2024-03-06T07:32:46.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.976+0000 7f267cb1e640 1 -- 172.21.15.83:0/2958343914 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f26780726f0 con 0x7f265806b4d0 2024-03-06T07:32:46.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.977+0000 7f266effd640 1 -- 172.21.15.83:0/2958343914 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13508 (secure 0 0 0) 0x7f26780726f0 con 0x7f265806b4d0 2024-03-06T07:32:46.979 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:32:46.979 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-06T07:32:46.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.979+0000 7f267cb1e640 1 -- 172.21.15.83:0/2958343914 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f265806b4d0 msgr2=0x7f265806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.979+0000 7f267cb1e640 1 --2- 172.21.15.83:0/2958343914 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f265806b4d0 0x7f265806d990 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f26680035f0 tx=0x7f2668009040 comp rx=0 tx=0).stop 2024-03-06T07:32:46.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.979+0000 7f267cb1e640 1 -- 172.21.15.83:0/2958343914 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2678140cd0 msgr2=0x7f267815d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.979+0000 7f267cb1e640 1 --2- 172.21.15.83:0/2958343914 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2678140cd0 0x7f267815d260 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7f266000b820 tx=0x7f266000bce0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.979+0000 7f267cb1e640 1 -- 172.21.15.83:0/2958343914 shutdown_connections 2024-03-06T07:32:46.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.979+0000 7f267cb1e640 1 --2- 172.21.15.83:0/2958343914 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f265806b4d0 0x7f265806d990 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.979+0000 7f267cb1e640 1 --2- 172.21.15.83:0/2958343914 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2678141150 0x7f267815d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.979+0000 7f267cb1e640 1 --2- 172.21.15.83:0/2958343914 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2678140cd0 0x7f267815d260 unknown :-1 s=CLOSED pgs=2034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.979+0000 7f267cb1e640 1 --2- 172.21.15.83:0/2958343914 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2664006f50 0x7f267813d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.979+0000 7f267cb1e640 1 -- 172.21.15.83:0/2958343914 >> 172.21.15.83:0/2958343914 conn(0x7f26780cf8a0 msgr2=0x7f267815c400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:46.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.979+0000 7f267cb1e640 1 -- 172.21.15.83:0/2958343914 shutdown_connections 2024-03-06T07:32:46.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:46.979+0000 7f267cb1e640 1 -- 172.21.15.83:0/2958343914 wait complete. 2024-03-06T07:32:46.991 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":459,"stamp":"2024-03-06T07:32:46.436384+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":124,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":330392,"kb_used_data":14376,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754644328,"statfs":{"total":773094113280,"available":772755791872,"internally_reserved":0,"allocated":14721024,"data_stored":7086712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12757,"internal_metadata":323472939},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-33900,"num_objects":-44,"num_object_clones":0,"num_object_copies":-88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-4,"num_read_kb":0,"num_write":-148,"num_write_kb":-83,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001134"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":380,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:32:44.163214+0000","last_change":"2024-03-06T07:21:47.553693+0000","last_active":"2024-03-06T07:32:44.163214+0000","last_peered":"2024-03-06T07:32:44.163214+0000","last_clean":"2024-03-06T07:32:44.163214+0000","last_became_active":"2024-03-06T07:21:47.552664+0000","last_became_peered":"2024-03-06T07:21:47.552664+0000","last_unstale":"2024-03-06T07:32:44.163214+0000","last_undegraded":"2024-03-06T07:32:44.163214+0000","last_fullsized":"2024-03-06T07:32:44.163214+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_clean_scrub_stamp":"2024-03-06T07:21:46.362028+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:02.916426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574985,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41736,"kb_used_data":2232,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330104,"statfs":{"total":96636764160,"available":96594026496,"internally_reserved":0,"allocated":2285568,"data_stored":1328615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":40434121},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574983,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41152,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330688,"statfs":{"total":96636764160,"available":96594624512,"internally_reserved":0,"allocated":1691648,"data_stored":738247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574984,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41152,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330688,"statfs":{"total":96636764160,"available":96594624512,"internally_reserved":0,"allocated":1691648,"data_stored":738247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574985,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41156,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330684,"statfs":{"total":96636764160,"available":96594620416,"internally_reserved":0,"allocated":1691648,"data_stored":738247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574984,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41156,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330684,"statfs":{"total":96636764160,"available":96594620416,"internally_reserved":0,"allocated":1691648,"data_stored":738247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574984,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41156,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330684,"statfs":{"total":96636764160,"available":96594620416,"internally_reserved":0,"allocated":1691648,"data_stored":738247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574985,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41152,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330688,"statfs":{"total":96636764160,"available":96594624512,"internally_reserved":0,"allocated":1691648,"data_stored":738247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574984,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41732,"kb_used_data":2232,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330108,"statfs":{"total":96636764160,"available":96594030592,"internally_reserved":0,"allocated":2285568,"data_stored":1328615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:32:46.992 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:47.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.090+0000 7f342b44e640 1 Processor -- start 2024-03-06T07:32:47.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.091+0000 7f342b44e640 1 -- start start 2024-03-06T07:32:47.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.091+0000 7f342b44e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f34240b4170 0x7f34240b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.091+0000 7f342b44e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f34240d4760 0x7f34240d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.091+0000 7f342b44e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34240d25d0 con 0x7f34240b4dd0 2024-03-06T07:32:47.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.091+0000 7f342b44e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34240d4bf0 con 0x7f34240d4760 2024-03-06T07:32:47.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.091+0000 7f342b44e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34240ce9e0 con 0x7f34240b4170 2024-03-06T07:32:47.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.092+0000 7f34289e9640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f34240d4760 0x7f34240d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:47.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.092+0000 7f34291ea640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f34240b4170 0x7f34240b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:47.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.092+0000 7f34289e9640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f34240d4760 0x7f34240d0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:51790/0 (socket says 172.21.15.83:51790) 2024-03-06T07:32:47.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.092+0000 7f34289e9640 1 -- 172.21.15.83:0/485744847 learned_addr learned my addr 172.21.15.83:0/485744847 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:47.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.092+0000 7f34289e9640 1 -- 172.21.15.83:0/485744847 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f34240b4170 msgr2=0x7f34240b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:47.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.092+0000 7f34289e9640 1 --2- 172.21.15.83:0/485744847 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f34240b4170 0x7f34240b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.092+0000 7f34289e9640 1 -- 172.21.15.83:0/485744847 >> v1:172.21.15.183:6789/0 conn(0x7f34240b4dd0 legacy=0x7f34240cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.092+0000 7f34289e9640 1 -- 172.21.15.83:0/485744847 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34240d2870 con 0x7f34240d4760 2024-03-06T07:32:47.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.092+0000 7f34291ea640 1 --2- 172.21.15.83:0/485744847 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f34240b4170 0x7f34240b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:47.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.093+0000 7f34289e9640 1 --2- 172.21.15.83:0/485744847 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f34240d4760 0x7f34240d0370 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f3420009870 tx=0x7f342002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ba9ff6d9cda94d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:47.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.093+0000 7f3417fff640 1 -- 172.21.15.83:0/485744847 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3420034070 con 0x7f34240d4760 2024-03-06T07:32:47.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.093+0000 7f3417fff640 1 -- 172.21.15.83:0/485744847 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3420030840 con 0x7f34240d4760 2024-03-06T07:32:47.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.093+0000 7f3417fff640 1 -- 172.21.15.83:0/485744847 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3420030c10 con 0x7f34240d4760 2024-03-06T07:32:47.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.094+0000 7f342b44e640 1 -- 172.21.15.83:0/485744847 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f34240d4760 msgr2=0x7f34240d0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.094+0000 7f342b44e640 1 --2- 172.21.15.83:0/485744847 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f34240d4760 0x7f34240d0370 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f3420009870 tx=0x7f342002b980 comp rx=0 tx=0).stop 2024-03-06T07:32:47.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.094+0000 7f342b44e640 1 -- 172.21.15.83:0/485744847 shutdown_connections 2024-03-06T07:32:47.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.094+0000 7f342b44e640 1 --2- 172.21.15.83:0/485744847 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f34240d4760 0x7f34240d0370 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.094+0000 7f342b44e640 1 --2- 172.21.15.83:0/485744847 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f34240b4170 0x7f34240b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.094+0000 7f342b44e640 1 -- 172.21.15.83:0/485744847 >> 172.21.15.83:0/485744847 conn(0x7f34240d3c40 msgr2=0x7f34240d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:47.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.094+0000 7f342b44e640 1 -- 172.21.15.83:0/485744847 shutdown_connections 2024-03-06T07:32:47.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.094+0000 7f342b44e640 1 -- 172.21.15.83:0/485744847 wait complete. 2024-03-06T07:32:47.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.095+0000 7f342b44e640 1 Processor -- start 2024-03-06T07:32:47.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.095+0000 7f342b44e640 1 -- start start 2024-03-06T07:32:47.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.096+0000 7f342b44e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34240b4170 0x7f34240cd780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.096+0000 7f342b44e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f34240b4dd0 0x7f34240c98a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.096+0000 7f342b44e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f34240c7940 0x7f3424154cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.096+0000 7f342b44e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34240b47d0 con 0x7f34240b4170 2024-03-06T07:32:47.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.096+0000 7f342b44e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34240b4550 con 0x7f34240b4dd0 2024-03-06T07:32:47.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.096+0000 7f342b44e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34240b4950 con 0x7f34240c7940 2024-03-06T07:32:47.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.096+0000 7f34291ea640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34240b4170 0x7f34240cd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:47.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.096+0000 7f34291ea640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34240b4170 0x7f34240cd780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36260/0 (socket says 172.21.15.83:36260) 2024-03-06T07:32:47.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.096+0000 7f34291ea640 1 -- 172.21.15.83:0/2337278665 learned_addr learned my addr 172.21.15.83:0/2337278665 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:47.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.096+0000 7f34289e9640 1 --2- 172.21.15.83:0/2337278665 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f34240b4dd0 0x7f34240c98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:47.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.096+0000 7f34299eb640 1 --2- 172.21.15.83:0/2337278665 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f34240c7940 0x7f3424154cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:47.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.096+0000 7f34291ea640 1 -- 172.21.15.83:0/2337278665 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f34240c7940 msgr2=0x7f3424154cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.096+0000 7f34291ea640 1 --2- 172.21.15.83:0/2337278665 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f34240c7940 0x7f3424154cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.096+0000 7f34291ea640 1 -- 172.21.15.83:0/2337278665 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f34240b4dd0 msgr2=0x7f34240c98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.096+0000 7f34291ea640 1 --2- 172.21.15.83:0/2337278665 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f34240b4dd0 0x7f34240c98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.096+0000 7f34291ea640 1 -- 172.21.15.83:0/2337278665 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34240c7f00 con 0x7f34240b4170 2024-03-06T07:32:47.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.096+0000 7f34289e9640 1 --2- 172.21.15.83:0/2337278665 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f34240b4dd0 0x7f34240c98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:47.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.096+0000 7f34291ea640 1 --2- 172.21.15.83:0/2337278665 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34240b4170 0x7f34240cd780 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7f341800ebe0 tx=0x7f341800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:47.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.096+0000 7f3415ffb640 1 -- 172.21.15.83:0/2337278665 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3418017070 con 0x7f34240b4170 2024-03-06T07:32:47.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.096+0000 7f3415ffb640 1 -- 172.21.15.83:0/2337278665 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f341800cc30 con 0x7f34240b4170 2024-03-06T07:32:47.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.096+0000 7f3415ffb640 1 -- 172.21.15.83:0/2337278665 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3418013470 con 0x7f34240b4170 2024-03-06T07:32:47.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.096+0000 7f342b44e640 1 -- 172.21.15.83:0/2337278665 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34240cb770 con 0x7f34240b4170 2024-03-06T07:32:47.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.097+0000 7f342b44e640 1 -- 172.21.15.83:0/2337278665 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34240cbd30 con 0x7f34240b4170 2024-03-06T07:32:47.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.098+0000 7f342b44e640 1 -- 172.21.15.83:0/2337278665 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f342404f330 con 0x7f34240b4170 2024-03-06T07:32:47.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.098+0000 7f3415ffb640 1 -- 172.21.15.83:0/2337278665 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f341801a070 con 0x7f34240b4170 2024-03-06T07:32:47.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.099+0000 7f3415ffb640 1 --2- 172.21.15.83:0/2337278665 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f340806b4d0 0x7f340806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.099+0000 7f34289e9640 1 --2- 172.21.15.83:0/2337278665 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f340806b4d0 0x7f340806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:47.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.099+0000 7f3415ffb640 1 -- 172.21.15.83:0/2337278665 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f341808a1c0 con 0x7f34240b4170 2024-03-06T07:32:47.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.099+0000 7f34289e9640 1 --2- 172.21.15.83:0/2337278665 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f340806b4d0 0x7f340806d990 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f3420036000 tx=0x7f342003b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:47.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.103+0000 7f3415ffb640 1 -- 172.21.15.83:0/2337278665 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3418059370 con 0x7f34240b4170 2024-03-06T07:32:47.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.275+0000 7f342b44e640 1 -- 172.21.15.83:0/2337278665 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34240733b0 con 0x7f34240b4170 2024-03-06T07:32:47.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.276+0000 7f3415ffb640 1 -- 172.21.15.83:0/2337278665 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v135) v1 ==== 74+0+13172 (secure 0 0 0) 0x7f3418058c70 con 0x7f34240b4170 2024-03-06T07:32:47.278 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:32:47.278 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":135,"fsid":"039877dd-d2ce-45d2-8e77-ac5eb7e1d55a","created":"2024-03-06T07:21:34.186715+0000","modified":"2024-03-06T07:32:43.991432+0000","last_up_change":"2024-03-06T07:21:45.230363+0000","last_in_change":"2024-03-06T07:21:36.411792+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:46.361929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c61c90c5-4903-4b8c-acf9-0544bf2d2d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6809","nonce":1635427425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6811","nonce":1635427425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6819","nonce":1635427425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":1635427425},{"type":"v1","addr":"172.21.15.83:6813","nonce":1635427425}]},"public_addr":"172.21.15.83:6809/1635427425","cluster_addr":"172.21.15.83:6811/1635427425","heartbeat_back_addr":"172.21.15.83:6819/1635427425","heartbeat_front_addr":"172.21.15.83:6813/1635427425","state":["exists","up"]},{"osd":1,"uuid":"9d6eddc0-6755-4bfc-bdf8-5d76456e7b99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6817","nonce":1822925240}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6821","nonce":1822925240}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6828","nonce":1822925240}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6823","nonce":1822925240},{"type":"v1","addr":"172.21.15.83:6825","nonce":1822925240}]},"public_addr":"172.21.15.83:6817/1822925240","cluster_addr":"172.21.15.83:6821/1822925240","heartbeat_back_addr":"172.21.15.83:6828/1822925240","heartbeat_front_addr":"172.21.15.83:6825/1822925240","state":["exists","up"]},{"osd":2,"uuid":"6d74c643-7301-4daf-91f4-cfd3203a7ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6801","nonce":2591342133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6803","nonce":2591342133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6807","nonce":2591342133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2591342133},{"type":"v1","addr":"172.21.15.83:6805","nonce":2591342133}]},"public_addr":"172.21.15.83:6801/2591342133","cluster_addr":"172.21.15.83:6803/2591342133","heartbeat_back_addr":"172.21.15.83:6807/2591342133","heartbeat_front_addr":"172.21.15.83:6805/2591342133","state":["exists","up"]},{"osd":3,"uuid":"fa07c172-62c0-485f-89ec-495300de270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6818","nonce":2656563756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6824","nonce":2656563756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6831","nonce":2656563756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6827","nonce":2656563756},{"type":"v1","addr":"172.21.15.83:6829","nonce":2656563756}]},"public_addr":"172.21.15.83:6818/2656563756","cluster_addr":"172.21.15.83:6824/2656563756","heartbeat_back_addr":"172.21.15.83:6831/2656563756","heartbeat_front_addr":"172.21.15.83:6829/2656563756","state":["exists","up"]},{"osd":4,"uuid":"3b420d33-2d34-4f12-ae2e-98ed0188d304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6817","nonce":3256618830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6819","nonce":3256618830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6827","nonce":3256618830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":3256618830},{"type":"v1","addr":"172.21.15.183:6823","nonce":3256618830}]},"public_addr":"172.21.15.183:6817/3256618830","cluster_addr":"172.21.15.183:6819/3256618830","heartbeat_back_addr":"172.21.15.183:6827/3256618830","heartbeat_front_addr":"172.21.15.183:6823/3256618830","state":["exists","up"]},{"osd":5,"uuid":"0b171206-23f4-4d38-b977-4a88e15eebb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2957306877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2957306877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2957306877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2957306877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2957306877}]},"public_addr":"172.21.15.183:6809/2957306877","cluster_addr":"172.21.15.183:6811/2957306877","heartbeat_back_addr":"172.21.15.183:6815/2957306877","heartbeat_front_addr":"172.21.15.183:6813/2957306877","state":["exists","up"]},{"osd":6,"uuid":"5573c862-78b3-4c8a-8443-9288d6872d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6801","nonce":2649185225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6803","nonce":2649185225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6807","nonce":2649185225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2649185225},{"type":"v1","addr":"172.21.15.183:6805","nonce":2649185225}]},"public_addr":"172.21.15.183:6801/2649185225","cluster_addr":"172.21.15.183:6803/2649185225","heartbeat_back_addr":"172.21.15.183:6807/2649185225","heartbeat_front_addr":"172.21.15.183:6805/2649185225","state":["exists","up"]},{"osd":7,"uuid":"6f176f58-87b8-4b3c-9526-316b7024e6d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6822","nonce":3336566695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6826","nonce":3336566695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6831","nonce":3336566695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3336566695},{"type":"v1","addr":"172.21.15.183:6829","nonce":3336566695}]},"public_addr":"172.21.15.183:6822/3336566695","cluster_addr":"172.21.15.183:6826/3336566695","heartbeat_back_addr":"172.21.15.183:6831/3336566695","heartbeat_front_addr":"172.21.15.183:6829/3336566695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.617126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.559604+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.581121+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.631357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.613614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:41.564385+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.83:6837/1401317546":"2024-03-07T07:32:40.983867+0000","172.21.15.83:6835/1401317546":"2024-03-07T07:32:40.983867+0000","172.21.15.83:6836/3544993654":"2024-03-07T07:32:40.983854+0000","172.21.15.83:6834/3544993654":"2024-03-07T07:32:40.983854+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:47.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.278+0000 7f342b44e640 1 -- 172.21.15.83:0/2337278665 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f340806b4d0 msgr2=0x7f340806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.278+0000 7f342b44e640 1 --2- 172.21.15.83:0/2337278665 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f340806b4d0 0x7f340806d990 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f3420036000 tx=0x7f342003b040 comp rx=0 tx=0).stop 2024-03-06T07:32:47.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.278+0000 7f342b44e640 1 -- 172.21.15.83:0/2337278665 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34240b4170 msgr2=0x7f34240cd780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.278+0000 7f342b44e640 1 --2- 172.21.15.83:0/2337278665 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34240b4170 0x7f34240cd780 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7f341800ebe0 tx=0x7f341800c520 comp rx=0 tx=0).stop 2024-03-06T07:32:47.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.278+0000 7f342b44e640 1 -- 172.21.15.83:0/2337278665 shutdown_connections 2024-03-06T07:32:47.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.278+0000 7f342b44e640 1 --2- 172.21.15.83:0/2337278665 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f340806b4d0 0x7f340806d990 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.278+0000 7f342b44e640 1 --2- 172.21.15.83:0/2337278665 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f34240c7940 0x7f3424154cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.278+0000 7f342b44e640 1 --2- 172.21.15.83:0/2337278665 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f34240b4dd0 0x7f34240c98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.278+0000 7f342b44e640 1 --2- 172.21.15.83:0/2337278665 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34240b4170 0x7f34240cd780 unknown :-1 s=CLOSED pgs=2035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.278+0000 7f342b44e640 1 -- 172.21.15.83:0/2337278665 >> 172.21.15.83:0/2337278665 conn(0x7f34240d3c40 msgr2=0x7f34240b70f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:47.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.278+0000 7f342b44e640 1 -- 172.21.15.83:0/2337278665 shutdown_connections 2024-03-06T07:32:47.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:47.278+0000 7f342b44e640 1 -- 172.21.15.83:0/2337278665 wait complete. 2024-03-06T07:32:48.291 INFO:tasks.ceph:Scrubbing osd.0 2024-03-06T07:32:48.292 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:32:48.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.400+0000 7f28e9502640 1 Processor -- start 2024-03-06T07:32:48.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.400+0000 7f28e9502640 1 -- start start 2024-03-06T07:32:48.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.400+0000 7f28e9502640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f28e415dfa0 0x7f28e40d18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.400+0000 7f28e9502640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f28e40d4630 0x7f28e40d22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.401+0000 7f28e9502640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f28e40d2820 0x7f28e40d05f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.401+0000 7f28e9502640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28e40d2100 con 0x7f28e40d4630 2024-03-06T07:32:48.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.401+0000 7f28e9502640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28e415e380 con 0x7f28e415dfa0 2024-03-06T07:32:48.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.401+0000 7f28e9502640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28e40d1e30 con 0x7f28e40d2820 2024-03-06T07:32:48.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.401+0000 7f28dbfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f28e40d4630 0x7f28e40d22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.401+0000 7f28dbfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f28e40d4630 0x7f28e40d22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36274/0 (socket says 172.21.15.83:36274) 2024-03-06T07:32:48.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.401+0000 7f28dbfff640 1 -- 172.21.15.83:0/2826012195 learned_addr learned my addr 172.21.15.83:0/2826012195 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:48.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.401+0000 7f28dbfff640 1 -- 172.21.15.83:0/2826012195 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f28e40d2820 msgr2=0x7f28e40d05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:48.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.401+0000 7f28dbfff640 1 --2- 172.21.15.83:0/2826012195 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f28e40d2820 0x7f28e40d05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.401+0000 7f28dbfff640 1 -- 172.21.15.83:0/2826012195 >> v2:172.21.15.183:3300/0 conn(0x7f28e415dfa0 msgr2=0x7f28e40d18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:48.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.401+0000 7f28dbfff640 1 --2- 172.21.15.83:0/2826012195 >> v2:172.21.15.183:3300/0 conn(0x7f28e415dfa0 0x7f28e40d18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.401+0000 7f28dbfff640 1 -- 172.21.15.83:0/2826012195 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28e40cf470 con 0x7f28e40d4630 2024-03-06T07:32:48.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.402+0000 7f28dbfff640 1 --2- 172.21.15.83:0/2826012195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28e40d4630 0x7f28e40d22e0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f28e408d560 tx=0x7f28cc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a0abe1de666a7d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.402+0000 7f28e27fc640 1 -- 172.21.15.83:0/2826012195 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28cc02e070 con 0x7f28e40d4630 2024-03-06T07:32:48.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.402+0000 7f28e27fc640 1 -- 172.21.15.83:0/2826012195 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28cc037070 con 0x7f28e40d4630 2024-03-06T07:32:48.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.402+0000 7f28e27fc640 1 -- 172.21.15.83:0/2826012195 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28cc031a00 con 0x7f28e40d4630 2024-03-06T07:32:48.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.402+0000 7f28e9502640 1 -- 172.21.15.83:0/2826012195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28e40d4630 msgr2=0x7f28e40d22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.402+0000 7f28e9502640 1 --2- 172.21.15.83:0/2826012195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28e40d4630 0x7f28e40d22e0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f28e408d560 tx=0x7f28cc02b880 comp rx=0 tx=0).stop 2024-03-06T07:32:48.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.402+0000 7f28e9502640 1 -- 172.21.15.83:0/2826012195 shutdown_connections 2024-03-06T07:32:48.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.402+0000 7f28e9502640 1 --2- 172.21.15.83:0/2826012195 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f28e40d2820 0x7f28e40d05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.402+0000 7f28e9502640 1 --2- 172.21.15.83:0/2826012195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28e40d4630 0x7f28e40d22e0 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.402+0000 7f28e9502640 1 --2- 172.21.15.83:0/2826012195 >> v2:172.21.15.183:3300/0 conn(0x7f28e415dfa0 0x7f28e40d18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.402+0000 7f28e9502640 1 -- 172.21.15.83:0/2826012195 >> 172.21.15.83:0/2826012195 conn(0x7f28e409d000 msgr2=0x7f28e40d3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:48.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.402+0000 7f28e9502640 1 -- 172.21.15.83:0/2826012195 shutdown_connections 2024-03-06T07:32:48.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.402+0000 7f28e9502640 1 -- 172.21.15.83:0/2826012195 wait complete. 2024-03-06T07:32:48.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.404+0000 7f28e9502640 1 Processor -- start 2024-03-06T07:32:48.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.404+0000 7f28e9502640 1 -- start start 2024-03-06T07:32:48.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.404+0000 7f28e9502640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f28e40d2820 0x7f28e415be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.404+0000 7f28e9502640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28e40d4630 0x7f28e40b72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.404+0000 7f28e9502640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f28e415dfa0 0x7f28e40a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.404+0000 7f28e9502640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e40d0b30 con 0x7f28e40d4630 2024-03-06T07:32:48.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.404+0000 7f28e9502640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e40cfa30 con 0x7f28e40d2820 2024-03-06T07:32:48.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.404+0000 7f28e9502640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28e40d0cb0 con 0x7f28e415dfa0 2024-03-06T07:32:48.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.404+0000 7f28dbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28e40d4630 0x7f28e40b72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.404+0000 7f28dbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28e40d4630 0x7f28e40b72d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36276/0 (socket says 172.21.15.83:36276) 2024-03-06T07:32:48.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.404+0000 7f28dbfff640 1 -- 172.21.15.83:0/1775211002 learned_addr learned my addr 172.21.15.83:0/1775211002 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:48.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.404+0000 7f28e2ffd640 1 --2- 172.21.15.83:0/1775211002 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f28e40d2820 0x7f28e415be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.404+0000 7f28e37fe640 1 --2- 172.21.15.83:0/1775211002 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f28e415dfa0 0x7f28e40a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.405+0000 7f28dbfff640 1 -- 172.21.15.83:0/1775211002 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f28e415dfa0 msgr2=0x7f28e40a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.405+0000 7f28dbfff640 1 --2- 172.21.15.83:0/1775211002 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f28e415dfa0 0x7f28e40a0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.405+0000 7f28dbfff640 1 -- 172.21.15.83:0/1775211002 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f28e40d2820 msgr2=0x7f28e415be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.405+0000 7f28dbfff640 1 --2- 172.21.15.83:0/1775211002 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f28e40d2820 0x7f28e415be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.405+0000 7f28dbfff640 1 -- 172.21.15.83:0/1775211002 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28e40b79b0 con 0x7f28e40d4630 2024-03-06T07:32:48.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.405+0000 7f28dbfff640 1 --2- 172.21.15.83:0/1775211002 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28e40d4630 0x7f28e40b72d0 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7f28cc02b8c0 tx=0x7f28cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.405+0000 7f28db7fe640 1 -- 172.21.15.83:0/1775211002 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28cc032880 con 0x7f28e40d4630 2024-03-06T07:32:48.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.405+0000 7f28db7fe640 1 -- 172.21.15.83:0/1775211002 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28cc032a20 con 0x7f28e40d4630 2024-03-06T07:32:48.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.405+0000 7f28e9502640 1 -- 172.21.15.83:0/1775211002 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28e40b7c90 con 0x7f28e40d4630 2024-03-06T07:32:48.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.405+0000 7f28db7fe640 1 -- 172.21.15.83:0/1775211002 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28cc032e10 con 0x7f28e40d4630 2024-03-06T07:32:48.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.405+0000 7f28e9502640 1 -- 172.21.15.83:0/1775211002 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28e40b8250 con 0x7f28e40d4630 2024-03-06T07:32:48.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.406+0000 7f28e9502640 1 -- 172.21.15.83:0/1775211002 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f28e404f330 con 0x7f28e40d4630 2024-03-06T07:32:48.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.406+0000 7f28db7fe640 1 -- 172.21.15.83:0/1775211002 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f28cc04c060 con 0x7f28e40d4630 2024-03-06T07:32:48.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.407+0000 7f28db7fe640 1 --2- 172.21.15.83:0/1775211002 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f28c406b500 0x7f28c406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.407+0000 7f28db7fe640 1 -- 172.21.15.83:0/1775211002 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f28cc0ab9b0 con 0x7f28e40d4630 2024-03-06T07:32:48.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.407+0000 7f28e2ffd640 1 --2- 172.21.15.83:0/1775211002 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f28c406b500 0x7f28c406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.407+0000 7f28db7fe640 1 --2- 172.21.15.83:0/1775211002 >> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] conn(0x7f28c4071460 0x7f28c4073880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.407+0000 7f28db7fe640 1 -- 172.21.15.83:0/1775211002 --> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f28cc039e50 con 0x7f28c4071460 2024-03-06T07:32:48.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.407+0000 7f28db7fe640 1 -- 172.21.15.83:0/1775211002 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_get_version_reply(handle=1 version=135) v2 ==== 24+0+0 (secure 0 0 0) 0x7f28cc0b02c0 con 0x7f28e40d4630 2024-03-06T07:32:48.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.407+0000 7f28e37fe640 1 --2- 172.21.15.83:0/1775211002 >> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] conn(0x7f28c4071460 0x7f28c4073880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.407+0000 7f28e2ffd640 1 --2- 172.21.15.83:0/1775211002 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f28c406b500 0x7f28c406d9c0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f28e409db90 tx=0x7f28d4009060 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.407+0000 7f28e37fe640 1 --2- 172.21.15.83:0/1775211002 >> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] conn(0x7f28c4071460 0x7f28c4073880 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.408+0000 7f28db7fe640 1 -- 172.21.15.83:0/1775211002 <== osd.0 v2:172.21.15.83:6808/1635427425 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f28cc039e50 con 0x7f28c4071460 2024-03-06T07:32:48.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.421+0000 7f28e9502640 1 -- 172.21.15.83:0/1775211002 --> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f28e40b9ba0 con 0x7f28c4071460 2024-03-06T07:32:48.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.421+0000 7f28db7fe640 1 -- 172.21.15.83:0/1775211002 <== osd.0 v2:172.21.15.83:6808/1635427425 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f28e40b9ba0 con 0x7f28c4071460 2024-03-06T07:32:48.423 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:32:48.424 INFO:teuthology.orchestra.run.smithi083.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:32:48.424 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:32:48.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.422+0000 7f28e9502640 1 -- 172.21.15.83:0/1775211002 >> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] conn(0x7f28c4071460 msgr2=0x7f28c4073880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.422+0000 7f28e9502640 1 --2- 172.21.15.83:0/1775211002 >> [v2:172.21.15.83:6808/1635427425,v1:172.21.15.83:6809/1635427425] conn(0x7f28c4071460 0x7f28c4073880 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.422+0000 7f28e9502640 1 -- 172.21.15.83:0/1775211002 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f28c406b500 msgr2=0x7f28c406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.422+0000 7f28e9502640 1 --2- 172.21.15.83:0/1775211002 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f28c406b500 0x7f28c406d9c0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f28e409db90 tx=0x7f28d4009060 comp rx=0 tx=0).stop 2024-03-06T07:32:48.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.422+0000 7f28e9502640 1 -- 172.21.15.83:0/1775211002 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28e40d4630 msgr2=0x7f28e40b72d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.422+0000 7f28e9502640 1 --2- 172.21.15.83:0/1775211002 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28e40d4630 0x7f28e40b72d0 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7f28cc02b8c0 tx=0x7f28cc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.422+0000 7f28e37fe640 1 -- 172.21.15.83:0/1775211002 reap_dead start 2024-03-06T07:32:48.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.422+0000 7f28e9502640 1 -- 172.21.15.83:0/1775211002 shutdown_connections 2024-03-06T07:32:48.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.422+0000 7f28e9502640 1 -- 172.21.15.83:0/1775211002 >> 172.21.15.83:0/1775211002 conn(0x7f28e409d000 msgr2=0x7f28e40cc3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:48.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.422+0000 7f28e9502640 1 -- 172.21.15.83:0/1775211002 shutdown_connections 2024-03-06T07:32:48.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.422+0000 7f28e9502640 1 -- 172.21.15.83:0/1775211002 wait complete. 2024-03-06T07:32:48.434 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-06T07:32:48.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.533+0000 7fda4b0c2640 1 Processor -- start 2024-03-06T07:32:48.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.533+0000 7fda4b0c2640 1 -- start start 2024-03-06T07:32:48.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.534+0000 7fda4b0c2640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fda440cf560 0x7fda4406d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.534+0000 7fda4b0c2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fda440d06b0 0x7fda440adfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.534+0000 7fda4b0c2640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda4406d860 con 0x7fda4415f1c0 2024-03-06T07:32:48.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.534+0000 7fda4b0c2640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda440cf940 con 0x7fda440d06b0 2024-03-06T07:32:48.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.534+0000 7fda4b0c2640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda440ad860 con 0x7fda440cf560 2024-03-06T07:32:48.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.534+0000 7fda4965f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fda440d06b0 0x7fda440adfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.534+0000 7fda43fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fda4415f1c0 0x7fda440cffa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:53828/0 (socket says 172.21.15.83:53828) 2024-03-06T07:32:48.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.534+0000 7fda43fff640 1 -- 172.21.15.83:0/2678606521 learned_addr learned my addr 172.21.15.83:0/2678606521 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:32:48.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.534+0000 7fda4965f640 1 -- 172.21.15.83:0/2678606521 >> v2:172.21.15.183:3300/0 conn(0x7fda440cf560 msgr2=0x7fda4406d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:48.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.534+0000 7fda4965f640 1 --2- 172.21.15.83:0/2678606521 >> v2:172.21.15.183:3300/0 conn(0x7fda440cf560 0x7fda4406d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.534+0000 7fda4965f640 1 -- 172.21.15.83:0/2678606521 >> v1:172.21.15.83:6789/0 conn(0x7fda4415f1c0 legacy=0x7fda440cffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.534+0000 7fda437fe640 1 -- 172.21.15.83:0/2678606521 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2145938465 0 0) 0x7fda4406d860 con 0x7fda4415f1c0 2024-03-06T07:32:48.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.534+0000 7fda4965f640 1 -- 172.21.15.83:0/2678606521 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda4406db40 con 0x7fda440d06b0 2024-03-06T07:32:48.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.535+0000 7fda4965f640 1 --2- 172.21.15.83:0/2678606521 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fda440d06b0 0x7fda440adfd0 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7fda44068410 tx=0x7fda3c02bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67542a5df003c9d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.535+0000 7fda437fe640 1 -- 172.21.15.83:0/2678606521 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda3c039070 con 0x7fda440d06b0 2024-03-06T07:32:48.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.535+0000 7fda437fe640 1 -- 172.21.15.83:0/2678606521 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda3c02cb60 con 0x7fda440d06b0 2024-03-06T07:32:48.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.535+0000 7fda437fe640 1 -- 172.21.15.83:0/2678606521 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda3c034440 con 0x7fda440d06b0 2024-03-06T07:32:48.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.536+0000 7fda4b0c2640 1 -- 172.21.15.83:0/2678606521 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fda440d06b0 msgr2=0x7fda440adfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.536+0000 7fda4b0c2640 1 --2- 172.21.15.83:0/2678606521 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fda440d06b0 0x7fda440adfd0 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7fda44068410 tx=0x7fda3c02bdb0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.536+0000 7fda4b0c2640 1 -- 172.21.15.83:0/2678606521 shutdown_connections 2024-03-06T07:32:48.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.536+0000 7fda4b0c2640 1 --2- 172.21.15.83:0/2678606521 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fda440d06b0 0x7fda440adfd0 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.536+0000 7fda4b0c2640 1 --2- 172.21.15.83:0/2678606521 >> v2:172.21.15.183:3300/0 conn(0x7fda440cf560 0x7fda4406d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.536+0000 7fda4b0c2640 1 -- 172.21.15.83:0/2678606521 >> 172.21.15.83:0/2678606521 conn(0x7fda4406b190 msgr2=0x7fda4406b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:48.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.536+0000 7fda4b0c2640 1 -- 172.21.15.83:0/2678606521 shutdown_connections 2024-03-06T07:32:48.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.536+0000 7fda4b0c2640 1 -- 172.21.15.83:0/2678606521 wait complete. 2024-03-06T07:32:48.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.537+0000 7fda4b0c2640 1 Processor -- start 2024-03-06T07:32:48.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.537+0000 7fda4b0c2640 1 -- start start 2024-03-06T07:32:48.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.538+0000 7fda4b0c2640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fda440cf560 0x7fda441536f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.538+0000 7fda4b0c2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fda440d06b0 0x7fda441f3b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.538+0000 7fda4b0c2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fda4415f1c0 0x7fda44071ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.538+0000 7fda4b0c2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda4406d550 con 0x7fda440d06b0 2024-03-06T07:32:48.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.538+0000 7fda4b0c2640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda440adab0 con 0x7fda440cf560 2024-03-06T07:32:48.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.538+0000 7fda4b0c2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda440ae510 con 0x7fda4415f1c0 2024-03-06T07:32:48.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.538+0000 7fda43fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fda440d06b0 0x7fda441f3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.538+0000 7fda43fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fda440d06b0 0x7fda441f3b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36286/0 (socket says 172.21.15.83:36286) 2024-03-06T07:32:48.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.538+0000 7fda43fff640 1 -- 172.21.15.83:0/2291592845 learned_addr learned my addr 172.21.15.83:0/2291592845 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:48.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.538+0000 7fda48e5e640 1 --2- 172.21.15.83:0/2291592845 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fda440cf560 0x7fda441536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.538+0000 7fda4965f640 1 --2- 172.21.15.83:0/2291592845 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fda4415f1c0 0x7fda44071ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.538+0000 7fda43fff640 1 -- 172.21.15.83:0/2291592845 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fda4415f1c0 msgr2=0x7fda44071ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.538+0000 7fda43fff640 1 --2- 172.21.15.83:0/2291592845 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fda4415f1c0 0x7fda44071ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.538+0000 7fda43fff640 1 -- 172.21.15.83:0/2291592845 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fda440cf560 msgr2=0x7fda441536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.538+0000 7fda43fff640 1 --2- 172.21.15.83:0/2291592845 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fda440cf560 0x7fda441536f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.538+0000 7fda43fff640 1 -- 172.21.15.83:0/2291592845 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda4415c130 con 0x7fda440d06b0 2024-03-06T07:32:48.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.538+0000 7fda43fff640 1 --2- 172.21.15.83:0/2291592845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fda440d06b0 0x7fda441f3b20 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7fda3800c460 tx=0x7fda3800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.539+0000 7fda417fa640 1 -- 172.21.15.83:0/2291592845 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda38018070 con 0x7fda440d06b0 2024-03-06T07:32:48.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.539+0000 7fda417fa640 1 -- 172.21.15.83:0/2291592845 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda38007500 con 0x7fda440d06b0 2024-03-06T07:32:48.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.539+0000 7fda4b0c2640 1 -- 172.21.15.83:0/2291592845 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda4415c6b0 con 0x7fda440d06b0 2024-03-06T07:32:48.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.539+0000 7fda417fa640 1 -- 172.21.15.83:0/2291592845 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda38004a20 con 0x7fda440d06b0 2024-03-06T07:32:48.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.539+0000 7fda4b0c2640 1 -- 172.21.15.83:0/2291592845 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda4415cc70 con 0x7fda440d06b0 2024-03-06T07:32:48.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.540+0000 7fda4b0c2640 1 -- 172.21.15.83:0/2291592845 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda4404f330 con 0x7fda440d06b0 2024-03-06T07:32:48.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.540+0000 7fda417fa640 1 -- 172.21.15.83:0/2291592845 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fda3801b070 con 0x7fda440d06b0 2024-03-06T07:32:48.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.540+0000 7fda417fa640 1 --2- 172.21.15.83:0/2291592845 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fda3006b4d0 0x7fda3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.541+0000 7fda417fa640 1 -- 172.21.15.83:0/2291592845 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fda380891c0 con 0x7fda440d06b0 2024-03-06T07:32:48.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.541+0000 7fda48e5e640 1 --2- 172.21.15.83:0/2291592845 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fda3006b4d0 0x7fda3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.541+0000 7fda48e5e640 1 --2- 172.21.15.83:0/2291592845 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fda3006b4d0 0x7fda3006d990 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fda34009890 tx=0x7fda34008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.545+0000 7fda417fa640 1 -- 172.21.15.83:0/2291592845 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fda380582e0 con 0x7fda440d06b0 2024-03-06T07:32:48.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.714+0000 7fda4b0c2640 1 -- 172.21.15.83:0/2291592845 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7fda441586f0 con 0x7fda3006b4d0 2024-03-06T07:32:48.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.714+0000 7fda417fa640 1 -- 172.21.15.83:0/2291592845 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fda441586f0 con 0x7fda3006b4d0 2024-03-06T07:32:48.716 INFO:teuthology.orchestra.run.smithi083.stderr:instructed osd(s) 0 to deep-scrub 2024-03-06T07:32:48.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.716+0000 7fda4b0c2640 1 -- 172.21.15.83:0/2291592845 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fda3006b4d0 msgr2=0x7fda3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.716+0000 7fda4b0c2640 1 --2- 172.21.15.83:0/2291592845 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fda3006b4d0 0x7fda3006d990 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fda34009890 tx=0x7fda34008040 comp rx=0 tx=0).stop 2024-03-06T07:32:48.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.716+0000 7fda4b0c2640 1 -- 172.21.15.83:0/2291592845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fda440d06b0 msgr2=0x7fda441f3b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.716+0000 7fda4b0c2640 1 --2- 172.21.15.83:0/2291592845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fda440d06b0 0x7fda441f3b20 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7fda3800c460 tx=0x7fda3800c920 comp rx=0 tx=0).stop 2024-03-06T07:32:48.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.716+0000 7fda4b0c2640 1 -- 172.21.15.83:0/2291592845 shutdown_connections 2024-03-06T07:32:48.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.716+0000 7fda4b0c2640 1 --2- 172.21.15.83:0/2291592845 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fda3006b4d0 0x7fda3006d990 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.716+0000 7fda4b0c2640 1 --2- 172.21.15.83:0/2291592845 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fda4415f1c0 0x7fda44071ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.716+0000 7fda4b0c2640 1 --2- 172.21.15.83:0/2291592845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fda440d06b0 0x7fda441f3b20 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.716+0000 7fda4b0c2640 1 --2- 172.21.15.83:0/2291592845 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fda440cf560 0x7fda441536f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.716+0000 7fda4b0c2640 1 -- 172.21.15.83:0/2291592845 >> 172.21.15.83:0/2291592845 conn(0x7fda4406b190 msgr2=0x7fda440b05d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:48.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.716+0000 7fda4b0c2640 1 -- 172.21.15.83:0/2291592845 shutdown_connections 2024-03-06T07:32:48.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.716+0000 7fda4b0c2640 1 -- 172.21.15.83:0/2291592845 wait complete. 2024-03-06T07:32:48.729 INFO:tasks.ceph:Scrubbing osd.1 2024-03-06T07:32:48.729 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:32:48.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.830+0000 7fb2ca281640 1 Processor -- start 2024-03-06T07:32:48.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.830+0000 7fb2ca281640 1 -- start start 2024-03-06T07:32:48.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.831+0000 7fb2ca281640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb2c415ac00 0x7fb2c406c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.831+0000 7fb2ca281640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c415f5c0 con 0x7fb2c40708b0 2024-03-06T07:32:48.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.831+0000 7fb2ca281640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c415afe0 con 0x7fb2c4070d10 2024-03-06T07:32:48.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.831+0000 7fb2ca281640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c40cdc10 con 0x7fb2c415ac00 2024-03-06T07:32:48.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.831+0000 7fb2c37fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fb2c40708b0 0x7fb2c415e7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:53840/0 (socket says 172.21.15.83:53840) 2024-03-06T07:32:48.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.831+0000 7fb2c37fe640 1 -- 172.21.15.83:0/2613723328 learned_addr learned my addr 172.21.15.83:0/2613723328 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:32:48.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.831+0000 7fb2c2ffd640 1 -- 172.21.15.83:0/2613723328 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3966584918 0 0) 0x7fb2c415f5c0 con 0x7fb2c40708b0 2024-03-06T07:32:48.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.831+0000 7fb2c2ffd640 1 -- 172.21.15.83:0/2613723328 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2ac003520 con 0x7fb2c40708b0 2024-03-06T07:32:48.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.831+0000 7fb2c3fff640 1 --2- 172.21.15.83:0/2613723328 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb2c415ac00 0x7fb2c406c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.832+0000 7fb2c2ffd640 1 -- 172.21.15.83:0/2613723328 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4092717153 0 0) 0x7fb2ac003520 con 0x7fb2c40708b0 2024-03-06T07:32:48.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.832+0000 7fb2c2ffd640 1 -- 172.21.15.83:0/2613723328 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb2c415ac00 msgr2=0x7fb2c406c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:48.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.832+0000 7fb2c2ffd640 1 --2- 172.21.15.83:0/2613723328 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb2c415ac00 0x7fb2c406c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.832+0000 7fb2c2ffd640 1 -- 172.21.15.83:0/2613723328 >> v1:172.21.15.183:6789/0 conn(0x7fb2c4070d10 legacy=0x7fb2c415eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.832+0000 7fb2c2ffd640 1 -- 172.21.15.83:0/2613723328 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c40cd000 con 0x7fb2c40708b0 2024-03-06T07:32:48.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.832+0000 7fb2c2ffd640 1 -- 172.21.15.83:0/2613723328 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7fb2b8001670 con 0x7fb2c40708b0 2024-03-06T07:32:48.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.832+0000 7fb2c2ffd640 1 -- 172.21.15.83:0/2613723328 >> v1:172.21.15.83:6789/0 conn(0x7fb2c40708b0 legacy=0x7fb2c415e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.832+0000 7fb2c2ffd640 1 --2- 172.21.15.83:0/2613723328 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2ac0039e0 0x7fb2ac003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.832+0000 7fb2c2ffd640 1 --2- 172.21.15.83:0/2613723328 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb2ac004800 0x7fb2ac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.832+0000 7fb2c2ffd640 1 --2- 172.21.15.83:0/2613723328 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb2c415ac00 0x7fb2ac005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.832+0000 7fb2c2ffd640 1 -- 172.21.15.83:0/2613723328 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c40cdc10 con 0x7fb2ac0039e0 2024-03-06T07:32:48.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.832+0000 7fb2c2ffd640 1 -- 172.21.15.83:0/2613723328 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ac00baf0 con 0x7fb2ac004800 2024-03-06T07:32:48.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.832+0000 7fb2c2ffd640 1 -- 172.21.15.83:0/2613723328 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ac00bcf0 con 0x7fb2c415ac00 2024-03-06T07:32:48.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.832+0000 7fb2c3fff640 1 --2- 172.21.15.83:0/2613723328 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2ac0039e0 0x7fb2ac003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.832+0000 7fb2c37fe640 1 --2- 172.21.15.83:0/2613723328 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb2ac004800 0x7fb2ac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.832+0000 7fb2c881e640 1 --2- 172.21.15.83:0/2613723328 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb2c415ac00 0x7fb2ac005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.832+0000 7fb2c3fff640 1 -- 172.21.15.83:0/2613723328 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb2c415ac00 msgr2=0x7fb2ac005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:48.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.832+0000 7fb2c3fff640 1 --2- 172.21.15.83:0/2613723328 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb2c415ac00 0x7fb2ac005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.832+0000 7fb2c3fff640 1 -- 172.21.15.83:0/2613723328 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb2ac004800 msgr2=0x7fb2ac004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:48.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.832+0000 7fb2c3fff640 1 --2- 172.21.15.83:0/2613723328 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb2ac004800 0x7fb2ac004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.832+0000 7fb2c3fff640 1 -- 172.21.15.83:0/2613723328 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2ac003520 con 0x7fb2ac0039e0 2024-03-06T07:32:48.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.832+0000 7fb2c37fe640 1 --2- 172.21.15.83:0/2613723328 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb2ac004800 0x7fb2ac004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:48.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.833+0000 7fb2c3fff640 1 --2- 172.21.15.83:0/2613723328 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2ac0039e0 0x7fb2ac003e00 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7fb2b4009ac0 tx=0x7fb2ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f580601fa8513f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.833+0000 7fb2c2ffd640 1 -- 172.21.15.83:0/2613723328 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b4039070 con 0x7fb2ac0039e0 2024-03-06T07:32:48.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.833+0000 7fb2c2ffd640 1 -- 172.21.15.83:0/2613723328 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2b402c6b0 con 0x7fb2ac0039e0 2024-03-06T07:32:48.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.833+0000 7fb2c2ffd640 1 -- 172.21.15.83:0/2613723328 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b4009690 con 0x7fb2ac0039e0 2024-03-06T07:32:48.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.833+0000 7fb2ca281640 1 -- 172.21.15.83:0/2613723328 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2ac0039e0 msgr2=0x7fb2ac003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.833+0000 7fb2ca281640 1 --2- 172.21.15.83:0/2613723328 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2ac0039e0 0x7fb2ac003e00 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7fb2b4009ac0 tx=0x7fb2ac005830 comp rx=0 tx=0).stop 2024-03-06T07:32:48.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.833+0000 7fb2c881e640 1 -- 172.21.15.83:0/2613723328 reap_dead start 2024-03-06T07:32:48.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.833+0000 7fb2ca281640 1 -- 172.21.15.83:0/2613723328 shutdown_connections 2024-03-06T07:32:48.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.833+0000 7fb2ca281640 1 -- 172.21.15.83:0/2613723328 >> 172.21.15.83:0/2613723328 conn(0x7fb2c40b1650 msgr2=0x7fb2c40b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:48.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.833+0000 7fb2ca281640 1 -- 172.21.15.83:0/2613723328 shutdown_connections 2024-03-06T07:32:48.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.833+0000 7fb2ca281640 1 -- 172.21.15.83:0/2613723328 wait complete. 2024-03-06T07:32:48.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.834+0000 7fb2ca281640 1 Processor -- start 2024-03-06T07:32:48.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.835+0000 7fb2ca281640 1 -- start start 2024-03-06T07:32:48.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.835+0000 7fb2ca281640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb2ac0039e0 0x7fb2c414a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.835+0000 7fb2ca281640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2c41ef850 0x7fb2c415b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.835+0000 7fb2ca281640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb2c4149e10 0x7fb2c41f0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.835+0000 7fb2ca281640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c40cd8a0 con 0x7fb2c41ef850 2024-03-06T07:32:48.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.835+0000 7fb2ca281640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c40cd5c0 con 0x7fb2ac0039e0 2024-03-06T07:32:48.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.835+0000 7fb2ca281640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c4071190 con 0x7fb2c4149e10 2024-03-06T07:32:48.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.835+0000 7fb2c37fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2c41ef850 0x7fb2c415b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.835+0000 7fb2c37fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2c41ef850 0x7fb2c415b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36310/0 (socket says 172.21.15.83:36310) 2024-03-06T07:32:48.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.835+0000 7fb2c37fe640 1 -- 172.21.15.83:0/3921436766 learned_addr learned my addr 172.21.15.83:0/3921436766 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:48.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.836+0000 7fb2c881e640 1 --2- 172.21.15.83:0/3921436766 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb2c4149e10 0x7fb2c41f0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.836+0000 7fb2c3fff640 1 --2- 172.21.15.83:0/3921436766 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb2ac0039e0 0x7fb2c414a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.836+0000 7fb2c37fe640 1 -- 172.21.15.83:0/3921436766 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb2c4149e10 msgr2=0x7fb2c41f0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.836+0000 7fb2c37fe640 1 --2- 172.21.15.83:0/3921436766 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb2c4149e10 0x7fb2c41f0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.836+0000 7fb2c37fe640 1 -- 172.21.15.83:0/3921436766 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb2ac0039e0 msgr2=0x7fb2c414a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.836+0000 7fb2c37fe640 1 --2- 172.21.15.83:0/3921436766 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb2ac0039e0 0x7fb2c414a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.836+0000 7fb2c37fe640 1 -- 172.21.15.83:0/3921436766 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c41efd50 con 0x7fb2c41ef850 2024-03-06T07:32:48.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.836+0000 7fb2c37fe640 1 --2- 172.21.15.83:0/3921436766 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2c41ef850 0x7fb2c415b220 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8001b60 tx=0x7fb2b800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.836+0000 7fb2c0ff9640 1 -- 172.21.15.83:0/3921436766 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b800c8d0 con 0x7fb2c41ef850 2024-03-06T07:32:48.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.836+0000 7fb2c0ff9640 1 -- 172.21.15.83:0/3921436766 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2b8018070 con 0x7fb2c41ef850 2024-03-06T07:32:48.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.836+0000 7fb2ca281640 1 -- 172.21.15.83:0/3921436766 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2c41f0030 con 0x7fb2c41ef850 2024-03-06T07:32:48.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.836+0000 7fb2c0ff9640 1 -- 172.21.15.83:0/3921436766 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b800ccb0 con 0x7fb2c41ef850 2024-03-06T07:32:48.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.836+0000 7fb2ca281640 1 -- 172.21.15.83:0/3921436766 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2c41f05f0 con 0x7fb2c41ef850 2024-03-06T07:32:48.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.837+0000 7fb2ca281640 1 -- 172.21.15.83:0/3921436766 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb2c404f330 con 0x7fb2c41ef850 2024-03-06T07:32:48.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.838+0000 7fb2c0ff9640 1 -- 172.21.15.83:0/3921436766 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb2b801d460 con 0x7fb2c41ef850 2024-03-06T07:32:48.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.838+0000 7fb2c0ff9640 1 --2- 172.21.15.83:0/3921436766 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb2b006b4d0 0x7fb2b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.838+0000 7fb2c0ff9640 1 -- 172.21.15.83:0/3921436766 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fb2b8089c40 con 0x7fb2c41ef850 2024-03-06T07:32:48.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.838+0000 7fb2c3fff640 1 --2- 172.21.15.83:0/3921436766 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb2b006b4d0 0x7fb2b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.838+0000 7fb2c0ff9640 1 --2- 172.21.15.83:0/3921436766 >> [v2:172.21.15.83:6814/1822925240,v1:172.21.15.83:6817/1822925240] conn(0x7fb2b0071430 0x7fb2b0073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.838+0000 7fb2c0ff9640 1 -- 172.21.15.83:0/3921436766 --> [v2:172.21.15.83:6814/1822925240,v1:172.21.15.83:6817/1822925240] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb2b8021ec0 con 0x7fb2b0071430 2024-03-06T07:32:48.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.838+0000 7fb2c0ff9640 1 -- 172.21.15.83:0/3921436766 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_get_version_reply(handle=1 version=135) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb2b808c340 con 0x7fb2c41ef850 2024-03-06T07:32:48.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.838+0000 7fb2c881e640 1 --2- 172.21.15.83:0/3921436766 >> [v2:172.21.15.83:6814/1822925240,v1:172.21.15.83:6817/1822925240] conn(0x7fb2b0071430 0x7fb2b0073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.838+0000 7fb2c3fff640 1 --2- 172.21.15.83:0/3921436766 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb2b006b4d0 0x7fb2b006d990 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fb2b4036000 tx=0x7fb2b4030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.839+0000 7fb2c881e640 1 --2- 172.21.15.83:0/3921436766 >> [v2:172.21.15.83:6814/1822925240,v1:172.21.15.83:6817/1822925240] conn(0x7fb2b0071430 0x7fb2b0073850 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.839+0000 7fb2c0ff9640 1 -- 172.21.15.83:0/3921436766 <== osd.1 v2:172.21.15.83:6814/1822925240 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb2b8021ec0 con 0x7fb2b0071430 2024-03-06T07:32:48.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.852+0000 7fb2ca281640 1 -- 172.21.15.83:0/3921436766 --> [v2:172.21.15.83:6814/1822925240,v1:172.21.15.83:6817/1822925240] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb2c40b11b0 con 0x7fb2b0071430 2024-03-06T07:32:48.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.852+0000 7fb2c0ff9640 1 -- 172.21.15.83:0/3921436766 <== osd.1 v2:172.21.15.83:6814/1822925240 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fb2c40b11b0 con 0x7fb2b0071430 2024-03-06T07:32:48.855 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:32:48.855 INFO:teuthology.orchestra.run.smithi083.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:32:48.855 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:32:48.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.853+0000 7fb2ca281640 1 -- 172.21.15.83:0/3921436766 >> [v2:172.21.15.83:6814/1822925240,v1:172.21.15.83:6817/1822925240] conn(0x7fb2b0071430 msgr2=0x7fb2b0073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.853+0000 7fb2ca281640 1 --2- 172.21.15.83:0/3921436766 >> [v2:172.21.15.83:6814/1822925240,v1:172.21.15.83:6817/1822925240] conn(0x7fb2b0071430 0x7fb2b0073850 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.853+0000 7fb2ca281640 1 -- 172.21.15.83:0/3921436766 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb2b006b4d0 msgr2=0x7fb2b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.853+0000 7fb2ca281640 1 --2- 172.21.15.83:0/3921436766 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fb2b006b4d0 0x7fb2b006d990 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fb2b4036000 tx=0x7fb2b4030000 comp rx=0 tx=0).stop 2024-03-06T07:32:48.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.853+0000 7fb2ca281640 1 -- 172.21.15.83:0/3921436766 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2c41ef850 msgr2=0x7fb2c415b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.853+0000 7fb2ca281640 1 --2- 172.21.15.83:0/3921436766 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2c41ef850 0x7fb2c415b220 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8001b60 tx=0x7fb2b800ecd0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.853+0000 7fb2c881e640 1 -- 172.21.15.83:0/3921436766 reap_dead start 2024-03-06T07:32:48.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.853+0000 7fb2ca281640 1 -- 172.21.15.83:0/3921436766 shutdown_connections 2024-03-06T07:32:48.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.853+0000 7fb2ca281640 1 -- 172.21.15.83:0/3921436766 >> 172.21.15.83:0/3921436766 conn(0x7fb2c40b1650 msgr2=0x7fb2c40716b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:48.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.853+0000 7fb2ca281640 1 -- 172.21.15.83:0/3921436766 shutdown_connections 2024-03-06T07:32:48.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.853+0000 7fb2ca281640 1 -- 172.21.15.83:0/3921436766 wait complete. 2024-03-06T07:32:48.865 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-06T07:32:48.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.964+0000 7fc1f328d640 1 Processor -- start 2024-03-06T07:32:48.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.965+0000 7fc1f328d640 1 -- start start 2024-03-06T07:32:48.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.965+0000 7fc1f328d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc1ec162970 0x7fc1ec0d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.965+0000 7fc1f328d640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fc1ec161cf0 0x7fc1ec162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.965+0000 7fc1f328d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1ec0d22c0 con 0x7fc1ec0b4460 2024-03-06T07:32:48.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.965+0000 7fc1f328d640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1ec161b00 con 0x7fc1ec161cf0 2024-03-06T07:32:48.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.965+0000 7fc1f328d640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ec0cecc0 con 0x7fc1ec162970 2024-03-06T07:32:48.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.966+0000 7fc1f182a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fc1ec161cf0 0x7fc1ec162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.966+0000 7fc1f0828640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc1ec162970 0x7fc1ec0d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.966+0000 7fc1f182a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fc1ec161cf0 0x7fc1ec162180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.83:51858/0 (socket says 172.21.15.83:51858) 2024-03-06T07:32:48.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.966+0000 7fc1f0828640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc1ec162970 0x7fc1ec0d4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:40330/0 (socket says 172.21.15.83:40330) 2024-03-06T07:32:48.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.966+0000 7fc1f182a640 1 -- 172.21.15.83:0/2334315095 learned_addr learned my addr 172.21.15.83:0/2334315095 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:48.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.966+0000 7fc1f1029640 1 --1- 172.21.15.83:0/2334315095 >> v1:172.21.15.183:6789/0 conn(0x7fc1ec0b4460 0x7fc1ec0b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:39912/0 (socket says 172.21.15.83:39912) 2024-03-06T07:32:48.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.966+0000 7fc1f182a640 1 -- 172.21.15.83:0/2334315095 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc1ec162970 msgr2=0x7fc1ec0d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:48.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.966+0000 7fc1f182a640 1 --2- 172.21.15.83:0/2334315095 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc1ec162970 0x7fc1ec0d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.966+0000 7fc1f182a640 1 -- 172.21.15.83:0/2334315095 >> v1:172.21.15.183:6789/0 conn(0x7fc1ec0b4460 legacy=0x7fc1ec0b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.966+0000 7fc1f182a640 1 -- 172.21.15.83:0/2334315095 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1ec0d25a0 con 0x7fc1ec161cf0 2024-03-06T07:32:48.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.966+0000 7fc1f0828640 1 --2- 172.21.15.83:0/2334315095 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc1ec162970 0x7fc1ec0d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:48.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.967+0000 7fc1f182a640 1 --2- 172.21.15.83:0/2334315095 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc1ec161cf0 0x7fc1ec162180 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fc1e4009e50 tx=0x7fc1e402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8782f954317464d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.967+0000 7fc1e3fff640 1 -- 172.21.15.83:0/2334315095 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e402cb00 con 0x7fc1ec161cf0 2024-03-06T07:32:48.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.967+0000 7fc1e3fff640 1 -- 172.21.15.83:0/2334315095 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1e402cca0 con 0x7fc1ec161cf0 2024-03-06T07:32:48.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.967+0000 7fc1e3fff640 1 -- 172.21.15.83:0/2334315095 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e4034610 con 0x7fc1ec161cf0 2024-03-06T07:32:48.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.968+0000 7fc1f328d640 1 -- 172.21.15.83:0/2334315095 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc1ec161cf0 msgr2=0x7fc1ec162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.968+0000 7fc1f328d640 1 --2- 172.21.15.83:0/2334315095 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc1ec161cf0 0x7fc1ec162180 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fc1e4009e50 tx=0x7fc1e402bf60 comp rx=0 tx=0).stop 2024-03-06T07:32:48.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.968+0000 7fc1f328d640 1 -- 172.21.15.83:0/2334315095 shutdown_connections 2024-03-06T07:32:48.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.968+0000 7fc1f328d640 1 --2- 172.21.15.83:0/2334315095 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc1ec161cf0 0x7fc1ec162180 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.968+0000 7fc1f328d640 1 --2- 172.21.15.83:0/2334315095 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc1ec162970 0x7fc1ec0d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.968+0000 7fc1f328d640 1 -- 172.21.15.83:0/2334315095 >> 172.21.15.83:0/2334315095 conn(0x7fc1ec0d3ce0 msgr2=0x7fc1ec0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:48.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.968+0000 7fc1f328d640 1 -- 172.21.15.83:0/2334315095 shutdown_connections 2024-03-06T07:32:48.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.968+0000 7fc1f328d640 1 -- 172.21.15.83:0/2334315095 wait complete. 2024-03-06T07:32:48.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.969+0000 7fc1f328d640 1 Processor -- start 2024-03-06T07:32:48.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.969+0000 7fc1f328d640 1 -- start start 2024-03-06T07:32:48.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.970+0000 7fc1f328d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc1ec0b4460 0x7fc1ec15ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.970+0000 7fc1f328d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc1ec161cf0 0x7fc1ec0cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.970+0000 7fc1f328d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc1ec162970 0x7fc1ec0cb940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.970+0000 7fc1f328d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ec0ce950 con 0x7fc1ec162970 2024-03-06T07:32:48.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.970+0000 7fc1f328d640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ec0d2b60 con 0x7fc1ec0b4460 2024-03-06T07:32:48.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.970+0000 7fc1f328d640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ec0b4840 con 0x7fc1ec161cf0 2024-03-06T07:32:48.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.970+0000 7fc1f182a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc1ec162970 0x7fc1ec0cb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.970+0000 7fc1f182a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc1ec162970 0x7fc1ec0cb940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36320/0 (socket says 172.21.15.83:36320) 2024-03-06T07:32:48.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.970+0000 7fc1f182a640 1 -- 172.21.15.83:0/4045279570 learned_addr learned my addr 172.21.15.83:0/4045279570 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:48.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.970+0000 7fc1f0828640 1 --2- 172.21.15.83:0/4045279570 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc1ec161cf0 0x7fc1ec0cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.970+0000 7fc1f1029640 1 --2- 172.21.15.83:0/4045279570 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc1ec0b4460 0x7fc1ec15ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.970+0000 7fc1f182a640 1 -- 172.21.15.83:0/4045279570 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc1ec161cf0 msgr2=0x7fc1ec0cd6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.970+0000 7fc1f182a640 1 --2- 172.21.15.83:0/4045279570 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc1ec161cf0 0x7fc1ec0cd6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.970+0000 7fc1f182a640 1 -- 172.21.15.83:0/4045279570 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc1ec0b4460 msgr2=0x7fc1ec15ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.970+0000 7fc1f182a640 1 --2- 172.21.15.83:0/4045279570 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc1ec0b4460 0x7fc1ec15ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.970+0000 7fc1f182a640 1 -- 172.21.15.83:0/4045279570 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1ec15f0f0 con 0x7fc1ec162970 2024-03-06T07:32:48.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.970+0000 7fc1f182a640 1 --2- 172.21.15.83:0/4045279570 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc1ec162970 0x7fc1ec0cb940 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7fc1e4036040 tx=0x7fc1e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.971+0000 7fc1e1ffb640 1 -- 172.21.15.83:0/4045279570 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e4043020 con 0x7fc1ec162970 2024-03-06T07:32:48.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.971+0000 7fc1e1ffb640 1 -- 172.21.15.83:0/4045279570 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1e402ce50 con 0x7fc1ec162970 2024-03-06T07:32:48.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.971+0000 7fc1f328d640 1 -- 172.21.15.83:0/4045279570 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1ec0cc600 con 0x7fc1ec162970 2024-03-06T07:32:48.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.971+0000 7fc1e1ffb640 1 -- 172.21.15.83:0/4045279570 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e4009510 con 0x7fc1ec162970 2024-03-06T07:32:48.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.971+0000 7fc1f328d640 1 -- 172.21.15.83:0/4045279570 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1ec0ccbc0 con 0x7fc1ec162970 2024-03-06T07:32:48.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.972+0000 7fc1f328d640 1 -- 172.21.15.83:0/4045279570 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1ec04f330 con 0x7fc1ec162970 2024-03-06T07:32:48.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.972+0000 7fc1e1ffb640 1 -- 172.21.15.83:0/4045279570 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc1e4046020 con 0x7fc1ec162970 2024-03-06T07:32:48.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.973+0000 7fc1e1ffb640 1 --2- 172.21.15.83:0/4045279570 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc1d006b4d0 0x7fc1d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.973+0000 7fc1e1ffb640 1 -- 172.21.15.83:0/4045279570 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fc1e40ac330 con 0x7fc1ec162970 2024-03-06T07:32:48.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.973+0000 7fc1f1029640 1 --2- 172.21.15.83:0/4045279570 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc1d006b4d0 0x7fc1d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.973+0000 7fc1f1029640 1 --2- 172.21.15.83:0/4045279570 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc1d006b4d0 0x7fc1d006d990 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fc1e8007fd0 tx=0x7fc1e8007e80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:48.977+0000 7fc1e1ffb640 1 -- 172.21.15.83:0/4045279570 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc1e404d030 con 0x7fc1ec162970 2024-03-06T07:32:49.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.151+0000 7fc1f328d640 1 -- 172.21.15.83:0/4045279570 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7fc1ec06b180 con 0x7fc1d006b4d0 2024-03-06T07:32:49.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.151+0000 7fc1e1ffb640 1 -- 172.21.15.83:0/4045279570 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc1ec06b180 con 0x7fc1d006b4d0 2024-03-06T07:32:49.153 INFO:teuthology.orchestra.run.smithi083.stderr:instructed osd(s) 1 to deep-scrub 2024-03-06T07:32:49.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.153+0000 7fc1f328d640 1 -- 172.21.15.83:0/4045279570 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc1d006b4d0 msgr2=0x7fc1d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.153+0000 7fc1f328d640 1 --2- 172.21.15.83:0/4045279570 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc1d006b4d0 0x7fc1d006d990 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fc1e8007fd0 tx=0x7fc1e8007e80 comp rx=0 tx=0).stop 2024-03-06T07:32:49.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.153+0000 7fc1f328d640 1 -- 172.21.15.83:0/4045279570 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc1ec162970 msgr2=0x7fc1ec0cb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.153+0000 7fc1f328d640 1 --2- 172.21.15.83:0/4045279570 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc1ec162970 0x7fc1ec0cb940 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7fc1e4036040 tx=0x7fc1e4004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:49.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.153+0000 7fc1f328d640 1 -- 172.21.15.83:0/4045279570 shutdown_connections 2024-03-06T07:32:49.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.153+0000 7fc1f328d640 1 --2- 172.21.15.83:0/4045279570 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fc1d006b4d0 0x7fc1d006d990 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.153+0000 7fc1f328d640 1 --2- 172.21.15.83:0/4045279570 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc1ec162970 0x7fc1ec0cb940 unknown :-1 s=CLOSED pgs=2044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.153+0000 7fc1f328d640 1 --2- 172.21.15.83:0/4045279570 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc1ec161cf0 0x7fc1ec0cd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.153+0000 7fc1f328d640 1 --2- 172.21.15.83:0/4045279570 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc1ec0b4460 0x7fc1ec15ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.153+0000 7fc1f328d640 1 -- 172.21.15.83:0/4045279570 >> 172.21.15.83:0/4045279570 conn(0x7fc1ec0d3ce0 msgr2=0x7fc1ec06caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:49.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.153+0000 7fc1f328d640 1 -- 172.21.15.83:0/4045279570 shutdown_connections 2024-03-06T07:32:49.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.153+0000 7fc1f328d640 1 -- 172.21.15.83:0/4045279570 wait complete. 2024-03-06T07:32:49.166 INFO:tasks.ceph:Scrubbing osd.2 2024-03-06T07:32:49.166 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:32:49.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.268+0000 7efd83c6f640 1 Processor -- start 2024-03-06T07:32:49.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.268+0000 7efd83c6f640 1 -- start start 2024-03-06T07:32:49.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.269+0000 7efd83c6f640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7efd7c15dff0 0x7efd7c0cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.269+0000 7efd83c6f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7efd7c0d1250 0x7efd7c0b3e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.269+0000 7efd83c6f640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd7c0cf8b0 con 0x7efd7c0d06b0 2024-03-06T07:32:49.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.269+0000 7efd83c6f640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd7c0cfb30 con 0x7efd7c0d1250 2024-03-06T07:32:49.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.269+0000 7efd83c6f640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd7c0b3320 con 0x7efd7c15dff0 2024-03-06T07:32:49.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.269+0000 7efd8220c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7efd7c0d1250 0x7efd7c0b3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.269+0000 7efd8220c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7efd7c0d1250 0x7efd7c0b3e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36336/0 (socket says 172.21.15.83:36336) 2024-03-06T07:32:49.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.269+0000 7efd8220c640 1 -- 172.21.15.83:0/3224054013 learned_addr learned my addr 172.21.15.83:0/3224054013 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:49.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.269+0000 7efd8220c640 1 -- 172.21.15.83:0/3224054013 >> v2:172.21.15.183:3300/0 conn(0x7efd7c15dff0 msgr2=0x7efd7c0cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:49.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.269+0000 7efd8220c640 1 --2- 172.21.15.83:0/3224054013 >> v2:172.21.15.183:3300/0 conn(0x7efd7c15dff0 0x7efd7c0cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.269+0000 7efd8220c640 1 -- 172.21.15.83:0/3224054013 >> v1:172.21.15.183:6789/0 conn(0x7efd7c0d06b0 legacy=0x7efd7c0d0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:49.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.269+0000 7efd8220c640 1 -- 172.21.15.83:0/3224054013 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd7c0b3600 con 0x7efd7c0d1250 2024-03-06T07:32:49.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.270+0000 7efd8220c640 1 --2- 172.21.15.83:0/3224054013 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efd7c0d1250 0x7efd7c0b3e00 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7efd7c08e3a0 tx=0x7efd7402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0981369fde830af server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:49.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.270+0000 7efd80a09640 1 -- 172.21.15.83:0/3224054013 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd7402cb20 con 0x7efd7c0d1250 2024-03-06T07:32:49.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.270+0000 7efd80a09640 1 -- 172.21.15.83:0/3224054013 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd7402ccc0 con 0x7efd7c0d1250 2024-03-06T07:32:49.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.270+0000 7efd80a09640 1 -- 172.21.15.83:0/3224054013 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd74034610 con 0x7efd7c0d1250 2024-03-06T07:32:49.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.270+0000 7efd83c6f640 1 -- 172.21.15.83:0/3224054013 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efd7c0d1250 msgr2=0x7efd7c0b3e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.270+0000 7efd83c6f640 1 --2- 172.21.15.83:0/3224054013 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efd7c0d1250 0x7efd7c0b3e00 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7efd7c08e3a0 tx=0x7efd7402bf80 comp rx=0 tx=0).stop 2024-03-06T07:32:49.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.271+0000 7efd83c6f640 1 -- 172.21.15.83:0/3224054013 shutdown_connections 2024-03-06T07:32:49.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.271+0000 7efd83c6f640 1 --2- 172.21.15.83:0/3224054013 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efd7c0d1250 0x7efd7c0b3e00 unknown :-1 s=CLOSED pgs=2045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.271+0000 7efd83c6f640 1 --2- 172.21.15.83:0/3224054013 >> v2:172.21.15.183:3300/0 conn(0x7efd7c15dff0 0x7efd7c0cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.271+0000 7efd83c6f640 1 -- 172.21.15.83:0/3224054013 >> 172.21.15.83:0/3224054013 conn(0x7efd7c0d3d50 msgr2=0x7efd7c0d4160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:49.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.271+0000 7efd83c6f640 1 -- 172.21.15.83:0/3224054013 shutdown_connections 2024-03-06T07:32:49.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.271+0000 7efd83c6f640 1 -- 172.21.15.83:0/3224054013 wait complete. 2024-03-06T07:32:49.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.272+0000 7efd83c6f640 1 Processor -- start 2024-03-06T07:32:49.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.272+0000 7efd83c6f640 1 -- start start 2024-03-06T07:32:49.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.272+0000 7efd83c6f640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efd7c0d06b0 0x7efd7c153c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.272+0000 7efd83c6f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efd7c0d1250 0x7efd7c152d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.272+0000 7efd83c6f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efd7c15dff0 0x7efd7c0b8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.272+0000 7efd83c6f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd7c15e550 con 0x7efd7c15dff0 2024-03-06T07:32:49.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.272+0000 7efd83c6f640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd7c15e3d0 con 0x7efd7c0d06b0 2024-03-06T07:32:49.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.272+0000 7efd83c6f640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd7c0b4340 con 0x7efd7c0d1250 2024-03-06T07:32:49.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.273+0000 7efd8220c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efd7c15dff0 0x7efd7c0b8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.273+0000 7efd8220c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efd7c15dff0 0x7efd7c0b8ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36346/0 (socket says 172.21.15.83:36346) 2024-03-06T07:32:49.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.273+0000 7efd8220c640 1 -- 172.21.15.83:0/1575013715 learned_addr learned my addr 172.21.15.83:0/1575013715 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:49.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.273+0000 7efd81a0b640 1 --2- 172.21.15.83:0/1575013715 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efd7c0d06b0 0x7efd7c153c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.273+0000 7efd8120a640 1 --2- 172.21.15.83:0/1575013715 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efd7c0d1250 0x7efd7c152d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.273+0000 7efd8220c640 1 -- 172.21.15.83:0/1575013715 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efd7c0d1250 msgr2=0x7efd7c152d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.273+0000 7efd8220c640 1 --2- 172.21.15.83:0/1575013715 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efd7c0d1250 0x7efd7c152d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.273+0000 7efd8220c640 1 -- 172.21.15.83:0/1575013715 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efd7c0d06b0 msgr2=0x7efd7c153c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.273+0000 7efd8220c640 1 --2- 172.21.15.83:0/1575013715 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efd7c0d06b0 0x7efd7c153c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.273+0000 7efd8220c640 1 -- 172.21.15.83:0/1575013715 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd7c0b7840 con 0x7efd7c15dff0 2024-03-06T07:32:49.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.273+0000 7efd81a0b640 1 --2- 172.21.15.83:0/1575013715 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efd7c0d06b0 0x7efd7c153c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:49.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.273+0000 7efd8220c640 1 --2- 172.21.15.83:0/1575013715 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efd7c15dff0 0x7efd7c0b8ca0 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7efd74036040 tx=0x7efd74004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:49.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.273+0000 7efd727fc640 1 -- 172.21.15.83:0/1575013715 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd74043020 con 0x7efd7c15dff0 2024-03-06T07:32:49.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.273+0000 7efd727fc640 1 -- 172.21.15.83:0/1575013715 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd7403e450 con 0x7efd7c15dff0 2024-03-06T07:32:49.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.273+0000 7efd83c6f640 1 -- 172.21.15.83:0/1575013715 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd7c1532d0 con 0x7efd7c15dff0 2024-03-06T07:32:49.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.273+0000 7efd727fc640 1 -- 172.21.15.83:0/1575013715 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd7403e820 con 0x7efd7c15dff0 2024-03-06T07:32:49.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.273+0000 7efd83c6f640 1 -- 172.21.15.83:0/1575013715 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd7c0b7dc0 con 0x7efd7c15dff0 2024-03-06T07:32:49.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.274+0000 7efd83c6f640 1 -- 172.21.15.83:0/1575013715 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7efd7c04f330 con 0x7efd7c15dff0 2024-03-06T07:32:49.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.275+0000 7efd727fc640 1 -- 172.21.15.83:0/1575013715 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efd74046020 con 0x7efd7c15dff0 2024-03-06T07:32:49.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.275+0000 7efd727fc640 1 --2- 172.21.15.83:0/1575013715 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efd6006b4d0 0x7efd6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.275+0000 7efd727fc640 1 -- 172.21.15.83:0/1575013715 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5029+0+0 (secure 0 0 0) 0x7efd740ac960 con 0x7efd7c15dff0 2024-03-06T07:32:49.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.275+0000 7efd81a0b640 1 --2- 172.21.15.83:0/1575013715 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efd6006b4d0 0x7efd6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.275+0000 7efd727fc640 1 --2- 172.21.15.83:0/1575013715 >> [v2:172.21.15.83:6800/2591342133,v1:172.21.15.83:6801/2591342133] conn(0x7efd60071430 0x7efd60073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.275+0000 7efd727fc640 1 -- 172.21.15.83:0/1575013715 --> [v2:172.21.15.83:6800/2591342133,v1:172.21.15.83:6801/2591342133] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efd7403ae50 con 0x7efd60071430 2024-03-06T07:32:49.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.275+0000 7efd727fc640 1 -- 172.21.15.83:0/1575013715 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_get_version_reply(handle=1 version=135) v2 ==== 24+0+0 (secure 0 0 0) 0x7efd740b12c0 con 0x7efd7c15dff0 2024-03-06T07:32:49.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.275+0000 7efd8120a640 1 --2- 172.21.15.83:0/1575013715 >> [v2:172.21.15.83:6800/2591342133,v1:172.21.15.83:6801/2591342133] conn(0x7efd60071430 0x7efd60073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.275+0000 7efd81a0b640 1 --2- 172.21.15.83:0/1575013715 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efd6006b4d0 0x7efd6006d990 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7efd7800a870 tx=0x7efd78008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:49.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.276+0000 7efd8120a640 1 --2- 172.21.15.83:0/1575013715 >> [v2:172.21.15.83:6800/2591342133,v1:172.21.15.83:6801/2591342133] conn(0x7efd60071430 0x7efd60073850 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:49.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.276+0000 7efd727fc640 1 -- 172.21.15.83:0/1575013715 <== osd.2 v2:172.21.15.83:6800/2591342133 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7efd7403ae50 con 0x7efd60071430 2024-03-06T07:32:49.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.289+0000 7efd83c6f640 1 -- 172.21.15.83:0/1575013715 --> [v2:172.21.15.83:6800/2591342133,v1:172.21.15.83:6801/2591342133] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7efd7c0ce820 con 0x7efd60071430 2024-03-06T07:32:49.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.289+0000 7efd727fc640 1 -- 172.21.15.83:0/1575013715 <== osd.2 v2:172.21.15.83:6800/2591342133 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7efd7c0ce820 con 0x7efd60071430 2024-03-06T07:32:49.291 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:32:49.292 INFO:teuthology.orchestra.run.smithi083.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:32:49.292 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:32:49.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.290+0000 7efd83c6f640 1 -- 172.21.15.83:0/1575013715 >> [v2:172.21.15.83:6800/2591342133,v1:172.21.15.83:6801/2591342133] conn(0x7efd60071430 msgr2=0x7efd60073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.290+0000 7efd83c6f640 1 --2- 172.21.15.83:0/1575013715 >> [v2:172.21.15.83:6800/2591342133,v1:172.21.15.83:6801/2591342133] conn(0x7efd60071430 0x7efd60073850 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.290+0000 7efd83c6f640 1 -- 172.21.15.83:0/1575013715 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efd6006b4d0 msgr2=0x7efd6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.290+0000 7efd83c6f640 1 --2- 172.21.15.83:0/1575013715 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7efd6006b4d0 0x7efd6006d990 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7efd7800a870 tx=0x7efd78008040 comp rx=0 tx=0).stop 2024-03-06T07:32:49.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.290+0000 7efd83c6f640 1 -- 172.21.15.83:0/1575013715 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efd7c15dff0 msgr2=0x7efd7c0b8ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.290+0000 7efd83c6f640 1 --2- 172.21.15.83:0/1575013715 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efd7c15dff0 0x7efd7c0b8ca0 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7efd74036040 tx=0x7efd74004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:49.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.290+0000 7efd8220c640 1 -- 172.21.15.83:0/1575013715 reap_dead start 2024-03-06T07:32:49.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.290+0000 7efd83c6f640 1 -- 172.21.15.83:0/1575013715 shutdown_connections 2024-03-06T07:32:49.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.290+0000 7efd83c6f640 1 -- 172.21.15.83:0/1575013715 >> 172.21.15.83:0/1575013715 conn(0x7efd7c0d3d50 msgr2=0x7efd7c0b5560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:49.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.290+0000 7efd83c6f640 1 -- 172.21.15.83:0/1575013715 shutdown_connections 2024-03-06T07:32:49.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.290+0000 7efd83c6f640 1 -- 172.21.15.83:0/1575013715 wait complete. 2024-03-06T07:32:49.302 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-06T07:32:49.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.402+0000 7f3ea784e640 1 Processor -- start 2024-03-06T07:32:49.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.402+0000 7f3ea784e640 1 -- start start 2024-03-06T07:32:49.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.403+0000 7f3ea784e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3ea015bed0 0x7f3ea00b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.403+0000 7f3ea784e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ea00b2d80 con 0x7f3ea00d21b0 2024-03-06T07:32:49.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.403+0000 7f3ea784e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ea00ae9c0 con 0x7f3ea00d04e0 2024-03-06T07:32:49.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.403+0000 7f3ea784e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea00aeca0 con 0x7f3ea015bed0 2024-03-06T07:32:49.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.403+0000 7f3ea5deb640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f3ea00d21b0 0x7f3ea00af780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:53846/0 (socket says 172.21.15.83:53846) 2024-03-06T07:32:49.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.403+0000 7f3ea5deb640 1 -- 172.21.15.83:0/3452141552 learned_addr learned my addr 172.21.15.83:0/3452141552 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:32:49.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.403+0000 7f3ea55ea640 1 --2- 172.21.15.83:0/3452141552 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3ea015bed0 0x7f3ea00b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.403+0000 7f3e97fff640 1 -- 172.21.15.83:0/3452141552 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3965221405 0 0) 0x7f3ea00b2d80 con 0x7f3ea00d21b0 2024-03-06T07:32:49.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.403+0000 7f3e97fff640 1 -- 172.21.15.83:0/3452141552 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e8c003520 con 0x7f3ea00d21b0 2024-03-06T07:32:49.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.403+0000 7f3e97fff640 1 -- 172.21.15.83:0/3452141552 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1911294346 0 0) 0x7f3e8c003520 con 0x7f3ea00d21b0 2024-03-06T07:32:49.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.403+0000 7f3e97fff640 1 -- 172.21.15.83:0/3452141552 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3ea015bed0 msgr2=0x7f3ea00b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:49.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.403+0000 7f3e97fff640 1 --2- 172.21.15.83:0/3452141552 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3ea015bed0 0x7f3ea00b2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.403+0000 7f3e97fff640 1 -- 172.21.15.83:0/3452141552 >> v1:172.21.15.183:6789/0 conn(0x7f3ea00d04e0 legacy=0x7f3ea00d1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.403+0000 7f3e97fff640 1 -- 172.21.15.83:0/3452141552 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea00aef80 con 0x7f3ea00d21b0 2024-03-06T07:32:49.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.403+0000 7f3e97fff640 1 -- 172.21.15.83:0/3452141552 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f3e980019f0 con 0x7f3ea00d21b0 2024-03-06T07:32:49.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.403+0000 7f3e97fff640 1 -- 172.21.15.83:0/3452141552 >> v1:172.21.15.83:6789/0 conn(0x7f3ea00d21b0 legacy=0x7f3ea00af780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.403+0000 7f3e97fff640 1 --2- 172.21.15.83:0/3452141552 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3e8c0039e0 0x7f3e8c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.403+0000 7f3e97fff640 1 --2- 172.21.15.83:0/3452141552 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e8c004800 0x7f3e8c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.403+0000 7f3e97fff640 1 --2- 172.21.15.83:0/3452141552 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3ea015bed0 0x7f3e8c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.403+0000 7f3e97fff640 1 -- 172.21.15.83:0/3452141552 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea00aeca0 con 0x7f3e8c004800 2024-03-06T07:32:49.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.403+0000 7f3e97fff640 1 -- 172.21.15.83:0/3452141552 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8c00baf0 con 0x7f3e8c0039e0 2024-03-06T07:32:49.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.404+0000 7f3e97fff640 1 -- 172.21.15.83:0/3452141552 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8c00bcf0 con 0x7f3ea015bed0 2024-03-06T07:32:49.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.404+0000 7f3ea4de9640 1 --2- 172.21.15.83:0/3452141552 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e8c004800 0x7f3e8c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.404+0000 7f3ea55ea640 1 --2- 172.21.15.83:0/3452141552 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3e8c0039e0 0x7f3e8c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.404+0000 7f3ea5deb640 1 --2- 172.21.15.83:0/3452141552 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3ea015bed0 0x7f3e8c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.404+0000 7f3ea4de9640 1 -- 172.21.15.83:0/3452141552 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3ea015bed0 msgr2=0x7f3e8c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:49.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.404+0000 7f3ea4de9640 1 --2- 172.21.15.83:0/3452141552 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3ea015bed0 0x7f3e8c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.404+0000 7f3ea4de9640 1 -- 172.21.15.83:0/3452141552 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3e8c0039e0 msgr2=0x7f3e8c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:49.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.404+0000 7f3ea4de9640 1 --2- 172.21.15.83:0/3452141552 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3e8c0039e0 0x7f3e8c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.404+0000 7f3ea4de9640 1 -- 172.21.15.83:0/3452141552 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e8c003520 con 0x7f3e8c004800 2024-03-06T07:32:49.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.405+0000 7f3ea4de9640 1 --2- 172.21.15.83:0/3452141552 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e8c004800 0x7f3e8c004c80 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f3e90001750 tx=0x7f3e8c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f85f154c73661e84 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:49.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.405+0000 7f3e97fff640 1 -- 172.21.15.83:0/3452141552 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e90001370 con 0x7f3e8c004800 2024-03-06T07:32:49.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.405+0000 7f3e97fff640 1 -- 172.21.15.83:0/3452141552 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e900052d0 con 0x7f3e8c004800 2024-03-06T07:32:49.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.405+0000 7f3e97fff640 1 -- 172.21.15.83:0/3452141552 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e90003830 con 0x7f3e8c004800 2024-03-06T07:32:49.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.405+0000 7f3ea784e640 1 -- 172.21.15.83:0/3452141552 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e8c004800 msgr2=0x7f3e8c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.405+0000 7f3ea784e640 1 --2- 172.21.15.83:0/3452141552 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e8c004800 0x7f3e8c004c80 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f3e90001750 tx=0x7f3e8c005830 comp rx=0 tx=0).stop 2024-03-06T07:32:49.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.405+0000 7f3ea5deb640 1 -- 172.21.15.83:0/3452141552 reap_dead start 2024-03-06T07:32:49.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.405+0000 7f3ea784e640 1 -- 172.21.15.83:0/3452141552 shutdown_connections 2024-03-06T07:32:49.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.405+0000 7f3ea784e640 1 -- 172.21.15.83:0/3452141552 >> 172.21.15.83:0/3452141552 conn(0x7f3ea009a9f0 msgr2=0x7f3ea009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:49.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.405+0000 7f3ea784e640 1 -- 172.21.15.83:0/3452141552 shutdown_connections 2024-03-06T07:32:49.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.405+0000 7f3ea784e640 1 -- 172.21.15.83:0/3452141552 wait complete. 2024-03-06T07:32:49.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.407+0000 7f3ea784e640 1 Processor -- start 2024-03-06T07:32:49.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.407+0000 7f3ea784e640 1 -- start start 2024-03-06T07:32:49.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.407+0000 7f3ea784e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3e8c004800 0x7f3ea014e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.407+0000 7f3ea784e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3ea01f2c70 0x7f3ea00b69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.407+0000 7f3ea784e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ea00b6ef0 0x7f3ea00cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.407+0000 7f3ea784e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea015c2b0 con 0x7f3ea00b6ef0 2024-03-06T07:32:49.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.407+0000 7f3ea784e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea00b2480 con 0x7f3ea01f2c70 2024-03-06T07:32:49.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.407+0000 7f3ea784e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea014e6f0 con 0x7f3e8c004800 2024-03-06T07:32:49.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.407+0000 7f3ea5deb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ea00b6ef0 0x7f3ea00cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.407+0000 7f3ea5deb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ea00b6ef0 0x7f3ea00cc3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36362/0 (socket says 172.21.15.83:36362) 2024-03-06T07:32:49.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.407+0000 7f3ea5deb640 1 -- 172.21.15.83:0/751970409 learned_addr learned my addr 172.21.15.83:0/751970409 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:49.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.408+0000 7f3ea55ea640 1 --2- 172.21.15.83:0/751970409 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3e8c004800 0x7f3ea014e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.408+0000 7f3ea4de9640 1 --2- 172.21.15.83:0/751970409 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3ea01f2c70 0x7f3ea00b69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.408+0000 7f3ea5deb640 1 -- 172.21.15.83:0/751970409 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3e8c004800 msgr2=0x7f3ea014e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.408+0000 7f3ea5deb640 1 --2- 172.21.15.83:0/751970409 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3e8c004800 0x7f3ea014e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.408+0000 7f3ea5deb640 1 -- 172.21.15.83:0/751970409 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3ea01f2c70 msgr2=0x7f3ea00b69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.408+0000 7f3ea5deb640 1 --2- 172.21.15.83:0/751970409 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3ea01f2c70 0x7f3ea00b69b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.408+0000 7f3ea5deb640 1 -- 172.21.15.83:0/751970409 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea00cd460 con 0x7f3ea00b6ef0 2024-03-06T07:32:49.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.408+0000 7f3ea5deb640 1 --2- 172.21.15.83:0/751970409 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ea00b6ef0 0x7f3ea00cc3c0 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7f3ea0160b60 tx=0x7f3e9800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:49.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.408+0000 7f3e95ffb640 1 -- 172.21.15.83:0/751970409 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e9800ebb0 con 0x7f3ea00b6ef0 2024-03-06T07:32:49.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.408+0000 7f3e95ffb640 1 -- 172.21.15.83:0/751970409 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e98018070 con 0x7f3ea00b6ef0 2024-03-06T07:32:49.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.408+0000 7f3ea784e640 1 -- 172.21.15.83:0/751970409 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ea00cd180 con 0x7f3ea00b6ef0 2024-03-06T07:32:49.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.408+0000 7f3e95ffb640 1 -- 172.21.15.83:0/751970409 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e98013420 con 0x7f3ea00b6ef0 2024-03-06T07:32:49.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.408+0000 7f3ea784e640 1 -- 172.21.15.83:0/751970409 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ea00b4a40 con 0x7f3ea00b6ef0 2024-03-06T07:32:49.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.409+0000 7f3ea784e640 1 -- 172.21.15.83:0/751970409 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ea004f330 con 0x7f3ea00b6ef0 2024-03-06T07:32:49.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.410+0000 7f3e95ffb640 1 -- 172.21.15.83:0/751970409 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3e980135c0 con 0x7f3ea00b6ef0 2024-03-06T07:32:49.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.410+0000 7f3e95ffb640 1 --2- 172.21.15.83:0/751970409 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3e8406b4d0 0x7f3e8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.410+0000 7f3ea55ea640 1 --2- 172.21.15.83:0/751970409 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3e8406b4d0 0x7f3e8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.410+0000 7f3e95ffb640 1 -- 172.21.15.83:0/751970409 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f3e9808a230 con 0x7f3ea00b6ef0 2024-03-06T07:32:49.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.411+0000 7f3ea55ea640 1 --2- 172.21.15.83:0/751970409 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3e8406b4d0 0x7f3e8406d990 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f3e9c009bc0 tx=0x7f3e9c009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:49.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.415+0000 7f3e95ffb640 1 -- 172.21.15.83:0/751970409 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3e98059410 con 0x7f3ea00b6ef0 2024-03-06T07:32:49.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.585+0000 7f3ea784e640 1 -- 172.21.15.83:0/751970409 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f3ea00d28b0 con 0x7f3e8406b4d0 2024-03-06T07:32:49.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.586+0000 7f3e95ffb640 1 -- 172.21.15.83:0/751970409 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3ea00d28b0 con 0x7f3e8406b4d0 2024-03-06T07:32:49.588 INFO:teuthology.orchestra.run.smithi083.stderr:instructed osd(s) 2 to deep-scrub 2024-03-06T07:32:49.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.588+0000 7f3ea784e640 1 -- 172.21.15.83:0/751970409 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3e8406b4d0 msgr2=0x7f3e8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.588+0000 7f3ea784e640 1 --2- 172.21.15.83:0/751970409 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3e8406b4d0 0x7f3e8406d990 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f3e9c009bc0 tx=0x7f3e9c009190 comp rx=0 tx=0).stop 2024-03-06T07:32:49.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.588+0000 7f3ea784e640 1 -- 172.21.15.83:0/751970409 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ea00b6ef0 msgr2=0x7f3ea00cc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.588+0000 7f3ea784e640 1 --2- 172.21.15.83:0/751970409 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ea00b6ef0 0x7f3ea00cc3c0 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7f3ea0160b60 tx=0x7f3e9800e530 comp rx=0 tx=0).stop 2024-03-06T07:32:49.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.588+0000 7f3ea784e640 1 -- 172.21.15.83:0/751970409 shutdown_connections 2024-03-06T07:32:49.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.588+0000 7f3ea784e640 1 --2- 172.21.15.83:0/751970409 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3e8406b4d0 0x7f3e8406d990 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.588+0000 7f3ea784e640 1 --2- 172.21.15.83:0/751970409 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ea00b6ef0 0x7f3ea00cc3c0 unknown :-1 s=CLOSED pgs=2049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.588+0000 7f3ea784e640 1 --2- 172.21.15.83:0/751970409 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3ea01f2c70 0x7f3ea00b69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.588+0000 7f3ea784e640 1 --2- 172.21.15.83:0/751970409 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3e8c004800 0x7f3ea014e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.588+0000 7f3ea784e640 1 -- 172.21.15.83:0/751970409 >> 172.21.15.83:0/751970409 conn(0x7f3ea009a9f0 msgr2=0x7f3ea00b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:49.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.588+0000 7f3ea784e640 1 -- 172.21.15.83:0/751970409 shutdown_connections 2024-03-06T07:32:49.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.588+0000 7f3ea784e640 1 -- 172.21.15.83:0/751970409 wait complete. 2024-03-06T07:32:49.601 INFO:tasks.ceph:Scrubbing osd.3 2024-03-06T07:32:49.601 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:32:49.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.703+0000 7f2846ad4640 1 Processor -- start 2024-03-06T07:32:49.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.703+0000 7f2846ad4640 1 -- start start 2024-03-06T07:32:49.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.704+0000 7f2846ad4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f28400ce8c0 0x7f28400cc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.704+0000 7f2846ad4640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f284015c980 0x7f2840071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.704+0000 7f2846ad4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28400ced50 con 0x7f284015c5a0 2024-03-06T07:32:49.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.704+0000 7f2846ad4640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2840072a20 con 0x7f284015c980 2024-03-06T07:32:49.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.704+0000 7f2846ad4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28400708b0 con 0x7f28400ce8c0 2024-03-06T07:32:49.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.704+0000 7f283ffff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f28400ce8c0 0x7f28400cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.704+0000 7f2845071640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f284015c980 0x7f2840071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.704+0000 7f2844870640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f284015c5a0 0x7f284006c910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:39946/0 (socket says 172.21.15.83:39946) 2024-03-06T07:32:49.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.704+0000 7f2844870640 1 -- 172.21.15.83:0/584009688 learned_addr learned my addr 172.21.15.83:0/584009688 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:32:49.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.704+0000 7f283f7fe640 1 -- 172.21.15.83:0/584009688 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3851940098 0 0) 0x7f28400ced50 con 0x7f284015c5a0 2024-03-06T07:32:49.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.704+0000 7f283f7fe640 1 -- 172.21.15.83:0/584009688 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2830003520 con 0x7f284015c5a0 2024-03-06T07:32:49.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.704+0000 7f283ffff640 1 -- 172.21.15.83:0/584009688 >> v2:172.21.15.183:3300/0 conn(0x7f284015c980 msgr2=0x7f2840071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:49.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.704+0000 7f283ffff640 1 --2- 172.21.15.83:0/584009688 >> v2:172.21.15.183:3300/0 conn(0x7f284015c980 0x7f2840071300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.704+0000 7f283ffff640 1 -- 172.21.15.83:0/584009688 >> v1:172.21.15.183:6789/0 conn(0x7f284015c5a0 legacy=0x7f284006c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.704+0000 7f283ffff640 1 -- 172.21.15.83:0/584009688 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28400cf030 con 0x7f28400ce8c0 2024-03-06T07:32:49.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.705+0000 7f2845071640 1 --2- 172.21.15.83:0/584009688 >> v2:172.21.15.183:3300/0 conn(0x7f284015c980 0x7f2840071300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:49.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.705+0000 7f283ffff640 1 --2- 172.21.15.83:0/584009688 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f28400ce8c0 0x7f28400cc330 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f282c0097a0 tx=0x7f282c02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b0d756c54fc6ea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:49.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.705+0000 7f283f7fe640 1 -- 172.21.15.83:0/584009688 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282c0315a0 con 0x7f28400ce8c0 2024-03-06T07:32:49.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.705+0000 7f283f7fe640 1 -- 172.21.15.83:0/584009688 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f282c037070 con 0x7f28400ce8c0 2024-03-06T07:32:49.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.706+0000 7f283f7fe640 1 -- 172.21.15.83:0/584009688 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282c031960 con 0x7f28400ce8c0 2024-03-06T07:32:49.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.706+0000 7f2846ad4640 1 -- 172.21.15.83:0/584009688 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f28400ce8c0 msgr2=0x7f28400cc330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.706+0000 7f2846ad4640 1 --2- 172.21.15.83:0/584009688 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f28400ce8c0 0x7f28400cc330 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f282c0097a0 tx=0x7f282c02b840 comp rx=0 tx=0).stop 2024-03-06T07:32:49.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.706+0000 7f2846ad4640 1 -- 172.21.15.83:0/584009688 shutdown_connections 2024-03-06T07:32:49.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.706+0000 7f2846ad4640 1 --2- 172.21.15.83:0/584009688 >> v2:172.21.15.183:3300/0 conn(0x7f284015c980 0x7f2840071300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.706+0000 7f2846ad4640 1 --2- 172.21.15.83:0/584009688 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f28400ce8c0 0x7f28400cc330 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.706+0000 7f2846ad4640 1 -- 172.21.15.83:0/584009688 >> 172.21.15.83:0/584009688 conn(0x7f28400b1650 msgr2=0x7f28400b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:49.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.706+0000 7f2846ad4640 1 -- 172.21.15.83:0/584009688 shutdown_connections 2024-03-06T07:32:49.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.706+0000 7f2846ad4640 1 -- 172.21.15.83:0/584009688 wait complete. 2024-03-06T07:32:49.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.707+0000 7f2846ad4640 1 Processor -- start 2024-03-06T07:32:49.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.707+0000 7f2846ad4640 1 -- start start 2024-03-06T07:32:49.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.708+0000 7f2846ad4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f284015c5a0 0x7f28401e7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.708+0000 7f2846ad4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f284015c980 0x7f28401e4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.708+0000 7f2846ad4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2840143cd0 0x7f284006da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.708+0000 7f2846ad4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28400cc870 con 0x7f284015c980 2024-03-06T07:32:49.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.708+0000 7f2846ad4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2840070e70 con 0x7f284015c5a0 2024-03-06T07:32:49.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.708+0000 7f2846ad4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f284015ce30 con 0x7f2840143cd0 2024-03-06T07:32:49.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.708+0000 7f283ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f284015c980 0x7f28401e4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.708+0000 7f283ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f284015c980 0x7f28401e4dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36374/0 (socket says 172.21.15.83:36374) 2024-03-06T07:32:49.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.708+0000 7f283ffff640 1 -- 172.21.15.83:0/4198438314 learned_addr learned my addr 172.21.15.83:0/4198438314 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:49.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.708+0000 7f2845071640 1 --2- 172.21.15.83:0/4198438314 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2840143cd0 0x7f284006da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.708+0000 7f2844870640 1 --2- 172.21.15.83:0/4198438314 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f284015c5a0 0x7f28401e7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.708+0000 7f283ffff640 1 -- 172.21.15.83:0/4198438314 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2840143cd0 msgr2=0x7f284006da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.708+0000 7f283ffff640 1 --2- 172.21.15.83:0/4198438314 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2840143cd0 0x7f284006da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.708+0000 7f283ffff640 1 -- 172.21.15.83:0/4198438314 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f284015c5a0 msgr2=0x7f28401e7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.708+0000 7f283ffff640 1 --2- 172.21.15.83:0/4198438314 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f284015c5a0 0x7f28401e7290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.708+0000 7f283ffff640 1 -- 172.21.15.83:0/4198438314 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28401e5570 con 0x7f284015c980 2024-03-06T07:32:49.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.708+0000 7f2845071640 1 --2- 172.21.15.83:0/4198438314 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2840143cd0 0x7f284006da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:49.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.708+0000 7f2844870640 1 --2- 172.21.15.83:0/4198438314 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f284015c5a0 0x7f28401e7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:49.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.709+0000 7f283ffff640 1 --2- 172.21.15.83:0/4198438314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f284015c980 0x7f28401e4dd0 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7f282c034040 tx=0x7f282c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:49.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.709+0000 7f283d7fa640 1 -- 172.21.15.83:0/4198438314 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282c032630 con 0x7f284015c980 2024-03-06T07:32:49.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.709+0000 7f283d7fa640 1 -- 172.21.15.83:0/4198438314 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f282c0327d0 con 0x7f284015c980 2024-03-06T07:32:49.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.709+0000 7f283d7fa640 1 -- 172.21.15.83:0/4198438314 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282c032bc0 con 0x7f284015c980 2024-03-06T07:32:49.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.709+0000 7f2846ad4640 1 -- 172.21.15.83:0/4198438314 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28401e5850 con 0x7f284015c980 2024-03-06T07:32:49.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.709+0000 7f2846ad4640 1 -- 172.21.15.83:0/4198438314 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28401f1d60 con 0x7f284015c980 2024-03-06T07:32:49.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.710+0000 7f2846ad4640 1 -- 172.21.15.83:0/4198438314 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f28401e5310 con 0x7f284015c980 2024-03-06T07:32:49.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.710+0000 7f283d7fa640 1 -- 172.21.15.83:0/4198438314 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f282c04e430 con 0x7f284015c980 2024-03-06T07:32:49.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.710+0000 7f283d7fa640 1 --2- 172.21.15.83:0/4198438314 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f282406b4d0 0x7f282406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.711+0000 7f283d7fa640 1 -- 172.21.15.83:0/4198438314 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f282c02f030 con 0x7f284015c980 2024-03-06T07:32:49.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.711+0000 7f2844870640 1 --2- 172.21.15.83:0/4198438314 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f282406b4d0 0x7f282406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.711+0000 7f283d7fa640 1 --2- 172.21.15.83:0/4198438314 >> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] conn(0x7f2824071430 0x7f2824073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.711+0000 7f283d7fa640 1 -- 172.21.15.83:0/4198438314 --> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2824073f00 con 0x7f2824071430 2024-03-06T07:32:49.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.711+0000 7f283d7fa640 1 -- 172.21.15.83:0/4198438314 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_get_version_reply(handle=1 version=135) v2 ==== 24+0+0 (secure 0 0 0) 0x7f282c0b02c0 con 0x7f284015c980 2024-03-06T07:32:49.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.711+0000 7f2845071640 1 --2- 172.21.15.83:0/4198438314 >> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] conn(0x7f2824071430 0x7f2824073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.711+0000 7f2844870640 1 --2- 172.21.15.83:0/4198438314 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f282406b4d0 0x7f282406d990 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f2834007fd0 tx=0x7f2834009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:49.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.711+0000 7f2845071640 1 --2- 172.21.15.83:0/4198438314 >> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] conn(0x7f2824071430 0x7f2824073850 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:49.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.712+0000 7f283d7fa640 1 -- 172.21.15.83:0/4198438314 <== osd.3 v2:172.21.15.83:6815/2656563756 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2824073f00 con 0x7f2824071430 2024-03-06T07:32:49.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.725+0000 7f2846ad4640 1 -- 172.21.15.83:0/4198438314 --> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f284004f330 con 0x7f2824071430 2024-03-06T07:32:49.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.725+0000 7f283d7fa640 1 -- 172.21.15.83:0/4198438314 <== osd.3 v2:172.21.15.83:6815/2656563756 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f284004f330 con 0x7f2824071430 2024-03-06T07:32:49.727 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:32:49.727 INFO:teuthology.orchestra.run.smithi083.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:32:49.727 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:32:49.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.725+0000 7f2846ad4640 1 -- 172.21.15.83:0/4198438314 >> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] conn(0x7f2824071430 msgr2=0x7f2824073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.725+0000 7f2846ad4640 1 --2- 172.21.15.83:0/4198438314 >> [v2:172.21.15.83:6815/2656563756,v1:172.21.15.83:6818/2656563756] conn(0x7f2824071430 0x7f2824073850 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.725+0000 7f2846ad4640 1 -- 172.21.15.83:0/4198438314 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f282406b4d0 msgr2=0x7f282406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.725+0000 7f2846ad4640 1 --2- 172.21.15.83:0/4198438314 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f282406b4d0 0x7f282406d990 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f2834007fd0 tx=0x7f2834009040 comp rx=0 tx=0).stop 2024-03-06T07:32:49.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.725+0000 7f2846ad4640 1 -- 172.21.15.83:0/4198438314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f284015c980 msgr2=0x7f28401e4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.725+0000 7f2846ad4640 1 --2- 172.21.15.83:0/4198438314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f284015c980 0x7f28401e4dd0 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7f282c034040 tx=0x7f282c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.725+0000 7f2845071640 1 -- 172.21.15.83:0/4198438314 reap_dead start 2024-03-06T07:32:49.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.725+0000 7f2846ad4640 1 -- 172.21.15.83:0/4198438314 shutdown_connections 2024-03-06T07:32:49.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.725+0000 7f2846ad4640 1 -- 172.21.15.83:0/4198438314 >> 172.21.15.83:0/4198438314 conn(0x7f28400b1650 msgr2=0x7f28400b1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:49.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.725+0000 7f2846ad4640 1 -- 172.21.15.83:0/4198438314 shutdown_connections 2024-03-06T07:32:49.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.725+0000 7f2846ad4640 1 -- 172.21.15.83:0/4198438314 wait complete. 2024-03-06T07:32:49.738 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-03-06T07:32:49.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.837+0000 7f552f09a640 1 Processor -- start 2024-03-06T07:32:49.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.837+0000 7f552f09a640 1 -- start start 2024-03-06T07:32:49.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.838+0000 7f552f09a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f55280cec30 0x7f55280cf010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.838+0000 7f552f09a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f55280723f0 0x7f55280ad860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.838+0000 7f552f09a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55280cfff0 con 0x7f5528072880 2024-03-06T07:32:49.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.838+0000 7f552f09a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55280d0850 con 0x7f55280723f0 2024-03-06T07:32:49.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.838+0000 7f552f09a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55280cf550 con 0x7f55280cec30 2024-03-06T07:32:49.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.838+0000 7f551ffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f55280723f0 0x7f55280ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.838+0000 7f552d637640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f5528072880 0x7f55280adda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:53850/0 (socket says 172.21.15.83:53850) 2024-03-06T07:32:49.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.838+0000 7f552d637640 1 -- 172.21.15.83:0/1435435075 learned_addr learned my addr 172.21.15.83:0/1435435075 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:32:49.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.838+0000 7f551ffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f55280723f0 0x7f55280ad860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36384/0 (socket says 172.21.15.83:36384) 2024-03-06T07:32:49.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.838+0000 7f552ce36640 1 --2- 172.21.15.83:0/1435435075 >> v2:172.21.15.183:3300/0 conn(0x7f55280cec30 0x7f55280cf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.838+0000 7f551ffff640 1 -- 172.21.15.83:0/1435435075 >> v2:172.21.15.183:3300/0 conn(0x7f55280cec30 msgr2=0x7f55280cf010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:49.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.838+0000 7f551ffff640 1 --2- 172.21.15.83:0/1435435075 >> v2:172.21.15.183:3300/0 conn(0x7f55280cec30 0x7f55280cf010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.838+0000 7f551ffff640 1 -- 172.21.15.83:0/1435435075 >> v1:172.21.15.83:6789/0 conn(0x7f5528072880 legacy=0x7f55280adda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.838+0000 7f551ffff640 1 -- 172.21.15.83:0/1435435075 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f552815e520 con 0x7f55280723f0 2024-03-06T07:32:49.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.839+0000 7f551ffff640 1 --2- 172.21.15.83:0/1435435075 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55280723f0 0x7f55280ad860 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f55280737d0 tx=0x7f551802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e293794075b0870b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:49.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.839+0000 7f551f7fe640 1 -- 172.21.15.83:0/1435435075 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5518035020 con 0x7f55280723f0 2024-03-06T07:32:49.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.839+0000 7f551f7fe640 1 -- 172.21.15.83:0/1435435075 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5518031830 con 0x7f55280723f0 2024-03-06T07:32:49.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.839+0000 7f551f7fe640 1 -- 172.21.15.83:0/1435435075 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5518031c00 con 0x7f55280723f0 2024-03-06T07:32:49.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.839+0000 7f552f09a640 1 -- 172.21.15.83:0/1435435075 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55280723f0 msgr2=0x7f55280ad860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.839+0000 7f552f09a640 1 --2- 172.21.15.83:0/1435435075 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55280723f0 0x7f55280ad860 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f55280737d0 tx=0x7f551802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.840+0000 7f552f09a640 1 -- 172.21.15.83:0/1435435075 shutdown_connections 2024-03-06T07:32:49.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.840+0000 7f552f09a640 1 --2- 172.21.15.83:0/1435435075 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55280723f0 0x7f55280ad860 unknown :-1 s=CLOSED pgs=2052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.840+0000 7f552f09a640 1 --2- 172.21.15.83:0/1435435075 >> v2:172.21.15.183:3300/0 conn(0x7f55280cec30 0x7f55280cf010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.840+0000 7f552f09a640 1 -- 172.21.15.83:0/1435435075 >> 172.21.15.83:0/1435435075 conn(0x7f55280b1c80 msgr2=0x7f55280cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:49.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.840+0000 7f552f09a640 1 -- 172.21.15.83:0/1435435075 shutdown_connections 2024-03-06T07:32:49.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.840+0000 7f552f09a640 1 -- 172.21.15.83:0/1435435075 wait complete. 2024-03-06T07:32:49.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.841+0000 7f552f09a640 1 Processor -- start 2024-03-06T07:32:49.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.841+0000 7f552f09a640 1 -- start start 2024-03-06T07:32:49.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.842+0000 7f552f09a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55280723f0 0x7f55281472d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.842+0000 7f552f09a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5528072880 0x7f552815c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.842+0000 7f552f09a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f55280cec30 0x7f5528158c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.842+0000 7f552f09a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55280cc4d0 con 0x7f55280723f0 2024-03-06T07:32:49.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.842+0000 7f552f09a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55280d0570 con 0x7f55280cec30 2024-03-06T07:32:49.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.842+0000 7f552f09a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55280cc7b0 con 0x7f5528072880 2024-03-06T07:32:49.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.842+0000 7f552ce36640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55280723f0 0x7f55281472d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.842+0000 7f552ce36640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55280723f0 0x7f55281472d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36388/0 (socket says 172.21.15.83:36388) 2024-03-06T07:32:49.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.842+0000 7f552ce36640 1 -- 172.21.15.83:0/420212108 learned_addr learned my addr 172.21.15.83:0/420212108 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:49.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.842+0000 7f551ffff640 1 --2- 172.21.15.83:0/420212108 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5528072880 0x7f552815c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.842+0000 7f552d637640 1 --2- 172.21.15.83:0/420212108 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f55280cec30 0x7f5528158c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.842+0000 7f552ce36640 1 -- 172.21.15.83:0/420212108 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5528072880 msgr2=0x7f552815c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.842+0000 7f552ce36640 1 --2- 172.21.15.83:0/420212108 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5528072880 0x7f552815c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.842+0000 7f552ce36640 1 -- 172.21.15.83:0/420212108 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f55280cec30 msgr2=0x7f5528158c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.842+0000 7f552ce36640 1 --2- 172.21.15.83:0/420212108 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f55280cec30 0x7f5528158c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.842+0000 7f552ce36640 1 -- 172.21.15.83:0/420212108 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f552815cf40 con 0x7f55280723f0 2024-03-06T07:32:49.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.842+0000 7f551ffff640 1 --2- 172.21.15.83:0/420212108 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5528072880 0x7f552815c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:49.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.842+0000 7f552d637640 1 --2- 172.21.15.83:0/420212108 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f55280cec30 0x7f5528158c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:49.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.842+0000 7f552ce36640 1 --2- 172.21.15.83:0/420212108 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55280723f0 0x7f55281472d0 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7f552400d5f0 tx=0x7f552400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:49.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.842+0000 7f551d7fa640 1 -- 172.21.15.83:0/420212108 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55240125f0 con 0x7f55280723f0 2024-03-06T07:32:49.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.842+0000 7f551d7fa640 1 -- 172.21.15.83:0/420212108 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5524019070 con 0x7f55280723f0 2024-03-06T07:32:49.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.842+0000 7f552f09a640 1 -- 172.21.15.83:0/420212108 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f552815d220 con 0x7f55280723f0 2024-03-06T07:32:49.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.842+0000 7f551d7fa640 1 -- 172.21.15.83:0/420212108 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55240129d0 con 0x7f55280723f0 2024-03-06T07:32:49.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.842+0000 7f552f09a640 1 -- 172.21.15.83:0/420212108 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f552815d7e0 con 0x7f55280723f0 2024-03-06T07:32:49.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.843+0000 7f552f09a640 1 -- 172.21.15.83:0/420212108 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f552804f330 con 0x7f55280723f0 2024-03-06T07:32:49.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.844+0000 7f551d7fa640 1 -- 172.21.15.83:0/420212108 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5524012b70 con 0x7f55280723f0 2024-03-06T07:32:49.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.844+0000 7f551d7fa640 1 --2- 172.21.15.83:0/420212108 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f551406b4d0 0x7f551406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.844+0000 7f551d7fa640 1 -- 172.21.15.83:0/420212108 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f552408a370 con 0x7f55280723f0 2024-03-06T07:32:49.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.844+0000 7f551ffff640 1 --2- 172.21.15.83:0/420212108 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f551406b4d0 0x7f551406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.845+0000 7f551ffff640 1 --2- 172.21.15.83:0/420212108 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f551406b4d0 0x7f551406d990 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f5518036000 tx=0x7f5518008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:49.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:49.848+0000 7f551d7fa640 1 -- 172.21.15.83:0/420212108 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5524059490 con 0x7f55280723f0 2024-03-06T07:32:50.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.022+0000 7f552f09a640 1 -- 172.21.15.83:0/420212108 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f55280cc110 con 0x7f551406b4d0 2024-03-06T07:32:50.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.023+0000 7f551d7fa640 1 -- 172.21.15.83:0/420212108 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f55280cc110 con 0x7f551406b4d0 2024-03-06T07:32:50.025 INFO:teuthology.orchestra.run.smithi083.stderr:instructed osd(s) 3 to deep-scrub 2024-03-06T07:32:50.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.025+0000 7f552f09a640 1 -- 172.21.15.83:0/420212108 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f551406b4d0 msgr2=0x7f551406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.025+0000 7f552f09a640 1 --2- 172.21.15.83:0/420212108 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f551406b4d0 0x7f551406d990 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f5518036000 tx=0x7f5518008030 comp rx=0 tx=0).stop 2024-03-06T07:32:50.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.025+0000 7f552f09a640 1 -- 172.21.15.83:0/420212108 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55280723f0 msgr2=0x7f55281472d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.025+0000 7f552f09a640 1 --2- 172.21.15.83:0/420212108 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55280723f0 0x7f55281472d0 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7f552400d5f0 tx=0x7f552400dab0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.025+0000 7f552f09a640 1 -- 172.21.15.83:0/420212108 shutdown_connections 2024-03-06T07:32:50.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.025+0000 7f552f09a640 1 --2- 172.21.15.83:0/420212108 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f551406b4d0 0x7f551406d990 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.025+0000 7f552f09a640 1 --2- 172.21.15.83:0/420212108 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f55280cec30 0x7f5528158c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.025+0000 7f552f09a640 1 --2- 172.21.15.83:0/420212108 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5528072880 0x7f552815c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.025+0000 7f552f09a640 1 --2- 172.21.15.83:0/420212108 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55280723f0 0x7f55281472d0 unknown :-1 s=CLOSED pgs=2053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.025+0000 7f552f09a640 1 -- 172.21.15.83:0/420212108 >> 172.21.15.83:0/420212108 conn(0x7f55280b1c80 msgr2=0x7f55280cf8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:50.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.025+0000 7f552f09a640 1 -- 172.21.15.83:0/420212108 shutdown_connections 2024-03-06T07:32:50.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.025+0000 7f552f09a640 1 -- 172.21.15.83:0/420212108 wait complete. 2024-03-06T07:32:50.037 INFO:tasks.ceph:Scrubbing osd.4 2024-03-06T07:32:50.038 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:32:50.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.140+0000 7ff81b623640 1 Processor -- start 2024-03-06T07:32:50.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.140+0000 7ff81b623640 1 -- start start 2024-03-06T07:32:50.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.141+0000 7ff81b623640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ff81415a490 0x7ff81406df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.141+0000 7ff81b623640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8140b3fd0 0x7ff8140b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.141+0000 7ff81b623640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff8140b2e90 0x7ff8140708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.141+0000 7ff81b623640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff81406e4d0 con 0x7ff8140b2e90 2024-03-06T07:32:50.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.141+0000 7ff81b623640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8140b4490 con 0x7ff81415a490 2024-03-06T07:32:50.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.141+0000 7ff81b623640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8140cf4a0 con 0x7ff8140b3fd0 2024-03-06T07:32:50.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.141+0000 7ff819bc0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff8140b2e90 0x7ff8140708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.141+0000 7ff819bc0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff8140b2e90 0x7ff8140708b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36398/0 (socket says 172.21.15.83:36398) 2024-03-06T07:32:50.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.141+0000 7ff819bc0640 1 -- 172.21.15.83:0/1022044123 learned_addr learned my addr 172.21.15.83:0/1022044123 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:50.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.141+0000 7ff818bbe640 1 --2- 172.21.15.83:0/1022044123 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8140b3fd0 0x7ff8140b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.141+0000 7ff819bc0640 1 -- 172.21.15.83:0/1022044123 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8140b3fd0 msgr2=0x7ff8140b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:50.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.141+0000 7ff8193bf640 1 --2- 172.21.15.83:0/1022044123 >> v2:172.21.15.183:3300/0 conn(0x7ff81415a490 0x7ff81406df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.141+0000 7ff819bc0640 1 --2- 172.21.15.83:0/1022044123 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8140b3fd0 0x7ff8140b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.141+0000 7ff819bc0640 1 -- 172.21.15.83:0/1022044123 >> v2:172.21.15.183:3300/0 conn(0x7ff81415a490 msgr2=0x7ff81406df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:50.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.141+0000 7ff819bc0640 1 --2- 172.21.15.83:0/1022044123 >> v2:172.21.15.183:3300/0 conn(0x7ff81415a490 0x7ff81406df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.141+0000 7ff819bc0640 1 -- 172.21.15.83:0/1022044123 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8140b4750 con 0x7ff8140b2e90 2024-03-06T07:32:50.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.142+0000 7ff819bc0640 1 --2- 172.21.15.83:0/1022044123 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff8140b2e90 0x7ff8140708b0 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7ff81406bf40 tx=0x7ff81002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60c232ebd832e611 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.142+0000 7ff80bfff640 1 -- 172.21.15.83:0/1022044123 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff810035020 con 0x7ff8140b2e90 2024-03-06T07:32:50.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.142+0000 7ff80bfff640 1 -- 172.21.15.83:0/1022044123 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff810031870 con 0x7ff8140b2e90 2024-03-06T07:32:50.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.142+0000 7ff80bfff640 1 -- 172.21.15.83:0/1022044123 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff810031c40 con 0x7ff8140b2e90 2024-03-06T07:32:50.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.143+0000 7ff81b623640 1 -- 172.21.15.83:0/1022044123 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff8140b2e90 msgr2=0x7ff8140708b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.143+0000 7ff81b623640 1 --2- 172.21.15.83:0/1022044123 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff8140b2e90 0x7ff8140708b0 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7ff81406bf40 tx=0x7ff81002b910 comp rx=0 tx=0).stop 2024-03-06T07:32:50.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.143+0000 7ff81b623640 1 -- 172.21.15.83:0/1022044123 shutdown_connections 2024-03-06T07:32:50.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.143+0000 7ff81b623640 1 --2- 172.21.15.83:0/1022044123 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff8140b2e90 0x7ff8140708b0 unknown :-1 s=CLOSED pgs=2054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.143+0000 7ff81b623640 1 --2- 172.21.15.83:0/1022044123 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8140b3fd0 0x7ff8140b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.143+0000 7ff81b623640 1 --2- 172.21.15.83:0/1022044123 >> v2:172.21.15.183:3300/0 conn(0x7ff81415a490 0x7ff81406df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.143+0000 7ff81b623640 1 -- 172.21.15.83:0/1022044123 >> 172.21.15.83:0/1022044123 conn(0x7ff8140b1650 msgr2=0x7ff8140b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:50.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.143+0000 7ff81b623640 1 -- 172.21.15.83:0/1022044123 shutdown_connections 2024-03-06T07:32:50.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.143+0000 7ff81b623640 1 -- 172.21.15.83:0/1022044123 wait complete. 2024-03-06T07:32:50.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.144+0000 7ff81b623640 1 Processor -- start 2024-03-06T07:32:50.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.144+0000 7ff81b623640 1 -- start start 2024-03-06T07:32:50.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.145+0000 7ff81b623640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff8140b2e90 0x7ff81414fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.145+0000 7ff81b623640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff8140b3fd0 0x7ff814153540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.145+0000 7ff81b623640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff81415a490 0x7ff81414bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.145+0000 7ff81b623640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8140cf260 con 0x7ff8140b3fd0 2024-03-06T07:32:50.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.145+0000 7ff81b623640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8140cf0e0 con 0x7ff8140b2e90 2024-03-06T07:32:50.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.145+0000 7ff81b623640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8140b3320 con 0x7ff81415a490 2024-03-06T07:32:50.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.145+0000 7ff818bbe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff8140b3fd0 0x7ff814153540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.145+0000 7ff818bbe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff8140b3fd0 0x7ff814153540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36406/0 (socket says 172.21.15.83:36406) 2024-03-06T07:32:50.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.145+0000 7ff818bbe640 1 -- 172.21.15.83:0/121348447 learned_addr learned my addr 172.21.15.83:0/121348447 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:50.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.145+0000 7ff819bc0640 1 --2- 172.21.15.83:0/121348447 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff81415a490 0x7ff81414bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.145+0000 7ff8193bf640 1 --2- 172.21.15.83:0/121348447 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff8140b2e90 0x7ff81414fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.145+0000 7ff818bbe640 1 -- 172.21.15.83:0/121348447 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff81415a490 msgr2=0x7ff81414bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.145+0000 7ff818bbe640 1 --2- 172.21.15.83:0/121348447 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff81415a490 0x7ff81414bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.145+0000 7ff818bbe640 1 -- 172.21.15.83:0/121348447 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff8140b2e90 msgr2=0x7ff81414fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.145+0000 7ff818bbe640 1 --2- 172.21.15.83:0/121348447 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff8140b2e90 0x7ff81414fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.145+0000 7ff818bbe640 1 -- 172.21.15.83:0/121348447 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8141f5160 con 0x7ff8140b3fd0 2024-03-06T07:32:50.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.145+0000 7ff818bbe640 1 --2- 172.21.15.83:0/121348447 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff8140b3fd0 0x7ff814153540 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7ff80400c7f0 tx=0x7ff80400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.145+0000 7ff809ffb640 1 -- 172.21.15.83:0/121348447 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff804015070 con 0x7ff8140b3fd0 2024-03-06T07:32:50.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.145+0000 7ff809ffb640 1 -- 172.21.15.83:0/121348447 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff804004240 con 0x7ff8140b3fd0 2024-03-06T07:32:50.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.145+0000 7ff81b623640 1 -- 172.21.15.83:0/121348447 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8141f5440 con 0x7ff8140b3fd0 2024-03-06T07:32:50.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.145+0000 7ff809ffb640 1 -- 172.21.15.83:0/121348447 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff804007620 con 0x7ff8140b3fd0 2024-03-06T07:32:50.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.146+0000 7ff81b623640 1 -- 172.21.15.83:0/121348447 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8141f5a00 con 0x7ff8140b3fd0 2024-03-06T07:32:50.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.147+0000 7ff81b623640 1 -- 172.21.15.83:0/121348447 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff81404f330 con 0x7ff8140b3fd0 2024-03-06T07:32:50.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.147+0000 7ff809ffb640 1 -- 172.21.15.83:0/121348447 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff8040079d0 con 0x7ff8140b3fd0 2024-03-06T07:32:50.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.147+0000 7ff809ffb640 1 --2- 172.21.15.83:0/121348447 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff7f806b4d0 0x7ff7f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.147+0000 7ff809ffb640 1 -- 172.21.15.83:0/121348447 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5029+0+0 (secure 0 0 0) 0x7ff80408a300 con 0x7ff8140b3fd0 2024-03-06T07:32:50.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.147+0000 7ff8193bf640 1 --2- 172.21.15.83:0/121348447 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff7f806b4d0 0x7ff7f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.147+0000 7ff809ffb640 1 --2- 172.21.15.83:0/121348447 >> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] conn(0x7ff7f8071430 0x7ff7f8073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.147+0000 7ff809ffb640 1 -- 172.21.15.83:0/121348447 --> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff7f8073f00 con 0x7ff7f8071430 2024-03-06T07:32:50.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.147+0000 7ff809ffb640 1 -- 172.21.15.83:0/121348447 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_get_version_reply(handle=1 version=135) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff804091050 con 0x7ff8140b3fd0 2024-03-06T07:32:50.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.148+0000 7ff819bc0640 1 --2- 172.21.15.83:0/121348447 >> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] conn(0x7ff7f8071430 0x7ff7f8073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.148+0000 7ff8193bf640 1 --2- 172.21.15.83:0/121348447 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff7f806b4d0 0x7ff7f806d990 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7ff80c007130 tx=0x7ff80c007040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.148+0000 7ff819bc0640 1 --2- 172.21.15.83:0/121348447 >> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] conn(0x7ff7f8071430 0x7ff7f8073850 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.149+0000 7ff809ffb640 1 -- 172.21.15.83:0/121348447 <== osd.4 v2:172.21.15.183:6816/3256618830 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff7f8073f00 con 0x7ff7f8071430 2024-03-06T07:32:50.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.161+0000 7ff81b623640 1 -- 172.21.15.83:0/121348447 --> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7ff8140cbbd0 con 0x7ff7f8071430 2024-03-06T07:32:50.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.162+0000 7ff809ffb640 1 -- 172.21.15.83:0/121348447 <== osd.4 v2:172.21.15.183:6816/3256618830 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7ff8140cbbd0 con 0x7ff7f8071430 2024-03-06T07:32:50.164 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:32:50.164 INFO:teuthology.orchestra.run.smithi083.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:32:50.165 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:32:50.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.163+0000 7ff81b623640 1 -- 172.21.15.83:0/121348447 >> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] conn(0x7ff7f8071430 msgr2=0x7ff7f8073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.163+0000 7ff81b623640 1 --2- 172.21.15.83:0/121348447 >> [v2:172.21.15.183:6816/3256618830,v1:172.21.15.183:6817/3256618830] conn(0x7ff7f8071430 0x7ff7f8073850 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.163+0000 7ff81b623640 1 -- 172.21.15.83:0/121348447 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff7f806b4d0 msgr2=0x7ff7f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.163+0000 7ff81b623640 1 --2- 172.21.15.83:0/121348447 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7ff7f806b4d0 0x7ff7f806d990 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7ff80c007130 tx=0x7ff80c007040 comp rx=0 tx=0).stop 2024-03-06T07:32:50.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.163+0000 7ff81b623640 1 -- 172.21.15.83:0/121348447 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff8140b3fd0 msgr2=0x7ff814153540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.163+0000 7ff81b623640 1 --2- 172.21.15.83:0/121348447 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff8140b3fd0 0x7ff814153540 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7ff80400c7f0 tx=0x7ff80400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.163+0000 7ff819bc0640 1 -- 172.21.15.83:0/121348447 reap_dead start 2024-03-06T07:32:50.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.163+0000 7ff81b623640 1 -- 172.21.15.83:0/121348447 shutdown_connections 2024-03-06T07:32:50.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.163+0000 7ff81b623640 1 -- 172.21.15.83:0/121348447 >> 172.21.15.83:0/121348447 conn(0x7ff8140b1650 msgr2=0x7ff8140af5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:50.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.163+0000 7ff81b623640 1 -- 172.21.15.83:0/121348447 shutdown_connections 2024-03-06T07:32:50.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.163+0000 7ff81b623640 1 -- 172.21.15.83:0/121348447 wait complete. 2024-03-06T07:32:50.175 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-03-06T07:32:50.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.274+0000 7f3a13efa640 1 Processor -- start 2024-03-06T07:32:50.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.275+0000 7f3a13efa640 1 -- start start 2024-03-06T07:32:50.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.275+0000 7f3a13efa640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f3a0c0b4460 0x7f3a0c0d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.275+0000 7f3a13efa640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a0c161cf0 0x7f3a0c162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.275+0000 7f3a13efa640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a0c0d22c0 con 0x7f3a0c162970 2024-03-06T07:32:50.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.275+0000 7f3a13efa640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a0c161b00 con 0x7f3a0c0b4460 2024-03-06T07:32:50.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.275+0000 7f3a13efa640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0c0cecc0 con 0x7f3a0c161cf0 2024-03-06T07:32:50.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.276+0000 7f3a11c96640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f3a0c0b4460 0x7f3a0c0d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.276+0000 7f3a11c96640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f3a0c0b4460 0x7f3a0c0d4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36408/0 (socket says 172.21.15.83:36408) 2024-03-06T07:32:50.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.276+0000 7f3a11c96640 1 -- 172.21.15.83:0/3752952821 learned_addr learned my addr 172.21.15.83:0/3752952821 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:50.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.276+0000 7f3a12497640 1 --2- 172.21.15.83:0/3752952821 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a0c161cf0 0x7f3a0c162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.276+0000 7f3a11c96640 1 -- 172.21.15.83:0/3752952821 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a0c161cf0 msgr2=0x7f3a0c162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:50.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.276+0000 7f3a11c96640 1 --2- 172.21.15.83:0/3752952821 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a0c161cf0 0x7f3a0c162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.276+0000 7f3a11c96640 1 -- 172.21.15.83:0/3752952821 >> v1:172.21.15.183:6789/0 conn(0x7f3a0c162970 legacy=0x7f3a0c0b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:50.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.276+0000 7f3a11c96640 1 -- 172.21.15.83:0/3752952821 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a0c0d25a0 con 0x7f3a0c0b4460 2024-03-06T07:32:50.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.277+0000 7f3a11c96640 1 --2- 172.21.15.83:0/3752952821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a0c0b4460 0x7f3a0c0d4800 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7f3a04009870 tx=0x7f3a0402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=805b2a52c104c28d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.277+0000 7f3a10c94640 1 -- 172.21.15.83:0/3752952821 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a04035020 con 0x7f3a0c0b4460 2024-03-06T07:32:50.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.277+0000 7f3a10c94640 1 -- 172.21.15.83:0/3752952821 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a04031640 con 0x7f3a0c0b4460 2024-03-06T07:32:50.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.277+0000 7f3a10c94640 1 -- 172.21.15.83:0/3752952821 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0402d040 con 0x7f3a0c0b4460 2024-03-06T07:32:50.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.277+0000 7f3a13efa640 1 -- 172.21.15.83:0/3752952821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a0c0b4460 msgr2=0x7f3a0c0d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.277+0000 7f3a13efa640 1 --2- 172.21.15.83:0/3752952821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a0c0b4460 0x7f3a0c0d4800 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7f3a04009870 tx=0x7f3a0402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.277+0000 7f3a13efa640 1 -- 172.21.15.83:0/3752952821 shutdown_connections 2024-03-06T07:32:50.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.277+0000 7f3a13efa640 1 --2- 172.21.15.83:0/3752952821 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a0c161cf0 0x7f3a0c162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.277+0000 7f3a13efa640 1 --2- 172.21.15.83:0/3752952821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a0c0b4460 0x7f3a0c0d4800 unknown :-1 s=CLOSED pgs=2056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.277+0000 7f3a13efa640 1 -- 172.21.15.83:0/3752952821 >> 172.21.15.83:0/3752952821 conn(0x7f3a0c0d3ce0 msgr2=0x7f3a0c0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:50.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.277+0000 7f3a13efa640 1 -- 172.21.15.83:0/3752952821 shutdown_connections 2024-03-06T07:32:50.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.277+0000 7f3a13efa640 1 -- 172.21.15.83:0/3752952821 wait complete. 2024-03-06T07:32:50.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.278+0000 7f3a13efa640 1 Processor -- start 2024-03-06T07:32:50.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.279+0000 7f3a13efa640 1 -- start start 2024-03-06T07:32:50.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.279+0000 7f3a13efa640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a0c0b4460 0x7f3a0c1561a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.279+0000 7f3a13efa640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a0c161cf0 0x7f3a0c156f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.279+0000 7f3a13efa640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a0c162970 0x7f3a0c0b7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.279+0000 7f3a13efa640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0c0ce950 con 0x7f3a0c162970 2024-03-06T07:32:50.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.279+0000 7f3a13efa640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0c0d2b60 con 0x7f3a0c0b4460 2024-03-06T07:32:50.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.279+0000 7f3a13efa640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0c0b4840 con 0x7f3a0c161cf0 2024-03-06T07:32:50.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.279+0000 7f3a12497640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a0c162970 0x7f3a0c0b7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.279+0000 7f3a12497640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a0c162970 0x7f3a0c0b7310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36410/0 (socket says 172.21.15.83:36410) 2024-03-06T07:32:50.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.279+0000 7f3a12497640 1 -- 172.21.15.83:0/124771858 learned_addr learned my addr 172.21.15.83:0/124771858 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:50.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.280+0000 7f3a11495640 1 --2- 172.21.15.83:0/124771858 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a0c161cf0 0x7f3a0c156f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.280+0000 7f3a11c96640 1 --2- 172.21.15.83:0/124771858 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a0c0b4460 0x7f3a0c1561a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.280+0000 7f3a12497640 1 -- 172.21.15.83:0/124771858 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a0c161cf0 msgr2=0x7f3a0c156f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.280+0000 7f3a12497640 1 --2- 172.21.15.83:0/124771858 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a0c161cf0 0x7f3a0c156f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.280+0000 7f3a12497640 1 -- 172.21.15.83:0/124771858 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a0c0b4460 msgr2=0x7f3a0c1561a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.280+0000 7f3a12497640 1 --2- 172.21.15.83:0/124771858 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a0c0b4460 0x7f3a0c1561a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.280+0000 7f3a12497640 1 -- 172.21.15.83:0/124771858 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a0c0cadb0 con 0x7f3a0c162970 2024-03-06T07:32:50.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.280+0000 7f3a12497640 1 --2- 172.21.15.83:0/124771858 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a0c162970 0x7f3a0c0b7310 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f3a08007e40 tx=0x7f3a0800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.280+0000 7f3a027fc640 1 -- 172.21.15.83:0/124771858 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a08014620 con 0x7f3a0c162970 2024-03-06T07:32:50.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.280+0000 7f3a027fc640 1 -- 172.21.15.83:0/124771858 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a08019070 con 0x7f3a0c162970 2024-03-06T07:32:50.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.280+0000 7f3a13efa640 1 -- 172.21.15.83:0/124771858 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a0c1566e0 con 0x7f3a0c162970 2024-03-06T07:32:50.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.280+0000 7f3a027fc640 1 -- 172.21.15.83:0/124771858 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a08014a20 con 0x7f3a0c162970 2024-03-06T07:32:50.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.280+0000 7f3a13efa640 1 -- 172.21.15.83:0/124771858 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a0c0cb610 con 0x7f3a0c162970 2024-03-06T07:32:50.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.281+0000 7f3a13efa640 1 -- 172.21.15.83:0/124771858 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a0c08db00 con 0x7f3a0c162970 2024-03-06T07:32:50.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.281+0000 7f3a027fc640 1 -- 172.21.15.83:0/124771858 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3a08014bc0 con 0x7f3a0c162970 2024-03-06T07:32:50.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.282+0000 7f3a027fc640 1 --2- 172.21.15.83:0/124771858 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f39f006b4d0 0x7f39f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.282+0000 7f3a027fc640 1 -- 172.21.15.83:0/124771858 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f3a0808a810 con 0x7f3a0c162970 2024-03-06T07:32:50.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.282+0000 7f3a11c96640 1 --2- 172.21.15.83:0/124771858 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f39f006b4d0 0x7f39f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.282+0000 7f3a11c96640 1 --2- 172.21.15.83:0/124771858 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f39f006b4d0 0x7f39f006d990 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f3a04036000 tx=0x7f3a04008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.286+0000 7f3a027fc640 1 -- 172.21.15.83:0/124771858 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3a0c08db00 con 0x7f3a0c162970 2024-03-06T07:32:50.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.459+0000 7f3a13efa640 1 -- 172.21.15.83:0/124771858 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f3a0c04f330 con 0x7f39f006b4d0 2024-03-06T07:32:50.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.459+0000 7f3a027fc640 1 -- 172.21.15.83:0/124771858 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3a0c04f330 con 0x7f39f006b4d0 2024-03-06T07:32:50.461 INFO:teuthology.orchestra.run.smithi083.stderr:instructed osd(s) 4 to deep-scrub 2024-03-06T07:32:50.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.461+0000 7f3a13efa640 1 -- 172.21.15.83:0/124771858 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f39f006b4d0 msgr2=0x7f39f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.461+0000 7f3a13efa640 1 --2- 172.21.15.83:0/124771858 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f39f006b4d0 0x7f39f006d990 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f3a04036000 tx=0x7f3a04008030 comp rx=0 tx=0).stop 2024-03-06T07:32:50.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.461+0000 7f3a13efa640 1 -- 172.21.15.83:0/124771858 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a0c162970 msgr2=0x7f3a0c0b7310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.461+0000 7f3a13efa640 1 --2- 172.21.15.83:0/124771858 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a0c162970 0x7f3a0c0b7310 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f3a08007e40 tx=0x7f3a0800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.461+0000 7f3a13efa640 1 -- 172.21.15.83:0/124771858 shutdown_connections 2024-03-06T07:32:50.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.461+0000 7f3a13efa640 1 --2- 172.21.15.83:0/124771858 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f39f006b4d0 0x7f39f006d990 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.461+0000 7f3a13efa640 1 --2- 172.21.15.83:0/124771858 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3a0c162970 0x7f3a0c0b7310 unknown :-1 s=CLOSED pgs=2057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.461+0000 7f3a13efa640 1 --2- 172.21.15.83:0/124771858 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a0c161cf0 0x7f3a0c156f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.461+0000 7f3a13efa640 1 --2- 172.21.15.83:0/124771858 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a0c0b4460 0x7f3a0c1561a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.461+0000 7f3a13efa640 1 -- 172.21.15.83:0/124771858 >> 172.21.15.83:0/124771858 conn(0x7f3a0c0d3ce0 msgr2=0x7f3a0c0d0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:50.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.461+0000 7f3a13efa640 1 -- 172.21.15.83:0/124771858 shutdown_connections 2024-03-06T07:32:50.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.461+0000 7f3a13efa640 1 -- 172.21.15.83:0/124771858 wait complete. 2024-03-06T07:32:50.474 INFO:tasks.ceph:Scrubbing osd.5 2024-03-06T07:32:50.474 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:32:50.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.577+0000 7f1ead3e9640 1 Processor -- start 2024-03-06T07:32:50.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.577+0000 7f1ead3e9640 1 -- start start 2024-03-06T07:32:50.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.577+0000 7f1ead3e9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1ea815dfa0 0x7f1ea80d18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.577+0000 7f1ead3e9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1ea80d29f0 0x7f1ea80cb190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.577+0000 7f1ead3e9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ea815e380 con 0x7f1ea80d4630 2024-03-06T07:32:50.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.577+0000 7f1ead3e9640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ea80d2100 con 0x7f1ea815dfa0 2024-03-06T07:32:50.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.577+0000 7f1ead3e9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea80d1e30 con 0x7f1ea80d29f0 2024-03-06T07:32:50.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.578+0000 7f1ea6ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1ea815dfa0 0x7f1ea80d18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.578+0000 7f1ea6ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1ea815dfa0 0x7f1ea80d18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36412/0 (socket says 172.21.15.83:36412) 2024-03-06T07:32:50.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.578+0000 7f1ea6ffd640 1 -- 172.21.15.83:0/1752122336 learned_addr learned my addr 172.21.15.83:0/1752122336 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:50.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.578+0000 7f1ea6ffd640 1 -- 172.21.15.83:0/1752122336 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1ea80d29f0 msgr2=0x7f1ea80cb190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:50.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.578+0000 7f1ea6ffd640 1 --2- 172.21.15.83:0/1752122336 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1ea80d29f0 0x7f1ea80cb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.578+0000 7f1ea6ffd640 1 -- 172.21.15.83:0/1752122336 >> v1:172.21.15.183:6789/0 conn(0x7f1ea80d4630 legacy=0x7f1ea80d22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:50.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.578+0000 7f1ea6ffd640 1 -- 172.21.15.83:0/1752122336 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea80cf470 con 0x7f1ea815dfa0 2024-03-06T07:32:50.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.579+0000 7f1ea6ffd640 1 --2- 172.21.15.83:0/1752122336 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ea815dfa0 0x7f1ea80d18f0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f1e98009870 tx=0x7f1e9802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1359c8ee89c675f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.579+0000 7f1ea5ffb640 1 -- 172.21.15.83:0/1752122336 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e9802e070 con 0x7f1ea815dfa0 2024-03-06T07:32:50.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.579+0000 7f1ea5ffb640 1 -- 172.21.15.83:0/1752122336 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e98037070 con 0x7f1ea815dfa0 2024-03-06T07:32:50.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.579+0000 7f1ea5ffb640 1 -- 172.21.15.83:0/1752122336 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e98031a00 con 0x7f1ea815dfa0 2024-03-06T07:32:50.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.579+0000 7f1ead3e9640 1 -- 172.21.15.83:0/1752122336 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ea815dfa0 msgr2=0x7f1ea80d18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.579+0000 7f1ead3e9640 1 --2- 172.21.15.83:0/1752122336 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ea815dfa0 0x7f1ea80d18f0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f1e98009870 tx=0x7f1e9802b880 comp rx=0 tx=0).stop 2024-03-06T07:32:50.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.579+0000 7f1ead3e9640 1 -- 172.21.15.83:0/1752122336 shutdown_connections 2024-03-06T07:32:50.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.579+0000 7f1ead3e9640 1 --2- 172.21.15.83:0/1752122336 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1ea80d29f0 0x7f1ea80cb190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.579+0000 7f1ead3e9640 1 --2- 172.21.15.83:0/1752122336 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ea815dfa0 0x7f1ea80d18f0 unknown :-1 s=CLOSED pgs=2058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.579+0000 7f1ead3e9640 1 -- 172.21.15.83:0/1752122336 >> 172.21.15.83:0/1752122336 conn(0x7f1ea809d000 msgr2=0x7f1ea80d3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:50.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.579+0000 7f1ead3e9640 1 -- 172.21.15.83:0/1752122336 shutdown_connections 2024-03-06T07:32:50.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.579+0000 7f1ead3e9640 1 -- 172.21.15.83:0/1752122336 wait complete. 2024-03-06T07:32:50.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.581+0000 7f1ead3e9640 1 Processor -- start 2024-03-06T07:32:50.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.581+0000 7f1ead3e9640 1 -- start start 2024-03-06T07:32:50.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.581+0000 7f1ead3e9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ea80d29f0 0x7f1ea8159dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.581+0000 7f1ead3e9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1ea80d4630 0x7f1ea80b4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.581+0000 7f1ead3e9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1ea815dfa0 0x7f1ea80b5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.581+0000 7f1ead3e9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea80cb6d0 con 0x7f1ea80d29f0 2024-03-06T07:32:50.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.581+0000 7f1ead3e9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea80cfa30 con 0x7f1ea815dfa0 2024-03-06T07:32:50.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.581+0000 7f1ead3e9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea80cb850 con 0x7f1ea80d4630 2024-03-06T07:32:50.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.581+0000 7f1ea6ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ea80d29f0 0x7f1ea8159dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.581+0000 7f1ea6ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ea80d29f0 0x7f1ea8159dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36428/0 (socket says 172.21.15.83:36428) 2024-03-06T07:32:50.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.581+0000 7f1ea6ffd640 1 -- 172.21.15.83:0/2643990796 learned_addr learned my addr 172.21.15.83:0/2643990796 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:50.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.582+0000 7f1ea67fc640 1 --2- 172.21.15.83:0/2643990796 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1ea80d4630 0x7f1ea80b4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.582+0000 7f1ea77fe640 1 --2- 172.21.15.83:0/2643990796 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1ea815dfa0 0x7f1ea80b5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.582+0000 7f1ea6ffd640 1 -- 172.21.15.83:0/2643990796 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1ea80d4630 msgr2=0x7f1ea80b4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.582+0000 7f1ea6ffd640 1 --2- 172.21.15.83:0/2643990796 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1ea80d4630 0x7f1ea80b4b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.582+0000 7f1ea6ffd640 1 -- 172.21.15.83:0/2643990796 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1ea815dfa0 msgr2=0x7f1ea80b5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.582+0000 7f1ea6ffd640 1 --2- 172.21.15.83:0/2643990796 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1ea815dfa0 0x7f1ea80b5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.582+0000 7f1ea6ffd640 1 -- 172.21.15.83:0/2643990796 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea80b6b90 con 0x7f1ea80d29f0 2024-03-06T07:32:50.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.582+0000 7f1ea77fe640 1 --2- 172.21.15.83:0/2643990796 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1ea815dfa0 0x7f1ea80b5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:50.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.582+0000 7f1ea6ffd640 1 --2- 172.21.15.83:0/2643990796 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ea80d29f0 0x7f1ea8159dd0 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f1e98000c00 tx=0x7f1e980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.582+0000 7f1e8f7fe640 1 -- 172.21.15.83:0/2643990796 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e98032880 con 0x7f1ea80d29f0 2024-03-06T07:32:50.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.582+0000 7f1e8f7fe640 1 -- 172.21.15.83:0/2643990796 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e98032a20 con 0x7f1ea80d29f0 2024-03-06T07:32:50.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.582+0000 7f1ead3e9640 1 -- 172.21.15.83:0/2643990796 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ea80b6e70 con 0x7f1ea80d29f0 2024-03-06T07:32:50.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.582+0000 7f1e8f7fe640 1 -- 172.21.15.83:0/2643990796 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e98032df0 con 0x7f1ea80d29f0 2024-03-06T07:32:50.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.582+0000 7f1ead3e9640 1 -- 172.21.15.83:0/2643990796 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ea80b7950 con 0x7f1ea80d29f0 2024-03-06T07:32:50.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.583+0000 7f1ead3e9640 1 -- 172.21.15.83:0/2643990796 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1ea80b6940 con 0x7f1ea80d29f0 2024-03-06T07:32:50.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.583+0000 7f1e8f7fe640 1 -- 172.21.15.83:0/2643990796 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1e9804c060 con 0x7f1ea80d29f0 2024-03-06T07:32:50.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.584+0000 7f1e8f7fe640 1 --2- 172.21.15.83:0/2643990796 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1e9406b4d0 0x7f1e9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.584+0000 7f1e8f7fe640 1 -- 172.21.15.83:0/2643990796 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f1e980ab780 con 0x7f1ea80d29f0 2024-03-06T07:32:50.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.584+0000 7f1ea67fc640 1 --2- 172.21.15.83:0/2643990796 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1e9406b4d0 0x7f1e9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.584+0000 7f1e8f7fe640 1 --2- 172.21.15.83:0/2643990796 >> [v2:172.21.15.183:6808/2957306877,v1:172.21.15.183:6809/2957306877] conn(0x7f1e94071430 0x7f1e94073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.584+0000 7f1e8f7fe640 1 -- 172.21.15.83:0/2643990796 --> [v2:172.21.15.183:6808/2957306877,v1:172.21.15.183:6809/2957306877] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1e98033e90 con 0x7f1e94071430 2024-03-06T07:32:50.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.584+0000 7f1e8f7fe640 1 -- 172.21.15.83:0/2643990796 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_get_version_reply(handle=1 version=135) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1e980b2050 con 0x7f1ea80d29f0 2024-03-06T07:32:50.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.584+0000 7f1ea77fe640 1 --2- 172.21.15.83:0/2643990796 >> [v2:172.21.15.183:6808/2957306877,v1:172.21.15.183:6809/2957306877] conn(0x7f1e94071430 0x7f1e94073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.584+0000 7f1ea67fc640 1 --2- 172.21.15.83:0/2643990796 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1e9406b4d0 0x7f1e9406d990 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f1ea806bdf0 tx=0x7f1e9c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.585+0000 7f1ea77fe640 1 --2- 172.21.15.83:0/2643990796 >> [v2:172.21.15.183:6808/2957306877,v1:172.21.15.183:6809/2957306877] conn(0x7f1e94071430 0x7f1e94073850 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.585+0000 7f1e8f7fe640 1 -- 172.21.15.83:0/2643990796 <== osd.5 v2:172.21.15.183:6808/2957306877 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f1e98033e90 con 0x7f1e94071430 2024-03-06T07:32:50.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.598+0000 7f1ead3e9640 1 -- 172.21.15.83:0/2643990796 --> [v2:172.21.15.183:6808/2957306877,v1:172.21.15.183:6809/2957306877] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f1ea804f330 con 0x7f1e94071430 2024-03-06T07:32:50.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.599+0000 7f1e8f7fe640 1 -- 172.21.15.83:0/2643990796 <== osd.5 v2:172.21.15.183:6808/2957306877 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f1ea804f330 con 0x7f1e94071430 2024-03-06T07:32:50.601 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:32:50.601 INFO:teuthology.orchestra.run.smithi083.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:32:50.601 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:32:50.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.599+0000 7f1ead3e9640 1 -- 172.21.15.83:0/2643990796 >> [v2:172.21.15.183:6808/2957306877,v1:172.21.15.183:6809/2957306877] conn(0x7f1e94071430 msgr2=0x7f1e94073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.599+0000 7f1ead3e9640 1 --2- 172.21.15.83:0/2643990796 >> [v2:172.21.15.183:6808/2957306877,v1:172.21.15.183:6809/2957306877] conn(0x7f1e94071430 0x7f1e94073850 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.599+0000 7f1ead3e9640 1 -- 172.21.15.83:0/2643990796 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1e9406b4d0 msgr2=0x7f1e9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.599+0000 7f1ead3e9640 1 --2- 172.21.15.83:0/2643990796 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f1e9406b4d0 0x7f1e9406d990 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f1ea806bdf0 tx=0x7f1e9c009040 comp rx=0 tx=0).stop 2024-03-06T07:32:50.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.599+0000 7f1ead3e9640 1 -- 172.21.15.83:0/2643990796 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ea80d29f0 msgr2=0x7f1ea8159dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.599+0000 7f1ead3e9640 1 --2- 172.21.15.83:0/2643990796 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ea80d29f0 0x7f1ea8159dd0 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f1e98000c00 tx=0x7f1e980046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.599+0000 7f1ea77fe640 1 -- 172.21.15.83:0/2643990796 reap_dead start 2024-03-06T07:32:50.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.599+0000 7f1ead3e9640 1 -- 172.21.15.83:0/2643990796 shutdown_connections 2024-03-06T07:32:50.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.599+0000 7f1ead3e9640 1 -- 172.21.15.83:0/2643990796 >> 172.21.15.83:0/2643990796 conn(0x7f1ea809d000 msgr2=0x7f1ea80cc530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:50.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.599+0000 7f1ead3e9640 1 -- 172.21.15.83:0/2643990796 shutdown_connections 2024-03-06T07:32:50.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.599+0000 7f1ead3e9640 1 -- 172.21.15.83:0/2643990796 wait complete. 2024-03-06T07:32:50.611 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-03-06T07:32:50.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.710+0000 7f99ecda0640 1 Processor -- start 2024-03-06T07:32:50.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.711+0000 7f99ecda0640 1 -- start start 2024-03-06T07:32:50.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.711+0000 7f99ecda0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f99e81456f0 0x7f99e8070940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.711+0000 7f99ecda0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f99e815c2c0 0x7f99e80cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.711+0000 7f99ecda0640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f99e8070e80 0x7f99e80cc830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.712+0000 7f99ecda0640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99e80b3680 con 0x7f99e81456f0 2024-03-06T07:32:50.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.712+0000 7f99ecda0640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99e815c760 con 0x7f99e8070e80 2024-03-06T07:32:50.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.712+0000 7f99ecda0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99e815ca40 con 0x7f99e815c2c0 2024-03-06T07:32:50.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.712+0000 7f99e659c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f99e81456f0 0x7f99e8070940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.712+0000 7f99e659c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f99e81456f0 0x7f99e8070940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36434/0 (socket says 172.21.15.83:36434) 2024-03-06T07:32:50.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.712+0000 7f99e659c640 1 -- 172.21.15.83:0/3535104203 learned_addr learned my addr 172.21.15.83:0/3535104203 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:50.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.712+0000 7f99e5d9b640 1 --2- 172.21.15.83:0/3535104203 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f99e815c2c0 0x7f99e80cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.712+0000 7f99e659c640 1 -- 172.21.15.83:0/3535104203 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f99e815c2c0 msgr2=0x7f99e80cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:50.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.712+0000 7f99e6d9d640 1 --2- 172.21.15.83:0/3535104203 >> v2:172.21.15.183:3300/0 conn(0x7f99e8070e80 0x7f99e80cc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.712+0000 7f99e659c640 1 --2- 172.21.15.83:0/3535104203 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f99e815c2c0 0x7f99e80cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.712+0000 7f99e659c640 1 -- 172.21.15.83:0/3535104203 >> v2:172.21.15.183:3300/0 conn(0x7f99e8070e80 msgr2=0x7f99e80cc830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:50.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.712+0000 7f99e659c640 1 --2- 172.21.15.83:0/3535104203 >> v2:172.21.15.183:3300/0 conn(0x7f99e8070e80 0x7f99e80cc830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.712+0000 7f99e659c640 1 -- 172.21.15.83:0/3535104203 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e815cd20 con 0x7f99e81456f0 2024-03-06T07:32:50.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.712+0000 7f99e5d9b640 1 --2- 172.21.15.83:0/3535104203 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f99e815c2c0 0x7f99e80cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:50.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.712+0000 7f99e6d9d640 1 --2- 172.21.15.83:0/3535104203 >> v2:172.21.15.183:3300/0 conn(0x7f99e8070e80 0x7f99e80cc830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:50.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.713+0000 7f99e659c640 1 --2- 172.21.15.83:0/3535104203 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99e81456f0 0x7f99e8070940 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f99e80b2c30 tx=0x7f99e002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eba5b63cea2ba87c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.713+0000 7f99e559a640 1 -- 172.21.15.83:0/3535104203 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e0035020 con 0x7f99e81456f0 2024-03-06T07:32:50.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.713+0000 7f99e559a640 1 -- 172.21.15.83:0/3535104203 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99e002d040 con 0x7f99e81456f0 2024-03-06T07:32:50.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.713+0000 7f99e559a640 1 -- 172.21.15.83:0/3535104203 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e0031a80 con 0x7f99e81456f0 2024-03-06T07:32:50.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.713+0000 7f99ecda0640 1 -- 172.21.15.83:0/3535104203 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99e81456f0 msgr2=0x7f99e8070940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.713+0000 7f99ecda0640 1 --2- 172.21.15.83:0/3535104203 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99e81456f0 0x7f99e8070940 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f99e80b2c30 tx=0x7f99e002b900 comp rx=0 tx=0).stop 2024-03-06T07:32:50.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.713+0000 7f99ecda0640 1 -- 172.21.15.83:0/3535104203 shutdown_connections 2024-03-06T07:32:50.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.713+0000 7f99ecda0640 1 --2- 172.21.15.83:0/3535104203 >> v2:172.21.15.183:3300/0 conn(0x7f99e8070e80 0x7f99e80cc830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.713+0000 7f99ecda0640 1 --2- 172.21.15.83:0/3535104203 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f99e815c2c0 0x7f99e80cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.713+0000 7f99ecda0640 1 --2- 172.21.15.83:0/3535104203 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99e81456f0 0x7f99e8070940 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.713+0000 7f99ecda0640 1 -- 172.21.15.83:0/3535104203 >> 172.21.15.83:0/3535104203 conn(0x7f99e80b1c80 msgr2=0x7f99e80cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:50.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.713+0000 7f99ecda0640 1 -- 172.21.15.83:0/3535104203 shutdown_connections 2024-03-06T07:32:50.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.714+0000 7f99ecda0640 1 -- 172.21.15.83:0/3535104203 wait complete. 2024-03-06T07:32:50.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.715+0000 7f99ecda0640 1 Processor -- start 2024-03-06T07:32:50.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.715+0000 7f99ecda0640 1 -- start start 2024-03-06T07:32:50.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.715+0000 7f99ecda0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99e8070e80 0x7f99e815b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.715+0000 7f99ecda0640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f99e81456f0 0x7f99e81430d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.715+0000 7f99ecda0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f99e815c2c0 0x7f99e8143610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.715+0000 7f99ecda0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e815d980 con 0x7f99e8070e80 2024-03-06T07:32:50.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.715+0000 7f99ecda0640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e815d800 con 0x7f99e81456f0 2024-03-06T07:32:50.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.715+0000 7f99ecda0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99e815f900 con 0x7f99e815c2c0 2024-03-06T07:32:50.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.716+0000 7f99e659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99e8070e80 0x7f99e815b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.716+0000 7f99e659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99e8070e80 0x7f99e815b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36438/0 (socket says 172.21.15.83:36438) 2024-03-06T07:32:50.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.716+0000 7f99e659c640 1 -- 172.21.15.83:0/2076082644 learned_addr learned my addr 172.21.15.83:0/2076082644 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:50.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.716+0000 7f99e6d9d640 1 --2- 172.21.15.83:0/2076082644 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f99e815c2c0 0x7f99e8143610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.716+0000 7f99e5d9b640 1 --2- 172.21.15.83:0/2076082644 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f99e81456f0 0x7f99e81430d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.716+0000 7f99e659c640 1 -- 172.21.15.83:0/2076082644 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f99e815c2c0 msgr2=0x7f99e8143610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.716+0000 7f99e659c640 1 --2- 172.21.15.83:0/2076082644 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f99e815c2c0 0x7f99e8143610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.716+0000 7f99e659c640 1 -- 172.21.15.83:0/2076082644 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f99e81456f0 msgr2=0x7f99e81430d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.716+0000 7f99e659c640 1 --2- 172.21.15.83:0/2076082644 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f99e81456f0 0x7f99e81430d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.716+0000 7f99e659c640 1 -- 172.21.15.83:0/2076082644 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e815bdc0 con 0x7f99e8070e80 2024-03-06T07:32:50.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.716+0000 7f99e5d9b640 1 --2- 172.21.15.83:0/2076082644 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f99e81456f0 0x7f99e81430d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:50.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.716+0000 7f99e659c640 1 --2- 172.21.15.83:0/2076082644 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99e8070e80 0x7f99e815b880 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7f99e002b940 tx=0x7f99e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.716+0000 7f99deffd640 1 -- 172.21.15.83:0/2076082644 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e0040070 con 0x7f99e8070e80 2024-03-06T07:32:50.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.716+0000 7f99deffd640 1 -- 172.21.15.83:0/2076082644 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99e0039c30 con 0x7f99e8070e80 2024-03-06T07:32:50.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.716+0000 7f99ecda0640 1 -- 172.21.15.83:0/2076082644 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99e81449f0 con 0x7f99e8070e80 2024-03-06T07:32:50.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.716+0000 7f99deffd640 1 -- 172.21.15.83:0/2076082644 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e0035040 con 0x7f99e8070e80 2024-03-06T07:32:50.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.716+0000 7f99ecda0640 1 -- 172.21.15.83:0/2076082644 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99e8144150 con 0x7f99e8070e80 2024-03-06T07:32:50.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.717+0000 7f99ecda0640 1 -- 172.21.15.83:0/2076082644 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99e80cca20 con 0x7f99e8070e80 2024-03-06T07:32:50.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.718+0000 7f99deffd640 1 -- 172.21.15.83:0/2076082644 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f99e0038020 con 0x7f99e8070e80 2024-03-06T07:32:50.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.718+0000 7f99deffd640 1 --2- 172.21.15.83:0/2076082644 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f99c806b4d0 0x7f99c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.718+0000 7f99deffd640 1 -- 172.21.15.83:0/2076082644 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f99e00abd40 con 0x7f99e8070e80 2024-03-06T07:32:50.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.718+0000 7f99e5d9b640 1 --2- 172.21.15.83:0/2076082644 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f99c806b4d0 0x7f99c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.718+0000 7f99e5d9b640 1 --2- 172.21.15.83:0/2076082644 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f99c806b4d0 0x7f99c806d990 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f99d0006fd0 tx=0x7f99d0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.722+0000 7f99deffd640 1 -- 172.21.15.83:0/2076082644 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f99e80cca20 con 0x7f99e8070e80 2024-03-06T07:32:50.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.892+0000 7f99ecda0640 1 -- 172.21.15.83:0/2076082644 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f99e8143b50 con 0x7f99c806b4d0 2024-03-06T07:32:50.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.892+0000 7f99deffd640 1 -- 172.21.15.83:0/2076082644 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f99e8143b50 con 0x7f99c806b4d0 2024-03-06T07:32:50.894 INFO:teuthology.orchestra.run.smithi083.stderr:instructed osd(s) 5 to deep-scrub 2024-03-06T07:32:50.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.894+0000 7f99ecda0640 1 -- 172.21.15.83:0/2076082644 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f99c806b4d0 msgr2=0x7f99c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.894+0000 7f99ecda0640 1 --2- 172.21.15.83:0/2076082644 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f99c806b4d0 0x7f99c806d990 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f99d0006fd0 tx=0x7f99d0008040 comp rx=0 tx=0).stop 2024-03-06T07:32:50.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.894+0000 7f99ecda0640 1 -- 172.21.15.83:0/2076082644 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99e8070e80 msgr2=0x7f99e815b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.894+0000 7f99ecda0640 1 --2- 172.21.15.83:0/2076082644 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99e8070e80 0x7f99e815b880 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7f99e002b940 tx=0x7f99e00046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.894+0000 7f99ecda0640 1 -- 172.21.15.83:0/2076082644 shutdown_connections 2024-03-06T07:32:50.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.894+0000 7f99ecda0640 1 --2- 172.21.15.83:0/2076082644 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f99c806b4d0 0x7f99c806d990 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.894+0000 7f99ecda0640 1 --2- 172.21.15.83:0/2076082644 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f99e815c2c0 0x7f99e8143610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.894+0000 7f99ecda0640 1 --2- 172.21.15.83:0/2076082644 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f99e81456f0 0x7f99e81430d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.894+0000 7f99ecda0640 1 --2- 172.21.15.83:0/2076082644 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99e8070e80 0x7f99e815b880 unknown :-1 s=CLOSED pgs=2061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.894+0000 7f99ecda0640 1 -- 172.21.15.83:0/2076082644 >> 172.21.15.83:0/2076082644 conn(0x7f99e80b1c80 msgr2=0x7f99e815e730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:50.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.894+0000 7f99ecda0640 1 -- 172.21.15.83:0/2076082644 shutdown_connections 2024-03-06T07:32:50.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:50.894+0000 7f99ecda0640 1 -- 172.21.15.83:0/2076082644 wait complete. 2024-03-06T07:32:50.907 INFO:tasks.ceph:Scrubbing osd.6 2024-03-06T07:32:50.907 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:32:51.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.009+0000 7f9bb03ac640 1 Processor -- start 2024-03-06T07:32:51.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.009+0000 7f9bb03ac640 1 -- start start 2024-03-06T07:32:51.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.010+0000 7f9bb03ac640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f9ba80708f0 0x7f9ba80d0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.010+0000 7f9bb03ac640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ba806f420 con 0x7f9ba815bdd0 2024-03-06T07:32:51.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.010+0000 7f9bb03ac640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ba806f700 con 0x7f9ba81602b0 2024-03-06T07:32:51.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.010+0000 7f9bb03ac640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ba815da40 con 0x7f9ba80708f0 2024-03-06T07:32:51.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.010+0000 7f9bae148640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f9ba815bdd0 0x7f9ba815d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:53858/0 (socket says 172.21.15.83:53858) 2024-03-06T07:32:51.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.010+0000 7f9bae148640 1 -- 172.21.15.83:0/3337975310 learned_addr learned my addr 172.21.15.83:0/3337975310 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:32:51.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.011+0000 7f9bae949640 1 --2- 172.21.15.83:0/3337975310 >> v2:172.21.15.183:3300/0 conn(0x7f9ba80708f0 0x7f9ba80d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.011+0000 7f9bae949640 1 -- 172.21.15.83:0/3337975310 >> v1:172.21.15.183:6789/0 conn(0x7f9ba81602b0 legacy=0x7f9ba80701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.011+0000 7f9bae949640 1 -- 172.21.15.83:0/3337975310 >> v1:172.21.15.83:6789/0 conn(0x7f9ba815bdd0 legacy=0x7f9ba815d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.011+0000 7f9bae949640 1 -- 172.21.15.83:0/3337975310 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ba806f9e0 con 0x7f9ba80708f0 2024-03-06T07:32:51.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.012+0000 7f9bae949640 1 --2- 172.21.15.83:0/3337975310 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9ba80708f0 0x7f9ba80d0130 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f9ba0009d90 tx=0x7f9ba002bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c69868a9e5a09b57 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:51.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.012+0000 7f9bad146640 1 -- 172.21.15.83:0/3337975310 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba0039070 con 0x7f9ba80708f0 2024-03-06T07:32:51.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.012+0000 7f9bad146640 1 -- 172.21.15.83:0/3337975310 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ba002cd00 con 0x7f9ba80708f0 2024-03-06T07:32:51.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.012+0000 7f9bad146640 1 -- 172.21.15.83:0/3337975310 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba0034660 con 0x7f9ba80708f0 2024-03-06T07:32:51.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.013+0000 7f9bb03ac640 1 -- 172.21.15.83:0/3337975310 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9ba80708f0 msgr2=0x7f9ba80d0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.013+0000 7f9bb03ac640 1 --2- 172.21.15.83:0/3337975310 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9ba80708f0 0x7f9ba80d0130 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f9ba0009d90 tx=0x7f9ba002bf50 comp rx=0 tx=0).stop 2024-03-06T07:32:51.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.013+0000 7f9bb03ac640 1 -- 172.21.15.83:0/3337975310 shutdown_connections 2024-03-06T07:32:51.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.013+0000 7f9bb03ac640 1 --2- 172.21.15.83:0/3337975310 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9ba80708f0 0x7f9ba80d0130 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.013+0000 7f9bb03ac640 1 -- 172.21.15.83:0/3337975310 >> 172.21.15.83:0/3337975310 conn(0x7f9ba809a9f0 msgr2=0x7f9ba809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:51.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.013+0000 7f9bb03ac640 1 -- 172.21.15.83:0/3337975310 shutdown_connections 2024-03-06T07:32:51.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.013+0000 7f9bb03ac640 1 -- 172.21.15.83:0/3337975310 wait complete. 2024-03-06T07:32:51.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.014+0000 7f9bb03ac640 1 Processor -- start 2024-03-06T07:32:51.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.014+0000 7f9bb03ac640 1 -- start start 2024-03-06T07:32:51.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.015+0000 7f9bb03ac640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ba80708f0 0x7f9ba8150f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.015+0000 7f9bb03ac640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9ba815bdd0 0x7f9ba80b67d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.015+0000 7f9bb03ac640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ba81602b0 0x7f9ba80b6d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.015+0000 7f9bb03ac640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba80d0670 con 0x7f9ba81602b0 2024-03-06T07:32:51.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.015+0000 7f9bb03ac640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba80d08d0 con 0x7f9ba815bdd0 2024-03-06T07:32:51.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.015+0000 7f9bb03ac640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba80d0a50 con 0x7f9ba80708f0 2024-03-06T07:32:51.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.015+0000 7f9bae949640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ba81602b0 0x7f9ba80b6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.015+0000 7f9bae949640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ba81602b0 0x7f9ba80b6d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36446/0 (socket says 172.21.15.83:36446) 2024-03-06T07:32:51.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.015+0000 7f9bae949640 1 -- 172.21.15.83:0/1284817014 learned_addr learned my addr 172.21.15.83:0/1284817014 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:51.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.015+0000 7f9bad947640 1 --2- 172.21.15.83:0/1284817014 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9ba815bdd0 0x7f9ba80b67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.015+0000 7f9bae148640 1 --2- 172.21.15.83:0/1284817014 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ba80708f0 0x7f9ba8150f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.015+0000 7f9bae949640 1 -- 172.21.15.83:0/1284817014 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ba80708f0 msgr2=0x7f9ba8150f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.015+0000 7f9bae949640 1 --2- 172.21.15.83:0/1284817014 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ba80708f0 0x7f9ba8150f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.015+0000 7f9bae949640 1 -- 172.21.15.83:0/1284817014 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9ba815bdd0 msgr2=0x7f9ba80b67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.015+0000 7f9bae949640 1 --2- 172.21.15.83:0/1284817014 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9ba815bdd0 0x7f9ba80b67d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.015+0000 7f9bae949640 1 -- 172.21.15.83:0/1284817014 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ba80c58d0 con 0x7f9ba81602b0 2024-03-06T07:32:51.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.015+0000 7f9bae949640 1 --2- 172.21.15.83:0/1284817014 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ba81602b0 0x7f9ba80b6d10 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f9ba0009d90 tx=0x7f9ba0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:51.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.017+0000 7f9b9effd640 1 -- 172.21.15.83:0/1284817014 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba0039050 con 0x7f9ba81602b0 2024-03-06T07:32:51.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.017+0000 7f9b9effd640 1 -- 172.21.15.83:0/1284817014 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ba0045070 con 0x7f9ba81602b0 2024-03-06T07:32:51.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.017+0000 7f9b9effd640 1 -- 172.21.15.83:0/1284817014 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba003e830 con 0x7f9ba81602b0 2024-03-06T07:32:51.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.017+0000 7f9bb03ac640 1 -- 172.21.15.83:0/1284817014 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ba80c5bb0 con 0x7f9ba81602b0 2024-03-06T07:32:51.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.017+0000 7f9bb03ac640 1 -- 172.21.15.83:0/1284817014 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ba80c6170 con 0x7f9ba81602b0 2024-03-06T07:32:51.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.018+0000 7f9bb03ac640 1 -- 172.21.15.83:0/1284817014 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9ba80c5680 con 0x7f9ba81602b0 2024-03-06T07:32:51.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.018+0000 7f9b9effd640 1 -- 172.21.15.83:0/1284817014 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9ba003e9d0 con 0x7f9ba81602b0 2024-03-06T07:32:51.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.018+0000 7f9b9effd640 1 --2- 172.21.15.83:0/1284817014 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9b8c06b500 0x7f9b8c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.019+0000 7f9b9effd640 1 -- 172.21.15.83:0/1284817014 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f9ba00ace70 con 0x7f9ba81602b0 2024-03-06T07:32:51.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.019+0000 7f9bae148640 1 --2- 172.21.15.83:0/1284817014 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9b8c06b500 0x7f9b8c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.019+0000 7f9b9effd640 1 --2- 172.21.15.83:0/1284817014 >> [v2:172.21.15.183:6800/2649185225,v1:172.21.15.183:6801/2649185225] conn(0x7f9b8c071460 0x7f9b8c073880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.019+0000 7f9b9effd640 1 -- 172.21.15.83:0/1284817014 --> [v2:172.21.15.183:6800/2649185225,v1:172.21.15.183:6801/2649185225] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9ba002cd00 con 0x7f9b8c071460 2024-03-06T07:32:51.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.019+0000 7f9b9effd640 1 -- 172.21.15.83:0/1284817014 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_get_version_reply(handle=1 version=135) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9ba00af570 con 0x7f9ba81602b0 2024-03-06T07:32:51.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.019+0000 7f9bad947640 1 --2- 172.21.15.83:0/1284817014 >> [v2:172.21.15.183:6800/2649185225,v1:172.21.15.183:6801/2649185225] conn(0x7f9b8c071460 0x7f9b8c073880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.019+0000 7f9bae148640 1 --2- 172.21.15.83:0/1284817014 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9b8c06b500 0x7f9b8c06d9c0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f9ba4000f60 tx=0x7f9ba4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:51.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.019+0000 7f9bad947640 1 --2- 172.21.15.83:0/1284817014 >> [v2:172.21.15.183:6800/2649185225,v1:172.21.15.183:6801/2649185225] conn(0x7f9b8c071460 0x7f9b8c073880 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:51.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.020+0000 7f9b9effd640 1 -- 172.21.15.83:0/1284817014 <== osd.6 v2:172.21.15.183:6800/2649185225 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9ba002cd00 con 0x7f9b8c071460 2024-03-06T07:32:51.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.033+0000 7f9bb03ac640 1 -- 172.21.15.83:0/1284817014 --> [v2:172.21.15.183:6800/2649185225,v1:172.21.15.183:6801/2649185225] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f9ba804f330 con 0x7f9b8c071460 2024-03-06T07:32:51.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.034+0000 7f9b9effd640 1 -- 172.21.15.83:0/1284817014 <== osd.6 v2:172.21.15.183:6800/2649185225 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f9ba804f330 con 0x7f9b8c071460 2024-03-06T07:32:51.036 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:32:51.036 INFO:teuthology.orchestra.run.smithi083.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:32:51.036 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:32:51.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.034+0000 7f9bb03ac640 1 -- 172.21.15.83:0/1284817014 >> [v2:172.21.15.183:6800/2649185225,v1:172.21.15.183:6801/2649185225] conn(0x7f9b8c071460 msgr2=0x7f9b8c073880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.034+0000 7f9bb03ac640 1 --2- 172.21.15.83:0/1284817014 >> [v2:172.21.15.183:6800/2649185225,v1:172.21.15.183:6801/2649185225] conn(0x7f9b8c071460 0x7f9b8c073880 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.034+0000 7f9bb03ac640 1 -- 172.21.15.83:0/1284817014 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9b8c06b500 msgr2=0x7f9b8c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.034+0000 7f9bb03ac640 1 --2- 172.21.15.83:0/1284817014 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f9b8c06b500 0x7f9b8c06d9c0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f9ba4000f60 tx=0x7f9ba4009040 comp rx=0 tx=0).stop 2024-03-06T07:32:51.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.034+0000 7f9bb03ac640 1 -- 172.21.15.83:0/1284817014 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ba81602b0 msgr2=0x7f9ba80b6d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.034+0000 7f9bb03ac640 1 --2- 172.21.15.83:0/1284817014 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ba81602b0 0x7f9ba80b6d10 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f9ba0009d90 tx=0x7f9ba0004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:51.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.034+0000 7f9bae949640 1 -- 172.21.15.83:0/1284817014 reap_dead start 2024-03-06T07:32:51.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.034+0000 7f9bb03ac640 1 -- 172.21.15.83:0/1284817014 shutdown_connections 2024-03-06T07:32:51.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.034+0000 7f9bb03ac640 1 -- 172.21.15.83:0/1284817014 >> 172.21.15.83:0/1284817014 conn(0x7f9ba809a9f0 msgr2=0x7f9ba80b2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:51.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.034+0000 7f9bb03ac640 1 -- 172.21.15.83:0/1284817014 shutdown_connections 2024-03-06T07:32:51.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.034+0000 7f9bb03ac640 1 -- 172.21.15.83:0/1284817014 wait complete. 2024-03-06T07:32:51.046 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-03-06T07:32:51.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.146+0000 7fa8a92aa640 1 Processor -- start 2024-03-06T07:32:51.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.146+0000 7fa8a92aa640 1 -- start start 2024-03-06T07:32:51.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.147+0000 7fa8a92aa640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fa8a4162970 0x7fa8a40d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.147+0000 7fa8a92aa640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8a40d22c0 con 0x7fa8a40b4460 2024-03-06T07:32:51.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.147+0000 7fa8a92aa640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8a40d25a0 con 0x7fa8a4161cf0 2024-03-06T07:32:51.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.147+0000 7fa8a92aa640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8a4161b00 con 0x7fa8a4162970 2024-03-06T07:32:51.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.147+0000 7fa8a2ffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fa8a40b4460 0x7fa8a40b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:53862/0 (socket says 172.21.15.83:53862) 2024-03-06T07:32:51.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.147+0000 7fa8a2ffd640 1 -- 172.21.15.83:0/1701862523 learned_addr learned my addr 172.21.15.83:0/1701862523 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:32:51.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.147+0000 7fa8a27fc640 1 --2- 172.21.15.83:0/1701862523 >> v2:172.21.15.183:3300/0 conn(0x7fa8a4162970 0x7fa8a40d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.147+0000 7fa8a1ffb640 1 -- 172.21.15.83:0/1701862523 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2629264837 0 0) 0x7fa8a40d22c0 con 0x7fa8a40b4460 2024-03-06T07:32:51.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.147+0000 7fa8a1ffb640 1 -- 172.21.15.83:0/1701862523 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa88c003520 con 0x7fa8a40b4460 2024-03-06T07:32:51.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.147+0000 7fa8a1ffb640 1 -- 172.21.15.83:0/1701862523 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2002273027 0 0) 0x7fa8a40d25a0 con 0x7fa8a4161cf0 2024-03-06T07:32:51.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.147+0000 7fa8a1ffb640 1 -- 172.21.15.83:0/1701862523 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8a40d22c0 con 0x7fa8a4161cf0 2024-03-06T07:32:51.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.147+0000 7fa8a1ffb640 1 -- 172.21.15.83:0/1701862523 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2158209932 0 0) 0x7fa88c003520 con 0x7fa8a40b4460 2024-03-06T07:32:51.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.147+0000 7fa8a1ffb640 1 -- 172.21.15.83:0/1701862523 >> v2:172.21.15.183:3300/0 conn(0x7fa8a4162970 msgr2=0x7fa8a40d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:51.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.147+0000 7fa8a1ffb640 1 --2- 172.21.15.83:0/1701862523 >> v2:172.21.15.183:3300/0 conn(0x7fa8a4162970 0x7fa8a40d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.147+0000 7fa8a1ffb640 1 -- 172.21.15.83:0/1701862523 >> v1:172.21.15.183:6789/0 conn(0x7fa8a4161cf0 legacy=0x7fa8a40cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.147+0000 7fa8a1ffb640 1 -- 172.21.15.83:0/1701862523 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a40d2880 con 0x7fa8a40b4460 2024-03-06T07:32:51.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.147+0000 7fa8a1ffb640 1 -- 172.21.15.83:0/1701862523 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7fa8940019f0 con 0x7fa8a40b4460 2024-03-06T07:32:51.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.148+0000 7fa8a1ffb640 1 -- 172.21.15.83:0/1701862523 >> v1:172.21.15.83:6789/0 conn(0x7fa8a40b4460 legacy=0x7fa8a40b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.148+0000 7fa8a1ffb640 1 --2- 172.21.15.83:0/1701862523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa88c003b90 0x7fa88c003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.148+0000 7fa8a1ffb640 1 --2- 172.21.15.83:0/1701862523 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa88c004800 0x7fa88c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.148+0000 7fa8a1ffb640 1 --2- 172.21.15.83:0/1701862523 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa88c0051c0 0x7fa88c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.148+0000 7fa8a1ffb640 1 -- 172.21.15.83:0/1701862523 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a4161b00 con 0x7fa88c003b90 2024-03-06T07:32:51.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.148+0000 7fa8a1ffb640 1 -- 172.21.15.83:0/1701862523 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa88c00c090 con 0x7fa88c0051c0 2024-03-06T07:32:51.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.148+0000 7fa8a1ffb640 1 -- 172.21.15.83:0/1701862523 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa88c00c290 con 0x7fa88c004800 2024-03-06T07:32:51.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.148+0000 7fa8a2ffd640 1 --2- 172.21.15.83:0/1701862523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa88c003b90 0x7fa88c003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.148+0000 7fa8a27fc640 1 --2- 172.21.15.83:0/1701862523 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa88c004800 0x7fa88c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.148+0000 7fa8a37fe640 1 --2- 172.21.15.83:0/1701862523 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa88c0051c0 0x7fa88c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.148+0000 7fa8a2ffd640 1 -- 172.21.15.83:0/1701862523 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa88c004800 msgr2=0x7fa88c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:51.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.148+0000 7fa8a2ffd640 1 --2- 172.21.15.83:0/1701862523 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa88c004800 0x7fa88c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.148+0000 7fa8a2ffd640 1 -- 172.21.15.83:0/1701862523 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa88c0051c0 msgr2=0x7fa88c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:51.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.148+0000 7fa8a2ffd640 1 --2- 172.21.15.83:0/1701862523 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa88c0051c0 0x7fa88c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.148+0000 7fa8a2ffd640 1 -- 172.21.15.83:0/1701862523 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa88c003520 con 0x7fa88c003b90 2024-03-06T07:32:51.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.148+0000 7fa8a37fe640 1 -- 172.21.15.83:0/1701862523 reap_dead start 2024-03-06T07:32:51.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.149+0000 7fa8a2ffd640 1 --2- 172.21.15.83:0/1701862523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa88c003b90 0x7fa88c003f90 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7fa89400a750 tx=0x7fa88c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26305908c789c06d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:51.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.149+0000 7fa8a1ffb640 1 -- 172.21.15.83:0/1701862523 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa89400a370 con 0x7fa88c003b90 2024-03-06T07:32:51.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.149+0000 7fa8a1ffb640 1 -- 172.21.15.83:0/1701862523 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa894034630 con 0x7fa88c003b90 2024-03-06T07:32:51.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.149+0000 7fa8a1ffb640 1 -- 172.21.15.83:0/1701862523 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa894034a20 con 0x7fa88c003b90 2024-03-06T07:32:51.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.149+0000 7fa8a92aa640 1 -- 172.21.15.83:0/1701862523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa88c003b90 msgr2=0x7fa88c003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.149+0000 7fa8a92aa640 1 --2- 172.21.15.83:0/1701862523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa88c003b90 0x7fa88c003f90 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7fa89400a750 tx=0x7fa88c005830 comp rx=0 tx=0).stop 2024-03-06T07:32:51.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.149+0000 7fa8a92aa640 1 -- 172.21.15.83:0/1701862523 shutdown_connections 2024-03-06T07:32:51.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.149+0000 7fa8a92aa640 1 --2- 172.21.15.83:0/1701862523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa88c003b90 0x7fa88c003f90 unknown :-1 s=CLOSED pgs=2065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.149+0000 7fa8a92aa640 1 -- 172.21.15.83:0/1701862523 >> 172.21.15.83:0/1701862523 conn(0x7fa8a40d3ce0 msgr2=0x7fa8a40d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:51.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.149+0000 7fa8a92aa640 1 -- 172.21.15.83:0/1701862523 shutdown_connections 2024-03-06T07:32:51.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.149+0000 7fa8a92aa640 1 -- 172.21.15.83:0/1701862523 wait complete. 2024-03-06T07:32:51.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.151+0000 7fa8a92aa640 1 Processor -- start 2024-03-06T07:32:51.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.151+0000 7fa8a92aa640 1 -- start start 2024-03-06T07:32:51.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.151+0000 7fa8a92aa640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa88c003b90 0x7fa8a4156230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.151+0000 7fa8a92aa640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8a40cadb0 0x7fa8a40cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.151+0000 7fa8a92aa640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8a40cb230 0x7fa8a40cd730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.151+0000 7fa8a92aa640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a40ce950 con 0x7fa8a40cadb0 2024-03-06T07:32:51.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.151+0000 7fa8a92aa640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a40cecc0 con 0x7fa88c003b90 2024-03-06T07:32:51.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.151+0000 7fa8a92aa640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a40b4840 con 0x7fa8a40cb230 2024-03-06T07:32:51.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.152+0000 7fa8a27fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8a40cadb0 0x7fa8a40cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.152+0000 7fa8a27fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8a40cadb0 0x7fa8a40cd1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36468/0 (socket says 172.21.15.83:36468) 2024-03-06T07:32:51.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.152+0000 7fa8a27fc640 1 -- 172.21.15.83:0/2953376236 learned_addr learned my addr 172.21.15.83:0/2953376236 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:51.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.152+0000 7fa8a37fe640 1 --2- 172.21.15.83:0/2953376236 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8a40cb230 0x7fa8a40cd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.152+0000 7fa8a2ffd640 1 --2- 172.21.15.83:0/2953376236 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa88c003b90 0x7fa8a4156230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.152+0000 7fa8a27fc640 1 -- 172.21.15.83:0/2953376236 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8a40cb230 msgr2=0x7fa8a40cd730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.152+0000 7fa8a27fc640 1 --2- 172.21.15.83:0/2953376236 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8a40cb230 0x7fa8a40cd730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.152+0000 7fa8a27fc640 1 -- 172.21.15.83:0/2953376236 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa88c003b90 msgr2=0x7fa8a4156230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.152+0000 7fa8a27fc640 1 --2- 172.21.15.83:0/2953376236 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa88c003b90 0x7fa8a4156230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.152+0000 7fa8a27fc640 1 -- 172.21.15.83:0/2953376236 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a40cb790 con 0x7fa8a40cadb0 2024-03-06T07:32:51.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.152+0000 7fa8a37fe640 1 --2- 172.21.15.83:0/2953376236 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8a40cb230 0x7fa8a40cd730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:51.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.152+0000 7fa8a27fc640 1 --2- 172.21.15.83:0/2953376236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8a40cadb0 0x7fa8a40cd1f0 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7fa8980119c0 tx=0x7fa898011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:51.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.152+0000 7fa88b7fe640 1 -- 172.21.15.83:0/2953376236 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa898019070 con 0x7fa8a40cadb0 2024-03-06T07:32:51.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.152+0000 7fa88b7fe640 1 -- 172.21.15.83:0/2953376236 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa898004240 con 0x7fa8a40cadb0 2024-03-06T07:32:51.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.152+0000 7fa8a92aa640 1 -- 172.21.15.83:0/2953376236 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8a40cba70 con 0x7fa8a40cadb0 2024-03-06T07:32:51.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.152+0000 7fa88b7fe640 1 -- 172.21.15.83:0/2953376236 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa89800f8e0 con 0x7fa8a40cadb0 2024-03-06T07:32:51.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.152+0000 7fa8a92aa640 1 -- 172.21.15.83:0/2953376236 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8a40cc030 con 0x7fa8a40cadb0 2024-03-06T07:32:51.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.153+0000 7fa8a92aa640 1 -- 172.21.15.83:0/2953376236 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8a404f330 con 0x7fa8a40cadb0 2024-03-06T07:32:51.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.154+0000 7fa88b7fe640 1 -- 172.21.15.83:0/2953376236 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa89801c070 con 0x7fa8a40cadb0 2024-03-06T07:32:51.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.154+0000 7fa88b7fe640 1 --2- 172.21.15.83:0/2953376236 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa89006b4d0 0x7fa89006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.154+0000 7fa8a2ffd640 1 --2- 172.21.15.83:0/2953376236 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa89006b4d0 0x7fa89006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.154+0000 7fa88b7fe640 1 -- 172.21.15.83:0/2953376236 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fa89808a410 con 0x7fa8a40cadb0 2024-03-06T07:32:51.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.154+0000 7fa8a2ffd640 1 --2- 172.21.15.83:0/2953376236 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa89006b4d0 0x7fa89006d990 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fa894034000 tx=0x7fa894036040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:51.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.158+0000 7fa88b7fe640 1 -- 172.21.15.83:0/2953376236 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa898059540 con 0x7fa8a40cadb0 2024-03-06T07:32:51.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.332+0000 7fa8a92aa640 1 -- 172.21.15.83:0/2953376236 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7fa8a4156770 con 0x7fa89006b4d0 2024-03-06T07:32:51.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.332+0000 7fa88b7fe640 1 -- 172.21.15.83:0/2953376236 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fa89403a040 con 0x7fa89006b4d0 2024-03-06T07:32:51.334 INFO:teuthology.orchestra.run.smithi083.stderr:instructed osd(s) 6 to deep-scrub 2024-03-06T07:32:51.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.334+0000 7fa8a92aa640 1 -- 172.21.15.83:0/2953376236 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa89006b4d0 msgr2=0x7fa89006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.334+0000 7fa8a92aa640 1 --2- 172.21.15.83:0/2953376236 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa89006b4d0 0x7fa89006d990 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fa894034000 tx=0x7fa894036040 comp rx=0 tx=0).stop 2024-03-06T07:32:51.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.334+0000 7fa8a92aa640 1 -- 172.21.15.83:0/2953376236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8a40cadb0 msgr2=0x7fa8a40cd1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.334+0000 7fa8a92aa640 1 --2- 172.21.15.83:0/2953376236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8a40cadb0 0x7fa8a40cd1f0 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7fa8980119c0 tx=0x7fa898011e80 comp rx=0 tx=0).stop 2024-03-06T07:32:51.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.335+0000 7fa8a92aa640 1 -- 172.21.15.83:0/2953376236 shutdown_connections 2024-03-06T07:32:51.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.335+0000 7fa8a92aa640 1 --2- 172.21.15.83:0/2953376236 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7fa89006b4d0 0x7fa89006d990 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.335+0000 7fa8a92aa640 1 --2- 172.21.15.83:0/2953376236 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa8a40cb230 0x7fa8a40cd730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.335+0000 7fa8a92aa640 1 --2- 172.21.15.83:0/2953376236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8a40cadb0 0x7fa8a40cd1f0 unknown :-1 s=CLOSED pgs=2066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.335+0000 7fa8a92aa640 1 --2- 172.21.15.83:0/2953376236 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa88c003b90 0x7fa8a4156230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.335+0000 7fa8a92aa640 1 -- 172.21.15.83:0/2953376236 >> 172.21.15.83:0/2953376236 conn(0x7fa8a40d3ce0 msgr2=0x7fa8a406cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:51.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.335+0000 7fa8a92aa640 1 -- 172.21.15.83:0/2953376236 shutdown_connections 2024-03-06T07:32:51.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.335+0000 7fa8a92aa640 1 -- 172.21.15.83:0/2953376236 wait complete. 2024-03-06T07:32:51.347 INFO:tasks.ceph:Scrubbing osd.7 2024-03-06T07:32:51.348 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:32:51.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.450+0000 7f97ea9d0640 1 Processor -- start 2024-03-06T07:32:51.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.450+0000 7f97ea9d0640 1 -- start start 2024-03-06T07:32:51.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.451+0000 7f97ea9d0640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f97e415ac00 0x7f97e406c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.451+0000 7f97ea9d0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f97e40708b0 0x7f97e4070d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.451+0000 7f97ea9d0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97e415e7a0 0x7f97e415ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.451+0000 7f97ea9d0640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97e4071250 con 0x7f97e40708b0 2024-03-06T07:32:51.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.451+0000 7f97ea9d0640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97e40cdc10 con 0x7f97e415ac00 2024-03-06T07:32:51.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.451+0000 7f97ea9d0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97e415afe0 con 0x7f97e415e7a0 2024-03-06T07:32:51.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.451+0000 7f97e37fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f97e40708b0 0x7f97e4070d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.451+0000 7f97e37fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f97e40708b0 0x7f97e4070d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36474/0 (socket says 172.21.15.83:36474) 2024-03-06T07:32:51.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.451+0000 7f97e37fe640 1 -- 172.21.15.83:0/2664620501 learned_addr learned my addr 172.21.15.83:0/2664620501 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:51.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.452+0000 7f97e37fe640 1 -- 172.21.15.83:0/2664620501 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97e415e7a0 msgr2=0x7f97e415ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:51.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.452+0000 7f97e37fe640 1 --2- 172.21.15.83:0/2664620501 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97e415e7a0 0x7f97e415ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.452+0000 7f97e37fe640 1 -- 172.21.15.83:0/2664620501 >> v2:172.21.15.183:3300/0 conn(0x7f97e415ac00 msgr2=0x7f97e406c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:51.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.452+0000 7f97e3fff640 1 --2- 172.21.15.83:0/2664620501 >> v2:172.21.15.183:3300/0 conn(0x7f97e415ac00 0x7f97e406c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.452+0000 7f97e37fe640 1 --2- 172.21.15.83:0/2664620501 >> v2:172.21.15.183:3300/0 conn(0x7f97e415ac00 0x7f97e406c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.452+0000 7f97e37fe640 1 -- 172.21.15.83:0/2664620501 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e40cd000 con 0x7f97e40708b0 2024-03-06T07:32:51.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.452+0000 7f97e37fe640 1 --2- 172.21.15.83:0/2664620501 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97e40708b0 0x7f97e4070d10 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f97e40730c0 tx=0x7f97d802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de9a9e7977ce357 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:51.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.452+0000 7f97e2ffd640 1 -- 172.21.15.83:0/2664620501 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d802e070 con 0x7f97e40708b0 2024-03-06T07:32:51.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.452+0000 7f97e2ffd640 1 -- 172.21.15.83:0/2664620501 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97d8037070 con 0x7f97e40708b0 2024-03-06T07:32:51.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.452+0000 7f97e2ffd640 1 -- 172.21.15.83:0/2664620501 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d8031a00 con 0x7f97e40708b0 2024-03-06T07:32:51.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.453+0000 7f97ea9d0640 1 -- 172.21.15.83:0/2664620501 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97e40708b0 msgr2=0x7f97e4070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.453+0000 7f97ea9d0640 1 --2- 172.21.15.83:0/2664620501 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97e40708b0 0x7f97e4070d10 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f97e40730c0 tx=0x7f97d802b880 comp rx=0 tx=0).stop 2024-03-06T07:32:51.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.453+0000 7f97ea9d0640 1 -- 172.21.15.83:0/2664620501 shutdown_connections 2024-03-06T07:32:51.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.453+0000 7f97ea9d0640 1 --2- 172.21.15.83:0/2664620501 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97e415e7a0 0x7f97e415ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.453+0000 7f97ea9d0640 1 --2- 172.21.15.83:0/2664620501 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97e40708b0 0x7f97e4070d10 unknown :-1 s=CLOSED pgs=2067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.453+0000 7f97ea9d0640 1 --2- 172.21.15.83:0/2664620501 >> v2:172.21.15.183:3300/0 conn(0x7f97e415ac00 0x7f97e406c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.453+0000 7f97ea9d0640 1 -- 172.21.15.83:0/2664620501 >> 172.21.15.83:0/2664620501 conn(0x7f97e40b1650 msgr2=0x7f97e40b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:51.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.453+0000 7f97ea9d0640 1 -- 172.21.15.83:0/2664620501 shutdown_connections 2024-03-06T07:32:51.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.453+0000 7f97ea9d0640 1 -- 172.21.15.83:0/2664620501 wait complete. 2024-03-06T07:32:51.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.454+0000 7f97ea9d0640 1 Processor -- start 2024-03-06T07:32:51.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.454+0000 7f97ea9d0640 1 -- start start 2024-03-06T07:32:51.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.455+0000 7f97ea9d0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97e40708b0 0x7f97e414b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.455+0000 7f97ea9d0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97e415ac00 0x7f97e40b4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.455+0000 7f97ea9d0640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97e415e7a0 0x7f97e40b4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.455+0000 7f97ea9d0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e40cd8a0 con 0x7f97e40708b0 2024-03-06T07:32:51.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.455+0000 7f97ea9d0640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e40cd5c0 con 0x7f97e415e7a0 2024-03-06T07:32:51.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.455+0000 7f97ea9d0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97e406ce50 con 0x7f97e415ac00 2024-03-06T07:32:51.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.455+0000 7f97e3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97e40708b0 0x7f97e414b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.455+0000 7f97e3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97e40708b0 0x7f97e414b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36490/0 (socket says 172.21.15.83:36490) 2024-03-06T07:32:51.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.455+0000 7f97e3fff640 1 -- 172.21.15.83:0/4120548445 learned_addr learned my addr 172.21.15.83:0/4120548445 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:51.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.455+0000 7f97e3fff640 1 -- 172.21.15.83:0/4120548445 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97e415ac00 msgr2=0x7f97e40b4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.455+0000 7f97e37fe640 1 --2- 172.21.15.83:0/4120548445 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97e415ac00 0x7f97e40b4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.455+0000 7f97e3fff640 1 --2- 172.21.15.83:0/4120548445 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97e415ac00 0x7f97e40b4030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.455+0000 7f97e3fff640 1 -- 172.21.15.83:0/4120548445 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97e415e7a0 msgr2=0x7f97e40b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.455+0000 7f97e3fff640 1 --2- 172.21.15.83:0/4120548445 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97e415e7a0 0x7f97e40b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.455+0000 7f97e3fff640 1 -- 172.21.15.83:0/4120548445 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e41568c0 con 0x7f97e40708b0 2024-03-06T07:32:51.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.456+0000 7f97e3fff640 1 --2- 172.21.15.83:0/4120548445 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97e40708b0 0x7f97e414b0e0 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f97d400eac0 tx=0x7f97d400ef80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:51.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.456+0000 7f97e0ff9640 1 -- 172.21.15.83:0/4120548445 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d4004220 con 0x7f97e40708b0 2024-03-06T07:32:51.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.456+0000 7f97e0ff9640 1 -- 172.21.15.83:0/4120548445 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97d40043c0 con 0x7f97e40708b0 2024-03-06T07:32:51.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.456+0000 7f97ea9d0640 1 -- 172.21.15.83:0/4120548445 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97e4156ba0 con 0x7f97e40708b0 2024-03-06T07:32:51.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.456+0000 7f97e0ff9640 1 -- 172.21.15.83:0/4120548445 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d4007bd0 con 0x7f97e40708b0 2024-03-06T07:32:51.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.456+0000 7f97ea9d0640 1 -- 172.21.15.83:0/4120548445 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97e4157160 con 0x7f97e40708b0 2024-03-06T07:32:51.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.457+0000 7f97ea9d0640 1 -- 172.21.15.83:0/4120548445 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f97e404f330 con 0x7f97e40708b0 2024-03-06T07:32:51.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.457+0000 7f97e0ff9640 1 -- 172.21.15.83:0/4120548445 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f97d400b030 con 0x7f97e40708b0 2024-03-06T07:32:51.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.457+0000 7f97e0ff9640 1 --2- 172.21.15.83:0/4120548445 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f97d006b4d0 0x7f97d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.458+0000 7f97e0ff9640 1 -- 172.21.15.83:0/4120548445 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f97d408c320 con 0x7f97e40708b0 2024-03-06T07:32:51.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.458+0000 7f97e37fe640 1 --2- 172.21.15.83:0/4120548445 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f97d006b4d0 0x7f97d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.458+0000 7f97e0ff9640 1 --2- 172.21.15.83:0/4120548445 >> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] conn(0x7f97d0071430 0x7f97d0073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.458+0000 7f97e0ff9640 1 -- 172.21.15.83:0/4120548445 --> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f97d0073f00 con 0x7f97d0071430 2024-03-06T07:32:51.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.458+0000 7f97e0ff9640 1 -- 172.21.15.83:0/4120548445 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_get_version_reply(handle=1 version=135) v2 ==== 24+0+0 (secure 0 0 0) 0x7f97d4093050 con 0x7f97e40708b0 2024-03-06T07:32:51.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.458+0000 7f97e37fe640 1 --2- 172.21.15.83:0/4120548445 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f97d006b4d0 0x7f97d006d990 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f97d8036000 tx=0x7f97d8008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:51.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.458+0000 7f97e8f6d640 1 --2- 172.21.15.83:0/4120548445 >> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] conn(0x7f97d0071430 0x7f97d0073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.458+0000 7f97e8f6d640 1 --2- 172.21.15.83:0/4120548445 >> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] conn(0x7f97d0071430 0x7f97d0073850 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:51.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.459+0000 7f97e0ff9640 1 -- 172.21.15.83:0/4120548445 <== osd.7 v2:172.21.15.183:6820/3336566695 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f97d0073f00 con 0x7f97d0071430 2024-03-06T07:32:51.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.472+0000 7f97ea9d0640 1 -- 172.21.15.83:0/4120548445 --> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f97e40cfde0 con 0x7f97d0071430 2024-03-06T07:32:51.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.472+0000 7f97e0ff9640 1 -- 172.21.15.83:0/4120548445 <== osd.7 v2:172.21.15.183:6820/3336566695 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f97e40cfde0 con 0x7f97d0071430 2024-03-06T07:32:51.475 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-06T07:32:51.475 INFO:teuthology.orchestra.run.smithi083.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:32:51.475 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-06T07:32:51.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.473+0000 7f97ea9d0640 1 -- 172.21.15.83:0/4120548445 >> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] conn(0x7f97d0071430 msgr2=0x7f97d0073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.473+0000 7f97ea9d0640 1 --2- 172.21.15.83:0/4120548445 >> [v2:172.21.15.183:6820/3336566695,v1:172.21.15.183:6822/3336566695] conn(0x7f97d0071430 0x7f97d0073850 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.473+0000 7f97ea9d0640 1 -- 172.21.15.83:0/4120548445 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f97d006b4d0 msgr2=0x7f97d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.473+0000 7f97ea9d0640 1 --2- 172.21.15.83:0/4120548445 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f97d006b4d0 0x7f97d006d990 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f97d8036000 tx=0x7f97d8008030 comp rx=0 tx=0).stop 2024-03-06T07:32:51.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.473+0000 7f97ea9d0640 1 -- 172.21.15.83:0/4120548445 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97e40708b0 msgr2=0x7f97e414b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.473+0000 7f97ea9d0640 1 --2- 172.21.15.83:0/4120548445 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97e40708b0 0x7f97e414b0e0 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f97d400eac0 tx=0x7f97d400ef80 comp rx=0 tx=0).stop 2024-03-06T07:32:51.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.473+0000 7f97e8f6d640 1 -- 172.21.15.83:0/4120548445 reap_dead start 2024-03-06T07:32:51.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.473+0000 7f97ea9d0640 1 -- 172.21.15.83:0/4120548445 shutdown_connections 2024-03-06T07:32:51.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.473+0000 7f97ea9d0640 1 -- 172.21.15.83:0/4120548445 >> 172.21.15.83:0/4120548445 conn(0x7f97e40b1650 msgr2=0x7f97e40ce680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:51.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.473+0000 7f97ea9d0640 1 -- 172.21.15.83:0/4120548445 shutdown_connections 2024-03-06T07:32:51.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.473+0000 7f97ea9d0640 1 -- 172.21.15.83:0/4120548445 wait complete. 2024-03-06T07:32:51.485 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-03-06T07:32:51.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.584+0000 7f4dea6fb640 1 Processor -- start 2024-03-06T07:32:51.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.584+0000 7f4dea6fb640 1 -- start start 2024-03-06T07:32:51.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.585+0000 7f4dea6fb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4de415b2d0 0x7f4de40708d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.585+0000 7f4dea6fb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4de4070e10 0x7f4de40cda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.585+0000 7f4dea6fb640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4de406d370 con 0x7f4de41444c0 2024-03-06T07:32:51.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.585+0000 7f4dea6fb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4de41448a0 con 0x7f4de4070e10 2024-03-06T07:32:51.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.585+0000 7f4dea6fb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4de4071260 con 0x7f4de415b2d0 2024-03-06T07:32:51.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.585+0000 7f4de8c98640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4de4070e10 0x7f4de40cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.585+0000 7f4de8c98640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4de4070e10 0x7f4de40cda60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36492/0 (socket says 172.21.15.83:36492) 2024-03-06T07:32:51.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.585+0000 7f4de8c98640 1 -- 172.21.15.83:0/109084191 learned_addr learned my addr 172.21.15.83:0/109084191 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:51.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.585+0000 7f4de37fe640 1 --2- 172.21.15.83:0/109084191 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4de415b2d0 0x7f4de40708d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.585+0000 7f4de8c98640 1 -- 172.21.15.83:0/109084191 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4de415b2d0 msgr2=0x7f4de40708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:51.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.585+0000 7f4de8c98640 1 --2- 172.21.15.83:0/109084191 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4de415b2d0 0x7f4de40708d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.585+0000 7f4de8c98640 1 -- 172.21.15.83:0/109084191 >> v1:172.21.15.183:6789/0 conn(0x7f4de41444c0 legacy=0x7f4de406ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:51.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.585+0000 7f4de8c98640 1 -- 172.21.15.83:0/109084191 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de406d650 con 0x7f4de4070e10 2024-03-06T07:32:51.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.586+0000 7f4de8c98640 1 --2- 172.21.15.83:0/109084191 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4de4070e10 0x7f4de40cda60 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc00d150 tx=0x7f4ddc02e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c02245589367aaa7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:51.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.586+0000 7f4de2ffd640 1 -- 172.21.15.83:0/109084191 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ddc02ea80 con 0x7f4de4070e10 2024-03-06T07:32:51.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.586+0000 7f4de2ffd640 1 -- 172.21.15.83:0/109084191 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ddc02ec20 con 0x7f4de4070e10 2024-03-06T07:32:51.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.586+0000 7f4de2ffd640 1 -- 172.21.15.83:0/109084191 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ddc009600 con 0x7f4de4070e10 2024-03-06T07:32:51.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.587+0000 7f4dea6fb640 1 -- 172.21.15.83:0/109084191 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4de4070e10 msgr2=0x7f4de40cda60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.587+0000 7f4dea6fb640 1 --2- 172.21.15.83:0/109084191 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4de4070e10 0x7f4de40cda60 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc00d150 tx=0x7f4ddc02e0e0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.587+0000 7f4dea6fb640 1 -- 172.21.15.83:0/109084191 shutdown_connections 2024-03-06T07:32:51.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.587+0000 7f4dea6fb640 1 --2- 172.21.15.83:0/109084191 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4de4070e10 0x7f4de40cda60 unknown :-1 s=CLOSED pgs=2069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.587+0000 7f4dea6fb640 1 --2- 172.21.15.83:0/109084191 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4de415b2d0 0x7f4de40708d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.587+0000 7f4dea6fb640 1 -- 172.21.15.83:0/109084191 >> 172.21.15.83:0/109084191 conn(0x7f4de40b1c80 msgr2=0x7f4de40cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:51.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.587+0000 7f4dea6fb640 1 -- 172.21.15.83:0/109084191 shutdown_connections 2024-03-06T07:32:51.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.587+0000 7f4dea6fb640 1 -- 172.21.15.83:0/109084191 wait complete. 2024-03-06T07:32:51.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.588+0000 7f4dea6fb640 1 Processor -- start 2024-03-06T07:32:51.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.588+0000 7f4dea6fb640 1 -- start start 2024-03-06T07:32:51.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.589+0000 7f4dea6fb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4de4070e10 0x7f4de4132d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.589+0000 7f4dea6fb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4de41444c0 0x7f4de41313d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.589+0000 7f4dea6fb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4de415b2d0 0x7f4de415e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.589+0000 7f4dea6fb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de406dc10 con 0x7f4de415b2d0 2024-03-06T07:32:51.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.589+0000 7f4dea6fb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de406d110 con 0x7f4de41444c0 2024-03-06T07:32:51.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.589+0000 7f4dea6fb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4de406def0 con 0x7f4de4070e10 2024-03-06T07:32:51.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.589+0000 7f4de8c98640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4de415b2d0 0x7f4de415e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.589+0000 7f4de8c98640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4de415b2d0 0x7f4de415e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36508/0 (socket says 172.21.15.83:36508) 2024-03-06T07:32:51.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.589+0000 7f4de8c98640 1 -- 172.21.15.83:0/3765199428 learned_addr learned my addr 172.21.15.83:0/3765199428 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:51.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.589+0000 7f4de3fff640 1 --2- 172.21.15.83:0/3765199428 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4de4070e10 0x7f4de4132d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.589+0000 7f4de37fe640 1 --2- 172.21.15.83:0/3765199428 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4de41444c0 0x7f4de41313d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.589+0000 7f4de8c98640 1 -- 172.21.15.83:0/3765199428 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4de4070e10 msgr2=0x7f4de4132d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.589+0000 7f4de8c98640 1 --2- 172.21.15.83:0/3765199428 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4de4070e10 0x7f4de4132d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.589+0000 7f4de8c98640 1 -- 172.21.15.83:0/3765199428 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4de41444c0 msgr2=0x7f4de41313d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.589+0000 7f4de8c98640 1 --2- 172.21.15.83:0/3765199428 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4de41444c0 0x7f4de41313d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.589+0000 7f4de8c98640 1 -- 172.21.15.83:0/3765199428 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de4131cd0 con 0x7f4de415b2d0 2024-03-06T07:32:51.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.589+0000 7f4de3fff640 1 --2- 172.21.15.83:0/3765199428 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4de4070e10 0x7f4de4132d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:51.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.589+0000 7f4de8c98640 1 --2- 172.21.15.83:0/3765199428 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4de415b2d0 0x7f4de415e700 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc038040 tx=0x7f4ddc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:51.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.589+0000 7f4de0ff9640 1 -- 172.21.15.83:0/3765199428 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ddc02ea80 con 0x7f4de415b2d0 2024-03-06T07:32:51.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.589+0000 7f4de0ff9640 1 -- 172.21.15.83:0/3765199428 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ddc045070 con 0x7f4de415b2d0 2024-03-06T07:32:51.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.590+0000 7f4de0ff9640 1 -- 172.21.15.83:0/3765199428 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ddc00b720 con 0x7f4de415b2d0 2024-03-06T07:32:51.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.590+0000 7f4dea6fb640 1 -- 172.21.15.83:0/3765199428 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4de4131fb0 con 0x7f4de415b2d0 2024-03-06T07:32:51.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.590+0000 7f4dea6fb640 1 -- 172.21.15.83:0/3765199428 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4de4132570 con 0x7f4de415b2d0 2024-03-06T07:32:51.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.590+0000 7f4dea6fb640 1 -- 172.21.15.83:0/3765199428 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4de404f330 con 0x7f4de415b2d0 2024-03-06T07:32:51.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.591+0000 7f4de0ff9640 1 -- 172.21.15.83:0/3765199428 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4ddc036a60 con 0x7f4de415b2d0 2024-03-06T07:32:51.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.591+0000 7f4de0ff9640 1 --2- 172.21.15.83:0/3765199428 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4dd006b4d0 0x7f4dd006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.591+0000 7f4de3fff640 1 --2- 172.21.15.83:0/3765199428 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4dd006b4d0 0x7f4dd006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.591+0000 7f4de0ff9640 1 -- 172.21.15.83:0/3765199428 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f4ddc0ad000 con 0x7f4de415b2d0 2024-03-06T07:32:51.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.592+0000 7f4de3fff640 1 --2- 172.21.15.83:0/3765199428 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4dd006b4d0 0x7f4dd006d990 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f4dd4003eb0 tx=0x7f4dd4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:51.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.596+0000 7f4de0ff9640 1 -- 172.21.15.83:0/3765199428 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4ddc07c1c0 con 0x7f4de415b2d0 2024-03-06T07:32:51.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.768+0000 7f4dea6fb640 1 -- 172.21.15.83:0/3765199428 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f4de41414e0 con 0x7f4dd006b4d0 2024-03-06T07:32:51.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.768+0000 7f4de0ff9640 1 -- 172.21.15.83:0/3765199428 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f4de41414e0 con 0x7f4dd006b4d0 2024-03-06T07:32:51.770 INFO:teuthology.orchestra.run.smithi083.stderr:instructed osd(s) 7 to deep-scrub 2024-03-06T07:32:51.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.770+0000 7f4dea6fb640 1 -- 172.21.15.83:0/3765199428 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4dd006b4d0 msgr2=0x7f4dd006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.770+0000 7f4dea6fb640 1 --2- 172.21.15.83:0/3765199428 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4dd006b4d0 0x7f4dd006d990 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f4dd4003eb0 tx=0x7f4dd4009040 comp rx=0 tx=0).stop 2024-03-06T07:32:51.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.770+0000 7f4dea6fb640 1 -- 172.21.15.83:0/3765199428 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4de415b2d0 msgr2=0x7f4de415e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.770+0000 7f4dea6fb640 1 --2- 172.21.15.83:0/3765199428 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4de415b2d0 0x7f4de415e700 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc038040 tx=0x7f4ddc004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:51.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.770+0000 7f4dea6fb640 1 -- 172.21.15.83:0/3765199428 shutdown_connections 2024-03-06T07:32:51.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.770+0000 7f4dea6fb640 1 --2- 172.21.15.83:0/3765199428 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f4dd006b4d0 0x7f4dd006d990 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.770+0000 7f4dea6fb640 1 --2- 172.21.15.83:0/3765199428 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4de415b2d0 0x7f4de415e700 unknown :-1 s=CLOSED pgs=2070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.770+0000 7f4dea6fb640 1 --2- 172.21.15.83:0/3765199428 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4de41444c0 0x7f4de41313d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.770+0000 7f4dea6fb640 1 --2- 172.21.15.83:0/3765199428 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4de4070e10 0x7f4de4132d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.770+0000 7f4dea6fb640 1 -- 172.21.15.83:0/3765199428 >> 172.21.15.83:0/3765199428 conn(0x7f4de40b1c80 msgr2=0x7f4de406ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:51.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.770+0000 7f4dea6fb640 1 -- 172.21.15.83:0/3765199428 shutdown_connections 2024-03-06T07:32:51.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.770+0000 7f4dea6fb640 1 -- 172.21.15.83:0/3765199428 wait complete. 2024-03-06T07:32:51.783 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:32:51.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.882+0000 7f8d65b93640 1 Processor -- start 2024-03-06T07:32:51.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.882+0000 7f8d65b93640 1 -- start start 2024-03-06T07:32:51.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.882+0000 7f8d65b93640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f8d600b3060 0x7f8d6015e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.882+0000 7f8d65b93640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d6015f580 con 0x7f8d6015bf30 2024-03-06T07:32:51.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.882+0000 7f8d65b93640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d6015b450 con 0x7f8d6013dc10 2024-03-06T07:32:51.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.882+0000 7f8d65b93640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d600cf3f0 con 0x7f8d600b3060 2024-03-06T07:32:51.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.883+0000 7f8d5ffff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f8d6015bf30 0x7f8d6015ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:53870/0 (socket says 172.21.15.83:53870) 2024-03-06T07:32:51.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.883+0000 7f8d5ffff640 1 -- 172.21.15.83:0/2596670373 learned_addr learned my addr 172.21.15.83:0/2596670373 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:32:51.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.883+0000 7f8d56ffd640 1 --2- 172.21.15.83:0/2596670373 >> v2:172.21.15.183:3300/0 conn(0x7f8d600b3060 0x7f8d6015e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.883+0000 7f8d5effd640 1 -- 172.21.15.83:0/2596670373 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3601860136 0 0) 0x7f8d6015f580 con 0x7f8d6015bf30 2024-03-06T07:32:51.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.883+0000 7f8d5effd640 1 -- 172.21.15.83:0/2596670373 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d4c003520 con 0x7f8d6015bf30 2024-03-06T07:32:51.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.883+0000 7f8d5effd640 1 -- 172.21.15.83:0/2596670373 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4093571603 0 0) 0x7f8d6015b450 con 0x7f8d6013dc10 2024-03-06T07:32:51.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.883+0000 7f8d5effd640 1 -- 172.21.15.83:0/2596670373 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d6015f580 con 0x7f8d6013dc10 2024-03-06T07:32:51.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.883+0000 7f8d5effd640 1 -- 172.21.15.83:0/2596670373 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 912376280 0 0) 0x7f8d4c003520 con 0x7f8d6015bf30 2024-03-06T07:32:51.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.883+0000 7f8d56ffd640 1 -- 172.21.15.83:0/2596670373 >> v1:172.21.15.183:6789/0 conn(0x7f8d6013dc10 legacy=0x7f8d600b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.883+0000 7f8d56ffd640 1 -- 172.21.15.83:0/2596670373 >> v1:172.21.15.83:6789/0 conn(0x7f8d6015bf30 legacy=0x7f8d6015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.883+0000 7f8d56ffd640 1 -- 172.21.15.83:0/2596670373 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d6015b730 con 0x7f8d600b3060 2024-03-06T07:32:51.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.884+0000 7f8d56ffd640 1 --2- 172.21.15.83:0/2596670373 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8d600b3060 0x7f8d6015e850 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f8d480097a0 tx=0x7f8d4802b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db7780e49783e2ce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:51.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.884+0000 7f8d5effd640 1 -- 172.21.15.83:0/2596670373 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d48037070 con 0x7f8d600b3060 2024-03-06T07:32:51.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.884+0000 7f8d5effd640 1 -- 172.21.15.83:0/2596670373 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d48030510 con 0x7f8d600b3060 2024-03-06T07:32:51.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.884+0000 7f8d5effd640 1 -- 172.21.15.83:0/2596670373 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d580019f0 con 0x7f8d600b3060 2024-03-06T07:32:51.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.885+0000 7f8d65b93640 1 -- 172.21.15.83:0/2596670373 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8d600b3060 msgr2=0x7f8d6015e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.885+0000 7f8d65b93640 1 --2- 172.21.15.83:0/2596670373 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8d600b3060 0x7f8d6015e850 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f8d480097a0 tx=0x7f8d4802b790 comp rx=0 tx=0).stop 2024-03-06T07:32:51.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.885+0000 7f8d65b93640 1 -- 172.21.15.83:0/2596670373 shutdown_connections 2024-03-06T07:32:51.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.885+0000 7f8d65b93640 1 --2- 172.21.15.83:0/2596670373 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8d600b3060 0x7f8d6015e850 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.885+0000 7f8d65b93640 1 -- 172.21.15.83:0/2596670373 >> 172.21.15.83:0/2596670373 conn(0x7f8d600cf8a0 msgr2=0x7f8d600cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:51.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.885+0000 7f8d65b93640 1 -- 172.21.15.83:0/2596670373 shutdown_connections 2024-03-06T07:32:51.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.885+0000 7f8d65b93640 1 -- 172.21.15.83:0/2596670373 wait complete. 2024-03-06T07:32:51.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.886+0000 7f8d65b93640 1 Processor -- start 2024-03-06T07:32:51.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.886+0000 7f8d65b93640 1 -- start start 2024-03-06T07:32:51.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.887+0000 7f8d65b93640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8d600b3060 0x7f8d601f3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.887+0000 7f8d65b93640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d6013dc10 0x7f8d601440a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.887+0000 7f8d65b93640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8d6015bf30 0x7f8d601f7820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.887+0000 7f8d65b93640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d600718e0 con 0x7f8d6013dc10 2024-03-06T07:32:51.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.887+0000 7f8d65b93640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d6015b210 con 0x7f8d6015bf30 2024-03-06T07:32:51.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.887+0000 7f8d65b93640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d60071a60 con 0x7f8d600b3060 2024-03-06T07:32:51.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.887+0000 7f8d56ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d6013dc10 0x7f8d601440a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.887+0000 7f8d56ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d6013dc10 0x7f8d601440a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36524/0 (socket says 172.21.15.83:36524) 2024-03-06T07:32:51.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.887+0000 7f8d56ffd640 1 -- 172.21.15.83:0/1102311017 learned_addr learned my addr 172.21.15.83:0/1102311017 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:32:51.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.887+0000 7f8d5ffff640 1 --2- 172.21.15.83:0/1102311017 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8d6015bf30 0x7f8d601f7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.887+0000 7f8d5f7fe640 1 --2- 172.21.15.83:0/1102311017 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8d600b3060 0x7f8d601f3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.887+0000 7f8d56ffd640 1 -- 172.21.15.83:0/1102311017 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8d600b3060 msgr2=0x7f8d601f3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.887+0000 7f8d56ffd640 1 --2- 172.21.15.83:0/1102311017 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8d600b3060 0x7f8d601f3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.887+0000 7f8d56ffd640 1 -- 172.21.15.83:0/1102311017 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8d6015bf30 msgr2=0x7f8d601f7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.887+0000 7f8d56ffd640 1 --2- 172.21.15.83:0/1102311017 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8d6015bf30 0x7f8d601f7820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.887+0000 7f8d56ffd640 1 -- 172.21.15.83:0/1102311017 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d601f4300 con 0x7f8d6013dc10 2024-03-06T07:32:51.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.887+0000 7f8d5ffff640 1 --2- 172.21.15.83:0/1102311017 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8d6015bf30 0x7f8d601f7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:51.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.887+0000 7f8d56ffd640 1 --2- 172.21.15.83:0/1102311017 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d6013dc10 0x7f8d601440a0 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f8d48000c00 tx=0x7f8d480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:51.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.888+0000 7f8d5cff9640 1 -- 172.21.15.83:0/1102311017 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d48040020 con 0x7f8d6013dc10 2024-03-06T07:32:51.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.888+0000 7f8d5cff9640 1 -- 172.21.15.83:0/1102311017 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d48038040 con 0x7f8d6013dc10 2024-03-06T07:32:51.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.888+0000 7f8d5cff9640 1 -- 172.21.15.83:0/1102311017 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d48037070 con 0x7f8d6013dc10 2024-03-06T07:32:51.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.888+0000 7f8d65b93640 1 -- 172.21.15.83:0/1102311017 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d601f45e0 con 0x7f8d6013dc10 2024-03-06T07:32:51.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.888+0000 7f8d65b93640 1 -- 172.21.15.83:0/1102311017 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d601f7300 con 0x7f8d6013dc10 2024-03-06T07:32:51.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.889+0000 7f8d65b93640 1 -- 172.21.15.83:0/1102311017 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d6004f330 con 0x7f8d6013dc10 2024-03-06T07:32:51.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.889+0000 7f8d5cff9640 1 -- 172.21.15.83:0/1102311017 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8d48043020 con 0x7f8d6013dc10 2024-03-06T07:32:51.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.889+0000 7f8d5cff9640 1 --2- 172.21.15.83:0/1102311017 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8d4006b4d0 0x7f8d4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.889+0000 7f8d5cff9640 1 -- 172.21.15.83:0/1102311017 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f8d480ab2b0 con 0x7f8d6013dc10 2024-03-06T07:32:51.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.889+0000 7f8d5f7fe640 1 --2- 172.21.15.83:0/1102311017 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8d4006b4d0 0x7f8d4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.890+0000 7f8d5f7fe640 1 --2- 172.21.15.83:0/1102311017 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8d4006b4d0 0x7f8d4006d990 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f8d500011e0 tx=0x7f8d50009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:51.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:51.894+0000 7f8d5cff9640 1 -- 172.21.15.83:0/1102311017 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8d4804a030 con 0x7f8d6013dc10 2024-03-06T07:32:52.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:52.063+0000 7f8d65b93640 1 -- 172.21.15.83:0/1102311017 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8d6006eb00 con 0x7f8d4006b4d0 2024-03-06T07:32:52.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:52.063+0000 7f8d5cff9640 1 -- 172.21.15.83:0/1102311017 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13389 (secure 0 0 0) 0x7f8d6006eb00 con 0x7f8d4006b4d0 2024-03-06T07:32:52.065 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:32:52.066 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-06T07:32:52.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:52.065+0000 7f8d65b93640 1 -- 172.21.15.83:0/1102311017 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8d4006b4d0 msgr2=0x7f8d4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:52.065+0000 7f8d65b93640 1 --2- 172.21.15.83:0/1102311017 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8d4006b4d0 0x7f8d4006d990 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f8d500011e0 tx=0x7f8d50009040 comp rx=0 tx=0).stop 2024-03-06T07:32:52.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:52.065+0000 7f8d65b93640 1 -- 172.21.15.83:0/1102311017 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d6013dc10 msgr2=0x7f8d601440a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:52.065+0000 7f8d65b93640 1 --2- 172.21.15.83:0/1102311017 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d6013dc10 0x7f8d601440a0 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f8d48000c00 tx=0x7f8d480046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:52.065+0000 7f8d65b93640 1 -- 172.21.15.83:0/1102311017 shutdown_connections 2024-03-06T07:32:52.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:52.065+0000 7f8d65b93640 1 --2- 172.21.15.83:0/1102311017 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f8d4006b4d0 0x7f8d4006d990 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:52.065+0000 7f8d65b93640 1 --2- 172.21.15.83:0/1102311017 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8d6015bf30 0x7f8d601f7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:52.065+0000 7f8d65b93640 1 --2- 172.21.15.83:0/1102311017 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d6013dc10 0x7f8d601440a0 unknown :-1 s=CLOSED pgs=2072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:52.065+0000 7f8d65b93640 1 --2- 172.21.15.83:0/1102311017 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8d600b3060 0x7f8d601f3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:52.066+0000 7f8d65b93640 1 -- 172.21.15.83:0/1102311017 >> 172.21.15.83:0/1102311017 conn(0x7f8d600cf8a0 msgr2=0x7f8d600afe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:52.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:52.066+0000 7f8d65b93640 1 -- 172.21.15.83:0/1102311017 shutdown_connections 2024-03-06T07:32:52.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:32:52.066+0000 7f8d65b93640 1 -- 172.21.15.83:0/1102311017 wait complete. 2024-03-06T07:32:52.078 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":461,"stamp":"2024-03-06T07:32:50.436999+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":330392,"kb_used_data":14376,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754644328,"statfs":{"total":773094113280,"available":772755791872,"internally_reserved":0,"allocated":14721024,"data_stored":7086712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12757,"internal_metadata":323472939},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-33900,"num_objects":-44,"num_object_clones":0,"num_object_copies":-88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-4,"num_read_kb":0,"num_write":-148,"num_write_kb":-83,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001288"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":380,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:32:44.163214+0000","last_change":"2024-03-06T07:21:47.553693+0000","last_active":"2024-03-06T07:32:44.163214+0000","last_peered":"2024-03-06T07:32:44.163214+0000","last_clean":"2024-03-06T07:32:44.163214+0000","last_became_active":"2024-03-06T07:21:47.552664+0000","last_became_peered":"2024-03-06T07:21:47.552664+0000","last_unstale":"2024-03-06T07:32:44.163214+0000","last_undegraded":"2024-03-06T07:32:44.163214+0000","last_fullsized":"2024-03-06T07:32:44.163214+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:46.362028+0000","last_clean_scrub_stamp":"2024-03-06T07:21:46.362028+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:02.916426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574986,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41736,"kb_used_data":2232,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330104,"statfs":{"total":96636764160,"available":96594026496,"internally_reserved":0,"allocated":2285568,"data_stored":1328615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":40434121},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574984,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41152,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330688,"statfs":{"total":96636764160,"available":96594624512,"internally_reserved":0,"allocated":1691648,"data_stored":738247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574985,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41152,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330688,"statfs":{"total":96636764160,"available":96594624512,"internally_reserved":0,"allocated":1691648,"data_stored":738247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574986,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41156,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330684,"statfs":{"total":96636764160,"available":96594620416,"internally_reserved":0,"allocated":1691648,"data_stored":738247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574985,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41156,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330684,"statfs":{"total":96636764160,"available":96594620416,"internally_reserved":0,"allocated":1691648,"data_stored":738247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574985,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41156,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330684,"statfs":{"total":96636764160,"available":96594620416,"internally_reserved":0,"allocated":1691648,"data_stored":738247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574986,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41152,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330688,"statfs":{"total":96636764160,"available":96594624512,"internally_reserved":0,"allocated":1691648,"data_stored":738247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574985,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41732,"kb_used_data":2232,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330108,"statfs":{"total":96636764160,"available":96594030592,"internally_reserved":0,"allocated":2285568,"data_stored":1328615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:32:52.081 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-06T07:21:46.362028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=6, tm_hour=7, tm_min=21, tm_sec=46, tm_wday=2, tm_yday=66, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=6, tm_hour=7, tm_min=32, tm_sec=47, tm_wday=2, tm_yday=66, tm_isdst=0) 2024-03-06T07:32:52.081 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-06T07:33:01.182 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:01.185 DEBUG:teuthology.orchestra.run.smithi183:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:12.082 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:33:12.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.191+0000 7f3c3d77a640 1 Processor -- start 2024-03-06T07:33:12.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.191+0000 7f3c3d77a640 1 -- start start 2024-03-06T07:33:12.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.192+0000 7f3c3d77a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3c3815dfe0 0x7f3c380cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.192+0000 7f3c3d77a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f3c380d29c0 0x7f3c380b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.192+0000 7f3c3d77a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c380b3430 con 0x7f3c380cf830 2024-03-06T07:33:12.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.192+0000 7f3c3d77a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c380b31e0 con 0x7f3c380d29c0 2024-03-06T07:33:12.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.192+0000 7f3c3d77a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c380b3d60 con 0x7f3c3815dfe0 2024-03-06T07:33:12.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.193+0000 7f3c36ffd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3c3815dfe0 0x7f3c380cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.193+0000 7f3c377fe640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f3c380d29c0 0x7f3c380b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.193+0000 7f3c367fc640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f3c380cf830 0x7f3c380d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.83:35710/0 (socket says 172.21.15.83:35710) 2024-03-06T07:33:12.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.193+0000 7f3c36ffd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3c3815dfe0 0x7f3c380cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.83:50424/0 (socket says 172.21.15.83:50424) 2024-03-06T07:33:12.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.193+0000 7f3c367fc640 1 -- 172.21.15.83:0/2103148309 learned_addr learned my addr 172.21.15.83:0/2103148309 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:33:12.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.193+0000 7f3c377fe640 1 -- 172.21.15.83:0/2103148309 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3c3815dfe0 msgr2=0x7f3c380cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:12.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.193+0000 7f3c377fe640 1 --2- 172.21.15.83:0/2103148309 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3c3815dfe0 0x7f3c380cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.193+0000 7f3c377fe640 1 -- 172.21.15.83:0/2103148309 >> v1:172.21.15.183:6789/0 conn(0x7f3c380cf830 legacy=0x7f3c380d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.193+0000 7f3c377fe640 1 -- 172.21.15.83:0/2103148309 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c380b3710 con 0x7f3c380d29c0 2024-03-06T07:33:12.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.193+0000 7f3c35ffb640 1 -- 172.21.15.83:0/2103148309 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1627594667 0 0) 0x7f3c380b3430 con 0x7f3c380cf830 2024-03-06T07:33:12.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.193+0000 7f3c36ffd640 1 --2- 172.21.15.83:0/2103148309 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3c3815dfe0 0x7f3c380cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:33:12.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.194+0000 7f3c377fe640 1 --2- 172.21.15.83:0/2103148309 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3c380d29c0 0x7f3c380b41f0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f3c300097a0 tx=0x7f3c3002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5858adedeaa47204 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:12.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.194+0000 7f3c35ffb640 1 -- 172.21.15.83:0/2103148309 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c30035020 con 0x7f3c380d29c0 2024-03-06T07:33:12.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.194+0000 7f3c35ffb640 1 -- 172.21.15.83:0/2103148309 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c30031830 con 0x7f3c380d29c0 2024-03-06T07:33:12.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.194+0000 7f3c35ffb640 1 -- 172.21.15.83:0/2103148309 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c30031c00 con 0x7f3c380d29c0 2024-03-06T07:33:12.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.195+0000 7f3c3d77a640 1 -- 172.21.15.83:0/2103148309 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3c380d29c0 msgr2=0x7f3c380b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.195+0000 7f3c3d77a640 1 --2- 172.21.15.83:0/2103148309 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3c380d29c0 0x7f3c380b41f0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f3c300097a0 tx=0x7f3c3002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.195+0000 7f3c3d77a640 1 -- 172.21.15.83:0/2103148309 shutdown_connections 2024-03-06T07:33:12.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.195+0000 7f3c3d77a640 1 --2- 172.21.15.83:0/2103148309 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3c380d29c0 0x7f3c380b41f0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.195+0000 7f3c3d77a640 1 --2- 172.21.15.83:0/2103148309 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3c3815dfe0 0x7f3c380cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.195+0000 7f3c3d77a640 1 -- 172.21.15.83:0/2103148309 >> 172.21.15.83:0/2103148309 conn(0x7f3c380d3cd0 msgr2=0x7f3c380d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:12.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.195+0000 7f3c3d77a640 1 -- 172.21.15.83:0/2103148309 shutdown_connections 2024-03-06T07:33:12.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.195+0000 7f3c3d77a640 1 -- 172.21.15.83:0/2103148309 wait complete. 2024-03-06T07:33:12.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.196+0000 7f3c3d77a640 1 Processor -- start 2024-03-06T07:33:12.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.196+0000 7f3c3d77a640 1 -- start start 2024-03-06T07:33:12.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.197+0000 7f3c3d77a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3c380cf830 0x7f3c3814f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.197+0000 7f3c3d77a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3c380d29c0 0x7f3c3814fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.197+0000 7f3c3d77a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3c3815dfe0 0x7f3c380b7520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.197+0000 7f3c3d77a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c380d1df0 con 0x7f3c380d29c0 2024-03-06T07:33:12.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.197+0000 7f3c3d77a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c380d1c70 con 0x7f3c3815dfe0 2024-03-06T07:33:12.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.197+0000 7f3c3d77a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c3815e3c0 con 0x7f3c380cf830 2024-03-06T07:33:12.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.197+0000 7f3c367fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3c380d29c0 0x7f3c3814fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.197+0000 7f3c367fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3c380d29c0 0x7f3c3814fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49292/0 (socket says 172.21.15.83:49292) 2024-03-06T07:33:12.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.197+0000 7f3c367fc640 1 -- 172.21.15.83:0/289209483 learned_addr learned my addr 172.21.15.83:0/289209483 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:33:12.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.197+0000 7f3c377fe640 1 --2- 172.21.15.83:0/289209483 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3c3815dfe0 0x7f3c380b7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.197+0000 7f3c36ffd640 1 --2- 172.21.15.83:0/289209483 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3c380cf830 0x7f3c3814f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.197+0000 7f3c367fc640 1 -- 172.21.15.83:0/289209483 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3c380cf830 msgr2=0x7f3c3814f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.197+0000 7f3c367fc640 1 --2- 172.21.15.83:0/289209483 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3c380cf830 0x7f3c3814f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.197+0000 7f3c367fc640 1 -- 172.21.15.83:0/289209483 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3c3815dfe0 msgr2=0x7f3c380b7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.197+0000 7f3c367fc640 1 --2- 172.21.15.83:0/289209483 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3c3815dfe0 0x7f3c380b7520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.197+0000 7f3c367fc640 1 -- 172.21.15.83:0/289209483 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c3815b950 con 0x7f3c380d29c0 2024-03-06T07:33:12.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.198+0000 7f3c367fc640 1 --2- 172.21.15.83:0/289209483 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3c380d29c0 0x7f3c3814fb70 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c00c4b0 tx=0x7f3c2c00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:12.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.198+0000 7f3c1f7fe640 1 -- 172.21.15.83:0/289209483 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c2c018070 con 0x7f3c380d29c0 2024-03-06T07:33:12.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.198+0000 7f3c1f7fe640 1 -- 172.21.15.83:0/289209483 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c2c007500 con 0x7f3c380d29c0 2024-03-06T07:33:12.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.198+0000 7f3c3d77a640 1 -- 172.21.15.83:0/289209483 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c380b8070 con 0x7f3c380d29c0 2024-03-06T07:33:12.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.198+0000 7f3c1f7fe640 1 -- 172.21.15.83:0/289209483 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c2c004a20 con 0x7f3c380d29c0 2024-03-06T07:33:12.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.198+0000 7f3c3d77a640 1 -- 172.21.15.83:0/289209483 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c380cca70 con 0x7f3c380d29c0 2024-03-06T07:33:12.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.199+0000 7f3c3d77a640 1 -- 172.21.15.83:0/289209483 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c380d3770 con 0x7f3c380d29c0 2024-03-06T07:33:12.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.199+0000 7f3c1f7fe640 1 -- 172.21.15.83:0/289209483 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3c2c01b070 con 0x7f3c380d29c0 2024-03-06T07:33:12.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.199+0000 7f3c1f7fe640 1 --2- 172.21.15.83:0/289209483 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3c2406b500 0x7f3c2406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.200+0000 7f3c1f7fe640 1 -- 172.21.15.83:0/289209483 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f3c2c089110 con 0x7f3c380d29c0 2024-03-06T07:33:12.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.200+0000 7f3c36ffd640 1 --2- 172.21.15.83:0/289209483 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3c2406b500 0x7f3c2406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.200+0000 7f3c36ffd640 1 --2- 172.21.15.83:0/289209483 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3c2406b500 0x7f3c2406d9c0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f3c28009bd0 tx=0x7f3c2800b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:12.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.204+0000 7f3c1f7fe640 1 -- 172.21.15.83:0/289209483 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3c380d3770 con 0x7f3c380d29c0 2024-03-06T07:33:12.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.371+0000 7f3c3d77a640 1 -- 172.21.15.83:0/289209483 --> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3c3806a190 con 0x7f3c2406b500 2024-03-06T07:33:12.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.372+0000 7f3c1f7fe640 1 -- 172.21.15.83:0/289209483 <== mgr.4110 v2:172.21.15.83:6832/2272295342 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13379 (secure 0 0 0) 0x7f3c3806a190 con 0x7f3c2406b500 2024-03-06T07:33:12.374 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:33:12.374 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-06T07:33:12.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.374+0000 7f3c3d77a640 1 -- 172.21.15.83:0/289209483 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3c2406b500 msgr2=0x7f3c2406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.374+0000 7f3c3d77a640 1 --2- 172.21.15.83:0/289209483 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3c2406b500 0x7f3c2406d9c0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f3c28009bd0 tx=0x7f3c2800b040 comp rx=0 tx=0).stop 2024-03-06T07:33:12.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.374+0000 7f3c3d77a640 1 -- 172.21.15.83:0/289209483 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3c380d29c0 msgr2=0x7f3c3814fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.374+0000 7f3c3d77a640 1 --2- 172.21.15.83:0/289209483 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3c380d29c0 0x7f3c3814fb70 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c00c4b0 tx=0x7f3c2c00c970 comp rx=0 tx=0).stop 2024-03-06T07:33:12.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.374+0000 7f3c3d77a640 1 -- 172.21.15.83:0/289209483 shutdown_connections 2024-03-06T07:33:12.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.374+0000 7f3c3d77a640 1 --2- 172.21.15.83:0/289209483 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f3c2406b500 0x7f3c2406d9c0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.374+0000 7f3c3d77a640 1 --2- 172.21.15.83:0/289209483 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3c3815dfe0 0x7f3c380b7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.374+0000 7f3c3d77a640 1 --2- 172.21.15.83:0/289209483 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3c380d29c0 0x7f3c3814fb70 unknown :-1 s=CLOSED pgs=2073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.374+0000 7f3c3d77a640 1 --2- 172.21.15.83:0/289209483 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3c380cf830 0x7f3c3814f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.374+0000 7f3c3d77a640 1 -- 172.21.15.83:0/289209483 >> 172.21.15.83:0/289209483 conn(0x7f3c380d3cd0 msgr2=0x7f3c3806c940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:12.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.374+0000 7f3c3d77a640 1 -- 172.21.15.83:0/289209483 shutdown_connections 2024-03-06T07:33:12.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.374+0000 7f3c3d77a640 1 -- 172.21.15.83:0/289209483 wait complete. 2024-03-06T07:33:12.387 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":471,"stamp":"2024-03-06T07:33:10.441004+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":330380,"kb_used_data":14376,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754644340,"statfs":{"total":773094113280,"available":772755804160,"internally_reserved":0,"allocated":14721024,"data_stored":7086712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12757,"internal_metadata":323472939},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002629"},"pg_stats":[{"pgid":"1.0","version":"135'119","reported_seq":391,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:32:51.790321+0000","last_change":"2024-03-06T07:32:51.790364+0000","last_active":"2024-03-06T07:32:51.790321+0000","last_peered":"2024-03-06T07:32:51.790321+0000","last_clean":"2024-03-06T07:32:51.790321+0000","last_became_active":"2024-03-06T07:21:47.552664+0000","last_became_peered":"2024-03-06T07:21:47.552664+0000","last_unstale":"2024-03-06T07:32:51.790321+0000","last_undegraded":"2024-03-06T07:32:51.790321+0000","last_fullsized":"2024-03-06T07:32:51.790321+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'119","last_scrub_stamp":"2024-03-06T07:32:51.790225+0000","last_deep_scrub":"135'119","last_deep_scrub_stamp":"2024-03-06T07:32:51.790225+0000","last_clean_scrub_stamp":"2024-03-06T07:32:51.790225+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:54.117863+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574990,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41728,"kb_used_data":2232,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330112,"statfs":{"total":96636764160,"available":96594034688,"internally_reserved":0,"allocated":2285568,"data_stored":1328615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":40434121},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574988,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41152,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330688,"statfs":{"total":96636764160,"available":96594624512,"internally_reserved":0,"allocated":1691648,"data_stored":738247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574989,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41152,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330688,"statfs":{"total":96636764160,"available":96594624512,"internally_reserved":0,"allocated":1691648,"data_stored":738247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574990,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41156,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330684,"statfs":{"total":96636764160,"available":96594620416,"internally_reserved":0,"allocated":1691648,"data_stored":738247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574989,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41156,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330684,"statfs":{"total":96636764160,"available":96594620416,"internally_reserved":0,"allocated":1691648,"data_stored":738247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574989,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41156,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330684,"statfs":{"total":96636764160,"available":96594620416,"internally_reserved":0,"allocated":1691648,"data_stored":738247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574990,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41152,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330688,"statfs":{"total":96636764160,"available":96594624512,"internally_reserved":0,"allocated":1691648,"data_stored":738247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574989,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41728,"kb_used_data":2232,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330112,"statfs":{"total":96636764160,"available":96594034688,"internally_reserved":0,"allocated":2285568,"data_stored":1328615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:33:12.388 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-06T07:33:12.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.486+0000 7f0cd7916640 1 Processor -- start 2024-03-06T07:33:12.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.486+0000 7f0cd7916640 1 -- start start 2024-03-06T07:33:12.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.487+0000 7f0cd7916640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f0cd01614d0 0x7f0cd00d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.487+0000 7f0cd7916640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cd01618b0 con 0x7f0cd00d1910 2024-03-06T07:33:12.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.487+0000 7f0cd7916640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cd00cf400 con 0x7f0cd00d24b0 2024-03-06T07:33:12.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.487+0000 7f0cd7916640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cd00cf6a0 con 0x7f0cd01614d0 2024-03-06T07:33:12.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.487+0000 7f0cd4eb1640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f0cd00d1910 0x7f0cd00d1da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60888/0 (socket says 172.21.15.83:60888) 2024-03-06T07:33:12.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.487+0000 7f0cd4eb1640 1 -- 172.21.15.83:0/585478067 learned_addr learned my addr 172.21.15.83:0/585478067 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-06T07:33:12.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.488+0000 7f0cc7fff640 1 -- 172.21.15.83:0/585478067 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 850301065 0 0) 0x7f0cd01618b0 con 0x7f0cd00d1910 2024-03-06T07:33:12.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.488+0000 7f0cd56b2640 1 --2- 172.21.15.83:0/585478067 >> v2:172.21.15.183:3300/0 conn(0x7f0cd01614d0 0x7f0cd00d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.488+0000 7f0cc7fff640 1 -- 172.21.15.83:0/585478067 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cbc003520 con 0x7f0cd00d1910 2024-03-06T07:33:12.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.488+0000 7f0cc7fff640 1 -- 172.21.15.83:0/585478067 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 825424212 0 0) 0x7f0cbc003520 con 0x7f0cd00d1910 2024-03-06T07:33:12.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.488+0000 7f0cc7fff640 1 -- 172.21.15.83:0/585478067 >> v2:172.21.15.183:3300/0 conn(0x7f0cd01614d0 msgr2=0x7f0cd00d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:12.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.488+0000 7f0cc7fff640 1 --2- 172.21.15.83:0/585478067 >> v2:172.21.15.183:3300/0 conn(0x7f0cd01614d0 0x7f0cd00d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.488+0000 7f0cc7fff640 1 -- 172.21.15.83:0/585478067 >> v1:172.21.15.183:6789/0 conn(0x7f0cd00d24b0 legacy=0x7f0cd00b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.488+0000 7f0cc7fff640 1 -- 172.21.15.83:0/585478067 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd00cf980 con 0x7f0cd00d1910 2024-03-06T07:33:12.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.488+0000 7f0cc7fff640 1 -- 172.21.15.83:0/585478067 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1019246290 0 0) 0x7f0cc0001670 con 0x7f0cd00d1910 2024-03-06T07:33:12.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.488+0000 7f0cc7fff640 1 -- 172.21.15.83:0/585478067 >> v1:172.21.15.83:6789/0 conn(0x7f0cd00d1910 legacy=0x7f0cd00d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.488+0000 7f0cc7fff640 1 --2- 172.21.15.83:0/585478067 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0cbc0039b0 0x7f0cbc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.488+0000 7f0cc7fff640 1 --2- 172.21.15.83:0/585478067 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0cbc004800 0x7f0cbc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.488+0000 7f0cc7fff640 1 --2- 172.21.15.83:0/585478067 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0cbc0051c0 0x7f0cbc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.488+0000 7f0cc7fff640 1 -- 172.21.15.83:0/585478067 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd00cf6a0 con 0x7f0cbc0039b0 2024-03-06T07:33:12.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.488+0000 7f0cc7fff640 1 -- 172.21.15.83:0/585478067 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc00c060 con 0x7f0cbc0051c0 2024-03-06T07:33:12.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.488+0000 7f0cc7fff640 1 -- 172.21.15.83:0/585478067 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc00c260 con 0x7f0cbc004800 2024-03-06T07:33:12.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.488+0000 7f0cd56b2640 1 --2- 172.21.15.83:0/585478067 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0cbc0039b0 0x7f0cbc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.488+0000 7f0cd4eb1640 1 --2- 172.21.15.83:0/585478067 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0cbc004800 0x7f0cbc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.488+0000 7f0cd5eb3640 1 --2- 172.21.15.83:0/585478067 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0cbc0051c0 0x7f0cbc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.488+0000 7f0cd56b2640 1 -- 172.21.15.83:0/585478067 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0cbc004800 msgr2=0x7f0cbc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:12.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.488+0000 7f0cd56b2640 1 --2- 172.21.15.83:0/585478067 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0cbc004800 0x7f0cbc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.488+0000 7f0cd56b2640 1 -- 172.21.15.83:0/585478067 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0cbc0051c0 msgr2=0x7f0cbc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:12.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.488+0000 7f0cd56b2640 1 --2- 172.21.15.83:0/585478067 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0cbc0051c0 0x7f0cbc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.488+0000 7f0cd5eb3640 1 -- 172.21.15.83:0/585478067 reap_dead start 2024-03-06T07:33:12.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.488+0000 7f0cd56b2640 1 -- 172.21.15.83:0/585478067 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cbc003520 con 0x7f0cbc0039b0 2024-03-06T07:33:12.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.489+0000 7f0cd56b2640 1 --2- 172.21.15.83:0/585478067 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0cbc0039b0 0x7f0cbc003dd0 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc009fc0 tx=0x7f0cbc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcf09545b9f3db1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:12.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.489+0000 7f0cc7fff640 1 -- 172.21.15.83:0/585478067 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ccc039070 con 0x7f0cbc0039b0 2024-03-06T07:33:12.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.489+0000 7f0cc7fff640 1 -- 172.21.15.83:0/585478067 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ccc02cc20 con 0x7f0cbc0039b0 2024-03-06T07:33:12.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.489+0000 7f0cc7fff640 1 -- 172.21.15.83:0/585478067 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ccc009c10 con 0x7f0cbc0039b0 2024-03-06T07:33:12.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.490+0000 7f0cd7916640 1 -- 172.21.15.83:0/585478067 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0cbc0039b0 msgr2=0x7f0cbc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.490+0000 7f0cd7916640 1 --2- 172.21.15.83:0/585478067 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0cbc0039b0 0x7f0cbc003dd0 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc009fc0 tx=0x7f0cbc005830 comp rx=0 tx=0).stop 2024-03-06T07:33:12.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.490+0000 7f0cd7916640 1 -- 172.21.15.83:0/585478067 shutdown_connections 2024-03-06T07:33:12.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.490+0000 7f0cd7916640 1 --2- 172.21.15.83:0/585478067 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0cbc0039b0 0x7f0cbc003dd0 unknown :-1 s=CLOSED pgs=2075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.490+0000 7f0cd7916640 1 -- 172.21.15.83:0/585478067 >> 172.21.15.83:0/585478067 conn(0x7f0cd00d3d40 msgr2=0x7f0cd00d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:12.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.490+0000 7f0cd7916640 1 -- 172.21.15.83:0/585478067 shutdown_connections 2024-03-06T07:33:12.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.490+0000 7f0cd7916640 1 -- 172.21.15.83:0/585478067 wait complete. 2024-03-06T07:33:12.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.491+0000 7f0cd7916640 1 Processor -- start 2024-03-06T07:33:12.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.491+0000 7f0cd7916640 1 -- start start 2024-03-06T07:33:12.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.492+0000 7f0cd7916640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0cbc0039b0 0x7f0cd015f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.492+0000 7f0cd7916640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0cd015fba0 0x7f0cd00c9960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.492+0000 7f0cd7916640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0cd01600f0 0x7f0cd00c9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.492+0000 7f0cd7916640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd00d4860 con 0x7f0cd015fba0 2024-03-06T07:33:12.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.492+0000 7f0cd7916640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd00b0910 con 0x7f0cd01600f0 2024-03-06T07:33:12.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.492+0000 7f0cd7916640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd00d49e0 con 0x7f0cbc0039b0 2024-03-06T07:33:12.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.492+0000 7f0cd4eb1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0cd015fba0 0x7f0cd00c9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.492+0000 7f0cd4eb1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0cd015fba0 0x7f0cd00c9960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49320/0 (socket says 172.21.15.83:49320) 2024-03-06T07:33:12.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.492+0000 7f0cd4eb1640 1 -- 172.21.15.83:0/1499061546 learned_addr learned my addr 172.21.15.83:0/1499061546 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:33:12.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.492+0000 7f0cd56b2640 1 --2- 172.21.15.83:0/1499061546 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0cbc0039b0 0x7f0cd015f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.492+0000 7f0cd5eb3640 1 --2- 172.21.15.83:0/1499061546 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0cd01600f0 0x7f0cd00c9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.492+0000 7f0cd4eb1640 1 -- 172.21.15.83:0/1499061546 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0cbc0039b0 msgr2=0x7f0cd015f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.492+0000 7f0cd4eb1640 1 --2- 172.21.15.83:0/1499061546 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0cbc0039b0 0x7f0cd015f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.492+0000 7f0cd4eb1640 1 -- 172.21.15.83:0/1499061546 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0cd01600f0 msgr2=0x7f0cd00c9ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.492+0000 7f0cd4eb1640 1 --2- 172.21.15.83:0/1499061546 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0cd01600f0 0x7f0cd00c9ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.492+0000 7f0cd4eb1640 1 -- 172.21.15.83:0/1499061546 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd00c87f0 con 0x7f0cd015fba0 2024-03-06T07:33:12.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.492+0000 7f0cd4eb1640 1 --2- 172.21.15.83:0/1499061546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0cd015fba0 0x7f0cd00c9960 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f0cc000e830 tx=0x7f0cc000ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:12.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.492+0000 7f0cc5ffb640 1 -- 172.21.15.83:0/1499061546 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc000c870 con 0x7f0cd015fba0 2024-03-06T07:33:12.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.492+0000 7f0cc5ffb640 1 -- 172.21.15.83:0/1499061546 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cc0018070 con 0x7f0cd015fba0 2024-03-06T07:33:12.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.492+0000 7f0cd7916640 1 -- 172.21.15.83:0/1499061546 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cd00c8ad0 con 0x7f0cd015fba0 2024-03-06T07:33:12.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.492+0000 7f0cc5ffb640 1 -- 172.21.15.83:0/1499061546 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc000cc50 con 0x7f0cd015fba0 2024-03-06T07:33:12.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.493+0000 7f0cd7916640 1 -- 172.21.15.83:0/1499061546 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cd00c9090 con 0x7f0cd015fba0 2024-03-06T07:33:12.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.494+0000 7f0cd7916640 1 -- 172.21.15.83:0/1499061546 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cd00c7b20 con 0x7f0cd015fba0 2024-03-06T07:33:12.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.494+0000 7f0cc5ffb640 1 -- 172.21.15.83:0/1499061546 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0cc0013420 con 0x7f0cd015fba0 2024-03-06T07:33:12.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.494+0000 7f0cc5ffb640 1 --2- 172.21.15.83:0/1499061546 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0cb406b500 0x7f0cb406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.494+0000 7f0cc5ffb640 1 -- 172.21.15.83:0/1499061546 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f0cc0089b30 con 0x7f0cd015fba0 2024-03-06T07:33:12.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.494+0000 7f0cd56b2640 1 --2- 172.21.15.83:0/1499061546 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0cb406b500 0x7f0cb406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.495+0000 7f0cd56b2640 1 --2- 172.21.15.83:0/1499061546 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0cb406b500 0x7f0cb406d9c0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc036000 tx=0x7f0ccc02e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:12.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.498+0000 7f0cc5ffb640 1 -- 172.21.15.83:0/1499061546 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0cc0058c50 con 0x7f0cd015fba0 2024-03-06T07:33:12.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.670+0000 7f0cd7916640 1 -- 172.21.15.83:0/1499061546 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f0cd0068ff0 con 0x7f0cd015fba0 2024-03-06T07:33:12.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.674+0000 7f0cc5ffb640 1 -- 172.21.15.83:0/1499061546 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v34)=0 v34) v1 ==== 123+0+0 (secure 0 0 0) 0x7f0cc0058550 con 0x7f0cd015fba0 2024-03-06T07:33:12.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.675+0000 7f0cc5ffb640 1 -- 172.21.15.83:0/1499061546 <== mon.0 v2:172.21.15.83:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f0cc005c6a0 con 0x7f0cd015fba0 2024-03-06T07:33:12.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.677+0000 7f0cd7916640 1 -- 172.21.15.83:0/1499061546 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0cb406b500 msgr2=0x7f0cb406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.677+0000 7f0cd7916640 1 --2- 172.21.15.83:0/1499061546 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0cb406b500 0x7f0cb406d9c0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc036000 tx=0x7f0ccc02e000 comp rx=0 tx=0).stop 2024-03-06T07:33:12.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.677+0000 7f0cd7916640 1 -- 172.21.15.83:0/1499061546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0cd015fba0 msgr2=0x7f0cd00c9960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.677+0000 7f0cd7916640 1 --2- 172.21.15.83:0/1499061546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0cd015fba0 0x7f0cd00c9960 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f0cc000e830 tx=0x7f0cc000ecf0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.677+0000 7f0cd7916640 1 -- 172.21.15.83:0/1499061546 shutdown_connections 2024-03-06T07:33:12.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.677+0000 7f0cd7916640 1 --2- 172.21.15.83:0/1499061546 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f0cb406b500 0x7f0cb406d9c0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.677+0000 7f0cd7916640 1 --2- 172.21.15.83:0/1499061546 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0cd01600f0 0x7f0cd00c9ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.677+0000 7f0cd7916640 1 --2- 172.21.15.83:0/1499061546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0cd015fba0 0x7f0cd00c9960 unknown :-1 s=CLOSED pgs=2076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.677+0000 7f0cd7916640 1 --2- 172.21.15.83:0/1499061546 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0cbc0039b0 0x7f0cd015f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.677+0000 7f0cd7916640 1 -- 172.21.15.83:0/1499061546 >> 172.21.15.83:0/1499061546 conn(0x7f0cd00d3d40 msgr2=0x7f0cd006c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:12.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.677+0000 7f0cd7916640 1 -- 172.21.15.83:0/1499061546 shutdown_connections 2024-03-06T07:33:12.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.677+0000 7f0cd7916640 1 -- 172.21.15.83:0/1499061546 wait complete. 2024-03-06T07:33:12.689 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:33:12.690 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:12.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.787+0000 7f270bb56640 1 Processor -- start 2024-03-06T07:33:12.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.787+0000 7f270bb56640 1 -- start start 2024-03-06T07:33:12.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.788+0000 7f270bb56640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2704158920 0x7f270415c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.788+0000 7f270bb56640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f27040d14f0 0x7f270406f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.788+0000 7f270bb56640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f27040af6c0 0x7f27040afb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.788+0000 7f270bb56640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f270406fae0 con 0x7f27040af6c0 2024-03-06T07:33:12.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.788+0000 7f270bb56640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f270415f5f0 con 0x7f27040d14f0 2024-03-06T07:33:12.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.788+0000 7f270bb56640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f270415e740 con 0x7f2704158920 2024-03-06T07:33:12.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.788+0000 7f270a0f3640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f27040af6c0 0x7f27040afb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.788+0000 7f270a0f3640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f27040af6c0 0x7f27040afb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49322/0 (socket says 172.21.15.83:49322) 2024-03-06T07:33:12.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.788+0000 7f270a0f3640 1 -- 172.21.15.83:0/1099438840 learned_addr learned my addr 172.21.15.83:0/1099438840 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:33:12.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.788+0000 7f27098f2640 1 --2- 172.21.15.83:0/1099438840 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2704158920 0x7f270415c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.788+0000 7f27090f1640 1 --2- 172.21.15.83:0/1099438840 >> v2:172.21.15.183:3300/0 conn(0x7f27040d14f0 0x7f270406f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.788+0000 7f270a0f3640 1 -- 172.21.15.83:0/1099438840 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2704158920 msgr2=0x7f270415c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:12.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.788+0000 7f270a0f3640 1 --2- 172.21.15.83:0/1099438840 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2704158920 0x7f270415c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.788+0000 7f270a0f3640 1 -- 172.21.15.83:0/1099438840 >> v2:172.21.15.183:3300/0 conn(0x7f27040d14f0 msgr2=0x7f270406f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:12.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.788+0000 7f270a0f3640 1 --2- 172.21.15.83:0/1099438840 >> v2:172.21.15.183:3300/0 conn(0x7f27040d14f0 0x7f270406f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.788+0000 7f270a0f3640 1 -- 172.21.15.83:0/1099438840 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f270415e9e0 con 0x7f27040af6c0 2024-03-06T07:33:12.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.789+0000 7f270a0f3640 1 --2- 172.21.15.83:0/1099438840 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27040af6c0 0x7f27040afb50 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f2704067ce0 tx=0x7f26fc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=782543e8cc879fd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:12.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.789+0000 7f27088f0640 1 -- 172.21.15.83:0/1099438840 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26fc02ca70 con 0x7f27040af6c0 2024-03-06T07:33:12.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.789+0000 7f27088f0640 1 -- 172.21.15.83:0/1099438840 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f26fc02cc10 con 0x7f27040af6c0 2024-03-06T07:33:12.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.789+0000 7f27088f0640 1 -- 172.21.15.83:0/1099438840 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26fc0343f0 con 0x7f27040af6c0 2024-03-06T07:33:12.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.790+0000 7f270bb56640 1 -- 172.21.15.83:0/1099438840 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27040af6c0 msgr2=0x7f27040afb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.790+0000 7f270bb56640 1 --2- 172.21.15.83:0/1099438840 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27040af6c0 0x7f27040afb50 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f2704067ce0 tx=0x7f26fc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.790+0000 7f270bb56640 1 -- 172.21.15.83:0/1099438840 shutdown_connections 2024-03-06T07:33:12.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.790+0000 7f270bb56640 1 --2- 172.21.15.83:0/1099438840 >> v2:172.21.15.183:3300/0 conn(0x7f27040d14f0 0x7f270406f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.790+0000 7f270bb56640 1 --2- 172.21.15.83:0/1099438840 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27040af6c0 0x7f27040afb50 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.790+0000 7f270bb56640 1 --2- 172.21.15.83:0/1099438840 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2704158920 0x7f270415c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.790+0000 7f270bb56640 1 -- 172.21.15.83:0/1099438840 >> 172.21.15.83:0/1099438840 conn(0x7f27040d0960 msgr2=0x7f27040d0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:12.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.790+0000 7f270bb56640 1 -- 172.21.15.83:0/1099438840 shutdown_connections 2024-03-06T07:33:12.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.790+0000 7f270bb56640 1 -- 172.21.15.83:0/1099438840 wait complete. 2024-03-06T07:33:12.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.791+0000 7f270bb56640 1 Processor -- start 2024-03-06T07:33:12.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.791+0000 7f270bb56640 1 -- start start 2024-03-06T07:33:12.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.792+0000 7f270bb56640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f27040af6c0 0x7f2704151f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.792+0000 7f270bb56640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f27040d14f0 0x7f270415b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.792+0000 7f270bb56640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2704158920 0x7f270415bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.792+0000 7f270bb56640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f270415f280 con 0x7f2704158920 2024-03-06T07:33:12.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.792+0000 7f270bb56640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f270415efa0 con 0x7f27040d14f0 2024-03-06T07:33:12.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.792+0000 7f270bb56640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f270415b400 con 0x7f27040af6c0 2024-03-06T07:33:12.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.792+0000 7f270a0f3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2704158920 0x7f270415bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.792+0000 7f270a0f3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2704158920 0x7f270415bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49338/0 (socket says 172.21.15.83:49338) 2024-03-06T07:33:12.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.792+0000 7f270a0f3640 1 -- 172.21.15.83:0/1493649595 learned_addr learned my addr 172.21.15.83:0/1493649595 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-06T07:33:12.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.792+0000 7f27098f2640 1 --2- 172.21.15.83:0/1493649595 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f27040af6c0 0x7f2704151f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.792+0000 7f27090f1640 1 --2- 172.21.15.83:0/1493649595 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f27040d14f0 0x7f270415b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.792+0000 7f270a0f3640 1 -- 172.21.15.83:0/1493649595 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f27040af6c0 msgr2=0x7f2704151f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.792+0000 7f270a0f3640 1 --2- 172.21.15.83:0/1493649595 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f27040af6c0 0x7f2704151f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.792+0000 7f270a0f3640 1 -- 172.21.15.83:0/1493649595 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f27040d14f0 msgr2=0x7f270415b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.792+0000 7f270a0f3640 1 --2- 172.21.15.83:0/1493649595 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f27040d14f0 0x7f270415b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.792+0000 7f270a0f3640 1 -- 172.21.15.83:0/1493649595 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f270415c3b0 con 0x7f2704158920 2024-03-06T07:33:12.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.792+0000 7f270a0f3640 1 --2- 172.21.15.83:0/1493649595 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2704158920 0x7f270415bd30 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f26fc02bf10 tx=0x7f26fc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:12.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.792+0000 7f26fa7fc640 1 -- 172.21.15.83:0/1493649595 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26fc043020 con 0x7f2704158920 2024-03-06T07:33:12.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.792+0000 7f26fa7fc640 1 -- 172.21.15.83:0/1493649595 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f26fc03e450 con 0x7f2704158920 2024-03-06T07:33:12.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.792+0000 7f26fa7fc640 1 -- 172.21.15.83:0/1493649595 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26fc009510 con 0x7f2704158920 2024-03-06T07:33:12.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.792+0000 7f270bb56640 1 -- 172.21.15.83:0/1493649595 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f270415c690 con 0x7f2704158920 2024-03-06T07:33:12.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.792+0000 7f270bb56640 1 -- 172.21.15.83:0/1493649595 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f270415cc50 con 0x7f2704158920 2024-03-06T07:33:12.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.793+0000 7f270bb56640 1 -- 172.21.15.83:0/1493649595 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f270404f330 con 0x7f2704158920 2024-03-06T07:33:12.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.794+0000 7f26fa7fc640 1 -- 172.21.15.83:0/1493649595 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f26fc046020 con 0x7f2704158920 2024-03-06T07:33:12.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.794+0000 7f26fa7fc640 1 --2- 172.21.15.83:0/1493649595 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f26e806b4d0 0x7f26e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.794+0000 7f26fa7fc640 1 -- 172.21.15.83:0/1493649595 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f26fc0ac6d0 con 0x7f2704158920 2024-03-06T07:33:12.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.794+0000 7f27098f2640 1 --2- 172.21.15.83:0/1493649595 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f26e806b4d0 0x7f26e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.795+0000 7f27098f2640 1 --2- 172.21.15.83:0/1493649595 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f26e806b4d0 0x7f26e806d990 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f2700009aa0 tx=0x7f2700009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:12.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:12.798+0000 7f26fa7fc640 1 -- 172.21.15.83:0/1493649595 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f26fc04d030 con 0x7f2704158920 2024-03-06T07:33:13.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:13.006+0000 7f270bb56640 1 -- 172.21.15.83:0/1493649595 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2704072b80 con 0x7f2704158920 2024-03-06T07:33:13.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:13.007+0000 7f26fa7fc640 1 -- 172.21.15.83:0/1493649595 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+1326 (secure 0 0 0) 0x7f26fc07b110 con 0x7f2704158920 2024-03-06T07:33:13.009 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-06T07:33:13.009 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":85,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9825,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/3006010640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6835","nonce":3006010640},{"type":"v1","addr":"172.21.15.83:6837","nonce":3006010640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[]} 2024-03-06T07:33:13.009 INFO:teuthology.orchestra.run.smithi083.stderr:dumped fsmap epoch 85 2024-03-06T07:33:13.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:13.009+0000 7f270bb56640 1 -- 172.21.15.83:0/1493649595 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f26e806b4d0 msgr2=0x7f26e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:13.009+0000 7f270bb56640 1 --2- 172.21.15.83:0/1493649595 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f26e806b4d0 0x7f26e806d990 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f2700009aa0 tx=0x7f2700009190 comp rx=0 tx=0).stop 2024-03-06T07:33:13.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:13.009+0000 7f270bb56640 1 -- 172.21.15.83:0/1493649595 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2704158920 msgr2=0x7f270415bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:13.009+0000 7f270bb56640 1 --2- 172.21.15.83:0/1493649595 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2704158920 0x7f270415bd30 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f26fc02bf10 tx=0x7f26fc004a40 comp rx=0 tx=0).stop 2024-03-06T07:33:13.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:13.009+0000 7f270bb56640 1 -- 172.21.15.83:0/1493649595 shutdown_connections 2024-03-06T07:33:13.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:13.009+0000 7f270bb56640 1 --2- 172.21.15.83:0/1493649595 >> [v2:172.21.15.83:6832/2272295342,v1:172.21.15.83:6833/2272295342] conn(0x7f26e806b4d0 0x7f26e806d990 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:13.009+0000 7f270bb56640 1 --2- 172.21.15.83:0/1493649595 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2704158920 0x7f270415bd30 unknown :-1 s=CLOSED pgs=2078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:13.009+0000 7f270bb56640 1 --2- 172.21.15.83:0/1493649595 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f27040d14f0 0x7f270415b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:13.009+0000 7f270bb56640 1 --2- 172.21.15.83:0/1493649595 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f27040af6c0 0x7f2704151f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:13.009+0000 7f270bb56640 1 -- 172.21.15.83:0/1493649595 >> 172.21.15.83:0/1493649595 conn(0x7f27040d0960 msgr2=0x7f270406afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:13.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:13.009+0000 7f270bb56640 1 -- 172.21.15.83:0/1493649595 shutdown_connections 2024-03-06T07:33:13.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-06T07:33:13.009+0000 7f270bb56640 1 -- 172.21.15.83:0/1493649595 wait complete. 2024-03-06T07:33:13.021 DEBUG:tasks.cephfs.filesystem:already dead... 2024-03-06T07:33:13.022 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-06T07:33:13.022 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:33:13.022 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:13.030 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:33:13.030 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:33:13.030 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:13.080 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:33:13.081 ERROR:tasks.ceph.mds.b:tried to stop a non-running daemon 2024-03-06T07:33:13.085 ERROR:tasks.ceph.mds.d:tried to stop a non-running daemon 2024-03-06T07:33:13.091 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-06T07:33:13.091 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-06T07:33:13.091 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:13.154 INFO:tasks.ceph.osd.0:Stopped 2024-03-06T07:33:13.154 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-06T07:33:13.154 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:13.222 INFO:tasks.ceph.osd.1:Stopped 2024-03-06T07:33:13.222 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-06T07:33:13.222 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:13.292 INFO:tasks.ceph.osd.2:Stopped 2024-03-06T07:33:13.292 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-06T07:33:13.293 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:13.367 INFO:tasks.ceph.osd.3:Stopped 2024-03-06T07:33:13.368 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-06T07:33:13.368 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:13.400 INFO:tasks.ceph.osd.4:Stopped 2024-03-06T07:33:13.400 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-06T07:33:13.400 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:13.478 INFO:tasks.ceph.osd.5:Stopped 2024-03-06T07:33:13.478 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-06T07:33:13.479 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:13.546 INFO:tasks.ceph.osd.6:Stopped 2024-03-06T07:33:13.547 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-06T07:33:13.547 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:13.617 INFO:tasks.ceph.osd.7:Stopped 2024-03-06T07:33:13.617 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-06T07:33:13.618 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-06T07:33:13.618 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:13.645 INFO:tasks.ceph.mgr.y:Stopped 2024-03-06T07:33:13.645 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-06T07:33:13.645 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:13.686 INFO:tasks.ceph.mgr.x:Stopped 2024-03-06T07:33:13.686 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-06T07:33:13.686 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-06T07:33:13.686 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:13.697 INFO:tasks.ceph.mon.a:Stopped 2024-03-06T07:33:13.698 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-06T07:33:13.698 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:13.738 INFO:tasks.ceph.mon.b:Stopped 2024-03-06T07:33:13.739 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-06T07:33:13.739 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:13.790 INFO:tasks.ceph.mon.c:Stopped 2024-03-06T07:33:13.791 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-06T07:33:13.791 DEBUG:teuthology.orchestra.run.smithi083:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-03-06T07:33:13.824 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi083.front.sepia.ceph.com 2024-03-06T07:33:13.824 DEBUG:teuthology.orchestra.run.smithi083:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-06T07:33:14.045 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi083.front.sepia.ceph.com 2024-03-06T07:33:14.045 DEBUG:teuthology.orchestra.run.smithi083:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-06T07:33:14.147 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi083.front.sepia.ceph.com 2024-03-06T07:33:14.147 DEBUG:teuthology.orchestra.run.smithi083:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-06T07:33:14.198 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi083.front.sepia.ceph.com 2024-03-06T07:33:14.198 DEBUG:teuthology.orchestra.run.smithi083:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-06T07:33:14.250 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-06T07:33:14.250 DEBUG:teuthology.orchestra.run.smithi183:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-06T07:33:14.597 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-06T07:33:14.597 DEBUG:teuthology.orchestra.run.smithi183:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-06T07:33:14.673 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-06T07:33:14.673 DEBUG:teuthology.orchestra.run.smithi183:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-06T07:33:14.733 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-06T07:33:14.733 DEBUG:teuthology.orchestra.run.smithi183:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-06T07:33:14.807 INFO:tasks.ceph:Archiving mon data... 2024-03-06T07:33:14.808 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583062/data/mon.a.tgz 2024-03-06T07:33:14.809 DEBUG:teuthology.orchestra.run.smithi083:> mktemp 2024-03-06T07:33:14.824 INFO:teuthology.orchestra.run.smithi083.stdout:/tmp/tmp.ZN9sbD76vT 2024-03-06T07:33:14.825 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.ZN9sbD76vT 2024-03-06T07:33:14.977 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0666 /tmp/tmp.ZN9sbD76vT 2024-03-06T07:33:15.062 DEBUG:teuthology.orchestra.remote:smithi083:/tmp/tmp.ZN9sbD76vT is 394KB 2024-03-06T07:33:15.158 DEBUG:teuthology.orchestra.run.smithi083:> rm -fr /tmp/tmp.ZN9sbD76vT 2024-03-06T07:33:15.173 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583062/data/mon.b.tgz 2024-03-06T07:33:15.173 DEBUG:teuthology.orchestra.run.smithi183:> mktemp 2024-03-06T07:33:15.191 INFO:teuthology.orchestra.run.smithi183.stdout:/tmp/tmp.RFtGqJUSpO 2024-03-06T07:33:15.191 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.RFtGqJUSpO 2024-03-06T07:33:15.355 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0666 /tmp/tmp.RFtGqJUSpO 2024-03-06T07:33:15.441 DEBUG:teuthology.orchestra.remote:smithi183:/tmp/tmp.RFtGqJUSpO is 399KB 2024-03-06T07:33:15.540 DEBUG:teuthology.orchestra.run.smithi183:> rm -fr /tmp/tmp.RFtGqJUSpO 2024-03-06T07:33:15.554 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583062/data/mon.c.tgz 2024-03-06T07:33:15.555 DEBUG:teuthology.orchestra.run.smithi183:> mktemp 2024-03-06T07:33:15.608 INFO:teuthology.orchestra.run.smithi183.stdout:/tmp/tmp.JGl3OvSjRf 2024-03-06T07:33:15.609 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.JGl3OvSjRf 2024-03-06T07:33:15.749 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0666 /tmp/tmp.JGl3OvSjRf 2024-03-06T07:33:15.834 DEBUG:teuthology.orchestra.remote:smithi183:/tmp/tmp.JGl3OvSjRf is 399KB 2024-03-06T07:33:15.901 DEBUG:teuthology.orchestra.run.smithi183:> rm -fr /tmp/tmp.JGl3OvSjRf 2024-03-06T07:33:15.915 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-06T07:33:15.915 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-06T07:33:15.918 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-06T07:33:16.002 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-06T07:33:16.035 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-06T07:33:16.035 INFO:tasks.ceph:Archiving crash dumps... 2024-03-06T07:33:16.037 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583062/remote/smithi083/crash 2024-03-06T07:33:16.038 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-06T07:33:16.071 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583062/remote/smithi183/crash 2024-03-06T07:33:16.072 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-06T07:33:16.101 INFO:tasks.ceph:Shutting down logrotate 2024-03-06T07:33:16.101 DEBUG:teuthology.orchestra.run.smithi083:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:16.116 DEBUG:teuthology.orchestra.run.smithi183:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:16.193 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:16.221 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:16.247 INFO:tasks.ceph:Compressing logs... 2024-03-06T07:33:16.247 DEBUG:teuthology.orchestra.run.smithi083:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-06T07:33:16.263 DEBUG:teuthology.orchestra.run.smithi183:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-06T07:33:16.287 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85835.log 2024-03-06T07:33:16.287 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70903.log 2024-03-06T07:33:16.332 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38335.log 2024-03-06T07:33:16.332 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-06T07:33:16.332 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.38335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38135.log 2024-03-06T07:33:16.332 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38295.log 2024-03-06T07:33:16.332 INFO:teuthology.orchestra.run.smithi183.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38335.log.gz 2024-03-06T07:33:16.332 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.38135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.44426.log 2024-03-06T07:33:16.332 INFO:teuthology.orchestra.run.smithi183.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38135.log.gz 2024-03-06T07:33:16.333 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.38295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.44879.log 2024-03-06T07:33:16.333 INFO:teuthology.orchestra.run.smithi183.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38295.log.gz 2024-03-06T07:33:16.333 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.1.44426.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-06T07:33:16.333 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.3.44879.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-03-06T07:33:16.333 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38055.log 2024-03-06T07:33:16.333 INFO:teuthology.orchestra.run.smithi183.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.1.44426.log.gz 2024-03-06T07:33:16.333 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-mds.b.log: 90.8% -- replaced with /var/log/ceph/ceph-client.3.44879.log.gz 2024-03-06T07:33:16.333 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-06T07:33:16.333 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.38055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38175.log 2024-03-06T07:33:16.334 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-osd.4.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38055.log.gz 2024-03-06T07:33:16.334 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38095.log 2024-03-06T07:33:16.334 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.38175.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-06T07:33:16.334 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.38095.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38175.log.gz 2024-03-06T07:33:16.334 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-06T07:33:16.334 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-osd.7.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38095.log.gz 2024-03-06T07:33:16.334 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-06T07:33:16.334 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-06T07:33:16.334 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-06T07:33:16.335 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph.log: 93.3% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-03-06T07:33:16.335 INFO:teuthology.orchestra.run.smithi183.stderr: 93.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-06T07:33:16.335 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38255.log 2024-03-06T07:33:16.335 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-03-06T07:33:16.335 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.38255.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38255.log.gz 2024-03-06T07:33:16.335 INFO:teuthology.orchestra.run.smithi183.stderr: 92.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-06T07:33:16.335 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38215.log 2024-03-06T07:33:16.335 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-06T07:33:16.335 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.38215.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38215.log.gz 2024-03-06T07:33:16.335 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-mon.c.log: 93.2% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-03-06T07:33:16.336 INFO:teuthology.orchestra.run.smithi183.stderr: 94.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-06T07:33:16.618 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.85835.log: -- /var/log/ceph/ceph-client.admin.81606.log 2024-03-06T07:33:16.618 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48710.log 2024-03-06T07:33:16.619 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.70903.log: /var/log/ceph/ceph-client.admin.81606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86681.log 2024-03-06T07:33:16.619 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.85835.log.gz 2024-03-06T07:33:16.619 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.70903.log.gz 2024-03-06T07:33:16.619 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48710.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.81606.log.gz 2024-03-06T07:33:16.619 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76258.log 2024-03-06T07:33:16.619 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.86681.log: gzip 88.3% -5 -- replaced with /var/log/ceph/ceph-client.admin.48710.log.gz 2024-03-06T07:33:16.619 INFO:teuthology.orchestra.run.smithi083.stderr: --verbose -- /var/log/ceph/ceph-client.admin.55966.log 2024-03-06T07:33:16.620 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.76258.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.86681.log.gz 2024-03-06T07:33:16.620 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57128.log 2024-03-06T07:33:16.620 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.55966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69249.log 2024-03-06T07:33:16.620 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.76258.log.gz 2024-03-06T07:33:16.620 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.57128.log: gzip -5 --verbose -- 89.2% /var/log/ceph/ceph-client.0.79136.log 2024-03-06T07:33:16.620 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55966.log.gz 2024-03-06T07:33:16.620 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57128.log.gz 2024-03-06T07:33:16.620 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.69249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71155.log 2024-03-06T07:33:16.621 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.0.79136.log: gzip 88.7% -5 -- replaced with /var/log/ceph/ceph-client.admin.69249.log.gz 2024-03-06T07:33:16.621 INFO:teuthology.orchestra.run.smithi083.stderr: --verbose -- /var/log/ceph/ceph-client.admin.70735.log 2024-03-06T07:33:16.621 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.71155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65497.log 2024-03-06T07:33:16.621 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71830.log 2024-03-06T07:33:16.621 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.70735.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.71155.log.gz 2024-03-06T07:33:16.621 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.65497.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70735.log.gz 2024-03-06T07:33:16.621 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54534.log 2024-03-06T07:33:16.621 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.71830.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-06T07:33:16.621 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.65497.log.gz 2024-03-06T07:33:16.622 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.54534.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.71830.log.gz 2024-03-06T07:33:16.622 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46061.log 2024-03-06T07:33:16.622 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54534.log.gz 2024-03-06T07:33:16.622 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83407.log 2024-03-06T07:33:16.622 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.46061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85430.log 2024-03-06T07:33:16.622 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.83407.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46061.log.gz 2024-03-06T07:33:16.622 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49697.log 2024-03-06T07:33:16.622 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.85430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48374.log 2024-03-06T07:33:16.623 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.83407.log.gz 2024-03-06T07:33:16.623 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.85430.log.gz 2024-03-06T07:33:16.623 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50243.log 2024-03-06T07:33:16.623 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49697.log.gz 2024-03-06T07:33:16.623 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86807.log 2024-03-06T07:33:16.623 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.50243.log: gzip -5 --verbose -- 87.9% /var/log/ceph/ceph-client.admin.43925.log 2024-03-06T07:33:16.623 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48374.log.gz 2024-03-06T07:33:16.623 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.86807.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50243.log.gz 2024-03-06T07:33:16.623 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45618.log 2024-03-06T07:33:16.624 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.43925.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.86807.log.gz 2024-03-06T07:33:16.624 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81564.log 2024-03-06T07:33:16.624 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45618.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43925.log.gz 2024-03-06T07:33:16.624 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85309.log 2024-03-06T07:33:16.624 INFO:teuthology.orchestra.run.smithi083.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45618.log.gz 2024-03-06T07:33:16.627 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.81564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59483.log 2024-03-06T07:33:16.627 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.85309.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.81564.log.gz 2024-03-06T07:33:16.627 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52637.log 2024-03-06T07:33:16.627 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.59483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61978.log 2024-03-06T07:33:16.627 INFO:teuthology.orchestra.run.smithi083.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.85309.log.gz 2024-03-06T07:33:16.627 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.59483.log.gz 2024-03-06T07:33:16.628 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.52637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68433.log 2024-03-06T07:33:16.628 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.61978.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52637.log.gz 2024-03-06T07:33:16.628 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58076.log 2024-03-06T07:33:16.628 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.68433.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.61978.log.gz 2024-03-06T07:33:16.628 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69415.log 2024-03-06T07:33:16.628 INFO:teuthology.orchestra.run.smithi083.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.68433.log.gz 2024-03-06T07:33:16.628 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.58076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64761.log 2024-03-06T07:33:16.629 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.69415.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58076.log.gz 2024-03-06T07:33:16.629 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45399.log 2024-03-06T07:33:16.629 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.64761.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.69415.log.gz 2024-03-06T07:33:16.629 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62020.log 2024-03-06T07:33:16.629 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.64761.log.gz 2024-03-06T07:33:16.629 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63986.log 2024-03-06T07:33:16.629 INFO:teuthology.orchestra.run.smithi083.stderr: 94.5% -- replaced with /var/log/ceph/ceph-client.0.79136.log.gz 2024-03-06T07:33:16.629 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.62020.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45399.log.gz 2024-03-06T07:33:16.629 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67487.log 2024-03-06T07:33:16.630 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.63986.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62020.log.gz 2024-03-06T07:33:16.630 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76620.log 2024-03-06T07:33:16.630 INFO:teuthology.orchestra.run.smithi083.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.63986.log.gz 2024-03-06T07:33:16.630 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.67487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77849.log 2024-03-06T07:33:16.630 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.76620.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.67487.log.gz 2024-03-06T07:33:16.630 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57609.log 2024-03-06T07:33:16.630 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.77849.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.76620.log.gz 2024-03-06T07:33:16.630 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.63034.log 2024-03-06T07:33:16.631 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.77849.log.gz 2024-03-06T07:33:16.631 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.57609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68261.log 2024-03-06T07:33:16.631 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.0.63034.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57609.log.gz 2024-03-06T07:33:16.631 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84139.log 2024-03-06T07:33:16.631 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.68261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72168.log 2024-03-06T07:33:16.631 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.84139.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.68261.log.gz 2024-03-06T07:33:16.631 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49529.log 2024-03-06T07:33:16.631 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.72168.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.84139.log.gz 2024-03-06T07:33:16.631 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44507.log 2024-03-06T07:33:16.632 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.72168.log.gz 2024-03-06T07:33:16.632 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51240.log 2024-03-06T07:33:16.632 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49529.log: /var/log/ceph/ceph-client.admin.44507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57567.log 2024-03-06T07:33:16.632 INFO:teuthology.orchestra.run.smithi083.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49529.log.gz 2024-03-06T07:33:16.632 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.51240.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44507.log.gz 2024-03-06T07:33:16.632 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73769.log 2024-03-06T07:33:16.632 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51240.log.gz 2024-03-06T07:33:16.632 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51570.log 2024-03-06T07:33:16.633 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.57567.log: /var/log/ceph/ceph-client.admin.73769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54405.log 2024-03-06T07:33:16.633 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57567.log.gz 2024-03-06T07:33:16.633 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.51570.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.73769.log.gz 2024-03-06T07:33:16.633 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71654.log 2024-03-06T07:33:16.633 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48146.log 2024-03-06T07:33:16.633 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.54405.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51570.log.gz 2024-03-06T07:33:16.633 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.71654.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54405.log.gz 2024-03-06T07:33:16.633 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76010.log 2024-03-06T07:33:16.633 INFO:teuthology.orchestra.run.smithi083.stderr:gzip 88.7% -5 -- replaced with /var/log/ceph/ceph-client.admin.71654.log.gz --verbose 2024-03-06T07:33:16.634 INFO:teuthology.orchestra.run.smithi083.stderr: -- /var/log/ceph/ceph-client.admin.76815.log 2024-03-06T07:33:16.634 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48146.log: /var/log/ceph/ceph-client.admin.76010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73937.log 2024-03-06T07:33:16.634 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48146.log.gz 2024-03-06T07:33:16.634 INFO:teuthology.orchestra.run.smithi083.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.76010.log.gz 2024-03-06T07:33:16.634 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.76815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52571.log 2024-03-06T07:33:16.634 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.73937.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.76815.log.gz 2024-03-06T07:33:16.634 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48458.log 2024-03-06T07:33:16.634 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.73937.log.gz 2024-03-06T07:33:16.635 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65623.log 2024-03-06T07:33:16.635 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.52571.log: /var/log/ceph/ceph-client.admin.48458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64364.log 2024-03-06T07:33:16.635 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.65623.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48458.log.gz 2024-03-06T07:33:16.635 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53714.log 2024-03-06T07:33:16.635 INFO:teuthology.orchestra.run.smithi083.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.52571.log.gz 2024-03-06T07:33:16.635 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50537.log 2024-03-06T07:33:16.635 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.64364.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.65623.log.gz 2024-03-06T07:33:16.635 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.53714.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.64364.log.gz 2024-03-06T07:33:16.636 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81480.log 2024-03-06T07:33:16.636 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53714.log.gz 2024-03-06T07:33:16.636 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49241.log 2024-03-06T07:33:16.636 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.50537.log: /var/log/ceph/ceph-client.admin.81480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84535.log 2024-03-06T07:33:16.636 INFO:teuthology.orchestra.run.smithi083.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50537.log.gz 2024-03-06T07:33:16.636 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49241.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.81480.log.gz 2024-03-06T07:33:16.636 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82045.log 2024-03-06T07:33:16.636 INFO:teuthology.orchestra.run.smithi083.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.49241.log.gz 2024-03-06T07:33:16.637 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.84535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84619.log 2024-03-06T07:33:16.637 INFO:teuthology.orchestra.run.smithi083.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.84535.log.gz 2024-03-06T07:33:16.637 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72038.log 2024-03-06T07:33:16.637 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.82045.log: gzip -5 --verbose/var/log/ceph/ceph-client.admin.84619.log: -- /var/log/ceph/ceph-client.admin.79474.log 2024-03-06T07:33:16.637 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.82045.log.gz 2024-03-06T07:33:16.637 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.72038.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.84619.log.gz 2024-03-06T07:33:16.637 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44717.log 2024-03-06T07:33:16.637 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.72038.log.gz 2024-03-06T07:33:16.638 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50453.log 2024-03-06T07:33:16.638 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.79474.log: /var/log/ceph/ceph-client.admin.44717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63749.log 2024-03-06T07:33:16.638 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.79474.log.gz 2024-03-06T07:33:16.638 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.50453.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44717.log.gz 2024-03-06T07:33:16.638 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80608.log 2024-03-06T07:33:16.638 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50453.log.gz 2024-03-06T07:33:16.638 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.63749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54263.log 2024-03-06T07:33:16.638 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.63749.log.gz 2024-03-06T07:33:16.639 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44885.log 2024-03-06T07:33:16.639 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.80608.log: /var/log/ceph/ceph-client.admin.54263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77422.log 2024-03-06T07:33:16.639 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9%/var/log/ceph/ceph-client.admin.44885.log: -- replaced with /var/log/ceph/ceph-client.admin.80608.log.gz 2024-03-06T07:33:16.639 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.54263.log.gz 2024-03-06T07:33:16.639 INFO:teuthology.orchestra.run.smithi083.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49939.log 2024-03-06T07:33:16.639 INFO:teuthology.orchestra.run.smithi083.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44885.log.gz 2024-03-06T07:33:16.639 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.77422.log: /var/log/ceph/ceph-client.admin.55900.log 2024-03-06T07:33:16.639 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52021.log 2024-03-06T07:33:16.640 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49939.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.77422.log.gz 2024-03-06T07:33:16.640 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79516.log 2024-03-06T07:33:16.640 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.55900.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49939.log.gz 2024-03-06T07:33:16.640 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.52021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83449.log 2024-03-06T07:33:16.640 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52021.log.gz 2024-03-06T07:33:16.640 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.79516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58471.log 2024-03-06T07:33:16.640 INFO:teuthology.orchestra.run.smithi083.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.55900.log.gz 2024-03-06T07:33:16.640 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51443.log 2024-03-06T07:33:16.641 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.83449.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.79516.log.gz 2024-03-06T07:33:16.641 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.58471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55392.log 2024-03-06T07:33:16.641 INFO:teuthology.orchestra.run.smithi083.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.83449.log.gz 2024-03-06T07:33:16.641 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.51443.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.58471.log.gz 2024-03-06T07:33:16.641 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87017.log 2024-03-06T07:33:16.641 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.55392.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51443.log.gz 2024-03-06T07:33:16.641 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49445.log 2024-03-06T07:33:16.641 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55392.log.gz 2024-03-06T07:33:16.641 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71954.log 2024-03-06T07:33:16.642 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.87017.log: /var/log/ceph/ceph-client.admin.49445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57357.log 2024-03-06T07:33:16.642 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.87017.log.gz 2024-03-06T07:33:16.642 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.71954.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49445.log.gz 2024-03-06T07:33:16.642 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74442.log 2024-03-06T07:33:16.642 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.71954.log.gz 2024-03-06T07:33:16.642 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56555.log 2024-03-06T07:33:16.642 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.57357.log: /var/log/ceph/ceph-client.admin.74442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52151.log 2024-03-06T07:33:16.642 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57357.log.gz 2024-03-06T07:33:16.643 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.56555.log: 89.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.74442.log.gz -5 2024-03-06T07:33:16.643 INFO:teuthology.orchestra.run.smithi083.stderr: --verbose -- /var/log/ceph/ceph-client.admin.55085.log 2024-03-06T07:33:16.643 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.52151.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56555.log.gz 2024-03-06T07:33:16.643 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44136.log 2024-03-06T07:33:16.643 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.55085.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52151.log.gz 2024-03-06T07:33:16.643 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48752.log 2024-03-06T07:33:16.643 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55085.log.gz 2024-03-06T07:33:16.643 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44293.log 2024-03-06T07:33:16.644 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68219.log 2024-03-06T07:33:16.644 INFO:teuthology.orchestra.run.smithi083.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44136.log.gz 2024-03-06T07:33:16.644 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44293.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48752.log.gz 2024-03-06T07:33:16.644 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82129.log 2024-03-06T07:33:16.644 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.68219.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44293.log.gz 2024-03-06T07:33:16.644 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65316.log 2024-03-06T07:33:16.644 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.68219.log.gz 2024-03-06T07:33:16.644 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53328.log 2024-03-06T07:33:16.645 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.82129.log: /var/log/ceph/ceph-client.admin.65316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80086.log 2024-03-06T07:33:16.645 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.82129.log.gz 2024-03-06T07:33:16.645 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.53328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61114.log 2024-03-06T07:33:16.645 INFO:teuthology.orchestra.run.smithi083.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.65316.log.gz 2024-03-06T07:33:16.645 INFO:teuthology.orchestra.run.smithi083.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.53328.log.gz 2024-03-06T07:33:16.645 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.80086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82003.log 2024-03-06T07:33:16.645 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.80086.log.gz 2024-03-06T07:33:16.645 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75419.log 2024-03-06T07:33:16.645 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.61114.log: 92.7% -- replaced with /var/log/ceph/ceph-client.0.63034.log.gz 2024-03-06T07:33:16.646 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.82003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45619.log 2024-03-06T07:33:16.646 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61114.log.gz 2024-03-06T07:33:16.646 INFO:teuthology.orchestra.run.smithi083.stderr: 89.3%/var/log/ceph/ceph-client.admin.75419.log: -- replaced with /var/log/ceph/ceph-client.admin.82003.log.gz 2024-03-06T07:33:16.646 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80356.log 2024-03-06T07:33:16.646 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47936.log 2024-03-06T07:33:16.646 INFO:teuthology.orchestra.run.smithi083.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.75419.log.gz 2024-03-06T07:33:16.646 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.80356.log: gzip 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45619.log.gz -5 2024-03-06T07:33:16.646 INFO:teuthology.orchestra.run.smithi083.stderr: --verbose -- /var/log/ceph/ceph-client.admin.38073.log 2024-03-06T07:33:16.647 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.47936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58120.log 2024-03-06T07:33:16.647 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.80356.log.gz 2024-03-06T07:33:16.647 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47936.log.gz/var/log/ceph/ceph-client.admin.38073.log: 2024-03-06T07:33:16.647 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83113.log 2024-03-06T07:33:16.647 INFO:teuthology.orchestra.run.smithi083.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38073.log.gz 2024-03-06T07:33:16.647 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.58120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55480.log 2024-03-06T07:33:16.647 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.83113.log: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.58120.log.gz -5 2024-03-06T07:33:16.647 INFO:teuthology.orchestra.run.smithi083.stderr: --verbose -- /var/log/ceph/ceph-client.admin.86849.log 2024-03-06T07:33:16.648 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.55480.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.83113.log.gz 2024-03-06T07:33:16.648 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45403.log 2024-03-06T07:33:16.648 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55480.log.gz 2024-03-06T07:33:16.648 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.86849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81071.log 2024-03-06T07:33:16.648 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83239.log 2024-03-06T07:33:16.648 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.86849.log.gz 2024-03-06T07:33:16.648 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.81071.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45403.log.gz 2024-03-06T07:33:16.648 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79293.log 2024-03-06T07:33:16.649 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.81071.log.gz 2024-03-06T07:33:16.649 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.83239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65581.log 2024-03-06T07:33:16.649 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.79293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58215.log 2024-03-06T07:33:16.649 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.83239.log.gz 2024-03-06T07:33:16.649 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.65581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47852.log 2024-03-06T07:33:16.649 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.79293.log.gz 2024-03-06T07:33:16.649 INFO:teuthology.orchestra.run.smithi083.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.65581.log.gz 2024-03-06T07:33:16.650 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.58215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53909.log 2024-03-06T07:33:16.650 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.47852.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58215.log.gz 2024-03-06T07:33:16.650 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81648.log 2024-03-06T07:33:16.650 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.53909.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47852.log.gz 2024-03-06T07:33:16.650 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44675.log 2024-03-06T07:33:16.650 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53909.log.gz 2024-03-06T07:33:16.650 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.81648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69569.log 2024-03-06T07:33:16.650 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44675.log: 89.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.81648.log.gz 2024-03-06T07:33:16.651 INFO:teuthology.orchestra.run.smithi083.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.48668.log 2024-03-06T07:33:16.651 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.69569.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.44675.log.gz 2024-03-06T07:33:16.651 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76518.log 2024-03-06T07:33:16.651 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.69569.log.gz 2024-03-06T07:33:16.651 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55350.log 2024-03-06T07:33:16.651 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48668.log.gz 2024-03-06T07:33:16.651 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.76518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80734.log 2024-03-06T07:33:16.651 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.55350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61343.log 2024-03-06T07:33:16.651 INFO:teuthology.orchestra.run.smithi083.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.76518.log.gz 2024-03-06T07:33:16.652 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55350.log.gz 2024-03-06T07:33:16.652 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.80734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74484.log 2024-03-06T07:33:16.652 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.80734.log.gz 2024-03-06T07:33:16.652 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.61343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76301.log 2024-03-06T07:33:16.652 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.74484.log: gzip -5 --verbose 88.6% -- -- replaced with /var/log/ceph/ceph-client.admin.61343.log.gz /var/log/ceph/ceph-client.admin.54095.log 2024-03-06T07:33:16.652 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-06T07:33:16.652 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.76301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67276.log 2024-03-06T07:33:16.652 INFO:teuthology.orchestra.run.smithi083.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.74484.log.gz/var/log/ceph/ceph-client.admin.54095.log: 2024-03-06T07:33:16.652 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62583.log 2024-03-06T07:33:16.653 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.67276.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54095.log.gz 2024-03-06T07:33:16.653 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73543.log 2024-03-06T07:33:16.653 INFO:teuthology.orchestra.run.smithi083.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.76301.log.gz 2024-03-06T07:33:16.653 INFO:teuthology.orchestra.run.smithi083.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.67276.log.gz 2024-03-06T07:33:16.653 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.62583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63251.log 2024-03-06T07:33:16.653 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.62583.log.gz 2024-03-06T07:33:16.653 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81311.log 2024-03-06T07:33:16.653 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.73543.log: /var/log/ceph/ceph-client.admin.63251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72877.log 2024-03-06T07:33:16.653 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.73543.log.gz 2024-03-06T07:33:16.654 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.81311.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.63251.log.gz 2024-03-06T07:33:16.654 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37954.log 2024-03-06T07:33:16.654 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.72877.log: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.81311.log.gz -5 2024-03-06T07:33:16.654 INFO:teuthology.orchestra.run.smithi083.stderr: --verbose -- /var/log/ceph/ceph-client.admin.72124.log 2024-03-06T07:33:16.654 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.37954.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.72877.log.gz 2024-03-06T07:33:16.654 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67090.log 2024-03-06T07:33:16.654 INFO:teuthology.orchestra.run.smithi083.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.37954.log.gz 2024-03-06T07:33:16.654 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.72124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85751.log 2024-03-06T07:33:16.654 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.67090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74152.log 2024-03-06T07:33:16.655 INFO:teuthology.orchestra.run.smithi083.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.72124.log.gz 2024-03-06T07:33:16.655 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.85751.log: 87.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.67090.log.gz -5 2024-03-06T07:33:16.655 INFO:teuthology.orchestra.run.smithi083.stderr: --verbose -- /var/log/ceph/ceph-client.admin.83029.log 2024-03-06T07:33:16.655 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.74152.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.85751.log.gz 2024-03-06T07:33:16.655 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71612.log 2024-03-06T07:33:16.655 INFO:teuthology.orchestra.run.smithi083.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.74152.log.gz 2024-03-06T07:33:16.655 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.83029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65539.log 2024-03-06T07:33:16.655 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.71612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61072.log 2024-03-06T07:33:16.656 INFO:teuthology.orchestra.run.smithi083.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.83029.log.gz 2024-03-06T07:33:16.656 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.65539.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.71612.log.gz 2024-03-06T07:33:16.656 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.60304.log 2024-03-06T07:33:16.656 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.61072.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.65539.log.gz 2024-03-06T07:33:16.656 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67708.log 2024-03-06T07:33:16.656 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61072.log.gz 2024-03-06T07:33:16.656 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.0.60304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84493.log 2024-03-06T07:33:16.656 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.67708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55816.log 2024-03-06T07:33:16.657 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.84493.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.67708.log.gz 2024-03-06T07:33:16.657 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54919.log 2024-03-06T07:33:16.657 INFO:teuthology.orchestra.run.smithi083.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.84493.log.gz 2024-03-06T07:33:16.657 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51401.log 2024-03-06T07:33:16.657 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.55816.log: /var/log/ceph/ceph-client.admin.54919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72080.log 2024-03-06T07:33:16.657 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55816.log.gz 2024-03-06T07:33:16.657 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.51401.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54919.log.gz 2024-03-06T07:33:16.657 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48416.log 2024-03-06T07:33:16.657 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87411.log 2024-03-06T07:33:16.658 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.72080.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51401.log.gz 2024-03-06T07:33:16.658 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75878.log 2024-03-06T07:33:16.658 INFO:teuthology.orchestra.run.smithi083.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.72080.log.gz 2024-03-06T07:33:16.658 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48416.log.gz 2024-03-06T07:33:16.658 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.87411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85556.log 2024-03-06T07:33:16.658 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.75878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46103.log 2024-03-06T07:33:16.658 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.87411.log.gz 2024-03-06T07:33:16.658 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.85556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80482.log 2024-03-06T07:33:16.659 INFO:teuthology.orchestra.run.smithi083.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.75878.log.gz 2024-03-06T07:33:16.659 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.85556.log.gz 2024-03-06T07:33:16.659 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.46103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50327.log 2024-03-06T07:33:16.659 INFO:teuthology.orchestra.run.smithi083.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46103.log.gz 2024-03-06T07:33:16.659 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49403.log 2024-03-06T07:33:16.659 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.80482.log: /var/log/ceph/ceph-client.admin.50327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54744.log 2024-03-06T07:33:16.659 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49403.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.80482.log.gz 2024-03-06T07:33:16.659 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50327.log.gz 2024-03-06T07:33:16.659 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.56225.log 2024-03-06T07:33:16.660 INFO:teuthology.orchestra.run.smithi083.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49403.log.gz 2024-03-06T07:33:16.660 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62541.log 2024-03-06T07:33:16.660 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.54744.log: /var/log/ceph/ceph-client.0.56225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78059.log 2024-03-06T07:33:16.660 INFO:teuthology.orchestra.run.smithi083.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54744.log.gz 2024-03-06T07:33:16.660 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.62541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62062.log 2024-03-06T07:33:16.660 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.62541.log.gz 2024-03-06T07:33:16.660 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.78059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.73136.log 2024-03-06T07:33:16.660 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44801.log 2024-03-06T07:33:16.661 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.62062.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.78059.log.gz 2024-03-06T07:33:16.661 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.0.73136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78541.log 2024-03-06T07:33:16.661 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.62062.log.gz 2024-03-06T07:33:16.661 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77464.log 2024-03-06T07:33:16.661 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.78541.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44801.log.gz 2024-03-06T07:33:16.661 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45832.log 2024-03-06T07:33:16.661 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.77464.log: gzip -5 --verbose -- 88.8% -- replaced with /var/log/ceph/ceph-client.admin.78541.log.gz 2024-03-06T07:33:16.661 INFO:teuthology.orchestra.run.smithi083.stderr: /var/log/ceph/ceph-client.admin.49813.log 2024-03-06T07:33:16.662 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45832.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.77464.log.gz 2024-03-06T07:33:16.662 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60970.log 2024-03-06T07:33:16.662 INFO:teuthology.orchestra.run.smithi083.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45832.log.gz 2024-03-06T07:33:16.662 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58723.log 2024-03-06T07:33:16.662 INFO:teuthology.orchestra.run.smithi083.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49813.log.gz 2024-03-06T07:33:16.662 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.60970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82507.log 2024-03-06T07:33:16.662 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.58723.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60970.log.gz 2024-03-06T07:33:16.662 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60802.log 2024-03-06T07:33:16.662 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.82507.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.58723.log.gz 2024-03-06T07:33:16.663 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86765.log 2024-03-06T07:33:16.663 INFO:teuthology.orchestra.run.smithi083.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.82507.log.gz 2024-03-06T07:33:16.663 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.60802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44423.log 2024-03-06T07:33:16.663 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.86765.log: gzip 88.1% -- replaced with /var/log/ceph/ceph-client.admin.60802.log.gz 2024-03-06T07:33:16.663 INFO:teuthology.orchestra.run.smithi083.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.49655.log 2024-03-06T07:33:16.663 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44423.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.86765.log.gz 2024-03-06T07:33:16.663 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84391.log 2024-03-06T07:33:16.663 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44423.log.gz 2024-03-06T07:33:16.663 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83929.log 2024-03-06T07:33:16.664 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.84391.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49655.log.gz 2024-03-06T07:33:16.664 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43968.log 2024-03-06T07:33:16.664 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.83929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61198.log 2024-03-06T07:33:16.664 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.84391.log.gz 2024-03-06T07:33:16.664 INFO:teuthology.orchestra.run.smithi083.stderr:gzip 87.9%/var/log/ceph/ceph-client.admin.43968.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.81353.log 2024-03-06T07:33:16.664 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.83929.log.gz 2024-03-06T07:33:16.664 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.61198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73455.log 2024-03-06T07:33:16.664 INFO:teuthology.orchestra.run.smithi083.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43968.log.gz 2024-03-06T07:33:16.665 INFO:teuthology.orchestra.run.smithi083.stderr: 94.1% -- replaced with /var/log/ceph/ceph-client.0.73136.log.gz 2024-03-06T07:33:16.665 INFO:teuthology.orchestra.run.smithi083.stderr:gzip 89.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.61198.log.gz 2024-03-06T07:33:16.665 INFO:teuthology.orchestra.run.smithi083.stderr: --verbose -- /var/log/ceph/ceph-client.admin.77109.log 2024-03-06T07:33:16.665 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.81353.log: gzip -5 --verbose/var/log/ceph/ceph-client.admin.73455.log: -- /var/log/ceph/ceph-client.admin.80650.log 2024-03-06T07:33:16.665 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.77109.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.81353.log.gz 2024-03-06T07:33:16.665 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81143.log 2024-03-06T07:33:16.665 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.77109.log.gz 2024-03-06T07:33:16.665 INFO:teuthology.orchestra.run.smithi083.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.73455.log.gz 2024-03-06T07:33:16.666 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.80650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69527.log 2024-03-06T07:33:16.666 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.81143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48542.log 2024-03-06T07:33:16.666 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.80650.log.gz 2024-03-06T07:33:16.666 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.69527.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.81143.log.gzgzip 2024-03-06T07:33:16.666 INFO:teuthology.orchestra.run.smithi083.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.77235.log 2024-03-06T07:33:16.666 INFO:teuthology.orchestra.run.smithi083.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.69527.log.gz 2024-03-06T07:33:16.666 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64978.log 2024-03-06T07:33:16.666 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51738.log 2024-03-06T07:33:16.666 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.77235.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.48542.log.gz 2024-03-06T07:33:16.667 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.64978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79779.log 2024-03-06T07:33:16.667 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.77235.log.gz 2024-03-06T07:33:16.667 INFO:teuthology.orchestra.run.smithi083.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.64978.log.gz 2024-03-06T07:33:16.667 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.51738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59307.log 2024-03-06T07:33:16.667 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.79779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82717.log 2024-03-06T07:33:16.667 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51738.log.gz 2024-03-06T07:33:16.667 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1%/var/log/ceph/ceph-client.admin.59307.log: -- replaced with /var/log/ceph/ceph-client.admin.79779.log.gz 2024-03-06T07:33:16.667 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82861.log 2024-03-06T07:33:16.667 INFO:teuthology.orchestra.run.smithi083.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.59307.log.gz 2024-03-06T07:33:16.668 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.82717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76419.log 2024-03-06T07:33:16.668 INFO:teuthology.orchestra.run.smithi083.stderr:gzip/var/log/ceph/ceph-client.admin.82861.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.78240.log 2024-03-06T07:33:16.668 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.82717.log.gz 2024-03-06T07:33:16.668 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.76419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66964.log 2024-03-06T07:33:16.668 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.82861.log.gz 2024-03-06T07:33:16.668 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.76419.log.gz 2024-03-06T07:33:16.668 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.78240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59567.log 2024-03-06T07:33:16.669 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.66964.log: gzip -5 88.9% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.78240.log.gz /var/log/ceph/ceph-client.admin.68475.log 2024-03-06T07:33:16.669 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-06T07:33:16.669 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.59567.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.66964.log.gz 2024-03-06T07:33:16.669 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80902.log 2024-03-06T07:33:16.669 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59567.log.gz 2024-03-06T07:33:16.669 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.68475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52277.log 2024-03-06T07:33:16.669 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45065.log 2024-03-06T07:33:16.669 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.80902.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.68475.log.gz 2024-03-06T07:33:16.669 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.52277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79344.log 89.0% -- replaced with /var/log/ceph/ceph-client.admin.80902.log.gz 2024-03-06T07:33:16.670 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-06T07:33:16.670 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45065.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52277.log.gz 2024-03-06T07:33:16.670 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45065.log.gz 2024-03-06T07:33:16.670 INFO:teuthology.orchestra.run.smithi083.stderr: /var/log/ceph/ceph-client.admin.49771.log 2024-03-06T07:33:16.670 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.79344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71570.log 2024-03-06T07:33:16.670 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49771.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.79344.log.gz 2024-03-06T07:33:16.670 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82339.log 2024-03-06T07:33:16.670 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49771.log.gz 2024-03-06T07:33:16.670 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.71570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43706.log 2024-03-06T07:33:16.671 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.82339.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.71570.log.gz 2024-03-06T07:33:16.671 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53131.log 2024-03-06T07:33:16.671 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.82339.log.gz 2024-03-06T07:33:16.671 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.43706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64196.log 2024-03-06T07:33:16.671 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64509.log 2024-03-06T07:33:16.671 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.53131.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43706.log.gz 2024-03-06T07:33:16.671 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.64196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53993.log 2024-03-06T07:33:16.671 INFO:teuthology.orchestra.run.smithi083.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.64196.log.gz 2024-03-06T07:33:16.671 INFO:teuthology.orchestra.run.smithi083.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.53131.log.gz 2024-03-06T07:33:16.672 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.64509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66517.log 2024-03-06T07:33:16.672 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51654.log 2024-03-06T07:33:16.672 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.53993.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.64509.log.gz 2024-03-06T07:33:16.672 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.66517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64424.log 2024-03-06T07:33:16.672 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53993.log.gz 2024-03-06T07:33:16.672 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.51654.log: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58257.log 2024-03-06T07:33:16.672 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.66517.log.gz 2024-03-06T07:33:16.672 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.64424.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51654.log.gz 2024-03-06T07:33:16.673 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52403.log 2024-03-06T07:33:16.673 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37704.log 2024-03-06T07:33:16.673 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.58257.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64424.log.gz 2024-03-06T07:33:16.673 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.52403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80212.log 2024-03-06T07:33:16.673 INFO:teuthology.orchestra.run.smithi083.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.58257.log.gz 2024-03-06T07:33:16.673 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52403.log.gz 2024-03-06T07:33:16.673 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.37704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60928.log 2024-03-06T07:33:16.673 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.80212.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.37704.log.gz 2024-03-06T07:33:16.673 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47894.log 2024-03-06T07:33:16.674 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.80212.log.gz 2024-03-06T07:33:16.674 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80272.log 2024-03-06T07:33:16.674 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.60928.log: /var/log/ceph/ceph-client.admin.47894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60692.log 2024-03-06T07:33:16.674 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60928.log.gz 2024-03-06T07:33:16.674 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.80272.log: gzip 87.9% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.47894.log.gz -- 2024-03-06T07:33:16.674 INFO:teuthology.orchestra.run.smithi083.stderr: /var/log/ceph/ceph-client.admin.80566.log 2024-03-06T07:33:16.674 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.80272.log.gz 2024-03-06T07:33:16.674 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.60692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80986.log 2024-03-06T07:33:16.675 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65272.log 2024-03-06T07:33:16.675 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.80566.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.60692.log.gz 2024-03-06T07:33:16.675 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.80986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75816.log 2024-03-06T07:33:16.675 INFO:teuthology.orchestra.run.smithi083.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.80566.log.gz 2024-03-06T07:33:16.675 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.65272.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.80986.log.gz 2024-03-06T07:33:16.675 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64803.log 2024-03-06T07:33:16.675 INFO:teuthology.orchestra.run.smithi083.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.65272.log.gz 2024-03-06T07:33:16.675 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.75816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65102.log 2024-03-06T07:33:16.675 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.64803.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.75816.log.gz 2024-03-06T07:33:16.676 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57950.log 2024-03-06T07:33:16.676 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.65102.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.64803.log.gz 2024-03-06T07:33:16.676 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.52896.log 2024-03-06T07:33:16.676 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.65102.log.gz 2024-03-06T07:33:16.676 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.57950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55858.log 2024-03-06T07:33:16.676 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57950.log.gz 2024-03-06T07:33:16.676 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.0.52896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72811.log 2024-03-06T07:33:16.677 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.55858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46523.log 2024-03-06T07:33:16.677 INFO:teuthology.orchestra.run.smithi083.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55858.log.gz 2024-03-06T07:33:16.677 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.72811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70109.log 2024-03-06T07:33:16.677 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.46523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81733.log 2024-03-06T07:33:16.677 INFO:teuthology.orchestra.run.smithi083.stderr: 91.1% 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46523.log.gz 2024-03-06T07:33:16.677 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.72811.log.gz 2024-03-06T07:33:16.677 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.70109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48889.log 2024-03-06T07:33:16.677 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.70109.log.gzgzip 2024-03-06T07:33:16.677 INFO:teuthology.orchestra.run.smithi083.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.86975.log 2024-03-06T07:33:16.678 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.81733.log: /var/log/ceph/ceph-client.admin.48889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82381.log 2024-03-06T07:33:16.678 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8%/var/log/ceph/ceph-client.admin.86975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50495.log 2024-03-06T07:33:16.678 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48889.log.gz 2024-03-06T07:33:16.678 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.81733.log.gz 2024-03-06T07:33:16.678 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.82381.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.86975.log.gz 2024-03-06T07:33:16.678 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55774.log 2024-03-06T07:33:16.678 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.82381.log.gz 2024-03-06T07:33:16.679 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.50495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68391.log 2024-03-06T07:33:16.679 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50495.log.gz 2024-03-06T07:33:16.679 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.55774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80314.log 2024-03-06T07:33:16.679 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55774.log.gz 2024-03-06T07:33:16.679 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.68391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68667.log 2024-03-06T07:33:16.679 INFO:teuthology.orchestra.run.smithi083.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.68391.log.gz 2024-03-06T07:33:16.679 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.80314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86639.log 2024-03-06T07:33:16.679 INFO:teuthology.orchestra.run.smithi083.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.80314.log.gz 2024-03-06T07:33:16.680 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.68667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53951.log 2024-03-06T07:33:16.680 INFO:teuthology.orchestra.run.smithi083.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.68667.log.gz 2024-03-06T07:33:16.680 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.86639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52319.log 2024-03-06T07:33:16.680 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.86639.log.gz 2024-03-06T07:33:16.680 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.53951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82903.log 2024-03-06T07:33:16.680 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53951.log.gz 2024-03-06T07:33:16.680 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.52319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84829.log 2024-03-06T07:33:16.680 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.82903.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.52319.log.gz 2024-03-06T07:33:16.681 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44549.log 2024-03-06T07:33:16.681 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.82903.log.gz 2024-03-06T07:33:16.681 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.84829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84307.log 2024-03-06T07:33:16.681 INFO:teuthology.orchestra.run.smithi083.stderr: 93.3% -- replaced with /var/log/ceph/ceph-client.0.56225.log.gz 2024-03-06T07:33:16.681 INFO:teuthology.orchestra.run.smithi083.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.84829.log.gz 2024-03-06T07:33:16.681 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82255.log 2024-03-06T07:33:16.681 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.84307.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44549.log.gz 2024-03-06T07:33:16.681 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.84307.log.gz 2024-03-06T07:33:16.681 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64719.log 2024-03-06T07:33:16.682 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59751.log 2024-03-06T07:33:16.682 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.82255.log: /var/log/ceph/ceph-client.admin.64719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86190.log 2024-03-06T07:33:16.682 INFO:teuthology.orchestra.run.smithi083.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.82255.log.gz 2024-03-06T07:33:16.682 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.59751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46271.log 2024-03-06T07:33:16.682 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.64719.log.gz 2024-03-06T07:33:16.682 INFO:teuthology.orchestra.run.smithi083.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.59751.log.gz 2024-03-06T07:33:16.682 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59441.log 2024-03-06T07:33:16.683 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.86190.log: /var/log/ceph/ceph-client.admin.46271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75046.log 2024-03-06T07:33:16.683 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.86190.log.gz 2024-03-06T07:33:16.683 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.59441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80860.log 2024-03-06T07:33:16.683 INFO:teuthology.orchestra.run.smithi083.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46271.log.gz 2024-03-06T07:33:16.683 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.75046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84913.log 2024-03-06T07:33:16.683 INFO:teuthology.orchestra.run.smithi083.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.59441.log.gz 2024-03-06T07:33:16.684 INFO:teuthology.orchestra.run.smithi083.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.75046.log.gz 2024-03-06T07:33:16.684 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80170.log 2024-03-06T07:33:16.684 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.80860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45679.log 2024-03-06T07:33:16.684 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.84913.log: /var/log/ceph/ceph-client.admin.80170.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.80860.log.gz 2024-03-06T07:33:16.684 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 88.0% -- replaced with /var/log/ceph/ceph-client.admin.84913.log.gz 2024-03-06T07:33:16.684 INFO:teuthology.orchestra.run.smithi083.stderr: --verbose -- /var/log/ceph/ceph-client.admin.77728.log 2024-03-06T07:33:16.684 INFO:teuthology.orchestra.run.smithi083.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.80170.log.gz 2024-03-06T07:33:16.684 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75268.log 2024-03-06T07:33:16.685 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.77728.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45679.log.gz 2024-03-06T07:33:16.685 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76941.log 2024-03-06T07:33:16.685 INFO:teuthology.orchestra.run.smithi083.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.77728.log.gz 2024-03-06T07:33:16.685 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.75268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85709.log 2024-03-06T07:33:16.685 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84745.log 2024-03-06T07:33:16.685 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.76941.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.75268.log.gz 2024-03-06T07:33:16.685 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.85709.log: gzip -5 88.7% -- replaced with /var/log/ceph/ceph-client.admin.76941.log.gz 2024-03-06T07:33:16.685 INFO:teuthology.orchestra.run.smithi083.stderr: --verbose -- /var/log/ceph/ceph-client.admin.46355.log 2024-03-06T07:33:16.686 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.84745.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.85709.log.gz 2024-03-06T07:33:16.686 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- 89.0% -- replaced with /var/log/ceph/ceph-client.admin.84745.log.gz 2024-03-06T07:33:16.686 INFO:teuthology.orchestra.run.smithi083.stderr: /var/log/ceph/ceph-client.admin.50039.log 2024-03-06T07:33:16.686 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.46355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45341.log 2024-03-06T07:33:16.686 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.50039.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46355.log.gz 2024-03-06T07:33:16.686 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43614.log 2024-03-06T07:33:16.686 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45341.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50039.log.gz 2024-03-06T07:33:16.687 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61553.log 2024-03-06T07:33:16.687 INFO:teuthology.orchestra.run.smithi083.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45341.log.gz 2024-03-06T07:33:16.687 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.43614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86087.log 2024-03-06T07:33:16.687 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48230.log 2024-03-06T07:33:16.687 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.61553.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43614.log.gz 2024-03-06T07:33:16.687 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.86087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54492.log 2024-03-06T07:33:16.687 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61553.log.gz 2024-03-06T07:33:16.687 INFO:teuthology.orchestra.run.smithi083.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.86087.log.gz 2024-03-06T07:33:16.687 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-06T07:33:16.688 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48230.log: /var/log/ceph/ceph-client.admin.54492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57826.log 2024-03-06T07:33:16.688 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48230.log.gz 2024-03-06T07:33:16.688 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54492.log.gz 2024-03-06T07:33:16.688 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68349.log 2024-03-06T07:33:16.688 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44335.log 2024-03-06T07:33:16.688 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.57826.log: /var/log/ceph/ceph-client.admin.68349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44465.log 2024-03-06T07:33:16.688 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57826.log.gz 2024-03-06T07:33:16.688 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76578.log 2024-03-06T07:33:16.689 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.68349.log.gz 2024-03-06T07:33:16.689 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70233.log 2024-03-06T07:33:16.689 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44335.log.gz 2024-03-06T07:33:16.689 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44465.log: /var/log/ceph/ceph-client.admin.76578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54137.log 2024-03-06T07:33:16.689 INFO:teuthology.orchestra.run.smithi083.stderr: 89.3%/var/log/ceph/ceph-client.admin.70233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58034.log 2024-03-06T07:33:16.689 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44465.log.gz 2024-03-06T07:33:16.689 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.76578.log.gz 2024-03-06T07:33:16.689 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.54137.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.70233.log.gz 2024-03-06T07:33:16.689 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57525.log 2024-03-06T07:33:16.690 INFO:teuthology.orchestra.run.smithi083.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54137.log.gz 2024-03-06T07:33:16.690 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.58034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71029.log 2024-03-06T07:33:16.690 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.58034.log.gz 2024-03-06T07:33:16.690 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.57525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84703.log 2024-03-06T07:33:16.690 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5/var/log/ceph/ceph-client.admin.71029.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57525.log.gz 2024-03-06T07:33:16.690 INFO:teuthology.orchestra.run.smithi083.stderr: --verbose -- /var/log/ceph/ceph-client.admin.69248.log 89.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-06T07:33:16.690 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-06T07:33:16.690 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.84703.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.71029.log.gz 2024-03-06T07:33:16.691 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose 88.0% -- replaced with /var/log/ceph/ceph-client.admin.84703.log.gz 2024-03-06T07:33:16.691 INFO:teuthology.orchestra.run.smithi083.stderr: -- /var/log/ceph/ceph-client.admin.49361.log 2024-03-06T07:33:16.691 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.69248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57651.log 2024-03-06T07:33:16.691 INFO:teuthology.orchestra.run.smithi083.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.69248.log.gz 2024-03-06T07:33:16.691 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62457.log 2024-03-06T07:33:16.691 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49361.log.gz 2024-03-06T07:33:16.691 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.57651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52193.log 2024-03-06T07:33:16.691 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77067.log 2024-03-06T07:33:16.692 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.62457.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.57651.log.gz 2024-03-06T07:33:16.692 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.52193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67403.log 2024-03-06T07:33:16.692 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.62457.log.gz 2024-03-06T07:33:16.692 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.77067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54305.log 2024-03-06T07:33:16.692 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52193.log.gz 2024-03-06T07:33:16.692 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.77067.log.gz 2024-03-06T07:33:16.692 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.67403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75921.log 2024-03-06T07:33:16.692 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45059.log 2024-03-06T07:33:16.693 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.54305.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.67403.log.gz 2024-03-06T07:33:16.693 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.75921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48104.log 2024-03-06T07:33:16.693 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54305.log.gz 2024-03-06T07:33:16.693 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84013.log 2024-03-06T07:33:16.693 INFO:teuthology.orchestra.run.smithi083.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.75921.log.gz 2024-03-06T07:33:16.693 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46229.log 2024-03-06T07:33:16.693 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2%/var/log/ceph/ceph-client.admin.48104.log: -- replaced with /var/log/ceph/ceph-client.admin.45059.log.gz 2024-03-06T07:33:16.693 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.84013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56918.log 2024-03-06T07:33:16.694 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48104.log.gz 2024-03-06T07:33:16.694 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51612.log 2024-03-06T07:33:16.694 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.46229.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.84013.log.gz 2024-03-06T07:33:16.694 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.56918.log: gzip -5 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46229.log.gz 2024-03-06T07:33:16.694 INFO:teuthology.orchestra.run.smithi083.stderr: --verbose -- /var/log/ceph/ceph-client.admin.77506.log 2024-03-06T07:33:16.694 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56918.log.gz 2024-03-06T07:33:16.694 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.51612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84955.log 2024-03-06T07:33:16.694 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51612.log.gz 2024-03-06T07:33:16.694 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67613.log 2024-03-06T07:33:16.695 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.77506.log: 93.2% -- replaced with /var/log/ceph/ceph-client.0.60304.log.gz 2024-03-06T07:33:16.695 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.84955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70149.log 2024-03-06T07:33:16.695 INFO:teuthology.orchestra.run.smithi083.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.77506.log.gz 2024-03-06T07:33:16.695 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.84955.log.gz 2024-03-06T07:33:16.695 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.67613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83281.log 2024-03-06T07:33:16.695 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.67613.log.gz 2024-03-06T07:33:16.695 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.70149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51696.log 2024-03-06T07:33:16.695 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.83281.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.70149.log.gz 2024-03-06T07:33:16.695 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86003.log 2024-03-06T07:33:16.696 INFO:teuthology.orchestra.run.smithi083.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.83281.log.gz 2024-03-06T07:33:16.696 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.51696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87311.log 2024-03-06T07:33:16.696 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63533.log 2024-03-06T07:33:16.696 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.86003.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51696.log.gz 2024-03-06T07:33:16.696 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.87311.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.86003.log.gz 2024-03-06T07:33:16.696 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71528.log 2024-03-06T07:33:16.696 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53798.log 2024-03-06T07:33:16.696 INFO:teuthology.orchestra.run.smithi083.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.87311.log.gz 2024-03-06T07:33:16.697 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.63533.log: /var/log/ceph/ceph-client.admin.71528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64070.log 2024-03-06T07:33:16.697 INFO:teuthology.orchestra.run.smithi083.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.63533.log.gz 2024-03-06T07:33:16.697 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62285.log 2024-03-06T07:33:16.697 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.53798.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.71528.log.gz 2024-03-06T07:33:16.697 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.64070.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53798.log.gz 2024-03-06T07:33:16.697 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48062.log 2024-03-06T07:33:16.697 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45398.log 2024-03-06T07:33:16.697 INFO:teuthology.orchestra.run.smithi083.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.64070.log.gz 2024-03-06T07:33:16.697 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.62285.log: /var/log/ceph/ceph-client.admin.48062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75692.log 2024-03-06T07:33:16.698 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.62285.log.gz 2024-03-06T07:33:16.698 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.59150.log 2024-03-06T07:33:16.698 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45398.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48062.log.gz 2024-03-06T07:33:16.698 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.75692.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45398.log.gz 2024-03-06T07:33:16.698 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70357.log 2024-03-06T07:33:16.698 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77338.log 2024-03-06T07:33:16.698 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.75692.log.gz 2024-03-06T07:33:16.698 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.0.59150.log: /var/log/ceph/ceph-client.admin.70357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49015.log 2024-03-06T07:33:16.698 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.77338.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.70357.log.gz 2024-03-06T07:33:16.699 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84055.log 2024-03-06T07:33:16.699 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58387.log 2024-03-06T07:33:16.699 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.77338.log.gz 2024-03-06T07:33:16.699 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.84055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62775.log 2024-03-06T07:33:16.699 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49015.log.gz 2024-03-06T07:33:16.699 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.58387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73585.log 2024-03-06T07:33:16.699 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.84055.log.gz 2024-03-06T07:33:16.699 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.62775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61290.log 2024-03-06T07:33:16.700 INFO:teuthology.orchestra.run.smithi083.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.58387.log.gz 2024-03-06T07:33:16.700 INFO:teuthology.orchestra.run.smithi083.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.62775.log.gz 2024-03-06T07:33:16.700 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.73585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52445.log 2024-03-06T07:33:16.700 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.61290.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.73585.log.gz 2024-03-06T07:33:16.700 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67006.log 2024-03-06T07:33:16.700 INFO:teuthology.orchestra.run.smithi083.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.61290.log.gz 2024-03-06T07:33:16.700 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.52445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63833.log 2024-03-06T07:33:16.700 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.67006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48272.log 2024-03-06T07:33:16.700 INFO:teuthology.orchestra.run.smithi083.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.52445.log.gz 2024-03-06T07:33:16.701 INFO:teuthology.orchestra.run.smithi083.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.67006.log.gz 2024-03-06T07:33:16.701 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.63833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85472.log 2024-03-06T07:33:16.701 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48272.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.63833.log.gzgzip 2024-03-06T07:33:16.701 INFO:teuthology.orchestra.run.smithi083.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.56681.log 2024-03-06T07:33:16.701 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.85472.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48272.log.gz 2024-03-06T07:33:16.701 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70456.log 2024-03-06T07:33:16.701 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.56681.log: 88.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.85472.log.gz -5 2024-03-06T07:33:16.701 INFO:teuthology.orchestra.run.smithi083.stderr: --verbose -- /var/log/ceph/ceph-client.admin.82087.log 2024-03-06T07:33:16.702 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.70456.log: 87.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87185.log 2024-03-06T07:33:16.702 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56681.log.gz 2024-03-06T07:33:16.702 INFO:teuthology.orchestra.run.smithi083.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.70456.log.gz 2024-03-06T07:33:16.702 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.82087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44010.log 2024-03-06T07:33:16.702 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.87185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72643.log 2024-03-06T07:33:16.702 INFO:teuthology.orchestra.run.smithi083.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.82087.log.gz 2024-03-06T07:33:16.702 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78811.log 2024-03-06T07:33:16.702 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.87185.log.gz 2024-03-06T07:33:16.702 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.72643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60886.log 2024-03-06T07:33:16.703 INFO:teuthology.orchestra.run.smithi083.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44010.log.gz 2024-03-06T07:33:16.703 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.78811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85207.log 2024-03-06T07:33:16.703 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.72643.log.gz 2024-03-06T07:33:16.703 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.60886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45058.log 2024-03-06T07:33:16.703 INFO:teuthology.orchestra.run.smithi083.stderr: 89.3%/var/log/ceph/ceph-client.admin.85207.log: gzip -- replaced with /var/log/ceph/ceph-client.admin.60886.log.gz 2024-03-06T07:33:16.703 INFO:teuthology.orchestra.run.smithi083.stderr: -5 --verbose 91.1% -- -- replaced with /var/log/ceph/ceph-client.admin.78811.log.gz /var/log/ceph/ceph-client.admin.61637.log 2024-03-06T07:33:16.703 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-06T07:33:16.703 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45058.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.85207.log.gz 2024-03-06T07:33:16.703 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72433.log 2024-03-06T07:33:16.704 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45058.log.gz 2024-03-06T07:33:16.704 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.61637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86274.log 2024-03-06T07:33:16.704 INFO:teuthology.orchestra.run.smithi083.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.61637.log.gz 2024-03-06T07:33:16.704 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.72433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83719.log 2024-03-06T07:33:16.704 INFO:teuthology.orchestra.run.smithi083.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.72433.log.gz 2024-03-06T07:33:16.704 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.86274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76134.log 2024-03-06T07:33:16.704 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.83719.log: gzip 88.4% -- replaced with /var/log/ceph/ceph-client.admin.86274.log.gz 2024-03-06T07:33:16.705 INFO:teuthology.orchestra.run.smithi083.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.64551.log 2024-03-06T07:33:16.705 INFO:teuthology.orchestra.run.smithi083.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.83719.log.gz 2024-03-06T07:33:16.705 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.76134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51011.log 2024-03-06T07:33:16.705 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.64551.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.76134.log.gz 2024-03-06T07:33:16.705 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58164.log 2024-03-06T07:33:16.705 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64551.log.gz 2024-03-06T07:33:16.705 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.51011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81961.log 2024-03-06T07:33:16.705 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.58164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83197.log 2024-03-06T07:33:16.705 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51011.log.gz 2024-03-06T07:33:16.706 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.58164.log.gz 2024-03-06T07:33:16.706 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.81961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70540.log 2024-03-06T07:33:16.706 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71318.log 2024-03-06T07:33:16.706 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.81961.log.gz 2024-03-06T07:33:16.706 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.83197.log: /var/log/ceph/ceph-client.admin.70540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85267.log 2024-03-06T07:33:16.706 INFO:teuthology.orchestra.run.smithi083.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.83197.log.gz 2024-03-06T07:33:16.706 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.71318.log: 88.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69139.log 2024-03-06T07:33:16.706 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.70540.log.gz 2024-03-06T07:33:16.707 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.71318.log.gz 2024-03-06T07:33:16.707 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.85267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67174.log 2024-03-06T07:33:16.707 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86045.log 2024-03-06T07:33:16.707 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.69139.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.85267.log.gz 2024-03-06T07:33:16.707 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.67174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50968.log 2024-03-06T07:33:16.707 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.69139.log.gz 2024-03-06T07:33:16.707 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.86045.log: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51198.log 2024-03-06T07:33:16.707 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.67174.log.gz 2024-03-06T07:33:16.707 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.86045.log.gz 2024-03-06T07:33:16.707 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.50968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84097.log 2024-03-06T07:33:16.708 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73853.log 2024-03-06T07:33:16.708 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.51198.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50968.log.gz 2024-03-06T07:33:16.708 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.84097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51884.log 2024-03-06T07:33:16.708 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51198.log.gz 2024-03-06T07:33:16.708 INFO:teuthology.orchestra.run.smithi083.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.84097.log.gz 2024-03-06T07:33:16.708 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.73853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.74809.log 2024-03-06T07:33:16.708 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74153.log 2024-03-06T07:33:16.708 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.51884.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.73853.log.gz 2024-03-06T07:33:16.709 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.0.74809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74984.log 2024-03-06T07:33:16.709 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51884.log.gz 2024-03-06T07:33:16.709 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.74153.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-06T07:33:16.709 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.74984.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.74153.log.gz 2024-03-06T07:33:16.709 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64677.log 2024-03-06T07:33:16.709 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-osd.1.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.74984.log.gz 2024-03-06T07:33:16.709 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84223.log 2024-03-06T07:33:16.709 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.64677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50369.log 2024-03-06T07:33:16.709 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.84223.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.64677.log.gz 2024-03-06T07:33:16.710 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75481.log 2024-03-06T07:33:16.710 INFO:teuthology.orchestra.run.smithi083.stderr: 89.5%/var/log/ceph/ceph-client.admin.50369.log: -- replaced with /var/log/ceph/ceph-client.admin.84223.log.gz 2024-03-06T07:33:16.710 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67788.log 2024-03-06T07:33:16.710 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50369.log.gz/var/log/ceph/ceph-client.admin.75481.log: 2024-03-06T07:33:16.710 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70861.log 2024-03-06T07:33:16.710 INFO:teuthology.orchestra.run.smithi083.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.75481.log.gz 2024-03-06T07:33:16.710 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54618.log 2024-03-06T07:33:16.710 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.67788.log: /var/log/ceph/ceph-client.admin.70861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68082.log 2024-03-06T07:33:16.710 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.54618.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.67788.log.gz 2024-03-06T07:33:16.711 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.70861.log.gz 2024-03-06T07:33:16.711 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 88.7% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.54618.log.gz /var/log/ceph/ceph-client.admin.87371.log 2024-03-06T07:33:16.711 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-06T07:33:16.711 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.68082.log: -- /var/log/ceph/ceph-client.admin.58555.log 2024-03-06T07:33:16.711 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.87371.log: gzip 88.4% -- replaced with /var/log/ceph/ceph-client.admin.68082.log.gz -5 2024-03-06T07:33:16.711 INFO:teuthology.orchestra.run.smithi083.stderr: --verbose -- /var/log/ceph/ceph-client.admin.67361.log 2024-03-06T07:33:16.711 INFO:teuthology.orchestra.run.smithi083.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.87371.log.gz 2024-03-06T07:33:16.711 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.58555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54660.log 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58555.log.gz 2024-03-06T07:33:16.712 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-06T07:33:16.712 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.67361.log: 93.4% -- replaced with /var/log/ceph/ceph-client.0.74809.log.gz 2024-03-06T07:33:16.712 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48931.log 2024-03-06T07:33:16.712 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.54660.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67361.log.gz 2024-03-06T07:33:16.712 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46667.log 2024-03-06T07:33:16.712 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54660.log.gz 2024-03-06T07:33:16.712 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76072.log 2024-03-06T07:33:16.712 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48931.log: /var/log/ceph/ceph-client.admin.46667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62327.log 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48931.log.gz 2024-03-06T07:33:16.713 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-06T07:33:16.713 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46667.log.gz 2024-03-06T07:33:16.713 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.76072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78017.log 2024-03-06T07:33:16.713 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.76072.log.gz 2024-03-06T07:33:16.713 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.62327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74155.log 2024-03-06T07:33:16.713 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62327.log.gz 2024-03-06T07:33:16.713 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.78017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51053.log 2024-03-06T07:33:16.713 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.78017.log.gz 2024-03-06T07:33:16.713 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5/var/log/ceph/ceph-client.admin.74155.log: --verbose -- /var/log/ceph/ceph-client.admin.71197.log 2024-03-06T07:33:16.714 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.51053.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.74155.log.gz 2024-03-06T07:33:16.714 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72219.log 2024-03-06T07:33:16.714 INFO:teuthology.orchestra.run.smithi083.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51053.log.gz 2024-03-06T07:33:16.714 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.71197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68601.log 2024-03-06T07:33:16.714 INFO:teuthology.orchestra.run.smithi083.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.71197.log.gz/var/log/ceph/ceph-client.admin.72219.log: 2024-03-06T07:33:16.714 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86597.log 2024-03-06T07:33:16.714 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.72219.log.gz 2024-03-06T07:33:16.714 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.68601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66376.log 2024-03-06T07:33:16.715 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5/var/log/ceph/ceph-client.admin.86597.log: --verbose -- /var/log/ceph/ceph.tmp-client.admin.31630.log 2024-03-06T07:33:16.715 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.66376.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.86597.log.gz 2024-03-06T07:33:16.715 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61595.log 2024-03-06T07:33:16.715 INFO:teuthology.orchestra.run.smithi083.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.68601.log.gz 2024-03-06T07:33:16.715 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66376.log.gz 2024-03-06T07:33:16.715 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph.tmp-client.admin.31630.log: gzip 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.31630.log.gz 2024-03-06T07:33:16.715 INFO:teuthology.orchestra.run.smithi083.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.44224.log 2024-03-06T07:33:16.715 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.61595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80692.log 2024-03-06T07:33:16.716 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44224.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61595.log.gz 2024-03-06T07:33:16.716 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49115.log 2024-03-06T07:33:16.716 INFO:teuthology.orchestra.run.smithi083.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44224.log.gz 2024-03-06T07:33:16.716 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.80692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72303.log 2024-03-06T07:33:16.716 INFO:teuthology.orchestra.run.smithi083.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.80692.log.gz 2024-03-06T07:33:16.716 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5/var/log/ceph/ceph-client.admin.49115.log: --verbose -- /var/log/ceph/ceph-client.admin.76357.log 2024-03-06T07:33:16.716 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.72303.log: gzip 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49115.log.gz 2024-03-06T07:33:16.716 INFO:teuthology.orchestra.run.smithi083.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.46145.log 2024-03-06T07:33:16.717 INFO:teuthology.orchestra.run.smithi083.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.72303.log.gz 2024-03-06T07:33:16.717 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.76357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53551.log 2024-03-06T07:33:16.717 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.76357.log.gz 2024-03-06T07:33:16.717 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.46145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85123.log 2024-03-06T07:33:16.717 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46145.log.gz 2024-03-06T07:33:16.717 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.53551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71486.log 2024-03-06T07:33:16.717 INFO:teuthology.orchestra.run.smithi083.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.53551.log.gz 2024-03-06T07:33:16.717 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.85123.log: gzip -5 --verbose -- 89.0% -- replaced with /var/log/ceph/ceph-client.admin.85123.log.gz 2024-03-06T07:33:16.717 INFO:teuthology.orchestra.run.smithi083.stderr: /var/log/ceph/ceph-client.admin.86316.log 2024-03-06T07:33:16.718 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.71486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66922.log 2024-03-06T07:33:16.718 INFO:teuthology.orchestra.run.smithi083.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.71486.log.gz 2024-03-06T07:33:16.718 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.86316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60608.log 2024-03-06T07:33:16.718 INFO:teuthology.orchestra.run.smithi083.stderr: 93.3% -- replaced with /var/log/ceph/ceph-client.0.59150.log.gz 2024-03-06T07:33:16.718 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.66922.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.86316.log.gz 2024-03-06T07:33:16.718 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45342.log 2024-03-06T07:33:16.718 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.66922.log.gz 2024-03-06T07:33:16.718 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.60608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85081.log 2024-03-06T07:33:16.718 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45342.log: gzip 89.2% -- replaced with /var/log/ceph/ceph-client.admin.60608.log.gz 2024-03-06T07:33:16.719 INFO:teuthology.orchestra.run.smithi083.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.49073.log 2024-03-06T07:33:16.719 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.85081.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45342.log.gz 2024-03-06T07:33:16.719 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62709.log 2024-03-06T07:33:16.719 INFO:teuthology.orchestra.run.smithi083.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.85081.log.gz 2024-03-06T07:33:16.719 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56597.log 2024-03-06T07:33:16.719 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.62709.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.49073.log.gz 2024-03-06T07:33:16.719 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45740.log 2024-03-06T07:33:16.719 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51528.log 2024-03-06T07:33:16.719 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.56597.log: /var/log/ceph/ceph-client.admin.45740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72559.log 2024-03-06T07:33:16.720 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1% 91.2% -- replaced with /var/log/ceph/ceph-client.admin.62709.log.gz 2024-03-06T07:33:16.720 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56597.log.gz 2024-03-06T07:33:16.720 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62667.log 2024-03-06T07:33:16.720 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45740.log.gz 2024-03-06T07:33:16.720 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.51528.log: gzip/var/log/ceph/ceph-client.admin.72559.log: -5 --verbose -- 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51528.log.gz 2024-03-06T07:33:16.720 INFO:teuthology.orchestra.run.smithi083.stderr: /var/log/ceph/ceph-client.admin.87101.log 2024-03-06T07:33:16.720 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.62667.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.72559.log.gz 2024-03-06T07:33:16.720 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64238.log 2024-03-06T07:33:16.721 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62667.log.gz 2024-03-06T07:33:16.721 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.87101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64112.log 2024-03-06T07:33:16.721 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47742.log 2024-03-06T07:33:16.721 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.64238.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.87101.log.gz 2024-03-06T07:33:16.721 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.64112.log: gzip 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64238.log.gz 2024-03-06T07:33:16.721 INFO:teuthology.orchestra.run.smithi083.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.77025.log 2024-03-06T07:33:16.721 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.64112.log.gz 2024-03-06T07:33:16.721 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.47742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67996.log 2024-03-06T07:33:16.722 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.77025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78877.log 2024-03-06T07:33:16.722 INFO:teuthology.orchestra.run.smithi083.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47742.log.gz 2024-03-06T07:33:16.722 INFO:teuthology.orchestra.run.smithi083.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.77025.log.gz 2024-03-06T07:33:16.722 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.67996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49855.log 2024-03-06T07:33:16.722 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62146.log 2024-03-06T07:33:16.722 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.78877.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.67996.log.gz 2024-03-06T07:33:16.722 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50877.log 2024-03-06T07:33:16.722 INFO:teuthology.orchestra.run.smithi083.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.78877.log.gz 2024-03-06T07:33:16.722 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.62146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87269.log 2024-03-06T07:33:16.723 INFO:teuthology.orchestra.run.smithi083.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49855.log.gz 2024-03-06T07:33:16.723 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.50877.log: gzip -5 --verbose 88.9% -- replaced with /var/log/ceph/ceph-client.admin.62146.log.gz 2024-03-06T07:33:16.723 INFO:teuthology.orchestra.run.smithi083.stderr: -- /var/log/ceph/ceph-client.admin.71996.log 2024-03-06T07:33:16.723 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.87269.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50877.log.gz 2024-03-06T07:33:16.723 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64898.log 2024-03-06T07:33:16.723 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.71996.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.87269.log.gz 2024-03-06T07:33:16.723 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72349.log 2024-03-06T07:33:16.723 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.71996.log.gz 2024-03-06T07:33:16.724 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.64898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81775.log 2024-03-06T07:33:16.724 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46565.log 2024-03-06T07:33:16.724 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64898.log.gz 2024-03-06T07:33:16.724 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.72349.log: /var/log/ceph/ceph-client.admin.81775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82423.log 2024-03-06T07:33:16.724 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.72349.log.gz 2024-03-06T07:33:16.724 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.46565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56834.log 2024-03-06T07:33:16.724 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.81775.log.gz 2024-03-06T07:33:16.724 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.82423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67529.log 2024-03-06T07:33:16.724 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.56834.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.46565.log.gz 2024-03-06T07:33:16.725 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62234.log 2024-03-06T07:33:16.725 INFO:teuthology.orchestra.run.smithi083.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.82423.log.gz 2024-03-06T07:33:16.725 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.67529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45062.log 2024-03-06T07:33:16.725 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56834.log.gz 2024-03-06T07:33:16.725 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.62234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81438.log 2024-03-06T07:33:16.725 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.67529.log.gz 2024-03-06T07:33:16.725 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45062.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.62234.log.gz 2024-03-06T07:33:16.725 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59835.log 2024-03-06T07:33:16.725 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.81438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83323.log 2024-03-06T07:33:16.726 INFO:teuthology.orchestra.run.smithi083.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.45062.log.gz 2024-03-06T07:33:16.726 INFO:teuthology.orchestra.run.smithi083.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.81438.log.gz 2024-03-06T07:33:16.726 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.59835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71750.log 2024-03-06T07:33:16.726 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49571.log 2024-03-06T07:33:16.726 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.83323.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59835.log.gz 2024-03-06T07:33:16.726 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.71750.log: gzip -5 --verbose 88.7% -- /var/log/ceph/ceph-client.admin.86513.log -- replaced with /var/log/ceph/ceph-client.admin.83323.log.gz 2024-03-06T07:33:16.726 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-06T07:33:16.726 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.71750.log.gz 2024-03-06T07:33:16.727 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82777.log 2024-03-06T07:33:16.727 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49571.log: /var/log/ceph/ceph-client.admin.86513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79717.log 2024-03-06T07:33:16.727 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49571.log.gz 2024-03-06T07:33:16.727 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.82777.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.86513.log.gz 2024-03-06T07:33:16.727 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78727.log 2024-03-06T07:33:16.727 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.79717.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.82777.log.gz 2024-03-06T07:33:16.727 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50579.log 2024-03-06T07:33:16.727 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.79717.log.gz 2024-03-06T07:33:16.727 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77151.log 2024-03-06T07:33:16.728 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.78727.log: /var/log/ceph/ceph-client.admin.50579.log: 87.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84265.log 2024-03-06T07:33:16.728 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.78727.log.gz 2024-03-06T07:33:16.728 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50579.log.gz 2024-03-06T07:33:16.728 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45056.log 2024-03-06T07:33:16.728 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.77151.log: /var/log/ceph/ceph-client.admin.84265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71113.log 2024-03-06T07:33:16.728 INFO:teuthology.orchestra.run.smithi083.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.77151.log.gz 2024-03-06T07:33:16.728 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85514.log 2024-03-06T07:33:16.728 INFO:teuthology.orchestra.run.smithi083.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.84265.log.gz 2024-03-06T07:33:16.729 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.71113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53071.log 2024-03-06T07:33:16.729 INFO:teuthology.orchestra.run.smithi083.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45056.log.gz 2024-03-06T07:33:16.729 INFO:teuthology.orchestra.run.smithi083.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.71113.log.gz 2024-03-06T07:33:16.729 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.85514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73412.log 2024-03-06T07:33:16.729 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64593.log 2024-03-06T07:33:16.729 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.53071.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.85514.log.gz 2024-03-06T07:33:16.729 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.73412.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53071.log.gz 2024-03-06T07:33:16.729 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81859.log 2024-03-06T07:33:16.729 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62625.log 2024-03-06T07:33:16.730 INFO:teuthology.orchestra.run.smithi083.stderr: 87.6%/var/log/ceph/ceph-client.admin.64593.log: -- replaced with /var/log/ceph/ceph-client.admin.73412.log.gz 2024-03-06T07:33:16.730 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.81859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67132.log 2024-03-06T07:33:16.730 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.64593.log.gz 2024-03-06T07:33:16.730 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.62625.log: gzip 88.2% -5 -- replaced with /var/log/ceph/ceph-client.admin.81859.log.gz --verbose 2024-03-06T07:33:16.730 INFO:teuthology.orchestra.run.smithi083.stderr: -- /var/log/ceph/ceph-client.admin.79841.log 2024-03-06T07:33:16.730 INFO:teuthology.orchestra.run.smithi083.stderr:gzip 88.3% -- replaced with /var/log/ceph/ceph-client.admin.62625.log.gz 2024-03-06T07:33:16.730 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.67132.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.53266.log 2024-03-06T07:33:16.730 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.79841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44052.log 2024-03-06T07:33:16.730 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.67132.log.gz 2024-03-06T07:33:16.731 INFO:teuthology.orchestra.run.smithi083.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.79841.log.gz 2024-03-06T07:33:16.731 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.53266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58783.log 2024-03-06T07:33:16.731 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85793.log 2024-03-06T07:33:16.731 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44052.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53266.log.gz 2024-03-06T07:33:16.731 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.58783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62369.log 2024-03-06T07:33:16.731 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44052.log.gz 2024-03-06T07:33:16.731 INFO:teuthology.orchestra.run.smithi083.stderr:gzip 88.8% -5/var/log/ceph/ceph-client.admin.85793.log: --verbose -- replaced with /var/log/ceph/ceph-client.admin.58783.log.gz -- 2024-03-06T07:33:16.731 INFO:teuthology.orchestra.run.smithi083.stderr: /var/log/ceph/ceph-client.admin.67445.log 2024-03-06T07:33:16.732 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.62369.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.85793.log.gz 2024-03-06T07:33:16.732 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57315.log 2024-03-06T07:33:16.732 INFO:teuthology.orchestra.run.smithi083.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.62369.log.gz 2024-03-06T07:33:16.732 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72391.log 2024-03-06T07:33:16.732 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.67445.log: /var/log/ceph/ceph-client.admin.57315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54053.log 2024-03-06T07:33:16.732 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.67445.log.gz 2024-03-06T07:33:16.732 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.72391.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57315.log.gz 2024-03-06T07:33:16.732 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59349.log 2024-03-06T07:33:16.733 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.54053.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.72391.log.gz 2024-03-06T07:33:16.733 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65451.log 2024-03-06T07:33:16.733 INFO:teuthology.orchestra.run.smithi083.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.54053.log.gz 2024-03-06T07:33:16.733 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.59349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62415.log 2024-03-06T07:33:16.733 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.65451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50123.log 2024-03-06T07:33:16.733 INFO:teuthology.orchestra.run.smithi083.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.59349.log.gz 2024-03-06T07:33:16.733 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.62415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56382.log 2024-03-06T07:33:16.733 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.65451.log.gz 2024-03-06T07:33:16.733 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.50123.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.62415.log.gz 2024-03-06T07:33:16.734 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83845.log 2024-03-06T07:33:16.734 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 88.1% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.50123.log.gz 2024-03-06T07:33:16.734 INFO:teuthology.orchestra.run.smithi083.stderr: /var/log/ceph/ceph-client.admin.78625.log 2024-03-06T07:33:16.734 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.56382.log: /var/log/ceph/ceph-client.admin.83845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80128.log 2024-03-06T07:33:16.734 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56382.log.gz 2024-03-06T07:33:16.734 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.83845.log.gz 2024-03-06T07:33:16.734 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64635.log 2024-03-06T07:33:16.734 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.78625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65186.log 2024-03-06T07:33:16.734 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.80128.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.78625.log.gz 2024-03-06T07:33:16.735 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.64635.log: gzip 89.5% -- replaced with /var/log/ceph/ceph-client.admin.80128.log.gz 2024-03-06T07:33:16.735 INFO:teuthology.orchestra.run.smithi083.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.70777.log 2024-03-06T07:33:16.735 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.65186.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.64635.log.gz 2024-03-06T07:33:16.735 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84451.log 2024-03-06T07:33:16.735 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.70777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56513.log 2024-03-06T07:33:16.735 INFO:teuthology.orchestra.run.smithi083.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.65186.log.gz 2024-03-06T07:33:16.735 INFO:teuthology.orchestra.run.smithi083.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.70777.log.gz 2024-03-06T07:33:16.735 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.84451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72475.log 2024-03-06T07:33:16.736 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70945.log 2024-03-06T07:33:16.736 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.56513.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.84451.log.gz 2024-03-06T07:33:16.736 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.72475.log: gzip 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56513.log.gz 2024-03-06T07:33:16.736 INFO:teuthology.orchestra.run.smithi083.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.60566.log 2024-03-06T07:33:16.736 INFO:teuthology.orchestra.run.smithi083.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.72475.log.gz 2024-03-06T07:33:16.736 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.70945.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-06T07:33:16.736 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.60566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58825.log 2024-03-06T07:33:16.736 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.70945.log.gz 2024-03-06T07:33:16.737 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-osd.3.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.60566.log.gz 2024-03-06T07:33:16.737 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69923.log 2024-03-06T07:33:16.737 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.58825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73811.log 2024-03-06T07:33:16.737 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.69923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54450.log 88.2% -- replaced with /var/log/ceph/ceph-client.admin.69923.log.gz 2024-03-06T07:33:16.737 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-06T07:33:16.737 INFO:teuthology.orchestra.run.smithi083.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.58825.log.gz 2024-03-06T07:33:16.737 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.73811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45067.log 2024-03-06T07:33:16.737 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.73811.log.gz 2024-03-06T07:33:16.738 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.54450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82819.log 2024-03-06T07:33:16.738 INFO:teuthology.orchestra.run.smithi083.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54450.log.gz 2024-03-06T07:33:16.738 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59913.log 2024-03-06T07:33:16.738 INFO:teuthology.orchestra.run.smithi083.stderr:gzip/var/log/ceph/ceph-client.admin.82819.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.58299.log 2024-03-06T07:33:16.738 INFO:teuthology.orchestra.run.smithi083.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45067.log.gz 2024-03-06T07:33:16.738 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.59913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64154.log 2024-03-06T07:33:16.738 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.82819.log.gz 2024-03-06T07:33:16.738 INFO:teuthology.orchestra.run.smithi083.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59913.log.gzgzip 2024-03-06T07:33:16.739 INFO:teuthology.orchestra.run.smithi083.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.55308.log 2024-03-06T07:33:16.739 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.58299.log: /var/log/ceph/ceph-client.admin.64154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83971.log 2024-03-06T07:33:16.739 INFO:teuthology.orchestra.run.smithi083.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58299.log.gz 2024-03-06T07:33:16.739 INFO:teuthology.orchestra.run.smithi083.stderr: 87.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57908.log 2024-03-06T07:33:16.739 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.64154.log.gz 2024-03-06T07:33:16.739 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.55308.log: /var/log/ceph/ceph-client.admin.83971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78103.log 2024-03-06T07:33:16.739 INFO:teuthology.orchestra.run.smithi083.stderr: 89.3%/var/log/ceph/ceph-client.admin.57908.log: -- replaced with /var/log/ceph/ceph-client.admin.55308.log.gz 2024-03-06T07:33:16.739 INFO:teuthology.orchestra.run.smithi083.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.83971.log.gz 2024-03-06T07:33:16.739 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86933.log 2024-03-06T07:33:16.740 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57908.log.gz 2024-03-06T07:33:16.740 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.78103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63471.log 2024-03-06T07:33:16.740 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.86933.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.78103.log.gz 2024-03-06T07:33:16.740 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45054.log 2024-03-06T07:33:16.740 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.86933.log.gz 2024-03-06T07:33:16.740 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68038.log 2024-03-06T07:33:16.740 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.63471.log: /var/log/ceph/ceph-client.admin.45054.log: gzip -5 --verbose -- 87.6% -- replaced with /var/log/ceph/ceph-client.admin.63471.log.gz 2024-03-06T07:33:16.740 INFO:teuthology.orchestra.run.smithi083.stderr: /var/log/ceph/ceph-client.admin.83533.log 2024-03-06T07:33:16.741 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45054.log.gz 2024-03-06T07:33:16.741 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.68038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45935.log 2024-03-06T07:33:16.741 INFO:teuthology.orchestra.run.smithi083.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.68038.log.gz 2024-03-06T07:33:16.741 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.83533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51822.log 2024-03-06T07:33:16.741 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45935.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.83533.log.gz 2024-03-06T07:33:16.741 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71444.log 2024-03-06T07:33:16.741 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45935.log.gz 2024-03-06T07:33:16.741 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.51822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85961.log 2024-03-06T07:33:16.742 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51822.log.gz 2024-03-06T07:33:16.742 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.71444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76704.log 2024-03-06T07:33:16.742 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.85961.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.71444.log.gz 2024-03-06T07:33:16.742 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78198.log 2024-03-06T07:33:16.742 INFO:teuthology.orchestra.run.smithi083.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.85961.log.gz 2024-03-06T07:33:16.742 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.76704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55648.log 2024-03-06T07:33:16.742 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.76704.log.gz 2024-03-06T07:33:16.742 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.78198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63409.log 2024-03-06T07:33:16.742 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.55648.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.78198.log.gz 2024-03-06T07:33:16.743 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51780.log 2024-03-06T07:33:16.743 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55648.log.gz 2024-03-06T07:33:16.743 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.63409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51095.log 2024-03-06T07:33:16.743 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63409.log.gz 2024-03-06T07:33:16.743 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.51780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83155.log 2024-03-06T07:33:16.743 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.51095.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51780.log.gz 2024-03-06T07:33:16.743 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63944.log 2024-03-06T07:33:16.743 INFO:teuthology.orchestra.run.smithi083.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51095.log.gz 2024-03-06T07:33:16.744 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.83155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44094.log 2024-03-06T07:33:16.744 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.83155.log.gz 2024-03-06T07:33:16.744 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.63944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84181.log 2024-03-06T07:33:16.744 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44094.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.63944.log.gz 2024-03-06T07:33:16.744 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65809.log 2024-03-06T07:33:16.744 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44094.log.gz 2024-03-06T07:33:16.744 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.84181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66438.log 2024-03-06T07:33:16.744 INFO:teuthology.orchestra.run.smithi083.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.84181.log.gz 2024-03-06T07:33:16.744 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.65809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59709.log 2024-03-06T07:33:16.745 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.66438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58681.log 2024-03-06T07:33:16.745 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66438.log.gz 2024-03-06T07:33:16.745 INFO:teuthology.orchestra.run.smithi083.stderr: 90.8%/var/log/ceph/ceph-client.admin.59709.log: -- replaced with /var/log/ceph/ceph-client.admin.65809.log.gz 2024-03-06T07:33:16.745 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.66134.log 2024-03-06T07:33:16.745 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59709.log.gz 2024-03-06T07:33:16.745 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.58681.log: /var/log/ceph/ceph-client.admin.80776.log 2024-03-06T07:33:16.745 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.0.66134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74395.log 2024-03-06T07:33:16.746 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58681.log.gz 2024-03-06T07:33:16.746 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58639.log 2024-03-06T07:33:16.746 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.80776.log: /var/log/ceph/ceph-client.admin.74395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50747.log 2024-03-06T07:33:16.746 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.58639.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.74395.log.gz 88.9% -- replaced with /var/log/ceph/ceph-client.admin.80776.log.gz 2024-03-06T07:33:16.746 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-06T07:33:16.746 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58639.log.gz 2024-03-06T07:33:16.746 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59609.log 2024-03-06T07:33:16.746 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.50747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65228.log 2024-03-06T07:33:16.747 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.59609.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50747.log.gz 2024-03-06T07:33:16.747 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70047.log 2024-03-06T07:33:16.747 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59609.log.gz 2024-03-06T07:33:16.747 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.65228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61427.log 2024-03-06T07:33:16.747 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.70047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57002.log 2024-03-06T07:33:16.747 INFO:teuthology.orchestra.run.smithi083.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.65228.log.gz 2024-03-06T07:33:16.747 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.61427.log: 88.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55522.log 2024-03-06T07:33:16.747 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.70047.log.gz 2024-03-06T07:33:16.748 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.61427.log.gz 2024-03-06T07:33:16.748 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.57002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67319.log 2024-03-06T07:33:16.748 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.55522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53390.log 2024-03-06T07:33:16.748 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57002.log.gz 2024-03-06T07:33:16.748 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.67319.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55522.log.gz 2024-03-06T07:33:16.748 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64322.log 2024-03-06T07:33:16.748 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67319.log.gz 2024-03-06T07:33:16.748 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46397.log 2024-03-06T07:33:16.749 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.53390.log: /var/log/ceph/ceph-client.admin.64322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78329.log 2024-03-06T07:33:16.749 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.46397.log: 88.7% 89.2% -- replaced with /var/log/ceph/ceph-client.admin.64322.log.gz -- replaced with /var/log/ceph/ceph-client.admin.53390.log.gz 2024-03-06T07:33:16.749 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-06T07:33:16.749 INFO:teuthology.orchestra.run.smithi083.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46397.log.gz 2024-03-06T07:33:16.749 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83071.log 2024-03-06T07:33:16.749 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77548.log 2024-03-06T07:33:16.750 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.78329.log: /var/log/ceph/ceph-client.admin.83071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61385.log 2024-03-06T07:33:16.750 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.77548.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.83071.log.gz 89.1% -- replaced with /var/log/ceph/ceph-client.admin.78329.log.gz 2024-03-06T07:33:16.750 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-06T07:33:16.750 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85165.log 2024-03-06T07:33:16.750 INFO:teuthology.orchestra.run.smithi083.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.77548.log.gz 2024-03-06T07:33:16.750 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50621.log 2024-03-06T07:33:16.750 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.61385.log: /var/log/ceph/ceph-client.admin.85165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69799.log 2024-03-06T07:33:16.751 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.50621.log: 89.3% 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61385.log.gz 2024-03-06T07:33:16.751 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.85165.log.gz 2024-03-06T07:33:16.751 INFO:teuthology.orchestra.run.smithi083.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50621.log.gz 2024-03-06T07:33:16.751 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70819.log 2024-03-06T07:33:16.751 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74320.log 2024-03-06T07:33:16.751 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.69799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61816.log 2024-03-06T07:33:16.751 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.70819.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.69799.log.gz 88.8% -- replaced with /var/log/ceph/ceph-client.admin.70819.log.gz 2024-03-06T07:33:16.752 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-06T07:33:16.752 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.74320.log: gzip -5 --verbose -- 88.5% -- replaced with /var/log/ceph/ceph-client.admin.74320.log.gz 2024-03-06T07:33:16.752 INFO:teuthology.orchestra.run.smithi083.stderr: /var/log/ceph/ceph-client.admin.75754.log 2024-03-06T07:33:16.752 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.61816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65409.log 2024-03-06T07:33:16.752 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61816.log.gz 2024-03-06T07:33:16.752 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.75754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75107.log 2024-03-06T07:33:16.752 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.75754.log.gz 2024-03-06T07:33:16.752 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82675.log 2024-03-06T07:33:16.753 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.65409.log: /var/log/ceph/ceph-client.admin.75107.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.65409.log.gz 2024-03-06T07:33:16.753 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54702.log 2024-03-06T07:33:16.753 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81691.log 2024-03-06T07:33:16.753 INFO:teuthology.orchestra.run.smithi083.stderr: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.75107.log.gz 2024-03-06T07:33:16.753 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.82675.log: /var/log/ceph/ceph-client.admin.54702.log: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76899.log 2024-03-06T07:33:16.753 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.82675.log.gz 2024-03-06T07:33:16.754 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.81691.log: -- replaced with /var/log/ceph/ceph-client.admin.54702.log.gz 2024-03-06T07:33:16.754 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48314.log 2024-03-06T07:33:16.754 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.81691.log.gz 2024-03-06T07:33:16.754 INFO:teuthology.orchestra.run.smithi083.stderr: 95.5% -- replaced with /var/log/ceph/ceph-client.0.66134.log.gz 2024-03-06T07:33:16.754 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45977.log 2024-03-06T07:33:16.754 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.76899.log: /var/log/ceph/ceph-client.admin.48314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53630.log 2024-03-06T07:33:16.754 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9%/var/log/ceph/ceph-client.admin.45977.log: -- replaced with /var/log/ceph/ceph-client.admin.76899.log.gz 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48314.log.gz 2024-03-06T07:33:16.754 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-06T07:33:16.755 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45063.log 2024-03-06T07:33:16.755 INFO:teuthology.orchestra.run.smithi083.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45977.log.gz 2024-03-06T07:33:16.755 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.53630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55257.log 2024-03-06T07:33:16.755 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53630.log.gz 2024-03-06T07:33:16.755 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44180.log 2024-03-06T07:33:16.755 INFO:teuthology.orchestra.run.smithi083.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45063.log.gz 2024-03-06T07:33:16.755 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.55257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47206.log 2024-03-06T07:33:16.756 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55257.log.gz 2024-03-06T07:33:16.756 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70624.log 2024-03-06T07:33:16.756 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44180.log.gz 2024-03-06T07:33:16.756 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.47206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54221.log 2024-03-06T07:33:16.756 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47206.log.gz 2024-03-06T07:33:16.756 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.70624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50705.log 2024-03-06T07:33:16.756 INFO:teuthology.orchestra.run.smithi083.stderr:gzip/var/log/ceph/ceph-client.admin.54221.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.82591.log 2024-03-06T07:33:16.757 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.70624.log.gz 2024-03-06T07:33:16.757 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.50705.log: gzip -5 --verbose 89.2% -- replaced with /var/log/ceph/ceph-client.admin.54221.log.gz 2024-03-06T07:33:16.757 INFO:teuthology.orchestra.run.smithi083.stderr: -- /var/log/ceph/ceph-client.admin.55001.log 2024-03-06T07:33:16.757 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50705.log.gz 2024-03-06T07:33:16.757 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.82591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87227.log 2024-03-06T07:33:16.757 INFO:teuthology.orchestra.run.smithi083.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.82591.log.gz 2024-03-06T07:33:16.757 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.55001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61679.log 2024-03-06T07:33:16.758 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.87227.log: gzip 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55001.log.gz 2024-03-06T07:33:16.758 INFO:teuthology.orchestra.run.smithi083.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.84997.log 2024-03-06T07:33:16.758 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.61679.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.87227.log.gz 2024-03-06T07:33:16.758 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86723.log 2024-03-06T07:33:16.758 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61679.log.gz 2024-03-06T07:33:16.758 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.84997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72601.log 2024-03-06T07:33:16.758 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.86723.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.84997.log.gz 2024-03-06T07:33:16.759 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73669.log 2024-03-06T07:33:16.759 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.86723.log.gz 2024-03-06T07:33:16.759 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.72601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48020.log 2024-03-06T07:33:16.759 INFO:teuthology.orchestra.run.smithi083.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.72601.log.gz 2024-03-06T07:33:16.759 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83491.log 2024-03-06T07:33:16.759 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.73669.log: gzip/var/log/ceph/ceph-client.admin.48020.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.84661.log 2024-03-06T07:33:16.759 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2%/var/log/ceph/ceph-client.admin.83491.log: -- replaced with /var/log/ceph/ceph-client.admin.73669.log.gz 2024-03-06T07:33:16.759 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48020.log.gz 2024-03-06T07:33:16.759 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54347.log 2024-03-06T07:33:16.760 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.83491.log.gz/var/log/ceph/ceph-client.admin.84661.log: 2024-03-06T07:33:16.760 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85388.log 2024-03-06T07:33:16.760 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.84661.log.gz 2024-03-06T07:33:16.760 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.54347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70295.log 2024-03-06T07:33:16.760 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 88.6% --verbose -- /var/log/ceph/ceph-client.admin.80440.log 2024-03-06T07:33:16.760 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54347.log.gz/var/log/ceph/ceph-client.admin.85388.log: 2024-03-06T07:33:16.760 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.70295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61469.log 2024-03-06T07:33:16.760 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.85388.log.gz 2024-03-06T07:33:16.761 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.70295.log.gz 2024-03-06T07:33:16.761 INFO:teuthology.orchestra.run.smithi083.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.66685.log 2024-03-06T07:33:16.761 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.80440.log: /var/log/ceph/ceph-client.admin.61469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65707.log 2024-03-06T07:33:16.820 INFO:teuthology.orchestra.run.smithi083.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.80440.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77295.log 2024-03-06T07:33:16.820 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.61469.log.gz 2024-03-06T07:33:16.820 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-06T07:33:16.820 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.66685.log: /var/log/ceph/ceph-client.admin.65707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72769.log 2024-03-06T07:33:16.820 INFO:teuthology.orchestra.run.smithi083.stderr: 89.3% 88.1% -- replaced with /var/log/ceph/ceph-client.admin.65707.log.gz 2024-03-06T07:33:16.820 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.77295.log: -- replaced with /var/log/ceph/ceph-client.admin.66685.log.gz 2024-03-06T07:33:16.820 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73627.log 2024-03-06T07:33:16.821 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.77295.log.gz 2024-03-06T07:33:16.821 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.72769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56876.log 2024-03-06T07:33:16.821 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.72769.log.gz 2024-03-06T07:33:16.821 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.73627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52235.log 2024-03-06T07:33:16.821 INFO:teuthology.orchestra.run.smithi083.stderr:gzip 89.1% -- replaced with /var/log/ceph/ceph-client.admin.73627.log.gz 2024-03-06T07:33:16.821 INFO:teuthology.orchestra.run.smithi083.stderr: -5 --verbose/var/log/ceph/ceph-client.admin.56876.log: -- /var/log/ceph/ceph-client.admin.50926.log 2024-03-06T07:33:16.821 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.52235.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56876.log.gz 2024-03-06T07:33:16.821 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- 88.6% /var/log/ceph/ceph-client.admin.65144.log -- replaced with /var/log/ceph/ceph-client.admin.52235.log.gz 2024-03-06T07:33:16.822 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-06T07:33:16.822 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.50926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77193.log 2024-03-06T07:33:16.822 INFO:teuthology.orchestra.run.smithi083.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50926.log.gz 2024-03-06T07:33:16.822 INFO:teuthology.orchestra.run.smithi083.stderr:gzip/var/log/ceph/ceph-client.admin.65144.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.60650.log 2024-03-06T07:33:16.822 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.77193.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.65144.log.gz 2024-03-06T07:33:16.822 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose 87.6% -- replaced with /var/log/ceph/ceph-client.admin.77193.log.gz 2024-03-06T07:33:16.822 INFO:teuthology.orchestra.run.smithi083.stderr: -- /var/log/ceph/ceph-client.admin.46187.log 2024-03-06T07:33:16.822 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.60650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49613.log 2024-03-06T07:33:16.823 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.46187.log: 87.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.60650.log.gz 2024-03-06T07:33:16.823 INFO:teuthology.orchestra.run.smithi083.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.43883.log 2024-03-06T07:33:16.823 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70582.log 2024-03-06T07:33:16.823 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46187.log.gz 2024-03-06T07:33:16.824 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83575.log 2024-03-06T07:33:16.824 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49613.log.gz 2024-03-06T07:33:16.824 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.43883.log: /var/log/ceph/ceph-client.admin.70582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67216.log 2024-03-06T07:33:16.824 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.83575.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-06T07:33:16.824 INFO:teuthology.orchestra.run.smithi083.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.70582.log.gz 2024-03-06T07:33:16.824 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43883.log.gz 2024-03-06T07:33:16.824 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.67216.log: gzip 88.5% -- replaced with /var/log/ceph/ceph-client.admin.83575.log.gz 2024-03-06T07:33:16.825 INFO:teuthology.orchestra.run.smithi083.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.81901.log 2024-03-06T07:33:16.825 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-mgr.y.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.67216.log.gz 2024-03-06T07:33:16.825 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86145.log 2024-03-06T07:33:16.825 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.81901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65875.log 2024-03-06T07:33:16.825 INFO:teuthology.orchestra.run.smithi083.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.81901.log.gz 2024-03-06T07:33:16.825 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.86145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79655.log 2024-03-06T07:33:16.825 INFO:teuthology.orchestra.run.smithi083.stderr: 88.0%/var/log/ceph/ceph-client.admin.65875.log: -- replaced with /var/log/ceph/ceph-client.admin.86145.log.gz 2024-03-06T07:33:16.825 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.68926.log 2024-03-06T07:33:16.826 INFO:teuthology.orchestra.run.smithi083.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.65875.log.gz/var/log/ceph/ceph-client.admin.79655.log: 2024-03-06T07:33:16.826 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69247.log 2024-03-06T07:33:16.826 INFO:teuthology.orchestra.run.smithi083.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.79655.log.gz 2024-03-06T07:33:16.826 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.0.68926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73711.log 2024-03-06T07:33:16.826 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.69247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52529.log 2024-03-06T07:33:16.826 INFO:teuthology.orchestra.run.smithi083.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.69247.log.gz 2024-03-06T07:33:16.826 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.73711.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.73711.log.gz 2024-03-06T07:33:16.827 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66643.log 2024-03-06T07:33:16.827 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.52529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58597.log 2024-03-06T07:33:16.827 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52529.log.gz 2024-03-06T07:33:16.827 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.66643.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.66643.log.gz 2024-03-06T07:33:16.827 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82297.log 2024-03-06T07:33:16.827 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52109.log 2024-03-06T07:33:16.828 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.58597.log: /var/log/ceph/ceph-client.admin.82297.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.58597.log.gz 2024-03-06T07:33:16.828 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.82297.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.48500.log 2024-03-06T07:33:16.828 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-06T07:33:16.828 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.52109.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52109.log.gz 2024-03-06T07:33:16.828 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71276.log 2024-03-06T07:33:16.828 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73895.log 2024-03-06T07:33:16.828 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48500.log: /var/log/ceph/ceph-client.admin.71276.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48500.log.gz 2024-03-06T07:33:16.828 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57483.log 2024-03-06T07:33:16.829 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.71276.log.gz 2024-03-06T07:33:16.829 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.73895.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.73895.log.gz 2024-03-06T07:33:16.829 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81028.log 2024-03-06T07:33:16.829 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74154.log 2024-03-06T07:33:16.829 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.57483.log: /var/log/ceph/ceph-client.admin.81028.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.57483.log.gz 2024-03-06T07:33:16.829 INFO:teuthology.orchestra.run.smithi083.stderr: 89.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46439.log 2024-03-06T07:33:16.829 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.81028.log.gz 2024-03-06T07:33:16.829 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.74154.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.74154.log.gz 2024-03-06T07:33:16.830 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45343.log 2024-03-06T07:33:16.830 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87059.log 2024-03-06T07:33:16.830 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.46439.log: /var/log/ceph/ceph-client.admin.45343.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46439.log.gz 2024-03-06T07:33:16.830 INFO:teuthology.orchestra.run.smithi083.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45343.log.gz 2024-03-06T07:33:16.830 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87453.log 2024-03-06T07:33:16.830 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.87059.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.87059.log.gz 2024-03-06T07:33:16.830 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55127.log 2024-03-06T07:33:16.830 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60037.log 2024-03-06T07:33:16.831 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.87453.log: /var/log/ceph/ceph-client.admin.55127.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.87453.log.gz 2024-03-06T07:33:16.831 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55127.log.gz 2024-03-06T07:33:16.831 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70498.log 2024-03-06T07:33:16.831 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.60037.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60037.log.gz 2024-03-06T07:33:16.831 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55564.log 2024-03-06T07:33:16.831 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75586.log 2024-03-06T07:33:16.832 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.70498.log: /var/log/ceph/ceph-client.admin.55564.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70498.log.gz 2024-03-06T07:33:16.832 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55564.log.gz 2024-03-06T07:33:16.832 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86232.log 2024-03-06T07:33:16.832 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.75586.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.75586.log.gz 2024-03-06T07:33:16.832 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77975.log 2024-03-06T07:33:16.832 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48584.log 2024-03-06T07:33:16.832 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.86232.log: /var/log/ceph/ceph-client.admin.77975.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.86232.log.gz 2024-03-06T07:33:16.832 INFO:teuthology.orchestra.run.smithi083.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.77975.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77933.log 2024-03-06T07:33:16.833 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-06T07:33:16.833 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48584.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.48584.log.gz 2024-03-06T07:33:16.833 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50832.log 2024-03-06T07:33:16.833 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58513.log 2024-03-06T07:33:16.833 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.77933.log: /var/log/ceph/ceph-client.admin.50832.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.77933.log.gz 2024-03-06T07:33:16.833 INFO:teuthology.orchestra.run.smithi083.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50832.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78583.log 2024-03-06T07:33:16.833 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-06T07:33:16.834 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.58513.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.58513.log.gz 2024-03-06T07:33:16.834 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55690.log 2024-03-06T07:33:16.834 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54576.log 2024-03-06T07:33:16.834 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.78583.log: /var/log/ceph/ceph-client.admin.55690.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.78583.log.gz 2024-03-06T07:33:16.834 INFO:teuthology.orchestra.run.smithi083.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55690.log.gz 2024-03-06T07:33:16.834 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47700.log 2024-03-06T07:33:16.834 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.54576.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54576.log.gz 2024-03-06T07:33:16.834 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59793.log 2024-03-06T07:33:16.835 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77590.log 2024-03-06T07:33:16.835 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.47700.log: /var/log/ceph/ceph-client.admin.59793.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47700.log.gz 2024-03-06T07:33:16.835 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.59793.log.gz 2024-03-06T07:33:16.835 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72685.log 2024-03-06T07:33:16.835 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.77590.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.77590.log.gz 2024-03-06T07:33:16.835 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83761.log 2024-03-06T07:33:16.835 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83677.log 2024-03-06T07:33:16.835 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.72685.log: /var/log/ceph/ceph-client.admin.83761.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.72685.log.gz 2024-03-06T07:33:16.836 INFO:teuthology.orchestra.run.smithi083.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.83761.log.gz 2024-03-06T07:33:16.836 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75163.log 2024-03-06T07:33:16.836 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.83677.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.83677.log.gz 2024-03-06T07:33:16.836 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82465.log 2024-03-06T07:33:16.836 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80818.log 2024-03-06T07:33:16.836 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.75163.log: /var/log/ceph/ceph-client.admin.82465.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.75163.log.gz 2024-03-06T07:33:16.836 INFO:teuthology.orchestra.run.smithi083.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.82465.log.gz 2024-03-06T07:33:16.836 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86555.log 2024-03-06T07:33:16.837 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.80818.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.80818.log.gz 2024-03-06T07:33:16.837 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55043.log 2024-03-06T07:33:16.837 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78282.log 2024-03-06T07:33:16.837 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.86555.log: /var/log/ceph/ceph-client.admin.55043.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.86555.log.gz 2024-03-06T07:33:16.837 INFO:teuthology.orchestra.run.smithi083.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55043.log.gz 2024-03-06T07:33:16.837 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49199.log 2024-03-06T07:33:16.837 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.78282.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.78282.log.gz 2024-03-06T07:33:16.837 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53672.log 2024-03-06T07:33:16.838 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80398.log 2024-03-06T07:33:16.838 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49199.log: /var/log/ceph/ceph-client.admin.53672.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49199.log.gz 2024-03-06T07:33:16.838 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53672.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-06T07:33:16.838 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-06T07:33:16.838 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.80398.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.80398.log.gz 2024-03-06T07:33:16.838 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57212.log 2024-03-06T07:33:16.838 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84349.log 2024-03-06T07:33:16.838 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.57212.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.57212.log.gz 2024-03-06T07:33:16.838 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44591.log 2024-03-06T07:33:16.839 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.84349.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.84349.log.gz 2024-03-06T07:33:16.839 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.47084.log 2024-03-06T07:33:16.839 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44591.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44591.log.gz 2024-03-06T07:33:16.839 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57992.log 2024-03-06T07:33:16.839 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.0.47084.log: 90.3% -- replaced with /var/log/ceph/ceph-client.0.47084.log.gz 2024-03-06T07:33:16.839 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51979.log 2024-03-06T07:33:16.839 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.57992.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.57992.log.gz 2024-03-06T07:33:16.839 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82171.log 2024-03-06T07:33:16.840 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.51979.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51979.log.gz 2024-03-06T07:33:16.840 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57399.log 2024-03-06T07:33:16.840 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.82171.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.82171.log.gz 2024-03-06T07:33:16.840 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68517.log 2024-03-06T07:33:16.840 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.57399.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57399.log.gz 2024-03-06T07:33:16.840 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55169.log 2024-03-06T07:33:16.840 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.68517.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.68517.log.gz 2024-03-06T07:33:16.840 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79390.log 2024-03-06T07:33:16.841 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.55169.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55169.log.gz 2024-03-06T07:33:16.841 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85919.log 2024-03-06T07:33:16.841 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.79390.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.79390.log.gz 2024-03-06T07:33:16.841 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82987.log 2024-03-06T07:33:16.841 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.85919.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.85919.log.gz 2024-03-06T07:33:16.841 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72517.log 2024-03-06T07:33:16.841 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.82987.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.82987.log.gz 2024-03-06T07:33:16.842 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69985.log 2024-03-06T07:33:16.842 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.72517.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.72517.log.gz 2024-03-06T07:33:16.842 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53495.log 2024-03-06T07:33:16.842 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.69985.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.69985.log.gz 2024-03-06T07:33:16.842 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58891.log 2024-03-06T07:33:16.842 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.53495.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.53495.log.gz 2024-03-06T07:33:16.842 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65060.log 2024-03-06T07:33:16.842 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.58891.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.58891.log.gz 2024-03-06T07:33:16.843 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81817.log 2024-03-06T07:33:16.843 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.65060.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.65060.log.gz 2024-03-06T07:33:16.843 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81395.log 2024-03-06T07:33:16.843 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.81817.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.81817.log.gz 2024-03-06T07:33:16.843 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85039.log 2024-03-06T07:33:16.843 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.81395.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.81395.log.gz 2024-03-06T07:33:16.843 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51928.log 2024-03-06T07:33:16.844 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.85039.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.85039.log.gz 2024-03-06T07:33:16.844 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65367.log 2024-03-06T07:33:16.844 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.51928.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51928.log.gz 2024-03-06T07:33:16.844 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66838.log 2024-03-06T07:33:16.844 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.65367.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.65367.log.gz 2024-03-06T07:33:16.845 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83635.log 2024-03-06T07:33:16.845 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.66838.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.66838.log.gz 2024-03-06T07:33:16.845 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53204.log 2024-03-06T07:33:16.845 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.83635.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.83635.log.gz 2024-03-06T07:33:16.845 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81522.log 2024-03-06T07:33:16.845 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.53204.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53204.log.gz 2024-03-06T07:33:16.845 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51320.log 2024-03-06T07:33:16.846 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.81522.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.81522.log.gz 2024-03-06T07:33:16.846 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72261.log 2024-03-06T07:33:16.846 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.51320.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51320.log.gz 2024-03-06T07:33:16.846 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66880.log 2024-03-06T07:33:16.846 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.72261.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.72261.log.gz 2024-03-06T07:33:16.846 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56960.log 2024-03-06T07:33:16.846 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.66880.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.66880.log.gz 2024-03-06T07:33:16.846 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68307.log 2024-03-06T07:33:16.847 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.56960.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56960.log.gz 2024-03-06T07:33:16.847 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69269.log 2024-03-06T07:33:16.847 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.68307.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.68307.log.gz 2024-03-06T07:33:16.847 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48847.log 2024-03-06T07:33:16.847 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.69269.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.69269.log.gz 2024-03-06T07:33:16.847 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-06T07:33:16.847 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48847.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48847.log.gz 2024-03-06T07:33:16.847 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63191.log 2024-03-06T07:33:16.848 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph.log: 86.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-06T07:33:16.848 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75340.log 2024-03-06T07:33:16.848 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.63191.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.63191.log.gz 2024-03-06T07:33:16.848 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81185.log 2024-03-06T07:33:16.848 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.75340.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.75340.log.gz 2024-03-06T07:33:16.848 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49997.log 2024-03-06T07:33:16.848 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.81185.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.81185.log.gz 2024-03-06T07:33:16.848 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67954.log 2024-03-06T07:33:16.849 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49997.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49997.log.gz 2024-03-06T07:33:16.849 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66601.log 2024-03-06T07:33:16.849 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.67954.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.67954.log.gz 2024-03-06T07:33:16.849 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76662.log 2024-03-06T07:33:16.849 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.66601.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.66601.log.gz 2024-03-06T07:33:16.849 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67870.log 2024-03-06T07:33:16.849 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.76662.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.76662.log.gz 2024-03-06T07:33:16.850 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-06T07:33:16.850 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.67870.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.67870.log.gz 2024-03-06T07:33:16.850 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56639.log 2024-03-06T07:33:16.850 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51485.log 2024-03-06T07:33:16.851 INFO:teuthology.orchestra.run.smithi183.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-06T07:33:16.851 INFO:teuthology.orchestra.run.smithi183.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-06T07:33:16.947 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.56639.log: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53756.log 2024-03-06T07:33:16.947 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56639.log.gz 2024-03-06T07:33:16.947 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.51485.log: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84871.log 2024-03-06T07:33:16.947 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51485.log.gz 2024-03-06T07:33:16.947 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.53756.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53756.log.gz 2024-03-06T07:33:16.947 INFO:teuthology.orchestra.run.smithi083.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-06T07:33:16.947 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61896.log 2024-03-06T07:33:16.948 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.84871.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.84871.log.gz 2024-03-06T07:33:16.948 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80944.log 2024-03-06T07:33:16.948 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.61896.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61896.log.gz 2024-03-06T07:33:16.948 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76196.log 2024-03-06T07:33:16.948 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.80944.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.80944.log.gz 2024-03-06T07:33:16.948 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46313.log 2024-03-06T07:33:16.948 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.76196.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.76196.log.gz 2024-03-06T07:33:16.949 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81269.log 2024-03-06T07:33:16.949 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.46313.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46313.log.gz 2024-03-06T07:33:16.949 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82549.log 2024-03-06T07:33:16.949 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.81269.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.81269.log.gz 2024-03-06T07:33:16.949 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86891.log 2024-03-06T07:33:16.949 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.82549.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.82549.log.gz 2024-03-06T07:33:16.949 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-03-06T07:33:16.949 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.86891.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.86891.log.gz 2024-03-06T07:33:16.950 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61012.log 2024-03-06T07:33:16.950 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78147.log 2024-03-06T07:33:16.950 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.61012.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.61012.log.gz 2024-03-06T07:33:16.950 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48626.log 2024-03-06T07:33:16.950 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.78147.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.78147.log.gz 2024-03-06T07:33:16.950 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60844.log 2024-03-06T07:33:16.950 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48626.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48626.log.gz 2024-03-06T07:33:16.950 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75543.log 2024-03-06T07:33:16.950 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.60844.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.60844.log.gz 2024-03-06T07:33:16.951 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74550.log 2024-03-06T07:33:16.951 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.75543.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.75543.log.gz 2024-03-06T07:33:16.951 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63707.log 2024-03-06T07:33:16.951 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.74550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50411.log 2024-03-06T07:33:16.951 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.63707.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.63707.log.gz 2024-03-06T07:33:16.951 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.74550.log.gz 2024-03-06T07:33:16.951 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44633.log 2024-03-06T07:33:16.951 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.50411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44759.log 2024-03-06T07:33:16.952 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44633.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44633.log.gz 2024-03-06T07:33:16.952 INFO:teuthology.orchestra.run.smithi083.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50411.log.gz 2024-03-06T07:33:16.952 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43802.log 2024-03-06T07:33:16.952 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44759.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44759.log.gz 2024-03-06T07:33:16.952 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70987.log 2024-03-06T07:33:16.952 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.43802.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43802.log.gz 2024-03-06T07:33:16.952 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55606.log 2024-03-06T07:33:16.952 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.70987.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.70987.log.gz 2024-03-06T07:33:16.953 INFO:teuthology.orchestra.run.smithi083.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-03-06T07:33:16.953 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78457.log 2024-03-06T07:33:16.953 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.55606.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.55606.log.gz 2024-03-06T07:33:16.953 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66559.log 2024-03-06T07:33:16.953 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.78457.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.78457.log.gz 2024-03-06T07:33:16.953 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44377.log 2024-03-06T07:33:16.953 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.66559.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.66559.log.gz 2024-03-06T07:33:16.953 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57086.log 2024-03-06T07:33:16.953 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44377.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44377.log.gz 2024-03-06T07:33:16.953 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52487.log 2024-03-06T07:33:16.954 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.57086.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57086.log.gz 2024-03-06T07:33:16.954 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82213.log 2024-03-06T07:33:16.954 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.52487.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52487.log.gz 2024-03-06T07:33:16.954 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79593.log 2024-03-06T07:33:16.954 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.82213.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.82213.log.gz 2024-03-06T07:33:16.954 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64028.log 2024-03-06T07:33:16.954 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.79593.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.79593.log.gz 2024-03-06T07:33:16.954 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83887.log 2024-03-06T07:33:16.955 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.64028.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.64028.log.gz 2024-03-06T07:33:16.955 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58429.log 2024-03-06T07:33:16.955 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.83887.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.83887.log.gz 2024-03-06T07:33:16.955 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47978.log 2024-03-06T07:33:16.955 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.58429.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58429.log.gz 2024-03-06T07:33:16.955 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63791.log 2024-03-06T07:33:16.955 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.47978.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47978.log.gz 2024-03-06T07:33:16.955 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78685.log 2024-03-06T07:33:16.955 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.63791.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.63791.log.gz 2024-03-06T07:33:16.956 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84787.log 2024-03-06T07:33:16.956 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.78685.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.78685.log.gz 2024-03-06T07:33:16.956 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66796.log 2024-03-06T07:33:16.956 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.84787.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.84787.log.gz 2024-03-06T07:33:16.956 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45678.log 2024-03-06T07:33:16.956 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.66796.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.66796.log.gz 2024-03-06T07:33:16.956 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45677.log 2024-03-06T07:33:16.956 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45678.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45678.log.gz 2024-03-06T07:33:16.957 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52063.log 2024-03-06T07:33:16.957 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45677.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.45677.log.gz 2024-03-06T07:33:16.957 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62499.log 2024-03-06T07:33:16.957 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.52063.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52063.log.gz 2024-03-06T07:33:16.957 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82633.log 2024-03-06T07:33:16.959 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.62499.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.62499.log.gz 2024-03-06T07:33:16.959 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79432.log 2024-03-06T07:33:16.959 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.82633.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.82633.log.gz 2024-03-06T07:33:16.960 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57272.log 2024-03-06T07:33:16.960 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.79432.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.79432.log.gz 2024-03-06T07:33:16.960 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63347.log 2024-03-06T07:33:16.960 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.57272.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57272.log.gz 2024-03-06T07:33:16.960 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65665.log 2024-03-06T07:33:16.960 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.63347.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.63347.log.gz 2024-03-06T07:33:16.960 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51155.log 2024-03-06T07:33:16.960 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.65665.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.65665.log.gz 2024-03-06T07:33:16.961 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72727.log 2024-03-06T07:33:16.961 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.51155.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51155.log.gz 2024-03-06T07:33:16.961 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64280.log 2024-03-06T07:33:16.961 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.72727.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.72727.log.gz 2024-03-06T07:33:16.961 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48188.log 2024-03-06T07:33:16.961 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.64280.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.64280.log.gz 2024-03-06T07:33:16.961 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61240.log 2024-03-06T07:33:16.961 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48188.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48188.log.gz 2024-03-06T07:33:16.962 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69861.log 2024-03-06T07:33:16.962 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.61240.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.61240.log.gz 2024-03-06T07:33:16.962 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86471.log 2024-03-06T07:33:16.962 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.69861.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.69861.log.gz 2024-03-06T07:33:16.962 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57746.log 2024-03-06T07:33:16.962 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.86471.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.86471.log.gz 2024-03-06T07:33:16.962 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55732.log 2024-03-06T07:33:16.962 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.57746.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57746.log.gz 2024-03-06T07:33:16.963 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62190.log 2024-03-06T07:33:16.963 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.55732.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.55732.log.gz 2024-03-06T07:33:16.963 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59975.log 2024-03-06T07:33:16.963 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.62190.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.62190.log.gz 2024-03-06T07:33:16.963 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78373.log 2024-03-06T07:33:16.963 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.59975.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59975.log.gz 2024-03-06T07:33:16.963 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.47578.log 2024-03-06T07:33:16.963 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.78373.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.78373.log.gz 2024-03-06T07:33:16.963 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49319.log 2024-03-06T07:33:16.964 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.2.47578.log: 91.0% -- replaced with /var/log/ceph/ceph-client.2.47578.log.gz 2024-03-06T07:33:16.964 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78415.log 2024-03-06T07:33:16.964 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49319.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49319.log.gz 2024-03-06T07:33:16.964 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56792.log 2024-03-06T07:33:16.964 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.78415.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.78415.log.gz 2024-03-06T07:33:16.964 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54839.log 2024-03-06T07:33:16.964 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.56792.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56792.log.gz 2024-03-06T07:33:16.964 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49897.log 2024-03-06T07:33:16.965 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.54839.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54839.log.gz 2024-03-06T07:33:16.965 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55213.log 2024-03-06T07:33:16.965 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49897.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49897.log.gz 2024-03-06T07:33:16.965 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50789.log 2024-03-06T07:33:16.965 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.55213.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.55213.log.gz 2024-03-06T07:33:16.965 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77380.log 2024-03-06T07:33:16.965 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.50789.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50789.log.gz 2024-03-06T07:33:16.965 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67571.log 2024-03-06T07:33:16.966 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.77380.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.77380.log.gz 2024-03-06T07:33:16.966 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83803.log 2024-03-06T07:33:16.966 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.67571.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.67571.log.gz 2024-03-06T07:33:16.966 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57044.log 2024-03-06T07:33:16.966 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.83803.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.83803.log.gz 2024-03-06T07:33:16.966 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69616.log 2024-03-06T07:33:16.966 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.57044.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57044.log.gz 2024-03-06T07:33:16.966 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49157.log 2024-03-06T07:33:16.966 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.69616.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.69616.log.gz 2024-03-06T07:33:16.967 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55438.log 2024-03-06T07:33:16.967 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49157.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49157.log.gz 2024-03-06T07:33:16.967 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52361.log 2024-03-06T07:33:16.967 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.55438.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55438.log.gz 2024-03-06T07:33:16.967 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78499.log 2024-03-06T07:33:16.967 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.52361.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52361.log.gz 2024-03-06T07:33:16.967 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62104.log 2024-03-06T07:33:16.967 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.78499.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.78499.log.gz 2024-03-06T07:33:16.968 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84577.log 2024-03-06T07:33:16.968 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.62104.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62104.log.gz 2024-03-06T07:33:16.968 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76983.log 2024-03-06T07:33:16.968 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.84577.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.84577.log.gz 2024-03-06T07:33:16.968 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45339.log 2024-03-06T07:33:16.968 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.76983.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.76983.log.gz 2024-03-06T07:33:16.968 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57441.log 2024-03-06T07:33:16.968 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45339.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.45339.log.gz 2024-03-06T07:33:16.969 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60524.log 2024-03-06T07:33:16.969 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.57441.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57441.log.gz 2024-03-06T07:33:16.969 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61721.log 2024-03-06T07:33:16.969 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.60524.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.60524.log.gz 2024-03-06T07:33:16.969 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45340.log 2024-03-06T07:33:16.969 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.61721.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.61721.log.gz 2024-03-06T07:33:16.969 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64467.log 2024-03-06T07:33:16.969 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45340.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45340.log.gz 2024-03-06T07:33:16.969 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48973.log 2024-03-06T07:33:16.970 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.64467.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64467.log.gz 2024-03-06T07:33:16.970 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63665.log 2024-03-06T07:33:16.970 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.48973.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.48973.log.gz 2024-03-06T07:33:16.970 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71912.log 2024-03-06T07:33:16.970 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.63665.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.63665.log.gz 2024-03-06T07:33:16.970 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67048.log 2024-03-06T07:33:16.970 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.71912.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.71912.log.gz 2024-03-06T07:33:16.970 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44843.log 2024-03-06T07:33:16.970 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.67048.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.67048.log.gz 2024-03-06T07:33:16.971 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82945.log 2024-03-06T07:33:16.971 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.44843.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44843.log.gz 2024-03-06T07:33:16.971 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50081.log 2024-03-06T07:33:16.971 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.82945.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.82945.log.gz 2024-03-06T07:33:16.971 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50663.log 2024-03-06T07:33:16.971 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.50081.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50081.log.gz 2024-03-06T07:33:16.971 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76857.log 2024-03-06T07:33:16.971 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.50663.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50663.log.gz 2024-03-06T07:33:16.971 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57170.log 2024-03-06T07:33:16.972 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.76857.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.76857.log.gz 2024-03-06T07:33:16.972 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68559.log 2024-03-06T07:33:16.972 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.57170.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.57170.log.gz 2024-03-06T07:33:16.972 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53452.log 2024-03-06T07:33:16.972 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.68559.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.68559.log.gz 2024-03-06T07:33:16.972 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46019.log 2024-03-06T07:33:16.972 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.53452.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53452.log.gz 2024-03-06T07:33:16.972 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50285.log 2024-03-06T07:33:16.973 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.46019.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46019.log.gz 2024-03-06T07:33:16.973 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77891.log 2024-03-06T07:33:16.973 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.50285.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50285.log.gz 2024-03-06T07:33:16.973 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85667.log 2024-03-06T07:33:16.973 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.77891.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.77891.log.gz 2024-03-06T07:33:16.973 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59525.log 2024-03-06T07:33:16.973 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.85667.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.85667.log.gz 2024-03-06T07:33:16.973 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54179.log 2024-03-06T07:33:16.973 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.59525.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.59525.log.gz 2024-03-06T07:33:16.974 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61156.log 2024-03-06T07:33:16.974 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.54179.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54179.log.gz 2024-03-06T07:33:16.974 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58345.log 2024-03-06T07:33:16.974 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.61156.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61156.log.gz 2024-03-06T07:33:16.974 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77632.log 2024-03-06T07:33:16.974 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.58345.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58345.log.gz 2024-03-06T07:33:16.974 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81227.log 2024-03-06T07:33:16.974 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.77632.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.77632.log.gz 2024-03-06T07:33:16.975 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49487.log 2024-03-06T07:33:16.975 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.81227.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.81227.log.gz 2024-03-06T07:33:16.975 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59667.log 2024-03-06T07:33:16.975 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.49487.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49487.log.gz 2024-03-06T07:33:16.975 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-03-06T07:33:16.975 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.59667.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59667.log.gz 2024-03-06T07:33:16.975 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45759.log 2024-03-06T07:33:16.976 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87143.log 2024-03-06T07:33:16.976 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.45759.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45759.log.gz 2024-03-06T07:33:16.976 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68126.log 2024-03-06T07:33:16.976 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.87143.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.87143.log.gz 2024-03-06T07:33:16.976 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83365.log 2024-03-06T07:33:16.976 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.68126.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.68126.log.gz 2024-03-06T07:33:16.976 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61511.log 2024-03-06T07:33:16.977 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.83365.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.83365.log.gz 2024-03-06T07:33:16.977 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67912.log 2024-03-06T07:33:16.977 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.61511.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.61511.log.gz 2024-03-06T07:33:16.979 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68177.log 2024-03-06T07:33:16.979 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.67912.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.67912.log.gz 2024-03-06T07:33:16.979 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65767.log 2024-03-06T07:33:16.979 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.68177.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.68177.log.gz 2024-03-06T07:33:16.981 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75225.log 2024-03-06T07:33:16.981 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.65767.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.65767.log.gz 2024-03-06T07:33:16.990 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78769.log 2024-03-06T07:33:16.990 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.75225.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.75225.log.gz 2024-03-06T07:33:16.999 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46481.log 2024-03-06T07:33:16.999 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.78769.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.78769.log.gz 2024-03-06T07:33:17.015 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74033.log 2024-03-06T07:33:17.015 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.46481.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46481.log.gz 2024-03-06T07:33:17.026 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50165.log 2024-03-06T07:33:17.026 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.74033.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.74033.log.gz 2024-03-06T07:33:17.045 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69658.log 2024-03-06T07:33:17.045 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.50165.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50165.log.gz 2024-03-06T07:33:17.056 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80524.log 2024-03-06T07:33:17.056 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.69658.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.69658.log.gz 2024-03-06T07:33:17.064 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85877.log 2024-03-06T07:33:17.070 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.80524.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.80524.log.gz 2024-03-06T07:33:17.073 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71071.log 2024-03-06T07:33:17.087 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.85877.log: 88.2%/var/log/ceph/ceph-client.admin.71071.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.85877.log.gz 2024-03-06T07:33:17.096 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.71071.log.gz 2024-03-06T07:33:17.102 INFO:teuthology.orchestra.run.smithi183.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-06T07:33:17.245 INFO:teuthology.orchestra.run.smithi083.stderr: 93.9% -- replaced with /var/log/ceph/ceph-client.0.52896.log.gz 2024-03-06T07:33:17.266 INFO:teuthology.orchestra.run.smithi083.stderr: 90.4% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-06T07:33:17.312 INFO:teuthology.orchestra.run.smithi183.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-06T07:33:17.315 INFO:teuthology.orchestra.run.smithi183.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-06T07:33:17.419 INFO:teuthology.orchestra.run.smithi183.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-06T07:33:17.421 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-06T07:33:17.422 INFO:teuthology.orchestra.run.smithi183.stderr:real 0m1.121s 2024-03-06T07:33:17.422 INFO:teuthology.orchestra.run.smithi183.stderr:user 0m4.412s 2024-03-06T07:33:17.422 INFO:teuthology.orchestra.run.smithi183.stderr:sys 0m0.186s 2024-03-06T07:33:17.619 INFO:teuthology.orchestra.run.smithi083.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-06T07:33:17.759 INFO:teuthology.orchestra.run.smithi083.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-06T07:33:17.894 INFO:teuthology.orchestra.run.smithi083.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-06T07:33:17.942 INFO:teuthology.orchestra.run.smithi083.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-06T07:33:18.684 INFO:teuthology.orchestra.run.smithi083.stderr: 93.6% -- replaced with /var/log/ceph/ceph-client.0.68926.log.gz 2024-03-06T07:33:25.703 INFO:teuthology.orchestra.run.smithi083.stderr: 93.4% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-03-06T07:33:25.705 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-06T07:33:25.705 INFO:teuthology.orchestra.run.smithi083.stderr:real 0m9.430s 2024-03-06T07:33:25.705 INFO:teuthology.orchestra.run.smithi083.stderr:user 0m18.319s 2024-03-06T07:33:25.705 INFO:teuthology.orchestra.run.smithi083.stderr:sys 0m1.038s 2024-03-06T07:33:25.706 INFO:tasks.ceph:Archiving logs... 2024-03-06T07:33:25.706 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583062/remote/smithi083/log 2024-03-06T07:33:25.707 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-06T07:33:29.875 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583062/remote/smithi183/log 2024-03-06T07:33:29.876 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-06T07:33:30.528 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-06T07:33:30.537 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-06T07:33:30.537 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-06T07:33:30.540 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-06T07:33:30.618 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-06T07:33:30.626 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-06T07:33:30.627 DEBUG:teuthology.orchestra.run.smithi083:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-06T07:33:30.629 DEBUG:teuthology.orchestra.run.smithi183:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-06T07:33:30.643 INFO:teuthology.orchestra.run.smithi083.stderr:bash: line 1: ntpq: command not found 2024-03-06T07:33:30.645 INFO:teuthology.orchestra.run.smithi183.stderr:bash: line 1: ntpq: command not found 2024-03-06T07:33:30.647 INFO:teuthology.orchestra.run.smithi083.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-06T07:33:30.647 INFO:teuthology.orchestra.run.smithi083.stdout:=============================================================================== 2024-03-06T07:33:30.648 INFO:teuthology.orchestra.run.smithi083.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 46 +1845us[+1845us] +/- 99ms 2024-03-06T07:33:30.648 INFO:teuthology.orchestra.run.smithi083.stdout:^* hv02.front.sepia.ceph.com 3 6 377 48 -1051us[-1049us] +/- 53ms 2024-03-06T07:33:30.648 INFO:teuthology.orchestra.run.smithi083.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 48 +108us[ +110us] +/- 89ms 2024-03-06T07:33:30.648 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:33:30.649 INFO:teuthology.orchestra.run.smithi183.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-06T07:33:30.650 INFO:teuthology.orchestra.run.smithi183.stdout:=============================================================================== 2024-03-06T07:33:30.650 INFO:teuthology.orchestra.run.smithi183.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 49 +1840us[+1841us] +/- 99ms 2024-03-06T07:33:30.650 INFO:teuthology.orchestra.run.smithi183.stdout:^* hv02.front.sepia.ceph.com 3 6 377 48 -1048us[-1047us] +/- 53ms 2024-03-06T07:33:30.650 INFO:teuthology.orchestra.run.smithi183.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 49 +117us[ +118us] +/- 89ms 2024-03-06T07:33:30.650 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:33:30.651 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-06T07:33:30.660 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-06T07:33:30.661 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-06T07:33:30.669 DEBUG:teuthology.orchestra.run.smithi083:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-06T07:33:30.786 DEBUG:teuthology.orchestra.run.smithi183:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-06T07:33:30.861 DEBUG:teuthology.orchestra.run.smithi083:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-06T07:33:30.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:33:30.893 DEBUG:teuthology.orchestra.run.smithi183:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-06T07:33:30.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:33:30.924 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-06T07:33:30.945 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-06T07:33:30.958 INFO:teuthology.task.internal:Duration was 1151.046590 seconds 2024-03-06T07:33:30.958 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-06T07:33:30.977 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-06T07:33:30.978 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-06T07:33:30.981 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-06T07:33:31.023 INFO:teuthology.orchestra.run.smithi083.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-06T07:33:31.026 INFO:teuthology.orchestra.run.smithi183.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-06T07:33:31.191 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-06T07:33:31.192 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi083.front.sepia.ceph.com 2024-03-06T07:33:31.192 DEBUG:teuthology.orchestra.run.smithi083:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-06T07:33:31.251 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi183.front.sepia.ceph.com 2024-03-06T07:33:31.251 DEBUG:teuthology.orchestra.run.smithi183:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-06T07:33:31.278 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-06T07:33:31.279 DEBUG:teuthology.orchestra.run.smithi083:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-06T07:33:31.293 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-06T07:33:31.347 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-06T07:33:31.347 DEBUG:teuthology.orchestra.run.smithi083:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-06T07:33:31.361 DEBUG:teuthology.orchestra.run.smithi183:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-06T07:33:31.496 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-06T07:33:31.515 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-06T07:33:31.515 DEBUG:teuthology.orchestra.run.smithi083:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-06T07:33:31.542 DEBUG:teuthology.orchestra.run.smithi183:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-06T07:33:31.570 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-06T07:33:31.583 DEBUG:teuthology.orchestra.run.smithi083:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-06T07:33:31.586 DEBUG:teuthology.orchestra.run.smithi183:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-06T07:33:31.609 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern = core 2024-03-06T07:33:31.635 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern = core 2024-03-06T07:33:31.651 DEBUG:teuthology.orchestra.run.smithi083:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-06T07:33:31.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:33:31.678 DEBUG:teuthology.orchestra.run.smithi183:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-06T07:33:31.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:33:31.707 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-06T07:33:31.718 INFO:teuthology.task.internal:Transferring archived files... 2024-03-06T07:33:31.719 DEBUG:teuthology.misc:Transferring archived files from smithi083:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583062/remote/smithi083 2024-03-06T07:33:31.719 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-06T07:33:31.757 DEBUG:teuthology.misc:Transferring archived files from smithi183:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583062/remote/smithi183 2024-03-06T07:33:31.757 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-06T07:33:31.792 INFO:teuthology.task.internal:Removing archive directory... 2024-03-06T07:33:31.793 DEBUG:teuthology.orchestra.run.smithi083:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-06T07:33:31.795 DEBUG:teuthology.orchestra.run.smithi183:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-06T07:33:31.842 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-06T07:33:31.852 INFO:teuthology.task.internal:Not uploading archives. 2024-03-06T07:33:31.852 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-06T07:33:31.863 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-06T07:33:31.864 DEBUG:teuthology.orchestra.run.smithi083:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-06T07:33:31.866 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-06T07:33:31.878 INFO:teuthology.orchestra.run.smithi083.stdout: 265484 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 6 07:33 /home/ubuntu/cephtest 2024-03-06T07:33:31.896 INFO:teuthology.orchestra.run.smithi183.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 6 07:33 /home/ubuntu/cephtest 2024-03-06T07:33:31.897 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-06T07:33:31.908 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-06T07:33:31.931 INFO:teuthology.nuke:Checking targets against current locks 2024-03-06T07:33:31.960 DEBUG:teuthology.nuke:shortname: smithi083 2024-03-06T07:33:31.960 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-06T07:33:31.983 DEBUG:teuthology.nuke:shortname: smithi183 2024-03-06T07:33:31.984 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-06T07:33:31.988 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi083.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583062', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-06 07:04:41.435642', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE1mL0BgMoVtpCUXipFaslZ37m3Ygh93+0MQR210oMZBMyGW4vvH8MwfKedU06uBiOY5eGNbiCMsbp+TQAsy7ws='} 2024-03-06T07:33:32.007 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi183.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583062', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-06 07:04:41.433984', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAKTQjnjlPE3geR+z/SVB8x392UFEtbZMHxHz917JIHBT0aUi9su9UZc5NVIVWcumwM6TE0zNyML9E9hoUUTpSc='} 2024-03-06T07:33:32.035 INFO:teuthology.orchestra.console.smithi083:Power off 2024-03-06T07:33:32.036 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-06T07:33:32.059 DEBUG:teuthology.orchestra.console.smithi083:power off output: Chassis Power Control: Down/Off 2024-03-06T07:33:32.059 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:33:32.078 INFO:teuthology.orchestra.console.smithi183:Power off 2024-03-06T07:33:32.078 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-06T07:33:32.088 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is on 2024-03-06T07:33:32.101 DEBUG:teuthology.orchestra.console.smithi183:power off output: Chassis Power Control: Down/Off 2024-03-06T07:33:32.151 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:33:32.177 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is on 2024-03-06T07:33:36.089 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:33:36.178 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:33:36.200 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is on 2024-03-06T07:33:36.289 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is on 2024-03-06T07:33:40.204 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:33:40.290 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:33:40.314 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is off 2024-03-06T07:33:40.401 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is on 2024-03-06T07:33:40.416 INFO:teuthology.orchestra.console.smithi083:Power off completed 2024-03-06T07:33:44.404 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:33:44.515 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is off 2024-03-06T07:33:44.616 INFO:teuthology.orchestra.console.smithi183:Power off completed 2024-03-06T07:33:44.721 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays} duration: 1151.046590089798 failure_reason: 'Test failure: test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays)' flavor: default owner: scheduled_leonidus@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=3c11dada7ce94582beb21ab348035e72 status: fail success: false 2024-03-06T07:33:44.721 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-06T07:33:44.800 INFO:teuthology.run:FAIL